2024-09-21T13:30:47.406 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-21T13:30:47.423 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-21T13:30:47.507 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/gabrioux-2024-09-21_11:23:40-orch:cephadm-wip-guits-main-2024-09-21-0842-distro-default-smithi/7914033 branch: wip-guits-main-2024-09-21-0842 description: orch:cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root} email: gabrioux@ibm.com first_in_suite: false job_id: '7914033' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: gabrioux-2024-09-21_11:23:40-orch:cephadm-wip-guits-main-2024-09-21-0842-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-guits-main-2024-09-21-0842 ceph: conf: client.0: debug ms: 1 debug objecter: 20 debug rados: 20 global: ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug ms: 1 debug osd: 20 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - CEPHADM_STRAY_DAEMON - CEPHADM_FAILED_DAEMON - but it is still running - objects unfound and apparently lost - \(POOL_APP_NOT_ENABLED\) log-only-match: - CEPHADM_ sha1: 441784946648ff53dd310293f9dd752afeeea72a ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: root install: ceph: flavor: default sha1: 441784946648ff53dd310293f9dd752afeeea72a selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-guits-main-2024-09-21-0842 sha1: 441784946648ff53dd310293f9dd752afeeea72a owner: scheduled_gabrioux@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 5896 sha1: 441784946648ff53dd310293f9dd752afeeea72a sleep_before_teardown: 0 subset: 1/8 suite: orch:cephadm suite_branch: wip-guits-main-2024-09-21-0842 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_441784946648ff53dd310293f9dd752afeeea72a/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 441784946648ff53dd310293f9dd752afeeea72a targets: smithi049.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHp/JENmQKxa8uNEoHjdEzMayZbhpzkT10cO/uB75XaW4SxrE22JZmXjNsbssdXpop+z4/4lIbUwF1x02PY/TJ8= smithi187.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPj3wQY8LOyvgb+F+grKmSpDhfvxkQVsKKcQbGzXk52BnWzGc8H3kScMclrfRSfu3AHuOmLX86PTM4QQ7t7m1N0= tasks: - pexec: all: - sudo dnf remove nvme-cli -y - sudo dnf install runc nvmetcli nvme-cli -y - sudo sed -i 's/^#runtime = "crun"/runtime = "runc"/g' /usr/share/containers/containers.conf - sudo sed -i 's/runtime = "crun"/#runtime = "crun"/g' /usr/share/containers/containers.conf - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-21_11:23:40 tube: smithi user: gabrioux verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2902 2024-09-21T13:30:47.507 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_441784946648ff53dd310293f9dd752afeeea72a/qa; will attempt to use it 2024-09-21T13:30:47.508 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_441784946648ff53dd310293f9dd752afeeea72a/qa/tasks 2024-09-21T13:30:47.508 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-21T13:30:47.509 INFO:teuthology.task.internal:Checking packages... 2024-09-21T13:30:47.529 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '441784946648ff53dd310293f9dd752afeeea72a' 2024-09-21T13:30:47.530 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-21T13:30:47.530 INFO:teuthology.packaging:ref: None 2024-09-21T13:30:47.530 INFO:teuthology.packaging:tag: None 2024-09-21T13:30:47.530 INFO:teuthology.packaging:branch: wip-guits-main-2024-09-21-0842 2024-09-21T13:30:47.530 INFO:teuthology.packaging:sha1: 441784946648ff53dd310293f9dd752afeeea72a 2024-09-21T13:30:47.530 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-guits-main-2024-09-21-0842 2024-09-21T13:30:47.679 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-5104.g44178494 2024-09-21T13:30:47.680 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-21T13:30:47.687 INFO:teuthology.task.internal:no buildpackages task found 2024-09-21T13:30:47.687 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-21T13:30:47.693 INFO:teuthology.task.internal:Saving configuration 2024-09-21T13:30:47.706 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-21T13:30:47.718 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-21T13:30:47.744 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi049.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-09-21_11:23:40-orch:cephadm-wip-guits-main-2024-09-21-0842-distro-default-smithi/7914033', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-21 13:24:13.762670', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHp/JENmQKxa8uNEoHjdEzMayZbhpzkT10cO/uB75XaW4SxrE22JZmXjNsbssdXpop+z4/4lIbUwF1x02PY/TJ8='} 2024-09-21T13:30:47.768 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi187.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-09-21_11:23:40-orch:cephadm-wip-guits-main-2024-09-21-0842-distro-default-smithi/7914033', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-21 13:24:13.760665', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPj3wQY8LOyvgb+F+grKmSpDhfvxkQVsKKcQbGzXk52BnWzGc8H3kScMclrfRSfu3AHuOmLX86PTM4QQ7t7m1N0='} 2024-09-21T13:30:47.769 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-21T13:30:47.775 INFO:teuthology.task.internal:roles: ubuntu@smithi049.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-09-21T13:30:47.775 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-09-21T13:30:47.775 INFO:teuthology.run_tasks:Running task console_log... 2024-09-21T13:30:47.832 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fad7c29c280>, signals=[15]) 2024-09-21T13:30:47.832 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-21T13:30:47.839 INFO:teuthology.task.internal:Opening connections... 2024-09-21T13:30:47.839 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi049.front.sepia.ceph.com 2024-09-21T13:30:47.840 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T13:30:47.917 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi187.front.sepia.ceph.com 2024-09-21T13:30:47.918 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T13:30:47.995 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-21T13:30:48.004 DEBUG:teuthology.orchestra.run.smithi049:> uname -m 2024-09-21T13:30:48.023 INFO:teuthology.orchestra.run.smithi049.stdout:x86_64 2024-09-21T13:30:48.023 DEBUG:teuthology.orchestra.run.smithi049:> cat /etc/os-release 2024-09-21T13:30:48.082 INFO:teuthology.orchestra.run.smithi049.stdout:NAME="CentOS Stream" 2024-09-21T13:30:48.082 INFO:teuthology.orchestra.run.smithi049.stdout:VERSION="9" 2024-09-21T13:30:48.082 INFO:teuthology.orchestra.run.smithi049.stdout:ID="centos" 2024-09-21T13:30:48.082 INFO:teuthology.orchestra.run.smithi049.stdout:ID_LIKE="rhel fedora" 2024-09-21T13:30:48.082 INFO:teuthology.orchestra.run.smithi049.stdout:VERSION_ID="9" 2024-09-21T13:30:48.082 INFO:teuthology.orchestra.run.smithi049.stdout:PLATFORM_ID="platform:el9" 2024-09-21T13:30:48.082 INFO:teuthology.orchestra.run.smithi049.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-21T13:30:48.083 INFO:teuthology.orchestra.run.smithi049.stdout:ANSI_COLOR="0;31" 2024-09-21T13:30:48.083 INFO:teuthology.orchestra.run.smithi049.stdout:LOGO="fedora-logo-icon" 2024-09-21T13:30:48.083 INFO:teuthology.orchestra.run.smithi049.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-21T13:30:48.083 INFO:teuthology.orchestra.run.smithi049.stdout:HOME_URL="https://centos.org/" 2024-09-21T13:30:48.083 INFO:teuthology.orchestra.run.smithi049.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-21T13:30:48.083 INFO:teuthology.orchestra.run.smithi049.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-21T13:30:48.083 INFO:teuthology.orchestra.run.smithi049.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-21T13:30:48.084 INFO:teuthology.lock.ops:Updating smithi049.front.sepia.ceph.com on lock server 2024-09-21T13:30:48.108 DEBUG:teuthology.orchestra.run.smithi187:> uname -m 2024-09-21T13:30:48.126 INFO:teuthology.orchestra.run.smithi187.stdout:x86_64 2024-09-21T13:30:48.127 DEBUG:teuthology.orchestra.run.smithi187:> cat /etc/os-release 2024-09-21T13:30:48.184 INFO:teuthology.orchestra.run.smithi187.stdout:NAME="CentOS Stream" 2024-09-21T13:30:48.184 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION="9" 2024-09-21T13:30:48.184 INFO:teuthology.orchestra.run.smithi187.stdout:ID="centos" 2024-09-21T13:30:48.184 INFO:teuthology.orchestra.run.smithi187.stdout:ID_LIKE="rhel fedora" 2024-09-21T13:30:48.184 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION_ID="9" 2024-09-21T13:30:48.184 INFO:teuthology.orchestra.run.smithi187.stdout:PLATFORM_ID="platform:el9" 2024-09-21T13:30:48.184 INFO:teuthology.orchestra.run.smithi187.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-21T13:30:48.184 INFO:teuthology.orchestra.run.smithi187.stdout:ANSI_COLOR="0;31" 2024-09-21T13:30:48.185 INFO:teuthology.orchestra.run.smithi187.stdout:LOGO="fedora-logo-icon" 2024-09-21T13:30:48.185 INFO:teuthology.orchestra.run.smithi187.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-21T13:30:48.185 INFO:teuthology.orchestra.run.smithi187.stdout:HOME_URL="https://centos.org/" 2024-09-21T13:30:48.185 INFO:teuthology.orchestra.run.smithi187.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-21T13:30:48.185 INFO:teuthology.orchestra.run.smithi187.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-21T13:30:48.185 INFO:teuthology.orchestra.run.smithi187.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-21T13:30:48.185 INFO:teuthology.lock.ops:Updating smithi187.front.sepia.ceph.com on lock server 2024-09-21T13:30:48.208 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-21T13:30:48.218 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-21T13:30:48.251 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-21T13:30:48.251 DEBUG:teuthology.orchestra.run.smithi049:> test '!' -e /home/ubuntu/cephtest 2024-09-21T13:30:48.253 DEBUG:teuthology.orchestra.run.smithi187:> test '!' -e /home/ubuntu/cephtest 2024-09-21T13:30:48.270 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-21T13:30:48.276 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-21T13:30:48.276 DEBUG:teuthology.orchestra.run.smithi049:> test -z $(ls -A /var/lib/ceph) 2024-09-21T13:30:48.311 DEBUG:teuthology.orchestra.run.smithi187:> test -z $(ls -A /var/lib/ceph) 2024-09-21T13:30:48.347 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-21T13:30:48.403 INFO:teuthology.run_tasks:Running task kernel... 2024-09-21T13:30:48.417 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-21T13:30:48.417 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-21T13:30:48.418 DEBUG:teuthology.orchestra.run.smithi049:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-21T13:30:48.418 DEBUG:teuthology.orchestra.run.smithi187:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-21T13:30:48.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:30:48.435 DEBUG:teuthology.orchestra.run.smithi187:> uname -r 2024-09-21T13:30:48.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:30:48.436 DEBUG:teuthology.orchestra.run.smithi049:> uname -r 2024-09-21T13:30:48.492 INFO:teuthology.orchestra.run.smithi187.stdout:5.14.0-503.el9.x86_64 2024-09-21T13:30:48.492 INFO:teuthology.task.kernel:Running kernel on smithi187: 5.14.0-503.el9.x86_64 2024-09-21T13:30:48.492 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum install -y kernel 2024-09-21T13:30:48.495 INFO:teuthology.orchestra.run.smithi049.stdout:5.14.0-503.el9.x86_64 2024-09-21T13:30:48.495 INFO:teuthology.task.kernel:Running kernel on smithi049: 5.14.0-503.el9.x86_64 2024-09-21T13:30:48.496 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum install -y kernel 2024-09-21T13:30:50.820 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.3 MB 00:00 2024-09-21T13:30:52.584 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - AppStream 30 MB/s | 20 MB 00:00 2024-09-21T13:30:52.969 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - BaseOS 3.0 MB/s | 8.3 MB 00:02 2024-09-21T13:30:56.061 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - AppStream 11 MB/s | 20 MB 00:01 2024-09-21T13:30:57.060 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - CRB 9.2 MB/s | 6.5 MB 00:00 2024-09-21T13:30:59.136 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 19 kB 00:00 2024-09-21T13:31:00.081 INFO:teuthology.orchestra.run.smithi187.stdout:Extra Packages for Enterprise Linux 28 MB/s | 23 MB 00:00 2024-09-21T13:31:03.778 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - CRB 1.8 MB/s | 6.5 MB 00:03 2024-09-21T13:31:05.362 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - Extras packages 71 kB/s | 19 kB 00:00 2024-09-21T13:31:05.537 INFO:teuthology.orchestra.run.smithi187.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-21T13:31:06.317 INFO:teuthology.orchestra.run.smithi049.stdout:Extra Packages for Enterprise Linux 27 MB/s | 23 MB 00:00 2024-09-21T13:31:07.427 INFO:teuthology.orchestra.run.smithi187.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-21T13:31:07.427 INFO:teuthology.orchestra.run.smithi187.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-21T13:31:07.530 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-21T13:31:07.535 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-21T13:31:07.535 INFO:teuthology.orchestra.run.smithi187.stdout: Package Architecture Version Repository Size 2024-09-21T13:31:07.535 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-21T13:31:07.535 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-09-21T13:31:07.535 INFO:teuthology.orchestra.run.smithi187.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-21T13:31:07.536 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2024-09-21T13:31:07.536 INFO:teuthology.orchestra.run.smithi187.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-21T13:31:07.536 INFO:teuthology.orchestra.run.smithi187.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-21T13:31:07.536 INFO:teuthology.orchestra.run.smithi187.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-21T13:31:07.536 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:31:07.536 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-09-21T13:31:07.536 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-21T13:31:07.536 INFO:teuthology.orchestra.run.smithi187.stdout:Install 4 Packages 2024-09-21T13:31:07.536 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:31:07.537 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 79 M 2024-09-21T13:31:07.537 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 126 M 2024-09-21T13:31:07.537 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-09-21T13:31:07.630 INFO:teuthology.orchestra.run.smithi187.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 501 kB/s | 36 kB 00:00 2024-09-21T13:31:08.347 INFO:teuthology.orchestra.run.smithi187.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 20 MB/s | 16 MB 00:00 2024-09-21T13:31:08.664 INFO:teuthology.orchestra.run.smithi187.stdout:(3/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 31 MB/s | 35 MB 00:01 2024-09-21T13:31:09.064 INFO:teuthology.orchestra.run.smithi187.stdout:(4/4): kernel-modules-core-5.14.0-508.el9.x86_6 20 MB/s | 29 MB 00:01 2024-09-21T13:31:09.064 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-09-21T13:31:09.064 INFO:teuthology.orchestra.run.smithi187.stdout:Total 52 MB/s | 79 MB 00:01 2024-09-21T13:31:09.748 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-09-21T13:31:10.111 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-09-21T13:31:10.111 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-09-21T13:31:11.095 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-09-21T13:31:11.095 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-09-21T13:31:11.990 INFO:teuthology.orchestra.run.smithi049.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-09-21T13:31:13.511 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-09-21T13:31:13.995 INFO:teuthology.orchestra.run.smithi049.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-21T13:31:13.995 INFO:teuthology.orchestra.run.smithi049.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-21T13:31:14.093 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-09-21T13:31:14.103 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-09-21T13:31:14.103 INFO:teuthology.orchestra.run.smithi049.stdout: Package Architecture Version Repository Size 2024-09-21T13:31:14.103 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-09-21T13:31:14.103 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-09-21T13:31:14.103 INFO:teuthology.orchestra.run.smithi049.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-21T13:31:14.104 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2024-09-21T13:31:14.104 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-21T13:31:14.104 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-21T13:31:14.104 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-21T13:31:14.104 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:31:14.104 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-09-21T13:31:14.104 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-09-21T13:31:14.104 INFO:teuthology.orchestra.run.smithi049.stdout:Install 4 Packages 2024-09-21T13:31:14.104 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:31:14.105 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 79 M 2024-09-21T13:31:14.105 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 126 M 2024-09-21T13:31:14.105 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-09-21T13:31:14.155 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-21T13:31:14.231 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T13:31:14.332 INFO:teuthology.orchestra.run.smithi049.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 401 kB/s | 36 kB 00:00 2024-09-21T13:31:14.790 INFO:teuthology.orchestra.run.smithi049.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 29 MB/s | 16 MB 00:00 2024-09-21T13:31:15.341 INFO:teuthology.orchestra.run.smithi049.stdout:(3/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 31 MB/s | 35 MB 00:01 2024-09-21T13:31:15.766 INFO:teuthology.orchestra.run.smithi049.stdout:(4/4): kernel-modules-core-5.14.0-508.el9.x86_6 20 MB/s | 29 MB 00:01 2024-09-21T13:31:15.766 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-09-21T13:31:15.766 INFO:teuthology.orchestra.run.smithi049.stdout:Total 47 MB/s | 79 MB 00:01 2024-09-21T13:31:16.480 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-09-21T13:31:16.918 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-09-21T13:31:16.918 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-09-21T13:31:16.980 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T13:31:17.107 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T13:31:17.914 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-09-21T13:31:17.914 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-09-21T13:31:20.490 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-09-21T13:31:21.155 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-21T13:31:21.242 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T13:31:21.921 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T13:31:22.007 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T13:31:24.143 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T13:31:24.270 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T13:31:26.847 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T13:31:29.485 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T13:31:29.529 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T13:31:34.763 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T13:32:05.180 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T13:32:05.224 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-21T13:32:05.581 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T13:32:05.581 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-21T13:32:05.581 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T13:32:05.581 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T13:32:06.467 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T13:32:06.467 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:32:06.467 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-09-21T13:32:06.467 INFO:teuthology.orchestra.run.smithi187.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-21T13:32:06.467 INFO:teuthology.orchestra.run.smithi187.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-21T13:32:06.467 INFO:teuthology.orchestra.run.smithi187.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-21T13:32:06.467 INFO:teuthology.orchestra.run.smithi187.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-21T13:32:06.467 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:32:06.467 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-21T13:32:06.797 DEBUG:teuthology.orchestra.run.smithi187:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T13:32:06.886 INFO:teuthology.orchestra.run.smithi187.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T13:32:06.886 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T13:32:06.887 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-21T13:32:06.887 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-21T13:32:06.887 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-21T13:32:06.887 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-21T13:32:06.887 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi187.front.sepia.ceph.com, path=None, version=distro) 2024-09-21T13:32:06.887 DEBUG:teuthology.orchestra.run.smithi187:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T13:32:06.947 INFO:teuthology.orchestra.run.smithi187.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T13:32:06.947 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T13:32:06.947 DEBUG:teuthology.orchestra.run.smithi187:> sudo rpm -qi grub2-tools 2024-09-21T13:32:06.987 INFO:teuthology.orchestra.run.smithi187.stdout:Name : grub2-tools 2024-09-21T13:32:06.988 INFO:teuthology.orchestra.run.smithi187.stdout:Epoch : 1 2024-09-21T13:32:06.988 INFO:teuthology.orchestra.run.smithi187.stdout:Version : 2.06 2024-09-21T13:32:06.988 INFO:teuthology.orchestra.run.smithi187.stdout:Release : 82.el9 2024-09-21T13:32:06.988 INFO:teuthology.orchestra.run.smithi187.stdout:Architecture: x86_64 2024-09-21T13:32:06.988 INFO:teuthology.orchestra.run.smithi187.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-21T13:32:06.988 INFO:teuthology.orchestra.run.smithi187.stdout:Group : Unspecified 2024-09-21T13:32:06.988 INFO:teuthology.orchestra.run.smithi187.stdout:Size : 8274670 2024-09-21T13:32:06.988 INFO:teuthology.orchestra.run.smithi187.stdout:License : GPLv3+ 2024-09-21T13:32:06.988 INFO:teuthology.orchestra.run.smithi187.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-21T13:32:06.988 INFO:teuthology.orchestra.run.smithi187.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-21T13:32:06.988 INFO:teuthology.orchestra.run.smithi187.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-21T13:32:06.989 INFO:teuthology.orchestra.run.smithi187.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-21T13:32:06.989 INFO:teuthology.orchestra.run.smithi187.stdout:Packager : builder@centos.org 2024-09-21T13:32:06.989 INFO:teuthology.orchestra.run.smithi187.stdout:Vendor : CentOS 2024-09-21T13:32:06.989 INFO:teuthology.orchestra.run.smithi187.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-21T13:32:06.989 INFO:teuthology.orchestra.run.smithi187.stdout:Summary : Support tools for GRUB. 2024-09-21T13:32:06.989 INFO:teuthology.orchestra.run.smithi187.stdout:Description : 2024-09-21T13:32:06.989 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:32:06.989 INFO:teuthology.orchestra.run.smithi187.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-21T13:32:06.989 INFO:teuthology.orchestra.run.smithi187.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-21T13:32:06.989 INFO:teuthology.orchestra.run.smithi187.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-21T13:32:06.989 INFO:teuthology.orchestra.run.smithi187.stdout:hardware devices. 2024-09-21T13:32:06.989 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:32:06.989 INFO:teuthology.orchestra.run.smithi187.stdout:This subpackage provides tools for support of all platforms. 2024-09-21T13:32:06.990 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-21T13:32:06.990 INFO:teuthology.task.kernel:Updating grub on smithi187 to boot 5.14.0-508.el9.x86_64 2024-09-21T13:32:06.990 DEBUG:teuthology.orchestra.run.smithi187:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-21T13:32:07.737 INFO:teuthology.orchestra.run.smithi187.stderr:Generating grub configuration file ... 2024-09-21T13:32:09.076 INFO:teuthology.orchestra.run.smithi187.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-21T13:32:09.106 INFO:teuthology.orchestra.run.smithi187.stderr:done 2024-09-21T13:32:09.108 DEBUG:teuthology.orchestra.run.smithi187:> mktemp 2024-09-21T13:32:09.164 INFO:teuthology.orchestra.run.smithi187.stdout:/tmp/tmp.NNyYrHw3k7 2024-09-21T13:32:09.165 DEBUG:teuthology.orchestra.run.smithi187:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.NNyYrHw3k7 2024-09-21T13:32:09.241 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod 0666 /tmp/tmp.NNyYrHw3k7 2024-09-21T13:32:09.376 DEBUG:teuthology.orchestra.remote:smithi187:/tmp/tmp.NNyYrHw3k7 is 6KB 2024-09-21T13:32:09.423 DEBUG:teuthology.orchestra.run.smithi187:> rm -fr /tmp/tmp.NNyYrHw3k7 2024-09-21T13:32:09.438 DEBUG:teuthology.orchestra.run.smithi187:> sudo /bin/ls /boot/loader/entries || true 2024-09-21T13:32:09.499 INFO:teuthology.orchestra.run.smithi187.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-21T13:32:09.500 INFO:teuthology.orchestra.run.smithi187.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-21T13:32:09.500 INFO:teuthology.orchestra.run.smithi187.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-21T13:32:09.500 INFO:teuthology.orchestra.run.smithi187.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-21T13:32:09.501 DEBUG:teuthology.orchestra.run.smithi187:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-21T13:32:09.635 DEBUG:teuthology.orchestra.run.smithi187:> sudo shutdown -r now 2024-09-21T13:32:14.843 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T13:32:14.885 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-21T13:32:15.279 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T13:32:15.279 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-21T13:32:15.279 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T13:32:15.279 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T13:32:16.144 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T13:32:16.144 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:32:16.144 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-09-21T13:32:16.144 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-21T13:32:16.144 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-21T13:32:16.144 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-21T13:32:16.145 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-21T13:32:16.145 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:32:16.145 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-09-21T13:32:16.516 DEBUG:teuthology.orchestra.run.smithi049:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T13:32:16.594 INFO:teuthology.orchestra.run.smithi049.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T13:32:16.595 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T13:32:16.595 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-21T13:32:16.595 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-21T13:32:16.595 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-21T13:32:16.595 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-21T13:32:16.595 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi049.front.sepia.ceph.com, path=None, version=distro) 2024-09-21T13:32:16.595 DEBUG:teuthology.orchestra.run.smithi049:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T13:32:16.655 INFO:teuthology.orchestra.run.smithi049.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T13:32:16.656 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T13:32:16.656 DEBUG:teuthology.orchestra.run.smithi049:> sudo rpm -qi grub2-tools 2024-09-21T13:32:16.695 INFO:teuthology.orchestra.run.smithi049.stdout:Name : grub2-tools 2024-09-21T13:32:16.695 INFO:teuthology.orchestra.run.smithi049.stdout:Epoch : 1 2024-09-21T13:32:16.695 INFO:teuthology.orchestra.run.smithi049.stdout:Version : 2.06 2024-09-21T13:32:16.695 INFO:teuthology.orchestra.run.smithi049.stdout:Release : 82.el9 2024-09-21T13:32:16.695 INFO:teuthology.orchestra.run.smithi049.stdout:Architecture: x86_64 2024-09-21T13:32:16.695 INFO:teuthology.orchestra.run.smithi049.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-21T13:32:16.695 INFO:teuthology.orchestra.run.smithi049.stdout:Group : Unspecified 2024-09-21T13:32:16.695 INFO:teuthology.orchestra.run.smithi049.stdout:Size : 8274670 2024-09-21T13:32:16.695 INFO:teuthology.orchestra.run.smithi049.stdout:License : GPLv3+ 2024-09-21T13:32:16.695 INFO:teuthology.orchestra.run.smithi049.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-21T13:32:16.695 INFO:teuthology.orchestra.run.smithi049.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-21T13:32:16.695 INFO:teuthology.orchestra.run.smithi049.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-21T13:32:16.696 INFO:teuthology.orchestra.run.smithi049.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-21T13:32:16.696 INFO:teuthology.orchestra.run.smithi049.stdout:Packager : builder@centos.org 2024-09-21T13:32:16.696 INFO:teuthology.orchestra.run.smithi049.stdout:Vendor : CentOS 2024-09-21T13:32:16.696 INFO:teuthology.orchestra.run.smithi049.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-21T13:32:16.696 INFO:teuthology.orchestra.run.smithi049.stdout:Summary : Support tools for GRUB. 2024-09-21T13:32:16.696 INFO:teuthology.orchestra.run.smithi049.stdout:Description : 2024-09-21T13:32:16.696 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:32:16.696 INFO:teuthology.orchestra.run.smithi049.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-21T13:32:16.696 INFO:teuthology.orchestra.run.smithi049.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-21T13:32:16.696 INFO:teuthology.orchestra.run.smithi049.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-21T13:32:16.696 INFO:teuthology.orchestra.run.smithi049.stdout:hardware devices. 2024-09-21T13:32:16.696 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:32:16.696 INFO:teuthology.orchestra.run.smithi049.stdout:This subpackage provides tools for support of all platforms. 2024-09-21T13:32:16.698 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-21T13:32:16.698 INFO:teuthology.task.kernel:Updating grub on smithi049 to boot 5.14.0-508.el9.x86_64 2024-09-21T13:32:16.698 DEBUG:teuthology.orchestra.run.smithi049:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-21T13:32:17.415 INFO:teuthology.orchestra.run.smithi049.stderr:Generating grub configuration file ... 2024-09-21T13:32:18.771 INFO:teuthology.orchestra.run.smithi049.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-21T13:32:18.810 INFO:teuthology.orchestra.run.smithi049.stderr:done 2024-09-21T13:32:18.811 DEBUG:teuthology.orchestra.run.smithi049:> mktemp 2024-09-21T13:32:18.827 INFO:teuthology.orchestra.run.smithi049.stdout:/tmp/tmp.ovbLzG3dzV 2024-09-21T13:32:18.827 DEBUG:teuthology.orchestra.run.smithi049:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ovbLzG3dzV 2024-09-21T13:32:18.895 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 0666 /tmp/tmp.ovbLzG3dzV 2024-09-21T13:32:19.030 DEBUG:teuthology.orchestra.remote:smithi049:/tmp/tmp.ovbLzG3dzV is 6KB 2024-09-21T13:32:19.050 DEBUG:teuthology.orchestra.run.smithi049:> rm -fr /tmp/tmp.ovbLzG3dzV 2024-09-21T13:32:19.066 DEBUG:teuthology.orchestra.run.smithi049:> sudo /bin/ls /boot/loader/entries || true 2024-09-21T13:32:19.133 INFO:teuthology.orchestra.run.smithi049.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-21T13:32:19.133 INFO:teuthology.orchestra.run.smithi049.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-21T13:32:19.133 INFO:teuthology.orchestra.run.smithi049.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-21T13:32:19.133 INFO:teuthology.orchestra.run.smithi049.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-21T13:32:19.134 DEBUG:teuthology.orchestra.run.smithi049:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-21T13:32:19.280 DEBUG:teuthology.orchestra.run.smithi049:> sudo shutdown -r now 2024-09-21T13:32:39.656 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-21T13:32:39.657 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2024-09-21T13:32:39.657 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T13:32:49.290 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-21T13:32:49.291 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-09-21T13:32:49.291 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T13:32:58.227 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2024-09-21T13:33:07.234 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2024-09-21T13:33:07.235 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T13:33:07.695 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.49 2024-09-21T13:33:10.291 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2024-09-21T13:33:16.699 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-09-21T13:33:16.700 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T13:33:19.759 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.49 2024-09-21T13:33:22.293 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2024-09-21T13:33:22.294 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T13:33:31.769 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-09-21T13:33:31.770 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T13:33:34.831 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.49 2024-09-21T13:33:40.723 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2024-09-21T13:33:49.841 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-09-21T13:33:49.841 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T13:33:52.911 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.49 2024-09-21T13:33:55.726 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2024-09-21T13:33:55.726 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T13:33:56.080 DEBUG:teuthology.orchestra.run.smithi187:> true 2024-09-21T13:33:56.591 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi187.front.sepia.ceph.com' 2024-09-21T13:33:56.591 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-508.el9.x86_64"... 2024-09-21T13:33:56.591 DEBUG:teuthology.orchestra.run.smithi187:> uname -r 2024-09-21T13:33:56.606 INFO:teuthology.orchestra.run.smithi187.stdout:5.14.0-508.el9.x86_64 2024-09-21T13:33:56.606 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-21T13:33:56.606 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-21T13:33:56.606 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-21T13:33:57.607 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-21T13:33:57.608 DEBUG:teuthology.orchestra.run.smithi187:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-21T13:33:57.693 INFO:teuthology.orchestra.run.smithi187.stdout:ttyS1 2024-09-21T13:33:57.722 DEBUG:teuthology.parallel:result is None 2024-09-21T13:34:10.912 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-09-21T13:34:10.913 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T13:35:10.915 DEBUG:teuthology.orchestra.remote:timed out 2024-09-21T13:35:31.916 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-09-21T13:35:31.917 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T13:35:32.291 DEBUG:teuthology.orchestra.run.smithi049:> true 2024-09-21T13:35:32.646 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-09-21T13:35:32.646 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-21T13:35:32.646 DEBUG:teuthology.orchestra.run.smithi049:> uname -r 2024-09-21T13:35:32.663 INFO:teuthology.orchestra.run.smithi049.stdout:5.14.0-508.el9.x86_64 2024-09-21T13:35:32.663 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-21T13:35:32.663 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-21T13:35:32.663 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-21T13:35:33.664 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-21T13:35:33.664 DEBUG:teuthology.orchestra.run.smithi049:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-21T13:35:33.740 INFO:teuthology.orchestra.run.smithi049.stdout:ttyS1 2024-09-21T13:35:33.768 DEBUG:teuthology.parallel:result is None 2024-09-21T13:35:33.768 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-21T13:35:33.776 INFO:teuthology.task.internal:Creating test directory... 2024-09-21T13:35:33.776 DEBUG:teuthology.orchestra.run.smithi049:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-21T13:35:33.778 DEBUG:teuthology.orchestra.run.smithi187:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-21T13:35:33.797 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-21T13:35:33.804 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-21T13:35:33.813 INFO:teuthology.task.internal:Creating archive directory... 2024-09-21T13:35:33.813 DEBUG:teuthology.orchestra.run.smithi049:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-21T13:35:33.836 DEBUG:teuthology.orchestra.run.smithi187:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-21T13:35:33.866 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-21T13:35:33.871 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-21T13:35:33.872 DEBUG:teuthology.orchestra.run.smithi049:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-21T13:35:33.901 DEBUG:teuthology.orchestra.run.smithi187:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-21T13:35:33.933 INFO:teuthology.orchestra.run.smithi049.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T13:35:33.942 INFO:teuthology.orchestra.run.smithi187.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T13:35:33.945 INFO:teuthology.orchestra.run.smithi049.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T13:35:33.954 INFO:teuthology.orchestra.run.smithi187.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T13:35:33.955 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-21T13:35:33.962 INFO:teuthology.task.internal:Configuring sudo... 2024-09-21T13:35:33.962 DEBUG:teuthology.orchestra.run.smithi049:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-21T13:35:33.990 DEBUG:teuthology.orchestra.run.smithi187:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-21T13:35:34.023 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-21T13:35:34.032 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-21T13:35:34.032 DEBUG:teuthology.orchestra.run.smithi049:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-21T13:35:34.058 DEBUG:teuthology.orchestra.run.smithi187:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-21T13:35:34.080 DEBUG:teuthology.orchestra.run.smithi049:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T13:35:34.155 DEBUG:teuthology.orchestra.run.smithi049:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T13:35:34.208 DEBUG:teuthology.orchestra.run.smithi049:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T13:35:34.290 DEBUG:teuthology.orchestra.run.smithi049:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T13:35:34.334 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-09-21T13:35:34.334 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-21T13:35:34.400 DEBUG:teuthology.orchestra.run.smithi187:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T13:35:34.444 DEBUG:teuthology.orchestra.run.smithi187:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T13:35:34.500 DEBUG:teuthology.orchestra.run.smithi187:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T13:35:34.583 DEBUG:teuthology.orchestra.run.smithi187:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T13:35:34.628 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-09-21T13:35:34.629 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-21T13:35:34.695 DEBUG:teuthology.orchestra.run.smithi049:> sudo service rsyslog restart 2024-09-21T13:35:34.697 DEBUG:teuthology.orchestra.run.smithi187:> sudo service rsyslog restart 2024-09-21T13:35:34.737 INFO:teuthology.orchestra.run.smithi049.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T13:35:34.785 INFO:teuthology.orchestra.run.smithi187.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T13:35:35.233 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-21T13:35:35.241 INFO:teuthology.task.internal:Starting timer... 2024-09-21T13:35:35.241 INFO:teuthology.run_tasks:Running task pcp... 2024-09-21T13:35:35.250 INFO:teuthology.run_tasks:Running task selinux... 2024-09-21T13:35:35.258 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-09-21T13:35:35.258 DEBUG:teuthology.orchestra.run.smithi049:> sudo service auditd rotate 2024-09-21T13:35:35.308 INFO:teuthology.orchestra.run.smithi049.stdout:Rotating logs: 2024-09-21T13:35:35.310 DEBUG:teuthology.orchestra.run.smithi187:> sudo service auditd rotate 2024-09-21T13:35:35.375 INFO:teuthology.orchestra.run.smithi187.stdout:Rotating logs: 2024-09-21T13:35:35.377 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-21T13:35:35.378 DEBUG:teuthology.orchestra.run.smithi049:> /usr/sbin/getenforce 2024-09-21T13:35:35.399 INFO:teuthology.orchestra.run.smithi049.stdout:Permissive 2024-09-21T13:35:35.400 DEBUG:teuthology.orchestra.run.smithi187:> /usr/sbin/getenforce 2024-09-21T13:35:35.434 INFO:teuthology.orchestra.run.smithi187.stdout:Permissive 2024-09-21T13:35:35.434 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi049.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi187.front.sepia.ceph.com': 'permissive'} 2024-09-21T13:35:35.434 DEBUG:teuthology.orchestra.run.smithi049:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-21T13:35:35.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:35:35.469 DEBUG:teuthology.orchestra.run.smithi187:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-21T13:35:35.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:35:35.501 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-21T13:35:35.501 DEBUG:teuthology.orchestra.run.smithi049:> sudo /usr/sbin/setenforce permissive 2024-09-21T13:35:35.556 DEBUG:teuthology.orchestra.run.smithi187:> sudo /usr/sbin/setenforce permissive 2024-09-21T13:35:35.589 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-21T13:35:35.598 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-21T13:35:35.609 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-21T13:35:35.609 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi049.front.sepia.ceph.com,smithi187.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-21T13:39:18.292 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi049.front.sepia.ceph.com'), Remote(name='ubuntu@smithi187.front.sepia.ceph.com')] 2024-09-21T13:39:18.293 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-09-21T13:39:18.294 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T13:39:18.369 DEBUG:teuthology.orchestra.run.smithi049:> true 2024-09-21T13:39:18.446 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-09-21T13:39:18.447 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2024-09-21T13:39:18.447 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T13:39:18.521 DEBUG:teuthology.orchestra.run.smithi187:> true 2024-09-21T13:39:18.609 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi187.front.sepia.ceph.com' 2024-09-21T13:39:18.609 INFO:teuthology.run_tasks:Running task clock... 2024-09-21T13:39:18.620 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-21T13:39:18.620 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-21T13:39:18.620 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T13:39:18.622 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-21T13:39:18.623 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T13:39:18.650 INFO:teuthology.orchestra.run.smithi049.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-21T13:39:18.666 INFO:teuthology.orchestra.run.smithi049.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-21T13:39:18.678 INFO:teuthology.orchestra.run.smithi187.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-21T13:39:18.691 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: ntpd: command not found 2024-09-21T13:39:18.693 INFO:teuthology.orchestra.run.smithi187.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-21T13:39:18.703 INFO:teuthology.orchestra.run.smithi049.stdout:506 Cannot talk to daemon 2024-09-21T13:39:18.717 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: ntpd: command not found 2024-09-21T13:39:18.718 INFO:teuthology.orchestra.run.smithi049.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-21T13:39:18.728 INFO:teuthology.orchestra.run.smithi187.stdout:506 Cannot talk to daemon 2024-09-21T13:39:18.732 INFO:teuthology.orchestra.run.smithi049.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-21T13:39:18.743 INFO:teuthology.orchestra.run.smithi187.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-21T13:39:18.757 INFO:teuthology.orchestra.run.smithi187.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-21T13:39:18.797 INFO:teuthology.orchestra.run.smithi049.stderr:bash: line 1: ntpq: command not found 2024-09-21T13:39:18.800 INFO:teuthology.orchestra.run.smithi049.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T13:39:18.800 INFO:teuthology.orchestra.run.smithi049.stdout:=============================================================================== 2024-09-21T13:39:18.800 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T13:39:18.801 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T13:39:18.801 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T13:39:18.801 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T13:39:18.823 INFO:teuthology.orchestra.run.smithi187.stderr:bash: line 1: ntpq: command not found 2024-09-21T13:39:18.826 INFO:teuthology.orchestra.run.smithi187.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T13:39:18.826 INFO:teuthology.orchestra.run.smithi187.stdout:=============================================================================== 2024-09-21T13:39:18.827 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T13:39:18.827 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T13:39:18.827 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T13:39:18.827 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T13:39:18.828 INFO:teuthology.run_tasks:Running task pexec... 2024-09-21T13:39:18.839 INFO:teuthology.task.pexec:Executing custom commands... 2024-09-21T13:39:18.840 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi049.front.sepia.ceph.com 2024-09-21T13:39:18.840 DEBUG:teuthology.orchestra.run.smithi049:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-09-21T13:39:18.840 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi187.front.sepia.ceph.com 2024-09-21T13:39:18.840 DEBUG:teuthology.orchestra.run.smithi187:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-09-21T13:39:19.209 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-09-21T13:39:19.218 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-21T13:39:19.229 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-21T13:39:19.230 INFO:teuthology.orchestra.run.smithi187.stdout: Package Architecture Version Repository Size 2024-09-21T13:39:19.230 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-21T13:39:19.230 INFO:teuthology.orchestra.run.smithi187.stdout:Removing: 2024-09-21T13:39:19.230 INFO:teuthology.orchestra.run.smithi187.stdout: nvme-cli x86_64 2.9.1-6.el9 @baseos 5.3 M 2024-09-21T13:39:19.230 INFO:teuthology.orchestra.run.smithi187.stdout:Removing unused dependencies: 2024-09-21T13:39:19.230 INFO:teuthology.orchestra.run.smithi187.stdout: libnvme x86_64 1.9-3.el9 @baseos 273 k 2024-09-21T13:39:19.230 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:39:19.230 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-09-21T13:39:19.230 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-21T13:39:19.230 INFO:teuthology.orchestra.run.smithi187.stdout:Remove 2 Packages 2024-09-21T13:39:19.230 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:39:19.230 INFO:teuthology.orchestra.run.smithi187.stdout:Freed space: 5.5 M 2024-09-21T13:39:19.231 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-09-21T13:39:19.231 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-09-21T13:39:19.231 INFO:teuthology.orchestra.run.smithi049.stdout: Package Architecture Version Repository Size 2024-09-21T13:39:19.231 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-09-21T13:39:19.231 INFO:teuthology.orchestra.run.smithi049.stdout:Removing: 2024-09-21T13:39:19.231 INFO:teuthology.orchestra.run.smithi049.stdout: nvme-cli x86_64 2.9.1-6.el9 @baseos 5.3 M 2024-09-21T13:39:19.232 INFO:teuthology.orchestra.run.smithi049.stdout:Removing unused dependencies: 2024-09-21T13:39:19.232 INFO:teuthology.orchestra.run.smithi049.stdout: libnvme x86_64 1.9-3.el9 @baseos 273 k 2024-09-21T13:39:19.232 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:39:19.232 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-09-21T13:39:19.232 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-09-21T13:39:19.232 INFO:teuthology.orchestra.run.smithi049.stdout:Remove 2 Packages 2024-09-21T13:39:19.232 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:39:19.232 INFO:teuthology.orchestra.run.smithi049.stdout:Freed space: 5.5 M 2024-09-21T13:39:19.232 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-09-21T13:39:19.234 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-09-21T13:39:19.234 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-09-21T13:39:19.235 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-09-21T13:39:19.235 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-09-21T13:39:19.294 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-09-21T13:39:19.295 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-09-21T13:39:19.299 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-09-21T13:39:19.299 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-09-21T13:39:19.437 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-09-21T13:39:19.447 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-09-21T13:39:19.531 INFO:teuthology.orchestra.run.smithi187.stdout: Erasing : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-21T13:39:19.534 INFO:teuthology.orchestra.run.smithi049.stdout: Erasing : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-21T13:39:19.589 INFO:teuthology.orchestra.run.smithi187.stdout: Erasing : libnvme-1.9-3.el9.x86_64 2/2 2024-09-21T13:39:19.594 INFO:teuthology.orchestra.run.smithi049.stdout: Erasing : libnvme-1.9-3.el9.x86_64 2/2 2024-09-21T13:39:20.098 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libnvme-1.9-3.el9.x86_64 2/2 2024-09-21T13:39:20.098 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/2 2024-09-21T13:39:20.114 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libnvme-1.9-3.el9.x86_64 2/2 2024-09-21T13:39:20.114 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/2 2024-09-21T13:39:20.391 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-21T13:39:20.391 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:39:20.391 INFO:teuthology.orchestra.run.smithi187.stdout:Removed: 2024-09-21T13:39:20.391 INFO:teuthology.orchestra.run.smithi187.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-21T13:39:20.391 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:39:20.391 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-21T13:39:20.502 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-21T13:39:20.502 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:39:20.502 INFO:teuthology.orchestra.run.smithi049.stdout:Removed: 2024-09-21T13:39:20.502 INFO:teuthology.orchestra.run.smithi049.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-21T13:39:20.502 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:39:20.502 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-09-21T13:39:21.239 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:00:49 ago on Sat 21 Sep 2024 01:38:32 PM UTC. 2024-09-21T13:39:21.334 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:00:46 ago on Sat 21 Sep 2024 01:38:35 PM UTC. 2024-09-21T13:39:21.376 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-21T13:39:21.377 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-21T13:39:21.377 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2024-09-21T13:39:21.377 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-21T13:39:21.377 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-09-21T13:39:21.377 INFO:teuthology.orchestra.run.smithi187.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-21T13:39:21.377 INFO:teuthology.orchestra.run.smithi187.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-09-21T13:39:21.377 INFO:teuthology.orchestra.run.smithi187.stdout: runc x86_64 4:1.1.13-4.el9 appstream 3.2 M 2024-09-21T13:39:21.377 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2024-09-21T13:39:21.377 INFO:teuthology.orchestra.run.smithi187.stdout: libnvme x86_64 1.9-3.el9 baseos 106 k 2024-09-21T13:39:21.377 INFO:teuthology.orchestra.run.smithi187.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-09-21T13:39:21.377 INFO:teuthology.orchestra.run.smithi187.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-09-21T13:39:21.377 INFO:teuthology.orchestra.run.smithi187.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-09-21T13:39:21.377 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:39:21.377 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-09-21T13:39:21.378 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-21T13:39:21.378 INFO:teuthology.orchestra.run.smithi187.stdout:Install 7 Packages 2024-09-21T13:39:21.378 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:39:21.391 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 5.2 M 2024-09-21T13:39:21.391 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 19 M 2024-09-21T13:39:21.391 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-09-21T13:39:21.470 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-09-21T13:39:21.471 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-09-21T13:39:21.471 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2024-09-21T13:39:21.471 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-09-21T13:39:21.471 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-09-21T13:39:21.471 INFO:teuthology.orchestra.run.smithi049.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-21T13:39:21.471 INFO:teuthology.orchestra.run.smithi049.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-09-21T13:39:21.471 INFO:teuthology.orchestra.run.smithi049.stdout: runc x86_64 4:1.1.13-4.el9 appstream 3.2 M 2024-09-21T13:39:21.472 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2024-09-21T13:39:21.472 INFO:teuthology.orchestra.run.smithi049.stdout: libnvme x86_64 1.9-3.el9 baseos 106 k 2024-09-21T13:39:21.472 INFO:teuthology.orchestra.run.smithi049.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-09-21T13:39:21.472 INFO:teuthology.orchestra.run.smithi049.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-09-21T13:39:21.472 INFO:teuthology.orchestra.run.smithi049.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-09-21T13:39:21.472 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:39:21.472 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-09-21T13:39:21.472 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-09-21T13:39:21.472 INFO:teuthology.orchestra.run.smithi049.stdout:Install 7 Packages 2024-09-21T13:39:21.472 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:39:21.481 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 5.2 M 2024-09-21T13:39:21.481 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 19 M 2024-09-21T13:39:21.481 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-09-21T13:39:21.981 INFO:teuthology.orchestra.run.smithi049.stdout:(1/7): nvmetcli-0.7-3.el9.noarch.rpm 262 kB/s | 44 kB 00:00 2024-09-21T13:39:22.006 INFO:teuthology.orchestra.run.smithi049.stdout:(2/7): libnvme-1.9-3.el9.x86_64.rpm 545 kB/s | 106 kB 00:00 2024-09-21T13:39:22.056 INFO:teuthology.orchestra.run.smithi049.stdout:(3/7): python3-kmod-0.9-32.el9.x86_64.rpm 1.6 MB/s | 84 kB 00:00 2024-09-21T13:39:22.081 INFO:teuthology.orchestra.run.smithi049.stdout:(4/7): python3-configshell-1.1.30-1.el9.noarch. 714 kB/s | 72 kB 00:00 2024-09-21T13:39:22.123 INFO:teuthology.orchestra.run.smithi049.stdout:(5/7): nvme-cli-2.9.1-6.el9.x86_64.rpm 2.8 MB/s | 906 kB 00:00 2024-09-21T13:39:22.182 INFO:teuthology.orchestra.run.smithi049.stdout:(6/7): python3-urwid-2.1.2-4.el9.x86_64.rpm 6.5 MB/s | 837 kB 00:00 2024-09-21T13:39:22.341 INFO:teuthology.orchestra.run.smithi049.stdout:(7/7): runc-1.1.13-4.el9.x86_64.rpm 12 MB/s | 3.2 MB 00:00 2024-09-21T13:39:22.341 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-09-21T13:39:22.341 INFO:teuthology.orchestra.run.smithi049.stdout:Total 6.0 MB/s | 5.2 MB 00:00 2024-09-21T13:39:22.362 INFO:teuthology.orchestra.run.smithi187.stdout:(1/7): nvmetcli-0.7-3.el9.noarch.rpm 75 kB/s | 44 kB 00:00 2024-09-21T13:39:22.455 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-09-21T13:39:22.471 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-09-21T13:39:22.471 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-09-21T13:39:22.487 INFO:teuthology.orchestra.run.smithi187.stdout:(2/7): libnvme-1.9-3.el9.x86_64.rpm 147 kB/s | 106 kB 00:00 2024-09-21T13:39:22.688 INFO:teuthology.orchestra.run.smithi187.stdout:(3/7): nvme-cli-2.9.1-6.el9.x86_64.rpm 983 kB/s | 906 kB 00:00 2024-09-21T13:39:22.777 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-09-21T13:39:22.778 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-09-21T13:39:22.988 INFO:teuthology.orchestra.run.smithi187.stdout:(4/7): python3-configshell-1.1.30-1.el9.noarch. 115 kB/s | 72 kB 00:00 2024-09-21T13:39:23.130 INFO:teuthology.orchestra.run.smithi187.stdout:(5/7): python3-kmod-0.9-32.el9.x86_64.rpm 131 kB/s | 84 kB 00:00 2024-09-21T13:39:23.277 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-09-21T13:39:23.330 INFO:teuthology.orchestra.run.smithi187.stdout:(6/7): runc-1.1.13-4.el9.x86_64.rpm 9.3 MB/s | 3.2 MB 00:00 2024-09-21T13:39:23.363 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/7 2024-09-21T13:39:23.397 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/7 2024-09-21T13:39:23.425 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/7 2024-09-21T13:39:23.589 INFO:teuthology.orchestra.run.smithi187.stdout:(7/7): python3-urwid-2.1.2-4.el9.x86_64.rpm 930 kB/s | 837 kB 00:00 2024-09-21T13:39:23.589 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-09-21T13:39:23.589 INFO:teuthology.orchestra.run.smithi187.stdout:Total 2.4 MB/s | 5.2 MB 00:02 2024-09-21T13:39:23.674 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libnvme-1.9-3.el9.x86_64 4/7 2024-09-21T13:39:23.704 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-09-21T13:39:23.706 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : nvme-cli-2.9.1-6.el9.x86_64 5/7 2024-09-21T13:39:23.719 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-09-21T13:39:23.719 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-09-21T13:39:24.001 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-09-21T13:39:24.001 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-09-21T13:39:24.193 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 5/7 2024-09-21T13:39:24.225 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : nvmetcli-0.7-3.el9.noarch 6/7 2024-09-21T13:39:24.356 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 6/7 2024-09-21T13:39:24.424 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-21T13:39:24.482 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-09-21T13:39:24.574 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/7 2024-09-21T13:39:24.609 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/7 2024-09-21T13:39:24.638 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/7 2024-09-21T13:39:24.844 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libnvme-1.9-3.el9.x86_64 4/7 2024-09-21T13:39:24.952 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : nvme-cli-2.9.1-6.el9.x86_64 5/7 2024-09-21T13:39:25.378 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 5/7 2024-09-21T13:39:25.407 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : nvmetcli-0.7-3.el9.noarch 6/7 2024-09-21T13:39:25.434 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-21T13:39:25.434 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/7 2024-09-21T13:39:25.434 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/7 2024-09-21T13:39:25.434 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 3/7 2024-09-21T13:39:25.434 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/7 2024-09-21T13:39:25.434 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/7 2024-09-21T13:39:25.434 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/7 2024-09-21T13:39:25.521 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 6/7 2024-09-21T13:39:25.587 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-21T13:39:25.856 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-21T13:39:25.856 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:39:25.856 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-09-21T13:39:25.856 INFO:teuthology.orchestra.run.smithi049.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-21T13:39:25.856 INFO:teuthology.orchestra.run.smithi049.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-09-21T13:39:25.856 INFO:teuthology.orchestra.run.smithi049.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-09-21T13:39:25.856 INFO:teuthology.orchestra.run.smithi049.stdout: runc-4:1.1.13-4.el9.x86_64 2024-09-21T13:39:25.856 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:39:25.856 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-09-21T13:39:26.204 DEBUG:teuthology.parallel:result is None 2024-09-21T13:39:26.487 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-21T13:39:26.487 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/7 2024-09-21T13:39:26.488 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/7 2024-09-21T13:39:26.488 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 3/7 2024-09-21T13:39:26.488 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/7 2024-09-21T13:39:26.488 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/7 2024-09-21T13:39:26.488 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/7 2024-09-21T13:39:26.878 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-21T13:39:26.878 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:39:26.878 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-09-21T13:39:26.878 INFO:teuthology.orchestra.run.smithi187.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-21T13:39:26.878 INFO:teuthology.orchestra.run.smithi187.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-09-21T13:39:26.878 INFO:teuthology.orchestra.run.smithi187.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-09-21T13:39:26.878 INFO:teuthology.orchestra.run.smithi187.stdout: runc-4:1.1.13-4.el9.x86_64 2024-09-21T13:39:26.879 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:39:26.879 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-21T13:39:27.291 DEBUG:teuthology.parallel:result is None 2024-09-21T13:39:27.291 INFO:teuthology.run_tasks:Running task install... 2024-09-21T13:39:27.300 DEBUG:teuthology.task.install:project ceph 2024-09-21T13:39:27.300 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '441784946648ff53dd310293f9dd752afeeea72a'}} 2024-09-21T13:39:27.300 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '441784946648ff53dd310293f9dd752afeeea72a'} 2024-09-21T13:39:27.300 INFO:teuthology.task.install:Using flavor: default 2024-09-21T13:39:27.305 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-21T13:39:27.305 INFO:teuthology.task.install:extra packages: [] 2024-09-21T13:39:27.306 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '441784946648ff53dd310293f9dd752afeeea72a', 'tag': None, 'wait_for_package': False} 2024-09-21T13:39:27.306 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=441784946648ff53dd310293f9dd752afeeea72a 2024-09-21T13:39:27.308 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '441784946648ff53dd310293f9dd752afeeea72a', 'tag': None, 'wait_for_package': False} 2024-09-21T13:39:27.308 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=441784946648ff53dd310293f9dd752afeeea72a 2024-09-21T13:39:27.445 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-21-0842/441784946648ff53dd310293f9dd752afeeea72a/centos/9/flavors/default/ 2024-09-21T13:39:27.446 INFO:teuthology.task.install.rpm:Package version is 19.3.0-5104.g44178494 2024-09-21T13:39:27.454 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-21-0842/441784946648ff53dd310293f9dd752afeeea72a/centos/9/flavors/default/ 2024-09-21T13:39:27.455 INFO:teuthology.task.install.rpm:Package version is 19.3.0-5104.g44178494 2024-09-21T13:39:27.556 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-21-0842/441784946648ff53dd310293f9dd752afeeea72a/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-21-0842/441784946648ff53dd310293f9dd752afeeea72a/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-21-0842/441784946648ff53dd310293f9dd752afeeea72a/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-21T13:39:27.556 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-09-21T13:39:27.556 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-21T13:39:27.564 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-21-0842/441784946648ff53dd310293f9dd752afeeea72a/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-21-0842/441784946648ff53dd310293f9dd752afeeea72a/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-21-0842/441784946648ff53dd310293f9dd752afeeea72a/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-21T13:39:27.564 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-09-21T13:39:27.564 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-21T13:39:27.589 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-21T13:39:27.590 DEBUG:teuthology.orchestra.run.smithi049:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/441784946648ff53dd310293f9dd752afeeea72a/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-21T13:39:27.591 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-21T13:39:27.591 DEBUG:teuthology.orchestra.run.smithi187:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/441784946648ff53dd310293f9dd752afeeea72a/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-21T13:39:27.658 DEBUG:teuthology.orchestra.run.smithi187:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-21T13:39:27.660 DEBUG:teuthology.orchestra.run.smithi049:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-21T13:39:27.732 DEBUG:teuthology.orchestra.run.smithi187:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-21T13:39:27.743 DEBUG:teuthology.orchestra.run.smithi049:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-21T13:39:27.812 INFO:teuthology.orchestra.run.smithi049.stdout:check_obsoletes = 1 2024-09-21T13:39:27.814 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum clean all 2024-09-21T13:39:27.823 INFO:teuthology.orchestra.run.smithi187.stdout:check_obsoletes = 1 2024-09-21T13:39:27.825 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum clean all 2024-09-21T13:39:28.015 INFO:teuthology.orchestra.run.smithi187.stdout:45 files removed 2024-09-21T13:39:28.042 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-radosgw 2024-09-21T13:39:28.074 INFO:teuthology.orchestra.run.smithi049.stdout:45 files removed 2024-09-21T13:39:28.101 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-radosgw 2024-09-21T13:39:28.597 INFO:teuthology.orchestra.run.smithi187.stdout:ceph packages for x86_64 311 kB/s | 84 kB 00:00 2024-09-21T13:39:28.666 INFO:teuthology.orchestra.run.smithi049.stdout:ceph packages for x86_64 317 kB/s | 84 kB 00:00 2024-09-21T13:39:28.872 INFO:teuthology.orchestra.run.smithi187.stdout:ceph noarch packages 104 kB/s | 24 kB 00:00 2024-09-21T13:39:28.950 INFO:teuthology.orchestra.run.smithi049.stdout:ceph noarch packages 102 kB/s | 24 kB 00:00 2024-09-21T13:39:29.148 INFO:teuthology.orchestra.run.smithi187.stdout:ceph source packages 8.1 kB/s | 1.9 kB 00:00 2024-09-21T13:39:29.217 INFO:teuthology.orchestra.run.smithi049.stdout:ceph source packages 8.5 kB/s | 1.9 kB 00:00 2024-09-21T13:39:29.859 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.3 MB 00:00 2024-09-21T13:39:30.698 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - BaseOS 5.4 MB/s | 8.3 MB 00:01 2024-09-21T13:39:33.302 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - AppStream 8.8 MB/s | 20 MB 00:02 2024-09-21T13:39:36.090 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - AppStream 4.7 MB/s | 20 MB 00:04 2024-09-21T13:39:40.094 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - CRB 2.3 MB/s | 6.5 MB 00:02 2024-09-21T13:39:40.249 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - CRB 18 MB/s | 6.5 MB 00:00 2024-09-21T13:39:41.653 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - Extras packages 78 kB/s | 19 kB 00:00 2024-09-21T13:39:41.817 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - Extras packages 100 kB/s | 19 kB 00:00 2024-09-21T13:39:42.632 INFO:teuthology.orchestra.run.smithi049.stdout:Extra Packages for Enterprise Linux 27 MB/s | 23 MB 00:00 2024-09-21T13:39:42.821 INFO:teuthology.orchestra.run.smithi187.stdout:Extra Packages for Enterprise Linux 26 MB/s | 23 MB 00:00 2024-09-21T13:39:48.260 INFO:teuthology.orchestra.run.smithi187.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-09-21T13:39:48.271 INFO:teuthology.orchestra.run.smithi049.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-21T13:39:50.459 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-21T13:39:50.462 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-21T13:39:50.462 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2024-09-21T13:39:50.462 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-21T13:39:50.462 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-09-21T13:39:50.462 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-radosgw x86_64 2:19.3.0-5104.g44178494.el9 ceph 11 M 2024-09-21T13:39:50.462 INFO:teuthology.orchestra.run.smithi187.stdout:Upgrading: 2024-09-21T13:39:50.462 INFO:teuthology.orchestra.run.smithi187.stdout: librados2 x86_64 2:19.3.0-5104.g44178494.el9 ceph 3.5 M 2024-09-21T13:39:50.462 INFO:teuthology.orchestra.run.smithi187.stdout: librbd1 x86_64 2:19.3.0-5104.g44178494.el9 ceph 3.2 M 2024-09-21T13:39:50.462 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2024-09-21T13:39:50.462 INFO:teuthology.orchestra.run.smithi187.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-21T13:39:50.462 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-base x86_64 2:19.3.0-5104.g44178494.el9 ceph 5.5 M 2024-09-21T13:39:50.462 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-common x86_64 2:19.3.0-5104.g44178494.el9 ceph 22 M 2024-09-21T13:39:50.463 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-selinux x86_64 2:19.3.0-5104.g44178494.el9 ceph 25 k 2024-09-21T13:39:50.463 INFO:teuthology.orchestra.run.smithi187.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-21T13:39:50.463 INFO:teuthology.orchestra.run.smithi187.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-21T13:39:50.463 INFO:teuthology.orchestra.run.smithi187.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-21T13:39:50.463 INFO:teuthology.orchestra.run.smithi187.stdout: libcephfs2 x86_64 2:19.3.0-5104.g44178494.el9 ceph 754 k 2024-09-21T13:39:50.463 INFO:teuthology.orchestra.run.smithi187.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-21T13:39:50.463 INFO:teuthology.orchestra.run.smithi187.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-21T13:39:50.463 INFO:teuthology.orchestra.run.smithi187.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-21T13:39:50.463 INFO:teuthology.orchestra.run.smithi187.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-21T13:39:50.463 INFO:teuthology.orchestra.run.smithi187.stdout: libradosstriper1 x86_64 2:19.3.0-5104.g44178494.el9 ceph 508 k 2024-09-21T13:39:50.463 INFO:teuthology.orchestra.run.smithi187.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-21T13:39:50.463 INFO:teuthology.orchestra.run.smithi187.stdout: librgw2 x86_64 2:19.3.0-5104.g44178494.el9 ceph 5.5 M 2024-09-21T13:39:50.463 INFO:teuthology.orchestra.run.smithi187.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-21T13:39:50.464 INFO:teuthology.orchestra.run.smithi187.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-21T13:39:50.464 INFO:teuthology.orchestra.run.smithi187.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-21T13:39:50.464 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ceph-argparse x86_64 2:19.3.0-5104.g44178494.el9 ceph 45 k 2024-09-21T13:39:50.464 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ceph-common x86_64 2:19.3.0-5104.g44178494.el9 ceph 148 k 2024-09-21T13:39:50.464 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cephfs x86_64 2:19.3.0-5104.g44178494.el9 ceph 161 k 2024-09-21T13:39:50.464 INFO:teuthology.orchestra.run.smithi187.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-21T13:39:50.464 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rados x86_64 2:19.3.0-5104.g44178494.el9 ceph 320 k 2024-09-21T13:39:50.464 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rbd x86_64 2:19.3.0-5104.g44178494.el9 ceph 307 k 2024-09-21T13:39:50.464 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rgw x86_64 2:19.3.0-5104.g44178494.el9 ceph 99 k 2024-09-21T13:39:50.464 INFO:teuthology.orchestra.run.smithi187.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-21T13:39:50.464 INFO:teuthology.orchestra.run.smithi187.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-21T13:39:50.464 INFO:teuthology.orchestra.run.smithi187.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-21T13:39:50.464 INFO:teuthology.orchestra.run.smithi187.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-21T13:39:50.464 INFO:teuthology.orchestra.run.smithi187.stdout:Installing weak dependencies: 2024-09-21T13:39:50.464 INFO:teuthology.orchestra.run.smithi187.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-21T13:39:50.465 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:39:50.465 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-09-21T13:39:50.465 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-21T13:39:50.465 INFO:teuthology.orchestra.run.smithi187.stdout:Install 31 Packages 2024-09-21T13:39:50.465 INFO:teuthology.orchestra.run.smithi187.stdout:Upgrade 2 Packages 2024-09-21T13:39:50.465 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:39:50.465 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 62 M 2024-09-21T13:39:50.465 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-09-21T13:39:50.645 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-09-21T13:39:50.647 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-09-21T13:39:50.647 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2024-09-21T13:39:50.647 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-09-21T13:39:50.647 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-09-21T13:39:50.647 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-radosgw x86_64 2:19.3.0-5104.g44178494.el9 ceph 11 M 2024-09-21T13:39:50.647 INFO:teuthology.orchestra.run.smithi049.stdout:Upgrading: 2024-09-21T13:39:50.648 INFO:teuthology.orchestra.run.smithi049.stdout: librados2 x86_64 2:19.3.0-5104.g44178494.el9 ceph 3.5 M 2024-09-21T13:39:50.648 INFO:teuthology.orchestra.run.smithi049.stdout: librbd1 x86_64 2:19.3.0-5104.g44178494.el9 ceph 3.2 M 2024-09-21T13:39:50.648 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2024-09-21T13:39:50.648 INFO:teuthology.orchestra.run.smithi049.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-21T13:39:50.648 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-base x86_64 2:19.3.0-5104.g44178494.el9 ceph 5.5 M 2024-09-21T13:39:50.648 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-common x86_64 2:19.3.0-5104.g44178494.el9 ceph 22 M 2024-09-21T13:39:50.648 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-selinux x86_64 2:19.3.0-5104.g44178494.el9 ceph 25 k 2024-09-21T13:39:50.648 INFO:teuthology.orchestra.run.smithi049.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-21T13:39:50.648 INFO:teuthology.orchestra.run.smithi049.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-21T13:39:50.648 INFO:teuthology.orchestra.run.smithi049.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-21T13:39:50.648 INFO:teuthology.orchestra.run.smithi049.stdout: libcephfs2 x86_64 2:19.3.0-5104.g44178494.el9 ceph 754 k 2024-09-21T13:39:50.648 INFO:teuthology.orchestra.run.smithi049.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-21T13:39:50.648 INFO:teuthology.orchestra.run.smithi049.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-21T13:39:50.648 INFO:teuthology.orchestra.run.smithi049.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-21T13:39:50.649 INFO:teuthology.orchestra.run.smithi049.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-21T13:39:50.649 INFO:teuthology.orchestra.run.smithi049.stdout: libradosstriper1 x86_64 2:19.3.0-5104.g44178494.el9 ceph 508 k 2024-09-21T13:39:50.649 INFO:teuthology.orchestra.run.smithi049.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-21T13:39:50.649 INFO:teuthology.orchestra.run.smithi049.stdout: librgw2 x86_64 2:19.3.0-5104.g44178494.el9 ceph 5.5 M 2024-09-21T13:39:50.649 INFO:teuthology.orchestra.run.smithi049.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-21T13:39:50.649 INFO:teuthology.orchestra.run.smithi049.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-21T13:39:50.649 INFO:teuthology.orchestra.run.smithi049.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-21T13:39:50.649 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ceph-argparse x86_64 2:19.3.0-5104.g44178494.el9 ceph 45 k 2024-09-21T13:39:50.649 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ceph-common x86_64 2:19.3.0-5104.g44178494.el9 ceph 148 k 2024-09-21T13:39:50.649 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cephfs x86_64 2:19.3.0-5104.g44178494.el9 ceph 161 k 2024-09-21T13:39:50.649 INFO:teuthology.orchestra.run.smithi049.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-21T13:39:50.649 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rados x86_64 2:19.3.0-5104.g44178494.el9 ceph 320 k 2024-09-21T13:39:50.649 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rbd x86_64 2:19.3.0-5104.g44178494.el9 ceph 307 k 2024-09-21T13:39:50.649 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rgw x86_64 2:19.3.0-5104.g44178494.el9 ceph 99 k 2024-09-21T13:39:50.650 INFO:teuthology.orchestra.run.smithi049.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-21T13:39:50.650 INFO:teuthology.orchestra.run.smithi049.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-21T13:39:50.650 INFO:teuthology.orchestra.run.smithi049.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-21T13:39:50.650 INFO:teuthology.orchestra.run.smithi049.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-21T13:39:50.650 INFO:teuthology.orchestra.run.smithi049.stdout:Installing weak dependencies: 2024-09-21T13:39:50.650 INFO:teuthology.orchestra.run.smithi049.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-21T13:39:50.650 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:39:50.650 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-09-21T13:39:50.650 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-09-21T13:39:50.650 INFO:teuthology.orchestra.run.smithi049.stdout:Install 31 Packages 2024-09-21T13:39:50.650 INFO:teuthology.orchestra.run.smithi049.stdout:Upgrade 2 Packages 2024-09-21T13:39:50.650 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:39:50.651 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 62 M 2024-09-21T13:39:50.651 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-09-21T13:39:51.270 INFO:teuthology.orchestra.run.smithi187.stdout:(1/33): ceph-base-19.3.0-5104.g44178494.el9.x86 9.0 MB/s | 5.5 MB 00:00 2024-09-21T13:39:51.336 INFO:teuthology.orchestra.run.smithi187.stdout:(2/33): ceph-selinux-19.3.0-5104.g44178494.el9. 379 kB/s | 25 kB 00:00 2024-09-21T13:39:51.475 INFO:teuthology.orchestra.run.smithi187.stdout:(3/33): ceph-radosgw-19.3.0-5104.g44178494.el9. 13 MB/s | 11 MB 00:00 2024-09-21T13:39:51.512 INFO:teuthology.orchestra.run.smithi187.stdout:(4/33): libcephfs2-19.3.0-5104.g44178494.el9.x8 4.2 MB/s | 754 kB 00:00 2024-09-21T13:39:51.546 INFO:teuthology.orchestra.run.smithi187.stdout:(5/33): libradosstriper1-19.3.0-5104.g44178494. 7.0 MB/s | 508 kB 00:00 2024-09-21T13:39:51.605 INFO:teuthology.orchestra.run.smithi187.stdout:(6/33): python3-ceph-argparse-19.3.0-5104.g4417 771 kB/s | 45 kB 00:00 2024-09-21T13:39:51.680 INFO:teuthology.orchestra.run.smithi187.stdout:(7/33): python3-ceph-common-19.3.0-5104.g441784 1.9 MB/s | 148 kB 00:00 2024-09-21T13:39:51.755 INFO:teuthology.orchestra.run.smithi187.stdout:(8/33): python3-cephfs-19.3.0-5104.g44178494.el 2.1 MB/s | 161 kB 00:00 2024-09-21T13:39:51.839 INFO:teuthology.orchestra.run.smithi187.stdout:(9/33): librgw2-19.3.0-5104.g44178494.el9.x86_6 17 MB/s | 5.5 MB 00:00 2024-09-21T13:39:51.864 INFO:teuthology.orchestra.run.smithi187.stdout:(10/33): python3-rados-19.3.0-5104.g44178494.el 2.9 MB/s | 320 kB 00:00 2024-09-21T13:39:51.973 INFO:teuthology.orchestra.run.smithi049.stdout:(1/33): ceph-base-19.3.0-5104.g44178494.el9.x86 4.6 MB/s | 5.5 MB 00:01 2024-09-21T13:39:52.014 INFO:teuthology.orchestra.run.smithi049.stdout:(2/33): ceph-selinux-19.3.0-5104.g44178494.el9. 610 kB/s | 25 kB 00:00 2024-09-21T13:39:52.056 INFO:teuthology.orchestra.run.smithi187.stdout:(11/33): ceph-common-19.3.0-5104.g44178494.el9. 15 MB/s | 22 MB 00:01 2024-09-21T13:39:52.081 INFO:teuthology.orchestra.run.smithi187.stdout:(12/33): python3-rgw-19.3.0-5104.g44178494.el9. 457 kB/s | 99 kB 00:00 2024-09-21T13:39:52.081 INFO:teuthology.orchestra.run.smithi049.stdout:(3/33): libcephfs2-19.3.0-5104.g44178494.el9.x8 11 MB/s | 754 kB 00:00 2024-09-21T13:39:52.115 INFO:teuthology.orchestra.run.smithi187.stdout:(13/33): python3-rbd-19.3.0-5104.g44178494.el9. 1.1 MB/s | 307 kB 00:00 2024-09-21T13:39:52.156 INFO:teuthology.orchestra.run.smithi049.stdout:(4/33): libradosstriper1-19.3.0-5104.g44178494. 6.6 MB/s | 508 kB 00:00 2024-09-21T13:39:52.198 INFO:teuthology.orchestra.run.smithi187.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 753 kB/s | 107 kB 00:00 2024-09-21T13:39:52.249 INFO:teuthology.orchestra.run.smithi187.stdout:(15/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 903 kB/s | 45 kB 00:00 2024-09-21T13:39:52.274 INFO:teuthology.orchestra.run.smithi187.stdout:(16/33): libnbd-1.20.2-2.el9.x86_64.rpm 886 kB/s | 171 kB 00:00 2024-09-21T13:39:52.299 INFO:teuthology.orchestra.run.smithi187.stdout:(17/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 870 kB/s | 160 kB 00:00 2024-09-21T13:39:52.341 INFO:teuthology.orchestra.run.smithi187.stdout:(18/33): python3-prettytable-0.7.2-27.el9.noarc 1.0 MB/s | 42 kB 00:00 2024-09-21T13:39:52.366 INFO:teuthology.orchestra.run.smithi187.stdout:(19/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.1 MB/s | 292 kB 00:00 2024-09-21T13:39:52.392 INFO:teuthology.orchestra.run.smithi187.stdout:(20/33): librdkafka-1.6.1-102.el9.x86_64.rpm 4.5 MB/s | 662 kB 00:00 2024-09-21T13:39:52.408 INFO:teuthology.orchestra.run.smithi187.stdout:(21/33): qatlib-service-24.02.0-1.el9.x86_64.rp 860 kB/s | 36 kB 00:00 2024-09-21T13:39:52.434 INFO:teuthology.orchestra.run.smithi187.stdout:(22/33): qatlib-24.02.0-1.el9.x86_64.rpm 2.3 MB/s | 221 kB 00:00 2024-09-21T13:39:52.451 INFO:teuthology.orchestra.run.smithi187.stdout:(23/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 810 kB/s | 47 kB 00:00 2024-09-21T13:39:52.457 INFO:teuthology.orchestra.run.smithi049.stdout:(5/33): librgw2-19.3.0-5104.g44178494.el9.x86_6 18 MB/s | 5.5 MB 00:00 2024-09-21T13:39:52.499 INFO:teuthology.orchestra.run.smithi049.stdout:(6/33): python3-ceph-argparse-19.3.0-5104.g4417 1.1 MB/s | 45 kB 00:00 2024-09-21T13:39:52.509 INFO:teuthology.orchestra.run.smithi187.stdout:(24/33): gperftools-libs-2.9.1-3.el9.x86_64.rpm 3.0 MB/s | 308 kB 00:00 2024-09-21T13:39:52.526 INFO:teuthology.orchestra.run.smithi187.stdout:(25/33): liboath-2.6.7-2.el9.x86_64.rpm 3.2 MB/s | 49 kB 00:00 2024-09-21T13:39:52.543 INFO:teuthology.orchestra.run.smithi187.stdout:(26/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 336 kB/s | 26 kB 00:00 2024-09-21T13:39:52.549 INFO:teuthology.orchestra.run.smithi049.stdout:(7/33): python3-ceph-common-19.3.0-5104.g441784 2.9 MB/s | 148 kB 00:00 2024-09-21T13:39:52.568 INFO:teuthology.orchestra.run.smithi187.stdout:(27/33): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-09-21T13:39:52.594 INFO:teuthology.orchestra.run.smithi187.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 7.5 MB/s | 191 kB 00:00 2024-09-21T13:39:52.599 INFO:teuthology.orchestra.run.smithi049.stdout:(8/33): python3-cephfs-19.3.0-5104.g44178494.el 3.2 MB/s | 161 kB 00:00 2024-09-21T13:39:52.650 INFO:teuthology.orchestra.run.smithi049.stdout:(9/33): python3-rados-19.3.0-5104.g44178494.el9 6.3 MB/s | 320 kB 00:00 2024-09-21T13:39:52.652 INFO:teuthology.orchestra.run.smithi187.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.5 MB/s | 839 kB 00:00 2024-09-21T13:39:52.700 INFO:teuthology.orchestra.run.smithi049.stdout:(10/33): python3-rbd-19.3.0-5104.g44178494.el9. 6.0 MB/s | 307 kB 00:00 2024-09-21T13:39:52.702 INFO:teuthology.orchestra.run.smithi187.stdout:(30/33): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-09-21T13:39:52.759 INFO:teuthology.orchestra.run.smithi049.stdout:(11/33): python3-rgw-19.3.0-5104.g44178494.el9. 1.7 MB/s | 99 kB 00:00 2024-09-21T13:39:52.778 INFO:teuthology.orchestra.run.smithi187.stdout:(31/33): libarrow-9.0.0-11.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-09-21T13:39:52.892 INFO:teuthology.orchestra.run.smithi049.stdout:(12/33): boost-program-options-1.75.0-8.el9.x86 802 kB/s | 107 kB 00:00 2024-09-21T13:39:52.903 INFO:teuthology.orchestra.run.smithi187.stdout:(32/33): librados2-19.3.0-5104.g44178494.el9.x8 14 MB/s | 3.5 MB 00:00 2024-09-21T13:39:52.951 INFO:teuthology.orchestra.run.smithi049.stdout:(13/33): libnbd-1.20.2-2.el9.x86_64.rpm 2.9 MB/s | 171 kB 00:00 2024-09-21T13:39:52.970 INFO:teuthology.orchestra.run.smithi187.stdout:(33/33): librbd1-19.3.0-5104.g44178494.el9.x86_ 12 MB/s | 3.2 MB 00:00 2024-09-21T13:39:52.976 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-09-21T13:39:52.976 INFO:teuthology.orchestra.run.smithi187.stdout:Total 25 MB/s | 62 MB 00:02 2024-09-21T13:39:53.001 INFO:teuthology.orchestra.run.smithi049.stdout:(14/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 160 kB 00:00 2024-09-21T13:39:53.043 INFO:teuthology.orchestra.run.smithi049.stdout:(15/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-09-21T13:39:53.092 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-09-21T13:39:53.118 INFO:teuthology.orchestra.run.smithi049.stdout:(16/33): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 662 kB 00:00 2024-09-21T13:39:53.128 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-09-21T13:39:53.129 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-09-21T13:39:53.160 INFO:teuthology.orchestra.run.smithi049.stdout:(17/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.9 MB/s | 292 kB 00:00 2024-09-21T13:39:53.205 INFO:teuthology.orchestra.run.smithi049.stdout:(18/33): python3-prettytable-0.7.2-27.el9.noarc 1.0 MB/s | 42 kB 00:00 2024-09-21T13:39:53.279 INFO:teuthology.orchestra.run.smithi049.stdout:(19/33): qatlib-24.02.0-1.el9.x86_64.rpm 5.2 MB/s | 221 kB 00:00 2024-09-21T13:39:53.286 INFO:teuthology.orchestra.run.smithi049.stdout:(20/33): qatlib-service-24.02.0-1.el9.x86_64.rp 858 kB/s | 36 kB 00:00 2024-09-21T13:39:53.328 INFO:teuthology.orchestra.run.smithi049.stdout:(21/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2024-09-21T13:39:53.378 INFO:teuthology.orchestra.run.smithi049.stdout:(22/33): gperftools-libs-2.9.1-3.el9.x86_64.rpm 6.0 MB/s | 308 kB 00:00 2024-09-21T13:39:53.487 INFO:teuthology.orchestra.run.smithi049.stdout:(23/33): libarrow-9.0.0-11.el9.x86_64.rpm 41 MB/s | 4.4 MB 00:00 2024-09-21T13:39:53.513 INFO:teuthology.orchestra.run.smithi049.stdout:(24/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.0 MB/s | 26 kB 00:00 2024-09-21T13:39:53.538 INFO:teuthology.orchestra.run.smithi049.stdout:(25/33): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-09-21T13:39:53.572 INFO:teuthology.orchestra.run.smithi049.stdout:(26/33): libunwind-1.6.2-1.el9.x86_64.rpm 2.0 MB/s | 67 kB 00:00 2024-09-21T13:39:53.614 INFO:teuthology.orchestra.run.smithi049.stdout:(27/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 20 MB/s | 839 kB 00:00 2024-09-21T13:39:53.635 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-09-21T13:39:53.635 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-09-21T13:39:53.647 INFO:teuthology.orchestra.run.smithi049.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-21T13:39:53.706 INFO:teuthology.orchestra.run.smithi049.stdout:(29/33): thrift-0.15.0-2.el9.x86_64.rpm 27 MB/s | 1.6 MB 00:00 2024-09-21T13:39:53.948 INFO:teuthology.orchestra.run.smithi049.stdout:(30/33): librados2-19.3.0-5104.g44178494.el9.x8 14 MB/s | 3.5 MB 00:00 2024-09-21T13:39:54.115 INFO:teuthology.orchestra.run.smithi049.stdout:(31/33): librbd1-19.3.0-5104.g44178494.el9.x86_ 19 MB/s | 3.2 MB 00:00 2024-09-21T13:39:54.632 INFO:teuthology.orchestra.run.smithi049.stdout:(32/33): ceph-radosgw-19.3.0-5104.g44178494.el9 2.8 MB/s | 11 MB 00:03 2024-09-21T13:39:55.101 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-09-21T13:39:55.194 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-21T13:39:55.226 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/35 2024-09-21T13:39:55.251 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/35 2024-09-21T13:39:55.274 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-21T13:39:55.309 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-21T13:39:55.501 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-21T13:39:55.553 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-21T13:39:55.940 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/35 2024-09-21T13:39:56.011 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : librados2-2:19.3.0-5104.g44178494.el9.x86_64 7/35 2024-09-21T13:39:56.040 INFO:teuthology.orchestra.run.smithi049.stdout:(33/33): ceph-common-19.3.0-5104.g44178494.el9. 4.1 MB/s | 22 MB 00:05 2024-09-21T13:39:56.045 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-09-21T13:39:56.045 INFO:teuthology.orchestra.run.smithi049.stdout:Total 11 MB/s | 62 MB 00:05 2024-09-21T13:39:56.130 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librados2-2:19.3.0-5104.g44178494.el9.x86_64 7/35 2024-09-21T13:39:56.170 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-09-21T13:39:56.212 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-09-21T13:39:56.213 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-09-21T13:39:56.235 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-rados-2:19.3.0-5104.g44178494.el9.x86_64 8/35 2024-09-21T13:39:56.262 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libcephfs2-2:19.3.0-5104.g44178494.el9.x86_64 9/35 2024-09-21T13:39:56.325 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libcephfs2-2:19.3.0-5104.g44178494.el9.x86_64 9/35 2024-09-21T13:39:56.362 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/35 2024-09-21T13:39:56.406 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-ceph-argparse-2:19.3.0-5104.g44178494.el9. 11/35 2024-09-21T13:39:56.475 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cephfs-2:19.3.0-5104.g44178494.el9.x86_64 12/35 2024-09-21T13:39:56.506 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libradosstriper1-2:19.3.0-5104.g44178494.el9.x86_6 13/35 2024-09-21T13:39:56.570 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libradosstriper1-2:19.3.0-5104.g44178494.el9.x86_6 13/35 2024-09-21T13:39:56.613 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/35 2024-09-21T13:39:56.730 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/35 2024-09-21T13:39:56.765 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-09-21T13:39:56.765 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-09-21T13:39:56.772 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/35 2024-09-21T13:39:56.813 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/35 2024-09-21T13:39:56.854 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/35 2024-09-21T13:39:56.883 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/35 2024-09-21T13:39:56.917 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-21T13:39:56.953 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/35 2024-09-21T13:39:57.285 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-21T13:39:57.330 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : librbd1-2:19.3.0-5104.g44178494.el9.x86_64 23/35 2024-09-21T13:39:57.420 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librbd1-2:19.3.0-5104.g44178494.el9.x86_64 23/35 2024-09-21T13:39:57.451 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-rbd-2:19.3.0-5104.g44178494.el9.x86_64 24/35 2024-09-21T13:39:57.487 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-21T13:39:57.975 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-21T13:39:57.999 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : librgw2-2:19.3.0-5104.g44178494.el9.x86_64 27/35 2024-09-21T13:39:58.070 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librgw2-2:19.3.0-5104.g44178494.el9.x86_64 27/35 2024-09-21T13:39:58.160 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-rgw-2:19.3.0-5104.g44178494.el9.x86_64 28/35 2024-09-21T13:39:58.184 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-ceph-common-2:19.3.0-5104.g44178494.el9.x8 29/35 2024-09-21T13:39:58.265 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-09-21T13:39:58.343 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-21T13:39:58.379 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/35 2024-09-21T13:39:58.404 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/35 2024-09-21T13:39:58.426 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-21T13:39:58.453 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-21T13:39:58.656 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-21T13:39:58.707 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-21T13:39:59.061 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/35 2024-09-21T13:39:59.164 INFO:teuthology.orchestra.run.smithi049.stdout: Upgrading : librados2-2:19.3.0-5104.g44178494.el9.x86_64 7/35 2024-09-21T13:39:59.262 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librados2-2:19.3.0-5104.g44178494.el9.x86_64 7/35 2024-09-21T13:39:59.360 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-rados-2:19.3.0-5104.g44178494.el9.x86_64 8/35 2024-09-21T13:39:59.390 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libcephfs2-2:19.3.0-5104.g44178494.el9.x86_64 9/35 2024-09-21T13:39:59.454 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libcephfs2-2:19.3.0-5104.g44178494.el9.x86_64 9/35 2024-09-21T13:39:59.499 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/35 2024-09-21T13:39:59.546 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-ceph-argparse-2:19.3.0-5104.g44178494.el9. 11/35 2024-09-21T13:39:59.622 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cephfs-2:19.3.0-5104.g44178494.el9.x86_64 12/35 2024-09-21T13:39:59.651 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libradosstriper1-2:19.3.0-5104.g44178494.el9.x86_6 13/35 2024-09-21T13:39:59.715 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libradosstriper1-2:19.3.0-5104.g44178494.el9.x86_6 13/35 2024-09-21T13:39:59.749 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/35 2024-09-21T13:39:59.860 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/35 2024-09-21T13:39:59.901 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/35 2024-09-21T13:39:59.950 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/35 2024-09-21T13:39:59.982 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/35 2024-09-21T13:40:00.012 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/35 2024-09-21T13:40:00.053 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-21T13:40:00.099 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/35 2024-09-21T13:40:00.435 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-21T13:40:00.483 INFO:teuthology.orchestra.run.smithi049.stdout: Upgrading : librbd1-2:19.3.0-5104.g44178494.el9.x86_64 23/35 2024-09-21T13:40:00.535 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-common-2:19.3.0-5104.g44178494.el9.x86_64 30/35 2024-09-21T13:40:00.573 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-common-2:19.3.0-5104.g44178494.el9.x86_64 30/35 2024-09-21T13:40:00.593 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librbd1-2:19.3.0-5104.g44178494.el9.x86_64 23/35 2024-09-21T13:40:00.621 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-rbd-2:19.3.0-5104.g44178494.el9.x86_64 24/35 2024-09-21T13:40:00.658 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-21T13:40:01.147 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-common-2:19.3.0-5104.g44178494.el9.x86_64 30/35 2024-09-21T13:40:01.175 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-21T13:40:01.192 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-base-2:19.3.0-5104.g44178494.el9.x86_64 31/35 2024-09-21T13:40:01.202 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : librgw2-2:19.3.0-5104.g44178494.el9.x86_64 27/35 2024-09-21T13:40:01.274 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librgw2-2:19.3.0-5104.g44178494.el9.x86_64 27/35 2024-09-21T13:40:01.288 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-base-2:19.3.0-5104.g44178494.el9.x86_64 31/35 2024-09-21T13:40:01.288 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-21T13:40:01.288 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-21T13:40:01.289 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:40:01.335 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-selinux-2:19.3.0-5104.g44178494.el9.x86_64 32/35 2024-09-21T13:40:01.359 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-rgw-2:19.3.0-5104.g44178494.el9.x86_64 28/35 2024-09-21T13:40:01.421 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-ceph-common-2:19.3.0-5104.g44178494.el9.x8 29/35 2024-09-21T13:40:03.875 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-common-2:19.3.0-5104.g44178494.el9.x86_64 30/35 2024-09-21T13:40:03.926 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-common-2:19.3.0-5104.g44178494.el9.x86_64 30/35 2024-09-21T13:40:04.534 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-common-2:19.3.0-5104.g44178494.el9.x86_64 30/35 2024-09-21T13:40:04.571 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-base-2:19.3.0-5104.g44178494.el9.x86_64 31/35 2024-09-21T13:40:04.657 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-base-2:19.3.0-5104.g44178494.el9.x86_64 31/35 2024-09-21T13:40:04.657 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-21T13:40:04.657 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-21T13:40:04.657 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:40:04.689 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-selinux-2:19.3.0-5104.g44178494.el9.x86_64 32/35 2024-09-21T13:40:11.526 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-selinux-2:19.3.0-5104.g44178494.el9.x86_64 32/35 2024-09-21T13:40:11.526 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /sys 2024-09-21T13:40:11.526 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /proc 2024-09-21T13:40:11.526 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /mnt 2024-09-21T13:40:11.526 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /var/tmp 2024-09-21T13:40:11.526 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /home 2024-09-21T13:40:11.527 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /root 2024-09-21T13:40:11.527 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /tmp 2024-09-21T13:40:11.527 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:40:12.559 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-radosgw-2:19.3.0-5104.g44178494.el9.x86_64 33/35 2024-09-21T13:40:12.586 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-5104.g44178494.el9.x86_64 33/35 2024-09-21T13:40:12.586 INFO:teuthology.orchestra.run.smithi187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T13:40:12.586 INFO:teuthology.orchestra.run.smithi187.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-21T13:40:12.586 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T13:40:12.586 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T13:40:12.586 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:40:12.587 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-21T13:40:12.671 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-21T13:40:12.671 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-21T13:40:13.827 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-21T13:40:13.827 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-base-2:19.3.0-5104.g44178494.el9.x86_64 1/35 2024-09-21T13:40:13.827 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-common-2:19.3.0-5104.g44178494.el9.x86_64 2/35 2024-09-21T13:40:13.827 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-radosgw-2:19.3.0-5104.g44178494.el9.x86_64 3/35 2024-09-21T13:40:13.827 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-selinux-2:19.3.0-5104.g44178494.el9.x86_64 4/35 2024-09-21T13:40:13.827 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libcephfs2-2:19.3.0-5104.g44178494.el9.x86_64 5/35 2024-09-21T13:40:13.827 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libradosstriper1-2:19.3.0-5104.g44178494.el9.x86_6 6/35 2024-09-21T13:40:13.827 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librgw2-2:19.3.0-5104.g44178494.el9.x86_64 7/35 2024-09-21T13:40:13.827 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-ceph-argparse-2:19.3.0-5104.g44178494.el9. 8/35 2024-09-21T13:40:13.827 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-ceph-common-2:19.3.0-5104.g44178494.el9.x8 9/35 2024-09-21T13:40:13.827 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cephfs-2:19.3.0-5104.g44178494.el9.x86_64 10/35 2024-09-21T13:40:13.827 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-rados-2:19.3.0-5104.g44178494.el9.x86_64 11/35 2024-09-21T13:40:13.828 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-rbd-2:19.3.0-5104.g44178494.el9.x86_64 12/35 2024-09-21T13:40:13.828 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-rgw-2:19.3.0-5104.g44178494.el9.x86_64 13/35 2024-09-21T13:40:13.828 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-21T13:40:13.828 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-21T13:40:13.828 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-21T13:40:13.828 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-21T13:40:13.828 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-21T13:40:13.828 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-21T13:40:13.828 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-21T13:40:13.828 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-21T13:40:13.828 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-21T13:40:13.828 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-21T13:40:13.828 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 24/35 2024-09-21T13:40:13.828 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-21T13:40:13.829 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-21T13:40:13.829 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-21T13:40:13.829 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-21T13:40:13.829 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-21T13:40:13.829 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-21T13:40:13.829 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-21T13:40:13.829 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librados2-2:19.3.0-5104.g44178494.el9.x86_64 32/35 2024-09-21T13:40:13.829 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-21T13:40:13.829 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librbd1-2:19.3.0-5104.g44178494.el9.x86_64 34/35 2024-09-21T13:40:14.760 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-21T13:40:14.760 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:40:14.760 INFO:teuthology.orchestra.run.smithi187.stdout:Upgraded: 2024-09-21T13:40:14.760 INFO:teuthology.orchestra.run.smithi187.stdout: librados2-2:19.3.0-5104.g44178494.el9.x86_64 2024-09-21T13:40:14.760 INFO:teuthology.orchestra.run.smithi187.stdout: librbd1-2:19.3.0-5104.g44178494.el9.x86_64 2024-09-21T13:40:14.761 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-09-21T13:40:14.761 INFO:teuthology.orchestra.run.smithi187.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-21T13:40:14.761 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-base-2:19.3.0-5104.g44178494.el9.x86_64 2024-09-21T13:40:14.761 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-common-2:19.3.0-5104.g44178494.el9.x86_64 2024-09-21T13:40:14.761 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-radosgw-2:19.3.0-5104.g44178494.el9.x86_64 2024-09-21T13:40:14.761 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-selinux-2:19.3.0-5104.g44178494.el9.x86_64 2024-09-21T13:40:14.761 INFO:teuthology.orchestra.run.smithi187.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-21T13:40:14.761 INFO:teuthology.orchestra.run.smithi187.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-21T13:40:14.761 INFO:teuthology.orchestra.run.smithi187.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-21T13:40:14.761 INFO:teuthology.orchestra.run.smithi187.stdout: libcephfs2-2:19.3.0-5104.g44178494.el9.x86_64 2024-09-21T13:40:14.761 INFO:teuthology.orchestra.run.smithi187.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-21T13:40:14.761 INFO:teuthology.orchestra.run.smithi187.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-21T13:40:14.761 INFO:teuthology.orchestra.run.smithi187.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-21T13:40:14.761 INFO:teuthology.orchestra.run.smithi187.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-21T13:40:14.762 INFO:teuthology.orchestra.run.smithi187.stdout: libradosstriper1-2:19.3.0-5104.g44178494.el9.x86_64 2024-09-21T13:40:14.762 INFO:teuthology.orchestra.run.smithi187.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-21T13:40:14.762 INFO:teuthology.orchestra.run.smithi187.stdout: librgw2-2:19.3.0-5104.g44178494.el9.x86_64 2024-09-21T13:40:14.762 INFO:teuthology.orchestra.run.smithi187.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-21T13:40:14.762 INFO:teuthology.orchestra.run.smithi187.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-21T13:40:14.762 INFO:teuthology.orchestra.run.smithi187.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-21T13:40:14.762 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ceph-argparse-2:19.3.0-5104.g44178494.el9.x86_64 2024-09-21T13:40:14.762 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ceph-common-2:19.3.0-5104.g44178494.el9.x86_64 2024-09-21T13:40:14.762 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cephfs-2:19.3.0-5104.g44178494.el9.x86_64 2024-09-21T13:40:14.762 INFO:teuthology.orchestra.run.smithi187.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-21T13:40:14.762 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rados-2:19.3.0-5104.g44178494.el9.x86_64 2024-09-21T13:40:14.762 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rbd-2:19.3.0-5104.g44178494.el9.x86_64 2024-09-21T13:40:14.762 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rgw-2:19.3.0-5104.g44178494.el9.x86_64 2024-09-21T13:40:14.762 INFO:teuthology.orchestra.run.smithi187.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-21T13:40:14.763 INFO:teuthology.orchestra.run.smithi187.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-21T13:40:14.763 INFO:teuthology.orchestra.run.smithi187.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-21T13:40:14.763 INFO:teuthology.orchestra.run.smithi187.stdout: re2-1:20211101-20.el9.x86_64 2024-09-21T13:40:14.763 INFO:teuthology.orchestra.run.smithi187.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-21T13:40:14.763 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:40:14.763 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-21T13:40:14.935 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-selinux-2:19.3.0-5104.g44178494.el9.x86_64 32/35 2024-09-21T13:40:14.935 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /sys 2024-09-21T13:40:14.936 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /proc 2024-09-21T13:40:14.936 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /mnt 2024-09-21T13:40:14.936 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /var/tmp 2024-09-21T13:40:14.936 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /home 2024-09-21T13:40:14.936 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /root 2024-09-21T13:40:14.936 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /tmp 2024-09-21T13:40:14.936 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:40:15.149 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-test 2024-09-21T13:40:15.731 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:00:27 ago on Sat 21 Sep 2024 01:39:48 PM UTC. 2024-09-21T13:40:15.863 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-21T13:40:15.863 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-21T13:40:15.864 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2024-09-21T13:40:15.864 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-21T13:40:15.864 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-09-21T13:40:15.864 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-test x86_64 2:19.3.0-5104.g44178494.el9 ceph 46 M 2024-09-21T13:40:15.864 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2024-09-21T13:40:15.864 INFO:teuthology.orchestra.run.smithi187.stdout: libcephsqlite x86_64 2:19.3.0-5104.g44178494.el9 ceph 161 k 2024-09-21T13:40:15.864 INFO:teuthology.orchestra.run.smithi187.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-21T13:40:15.864 INFO:teuthology.orchestra.run.smithi187.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-21T13:40:15.864 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:40:15.864 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-09-21T13:40:15.864 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-21T13:40:15.864 INFO:teuthology.orchestra.run.smithi187.stdout:Install 4 Packages 2024-09-21T13:40:15.865 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:40:15.865 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 46 M 2024-09-21T13:40:15.865 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 201 M 2024-09-21T13:40:15.865 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-09-21T13:40:15.976 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-radosgw-2:19.3.0-5104.g44178494.el9.x86_64 33/35 2024-09-21T13:40:16.004 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-5104.g44178494.el9.x86_64 33/35 2024-09-21T13:40:16.004 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T13:40:16.004 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-21T13:40:16.004 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T13:40:16.004 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T13:40:16.004 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:40:16.006 INFO:teuthology.orchestra.run.smithi049.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-21T13:40:16.103 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-21T13:40:16.103 INFO:teuthology.orchestra.run.smithi049.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-21T13:40:16.252 INFO:teuthology.orchestra.run.smithi187.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.3 MB/s | 304 kB 00:00 2024-09-21T13:40:16.285 INFO:teuthology.orchestra.run.smithi187.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 64 kB 00:00 2024-09-21T13:40:16.352 INFO:teuthology.orchestra.run.smithi187.stdout:(3/4): libcephsqlite-19.3.0-5104.g44178494.el9. 706 kB/s | 161 kB 00:00 2024-09-21T13:40:17.317 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-21T13:40:17.317 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-base-2:19.3.0-5104.g44178494.el9.x86_64 1/35 2024-09-21T13:40:17.317 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-common-2:19.3.0-5104.g44178494.el9.x86_64 2/35 2024-09-21T13:40:17.317 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-radosgw-2:19.3.0-5104.g44178494.el9.x86_64 3/35 2024-09-21T13:40:17.318 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-selinux-2:19.3.0-5104.g44178494.el9.x86_64 4/35 2024-09-21T13:40:17.318 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libcephfs2-2:19.3.0-5104.g44178494.el9.x86_64 5/35 2024-09-21T13:40:17.318 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libradosstriper1-2:19.3.0-5104.g44178494.el9.x86_6 6/35 2024-09-21T13:40:17.318 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librgw2-2:19.3.0-5104.g44178494.el9.x86_64 7/35 2024-09-21T13:40:17.318 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-ceph-argparse-2:19.3.0-5104.g44178494.el9. 8/35 2024-09-21T13:40:17.318 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-ceph-common-2:19.3.0-5104.g44178494.el9.x8 9/35 2024-09-21T13:40:17.318 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cephfs-2:19.3.0-5104.g44178494.el9.x86_64 10/35 2024-09-21T13:40:17.318 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-rados-2:19.3.0-5104.g44178494.el9.x86_64 11/35 2024-09-21T13:40:17.318 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-rbd-2:19.3.0-5104.g44178494.el9.x86_64 12/35 2024-09-21T13:40:17.318 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-rgw-2:19.3.0-5104.g44178494.el9.x86_64 13/35 2024-09-21T13:40:17.318 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-21T13:40:17.318 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-21T13:40:17.318 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-21T13:40:17.319 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-21T13:40:17.319 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-21T13:40:17.319 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-21T13:40:17.319 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-21T13:40:17.319 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-21T13:40:17.319 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-21T13:40:17.319 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-21T13:40:17.319 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 24/35 2024-09-21T13:40:17.319 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-21T13:40:17.319 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-21T13:40:17.319 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-21T13:40:17.320 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-21T13:40:17.320 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-21T13:40:17.320 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-21T13:40:17.321 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-21T13:40:17.321 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librados2-2:19.3.0-5104.g44178494.el9.x86_64 32/35 2024-09-21T13:40:17.321 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-21T13:40:17.321 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librbd1-2:19.3.0-5104.g44178494.el9.x86_64 34/35 2024-09-21T13:40:18.168 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-21T13:40:18.168 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:40:18.168 INFO:teuthology.orchestra.run.smithi049.stdout:Upgraded: 2024-09-21T13:40:18.168 INFO:teuthology.orchestra.run.smithi049.stdout: librados2-2:19.3.0-5104.g44178494.el9.x86_64 2024-09-21T13:40:18.168 INFO:teuthology.orchestra.run.smithi049.stdout: librbd1-2:19.3.0-5104.g44178494.el9.x86_64 2024-09-21T13:40:18.168 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-09-21T13:40:18.169 INFO:teuthology.orchestra.run.smithi049.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-21T13:40:18.169 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-base-2:19.3.0-5104.g44178494.el9.x86_64 2024-09-21T13:40:18.169 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-common-2:19.3.0-5104.g44178494.el9.x86_64 2024-09-21T13:40:18.169 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-radosgw-2:19.3.0-5104.g44178494.el9.x86_64 2024-09-21T13:40:18.169 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-selinux-2:19.3.0-5104.g44178494.el9.x86_64 2024-09-21T13:40:18.169 INFO:teuthology.orchestra.run.smithi049.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-21T13:40:18.169 INFO:teuthology.orchestra.run.smithi049.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-21T13:40:18.169 INFO:teuthology.orchestra.run.smithi049.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-21T13:40:18.169 INFO:teuthology.orchestra.run.smithi049.stdout: libcephfs2-2:19.3.0-5104.g44178494.el9.x86_64 2024-09-21T13:40:18.169 INFO:teuthology.orchestra.run.smithi049.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-21T13:40:18.169 INFO:teuthology.orchestra.run.smithi049.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-21T13:40:18.169 INFO:teuthology.orchestra.run.smithi049.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-21T13:40:18.169 INFO:teuthology.orchestra.run.smithi049.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-21T13:40:18.169 INFO:teuthology.orchestra.run.smithi049.stdout: libradosstriper1-2:19.3.0-5104.g44178494.el9.x86_64 2024-09-21T13:40:18.169 INFO:teuthology.orchestra.run.smithi049.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-21T13:40:18.170 INFO:teuthology.orchestra.run.smithi049.stdout: librgw2-2:19.3.0-5104.g44178494.el9.x86_64 2024-09-21T13:40:18.170 INFO:teuthology.orchestra.run.smithi049.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-21T13:40:18.170 INFO:teuthology.orchestra.run.smithi049.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-21T13:40:18.170 INFO:teuthology.orchestra.run.smithi049.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-21T13:40:18.170 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ceph-argparse-2:19.3.0-5104.g44178494.el9.x86_64 2024-09-21T13:40:18.170 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ceph-common-2:19.3.0-5104.g44178494.el9.x86_64 2024-09-21T13:40:18.170 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cephfs-2:19.3.0-5104.g44178494.el9.x86_64 2024-09-21T13:40:18.170 INFO:teuthology.orchestra.run.smithi049.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-21T13:40:18.170 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rados-2:19.3.0-5104.g44178494.el9.x86_64 2024-09-21T13:40:18.170 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rbd-2:19.3.0-5104.g44178494.el9.x86_64 2024-09-21T13:40:18.170 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rgw-2:19.3.0-5104.g44178494.el9.x86_64 2024-09-21T13:40:18.170 INFO:teuthology.orchestra.run.smithi049.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-21T13:40:18.170 INFO:teuthology.orchestra.run.smithi049.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-21T13:40:18.171 INFO:teuthology.orchestra.run.smithi049.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-21T13:40:18.171 INFO:teuthology.orchestra.run.smithi049.stdout: re2-1:20211101-20.el9.x86_64 2024-09-21T13:40:18.171 INFO:teuthology.orchestra.run.smithi049.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-21T13:40:18.171 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:40:18.171 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-09-21T13:40:18.219 INFO:teuthology.orchestra.run.smithi187.stdout:(4/4): ceph-test-19.3.0-5104.g44178494.el9.x86_ 22 MB/s | 46 MB 00:02 2024-09-21T13:40:18.221 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-09-21T13:40:18.221 INFO:teuthology.orchestra.run.smithi187.stdout:Total 20 MB/s | 46 MB 00:02 2024-09-21T13:40:18.250 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-09-21T13:40:18.270 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-09-21T13:40:18.270 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-09-21T13:40:18.523 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-test 2024-09-21T13:40:18.632 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-09-21T13:40:18.632 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-09-21T13:40:19.131 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:00:31 ago on Sat 21 Sep 2024 01:39:48 PM UTC. 2024-09-21T13:40:19.189 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-09-21T13:40:19.267 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-09-21T13:40:19.268 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-09-21T13:40:19.268 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2024-09-21T13:40:19.268 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-09-21T13:40:19.268 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-09-21T13:40:19.268 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-test x86_64 2:19.3.0-5104.g44178494.el9 ceph 46 M 2024-09-21T13:40:19.268 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2024-09-21T13:40:19.268 INFO:teuthology.orchestra.run.smithi049.stdout: libcephsqlite x86_64 2:19.3.0-5104.g44178494.el9 ceph 161 k 2024-09-21T13:40:19.268 INFO:teuthology.orchestra.run.smithi049.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-21T13:40:19.269 INFO:teuthology.orchestra.run.smithi049.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-21T13:40:19.269 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:40:19.269 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-09-21T13:40:19.269 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-09-21T13:40:19.269 INFO:teuthology.orchestra.run.smithi049.stdout:Install 4 Packages 2024-09-21T13:40:19.269 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:40:19.269 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 46 M 2024-09-21T13:40:19.270 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 201 M 2024-09-21T13:40:19.270 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-09-21T13:40:19.288 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-21T13:40:19.334 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-21T13:40:19.363 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libcephsqlite-2:19.3.0-5104.g44178494.el9.x86_64 3/4 2024-09-21T13:40:19.582 INFO:teuthology.orchestra.run.smithi049.stdout:(1/4): libcephsqlite-19.3.0-5104.g44178494.el9. 762 kB/s | 161 kB 00:00 2024-09-21T13:40:20.166 INFO:teuthology.orchestra.run.smithi049.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 382 kB/s | 304 kB 00:00 2024-09-21T13:40:20.207 INFO:teuthology.orchestra.run.smithi049.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 102 kB/s | 64 kB 00:00 2024-09-21T13:40:20.808 INFO:teuthology.orchestra.run.smithi049.stdout:(4/4): ceph-test-19.3.0-5104.g44178494.el9.x86_ 32 MB/s | 46 MB 00:01 2024-09-21T13:40:20.810 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-09-21T13:40:20.810 INFO:teuthology.orchestra.run.smithi049.stdout:Total 30 MB/s | 46 MB 00:01 2024-09-21T13:40:20.839 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-09-21T13:40:20.862 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-09-21T13:40:20.862 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-09-21T13:40:21.245 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-09-21T13:40:21.245 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-09-21T13:40:21.824 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-09-21T13:40:21.916 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-21T13:40:21.965 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-21T13:40:21.993 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libcephsqlite-2:19.3.0-5104.g44178494.el9.x86_64 3/4 2024-09-21T13:40:23.805 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libcephsqlite-2:19.3.0-5104.g44178494.el9.x86_64 3/4 2024-09-21T13:40:23.877 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-test-2:19.3.0-5104.g44178494.el9.x86_64 4/4 2024-09-21T13:40:26.271 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-test-2:19.3.0-5104.g44178494.el9.x86_64 4/4 2024-09-21T13:40:26.271 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-test-2:19.3.0-5104.g44178494.el9.x86_64 1/4 2024-09-21T13:40:26.271 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libcephsqlite-2:19.3.0-5104.g44178494.el9.x86_64 2/4 2024-09-21T13:40:26.271 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-21T13:40:26.563 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-21T13:40:26.563 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:40:26.563 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-09-21T13:40:26.563 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-test-2:19.3.0-5104.g44178494.el9.x86_64 2024-09-21T13:40:26.563 INFO:teuthology.orchestra.run.smithi187.stdout: libcephsqlite-2:19.3.0-5104.g44178494.el9.x86_64 2024-09-21T13:40:26.563 INFO:teuthology.orchestra.run.smithi187.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-21T13:40:26.563 INFO:teuthology.orchestra.run.smithi187.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-21T13:40:26.564 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:40:26.564 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-21T13:40:26.639 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libcephsqlite-2:19.3.0-5104.g44178494.el9.x86_64 3/4 2024-09-21T13:40:26.736 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-test-2:19.3.0-5104.g44178494.el9.x86_64 4/4 2024-09-21T13:40:26.853 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph 2024-09-21T13:40:27.459 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:00:39 ago on Sat 21 Sep 2024 01:39:48 PM UTC. 2024-09-21T13:40:27.603 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-21T13:40:27.606 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-21T13:40:27.606 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2024-09-21T13:40:27.606 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-21T13:40:27.606 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-09-21T13:40:27.606 INFO:teuthology.orchestra.run.smithi187.stdout: ceph x86_64 2:19.3.0-5104.g44178494.el9 ceph 6.6 k 2024-09-21T13:40:27.606 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2024-09-21T13:40:27.606 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mds x86_64 2:19.3.0-5104.g44178494.el9 ceph 2.4 M 2024-09-21T13:40:27.606 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr x86_64 2:19.3.0-5104.g44178494.el9 ceph 1.5 M 2024-09-21T13:40:27.606 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-modules-core noarch 2:19.3.0-5104.g44178494.el9 ceph-noarch 292 k 2024-09-21T13:40:27.606 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mon x86_64 2:19.3.0-5104.g44178494.el9 ceph 4.8 M 2024-09-21T13:40:27.606 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-osd x86_64 2:19.3.0-5104.g44178494.el9 ceph 17 M 2024-09-21T13:40:27.607 INFO:teuthology.orchestra.run.smithi187.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-21T13:40:27.607 INFO:teuthology.orchestra.run.smithi187.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-21T13:40:27.607 INFO:teuthology.orchestra.run.smithi187.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-21T13:40:27.607 INFO:teuthology.orchestra.run.smithi187.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-21T13:40:27.607 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-21T13:40:27.607 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-21T13:40:27.607 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-21T13:40:27.607 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-21T13:40:27.607 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-21T13:40:27.607 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-21T13:40:27.607 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-collections 2024-09-21T13:40:27.607 INFO:teuthology.orchestra.run.smithi187.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-21T13:40:27.607 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-21T13:40:27.608 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-21T13:40:27.608 INFO:teuthology.orchestra.run.smithi187.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-21T13:40:27.608 INFO:teuthology.orchestra.run.smithi187.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-21T13:40:27.608 INFO:teuthology.orchestra.run.smithi187.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-21T13:40:27.608 INFO:teuthology.orchestra.run.smithi187.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-21T13:40:27.608 INFO:teuthology.orchestra.run.smithi187.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-21T13:40:27.608 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-21T13:40:27.608 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-21T13:40:27.608 INFO:teuthology.orchestra.run.smithi187.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-21T13:40:27.608 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-21T13:40:27.608 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-21T13:40:27.608 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-21T13:40:27.608 INFO:teuthology.orchestra.run.smithi187.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-21T13:40:27.608 INFO:teuthology.orchestra.run.smithi187.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-21T13:40:27.608 INFO:teuthology.orchestra.run.smithi187.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-21T13:40:27.609 INFO:teuthology.orchestra.run.smithi187.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-21T13:40:27.609 INFO:teuthology.orchestra.run.smithi187.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-21T13:40:27.609 INFO:teuthology.orchestra.run.smithi187.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-21T13:40:27.609 INFO:teuthology.orchestra.run.smithi187.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-21T13:40:27.609 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:40:27.609 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-09-21T13:40:27.609 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-21T13:40:27.609 INFO:teuthology.orchestra.run.smithi187.stdout:Install 37 Packages 2024-09-21T13:40:27.609 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:40:27.610 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 31 M 2024-09-21T13:40:27.610 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 109 M 2024-09-21T13:40:27.610 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-09-21T13:40:28.245 INFO:teuthology.orchestra.run.smithi187.stdout:(1/37): ceph-19.3.0-5104.g44178494.el9.x86_64.r 56 kB/s | 6.6 kB 00:00 2024-09-21T13:40:28.437 INFO:teuthology.orchestra.run.smithi187.stdout:(2/37): ceph-mgr-19.3.0-5104.g44178494.el9.x86_ 4.9 MB/s | 1.5 MB 00:00 2024-09-21T13:40:28.496 INFO:teuthology.orchestra.run.smithi187.stdout:(3/37): ceph-mds-19.3.0-5104.g44178494.el9.x86_ 6.5 MB/s | 2.4 MB 00:00 2024-09-21T13:40:28.571 INFO:teuthology.orchestra.run.smithi187.stdout:(4/37): ceph-mgr-modules-core-19.3.0-5104.g4417 3.8 MB/s | 292 kB 00:00 2024-09-21T13:40:28.646 INFO:teuthology.orchestra.run.smithi187.stdout:(5/37): ceph-mon-19.3.0-5104.g44178494.el9.x86_ 12 MB/s | 4.8 MB 00:00 2024-09-21T13:40:28.788 INFO:teuthology.orchestra.run.smithi187.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.1 MB/s | 253 kB 00:00 2024-09-21T13:40:28.838 INFO:teuthology.orchestra.run.smithi187.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-09-21T13:40:28.897 INFO:teuthology.orchestra.run.smithi187.stdout:(8/37): python3-cryptography-36.0.1-4.el9.x86_6 4.9 MB/s | 1.2 MB 00:00 2024-09-21T13:40:28.931 INFO:teuthology.orchestra.run.smithi187.stdout:(9/37): python3-pycparser-2.20-6.el9.noarch.rpm 1.4 MB/s | 135 kB 00:00 2024-09-21T13:40:28.956 INFO:teuthology.orchestra.run.smithi187.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 2.1 MB/s | 125 kB 00:00 2024-09-21T13:40:29.123 INFO:teuthology.orchestra.run.smithi187.stdout:(11/37): ceph-osd-19.3.0-5104.g44178494.el9.x86 25 MB/s | 17 MB 00:00 2024-09-21T13:40:29.148 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-test-2:19.3.0-5104.g44178494.el9.x86_64 4/4 2024-09-21T13:40:29.148 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-test-2:19.3.0-5104.g44178494.el9.x86_64 1/4 2024-09-21T13:40:29.148 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libcephsqlite-2:19.3.0-5104.g44178494.el9.x86_64 2/4 2024-09-21T13:40:29.148 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-21T13:40:29.148 INFO:teuthology.orchestra.run.smithi187.stdout:(12/37): python3-urllib3-1.26.5-6.el9.noarch.rp 988 kB/s | 215 kB 00:00 2024-09-21T13:40:29.424 INFO:teuthology.orchestra.run.smithi187.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 126 kB/s | 35 kB 00:00 2024-09-21T13:40:29.466 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-21T13:40:29.466 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:40:29.466 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-09-21T13:40:29.467 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-test-2:19.3.0-5104.g44178494.el9.x86_64 2024-09-21T13:40:29.467 INFO:teuthology.orchestra.run.smithi049.stdout: libcephsqlite-2:19.3.0-5104.g44178494.el9.x86_64 2024-09-21T13:40:29.467 INFO:teuthology.orchestra.run.smithi049.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-21T13:40:29.467 INFO:teuthology.orchestra.run.smithi049.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-21T13:40:29.467 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:40:29.467 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-09-21T13:40:29.624 INFO:teuthology.orchestra.run.smithi187.stdout:(14/37): python3-packaging-20.9-5.el9.noarch.rp 386 kB/s | 77 kB 00:00 2024-09-21T13:40:29.666 INFO:teuthology.orchestra.run.smithi187.stdout:(15/37): python3-mako-1.1.4-6.el9.noarch.rpm 317 kB/s | 172 kB 00:00 2024-09-21T13:40:29.691 INFO:teuthology.orchestra.run.smithi187.stdout:(16/37): lua-5.4.4-4.el9.x86_64.rpm 256 kB/s | 188 kB 00:00 2024-09-21T13:40:29.693 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph 2024-09-21T13:40:29.733 INFO:teuthology.orchestra.run.smithi187.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 468 kB/s | 51 kB 00:00 2024-09-21T13:40:29.775 INFO:teuthology.orchestra.run.smithi187.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 382 kB/s | 42 kB 00:00 2024-09-21T13:40:29.800 INFO:teuthology.orchestra.run.smithi187.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-09-21T13:40:29.858 INFO:teuthology.orchestra.run.smithi187.stdout:(20/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 3.4 MB/s | 172 kB 00:00 2024-09-21T13:40:29.884 INFO:teuthology.orchestra.run.smithi187.stdout:(21/37): lua-devel-5.4.4-4.el9.x86_64.rpm 116 kB/s | 22 kB 00:00 2024-09-21T13:40:29.909 INFO:teuthology.orchestra.run.smithi187.stdout:(22/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 324 kB/s | 43 kB 00:00 2024-09-21T13:40:29.943 INFO:teuthology.orchestra.run.smithi187.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 689 kB/s | 18 kB 00:00 2024-09-21T13:40:29.968 INFO:teuthology.orchestra.run.smithi187.stdout:(24/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 128 kB/s | 11 kB 00:00 2024-09-21T13:40:29.993 INFO:teuthology.orchestra.run.smithi187.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 478 kB/s | 23 kB 00:00 2024-09-21T13:40:30.018 INFO:teuthology.orchestra.run.smithi187.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 783 kB/s | 19 kB 00:00 2024-09-21T13:40:30.044 INFO:teuthology.orchestra.run.smithi187.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-09-21T13:40:30.069 INFO:teuthology.orchestra.run.smithi187.stdout:(28/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.7 MB/s | 358 kB 00:00 2024-09-21T13:40:30.095 INFO:teuthology.orchestra.run.smithi187.stdout:(29/37): python3-logutils-0.3.5-21.el9.noarch.r 611 kB/s | 46 kB 00:00 2024-09-21T13:40:30.120 INFO:teuthology.orchestra.run.smithi187.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-09-21T13:40:30.145 INFO:teuthology.orchestra.run.smithi187.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 327 kB/s | 16 kB 00:00 2024-09-21T13:40:30.170 INFO:teuthology.orchestra.run.smithi187.stdout:(32/37): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-09-21T13:40:30.196 INFO:teuthology.orchestra.run.smithi187.stdout:(33/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-09-21T13:40:30.229 INFO:teuthology.orchestra.run.smithi187.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-09-21T13:40:30.246 INFO:teuthology.orchestra.run.smithi187.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 1.2 MB/s | 20 kB 00:00 2024-09-21T13:40:30.271 INFO:teuthology.orchestra.run.smithi187.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-21T13:40:30.305 INFO:teuthology.orchestra.run.smithi187.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-09-21T13:40:30.309 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-09-21T13:40:30.310 INFO:teuthology.orchestra.run.smithi187.stdout:Total 11 MB/s | 31 MB 00:02 2024-09-21T13:40:30.312 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:00:42 ago on Sat 21 Sep 2024 01:39:48 PM UTC. 2024-09-21T13:40:30.456 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-09-21T13:40:30.459 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-09-21T13:40:30.459 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2024-09-21T13:40:30.459 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-09-21T13:40:30.459 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-09-21T13:40:30.459 INFO:teuthology.orchestra.run.smithi049.stdout: ceph x86_64 2:19.3.0-5104.g44178494.el9 ceph 6.6 k 2024-09-21T13:40:30.459 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2024-09-21T13:40:30.459 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mds x86_64 2:19.3.0-5104.g44178494.el9 ceph 2.4 M 2024-09-21T13:40:30.460 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr x86_64 2:19.3.0-5104.g44178494.el9 ceph 1.5 M 2024-09-21T13:40:30.460 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-modules-core noarch 2:19.3.0-5104.g44178494.el9 ceph-noarch 292 k 2024-09-21T13:40:30.460 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mon x86_64 2:19.3.0-5104.g44178494.el9 ceph 4.8 M 2024-09-21T13:40:30.460 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-osd x86_64 2:19.3.0-5104.g44178494.el9 ceph 17 M 2024-09-21T13:40:30.460 INFO:teuthology.orchestra.run.smithi049.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-21T13:40:30.460 INFO:teuthology.orchestra.run.smithi049.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-21T13:40:30.460 INFO:teuthology.orchestra.run.smithi049.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-21T13:40:30.460 INFO:teuthology.orchestra.run.smithi049.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-21T13:40:30.460 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-21T13:40:30.460 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-21T13:40:30.460 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-21T13:40:30.460 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-21T13:40:30.460 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-21T13:40:30.460 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-21T13:40:30.461 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-collections 2024-09-21T13:40:30.461 INFO:teuthology.orchestra.run.smithi049.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-21T13:40:30.461 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-21T13:40:30.461 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-21T13:40:30.461 INFO:teuthology.orchestra.run.smithi049.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-21T13:40:30.461 INFO:teuthology.orchestra.run.smithi049.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-21T13:40:30.461 INFO:teuthology.orchestra.run.smithi049.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-21T13:40:30.461 INFO:teuthology.orchestra.run.smithi049.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-21T13:40:30.461 INFO:teuthology.orchestra.run.smithi049.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-21T13:40:30.461 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-21T13:40:30.461 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-21T13:40:30.461 INFO:teuthology.orchestra.run.smithi049.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-21T13:40:30.461 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-21T13:40:30.462 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-21T13:40:30.462 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-21T13:40:30.462 INFO:teuthology.orchestra.run.smithi049.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-21T13:40:30.462 INFO:teuthology.orchestra.run.smithi049.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-21T13:40:30.462 INFO:teuthology.orchestra.run.smithi049.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-21T13:40:30.462 INFO:teuthology.orchestra.run.smithi049.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-21T13:40:30.462 INFO:teuthology.orchestra.run.smithi049.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-21T13:40:30.462 INFO:teuthology.orchestra.run.smithi049.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-21T13:40:30.462 INFO:teuthology.orchestra.run.smithi049.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-21T13:40:30.463 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:40:30.463 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-09-21T13:40:30.463 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-09-21T13:40:30.463 INFO:teuthology.orchestra.run.smithi049.stdout:Install 37 Packages 2024-09-21T13:40:30.463 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:40:30.463 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-09-21T13:40:30.464 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 31 M 2024-09-21T13:40:30.464 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 109 M 2024-09-21T13:40:30.464 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-09-21T13:40:30.503 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-09-21T13:40:30.503 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-09-21T13:40:30.810 INFO:teuthology.orchestra.run.smithi049.stdout:(1/37): ceph-19.3.0-5104.g44178494.el9.x86_64.r 59 kB/s | 6.6 kB 00:00 2024-09-21T13:40:30.963 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-09-21T13:40:30.964 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-09-21T13:40:31.002 INFO:teuthology.orchestra.run.smithi049.stdout:(2/37): ceph-mgr-19.3.0-5104.g44178494.el9.x86_ 5.0 MB/s | 1.5 MB 00:00 2024-09-21T13:40:31.052 INFO:teuthology.orchestra.run.smithi049.stdout:(3/37): ceph-mds-19.3.0-5104.g44178494.el9.x86_ 6.7 MB/s | 2.4 MB 00:00 2024-09-21T13:40:31.119 INFO:teuthology.orchestra.run.smithi049.stdout:(4/37): ceph-mgr-modules-core-19.3.0-5104.g4417 4.3 MB/s | 292 kB 00:00 2024-09-21T13:40:31.186 INFO:teuthology.orchestra.run.smithi049.stdout:(5/37): ceph-mon-19.3.0-5104.g44178494.el9.x86_ 13 MB/s | 4.8 MB 00:00 2024-09-21T13:40:31.553 INFO:teuthology.orchestra.run.smithi049.stdout:(6/37): ceph-osd-19.3.0-5104.g44178494.el9.x86_ 31 MB/s | 17 MB 00:00 2024-09-21T13:40:32.145 INFO:teuthology.orchestra.run.smithi049.stdout:(7/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 247 kB/s | 253 kB 00:01 2024-09-21T13:40:32.253 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-09-21T13:40:32.270 INFO:teuthology.orchestra.run.smithi049.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 152 kB/s | 106 kB 00:00 2024-09-21T13:40:32.320 INFO:teuthology.orchestra.run.smithi049.stdout:(9/37): python3-cryptography-36.0.1-4.el9.x86_6 1.1 MB/s | 1.2 MB 00:01 2024-09-21T13:40:32.324 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-21T13:40:32.357 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-21T13:40:32.385 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-21T13:40:32.465 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-21T13:40:32.500 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-21T13:40:32.533 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-21T13:40:32.626 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-21T13:40:32.704 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-21T13:40:32.744 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-21T13:40:32.779 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-21T13:40:32.809 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-21T13:40:32.843 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-21T13:40:32.877 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-21T13:40:32.926 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-21T13:40:32.929 INFO:teuthology.orchestra.run.smithi049.stdout:(10/37): python3-pycparser-2.20-6.el9.noarch.rp 172 kB/s | 135 kB 00:00 2024-09-21T13:40:32.969 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-21T13:40:33.004 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-21T13:40:33.004 INFO:teuthology.orchestra.run.smithi049.stdout:(11/37): python3-requests-2.25.1-8.el9.noarch.r 170 kB/s | 125 kB 00:00 2024-09-21T13:40:33.052 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-21T13:40:33.063 INFO:teuthology.orchestra.run.smithi049.stdout:(12/37): python3-urllib3-1.26.5-6.el9.noarch.rp 289 kB/s | 215 kB 00:00 2024-09-21T13:40:33.084 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-21T13:40:33.113 INFO:teuthology.orchestra.run.smithi049.stdout:(13/37): lua-5.4.4-4.el9.x86_64.rpm 1.0 MB/s | 188 kB 00:00 2024-09-21T13:40:33.157 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-21T13:40:33.172 INFO:teuthology.orchestra.run.smithi049.stdout:(14/37): python3-mako-1.1.4-6.el9.noarch.rpm 1.0 MB/s | 172 kB 00:00 2024-09-21T13:40:33.197 INFO:teuthology.orchestra.run.smithi049.stdout:(15/37): python3-markupsafe-1.1.1-12.el9.x86_64 260 kB/s | 35 kB 00:00 2024-09-21T13:40:33.223 INFO:teuthology.orchestra.run.smithi049.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 710 kB/s | 77 kB 00:00 2024-09-21T13:40:33.265 INFO:teuthology.orchestra.run.smithi049.stdout:(17/37): python3-toml-0.10.2-6.el9.noarch.rpm 624 kB/s | 42 kB 00:00 2024-09-21T13:40:33.290 INFO:teuthology.orchestra.run.smithi049.stdout:(18/37): python3-pytz-2021.1-5.el9.noarch.rpm 435 kB/s | 51 kB 00:00 2024-09-21T13:40:33.298 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-21T13:40:33.332 INFO:teuthology.orchestra.run.smithi049.stdout:(19/37): lua-devel-5.4.4-4.el9.x86_64.rpm 205 kB/s | 22 kB 00:00 2024-09-21T13:40:33.365 INFO:teuthology.orchestra.run.smithi049.stdout:(20/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 578 kB/s | 43 kB 00:00 2024-09-21T13:40:33.385 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-21T13:40:33.390 INFO:teuthology.orchestra.run.smithi049.stdout:(21/37): luarocks-3.9.2-5.el9.noarch.rpm 1.2 MB/s | 151 kB 00:00 2024-09-21T13:40:33.416 INFO:teuthology.orchestra.run.smithi049.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 433 kB/s | 11 kB 00:00 2024-09-21T13:40:33.441 INFO:teuthology.orchestra.run.smithi049.stdout:(23/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.6 MB/s | 172 kB 00:00 2024-09-21T13:40:33.445 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-21T13:40:33.466 INFO:teuthology.orchestra.run.smithi049.stdout:(24/37): python3-jaraco-classes-3.2.1-5.el9.noa 352 kB/s | 18 kB 00:00 2024-09-21T13:40:33.483 INFO:teuthology.orchestra.run.smithi049.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 1.4 MB/s | 23 kB 00:00 2024-09-21T13:40:33.483 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-21T13:40:33.500 INFO:teuthology.orchestra.run.smithi049.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 584 kB/s | 19 kB 00:00 2024-09-21T13:40:33.517 INFO:teuthology.orchestra.run.smithi049.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 586 kB/s | 20 kB 00:00 2024-09-21T13:40:33.533 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-21T13:40:33.534 INFO:teuthology.orchestra.run.smithi049.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 1.4 MB/s | 46 kB 00:00 2024-09-21T13:40:33.567 INFO:teuthology.orchestra.run.smithi049.stdout:(29/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.7 MB/s | 358 kB 00:00 2024-09-21T13:40:33.589 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-21T13:40:33.593 INFO:teuthology.orchestra.run.smithi049.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-09-21T13:40:33.610 INFO:teuthology.orchestra.run.smithi049.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 391 kB/s | 16 kB 00:00 2024-09-21T13:40:33.635 INFO:teuthology.orchestra.run.smithi049.stdout:(32/37): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-09-21T13:40:33.660 INFO:teuthology.orchestra.run.smithi049.stdout:(33/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.3 MB/s | 90 kB 00:00 2024-09-21T13:40:33.686 INFO:teuthology.orchestra.run.smithi049.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-21T13:40:33.711 INFO:teuthology.orchestra.run.smithi049.stdout:(35/37): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-09-21T13:40:33.736 INFO:teuthology.orchestra.run.smithi049.stdout:(36/37): python3-zc-lockfile-2.0-10.el9.noarch. 398 kB/s | 20 kB 00:00 2024-09-21T13:40:33.761 INFO:teuthology.orchestra.run.smithi049.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-09-21T13:40:33.767 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-09-21T13:40:33.767 INFO:teuthology.orchestra.run.smithi049.stdout:Total 9.3 MB/s | 31 MB 00:03 2024-09-21T13:40:33.779 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-21T13:40:33.829 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-21T13:40:33.908 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-21T13:40:33.929 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-09-21T13:40:33.974 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-09-21T13:40:33.974 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-09-21T13:40:34.005 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-21T13:40:34.053 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-21T13:40:34.186 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-21T13:40:34.343 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-5104.g44178494.el9. 32/37 2024-09-21T13:40:34.380 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-2:19.3.0-5104.g44178494.el9.x86_64 33/37 2024-09-21T13:40:34.407 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mgr-2:19.3.0-5104.g44178494.el9.x86_64 33/37 2024-09-21T13:40:34.407 INFO:teuthology.orchestra.run.smithi187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T13:40:34.407 INFO:teuthology.orchestra.run.smithi187.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-21T13:40:34.407 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T13:40:34.407 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T13:40:34.407 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:40:34.466 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-09-21T13:40:34.466 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-09-21T13:40:35.790 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-09-21T13:40:35.856 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-21T13:40:35.890 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-osd-2:19.3.0-5104.g44178494.el9.x86_64 34/37 2024-09-21T13:40:35.897 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-21T13:40:35.917 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-osd-2:19.3.0-5104.g44178494.el9.x86_64 34/37 2024-09-21T13:40:35.917 INFO:teuthology.orchestra.run.smithi187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T13:40:35.917 INFO:teuthology.orchestra.run.smithi187.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-21T13:40:35.917 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T13:40:35.917 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T13:40:35.917 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:40:35.933 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-21T13:40:36.008 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-21T13:40:36.058 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-21T13:40:36.090 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-21T13:40:36.187 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-21T13:40:36.281 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-21T13:40:36.327 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-21T13:40:36.362 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-21T13:40:36.373 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mon-2:19.3.0-5104.g44178494.el9.x86_64 35/37 2024-09-21T13:40:36.391 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-21T13:40:36.397 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mon-2:19.3.0-5104.g44178494.el9.x86_64 35/37 2024-09-21T13:40:36.397 INFO:teuthology.orchestra.run.smithi187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T13:40:36.397 INFO:teuthology.orchestra.run.smithi187.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-21T13:40:36.397 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T13:40:36.397 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T13:40:36.397 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:40:36.426 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-21T13:40:36.459 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-21T13:40:36.500 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-21T13:40:36.552 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-21T13:40:36.594 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-21T13:40:36.624 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mds-2:19.3.0-5104.g44178494.el9.x86_64 36/37 2024-09-21T13:40:36.644 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-21T13:40:36.648 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mds-2:19.3.0-5104.g44178494.el9.x86_64 36/37 2024-09-21T13:40:36.648 INFO:teuthology.orchestra.run.smithi187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T13:40:36.648 INFO:teuthology.orchestra.run.smithi187.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-21T13:40:36.648 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T13:40:36.648 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T13:40:36.648 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:40:36.691 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-21T13:40:36.721 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-2:19.3.0-5104.g44178494.el9.x86_64 37/37 2024-09-21T13:40:36.768 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-21T13:40:36.920 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-21T13:40:37.011 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-21T13:40:37.070 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-21T13:40:37.108 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-21T13:40:37.174 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-21T13:40:37.247 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-21T13:40:37.453 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-21T13:40:37.495 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-21T13:40:37.576 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-21T13:40:37.674 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-21T13:40:37.727 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-21T13:40:37.864 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-21T13:40:38.032 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-5104.g44178494.el9. 32/37 2024-09-21T13:40:38.062 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-2:19.3.0-5104.g44178494.el9.x86_64 33/37 2024-09-21T13:40:38.090 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mgr-2:19.3.0-5104.g44178494.el9.x86_64 33/37 2024-09-21T13:40:38.090 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T13:40:38.090 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-21T13:40:38.090 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T13:40:38.090 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T13:40:38.090 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:40:38.622 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-2:19.3.0-5104.g44178494.el9.x86_64 37/37 2024-09-21T13:40:38.623 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-2:19.3.0-5104.g44178494.el9.x86_64 1/37 2024-09-21T13:40:38.623 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mds-2:19.3.0-5104.g44178494.el9.x86_64 2/37 2024-09-21T13:40:38.623 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-2:19.3.0-5104.g44178494.el9.x86_64 3/37 2024-09-21T13:40:38.623 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mon-2:19.3.0-5104.g44178494.el9.x86_64 4/37 2024-09-21T13:40:38.623 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-osd-2:19.3.0-5104.g44178494.el9.x86_64 5/37 2024-09-21T13:40:38.623 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-5104.g44178494.el9. 6/37 2024-09-21T13:40:38.623 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-21T13:40:38.623 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-21T13:40:38.623 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-21T13:40:38.623 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-21T13:40:38.623 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-21T13:40:38.624 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-21T13:40:38.624 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-21T13:40:38.624 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-21T13:40:38.624 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-21T13:40:38.624 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-21T13:40:38.624 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-21T13:40:38.624 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-21T13:40:38.624 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-21T13:40:38.624 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-21T13:40:38.624 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-21T13:40:38.624 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-21T13:40:38.624 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-21T13:40:38.624 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-21T13:40:38.624 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-21T13:40:38.625 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-21T13:40:38.625 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-21T13:40:38.625 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-21T13:40:38.625 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-21T13:40:38.625 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-21T13:40:38.625 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-21T13:40:38.625 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-21T13:40:38.625 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-21T13:40:38.625 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-21T13:40:38.625 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-21T13:40:38.625 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-21T13:40:39.640 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-osd-2:19.3.0-5104.g44178494.el9.x86_64 34/37 2024-09-21T13:40:39.669 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-osd-2:19.3.0-5104.g44178494.el9.x86_64 34/37 2024-09-21T13:40:39.669 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T13:40:39.669 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-21T13:40:39.670 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T13:40:39.670 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T13:40:39.670 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:40:39.744 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-21T13:40:39.744 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:40:39.744 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-09-21T13:40:39.744 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-2:19.3.0-5104.g44178494.el9.x86_64 2024-09-21T13:40:39.744 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mds-2:19.3.0-5104.g44178494.el9.x86_64 2024-09-21T13:40:39.744 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-2:19.3.0-5104.g44178494.el9.x86_64 2024-09-21T13:40:39.744 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-modules-core-2:19.3.0-5104.g44178494.el9.noarch 2024-09-21T13:40:39.744 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mon-2:19.3.0-5104.g44178494.el9.x86_64 2024-09-21T13:40:39.744 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-osd-2:19.3.0-5104.g44178494.el9.x86_64 2024-09-21T13:40:39.744 INFO:teuthology.orchestra.run.smithi187.stdout: lua-5.4.4-4.el9.x86_64 2024-09-21T13:40:39.744 INFO:teuthology.orchestra.run.smithi187.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-21T13:40:39.745 INFO:teuthology.orchestra.run.smithi187.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-21T13:40:39.745 INFO:teuthology.orchestra.run.smithi187.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-21T13:40:39.745 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-21T13:40:39.745 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-21T13:40:39.745 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-21T13:40:39.745 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-21T13:40:39.745 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-21T13:40:39.745 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-21T13:40:39.745 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-21T13:40:39.745 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-21T13:40:39.745 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-21T13:40:39.745 INFO:teuthology.orchestra.run.smithi187.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-21T13:40:39.745 INFO:teuthology.orchestra.run.smithi187.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-21T13:40:39.745 INFO:teuthology.orchestra.run.smithi187.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-21T13:40:39.746 INFO:teuthology.orchestra.run.smithi187.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-21T13:40:39.746 INFO:teuthology.orchestra.run.smithi187.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-21T13:40:39.746 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-21T13:40:39.746 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ply-3.11-14.el9.noarch 2024-09-21T13:40:39.746 INFO:teuthology.orchestra.run.smithi187.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-21T13:40:39.746 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-21T13:40:39.746 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-21T13:40:39.746 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-21T13:40:39.746 INFO:teuthology.orchestra.run.smithi187.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-21T13:40:39.746 INFO:teuthology.orchestra.run.smithi187.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-21T13:40:39.746 INFO:teuthology.orchestra.run.smithi187.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-21T13:40:39.746 INFO:teuthology.orchestra.run.smithi187.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-21T13:40:39.746 INFO:teuthology.orchestra.run.smithi187.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-21T13:40:39.747 INFO:teuthology.orchestra.run.smithi187.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-21T13:40:39.747 INFO:teuthology.orchestra.run.smithi187.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-21T13:40:39.747 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:40:39.747 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-21T13:40:40.158 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mon-2:19.3.0-5104.g44178494.el9.x86_64 35/37 2024-09-21T13:40:40.163 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-base 2024-09-21T13:40:40.183 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mon-2:19.3.0-5104.g44178494.el9.x86_64 35/37 2024-09-21T13:40:40.183 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T13:40:40.183 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-21T13:40:40.183 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T13:40:40.183 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T13:40:40.183 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:40:40.417 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mds-2:19.3.0-5104.g44178494.el9.x86_64 36/37 2024-09-21T13:40:40.442 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mds-2:19.3.0-5104.g44178494.el9.x86_64 36/37 2024-09-21T13:40:40.442 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T13:40:40.442 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-21T13:40:40.442 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T13:40:40.442 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T13:40:40.442 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:40:40.509 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-2:19.3.0-5104.g44178494.el9.x86_64 37/37 2024-09-21T13:40:40.749 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:00:52 ago on Sat 21 Sep 2024 01:39:48 PM UTC. 2024-09-21T13:40:40.838 INFO:teuthology.orchestra.run.smithi187.stdout:Package ceph-base-2:19.3.0-5104.g44178494.el9.x86_64 is already installed. 2024-09-21T13:40:40.881 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-21T13:40:40.882 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2024-09-21T13:40:40.882 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-21T13:40:40.954 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install cephadm 2024-09-21T13:40:41.533 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:00:53 ago on Sat 21 Sep 2024 01:39:48 PM UTC. 2024-09-21T13:40:41.664 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-21T13:40:41.665 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-21T13:40:41.665 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2024-09-21T13:40:41.666 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-21T13:40:41.666 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-09-21T13:40:41.666 INFO:teuthology.orchestra.run.smithi187.stdout: cephadm noarch 2:19.3.0-5104.g44178494.el9 ceph-noarch 784 k 2024-09-21T13:40:41.666 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:40:41.666 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-09-21T13:40:41.666 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-21T13:40:41.666 INFO:teuthology.orchestra.run.smithi187.stdout:Install 1 Package 2024-09-21T13:40:41.666 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:40:41.666 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 784 k 2024-09-21T13:40:41.666 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 791 k 2024-09-21T13:40:41.666 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-09-21T13:40:41.949 INFO:teuthology.orchestra.run.smithi187.stdout:cephadm-19.3.0-5104.g44178494.el9.noarch.rpm 2.7 MB/s | 784 kB 00:00 2024-09-21T13:40:41.950 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-09-21T13:40:41.950 INFO:teuthology.orchestra.run.smithi187.stdout:Total 2.7 MB/s | 784 kB 00:00 2024-09-21T13:40:41.950 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-09-21T13:40:41.955 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-09-21T13:40:41.955 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-09-21T13:40:41.961 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-09-21T13:40:41.962 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-09-21T13:40:42.033 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-09-21T13:40:42.485 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-2:19.3.0-5104.g44178494.el9.x86_64 37/37 2024-09-21T13:40:42.486 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-2:19.3.0-5104.g44178494.el9.x86_64 1/37 2024-09-21T13:40:42.486 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mds-2:19.3.0-5104.g44178494.el9.x86_64 2/37 2024-09-21T13:40:42.486 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-2:19.3.0-5104.g44178494.el9.x86_64 3/37 2024-09-21T13:40:42.486 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mon-2:19.3.0-5104.g44178494.el9.x86_64 4/37 2024-09-21T13:40:42.486 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-osd-2:19.3.0-5104.g44178494.el9.x86_64 5/37 2024-09-21T13:40:42.486 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-5104.g44178494.el9. 6/37 2024-09-21T13:40:42.486 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-21T13:40:42.486 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-21T13:40:42.486 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-21T13:40:42.486 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-21T13:40:42.486 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-21T13:40:42.486 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-21T13:40:42.487 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-21T13:40:42.487 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-21T13:40:42.487 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-21T13:40:42.487 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-21T13:40:42.487 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-21T13:40:42.487 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-21T13:40:42.487 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-21T13:40:42.487 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-21T13:40:42.487 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-21T13:40:42.487 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-21T13:40:42.487 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-21T13:40:42.487 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-21T13:40:42.487 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-21T13:40:42.487 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-21T13:40:42.488 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-21T13:40:42.488 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-21T13:40:42.488 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-21T13:40:42.489 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-21T13:40:42.489 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-21T13:40:42.489 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-21T13:40:42.489 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-21T13:40:42.489 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-21T13:40:42.489 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-21T13:40:42.489 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-21T13:40:42.507 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: cephadm-2:19.3.0-5104.g44178494.el9.noarch 1/1 2024-09-21T13:40:42.595 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : cephadm-2:19.3.0-5104.g44178494.el9.noarch 1/1 2024-09-21T13:40:42.983 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: cephadm-2:19.3.0-5104.g44178494.el9.noarch 1/1 2024-09-21T13:40:43.226 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : cephadm-2:19.3.0-5104.g44178494.el9.noarch 1/1 2024-09-21T13:40:43.227 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:40:43.227 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-09-21T13:40:43.227 INFO:teuthology.orchestra.run.smithi187.stdout: cephadm-2:19.3.0-5104.g44178494.el9.noarch 2024-09-21T13:40:43.227 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:40:43.227 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-21T13:40:43.445 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-21T13:40:43.445 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:40:43.445 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-09-21T13:40:43.445 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-2:19.3.0-5104.g44178494.el9.x86_64 2024-09-21T13:40:43.446 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mds-2:19.3.0-5104.g44178494.el9.x86_64 2024-09-21T13:40:43.446 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-2:19.3.0-5104.g44178494.el9.x86_64 2024-09-21T13:40:43.446 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-modules-core-2:19.3.0-5104.g44178494.el9.noarch 2024-09-21T13:40:43.446 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mon-2:19.3.0-5104.g44178494.el9.x86_64 2024-09-21T13:40:43.446 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-osd-2:19.3.0-5104.g44178494.el9.x86_64 2024-09-21T13:40:43.446 INFO:teuthology.orchestra.run.smithi049.stdout: lua-5.4.4-4.el9.x86_64 2024-09-21T13:40:43.446 INFO:teuthology.orchestra.run.smithi049.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-21T13:40:43.446 INFO:teuthology.orchestra.run.smithi049.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-21T13:40:43.446 INFO:teuthology.orchestra.run.smithi049.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-21T13:40:43.446 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-21T13:40:43.446 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-21T13:40:43.446 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-21T13:40:43.446 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-21T13:40:43.447 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-21T13:40:43.447 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-21T13:40:43.447 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-21T13:40:43.447 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-21T13:40:43.447 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-21T13:40:43.447 INFO:teuthology.orchestra.run.smithi049.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-21T13:40:43.447 INFO:teuthology.orchestra.run.smithi049.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-21T13:40:43.447 INFO:teuthology.orchestra.run.smithi049.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-21T13:40:43.447 INFO:teuthology.orchestra.run.smithi049.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-21T13:40:43.447 INFO:teuthology.orchestra.run.smithi049.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-21T13:40:43.447 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-21T13:40:43.447 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ply-3.11-14.el9.noarch 2024-09-21T13:40:43.447 INFO:teuthology.orchestra.run.smithi049.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-21T13:40:43.447 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-21T13:40:43.447 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-21T13:40:43.448 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-21T13:40:43.448 INFO:teuthology.orchestra.run.smithi049.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-21T13:40:43.448 INFO:teuthology.orchestra.run.smithi049.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-21T13:40:43.448 INFO:teuthology.orchestra.run.smithi049.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-21T13:40:43.448 INFO:teuthology.orchestra.run.smithi049.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-21T13:40:43.448 INFO:teuthology.orchestra.run.smithi049.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-21T13:40:43.448 INFO:teuthology.orchestra.run.smithi049.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-21T13:40:43.448 INFO:teuthology.orchestra.run.smithi049.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-21T13:40:43.448 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:40:43.448 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-09-21T13:40:43.453 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-immutable-object-cache 2024-09-21T13:40:43.863 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-base 2024-09-21T13:40:44.041 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:00:56 ago on Sat 21 Sep 2024 01:39:48 PM UTC. 2024-09-21T13:40:44.176 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-21T13:40:44.177 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-21T13:40:44.177 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repo Size 2024-09-21T13:40:44.177 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-21T13:40:44.177 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-09-21T13:40:44.177 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-5104.g44178494.el9 ceph 145 k 2024-09-21T13:40:44.177 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:40:44.177 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-09-21T13:40:44.177 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-21T13:40:44.177 INFO:teuthology.orchestra.run.smithi187.stdout:Install 1 Package 2024-09-21T13:40:44.177 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:40:44.178 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 145 k 2024-09-21T13:40:44.178 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 432 k 2024-09-21T13:40:44.178 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-09-21T13:40:44.380 INFO:teuthology.orchestra.run.smithi187.stdout:ceph-immutable-object-cache-19.3.0-5104.g441784 717 kB/s | 145 kB 00:00 2024-09-21T13:40:44.381 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-09-21T13:40:44.381 INFO:teuthology.orchestra.run.smithi187.stdout:Total 712 kB/s | 145 kB 00:00 2024-09-21T13:40:44.381 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-09-21T13:40:44.390 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-09-21T13:40:44.390 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-09-21T13:40:44.447 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-09-21T13:40:44.447 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-09-21T13:40:44.477 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:00:56 ago on Sat 21 Sep 2024 01:39:48 PM UTC. 2024-09-21T13:40:44.572 INFO:teuthology.orchestra.run.smithi049.stdout:Package ceph-base-2:19.3.0-5104.g44178494.el9.x86_64 is already installed. 2024-09-21T13:40:44.593 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-09-21T13:40:44.618 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-09-21T13:40:44.619 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2024-09-21T13:40:44.619 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-09-21T13:40:44.681 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install cephadm 2024-09-21T13:40:44.681 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-5104.g44178494. 1/1 2024-09-21T13:40:44.707 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-5104.g44178494. 1/1 2024-09-21T13:40:44.707 INFO:teuthology.orchestra.run.smithi187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T13:40:44.707 INFO:teuthology.orchestra.run.smithi187.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-21T13:40:44.707 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:40:45.283 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:00:57 ago on Sat 21 Sep 2024 01:39:48 PM UTC. 2024-09-21T13:40:45.419 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-09-21T13:40:45.419 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-09-21T13:40:45.419 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2024-09-21T13:40:45.420 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-09-21T13:40:45.420 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-09-21T13:40:45.420 INFO:teuthology.orchestra.run.smithi049.stdout: cephadm noarch 2:19.3.0-5104.g44178494.el9 ceph-noarch 784 k 2024-09-21T13:40:45.420 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:40:45.420 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-09-21T13:40:45.420 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-09-21T13:40:45.420 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-09-21T13:40:45.420 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:40:45.420 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 784 k 2024-09-21T13:40:45.420 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 791 k 2024-09-21T13:40:45.420 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-09-21T13:40:45.674 INFO:teuthology.orchestra.run.smithi049.stdout:cephadm-19.3.0-5104.g44178494.el9.noarch.rpm 3.0 MB/s | 784 kB 00:00 2024-09-21T13:40:45.675 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-09-21T13:40:45.675 INFO:teuthology.orchestra.run.smithi049.stdout:Total 3.0 MB/s | 784 kB 00:00 2024-09-21T13:40:45.675 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-09-21T13:40:45.679 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-5104.g44178494. 1/1 2024-09-21T13:40:45.679 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:40:45.679 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-09-21T13:40:45.679 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-immutable-object-cache-2:19.3.0-5104.g44178494.el9.x86_64 2024-09-21T13:40:45.680 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:40:45.680 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-21T13:40:45.681 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-09-21T13:40:45.681 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-09-21T13:40:45.687 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-09-21T13:40:45.687 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-09-21T13:40:45.761 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-09-21T13:40:45.892 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-mgr 2024-09-21T13:40:46.241 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: cephadm-2:19.3.0-5104.g44178494.el9.noarch 1/1 2024-09-21T13:40:46.339 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : cephadm-2:19.3.0-5104.g44178494.el9.noarch 1/1 2024-09-21T13:40:46.483 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:00:58 ago on Sat 21 Sep 2024 01:39:48 PM UTC. 2024-09-21T13:40:46.575 INFO:teuthology.orchestra.run.smithi187.stdout:Package ceph-mgr-2:19.3.0-5104.g44178494.el9.x86_64 is already installed. 2024-09-21T13:40:46.619 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-21T13:40:46.620 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2024-09-21T13:40:46.620 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-21T13:40:46.687 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-mgr-dashboard 2024-09-21T13:40:46.724 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: cephadm-2:19.3.0-5104.g44178494.el9.noarch 1/1 2024-09-21T13:40:47.025 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : cephadm-2:19.3.0-5104.g44178494.el9.noarch 1/1 2024-09-21T13:40:47.025 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:40:47.025 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-09-21T13:40:47.025 INFO:teuthology.orchestra.run.smithi049.stdout: cephadm-2:19.3.0-5104.g44178494.el9.noarch 2024-09-21T13:40:47.026 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:40:47.026 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-09-21T13:40:47.266 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:00:59 ago on Sat 21 Sep 2024 01:39:48 PM UTC. 2024-09-21T13:40:47.357 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-immutable-object-cache 2024-09-21T13:40:47.401 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-21T13:40:47.402 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-21T13:40:47.402 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2024-09-21T13:40:47.402 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-21T13:40:47.402 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-09-21T13:40:47.402 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-dashboard noarch 2:19.3.0-5104.g44178494.el9 ceph-noarch 14 M 2024-09-21T13:40:47.402 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2024-09-21T13:40:47.402 INFO:teuthology.orchestra.run.smithi187.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-21T13:40:47.402 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-grafana-dashboards noarch 2:19.3.0-5104.g44178494.el9 ceph-noarch 34 k 2024-09-21T13:40:47.402 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-prometheus-alerts noarch 2:19.3.0-5104.g44178494.el9 ceph-noarch 16 k 2024-09-21T13:40:47.403 INFO:teuthology.orchestra.run.smithi187.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-21T13:40:47.403 INFO:teuthology.orchestra.run.smithi187.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-21T13:40:47.403 INFO:teuthology.orchestra.run.smithi187.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-21T13:40:47.403 INFO:teuthology.orchestra.run.smithi187.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-21T13:40:47.403 INFO:teuthology.orchestra.run.smithi187.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-21T13:40:47.403 INFO:teuthology.orchestra.run.smithi187.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-21T13:40:47.403 INFO:teuthology.orchestra.run.smithi187.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-21T13:40:47.403 INFO:teuthology.orchestra.run.smithi187.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-21T13:40:47.403 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:40:47.403 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-09-21T13:40:47.403 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-21T13:40:47.403 INFO:teuthology.orchestra.run.smithi187.stdout:Install 12 Packages 2024-09-21T13:40:47.404 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:40:47.404 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 19 M 2024-09-21T13:40:47.404 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 159 M 2024-09-21T13:40:47.404 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-09-21T13:40:47.892 INFO:teuthology.orchestra.run.smithi187.stdout:(1/12): ceph-grafana-dashboards-19.3.0-5104.g44 224 kB/s | 34 kB 00:00 2024-09-21T13:40:47.925 INFO:teuthology.orchestra.run.smithi187.stdout:(2/12): ceph-prometheus-alerts-19.3.0-5104.g441 88 kB/s | 16 kB 00:00 2024-09-21T13:40:47.967 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:00:59 ago on Sat 21 Sep 2024 01:39:48 PM UTC. 2024-09-21T13:40:48.107 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-09-21T13:40:48.107 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-09-21T13:40:48.107 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repo Size 2024-09-21T13:40:48.108 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-09-21T13:40:48.108 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-09-21T13:40:48.108 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-5104.g44178494.el9 ceph 145 k 2024-09-21T13:40:48.108 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:40:48.108 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-09-21T13:40:48.108 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-09-21T13:40:48.108 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-09-21T13:40:48.108 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:40:48.109 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 145 k 2024-09-21T13:40:48.109 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 432 k 2024-09-21T13:40:48.109 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-09-21T13:40:48.192 INFO:teuthology.orchestra.run.smithi187.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.0 MB/s | 269 kB 00:00 2024-09-21T13:40:48.226 INFO:teuthology.orchestra.run.smithi187.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.0 MB/s | 1.0 MB 00:00 2024-09-21T13:40:48.292 INFO:teuthology.orchestra.run.smithi187.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 8.1 MB/s | 548 kB 00:00 2024-09-21T13:40:48.318 INFO:teuthology.orchestra.run.smithi187.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 766 kB/s | 19 kB 00:00 2024-09-21T13:40:48.355 INFO:teuthology.orchestra.run.smithi049.stdout:ceph-immutable-object-cache-19.3.0-5104.g441784 589 kB/s | 145 kB 00:00 2024-09-21T13:40:48.355 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-09-21T13:40:48.356 INFO:teuthology.orchestra.run.smithi049.stdout:Total 585 kB/s | 145 kB 00:00 2024-09-21T13:40:48.356 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-09-21T13:40:48.364 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-09-21T13:40:48.364 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-09-21T13:40:48.425 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-09-21T13:40:48.425 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-09-21T13:40:48.494 INFO:teuthology.orchestra.run.smithi187.stdout:(7/12): ceph-mgr-dashboard-19.3.0-5104.g4417849 19 MB/s | 14 MB 00:00 2024-09-21T13:40:48.519 INFO:teuthology.orchestra.run.smithi187.stdout:(8/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.6 MB/s | 863 kB 00:00 2024-09-21T13:40:48.561 INFO:teuthology.orchestra.run.smithi187.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.1 MB/s | 144 kB 00:00 2024-09-21T13:40:48.576 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-09-21T13:40:48.586 INFO:teuthology.orchestra.run.smithi187.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 515 kB/s | 31 kB 00:00 2024-09-21T13:40:48.628 INFO:teuthology.orchestra.run.smithi187.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.8 MB/s | 188 kB 00:00 2024-09-21T13:40:48.648 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-5104.g44178494. 1/1 2024-09-21T13:40:48.675 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-5104.g44178494. 1/1 2024-09-21T13:40:48.675 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T13:40:48.675 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-21T13:40:48.675 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:40:48.678 INFO:teuthology.orchestra.run.smithi187.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 5.7 MB/s | 2.0 MB 00:00 2024-09-21T13:40:48.683 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-09-21T13:40:48.683 INFO:teuthology.orchestra.run.smithi187.stdout:Total 15 MB/s | 19 MB 00:01 2024-09-21T13:40:48.733 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-09-21T13:40:48.751 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-09-21T13:40:48.751 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-09-21T13:40:49.042 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-09-21T13:40:49.042 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-09-21T13:40:49.599 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-5104.g44178494. 1/1 2024-09-21T13:40:49.599 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:40:49.600 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-09-21T13:40:49.600 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-immutable-object-cache-2:19.3.0-5104.g44178494.el9.x86_64 2024-09-21T13:40:49.600 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:40:49.600 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-09-21T13:40:49.649 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-09-21T13:40:49.712 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-21T13:40:49.747 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-21T13:40:49.795 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-21T13:40:49.818 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-mgr 2024-09-21T13:40:49.841 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-21T13:40:49.944 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-21T13:40:50.085 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-21T13:40:50.179 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-21T13:40:50.221 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-21T13:40:50.249 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-21T13:40:50.281 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-5104.g44178494.el9 10/12 2024-09-21T13:40:50.425 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:02 ago on Sat 21 Sep 2024 01:39:48 PM UTC. 2024-09-21T13:40:50.519 INFO:teuthology.orchestra.run.smithi049.stdout:Package ceph-mgr-2:19.3.0-5104.g44178494.el9.x86_64 is already installed. 2024-09-21T13:40:50.563 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-09-21T13:40:50.564 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2024-09-21T13:40:50.565 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-09-21T13:40:50.622 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-mgr-dashboard 2024-09-21T13:40:51.227 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:03 ago on Sat 21 Sep 2024 01:39:48 PM UTC. 2024-09-21T13:40:51.366 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-09-21T13:40:51.367 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-09-21T13:40:51.367 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2024-09-21T13:40:51.367 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-09-21T13:40:51.367 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-09-21T13:40:51.367 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-dashboard noarch 2:19.3.0-5104.g44178494.el9 ceph-noarch 14 M 2024-09-21T13:40:51.367 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2024-09-21T13:40:51.367 INFO:teuthology.orchestra.run.smithi049.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-21T13:40:51.367 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-grafana-dashboards noarch 2:19.3.0-5104.g44178494.el9 ceph-noarch 34 k 2024-09-21T13:40:51.367 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-prometheus-alerts noarch 2:19.3.0-5104.g44178494.el9 ceph-noarch 16 k 2024-09-21T13:40:51.367 INFO:teuthology.orchestra.run.smithi049.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-21T13:40:51.367 INFO:teuthology.orchestra.run.smithi049.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-21T13:40:51.368 INFO:teuthology.orchestra.run.smithi049.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-21T13:40:51.368 INFO:teuthology.orchestra.run.smithi049.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-21T13:40:51.368 INFO:teuthology.orchestra.run.smithi049.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-21T13:40:51.368 INFO:teuthology.orchestra.run.smithi049.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-21T13:40:51.368 INFO:teuthology.orchestra.run.smithi049.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-21T13:40:51.368 INFO:teuthology.orchestra.run.smithi049.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-21T13:40:51.368 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:40:51.368 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-09-21T13:40:51.368 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-09-21T13:40:51.368 INFO:teuthology.orchestra.run.smithi049.stdout:Install 12 Packages 2024-09-21T13:40:51.368 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:40:51.369 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 19 M 2024-09-21T13:40:51.369 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 159 M 2024-09-21T13:40:51.369 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-09-21T13:40:51.934 INFO:teuthology.orchestra.run.smithi049.stdout:(1/12): ceph-prometheus-alerts-19.3.0-5104.g441 102 kB/s | 16 kB 00:00 2024-09-21T13:40:51.951 INFO:teuthology.orchestra.run.smithi049.stdout:(2/12): ceph-grafana-dashboards-19.3.0-5104.g44 191 kB/s | 34 kB 00:00 2024-09-21T13:40:52.151 INFO:teuthology.orchestra.run.smithi049.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.3 MB/s | 269 kB 00:00 2024-09-21T13:40:52.276 INFO:teuthology.orchestra.run.smithi049.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 2.9 MB/s | 1.0 MB 00:00 2024-09-21T13:40:52.427 INFO:teuthology.orchestra.run.smithi049.stdout:(5/12): ceph-mgr-dashboard-19.3.0-5104.g4417849 22 MB/s | 14 MB 00:00 2024-09-21T13:40:52.468 INFO:teuthology.orchestra.run.smithi049.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.8 MB/s | 548 kB 00:00 2024-09-21T13:40:52.494 INFO:teuthology.orchestra.run.smithi049.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 301 kB/s | 19 kB 00:00 2024-09-21T13:40:52.536 INFO:teuthology.orchestra.run.smithi049.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 3.4 MB/s | 144 kB 00:00 2024-09-21T13:40:52.561 INFO:teuthology.orchestra.run.smithi049.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-21T13:40:52.611 INFO:teuthology.orchestra.run.smithi049.stdout:(10/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 15 MB/s | 2.0 MB 00:00 2024-09-21T13:40:52.636 INFO:teuthology.orchestra.run.smithi049.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.4 MB/s | 188 kB 00:00 2024-09-21T13:40:52.961 INFO:teuthology.orchestra.run.smithi049.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 1.0 MB/s | 863 kB 00:00 2024-09-21T13:40:52.965 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-09-21T13:40:52.965 INFO:teuthology.orchestra.run.smithi049.stdout:Total 12 MB/s | 19 MB 00:01 2024-09-21T13:40:53.016 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-09-21T13:40:53.036 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-09-21T13:40:53.036 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-09-21T13:40:53.328 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-09-21T13:40:53.328 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-09-21T13:40:53.520 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-5104.g44178494.el 11/12 2024-09-21T13:40:53.577 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-5104.g44178494.el9.noa 12/12 2024-09-21T13:40:53.938 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-09-21T13:40:54.022 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-21T13:40:54.064 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-21T13:40:54.112 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-21T13:40:54.150 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-21T13:40:54.247 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-21T13:40:54.382 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-21T13:40:54.482 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-21T13:40:54.530 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-21T13:40:54.558 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-21T13:40:54.616 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-5104.g44178494.el9 10/12 2024-09-21T13:40:55.564 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-5104.g44178494.el9.noa 12/12 2024-09-21T13:40:55.564 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-5104.g44178494.el 1/12 2024-09-21T13:40:55.565 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-5104.g44178494.el9.noa 2/12 2024-09-21T13:40:55.565 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-5104.g44178494.el9 3/12 2024-09-21T13:40:55.565 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-21T13:40:55.565 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-21T13:40:55.565 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-21T13:40:55.565 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-21T13:40:55.565 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-21T13:40:55.565 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-21T13:40:55.565 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-21T13:40:55.565 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-21T13:40:55.881 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-21T13:40:55.881 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:40:55.881 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-09-21T13:40:55.881 INFO:teuthology.orchestra.run.smithi187.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-21T13:40:55.881 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-grafana-dashboards-2:19.3.0-5104.g44178494.el9.noarch 2024-09-21T13:40:55.881 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-dashboard-2:19.3.0-5104.g44178494.el9.noarch 2024-09-21T13:40:55.881 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-prometheus-alerts-2:19.3.0-5104.g44178494.el9.noarch 2024-09-21T13:40:55.881 INFO:teuthology.orchestra.run.smithi187.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-21T13:40:55.881 INFO:teuthology.orchestra.run.smithi187.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-21T13:40:55.881 INFO:teuthology.orchestra.run.smithi187.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-21T13:40:55.881 INFO:teuthology.orchestra.run.smithi187.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-21T13:40:55.881 INFO:teuthology.orchestra.run.smithi187.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-21T13:40:55.882 INFO:teuthology.orchestra.run.smithi187.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-21T13:40:55.882 INFO:teuthology.orchestra.run.smithi187.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-21T13:40:55.882 INFO:teuthology.orchestra.run.smithi187.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-21T13:40:55.882 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:40:55.882 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-21T13:40:56.132 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-21T13:40:56.711 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:08 ago on Sat 21 Sep 2024 01:39:48 PM UTC. 2024-09-21T13:40:56.845 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-21T13:40:56.846 INFO:teuthology.orchestra.run.smithi187.stdout:======================================================================================= 2024-09-21T13:40:56.846 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2024-09-21T13:40:56.846 INFO:teuthology.orchestra.run.smithi187.stdout:======================================================================================= 2024-09-21T13:40:56.846 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-09-21T13:40:56.846 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-5104.g44178494.el9 ceph-noarch 7.4 M 2024-09-21T13:40:56.846 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2024-09-21T13:40:56.846 INFO:teuthology.orchestra.run.smithi187.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-21T13:40:56.846 INFO:teuthology.orchestra.run.smithi187.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-21T13:40:56.846 INFO:teuthology.orchestra.run.smithi187.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-21T13:40:56.846 INFO:teuthology.orchestra.run.smithi187.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-21T13:40:56.847 INFO:teuthology.orchestra.run.smithi187.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-21T13:40:56.847 INFO:teuthology.orchestra.run.smithi187.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-21T13:40:56.847 INFO:teuthology.orchestra.run.smithi187.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-21T13:40:56.847 INFO:teuthology.orchestra.run.smithi187.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-21T13:40:56.847 INFO:teuthology.orchestra.run.smithi187.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-21T13:40:56.847 INFO:teuthology.orchestra.run.smithi187.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-21T13:40:56.847 INFO:teuthology.orchestra.run.smithi187.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-21T13:40:56.847 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:40:56.847 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-09-21T13:40:56.847 INFO:teuthology.orchestra.run.smithi187.stdout:======================================================================================= 2024-09-21T13:40:56.847 INFO:teuthology.orchestra.run.smithi187.stdout:Install 12 Packages 2024-09-21T13:40:56.847 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:40:56.848 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 38 M 2024-09-21T13:40:56.848 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 212 M 2024-09-21T13:40:56.848 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-09-21T13:40:57.301 INFO:teuthology.orchestra.run.smithi187.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.1 MB/s | 191 kB 00:00 2024-09-21T13:40:57.392 INFO:teuthology.orchestra.run.smithi187.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 2.9 MB/s | 801 kB 00:00 2024-09-21T13:40:57.409 INFO:teuthology.orchestra.run.smithi187.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 301 kB/s | 33 kB 00:00 2024-09-21T13:40:57.451 INFO:teuthology.orchestra.run.smithi187.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 436 kB/s | 18 kB 00:00 2024-09-21T13:40:57.585 INFO:teuthology.orchestra.run.smithi187.stdout:(5/12): ceph-mgr-diskprediction-local-19.3.0-51 16 MB/s | 7.4 MB 00:00 2024-09-21T13:40:57.602 INFO:teuthology.orchestra.run.smithi187.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 256 kB/s | 38 kB 00:00 2024-09-21T13:40:57.786 INFO:teuthology.orchestra.run.smithi187.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 7.6 MB/s | 3.0 MB 00:00 2024-09-21T13:40:57.811 INFO:teuthology.orchestra.run.smithi187.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.1 MB/s | 245 kB 00:00 2024-09-21T13:40:57.886 INFO:teuthology.orchestra.run.smithi187.stdout:(9/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 16 MB/s | 4.9 MB 00:00 2024-09-21T13:40:57.911 INFO:teuthology.orchestra.run.smithi187.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 2.4 MB/s | 250 kB 00:00 2024-09-21T13:40:57.995 INFO:teuthology.orchestra.run.smithi187.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 24 MB/s | 5.1 MB 00:00 2024-09-21T13:40:58.020 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-5104.g44178494.el 11/12 2024-09-21T13:40:58.078 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-5104.g44178494.el9.noa 12/12 2024-09-21T13:40:58.237 INFO:teuthology.orchestra.run.smithi187.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 46 MB/s | 16 MB 00:00 2024-09-21T13:40:58.238 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-09-21T13:40:58.238 INFO:teuthology.orchestra.run.smithi187.stdout:Total 27 MB/s | 38 MB 00:01 2024-09-21T13:40:58.590 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-09-21T13:40:58.617 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-09-21T13:40:58.617 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-09-21T13:40:59.049 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-09-21T13:40:59.049 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-09-21T13:40:59.816 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-09-21T13:40:59.894 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-21T13:40:59.926 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-21T13:41:00.019 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-21T13:41:00.052 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-21T13:41:00.075 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-5104.g44178494.el9.noa 12/12 2024-09-21T13:41:00.075 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-5104.g44178494.el 1/12 2024-09-21T13:41:00.075 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-5104.g44178494.el9.noa 2/12 2024-09-21T13:41:00.075 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-5104.g44178494.el9 3/12 2024-09-21T13:41:00.075 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-21T13:41:00.075 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-21T13:41:00.075 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-21T13:41:00.076 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-21T13:41:00.076 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-21T13:41:00.076 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-21T13:41:00.076 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-21T13:41:00.076 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-21T13:41:00.236 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-21T13:41:00.268 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-21T13:41:00.344 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-21T13:41:00.414 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-21T13:41:00.415 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:41:00.415 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-09-21T13:41:00.415 INFO:teuthology.orchestra.run.smithi049.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-21T13:41:00.415 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-grafana-dashboards-2:19.3.0-5104.g44178494.el9.noarch 2024-09-21T13:41:00.415 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-dashboard-2:19.3.0-5104.g44178494.el9.noarch 2024-09-21T13:41:00.415 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-prometheus-alerts-2:19.3.0-5104.g44178494.el9.noarch 2024-09-21T13:41:00.415 INFO:teuthology.orchestra.run.smithi049.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-21T13:41:00.415 INFO:teuthology.orchestra.run.smithi049.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-21T13:41:00.415 INFO:teuthology.orchestra.run.smithi049.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-21T13:41:00.415 INFO:teuthology.orchestra.run.smithi049.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-21T13:41:00.415 INFO:teuthology.orchestra.run.smithi049.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-21T13:41:00.415 INFO:teuthology.orchestra.run.smithi049.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-21T13:41:00.415 INFO:teuthology.orchestra.run.smithi049.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-21T13:41:00.415 INFO:teuthology.orchestra.run.smithi049.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-21T13:41:00.416 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:41:00.416 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-09-21T13:41:00.658 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-21T13:41:01.002 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-21T13:41:01.106 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-21T13:41:01.279 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:13 ago on Sat 21 Sep 2024 01:39:48 PM UTC. 2024-09-21T13:41:01.420 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-09-21T13:41:01.421 INFO:teuthology.orchestra.run.smithi049.stdout:======================================================================================= 2024-09-21T13:41:01.421 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2024-09-21T13:41:01.421 INFO:teuthology.orchestra.run.smithi049.stdout:======================================================================================= 2024-09-21T13:41:01.421 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-09-21T13:41:01.421 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-5104.g44178494.el9 ceph-noarch 7.4 M 2024-09-21T13:41:01.421 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2024-09-21T13:41:01.422 INFO:teuthology.orchestra.run.smithi049.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-21T13:41:01.422 INFO:teuthology.orchestra.run.smithi049.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-21T13:41:01.422 INFO:teuthology.orchestra.run.smithi049.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-21T13:41:01.422 INFO:teuthology.orchestra.run.smithi049.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-21T13:41:01.422 INFO:teuthology.orchestra.run.smithi049.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-21T13:41:01.422 INFO:teuthology.orchestra.run.smithi049.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-21T13:41:01.422 INFO:teuthology.orchestra.run.smithi049.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-21T13:41:01.422 INFO:teuthology.orchestra.run.smithi049.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-21T13:41:01.422 INFO:teuthology.orchestra.run.smithi049.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-21T13:41:01.422 INFO:teuthology.orchestra.run.smithi049.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-21T13:41:01.422 INFO:teuthology.orchestra.run.smithi049.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-21T13:41:01.422 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:41:01.422 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-09-21T13:41:01.422 INFO:teuthology.orchestra.run.smithi049.stdout:======================================================================================= 2024-09-21T13:41:01.423 INFO:teuthology.orchestra.run.smithi049.stdout:Install 12 Packages 2024-09-21T13:41:01.423 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:41:01.423 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 38 M 2024-09-21T13:41:01.423 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 212 M 2024-09-21T13:41:01.423 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-09-21T13:41:01.769 INFO:teuthology.orchestra.run.smithi049.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.0 MB/s | 191 kB 00:00 2024-09-21T13:41:01.886 INFO:teuthology.orchestra.run.smithi049.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 2.6 MB/s | 801 kB 00:00 2024-09-21T13:41:02.019 INFO:teuthology.orchestra.run.smithi049.stdout:(3/12): ceph-mgr-diskprediction-local-19.3.0-51 17 MB/s | 7.4 MB 00:00 2024-09-21T13:41:02.360 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-21T13:41:02.378 INFO:teuthology.orchestra.run.smithi049.stdout:(4/12): flexiblas-3.0.4-8.el9.x86_64.rpm 54 kB/s | 33 kB 00:00 2024-09-21T13:41:02.520 INFO:teuthology.orchestra.run.smithi049.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 37 kB/s | 18 kB 00:00 2024-09-21T13:41:02.937 INFO:teuthology.orchestra.run.smithi049.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 69 kB/s | 38 kB 00:00 2024-09-21T13:41:02.995 INFO:teuthology.orchestra.run.smithi049.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.7 MB/s | 3.0 MB 00:01 2024-09-21T13:41:03.395 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-21T13:41:03.425 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-5104.g44178 12/12 2024-09-21T13:41:03.654 INFO:teuthology.orchestra.run.smithi049.stdout:(8/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 4.4 MB/s | 4.9 MB 00:01 2024-09-21T13:41:03.779 INFO:teuthology.orchestra.run.smithi049.stdout:(9/12): python3-devel-3.9.19-8.el9.x86_64.rpm 291 kB/s | 245 kB 00:00 2024-09-21T13:41:04.147 INFO:teuthology.orchestra.run.smithi049.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.4 MB/s | 5.1 MB 00:01 2024-09-21T13:41:04.447 INFO:teuthology.orchestra.run.smithi049.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 316 kB/s | 250 kB 00:00 2024-09-21T13:41:05.222 INFO:teuthology.orchestra.run.smithi049.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 11 MB/s | 16 MB 00:01 2024-09-21T13:41:05.223 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-09-21T13:41:05.223 INFO:teuthology.orchestra.run.smithi049.stdout:Total 10 MB/s | 38 MB 00:03 2024-09-21T13:41:05.597 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-09-21T13:41:05.625 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-09-21T13:41:05.625 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-09-21T13:41:06.078 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-09-21T13:41:06.078 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-09-21T13:41:06.122 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-5104.g44178 12/12 2024-09-21T13:41:06.122 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-5104.g44178 1/12 2024-09-21T13:41:06.122 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-21T13:41:06.122 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-21T13:41:06.122 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-21T13:41:06.122 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-21T13:41:06.123 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-21T13:41:06.123 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-21T13:41:06.123 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-21T13:41:06.123 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-21T13:41:06.123 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-21T13:41:06.123 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-21T13:41:06.554 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-21T13:41:06.555 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:41:06.555 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-09-21T13:41:06.555 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-diskprediction-local-2:19.3.0-5104.g44178494.el9.noarch 2024-09-21T13:41:06.555 INFO:teuthology.orchestra.run.smithi187.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-21T13:41:06.555 INFO:teuthology.orchestra.run.smithi187.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-21T13:41:06.555 INFO:teuthology.orchestra.run.smithi187.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-21T13:41:06.555 INFO:teuthology.orchestra.run.smithi187.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-21T13:41:06.555 INFO:teuthology.orchestra.run.smithi187.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-21T13:41:06.555 INFO:teuthology.orchestra.run.smithi187.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-21T13:41:06.555 INFO:teuthology.orchestra.run.smithi187.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-21T13:41:06.555 INFO:teuthology.orchestra.run.smithi187.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-21T13:41:06.555 INFO:teuthology.orchestra.run.smithi187.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-21T13:41:06.556 INFO:teuthology.orchestra.run.smithi187.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-21T13:41:06.556 INFO:teuthology.orchestra.run.smithi187.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-21T13:41:06.556 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:41:06.556 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-21T13:41:06.914 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-mgr-rook 2024-09-21T13:41:06.990 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-09-21T13:41:07.056 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-21T13:41:07.104 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-21T13:41:07.208 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-21T13:41:07.247 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-21T13:41:07.438 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-21T13:41:07.479 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-21T13:41:07.500 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:19 ago on Sat 21 Sep 2024 01:39:48 PM UTC. 2024-09-21T13:41:07.557 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-21T13:41:07.633 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-21T13:41:07.634 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-21T13:41:07.634 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2024-09-21T13:41:07.635 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-21T13:41:07.635 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-09-21T13:41:07.635 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-rook noarch 2:19.3.0-5104.g44178494.el9 ceph-noarch 49 k 2024-09-21T13:41:07.635 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2024-09-21T13:41:07.635 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-21T13:41:07.635 INFO:teuthology.orchestra.run.smithi187.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-21T13:41:07.635 INFO:teuthology.orchestra.run.smithi187.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-21T13:41:07.635 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-21T13:41:07.635 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-21T13:41:07.635 INFO:teuthology.orchestra.run.smithi187.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-21T13:41:07.635 INFO:teuthology.orchestra.run.smithi187.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-21T13:41:07.635 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-21T13:41:07.635 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-21T13:41:07.636 INFO:teuthology.orchestra.run.smithi187.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-21T13:41:07.636 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-21T13:41:07.636 INFO:teuthology.orchestra.run.smithi187.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-21T13:41:07.636 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:41:07.636 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-09-21T13:41:07.636 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-21T13:41:07.636 INFO:teuthology.orchestra.run.smithi187.stdout:Install 13 Packages 2024-09-21T13:41:07.636 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:41:07.636 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 2.2 M 2024-09-21T13:41:07.636 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 26 M 2024-09-21T13:41:07.637 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-09-21T13:41:08.027 INFO:teuthology.orchestra.run.smithi187.stdout:(1/13): ceph-mgr-rook-19.3.0-5104.g44178494.el9 296 kB/s | 49 kB 00:00 2024-09-21T13:41:08.258 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-21T13:41:08.369 INFO:teuthology.orchestra.run.smithi187.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 38 kB/s | 19 kB 00:00 2024-09-21T13:41:08.388 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-21T13:41:08.427 INFO:teuthology.orchestra.run.smithi187.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 45 kB/s | 26 kB 00:00 2024-09-21T13:41:08.769 INFO:teuthology.orchestra.run.smithi187.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 299 kB/s | 222 kB 00:00 2024-09-21T13:41:09.069 INFO:teuthology.orchestra.run.smithi187.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 227 kB/s | 159 kB 00:00 2024-09-21T13:41:09.111 INFO:teuthology.orchestra.run.smithi187.stdout:(6/13): python3-cachetools-4.2.4-1.el9.noarch.r 781 kB/s | 32 kB 00:00 2024-09-21T13:41:09.145 INFO:teuthology.orchestra.run.smithi187.stdout:(7/13): python3-certifi-2023.05.07-4.el9.noarch 424 kB/s | 14 kB 00:00 2024-09-21T13:41:09.178 INFO:teuthology.orchestra.run.smithi187.stdout:(8/13): python3-google-auth-2.34.0-1.el9.noarch 6.5 MB/s | 220 kB 00:00 2024-09-21T13:41:09.204 INFO:teuthology.orchestra.run.smithi187.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 359 kB/s | 279 kB 00:00 2024-09-21T13:41:09.245 INFO:teuthology.orchestra.run.smithi187.stdout:(10/13): python3-kubernetes-26.1.0-2.el9.noarch 15 MB/s | 1.0 MB 00:00 2024-09-21T13:41:09.271 INFO:teuthology.orchestra.run.smithi187.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.8 MB/s | 90 kB 00:00 2024-09-21T13:41:09.296 INFO:teuthology.orchestra.run.smithi187.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 643 kB/s | 59 kB 00:00 2024-09-21T13:41:09.388 INFO:teuthology.orchestra.run.smithi187.stdout:(13/13): python3-requests-oauthlib-1.3.0-12.el9 87 kB/s | 54 kB 00:00 2024-09-21T13:41:09.391 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-09-21T13:41:09.392 INFO:teuthology.orchestra.run.smithi187.stdout:Total 1.3 MB/s | 2.2 MB 00:01 2024-09-21T13:41:09.461 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-09-21T13:41:09.477 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-09-21T13:41:09.477 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-09-21T13:41:09.567 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-09-21T13:41:09.567 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-09-21T13:41:09.715 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-21T13:41:10.077 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-09-21T13:41:10.268 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-21T13:41:10.340 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-21T13:41:10.403 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-21T13:41:10.440 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-21T13:41:10.481 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-21T13:41:10.583 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-21T13:41:10.684 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-21T13:41:10.750 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-21T13:41:10.809 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-21T13:41:10.836 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-5104.g44178 12/12 2024-09-21T13:41:11.469 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-21T13:41:11.514 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-21T13:41:11.547 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-21T13:41:11.596 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-21T13:41:11.625 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-rook-2:19.3.0-5104.g44178494.el9.noarch 13/13 2024-09-21T13:41:12.410 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-5104.g44178494.el9.noarch 13/13 2024-09-21T13:41:12.410 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-rook-2:19.3.0-5104.g44178494.el9.noarch 1/13 2024-09-21T13:41:12.410 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-21T13:41:12.410 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-21T13:41:12.410 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-21T13:41:12.410 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-21T13:41:12.410 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-21T13:41:12.410 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-21T13:41:12.411 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-21T13:41:12.411 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-21T13:41:12.411 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-21T13:41:12.411 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-21T13:41:12.411 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-21T13:41:12.744 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-21T13:41:12.744 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:41:12.744 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-09-21T13:41:12.744 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-rook-2:19.3.0-5104.g44178494.el9.noarch 2024-09-21T13:41:12.744 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-21T13:41:12.744 INFO:teuthology.orchestra.run.smithi187.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-21T13:41:12.744 INFO:teuthology.orchestra.run.smithi187.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-21T13:41:12.744 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-21T13:41:12.744 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-21T13:41:12.744 INFO:teuthology.orchestra.run.smithi187.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-21T13:41:12.744 INFO:teuthology.orchestra.run.smithi187.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-21T13:41:12.744 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-21T13:41:12.745 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-21T13:41:12.745 INFO:teuthology.orchestra.run.smithi187.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-21T13:41:12.745 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-21T13:41:12.745 INFO:teuthology.orchestra.run.smithi187.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-21T13:41:12.745 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:41:12.745 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-21T13:41:12.993 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-mgr-cephadm 2024-09-21T13:41:13.441 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-5104.g44178 12/12 2024-09-21T13:41:13.441 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-5104.g44178 1/12 2024-09-21T13:41:13.441 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-21T13:41:13.441 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-21T13:41:13.441 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-21T13:41:13.441 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-21T13:41:13.441 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-21T13:41:13.441 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-21T13:41:13.441 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-21T13:41:13.441 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-21T13:41:13.442 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-21T13:41:13.442 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-21T13:41:13.573 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:25 ago on Sat 21 Sep 2024 01:39:48 PM UTC. 2024-09-21T13:41:13.705 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-21T13:41:13.706 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-21T13:41:13.706 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2024-09-21T13:41:13.706 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-21T13:41:13.706 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-09-21T13:41:13.707 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-cephadm noarch 2:19.3.0-5104.g44178494.el9 ceph-noarch 154 k 2024-09-21T13:41:13.707 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2024-09-21T13:41:13.707 INFO:teuthology.orchestra.run.smithi187.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-21T13:41:13.707 INFO:teuthology.orchestra.run.smithi187.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-21T13:41:13.707 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-21T13:41:13.707 INFO:teuthology.orchestra.run.smithi187.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-21T13:41:13.707 INFO:teuthology.orchestra.run.smithi187.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-21T13:41:13.707 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:41:13.707 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-09-21T13:41:13.707 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-21T13:41:13.707 INFO:teuthology.orchestra.run.smithi187.stdout:Install 6 Packages 2024-09-21T13:41:13.707 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:41:13.708 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 7.0 M 2024-09-21T13:41:13.708 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 33 M 2024-09-21T13:41:13.708 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-09-21T13:41:13.888 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-21T13:41:13.888 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:41:13.889 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-09-21T13:41:13.889 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-diskprediction-local-2:19.3.0-5104.g44178494.el9.noarch 2024-09-21T13:41:13.889 INFO:teuthology.orchestra.run.smithi049.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-21T13:41:13.889 INFO:teuthology.orchestra.run.smithi049.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-21T13:41:13.889 INFO:teuthology.orchestra.run.smithi049.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-21T13:41:13.889 INFO:teuthology.orchestra.run.smithi049.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-21T13:41:13.889 INFO:teuthology.orchestra.run.smithi049.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-21T13:41:13.889 INFO:teuthology.orchestra.run.smithi049.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-21T13:41:13.889 INFO:teuthology.orchestra.run.smithi049.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-21T13:41:13.889 INFO:teuthology.orchestra.run.smithi049.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-21T13:41:13.889 INFO:teuthology.orchestra.run.smithi049.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-21T13:41:13.889 INFO:teuthology.orchestra.run.smithi049.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-21T13:41:13.890 INFO:teuthology.orchestra.run.smithi049.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-21T13:41:13.890 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:41:13.890 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-09-21T13:41:14.014 INFO:teuthology.orchestra.run.smithi187.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.7 MB/s | 248 kB 00:00 2024-09-21T13:41:14.082 INFO:teuthology.orchestra.run.smithi187.stdout:(2/6): ceph-mgr-cephadm-19.3.0-5104.g44178494.e 722 kB/s | 154 kB 00:00 2024-09-21T13:41:14.182 INFO:teuthology.orchestra.run.smithi187.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 6.0 MB 00:00 2024-09-21T13:41:14.207 INFO:teuthology.orchestra.run.smithi187.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 2.8 MB/s | 548 kB 00:00 2024-09-21T13:41:14.224 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-mgr-rook 2024-09-21T13:41:14.240 INFO:teuthology.orchestra.run.smithi187.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 364 kB/s | 58 kB 00:00 2024-09-21T13:41:14.274 INFO:teuthology.orchestra.run.smithi187.stdout:(6/6): python3-typing-extensions-4.12.2-2.el9.n 827 kB/s | 74 kB 00:00 2024-09-21T13:41:14.277 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-09-21T13:41:14.277 INFO:teuthology.orchestra.run.smithi187.stdout:Total 12 MB/s | 7.0 MB 00:00 2024-09-21T13:41:14.355 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-09-21T13:41:14.367 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-09-21T13:41:14.367 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-09-21T13:41:14.468 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-09-21T13:41:14.469 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-09-21T13:41:14.796 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-09-21T13:41:14.836 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:26 ago on Sat 21 Sep 2024 01:39:48 PM UTC. 2024-09-21T13:41:14.974 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-09-21T13:41:14.975 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-09-21T13:41:14.976 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2024-09-21T13:41:14.976 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-09-21T13:41:14.976 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-09-21T13:41:14.976 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-rook noarch 2:19.3.0-5104.g44178494.el9 ceph-noarch 49 k 2024-09-21T13:41:14.976 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2024-09-21T13:41:14.976 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-21T13:41:14.976 INFO:teuthology.orchestra.run.smithi049.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-21T13:41:14.976 INFO:teuthology.orchestra.run.smithi049.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-21T13:41:14.976 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-21T13:41:14.976 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-21T13:41:14.976 INFO:teuthology.orchestra.run.smithi049.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-21T13:41:14.976 INFO:teuthology.orchestra.run.smithi049.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-21T13:41:14.977 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-21T13:41:14.977 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-21T13:41:14.977 INFO:teuthology.orchestra.run.smithi049.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-21T13:41:14.977 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-21T13:41:14.977 INFO:teuthology.orchestra.run.smithi049.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-21T13:41:14.977 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:41:14.977 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-09-21T13:41:14.977 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-09-21T13:41:14.977 INFO:teuthology.orchestra.run.smithi049.stdout:Install 13 Packages 2024-09-21T13:41:14.977 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:41:14.978 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 2.2 M 2024-09-21T13:41:14.978 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 26 M 2024-09-21T13:41:14.978 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-09-21T13:41:14.978 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-21T13:41:15.025 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-21T13:41:15.269 INFO:teuthology.orchestra.run.smithi049.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 166 kB/s | 19 kB 00:00 2024-09-21T13:41:15.294 INFO:teuthology.orchestra.run.smithi049.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 183 kB/s | 26 kB 00:00 2024-09-21T13:41:15.320 INFO:teuthology.orchestra.run.smithi049.stdout:(3/13): ceph-mgr-rook-19.3.0-5104.g44178494.el9 298 kB/s | 49 kB 00:00 2024-09-21T13:41:15.412 INFO:teuthology.orchestra.run.smithi049.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.5 MB/s | 222 kB 00:00 2024-09-21T13:41:15.437 INFO:teuthology.orchestra.run.smithi049.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.1 MB/s | 159 kB 00:00 2024-09-21T13:41:15.479 INFO:teuthology.orchestra.run.smithi049.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 798 kB/s | 54 kB 00:00 2024-09-21T13:41:15.513 INFO:teuthology.orchestra.run.smithi049.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 427 kB/s | 32 kB 00:00 2024-09-21T13:41:15.538 INFO:teuthology.orchestra.run.smithi049.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 249 kB/s | 14 kB 00:00 2024-09-21T13:41:15.556 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-21T13:41:15.563 INFO:teuthology.orchestra.run.smithi049.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.1 MB/s | 279 kB 00:00 2024-09-21T13:41:15.597 INFO:teuthology.orchestra.run.smithi049.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.6 MB/s | 220 kB 00:00 2024-09-21T13:41:15.639 INFO:teuthology.orchestra.run.smithi049.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 2.2 MB/s | 90 kB 00:00 2024-09-21T13:41:15.653 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-21T13:41:15.664 INFO:teuthology.orchestra.run.smithi049.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 588 kB/s | 59 kB 00:00 2024-09-21T13:41:15.697 INFO:teuthology.orchestra.run.smithi049.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.4 MB/s | 1.0 MB 00:00 2024-09-21T13:41:15.700 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-09-21T13:41:15.700 INFO:teuthology.orchestra.run.smithi049.stdout:Total 3.1 MB/s | 2.2 MB 00:00 2024-09-21T13:41:15.729 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-21T13:41:15.755 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-5104.g44178494.el9.noarch 6/6 2024-09-21T13:41:15.767 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-09-21T13:41:15.782 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-09-21T13:41:15.782 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-09-21T13:41:15.880 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-09-21T13:41:15.881 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-09-21T13:41:16.518 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-09-21T13:41:16.562 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-5104.g44178494.el9.noarch 6/6 2024-09-21T13:41:16.562 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-5104.g44178494.el9.noarch 1/6 2024-09-21T13:41:16.562 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-21T13:41:16.562 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-21T13:41:16.562 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-21T13:41:16.562 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-21T13:41:16.680 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-21T13:41:16.844 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-21T13:41:16.844 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:41:16.844 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-09-21T13:41:16.844 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-cephadm-2:19.3.0-5104.g44178494.el9.noarch 2024-09-21T13:41:16.844 INFO:teuthology.orchestra.run.smithi187.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-21T13:41:16.844 INFO:teuthology.orchestra.run.smithi187.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-21T13:41:16.844 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-21T13:41:16.844 INFO:teuthology.orchestra.run.smithi187.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-21T13:41:16.844 INFO:teuthology.orchestra.run.smithi187.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-21T13:41:16.844 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:41:16.845 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-21T13:41:16.851 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-21T13:41:16.915 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-21T13:41:16.950 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-21T13:41:16.993 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-21T13:41:17.079 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-fuse 2024-09-21T13:41:17.107 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-21T13:41:17.226 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-21T13:41:17.278 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-21T13:41:17.667 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:29 ago on Sat 21 Sep 2024 01:39:48 PM UTC. 2024-09-21T13:41:17.800 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-21T13:41:17.800 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-21T13:41:17.800 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2024-09-21T13:41:17.800 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-21T13:41:17.800 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-09-21T13:41:17.801 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-fuse x86_64 2:19.3.0-5104.g44178494.el9 ceph 900 k 2024-09-21T13:41:17.801 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2024-09-21T13:41:17.801 INFO:teuthology.orchestra.run.smithi187.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-21T13:41:17.801 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:41:17.801 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-09-21T13:41:17.801 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-21T13:41:17.801 INFO:teuthology.orchestra.run.smithi187.stdout:Install 2 Packages 2024-09-21T13:41:17.801 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:41:17.802 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 980 k 2024-09-21T13:41:17.802 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 2.9 M 2024-09-21T13:41:17.802 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-09-21T13:41:17.993 INFO:teuthology.orchestra.run.smithi187.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 472 kB/s | 80 kB 00:00 2024-09-21T13:41:18.044 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-21T13:41:18.108 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-21T13:41:18.126 INFO:teuthology.orchestra.run.smithi187.stdout:(2/2): ceph-fuse-19.3.0-5104.g44178494.el9.x86_ 2.9 MB/s | 900 kB 00:00 2024-09-21T13:41:18.127 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-09-21T13:41:18.127 INFO:teuthology.orchestra.run.smithi187.stdout:Total 2.9 MB/s | 980 kB 00:00 2024-09-21T13:41:18.139 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-09-21T13:41:18.149 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-21T13:41:18.161 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-09-21T13:41:18.161 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-09-21T13:41:18.198 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-21T13:41:18.227 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-rook-2:19.3.0-5104.g44178494.el9.noarch 13/13 2024-09-21T13:41:18.243 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-09-21T13:41:18.244 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-09-21T13:41:18.430 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-09-21T13:41:18.561 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-21T13:41:18.626 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-fuse-2:19.3.0-5104.g44178494.el9.x86_64 2/2 2024-09-21T13:41:18.994 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-5104.g44178494.el9.noarch 13/13 2024-09-21T13:41:18.994 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-rook-2:19.3.0-5104.g44178494.el9.noarch 1/13 2024-09-21T13:41:18.994 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-21T13:41:18.994 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-21T13:41:18.994 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-21T13:41:18.994 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-21T13:41:18.994 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-21T13:41:18.995 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-21T13:41:18.995 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-21T13:41:18.995 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-21T13:41:18.995 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-21T13:41:18.995 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-21T13:41:18.995 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-21T13:41:19.335 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-21T13:41:19.335 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:41:19.335 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-09-21T13:41:19.335 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-rook-2:19.3.0-5104.g44178494.el9.noarch 2024-09-21T13:41:19.336 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-21T13:41:19.336 INFO:teuthology.orchestra.run.smithi049.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-21T13:41:19.336 INFO:teuthology.orchestra.run.smithi049.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-21T13:41:19.336 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-21T13:41:19.336 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-21T13:41:19.336 INFO:teuthology.orchestra.run.smithi049.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-21T13:41:19.336 INFO:teuthology.orchestra.run.smithi049.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-21T13:41:19.336 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-21T13:41:19.336 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-21T13:41:19.336 INFO:teuthology.orchestra.run.smithi049.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-21T13:41:19.336 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-21T13:41:19.336 INFO:teuthology.orchestra.run.smithi049.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-21T13:41:19.337 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:41:19.337 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-09-21T13:41:19.343 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-fuse-2:19.3.0-5104.g44178494.el9.x86_64 2/2 2024-09-21T13:41:19.343 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-fuse-2:19.3.0-5104.g44178494.el9.x86_64 1/2 2024-09-21T13:41:19.580 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-mgr-cephadm 2024-09-21T13:41:19.653 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-21T13:41:19.654 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:41:19.654 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-09-21T13:41:19.654 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-fuse-2:19.3.0-5104.g44178494.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-21T13:41:19.654 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:41:19.654 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-21T13:41:19.927 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-volume 2024-09-21T13:41:20.192 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:32 ago on Sat 21 Sep 2024 01:39:48 PM UTC. 2024-09-21T13:41:20.333 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-09-21T13:41:20.333 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-09-21T13:41:20.333 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2024-09-21T13:41:20.333 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-09-21T13:41:20.333 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-09-21T13:41:20.333 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-cephadm noarch 2:19.3.0-5104.g44178494.el9 ceph-noarch 154 k 2024-09-21T13:41:20.334 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2024-09-21T13:41:20.334 INFO:teuthology.orchestra.run.smithi049.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-21T13:41:20.334 INFO:teuthology.orchestra.run.smithi049.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-21T13:41:20.334 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-21T13:41:20.334 INFO:teuthology.orchestra.run.smithi049.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-21T13:41:20.334 INFO:teuthology.orchestra.run.smithi049.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-21T13:41:20.334 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:41:20.334 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-09-21T13:41:20.334 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-09-21T13:41:20.334 INFO:teuthology.orchestra.run.smithi049.stdout:Install 6 Packages 2024-09-21T13:41:20.334 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:41:20.335 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 7.0 M 2024-09-21T13:41:20.335 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 33 M 2024-09-21T13:41:20.335 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-09-21T13:41:20.524 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:32 ago on Sat 21 Sep 2024 01:39:48 PM UTC. 2024-09-21T13:41:20.658 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-21T13:41:20.659 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-21T13:41:20.659 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2024-09-21T13:41:20.659 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-21T13:41:20.659 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-09-21T13:41:20.659 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-volume noarch 2:19.3.0-5104.g44178494.el9 ceph-noarch 287 k 2024-09-21T13:41:20.659 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:41:20.659 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-09-21T13:41:20.659 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-21T13:41:20.659 INFO:teuthology.orchestra.run.smithi187.stdout:Install 1 Package 2024-09-21T13:41:20.659 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:41:20.660 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 287 k 2024-09-21T13:41:20.660 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 1.4 M 2024-09-21T13:41:20.660 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-09-21T13:41:20.889 INFO:teuthology.orchestra.run.smithi187.stdout:ceph-volume-19.3.0-5104.g44178494.el9.noarch.rp 1.2 MB/s | 287 kB 00:00 2024-09-21T13:41:20.889 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-09-21T13:41:20.889 INFO:teuthology.orchestra.run.smithi187.stdout:Total 1.2 MB/s | 287 kB 00:00 2024-09-21T13:41:20.890 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-09-21T13:41:20.898 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-09-21T13:41:20.898 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-09-21T13:41:20.925 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-09-21T13:41:20.925 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-09-21T13:41:21.033 INFO:teuthology.orchestra.run.smithi049.stdout:(1/6): ceph-mgr-cephadm-19.3.0-5104.g44178494.e 714 kB/s | 154 kB 00:00 2024-09-21T13:41:21.092 INFO:teuthology.orchestra.run.smithi049.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 9.2 MB/s | 548 kB 00:00 2024-09-21T13:41:21.117 INFO:teuthology.orchestra.run.smithi049.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-09-21T13:41:21.147 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-09-21T13:41:21.150 INFO:teuthology.orchestra.run.smithi049.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.2 MB/s | 74 kB 00:00 2024-09-21T13:41:21.216 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-volume-2:19.3.0-5104.g44178494.el9.noarch 1/1 2024-09-21T13:41:21.232 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-volume-2:19.3.0-5104.g44178494.el9.noarch 1/1 2024-09-21T13:41:21.233 INFO:teuthology.orchestra.run.smithi187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T13:41:21.233 INFO:teuthology.orchestra.run.smithi187.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-21T13:41:21.233 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:41:21.451 INFO:teuthology.orchestra.run.smithi049.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 392 kB/s | 248 kB 00:00 2024-09-21T13:41:22.184 INFO:teuthology.orchestra.run.smithi049.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 4.4 MB/s | 6.0 MB 00:01 2024-09-21T13:41:22.186 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-09-21T13:41:22.187 INFO:teuthology.orchestra.run.smithi049.stdout:Total 3.8 MB/s | 7.0 MB 00:01 2024-09-21T13:41:22.209 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-volume-2:19.3.0-5104.g44178494.el9.noarch 1/1 2024-09-21T13:41:22.209 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:41:22.209 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-09-21T13:41:22.209 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-volume-2:19.3.0-5104.g44178494.el9.noarch 2024-09-21T13:41:22.209 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:41:22.209 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-21T13:41:22.263 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-09-21T13:41:22.276 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-09-21T13:41:22.276 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-09-21T13:41:22.384 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-09-21T13:41:22.386 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-09-21T13:41:22.508 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install librados-devel 2024-09-21T13:41:22.706 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-09-21T13:41:22.893 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-21T13:41:22.954 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-21T13:41:23.102 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:35 ago on Sat 21 Sep 2024 01:39:48 PM UTC. 2024-09-21T13:41:23.236 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-21T13:41:23.236 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-21T13:41:23.236 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repo Size 2024-09-21T13:41:23.236 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-21T13:41:23.236 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-09-21T13:41:23.236 INFO:teuthology.orchestra.run.smithi187.stdout: librados-devel x86_64 2:19.3.0-5104.g44178494.el9 ceph 126 k 2024-09-21T13:41:23.237 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:41:23.237 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-09-21T13:41:23.237 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-21T13:41:23.237 INFO:teuthology.orchestra.run.smithi187.stdout:Install 1 Package 2024-09-21T13:41:23.237 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:41:23.238 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 126 k 2024-09-21T13:41:23.238 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 456 k 2024-09-21T13:41:23.238 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-09-21T13:41:23.452 INFO:teuthology.orchestra.run.smithi187.stdout:librados-devel-19.3.0-5104.g44178494.el9.x86_64 590 kB/s | 126 kB 00:00 2024-09-21T13:41:23.453 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-09-21T13:41:23.453 INFO:teuthology.orchestra.run.smithi187.stdout:Total 585 kB/s | 126 kB 00:00 2024-09-21T13:41:23.453 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-09-21T13:41:23.459 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-09-21T13:41:23.459 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-09-21T13:41:23.509 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-21T13:41:23.523 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-09-21T13:41:23.523 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-09-21T13:41:23.593 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-21T13:41:23.670 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-09-21T13:41:23.676 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-21T13:41:23.715 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-5104.g44178494.el9.noarch 6/6 2024-09-21T13:41:23.775 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : librados-devel-2:19.3.0-5104.g44178494.el9.x86_64 1/1 2024-09-21T13:41:24.247 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librados-devel-2:19.3.0-5104.g44178494.el9.x86_64 1/1 2024-09-21T13:41:24.518 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-5104.g44178494.el9.noarch 6/6 2024-09-21T13:41:24.518 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-5104.g44178494.el9.noarch 1/6 2024-09-21T13:41:24.518 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-21T13:41:24.518 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-21T13:41:24.518 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-21T13:41:24.518 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-21T13:41:24.518 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librados-devel-2:19.3.0-5104.g44178494.el9.x86_64 1/1 2024-09-21T13:41:24.519 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:41:24.519 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-09-21T13:41:24.519 INFO:teuthology.orchestra.run.smithi187.stdout: librados-devel-2:19.3.0-5104.g44178494.el9.x86_64 2024-09-21T13:41:24.519 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:41:24.519 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-21T13:41:24.788 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install libcephfs2 2024-09-21T13:41:24.861 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-21T13:41:24.861 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:41:24.861 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-09-21T13:41:24.861 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-cephadm-2:19.3.0-5104.g44178494.el9.noarch 2024-09-21T13:41:24.862 INFO:teuthology.orchestra.run.smithi049.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-21T13:41:24.862 INFO:teuthology.orchestra.run.smithi049.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-21T13:41:24.862 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-21T13:41:24.862 INFO:teuthology.orchestra.run.smithi049.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-21T13:41:24.862 INFO:teuthology.orchestra.run.smithi049.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-21T13:41:24.862 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:41:24.862 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-09-21T13:41:25.111 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-fuse 2024-09-21T13:41:25.381 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:37 ago on Sat 21 Sep 2024 01:39:48 PM UTC. 2024-09-21T13:41:25.474 INFO:teuthology.orchestra.run.smithi187.stdout:Package libcephfs2-2:19.3.0-5104.g44178494.el9.x86_64 is already installed. 2024-09-21T13:41:25.518 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-21T13:41:25.520 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2024-09-21T13:41:25.520 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-21T13:41:25.593 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install libcephfs-devel 2024-09-21T13:41:25.720 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:37 ago on Sat 21 Sep 2024 01:39:48 PM UTC. 2024-09-21T13:41:25.859 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-09-21T13:41:25.860 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-09-21T13:41:25.860 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2024-09-21T13:41:25.860 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-09-21T13:41:25.860 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-09-21T13:41:25.860 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-fuse x86_64 2:19.3.0-5104.g44178494.el9 ceph 900 k 2024-09-21T13:41:25.860 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2024-09-21T13:41:25.860 INFO:teuthology.orchestra.run.smithi049.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-21T13:41:25.860 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:41:25.860 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-09-21T13:41:25.861 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-09-21T13:41:25.861 INFO:teuthology.orchestra.run.smithi049.stdout:Install 2 Packages 2024-09-21T13:41:25.861 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:41:25.861 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 980 k 2024-09-21T13:41:25.861 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 2.9 M 2024-09-21T13:41:25.861 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-09-21T13:41:26.128 INFO:teuthology.orchestra.run.smithi049.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 563 kB/s | 80 kB 00:00 2024-09-21T13:41:26.166 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:38 ago on Sat 21 Sep 2024 01:39:48 PM UTC. 2024-09-21T13:41:26.300 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-21T13:41:26.301 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-21T13:41:26.301 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repo Size 2024-09-21T13:41:26.301 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-21T13:41:26.301 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-09-21T13:41:26.301 INFO:teuthology.orchestra.run.smithi187.stdout: libcephfs-devel x86_64 2:19.3.0-5104.g44178494.el9 ceph 32 k 2024-09-21T13:41:26.301 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:41:26.301 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-09-21T13:41:26.301 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-21T13:41:26.301 INFO:teuthology.orchestra.run.smithi187.stdout:Install 1 Package 2024-09-21T13:41:26.301 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:41:26.301 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 32 k 2024-09-21T13:41:26.302 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 142 k 2024-09-21T13:41:26.302 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-09-21T13:41:26.303 INFO:teuthology.orchestra.run.smithi049.stdout:(2/2): ceph-fuse-19.3.0-5104.g44178494.el9.x86_ 2.8 MB/s | 900 kB 00:00 2024-09-21T13:41:26.303 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-09-21T13:41:26.304 INFO:teuthology.orchestra.run.smithi049.stdout:Total 2.2 MB/s | 980 kB 00:00 2024-09-21T13:41:26.317 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-09-21T13:41:26.341 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-09-21T13:41:26.341 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-09-21T13:41:26.428 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-09-21T13:41:26.429 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-09-21T13:41:26.589 INFO:teuthology.orchestra.run.smithi187.stdout:libcephfs-devel-19.3.0-5104.g44178494.el9.x86_6 110 kB/s | 32 kB 00:00 2024-09-21T13:41:26.590 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-09-21T13:41:26.590 INFO:teuthology.orchestra.run.smithi187.stdout:Total 110 kB/s | 32 kB 00:00 2024-09-21T13:41:26.590 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-09-21T13:41:26.594 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-09-21T13:41:26.594 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-09-21T13:41:26.610 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-09-21T13:41:26.610 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-09-21T13:41:26.622 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-09-21T13:41:26.697 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-09-21T13:41:26.760 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-21T13:41:26.821 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libcephfs-devel-2:19.3.0-5104.g44178494.el9.x86_64 1/1 2024-09-21T13:41:26.839 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-fuse-2:19.3.0-5104.g44178494.el9.x86_64 2/2 2024-09-21T13:41:27.204 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-5104.g44178494.el9.x86_64 1/1 2024-09-21T13:41:27.439 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libcephfs-devel-2:19.3.0-5104.g44178494.el9.x86_64 1/1 2024-09-21T13:41:27.440 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:41:27.440 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-09-21T13:41:27.440 INFO:teuthology.orchestra.run.smithi187.stdout: libcephfs-devel-2:19.3.0-5104.g44178494.el9.x86_64 2024-09-21T13:41:27.440 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:41:27.440 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-21T13:41:27.543 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-fuse-2:19.3.0-5104.g44178494.el9.x86_64 2/2 2024-09-21T13:41:27.543 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-fuse-2:19.3.0-5104.g44178494.el9.x86_64 1/2 2024-09-21T13:41:27.635 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install librados2 2024-09-21T13:41:27.834 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-21T13:41:27.835 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:41:27.835 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-09-21T13:41:27.835 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-fuse-2:19.3.0-5104.g44178494.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-21T13:41:27.835 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:41:27.835 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-09-21T13:41:28.101 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-volume 2024-09-21T13:41:28.222 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:40 ago on Sat 21 Sep 2024 01:39:48 PM UTC. 2024-09-21T13:41:28.311 INFO:teuthology.orchestra.run.smithi187.stdout:Package librados2-2:19.3.0-5104.g44178494.el9.x86_64 is already installed. 2024-09-21T13:41:28.355 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-21T13:41:28.356 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2024-09-21T13:41:28.356 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-21T13:41:28.431 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install librbd1 2024-09-21T13:41:28.714 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:40 ago on Sat 21 Sep 2024 01:39:48 PM UTC. 2024-09-21T13:41:28.851 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-09-21T13:41:28.852 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-09-21T13:41:28.852 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2024-09-21T13:41:28.852 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-09-21T13:41:28.852 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-09-21T13:41:28.852 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-volume noarch 2:19.3.0-5104.g44178494.el9 ceph-noarch 287 k 2024-09-21T13:41:28.852 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:41:28.852 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-09-21T13:41:28.853 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-09-21T13:41:28.853 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-09-21T13:41:28.853 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:41:28.853 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 287 k 2024-09-21T13:41:28.853 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 1.4 M 2024-09-21T13:41:28.853 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-09-21T13:41:29.012 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:41 ago on Sat 21 Sep 2024 01:39:48 PM UTC. 2024-09-21T13:41:29.074 INFO:teuthology.orchestra.run.smithi049.stdout:ceph-volume-19.3.0-5104.g44178494.el9.noarch.rp 1.3 MB/s | 287 kB 00:00 2024-09-21T13:41:29.075 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-09-21T13:41:29.075 INFO:teuthology.orchestra.run.smithi049.stdout:Total 1.3 MB/s | 287 kB 00:00 2024-09-21T13:41:29.075 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-09-21T13:41:29.084 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-09-21T13:41:29.084 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-09-21T13:41:29.102 INFO:teuthology.orchestra.run.smithi187.stdout:Package librbd1-2:19.3.0-5104.g44178494.el9.x86_64 is already installed. 2024-09-21T13:41:29.116 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-09-21T13:41:29.116 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-09-21T13:41:29.145 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-21T13:41:29.146 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2024-09-21T13:41:29.146 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-21T13:41:29.216 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install python3-rados 2024-09-21T13:41:29.344 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-09-21T13:41:29.418 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-volume-2:19.3.0-5104.g44178494.el9.noarch 1/1 2024-09-21T13:41:29.435 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-volume-2:19.3.0-5104.g44178494.el9.noarch 1/1 2024-09-21T13:41:29.435 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T13:41:29.435 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-21T13:41:29.436 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:41:29.799 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:41 ago on Sat 21 Sep 2024 01:39:48 PM UTC. 2024-09-21T13:41:29.890 INFO:teuthology.orchestra.run.smithi187.stdout:Package python3-rados-2:19.3.0-5104.g44178494.el9.x86_64 is already installed. 2024-09-21T13:41:29.933 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-21T13:41:29.934 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2024-09-21T13:41:29.934 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-21T13:41:30.000 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install python3-rgw 2024-09-21T13:41:30.484 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-volume-2:19.3.0-5104.g44178494.el9.noarch 1/1 2024-09-21T13:41:30.575 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:41:30.575 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-09-21T13:41:30.575 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-volume-2:19.3.0-5104.g44178494.el9.noarch 2024-09-21T13:41:30.575 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:41:30.576 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-09-21T13:41:30.580 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:42 ago on Sat 21 Sep 2024 01:39:48 PM UTC. 2024-09-21T13:41:30.670 INFO:teuthology.orchestra.run.smithi187.stdout:Package python3-rgw-2:19.3.0-5104.g44178494.el9.x86_64 is already installed. 2024-09-21T13:41:30.714 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-21T13:41:30.715 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2024-09-21T13:41:30.715 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-21T13:41:30.764 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install librados-devel 2024-09-21T13:41:30.786 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install python3-cephfs 2024-09-21T13:41:31.365 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:43 ago on Sat 21 Sep 2024 01:39:48 PM UTC. 2024-09-21T13:41:31.378 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:43 ago on Sat 21 Sep 2024 01:39:48 PM UTC. 2024-09-21T13:41:31.454 INFO:teuthology.orchestra.run.smithi187.stdout:Package python3-cephfs-2:19.3.0-5104.g44178494.el9.x86_64 is already installed. 2024-09-21T13:41:31.497 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-21T13:41:31.498 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2024-09-21T13:41:31.498 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-21T13:41:31.517 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-09-21T13:41:31.518 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-09-21T13:41:31.518 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repo Size 2024-09-21T13:41:31.518 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-09-21T13:41:31.518 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-09-21T13:41:31.518 INFO:teuthology.orchestra.run.smithi049.stdout: librados-devel x86_64 2:19.3.0-5104.g44178494.el9 ceph 126 k 2024-09-21T13:41:31.518 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:41:31.518 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-09-21T13:41:31.518 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-09-21T13:41:31.519 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-09-21T13:41:31.519 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:41:31.519 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 126 k 2024-09-21T13:41:31.519 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 456 k 2024-09-21T13:41:31.519 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-09-21T13:41:31.562 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install python3-rbd 2024-09-21T13:41:31.729 INFO:teuthology.orchestra.run.smithi049.stdout:librados-devel-19.3.0-5104.g44178494.el9.x86_64 603 kB/s | 126 kB 00:00 2024-09-21T13:41:31.730 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-09-21T13:41:31.730 INFO:teuthology.orchestra.run.smithi049.stdout:Total 598 kB/s | 126 kB 00:00 2024-09-21T13:41:31.730 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-09-21T13:41:31.736 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-09-21T13:41:31.736 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-09-21T13:41:31.805 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-09-21T13:41:31.805 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-09-21T13:41:31.963 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-09-21T13:41:32.065 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : librados-devel-2:19.3.0-5104.g44178494.el9.x86_64 1/1 2024-09-21T13:41:32.142 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:44 ago on Sat 21 Sep 2024 01:39:48 PM UTC. 2024-09-21T13:41:32.231 INFO:teuthology.orchestra.run.smithi187.stdout:Package python3-rbd-2:19.3.0-5104.g44178494.el9.x86_64 is already installed. 2024-09-21T13:41:32.274 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-21T13:41:32.276 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2024-09-21T13:41:32.276 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-21T13:41:32.346 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install rbd-fuse 2024-09-21T13:41:32.592 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librados-devel-2:19.3.0-5104.g44178494.el9.x86_64 1/1 2024-09-21T13:41:32.868 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librados-devel-2:19.3.0-5104.g44178494.el9.x86_64 1/1 2024-09-21T13:41:32.868 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:41:32.868 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-09-21T13:41:32.868 INFO:teuthology.orchestra.run.smithi049.stdout: librados-devel-2:19.3.0-5104.g44178494.el9.x86_64 2024-09-21T13:41:32.868 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:41:32.868 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-09-21T13:41:32.930 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:44 ago on Sat 21 Sep 2024 01:39:48 PM UTC. 2024-09-21T13:41:33.063 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-21T13:41:33.064 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-21T13:41:33.064 INFO:teuthology.orchestra.run.smithi187.stdout: Package Architecture Version Repository Size 2024-09-21T13:41:33.064 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-21T13:41:33.064 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-09-21T13:41:33.064 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-fuse x86_64 2:19.3.0-5104.g44178494.el9 ceph 85 k 2024-09-21T13:41:33.064 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:41:33.064 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-09-21T13:41:33.064 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-21T13:41:33.064 INFO:teuthology.orchestra.run.smithi187.stdout:Install 1 Package 2024-09-21T13:41:33.064 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:41:33.064 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 85 k 2024-09-21T13:41:33.065 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 231 k 2024-09-21T13:41:33.065 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-09-21T13:41:33.185 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install libcephfs2 2024-09-21T13:41:33.267 INFO:teuthology.orchestra.run.smithi187.stdout:rbd-fuse-19.3.0-5104.g44178494.el9.x86_64.rpm 421 kB/s | 85 kB 00:00 2024-09-21T13:41:33.268 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-09-21T13:41:33.268 INFO:teuthology.orchestra.run.smithi187.stdout:Total 418 kB/s | 85 kB 00:00 2024-09-21T13:41:33.268 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-09-21T13:41:33.275 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-09-21T13:41:33.275 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-09-21T13:41:33.343 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-09-21T13:41:33.343 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-09-21T13:41:33.520 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-09-21T13:41:33.630 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : rbd-fuse-2:19.3.0-5104.g44178494.el9.x86_64 1/1 2024-09-21T13:41:33.794 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:45 ago on Sat 21 Sep 2024 01:39:48 PM UTC. 2024-09-21T13:41:33.890 INFO:teuthology.orchestra.run.smithi049.stdout:Package libcephfs2-2:19.3.0-5104.g44178494.el9.x86_64 is already installed. 2024-09-21T13:41:33.937 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-09-21T13:41:33.938 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2024-09-21T13:41:33.938 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-09-21T13:41:34.007 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install libcephfs-devel 2024-09-21T13:41:34.023 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: rbd-fuse-2:19.3.0-5104.g44178494.el9.x86_64 1/1 2024-09-21T13:41:34.266 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : rbd-fuse-2:19.3.0-5104.g44178494.el9.x86_64 1/1 2024-09-21T13:41:34.267 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:41:34.267 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-09-21T13:41:34.267 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-fuse-2:19.3.0-5104.g44178494.el9.x86_64 2024-09-21T13:41:34.267 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:41:34.268 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-21T13:41:34.528 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install rbd-mirror 2024-09-21T13:41:34.616 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:46 ago on Sat 21 Sep 2024 01:39:48 PM UTC. 2024-09-21T13:41:34.756 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-09-21T13:41:34.756 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-09-21T13:41:34.756 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repo Size 2024-09-21T13:41:34.756 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-09-21T13:41:34.756 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-09-21T13:41:34.757 INFO:teuthology.orchestra.run.smithi049.stdout: libcephfs-devel x86_64 2:19.3.0-5104.g44178494.el9 ceph 32 k 2024-09-21T13:41:34.757 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:41:34.757 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-09-21T13:41:34.757 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-09-21T13:41:34.757 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-09-21T13:41:34.757 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:41:34.758 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 32 k 2024-09-21T13:41:34.758 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 142 k 2024-09-21T13:41:34.758 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-09-21T13:41:34.900 INFO:teuthology.orchestra.run.smithi049.stdout:libcephfs-devel-19.3.0-5104.g44178494.el9.x86_6 222 kB/s | 32 kB 00:00 2024-09-21T13:41:34.901 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-09-21T13:41:34.901 INFO:teuthology.orchestra.run.smithi049.stdout:Total 220 kB/s | 32 kB 00:00 2024-09-21T13:41:34.901 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-09-21T13:41:34.905 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-09-21T13:41:34.905 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-09-21T13:41:34.923 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-09-21T13:41:34.923 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-09-21T13:41:35.014 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-09-21T13:41:35.126 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libcephfs-devel-2:19.3.0-5104.g44178494.el9.x86_64 1/1 2024-09-21T13:41:35.148 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:47 ago on Sat 21 Sep 2024 01:39:48 PM UTC. 2024-09-21T13:41:35.290 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-21T13:41:35.291 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-21T13:41:35.291 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repo Size 2024-09-21T13:41:35.291 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-21T13:41:35.291 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-09-21T13:41:35.291 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-mirror x86_64 2:19.3.0-5104.g44178494.el9 ceph 3.2 M 2024-09-21T13:41:35.291 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:41:35.291 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-09-21T13:41:35.291 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-21T13:41:35.291 INFO:teuthology.orchestra.run.smithi187.stdout:Install 1 Package 2024-09-21T13:41:35.292 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:41:35.292 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 3.2 M 2024-09-21T13:41:35.292 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 14 M 2024-09-21T13:41:35.292 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-09-21T13:41:35.530 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-5104.g44178494.el9.x86_64 1/1 2024-09-21T13:41:35.629 INFO:teuthology.orchestra.run.smithi187.stdout:rbd-mirror-19.3.0-5104.g44178494.el9.x86_64.rpm 9.5 MB/s | 3.2 MB 00:00 2024-09-21T13:41:35.629 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-09-21T13:41:35.630 INFO:teuthology.orchestra.run.smithi187.stdout:Total 9.4 MB/s | 3.2 MB 00:00 2024-09-21T13:41:35.630 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-09-21T13:41:35.640 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-09-21T13:41:35.640 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-09-21T13:41:35.725 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-09-21T13:41:35.726 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-09-21T13:41:35.774 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libcephfs-devel-2:19.3.0-5104.g44178494.el9.x86_64 1/1 2024-09-21T13:41:35.774 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:41:35.774 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-09-21T13:41:35.774 INFO:teuthology.orchestra.run.smithi049.stdout: libcephfs-devel-2:19.3.0-5104.g44178494.el9.x86_64 2024-09-21T13:41:35.774 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:41:35.775 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-09-21T13:41:35.958 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install librados2 2024-09-21T13:41:36.170 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-09-21T13:41:36.238 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : rbd-mirror-2:19.3.0-5104.g44178494.el9.x86_64 1/1 2024-09-21T13:41:36.263 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: rbd-mirror-2:19.3.0-5104.g44178494.el9.x86_64 1/1 2024-09-21T13:41:36.264 INFO:teuthology.orchestra.run.smithi187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T13:41:36.264 INFO:teuthology.orchestra.run.smithi187.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-21T13:41:36.264 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T13:41:36.264 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T13:41:36.264 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:41:36.564 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:48 ago on Sat 21 Sep 2024 01:39:48 PM UTC. 2024-09-21T13:41:36.656 INFO:teuthology.orchestra.run.smithi049.stdout:Package librados2-2:19.3.0-5104.g44178494.el9.x86_64 is already installed. 2024-09-21T13:41:36.701 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-09-21T13:41:36.703 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2024-09-21T13:41:36.703 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-09-21T13:41:36.767 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install librbd1 2024-09-21T13:41:37.373 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:49 ago on Sat 21 Sep 2024 01:39:48 PM UTC. 2024-09-21T13:41:37.430 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : rbd-mirror-2:19.3.0-5104.g44178494.el9.x86_64 1/1 2024-09-21T13:41:37.430 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:41:37.430 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-09-21T13:41:37.430 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-mirror-2:19.3.0-5104.g44178494.el9.x86_64 2024-09-21T13:41:37.430 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:41:37.430 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-21T13:41:37.466 INFO:teuthology.orchestra.run.smithi049.stdout:Package librbd1-2:19.3.0-5104.g44178494.el9.x86_64 is already installed. 2024-09-21T13:41:37.512 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-09-21T13:41:37.513 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2024-09-21T13:41:37.513 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-09-21T13:41:37.574 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install python3-rados 2024-09-21T13:41:37.652 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install rbd-nbd 2024-09-21T13:41:38.175 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:50 ago on Sat 21 Sep 2024 01:39:48 PM UTC. 2024-09-21T13:41:38.245 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:50 ago on Sat 21 Sep 2024 01:39:48 PM UTC. 2024-09-21T13:41:38.268 INFO:teuthology.orchestra.run.smithi049.stdout:Package python3-rados-2:19.3.0-5104.g44178494.el9.x86_64 is already installed. 2024-09-21T13:41:38.315 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-09-21T13:41:38.316 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2024-09-21T13:41:38.316 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-09-21T13:41:38.377 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install python3-rgw 2024-09-21T13:41:38.379 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-21T13:41:38.380 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-21T13:41:38.380 INFO:teuthology.orchestra.run.smithi187.stdout: Package Architecture Version Repository Size 2024-09-21T13:41:38.380 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-21T13:41:38.380 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-09-21T13:41:38.380 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-nbd x86_64 2:19.3.0-5104.g44178494.el9 ceph 170 k 2024-09-21T13:41:38.380 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:41:38.380 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-09-21T13:41:38.380 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-21T13:41:38.380 INFO:teuthology.orchestra.run.smithi187.stdout:Install 1 Package 2024-09-21T13:41:38.381 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:41:38.381 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 170 k 2024-09-21T13:41:38.381 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 490 k 2024-09-21T13:41:38.381 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-09-21T13:41:38.599 INFO:teuthology.orchestra.run.smithi187.stdout:rbd-nbd-19.3.0-5104.g44178494.el9.x86_64.rpm 783 kB/s | 170 kB 00:00 2024-09-21T13:41:38.600 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-09-21T13:41:38.600 INFO:teuthology.orchestra.run.smithi187.stdout:Total 777 kB/s | 170 kB 00:00 2024-09-21T13:41:38.600 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-09-21T13:41:38.608 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-09-21T13:41:38.608 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-09-21T13:41:38.678 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-09-21T13:41:38.678 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-09-21T13:41:38.838 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-09-21T13:41:38.939 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : rbd-nbd-2:19.3.0-5104.g44178494.el9.x86_64 1/1 2024-09-21T13:41:38.987 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:50 ago on Sat 21 Sep 2024 01:39:48 PM UTC. 2024-09-21T13:41:39.081 INFO:teuthology.orchestra.run.smithi049.stdout:Package python3-rgw-2:19.3.0-5104.g44178494.el9.x86_64 is already installed. 2024-09-21T13:41:39.127 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-09-21T13:41:39.128 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2024-09-21T13:41:39.128 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-09-21T13:41:39.192 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install python3-cephfs 2024-09-21T13:41:39.406 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: rbd-nbd-2:19.3.0-5104.g44178494.el9.x86_64 1/1 2024-09-21T13:41:39.681 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : rbd-nbd-2:19.3.0-5104.g44178494.el9.x86_64 1/1 2024-09-21T13:41:39.681 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:41:39.681 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-09-21T13:41:39.681 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-nbd-2:19.3.0-5104.g44178494.el9.x86_64 2024-09-21T13:41:39.681 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:41:39.682 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-21T13:41:39.797 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:51 ago on Sat 21 Sep 2024 01:39:48 PM UTC. 2024-09-21T13:41:39.890 INFO:teuthology.orchestra.run.smithi049.stdout:Package python3-cephfs-2:19.3.0-5104.g44178494.el9.x86_64 is already installed. 2024-09-21T13:41:39.909 DEBUG:teuthology.parallel:result is None 2024-09-21T13:41:39.936 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-09-21T13:41:39.937 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2024-09-21T13:41:39.937 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-09-21T13:41:39.999 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install python3-rbd 2024-09-21T13:41:40.611 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:52 ago on Sat 21 Sep 2024 01:39:48 PM UTC. 2024-09-21T13:41:40.704 INFO:teuthology.orchestra.run.smithi049.stdout:Package python3-rbd-2:19.3.0-5104.g44178494.el9.x86_64 is already installed. 2024-09-21T13:41:40.750 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-09-21T13:41:40.751 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2024-09-21T13:41:40.751 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-09-21T13:41:40.817 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install rbd-fuse 2024-09-21T13:41:41.424 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:53 ago on Sat 21 Sep 2024 01:39:48 PM UTC. 2024-09-21T13:41:41.563 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-09-21T13:41:41.563 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-09-21T13:41:41.563 INFO:teuthology.orchestra.run.smithi049.stdout: Package Architecture Version Repository Size 2024-09-21T13:41:41.563 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-09-21T13:41:41.563 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-09-21T13:41:41.563 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-fuse x86_64 2:19.3.0-5104.g44178494.el9 ceph 85 k 2024-09-21T13:41:41.564 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:41:41.564 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-09-21T13:41:41.564 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-09-21T13:41:41.564 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-09-21T13:41:41.564 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:41:41.564 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 85 k 2024-09-21T13:41:41.564 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 231 k 2024-09-21T13:41:41.564 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-09-21T13:41:41.762 INFO:teuthology.orchestra.run.smithi049.stdout:rbd-fuse-19.3.0-5104.g44178494.el9.x86_64.rpm 432 kB/s | 85 kB 00:00 2024-09-21T13:41:41.762 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-09-21T13:41:41.762 INFO:teuthology.orchestra.run.smithi049.stdout:Total 430 kB/s | 85 kB 00:00 2024-09-21T13:41:41.763 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-09-21T13:41:41.770 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-09-21T13:41:41.770 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-09-21T13:41:41.843 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-09-21T13:41:41.843 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-09-21T13:41:42.028 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-09-21T13:41:42.155 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : rbd-fuse-2:19.3.0-5104.g44178494.el9.x86_64 1/1 2024-09-21T13:41:42.567 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: rbd-fuse-2:19.3.0-5104.g44178494.el9.x86_64 1/1 2024-09-21T13:41:42.840 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : rbd-fuse-2:19.3.0-5104.g44178494.el9.x86_64 1/1 2024-09-21T13:41:42.840 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:41:42.840 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-09-21T13:41:42.840 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-fuse-2:19.3.0-5104.g44178494.el9.x86_64 2024-09-21T13:41:42.840 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:41:42.840 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-09-21T13:41:43.047 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install rbd-mirror 2024-09-21T13:41:43.665 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:55 ago on Sat 21 Sep 2024 01:39:48 PM UTC. 2024-09-21T13:41:43.803 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-09-21T13:41:43.806 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-09-21T13:41:43.806 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repo Size 2024-09-21T13:41:43.806 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-09-21T13:41:43.806 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-09-21T13:41:43.806 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-mirror x86_64 2:19.3.0-5104.g44178494.el9 ceph 3.2 M 2024-09-21T13:41:43.806 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:41:43.807 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-09-21T13:41:43.807 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-09-21T13:41:43.807 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-09-21T13:41:43.807 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:41:43.807 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 3.2 M 2024-09-21T13:41:43.807 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 14 M 2024-09-21T13:41:43.807 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-09-21T13:41:44.148 INFO:teuthology.orchestra.run.smithi049.stdout:rbd-mirror-19.3.0-5104.g44178494.el9.x86_64.rpm 9.3 MB/s | 3.2 MB 00:00 2024-09-21T13:41:44.149 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-09-21T13:41:44.149 INFO:teuthology.orchestra.run.smithi049.stdout:Total 9.3 MB/s | 3.2 MB 00:00 2024-09-21T13:41:44.149 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-09-21T13:41:44.159 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-09-21T13:41:44.160 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-09-21T13:41:44.246 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-09-21T13:41:44.247 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-09-21T13:41:44.716 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-09-21T13:41:44.774 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : rbd-mirror-2:19.3.0-5104.g44178494.el9.x86_64 1/1 2024-09-21T13:41:44.800 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: rbd-mirror-2:19.3.0-5104.g44178494.el9.x86_64 1/1 2024-09-21T13:41:44.800 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T13:41:44.801 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-21T13:41:44.801 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T13:41:44.801 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T13:41:44.801 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:41:45.923 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : rbd-mirror-2:19.3.0-5104.g44178494.el9.x86_64 1/1 2024-09-21T13:41:45.923 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:41:45.923 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-09-21T13:41:45.923 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-mirror-2:19.3.0-5104.g44178494.el9.x86_64 2024-09-21T13:41:45.923 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:41:45.923 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-09-21T13:41:46.145 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install rbd-nbd 2024-09-21T13:41:46.759 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:58 ago on Sat 21 Sep 2024 01:39:48 PM UTC. 2024-09-21T13:41:46.895 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-09-21T13:41:46.896 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-09-21T13:41:46.896 INFO:teuthology.orchestra.run.smithi049.stdout: Package Architecture Version Repository Size 2024-09-21T13:41:46.896 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-09-21T13:41:46.896 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-09-21T13:41:46.896 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-nbd x86_64 2:19.3.0-5104.g44178494.el9 ceph 170 k 2024-09-21T13:41:46.896 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:41:46.896 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-09-21T13:41:46.896 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-09-21T13:41:46.897 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-09-21T13:41:46.897 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:41:46.897 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 170 k 2024-09-21T13:41:46.897 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 490 k 2024-09-21T13:41:46.897 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-09-21T13:41:47.110 INFO:teuthology.orchestra.run.smithi049.stdout:rbd-nbd-19.3.0-5104.g44178494.el9.x86_64.rpm 801 kB/s | 170 kB 00:00 2024-09-21T13:41:47.110 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-09-21T13:41:47.111 INFO:teuthology.orchestra.run.smithi049.stdout:Total 795 kB/s | 170 kB 00:00 2024-09-21T13:41:47.111 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-09-21T13:41:47.119 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-09-21T13:41:47.119 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-09-21T13:41:47.194 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-09-21T13:41:47.194 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-09-21T13:41:47.357 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-09-21T13:41:47.469 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : rbd-nbd-2:19.3.0-5104.g44178494.el9.x86_64 1/1 2024-09-21T13:41:47.909 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: rbd-nbd-2:19.3.0-5104.g44178494.el9.x86_64 1/1 2024-09-21T13:41:48.268 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : rbd-nbd-2:19.3.0-5104.g44178494.el9.x86_64 1/1 2024-09-21T13:41:48.268 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:41:48.268 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-09-21T13:41:48.268 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-nbd-2:19.3.0-5104.g44178494.el9.x86_64 2024-09-21T13:41:48.268 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:41:48.269 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-09-21T13:41:48.505 DEBUG:teuthology.parallel:result is None 2024-09-21T13:41:48.506 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=441784946648ff53dd310293f9dd752afeeea72a 2024-09-21T13:41:48.644 DEBUG:teuthology.orchestra.run.smithi049:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-21T13:41:48.667 INFO:teuthology.orchestra.run.smithi049.stdout:19.3.0-5104.g44178494.el9 2024-09-21T13:41:48.667 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-5104.g44178494.el9 2024-09-21T13:41:48.667 INFO:teuthology.task.install:The correct ceph version 19.3.0-5104.g44178494 is installed. 2024-09-21T13:41:48.669 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=441784946648ff53dd310293f9dd752afeeea72a 2024-09-21T13:41:48.801 DEBUG:teuthology.orchestra.run.smithi187:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-21T13:41:48.827 INFO:teuthology.orchestra.run.smithi187.stdout:19.3.0-5104.g44178494.el9 2024-09-21T13:41:48.827 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-5104.g44178494.el9 2024-09-21T13:41:48.828 INFO:teuthology.task.install:The correct ceph version 19.3.0-5104.g44178494 is installed. 2024-09-21T13:41:48.829 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-21T13:41:48.829 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-09-21T13:41:48.830 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-21T13:41:48.859 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-09-21T13:41:48.859 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-21T13:41:48.897 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-21T13:41:48.898 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-09-21T13:41:48.898 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/usr/bin/daemon-helper 2024-09-21T13:41:48.926 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-21T13:41:48.994 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-09-21T13:41:48.994 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/usr/bin/daemon-helper 2024-09-21T13:41:49.024 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-21T13:41:49.094 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-21T13:41:49.094 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-09-21T13:41:49.094 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-21T13:41:49.121 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-21T13:41:49.187 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-09-21T13:41:49.188 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-21T13:41:49.216 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-21T13:41:49.283 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-21T13:41:49.284 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-09-21T13:41:49.284 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/usr/bin/stdin-killer 2024-09-21T13:41:49.310 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-21T13:41:49.376 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-09-21T13:41:49.376 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/usr/bin/stdin-killer 2024-09-21T13:41:49.403 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-21T13:41:49.470 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-21T13:41:49.556 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'CEPHADM_STRAY_DAEMON', 'CEPHADM_FAILED_DAEMON', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'log-only-match': ['CEPHADM_'], 'sha1': '441784946648ff53dd310293f9dd752afeeea72a', 'cephadm_mode': 'root'} 2024-09-21T13:41:49.556 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a 2024-09-21T13:41:49.557 INFO:tasks.cephadm:Cluster fsid is 40a3d2e8-781f-11ef-baf6-efdc52797490 2024-09-21T13:41:49.557 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-21T13:41:49.557 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.49', 'mon.c': '[v2:172.21.15.49:3301,v1:172.21.15.49:6790]', 'mon.b': '172.21.15.187'} 2024-09-21T13:41:49.557 INFO:tasks.cephadm:First mon is mon.a on smithi049 2024-09-21T13:41:49.557 INFO:tasks.cephadm:First mgr is y 2024-09-21T13:41:49.557 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-21T13:41:49.558 DEBUG:teuthology.orchestra.run.smithi049:> sudo hostname $(hostname -s) 2024-09-21T13:41:49.585 DEBUG:teuthology.orchestra.run.smithi187:> sudo hostname $(hostname -s) 2024-09-21T13:41:49.612 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-09-21T13:41:49.612 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=441784946648ff53dd310293f9dd752afeeea72a 2024-09-21T13:41:49.751 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '441784946648ff53dd310293f9dd752afeeea72a', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/83203/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.3.0-5104-g44178494', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.3.0-5104.g44178494'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-21-0842/441784946648ff53dd310293f9dd752afeeea72a/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-09-21 10:26:40.656703', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-guits-main-2024-09-21-0842', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-guits-main-2024-09-21-0842/441784946648ff53dd310293f9dd752afeeea72a/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-09-21T13:41:49.885 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref wip-guits-main-2024-09-21-0842, sha1 441784946648ff53dd310293f9dd752afeeea72a from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=441784946648ff53dd310293f9dd752afeeea72a 2024-09-21T13:41:49.887 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/wip-guits-main-2024-09-21-0842/441784946648ff53dd310293f9dd752afeeea72a/centos/9/x86_64/flavors/default/cephadm 2024-09-21T13:41:49.887 INFO:tasks.cephadm:Downloading cephadm from url: https://3.chacra.ceph.com/binaries/ceph/wip-guits-main-2024-09-21-0842/441784946648ff53dd310293f9dd752afeeea72a/centos/9/x86_64/flavors/default/cephadm 2024-09-21T13:41:49.887 DEBUG:teuthology.orchestra.run.smithi049:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-guits-main-2024-09-21-0842/441784946648ff53dd310293f9dd752afeeea72a/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-21T13:41:50.395 INFO:teuthology.orchestra.run.smithi049.stdout:-rw-r--r--. 1 ubuntu ubuntu 804351 Sep 21 13:41 /home/ubuntu/cephtest/cephadm 2024-09-21T13:41:50.396 DEBUG:teuthology.orchestra.run.smithi187:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-guits-main-2024-09-21-0842/441784946648ff53dd310293f9dd752afeeea72a/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-21T13:41:51.004 INFO:teuthology.orchestra.run.smithi187.stdout:-rw-r--r--. 1 ubuntu ubuntu 804351 Sep 21 13:41 /home/ubuntu/cephtest/cephadm 2024-09-21T13:41:51.069 DEBUG:teuthology.orchestra.run.smithi049:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-21T13:41:51.090 DEBUG:teuthology.orchestra.run.smithi187:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-21T13:41:51.125 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a on all hosts... 2024-09-21T13:41:51.126 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a pull 2024-09-21T13:41:51.132 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a pull 2024-09-21T13:41:52.806 INFO:teuthology.orchestra.run.smithi049.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a... 2024-09-21T13:41:52.827 INFO:teuthology.orchestra.run.smithi187.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a... 2024-09-21T13:42:32.023 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T13:42:32.024 INFO:teuthology.orchestra.run.smithi187.stdout: "ceph_version": "ceph version 19.3.0-5104-g44178494 (441784946648ff53dd310293f9dd752afeeea72a) squid (dev)", 2024-09-21T13:42:32.024 INFO:teuthology.orchestra.run.smithi187.stdout: "image_id": "b415816db8b41f544cdfdc71d193ff8b1ee008bf7b0ec4f64da8eb8f71b62e85", 2024-09-21T13:42:32.024 INFO:teuthology.orchestra.run.smithi187.stdout: "repo_digests": [ 2024-09-21T13:42:32.024 INFO:teuthology.orchestra.run.smithi187.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5" 2024-09-21T13:42:32.024 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:42:32.024 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T13:42:44.751 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T13:42:44.751 INFO:teuthology.orchestra.run.smithi049.stdout: "ceph_version": "ceph version 19.3.0-5104-g44178494 (441784946648ff53dd310293f9dd752afeeea72a) squid (dev)", 2024-09-21T13:42:44.752 INFO:teuthology.orchestra.run.smithi049.stdout: "image_id": "b415816db8b41f544cdfdc71d193ff8b1ee008bf7b0ec4f64da8eb8f71b62e85", 2024-09-21T13:42:44.752 INFO:teuthology.orchestra.run.smithi049.stdout: "repo_digests": [ 2024-09-21T13:42:44.752 INFO:teuthology.orchestra.run.smithi049.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5" 2024-09-21T13:42:44.752 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:42:44.752 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T13:42:44.777 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /etc/ceph 2024-09-21T13:42:44.828 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /etc/ceph 2024-09-21T13:42:44.862 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 777 /etc/ceph 2024-09-21T13:42:44.887 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod 777 /etc/ceph 2024-09-21T13:42:44.930 INFO:tasks.cephadm:Writing seed config... 2024-09-21T13:42:44.931 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-21T13:42:44.931 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-21T13:42:44.931 INFO:tasks.cephadm: override: [client.0] debug ms = 1 2024-09-21T13:42:44.931 INFO:tasks.cephadm: override: [client.0] debug objecter = 20 2024-09-21T13:42:44.931 INFO:tasks.cephadm: override: [client.0] debug rados = 20 2024-09-21T13:42:44.931 INFO:tasks.cephadm: override: [global] ms type = async 2024-09-21T13:42:44.931 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-21T13:42:44.931 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-21T13:42:44.931 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-21T13:42:44.931 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-09-21T13:42:44.932 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-09-21T13:42:44.932 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-09-21T13:42:44.932 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-09-21T13:42:44.932 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-09-21T13:42:44.932 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-21T13:42:44.932 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-21T13:42:44.932 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2024-09-21T13:42:44.932 INFO:tasks.cephadm: override: [osd] osd delete sleep = 1 2024-09-21T13:42:44.932 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2024-09-21T13:42:44.932 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-09-21T13:42:44.932 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-09-21T13:42:44.932 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-09-21T13:42:44.932 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2024-09-21T13:42:44.933 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-09-21T13:42:44.933 DEBUG:teuthology.orchestra.run.smithi049:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-21T13:42:44.949 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 40a3d2e8-781f-11ef-baf6-efdc52797490 ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd debug reject backfill probability = 0.3 osd delete sleep = 1 osd max backfills = 3 osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client.0] debug ms = 1 debug objecter = 20 debug rados = 20 2024-09-21T13:42:44.949 DEBUG:teuthology.orchestra.run.smithi049:mon.a> sudo journalctl -f -n 0 -u ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@mon.a.service 2024-09-21T13:42:44.992 DEBUG:teuthology.orchestra.run.smithi049:mgr.y> sudo journalctl -f -n 0 -u ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@mgr.y.service 2024-09-21T13:42:45.034 INFO:tasks.cephadm:Bootstrapping... 2024-09-21T13:42:45.034 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a -v bootstrap --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.49 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-21T13:42:45.244 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-09-21T13:42:45.244 INFO:teuthology.orchestra.run.smithi049.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a', '-v', 'bootstrap', '--fsid', '40a3d2e8-781f-11ef-baf6-efdc52797490', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.49', '--skip-admin-label'] 2024-09-21T13:42:45.245 INFO:teuthology.orchestra.run.smithi049.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-09-21T13:42:45.245 INFO:teuthology.orchestra.run.smithi049.stdout:Verifying podman|docker is present... 2024-09-21T13:42:45.268 INFO:teuthology.orchestra.run.smithi049.stdout:/bin/podman: stdout 5.2.2 2024-09-21T13:42:45.269 INFO:teuthology.orchestra.run.smithi049.stdout:Verifying lvm2 is present... 2024-09-21T13:42:45.269 INFO:teuthology.orchestra.run.smithi049.stdout:Verifying time synchronization is in place... 2024-09-21T13:42:45.277 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-21T13:42:45.277 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-21T13:42:45.283 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-21T13:42:45.283 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout inactive 2024-09-21T13:42:45.290 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout enabled 2024-09-21T13:42:45.296 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout active 2024-09-21T13:42:45.296 INFO:teuthology.orchestra.run.smithi049.stdout:Unit chronyd.service is enabled and running 2024-09-21T13:42:45.296 INFO:teuthology.orchestra.run.smithi049.stdout:Repeating the final host check... 2024-09-21T13:42:45.318 INFO:teuthology.orchestra.run.smithi049.stdout:/bin/podman: stdout 5.2.2 2024-09-21T13:42:45.318 INFO:teuthology.orchestra.run.smithi049.stdout:podman (/bin/podman) version 5.2.2 is present 2024-09-21T13:42:45.318 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl is present 2024-09-21T13:42:45.319 INFO:teuthology.orchestra.run.smithi049.stdout:lvcreate is present 2024-09-21T13:42:45.325 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-21T13:42:45.325 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-21T13:42:45.332 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-21T13:42:45.332 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout inactive 2024-09-21T13:42:45.338 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout enabled 2024-09-21T13:42:45.345 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout active 2024-09-21T13:42:45.345 INFO:teuthology.orchestra.run.smithi049.stdout:Unit chronyd.service is enabled and running 2024-09-21T13:42:45.345 INFO:teuthology.orchestra.run.smithi049.stdout:Host looks OK 2024-09-21T13:42:45.345 INFO:teuthology.orchestra.run.smithi049.stdout:Cluster fsid: 40a3d2e8-781f-11ef-baf6-efdc52797490 2024-09-21T13:42:45.345 INFO:teuthology.orchestra.run.smithi049.stdout:Acquiring lock 140085882642048 on /run/cephadm/40a3d2e8-781f-11ef-baf6-efdc52797490.lock 2024-09-21T13:42:45.345 INFO:teuthology.orchestra.run.smithi049.stdout:Lock 140085882642048 acquired on /run/cephadm/40a3d2e8-781f-11ef-baf6-efdc52797490.lock 2024-09-21T13:42:45.346 INFO:teuthology.orchestra.run.smithi049.stdout:Verifying IP 172.21.15.49 port 3300 ... 2024-09-21T13:42:45.346 INFO:teuthology.orchestra.run.smithi049.stdout:Verifying IP 172.21.15.49 port 6789 ... 2024-09-21T13:42:45.346 INFO:teuthology.orchestra.run.smithi049.stdout:Base mon IP(s) is [172.21.15.49:3300, 172.21.15.49:6789], mon addrv is [v2:172.21.15.49:3300,v1:172.21.15.49:6789] 2024-09-21T13:42:45.349 INFO:teuthology.orchestra.run.smithi049.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.49 metric 100 2024-09-21T13:42:45.349 INFO:teuthology.orchestra.run.smithi049.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.49 metric 100 2024-09-21T13:42:45.352 INFO:teuthology.orchestra.run.smithi049.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-09-21T13:42:45.352 INFO:teuthology.orchestra.run.smithi049.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2024-09-21T13:42:45.352 INFO:teuthology.orchestra.run.smithi049.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1797sec hoplimit 64 pref medium 2024-09-21T13:42:45.355 INFO:teuthology.orchestra.run.smithi049.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-21T13:42:45.355 INFO:teuthology.orchestra.run.smithi049.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-09-21T13:42:45.355 INFO:teuthology.orchestra.run.smithi049.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-21T13:42:45.355 INFO:teuthology.orchestra.run.smithi049.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-09-21T13:42:45.355 INFO:teuthology.orchestra.run.smithi049.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:1566/64 scope link 2024-09-21T13:42:45.355 INFO:teuthology.orchestra.run.smithi049.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-21T13:42:45.356 INFO:teuthology.orchestra.run.smithi049.stdout:Mon IP `172.21.15.49` is in CIDR network `172.21.0.0/20` 2024-09-21T13:42:45.356 INFO:teuthology.orchestra.run.smithi049.stdout:Mon IP `172.21.15.49` is in CIDR network `172.21.0.0/20` 2024-09-21T13:42:45.356 INFO:teuthology.orchestra.run.smithi049.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-09-21T13:42:45.356 INFO:teuthology.orchestra.run.smithi049.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-21T13:42:45.357 INFO:teuthology.orchestra.run.smithi049.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a... 2024-09-21T13:42:45.862 INFO:teuthology.orchestra.run.smithi049.stdout:/bin/podman: stdout b415816db8b41f544cdfdc71d193ff8b1ee008bf7b0ec4f64da8eb8f71b62e85 2024-09-21T13:42:45.863 INFO:teuthology.orchestra.run.smithi049.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a... 2024-09-21T13:42:45.863 INFO:teuthology.orchestra.run.smithi049.stdout:/bin/podman: stderr Getting image source signatures 2024-09-21T13:42:45.863 INFO:teuthology.orchestra.run.smithi049.stdout:/bin/podman: stderr Copying blob sha256:d3c61a6c378e0e3b0f2d0c744b48039813ea4b374665619799f973b00e21a218 2024-09-21T13:42:45.863 INFO:teuthology.orchestra.run.smithi049.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-09-21T13:42:45.863 INFO:teuthology.orchestra.run.smithi049.stdout:/bin/podman: stderr Copying config sha256:b415816db8b41f544cdfdc71d193ff8b1ee008bf7b0ec4f64da8eb8f71b62e85 2024-09-21T13:42:45.863 INFO:teuthology.orchestra.run.smithi049.stdout:/bin/podman: stderr Writing manifest to image destination 2024-09-21T13:42:46.850 INFO:teuthology.orchestra.run.smithi049.stdout:ceph: stdout ceph version 19.3.0-5104-g44178494 (441784946648ff53dd310293f9dd752afeeea72a) squid (dev) 2024-09-21T13:42:46.850 INFO:teuthology.orchestra.run.smithi049.stdout:Ceph version: ceph version 19.3.0-5104-g44178494 (441784946648ff53dd310293f9dd752afeeea72a) squid (dev) 2024-09-21T13:42:46.850 INFO:teuthology.orchestra.run.smithi049.stdout:Extracting ceph user uid/gid from container image... 2024-09-21T13:42:47.906 INFO:teuthology.orchestra.run.smithi049.stdout:stat: stdout 167 167 2024-09-21T13:42:47.906 INFO:teuthology.orchestra.run.smithi049.stdout:Creating initial keys... 2024-09-21T13:42:48.852 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph-authtool: stdout AQBYze5mxM9NDxAA4aLRJjxKBI6eyL1xSuinTw== 2024-09-21T13:42:49.783 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph-authtool: stdout AQBZze5mFCEwCxAAeCarcoUw+rzWvGngEQ9tXQ== 2024-09-21T13:42:50.705 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph-authtool: stdout AQBaze5mqiyFBxAAFL9uuuaKphMW68e0k3lb7A== 2024-09-21T13:42:50.706 INFO:teuthology.orchestra.run.smithi049.stdout:Creating initial monmap... 2024-09-21T13:42:51.594 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-21T13:42:51.594 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-09-21T13:42:51.594 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 40a3d2e8-781f-11ef-baf6-efdc52797490 2024-09-21T13:42:51.595 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-21T13:42:51.595 INFO:teuthology.orchestra.run.smithi049.stdout:monmaptool for a [v2:172.21.15.49:3300,v1:172.21.15.49:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-21T13:42:51.595 INFO:teuthology.orchestra.run.smithi049.stdout:setting min_mon_release = quincy 2024-09-21T13:42:51.595 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/monmaptool: set fsid to 40a3d2e8-781f-11ef-baf6-efdc52797490 2024-09-21T13:42:51.595 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-21T13:42:51.595 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:42:51.595 INFO:teuthology.orchestra.run.smithi049.stdout:Creating mon... 2024-09-21T13:42:52.517 INFO:teuthology.orchestra.run.smithi049.stdout:create mon.a on 2024-09-21T13:42:52.733 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-09-21T13:42:52.883 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-09-21T13:42:53.631 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-40a3d2e8-781f-11ef-baf6-efdc52797490.target → /etc/systemd/system/ceph-40a3d2e8-781f-11ef-baf6-efdc52797490.target. 2024-09-21T13:42:53.631 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-40a3d2e8-781f-11ef-baf6-efdc52797490.target → /etc/systemd/system/ceph-40a3d2e8-781f-11ef-baf6-efdc52797490.target. 2024-09-21T13:42:54.457 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@mon.a 2024-09-21T13:42:54.457 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stderr Failed to reset failed state of unit ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@mon.a.service: Unit ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@mon.a.service not loaded. 2024-09-21T13:42:54.687 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-40a3d2e8-781f-11ef-baf6-efdc52797490.target.wants/ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@mon.a.service → /etc/systemd/system/ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@.service. 2024-09-21T13:42:55.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:42:55 smithi049 podman[29574]: 2024-09-21 13:42:54.952288303 +0000 UTC m=+0.016679520 image pull b415816db8b41f544cdfdc71d193ff8b1ee008bf7b0ec4f64da8eb8f71b62e85 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a 2024-09-21T13:42:56.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:42:55 smithi049 podman[29574]: 2024-09-21 13:42:55.95650351 +0000 UTC m=+1.020894717 container create a6698a9b1964b03790c3fdee1abfeafabe7c2bfce78f607d63610d59aac9ce0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mon-a, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-21T13:42:58.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:42:57 smithi049 podman[29574]: 2024-09-21 13:42:57.787888205 +0000 UTC m=+2.852279423 container init a6698a9b1964b03790c3fdee1abfeafabe7c2bfce78f607d63610d59aac9ce0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mon-a, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731) 2024-09-21T13:42:58.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:42:57 smithi049 podman[29574]: 2024-09-21 13:42:57.793766143 +0000 UTC m=+2.858157351 container start a6698a9b1964b03790c3fdee1abfeafabe7c2bfce78f607d63610d59aac9ce0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mon-a, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-21-0842-4417849, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T13:42:58.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:42:57 smithi049 ceph-mon[29612]: mkfs 40a3d2e8-781f-11ef-baf6-efdc52797490 2024-09-21T13:42:58.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:42:57 smithi049 ceph-mon[29612]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-21T13:42:58.807 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-21T13:42:58.807 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout disabled 2024-09-21T13:42:58.816 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-21T13:42:58.816 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout inactive 2024-09-21T13:42:58.816 INFO:teuthology.orchestra.run.smithi049.stdout:firewalld.service is not enabled 2024-09-21T13:42:58.816 INFO:teuthology.orchestra.run.smithi049.stdout:Not possible to enable service . firewalld.service is not available 2024-09-21T13:42:58.817 INFO:teuthology.orchestra.run.smithi049.stdout:Waiting for mon to start... 2024-09-21T13:42:58.817 INFO:teuthology.orchestra.run.smithi049.stdout:Waiting for mon... 2024-09-21T13:42:59.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:42:58 smithi049 bash[29574]: a6698a9b1964b03790c3fdee1abfeafabe7c2bfce78f607d63610d59aac9ce0f 2024-09-21T13:42:59.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:42:58 smithi049 systemd[1]: Started Ceph mon.a for 40a3d2e8-781f-11ef-baf6-efdc52797490. 2024-09-21T13:42:59.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:42:58 smithi049 ceph-mon[29612]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-21T13:42:59.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:42:58 smithi049 ceph-mon[29612]: monmap epoch 1 2024-09-21T13:42:59.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:42:58 smithi049 ceph-mon[29612]: fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 2024-09-21T13:42:59.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:42:58 smithi049 ceph-mon[29612]: last_changed 2024-09-21T13:42:51.046146+0000 2024-09-21T13:42:59.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:42:58 smithi049 ceph-mon[29612]: created 2024-09-21T13:42:51.046146+0000 2024-09-21T13:42:59.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:42:58 smithi049 ceph-mon[29612]: min_mon_release 19 (squid) 2024-09-21T13:42:59.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:42:58 smithi049 ceph-mon[29612]: election_strategy: 1 2024-09-21T13:42:59.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:42:58 smithi049 ceph-mon[29612]: 0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-09-21T13:42:59.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:42:58 smithi049 ceph-mon[29612]: fsmap 2024-09-21T13:42:59.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:42:58 smithi049 ceph-mon[29612]: osdmap e1: 0 total, 0 up, 0 in 2024-09-21T13:42:59.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:42:58 smithi049 ceph-mon[29612]: mgrmap e1: no daemons active 2024-09-21T13:43:02.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:02 smithi049 ceph-mon[29612]: from='client.? 172.21.15.49:0/1740664834' entity='client.admin' cmd={"prefix": "status"} : dispatch 2024-09-21T13:43:05.928 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout cluster: 2024-09-21T13:43:05.930 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout id: 40a3d2e8-781f-11ef-baf6-efdc52797490 2024-09-21T13:43:05.930 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-09-21T13:43:05.931 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2024-09-21T13:43:05.931 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout services: 2024-09-21T13:43:05.931 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 4s) 2024-09-21T13:43:05.931 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-09-21T13:43:05.931 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-09-21T13:43:05.931 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2024-09-21T13:43:05.931 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout data: 2024-09-21T13:43:05.931 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-09-21T13:43:05.931 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-09-21T13:43:05.931 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-09-21T13:43:05.931 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout pgs: 2024-09-21T13:43:05.931 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2024-09-21T13:43:05.931 INFO:teuthology.orchestra.run.smithi049.stdout:mon is available 2024-09-21T13:43:05.931 INFO:teuthology.orchestra.run.smithi049.stdout:Assimilating anything we can from ceph.conf... 2024-09-21T13:43:06.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:06 smithi049 ceph-mon[29612]: from='client.? 172.21.15.49:0/1169388490' entity='client.admin' cmd={"prefix": "config assimilate-conf"} : dispatch 2024-09-21T13:43:06.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:06 smithi049 ceph-mon[29612]: from='client.? 172.21.15.49:0/1169388490' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-21T13:43:07.010 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2024-09-21T13:43:07.011 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout [global] 2024-09-21T13:43:07.011 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout fsid = 40a3d2e8-781f-11ef-baf6-efdc52797490 2024-09-21T13:43:07.011 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-21T13:43:07.011 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.49:3300,v1:172.21.15.49:6789] 2024-09-21T13:43:07.011 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-21T13:43:07.011 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-21T13:43:07.011 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-21T13:43:07.011 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-21T13:43:07.011 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2024-09-21T13:43:07.011 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-21T13:43:07.011 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-21T13:43:07.011 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2024-09-21T13:43:07.012 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout [osd] 2024-09-21T13:43:07.012 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-21T13:43:07.012 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-21T13:43:07.012 INFO:teuthology.orchestra.run.smithi049.stdout:Generating new minimal ceph.conf... 2024-09-21T13:43:07.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:07 smithi049 ceph-mon[29612]: from='client.? 172.21.15.49:0/3807268498' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:43:08.057 INFO:teuthology.orchestra.run.smithi049.stdout:Restarting the monitor... 2024-09-21T13:43:08.351 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:08 smithi049 systemd[1]: Stopping Ceph mon.a for 40a3d2e8-781f-11ef-baf6-efdc52797490... 2024-09-21T13:43:08.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:08 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mon-a[29589]: 2024-09-21T13:43:08.349+0000 7f8075021640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T13:43:08.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:08 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mon-a[29589]: 2024-09-21T13:43:08.349+0000 7f8075021640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-21T13:43:08.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:08 smithi049 podman[29931]: 2024-09-21 13:43:08.53428976 +0000 UTC m=+0.308277534 container died a6698a9b1964b03790c3fdee1abfeafabe7c2bfce78f607d63610d59aac9ce0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True) 2024-09-21T13:43:09.141 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:08 smithi049 podman[29931]: 2024-09-21 13:43:08.881529137 +0000 UTC m=+0.655516917 container cleanup a6698a9b1964b03790c3fdee1abfeafabe7c2bfce78f607d63610d59aac9ce0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mon-a, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.build-date=20240731) 2024-09-21T13:43:09.141 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:08 smithi049 bash[29931]: ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mon-a 2024-09-21T13:43:09.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 podman[29949]: 2024-09-21 13:43:09.140739433 +0000 UTC m=+0.605187637 container remove a6698a9b1964b03790c3fdee1abfeafabe7c2bfce78f607d63610d59aac9ce0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mon-a, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-21-0842-4417849) 2024-09-21T13:43:09.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 systemd[1]: ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@mon.a.service: Deactivated successfully. 2024-09-21T13:43:09.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 systemd[1]: Stopped Ceph mon.a for 40a3d2e8-781f-11ef-baf6-efdc52797490. 2024-09-21T13:43:09.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 systemd[1]: Starting Ceph mon.a for 40a3d2e8-781f-11ef-baf6-efdc52797490... 2024-09-21T13:43:09.771 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 podman[30042]: 2024-09-21 13:43:09.422163434 +0000 UTC m=+0.016428885 image pull b415816db8b41f544cdfdc71d193ff8b1ee008bf7b0ec4f64da8eb8f71b62e85 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a 2024-09-21T13:43:09.771 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 podman[30042]: 2024-09-21 13:43:09.52519707 +0000 UTC m=+0.119462562 container create 5ef18ce25de2fa63e69656e4aff529955e8cc0a86e7f783e183cc4634d08d806 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-21-0842-4417849, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T13:43:09.771 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 podman[30042]: 2024-09-21 13:43:09.694383884 +0000 UTC m=+0.288649342 container init 5ef18ce25de2fa63e69656e4aff529955e8cc0a86e7f783e183cc4634d08d806 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mon-a, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2) 2024-09-21T13:43:09.771 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 podman[30042]: 2024-09-21 13:43:09.699032244 +0000 UTC m=+0.293297695 container start 5ef18ce25de2fa63e69656e4aff529955e8cc0a86e7f783e183cc4634d08d806 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mon-a, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.build-date=20240731) 2024-09-21T13:43:09.771 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: set uid:gid to 167:167 (ceph:ceph) 2024-09-21T13:43:09.772 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: ceph version 19.3.0-5104-g44178494 (441784946648ff53dd310293f9dd752afeeea72a) squid (dev), process ceph-mon, pid 7 2024-09-21T13:43:09.772 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: pidfile_write: ignore empty --pid-file 2024-09-21T13:43:09.772 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: load: jerasure load: lrc 2024-09-21T13:43:09.772 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: RocksDB version: 7.9.2 2024-09-21T13:43:09.772 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Git sha 0 2024-09-21T13:43:09.772 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Compile date 2024-09-21 09:24:23 2024-09-21T13:43:09.772 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: DB SUMMARY 2024-09-21T13:43:09.772 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: DB Session ID: CQH6ZN0ZCZO36GKTSQU3 2024-09-21T13:43:09.772 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: CURRENT file: CURRENT 2024-09-21T13:43:09.772 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: IDENTITY file: IDENTITY 2024-09-21T13:43:09.772 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-09-21T13:43:09.772 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-09-21T13:43:09.772 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 95303 ; 2024-09-21T13:43:09.772 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.error_if_exists: 0 2024-09-21T13:43:09.773 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.create_if_missing: 0 2024-09-21T13:43:09.773 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.paranoid_checks: 1 2024-09-21T13:43:09.773 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-21T13:43:09.773 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-21T13:43:09.773 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-21T13:43:09.773 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.env: 0x56242e73f120 2024-09-21T13:43:09.773 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.fs: PosixFileSystem 2024-09-21T13:43:09.774 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.info_log: 0x562430667700 2024-09-21T13:43:09.774 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.max_file_opening_threads: 16 2024-09-21T13:43:09.774 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.statistics: (nil) 2024-09-21T13:43:09.774 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.use_fsync: 0 2024-09-21T13:43:09.774 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.max_log_file_size: 0 2024-09-21T13:43:09.774 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-21T13:43:09.774 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-21T13:43:09.774 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.keep_log_file_num: 1000 2024-09-21T13:43:09.774 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.recycle_log_file_num: 0 2024-09-21T13:43:09.774 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.allow_fallocate: 1 2024-09-21T13:43:09.774 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.allow_mmap_reads: 0 2024-09-21T13:43:09.774 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.allow_mmap_writes: 0 2024-09-21T13:43:09.775 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.use_direct_reads: 0 2024-09-21T13:43:09.775 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-21T13:43:09.775 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.create_missing_column_families: 0 2024-09-21T13:43:09.775 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.db_log_dir: 2024-09-21T13:43:09.775 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.wal_dir: 2024-09-21T13:43:09.775 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-21T13:43:09.775 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-21T13:43:09.775 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-21T13:43:09.775 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-21T13:43:09.775 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-21T13:43:09.775 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-21T13:43:09.776 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.advise_random_on_open: 1 2024-09-21T13:43:09.776 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.db_write_buffer_size: 0 2024-09-21T13:43:09.776 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.write_buffer_manager: 0x56243066bb80 2024-09-21T13:43:09.776 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-21T13:43:09.776 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-21T13:43:09.776 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-21T13:43:09.776 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.rate_limiter: (nil) 2024-09-21T13:43:09.776 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-21T13:43:09.776 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.wal_recovery_mode: 2 2024-09-21T13:43:09.776 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.enable_thread_tracking: 0 2024-09-21T13:43:09.776 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.enable_pipelined_write: 0 2024-09-21T13:43:09.776 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.unordered_write: 0 2024-09-21T13:43:09.776 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-21T13:43:09.776 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-21T13:43:09.776 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-21T13:43:09.777 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-21T13:43:09.777 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.row_cache: None 2024-09-21T13:43:09.777 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.wal_filter: None 2024-09-21T13:43:09.777 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-21T13:43:09.777 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.allow_ingest_behind: 0 2024-09-21T13:43:09.777 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.two_write_queues: 0 2024-09-21T13:43:09.777 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.manual_wal_flush: 0 2024-09-21T13:43:09.777 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.wal_compression: 0 2024-09-21T13:43:09.777 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.atomic_flush: 0 2024-09-21T13:43:09.777 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-21T13:43:09.777 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-21T13:43:09.777 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-21T13:43:09.777 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.log_readahead_size: 0 2024-09-21T13:43:09.777 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-21T13:43:09.778 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.best_efforts_recovery: 0 2024-09-21T13:43:09.778 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-21T13:43:09.778 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-21T13:43:09.778 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.allow_data_in_errors: 0 2024-09-21T13:43:09.778 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.db_host_id: __hostname__ 2024-09-21T13:43:09.778 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-21T13:43:09.778 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.max_background_jobs: 2 2024-09-21T13:43:09.778 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.max_background_compactions: -1 2024-09-21T13:43:09.778 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.max_subcompactions: 1 2024-09-21T13:43:09.778 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-21T13:43:09.778 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-21T13:43:09.778 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-21T13:43:09.778 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.max_total_wal_size: 0 2024-09-21T13:43:09.778 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-21T13:43:09.779 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-21T13:43:09.779 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-21T13:43:09.779 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-21T13:43:09.779 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.max_open_files: -1 2024-09-21T13:43:09.779 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.bytes_per_sync: 0 2024-09-21T13:43:09.779 INFO:teuthology.orchestra.run.smithi049.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-09-21T13:43:09.780 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-21T13:43:09.781 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-21T13:43:09.781 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.compaction_readahead_size: 0 2024-09-21T13:43:09.781 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.max_background_flushes: -1 2024-09-21T13:43:09.781 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Compression algorithms supported: 2024-09-21T13:43:09.781 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: kZSTD supported: 0 2024-09-21T13:43:09.781 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: kXpressCompression supported: 0 2024-09-21T13:43:09.781 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: kBZip2Compression supported: 0 2024-09-21T13:43:09.781 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-21T13:43:09.781 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: kLZ4Compression supported: 1 2024-09-21T13:43:09.781 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: kZlibCompression supported: 1 2024-09-21T13:43:09.782 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: kLZ4HCCompression supported: 1 2024-09-21T13:43:09.782 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: kSnappyCompression supported: 1 2024-09-21T13:43:09.782 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-21T13:43:09.782 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-21T13:43:09.782 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-09-21T13:43:09.782 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-21T13:43:09.782 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-21T13:43:09.782 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.merge_operator: 2024-09-21T13:43:09.782 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.compaction_filter: None 2024-09-21T13:43:09.782 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.compaction_filter_factory: None 2024-09-21T13:43:09.782 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.sst_partitioner_factory: None 2024-09-21T13:43:09.782 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-21T13:43:09.783 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-21T13:43:09.783 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x562430667320) 2024-09-21T13:43:09.783 INFO:journalctl@ceph.mon.a.smithi049.stdout: cache_index_and_filter_blocks: 1 2024-09-21T13:43:09.783 INFO:journalctl@ceph.mon.a.smithi049.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-21T13:43:09.783 INFO:journalctl@ceph.mon.a.smithi049.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-21T13:43:09.783 INFO:journalctl@ceph.mon.a.smithi049.stdout: pin_top_level_index_and_filter: 1 2024-09-21T13:43:09.783 INFO:journalctl@ceph.mon.a.smithi049.stdout: index_type: 0 2024-09-21T13:43:09.784 INFO:journalctl@ceph.mon.a.smithi049.stdout: data_block_index_type: 0 2024-09-21T13:43:09.784 INFO:journalctl@ceph.mon.a.smithi049.stdout: index_shortening: 1 2024-09-21T13:43:09.784 INFO:journalctl@ceph.mon.a.smithi049.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-21T13:43:09.784 INFO:journalctl@ceph.mon.a.smithi049.stdout: checksum: 4 2024-09-21T13:43:09.784 INFO:journalctl@ceph.mon.a.smithi049.stdout: no_block_cache: 0 2024-09-21T13:43:09.784 INFO:journalctl@ceph.mon.a.smithi049.stdout: block_cache: 0x56243065b610 2024-09-21T13:43:09.784 INFO:journalctl@ceph.mon.a.smithi049.stdout: block_cache_name: BinnedLRUCache 2024-09-21T13:43:09.784 INFO:journalctl@ceph.mon.a.smithi049.stdout: block_cache_options: 2024-09-21T13:43:09.784 INFO:journalctl@ceph.mon.a.smithi049.stdout: capacity : 536870912 2024-09-21T13:43:09.784 INFO:journalctl@ceph.mon.a.smithi049.stdout: num_shard_bits : 4 2024-09-21T13:43:09.784 INFO:journalctl@ceph.mon.a.smithi049.stdout: strict_capacity_limit : 0 2024-09-21T13:43:09.784 INFO:journalctl@ceph.mon.a.smithi049.stdout: high_pri_pool_ratio: 0.000 2024-09-21T13:43:09.784 INFO:journalctl@ceph.mon.a.smithi049.stdout: block_cache_compressed: (nil) 2024-09-21T13:43:09.784 INFO:journalctl@ceph.mon.a.smithi049.stdout: persistent_cache: (nil) 2024-09-21T13:43:09.784 INFO:journalctl@ceph.mon.a.smithi049.stdout: block_size: 4096 2024-09-21T13:43:09.785 INFO:journalctl@ceph.mon.a.smithi049.stdout: block_size_deviation: 10 2024-09-21T13:43:09.785 INFO:journalctl@ceph.mon.a.smithi049.stdout: block_restart_interval: 16 2024-09-21T13:43:09.785 INFO:journalctl@ceph.mon.a.smithi049.stdout: index_block_restart_interval: 1 2024-09-21T13:43:09.785 INFO:journalctl@ceph.mon.a.smithi049.stdout: metadata_block_size: 4096 2024-09-21T13:43:09.785 INFO:journalctl@ceph.mon.a.smithi049.stdout: partition_filters: 0 2024-09-21T13:43:09.785 INFO:journalctl@ceph.mon.a.smithi049.stdout: use_delta_encoding: 1 2024-09-21T13:43:09.785 INFO:journalctl@ceph.mon.a.smithi049.stdout: filter_policy: bloomfilter 2024-09-21T13:43:09.785 INFO:journalctl@ceph.mon.a.smithi049.stdout: whole_key_filtering: 1 2024-09-21T13:43:09.785 INFO:journalctl@ceph.mon.a.smithi049.stdout: verify_compression: 0 2024-09-21T13:43:09.785 INFO:journalctl@ceph.mon.a.smithi049.stdout: read_amp_bytes_per_bit: 0 2024-09-21T13:43:09.785 INFO:journalctl@ceph.mon.a.smithi049.stdout: format_version: 5 2024-09-21T13:43:09.785 INFO:journalctl@ceph.mon.a.smithi049.stdout: enable_index_compression: 1 2024-09-21T13:43:09.785 INFO:journalctl@ceph.mon.a.smithi049.stdout: block_align: 0 2024-09-21T13:43:09.785 INFO:journalctl@ceph.mon.a.smithi049.stdout: max_auto_readahead_size: 262144 2024-09-21T13:43:09.785 INFO:journalctl@ceph.mon.a.smithi049.stdout: prepopulate_block_cache: 0 2024-09-21T13:43:09.786 INFO:journalctl@ceph.mon.a.smithi049.stdout: initial_auto_readahead_size: 8192 2024-09-21T13:43:09.786 INFO:journalctl@ceph.mon.a.smithi049.stdout: num_file_reads_for_auto_readahead: 2 2024-09-21T13:43:09.786 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.write_buffer_size: 33554432 2024-09-21T13:43:09.786 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.max_write_buffer_number: 2 2024-09-21T13:43:09.786 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.compression: NoCompression 2024-09-21T13:43:09.786 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.bottommost_compression: Disabled 2024-09-21T13:43:09.786 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.prefix_extractor: nullptr 2024-09-21T13:43:09.786 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-21T13:43:09.786 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.num_levels: 7 2024-09-21T13:43:09.786 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-21T13:43:09.786 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-21T13:43:09.786 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-21T13:43:09.786 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-21T13:43:09.786 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-21T13:43:09.787 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-21T13:43:09.787 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-21T13:43:09.787 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-21T13:43:09.787 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-21T13:43:09.787 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-21T13:43:09.787 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-21T13:43:09.787 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-21T13:43:09.787 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-21T13:43:09.787 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.compression_opts.level: 32767 2024-09-21T13:43:09.787 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.compression_opts.strategy: 0 2024-09-21T13:43:09.787 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-21T13:43:09.787 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-21T13:43:09.787 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-21T13:43:09.787 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-21T13:43:09.788 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.compression_opts.enabled: false 2024-09-21T13:43:09.788 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-21T13:43:09.788 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-21T13:43:09.788 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-21T13:43:09.788 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-21T13:43:09.788 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.target_file_size_base: 67108864 2024-09-21T13:43:09.788 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-21T13:43:09.788 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-21T13:43:09.788 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-21T13:43:09.788 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-21T13:43:09.788 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-21T13:43:09.788 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-21T13:43:09.788 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-21T13:43:09.788 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-21T13:43:09.789 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-21T13:43:09.789 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-21T13:43:09.789 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-21T13:43:09.789 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-21T13:43:09.789 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-21T13:43:09.789 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-21T13:43:09.789 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.arena_block_size: 1048576 2024-09-21T13:43:09.789 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-21T13:43:09.789 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-21T13:43:09.789 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.disable_auto_compactions: 0 2024-09-21T13:43:09.789 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-21T13:43:09.789 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-21T13:43:09.789 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-21T13:43:09.789 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-21T13:43:09.790 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-21T13:43:09.790 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-21T13:43:09.790 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-21T13:43:09.790 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-21T13:43:09.790 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-21T13:43:09.790 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-21T13:43:09.790 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-21T13:43:09.790 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.inplace_update_support: 0 2024-09-21T13:43:09.790 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-21T13:43:09.790 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-21T13:43:09.790 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-21T13:43:09.791 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-21T13:43:09.791 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.bloom_locality: 0 2024-09-21T13:43:09.791 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.max_successive_merges: 0 2024-09-21T13:43:09.791 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-21T13:43:09.791 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.paranoid_file_checks: 0 2024-09-21T13:43:09.791 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.force_consistency_checks: 1 2024-09-21T13:43:09.791 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.report_bg_io_stats: 0 2024-09-21T13:43:09.791 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.ttl: 2592000 2024-09-21T13:43:09.791 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-21T13:43:09.791 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-21T13:43:09.791 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-21T13:43:09.791 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.enable_blob_files: false 2024-09-21T13:43:09.791 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.min_blob_size: 0 2024-09-21T13:43:09.791 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.blob_file_size: 268435456 2024-09-21T13:43:09.792 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-21T13:43:09.792 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-21T13:43:09.792 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-21T13:43:09.792 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-21T13:43:09.792 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-21T13:43:09.792 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.blob_file_starting_level: 0 2024-09-21T13:43:09.792 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-21T13:43:09.792 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-09-21T13:43:09.792 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-09-21T13:43:09.792 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 4dcae047-82f2-4092-82c9-021d04105b95 2024-09-21T13:43:09.792 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726926189726150, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-09-21T13:43:09.792 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-09-21T13:43:09.792 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726926189727097, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 91212, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 291, "table_properties": {"data_size": 89257, "index_size": 275, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 11728, "raw_average_key_size": 47, "raw_value_size": 82738, "raw_average_value_size": 336, "num_data_blocks": 12, "num_entries": 246, "num_filter_entries": 246, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726926189, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "4dcae047-82f2-4092-82c9-021d04105b95", "db_session_id": "CQH6ZN0ZCZO36GKTSQU3", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-09-21T13:43:09.792 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726926189727202, "job": 1, "event": "recovery_finished"} 2024-09-21T13:43:09.793 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-09-21T13:43:09.793 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-21T13:43:09.793 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x56243068ae00 2024-09-21T13:43:09.793 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: DB pointer 0x56243078c000 2024-09-21T13:43:09.793 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-21T13:43:09.793 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-21T13:43:09.793 INFO:journalctl@ceph.mon.a.smithi049.stdout: ** DB Stats ** 2024-09-21T13:43:09.793 INFO:journalctl@ceph.mon.a.smithi049.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T13:43:09.793 INFO:journalctl@ceph.mon.a.smithi049.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-21T13:43:09.793 INFO:journalctl@ceph.mon.a.smithi049.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T13:43:09.793 INFO:journalctl@ceph.mon.a.smithi049.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T13:43:09.793 INFO:journalctl@ceph.mon.a.smithi049.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-21T13:43:09.793 INFO:journalctl@ceph.mon.a.smithi049.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T13:43:09.793 INFO:journalctl@ceph.mon.a.smithi049.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T13:43:09.794 INFO:journalctl@ceph.mon.a.smithi049.stdout: 2024-09-21T13:43:09.794 INFO:journalctl@ceph.mon.a.smithi049.stdout: ** Compaction Stats [default] ** 2024-09-21T13:43:09.794 INFO:journalctl@ceph.mon.a.smithi049.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T13:43:09.794 INFO:journalctl@ceph.mon.a.smithi049.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-21T13:43:09.794 INFO:journalctl@ceph.mon.a.smithi049.stdout: L0 2/0 90.93 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 150.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-21T13:43:09.794 INFO:journalctl@ceph.mon.a.smithi049.stdout: Sum 2/0 90.93 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 150.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-21T13:43:09.794 INFO:journalctl@ceph.mon.a.smithi049.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 150.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-21T13:43:09.794 INFO:journalctl@ceph.mon.a.smithi049.stdout: 2024-09-21T13:43:09.794 INFO:journalctl@ceph.mon.a.smithi049.stdout: ** Compaction Stats [default] ** 2024-09-21T13:43:09.794 INFO:journalctl@ceph.mon.a.smithi049.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T13:43:09.794 INFO:journalctl@ceph.mon.a.smithi049.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T13:43:09.794 INFO:journalctl@ceph.mon.a.smithi049.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 150.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-21T13:43:09.794 INFO:journalctl@ceph.mon.a.smithi049.stdout: 2024-09-21T13:43:09.794 INFO:journalctl@ceph.mon.a.smithi049.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-21T13:43:09.795 INFO:journalctl@ceph.mon.a.smithi049.stdout: 2024-09-21T13:43:09.795 INFO:journalctl@ceph.mon.a.smithi049.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T13:43:09.795 INFO:journalctl@ceph.mon.a.smithi049.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-21T13:43:09.795 INFO:journalctl@ceph.mon.a.smithi049.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-21T13:43:09.795 INFO:journalctl@ceph.mon.a.smithi049.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-21T13:43:09.795 INFO:journalctl@ceph.mon.a.smithi049.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-21T13:43:09.795 INFO:journalctl@ceph.mon.a.smithi049.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-21T13:43:09.795 INFO:journalctl@ceph.mon.a.smithi049.stdout: Cumulative compaction: 0.00 GB write, 28.45 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T13:43:09.795 INFO:journalctl@ceph.mon.a.smithi049.stdout: Interval compaction: 0.00 GB write, 28.45 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T13:43:09.795 INFO:journalctl@ceph.mon.a.smithi049.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-21T13:43:09.795 INFO:journalctl@ceph.mon.a.smithi049.stdout: Block cache BinnedLRUCache@0x56243065b610#7 capacity: 512.00 MB usage: 1.30 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-09-21T13:43:09.795 INFO:journalctl@ceph.mon.a.smithi049.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.83 KB,0.000157952%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-09-21T13:43:09.795 INFO:journalctl@ceph.mon.a.smithi049.stdout: 2024-09-21T13:43:09.795 INFO:journalctl@ceph.mon.a.smithi049.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-21T13:43:09.795 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: starting mon.a rank 0 at public addrs [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] at bind addrs [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 2024-09-21T13:43:09.796 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: mon.a@-1(???) e1 preinit fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 2024-09-21T13:43:09.796 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: mon.a@-1(???).mds e1 new map 2024-09-21T13:43:09.796 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: mon.a@-1(???).mds e1 print_map 2024-09-21T13:43:09.796 INFO:journalctl@ceph.mon.a.smithi049.stdout: e1 2024-09-21T13:43:09.796 INFO:journalctl@ceph.mon.a.smithi049.stdout: btime 2024-09-21T13:42:57:838094+0000 2024-09-21T13:43:09.796 INFO:journalctl@ceph.mon.a.smithi049.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-21T13:43:09.796 INFO:journalctl@ceph.mon.a.smithi049.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-21T13:43:09.796 INFO:journalctl@ceph.mon.a.smithi049.stdout: legacy client fscid: -1 2024-09-21T13:43:09.796 INFO:journalctl@ceph.mon.a.smithi049.stdout: 2024-09-21T13:43:09.796 INFO:journalctl@ceph.mon.a.smithi049.stdout: No filesystems configured 2024-09-21T13:43:09.796 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-21T13:43:09.796 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T13:43:09.796 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T13:43:09.796 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T13:43:09.796 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-21T13:43:09.796 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 bash[30042]: 5ef18ce25de2fa63e69656e4aff529955e8cc0a86e7f783e183cc4634d08d806 2024-09-21T13:43:09.796 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 systemd[1]: Started Ceph mon.a for 40a3d2e8-781f-11ef-baf6-efdc52797490. 2024-09-21T13:43:10.023 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-21T13:43:10.024 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: monmap epoch 1 2024-09-21T13:43:10.024 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 2024-09-21T13:43:10.024 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: last_changed 2024-09-21T13:42:51.046146+0000 2024-09-21T13:43:10.024 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: created 2024-09-21T13:42:51.046146+0000 2024-09-21T13:43:10.024 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: min_mon_release 19 (squid) 2024-09-21T13:43:10.024 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: election_strategy: 1 2024-09-21T13:43:10.024 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: 0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-09-21T13:43:10.024 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: fsmap 2024-09-21T13:43:10.024 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: osdmap e1: 0 total, 0 up, 0 in 2024-09-21T13:43:10.024 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:09 smithi049 ceph-mon[30080]: mgrmap e1: no daemons active 2024-09-21T13:43:11.666 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:11 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/4005809325' entity='client.admin' 2024-09-21T13:43:11.669 INFO:teuthology.orchestra.run.smithi049.stdout:Wrote config to /etc/ceph/ceph.conf 2024-09-21T13:43:11.698 INFO:teuthology.orchestra.run.smithi049.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-21T13:43:11.699 INFO:teuthology.orchestra.run.smithi049.stdout:Creating mgr... 2024-09-21T13:43:11.699 INFO:teuthology.orchestra.run.smithi049.stdout:Verifying port 0.0.0.0:9283 ... 2024-09-21T13:43:11.699 INFO:teuthology.orchestra.run.smithi049.stdout:Verifying port 0.0.0.0:8765 ... 2024-09-21T13:43:11.957 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@mgr.y 2024-09-21T13:43:11.957 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stderr Failed to reset failed state of unit ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@mgr.y.service: Unit ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@mgr.y.service not loaded. 2024-09-21T13:43:12.111 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-40a3d2e8-781f-11ef-baf6-efdc52797490.target.wants/ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@mgr.y.service → /etc/systemd/system/ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@.service. 2024-09-21T13:43:12.255 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:12 smithi049 systemd[1]: Starting Ceph mgr.y for 40a3d2e8-781f-11ef-baf6-efdc52797490... 2024-09-21T13:43:12.620 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:12 smithi049 podman[30318]: 2024-09-21 13:43:12.342204985 +0000 UTC m=+0.111956988 container create 0bf9f1993273ed20cf3091d6c7746dde435ba745f53b058916a879c6b6067c50 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-21-0842-4417849, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T13:43:12.620 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:12 smithi049 podman[30318]: 2024-09-21 13:43:12.246479474 +0000 UTC m=+0.016231494 image pull b415816db8b41f544cdfdc71d193ff8b1ee008bf7b0ec4f64da8eb8f71b62e85 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a 2024-09-21T13:43:12.620 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:12 smithi049 podman[30318]: 2024-09-21 13:43:12.544931669 +0000 UTC m=+0.314683681 container init 0bf9f1993273ed20cf3091d6c7746dde435ba745f53b058916a879c6b6067c50 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-21-0842-4417849, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-09-21T13:43:12.620 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:12 smithi049 podman[30318]: 2024-09-21 13:43:12.550087937 +0000 UTC m=+0.319839940 container start 0bf9f1993273ed20cf3091d6c7746dde435ba745f53b058916a879c6b6067c50 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-09-21T13:43:12.620 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:12 smithi049 bash[30318]: 0bf9f1993273ed20cf3091d6c7746dde435ba745f53b058916a879c6b6067c50 2024-09-21T13:43:12.635 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-21T13:43:12.635 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout disabled 2024-09-21T13:43:12.643 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-21T13:43:12.645 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout inactive 2024-09-21T13:43:12.645 INFO:teuthology.orchestra.run.smithi049.stdout:firewalld.service is not enabled 2024-09-21T13:43:12.645 INFO:teuthology.orchestra.run.smithi049.stdout:Not possible to enable service . firewalld.service is not available 2024-09-21T13:43:12.653 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-21T13:43:12.653 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout disabled 2024-09-21T13:43:12.661 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-21T13:43:12.661 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout inactive 2024-09-21T13:43:12.661 INFO:teuthology.orchestra.run.smithi049.stdout:firewalld.service is not enabled 2024-09-21T13:43:12.661 INFO:teuthology.orchestra.run.smithi049.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-09-21T13:43:12.661 INFO:teuthology.orchestra.run.smithi049.stdout:Waiting for mgr to start... 2024-09-21T13:43:12.661 INFO:teuthology.orchestra.run.smithi049.stdout:Waiting for mgr... 2024-09-21T13:43:12.900 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:12 smithi049 systemd[1]: Started Ceph mgr.y for 40a3d2e8-781f-11ef-baf6-efdc52797490. 2024-09-21T13:43:13.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:13 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1591397978' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-21T13:43:13.839 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2024-09-21T13:43:13.840 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout { 2024-09-21T13:43:13.840 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "fsid": "40a3d2e8-781f-11ef-baf6-efdc52797490", 2024-09-21T13:43:13.840 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "health": { 2024-09-21T13:43:13.840 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-21T13:43:13.840 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-21T13:43:13.840 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-21T13:43:13.840 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-09-21T13:43:13.840 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-21T13:43:13.840 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-21T13:43:13.840 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 0 2024-09-21T13:43:13.840 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout ], 2024-09-21T13:43:13.840 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-21T13:43:13.840 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "a" 2024-09-21T13:43:13.840 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout ], 2024-09-21T13:43:13.841 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-09-21T13:43:13.841 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-21T13:43:13.841 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T13:43:13.841 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-21T13:43:13.841 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-21T13:43:13.841 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-09-21T13:43:13.841 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-21T13:43:13.841 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T13:43:13.841 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-21T13:43:13.841 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-21T13:43:13.841 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-21T13:43:13.844 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-21T13:43:13.844 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-21T13:43:13.844 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-21T13:43:13.845 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-09-21T13:43:13.845 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-21T13:43:13.845 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-21T13:43:13.845 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-21T13:43:13.845 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-21T13:43:13.845 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-21T13:43:13.845 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-21T13:43:13.845 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-21T13:43:13.845 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-21T13:43:13.845 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-21T13:43:13.845 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-09-21T13:43:13.845 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-21T13:43:13.845 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T13:43:13.845 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "btime": "2024-09-21T13:42:57:838094+0000", 2024-09-21T13:43:13.846 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-21T13:43:13.846 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-21T13:43:13.846 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-09-21T13:43:13.846 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-21T13:43:13.846 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-21T13:43:13.846 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-21T13:43:13.846 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-21T13:43:13.846 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-21T13:43:13.846 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-21T13:43:13.846 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "restful" 2024-09-21T13:43:13.846 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout ], 2024-09-21T13:43:13.846 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T13:43:13.846 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-09-21T13:43:13.846 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-21T13:43:13.847 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T13:43:13.847 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "modified": "2024-09-21T13:42:57.838836+0000", 2024-09-21T13:43:13.847 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T13:43:13.847 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-09-21T13:43:13.847 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-21T13:43:13.847 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout } 2024-09-21T13:43:13.847 INFO:teuthology.orchestra.run.smithi049.stdout:mgr not available, waiting (1/15)... 2024-09-21T13:43:16.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:16 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2769300075' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-21T13:43:17.144 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2024-09-21T13:43:17.150 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout { 2024-09-21T13:43:17.150 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "fsid": "40a3d2e8-781f-11ef-baf6-efdc52797490", 2024-09-21T13:43:17.150 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "health": { 2024-09-21T13:43:17.150 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-21T13:43:17.150 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-21T13:43:17.150 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-21T13:43:17.150 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-09-21T13:43:17.150 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-21T13:43:17.150 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-21T13:43:17.150 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 0 2024-09-21T13:43:17.150 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout ], 2024-09-21T13:43:17.151 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-21T13:43:17.151 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "a" 2024-09-21T13:43:17.151 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout ], 2024-09-21T13:43:17.151 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-09-21T13:43:17.151 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-21T13:43:17.151 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T13:43:17.151 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-21T13:43:17.151 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-21T13:43:17.151 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-09-21T13:43:17.151 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-21T13:43:17.151 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T13:43:17.151 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-21T13:43:17.151 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-21T13:43:17.151 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-21T13:43:17.152 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-21T13:43:17.152 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-21T13:43:17.152 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-21T13:43:17.152 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-09-21T13:43:17.152 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-21T13:43:17.152 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-21T13:43:17.152 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-21T13:43:17.152 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-21T13:43:17.152 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-21T13:43:17.152 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-21T13:43:17.152 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-21T13:43:17.152 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-21T13:43:17.152 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-21T13:43:17.152 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-09-21T13:43:17.152 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-21T13:43:17.153 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T13:43:17.153 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "btime": "2024-09-21T13:42:57:838094+0000", 2024-09-21T13:43:17.153 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-21T13:43:17.153 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-21T13:43:17.153 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-09-21T13:43:17.153 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-21T13:43:17.153 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-21T13:43:17.153 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-21T13:43:17.153 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-21T13:43:17.153 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-21T13:43:17.153 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-21T13:43:17.153 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "restful" 2024-09-21T13:43:17.153 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout ], 2024-09-21T13:43:17.153 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T13:43:17.153 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-09-21T13:43:17.154 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-21T13:43:17.154 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T13:43:17.154 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "modified": "2024-09-21T13:42:57.838836+0000", 2024-09-21T13:43:17.154 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T13:43:17.154 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-09-21T13:43:17.154 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-21T13:43:17.154 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout } 2024-09-21T13:43:17.154 INFO:teuthology.orchestra.run.smithi049.stdout:mgr not available, waiting (2/15)... 2024-09-21T13:43:18.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:17 smithi049 ceph-mon[30080]: Activating manager daemon y 2024-09-21T13:43:18.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:17 smithi049 ceph-mon[30080]: mgrmap e2: y(active, starting, since 0.00316307s) 2024-09-21T13:43:18.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:17 smithi049 ceph-mon[30080]: from='mgr.14100 172.21.15.49:0/4035168193' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-21T13:43:18.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:17 smithi049 ceph-mon[30080]: from='mgr.14100 172.21.15.49:0/4035168193' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-21T13:43:18.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:17 smithi049 ceph-mon[30080]: from='mgr.14100 172.21.15.49:0/4035168193' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-21T13:43:18.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:17 smithi049 ceph-mon[30080]: from='mgr.14100 172.21.15.49:0/4035168193' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T13:43:18.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:17 smithi049 ceph-mon[30080]: from='mgr.14100 172.21.15.49:0/4035168193' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-21T13:43:18.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:17 smithi049 ceph-mon[30080]: Manager daemon y is now available 2024-09-21T13:43:18.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:17 smithi049 ceph-mon[30080]: from='mgr.14100 172.21.15.49:0/4035168193' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-21T13:43:18.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:17 smithi049 ceph-mon[30080]: from='mgr.14100 172.21.15.49:0/4035168193' entity='mgr.y' 2024-09-21T13:43:18.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:17 smithi049 ceph-mon[30080]: from='mgr.14100 172.21.15.49:0/4035168193' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-21T13:43:18.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:17 smithi049 ceph-mon[30080]: from='mgr.14100 172.21.15.49:0/4035168193' entity='mgr.y' 2024-09-21T13:43:18.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:17 smithi049 ceph-mon[30080]: from='mgr.14100 172.21.15.49:0/4035168193' entity='mgr.y' 2024-09-21T13:43:20.205 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:19 smithi049 ceph-mon[30080]: mgrmap e3: y(active, since 1.00781s) 2024-09-21T13:43:20.514 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2024-09-21T13:43:20.514 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout { 2024-09-21T13:43:20.514 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "fsid": "40a3d2e8-781f-11ef-baf6-efdc52797490", 2024-09-21T13:43:20.514 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "health": { 2024-09-21T13:43:20.514 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-21T13:43:20.514 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-21T13:43:20.514 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-21T13:43:20.515 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-09-21T13:43:20.515 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-21T13:43:20.515 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-21T13:43:20.515 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 0 2024-09-21T13:43:20.515 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout ], 2024-09-21T13:43:20.515 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-21T13:43:20.515 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "a" 2024-09-21T13:43:20.515 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout ], 2024-09-21T13:43:20.515 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-09-21T13:43:20.515 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-21T13:43:20.515 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T13:43:20.515 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-21T13:43:20.515 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-21T13:43:20.515 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-09-21T13:43:20.516 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-21T13:43:20.516 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T13:43:20.516 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-21T13:43:20.516 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-21T13:43:20.516 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-21T13:43:20.516 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-21T13:43:20.516 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-21T13:43:20.516 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-21T13:43:20.516 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-09-21T13:43:20.516 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-21T13:43:20.516 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-21T13:43:20.516 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-21T13:43:20.516 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-21T13:43:20.516 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-21T13:43:20.516 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-21T13:43:20.517 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-21T13:43:20.517 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-21T13:43:20.517 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-21T13:43:20.517 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-09-21T13:43:20.517 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-21T13:43:20.517 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T13:43:20.517 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "btime": "2024-09-21T13:42:57:838094+0000", 2024-09-21T13:43:20.517 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-21T13:43:20.517 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-21T13:43:20.517 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-09-21T13:43:20.517 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-21T13:43:20.517 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-21T13:43:20.517 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-21T13:43:20.517 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-21T13:43:20.518 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-21T13:43:20.518 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-21T13:43:20.518 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "restful" 2024-09-21T13:43:20.518 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout ], 2024-09-21T13:43:20.518 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T13:43:20.518 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-09-21T13:43:20.518 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-21T13:43:20.518 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T13:43:20.518 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "modified": "2024-09-21T13:42:57.838836+0000", 2024-09-21T13:43:20.518 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T13:43:20.518 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-09-21T13:43:20.518 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-21T13:43:20.518 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout } 2024-09-21T13:43:20.518 INFO:teuthology.orchestra.run.smithi049.stdout:mgr is available 2024-09-21T13:43:21.173 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:20 smithi049 ceph-mon[30080]: mgrmap e4: y(active, since 2s) 2024-09-21T13:43:21.173 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:20 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1387975719' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-21T13:43:21.759 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2024-09-21T13:43:21.759 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout [global] 2024-09-21T13:43:21.759 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout fsid = 40a3d2e8-781f-11ef-baf6-efdc52797490 2024-09-21T13:43:21.759 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-21T13:43:21.759 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.49:3300,v1:172.21.15.49:6789] 2024-09-21T13:43:21.759 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-21T13:43:21.759 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-21T13:43:21.759 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-21T13:43:21.759 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-21T13:43:21.759 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2024-09-21T13:43:21.759 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-21T13:43:21.760 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-21T13:43:21.760 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2024-09-21T13:43:21.760 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout [osd] 2024-09-21T13:43:21.760 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-21T13:43:21.760 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-21T13:43:21.760 INFO:teuthology.orchestra.run.smithi049.stdout:Enabling cephadm module... 2024-09-21T13:43:22.007 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:21 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/4203813115' entity='client.admin' cmd={"prefix": "config assimilate-conf"} : dispatch 2024-09-21T13:43:23.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:22 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2447413985' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "cephadm"} : dispatch 2024-09-21T13:43:23.228 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:22 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ignoring --setuser ceph since I am not root 2024-09-21T13:43:23.228 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:22 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ignoring --setgroup ceph since I am not root 2024-09-21T13:43:24.196 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:23 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2447413985' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-21T13:43:24.196 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:23 smithi049 ceph-mon[30080]: mgrmap e5: y(active, since 5s) 2024-09-21T13:43:24.829 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout { 2024-09-21T13:43:24.830 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-09-21T13:43:24.830 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-21T13:43:24.830 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-21T13:43:24.830 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-21T13:43:24.830 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout } 2024-09-21T13:43:24.830 INFO:teuthology.orchestra.run.smithi049.stdout:Waiting for the mgr to restart... 2024-09-21T13:43:24.830 INFO:teuthology.orchestra.run.smithi049.stdout:Waiting for mgr epoch 5... 2024-09-21T13:43:25.105 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:24 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1551373008' entity='client.admin' cmd={"prefix": "mgr stat"} : dispatch 2024-09-21T13:43:28.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:28 smithi049 ceph-mon[30080]: Active manager daemon y restarted 2024-09-21T13:43:28.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:28 smithi049 ceph-mon[30080]: Activating manager daemon y 2024-09-21T13:43:28.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:28 smithi049 ceph-mon[30080]: osdmap e2: 0 total, 0 up, 0 in 2024-09-21T13:43:28.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:28 smithi049 ceph-mon[30080]: mgrmap e6: y(active, starting, since 0.00342714s) 2024-09-21T13:43:28.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:28 smithi049 ceph-mon[30080]: from='mgr.14118 172.21.15.49:0/2880154351' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T13:43:28.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:28 smithi049 ceph-mon[30080]: from='mgr.14118 172.21.15.49:0/2880154351' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-21T13:43:28.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:28 smithi049 ceph-mon[30080]: from='mgr.14118 172.21.15.49:0/2880154351' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-21T13:43:28.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:28 smithi049 ceph-mon[30080]: from='mgr.14118 172.21.15.49:0/2880154351' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-21T13:43:28.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:28 smithi049 ceph-mon[30080]: from='mgr.14118 172.21.15.49:0/2880154351' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-21T13:43:28.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:28 smithi049 ceph-mon[30080]: Manager daemon y is now available 2024-09-21T13:43:29.694 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout { 2024-09-21T13:43:29.694 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-09-21T13:43:29.694 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-21T13:43:29.694 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout } 2024-09-21T13:43:29.695 INFO:teuthology.orchestra.run.smithi049.stdout:mgr epoch 5 is available 2024-09-21T13:43:29.695 INFO:teuthology.orchestra.run.smithi049.stdout:Verifying orchestrator module is enabled... 2024-09-21T13:43:30.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:30 smithi049 ceph-mon[30080]: mgrmap e7: y(active, since 1.00517s) 2024-09-21T13:43:30.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:30 smithi049 ceph-mon[30080]: from='mgr.14118 172.21.15.49:0/2880154351' entity='mgr.y' 2024-09-21T13:43:30.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:30 smithi049 ceph-mon[30080]: from='mgr.14118 172.21.15.49:0/2880154351' entity='mgr.y' 2024-09-21T13:43:30.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:30 smithi049 ceph-mon[30080]: Found migration_current of "None". Setting to last migration. 2024-09-21T13:43:30.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:30 smithi049 ceph-mon[30080]: from='mgr.14118 172.21.15.49:0/2880154351' entity='mgr.y' 2024-09-21T13:43:30.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:30 smithi049 ceph-mon[30080]: from='mgr.14118 172.21.15.49:0/2880154351' entity='mgr.y' 2024-09-21T13:43:30.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:30 smithi049 ceph-mon[30080]: from='mgr.14118 172.21.15.49:0/2880154351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:43:30.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:30 smithi049 ceph-mon[30080]: from='mgr.14118 172.21.15.49:0/2880154351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:43:30.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:30 smithi049 ceph-mon[30080]: from='mgr.14118 172.21.15.49:0/2880154351' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-21T13:43:30.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:30 smithi049 ceph-mon[30080]: from='mgr.14118 172.21.15.49:0/2880154351' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-21T13:43:31.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:31 smithi049 ceph-mon[30080]: mgrmap e8: y(active, since 2s) 2024-09-21T13:43:31.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:31 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2854856067' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "orchestrator"} : dispatch 2024-09-21T13:43:31.800 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr module 'orchestrator' is already enabled (always-on) 2024-09-21T13:43:31.800 INFO:teuthology.orchestra.run.smithi049.stdout:Setting orchestrator backend to cephadm... 2024-09-21T13:43:32.159 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:31 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: 2024-09-21T13:43:31.809+0000 7f01eee2c640 -1 log_channel(cephadm) log [ERR] : [21/Sep/2024:13:43:31] ENGINE Error in HTTPServer.serve 2024-09-21T13:43:32.159 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:31 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: Traceback (most recent call last): 2024-09-21T13:43:32.160 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:31 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T13:43:32.160 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:31 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: self._connections.run(self.expiration_interval) 2024-09-21T13:43:32.160 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:31 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T13:43:32.160 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:31 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: self._run(expiration_interval) 2024-09-21T13:43:32.160 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:31 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T13:43:32.160 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:31 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T13:43:32.160 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:31 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T13:43:32.160 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:31 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T13:43:32.160 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:31 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T13:43:32.160 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:31 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: s = self.context.wrap_socket( 2024-09-21T13:43:32.160 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:31 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T13:43:32.160 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:31 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: return self.sslsocket_class._create( 2024-09-21T13:43:32.160 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:31 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T13:43:32.160 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:31 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: self.do_handshake() 2024-09-21T13:43:32.161 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:31 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T13:43:32.161 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:31 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: self._sslobj.do_handshake() 2024-09-21T13:43:32.161 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:31 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T13:43:32.161 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:31 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: 2024-09-21T13:43:32.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:32 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2854856067' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "orchestrator"}]': finished 2024-09-21T13:43:32.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:32 smithi049 ceph-mon[30080]: mgrmap e9: y(active, since 3s) 2024-09-21T13:43:32.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:32 smithi049 ceph-mon[30080]: [21/Sep/2024:13:43:31] ENGINE Bus STARTING 2024-09-21T13:43:32.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:32 smithi049 ceph-mon[30080]: [21/Sep/2024:13:43:31] ENGINE Serving on http://172.21.15.49:8765 2024-09-21T13:43:32.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:32 smithi049 ceph-mon[30080]: [21/Sep/2024:13:43:31] ENGINE Serving on https://172.21.15.49:7150 2024-09-21T13:43:32.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:32 smithi049 ceph-mon[30080]: [21/Sep/2024:13:43:31] ENGINE Bus STARTED 2024-09-21T13:43:32.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:32 smithi049 ceph-mon[30080]: from='mgr.14118 172.21.15.49:0/2880154351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:43:32.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:32 smithi049 ceph-mon[30080]: [21/Sep/2024:13:43:31] ENGINE Error in HTTPServer.serve 2024-09-21T13:43:32.472 INFO:journalctl@ceph.mon.a.smithi049.stdout: Traceback (most recent call last): 2024-09-21T13:43:32.472 INFO:journalctl@ceph.mon.a.smithi049.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T13:43:32.472 INFO:journalctl@ceph.mon.a.smithi049.stdout: self._connections.run(self.expiration_interval) 2024-09-21T13:43:32.472 INFO:journalctl@ceph.mon.a.smithi049.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T13:43:32.472 INFO:journalctl@ceph.mon.a.smithi049.stdout: self._run(expiration_interval) 2024-09-21T13:43:32.472 INFO:journalctl@ceph.mon.a.smithi049.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T13:43:32.472 INFO:journalctl@ceph.mon.a.smithi049.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T13:43:32.472 INFO:journalctl@ceph.mon.a.smithi049.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T13:43:32.473 INFO:journalctl@ceph.mon.a.smithi049.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T13:43:32.473 INFO:journalctl@ceph.mon.a.smithi049.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T13:43:32.473 INFO:journalctl@ceph.mon.a.smithi049.stdout: s = self.context.wrap_socket( 2024-09-21T13:43:32.473 INFO:journalctl@ceph.mon.a.smithi049.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T13:43:32.473 INFO:journalctl@ceph.mon.a.smithi049.stdout: return self.sslsocket_class._create( 2024-09-21T13:43:32.473 INFO:journalctl@ceph.mon.a.smithi049.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T13:43:32.473 INFO:journalctl@ceph.mon.a.smithi049.stdout: self.do_handshake() 2024-09-21T13:43:32.473 INFO:journalctl@ceph.mon.a.smithi049.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T13:43:32.473 INFO:journalctl@ceph.mon.a.smithi049.stdout: self._sslobj.do_handshake() 2024-09-21T13:43:32.473 INFO:journalctl@ceph.mon.a.smithi049.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T13:43:33.698 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:33 smithi049 ceph-mon[30080]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:43:33.698 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:33 smithi049 ceph-mon[30080]: from='mgr.14118 172.21.15.49:0/2880154351' entity='mgr.y' 2024-09-21T13:43:33.698 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:33 smithi049 ceph-mon[30080]: from='mgr.14118 172.21.15.49:0/2880154351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:43:34.222 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout value unchanged 2024-09-21T13:43:34.222 INFO:teuthology.orchestra.run.smithi049.stdout:Generating ssh key... 2024-09-21T13:43:34.671 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:34 smithi049 ceph-mon[30080]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:43:35.459 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:35 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: Generating public/private rsa key pair. 2024-09-21T13:43:35.459 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:35 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: Your identification has been saved in /tmp/tmpfvy3_4ns/key 2024-09-21T13:43:35.459 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:35 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: Your public key has been saved in /tmp/tmpfvy3_4ns/key.pub 2024-09-21T13:43:35.459 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:35 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: The key fingerprint is: 2024-09-21T13:43:35.460 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:35 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: SHA256:sOCFozmOfgpqkQ2sfsEurc5MBLVjg47xOoY/uaW6jbw ceph-40a3d2e8-781f-11ef-baf6-efdc52797490 2024-09-21T13:43:35.460 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:35 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: The key's randomart image is: 2024-09-21T13:43:35.460 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:35 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: +---[RSA 3072]----+ 2024-09-21T13:43:35.460 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:35 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: | . | 2024-09-21T13:43:35.460 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:35 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: | o . . | 2024-09-21T13:43:35.460 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:35 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: |= = + o | 2024-09-21T13:43:35.460 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:35 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: |+* * + o | 2024-09-21T13:43:35.460 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:35 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: |ooX . . S | 2024-09-21T13:43:35.460 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:35 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: |+* = | 2024-09-21T13:43:35.460 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:35 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: |Bo=.o | 2024-09-21T13:43:35.460 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:35 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: |XOoB | 2024-09-21T13:43:35.460 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:35 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: |BE%o | 2024-09-21T13:43:35.460 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:35 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: +----[SHA256]-----+ 2024-09-21T13:43:35.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:35 smithi049 ceph-mon[30080]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:43:35.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:35 smithi049 ceph-mon[30080]: Generating ssh key... 2024-09-21T13:43:35.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:35 smithi049 ceph-mon[30080]: from='mgr.14118 172.21.15.49:0/2880154351' entity='mgr.y' 2024-09-21T13:43:35.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:35 smithi049 ceph-mon[30080]: from='mgr.14118 172.21.15.49:0/2880154351' entity='mgr.y' 2024-09-21T13:43:37.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:36 smithi049 ceph-mon[30080]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:43:37.017 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCqODoVC5IghBUlo5SlJDI5bIn2ngTCcf/e9DQHDy4uNOPii0ZdbUPqEBAlfWIljU9kDlW9HzOyBaIRldJp61V3k8P74f9WxD+WOWrj4aCpr+xZtpDccdjj3rJ4u9p5ivOmhTj8Np8j7tx9YOxmvQiJrkN9aaFRoYSkFeZwcBq8r3PCZCOifhuLjbER3l+2pxGopr7iN3c65sI/wDGqYVDC2Pz/BEccun5zEwlYr1iYzDfreeLjtprH0A7Z7vSxIHQ8Uc4oR1lhE5hMf2iL3EILKROlwfiG8lSC7CvjhEtrUUkFfT8tu11yrRiQ55h5xBlxMHffj1UuXmYkxGPJg4mjKdvT64LOM0uSVmjgr5JyKyAj5MpTQUZ3ut8YrUkLeA+iK1VEu5DWYR/bY++Vblemad6IkOdIAxqstlZUeaJ9TRmxvHniJNyZfCSjXxq3FB1LYfX23tmflkuY9Bedx+YP+jqkPpbqej39PGjhJcFanptunCfGhWvQnO/1jfTOjts= ceph-40a3d2e8-781f-11ef-baf6-efdc52797490 2024-09-21T13:43:37.018 INFO:teuthology.orchestra.run.smithi049.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-21T13:43:37.018 INFO:teuthology.orchestra.run.smithi049.stdout:Adding key to root@localhost authorized_keys... 2024-09-21T13:43:37.029 INFO:teuthology.orchestra.run.smithi049.stdout:Adding host smithi049... 2024-09-21T13:43:38.146 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:37 smithi049 ceph-mon[30080]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi049", "addr": "172.21.15.49", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:43:39.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:38 smithi049 ceph-mon[30080]: Deploying cephadm binary to smithi049 2024-09-21T13:43:42.237 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout Added host 'smithi049' with addr '172.21.15.49' 2024-09-21T13:43:42.237 INFO:teuthology.orchestra.run.smithi049.stdout:Deploying unmanaged mon service... 2024-09-21T13:43:42.842 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:42 smithi049 ceph-mon[30080]: from='mgr.14118 172.21.15.49:0/2880154351' entity='mgr.y' 2024-09-21T13:43:42.842 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:42 smithi049 ceph-mon[30080]: Added host smithi049 2024-09-21T13:43:42.842 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:42 smithi049 ceph-mon[30080]: from='mgr.14118 172.21.15.49:0/2880154351' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:43:44.321 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:44 smithi049 ceph-mon[30080]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:43:44.322 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:44 smithi049 ceph-mon[30080]: Saving service mon spec with placement count:5 2024-09-21T13:43:44.322 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:44 smithi049 ceph-mon[30080]: from='mgr.14118 172.21.15.49:0/2880154351' entity='mgr.y' 2024-09-21T13:43:44.322 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:44 smithi049 ceph-mon[30080]: from='mgr.14118 172.21.15.49:0/2880154351' entity='mgr.y' 2024-09-21T13:43:44.441 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-09-21T13:43:44.441 INFO:teuthology.orchestra.run.smithi049.stdout:Deploying unmanaged mgr service... 2024-09-21T13:43:45.923 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:45 smithi049 ceph-mon[30080]: from='mgr.14118 172.21.15.49:0/2880154351' entity='mgr.y' 2024-09-21T13:43:45.923 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:45 smithi049 ceph-mon[30080]: from='mgr.14118 172.21.15.49:0/2880154351' entity='mgr.y' 2024-09-21T13:43:45.925 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-09-21T13:43:46.762 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:46 smithi049 ceph-mon[30080]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:43:46.762 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:46 smithi049 ceph-mon[30080]: Saving service mgr spec with placement count:2 2024-09-21T13:43:46.762 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:46 smithi049 ceph-mon[30080]: from='mgr.14118 172.21.15.49:0/2880154351' entity='mgr.y' 2024-09-21T13:43:48.118 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:48 smithi049 ceph-mon[30080]: from='mgr.14118 172.21.15.49:0/2880154351' entity='mgr.y' 2024-09-21T13:43:48.118 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:48 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3187541670' entity='client.admin' 2024-09-21T13:43:49.670 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:49 smithi049 ceph-mon[30080]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:43:49.670 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:49 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1665895386' entity='client.admin' 2024-09-21T13:43:49.673 INFO:teuthology.orchestra.run.smithi049.stdout:Enabling the dashboard module... 2024-09-21T13:43:51.643 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:51 smithi049 ceph-mon[30080]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:43:51.643 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:51 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1760707215' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "dashboard"} : dispatch 2024-09-21T13:43:51.643 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:51 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ignoring --setuser ceph since I am not root 2024-09-21T13:43:51.643 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:51 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ignoring --setgroup ceph since I am not root 2024-09-21T13:43:52.684 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:52 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1760707215' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-21T13:43:52.685 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:52 smithi049 ceph-mon[30080]: mgrmap e10: y(active, since 23s) 2024-09-21T13:43:53.295 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout { 2024-09-21T13:43:53.295 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 10, 2024-09-21T13:43:53.295 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-21T13:43:53.295 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-21T13:43:53.295 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-21T13:43:53.295 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout } 2024-09-21T13:43:53.295 INFO:teuthology.orchestra.run.smithi049.stdout:Waiting for the mgr to restart... 2024-09-21T13:43:53.295 INFO:teuthology.orchestra.run.smithi049.stdout:Waiting for mgr epoch 10... 2024-09-21T13:43:53.542 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:53 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/367008127' entity='client.admin' cmd={"prefix": "mgr stat"} : dispatch 2024-09-21T13:43:57.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:56 smithi049 ceph-mon[30080]: Active manager daemon y restarted 2024-09-21T13:43:57.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:56 smithi049 ceph-mon[30080]: Activating manager daemon y 2024-09-21T13:43:57.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:56 smithi049 ceph-mon[30080]: osdmap e3: 0 total, 0 up, 0 in 2024-09-21T13:43:57.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:56 smithi049 ceph-mon[30080]: mgrmap e11: y(active, starting, since 0.00369844s) 2024-09-21T13:43:57.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:56 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T13:43:57.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:56 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-21T13:43:57.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:56 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-21T13:43:57.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:56 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-21T13:43:57.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:56 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-21T13:43:57.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:56 smithi049 ceph-mon[30080]: Manager daemon y is now available 2024-09-21T13:43:57.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:57 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:43:57.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:57 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-21T13:43:57.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:57 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-21T13:43:57.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:57 smithi049 ceph-mon[30080]: mgrmap e12: y(active, since 1.00764s) 2024-09-21T13:43:58.210 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout { 2024-09-21T13:43:58.210 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 12, 2024-09-21T13:43:58.210 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-21T13:43:58.210 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout } 2024-09-21T13:43:58.211 INFO:teuthology.orchestra.run.smithi049.stdout:mgr epoch 10 is available 2024-09-21T13:43:58.211 INFO:teuthology.orchestra.run.smithi049.stdout:Using certmgr to generate dashboard self-signed certificate... 2024-09-21T13:43:58.945 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:58 smithi049 ceph-mon[30080]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-21T13:43:58.945 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:58 smithi049 ceph-mon[30080]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-21T13:43:59.689 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:59 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: 2024-09-21T13:43:59.364+0000 7f0d6409b640 -1 log_channel(cephadm) log [ERR] : [21/Sep/2024:13:43:59] ENGINE Error in HTTPServer.serve 2024-09-21T13:43:59.690 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:59 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: Traceback (most recent call last): 2024-09-21T13:43:59.690 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:59 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T13:43:59.690 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:59 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: self._connections.run(self.expiration_interval) 2024-09-21T13:43:59.690 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:59 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T13:43:59.690 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:59 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: self._run(expiration_interval) 2024-09-21T13:43:59.690 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:59 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T13:43:59.690 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:59 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T13:43:59.690 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:59 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T13:43:59.690 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:59 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T13:43:59.690 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:59 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T13:43:59.690 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:59 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: s = self.context.wrap_socket( 2024-09-21T13:43:59.690 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:59 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T13:43:59.690 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:59 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: return self.sslsocket_class._create( 2024-09-21T13:43:59.691 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:59 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T13:43:59.691 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:59 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: self.do_handshake() 2024-09-21T13:43:59.691 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:59 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T13:43:59.691 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:59 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: self._sslobj.do_handshake() 2024-09-21T13:43:59.691 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:59 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T13:43:59.691 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:43:59 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: 2024-09-21T13:44:00.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:59 smithi049 ceph-mon[30080]: mgrmap e13: y(active, since 2s) 2024-09-21T13:44:00.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:59 smithi049 ceph-mon[30080]: [21/Sep/2024:13:43:59] ENGINE Bus STARTING 2024-09-21T13:44:00.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:59 smithi049 ceph-mon[30080]: [21/Sep/2024:13:43:59] ENGINE Serving on https://172.21.15.49:7150 2024-09-21T13:44:00.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:59 smithi049 ceph-mon[30080]: [21/Sep/2024:13:43:59] ENGINE Error in HTTPServer.serve 2024-09-21T13:44:00.005 INFO:journalctl@ceph.mon.a.smithi049.stdout: Traceback (most recent call last): 2024-09-21T13:44:00.005 INFO:journalctl@ceph.mon.a.smithi049.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T13:44:00.005 INFO:journalctl@ceph.mon.a.smithi049.stdout: self._connections.run(self.expiration_interval) 2024-09-21T13:44:00.005 INFO:journalctl@ceph.mon.a.smithi049.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T13:44:00.005 INFO:journalctl@ceph.mon.a.smithi049.stdout: self._run(expiration_interval) 2024-09-21T13:44:00.006 INFO:journalctl@ceph.mon.a.smithi049.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T13:44:00.006 INFO:journalctl@ceph.mon.a.smithi049.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T13:44:00.006 INFO:journalctl@ceph.mon.a.smithi049.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T13:44:00.006 INFO:journalctl@ceph.mon.a.smithi049.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T13:44:00.006 INFO:journalctl@ceph.mon.a.smithi049.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T13:44:00.006 INFO:journalctl@ceph.mon.a.smithi049.stdout: s = self.context.wrap_socket( 2024-09-21T13:44:00.006 INFO:journalctl@ceph.mon.a.smithi049.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T13:44:00.006 INFO:journalctl@ceph.mon.a.smithi049.stdout: return self.sslsocket_class._create( 2024-09-21T13:44:00.006 INFO:journalctl@ceph.mon.a.smithi049.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T13:44:00.006 INFO:journalctl@ceph.mon.a.smithi049.stdout: self.do_handshake() 2024-09-21T13:44:00.006 INFO:journalctl@ceph.mon.a.smithi049.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T13:44:00.006 INFO:journalctl@ceph.mon.a.smithi049.stdout: self._sslobj.do_handshake() 2024-09-21T13:44:00.006 INFO:journalctl@ceph.mon.a.smithi049.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T13:44:00.007 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:43:59 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:01.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:00 smithi049 ceph-mon[30080]: [21/Sep/2024:13:43:59] ENGINE Serving on http://172.21.15.49:8765 2024-09-21T13:44:01.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:00 smithi049 ceph-mon[30080]: [21/Sep/2024:13:43:59] ENGINE Bus STARTED 2024-09-21T13:44:01.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:00 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:01.901 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout {"cert": "-----BEGIN CERTIFICATE-----\nMIIE+zCCAuOgAwIBAgIUT9+6ijGxPQ2ldc0z7hPs1xYwRHkwDQYJKoZIhvcNAQEL\nBQAwFzEVMBMGA1UEAwwMY2VwaGFkbS1yb290MB4XDTI0MDkyMTEzNDQwMFoXDTM0\nMDkyMjEzNDQwMFowFzEVMBMGA1UEAwwMMTcyLjIxLjE1LjQ5MIICIjANBgkqhkiG\n9w0BAQEFAAOCAg8AMIICCgKCAgEAoEj/9A6n/b6rGk80lz3ECkpwipNCTRC3VlYP\nqP4x+a2dSqUn0EKwazZMcUEtBbUaLeXoVTN7M+7qtbZ+rUJujWrbf1jq8p38yXBx\nFmxOkFMA/+vlw6H6biL14KdUGXt2FMaxpsBhUv93rFbbyQdR3NbFtjFQzR5fl62k\nQfwne+n7Vj9Abdx9tLFCj4yt4R+o8JUeiFJaJjgKe8KjGwYs2t9fJejFSv+0Y9Yx\nYe66S8jwqSWDbtN2lgk1ScOyKDPxJXbSOntBRO7b16mIR2TYCFfkhUkS2oxWkyxn\nhNbpTH8wXeHx0yiI70XzsyiaIcNDNitv+bZsXkCkxdgXFK3u33T84xnOGe0TfBB+\nLYYr/VlZS8kkRu7JBZQD5I742U12I8i707d/tZ0LhyB6Msksa/3g1+0Qvw2apzvC\nhBLXkBwUUNOyxrlYZDAfHat+PdO4Y0MHiY+RuzevUrd8MsxoYGJVKJYvPGg12f7k\nANAcVMjWOU3YHLX4gYPVolIbx0hn/cnD2cIT4FgrZ4KEfGykCMACTa23oaYe6yQN\nv1qvGqWPpA0fuv3aR+7r+qtJDxwt8krn2tPq2LwH5tnO3nYFf1Z4RtSRdFys/Myg\nrqBgcVAL/SkH2N0LX5Nd29oiRzSOK/bdLUhLA6EuXWsR8aBDwzgAZ/kHOhKiTAiB\nagk67aUCAwEAAaM/MD0wLQYDVR0RBCYwJIIJc21pdGhpMDQ5ghFkYXNoYm9hcmRf\nc2VydmVyc4cErBUPMTAMBgNVHRMBAf8EAjAAMA0GCSqGSIb3DQEBCwUAA4ICAQBa\nThi6rPuU3DWzPn0kog4EhYWDRgEqxKaKfqfqlWEo1SBaRGgBA+kKdjhto81974dW\n4/8JVsrBzuQ8+GQLmXqVCu9+cZdZ1hWqhUmj7lw5VcW6AAlT0gq4hR25vdvfrrJa\nivy9gqcpG6wvPWAeuSr39PeGsetnkjK0Qnfqzi/To2ZzoG1YYiPuJtGRV8Xm65lj\nj4me/7czW852qC+jAQR0sU5Alt7BaowawUNB9EKH1fXS8vJRZJRR0dTsIHMjXXXr\nTXmIrzVGYhevll4DqtSJn5RDneBEygHfkZ95hhqaNNIPwltfzTY3DaTTGenLtJPo\nb+Nu5lMpRdchCCSmnqer6gLNo8g5oclQ//IFa6GZEWlONNpn4j+jdsU4MYsE+3Ek\nzBrE5ERiBAkBJCctd9Jox6TZxbB0hZn5yfoTn4+lYqlHpxQnlkllhlAM7J02Eeu9\nTLX5hOIzP4WSjID9lmmqNdqyjlmU+b+GyGT2GgByb+Q8Ari961R1mG63++PDeWtY\nCi8kE+vTeGpVOWwFd7hDJc7s0BAvq79EdxPpcgYyJCL/iwOiWAQg6ls7GxnPsolP\nRQw0CiEvgBF67ypczyF0GTaHbPKn5VigpElUSqSZDJsof0pIEzk3zWrZGFoZo9lM\npQshyRUfJBmyyITIHjvJhbruudYaSzq89cgy6z6BCA==\n-----END CERTIFICATE-----\n", "key": "-----BEGIN RSA PRIVATE KEY-----\nMIIJKAIBAAKCAgEAoEj/9A6n/b6rGk80lz3ECkpwipNCTRC3VlYPqP4x+a2dSqUn\n0EKwazZMcUEtBbUaLeXoVTN7M+7qtbZ+rUJujWrbf1jq8p38yXBxFmxOkFMA/+vl\nw6H6biL14KdUGXt2FMaxpsBhUv93rFbbyQdR3NbFtjFQzR5fl62kQfwne+n7Vj9A\nbdx9tLFCj4yt4R+o8JUeiFJaJjgKe8KjGwYs2t9fJejFSv+0Y9YxYe66S8jwqSWD\nbtN2lgk1ScOyKDPxJXbSOntBRO7b16mIR2TYCFfkhUkS2oxWkyxnhNbpTH8wXeHx\n0yiI70XzsyiaIcNDNitv+bZsXkCkxdgXFK3u33T84xnOGe0TfBB+LYYr/VlZS8kk\nRu7JBZQD5I742U12I8i707d/tZ0LhyB6Msksa/3g1+0Qvw2apzvChBLXkBwUUNOy\nxrlYZDAfHat+PdO4Y0MHiY+RuzevUrd8MsxoYGJVKJYvPGg12f7kANAcVMjWOU3Y\nHLX4gYPVolIbx0hn/cnD2cIT4FgrZ4KEfGykCMACTa23oaYe6yQNv1qvGqWPpA0f\nuv3aR+7r+qtJDxwt8krn2tPq2LwH5tnO3nYFf1Z4RtSRdFys/MygrqBgcVAL/SkH\n2N0LX5Nd29oiRzSOK/bdLUhLA6EuXWsR8aBDwzgAZ/kHOhKiTAiBagk67aUCAwEA\nAQKCAgAFADb9EQGwPTjJzMYjx9vnJsDgXuD5fZBLaI/ASetB8ASRf5neDXESClWK\nJuWdb+7gNJGeeQ44Q2QPDz8lPgaOcYPvlZALolTwsk8OLbJa4xaLtFO7gY/UBypc\n8hzGua33CebmClR3ouESZrX5JjAzPpNsDehhH8vl9BdO6WvVLXlD+S/FTvKI8KiR\njerj+Vbc1ENzkXVGM0vzNMhd5ud1l/Tpb5vfStCKgGp/oQ6k2uX77fOBpC+/pD0P\nWP+HQJTZ2x1PzC2DpJHehBkAM2SIG/RfBecOVxO6G9usqdhYGsTcDBRHPS0tKuCE\nL7vXqSLUqPNI2KZ61ZFCNAtpiaTIWtcGvhIOOuXOZXUdZ1qm2ULqd/jTw4rG8LTB\naABgVMPYYcXa8KRC3Vv7SLb88D1HJpBaypKLz5x2WdIiUP6sUIHnowl+G1F0Hyke\nt2PMHohwZ2+wn+KpH42SEWG+sZmI6wAExYSu7q1sRdzMr7F4twU0ZST3zNKoVljD\noinXjcimWyhfyBQ9/fJjzocQV44cYn0AetllE5jG/SFcmKs+8+b+JVJTe1bBzuef\nRFsJ6p2wpbJ7/g19HOkn+TSXTTGYjO0TzUV1zTm6rDQ4jPTBsPbQBLfMNHRY9jyA\nP2hNdBLG0seiB6KIZbQVL33HNoBgdHAI3sETgzzw/C/nnySIawKCAQEAzSaDBelG\nYjUhJWfHDyfsFP3Bj/Gk7H/39HJR5pULEVT5U7McQwyYKPVMxSzCEqQ+wpE+4BCl\neZQZCKq5yD5mkBnq/4T4qfc4tc9XhIlEHxZkE3KJ2G6wwoatVGwOZWdYxLWr1Y/N\nmRbnQaBz6MqKfKapy6niwOxVLeh63JAnHzTfMp3CuQUeOu6akB4QzD5mlwoya6r4\nWkgpqrOUEvcp37/Y4IbdauJpvQMalSibKYvWJ1CoYSrsN3u5nScgfU15vagFQwRo\n8p1d1yL4Gby7ApMlwCTjd7ifsLlmZfO+N7aSql2GoXhnI/Srt9xrrs01pt98b77Q\nlGUKiJE3u+MfGwKCAQEAyAOiZb33lPAT1F39FY9jtl34x4DbqRqu7lVXrjg5KrC0\nGrw6AA0cLnsCpaMcW41N7IB0HIunkaHBTHPd/UsjdPFZ0lycliPZQ9Q8hwRoVgq0\nmz84QMDr7ieUxmnPPhj/nO2KHEjLwPF6XMiS7u/aIusdoRafoF1iPkegwgusTdQv\noJPObh1zYhfL3TbTvA7AlLBe9iy9tHJ9JU8R88XChskDK+IWtkTN2b51kDsHVP0M\nkhFdEvvaySIOMLR8zZc3wsN2RwKBsf59rbi2fUTSMc6k4MxOm46euI+HQhTC6Hub\n73YbTexHLeFK+VIbFrGq0I7ennQCxUgRPQclffkyPwKCAQEAw4ZXDkU8/dDhXINC\nZ6GHEqKf3w9SPs3ghyyF/pxCxiwUzb9Bym2wgCj7GFoGRjogTtSb9nYnIG8URPxi\nwQUoP2NUZGBZl7S2aiHJqdoiAPcYPqjF1G/7LAusG3A5cFBFfIYh1hOOwr/ffN7u\nWgrgXtmBr0ilg3Y74NtYbes+PjHoIJfXxPupykUrLvxYyjEXGhFv6Mo0led9PuKU\n2FaVpFl23NftJDfSy4Cvz0W6TKioXSQ0txdvt5nGWI+4GXTD+QywpyUG2WU+0RLV\nH8KPAm84ZZ9C5RIjsQ2jGLxsqTDrLFCjRjY5Jz3EhCIix6Fx5y/Nq7tTlHuQxIST\neBGJzQKCAQBEDUiByWKlYBG/HcNwRHsun0Z3ypXyMUv3dz3/3s1Egx6kQgqR3eQK\nxHugE6EgXTF4GDRpSfLXj+uOsZWECr6jmUJwWmZEhyny9BEr9wnwzG+qqpXfIkSj\noP/ziiii/ipqID/vY60/lmmFjBDCDwVvAAPyvRnOnDdC7Wf63bHgQSN7RQ7lVEvf\nwizwtSjRQUuwy1lj9NEOxi1iNkbrZAvNhDL6CVCk70ACp9SkIBFcj0RpplK+lMZZ\nzKZ7V81mvd6Q+6QQgP/TqbzlEIW2lWXW4HIksh7Eun+yV9MzXpD6PKRPkyWzG4MU\nTyUNMyxYSlbN4u5Hhz0yFlUXKIlPFSxNAoIBABarcl9Fl1FSSSR3C6nN5gqLdd7o\n4M/+pn3Tw7ZSfWUUqXRx5F+gTn5F5iSSlGB19ES/HV5RcV+RfQ2Lhn6mkBd7IK1q\nJcRDOxVjCWnkNNC0voJIXxGbfKZiaulEPj/bOm8HaKVmkJ1BiipbVbP1owj+U9x6\nxEAfBUB8mmTgzMTU9cPTcmQjGtn8x7oyjpNJ3gEjqASvRieoh1IlFCSCLnEIiD09\nbmmK1aI8ig9yheXnuTwHEt57My9l51AW48hgBBqJACf11vhFaoQLbbLNmJyVqLh5\nhfjaETNdGGwSbgD8GTKzipIWiUuYILY0VhGsCtYFnhIzXqmE4+yNVer3Ms4=\n-----END RSA PRIVATE KEY-----\n"} 2024-09-21T13:44:03.308 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout SSL certificate updated 2024-09-21T13:44:04.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:03 smithi049 ceph-mon[30080]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard set-ssl-certificate", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:44:04.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:03 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:04.597 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout SSL certificate key updated 2024-09-21T13:44:04.597 INFO:teuthology.orchestra.run.smithi049.stdout:Creating initial admin user... 2024-09-21T13:44:05.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:05 smithi049 ceph-mon[30080]: from='client.14168 -' entity='client.admin' cmd=[{"prefix": "dashboard set-ssl-certificate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:44:05.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:05 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:06.143 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:06 smithi049 ceph-mon[30080]: from='client.14170 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:44:06.143 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:06 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:06.146 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$aJucqB1DvJkobCUzT7upaO06QlWz3/4aRCTrhCWOKGV7znMTFqZ3O", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1726926245, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-21T13:44:06.146 INFO:teuthology.orchestra.run.smithi049.stdout:Fetching dashboard port number... 2024-09-21T13:44:07.091 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:07 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1973906123' entity='client.admin' cmd={"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} : dispatch 2024-09-21T13:44:07.368 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 8443 2024-09-21T13:44:07.380 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-21T13:44:07.380 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout disabled 2024-09-21T13:44:07.388 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-21T13:44:07.388 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout inactive 2024-09-21T13:44:07.388 INFO:teuthology.orchestra.run.smithi049.stdout:firewalld.service is not enabled 2024-09-21T13:44:07.388 INFO:teuthology.orchestra.run.smithi049.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-21T13:44:07.390 INFO:teuthology.orchestra.run.smithi049.stdout:Ceph Dashboard is now available at: 2024-09-21T13:44:07.390 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:44:07.390 INFO:teuthology.orchestra.run.smithi049.stdout: URL: https://smithi049.front.sepia.ceph.com:8443/ 2024-09-21T13:44:07.390 INFO:teuthology.orchestra.run.smithi049.stdout: User: admin 2024-09-21T13:44:07.390 INFO:teuthology.orchestra.run.smithi049.stdout: Password: h9rgxic6hl 2024-09-21T13:44:07.390 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:44:07.391 INFO:teuthology.orchestra.run.smithi049.stdout:Saving cluster configuration to /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/config directory 2024-09-21T13:44:08.759 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-09-21T13:44:08.760 INFO:teuthology.orchestra.run.smithi049.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-09-21T13:44:08.760 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:44:08.760 INFO:teuthology.orchestra.run.smithi049.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-21T13:44:08.760 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:44:08.760 INFO:teuthology.orchestra.run.smithi049.stdout:Or, if you are only running a single cluster on this host: 2024-09-21T13:44:08.760 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:44:08.760 INFO:teuthology.orchestra.run.smithi049.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-09-21T13:44:08.760 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:44:08.760 INFO:teuthology.orchestra.run.smithi049.stdout:Please consider enabling telemetry to help improve Ceph: 2024-09-21T13:44:08.760 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:44:08.760 INFO:teuthology.orchestra.run.smithi049.stdout: ceph telemetry on 2024-09-21T13:44:08.760 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:44:08.760 INFO:teuthology.orchestra.run.smithi049.stdout:For more information see: 2024-09-21T13:44:08.760 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:44:08.761 INFO:teuthology.orchestra.run.smithi049.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-09-21T13:44:08.761 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:44:08.761 INFO:teuthology.orchestra.run.smithi049.stdout:Bootstrap complete. 2024-09-21T13:44:08.800 INFO:tasks.cephadm:Fetching config... 2024-09-21T13:44:08.801 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-09-21T13:44:08.801 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-21T13:44:08.817 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-21T13:44:08.817 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-09-21T13:44:08.817 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-21T13:44:08.873 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-21T13:44:08.873 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-09-21T13:44:08.873 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/keyring of=/dev/stdout 2024-09-21T13:44:08.938 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-21T13:44:08.939 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-09-21T13:44:08.939 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-21T13:44:08.994 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-21T13:44:08.995 DEBUG:teuthology.orchestra.run.smithi049:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCqODoVC5IghBUlo5SlJDI5bIn2ngTCcf/e9DQHDy4uNOPii0ZdbUPqEBAlfWIljU9kDlW9HzOyBaIRldJp61V3k8P74f9WxD+WOWrj4aCpr+xZtpDccdjj3rJ4u9p5ivOmhTj8Np8j7tx9YOxmvQiJrkN9aaFRoYSkFeZwcBq8r3PCZCOifhuLjbER3l+2pxGopr7iN3c65sI/wDGqYVDC2Pz/BEccun5zEwlYr1iYzDfreeLjtprH0A7Z7vSxIHQ8Uc4oR1lhE5hMf2iL3EILKROlwfiG8lSC7CvjhEtrUUkFfT8tu11yrRiQ55h5xBlxMHffj1UuXmYkxGPJg4mjKdvT64LOM0uSVmjgr5JyKyAj5MpTQUZ3ut8YrUkLeA+iK1VEu5DWYR/bY++Vblemad6IkOdIAxqstlZUeaJ9TRmxvHniJNyZfCSjXxq3FB1LYfX23tmflkuY9Bedx+YP+jqkPpbqej39PGjhJcFanptunCfGhWvQnO/1jfTOjts= ceph-40a3d2e8-781f-11ef-baf6-efdc52797490' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-21T13:44:09.069 INFO:teuthology.orchestra.run.smithi049.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCqODoVC5IghBUlo5SlJDI5bIn2ngTCcf/e9DQHDy4uNOPii0ZdbUPqEBAlfWIljU9kDlW9HzOyBaIRldJp61V3k8P74f9WxD+WOWrj4aCpr+xZtpDccdjj3rJ4u9p5ivOmhTj8Np8j7tx9YOxmvQiJrkN9aaFRoYSkFeZwcBq8r3PCZCOifhuLjbER3l+2pxGopr7iN3c65sI/wDGqYVDC2Pz/BEccun5zEwlYr1iYzDfreeLjtprH0A7Z7vSxIHQ8Uc4oR1lhE5hMf2iL3EILKROlwfiG8lSC7CvjhEtrUUkFfT8tu11yrRiQ55h5xBlxMHffj1UuXmYkxGPJg4mjKdvT64LOM0uSVmjgr5JyKyAj5MpTQUZ3ut8YrUkLeA+iK1VEu5DWYR/bY++Vblemad6IkOdIAxqstlZUeaJ9TRmxvHniJNyZfCSjXxq3FB1LYfX23tmflkuY9Bedx+YP+jqkPpbqej39PGjhJcFanptunCfGhWvQnO/1jfTOjts= ceph-40a3d2e8-781f-11ef-baf6-efdc52797490 2024-09-21T13:44:09.083 DEBUG:teuthology.orchestra.run.smithi187:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCqODoVC5IghBUlo5SlJDI5bIn2ngTCcf/e9DQHDy4uNOPii0ZdbUPqEBAlfWIljU9kDlW9HzOyBaIRldJp61V3k8P74f9WxD+WOWrj4aCpr+xZtpDccdjj3rJ4u9p5ivOmhTj8Np8j7tx9YOxmvQiJrkN9aaFRoYSkFeZwcBq8r3PCZCOifhuLjbER3l+2pxGopr7iN3c65sI/wDGqYVDC2Pz/BEccun5zEwlYr1iYzDfreeLjtprH0A7Z7vSxIHQ8Uc4oR1lhE5hMf2iL3EILKROlwfiG8lSC7CvjhEtrUUkFfT8tu11yrRiQ55h5xBlxMHffj1UuXmYkxGPJg4mjKdvT64LOM0uSVmjgr5JyKyAj5MpTQUZ3ut8YrUkLeA+iK1VEu5DWYR/bY++Vblemad6IkOdIAxqstlZUeaJ9TRmxvHniJNyZfCSjXxq3FB1LYfX23tmflkuY9Bedx+YP+jqkPpbqej39PGjhJcFanptunCfGhWvQnO/1jfTOjts= ceph-40a3d2e8-781f-11ef-baf6-efdc52797490' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-21T13:44:09.159 INFO:teuthology.orchestra.run.smithi187.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCqODoVC5IghBUlo5SlJDI5bIn2ngTCcf/e9DQHDy4uNOPii0ZdbUPqEBAlfWIljU9kDlW9HzOyBaIRldJp61V3k8P74f9WxD+WOWrj4aCpr+xZtpDccdjj3rJ4u9p5ivOmhTj8Np8j7tx9YOxmvQiJrkN9aaFRoYSkFeZwcBq8r3PCZCOifhuLjbER3l+2pxGopr7iN3c65sI/wDGqYVDC2Pz/BEccun5zEwlYr1iYzDfreeLjtprH0A7Z7vSxIHQ8Uc4oR1lhE5hMf2iL3EILKROlwfiG8lSC7CvjhEtrUUkFfT8tu11yrRiQ55h5xBlxMHffj1UuXmYkxGPJg4mjKdvT64LOM0uSVmjgr5JyKyAj5MpTQUZ3ut8YrUkLeA+iK1VEu5DWYR/bY++Vblemad6IkOdIAxqstlZUeaJ9TRmxvHniJNyZfCSjXxq3FB1LYfX23tmflkuY9Bedx+YP+jqkPpbqej39PGjhJcFanptunCfGhWvQnO/1jfTOjts= ceph-40a3d2e8-781f-11ef-baf6-efdc52797490 2024-09-21T13:44:09.172 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-21T13:44:09.352 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:09 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/482804805' entity='client.admin' 2024-09-21T13:44:09.426 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:44:10.443 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:10 smithi049 ceph-mon[30080]: mgrmap e14: y(active, since 12s) 2024-09-21T13:44:10.443 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:10 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3222140621' entity='client.admin' 2024-09-21T13:44:10.767 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-21T13:44:10.767 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-21T13:44:11.264 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:44:12.327 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:12 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:12.327 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:12 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:12.327 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:12 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi049", "name": "osd_memory_target"} : dispatch 2024-09-21T13:44:12.327 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:12 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:12.327 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:12 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:44:12.327 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:12 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:13.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:13 smithi049 ceph-mon[30080]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:44:13.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:13 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:13.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:13 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:44:13.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:13 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:44:13.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:13 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:44:13.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:13 smithi049 ceph-mon[30080]: Updating smithi049:/etc/ceph/ceph.conf 2024-09-21T13:44:13.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:13 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:13.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:13 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:13.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:13 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:13.641 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi187 2024-09-21T13:44:13.641 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-09-21T13:44:13.641 DEBUG:teuthology.orchestra.run.smithi187:> dd of=/etc/ceph/ceph.conf 2024-09-21T13:44:13.661 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-09-21T13:44:13.661 DEBUG:teuthology.orchestra.run.smithi187:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-21T13:44:13.719 INFO:tasks.cephadm:Adding host smithi187 to orchestrator... 2024-09-21T13:44:13.719 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph orch host add smithi187 2024-09-21T13:44:13.972 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:44:14.408 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:14 smithi049 ceph-mon[30080]: Updating smithi049:/var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T13:44:14.408 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:14 smithi049 ceph-mon[30080]: Updating smithi049:/etc/ceph/ceph.client.admin.keyring 2024-09-21T13:44:14.408 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:14 smithi049 ceph-mon[30080]: Updating smithi049:/var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T13:44:15.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:15 smithi049 ceph-mon[30080]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi187", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:44:16.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:16 smithi049 ceph-mon[30080]: Deploying cephadm binary to smithi187 2024-09-21T13:44:17.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:17 smithi049 ceph-mon[30080]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:18.393 INFO:teuthology.orchestra.run.smithi049.stdout:Added host 'smithi187' with addr '172.21.15.187' 2024-09-21T13:44:19.013 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph orch host ls --format=json 2024-09-21T13:44:19.267 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:44:19.656 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:19 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:19.656 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:19 smithi049 ceph-mon[30080]: Added host smithi187 2024-09-21T13:44:19.656 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:19 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:44:19.656 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:19 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:19.656 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:19 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:19.944 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:44:19.944 INFO:teuthology.orchestra.run.smithi049.stdout:[{"addr": "172.21.15.49", "hostname": "smithi049", "labels": [], "status": ""}, {"addr": "172.21.15.187", "hostname": "smithi187", "labels": [], "status": ""}] 2024-09-21T13:44:20.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:20 smithi049 ceph-mon[30080]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:20.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:20 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:20.566 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-21T13:44:20.566 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd crush tunables default 2024-09-21T13:44:20.817 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:44:21.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:21 smithi049 ceph-mon[30080]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T13:44:22.395 INFO:teuthology.orchestra.run.smithi049.stderr:adjusted tunables profile to default 2024-09-21T13:44:22.733 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:22 smithi049 ceph-mon[30080]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:22.733 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:22 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1368541128' entity='client.admin' cmd={"prefix": "osd crush tunables", "profile": "default"} : dispatch 2024-09-21T13:44:23.119 INFO:tasks.cephadm:Adding mon.a on smithi049 2024-09-21T13:44:23.119 INFO:tasks.cephadm:Adding mon.c on smithi049 2024-09-21T13:44:23.119 INFO:tasks.cephadm:Adding mon.b on smithi187 2024-09-21T13:44:23.119 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph orch apply mon '3;smithi049:172.21.15.49=a;smithi049:[v2:172.21.15.49:3301,v1:172.21.15.49:6790]=c;smithi187:172.21.15.187=b' 2024-09-21T13:44:23.352 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T13:44:23.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:23 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1368541128' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-21T13:44:23.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:23 smithi049 ceph-mon[30080]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T13:44:23.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:23 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:23.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:23 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:23.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:23 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:23.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:23 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:23.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:23 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi187", "name": "osd_memory_target"} : dispatch 2024-09-21T13:44:23.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:23 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:44:23.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:23 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:44:23.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:23 smithi049 ceph-mon[30080]: Updating smithi187:/etc/ceph/ceph.conf 2024-09-21T13:44:23.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:23 smithi049 ceph-mon[30080]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:23.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:23 smithi049 ceph-mon[30080]: Updating smithi187:/var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T13:44:23.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:23 smithi049 ceph-mon[30080]: Updating smithi187:/etc/ceph/ceph.client.admin.keyring 2024-09-21T13:44:23.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:23 smithi049 ceph-mon[30080]: Updating smithi187:/var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T13:44:23.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:23 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:23.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:23 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:23.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:23 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:23.939 INFO:teuthology.orchestra.run.smithi187.stdout:Scheduled mon update... 2024-09-21T13:44:24.505 DEBUG:teuthology.orchestra.run.smithi049:mon.c> sudo journalctl -f -n 0 -u ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@mon.c.service 2024-09-21T13:44:24.508 DEBUG:teuthology.orchestra.run.smithi187:mon.b> sudo journalctl -f -n 0 -u ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@mon.b.service 2024-09-21T13:44:24.511 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-21T13:44:24.511 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph mon dump -f json 2024-09-21T13:44:25.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:24 smithi049 ceph-mon[30080]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi049:172.21.15.49=a;smithi049:[v2:172.21.15.49:3301,v1:172.21.15.49:6790]=c;smithi187:172.21.15.187=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:44:25.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:24 smithi049 ceph-mon[30080]: Saving service mon spec with placement smithi049:172.21.15.49=a;smithi049:[v2:172.21.15.49:3301,v1:172.21.15.49:6790]=c;smithi187:172.21.15.187=b;count:3 2024-09-21T13:44:25.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:24 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:25.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:24 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:44:25.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:24 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:44:25.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:24 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:44:25.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:24 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:25.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:24 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T13:44:25.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:24 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:44:25.257 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:24 smithi049 ceph-mon[30080]: Deploying daemon mon.b on smithi187 2024-09-21T13:44:25.486 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T13:44:26.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:25 smithi049 ceph-mon[30080]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:26.385 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:44:26.386 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":1,"fsid":"40a3d2e8-781f-11ef-baf6-efdc52797490","modified":"2024-09-21T13:42:51.046146Z","created":"2024-09-21T13:42:51.046146Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:3300","nonce":0},{"type":"v1","addr":"172.21.15.49:6789","nonce":0}]},"addr":"172.21.15.49:6789/0","public_addr":"172.21.15.49:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-21T13:44:26.386 INFO:teuthology.orchestra.run.smithi187.stderr:dumped monmap epoch 1 2024-09-21T13:44:27.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:26 smithi049 ceph-mon[30080]: from='client.? 172.21.15.187:0/2206259496' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T13:44:28.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:27 smithi049 ceph-mon[30080]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:28.672 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:28 smithi187 ceph-mon[33032]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-21T13:44:28.703 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-21T13:44:28.703 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph mon dump -f json 2024-09-21T13:44:28.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:28 smithi187 ceph-mon[33032]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-21T13:44:28.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:28 smithi187 ceph-mon[33032]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-21T13:44:28.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:28 smithi187 ceph-mon[33032]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-21T13:44:28.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:28 smithi187 ceph-mon[33032]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-21T13:44:28.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:28 smithi187 ceph-mon[33032]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-21T13:44:28.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:28 smithi187 ceph-mon[33032]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-21T13:44:28.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:28 smithi187 ceph-mon[33032]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T13:44:28.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:28 smithi187 ceph-mon[33032]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T13:44:28.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:28 smithi187 ceph-mon[33032]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T13:44:28.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:28 smithi187 ceph-mon[33032]: Deploying cephadm binary to smithi187 2024-09-21T13:44:28.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:28 smithi187 ceph-mon[33032]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:28.947 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:28 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:28.947 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:28 smithi187 ceph-mon[33032]: Added host smithi187 2024-09-21T13:44:28.947 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:28 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:44:28.947 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:28 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:28.947 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:28 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:28.947 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:28 smithi187 ceph-mon[33032]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:28.947 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:28 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:28.948 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:28 smithi187 ceph-mon[33032]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T13:44:28.948 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:28 smithi187 ceph-mon[33032]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:28.948 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:28 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1368541128' entity='client.admin' cmd={"prefix": "osd crush tunables", "profile": "default"} : dispatch 2024-09-21T13:44:28.948 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:28 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1368541128' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-21T13:44:28.948 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:28 smithi187 ceph-mon[33032]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T13:44:28.948 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:28 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:28.948 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:28 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:28.949 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:28 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:28.949 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:28 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:28.949 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:28 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi187", "name": "osd_memory_target"} : dispatch 2024-09-21T13:44:28.949 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:28 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:44:28.949 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:28 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:44:28.949 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:28 smithi187 ceph-mon[33032]: Updating smithi187:/etc/ceph/ceph.conf 2024-09-21T13:44:28.949 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:28 smithi187 ceph-mon[33032]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:28.950 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:28 smithi187 ceph-mon[33032]: Updating smithi187:/var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T13:44:28.950 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:28 smithi187 ceph-mon[33032]: Updating smithi187:/etc/ceph/ceph.client.admin.keyring 2024-09-21T13:44:28.950 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:28 smithi187 ceph-mon[33032]: Updating smithi187:/var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T13:44:28.950 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:28 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:28.950 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:28 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:28.950 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:28 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:28.950 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:28 smithi187 ceph-mon[33032]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi049:172.21.15.49=a;smithi049:[v2:172.21.15.49:3301,v1:172.21.15.49:6790]=c;smithi187:172.21.15.187=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:44:28.951 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:28 smithi187 ceph-mon[33032]: Saving service mon spec with placement smithi049:172.21.15.49=a;smithi049:[v2:172.21.15.49:3301,v1:172.21.15.49:6790]=c;smithi187:172.21.15.187=b;count:3 2024-09-21T13:44:28.951 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:28 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:28.951 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:28 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:44:28.951 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:28 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:44:28.951 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:28 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:44:28.951 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:28 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:28.951 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:28 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T13:44:28.952 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:28 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:44:28.952 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:28 smithi187 ceph-mon[33032]: Deploying daemon mon.b on smithi187 2024-09-21T13:44:28.952 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:28 smithi187 ceph-mon[33032]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:28.952 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:28 smithi187 ceph-mon[33032]: from='client.? 172.21.15.187:0/2206259496' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T13:44:28.952 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:28 smithi187 ceph-mon[33032]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:28.952 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:28 smithi187 ceph-mon[33032]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-21T13:44:28.952 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:28 smithi187 bash[32994]: f5f4713510d2b18e55f23a9e0332d00d119cc061ac1e4407c2215fc553576ab4 2024-09-21T13:44:28.953 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:28 smithi187 systemd[1]: Started Ceph mon.b for 40a3d2e8-781f-11ef-baf6-efdc52797490. 2024-09-21T13:44:28.954 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:44:34.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:33 smithi049 ceph-mon[30080]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:34.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:33 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T13:44:34.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:33 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T13:44:34.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:33 smithi049 ceph-mon[30080]: mon.a calling monitor election 2024-09-21T13:44:34.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:33 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T13:44:34.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:33 smithi049 ceph-mon[30080]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:34.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:33 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T13:44:34.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:33 smithi049 ceph-mon[30080]: mon.b calling monitor election 2024-09-21T13:44:34.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:33 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T13:44:34.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:33 smithi049 ceph-mon[30080]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:34.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:33 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T13:44:34.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:33 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T13:44:34.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:33 smithi049 ceph-mon[30080]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-21T13:44:34.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:33 smithi049 ceph-mon[30080]: monmap epoch 2 2024-09-21T13:44:34.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:33 smithi049 ceph-mon[30080]: fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 2024-09-21T13:44:34.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:33 smithi049 ceph-mon[30080]: last_changed 2024-09-21T13:44:28.677887+0000 2024-09-21T13:44:34.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:33 smithi049 ceph-mon[30080]: created 2024-09-21T13:42:51.046146+0000 2024-09-21T13:44:34.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:33 smithi049 ceph-mon[30080]: min_mon_release 19 (squid) 2024-09-21T13:44:34.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:33 smithi049 ceph-mon[30080]: election_strategy: 1 2024-09-21T13:44:34.007 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:33 smithi049 ceph-mon[30080]: 0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-09-21T13:44:34.007 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:33 smithi049 ceph-mon[30080]: 1: [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] mon.b 2024-09-21T13:44:34.007 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:33 smithi049 ceph-mon[30080]: fsmap 2024-09-21T13:44:34.007 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:33 smithi049 ceph-mon[30080]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T13:44:34.007 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:33 smithi049 ceph-mon[30080]: mgrmap e14: y(active, since 37s) 2024-09-21T13:44:34.007 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:33 smithi049 ceph-mon[30080]: overall HEALTH_OK 2024-09-21T13:44:34.007 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:33 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:34.007 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:33 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:34.007 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:33 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:34.007 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:33 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T13:44:34.007 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:33 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:44:34.069 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:44:34.069 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":2,"fsid":"40a3d2e8-781f-11ef-baf6-efdc52797490","modified":"2024-09-21T13:44:28.677887Z","created":"2024-09-21T13:42:51.046146Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:3300","nonce":0},{"type":"v1","addr":"172.21.15.49:6789","nonce":0}]},"addr":"172.21.15.49:6789/0","public_addr":"172.21.15.49:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:3300","nonce":0},{"type":"v1","addr":"172.21.15.187:6789","nonce":0}]},"addr":"172.21.15.187:6789/0","public_addr":"172.21.15.187:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-21T13:44:34.069 INFO:teuthology.orchestra.run.smithi187.stderr:dumped monmap epoch 2 2024-09-21T13:44:34.070 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:33 smithi187 ceph-mon[33032]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:34.070 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:33 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T13:44:34.070 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:33 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T13:44:34.070 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:33 smithi187 ceph-mon[33032]: mon.a calling monitor election 2024-09-21T13:44:34.070 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:33 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T13:44:34.070 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:33 smithi187 ceph-mon[33032]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:34.070 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:33 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T13:44:34.070 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:33 smithi187 ceph-mon[33032]: mon.b calling monitor election 2024-09-21T13:44:34.070 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:33 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T13:44:34.070 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:33 smithi187 ceph-mon[33032]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:34.071 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:33 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T13:44:34.071 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:33 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T13:44:34.071 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:33 smithi187 ceph-mon[33032]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-21T13:44:34.071 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:33 smithi187 ceph-mon[33032]: monmap epoch 2 2024-09-21T13:44:34.071 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:33 smithi187 ceph-mon[33032]: fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 2024-09-21T13:44:34.071 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:33 smithi187 ceph-mon[33032]: last_changed 2024-09-21T13:44:28.677887+0000 2024-09-21T13:44:34.071 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:33 smithi187 ceph-mon[33032]: created 2024-09-21T13:42:51.046146+0000 2024-09-21T13:44:34.071 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:33 smithi187 ceph-mon[33032]: min_mon_release 19 (squid) 2024-09-21T13:44:34.071 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:33 smithi187 ceph-mon[33032]: election_strategy: 1 2024-09-21T13:44:34.071 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:33 smithi187 ceph-mon[33032]: 0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-09-21T13:44:34.071 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:33 smithi187 ceph-mon[33032]: 1: [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] mon.b 2024-09-21T13:44:34.071 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:33 smithi187 ceph-mon[33032]: fsmap 2024-09-21T13:44:34.071 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:33 smithi187 ceph-mon[33032]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T13:44:34.071 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:33 smithi187 ceph-mon[33032]: mgrmap e14: y(active, since 37s) 2024-09-21T13:44:34.071 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:33 smithi187 ceph-mon[33032]: overall HEALTH_OK 2024-09-21T13:44:34.072 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:33 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:34.072 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:33 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:34.072 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:33 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:34.072 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:33 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T13:44:34.072 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:33 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:44:34.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:34 smithi187 ceph-mon[33032]: Deploying daemon mon.c on smithi049 2024-09-21T13:44:34.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:34 smithi187 ceph-mon[33032]: from='client.? 172.21.15.187:0/2329667103' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T13:44:34.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:34 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T13:44:35.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:34 smithi049 ceph-mon[30080]: Deploying daemon mon.c on smithi049 2024-09-21T13:44:35.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:34 smithi049 ceph-mon[30080]: from='client.? 172.21.15.187:0/2329667103' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T13:44:35.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:34 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T13:44:35.661 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-21T13:44:35.661 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph mon dump -f json 2024-09-21T13:44:35.902 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:44:35.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:35 smithi187 ceph-mon[33032]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:36.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:35 smithi049 ceph-mon[30080]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:36.572 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:44:36.572 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":2,"fsid":"40a3d2e8-781f-11ef-baf6-efdc52797490","modified":"2024-09-21T13:44:28.677887Z","created":"2024-09-21T13:42:51.046146Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:3300","nonce":0},{"type":"v1","addr":"172.21.15.49:6789","nonce":0}]},"addr":"172.21.15.49:6789/0","public_addr":"172.21.15.49:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:3300","nonce":0},{"type":"v1","addr":"172.21.15.187:6789","nonce":0}]},"addr":"172.21.15.187:6789/0","public_addr":"172.21.15.187:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-21T13:44:36.572 INFO:teuthology.orchestra.run.smithi187.stderr:dumped monmap epoch 2 2024-09-21T13:44:36.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:36 smithi049 systemd[1]: Starting Ceph mon.c for 40a3d2e8-781f-11ef-baf6-efdc52797490... 2024-09-21T13:44:36.872 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:36 smithi187 ceph-mon[33032]: from='client.? 172.21.15.187:0/2350592671' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T13:44:37.094 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:36 smithi049 ceph-mon[30080]: from='client.? 172.21.15.187:0/2350592671' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T13:44:37.095 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:36 smithi049 podman[40451]: 2024-09-21 13:44:36.743670263 +0000 UTC m=+0.016575848 image pull b415816db8b41f544cdfdc71d193ff8b1ee008bf7b0ec4f64da8eb8f71b62e85 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5 2024-09-21T13:44:37.095 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:36 smithi049 podman[40451]: 2024-09-21 13:44:36.867627902 +0000 UTC m=+0.140533469 container create cd9887925ce4b247db9da2687f910e68500abf41fdcbc3c776120dd25f835f91 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mon-c, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-guits-main-2024-09-21-0842-4417849, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T13:44:37.095 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 podman[40451]: 2024-09-21 13:44:37.094814088 +0000 UTC m=+0.367719668 container init cd9887925ce4b247db9da2687f910e68500abf41fdcbc3c776120dd25f835f91 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mon-c, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=) 2024-09-21T13:44:37.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 podman[40451]: 2024-09-21 13:44:37.099654168 +0000 UTC m=+0.372559733 container start cd9887925ce4b247db9da2687f910e68500abf41fdcbc3c776120dd25f835f91 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mon-c, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-main-2024-09-21-0842-4417849, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T13:44:37.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: set uid:gid to 167:167 (ceph:ceph) 2024-09-21T13:44:37.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: ceph version 19.3.0-5104-g44178494 (441784946648ff53dd310293f9dd752afeeea72a) squid (dev), process ceph-mon, pid 7 2024-09-21T13:44:37.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: pidfile_write: ignore empty --pid-file 2024-09-21T13:44:37.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: load: jerasure load: lrc 2024-09-21T13:44:37.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: RocksDB version: 7.9.2 2024-09-21T13:44:37.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Git sha 0 2024-09-21T13:44:37.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Compile date 2024-09-21 09:24:23 2024-09-21T13:44:37.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: DB SUMMARY 2024-09-21T13:44:37.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: DB Session ID: 37X61BZBI1C4VFKMSETX 2024-09-21T13:44:37.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: CURRENT file: CURRENT 2024-09-21T13:44:37.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: IDENTITY file: IDENTITY 2024-09-21T13:44:37.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-09-21T13:44:37.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-09-21T13:44:37.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 636 ; 2024-09-21T13:44:37.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.error_if_exists: 0 2024-09-21T13:44:37.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.create_if_missing: 0 2024-09-21T13:44:37.507 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.paranoid_checks: 1 2024-09-21T13:44:37.507 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-21T13:44:37.507 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-21T13:44:37.507 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-21T13:44:37.507 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.env: 0x55e92f37f120 2024-09-21T13:44:37.507 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.fs: PosixFileSystem 2024-09-21T13:44:37.507 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.info_log: 0x55e9308a5700 2024-09-21T13:44:37.507 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.max_file_opening_threads: 16 2024-09-21T13:44:37.507 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.statistics: (nil) 2024-09-21T13:44:37.507 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.use_fsync: 0 2024-09-21T13:44:37.507 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.max_log_file_size: 0 2024-09-21T13:44:37.507 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-21T13:44:37.507 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-21T13:44:37.507 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.keep_log_file_num: 1000 2024-09-21T13:44:37.507 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.recycle_log_file_num: 0 2024-09-21T13:44:37.508 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.allow_fallocate: 1 2024-09-21T13:44:37.509 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.allow_mmap_reads: 0 2024-09-21T13:44:37.509 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.allow_mmap_writes: 0 2024-09-21T13:44:37.509 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.use_direct_reads: 0 2024-09-21T13:44:37.509 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-21T13:44:37.509 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.create_missing_column_families: 0 2024-09-21T13:44:37.509 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.db_log_dir: 2024-09-21T13:44:37.509 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.wal_dir: 2024-09-21T13:44:37.510 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-21T13:44:37.510 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-21T13:44:37.510 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-21T13:44:37.510 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-21T13:44:37.510 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-21T13:44:37.510 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-21T13:44:37.510 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.advise_random_on_open: 1 2024-09-21T13:44:37.510 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.db_write_buffer_size: 0 2024-09-21T13:44:37.510 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.write_buffer_manager: 0x55e9308a9b80 2024-09-21T13:44:37.510 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-21T13:44:37.510 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-21T13:44:37.510 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-21T13:44:37.510 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.rate_limiter: (nil) 2024-09-21T13:44:37.511 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-21T13:44:37.511 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.wal_recovery_mode: 2 2024-09-21T13:44:37.511 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.enable_thread_tracking: 0 2024-09-21T13:44:37.511 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.enable_pipelined_write: 0 2024-09-21T13:44:37.511 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.unordered_write: 0 2024-09-21T13:44:37.511 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-21T13:44:37.511 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-21T13:44:37.511 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-21T13:44:37.511 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-21T13:44:37.511 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.row_cache: None 2024-09-21T13:44:37.511 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.wal_filter: None 2024-09-21T13:44:37.511 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-21T13:44:37.511 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.allow_ingest_behind: 0 2024-09-21T13:44:37.511 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.two_write_queues: 0 2024-09-21T13:44:37.511 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.manual_wal_flush: 0 2024-09-21T13:44:37.513 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.wal_compression: 0 2024-09-21T13:44:37.513 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.atomic_flush: 0 2024-09-21T13:44:37.513 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-21T13:44:37.513 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-21T13:44:37.513 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-21T13:44:37.514 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.log_readahead_size: 0 2024-09-21T13:44:37.514 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-21T13:44:37.514 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.best_efforts_recovery: 0 2024-09-21T13:44:37.514 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-21T13:44:37.514 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-21T13:44:37.514 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.allow_data_in_errors: 0 2024-09-21T13:44:37.514 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.db_host_id: __hostname__ 2024-09-21T13:44:37.514 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-21T13:44:37.514 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.max_background_jobs: 2 2024-09-21T13:44:37.514 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.max_background_compactions: -1 2024-09-21T13:44:37.514 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.max_subcompactions: 1 2024-09-21T13:44:37.514 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-21T13:44:37.514 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-21T13:44:37.514 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-21T13:44:37.515 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.max_total_wal_size: 0 2024-09-21T13:44:37.515 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-21T13:44:37.515 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-21T13:44:37.515 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-21T13:44:37.515 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-21T13:44:37.515 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.max_open_files: -1 2024-09-21T13:44:37.515 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.bytes_per_sync: 0 2024-09-21T13:44:37.515 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-21T13:44:37.515 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-21T13:44:37.515 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.compaction_readahead_size: 0 2024-09-21T13:44:37.515 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.max_background_flushes: -1 2024-09-21T13:44:37.515 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Compression algorithms supported: 2024-09-21T13:44:37.515 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: kZSTD supported: 0 2024-09-21T13:44:37.515 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: kXpressCompression supported: 0 2024-09-21T13:44:37.515 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: kBZip2Compression supported: 0 2024-09-21T13:44:37.516 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-21T13:44:37.516 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: kLZ4Compression supported: 1 2024-09-21T13:44:37.516 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: kZlibCompression supported: 1 2024-09-21T13:44:37.516 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: kLZ4HCCompression supported: 1 2024-09-21T13:44:37.516 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: kSnappyCompression supported: 1 2024-09-21T13:44:37.516 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-21T13:44:37.516 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-21T13:44:37.516 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-09-21T13:44:37.516 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-21T13:44:37.516 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-21T13:44:37.516 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.merge_operator: 2024-09-21T13:44:37.516 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.compaction_filter: None 2024-09-21T13:44:37.516 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.compaction_filter_factory: None 2024-09-21T13:44:37.516 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.sst_partitioner_factory: None 2024-09-21T13:44:37.516 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-21T13:44:37.517 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-21T13:44:37.517 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55e9308a5320) 2024-09-21T13:44:37.517 INFO:journalctl@ceph.mon.c.smithi049.stdout: cache_index_and_filter_blocks: 1 2024-09-21T13:44:37.517 INFO:journalctl@ceph.mon.c.smithi049.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-21T13:44:37.517 INFO:journalctl@ceph.mon.c.smithi049.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-21T13:44:37.517 INFO:journalctl@ceph.mon.c.smithi049.stdout: pin_top_level_index_and_filter: 1 2024-09-21T13:44:37.517 INFO:journalctl@ceph.mon.c.smithi049.stdout: index_type: 0 2024-09-21T13:44:37.517 INFO:journalctl@ceph.mon.c.smithi049.stdout: data_block_index_type: 0 2024-09-21T13:44:37.517 INFO:journalctl@ceph.mon.c.smithi049.stdout: index_shortening: 1 2024-09-21T13:44:37.517 INFO:journalctl@ceph.mon.c.smithi049.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-21T13:44:37.517 INFO:journalctl@ceph.mon.c.smithi049.stdout: checksum: 4 2024-09-21T13:44:37.517 INFO:journalctl@ceph.mon.c.smithi049.stdout: no_block_cache: 0 2024-09-21T13:44:37.517 INFO:journalctl@ceph.mon.c.smithi049.stdout: block_cache: 0x55e930899610 2024-09-21T13:44:37.517 INFO:journalctl@ceph.mon.c.smithi049.stdout: block_cache_name: BinnedLRUCache 2024-09-21T13:44:37.518 INFO:journalctl@ceph.mon.c.smithi049.stdout: block_cache_options: 2024-09-21T13:44:37.518 INFO:journalctl@ceph.mon.c.smithi049.stdout: capacity : 536870912 2024-09-21T13:44:37.518 INFO:journalctl@ceph.mon.c.smithi049.stdout: num_shard_bits : 4 2024-09-21T13:44:37.518 INFO:journalctl@ceph.mon.c.smithi049.stdout: strict_capacity_limit : 0 2024-09-21T13:44:37.518 INFO:journalctl@ceph.mon.c.smithi049.stdout: high_pri_pool_ratio: 0.000 2024-09-21T13:44:37.518 INFO:journalctl@ceph.mon.c.smithi049.stdout: block_cache_compressed: (nil) 2024-09-21T13:44:37.518 INFO:journalctl@ceph.mon.c.smithi049.stdout: persistent_cache: (nil) 2024-09-21T13:44:37.518 INFO:journalctl@ceph.mon.c.smithi049.stdout: block_size: 4096 2024-09-21T13:44:37.518 INFO:journalctl@ceph.mon.c.smithi049.stdout: block_size_deviation: 10 2024-09-21T13:44:37.518 INFO:journalctl@ceph.mon.c.smithi049.stdout: block_restart_interval: 16 2024-09-21T13:44:37.518 INFO:journalctl@ceph.mon.c.smithi049.stdout: index_block_restart_interval: 1 2024-09-21T13:44:37.518 INFO:journalctl@ceph.mon.c.smithi049.stdout: metadata_block_size: 4096 2024-09-21T13:44:37.518 INFO:journalctl@ceph.mon.c.smithi049.stdout: partition_filters: 0 2024-09-21T13:44:37.518 INFO:journalctl@ceph.mon.c.smithi049.stdout: use_delta_encoding: 1 2024-09-21T13:44:37.518 INFO:journalctl@ceph.mon.c.smithi049.stdout: filter_policy: bloomfilter 2024-09-21T13:44:37.519 INFO:journalctl@ceph.mon.c.smithi049.stdout: whole_key_filtering: 1 2024-09-21T13:44:37.519 INFO:journalctl@ceph.mon.c.smithi049.stdout: verify_compression: 0 2024-09-21T13:44:37.519 INFO:journalctl@ceph.mon.c.smithi049.stdout: read_amp_bytes_per_bit: 0 2024-09-21T13:44:37.519 INFO:journalctl@ceph.mon.c.smithi049.stdout: format_version: 5 2024-09-21T13:44:37.519 INFO:journalctl@ceph.mon.c.smithi049.stdout: enable_index_compression: 1 2024-09-21T13:44:37.519 INFO:journalctl@ceph.mon.c.smithi049.stdout: block_align: 0 2024-09-21T13:44:37.519 INFO:journalctl@ceph.mon.c.smithi049.stdout: max_auto_readahead_size: 262144 2024-09-21T13:44:37.519 INFO:journalctl@ceph.mon.c.smithi049.stdout: prepopulate_block_cache: 0 2024-09-21T13:44:37.519 INFO:journalctl@ceph.mon.c.smithi049.stdout: initial_auto_readahead_size: 8192 2024-09-21T13:44:37.519 INFO:journalctl@ceph.mon.c.smithi049.stdout: num_file_reads_for_auto_readahead: 2 2024-09-21T13:44:37.519 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.write_buffer_size: 33554432 2024-09-21T13:44:37.519 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.max_write_buffer_number: 2 2024-09-21T13:44:37.519 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.compression: NoCompression 2024-09-21T13:44:37.519 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.bottommost_compression: Disabled 2024-09-21T13:44:37.519 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.prefix_extractor: nullptr 2024-09-21T13:44:37.520 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-21T13:44:37.520 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.num_levels: 7 2024-09-21T13:44:37.520 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-21T13:44:37.520 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-21T13:44:37.520 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-21T13:44:37.520 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-21T13:44:37.520 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-21T13:44:37.520 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-21T13:44:37.520 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-21T13:44:37.520 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-21T13:44:37.520 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-21T13:44:37.520 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-21T13:44:37.520 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-21T13:44:37.520 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-21T13:44:37.520 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-21T13:44:37.521 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.compression_opts.level: 32767 2024-09-21T13:44:37.521 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.compression_opts.strategy: 0 2024-09-21T13:44:37.521 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-21T13:44:37.521 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-21T13:44:37.521 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-21T13:44:37.521 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-21T13:44:37.521 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.compression_opts.enabled: false 2024-09-21T13:44:37.521 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-21T13:44:37.521 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-21T13:44:37.521 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-21T13:44:37.521 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-21T13:44:37.521 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.target_file_size_base: 67108864 2024-09-21T13:44:37.521 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-21T13:44:37.521 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-21T13:44:37.521 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-21T13:44:37.521 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-21T13:44:37.522 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-21T13:44:37.522 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-21T13:44:37.522 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-21T13:44:37.522 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-21T13:44:37.522 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-21T13:44:37.522 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-21T13:44:37.522 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-21T13:44:37.522 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-21T13:44:37.522 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-21T13:44:37.522 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-21T13:44:37.522 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.arena_block_size: 1048576 2024-09-21T13:44:37.522 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-21T13:44:37.522 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-21T13:44:37.522 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.disable_auto_compactions: 0 2024-09-21T13:44:37.523 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-21T13:44:37.523 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-21T13:44:37.523 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-21T13:44:37.523 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-21T13:44:37.523 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-21T13:44:37.523 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-21T13:44:37.523 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-21T13:44:37.523 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-21T13:44:37.523 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-21T13:44:37.523 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-21T13:44:37.523 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-21T13:44:37.523 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.inplace_update_support: 0 2024-09-21T13:44:37.523 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-21T13:44:37.523 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-21T13:44:37.524 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-21T13:44:37.524 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-21T13:44:37.524 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.bloom_locality: 0 2024-09-21T13:44:37.524 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.max_successive_merges: 0 2024-09-21T13:44:37.524 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-21T13:44:37.524 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.paranoid_file_checks: 0 2024-09-21T13:44:37.524 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.force_consistency_checks: 1 2024-09-21T13:44:37.524 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.report_bg_io_stats: 0 2024-09-21T13:44:37.524 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.ttl: 2592000 2024-09-21T13:44:37.524 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-21T13:44:37.524 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-21T13:44:37.524 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-21T13:44:37.524 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.enable_blob_files: false 2024-09-21T13:44:37.524 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.min_blob_size: 0 2024-09-21T13:44:37.524 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.blob_file_size: 268435456 2024-09-21T13:44:37.525 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-21T13:44:37.525 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-21T13:44:37.525 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-21T13:44:37.525 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-21T13:44:37.525 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-21T13:44:37.525 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.blob_file_starting_level: 0 2024-09-21T13:44:37.525 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-21T13:44:37.525 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-21T13:44:37.525 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-09-21T13:44:37.525 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 8bb33ee5-da0b-43cd-a576-4aaec5e6c29f 2024-09-21T13:44:37.525 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726926277127416, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-21T13:44:37.525 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-09-21T13:44:37.525 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726926277127813, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1773, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726926277, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "8bb33ee5-da0b-43cd-a576-4aaec5e6c29f", "db_session_id": "37X61BZBI1C4VFKMSETX", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-09-21T13:44:37.525 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726926277127910, "job": 1, "event": "recovery_finished"} 2024-09-21T13:44:37.525 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-09-21T13:44:37.526 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-21T13:44:37.526 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55e9308c8e00 2024-09-21T13:44:37.526 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: DB pointer 0x55e9309ca000 2024-09-21T13:44:37.526 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-21T13:44:37.526 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-21T13:44:37.526 INFO:journalctl@ceph.mon.c.smithi049.stdout: ** DB Stats ** 2024-09-21T13:44:37.526 INFO:journalctl@ceph.mon.c.smithi049.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T13:44:37.526 INFO:journalctl@ceph.mon.c.smithi049.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-21T13:44:37.526 INFO:journalctl@ceph.mon.c.smithi049.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T13:44:37.526 INFO:journalctl@ceph.mon.c.smithi049.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T13:44:37.526 INFO:journalctl@ceph.mon.c.smithi049.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-21T13:44:37.526 INFO:journalctl@ceph.mon.c.smithi049.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T13:44:37.526 INFO:journalctl@ceph.mon.c.smithi049.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T13:44:37.526 INFO:journalctl@ceph.mon.c.smithi049.stdout: 2024-09-21T13:44:37.527 INFO:journalctl@ceph.mon.c.smithi049.stdout: ** Compaction Stats [default] ** 2024-09-21T13:44:37.527 INFO:journalctl@ceph.mon.c.smithi049.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T13:44:37.527 INFO:journalctl@ceph.mon.c.smithi049.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-21T13:44:37.527 INFO:journalctl@ceph.mon.c.smithi049.stdout: L0 1/0 1.73 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-21T13:44:37.527 INFO:journalctl@ceph.mon.c.smithi049.stdout: Sum 1/0 1.73 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-21T13:44:37.527 INFO:journalctl@ceph.mon.c.smithi049.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-21T13:44:37.527 INFO:journalctl@ceph.mon.c.smithi049.stdout: 2024-09-21T13:44:37.527 INFO:journalctl@ceph.mon.c.smithi049.stdout: ** Compaction Stats [default] ** 2024-09-21T13:44:37.527 INFO:journalctl@ceph.mon.c.smithi049.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T13:44:37.527 INFO:journalctl@ceph.mon.c.smithi049.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T13:44:37.527 INFO:journalctl@ceph.mon.c.smithi049.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-21T13:44:37.527 INFO:journalctl@ceph.mon.c.smithi049.stdout: 2024-09-21T13:44:37.527 INFO:journalctl@ceph.mon.c.smithi049.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-21T13:44:37.527 INFO:journalctl@ceph.mon.c.smithi049.stdout: 2024-09-21T13:44:37.528 INFO:journalctl@ceph.mon.c.smithi049.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T13:44:37.528 INFO:journalctl@ceph.mon.c.smithi049.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-21T13:44:37.528 INFO:journalctl@ceph.mon.c.smithi049.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-21T13:44:37.528 INFO:journalctl@ceph.mon.c.smithi049.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-21T13:44:37.528 INFO:journalctl@ceph.mon.c.smithi049.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-21T13:44:37.528 INFO:journalctl@ceph.mon.c.smithi049.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-21T13:44:37.528 INFO:journalctl@ceph.mon.c.smithi049.stdout: Cumulative compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T13:44:37.528 INFO:journalctl@ceph.mon.c.smithi049.stdout: Interval compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T13:44:37.528 INFO:journalctl@ceph.mon.c.smithi049.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-21T13:44:37.528 INFO:journalctl@ceph.mon.c.smithi049.stdout: Block cache BinnedLRUCache@0x55e930899610#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-09-21T13:44:37.528 INFO:journalctl@ceph.mon.c.smithi049.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-09-21T13:44:37.528 INFO:journalctl@ceph.mon.c.smithi049.stdout: 2024-09-21T13:44:37.528 INFO:journalctl@ceph.mon.c.smithi049.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-21T13:44:37.529 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-09-21T13:44:37.529 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: using public_addrv [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-09-21T13:44:37.529 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: starting mon.c rank -1 at public addrs [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] at bind addrs [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 2024-09-21T13:44:37.529 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: mon.c@-1(???) e0 preinit fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 2024-09-21T13:44:37.529 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: mon.c@-1(synchronizing).mds e1 new map 2024-09-21T13:44:37.529 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: mon.c@-1(synchronizing).mds e1 print_map 2024-09-21T13:44:37.529 INFO:journalctl@ceph.mon.c.smithi049.stdout: e1 2024-09-21T13:44:37.529 INFO:journalctl@ceph.mon.c.smithi049.stdout: btime 2024-09-21T13:42:57:838094+0000 2024-09-21T13:44:37.529 INFO:journalctl@ceph.mon.c.smithi049.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-21T13:44:37.529 INFO:journalctl@ceph.mon.c.smithi049.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-21T13:44:37.529 INFO:journalctl@ceph.mon.c.smithi049.stdout: legacy client fscid: -1 2024-09-21T13:44:37.529 INFO:journalctl@ceph.mon.c.smithi049.stdout: 2024-09-21T13:44:37.529 INFO:journalctl@ceph.mon.c.smithi049.stdout: No filesystems configured 2024-09-21T13:44:37.529 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-21T13:44:37.529 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-21T13:44:37.530 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-21T13:44:37.530 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-21T13:44:37.530 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-21T13:44:37.530 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-21T13:44:37.530 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-21T13:44:37.530 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T13:44:37.530 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T13:44:37.530 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T13:44:37.530 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: Deploying cephadm binary to smithi187 2024-09-21T13:44:37.530 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:37.530 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:37.530 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: Added host smithi187 2024-09-21T13:44:37.530 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:44:37.530 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:37.531 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:37.531 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:37.531 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:37.531 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T13:44:37.531 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:37.531 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1368541128' entity='client.admin' cmd={"prefix": "osd crush tunables", "profile": "default"} : dispatch 2024-09-21T13:44:37.531 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1368541128' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-21T13:44:37.531 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T13:44:37.531 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:37.531 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:37.531 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:37.531 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:37.531 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi187", "name": "osd_memory_target"} : dispatch 2024-09-21T13:44:37.531 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:44:37.531 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:44:37.532 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: Updating smithi187:/etc/ceph/ceph.conf 2024-09-21T13:44:37.532 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:37.532 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: Updating smithi187:/var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T13:44:37.532 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: Updating smithi187:/etc/ceph/ceph.client.admin.keyring 2024-09-21T13:44:37.532 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: Updating smithi187:/var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T13:44:37.532 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:37.532 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:37.532 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:37.532 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi049:172.21.15.49=a;smithi049:[v2:172.21.15.49:3301,v1:172.21.15.49:6790]=c;smithi187:172.21.15.187=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:44:37.532 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: Saving service mon spec with placement smithi049:172.21.15.49=a;smithi049:[v2:172.21.15.49:3301,v1:172.21.15.49:6790]=c;smithi187:172.21.15.187=b;count:3 2024-09-21T13:44:37.532 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:37.532 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:44:37.532 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:44:37.532 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:44:37.532 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:37.533 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T13:44:37.533 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:44:37.533 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: Deploying daemon mon.b on smithi187 2024-09-21T13:44:37.533 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:37.533 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: from='client.? 172.21.15.187:0/2206259496' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T13:44:37.533 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:37.533 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:37.533 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T13:44:37.533 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T13:44:37.533 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: mon.a calling monitor election 2024-09-21T13:44:37.533 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T13:44:37.533 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:37.533 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T13:44:37.533 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: mon.b calling monitor election 2024-09-21T13:44:37.533 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T13:44:37.534 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:37.534 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T13:44:37.534 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T13:44:37.534 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-21T13:44:37.534 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: monmap epoch 2 2024-09-21T13:44:37.534 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 2024-09-21T13:44:37.534 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: last_changed 2024-09-21T13:44:28.677887+0000 2024-09-21T13:44:37.534 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: created 2024-09-21T13:42:51.046146+0000 2024-09-21T13:44:37.534 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: min_mon_release 19 (squid) 2024-09-21T13:44:37.534 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: election_strategy: 1 2024-09-21T13:44:37.534 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: 0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-09-21T13:44:37.534 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: 1: [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] mon.b 2024-09-21T13:44:37.534 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: fsmap 2024-09-21T13:44:37.534 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T13:44:37.535 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: mgrmap e14: y(active, since 37s) 2024-09-21T13:44:37.535 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: overall HEALTH_OK 2024-09-21T13:44:37.535 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:37.535 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:37.535 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:37.535 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T13:44:37.535 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:44:37.535 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: Deploying daemon mon.c on smithi049 2024-09-21T13:44:37.535 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: from='client.? 172.21.15.187:0/2329667103' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T13:44:37.535 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T13:44:37.535 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:37.535 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: from='client.? 172.21.15.187:0/2350592671' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T13:44:37.535 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 ceph-mon[40489]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-21T13:44:37.535 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 bash[40451]: cd9887925ce4b247db9da2687f910e68500abf41fdcbc3c776120dd25f835f91 2024-09-21T13:44:37.535 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:37 smithi049 systemd[1]: Started Ceph mon.c for 40a3d2e8-781f-11ef-baf6-efdc52797490. 2024-09-21T13:44:38.152 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-21T13:44:38.153 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph mon dump -f json 2024-09-21T13:44:38.407 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:44:42.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:42 smithi049 ceph-mon[30080]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:42.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:42 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T13:44:42.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:42 smithi049 ceph-mon[30080]: mon.a calling monitor election 2024-09-21T13:44:42.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:42 smithi049 ceph-mon[30080]: mon.b calling monitor election 2024-09-21T13:44:42.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:42 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T13:44:42.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:42 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T13:44:42.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:42 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T13:44:42.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:42 smithi049 ceph-mon[30080]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:42.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:42 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T13:44:42.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:42 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T13:44:42.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:42 smithi049 ceph-mon[30080]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:42.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:42 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T13:44:42.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:42 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T13:44:42.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:42 smithi049 ceph-mon[30080]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-21T13:44:42.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:42 smithi049 ceph-mon[30080]: monmap epoch 3 2024-09-21T13:44:42.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:42 smithi049 ceph-mon[30080]: fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 2024-09-21T13:44:42.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:42 smithi049 ceph-mon[30080]: last_changed 2024-09-21T13:44:37.153888+0000 2024-09-21T13:44:42.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:42 smithi049 ceph-mon[30080]: created 2024-09-21T13:42:51.046146+0000 2024-09-21T13:44:42.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:42 smithi049 ceph-mon[30080]: min_mon_release 19 (squid) 2024-09-21T13:44:42.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:42 smithi049 ceph-mon[30080]: election_strategy: 1 2024-09-21T13:44:42.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:42 smithi049 ceph-mon[30080]: 0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-09-21T13:44:42.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:42 smithi049 ceph-mon[30080]: 1: [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] mon.b 2024-09-21T13:44:42.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:42 smithi049 ceph-mon[30080]: 2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-09-21T13:44:42.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:42 smithi049 ceph-mon[30080]: fsmap 2024-09-21T13:44:42.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:42 smithi049 ceph-mon[30080]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T13:44:42.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:42 smithi049 ceph-mon[30080]: mgrmap e14: y(active, since 45s) 2024-09-21T13:44:42.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:42 smithi049 ceph-mon[30080]: Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-09-21T13:44:42.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:42 smithi049 ceph-mon[30080]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-09-21T13:44:42.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:42 smithi049 ceph-mon[30080]: [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-09-21T13:44:42.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:42 smithi049 ceph-mon[30080]: mon.c (rank 2) addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] is down (out of quorum) 2024-09-21T13:44:42.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:42 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:42.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:42 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:42.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:42 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:42.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:42 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:42.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:42 smithi187 ceph-mon[33032]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:42.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:42 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T13:44:42.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:42 smithi187 ceph-mon[33032]: mon.a calling monitor election 2024-09-21T13:44:42.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:42 smithi187 ceph-mon[33032]: mon.b calling monitor election 2024-09-21T13:44:42.696 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:42 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T13:44:42.696 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:42 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T13:44:42.696 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:42 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T13:44:42.696 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:42 smithi187 ceph-mon[33032]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:42.696 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:42 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T13:44:42.696 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:42 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T13:44:42.696 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:42 smithi187 ceph-mon[33032]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:42.696 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:42 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T13:44:42.696 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:42 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T13:44:42.696 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:42 smithi187 ceph-mon[33032]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-21T13:44:42.696 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:42 smithi187 ceph-mon[33032]: monmap epoch 3 2024-09-21T13:44:42.696 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:42 smithi187 ceph-mon[33032]: fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 2024-09-21T13:44:42.696 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:42 smithi187 ceph-mon[33032]: last_changed 2024-09-21T13:44:37.153888+0000 2024-09-21T13:44:42.696 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:42 smithi187 ceph-mon[33032]: created 2024-09-21T13:42:51.046146+0000 2024-09-21T13:44:42.697 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:42 smithi187 ceph-mon[33032]: min_mon_release 19 (squid) 2024-09-21T13:44:42.697 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:42 smithi187 ceph-mon[33032]: election_strategy: 1 2024-09-21T13:44:42.697 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:42 smithi187 ceph-mon[33032]: 0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-09-21T13:44:42.697 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:42 smithi187 ceph-mon[33032]: 1: [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] mon.b 2024-09-21T13:44:42.697 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:42 smithi187 ceph-mon[33032]: 2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-09-21T13:44:42.697 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:42 smithi187 ceph-mon[33032]: fsmap 2024-09-21T13:44:42.697 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:42 smithi187 ceph-mon[33032]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T13:44:42.697 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:42 smithi187 ceph-mon[33032]: mgrmap e14: y(active, since 45s) 2024-09-21T13:44:42.697 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:42 smithi187 ceph-mon[33032]: Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-09-21T13:44:42.697 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:42 smithi187 ceph-mon[33032]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-09-21T13:44:42.697 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:42 smithi187 ceph-mon[33032]: [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-09-21T13:44:42.697 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:42 smithi187 ceph-mon[33032]: mon.c (rank 2) addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] is down (out of quorum) 2024-09-21T13:44:42.697 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:42 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:42.697 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:42 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:42.698 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:42 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:42.698 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:42 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:42.701 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:44:42.701 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":3,"fsid":"40a3d2e8-781f-11ef-baf6-efdc52797490","modified":"2024-09-21T13:44:37.153888Z","created":"2024-09-21T13:42:51.046146Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:3300","nonce":0},{"type":"v1","addr":"172.21.15.49:6789","nonce":0}]},"addr":"172.21.15.49:6789/0","public_addr":"172.21.15.49:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:3300","nonce":0},{"type":"v1","addr":"172.21.15.187:6789","nonce":0}]},"addr":"172.21.15.187:6789/0","public_addr":"172.21.15.187:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:3301","nonce":0},{"type":"v1","addr":"172.21.15.49:6790","nonce":0}]},"addr":"172.21.15.49:6790/0","public_addr":"172.21.15.49:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-21T13:44:42.701 INFO:teuthology.orchestra.run.smithi187.stderr:dumped monmap epoch 3 2024-09-21T13:44:43.322 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:43 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:44:43.322 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:43 smithi187 ceph-mon[33032]: from='client.? 172.21.15.187:0/3399547578' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T13:44:43.323 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:43 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T13:44:43.350 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-21T13:44:43.350 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph config generate-minimal-conf 2024-09-21T13:44:43.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:43 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:44:43.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:43 smithi049 ceph-mon[30080]: from='client.? 172.21.15.187:0/3399547578' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T13:44:43.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:43 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T13:44:43.616 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:44:44.339 INFO:teuthology.orchestra.run.smithi049.stdout:# minimal ceph.conf for 40a3d2e8-781f-11ef-baf6-efdc52797490 2024-09-21T13:44:44.339 INFO:teuthology.orchestra.run.smithi049.stdout:[global] 2024-09-21T13:44:44.339 INFO:teuthology.orchestra.run.smithi049.stdout: fsid = 40a3d2e8-781f-11ef-baf6-efdc52797490 2024-09-21T13:44:44.339 INFO:teuthology.orchestra.run.smithi049.stdout: mon_host = [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] 2024-09-21T13:44:44.427 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:44 smithi049 ceph-mon[40489]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:44.427 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:44 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T13:44:44.427 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:44 smithi049 ceph-mon[40489]: mon.a calling monitor election 2024-09-21T13:44:44.427 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:44 smithi049 ceph-mon[40489]: mon.b calling monitor election 2024-09-21T13:44:44.428 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:44 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T13:44:44.428 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:44 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T13:44:44.428 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:44 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T13:44:44.428 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:44 smithi049 ceph-mon[40489]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:44.428 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:44 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T13:44:44.428 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:44 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T13:44:44.428 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:44 smithi049 ceph-mon[40489]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:44.428 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:44 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T13:44:44.428 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:44 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T13:44:44.428 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:44 smithi049 ceph-mon[40489]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-21T13:44:44.428 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:44 smithi049 ceph-mon[40489]: monmap epoch 3 2024-09-21T13:44:44.428 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:44 smithi049 ceph-mon[40489]: fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 2024-09-21T13:44:44.428 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:44 smithi049 ceph-mon[40489]: last_changed 2024-09-21T13:44:37.153888+0000 2024-09-21T13:44:44.428 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:44 smithi049 ceph-mon[40489]: created 2024-09-21T13:42:51.046146+0000 2024-09-21T13:44:44.429 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:44 smithi049 ceph-mon[40489]: min_mon_release 19 (squid) 2024-09-21T13:44:44.429 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:44 smithi049 ceph-mon[40489]: election_strategy: 1 2024-09-21T13:44:44.429 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:44 smithi049 ceph-mon[40489]: 0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-09-21T13:44:44.429 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:44 smithi049 ceph-mon[40489]: 1: [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] mon.b 2024-09-21T13:44:44.429 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:44 smithi049 ceph-mon[40489]: 2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-09-21T13:44:44.429 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:44 smithi049 ceph-mon[40489]: fsmap 2024-09-21T13:44:44.429 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:44 smithi049 ceph-mon[40489]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T13:44:44.429 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:44 smithi049 ceph-mon[40489]: mgrmap e14: y(active, since 45s) 2024-09-21T13:44:44.429 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:44 smithi049 ceph-mon[40489]: Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-09-21T13:44:44.429 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:44 smithi049 ceph-mon[40489]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-09-21T13:44:44.429 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:44 smithi049 ceph-mon[40489]: [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-09-21T13:44:44.429 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:44 smithi049 ceph-mon[40489]: mon.c (rank 2) addr [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] is down (out of quorum) 2024-09-21T13:44:44.429 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:44 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:44.429 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:44 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:44.429 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:44 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:44.430 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:44 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:44.430 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:44 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:44:44.430 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:44 smithi049 ceph-mon[40489]: from='client.? 172.21.15.187:0/3399547578' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T13:44:44.430 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:44 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T13:44:45.110 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:45 smithi049 ceph-mon[30080]: mon.c calling monitor election 2024-09-21T13:44:45.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:45 smithi049 ceph-mon[30080]: Updating smithi049:/etc/ceph/ceph.conf 2024-09-21T13:44:45.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:45 smithi049 ceph-mon[30080]: Updating smithi187:/etc/ceph/ceph.conf 2024-09-21T13:44:45.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:45 smithi049 ceph-mon[30080]: Updating smithi187:/var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T13:44:45.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:45 smithi049 ceph-mon[30080]: Updating smithi049:/var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T13:44:45.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:45 smithi049 ceph-mon[30080]: Reconfiguring mon.a (unknown last config time)... 2024-09-21T13:44:45.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:45 smithi049 ceph-mon[30080]: Reconfiguring daemon mon.a on smithi049 2024-09-21T13:44:45.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:45 smithi049 ceph-mon[30080]: mon.c calling monitor election 2024-09-21T13:44:45.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:45 smithi049 ceph-mon[30080]: mon.b calling monitor election 2024-09-21T13:44:45.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:45 smithi049 ceph-mon[30080]: mon.a calling monitor election 2024-09-21T13:44:45.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:45 smithi049 ceph-mon[30080]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T13:44:45.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:45 smithi049 ceph-mon[30080]: monmap epoch 3 2024-09-21T13:44:45.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:45 smithi049 ceph-mon[30080]: fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 2024-09-21T13:44:45.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:45 smithi049 ceph-mon[30080]: last_changed 2024-09-21T13:44:37.153888+0000 2024-09-21T13:44:45.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:45 smithi049 ceph-mon[30080]: created 2024-09-21T13:42:51.046146+0000 2024-09-21T13:44:45.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:45 smithi049 ceph-mon[30080]: min_mon_release 19 (squid) 2024-09-21T13:44:45.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:45 smithi049 ceph-mon[30080]: election_strategy: 1 2024-09-21T13:44:45.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:45 smithi049 ceph-mon[30080]: 0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-09-21T13:44:45.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:45 smithi049 ceph-mon[30080]: 1: [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] mon.b 2024-09-21T13:44:45.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:45 smithi049 ceph-mon[30080]: 2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-09-21T13:44:45.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:45 smithi049 ceph-mon[30080]: fsmap 2024-09-21T13:44:45.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:45 smithi049 ceph-mon[30080]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T13:44:45.127 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:45 smithi049 ceph-mon[30080]: mgrmap e14: y(active, since 47s) 2024-09-21T13:44:45.127 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:45 smithi049 ceph-mon[30080]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-09-21T13:44:45.127 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:45 smithi049 ceph-mon[30080]: Cluster is now healthy 2024-09-21T13:44:45.127 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:45 smithi049 ceph-mon[30080]: overall HEALTH_OK 2024-09-21T13:44:45.127 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:45 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/559674013' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:44:45.127 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:45 smithi049 ceph-mon[40489]: mon.c calling monitor election 2024-09-21T13:44:45.127 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:45 smithi049 ceph-mon[40489]: Updating smithi049:/etc/ceph/ceph.conf 2024-09-21T13:44:45.127 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:45 smithi049 ceph-mon[40489]: Updating smithi187:/etc/ceph/ceph.conf 2024-09-21T13:44:45.127 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:45 smithi049 ceph-mon[40489]: Updating smithi187:/var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T13:44:45.127 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:45 smithi049 ceph-mon[40489]: Updating smithi049:/var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T13:44:45.127 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:45 smithi049 ceph-mon[40489]: Reconfiguring mon.a (unknown last config time)... 2024-09-21T13:44:45.127 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:45 smithi049 ceph-mon[40489]: Reconfiguring daemon mon.a on smithi049 2024-09-21T13:44:45.128 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:45 smithi049 ceph-mon[40489]: mon.c calling monitor election 2024-09-21T13:44:45.128 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:45 smithi049 ceph-mon[40489]: mon.b calling monitor election 2024-09-21T13:44:45.128 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:45 smithi049 ceph-mon[40489]: mon.a calling monitor election 2024-09-21T13:44:45.128 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:45 smithi049 ceph-mon[40489]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T13:44:45.128 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:45 smithi049 ceph-mon[40489]: monmap epoch 3 2024-09-21T13:44:45.128 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:45 smithi049 ceph-mon[40489]: fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 2024-09-21T13:44:45.128 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:45 smithi049 ceph-mon[40489]: last_changed 2024-09-21T13:44:37.153888+0000 2024-09-21T13:44:45.128 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:45 smithi049 ceph-mon[40489]: created 2024-09-21T13:42:51.046146+0000 2024-09-21T13:44:45.128 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:45 smithi049 ceph-mon[40489]: min_mon_release 19 (squid) 2024-09-21T13:44:45.128 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:45 smithi049 ceph-mon[40489]: election_strategy: 1 2024-09-21T13:44:45.128 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:45 smithi049 ceph-mon[40489]: 0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-09-21T13:44:45.128 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:45 smithi049 ceph-mon[40489]: 1: [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] mon.b 2024-09-21T13:44:45.128 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:45 smithi049 ceph-mon[40489]: 2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-09-21T13:44:45.128 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:45 smithi049 ceph-mon[40489]: fsmap 2024-09-21T13:44:45.128 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:45 smithi049 ceph-mon[40489]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T13:44:45.129 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:45 smithi049 ceph-mon[40489]: mgrmap e14: y(active, since 47s) 2024-09-21T13:44:45.129 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:45 smithi049 ceph-mon[40489]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-09-21T13:44:45.129 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:45 smithi049 ceph-mon[40489]: Cluster is now healthy 2024-09-21T13:44:45.129 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:45 smithi049 ceph-mon[40489]: overall HEALTH_OK 2024-09-21T13:44:45.129 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:45 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/559674013' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:44:45.136 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-21T13:44:45.137 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-09-21T13:44:45.137 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/etc/ceph/ceph.conf 2024-09-21T13:44:45.165 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-09-21T13:44:45.165 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-21T13:44:45.232 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-09-21T13:44:45.232 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/ceph/ceph.conf 2024-09-21T13:44:45.265 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-09-21T13:44:45.265 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-21T13:44:45.332 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:45 smithi187 ceph-mon[33032]: mon.c calling monitor election 2024-09-21T13:44:45.332 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:45 smithi187 ceph-mon[33032]: Updating smithi049:/etc/ceph/ceph.conf 2024-09-21T13:44:45.332 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:45 smithi187 ceph-mon[33032]: Updating smithi187:/etc/ceph/ceph.conf 2024-09-21T13:44:45.332 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:45 smithi187 ceph-mon[33032]: Updating smithi187:/var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T13:44:45.332 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:45 smithi187 ceph-mon[33032]: Updating smithi049:/var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T13:44:45.332 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:45 smithi187 ceph-mon[33032]: Reconfiguring mon.a (unknown last config time)... 2024-09-21T13:44:45.332 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:45 smithi187 ceph-mon[33032]: Reconfiguring daemon mon.a on smithi049 2024-09-21T13:44:45.332 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:45 smithi187 ceph-mon[33032]: mon.c calling monitor election 2024-09-21T13:44:45.332 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:45 smithi187 ceph-mon[33032]: mon.b calling monitor election 2024-09-21T13:44:45.332 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:45 smithi187 ceph-mon[33032]: mon.a calling monitor election 2024-09-21T13:44:45.333 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:45 smithi187 ceph-mon[33032]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T13:44:45.333 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:45 smithi187 ceph-mon[33032]: monmap epoch 3 2024-09-21T13:44:45.333 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:45 smithi187 ceph-mon[33032]: fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 2024-09-21T13:44:45.333 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:45 smithi187 ceph-mon[33032]: last_changed 2024-09-21T13:44:37.153888+0000 2024-09-21T13:44:45.333 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:45 smithi187 ceph-mon[33032]: created 2024-09-21T13:42:51.046146+0000 2024-09-21T13:44:45.333 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:45 smithi187 ceph-mon[33032]: min_mon_release 19 (squid) 2024-09-21T13:44:45.333 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:45 smithi187 ceph-mon[33032]: election_strategy: 1 2024-09-21T13:44:45.333 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:45 smithi187 ceph-mon[33032]: 0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-09-21T13:44:45.333 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:45 smithi187 ceph-mon[33032]: 1: [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] mon.b 2024-09-21T13:44:45.333 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:45 smithi187 ceph-mon[33032]: 2: [v2:172.21.15.49:3301/0,v1:172.21.15.49:6790/0] mon.c 2024-09-21T13:44:45.333 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:45 smithi187 ceph-mon[33032]: fsmap 2024-09-21T13:44:45.333 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:45 smithi187 ceph-mon[33032]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T13:44:45.333 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:45 smithi187 ceph-mon[33032]: mgrmap e14: y(active, since 47s) 2024-09-21T13:44:45.333 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:45 smithi187 ceph-mon[33032]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-09-21T13:44:45.333 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:45 smithi187 ceph-mon[33032]: Cluster is now healthy 2024-09-21T13:44:45.334 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:45 smithi187 ceph-mon[33032]: overall HEALTH_OK 2024-09-21T13:44:45.334 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:45 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/559674013' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:44:45.336 INFO:tasks.cephadm:Adding mgr.y on smithi049 2024-09-21T13:44:45.336 INFO:tasks.cephadm:Adding mgr.x on smithi187 2024-09-21T13:44:45.336 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph orch apply mgr '2;smithi049=y;smithi187=x' 2024-09-21T13:44:45.623 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:44:46.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:46 smithi049 ceph-mon[30080]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:46.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:46 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T13:44:46.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:46 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:46.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:46 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:46.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:46 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T13:44:46.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:46 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T13:44:46.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:46 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:44:46.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:46 smithi049 ceph-mon[40489]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:46.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:46 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T13:44:46.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:46 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:46.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:46 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:46.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:46 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T13:44:46.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:46 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T13:44:46.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:46 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:44:46.291 INFO:teuthology.orchestra.run.smithi187.stdout:Scheduled mgr update... 2024-09-21T13:44:46.292 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:46 smithi187 ceph-mon[33032]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:46.292 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:46 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T13:44:46.292 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:46 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:46.292 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:46 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:46.292 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:46 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T13:44:46.292 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:46 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T13:44:46.292 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:46 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:44:46.901 DEBUG:teuthology.orchestra.run.smithi187:mgr.x> sudo journalctl -f -n 0 -u ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@mgr.x.service 2024-09-21T13:44:46.904 DEBUG:tasks.cephadm:set 0 configs 2024-09-21T13:44:46.904 INFO:tasks.cephadm:Deploying OSDs... 2024-09-21T13:44:46.904 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-09-21T13:44:46.904 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/scratch_devs of=/dev/stdout 2024-09-21T13:44:46.921 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-21T13:44:46.922 DEBUG:teuthology.orchestra.run.smithi049:> stat /dev/vg_nvme/lv_1 2024-09-21T13:44:46.978 INFO:teuthology.orchestra.run.smithi049.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-21T13:44:46.978 INFO:teuthology.orchestra.run.smithi049.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T13:44:46.979 INFO:teuthology.orchestra.run.smithi049.stdout:Device: 5h/5d Inode: 973 Links: 1 2024-09-21T13:44:46.979 INFO:teuthology.orchestra.run.smithi049.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T13:44:46.979 INFO:teuthology.orchestra.run.smithi049.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T13:44:46.979 INFO:teuthology.orchestra.run.smithi049.stdout:Access: 2024-09-21 13:44:36.565425736 +0000 2024-09-21T13:44:46.979 INFO:teuthology.orchestra.run.smithi049.stdout:Modify: 2024-09-21 13:39:24.316868441 +0000 2024-09-21T13:44:46.979 INFO:teuthology.orchestra.run.smithi049.stdout:Change: 2024-09-21 13:39:24.316868441 +0000 2024-09-21T13:44:46.979 INFO:teuthology.orchestra.run.smithi049.stdout: Birth: 2024-09-21 13:39:24.316868441 +0000 2024-09-21T13:44:46.980 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-21T13:44:47.044 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records in 2024-09-21T13:44:47.044 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records out 2024-09-21T13:44:47.044 INFO:teuthology.orchestra.run.smithi049.stderr:512 bytes copied, 0.000112989 s, 4.5 MB/s 2024-09-21T13:44:47.045 DEBUG:teuthology.orchestra.run.smithi049:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-21T13:44:47.103 DEBUG:teuthology.orchestra.run.smithi049:> stat /dev/vg_nvme/lv_2 2024-09-21T13:44:47.159 INFO:teuthology.orchestra.run.smithi049.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-21T13:44:47.159 INFO:teuthology.orchestra.run.smithi049.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T13:44:47.159 INFO:teuthology.orchestra.run.smithi049.stdout:Device: 5h/5d Inode: 971 Links: 1 2024-09-21T13:44:47.159 INFO:teuthology.orchestra.run.smithi049.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T13:44:47.159 INFO:teuthology.orchestra.run.smithi049.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T13:44:47.159 INFO:teuthology.orchestra.run.smithi049.stdout:Access: 2024-09-21 13:44:36.565425736 +0000 2024-09-21T13:44:47.159 INFO:teuthology.orchestra.run.smithi049.stdout:Modify: 2024-09-21 13:39:24.316868441 +0000 2024-09-21T13:44:47.159 INFO:teuthology.orchestra.run.smithi049.stdout:Change: 2024-09-21 13:39:24.316868441 +0000 2024-09-21T13:44:47.160 INFO:teuthology.orchestra.run.smithi049.stdout: Birth: 2024-09-21 13:39:24.316868441 +0000 2024-09-21T13:44:47.160 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-21T13:44:47.224 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records in 2024-09-21T13:44:47.224 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records out 2024-09-21T13:44:47.224 INFO:teuthology.orchestra.run.smithi049.stderr:512 bytes copied, 0.000115143 s, 4.4 MB/s 2024-09-21T13:44:47.225 DEBUG:teuthology.orchestra.run.smithi049:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-21T13:44:47.281 DEBUG:teuthology.orchestra.run.smithi049:> stat /dev/vg_nvme/lv_3 2024-09-21T13:44:47.339 INFO:teuthology.orchestra.run.smithi049.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-21T13:44:47.339 INFO:teuthology.orchestra.run.smithi049.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T13:44:47.339 INFO:teuthology.orchestra.run.smithi049.stdout:Device: 5h/5d Inode: 965 Links: 1 2024-09-21T13:44:47.339 INFO:teuthology.orchestra.run.smithi049.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T13:44:47.339 INFO:teuthology.orchestra.run.smithi049.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T13:44:47.339 INFO:teuthology.orchestra.run.smithi049.stdout:Access: 2024-09-21 13:44:36.565425736 +0000 2024-09-21T13:44:47.339 INFO:teuthology.orchestra.run.smithi049.stdout:Modify: 2024-09-21 13:39:24.316868441 +0000 2024-09-21T13:44:47.339 INFO:teuthology.orchestra.run.smithi049.stdout:Change: 2024-09-21 13:39:24.316868441 +0000 2024-09-21T13:44:47.339 INFO:teuthology.orchestra.run.smithi049.stdout: Birth: 2024-09-21 13:39:24.316868441 +0000 2024-09-21T13:44:47.340 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-21T13:44:47.405 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records in 2024-09-21T13:44:47.405 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records out 2024-09-21T13:44:47.405 INFO:teuthology.orchestra.run.smithi049.stderr:512 bytes copied, 9.4958e-05 s, 5.4 MB/s 2024-09-21T13:44:47.407 DEBUG:teuthology.orchestra.run.smithi049:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-21T13:44:47.464 DEBUG:teuthology.orchestra.run.smithi049:> stat /dev/vg_nvme/lv_4 2024-09-21T13:44:47.520 INFO:teuthology.orchestra.run.smithi049.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-21T13:44:47.520 INFO:teuthology.orchestra.run.smithi049.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T13:44:47.520 INFO:teuthology.orchestra.run.smithi049.stdout:Device: 5h/5d Inode: 978 Links: 1 2024-09-21T13:44:47.520 INFO:teuthology.orchestra.run.smithi049.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T13:44:47.520 INFO:teuthology.orchestra.run.smithi049.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T13:44:47.520 INFO:teuthology.orchestra.run.smithi049.stdout:Access: 2024-09-21 13:44:36.566425709 +0000 2024-09-21T13:44:47.520 INFO:teuthology.orchestra.run.smithi049.stdout:Modify: 2024-09-21 13:39:24.316868441 +0000 2024-09-21T13:44:47.520 INFO:teuthology.orchestra.run.smithi049.stdout:Change: 2024-09-21 13:39:24.316868441 +0000 2024-09-21T13:44:47.521 INFO:teuthology.orchestra.run.smithi049.stdout: Birth: 2024-09-21 13:39:24.316868441 +0000 2024-09-21T13:44:47.521 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-21T13:44:47.582 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:47 smithi049 ceph-mon[30080]: Reconfiguring mon.c (monmap changed)... 2024-09-21T13:44:47.582 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:47 smithi049 ceph-mon[30080]: Reconfiguring daemon mon.c on smithi049 2024-09-21T13:44:47.582 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:47 smithi049 ceph-mon[30080]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi049=y;smithi187=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:44:47.582 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:47 smithi049 ceph-mon[30080]: Saving service mgr spec with placement smithi049=y;smithi187=x;count:2 2024-09-21T13:44:47.582 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:47 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:47.582 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:47 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:47.582 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:47 smithi049 ceph-mon[40489]: Reconfiguring mon.c (monmap changed)... 2024-09-21T13:44:47.583 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:47 smithi049 ceph-mon[40489]: Reconfiguring daemon mon.c on smithi049 2024-09-21T13:44:47.583 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:47 smithi049 ceph-mon[40489]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi049=y;smithi187=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:44:47.583 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:47 smithi049 ceph-mon[40489]: Saving service mgr spec with placement smithi049=y;smithi187=x;count:2 2024-09-21T13:44:47.583 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:47 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:47.583 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:47 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:47.584 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records in 2024-09-21T13:44:47.584 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records out 2024-09-21T13:44:47.584 INFO:teuthology.orchestra.run.smithi049.stderr:512 bytes copied, 0.00010632 s, 4.8 MB/s 2024-09-21T13:44:47.585 DEBUG:teuthology.orchestra.run.smithi049:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-21T13:44:47.642 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-09-21T13:44:47.643 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/scratch_devs of=/dev/stdout 2024-09-21T13:44:47.659 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-21T13:44:47.659 DEBUG:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_1 2024-09-21T13:44:47.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:47 smithi187 ceph-mon[33032]: Reconfiguring mon.c (monmap changed)... 2024-09-21T13:44:47.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:47 smithi187 ceph-mon[33032]: Reconfiguring daemon mon.c on smithi049 2024-09-21T13:44:47.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:47 smithi187 ceph-mon[33032]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi049=y;smithi187=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:44:47.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:47 smithi187 ceph-mon[33032]: Saving service mgr spec with placement smithi049=y;smithi187=x;count:2 2024-09-21T13:44:47.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:47 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:47.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:47 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:47.710 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-21T13:44:47.710 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T13:44:47.710 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 5h/5d Inode: 960 Links: 1 2024-09-21T13:44:47.710 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T13:44:47.710 INFO:teuthology.orchestra.run.smithi187.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T13:44:47.710 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2024-09-21 13:44:28.122081657 +0000 2024-09-21T13:44:47.710 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2024-09-21 13:39:25.489601112 +0000 2024-09-21T13:44:47.710 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2024-09-21 13:39:25.489601112 +0000 2024-09-21T13:44:47.711 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: 2024-09-21 13:39:25.489601112 +0000 2024-09-21T13:44:47.711 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-21T13:44:47.774 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2024-09-21T13:44:47.774 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2024-09-21T13:44:47.775 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.000212328 s, 2.4 MB/s 2024-09-21T13:44:47.776 DEBUG:teuthology.orchestra.run.smithi187:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-21T13:44:47.831 DEBUG:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_2 2024-09-21T13:44:47.887 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-21T13:44:47.887 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T13:44:47.887 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 5h/5d Inode: 962 Links: 1 2024-09-21T13:44:47.888 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T13:44:47.888 INFO:teuthology.orchestra.run.smithi187.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T13:44:47.888 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2024-09-21 13:44:28.123081642 +0000 2024-09-21T13:44:47.888 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2024-09-21 13:39:25.489601112 +0000 2024-09-21T13:44:47.888 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2024-09-21 13:39:25.489601112 +0000 2024-09-21T13:44:47.888 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: 2024-09-21 13:39:25.489601112 +0000 2024-09-21T13:44:47.889 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-21T13:44:47.952 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2024-09-21T13:44:47.953 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2024-09-21T13:44:47.953 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.000197885 s, 2.6 MB/s 2024-09-21T13:44:47.954 DEBUG:teuthology.orchestra.run.smithi187:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-21T13:44:48.010 DEBUG:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_3 2024-09-21T13:44:48.066 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-21T13:44:48.066 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T13:44:48.066 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 5h/5d Inode: 979 Links: 1 2024-09-21T13:44:48.067 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T13:44:48.067 INFO:teuthology.orchestra.run.smithi187.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T13:44:48.067 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2024-09-21 13:44:28.123081642 +0000 2024-09-21T13:44:48.067 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2024-09-21 13:39:25.490601097 +0000 2024-09-21T13:44:48.067 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2024-09-21 13:39:25.490601097 +0000 2024-09-21T13:44:48.067 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: 2024-09-21 13:39:25.490601097 +0000 2024-09-21T13:44:48.067 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-21T13:44:48.129 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2024-09-21T13:44:48.129 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2024-09-21T13:44:48.129 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.000221224 s, 2.3 MB/s 2024-09-21T13:44:48.130 DEBUG:teuthology.orchestra.run.smithi187:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-21T13:44:48.186 DEBUG:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_4 2024-09-21T13:44:48.241 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-21T13:44:48.241 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T13:44:48.241 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 5h/5d Inode: 982 Links: 1 2024-09-21T13:44:48.242 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T13:44:48.242 INFO:teuthology.orchestra.run.smithi187.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T13:44:48.242 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2024-09-21 13:44:28.123081642 +0000 2024-09-21T13:44:48.242 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2024-09-21 13:39:25.491601082 +0000 2024-09-21T13:44:48.242 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2024-09-21 13:39:25.491601082 +0000 2024-09-21T13:44:48.242 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: 2024-09-21 13:39:25.491601082 +0000 2024-09-21T13:44:48.242 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-21T13:44:48.305 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2024-09-21T13:44:48.305 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2024-09-21T13:44:48.305 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.000198841 s, 2.6 MB/s 2024-09-21T13:44:48.306 DEBUG:teuthology.orchestra.run.smithi187:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-21T13:44:48.362 INFO:tasks.cephadm:Deploying osd.0 on smithi049 with /dev/vg_nvme/lv_4... 2024-09-21T13:44:48.363 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_4 2024-09-21T13:44:48.625 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:44:48.674 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:48 smithi187 ceph-mon[33032]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:48.674 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:48 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:48.674 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:48 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:48.674 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:48 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T13:44:48.674 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:48 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T13:44:48.674 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:48 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:44:48.746 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:48 smithi049 ceph-mon[30080]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:48.747 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:48 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:48.747 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:48 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:48.747 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:48 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T13:44:48.747 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:48 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T13:44:48.747 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:48 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:44:48.747 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:48 smithi049 ceph-mon[40489]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:48.747 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:48 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:48.747 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:48 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:48.747 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:48 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T13:44:48.747 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:48 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T13:44:48.748 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:48 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:44:49.418 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:49 smithi187 ceph-mon[33032]: Reconfiguring mon.b (monmap changed)... 2024-09-21T13:44:49.419 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:49 smithi187 ceph-mon[33032]: Reconfiguring daemon mon.b on smithi187 2024-09-21T13:44:49.419 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:49 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:49.419 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:49 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:49.419 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:49 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:44:49.419 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:49 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:44:49.419 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:49 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:44:49.419 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:49 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:49.419 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:49 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T13:44:49.419 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:49 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-21T13:44:49.419 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:49 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T13:44:49.419 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:49 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:44:49.711 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:49 smithi049 ceph-mon[40489]: Reconfiguring mon.b (monmap changed)... 2024-09-21T13:44:49.711 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:49 smithi049 ceph-mon[40489]: Reconfiguring daemon mon.b on smithi187 2024-09-21T13:44:49.711 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:49 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:49.711 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:49 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:49.711 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:49 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:44:49.712 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:49 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:44:49.712 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:49 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:44:49.712 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:49 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:49.712 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:49 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T13:44:49.712 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:49 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-21T13:44:49.712 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:49 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T13:44:49.712 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:49 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:44:49.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:49 smithi049 ceph-mon[30080]: Reconfiguring mon.b (monmap changed)... 2024-09-21T13:44:49.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:49 smithi049 ceph-mon[30080]: Reconfiguring daemon mon.b on smithi187 2024-09-21T13:44:49.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:49 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:49.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:49 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:49.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:49 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:44:49.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:49 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:44:49.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:49 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:44:49.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:49 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:49.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:49 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T13:44:49.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:49 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-21T13:44:49.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:49 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T13:44:49.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:49 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:44:50.438 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:50 smithi187 ceph-mon[33032]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:50.438 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:50 smithi187 ceph-mon[33032]: Deploying daemon mgr.x on smithi187 2024-09-21T13:44:50.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:50 smithi049 ceph-mon[30080]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:50.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:50 smithi049 ceph-mon[30080]: Deploying daemon mgr.x on smithi187 2024-09-21T13:44:50.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:50 smithi049 ceph-mon[40489]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:50.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:50 smithi049 ceph-mon[40489]: Deploying daemon mgr.x on smithi187 2024-09-21T13:44:50.695 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Sep 21 13:44:50 smithi187 podman[34676]: 2024-09-21 13:44:50.649042134 +0000 UTC m=+0.113297212 container create 3c06c2000419200307e04376a168148d371d904454d105011517df8eab755906 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-x, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True) 2024-09-21T13:44:51.195 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Sep 21 13:44:50 smithi187 podman[34676]: 2024-09-21 13:44:50.868011123 +0000 UTC m=+0.332266203 container init 3c06c2000419200307e04376a168148d371d904454d105011517df8eab755906 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-x, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-21-0842-4417849, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T13:44:51.195 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Sep 21 13:44:50 smithi187 podman[34676]: 2024-09-21 13:44:50.872830731 +0000 UTC m=+0.337085807 container start 3c06c2000419200307e04376a168148d371d904454d105011517df8eab755906 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-x, ceph=True, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-21-0842-4417849) 2024-09-21T13:44:51.195 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Sep 21 13:44:50 smithi187 ceph-mgr[34714]: -- 172.21.15.187:0/2851109627 <== mon.1 v2:172.21.15.187:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555a660eb0e0 con 0x555a660c9000 2024-09-21T13:44:51.195 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Sep 21 13:44:50 smithi187 bash[34676]: 3c06c2000419200307e04376a168148d371d904454d105011517df8eab755906 2024-09-21T13:44:51.195 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Sep 21 13:44:50 smithi187 systemd[1]: Started Ceph mgr.x for 40a3d2e8-781f-11ef-baf6-efdc52797490. 2024-09-21T13:44:51.405 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:44:51.431 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi049:vg_nvme/lv_4 2024-09-21T13:44:51.685 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:44:52.121 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:52 smithi049 ceph-mon[30080]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:52.121 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:52 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:52.121 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:52 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:52.122 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:52 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:52.122 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:52 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:52.122 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:52 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:44:52.122 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:52 smithi049 ceph-mon[40489]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:52.122 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:52 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:52.122 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:52 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:52.122 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:52 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:52.122 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:52 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:52.123 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:52 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:44:52.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:52 smithi187 ceph-mon[33032]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:52.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:52 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:52.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:52 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:52.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:52 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:52.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:52 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:52.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:52 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:44:53.396 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:53 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:53.396 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:53 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:53.396 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:53 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:53.397 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:53 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:44:53.397 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:53 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:44:53.397 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:53 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:53.397 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:53 smithi049 ceph-mon[30080]: Reconfiguring mgr.y (unknown last config time)... 2024-09-21T13:44:53.397 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:53 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T13:44:53.397 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:53 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T13:44:53.397 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:53 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:44:53.397 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:53 smithi049 ceph-mon[30080]: Reconfiguring daemon mgr.y on smithi049 2024-09-21T13:44:53.397 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:53 smithi049 ceph-mon[30080]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:44:53.397 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:53 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T13:44:53.397 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:53 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T13:44:53.397 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:53 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:44:53.397 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:53 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:53.398 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:53 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:53.398 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:53 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:53.398 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:53 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:44:53.398 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:53 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:44:53.398 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:53 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:53.398 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:53 smithi049 ceph-mon[40489]: Reconfiguring mgr.y (unknown last config time)... 2024-09-21T13:44:53.398 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:53 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T13:44:53.398 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:53 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T13:44:53.398 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:53 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:44:53.398 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:53 smithi049 ceph-mon[40489]: Reconfiguring daemon mgr.y on smithi049 2024-09-21T13:44:53.399 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:53 smithi049 ceph-mon[40489]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:44:53.399 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:53 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T13:44:53.399 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:53 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T13:44:53.399 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:53 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:44:53.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:53 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:53.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:53 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:53.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:53 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:53.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:53 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:44:53.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:53 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:44:53.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:53 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:53.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:53 smithi187 ceph-mon[33032]: Reconfiguring mgr.y (unknown last config time)... 2024-09-21T13:44:53.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:53 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T13:44:53.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:53 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T13:44:53.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:53 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:44:53.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:53 smithi187 ceph-mon[33032]: Reconfiguring daemon mgr.y on smithi049 2024-09-21T13:44:53.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:53 smithi187 ceph-mon[33032]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:44:53.446 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:53 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T13:44:53.446 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:53 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T13:44:53.446 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:53 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:44:54.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:54 smithi187 ceph-mon[33032]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:54.501 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:54 smithi049 ceph-mon[30080]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:54.501 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:54 smithi049 ceph-mon[40489]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:55.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:55 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:55.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:55 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:55.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:55 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:44:55.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:55 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:44:55.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:55 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:44:55.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:55 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:55.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:55 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:55.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:55 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:55.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:55 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:44:55.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:55 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:44:55.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:55 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:44:55.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:55 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:55.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:55 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:55.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:55 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:55.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:55 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:44:55.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:55 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:44:55.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:55 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:44:55.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:55 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:44:56.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:56 smithi187 ceph-mon[33032]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:56.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:56 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3126466817' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "2595dca9-6219-4e0c-8728-751eb102e73a"} : dispatch 2024-09-21T13:44:56.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:56 smithi187 ceph-mon[33032]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "2595dca9-6219-4e0c-8728-751eb102e73a"} : dispatch 2024-09-21T13:44:56.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:56 smithi187 ceph-mon[33032]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2595dca9-6219-4e0c-8728-751eb102e73a"}]': finished 2024-09-21T13:44:56.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:56 smithi187 ceph-mon[33032]: osdmap e5: 1 total, 0 up, 1 in 2024-09-21T13:44:56.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:56 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T13:44:56.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:56 smithi187 ceph-mon[33032]: Standby manager daemon x started 2024-09-21T13:44:56.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:56 smithi187 ceph-mon[33032]: from='mgr.? 172.21.15.187:0/1694502207' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-21T13:44:56.446 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:56 smithi187 ceph-mon[33032]: from='mgr.? 172.21.15.187:0/1694502207' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-21T13:44:56.446 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:56 smithi187 ceph-mon[33032]: from='mgr.? 172.21.15.187:0/1694502207' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-21T13:44:56.446 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:56 smithi187 ceph-mon[33032]: from='mgr.? 172.21.15.187:0/1694502207' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-21T13:44:56.446 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:56 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2533734148' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T13:44:56.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:56 smithi049 ceph-mon[30080]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:56.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:56 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3126466817' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "2595dca9-6219-4e0c-8728-751eb102e73a"} : dispatch 2024-09-21T13:44:56.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:56 smithi049 ceph-mon[30080]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "2595dca9-6219-4e0c-8728-751eb102e73a"} : dispatch 2024-09-21T13:44:56.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:56 smithi049 ceph-mon[30080]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2595dca9-6219-4e0c-8728-751eb102e73a"}]': finished 2024-09-21T13:44:56.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:56 smithi049 ceph-mon[30080]: osdmap e5: 1 total, 0 up, 1 in 2024-09-21T13:44:56.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:56 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T13:44:56.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:56 smithi049 ceph-mon[30080]: Standby manager daemon x started 2024-09-21T13:44:56.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:56 smithi049 ceph-mon[30080]: from='mgr.? 172.21.15.187:0/1694502207' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-21T13:44:56.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:56 smithi049 ceph-mon[30080]: from='mgr.? 172.21.15.187:0/1694502207' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-21T13:44:56.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:56 smithi049 ceph-mon[30080]: from='mgr.? 172.21.15.187:0/1694502207' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-21T13:44:56.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:56 smithi049 ceph-mon[30080]: from='mgr.? 172.21.15.187:0/1694502207' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-21T13:44:56.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:56 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2533734148' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T13:44:56.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:56 smithi049 ceph-mon[40489]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:56.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:56 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3126466817' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "2595dca9-6219-4e0c-8728-751eb102e73a"} : dispatch 2024-09-21T13:44:56.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:56 smithi049 ceph-mon[40489]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "2595dca9-6219-4e0c-8728-751eb102e73a"} : dispatch 2024-09-21T13:44:56.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:56 smithi049 ceph-mon[40489]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2595dca9-6219-4e0c-8728-751eb102e73a"}]': finished 2024-09-21T13:44:56.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:56 smithi049 ceph-mon[40489]: osdmap e5: 1 total, 0 up, 1 in 2024-09-21T13:44:56.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:56 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T13:44:56.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:56 smithi049 ceph-mon[40489]: Standby manager daemon x started 2024-09-21T13:44:56.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:56 smithi049 ceph-mon[40489]: from='mgr.? 172.21.15.187:0/1694502207' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-21T13:44:56.507 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:56 smithi049 ceph-mon[40489]: from='mgr.? 172.21.15.187:0/1694502207' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-21T13:44:56.507 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:56 smithi049 ceph-mon[40489]: from='mgr.? 172.21.15.187:0/1694502207' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-21T13:44:56.507 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:56 smithi049 ceph-mon[40489]: from='mgr.? 172.21.15.187:0/1694502207' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-21T13:44:56.507 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:56 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2533734148' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T13:44:57.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:57 smithi187 ceph-mon[33032]: mgrmap e15: y(active, since 59s), standbys: x 2024-09-21T13:44:57.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:57 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-21T13:44:57.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:57 smithi187 ceph-mon[33032]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:58.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:57 smithi049 ceph-mon[40489]: mgrmap e15: y(active, since 59s), standbys: x 2024-09-21T13:44:58.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:57 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-21T13:44:58.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:57 smithi049 ceph-mon[40489]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:44:58.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:57 smithi049 ceph-mon[30080]: mgrmap e15: y(active, since 59s), standbys: x 2024-09-21T13:44:58.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:57 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-21T13:44:58.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:57 smithi049 ceph-mon[30080]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:45:00.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:44:59 smithi049 ceph-mon[40489]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:45:00.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:44:59 smithi049 ceph-mon[30080]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:45:00.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:44:59 smithi187 ceph-mon[33032]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:45:01.940 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:01 smithi049 ceph-mon[30080]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:45:01.941 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:01 smithi049 ceph-mon[40489]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:45:02.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:01 smithi187 ceph-mon[33032]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:45:04.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:03 smithi049 ceph-mon[30080]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:45:04.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:03 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-21T13:45:04.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:03 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:45:04.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:03 smithi049 ceph-mon[30080]: Deploying daemon osd.0 on smithi049 2024-09-21T13:45:04.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:03 smithi049 ceph-mon[40489]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:45:04.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:03 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-21T13:45:04.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:03 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:45:04.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:03 smithi049 ceph-mon[40489]: Deploying daemon osd.0 on smithi049 2024-09-21T13:45:04.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:03 smithi187 ceph-mon[33032]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:45:04.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:03 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-21T13:45:04.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:03 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:45:04.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:03 smithi187 ceph-mon[33032]: Deploying daemon osd.0 on smithi049 2024-09-21T13:45:05.885 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:05 smithi049 ceph-mon[40489]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:45:05.886 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:05 smithi049 ceph-mon[30080]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:45:06.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:05 smithi187 ceph-mon[33032]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:45:08.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:07 smithi049 ceph-mon[30080]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:45:08.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:07 smithi049 ceph-mon[40489]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:45:08.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:07 smithi187 ceph-mon[33032]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:45:09.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:09 smithi049 ceph-mon[30080]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:45:09.960 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:09 smithi049 ceph-mon[40489]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:45:10.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:09 smithi187 ceph-mon[33032]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:45:12.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:11 smithi049 ceph-mon[30080]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:45:12.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:11 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:45:12.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:11 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:12.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:11 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:12.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:11 smithi049 ceph-mon[40489]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:45:12.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:11 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:45:12.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:11 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:12.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:11 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:12.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:11 smithi187 ceph-mon[33032]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:45:12.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:11 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:45:12.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:11 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:12.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:11 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:13.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:13 smithi049 ceph-mon[30080]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:45:14.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:13 smithi049 ceph-mon[40489]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:45:14.181 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:13 smithi187 ceph-mon[33032]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:45:15.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:14 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:15.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:14 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:15.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:14 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:45:15.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:14 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:45:15.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:14 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:15.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:14 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:15.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:14 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:15.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:14 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:45:15.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:14 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:45:15.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:14 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:15.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:14 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:15.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:14 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:15.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:14 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:45:15.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:14 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:45:15.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:14 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:16.439 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:16 smithi049 ceph-mon[30080]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:45:16.439 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:16 smithi049 ceph-mon[30080]: from='osd.0 [v2:172.21.15.49:6802/2095561412,v1:172.21.15.49:6803/2095561412]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T13:45:16.439 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:16 smithi049 ceph-mon[30080]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T13:45:16.440 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:16 smithi049 ceph-mon[40489]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:45:16.440 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:16 smithi049 ceph-mon[40489]: from='osd.0 [v2:172.21.15.49:6802/2095561412,v1:172.21.15.49:6803/2095561412]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T13:45:16.440 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:16 smithi049 ceph-mon[40489]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T13:45:16.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:16 smithi187 ceph-mon[33032]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:45:16.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:16 smithi187 ceph-mon[33032]: from='osd.0 [v2:172.21.15.49:6802/2095561412,v1:172.21.15.49:6803/2095561412]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T13:45:16.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:16 smithi187 ceph-mon[33032]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T13:45:16.754 INFO:teuthology.orchestra.run.smithi049.stdout:Created osd(s) 0 on host 'smithi049' 2024-09-21T13:45:17.354 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:17 smithi049 ceph-mon[40489]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T13:45:17.354 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:17 smithi049 ceph-mon[40489]: osdmap e6: 1 total, 0 up, 1 in 2024-09-21T13:45:17.354 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:17 smithi049 ceph-mon[40489]: from='osd.0 [v2:172.21.15.49:6802/2095561412,v1:172.21.15.49:6803/2095561412]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi049", "root=default"]} : dispatch 2024-09-21T13:45:17.354 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:17 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T13:45:17.354 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:17 smithi049 ceph-mon[40489]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi049", "root=default"]} : dispatch 2024-09-21T13:45:17.354 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:17 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:45:17.354 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:17 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:17.354 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:17 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:17.354 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:17 smithi049 ceph-mon[40489]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2024-09-21T13:45:17.354 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:17 smithi049 ceph-mon[40489]: osdmap e7: 1 total, 0 up, 1 in 2024-09-21T13:45:17.355 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:17 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T13:45:17.355 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:17 smithi049 ceph-mon[30080]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T13:45:17.355 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:17 smithi049 ceph-mon[30080]: osdmap e6: 1 total, 0 up, 1 in 2024-09-21T13:45:17.355 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:17 smithi049 ceph-mon[30080]: from='osd.0 [v2:172.21.15.49:6802/2095561412,v1:172.21.15.49:6803/2095561412]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi049", "root=default"]} : dispatch 2024-09-21T13:45:17.355 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:17 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T13:45:17.355 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:17 smithi049 ceph-mon[30080]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi049", "root=default"]} : dispatch 2024-09-21T13:45:17.355 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:17 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:45:17.355 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:17 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:17.355 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:17 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:17.355 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:17 smithi049 ceph-mon[30080]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2024-09-21T13:45:17.355 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:17 smithi049 ceph-mon[30080]: osdmap e7: 1 total, 0 up, 1 in 2024-09-21T13:45:17.356 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:17 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T13:45:17.356 DEBUG:teuthology.orchestra.run.smithi049:osd.0> sudo journalctl -f -n 0 -u ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@osd.0.service 2024-09-21T13:45:17.359 INFO:tasks.cephadm:Deploying osd.1 on smithi049 with /dev/vg_nvme/lv_3... 2024-09-21T13:45:17.359 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_3 2024-09-21T13:45:17.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:17 smithi187 ceph-mon[33032]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T13:45:17.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:17 smithi187 ceph-mon[33032]: osdmap e6: 1 total, 0 up, 1 in 2024-09-21T13:45:17.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:17 smithi187 ceph-mon[33032]: from='osd.0 [v2:172.21.15.49:6802/2095561412,v1:172.21.15.49:6803/2095561412]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi049", "root=default"]} : dispatch 2024-09-21T13:45:17.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:17 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T13:45:17.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:17 smithi187 ceph-mon[33032]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi049", "root=default"]} : dispatch 2024-09-21T13:45:17.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:17 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:45:17.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:17 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:17.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:17 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:17.446 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:17 smithi187 ceph-mon[33032]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2024-09-21T13:45:17.446 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:17 smithi187 ceph-mon[33032]: osdmap e7: 1 total, 0 up, 1 in 2024-09-21T13:45:17.446 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:17 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T13:45:18.174 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:45:18.424 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:18 smithi049 ceph-mon[40489]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:45:18.424 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:18 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T13:45:18.425 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:18 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T13:45:18.425 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:18 smithi049 ceph-mon[30080]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:45:18.425 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:18 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T13:45:18.425 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:18 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T13:45:18.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:18 smithi187 ceph-mon[33032]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:45:18.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:18 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T13:45:18.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:18 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T13:45:19.415 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:19 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:45:19.415 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:19 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:45:19.415 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:19 smithi049 ceph-mon[30080]: osd.0 [v2:172.21.15.49:6802/2095561412,v1:172.21.15.49:6803/2095561412] boot 2024-09-21T13:45:19.415 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:19 smithi049 ceph-mon[30080]: osdmap e8: 1 total, 1 up, 1 in 2024-09-21T13:45:19.415 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:19 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T13:45:19.416 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:19 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:45:19.416 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:19 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:45:19.416 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:19 smithi049 ceph-mon[40489]: osd.0 [v2:172.21.15.49:6802/2095561412,v1:172.21.15.49:6803/2095561412] boot 2024-09-21T13:45:19.416 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:19 smithi049 ceph-mon[40489]: osdmap e8: 1 total, 1 up, 1 in 2024-09-21T13:45:19.416 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:19 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T13:45:19.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:19 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:45:19.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:19 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:45:19.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:19 smithi187 ceph-mon[33032]: osd.0 [v2:172.21.15.49:6802/2095561412,v1:172.21.15.49:6803/2095561412] boot 2024-09-21T13:45:19.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:19 smithi187 ceph-mon[33032]: osdmap e8: 1 total, 1 up, 1 in 2024-09-21T13:45:19.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:19 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T13:45:20.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:20 smithi187 ceph-mon[33032]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:45:20.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:20 smithi187 ceph-mon[33032]: osdmap e9: 1 total, 1 up, 1 in 2024-09-21T13:45:20.455 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:20 smithi049 ceph-mon[40489]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:45:20.455 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:20 smithi049 ceph-mon[40489]: osdmap e9: 1 total, 1 up, 1 in 2024-09-21T13:45:20.455 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:20 smithi049 ceph-mon[30080]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T13:45:20.455 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:20 smithi049 ceph-mon[30080]: osdmap e9: 1 total, 1 up, 1 in 2024-09-21T13:45:22.154 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:21 smithi049 ceph-mon[30080]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T13:45:22.155 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:21 smithi049 ceph-mon[30080]: Detected new or changed devices on smithi049 2024-09-21T13:45:22.155 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:21 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:22.155 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:21 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:22.155 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:21 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T13:45:22.155 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:21 smithi049 ceph-mon[30080]: Adjusting osd_memory_target on smithi049 to 16064M 2024-09-21T13:45:22.155 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:21 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:22.155 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:21 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:45:22.155 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:21 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:45:22.155 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:21 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:22.156 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:21 smithi049 ceph-mon[40489]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T13:45:22.156 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:21 smithi049 ceph-mon[40489]: Detected new or changed devices on smithi049 2024-09-21T13:45:22.156 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:21 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:22.156 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:21 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:22.156 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:21 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T13:45:22.156 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:21 smithi049 ceph-mon[40489]: Adjusting osd_memory_target on smithi049 to 16064M 2024-09-21T13:45:22.156 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:21 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:22.156 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:21 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:45:22.156 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:21 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:45:22.156 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:21 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:22.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:21 smithi187 ceph-mon[33032]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T13:45:22.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:21 smithi187 ceph-mon[33032]: Detected new or changed devices on smithi049 2024-09-21T13:45:22.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:21 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:22.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:21 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:22.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:21 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T13:45:22.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:21 smithi187 ceph-mon[33032]: Adjusting osd_memory_target on smithi049 to 16064M 2024-09-21T13:45:22.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:21 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:22.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:21 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:45:22.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:21 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:45:22.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:21 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:22.433 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:45:22.459 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi049:vg_nvme/lv_3 2024-09-21T13:45:22.713 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:45:24.044 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:23 smithi049 ceph-mon[30080]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T13:45:24.044 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:23 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T13:45:24.044 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:23 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T13:45:24.044 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:23 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:45:24.044 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:23 smithi049 ceph-mon[40489]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T13:45:24.044 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:23 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T13:45:24.045 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:23 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T13:45:24.045 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:23 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:45:24.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:23 smithi187 ceph-mon[33032]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T13:45:24.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:23 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T13:45:24.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:23 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T13:45:24.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:23 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:45:25.042 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:24 smithi049 ceph-mon[30080]: from='client.24137 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:45:25.043 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:24 smithi049 ceph-mon[40489]: from='client.24137 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:45:25.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:24 smithi187 ceph-mon[33032]: from='client.24137 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:45:26.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:25 smithi187 ceph-mon[33032]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T13:45:26.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:25 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1227919841' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "0ba327f9-5524-4a40-abcc-729ded0b1917"} : dispatch 2024-09-21T13:45:26.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:25 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1227919841' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0ba327f9-5524-4a40-abcc-729ded0b1917"}]': finished 2024-09-21T13:45:26.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:25 smithi187 ceph-mon[33032]: osdmap e10: 2 total, 1 up, 2 in 2024-09-21T13:45:26.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:25 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T13:45:26.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:25 smithi049 ceph-mon[40489]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T13:45:26.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:25 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1227919841' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "0ba327f9-5524-4a40-abcc-729ded0b1917"} : dispatch 2024-09-21T13:45:26.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:25 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1227919841' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0ba327f9-5524-4a40-abcc-729ded0b1917"}]': finished 2024-09-21T13:45:26.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:25 smithi049 ceph-mon[40489]: osdmap e10: 2 total, 1 up, 2 in 2024-09-21T13:45:26.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:25 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T13:45:26.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:25 smithi049 ceph-mon[30080]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T13:45:26.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:25 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1227919841' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "0ba327f9-5524-4a40-abcc-729ded0b1917"} : dispatch 2024-09-21T13:45:26.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:25 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1227919841' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0ba327f9-5524-4a40-abcc-729ded0b1917"}]': finished 2024-09-21T13:45:26.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:25 smithi049 ceph-mon[30080]: osdmap e10: 2 total, 1 up, 2 in 2024-09-21T13:45:26.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:25 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T13:45:27.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:26 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3232594211' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T13:45:27.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:26 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3232594211' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T13:45:27.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:26 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3232594211' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T13:45:28.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:27 smithi187 ceph-mon[33032]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T13:45:28.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:27 smithi049 ceph-mon[30080]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T13:45:28.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:27 smithi049 ceph-mon[40489]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T13:45:30.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:29 smithi187 ceph-mon[33032]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T13:45:30.198 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:29 smithi049 ceph-mon[30080]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T13:45:30.199 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:29 smithi049 ceph-mon[40489]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T13:45:31.880 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:31 smithi049 ceph-mon[40489]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T13:45:31.880 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:31 smithi049 ceph-mon[30080]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T13:45:32.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:31 smithi187 ceph-mon[33032]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T13:45:34.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:33 smithi049 ceph-mon[30080]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T13:45:34.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:33 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-21T13:45:34.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:33 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:45:34.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:33 smithi049 ceph-mon[30080]: Deploying daemon osd.1 on smithi049 2024-09-21T13:45:34.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:33 smithi049 ceph-mon[40489]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T13:45:34.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:33 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-21T13:45:34.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:33 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:45:34.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:33 smithi049 ceph-mon[40489]: Deploying daemon osd.1 on smithi049 2024-09-21T13:45:34.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:33 smithi187 ceph-mon[33032]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T13:45:34.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:33 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-21T13:45:34.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:33 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:45:34.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:33 smithi187 ceph-mon[33032]: Deploying daemon osd.1 on smithi049 2024-09-21T13:45:35.949 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:35 smithi049 ceph-mon[40489]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T13:45:35.950 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:35 smithi049 ceph-mon[30080]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T13:45:36.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:35 smithi187 ceph-mon[33032]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T13:45:38.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:37 smithi187 ceph-mon[33032]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T13:45:38.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:37 smithi049 ceph-mon[30080]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T13:45:38.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:37 smithi049 ceph-mon[40489]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T13:45:39.892 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:39 smithi049 ceph-mon[30080]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T13:45:39.892 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:39 smithi049 ceph-mon[40489]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T13:45:40.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:39 smithi187 ceph-mon[33032]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T13:45:41.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:40 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:45:41.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:40 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:41.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:40 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:41.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:40 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:45:41.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:40 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:41.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:40 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:41.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:40 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:45:41.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:40 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:41.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:40 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:42.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:41 smithi187 ceph-mon[33032]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T13:45:42.195 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:41 smithi049 ceph-mon[40489]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T13:45:42.195 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:41 smithi049 ceph-mon[30080]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T13:45:44.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:43 smithi049 ceph-mon[40489]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T13:45:44.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:43 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:44.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:43 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:44.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:43 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:45:44.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:43 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:45:44.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:43 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:44.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:43 smithi049 ceph-mon[30080]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T13:45:44.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:43 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:44.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:43 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:44.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:43 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:45:44.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:43 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:45:44.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:43 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:44.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:43 smithi187 ceph-mon[33032]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T13:45:44.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:43 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:44.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:43 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:44.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:43 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:45:44.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:43 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:45:44.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:43 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:46.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:45 smithi049 ceph-mon[30080]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T13:45:46.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:45 smithi049 ceph-mon[30080]: from='osd.1 [v2:172.21.15.49:6810/1382840527,v1:172.21.15.49:6811/1382840527]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-21T13:45:46.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:45 smithi049 ceph-mon[40489]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T13:45:46.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:45 smithi049 ceph-mon[40489]: from='osd.1 [v2:172.21.15.49:6810/1382840527,v1:172.21.15.49:6811/1382840527]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-21T13:45:46.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:45 smithi187 ceph-mon[33032]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T13:45:46.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:45 smithi187 ceph-mon[33032]: from='osd.1 [v2:172.21.15.49:6810/1382840527,v1:172.21.15.49:6811/1382840527]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-21T13:45:46.209 INFO:teuthology.orchestra.run.smithi049.stdout:Created osd(s) 1 on host 'smithi049' 2024-09-21T13:45:46.849 DEBUG:teuthology.orchestra.run.smithi049:osd.1> sudo journalctl -f -n 0 -u ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@osd.1.service 2024-09-21T13:45:46.856 INFO:tasks.cephadm:Deploying osd.2 on smithi049 with /dev/vg_nvme/lv_2... 2024-09-21T13:45:46.856 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_2 2024-09-21T13:45:47.071 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:46 smithi049 ceph-mon[40489]: from='osd.1 [v2:172.21.15.49:6810/1382840527,v1:172.21.15.49:6811/1382840527]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-21T13:45:47.071 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:46 smithi049 ceph-mon[40489]: osdmap e11: 2 total, 1 up, 2 in 2024-09-21T13:45:47.071 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:46 smithi049 ceph-mon[40489]: from='osd.1 [v2:172.21.15.49:6810/1382840527,v1:172.21.15.49:6811/1382840527]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi049", "root=default"]} : dispatch 2024-09-21T13:45:47.071 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:46 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T13:45:47.072 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:46 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:45:47.072 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:46 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:47.072 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:46 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:47.072 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:46 smithi049 ceph-mon[40489]: from='osd.1 [v2:172.21.15.49:6810/1382840527,v1:172.21.15.49:6811/1382840527]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2024-09-21T13:45:47.072 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:46 smithi049 ceph-mon[30080]: from='osd.1 [v2:172.21.15.49:6810/1382840527,v1:172.21.15.49:6811/1382840527]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-21T13:45:47.072 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:46 smithi049 ceph-mon[30080]: osdmap e11: 2 total, 1 up, 2 in 2024-09-21T13:45:47.072 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:46 smithi049 ceph-mon[30080]: from='osd.1 [v2:172.21.15.49:6810/1382840527,v1:172.21.15.49:6811/1382840527]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi049", "root=default"]} : dispatch 2024-09-21T13:45:47.072 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:46 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T13:45:47.073 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:46 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:45:47.073 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:46 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:47.073 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:46 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:47.073 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:46 smithi049 ceph-mon[30080]: from='osd.1 [v2:172.21.15.49:6810/1382840527,v1:172.21.15.49:6811/1382840527]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2024-09-21T13:45:47.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:46 smithi187 ceph-mon[33032]: from='osd.1 [v2:172.21.15.49:6810/1382840527,v1:172.21.15.49:6811/1382840527]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-21T13:45:47.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:46 smithi187 ceph-mon[33032]: osdmap e11: 2 total, 1 up, 2 in 2024-09-21T13:45:47.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:46 smithi187 ceph-mon[33032]: from='osd.1 [v2:172.21.15.49:6810/1382840527,v1:172.21.15.49:6811/1382840527]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi049", "root=default"]} : dispatch 2024-09-21T13:45:47.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:46 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T13:45:47.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:46 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:45:47.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:46 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:47.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:46 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:47.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:46 smithi187 ceph-mon[33032]: from='osd.1 [v2:172.21.15.49:6810/1382840527,v1:172.21.15.49:6811/1382840527]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2024-09-21T13:45:47.396 INFO:journalctl@ceph.osd.1.smithi049.stdout:Sep 21 13:45:47 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-1[49970]: 2024-09-21T13:45:47.069+0000 7fe19c679640 -1 osd.1 0 waiting for initial osdmap 2024-09-21T13:45:47.396 INFO:journalctl@ceph.osd.1.smithi049.stdout:Sep 21 13:45:47 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-1[49970]: 2024-09-21T13:45:47.072+0000 7fe194498640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T13:45:47.596 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:45:48.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:47 smithi049 ceph-mon[30080]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T13:45:48.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:47 smithi049 ceph-mon[30080]: osdmap e12: 2 total, 1 up, 2 in 2024-09-21T13:45:48.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:47 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T13:45:48.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:47 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T13:45:48.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:47 smithi049 ceph-mon[30080]: from='osd.1 [v2:172.21.15.49:6810/1382840527,v1:172.21.15.49:6811/1382840527]' entity='osd.1' 2024-09-21T13:45:48.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:47 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T13:45:48.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:47 smithi049 ceph-mon[40489]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T13:45:48.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:47 smithi049 ceph-mon[40489]: osdmap e12: 2 total, 1 up, 2 in 2024-09-21T13:45:48.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:47 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T13:45:48.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:47 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T13:45:48.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:47 smithi049 ceph-mon[40489]: from='osd.1 [v2:172.21.15.49:6810/1382840527,v1:172.21.15.49:6811/1382840527]' entity='osd.1' 2024-09-21T13:45:48.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:47 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T13:45:48.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:47 smithi187 ceph-mon[33032]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T13:45:48.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:47 smithi187 ceph-mon[33032]: osdmap e12: 2 total, 1 up, 2 in 2024-09-21T13:45:48.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:47 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T13:45:48.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:47 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T13:45:48.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:47 smithi187 ceph-mon[33032]: from='osd.1 [v2:172.21.15.49:6810/1382840527,v1:172.21.15.49:6811/1382840527]' entity='osd.1' 2024-09-21T13:45:48.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:47 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T13:45:49.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:49 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:45:49.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:49 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:45:49.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:49 smithi049 ceph-mon[30080]: osd.1 [v2:172.21.15.49:6810/1382840527,v1:172.21.15.49:6811/1382840527] boot 2024-09-21T13:45:49.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:49 smithi049 ceph-mon[30080]: osdmap e13: 2 total, 2 up, 2 in 2024-09-21T13:45:49.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:49 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T13:45:49.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:49 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:45:49.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:49 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:45:49.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:49 smithi049 ceph-mon[40489]: osd.1 [v2:172.21.15.49:6810/1382840527,v1:172.21.15.49:6811/1382840527] boot 2024-09-21T13:45:49.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:49 smithi049 ceph-mon[40489]: osdmap e13: 2 total, 2 up, 2 in 2024-09-21T13:45:49.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:49 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T13:45:49.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:49 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:45:49.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:49 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:45:49.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:49 smithi187 ceph-mon[33032]: osd.1 [v2:172.21.15.49:6810/1382840527,v1:172.21.15.49:6811/1382840527] boot 2024-09-21T13:45:49.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:49 smithi187 ceph-mon[33032]: osdmap e13: 2 total, 2 up, 2 in 2024-09-21T13:45:49.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:49 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T13:45:50.421 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:50 smithi049 ceph-mon[40489]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T13:45:50.421 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:50 smithi049 ceph-mon[40489]: osdmap e14: 2 total, 2 up, 2 in 2024-09-21T13:45:50.421 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:50 smithi049 ceph-mon[30080]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T13:45:50.421 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:50 smithi049 ceph-mon[30080]: osdmap e14: 2 total, 2 up, 2 in 2024-09-21T13:45:50.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:50 smithi187 ceph-mon[33032]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T13:45:50.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:50 smithi187 ceph-mon[33032]: osdmap e14: 2 total, 2 up, 2 in 2024-09-21T13:45:52.154 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:51 smithi049 ceph-mon[40489]: pgmap v61: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-21T13:45:52.154 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:51 smithi049 ceph-mon[40489]: Detected new or changed devices on smithi049 2024-09-21T13:45:52.154 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:51 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:52.154 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:51 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:52.154 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:51 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T13:45:52.154 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:51 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T13:45:52.154 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:51 smithi049 ceph-mon[40489]: Adjusting osd_memory_target on smithi049 to 8032M 2024-09-21T13:45:52.154 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:51 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:52.155 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:51 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:45:52.155 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:51 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:45:52.155 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:51 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:52.155 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:51 smithi049 ceph-mon[30080]: pgmap v61: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-21T13:45:52.155 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:51 smithi049 ceph-mon[30080]: Detected new or changed devices on smithi049 2024-09-21T13:45:52.155 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:51 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:52.155 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:51 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:52.155 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:51 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T13:45:52.155 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:51 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T13:45:52.155 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:51 smithi049 ceph-mon[30080]: Adjusting osd_memory_target on smithi049 to 8032M 2024-09-21T13:45:52.155 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:51 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:52.155 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:51 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:45:52.155 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:51 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:45:52.155 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:51 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:52.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:51 smithi187 ceph-mon[33032]: pgmap v61: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-21T13:45:52.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:51 smithi187 ceph-mon[33032]: Detected new or changed devices on smithi049 2024-09-21T13:45:52.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:51 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:52.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:51 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:52.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:51 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T13:45:52.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:51 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T13:45:52.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:51 smithi187 ceph-mon[33032]: Adjusting osd_memory_target on smithi049 to 8032M 2024-09-21T13:45:52.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:51 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:52.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:51 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:45:52.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:51 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:45:52.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:51 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:45:52.449 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:45:52.478 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi049:vg_nvme/lv_2 2024-09-21T13:45:52.734 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:45:54.034 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:53 smithi049 ceph-mon[30080]: pgmap v62: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-21T13:45:54.034 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:53 smithi049 ceph-mon[30080]: from='client.24164 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:45:54.035 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:53 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T13:45:54.035 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:53 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T13:45:54.035 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:53 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:45:54.035 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:53 smithi049 ceph-mon[40489]: pgmap v62: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-21T13:45:54.035 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:53 smithi049 ceph-mon[40489]: from='client.24164 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:45:54.035 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:53 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T13:45:54.035 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:53 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T13:45:54.035 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:53 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:45:54.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:53 smithi187 ceph-mon[33032]: pgmap v62: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-21T13:45:54.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:53 smithi187 ceph-mon[33032]: from='client.24164 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:45:54.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:53 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T13:45:54.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:53 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T13:45:54.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:53 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:45:56.070 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:55 smithi049 ceph-mon[30080]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-21T13:45:56.071 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:55 smithi049 ceph-mon[40489]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-21T13:45:56.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:55 smithi187 ceph-mon[33032]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-21T13:45:57.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:56 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/203214379' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "27a45750-204d-48fd-bb8f-ebb01273343d"} : dispatch 2024-09-21T13:45:57.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:56 smithi187 ceph-mon[33032]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "27a45750-204d-48fd-bb8f-ebb01273343d"} : dispatch 2024-09-21T13:45:57.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:56 smithi187 ceph-mon[33032]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "27a45750-204d-48fd-bb8f-ebb01273343d"}]': finished 2024-09-21T13:45:57.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:56 smithi187 ceph-mon[33032]: osdmap e15: 3 total, 2 up, 3 in 2024-09-21T13:45:57.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:56 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T13:45:57.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:56 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1074687583' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T13:45:57.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:56 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/203214379' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "27a45750-204d-48fd-bb8f-ebb01273343d"} : dispatch 2024-09-21T13:45:57.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:56 smithi049 ceph-mon[30080]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "27a45750-204d-48fd-bb8f-ebb01273343d"} : dispatch 2024-09-21T13:45:57.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:56 smithi049 ceph-mon[30080]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "27a45750-204d-48fd-bb8f-ebb01273343d"}]': finished 2024-09-21T13:45:57.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:56 smithi049 ceph-mon[30080]: osdmap e15: 3 total, 2 up, 3 in 2024-09-21T13:45:57.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:56 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T13:45:57.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:56 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1074687583' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T13:45:57.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:56 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/203214379' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "27a45750-204d-48fd-bb8f-ebb01273343d"} : dispatch 2024-09-21T13:45:57.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:56 smithi049 ceph-mon[40489]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "27a45750-204d-48fd-bb8f-ebb01273343d"} : dispatch 2024-09-21T13:45:57.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:56 smithi049 ceph-mon[40489]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "27a45750-204d-48fd-bb8f-ebb01273343d"}]': finished 2024-09-21T13:45:57.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:56 smithi049 ceph-mon[40489]: osdmap e15: 3 total, 2 up, 3 in 2024-09-21T13:45:57.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:56 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T13:45:57.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:56 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1074687583' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T13:45:58.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:57 smithi187 ceph-mon[33032]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-21T13:45:58.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:57 smithi049 ceph-mon[40489]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-21T13:45:58.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:57 smithi049 ceph-mon[30080]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-21T13:46:00.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:45:59 smithi049 ceph-mon[30080]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-21T13:46:00.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:45:59 smithi049 ceph-mon[40489]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-21T13:46:00.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:45:59 smithi187 ceph-mon[33032]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-21T13:46:02.188 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:01 smithi049 ceph-mon[30080]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-21T13:46:02.189 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:01 smithi049 ceph-mon[40489]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-21T13:46:02.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:01 smithi187 ceph-mon[33032]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-21T13:46:04.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:03 smithi049 ceph-mon[30080]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-21T13:46:04.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:03 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-21T13:46:04.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:03 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:46:04.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:03 smithi049 ceph-mon[30080]: Deploying daemon osd.2 on smithi049 2024-09-21T13:46:04.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:03 smithi049 ceph-mon[40489]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-21T13:46:04.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:03 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-21T13:46:04.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:03 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:46:04.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:03 smithi049 ceph-mon[40489]: Deploying daemon osd.2 on smithi049 2024-09-21T13:46:04.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:03 smithi187 ceph-mon[33032]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-21T13:46:04.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:03 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-21T13:46:04.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:03 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:46:04.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:03 smithi187 ceph-mon[33032]: Deploying daemon osd.2 on smithi049 2024-09-21T13:46:05.953 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:05 smithi049 ceph-mon[30080]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-21T13:46:05.953 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:05 smithi049 ceph-mon[40489]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-21T13:46:06.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:05 smithi187 ceph-mon[33032]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-21T13:46:08.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:07 smithi187 ceph-mon[33032]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-21T13:46:08.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:07 smithi049 ceph-mon[30080]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-21T13:46:08.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:07 smithi049 ceph-mon[40489]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-21T13:46:10.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:09 smithi187 ceph-mon[33032]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-21T13:46:10.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:09 smithi049 ceph-mon[30080]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-21T13:46:10.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:09 smithi049 ceph-mon[40489]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-21T13:46:12.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:11 smithi187 ceph-mon[33032]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-21T13:46:12.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:11 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:46:12.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:11 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:12.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:11 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:12.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:11 smithi049 ceph-mon[30080]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-21T13:46:12.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:11 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:46:12.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:11 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:12.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:11 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:12.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:11 smithi049 ceph-mon[40489]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-21T13:46:12.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:11 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:46:12.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:11 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:12.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:11 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:14.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:13 smithi187 ceph-mon[33032]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-21T13:46:14.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:13 smithi049 ceph-mon[30080]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-21T13:46:14.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:13 smithi049 ceph-mon[40489]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-21T13:46:15.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:14 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:15.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:14 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:15.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:14 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:46:15.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:14 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:46:15.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:14 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:15.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:14 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:15.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:14 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:15.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:14 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:46:15.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:14 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:46:15.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:14 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:15.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:14 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:15.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:14 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:15.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:14 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:46:15.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:14 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:46:15.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:14 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:15.997 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:15 smithi049 ceph-mon[40489]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-21T13:46:15.998 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:15 smithi049 ceph-mon[30080]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-21T13:46:16.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:15 smithi187 ceph-mon[33032]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-21T13:46:16.356 INFO:teuthology.orchestra.run.smithi049.stdout:Created osd(s) 2 on host 'smithi049' 2024-09-21T13:46:16.927 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:16 smithi049 ceph-mon[40489]: from='osd.2 [v2:172.21.15.49:6818/2189046898,v1:172.21.15.49:6819/2189046898]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T13:46:16.927 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:16 smithi049 ceph-mon[40489]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T13:46:16.927 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:16 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:46:16.927 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:16 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:16.927 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:16 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:16.927 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:16 smithi049 ceph-mon[30080]: from='osd.2 [v2:172.21.15.49:6818/2189046898,v1:172.21.15.49:6819/2189046898]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T13:46:16.927 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:16 smithi049 ceph-mon[30080]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T13:46:16.928 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:16 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:46:16.928 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:16 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:16.928 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:16 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:16.957 DEBUG:teuthology.orchestra.run.smithi049:osd.2> sudo journalctl -f -n 0 -u ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@osd.2.service 2024-09-21T13:46:16.959 INFO:tasks.cephadm:Deploying osd.3 on smithi049 with /dev/vg_nvme/lv_1... 2024-09-21T13:46:16.959 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_1 2024-09-21T13:46:17.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:16 smithi187 ceph-mon[33032]: from='osd.2 [v2:172.21.15.49:6818/2189046898,v1:172.21.15.49:6819/2189046898]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T13:46:17.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:16 smithi187 ceph-mon[33032]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T13:46:17.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:16 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:46:17.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:16 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:17.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:16 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:17.714 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:46:18.158 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:17 smithi049 ceph-mon[40489]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-21T13:46:18.158 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:17 smithi049 ceph-mon[40489]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T13:46:18.158 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:17 smithi049 ceph-mon[40489]: osdmap e16: 3 total, 2 up, 3 in 2024-09-21T13:46:18.158 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:17 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T13:46:18.158 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:17 smithi049 ceph-mon[40489]: from='osd.2 [v2:172.21.15.49:6818/2189046898,v1:172.21.15.49:6819/2189046898]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi049", "root=default"]} : dispatch 2024-09-21T13:46:18.158 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:17 smithi049 ceph-mon[40489]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi049", "root=default"]} : dispatch 2024-09-21T13:46:18.159 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:17 smithi049 ceph-mon[30080]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-21T13:46:18.159 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:17 smithi049 ceph-mon[30080]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T13:46:18.159 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:17 smithi049 ceph-mon[30080]: osdmap e16: 3 total, 2 up, 3 in 2024-09-21T13:46:18.159 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:17 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T13:46:18.159 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:17 smithi049 ceph-mon[30080]: from='osd.2 [v2:172.21.15.49:6818/2189046898,v1:172.21.15.49:6819/2189046898]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi049", "root=default"]} : dispatch 2024-09-21T13:46:18.159 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:17 smithi049 ceph-mon[30080]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi049", "root=default"]} : dispatch 2024-09-21T13:46:18.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:17 smithi187 ceph-mon[33032]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-21T13:46:18.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:17 smithi187 ceph-mon[33032]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T13:46:18.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:17 smithi187 ceph-mon[33032]: osdmap e16: 3 total, 2 up, 3 in 2024-09-21T13:46:18.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:17 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T13:46:18.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:17 smithi187 ceph-mon[33032]: from='osd.2 [v2:172.21.15.49:6818/2189046898,v1:172.21.15.49:6819/2189046898]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi049", "root=default"]} : dispatch 2024-09-21T13:46:18.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:17 smithi187 ceph-mon[33032]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi049", "root=default"]} : dispatch 2024-09-21T13:46:18.470 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:46:18 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-2[55757]: 2024-09-21T13:46:18.155+0000 7f636f766640 -1 osd.2 0 waiting for initial osdmap 2024-09-21T13:46:18.470 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:46:18 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-2[55757]: 2024-09-21T13:46:18.158+0000 7f6367585640 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T13:46:19.082 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:18 smithi049 ceph-mon[30080]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2024-09-21T13:46:19.082 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:18 smithi049 ceph-mon[30080]: osdmap e17: 3 total, 2 up, 3 in 2024-09-21T13:46:19.082 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:18 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T13:46:19.082 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:18 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T13:46:19.082 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:18 smithi049 ceph-mon[30080]: from='osd.2 ' entity='osd.2' 2024-09-21T13:46:19.082 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:18 smithi049 ceph-mon[40489]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2024-09-21T13:46:19.082 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:18 smithi049 ceph-mon[40489]: osdmap e17: 3 total, 2 up, 3 in 2024-09-21T13:46:19.082 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:18 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T13:46:19.082 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:18 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T13:46:19.083 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:18 smithi049 ceph-mon[40489]: from='osd.2 ' entity='osd.2' 2024-09-21T13:46:19.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:18 smithi187 ceph-mon[33032]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2024-09-21T13:46:19.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:18 smithi187 ceph-mon[33032]: osdmap e17: 3 total, 2 up, 3 in 2024-09-21T13:46:19.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:18 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T13:46:19.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:18 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T13:46:19.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:18 smithi187 ceph-mon[33032]: from='osd.2 ' entity='osd.2' 2024-09-21T13:46:20.155 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:19 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:46:20.155 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:19 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:46:20.155 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:19 smithi049 ceph-mon[40489]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-21T13:46:20.155 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:19 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T13:46:20.155 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:19 smithi049 ceph-mon[40489]: osd.2 [v2:172.21.15.49:6818/2189046898,v1:172.21.15.49:6819/2189046898] boot 2024-09-21T13:46:20.155 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:19 smithi049 ceph-mon[40489]: osdmap e18: 3 total, 3 up, 3 in 2024-09-21T13:46:20.156 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:19 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T13:46:20.156 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:19 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:46:20.156 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:19 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:46:20.156 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:19 smithi049 ceph-mon[30080]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-21T13:46:20.156 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:19 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T13:46:20.157 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:19 smithi049 ceph-mon[30080]: osd.2 [v2:172.21.15.49:6818/2189046898,v1:172.21.15.49:6819/2189046898] boot 2024-09-21T13:46:20.157 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:19 smithi049 ceph-mon[30080]: osdmap e18: 3 total, 3 up, 3 in 2024-09-21T13:46:20.157 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:19 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T13:46:20.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:19 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:46:20.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:19 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:46:20.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:19 smithi187 ceph-mon[33032]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-21T13:46:20.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:19 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T13:46:20.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:19 smithi187 ceph-mon[33032]: osd.2 [v2:172.21.15.49:6818/2189046898,v1:172.21.15.49:6819/2189046898] boot 2024-09-21T13:46:20.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:19 smithi187 ceph-mon[33032]: osdmap e18: 3 total, 3 up, 3 in 2024-09-21T13:46:20.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:19 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T13:46:21.171 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:21 smithi049 ceph-mon[30080]: osdmap e19: 3 total, 3 up, 3 in 2024-09-21T13:46:21.172 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:21 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:21.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:21 smithi187 ceph-mon[33032]: osdmap e19: 3 total, 3 up, 3 in 2024-09-21T13:46:21.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:21 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:21.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:21 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:21.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:21 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T13:46:21.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:21 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T13:46:21.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:21 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T13:46:21.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:21 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:21.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:21 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:46:21.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:21 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:46:21.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:21 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:21.446 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:21 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-21T13:46:21.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:21 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:21.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:21 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T13:46:21.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:21 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T13:46:21.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:21 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T13:46:21.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:21 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:21.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:21 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:46:21.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:21 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:46:21.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:21 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:21.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:21 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-21T13:46:21.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:21 smithi049 ceph-mon[40489]: osdmap e19: 3 total, 3 up, 3 in 2024-09-21T13:46:21.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:21 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:21.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:21 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:21.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:21 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T13:46:21.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:21 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T13:46:21.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:21 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T13:46:21.507 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:21 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:21.507 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:21 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:46:21.507 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:21 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:46:21.507 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:21 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:21.507 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:21 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-21T13:46:22.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:22 smithi049 ceph-mon[30080]: pgmap v81: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-21T13:46:22.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:22 smithi049 ceph-mon[30080]: Detected new or changed devices on smithi049 2024-09-21T13:46:22.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:22 smithi049 ceph-mon[30080]: Adjusting osd_memory_target on smithi049 to 5354M 2024-09-21T13:46:22.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:22 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-21T13:46:22.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:22 smithi049 ceph-mon[30080]: osdmap e20: 3 total, 3 up, 3 in 2024-09-21T13:46:22.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:22 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-21T13:46:22.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:22 smithi049 ceph-mon[40489]: pgmap v81: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-21T13:46:22.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:22 smithi049 ceph-mon[40489]: Detected new or changed devices on smithi049 2024-09-21T13:46:22.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:22 smithi049 ceph-mon[40489]: Adjusting osd_memory_target on smithi049 to 5354M 2024-09-21T13:46:22.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:22 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-21T13:46:22.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:22 smithi049 ceph-mon[40489]: osdmap e20: 3 total, 3 up, 3 in 2024-09-21T13:46:22.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:22 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-21T13:46:22.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:22 smithi187 ceph-mon[33032]: pgmap v81: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-21T13:46:22.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:22 smithi187 ceph-mon[33032]: Detected new or changed devices on smithi049 2024-09-21T13:46:22.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:22 smithi187 ceph-mon[33032]: Adjusting osd_memory_target on smithi049 to 5354M 2024-09-21T13:46:22.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:22 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-21T13:46:22.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:22 smithi187 ceph-mon[33032]: osdmap e20: 3 total, 3 up, 3 in 2024-09-21T13:46:22.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:22 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-21T13:46:22.641 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:46:22.669 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi049:vg_nvme/lv_1 2024-09-21T13:46:22.928 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:46:23.589 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:46:23 smithi049 sudo[59700]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T13:46:23.590 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:46:23 smithi049 sudo[59700]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T13:46:23.590 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:46:23 smithi049 sudo[59700]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T13:46:23.590 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:46:23 smithi049 sudo[59700]: pam_unix(sudo:session): session closed for user root 2024-09-21T13:46:23.590 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:46:23 smithi049 sudo[59704]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T13:46:23.590 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:46:23 smithi049 sudo[59704]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T13:46:23.590 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:46:23 smithi049 sudo[59704]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T13:46:23.590 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:46:23 smithi049 sudo[59704]: pam_unix(sudo:session): session closed for user root 2024-09-21T13:46:23.590 INFO:journalctl@ceph.osd.0.smithi049.stdout:Sep 21 13:46:23 smithi049 sudo[59688]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T13:46:23.590 INFO:journalctl@ceph.osd.0.smithi049.stdout:Sep 21 13:46:23 smithi049 sudo[59688]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T13:46:23.591 INFO:journalctl@ceph.osd.0.smithi049.stdout:Sep 21 13:46:23 smithi049 sudo[59688]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T13:46:23.591 INFO:journalctl@ceph.osd.0.smithi049.stdout:Sep 21 13:46:23 smithi049 sudo[59688]: pam_unix(sudo:session): session closed for user root 2024-09-21T13:46:23.591 INFO:journalctl@ceph.osd.0.smithi049.stdout:Sep 21 13:46:23 smithi049 sudo[59691]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T13:46:23.591 INFO:journalctl@ceph.osd.0.smithi049.stdout:Sep 21 13:46:23 smithi049 sudo[59691]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T13:46:23.591 INFO:journalctl@ceph.osd.0.smithi049.stdout:Sep 21 13:46:23 smithi049 sudo[59691]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T13:46:23.591 INFO:journalctl@ceph.osd.0.smithi049.stdout:Sep 21 13:46:23 smithi049 sudo[59691]: pam_unix(sudo:session): session closed for user root 2024-09-21T13:46:23.591 INFO:journalctl@ceph.osd.1.smithi049.stdout:Sep 21 13:46:23 smithi049 sudo[59694]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T13:46:23.591 INFO:journalctl@ceph.osd.1.smithi049.stdout:Sep 21 13:46:23 smithi049 sudo[59694]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T13:46:23.591 INFO:journalctl@ceph.osd.1.smithi049.stdout:Sep 21 13:46:23 smithi049 sudo[59694]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T13:46:23.591 INFO:journalctl@ceph.osd.1.smithi049.stdout:Sep 21 13:46:23 smithi049 sudo[59694]: pam_unix(sudo:session): session closed for user root 2024-09-21T13:46:23.591 INFO:journalctl@ceph.osd.1.smithi049.stdout:Sep 21 13:46:23 smithi049 sudo[59697]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T13:46:23.591 INFO:journalctl@ceph.osd.1.smithi049.stdout:Sep 21 13:46:23 smithi049 sudo[59697]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T13:46:23.591 INFO:journalctl@ceph.osd.1.smithi049.stdout:Sep 21 13:46:23 smithi049 sudo[59697]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T13:46:23.592 INFO:journalctl@ceph.osd.1.smithi049.stdout:Sep 21 13:46:23 smithi049 sudo[59697]: pam_unix(sudo:session): session closed for user root 2024-09-21T13:46:23.927 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:23 smithi049 sudo[59723]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T13:46:23.927 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:23 smithi049 sudo[59723]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T13:46:23.927 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:23 smithi049 sudo[59723]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T13:46:23.927 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:23 smithi049 sudo[59723]: pam_unix(sudo:session): session closed for user root 2024-09-21T13:46:23.927 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:23 smithi049 sudo[59744]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T13:46:23.927 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:23 smithi049 sudo[59744]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T13:46:23.928 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:23 smithi049 sudo[59744]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T13:46:23.928 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:23 smithi049 sudo[59744]: pam_unix(sudo:session): session closed for user root 2024-09-21T13:46:23.928 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:23 smithi049 sudo[59747]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T13:46:23.928 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:23 smithi049 sudo[59747]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T13:46:23.928 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:23 smithi049 sudo[59747]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T13:46:23.928 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:23 smithi049 sudo[59747]: pam_unix(sudo:session): session closed for user root 2024-09-21T13:46:23.929 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:23 smithi049 sudo[59750]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T13:46:23.929 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:23 smithi049 sudo[59750]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T13:46:23.929 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:23 smithi049 sudo[59750]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T13:46:23.929 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:23 smithi049 sudo[59750]: pam_unix(sudo:session): session closed for user root 2024-09-21T13:46:23.930 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:23 smithi187 sudo[35156]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T13:46:23.930 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:23 smithi187 sudo[35156]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T13:46:23.930 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:23 smithi187 sudo[35156]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T13:46:23.930 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:23 smithi187 sudo[35156]: pam_unix(sudo:session): session closed for user root 2024-09-21T13:46:23.930 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:23 smithi187 sudo[35159]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T13:46:23.930 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:23 smithi187 sudo[35159]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T13:46:23.931 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:23 smithi187 sudo[35159]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T13:46:23.931 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:23 smithi187 sudo[35159]: pam_unix(sudo:session): session closed for user root 2024-09-21T13:46:24.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:23 smithi187 ceph-mon[33032]: pgmap v83: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-21T13:46:24.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:23 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-21T13:46:24.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:23 smithi187 ceph-mon[33032]: osdmap e21: 3 total, 3 up, 3 in 2024-09-21T13:46:24.196 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:23 smithi187 ceph-mon[33032]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T13:46:24.196 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:23 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:24.196 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:23 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T13:46:24.196 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:23 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T13:46:24.196 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:23 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:46:24.196 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:23 smithi187 ceph-mon[33032]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T13:46:24.197 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:23 smithi187 ceph-mon[33032]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T13:46:24.197 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:23 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T13:46:24.197 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:23 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T13:46:24.197 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:23 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T13:46:24.197 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:23 smithi187 ceph-mon[33032]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T13:46:24.197 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:23 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T13:46:24.198 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:23 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T13:46:24.198 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:23 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T13:46:24.198 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:23 smithi187 ceph-mon[33032]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T13:46:24.198 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:23 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:24.198 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:23 smithi187 ceph-mon[33032]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T13:46:24.198 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:23 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T13:46:24.199 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:23 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T13:46:24.199 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:23 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T13:46:24.199 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:23 smithi187 ceph-mon[33032]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T13:46:24.220 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:23 smithi049 ceph-mon[40489]: pgmap v83: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-21T13:46:24.220 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:23 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-21T13:46:24.220 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:23 smithi049 ceph-mon[40489]: osdmap e21: 3 total, 3 up, 3 in 2024-09-21T13:46:24.220 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:23 smithi049 ceph-mon[40489]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T13:46:24.220 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:23 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:24.221 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:23 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T13:46:24.221 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:23 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T13:46:24.221 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:23 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:46:24.221 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:23 smithi049 ceph-mon[40489]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T13:46:24.221 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:23 smithi049 ceph-mon[40489]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T13:46:24.221 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:23 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T13:46:24.221 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:23 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T13:46:24.221 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:23 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T13:46:24.221 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:23 smithi049 ceph-mon[40489]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T13:46:24.221 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:23 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T13:46:24.221 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:23 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T13:46:24.221 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:23 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T13:46:24.221 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:23 smithi049 ceph-mon[40489]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T13:46:24.221 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:23 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:24.221 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:23 smithi049 ceph-mon[40489]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T13:46:24.222 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:23 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T13:46:24.222 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:23 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T13:46:24.222 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:23 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T13:46:24.222 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:23 smithi049 ceph-mon[40489]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T13:46:24.222 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:23 smithi049 ceph-mon[30080]: pgmap v83: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-21T13:46:24.222 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:23 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-21T13:46:24.222 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:23 smithi049 ceph-mon[30080]: osdmap e21: 3 total, 3 up, 3 in 2024-09-21T13:46:24.222 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:23 smithi049 ceph-mon[30080]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T13:46:24.222 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:23 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:24.222 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:23 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T13:46:24.222 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:23 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T13:46:24.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:23 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:46:24.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:23 smithi049 ceph-mon[30080]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T13:46:24.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:23 smithi049 ceph-mon[30080]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T13:46:24.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:23 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T13:46:24.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:23 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T13:46:24.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:23 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T13:46:24.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:23 smithi049 ceph-mon[30080]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T13:46:24.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:23 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T13:46:24.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:23 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T13:46:24.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:23 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T13:46:24.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:23 smithi049 ceph-mon[30080]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T13:46:24.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:23 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:24.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:23 smithi049 ceph-mon[30080]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T13:46:24.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:23 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T13:46:24.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:23 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T13:46:24.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:23 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T13:46:24.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:23 smithi049 ceph-mon[30080]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T13:46:25.215 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:25 smithi049 ceph-mon[40489]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:46:25.215 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:25 smithi049 ceph-mon[40489]: osdmap e22: 3 total, 3 up, 3 in 2024-09-21T13:46:25.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:25 smithi049 ceph-mon[30080]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:46:25.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:25 smithi049 ceph-mon[30080]: osdmap e22: 3 total, 3 up, 3 in 2024-09-21T13:46:25.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:25 smithi187 ceph-mon[33032]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:46:25.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:25 smithi187 ceph-mon[33032]: osdmap e22: 3 total, 3 up, 3 in 2024-09-21T13:46:26.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:26 smithi187 ceph-mon[33032]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-21T13:46:26.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:26 smithi187 ceph-mon[33032]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T13:46:26.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:26 smithi187 ceph-mon[33032]: Cluster is now healthy 2024-09-21T13:46:26.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:26 smithi187 ceph-mon[33032]: mgrmap e16: y(active, since 2m), standbys: x 2024-09-21T13:46:26.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:26 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2684882478' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "528bc390-3dc9-4f3b-ab75-d7412c9625db"} : dispatch 2024-09-21T13:46:26.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:26 smithi187 ceph-mon[33032]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "528bc390-3dc9-4f3b-ab75-d7412c9625db"} : dispatch 2024-09-21T13:46:26.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:26 smithi187 ceph-mon[33032]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "528bc390-3dc9-4f3b-ab75-d7412c9625db"}]': finished 2024-09-21T13:46:26.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:26 smithi187 ceph-mon[33032]: osdmap e23: 4 total, 3 up, 4 in 2024-09-21T13:46:26.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:26 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T13:46:26.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:26 smithi049 ceph-mon[30080]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-21T13:46:26.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:26 smithi049 ceph-mon[30080]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T13:46:26.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:26 smithi049 ceph-mon[30080]: Cluster is now healthy 2024-09-21T13:46:26.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:26 smithi049 ceph-mon[30080]: mgrmap e16: y(active, since 2m), standbys: x 2024-09-21T13:46:26.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:26 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2684882478' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "528bc390-3dc9-4f3b-ab75-d7412c9625db"} : dispatch 2024-09-21T13:46:26.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:26 smithi049 ceph-mon[30080]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "528bc390-3dc9-4f3b-ab75-d7412c9625db"} : dispatch 2024-09-21T13:46:26.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:26 smithi049 ceph-mon[30080]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "528bc390-3dc9-4f3b-ab75-d7412c9625db"}]': finished 2024-09-21T13:46:26.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:26 smithi049 ceph-mon[30080]: osdmap e23: 4 total, 3 up, 4 in 2024-09-21T13:46:26.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:26 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T13:46:26.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:26 smithi049 ceph-mon[40489]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-21T13:46:26.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:26 smithi049 ceph-mon[40489]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T13:46:26.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:26 smithi049 ceph-mon[40489]: Cluster is now healthy 2024-09-21T13:46:26.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:26 smithi049 ceph-mon[40489]: mgrmap e16: y(active, since 2m), standbys: x 2024-09-21T13:46:26.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:26 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2684882478' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "528bc390-3dc9-4f3b-ab75-d7412c9625db"} : dispatch 2024-09-21T13:46:26.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:26 smithi049 ceph-mon[40489]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "528bc390-3dc9-4f3b-ab75-d7412c9625db"} : dispatch 2024-09-21T13:46:26.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:26 smithi049 ceph-mon[40489]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "528bc390-3dc9-4f3b-ab75-d7412c9625db"}]': finished 2024-09-21T13:46:26.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:26 smithi049 ceph-mon[40489]: osdmap e23: 4 total, 3 up, 4 in 2024-09-21T13:46:26.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:26 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T13:46:27.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:27 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3643490509' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T13:46:27.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:27 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3643490509' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T13:46:27.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:27 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3643490509' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T13:46:28.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:28 smithi187 ceph-mon[33032]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-21T13:46:28.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:28 smithi049 ceph-mon[40489]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-21T13:46:28.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:28 smithi049 ceph-mon[30080]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-21T13:46:30.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:30 smithi187 ceph-mon[33032]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-21T13:46:30.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:30 smithi049 ceph-mon[30080]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-21T13:46:30.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:30 smithi049 ceph-mon[40489]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-21T13:46:32.432 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:32 smithi049 ceph-mon[40489]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-21T13:46:32.432 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:32 smithi049 ceph-mon[30080]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-21T13:46:32.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:32 smithi187 ceph-mon[33032]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-21T13:46:34.297 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:34 smithi049 ceph-mon[30080]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-21T13:46:34.297 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:34 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-21T13:46:34.297 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:34 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:46:34.298 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:34 smithi049 ceph-mon[40489]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-21T13:46:34.298 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:34 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-21T13:46:34.298 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:34 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:46:34.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:34 smithi187 ceph-mon[33032]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-21T13:46:34.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:34 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-21T13:46:34.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:34 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:46:35.454 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:35 smithi049 ceph-mon[40489]: Deploying daemon osd.3 on smithi049 2024-09-21T13:46:35.454 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:35 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:46:35.454 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:35 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:46:35.454 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:35 smithi049 ceph-mon[30080]: Deploying daemon osd.3 on smithi049 2024-09-21T13:46:35.455 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:35 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:46:35.455 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:35 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:46:35.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:35 smithi187 ceph-mon[33032]: Deploying daemon osd.3 on smithi049 2024-09-21T13:46:35.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:35 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:46:35.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:35 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:46:36.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:36 smithi049 ceph-mon[30080]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-21T13:46:36.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:36 smithi049 ceph-mon[30080]: osdmap e24: 4 total, 3 up, 4 in 2024-09-21T13:46:36.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:36 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T13:46:36.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:36 smithi049 ceph-mon[40489]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-21T13:46:36.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:36 smithi049 ceph-mon[40489]: osdmap e24: 4 total, 3 up, 4 in 2024-09-21T13:46:36.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:36 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T13:46:36.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:36 smithi187 ceph-mon[33032]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-21T13:46:36.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:36 smithi187 ceph-mon[33032]: osdmap e24: 4 total, 3 up, 4 in 2024-09-21T13:46:36.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:36 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T13:46:38.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:38 smithi049 ceph-mon[30080]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-21T13:46:38.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:38 smithi049 ceph-mon[40489]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-21T13:46:38.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:38 smithi187 ceph-mon[33032]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-21T13:46:40.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:40 smithi049 ceph-mon[30080]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-21T13:46:40.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:40 smithi049 ceph-mon[40489]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-21T13:46:40.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:40 smithi187 ceph-mon[33032]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-21T13:46:42.471 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:42 smithi049 ceph-mon[40489]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-21T13:46:42.471 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:42 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:46:42.471 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:42 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:42.472 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:42 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:42.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:42 smithi049 ceph-mon[30080]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-21T13:46:42.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:42 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:46:42.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:42 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:42.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:42 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:42.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:42 smithi187 ceph-mon[33032]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-21T13:46:42.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:42 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:46:42.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:42 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:42.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:42 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:44.466 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:44 smithi049 ceph-mon[40489]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-21T13:46:44.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:44 smithi049 ceph-mon[30080]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-21T13:46:44.497 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:44 smithi187 ceph-mon[33032]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-21T13:46:46.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:45 smithi049 ceph-mon[30080]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-21T13:46:46.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:45 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:46.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:45 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:46.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:45 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:46:46.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:45 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:46:46.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:45 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:46.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:45 smithi049 ceph-mon[40489]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-21T13:46:46.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:45 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:46.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:45 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:46.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:45 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:46:46.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:45 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:46:46.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:45 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:46.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:45 smithi187 ceph-mon[33032]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-21T13:46:46.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:45 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:46.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:45 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:46.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:45 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:46:46.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:45 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:46:46.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:45 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:46.935 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:46 smithi049 ceph-mon[40489]: from='osd.3 [v2:172.21.15.49:6826/1981113123,v1:172.21.15.49:6827/1981113123]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-21T13:46:46.935 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:46 smithi049 ceph-mon[40489]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-21T13:46:46.935 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:46 smithi049 ceph-mon[30080]: from='osd.3 [v2:172.21.15.49:6826/1981113123,v1:172.21.15.49:6827/1981113123]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-21T13:46:46.935 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:46 smithi049 ceph-mon[30080]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-21T13:46:47.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:46 smithi187 ceph-mon[33032]: from='osd.3 [v2:172.21.15.49:6826/1981113123,v1:172.21.15.49:6827/1981113123]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-21T13:46:47.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:46 smithi187 ceph-mon[33032]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-21T13:46:47.652 INFO:teuthology.orchestra.run.smithi049.stdout:Created osd(s) 3 on host 'smithi049' 2024-09-21T13:46:47.955 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:47 smithi049 ceph-mon[30080]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-21T13:46:47.955 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:47 smithi049 ceph-mon[30080]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-21T13:46:47.955 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:47 smithi049 ceph-mon[30080]: osdmap e25: 4 total, 3 up, 4 in 2024-09-21T13:46:47.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:47 smithi049 ceph-mon[30080]: from='osd.3 [v2:172.21.15.49:6826/1981113123,v1:172.21.15.49:6827/1981113123]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi049", "root=default"]} : dispatch 2024-09-21T13:46:47.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:47 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T13:46:47.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:47 smithi049 ceph-mon[30080]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi049", "root=default"]} : dispatch 2024-09-21T13:46:47.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:47 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:46:47.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:47 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:47.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:47 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:47.956 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:47 smithi049 ceph-mon[40489]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-21T13:46:47.956 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:47 smithi049 ceph-mon[40489]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-21T13:46:47.956 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:47 smithi049 ceph-mon[40489]: osdmap e25: 4 total, 3 up, 4 in 2024-09-21T13:46:47.956 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:47 smithi049 ceph-mon[40489]: from='osd.3 [v2:172.21.15.49:6826/1981113123,v1:172.21.15.49:6827/1981113123]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi049", "root=default"]} : dispatch 2024-09-21T13:46:47.956 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:47 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T13:46:47.957 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:47 smithi049 ceph-mon[40489]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi049", "root=default"]} : dispatch 2024-09-21T13:46:47.957 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:47 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:46:47.957 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:47 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:47.957 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:47 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:48.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:47 smithi187 ceph-mon[33032]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-21T13:46:48.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:47 smithi187 ceph-mon[33032]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-21T13:46:48.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:47 smithi187 ceph-mon[33032]: osdmap e25: 4 total, 3 up, 4 in 2024-09-21T13:46:48.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:47 smithi187 ceph-mon[33032]: from='osd.3 [v2:172.21.15.49:6826/1981113123,v1:172.21.15.49:6827/1981113123]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi049", "root=default"]} : dispatch 2024-09-21T13:46:48.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:47 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T13:46:48.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:47 smithi187 ceph-mon[33032]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi049", "root=default"]} : dispatch 2024-09-21T13:46:48.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:47 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:46:48.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:47 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:48.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:47 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:48.308 DEBUG:teuthology.orchestra.run.smithi049:osd.3> sudo journalctl -f -n 0 -u ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@osd.3.service 2024-09-21T13:46:48.311 INFO:tasks.cephadm:Deploying osd.4 on smithi187 with /dev/vg_nvme/lv_4... 2024-09-21T13:46:48.311 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_4 2024-09-21T13:46:48.564 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:46:48.850 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:48 smithi049 ceph-mon[30080]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2024-09-21T13:46:48.850 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:48 smithi049 ceph-mon[30080]: osdmap e26: 4 total, 3 up, 4 in 2024-09-21T13:46:48.850 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:48 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T13:46:48.850 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:48 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T13:46:48.851 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:48 smithi049 ceph-mon[40489]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2024-09-21T13:46:48.851 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:48 smithi049 ceph-mon[40489]: osdmap e26: 4 total, 3 up, 4 in 2024-09-21T13:46:48.851 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:48 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T13:46:48.851 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:48 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T13:46:48.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:48 smithi187 ceph-mon[33032]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2024-09-21T13:46:48.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:48 smithi187 ceph-mon[33032]: osdmap e26: 4 total, 3 up, 4 in 2024-09-21T13:46:48.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:48 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T13:46:48.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:48 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T13:46:49.873 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:49 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:46:49.874 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:49 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:46:49.874 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:49 smithi049 ceph-mon[40489]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-21T13:46:49.874 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:49 smithi049 ceph-mon[40489]: osd.3 [v2:172.21.15.49:6826/1981113123,v1:172.21.15.49:6827/1981113123] boot 2024-09-21T13:46:49.874 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:49 smithi049 ceph-mon[40489]: osdmap e27: 4 total, 4 up, 4 in 2024-09-21T13:46:49.874 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:49 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T13:46:49.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:49 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:46:49.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:49 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:46:49.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:49 smithi049 ceph-mon[30080]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-21T13:46:49.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:49 smithi049 ceph-mon[30080]: osd.3 [v2:172.21.15.49:6826/1981113123,v1:172.21.15.49:6827/1981113123] boot 2024-09-21T13:46:49.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:49 smithi049 ceph-mon[30080]: osdmap e27: 4 total, 4 up, 4 in 2024-09-21T13:46:49.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:49 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T13:46:50.042 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:49 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:46:50.042 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:49 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:46:50.042 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:49 smithi187 ceph-mon[33032]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-21T13:46:50.042 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:49 smithi187 ceph-mon[33032]: osd.3 [v2:172.21.15.49:6826/1981113123,v1:172.21.15.49:6827/1981113123] boot 2024-09-21T13:46:50.042 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:49 smithi187 ceph-mon[33032]: osdmap e27: 4 total, 4 up, 4 in 2024-09-21T13:46:50.043 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:49 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T13:46:50.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:50 smithi187 ceph-mon[33032]: osdmap e28: 4 total, 4 up, 4 in 2024-09-21T13:46:51.077 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:50 smithi049 ceph-mon[40489]: osdmap e28: 4 total, 4 up, 4 in 2024-09-21T13:46:51.078 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:50 smithi049 ceph-mon[30080]: osdmap e28: 4 total, 4 up, 4 in 2024-09-21T13:46:51.223 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:46:51.249 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi187:vg_nvme/lv_4 2024-09-21T13:46:51.490 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:46:52.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:52 smithi187 ceph-mon[33032]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-21T13:46:52.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:52 smithi187 ceph-mon[33032]: Detected new or changed devices on smithi049 2024-09-21T13:46:52.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:52 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:52.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:52 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:52.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:52 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T13:46:52.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:52 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T13:46:52.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:52 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T13:46:52.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:52 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T13:46:52.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:52 smithi187 ceph-mon[33032]: Adjusting osd_memory_target on smithi049 to 4016M 2024-09-21T13:46:52.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:52 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:52.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:52 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:46:52.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:52 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:46:52.196 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:52 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:52.454 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:52 smithi049 ceph-mon[40489]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-21T13:46:52.454 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:52 smithi049 ceph-mon[40489]: Detected new or changed devices on smithi049 2024-09-21T13:46:52.455 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:52 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:52.455 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:52 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:52.455 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:52 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T13:46:52.455 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:52 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T13:46:52.455 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:52 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T13:46:52.455 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:52 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T13:46:52.455 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:52 smithi049 ceph-mon[40489]: Adjusting osd_memory_target on smithi049 to 4016M 2024-09-21T13:46:52.455 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:52 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:52.455 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:52 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:46:52.455 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:52 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:46:52.455 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:52 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:52.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:52 smithi049 ceph-mon[30080]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-21T13:46:52.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:52 smithi049 ceph-mon[30080]: Detected new or changed devices on smithi049 2024-09-21T13:46:52.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:52 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:52.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:52 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:52.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:52 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T13:46:52.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:52 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T13:46:52.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:52 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T13:46:52.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:52 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T13:46:52.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:52 smithi049 ceph-mon[30080]: Adjusting osd_memory_target on smithi049 to 4016M 2024-09-21T13:46:52.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:52 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:52.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:52 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:46:52.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:52 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:46:52.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:52 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:46:53.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:53 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:46:53.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:53 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:46:53.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:53 smithi187 ceph-mon[33032]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:46:53.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:53 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T13:46:53.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:53 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T13:46:53.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:53 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:46:53.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:53 smithi187 ceph-mon[33032]: osdmap e29: 4 total, 4 up, 4 in 2024-09-21T13:46:53.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:53 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:46:53.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:53 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:46:53.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:53 smithi049 ceph-mon[40489]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:46:53.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:53 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T13:46:53.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:53 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T13:46:53.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:53 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:46:53.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:53 smithi049 ceph-mon[40489]: osdmap e29: 4 total, 4 up, 4 in 2024-09-21T13:46:53.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:53 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:46:53.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:53 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:46:53.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:53 smithi049 ceph-mon[30080]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:46:53.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:53 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T13:46:53.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:53 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T13:46:53.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:53 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:46:53.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:53 smithi049 ceph-mon[30080]: osdmap e29: 4 total, 4 up, 4 in 2024-09-21T13:46:54.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:54 smithi187 ceph-mon[33032]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-21T13:46:54.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:54 smithi049 ceph-mon[30080]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-21T13:46:54.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:54 smithi049 ceph-mon[40489]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-21T13:46:55.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:55 smithi187 ceph-mon[33032]: from='client.? 172.21.15.187:0/1344775706' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6fc64698-2d37-4db0-bf3d-c7e612c45163"} : dispatch 2024-09-21T13:46:55.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:55 smithi187 ceph-mon[33032]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6fc64698-2d37-4db0-bf3d-c7e612c45163"} : dispatch 2024-09-21T13:46:55.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:55 smithi187 ceph-mon[33032]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6fc64698-2d37-4db0-bf3d-c7e612c45163"}]': finished 2024-09-21T13:46:55.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:55 smithi187 ceph-mon[33032]: osdmap e30: 5 total, 4 up, 5 in 2024-09-21T13:46:55.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:55 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T13:46:55.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:55 smithi187 ceph-mon[33032]: from='client.? 172.21.15.187:0/3838528063' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T13:46:55.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:55 smithi049 ceph-mon[30080]: from='client.? 172.21.15.187:0/1344775706' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6fc64698-2d37-4db0-bf3d-c7e612c45163"} : dispatch 2024-09-21T13:46:55.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:55 smithi049 ceph-mon[30080]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6fc64698-2d37-4db0-bf3d-c7e612c45163"} : dispatch 2024-09-21T13:46:55.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:55 smithi049 ceph-mon[30080]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6fc64698-2d37-4db0-bf3d-c7e612c45163"}]': finished 2024-09-21T13:46:55.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:55 smithi049 ceph-mon[30080]: osdmap e30: 5 total, 4 up, 5 in 2024-09-21T13:46:55.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:55 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T13:46:55.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:55 smithi049 ceph-mon[30080]: from='client.? 172.21.15.187:0/3838528063' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T13:46:55.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:55 smithi049 ceph-mon[40489]: from='client.? 172.21.15.187:0/1344775706' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6fc64698-2d37-4db0-bf3d-c7e612c45163"} : dispatch 2024-09-21T13:46:55.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:55 smithi049 ceph-mon[40489]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6fc64698-2d37-4db0-bf3d-c7e612c45163"} : dispatch 2024-09-21T13:46:55.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:55 smithi049 ceph-mon[40489]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6fc64698-2d37-4db0-bf3d-c7e612c45163"}]': finished 2024-09-21T13:46:55.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:55 smithi049 ceph-mon[40489]: osdmap e30: 5 total, 4 up, 5 in 2024-09-21T13:46:55.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:55 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T13:46:55.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:55 smithi049 ceph-mon[40489]: from='client.? 172.21.15.187:0/3838528063' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T13:46:56.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:56 smithi187 ceph-mon[33032]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-21T13:46:56.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:56 smithi049 ceph-mon[30080]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-21T13:46:56.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:56 smithi049 ceph-mon[40489]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-21T13:46:58.596 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:46:58 smithi187 ceph-mon[33032]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-21T13:46:58.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:46:58 smithi049 ceph-mon[30080]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-21T13:46:58.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:46:58 smithi049 ceph-mon[40489]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-21T13:47:00.577 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:00 smithi187 ceph-mon[33032]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-21T13:47:00.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:00 smithi049 ceph-mon[30080]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-21T13:47:00.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:00 smithi049 ceph-mon[40489]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-21T13:47:01.631 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:01 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-21T13:47:01.631 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:01 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:47:01.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:01 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-21T13:47:01.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:01 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:47:01.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:01 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-21T13:47:01.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:01 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:47:02.589 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:02 smithi187 ceph-mon[33032]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-21T13:47:02.589 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:02 smithi187 ceph-mon[33032]: Deploying daemon osd.4 on smithi187 2024-09-21T13:47:02.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:02 smithi049 ceph-mon[30080]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-21T13:47:02.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:02 smithi049 ceph-mon[30080]: Deploying daemon osd.4 on smithi187 2024-09-21T13:47:02.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:02 smithi049 ceph-mon[40489]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-21T13:47:02.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:02 smithi049 ceph-mon[40489]: Deploying daemon osd.4 on smithi187 2024-09-21T13:47:04.683 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:04 smithi187 ceph-mon[33032]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-21T13:47:04.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:04 smithi049 ceph-mon[30080]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-21T13:47:04.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:04 smithi049 ceph-mon[40489]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-21T13:47:06.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:06 smithi187 ceph-mon[33032]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-21T13:47:06.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:06 smithi049 ceph-mon[30080]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-21T13:47:06.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:06 smithi049 ceph-mon[40489]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-21T13:47:08.674 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:08 smithi187 ceph-mon[33032]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-21T13:47:08.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:08 smithi049 ceph-mon[30080]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-21T13:47:08.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:08 smithi049 ceph-mon[40489]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-21T13:47:09.598 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:09 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:47:09.599 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:09 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:09.599 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:09 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:09.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:09 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:47:09.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:09 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:09.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:09 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:09.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:09 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:47:09.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:09 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:09.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:09 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:10.665 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:10 smithi187 ceph-mon[33032]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-21T13:47:10.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:10 smithi049 ceph-mon[30080]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-21T13:47:10.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:10 smithi049 ceph-mon[40489]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-21T13:47:12.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:12 smithi187 ceph-mon[33032]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-21T13:47:12.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:12 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:12.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:12 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:12.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:12 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:47:12.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:12 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:47:12.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:12 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:12.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:12 smithi049 ceph-mon[30080]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-21T13:47:12.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:12 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:12.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:12 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:12.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:12 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:47:12.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:12 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:47:12.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:12 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:12.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:12 smithi049 ceph-mon[40489]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-21T13:47:12.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:12 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:12.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:12 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:12.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:12 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:47:12.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:12 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:47:12.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:12 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:14.051 INFO:teuthology.orchestra.run.smithi187.stdout:Created osd(s) 4 on host 'smithi187' 2024-09-21T13:47:14.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:14 smithi187 ceph-mon[33032]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-21T13:47:14.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:14 smithi187 ceph-mon[33032]: from='osd.4 [v2:172.21.15.187:6800/1702447776,v1:172.21.15.187:6801/1702447776]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-21T13:47:14.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:14 smithi187 ceph-mon[33032]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-21T13:47:14.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:14 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:47:14.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:14 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:14.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:14 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:14.736 DEBUG:teuthology.orchestra.run.smithi187:osd.4> sudo journalctl -f -n 0 -u ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@osd.4.service 2024-09-21T13:47:14.738 INFO:tasks.cephadm:Deploying osd.5 on smithi187 with /dev/vg_nvme/lv_3... 2024-09-21T13:47:14.738 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_3 2024-09-21T13:47:14.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:14 smithi049 ceph-mon[30080]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-21T13:47:14.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:14 smithi049 ceph-mon[30080]: from='osd.4 [v2:172.21.15.187:6800/1702447776,v1:172.21.15.187:6801/1702447776]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-21T13:47:14.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:14 smithi049 ceph-mon[30080]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-21T13:47:14.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:14 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:47:14.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:14 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:14.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:14 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:14.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:14 smithi049 ceph-mon[40489]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-21T13:47:14.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:14 smithi049 ceph-mon[40489]: from='osd.4 [v2:172.21.15.187:6800/1702447776,v1:172.21.15.187:6801/1702447776]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-21T13:47:14.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:14 smithi049 ceph-mon[40489]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-21T13:47:14.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:14 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:47:14.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:14 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:14.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:14 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:15.463 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:47:15.583 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:15 smithi187 ceph-mon[33032]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-21T13:47:15.583 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:15 smithi187 ceph-mon[33032]: from='osd.4 [v2:172.21.15.187:6800/1702447776,v1:172.21.15.187:6801/1702447776]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-21T13:47:15.584 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:15 smithi187 ceph-mon[33032]: osdmap e31: 5 total, 4 up, 5 in 2024-09-21T13:47:15.584 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:15 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T13:47:15.584 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:15 smithi187 ceph-mon[33032]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-21T13:47:15.584 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:15 smithi187 ceph-mon[33032]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2024-09-21T13:47:15.584 INFO:journalctl@ceph.osd.4.smithi187.stdout:Sep 21 13:47:15 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-4[37666]: 2024-09-21T13:47:15.424+0000 7fdf1fa19640 -1 osd.4 0 waiting for initial osdmap 2024-09-21T13:47:15.584 INFO:journalctl@ceph.osd.4.smithi187.stdout:Sep 21 13:47:15 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-4[37666]: 2024-09-21T13:47:15.428+0000 7fdf1b840640 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T13:47:15.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:15 smithi049 ceph-mon[40489]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-21T13:47:15.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:15 smithi049 ceph-mon[40489]: from='osd.4 [v2:172.21.15.187:6800/1702447776,v1:172.21.15.187:6801/1702447776]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-21T13:47:15.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:15 smithi049 ceph-mon[40489]: osdmap e31: 5 total, 4 up, 5 in 2024-09-21T13:47:15.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:15 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T13:47:15.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:15 smithi049 ceph-mon[40489]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-21T13:47:15.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:15 smithi049 ceph-mon[40489]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2024-09-21T13:47:15.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:15 smithi049 ceph-mon[30080]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-21T13:47:15.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:15 smithi049 ceph-mon[30080]: from='osd.4 [v2:172.21.15.187:6800/1702447776,v1:172.21.15.187:6801/1702447776]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-21T13:47:15.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:15 smithi049 ceph-mon[30080]: osdmap e31: 5 total, 4 up, 5 in 2024-09-21T13:47:15.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:15 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T13:47:15.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:15 smithi049 ceph-mon[30080]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-21T13:47:15.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:15 smithi049 ceph-mon[30080]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2024-09-21T13:47:16.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:16 smithi187 ceph-mon[33032]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-21T13:47:16.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:16 smithi187 ceph-mon[33032]: osdmap e32: 5 total, 4 up, 5 in 2024-09-21T13:47:16.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:16 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T13:47:16.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:16 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T13:47:16.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:16 smithi187 ceph-mon[33032]: osd.4 [v2:172.21.15.187:6800/1702447776,v1:172.21.15.187:6801/1702447776] boot 2024-09-21T13:47:16.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:16 smithi187 ceph-mon[33032]: osdmap e33: 5 total, 5 up, 5 in 2024-09-21T13:47:16.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:16 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T13:47:16.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:16 smithi049 ceph-mon[40489]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-21T13:47:16.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:16 smithi049 ceph-mon[40489]: osdmap e32: 5 total, 4 up, 5 in 2024-09-21T13:47:16.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:16 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T13:47:16.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:16 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T13:47:16.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:16 smithi049 ceph-mon[40489]: osd.4 [v2:172.21.15.187:6800/1702447776,v1:172.21.15.187:6801/1702447776] boot 2024-09-21T13:47:16.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:16 smithi049 ceph-mon[40489]: osdmap e33: 5 total, 5 up, 5 in 2024-09-21T13:47:16.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:16 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T13:47:16.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:16 smithi049 ceph-mon[30080]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-21T13:47:16.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:16 smithi049 ceph-mon[30080]: osdmap e32: 5 total, 4 up, 5 in 2024-09-21T13:47:16.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:16 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T13:47:16.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:16 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T13:47:16.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:16 smithi049 ceph-mon[30080]: osd.4 [v2:172.21.15.187:6800/1702447776,v1:172.21.15.187:6801/1702447776] boot 2024-09-21T13:47:16.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:16 smithi049 ceph-mon[30080]: osdmap e33: 5 total, 5 up, 5 in 2024-09-21T13:47:16.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:16 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T13:47:17.697 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:17 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:47:17.697 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:17 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:47:17.698 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:17 smithi187 ceph-mon[33032]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-21T13:47:17.698 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:17 smithi187 ceph-mon[33032]: osdmap e34: 5 total, 5 up, 5 in 2024-09-21T13:47:17.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:17 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:47:17.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:17 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:47:17.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:17 smithi049 ceph-mon[30080]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-21T13:47:17.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:17 smithi049 ceph-mon[30080]: osdmap e34: 5 total, 5 up, 5 in 2024-09-21T13:47:17.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:17 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:47:17.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:17 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:47:17.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:17 smithi049 ceph-mon[40489]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-21T13:47:17.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:17 smithi049 ceph-mon[40489]: osdmap e34: 5 total, 5 up, 5 in 2024-09-21T13:47:19.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:19 smithi187 ceph-mon[33032]: osdmap e35: 5 total, 5 up, 5 in 2024-09-21T13:47:19.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:19 smithi187 ceph-mon[33032]: Detected new or changed devices on smithi187 2024-09-21T13:47:19.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:19 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:19.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:19 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:19.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:19 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T13:47:19.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:19 smithi187 ceph-mon[33032]: Adjusting osd_memory_target on smithi187 to 17088M 2024-09-21T13:47:19.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:19 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:19.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:19 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:47:19.446 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:19 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:47:19.446 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:19 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:19.446 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:19 smithi187 ceph-mon[33032]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-21T13:47:19.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:19 smithi049 ceph-mon[40489]: osdmap e35: 5 total, 5 up, 5 in 2024-09-21T13:47:19.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:19 smithi049 ceph-mon[40489]: Detected new or changed devices on smithi187 2024-09-21T13:47:19.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:19 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:19.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:19 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:19.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:19 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T13:47:19.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:19 smithi049 ceph-mon[40489]: Adjusting osd_memory_target on smithi187 to 17088M 2024-09-21T13:47:19.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:19 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:19.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:19 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:47:19.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:19 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:47:19.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:19 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:19.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:19 smithi049 ceph-mon[40489]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-21T13:47:19.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:19 smithi049 ceph-mon[30080]: osdmap e35: 5 total, 5 up, 5 in 2024-09-21T13:47:19.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:19 smithi049 ceph-mon[30080]: Detected new or changed devices on smithi187 2024-09-21T13:47:19.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:19 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:19.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:19 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:19.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:19 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T13:47:19.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:19 smithi049 ceph-mon[30080]: Adjusting osd_memory_target on smithi187 to 17088M 2024-09-21T13:47:19.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:19 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:19.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:19 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:47:19.757 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:19 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:47:19.757 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:19 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:19.757 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:19 smithi049 ceph-mon[30080]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-21T13:47:20.223 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:47:20.248 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi187:vg_nvme/lv_3 2024-09-21T13:47:20.489 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:47:22.078 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:21 smithi187 ceph-mon[33032]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 110 KiB/s, 0 objects/s recovering 2024-09-21T13:47:22.078 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:21 smithi187 ceph-mon[33032]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:47:22.078 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:21 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T13:47:22.078 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:21 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T13:47:22.078 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:21 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:47:22.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:21 smithi049 ceph-mon[30080]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 110 KiB/s, 0 objects/s recovering 2024-09-21T13:47:22.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:21 smithi049 ceph-mon[30080]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:47:22.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:21 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T13:47:22.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:21 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T13:47:22.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:21 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:47:22.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:21 smithi049 ceph-mon[40489]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 110 KiB/s, 0 objects/s recovering 2024-09-21T13:47:22.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:21 smithi049 ceph-mon[40489]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:47:22.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:21 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T13:47:22.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:21 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T13:47:22.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:21 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:47:23.909 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:23 smithi187 ceph-mon[33032]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-09-21T13:47:23.909 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:23 smithi187 ceph-mon[33032]: from='client.? 172.21.15.187:0/4190685535' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "8d7f62a1-7e92-4e06-9cad-86b96be514d8"} : dispatch 2024-09-21T13:47:23.909 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:23 smithi187 ceph-mon[33032]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "8d7f62a1-7e92-4e06-9cad-86b96be514d8"} : dispatch 2024-09-21T13:47:23.909 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:23 smithi187 ceph-mon[33032]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8d7f62a1-7e92-4e06-9cad-86b96be514d8"}]': finished 2024-09-21T13:47:23.910 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:23 smithi187 ceph-mon[33032]: osdmap e36: 6 total, 5 up, 6 in 2024-09-21T13:47:23.910 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:23 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T13:47:24.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:23 smithi049 ceph-mon[30080]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-09-21T13:47:24.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:23 smithi049 ceph-mon[30080]: from='client.? 172.21.15.187:0/4190685535' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "8d7f62a1-7e92-4e06-9cad-86b96be514d8"} : dispatch 2024-09-21T13:47:24.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:23 smithi049 ceph-mon[30080]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "8d7f62a1-7e92-4e06-9cad-86b96be514d8"} : dispatch 2024-09-21T13:47:24.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:23 smithi049 ceph-mon[30080]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8d7f62a1-7e92-4e06-9cad-86b96be514d8"}]': finished 2024-09-21T13:47:24.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:23 smithi049 ceph-mon[30080]: osdmap e36: 6 total, 5 up, 6 in 2024-09-21T13:47:24.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:23 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T13:47:24.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:23 smithi049 ceph-mon[40489]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-09-21T13:47:24.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:23 smithi049 ceph-mon[40489]: from='client.? 172.21.15.187:0/4190685535' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "8d7f62a1-7e92-4e06-9cad-86b96be514d8"} : dispatch 2024-09-21T13:47:24.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:23 smithi049 ceph-mon[40489]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "8d7f62a1-7e92-4e06-9cad-86b96be514d8"} : dispatch 2024-09-21T13:47:24.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:23 smithi049 ceph-mon[40489]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8d7f62a1-7e92-4e06-9cad-86b96be514d8"}]': finished 2024-09-21T13:47:24.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:23 smithi049 ceph-mon[40489]: osdmap e36: 6 total, 5 up, 6 in 2024-09-21T13:47:24.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:23 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T13:47:25.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:24 smithi187 ceph-mon[33032]: from='client.? 172.21.15.187:0/3995921628' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T13:47:25.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:24 smithi049 ceph-mon[30080]: from='client.? 172.21.15.187:0/3995921628' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T13:47:25.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:24 smithi049 ceph-mon[40489]: from='client.? 172.21.15.187:0/3995921628' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T13:47:26.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:25 smithi187 ceph-mon[33032]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-09-21T13:47:26.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:25 smithi049 ceph-mon[30080]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-09-21T13:47:26.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:25 smithi049 ceph-mon[40489]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-09-21T13:47:27.819 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:27 smithi187 ceph-mon[33032]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-09-21T13:47:28.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:27 smithi049 ceph-mon[40489]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-09-21T13:47:28.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:27 smithi049 ceph-mon[30080]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-09-21T13:47:30.149 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:29 smithi187 ceph-mon[33032]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T13:47:30.149 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:29 smithi187 ceph-mon[33032]: 1.0 scrub starts 2024-09-21T13:47:30.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:29 smithi049 ceph-mon[30080]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T13:47:30.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:29 smithi049 ceph-mon[30080]: 1.0 scrub starts 2024-09-21T13:47:30.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:29 smithi049 ceph-mon[40489]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T13:47:30.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:29 smithi049 ceph-mon[40489]: 1.0 scrub starts 2024-09-21T13:47:31.135 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:30 smithi187 ceph-mon[33032]: 1.0 scrub ok 2024-09-21T13:47:31.135 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:30 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-21T13:47:31.135 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:30 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:47:31.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:30 smithi049 ceph-mon[30080]: 1.0 scrub ok 2024-09-21T13:47:31.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:30 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-21T13:47:31.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:30 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:47:31.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:30 smithi049 ceph-mon[40489]: 1.0 scrub ok 2024-09-21T13:47:31.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:30 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-21T13:47:31.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:30 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:47:31.939 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:31 smithi187 ceph-mon[33032]: Deploying daemon osd.5 on smithi187 2024-09-21T13:47:31.939 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:31 smithi187 ceph-mon[33032]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-21T13:47:32.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:31 smithi049 ceph-mon[30080]: Deploying daemon osd.5 on smithi187 2024-09-21T13:47:32.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:31 smithi049 ceph-mon[30080]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-21T13:47:32.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:31 smithi049 ceph-mon[40489]: Deploying daemon osd.5 on smithi187 2024-09-21T13:47:32.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:31 smithi049 ceph-mon[40489]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-21T13:47:34.091 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:33 smithi187 ceph-mon[33032]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-21T13:47:34.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:33 smithi049 ceph-mon[30080]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-21T13:47:34.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:33 smithi049 ceph-mon[40489]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-21T13:47:36.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:35 smithi187 ceph-mon[33032]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-21T13:47:36.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:35 smithi049 ceph-mon[30080]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-21T13:47:36.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:35 smithi049 ceph-mon[40489]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-21T13:47:38.159 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:37 smithi187 ceph-mon[33032]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-21T13:47:38.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:37 smithi049 ceph-mon[30080]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-21T13:47:38.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:37 smithi049 ceph-mon[40489]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-21T13:47:39.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:38 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:47:39.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:38 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:39.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:38 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:39.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:38 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:47:39.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:38 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:39.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:38 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:39.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:38 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:47:39.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:38 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:39.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:38 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:40.515 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:40 smithi187 ceph-mon[33032]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-21T13:47:40.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:40 smithi049 ceph-mon[30080]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-21T13:47:40.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:40 smithi049 ceph-mon[40489]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-21T13:47:42.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:41 smithi187 ceph-mon[33032]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-21T13:47:42.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:41 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:42.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:41 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:42.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:41 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:47:42.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:41 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:47:42.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:41 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:42.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:41 smithi049 ceph-mon[40489]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-21T13:47:42.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:41 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:42.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:41 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:42.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:41 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:47:42.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:41 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:47:42.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:41 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:42.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:41 smithi049 ceph-mon[30080]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-21T13:47:42.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:41 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:42.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:41 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:42.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:41 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:47:42.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:41 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:47:42.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:41 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:44.035 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:43 smithi187 ceph-mon[33032]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-21T13:47:44.035 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:43 smithi187 ceph-mon[33032]: from='osd.5 [v2:172.21.15.187:6808/3659050776,v1:172.21.15.187:6809/3659050776]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T13:47:44.035 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:43 smithi187 ceph-mon[33032]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T13:47:44.081 INFO:teuthology.orchestra.run.smithi187.stdout:Created osd(s) 5 on host 'smithi187' 2024-09-21T13:47:44.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:43 smithi049 ceph-mon[30080]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-21T13:47:44.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:43 smithi049 ceph-mon[30080]: from='osd.5 [v2:172.21.15.187:6808/3659050776,v1:172.21.15.187:6809/3659050776]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T13:47:44.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:43 smithi049 ceph-mon[30080]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T13:47:44.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:43 smithi049 ceph-mon[40489]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-21T13:47:44.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:43 smithi049 ceph-mon[40489]: from='osd.5 [v2:172.21.15.187:6808/3659050776,v1:172.21.15.187:6809/3659050776]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T13:47:44.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:43 smithi049 ceph-mon[40489]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T13:47:44.740 DEBUG:teuthology.orchestra.run.smithi187:osd.5> sudo journalctl -f -n 0 -u ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@osd.5.service 2024-09-21T13:47:44.742 INFO:tasks.cephadm:Deploying osd.6 on smithi187 with /dev/vg_nvme/lv_2... 2024-09-21T13:47:44.742 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_2 2024-09-21T13:47:45.002 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 13:47:44 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5[43330]: 2024-09-21T13:47:44.838+0000 7f7fe439c640 -1 osd.5 0 waiting for initial osdmap 2024-09-21T13:47:45.002 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 13:47:44 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5[43330]: 2024-09-21T13:47:44.843+0000 7f7fe01c3640 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T13:47:45.002 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:44 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:47:45.002 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:44 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:47:45.002 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:44 smithi187 ceph-mon[33032]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-21T13:47:45.002 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:44 smithi187 ceph-mon[33032]: from='osd.5 [v2:172.21.15.187:6808/3659050776,v1:172.21.15.187:6809/3659050776]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-21T13:47:45.002 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:44 smithi187 ceph-mon[33032]: osdmap e37: 6 total, 5 up, 6 in 2024-09-21T13:47:45.003 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:44 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T13:47:45.003 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:44 smithi187 ceph-mon[33032]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-21T13:47:45.003 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:44 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:47:45.003 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:44 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:45.003 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:44 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:45.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:44 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:47:45.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:44 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:47:45.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:44 smithi049 ceph-mon[40489]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-21T13:47:45.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:44 smithi049 ceph-mon[40489]: from='osd.5 [v2:172.21.15.187:6808/3659050776,v1:172.21.15.187:6809/3659050776]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-21T13:47:45.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:44 smithi049 ceph-mon[40489]: osdmap e37: 6 total, 5 up, 6 in 2024-09-21T13:47:45.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:44 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T13:47:45.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:44 smithi049 ceph-mon[40489]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-21T13:47:45.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:44 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:47:45.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:44 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:45.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:44 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:45.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:44 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:47:45.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:44 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:47:45.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:44 smithi049 ceph-mon[30080]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-21T13:47:45.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:44 smithi049 ceph-mon[30080]: from='osd.5 [v2:172.21.15.187:6808/3659050776,v1:172.21.15.187:6809/3659050776]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-21T13:47:45.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:44 smithi049 ceph-mon[30080]: osdmap e37: 6 total, 5 up, 6 in 2024-09-21T13:47:45.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:44 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T13:47:45.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:44 smithi049 ceph-mon[30080]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-21T13:47:45.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:44 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:47:45.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:44 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:45.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:44 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:45.472 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:47:46.118 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:45 smithi187 ceph-mon[33032]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-21T13:47:46.118 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:45 smithi187 ceph-mon[33032]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2024-09-21T13:47:46.118 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:45 smithi187 ceph-mon[33032]: osdmap e38: 6 total, 5 up, 6 in 2024-09-21T13:47:46.118 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:45 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T13:47:46.119 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:45 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T13:47:46.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:45 smithi049 ceph-mon[40489]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-21T13:47:46.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:45 smithi049 ceph-mon[40489]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2024-09-21T13:47:46.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:45 smithi049 ceph-mon[40489]: osdmap e38: 6 total, 5 up, 6 in 2024-09-21T13:47:46.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:45 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T13:47:46.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:45 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T13:47:46.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:45 smithi049 ceph-mon[30080]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-09-21T13:47:46.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:45 smithi049 ceph-mon[30080]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2024-09-21T13:47:46.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:45 smithi049 ceph-mon[30080]: osdmap e38: 6 total, 5 up, 6 in 2024-09-21T13:47:46.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:45 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T13:47:46.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:45 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T13:47:47.184 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:46 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:47:47.184 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:46 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:47:47.184 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:46 smithi187 ceph-mon[33032]: osd.5 [v2:172.21.15.187:6808/3659050776,v1:172.21.15.187:6809/3659050776] boot 2024-09-21T13:47:47.184 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:46 smithi187 ceph-mon[33032]: osdmap e39: 6 total, 6 up, 6 in 2024-09-21T13:47:47.184 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:46 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T13:47:47.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:46 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:47:47.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:46 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:47:47.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:46 smithi049 ceph-mon[30080]: osd.5 [v2:172.21.15.187:6808/3659050776,v1:172.21.15.187:6809/3659050776] boot 2024-09-21T13:47:47.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:46 smithi049 ceph-mon[30080]: osdmap e39: 6 total, 6 up, 6 in 2024-09-21T13:47:47.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:46 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T13:47:47.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:46 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:47:47.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:46 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:47:47.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:46 smithi049 ceph-mon[40489]: osd.5 [v2:172.21.15.187:6808/3659050776,v1:172.21.15.187:6809/3659050776] boot 2024-09-21T13:47:47.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:46 smithi049 ceph-mon[40489]: osdmap e39: 6 total, 6 up, 6 in 2024-09-21T13:47:47.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:46 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T13:47:48.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:47 smithi187 ceph-mon[33032]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-21T13:47:48.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:47 smithi187 ceph-mon[33032]: osdmap e40: 6 total, 6 up, 6 in 2024-09-21T13:47:48.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:47 smithi049 ceph-mon[30080]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-21T13:47:48.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:47 smithi049 ceph-mon[30080]: osdmap e40: 6 total, 6 up, 6 in 2024-09-21T13:47:48.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:47 smithi049 ceph-mon[40489]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-21T13:47:48.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:47 smithi049 ceph-mon[40489]: osdmap e40: 6 total, 6 up, 6 in 2024-09-21T13:47:49.068 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:48 smithi187 ceph-mon[33032]: osdmap e41: 6 total, 6 up, 6 in 2024-09-21T13:47:49.068 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:48 smithi187 ceph-mon[33032]: Detected new or changed devices on smithi187 2024-09-21T13:47:49.068 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:48 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:49.068 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:48 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:49.068 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:48 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T13:47:49.069 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:48 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T13:47:49.069 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:48 smithi187 ceph-mon[33032]: Adjusting osd_memory_target on smithi187 to 8544M 2024-09-21T13:47:49.069 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:48 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:49.069 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:48 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:47:49.069 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:48 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:47:49.069 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:48 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:49.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:48 smithi049 ceph-mon[30080]: osdmap e41: 6 total, 6 up, 6 in 2024-09-21T13:47:49.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:48 smithi049 ceph-mon[30080]: Detected new or changed devices on smithi187 2024-09-21T13:47:49.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:48 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:49.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:48 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:49.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:48 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T13:47:49.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:48 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T13:47:49.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:48 smithi049 ceph-mon[30080]: Adjusting osd_memory_target on smithi187 to 8544M 2024-09-21T13:47:49.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:48 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:49.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:48 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:47:49.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:48 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:47:49.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:48 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:49.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:48 smithi049 ceph-mon[40489]: osdmap e41: 6 total, 6 up, 6 in 2024-09-21T13:47:49.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:48 smithi049 ceph-mon[40489]: Detected new or changed devices on smithi187 2024-09-21T13:47:49.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:48 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:49.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:48 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:49.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:48 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T13:47:49.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:48 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T13:47:49.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:48 smithi049 ceph-mon[40489]: Adjusting osd_memory_target on smithi187 to 8544M 2024-09-21T13:47:49.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:48 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:49.257 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:48 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:47:49.257 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:48 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:47:49.257 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:48 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:47:50.082 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:47:50.110 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi187:vg_nvme/lv_2 2024-09-21T13:47:50.356 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:47:50.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:50 smithi187 ceph-mon[33032]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-21T13:47:50.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:50 smithi049 ceph-mon[40489]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-21T13:47:50.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:50 smithi049 ceph-mon[30080]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-21T13:47:51.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:51 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T13:47:51.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:51 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T13:47:51.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:51 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:47:51.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:51 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T13:47:51.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:51 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T13:47:51.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:51 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:47:51.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:51 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T13:47:51.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:51 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T13:47:51.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:51 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:47:52.631 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:52 smithi187 ceph-mon[33032]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-21T13:47:52.631 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:52 smithi187 ceph-mon[33032]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:47:52.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:52 smithi049 ceph-mon[30080]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-21T13:47:52.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:52 smithi049 ceph-mon[30080]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:47:52.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:52 smithi049 ceph-mon[40489]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-21T13:47:52.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:52 smithi049 ceph-mon[40489]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:47:53.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:53 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:47:53.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:53 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:47:53.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:53 smithi187 ceph-mon[33032]: osdmap e42: 6 total, 6 up, 6 in 2024-09-21T13:47:53.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:53 smithi187 ceph-mon[33032]: from='client.? 172.21.15.187:0/3340250112' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "3a7afd13-26ce-444c-9cc0-81ce2eab6ce5"} : dispatch 2024-09-21T13:47:53.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:53 smithi187 ceph-mon[33032]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "3a7afd13-26ce-444c-9cc0-81ce2eab6ce5"} : dispatch 2024-09-21T13:47:53.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:53 smithi187 ceph-mon[33032]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3a7afd13-26ce-444c-9cc0-81ce2eab6ce5"}]': finished 2024-09-21T13:47:53.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:53 smithi187 ceph-mon[33032]: osdmap e43: 7 total, 6 up, 7 in 2024-09-21T13:47:53.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:53 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T13:47:53.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:53 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:47:53.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:53 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:47:53.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:53 smithi049 ceph-mon[40489]: osdmap e42: 6 total, 6 up, 6 in 2024-09-21T13:47:53.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:53 smithi049 ceph-mon[40489]: from='client.? 172.21.15.187:0/3340250112' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "3a7afd13-26ce-444c-9cc0-81ce2eab6ce5"} : dispatch 2024-09-21T13:47:53.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:53 smithi049 ceph-mon[40489]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "3a7afd13-26ce-444c-9cc0-81ce2eab6ce5"} : dispatch 2024-09-21T13:47:53.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:53 smithi049 ceph-mon[40489]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3a7afd13-26ce-444c-9cc0-81ce2eab6ce5"}]': finished 2024-09-21T13:47:53.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:53 smithi049 ceph-mon[40489]: osdmap e43: 7 total, 6 up, 7 in 2024-09-21T13:47:53.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:53 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T13:47:53.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:53 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:47:53.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:53 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:47:53.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:53 smithi049 ceph-mon[30080]: osdmap e42: 6 total, 6 up, 6 in 2024-09-21T13:47:53.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:53 smithi049 ceph-mon[30080]: from='client.? 172.21.15.187:0/3340250112' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "3a7afd13-26ce-444c-9cc0-81ce2eab6ce5"} : dispatch 2024-09-21T13:47:53.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:53 smithi049 ceph-mon[30080]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "3a7afd13-26ce-444c-9cc0-81ce2eab6ce5"} : dispatch 2024-09-21T13:47:53.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:53 smithi049 ceph-mon[30080]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3a7afd13-26ce-444c-9cc0-81ce2eab6ce5"}]': finished 2024-09-21T13:47:53.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:53 smithi049 ceph-mon[30080]: osdmap e43: 7 total, 6 up, 7 in 2024-09-21T13:47:53.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:53 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T13:47:54.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:54 smithi187 ceph-mon[33032]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-21T13:47:54.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:54 smithi187 ceph-mon[33032]: from='client.? 172.21.15.187:0/3387873213' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T13:47:54.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:54 smithi049 ceph-mon[30080]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-21T13:47:54.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:54 smithi049 ceph-mon[30080]: from='client.? 172.21.15.187:0/3387873213' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T13:47:54.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:54 smithi049 ceph-mon[40489]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-21T13:47:54.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:54 smithi049 ceph-mon[40489]: from='client.? 172.21.15.187:0/3387873213' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T13:47:55.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:55 smithi049 ceph-mon[30080]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-21T13:47:55.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:55 smithi049 ceph-mon[40489]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-21T13:47:55.790 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:55 smithi187 ceph-mon[33032]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-21T13:47:58.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:57 smithi049 ceph-mon[40489]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-21T13:47:58.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:57 smithi049 ceph-mon[30080]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-21T13:47:58.089 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:57 smithi187 ceph-mon[33032]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-21T13:47:59.853 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:47:59 smithi187 ceph-mon[33032]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-21T13:48:00.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:47:59 smithi049 ceph-mon[30080]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-21T13:48:00.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:47:59 smithi049 ceph-mon[40489]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-21T13:48:02.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:01 smithi049 ceph-mon[30080]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-21T13:48:02.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:01 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-21T13:48:02.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:01 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:48:02.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:01 smithi049 ceph-mon[30080]: Deploying daemon osd.6 on smithi187 2024-09-21T13:48:02.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:01 smithi049 ceph-mon[40489]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-21T13:48:02.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:01 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-21T13:48:02.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:01 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:48:02.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:01 smithi049 ceph-mon[40489]: Deploying daemon osd.6 on smithi187 2024-09-21T13:48:02.183 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:01 smithi187 ceph-mon[33032]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-21T13:48:02.184 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:01 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-21T13:48:02.184 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:01 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:48:02.184 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:01 smithi187 ceph-mon[33032]: Deploying daemon osd.6 on smithi187 2024-09-21T13:48:03.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:03 smithi187 ceph-mon[33032]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-21T13:48:04.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:03 smithi049 ceph-mon[30080]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-21T13:48:04.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:03 smithi049 ceph-mon[40489]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-21T13:48:06.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:05 smithi049 ceph-mon[30080]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-21T13:48:06.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:05 smithi049 ceph-mon[40489]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-21T13:48:06.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:05 smithi187 ceph-mon[33032]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-21T13:48:07.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:06 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:48:07.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:06 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:48:07.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:06 smithi049 ceph-mon[40489]: osdmap e44: 7 total, 6 up, 7 in 2024-09-21T13:48:07.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:06 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T13:48:07.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:06 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:48:07.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:06 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:48:07.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:06 smithi049 ceph-mon[30080]: osdmap e44: 7 total, 6 up, 7 in 2024-09-21T13:48:07.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:06 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T13:48:07.188 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:06 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:48:07.188 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:06 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:48:07.188 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:06 smithi187 ceph-mon[33032]: osdmap e44: 7 total, 6 up, 7 in 2024-09-21T13:48:07.188 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:06 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T13:48:07.848 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:07 smithi187 ceph-mon[33032]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-21T13:48:08.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:07 smithi049 ceph-mon[30080]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-21T13:48:08.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:07 smithi049 ceph-mon[40489]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-21T13:48:08.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:08 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:48:08.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:08 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:08.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:08 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:09.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:08 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:48:09.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:08 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:09.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:08 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:09.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:08 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:48:09.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:08 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:09.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:08 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:10.058 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:09 smithi187 ceph-mon[33032]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-21T13:48:10.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:09 smithi049 ceph-mon[30080]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-21T13:48:10.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:09 smithi049 ceph-mon[40489]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-21T13:48:12.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:12 smithi187 ceph-mon[33032]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-21T13:48:12.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:12 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:12.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:12 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:12.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:12 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:48:12.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:12 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:48:12.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:12 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:12.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:12 smithi049 ceph-mon[40489]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-21T13:48:12.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:12 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:12.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:12 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:12.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:12 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:48:12.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:12 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:48:12.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:12 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:12.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:12 smithi049 ceph-mon[30080]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-21T13:48:12.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:12 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:12.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:12 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:12.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:12 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:48:12.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:12 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:48:12.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:12 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:14.385 INFO:teuthology.orchestra.run.smithi187.stdout:Created osd(s) 6 on host 'smithi187' 2024-09-21T13:48:14.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:14 smithi187 ceph-mon[33032]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-21T13:48:14.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:14 smithi187 ceph-mon[33032]: from='osd.6 [v2:172.21.15.187:6816/2728426525,v1:172.21.15.187:6817/2728426525]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-21T13:48:14.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:14 smithi187 ceph-mon[33032]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-21T13:48:14.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:14 smithi049 ceph-mon[40489]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-21T13:48:14.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:14 smithi049 ceph-mon[40489]: from='osd.6 [v2:172.21.15.187:6816/2728426525,v1:172.21.15.187:6817/2728426525]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-21T13:48:14.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:14 smithi049 ceph-mon[40489]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-21T13:48:14.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:14 smithi049 ceph-mon[30080]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-21T13:48:14.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:14 smithi049 ceph-mon[30080]: from='osd.6 [v2:172.21.15.187:6816/2728426525,v1:172.21.15.187:6817/2728426525]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-21T13:48:14.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:14 smithi049 ceph-mon[30080]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-21T13:48:15.000 DEBUG:teuthology.orchestra.run.smithi187:osd.6> sudo journalctl -f -n 0 -u ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@osd.6.service 2024-09-21T13:48:15.002 INFO:tasks.cephadm:Deploying osd.7 on smithi187 with /dev/vg_nvme/lv_1... 2024-09-21T13:48:15.003 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_1 2024-09-21T13:48:15.600 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:15 smithi187 ceph-mon[33032]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-21T13:48:15.600 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:15 smithi187 ceph-mon[33032]: osdmap e45: 7 total, 6 up, 7 in 2024-09-21T13:48:15.600 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:15 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T13:48:15.600 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:15 smithi187 ceph-mon[33032]: from='osd.6 [v2:172.21.15.187:6816/2728426525,v1:172.21.15.187:6817/2728426525]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-21T13:48:15.600 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:15 smithi187 ceph-mon[33032]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-21T13:48:15.600 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:15 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:48:15.600 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:15 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:15.600 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:15 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:15.601 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 13:48:15 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T13:48:15.362+0000 7f3a3e8ef640 -1 osd.6 0 waiting for initial osdmap 2024-09-21T13:48:15.601 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 13:48:15 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T13:48:15.369+0000 7f3a3a716640 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T13:48:15.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:15 smithi049 ceph-mon[30080]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-21T13:48:15.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:15 smithi049 ceph-mon[30080]: osdmap e45: 7 total, 6 up, 7 in 2024-09-21T13:48:15.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:15 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T13:48:15.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:15 smithi049 ceph-mon[30080]: from='osd.6 [v2:172.21.15.187:6816/2728426525,v1:172.21.15.187:6817/2728426525]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-21T13:48:15.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:15 smithi049 ceph-mon[30080]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-21T13:48:15.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:15 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:48:15.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:15 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:15.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:15 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:15.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:15 smithi049 ceph-mon[40489]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-21T13:48:15.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:15 smithi049 ceph-mon[40489]: osdmap e45: 7 total, 6 up, 7 in 2024-09-21T13:48:15.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:15 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T13:48:15.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:15 smithi049 ceph-mon[40489]: from='osd.6 [v2:172.21.15.187:6816/2728426525,v1:172.21.15.187:6817/2728426525]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-21T13:48:15.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:15 smithi049 ceph-mon[40489]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-21T13:48:15.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:15 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:48:15.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:15 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:15.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:15 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:16.175 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:48:16.539 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:16 smithi187 ceph-mon[33032]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-21T13:48:16.540 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:16 smithi187 ceph-mon[33032]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2024-09-21T13:48:16.540 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:16 smithi187 ceph-mon[33032]: osdmap e46: 7 total, 6 up, 7 in 2024-09-21T13:48:16.540 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:16 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T13:48:16.540 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:16 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T13:48:16.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:16 smithi049 ceph-mon[30080]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-21T13:48:16.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:16 smithi049 ceph-mon[30080]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2024-09-21T13:48:16.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:16 smithi049 ceph-mon[30080]: osdmap e46: 7 total, 6 up, 7 in 2024-09-21T13:48:16.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:16 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T13:48:16.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:16 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T13:48:16.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:16 smithi049 ceph-mon[40489]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-21T13:48:16.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:16 smithi049 ceph-mon[40489]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2024-09-21T13:48:16.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:16 smithi049 ceph-mon[40489]: osdmap e46: 7 total, 6 up, 7 in 2024-09-21T13:48:16.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:16 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T13:48:16.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:16 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T13:48:17.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:17 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:48:17.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:17 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:48:17.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:17 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T13:48:17.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:17 smithi187 ceph-mon[33032]: osd.6 [v2:172.21.15.187:6816/2728426525,v1:172.21.15.187:6817/2728426525] boot 2024-09-21T13:48:17.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:17 smithi187 ceph-mon[33032]: osdmap e47: 7 total, 7 up, 7 in 2024-09-21T13:48:17.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:17 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T13:48:17.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:17 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:48:17.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:17 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:48:17.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:17 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T13:48:17.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:17 smithi049 ceph-mon[30080]: osd.6 [v2:172.21.15.187:6816/2728426525,v1:172.21.15.187:6817/2728426525] boot 2024-09-21T13:48:17.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:17 smithi049 ceph-mon[30080]: osdmap e47: 7 total, 7 up, 7 in 2024-09-21T13:48:17.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:17 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T13:48:17.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:17 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:48:17.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:17 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:48:17.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:17 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T13:48:17.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:17 smithi049 ceph-mon[40489]: osd.6 [v2:172.21.15.187:6816/2728426525,v1:172.21.15.187:6817/2728426525] boot 2024-09-21T13:48:17.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:17 smithi049 ceph-mon[40489]: osdmap e47: 7 total, 7 up, 7 in 2024-09-21T13:48:17.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:17 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T13:48:18.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:18 smithi187 ceph-mon[33032]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-21T13:48:18.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:18 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:48:18.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:18 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:48:18.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:18 smithi187 ceph-mon[33032]: osdmap e48: 7 total, 7 up, 7 in 2024-09-21T13:48:18.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:18 smithi049 ceph-mon[30080]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-21T13:48:18.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:18 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:48:18.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:18 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:48:18.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:18 smithi049 ceph-mon[30080]: osdmap e48: 7 total, 7 up, 7 in 2024-09-21T13:48:18.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:18 smithi049 ceph-mon[40489]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-09-21T13:48:18.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:18 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:48:18.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:18 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:48:18.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:18 smithi049 ceph-mon[40489]: osdmap e48: 7 total, 7 up, 7 in 2024-09-21T13:48:19.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:19 smithi187 ceph-mon[33032]: osdmap e49: 7 total, 7 up, 7 in 2024-09-21T13:48:19.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:19 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:19.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:19 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:19.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:19 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T13:48:19.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:19 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T13:48:19.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:19 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T13:48:19.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:19 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:19.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:19 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:48:19.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:19 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:48:19.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:19 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:19.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:19 smithi049 ceph-mon[30080]: osdmap e49: 7 total, 7 up, 7 in 2024-09-21T13:48:19.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:19 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:19.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:19 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:19.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:19 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T13:48:19.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:19 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T13:48:19.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:19 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T13:48:19.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:19 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:19.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:19 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:48:19.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:19 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:48:19.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:19 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:19.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:19 smithi049 ceph-mon[40489]: osdmap e49: 7 total, 7 up, 7 in 2024-09-21T13:48:19.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:19 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:19.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:19 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:19.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:19 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T13:48:19.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:19 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T13:48:19.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:19 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T13:48:19.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:19 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:19.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:19 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:48:19.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:19 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:48:19.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:19 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:20.202 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-21T13:48:20.227 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi187:vg_nvme/lv_1 2024-09-21T13:48:20.473 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:48:20.593 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:20 smithi187 ceph-mon[33032]: Detected new or changed devices on smithi187 2024-09-21T13:48:20.593 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:20 smithi187 ceph-mon[33032]: Adjusting osd_memory_target on smithi187 to 5696M 2024-09-21T13:48:20.593 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:20 smithi187 ceph-mon[33032]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-21T13:48:20.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:20 smithi049 ceph-mon[30080]: Detected new or changed devices on smithi187 2024-09-21T13:48:20.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:20 smithi049 ceph-mon[30080]: Adjusting osd_memory_target on smithi187 to 5696M 2024-09-21T13:48:20.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:20 smithi049 ceph-mon[30080]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-21T13:48:20.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:20 smithi049 ceph-mon[40489]: Detected new or changed devices on smithi187 2024-09-21T13:48:20.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:20 smithi049 ceph-mon[40489]: Adjusting osd_memory_target on smithi187 to 5696M 2024-09-21T13:48:20.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:20 smithi049 ceph-mon[40489]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-09-21T13:48:21.678 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:21 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T13:48:21.678 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:21 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T13:48:21.679 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:21 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:48:21.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:21 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T13:48:21.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:21 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T13:48:21.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:21 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:48:21.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:21 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T13:48:21.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:21 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T13:48:21.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:21 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:48:22.653 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:22 smithi187 ceph-mon[33032]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-21T13:48:22.653 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:22 smithi187 ceph-mon[33032]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:48:22.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:22 smithi049 ceph-mon[30080]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-21T13:48:22.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:22 smithi049 ceph-mon[30080]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:48:22.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:22 smithi049 ceph-mon[40489]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-21T13:48:22.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:22 smithi049 ceph-mon[40489]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:48:24.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:24 smithi187 ceph-mon[33032]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-21T13:48:24.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:24 smithi187 ceph-mon[33032]: from='client.? 172.21.15.187:0/2957662230' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1e08d891-8747-402e-b4c2-aa3802ce8537"} : dispatch 2024-09-21T13:48:24.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:24 smithi187 ceph-mon[33032]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1e08d891-8747-402e-b4c2-aa3802ce8537"} : dispatch 2024-09-21T13:48:24.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:24 smithi187 ceph-mon[33032]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1e08d891-8747-402e-b4c2-aa3802ce8537"}]': finished 2024-09-21T13:48:24.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:24 smithi187 ceph-mon[33032]: osdmap e50: 8 total, 7 up, 8 in 2024-09-21T13:48:24.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:24 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T13:48:24.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:24 smithi187 ceph-mon[33032]: from='client.? 172.21.15.187:0/3231635727' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T13:48:24.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:24 smithi049 ceph-mon[40489]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-21T13:48:24.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:24 smithi049 ceph-mon[40489]: from='client.? 172.21.15.187:0/2957662230' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1e08d891-8747-402e-b4c2-aa3802ce8537"} : dispatch 2024-09-21T13:48:24.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:24 smithi049 ceph-mon[40489]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1e08d891-8747-402e-b4c2-aa3802ce8537"} : dispatch 2024-09-21T13:48:24.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:24 smithi049 ceph-mon[40489]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1e08d891-8747-402e-b4c2-aa3802ce8537"}]': finished 2024-09-21T13:48:24.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:24 smithi049 ceph-mon[40489]: osdmap e50: 8 total, 7 up, 8 in 2024-09-21T13:48:24.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:24 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T13:48:24.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:24 smithi049 ceph-mon[40489]: from='client.? 172.21.15.187:0/3231635727' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T13:48:24.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:24 smithi049 ceph-mon[30080]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-21T13:48:24.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:24 smithi049 ceph-mon[30080]: from='client.? 172.21.15.187:0/2957662230' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1e08d891-8747-402e-b4c2-aa3802ce8537"} : dispatch 2024-09-21T13:48:24.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:24 smithi049 ceph-mon[30080]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1e08d891-8747-402e-b4c2-aa3802ce8537"} : dispatch 2024-09-21T13:48:24.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:24 smithi049 ceph-mon[30080]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1e08d891-8747-402e-b4c2-aa3802ce8537"}]': finished 2024-09-21T13:48:24.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:24 smithi049 ceph-mon[30080]: osdmap e50: 8 total, 7 up, 8 in 2024-09-21T13:48:24.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:24 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T13:48:24.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:24 smithi049 ceph-mon[30080]: from='client.? 172.21.15.187:0/3231635727' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T13:48:26.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:26 smithi187 ceph-mon[33032]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-21T13:48:26.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:26 smithi049 ceph-mon[30080]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-21T13:48:26.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:26 smithi049 ceph-mon[40489]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-21T13:48:28.670 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:28 smithi187 ceph-mon[33032]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-21T13:48:28.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:28 smithi049 ceph-mon[30080]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-21T13:48:28.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:28 smithi049 ceph-mon[40489]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-21T13:48:30.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:30 smithi187 ceph-mon[33032]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-21T13:48:30.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:30 smithi049 ceph-mon[40489]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-21T13:48:30.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:30 smithi049 ceph-mon[30080]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-21T13:48:31.634 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:31 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-21T13:48:31.635 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:31 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:48:31.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:31 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-21T13:48:31.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:31 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:48:31.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:31 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-21T13:48:31.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:31 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:48:32.630 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:32 smithi187 ceph-mon[33032]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-21T13:48:32.631 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:32 smithi187 ceph-mon[33032]: Deploying daemon osd.7 on smithi187 2024-09-21T13:48:32.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:32 smithi049 ceph-mon[30080]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-21T13:48:32.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:32 smithi049 ceph-mon[30080]: Deploying daemon osd.7 on smithi187 2024-09-21T13:48:32.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:32 smithi049 ceph-mon[40489]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-21T13:48:32.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:32 smithi049 ceph-mon[40489]: Deploying daemon osd.7 on smithi187 2024-09-21T13:48:34.639 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:34 smithi187 ceph-mon[33032]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-21T13:48:34.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:34 smithi049 ceph-mon[30080]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-21T13:48:34.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:34 smithi049 ceph-mon[40489]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-21T13:48:36.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:36 smithi187 ceph-mon[33032]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-21T13:48:36.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:36 smithi049 ceph-mon[30080]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-21T13:48:36.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:36 smithi049 ceph-mon[40489]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-21T13:48:37.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:37 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:48:37.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:37 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:48:37.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:37 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:48:37.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:37 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:48:37.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:37 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:48:37.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:37 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:48:38.534 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:38 smithi187 ceph-mon[33032]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-21T13:48:38.535 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:38 smithi187 ceph-mon[33032]: osdmap e51: 8 total, 7 up, 8 in 2024-09-21T13:48:38.535 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:38 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T13:48:38.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:38 smithi049 ceph-mon[30080]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-21T13:48:38.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:38 smithi049 ceph-mon[30080]: osdmap e51: 8 total, 7 up, 8 in 2024-09-21T13:48:38.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:38 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T13:48:38.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:38 smithi049 ceph-mon[40489]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-21T13:48:38.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:38 smithi049 ceph-mon[40489]: osdmap e51: 8 total, 7 up, 8 in 2024-09-21T13:48:38.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:38 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T13:48:39.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:39 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:48:39.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:39 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:39.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:39 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:39.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:39 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:48:39.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:39 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:39.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:39 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:39.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:39 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:48:39.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:39 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:39.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:39 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:40.686 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:40 smithi187 ceph-mon[33032]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-21T13:48:40.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:40 smithi049 ceph-mon[30080]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-21T13:48:40.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:40 smithi049 ceph-mon[40489]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-21T13:48:42.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:42 smithi187 ceph-mon[33032]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-21T13:48:42.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:42 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:42.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:42 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:42.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:42 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:48:42.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:42 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:48:42.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:42 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:42.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:42 smithi049 ceph-mon[30080]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-21T13:48:42.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:42 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:42.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:42 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:42.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:42 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:48:42.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:42 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:48:42.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:42 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:42.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:42 smithi049 ceph-mon[40489]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-21T13:48:42.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:42 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:42.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:42 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:42.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:42 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:48:42.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:42 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:48:42.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:42 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:44.462 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:44 smithi187 ceph-mon[33032]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-21T13:48:44.462 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:44 smithi187 ceph-mon[33032]: from='osd.7 [v2:172.21.15.187:6824/3680850037,v1:172.21.15.187:6825/3680850037]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-21T13:48:44.462 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:44 smithi187 ceph-mon[33032]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-21T13:48:44.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:44 smithi049 ceph-mon[30080]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-21T13:48:44.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:44 smithi049 ceph-mon[30080]: from='osd.7 [v2:172.21.15.187:6824/3680850037,v1:172.21.15.187:6825/3680850037]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-21T13:48:44.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:44 smithi049 ceph-mon[30080]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-21T13:48:44.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:44 smithi049 ceph-mon[40489]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-21T13:48:44.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:44 smithi049 ceph-mon[40489]: from='osd.7 [v2:172.21.15.187:6824/3680850037,v1:172.21.15.187:6825/3680850037]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-21T13:48:44.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:44 smithi049 ceph-mon[40489]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-21T13:48:44.770 INFO:teuthology.orchestra.run.smithi187.stdout:Created osd(s) 7 on host 'smithi187' 2024-09-21T13:48:45.427 DEBUG:teuthology.orchestra.run.smithi187:osd.7> sudo journalctl -f -n 0 -u ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@osd.7.service 2024-09-21T13:48:45.429 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-09-21T13:48:45.430 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd stat -f json 2024-09-21T13:48:45.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:45 smithi187 ceph-mon[33032]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-21T13:48:45.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:45 smithi187 ceph-mon[33032]: osdmap e52: 8 total, 7 up, 8 in 2024-09-21T13:48:45.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:45 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T13:48:45.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:45 smithi187 ceph-mon[33032]: from='osd.7 [v2:172.21.15.187:6824/3680850037,v1:172.21.15.187:6825/3680850037]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-21T13:48:45.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:45 smithi187 ceph-mon[33032]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-21T13:48:45.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:45 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:48:45.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:45 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:45.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:45 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:45.700 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:48:45.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:45 smithi049 ceph-mon[30080]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-21T13:48:45.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:45 smithi049 ceph-mon[30080]: osdmap e52: 8 total, 7 up, 8 in 2024-09-21T13:48:45.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:45 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T13:48:45.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:45 smithi049 ceph-mon[30080]: from='osd.7 [v2:172.21.15.187:6824/3680850037,v1:172.21.15.187:6825/3680850037]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-21T13:48:45.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:45 smithi049 ceph-mon[30080]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-21T13:48:45.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:45 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:48:45.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:45 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:45.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:45 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:45.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:45 smithi049 ceph-mon[40489]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-21T13:48:45.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:45 smithi049 ceph-mon[40489]: osdmap e52: 8 total, 7 up, 8 in 2024-09-21T13:48:45.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:45 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T13:48:45.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:45 smithi049 ceph-mon[40489]: from='osd.7 [v2:172.21.15.187:6824/3680850037,v1:172.21.15.187:6825/3680850037]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-21T13:48:45.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:45 smithi049 ceph-mon[40489]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-21T13:48:45.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:45 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:48:45.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:45 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:45.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:45 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:45.969 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 13:48:45 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[55522]: 2024-09-21T13:48:45.867+0000 7fadfd923640 -1 osd.7 0 waiting for initial osdmap 2024-09-21T13:48:46.038 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 13:48:45 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[55522]: 2024-09-21T13:48:45.874+0000 7fadf5742640 -1 osd.7 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T13:48:46.338 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:48:46.605 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:46 smithi187 ceph-mon[33032]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-21T13:48:46.605 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:46 smithi187 ceph-mon[33032]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2024-09-21T13:48:46.605 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:46 smithi187 ceph-mon[33032]: osdmap e53: 8 total, 7 up, 8 in 2024-09-21T13:48:46.605 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:46 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T13:48:46.605 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:46 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T13:48:46.605 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:46 smithi187 ceph-mon[33032]: from='osd.7 ' entity='osd.7' 2024-09-21T13:48:46.605 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:46 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2424187656' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-21T13:48:46.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:46 smithi049 ceph-mon[40489]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-21T13:48:46.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:46 smithi049 ceph-mon[40489]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2024-09-21T13:48:46.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:46 smithi049 ceph-mon[40489]: osdmap e53: 8 total, 7 up, 8 in 2024-09-21T13:48:46.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:46 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T13:48:46.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:46 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T13:48:46.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:46 smithi049 ceph-mon[40489]: from='osd.7 ' entity='osd.7' 2024-09-21T13:48:46.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:46 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2424187656' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-21T13:48:46.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:46 smithi049 ceph-mon[30080]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-09-21T13:48:46.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:46 smithi049 ceph-mon[30080]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2024-09-21T13:48:46.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:46 smithi049 ceph-mon[30080]: osdmap e53: 8 total, 7 up, 8 in 2024-09-21T13:48:46.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:46 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T13:48:46.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:46 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T13:48:46.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:46 smithi049 ceph-mon[30080]: from='osd.7 ' entity='osd.7' 2024-09-21T13:48:46.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:46 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2424187656' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-21T13:48:47.166 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":53,"num_osds":8,"num_up_osds":7,"osd_up_since":1726926496,"num_in_osds":8,"osd_in_since":1726926503,"num_remapped_pgs":0} 2024-09-21T13:48:47.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:47 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:48:47.757 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:47 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:48:47.757 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:47 smithi049 ceph-mon[30080]: osd.7 [v2:172.21.15.187:6824/3680850037,v1:172.21.15.187:6825/3680850037] boot 2024-09-21T13:48:47.757 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:47 smithi049 ceph-mon[30080]: osdmap e54: 8 total, 8 up, 8 in 2024-09-21T13:48:47.757 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:47 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T13:48:47.757 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:47 smithi049 ceph-mon[30080]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-21T13:48:47.758 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:47 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:48:47.758 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:47 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:48:47.758 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:47 smithi049 ceph-mon[40489]: osd.7 [v2:172.21.15.187:6824/3680850037,v1:172.21.15.187:6825/3680850037] boot 2024-09-21T13:48:47.758 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:47 smithi049 ceph-mon[40489]: osdmap e54: 8 total, 8 up, 8 in 2024-09-21T13:48:47.758 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:47 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T13:48:47.759 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:47 smithi049 ceph-mon[40489]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-21T13:48:47.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:47 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:48:47.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:47 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:48:47.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:47 smithi187 ceph-mon[33032]: osd.7 [v2:172.21.15.187:6824/3680850037,v1:172.21.15.187:6825/3680850037] boot 2024-09-21T13:48:47.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:47 smithi187 ceph-mon[33032]: osdmap e54: 8 total, 8 up, 8 in 2024-09-21T13:48:47.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:47 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T13:48:47.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:47 smithi187 ceph-mon[33032]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-21T13:48:48.167 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd stat -f json 2024-09-21T13:48:48.445 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:48:48.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:48 smithi187 ceph-mon[33032]: osdmap e55: 8 total, 8 up, 8 in 2024-09-21T13:48:48.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:48 smithi187 ceph-mon[33032]: Detected new or changed devices on smithi187 2024-09-21T13:48:48.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:48 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:48.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:48 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:48.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:48 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T13:48:48.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:48 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T13:48:48.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:48 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T13:48:48.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:48 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T13:48:48.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:48 smithi187 ceph-mon[33032]: Adjusting osd_memory_target on smithi187 to 4272M 2024-09-21T13:48:48.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:48 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:48.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:48 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:48:48.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:48 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:48:48.696 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:48 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:48.711 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:48 smithi049 ceph-mon[40489]: osdmap e55: 8 total, 8 up, 8 in 2024-09-21T13:48:48.711 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:48 smithi049 ceph-mon[40489]: Detected new or changed devices on smithi187 2024-09-21T13:48:48.711 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:48 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:48.711 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:48 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:48.711 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:48 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T13:48:48.711 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:48 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T13:48:48.711 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:48 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T13:48:48.712 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:48 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T13:48:48.712 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:48 smithi049 ceph-mon[40489]: Adjusting osd_memory_target on smithi187 to 4272M 2024-09-21T13:48:48.712 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:48 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:48.712 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:48 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:48:48.712 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:48 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:48:48.712 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:48 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:48.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:48 smithi049 ceph-mon[30080]: osdmap e55: 8 total, 8 up, 8 in 2024-09-21T13:48:48.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:48 smithi049 ceph-mon[30080]: Detected new or changed devices on smithi187 2024-09-21T13:48:48.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:48 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:48.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:48 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:48.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:48 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T13:48:48.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:48 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T13:48:48.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:48 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T13:48:48.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:48 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T13:48:48.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:48 smithi049 ceph-mon[30080]: Adjusting osd_memory_target on smithi187 to 4272M 2024-09-21T13:48:48.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:48 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:48.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:48 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:48:48.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:48 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:48:48.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:48 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:49.118 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:48:49.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:49 smithi049 ceph-mon[40489]: osdmap e56: 8 total, 8 up, 8 in 2024-09-21T13:48:49.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:49 smithi049 ceph-mon[40489]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-21T13:48:49.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:49 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:48:49.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:49 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:48:49.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:49 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2525041816' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-21T13:48:49.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:49 smithi049 ceph-mon[30080]: osdmap e56: 8 total, 8 up, 8 in 2024-09-21T13:48:49.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:49 smithi049 ceph-mon[30080]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-21T13:48:49.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:49 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:48:49.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:49 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:48:49.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:49 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2525041816' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-21T13:48:49.842 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":56,"num_osds":8,"num_up_osds":8,"osd_up_since":1726926526,"num_in_osds":8,"osd_in_since":1726926503,"num_remapped_pgs":0} 2024-09-21T13:48:49.843 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd dump --format=json 2024-09-21T13:48:49.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:49 smithi187 ceph-mon[33032]: osdmap e56: 8 total, 8 up, 8 in 2024-09-21T13:48:49.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:49 smithi187 ceph-mon[33032]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-21T13:48:49.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:49 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:48:49.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:49 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:48:49.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:49 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2525041816' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-21T13:48:50.097 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:48:50.473 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:50 smithi049 ceph-mon[40489]: osdmap e57: 8 total, 8 up, 8 in 2024-09-21T13:48:50.473 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:50 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:48:50.473 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:50 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:48:50.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:50 smithi049 ceph-mon[30080]: osdmap e57: 8 total, 8 up, 8 in 2024-09-21T13:48:50.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:50 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:48:50.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:50 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:48:50.754 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:48:50.754 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":57,"fsid":"40a3d2e8-781f-11ef-baf6-efdc52797490","created":"2024-09-21T13:42:57.838275+0000","modified":"2024-09-21T13:48:49.454858+0000","last_up_change":"2024-09-21T13:48:46.439884+0000","last_in_change":"2024-09-21T13:48:23.514355+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T13:46:21.096244+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2595dca9-6219-4e0c-8728-751eb102e73a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":2095561412},{"type":"v1","addr":"172.21.15.49:6803","nonce":2095561412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":2095561412},{"type":"v1","addr":"172.21.15.49:6805","nonce":2095561412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":2095561412},{"type":"v1","addr":"172.21.15.49:6809","nonce":2095561412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":2095561412},{"type":"v1","addr":"172.21.15.49:6807","nonce":2095561412}]},"public_addr":"172.21.15.49:6803/2095561412","cluster_addr":"172.21.15.49:6805/2095561412","heartbeat_back_addr":"172.21.15.49:6809/2095561412","heartbeat_front_addr":"172.21.15.49:6807/2095561412","state":["exists","up"]},{"osd":1,"uuid":"0ba327f9-5524-4a40-abcc-729ded0b1917","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":1382840527},{"type":"v1","addr":"172.21.15.49:6811","nonce":1382840527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":1382840527},{"type":"v1","addr":"172.21.15.49:6813","nonce":1382840527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":1382840527},{"type":"v1","addr":"172.21.15.49:6817","nonce":1382840527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":1382840527},{"type":"v1","addr":"172.21.15.49:6815","nonce":1382840527}]},"public_addr":"172.21.15.49:6811/1382840527","cluster_addr":"172.21.15.49:6813/1382840527","heartbeat_back_addr":"172.21.15.49:6817/1382840527","heartbeat_front_addr":"172.21.15.49:6815/1382840527","state":["exists","up"]},{"osd":2,"uuid":"27a45750-204d-48fd-bb8f-ebb01273343d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":2189046898},{"type":"v1","addr":"172.21.15.49:6819","nonce":2189046898}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":2189046898},{"type":"v1","addr":"172.21.15.49:6821","nonce":2189046898}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2189046898},{"type":"v1","addr":"172.21.15.49:6825","nonce":2189046898}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":2189046898},{"type":"v1","addr":"172.21.15.49:6823","nonce":2189046898}]},"public_addr":"172.21.15.49:6819/2189046898","cluster_addr":"172.21.15.49:6821/2189046898","heartbeat_back_addr":"172.21.15.49:6825/2189046898","heartbeat_front_addr":"172.21.15.49:6823/2189046898","state":["exists","up"]},{"osd":3,"uuid":"528bc390-3dc9-4f3b-ab75-d7412c9625db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":1981113123},{"type":"v1","addr":"172.21.15.49:6827","nonce":1981113123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":1981113123},{"type":"v1","addr":"172.21.15.49:6829","nonce":1981113123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":1981113123},{"type":"v1","addr":"172.21.15.49:6833","nonce":1981113123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":1981113123},{"type":"v1","addr":"172.21.15.49:6831","nonce":1981113123}]},"public_addr":"172.21.15.49:6827/1981113123","cluster_addr":"172.21.15.49:6829/1981113123","heartbeat_back_addr":"172.21.15.49:6833/1981113123","heartbeat_front_addr":"172.21.15.49:6831/1981113123","state":["exists","up"]},{"osd":4,"uuid":"6fc64698-2d37-4db0-bf3d-c7e612c45163","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":1702447776},{"type":"v1","addr":"172.21.15.187:6801","nonce":1702447776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":1702447776},{"type":"v1","addr":"172.21.15.187:6803","nonce":1702447776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":1702447776},{"type":"v1","addr":"172.21.15.187:6807","nonce":1702447776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":1702447776},{"type":"v1","addr":"172.21.15.187:6805","nonce":1702447776}]},"public_addr":"172.21.15.187:6801/1702447776","cluster_addr":"172.21.15.187:6803/1702447776","heartbeat_back_addr":"172.21.15.187:6807/1702447776","heartbeat_front_addr":"172.21.15.187:6805/1702447776","state":["exists","up"]},{"osd":5,"uuid":"8d7f62a1-7e92-4e06-9cad-86b96be514d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3659050776},{"type":"v1","addr":"172.21.15.187:6809","nonce":3659050776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3659050776},{"type":"v1","addr":"172.21.15.187:6811","nonce":3659050776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3659050776},{"type":"v1","addr":"172.21.15.187:6815","nonce":3659050776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3659050776},{"type":"v1","addr":"172.21.15.187:6813","nonce":3659050776}]},"public_addr":"172.21.15.187:6809/3659050776","cluster_addr":"172.21.15.187:6811/3659050776","heartbeat_back_addr":"172.21.15.187:6815/3659050776","heartbeat_front_addr":"172.21.15.187:6813/3659050776","state":["exists","up"]},{"osd":6,"uuid":"3a7afd13-26ce-444c-9cc0-81ce2eab6ce5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":2728426525},{"type":"v1","addr":"172.21.15.187:6817","nonce":2728426525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":2728426525},{"type":"v1","addr":"172.21.15.187:6819","nonce":2728426525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":2728426525},{"type":"v1","addr":"172.21.15.187:6823","nonce":2728426525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":2728426525},{"type":"v1","addr":"172.21.15.187:6821","nonce":2728426525}]},"public_addr":"172.21.15.187:6817/2728426525","cluster_addr":"172.21.15.187:6819/2728426525","heartbeat_back_addr":"172.21.15.187:6823/2728426525","heartbeat_front_addr":"172.21.15.187:6821/2728426525","state":["exists","up"]},{"osd":7,"uuid":"1e08d891-8747-402e-b4c2-aa3802ce8537","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":3680850037},{"type":"v1","addr":"172.21.15.187:6825","nonce":3680850037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":3680850037},{"type":"v1","addr":"172.21.15.187:6827","nonce":3680850037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":3680850037},{"type":"v1","addr":"172.21.15.187:6831","nonce":3680850037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":3680850037},{"type":"v1","addr":"172.21.15.187:6829","nonce":3680850037}]},"public_addr":"172.21.15.187:6825/3680850037","cluster_addr":"172.21.15.187:6827/3680850037","heartbeat_back_addr":"172.21.15.187:6831/3680850037","heartbeat_front_addr":"172.21.15.187:6829/3680850037","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T13:47:52.198887+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T13:48:05.624549+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T13:48:49.018318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T13:48:16.805740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T13:48:36.764542+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T13:47:43.919949+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T13:48:14.651488+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T13:48:44.977550+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.49:0/3600977363":"2024-09-22T13:43:56.633723+0000","172.21.15.49:0/3125743776":"2024-09-22T13:43:56.633723+0000","172.21.15.49:6801/4129636473":"2024-09-22T13:43:56.633723+0000","172.21.15.49:6800/4129636473":"2024-09-22T13:43:56.633723+0000","172.21.15.49:0/202325455":"2024-09-22T13:43:28.110499+0000","172.21.15.49:0/3514741273":"2024-09-22T13:43:56.633723+0000","172.21.15.49:0/231530138":"2024-09-22T13:43:28.110499+0000","172.21.15.49:0/3556829486":"2024-09-22T13:43:28.110499+0000","172.21.15.49:6801/2518197280":"2024-09-22T13:43:28.110499+0000","172.21.15.49:6800/2518197280":"2024-09-22T13:43:28.110499+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T13:48:50.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:50 smithi187 ceph-mon[33032]: osdmap e57: 8 total, 8 up, 8 in 2024-09-21T13:48:50.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:50 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:48:50.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:50 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:48:51.430 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T13:46:21.096244+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-21T13:48:51.430 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd pool get .mgr pg_num 2024-09-21T13:48:51.684 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:48:51.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:51 smithi049 ceph-mon[30080]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-21T13:48:51.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:51 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/212501423' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T13:48:51.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:51 smithi049 ceph-mon[30080]: osdmap e58: 8 total, 8 up, 8 in 2024-09-21T13:48:51.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:51 smithi049 ceph-mon[40489]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-21T13:48:51.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:51 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/212501423' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T13:48:51.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:51 smithi049 ceph-mon[40489]: osdmap e58: 8 total, 8 up, 8 in 2024-09-21T13:48:51.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:51 smithi187 ceph-mon[33032]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-21T13:48:51.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:51 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/212501423' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T13:48:51.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:51 smithi187 ceph-mon[33032]: osdmap e58: 8 total, 8 up, 8 in 2024-09-21T13:48:52.386 INFO:teuthology.orchestra.run.smithi049.stdout:pg_num: 1 2024-09-21T13:48:52.667 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:52 smithi049 ceph-mon[40489]: 1.0 scrub starts 2024-09-21T13:48:52.667 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:52 smithi049 ceph-mon[40489]: 1.0 scrub ok 2024-09-21T13:48:52.667 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:52 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1084756878' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-21T13:48:52.667 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:52 smithi049 ceph-mon[30080]: 1.0 scrub starts 2024-09-21T13:48:52.667 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:52 smithi049 ceph-mon[30080]: 1.0 scrub ok 2024-09-21T13:48:52.667 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:52 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1084756878' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-21T13:48:52.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:52 smithi187 ceph-mon[33032]: 1.0 scrub starts 2024-09-21T13:48:52.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:52 smithi187 ceph-mon[33032]: 1.0 scrub ok 2024-09-21T13:48:52.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:52 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1084756878' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-21T13:48:53.074 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi049 2024-09-21T13:48:53.074 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph orch apply rgw foo.a --placement '1;smithi049=foo.a' 2024-09-21T13:48:53.334 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:48:53.724 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:53 smithi187 ceph-mon[33032]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-21T13:48:53.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:53 smithi049 ceph-mon[30080]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-21T13:48:53.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:53 smithi049 ceph-mon[40489]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-21T13:48:54.012 INFO:teuthology.orchestra.run.smithi187.stdout:Scheduled rgw.foo.a update... 2024-09-21T13:48:54.615 DEBUG:teuthology.orchestra.run.smithi049:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@rgw.foo.a.service 2024-09-21T13:48:54.618 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi187 2024-09-21T13:48:54.618 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd pool create datapool 3 3 replicated 2024-09-21T13:48:54.863 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:48:55.264 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:55 smithi187 ceph-mon[33032]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi049=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:48:55.264 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:55 smithi187 ceph-mon[33032]: Saving service rgw.foo.a spec with placement smithi049=foo.a;count:1 2024-09-21T13:48:55.264 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:55 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:55.264 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:55 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:48:55.264 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:55 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:48:55.264 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:55 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:48:55.264 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:55 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:55.264 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:55 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-21T13:48:55.265 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:55 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-21T13:48:55.265 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:55 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:55.265 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:55 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:48:55.265 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:55 smithi187 ceph-mon[33032]: Deploying daemon rgw.foo.a on smithi049 2024-09-21T13:48:55.391 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:55 smithi049 ceph-mon[40489]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi049=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:48:55.391 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:55 smithi049 ceph-mon[40489]: Saving service rgw.foo.a spec with placement smithi049=foo.a;count:1 2024-09-21T13:48:55.391 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:55 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:55.391 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:55 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:48:55.391 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:55 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:48:55.391 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:55 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:48:55.391 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:55 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:55.392 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:55 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-21T13:48:55.392 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:55 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-21T13:48:55.392 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:55 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:55.392 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:55 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:48:55.392 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:55 smithi049 ceph-mon[40489]: Deploying daemon rgw.foo.a on smithi049 2024-09-21T13:48:55.392 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:55 smithi049 ceph-mon[30080]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi049=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:48:55.392 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:55 smithi049 ceph-mon[30080]: Saving service rgw.foo.a spec with placement smithi049=foo.a;count:1 2024-09-21T13:48:55.392 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:55 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:55.392 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:55 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:48:55.392 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:55 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:48:55.393 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:55 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:48:55.393 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:55 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:55.393 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:55 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]} : dispatch 2024-09-21T13:48:55.393 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:55 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-21T13:48:55.393 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:55 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:55.393 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:55 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:48:55.393 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:55 smithi049 ceph-mon[30080]: Deploying daemon rgw.foo.a on smithi049 2024-09-21T13:48:56.005 INFO:journalctl@ceph.rgw.foo.a.smithi049.stdout:Sep 21 13:48:55 smithi049 systemd[1]: Starting Ceph rgw.foo.a for 40a3d2e8-781f-11ef-baf6-efdc52797490... 2024-09-21T13:48:56.040 INFO:teuthology.orchestra.run.smithi187.stderr:pool 'datapool' created 2024-09-21T13:48:56.334 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:56 smithi187 ceph-mon[33032]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-21T13:48:56.334 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:56 smithi187 ceph-mon[33032]: from='client.? 172.21.15.187:0/1551275519' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-21T13:48:56.334 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:56 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-21T13:48:56.372 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:56 smithi049 ceph-mon[40489]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-21T13:48:56.372 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:56 smithi049 ceph-mon[40489]: from='client.? 172.21.15.187:0/1551275519' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-21T13:48:56.372 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:56 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-21T13:48:56.372 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:56 smithi049 ceph-mon[30080]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-21T13:48:56.372 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:56 smithi049 ceph-mon[30080]: from='client.? 172.21.15.187:0/1551275519' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-21T13:48:56.372 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:56 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} : dispatch 2024-09-21T13:48:56.373 INFO:journalctl@ceph.rgw.foo.a.smithi049.stdout:Sep 21 13:48:56 smithi049 podman[66352]: 2024-09-21 13:48:55.987245013 +0000 UTC m=+0.016039858 image pull b415816db8b41f544cdfdc71d193ff8b1ee008bf7b0ec4f64da8eb8f71b62e85 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5 2024-09-21T13:48:56.373 INFO:journalctl@ceph.rgw.foo.a.smithi049.stdout:Sep 21 13:48:56 smithi049 podman[66352]: 2024-09-21 13:48:56.135939269 +0000 UTC m=+0.164734104 container create 0a792214d2a97ae87eb69b0a046de7ca87bfc08396f1dd1e54383930fe649673 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-rgw-foo-a, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-21-0842-4417849, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2) 2024-09-21T13:48:56.668 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- rbd pool init datapool 2024-09-21T13:48:56.755 INFO:journalctl@ceph.rgw.foo.a.smithi049.stdout:Sep 21 13:48:56 smithi049 podman[66352]: 2024-09-21 13:48:56.372058779 +0000 UTC m=+0.400853624 container init 0a792214d2a97ae87eb69b0a046de7ca87bfc08396f1dd1e54383930fe649673 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-rgw-foo-a, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD) 2024-09-21T13:48:56.755 INFO:journalctl@ceph.rgw.foo.a.smithi049.stdout:Sep 21 13:48:56 smithi049 podman[66352]: 2024-09-21 13:48:56.376883746 +0000 UTC m=+0.405678584 container start 0a792214d2a97ae87eb69b0a046de7ca87bfc08396f1dd1e54383930fe649673 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-rgw-foo-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-09-21T13:48:56.755 INFO:journalctl@ceph.rgw.foo.a.smithi049.stdout:Sep 21 13:48:56 smithi049 bash[66352]: 0a792214d2a97ae87eb69b0a046de7ca87bfc08396f1dd1e54383930fe649673 2024-09-21T13:48:56.755 INFO:journalctl@ceph.rgw.foo.a.smithi049.stdout:Sep 21 13:48:56 smithi049 systemd[1]: Started Ceph rgw.foo.a for 40a3d2e8-781f-11ef-baf6-efdc52797490. 2024-09-21T13:48:56.913 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:48:57.300 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:57 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-21T13:48:57.300 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:57 smithi049 ceph-mon[40489]: osdmap e59: 8 total, 8 up, 8 in 2024-09-21T13:48:57.300 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:57 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:57.300 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:57 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:57.300 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:57 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:57.300 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:57 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:57.300 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:57 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:57.300 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:57 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:48:57.300 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:57 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-21T13:48:57.300 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:57 smithi049 ceph-mon[30080]: osdmap e59: 8 total, 8 up, 8 in 2024-09-21T13:48:57.301 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:57 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:57.301 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:57 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:57.301 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:57 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:57.301 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:57 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:57.301 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:57 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:57.301 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:57 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:48:57.337 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:57 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-21T13:48:57.337 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:57 smithi187 ceph-mon[33032]: osdmap e59: 8 total, 8 up, 8 in 2024-09-21T13:48:57.338 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:57 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:57.338 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:57 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:57.338 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:57 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:57.338 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:57 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:57.338 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:57 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:57.338 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:57 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:48:58.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:58 smithi187 ceph-mon[33032]: Saving service rgw.foo.a spec with placement smithi049=foo.a;count:1 2024-09-21T13:48:58.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:58 smithi187 ceph-mon[33032]: pgmap v199: 4 pgs: 1 creating+peering, 1 active+clean, 2 unknown; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-21T13:48:58.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:58 smithi187 ceph-mon[33032]: osdmap e60: 8 total, 8 up, 8 in 2024-09-21T13:48:58.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:58 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3156966068' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"} : dispatch 2024-09-21T13:48:58.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:58 smithi187 ceph-mon[33032]: from='client.? ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"} : dispatch 2024-09-21T13:48:58.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:58 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:58.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:58 smithi187 ceph-mon[33032]: from='client.? 172.21.15.187:0/445563831' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-21T13:48:58.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:58 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-21T13:48:58.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:58 smithi187 ceph-mon[33032]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T13:48:58.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:58 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:58.446 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:58 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:58.446 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:58 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:48:58.446 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:58 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:48:58.446 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:58 smithi187 ceph-mon[33032]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-21T13:48:58.446 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:58 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-21T13:48:58.446 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:58 smithi187 ceph-mon[33032]: osdmap e61: 8 total, 8 up, 8 in 2024-09-21T13:48:58.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:58 smithi049 ceph-mon[30080]: Saving service rgw.foo.a spec with placement smithi049=foo.a;count:1 2024-09-21T13:48:58.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:58 smithi049 ceph-mon[30080]: pgmap v199: 4 pgs: 1 creating+peering, 1 active+clean, 2 unknown; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-21T13:48:58.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:58 smithi049 ceph-mon[30080]: osdmap e60: 8 total, 8 up, 8 in 2024-09-21T13:48:58.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:58 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3156966068' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"} : dispatch 2024-09-21T13:48:58.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:58 smithi049 ceph-mon[30080]: from='client.? ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"} : dispatch 2024-09-21T13:48:58.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:58 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:58.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:58 smithi049 ceph-mon[30080]: from='client.? 172.21.15.187:0/445563831' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-21T13:48:58.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:58 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-21T13:48:58.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:58 smithi049 ceph-mon[30080]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T13:48:58.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:58 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:58.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:58 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:58.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:58 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:48:58.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:58 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:48:58.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:58 smithi049 ceph-mon[30080]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-21T13:48:58.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:58 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-21T13:48:58.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:58 smithi049 ceph-mon[30080]: osdmap e61: 8 total, 8 up, 8 in 2024-09-21T13:48:58.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:58 smithi049 ceph-mon[40489]: Saving service rgw.foo.a spec with placement smithi049=foo.a;count:1 2024-09-21T13:48:58.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:58 smithi049 ceph-mon[40489]: pgmap v199: 4 pgs: 1 creating+peering, 1 active+clean, 2 unknown; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-21T13:48:58.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:58 smithi049 ceph-mon[40489]: osdmap e60: 8 total, 8 up, 8 in 2024-09-21T13:48:58.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:58 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3156966068' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"} : dispatch 2024-09-21T13:48:58.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:58 smithi049 ceph-mon[40489]: from='client.? ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"} : dispatch 2024-09-21T13:48:58.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:58 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:58.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:58 smithi049 ceph-mon[40489]: from='client.? 172.21.15.187:0/445563831' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-21T13:48:58.507 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:58 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "datapool","app": "rbd"} : dispatch 2024-09-21T13:48:58.507 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:58 smithi049 ceph-mon[40489]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T13:48:58.507 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:58 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:58.507 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:58 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:48:58.507 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:58 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:48:58.507 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:58 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:48:58.507 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:58 smithi049 ceph-mon[40489]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-21T13:48:58.507 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:58 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-21T13:48:58.507 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:58 smithi049 ceph-mon[40489]: osdmap e61: 8 total, 8 up, 8 in 2024-09-21T13:48:59.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:59 smithi187 ceph-mon[33032]: Checking dashboard <-> RGW credentials 2024-09-21T13:48:59.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:59 smithi187 ceph-mon[33032]: osdmap e62: 8 total, 8 up, 8 in 2024-09-21T13:48:59.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:59 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3404764347' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-21T13:48:59.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:59 smithi187 ceph-mon[33032]: from='client.? ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-21T13:48:59.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:59 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/130352508' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-21T13:48:59.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:48:59 smithi187 ceph-mon[33032]: from='client.? ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-21T13:48:59.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:59 smithi049 ceph-mon[30080]: Checking dashboard <-> RGW credentials 2024-09-21T13:48:59.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:59 smithi049 ceph-mon[30080]: osdmap e62: 8 total, 8 up, 8 in 2024-09-21T13:48:59.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:59 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3404764347' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-21T13:48:59.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:59 smithi049 ceph-mon[30080]: from='client.? ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-21T13:48:59.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:59 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/130352508' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-21T13:48:59.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:48:59 smithi049 ceph-mon[30080]: from='client.? ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-21T13:48:59.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:59 smithi049 ceph-mon[40489]: Checking dashboard <-> RGW credentials 2024-09-21T13:48:59.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:59 smithi049 ceph-mon[40489]: osdmap e62: 8 total, 8 up, 8 in 2024-09-21T13:48:59.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:59 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3404764347' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-21T13:48:59.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:59 smithi049 ceph-mon[40489]: from='client.? ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-21T13:48:59.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:59 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/130352508' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-21T13:48:59.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:48:59 smithi049 ceph-mon[40489]: from='client.? ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"} : dispatch 2024-09-21T13:49:00.402 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:00 smithi187 ceph-mon[33032]: pgmap v202: 36 pgs: 3 creating+peering, 1 active+clean, 32 unknown; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-21T13:49:00.402 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:00 smithi187 ceph-mon[33032]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T13:49:00.402 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:00 smithi187 ceph-mon[33032]: Cluster is now healthy 2024-09-21T13:49:00.402 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:00 smithi187 ceph-mon[33032]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-21T13:49:00.402 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:00 smithi187 ceph-mon[33032]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-21T13:49:00.402 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:00 smithi187 ceph-mon[33032]: osdmap e63: 8 total, 8 up, 8 in 2024-09-21T13:49:00.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:00 smithi049 ceph-mon[30080]: pgmap v202: 36 pgs: 3 creating+peering, 1 active+clean, 32 unknown; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-21T13:49:00.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:00 smithi049 ceph-mon[30080]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T13:49:00.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:00 smithi049 ceph-mon[30080]: Cluster is now healthy 2024-09-21T13:49:00.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:00 smithi049 ceph-mon[30080]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-21T13:49:00.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:00 smithi049 ceph-mon[30080]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-21T13:49:00.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:00 smithi049 ceph-mon[30080]: osdmap e63: 8 total, 8 up, 8 in 2024-09-21T13:49:00.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:00 smithi049 ceph-mon[40489]: pgmap v202: 36 pgs: 3 creating+peering, 1 active+clean, 32 unknown; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-09-21T13:49:00.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:00 smithi049 ceph-mon[40489]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T13:49:00.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:00 smithi049 ceph-mon[40489]: Cluster is now healthy 2024-09-21T13:49:00.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:00 smithi049 ceph-mon[40489]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-21T13:49:00.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:00 smithi049 ceph-mon[40489]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-21T13:49:00.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:00 smithi049 ceph-mon[40489]: osdmap e63: 8 total, 8 up, 8 in 2024-09-21T13:49:00.667 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.187 --placement '1;smithi187=iscsi.a' 2024-09-21T13:49:00.915 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:49:01.600 INFO:teuthology.orchestra.run.smithi187.stdout:Scheduled iscsi.datapool update... 2024-09-21T13:49:02.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:02 smithi187 ceph-mon[33032]: pgmap v205: 68 pgs: 16 creating+peering, 17 active+clean, 35 unknown; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 511 B/s wr, 0 op/s 2024-09-21T13:49:02.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:02 smithi187 ceph-mon[33032]: osdmap e64: 8 total, 8 up, 8 in 2024-09-21T13:49:02.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:02 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3404764347' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-21T13:49:02.196 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:02 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/130352508' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-21T13:49:02.196 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:02 smithi187 ceph-mon[33032]: from='client.? ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-21T13:49:02.196 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:02 smithi187 ceph-mon[33032]: from='client.? ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-21T13:49:02.196 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:02 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:49:02.233 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-09-21T13:49:02.233 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-09-21T13:49:02.233 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-09-21T13:49:02.264 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-09-21T13:49:02.264 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-09-21T13:49:02.290 DEBUG:teuthology.orchestra.run.smithi187:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@iscsi.iscsi.a.service 2024-09-21T13:49:02.333 INFO:tasks.cephadm:Adding prometheus.a on smithi187 2024-09-21T13:49:02.333 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph orch apply prometheus '1;smithi187=a' 2024-09-21T13:49:02.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:02 smithi049 ceph-mon[30080]: pgmap v205: 68 pgs: 16 creating+peering, 17 active+clean, 35 unknown; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 511 B/s wr, 0 op/s 2024-09-21T13:49:02.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:02 smithi049 ceph-mon[30080]: osdmap e64: 8 total, 8 up, 8 in 2024-09-21T13:49:02.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:02 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3404764347' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-21T13:49:02.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:02 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/130352508' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-21T13:49:02.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:02 smithi049 ceph-mon[30080]: from='client.? ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-21T13:49:02.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:02 smithi049 ceph-mon[30080]: from='client.? ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-21T13:49:02.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:02 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:49:02.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:02 smithi049 ceph-mon[40489]: pgmap v205: 68 pgs: 16 creating+peering, 17 active+clean, 35 unknown; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 511 B/s wr, 0 op/s 2024-09-21T13:49:02.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:02 smithi049 ceph-mon[40489]: osdmap e64: 8 total, 8 up, 8 in 2024-09-21T13:49:02.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:02 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3404764347' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-21T13:49:02.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:02 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/130352508' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-21T13:49:02.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:02 smithi049 ceph-mon[40489]: from='client.? ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-21T13:49:02.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:02 smithi049 ceph-mon[40489]: from='client.? ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"} : dispatch 2024-09-21T13:49:02.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:02 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:49:02.615 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:49:03.299 INFO:teuthology.orchestra.run.smithi187.stdout:Scheduled prometheus update... 2024-09-21T13:49:03.299 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:03 smithi187 ceph-mon[33032]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.187", "placement": "1;smithi187=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:49:03.299 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:03 smithi187 ceph-mon[33032]: Saving service iscsi.datapool spec with placement smithi187=iscsi.a;count:1 2024-09-21T13:49:03.299 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:03 smithi187 ceph-mon[33032]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-21T13:49:03.299 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:03 smithi187 ceph-mon[33032]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-21T13:49:03.299 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:03 smithi187 ceph-mon[33032]: osdmap e65: 8 total, 8 up, 8 in 2024-09-21T13:49:03.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:03 smithi049 ceph-mon[30080]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.187", "placement": "1;smithi187=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:49:03.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:03 smithi049 ceph-mon[30080]: Saving service iscsi.datapool spec with placement smithi187=iscsi.a;count:1 2024-09-21T13:49:03.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:03 smithi049 ceph-mon[30080]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-21T13:49:03.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:03 smithi049 ceph-mon[30080]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-21T13:49:03.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:03 smithi049 ceph-mon[30080]: osdmap e65: 8 total, 8 up, 8 in 2024-09-21T13:49:03.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:03 smithi049 ceph-mon[40489]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.187", "placement": "1;smithi187=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:49:03.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:03 smithi049 ceph-mon[40489]: Saving service iscsi.datapool spec with placement smithi187=iscsi.a;count:1 2024-09-21T13:49:03.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:03 smithi049 ceph-mon[40489]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-21T13:49:03.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:03 smithi049 ceph-mon[40489]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-21T13:49:03.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:03 smithi049 ceph-mon[40489]: osdmap e65: 8 total, 8 up, 8 in 2024-09-21T13:49:03.892 DEBUG:teuthology.orchestra.run.smithi187:prometheus.a> sudo journalctl -f -n 0 -u ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@prometheus.a.service 2024-09-21T13:49:03.895 INFO:tasks.cephadm:Adding node-exporter.a on smithi049 2024-09-21T13:49:03.895 INFO:tasks.cephadm:Adding node-exporter.b on smithi187 2024-09-21T13:49:03.895 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph orch apply node-exporter '2;smithi049=a;smithi187=b' 2024-09-21T13:49:04.186 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:49:04.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:04 smithi187 ceph-mon[33032]: pgmap v208: 100 pgs: 21 creating+peering, 53 active+clean, 26 unknown; 578 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2024-09-21T13:49:04.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:04 smithi187 ceph-mon[33032]: osdmap e66: 8 total, 8 up, 8 in 2024-09-21T13:49:04.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:04 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3404764347' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-21T13:49:04.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:04 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/130352508' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-21T13:49:04.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:04 smithi187 ceph-mon[33032]: from='client.? ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-21T13:49:04.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:04 smithi187 ceph-mon[33032]: from='client.? ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-21T13:49:04.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:04 smithi187 ceph-mon[33032]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi187=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:49:04.196 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:04 smithi187 ceph-mon[33032]: Saving service prometheus spec with placement smithi187=a;count:1 2024-09-21T13:49:04.196 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:04 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:49:04.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:04 smithi049 ceph-mon[40489]: pgmap v208: 100 pgs: 21 creating+peering, 53 active+clean, 26 unknown; 578 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2024-09-21T13:49:04.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:04 smithi049 ceph-mon[40489]: osdmap e66: 8 total, 8 up, 8 in 2024-09-21T13:49:04.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:04 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3404764347' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-21T13:49:04.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:04 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/130352508' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-21T13:49:04.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:04 smithi049 ceph-mon[40489]: from='client.? ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-21T13:49:04.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:04 smithi049 ceph-mon[40489]: from='client.? ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-21T13:49:04.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:04 smithi049 ceph-mon[40489]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi187=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:49:04.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:04 smithi049 ceph-mon[40489]: Saving service prometheus spec with placement smithi187=a;count:1 2024-09-21T13:49:04.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:04 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:49:04.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:04 smithi049 ceph-mon[30080]: pgmap v208: 100 pgs: 21 creating+peering, 53 active+clean, 26 unknown; 578 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2024-09-21T13:49:04.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:04 smithi049 ceph-mon[30080]: osdmap e66: 8 total, 8 up, 8 in 2024-09-21T13:49:04.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:04 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3404764347' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-21T13:49:04.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:04 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/130352508' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-21T13:49:04.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:04 smithi049 ceph-mon[30080]: from='client.? ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-21T13:49:04.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:04 smithi049 ceph-mon[30080]: from='client.? ' entity='mgr.y' cmd={"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"} : dispatch 2024-09-21T13:49:04.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:04 smithi049 ceph-mon[30080]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi187=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:49:04.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:04 smithi049 ceph-mon[30080]: Saving service prometheus spec with placement smithi187=a;count:1 2024-09-21T13:49:04.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:04 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:49:04.851 INFO:teuthology.orchestra.run.smithi187.stdout:Scheduled node-exporter update... 2024-09-21T13:49:05.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:05 smithi187 ceph-mon[33032]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-21T13:49:05.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:05 smithi187 ceph-mon[33032]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-21T13:49:05.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:05 smithi187 ceph-mon[33032]: osdmap e67: 8 total, 8 up, 8 in 2024-09-21T13:49:05.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:05 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3404764347' entity='client.rgw.foo.a' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-21T13:49:05.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:05 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/130352508' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-21T13:49:05.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:05 smithi187 ceph-mon[33032]: from='client.? ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-21T13:49:05.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:05 smithi187 ceph-mon[33032]: from='client.? ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-21T13:49:05.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:05 smithi187 ceph-mon[33032]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-21T13:49:05.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:05 smithi187 ceph-mon[33032]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-21T13:49:05.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:05 smithi187 ceph-mon[33032]: osdmap e68: 8 total, 8 up, 8 in 2024-09-21T13:49:05.446 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:05 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:49:05.446 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:05 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:49:05.446 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:05 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:49:05.446 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:05 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:49:05.446 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:05 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]} : dispatch 2024-09-21T13:49:05.446 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:05 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-21T13:49:05.446 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:05 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:49:05.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:05 smithi049 ceph-mon[40489]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-21T13:49:05.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:05 smithi049 ceph-mon[40489]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-21T13:49:05.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:05 smithi049 ceph-mon[40489]: osdmap e67: 8 total, 8 up, 8 in 2024-09-21T13:49:05.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:05 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3404764347' entity='client.rgw.foo.a' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-21T13:49:05.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:05 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/130352508' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-21T13:49:05.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:05 smithi049 ceph-mon[40489]: from='client.? ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-21T13:49:05.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:05 smithi049 ceph-mon[40489]: from='client.? ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-21T13:49:05.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:05 smithi049 ceph-mon[40489]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-21T13:49:05.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:05 smithi049 ceph-mon[40489]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-21T13:49:05.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:05 smithi049 ceph-mon[40489]: osdmap e68: 8 total, 8 up, 8 in 2024-09-21T13:49:05.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:05 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:49:05.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:05 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:49:05.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:05 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:49:05.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:05 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:49:05.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:05 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]} : dispatch 2024-09-21T13:49:05.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:05 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-21T13:49:05.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:05 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:49:05.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:05 smithi049 ceph-mon[30080]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-21T13:49:05.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:05 smithi049 ceph-mon[30080]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-21T13:49:05.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:05 smithi049 ceph-mon[30080]: osdmap e67: 8 total, 8 up, 8 in 2024-09-21T13:49:05.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:05 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3404764347' entity='client.rgw.foo.a' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-21T13:49:05.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:05 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/130352508' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-21T13:49:05.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:05 smithi049 ceph-mon[30080]: from='client.? ' entity='client.rgw.foo.a' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-21T13:49:05.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:05 smithi049 ceph-mon[30080]: from='client.? ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"} : dispatch 2024-09-21T13:49:05.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:05 smithi049 ceph-mon[30080]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-21T13:49:05.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:05 smithi049 ceph-mon[30080]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-21T13:49:05.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:05 smithi049 ceph-mon[30080]: osdmap e68: 8 total, 8 up, 8 in 2024-09-21T13:49:05.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:05 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:49:05.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:05 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:49:05.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:05 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:49:05.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:05 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:49:05.508 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:05 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]} : dispatch 2024-09-21T13:49:05.508 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:05 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-21T13:49:05.508 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:05 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:49:05.552 DEBUG:teuthology.orchestra.run.smithi049:node-exporter.a> sudo journalctl -f -n 0 -u ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@node-exporter.a.service 2024-09-21T13:49:05.555 DEBUG:teuthology.orchestra.run.smithi187:node-exporter.b> sudo journalctl -f -n 0 -u ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@node-exporter.b.service 2024-09-21T13:49:05.557 INFO:tasks.cephadm:Adding alertmanager.a on smithi049 2024-09-21T13:49:05.557 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph orch apply alertmanager '1;smithi049=a' 2024-09-21T13:49:06.101 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:06 smithi187 ceph-mon[33032]: pgmap v211: 132 pgs: 19 creating+peering, 75 active+clean, 38 unknown; 578 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-09-21T13:49:06.101 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:06 smithi187 ceph-mon[33032]: from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi049=a;smithi187=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:49:06.101 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:06 smithi187 ceph-mon[33032]: Saving service node-exporter spec with placement smithi049=a;smithi187=b;count:2 2024-09-21T13:49:06.101 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:06 smithi187 ceph-mon[33032]: Deploying daemon iscsi.iscsi.a on smithi187 2024-09-21T13:49:06.326 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:49:06.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:06 smithi049 ceph-mon[40489]: pgmap v211: 132 pgs: 19 creating+peering, 75 active+clean, 38 unknown; 578 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-09-21T13:49:06.511 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:06 smithi049 ceph-mon[40489]: from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi049=a;smithi187=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:49:06.511 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:06 smithi049 ceph-mon[40489]: Saving service node-exporter spec with placement smithi049=a;smithi187=b;count:2 2024-09-21T13:49:06.511 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:06 smithi049 ceph-mon[40489]: Deploying daemon iscsi.iscsi.a on smithi187 2024-09-21T13:49:06.512 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:06 smithi049 ceph-mon[30080]: pgmap v211: 132 pgs: 19 creating+peering, 75 active+clean, 38 unknown; 578 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-09-21T13:49:06.512 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:06 smithi049 ceph-mon[30080]: from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi049=a;smithi187=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:49:06.512 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:06 smithi049 ceph-mon[30080]: Saving service node-exporter spec with placement smithi049=a;smithi187=b;count:2 2024-09-21T13:49:06.512 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:06 smithi049 ceph-mon[30080]: Deploying daemon iscsi.iscsi.a on smithi187 2024-09-21T13:49:07.527 INFO:teuthology.orchestra.run.smithi187.stdout:Scheduled alertmanager update... 2024-09-21T13:49:07.854 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:49:07 smithi187 systemd[1]: Starting Ceph iscsi.iscsi.a for 40a3d2e8-781f-11ef-baf6-efdc52797490... 2024-09-21T13:49:08.137 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:08 smithi187 ceph-mon[33032]: pgmap v213: 132 pgs: 3 creating+peering, 129 active+clean; 580 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 3.2 KiB/s wr, 83 op/s 2024-09-21T13:49:08.138 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:08 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:49:08.164 DEBUG:teuthology.orchestra.run.smithi049:alertmanager.a> sudo journalctl -f -n 0 -u ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@alertmanager.a.service 2024-09-21T13:49:08.167 INFO:tasks.cephadm:Adding grafana.a on smithi187 2024-09-21T13:49:08.167 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph orch apply grafana '1;smithi187=a' 2024-09-21T13:49:08.445 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:49:08 smithi187 podman[60763]: 2024-09-21 13:49:08.265620839 +0000 UTC m=+0.021520957 image pull b415816db8b41f544cdfdc71d193ff8b1ee008bf7b0ec4f64da8eb8f71b62e85 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5 2024-09-21T13:49:08.445 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:49:08 smithi187 podman[60763]: 2024-09-21 13:49:08.363825388 +0000 UTC m=+0.119725506 container create 660ba721898d9859f179ca37d531124ce7ccfacc12af9c252f61bcf28aa2a2b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-guits-main-2024-09-21-0842-4417849, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2) 2024-09-21T13:49:08.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:08 smithi049 ceph-mon[40489]: pgmap v213: 132 pgs: 3 creating+peering, 129 active+clean; 580 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 3.2 KiB/s wr, 83 op/s 2024-09-21T13:49:08.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:08 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:49:08.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:08 smithi049 ceph-mon[30080]: pgmap v213: 132 pgs: 3 creating+peering, 129 active+clean; 580 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 3.2 KiB/s wr, 83 op/s 2024-09-21T13:49:08.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:08 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:49:08.703 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:49:08.830 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:49:08 smithi187 podman[60763]: 2024-09-21 13:49:08.565820378 +0000 UTC m=+0.321720495 container init 660ba721898d9859f179ca37d531124ce7ccfacc12af9c252f61bcf28aa2a2b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-21T13:49:08.830 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:49:08 smithi187 podman[60763]: 2024-09-21 13:49:08.570768269 +0000 UTC m=+0.326668394 container start 660ba721898d9859f179ca37d531124ce7ccfacc12af9c252f61bcf28aa2a2b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a, RELEASE=wip-guits-main-2024-09-21-0842-4417849, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-09-21T13:49:08.830 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:49:08 smithi187 bash[60763]: 660ba721898d9859f179ca37d531124ce7ccfacc12af9c252f61bcf28aa2a2b1 2024-09-21T13:49:08.830 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:49:08 smithi187 systemd[1]: Started Ceph iscsi.iscsi.a for 40a3d2e8-781f-11ef-baf6-efdc52797490. 2024-09-21T13:49:09.082 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:49:08 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug Started the configuration object watcher 2024-09-21T13:49:09.083 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:49:08 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug Checking for config object changes every 1s 2024-09-21T13:49:09.083 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:49:08 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug Processing osd blocklist entries for this node 2024-09-21T13:49:09.339 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:09 smithi187 ceph-mon[33032]: from='client.24418 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi049=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:49:09.339 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:09 smithi187 ceph-mon[33032]: Saving service alertmanager spec with placement smithi049=a;count:1 2024-09-21T13:49:09.339 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:09 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:49:09.339 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:09 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:49:09.339 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:09 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:49:09.339 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:09 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:49:09.340 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:49:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug Reading the configuration object to update local LIO configuration 2024-09-21T13:49:09.340 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:49:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug Configuration does not have an entry for this host(smithi187.front.sepia.ceph.com) - nothing to define to LIO 2024-09-21T13:49:09.340 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:49:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: * Serving Flask app 'rbd-target-api' (lazy loading) 2024-09-21T13:49:09.340 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:49:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: * Environment: production 2024-09-21T13:49:09.340 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:49:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: WARNING: This is a development server. Do not use it in a production deployment. 2024-09-21T13:49:09.340 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:49:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: Use a production WSGI server instead. 2024-09-21T13:49:09.340 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:49:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: * Debug mode: off 2024-09-21T13:49:09.340 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:49:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug * Running on all addresses. 2024-09-21T13:49:09.340 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:49:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: WARNING: This is a development server. Do not use it in a production deployment. 2024-09-21T13:49:09.340 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:49:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: * Running on all addresses. 2024-09-21T13:49:09.340 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:49:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: WARNING: This is a development server. Do not use it in a production deployment. 2024-09-21T13:49:09.340 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:49:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug * Running on http://[fe80::ec4:7aff:fe88:77b1]:5000/ (Press CTRL+C to quit) 2024-09-21T13:49:09.340 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:49:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: * Running on http://[fe80::ec4:7aff:fe88:77b1]:5000/ (Press CTRL+C to quit) 2024-09-21T13:49:09.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:09 smithi049 ceph-mon[40489]: from='client.24418 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi049=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:49:09.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:09 smithi049 ceph-mon[40489]: Saving service alertmanager spec with placement smithi049=a;count:1 2024-09-21T13:49:09.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:09 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:49:09.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:09 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:49:09.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:09 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:49:09.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:09 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:49:09.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:09 smithi049 ceph-mon[30080]: from='client.24418 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi049=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:49:09.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:09 smithi049 ceph-mon[30080]: Saving service alertmanager spec with placement smithi049=a;count:1 2024-09-21T13:49:09.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:09 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:49:09.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:09 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:49:09.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:09 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:49:09.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:09 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:49:09.998 INFO:teuthology.orchestra.run.smithi187.stdout:Scheduled grafana update... 2024-09-21T13:49:10.280 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:10 smithi187 ceph-mon[33032]: pgmap v214: 132 pgs: 3 creating+peering, 129 active+clean; 580 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 2.7 KiB/s wr, 82 op/s 2024-09-21T13:49:10.281 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:10 smithi187 ceph-mon[33032]: Checking pool "datapool" exists for service iscsi.datapool 2024-09-21T13:49:10.281 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:10 smithi187 ceph-mon[33032]: Deploying daemon prometheus.a on smithi187 2024-09-21T13:49:10.281 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:10 smithi187 ceph-mon[33032]: from='client.? 172.21.15.187:0/342375722' entity='client.iscsi.iscsi.a' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-21T13:49:10.281 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:10 smithi187 ceph-mon[33032]: osdmap e69: 8 total, 8 up, 8 in 2024-09-21T13:49:10.281 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:10 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:49:10.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:10 smithi049 ceph-mon[40489]: pgmap v214: 132 pgs: 3 creating+peering, 129 active+clean; 580 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 2.7 KiB/s wr, 82 op/s 2024-09-21T13:49:10.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:10 smithi049 ceph-mon[40489]: Checking pool "datapool" exists for service iscsi.datapool 2024-09-21T13:49:10.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:10 smithi049 ceph-mon[40489]: Deploying daemon prometheus.a on smithi187 2024-09-21T13:49:10.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:10 smithi049 ceph-mon[40489]: from='client.? 172.21.15.187:0/342375722' entity='client.iscsi.iscsi.a' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-21T13:49:10.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:10 smithi049 ceph-mon[40489]: osdmap e69: 8 total, 8 up, 8 in 2024-09-21T13:49:10.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:10 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:49:10.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:10 smithi049 ceph-mon[30080]: pgmap v214: 132 pgs: 3 creating+peering, 129 active+clean; 580 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 2.7 KiB/s wr, 82 op/s 2024-09-21T13:49:10.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:10 smithi049 ceph-mon[30080]: Checking pool "datapool" exists for service iscsi.datapool 2024-09-21T13:49:10.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:10 smithi049 ceph-mon[30080]: Deploying daemon prometheus.a on smithi187 2024-09-21T13:49:10.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:10 smithi049 ceph-mon[30080]: from='client.? 172.21.15.187:0/342375722' entity='client.iscsi.iscsi.a' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-21T13:49:10.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:10 smithi049 ceph-mon[30080]: osdmap e69: 8 total, 8 up, 8 in 2024-09-21T13:49:10.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:10 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:49:10.700 DEBUG:teuthology.orchestra.run.smithi187:grafana.a> sudo journalctl -f -n 0 -u ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@grafana.a.service 2024-09-21T13:49:10.702 INFO:tasks.cephadm:Setting up client nodes... 2024-09-21T13:49:10.703 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-21T13:49:10.969 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:49:11.363 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:11 smithi187 ceph-mon[33032]: from='client.24454 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi187=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:49:11.363 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:11 smithi187 ceph-mon[33032]: Saving service grafana spec with placement smithi187=a;count:1 2024-09-21T13:49:11.387 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:11 smithi049 ceph-mon[40489]: from='client.24454 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi187=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:49:11.387 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:11 smithi049 ceph-mon[40489]: Saving service grafana spec with placement smithi187=a;count:1 2024-09-21T13:49:11.387 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:11 smithi049 ceph-mon[30080]: from='client.24454 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi187=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:49:11.387 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:11 smithi049 ceph-mon[30080]: Saving service grafana spec with placement smithi187=a;count:1 2024-09-21T13:49:11.753 INFO:teuthology.orchestra.run.smithi049.stdout:[client.0] 2024-09-21T13:49:11.753 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQDXzu5mKd7SLBAA6mukW14oyabNUBDGu7OTuw== 2024-09-21T13:49:12.416 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:12 smithi049 ceph-mon[40489]: pgmap v216: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 5.1 KiB/s wr, 168 op/s 2024-09-21T13:49:12.416 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:12 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:49:12.416 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:12 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:49:12.416 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:12 smithi049 ceph-mon[40489]: mgrmap e17: y(active, since 5m), standbys: x 2024-09-21T13:49:12.416 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:12 smithi049 ceph-mon[40489]: osdmap e70: 8 total, 8 up, 8 in 2024-09-21T13:49:12.416 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:12 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/377563311' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-21T13:49:12.416 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:12 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/377563311' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T13:49:12.417 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:12 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:49:12.417 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:12 smithi049 ceph-mon[30080]: pgmap v216: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 5.1 KiB/s wr, 168 op/s 2024-09-21T13:49:12.417 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:12 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:49:12.417 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:12 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:49:12.417 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:12 smithi049 ceph-mon[30080]: mgrmap e17: y(active, since 5m), standbys: x 2024-09-21T13:49:12.417 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:12 smithi049 ceph-mon[30080]: osdmap e70: 8 total, 8 up, 8 in 2024-09-21T13:49:12.417 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:12 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/377563311' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-21T13:49:12.417 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:12 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/377563311' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T13:49:12.417 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:12 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:49:12.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:12 smithi187 ceph-mon[33032]: pgmap v216: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 5.1 KiB/s wr, 168 op/s 2024-09-21T13:49:12.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:12 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:49:12.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:12 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:49:12.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:12 smithi187 ceph-mon[33032]: mgrmap e17: y(active, since 5m), standbys: x 2024-09-21T13:49:12.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:12 smithi187 ceph-mon[33032]: osdmap e70: 8 total, 8 up, 8 in 2024-09-21T13:49:12.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:12 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/377563311' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-21T13:49:12.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:12 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/377563311' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T13:49:12.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:12 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:49:12.446 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-09-21T13:49:12.446 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-21T13:49:12.446 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-21T13:49:12.485 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-21T13:49:14.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:14 smithi187 ceph-mon[33032]: pgmap v218: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.4 KiB/s wr, 142 op/s 2024-09-21T13:49:14.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:14 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:49:14.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:14 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:49:14.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:14 smithi187 ceph-mon[33032]: osdmap e71: 8 total, 8 up, 8 in 2024-09-21T13:49:14.464 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:49:14.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:14 smithi049 ceph-mon[30080]: pgmap v218: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.4 KiB/s wr, 142 op/s 2024-09-21T13:49:14.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:14 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:49:14.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:14 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:49:14.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:14 smithi049 ceph-mon[30080]: osdmap e71: 8 total, 8 up, 8 in 2024-09-21T13:49:14.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:14 smithi049 ceph-mon[40489]: pgmap v218: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.4 KiB/s wr, 142 op/s 2024-09-21T13:49:14.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:14 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:49:14.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:14 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:49:14.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:14 smithi049 ceph-mon[40489]: osdmap e71: 8 total, 8 up, 8 in 2024-09-21T13:49:16.079 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:15 smithi187 ceph-mon[33032]: pgmap v220: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 3.3 KiB/s wr, 105 op/s 2024-09-21T13:49:16.080 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:15 smithi187 ceph-mon[33032]: osdmap e72: 8 total, 8 up, 8 in 2024-09-21T13:49:16.148 INFO:teuthology.orchestra.run.smithi187.stdout:[client.1] 2024-09-21T13:49:16.149 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQDczu5mu1+PCBAAvfcP2gZcuJthWBjqtjx54g== 2024-09-21T13:49:16.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:15 smithi049 ceph-mon[30080]: pgmap v220: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 3.3 KiB/s wr, 105 op/s 2024-09-21T13:49:16.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:15 smithi049 ceph-mon[30080]: osdmap e72: 8 total, 8 up, 8 in 2024-09-21T13:49:16.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:15 smithi049 ceph-mon[40489]: pgmap v220: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 3.3 KiB/s wr, 105 op/s 2024-09-21T13:49:16.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:15 smithi049 ceph-mon[40489]: osdmap e72: 8 total, 8 up, 8 in 2024-09-21T13:49:17.053 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:16 smithi187 ceph-mon[33032]: from='client.? 172.21.15.187:0/3683226446' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-21T13:49:17.053 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:16 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-21T13:49:17.053 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:16 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T13:49:17.082 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-09-21T13:49:17.083 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-09-21T13:49:17.083 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-21T13:49:17.122 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph config log 1 --format=json 2024-09-21T13:49:17.154 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:16 smithi049 ceph-mon[40489]: from='client.? 172.21.15.187:0/3683226446' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-21T13:49:17.154 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:16 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-21T13:49:17.154 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:16 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T13:49:17.154 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:16 smithi049 ceph-mon[30080]: from='client.? 172.21.15.187:0/3683226446' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-21T13:49:17.154 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:16 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-21T13:49:17.155 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:16 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T13:49:17.393 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:49:18.045 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:49:18.262 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:18 smithi187 ceph-mon[33032]: pgmap v222: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 170 B/s wr, 2 op/s 2024-09-21T13:49:18.262 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:18 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:49:18.262 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:18 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3489519975' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-21T13:49:18.339 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:18 smithi049 ceph-mon[30080]: pgmap v222: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 170 B/s wr, 2 op/s 2024-09-21T13:49:18.339 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:18 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:49:18.339 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:18 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3489519975' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-21T13:49:18.339 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:18 smithi049 ceph-mon[40489]: pgmap v222: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 170 B/s wr, 2 op/s 2024-09-21T13:49:18.340 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:18 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:49:18.340 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:18 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3489519975' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-21T13:49:18.862 INFO:teuthology.orchestra.run.smithi049.stdout:[{"version":23,"timestamp":"2024-09-21T13:49:05.031534+0000","name":"","changes":[{"name":"mgr/mgr/dashboard/RGW_API_SECRET_KEY","new_value":"kECXBuc5FkTYtkxiABg4GE0s5ct7KqHXiotCA1uk"}]}] 2024-09-21T13:49:18.863 INFO:tasks.ceph_manager:config epoch is 23 2024-09-21T13:49:18.863 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-21T13:49:18.863 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-21T13:49:18.863 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph mgr dump --format=json 2024-09-21T13:49:19.133 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:49:18 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:49:19.162 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:49:19.881 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:49:20.091 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:49:19 smithi187 systemd[1]: Starting Ceph prometheus.a for 40a3d2e8-781f-11ef-baf6-efdc52797490... 2024-09-21T13:49:20.246 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:20 smithi049 ceph-mon[40489]: pgmap v223: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 134 B/s wr, 2 op/s 2024-09-21T13:49:20.246 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:20 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:49:20.246 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:20 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/610232730' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-21T13:49:20.246 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:20 smithi049 ceph-mon[30080]: pgmap v223: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 134 B/s wr, 2 op/s 2024-09-21T13:49:20.246 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:20 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:49:20.246 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:20 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/610232730' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-21T13:49:20.343 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:20 smithi187 ceph-mon[33032]: pgmap v223: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 134 B/s wr, 2 op/s 2024-09-21T13:49:20.343 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:20 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:49:20.343 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:20 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/610232730' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-21T13:49:20.343 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:49:20 smithi187 podman[61805]: 2024-09-21 13:49:20.083589827 +0000 UTC m=+0.018117843 image pull 1d3b7f56885b6dd623f1785be963aa9c195f86bc256ea454e8d02a7980b79c53 quay.io/prometheus/prometheus:v2.51.0 2024-09-21T13:49:20.343 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:49:20 smithi187 podman[61805]: 2024-09-21 13:49:20.194707224 +0000 UTC m=+0.129235242 container create f6d988baca2083ec6576667c7fc0aa84f7e284effa48c99afc12971935f1cd27 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T13:49:20.625 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":3453571598},{"type":"v1","addr":"172.21.15.49:6801","nonce":3453571598}]},"active_addr":"172.21.15.49:6801/3453571598","active_change":"2024-09-21T13:43:56.633869+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[{"gid":24109,"name":"x","mgr_features":4540719139924082687,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.8","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.8","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.49:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.49:0","nonce":1249926115}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.49:0","nonce":4220781134}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.49:0","nonce":2731793917}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.49:0","nonce":3553734432}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.49:0","nonce":799831425}]}]} 2024-09-21T13:49:20.629 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-21T13:49:20.630 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-21T13:49:20.630 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd dump --format=json 2024-09-21T13:49:20.695 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:49:20 smithi187 podman[61805]: 2024-09-21 13:49:20.438661865 +0000 UTC m=+0.373189898 container init f6d988baca2083ec6576667c7fc0aa84f7e284effa48c99afc12971935f1cd27 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T13:49:20.695 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:49:20 smithi187 podman[61805]: 2024-09-21 13:49:20.44365157 +0000 UTC m=+0.378179587 container start f6d988baca2083ec6576667c7fc0aa84f7e284effa48c99afc12971935f1cd27 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T13:49:20.695 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:49:20 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[61820]: ts=2024-09-21T13:49:20.463Z caller=main.go:617 level=info msg="Starting Prometheus Server" mode=server version="(version=2.51.0, branch=HEAD, revision=c05c15512acb675e3f6cd662a6727854e93fc024)" 2024-09-21T13:49:20.696 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:49:20 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[61820]: ts=2024-09-21T13:49:20.463Z caller=main.go:622 level=info build_context="(go=go1.22.1, platform=linux/amd64, user=root@b5723e458358, date=20240319-10:54:45, tags=netgo,builtinassets,stringlabels)" 2024-09-21T13:49:20.696 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:49:20 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[61820]: ts=2024-09-21T13:49:20.463Z caller=main.go:623 level=info host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi187 (none))" 2024-09-21T13:49:20.696 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:49:20 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[61820]: ts=2024-09-21T13:49:20.463Z caller=main.go:624 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-21T13:49:20.696 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:49:20 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[61820]: ts=2024-09-21T13:49:20.463Z caller=main.go:625 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-21T13:49:20.696 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:49:20 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[61820]: ts=2024-09-21T13:49:20.464Z caller=web.go:568 level=info component=web msg="Start listening for connections" address=:9095 2024-09-21T13:49:20.696 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:49:20 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[61820]: ts=2024-09-21T13:49:20.465Z caller=main.go:1129 level=info msg="Starting TSDB ..." 2024-09-21T13:49:20.696 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:49:20 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[61820]: ts=2024-09-21T13:49:20.467Z caller=tls_config.go:313 level=info component=web msg="Listening on" address=[::]:9095 2024-09-21T13:49:20.696 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:49:20 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[61820]: ts=2024-09-21T13:49:20.467Z caller=tls_config.go:316 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-21T13:49:20.696 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:49:20 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[61820]: ts=2024-09-21T13:49:20.468Z caller=head.go:616 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-21T13:49:20.696 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:49:20 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[61820]: ts=2024-09-21T13:49:20.468Z caller=head.go:698 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.554µs 2024-09-21T13:49:20.696 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:49:20 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[61820]: ts=2024-09-21T13:49:20.468Z caller=head.go:706 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-21T13:49:20.696 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:49:20 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[61820]: ts=2024-09-21T13:49:20.468Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-09-21T13:49:20.696 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:49:20 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[61820]: ts=2024-09-21T13:49:20.468Z caller=head.go:815 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=21.688µs wal_replay_duration=143.973µs wbl_replay_duration=110ns total_replay_duration=183.337µs 2024-09-21T13:49:20.696 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:49:20 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[61820]: ts=2024-09-21T13:49:20.470Z caller=main.go:1150 level=info fs_type=XFS_SUPER_MAGIC 2024-09-21T13:49:20.697 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:49:20 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[61820]: ts=2024-09-21T13:49:20.470Z caller=main.go:1153 level=info msg="TSDB started" 2024-09-21T13:49:20.697 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:49:20 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[61820]: ts=2024-09-21T13:49:20.470Z caller=main.go:1335 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-21T13:49:20.697 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:49:20 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[61820]: ts=2024-09-21T13:49:20.485Z caller=main.go:1372 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=15.274695ms db_storage=768ns remote_storage=1.119µs web_handler=306ns query_engine=672ns scrape=380.476µs scrape_sd=91.649µs notify=932ns notify_sd=1.028µs rules=14.491544ms tracing=6.422µs 2024-09-21T13:49:20.697 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:49:20 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[61820]: ts=2024-09-21T13:49:20.485Z caller=main.go:1114 level=info msg="Server is ready to receive web requests." 2024-09-21T13:49:20.697 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:49:20 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[61820]: ts=2024-09-21T13:49:20.485Z caller=manager.go:163 level=info component="rule manager" msg="Starting rule manager..." 2024-09-21T13:49:20.697 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:49:20 smithi187 bash[61805]: f6d988baca2083ec6576667c7fc0aa84f7e284effa48c99afc12971935f1cd27 2024-09-21T13:49:20.697 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:49:20 smithi187 systemd[1]: Started Ceph prometheus.a for 40a3d2e8-781f-11ef-baf6-efdc52797490. 2024-09-21T13:49:20.884 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:49:21.582 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:49:21.582 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":72,"fsid":"40a3d2e8-781f-11ef-baf6-efdc52797490","created":"2024-09-21T13:42:57.838275+0000","modified":"2024-09-21T13:49:14.796992+0000","last_up_change":"2024-09-21T13:48:46.439884+0000","last_in_change":"2024-09-21T13:48:23.514355+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T13:46:21.096244+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-09-21T13:48:55.540712+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":63,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-09-21T13:48:56.417456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-21T13:48:58.128455+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-21T13:49:00.069361+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-21T13:49:02.127399+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2595dca9-6219-4e0c-8728-751eb102e73a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":2095561412},{"type":"v1","addr":"172.21.15.49:6803","nonce":2095561412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":2095561412},{"type":"v1","addr":"172.21.15.49:6805","nonce":2095561412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":2095561412},{"type":"v1","addr":"172.21.15.49:6809","nonce":2095561412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":2095561412},{"type":"v1","addr":"172.21.15.49:6807","nonce":2095561412}]},"public_addr":"172.21.15.49:6803/2095561412","cluster_addr":"172.21.15.49:6805/2095561412","heartbeat_back_addr":"172.21.15.49:6809/2095561412","heartbeat_front_addr":"172.21.15.49:6807/2095561412","state":["exists","up"]},{"osd":1,"uuid":"0ba327f9-5524-4a40-abcc-729ded0b1917","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":1382840527},{"type":"v1","addr":"172.21.15.49:6811","nonce":1382840527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":1382840527},{"type":"v1","addr":"172.21.15.49:6813","nonce":1382840527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":1382840527},{"type":"v1","addr":"172.21.15.49:6817","nonce":1382840527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":1382840527},{"type":"v1","addr":"172.21.15.49:6815","nonce":1382840527}]},"public_addr":"172.21.15.49:6811/1382840527","cluster_addr":"172.21.15.49:6813/1382840527","heartbeat_back_addr":"172.21.15.49:6817/1382840527","heartbeat_front_addr":"172.21.15.49:6815/1382840527","state":["exists","up"]},{"osd":2,"uuid":"27a45750-204d-48fd-bb8f-ebb01273343d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":2189046898},{"type":"v1","addr":"172.21.15.49:6819","nonce":2189046898}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":2189046898},{"type":"v1","addr":"172.21.15.49:6821","nonce":2189046898}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2189046898},{"type":"v1","addr":"172.21.15.49:6825","nonce":2189046898}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":2189046898},{"type":"v1","addr":"172.21.15.49:6823","nonce":2189046898}]},"public_addr":"172.21.15.49:6819/2189046898","cluster_addr":"172.21.15.49:6821/2189046898","heartbeat_back_addr":"172.21.15.49:6825/2189046898","heartbeat_front_addr":"172.21.15.49:6823/2189046898","state":["exists","up"]},{"osd":3,"uuid":"528bc390-3dc9-4f3b-ab75-d7412c9625db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":1981113123},{"type":"v1","addr":"172.21.15.49:6827","nonce":1981113123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":1981113123},{"type":"v1","addr":"172.21.15.49:6829","nonce":1981113123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":1981113123},{"type":"v1","addr":"172.21.15.49:6833","nonce":1981113123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":1981113123},{"type":"v1","addr":"172.21.15.49:6831","nonce":1981113123}]},"public_addr":"172.21.15.49:6827/1981113123","cluster_addr":"172.21.15.49:6829/1981113123","heartbeat_back_addr":"172.21.15.49:6833/1981113123","heartbeat_front_addr":"172.21.15.49:6831/1981113123","state":["exists","up"]},{"osd":4,"uuid":"6fc64698-2d37-4db0-bf3d-c7e612c45163","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":1702447776},{"type":"v1","addr":"172.21.15.187:6801","nonce":1702447776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":1702447776},{"type":"v1","addr":"172.21.15.187:6803","nonce":1702447776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":1702447776},{"type":"v1","addr":"172.21.15.187:6807","nonce":1702447776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":1702447776},{"type":"v1","addr":"172.21.15.187:6805","nonce":1702447776}]},"public_addr":"172.21.15.187:6801/1702447776","cluster_addr":"172.21.15.187:6803/1702447776","heartbeat_back_addr":"172.21.15.187:6807/1702447776","heartbeat_front_addr":"172.21.15.187:6805/1702447776","state":["exists","up"]},{"osd":5,"uuid":"8d7f62a1-7e92-4e06-9cad-86b96be514d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3659050776},{"type":"v1","addr":"172.21.15.187:6809","nonce":3659050776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3659050776},{"type":"v1","addr":"172.21.15.187:6811","nonce":3659050776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3659050776},{"type":"v1","addr":"172.21.15.187:6815","nonce":3659050776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3659050776},{"type":"v1","addr":"172.21.15.187:6813","nonce":3659050776}]},"public_addr":"172.21.15.187:6809/3659050776","cluster_addr":"172.21.15.187:6811/3659050776","heartbeat_back_addr":"172.21.15.187:6815/3659050776","heartbeat_front_addr":"172.21.15.187:6813/3659050776","state":["exists","up"]},{"osd":6,"uuid":"3a7afd13-26ce-444c-9cc0-81ce2eab6ce5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":2728426525},{"type":"v1","addr":"172.21.15.187:6817","nonce":2728426525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":2728426525},{"type":"v1","addr":"172.21.15.187:6819","nonce":2728426525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":2728426525},{"type":"v1","addr":"172.21.15.187:6823","nonce":2728426525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":2728426525},{"type":"v1","addr":"172.21.15.187:6821","nonce":2728426525}]},"public_addr":"172.21.15.187:6817/2728426525","cluster_addr":"172.21.15.187:6819/2728426525","heartbeat_back_addr":"172.21.15.187:6823/2728426525","heartbeat_front_addr":"172.21.15.187:6821/2728426525","state":["exists","up"]},{"osd":7,"uuid":"1e08d891-8747-402e-b4c2-aa3802ce8537","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":3680850037},{"type":"v1","addr":"172.21.15.187:6825","nonce":3680850037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":3680850037},{"type":"v1","addr":"172.21.15.187:6827","nonce":3680850037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":3680850037},{"type":"v1","addr":"172.21.15.187:6831","nonce":3680850037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":3680850037},{"type":"v1","addr":"172.21.15.187:6829","nonce":3680850037}]},"public_addr":"172.21.15.187:6825/3680850037","cluster_addr":"172.21.15.187:6827/3680850037","heartbeat_back_addr":"172.21.15.187:6831/3680850037","heartbeat_front_addr":"172.21.15.187:6829/3680850037","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T13:49:10.973314+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T13:49:12.923926+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T13:48:49.018318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T13:48:16.805740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T13:48:36.764542+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T13:48:49.874648+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T13:48:14.651488+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T13:48:44.977550+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.49:0/3600977363":"2024-09-22T13:43:56.633723+0000","172.21.15.49:0/3125743776":"2024-09-22T13:43:56.633723+0000","172.21.15.49:6801/4129636473":"2024-09-22T13:43:56.633723+0000","172.21.15.49:6800/4129636473":"2024-09-22T13:43:56.633723+0000","172.21.15.49:0/202325455":"2024-09-22T13:43:28.110499+0000","172.21.15.49:0/3514741273":"2024-09-22T13:43:56.633723+0000","172.21.15.49:0/231530138":"2024-09-22T13:43:28.110499+0000","172.21.15.49:0/3556829486":"2024-09-22T13:43:28.110499+0000","172.21.15.49:6801/2518197280":"2024-09-22T13:43:28.110499+0000","172.21.15.49:6800/2518197280":"2024-09-22T13:43:28.110499+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T13:49:21.913 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:49:21 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ignoring --setuser ceph since I am not root 2024-09-21T13:49:21.913 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:49:21 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ignoring --setgroup ceph since I am not root 2024-09-21T13:49:21.914 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:21 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:49:21.914 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:21 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:49:21.914 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:21 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:49:21.914 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:21 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mgr module enable", "module": "prometheus"} : dispatch 2024-09-21T13:49:21.914 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:21 smithi049 ceph-mon[40489]: pgmap v224: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-21T13:49:21.914 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:21 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3166448355' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T13:49:21.914 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:21 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:49:21.914 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:21 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:49:21.914 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:21 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:49:21.914 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:21 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mgr module enable", "module": "prometheus"} : dispatch 2024-09-21T13:49:21.914 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:21 smithi049 ceph-mon[30080]: pgmap v224: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-21T13:49:21.915 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:21 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3166448355' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T13:49:21.944 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Sep 21 13:49:21 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-x[34691]: ignoring --setuser ceph since I am not root 2024-09-21T13:49:21.945 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Sep 21 13:49:21 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-x[34691]: ignoring --setgroup ceph since I am not root 2024-09-21T13:49:21.945 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Sep 21 13:49:21 smithi187 ceph-mgr[34714]: -- 172.21.15.187:0/1243448145 <== mon.2 v2:172.21.15.49:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55eeb6bb90e0 con 0x55eeb6b96800 2024-09-21T13:49:21.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:21 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:49:21.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:21 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:49:21.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:21 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' 2024-09-21T13:49:21.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:21 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd={"prefix": "mgr module enable", "module": "prometheus"} : dispatch 2024-09-21T13:49:21.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:21 smithi187 ceph-mon[33032]: pgmap v224: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-21T13:49:21.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:21 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3166448355' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T13:49:22.352 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-21T13:49:22.353 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd dump --format=json 2024-09-21T13:49:22.628 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:49:22.867 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:22 smithi049 ceph-mon[40489]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-21T13:49:22.867 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:22 smithi049 ceph-mon[40489]: mgrmap e18: y(active, since 5m), standbys: x 2024-09-21T13:49:22.867 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:22 smithi049 ceph-mon[30080]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-21T13:49:22.867 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:22 smithi049 ceph-mon[30080]: mgrmap e18: y(active, since 5m), standbys: x 2024-09-21T13:49:22.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:22 smithi187 ceph-mon[33032]: from='mgr.14152 172.21.15.49:0/1459567586' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-21T13:49:22.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:22 smithi187 ceph-mon[33032]: mgrmap e18: y(active, since 5m), standbys: x 2024-09-21T13:49:23.303 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:49:23.304 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":73,"fsid":"40a3d2e8-781f-11ef-baf6-efdc52797490","created":"2024-09-21T13:42:57.838275+0000","modified":"2024-09-21T13:49:22.601464+0000","last_up_change":"2024-09-21T13:48:46.439884+0000","last_in_change":"2024-09-21T13:48:23.514355+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T13:46:21.096244+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-09-21T13:48:55.540712+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":63,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-09-21T13:48:56.417456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-21T13:48:58.128455+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-21T13:49:00.069361+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-21T13:49:02.127399+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2595dca9-6219-4e0c-8728-751eb102e73a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":2095561412},{"type":"v1","addr":"172.21.15.49:6803","nonce":2095561412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":2095561412},{"type":"v1","addr":"172.21.15.49:6805","nonce":2095561412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":2095561412},{"type":"v1","addr":"172.21.15.49:6809","nonce":2095561412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":2095561412},{"type":"v1","addr":"172.21.15.49:6807","nonce":2095561412}]},"public_addr":"172.21.15.49:6803/2095561412","cluster_addr":"172.21.15.49:6805/2095561412","heartbeat_back_addr":"172.21.15.49:6809/2095561412","heartbeat_front_addr":"172.21.15.49:6807/2095561412","state":["exists","up"]},{"osd":1,"uuid":"0ba327f9-5524-4a40-abcc-729ded0b1917","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":1382840527},{"type":"v1","addr":"172.21.15.49:6811","nonce":1382840527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":1382840527},{"type":"v1","addr":"172.21.15.49:6813","nonce":1382840527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":1382840527},{"type":"v1","addr":"172.21.15.49:6817","nonce":1382840527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":1382840527},{"type":"v1","addr":"172.21.15.49:6815","nonce":1382840527}]},"public_addr":"172.21.15.49:6811/1382840527","cluster_addr":"172.21.15.49:6813/1382840527","heartbeat_back_addr":"172.21.15.49:6817/1382840527","heartbeat_front_addr":"172.21.15.49:6815/1382840527","state":["exists","up"]},{"osd":2,"uuid":"27a45750-204d-48fd-bb8f-ebb01273343d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":2189046898},{"type":"v1","addr":"172.21.15.49:6819","nonce":2189046898}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":2189046898},{"type":"v1","addr":"172.21.15.49:6821","nonce":2189046898}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":2189046898},{"type":"v1","addr":"172.21.15.49:6825","nonce":2189046898}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":2189046898},{"type":"v1","addr":"172.21.15.49:6823","nonce":2189046898}]},"public_addr":"172.21.15.49:6819/2189046898","cluster_addr":"172.21.15.49:6821/2189046898","heartbeat_back_addr":"172.21.15.49:6825/2189046898","heartbeat_front_addr":"172.21.15.49:6823/2189046898","state":["exists","up"]},{"osd":3,"uuid":"528bc390-3dc9-4f3b-ab75-d7412c9625db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":1981113123},{"type":"v1","addr":"172.21.15.49:6827","nonce":1981113123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":1981113123},{"type":"v1","addr":"172.21.15.49:6829","nonce":1981113123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":1981113123},{"type":"v1","addr":"172.21.15.49:6833","nonce":1981113123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":1981113123},{"type":"v1","addr":"172.21.15.49:6831","nonce":1981113123}]},"public_addr":"172.21.15.49:6827/1981113123","cluster_addr":"172.21.15.49:6829/1981113123","heartbeat_back_addr":"172.21.15.49:6833/1981113123","heartbeat_front_addr":"172.21.15.49:6831/1981113123","state":["exists","up"]},{"osd":4,"uuid":"6fc64698-2d37-4db0-bf3d-c7e612c45163","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":1702447776},{"type":"v1","addr":"172.21.15.187:6801","nonce":1702447776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":1702447776},{"type":"v1","addr":"172.21.15.187:6803","nonce":1702447776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":1702447776},{"type":"v1","addr":"172.21.15.187:6807","nonce":1702447776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":1702447776},{"type":"v1","addr":"172.21.15.187:6805","nonce":1702447776}]},"public_addr":"172.21.15.187:6801/1702447776","cluster_addr":"172.21.15.187:6803/1702447776","heartbeat_back_addr":"172.21.15.187:6807/1702447776","heartbeat_front_addr":"172.21.15.187:6805/1702447776","state":["exists","up"]},{"osd":5,"uuid":"8d7f62a1-7e92-4e06-9cad-86b96be514d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3659050776},{"type":"v1","addr":"172.21.15.187:6809","nonce":3659050776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3659050776},{"type":"v1","addr":"172.21.15.187:6811","nonce":3659050776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3659050776},{"type":"v1","addr":"172.21.15.187:6815","nonce":3659050776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3659050776},{"type":"v1","addr":"172.21.15.187:6813","nonce":3659050776}]},"public_addr":"172.21.15.187:6809/3659050776","cluster_addr":"172.21.15.187:6811/3659050776","heartbeat_back_addr":"172.21.15.187:6815/3659050776","heartbeat_front_addr":"172.21.15.187:6813/3659050776","state":["exists","up"]},{"osd":6,"uuid":"3a7afd13-26ce-444c-9cc0-81ce2eab6ce5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":2728426525},{"type":"v1","addr":"172.21.15.187:6817","nonce":2728426525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":2728426525},{"type":"v1","addr":"172.21.15.187:6819","nonce":2728426525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":2728426525},{"type":"v1","addr":"172.21.15.187:6823","nonce":2728426525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":2728426525},{"type":"v1","addr":"172.21.15.187:6821","nonce":2728426525}]},"public_addr":"172.21.15.187:6817/2728426525","cluster_addr":"172.21.15.187:6819/2728426525","heartbeat_back_addr":"172.21.15.187:6823/2728426525","heartbeat_front_addr":"172.21.15.187:6821/2728426525","state":["exists","up"]},{"osd":7,"uuid":"1e08d891-8747-402e-b4c2-aa3802ce8537","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":3680850037},{"type":"v1","addr":"172.21.15.187:6825","nonce":3680850037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":3680850037},{"type":"v1","addr":"172.21.15.187:6827","nonce":3680850037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":3680850037},{"type":"v1","addr":"172.21.15.187:6831","nonce":3680850037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":3680850037},{"type":"v1","addr":"172.21.15.187:6829","nonce":3680850037}]},"public_addr":"172.21.15.187:6825/3680850037","cluster_addr":"172.21.15.187:6827/3680850037","heartbeat_back_addr":"172.21.15.187:6831/3680850037","heartbeat_front_addr":"172.21.15.187:6829/3680850037","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T13:49:10.973314+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T13:49:12.923926+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T13:48:49.018318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T13:48:16.805740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T13:48:36.764542+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T13:48:49.874648+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T13:49:22.303774+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T13:48:44.977550+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.49:0/3600977363":"2024-09-22T13:43:56.633723+0000","172.21.15.49:0/3125743776":"2024-09-22T13:43:56.633723+0000","172.21.15.49:6801/4129636473":"2024-09-22T13:43:56.633723+0000","172.21.15.49:6800/4129636473":"2024-09-22T13:43:56.633723+0000","172.21.15.49:0/202325455":"2024-09-22T13:43:28.110499+0000","172.21.15.49:0/3514741273":"2024-09-22T13:43:56.633723+0000","172.21.15.49:0/231530138":"2024-09-22T13:43:28.110499+0000","172.21.15.49:0/3556829486":"2024-09-22T13:43:28.110499+0000","172.21.15.49:6801/2518197280":"2024-09-22T13:43:28.110499+0000","172.21.15.49:6800/2518197280":"2024-09-22T13:43:28.110499+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T13:49:23.921 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:23 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:49:23.921 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:23 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:49:23.921 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:23 smithi049 ceph-mon[40489]: osdmap e73: 8 total, 8 up, 8 in 2024-09-21T13:49:23.921 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:23 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/114888012' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T13:49:23.921 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:23 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:49:23.921 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:23 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:49:23.921 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:23 smithi049 ceph-mon[30080]: osdmap e73: 8 total, 8 up, 8 in 2024-09-21T13:49:23.921 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:23 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/114888012' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T13:49:23.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:23 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:49:23.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:23 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:49:23.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:23 smithi187 ceph-mon[33032]: osdmap e73: 8 total, 8 up, 8 in 2024-09-21T13:49:23.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:23 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/114888012' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T13:49:23.951 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph tell osd.0 flush_pg_stats 2024-09-21T13:49:23.951 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph tell osd.1 flush_pg_stats 2024-09-21T13:49:23.952 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph tell osd.2 flush_pg_stats 2024-09-21T13:49:23.952 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph tell osd.3 flush_pg_stats 2024-09-21T13:49:23.952 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph tell osd.4 flush_pg_stats 2024-09-21T13:49:23.953 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph tell osd.5 flush_pg_stats 2024-09-21T13:49:23.953 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph tell osd.6 flush_pg_stats 2024-09-21T13:49:23.953 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph tell osd.7 flush_pg_stats 2024-09-21T13:49:24.494 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:49:24.494 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:49:24.494 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:49:24.499 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:49:25.344 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:49:25.344 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:49:25.345 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:49:25.347 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:49:26.160 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:25 smithi049 ceph-mon[40489]: osdmap e74: 8 total, 8 up, 8 in 2024-09-21T13:49:26.160 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:25 smithi049 ceph-mon[30080]: osdmap e74: 8 total, 8 up, 8 in 2024-09-21T13:49:26.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:25 smithi187 ceph-mon[33032]: osdmap e74: 8 total, 8 up, 8 in 2024-09-21T13:49:26.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:26 smithi187 ceph-mon[33032]: Standby manager daemon x restarted 2024-09-21T13:49:26.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:26 smithi187 ceph-mon[33032]: Standby manager daemon x started 2024-09-21T13:49:26.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:26 smithi187 ceph-mon[33032]: from='mgr.? 172.21.15.187:0/3654822744' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-21T13:49:26.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:26 smithi187 ceph-mon[33032]: from='mgr.? 172.21.15.187:0/3654822744' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-21T13:49:26.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:26 smithi187 ceph-mon[33032]: from='mgr.? 172.21.15.187:0/3654822744' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-21T13:49:26.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:26 smithi187 ceph-mon[33032]: from='mgr.? 172.21.15.187:0/3654822744' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-21T13:49:26.945 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Sep 21 13:49:26 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-x[34691]: [21/Sep/2024:13:49:26] ENGINE Bus STARTING 2024-09-21T13:49:26.946 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Sep 21 13:49:26 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-x[34691]: CherryPy Checker: 2024-09-21T13:49:26.946 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Sep 21 13:49:26 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-x[34691]: The Application mounted at '' has an empty config. 2024-09-21T13:49:26.946 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Sep 21 13:49:26 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-x[34691]: 2024-09-21T13:49:26.946 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Sep 21 13:49:26 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-x[34691]: [21/Sep/2024:13:49:26] ENGINE Serving on http://:::9283 2024-09-21T13:49:26.946 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Sep 21 13:49:26 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-x[34691]: [21/Sep/2024:13:49:26] ENGINE Bus STARTED 2024-09-21T13:49:27.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:26 smithi049 ceph-mon[40489]: Standby manager daemon x restarted 2024-09-21T13:49:27.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:26 smithi049 ceph-mon[40489]: Standby manager daemon x started 2024-09-21T13:49:27.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:26 smithi049 ceph-mon[40489]: from='mgr.? 172.21.15.187:0/3654822744' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-21T13:49:27.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:26 smithi049 ceph-mon[40489]: from='mgr.? 172.21.15.187:0/3654822744' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-21T13:49:27.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:26 smithi049 ceph-mon[40489]: from='mgr.? 172.21.15.187:0/3654822744' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-21T13:49:27.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:26 smithi049 ceph-mon[40489]: from='mgr.? 172.21.15.187:0/3654822744' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-21T13:49:27.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:26 smithi049 ceph-mon[30080]: Standby manager daemon x restarted 2024-09-21T13:49:27.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:26 smithi049 ceph-mon[30080]: Standby manager daemon x started 2024-09-21T13:49:27.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:26 smithi049 ceph-mon[30080]: from='mgr.? 172.21.15.187:0/3654822744' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-21T13:49:27.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:26 smithi049 ceph-mon[30080]: from='mgr.? 172.21.15.187:0/3654822744' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-21T13:49:27.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:26 smithi049 ceph-mon[30080]: from='mgr.? 172.21.15.187:0/3654822744' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-21T13:49:27.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:26 smithi049 ceph-mon[30080]: from='mgr.? 172.21.15.187:0/3654822744' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-21T13:49:27.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:27 smithi049 ceph-mon[30080]: mgrmap e19: y(active, since 5m), standbys: x 2024-09-21T13:49:27.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:27 smithi049 ceph-mon[30080]: Active manager daemon y restarted 2024-09-21T13:49:27.964 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:27 smithi049 ceph-mon[30080]: Activating manager daemon y 2024-09-21T13:49:27.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:27 smithi049 ceph-mon[30080]: osdmap e75: 8 total, 8 up, 8 in 2024-09-21T13:49:27.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:27 smithi049 ceph-mon[30080]: mgrmap e20: y(active, starting, since 0.00867815s), standbys: x 2024-09-21T13:49:27.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:27 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T13:49:27.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:27 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T13:49:27.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:27 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T13:49:27.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:27 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-21T13:49:27.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:27 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-21T13:49:27.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:27 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T13:49:27.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:27 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T13:49:27.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:27 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T13:49:27.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:27 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T13:49:27.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:27 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T13:49:27.965 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:27 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T13:49:27.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:27 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T13:49:27.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:27 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T13:49:27.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:27 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-21T13:49:27.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:27 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-21T13:49:27.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:27 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-21T13:49:27.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:27 smithi049 ceph-mon[30080]: Manager daemon y is now available 2024-09-21T13:49:27.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:27 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:49:27.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:27 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:27.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:27 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:49:27.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:27 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T13:49:27.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:27 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-21T13:49:27.966 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:27 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-21T13:49:27.967 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:27 smithi049 ceph-mon[40489]: mgrmap e19: y(active, since 5m), standbys: x 2024-09-21T13:49:27.967 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:27 smithi049 ceph-mon[40489]: Active manager daemon y restarted 2024-09-21T13:49:27.967 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:27 smithi049 ceph-mon[40489]: Activating manager daemon y 2024-09-21T13:49:27.967 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:27 smithi049 ceph-mon[40489]: osdmap e75: 8 total, 8 up, 8 in 2024-09-21T13:49:27.967 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:27 smithi049 ceph-mon[40489]: mgrmap e20: y(active, starting, since 0.00867815s), standbys: x 2024-09-21T13:49:27.967 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:27 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T13:49:27.967 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:27 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T13:49:27.967 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:27 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T13:49:27.967 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:27 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-21T13:49:27.967 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:27 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-21T13:49:27.967 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:27 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T13:49:27.967 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:27 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T13:49:27.968 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:27 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T13:49:27.968 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:27 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T13:49:27.968 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:27 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T13:49:27.968 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:27 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T13:49:27.968 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:27 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T13:49:27.968 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:27 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T13:49:27.968 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:27 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-21T13:49:27.968 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:27 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-21T13:49:27.968 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:27 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-21T13:49:27.968 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:27 smithi049 ceph-mon[40489]: Manager daemon y is now available 2024-09-21T13:49:27.968 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:27 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:49:27.968 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:27 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:27.968 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:27 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:49:27.968 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:27 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T13:49:27.968 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:27 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-21T13:49:27.969 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:27 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-21T13:49:28.070 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:27 smithi187 ceph-mon[33032]: mgrmap e19: y(active, since 5m), standbys: x 2024-09-21T13:49:28.071 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:27 smithi187 ceph-mon[33032]: Active manager daemon y restarted 2024-09-21T13:49:28.071 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:27 smithi187 ceph-mon[33032]: Activating manager daemon y 2024-09-21T13:49:28.071 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:27 smithi187 ceph-mon[33032]: osdmap e75: 8 total, 8 up, 8 in 2024-09-21T13:49:28.071 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:27 smithi187 ceph-mon[33032]: mgrmap e20: y(active, starting, since 0.00867815s), standbys: x 2024-09-21T13:49:28.071 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:27 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T13:49:28.071 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:27 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T13:49:28.071 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:27 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T13:49:28.071 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:27 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-21T13:49:28.071 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:27 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-21T13:49:28.071 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:27 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T13:49:28.071 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:27 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T13:49:28.071 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:27 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T13:49:28.071 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:27 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T13:49:28.071 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:27 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T13:49:28.072 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:27 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T13:49:28.072 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:27 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T13:49:28.072 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:27 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T13:49:28.072 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:27 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-21T13:49:28.072 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:27 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-21T13:49:28.072 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:27 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-21T13:49:28.072 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:27 smithi187 ceph-mon[33032]: Manager daemon y is now available 2024-09-21T13:49:28.072 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:27 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:49:28.072 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:27 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:28.072 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:27 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:49:28.072 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:27 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T13:49:28.072 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:27 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-21T13:49:28.072 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:27 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-21T13:49:28.481 INFO:teuthology.orchestra.run.smithi049.stdout:34359738420 2024-09-21T13:49:28.481 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.0 2024-09-21T13:49:28.755 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:49:28 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: [21/Sep/2024:13:49:28] ENGINE Bus STARTING 2024-09-21T13:49:28.755 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:49:28 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: CherryPy Checker: 2024-09-21T13:49:28.755 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:49:28 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: The Application mounted at '' has an empty config. 2024-09-21T13:49:28.755 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:49:28 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: 2024-09-21T13:49:28.755 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:49:28 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: [21/Sep/2024:13:49:28] ENGINE Serving on http://:::9283 2024-09-21T13:49:28.755 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:49:28 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: [21/Sep/2024:13:49:28] ENGINE Bus STARTED 2024-09-21T13:49:28.903 INFO:teuthology.orchestra.run.smithi049.stdout:167503724566 2024-09-21T13:49:28.903 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.5 2024-09-21T13:49:29.195 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:49:28 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:49:29.455 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:29 smithi049 ceph-mon[40489]: mgrmap e21: y(active, since 1.0137s), standbys: x 2024-09-21T13:49:29.456 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:29 smithi049 ceph-mon[40489]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T13:49:29.456 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:29 smithi049 ceph-mon[40489]: pgmap v3: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-09-21T13:49:29.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:29 smithi049 ceph-mon[30080]: mgrmap e21: y(active, since 1.0137s), standbys: x 2024-09-21T13:49:29.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:29 smithi049 ceph-mon[30080]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T13:49:29.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:29 smithi049 ceph-mon[30080]: pgmap v3: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-09-21T13:49:29.678 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:29 smithi187 ceph-mon[33032]: mgrmap e21: y(active, since 1.0137s), standbys: x 2024-09-21T13:49:29.678 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:29 smithi187 ceph-mon[33032]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T13:49:29.678 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:29 smithi187 ceph-mon[33032]: pgmap v3: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-09-21T13:49:29.755 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:49:29 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: 2024-09-21T13:49:29.531+0000 7f6a49203640 -1 log_channel(cephadm) log [ERR] : [21/Sep/2024:13:49:29] ENGINE Error in HTTPServer.serve 2024-09-21T13:49:29.755 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:49:29 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: Traceback (most recent call last): 2024-09-21T13:49:29.755 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:49:29 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T13:49:29.755 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:49:29 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: self._connections.run(self.expiration_interval) 2024-09-21T13:49:29.755 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:49:29 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T13:49:29.755 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:49:29 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: self._run(expiration_interval) 2024-09-21T13:49:29.755 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:49:29 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T13:49:29.755 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:49:29 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T13:49:29.755 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:49:29 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T13:49:29.756 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:49:29 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T13:49:29.756 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:49:29 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T13:49:29.756 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:49:29 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: s = self.context.wrap_socket( 2024-09-21T13:49:29.756 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:49:29 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T13:49:29.756 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:49:29 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: return self.sslsocket_class._create( 2024-09-21T13:49:29.756 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:49:29 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T13:49:29.756 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:49:29 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: self.do_handshake() 2024-09-21T13:49:29.756 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:49:29 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T13:49:29.756 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:49:29 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: self._sslobj.do_handshake() 2024-09-21T13:49:29.756 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:49:29 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T13:49:29.756 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:49:29 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: 2024-09-21T13:49:30.658 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:30 smithi187 ceph-mon[33032]: mgrmap e22: y(active, since 2s), standbys: x 2024-09-21T13:49:30.658 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:30 smithi187 ceph-mon[33032]: [21/Sep/2024:13:49:29] ENGINE Bus STARTING 2024-09-21T13:49:30.658 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:30 smithi187 ceph-mon[33032]: [21/Sep/2024:13:49:29] ENGINE Serving on https://172.21.15.49:7150 2024-09-21T13:49:30.658 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:30 smithi187 ceph-mon[33032]: [21/Sep/2024:13:49:29] ENGINE Error in HTTPServer.serve 2024-09-21T13:49:30.658 INFO:journalctl@ceph.mon.b.smithi187.stdout: Traceback (most recent call last): 2024-09-21T13:49:30.658 INFO:journalctl@ceph.mon.b.smithi187.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T13:49:30.658 INFO:journalctl@ceph.mon.b.smithi187.stdout: self._connections.run(self.expiration_interval) 2024-09-21T13:49:30.659 INFO:journalctl@ceph.mon.b.smithi187.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T13:49:30.659 INFO:journalctl@ceph.mon.b.smithi187.stdout: self._run(expiration_interval) 2024-09-21T13:49:30.659 INFO:journalctl@ceph.mon.b.smithi187.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T13:49:30.659 INFO:journalctl@ceph.mon.b.smithi187.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T13:49:30.659 INFO:journalctl@ceph.mon.b.smithi187.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T13:49:30.659 INFO:journalctl@ceph.mon.b.smithi187.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T13:49:30.659 INFO:journalctl@ceph.mon.b.smithi187.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T13:49:30.659 INFO:journalctl@ceph.mon.b.smithi187.stdout: s = self.context.wrap_socket( 2024-09-21T13:49:30.659 INFO:journalctl@ceph.mon.b.smithi187.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T13:49:30.659 INFO:journalctl@ceph.mon.b.smithi187.stdout: return self.sslsocket_class._create( 2024-09-21T13:49:30.659 INFO:journalctl@ceph.mon.b.smithi187.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T13:49:30.660 INFO:journalctl@ceph.mon.b.smithi187.stdout: self.do_handshake() 2024-09-21T13:49:30.660 INFO:journalctl@ceph.mon.b.smithi187.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T13:49:30.660 INFO:journalctl@ceph.mon.b.smithi187.stdout: self._sslobj.do_handshake() 2024-09-21T13:49:30.660 INFO:journalctl@ceph.mon.b.smithi187.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T13:49:30.660 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:30 smithi187 ceph-mon[33032]: [21/Sep/2024:13:49:29] ENGINE Serving on http://172.21.15.49:8765 2024-09-21T13:49:30.660 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:30 smithi187 ceph-mon[33032]: [21/Sep/2024:13:49:29] ENGINE Bus STARTED 2024-09-21T13:49:30.660 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:30 smithi187 ceph-mon[33032]: osdmap e76: 8 total, 8 up, 8 in 2024-09-21T13:49:30.755 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:49:30 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:49:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:49:30.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:30 smithi049 ceph-mon[40489]: mgrmap e22: y(active, since 2s), standbys: x 2024-09-21T13:49:30.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:30 smithi049 ceph-mon[40489]: [21/Sep/2024:13:49:29] ENGINE Bus STARTING 2024-09-21T13:49:30.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:30 smithi049 ceph-mon[40489]: [21/Sep/2024:13:49:29] ENGINE Serving on https://172.21.15.49:7150 2024-09-21T13:49:30.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:30 smithi049 ceph-mon[40489]: [21/Sep/2024:13:49:29] ENGINE Error in HTTPServer.serve 2024-09-21T13:49:30.755 INFO:journalctl@ceph.mon.c.smithi049.stdout: Traceback (most recent call last): 2024-09-21T13:49:30.755 INFO:journalctl@ceph.mon.c.smithi049.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T13:49:30.756 INFO:journalctl@ceph.mon.c.smithi049.stdout: self._connections.run(self.expiration_interval) 2024-09-21T13:49:30.756 INFO:journalctl@ceph.mon.c.smithi049.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T13:49:30.756 INFO:journalctl@ceph.mon.c.smithi049.stdout: self._run(expiration_interval) 2024-09-21T13:49:30.756 INFO:journalctl@ceph.mon.c.smithi049.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T13:49:30.756 INFO:journalctl@ceph.mon.c.smithi049.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T13:49:30.756 INFO:journalctl@ceph.mon.c.smithi049.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T13:49:30.756 INFO:journalctl@ceph.mon.c.smithi049.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T13:49:30.756 INFO:journalctl@ceph.mon.c.smithi049.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T13:49:30.756 INFO:journalctl@ceph.mon.c.smithi049.stdout: s = self.context.wrap_socket( 2024-09-21T13:49:30.756 INFO:journalctl@ceph.mon.c.smithi049.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T13:49:30.756 INFO:journalctl@ceph.mon.c.smithi049.stdout: return self.sslsocket_class._create( 2024-09-21T13:49:30.756 INFO:journalctl@ceph.mon.c.smithi049.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T13:49:30.756 INFO:journalctl@ceph.mon.c.smithi049.stdout: self.do_handshake() 2024-09-21T13:49:30.756 INFO:journalctl@ceph.mon.c.smithi049.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T13:49:30.757 INFO:journalctl@ceph.mon.c.smithi049.stdout: self._sslobj.do_handshake() 2024-09-21T13:49:30.757 INFO:journalctl@ceph.mon.c.smithi049.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T13:49:30.757 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:30 smithi049 ceph-mon[40489]: [21/Sep/2024:13:49:29] ENGINE Serving on http://172.21.15.49:8765 2024-09-21T13:49:30.757 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:30 smithi049 ceph-mon[40489]: [21/Sep/2024:13:49:29] ENGINE Bus STARTED 2024-09-21T13:49:30.757 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:30 smithi049 ceph-mon[40489]: osdmap e76: 8 total, 8 up, 8 in 2024-09-21T13:49:30.757 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:30 smithi049 ceph-mon[30080]: mgrmap e22: y(active, since 2s), standbys: x 2024-09-21T13:49:30.757 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:30 smithi049 ceph-mon[30080]: [21/Sep/2024:13:49:29] ENGINE Bus STARTING 2024-09-21T13:49:30.757 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:30 smithi049 ceph-mon[30080]: [21/Sep/2024:13:49:29] ENGINE Serving on https://172.21.15.49:7150 2024-09-21T13:49:30.757 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:30 smithi049 ceph-mon[30080]: [21/Sep/2024:13:49:29] ENGINE Error in HTTPServer.serve 2024-09-21T13:49:30.757 INFO:journalctl@ceph.mon.a.smithi049.stdout: Traceback (most recent call last): 2024-09-21T13:49:30.757 INFO:journalctl@ceph.mon.a.smithi049.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T13:49:30.757 INFO:journalctl@ceph.mon.a.smithi049.stdout: self._connections.run(self.expiration_interval) 2024-09-21T13:49:30.757 INFO:journalctl@ceph.mon.a.smithi049.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T13:49:30.758 INFO:journalctl@ceph.mon.a.smithi049.stdout: self._run(expiration_interval) 2024-09-21T13:49:30.758 INFO:journalctl@ceph.mon.a.smithi049.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T13:49:30.758 INFO:journalctl@ceph.mon.a.smithi049.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T13:49:30.758 INFO:journalctl@ceph.mon.a.smithi049.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T13:49:30.758 INFO:journalctl@ceph.mon.a.smithi049.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T13:49:30.758 INFO:journalctl@ceph.mon.a.smithi049.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T13:49:30.758 INFO:journalctl@ceph.mon.a.smithi049.stdout: s = self.context.wrap_socket( 2024-09-21T13:49:30.758 INFO:journalctl@ceph.mon.a.smithi049.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T13:49:30.758 INFO:journalctl@ceph.mon.a.smithi049.stdout: return self.sslsocket_class._create( 2024-09-21T13:49:30.758 INFO:journalctl@ceph.mon.a.smithi049.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T13:49:30.758 INFO:journalctl@ceph.mon.a.smithi049.stdout: self.do_handshake() 2024-09-21T13:49:30.758 INFO:journalctl@ceph.mon.a.smithi049.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T13:49:30.759 INFO:journalctl@ceph.mon.a.smithi049.stdout: self._sslobj.do_handshake() 2024-09-21T13:49:30.759 INFO:journalctl@ceph.mon.a.smithi049.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T13:49:30.759 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:30 smithi049 ceph-mon[30080]: [21/Sep/2024:13:49:29] ENGINE Serving on http://172.21.15.49:8765 2024-09-21T13:49:30.759 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:30 smithi049 ceph-mon[30080]: [21/Sep/2024:13:49:29] ENGINE Bus STARTED 2024-09-21T13:49:30.759 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:30 smithi049 ceph-mon[30080]: osdmap e76: 8 total, 8 up, 8 in 2024-09-21T13:49:31.078 INFO:teuthology.orchestra.run.smithi049.stdout:141733920796 2024-09-21T13:49:31.078 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.4 2024-09-21T13:49:32.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:31 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:32.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:31 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:32.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:31 smithi049 ceph-mon[40489]: pgmap v6: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-09-21T13:49:32.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:31 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:32.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:31 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:32.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:31 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T13:49:32.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:31 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T13:49:32.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:31 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T13:49:32.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:31 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T13:49:32.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:31 smithi049 ceph-mon[40489]: Adjusting osd_memory_target on smithi187 to 3760M 2024-09-21T13:49:32.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:31 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:32.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:31 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:32.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:31 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:32.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:31 smithi049 ceph-mon[30080]: pgmap v6: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-09-21T13:49:32.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:31 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:32.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:31 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:32.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:31 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T13:49:32.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:31 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T13:49:32.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:31 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T13:49:32.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:31 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T13:49:32.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:31 smithi049 ceph-mon[30080]: Adjusting osd_memory_target on smithi187 to 3760M 2024-09-21T13:49:32.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:31 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:32.039 INFO:teuthology.orchestra.run.smithi049.stdout:115964117026 2024-09-21T13:49:32.040 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.3 2024-09-21T13:49:32.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:31 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:32.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:31 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:32.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:31 smithi187 ceph-mon[33032]: pgmap v6: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-09-21T13:49:32.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:31 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:32.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:31 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:32.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:31 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T13:49:32.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:31 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T13:49:32.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:31 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T13:49:32.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:31 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T13:49:32.196 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:31 smithi187 ceph-mon[33032]: Adjusting osd_memory_target on smithi187 to 3760M 2024-09-21T13:49:32.196 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:31 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:32.467 INFO:teuthology.orchestra.run.smithi049.stdout:231928233994 2024-09-21T13:49:32.468 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.7 2024-09-21T13:49:32.788 INFO:teuthology.orchestra.run.smithi049.stdout:77309411366 2024-09-21T13:49:32.789 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.2 2024-09-21T13:49:33.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:32 smithi049 ceph-mon[40489]: mgrmap e23: y(active, since 4s), standbys: x 2024-09-21T13:49:33.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:32 smithi049 ceph-mon[30080]: mgrmap e23: y(active, since 4s), standbys: x 2024-09-21T13:49:33.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:32 smithi187 ceph-mon[33032]: mgrmap e23: y(active, since 4s), standbys: x 2024-09-21T13:49:33.250 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:49:33.252 INFO:teuthology.orchestra.run.smithi049.stdout:201863462928 2024-09-21T13:49:33.253 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.6 2024-09-21T13:49:33.658 INFO:teuthology.orchestra.run.smithi049.stdout:55834574895 2024-09-21T13:49:33.659 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.1 2024-09-21T13:49:33.774 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:49:33.775 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:49:33.776 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:49:33.776 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:49:33.783 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:49:33.845 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:49:34.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:33 smithi187 ceph-mon[33032]: pgmap v7: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-09-21T13:49:34.195 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:33 smithi049 ceph-mon[40489]: pgmap v7: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-09-21T13:49:34.195 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:33 smithi049 ceph-mon[30080]: pgmap v7: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-09-21T13:49:34.823 INFO:teuthology.orchestra.run.smithi049.stdout:115964117026 2024-09-21T13:49:35.143 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:34 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:49:35.143 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:34 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:49:35.143 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:34 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2517051164' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-21T13:49:35.144 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:34 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:49:35.144 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:34 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:49:35.144 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:34 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2517051164' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-21T13:49:35.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:34 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:49:35.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:34 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:49:35.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:34 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2517051164' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-21T13:49:35.427 INFO:teuthology.orchestra.run.smithi049.stdout:167503724567 2024-09-21T13:49:36.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:35 smithi187 ceph-mon[33032]: osdmap e77: 8 total, 8 up, 8 in 2024-09-21T13:49:36.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:35 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/670144478' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-21T13:49:36.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:35 smithi187 ceph-mon[33032]: pgmap v9: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-21T13:49:36.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:35 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/4065934770' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-21T13:49:36.232 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:35 smithi049 ceph-mon[40489]: osdmap e77: 8 total, 8 up, 8 in 2024-09-21T13:49:36.233 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:35 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/670144478' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-21T13:49:36.233 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:35 smithi049 ceph-mon[40489]: pgmap v9: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-21T13:49:36.233 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:35 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/4065934770' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-21T13:49:36.233 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:35 smithi049 ceph-mon[30080]: osdmap e77: 8 total, 8 up, 8 in 2024-09-21T13:49:36.233 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:35 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/670144478' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-21T13:49:36.233 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:35 smithi049 ceph-mon[30080]: pgmap v9: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-21T13:49:36.233 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:35 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/4065934770' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-21T13:49:36.261 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724566 got 167503724567 for osd.5 2024-09-21T13:49:36.261 DEBUG:teuthology.parallel:result is None 2024-09-21T13:49:36.350 INFO:teuthology.orchestra.run.smithi049.stdout:201863462928 2024-09-21T13:49:36.990 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462928 got 201863462928 for osd.6 2024-09-21T13:49:36.990 DEBUG:teuthology.parallel:result is None 2024-09-21T13:49:37.467 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117026 got 115964117026 for osd.3 2024-09-21T13:49:37.467 DEBUG:teuthology.parallel:result is None 2024-09-21T13:49:37.652 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:49:37.829 INFO:teuthology.orchestra.run.smithi049.stdout:34359738422 2024-09-21T13:49:38.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:38 smithi049 ceph-mon[40489]: pgmap v10: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T13:49:38.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:38 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3319358032' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-21T13:49:38.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:38 smithi049 ceph-mon[30080]: pgmap v10: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T13:49:38.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:38 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3319358032' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-21T13:49:38.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:38 smithi187 ceph-mon[33032]: pgmap v10: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T13:49:38.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:38 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3319358032' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-21T13:49:38.791 INFO:teuthology.orchestra.run.smithi049.stdout:55834574896 2024-09-21T13:49:39.194 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:49:38 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:49:39.496 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:39 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2483051809' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-21T13:49:39.496 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:39 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2483051809' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-21T13:49:39.527 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574895 got 55834574896 for osd.1 2024-09-21T13:49:39.530 DEBUG:teuthology.parallel:result is None 2024-09-21T13:49:39.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:39 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2483051809' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-21T13:49:40.056 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738420 got 34359738422 for osd.0 2024-09-21T13:49:40.056 DEBUG:teuthology.parallel:result is None 2024-09-21T13:49:40.567 INFO:teuthology.orchestra.run.smithi049.stdout:141733920799 2024-09-21T13:49:40.567 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:40 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:49:40.567 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:40 smithi049 ceph-mon[40489]: pgmap v11: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-21T13:49:40.567 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:40 smithi049 ceph-mon[40489]: osdmap e78: 8 total, 8 up, 8 in 2024-09-21T13:49:40.567 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:40 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:49:40.567 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:40 smithi049 ceph-mon[30080]: pgmap v11: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-21T13:49:40.567 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:40 smithi049 ceph-mon[30080]: osdmap e78: 8 total, 8 up, 8 in 2024-09-21T13:49:40.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:40 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:49:40.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:40 smithi187 ceph-mon[33032]: pgmap v11: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-21T13:49:40.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:40 smithi187 ceph-mon[33032]: osdmap e78: 8 total, 8 up, 8 in 2024-09-21T13:49:40.895 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:49:40 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:49:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:49:41.228 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920796 got 141733920799 for osd.4 2024-09-21T13:49:41.229 DEBUG:teuthology.parallel:result is None 2024-09-21T13:49:41.332 INFO:teuthology.orchestra.run.smithi049.stdout:231928233997 2024-09-21T13:49:41.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:41 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/4265755713' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-21T13:49:41.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:41 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1666729452' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-21T13:49:41.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:41 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3368375467' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-21T13:49:41.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:41 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/4265755713' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-21T13:49:41.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:41 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1666729452' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-21T13:49:41.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:41 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3368375467' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-21T13:49:41.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:41 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/4265755713' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-21T13:49:41.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:41 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1666729452' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-21T13:49:41.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:41 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3368375467' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-21T13:49:41.906 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928233994 got 231928233997 for osd.7 2024-09-21T13:49:41.906 DEBUG:teuthology.parallel:result is None 2024-09-21T13:49:42.145 INFO:teuthology.orchestra.run.smithi049.stdout:77309411369 2024-09-21T13:49:42.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:42 smithi187 ceph-mon[33032]: pgmap v13: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-21T13:49:42.707 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:42 smithi049 ceph-mon[40489]: pgmap v13: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-21T13:49:42.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:42 smithi049 ceph-mon[30080]: pgmap v13: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-21T13:49:42.743 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411366 got 77309411369 for osd.2 2024-09-21T13:49:42.743 DEBUG:teuthology.parallel:result is None 2024-09-21T13:49:42.743 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-21T13:49:42.743 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T13:49:43.193 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:49:43.596 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:43 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:49:43.596 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:43 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:49:43.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:43 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:49:43.897 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:49:43.899 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-09-21T13:49:44.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:44 smithi049 ceph-mon[40489]: pgmap v14: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 607 B/s rd, 0 op/s 2024-09-21T13:49:44.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:44 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:44.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:44 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:44.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:44 smithi049 ceph-mon[30080]: pgmap v14: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 607 B/s rd, 0 op/s 2024-09-21T13:49:44.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:44 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:44.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:44 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:44.663 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-09-21T13:49:43.332507+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":883,"num_read_kb":740,"num_write":549,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":552,"ondisk_log_size":552,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":237280,"kb_used_data":8576,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":749723936,"statfs":{"total":767960285184,"available":767717310464,"internally_reserved":0,"allocated":8781824,"data_stored":4771504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.421721"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":44,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:35.374707+0000","last_change":"2024-09-21T13:48:58.061797+0000","last_active":"2024-09-21T13:49:35.374707+0000","last_peered":"2024-09-21T13:49:35.374707+0000","last_clean":"2024-09-21T13:49:35.374707+0000","last_became_active":"2024-09-21T13:48:58.061415+0000","last_became_peered":"2024-09-21T13:48:58.061415+0000","last_unstale":"2024-09-21T13:49:35.374707+0000","last_undegraded":"2024-09-21T13:49:35.374707+0000","last_fullsized":"2024-09-21T13:49:35.374707+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:07.025619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"68'9","reported_seq":56,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:34.912149+0000","last_change":"2024-09-21T13:49:00.055717+0000","last_active":"2024-09-21T13:49:34.912149+0000","last_peered":"2024-09-21T13:49:34.912149+0000","last_clean":"2024-09-21T13:49:34.912149+0000","last_became_active":"2024-09-21T13:49:00.055210+0000","last_became_peered":"2024-09-21T13:49:00.055210+0000","last_unstale":"2024-09-21T13:49:34.912149+0000","last_undegraded":"2024-09-21T13:49:34.912149+0000","last_fullsized":"2024-09-21T13:49:34.912149+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:13.541017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.806595+0000","last_change":"2024-09-21T13:49:02.061394+0000","last_active":"2024-09-21T13:49:39.806595+0000","last_peered":"2024-09-21T13:49:39.806595+0000","last_clean":"2024-09-21T13:49:39.806595+0000","last_became_active":"2024-09-21T13:49:02.060984+0000","last_became_peered":"2024-09-21T13:49:02.060984+0000","last_unstale":"2024-09-21T13:49:39.806595+0000","last_undegraded":"2024-09-21T13:49:39.806595+0000","last_fullsized":"2024-09-21T13:49:39.806595+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:18.936236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":32,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:34.912312+0000","last_change":"2024-09-21T13:49:04.072734+0000","last_active":"2024-09-21T13:49:34.912312+0000","last_peered":"2024-09-21T13:49:34.912312+0000","last_clean":"2024-09-21T13:49:34.912312+0000","last_became_active":"2024-09-21T13:49:04.072460+0000","last_became_peered":"2024-09-21T13:49:04.072460+0000","last_unstale":"2024-09-21T13:49:34.912312+0000","last_undegraded":"2024-09-21T13:49:34.912312+0000","last_fullsized":"2024-09-21T13:49:34.912312+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:30.526897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":32,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:34.912526+0000","last_change":"2024-09-21T13:49:04.434312+0000","last_active":"2024-09-21T13:49:34.912526+0000","last_peered":"2024-09-21T13:49:34.912526+0000","last_clean":"2024-09-21T13:49:34.912526+0000","last_became_active":"2024-09-21T13:49:04.433965+0000","last_became_peered":"2024-09-21T13:49:04.433965+0000","last_unstale":"2024-09-21T13:49:34.912526+0000","last_undegraded":"2024-09-21T13:49:34.912526+0000","last_fullsized":"2024-09-21T13:49:34.912526+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:06.082699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":44,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:34.914018+0000","last_change":"2024-09-21T13:48:58.051225+0000","last_active":"2024-09-21T13:49:34.914018+0000","last_peered":"2024-09-21T13:49:34.914018+0000","last_clean":"2024-09-21T13:49:34.914018+0000","last_became_active":"2024-09-21T13:48:58.050479+0000","last_became_peered":"2024-09-21T13:48:58.050479+0000","last_unstale":"2024-09-21T13:49:34.914018+0000","last_undegraded":"2024-09-21T13:49:34.914018+0000","last_fullsized":"2024-09-21T13:49:34.914018+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:19.515469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"68'15","reported_seq":65,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:34.913375+0000","last_change":"2024-09-21T13:49:00.063787+0000","last_active":"2024-09-21T13:49:34.913375+0000","last_peered":"2024-09-21T13:49:34.913375+0000","last_clean":"2024-09-21T13:49:34.913375+0000","last_became_active":"2024-09-21T13:49:00.063460+0000","last_became_peered":"2024-09-21T13:49:00.063460+0000","last_unstale":"2024-09-21T13:49:34.913375+0000","last_undegraded":"2024-09-21T13:49:34.913375+0000","last_fullsized":"2024-09-21T13:49:34.913375+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:14.007409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":36,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:34.912142+0000","last_change":"2024-09-21T13:49:02.064007+0000","last_active":"2024-09-21T13:49:34.912142+0000","last_peered":"2024-09-21T13:49:34.912142+0000","last_clean":"2024-09-21T13:49:34.912142+0000","last_became_active":"2024-09-21T13:49:02.063687+0000","last_became_peered":"2024-09-21T13:49:02.063687+0000","last_unstale":"2024-09-21T13:49:34.912142+0000","last_undegraded":"2024-09-21T13:49:34.912142+0000","last_fullsized":"2024-09-21T13:49:34.912142+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:05.421236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":44,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:35.234930+0000","last_change":"2024-09-21T13:48:58.051001+0000","last_active":"2024-09-21T13:49:35.234930+0000","last_peered":"2024-09-21T13:49:35.234930+0000","last_clean":"2024-09-21T13:49:35.234930+0000","last_became_active":"2024-09-21T13:48:58.050695+0000","last_became_peered":"2024-09-21T13:48:58.050695+0000","last_unstale":"2024-09-21T13:49:35.234930+0000","last_undegraded":"2024-09-21T13:49:35.234930+0000","last_fullsized":"2024-09-21T13:49:35.234930+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:09.186887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"68'9","reported_seq":56,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:34.912520+0000","last_change":"2024-09-21T13:49:00.059748+0000","last_active":"2024-09-21T13:49:34.912520+0000","last_peered":"2024-09-21T13:49:34.912520+0000","last_clean":"2024-09-21T13:49:34.912520+0000","last_became_active":"2024-09-21T13:49:00.059382+0000","last_became_peered":"2024-09-21T13:49:00.059382+0000","last_unstale":"2024-09-21T13:49:34.912520+0000","last_undegraded":"2024-09-21T13:49:34.912520+0000","last_fullsized":"2024-09-21T13:49:34.912520+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:05.303397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":36,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:35.234486+0000","last_change":"2024-09-21T13:49:02.066978+0000","last_active":"2024-09-21T13:49:35.234486+0000","last_peered":"2024-09-21T13:49:35.234486+0000","last_clean":"2024-09-21T13:49:35.234486+0000","last_became_active":"2024-09-21T13:49:02.066693+0000","last_became_peered":"2024-09-21T13:49:02.066693+0000","last_unstale":"2024-09-21T13:49:35.234486+0000","last_undegraded":"2024-09-21T13:49:35.234486+0000","last_fullsized":"2024-09-21T13:49:35.234486+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:28.710214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":32,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:35.373982+0000","last_change":"2024-09-21T13:49:04.080709+0000","last_active":"2024-09-21T13:49:35.373982+0000","last_peered":"2024-09-21T13:49:35.373982+0000","last_clean":"2024-09-21T13:49:35.373982+0000","last_became_active":"2024-09-21T13:49:04.080359+0000","last_became_peered":"2024-09-21T13:49:04.080359+0000","last_unstale":"2024-09-21T13:49:35.373982+0000","last_undegraded":"2024-09-21T13:49:35.373982+0000","last_fullsized":"2024-09-21T13:49:35.373982+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:09.590774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":44,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:35.234635+0000","last_change":"2024-09-21T13:48:58.052581+0000","last_active":"2024-09-21T13:49:35.234635+0000","last_peered":"2024-09-21T13:49:35.234635+0000","last_clean":"2024-09-21T13:49:35.234635+0000","last_became_active":"2024-09-21T13:48:58.050906+0000","last_became_peered":"2024-09-21T13:48:58.050906+0000","last_unstale":"2024-09-21T13:49:35.234635+0000","last_undegraded":"2024-09-21T13:49:35.234635+0000","last_fullsized":"2024-09-21T13:49:35.234635+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:19.977336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"68'5","reported_seq":50,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:34.912667+0000","last_change":"2024-09-21T13:49:00.060913+0000","last_active":"2024-09-21T13:49:34.912667+0000","last_peered":"2024-09-21T13:49:34.912667+0000","last_clean":"2024-09-21T13:49:34.912667+0000","last_became_active":"2024-09-21T13:49:00.060497+0000","last_became_peered":"2024-09-21T13:49:00.060497+0000","last_unstale":"2024-09-21T13:49:34.912667+0000","last_undegraded":"2024-09-21T13:49:34.912667+0000","last_fullsized":"2024-09-21T13:49:34.912667+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:10.507839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":36,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:35.037719+0000","last_change":"2024-09-21T13:49:02.060484+0000","last_active":"2024-09-21T13:49:35.037719+0000","last_peered":"2024-09-21T13:49:35.037719+0000","last_clean":"2024-09-21T13:49:35.037719+0000","last_became_active":"2024-09-21T13:49:02.060212+0000","last_became_peered":"2024-09-21T13:49:02.060212+0000","last_unstale":"2024-09-21T13:49:35.037719+0000","last_undegraded":"2024-09-21T13:49:35.037719+0000","last_fullsized":"2024-09-21T13:49:35.037719+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:17.976156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":32,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:35.234276+0000","last_change":"2024-09-21T13:49:04.078296+0000","last_active":"2024-09-21T13:49:35.234276+0000","last_peered":"2024-09-21T13:49:35.234276+0000","last_clean":"2024-09-21T13:49:35.234276+0000","last_became_active":"2024-09-21T13:49:04.078026+0000","last_became_peered":"2024-09-21T13:49:04.078026+0000","last_unstale":"2024-09-21T13:49:35.234276+0000","last_undegraded":"2024-09-21T13:49:35.234276+0000","last_fullsized":"2024-09-21T13:49:35.234276+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:05.284846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":32,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:34.913027+0000","last_change":"2024-09-21T13:49:04.432458+0000","last_active":"2024-09-21T13:49:34.913027+0000","last_peered":"2024-09-21T13:49:34.913027+0000","last_clean":"2024-09-21T13:49:34.913027+0000","last_became_active":"2024-09-21T13:49:04.431422+0000","last_became_peered":"2024-09-21T13:49:04.431422+0000","last_unstale":"2024-09-21T13:49:34.913027+0000","last_undegraded":"2024-09-21T13:49:34.913027+0000","last_fullsized":"2024-09-21T13:49:34.913027+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:08.067421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":44,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:35.374580+0000","last_change":"2024-09-21T13:48:58.057667+0000","last_active":"2024-09-21T13:49:35.374580+0000","last_peered":"2024-09-21T13:49:35.374580+0000","last_clean":"2024-09-21T13:49:35.374580+0000","last_became_active":"2024-09-21T13:48:58.057336+0000","last_became_peered":"2024-09-21T13:48:58.057336+0000","last_unstale":"2024-09-21T13:49:35.374580+0000","last_undegraded":"2024-09-21T13:49:35.374580+0000","last_fullsized":"2024-09-21T13:49:35.374580+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:18.555629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"68'15","reported_seq":67,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.807764+0000","last_change":"2024-09-21T13:49:00.059068+0000","last_active":"2024-09-21T13:49:39.807764+0000","last_peered":"2024-09-21T13:49:39.807764+0000","last_clean":"2024-09-21T13:49:39.807764+0000","last_became_active":"2024-09-21T13:49:00.058730+0000","last_became_peered":"2024-09-21T13:49:00.058730+0000","last_unstale":"2024-09-21T13:49:39.807764+0000","last_undegraded":"2024-09-21T13:49:39.807764+0000","last_fullsized":"2024-09-21T13:49:39.807764+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:18.796269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.808006+0000","last_change":"2024-09-21T13:49:02.067253+0000","last_active":"2024-09-21T13:49:39.808006+0000","last_peered":"2024-09-21T13:49:39.808006+0000","last_clean":"2024-09-21T13:49:39.808006+0000","last_became_active":"2024-09-21T13:49:02.066796+0000","last_became_peered":"2024-09-21T13:49:02.066796+0000","last_unstale":"2024-09-21T13:49:39.808006+0000","last_undegraded":"2024-09-21T13:49:39.808006+0000","last_fullsized":"2024-09-21T13:49:39.808006+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:24.625163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":32,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:34.914838+0000","last_change":"2024-09-21T13:49:04.434429+0000","last_active":"2024-09-21T13:49:34.914838+0000","last_peered":"2024-09-21T13:49:34.914838+0000","last_clean":"2024-09-21T13:49:34.914838+0000","last_became_active":"2024-09-21T13:49:04.434049+0000","last_became_peered":"2024-09-21T13:49:04.434049+0000","last_unstale":"2024-09-21T13:49:34.914838+0000","last_undegraded":"2024-09-21T13:49:34.914838+0000","last_fullsized":"2024-09-21T13:49:34.914838+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:25.759474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":44,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:34.912634+0000","last_change":"2024-09-21T13:48:58.043457+0000","last_active":"2024-09-21T13:49:34.912634+0000","last_peered":"2024-09-21T13:49:34.912634+0000","last_clean":"2024-09-21T13:49:34.912634+0000","last_became_active":"2024-09-21T13:48:58.043088+0000","last_became_peered":"2024-09-21T13:48:58.043088+0000","last_unstale":"2024-09-21T13:49:34.912634+0000","last_undegraded":"2024-09-21T13:49:34.912634+0000","last_fullsized":"2024-09-21T13:49:34.912634+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:21.622429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"68'12","reported_seq":63,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:34.914502+0000","last_change":"2024-09-21T13:49:00.062162+0000","last_active":"2024-09-21T13:49:34.914502+0000","last_peered":"2024-09-21T13:49:34.914502+0000","last_clean":"2024-09-21T13:49:34.914502+0000","last_became_active":"2024-09-21T13:49:00.061867+0000","last_became_peered":"2024-09-21T13:49:00.061867+0000","last_unstale":"2024-09-21T13:49:34.914502+0000","last_undegraded":"2024-09-21T13:49:34.914502+0000","last_fullsized":"2024-09-21T13:49:34.914502+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:27.206067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":36,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:34.912744+0000","last_change":"2024-09-21T13:49:02.063090+0000","last_active":"2024-09-21T13:49:34.912744+0000","last_peered":"2024-09-21T13:49:34.912744+0000","last_clean":"2024-09-21T13:49:34.912744+0000","last_became_active":"2024-09-21T13:49:02.062802+0000","last_became_peered":"2024-09-21T13:49:02.062802+0000","last_unstale":"2024-09-21T13:49:34.912744+0000","last_undegraded":"2024-09-21T13:49:34.912744+0000","last_fullsized":"2024-09-21T13:49:34.912744+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:20.369055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:35.037980+0000","last_change":"2024-09-21T13:49:04.076090+0000","last_active":"2024-09-21T13:49:35.037980+0000","last_peered":"2024-09-21T13:49:35.037980+0000","last_clean":"2024-09-21T13:49:35.037980+0000","last_became_active":"2024-09-21T13:49:04.075792+0000","last_became_peered":"2024-09-21T13:49:04.075792+0000","last_unstale":"2024-09-21T13:49:35.037980+0000","last_undegraded":"2024-09-21T13:49:35.037980+0000","last_fullsized":"2024-09-21T13:49:35.037980+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:07.846061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":47,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.806690+0000","last_change":"2024-09-21T13:48:58.049463+0000","last_active":"2024-09-21T13:49:39.806690+0000","last_peered":"2024-09-21T13:49:39.806690+0000","last_clean":"2024-09-21T13:49:39.806690+0000","last_became_active":"2024-09-21T13:48:58.048972+0000","last_became_peered":"2024-09-21T13:48:58.048972+0000","last_unstale":"2024-09-21T13:49:39.806690+0000","last_undegraded":"2024-09-21T13:49:39.806690+0000","last_fullsized":"2024-09-21T13:49:39.806690+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:03.427338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"68'10","reported_seq":55,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:35.374594+0000","last_change":"2024-09-21T13:49:00.426837+0000","last_active":"2024-09-21T13:49:35.374594+0000","last_peered":"2024-09-21T13:49:35.374594+0000","last_clean":"2024-09-21T13:49:35.374594+0000","last_became_active":"2024-09-21T13:49:00.426178+0000","last_became_peered":"2024-09-21T13:49:00.426178+0000","last_unstale":"2024-09-21T13:49:35.374594+0000","last_undegraded":"2024-09-21T13:49:35.374594+0000","last_fullsized":"2024-09-21T13:49:35.374594+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:22.029884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"68'8","reported_seq":59,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836690+0000","last_change":"2024-09-21T13:49:02.059979+0000","last_active":"2024-09-21T13:49:39.836690+0000","last_peered":"2024-09-21T13:49:39.836690+0000","last_clean":"2024-09-21T13:49:39.836690+0000","last_became_active":"2024-09-21T13:49:02.059638+0000","last_became_peered":"2024-09-21T13:49:02.059638+0000","last_unstale":"2024-09-21T13:49:39.836690+0000","last_undegraded":"2024-09-21T13:49:39.836690+0000","last_fullsized":"2024-09-21T13:49:39.836690+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:26.974288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"68'15","reported_seq":67,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.808011+0000","last_change":"2024-09-21T13:49:00.060620+0000","last_active":"2024-09-21T13:49:39.808011+0000","last_peered":"2024-09-21T13:49:39.808011+0000","last_clean":"2024-09-21T13:49:39.808011+0000","last_became_active":"2024-09-21T13:49:00.060324+0000","last_became_peered":"2024-09-21T13:49:00.060324+0000","last_unstale":"2024-09-21T13:49:39.808011+0000","last_undegraded":"2024-09-21T13:49:39.808011+0000","last_fullsized":"2024-09-21T13:49:39.808011+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:20.149960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":44,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:34.914870+0000","last_change":"2024-09-21T13:48:58.052171+0000","last_active":"2024-09-21T13:49:34.914870+0000","last_peered":"2024-09-21T13:49:34.914870+0000","last_clean":"2024-09-21T13:49:34.914870+0000","last_became_active":"2024-09-21T13:48:58.051911+0000","last_became_peered":"2024-09-21T13:48:58.051911+0000","last_unstale":"2024-09-21T13:49:34.914870+0000","last_undegraded":"2024-09-21T13:49:34.914870+0000","last_fullsized":"2024-09-21T13:49:34.914870+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:49:58.457410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"68'8","reported_seq":56,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:34.913230+0000","last_change":"2024-09-21T13:49:02.061966+0000","last_active":"2024-09-21T13:49:34.913230+0000","last_peered":"2024-09-21T13:49:34.913230+0000","last_clean":"2024-09-21T13:49:34.913230+0000","last_became_active":"2024-09-21T13:49:02.061679+0000","last_became_peered":"2024-09-21T13:49:02.061679+0000","last_unstale":"2024-09-21T13:49:34.913230+0000","last_undegraded":"2024-09-21T13:49:34.913230+0000","last_fullsized":"2024-09-21T13:49:34.913230+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:28.854727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":32,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:35.234611+0000","last_change":"2024-09-21T13:49:04.082213+0000","last_active":"2024-09-21T13:49:35.234611+0000","last_peered":"2024-09-21T13:49:35.234611+0000","last_clean":"2024-09-21T13:49:35.234611+0000","last_became_active":"2024-09-21T13:49:04.081905+0000","last_became_peered":"2024-09-21T13:49:04.081905+0000","last_unstale":"2024-09-21T13:49:35.234611+0000","last_undegraded":"2024-09-21T13:49:35.234611+0000","last_fullsized":"2024-09-21T13:49:35.234611+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:17.499009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"68'18","reported_seq":72,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:34.913812+0000","last_change":"2024-09-21T13:49:00.429686+0000","last_active":"2024-09-21T13:49:34.913812+0000","last_peered":"2024-09-21T13:49:34.913812+0000","last_clean":"2024-09-21T13:49:34.913812+0000","last_became_active":"2024-09-21T13:49:00.429314+0000","last_became_peered":"2024-09-21T13:49:00.429314+0000","last_unstale":"2024-09-21T13:49:34.913812+0000","last_undegraded":"2024-09-21T13:49:34.913812+0000","last_fullsized":"2024-09-21T13:49:34.913812+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:09.824746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":44,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:34.915036+0000","last_change":"2024-09-21T13:48:58.056669+0000","last_active":"2024-09-21T13:49:34.915036+0000","last_peered":"2024-09-21T13:49:34.915036+0000","last_clean":"2024-09-21T13:49:34.915036+0000","last_became_active":"2024-09-21T13:48:58.056375+0000","last_became_peered":"2024-09-21T13:48:58.056375+0000","last_unstale":"2024-09-21T13:49:34.915036+0000","last_undegraded":"2024-09-21T13:49:34.915036+0000","last_fullsized":"2024-09-21T13:49:34.915036+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:21.686532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":36,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:34.912202+0000","last_change":"2024-09-21T13:49:02.063237+0000","last_active":"2024-09-21T13:49:34.912202+0000","last_peered":"2024-09-21T13:49:34.912202+0000","last_clean":"2024-09-21T13:49:34.912202+0000","last_became_active":"2024-09-21T13:49:02.062955+0000","last_became_peered":"2024-09-21T13:49:02.062955+0000","last_unstale":"2024-09-21T13:49:34.912202+0000","last_undegraded":"2024-09-21T13:49:34.912202+0000","last_fullsized":"2024-09-21T13:49:34.912202+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:05.393016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":32,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:34.912495+0000","last_change":"2024-09-21T13:49:04.076445+0000","last_active":"2024-09-21T13:49:34.912495+0000","last_peered":"2024-09-21T13:49:34.912495+0000","last_clean":"2024-09-21T13:49:34.912495+0000","last_became_active":"2024-09-21T13:49:04.075818+0000","last_became_peered":"2024-09-21T13:49:04.075818+0000","last_unstale":"2024-09-21T13:49:34.912495+0000","last_undegraded":"2024-09-21T13:49:34.912495+0000","last_fullsized":"2024-09-21T13:49:34.912495+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:31.482751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"68'14","reported_seq":61,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:34.912236+0000","last_change":"2024-09-21T13:49:00.055423+0000","last_active":"2024-09-21T13:49:34.912236+0000","last_peered":"2024-09-21T13:49:34.912236+0000","last_clean":"2024-09-21T13:49:34.912236+0000","last_became_active":"2024-09-21T13:49:00.055031+0000","last_became_peered":"2024-09-21T13:49:00.055031+0000","last_unstale":"2024-09-21T13:49:34.912236+0000","last_undegraded":"2024-09-21T13:49:34.912236+0000","last_fullsized":"2024-09-21T13:49:34.912236+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:06.138147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":45,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:35.374766+0000","last_change":"2024-09-21T13:48:58.053829+0000","last_active":"2024-09-21T13:49:35.374766+0000","last_peered":"2024-09-21T13:49:35.374766+0000","last_clean":"2024-09-21T13:49:35.374766+0000","last_became_active":"2024-09-21T13:48:58.053533+0000","last_became_peered":"2024-09-21T13:48:58.053533+0000","last_unstale":"2024-09-21T13:49:35.374766+0000","last_undegraded":"2024-09-21T13:49:35.374766+0000","last_fullsized":"2024-09-21T13:49:35.374766+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:49:58.639672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"68'8","reported_seq":53,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:34.913666+0000","last_change":"2024-09-21T13:49:02.070405+0000","last_active":"2024-09-21T13:49:34.913666+0000","last_peered":"2024-09-21T13:49:34.913666+0000","last_clean":"2024-09-21T13:49:34.913666+0000","last_became_active":"2024-09-21T13:49:02.070064+0000","last_became_peered":"2024-09-21T13:49:02.070064+0000","last_unstale":"2024-09-21T13:49:34.913666+0000","last_undegraded":"2024-09-21T13:49:34.913666+0000","last_fullsized":"2024-09-21T13:49:34.913666+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:13.746404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":32,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:35.037905+0000","last_change":"2024-09-21T13:49:04.434498+0000","last_active":"2024-09-21T13:49:35.037905+0000","last_peered":"2024-09-21T13:49:35.037905+0000","last_clean":"2024-09-21T13:49:35.037905+0000","last_became_active":"2024-09-21T13:49:04.433920+0000","last_became_peered":"2024-09-21T13:49:04.433920+0000","last_unstale":"2024-09-21T13:49:35.037905+0000","last_undegraded":"2024-09-21T13:49:35.037905+0000","last_fullsized":"2024-09-21T13:49:35.037905+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:22.847243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"68'10","reported_seq":57,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.807473+0000","last_change":"2024-09-21T13:49:00.054972+0000","last_active":"2024-09-21T13:49:39.807473+0000","last_peered":"2024-09-21T13:49:39.807473+0000","last_clean":"2024-09-21T13:49:39.807473+0000","last_became_active":"2024-09-21T13:49:00.054564+0000","last_became_peered":"2024-09-21T13:49:00.054564+0000","last_unstale":"2024-09-21T13:49:39.807473+0000","last_undegraded":"2024-09-21T13:49:39.807473+0000","last_fullsized":"2024-09-21T13:49:39.807473+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:18.647498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"61'1","reported_seq":50,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:35.235020+0000","last_change":"2024-09-21T13:48:58.050844+0000","last_active":"2024-09-21T13:49:35.235020+0000","last_peered":"2024-09-21T13:49:35.235020+0000","last_clean":"2024-09-21T13:49:35.235020+0000","last_became_active":"2024-09-21T13:48:58.050548+0000","last_became_peered":"2024-09-21T13:48:58.050548+0000","last_unstale":"2024-09-21T13:49:35.235020+0000","last_undegraded":"2024-09-21T13:49:35.235020+0000","last_fullsized":"2024-09-21T13:49:35.235020+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:17.587266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"68'8","reported_seq":53,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:35.374355+0000","last_change":"2024-09-21T13:49:02.068926+0000","last_active":"2024-09-21T13:49:35.374355+0000","last_peered":"2024-09-21T13:49:35.374355+0000","last_clean":"2024-09-21T13:49:35.374355+0000","last_became_active":"2024-09-21T13:49:02.067664+0000","last_became_peered":"2024-09-21T13:49:02.067664+0000","last_unstale":"2024-09-21T13:49:35.374355+0000","last_undegraded":"2024-09-21T13:49:35.374355+0000","last_fullsized":"2024-09-21T13:49:35.374355+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:09.630253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":32,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:35.374033+0000","last_change":"2024-09-21T13:49:04.081771+0000","last_active":"2024-09-21T13:49:35.374033+0000","last_peered":"2024-09-21T13:49:35.374033+0000","last_clean":"2024-09-21T13:49:35.374033+0000","last_became_active":"2024-09-21T13:49:04.081462+0000","last_became_peered":"2024-09-21T13:49:04.081462+0000","last_unstale":"2024-09-21T13:49:35.374033+0000","last_undegraded":"2024-09-21T13:49:35.374033+0000","last_fullsized":"2024-09-21T13:49:35.374033+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:25.632205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"68'19","reported_seq":76,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:35.374393+0000","last_change":"2024-09-21T13:49:00.426917+0000","last_active":"2024-09-21T13:49:35.374393+0000","last_peered":"2024-09-21T13:49:35.374393+0000","last_clean":"2024-09-21T13:49:35.374393+0000","last_became_active":"2024-09-21T13:49:00.426579+0000","last_became_peered":"2024-09-21T13:49:00.426579+0000","last_unstale":"2024-09-21T13:49:35.374393+0000","last_undegraded":"2024-09-21T13:49:35.374393+0000","last_fullsized":"2024-09-21T13:49:35.374393+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:15.913331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.808545+0000","last_change":"2024-09-21T13:48:58.049961+0000","last_active":"2024-09-21T13:49:39.808545+0000","last_peered":"2024-09-21T13:49:39.808545+0000","last_clean":"2024-09-21T13:49:39.808545+0000","last_became_active":"2024-09-21T13:48:58.049682+0000","last_became_peered":"2024-09-21T13:48:58.049682+0000","last_unstale":"2024-09-21T13:49:39.808545+0000","last_undegraded":"2024-09-21T13:49:39.808545+0000","last_fullsized":"2024-09-21T13:49:39.808545+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:02.280644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835812+0000","last_change":"2024-09-21T13:49:02.061604+0000","last_active":"2024-09-21T13:49:39.835812+0000","last_peered":"2024-09-21T13:49:39.835812+0000","last_clean":"2024-09-21T13:49:39.835812+0000","last_became_active":"2024-09-21T13:49:02.061187+0000","last_became_peered":"2024-09-21T13:49:02.061187+0000","last_unstale":"2024-09-21T13:49:39.835812+0000","last_undegraded":"2024-09-21T13:49:39.835812+0000","last_fullsized":"2024-09-21T13:49:39.835812+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:08.846428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.806520+0000","last_change":"2024-09-21T13:49:04.431651+0000","last_active":"2024-09-21T13:49:39.806520+0000","last_peered":"2024-09-21T13:49:39.806520+0000","last_clean":"2024-09-21T13:49:39.806520+0000","last_became_active":"2024-09-21T13:49:04.431079+0000","last_became_peered":"2024-09-21T13:49:04.431079+0000","last_unstale":"2024-09-21T13:49:39.806520+0000","last_undegraded":"2024-09-21T13:49:39.806520+0000","last_fullsized":"2024-09-21T13:49:39.806520+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:05.155255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"68'28","reported_seq":100,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.808393+0000","last_change":"2024-09-21T13:49:00.060616+0000","last_active":"2024-09-21T13:49:39.808393+0000","last_peered":"2024-09-21T13:49:39.808393+0000","last_clean":"2024-09-21T13:49:39.808393+0000","last_became_active":"2024-09-21T13:49:00.060355+0000","last_became_peered":"2024-09-21T13:49:00.060355+0000","last_unstale":"2024-09-21T13:49:39.808393+0000","last_undegraded":"2024-09-21T13:49:39.808393+0000","last_fullsized":"2024-09-21T13:49:39.808393+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:26.214000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":44,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:34.912867+0000","last_change":"2024-09-21T13:48:58.055640+0000","last_active":"2024-09-21T13:49:34.912867+0000","last_peered":"2024-09-21T13:49:34.912867+0000","last_clean":"2024-09-21T13:49:34.912867+0000","last_became_active":"2024-09-21T13:48:58.055366+0000","last_became_peered":"2024-09-21T13:48:58.055366+0000","last_unstale":"2024-09-21T13:49:34.912867+0000","last_undegraded":"2024-09-21T13:49:34.912867+0000","last_fullsized":"2024-09-21T13:49:34.912867+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:17.442979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":54,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:35.234574+0000","last_change":"2024-09-21T13:49:00.053256+0000","last_active":"2024-09-21T13:49:35.234574+0000","last_peered":"2024-09-21T13:49:35.234574+0000","last_clean":"2024-09-21T13:49:35.234574+0000","last_became_active":"2024-09-21T13:48:57.046246+0000","last_became_peered":"2024-09-21T13:48:57.046246+0000","last_unstale":"2024-09-21T13:49:35.234574+0000","last_undegraded":"2024-09-21T13:49:35.234574+0000","last_fullsized":"2024-09-21T13:49:35.234574+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:48:56.029323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:04.022287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015946669999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":36,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:35.374505+0000","last_change":"2024-09-21T13:49:02.068949+0000","last_active":"2024-09-21T13:49:35.374505+0000","last_peered":"2024-09-21T13:49:35.374505+0000","last_clean":"2024-09-21T13:49:35.374505+0000","last_became_active":"2024-09-21T13:49:02.068611+0000","last_became_peered":"2024-09-21T13:49:02.068611+0000","last_unstale":"2024-09-21T13:49:35.374505+0000","last_undegraded":"2024-09-21T13:49:35.374505+0000","last_fullsized":"2024-09-21T13:49:35.374505+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:01.307406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":32,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:34.913730+0000","last_change":"2024-09-21T13:49:04.080755+0000","last_active":"2024-09-21T13:49:34.913730+0000","last_peered":"2024-09-21T13:49:34.913730+0000","last_clean":"2024-09-21T13:49:34.913730+0000","last_became_active":"2024-09-21T13:49:04.080466+0000","last_became_peered":"2024-09-21T13:49:04.080466+0000","last_unstale":"2024-09-21T13:49:34.913730+0000","last_undegraded":"2024-09-21T13:49:34.913730+0000","last_fullsized":"2024-09-21T13:49:34.913730+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:06.570558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"68'13","reported_seq":69,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.808039+0000","last_change":"2024-09-21T13:49:00.053645+0000","last_active":"2024-09-21T13:49:39.808039+0000","last_peered":"2024-09-21T13:49:39.808039+0000","last_clean":"2024-09-21T13:49:39.808039+0000","last_became_active":"2024-09-21T13:49:00.053323+0000","last_became_peered":"2024-09-21T13:49:00.053323+0000","last_unstale":"2024-09-21T13:49:39.808039+0000","last_undegraded":"2024-09-21T13:49:39.808039+0000","last_fullsized":"2024-09-21T13:49:39.808039+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:02.019380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.806770+0000","last_change":"2024-09-21T13:48:58.050687+0000","last_active":"2024-09-21T13:49:39.806770+0000","last_peered":"2024-09-21T13:49:39.806770+0000","last_clean":"2024-09-21T13:49:39.806770+0000","last_became_active":"2024-09-21T13:48:58.050371+0000","last_became_peered":"2024-09-21T13:48:58.050371+0000","last_unstale":"2024-09-21T13:49:39.806770+0000","last_undegraded":"2024-09-21T13:49:39.806770+0000","last_fullsized":"2024-09-21T13:49:39.806770+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:19.757852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":55,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.807250+0000","last_change":"2024-09-21T13:49:00.045590+0000","last_active":"2024-09-21T13:49:39.807250+0000","last_peered":"2024-09-21T13:49:39.807250+0000","last_clean":"2024-09-21T13:49:39.807250+0000","last_became_active":"2024-09-21T13:48:57.047067+0000","last_became_peered":"2024-09-21T13:48:57.047067+0000","last_unstale":"2024-09-21T13:49:39.807250+0000","last_undegraded":"2024-09-21T13:49:39.807250+0000","last_fullsized":"2024-09-21T13:49:39.807250+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:48:56.029323+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:49:56.701182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065666300000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.807847+0000","last_change":"2024-09-21T13:49:02.057270+0000","last_active":"2024-09-21T13:49:39.807847+0000","last_peered":"2024-09-21T13:49:39.807847+0000","last_clean":"2024-09-21T13:49:39.807847+0000","last_became_active":"2024-09-21T13:49:02.056806+0000","last_became_peered":"2024-09-21T13:49:02.056806+0000","last_unstale":"2024-09-21T13:49:39.807847+0000","last_undegraded":"2024-09-21T13:49:39.807847+0000","last_fullsized":"2024-09-21T13:49:39.807847+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:17.996686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":32,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:35.038114+0000","last_change":"2024-09-21T13:49:04.431351+0000","last_active":"2024-09-21T13:49:35.038114+0000","last_peered":"2024-09-21T13:49:35.038114+0000","last_clean":"2024-09-21T13:49:35.038114+0000","last_became_active":"2024-09-21T13:49:04.431035+0000","last_became_peered":"2024-09-21T13:49:04.431035+0000","last_unstale":"2024-09-21T13:49:35.038114+0000","last_undegraded":"2024-09-21T13:49:35.038114+0000","last_fullsized":"2024-09-21T13:49:35.038114+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:03.519988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"68'12","reported_seq":58,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:35.374654+0000","last_change":"2024-09-21T13:49:00.053466+0000","last_active":"2024-09-21T13:49:35.374654+0000","last_peered":"2024-09-21T13:49:35.374654+0000","last_clean":"2024-09-21T13:49:35.374654+0000","last_became_active":"2024-09-21T13:49:00.053080+0000","last_became_peered":"2024-09-21T13:49:00.053080+0000","last_unstale":"2024-09-21T13:49:35.374654+0000","last_undegraded":"2024-09-21T13:49:35.374654+0000","last_fullsized":"2024-09-21T13:49:35.374654+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:03.884409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":44,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:35.374183+0000","last_change":"2024-09-21T13:48:58.051534+0000","last_active":"2024-09-21T13:49:35.374183+0000","last_peered":"2024-09-21T13:49:35.374183+0000","last_clean":"2024-09-21T13:49:35.374183+0000","last_became_active":"2024-09-21T13:48:58.051249+0000","last_became_peered":"2024-09-21T13:48:58.051249+0000","last_unstale":"2024-09-21T13:49:35.374183+0000","last_undegraded":"2024-09-21T13:49:35.374183+0000","last_fullsized":"2024-09-21T13:49:35.374183+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:04.856323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":90,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:36.930859+0000","last_change":"2024-09-21T13:49:00.426330+0000","last_active":"2024-09-21T13:49:36.930859+0000","last_peered":"2024-09-21T13:49:36.930859+0000","last_clean":"2024-09-21T13:49:36.930859+0000","last_became_active":"2024-09-21T13:48:57.046084+0000","last_became_peered":"2024-09-21T13:48:57.046084+0000","last_unstale":"2024-09-21T13:49:36.930859+0000","last_undegraded":"2024-09-21T13:49:36.930859+0000","last_fullsized":"2024-09-21T13:49:36.930859+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:48:56.029323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:08.036165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047048000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":35,"num_read_kb":30,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":36,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:35.235095+0000","last_change":"2024-09-21T13:49:02.066942+0000","last_active":"2024-09-21T13:49:35.235095+0000","last_peered":"2024-09-21T13:49:35.235095+0000","last_clean":"2024-09-21T13:49:35.235095+0000","last_became_active":"2024-09-21T13:49:02.066641+0000","last_became_peered":"2024-09-21T13:49:02.066641+0000","last_unstale":"2024-09-21T13:49:35.235095+0000","last_undegraded":"2024-09-21T13:49:35.235095+0000","last_fullsized":"2024-09-21T13:49:35.235095+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:22.324406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.808219+0000","last_change":"2024-09-21T13:49:04.078686+0000","last_active":"2024-09-21T13:49:39.808219+0000","last_peered":"2024-09-21T13:49:39.808219+0000","last_clean":"2024-09-21T13:49:39.808219+0000","last_became_active":"2024-09-21T13:49:04.078372+0000","last_became_peered":"2024-09-21T13:49:04.078372+0000","last_unstale":"2024-09-21T13:49:39.808219+0000","last_undegraded":"2024-09-21T13:49:39.808219+0000","last_fullsized":"2024-09-21T13:49:39.808219+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:25.621989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"68'16","reported_seq":74,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836782+0000","last_change":"2024-09-21T13:49:00.427430+0000","last_active":"2024-09-21T13:49:39.836782+0000","last_peered":"2024-09-21T13:49:39.836782+0000","last_clean":"2024-09-21T13:49:39.836782+0000","last_became_active":"2024-09-21T13:49:00.426867+0000","last_became_peered":"2024-09-21T13:49:00.426867+0000","last_unstale":"2024-09-21T13:49:39.836782+0000","last_undegraded":"2024-09-21T13:49:39.836782+0000","last_fullsized":"2024-09-21T13:49:39.836782+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:11.516725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":44,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:34.912425+0000","last_change":"2024-09-21T13:48:58.052925+0000","last_active":"2024-09-21T13:49:34.912425+0000","last_peered":"2024-09-21T13:49:34.912425+0000","last_clean":"2024-09-21T13:49:34.912425+0000","last_became_active":"2024-09-21T13:48:58.052659+0000","last_became_peered":"2024-09-21T13:48:58.052659+0000","last_unstale":"2024-09-21T13:49:34.912425+0000","last_undegraded":"2024-09-21T13:49:34.912425+0000","last_fullsized":"2024-09-21T13:49:34.912425+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:03.529879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"75'79","reported_seq":93,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:35.037863+0000","last_change":"2024-09-21T13:48:51.893262+0000","last_active":"2024-09-21T13:49:35.037863+0000","last_peered":"2024-09-21T13:49:35.037863+0000","last_clean":"2024-09-21T13:49:35.037863+0000","last_became_active":"2024-09-21T13:48:48.454646+0000","last_became_peered":"2024-09-21T13:48:48.454646+0000","last_unstale":"2024-09-21T13:49:35.037863+0000","last_undegraded":"2024-09-21T13:49:35.037863+0000","last_fullsized":"2024-09-21T13:49:35.037863+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'72","last_scrub_stamp":"2024-09-21T13:48:51.893178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:48:51.893178+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:10.164272+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":36,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:35.038075+0000","last_change":"2024-09-21T13:49:02.061930+0000","last_active":"2024-09-21T13:49:35.038075+0000","last_peered":"2024-09-21T13:49:35.038075+0000","last_clean":"2024-09-21T13:49:35.038075+0000","last_became_active":"2024-09-21T13:49:02.061454+0000","last_became_peered":"2024-09-21T13:49:02.061454+0000","last_unstale":"2024-09-21T13:49:35.038075+0000","last_undegraded":"2024-09-21T13:49:35.038075+0000","last_fullsized":"2024-09-21T13:49:35.038075+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:23.000364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":32,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:35.234955+0000","last_change":"2024-09-21T13:49:04.078343+0000","last_active":"2024-09-21T13:49:35.234955+0000","last_peered":"2024-09-21T13:49:35.234955+0000","last_clean":"2024-09-21T13:49:35.234955+0000","last_became_active":"2024-09-21T13:49:04.078091+0000","last_became_peered":"2024-09-21T13:49:04.078091+0000","last_unstale":"2024-09-21T13:49:35.234955+0000","last_undegraded":"2024-09-21T13:49:35.234955+0000","last_fullsized":"2024-09-21T13:49:35.234955+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:17.101054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"68'11","reported_seq":59,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:34.913166+0000","last_change":"2024-09-21T13:49:00.054827+0000","last_active":"2024-09-21T13:49:34.913166+0000","last_peered":"2024-09-21T13:49:34.913166+0000","last_clean":"2024-09-21T13:49:34.913166+0000","last_became_active":"2024-09-21T13:49:00.053268+0000","last_became_peered":"2024-09-21T13:49:00.053268+0000","last_unstale":"2024-09-21T13:49:34.913166+0000","last_undegraded":"2024-09-21T13:49:34.913166+0000","last_fullsized":"2024-09-21T13:49:34.913166+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:28.403874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":44,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:34.912429+0000","last_change":"2024-09-21T13:48:58.052364+0000","last_active":"2024-09-21T13:49:34.912429+0000","last_peered":"2024-09-21T13:49:34.912429+0000","last_clean":"2024-09-21T13:49:34.912429+0000","last_became_active":"2024-09-21T13:48:58.052092+0000","last_became_peered":"2024-09-21T13:48:58.052092+0000","last_unstale":"2024-09-21T13:49:34.912429+0000","last_undegraded":"2024-09-21T13:49:34.912429+0000","last_fullsized":"2024-09-21T13:49:34.912429+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:10.772378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":36,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:35.234927+0000","last_change":"2024-09-21T13:49:02.062262+0000","last_active":"2024-09-21T13:49:35.234927+0000","last_peered":"2024-09-21T13:49:35.234927+0000","last_clean":"2024-09-21T13:49:35.234927+0000","last_became_active":"2024-09-21T13:49:02.061901+0000","last_became_peered":"2024-09-21T13:49:02.061901+0000","last_unstale":"2024-09-21T13:49:35.234927+0000","last_undegraded":"2024-09-21T13:49:35.234927+0000","last_fullsized":"2024-09-21T13:49:35.234927+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:20.147536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":32,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:34.913782+0000","last_change":"2024-09-21T13:49:04.431695+0000","last_active":"2024-09-21T13:49:34.913782+0000","last_peered":"2024-09-21T13:49:34.913782+0000","last_clean":"2024-09-21T13:49:34.913782+0000","last_became_active":"2024-09-21T13:49:04.431268+0000","last_became_peered":"2024-09-21T13:49:04.431268+0000","last_unstale":"2024-09-21T13:49:34.913782+0000","last_undegraded":"2024-09-21T13:49:34.913782+0000","last_fullsized":"2024-09-21T13:49:34.913782+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:27.118735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"68'17","reported_seq":68,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:34.913130+0000","last_change":"2024-09-21T13:49:00.053077+0000","last_active":"2024-09-21T13:49:34.913130+0000","last_peered":"2024-09-21T13:49:34.913130+0000","last_clean":"2024-09-21T13:49:34.913130+0000","last_became_active":"2024-09-21T13:49:00.052608+0000","last_became_peered":"2024-09-21T13:49:00.052608+0000","last_unstale":"2024-09-21T13:49:34.913130+0000","last_undegraded":"2024-09-21T13:49:34.913130+0000","last_fullsized":"2024-09-21T13:49:34.913130+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:14.266475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836235+0000","last_change":"2024-09-21T13:48:58.051751+0000","last_active":"2024-09-21T13:49:39.836235+0000","last_peered":"2024-09-21T13:49:39.836235+0000","last_clean":"2024-09-21T13:49:39.836235+0000","last_became_active":"2024-09-21T13:48:58.049142+0000","last_became_peered":"2024-09-21T13:48:58.049142+0000","last_unstale":"2024-09-21T13:49:39.836235+0000","last_undegraded":"2024-09-21T13:49:39.836235+0000","last_fullsized":"2024-09-21T13:49:39.836235+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:07.453552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.808232+0000","last_change":"2024-09-21T13:49:02.067811+0000","last_active":"2024-09-21T13:49:39.808232+0000","last_peered":"2024-09-21T13:49:39.808232+0000","last_clean":"2024-09-21T13:49:39.808232+0000","last_became_active":"2024-09-21T13:49:02.067472+0000","last_became_peered":"2024-09-21T13:49:02.067472+0000","last_unstale":"2024-09-21T13:49:39.808232+0000","last_undegraded":"2024-09-21T13:49:39.808232+0000","last_fullsized":"2024-09-21T13:49:39.808232+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:29.225485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.808029+0000","last_change":"2024-09-21T13:49:04.078285+0000","last_active":"2024-09-21T13:49:39.808029+0000","last_peered":"2024-09-21T13:49:39.808029+0000","last_clean":"2024-09-21T13:49:39.808029+0000","last_became_active":"2024-09-21T13:49:04.077962+0000","last_became_peered":"2024-09-21T13:49:04.077962+0000","last_unstale":"2024-09-21T13:49:39.808029+0000","last_undegraded":"2024-09-21T13:49:39.808029+0000","last_fullsized":"2024-09-21T13:49:39.808029+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:13.659310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"68'10","reported_seq":55,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:34.912813+0000","last_change":"2024-09-21T13:49:00.060182+0000","last_active":"2024-09-21T13:49:34.912813+0000","last_peered":"2024-09-21T13:49:34.912813+0000","last_clean":"2024-09-21T13:49:34.912813+0000","last_became_active":"2024-09-21T13:49:00.059850+0000","last_became_peered":"2024-09-21T13:49:00.059850+0000","last_unstale":"2024-09-21T13:49:34.912813+0000","last_undegraded":"2024-09-21T13:49:34.912813+0000","last_fullsized":"2024-09-21T13:49:34.912813+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:16.066823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":44,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:34.912940+0000","last_change":"2024-09-21T13:48:58.058603+0000","last_active":"2024-09-21T13:49:34.912940+0000","last_peered":"2024-09-21T13:49:34.912940+0000","last_clean":"2024-09-21T13:49:34.912940+0000","last_became_active":"2024-09-21T13:48:58.058309+0000","last_became_peered":"2024-09-21T13:48:58.058309+0000","last_unstale":"2024-09-21T13:49:34.912940+0000","last_undegraded":"2024-09-21T13:49:34.912940+0000","last_fullsized":"2024-09-21T13:49:34.912940+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:09.079397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"68'8","reported_seq":56,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.807442+0000","last_change":"2024-09-21T13:49:02.065220+0000","last_active":"2024-09-21T13:49:39.807442+0000","last_peered":"2024-09-21T13:49:39.807442+0000","last_clean":"2024-09-21T13:49:39.807442+0000","last_became_active":"2024-09-21T13:49:02.064817+0000","last_became_peered":"2024-09-21T13:49:02.064817+0000","last_unstale":"2024-09-21T13:49:39.807442+0000","last_undegraded":"2024-09-21T13:49:39.807442+0000","last_fullsized":"2024-09-21T13:49:39.807442+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:11.912943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"68'1","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:34.913036+0000","last_change":"2024-09-21T13:49:04.072175+0000","last_active":"2024-09-21T13:49:34.913036+0000","last_peered":"2024-09-21T13:49:34.913036+0000","last_clean":"2024-09-21T13:49:34.913036+0000","last_became_active":"2024-09-21T13:49:04.071885+0000","last_became_peered":"2024-09-21T13:49:04.071885+0000","last_unstale":"2024-09-21T13:49:34.913036+0000","last_undegraded":"2024-09-21T13:49:34.913036+0000","last_fullsized":"2024-09-21T13:49:34.913036+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:03.670445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"68'9","reported_seq":56,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:35.374367+0000","last_change":"2024-09-21T13:49:00.053349+0000","last_active":"2024-09-21T13:49:35.374367+0000","last_peered":"2024-09-21T13:49:35.374367+0000","last_clean":"2024-09-21T13:49:35.374367+0000","last_became_active":"2024-09-21T13:49:00.053016+0000","last_became_peered":"2024-09-21T13:49:00.053016+0000","last_unstale":"2024-09-21T13:49:35.374367+0000","last_undegraded":"2024-09-21T13:49:35.374367+0000","last_fullsized":"2024-09-21T13:49:35.374367+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:06.374913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":44,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:35.234742+0000","last_change":"2024-09-21T13:48:58.050243+0000","last_active":"2024-09-21T13:49:35.234742+0000","last_peered":"2024-09-21T13:49:35.234742+0000","last_clean":"2024-09-21T13:49:35.234742+0000","last_became_active":"2024-09-21T13:48:58.049543+0000","last_became_peered":"2024-09-21T13:48:58.049543+0000","last_unstale":"2024-09-21T13:49:35.234742+0000","last_undegraded":"2024-09-21T13:49:35.234742+0000","last_fullsized":"2024-09-21T13:49:35.234742+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:17.729176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.807325+0000","last_change":"2024-09-21T13:49:02.063877+0000","last_active":"2024-09-21T13:49:39.807325+0000","last_peered":"2024-09-21T13:49:39.807325+0000","last_clean":"2024-09-21T13:49:39.807325+0000","last_became_active":"2024-09-21T13:49:02.063525+0000","last_became_peered":"2024-09-21T13:49:02.063525+0000","last_unstale":"2024-09-21T13:49:39.807325+0000","last_undegraded":"2024-09-21T13:49:39.807325+0000","last_fullsized":"2024-09-21T13:49:39.807325+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:30.535568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":32,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:35.374137+0000","last_change":"2024-09-21T13:49:04.081924+0000","last_active":"2024-09-21T13:49:35.374137+0000","last_peered":"2024-09-21T13:49:35.374137+0000","last_clean":"2024-09-21T13:49:35.374137+0000","last_became_active":"2024-09-21T13:49:04.081617+0000","last_became_peered":"2024-09-21T13:49:04.081617+0000","last_unstale":"2024-09-21T13:49:35.374137+0000","last_undegraded":"2024-09-21T13:49:35.374137+0000","last_fullsized":"2024-09-21T13:49:35.374137+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:25.513722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"68'19","reported_seq":73,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835941+0000","last_change":"2024-09-21T13:49:00.426272+0000","last_active":"2024-09-21T13:49:39.835941+0000","last_peered":"2024-09-21T13:49:39.835941+0000","last_clean":"2024-09-21T13:49:39.835941+0000","last_became_active":"2024-09-21T13:49:00.425955+0000","last_became_peered":"2024-09-21T13:49:00.425955+0000","last_unstale":"2024-09-21T13:49:39.835941+0000","last_undegraded":"2024-09-21T13:49:39.835941+0000","last_fullsized":"2024-09-21T13:49:39.835941+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:02.788196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":44,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:35.038380+0000","last_change":"2024-09-21T13:48:58.053940+0000","last_active":"2024-09-21T13:49:35.038380+0000","last_peered":"2024-09-21T13:49:35.038380+0000","last_clean":"2024-09-21T13:49:35.038380+0000","last_became_active":"2024-09-21T13:48:58.053641+0000","last_became_peered":"2024-09-21T13:48:58.053641+0000","last_unstale":"2024-09-21T13:49:35.038380+0000","last_undegraded":"2024-09-21T13:49:35.038380+0000","last_fullsized":"2024-09-21T13:49:35.038380+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:25.094666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.807300+0000","last_change":"2024-09-21T13:49:02.066996+0000","last_active":"2024-09-21T13:49:39.807300+0000","last_peered":"2024-09-21T13:49:39.807300+0000","last_clean":"2024-09-21T13:49:39.807300+0000","last_became_active":"2024-09-21T13:49:02.066676+0000","last_became_peered":"2024-09-21T13:49:02.066676+0000","last_unstale":"2024-09-21T13:49:39.807300+0000","last_undegraded":"2024-09-21T13:49:39.807300+0000","last_fullsized":"2024-09-21T13:49:39.807300+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:19.241359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":32,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:35.037730+0000","last_change":"2024-09-21T13:49:04.078412+0000","last_active":"2024-09-21T13:49:35.037730+0000","last_peered":"2024-09-21T13:49:35.037730+0000","last_clean":"2024-09-21T13:49:35.037730+0000","last_became_active":"2024-09-21T13:49:04.078084+0000","last_became_peered":"2024-09-21T13:49:04.078084+0000","last_unstale":"2024-09-21T13:49:35.037730+0000","last_undegraded":"2024-09-21T13:49:35.037730+0000","last_fullsized":"2024-09-21T13:49:35.037730+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:11.517342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"68'12","reported_seq":63,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:34.912333+0000","last_change":"2024-09-21T13:49:00.060188+0000","last_active":"2024-09-21T13:49:34.912333+0000","last_peered":"2024-09-21T13:49:34.912333+0000","last_clean":"2024-09-21T13:49:34.912333+0000","last_became_active":"2024-09-21T13:49:00.059720+0000","last_became_peered":"2024-09-21T13:49:00.059720+0000","last_unstale":"2024-09-21T13:49:34.912333+0000","last_undegraded":"2024-09-21T13:49:34.912333+0000","last_fullsized":"2024-09-21T13:49:34.912333+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:17.978182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":44,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:35.038485+0000","last_change":"2024-09-21T13:48:58.058654+0000","last_active":"2024-09-21T13:49:35.038485+0000","last_peered":"2024-09-21T13:49:35.038485+0000","last_clean":"2024-09-21T13:49:35.038485+0000","last_became_active":"2024-09-21T13:48:58.057641+0000","last_became_peered":"2024-09-21T13:48:58.057641+0000","last_unstale":"2024-09-21T13:49:35.038485+0000","last_undegraded":"2024-09-21T13:49:35.038485+0000","last_fullsized":"2024-09-21T13:49:35.038485+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:11.120209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.807118+0000","last_change":"2024-09-21T13:49:02.057568+0000","last_active":"2024-09-21T13:49:39.807118+0000","last_peered":"2024-09-21T13:49:39.807118+0000","last_clean":"2024-09-21T13:49:39.807118+0000","last_became_active":"2024-09-21T13:49:02.057195+0000","last_became_peered":"2024-09-21T13:49:02.057195+0000","last_unstale":"2024-09-21T13:49:39.807118+0000","last_undegraded":"2024-09-21T13:49:39.807118+0000","last_fullsized":"2024-09-21T13:49:39.807118+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:01.498483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":32,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:34.912615+0000","last_change":"2024-09-21T13:49:04.078422+0000","last_active":"2024-09-21T13:49:34.912615+0000","last_peered":"2024-09-21T13:49:34.912615+0000","last_clean":"2024-09-21T13:49:34.912615+0000","last_became_active":"2024-09-21T13:49:04.078089+0000","last_became_peered":"2024-09-21T13:49:04.078089+0000","last_unstale":"2024-09-21T13:49:34.912615+0000","last_undegraded":"2024-09-21T13:49:34.912615+0000","last_fullsized":"2024-09-21T13:49:34.912615+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:18.974401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"68'15","reported_seq":65,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:35.234189+0000","last_change":"2024-09-21T13:49:00.426358+0000","last_active":"2024-09-21T13:49:35.234189+0000","last_peered":"2024-09-21T13:49:35.234189+0000","last_clean":"2024-09-21T13:49:35.234189+0000","last_became_active":"2024-09-21T13:49:00.425997+0000","last_became_peered":"2024-09-21T13:49:00.425997+0000","last_unstale":"2024-09-21T13:49:35.234189+0000","last_undegraded":"2024-09-21T13:49:35.234189+0000","last_fullsized":"2024-09-21T13:49:35.234189+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:03.382004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":56,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:35.038493+0000","last_change":"2024-09-21T13:48:58.058778+0000","last_active":"2024-09-21T13:49:35.038493+0000","last_peered":"2024-09-21T13:49:35.038493+0000","last_clean":"2024-09-21T13:49:35.038493+0000","last_became_active":"2024-09-21T13:48:58.057632+0000","last_became_peered":"2024-09-21T13:48:58.057632+0000","last_unstale":"2024-09-21T13:49:35.038493+0000","last_undegraded":"2024-09-21T13:49:35.038493+0000","last_fullsized":"2024-09-21T13:49:35.038493+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:06.995889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"68'8","reported_seq":53,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:35.037737+0000","last_change":"2024-09-21T13:49:02.056471+0000","last_active":"2024-09-21T13:49:35.037737+0000","last_peered":"2024-09-21T13:49:35.037737+0000","last_clean":"2024-09-21T13:49:35.037737+0000","last_became_active":"2024-09-21T13:49:02.056167+0000","last_became_peered":"2024-09-21T13:49:02.056167+0000","last_unstale":"2024-09-21T13:49:35.037737+0000","last_undegraded":"2024-09-21T13:49:35.037737+0000","last_fullsized":"2024-09-21T13:49:35.037737+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:10.093140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":32,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:35.234326+0000","last_change":"2024-09-21T13:49:04.432809+0000","last_active":"2024-09-21T13:49:35.234326+0000","last_peered":"2024-09-21T13:49:35.234326+0000","last_clean":"2024-09-21T13:49:35.234326+0000","last_became_active":"2024-09-21T13:49:04.432546+0000","last_became_peered":"2024-09-21T13:49:04.432546+0000","last_unstale":"2024-09-21T13:49:35.234326+0000","last_undegraded":"2024-09-21T13:49:35.234326+0000","last_fullsized":"2024-09-21T13:49:35.234326+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:08.569714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836980+0000","last_change":"2024-09-21T13:48:58.056383+0000","last_active":"2024-09-21T13:49:39.836980+0000","last_peered":"2024-09-21T13:49:39.836980+0000","last_clean":"2024-09-21T13:49:39.836980+0000","last_became_active":"2024-09-21T13:48:58.056080+0000","last_became_peered":"2024-09-21T13:48:58.056080+0000","last_unstale":"2024-09-21T13:49:39.836980+0000","last_undegraded":"2024-09-21T13:49:39.836980+0000","last_fullsized":"2024-09-21T13:49:39.836980+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:26.306822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"68'6","reported_seq":49,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:34.914970+0000","last_change":"2024-09-21T13:49:00.427204+0000","last_active":"2024-09-21T13:49:34.914970+0000","last_peered":"2024-09-21T13:49:34.914970+0000","last_clean":"2024-09-21T13:49:34.914970+0000","last_became_active":"2024-09-21T13:49:00.426829+0000","last_became_peered":"2024-09-21T13:49:00.426829+0000","last_unstale":"2024-09-21T13:49:34.914970+0000","last_undegraded":"2024-09-21T13:49:34.914970+0000","last_fullsized":"2024-09-21T13:49:34.914970+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:20.014163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":36,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:35.234833+0000","last_change":"2024-09-21T13:49:02.062503+0000","last_active":"2024-09-21T13:49:35.234833+0000","last_peered":"2024-09-21T13:49:35.234833+0000","last_clean":"2024-09-21T13:49:35.234833+0000","last_became_active":"2024-09-21T13:49:02.062231+0000","last_became_peered":"2024-09-21T13:49:02.062231+0000","last_unstale":"2024-09-21T13:49:35.234833+0000","last_undegraded":"2024-09-21T13:49:35.234833+0000","last_fullsized":"2024-09-21T13:49:35.234833+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:21.288932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":32,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:35.038067+0000","last_change":"2024-09-21T13:49:04.432777+0000","last_active":"2024-09-21T13:49:35.038067+0000","last_peered":"2024-09-21T13:49:35.038067+0000","last_clean":"2024-09-21T13:49:35.038067+0000","last_became_active":"2024-09-21T13:49:04.432507+0000","last_became_peered":"2024-09-21T13:49:04.432507+0000","last_unstale":"2024-09-21T13:49:35.038067+0000","last_undegraded":"2024-09-21T13:49:35.038067+0000","last_fullsized":"2024-09-21T13:49:35.038067+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:03.727186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"68'9","reported_seq":56,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:35.374689+0000","last_change":"2024-09-21T13:49:00.428704+0000","last_active":"2024-09-21T13:49:35.374689+0000","last_peered":"2024-09-21T13:49:35.374689+0000","last_clean":"2024-09-21T13:49:35.374689+0000","last_became_active":"2024-09-21T13:49:00.428330+0000","last_became_peered":"2024-09-21T13:49:00.428330+0000","last_unstale":"2024-09-21T13:49:35.374689+0000","last_undegraded":"2024-09-21T13:49:35.374689+0000","last_fullsized":"2024-09-21T13:49:35.374689+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:06.325729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"61'1","reported_seq":50,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:35.037919+0000","last_change":"2024-09-21T13:48:58.050948+0000","last_active":"2024-09-21T13:49:35.037919+0000","last_peered":"2024-09-21T13:49:35.037919+0000","last_clean":"2024-09-21T13:49:35.037919+0000","last_became_active":"2024-09-21T13:48:58.050645+0000","last_became_peered":"2024-09-21T13:48:58.050645+0000","last_unstale":"2024-09-21T13:49:35.037919+0000","last_undegraded":"2024-09-21T13:49:35.037919+0000","last_fullsized":"2024-09-21T13:49:35.037919+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:24.159998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":36,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:34.914837+0000","last_change":"2024-09-21T13:49:02.062075+0000","last_active":"2024-09-21T13:49:34.914837+0000","last_peered":"2024-09-21T13:49:34.914837+0000","last_clean":"2024-09-21T13:49:34.914837+0000","last_became_active":"2024-09-21T13:49:02.061775+0000","last_became_peered":"2024-09-21T13:49:02.061775+0000","last_unstale":"2024-09-21T13:49:34.914837+0000","last_undegraded":"2024-09-21T13:49:34.914837+0000","last_fullsized":"2024-09-21T13:49:34.914837+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:16.428002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.807270+0000","last_change":"2024-09-21T13:49:04.069452+0000","last_active":"2024-09-21T13:49:39.807270+0000","last_peered":"2024-09-21T13:49:39.807270+0000","last_clean":"2024-09-21T13:49:39.807270+0000","last_became_active":"2024-09-21T13:49:04.069142+0000","last_became_peered":"2024-09-21T13:49:04.069142+0000","last_unstale":"2024-09-21T13:49:39.807270+0000","last_undegraded":"2024-09-21T13:49:39.807270+0000","last_fullsized":"2024-09-21T13:49:39.807270+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:28.894052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"68'9","reported_seq":56,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:35.234894+0000","last_change":"2024-09-21T13:49:00.428080+0000","last_active":"2024-09-21T13:49:35.234894+0000","last_peered":"2024-09-21T13:49:35.234894+0000","last_clean":"2024-09-21T13:49:35.234894+0000","last_became_active":"2024-09-21T13:49:00.427766+0000","last_became_peered":"2024-09-21T13:49:00.427766+0000","last_unstale":"2024-09-21T13:49:35.234894+0000","last_undegraded":"2024-09-21T13:49:35.234894+0000","last_fullsized":"2024-09-21T13:49:35.234894+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:16.384643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":44,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:35.374272+0000","last_change":"2024-09-21T13:48:58.052465+0000","last_active":"2024-09-21T13:49:35.374272+0000","last_peered":"2024-09-21T13:49:35.374272+0000","last_clean":"2024-09-21T13:49:35.374272+0000","last_became_active":"2024-09-21T13:48:58.052151+0000","last_became_peered":"2024-09-21T13:48:58.052151+0000","last_unstale":"2024-09-21T13:49:35.374272+0000","last_undegraded":"2024-09-21T13:49:35.374272+0000","last_fullsized":"2024-09-21T13:49:35.374272+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:09.226461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"68'8","reported_seq":53,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:34.914027+0000","last_change":"2024-09-21T13:49:02.066442+0000","last_active":"2024-09-21T13:49:34.914027+0000","last_peered":"2024-09-21T13:49:34.914027+0000","last_clean":"2024-09-21T13:49:34.914027+0000","last_became_active":"2024-09-21T13:49:02.065602+0000","last_became_peered":"2024-09-21T13:49:02.065602+0000","last_unstale":"2024-09-21T13:49:34.914027+0000","last_undegraded":"2024-09-21T13:49:34.914027+0000","last_fullsized":"2024-09-21T13:49:34.914027+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:10.855295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":32,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:34.913143+0000","last_change":"2024-09-21T13:49:04.073461+0000","last_active":"2024-09-21T13:49:34.913143+0000","last_peered":"2024-09-21T13:49:34.913143+0000","last_clean":"2024-09-21T13:49:34.913143+0000","last_became_active":"2024-09-21T13:49:04.073188+0000","last_became_peered":"2024-09-21T13:49:04.073188+0000","last_unstale":"2024-09-21T13:49:34.913143+0000","last_undegraded":"2024-09-21T13:49:34.913143+0000","last_fullsized":"2024-09-21T13:49:34.913143+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:08.749766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"68'10","reported_seq":55,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:34.914204+0000","last_change":"2024-09-21T13:49:00.427262+0000","last_active":"2024-09-21T13:49:34.914204+0000","last_peered":"2024-09-21T13:49:34.914204+0000","last_clean":"2024-09-21T13:49:34.914204+0000","last_became_active":"2024-09-21T13:49:00.426071+0000","last_became_peered":"2024-09-21T13:49:00.426071+0000","last_unstale":"2024-09-21T13:49:34.914204+0000","last_undegraded":"2024-09-21T13:49:34.914204+0000","last_fullsized":"2024-09-21T13:49:34.914204+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:03.592724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":44,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:35.038181+0000","last_change":"2024-09-21T13:48:58.058198+0000","last_active":"2024-09-21T13:49:35.038181+0000","last_peered":"2024-09-21T13:49:35.038181+0000","last_clean":"2024-09-21T13:49:35.038181+0000","last_became_active":"2024-09-21T13:48:58.057587+0000","last_became_peered":"2024-09-21T13:48:58.057587+0000","last_unstale":"2024-09-21T13:49:35.038181+0000","last_undegraded":"2024-09-21T13:49:35.038181+0000","last_fullsized":"2024-09-21T13:49:35.038181+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:19.518354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"68'8","reported_seq":53,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:35.234812+0000","last_change":"2024-09-21T13:49:02.067827+0000","last_active":"2024-09-21T13:49:35.234812+0000","last_peered":"2024-09-21T13:49:35.234812+0000","last_clean":"2024-09-21T13:49:35.234812+0000","last_became_active":"2024-09-21T13:49:02.067487+0000","last_became_peered":"2024-09-21T13:49:02.067487+0000","last_unstale":"2024-09-21T13:49:35.234812+0000","last_undegraded":"2024-09-21T13:49:35.234812+0000","last_fullsized":"2024-09-21T13:49:35.234812+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:23.003843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":32,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:35.374465+0000","last_change":"2024-09-21T13:49:04.081928+0000","last_active":"2024-09-21T13:49:35.374465+0000","last_peered":"2024-09-21T13:49:35.374465+0000","last_clean":"2024-09-21T13:49:35.374465+0000","last_became_active":"2024-09-21T13:49:04.081609+0000","last_became_peered":"2024-09-21T13:49:04.081609+0000","last_unstale":"2024-09-21T13:49:35.374465+0000","last_undegraded":"2024-09-21T13:49:35.374465+0000","last_fullsized":"2024-09-21T13:49:35.374465+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:31.380912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"68'11","reported_seq":59,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:34.912681+0000","last_change":"2024-09-21T13:49:00.055420+0000","last_active":"2024-09-21T13:49:34.912681+0000","last_peered":"2024-09-21T13:49:34.912681+0000","last_clean":"2024-09-21T13:49:34.912681+0000","last_became_active":"2024-09-21T13:49:00.055024+0000","last_became_peered":"2024-09-21T13:49:00.055024+0000","last_unstale":"2024-09-21T13:49:34.912681+0000","last_undegraded":"2024-09-21T13:49:34.912681+0000","last_fullsized":"2024-09-21T13:49:34.912681+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:02.449612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":44,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:34.913008+0000","last_change":"2024-09-21T13:48:58.051845+0000","last_active":"2024-09-21T13:49:34.913008+0000","last_peered":"2024-09-21T13:49:34.913008+0000","last_clean":"2024-09-21T13:49:34.913008+0000","last_became_active":"2024-09-21T13:48:58.051541+0000","last_became_peered":"2024-09-21T13:48:58.051541+0000","last_unstale":"2024-09-21T13:49:34.913008+0000","last_undegraded":"2024-09-21T13:49:34.913008+0000","last_fullsized":"2024-09-21T13:49:34.913008+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:03.712482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.806822+0000","last_change":"2024-09-21T13:49:02.063328+0000","last_active":"2024-09-21T13:49:39.806822+0000","last_peered":"2024-09-21T13:49:39.806822+0000","last_clean":"2024-09-21T13:49:39.806822+0000","last_became_active":"2024-09-21T13:49:02.062868+0000","last_became_peered":"2024-09-21T13:49:02.062868+0000","last_unstale":"2024-09-21T13:49:39.806822+0000","last_undegraded":"2024-09-21T13:49:39.806822+0000","last_fullsized":"2024-09-21T13:49:39.806822+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:12.568168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":32,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:34.911897+0000","last_change":"2024-09-21T13:49:04.083254+0000","last_active":"2024-09-21T13:49:34.911897+0000","last_peered":"2024-09-21T13:49:34.911897+0000","last_clean":"2024-09-21T13:49:34.911897+0000","last_became_active":"2024-09-21T13:49:04.082961+0000","last_became_peered":"2024-09-21T13:49:04.082961+0000","last_unstale":"2024-09-21T13:49:34.911897+0000","last_undegraded":"2024-09-21T13:49:34.911897+0000","last_fullsized":"2024-09-21T13:49:34.911897+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:07.548695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"68'9","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.807587+0000","last_change":"2024-09-21T13:49:00.057817+0000","last_active":"2024-09-21T13:49:39.807587+0000","last_peered":"2024-09-21T13:49:39.807587+0000","last_clean":"2024-09-21T13:49:39.807587+0000","last_became_active":"2024-09-21T13:49:00.057392+0000","last_became_peered":"2024-09-21T13:49:00.057392+0000","last_unstale":"2024-09-21T13:49:39.807587+0000","last_undegraded":"2024-09-21T13:49:39.807587+0000","last_fullsized":"2024-09-21T13:49:39.807587+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:02.682820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":44,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:35.038282+0000","last_change":"2024-09-21T13:48:58.058032+0000","last_active":"2024-09-21T13:49:35.038282+0000","last_peered":"2024-09-21T13:49:35.038282+0000","last_clean":"2024-09-21T13:49:35.038282+0000","last_became_active":"2024-09-21T13:48:58.057516+0000","last_became_peered":"2024-09-21T13:48:58.057516+0000","last_unstale":"2024-09-21T13:49:35.038282+0000","last_undegraded":"2024-09-21T13:49:35.038282+0000","last_fullsized":"2024-09-21T13:49:35.038282+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:20.885486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":36,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:34.912702+0000","last_change":"2024-09-21T13:49:02.072264+0000","last_active":"2024-09-21T13:49:34.912702+0000","last_peered":"2024-09-21T13:49:34.912702+0000","last_clean":"2024-09-21T13:49:34.912702+0000","last_became_active":"2024-09-21T13:49:02.071922+0000","last_became_peered":"2024-09-21T13:49:02.071922+0000","last_unstale":"2024-09-21T13:49:34.912702+0000","last_undegraded":"2024-09-21T13:49:34.912702+0000","last_fullsized":"2024-09-21T13:49:34.912702+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:12.773964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":32,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:35.373922+0000","last_change":"2024-09-21T13:49:04.080300+0000","last_active":"2024-09-21T13:49:35.373922+0000","last_peered":"2024-09-21T13:49:35.373922+0000","last_clean":"2024-09-21T13:49:35.373922+0000","last_became_active":"2024-09-21T13:49:04.079950+0000","last_became_peered":"2024-09-21T13:49:04.079950+0000","last_unstale":"2024-09-21T13:49:35.373922+0000","last_undegraded":"2024-09-21T13:49:35.373922+0000","last_fullsized":"2024-09-21T13:49:35.373922+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:20.474523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"68'11","reported_seq":59,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:34.913110+0000","last_change":"2024-09-21T13:49:00.426487+0000","last_active":"2024-09-21T13:49:34.913110+0000","last_peered":"2024-09-21T13:49:34.913110+0000","last_clean":"2024-09-21T13:49:34.913110+0000","last_became_active":"2024-09-21T13:49:00.426091+0000","last_became_peered":"2024-09-21T13:49:00.426091+0000","last_unstale":"2024-09-21T13:49:34.913110+0000","last_undegraded":"2024-09-21T13:49:34.913110+0000","last_fullsized":"2024-09-21T13:49:34.913110+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:11.657590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":44,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:35.234902+0000","last_change":"2024-09-21T13:48:58.052281+0000","last_active":"2024-09-21T13:49:35.234902+0000","last_peered":"2024-09-21T13:49:35.234902+0000","last_clean":"2024-09-21T13:49:35.234902+0000","last_became_active":"2024-09-21T13:48:58.051974+0000","last_became_peered":"2024-09-21T13:48:58.051974+0000","last_unstale":"2024-09-21T13:49:35.234902+0000","last_undegraded":"2024-09-21T13:49:35.234902+0000","last_fullsized":"2024-09-21T13:49:35.234902+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:25.614472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":36,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:35.037480+0000","last_change":"2024-09-21T13:49:02.056666+0000","last_active":"2024-09-21T13:49:35.037480+0000","last_peered":"2024-09-21T13:49:35.037480+0000","last_clean":"2024-09-21T13:49:35.037480+0000","last_became_active":"2024-09-21T13:49:02.056006+0000","last_became_peered":"2024-09-21T13:49:02.056006+0000","last_unstale":"2024-09-21T13:49:35.037480+0000","last_undegraded":"2024-09-21T13:49:35.037480+0000","last_fullsized":"2024-09-21T13:49:35.037480+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:17.511468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":32,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:34.912850+0000","last_change":"2024-09-21T13:49:04.434380+0000","last_active":"2024-09-21T13:49:34.912850+0000","last_peered":"2024-09-21T13:49:34.912850+0000","last_clean":"2024-09-21T13:49:34.912850+0000","last_became_active":"2024-09-21T13:49:04.434034+0000","last_became_peered":"2024-09-21T13:49:04.434034+0000","last_unstale":"2024-09-21T13:49:34.912850+0000","last_undegraded":"2024-09-21T13:49:34.912850+0000","last_fullsized":"2024-09-21T13:49:34.912850+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:28.547927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"68'4","reported_seq":46,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:34.913719+0000","last_change":"2024-09-21T13:49:00.063374+0000","last_active":"2024-09-21T13:49:34.913719+0000","last_peered":"2024-09-21T13:49:34.913719+0000","last_clean":"2024-09-21T13:49:34.913719+0000","last_became_active":"2024-09-21T13:49:00.063066+0000","last_became_peered":"2024-09-21T13:49:00.063066+0000","last_unstale":"2024-09-21T13:49:34.913719+0000","last_undegraded":"2024-09-21T13:49:34.913719+0000","last_fullsized":"2024-09-21T13:49:34.913719+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:19.893798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":44,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:35.374721+0000","last_change":"2024-09-21T13:48:58.059979+0000","last_active":"2024-09-21T13:49:35.374721+0000","last_peered":"2024-09-21T13:49:35.374721+0000","last_clean":"2024-09-21T13:49:35.374721+0000","last_became_active":"2024-09-21T13:48:58.059674+0000","last_became_peered":"2024-09-21T13:48:58.059674+0000","last_unstale":"2024-09-21T13:49:35.374721+0000","last_undegraded":"2024-09-21T13:49:35.374721+0000","last_fullsized":"2024-09-21T13:49:35.374721+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:02.308942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835914+0000","last_change":"2024-09-21T13:49:02.065043+0000","last_active":"2024-09-21T13:49:39.835914+0000","last_peered":"2024-09-21T13:49:39.835914+0000","last_clean":"2024-09-21T13:49:39.835914+0000","last_became_active":"2024-09-21T13:49:02.064789+0000","last_became_peered":"2024-09-21T13:49:02.064789+0000","last_unstale":"2024-09-21T13:49:39.835914+0000","last_undegraded":"2024-09-21T13:49:39.835914+0000","last_fullsized":"2024-09-21T13:49:39.835914+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:10.569664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":32,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:35.037725+0000","last_change":"2024-09-21T13:49:04.075170+0000","last_active":"2024-09-21T13:49:35.037725+0000","last_peered":"2024-09-21T13:49:35.037725+0000","last_clean":"2024-09-21T13:49:35.037725+0000","last_became_active":"2024-09-21T13:49:04.074857+0000","last_became_peered":"2024-09-21T13:49:04.074857+0000","last_unstale":"2024-09-21T13:49:35.037725+0000","last_undegraded":"2024-09-21T13:49:35.037725+0000","last_fullsized":"2024-09-21T13:49:35.037725+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:17.971950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.808102+0000","last_change":"2024-09-21T13:49:04.075860+0000","last_active":"2024-09-21T13:49:39.808102+0000","last_peered":"2024-09-21T13:49:39.808102+0000","last_clean":"2024-09-21T13:49:39.808102+0000","last_became_active":"2024-09-21T13:49:04.075562+0000","last_became_peered":"2024-09-21T13:49:04.075562+0000","last_unstale":"2024-09-21T13:49:39.808102+0000","last_undegraded":"2024-09-21T13:49:39.808102+0000","last_fullsized":"2024-09-21T13:49:39.808102+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:21.041626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":44,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:34.913965+0000","last_change":"2024-09-21T13:48:58.055791+0000","last_active":"2024-09-21T13:49:34.913965+0000","last_peered":"2024-09-21T13:49:34.913965+0000","last_clean":"2024-09-21T13:49:34.913965+0000","last_became_active":"2024-09-21T13:48:58.055497+0000","last_became_peered":"2024-09-21T13:48:58.055497+0000","last_unstale":"2024-09-21T13:49:34.913965+0000","last_undegraded":"2024-09-21T13:49:34.913965+0000","last_fullsized":"2024-09-21T13:49:34.913965+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:19.570880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"68'11","reported_seq":61,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836752+0000","last_change":"2024-09-21T13:49:00.058623+0000","last_active":"2024-09-21T13:49:39.836752+0000","last_peered":"2024-09-21T13:49:39.836752+0000","last_clean":"2024-09-21T13:49:39.836752+0000","last_became_active":"2024-09-21T13:49:00.058259+0000","last_became_peered":"2024-09-21T13:49:00.058259+0000","last_unstale":"2024-09-21T13:49:39.836752+0000","last_undegraded":"2024-09-21T13:49:39.836752+0000","last_fullsized":"2024-09-21T13:49:39.836752+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:17.663571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":36,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T13:49:35.374563+0000","last_change":"2024-09-21T13:49:02.068530+0000","last_active":"2024-09-21T13:49:35.374563+0000","last_peered":"2024-09-21T13:49:35.374563+0000","last_clean":"2024-09-21T13:49:35.374563+0000","last_became_active":"2024-09-21T13:49:02.066159+0000","last_became_peered":"2024-09-21T13:49:02.066159+0000","last_unstale":"2024-09-21T13:49:35.374563+0000","last_undegraded":"2024-09-21T13:49:35.374563+0000","last_fullsized":"2024-09-21T13:49:35.374563+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:29.718157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":35,"num_read_kb":30,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928233997,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32616,"kb_used_data":1428,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712536,"statfs":{"total":95995035648,"available":95961636864,"internally_reserved":0,"allocated":1462272,"data_stored":962832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462931,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28448,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716704,"statfs":{"total":95995035648,"available":95965904896,"internally_reserved":0,"allocated":1454080,"data_stored":969368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724569,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27860,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717292,"statfs":{"total":95995035648,"available":95966507008,"internally_reserved":0,"allocated":851968,"data_stored":372295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920799,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":880640,"data_stored":371997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117028,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27892,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717260,"statfs":{"total":95995035648,"available":95966474240,"internally_reserved":0,"allocated":884736,"data_stored":372827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411370,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32028,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713124,"statfs":{"total":95995035648,"available":95962238976,"internally_reserved":0,"allocated":860160,"data_stored":379718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32092,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713060,"statfs":{"total":95995035648,"available":95962173440,"internally_reserved":0,"allocated":925696,"data_stored":380362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738423,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28456,"kb_used_data":1428,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716696,"statfs":{"total":95995035648,"available":95965896704,"internally_reserved":0,"allocated":1462272,"data_stored":962105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T13:49:44.669 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T13:49:44.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:44 smithi187 ceph-mon[33032]: pgmap v14: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 607 B/s rd, 0 op/s 2024-09-21T13:49:44.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:44 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:44.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:44 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:44.947 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:49:45.407 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:45 smithi049 ceph-mon[30080]: from='client.14733 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T13:49:45.407 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:45 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:45.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:45 smithi187 ceph-mon[33032]: from='client.14733 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T13:49:45.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:45 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:45.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:45 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:45.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:45 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T13:49:45.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:45 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T13:49:45.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:45 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T13:49:45.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:45 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T13:49:45.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:45 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:45.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:45 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:49:45.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:45 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:49:45.746 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:45 smithi049 ceph-mon[40489]: from='client.14733 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T13:49:45.746 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:45 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:45.746 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:45 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:45.747 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:45 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T13:49:45.747 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:45 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T13:49:45.747 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:45 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T13:49:45.747 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:45 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T13:49:45.747 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:45 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:45.747 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:45 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:49:45.747 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:45 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:49:45.747 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:45 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:45.747 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:45 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T13:49:45.747 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:45 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T13:49:45.747 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:45 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T13:49:45.747 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:45 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T13:49:45.748 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:45 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:45.748 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:45 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:49:45.748 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:45 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:49:45.828 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:49:45.830 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-09-21T13:49:46.565 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:46 smithi049 ceph-mon[40489]: Adjusting osd_memory_target on smithi049 to 3760M 2024-09-21T13:49:46.565 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:46 smithi049 ceph-mon[40489]: Updating smithi049:/etc/ceph/ceph.conf 2024-09-21T13:49:46.565 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:46 smithi049 ceph-mon[40489]: Updating smithi187:/etc/ceph/ceph.conf 2024-09-21T13:49:46.565 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:46 smithi049 ceph-mon[40489]: pgmap v15: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-21T13:49:46.566 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:46 smithi049 ceph-mon[40489]: Updating smithi187:/var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T13:49:46.566 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:46 smithi049 ceph-mon[40489]: Updating smithi049:/var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T13:49:46.566 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:46 smithi049 ceph-mon[40489]: Updating smithi187:/etc/ceph/ceph.client.admin.keyring 2024-09-21T13:49:46.566 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:46 smithi049 ceph-mon[40489]: Updating smithi049:/etc/ceph/ceph.client.admin.keyring 2024-09-21T13:49:46.566 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:46 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:46.566 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:46 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:46.566 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:46 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:46.566 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:46 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:46.566 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:46 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:46.566 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:46 smithi049 ceph-mon[30080]: Adjusting osd_memory_target on smithi049 to 3760M 2024-09-21T13:49:46.567 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:46 smithi049 ceph-mon[30080]: Updating smithi049:/etc/ceph/ceph.conf 2024-09-21T13:49:46.567 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:46 smithi049 ceph-mon[30080]: Updating smithi187:/etc/ceph/ceph.conf 2024-09-21T13:49:46.567 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:46 smithi049 ceph-mon[30080]: pgmap v15: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-21T13:49:46.567 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:46 smithi049 ceph-mon[30080]: Updating smithi187:/var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T13:49:46.567 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:46 smithi049 ceph-mon[30080]: Updating smithi049:/var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T13:49:46.567 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:46 smithi049 ceph-mon[30080]: Updating smithi187:/etc/ceph/ceph.client.admin.keyring 2024-09-21T13:49:46.567 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:46 smithi049 ceph-mon[30080]: Updating smithi049:/etc/ceph/ceph.client.admin.keyring 2024-09-21T13:49:46.567 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:46 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:46.567 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:46 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:46.567 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:46 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:46.567 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:46 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:46.567 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:46 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:46.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:46 smithi187 ceph-mon[33032]: Adjusting osd_memory_target on smithi049 to 3760M 2024-09-21T13:49:46.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:46 smithi187 ceph-mon[33032]: Updating smithi049:/etc/ceph/ceph.conf 2024-09-21T13:49:46.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:46 smithi187 ceph-mon[33032]: Updating smithi187:/etc/ceph/ceph.conf 2024-09-21T13:49:46.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:46 smithi187 ceph-mon[33032]: pgmap v15: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-21T13:49:46.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:46 smithi187 ceph-mon[33032]: Updating smithi187:/var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T13:49:46.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:46 smithi187 ceph-mon[33032]: Updating smithi049:/var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T13:49:46.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:46 smithi187 ceph-mon[33032]: Updating smithi187:/etc/ceph/ceph.client.admin.keyring 2024-09-21T13:49:46.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:46 smithi187 ceph-mon[33032]: Updating smithi049:/etc/ceph/ceph.client.admin.keyring 2024-09-21T13:49:46.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:46 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:46.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:46 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:46.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:46 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:46.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:46 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:46.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:46 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:46.798 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2024-09-21T13:49:45.332881+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":888,"num_read_kb":745,"num_write":549,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":552,"ondisk_log_size":552,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":237340,"kb_used_data":8636,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":749723876,"statfs":{"total":767960285184,"available":767717249024,"internally_reserved":0,"allocated":8843264,"data_stored":4809609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002103"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836141+0000","last_change":"2024-09-21T13:48:58.061797+0000","last_active":"2024-09-21T13:49:39.836141+0000","last_peered":"2024-09-21T13:49:39.836141+0000","last_clean":"2024-09-21T13:49:39.836141+0000","last_became_active":"2024-09-21T13:48:58.061415+0000","last_became_peered":"2024-09-21T13:48:58.061415+0000","last_unstale":"2024-09-21T13:49:39.836141+0000","last_undegraded":"2024-09-21T13:49:39.836141+0000","last_fullsized":"2024-09-21T13:49:39.836141+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:07.025619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"68'9","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.811507+0000","last_change":"2024-09-21T13:49:00.055717+0000","last_active":"2024-09-21T13:49:39.811507+0000","last_peered":"2024-09-21T13:49:39.811507+0000","last_clean":"2024-09-21T13:49:39.811507+0000","last_became_active":"2024-09-21T13:49:00.055210+0000","last_became_peered":"2024-09-21T13:49:00.055210+0000","last_unstale":"2024-09-21T13:49:39.811507+0000","last_undegraded":"2024-09-21T13:49:39.811507+0000","last_fullsized":"2024-09-21T13:49:39.811507+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:13.541017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.806595+0000","last_change":"2024-09-21T13:49:02.061394+0000","last_active":"2024-09-21T13:49:39.806595+0000","last_peered":"2024-09-21T13:49:39.806595+0000","last_clean":"2024-09-21T13:49:39.806595+0000","last_became_active":"2024-09-21T13:49:02.060984+0000","last_became_peered":"2024-09-21T13:49:02.060984+0000","last_unstale":"2024-09-21T13:49:39.806595+0000","last_undegraded":"2024-09-21T13:49:39.806595+0000","last_fullsized":"2024-09-21T13:49:39.806595+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:18.936236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.811533+0000","last_change":"2024-09-21T13:49:04.072734+0000","last_active":"2024-09-21T13:49:39.811533+0000","last_peered":"2024-09-21T13:49:39.811533+0000","last_clean":"2024-09-21T13:49:39.811533+0000","last_became_active":"2024-09-21T13:49:04.072460+0000","last_became_peered":"2024-09-21T13:49:04.072460+0000","last_unstale":"2024-09-21T13:49:39.811533+0000","last_undegraded":"2024-09-21T13:49:39.811533+0000","last_fullsized":"2024-09-21T13:49:39.811533+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:30.526897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836010+0000","last_change":"2024-09-21T13:49:04.434312+0000","last_active":"2024-09-21T13:49:39.836010+0000","last_peered":"2024-09-21T13:49:39.836010+0000","last_clean":"2024-09-21T13:49:39.836010+0000","last_became_active":"2024-09-21T13:49:04.433965+0000","last_became_peered":"2024-09-21T13:49:04.433965+0000","last_unstale":"2024-09-21T13:49:39.836010+0000","last_undegraded":"2024-09-21T13:49:39.836010+0000","last_fullsized":"2024-09-21T13:49:39.836010+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:06.082699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836648+0000","last_change":"2024-09-21T13:48:58.051225+0000","last_active":"2024-09-21T13:49:39.836648+0000","last_peered":"2024-09-21T13:49:39.836648+0000","last_clean":"2024-09-21T13:49:39.836648+0000","last_became_active":"2024-09-21T13:48:58.050479+0000","last_became_peered":"2024-09-21T13:48:58.050479+0000","last_unstale":"2024-09-21T13:49:39.836648+0000","last_undegraded":"2024-09-21T13:49:39.836648+0000","last_fullsized":"2024-09-21T13:49:39.836648+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:19.515469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"68'15","reported_seq":67,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835920+0000","last_change":"2024-09-21T13:49:00.063787+0000","last_active":"2024-09-21T13:49:39.835920+0000","last_peered":"2024-09-21T13:49:39.835920+0000","last_clean":"2024-09-21T13:49:39.835920+0000","last_became_active":"2024-09-21T13:49:00.063460+0000","last_became_peered":"2024-09-21T13:49:00.063460+0000","last_unstale":"2024-09-21T13:49:39.835920+0000","last_undegraded":"2024-09-21T13:49:39.835920+0000","last_fullsized":"2024-09-21T13:49:39.835920+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:14.007409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.811464+0000","last_change":"2024-09-21T13:49:02.064007+0000","last_active":"2024-09-21T13:49:39.811464+0000","last_peered":"2024-09-21T13:49:39.811464+0000","last_clean":"2024-09-21T13:49:39.811464+0000","last_became_active":"2024-09-21T13:49:02.063687+0000","last_became_peered":"2024-09-21T13:49:02.063687+0000","last_unstale":"2024-09-21T13:49:39.811464+0000","last_undegraded":"2024-09-21T13:49:39.811464+0000","last_fullsized":"2024-09-21T13:49:39.811464+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:05.421236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.837174+0000","last_change":"2024-09-21T13:48:58.051001+0000","last_active":"2024-09-21T13:49:39.837174+0000","last_peered":"2024-09-21T13:49:39.837174+0000","last_clean":"2024-09-21T13:49:39.837174+0000","last_became_active":"2024-09-21T13:48:58.050695+0000","last_became_peered":"2024-09-21T13:48:58.050695+0000","last_unstale":"2024-09-21T13:49:39.837174+0000","last_undegraded":"2024-09-21T13:49:39.837174+0000","last_fullsized":"2024-09-21T13:49:39.837174+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:09.186887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"68'9","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.811774+0000","last_change":"2024-09-21T13:49:00.059748+0000","last_active":"2024-09-21T13:49:39.811774+0000","last_peered":"2024-09-21T13:49:39.811774+0000","last_clean":"2024-09-21T13:49:39.811774+0000","last_became_active":"2024-09-21T13:49:00.059382+0000","last_became_peered":"2024-09-21T13:49:00.059382+0000","last_unstale":"2024-09-21T13:49:39.811774+0000","last_undegraded":"2024-09-21T13:49:39.811774+0000","last_fullsized":"2024-09-21T13:49:39.811774+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:05.303397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836250+0000","last_change":"2024-09-21T13:49:02.066978+0000","last_active":"2024-09-21T13:49:39.836250+0000","last_peered":"2024-09-21T13:49:39.836250+0000","last_clean":"2024-09-21T13:49:39.836250+0000","last_became_active":"2024-09-21T13:49:02.066693+0000","last_became_peered":"2024-09-21T13:49:02.066693+0000","last_unstale":"2024-09-21T13:49:39.836250+0000","last_undegraded":"2024-09-21T13:49:39.836250+0000","last_fullsized":"2024-09-21T13:49:39.836250+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:28.710214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835306+0000","last_change":"2024-09-21T13:49:04.080709+0000","last_active":"2024-09-21T13:49:39.835306+0000","last_peered":"2024-09-21T13:49:39.835306+0000","last_clean":"2024-09-21T13:49:39.835306+0000","last_became_active":"2024-09-21T13:49:04.080359+0000","last_became_peered":"2024-09-21T13:49:04.080359+0000","last_unstale":"2024-09-21T13:49:39.835306+0000","last_undegraded":"2024-09-21T13:49:39.835306+0000","last_fullsized":"2024-09-21T13:49:39.835306+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:09.590774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836673+0000","last_change":"2024-09-21T13:48:58.052581+0000","last_active":"2024-09-21T13:49:39.836673+0000","last_peered":"2024-09-21T13:49:39.836673+0000","last_clean":"2024-09-21T13:49:39.836673+0000","last_became_active":"2024-09-21T13:48:58.050906+0000","last_became_peered":"2024-09-21T13:48:58.050906+0000","last_unstale":"2024-09-21T13:49:39.836673+0000","last_undegraded":"2024-09-21T13:49:39.836673+0000","last_fullsized":"2024-09-21T13:49:39.836673+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:19.977336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"68'5","reported_seq":52,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.811724+0000","last_change":"2024-09-21T13:49:00.060913+0000","last_active":"2024-09-21T13:49:39.811724+0000","last_peered":"2024-09-21T13:49:39.811724+0000","last_clean":"2024-09-21T13:49:39.811724+0000","last_became_active":"2024-09-21T13:49:00.060497+0000","last_became_peered":"2024-09-21T13:49:00.060497+0000","last_unstale":"2024-09-21T13:49:39.811724+0000","last_undegraded":"2024-09-21T13:49:39.811724+0000","last_fullsized":"2024-09-21T13:49:39.811724+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:10.507839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835593+0000","last_change":"2024-09-21T13:49:02.060484+0000","last_active":"2024-09-21T13:49:39.835593+0000","last_peered":"2024-09-21T13:49:39.835593+0000","last_clean":"2024-09-21T13:49:39.835593+0000","last_became_active":"2024-09-21T13:49:02.060212+0000","last_became_peered":"2024-09-21T13:49:02.060212+0000","last_unstale":"2024-09-21T13:49:39.835593+0000","last_undegraded":"2024-09-21T13:49:39.835593+0000","last_fullsized":"2024-09-21T13:49:39.835593+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:17.976156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835659+0000","last_change":"2024-09-21T13:49:04.078296+0000","last_active":"2024-09-21T13:49:39.835659+0000","last_peered":"2024-09-21T13:49:39.835659+0000","last_clean":"2024-09-21T13:49:39.835659+0000","last_became_active":"2024-09-21T13:49:04.078026+0000","last_became_peered":"2024-09-21T13:49:04.078026+0000","last_unstale":"2024-09-21T13:49:39.835659+0000","last_undegraded":"2024-09-21T13:49:39.835659+0000","last_fullsized":"2024-09-21T13:49:39.835659+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:05.284846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.812185+0000","last_change":"2024-09-21T13:49:04.432458+0000","last_active":"2024-09-21T13:49:39.812185+0000","last_peered":"2024-09-21T13:49:39.812185+0000","last_clean":"2024-09-21T13:49:39.812185+0000","last_became_active":"2024-09-21T13:49:04.431422+0000","last_became_peered":"2024-09-21T13:49:04.431422+0000","last_unstale":"2024-09-21T13:49:39.812185+0000","last_undegraded":"2024-09-21T13:49:39.812185+0000","last_fullsized":"2024-09-21T13:49:39.812185+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:08.067421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836389+0000","last_change":"2024-09-21T13:48:58.057667+0000","last_active":"2024-09-21T13:49:39.836389+0000","last_peered":"2024-09-21T13:49:39.836389+0000","last_clean":"2024-09-21T13:49:39.836389+0000","last_became_active":"2024-09-21T13:48:58.057336+0000","last_became_peered":"2024-09-21T13:48:58.057336+0000","last_unstale":"2024-09-21T13:49:39.836389+0000","last_undegraded":"2024-09-21T13:49:39.836389+0000","last_fullsized":"2024-09-21T13:49:39.836389+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:18.555629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"68'15","reported_seq":67,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.807764+0000","last_change":"2024-09-21T13:49:00.059068+0000","last_active":"2024-09-21T13:49:39.807764+0000","last_peered":"2024-09-21T13:49:39.807764+0000","last_clean":"2024-09-21T13:49:39.807764+0000","last_became_active":"2024-09-21T13:49:00.058730+0000","last_became_peered":"2024-09-21T13:49:00.058730+0000","last_unstale":"2024-09-21T13:49:39.807764+0000","last_undegraded":"2024-09-21T13:49:39.807764+0000","last_fullsized":"2024-09-21T13:49:39.807764+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:18.796269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.808006+0000","last_change":"2024-09-21T13:49:02.067253+0000","last_active":"2024-09-21T13:49:39.808006+0000","last_peered":"2024-09-21T13:49:39.808006+0000","last_clean":"2024-09-21T13:49:39.808006+0000","last_became_active":"2024-09-21T13:49:02.066796+0000","last_became_peered":"2024-09-21T13:49:02.066796+0000","last_unstale":"2024-09-21T13:49:39.808006+0000","last_undegraded":"2024-09-21T13:49:39.808006+0000","last_fullsized":"2024-09-21T13:49:39.808006+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:24.625163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836826+0000","last_change":"2024-09-21T13:49:04.434429+0000","last_active":"2024-09-21T13:49:39.836826+0000","last_peered":"2024-09-21T13:49:39.836826+0000","last_clean":"2024-09-21T13:49:39.836826+0000","last_became_active":"2024-09-21T13:49:04.434049+0000","last_became_peered":"2024-09-21T13:49:04.434049+0000","last_unstale":"2024-09-21T13:49:39.836826+0000","last_undegraded":"2024-09-21T13:49:39.836826+0000","last_fullsized":"2024-09-21T13:49:39.836826+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:25.759474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.811889+0000","last_change":"2024-09-21T13:48:58.043457+0000","last_active":"2024-09-21T13:49:39.811889+0000","last_peered":"2024-09-21T13:49:39.811889+0000","last_clean":"2024-09-21T13:49:39.811889+0000","last_became_active":"2024-09-21T13:48:58.043088+0000","last_became_peered":"2024-09-21T13:48:58.043088+0000","last_unstale":"2024-09-21T13:49:39.811889+0000","last_undegraded":"2024-09-21T13:49:39.811889+0000","last_fullsized":"2024-09-21T13:49:39.811889+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:21.622429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"68'12","reported_seq":65,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836785+0000","last_change":"2024-09-21T13:49:00.062162+0000","last_active":"2024-09-21T13:49:39.836785+0000","last_peered":"2024-09-21T13:49:39.836785+0000","last_clean":"2024-09-21T13:49:39.836785+0000","last_became_active":"2024-09-21T13:49:00.061867+0000","last_became_peered":"2024-09-21T13:49:00.061867+0000","last_unstale":"2024-09-21T13:49:39.836785+0000","last_undegraded":"2024-09-21T13:49:39.836785+0000","last_fullsized":"2024-09-21T13:49:39.836785+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:27.206067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.812068+0000","last_change":"2024-09-21T13:49:02.063090+0000","last_active":"2024-09-21T13:49:39.812068+0000","last_peered":"2024-09-21T13:49:39.812068+0000","last_clean":"2024-09-21T13:49:39.812068+0000","last_became_active":"2024-09-21T13:49:02.062802+0000","last_became_peered":"2024-09-21T13:49:02.062802+0000","last_unstale":"2024-09-21T13:49:39.812068+0000","last_undegraded":"2024-09-21T13:49:39.812068+0000","last_fullsized":"2024-09-21T13:49:39.812068+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:20.369055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":35,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836298+0000","last_change":"2024-09-21T13:49:04.076090+0000","last_active":"2024-09-21T13:49:39.836298+0000","last_peered":"2024-09-21T13:49:39.836298+0000","last_clean":"2024-09-21T13:49:39.836298+0000","last_became_active":"2024-09-21T13:49:04.075792+0000","last_became_peered":"2024-09-21T13:49:04.075792+0000","last_unstale":"2024-09-21T13:49:39.836298+0000","last_undegraded":"2024-09-21T13:49:39.836298+0000","last_fullsized":"2024-09-21T13:49:39.836298+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:07.846061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":47,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.806690+0000","last_change":"2024-09-21T13:48:58.049463+0000","last_active":"2024-09-21T13:49:39.806690+0000","last_peered":"2024-09-21T13:49:39.806690+0000","last_clean":"2024-09-21T13:49:39.806690+0000","last_became_active":"2024-09-21T13:48:58.048972+0000","last_became_peered":"2024-09-21T13:48:58.048972+0000","last_unstale":"2024-09-21T13:49:39.806690+0000","last_undegraded":"2024-09-21T13:49:39.806690+0000","last_fullsized":"2024-09-21T13:49:39.806690+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:03.427338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"68'10","reported_seq":57,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836578+0000","last_change":"2024-09-21T13:49:00.426837+0000","last_active":"2024-09-21T13:49:39.836578+0000","last_peered":"2024-09-21T13:49:39.836578+0000","last_clean":"2024-09-21T13:49:39.836578+0000","last_became_active":"2024-09-21T13:49:00.426178+0000","last_became_peered":"2024-09-21T13:49:00.426178+0000","last_unstale":"2024-09-21T13:49:39.836578+0000","last_undegraded":"2024-09-21T13:49:39.836578+0000","last_fullsized":"2024-09-21T13:49:39.836578+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:22.029884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"68'8","reported_seq":59,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836690+0000","last_change":"2024-09-21T13:49:02.059979+0000","last_active":"2024-09-21T13:49:39.836690+0000","last_peered":"2024-09-21T13:49:39.836690+0000","last_clean":"2024-09-21T13:49:39.836690+0000","last_became_active":"2024-09-21T13:49:02.059638+0000","last_became_peered":"2024-09-21T13:49:02.059638+0000","last_unstale":"2024-09-21T13:49:39.836690+0000","last_undegraded":"2024-09-21T13:49:39.836690+0000","last_fullsized":"2024-09-21T13:49:39.836690+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:26.974288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"68'15","reported_seq":67,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.808011+0000","last_change":"2024-09-21T13:49:00.060620+0000","last_active":"2024-09-21T13:49:39.808011+0000","last_peered":"2024-09-21T13:49:39.808011+0000","last_clean":"2024-09-21T13:49:39.808011+0000","last_became_active":"2024-09-21T13:49:00.060324+0000","last_became_peered":"2024-09-21T13:49:00.060324+0000","last_unstale":"2024-09-21T13:49:39.808011+0000","last_undegraded":"2024-09-21T13:49:39.808011+0000","last_fullsized":"2024-09-21T13:49:39.808011+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:20.149960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835655+0000","last_change":"2024-09-21T13:48:58.052171+0000","last_active":"2024-09-21T13:49:39.835655+0000","last_peered":"2024-09-21T13:49:39.835655+0000","last_clean":"2024-09-21T13:49:39.835655+0000","last_became_active":"2024-09-21T13:48:58.051911+0000","last_became_peered":"2024-09-21T13:48:58.051911+0000","last_unstale":"2024-09-21T13:49:39.835655+0000","last_undegraded":"2024-09-21T13:49:39.835655+0000","last_fullsized":"2024-09-21T13:49:39.835655+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:49:58.457410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"68'8","reported_seq":59,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.812357+0000","last_change":"2024-09-21T13:49:02.061966+0000","last_active":"2024-09-21T13:49:39.812357+0000","last_peered":"2024-09-21T13:49:39.812357+0000","last_clean":"2024-09-21T13:49:39.812357+0000","last_became_active":"2024-09-21T13:49:02.061679+0000","last_became_peered":"2024-09-21T13:49:02.061679+0000","last_unstale":"2024-09-21T13:49:39.812357+0000","last_undegraded":"2024-09-21T13:49:39.812357+0000","last_fullsized":"2024-09-21T13:49:39.812357+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:28.854727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836647+0000","last_change":"2024-09-21T13:49:04.082213+0000","last_active":"2024-09-21T13:49:39.836647+0000","last_peered":"2024-09-21T13:49:39.836647+0000","last_clean":"2024-09-21T13:49:39.836647+0000","last_became_active":"2024-09-21T13:49:04.081905+0000","last_became_peered":"2024-09-21T13:49:04.081905+0000","last_unstale":"2024-09-21T13:49:39.836647+0000","last_undegraded":"2024-09-21T13:49:39.836647+0000","last_fullsized":"2024-09-21T13:49:39.836647+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:17.499009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"68'18","reported_seq":74,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835396+0000","last_change":"2024-09-21T13:49:00.429686+0000","last_active":"2024-09-21T13:49:39.835396+0000","last_peered":"2024-09-21T13:49:39.835396+0000","last_clean":"2024-09-21T13:49:39.835396+0000","last_became_active":"2024-09-21T13:49:00.429314+0000","last_became_peered":"2024-09-21T13:49:00.429314+0000","last_unstale":"2024-09-21T13:49:39.835396+0000","last_undegraded":"2024-09-21T13:49:39.835396+0000","last_fullsized":"2024-09-21T13:49:39.835396+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:09.824746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.837041+0000","last_change":"2024-09-21T13:48:58.056669+0000","last_active":"2024-09-21T13:49:39.837041+0000","last_peered":"2024-09-21T13:49:39.837041+0000","last_clean":"2024-09-21T13:49:39.837041+0000","last_became_active":"2024-09-21T13:48:58.056375+0000","last_became_peered":"2024-09-21T13:48:58.056375+0000","last_unstale":"2024-09-21T13:49:39.837041+0000","last_undegraded":"2024-09-21T13:49:39.837041+0000","last_fullsized":"2024-09-21T13:49:39.837041+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:21.686532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.811469+0000","last_change":"2024-09-21T13:49:02.063237+0000","last_active":"2024-09-21T13:49:39.811469+0000","last_peered":"2024-09-21T13:49:39.811469+0000","last_clean":"2024-09-21T13:49:39.811469+0000","last_became_active":"2024-09-21T13:49:02.062955+0000","last_became_peered":"2024-09-21T13:49:02.062955+0000","last_unstale":"2024-09-21T13:49:39.811469+0000","last_undegraded":"2024-09-21T13:49:39.811469+0000","last_fullsized":"2024-09-21T13:49:39.811469+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:05.393016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.811701+0000","last_change":"2024-09-21T13:49:04.076445+0000","last_active":"2024-09-21T13:49:39.811701+0000","last_peered":"2024-09-21T13:49:39.811701+0000","last_clean":"2024-09-21T13:49:39.811701+0000","last_became_active":"2024-09-21T13:49:04.075818+0000","last_became_peered":"2024-09-21T13:49:04.075818+0000","last_unstale":"2024-09-21T13:49:39.811701+0000","last_undegraded":"2024-09-21T13:49:39.811701+0000","last_fullsized":"2024-09-21T13:49:39.811701+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:31.482751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"68'14","reported_seq":63,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.811530+0000","last_change":"2024-09-21T13:49:00.055423+0000","last_active":"2024-09-21T13:49:39.811530+0000","last_peered":"2024-09-21T13:49:39.811530+0000","last_clean":"2024-09-21T13:49:39.811530+0000","last_became_active":"2024-09-21T13:49:00.055031+0000","last_became_peered":"2024-09-21T13:49:00.055031+0000","last_unstale":"2024-09-21T13:49:39.811530+0000","last_undegraded":"2024-09-21T13:49:39.811530+0000","last_fullsized":"2024-09-21T13:49:39.811530+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:06.138147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":47,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836778+0000","last_change":"2024-09-21T13:48:58.053829+0000","last_active":"2024-09-21T13:49:39.836778+0000","last_peered":"2024-09-21T13:49:39.836778+0000","last_clean":"2024-09-21T13:49:39.836778+0000","last_became_active":"2024-09-21T13:48:58.053533+0000","last_became_peered":"2024-09-21T13:48:58.053533+0000","last_unstale":"2024-09-21T13:49:39.836778+0000","last_undegraded":"2024-09-21T13:49:39.836778+0000","last_fullsized":"2024-09-21T13:49:39.836778+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:49:58.639672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"68'8","reported_seq":56,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835314+0000","last_change":"2024-09-21T13:49:02.070405+0000","last_active":"2024-09-21T13:49:39.835314+0000","last_peered":"2024-09-21T13:49:39.835314+0000","last_clean":"2024-09-21T13:49:39.835314+0000","last_became_active":"2024-09-21T13:49:02.070064+0000","last_became_peered":"2024-09-21T13:49:02.070064+0000","last_unstale":"2024-09-21T13:49:39.835314+0000","last_undegraded":"2024-09-21T13:49:39.835314+0000","last_fullsized":"2024-09-21T13:49:39.835314+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:13.746404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836389+0000","last_change":"2024-09-21T13:49:04.434498+0000","last_active":"2024-09-21T13:49:39.836389+0000","last_peered":"2024-09-21T13:49:39.836389+0000","last_clean":"2024-09-21T13:49:39.836389+0000","last_became_active":"2024-09-21T13:49:04.433920+0000","last_became_peered":"2024-09-21T13:49:04.433920+0000","last_unstale":"2024-09-21T13:49:39.836389+0000","last_undegraded":"2024-09-21T13:49:39.836389+0000","last_fullsized":"2024-09-21T13:49:39.836389+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:22.847243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"68'10","reported_seq":57,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.807473+0000","last_change":"2024-09-21T13:49:00.054972+0000","last_active":"2024-09-21T13:49:39.807473+0000","last_peered":"2024-09-21T13:49:39.807473+0000","last_clean":"2024-09-21T13:49:39.807473+0000","last_became_active":"2024-09-21T13:49:00.054564+0000","last_became_peered":"2024-09-21T13:49:00.054564+0000","last_unstale":"2024-09-21T13:49:39.807473+0000","last_undegraded":"2024-09-21T13:49:39.807473+0000","last_fullsized":"2024-09-21T13:49:39.807473+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:18.647498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"61'1","reported_seq":52,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.837265+0000","last_change":"2024-09-21T13:48:58.050844+0000","last_active":"2024-09-21T13:49:39.837265+0000","last_peered":"2024-09-21T13:49:39.837265+0000","last_clean":"2024-09-21T13:49:39.837265+0000","last_became_active":"2024-09-21T13:48:58.050548+0000","last_became_peered":"2024-09-21T13:48:58.050548+0000","last_unstale":"2024-09-21T13:49:39.837265+0000","last_undegraded":"2024-09-21T13:49:39.837265+0000","last_fullsized":"2024-09-21T13:49:39.837265+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:17.587266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"68'8","reported_seq":56,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836022+0000","last_change":"2024-09-21T13:49:02.068926+0000","last_active":"2024-09-21T13:49:39.836022+0000","last_peered":"2024-09-21T13:49:39.836022+0000","last_clean":"2024-09-21T13:49:39.836022+0000","last_became_active":"2024-09-21T13:49:02.067664+0000","last_became_peered":"2024-09-21T13:49:02.067664+0000","last_unstale":"2024-09-21T13:49:39.836022+0000","last_undegraded":"2024-09-21T13:49:39.836022+0000","last_fullsized":"2024-09-21T13:49:39.836022+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:09.630253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835435+0000","last_change":"2024-09-21T13:49:04.081771+0000","last_active":"2024-09-21T13:49:39.835435+0000","last_peered":"2024-09-21T13:49:39.835435+0000","last_clean":"2024-09-21T13:49:39.835435+0000","last_became_active":"2024-09-21T13:49:04.081462+0000","last_became_peered":"2024-09-21T13:49:04.081462+0000","last_unstale":"2024-09-21T13:49:39.835435+0000","last_undegraded":"2024-09-21T13:49:39.835435+0000","last_fullsized":"2024-09-21T13:49:39.835435+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:25.632205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"68'19","reported_seq":78,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836173+0000","last_change":"2024-09-21T13:49:00.426917+0000","last_active":"2024-09-21T13:49:39.836173+0000","last_peered":"2024-09-21T13:49:39.836173+0000","last_clean":"2024-09-21T13:49:39.836173+0000","last_became_active":"2024-09-21T13:49:00.426579+0000","last_became_peered":"2024-09-21T13:49:00.426579+0000","last_unstale":"2024-09-21T13:49:39.836173+0000","last_undegraded":"2024-09-21T13:49:39.836173+0000","last_fullsized":"2024-09-21T13:49:39.836173+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:15.913331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.808545+0000","last_change":"2024-09-21T13:48:58.049961+0000","last_active":"2024-09-21T13:49:39.808545+0000","last_peered":"2024-09-21T13:49:39.808545+0000","last_clean":"2024-09-21T13:49:39.808545+0000","last_became_active":"2024-09-21T13:48:58.049682+0000","last_became_peered":"2024-09-21T13:48:58.049682+0000","last_unstale":"2024-09-21T13:49:39.808545+0000","last_undegraded":"2024-09-21T13:49:39.808545+0000","last_fullsized":"2024-09-21T13:49:39.808545+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:02.280644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835812+0000","last_change":"2024-09-21T13:49:02.061604+0000","last_active":"2024-09-21T13:49:39.835812+0000","last_peered":"2024-09-21T13:49:39.835812+0000","last_clean":"2024-09-21T13:49:39.835812+0000","last_became_active":"2024-09-21T13:49:02.061187+0000","last_became_peered":"2024-09-21T13:49:02.061187+0000","last_unstale":"2024-09-21T13:49:39.835812+0000","last_undegraded":"2024-09-21T13:49:39.835812+0000","last_fullsized":"2024-09-21T13:49:39.835812+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:08.846428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.806520+0000","last_change":"2024-09-21T13:49:04.431651+0000","last_active":"2024-09-21T13:49:39.806520+0000","last_peered":"2024-09-21T13:49:39.806520+0000","last_clean":"2024-09-21T13:49:39.806520+0000","last_became_active":"2024-09-21T13:49:04.431079+0000","last_became_peered":"2024-09-21T13:49:04.431079+0000","last_unstale":"2024-09-21T13:49:39.806520+0000","last_undegraded":"2024-09-21T13:49:39.806520+0000","last_fullsized":"2024-09-21T13:49:39.806520+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:05.155255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"68'28","reported_seq":100,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.808393+0000","last_change":"2024-09-21T13:49:00.060616+0000","last_active":"2024-09-21T13:49:39.808393+0000","last_peered":"2024-09-21T13:49:39.808393+0000","last_clean":"2024-09-21T13:49:39.808393+0000","last_became_active":"2024-09-21T13:49:00.060355+0000","last_became_peered":"2024-09-21T13:49:00.060355+0000","last_unstale":"2024-09-21T13:49:39.808393+0000","last_undegraded":"2024-09-21T13:49:39.808393+0000","last_fullsized":"2024-09-21T13:49:39.808393+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:26.214000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.812060+0000","last_change":"2024-09-21T13:48:58.055640+0000","last_active":"2024-09-21T13:49:39.812060+0000","last_peered":"2024-09-21T13:49:39.812060+0000","last_clean":"2024-09-21T13:49:39.812060+0000","last_became_active":"2024-09-21T13:48:58.055366+0000","last_became_peered":"2024-09-21T13:48:58.055366+0000","last_unstale":"2024-09-21T13:49:39.812060+0000","last_undegraded":"2024-09-21T13:49:39.812060+0000","last_fullsized":"2024-09-21T13:49:39.812060+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:17.442979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":56,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.837470+0000","last_change":"2024-09-21T13:49:00.053256+0000","last_active":"2024-09-21T13:49:39.837470+0000","last_peered":"2024-09-21T13:49:39.837470+0000","last_clean":"2024-09-21T13:49:39.837470+0000","last_became_active":"2024-09-21T13:48:57.046246+0000","last_became_peered":"2024-09-21T13:48:57.046246+0000","last_unstale":"2024-09-21T13:49:39.837470+0000","last_undegraded":"2024-09-21T13:49:39.837470+0000","last_fullsized":"2024-09-21T13:49:39.837470+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:48:56.029323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:04.022287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015946669999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836059+0000","last_change":"2024-09-21T13:49:02.068949+0000","last_active":"2024-09-21T13:49:39.836059+0000","last_peered":"2024-09-21T13:49:39.836059+0000","last_clean":"2024-09-21T13:49:39.836059+0000","last_became_active":"2024-09-21T13:49:02.068611+0000","last_became_peered":"2024-09-21T13:49:02.068611+0000","last_unstale":"2024-09-21T13:49:39.836059+0000","last_undegraded":"2024-09-21T13:49:39.836059+0000","last_fullsized":"2024-09-21T13:49:39.836059+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:01.307406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836335+0000","last_change":"2024-09-21T13:49:04.080755+0000","last_active":"2024-09-21T13:49:39.836335+0000","last_peered":"2024-09-21T13:49:39.836335+0000","last_clean":"2024-09-21T13:49:39.836335+0000","last_became_active":"2024-09-21T13:49:04.080466+0000","last_became_peered":"2024-09-21T13:49:04.080466+0000","last_unstale":"2024-09-21T13:49:39.836335+0000","last_undegraded":"2024-09-21T13:49:39.836335+0000","last_fullsized":"2024-09-21T13:49:39.836335+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:06.570558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"68'13","reported_seq":69,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.808039+0000","last_change":"2024-09-21T13:49:00.053645+0000","last_active":"2024-09-21T13:49:39.808039+0000","last_peered":"2024-09-21T13:49:39.808039+0000","last_clean":"2024-09-21T13:49:39.808039+0000","last_became_active":"2024-09-21T13:49:00.053323+0000","last_became_peered":"2024-09-21T13:49:00.053323+0000","last_unstale":"2024-09-21T13:49:39.808039+0000","last_undegraded":"2024-09-21T13:49:39.808039+0000","last_fullsized":"2024-09-21T13:49:39.808039+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:02.019380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.806770+0000","last_change":"2024-09-21T13:48:58.050687+0000","last_active":"2024-09-21T13:49:39.806770+0000","last_peered":"2024-09-21T13:49:39.806770+0000","last_clean":"2024-09-21T13:49:39.806770+0000","last_became_active":"2024-09-21T13:48:58.050371+0000","last_became_peered":"2024-09-21T13:48:58.050371+0000","last_unstale":"2024-09-21T13:49:39.806770+0000","last_undegraded":"2024-09-21T13:49:39.806770+0000","last_fullsized":"2024-09-21T13:49:39.806770+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:19.757852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":55,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.807250+0000","last_change":"2024-09-21T13:49:00.045590+0000","last_active":"2024-09-21T13:49:39.807250+0000","last_peered":"2024-09-21T13:49:39.807250+0000","last_clean":"2024-09-21T13:49:39.807250+0000","last_became_active":"2024-09-21T13:48:57.047067+0000","last_became_peered":"2024-09-21T13:48:57.047067+0000","last_unstale":"2024-09-21T13:49:39.807250+0000","last_undegraded":"2024-09-21T13:49:39.807250+0000","last_fullsized":"2024-09-21T13:49:39.807250+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:48:56.029323+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:49:56.701182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065666300000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.807847+0000","last_change":"2024-09-21T13:49:02.057270+0000","last_active":"2024-09-21T13:49:39.807847+0000","last_peered":"2024-09-21T13:49:39.807847+0000","last_clean":"2024-09-21T13:49:39.807847+0000","last_became_active":"2024-09-21T13:49:02.056806+0000","last_became_peered":"2024-09-21T13:49:02.056806+0000","last_unstale":"2024-09-21T13:49:39.807847+0000","last_undegraded":"2024-09-21T13:49:39.807847+0000","last_fullsized":"2024-09-21T13:49:39.807847+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:17.996686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836856+0000","last_change":"2024-09-21T13:49:04.431351+0000","last_active":"2024-09-21T13:49:39.836856+0000","last_peered":"2024-09-21T13:49:39.836856+0000","last_clean":"2024-09-21T13:49:39.836856+0000","last_became_active":"2024-09-21T13:49:04.431035+0000","last_became_peered":"2024-09-21T13:49:04.431035+0000","last_unstale":"2024-09-21T13:49:39.836856+0000","last_undegraded":"2024-09-21T13:49:39.836856+0000","last_fullsized":"2024-09-21T13:49:39.836856+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:03.519988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"68'12","reported_seq":60,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836677+0000","last_change":"2024-09-21T13:49:00.053466+0000","last_active":"2024-09-21T13:49:39.836677+0000","last_peered":"2024-09-21T13:49:39.836677+0000","last_clean":"2024-09-21T13:49:39.836677+0000","last_became_active":"2024-09-21T13:49:00.053080+0000","last_became_peered":"2024-09-21T13:49:00.053080+0000","last_unstale":"2024-09-21T13:49:39.836677+0000","last_undegraded":"2024-09-21T13:49:39.836677+0000","last_fullsized":"2024-09-21T13:49:39.836677+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:03.884409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835630+0000","last_change":"2024-09-21T13:48:58.051534+0000","last_active":"2024-09-21T13:49:39.835630+0000","last_peered":"2024-09-21T13:49:39.835630+0000","last_clean":"2024-09-21T13:49:39.835630+0000","last_became_active":"2024-09-21T13:48:58.051249+0000","last_became_peered":"2024-09-21T13:48:58.051249+0000","last_unstale":"2024-09-21T13:49:39.835630+0000","last_undegraded":"2024-09-21T13:49:39.835630+0000","last_fullsized":"2024-09-21T13:49:39.835630+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:04.856323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":97,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:41.939250+0000","last_change":"2024-09-21T13:49:00.426330+0000","last_active":"2024-09-21T13:49:41.939250+0000","last_peered":"2024-09-21T13:49:41.939250+0000","last_clean":"2024-09-21T13:49:41.939250+0000","last_became_active":"2024-09-21T13:48:57.046084+0000","last_became_peered":"2024-09-21T13:48:57.046084+0000","last_unstale":"2024-09-21T13:49:41.939250+0000","last_undegraded":"2024-09-21T13:49:41.939250+0000","last_fullsized":"2024-09-21T13:49:41.939250+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:48:56.029323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:08.036165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047048000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":40,"num_read_kb":35,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.837653+0000","last_change":"2024-09-21T13:49:02.066942+0000","last_active":"2024-09-21T13:49:39.837653+0000","last_peered":"2024-09-21T13:49:39.837653+0000","last_clean":"2024-09-21T13:49:39.837653+0000","last_became_active":"2024-09-21T13:49:02.066641+0000","last_became_peered":"2024-09-21T13:49:02.066641+0000","last_unstale":"2024-09-21T13:49:39.837653+0000","last_undegraded":"2024-09-21T13:49:39.837653+0000","last_fullsized":"2024-09-21T13:49:39.837653+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:22.324406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.808219+0000","last_change":"2024-09-21T13:49:04.078686+0000","last_active":"2024-09-21T13:49:39.808219+0000","last_peered":"2024-09-21T13:49:39.808219+0000","last_clean":"2024-09-21T13:49:39.808219+0000","last_became_active":"2024-09-21T13:49:04.078372+0000","last_became_peered":"2024-09-21T13:49:04.078372+0000","last_unstale":"2024-09-21T13:49:39.808219+0000","last_undegraded":"2024-09-21T13:49:39.808219+0000","last_fullsized":"2024-09-21T13:49:39.808219+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:25.621989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"68'16","reported_seq":74,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836782+0000","last_change":"2024-09-21T13:49:00.427430+0000","last_active":"2024-09-21T13:49:39.836782+0000","last_peered":"2024-09-21T13:49:39.836782+0000","last_clean":"2024-09-21T13:49:39.836782+0000","last_became_active":"2024-09-21T13:49:00.426867+0000","last_became_peered":"2024-09-21T13:49:00.426867+0000","last_unstale":"2024-09-21T13:49:39.836782+0000","last_undegraded":"2024-09-21T13:49:39.836782+0000","last_fullsized":"2024-09-21T13:49:39.836782+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:11.516725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835899+0000","last_change":"2024-09-21T13:48:58.052925+0000","last_active":"2024-09-21T13:49:39.835899+0000","last_peered":"2024-09-21T13:49:39.835899+0000","last_clean":"2024-09-21T13:49:39.835899+0000","last_became_active":"2024-09-21T13:48:58.052659+0000","last_became_peered":"2024-09-21T13:48:58.052659+0000","last_unstale":"2024-09-21T13:49:39.835899+0000","last_undegraded":"2024-09-21T13:49:39.835899+0000","last_fullsized":"2024-09-21T13:49:39.835899+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:03.529879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"75'79","reported_seq":95,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836777+0000","last_change":"2024-09-21T13:48:51.893262+0000","last_active":"2024-09-21T13:49:39.836777+0000","last_peered":"2024-09-21T13:49:39.836777+0000","last_clean":"2024-09-21T13:49:39.836777+0000","last_became_active":"2024-09-21T13:48:48.454646+0000","last_became_peered":"2024-09-21T13:48:48.454646+0000","last_unstale":"2024-09-21T13:49:39.836777+0000","last_undegraded":"2024-09-21T13:49:39.836777+0000","last_fullsized":"2024-09-21T13:49:39.836777+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'72","last_scrub_stamp":"2024-09-21T13:48:51.893178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:48:51.893178+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:10.164272+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836466+0000","last_change":"2024-09-21T13:49:02.061930+0000","last_active":"2024-09-21T13:49:39.836466+0000","last_peered":"2024-09-21T13:49:39.836466+0000","last_clean":"2024-09-21T13:49:39.836466+0000","last_became_active":"2024-09-21T13:49:02.061454+0000","last_became_peered":"2024-09-21T13:49:02.061454+0000","last_unstale":"2024-09-21T13:49:39.836466+0000","last_undegraded":"2024-09-21T13:49:39.836466+0000","last_fullsized":"2024-09-21T13:49:39.836466+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:23.000364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.837469+0000","last_change":"2024-09-21T13:49:04.078343+0000","last_active":"2024-09-21T13:49:39.837469+0000","last_peered":"2024-09-21T13:49:39.837469+0000","last_clean":"2024-09-21T13:49:39.837469+0000","last_became_active":"2024-09-21T13:49:04.078091+0000","last_became_peered":"2024-09-21T13:49:04.078091+0000","last_unstale":"2024-09-21T13:49:39.837469+0000","last_undegraded":"2024-09-21T13:49:39.837469+0000","last_fullsized":"2024-09-21T13:49:39.837469+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:17.101054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"68'11","reported_seq":61,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.812470+0000","last_change":"2024-09-21T13:49:00.054827+0000","last_active":"2024-09-21T13:49:39.812470+0000","last_peered":"2024-09-21T13:49:39.812470+0000","last_clean":"2024-09-21T13:49:39.812470+0000","last_became_active":"2024-09-21T13:49:00.053268+0000","last_became_peered":"2024-09-21T13:49:00.053268+0000","last_unstale":"2024-09-21T13:49:39.812470+0000","last_undegraded":"2024-09-21T13:49:39.812470+0000","last_fullsized":"2024-09-21T13:49:39.812470+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:28.403874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.811957+0000","last_change":"2024-09-21T13:48:58.052364+0000","last_active":"2024-09-21T13:49:39.811957+0000","last_peered":"2024-09-21T13:49:39.811957+0000","last_clean":"2024-09-21T13:49:39.811957+0000","last_became_active":"2024-09-21T13:48:58.052092+0000","last_became_peered":"2024-09-21T13:48:58.052092+0000","last_unstale":"2024-09-21T13:49:39.811957+0000","last_undegraded":"2024-09-21T13:49:39.811957+0000","last_fullsized":"2024-09-21T13:49:39.811957+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:10.772378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.837459+0000","last_change":"2024-09-21T13:49:02.062262+0000","last_active":"2024-09-21T13:49:39.837459+0000","last_peered":"2024-09-21T13:49:39.837459+0000","last_clean":"2024-09-21T13:49:39.837459+0000","last_became_active":"2024-09-21T13:49:02.061901+0000","last_became_peered":"2024-09-21T13:49:02.061901+0000","last_unstale":"2024-09-21T13:49:39.837459+0000","last_undegraded":"2024-09-21T13:49:39.837459+0000","last_fullsized":"2024-09-21T13:49:39.837459+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:20.147536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836230+0000","last_change":"2024-09-21T13:49:04.431695+0000","last_active":"2024-09-21T13:49:39.836230+0000","last_peered":"2024-09-21T13:49:39.836230+0000","last_clean":"2024-09-21T13:49:39.836230+0000","last_became_active":"2024-09-21T13:49:04.431268+0000","last_became_peered":"2024-09-21T13:49:04.431268+0000","last_unstale":"2024-09-21T13:49:39.836230+0000","last_undegraded":"2024-09-21T13:49:39.836230+0000","last_fullsized":"2024-09-21T13:49:39.836230+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:27.118735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"68'17","reported_seq":70,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.812385+0000","last_change":"2024-09-21T13:49:00.053077+0000","last_active":"2024-09-21T13:49:39.812385+0000","last_peered":"2024-09-21T13:49:39.812385+0000","last_clean":"2024-09-21T13:49:39.812385+0000","last_became_active":"2024-09-21T13:49:00.052608+0000","last_became_peered":"2024-09-21T13:49:00.052608+0000","last_unstale":"2024-09-21T13:49:39.812385+0000","last_undegraded":"2024-09-21T13:49:39.812385+0000","last_fullsized":"2024-09-21T13:49:39.812385+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:14.266475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836235+0000","last_change":"2024-09-21T13:48:58.051751+0000","last_active":"2024-09-21T13:49:39.836235+0000","last_peered":"2024-09-21T13:49:39.836235+0000","last_clean":"2024-09-21T13:49:39.836235+0000","last_became_active":"2024-09-21T13:48:58.049142+0000","last_became_peered":"2024-09-21T13:48:58.049142+0000","last_unstale":"2024-09-21T13:49:39.836235+0000","last_undegraded":"2024-09-21T13:49:39.836235+0000","last_fullsized":"2024-09-21T13:49:39.836235+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:07.453552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.808232+0000","last_change":"2024-09-21T13:49:02.067811+0000","last_active":"2024-09-21T13:49:39.808232+0000","last_peered":"2024-09-21T13:49:39.808232+0000","last_clean":"2024-09-21T13:49:39.808232+0000","last_became_active":"2024-09-21T13:49:02.067472+0000","last_became_peered":"2024-09-21T13:49:02.067472+0000","last_unstale":"2024-09-21T13:49:39.808232+0000","last_undegraded":"2024-09-21T13:49:39.808232+0000","last_fullsized":"2024-09-21T13:49:39.808232+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:29.225485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.808029+0000","last_change":"2024-09-21T13:49:04.078285+0000","last_active":"2024-09-21T13:49:39.808029+0000","last_peered":"2024-09-21T13:49:39.808029+0000","last_clean":"2024-09-21T13:49:39.808029+0000","last_became_active":"2024-09-21T13:49:04.077962+0000","last_became_peered":"2024-09-21T13:49:04.077962+0000","last_unstale":"2024-09-21T13:49:39.808029+0000","last_undegraded":"2024-09-21T13:49:39.808029+0000","last_fullsized":"2024-09-21T13:49:39.808029+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:13.659310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"68'10","reported_seq":57,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.812106+0000","last_change":"2024-09-21T13:49:00.060182+0000","last_active":"2024-09-21T13:49:39.812106+0000","last_peered":"2024-09-21T13:49:39.812106+0000","last_clean":"2024-09-21T13:49:39.812106+0000","last_became_active":"2024-09-21T13:49:00.059850+0000","last_became_peered":"2024-09-21T13:49:00.059850+0000","last_unstale":"2024-09-21T13:49:39.812106+0000","last_undegraded":"2024-09-21T13:49:39.812106+0000","last_fullsized":"2024-09-21T13:49:39.812106+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:16.066823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836259+0000","last_change":"2024-09-21T13:48:58.058603+0000","last_active":"2024-09-21T13:49:39.836259+0000","last_peered":"2024-09-21T13:49:39.836259+0000","last_clean":"2024-09-21T13:49:39.836259+0000","last_became_active":"2024-09-21T13:48:58.058309+0000","last_became_peered":"2024-09-21T13:48:58.058309+0000","last_unstale":"2024-09-21T13:49:39.836259+0000","last_undegraded":"2024-09-21T13:49:39.836259+0000","last_fullsized":"2024-09-21T13:49:39.836259+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:09.079397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"68'8","reported_seq":56,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.807442+0000","last_change":"2024-09-21T13:49:02.065220+0000","last_active":"2024-09-21T13:49:39.807442+0000","last_peered":"2024-09-21T13:49:39.807442+0000","last_clean":"2024-09-21T13:49:39.807442+0000","last_became_active":"2024-09-21T13:49:02.064817+0000","last_became_peered":"2024-09-21T13:49:02.064817+0000","last_unstale":"2024-09-21T13:49:39.807442+0000","last_undegraded":"2024-09-21T13:49:39.807442+0000","last_fullsized":"2024-09-21T13:49:39.807442+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:11.912943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"68'1","reported_seq":35,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.812179+0000","last_change":"2024-09-21T13:49:04.072175+0000","last_active":"2024-09-21T13:49:39.812179+0000","last_peered":"2024-09-21T13:49:39.812179+0000","last_clean":"2024-09-21T13:49:39.812179+0000","last_became_active":"2024-09-21T13:49:04.071885+0000","last_became_peered":"2024-09-21T13:49:04.071885+0000","last_unstale":"2024-09-21T13:49:39.812179+0000","last_undegraded":"2024-09-21T13:49:39.812179+0000","last_fullsized":"2024-09-21T13:49:39.812179+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:03.670445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"68'9","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836079+0000","last_change":"2024-09-21T13:49:00.053349+0000","last_active":"2024-09-21T13:49:39.836079+0000","last_peered":"2024-09-21T13:49:39.836079+0000","last_clean":"2024-09-21T13:49:39.836079+0000","last_became_active":"2024-09-21T13:49:00.053016+0000","last_became_peered":"2024-09-21T13:49:00.053016+0000","last_unstale":"2024-09-21T13:49:39.836079+0000","last_undegraded":"2024-09-21T13:49:39.836079+0000","last_fullsized":"2024-09-21T13:49:39.836079+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:06.374913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836830+0000","last_change":"2024-09-21T13:48:58.050243+0000","last_active":"2024-09-21T13:49:39.836830+0000","last_peered":"2024-09-21T13:49:39.836830+0000","last_clean":"2024-09-21T13:49:39.836830+0000","last_became_active":"2024-09-21T13:48:58.049543+0000","last_became_peered":"2024-09-21T13:48:58.049543+0000","last_unstale":"2024-09-21T13:49:39.836830+0000","last_undegraded":"2024-09-21T13:49:39.836830+0000","last_fullsized":"2024-09-21T13:49:39.836830+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:17.729176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.807325+0000","last_change":"2024-09-21T13:49:02.063877+0000","last_active":"2024-09-21T13:49:39.807325+0000","last_peered":"2024-09-21T13:49:39.807325+0000","last_clean":"2024-09-21T13:49:39.807325+0000","last_became_active":"2024-09-21T13:49:02.063525+0000","last_became_peered":"2024-09-21T13:49:02.063525+0000","last_unstale":"2024-09-21T13:49:39.807325+0000","last_undegraded":"2024-09-21T13:49:39.807325+0000","last_fullsized":"2024-09-21T13:49:39.807325+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:30.535568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835588+0000","last_change":"2024-09-21T13:49:04.081924+0000","last_active":"2024-09-21T13:49:39.835588+0000","last_peered":"2024-09-21T13:49:39.835588+0000","last_clean":"2024-09-21T13:49:39.835588+0000","last_became_active":"2024-09-21T13:49:04.081617+0000","last_became_peered":"2024-09-21T13:49:04.081617+0000","last_unstale":"2024-09-21T13:49:39.835588+0000","last_undegraded":"2024-09-21T13:49:39.835588+0000","last_fullsized":"2024-09-21T13:49:39.835588+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:25.513722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"68'19","reported_seq":73,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835941+0000","last_change":"2024-09-21T13:49:00.426272+0000","last_active":"2024-09-21T13:49:39.835941+0000","last_peered":"2024-09-21T13:49:39.835941+0000","last_clean":"2024-09-21T13:49:39.835941+0000","last_became_active":"2024-09-21T13:49:00.425955+0000","last_became_peered":"2024-09-21T13:49:00.425955+0000","last_unstale":"2024-09-21T13:49:39.835941+0000","last_undegraded":"2024-09-21T13:49:39.835941+0000","last_fullsized":"2024-09-21T13:49:39.835941+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:02.788196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.837228+0000","last_change":"2024-09-21T13:48:58.053940+0000","last_active":"2024-09-21T13:49:39.837228+0000","last_peered":"2024-09-21T13:49:39.837228+0000","last_clean":"2024-09-21T13:49:39.837228+0000","last_became_active":"2024-09-21T13:48:58.053641+0000","last_became_peered":"2024-09-21T13:48:58.053641+0000","last_unstale":"2024-09-21T13:49:39.837228+0000","last_undegraded":"2024-09-21T13:49:39.837228+0000","last_fullsized":"2024-09-21T13:49:39.837228+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:25.094666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.807300+0000","last_change":"2024-09-21T13:49:02.066996+0000","last_active":"2024-09-21T13:49:39.807300+0000","last_peered":"2024-09-21T13:49:39.807300+0000","last_clean":"2024-09-21T13:49:39.807300+0000","last_became_active":"2024-09-21T13:49:02.066676+0000","last_became_peered":"2024-09-21T13:49:02.066676+0000","last_unstale":"2024-09-21T13:49:39.807300+0000","last_undegraded":"2024-09-21T13:49:39.807300+0000","last_fullsized":"2024-09-21T13:49:39.807300+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:19.241359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836031+0000","last_change":"2024-09-21T13:49:04.078412+0000","last_active":"2024-09-21T13:49:39.836031+0000","last_peered":"2024-09-21T13:49:39.836031+0000","last_clean":"2024-09-21T13:49:39.836031+0000","last_became_active":"2024-09-21T13:49:04.078084+0000","last_became_peered":"2024-09-21T13:49:04.078084+0000","last_unstale":"2024-09-21T13:49:39.836031+0000","last_undegraded":"2024-09-21T13:49:39.836031+0000","last_fullsized":"2024-09-21T13:49:39.836031+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:11.517342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"68'12","reported_seq":65,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.811928+0000","last_change":"2024-09-21T13:49:00.060188+0000","last_active":"2024-09-21T13:49:39.811928+0000","last_peered":"2024-09-21T13:49:39.811928+0000","last_clean":"2024-09-21T13:49:39.811928+0000","last_became_active":"2024-09-21T13:49:00.059720+0000","last_became_peered":"2024-09-21T13:49:00.059720+0000","last_unstale":"2024-09-21T13:49:39.811928+0000","last_undegraded":"2024-09-21T13:49:39.811928+0000","last_fullsized":"2024-09-21T13:49:39.811928+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:17.978182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.837628+0000","last_change":"2024-09-21T13:48:58.058654+0000","last_active":"2024-09-21T13:49:39.837628+0000","last_peered":"2024-09-21T13:49:39.837628+0000","last_clean":"2024-09-21T13:49:39.837628+0000","last_became_active":"2024-09-21T13:48:58.057641+0000","last_became_peered":"2024-09-21T13:48:58.057641+0000","last_unstale":"2024-09-21T13:49:39.837628+0000","last_undegraded":"2024-09-21T13:49:39.837628+0000","last_fullsized":"2024-09-21T13:49:39.837628+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:11.120209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.807118+0000","last_change":"2024-09-21T13:49:02.057568+0000","last_active":"2024-09-21T13:49:39.807118+0000","last_peered":"2024-09-21T13:49:39.807118+0000","last_clean":"2024-09-21T13:49:39.807118+0000","last_became_active":"2024-09-21T13:49:02.057195+0000","last_became_peered":"2024-09-21T13:49:02.057195+0000","last_unstale":"2024-09-21T13:49:39.807118+0000","last_undegraded":"2024-09-21T13:49:39.807118+0000","last_fullsized":"2024-09-21T13:49:39.807118+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:01.498483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836115+0000","last_change":"2024-09-21T13:49:04.078422+0000","last_active":"2024-09-21T13:49:39.836115+0000","last_peered":"2024-09-21T13:49:39.836115+0000","last_clean":"2024-09-21T13:49:39.836115+0000","last_became_active":"2024-09-21T13:49:04.078089+0000","last_became_peered":"2024-09-21T13:49:04.078089+0000","last_unstale":"2024-09-21T13:49:39.836115+0000","last_undegraded":"2024-09-21T13:49:39.836115+0000","last_fullsized":"2024-09-21T13:49:39.836115+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:18.974401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"68'15","reported_seq":67,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835435+0000","last_change":"2024-09-21T13:49:00.426358+0000","last_active":"2024-09-21T13:49:39.835435+0000","last_peered":"2024-09-21T13:49:39.835435+0000","last_clean":"2024-09-21T13:49:39.835435+0000","last_became_active":"2024-09-21T13:49:00.425997+0000","last_became_peered":"2024-09-21T13:49:00.425997+0000","last_unstale":"2024-09-21T13:49:39.835435+0000","last_undegraded":"2024-09-21T13:49:39.835435+0000","last_fullsized":"2024-09-21T13:49:39.835435+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:03.382004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.837692+0000","last_change":"2024-09-21T13:48:58.058778+0000","last_active":"2024-09-21T13:49:39.837692+0000","last_peered":"2024-09-21T13:49:39.837692+0000","last_clean":"2024-09-21T13:49:39.837692+0000","last_became_active":"2024-09-21T13:48:58.057632+0000","last_became_peered":"2024-09-21T13:48:58.057632+0000","last_unstale":"2024-09-21T13:49:39.837692+0000","last_undegraded":"2024-09-21T13:49:39.837692+0000","last_fullsized":"2024-09-21T13:49:39.837692+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:06.995889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"68'8","reported_seq":56,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835763+0000","last_change":"2024-09-21T13:49:02.056471+0000","last_active":"2024-09-21T13:49:39.835763+0000","last_peered":"2024-09-21T13:49:39.835763+0000","last_clean":"2024-09-21T13:49:39.835763+0000","last_became_active":"2024-09-21T13:49:02.056167+0000","last_became_peered":"2024-09-21T13:49:02.056167+0000","last_unstale":"2024-09-21T13:49:39.835763+0000","last_undegraded":"2024-09-21T13:49:39.835763+0000","last_fullsized":"2024-09-21T13:49:39.835763+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:10.093140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835662+0000","last_change":"2024-09-21T13:49:04.432809+0000","last_active":"2024-09-21T13:49:39.835662+0000","last_peered":"2024-09-21T13:49:39.835662+0000","last_clean":"2024-09-21T13:49:39.835662+0000","last_became_active":"2024-09-21T13:49:04.432546+0000","last_became_peered":"2024-09-21T13:49:04.432546+0000","last_unstale":"2024-09-21T13:49:39.835662+0000","last_undegraded":"2024-09-21T13:49:39.835662+0000","last_fullsized":"2024-09-21T13:49:39.835662+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:08.569714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836980+0000","last_change":"2024-09-21T13:48:58.056383+0000","last_active":"2024-09-21T13:49:39.836980+0000","last_peered":"2024-09-21T13:49:39.836980+0000","last_clean":"2024-09-21T13:49:39.836980+0000","last_became_active":"2024-09-21T13:48:58.056080+0000","last_became_peered":"2024-09-21T13:48:58.056080+0000","last_unstale":"2024-09-21T13:49:39.836980+0000","last_undegraded":"2024-09-21T13:49:39.836980+0000","last_fullsized":"2024-09-21T13:49:39.836980+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:26.306822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"68'6","reported_seq":51,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836952+0000","last_change":"2024-09-21T13:49:00.427204+0000","last_active":"2024-09-21T13:49:39.836952+0000","last_peered":"2024-09-21T13:49:39.836952+0000","last_clean":"2024-09-21T13:49:39.836952+0000","last_became_active":"2024-09-21T13:49:00.426829+0000","last_became_peered":"2024-09-21T13:49:00.426829+0000","last_unstale":"2024-09-21T13:49:39.836952+0000","last_undegraded":"2024-09-21T13:49:39.836952+0000","last_fullsized":"2024-09-21T13:49:39.836952+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:20.014163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.837140+0000","last_change":"2024-09-21T13:49:02.062503+0000","last_active":"2024-09-21T13:49:39.837140+0000","last_peered":"2024-09-21T13:49:39.837140+0000","last_clean":"2024-09-21T13:49:39.837140+0000","last_became_active":"2024-09-21T13:49:02.062231+0000","last_became_peered":"2024-09-21T13:49:02.062231+0000","last_unstale":"2024-09-21T13:49:39.837140+0000","last_undegraded":"2024-09-21T13:49:39.837140+0000","last_fullsized":"2024-09-21T13:49:39.837140+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:21.288932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836220+0000","last_change":"2024-09-21T13:49:04.432777+0000","last_active":"2024-09-21T13:49:39.836220+0000","last_peered":"2024-09-21T13:49:39.836220+0000","last_clean":"2024-09-21T13:49:39.836220+0000","last_became_active":"2024-09-21T13:49:04.432507+0000","last_became_peered":"2024-09-21T13:49:04.432507+0000","last_unstale":"2024-09-21T13:49:39.836220+0000","last_undegraded":"2024-09-21T13:49:39.836220+0000","last_fullsized":"2024-09-21T13:49:39.836220+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:03.727186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"68'9","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836720+0000","last_change":"2024-09-21T13:49:00.428704+0000","last_active":"2024-09-21T13:49:39.836720+0000","last_peered":"2024-09-21T13:49:39.836720+0000","last_clean":"2024-09-21T13:49:39.836720+0000","last_became_active":"2024-09-21T13:49:00.428330+0000","last_became_peered":"2024-09-21T13:49:00.428330+0000","last_unstale":"2024-09-21T13:49:39.836720+0000","last_undegraded":"2024-09-21T13:49:39.836720+0000","last_fullsized":"2024-09-21T13:49:39.836720+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:06.325729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"61'1","reported_seq":52,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835958+0000","last_change":"2024-09-21T13:48:58.050948+0000","last_active":"2024-09-21T13:49:39.835958+0000","last_peered":"2024-09-21T13:49:39.835958+0000","last_clean":"2024-09-21T13:49:39.835958+0000","last_became_active":"2024-09-21T13:48:58.050645+0000","last_became_peered":"2024-09-21T13:48:58.050645+0000","last_unstale":"2024-09-21T13:49:39.835958+0000","last_undegraded":"2024-09-21T13:49:39.835958+0000","last_fullsized":"2024-09-21T13:49:39.835958+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:24.159998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836864+0000","last_change":"2024-09-21T13:49:02.062075+0000","last_active":"2024-09-21T13:49:39.836864+0000","last_peered":"2024-09-21T13:49:39.836864+0000","last_clean":"2024-09-21T13:49:39.836864+0000","last_became_active":"2024-09-21T13:49:02.061775+0000","last_became_peered":"2024-09-21T13:49:02.061775+0000","last_unstale":"2024-09-21T13:49:39.836864+0000","last_undegraded":"2024-09-21T13:49:39.836864+0000","last_fullsized":"2024-09-21T13:49:39.836864+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:16.428002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.807270+0000","last_change":"2024-09-21T13:49:04.069452+0000","last_active":"2024-09-21T13:49:39.807270+0000","last_peered":"2024-09-21T13:49:39.807270+0000","last_clean":"2024-09-21T13:49:39.807270+0000","last_became_active":"2024-09-21T13:49:04.069142+0000","last_became_peered":"2024-09-21T13:49:04.069142+0000","last_unstale":"2024-09-21T13:49:39.807270+0000","last_undegraded":"2024-09-21T13:49:39.807270+0000","last_fullsized":"2024-09-21T13:49:39.807270+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:28.894052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"68'9","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.837072+0000","last_change":"2024-09-21T13:49:00.428080+0000","last_active":"2024-09-21T13:49:39.837072+0000","last_peered":"2024-09-21T13:49:39.837072+0000","last_clean":"2024-09-21T13:49:39.837072+0000","last_became_active":"2024-09-21T13:49:00.427766+0000","last_became_peered":"2024-09-21T13:49:00.427766+0000","last_unstale":"2024-09-21T13:49:39.837072+0000","last_undegraded":"2024-09-21T13:49:39.837072+0000","last_fullsized":"2024-09-21T13:49:39.837072+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:16.384643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836693+0000","last_change":"2024-09-21T13:48:58.052465+0000","last_active":"2024-09-21T13:49:39.836693+0000","last_peered":"2024-09-21T13:49:39.836693+0000","last_clean":"2024-09-21T13:49:39.836693+0000","last_became_active":"2024-09-21T13:48:58.052151+0000","last_became_peered":"2024-09-21T13:48:58.052151+0000","last_unstale":"2024-09-21T13:49:39.836693+0000","last_undegraded":"2024-09-21T13:49:39.836693+0000","last_fullsized":"2024-09-21T13:49:39.836693+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:09.226461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"68'8","reported_seq":56,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836463+0000","last_change":"2024-09-21T13:49:02.066442+0000","last_active":"2024-09-21T13:49:39.836463+0000","last_peered":"2024-09-21T13:49:39.836463+0000","last_clean":"2024-09-21T13:49:39.836463+0000","last_became_active":"2024-09-21T13:49:02.065602+0000","last_became_peered":"2024-09-21T13:49:02.065602+0000","last_unstale":"2024-09-21T13:49:39.836463+0000","last_undegraded":"2024-09-21T13:49:39.836463+0000","last_fullsized":"2024-09-21T13:49:39.836463+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:10.855295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.812396+0000","last_change":"2024-09-21T13:49:04.073461+0000","last_active":"2024-09-21T13:49:39.812396+0000","last_peered":"2024-09-21T13:49:39.812396+0000","last_clean":"2024-09-21T13:49:39.812396+0000","last_became_active":"2024-09-21T13:49:04.073188+0000","last_became_peered":"2024-09-21T13:49:04.073188+0000","last_unstale":"2024-09-21T13:49:39.812396+0000","last_undegraded":"2024-09-21T13:49:39.812396+0000","last_fullsized":"2024-09-21T13:49:39.812396+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:08.749766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"68'10","reported_seq":57,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836534+0000","last_change":"2024-09-21T13:49:00.427262+0000","last_active":"2024-09-21T13:49:39.836534+0000","last_peered":"2024-09-21T13:49:39.836534+0000","last_clean":"2024-09-21T13:49:39.836534+0000","last_became_active":"2024-09-21T13:49:00.426071+0000","last_became_peered":"2024-09-21T13:49:00.426071+0000","last_unstale":"2024-09-21T13:49:39.836534+0000","last_undegraded":"2024-09-21T13:49:39.836534+0000","last_fullsized":"2024-09-21T13:49:39.836534+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:03.592724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836695+0000","last_change":"2024-09-21T13:48:58.058198+0000","last_active":"2024-09-21T13:49:39.836695+0000","last_peered":"2024-09-21T13:49:39.836695+0000","last_clean":"2024-09-21T13:49:39.836695+0000","last_became_active":"2024-09-21T13:48:58.057587+0000","last_became_peered":"2024-09-21T13:48:58.057587+0000","last_unstale":"2024-09-21T13:49:39.836695+0000","last_undegraded":"2024-09-21T13:49:39.836695+0000","last_fullsized":"2024-09-21T13:49:39.836695+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:19.518354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"68'8","reported_seq":56,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836883+0000","last_change":"2024-09-21T13:49:02.067827+0000","last_active":"2024-09-21T13:49:39.836883+0000","last_peered":"2024-09-21T13:49:39.836883+0000","last_clean":"2024-09-21T13:49:39.836883+0000","last_became_active":"2024-09-21T13:49:02.067487+0000","last_became_peered":"2024-09-21T13:49:02.067487+0000","last_unstale":"2024-09-21T13:49:39.836883+0000","last_undegraded":"2024-09-21T13:49:39.836883+0000","last_fullsized":"2024-09-21T13:49:39.836883+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:23.003843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836447+0000","last_change":"2024-09-21T13:49:04.081928+0000","last_active":"2024-09-21T13:49:39.836447+0000","last_peered":"2024-09-21T13:49:39.836447+0000","last_clean":"2024-09-21T13:49:39.836447+0000","last_became_active":"2024-09-21T13:49:04.081609+0000","last_became_peered":"2024-09-21T13:49:04.081609+0000","last_unstale":"2024-09-21T13:49:39.836447+0000","last_undegraded":"2024-09-21T13:49:39.836447+0000","last_fullsized":"2024-09-21T13:49:39.836447+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:31.380912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"68'11","reported_seq":61,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.811729+0000","last_change":"2024-09-21T13:49:00.055420+0000","last_active":"2024-09-21T13:49:39.811729+0000","last_peered":"2024-09-21T13:49:39.811729+0000","last_clean":"2024-09-21T13:49:39.811729+0000","last_became_active":"2024-09-21T13:49:00.055024+0000","last_became_peered":"2024-09-21T13:49:00.055024+0000","last_unstale":"2024-09-21T13:49:39.811729+0000","last_undegraded":"2024-09-21T13:49:39.811729+0000","last_fullsized":"2024-09-21T13:49:39.811729+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:02.449612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.812476+0000","last_change":"2024-09-21T13:48:58.051845+0000","last_active":"2024-09-21T13:49:39.812476+0000","last_peered":"2024-09-21T13:49:39.812476+0000","last_clean":"2024-09-21T13:49:39.812476+0000","last_became_active":"2024-09-21T13:48:58.051541+0000","last_became_peered":"2024-09-21T13:48:58.051541+0000","last_unstale":"2024-09-21T13:49:39.812476+0000","last_undegraded":"2024-09-21T13:49:39.812476+0000","last_fullsized":"2024-09-21T13:49:39.812476+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:03.712482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.806822+0000","last_change":"2024-09-21T13:49:02.063328+0000","last_active":"2024-09-21T13:49:39.806822+0000","last_peered":"2024-09-21T13:49:39.806822+0000","last_clean":"2024-09-21T13:49:39.806822+0000","last_became_active":"2024-09-21T13:49:02.062868+0000","last_became_peered":"2024-09-21T13:49:02.062868+0000","last_unstale":"2024-09-21T13:49:39.806822+0000","last_undegraded":"2024-09-21T13:49:39.806822+0000","last_fullsized":"2024-09-21T13:49:39.806822+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:12.568168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835473+0000","last_change":"2024-09-21T13:49:04.083254+0000","last_active":"2024-09-21T13:49:39.835473+0000","last_peered":"2024-09-21T13:49:39.835473+0000","last_clean":"2024-09-21T13:49:39.835473+0000","last_became_active":"2024-09-21T13:49:04.082961+0000","last_became_peered":"2024-09-21T13:49:04.082961+0000","last_unstale":"2024-09-21T13:49:39.835473+0000","last_undegraded":"2024-09-21T13:49:39.835473+0000","last_fullsized":"2024-09-21T13:49:39.835473+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:07.548695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"68'9","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.807587+0000","last_change":"2024-09-21T13:49:00.057817+0000","last_active":"2024-09-21T13:49:39.807587+0000","last_peered":"2024-09-21T13:49:39.807587+0000","last_clean":"2024-09-21T13:49:39.807587+0000","last_became_active":"2024-09-21T13:49:00.057392+0000","last_became_peered":"2024-09-21T13:49:00.057392+0000","last_unstale":"2024-09-21T13:49:39.807587+0000","last_undegraded":"2024-09-21T13:49:39.807587+0000","last_fullsized":"2024-09-21T13:49:39.807587+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:02.682820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.837162+0000","last_change":"2024-09-21T13:48:58.058032+0000","last_active":"2024-09-21T13:49:39.837162+0000","last_peered":"2024-09-21T13:49:39.837162+0000","last_clean":"2024-09-21T13:49:39.837162+0000","last_became_active":"2024-09-21T13:48:58.057516+0000","last_became_peered":"2024-09-21T13:48:58.057516+0000","last_unstale":"2024-09-21T13:49:39.837162+0000","last_undegraded":"2024-09-21T13:49:39.837162+0000","last_fullsized":"2024-09-21T13:49:39.837162+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:20.885486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836103+0000","last_change":"2024-09-21T13:49:02.072264+0000","last_active":"2024-09-21T13:49:39.836103+0000","last_peered":"2024-09-21T13:49:39.836103+0000","last_clean":"2024-09-21T13:49:39.836103+0000","last_became_active":"2024-09-21T13:49:02.071922+0000","last_became_peered":"2024-09-21T13:49:02.071922+0000","last_unstale":"2024-09-21T13:49:39.836103+0000","last_undegraded":"2024-09-21T13:49:39.836103+0000","last_fullsized":"2024-09-21T13:49:39.836103+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:12.773964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835233+0000","last_change":"2024-09-21T13:49:04.080300+0000","last_active":"2024-09-21T13:49:39.835233+0000","last_peered":"2024-09-21T13:49:39.835233+0000","last_clean":"2024-09-21T13:49:39.835233+0000","last_became_active":"2024-09-21T13:49:04.079950+0000","last_became_peered":"2024-09-21T13:49:04.079950+0000","last_unstale":"2024-09-21T13:49:39.835233+0000","last_undegraded":"2024-09-21T13:49:39.835233+0000","last_fullsized":"2024-09-21T13:49:39.835233+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:20.474523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"68'11","reported_seq":61,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835652+0000","last_change":"2024-09-21T13:49:00.426487+0000","last_active":"2024-09-21T13:49:39.835652+0000","last_peered":"2024-09-21T13:49:39.835652+0000","last_clean":"2024-09-21T13:49:39.835652+0000","last_became_active":"2024-09-21T13:49:00.426091+0000","last_became_peered":"2024-09-21T13:49:00.426091+0000","last_unstale":"2024-09-21T13:49:39.835652+0000","last_undegraded":"2024-09-21T13:49:39.835652+0000","last_fullsized":"2024-09-21T13:49:39.835652+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:11.657590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.837231+0000","last_change":"2024-09-21T13:48:58.052281+0000","last_active":"2024-09-21T13:49:39.837231+0000","last_peered":"2024-09-21T13:49:39.837231+0000","last_clean":"2024-09-21T13:49:39.837231+0000","last_became_active":"2024-09-21T13:48:58.051974+0000","last_became_peered":"2024-09-21T13:48:58.051974+0000","last_unstale":"2024-09-21T13:49:39.837231+0000","last_undegraded":"2024-09-21T13:49:39.837231+0000","last_fullsized":"2024-09-21T13:49:39.837231+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:25.614472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835533+0000","last_change":"2024-09-21T13:49:02.056666+0000","last_active":"2024-09-21T13:49:39.835533+0000","last_peered":"2024-09-21T13:49:39.835533+0000","last_clean":"2024-09-21T13:49:39.835533+0000","last_became_active":"2024-09-21T13:49:02.056006+0000","last_became_peered":"2024-09-21T13:49:02.056006+0000","last_unstale":"2024-09-21T13:49:39.835533+0000","last_undegraded":"2024-09-21T13:49:39.835533+0000","last_fullsized":"2024-09-21T13:49:39.835533+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:17.511468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836235+0000","last_change":"2024-09-21T13:49:04.434380+0000","last_active":"2024-09-21T13:49:39.836235+0000","last_peered":"2024-09-21T13:49:39.836235+0000","last_clean":"2024-09-21T13:49:39.836235+0000","last_became_active":"2024-09-21T13:49:04.434034+0000","last_became_peered":"2024-09-21T13:49:04.434034+0000","last_unstale":"2024-09-21T13:49:39.836235+0000","last_undegraded":"2024-09-21T13:49:39.836235+0000","last_fullsized":"2024-09-21T13:49:39.836235+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:28.547927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"68'4","reported_seq":48,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835309+0000","last_change":"2024-09-21T13:49:00.063374+0000","last_active":"2024-09-21T13:49:39.835309+0000","last_peered":"2024-09-21T13:49:39.835309+0000","last_clean":"2024-09-21T13:49:39.835309+0000","last_became_active":"2024-09-21T13:49:00.063066+0000","last_became_peered":"2024-09-21T13:49:00.063066+0000","last_unstale":"2024-09-21T13:49:39.835309+0000","last_undegraded":"2024-09-21T13:49:39.835309+0000","last_fullsized":"2024-09-21T13:49:39.835309+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:19.893798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836232+0000","last_change":"2024-09-21T13:48:58.059979+0000","last_active":"2024-09-21T13:49:39.836232+0000","last_peered":"2024-09-21T13:49:39.836232+0000","last_clean":"2024-09-21T13:49:39.836232+0000","last_became_active":"2024-09-21T13:48:58.059674+0000","last_became_peered":"2024-09-21T13:48:58.059674+0000","last_unstale":"2024-09-21T13:49:39.836232+0000","last_undegraded":"2024-09-21T13:49:39.836232+0000","last_fullsized":"2024-09-21T13:49:39.836232+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:02.308942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835914+0000","last_change":"2024-09-21T13:49:02.065043+0000","last_active":"2024-09-21T13:49:39.835914+0000","last_peered":"2024-09-21T13:49:39.835914+0000","last_clean":"2024-09-21T13:49:39.835914+0000","last_became_active":"2024-09-21T13:49:02.064789+0000","last_became_peered":"2024-09-21T13:49:02.064789+0000","last_unstale":"2024-09-21T13:49:39.835914+0000","last_undegraded":"2024-09-21T13:49:39.835914+0000","last_fullsized":"2024-09-21T13:49:39.835914+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:10.569664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835522+0000","last_change":"2024-09-21T13:49:04.075170+0000","last_active":"2024-09-21T13:49:39.835522+0000","last_peered":"2024-09-21T13:49:39.835522+0000","last_clean":"2024-09-21T13:49:39.835522+0000","last_became_active":"2024-09-21T13:49:04.074857+0000","last_became_peered":"2024-09-21T13:49:04.074857+0000","last_unstale":"2024-09-21T13:49:39.835522+0000","last_undegraded":"2024-09-21T13:49:39.835522+0000","last_fullsized":"2024-09-21T13:49:39.835522+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:17.971950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.808102+0000","last_change":"2024-09-21T13:49:04.075860+0000","last_active":"2024-09-21T13:49:39.808102+0000","last_peered":"2024-09-21T13:49:39.808102+0000","last_clean":"2024-09-21T13:49:39.808102+0000","last_became_active":"2024-09-21T13:49:04.075562+0000","last_became_peered":"2024-09-21T13:49:04.075562+0000","last_unstale":"2024-09-21T13:49:39.808102+0000","last_undegraded":"2024-09-21T13:49:39.808102+0000","last_fullsized":"2024-09-21T13:49:39.808102+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:21.041626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835570+0000","last_change":"2024-09-21T13:48:58.055791+0000","last_active":"2024-09-21T13:49:39.835570+0000","last_peered":"2024-09-21T13:49:39.835570+0000","last_clean":"2024-09-21T13:49:39.835570+0000","last_became_active":"2024-09-21T13:48:58.055497+0000","last_became_peered":"2024-09-21T13:48:58.055497+0000","last_unstale":"2024-09-21T13:49:39.835570+0000","last_undegraded":"2024-09-21T13:49:39.835570+0000","last_fullsized":"2024-09-21T13:49:39.835570+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:19.570880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"68'11","reported_seq":61,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836752+0000","last_change":"2024-09-21T13:49:00.058623+0000","last_active":"2024-09-21T13:49:39.836752+0000","last_peered":"2024-09-21T13:49:39.836752+0000","last_clean":"2024-09-21T13:49:39.836752+0000","last_became_active":"2024-09-21T13:49:00.058259+0000","last_became_peered":"2024-09-21T13:49:00.058259+0000","last_unstale":"2024-09-21T13:49:39.836752+0000","last_undegraded":"2024-09-21T13:49:39.836752+0000","last_fullsized":"2024-09-21T13:49:39.836752+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:17.663571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836553+0000","last_change":"2024-09-21T13:49:02.068530+0000","last_active":"2024-09-21T13:49:39.836553+0000","last_peered":"2024-09-21T13:49:39.836553+0000","last_clean":"2024-09-21T13:49:39.836553+0000","last_became_active":"2024-09-21T13:49:02.066159+0000","last_became_peered":"2024-09-21T13:49:02.066159+0000","last_unstale":"2024-09-21T13:49:39.836553+0000","last_undegraded":"2024-09-21T13:49:39.836553+0000","last_fullsized":"2024-09-21T13:49:39.836553+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:29.718157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":40,"num_read_kb":35,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928233998,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32628,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712524,"statfs":{"total":95995035648,"available":95961624576,"internally_reserved":0,"allocated":1474560,"data_stored":970453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462931,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28448,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716704,"statfs":{"total":95995035648,"available":95965904896,"internally_reserved":0,"allocated":1454080,"data_stored":969368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724570,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27872,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717280,"statfs":{"total":95995035648,"available":95966494720,"internally_reserved":0,"allocated":864256,"data_stored":379916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920800,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27900,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717252,"statfs":{"total":95995035648,"available":95966466048,"internally_reserved":0,"allocated":892928,"data_stored":379618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117029,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27904,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717248,"statfs":{"total":95995035648,"available":95966461952,"internally_reserved":0,"allocated":897024,"data_stored":380448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411370,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32028,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713124,"statfs":{"total":95995035648,"available":95962238976,"internally_reserved":0,"allocated":860160,"data_stored":379718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32092,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713060,"statfs":{"total":95995035648,"available":95962173440,"internally_reserved":0,"allocated":925696,"data_stored":380362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738424,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28468,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716684,"statfs":{"total":95995035648,"available":95965884416,"internally_reserved":0,"allocated":1474560,"data_stored":969726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T13:49:46.804 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-21T13:49:46.804 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-21T13:49:46.804 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-21T13:49:46.804 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph health --format=json 2024-09-21T13:49:47.061 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:49:47.455 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:47 smithi049 ceph-mon[40489]: Updating smithi187:/var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T13:49:47.455 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:47 smithi049 ceph-mon[40489]: Updating smithi049:/var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T13:49:47.455 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:47 smithi049 ceph-mon[40489]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T13:49:47.455 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:47 smithi049 ceph-mon[40489]: Deploying daemon alertmanager.a on smithi049 2024-09-21T13:49:47.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:47 smithi049 ceph-mon[30080]: Updating smithi187:/var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T13:49:47.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:47 smithi049 ceph-mon[30080]: Updating smithi049:/var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T13:49:47.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:47 smithi049 ceph-mon[30080]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T13:49:47.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:47 smithi049 ceph-mon[30080]: Deploying daemon alertmanager.a on smithi049 2024-09-21T13:49:47.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:47 smithi187 ceph-mon[33032]: Updating smithi187:/var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T13:49:47.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:47 smithi187 ceph-mon[33032]: Updating smithi049:/var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T13:49:47.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:47 smithi187 ceph-mon[33032]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T13:49:47.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:47 smithi187 ceph-mon[33032]: Deploying daemon alertmanager.a on smithi049 2024-09-21T13:49:47.969 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:49:47.969 INFO:teuthology.orchestra.run.smithi049.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-21T13:49:48.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:48 smithi187 ceph-mon[33032]: pgmap v16: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-21T13:49:48.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:48 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/988776258' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-21T13:49:48.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:48 smithi049 ceph-mon[30080]: pgmap v16: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-21T13:49:48.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:48 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/988776258' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-21T13:49:48.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:48 smithi049 ceph-mon[40489]: pgmap v16: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-21T13:49:48.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:48 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/988776258' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-21T13:49:48.974 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-21T13:49:48.974 INFO:tasks.cephadm:Setup complete, yielding 2024-09-21T13:49:48.975 INFO:teuthology.run_tasks:Running task thrashosds... 2024-09-21T13:49:48.984 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-21T13:49:48.984 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-guits-main-2024-09-21-0842'}, 'ceph': {'conf': {'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'CEPHADM_STRAY_DAEMON', 'CEPHADM_FAILED_DAEMON', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'log-only-match': ['CEPHADM_'], 'sha1': '441784946648ff53dd310293f9dd752afeeea72a'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'cephadm': {'cephadm_mode': 'root'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '441784946648ff53dd310293f9dd752afeeea72a'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-guits-main-2024-09-21-0842', 'sha1': '441784946648ff53dd310293f9dd752afeeea72a'}} 2024-09-21T13:49:48.984 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-21T13:49:48.985 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-09-21T13:49:49.195 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:49:48 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:49:49.247 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:49:50.416 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:49:50 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: [21/Sep/2024:13:49:50] ENGINE Bus STOPPING 2024-09-21T13:49:50.416 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:50 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:49:50.416 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:50 smithi049 ceph-mon[30080]: pgmap v17: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-21T13:49:50.416 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:50 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' 2024-09-21T13:49:50.682 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:50 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:49:50.682 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:50 smithi049 ceph-mon[40489]: pgmap v17: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-21T13:49:50.682 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:50 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' 2024-09-21T13:49:50.683 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:49:50 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:49:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:49:50.683 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:49:50 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: [21/Sep/2024:13:49:50] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T13:49:50.683 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:49:50 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: [21/Sep/2024:13:49:50] ENGINE Bus STOPPED 2024-09-21T13:49:50.683 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:49:50 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: [21/Sep/2024:13:49:50] ENGINE Bus STARTING 2024-09-21T13:49:50.683 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:49:50 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: [21/Sep/2024:13:49:50] ENGINE Serving on http://:::9283 2024-09-21T13:49:50.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:50 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:49:50.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:50 smithi187 ceph-mon[33032]: pgmap v17: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-21T13:49:50.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:50 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' 2024-09-21T13:49:51.005 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:49:50 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: [21/Sep/2024:13:49:50] ENGINE Bus STARTED 2024-09-21T13:49:51.560 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-21T13:49:51.561 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-21T13:49:51.561 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T13:49:51.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:51 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T13:49:51.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:51 smithi187 ceph-mon[33032]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T13:49:51.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:51 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T13:49:51.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:51 smithi049 ceph-mon[40489]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T13:49:51.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:51 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T13:49:51.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:51 smithi049 ceph-mon[30080]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T13:49:52.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:52 smithi049 ceph-mon[30080]: pgmap v18: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 888 B/s rd, 0 op/s 2024-09-21T13:49:52.465 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:52 smithi049 ceph-mon[40489]: pgmap v18: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 888 B/s rd, 0 op/s 2024-09-21T13:49:52.694 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:49:52.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:52 smithi187 ceph-mon[33032]: pgmap v18: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 888 B/s rd, 0 op/s 2024-09-21T13:49:54.594 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:54 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:49:54.594 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:54 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:49:54.595 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:54 smithi049 ceph-mon[40489]: pgmap v19: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T13:49:54.595 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:54 smithi049 ceph-mon[40489]: osdmap e79: 8 total, 8 up, 8 in 2024-09-21T13:49:54.595 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:54 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:49:54.595 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:54 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:49:54.595 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:54 smithi049 ceph-mon[30080]: pgmap v19: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T13:49:54.595 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:54 smithi049 ceph-mon[30080]: osdmap e79: 8 total, 8 up, 8 in 2024-09-21T13:49:54.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:54 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:49:54.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:54 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:49:54.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:54 smithi187 ceph-mon[33032]: pgmap v19: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T13:49:54.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:54 smithi187 ceph-mon[33032]: osdmap e79: 8 total, 8 up, 8 in 2024-09-21T13:49:55.575 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:49:55.576 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-09-21T13:49:55.916 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:55 smithi049 ceph-mon[40489]: osdmap e80: 8 total, 8 up, 8 in 2024-09-21T13:49:55.917 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:55 smithi049 ceph-mon[40489]: pgmap v22: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T13:49:55.917 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:55 smithi049 ceph-mon[40489]: from='client.24650 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T13:49:55.917 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:55 smithi049 ceph-mon[30080]: osdmap e80: 8 total, 8 up, 8 in 2024-09-21T13:49:55.917 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:55 smithi049 ceph-mon[30080]: pgmap v22: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T13:49:55.917 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:55 smithi049 ceph-mon[30080]: from='client.24650 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T13:49:56.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:55 smithi187 ceph-mon[33032]: osdmap e80: 8 total, 8 up, 8 in 2024-09-21T13:49:56.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:55 smithi187 ceph-mon[33032]: pgmap v22: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T13:49:56.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:55 smithi187 ceph-mon[33032]: from='client.24650 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T13:49:56.252 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-09-21T13:49:55.335037+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":898,"num_read_kb":755,"num_write":549,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":552,"ondisk_log_size":552,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":237340,"kb_used_data":8636,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":749723876,"statfs":{"total":767960285184,"available":767717249024,"internally_reserved":0,"allocated":8843264,"data_stored":4809609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001591"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836141+0000","last_change":"2024-09-21T13:48:58.061797+0000","last_active":"2024-09-21T13:49:39.836141+0000","last_peered":"2024-09-21T13:49:39.836141+0000","last_clean":"2024-09-21T13:49:39.836141+0000","last_became_active":"2024-09-21T13:48:58.061415+0000","last_became_peered":"2024-09-21T13:48:58.061415+0000","last_unstale":"2024-09-21T13:49:39.836141+0000","last_undegraded":"2024-09-21T13:49:39.836141+0000","last_fullsized":"2024-09-21T13:49:39.836141+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:07.025619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"68'9","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.811507+0000","last_change":"2024-09-21T13:49:00.055717+0000","last_active":"2024-09-21T13:49:39.811507+0000","last_peered":"2024-09-21T13:49:39.811507+0000","last_clean":"2024-09-21T13:49:39.811507+0000","last_became_active":"2024-09-21T13:49:00.055210+0000","last_became_peered":"2024-09-21T13:49:00.055210+0000","last_unstale":"2024-09-21T13:49:39.811507+0000","last_undegraded":"2024-09-21T13:49:39.811507+0000","last_fullsized":"2024-09-21T13:49:39.811507+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:13.541017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.806595+0000","last_change":"2024-09-21T13:49:02.061394+0000","last_active":"2024-09-21T13:49:39.806595+0000","last_peered":"2024-09-21T13:49:39.806595+0000","last_clean":"2024-09-21T13:49:39.806595+0000","last_became_active":"2024-09-21T13:49:02.060984+0000","last_became_peered":"2024-09-21T13:49:02.060984+0000","last_unstale":"2024-09-21T13:49:39.806595+0000","last_undegraded":"2024-09-21T13:49:39.806595+0000","last_fullsized":"2024-09-21T13:49:39.806595+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:18.936236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.811533+0000","last_change":"2024-09-21T13:49:04.072734+0000","last_active":"2024-09-21T13:49:39.811533+0000","last_peered":"2024-09-21T13:49:39.811533+0000","last_clean":"2024-09-21T13:49:39.811533+0000","last_became_active":"2024-09-21T13:49:04.072460+0000","last_became_peered":"2024-09-21T13:49:04.072460+0000","last_unstale":"2024-09-21T13:49:39.811533+0000","last_undegraded":"2024-09-21T13:49:39.811533+0000","last_fullsized":"2024-09-21T13:49:39.811533+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:30.526897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836010+0000","last_change":"2024-09-21T13:49:04.434312+0000","last_active":"2024-09-21T13:49:39.836010+0000","last_peered":"2024-09-21T13:49:39.836010+0000","last_clean":"2024-09-21T13:49:39.836010+0000","last_became_active":"2024-09-21T13:49:04.433965+0000","last_became_peered":"2024-09-21T13:49:04.433965+0000","last_unstale":"2024-09-21T13:49:39.836010+0000","last_undegraded":"2024-09-21T13:49:39.836010+0000","last_fullsized":"2024-09-21T13:49:39.836010+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:06.082699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836648+0000","last_change":"2024-09-21T13:48:58.051225+0000","last_active":"2024-09-21T13:49:39.836648+0000","last_peered":"2024-09-21T13:49:39.836648+0000","last_clean":"2024-09-21T13:49:39.836648+0000","last_became_active":"2024-09-21T13:48:58.050479+0000","last_became_peered":"2024-09-21T13:48:58.050479+0000","last_unstale":"2024-09-21T13:49:39.836648+0000","last_undegraded":"2024-09-21T13:49:39.836648+0000","last_fullsized":"2024-09-21T13:49:39.836648+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:19.515469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"68'15","reported_seq":67,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835920+0000","last_change":"2024-09-21T13:49:00.063787+0000","last_active":"2024-09-21T13:49:39.835920+0000","last_peered":"2024-09-21T13:49:39.835920+0000","last_clean":"2024-09-21T13:49:39.835920+0000","last_became_active":"2024-09-21T13:49:00.063460+0000","last_became_peered":"2024-09-21T13:49:00.063460+0000","last_unstale":"2024-09-21T13:49:39.835920+0000","last_undegraded":"2024-09-21T13:49:39.835920+0000","last_fullsized":"2024-09-21T13:49:39.835920+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:14.007409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.811464+0000","last_change":"2024-09-21T13:49:02.064007+0000","last_active":"2024-09-21T13:49:39.811464+0000","last_peered":"2024-09-21T13:49:39.811464+0000","last_clean":"2024-09-21T13:49:39.811464+0000","last_became_active":"2024-09-21T13:49:02.063687+0000","last_became_peered":"2024-09-21T13:49:02.063687+0000","last_unstale":"2024-09-21T13:49:39.811464+0000","last_undegraded":"2024-09-21T13:49:39.811464+0000","last_fullsized":"2024-09-21T13:49:39.811464+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:05.421236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.837174+0000","last_change":"2024-09-21T13:48:58.051001+0000","last_active":"2024-09-21T13:49:39.837174+0000","last_peered":"2024-09-21T13:49:39.837174+0000","last_clean":"2024-09-21T13:49:39.837174+0000","last_became_active":"2024-09-21T13:48:58.050695+0000","last_became_peered":"2024-09-21T13:48:58.050695+0000","last_unstale":"2024-09-21T13:49:39.837174+0000","last_undegraded":"2024-09-21T13:49:39.837174+0000","last_fullsized":"2024-09-21T13:49:39.837174+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:09.186887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"68'9","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.811774+0000","last_change":"2024-09-21T13:49:00.059748+0000","last_active":"2024-09-21T13:49:39.811774+0000","last_peered":"2024-09-21T13:49:39.811774+0000","last_clean":"2024-09-21T13:49:39.811774+0000","last_became_active":"2024-09-21T13:49:00.059382+0000","last_became_peered":"2024-09-21T13:49:00.059382+0000","last_unstale":"2024-09-21T13:49:39.811774+0000","last_undegraded":"2024-09-21T13:49:39.811774+0000","last_fullsized":"2024-09-21T13:49:39.811774+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:05.303397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836250+0000","last_change":"2024-09-21T13:49:02.066978+0000","last_active":"2024-09-21T13:49:39.836250+0000","last_peered":"2024-09-21T13:49:39.836250+0000","last_clean":"2024-09-21T13:49:39.836250+0000","last_became_active":"2024-09-21T13:49:02.066693+0000","last_became_peered":"2024-09-21T13:49:02.066693+0000","last_unstale":"2024-09-21T13:49:39.836250+0000","last_undegraded":"2024-09-21T13:49:39.836250+0000","last_fullsized":"2024-09-21T13:49:39.836250+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:28.710214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835306+0000","last_change":"2024-09-21T13:49:04.080709+0000","last_active":"2024-09-21T13:49:39.835306+0000","last_peered":"2024-09-21T13:49:39.835306+0000","last_clean":"2024-09-21T13:49:39.835306+0000","last_became_active":"2024-09-21T13:49:04.080359+0000","last_became_peered":"2024-09-21T13:49:04.080359+0000","last_unstale":"2024-09-21T13:49:39.835306+0000","last_undegraded":"2024-09-21T13:49:39.835306+0000","last_fullsized":"2024-09-21T13:49:39.835306+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:09.590774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836673+0000","last_change":"2024-09-21T13:48:58.052581+0000","last_active":"2024-09-21T13:49:39.836673+0000","last_peered":"2024-09-21T13:49:39.836673+0000","last_clean":"2024-09-21T13:49:39.836673+0000","last_became_active":"2024-09-21T13:48:58.050906+0000","last_became_peered":"2024-09-21T13:48:58.050906+0000","last_unstale":"2024-09-21T13:49:39.836673+0000","last_undegraded":"2024-09-21T13:49:39.836673+0000","last_fullsized":"2024-09-21T13:49:39.836673+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:19.977336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"68'5","reported_seq":52,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.811724+0000","last_change":"2024-09-21T13:49:00.060913+0000","last_active":"2024-09-21T13:49:39.811724+0000","last_peered":"2024-09-21T13:49:39.811724+0000","last_clean":"2024-09-21T13:49:39.811724+0000","last_became_active":"2024-09-21T13:49:00.060497+0000","last_became_peered":"2024-09-21T13:49:00.060497+0000","last_unstale":"2024-09-21T13:49:39.811724+0000","last_undegraded":"2024-09-21T13:49:39.811724+0000","last_fullsized":"2024-09-21T13:49:39.811724+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:10.507839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835593+0000","last_change":"2024-09-21T13:49:02.060484+0000","last_active":"2024-09-21T13:49:39.835593+0000","last_peered":"2024-09-21T13:49:39.835593+0000","last_clean":"2024-09-21T13:49:39.835593+0000","last_became_active":"2024-09-21T13:49:02.060212+0000","last_became_peered":"2024-09-21T13:49:02.060212+0000","last_unstale":"2024-09-21T13:49:39.835593+0000","last_undegraded":"2024-09-21T13:49:39.835593+0000","last_fullsized":"2024-09-21T13:49:39.835593+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:17.976156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835659+0000","last_change":"2024-09-21T13:49:04.078296+0000","last_active":"2024-09-21T13:49:39.835659+0000","last_peered":"2024-09-21T13:49:39.835659+0000","last_clean":"2024-09-21T13:49:39.835659+0000","last_became_active":"2024-09-21T13:49:04.078026+0000","last_became_peered":"2024-09-21T13:49:04.078026+0000","last_unstale":"2024-09-21T13:49:39.835659+0000","last_undegraded":"2024-09-21T13:49:39.835659+0000","last_fullsized":"2024-09-21T13:49:39.835659+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:05.284846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.812185+0000","last_change":"2024-09-21T13:49:04.432458+0000","last_active":"2024-09-21T13:49:39.812185+0000","last_peered":"2024-09-21T13:49:39.812185+0000","last_clean":"2024-09-21T13:49:39.812185+0000","last_became_active":"2024-09-21T13:49:04.431422+0000","last_became_peered":"2024-09-21T13:49:04.431422+0000","last_unstale":"2024-09-21T13:49:39.812185+0000","last_undegraded":"2024-09-21T13:49:39.812185+0000","last_fullsized":"2024-09-21T13:49:39.812185+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:08.067421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836389+0000","last_change":"2024-09-21T13:48:58.057667+0000","last_active":"2024-09-21T13:49:39.836389+0000","last_peered":"2024-09-21T13:49:39.836389+0000","last_clean":"2024-09-21T13:49:39.836389+0000","last_became_active":"2024-09-21T13:48:58.057336+0000","last_became_peered":"2024-09-21T13:48:58.057336+0000","last_unstale":"2024-09-21T13:49:39.836389+0000","last_undegraded":"2024-09-21T13:49:39.836389+0000","last_fullsized":"2024-09-21T13:49:39.836389+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:18.555629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"68'15","reported_seq":67,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.807764+0000","last_change":"2024-09-21T13:49:00.059068+0000","last_active":"2024-09-21T13:49:39.807764+0000","last_peered":"2024-09-21T13:49:39.807764+0000","last_clean":"2024-09-21T13:49:39.807764+0000","last_became_active":"2024-09-21T13:49:00.058730+0000","last_became_peered":"2024-09-21T13:49:00.058730+0000","last_unstale":"2024-09-21T13:49:39.807764+0000","last_undegraded":"2024-09-21T13:49:39.807764+0000","last_fullsized":"2024-09-21T13:49:39.807764+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:18.796269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.808006+0000","last_change":"2024-09-21T13:49:02.067253+0000","last_active":"2024-09-21T13:49:39.808006+0000","last_peered":"2024-09-21T13:49:39.808006+0000","last_clean":"2024-09-21T13:49:39.808006+0000","last_became_active":"2024-09-21T13:49:02.066796+0000","last_became_peered":"2024-09-21T13:49:02.066796+0000","last_unstale":"2024-09-21T13:49:39.808006+0000","last_undegraded":"2024-09-21T13:49:39.808006+0000","last_fullsized":"2024-09-21T13:49:39.808006+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:24.625163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836826+0000","last_change":"2024-09-21T13:49:04.434429+0000","last_active":"2024-09-21T13:49:39.836826+0000","last_peered":"2024-09-21T13:49:39.836826+0000","last_clean":"2024-09-21T13:49:39.836826+0000","last_became_active":"2024-09-21T13:49:04.434049+0000","last_became_peered":"2024-09-21T13:49:04.434049+0000","last_unstale":"2024-09-21T13:49:39.836826+0000","last_undegraded":"2024-09-21T13:49:39.836826+0000","last_fullsized":"2024-09-21T13:49:39.836826+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:25.759474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.811889+0000","last_change":"2024-09-21T13:48:58.043457+0000","last_active":"2024-09-21T13:49:39.811889+0000","last_peered":"2024-09-21T13:49:39.811889+0000","last_clean":"2024-09-21T13:49:39.811889+0000","last_became_active":"2024-09-21T13:48:58.043088+0000","last_became_peered":"2024-09-21T13:48:58.043088+0000","last_unstale":"2024-09-21T13:49:39.811889+0000","last_undegraded":"2024-09-21T13:49:39.811889+0000","last_fullsized":"2024-09-21T13:49:39.811889+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:21.622429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"68'12","reported_seq":65,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836785+0000","last_change":"2024-09-21T13:49:00.062162+0000","last_active":"2024-09-21T13:49:39.836785+0000","last_peered":"2024-09-21T13:49:39.836785+0000","last_clean":"2024-09-21T13:49:39.836785+0000","last_became_active":"2024-09-21T13:49:00.061867+0000","last_became_peered":"2024-09-21T13:49:00.061867+0000","last_unstale":"2024-09-21T13:49:39.836785+0000","last_undegraded":"2024-09-21T13:49:39.836785+0000","last_fullsized":"2024-09-21T13:49:39.836785+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:27.206067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.812068+0000","last_change":"2024-09-21T13:49:02.063090+0000","last_active":"2024-09-21T13:49:39.812068+0000","last_peered":"2024-09-21T13:49:39.812068+0000","last_clean":"2024-09-21T13:49:39.812068+0000","last_became_active":"2024-09-21T13:49:02.062802+0000","last_became_peered":"2024-09-21T13:49:02.062802+0000","last_unstale":"2024-09-21T13:49:39.812068+0000","last_undegraded":"2024-09-21T13:49:39.812068+0000","last_fullsized":"2024-09-21T13:49:39.812068+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:20.369055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":35,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836298+0000","last_change":"2024-09-21T13:49:04.076090+0000","last_active":"2024-09-21T13:49:39.836298+0000","last_peered":"2024-09-21T13:49:39.836298+0000","last_clean":"2024-09-21T13:49:39.836298+0000","last_became_active":"2024-09-21T13:49:04.075792+0000","last_became_peered":"2024-09-21T13:49:04.075792+0000","last_unstale":"2024-09-21T13:49:39.836298+0000","last_undegraded":"2024-09-21T13:49:39.836298+0000","last_fullsized":"2024-09-21T13:49:39.836298+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:07.846061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":47,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.806690+0000","last_change":"2024-09-21T13:48:58.049463+0000","last_active":"2024-09-21T13:49:39.806690+0000","last_peered":"2024-09-21T13:49:39.806690+0000","last_clean":"2024-09-21T13:49:39.806690+0000","last_became_active":"2024-09-21T13:48:58.048972+0000","last_became_peered":"2024-09-21T13:48:58.048972+0000","last_unstale":"2024-09-21T13:49:39.806690+0000","last_undegraded":"2024-09-21T13:49:39.806690+0000","last_fullsized":"2024-09-21T13:49:39.806690+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:03.427338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"68'10","reported_seq":57,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836578+0000","last_change":"2024-09-21T13:49:00.426837+0000","last_active":"2024-09-21T13:49:39.836578+0000","last_peered":"2024-09-21T13:49:39.836578+0000","last_clean":"2024-09-21T13:49:39.836578+0000","last_became_active":"2024-09-21T13:49:00.426178+0000","last_became_peered":"2024-09-21T13:49:00.426178+0000","last_unstale":"2024-09-21T13:49:39.836578+0000","last_undegraded":"2024-09-21T13:49:39.836578+0000","last_fullsized":"2024-09-21T13:49:39.836578+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:22.029884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"68'8","reported_seq":61,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:48.114202+0000","last_change":"2024-09-21T13:49:02.059979+0000","last_active":"2024-09-21T13:49:48.114202+0000","last_peered":"2024-09-21T13:49:48.114202+0000","last_clean":"2024-09-21T13:49:48.114202+0000","last_became_active":"2024-09-21T13:49:02.059638+0000","last_became_peered":"2024-09-21T13:49:02.059638+0000","last_unstale":"2024-09-21T13:49:48.114202+0000","last_undegraded":"2024-09-21T13:49:48.114202+0000","last_fullsized":"2024-09-21T13:49:48.114202+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:26.974288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"68'15","reported_seq":67,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.808011+0000","last_change":"2024-09-21T13:49:00.060620+0000","last_active":"2024-09-21T13:49:39.808011+0000","last_peered":"2024-09-21T13:49:39.808011+0000","last_clean":"2024-09-21T13:49:39.808011+0000","last_became_active":"2024-09-21T13:49:00.060324+0000","last_became_peered":"2024-09-21T13:49:00.060324+0000","last_unstale":"2024-09-21T13:49:39.808011+0000","last_undegraded":"2024-09-21T13:49:39.808011+0000","last_fullsized":"2024-09-21T13:49:39.808011+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:20.149960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835655+0000","last_change":"2024-09-21T13:48:58.052171+0000","last_active":"2024-09-21T13:49:39.835655+0000","last_peered":"2024-09-21T13:49:39.835655+0000","last_clean":"2024-09-21T13:49:39.835655+0000","last_became_active":"2024-09-21T13:48:58.051911+0000","last_became_peered":"2024-09-21T13:48:58.051911+0000","last_unstale":"2024-09-21T13:49:39.835655+0000","last_undegraded":"2024-09-21T13:49:39.835655+0000","last_fullsized":"2024-09-21T13:49:39.835655+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:49:58.457410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"68'8","reported_seq":61,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:48.114264+0000","last_change":"2024-09-21T13:49:02.061966+0000","last_active":"2024-09-21T13:49:48.114264+0000","last_peered":"2024-09-21T13:49:48.114264+0000","last_clean":"2024-09-21T13:49:48.114264+0000","last_became_active":"2024-09-21T13:49:02.061679+0000","last_became_peered":"2024-09-21T13:49:02.061679+0000","last_unstale":"2024-09-21T13:49:48.114264+0000","last_undegraded":"2024-09-21T13:49:48.114264+0000","last_fullsized":"2024-09-21T13:49:48.114264+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:28.854727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836647+0000","last_change":"2024-09-21T13:49:04.082213+0000","last_active":"2024-09-21T13:49:39.836647+0000","last_peered":"2024-09-21T13:49:39.836647+0000","last_clean":"2024-09-21T13:49:39.836647+0000","last_became_active":"2024-09-21T13:49:04.081905+0000","last_became_peered":"2024-09-21T13:49:04.081905+0000","last_unstale":"2024-09-21T13:49:39.836647+0000","last_undegraded":"2024-09-21T13:49:39.836647+0000","last_fullsized":"2024-09-21T13:49:39.836647+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:17.499009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"68'18","reported_seq":74,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835396+0000","last_change":"2024-09-21T13:49:00.429686+0000","last_active":"2024-09-21T13:49:39.835396+0000","last_peered":"2024-09-21T13:49:39.835396+0000","last_clean":"2024-09-21T13:49:39.835396+0000","last_became_active":"2024-09-21T13:49:00.429314+0000","last_became_peered":"2024-09-21T13:49:00.429314+0000","last_unstale":"2024-09-21T13:49:39.835396+0000","last_undegraded":"2024-09-21T13:49:39.835396+0000","last_fullsized":"2024-09-21T13:49:39.835396+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:09.824746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.837041+0000","last_change":"2024-09-21T13:48:58.056669+0000","last_active":"2024-09-21T13:49:39.837041+0000","last_peered":"2024-09-21T13:49:39.837041+0000","last_clean":"2024-09-21T13:49:39.837041+0000","last_became_active":"2024-09-21T13:48:58.056375+0000","last_became_peered":"2024-09-21T13:48:58.056375+0000","last_unstale":"2024-09-21T13:49:39.837041+0000","last_undegraded":"2024-09-21T13:49:39.837041+0000","last_fullsized":"2024-09-21T13:49:39.837041+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:21.686532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.811469+0000","last_change":"2024-09-21T13:49:02.063237+0000","last_active":"2024-09-21T13:49:39.811469+0000","last_peered":"2024-09-21T13:49:39.811469+0000","last_clean":"2024-09-21T13:49:39.811469+0000","last_became_active":"2024-09-21T13:49:02.062955+0000","last_became_peered":"2024-09-21T13:49:02.062955+0000","last_unstale":"2024-09-21T13:49:39.811469+0000","last_undegraded":"2024-09-21T13:49:39.811469+0000","last_fullsized":"2024-09-21T13:49:39.811469+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:05.393016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.811701+0000","last_change":"2024-09-21T13:49:04.076445+0000","last_active":"2024-09-21T13:49:39.811701+0000","last_peered":"2024-09-21T13:49:39.811701+0000","last_clean":"2024-09-21T13:49:39.811701+0000","last_became_active":"2024-09-21T13:49:04.075818+0000","last_became_peered":"2024-09-21T13:49:04.075818+0000","last_unstale":"2024-09-21T13:49:39.811701+0000","last_undegraded":"2024-09-21T13:49:39.811701+0000","last_fullsized":"2024-09-21T13:49:39.811701+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:31.482751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"68'14","reported_seq":63,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.811530+0000","last_change":"2024-09-21T13:49:00.055423+0000","last_active":"2024-09-21T13:49:39.811530+0000","last_peered":"2024-09-21T13:49:39.811530+0000","last_clean":"2024-09-21T13:49:39.811530+0000","last_became_active":"2024-09-21T13:49:00.055031+0000","last_became_peered":"2024-09-21T13:49:00.055031+0000","last_unstale":"2024-09-21T13:49:39.811530+0000","last_undegraded":"2024-09-21T13:49:39.811530+0000","last_fullsized":"2024-09-21T13:49:39.811530+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:06.138147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":47,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836778+0000","last_change":"2024-09-21T13:48:58.053829+0000","last_active":"2024-09-21T13:49:39.836778+0000","last_peered":"2024-09-21T13:49:39.836778+0000","last_clean":"2024-09-21T13:49:39.836778+0000","last_became_active":"2024-09-21T13:48:58.053533+0000","last_became_peered":"2024-09-21T13:48:58.053533+0000","last_unstale":"2024-09-21T13:49:39.836778+0000","last_undegraded":"2024-09-21T13:49:39.836778+0000","last_fullsized":"2024-09-21T13:49:39.836778+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:49:58.639672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"68'8","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:48.114084+0000","last_change":"2024-09-21T13:49:02.070405+0000","last_active":"2024-09-21T13:49:48.114084+0000","last_peered":"2024-09-21T13:49:48.114084+0000","last_clean":"2024-09-21T13:49:48.114084+0000","last_became_active":"2024-09-21T13:49:02.070064+0000","last_became_peered":"2024-09-21T13:49:02.070064+0000","last_unstale":"2024-09-21T13:49:48.114084+0000","last_undegraded":"2024-09-21T13:49:48.114084+0000","last_fullsized":"2024-09-21T13:49:48.114084+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:13.746404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836389+0000","last_change":"2024-09-21T13:49:04.434498+0000","last_active":"2024-09-21T13:49:39.836389+0000","last_peered":"2024-09-21T13:49:39.836389+0000","last_clean":"2024-09-21T13:49:39.836389+0000","last_became_active":"2024-09-21T13:49:04.433920+0000","last_became_peered":"2024-09-21T13:49:04.433920+0000","last_unstale":"2024-09-21T13:49:39.836389+0000","last_undegraded":"2024-09-21T13:49:39.836389+0000","last_fullsized":"2024-09-21T13:49:39.836389+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:22.847243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"68'10","reported_seq":57,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.807473+0000","last_change":"2024-09-21T13:49:00.054972+0000","last_active":"2024-09-21T13:49:39.807473+0000","last_peered":"2024-09-21T13:49:39.807473+0000","last_clean":"2024-09-21T13:49:39.807473+0000","last_became_active":"2024-09-21T13:49:00.054564+0000","last_became_peered":"2024-09-21T13:49:00.054564+0000","last_unstale":"2024-09-21T13:49:39.807473+0000","last_undegraded":"2024-09-21T13:49:39.807473+0000","last_fullsized":"2024-09-21T13:49:39.807473+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:18.647498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"61'1","reported_seq":52,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.837265+0000","last_change":"2024-09-21T13:48:58.050844+0000","last_active":"2024-09-21T13:49:39.837265+0000","last_peered":"2024-09-21T13:49:39.837265+0000","last_clean":"2024-09-21T13:49:39.837265+0000","last_became_active":"2024-09-21T13:48:58.050548+0000","last_became_peered":"2024-09-21T13:48:58.050548+0000","last_unstale":"2024-09-21T13:49:39.837265+0000","last_undegraded":"2024-09-21T13:49:39.837265+0000","last_fullsized":"2024-09-21T13:49:39.837265+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:17.587266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"68'8","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:48.114036+0000","last_change":"2024-09-21T13:49:02.068926+0000","last_active":"2024-09-21T13:49:48.114036+0000","last_peered":"2024-09-21T13:49:48.114036+0000","last_clean":"2024-09-21T13:49:48.114036+0000","last_became_active":"2024-09-21T13:49:02.067664+0000","last_became_peered":"2024-09-21T13:49:02.067664+0000","last_unstale":"2024-09-21T13:49:48.114036+0000","last_undegraded":"2024-09-21T13:49:48.114036+0000","last_fullsized":"2024-09-21T13:49:48.114036+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:09.630253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835435+0000","last_change":"2024-09-21T13:49:04.081771+0000","last_active":"2024-09-21T13:49:39.835435+0000","last_peered":"2024-09-21T13:49:39.835435+0000","last_clean":"2024-09-21T13:49:39.835435+0000","last_became_active":"2024-09-21T13:49:04.081462+0000","last_became_peered":"2024-09-21T13:49:04.081462+0000","last_unstale":"2024-09-21T13:49:39.835435+0000","last_undegraded":"2024-09-21T13:49:39.835435+0000","last_fullsized":"2024-09-21T13:49:39.835435+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:25.632205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"68'19","reported_seq":78,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836173+0000","last_change":"2024-09-21T13:49:00.426917+0000","last_active":"2024-09-21T13:49:39.836173+0000","last_peered":"2024-09-21T13:49:39.836173+0000","last_clean":"2024-09-21T13:49:39.836173+0000","last_became_active":"2024-09-21T13:49:00.426579+0000","last_became_peered":"2024-09-21T13:49:00.426579+0000","last_unstale":"2024-09-21T13:49:39.836173+0000","last_undegraded":"2024-09-21T13:49:39.836173+0000","last_fullsized":"2024-09-21T13:49:39.836173+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:15.913331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.808545+0000","last_change":"2024-09-21T13:48:58.049961+0000","last_active":"2024-09-21T13:49:39.808545+0000","last_peered":"2024-09-21T13:49:39.808545+0000","last_clean":"2024-09-21T13:49:39.808545+0000","last_became_active":"2024-09-21T13:48:58.049682+0000","last_became_peered":"2024-09-21T13:48:58.049682+0000","last_unstale":"2024-09-21T13:49:39.808545+0000","last_undegraded":"2024-09-21T13:49:39.808545+0000","last_fullsized":"2024-09-21T13:49:39.808545+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:02.280644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835812+0000","last_change":"2024-09-21T13:49:02.061604+0000","last_active":"2024-09-21T13:49:39.835812+0000","last_peered":"2024-09-21T13:49:39.835812+0000","last_clean":"2024-09-21T13:49:39.835812+0000","last_became_active":"2024-09-21T13:49:02.061187+0000","last_became_peered":"2024-09-21T13:49:02.061187+0000","last_unstale":"2024-09-21T13:49:39.835812+0000","last_undegraded":"2024-09-21T13:49:39.835812+0000","last_fullsized":"2024-09-21T13:49:39.835812+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:08.846428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.806520+0000","last_change":"2024-09-21T13:49:04.431651+0000","last_active":"2024-09-21T13:49:39.806520+0000","last_peered":"2024-09-21T13:49:39.806520+0000","last_clean":"2024-09-21T13:49:39.806520+0000","last_became_active":"2024-09-21T13:49:04.431079+0000","last_became_peered":"2024-09-21T13:49:04.431079+0000","last_unstale":"2024-09-21T13:49:39.806520+0000","last_undegraded":"2024-09-21T13:49:39.806520+0000","last_fullsized":"2024-09-21T13:49:39.806520+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:05.155255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"68'28","reported_seq":102,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:48.114038+0000","last_change":"2024-09-21T13:49:00.060616+0000","last_active":"2024-09-21T13:49:48.114038+0000","last_peered":"2024-09-21T13:49:48.114038+0000","last_clean":"2024-09-21T13:49:48.114038+0000","last_became_active":"2024-09-21T13:49:00.060355+0000","last_became_peered":"2024-09-21T13:49:00.060355+0000","last_unstale":"2024-09-21T13:49:48.114038+0000","last_undegraded":"2024-09-21T13:49:48.114038+0000","last_fullsized":"2024-09-21T13:49:48.114038+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:26.214000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.812060+0000","last_change":"2024-09-21T13:48:58.055640+0000","last_active":"2024-09-21T13:49:39.812060+0000","last_peered":"2024-09-21T13:49:39.812060+0000","last_clean":"2024-09-21T13:49:39.812060+0000","last_became_active":"2024-09-21T13:48:58.055366+0000","last_became_peered":"2024-09-21T13:48:58.055366+0000","last_unstale":"2024-09-21T13:49:39.812060+0000","last_undegraded":"2024-09-21T13:49:39.812060+0000","last_fullsized":"2024-09-21T13:49:39.812060+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:17.442979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":56,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.837470+0000","last_change":"2024-09-21T13:49:00.053256+0000","last_active":"2024-09-21T13:49:39.837470+0000","last_peered":"2024-09-21T13:49:39.837470+0000","last_clean":"2024-09-21T13:49:39.837470+0000","last_became_active":"2024-09-21T13:48:57.046246+0000","last_became_peered":"2024-09-21T13:48:57.046246+0000","last_unstale":"2024-09-21T13:49:39.837470+0000","last_undegraded":"2024-09-21T13:49:39.837470+0000","last_fullsized":"2024-09-21T13:49:39.837470+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:48:56.029323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:04.022287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015946669999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836059+0000","last_change":"2024-09-21T13:49:02.068949+0000","last_active":"2024-09-21T13:49:39.836059+0000","last_peered":"2024-09-21T13:49:39.836059+0000","last_clean":"2024-09-21T13:49:39.836059+0000","last_became_active":"2024-09-21T13:49:02.068611+0000","last_became_peered":"2024-09-21T13:49:02.068611+0000","last_unstale":"2024-09-21T13:49:39.836059+0000","last_undegraded":"2024-09-21T13:49:39.836059+0000","last_fullsized":"2024-09-21T13:49:39.836059+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:01.307406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836335+0000","last_change":"2024-09-21T13:49:04.080755+0000","last_active":"2024-09-21T13:49:39.836335+0000","last_peered":"2024-09-21T13:49:39.836335+0000","last_clean":"2024-09-21T13:49:39.836335+0000","last_became_active":"2024-09-21T13:49:04.080466+0000","last_became_peered":"2024-09-21T13:49:04.080466+0000","last_unstale":"2024-09-21T13:49:39.836335+0000","last_undegraded":"2024-09-21T13:49:39.836335+0000","last_fullsized":"2024-09-21T13:49:39.836335+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:06.570558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"68'13","reported_seq":69,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.808039+0000","last_change":"2024-09-21T13:49:00.053645+0000","last_active":"2024-09-21T13:49:39.808039+0000","last_peered":"2024-09-21T13:49:39.808039+0000","last_clean":"2024-09-21T13:49:39.808039+0000","last_became_active":"2024-09-21T13:49:00.053323+0000","last_became_peered":"2024-09-21T13:49:00.053323+0000","last_unstale":"2024-09-21T13:49:39.808039+0000","last_undegraded":"2024-09-21T13:49:39.808039+0000","last_fullsized":"2024-09-21T13:49:39.808039+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:02.019380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.806770+0000","last_change":"2024-09-21T13:48:58.050687+0000","last_active":"2024-09-21T13:49:39.806770+0000","last_peered":"2024-09-21T13:49:39.806770+0000","last_clean":"2024-09-21T13:49:39.806770+0000","last_became_active":"2024-09-21T13:48:58.050371+0000","last_became_peered":"2024-09-21T13:48:58.050371+0000","last_unstale":"2024-09-21T13:49:39.806770+0000","last_undegraded":"2024-09-21T13:49:39.806770+0000","last_fullsized":"2024-09-21T13:49:39.806770+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:19.757852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":55,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.807250+0000","last_change":"2024-09-21T13:49:00.045590+0000","last_active":"2024-09-21T13:49:39.807250+0000","last_peered":"2024-09-21T13:49:39.807250+0000","last_clean":"2024-09-21T13:49:39.807250+0000","last_became_active":"2024-09-21T13:48:57.047067+0000","last_became_peered":"2024-09-21T13:48:57.047067+0000","last_unstale":"2024-09-21T13:49:39.807250+0000","last_undegraded":"2024-09-21T13:49:39.807250+0000","last_fullsized":"2024-09-21T13:49:39.807250+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:48:56.029323+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:49:56.701182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065666300000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.807847+0000","last_change":"2024-09-21T13:49:02.057270+0000","last_active":"2024-09-21T13:49:39.807847+0000","last_peered":"2024-09-21T13:49:39.807847+0000","last_clean":"2024-09-21T13:49:39.807847+0000","last_became_active":"2024-09-21T13:49:02.056806+0000","last_became_peered":"2024-09-21T13:49:02.056806+0000","last_unstale":"2024-09-21T13:49:39.807847+0000","last_undegraded":"2024-09-21T13:49:39.807847+0000","last_fullsized":"2024-09-21T13:49:39.807847+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:17.996686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836856+0000","last_change":"2024-09-21T13:49:04.431351+0000","last_active":"2024-09-21T13:49:39.836856+0000","last_peered":"2024-09-21T13:49:39.836856+0000","last_clean":"2024-09-21T13:49:39.836856+0000","last_became_active":"2024-09-21T13:49:04.431035+0000","last_became_peered":"2024-09-21T13:49:04.431035+0000","last_unstale":"2024-09-21T13:49:39.836856+0000","last_undegraded":"2024-09-21T13:49:39.836856+0000","last_fullsized":"2024-09-21T13:49:39.836856+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:03.519988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"68'12","reported_seq":60,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836677+0000","last_change":"2024-09-21T13:49:00.053466+0000","last_active":"2024-09-21T13:49:39.836677+0000","last_peered":"2024-09-21T13:49:39.836677+0000","last_clean":"2024-09-21T13:49:39.836677+0000","last_became_active":"2024-09-21T13:49:00.053080+0000","last_became_peered":"2024-09-21T13:49:00.053080+0000","last_unstale":"2024-09-21T13:49:39.836677+0000","last_undegraded":"2024-09-21T13:49:39.836677+0000","last_fullsized":"2024-09-21T13:49:39.836677+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:03.884409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835630+0000","last_change":"2024-09-21T13:48:58.051534+0000","last_active":"2024-09-21T13:49:39.835630+0000","last_peered":"2024-09-21T13:49:39.835630+0000","last_clean":"2024-09-21T13:49:39.835630+0000","last_became_active":"2024-09-21T13:48:58.051249+0000","last_became_peered":"2024-09-21T13:48:58.051249+0000","last_unstale":"2024-09-21T13:49:39.835630+0000","last_undegraded":"2024-09-21T13:49:39.835630+0000","last_fullsized":"2024-09-21T13:49:39.835630+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:04.856323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":107,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:51.955301+0000","last_change":"2024-09-21T13:49:00.426330+0000","last_active":"2024-09-21T13:49:51.955301+0000","last_peered":"2024-09-21T13:49:51.955301+0000","last_clean":"2024-09-21T13:49:51.955301+0000","last_became_active":"2024-09-21T13:48:57.046084+0000","last_became_peered":"2024-09-21T13:48:57.046084+0000","last_unstale":"2024-09-21T13:49:51.955301+0000","last_undegraded":"2024-09-21T13:49:51.955301+0000","last_fullsized":"2024-09-21T13:49:51.955301+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:48:56.029323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:08.036165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047048000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":50,"num_read_kb":45,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.837653+0000","last_change":"2024-09-21T13:49:02.066942+0000","last_active":"2024-09-21T13:49:39.837653+0000","last_peered":"2024-09-21T13:49:39.837653+0000","last_clean":"2024-09-21T13:49:39.837653+0000","last_became_active":"2024-09-21T13:49:02.066641+0000","last_became_peered":"2024-09-21T13:49:02.066641+0000","last_unstale":"2024-09-21T13:49:39.837653+0000","last_undegraded":"2024-09-21T13:49:39.837653+0000","last_fullsized":"2024-09-21T13:49:39.837653+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:22.324406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.808219+0000","last_change":"2024-09-21T13:49:04.078686+0000","last_active":"2024-09-21T13:49:39.808219+0000","last_peered":"2024-09-21T13:49:39.808219+0000","last_clean":"2024-09-21T13:49:39.808219+0000","last_became_active":"2024-09-21T13:49:04.078372+0000","last_became_peered":"2024-09-21T13:49:04.078372+0000","last_unstale":"2024-09-21T13:49:39.808219+0000","last_undegraded":"2024-09-21T13:49:39.808219+0000","last_fullsized":"2024-09-21T13:49:39.808219+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:25.621989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"68'16","reported_seq":76,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:48.114277+0000","last_change":"2024-09-21T13:49:00.427430+0000","last_active":"2024-09-21T13:49:48.114277+0000","last_peered":"2024-09-21T13:49:48.114277+0000","last_clean":"2024-09-21T13:49:48.114277+0000","last_became_active":"2024-09-21T13:49:00.426867+0000","last_became_peered":"2024-09-21T13:49:00.426867+0000","last_unstale":"2024-09-21T13:49:48.114277+0000","last_undegraded":"2024-09-21T13:49:48.114277+0000","last_fullsized":"2024-09-21T13:49:48.114277+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:11.516725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835899+0000","last_change":"2024-09-21T13:48:58.052925+0000","last_active":"2024-09-21T13:49:39.835899+0000","last_peered":"2024-09-21T13:49:39.835899+0000","last_clean":"2024-09-21T13:49:39.835899+0000","last_became_active":"2024-09-21T13:48:58.052659+0000","last_became_peered":"2024-09-21T13:48:58.052659+0000","last_unstale":"2024-09-21T13:49:39.835899+0000","last_undegraded":"2024-09-21T13:49:39.835899+0000","last_fullsized":"2024-09-21T13:49:39.835899+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:03.529879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"75'79","reported_seq":95,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836777+0000","last_change":"2024-09-21T13:48:51.893262+0000","last_active":"2024-09-21T13:49:39.836777+0000","last_peered":"2024-09-21T13:49:39.836777+0000","last_clean":"2024-09-21T13:49:39.836777+0000","last_became_active":"2024-09-21T13:48:48.454646+0000","last_became_peered":"2024-09-21T13:48:48.454646+0000","last_unstale":"2024-09-21T13:49:39.836777+0000","last_undegraded":"2024-09-21T13:49:39.836777+0000","last_fullsized":"2024-09-21T13:49:39.836777+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'72","last_scrub_stamp":"2024-09-21T13:48:51.893178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:48:51.893178+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:10.164272+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836466+0000","last_change":"2024-09-21T13:49:02.061930+0000","last_active":"2024-09-21T13:49:39.836466+0000","last_peered":"2024-09-21T13:49:39.836466+0000","last_clean":"2024-09-21T13:49:39.836466+0000","last_became_active":"2024-09-21T13:49:02.061454+0000","last_became_peered":"2024-09-21T13:49:02.061454+0000","last_unstale":"2024-09-21T13:49:39.836466+0000","last_undegraded":"2024-09-21T13:49:39.836466+0000","last_fullsized":"2024-09-21T13:49:39.836466+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:23.000364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.837469+0000","last_change":"2024-09-21T13:49:04.078343+0000","last_active":"2024-09-21T13:49:39.837469+0000","last_peered":"2024-09-21T13:49:39.837469+0000","last_clean":"2024-09-21T13:49:39.837469+0000","last_became_active":"2024-09-21T13:49:04.078091+0000","last_became_peered":"2024-09-21T13:49:04.078091+0000","last_unstale":"2024-09-21T13:49:39.837469+0000","last_undegraded":"2024-09-21T13:49:39.837469+0000","last_fullsized":"2024-09-21T13:49:39.837469+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:17.101054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"68'11","reported_seq":61,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.812470+0000","last_change":"2024-09-21T13:49:00.054827+0000","last_active":"2024-09-21T13:49:39.812470+0000","last_peered":"2024-09-21T13:49:39.812470+0000","last_clean":"2024-09-21T13:49:39.812470+0000","last_became_active":"2024-09-21T13:49:00.053268+0000","last_became_peered":"2024-09-21T13:49:00.053268+0000","last_unstale":"2024-09-21T13:49:39.812470+0000","last_undegraded":"2024-09-21T13:49:39.812470+0000","last_fullsized":"2024-09-21T13:49:39.812470+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:28.403874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.811957+0000","last_change":"2024-09-21T13:48:58.052364+0000","last_active":"2024-09-21T13:49:39.811957+0000","last_peered":"2024-09-21T13:49:39.811957+0000","last_clean":"2024-09-21T13:49:39.811957+0000","last_became_active":"2024-09-21T13:48:58.052092+0000","last_became_peered":"2024-09-21T13:48:58.052092+0000","last_unstale":"2024-09-21T13:49:39.811957+0000","last_undegraded":"2024-09-21T13:49:39.811957+0000","last_fullsized":"2024-09-21T13:49:39.811957+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:10.772378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.837459+0000","last_change":"2024-09-21T13:49:02.062262+0000","last_active":"2024-09-21T13:49:39.837459+0000","last_peered":"2024-09-21T13:49:39.837459+0000","last_clean":"2024-09-21T13:49:39.837459+0000","last_became_active":"2024-09-21T13:49:02.061901+0000","last_became_peered":"2024-09-21T13:49:02.061901+0000","last_unstale":"2024-09-21T13:49:39.837459+0000","last_undegraded":"2024-09-21T13:49:39.837459+0000","last_fullsized":"2024-09-21T13:49:39.837459+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:20.147536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836230+0000","last_change":"2024-09-21T13:49:04.431695+0000","last_active":"2024-09-21T13:49:39.836230+0000","last_peered":"2024-09-21T13:49:39.836230+0000","last_clean":"2024-09-21T13:49:39.836230+0000","last_became_active":"2024-09-21T13:49:04.431268+0000","last_became_peered":"2024-09-21T13:49:04.431268+0000","last_unstale":"2024-09-21T13:49:39.836230+0000","last_undegraded":"2024-09-21T13:49:39.836230+0000","last_fullsized":"2024-09-21T13:49:39.836230+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:27.118735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"68'17","reported_seq":70,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.812385+0000","last_change":"2024-09-21T13:49:00.053077+0000","last_active":"2024-09-21T13:49:39.812385+0000","last_peered":"2024-09-21T13:49:39.812385+0000","last_clean":"2024-09-21T13:49:39.812385+0000","last_became_active":"2024-09-21T13:49:00.052608+0000","last_became_peered":"2024-09-21T13:49:00.052608+0000","last_unstale":"2024-09-21T13:49:39.812385+0000","last_undegraded":"2024-09-21T13:49:39.812385+0000","last_fullsized":"2024-09-21T13:49:39.812385+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:14.266475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836235+0000","last_change":"2024-09-21T13:48:58.051751+0000","last_active":"2024-09-21T13:49:39.836235+0000","last_peered":"2024-09-21T13:49:39.836235+0000","last_clean":"2024-09-21T13:49:39.836235+0000","last_became_active":"2024-09-21T13:48:58.049142+0000","last_became_peered":"2024-09-21T13:48:58.049142+0000","last_unstale":"2024-09-21T13:49:39.836235+0000","last_undegraded":"2024-09-21T13:49:39.836235+0000","last_fullsized":"2024-09-21T13:49:39.836235+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:07.453552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.808232+0000","last_change":"2024-09-21T13:49:02.067811+0000","last_active":"2024-09-21T13:49:39.808232+0000","last_peered":"2024-09-21T13:49:39.808232+0000","last_clean":"2024-09-21T13:49:39.808232+0000","last_became_active":"2024-09-21T13:49:02.067472+0000","last_became_peered":"2024-09-21T13:49:02.067472+0000","last_unstale":"2024-09-21T13:49:39.808232+0000","last_undegraded":"2024-09-21T13:49:39.808232+0000","last_fullsized":"2024-09-21T13:49:39.808232+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:29.225485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.808029+0000","last_change":"2024-09-21T13:49:04.078285+0000","last_active":"2024-09-21T13:49:39.808029+0000","last_peered":"2024-09-21T13:49:39.808029+0000","last_clean":"2024-09-21T13:49:39.808029+0000","last_became_active":"2024-09-21T13:49:04.077962+0000","last_became_peered":"2024-09-21T13:49:04.077962+0000","last_unstale":"2024-09-21T13:49:39.808029+0000","last_undegraded":"2024-09-21T13:49:39.808029+0000","last_fullsized":"2024-09-21T13:49:39.808029+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:13.659310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"68'10","reported_seq":57,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.812106+0000","last_change":"2024-09-21T13:49:00.060182+0000","last_active":"2024-09-21T13:49:39.812106+0000","last_peered":"2024-09-21T13:49:39.812106+0000","last_clean":"2024-09-21T13:49:39.812106+0000","last_became_active":"2024-09-21T13:49:00.059850+0000","last_became_peered":"2024-09-21T13:49:00.059850+0000","last_unstale":"2024-09-21T13:49:39.812106+0000","last_undegraded":"2024-09-21T13:49:39.812106+0000","last_fullsized":"2024-09-21T13:49:39.812106+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:16.066823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836259+0000","last_change":"2024-09-21T13:48:58.058603+0000","last_active":"2024-09-21T13:49:39.836259+0000","last_peered":"2024-09-21T13:49:39.836259+0000","last_clean":"2024-09-21T13:49:39.836259+0000","last_became_active":"2024-09-21T13:48:58.058309+0000","last_became_peered":"2024-09-21T13:48:58.058309+0000","last_unstale":"2024-09-21T13:49:39.836259+0000","last_undegraded":"2024-09-21T13:49:39.836259+0000","last_fullsized":"2024-09-21T13:49:39.836259+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:09.079397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"68'8","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:48.114234+0000","last_change":"2024-09-21T13:49:02.065220+0000","last_active":"2024-09-21T13:49:48.114234+0000","last_peered":"2024-09-21T13:49:48.114234+0000","last_clean":"2024-09-21T13:49:48.114234+0000","last_became_active":"2024-09-21T13:49:02.064817+0000","last_became_peered":"2024-09-21T13:49:02.064817+0000","last_unstale":"2024-09-21T13:49:48.114234+0000","last_undegraded":"2024-09-21T13:49:48.114234+0000","last_fullsized":"2024-09-21T13:49:48.114234+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:11.912943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"68'1","reported_seq":35,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.812179+0000","last_change":"2024-09-21T13:49:04.072175+0000","last_active":"2024-09-21T13:49:39.812179+0000","last_peered":"2024-09-21T13:49:39.812179+0000","last_clean":"2024-09-21T13:49:39.812179+0000","last_became_active":"2024-09-21T13:49:04.071885+0000","last_became_peered":"2024-09-21T13:49:04.071885+0000","last_unstale":"2024-09-21T13:49:39.812179+0000","last_undegraded":"2024-09-21T13:49:39.812179+0000","last_fullsized":"2024-09-21T13:49:39.812179+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:03.670445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"68'9","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836079+0000","last_change":"2024-09-21T13:49:00.053349+0000","last_active":"2024-09-21T13:49:39.836079+0000","last_peered":"2024-09-21T13:49:39.836079+0000","last_clean":"2024-09-21T13:49:39.836079+0000","last_became_active":"2024-09-21T13:49:00.053016+0000","last_became_peered":"2024-09-21T13:49:00.053016+0000","last_unstale":"2024-09-21T13:49:39.836079+0000","last_undegraded":"2024-09-21T13:49:39.836079+0000","last_fullsized":"2024-09-21T13:49:39.836079+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:06.374913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836830+0000","last_change":"2024-09-21T13:48:58.050243+0000","last_active":"2024-09-21T13:49:39.836830+0000","last_peered":"2024-09-21T13:49:39.836830+0000","last_clean":"2024-09-21T13:49:39.836830+0000","last_became_active":"2024-09-21T13:48:58.049543+0000","last_became_peered":"2024-09-21T13:48:58.049543+0000","last_unstale":"2024-09-21T13:49:39.836830+0000","last_undegraded":"2024-09-21T13:49:39.836830+0000","last_fullsized":"2024-09-21T13:49:39.836830+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:17.729176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.807325+0000","last_change":"2024-09-21T13:49:02.063877+0000","last_active":"2024-09-21T13:49:39.807325+0000","last_peered":"2024-09-21T13:49:39.807325+0000","last_clean":"2024-09-21T13:49:39.807325+0000","last_became_active":"2024-09-21T13:49:02.063525+0000","last_became_peered":"2024-09-21T13:49:02.063525+0000","last_unstale":"2024-09-21T13:49:39.807325+0000","last_undegraded":"2024-09-21T13:49:39.807325+0000","last_fullsized":"2024-09-21T13:49:39.807325+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:30.535568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835588+0000","last_change":"2024-09-21T13:49:04.081924+0000","last_active":"2024-09-21T13:49:39.835588+0000","last_peered":"2024-09-21T13:49:39.835588+0000","last_clean":"2024-09-21T13:49:39.835588+0000","last_became_active":"2024-09-21T13:49:04.081617+0000","last_became_peered":"2024-09-21T13:49:04.081617+0000","last_unstale":"2024-09-21T13:49:39.835588+0000","last_undegraded":"2024-09-21T13:49:39.835588+0000","last_fullsized":"2024-09-21T13:49:39.835588+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:25.513722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"68'19","reported_seq":73,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835941+0000","last_change":"2024-09-21T13:49:00.426272+0000","last_active":"2024-09-21T13:49:39.835941+0000","last_peered":"2024-09-21T13:49:39.835941+0000","last_clean":"2024-09-21T13:49:39.835941+0000","last_became_active":"2024-09-21T13:49:00.425955+0000","last_became_peered":"2024-09-21T13:49:00.425955+0000","last_unstale":"2024-09-21T13:49:39.835941+0000","last_undegraded":"2024-09-21T13:49:39.835941+0000","last_fullsized":"2024-09-21T13:49:39.835941+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:02.788196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.837228+0000","last_change":"2024-09-21T13:48:58.053940+0000","last_active":"2024-09-21T13:49:39.837228+0000","last_peered":"2024-09-21T13:49:39.837228+0000","last_clean":"2024-09-21T13:49:39.837228+0000","last_became_active":"2024-09-21T13:48:58.053641+0000","last_became_peered":"2024-09-21T13:48:58.053641+0000","last_unstale":"2024-09-21T13:49:39.837228+0000","last_undegraded":"2024-09-21T13:49:39.837228+0000","last_fullsized":"2024-09-21T13:49:39.837228+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:25.094666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.807300+0000","last_change":"2024-09-21T13:49:02.066996+0000","last_active":"2024-09-21T13:49:39.807300+0000","last_peered":"2024-09-21T13:49:39.807300+0000","last_clean":"2024-09-21T13:49:39.807300+0000","last_became_active":"2024-09-21T13:49:02.066676+0000","last_became_peered":"2024-09-21T13:49:02.066676+0000","last_unstale":"2024-09-21T13:49:39.807300+0000","last_undegraded":"2024-09-21T13:49:39.807300+0000","last_fullsized":"2024-09-21T13:49:39.807300+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:19.241359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836031+0000","last_change":"2024-09-21T13:49:04.078412+0000","last_active":"2024-09-21T13:49:39.836031+0000","last_peered":"2024-09-21T13:49:39.836031+0000","last_clean":"2024-09-21T13:49:39.836031+0000","last_became_active":"2024-09-21T13:49:04.078084+0000","last_became_peered":"2024-09-21T13:49:04.078084+0000","last_unstale":"2024-09-21T13:49:39.836031+0000","last_undegraded":"2024-09-21T13:49:39.836031+0000","last_fullsized":"2024-09-21T13:49:39.836031+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:11.517342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"68'12","reported_seq":65,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.811928+0000","last_change":"2024-09-21T13:49:00.060188+0000","last_active":"2024-09-21T13:49:39.811928+0000","last_peered":"2024-09-21T13:49:39.811928+0000","last_clean":"2024-09-21T13:49:39.811928+0000","last_became_active":"2024-09-21T13:49:00.059720+0000","last_became_peered":"2024-09-21T13:49:00.059720+0000","last_unstale":"2024-09-21T13:49:39.811928+0000","last_undegraded":"2024-09-21T13:49:39.811928+0000","last_fullsized":"2024-09-21T13:49:39.811928+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:17.978182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.837628+0000","last_change":"2024-09-21T13:48:58.058654+0000","last_active":"2024-09-21T13:49:39.837628+0000","last_peered":"2024-09-21T13:49:39.837628+0000","last_clean":"2024-09-21T13:49:39.837628+0000","last_became_active":"2024-09-21T13:48:58.057641+0000","last_became_peered":"2024-09-21T13:48:58.057641+0000","last_unstale":"2024-09-21T13:49:39.837628+0000","last_undegraded":"2024-09-21T13:49:39.837628+0000","last_fullsized":"2024-09-21T13:49:39.837628+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:11.120209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.807118+0000","last_change":"2024-09-21T13:49:02.057568+0000","last_active":"2024-09-21T13:49:39.807118+0000","last_peered":"2024-09-21T13:49:39.807118+0000","last_clean":"2024-09-21T13:49:39.807118+0000","last_became_active":"2024-09-21T13:49:02.057195+0000","last_became_peered":"2024-09-21T13:49:02.057195+0000","last_unstale":"2024-09-21T13:49:39.807118+0000","last_undegraded":"2024-09-21T13:49:39.807118+0000","last_fullsized":"2024-09-21T13:49:39.807118+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:01.498483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836115+0000","last_change":"2024-09-21T13:49:04.078422+0000","last_active":"2024-09-21T13:49:39.836115+0000","last_peered":"2024-09-21T13:49:39.836115+0000","last_clean":"2024-09-21T13:49:39.836115+0000","last_became_active":"2024-09-21T13:49:04.078089+0000","last_became_peered":"2024-09-21T13:49:04.078089+0000","last_unstale":"2024-09-21T13:49:39.836115+0000","last_undegraded":"2024-09-21T13:49:39.836115+0000","last_fullsized":"2024-09-21T13:49:39.836115+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:18.974401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"68'15","reported_seq":67,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835435+0000","last_change":"2024-09-21T13:49:00.426358+0000","last_active":"2024-09-21T13:49:39.835435+0000","last_peered":"2024-09-21T13:49:39.835435+0000","last_clean":"2024-09-21T13:49:39.835435+0000","last_became_active":"2024-09-21T13:49:00.425997+0000","last_became_peered":"2024-09-21T13:49:00.425997+0000","last_unstale":"2024-09-21T13:49:39.835435+0000","last_undegraded":"2024-09-21T13:49:39.835435+0000","last_fullsized":"2024-09-21T13:49:39.835435+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:03.382004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.837692+0000","last_change":"2024-09-21T13:48:58.058778+0000","last_active":"2024-09-21T13:49:39.837692+0000","last_peered":"2024-09-21T13:49:39.837692+0000","last_clean":"2024-09-21T13:49:39.837692+0000","last_became_active":"2024-09-21T13:48:58.057632+0000","last_became_peered":"2024-09-21T13:48:58.057632+0000","last_unstale":"2024-09-21T13:49:39.837692+0000","last_undegraded":"2024-09-21T13:49:39.837692+0000","last_fullsized":"2024-09-21T13:49:39.837692+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:06.995889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"68'8","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:48.114491+0000","last_change":"2024-09-21T13:49:02.056471+0000","last_active":"2024-09-21T13:49:48.114491+0000","last_peered":"2024-09-21T13:49:48.114491+0000","last_clean":"2024-09-21T13:49:48.114491+0000","last_became_active":"2024-09-21T13:49:02.056167+0000","last_became_peered":"2024-09-21T13:49:02.056167+0000","last_unstale":"2024-09-21T13:49:48.114491+0000","last_undegraded":"2024-09-21T13:49:48.114491+0000","last_fullsized":"2024-09-21T13:49:48.114491+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:10.093140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835662+0000","last_change":"2024-09-21T13:49:04.432809+0000","last_active":"2024-09-21T13:49:39.835662+0000","last_peered":"2024-09-21T13:49:39.835662+0000","last_clean":"2024-09-21T13:49:39.835662+0000","last_became_active":"2024-09-21T13:49:04.432546+0000","last_became_peered":"2024-09-21T13:49:04.432546+0000","last_unstale":"2024-09-21T13:49:39.835662+0000","last_undegraded":"2024-09-21T13:49:39.835662+0000","last_fullsized":"2024-09-21T13:49:39.835662+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:08.569714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836980+0000","last_change":"2024-09-21T13:48:58.056383+0000","last_active":"2024-09-21T13:49:39.836980+0000","last_peered":"2024-09-21T13:49:39.836980+0000","last_clean":"2024-09-21T13:49:39.836980+0000","last_became_active":"2024-09-21T13:48:58.056080+0000","last_became_peered":"2024-09-21T13:48:58.056080+0000","last_unstale":"2024-09-21T13:49:39.836980+0000","last_undegraded":"2024-09-21T13:49:39.836980+0000","last_fullsized":"2024-09-21T13:49:39.836980+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:26.306822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"68'6","reported_seq":51,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836952+0000","last_change":"2024-09-21T13:49:00.427204+0000","last_active":"2024-09-21T13:49:39.836952+0000","last_peered":"2024-09-21T13:49:39.836952+0000","last_clean":"2024-09-21T13:49:39.836952+0000","last_became_active":"2024-09-21T13:49:00.426829+0000","last_became_peered":"2024-09-21T13:49:00.426829+0000","last_unstale":"2024-09-21T13:49:39.836952+0000","last_undegraded":"2024-09-21T13:49:39.836952+0000","last_fullsized":"2024-09-21T13:49:39.836952+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:20.014163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.837140+0000","last_change":"2024-09-21T13:49:02.062503+0000","last_active":"2024-09-21T13:49:39.837140+0000","last_peered":"2024-09-21T13:49:39.837140+0000","last_clean":"2024-09-21T13:49:39.837140+0000","last_became_active":"2024-09-21T13:49:02.062231+0000","last_became_peered":"2024-09-21T13:49:02.062231+0000","last_unstale":"2024-09-21T13:49:39.837140+0000","last_undegraded":"2024-09-21T13:49:39.837140+0000","last_fullsized":"2024-09-21T13:49:39.837140+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:21.288932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836220+0000","last_change":"2024-09-21T13:49:04.432777+0000","last_active":"2024-09-21T13:49:39.836220+0000","last_peered":"2024-09-21T13:49:39.836220+0000","last_clean":"2024-09-21T13:49:39.836220+0000","last_became_active":"2024-09-21T13:49:04.432507+0000","last_became_peered":"2024-09-21T13:49:04.432507+0000","last_unstale":"2024-09-21T13:49:39.836220+0000","last_undegraded":"2024-09-21T13:49:39.836220+0000","last_fullsized":"2024-09-21T13:49:39.836220+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:03.727186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"68'9","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836720+0000","last_change":"2024-09-21T13:49:00.428704+0000","last_active":"2024-09-21T13:49:39.836720+0000","last_peered":"2024-09-21T13:49:39.836720+0000","last_clean":"2024-09-21T13:49:39.836720+0000","last_became_active":"2024-09-21T13:49:00.428330+0000","last_became_peered":"2024-09-21T13:49:00.428330+0000","last_unstale":"2024-09-21T13:49:39.836720+0000","last_undegraded":"2024-09-21T13:49:39.836720+0000","last_fullsized":"2024-09-21T13:49:39.836720+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:06.325729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"61'1","reported_seq":52,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835958+0000","last_change":"2024-09-21T13:48:58.050948+0000","last_active":"2024-09-21T13:49:39.835958+0000","last_peered":"2024-09-21T13:49:39.835958+0000","last_clean":"2024-09-21T13:49:39.835958+0000","last_became_active":"2024-09-21T13:48:58.050645+0000","last_became_peered":"2024-09-21T13:48:58.050645+0000","last_unstale":"2024-09-21T13:49:39.835958+0000","last_undegraded":"2024-09-21T13:49:39.835958+0000","last_fullsized":"2024-09-21T13:49:39.835958+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:24.159998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836864+0000","last_change":"2024-09-21T13:49:02.062075+0000","last_active":"2024-09-21T13:49:39.836864+0000","last_peered":"2024-09-21T13:49:39.836864+0000","last_clean":"2024-09-21T13:49:39.836864+0000","last_became_active":"2024-09-21T13:49:02.061775+0000","last_became_peered":"2024-09-21T13:49:02.061775+0000","last_unstale":"2024-09-21T13:49:39.836864+0000","last_undegraded":"2024-09-21T13:49:39.836864+0000","last_fullsized":"2024-09-21T13:49:39.836864+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:16.428002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.807270+0000","last_change":"2024-09-21T13:49:04.069452+0000","last_active":"2024-09-21T13:49:39.807270+0000","last_peered":"2024-09-21T13:49:39.807270+0000","last_clean":"2024-09-21T13:49:39.807270+0000","last_became_active":"2024-09-21T13:49:04.069142+0000","last_became_peered":"2024-09-21T13:49:04.069142+0000","last_unstale":"2024-09-21T13:49:39.807270+0000","last_undegraded":"2024-09-21T13:49:39.807270+0000","last_fullsized":"2024-09-21T13:49:39.807270+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:28.894052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"68'9","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.837072+0000","last_change":"2024-09-21T13:49:00.428080+0000","last_active":"2024-09-21T13:49:39.837072+0000","last_peered":"2024-09-21T13:49:39.837072+0000","last_clean":"2024-09-21T13:49:39.837072+0000","last_became_active":"2024-09-21T13:49:00.427766+0000","last_became_peered":"2024-09-21T13:49:00.427766+0000","last_unstale":"2024-09-21T13:49:39.837072+0000","last_undegraded":"2024-09-21T13:49:39.837072+0000","last_fullsized":"2024-09-21T13:49:39.837072+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:16.384643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836693+0000","last_change":"2024-09-21T13:48:58.052465+0000","last_active":"2024-09-21T13:49:39.836693+0000","last_peered":"2024-09-21T13:49:39.836693+0000","last_clean":"2024-09-21T13:49:39.836693+0000","last_became_active":"2024-09-21T13:48:58.052151+0000","last_became_peered":"2024-09-21T13:48:58.052151+0000","last_unstale":"2024-09-21T13:49:39.836693+0000","last_undegraded":"2024-09-21T13:49:39.836693+0000","last_fullsized":"2024-09-21T13:49:39.836693+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:09.226461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"68'8","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:48.114142+0000","last_change":"2024-09-21T13:49:02.066442+0000","last_active":"2024-09-21T13:49:48.114142+0000","last_peered":"2024-09-21T13:49:48.114142+0000","last_clean":"2024-09-21T13:49:48.114142+0000","last_became_active":"2024-09-21T13:49:02.065602+0000","last_became_peered":"2024-09-21T13:49:02.065602+0000","last_unstale":"2024-09-21T13:49:48.114142+0000","last_undegraded":"2024-09-21T13:49:48.114142+0000","last_fullsized":"2024-09-21T13:49:48.114142+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:10.855295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.812396+0000","last_change":"2024-09-21T13:49:04.073461+0000","last_active":"2024-09-21T13:49:39.812396+0000","last_peered":"2024-09-21T13:49:39.812396+0000","last_clean":"2024-09-21T13:49:39.812396+0000","last_became_active":"2024-09-21T13:49:04.073188+0000","last_became_peered":"2024-09-21T13:49:04.073188+0000","last_unstale":"2024-09-21T13:49:39.812396+0000","last_undegraded":"2024-09-21T13:49:39.812396+0000","last_fullsized":"2024-09-21T13:49:39.812396+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:08.749766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"68'10","reported_seq":57,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836534+0000","last_change":"2024-09-21T13:49:00.427262+0000","last_active":"2024-09-21T13:49:39.836534+0000","last_peered":"2024-09-21T13:49:39.836534+0000","last_clean":"2024-09-21T13:49:39.836534+0000","last_became_active":"2024-09-21T13:49:00.426071+0000","last_became_peered":"2024-09-21T13:49:00.426071+0000","last_unstale":"2024-09-21T13:49:39.836534+0000","last_undegraded":"2024-09-21T13:49:39.836534+0000","last_fullsized":"2024-09-21T13:49:39.836534+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:03.592724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836695+0000","last_change":"2024-09-21T13:48:58.058198+0000","last_active":"2024-09-21T13:49:39.836695+0000","last_peered":"2024-09-21T13:49:39.836695+0000","last_clean":"2024-09-21T13:49:39.836695+0000","last_became_active":"2024-09-21T13:48:58.057587+0000","last_became_peered":"2024-09-21T13:48:58.057587+0000","last_unstale":"2024-09-21T13:49:39.836695+0000","last_undegraded":"2024-09-21T13:49:39.836695+0000","last_fullsized":"2024-09-21T13:49:39.836695+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:19.518354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"68'8","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:48.114248+0000","last_change":"2024-09-21T13:49:02.067827+0000","last_active":"2024-09-21T13:49:48.114248+0000","last_peered":"2024-09-21T13:49:48.114248+0000","last_clean":"2024-09-21T13:49:48.114248+0000","last_became_active":"2024-09-21T13:49:02.067487+0000","last_became_peered":"2024-09-21T13:49:02.067487+0000","last_unstale":"2024-09-21T13:49:48.114248+0000","last_undegraded":"2024-09-21T13:49:48.114248+0000","last_fullsized":"2024-09-21T13:49:48.114248+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:23.003843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836447+0000","last_change":"2024-09-21T13:49:04.081928+0000","last_active":"2024-09-21T13:49:39.836447+0000","last_peered":"2024-09-21T13:49:39.836447+0000","last_clean":"2024-09-21T13:49:39.836447+0000","last_became_active":"2024-09-21T13:49:04.081609+0000","last_became_peered":"2024-09-21T13:49:04.081609+0000","last_unstale":"2024-09-21T13:49:39.836447+0000","last_undegraded":"2024-09-21T13:49:39.836447+0000","last_fullsized":"2024-09-21T13:49:39.836447+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:31.380912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"68'11","reported_seq":61,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.811729+0000","last_change":"2024-09-21T13:49:00.055420+0000","last_active":"2024-09-21T13:49:39.811729+0000","last_peered":"2024-09-21T13:49:39.811729+0000","last_clean":"2024-09-21T13:49:39.811729+0000","last_became_active":"2024-09-21T13:49:00.055024+0000","last_became_peered":"2024-09-21T13:49:00.055024+0000","last_unstale":"2024-09-21T13:49:39.811729+0000","last_undegraded":"2024-09-21T13:49:39.811729+0000","last_fullsized":"2024-09-21T13:49:39.811729+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:02.449612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.812476+0000","last_change":"2024-09-21T13:48:58.051845+0000","last_active":"2024-09-21T13:49:39.812476+0000","last_peered":"2024-09-21T13:49:39.812476+0000","last_clean":"2024-09-21T13:49:39.812476+0000","last_became_active":"2024-09-21T13:48:58.051541+0000","last_became_peered":"2024-09-21T13:48:58.051541+0000","last_unstale":"2024-09-21T13:49:39.812476+0000","last_undegraded":"2024-09-21T13:49:39.812476+0000","last_fullsized":"2024-09-21T13:49:39.812476+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:03.712482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.806822+0000","last_change":"2024-09-21T13:49:02.063328+0000","last_active":"2024-09-21T13:49:39.806822+0000","last_peered":"2024-09-21T13:49:39.806822+0000","last_clean":"2024-09-21T13:49:39.806822+0000","last_became_active":"2024-09-21T13:49:02.062868+0000","last_became_peered":"2024-09-21T13:49:02.062868+0000","last_unstale":"2024-09-21T13:49:39.806822+0000","last_undegraded":"2024-09-21T13:49:39.806822+0000","last_fullsized":"2024-09-21T13:49:39.806822+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:12.568168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835473+0000","last_change":"2024-09-21T13:49:04.083254+0000","last_active":"2024-09-21T13:49:39.835473+0000","last_peered":"2024-09-21T13:49:39.835473+0000","last_clean":"2024-09-21T13:49:39.835473+0000","last_became_active":"2024-09-21T13:49:04.082961+0000","last_became_peered":"2024-09-21T13:49:04.082961+0000","last_unstale":"2024-09-21T13:49:39.835473+0000","last_undegraded":"2024-09-21T13:49:39.835473+0000","last_fullsized":"2024-09-21T13:49:39.835473+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:07.548695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"68'9","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.807587+0000","last_change":"2024-09-21T13:49:00.057817+0000","last_active":"2024-09-21T13:49:39.807587+0000","last_peered":"2024-09-21T13:49:39.807587+0000","last_clean":"2024-09-21T13:49:39.807587+0000","last_became_active":"2024-09-21T13:49:00.057392+0000","last_became_peered":"2024-09-21T13:49:00.057392+0000","last_unstale":"2024-09-21T13:49:39.807587+0000","last_undegraded":"2024-09-21T13:49:39.807587+0000","last_fullsized":"2024-09-21T13:49:39.807587+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:02.682820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.837162+0000","last_change":"2024-09-21T13:48:58.058032+0000","last_active":"2024-09-21T13:49:39.837162+0000","last_peered":"2024-09-21T13:49:39.837162+0000","last_clean":"2024-09-21T13:49:39.837162+0000","last_became_active":"2024-09-21T13:48:58.057516+0000","last_became_peered":"2024-09-21T13:48:58.057516+0000","last_unstale":"2024-09-21T13:49:39.837162+0000","last_undegraded":"2024-09-21T13:49:39.837162+0000","last_fullsized":"2024-09-21T13:49:39.837162+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:20.885486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836103+0000","last_change":"2024-09-21T13:49:02.072264+0000","last_active":"2024-09-21T13:49:39.836103+0000","last_peered":"2024-09-21T13:49:39.836103+0000","last_clean":"2024-09-21T13:49:39.836103+0000","last_became_active":"2024-09-21T13:49:02.071922+0000","last_became_peered":"2024-09-21T13:49:02.071922+0000","last_unstale":"2024-09-21T13:49:39.836103+0000","last_undegraded":"2024-09-21T13:49:39.836103+0000","last_fullsized":"2024-09-21T13:49:39.836103+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:12.773964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835233+0000","last_change":"2024-09-21T13:49:04.080300+0000","last_active":"2024-09-21T13:49:39.835233+0000","last_peered":"2024-09-21T13:49:39.835233+0000","last_clean":"2024-09-21T13:49:39.835233+0000","last_became_active":"2024-09-21T13:49:04.079950+0000","last_became_peered":"2024-09-21T13:49:04.079950+0000","last_unstale":"2024-09-21T13:49:39.835233+0000","last_undegraded":"2024-09-21T13:49:39.835233+0000","last_fullsized":"2024-09-21T13:49:39.835233+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:20.474523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"68'11","reported_seq":61,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835652+0000","last_change":"2024-09-21T13:49:00.426487+0000","last_active":"2024-09-21T13:49:39.835652+0000","last_peered":"2024-09-21T13:49:39.835652+0000","last_clean":"2024-09-21T13:49:39.835652+0000","last_became_active":"2024-09-21T13:49:00.426091+0000","last_became_peered":"2024-09-21T13:49:00.426091+0000","last_unstale":"2024-09-21T13:49:39.835652+0000","last_undegraded":"2024-09-21T13:49:39.835652+0000","last_fullsized":"2024-09-21T13:49:39.835652+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:11.657590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.837231+0000","last_change":"2024-09-21T13:48:58.052281+0000","last_active":"2024-09-21T13:49:39.837231+0000","last_peered":"2024-09-21T13:49:39.837231+0000","last_clean":"2024-09-21T13:49:39.837231+0000","last_became_active":"2024-09-21T13:48:58.051974+0000","last_became_peered":"2024-09-21T13:48:58.051974+0000","last_unstale":"2024-09-21T13:49:39.837231+0000","last_undegraded":"2024-09-21T13:49:39.837231+0000","last_fullsized":"2024-09-21T13:49:39.837231+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:25.614472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835533+0000","last_change":"2024-09-21T13:49:02.056666+0000","last_active":"2024-09-21T13:49:39.835533+0000","last_peered":"2024-09-21T13:49:39.835533+0000","last_clean":"2024-09-21T13:49:39.835533+0000","last_became_active":"2024-09-21T13:49:02.056006+0000","last_became_peered":"2024-09-21T13:49:02.056006+0000","last_unstale":"2024-09-21T13:49:39.835533+0000","last_undegraded":"2024-09-21T13:49:39.835533+0000","last_fullsized":"2024-09-21T13:49:39.835533+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:17.511468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836235+0000","last_change":"2024-09-21T13:49:04.434380+0000","last_active":"2024-09-21T13:49:39.836235+0000","last_peered":"2024-09-21T13:49:39.836235+0000","last_clean":"2024-09-21T13:49:39.836235+0000","last_became_active":"2024-09-21T13:49:04.434034+0000","last_became_peered":"2024-09-21T13:49:04.434034+0000","last_unstale":"2024-09-21T13:49:39.836235+0000","last_undegraded":"2024-09-21T13:49:39.836235+0000","last_fullsized":"2024-09-21T13:49:39.836235+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:28.547927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"68'4","reported_seq":48,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835309+0000","last_change":"2024-09-21T13:49:00.063374+0000","last_active":"2024-09-21T13:49:39.835309+0000","last_peered":"2024-09-21T13:49:39.835309+0000","last_clean":"2024-09-21T13:49:39.835309+0000","last_became_active":"2024-09-21T13:49:00.063066+0000","last_became_peered":"2024-09-21T13:49:00.063066+0000","last_unstale":"2024-09-21T13:49:39.835309+0000","last_undegraded":"2024-09-21T13:49:39.835309+0000","last_fullsized":"2024-09-21T13:49:39.835309+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:19.893798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836232+0000","last_change":"2024-09-21T13:48:58.059979+0000","last_active":"2024-09-21T13:49:39.836232+0000","last_peered":"2024-09-21T13:49:39.836232+0000","last_clean":"2024-09-21T13:49:39.836232+0000","last_became_active":"2024-09-21T13:48:58.059674+0000","last_became_peered":"2024-09-21T13:48:58.059674+0000","last_unstale":"2024-09-21T13:49:39.836232+0000","last_undegraded":"2024-09-21T13:49:39.836232+0000","last_fullsized":"2024-09-21T13:49:39.836232+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:02.308942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835914+0000","last_change":"2024-09-21T13:49:02.065043+0000","last_active":"2024-09-21T13:49:39.835914+0000","last_peered":"2024-09-21T13:49:39.835914+0000","last_clean":"2024-09-21T13:49:39.835914+0000","last_became_active":"2024-09-21T13:49:02.064789+0000","last_became_peered":"2024-09-21T13:49:02.064789+0000","last_unstale":"2024-09-21T13:49:39.835914+0000","last_undegraded":"2024-09-21T13:49:39.835914+0000","last_fullsized":"2024-09-21T13:49:39.835914+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:10.569664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835522+0000","last_change":"2024-09-21T13:49:04.075170+0000","last_active":"2024-09-21T13:49:39.835522+0000","last_peered":"2024-09-21T13:49:39.835522+0000","last_clean":"2024-09-21T13:49:39.835522+0000","last_became_active":"2024-09-21T13:49:04.074857+0000","last_became_peered":"2024-09-21T13:49:04.074857+0000","last_unstale":"2024-09-21T13:49:39.835522+0000","last_undegraded":"2024-09-21T13:49:39.835522+0000","last_fullsized":"2024-09-21T13:49:39.835522+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:17.971950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.808102+0000","last_change":"2024-09-21T13:49:04.075860+0000","last_active":"2024-09-21T13:49:39.808102+0000","last_peered":"2024-09-21T13:49:39.808102+0000","last_clean":"2024-09-21T13:49:39.808102+0000","last_became_active":"2024-09-21T13:49:04.075562+0000","last_became_peered":"2024-09-21T13:49:04.075562+0000","last_unstale":"2024-09-21T13:49:39.808102+0000","last_undegraded":"2024-09-21T13:49:39.808102+0000","last_fullsized":"2024-09-21T13:49:39.808102+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:21.041626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835570+0000","last_change":"2024-09-21T13:48:58.055791+0000","last_active":"2024-09-21T13:49:39.835570+0000","last_peered":"2024-09-21T13:49:39.835570+0000","last_clean":"2024-09-21T13:49:39.835570+0000","last_became_active":"2024-09-21T13:48:58.055497+0000","last_became_peered":"2024-09-21T13:48:58.055497+0000","last_unstale":"2024-09-21T13:49:39.835570+0000","last_undegraded":"2024-09-21T13:49:39.835570+0000","last_fullsized":"2024-09-21T13:49:39.835570+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:19.570880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"68'11","reported_seq":61,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836752+0000","last_change":"2024-09-21T13:49:00.058623+0000","last_active":"2024-09-21T13:49:39.836752+0000","last_peered":"2024-09-21T13:49:39.836752+0000","last_clean":"2024-09-21T13:49:39.836752+0000","last_became_active":"2024-09-21T13:49:00.058259+0000","last_became_peered":"2024-09-21T13:49:00.058259+0000","last_unstale":"2024-09-21T13:49:39.836752+0000","last_undegraded":"2024-09-21T13:49:39.836752+0000","last_fullsized":"2024-09-21T13:49:39.836752+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:17.663571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836553+0000","last_change":"2024-09-21T13:49:02.068530+0000","last_active":"2024-09-21T13:49:39.836553+0000","last_peered":"2024-09-21T13:49:39.836553+0000","last_clean":"2024-09-21T13:49:39.836553+0000","last_became_active":"2024-09-21T13:49:02.066159+0000","last_became_peered":"2024-09-21T13:49:02.066159+0000","last_unstale":"2024-09-21T13:49:39.836553+0000","last_undegraded":"2024-09-21T13:49:39.836553+0000","last_fullsized":"2024-09-21T13:49:39.836553+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:29.718157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":50,"num_read_kb":45,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234000,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32628,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712524,"statfs":{"total":95995035648,"available":95961624576,"internally_reserved":0,"allocated":1474560,"data_stored":970453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462933,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28448,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716704,"statfs":{"total":95995035648,"available":95965904896,"internally_reserved":0,"allocated":1454080,"data_stored":969368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724572,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27872,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717280,"statfs":{"total":95995035648,"available":95966494720,"internally_reserved":0,"allocated":864256,"data_stored":379916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920802,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27900,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717252,"statfs":{"total":95995035648,"available":95966466048,"internally_reserved":0,"allocated":892928,"data_stored":379618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117031,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27904,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717248,"statfs":{"total":95995035648,"available":95966461952,"internally_reserved":0,"allocated":897024,"data_stored":380448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411372,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32028,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713124,"statfs":{"total":95995035648,"available":95962238976,"internally_reserved":0,"allocated":860160,"data_stored":379718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574900,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32092,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713060,"statfs":{"total":95995035648,"available":95962173440,"internally_reserved":0,"allocated":925696,"data_stored":380362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738426,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28468,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716684,"statfs":{"total":95995035648,"available":95965884416,"internally_reserved":0,"allocated":1474560,"data_stored":969726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T13:49:56.258 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T13:49:56.505 INFO:journalctl@ceph.alertmanager.a.smithi049.stdout:Sep 21 13:49:56 smithi049 podman[74257]: 2024-09-21 13:49:56.35509182 +0000 UTC m=+0.016656325 image pull 11f11916f8cdfb17c1da74b20deb2075d778357b4d60474d82453303f8d461f5 quay.io/prometheus/alertmanager:v0.27.0 2024-09-21T13:49:56.863 INFO:journalctl@ceph.alertmanager.a.smithi049.stdout:Sep 21 13:49:56 smithi049 podman[74257]: 2024-09-21 13:49:56.543702292 +0000 UTC m=+0.205267364 container create 3529245bd961feeb62033134ae8a2e0a6d8b0fcef0af93b943484e6b78cedb09 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T13:49:57.064 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:49:57.192 INFO:journalctl@ceph.alertmanager.a.smithi049.stdout:Sep 21 13:49:56 smithi049 podman[74257]: 2024-09-21 13:49:56.86347595 +0000 UTC m=+0.525040450 container init 3529245bd961feeb62033134ae8a2e0a6d8b0fcef0af93b943484e6b78cedb09 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T13:49:57.192 INFO:journalctl@ceph.alertmanager.a.smithi049.stdout:Sep 21 13:49:56 smithi049 podman[74257]: 2024-09-21 13:49:56.868589047 +0000 UTC m=+0.530153543 container start 3529245bd961feeb62033134ae8a2e0a6d8b0fcef0af93b943484e6b78cedb09 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T13:49:57.192 INFO:journalctl@ceph.alertmanager.a.smithi049.stdout:Sep 21 13:49:56 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-alertmanager-a[74283]: ts=2024-09-21T13:49:56.880Z caller=main.go:181 level=info msg="Starting Alertmanager" version="(version=0.27.0, branch=HEAD, revision=0aa3c2aad14cff039931923ab16b26b7481783b5)" 2024-09-21T13:49:57.192 INFO:journalctl@ceph.alertmanager.a.smithi049.stdout:Sep 21 13:49:56 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-alertmanager-a[74283]: ts=2024-09-21T13:49:56.880Z caller=main.go:182 level=info build_context="(go=go1.21.7, platform=linux/amd64, user=root@22cd11f671e9, date=20240228-11:51:20, tags=netgo)" 2024-09-21T13:49:57.192 INFO:journalctl@ceph.alertmanager.a.smithi049.stdout:Sep 21 13:49:56 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-alertmanager-a[74283]: ts=2024-09-21T13:49:56.881Z caller=cluster.go:186 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.49 port=9094 2024-09-21T13:49:57.192 INFO:journalctl@ceph.alertmanager.a.smithi049.stdout:Sep 21 13:49:56 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-alertmanager-a[74283]: ts=2024-09-21T13:49:56.882Z caller=cluster.go:683 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-21T13:49:57.192 INFO:journalctl@ceph.alertmanager.a.smithi049.stdout:Sep 21 13:49:56 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-alertmanager-a[74283]: ts=2024-09-21T13:49:56.903Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-21T13:49:57.193 INFO:journalctl@ceph.alertmanager.a.smithi049.stdout:Sep 21 13:49:56 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-alertmanager-a[74283]: ts=2024-09-21T13:49:56.904Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-21T13:49:57.193 INFO:journalctl@ceph.alertmanager.a.smithi049.stdout:Sep 21 13:49:56 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-alertmanager-a[74283]: ts=2024-09-21T13:49:56.905Z caller=tls_config.go:313 level=info msg="Listening on" address=[::]:9093 2024-09-21T13:49:57.193 INFO:journalctl@ceph.alertmanager.a.smithi049.stdout:Sep 21 13:49:56 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-alertmanager-a[74283]: ts=2024-09-21T13:49:56.906Z caller=tls_config.go:316 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-21T13:49:57.193 INFO:journalctl@ceph.alertmanager.a.smithi049.stdout:Sep 21 13:49:56 smithi049 bash[74257]: 3529245bd961feeb62033134ae8a2e0a6d8b0fcef0af93b943484e6b78cedb09 2024-09-21T13:49:57.193 INFO:journalctl@ceph.alertmanager.a.smithi049.stdout:Sep 21 13:49:56 smithi049 systemd[1]: Started Ceph alertmanager.a for 40a3d2e8-781f-11ef-baf6-efdc52797490. 2024-09-21T13:49:57.747 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:49:57.750 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-09-21T13:49:58.346 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:58 smithi049 ceph-mon[40489]: 2.1 scrub starts 2024-09-21T13:49:58.346 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:58 smithi049 ceph-mon[40489]: 2.1 scrub ok 2024-09-21T13:49:58.346 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:58 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:58.346 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:58 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:58.346 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:58 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:58.346 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:58 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:58.346 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:58 smithi049 ceph-mon[40489]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-21T13:49:58.346 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:58 smithi049 ceph-mon[40489]: pgmap v23: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-21T13:49:58.346 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:58 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:58.347 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:58 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:49:58.347 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:58 smithi049 ceph-mon[30080]: 2.1 scrub starts 2024-09-21T13:49:58.347 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:58 smithi049 ceph-mon[30080]: 2.1 scrub ok 2024-09-21T13:49:58.347 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:58 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:58.347 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:58 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:58.347 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:58 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:58.347 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:58 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:58.347 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:58 smithi049 ceph-mon[30080]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-21T13:49:58.347 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:58 smithi049 ceph-mon[30080]: pgmap v23: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-21T13:49:58.347 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:58 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:58.347 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:58 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:49:58.348 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:49:58 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: [21/Sep/2024:13:49:58] ENGINE Bus STOPPING 2024-09-21T13:49:58.348 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:49:58 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: [21/Sep/2024:13:49:58] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T13:49:58.348 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:49:58 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: [21/Sep/2024:13:49:58] ENGINE Bus STOPPED 2024-09-21T13:49:58.348 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:49:58 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: [21/Sep/2024:13:49:58] ENGINE Bus STARTING 2024-09-21T13:49:58.348 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:49:58 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: [21/Sep/2024:13:49:58] ENGINE Serving on http://:::9283 2024-09-21T13:49:58.348 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:49:58 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: [21/Sep/2024:13:49:58] ENGINE Bus STARTED 2024-09-21T13:49:58.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:58 smithi187 ceph-mon[33032]: 2.1 scrub starts 2024-09-21T13:49:58.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:58 smithi187 ceph-mon[33032]: 2.1 scrub ok 2024-09-21T13:49:58.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:58 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:58.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:58 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:58.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:58 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:58.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:58 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:58.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:58 smithi187 ceph-mon[33032]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-21T13:49:58.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:58 smithi187 ceph-mon[33032]: pgmap v23: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-21T13:49:58.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:58 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:58.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:58 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:49:58.669 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-09-21T13:49:57.335346+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":898,"num_read_kb":755,"num_write":549,"num_write_kb":1673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":552,"ondisk_log_size":552,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":237340,"kb_used_data":8636,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":749723876,"statfs":{"total":767960285184,"available":767717249024,"internally_reserved":0,"allocated":8843264,"data_stored":4809609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001604"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836141+0000","last_change":"2024-09-21T13:48:58.061797+0000","last_active":"2024-09-21T13:49:39.836141+0000","last_peered":"2024-09-21T13:49:39.836141+0000","last_clean":"2024-09-21T13:49:39.836141+0000","last_became_active":"2024-09-21T13:48:58.061415+0000","last_became_peered":"2024-09-21T13:48:58.061415+0000","last_unstale":"2024-09-21T13:49:39.836141+0000","last_undegraded":"2024-09-21T13:49:39.836141+0000","last_fullsized":"2024-09-21T13:49:39.836141+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:07.025619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"68'9","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.811507+0000","last_change":"2024-09-21T13:49:00.055717+0000","last_active":"2024-09-21T13:49:39.811507+0000","last_peered":"2024-09-21T13:49:39.811507+0000","last_clean":"2024-09-21T13:49:39.811507+0000","last_became_active":"2024-09-21T13:49:00.055210+0000","last_became_peered":"2024-09-21T13:49:00.055210+0000","last_unstale":"2024-09-21T13:49:39.811507+0000","last_undegraded":"2024-09-21T13:49:39.811507+0000","last_fullsized":"2024-09-21T13:49:39.811507+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:13.541017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.806595+0000","last_change":"2024-09-21T13:49:02.061394+0000","last_active":"2024-09-21T13:49:39.806595+0000","last_peered":"2024-09-21T13:49:39.806595+0000","last_clean":"2024-09-21T13:49:39.806595+0000","last_became_active":"2024-09-21T13:49:02.060984+0000","last_became_peered":"2024-09-21T13:49:02.060984+0000","last_unstale":"2024-09-21T13:49:39.806595+0000","last_undegraded":"2024-09-21T13:49:39.806595+0000","last_fullsized":"2024-09-21T13:49:39.806595+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:18.936236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.811533+0000","last_change":"2024-09-21T13:49:04.072734+0000","last_active":"2024-09-21T13:49:39.811533+0000","last_peered":"2024-09-21T13:49:39.811533+0000","last_clean":"2024-09-21T13:49:39.811533+0000","last_became_active":"2024-09-21T13:49:04.072460+0000","last_became_peered":"2024-09-21T13:49:04.072460+0000","last_unstale":"2024-09-21T13:49:39.811533+0000","last_undegraded":"2024-09-21T13:49:39.811533+0000","last_fullsized":"2024-09-21T13:49:39.811533+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:30.526897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836010+0000","last_change":"2024-09-21T13:49:04.434312+0000","last_active":"2024-09-21T13:49:39.836010+0000","last_peered":"2024-09-21T13:49:39.836010+0000","last_clean":"2024-09-21T13:49:39.836010+0000","last_became_active":"2024-09-21T13:49:04.433965+0000","last_became_peered":"2024-09-21T13:49:04.433965+0000","last_unstale":"2024-09-21T13:49:39.836010+0000","last_undegraded":"2024-09-21T13:49:39.836010+0000","last_fullsized":"2024-09-21T13:49:39.836010+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:06.082699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836648+0000","last_change":"2024-09-21T13:48:58.051225+0000","last_active":"2024-09-21T13:49:39.836648+0000","last_peered":"2024-09-21T13:49:39.836648+0000","last_clean":"2024-09-21T13:49:39.836648+0000","last_became_active":"2024-09-21T13:48:58.050479+0000","last_became_peered":"2024-09-21T13:48:58.050479+0000","last_unstale":"2024-09-21T13:49:39.836648+0000","last_undegraded":"2024-09-21T13:49:39.836648+0000","last_fullsized":"2024-09-21T13:49:39.836648+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:19.515469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"68'15","reported_seq":67,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835920+0000","last_change":"2024-09-21T13:49:00.063787+0000","last_active":"2024-09-21T13:49:39.835920+0000","last_peered":"2024-09-21T13:49:39.835920+0000","last_clean":"2024-09-21T13:49:39.835920+0000","last_became_active":"2024-09-21T13:49:00.063460+0000","last_became_peered":"2024-09-21T13:49:00.063460+0000","last_unstale":"2024-09-21T13:49:39.835920+0000","last_undegraded":"2024-09-21T13:49:39.835920+0000","last_fullsized":"2024-09-21T13:49:39.835920+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:14.007409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.811464+0000","last_change":"2024-09-21T13:49:02.064007+0000","last_active":"2024-09-21T13:49:39.811464+0000","last_peered":"2024-09-21T13:49:39.811464+0000","last_clean":"2024-09-21T13:49:39.811464+0000","last_became_active":"2024-09-21T13:49:02.063687+0000","last_became_peered":"2024-09-21T13:49:02.063687+0000","last_unstale":"2024-09-21T13:49:39.811464+0000","last_undegraded":"2024-09-21T13:49:39.811464+0000","last_fullsized":"2024-09-21T13:49:39.811464+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:05.421236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.837174+0000","last_change":"2024-09-21T13:48:58.051001+0000","last_active":"2024-09-21T13:49:39.837174+0000","last_peered":"2024-09-21T13:49:39.837174+0000","last_clean":"2024-09-21T13:49:39.837174+0000","last_became_active":"2024-09-21T13:48:58.050695+0000","last_became_peered":"2024-09-21T13:48:58.050695+0000","last_unstale":"2024-09-21T13:49:39.837174+0000","last_undegraded":"2024-09-21T13:49:39.837174+0000","last_fullsized":"2024-09-21T13:49:39.837174+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:09.186887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"68'9","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.811774+0000","last_change":"2024-09-21T13:49:00.059748+0000","last_active":"2024-09-21T13:49:39.811774+0000","last_peered":"2024-09-21T13:49:39.811774+0000","last_clean":"2024-09-21T13:49:39.811774+0000","last_became_active":"2024-09-21T13:49:00.059382+0000","last_became_peered":"2024-09-21T13:49:00.059382+0000","last_unstale":"2024-09-21T13:49:39.811774+0000","last_undegraded":"2024-09-21T13:49:39.811774+0000","last_fullsized":"2024-09-21T13:49:39.811774+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:05.303397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836250+0000","last_change":"2024-09-21T13:49:02.066978+0000","last_active":"2024-09-21T13:49:39.836250+0000","last_peered":"2024-09-21T13:49:39.836250+0000","last_clean":"2024-09-21T13:49:39.836250+0000","last_became_active":"2024-09-21T13:49:02.066693+0000","last_became_peered":"2024-09-21T13:49:02.066693+0000","last_unstale":"2024-09-21T13:49:39.836250+0000","last_undegraded":"2024-09-21T13:49:39.836250+0000","last_fullsized":"2024-09-21T13:49:39.836250+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:28.710214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835306+0000","last_change":"2024-09-21T13:49:04.080709+0000","last_active":"2024-09-21T13:49:39.835306+0000","last_peered":"2024-09-21T13:49:39.835306+0000","last_clean":"2024-09-21T13:49:39.835306+0000","last_became_active":"2024-09-21T13:49:04.080359+0000","last_became_peered":"2024-09-21T13:49:04.080359+0000","last_unstale":"2024-09-21T13:49:39.835306+0000","last_undegraded":"2024-09-21T13:49:39.835306+0000","last_fullsized":"2024-09-21T13:49:39.835306+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:09.590774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836673+0000","last_change":"2024-09-21T13:48:58.052581+0000","last_active":"2024-09-21T13:49:39.836673+0000","last_peered":"2024-09-21T13:49:39.836673+0000","last_clean":"2024-09-21T13:49:39.836673+0000","last_became_active":"2024-09-21T13:48:58.050906+0000","last_became_peered":"2024-09-21T13:48:58.050906+0000","last_unstale":"2024-09-21T13:49:39.836673+0000","last_undegraded":"2024-09-21T13:49:39.836673+0000","last_fullsized":"2024-09-21T13:49:39.836673+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:19.977336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"68'5","reported_seq":52,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.811724+0000","last_change":"2024-09-21T13:49:00.060913+0000","last_active":"2024-09-21T13:49:39.811724+0000","last_peered":"2024-09-21T13:49:39.811724+0000","last_clean":"2024-09-21T13:49:39.811724+0000","last_became_active":"2024-09-21T13:49:00.060497+0000","last_became_peered":"2024-09-21T13:49:00.060497+0000","last_unstale":"2024-09-21T13:49:39.811724+0000","last_undegraded":"2024-09-21T13:49:39.811724+0000","last_fullsized":"2024-09-21T13:49:39.811724+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:10.507839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835593+0000","last_change":"2024-09-21T13:49:02.060484+0000","last_active":"2024-09-21T13:49:39.835593+0000","last_peered":"2024-09-21T13:49:39.835593+0000","last_clean":"2024-09-21T13:49:39.835593+0000","last_became_active":"2024-09-21T13:49:02.060212+0000","last_became_peered":"2024-09-21T13:49:02.060212+0000","last_unstale":"2024-09-21T13:49:39.835593+0000","last_undegraded":"2024-09-21T13:49:39.835593+0000","last_fullsized":"2024-09-21T13:49:39.835593+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:17.976156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835659+0000","last_change":"2024-09-21T13:49:04.078296+0000","last_active":"2024-09-21T13:49:39.835659+0000","last_peered":"2024-09-21T13:49:39.835659+0000","last_clean":"2024-09-21T13:49:39.835659+0000","last_became_active":"2024-09-21T13:49:04.078026+0000","last_became_peered":"2024-09-21T13:49:04.078026+0000","last_unstale":"2024-09-21T13:49:39.835659+0000","last_undegraded":"2024-09-21T13:49:39.835659+0000","last_fullsized":"2024-09-21T13:49:39.835659+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:05.284846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.812185+0000","last_change":"2024-09-21T13:49:04.432458+0000","last_active":"2024-09-21T13:49:39.812185+0000","last_peered":"2024-09-21T13:49:39.812185+0000","last_clean":"2024-09-21T13:49:39.812185+0000","last_became_active":"2024-09-21T13:49:04.431422+0000","last_became_peered":"2024-09-21T13:49:04.431422+0000","last_unstale":"2024-09-21T13:49:39.812185+0000","last_undegraded":"2024-09-21T13:49:39.812185+0000","last_fullsized":"2024-09-21T13:49:39.812185+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:08.067421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836389+0000","last_change":"2024-09-21T13:48:58.057667+0000","last_active":"2024-09-21T13:49:39.836389+0000","last_peered":"2024-09-21T13:49:39.836389+0000","last_clean":"2024-09-21T13:49:39.836389+0000","last_became_active":"2024-09-21T13:48:58.057336+0000","last_became_peered":"2024-09-21T13:48:58.057336+0000","last_unstale":"2024-09-21T13:49:39.836389+0000","last_undegraded":"2024-09-21T13:49:39.836389+0000","last_fullsized":"2024-09-21T13:49:39.836389+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:18.555629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"68'15","reported_seq":67,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.807764+0000","last_change":"2024-09-21T13:49:00.059068+0000","last_active":"2024-09-21T13:49:39.807764+0000","last_peered":"2024-09-21T13:49:39.807764+0000","last_clean":"2024-09-21T13:49:39.807764+0000","last_became_active":"2024-09-21T13:49:00.058730+0000","last_became_peered":"2024-09-21T13:49:00.058730+0000","last_unstale":"2024-09-21T13:49:39.807764+0000","last_undegraded":"2024-09-21T13:49:39.807764+0000","last_fullsized":"2024-09-21T13:49:39.807764+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:18.796269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.808006+0000","last_change":"2024-09-21T13:49:02.067253+0000","last_active":"2024-09-21T13:49:39.808006+0000","last_peered":"2024-09-21T13:49:39.808006+0000","last_clean":"2024-09-21T13:49:39.808006+0000","last_became_active":"2024-09-21T13:49:02.066796+0000","last_became_peered":"2024-09-21T13:49:02.066796+0000","last_unstale":"2024-09-21T13:49:39.808006+0000","last_undegraded":"2024-09-21T13:49:39.808006+0000","last_fullsized":"2024-09-21T13:49:39.808006+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:24.625163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836826+0000","last_change":"2024-09-21T13:49:04.434429+0000","last_active":"2024-09-21T13:49:39.836826+0000","last_peered":"2024-09-21T13:49:39.836826+0000","last_clean":"2024-09-21T13:49:39.836826+0000","last_became_active":"2024-09-21T13:49:04.434049+0000","last_became_peered":"2024-09-21T13:49:04.434049+0000","last_unstale":"2024-09-21T13:49:39.836826+0000","last_undegraded":"2024-09-21T13:49:39.836826+0000","last_fullsized":"2024-09-21T13:49:39.836826+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:25.759474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.811889+0000","last_change":"2024-09-21T13:48:58.043457+0000","last_active":"2024-09-21T13:49:39.811889+0000","last_peered":"2024-09-21T13:49:39.811889+0000","last_clean":"2024-09-21T13:49:39.811889+0000","last_became_active":"2024-09-21T13:48:58.043088+0000","last_became_peered":"2024-09-21T13:48:58.043088+0000","last_unstale":"2024-09-21T13:49:39.811889+0000","last_undegraded":"2024-09-21T13:49:39.811889+0000","last_fullsized":"2024-09-21T13:49:39.811889+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:21.622429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"68'12","reported_seq":65,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836785+0000","last_change":"2024-09-21T13:49:00.062162+0000","last_active":"2024-09-21T13:49:39.836785+0000","last_peered":"2024-09-21T13:49:39.836785+0000","last_clean":"2024-09-21T13:49:39.836785+0000","last_became_active":"2024-09-21T13:49:00.061867+0000","last_became_peered":"2024-09-21T13:49:00.061867+0000","last_unstale":"2024-09-21T13:49:39.836785+0000","last_undegraded":"2024-09-21T13:49:39.836785+0000","last_fullsized":"2024-09-21T13:49:39.836785+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:27.206067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.812068+0000","last_change":"2024-09-21T13:49:02.063090+0000","last_active":"2024-09-21T13:49:39.812068+0000","last_peered":"2024-09-21T13:49:39.812068+0000","last_clean":"2024-09-21T13:49:39.812068+0000","last_became_active":"2024-09-21T13:49:02.062802+0000","last_became_peered":"2024-09-21T13:49:02.062802+0000","last_unstale":"2024-09-21T13:49:39.812068+0000","last_undegraded":"2024-09-21T13:49:39.812068+0000","last_fullsized":"2024-09-21T13:49:39.812068+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:20.369055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":35,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836298+0000","last_change":"2024-09-21T13:49:04.076090+0000","last_active":"2024-09-21T13:49:39.836298+0000","last_peered":"2024-09-21T13:49:39.836298+0000","last_clean":"2024-09-21T13:49:39.836298+0000","last_became_active":"2024-09-21T13:49:04.075792+0000","last_became_peered":"2024-09-21T13:49:04.075792+0000","last_unstale":"2024-09-21T13:49:39.836298+0000","last_undegraded":"2024-09-21T13:49:39.836298+0000","last_fullsized":"2024-09-21T13:49:39.836298+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:07.846061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":47,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.806690+0000","last_change":"2024-09-21T13:48:58.049463+0000","last_active":"2024-09-21T13:49:39.806690+0000","last_peered":"2024-09-21T13:49:39.806690+0000","last_clean":"2024-09-21T13:49:39.806690+0000","last_became_active":"2024-09-21T13:48:58.048972+0000","last_became_peered":"2024-09-21T13:48:58.048972+0000","last_unstale":"2024-09-21T13:49:39.806690+0000","last_undegraded":"2024-09-21T13:49:39.806690+0000","last_fullsized":"2024-09-21T13:49:39.806690+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:03.427338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"68'10","reported_seq":57,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836578+0000","last_change":"2024-09-21T13:49:00.426837+0000","last_active":"2024-09-21T13:49:39.836578+0000","last_peered":"2024-09-21T13:49:39.836578+0000","last_clean":"2024-09-21T13:49:39.836578+0000","last_became_active":"2024-09-21T13:49:00.426178+0000","last_became_peered":"2024-09-21T13:49:00.426178+0000","last_unstale":"2024-09-21T13:49:39.836578+0000","last_undegraded":"2024-09-21T13:49:39.836578+0000","last_fullsized":"2024-09-21T13:49:39.836578+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:22.029884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"68'8","reported_seq":61,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:48.114202+0000","last_change":"2024-09-21T13:49:02.059979+0000","last_active":"2024-09-21T13:49:48.114202+0000","last_peered":"2024-09-21T13:49:48.114202+0000","last_clean":"2024-09-21T13:49:48.114202+0000","last_became_active":"2024-09-21T13:49:02.059638+0000","last_became_peered":"2024-09-21T13:49:02.059638+0000","last_unstale":"2024-09-21T13:49:48.114202+0000","last_undegraded":"2024-09-21T13:49:48.114202+0000","last_fullsized":"2024-09-21T13:49:48.114202+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:26.974288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"68'15","reported_seq":67,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.808011+0000","last_change":"2024-09-21T13:49:00.060620+0000","last_active":"2024-09-21T13:49:39.808011+0000","last_peered":"2024-09-21T13:49:39.808011+0000","last_clean":"2024-09-21T13:49:39.808011+0000","last_became_active":"2024-09-21T13:49:00.060324+0000","last_became_peered":"2024-09-21T13:49:00.060324+0000","last_unstale":"2024-09-21T13:49:39.808011+0000","last_undegraded":"2024-09-21T13:49:39.808011+0000","last_fullsized":"2024-09-21T13:49:39.808011+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:20.149960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835655+0000","last_change":"2024-09-21T13:48:58.052171+0000","last_active":"2024-09-21T13:49:39.835655+0000","last_peered":"2024-09-21T13:49:39.835655+0000","last_clean":"2024-09-21T13:49:39.835655+0000","last_became_active":"2024-09-21T13:48:58.051911+0000","last_became_peered":"2024-09-21T13:48:58.051911+0000","last_unstale":"2024-09-21T13:49:39.835655+0000","last_undegraded":"2024-09-21T13:49:39.835655+0000","last_fullsized":"2024-09-21T13:49:39.835655+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:49:58.457410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"68'8","reported_seq":61,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:48.114264+0000","last_change":"2024-09-21T13:49:02.061966+0000","last_active":"2024-09-21T13:49:48.114264+0000","last_peered":"2024-09-21T13:49:48.114264+0000","last_clean":"2024-09-21T13:49:48.114264+0000","last_became_active":"2024-09-21T13:49:02.061679+0000","last_became_peered":"2024-09-21T13:49:02.061679+0000","last_unstale":"2024-09-21T13:49:48.114264+0000","last_undegraded":"2024-09-21T13:49:48.114264+0000","last_fullsized":"2024-09-21T13:49:48.114264+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:28.854727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836647+0000","last_change":"2024-09-21T13:49:04.082213+0000","last_active":"2024-09-21T13:49:39.836647+0000","last_peered":"2024-09-21T13:49:39.836647+0000","last_clean":"2024-09-21T13:49:39.836647+0000","last_became_active":"2024-09-21T13:49:04.081905+0000","last_became_peered":"2024-09-21T13:49:04.081905+0000","last_unstale":"2024-09-21T13:49:39.836647+0000","last_undegraded":"2024-09-21T13:49:39.836647+0000","last_fullsized":"2024-09-21T13:49:39.836647+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:17.499009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"68'18","reported_seq":74,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835396+0000","last_change":"2024-09-21T13:49:00.429686+0000","last_active":"2024-09-21T13:49:39.835396+0000","last_peered":"2024-09-21T13:49:39.835396+0000","last_clean":"2024-09-21T13:49:39.835396+0000","last_became_active":"2024-09-21T13:49:00.429314+0000","last_became_peered":"2024-09-21T13:49:00.429314+0000","last_unstale":"2024-09-21T13:49:39.835396+0000","last_undegraded":"2024-09-21T13:49:39.835396+0000","last_fullsized":"2024-09-21T13:49:39.835396+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:09.824746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.837041+0000","last_change":"2024-09-21T13:48:58.056669+0000","last_active":"2024-09-21T13:49:39.837041+0000","last_peered":"2024-09-21T13:49:39.837041+0000","last_clean":"2024-09-21T13:49:39.837041+0000","last_became_active":"2024-09-21T13:48:58.056375+0000","last_became_peered":"2024-09-21T13:48:58.056375+0000","last_unstale":"2024-09-21T13:49:39.837041+0000","last_undegraded":"2024-09-21T13:49:39.837041+0000","last_fullsized":"2024-09-21T13:49:39.837041+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:21.686532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.811469+0000","last_change":"2024-09-21T13:49:02.063237+0000","last_active":"2024-09-21T13:49:39.811469+0000","last_peered":"2024-09-21T13:49:39.811469+0000","last_clean":"2024-09-21T13:49:39.811469+0000","last_became_active":"2024-09-21T13:49:02.062955+0000","last_became_peered":"2024-09-21T13:49:02.062955+0000","last_unstale":"2024-09-21T13:49:39.811469+0000","last_undegraded":"2024-09-21T13:49:39.811469+0000","last_fullsized":"2024-09-21T13:49:39.811469+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:05.393016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.811701+0000","last_change":"2024-09-21T13:49:04.076445+0000","last_active":"2024-09-21T13:49:39.811701+0000","last_peered":"2024-09-21T13:49:39.811701+0000","last_clean":"2024-09-21T13:49:39.811701+0000","last_became_active":"2024-09-21T13:49:04.075818+0000","last_became_peered":"2024-09-21T13:49:04.075818+0000","last_unstale":"2024-09-21T13:49:39.811701+0000","last_undegraded":"2024-09-21T13:49:39.811701+0000","last_fullsized":"2024-09-21T13:49:39.811701+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:31.482751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"68'14","reported_seq":63,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.811530+0000","last_change":"2024-09-21T13:49:00.055423+0000","last_active":"2024-09-21T13:49:39.811530+0000","last_peered":"2024-09-21T13:49:39.811530+0000","last_clean":"2024-09-21T13:49:39.811530+0000","last_became_active":"2024-09-21T13:49:00.055031+0000","last_became_peered":"2024-09-21T13:49:00.055031+0000","last_unstale":"2024-09-21T13:49:39.811530+0000","last_undegraded":"2024-09-21T13:49:39.811530+0000","last_fullsized":"2024-09-21T13:49:39.811530+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:06.138147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":47,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836778+0000","last_change":"2024-09-21T13:48:58.053829+0000","last_active":"2024-09-21T13:49:39.836778+0000","last_peered":"2024-09-21T13:49:39.836778+0000","last_clean":"2024-09-21T13:49:39.836778+0000","last_became_active":"2024-09-21T13:48:58.053533+0000","last_became_peered":"2024-09-21T13:48:58.053533+0000","last_unstale":"2024-09-21T13:49:39.836778+0000","last_undegraded":"2024-09-21T13:49:39.836778+0000","last_fullsized":"2024-09-21T13:49:39.836778+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:49:58.639672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"68'8","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:48.114084+0000","last_change":"2024-09-21T13:49:02.070405+0000","last_active":"2024-09-21T13:49:48.114084+0000","last_peered":"2024-09-21T13:49:48.114084+0000","last_clean":"2024-09-21T13:49:48.114084+0000","last_became_active":"2024-09-21T13:49:02.070064+0000","last_became_peered":"2024-09-21T13:49:02.070064+0000","last_unstale":"2024-09-21T13:49:48.114084+0000","last_undegraded":"2024-09-21T13:49:48.114084+0000","last_fullsized":"2024-09-21T13:49:48.114084+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:13.746404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836389+0000","last_change":"2024-09-21T13:49:04.434498+0000","last_active":"2024-09-21T13:49:39.836389+0000","last_peered":"2024-09-21T13:49:39.836389+0000","last_clean":"2024-09-21T13:49:39.836389+0000","last_became_active":"2024-09-21T13:49:04.433920+0000","last_became_peered":"2024-09-21T13:49:04.433920+0000","last_unstale":"2024-09-21T13:49:39.836389+0000","last_undegraded":"2024-09-21T13:49:39.836389+0000","last_fullsized":"2024-09-21T13:49:39.836389+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:22.847243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"68'10","reported_seq":57,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.807473+0000","last_change":"2024-09-21T13:49:00.054972+0000","last_active":"2024-09-21T13:49:39.807473+0000","last_peered":"2024-09-21T13:49:39.807473+0000","last_clean":"2024-09-21T13:49:39.807473+0000","last_became_active":"2024-09-21T13:49:00.054564+0000","last_became_peered":"2024-09-21T13:49:00.054564+0000","last_unstale":"2024-09-21T13:49:39.807473+0000","last_undegraded":"2024-09-21T13:49:39.807473+0000","last_fullsized":"2024-09-21T13:49:39.807473+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:18.647498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"61'1","reported_seq":52,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.837265+0000","last_change":"2024-09-21T13:48:58.050844+0000","last_active":"2024-09-21T13:49:39.837265+0000","last_peered":"2024-09-21T13:49:39.837265+0000","last_clean":"2024-09-21T13:49:39.837265+0000","last_became_active":"2024-09-21T13:48:58.050548+0000","last_became_peered":"2024-09-21T13:48:58.050548+0000","last_unstale":"2024-09-21T13:49:39.837265+0000","last_undegraded":"2024-09-21T13:49:39.837265+0000","last_fullsized":"2024-09-21T13:49:39.837265+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:17.587266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"68'8","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:48.114036+0000","last_change":"2024-09-21T13:49:02.068926+0000","last_active":"2024-09-21T13:49:48.114036+0000","last_peered":"2024-09-21T13:49:48.114036+0000","last_clean":"2024-09-21T13:49:48.114036+0000","last_became_active":"2024-09-21T13:49:02.067664+0000","last_became_peered":"2024-09-21T13:49:02.067664+0000","last_unstale":"2024-09-21T13:49:48.114036+0000","last_undegraded":"2024-09-21T13:49:48.114036+0000","last_fullsized":"2024-09-21T13:49:48.114036+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:09.630253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835435+0000","last_change":"2024-09-21T13:49:04.081771+0000","last_active":"2024-09-21T13:49:39.835435+0000","last_peered":"2024-09-21T13:49:39.835435+0000","last_clean":"2024-09-21T13:49:39.835435+0000","last_became_active":"2024-09-21T13:49:04.081462+0000","last_became_peered":"2024-09-21T13:49:04.081462+0000","last_unstale":"2024-09-21T13:49:39.835435+0000","last_undegraded":"2024-09-21T13:49:39.835435+0000","last_fullsized":"2024-09-21T13:49:39.835435+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:25.632205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"68'19","reported_seq":78,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836173+0000","last_change":"2024-09-21T13:49:00.426917+0000","last_active":"2024-09-21T13:49:39.836173+0000","last_peered":"2024-09-21T13:49:39.836173+0000","last_clean":"2024-09-21T13:49:39.836173+0000","last_became_active":"2024-09-21T13:49:00.426579+0000","last_became_peered":"2024-09-21T13:49:00.426579+0000","last_unstale":"2024-09-21T13:49:39.836173+0000","last_undegraded":"2024-09-21T13:49:39.836173+0000","last_fullsized":"2024-09-21T13:49:39.836173+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:15.913331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.808545+0000","last_change":"2024-09-21T13:48:58.049961+0000","last_active":"2024-09-21T13:49:39.808545+0000","last_peered":"2024-09-21T13:49:39.808545+0000","last_clean":"2024-09-21T13:49:39.808545+0000","last_became_active":"2024-09-21T13:48:58.049682+0000","last_became_peered":"2024-09-21T13:48:58.049682+0000","last_unstale":"2024-09-21T13:49:39.808545+0000","last_undegraded":"2024-09-21T13:49:39.808545+0000","last_fullsized":"2024-09-21T13:49:39.808545+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:02.280644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835812+0000","last_change":"2024-09-21T13:49:02.061604+0000","last_active":"2024-09-21T13:49:39.835812+0000","last_peered":"2024-09-21T13:49:39.835812+0000","last_clean":"2024-09-21T13:49:39.835812+0000","last_became_active":"2024-09-21T13:49:02.061187+0000","last_became_peered":"2024-09-21T13:49:02.061187+0000","last_unstale":"2024-09-21T13:49:39.835812+0000","last_undegraded":"2024-09-21T13:49:39.835812+0000","last_fullsized":"2024-09-21T13:49:39.835812+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:08.846428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.806520+0000","last_change":"2024-09-21T13:49:04.431651+0000","last_active":"2024-09-21T13:49:39.806520+0000","last_peered":"2024-09-21T13:49:39.806520+0000","last_clean":"2024-09-21T13:49:39.806520+0000","last_became_active":"2024-09-21T13:49:04.431079+0000","last_became_peered":"2024-09-21T13:49:04.431079+0000","last_unstale":"2024-09-21T13:49:39.806520+0000","last_undegraded":"2024-09-21T13:49:39.806520+0000","last_fullsized":"2024-09-21T13:49:39.806520+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:05.155255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"68'28","reported_seq":102,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:48.114038+0000","last_change":"2024-09-21T13:49:00.060616+0000","last_active":"2024-09-21T13:49:48.114038+0000","last_peered":"2024-09-21T13:49:48.114038+0000","last_clean":"2024-09-21T13:49:48.114038+0000","last_became_active":"2024-09-21T13:49:00.060355+0000","last_became_peered":"2024-09-21T13:49:00.060355+0000","last_unstale":"2024-09-21T13:49:48.114038+0000","last_undegraded":"2024-09-21T13:49:48.114038+0000","last_fullsized":"2024-09-21T13:49:48.114038+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:26.214000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.812060+0000","last_change":"2024-09-21T13:48:58.055640+0000","last_active":"2024-09-21T13:49:39.812060+0000","last_peered":"2024-09-21T13:49:39.812060+0000","last_clean":"2024-09-21T13:49:39.812060+0000","last_became_active":"2024-09-21T13:48:58.055366+0000","last_became_peered":"2024-09-21T13:48:58.055366+0000","last_unstale":"2024-09-21T13:49:39.812060+0000","last_undegraded":"2024-09-21T13:49:39.812060+0000","last_fullsized":"2024-09-21T13:49:39.812060+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:17.442979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":56,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.837470+0000","last_change":"2024-09-21T13:49:00.053256+0000","last_active":"2024-09-21T13:49:39.837470+0000","last_peered":"2024-09-21T13:49:39.837470+0000","last_clean":"2024-09-21T13:49:39.837470+0000","last_became_active":"2024-09-21T13:48:57.046246+0000","last_became_peered":"2024-09-21T13:48:57.046246+0000","last_unstale":"2024-09-21T13:49:39.837470+0000","last_undegraded":"2024-09-21T13:49:39.837470+0000","last_fullsized":"2024-09-21T13:49:39.837470+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:48:56.029323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:04.022287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015946669999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836059+0000","last_change":"2024-09-21T13:49:02.068949+0000","last_active":"2024-09-21T13:49:39.836059+0000","last_peered":"2024-09-21T13:49:39.836059+0000","last_clean":"2024-09-21T13:49:39.836059+0000","last_became_active":"2024-09-21T13:49:02.068611+0000","last_became_peered":"2024-09-21T13:49:02.068611+0000","last_unstale":"2024-09-21T13:49:39.836059+0000","last_undegraded":"2024-09-21T13:49:39.836059+0000","last_fullsized":"2024-09-21T13:49:39.836059+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:01.307406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836335+0000","last_change":"2024-09-21T13:49:04.080755+0000","last_active":"2024-09-21T13:49:39.836335+0000","last_peered":"2024-09-21T13:49:39.836335+0000","last_clean":"2024-09-21T13:49:39.836335+0000","last_became_active":"2024-09-21T13:49:04.080466+0000","last_became_peered":"2024-09-21T13:49:04.080466+0000","last_unstale":"2024-09-21T13:49:39.836335+0000","last_undegraded":"2024-09-21T13:49:39.836335+0000","last_fullsized":"2024-09-21T13:49:39.836335+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:06.570558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"68'13","reported_seq":69,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.808039+0000","last_change":"2024-09-21T13:49:00.053645+0000","last_active":"2024-09-21T13:49:39.808039+0000","last_peered":"2024-09-21T13:49:39.808039+0000","last_clean":"2024-09-21T13:49:39.808039+0000","last_became_active":"2024-09-21T13:49:00.053323+0000","last_became_peered":"2024-09-21T13:49:00.053323+0000","last_unstale":"2024-09-21T13:49:39.808039+0000","last_undegraded":"2024-09-21T13:49:39.808039+0000","last_fullsized":"2024-09-21T13:49:39.808039+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:02.019380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.806770+0000","last_change":"2024-09-21T13:48:58.050687+0000","last_active":"2024-09-21T13:49:39.806770+0000","last_peered":"2024-09-21T13:49:39.806770+0000","last_clean":"2024-09-21T13:49:39.806770+0000","last_became_active":"2024-09-21T13:48:58.050371+0000","last_became_peered":"2024-09-21T13:48:58.050371+0000","last_unstale":"2024-09-21T13:49:39.806770+0000","last_undegraded":"2024-09-21T13:49:39.806770+0000","last_fullsized":"2024-09-21T13:49:39.806770+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:19.757852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":55,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.807250+0000","last_change":"2024-09-21T13:49:00.045590+0000","last_active":"2024-09-21T13:49:39.807250+0000","last_peered":"2024-09-21T13:49:39.807250+0000","last_clean":"2024-09-21T13:49:39.807250+0000","last_became_active":"2024-09-21T13:48:57.047067+0000","last_became_peered":"2024-09-21T13:48:57.047067+0000","last_unstale":"2024-09-21T13:49:39.807250+0000","last_undegraded":"2024-09-21T13:49:39.807250+0000","last_fullsized":"2024-09-21T13:49:39.807250+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:48:56.029323+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:49:56.701182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065666300000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.807847+0000","last_change":"2024-09-21T13:49:02.057270+0000","last_active":"2024-09-21T13:49:39.807847+0000","last_peered":"2024-09-21T13:49:39.807847+0000","last_clean":"2024-09-21T13:49:39.807847+0000","last_became_active":"2024-09-21T13:49:02.056806+0000","last_became_peered":"2024-09-21T13:49:02.056806+0000","last_unstale":"2024-09-21T13:49:39.807847+0000","last_undegraded":"2024-09-21T13:49:39.807847+0000","last_fullsized":"2024-09-21T13:49:39.807847+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:17.996686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836856+0000","last_change":"2024-09-21T13:49:04.431351+0000","last_active":"2024-09-21T13:49:39.836856+0000","last_peered":"2024-09-21T13:49:39.836856+0000","last_clean":"2024-09-21T13:49:39.836856+0000","last_became_active":"2024-09-21T13:49:04.431035+0000","last_became_peered":"2024-09-21T13:49:04.431035+0000","last_unstale":"2024-09-21T13:49:39.836856+0000","last_undegraded":"2024-09-21T13:49:39.836856+0000","last_fullsized":"2024-09-21T13:49:39.836856+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:03.519988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"68'12","reported_seq":60,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836677+0000","last_change":"2024-09-21T13:49:00.053466+0000","last_active":"2024-09-21T13:49:39.836677+0000","last_peered":"2024-09-21T13:49:39.836677+0000","last_clean":"2024-09-21T13:49:39.836677+0000","last_became_active":"2024-09-21T13:49:00.053080+0000","last_became_peered":"2024-09-21T13:49:00.053080+0000","last_unstale":"2024-09-21T13:49:39.836677+0000","last_undegraded":"2024-09-21T13:49:39.836677+0000","last_fullsized":"2024-09-21T13:49:39.836677+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:03.884409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835630+0000","last_change":"2024-09-21T13:48:58.051534+0000","last_active":"2024-09-21T13:49:39.835630+0000","last_peered":"2024-09-21T13:49:39.835630+0000","last_clean":"2024-09-21T13:49:39.835630+0000","last_became_active":"2024-09-21T13:48:58.051249+0000","last_became_peered":"2024-09-21T13:48:58.051249+0000","last_unstale":"2024-09-21T13:49:39.835630+0000","last_undegraded":"2024-09-21T13:49:39.835630+0000","last_fullsized":"2024-09-21T13:49:39.835630+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:04.856323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":107,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:51.955301+0000","last_change":"2024-09-21T13:49:00.426330+0000","last_active":"2024-09-21T13:49:51.955301+0000","last_peered":"2024-09-21T13:49:51.955301+0000","last_clean":"2024-09-21T13:49:51.955301+0000","last_became_active":"2024-09-21T13:48:57.046084+0000","last_became_peered":"2024-09-21T13:48:57.046084+0000","last_unstale":"2024-09-21T13:49:51.955301+0000","last_undegraded":"2024-09-21T13:49:51.955301+0000","last_fullsized":"2024-09-21T13:49:51.955301+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:48:56.029323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:08.036165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047048000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":50,"num_read_kb":45,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.837653+0000","last_change":"2024-09-21T13:49:02.066942+0000","last_active":"2024-09-21T13:49:39.837653+0000","last_peered":"2024-09-21T13:49:39.837653+0000","last_clean":"2024-09-21T13:49:39.837653+0000","last_became_active":"2024-09-21T13:49:02.066641+0000","last_became_peered":"2024-09-21T13:49:02.066641+0000","last_unstale":"2024-09-21T13:49:39.837653+0000","last_undegraded":"2024-09-21T13:49:39.837653+0000","last_fullsized":"2024-09-21T13:49:39.837653+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:22.324406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.808219+0000","last_change":"2024-09-21T13:49:04.078686+0000","last_active":"2024-09-21T13:49:39.808219+0000","last_peered":"2024-09-21T13:49:39.808219+0000","last_clean":"2024-09-21T13:49:39.808219+0000","last_became_active":"2024-09-21T13:49:04.078372+0000","last_became_peered":"2024-09-21T13:49:04.078372+0000","last_unstale":"2024-09-21T13:49:39.808219+0000","last_undegraded":"2024-09-21T13:49:39.808219+0000","last_fullsized":"2024-09-21T13:49:39.808219+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:25.621989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"68'16","reported_seq":76,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:48.114277+0000","last_change":"2024-09-21T13:49:00.427430+0000","last_active":"2024-09-21T13:49:48.114277+0000","last_peered":"2024-09-21T13:49:48.114277+0000","last_clean":"2024-09-21T13:49:48.114277+0000","last_became_active":"2024-09-21T13:49:00.426867+0000","last_became_peered":"2024-09-21T13:49:00.426867+0000","last_unstale":"2024-09-21T13:49:48.114277+0000","last_undegraded":"2024-09-21T13:49:48.114277+0000","last_fullsized":"2024-09-21T13:49:48.114277+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:11.516725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835899+0000","last_change":"2024-09-21T13:48:58.052925+0000","last_active":"2024-09-21T13:49:39.835899+0000","last_peered":"2024-09-21T13:49:39.835899+0000","last_clean":"2024-09-21T13:49:39.835899+0000","last_became_active":"2024-09-21T13:48:58.052659+0000","last_became_peered":"2024-09-21T13:48:58.052659+0000","last_unstale":"2024-09-21T13:49:39.835899+0000","last_undegraded":"2024-09-21T13:49:39.835899+0000","last_fullsized":"2024-09-21T13:49:39.835899+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:03.529879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"75'79","reported_seq":95,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836777+0000","last_change":"2024-09-21T13:48:51.893262+0000","last_active":"2024-09-21T13:49:39.836777+0000","last_peered":"2024-09-21T13:49:39.836777+0000","last_clean":"2024-09-21T13:49:39.836777+0000","last_became_active":"2024-09-21T13:48:48.454646+0000","last_became_peered":"2024-09-21T13:48:48.454646+0000","last_unstale":"2024-09-21T13:49:39.836777+0000","last_undegraded":"2024-09-21T13:49:39.836777+0000","last_fullsized":"2024-09-21T13:49:39.836777+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'72","last_scrub_stamp":"2024-09-21T13:48:51.893178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:48:51.893178+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:10.164272+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836466+0000","last_change":"2024-09-21T13:49:02.061930+0000","last_active":"2024-09-21T13:49:39.836466+0000","last_peered":"2024-09-21T13:49:39.836466+0000","last_clean":"2024-09-21T13:49:39.836466+0000","last_became_active":"2024-09-21T13:49:02.061454+0000","last_became_peered":"2024-09-21T13:49:02.061454+0000","last_unstale":"2024-09-21T13:49:39.836466+0000","last_undegraded":"2024-09-21T13:49:39.836466+0000","last_fullsized":"2024-09-21T13:49:39.836466+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:23.000364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.837469+0000","last_change":"2024-09-21T13:49:04.078343+0000","last_active":"2024-09-21T13:49:39.837469+0000","last_peered":"2024-09-21T13:49:39.837469+0000","last_clean":"2024-09-21T13:49:39.837469+0000","last_became_active":"2024-09-21T13:49:04.078091+0000","last_became_peered":"2024-09-21T13:49:04.078091+0000","last_unstale":"2024-09-21T13:49:39.837469+0000","last_undegraded":"2024-09-21T13:49:39.837469+0000","last_fullsized":"2024-09-21T13:49:39.837469+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:17.101054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"68'11","reported_seq":61,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.812470+0000","last_change":"2024-09-21T13:49:00.054827+0000","last_active":"2024-09-21T13:49:39.812470+0000","last_peered":"2024-09-21T13:49:39.812470+0000","last_clean":"2024-09-21T13:49:39.812470+0000","last_became_active":"2024-09-21T13:49:00.053268+0000","last_became_peered":"2024-09-21T13:49:00.053268+0000","last_unstale":"2024-09-21T13:49:39.812470+0000","last_undegraded":"2024-09-21T13:49:39.812470+0000","last_fullsized":"2024-09-21T13:49:39.812470+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:28.403874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.811957+0000","last_change":"2024-09-21T13:48:58.052364+0000","last_active":"2024-09-21T13:49:39.811957+0000","last_peered":"2024-09-21T13:49:39.811957+0000","last_clean":"2024-09-21T13:49:39.811957+0000","last_became_active":"2024-09-21T13:48:58.052092+0000","last_became_peered":"2024-09-21T13:48:58.052092+0000","last_unstale":"2024-09-21T13:49:39.811957+0000","last_undegraded":"2024-09-21T13:49:39.811957+0000","last_fullsized":"2024-09-21T13:49:39.811957+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:10.772378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.837459+0000","last_change":"2024-09-21T13:49:02.062262+0000","last_active":"2024-09-21T13:49:39.837459+0000","last_peered":"2024-09-21T13:49:39.837459+0000","last_clean":"2024-09-21T13:49:39.837459+0000","last_became_active":"2024-09-21T13:49:02.061901+0000","last_became_peered":"2024-09-21T13:49:02.061901+0000","last_unstale":"2024-09-21T13:49:39.837459+0000","last_undegraded":"2024-09-21T13:49:39.837459+0000","last_fullsized":"2024-09-21T13:49:39.837459+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:20.147536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836230+0000","last_change":"2024-09-21T13:49:04.431695+0000","last_active":"2024-09-21T13:49:39.836230+0000","last_peered":"2024-09-21T13:49:39.836230+0000","last_clean":"2024-09-21T13:49:39.836230+0000","last_became_active":"2024-09-21T13:49:04.431268+0000","last_became_peered":"2024-09-21T13:49:04.431268+0000","last_unstale":"2024-09-21T13:49:39.836230+0000","last_undegraded":"2024-09-21T13:49:39.836230+0000","last_fullsized":"2024-09-21T13:49:39.836230+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:27.118735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"68'17","reported_seq":70,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.812385+0000","last_change":"2024-09-21T13:49:00.053077+0000","last_active":"2024-09-21T13:49:39.812385+0000","last_peered":"2024-09-21T13:49:39.812385+0000","last_clean":"2024-09-21T13:49:39.812385+0000","last_became_active":"2024-09-21T13:49:00.052608+0000","last_became_peered":"2024-09-21T13:49:00.052608+0000","last_unstale":"2024-09-21T13:49:39.812385+0000","last_undegraded":"2024-09-21T13:49:39.812385+0000","last_fullsized":"2024-09-21T13:49:39.812385+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:14.266475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836235+0000","last_change":"2024-09-21T13:48:58.051751+0000","last_active":"2024-09-21T13:49:39.836235+0000","last_peered":"2024-09-21T13:49:39.836235+0000","last_clean":"2024-09-21T13:49:39.836235+0000","last_became_active":"2024-09-21T13:48:58.049142+0000","last_became_peered":"2024-09-21T13:48:58.049142+0000","last_unstale":"2024-09-21T13:49:39.836235+0000","last_undegraded":"2024-09-21T13:49:39.836235+0000","last_fullsized":"2024-09-21T13:49:39.836235+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:07.453552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.808232+0000","last_change":"2024-09-21T13:49:02.067811+0000","last_active":"2024-09-21T13:49:39.808232+0000","last_peered":"2024-09-21T13:49:39.808232+0000","last_clean":"2024-09-21T13:49:39.808232+0000","last_became_active":"2024-09-21T13:49:02.067472+0000","last_became_peered":"2024-09-21T13:49:02.067472+0000","last_unstale":"2024-09-21T13:49:39.808232+0000","last_undegraded":"2024-09-21T13:49:39.808232+0000","last_fullsized":"2024-09-21T13:49:39.808232+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:29.225485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.808029+0000","last_change":"2024-09-21T13:49:04.078285+0000","last_active":"2024-09-21T13:49:39.808029+0000","last_peered":"2024-09-21T13:49:39.808029+0000","last_clean":"2024-09-21T13:49:39.808029+0000","last_became_active":"2024-09-21T13:49:04.077962+0000","last_became_peered":"2024-09-21T13:49:04.077962+0000","last_unstale":"2024-09-21T13:49:39.808029+0000","last_undegraded":"2024-09-21T13:49:39.808029+0000","last_fullsized":"2024-09-21T13:49:39.808029+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:13.659310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"68'10","reported_seq":57,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.812106+0000","last_change":"2024-09-21T13:49:00.060182+0000","last_active":"2024-09-21T13:49:39.812106+0000","last_peered":"2024-09-21T13:49:39.812106+0000","last_clean":"2024-09-21T13:49:39.812106+0000","last_became_active":"2024-09-21T13:49:00.059850+0000","last_became_peered":"2024-09-21T13:49:00.059850+0000","last_unstale":"2024-09-21T13:49:39.812106+0000","last_undegraded":"2024-09-21T13:49:39.812106+0000","last_fullsized":"2024-09-21T13:49:39.812106+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:16.066823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836259+0000","last_change":"2024-09-21T13:48:58.058603+0000","last_active":"2024-09-21T13:49:39.836259+0000","last_peered":"2024-09-21T13:49:39.836259+0000","last_clean":"2024-09-21T13:49:39.836259+0000","last_became_active":"2024-09-21T13:48:58.058309+0000","last_became_peered":"2024-09-21T13:48:58.058309+0000","last_unstale":"2024-09-21T13:49:39.836259+0000","last_undegraded":"2024-09-21T13:49:39.836259+0000","last_fullsized":"2024-09-21T13:49:39.836259+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:09.079397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"68'8","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:48.114234+0000","last_change":"2024-09-21T13:49:02.065220+0000","last_active":"2024-09-21T13:49:48.114234+0000","last_peered":"2024-09-21T13:49:48.114234+0000","last_clean":"2024-09-21T13:49:48.114234+0000","last_became_active":"2024-09-21T13:49:02.064817+0000","last_became_peered":"2024-09-21T13:49:02.064817+0000","last_unstale":"2024-09-21T13:49:48.114234+0000","last_undegraded":"2024-09-21T13:49:48.114234+0000","last_fullsized":"2024-09-21T13:49:48.114234+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:11.912943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"68'1","reported_seq":35,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.812179+0000","last_change":"2024-09-21T13:49:04.072175+0000","last_active":"2024-09-21T13:49:39.812179+0000","last_peered":"2024-09-21T13:49:39.812179+0000","last_clean":"2024-09-21T13:49:39.812179+0000","last_became_active":"2024-09-21T13:49:04.071885+0000","last_became_peered":"2024-09-21T13:49:04.071885+0000","last_unstale":"2024-09-21T13:49:39.812179+0000","last_undegraded":"2024-09-21T13:49:39.812179+0000","last_fullsized":"2024-09-21T13:49:39.812179+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:03.670445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"68'9","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836079+0000","last_change":"2024-09-21T13:49:00.053349+0000","last_active":"2024-09-21T13:49:39.836079+0000","last_peered":"2024-09-21T13:49:39.836079+0000","last_clean":"2024-09-21T13:49:39.836079+0000","last_became_active":"2024-09-21T13:49:00.053016+0000","last_became_peered":"2024-09-21T13:49:00.053016+0000","last_unstale":"2024-09-21T13:49:39.836079+0000","last_undegraded":"2024-09-21T13:49:39.836079+0000","last_fullsized":"2024-09-21T13:49:39.836079+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:06.374913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836830+0000","last_change":"2024-09-21T13:48:58.050243+0000","last_active":"2024-09-21T13:49:39.836830+0000","last_peered":"2024-09-21T13:49:39.836830+0000","last_clean":"2024-09-21T13:49:39.836830+0000","last_became_active":"2024-09-21T13:48:58.049543+0000","last_became_peered":"2024-09-21T13:48:58.049543+0000","last_unstale":"2024-09-21T13:49:39.836830+0000","last_undegraded":"2024-09-21T13:49:39.836830+0000","last_fullsized":"2024-09-21T13:49:39.836830+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:17.729176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.807325+0000","last_change":"2024-09-21T13:49:02.063877+0000","last_active":"2024-09-21T13:49:39.807325+0000","last_peered":"2024-09-21T13:49:39.807325+0000","last_clean":"2024-09-21T13:49:39.807325+0000","last_became_active":"2024-09-21T13:49:02.063525+0000","last_became_peered":"2024-09-21T13:49:02.063525+0000","last_unstale":"2024-09-21T13:49:39.807325+0000","last_undegraded":"2024-09-21T13:49:39.807325+0000","last_fullsized":"2024-09-21T13:49:39.807325+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:30.535568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835588+0000","last_change":"2024-09-21T13:49:04.081924+0000","last_active":"2024-09-21T13:49:39.835588+0000","last_peered":"2024-09-21T13:49:39.835588+0000","last_clean":"2024-09-21T13:49:39.835588+0000","last_became_active":"2024-09-21T13:49:04.081617+0000","last_became_peered":"2024-09-21T13:49:04.081617+0000","last_unstale":"2024-09-21T13:49:39.835588+0000","last_undegraded":"2024-09-21T13:49:39.835588+0000","last_fullsized":"2024-09-21T13:49:39.835588+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:25.513722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"68'19","reported_seq":73,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835941+0000","last_change":"2024-09-21T13:49:00.426272+0000","last_active":"2024-09-21T13:49:39.835941+0000","last_peered":"2024-09-21T13:49:39.835941+0000","last_clean":"2024-09-21T13:49:39.835941+0000","last_became_active":"2024-09-21T13:49:00.425955+0000","last_became_peered":"2024-09-21T13:49:00.425955+0000","last_unstale":"2024-09-21T13:49:39.835941+0000","last_undegraded":"2024-09-21T13:49:39.835941+0000","last_fullsized":"2024-09-21T13:49:39.835941+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:02.788196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.837228+0000","last_change":"2024-09-21T13:48:58.053940+0000","last_active":"2024-09-21T13:49:39.837228+0000","last_peered":"2024-09-21T13:49:39.837228+0000","last_clean":"2024-09-21T13:49:39.837228+0000","last_became_active":"2024-09-21T13:48:58.053641+0000","last_became_peered":"2024-09-21T13:48:58.053641+0000","last_unstale":"2024-09-21T13:49:39.837228+0000","last_undegraded":"2024-09-21T13:49:39.837228+0000","last_fullsized":"2024-09-21T13:49:39.837228+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:25.094666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.807300+0000","last_change":"2024-09-21T13:49:02.066996+0000","last_active":"2024-09-21T13:49:39.807300+0000","last_peered":"2024-09-21T13:49:39.807300+0000","last_clean":"2024-09-21T13:49:39.807300+0000","last_became_active":"2024-09-21T13:49:02.066676+0000","last_became_peered":"2024-09-21T13:49:02.066676+0000","last_unstale":"2024-09-21T13:49:39.807300+0000","last_undegraded":"2024-09-21T13:49:39.807300+0000","last_fullsized":"2024-09-21T13:49:39.807300+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:19.241359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836031+0000","last_change":"2024-09-21T13:49:04.078412+0000","last_active":"2024-09-21T13:49:39.836031+0000","last_peered":"2024-09-21T13:49:39.836031+0000","last_clean":"2024-09-21T13:49:39.836031+0000","last_became_active":"2024-09-21T13:49:04.078084+0000","last_became_peered":"2024-09-21T13:49:04.078084+0000","last_unstale":"2024-09-21T13:49:39.836031+0000","last_undegraded":"2024-09-21T13:49:39.836031+0000","last_fullsized":"2024-09-21T13:49:39.836031+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:11.517342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"68'12","reported_seq":65,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.811928+0000","last_change":"2024-09-21T13:49:00.060188+0000","last_active":"2024-09-21T13:49:39.811928+0000","last_peered":"2024-09-21T13:49:39.811928+0000","last_clean":"2024-09-21T13:49:39.811928+0000","last_became_active":"2024-09-21T13:49:00.059720+0000","last_became_peered":"2024-09-21T13:49:00.059720+0000","last_unstale":"2024-09-21T13:49:39.811928+0000","last_undegraded":"2024-09-21T13:49:39.811928+0000","last_fullsized":"2024-09-21T13:49:39.811928+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:17.978182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.837628+0000","last_change":"2024-09-21T13:48:58.058654+0000","last_active":"2024-09-21T13:49:39.837628+0000","last_peered":"2024-09-21T13:49:39.837628+0000","last_clean":"2024-09-21T13:49:39.837628+0000","last_became_active":"2024-09-21T13:48:58.057641+0000","last_became_peered":"2024-09-21T13:48:58.057641+0000","last_unstale":"2024-09-21T13:49:39.837628+0000","last_undegraded":"2024-09-21T13:49:39.837628+0000","last_fullsized":"2024-09-21T13:49:39.837628+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:11.120209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.807118+0000","last_change":"2024-09-21T13:49:02.057568+0000","last_active":"2024-09-21T13:49:39.807118+0000","last_peered":"2024-09-21T13:49:39.807118+0000","last_clean":"2024-09-21T13:49:39.807118+0000","last_became_active":"2024-09-21T13:49:02.057195+0000","last_became_peered":"2024-09-21T13:49:02.057195+0000","last_unstale":"2024-09-21T13:49:39.807118+0000","last_undegraded":"2024-09-21T13:49:39.807118+0000","last_fullsized":"2024-09-21T13:49:39.807118+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:01.498483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836115+0000","last_change":"2024-09-21T13:49:04.078422+0000","last_active":"2024-09-21T13:49:39.836115+0000","last_peered":"2024-09-21T13:49:39.836115+0000","last_clean":"2024-09-21T13:49:39.836115+0000","last_became_active":"2024-09-21T13:49:04.078089+0000","last_became_peered":"2024-09-21T13:49:04.078089+0000","last_unstale":"2024-09-21T13:49:39.836115+0000","last_undegraded":"2024-09-21T13:49:39.836115+0000","last_fullsized":"2024-09-21T13:49:39.836115+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:18.974401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"68'15","reported_seq":67,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835435+0000","last_change":"2024-09-21T13:49:00.426358+0000","last_active":"2024-09-21T13:49:39.835435+0000","last_peered":"2024-09-21T13:49:39.835435+0000","last_clean":"2024-09-21T13:49:39.835435+0000","last_became_active":"2024-09-21T13:49:00.425997+0000","last_became_peered":"2024-09-21T13:49:00.425997+0000","last_unstale":"2024-09-21T13:49:39.835435+0000","last_undegraded":"2024-09-21T13:49:39.835435+0000","last_fullsized":"2024-09-21T13:49:39.835435+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:03.382004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.837692+0000","last_change":"2024-09-21T13:48:58.058778+0000","last_active":"2024-09-21T13:49:39.837692+0000","last_peered":"2024-09-21T13:49:39.837692+0000","last_clean":"2024-09-21T13:49:39.837692+0000","last_became_active":"2024-09-21T13:48:58.057632+0000","last_became_peered":"2024-09-21T13:48:58.057632+0000","last_unstale":"2024-09-21T13:49:39.837692+0000","last_undegraded":"2024-09-21T13:49:39.837692+0000","last_fullsized":"2024-09-21T13:49:39.837692+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:06.995889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"68'8","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:48.114491+0000","last_change":"2024-09-21T13:49:02.056471+0000","last_active":"2024-09-21T13:49:48.114491+0000","last_peered":"2024-09-21T13:49:48.114491+0000","last_clean":"2024-09-21T13:49:48.114491+0000","last_became_active":"2024-09-21T13:49:02.056167+0000","last_became_peered":"2024-09-21T13:49:02.056167+0000","last_unstale":"2024-09-21T13:49:48.114491+0000","last_undegraded":"2024-09-21T13:49:48.114491+0000","last_fullsized":"2024-09-21T13:49:48.114491+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:10.093140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835662+0000","last_change":"2024-09-21T13:49:04.432809+0000","last_active":"2024-09-21T13:49:39.835662+0000","last_peered":"2024-09-21T13:49:39.835662+0000","last_clean":"2024-09-21T13:49:39.835662+0000","last_became_active":"2024-09-21T13:49:04.432546+0000","last_became_peered":"2024-09-21T13:49:04.432546+0000","last_unstale":"2024-09-21T13:49:39.835662+0000","last_undegraded":"2024-09-21T13:49:39.835662+0000","last_fullsized":"2024-09-21T13:49:39.835662+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:08.569714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836980+0000","last_change":"2024-09-21T13:48:58.056383+0000","last_active":"2024-09-21T13:49:39.836980+0000","last_peered":"2024-09-21T13:49:39.836980+0000","last_clean":"2024-09-21T13:49:39.836980+0000","last_became_active":"2024-09-21T13:48:58.056080+0000","last_became_peered":"2024-09-21T13:48:58.056080+0000","last_unstale":"2024-09-21T13:49:39.836980+0000","last_undegraded":"2024-09-21T13:49:39.836980+0000","last_fullsized":"2024-09-21T13:49:39.836980+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:26.306822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"68'6","reported_seq":51,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836952+0000","last_change":"2024-09-21T13:49:00.427204+0000","last_active":"2024-09-21T13:49:39.836952+0000","last_peered":"2024-09-21T13:49:39.836952+0000","last_clean":"2024-09-21T13:49:39.836952+0000","last_became_active":"2024-09-21T13:49:00.426829+0000","last_became_peered":"2024-09-21T13:49:00.426829+0000","last_unstale":"2024-09-21T13:49:39.836952+0000","last_undegraded":"2024-09-21T13:49:39.836952+0000","last_fullsized":"2024-09-21T13:49:39.836952+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:20.014163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.837140+0000","last_change":"2024-09-21T13:49:02.062503+0000","last_active":"2024-09-21T13:49:39.837140+0000","last_peered":"2024-09-21T13:49:39.837140+0000","last_clean":"2024-09-21T13:49:39.837140+0000","last_became_active":"2024-09-21T13:49:02.062231+0000","last_became_peered":"2024-09-21T13:49:02.062231+0000","last_unstale":"2024-09-21T13:49:39.837140+0000","last_undegraded":"2024-09-21T13:49:39.837140+0000","last_fullsized":"2024-09-21T13:49:39.837140+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:21.288932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836220+0000","last_change":"2024-09-21T13:49:04.432777+0000","last_active":"2024-09-21T13:49:39.836220+0000","last_peered":"2024-09-21T13:49:39.836220+0000","last_clean":"2024-09-21T13:49:39.836220+0000","last_became_active":"2024-09-21T13:49:04.432507+0000","last_became_peered":"2024-09-21T13:49:04.432507+0000","last_unstale":"2024-09-21T13:49:39.836220+0000","last_undegraded":"2024-09-21T13:49:39.836220+0000","last_fullsized":"2024-09-21T13:49:39.836220+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:03.727186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"68'9","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836720+0000","last_change":"2024-09-21T13:49:00.428704+0000","last_active":"2024-09-21T13:49:39.836720+0000","last_peered":"2024-09-21T13:49:39.836720+0000","last_clean":"2024-09-21T13:49:39.836720+0000","last_became_active":"2024-09-21T13:49:00.428330+0000","last_became_peered":"2024-09-21T13:49:00.428330+0000","last_unstale":"2024-09-21T13:49:39.836720+0000","last_undegraded":"2024-09-21T13:49:39.836720+0000","last_fullsized":"2024-09-21T13:49:39.836720+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:06.325729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"61'1","reported_seq":52,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835958+0000","last_change":"2024-09-21T13:48:58.050948+0000","last_active":"2024-09-21T13:49:39.835958+0000","last_peered":"2024-09-21T13:49:39.835958+0000","last_clean":"2024-09-21T13:49:39.835958+0000","last_became_active":"2024-09-21T13:48:58.050645+0000","last_became_peered":"2024-09-21T13:48:58.050645+0000","last_unstale":"2024-09-21T13:49:39.835958+0000","last_undegraded":"2024-09-21T13:49:39.835958+0000","last_fullsized":"2024-09-21T13:49:39.835958+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:24.159998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836864+0000","last_change":"2024-09-21T13:49:02.062075+0000","last_active":"2024-09-21T13:49:39.836864+0000","last_peered":"2024-09-21T13:49:39.836864+0000","last_clean":"2024-09-21T13:49:39.836864+0000","last_became_active":"2024-09-21T13:49:02.061775+0000","last_became_peered":"2024-09-21T13:49:02.061775+0000","last_unstale":"2024-09-21T13:49:39.836864+0000","last_undegraded":"2024-09-21T13:49:39.836864+0000","last_fullsized":"2024-09-21T13:49:39.836864+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:16.428002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.807270+0000","last_change":"2024-09-21T13:49:04.069452+0000","last_active":"2024-09-21T13:49:39.807270+0000","last_peered":"2024-09-21T13:49:39.807270+0000","last_clean":"2024-09-21T13:49:39.807270+0000","last_became_active":"2024-09-21T13:49:04.069142+0000","last_became_peered":"2024-09-21T13:49:04.069142+0000","last_unstale":"2024-09-21T13:49:39.807270+0000","last_undegraded":"2024-09-21T13:49:39.807270+0000","last_fullsized":"2024-09-21T13:49:39.807270+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:28.894052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"68'9","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.837072+0000","last_change":"2024-09-21T13:49:00.428080+0000","last_active":"2024-09-21T13:49:39.837072+0000","last_peered":"2024-09-21T13:49:39.837072+0000","last_clean":"2024-09-21T13:49:39.837072+0000","last_became_active":"2024-09-21T13:49:00.427766+0000","last_became_peered":"2024-09-21T13:49:00.427766+0000","last_unstale":"2024-09-21T13:49:39.837072+0000","last_undegraded":"2024-09-21T13:49:39.837072+0000","last_fullsized":"2024-09-21T13:49:39.837072+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:16.384643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836693+0000","last_change":"2024-09-21T13:48:58.052465+0000","last_active":"2024-09-21T13:49:39.836693+0000","last_peered":"2024-09-21T13:49:39.836693+0000","last_clean":"2024-09-21T13:49:39.836693+0000","last_became_active":"2024-09-21T13:48:58.052151+0000","last_became_peered":"2024-09-21T13:48:58.052151+0000","last_unstale":"2024-09-21T13:49:39.836693+0000","last_undegraded":"2024-09-21T13:49:39.836693+0000","last_fullsized":"2024-09-21T13:49:39.836693+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:09.226461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"68'8","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:48.114142+0000","last_change":"2024-09-21T13:49:02.066442+0000","last_active":"2024-09-21T13:49:48.114142+0000","last_peered":"2024-09-21T13:49:48.114142+0000","last_clean":"2024-09-21T13:49:48.114142+0000","last_became_active":"2024-09-21T13:49:02.065602+0000","last_became_peered":"2024-09-21T13:49:02.065602+0000","last_unstale":"2024-09-21T13:49:48.114142+0000","last_undegraded":"2024-09-21T13:49:48.114142+0000","last_fullsized":"2024-09-21T13:49:48.114142+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:10.855295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.812396+0000","last_change":"2024-09-21T13:49:04.073461+0000","last_active":"2024-09-21T13:49:39.812396+0000","last_peered":"2024-09-21T13:49:39.812396+0000","last_clean":"2024-09-21T13:49:39.812396+0000","last_became_active":"2024-09-21T13:49:04.073188+0000","last_became_peered":"2024-09-21T13:49:04.073188+0000","last_unstale":"2024-09-21T13:49:39.812396+0000","last_undegraded":"2024-09-21T13:49:39.812396+0000","last_fullsized":"2024-09-21T13:49:39.812396+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:08.749766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"68'10","reported_seq":57,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836534+0000","last_change":"2024-09-21T13:49:00.427262+0000","last_active":"2024-09-21T13:49:39.836534+0000","last_peered":"2024-09-21T13:49:39.836534+0000","last_clean":"2024-09-21T13:49:39.836534+0000","last_became_active":"2024-09-21T13:49:00.426071+0000","last_became_peered":"2024-09-21T13:49:00.426071+0000","last_unstale":"2024-09-21T13:49:39.836534+0000","last_undegraded":"2024-09-21T13:49:39.836534+0000","last_fullsized":"2024-09-21T13:49:39.836534+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:03.592724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836695+0000","last_change":"2024-09-21T13:48:58.058198+0000","last_active":"2024-09-21T13:49:39.836695+0000","last_peered":"2024-09-21T13:49:39.836695+0000","last_clean":"2024-09-21T13:49:39.836695+0000","last_became_active":"2024-09-21T13:48:58.057587+0000","last_became_peered":"2024-09-21T13:48:58.057587+0000","last_unstale":"2024-09-21T13:49:39.836695+0000","last_undegraded":"2024-09-21T13:49:39.836695+0000","last_fullsized":"2024-09-21T13:49:39.836695+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:19.518354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"68'8","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:48.114248+0000","last_change":"2024-09-21T13:49:02.067827+0000","last_active":"2024-09-21T13:49:48.114248+0000","last_peered":"2024-09-21T13:49:48.114248+0000","last_clean":"2024-09-21T13:49:48.114248+0000","last_became_active":"2024-09-21T13:49:02.067487+0000","last_became_peered":"2024-09-21T13:49:02.067487+0000","last_unstale":"2024-09-21T13:49:48.114248+0000","last_undegraded":"2024-09-21T13:49:48.114248+0000","last_fullsized":"2024-09-21T13:49:48.114248+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:23.003843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836447+0000","last_change":"2024-09-21T13:49:04.081928+0000","last_active":"2024-09-21T13:49:39.836447+0000","last_peered":"2024-09-21T13:49:39.836447+0000","last_clean":"2024-09-21T13:49:39.836447+0000","last_became_active":"2024-09-21T13:49:04.081609+0000","last_became_peered":"2024-09-21T13:49:04.081609+0000","last_unstale":"2024-09-21T13:49:39.836447+0000","last_undegraded":"2024-09-21T13:49:39.836447+0000","last_fullsized":"2024-09-21T13:49:39.836447+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:31.380912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"68'11","reported_seq":61,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.811729+0000","last_change":"2024-09-21T13:49:00.055420+0000","last_active":"2024-09-21T13:49:39.811729+0000","last_peered":"2024-09-21T13:49:39.811729+0000","last_clean":"2024-09-21T13:49:39.811729+0000","last_became_active":"2024-09-21T13:49:00.055024+0000","last_became_peered":"2024-09-21T13:49:00.055024+0000","last_unstale":"2024-09-21T13:49:39.811729+0000","last_undegraded":"2024-09-21T13:49:39.811729+0000","last_fullsized":"2024-09-21T13:49:39.811729+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:02.449612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.812476+0000","last_change":"2024-09-21T13:48:58.051845+0000","last_active":"2024-09-21T13:49:39.812476+0000","last_peered":"2024-09-21T13:49:39.812476+0000","last_clean":"2024-09-21T13:49:39.812476+0000","last_became_active":"2024-09-21T13:48:58.051541+0000","last_became_peered":"2024-09-21T13:48:58.051541+0000","last_unstale":"2024-09-21T13:49:39.812476+0000","last_undegraded":"2024-09-21T13:49:39.812476+0000","last_fullsized":"2024-09-21T13:49:39.812476+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:03.712482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.806822+0000","last_change":"2024-09-21T13:49:02.063328+0000","last_active":"2024-09-21T13:49:39.806822+0000","last_peered":"2024-09-21T13:49:39.806822+0000","last_clean":"2024-09-21T13:49:39.806822+0000","last_became_active":"2024-09-21T13:49:02.062868+0000","last_became_peered":"2024-09-21T13:49:02.062868+0000","last_unstale":"2024-09-21T13:49:39.806822+0000","last_undegraded":"2024-09-21T13:49:39.806822+0000","last_fullsized":"2024-09-21T13:49:39.806822+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:12.568168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835473+0000","last_change":"2024-09-21T13:49:04.083254+0000","last_active":"2024-09-21T13:49:39.835473+0000","last_peered":"2024-09-21T13:49:39.835473+0000","last_clean":"2024-09-21T13:49:39.835473+0000","last_became_active":"2024-09-21T13:49:04.082961+0000","last_became_peered":"2024-09-21T13:49:04.082961+0000","last_unstale":"2024-09-21T13:49:39.835473+0000","last_undegraded":"2024-09-21T13:49:39.835473+0000","last_fullsized":"2024-09-21T13:49:39.835473+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:07.548695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"68'9","reported_seq":58,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.807587+0000","last_change":"2024-09-21T13:49:00.057817+0000","last_active":"2024-09-21T13:49:39.807587+0000","last_peered":"2024-09-21T13:49:39.807587+0000","last_clean":"2024-09-21T13:49:39.807587+0000","last_became_active":"2024-09-21T13:49:00.057392+0000","last_became_peered":"2024-09-21T13:49:00.057392+0000","last_unstale":"2024-09-21T13:49:39.807587+0000","last_undegraded":"2024-09-21T13:49:39.807587+0000","last_fullsized":"2024-09-21T13:49:39.807587+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:02.682820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.837162+0000","last_change":"2024-09-21T13:48:58.058032+0000","last_active":"2024-09-21T13:49:39.837162+0000","last_peered":"2024-09-21T13:49:39.837162+0000","last_clean":"2024-09-21T13:49:39.837162+0000","last_became_active":"2024-09-21T13:48:58.057516+0000","last_became_peered":"2024-09-21T13:48:58.057516+0000","last_unstale":"2024-09-21T13:49:39.837162+0000","last_undegraded":"2024-09-21T13:49:39.837162+0000","last_fullsized":"2024-09-21T13:49:39.837162+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:20.885486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836103+0000","last_change":"2024-09-21T13:49:02.072264+0000","last_active":"2024-09-21T13:49:39.836103+0000","last_peered":"2024-09-21T13:49:39.836103+0000","last_clean":"2024-09-21T13:49:39.836103+0000","last_became_active":"2024-09-21T13:49:02.071922+0000","last_became_peered":"2024-09-21T13:49:02.071922+0000","last_unstale":"2024-09-21T13:49:39.836103+0000","last_undegraded":"2024-09-21T13:49:39.836103+0000","last_fullsized":"2024-09-21T13:49:39.836103+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:12.773964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835233+0000","last_change":"2024-09-21T13:49:04.080300+0000","last_active":"2024-09-21T13:49:39.835233+0000","last_peered":"2024-09-21T13:49:39.835233+0000","last_clean":"2024-09-21T13:49:39.835233+0000","last_became_active":"2024-09-21T13:49:04.079950+0000","last_became_peered":"2024-09-21T13:49:04.079950+0000","last_unstale":"2024-09-21T13:49:39.835233+0000","last_undegraded":"2024-09-21T13:49:39.835233+0000","last_fullsized":"2024-09-21T13:49:39.835233+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:20.474523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"68'11","reported_seq":61,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835652+0000","last_change":"2024-09-21T13:49:00.426487+0000","last_active":"2024-09-21T13:49:39.835652+0000","last_peered":"2024-09-21T13:49:39.835652+0000","last_clean":"2024-09-21T13:49:39.835652+0000","last_became_active":"2024-09-21T13:49:00.426091+0000","last_became_peered":"2024-09-21T13:49:00.426091+0000","last_unstale":"2024-09-21T13:49:39.835652+0000","last_undegraded":"2024-09-21T13:49:39.835652+0000","last_fullsized":"2024-09-21T13:49:39.835652+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:11.657590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.837231+0000","last_change":"2024-09-21T13:48:58.052281+0000","last_active":"2024-09-21T13:49:39.837231+0000","last_peered":"2024-09-21T13:49:39.837231+0000","last_clean":"2024-09-21T13:49:39.837231+0000","last_became_active":"2024-09-21T13:48:58.051974+0000","last_became_peered":"2024-09-21T13:48:58.051974+0000","last_unstale":"2024-09-21T13:49:39.837231+0000","last_undegraded":"2024-09-21T13:49:39.837231+0000","last_fullsized":"2024-09-21T13:49:39.837231+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:25.614472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835533+0000","last_change":"2024-09-21T13:49:02.056666+0000","last_active":"2024-09-21T13:49:39.835533+0000","last_peered":"2024-09-21T13:49:39.835533+0000","last_clean":"2024-09-21T13:49:39.835533+0000","last_became_active":"2024-09-21T13:49:02.056006+0000","last_became_peered":"2024-09-21T13:49:02.056006+0000","last_unstale":"2024-09-21T13:49:39.835533+0000","last_undegraded":"2024-09-21T13:49:39.835533+0000","last_fullsized":"2024-09-21T13:49:39.835533+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:17.511468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836235+0000","last_change":"2024-09-21T13:49:04.434380+0000","last_active":"2024-09-21T13:49:39.836235+0000","last_peered":"2024-09-21T13:49:39.836235+0000","last_clean":"2024-09-21T13:49:39.836235+0000","last_became_active":"2024-09-21T13:49:04.434034+0000","last_became_peered":"2024-09-21T13:49:04.434034+0000","last_unstale":"2024-09-21T13:49:39.836235+0000","last_undegraded":"2024-09-21T13:49:39.836235+0000","last_fullsized":"2024-09-21T13:49:39.836235+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:28.547927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"68'4","reported_seq":48,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835309+0000","last_change":"2024-09-21T13:49:00.063374+0000","last_active":"2024-09-21T13:49:39.835309+0000","last_peered":"2024-09-21T13:49:39.835309+0000","last_clean":"2024-09-21T13:49:39.835309+0000","last_became_active":"2024-09-21T13:49:00.063066+0000","last_became_peered":"2024-09-21T13:49:00.063066+0000","last_unstale":"2024-09-21T13:49:39.835309+0000","last_undegraded":"2024-09-21T13:49:39.835309+0000","last_fullsized":"2024-09-21T13:49:39.835309+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:19.893798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836232+0000","last_change":"2024-09-21T13:48:58.059979+0000","last_active":"2024-09-21T13:49:39.836232+0000","last_peered":"2024-09-21T13:49:39.836232+0000","last_clean":"2024-09-21T13:49:39.836232+0000","last_became_active":"2024-09-21T13:48:58.059674+0000","last_became_peered":"2024-09-21T13:48:58.059674+0000","last_unstale":"2024-09-21T13:49:39.836232+0000","last_undegraded":"2024-09-21T13:49:39.836232+0000","last_fullsized":"2024-09-21T13:49:39.836232+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:02.308942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835914+0000","last_change":"2024-09-21T13:49:02.065043+0000","last_active":"2024-09-21T13:49:39.835914+0000","last_peered":"2024-09-21T13:49:39.835914+0000","last_clean":"2024-09-21T13:49:39.835914+0000","last_became_active":"2024-09-21T13:49:02.064789+0000","last_became_peered":"2024-09-21T13:49:02.064789+0000","last_unstale":"2024-09-21T13:49:39.835914+0000","last_undegraded":"2024-09-21T13:49:39.835914+0000","last_fullsized":"2024-09-21T13:49:39.835914+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:10.569664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835522+0000","last_change":"2024-09-21T13:49:04.075170+0000","last_active":"2024-09-21T13:49:39.835522+0000","last_peered":"2024-09-21T13:49:39.835522+0000","last_clean":"2024-09-21T13:49:39.835522+0000","last_became_active":"2024-09-21T13:49:04.074857+0000","last_became_peered":"2024-09-21T13:49:04.074857+0000","last_unstale":"2024-09-21T13:49:39.835522+0000","last_undegraded":"2024-09-21T13:49:39.835522+0000","last_fullsized":"2024-09-21T13:49:39.835522+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:17.971950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":34,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.808102+0000","last_change":"2024-09-21T13:49:04.075860+0000","last_active":"2024-09-21T13:49:39.808102+0000","last_peered":"2024-09-21T13:49:39.808102+0000","last_clean":"2024-09-21T13:49:39.808102+0000","last_became_active":"2024-09-21T13:49:04.075562+0000","last_became_peered":"2024-09-21T13:49:04.075562+0000","last_unstale":"2024-09-21T13:49:39.808102+0000","last_undegraded":"2024-09-21T13:49:39.808102+0000","last_fullsized":"2024-09-21T13:49:39.808102+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:21.041626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":46,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.835570+0000","last_change":"2024-09-21T13:48:58.055791+0000","last_active":"2024-09-21T13:49:39.835570+0000","last_peered":"2024-09-21T13:49:39.835570+0000","last_clean":"2024-09-21T13:49:39.835570+0000","last_became_active":"2024-09-21T13:48:58.055497+0000","last_became_peered":"2024-09-21T13:48:58.055497+0000","last_unstale":"2024-09-21T13:49:39.835570+0000","last_undegraded":"2024-09-21T13:49:39.835570+0000","last_fullsized":"2024-09-21T13:49:39.835570+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:19.570880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"68'11","reported_seq":61,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836752+0000","last_change":"2024-09-21T13:49:00.058623+0000","last_active":"2024-09-21T13:49:39.836752+0000","last_peered":"2024-09-21T13:49:39.836752+0000","last_clean":"2024-09-21T13:49:39.836752+0000","last_became_active":"2024-09-21T13:49:00.058259+0000","last_became_peered":"2024-09-21T13:49:00.058259+0000","last_unstale":"2024-09-21T13:49:39.836752+0000","last_undegraded":"2024-09-21T13:49:39.836752+0000","last_fullsized":"2024-09-21T13:49:39.836752+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:17.663571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":38,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-09-21T13:49:39.836553+0000","last_change":"2024-09-21T13:49:02.068530+0000","last_active":"2024-09-21T13:49:39.836553+0000","last_peered":"2024-09-21T13:49:39.836553+0000","last_clean":"2024-09-21T13:49:39.836553+0000","last_became_active":"2024-09-21T13:49:02.066159+0000","last_became_peered":"2024-09-21T13:49:02.066159+0000","last_unstale":"2024-09-21T13:49:39.836553+0000","last_undegraded":"2024-09-21T13:49:39.836553+0000","last_fullsized":"2024-09-21T13:49:39.836553+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:29.718157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":50,"num_read_kb":45,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234000,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32628,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712524,"statfs":{"total":95995035648,"available":95961624576,"internally_reserved":0,"allocated":1474560,"data_stored":970453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462933,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28448,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716704,"statfs":{"total":95995035648,"available":95965904896,"internally_reserved":0,"allocated":1454080,"data_stored":969368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724572,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27872,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717280,"statfs":{"total":95995035648,"available":95966494720,"internally_reserved":0,"allocated":864256,"data_stored":379916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920802,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27900,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717252,"statfs":{"total":95995035648,"available":95966466048,"internally_reserved":0,"allocated":892928,"data_stored":379618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117031,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27904,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717248,"statfs":{"total":95995035648,"available":95966461952,"internally_reserved":0,"allocated":897024,"data_stored":380448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411372,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32028,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713124,"statfs":{"total":95995035648,"available":95962238976,"internally_reserved":0,"allocated":860160,"data_stored":379718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574900,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32092,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713060,"statfs":{"total":95995035648,"available":95962173440,"internally_reserved":0,"allocated":925696,"data_stored":380362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738426,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28468,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716684,"statfs":{"total":95995035648,"available":95965884416,"internally_reserved":0,"allocated":1474560,"data_stored":969726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T13:49:58.674 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-21T13:49:58.674 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd dump 2024-09-21T13:49:58.932 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:49:59.004 INFO:journalctl@ceph.alertmanager.a.smithi049.stdout:Sep 21 13:49:58 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-alertmanager-a[74283]: ts=2024-09-21T13:49:58.882Z caller=cluster.go:708 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.00075171s 2024-09-21T13:49:59.113 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:59 smithi049 ceph-mon[40489]: from='client.14763 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T13:49:59.113 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:59 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:59.113 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:59 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:59.113 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:59 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"} : dispatch 2024-09-21T13:49:59.114 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:59 smithi049 ceph-mon[40489]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-21T13:49:59.114 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:59 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:59.114 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:59 smithi049 ceph-mon[40489]: Deploying daemon grafana.a on smithi187 2024-09-21T13:49:59.114 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:59 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T13:49:59.114 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:59 smithi049 ceph-mon[40489]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T13:49:59.114 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:59 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:49:59.114 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:49:59 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:49:59.115 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:59 smithi049 ceph-mon[30080]: from='client.14763 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T13:49:59.115 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:59 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:59.115 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:59 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:59.115 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:59 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"} : dispatch 2024-09-21T13:49:59.115 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:59 smithi049 ceph-mon[30080]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-21T13:49:59.115 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:59 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:59.116 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:59 smithi049 ceph-mon[30080]: Deploying daemon grafana.a on smithi187 2024-09-21T13:49:59.116 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:59 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T13:49:59.116 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:59 smithi049 ceph-mon[30080]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T13:49:59.116 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:59 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:49:59.116 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:49:59 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:49:59.194 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:49:58 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:49:59.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:59 smithi187 ceph-mon[33032]: from='client.14763 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T13:49:59.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:59 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:59.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:59 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:59.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:59 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"} : dispatch 2024-09-21T13:49:59.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:59 smithi187 ceph-mon[33032]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-21T13:49:59.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:59 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:49:59.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:59 smithi187 ceph-mon[33032]: Deploying daemon grafana.a on smithi187 2024-09-21T13:49:59.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:59 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T13:49:59.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:59 smithi187 ceph-mon[33032]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T13:49:59.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:59 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:49:59.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:49:59 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:49:59.688 INFO:teuthology.orchestra.run.smithi049.stdout:epoch 81 2024-09-21T13:49:59.688 INFO:teuthology.orchestra.run.smithi049.stdout:fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 2024-09-21T13:49:59.688 INFO:teuthology.orchestra.run.smithi049.stdout:created 2024-09-21T13:42:57.838275+0000 2024-09-21T13:49:59.688 INFO:teuthology.orchestra.run.smithi049.stdout:modified 2024-09-21T13:49:59.118633+0000 2024-09-21T13:49:59.689 INFO:teuthology.orchestra.run.smithi049.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-21T13:49:59.689 INFO:teuthology.orchestra.run.smithi049.stdout:crush_version 18 2024-09-21T13:49:59.689 INFO:teuthology.orchestra.run.smithi049.stdout:full_ratio 0.95 2024-09-21T13:49:59.689 INFO:teuthology.orchestra.run.smithi049.stdout:backfillfull_ratio 0.9 2024-09-21T13:49:59.689 INFO:teuthology.orchestra.run.smithi049.stdout:nearfull_ratio 0.85 2024-09-21T13:49:59.689 INFO:teuthology.orchestra.run.smithi049.stdout:require_min_compat_client luminous 2024-09-21T13:49:59.689 INFO:teuthology.orchestra.run.smithi049.stdout:min_compat_client jewel 2024-09-21T13:49:59.689 INFO:teuthology.orchestra.run.smithi049.stdout:require_osd_release squid 2024-09-21T13:49:59.689 INFO:teuthology.orchestra.run.smithi049.stdout:stretch_mode_enabled false 2024-09-21T13:49:59.689 INFO:teuthology.orchestra.run.smithi049.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 7.89 2024-09-21T13:49:59.689 INFO:teuthology.orchestra.run.smithi049.stdout:pool 2 'datapool' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 3 pgp_num 3 autoscale_mode off last_change 63 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.65 2024-09-21T13:49:59.689 INFO:teuthology.orchestra.run.smithi049.stdout:pool 3 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 62 flags hashpspool stripe_width 0 application rgw read_balance_score 1.50 2024-09-21T13:49:59.689 INFO:teuthology.orchestra.run.smithi049.stdout:pool 4 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 64 flags hashpspool stripe_width 0 application rgw read_balance_score 2.25 2024-09-21T13:49:59.689 INFO:teuthology.orchestra.run.smithi049.stdout:pool 5 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 66 flags hashpspool stripe_width 0 application rgw read_balance_score 1.25 2024-09-21T13:49:59.690 INFO:teuthology.orchestra.run.smithi049.stdout:pool 6 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 68 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw read_balance_score 1.75 2024-09-21T13:49:59.690 INFO:teuthology.orchestra.run.smithi049.stdout:max_osd 8 2024-09-21T13:49:59.690 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0 up in weight 1 up_from 8 up_thru 66 down_at 0 last_clean_interval [0,0) [v2:172.21.15.49:6802/2095561412,v1:172.21.15.49:6803/2095561412] [v2:172.21.15.49:6804/2095561412,v1:172.21.15.49:6805/2095561412] exists,up 2595dca9-6219-4e0c-8728-751eb102e73a 2024-09-21T13:49:59.690 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1 up in weight 1 up_from 13 up_thru 66 down_at 0 last_clean_interval [0,0) [v2:172.21.15.49:6810/1382840527,v1:172.21.15.49:6811/1382840527] [v2:172.21.15.49:6812/1382840527,v1:172.21.15.49:6813/1382840527] exists,up 0ba327f9-5524-4a40-abcc-729ded0b1917 2024-09-21T13:49:59.690 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2 up in weight 1 up_from 18 up_thru 66 down_at 0 last_clean_interval [0,0) [v2:172.21.15.49:6818/2189046898,v1:172.21.15.49:6819/2189046898] [v2:172.21.15.49:6820/2189046898,v1:172.21.15.49:6821/2189046898] exists,up 27a45750-204d-48fd-bb8f-ebb01273343d 2024-09-21T13:49:59.690 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3 up in weight 1 up_from 27 up_thru 66 down_at 0 last_clean_interval [0,0) [v2:172.21.15.49:6826/1981113123,v1:172.21.15.49:6827/1981113123] [v2:172.21.15.49:6828/1981113123,v1:172.21.15.49:6829/1981113123] exists,up 528bc390-3dc9-4f3b-ab75-d7412c9625db 2024-09-21T13:49:59.690 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4 up in weight 1 up_from 33 up_thru 66 down_at 0 last_clean_interval [0,0) [v2:172.21.15.187:6800/1702447776,v1:172.21.15.187:6801/1702447776] [v2:172.21.15.187:6802/1702447776,v1:172.21.15.187:6803/1702447776] exists,up 6fc64698-2d37-4db0-bf3d-c7e612c45163 2024-09-21T13:49:59.690 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5 up in weight 1 up_from 39 up_thru 66 down_at 0 last_clean_interval [0,0) [v2:172.21.15.187:6808/3659050776,v1:172.21.15.187:6809/3659050776] [v2:172.21.15.187:6810/3659050776,v1:172.21.15.187:6811/3659050776] exists,up 8d7f62a1-7e92-4e06-9cad-86b96be514d8 2024-09-21T13:49:59.690 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6 up in weight 1 up_from 47 up_thru 64 down_at 0 last_clean_interval [0,0) [v2:172.21.15.187:6816/2728426525,v1:172.21.15.187:6817/2728426525] [v2:172.21.15.187:6818/2728426525,v1:172.21.15.187:6819/2728426525] exists,up 3a7afd13-26ce-444c-9cc0-81ce2eab6ce5 2024-09-21T13:49:59.690 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7 up in weight 1 up_from 54 up_thru 66 down_at 0 last_clean_interval [0,0) [v2:172.21.15.187:6824/3680850037,v1:172.21.15.187:6825/3680850037] [v2:172.21.15.187:6826/3680850037,v1:172.21.15.187:6827/3680850037] exists,up 1e08d891-8747-402e-b4c2-aa3802ce8537 2024-09-21T13:49:59.690 INFO:teuthology.orchestra.run.smithi049.stdout:blocklist 172.21.15.49:0/2731793917 expires 2024-09-22T13:49:27.311313+0000 2024-09-21T13:49:59.690 INFO:teuthology.orchestra.run.smithi049.stdout:blocklist 172.21.15.49:0/799831425 expires 2024-09-22T13:49:27.311313+0000 2024-09-21T13:49:59.690 INFO:teuthology.orchestra.run.smithi049.stdout:blocklist 172.21.15.49:6801/2518197280 expires 2024-09-22T13:43:28.110499+0000 2024-09-21T13:49:59.690 INFO:teuthology.orchestra.run.smithi049.stdout:blocklist 172.21.15.49:0/3556829486 expires 2024-09-22T13:43:28.110499+0000 2024-09-21T13:49:59.691 INFO:teuthology.orchestra.run.smithi049.stdout:blocklist 172.21.15.49:0/1249926115 expires 2024-09-22T13:49:27.311313+0000 2024-09-21T13:49:59.691 INFO:teuthology.orchestra.run.smithi049.stdout:blocklist 172.21.15.49:0/231530138 expires 2024-09-22T13:43:28.110499+0000 2024-09-21T13:49:59.691 INFO:teuthology.orchestra.run.smithi049.stdout:blocklist 172.21.15.49:0/4220781134 expires 2024-09-22T13:49:27.311313+0000 2024-09-21T13:49:59.691 INFO:teuthology.orchestra.run.smithi049.stdout:blocklist 172.21.15.49:0/202325455 expires 2024-09-22T13:43:28.110499+0000 2024-09-21T13:49:59.691 INFO:teuthology.orchestra.run.smithi049.stdout:blocklist 172.21.15.49:6800/4129636473 expires 2024-09-22T13:43:56.633723+0000 2024-09-21T13:49:59.691 INFO:teuthology.orchestra.run.smithi049.stdout:blocklist 172.21.15.49:6800/2518197280 expires 2024-09-22T13:43:28.110499+0000 2024-09-21T13:49:59.691 INFO:teuthology.orchestra.run.smithi049.stdout:blocklist 172.21.15.49:0/3514741273 expires 2024-09-22T13:43:56.633723+0000 2024-09-21T13:49:59.691 INFO:teuthology.orchestra.run.smithi049.stdout:blocklist 172.21.15.49:6801/4129636473 expires 2024-09-22T13:43:56.633723+0000 2024-09-21T13:49:59.691 INFO:teuthology.orchestra.run.smithi049.stdout:blocklist 172.21.15.49:0/3125743776 expires 2024-09-22T13:43:56.633723+0000 2024-09-21T13:49:59.691 INFO:teuthology.orchestra.run.smithi049.stdout:blocklist 172.21.15.49:0/3600977363 expires 2024-09-22T13:43:56.633723+0000 2024-09-21T13:49:59.691 INFO:teuthology.orchestra.run.smithi049.stdout:blocklist 172.21.15.49:6801/3453571598 expires 2024-09-22T13:49:27.311313+0000 2024-09-21T13:49:59.691 INFO:teuthology.orchestra.run.smithi049.stdout:blocklist 172.21.15.49:6800/3453571598 expires 2024-09-22T13:49:27.311313+0000 2024-09-21T13:49:59.691 INFO:teuthology.orchestra.run.smithi049.stdout:blocklist 172.21.15.49:0/3553734432 expires 2024-09-22T13:49:27.311313+0000 2024-09-21T13:50:00.441 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:00 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:50:00.441 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:00 smithi049 ceph-mon[40489]: osdmap e81: 8 total, 8 up, 8 in 2024-09-21T13:50:00.442 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:00 smithi049 ceph-mon[40489]: 3.6 scrub starts 2024-09-21T13:50:00.442 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:00 smithi049 ceph-mon[40489]: 3.6 scrub ok 2024-09-21T13:50:00.442 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:00 smithi049 ceph-mon[40489]: pgmap v25: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T13:50:00.442 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:00 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2097059434' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-21T13:50:00.442 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:00 smithi049 ceph-mon[40489]: osdmap e82: 8 total, 8 up, 8 in 2024-09-21T13:50:00.442 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:00 smithi049 ceph-mon[40489]: overall HEALTH_OK 2024-09-21T13:50:00.442 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:00 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:50:00.442 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:00 smithi049 ceph-mon[30080]: osdmap e81: 8 total, 8 up, 8 in 2024-09-21T13:50:00.442 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:00 smithi049 ceph-mon[30080]: 3.6 scrub starts 2024-09-21T13:50:00.442 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:00 smithi049 ceph-mon[30080]: 3.6 scrub ok 2024-09-21T13:50:00.442 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:00 smithi049 ceph-mon[30080]: pgmap v25: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T13:50:00.443 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:00 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2097059434' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-21T13:50:00.443 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:00 smithi049 ceph-mon[30080]: osdmap e82: 8 total, 8 up, 8 in 2024-09-21T13:50:00.443 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:00 smithi049 ceph-mon[30080]: overall HEALTH_OK 2024-09-21T13:50:00.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:00 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:50:00.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:00 smithi187 ceph-mon[33032]: osdmap e81: 8 total, 8 up, 8 in 2024-09-21T13:50:00.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:00 smithi187 ceph-mon[33032]: 3.6 scrub starts 2024-09-21T13:50:00.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:00 smithi187 ceph-mon[33032]: 3.6 scrub ok 2024-09-21T13:50:00.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:00 smithi187 ceph-mon[33032]: pgmap v25: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T13:50:00.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:00 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2097059434' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-21T13:50:00.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:00 smithi187 ceph-mon[33032]: osdmap e82: 8 total, 8 up, 8 in 2024-09-21T13:50:00.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:00 smithi187 ceph-mon[33032]: overall HEALTH_OK 2024-09-21T13:50:00.470 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 66 down_at 0 last_clean_interval [0,0) [v2:172.21.15.49:6802/2095561412,v1:172.21.15.49:6803/2095561412] [v2:172.21.15.49:6804/2095561412,v1:172.21.15.49:6805/2095561412] exists,up 2595dca9-6219-4e0c-8728-751eb102e73a', 'osd.1 up in weight 1 up_from 13 up_thru 66 down_at 0 last_clean_interval [0,0) [v2:172.21.15.49:6810/1382840527,v1:172.21.15.49:6811/1382840527] [v2:172.21.15.49:6812/1382840527,v1:172.21.15.49:6813/1382840527] exists,up 0ba327f9-5524-4a40-abcc-729ded0b1917', 'osd.2 up in weight 1 up_from 18 up_thru 66 down_at 0 last_clean_interval [0,0) [v2:172.21.15.49:6818/2189046898,v1:172.21.15.49:6819/2189046898] [v2:172.21.15.49:6820/2189046898,v1:172.21.15.49:6821/2189046898] exists,up 27a45750-204d-48fd-bb8f-ebb01273343d', 'osd.3 up in weight 1 up_from 27 up_thru 66 down_at 0 last_clean_interval [0,0) [v2:172.21.15.49:6826/1981113123,v1:172.21.15.49:6827/1981113123] [v2:172.21.15.49:6828/1981113123,v1:172.21.15.49:6829/1981113123] exists,up 528bc390-3dc9-4f3b-ab75-d7412c9625db', 'osd.4 up in weight 1 up_from 33 up_thru 66 down_at 0 last_clean_interval [0,0) [v2:172.21.15.187:6800/1702447776,v1:172.21.15.187:6801/1702447776] [v2:172.21.15.187:6802/1702447776,v1:172.21.15.187:6803/1702447776] exists,up 6fc64698-2d37-4db0-bf3d-c7e612c45163', 'osd.5 up in weight 1 up_from 39 up_thru 66 down_at 0 last_clean_interval [0,0) [v2:172.21.15.187:6808/3659050776,v1:172.21.15.187:6809/3659050776] [v2:172.21.15.187:6810/3659050776,v1:172.21.15.187:6811/3659050776] exists,up 8d7f62a1-7e92-4e06-9cad-86b96be514d8', 'osd.6 up in weight 1 up_from 47 up_thru 64 down_at 0 last_clean_interval [0,0) [v2:172.21.15.187:6816/2728426525,v1:172.21.15.187:6817/2728426525] [v2:172.21.15.187:6818/2728426525,v1:172.21.15.187:6819/2728426525] exists,up 3a7afd13-26ce-444c-9cc0-81ce2eab6ce5', 'osd.7 up in weight 1 up_from 54 up_thru 66 down_at 0 last_clean_interval [0,0) [v2:172.21.15.187:6824/3680850037,v1:172.21.15.187:6825/3680850037] [v2:172.21.15.187:6826/3680850037,v1:172.21.15.187:6827/3680850037] exists,up 1e08d891-8747-402e-b4c2-aa3802ce8537'] 2024-09-21T13:50:00.471 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon mon.a config show 2024-09-21T13:50:00.728 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:50:00.754 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:50:00 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:50:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:50:01.221 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T13:50:01.221 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "mon.a", 2024-09-21T13:50:01.221 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster": "ceph", 2024-09-21T13:50:01.221 INFO:teuthology.orchestra.run.smithi049.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-21T13:50:01.221 INFO:teuthology.orchestra.run.smithi049.stdout: "admin_socket_mode": "", 2024-09-21T13:50:01.221 INFO:teuthology.orchestra.run.smithi049.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-09-21T13:50:01.221 INFO:teuthology.orchestra.run.smithi049.stdout: "auth_client_required": "cephx, none", 2024-09-21T13:50:01.221 INFO:teuthology.orchestra.run.smithi049.stdout: "auth_cluster_required": "cephx", 2024-09-21T13:50:01.221 INFO:teuthology.orchestra.run.smithi049.stdout: "auth_debug": "true", 2024-09-21T13:50:01.222 INFO:teuthology.orchestra.run.smithi049.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-21T13:50:01.222 INFO:teuthology.orchestra.run.smithi049.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-21T13:50:01.222 INFO:teuthology.orchestra.run.smithi049.stdout: "auth_service_required": "cephx", 2024-09-21T13:50:01.222 INFO:teuthology.orchestra.run.smithi049.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-21T13:50:01.222 INFO:teuthology.orchestra.run.smithi049.stdout: "auth_supported": "", 2024-09-21T13:50:01.222 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_aio": "true", 2024-09-21T13:50:01.222 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-21T13:50:01.222 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_aio_poll_ms": "250", 2024-09-21T13:50:01.222 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_aio_reap_max": "16", 2024-09-21T13:50:01.222 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_async_discard_threads": "0", 2024-09-21T13:50:01.222 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_block_size": "4096", 2024-09-21T13:50:01.222 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_debug_aio": "false", 2024-09-21T13:50:01.222 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-21T13:50:01.223 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-21T13:50:01.223 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_debug_inflight_ios": "false", 2024-09-21T13:50:01.223 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_enable_discard": "false", 2024-09-21T13:50:01.223 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_flock_retry": "3", 2024-09-21T13:50:01.223 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-21T13:50:01.223 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_inject_crash": "0", 2024-09-21T13:50:01.223 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-21T13:50:01.223 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_ioring": "false", 2024-09-21T13:50:01.223 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_ioring_hipri": "false", 2024-09-21T13:50:01.223 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-21T13:50:01.223 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-21T13:50:01.223 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-21T13:50:01.223 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-21T13:50:01.224 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_stalled_read_warn_lifetime": "86400", 2024-09-21T13:50:01.224 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_stalled_read_warn_threshold": "1", 2024-09-21T13:50:01.224 INFO:teuthology.orchestra.run.smithi049.stdout: "bdev_type": "", 2024-09-21T13:50:01.224 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_alloc_size": "1048576", 2024-09-21T13:50:01.224 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_allocator": "hybrid", 2024-09-21T13:50:01.224 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_buffered_io": "true", 2024-09-21T13:50:01.224 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_check_for_zeros": "false", 2024-09-21T13:50:01.224 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-21T13:50:01.224 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-21T13:50:01.224 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_compact_log_sync": "false", 2024-09-21T13:50:01.224 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-21T13:50:01.224 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-21T13:50:01.224 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-21T13:50:01.224 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-21T13:50:01.225 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_max_log_runway": "4194304", 2024-09-21T13:50:01.225 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_max_prefetch": "1048576", 2024-09-21T13:50:01.225 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_min_flush_size": "524288", 2024-09-21T13:50:01.225 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_min_log_runway": "1048576", 2024-09-21T13:50:01.225 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_replay_recovery": "false", 2024-09-21T13:50:01.225 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-21T13:50:01.225 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-21T13:50:01.225 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_sync_write": "false", 2024-09-21T13:50:01.225 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-21T13:50:01.225 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-21T13:50:01.225 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-21T13:50:01.225 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_allocation_from_file": "true", 2024-09-21T13:50:01.225 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_allocator": "hybrid", 2024-09-21T13:50:01.225 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_async_db_compaction": "true", 2024-09-21T13:50:01.226 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-21T13:50:01.226 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-21T13:50:01.226 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-21T13:50:01.226 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-21T13:50:01.226 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_bdev_label_multi": "true", 2024-09-21T13:50:01.226 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_bdev_label_multi_upgrade": "false", 2024-09-21T13:50:01.226 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_bdev_label_require_all": "true", 2024-09-21T13:50:01.226 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-21T13:50:01.226 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-21T13:50:01.226 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-21T13:50:01.226 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_block_create": "true", 2024-09-21T13:50:01.226 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_block_db_create": "false", 2024-09-21T13:50:01.226 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_block_db_path": "", 2024-09-21T13:50:01.226 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_block_db_size": "0", 2024-09-21T13:50:01.227 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_block_path": "", 2024-09-21T13:50:01.227 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_block_preallocate_file": "false", 2024-09-21T13:50:01.227 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_block_size": "107374182400", 2024-09-21T13:50:01.227 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_block_wal_create": "false", 2024-09-21T13:50:01.227 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_block_wal_path": "", 2024-09-21T13:50:01.227 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_block_wal_size": "100663296", 2024-09-21T13:50:01.227 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_bluefs": "true", 2024-09-21T13:50:01.227 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-21T13:50:01.227 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-21T13:50:01.227 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-21T13:50:01.227 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_btree2_alloc_weight_factor": "2.000000", 2024-09-21T13:50:01.227 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-21T13:50:01.227 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-21T13:50:01.227 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-21T13:50:01.228 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-21T13:50:01.228 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-21T13:50:01.228 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_autotune": "true", 2024-09-21T13:50:01.228 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-21T13:50:01.228 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-21T13:50:01.228 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-21T13:50:01.228 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-21T13:50:01.228 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_size": "0", 2024-09-21T13:50:01.228 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-21T13:50:01.228 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-21T13:50:01.228 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-21T13:50:01.228 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-21T13:50:01.228 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_type": "2q", 2024-09-21T13:50:01.229 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-21T13:50:01.229 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_clone_cow": "true", 2024-09-21T13:50:01.229 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-21T13:50:01.229 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-21T13:50:01.229 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-21T13:50:01.229 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-21T13:50:01.229 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-21T13:50:01.229 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-21T13:50:01.229 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-21T13:50:01.229 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_compression_mode": "none", 2024-09-21T13:50:01.229 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-21T13:50:01.229 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_csum_type": "crc32c", 2024-09-21T13:50:01.229 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-09-21T13:50:01.229 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-21T13:50:01.230 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_debug_freelist": "false", 2024-09-21T13:50:01.230 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-21T13:50:01.230 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-21T13:50:01.230 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-21T13:50:01.230 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-21T13:50:01.230 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-21T13:50:01.230 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-21T13:50:01.230 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-21T13:50:01.230 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-21T13:50:01.230 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-21T13:50:01.230 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-21T13:50:01.230 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-21T13:50:01.231 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-21T13:50:01.231 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_debug_small_allocations": "0", 2024-09-21T13:50:01.231 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-21T13:50:01.231 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_default_buffered_read": "true", 2024-09-21T13:50:01.231 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_default_buffered_write": "false", 2024-09-21T13:50:01.231 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-21T13:50:01.231 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-21T13:50:01.231 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-21T13:50:01.231 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_elastic_shared_blobs": "true", 2024-09-21T13:50:01.231 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-21T13:50:01.231 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-21T13:50:01.231 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-21T13:50:01.231 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-21T13:50:01.231 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-21T13:50:01.231 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_fail_eio": "false", 2024-09-21T13:50:01.232 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-21T13:50:01.232 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-21T13:50:01.232 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-21T13:50:01.232 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-21T13:50:01.232 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-21T13:50:01.232 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-21T13:50:01.232 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_fsck_on_mount": "false", 2024-09-21T13:50:01.232 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-21T13:50:01.232 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_fsck_on_umount": "false", 2024-09-21T13:50:01.232 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-21T13:50:01.232 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-21T13:50:01.233 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-21T13:50:01.233 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-21T13:50:01.233 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-21T13:50:01.233 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-21T13:50:01.233 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-21T13:50:01.233 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-21T13:50:01.233 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_ignore_data_csum": "false", 2024-09-21T13:50:01.233 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-21T13:50:01.233 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-21T13:50:01.233 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-21T13:50:01.233 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-21T13:50:01.233 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_log_op_age": "5.000000", 2024-09-21T13:50:01.233 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_max_alloc_size": "0", 2024-09-21T13:50:01.234 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_max_blob_size": "0", 2024-09-21T13:50:01.234 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-21T13:50:01.234 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-21T13:50:01.234 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-21T13:50:01.234 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_max_deferred_txc": "32", 2024-09-21T13:50:01.234 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_min_alloc_size": "0", 2024-09-21T13:50:01.234 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-21T13:50:01.234 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-21T13:50:01.235 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_nid_prealloc": "1024", 2024-09-21T13:50:01.235 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-21T13:50:01.235 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-21T13:50:01.235 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-21T13:50:01.235 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-21T13:50:01.235 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_retry_disk_reads": "3", 2024-09-21T13:50:01.235 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_rocksdb_cf": "true", 2024-09-21T13:50:01.235 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-09-21T13:50:01.235 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-09-21T13:50:01.235 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-21T13:50:01.235 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_slow_ops_warn_lifetime": "86400", 2024-09-21T13:50:01.235 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_slow_ops_warn_threshold": "1", 2024-09-21T13:50:01.236 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-21T13:50:01.236 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-21T13:50:01.236 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-21T13:50:01.236 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_spdk_mem": "512", 2024-09-21T13:50:01.236 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-21T13:50:01.236 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-21T13:50:01.236 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-21T13:50:01.236 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-21T13:50:01.236 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-21T13:50:01.236 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-21T13:50:01.236 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-21T13:50:01.236 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_tracing": "false", 2024-09-21T13:50:01.236 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-21T13:50:01.237 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-21T13:50:01.237 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-21T13:50:01.237 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-21T13:50:01.237 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-21T13:50:01.237 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-21T13:50:01.237 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-21T13:50:01.237 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-21T13:50:01.237 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-21T13:50:01.237 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_write_v2": "false", 2024-09-21T13:50:01.237 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_write_v2_random": "false", 2024-09-21T13:50:01.237 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_zero_block_detection": "false", 2024-09-21T13:50:01.237 INFO:teuthology.orchestra.run.smithi049.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-21T13:50:01.237 INFO:teuthology.orchestra.run.smithi049.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-21T13:50:01.237 INFO:teuthology.orchestra.run.smithi049.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-21T13:50:01.238 INFO:teuthology.orchestra.run.smithi049.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-21T13:50:01.238 INFO:teuthology.orchestra.run.smithi049.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-21T13:50:01.238 INFO:teuthology.orchestra.run.smithi049.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-21T13:50:01.238 INFO:teuthology.orchestra.run.smithi049.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-21T13:50:01.238 INFO:teuthology.orchestra.run.smithi049.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-09-21T13:50:01.238 INFO:teuthology.orchestra.run.smithi049.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-21T13:50:01.238 INFO:teuthology.orchestra.run.smithi049.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-21T13:50:01.238 INFO:teuthology.orchestra.run.smithi049.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-21T13:50:01.238 INFO:teuthology.orchestra.run.smithi049.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-21T13:50:01.238 INFO:teuthology.orchestra.run.smithi049.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-21T13:50:01.238 INFO:teuthology.orchestra.run.smithi049.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-21T13:50:01.238 INFO:teuthology.orchestra.run.smithi049.stdout: "cephx_cluster_require_signatures": "false", 2024-09-21T13:50:01.239 INFO:teuthology.orchestra.run.smithi049.stdout: "cephx_cluster_require_version": "2", 2024-09-21T13:50:01.241 INFO:teuthology.orchestra.run.smithi049.stdout: "cephx_require_signatures": "false", 2024-09-21T13:50:01.241 INFO:teuthology.orchestra.run.smithi049.stdout: "cephx_require_version": "2", 2024-09-21T13:50:01.241 INFO:teuthology.orchestra.run.smithi049.stdout: "cephx_service_require_signatures": "false", 2024-09-21T13:50:01.242 INFO:teuthology.orchestra.run.smithi049.stdout: "cephx_service_require_version": "2", 2024-09-21T13:50:01.242 INFO:teuthology.orchestra.run.smithi049.stdout: "cephx_sign_messages": "true", 2024-09-21T13:50:01.242 INFO:teuthology.orchestra.run.smithi049.stdout: "chdir": "", 2024-09-21T13:50:01.242 INFO:teuthology.orchestra.run.smithi049.stdout: "client_acl_type": "", 2024-09-21T13:50:01.242 INFO:teuthology.orchestra.run.smithi049.stdout: "client_asio_thread_count": "2", 2024-09-21T13:50:01.242 INFO:teuthology.orchestra.run.smithi049.stdout: "client_cache_mid": "0.750000", 2024-09-21T13:50:01.242 INFO:teuthology.orchestra.run.smithi049.stdout: "client_cache_size": "16384", 2024-09-21T13:50:01.242 INFO:teuthology.orchestra.run.smithi049.stdout: "client_caps_release_delay": "5", 2024-09-21T13:50:01.242 INFO:teuthology.orchestra.run.smithi049.stdout: "client_check_pool_perm": "true", 2024-09-21T13:50:01.242 INFO:teuthology.orchestra.run.smithi049.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-21T13:50:01.242 INFO:teuthology.orchestra.run.smithi049.stdout: "client_debug_force_sync_read": "false", 2024-09-21T13:50:01.242 INFO:teuthology.orchestra.run.smithi049.stdout: "client_debug_getattr_caps": "false", 2024-09-21T13:50:01.242 INFO:teuthology.orchestra.run.smithi049.stdout: "client_debug_inject_features": "", 2024-09-21T13:50:01.243 INFO:teuthology.orchestra.run.smithi049.stdout: "client_debug_inject_tick_delay": "0", 2024-09-21T13:50:01.243 INFO:teuthology.orchestra.run.smithi049.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-21T13:50:01.243 INFO:teuthology.orchestra.run.smithi049.stdout: "client_die_on_failed_remount": "false", 2024-09-21T13:50:01.243 INFO:teuthology.orchestra.run.smithi049.stdout: "client_dirsize_rbytes": "true", 2024-09-21T13:50:01.243 INFO:teuthology.orchestra.run.smithi049.stdout: "client_force_lazyio": "false", 2024-09-21T13:50:01.243 INFO:teuthology.orchestra.run.smithi049.stdout: "client_fs": "", 2024-09-21T13:50:01.243 INFO:teuthology.orchestra.run.smithi049.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-21T13:50:01.243 INFO:teuthology.orchestra.run.smithi049.stdout: "client_inject_release_failure": "false", 2024-09-21T13:50:01.243 INFO:teuthology.orchestra.run.smithi049.stdout: "client_max_inline_size": "4096", 2024-09-21T13:50:01.243 INFO:teuthology.orchestra.run.smithi049.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-21T13:50:01.243 INFO:teuthology.orchestra.run.smithi049.stdout: "client_mds_namespace": "", 2024-09-21T13:50:01.243 INFO:teuthology.orchestra.run.smithi049.stdout: "client_metadata": "", 2024-09-21T13:50:01.243 INFO:teuthology.orchestra.run.smithi049.stdout: "client_mount_gid": "-1", 2024-09-21T13:50:01.243 INFO:teuthology.orchestra.run.smithi049.stdout: "client_mount_timeout": "300", 2024-09-21T13:50:01.244 INFO:teuthology.orchestra.run.smithi049.stdout: "client_mount_uid": "-1", 2024-09-21T13:50:01.244 INFO:teuthology.orchestra.run.smithi049.stdout: "client_mountpoint": "/", 2024-09-21T13:50:01.244 INFO:teuthology.orchestra.run.smithi049.stdout: "client_notify_timeout": "10", 2024-09-21T13:50:01.244 INFO:teuthology.orchestra.run.smithi049.stdout: "client_oc": "true", 2024-09-21T13:50:01.244 INFO:teuthology.orchestra.run.smithi049.stdout: "client_oc_max_dirty": "104857600", 2024-09-21T13:50:01.244 INFO:teuthology.orchestra.run.smithi049.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-21T13:50:01.244 INFO:teuthology.orchestra.run.smithi049.stdout: "client_oc_max_objects": "1000", 2024-09-21T13:50:01.244 INFO:teuthology.orchestra.run.smithi049.stdout: "client_oc_size": "209715200", 2024-09-21T13:50:01.244 INFO:teuthology.orchestra.run.smithi049.stdout: "client_oc_target_dirty": "8388608", 2024-09-21T13:50:01.244 INFO:teuthology.orchestra.run.smithi049.stdout: "client_permissions": "true", 2024-09-21T13:50:01.244 INFO:teuthology.orchestra.run.smithi049.stdout: "client_quota": "true", 2024-09-21T13:50:01.244 INFO:teuthology.orchestra.run.smithi049.stdout: "client_quota_df": "true", 2024-09-21T13:50:01.245 INFO:teuthology.orchestra.run.smithi049.stdout: "client_readahead_max_bytes": "0", 2024-09-21T13:50:01.245 INFO:teuthology.orchestra.run.smithi049.stdout: "client_readahead_max_periods": "4", 2024-09-21T13:50:01.245 INFO:teuthology.orchestra.run.smithi049.stdout: "client_readahead_min": "131072", 2024-09-21T13:50:01.245 INFO:teuthology.orchestra.run.smithi049.stdout: "client_reconnect_stale": "false", 2024-09-21T13:50:01.245 INFO:teuthology.orchestra.run.smithi049.stdout: "client_shutdown_timeout": "30", 2024-09-21T13:50:01.245 INFO:teuthology.orchestra.run.smithi049.stdout: "client_snapdir": ".snap", 2024-09-21T13:50:01.245 INFO:teuthology.orchestra.run.smithi049.stdout: "client_tick_interval": "1", 2024-09-21T13:50:01.245 INFO:teuthology.orchestra.run.smithi049.stdout: "client_trace": "", 2024-09-21T13:50:01.245 INFO:teuthology.orchestra.run.smithi049.stdout: "client_try_dentry_invalidate": "false", 2024-09-21T13:50:01.245 INFO:teuthology.orchestra.run.smithi049.stdout: "client_use_faked_inos": "false", 2024-09-21T13:50:01.245 INFO:teuthology.orchestra.run.smithi049.stdout: "client_use_random_mds": "false", 2024-09-21T13:50:01.245 INFO:teuthology.orchestra.run.smithi049.stdout: "clog_to_graylog": "false", 2024-09-21T13:50:01.245 INFO:teuthology.orchestra.run.smithi049.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-21T13:50:01.245 INFO:teuthology.orchestra.run.smithi049.stdout: "clog_to_graylog_port": "12201", 2024-09-21T13:50:01.246 INFO:teuthology.orchestra.run.smithi049.stdout: "clog_to_monitors": "default=true", 2024-09-21T13:50:01.246 INFO:teuthology.orchestra.run.smithi049.stdout: "clog_to_syslog": "false", 2024-09-21T13:50:01.246 INFO:teuthology.orchestra.run.smithi049.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-21T13:50:01.246 INFO:teuthology.orchestra.run.smithi049.stdout: "clog_to_syslog_level": "info", 2024-09-21T13:50:01.246 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "-", 2024-09-21T13:50:01.246 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_network": "", 2024-09-21T13:50:01.246 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_network_interface": "", 2024-09-21T13:50:01.246 INFO:teuthology.orchestra.run.smithi049.stdout: "compressor_zlib_isal": "false", 2024-09-21T13:50:01.246 INFO:teuthology.orchestra.run.smithi049.stdout: "compressor_zlib_level": "5", 2024-09-21T13:50:01.246 INFO:teuthology.orchestra.run.smithi049.stdout: "compressor_zlib_winsize": "-15", 2024-09-21T13:50:01.246 INFO:teuthology.orchestra.run.smithi049.stdout: "compressor_zstd_level": "1", 2024-09-21T13:50:01.246 INFO:teuthology.orchestra.run.smithi049.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5", 2024-09-21T13:50:01.246 INFO:teuthology.orchestra.run.smithi049.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-21T13:50:01.247 INFO:teuthology.orchestra.run.smithi049.stdout: "crimson_alien_op_num_threads": "6", 2024-09-21T13:50:01.247 INFO:teuthology.orchestra.run.smithi049.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-21T13:50:01.247 INFO:teuthology.orchestra.run.smithi049.stdout: "crimson_osd_obc_lru_size": "512", 2024-09-21T13:50:01.247 INFO:teuthology.orchestra.run.smithi049.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-21T13:50:01.247 INFO:teuthology.orchestra.run.smithi049.stdout: "crimson_osd_stat_interval": "0", 2024-09-21T13:50:01.247 INFO:teuthology.orchestra.run.smithi049.stdout: "crimson_seastar_cpu_cores": "", 2024-09-21T13:50:01.247 INFO:teuthology.orchestra.run.smithi049.stdout: "crimson_seastar_num_threads": "0", 2024-09-21T13:50:01.247 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_location": "", 2024-09-21T13:50:01.247 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_location_hook": "", 2024-09-21T13:50:01.247 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_location_hook_timeout": "10", 2024-09-21T13:50:01.247 INFO:teuthology.orchestra.run.smithi049.stdout: "daemonize": "false", 2024-09-21T13:50:01.247 INFO:teuthology.orchestra.run.smithi049.stdout: "daos_pool": "tank", 2024-09-21T13:50:01.247 INFO:teuthology.orchestra.run.smithi049.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-09-21T13:50:01.248 INFO:teuthology.orchestra.run.smithi049.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-09-21T13:50:01.248 INFO:teuthology.orchestra.run.smithi049.stdout: "dbstore_db_name_prefix": "dbstore", 2024-09-21T13:50:01.248 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_alienstore": "0/5", 2024-09-21T13:50:01.248 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_allow_any_pool_priority": "false", 2024-09-21T13:50:01.248 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_asok": "1/5", 2024-09-21T13:50:01.248 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_asok_assert_abort": "false", 2024-09-21T13:50:01.248 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_asserts_on_shutdown": "true", 2024-09-21T13:50:01.248 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_auth": "1/5", 2024-09-21T13:50:01.248 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_bdev": "1/3", 2024-09-21T13:50:01.248 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_bluefs": "1/5", 2024-09-21T13:50:01.248 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_bluestore": "1/5", 2024-09-21T13:50:01.248 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_buffer": "0/1", 2024-09-21T13:50:01.248 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_ceph_dedup": "0/5", 2024-09-21T13:50:01.248 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_ceph_exporter": "1/5", 2024-09-21T13:50:01.248 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_cephfs_mirror": "0/5", 2024-09-21T13:50:01.249 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_cephsqlite": "0/5", 2024-09-21T13:50:01.249 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_client": "0/5", 2024-09-21T13:50:01.249 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_compressor": "1/5", 2024-09-21T13:50:01.249 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_context": "0/1", 2024-09-21T13:50:01.249 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_crimson_interrupt": "0/5", 2024-09-21T13:50:01.249 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_crush": "1/1", 2024-09-21T13:50:01.249 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_crypto": "1/5", 2024-09-21T13:50:01.249 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_cyanstore": "0/5", 2024-09-21T13:50:01.249 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_deliberately_leak_memory": "false", 2024-09-21T13:50:01.249 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_disable_randomized_ping": "false", 2024-09-21T13:50:01.249 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_dpdk": "1/5", 2024-09-21T13:50:01.249 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_eventtrace": "1/5", 2024-09-21T13:50:01.249 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_filer": "0/1", 2024-09-21T13:50:01.250 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_filestore": "1/3", 2024-09-21T13:50:01.250 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_finisher": "1/1", 2024-09-21T13:50:01.250 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_fuse": "1/5", 2024-09-21T13:50:01.250 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_heartbeat_testing_span": "0", 2024-09-21T13:50:01.250 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_heartbeatmap": "1/5", 2024-09-21T13:50:01.250 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-21T13:50:01.250 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_javaclient": "1/5", 2024-09-21T13:50:01.250 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_journal": "1/3", 2024-09-21T13:50:01.250 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_journaler": "0/5", 2024-09-21T13:50:01.250 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_kstore": "1/5", 2024-09-21T13:50:01.250 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_lockdep": "0/1", 2024-09-21T13:50:01.250 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_mclock": "1/5", 2024-09-21T13:50:01.250 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_mds": "1/5", 2024-09-21T13:50:01.250 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_mds_balancer": "1/5", 2024-09-21T13:50:01.251 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_mds_locker": "1/5", 2024-09-21T13:50:01.251 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_mds_log": "1/5", 2024-09-21T13:50:01.251 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_mds_log_expire": "1/5", 2024-09-21T13:50:01.251 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_mds_migrator": "1/5", 2024-09-21T13:50:01.251 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_mds_quiesce": "3/5", 2024-09-21T13:50:01.251 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_memstore": "1/5", 2024-09-21T13:50:01.251 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_mgr": "2/5", 2024-09-21T13:50:01.251 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_mgrc": "1/5", 2024-09-21T13:50:01.251 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_mon": "20/20", 2024-09-21T13:50:01.251 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_monc": "0/10", 2024-09-21T13:50:01.251 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_ms": "1/1", 2024-09-21T13:50:01.251 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_none": "0/5", 2024-09-21T13:50:01.251 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_objclass": "0/5", 2024-09-21T13:50:01.251 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_objectcacher": "0/5", 2024-09-21T13:50:01.252 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_objecter": "0/1", 2024-09-21T13:50:01.252 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_optracker": "0/5", 2024-09-21T13:50:01.252 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_osd": "1/5", 2024-09-21T13:50:01.252 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_paxos": "20/20", 2024-09-21T13:50:01.252 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_perfcounter": "1/5", 2024-09-21T13:50:01.252 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_prioritycache": "1/5", 2024-09-21T13:50:01.252 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_rados": "0/5", 2024-09-21T13:50:01.252 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_rbd": "0/5", 2024-09-21T13:50:01.252 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_rbd_mirror": "0/5", 2024-09-21T13:50:01.252 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_rbd_pwl": "0/5", 2024-09-21T13:50:01.252 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_rbd_replay": "0/5", 2024-09-21T13:50:01.252 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_refs": "0/0", 2024-09-21T13:50:01.252 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_reserver": "1/1", 2024-09-21T13:50:01.252 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_rgw": "1/5", 2024-09-21T13:50:01.253 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_rgw_access": "1/5", 2024-09-21T13:50:01.253 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_rgw_datacache": "1/5", 2024-09-21T13:50:01.253 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_rgw_dbstore": "1/5", 2024-09-21T13:50:01.253 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_rgw_flight": "1/5", 2024-09-21T13:50:01.253 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_rgw_lifecycle": "1/5", 2024-09-21T13:50:01.253 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_rgw_notification": "1/5", 2024-09-21T13:50:01.253 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_rgw_sync": "1/5", 2024-09-21T13:50:01.253 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_rocksdb": "4/5", 2024-09-21T13:50:01.253 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_seastore": "0/5", 2024-09-21T13:50:01.253 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_seastore_backref": "0/5", 2024-09-21T13:50:01.253 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_seastore_cache": "0/5", 2024-09-21T13:50:01.253 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_seastore_cleaner": "0/5", 2024-09-21T13:50:01.253 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_seastore_device": "0/5", 2024-09-21T13:50:01.254 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_seastore_epm": "0/5", 2024-09-21T13:50:01.254 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-09-21T13:50:01.254 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_seastore_journal": "0/5", 2024-09-21T13:50:01.254 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_seastore_lba": "0/5", 2024-09-21T13:50:01.254 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_seastore_odata": "0/5", 2024-09-21T13:50:01.254 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_seastore_omap": "0/5", 2024-09-21T13:50:01.254 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_seastore_onode": "0/5", 2024-09-21T13:50:01.254 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_seastore_t": "0/5", 2024-09-21T13:50:01.254 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_seastore_tm": "0/5", 2024-09-21T13:50:01.254 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_striper": "0/1", 2024-09-21T13:50:01.254 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_test": "0/5", 2024-09-21T13:50:01.254 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_throttle": "1/1", 2024-09-21T13:50:01.254 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_timer": "0/1", 2024-09-21T13:50:01.255 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_tp": "0/5", 2024-09-21T13:50:01.255 INFO:teuthology.orchestra.run.smithi049.stdout: "debug_trace": "1/5", 2024-09-21T13:50:01.255 INFO:teuthology.orchestra.run.smithi049.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-21T13:50:01.255 INFO:teuthology.orchestra.run.smithi049.stdout: "device_failure_prediction_mode": "none", 2024-09-21T13:50:01.255 INFO:teuthology.orchestra.run.smithi049.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-09-21T13:50:01.255 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-09-21T13:50:01.255 INFO:teuthology.orchestra.run.smithi049.stdout: "err_to_graylog": "false", 2024-09-21T13:50:01.255 INFO:teuthology.orchestra.run.smithi049.stdout: "err_to_journald": "false", 2024-09-21T13:50:01.255 INFO:teuthology.orchestra.run.smithi049.stdout: "err_to_stderr": "true", 2024-09-21T13:50:01.255 INFO:teuthology.orchestra.run.smithi049.stdout: "err_to_syslog": "false", 2024-09-21T13:50:01.255 INFO:teuthology.orchestra.run.smithi049.stdout: "event_tracing": "false", 2024-09-21T13:50:01.255 INFO:teuthology.orchestra.run.smithi049.stdout: "exporter_addr": "0.0.0.0", 2024-09-21T13:50:01.255 INFO:teuthology.orchestra.run.smithi049.stdout: "exporter_cert_file": "", 2024-09-21T13:50:01.255 INFO:teuthology.orchestra.run.smithi049.stdout: "exporter_http_port": "9926", 2024-09-21T13:50:01.256 INFO:teuthology.orchestra.run.smithi049.stdout: "exporter_key_file": "", 2024-09-21T13:50:01.256 INFO:teuthology.orchestra.run.smithi049.stdout: "exporter_prio_limit": "5", 2024-09-21T13:50:01.256 INFO:teuthology.orchestra.run.smithi049.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-21T13:50:01.256 INFO:teuthology.orchestra.run.smithi049.stdout: "exporter_sort_metrics": "true", 2024-09-21T13:50:01.256 INFO:teuthology.orchestra.run.smithi049.stdout: "exporter_stats_period": "5", 2024-09-21T13:50:01.256 INFO:teuthology.orchestra.run.smithi049.stdout: "fake_statfs_for_testing": "0", 2024-09-21T13:50:01.256 INFO:teuthology.orchestra.run.smithi049.stdout: "fatal_signal_handlers": "true", 2024-09-21T13:50:01.256 INFO:teuthology.orchestra.run.smithi049.stdout: "filer_max_purge_ops": "10", 2024-09-21T13:50:01.256 INFO:teuthology.orchestra.run.smithi049.stdout: "filer_max_truncate_ops": "128", 2024-09-21T13:50:01.256 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_apply_finisher_threads": "1", 2024-09-21T13:50:01.256 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_blackhole": "false", 2024-09-21T13:50:01.256 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_btrfs_clone_range": "true", 2024-09-21T13:50:01.256 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_btrfs_snap": "true", 2024-09-21T13:50:01.257 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_caller_concurrency": "10", 2024-09-21T13:50:01.257 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_collect_device_partition_information": "true", 2024-09-21T13:50:01.257 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_commit_timeout": "600.000000", 2024-09-21T13:50:01.257 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_debug_inject_read_err": "false", 2024-09-21T13:50:01.257 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_debug_omap_check": "false", 2024-09-21T13:50:01.257 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_debug_verify_split": "false", 2024-09-21T13:50:01.257 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_dump_file": "", 2024-09-21T13:50:01.257 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-21T13:50:01.257 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-21T13:50:01.257 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_fadvise": "true", 2024-09-21T13:50:01.257 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_fail_eio": "true", 2024-09-21T13:50:01.257 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_fd_cache_shards": "16", 2024-09-21T13:50:01.257 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_fd_cache_size": "128", 2024-09-21T13:50:01.258 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_fiemap": "false", 2024-09-21T13:50:01.258 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_fiemap_threshold": "4096", 2024-09-21T13:50:01.258 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-21T13:50:01.258 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-21T13:50:01.258 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_inject_stall": "0", 2024-09-21T13:50:01.258 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_journal_parallel": "false", 2024-09-21T13:50:01.258 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_journal_trailing": "false", 2024-09-21T13:50:01.258 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_journal_writeahead": "false", 2024-09-21T13:50:01.258 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_kill_at": "0", 2024-09-21T13:50:01.258 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-21T13:50:01.258 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-21T13:50:01.259 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-21T13:50:01.259 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-21T13:50:01.259 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-21T13:50:01.259 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_max_inline_xattrs": "0", 2024-09-21T13:50:01.259 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-21T13:50:01.259 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-21T13:50:01.259 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-21T13:50:01.259 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-21T13:50:01.259 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_max_xattr_value_size": "0", 2024-09-21T13:50:01.259 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-21T13:50:01.259 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-21T13:50:01.259 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-21T13:50:01.259 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_merge_threshold": "-10", 2024-09-21T13:50:01.260 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-21T13:50:01.260 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_odsync_write": "false", 2024-09-21T13:50:01.260 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_omap_backend": "rocksdb", 2024-09-21T13:50:01.260 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_omap_backend_path": "", 2024-09-21T13:50:01.260 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-21T13:50:01.260 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-21T13:50:01.260 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-21T13:50:01.260 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_op_thread_timeout": "60", 2024-09-21T13:50:01.260 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_op_threads": "2", 2024-09-21T13:50:01.260 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_punch_hole": "false", 2024-09-21T13:50:01.260 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-21T13:50:01.260 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-21T13:50:01.260 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-21T13:50:01.261 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-21T13:50:01.261 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-21T13:50:01.261 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-21T13:50:01.261 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-21T13:50:01.261 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-21T13:50:01.261 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-21T13:50:01.261 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_queue_max_ops": "50", 2024-09-21T13:50:01.261 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-21T13:50:01.261 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_seek_data_hole": "false", 2024-09-21T13:50:01.261 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_sloppy_crc": "false", 2024-09-21T13:50:01.261 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-21T13:50:01.261 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_splice": "false", 2024-09-21T13:50:01.261 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_split_multiple": "2", 2024-09-21T13:50:01.261 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_split_rand_factor": "20", 2024-09-21T13:50:01.262 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_update_to": "1000", 2024-09-21T13:50:01.262 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-21T13:50:01.262 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-21T13:50:01.262 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-21T13:50:01.262 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-21T13:50:01.262 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-21T13:50:01.262 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-21T13:50:01.262 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_wbthrottle_enable": "true", 2024-09-21T13:50:01.262 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-21T13:50:01.262 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-21T13:50:01.262 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-21T13:50:01.262 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-21T13:50:01.262 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-21T13:50:01.263 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-21T13:50:01.263 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_xfs_extsize": "false", 2024-09-21T13:50:01.263 INFO:teuthology.orchestra.run.smithi049.stdout: "filestore_zfs_snap": "false", 2024-09-21T13:50:01.263 INFO:teuthology.orchestra.run.smithi049.stdout: "fio_dir": "/tmp/fio", 2024-09-21T13:50:01.263 INFO:teuthology.orchestra.run.smithi049.stdout: "fsid": "40a3d2e8-781f-11ef-baf6-efdc52797490", 2024-09-21T13:50:01.263 INFO:teuthology.orchestra.run.smithi049.stdout: "fuse_allow_other": "true", 2024-09-21T13:50:01.263 INFO:teuthology.orchestra.run.smithi049.stdout: "fuse_atomic_o_trunc": "true", 2024-09-21T13:50:01.263 INFO:teuthology.orchestra.run.smithi049.stdout: "fuse_big_writes": "true", 2024-09-21T13:50:01.263 INFO:teuthology.orchestra.run.smithi049.stdout: "fuse_debug": "false", 2024-09-21T13:50:01.263 INFO:teuthology.orchestra.run.smithi049.stdout: "fuse_default_permissions": "false", 2024-09-21T13:50:01.263 INFO:teuthology.orchestra.run.smithi049.stdout: "fuse_disable_pagecache": "false", 2024-09-21T13:50:01.263 INFO:teuthology.orchestra.run.smithi049.stdout: "fuse_max_write": "0", 2024-09-21T13:50:01.263 INFO:teuthology.orchestra.run.smithi049.stdout: "fuse_multithreaded": "true", 2024-09-21T13:50:01.263 INFO:teuthology.orchestra.run.smithi049.stdout: "fuse_require_active_mds": "true", 2024-09-21T13:50:01.264 INFO:teuthology.orchestra.run.smithi049.stdout: "fuse_set_user_groups": "true", 2024-09-21T13:50:01.264 INFO:teuthology.orchestra.run.smithi049.stdout: "fuse_splice_move": "true", 2024-09-21T13:50:01.264 INFO:teuthology.orchestra.run.smithi049.stdout: "fuse_splice_read": "true", 2024-09-21T13:50:01.264 INFO:teuthology.orchestra.run.smithi049.stdout: "fuse_splice_write": "true", 2024-09-21T13:50:01.264 INFO:teuthology.orchestra.run.smithi049.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-21T13:50:01.264 INFO:teuthology.orchestra.run.smithi049.stdout: "fuse_use_invalidate_cb": "true", 2024-09-21T13:50:01.264 INFO:teuthology.orchestra.run.smithi049.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-21T13:50:01.264 INFO:teuthology.orchestra.run.smithi049.stdout: "gss_target_name": "ceph", 2024-09-21T13:50:01.264 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_file": "", 2024-09-21T13:50:01.264 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_inject_failure": "0", 2024-09-21T13:50:01.264 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_interval": "5", 2024-09-21T13:50:01.264 INFO:teuthology.orchestra.run.smithi049.stdout: "host": "smithi049", 2024-09-21T13:50:01.264 INFO:teuthology.orchestra.run.smithi049.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-21T13:50:01.265 INFO:teuthology.orchestra.run.smithi049.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-21T13:50:01.265 INFO:teuthology.orchestra.run.smithi049.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-21T13:50:01.265 INFO:teuthology.orchestra.run.smithi049.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-21T13:50:01.265 INFO:teuthology.orchestra.run.smithi049.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-21T13:50:01.265 INFO:teuthology.orchestra.run.smithi049.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-21T13:50:01.265 INFO:teuthology.orchestra.run.smithi049.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-21T13:50:01.265 INFO:teuthology.orchestra.run.smithi049.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-21T13:50:01.265 INFO:teuthology.orchestra.run.smithi049.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-21T13:50:01.265 INFO:teuthology.orchestra.run.smithi049.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-21T13:50:01.265 INFO:teuthology.orchestra.run.smithi049.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-21T13:50:01.265 INFO:teuthology.orchestra.run.smithi049.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-21T13:50:01.265 INFO:teuthology.orchestra.run.smithi049.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-21T13:50:01.265 INFO:teuthology.orchestra.run.smithi049.stdout: "inject_early_sigterm": "false", 2024-09-21T13:50:01.265 INFO:teuthology.orchestra.run.smithi049.stdout: "jaeger_agent_port": "6799", 2024-09-21T13:50:01.266 INFO:teuthology.orchestra.run.smithi049.stdout: "jaeger_tracing_enable": "false", 2024-09-21T13:50:01.266 INFO:teuthology.orchestra.run.smithi049.stdout: "journal_aio": "true", 2024-09-21T13:50:01.266 INFO:teuthology.orchestra.run.smithi049.stdout: "journal_align_min_size": "65536", 2024-09-21T13:50:01.266 INFO:teuthology.orchestra.run.smithi049.stdout: "journal_block_align": "true", 2024-09-21T13:50:01.266 INFO:teuthology.orchestra.run.smithi049.stdout: "journal_block_size": "4096", 2024-09-21T13:50:01.266 INFO:teuthology.orchestra.run.smithi049.stdout: "journal_dio": "true", 2024-09-21T13:50:01.266 INFO:teuthology.orchestra.run.smithi049.stdout: "journal_discard": "false", 2024-09-21T13:50:01.266 INFO:teuthology.orchestra.run.smithi049.stdout: "journal_force_aio": "false", 2024-09-21T13:50:01.266 INFO:teuthology.orchestra.run.smithi049.stdout: "journal_ignore_corruption": "false", 2024-09-21T13:50:01.266 INFO:teuthology.orchestra.run.smithi049.stdout: "journal_max_write_bytes": "10485760", 2024-09-21T13:50:01.266 INFO:teuthology.orchestra.run.smithi049.stdout: "journal_max_write_entries": "100", 2024-09-21T13:50:01.266 INFO:teuthology.orchestra.run.smithi049.stdout: "journal_replay_from": "0", 2024-09-21T13:50:01.266 INFO:teuthology.orchestra.run.smithi049.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-21T13:50:01.267 INFO:teuthology.orchestra.run.smithi049.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-21T13:50:01.267 INFO:teuthology.orchestra.run.smithi049.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-21T13:50:01.267 INFO:teuthology.orchestra.run.smithi049.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-21T13:50:01.267 INFO:teuthology.orchestra.run.smithi049.stdout: "journal_write_header_frequency": "0", 2024-09-21T13:50:01.267 INFO:teuthology.orchestra.run.smithi049.stdout: "journal_zero_on_create": "false", 2024-09-21T13:50:01.267 INFO:teuthology.orchestra.run.smithi049.stdout: "journaler_prefetch_periods": "10", 2024-09-21T13:50:01.267 INFO:teuthology.orchestra.run.smithi049.stdout: "journaler_prezero_periods": "5", 2024-09-21T13:50:01.267 INFO:teuthology.orchestra.run.smithi049.stdout: "journaler_write_head_interval": "15", 2024-09-21T13:50:01.267 INFO:teuthology.orchestra.run.smithi049.stdout: "key": "", 2024-09-21T13:50:01.267 INFO:teuthology.orchestra.run.smithi049.stdout: "keyfile": "", 2024-09-21T13:50:01.267 INFO:teuthology.orchestra.run.smithi049.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-21T13:50:01.267 INFO:teuthology.orchestra.run.smithi049.stdout: "kstore_backend": "rocksdb", 2024-09-21T13:50:01.267 INFO:teuthology.orchestra.run.smithi049.stdout: "kstore_default_stripe_size": "65536", 2024-09-21T13:50:01.267 INFO:teuthology.orchestra.run.smithi049.stdout: "kstore_fsck_on_mount": "false", 2024-09-21T13:50:01.268 INFO:teuthology.orchestra.run.smithi049.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-21T13:50:01.268 INFO:teuthology.orchestra.run.smithi049.stdout: "kstore_max_bytes": "67108864", 2024-09-21T13:50:01.268 INFO:teuthology.orchestra.run.smithi049.stdout: "kstore_max_ops": "512", 2024-09-21T13:50:01.268 INFO:teuthology.orchestra.run.smithi049.stdout: "kstore_nid_prealloc": "1024", 2024-09-21T13:50:01.268 INFO:teuthology.orchestra.run.smithi049.stdout: "kstore_onode_map_size": "1024", 2024-09-21T13:50:01.268 INFO:teuthology.orchestra.run.smithi049.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-21T13:50:01.268 INFO:teuthology.orchestra.run.smithi049.stdout: "kstore_sync_submit_transaction": "false", 2024-09-21T13:50:01.268 INFO:teuthology.orchestra.run.smithi049.stdout: "kstore_sync_transaction": "false", 2024-09-21T13:50:01.268 INFO:teuthology.orchestra.run.smithi049.stdout: "librados_thread_count": "2", 2024-09-21T13:50:01.268 INFO:teuthology.orchestra.run.smithi049.stdout: "lockdep": "false", 2024-09-21T13:50:01.268 INFO:teuthology.orchestra.run.smithi049.stdout: "lockdep_force_backtrace": "false", 2024-09-21T13:50:01.268 INFO:teuthology.orchestra.run.smithi049.stdout: "log_coarse_timestamps": "true", 2024-09-21T13:50:01.268 INFO:teuthology.orchestra.run.smithi049.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-21T13:50:01.268 INFO:teuthology.orchestra.run.smithi049.stdout: "log_flush_on_exit": "false", 2024-09-21T13:50:01.269 INFO:teuthology.orchestra.run.smithi049.stdout: "log_graylog_host": "127.0.0.1", 2024-09-21T13:50:01.269 INFO:teuthology.orchestra.run.smithi049.stdout: "log_graylog_port": "12201", 2024-09-21T13:50:01.269 INFO:teuthology.orchestra.run.smithi049.stdout: "log_max_new": "1000", 2024-09-21T13:50:01.269 INFO:teuthology.orchestra.run.smithi049.stdout: "log_max_recent": "10000", 2024-09-21T13:50:01.269 INFO:teuthology.orchestra.run.smithi049.stdout: "log_stderr_prefix": "", 2024-09-21T13:50:01.269 INFO:teuthology.orchestra.run.smithi049.stdout: "log_stop_at_utilization": "0.970000", 2024-09-21T13:50:01.269 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": "true", 2024-09-21T13:50:01.269 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_graylog": "false", 2024-09-21T13:50:01.269 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_journald": "false", 2024-09-21T13:50:01.269 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_stderr": "false", 2024-09-21T13:50:01.269 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_syslog": "false", 2024-09-21T13:50:01.269 INFO:teuthology.orchestra.run.smithi049.stdout: "max_rotating_auth_attempts": "10", 2024-09-21T13:50:01.269 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-21T13:50:01.269 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_action_on_write_error": "1", 2024-09-21T13:50:01.270 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_alternate_name_max": "8192", 2024-09-21T13:50:01.270 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_asio_thread_count": "2", 2024-09-21T13:50:01.270 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_export_pin": "true", 2024-09-21T13:50:01.270 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_fragment_dirs": "true", 2024-09-21T13:50:01.270 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-21T13:50:01.270 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_fragment_interval": "5", 2024-09-21T13:50:01.270 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-21T13:50:01.270 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-21T13:50:01.270 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_interval": "10", 2024-09-21T13:50:01.270 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_max": "-1", 2024-09-21T13:50:01.270 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_max_until": "-1", 2024-09-21T13:50:01.270 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_merge_size": "50", 2024-09-21T13:50:01.270 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_midchunk": "0.300000", 2024-09-21T13:50:01.270 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-21T13:50:01.271 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_min_start": "0.200000", 2024-09-21T13:50:01.271 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_minchunk": "0.001000", 2024-09-21T13:50:01.271 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_mode": "0", 2024-09-21T13:50:01.271 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_need_max": "1.200000", 2024-09-21T13:50:01.271 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_need_min": "0.800000", 2024-09-21T13:50:01.271 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_overload_epochs": "2", 2024-09-21T13:50:01.271 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-21T13:50:01.271 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-21T13:50:01.271 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_split_bits": "3", 2024-09-21T13:50:01.271 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-21T13:50:01.271 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_split_size": "10000", 2024-09-21T13:50:01.271 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-21T13:50:01.272 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_target_decay": "10.000000", 2024-09-21T13:50:01.272 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-21T13:50:01.272 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_beacon_grace": "15.000000", 2024-09-21T13:50:01.272 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_beacon_interval": "4.000000", 2024-09-21T13:50:01.272 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-21T13:50:01.272 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-21T13:50:01.272 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_cache_mid": "0.700000", 2024-09-21T13:50:01.272 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-09-21T13:50:01.272 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_cache_quiesce_delay": "0", 2024-09-21T13:50:01.272 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_cache_quiesce_sleep": "200", 2024-09-21T13:50:01.272 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_cache_quiesce_splitauth": "true", 2024-09-21T13:50:01.272 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_cache_quiesce_threshold": "524288", 2024-09-21T13:50:01.272 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_cache_release_free_interval": "10", 2024-09-21T13:50:01.272 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_cache_reservation": "0.050000", 2024-09-21T13:50:01.273 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-21T13:50:01.273 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_cache_trim_interval": "1", 2024-09-21T13:50:01.273 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_cache_trim_threshold": "262144", 2024-09-21T13:50:01.273 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-21T13:50:01.273 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-21T13:50:01.273 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-21T13:50:01.273 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_client_prealloc_inos": "1000", 2024-09-21T13:50:01.273 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-21T13:50:01.273 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_connect_bootstrapping": "false", 2024-09-21T13:50:01.273 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_damage_table_max_entries": "10000", 2024-09-21T13:50:01.273 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-21T13:50:01.273 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_debug_auth_pins": "false", 2024-09-21T13:50:01.273 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_debug_frag": "false", 2024-09-21T13:50:01.273 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_debug_scatterstat": "false", 2024-09-21T13:50:01.274 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_debug_subtrees": "false", 2024-09-21T13:50:01.274 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_decay_halflife": "5.000000", 2024-09-21T13:50:01.274 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_default_dir_hash": "2", 2024-09-21T13:50:01.274 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_defer_session_stale": "true", 2024-09-21T13:50:01.274 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_deny_all_reconnect": "false", 2024-09-21T13:50:01.274 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_dir_keys_per_op": "16384", 2024-09-21T13:50:01.274 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_dir_max_commit_size": "10", 2024-09-21T13:50:01.274 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_dir_max_entries": "0", 2024-09-21T13:50:01.274 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_dir_prefetch": "true", 2024-09-21T13:50:01.274 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-21T13:50:01.274 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-21T13:50:01.274 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_dump_cache_on_map": "false", 2024-09-21T13:50:01.275 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-21T13:50:01.275 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-21T13:50:01.275 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_early_reply": "true", 2024-09-21T13:50:01.275 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_enable_op_tracker": "true", 2024-09-21T13:50:01.275 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_enforce_unique_name": "true", 2024-09-21T13:50:01.275 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-21T13:50:01.275 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-21T13:50:01.275 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_export_ephemeral_random": "true", 2024-09-21T13:50:01.275 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-21T13:50:01.275 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-09-21T13:50:01.275 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-21T13:50:01.275 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-21T13:50:01.275 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-09-21T13:50:01.275 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-21T13:50:01.276 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-21T13:50:01.276 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-21T13:50:01.276 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_health_summarize_threshold": "10", 2024-09-21T13:50:01.276 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-21T13:50:01.276 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-21T13:50:01.276 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_inject_health_dummy": "false", 2024-09-21T13:50:01.276 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-21T13:50:01.276 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_inject_migrator_session_race": "false", 2024-09-21T13:50:01.276 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-21T13:50:01.276 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-21T13:50:01.276 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-21T13:50:01.276 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_join_fs": "", 2024-09-21T13:50:01.276 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_journal_format": "1", 2024-09-21T13:50:01.276 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-09-21T13:50:01.277 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_kill_create_at": "0", 2024-09-21T13:50:01.277 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_kill_dirfrag_at": "0", 2024-09-21T13:50:01.277 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_kill_export_at": "0", 2024-09-21T13:50:01.277 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_kill_import_at": "0", 2024-09-21T13:50:01.277 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_kill_journal_at": "0", 2024-09-21T13:50:01.277 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_kill_journal_expire_at": "0", 2024-09-21T13:50:01.277 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_kill_journal_replay_at": "0", 2024-09-21T13:50:01.277 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_kill_link_at": "0", 2024-09-21T13:50:01.277 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_kill_mdstable_at": "0", 2024-09-21T13:50:01.277 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_kill_openc_at": "0", 2024-09-21T13:50:01.277 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_kill_rename_at": "0", 2024-09-21T13:50:01.277 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_kill_shutdown_at": "0", 2024-09-21T13:50:01.277 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_log_event_large_threshold": "524288", 2024-09-21T13:50:01.278 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_log_events_per_segment": "1024", 2024-09-21T13:50:01.278 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_log_major_segment_event_ratio": "12", 2024-09-21T13:50:01.278 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_log_max_events": "-1", 2024-09-21T13:50:01.278 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_log_max_segments": "128", 2024-09-21T13:50:01.278 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_log_pause": "false", 2024-09-21T13:50:01.278 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_log_segment_size": "0", 2024-09-21T13:50:01.278 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-21T13:50:01.278 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_log_skip_unbounded_events": "false", 2024-09-21T13:50:01.278 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-09-21T13:50:01.278 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_log_trim_threshold": "128", 2024-09-21T13:50:01.278 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-09-21T13:50:01.278 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_log_warn_factor": "2.000000", 2024-09-21T13:50:01.278 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_max_caps_per_client": "1048576", 2024-09-21T13:50:01.278 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_max_completed_flushes": "100000", 2024-09-21T13:50:01.279 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_max_completed_requests": "100000", 2024-09-21T13:50:01.279 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_max_export_size": "20971520", 2024-09-21T13:50:01.279 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_max_file_recover": "32", 2024-09-21T13:50:01.279 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_max_purge_files": "64", 2024-09-21T13:50:01.279 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_max_purge_ops": "8192", 2024-09-21T13:50:01.279 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-21T13:50:01.279 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-21T13:50:01.279 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_max_snaps_per_dir": "100", 2024-09-21T13:50:01.279 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_metrics_update_interval": "2", 2024-09-21T13:50:01.279 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_min_caps_per_client": "100", 2024-09-21T13:50:01.279 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_min_caps_working_set": "10000", 2024-09-21T13:50:01.279 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-21T13:50:01.279 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_numa_node": "-1", 2024-09-21T13:50:01.279 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-21T13:50:01.280 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_op_complaint_time": "30.000000", 2024-09-21T13:50:01.280 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_op_history_duration": "600", 2024-09-21T13:50:01.280 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_op_history_size": "20", 2024-09-21T13:50:01.280 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_op_history_slow_op_size": "20", 2024-09-21T13:50:01.280 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-21T13:50:01.280 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_op_log_threshold": "5", 2024-09-21T13:50:01.280 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_ping_grace": "15", 2024-09-21T13:50:01.280 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_ping_interval": "5", 2024-09-21T13:50:01.280 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-21T13:50:01.280 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-21T13:50:01.280 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_recall_max_caps": "30000", 2024-09-21T13:50:01.280 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-21T13:50:01.280 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-21T13:50:01.280 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-21T13:50:01.281 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_recall_warning_threshold": "262144", 2024-09-21T13:50:01.281 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-21T13:50:01.281 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_replay_interval": "1.000000", 2024-09-21T13:50:01.281 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-21T13:50:01.281 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-21T13:50:01.281 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_root_ino_gid": "0", 2024-09-21T13:50:01.281 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_root_ino_uid": "0", 2024-09-21T13:50:01.281 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-21T13:50:01.281 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_scrub_stats_review_period": "1", 2024-09-21T13:50:01.281 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_server_dispatch_client_request_delay": "0", 2024-09-21T13:50:01.281 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_server_dispatch_killpoint_random": "0.000000", 2024-09-21T13:50:01.281 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-21T13:50:01.281 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-21T13:50:01.281 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-21T13:50:01.282 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-21T13:50:01.282 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-21T13:50:01.282 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-21T13:50:01.282 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-21T13:50:01.282 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-21T13:50:01.282 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-21T13:50:01.282 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_shutdown_check": "0", 2024-09-21T13:50:01.282 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_skip_ino": "0", 2024-09-21T13:50:01.282 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-21T13:50:01.282 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_snap_max_uid": "4294967294", 2024-09-21T13:50:01.282 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_snap_min_uid": "0", 2024-09-21T13:50:01.282 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_snap_rstat": "false", 2024-09-21T13:50:01.284 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_standby_replay_damaged": "false", 2024-09-21T13:50:01.284 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_symlink_recovery": "true", 2024-09-21T13:50:01.284 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-21T13:50:01.284 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_thrash_exports": "0", 2024-09-21T13:50:01.285 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_thrash_fragments": "0", 2024-09-21T13:50:01.285 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_tick_interval": "5.000000", 2024-09-21T13:50:01.285 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_valgrind_exit": "false", 2024-09-21T13:50:01.285 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_verify_backtrace": "1", 2024-09-21T13:50:01.285 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_verify_scatter": "false", 2024-09-21T13:50:01.285 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-21T13:50:01.285 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_wipe_sessions": "false", 2024-09-21T13:50:01.285 INFO:teuthology.orchestra.run.smithi049.stdout: "mempool_debug": "false", 2024-09-21T13:50:01.285 INFO:teuthology.orchestra.run.smithi049.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-21T13:50:01.285 INFO:teuthology.orchestra.run.smithi049.stdout: "memstore_device_bytes": "1073741824", 2024-09-21T13:50:01.286 INFO:teuthology.orchestra.run.smithi049.stdout: "memstore_page_set": "false", 2024-09-21T13:50:01.286 INFO:teuthology.orchestra.run.smithi049.stdout: "memstore_page_size": "65536", 2024-09-21T13:50:01.286 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_client_bytes": "134217728", 2024-09-21T13:50:01.286 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_client_messages": "512", 2024-09-21T13:50:01.286 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-21T13:50:01.286 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-21T13:50:01.286 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-21T13:50:01.286 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-21T13:50:01.286 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_disabled_modules": "", 2024-09-21T13:50:01.286 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_enable_op_tracker": "true", 2024-09-21T13:50:01.286 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-21T13:50:01.286 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_max_pg_creating": "1024", 2024-09-21T13:50:01.287 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_max_pg_num_change": "128", 2024-09-21T13:50:01.287 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_mds_bytes": "134217728", 2024-09-21T13:50:01.287 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_mds_messages": "128", 2024-09-21T13:50:01.287 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-21T13:50:01.287 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_mon_bytes": "134217728", 2024-09-21T13:50:01.287 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_mon_messages": "128", 2024-09-21T13:50:01.287 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_num_op_tracker_shard": "32", 2024-09-21T13:50:01.287 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_op_complaint_time": "30.000000", 2024-09-21T13:50:01.287 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_op_history_duration": "600", 2024-09-21T13:50:01.287 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_op_history_size": "20", 2024-09-21T13:50:01.287 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_op_history_slow_op_size": "20", 2024-09-21T13:50:01.287 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2024-09-21T13:50:01.287 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_op_log_threshold": "5", 2024-09-21T13:50:01.287 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_osd_bytes": "536870912", 2024-09-21T13:50:01.288 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_osd_messages": "8192", 2024-09-21T13:50:01.288 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_pool": "true", 2024-09-21T13:50:01.288 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-21T13:50:01.288 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_standby_modules": "true", 2024-09-21T13:50:01.288 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_stats_period": "5", 2024-09-21T13:50:01.288 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_stats_threshold": "5", 2024-09-21T13:50:01.288 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_tick_period": "2", 2024-09-21T13:50:01.288 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-21T13:50:01.288 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-21T13:50:01.288 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_allow_pool_delete": "true", 2024-09-21T13:50:01.288 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_allow_pool_size_one": "false", 2024-09-21T13:50:01.288 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_auth_validate_all_caps": "true", 2024-09-21T13:50:01.288 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-21T13:50:01.288 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-21T13:50:01.288 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_client_bytes": "104857600", 2024-09-21T13:50:01.289 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_client_directed_command_retry": "2", 2024-09-21T13:50:01.289 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-21T13:50:01.289 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-21T13:50:01.289 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-21T13:50:01.289 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-21T13:50:01.289 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_client_hunt_parallel": "3", 2024-09-21T13:50:01.289 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_client_log_interval": "1.000000", 2024-09-21T13:50:01.289 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-21T13:50:01.289 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_client_ping_interval": "10.000000", 2024-09-21T13:50:01.289 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-21T13:50:01.289 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_client_target_rank": "-1", 2024-09-21T13:50:01.289 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-21T13:50:01.289 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-21T13:50:01.289 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-21T13:50:01.290 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_cluster_log_level": "debug", 2024-09-21T13:50:01.290 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_cluster_log_to_file": "true", 2024-09-21T13:50:01.290 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-21T13:50:01.290 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-21T13:50:01.290 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-21T13:50:01.290 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_cluster_log_to_journald": "true", 2024-09-21T13:50:01.290 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_cluster_log_to_stderr": "false", 2024-09-21T13:50:01.290 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-21T13:50:01.290 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-21T13:50:01.290 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_compact_on_bootstrap": "false", 2024-09-21T13:50:01.290 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_compact_on_start": "false", 2024-09-21T13:50:01.290 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_compact_on_trim": "true", 2024-09-21T13:50:01.291 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-21T13:50:01.291 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-21T13:50:01.291 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-21T13:50:01.291 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_cpu_threads": "4", 2024-09-21T13:50:01.291 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_crush_min_required_version": "hammer", 2024-09-21T13:50:01.291 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_daemon_bytes": "419430400", 2024-09-21T13:50:01.291 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-21T13:50:01.291 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_data_avail_crit": "5", 2024-09-21T13:50:01.291 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_data_avail_warn": "5", 2024-09-21T13:50:01.291 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_data_size_warn": "16106127360", 2024-09-21T13:50:01.291 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-21T13:50:01.291 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-21T13:50:01.291 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_debug_dump_json": "false", 2024-09-21T13:50:01.292 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-21T13:50:01.292 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_debug_dump_transactions": "false", 2024-09-21T13:50:01.292 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_debug_extra_checks": "false", 2024-09-21T13:50:01.292 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-21T13:50:01.292 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-21T13:50:01.292 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_debug_no_require_reef": "false", 2024-09-21T13:50:01.292 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_debug_no_require_squid": "false", 2024-09-21T13:50:01.292 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-21T13:50:01.292 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-21T13:50:01.292 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-21T13:50:01.293 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_down_mkfs_grace": "60", 2024-09-21T13:50:01.293 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_down_uptime_grace": "60", 2024-09-21T13:50:01.293 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_election_default_strategy": "1", 2024-09-21T13:50:01.293 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_election_timeout": "5.000000", 2024-09-21T13:50:01.293 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-21T13:50:01.293 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_elector_ping_divisor": "2", 2024-09-21T13:50:01.293 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-21T13:50:01.293 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_enable_op_tracker": "true", 2024-09-21T13:50:01.293 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_fake_pool_delete": "false", 2024-09-21T13:50:01.293 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_force_quorum_join": "false", 2024-09-21T13:50:01.293 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_fsmap_prune_threshold": "300", 2024-09-21T13:50:01.294 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_globalid_prealloc": "10000", 2024-09-21T13:50:01.294 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_health_detail_to_clog": "true", 2024-09-21T13:50:01.294 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_health_log_update_period": "5", 2024-09-21T13:50:01.294 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_health_max_detail": "50", 2024-09-21T13:50:01.294 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_health_to_clog": "true", 2024-09-21T13:50:01.294 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_health_to_clog_interval": "600", 2024-09-21T13:50:01.294 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-21T13:50:01.294 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_host": "[v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0]", 2024-09-21T13:50:01.294 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_host_override": "", 2024-09-21T13:50:01.294 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_initial_members": "", 2024-09-21T13:50:01.294 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-21T13:50:01.294 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-21T13:50:01.294 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-21T13:50:01.295 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-21T13:50:01.295 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-21T13:50:01.295 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_lease": "5.000000", 2024-09-21T13:50:01.295 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-21T13:50:01.295 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-21T13:50:01.295 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_log_full_interval": "50", 2024-09-21T13:50:01.295 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_log_max": "10000", 2024-09-21T13:50:01.295 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_log_max_summary": "50", 2024-09-21T13:50:01.295 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-21T13:50:01.295 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_max_log_epochs": "500", 2024-09-21T13:50:01.295 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-21T13:50:01.295 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-21T13:50:01.295 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_max_nvmeof_epochs": "500", 2024-09-21T13:50:01.295 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_max_osd": "10000", 2024-09-21T13:50:01.296 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_max_pg_per_osd": "10000", 2024-09-21T13:50:01.296 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_max_pool_pg_num": "65536", 2024-09-21T13:50:01.296 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-21T13:50:01.296 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-21T13:50:01.296 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_mds_force_trim_to": "0", 2024-09-21T13:50:01.296 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_mds_skip_sanity": "false", 2024-09-21T13:50:01.296 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_memory_autotune": "true", 2024-09-21T13:50:01.296 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_memory_target": "2147483648", 2024-09-21T13:50:01.296 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_mgr_beacon_grace": "30", 2024-09-21T13:50:01.296 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-21T13:50:01.296 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_mgr_digest_period": "5", 2024-09-21T13:50:01.296 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_mgr_inactive_grace": "60", 2024-09-21T13:50:01.296 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-21T13:50:01.296 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-21T13:50:01.297 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_min_osdmap_epochs": "50", 2024-09-21T13:50:01.297 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_nvmeofgw_beacon_grace": "10", 2024-09-21T13:50:01.297 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_nvmeofgw_set_group_id_retry": "1000", 2024-09-21T13:50:01.297 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_op_complaint_time": "30", 2024-09-21T13:50:01.297 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_op_history_duration": "600", 2024-09-21T13:50:01.297 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_op_history_size": "20", 2024-09-21T13:50:01.297 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_op_history_slow_op_size": "20", 2024-09-21T13:50:01.297 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-21T13:50:01.297 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_op_log_threshold": "5", 2024-09-21T13:50:01.297 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-21T13:50:01.297 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-21T13:50:01.297 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-21T13:50:01.297 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_auto_mark_in": "false", 2024-09-21T13:50:01.297 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-21T13:50:01.297 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-21T13:50:01.298 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-21T13:50:01.298 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_cache_size": "500", 2024-09-21T13:50:01.298 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-21T13:50:01.298 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-21T13:50:01.298 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-21T13:50:01.298 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_down_out_interval": "600", 2024-09-21T13:50:01.298 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-21T13:50:01.298 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-21T13:50:01.298 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_force_trim_to": "0", 2024-09-21T13:50:01.298 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-21T13:50:01.298 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-21T13:50:01.298 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-21T13:50:01.298 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-21T13:50:01.299 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-21T13:50:01.299 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-21T13:50:01.299 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-21T13:50:01.299 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_min_down_reporters": "2", 2024-09-21T13:50:01.299 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-21T13:50:01.299 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-21T13:50:01.299 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-21T13:50:01.299 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-21T13:50:01.299 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-21T13:50:01.299 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-21T13:50:01.299 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_report_timeout": "900", 2024-09-21T13:50:01.299 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-21T13:50:01.299 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-21T13:50:01.300 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-21T13:50:01.300 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-21T13:50:01.300 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-21T13:50:01.300 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osdmap_full_prune_interval": "2", 2024-09-21T13:50:01.300 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osdmap_full_prune_min": "15", 2024-09-21T13:50:01.300 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-09-21T13:50:01.300 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-21T13:50:01.300 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_pg_stuck_threshold": "60", 2024-09-21T13:50:01.300 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-21T13:50:01.300 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-21T13:50:01.300 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-21T13:50:01.300 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-21T13:50:01.300 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-21T13:50:01.300 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-21T13:50:01.301 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_probe_timeout": "2.000000", 2024-09-21T13:50:01.301 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_reweight_max_change": "0.050000", 2024-09-21T13:50:01.301 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_reweight_max_osds": "4", 2024-09-21T13:50:01.301 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-21T13:50:01.301 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-21T13:50:01.301 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-21T13:50:01.301 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-21T13:50:01.301 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-21T13:50:01.301 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_scrub_interval": "86400", 2024-09-21T13:50:01.301 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_scrub_max_keys": "100", 2024-09-21T13:50:01.301 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_scrub_timeout": "300", 2024-09-21T13:50:01.301 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_session_timeout": "300", 2024-09-21T13:50:01.301 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_smart_report_timeout": "5", 2024-09-21T13:50:01.301 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_stat_smooth_intervals": "6", 2024-09-21T13:50:01.302 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-21T13:50:01.302 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_stretch_pool_min_size": "2", 2024-09-21T13:50:01.302 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_stretch_pool_size": "4", 2024-09-21T13:50:01.302 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-21T13:50:01.302 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-21T13:50:01.302 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_sync_debug": "false", 2024-09-21T13:50:01.302 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-21T13:50:01.302 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-21T13:50:01.302 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_sync_provider_kill_at": "0", 2024-09-21T13:50:01.302 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_sync_requester_kill_at": "0", 2024-09-21T13:50:01.302 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_sync_timeout": "60.000000", 2024-09-21T13:50:01.302 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_target_pg_per_osd": "100", 2024-09-21T13:50:01.302 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_tick_interval": "5", 2024-09-21T13:50:01.302 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_timecheck_interval": "300.000000", 2024-09-21T13:50:01.303 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-21T13:50:01.303 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_use_min_delay_socket": "false", 2024-09-21T13:50:01.303 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_older_version_delay": "604800", 2024-09-21T13:50:01.303 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-21T13:50:01.303 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-21T13:50:01.303 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-21T13:50:01.303 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_on_filestore_osds": "true", 2024-09-21T13:50:01.303 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-21T13:50:01.303 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-21T13:50:01.303 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-21T13:50:01.303 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_on_misplaced": "false", 2024-09-21T13:50:01.303 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-21T13:50:01.304 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_on_older_version": "true", 2024-09-21T13:50:01.304 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-21T13:50:01.304 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_on_pool_no_app": "true", 2024-09-21T13:50:01.304 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-09-21T13:50:01.304 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-21T13:50:01.304 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-21T13:50:01.304 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-21T13:50:01.304 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-21T13:50:01.304 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-21T13:50:01.304 INFO:teuthology.orchestra.run.smithi049.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-21T13:50:01.304 INFO:teuthology.orchestra.run.smithi049.stdout: "monmap": "", 2024-09-21T13:50:01.304 INFO:teuthology.orchestra.run.smithi049.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-21T13:50:01.304 INFO:teuthology.orchestra.run.smithi049.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-09-21T13:50:01.304 INFO:teuthology.orchestra.run.smithi049.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-09-21T13:50:01.305 INFO:teuthology.orchestra.run.smithi049.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-21T13:50:01.305 INFO:teuthology.orchestra.run.smithi049.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-09-21T13:50:01.305 INFO:teuthology.orchestra.run.smithi049.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-09-21T13:50:01.305 INFO:teuthology.orchestra.run.smithi049.stdout: "motr_tracing_enabled": "false", 2024-09-21T13:50:01.305 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_async_op_threads": "3", 2024-09-21T13:50:01.305 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-21T13:50:01.305 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_async_rdma_cm": "false", 2024-09-21T13:50:01.305 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_async_rdma_device_name": "", 2024-09-21T13:50:01.305 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_async_rdma_dscp": "96", 2024-09-21T13:50:01.305 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-21T13:50:01.305 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-21T13:50:01.305 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_async_rdma_local_gid": "", 2024-09-21T13:50:01.305 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-21T13:50:01.306 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_async_rdma_port_num": "1", 2024-09-21T13:50:01.306 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-21T13:50:01.306 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-21T13:50:01.306 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-21T13:50:01.306 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-21T13:50:01.306 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_async_rdma_sl": "3", 2024-09-21T13:50:01.306 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_async_rdma_support_srq": "true", 2024-09-21T13:50:01.306 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_async_rdma_type": "ib", 2024-09-21T13:50:01.306 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_async_reap_threshold": "5", 2024-09-21T13:50:01.306 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_bind_before_connect": "false", 2024-09-21T13:50:01.306 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_bind_ipv4": "true", 2024-09-21T13:50:01.306 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_bind_ipv6": "false", 2024-09-21T13:50:01.306 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_bind_msgr1": "true", 2024-09-21T13:50:01.306 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_bind_msgr2": "true", 2024-09-21T13:50:01.306 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_bind_port_max": "7568", 2024-09-21T13:50:01.307 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_bind_port_min": "6800", 2024-09-21T13:50:01.307 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-21T13:50:01.307 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_bind_retry_count": "3", 2024-09-21T13:50:01.307 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_bind_retry_delay": "5", 2024-09-21T13:50:01.307 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_blackhole_client": "false", 2024-09-21T13:50:01.307 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_blackhole_mds": "false", 2024-09-21T13:50:01.307 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_blackhole_mgr": "false", 2024-09-21T13:50:01.307 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_blackhole_mon": "false", 2024-09-21T13:50:01.307 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_blackhole_osd": "false", 2024-09-21T13:50:01.307 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_client_mode": "crc secure", 2024-09-21T13:50:01.307 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_client_throttle_retry_time_interval": "5000", 2024-09-21T13:50:01.307 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_cluster_mode": "crc secure", 2024-09-21T13:50:01.307 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_cluster_type": "", 2024-09-21T13:50:01.307 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_compress_secure": "false", 2024-09-21T13:50:01.308 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_connection_idle_timeout": "900", 2024-09-21T13:50:01.308 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_connection_ready_timeout": "10", 2024-09-21T13:50:01.308 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_crc_data": "true", 2024-09-21T13:50:01.308 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_crc_header": "true", 2024-09-21T13:50:01.308 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_die_on_bad_msg": "false", 2024-09-21T13:50:01.308 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_die_on_bug": "true", 2024-09-21T13:50:01.308 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_die_on_old_message": "true", 2024-09-21T13:50:01.308 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_die_on_skipped_message": "false", 2024-09-21T13:50:01.308 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-21T13:50:01.308 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-21T13:50:01.308 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_dpdk_coremask": "0xF", 2024-09-21T13:50:01.308 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-21T13:50:01.308 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-21T13:50:01.309 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_dpdk_enable_tso": "true", 2024-09-21T13:50:01.309 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-21T13:50:01.309 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-21T13:50:01.309 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_dpdk_hugepages": "", 2024-09-21T13:50:01.309 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-21T13:50:01.309 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-21T13:50:01.309 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_dpdk_lro": "true", 2024-09-21T13:50:01.309 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_dpdk_memory_channel": "4", 2024-09-21T13:50:01.309 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-21T13:50:01.309 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_dpdk_pmd": "", 2024-09-21T13:50:01.309 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_dpdk_port_id": "0", 2024-09-21T13:50:01.309 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-21T13:50:01.309 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-21T13:50:01.309 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_dump_on_send": "false", 2024-09-21T13:50:01.310 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_initial_backoff": "0.200000", 2024-09-21T13:50:01.310 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_inject_delay_max": "1.000000", 2024-09-21T13:50:01.310 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_inject_delay_probability": "0.000000", 2024-09-21T13:50:01.310 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_inject_delay_type": "", 2024-09-21T13:50:01.310 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_inject_internal_delays": "0.000000", 2024-09-21T13:50:01.310 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_inject_network_congestion": "0", 2024-09-21T13:50:01.310 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_inject_socket_failures": "0", 2024-09-21T13:50:01.310 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_learn_addr_from_peer": "true", 2024-09-21T13:50:01.310 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_max_accept_failures": "4", 2024-09-21T13:50:01.310 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_max_backoff": "15.000000", 2024-09-21T13:50:01.310 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_mon_client_mode": "secure crc", 2024-09-21T13:50:01.310 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_mon_cluster_mode": "secure crc", 2024-09-21T13:50:01.311 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_mon_service_mode": "secure crc", 2024-09-21T13:50:01.311 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_osd_compress_min_size": "1024", 2024-09-21T13:50:01.311 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_osd_compress_mode": "none", 2024-09-21T13:50:01.311 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_osd_compression_algorithm": "snappy", 2024-09-21T13:50:01.311 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-21T13:50:01.311 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_pq_min_cost": "65536", 2024-09-21T13:50:01.311 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_public_type": "", 2024-09-21T13:50:01.311 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_service_mode": "crc secure", 2024-09-21T13:50:01.311 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_tcp_listen_backlog": "512", 2024-09-21T13:50:01.311 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_tcp_nodelay": "true", 2024-09-21T13:50:01.311 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-21T13:50:01.311 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_tcp_rcvbuf": "0", 2024-09-21T13:50:01.311 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_time_events_min_wait_interval": "1000", 2024-09-21T13:50:01.311 INFO:teuthology.orchestra.run.smithi049.stdout: "ms_type": "async+posix", 2024-09-21T13:50:01.312 INFO:teuthology.orchestra.run.smithi049.stdout: "no_config_file": "false", 2024-09-21T13:50:01.312 INFO:teuthology.orchestra.run.smithi049.stdout: "nvmeof_mon_client_disconnect_panic": "100", 2024-09-21T13:50:01.312 INFO:teuthology.orchestra.run.smithi049.stdout: "nvmeof_mon_client_tick_period": "2", 2024-09-21T13:50:01.312 INFO:teuthology.orchestra.run.smithi049.stdout: "objecter_completion_locks_per_session": "32", 2024-09-21T13:50:01.312 INFO:teuthology.orchestra.run.smithi049.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-21T13:50:01.312 INFO:teuthology.orchestra.run.smithi049.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-21T13:50:01.312 INFO:teuthology.orchestra.run.smithi049.stdout: "objecter_inflight_ops": "1024", 2024-09-21T13:50:01.312 INFO:teuthology.orchestra.run.smithi049.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-21T13:50:01.312 INFO:teuthology.orchestra.run.smithi049.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-21T13:50:01.312 INFO:teuthology.orchestra.run.smithi049.stdout: "objecter_tick_interval": "5.000000", 2024-09-21T13:50:01.312 INFO:teuthology.orchestra.run.smithi049.stdout: "objecter_timeout": "10.000000", 2024-09-21T13:50:01.312 INFO:teuthology.orchestra.run.smithi049.stdout: "objectstore_blackhole": "false", 2024-09-21T13:50:01.312 INFO:teuthology.orchestra.run.smithi049.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-09-21T13:50:01.313 INFO:teuthology.orchestra.run.smithi049.stdout: "openssl_engine_opts": "", 2024-09-21T13:50:01.313 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_agent_delay_time": "5.000000", 2024-09-21T13:50:01.313 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_agent_hist_halflife": "1000", 2024-09-21T13:50:01.313 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_agent_max_low_ops": "2", 2024-09-21T13:50:01.313 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_agent_max_ops": "4", 2024-09-21T13:50:01.313 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-21T13:50:01.313 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-21T13:50:01.313 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_agent_slop": "0.020000", 2024-09-21T13:50:01.313 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-21T13:50:01.313 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-21T13:50:01.313 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_asio_thread_count": "2", 2024-09-21T13:50:01.313 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_async_recovery_min_cost": "100", 2024-09-21T13:50:01.313 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-21T13:50:01.313 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-21T13:50:01.314 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_backfill_scan_max": "512", 2024-09-21T13:50:01.314 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_backfill_scan_min": "64", 2024-09-21T13:50:01.314 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_backoff_on_degraded": "false", 2024-09-21T13:50:01.314 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_backoff_on_peering": "false", 2024-09-21T13:50:01.314 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_backoff_on_unfound": "true", 2024-09-21T13:50:01.314 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_beacon_report_interval": "300", 2024-09-21T13:50:01.314 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_bench_duration": "30", 2024-09-21T13:50:01.314 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-21T13:50:01.314 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_bench_max_block_size": "67108864", 2024-09-21T13:50:01.314 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-21T13:50:01.314 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_blkin_trace_all": "false", 2024-09-21T13:50:01.314 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_blocked_scrub_grace_period": "120", 2024-09-21T13:50:01.314 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-21T13:50:01.314 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-09-21T13:50:01.315 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-21T13:50:01.315 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_check_for_log_corruption": "false", 2024-09-21T13:50:01.315 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-21T13:50:01.315 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-21T13:50:01.315 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-09-21T13:50:01.315 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-21T13:50:01.315 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_class_update_on_start": "true", 2024-09-21T13:50:01.315 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_client_message_cap": "256", 2024-09-21T13:50:01.315 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_client_message_size_cap": "524288000", 2024-09-21T13:50:01.315 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_client_op_priority": "63", 2024-09-21T13:50:01.315 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_client_watch_timeout": "30", 2024-09-21T13:50:01.315 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_command_max_records": "256", 2024-09-21T13:50:01.315 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_compact_on_start": "false", 2024-09-21T13:50:01.316 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-21T13:50:01.316 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_crush_chooseleaf_type": "1", 2024-09-21T13:50:01.316 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-21T13:50:01.316 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_crush_update_on_start": "true", 2024-09-21T13:50:01.316 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_crush_update_weight_set": "true", 2024-09-21T13:50:01.316 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-21T13:50:01.316 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-21T13:50:01.316 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-21T13:50:01.316 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-21T13:50:01.316 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-21T13:50:01.316 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_feed_pullee": "-1", 2024-09-21T13:50:01.316 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-21T13:50:01.316 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-21T13:50:01.316 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-09-21T13:50:01.316 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_misdirected_ops": "false", 2024-09-21T13:50:01.317 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_no_acting_change": "false", 2024-09-21T13:50:01.317 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_no_purge_strays": "false", 2024-09-21T13:50:01.317 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_op_order": "false", 2024-09-21T13:50:01.317 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-21T13:50:01.317 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-21T13:50:01.317 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-21T13:50:01.317 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-21T13:50:01.317 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_shutdown": "false", 2024-09-21T13:50:01.317 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-21T13:50:01.317 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-21T13:50:01.317 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_trim_objects": "false", 2024-09-21T13:50:01.317 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-21T13:50:01.317 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-21T13:50:01.317 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_verify_snaps": "false", 2024-09-21T13:50:01.318 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-21T13:50:01.318 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-21T13:50:01.318 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_deep_scrub_keys": "1024", 2024-09-21T13:50:01.318 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-21T13:50:01.318 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-21T13:50:01.318 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-21T13:50:01.318 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_deep_scrub_stride": "524288", 2024-09-21T13:50:01.318 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-21T13:50:01.318 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_default_data_pool_replay_window": "45", 2024-09-21T13:50:01.318 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_default_notify_timeout": "30", 2024-09-21T13:50:01.318 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_delete_sleep": "0.000000", 2024-09-21T13:50:01.318 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-21T13:50:01.318 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-21T13:50:01.318 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-21T13:50:01.319 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_discard_disconnected_ops": "true", 2024-09-21T13:50:01.319 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_ec_partial_reads": "true", 2024-09-21T13:50:01.319 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_enable_op_tracker": "true", 2024-09-21T13:50:01.319 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-21T13:50:01.319 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_extblkdev_plugins": "vdo", 2024-09-21T13:50:01.319 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-21T13:50:01.319 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-21T13:50:01.319 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_fast_info": "true", 2024-09-21T13:50:01.319 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_fast_shutdown": "true", 2024-09-21T13:50:01.319 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-21T13:50:01.319 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-21T13:50:01.319 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-21T13:50:01.319 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-21T13:50:01.319 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-21T13:50:01.320 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_function_tracing": "false", 2024-09-21T13:50:01.320 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_heartbeat_grace": "20", 2024-09-21T13:50:01.320 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_heartbeat_interval": "6", 2024-09-21T13:50:01.320 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-21T13:50:01.320 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_heartbeat_min_peers": "10", 2024-09-21T13:50:01.320 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_heartbeat_min_size": "2000", 2024-09-21T13:50:01.320 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_heartbeat_stale": "600", 2024-09-21T13:50:01.320 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-21T13:50:01.320 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_hit_set_max_size": "100000", 2024-09-21T13:50:01.320 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_hit_set_min_size": "1000", 2024-09-21T13:50:01.320 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-21T13:50:01.320 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-21T13:50:01.320 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-21T13:50:01.320 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-21T13:50:01.321 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-21T13:50:01.321 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-21T13:50:01.321 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_journal_size": "5120", 2024-09-21T13:50:01.321 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_kill_backfill_at": "0", 2024-09-21T13:50:01.321 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-21T13:50:01.321 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_map_cache_size": "50", 2024-09-21T13:50:01.321 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_map_dedup": "true", 2024-09-21T13:50:01.321 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_map_message_max": "40", 2024-09-21T13:50:01.321 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-21T13:50:01.321 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_map_share_max_epochs": "40", 2024-09-21T13:50:01.321 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_attr_name_len": "100", 2024-09-21T13:50:01.321 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_attr_size": "0", 2024-09-21T13:50:01.321 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_backfills": "1", 2024-09-21T13:50:01.321 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_markdown_count": "5", 2024-09-21T13:50:01.322 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_markdown_period": "600", 2024-09-21T13:50:01.322 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_object_name_len": "2048", 2024-09-21T13:50:01.322 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_object_namespace_len": "256", 2024-09-21T13:50:01.322 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_object_size": "134217728", 2024-09-21T13:50:01.322 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-21T13:50:01.322 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-21T13:50:01.322 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_pg_blocked_by": "16", 2024-09-21T13:50:01.322 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_pg_log_entries": "10000", 2024-09-21T13:50:01.322 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-21T13:50:01.322 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_pgls": "1024", 2024-09-21T13:50:01.322 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_push_cost": "8388608", 2024-09-21T13:50:01.322 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_push_objects": "10", 2024-09-21T13:50:01.322 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_scrubs": "3", 2024-09-21T13:50:01.322 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-21T13:50:01.323 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_trimming_pgs": "2", 2024-09-21T13:50:01.323 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_write_op_reply_len": "64", 2024-09-21T13:50:01.323 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_max_write_size": "90", 2024-09-21T13:50:01.323 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-21T13:50:01.323 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-21T13:50:01.323 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-21T13:50:01.323 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-21T13:50:01.323 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-21T13:50:01.323 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-21T13:50:01.323 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-21T13:50:01.323 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-21T13:50:01.323 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_profile": "balanced", 2024-09-21T13:50:01.323 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-21T13:50:01.323 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-21T13:50:01.324 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-21T13:50:01.324 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-21T13:50:01.324 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-21T13:50:01.324 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-21T13:50:01.324 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-21T13:50:01.324 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-21T13:50:01.324 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-21T13:50:01.324 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-21T13:50:01.324 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-21T13:50:01.324 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_memory_base": "805306368", 2024-09-21T13:50:01.324 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_memory_cache_min": "134217728", 2024-09-21T13:50:01.324 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-21T13:50:01.324 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-21T13:50:01.324 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_memory_target": "4294967296", 2024-09-21T13:50:01.325 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_memory_target_autotune": "false", 2024-09-21T13:50:01.325 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-21T13:50:01.325 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_min_pg_log_entries": "250", 2024-09-21T13:50:01.325 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_min_recovery_priority": "0", 2024-09-21T13:50:01.325 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-21T13:50:01.325 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-21T13:50:01.325 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mon_report_interval": "5", 2024-09-21T13:50:01.325 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-21T13:50:01.325 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-21T13:50:01.325 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_num_cache_shards": "32", 2024-09-21T13:50:01.325 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_num_op_tracker_shard": "32", 2024-09-21T13:50:01.325 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_numa_auto_affinity": "true", 2024-09-21T13:50:01.325 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_numa_node": "-1", 2024-09-21T13:50:01.325 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_numa_prefer_iface": "true", 2024-09-21T13:50:01.326 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-09-21T13:50:01.326 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_objecter_finishers": "1", 2024-09-21T13:50:01.326 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_objectstore": "bluestore", 2024-09-21T13:50:01.326 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_objectstore_fuse": "false", 2024-09-21T13:50:01.326 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_objectstore_ideal_list_max": "64", 2024-09-21T13:50:01.326 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_objectstore_tracing": "false", 2024-09-21T13:50:01.326 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_op_complaint_time": "30.000000", 2024-09-21T13:50:01.326 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_op_history_duration": "600", 2024-09-21T13:50:01.326 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_op_history_size": "20", 2024-09-21T13:50:01.326 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_op_history_slow_op_size": "20", 2024-09-21T13:50:01.326 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-21T13:50:01.326 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_op_log_threshold": "5", 2024-09-21T13:50:01.326 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_op_num_shards": "0", 2024-09-21T13:50:01.326 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_op_num_shards_hdd": "5", 2024-09-21T13:50:01.327 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_op_num_shards_ssd": "8", 2024-09-21T13:50:01.327 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-21T13:50:01.327 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-21T13:50:01.327 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-21T13:50:01.327 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-21T13:50:01.327 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_op_pq_min_cost": "65536", 2024-09-21T13:50:01.327 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-21T13:50:01.327 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_op_queue_cut_off": "high", 2024-09-21T13:50:01.327 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-21T13:50:01.327 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_op_thread_timeout": "15", 2024-09-21T13:50:01.327 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_open_classes_on_start": "true", 2024-09-21T13:50:01.327 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_os_flags": "0", 2024-09-21T13:50:01.327 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_peering_op_priority": "255", 2024-09-21T13:50:01.327 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pg_delete_cost": "1048576", 2024-09-21T13:50:01.328 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pg_delete_priority": "5", 2024-09-21T13:50:01.328 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-21T13:50:01.328 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-21T13:50:01.328 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-21T13:50:01.328 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pg_log_trim_max": "10000", 2024-09-21T13:50:01.328 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pg_log_trim_min": "100", 2024-09-21T13:50:01.328 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-21T13:50:01.328 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-21T13:50:01.328 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-09-21T13:50:01.328 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-09-21T13:50:01.328 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-21T13:50:01.328 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-21T13:50:01.328 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-21T13:50:01.329 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-21T13:50:01.329 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-21T13:50:01.329 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-21T13:50:01.329 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_crimson": "false", 2024-09-21T13:50:01.329 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-21T13:50:01.329 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-21T13:50:01.329 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-21T13:50:01.329 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-21T13:50:01.329 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-21T13:50:01.329 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-21T13:50:01.329 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-21T13:50:01.329 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-21T13:50:01.329 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_flags": "0", 2024-09-21T13:50:01.329 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-21T13:50:01.329 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_min_size": "0", 2024-09-21T13:50:01.330 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-21T13:50:01.330 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_pg_num": "32", 2024-09-21T13:50:01.330 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_pgp_num": "0", 2024-09-21T13:50:01.330 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-21T13:50:01.330 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_read_ratio": "70", 2024-09-21T13:50:01.330 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_size": "3", 2024-09-21T13:50:01.330 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_default_type": "replicated", 2024-09-21T13:50:01.330 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-21T13:50:01.330 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-21T13:50:01.330 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_push_per_object_cost": "1000", 2024-09-21T13:50:01.330 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-21T13:50:01.330 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_recover_clone_overlap": "true", 2024-09-21T13:50:01.330 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-21T13:50:01.330 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_recovery_cost": "20971520", 2024-09-21T13:50:01.331 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-21T13:50:01.331 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_recovery_max_active": "0", 2024-09-21T13:50:01.331 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-21T13:50:01.331 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-21T13:50:01.331 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-21T13:50:01.331 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-21T13:50:01.331 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_recovery_max_single_start": "1", 2024-09-21T13:50:01.331 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_recovery_op_priority": "3", 2024-09-21T13:50:01.331 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-21T13:50:01.331 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_recovery_priority": "5", 2024-09-21T13:50:01.331 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-21T13:50:01.331 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_recovery_sleep": "0.000000", 2024-09-21T13:50:01.331 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-21T13:50:01.331 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-21T13:50:01.332 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-21T13:50:01.332 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_repair_during_recovery": "false", 2024-09-21T13:50:01.332 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_requested_scrub_priority": "5", 2024-09-21T13:50:01.332 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-21T13:50:01.332 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-21T13:50:01.332 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_auto_repair": "false", 2024-09-21T13:50:01.332 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-21T13:50:01.332 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-21T13:50:01.332 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_begin_hour": "0", 2024-09-21T13:50:01.332 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_begin_week_day": "0", 2024-09-21T13:50:01.332 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_chunk_max": "15", 2024-09-21T13:50:01.332 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_chunk_min": "5", 2024-09-21T13:50:01.332 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_cost": "52428800", 2024-09-21T13:50:01.332 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-09-21T13:50:01.333 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_during_recovery": "false", 2024-09-21T13:50:01.333 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_end_hour": "0", 2024-09-21T13:50:01.333 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_end_week_day": "0", 2024-09-21T13:50:01.333 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_event_cost": "4096", 2024-09-21T13:50:01.333 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-21T13:50:01.333 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-21T13:50:01.333 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_invalid_stats": "true", 2024-09-21T13:50:01.333 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-21T13:50:01.333 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-21T13:50:01.333 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_max_preemptions": "5", 2024-09-21T13:50:01.333 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-21T13:50:01.333 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_priority": "5", 2024-09-21T13:50:01.333 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_retry_after_noscrub": "60", 2024-09-21T13:50:01.333 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_retry_delay": "30", 2024-09-21T13:50:01.334 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_retry_new_interval": "10", 2024-09-21T13:50:01.334 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_retry_pg_state": "60", 2024-09-21T13:50:01.334 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_retry_trimming": "10", 2024-09-21T13:50:01.334 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_scrub_sleep": "0.000000", 2024-09-21T13:50:01.334 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-09-21T13:50:01.334 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-09-21T13:50:01.334 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-21T13:50:01.334 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-09-21T13:50:01.334 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_skip_data_digest": "false", 2024-09-21T13:50:01.334 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_smart_report_timeout": "5", 2024-09-21T13:50:01.334 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_snap_trim_cost": "1048576", 2024-09-21T13:50:01.334 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_snap_trim_priority": "5", 2024-09-21T13:50:01.334 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-21T13:50:01.334 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-21T13:50:01.335 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-21T13:50:01.335 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-21T13:50:01.335 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-09-21T13:50:01.335 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_stats_update_period_scrubbing": "15", 2024-09-21T13:50:01.335 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-21T13:50:01.335 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_target_transaction_size": "30", 2024-09-21T13:50:01.335 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-21T13:50:01.335 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-21T13:50:01.335 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-21T13:50:01.335 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-21T13:50:01.335 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-21T13:50:01.335 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-21T13:50:01.335 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-21T13:50:01.336 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-21T13:50:01.336 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-21T13:50:01.336 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-21T13:50:01.336 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_tracing": "false", 2024-09-21T13:50:01.336 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_use_stale_snap": "false", 2024-09-21T13:50:01.336 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-21T13:50:01.336 INFO:teuthology.orchestra.run.smithi049.stdout: "osdc_blkin_trace_all": "false", 2024-09-21T13:50:01.336 INFO:teuthology.orchestra.run.smithi049.stdout: "paxos_kill_at": "0", 2024-09-21T13:50:01.336 INFO:teuthology.orchestra.run.smithi049.stdout: "paxos_max_join_drift": "10", 2024-09-21T13:50:01.336 INFO:teuthology.orchestra.run.smithi049.stdout: "paxos_min": "500", 2024-09-21T13:50:01.336 INFO:teuthology.orchestra.run.smithi049.stdout: "paxos_min_wait": "0.050000", 2024-09-21T13:50:01.336 INFO:teuthology.orchestra.run.smithi049.stdout: "paxos_propose_interval": "1.000000", 2024-09-21T13:50:01.336 INFO:teuthology.orchestra.run.smithi049.stdout: "paxos_service_trim_max": "500", 2024-09-21T13:50:01.336 INFO:teuthology.orchestra.run.smithi049.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-21T13:50:01.337 INFO:teuthology.orchestra.run.smithi049.stdout: "paxos_service_trim_min": "10", 2024-09-21T13:50:01.337 INFO:teuthology.orchestra.run.smithi049.stdout: "paxos_stash_full_interval": "25", 2024-09-21T13:50:01.337 INFO:teuthology.orchestra.run.smithi049.stdout: "paxos_trim_max": "500", 2024-09-21T13:50:01.337 INFO:teuthology.orchestra.run.smithi049.stdout: "paxos_trim_min": "250", 2024-09-21T13:50:01.337 INFO:teuthology.orchestra.run.smithi049.stdout: "perf": "true", 2024-09-21T13:50:01.337 INFO:teuthology.orchestra.run.smithi049.stdout: "pid_file": "", 2024-09-21T13:50:01.337 INFO:teuthology.orchestra.run.smithi049.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-21T13:50:01.337 INFO:teuthology.orchestra.run.smithi049.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-09-21T13:50:01.337 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "-", 2024-09-21T13:50:01.337 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrv": "", 2024-09-21T13:50:01.337 INFO:teuthology.orchestra.run.smithi049.stdout: "public_bind_addr": "-", 2024-09-21T13:50:01.337 INFO:teuthology.orchestra.run.smithi049.stdout: "public_network": "", 2024-09-21T13:50:01.337 INFO:teuthology.orchestra.run.smithi049.stdout: "public_network_interface": "", 2024-09-21T13:50:01.337 INFO:teuthology.orchestra.run.smithi049.stdout: "qat_compressor_busy_polling": "false", 2024-09-21T13:50:01.338 INFO:teuthology.orchestra.run.smithi049.stdout: "qat_compressor_enabled": "false", 2024-09-21T13:50:01.338 INFO:teuthology.orchestra.run.smithi049.stdout: "qat_compressor_session_max_number": "256", 2024-09-21T13:50:01.338 INFO:teuthology.orchestra.run.smithi049.stdout: "rados_mon_op_timeout": "0", 2024-09-21T13:50:01.338 INFO:teuthology.orchestra.run.smithi049.stdout: "rados_osd_op_timeout": "0", 2024-09-21T13:50:01.338 INFO:teuthology.orchestra.run.smithi049.stdout: "rados_tracing": "false", 2024-09-21T13:50:01.338 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_atime_update_interval": "60", 2024-09-21T13:50:01.338 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-21T13:50:01.338 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_balance_parent_reads": "false", 2024-09-21T13:50:01.338 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_balance_snap_reads": "false", 2024-09-21T13:50:01.338 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_blkin_trace_all": "false", 2024-09-21T13:50:01.338 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-21T13:50:01.338 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-21T13:50:01.338 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_cache": "true", 2024-09-21T13:50:01.339 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-21T13:50:01.339 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-21T13:50:01.339 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-21T13:50:01.339 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-21T13:50:01.339 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_cache_policy": "writearound", 2024-09-21T13:50:01.339 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_cache_size": "33554432", 2024-09-21T13:50:01.339 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-21T13:50:01.339 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-21T13:50:01.339 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_clone_copy_on_read": "false", 2024-09-21T13:50:01.339 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_compression_hint": "none", 2024-09-21T13:50:01.339 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_concurrent_management_ops": "10", 2024-09-21T13:50:01.339 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-09-21T13:50:01.339 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_default_clone_format": "auto", 2024-09-21T13:50:01.339 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_default_data_pool": "", 2024-09-21T13:50:01.340 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_default_features": "61", 2024-09-21T13:50:01.340 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_default_format": "2", 2024-09-21T13:50:01.340 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_default_map_options": "", 2024-09-21T13:50:01.340 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_default_order": "22", 2024-09-21T13:50:01.340 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_default_pool": "rbd", 2024-09-21T13:50:01.340 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-21T13:50:01.340 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_default_stripe_count": "0", 2024-09-21T13:50:01.340 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_default_stripe_unit": "0", 2024-09-21T13:50:01.340 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-21T13:50:01.340 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-21T13:50:01.340 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-21T13:50:01.340 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_enable_alloc_hint": "true", 2024-09-21T13:50:01.340 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-21T13:50:01.341 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_io_scheduler": "simple", 2024-09-21T13:50:01.341 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-21T13:50:01.341 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-21T13:50:01.341 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-21T13:50:01.341 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-21T13:50:01.341 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-21T13:50:01.341 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-21T13:50:01.341 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-21T13:50:01.341 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-21T13:50:01.341 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-21T13:50:01.341 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_journal_order": "24", 2024-09-21T13:50:01.341 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_journal_pool": "", 2024-09-21T13:50:01.341 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_journal_splay_width": "4", 2024-09-21T13:50:01.341 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_localize_parent_reads": "false", 2024-09-21T13:50:01.342 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_localize_snap_reads": "false", 2024-09-21T13:50:01.342 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-21T13:50:01.342 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-21T13:50:01.342 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-21T13:50:01.342 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-21T13:50:01.342 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-21T13:50:01.342 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-21T13:50:01.342 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-21T13:50:01.342 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-21T13:50:01.342 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-21T13:50:01.342 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-21T13:50:01.342 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-21T13:50:01.343 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-21T13:50:01.343 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-21T13:50:01.343 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-21T13:50:01.343 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-21T13:50:01.343 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-21T13:50:01.343 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-21T13:50:01.343 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-21T13:50:01.343 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-21T13:50:01.343 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-21T13:50:01.343 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-21T13:50:01.343 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-21T13:50:01.343 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-21T13:50:01.343 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-21T13:50:01.343 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-21T13:50:01.344 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-21T13:50:01.344 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-21T13:50:01.344 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-21T13:50:01.344 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-21T13:50:01.344 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-21T13:50:01.344 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-21T13:50:01.344 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_mtime_update_interval": "60", 2024-09-21T13:50:01.344 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_non_blocking_aio": "true", 2024-09-21T13:50:01.344 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_op_thread_timeout": "60", 2024-09-21T13:50:01.344 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_op_threads": "1", 2024-09-21T13:50:01.344 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_parent_cache_enabled": "false", 2024-09-21T13:50:01.344 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-21T13:50:01.345 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-21T13:50:01.345 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-21T13:50:01.345 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_plugins": "", 2024-09-21T13:50:01.345 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_bps_burst": "0", 2024-09-21T13:50:01.345 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-21T13:50:01.345 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_bps_limit": "0", 2024-09-21T13:50:01.345 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_exclude_ops": "0", 2024-09-21T13:50:01.345 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_iops_burst": "0", 2024-09-21T13:50:01.345 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-21T13:50:01.345 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_iops_limit": "0", 2024-09-21T13:50:01.345 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-21T13:50:01.345 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-21T13:50:01.345 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-21T13:50:01.345 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-21T13:50:01.346 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-21T13:50:01.346 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-21T13:50:01.346 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-21T13:50:01.346 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-21T13:50:01.346 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-21T13:50:01.346 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-21T13:50:01.346 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-21T13:50:01.346 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-21T13:50:01.346 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-21T13:50:01.346 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-21T13:50:01.346 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_read_from_replica_policy": "default", 2024-09-21T13:50:01.346 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-21T13:50:01.346 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-21T13:50:01.346 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-21T13:50:01.347 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-21T13:50:01.347 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_skip_partial_discard": "true", 2024-09-21T13:50:01.347 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-21T13:50:01.347 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_tracing": "false", 2024-09-21T13:50:01.347 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_validate_names": "true", 2024-09-21T13:50:01.347 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_validate_pool": "true", 2024-09-21T13:50:01.347 INFO:teuthology.orchestra.run.smithi049.stdout: "restapi_base_url": "", 2024-09-21T13:50:01.347 INFO:teuthology.orchestra.run.smithi049.stdout: "restapi_log_level": "", 2024-09-21T13:50:01.347 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-09-21T13:50:01.347 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_account_default_quota_max_size": "-1", 2024-09-21T13:50:01.347 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_acl_grants_max_num": "100", 2024-09-21T13:50:01.347 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_admin_entry": "admin", 2024-09-21T13:50:01.347 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-09-21T13:50:01.347 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_asio_assert_yielding": "false", 2024-09-21T13:50:01.348 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_backend_store": "rados", 2024-09-21T13:50:01.348 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_barbican_url": "", 2024-09-21T13:50:01.348 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_beast_enable_async": "true", 2024-09-21T13:50:01.348 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_bucket_counters_cache": "false", 2024-09-21T13:50:01.348 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-09-21T13:50:01.348 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-21T13:50:01.348 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-21T13:50:01.348 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-21T13:50:01.348 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-09-21T13:50:01.348 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-21T13:50:01.348 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-21T13:50:01.348 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-09-21T13:50:01.348 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_cache_enabled": "true", 2024-09-21T13:50:01.348 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_cache_expiry_interval": "900", 2024-09-21T13:50:01.349 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_cache_lru_size": "25000", 2024-09-21T13:50:01.349 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_config_store": "rados", 2024-09-21T13:50:01.349 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_content_length_compat": "false", 2024-09-21T13:50:01.349 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_copy_obj_progress": "true", 2024-09-21T13:50:01.349 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-21T13:50:01.349 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_cors_rules_max_num": "100", 2024-09-21T13:50:01.349 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_cross_domain_policy": "", 2024-09-21T13:50:01.349 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-21T13:50:01.349 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_kmip_addr": "", 2024-09-21T13:50:01.349 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-21T13:50:01.349 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-21T13:50:01.349 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-21T13:50:01.349 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-21T13:50:01.349 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_kmip_password": "", 2024-09-21T13:50:01.350 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-21T13:50:01.350 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_kmip_username": "", 2024-09-21T13:50:01.350 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_require_ssl": "true", 2024-09-21T13:50:01.350 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-21T13:50:01.350 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-21T13:50:01.350 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-21T13:50:01.350 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-21T13:50:01.350 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-21T13:50:01.350 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-21T13:50:01.350 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-21T13:50:01.350 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-21T13:50:01.350 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-21T13:50:01.350 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-21T13:50:01.350 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-21T13:50:01.351 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-21T13:50:01.351 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-21T13:50:01.351 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-21T13:50:01.351 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-21T13:50:01.351 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_vault_addr": "", 2024-09-21T13:50:01.351 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_vault_auth": "token", 2024-09-21T13:50:01.351 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_vault_namespace": "", 2024-09-21T13:50:01.351 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_vault_prefix": "", 2024-09-21T13:50:01.351 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-21T13:50:01.351 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-21T13:50:01.351 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-21T13:50:01.351 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-21T13:50:01.351 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_vault_token_file": "", 2024-09-21T13:50:01.351 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-21T13:50:01.352 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_curl_buffersize": "524288", 2024-09-21T13:50:01.352 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-21T13:50:01.352 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_curl_low_speed_time": "300", 2024-09-21T13:50:01.352 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_curl_tcp_keepalive": "0", 2024-09-21T13:50:01.352 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-21T13:50:01.352 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-21T13:50:01.352 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-21T13:50:01.352 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-21T13:50:01.352 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-21T13:50:01.352 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-21T13:50:01.352 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-21T13:50:01.352 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-21T13:50:01.352 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-21T13:50:01.352 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_d4n_address": "127.0.0.1:6379", 2024-09-21T13:50:01.353 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_d4n_l1_datacache_address": "127.0.0.1:6379", 2024-09-21T13:50:01.353 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2024-09-21T13:50:01.353 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_d4n_l1_datacache_size": "1073741824", 2024-09-21T13:50:01.353 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2024-09-21T13:50:01.353 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_d4n_l1_fadvise": "4", 2024-09-21T13:50:01.353 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_d4n_libaio_aio_num": "64", 2024-09-21T13:50:01.353 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_d4n_libaio_aio_threads": "20", 2024-09-21T13:50:01.353 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-21T13:50:01.353 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_data_log_changes_size": "1000", 2024-09-21T13:50:01.353 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_data_log_num_shards": "128", 2024-09-21T13:50:01.353 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-21T13:50:01.353 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_data_log_window": "30", 2024-09-21T13:50:01.354 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_data_notify_interval_msec": "0", 2024-09-21T13:50:01.354 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_data_sync_poll_interval": "20", 2024-09-21T13:50:01.354 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_data_sync_spawn_window": "20", 2024-09-21T13:50:01.354 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-09-21T13:50:01.354 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-21T13:50:01.354 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-21T13:50:01.354 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-21T13:50:01.354 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-21T13:50:01.354 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-21T13:50:01.354 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-21T13:50:01.354 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-21T13:50:01.354 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-21T13:50:01.354 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-21T13:50:01.355 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_disable_s3select": "false", 2024-09-21T13:50:01.355 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-21T13:50:01.355 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-21T13:50:01.355 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-21T13:50:01.355 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-21T13:50:01.355 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-21T13:50:01.355 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-21T13:50:01.355 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-21T13:50:01.355 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-21T13:50:01.355 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-21T13:50:01.355 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-21T13:50:01.355 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-21T13:50:01.355 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-21T13:50:01.356 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_dns_name": "", 2024-09-21T13:50:01.356 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_dns_s3website_name": "", 2024-09-21T13:50:01.356 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_dynamic_resharding": "true", 2024-09-21T13:50:01.356 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_dynamic_resharding_may_reduce": "true", 2024-09-21T13:50:01.356 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_dynamic_resharding_reduction_wait": "120", 2024-09-21T13:50:01.356 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-21T13:50:01.356 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_enable_gc_threads": "true", 2024-09-21T13:50:01.356 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_enable_lc_threads": "true", 2024-09-21T13:50:01.356 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_enable_mdsearch": "true", 2024-09-21T13:50:01.356 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_enable_ops_log": "false", 2024-09-21T13:50:01.356 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_enable_quota_threads": "true", 2024-09-21T13:50:01.356 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_enable_static_website": "false", 2024-09-21T13:50:01.356 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_enable_usage_log": "false", 2024-09-21T13:50:01.356 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_enforce_swift_acls": "true", 2024-09-21T13:50:01.357 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_exit_timeout_secs": "120", 2024-09-21T13:50:01.357 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_expose_bucket": "false", 2024-09-21T13:50:01.357 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_extended_http_attrs": "", 2024-09-21T13:50:01.357 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_filter": "none", 2024-09-21T13:50:01.357 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-21T13:50:01.357 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_frontends": "beast port=7480", 2024-09-21T13:50:01.357 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-21T13:50:01.357 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_gc_max_deferred": "50", 2024-09-21T13:50:01.357 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-21T13:50:01.357 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_gc_max_objs": "32", 2024-09-21T13:50:01.357 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-21T13:50:01.357 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-21T13:50:01.357 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-21T13:50:01.358 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-21T13:50:01.358 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_gc_processor_period": "3600", 2024-09-21T13:50:01.358 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-21T13:50:01.358 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-21T13:50:01.358 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-21T13:50:01.358 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-21T13:50:01.358 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_init_timeout": "300", 2024-09-21T13:50:01.358 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-21T13:50:01.358 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-09-21T13:50:01.358 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_kafka_connection_idle": "300", 2024-09-21T13:50:01.358 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_kafka_message_timeout": "5000", 2024-09-21T13:50:01.358 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_kafka_sleep_timeout": "10", 2024-09-21T13:50:01.358 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-21T13:50:01.359 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-09-21T13:50:01.359 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-21T13:50:01.359 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_admin_domain": "", 2024-09-21T13:50:01.359 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_admin_password": "", 2024-09-21T13:50:01.359 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_admin_password_path": "", 2024-09-21T13:50:01.359 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_admin_project": "", 2024-09-21T13:50:01.359 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_admin_tenant": "", 2024-09-21T13:50:01.359 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_admin_token": "", 2024-09-21T13:50:01.359 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_admin_token_path": "", 2024-09-21T13:50:01.359 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_admin_user": "", 2024-09-21T13:50:01.359 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_barbican_domain": "", 2024-09-21T13:50:01.359 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_barbican_password": "", 2024-09-21T13:50:01.359 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_barbican_project": "", 2024-09-21T13:50:01.359 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-21T13:50:01.360 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_barbican_user": "", 2024-09-21T13:50:01.360 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-21T13:50:01.360 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-21T13:50:01.360 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-21T13:50:01.360 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-21T13:50:01.360 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-21T13:50:01.360 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_url": "", 2024-09-21T13:50:01.360 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-21T13:50:01.360 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_lc_debug_interval": "-1", 2024-09-21T13:50:01.360 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_lc_lock_max_time": "90", 2024-09-21T13:50:01.360 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_lc_max_objs": "32", 2024-09-21T13:50:01.360 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_lc_max_rules": "1000", 2024-09-21T13:50:01.360 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_lc_max_worker": "3", 2024-09-21T13:50:01.360 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-21T13:50:01.361 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_lc_thread_delay": "0", 2024-09-21T13:50:01.361 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-21T13:50:01.361 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_ldap_dnattr": "uid", 2024-09-21T13:50:01.361 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-21T13:50:01.361 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_ldap_searchfilter": "", 2024-09-21T13:50:01.361 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-21T13:50:01.361 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_ldap_uri": "", 2024-09-21T13:50:01.361 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_lfuda_sync_frequency": "60", 2024-09-21T13:50:01.361 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-21T13:50:01.361 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-21T13:50:01.361 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-21T13:50:01.361 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_log_http_headers": "", 2024-09-21T13:50:01.361 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-21T13:50:01.362 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-21T13:50:01.362 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_log_object_name_utc": "false", 2024-09-21T13:50:01.362 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-09-21T13:50:01.362 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-09-21T13:50:01.362 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_max_attr_name_len": "0", 2024-09-21T13:50:01.362 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_max_attr_size": "0", 2024-09-21T13:50:01.362 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-21T13:50:01.362 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_max_chunk_size": "4194304", 2024-09-21T13:50:01.362 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-21T13:50:01.362 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-09-21T13:50:01.362 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-21T13:50:01.362 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_max_listing_results": "5000", 2024-09-21T13:50:01.362 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_max_notify_retries": "10", 2024-09-21T13:50:01.363 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-21T13:50:01.363 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_max_put_param_size": "1048576", 2024-09-21T13:50:01.363 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_max_put_size": "5368709120", 2024-09-21T13:50:01.363 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_max_slo_entries": "1000", 2024-09-21T13:50:01.363 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_md_log_max_shards": "64", 2024-09-21T13:50:01.363 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-21T13:50:01.363 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_meta_sync_poll_interval": "20", 2024-09-21T13:50:01.363 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_meta_sync_spawn_window": "20", 2024-09-21T13:50:01.363 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-21T13:50:01.363 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_mp_lock_max_time": "600", 2024-09-21T13:50:01.363 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-21T13:50:01.363 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-21T13:50:01.363 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-21T13:50:01.364 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-21T13:50:01.364 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-21T13:50:01.364 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-09-21T13:50:01.364 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-21T13:50:01.364 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-21T13:50:01.364 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_nfs_max_gc": "300", 2024-09-21T13:50:01.364 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-21T13:50:01.364 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-21T13:50:01.364 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-21T13:50:01.364 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-21T13:50:01.364 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-21T13:50:01.364 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-21T13:50:01.364 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-21T13:50:01.365 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_num_async_rados_threads": "32", 2024-09-21T13:50:01.365 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_num_control_oids": "8", 2024-09-21T13:50:01.365 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_numa_node": "-1", 2024-09-21T13:50:01.365 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-21T13:50:01.365 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-21T13:50:01.365 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_objexp_chunk_size": "100", 2024-09-21T13:50:01.365 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_objexp_gc_interval": "600", 2024-09-21T13:50:01.365 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-21T13:50:01.365 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-21T13:50:01.365 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-21T13:50:01.365 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_op_thread_timeout": "600", 2024-09-21T13:50:01.365 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_op_tracing": "false", 2024-09-21T13:50:01.365 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_opa_token": "", 2024-09-21T13:50:01.365 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_opa_url": "", 2024-09-21T13:50:01.366 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_opa_verify_ssl": "true", 2024-09-21T13:50:01.366 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-21T13:50:01.366 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-09-21T13:50:01.366 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_ops_log_rados": "false", 2024-09-21T13:50:01.366 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_ops_log_socket_path": "", 2024-09-21T13:50:01.366 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-21T13:50:01.366 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-09-21T13:50:01.366 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-21T13:50:01.366 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_period_push_interval": "2.000000", 2024-09-21T13:50:01.366 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-21T13:50:01.366 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-21T13:50:01.366 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-09-21T13:50:01.366 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-09-21T13:50:01.367 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_posix_cache_lanes": "3", 2024-09-21T13:50:01.367 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-09-21T13:50:01.367 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_posix_cache_max_buckets": "100", 2024-09-21T13:50:01.367 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_posix_cache_partitions": "3", 2024-09-21T13:50:01.367 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-09-21T13:50:01.367 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_print_continue": "true", 2024-09-21T13:50:01.367 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-21T13:50:01.367 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-21T13:50:01.367 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-21T13:50:01.367 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-21T13:50:01.367 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-21T13:50:01.367 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_rados_tracing": "false", 2024-09-21T13:50:01.367 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_realm": "", 2024-09-21T13:50:01.367 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_realm_id": "", 2024-09-21T13:50:01.368 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-21T13:50:01.368 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_region": "", 2024-09-21T13:50:01.368 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-21T13:50:01.368 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-21T13:50:01.368 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-21T13:50:01.368 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-21T13:50:01.368 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_request_uri": "", 2024-09-21T13:50:01.368 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_reshard_batch_size": "64", 2024-09-21T13:50:01.368 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-21T13:50:01.368 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_reshard_debug_interval": "-1", 2024-09-21T13:50:01.368 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_reshard_max_aio": "128", 2024-09-21T13:50:01.368 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_reshard_num_logs": "16", 2024-09-21T13:50:01.368 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_reshard_progress_judge_interval": "120", 2024-09-21T13:50:01.369 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_reshard_progress_judge_ratio": "0.500000", 2024-09-21T13:50:01.369 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_reshard_thread_interval": "600", 2024-09-21T13:50:01.369 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_reshardlog_threshold": "30000", 2024-09-21T13:50:01.369 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_resolve_cname": "false", 2024-09-21T13:50:01.369 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-21T13:50:01.369 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_run_sync_thread": "true", 2024-09-21T13:50:01.369 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-09-21T13:50:01.369 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-21T13:50:01.369 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-21T13:50:01.369 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-21T13:50:01.369 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-21T13:50:01.369 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-21T13:50:01.369 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-21T13:50:01.369 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-21T13:50:01.370 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-21T13:50:01.370 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_scheduler_type": "throttler", 2024-09-21T13:50:01.370 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_script_uri": "", 2024-09-21T13:50:01.370 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_service_provider_name": "", 2024-09-21T13:50:01.370 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-21T13:50:01.370 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_sts_client_id": "", 2024-09-21T13:50:01.370 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_sts_client_secret": "", 2024-09-21T13:50:01.370 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_sts_entry": "sts", 2024-09-21T13:50:01.370 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_sts_key": "sts", 2024-09-21T13:50:01.370 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-21T13:50:01.370 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_sts_min_session_duration": "900", 2024-09-21T13:50:01.370 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_sts_token_introspection_url": "", 2024-09-21T13:50:01.370 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_swift_account_in_url": "false", 2024-09-21T13:50:01.371 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_swift_auth_entry": "auth", 2024-09-21T13:50:01.371 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_swift_auth_url": "", 2024-09-21T13:50:01.371 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_swift_custom_header": "", 2024-09-21T13:50:01.371 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-21T13:50:01.371 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_swift_need_stats": "true", 2024-09-21T13:50:01.371 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_swift_tenant_name": "", 2024-09-21T13:50:01.371 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_swift_token_expiration": "86400", 2024-09-21T13:50:01.371 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_swift_url": "", 2024-09-21T13:50:01.371 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_swift_url_prefix": "swift", 2024-09-21T13:50:01.371 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-21T13:50:01.371 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-09-21T13:50:01.371 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-21T13:50:01.371 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_sync_lease_period": "120", 2024-09-21T13:50:01.371 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-21T13:50:01.372 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-21T13:50:01.372 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-21T13:50:01.372 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-21T13:50:01.372 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-21T13:50:01.372 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-21T13:50:01.372 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-21T13:50:01.372 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-21T13:50:01.372 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-21T13:50:01.372 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_thread_pool_size": "512", 2024-09-21T13:50:01.372 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_topic_persistency_max_retries": "0", 2024-09-21T13:50:01.372 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-09-21T13:50:01.372 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-09-21T13:50:01.372 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_topic_require_publish_policy": "false", 2024-09-21T13:50:01.372 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_torrent_comment": "", 2024-09-21T13:50:01.372 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_torrent_createby": "", 2024-09-21T13:50:01.373 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_torrent_encoding": "", 2024-09-21T13:50:01.373 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_torrent_flag": "false", 2024-09-21T13:50:01.373 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_torrent_max_size": "5368709120", 2024-09-21T13:50:01.373 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_torrent_origin": "", 2024-09-21T13:50:01.373 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-21T13:50:01.373 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_torrent_tracker": "", 2024-09-21T13:50:01.373 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_trust_forwarded_https": "false", 2024-09-21T13:50:01.373 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-21T13:50:01.373 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-21T13:50:01.373 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_usage_max_shards": "32", 2024-09-21T13:50:01.373 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_usage_max_user_shards": "1", 2024-09-21T13:50:01.373 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_use_opa_authz": "false", 2024-09-21T13:50:01.373 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_user_counters_cache": "false", 2024-09-21T13:50:01.374 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_user_counters_cache_size": "10000", 2024-09-21T13:50:01.374 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-21T13:50:01.374 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-21T13:50:01.374 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_user_max_buckets": "1000", 2024-09-21T13:50:01.374 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_user_policies_max_num": "100", 2024-09-21T13:50:01.374 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-21T13:50:01.374 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-21T13:50:01.374 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-21T13:50:01.374 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-21T13:50:01.374 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_user_unique_email": "true", 2024-09-21T13:50:01.374 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_verify_ssl": "true", 2024-09-21T13:50:01.375 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-21T13:50:01.375 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_zone": "", 2024-09-21T13:50:01.375 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_zone_id": "", 2024-09-21T13:50:01.375 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-21T13:50:01.375 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_zonegroup": "", 2024-09-21T13:50:01.375 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_zonegroup_id": "", 2024-09-21T13:50:01.375 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-21T13:50:01.375 INFO:teuthology.orchestra.run.smithi049.stdout: "rgwlc_auto_session_clear": "true", 2024-09-21T13:50:01.375 INFO:teuthology.orchestra.run.smithi049.stdout: "rgwlc_skip_bucket_step": "false", 2024-09-21T13:50:01.375 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_block_size": "4096", 2024-09-21T13:50:01.375 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-21T13:50:01.375 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-21T13:50:01.375 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-21T13:50:01.375 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-21T13:50:01.376 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-21T13:50:01.376 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_cache_size": "536870912", 2024-09-21T13:50:01.376 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-21T13:50:01.376 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-21T13:50:01.376 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-21T13:50:01.376 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-21T13:50:01.376 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-21T13:50:01.376 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-21T13:50:01.376 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-21T13:50:01.376 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-21T13:50:01.376 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_index_type": "binary_search", 2024-09-21T13:50:01.376 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-21T13:50:01.376 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-21T13:50:01.376 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_partition_filters": "false", 2024-09-21T13:50:01.377 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_perf": "false", 2024-09-21T13:50:01.377 INFO:teuthology.orchestra.run.smithi049.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-21T13:50:01.377 INFO:teuthology.orchestra.run.smithi049.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-21T13:50:01.377 INFO:teuthology.orchestra.run.smithi049.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-21T13:50:01.377 INFO:teuthology.orchestra.run.smithi049.stdout: "run_dir": "/var/run/ceph", 2024-09-21T13:50:01.377 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_block_create": "true", 2024-09-21T13:50:01.377 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_cache_lru_size": "67108864", 2024-09-21T13:50:01.377 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_cbjournal_size": "5368709120", 2024-09-21T13:50:01.377 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_data_delta_based_overwrite": "0", 2024-09-21T13:50:01.377 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_default_max_object_size": "16777216", 2024-09-21T13:50:01.377 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-21T13:50:01.377 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_device_size": "53687091200", 2024-09-21T13:50:01.377 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_disable_end_to_end_data_protection": "true", 2024-09-21T13:50:01.377 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_full_integrity_check": "false", 2024-09-21T13:50:01.378 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_journal_batch_capacity": "16", 2024-09-21T13:50:01.378 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-21T13:50:01.378 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-21T13:50:01.378 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-21T13:50:01.378 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_main_device_type": "SSD", 2024-09-21T13:50:01.378 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_max_concurrent_transactions": "8", 2024-09-21T13:50:01.378 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_max_data_allocation_size": "32768", 2024-09-21T13:50:01.378 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-09-21T13:50:01.378 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-09-21T13:50:01.378 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-09-21T13:50:01.378 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-09-21T13:50:01.378 INFO:teuthology.orchestra.run.smithi049.stdout: "seastore_segment_size": "67108864", 2024-09-21T13:50:01.378 INFO:teuthology.orchestra.run.smithi049.stdout: "set_keepcaps": "false", 2024-09-21T13:50:01.378 INFO:teuthology.orchestra.run.smithi049.stdout: "setgroup": "ceph", 2024-09-21T13:50:01.379 INFO:teuthology.orchestra.run.smithi049.stdout: "setuser": "ceph", 2024-09-21T13:50:01.379 INFO:teuthology.orchestra.run.smithi049.stdout: "setuser_match_path": "", 2024-09-21T13:50:01.379 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-21T13:50:01.379 INFO:teuthology.orchestra.run.smithi049.stdout: "thp": "false", 2024-09-21T13:50:01.379 INFO:teuthology.orchestra.run.smithi049.stdout: "threadpool_default_timeout": "60", 2024-09-21T13:50:01.379 INFO:teuthology.orchestra.run.smithi049.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-21T13:50:01.379 INFO:teuthology.orchestra.run.smithi049.stdout: "throttler_perf_counter": "true", 2024-09-21T13:50:01.379 INFO:teuthology.orchestra.run.smithi049.stdout: "tmp_dir": "/tmp", 2024-09-21T13:50:01.379 INFO:teuthology.orchestra.run.smithi049.stdout: "tmp_file_template": "/tmp/ceph-mon.a.XXXXXX", 2024-09-21T13:50:01.379 INFO:teuthology.orchestra.run.smithi049.stdout: "uadk_compressor_enabled": "false", 2024-09-21T13:50:01.379 INFO:teuthology.orchestra.run.smithi049.stdout: "uadk_wd_sync_ctx_num": "2" 2024-09-21T13:50:01.379 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T13:50:02.396 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-21T13:50:02.652 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:50:02.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:02 smithi187 ceph-mon[33032]: pgmap v27: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 784 B/s rd, 0 op/s 2024-09-21T13:50:02.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:02 smithi049 ceph-mon[40489]: pgmap v27: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 784 B/s rd, 0 op/s 2024-09-21T13:50:02.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:02 smithi049 ceph-mon[30080]: pgmap v27: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 784 B/s rd, 0 op/s 2024-09-21T13:50:03.171 INFO:teuthology.orchestra.run.smithi049.stdout:mon.a: {} 2024-09-21T13:50:03.171 INFO:teuthology.orchestra.run.smithi049.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T13:50:03.182 INFO:teuthology.orchestra.run.smithi049.stdout:mon.b: {} 2024-09-21T13:50:03.182 INFO:teuthology.orchestra.run.smithi049.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T13:50:03.190 INFO:teuthology.orchestra.run.smithi049.stdout:mon.c: {} 2024-09-21T13:50:03.190 INFO:teuthology.orchestra.run.smithi049.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T13:50:03.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:03 smithi187 ceph-mon[33032]: 4.13 scrub starts 2024-09-21T13:50:03.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:03 smithi187 ceph-mon[33032]: 4.13 scrub ok 2024-09-21T13:50:03.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:03 smithi187 ceph-mon[33032]: 5.8 scrub starts 2024-09-21T13:50:03.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:03 smithi187 ceph-mon[33032]: 5.8 scrub ok 2024-09-21T13:50:03.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:03 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3213816040' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T13:50:03.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:03 smithi049 ceph-mon[40489]: 4.13 scrub starts 2024-09-21T13:50:03.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:03 smithi049 ceph-mon[40489]: 4.13 scrub ok 2024-09-21T13:50:03.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:03 smithi049 ceph-mon[40489]: 5.8 scrub starts 2024-09-21T13:50:03.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:03 smithi049 ceph-mon[40489]: 5.8 scrub ok 2024-09-21T13:50:03.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:03 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3213816040' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T13:50:03.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:03 smithi049 ceph-mon[30080]: 4.13 scrub starts 2024-09-21T13:50:03.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:03 smithi049 ceph-mon[30080]: 4.13 scrub ok 2024-09-21T13:50:03.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:03 smithi049 ceph-mon[30080]: 5.8 scrub starts 2024-09-21T13:50:03.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:03 smithi049 ceph-mon[30080]: 5.8 scrub ok 2024-09-21T13:50:03.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:03 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3213816040' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T13:50:03.903 INFO:teuthology.run_tasks:Running task full_sequential... 2024-09-21T13:50:03.913 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-21T13:50:03.914 INFO:tasks.radosbench:Beginning radosbench... 2024-09-21T13:50:03.914 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-09-21T13:50:03.914 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd pool create unique_pool_0 16 2024-09-21T13:50:03.915 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-21T13:50:03.915 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T13:50:03.915 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T13:50:03.915 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-09-21T13:50:03.915 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T13:50:03.916 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T13:50:03.916 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-09-21T13:50:03.916 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-09-21T13:50:03.916 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:03.917 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-21T13:50:03.917 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T13:50:03.917 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-09-21T13:50:03.917 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T13:50:03.918 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-09-21T13:50:03.918 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T13:50:03.945 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:03.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:03.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:04.047 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:50:04.072 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:04.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:04.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:50:04.174 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:50:04.200 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:04.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:04.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:50:04.210 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:50:04.214 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:50:04.301 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:50:04.323 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:04.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:04.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:50:04.425 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:50:04.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:04.448 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:04.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:50:04.549 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:04.570 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:04.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:04.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:04.672 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:04.683 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:04 smithi049 ceph-mon[40489]: 5.5 scrub starts 2024-09-21T13:50:04.683 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:04 smithi049 ceph-mon[40489]: 5.5 scrub ok 2024-09-21T13:50:04.683 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:04 smithi049 ceph-mon[40489]: pgmap v28: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T13:50:04.683 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:04 smithi049 ceph-mon[30080]: 5.5 scrub starts 2024-09-21T13:50:04.683 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:04 smithi049 ceph-mon[30080]: 5.5 scrub ok 2024-09-21T13:50:04.683 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:04 smithi049 ceph-mon[30080]: pgmap v28: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T13:50:04.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:04 smithi187 ceph-mon[33032]: 5.5 scrub starts 2024-09-21T13:50:04.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:04 smithi187 ceph-mon[33032]: 5.5 scrub ok 2024-09-21T13:50:04.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:04 smithi187 ceph-mon[33032]: pgmap v28: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T13:50:04.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:04.695 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:04.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:04.796 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:50:04.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:04.820 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:04.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:50:04.921 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:50:04.943 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:04.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:04.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:50:04.970 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:50:05.035 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:50:05.035 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:50:05.038 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:50:05.039 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:50:05.045 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:50:05.055 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:50:05.056 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:50:05.067 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:05.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:05.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:50:05.072 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:50:05.072 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:50:05.089 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:50:05.089 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:50:05.106 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:50:05.107 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:50:05.123 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:50:05.123 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:50:05.140 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:50:05.140 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:50:05.169 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:50:05.191 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:05.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:05.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:50:05.292 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:50:05.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:05.316 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:05.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:50:05.417 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:50:05.439 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:05.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:05.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:50:05.541 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:05.557 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:05 smithi049 ceph-mon[40489]: 4.a scrub starts 2024-09-21T13:50:05.557 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:05 smithi049 ceph-mon[40489]: 4.a scrub ok 2024-09-21T13:50:05.557 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:05 smithi049 ceph-mon[40489]: 3.17 scrub starts 2024-09-21T13:50:05.557 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:05 smithi049 ceph-mon[40489]: 3.17 scrub ok 2024-09-21T13:50:05.557 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:05 smithi049 ceph-mon[40489]: 6.5 scrub starts 2024-09-21T13:50:05.557 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:05 smithi049 ceph-mon[40489]: 6.5 scrub ok 2024-09-21T13:50:05.557 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:05 smithi049 ceph-mon[40489]: 4.8 scrub starts 2024-09-21T13:50:05.557 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:05 smithi049 ceph-mon[40489]: 4.8 scrub ok 2024-09-21T13:50:05.557 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:05 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3851834264' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:50:05.557 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:05 smithi049 ceph-mon[30080]: 4.a scrub starts 2024-09-21T13:50:05.558 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:05 smithi049 ceph-mon[30080]: 4.a scrub ok 2024-09-21T13:50:05.558 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:05 smithi049 ceph-mon[30080]: 3.17 scrub starts 2024-09-21T13:50:05.558 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:05 smithi049 ceph-mon[30080]: 3.17 scrub ok 2024-09-21T13:50:05.558 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:05 smithi049 ceph-mon[30080]: 6.5 scrub starts 2024-09-21T13:50:05.558 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:05 smithi049 ceph-mon[30080]: 6.5 scrub ok 2024-09-21T13:50:05.558 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:05 smithi049 ceph-mon[30080]: 4.8 scrub starts 2024-09-21T13:50:05.558 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:05 smithi049 ceph-mon[30080]: 4.8 scrub ok 2024-09-21T13:50:05.558 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:05 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3851834264' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:50:05.563 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:05.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:05.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:05.632 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T13:50:05.632 INFO:teuthology.orchestra.run.smithi187.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-21T13:50:05.633 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T13:50:05.664 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:05.685 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:05.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:05.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:05.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:05 smithi187 ceph-mon[33032]: 4.a scrub starts 2024-09-21T13:50:05.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:05 smithi187 ceph-mon[33032]: 4.a scrub ok 2024-09-21T13:50:05.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:05 smithi187 ceph-mon[33032]: 3.17 scrub starts 2024-09-21T13:50:05.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:05 smithi187 ceph-mon[33032]: 3.17 scrub ok 2024-09-21T13:50:05.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:05 smithi187 ceph-mon[33032]: 6.5 scrub starts 2024-09-21T13:50:05.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:05 smithi187 ceph-mon[33032]: 6.5 scrub ok 2024-09-21T13:50:05.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:05 smithi187 ceph-mon[33032]: 4.8 scrub starts 2024-09-21T13:50:05.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:05 smithi187 ceph-mon[33032]: 4.8 scrub ok 2024-09-21T13:50:05.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:05 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3851834264' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:50:05.787 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:05.809 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:05.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:05.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:50:05.911 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:05.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:05.934 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:05.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:06.031 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:50:06.035 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:06.036 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:50:06.056 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:06.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:06.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:06.158 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:06.180 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:06.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:06.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:50:06.281 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:50:06.304 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:06.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:06.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:50:06.405 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'unique_pool_0' created 2024-09-21T13:50:06.405 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:06.429 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:06.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:06.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:06.530 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:06.552 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:06.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:06.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:06.625 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:06 smithi049 ceph-mon[40489]: pgmap v29: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T13:50:06.625 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:06 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3868022686' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-21T13:50:06.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:06 smithi049 ceph-mon[30080]: pgmap v29: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T13:50:06.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:06 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3868022686' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-21T13:50:06.654 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:06.676 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:06.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:06.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:06.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:06 smithi187 ceph-mon[33032]: pgmap v29: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T13:50:06.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:06 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3868022686' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-21T13:50:06.778 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:50:06.801 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:06.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:06.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:50:06.856 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T13:50:06.856 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:50:06.903 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:06.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:06.927 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:06.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:07.028 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:50:07.050 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:07.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:07.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:50:07.152 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:50:07.177 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:07.178 INFO:journalctl@ceph.alertmanager.a.smithi049.stdout:Sep 21 13:50:06 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-alertmanager-a[74283]: ts=2024-09-21T13:50:06.884Z caller=cluster.go:700 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002515229s 2024-09-21T13:50:07.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:07.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:50:07.279 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:07.304 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:07.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:07.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:50:07.406 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:50:07.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:07.435 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:07.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:50:07.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:07 smithi049 ceph-mon[40489]: 3.8 scrub starts 2024-09-21T13:50:07.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:07 smithi049 ceph-mon[40489]: 3.8 scrub ok 2024-09-21T13:50:07.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:07 smithi049 ceph-mon[40489]: 6.15 scrub starts 2024-09-21T13:50:07.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:07 smithi049 ceph-mon[40489]: 6.15 scrub ok 2024-09-21T13:50:07.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:07 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3868022686' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-21T13:50:07.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:07 smithi049 ceph-mon[40489]: osdmap e83: 8 total, 8 up, 8 in 2024-09-21T13:50:07.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:07 smithi049 ceph-mon[30080]: 3.8 scrub starts 2024-09-21T13:50:07.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:07 smithi049 ceph-mon[30080]: 3.8 scrub ok 2024-09-21T13:50:07.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:07 smithi049 ceph-mon[30080]: 6.15 scrub starts 2024-09-21T13:50:07.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:07 smithi049 ceph-mon[30080]: 6.15 scrub ok 2024-09-21T13:50:07.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:07 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3868022686' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-21T13:50:07.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:07 smithi049 ceph-mon[30080]: osdmap e83: 8 total, 8 up, 8 in 2024-09-21T13:50:07.536 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:50:07.558 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:07.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:07.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:50:07.608 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-09-21T13:50:07.643 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:07 smithi187 ceph-mon[33032]: 3.8 scrub starts 2024-09-21T13:50:07.643 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:07 smithi187 ceph-mon[33032]: 3.8 scrub ok 2024-09-21T13:50:07.643 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:07 smithi187 ceph-mon[33032]: 6.15 scrub starts 2024-09-21T13:50:07.643 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:07 smithi187 ceph-mon[33032]: 6.15 scrub ok 2024-09-21T13:50:07.643 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:07 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3868022686' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-21T13:50:07.643 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:07 smithi187 ceph-mon[33032]: osdmap e83: 8 total, 8 up, 8 in 2024-09-21T13:50:07.660 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:50:07.682 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:07.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:07.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:50:07.784 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:50:07.807 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:07.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:07.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:50:07.908 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:50:07.930 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:07.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:07.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:50:07.978 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T13:50:08.023 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T13:50:08.032 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:50:08.087 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:08.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:08.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:50:08.188 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:50:08.210 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:08.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:08.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:50:08.292 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:50:08.312 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:08.334 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:08.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:08.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:50:08.435 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:50:08.458 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:08.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:08.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:50:08.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:08 smithi049 ceph-mon[30080]: 4.7 scrub starts 2024-09-21T13:50:08.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:08 smithi049 ceph-mon[30080]: 4.7 scrub ok 2024-09-21T13:50:08.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:08 smithi049 ceph-mon[30080]: pgmap v31: 148 pgs: 16 unknown, 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-21T13:50:08.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:08 smithi049 ceph-mon[30080]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T13:50:08.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:08 smithi049 ceph-mon[30080]: osdmap e84: 8 total, 8 up, 8 in 2024-09-21T13:50:08.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:08 smithi049 ceph-mon[40489]: 4.7 scrub starts 2024-09-21T13:50:08.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:08 smithi049 ceph-mon[40489]: 4.7 scrub ok 2024-09-21T13:50:08.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:08 smithi049 ceph-mon[40489]: pgmap v31: 148 pgs: 16 unknown, 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-21T13:50:08.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:08 smithi049 ceph-mon[40489]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T13:50:08.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:08 smithi049 ceph-mon[40489]: osdmap e84: 8 total, 8 up, 8 in 2024-09-21T13:50:08.527 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:08 smithi187 ceph-mon[33032]: 4.7 scrub starts 2024-09-21T13:50:08.528 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:08 smithi187 ceph-mon[33032]: 4.7 scrub ok 2024-09-21T13:50:08.528 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:08 smithi187 ceph-mon[33032]: pgmap v31: 148 pgs: 16 unknown, 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-21T13:50:08.528 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:08 smithi187 ceph-mon[33032]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T13:50:08.528 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:08 smithi187 ceph-mon[33032]: osdmap e84: 8 total, 8 up, 8 in 2024-09-21T13:50:08.560 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:08.566 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:50:08.581 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:08.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:08.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:08.683 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:08.708 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:08.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:08.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:08.809 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:50:08.832 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:08.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:08.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:50:08.934 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:50:08.956 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:08.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:08.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:50:09.058 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:50:09.065 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:50:09.080 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:09.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:09.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:50:09.182 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:50:09.195 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 systemd[1]: Starting Ceph grafana.a for 40a3d2e8-781f-11ef-baf6-efdc52797490... 2024-09-21T13:50:09.195 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:50:08 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:50:09.204 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:09.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:09.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:50:09.284 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:50:09.285 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:50:09.287 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:50:09.288 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:50:09.304 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:50:09.304 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:50:09.306 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:50:09.322 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:50:09.322 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:50:09.329 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:09.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:09.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:50:09.340 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:50:09.340 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:50:09.358 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:50:09.358 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:50:09.375 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:50:09.375 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:50:09.391 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:50:09.392 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:50:09.420 INFO:teuthology.orchestra.run.smithi049.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-09-21T13:50:09.425 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-09-21T13:50:09.431 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:50:09.457 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:09.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:09.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:50:09.559 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:50:09.581 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:09.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:09.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:50:09.582 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-mon[33032]: 3.a scrub starts 2024-09-21T13:50:09.583 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-mon[33032]: 3.a scrub ok 2024-09-21T13:50:09.583 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-mon[33032]: 3.f scrub starts 2024-09-21T13:50:09.583 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-mon[33032]: 3.f scrub ok 2024-09-21T13:50:09.583 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-mon[33032]: osdmap e85: 8 total, 8 up, 8 in 2024-09-21T13:50:09.583 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1824675077' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:50:09.583 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:50:09.583 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/18041778' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T13:50:09.584 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3530805281' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:50:09.584 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 podman[63849]: 2024-09-21 13:50:09.145058014 +0000 UTC m=+0.016687144 image pull 5aad1d7cf004c80a4acb170e1c80fb85781b022f00f08c6bcd22608882b4c748 quay.io/ceph/grafana:10.4.8 2024-09-21T13:50:09.584 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 podman[63849]: 2024-09-21 13:50:09.264222118 +0000 UTC m=+0.135851248 container create bef1e288687e5c07030d19e57743fa0825b3890b53a3d8874b18d99c050f0be8 (image=quay.io/ceph/grafana:10.4.8, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a, maintainer=Grafana Labs ) 2024-09-21T13:50:09.585 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 podman[63849]: 2024-09-21 13:50:09.483558394 +0000 UTC m=+0.355187532 container init bef1e288687e5c07030d19e57743fa0825b3890b53a3d8874b18d99c050f0be8 (image=quay.io/ceph/grafana:10.4.8, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a, maintainer=Grafana Labs ) 2024-09-21T13:50:09.585 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 podman[63849]: 2024-09-21 13:50:09.488557312 +0000 UTC m=+0.360186445 container start bef1e288687e5c07030d19e57743fa0825b3890b53a3d8874b18d99c050f0be8 (image=quay.io/ceph/grafana:10.4.8, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a, maintainer=Grafana Labs ) 2024-09-21T13:50:09.585 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 bash[63849]: bef1e288687e5c07030d19e57743fa0825b3890b53a3d8874b18d99c050f0be8 2024-09-21T13:50:09.682 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:09.704 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:09.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:09.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:50:09.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:09 smithi049 ceph-mon[40489]: 3.a scrub starts 2024-09-21T13:50:09.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:09 smithi049 ceph-mon[40489]: 3.a scrub ok 2024-09-21T13:50:09.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:09 smithi049 ceph-mon[40489]: 3.f scrub starts 2024-09-21T13:50:09.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:09 smithi049 ceph-mon[40489]: 3.f scrub ok 2024-09-21T13:50:09.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:09 smithi049 ceph-mon[40489]: osdmap e85: 8 total, 8 up, 8 in 2024-09-21T13:50:09.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:09 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1824675077' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:50:09.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:09 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:50:09.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:09 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/18041778' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T13:50:09.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:09 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3530805281' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:50:09.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:09 smithi049 ceph-mon[30080]: 3.a scrub starts 2024-09-21T13:50:09.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:09 smithi049 ceph-mon[30080]: 3.a scrub ok 2024-09-21T13:50:09.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:09 smithi049 ceph-mon[30080]: 3.f scrub starts 2024-09-21T13:50:09.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:09 smithi049 ceph-mon[30080]: 3.f scrub ok 2024-09-21T13:50:09.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:09 smithi049 ceph-mon[30080]: osdmap e85: 8 total, 8 up, 8 in 2024-09-21T13:50:09.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:09 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1824675077' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:50:09.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:09 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:50:09.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:09 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/18041778' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T13:50:09.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:09 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3530805281' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:50:09.806 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:09.829 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:09.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:09.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:09.833 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 systemd[1]: Started Ceph grafana.a for 40a3d2e8-781f-11ef-baf6-efdc52797490. 2024-09-21T13:50:09.834 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=settings t=2024-09-21T13:50:09.607435779Z level=info msg="Starting Grafana" version=10.4.8 commit=8c384b71bde142a25ddc01ed5a183ca572b2b7f7 branch=v10.4.x compiled=2024-09-21T13:50:09Z 2024-09-21T13:50:09.834 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=settings t=2024-09-21T13:50:09.607725721Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-09-21T13:50:09.834 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=settings t=2024-09-21T13:50:09.607737185Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-09-21T13:50:09.834 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=settings t=2024-09-21T13:50:09.607741813Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana" 2024-09-21T13:50:09.835 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=settings t=2024-09-21T13:50:09.60774572Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana" 2024-09-21T13:50:09.835 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=settings t=2024-09-21T13:50:09.607749452Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins" 2024-09-21T13:50:09.835 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=settings t=2024-09-21T13:50:09.607753024Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning" 2024-09-21T13:50:09.835 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=settings t=2024-09-21T13:50:09.607756675Z level=info msg="Config overridden from command line" arg="default.log.mode=console" 2024-09-21T13:50:09.835 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=settings t=2024-09-21T13:50:09.607760445Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-09-21T13:50:09.835 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=settings t=2024-09-21T13:50:09.607764272Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-09-21T13:50:09.837 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=settings t=2024-09-21T13:50:09.607767711Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-09-21T13:50:09.837 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=settings t=2024-09-21T13:50:09.607771224Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-09-21T13:50:09.838 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=settings t=2024-09-21T13:50:09.607775549Z level=info msg=Target target=[all] 2024-09-21T13:50:09.838 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=settings t=2024-09-21T13:50:09.607783893Z level=info msg="Path Home" path=/usr/share/grafana 2024-09-21T13:50:09.838 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=settings t=2024-09-21T13:50:09.607788216Z level=info msg="Path Data" path=/var/lib/grafana 2024-09-21T13:50:09.838 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=settings t=2024-09-21T13:50:09.607792494Z level=info msg="Path Logs" path=/var/log/grafana 2024-09-21T13:50:09.838 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=settings t=2024-09-21T13:50:09.60779669Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-09-21T13:50:09.838 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=settings t=2024-09-21T13:50:09.607801144Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-09-21T13:50:09.843 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=settings t=2024-09-21T13:50:09.607805565Z level=info msg="App mode production" 2024-09-21T13:50:09.843 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=sqlstore t=2024-09-21T13:50:09.60817828Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-09-21T13:50:09.843 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=sqlstore t=2024-09-21T13:50:09.608196607Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-09-21T13:50:09.843 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.608823738Z level=info msg="Starting DB migrations" 2024-09-21T13:50:09.844 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.609714852Z level=info msg="Executing migration" id="create migration_log table" 2024-09-21T13:50:09.844 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.610480765Z level=info msg="Migration successfully executed" id="create migration_log table" duration=765.795µs 2024-09-21T13:50:09.844 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.610952516Z level=info msg="Executing migration" id="create user table" 2024-09-21T13:50:09.844 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.611494155Z level=info msg="Migration successfully executed" id="create user table" duration=541.895µs 2024-09-21T13:50:09.844 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.611895022Z level=info msg="Executing migration" id="add unique index user.login" 2024-09-21T13:50:09.844 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.61266265Z level=info msg="Migration successfully executed" id="add unique index user.login" duration=745.3µs 2024-09-21T13:50:09.844 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.614405385Z level=info msg="Executing migration" id="add unique index user.email" 2024-09-21T13:50:09.844 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.615264052Z level=info msg="Migration successfully executed" id="add unique index user.email" duration=859.548µs 2024-09-21T13:50:09.845 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.615785317Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-09-21T13:50:09.845 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.616346399Z level=info msg="Migration successfully executed" id="drop index UQE_user_login - v1" duration=561.282µs 2024-09-21T13:50:09.845 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.616889731Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-09-21T13:50:09.845 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.617369293Z level=info msg="Migration successfully executed" id="drop index UQE_user_email - v1" duration=479.985µs 2024-09-21T13:50:09.845 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.61789542Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-09-21T13:50:09.845 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.619547247Z level=info msg="Migration successfully executed" id="Rename table user to user_v1 - v1" duration=1.651022ms 2024-09-21T13:50:09.845 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.620010251Z level=info msg="Executing migration" id="create user table v2" 2024-09-21T13:50:09.845 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.620551697Z level=info msg="Migration successfully executed" id="create user table v2" duration=542.347µs 2024-09-21T13:50:09.846 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.620928129Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-09-21T13:50:09.846 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.621404528Z level=info msg="Migration successfully executed" id="create index UQE_user_login - v2" duration=476.881µs 2024-09-21T13:50:09.846 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.62174702Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-09-21T13:50:09.846 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.62221058Z level=info msg="Migration successfully executed" id="create index UQE_user_email - v2" duration=463.353µs 2024-09-21T13:50:09.846 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.622622347Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-09-21T13:50:09.846 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.622936557Z level=info msg="Migration successfully executed" id="copy data_source v1 to v2" duration=314.104µs 2024-09-21T13:50:09.846 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.62329997Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-09-21T13:50:09.846 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.623707558Z level=info msg="Migration successfully executed" id="Drop old table user_v1" duration=406.792µs 2024-09-21T13:50:09.847 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.624083134Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-09-21T13:50:09.847 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.624832769Z level=info msg="Migration successfully executed" id="Add column help_flags1 to user table" duration=749.311µs 2024-09-21T13:50:09.847 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.625253417Z level=info msg="Executing migration" id="Update user table charset" 2024-09-21T13:50:09.847 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.625269938Z level=info msg="Migration successfully executed" id="Update user table charset" duration=17.219µs 2024-09-21T13:50:09.847 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.625713071Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-09-21T13:50:09.847 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.626471699Z level=info msg="Migration successfully executed" id="Add last_seen_at column to user" duration=758.463µs 2024-09-21T13:50:09.847 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.626882633Z level=info msg="Executing migration" id="Add missing user data" 2024-09-21T13:50:09.848 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.627042207Z level=info msg="Migration successfully executed" id="Add missing user data" duration=159.455µs 2024-09-21T13:50:09.848 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.627464624Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-09-21T13:50:09.848 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.628206273Z level=info msg="Migration successfully executed" id="Add is_disabled column to user" duration=741.499µs 2024-09-21T13:50:09.848 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.628576566Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-09-21T13:50:09.848 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.629075752Z level=info msg="Migration successfully executed" id="Add index user.login/user.email" duration=497.184µs 2024-09-21T13:50:09.848 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.62944752Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-09-21T13:50:09.848 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.630229112Z level=info msg="Migration successfully executed" id="Add is_service_account column to user" duration=781.253µs 2024-09-21T13:50:09.849 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.630597983Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-09-21T13:50:09.849 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.635873467Z level=info msg="Migration successfully executed" id="Update is_service_account column to nullable" duration=5.272801ms 2024-09-21T13:50:09.849 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.636396013Z level=info msg="Executing migration" id="Add uid column to user" 2024-09-21T13:50:09.849 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.637208367Z level=info msg="Migration successfully executed" id="Add uid column to user" duration=812.393µs 2024-09-21T13:50:09.849 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.637600795Z level=info msg="Executing migration" id="Update uid column values for users" 2024-09-21T13:50:09.850 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.637754602Z level=info msg="Migration successfully executed" id="Update uid column values for users" duration=155.91µs 2024-09-21T13:50:09.850 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.638164064Z level=info msg="Executing migration" id="Add unique index user_uid" 2024-09-21T13:50:09.850 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.638643557Z level=info msg="Migration successfully executed" id="Add unique index user_uid" duration=478.943µs 2024-09-21T13:50:09.850 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.639082461Z level=info msg="Executing migration" id="update login field with orgid to allow for multiple service accounts with same name across orgs" 2024-09-21T13:50:09.850 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.639295498Z level=info msg="Migration successfully executed" id="update login field with orgid to allow for multiple service accounts with same name across orgs" duration=213.83µs 2024-09-21T13:50:09.850 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.639703079Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-09-21T13:50:09.851 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.640233094Z level=info msg="Migration successfully executed" id="create temp user table v1-7" duration=530.217µs 2024-09-21T13:50:09.851 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.64061599Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-09-21T13:50:09.851 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.64108674Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_email - v1-7" duration=470.465µs 2024-09-21T13:50:09.851 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.641426393Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-09-21T13:50:09.851 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.641894531Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_org_id - v1-7" duration=467.901µs 2024-09-21T13:50:09.851 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.642240753Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-09-21T13:50:09.852 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.642695555Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_code - v1-7" duration=454.562µs 2024-09-21T13:50:09.852 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.643051173Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-09-21T13:50:09.852 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.643497997Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_status - v1-7" duration=446.541µs 2024-09-21T13:50:09.852 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.643823844Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-09-21T13:50:09.852 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.643853854Z level=info msg="Migration successfully executed" id="Update temp_user table charset" duration=30.451µs 2024-09-21T13:50:09.852 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.644258898Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-09-21T13:50:09.852 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.644724504Z level=info msg="Migration successfully executed" id="drop index IDX_temp_user_email - v1" duration=464.639µs 2024-09-21T13:50:09.853 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.645066472Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-09-21T13:50:09.853 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.645524467Z level=info msg="Migration successfully executed" id="drop index IDX_temp_user_org_id - v1" duration=457.744µs 2024-09-21T13:50:09.853 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.645852669Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-09-21T13:50:09.853 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.646303264Z level=info msg="Migration successfully executed" id="drop index IDX_temp_user_code - v1" duration=465.616µs 2024-09-21T13:50:09.853 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.646636066Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-09-21T13:50:09.853 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.64709127Z level=info msg="Migration successfully executed" id="drop index IDX_temp_user_status - v1" duration=455.515µs 2024-09-21T13:50:09.853 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.647426189Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-09-21T13:50:09.853 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.649412599Z level=info msg="Migration successfully executed" id="Rename table temp_user to temp_user_tmp_qwerty - v1" duration=1.985802ms 2024-09-21T13:50:09.853 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.649731227Z level=info msg="Executing migration" id="create temp_user v2" 2024-09-21T13:50:09.854 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.650275308Z level=info msg="Migration successfully executed" id="create temp_user v2" duration=543.962µs 2024-09-21T13:50:09.854 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.650606919Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-09-21T13:50:09.854 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.651100781Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_email - v2" duration=493.615µs 2024-09-21T13:50:09.854 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.6514341Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-09-21T13:50:09.854 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.651916204Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_org_id - v2" duration=482.424µs 2024-09-21T13:50:09.854 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.652245004Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-09-21T13:50:09.854 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.652712007Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_code - v2" duration=466.852µs 2024-09-21T13:50:09.854 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.653055684Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-09-21T13:50:09.854 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.65352109Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_status - v2" duration=465.595µs 2024-09-21T13:50:09.855 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.65386015Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-09-21T13:50:09.855 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.654113921Z level=info msg="Migration successfully executed" id="copy temp_user v1 to v2" duration=253.668µs 2024-09-21T13:50:09.855 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.654444028Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-09-21T13:50:09.855 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.654791116Z level=info msg="Migration successfully executed" id="drop temp_user_tmp_qwerty" duration=356.085µs 2024-09-21T13:50:09.855 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.655154047Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-09-21T13:50:09.855 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.655405548Z level=info msg="Migration successfully executed" id="Set created for temp users that will otherwise prematurely expire" duration=251.403µs 2024-09-21T13:50:09.856 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.655781018Z level=info msg="Executing migration" id="create star table" 2024-09-21T13:50:09.856 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.656214207Z level=info msg="Migration successfully executed" id="create star table" duration=433.611µs 2024-09-21T13:50:09.856 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.656561675Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-09-21T13:50:09.856 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.657047789Z level=info msg="Migration successfully executed" id="add unique index star.user_id_dashboard_id" duration=486.337µs 2024-09-21T13:50:09.856 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.65738433Z level=info msg="Executing migration" id="create org table v1" 2024-09-21T13:50:09.856 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.657865172Z level=info msg="Migration successfully executed" id="create org table v1" duration=480.544µs 2024-09-21T13:50:09.856 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.658197758Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-09-21T13:50:09.857 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.658666192Z level=info msg="Migration successfully executed" id="create index UQE_org_name - v1" duration=468.781µs 2024-09-21T13:50:09.857 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.659014984Z level=info msg="Executing migration" id="create org_user table v1" 2024-09-21T13:50:09.857 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.659464029Z level=info msg="Migration successfully executed" id="create org_user table v1" duration=449.207µs 2024-09-21T13:50:09.857 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.659818626Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-09-21T13:50:09.857 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.66036835Z level=info msg="Migration successfully executed" id="create index IDX_org_user_org_id - v1" duration=547.806µs 2024-09-21T13:50:09.857 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.660765303Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-09-21T13:50:09.857 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.661291829Z level=info msg="Migration successfully executed" id="create index UQE_org_user_org_id_user_id - v1" duration=526.376µs 2024-09-21T13:50:09.858 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.661655246Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-09-21T13:50:09.858 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.662153902Z level=info msg="Migration successfully executed" id="create index IDX_org_user_user_id - v1" duration=498.39µs 2024-09-21T13:50:09.858 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.662510487Z level=info msg="Executing migration" id="Update org table charset" 2024-09-21T13:50:09.858 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.662525274Z level=info msg="Migration successfully executed" id="Update org table charset" duration=15.287µs 2024-09-21T13:50:09.858 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.663338194Z level=info msg="Executing migration" id="Update org_user table charset" 2024-09-21T13:50:09.858 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.66335722Z level=info msg="Migration successfully executed" id="Update org_user table charset" duration=19.956µs 2024-09-21T13:50:09.858 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.664307852Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-09-21T13:50:09.858 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.664521637Z level=info msg="Migration successfully executed" id="Migrate all Read Only Viewers to Viewers" duration=217.156µs 2024-09-21T13:50:09.858 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.665027642Z level=info msg="Executing migration" id="create dashboard table" 2024-09-21T13:50:09.858 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.665540162Z level=info msg="Migration successfully executed" id="create dashboard table" duration=512.068µs 2024-09-21T13:50:09.858 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.665913355Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-09-21T13:50:09.858 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.666434429Z level=info msg="Migration successfully executed" id="add index dashboard.account_id" duration=521.216µs 2024-09-21T13:50:09.859 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.666789629Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-09-21T13:50:09.859 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.667330257Z level=info msg="Migration successfully executed" id="add unique index dashboard_account_id_slug" duration=539.584µs 2024-09-21T13:50:09.859 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.667696237Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-09-21T13:50:09.859 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.668141635Z level=info msg="Migration successfully executed" id="create dashboard_tag table" duration=446.057µs 2024-09-21T13:50:09.859 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.668561769Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-09-21T13:50:09.859 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.669065345Z level=info msg="Migration successfully executed" id="add unique index dashboard_tag.dasboard_id_term" duration=503.215µs 2024-09-21T13:50:09.859 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.66941167Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-09-21T13:50:09.859 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.669968686Z level=info msg="Migration successfully executed" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" duration=556.781µs 2024-09-21T13:50:09.859 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.670327302Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-09-21T13:50:09.859 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.673718882Z level=info msg="Migration successfully executed" id="Rename table dashboard to dashboard_v1 - v1" duration=3.391107ms 2024-09-21T13:50:09.859 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.674100861Z level=info msg="Executing migration" id="create dashboard v2" 2024-09-21T13:50:09.860 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.674592038Z level=info msg="Migration successfully executed" id="create dashboard v2" duration=489.952µs 2024-09-21T13:50:09.860 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.676980333Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-09-21T13:50:09.860 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.677465944Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_org_id - v2" duration=484.934µs 2024-09-21T13:50:09.860 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.679876939Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-09-21T13:50:09.860 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.680397987Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_org_id_slug - v2" duration=521.202µs 2024-09-21T13:50:09.860 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.682727869Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-09-21T13:50:09.860 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.68300949Z level=info msg="Migration successfully executed" id="copy dashboard v1 to v2" duration=281.371µs 2024-09-21T13:50:09.860 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.683363525Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-09-21T13:50:09.860 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.683952351Z level=info msg="Migration successfully executed" id="drop table dashboard_v1" duration=588.517µs 2024-09-21T13:50:09.860 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.686328941Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-09-21T13:50:09.860 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.686404582Z level=info msg="Migration successfully executed" id="alter dashboard.data to mediumtext v1" duration=76.655µs 2024-09-21T13:50:09.860 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.689195324Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-09-21T13:50:09.860 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.690362784Z level=info msg="Migration successfully executed" id="Add column updated_by in dashboard - v2" duration=1.166916ms 2024-09-21T13:50:09.861 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.691970984Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-09-21T13:50:09.861 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.693065425Z level=info msg="Migration successfully executed" id="Add column created_by in dashboard - v2" duration=1.094037ms 2024-09-21T13:50:09.861 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.693420847Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-09-21T13:50:09.861 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.694498176Z level=info msg="Migration successfully executed" id="Add column gnetId in dashboard" duration=1.076914ms 2024-09-21T13:50:09.861 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.697417899Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-09-21T13:50:09.862 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.697935371Z level=info msg="Migration successfully executed" id="Add index for gnetId in dashboard" duration=517.283µs 2024-09-21T13:50:09.862 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.700263108Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-09-21T13:50:09.862 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.701397229Z level=info msg="Migration successfully executed" id="Add column plugin_id in dashboard" duration=1.133748ms 2024-09-21T13:50:09.862 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.70291269Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-09-21T13:50:09.862 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.70342702Z level=info msg="Migration successfully executed" id="Add index for plugin_id in dashboard" duration=513.913µs 2024-09-21T13:50:09.862 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.70379305Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-09-21T13:50:09.862 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.704295241Z level=info msg="Migration successfully executed" id="Add index for dashboard_id in dashboard_tag" duration=502.41µs 2024-09-21T13:50:09.862 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.70637996Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-09-21T13:50:09.862 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.706412368Z level=info msg="Migration successfully executed" id="Update dashboard table charset" duration=32.914µs 2024-09-21T13:50:09.863 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.709155454Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-09-21T13:50:09.863 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.709170698Z level=info msg="Migration successfully executed" id="Update dashboard_tag table charset" duration=15.737µs 2024-09-21T13:50:09.863 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.711962609Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-09-21T13:50:09.863 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.713300378Z level=info msg="Migration successfully executed" id="Add column folder_id in dashboard" duration=1.33772ms 2024-09-21T13:50:09.863 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.713690324Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-09-21T13:50:09.863 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.714888414Z level=info msg="Migration successfully executed" id="Add column isFolder in dashboard" duration=1.19775ms 2024-09-21T13:50:09.863 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.716515358Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-09-21T13:50:09.863 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.717741141Z level=info msg="Migration successfully executed" id="Add column has_acl in dashboard" duration=1.225551ms 2024-09-21T13:50:09.863 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.719070075Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-09-21T13:50:09.863 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.720274519Z level=info msg="Migration successfully executed" id="Add column uid in dashboard" duration=1.20429ms 2024-09-21T13:50:09.863 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.721808538Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-09-21T13:50:09.863 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.722005637Z level=info msg="Migration successfully executed" id="Update uid column values in dashboard" duration=195.826µs 2024-09-21T13:50:09.863 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.722454523Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-09-21T13:50:09.863 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.722953695Z level=info msg="Migration successfully executed" id="Add unique index dashboard_org_id_uid" duration=498.875µs 2024-09-21T13:50:09.864 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.725512117Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-09-21T13:50:09.864 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.72603155Z level=info msg="Migration successfully executed" id="Remove unique index org_id_slug" duration=517.885µs 2024-09-21T13:50:09.864 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.728163945Z level=info msg="Executing migration" id="Update dashboard title length" 2024-09-21T13:50:09.864 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.728202607Z level=info msg="Migration successfully executed" id="Update dashboard title length" duration=39.989µs 2024-09-21T13:50:09.864 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.730833081Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-09-21T13:50:09.864 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.73138155Z level=info msg="Migration successfully executed" id="Add unique index for dashboard_org_id_title_folder_id" duration=548.809µs 2024-09-21T13:50:09.864 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.731766582Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-09-21T13:50:09.864 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.732400124Z level=info msg="Migration successfully executed" id="create dashboard_provisioning" duration=633.159µs 2024-09-21T13:50:09.864 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.734536111Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-09-21T13:50:09.864 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.737723687Z level=info msg="Migration successfully executed" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" duration=3.188429ms 2024-09-21T13:50:09.864 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.738113858Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-09-21T13:50:09.864 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.738585066Z level=info msg="Migration successfully executed" id="create dashboard_provisioning v2" duration=471.742µs 2024-09-21T13:50:09.864 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.741026851Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-09-21T13:50:09.865 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.741534917Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_provisioning_dashboard_id - v2" duration=508.295µs 2024-09-21T13:50:09.865 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.741894745Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-09-21T13:50:09.865 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.742412236Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" duration=517.678µs 2024-09-21T13:50:09.865 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.744670014Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-09-21T13:50:09.865 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.744910377Z level=info msg="Migration successfully executed" id="copy dashboard_provisioning v1 to v2" duration=239.885µs 2024-09-21T13:50:09.865 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.747326205Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-09-21T13:50:09.865 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.747691688Z level=info msg="Migration successfully executed" id="drop dashboard_provisioning_tmp_qwerty" duration=365.204µs 2024-09-21T13:50:09.865 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.749907157Z level=info msg="Executing migration" id="Add check_sum column" 2024-09-21T13:50:09.865 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.751124967Z level=info msg="Migration successfully executed" id="Add check_sum column" duration=1.218383ms 2024-09-21T13:50:09.865 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.751481626Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-09-21T13:50:09.865 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.751967288Z level=info msg="Migration successfully executed" id="Add index for dashboard_title" duration=485.961µs 2024-09-21T13:50:09.865 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.754178788Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-09-21T13:50:09.865 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.754290182Z level=info msg="Migration successfully executed" id="delete tags for deleted dashboards" duration=111.442µs 2024-09-21T13:50:09.865 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.756868292Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-09-21T13:50:09.866 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.75704863Z level=info msg="Migration successfully executed" id="delete stars for deleted dashboards" duration=182.589µs 2024-09-21T13:50:09.866 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.759496873Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-09-21T13:50:09.866 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.760055465Z level=info msg="Migration successfully executed" id="Add index for dashboard_is_folder" duration=562.458µs 2024-09-21T13:50:09.866 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.760425909Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-09-21T13:50:09.866 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.76166207Z level=info msg="Migration successfully executed" id="Add isPublic for dashboard" duration=1.235966ms 2024-09-21T13:50:09.866 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.763271962Z level=info msg="Executing migration" id="create data_source table" 2024-09-21T13:50:09.866 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.763915368Z level=info msg="Migration successfully executed" id="create data_source table" duration=643.097µs 2024-09-21T13:50:09.866 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.765572525Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-09-21T13:50:09.866 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.766069218Z level=info msg="Migration successfully executed" id="add index data_source.account_id" duration=496.511µs 2024-09-21T13:50:09.866 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.76806582Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-09-21T13:50:09.866 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.76863768Z level=info msg="Migration successfully executed" id="add unique index data_source.account_id_name" duration=573.546µs 2024-09-21T13:50:09.866 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.768990639Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-09-21T13:50:09.866 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.769464527Z level=info msg="Migration successfully executed" id="drop index IDX_data_source_account_id - v1" duration=473.83µs 2024-09-21T13:50:09.866 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.771611222Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-09-21T13:50:09.867 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.77212664Z level=info msg="Migration successfully executed" id="drop index UQE_data_source_account_id_name - v1" duration=515.104µs 2024-09-21T13:50:09.867 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.772470516Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-09-21T13:50:09.867 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.775795998Z level=info msg="Migration successfully executed" id="Rename table data_source to data_source_v1 - v1" duration=3.325274ms 2024-09-21T13:50:09.867 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.776187156Z level=info msg="Executing migration" id="create data_source table v2" 2024-09-21T13:50:09.867 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.77676532Z level=info msg="Migration successfully executed" id="create data_source table v2" duration=578.07µs 2024-09-21T13:50:09.867 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.777125803Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-09-21T13:50:09.867 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.777644206Z level=info msg="Migration successfully executed" id="create index IDX_data_source_org_id - v2" duration=518.173µs 2024-09-21T13:50:09.867 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.777977475Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-09-21T13:50:09.867 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.778502433Z level=info msg="Migration successfully executed" id="create index UQE_data_source_org_id_name - v2" duration=524.446µs 2024-09-21T13:50:09.867 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.778862431Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-09-21T13:50:09.867 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.779232938Z level=info msg="Migration successfully executed" id="Drop old table data_source_v1 #2" duration=370.317µs 2024-09-21T13:50:09.867 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.779682516Z level=info msg="Executing migration" id="Add column with_credentials" 2024-09-21T13:50:09.867 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.78106798Z level=info msg="Migration successfully executed" id="Add column with_credentials" duration=1.385116ms 2024-09-21T13:50:09.867 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.781426391Z level=info msg="Executing migration" id="Add secure json data column" 2024-09-21T13:50:09.868 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.782795312Z level=info msg="Migration successfully executed" id="Add secure json data column" duration=1.36864ms 2024-09-21T13:50:09.868 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.783178819Z level=info msg="Executing migration" id="Update data_source table charset" 2024-09-21T13:50:09.868 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.783193885Z level=info msg="Migration successfully executed" id="Update data_source table charset" duration=15.75µs 2024-09-21T13:50:09.868 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.783947898Z level=info msg="Executing migration" id="Update initial version to 1" 2024-09-21T13:50:09.868 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.784095536Z level=info msg="Migration successfully executed" id="Update initial version to 1" duration=147.342µs 2024-09-21T13:50:09.868 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.784978327Z level=info msg="Executing migration" id="Add read_only data column" 2024-09-21T13:50:09.868 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.786412225Z level=info msg="Migration successfully executed" id="Add read_only data column" duration=1.43732ms 2024-09-21T13:50:09.868 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.786772395Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-09-21T13:50:09.868 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.786926792Z level=info msg="Migration successfully executed" id="Migrate logging ds to loki ds" duration=154.859µs 2024-09-21T13:50:09.868 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.787532947Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-09-21T13:50:09.868 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.787668352Z level=info msg="Migration successfully executed" id="Update json_data with nulls" duration=134.865µs 2024-09-21T13:50:09.868 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.788372081Z level=info msg="Executing migration" id="Add uid column" 2024-09-21T13:50:09.868 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.789778649Z level=info msg="Migration successfully executed" id="Add uid column" duration=1.406862ms 2024-09-21T13:50:09.868 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.790148162Z level=info msg="Executing migration" id="Update uid value" 2024-09-21T13:50:09.869 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.790282483Z level=info msg="Migration successfully executed" id="Update uid value" duration=134.779µs 2024-09-21T13:50:09.869 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.790674086Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-09-21T13:50:09.869 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.791255235Z level=info msg="Migration successfully executed" id="Add unique index datasource_org_id_uid" duration=581.349µs 2024-09-21T13:50:09.869 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.79160787Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-09-21T13:50:09.869 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.792238502Z level=info msg="Migration successfully executed" id="add unique index datasource_org_id_is_default" duration=630.061µs 2024-09-21T13:50:09.869 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.792613908Z level=info msg="Executing migration" id="create api_key table" 2024-09-21T13:50:09.869 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.793257886Z level=info msg="Migration successfully executed" id="create api_key table" duration=643.723µs 2024-09-21T13:50:09.869 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.793626906Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-09-21T13:50:09.869 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.794182763Z level=info msg="Migration successfully executed" id="add index api_key.account_id" duration=556.083µs 2024-09-21T13:50:09.869 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.794547611Z level=info msg="Executing migration" id="add index api_key.key" 2024-09-21T13:50:09.869 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.795099431Z level=info msg="Migration successfully executed" id="add index api_key.key" duration=552.053µs 2024-09-21T13:50:09.869 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.795525912Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-09-21T13:50:09.869 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.796069897Z level=info msg="Migration successfully executed" id="add index api_key.account_id_name" duration=543.675µs 2024-09-21T13:50:09.869 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.79645744Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-09-21T13:50:09.870 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.796957787Z level=info msg="Migration successfully executed" id="drop index IDX_api_key_account_id - v1" duration=499.112µs 2024-09-21T13:50:09.870 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.797328516Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-09-21T13:50:09.870 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.797820898Z level=info msg="Migration successfully executed" id="drop index UQE_api_key_key - v1" duration=492.208µs 2024-09-21T13:50:09.870 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.798204815Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-09-21T13:50:09.870 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.798694675Z level=info msg="Migration successfully executed" id="drop index UQE_api_key_account_id_name - v1" duration=489.766µs 2024-09-21T13:50:09.870 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.799076629Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-09-21T13:50:09.870 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.802785268Z level=info msg="Migration successfully executed" id="Rename table api_key to api_key_v1 - v1" duration=3.708073ms 2024-09-21T13:50:09.870 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.803179184Z level=info msg="Executing migration" id="create api_key table v2" 2024-09-21T13:50:09.870 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.803670676Z level=info msg="Migration successfully executed" id="create api_key table v2" duration=491.242µs 2024-09-21T13:50:09.870 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.804026746Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-09-21T13:50:09.870 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.8045171Z level=info msg="Migration successfully executed" id="create index IDX_api_key_org_id - v2" duration=490.238µs 2024-09-21T13:50:09.870 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.804892367Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-09-21T13:50:09.871 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.805382822Z level=info msg="Migration successfully executed" id="create index UQE_api_key_key - v2" duration=490.412µs 2024-09-21T13:50:09.871 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.805732357Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-09-21T13:50:09.871 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.806261056Z level=info msg="Migration successfully executed" id="create index UQE_api_key_org_id_name - v2" duration=529.299µs 2024-09-21T13:50:09.871 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.806622133Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-09-21T13:50:09.871 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.806875845Z level=info msg="Migration successfully executed" id="copy api_key v1 to v2" duration=253.357µs 2024-09-21T13:50:09.871 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.807422587Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-09-21T13:50:09.871 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.80780137Z level=info msg="Migration successfully executed" id="Drop old table api_key_v1" duration=378.613µs 2024-09-21T13:50:09.871 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.808314559Z level=info msg="Executing migration" id="Update api_key table charset" 2024-09-21T13:50:09.871 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.808346277Z level=info msg="Migration successfully executed" id="Update api_key table charset" duration=31.2µs 2024-09-21T13:50:09.871 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.809155425Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-09-21T13:50:09.871 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.810629875Z level=info msg="Migration successfully executed" id="Add expires to api_key table" duration=1.474102ms 2024-09-21T13:50:09.871 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.81097817Z level=info msg="Executing migration" id="Add service account foreign key" 2024-09-21T13:50:09.871 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.812502992Z level=info msg="Migration successfully executed" id="Add service account foreign key" duration=1.522899ms 2024-09-21T13:50:09.872 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.813602619Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-09-21T13:50:09.872 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.813777519Z level=info msg="Migration successfully executed" id="set service account foreign key to nil if 0" duration=175.432µs 2024-09-21T13:50:09.872 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.814345661Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-09-21T13:50:09.872 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.816124743Z level=info msg="Migration successfully executed" id="Add last_used_at to api_key table" duration=1.777624ms 2024-09-21T13:50:09.872 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.816558035Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-09-21T13:50:09.872 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.818170469Z level=info msg="Migration successfully executed" id="Add is_revoked column to api_key table" duration=1.6111ms 2024-09-21T13:50:09.872 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.819263797Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-09-21T13:50:09.872 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.819862246Z level=info msg="Migration successfully executed" id="create dashboard_snapshot table v4" duration=599.388µs 2024-09-21T13:50:09.872 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.82220729Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-09-21T13:50:09.872 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.822812093Z level=info msg="Migration successfully executed" id="drop table dashboard_snapshot_v4 #1" duration=607.478µs 2024-09-21T13:50:09.872 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.828896526Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-09-21T13:50:09.872 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.829555363Z level=info msg="Migration successfully executed" id="create dashboard_snapshot table v5 #2" duration=658.823µs 2024-09-21T13:50:09.872 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.829969278Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-09-21T13:50:09.872 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.830558221Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_snapshot_key - v5" duration=589.112µs 2024-09-21T13:50:09.873 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.830921752Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-09-21T13:50:09.873 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.831490294Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_snapshot_delete_key - v5" duration=568.306µs 2024-09-21T13:50:09.873 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.831977734Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-09-21T13:50:09.873 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.832538826Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_snapshot_user_id - v5" duration=561.801µs 2024-09-21T13:50:09.873 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.832954091Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-09-21T13:50:09.873 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.832993876Z level=info msg="Migration successfully executed" id="alter dashboard_snapshot to mediumtext v2" duration=40.238µs 2024-09-21T13:50:09.873 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.833714912Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-09-21T13:50:09.873 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.833731368Z level=info msg="Migration successfully executed" id="Update dashboard_snapshot table charset" duration=16.991µs 2024-09-21T13:50:09.873 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.834570086Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-09-21T13:50:09.873 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.836311012Z level=info msg="Migration successfully executed" id="Add column external_delete_url to dashboard_snapshots table" duration=1.740805ms 2024-09-21T13:50:09.873 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.836685199Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-09-21T13:50:09.873 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.838833545Z level=info msg="Migration successfully executed" id="Add encrypted dashboard json column" duration=2.148161ms 2024-09-21T13:50:09.873 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.839202331Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-09-21T13:50:09.873 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.839240321Z level=info msg="Migration successfully executed" id="Change dashboard_encrypted column to MEDIUMBLOB" duration=38.231µs 2024-09-21T13:50:09.874 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.840068551Z level=info msg="Executing migration" id="create quota table v1" 2024-09-21T13:50:09.874 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.840547229Z level=info msg="Migration successfully executed" id="create quota table v1" duration=477.989µs 2024-09-21T13:50:09.874 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.840961067Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-09-21T13:50:09.874 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.841497014Z level=info msg="Migration successfully executed" id="create index UQE_quota_org_id_user_id_target - v1" duration=536.067µs 2024-09-21T13:50:09.874 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.841848968Z level=info msg="Executing migration" id="Update quota table charset" 2024-09-21T13:50:09.874 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.8418657Z level=info msg="Migration successfully executed" id="Update quota table charset" duration=18.55µs 2024-09-21T13:50:09.930 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:50:09.951 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:09.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:09.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:50:10.053 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:50:10.077 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:10.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:10.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:50:10.084 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.842702049Z level=info msg="Executing migration" id="create plugin_setting table" 2024-09-21T13:50:10.084 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.843245609Z level=info msg="Migration successfully executed" id="create plugin_setting table" duration=543.29µs 2024-09-21T13:50:10.084 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.843595423Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-09-21T13:50:10.084 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.844114595Z level=info msg="Migration successfully executed" id="create index UQE_plugin_setting_org_id_plugin_id - v1" duration=518.856µs 2024-09-21T13:50:10.084 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.846323884Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-09-21T13:50:10.084 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.848052499Z level=info msg="Migration successfully executed" id="Add column plugin_version to plugin_settings" duration=1.728306ms 2024-09-21T13:50:10.084 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.849003049Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-09-21T13:50:10.084 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.849017599Z level=info msg="Migration successfully executed" id="Update plugin_setting table charset" duration=14.888µs 2024-09-21T13:50:10.084 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.851859838Z level=info msg="Executing migration" id="create session table" 2024-09-21T13:50:10.084 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.852440342Z level=info msg="Migration successfully executed" id="create session table" duration=580.689µs 2024-09-21T13:50:10.085 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.854731268Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-09-21T13:50:10.085 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.854797552Z level=info msg="Migration successfully executed" id="Drop old table playlist table" duration=65.214µs 2024-09-21T13:50:10.085 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.857485637Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-09-21T13:50:10.085 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.857566458Z level=info msg="Migration successfully executed" id="Drop old table playlist_item table" duration=81.453µs 2024-09-21T13:50:10.085 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.860376416Z level=info msg="Executing migration" id="create playlist table v2" 2024-09-21T13:50:10.085 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.860895624Z level=info msg="Migration successfully executed" id="create playlist table v2" duration=519.562µs 2024-09-21T13:50:10.085 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.863169001Z level=info msg="Executing migration" id="create playlist item table v2" 2024-09-21T13:50:10.085 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.86371591Z level=info msg="Migration successfully executed" id="create playlist item table v2" duration=547.035µs 2024-09-21T13:50:10.086 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.865933925Z level=info msg="Executing migration" id="Update playlist table charset" 2024-09-21T13:50:10.087 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.865952433Z level=info msg="Migration successfully executed" id="Update playlist table charset" duration=20.629µs 2024-09-21T13:50:10.087 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.868630133Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-09-21T13:50:10.087 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.868649316Z level=info msg="Migration successfully executed" id="Update playlist_item table charset" duration=20.165µs 2024-09-21T13:50:10.087 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.871492732Z level=info msg="Executing migration" id="Add playlist column created_at" 2024-09-21T13:50:10.087 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.873276522Z level=info msg="Migration successfully executed" id="Add playlist column created_at" duration=1.787115ms 2024-09-21T13:50:10.087 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.874210705Z level=info msg="Executing migration" id="Add playlist column updated_at" 2024-09-21T13:50:10.087 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.876162268Z level=info msg="Migration successfully executed" id="Add playlist column updated_at" duration=1.95484ms 2024-09-21T13:50:10.087 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.876828304Z level=info msg="Executing migration" id="drop preferences table v2" 2024-09-21T13:50:10.087 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.876918113Z level=info msg="Migration successfully executed" id="drop preferences table v2" duration=90.383µs 2024-09-21T13:50:10.087 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.879626553Z level=info msg="Executing migration" id="drop preferences table v3" 2024-09-21T13:50:10.088 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.879736003Z level=info msg="Migration successfully executed" id="drop preferences table v3" duration=110.931µs 2024-09-21T13:50:10.088 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.882393754Z level=info msg="Executing migration" id="create preferences table v3" 2024-09-21T13:50:10.088 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.882922128Z level=info msg="Migration successfully executed" id="create preferences table v3" duration=528.767µs 2024-09-21T13:50:10.088 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.885462746Z level=info msg="Executing migration" id="Update preferences table charset" 2024-09-21T13:50:10.088 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.885512385Z level=info msg="Migration successfully executed" id="Update preferences table charset" duration=52.883µs 2024-09-21T13:50:10.088 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.88822068Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-09-21T13:50:10.088 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.890080927Z level=info msg="Migration successfully executed" id="Add column team_id in preferences" duration=1.860456ms 2024-09-21T13:50:10.088 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.890920906Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-09-21T13:50:10.088 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.891039974Z level=info msg="Migration successfully executed" id="Update team_id column values in preferences" duration=119.548µs 2024-09-21T13:50:10.088 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.89375051Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-09-21T13:50:10.088 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.895663986Z level=info msg="Migration successfully executed" id="Add column week_start in preferences" duration=1.917044ms 2024-09-21T13:50:10.088 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.896505173Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-09-21T13:50:10.088 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.898363422Z level=info msg="Migration successfully executed" id="Add column preferences.json_data" duration=1.840121ms 2024-09-21T13:50:10.089 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.899074178Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-09-21T13:50:10.089 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.89920546Z level=info msg="Migration successfully executed" id="alter preferences.json_data to mediumtext v1" duration=135.187µs 2024-09-21T13:50:10.089 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.901656693Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-09-21T13:50:10.089 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.902319896Z level=info msg="Migration successfully executed" id="Add preferences index org_id" duration=663.246µs 2024-09-21T13:50:10.089 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.904162834Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-09-21T13:50:10.089 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.904699725Z level=info msg="Migration successfully executed" id="Add preferences index user_id" duration=536.566µs 2024-09-21T13:50:10.089 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.906735013Z level=info msg="Executing migration" id="create alert table v1" 2024-09-21T13:50:10.089 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.907480696Z level=info msg="Migration successfully executed" id="create alert table v1" duration=745.421µs 2024-09-21T13:50:10.089 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.909447369Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-09-21T13:50:10.089 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.910189975Z level=info msg="Migration successfully executed" id="add index alert org_id & id " duration=744.271µs 2024-09-21T13:50:10.089 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.912021441Z level=info msg="Executing migration" id="add index alert state" 2024-09-21T13:50:10.089 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.912567487Z level=info msg="Migration successfully executed" id="add index alert state" duration=545.894µs 2024-09-21T13:50:10.090 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.91478783Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-09-21T13:50:10.090 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.915380198Z level=info msg="Migration successfully executed" id="add index alert dashboard_id" duration=593.942µs 2024-09-21T13:50:10.090 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.917543838Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-09-21T13:50:10.090 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.918018575Z level=info msg="Migration successfully executed" id="Create alert_rule_tag table v1" duration=474.441µs 2024-09-21T13:50:10.090 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.920159537Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-09-21T13:50:10.090 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.920744597Z level=info msg="Migration successfully executed" id="Add unique index alert_rule_tag.alert_id_tag_id" duration=584.733µs 2024-09-21T13:50:10.090 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.922609033Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-09-21T13:50:10.090 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.923171489Z level=info msg="Migration successfully executed" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" duration=562.271µs 2024-09-21T13:50:10.090 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.923512198Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-09-21T13:50:10.090 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.929215323Z level=info msg="Migration successfully executed" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" duration=5.702835ms 2024-09-21T13:50:10.090 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.929584448Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-09-21T13:50:10.090 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.930059814Z level=info msg="Migration successfully executed" id="Create alert_rule_tag table v2" duration=475.202µs 2024-09-21T13:50:10.091 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.930403223Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-09-21T13:50:10.091 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.93093387Z level=info msg="Migration successfully executed" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" duration=531.499µs 2024-09-21T13:50:10.091 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.931276356Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-09-21T13:50:10.091 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.931475133Z level=info msg="Migration successfully executed" id="copy alert_rule_tag v1 to v2" duration=198.647µs 2024-09-21T13:50:10.091 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.931793047Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-09-21T13:50:10.091 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.932179134Z level=info msg="Migration successfully executed" id="drop table alert_rule_tag_v1" duration=385.853µs 2024-09-21T13:50:10.091 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.932534477Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-09-21T13:50:10.091 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.933025701Z level=info msg="Migration successfully executed" id="create alert_notification table v1" duration=490.957µs 2024-09-21T13:50:10.091 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.933364803Z level=info msg="Executing migration" id="Add column is_default" 2024-09-21T13:50:10.091 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.935551728Z level=info msg="Migration successfully executed" id="Add column is_default" duration=2.186172ms 2024-09-21T13:50:10.091 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.935928493Z level=info msg="Executing migration" id="Add column frequency" 2024-09-21T13:50:10.091 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.938160079Z level=info msg="Migration successfully executed" id="Add column frequency" duration=2.231246ms 2024-09-21T13:50:10.091 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.938515218Z level=info msg="Executing migration" id="Add column send_reminder" 2024-09-21T13:50:10.091 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.940826601Z level=info msg="Migration successfully executed" id="Add column send_reminder" duration=2.310974ms 2024-09-21T13:50:10.091 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.941188738Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-09-21T13:50:10.092 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.94325877Z level=info msg="Migration successfully executed" id="Add column disable_resolve_message" duration=2.069746ms 2024-09-21T13:50:10.092 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.943695329Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-09-21T13:50:10.092 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.944252491Z level=info msg="Migration successfully executed" id="add index alert_notification org_id & name" duration=556.978µs 2024-09-21T13:50:10.092 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.944612324Z level=info msg="Executing migration" id="Update alert table charset" 2024-09-21T13:50:10.092 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.944632729Z level=info msg="Migration successfully executed" id="Update alert table charset" duration=20.306µs 2024-09-21T13:50:10.092 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.945051469Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-09-21T13:50:10.092 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.945069462Z level=info msg="Migration successfully executed" id="Update alert_notification table charset" duration=17.996µs 2024-09-21T13:50:10.092 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.945462337Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-09-21T13:50:10.092 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.945958724Z level=info msg="Migration successfully executed" id="create notification_journal table v1" duration=496.054µs 2024-09-21T13:50:10.092 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.946324004Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-09-21T13:50:10.092 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.946858914Z level=info msg="Migration successfully executed" id="add index notification_journal org_id & alert_id & notifier_id" duration=533.443µs 2024-09-21T13:50:10.092 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.94723434Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-09-21T13:50:10.092 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.947788192Z level=info msg="Migration successfully executed" id="drop alert_notification_journal" duration=553.906µs 2024-09-21T13:50:10.092 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.948179172Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-09-21T13:50:10.092 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.948705697Z level=info msg="Migration successfully executed" id="create alert_notification_state table v1" duration=526.273µs 2024-09-21T13:50:10.093 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.949061642Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-09-21T13:50:10.093 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.94960196Z level=info msg="Migration successfully executed" id="add index alert_notification_state org_id & alert_id & notifier_id" duration=540.002µs 2024-09-21T13:50:10.093 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.949958735Z level=info msg="Executing migration" id="Add for to alert table" 2024-09-21T13:50:10.093 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.95210134Z level=info msg="Migration successfully executed" id="Add for to alert table" duration=2.142058ms 2024-09-21T13:50:10.093 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.952468418Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-09-21T13:50:10.093 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.954802106Z level=info msg="Migration successfully executed" id="Add column uid in alert_notification" duration=2.333405ms 2024-09-21T13:50:10.093 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.955183441Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-09-21T13:50:10.093 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.955304407Z level=info msg="Migration successfully executed" id="Update uid column values in alert_notification" duration=122.977µs 2024-09-21T13:50:10.093 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.956036945Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-09-21T13:50:10.093 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.956560579Z level=info msg="Migration successfully executed" id="Add unique index alert_notification_org_id_uid" duration=523.936µs 2024-09-21T13:50:10.093 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.956916348Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-09-21T13:50:10.093 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.957444405Z level=info msg="Migration successfully executed" id="Remove unique index org_id_name" duration=528.41µs 2024-09-21T13:50:10.093 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.957782003Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-09-21T13:50:10.093 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.959934479Z level=info msg="Migration successfully executed" id="Add column secure_settings in alert_notification" duration=2.152723ms 2024-09-21T13:50:10.094 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.960279783Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-09-21T13:50:10.094 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.960318502Z level=info msg="Migration successfully executed" id="alter alert.settings to mediumtext" duration=39.187µs 2024-09-21T13:50:10.094 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.961297048Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-09-21T13:50:10.094 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.961977114Z level=info msg="Migration successfully executed" id="Add non-unique index alert_notification_state_alert_id" duration=683.099µs 2024-09-21T13:50:10.094 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.962312999Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-09-21T13:50:10.094 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.96290742Z level=info msg="Migration successfully executed" id="Add non-unique index alert_rule_tag_alert_id" duration=594.096µs 2024-09-21T13:50:10.094 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.963269773Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-09-21T13:50:10.094 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.963328572Z level=info msg="Migration successfully executed" id="Drop old annotation table v4" duration=57.402µs 2024-09-21T13:50:10.094 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.96413374Z level=info msg="Executing migration" id="create annotation table v5" 2024-09-21T13:50:10.094 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.96472676Z level=info msg="Migration successfully executed" id="create annotation table v5" duration=593.22µs 2024-09-21T13:50:10.094 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.965106095Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-09-21T13:50:10.094 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.965661625Z level=info msg="Migration successfully executed" id="add index annotation 0 v3" duration=555.765µs 2024-09-21T13:50:10.094 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.966007281Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-09-21T13:50:10.094 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.966526766Z level=info msg="Migration successfully executed" id="add index annotation 1 v3" duration=519.217µs 2024-09-21T13:50:10.094 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.966902295Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-09-21T13:50:10.095 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.967437353Z level=info msg="Migration successfully executed" id="add index annotation 2 v3" duration=534.224µs 2024-09-21T13:50:10.095 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.967775795Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-09-21T13:50:10.095 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.968395145Z level=info msg="Migration successfully executed" id="add index annotation 3 v3" duration=619.525µs 2024-09-21T13:50:10.095 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.96875734Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-09-21T13:50:10.095 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.969387952Z level=info msg="Migration successfully executed" id="add index annotation 4 v3" duration=629.684µs 2024-09-21T13:50:10.095 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.969744363Z level=info msg="Executing migration" id="Update annotation table charset" 2024-09-21T13:50:10.095 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.969759389Z level=info msg="Migration successfully executed" id="Update annotation table charset" duration=15.636µs 2024-09-21T13:50:10.095 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.970612584Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-09-21T13:50:10.095 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.973033025Z level=info msg="Migration successfully executed" id="Add column region_id to annotation table" duration=2.42022ms 2024-09-21T13:50:10.095 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.973435354Z level=info msg="Executing migration" id="Drop category_id index" 2024-09-21T13:50:10.095 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.973974283Z level=info msg="Migration successfully executed" id="Drop category_id index" duration=538.73µs 2024-09-21T13:50:10.095 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.974311482Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-09-21T13:50:10.095 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.976572781Z level=info msg="Migration successfully executed" id="Add column tags to annotation table" duration=2.261025ms 2024-09-21T13:50:10.095 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.977077341Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-09-21T13:50:10.095 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.977519785Z level=info msg="Migration successfully executed" id="Create annotation_tag table v2" duration=442.249µs 2024-09-21T13:50:10.096 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.979926405Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-09-21T13:50:10.096 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.980551899Z level=info msg="Migration successfully executed" id="Add unique index annotation_tag.annotation_id_tag_id" duration=626µs 2024-09-21T13:50:10.096 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.982947499Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-09-21T13:50:10.096 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.983565072Z level=info msg="Migration successfully executed" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" duration=620.181µs 2024-09-21T13:50:10.096 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.985688712Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-09-21T13:50:10.096 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.992178709Z level=info msg="Migration successfully executed" id="Rename table annotation_tag to annotation_tag_v2 - v2" duration=6.492029ms 2024-09-21T13:50:10.096 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.992539082Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-09-21T13:50:10.096 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.993012613Z level=info msg="Migration successfully executed" id="Create annotation_tag table v3" duration=473.292µs 2024-09-21T13:50:10.096 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.995335028Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-09-21T13:50:10.096 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.995906978Z level=info msg="Migration successfully executed" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" duration=574.801µs 2024-09-21T13:50:10.096 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.998194654Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-09-21T13:50:10.096 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:09.998417434Z level=info msg="Migration successfully executed" id="copy annotation_tag v2 to v3" duration=223.717µs 2024-09-21T13:50:10.096 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.00089359Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-09-21T13:50:10.096 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.001283684Z level=info msg="Migration successfully executed" id="drop table annotation_tag_v2" duration=390.57µs 2024-09-21T13:50:10.096 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.003878955Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-09-21T13:50:10.097 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.003994963Z level=info msg="Migration successfully executed" id="Update alert annotations and set TEXT to empty" duration=116.653µs 2024-09-21T13:50:10.097 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.006827874Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-09-21T13:50:10.097 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.009148969Z level=info msg="Migration successfully executed" id="Add created time to annotation table" duration=2.321242ms 2024-09-21T13:50:10.097 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.00951013Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-09-21T13:50:10.097 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.011784461Z level=info msg="Migration successfully executed" id="Add updated time to annotation table" duration=2.275617ms 2024-09-21T13:50:10.097 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.012523003Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-09-21T13:50:10.097 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.013234007Z level=info msg="Migration successfully executed" id="Add index for created in annotation table" duration=712.725µs 2024-09-21T13:50:10.098 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.015242576Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-09-21T13:50:10.098 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.015782222Z level=info msg="Migration successfully executed" id="Add index for updated in annotation table" duration=539.438µs 2024-09-21T13:50:10.098 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.018011464Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-09-21T13:50:10.098 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.018155914Z level=info msg="Migration successfully executed" id="Convert existing annotations from seconds to milliseconds" duration=142.982µs 2024-09-21T13:50:10.098 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.020889765Z level=info msg="Executing migration" id="Add epoch_end column" 2024-09-21T13:50:10.098 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.023226981Z level=info msg="Migration successfully executed" id="Add epoch_end column" duration=2.337975ms 2024-09-21T13:50:10.098 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.023594632Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-09-21T13:50:10.098 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.024165375Z level=info msg="Migration successfully executed" id="Add index for epoch_end" duration=570.524µs 2024-09-21T13:50:10.098 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.026367572Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-09-21T13:50:10.098 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.02649082Z level=info msg="Migration successfully executed" id="Make epoch_end the same as epoch" duration=123.289µs 2024-09-21T13:50:10.098 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.029198044Z level=info msg="Executing migration" id="Move region to single row" 2024-09-21T13:50:10.098 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.029434966Z level=info msg="Migration successfully executed" id="Move region to single row" duration=236.843µs 2024-09-21T13:50:10.098 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.031936297Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-09-21T13:50:10.098 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.032463802Z level=info msg="Migration successfully executed" id="Remove index org_id_epoch from annotation table" duration=527.358µs 2024-09-21T13:50:10.099 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.034539475Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-09-21T13:50:10.099 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.035132581Z level=info msg="Migration successfully executed" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" duration=595.081µs 2024-09-21T13:50:10.099 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.037265448Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-09-21T13:50:10.099 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.037887631Z level=info msg="Migration successfully executed" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" duration=623.686µs 2024-09-21T13:50:10.099 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.039891222Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-09-21T13:50:10.099 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.040547991Z level=info msg="Migration successfully executed" id="Add index for org_id_epoch_end_epoch on annotation table" duration=658.243µs 2024-09-21T13:50:10.099 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.042428167Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-09-21T13:50:10.099 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.04297481Z level=info msg="Migration successfully executed" id="Remove index org_id_epoch_epoch_end from annotation table" duration=546.519µs 2024-09-21T13:50:10.099 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.045239924Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-09-21T13:50:10.099 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.045823917Z level=info msg="Migration successfully executed" id="Add index for alert_id on annotation table" duration=584.692µs 2024-09-21T13:50:10.099 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.04797258Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-09-21T13:50:10.099 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.048034279Z level=info msg="Migration successfully executed" id="Increase tags column to length 4096" duration=64.23µs 2024-09-21T13:50:10.099 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.048454969Z level=info msg="Executing migration" id="create test_data table" 2024-09-21T13:50:10.099 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.048987686Z level=info msg="Migration successfully executed" id="create test_data table" duration=533.002µs 2024-09-21T13:50:10.100 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.050521696Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-09-21T13:50:10.100 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.051078534Z level=info msg="Migration successfully executed" id="create dashboard_version table v1" duration=556.603µs 2024-09-21T13:50:10.100 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.053267218Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-09-21T13:50:10.100 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.053984714Z level=info msg="Migration successfully executed" id="add index dashboard_version.dashboard_id" duration=716.845µs 2024-09-21T13:50:10.100 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.05590536Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-09-21T13:50:10.100 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.056467508Z level=info msg="Migration successfully executed" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" duration=562.022µs 2024-09-21T13:50:10.100 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.058522731Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-09-21T13:50:10.100 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.058713912Z level=info msg="Migration successfully executed" id="Set dashboard version to 1 where 0" duration=190.601µs 2024-09-21T13:50:10.100 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.061287414Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-09-21T13:50:10.100 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.061538843Z level=info msg="Migration successfully executed" id="save existing dashboard data in dashboard_version table v1" duration=250.417µs 2024-09-21T13:50:10.100 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.063938555Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-09-21T13:50:10.100 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.064057283Z level=info msg="Migration successfully executed" id="alter dashboard_version.data to mediumtext v1" duration=121.311µs 2024-09-21T13:50:10.100 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.066488299Z level=info msg="Executing migration" id="create team table" 2024-09-21T13:50:10.100 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.066989599Z level=info msg="Migration successfully executed" id="create team table" duration=501.599µs 2024-09-21T13:50:10.101 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.069185758Z level=info msg="Executing migration" id="add index team.org_id" 2024-09-21T13:50:10.101 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.069826702Z level=info msg="Migration successfully executed" id="add index team.org_id" duration=641.047µs 2024-09-21T13:50:10.101 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.072053337Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-09-21T13:50:10.101 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.07260058Z level=info msg="Migration successfully executed" id="add unique index team_org_id_name" duration=547.374µs 2024-09-21T13:50:10.101 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.072948582Z level=info msg="Executing migration" id="Add column uid in team" 2024-09-21T13:50:10.101 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.075517461Z level=info msg="Migration successfully executed" id="Add column uid in team" duration=2.568691ms 2024-09-21T13:50:10.101 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.075886638Z level=info msg="Executing migration" id="Update uid column values in team" 2024-09-21T13:50:10.101 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.075999612Z level=info msg="Migration successfully executed" id="Update uid column values in team" duration=112.947µs 2024-09-21T13:50:10.101 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.07675354Z level=info msg="Executing migration" id="Add unique index team_org_id_uid" 2024-09-21T13:50:10.101 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.077330299Z level=info msg="Migration successfully executed" id="Add unique index team_org_id_uid" duration=578.231µs 2024-09-21T13:50:10.101 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.077682227Z level=info msg="Executing migration" id="create team member table" 2024-09-21T13:50:10.101 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.078193302Z level=info msg="Migration successfully executed" id="create team member table" duration=510.903µs 2024-09-21T13:50:10.101 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.078555296Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-09-21T13:50:10.101 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.079115494Z level=info msg="Migration successfully executed" id="add index team_member.org_id" duration=559.98µs 2024-09-21T13:50:10.101 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.07947507Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-09-21T13:50:10.101 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.080062562Z level=info msg="Migration successfully executed" id="add unique index team_member_org_id_team_id_user_id" duration=587.677µs 2024-09-21T13:50:10.102 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.080405968Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-09-21T13:50:10.102 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.080963826Z level=info msg="Migration successfully executed" id="add index team_member.team_id" duration=558.122µs 2024-09-21T13:50:10.102 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.081307134Z level=info msg="Executing migration" id="Add column email to team table" 2024-09-21T13:50:10.102 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.08404592Z level=info msg="Migration successfully executed" id="Add column email to team table" duration=2.738414ms 2024-09-21T13:50:10.102 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.084420597Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-09-21T13:50:10.102 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.087582283Z level=info msg="Migration successfully executed" id="Add column external to team_member table" duration=3.161936ms 2024-09-21T13:50:10.102 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.087966817Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-09-21T13:50:10.179 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:50:10.202 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:10.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:10.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:50:10.304 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:10.328 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:10.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:10.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:50:10.335 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.090744213Z level=info msg="Migration successfully executed" id="Add column permission to team_member table" duration=2.777626ms 2024-09-21T13:50:10.335 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.091101375Z level=info msg="Executing migration" id="create dashboard acl table" 2024-09-21T13:50:10.335 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.091726657Z level=info msg="Migration successfully executed" id="create dashboard acl table" duration=624.95µs 2024-09-21T13:50:10.335 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.092103246Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-09-21T13:50:10.336 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.092672797Z level=info msg="Migration successfully executed" id="add index dashboard_acl_dashboard_id" duration=569.394µs 2024-09-21T13:50:10.336 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.093042312Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-09-21T13:50:10.336 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.093671716Z level=info msg="Migration successfully executed" id="add unique index dashboard_acl_dashboard_id_user_id" duration=628.866µs 2024-09-21T13:50:10.336 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.094043745Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-09-21T13:50:10.336 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.094632018Z level=info msg="Migration successfully executed" id="add unique index dashboard_acl_dashboard_id_team_id" duration=587.798µs 2024-09-21T13:50:10.336 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.09497471Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-09-21T13:50:10.336 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.095532853Z level=info msg="Migration successfully executed" id="add index dashboard_acl_user_id" duration=557.82µs 2024-09-21T13:50:10.336 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.095889546Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-09-21T13:50:10.336 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.096551907Z level=info msg="Migration successfully executed" id="add index dashboard_acl_team_id" duration=662.112µs 2024-09-21T13:50:10.336 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.096919093Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-09-21T13:50:10.336 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.09751252Z level=info msg="Migration successfully executed" id="add index dashboard_acl_org_id_role" duration=593.761µs 2024-09-21T13:50:10.337 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.09789009Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-09-21T13:50:10.337 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.098495098Z level=info msg="Migration successfully executed" id="add index dashboard_permission" duration=606.547µs 2024-09-21T13:50:10.337 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.098859475Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-09-21T13:50:10.337 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.099195279Z level=info msg="Migration successfully executed" id="save default acl rules in dashboard_acl table" duration=336.222µs 2024-09-21T13:50:10.338 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.099693234Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-09-21T13:50:10.338 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.099856887Z level=info msg="Migration successfully executed" id="delete acl rules for deleted dashboards and folders" duration=164.122µs 2024-09-21T13:50:10.338 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.100595679Z level=info msg="Executing migration" id="create tag table" 2024-09-21T13:50:10.338 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.101130293Z level=info msg="Migration successfully executed" id="create tag table" duration=534.478µs 2024-09-21T13:50:10.338 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.101505006Z level=info msg="Executing migration" id="add index tag.key_value" 2024-09-21T13:50:10.338 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.102090936Z level=info msg="Migration successfully executed" id="add index tag.key_value" duration=584.423µs 2024-09-21T13:50:10.338 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.102476729Z level=info msg="Executing migration" id="create login attempt table" 2024-09-21T13:50:10.338 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.102972902Z level=info msg="Migration successfully executed" id="create login attempt table" duration=495.85µs 2024-09-21T13:50:10.338 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.103329115Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-09-21T13:50:10.338 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.103906916Z level=info msg="Migration successfully executed" id="add index login_attempt.username" duration=577.5µs 2024-09-21T13:50:10.338 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.104261805Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-09-21T13:50:10.338 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.104828187Z level=info msg="Migration successfully executed" id="drop index IDX_login_attempt_username - v1" duration=566.791µs 2024-09-21T13:50:10.338 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.105168202Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-09-21T13:50:10.339 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.113117107Z level=info msg="Migration successfully executed" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" duration=7.948508ms 2024-09-21T13:50:10.339 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.113480508Z level=info msg="Executing migration" id="create login_attempt v2" 2024-09-21T13:50:10.339 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.113958471Z level=info msg="Migration successfully executed" id="create login_attempt v2" duration=477.619µs 2024-09-21T13:50:10.339 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.114337526Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-09-21T13:50:10.339 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.115040903Z level=info msg="Migration successfully executed" id="create index IDX_login_attempt_username - v2" duration=702.34µs 2024-09-21T13:50:10.339 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.115488555Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-09-21T13:50:10.339 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.115756516Z level=info msg="Migration successfully executed" id="copy login_attempt v1 to v2" duration=267.727µs 2024-09-21T13:50:10.339 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.116359855Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-09-21T13:50:10.339 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.116948065Z level=info msg="Migration successfully executed" id="drop login_attempt_tmp_qwerty" duration=587.908µs 2024-09-21T13:50:10.339 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.117389863Z level=info msg="Executing migration" id="create user auth table" 2024-09-21T13:50:10.339 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.118094282Z level=info msg="Migration successfully executed" id="create user auth table" duration=703.99µs 2024-09-21T13:50:10.339 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.118555571Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-09-21T13:50:10.339 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.119368551Z level=info msg="Migration successfully executed" id="create index IDX_user_auth_auth_module_auth_id - v1" duration=812.562µs 2024-09-21T13:50:10.339 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.119803122Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-09-21T13:50:10.340 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.119865752Z level=info msg="Migration successfully executed" id="alter user_auth.auth_id to length 190" duration=63.013µs 2024-09-21T13:50:10.340 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.120413814Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-09-21T13:50:10.340 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.123448498Z level=info msg="Migration successfully executed" id="Add OAuth access token to user_auth" duration=3.034919ms 2024-09-21T13:50:10.340 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.123823132Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-09-21T13:50:10.340 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.126713621Z level=info msg="Migration successfully executed" id="Add OAuth refresh token to user_auth" duration=2.891147ms 2024-09-21T13:50:10.340 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.12709072Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-09-21T13:50:10.340 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.129973113Z level=info msg="Migration successfully executed" id="Add OAuth token type to user_auth" duration=2.883056ms 2024-09-21T13:50:10.340 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.130330812Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-09-21T13:50:10.340 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.133262772Z level=info msg="Migration successfully executed" id="Add OAuth expiry to user_auth" duration=2.93164ms 2024-09-21T13:50:10.340 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.133636147Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-09-21T13:50:10.340 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.134234314Z level=info msg="Migration successfully executed" id="Add index to user_id column in user_auth" duration=598.038µs 2024-09-21T13:50:10.340 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.134609698Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-09-21T13:50:10.340 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.137537192Z level=info msg="Migration successfully executed" id="Add OAuth ID token to user_auth" duration=2.927148ms 2024-09-21T13:50:10.340 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.137921494Z level=info msg="Executing migration" id="create server_lock table" 2024-09-21T13:50:10.341 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.138454058Z level=info msg="Migration successfully executed" id="create server_lock table" duration=532.392µs 2024-09-21T13:50:10.341 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.138826325Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-09-21T13:50:10.341 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.139432462Z level=info msg="Migration successfully executed" id="add index server_lock.operation_uid" duration=605.914µs 2024-09-21T13:50:10.341 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.139808667Z level=info msg="Executing migration" id="create user auth token table" 2024-09-21T13:50:10.341 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.140387444Z level=info msg="Migration successfully executed" id="create user auth token table" duration=578.553µs 2024-09-21T13:50:10.341 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.140756234Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-09-21T13:50:10.341 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.141357276Z level=info msg="Migration successfully executed" id="add unique index user_auth_token.auth_token" duration=601.662µs 2024-09-21T13:50:10.341 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.141729241Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-09-21T13:50:10.341 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.142334371Z level=info msg="Migration successfully executed" id="add unique index user_auth_token.prev_auth_token" duration=608.38µs 2024-09-21T13:50:10.341 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.142731074Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-09-21T13:50:10.341 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.143393245Z level=info msg="Migration successfully executed" id="add index user_auth_token.user_id" duration=662.044µs 2024-09-21T13:50:10.341 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.143791193Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-09-21T13:50:10.341 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.146915862Z level=info msg="Migration successfully executed" id="Add revoked_at to the user auth token" duration=3.124314ms 2024-09-21T13:50:10.341 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.147290488Z level=info msg="Executing migration" id="add index user_auth_token.revoked_at" 2024-09-21T13:50:10.342 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.147955109Z level=info msg="Migration successfully executed" id="add index user_auth_token.revoked_at" duration=664.231µs 2024-09-21T13:50:10.342 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.148323352Z level=info msg="Executing migration" id="create cache_data table" 2024-09-21T13:50:10.342 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.148923896Z level=info msg="Migration successfully executed" id="create cache_data table" duration=600.3µs 2024-09-21T13:50:10.342 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.149288319Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-09-21T13:50:10.342 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.149887354Z level=info msg="Migration successfully executed" id="add unique index cache_data.cache_key" duration=597.317µs 2024-09-21T13:50:10.342 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.150245418Z level=info msg="Executing migration" id="create short_url table v1" 2024-09-21T13:50:10.342 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.150826147Z level=info msg="Migration successfully executed" id="create short_url table v1" duration=580.492µs 2024-09-21T13:50:10.342 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.1512612Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-09-21T13:50:10.342 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.151924222Z level=info msg="Migration successfully executed" id="add index short_url.org_id-uid" duration=662.953µs 2024-09-21T13:50:10.342 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.15248936Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-09-21T13:50:10.342 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.152604221Z level=info msg="Migration successfully executed" id="alter table short_url alter column created_by type to bigint" duration=114.655µs 2024-09-21T13:50:10.342 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.153093816Z level=info msg="Executing migration" id="delete alert_definition table" 2024-09-21T13:50:10.342 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.153166563Z level=info msg="Migration successfully executed" id="delete alert_definition table" duration=72.312µs 2024-09-21T13:50:10.342 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.153588914Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-09-21T13:50:10.343 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.154178824Z level=info msg="Migration successfully executed" id="recreate alert_definition table" duration=589.777µs 2024-09-21T13:50:10.343 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.154559284Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-09-21T13:50:10.343 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.155200628Z level=info msg="Migration successfully executed" id="add index in alert_definition on org_id and title columns" duration=641.157µs 2024-09-21T13:50:10.343 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.157507418Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-09-21T13:50:10.343 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.158182163Z level=info msg="Migration successfully executed" id="add index in alert_definition on org_id and uid columns" duration=672.913µs 2024-09-21T13:50:10.343 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.160054218Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-09-21T13:50:10.343 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.160097604Z level=info msg="Migration successfully executed" id="alter alert_definition table data column to mediumtext in mysql" duration=43.057µs 2024-09-21T13:50:10.343 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.162799218Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-09-21T13:50:10.343 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.163442792Z level=info msg="Migration successfully executed" id="drop index in alert_definition on org_id and title columns" duration=645.69µs 2024-09-21T13:50:10.343 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.163786289Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-09-21T13:50:10.343 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.164375791Z level=info msg="Migration successfully executed" id="drop index in alert_definition on org_id and uid columns" duration=589.267µs 2024-09-21T13:50:10.343 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.166667561Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-09-21T13:50:10.343 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.167342973Z level=info msg="Migration successfully executed" id="add unique index in alert_definition on org_id and title columns" duration=678.544µs 2024-09-21T13:50:10.343 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.169407553Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-09-21T13:50:10.344 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.170194373Z level=info msg="Migration successfully executed" id="add unique index in alert_definition on org_id and uid columns" duration=788.584µs 2024-09-21T13:50:10.344 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.171822537Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-09-21T13:50:10.344 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.175008319Z level=info msg="Migration successfully executed" id="Add column paused in alert_definition" duration=3.185431ms 2024-09-21T13:50:10.345 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.175347428Z level=info msg="Executing migration" id="drop alert_definition table" 2024-09-21T13:50:10.345 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.176007566Z level=info msg="Migration successfully executed" id="drop alert_definition table" duration=659.702µs 2024-09-21T13:50:10.345 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.178029903Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-09-21T13:50:10.345 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.178092107Z level=info msg="Migration successfully executed" id="delete alert_definition_version table" duration=61.825µs 2024-09-21T13:50:10.345 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.180765929Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-09-21T13:50:10.345 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.18135994Z level=info msg="Migration successfully executed" id="recreate alert_definition_version table" duration=594.703µs 2024-09-21T13:50:10.345 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.183338721Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-09-21T13:50:10.345 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.183970347Z level=info msg="Migration successfully executed" id="add index in alert_definition_version table on alert_definition_id and version columns" duration=631.836µs 2024-09-21T13:50:10.345 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.184315592Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-09-21T13:50:10.345 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.184942231Z level=info msg="Migration successfully executed" id="add index in alert_definition_version table on alert_definition_uid and version columns" duration=624.705µs 2024-09-21T13:50:10.345 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.187001155Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-09-21T13:50:10.345 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.187044303Z level=info msg="Migration successfully executed" id="alter alert_definition_version table data column to mediumtext in mysql" duration=44.039µs 2024-09-21T13:50:10.346 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.189912522Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-09-21T13:50:10.346 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.190577309Z level=info msg="Migration successfully executed" id="drop alert_definition_version table" duration=683.733µs 2024-09-21T13:50:10.346 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.192654672Z level=info msg="Executing migration" id="create alert_instance table" 2024-09-21T13:50:10.346 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.193289741Z level=info msg="Migration successfully executed" id="create alert_instance table" duration=635.108µs 2024-09-21T13:50:10.346 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.193644897Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-09-21T13:50:10.346 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.194278469Z level=info msg="Migration successfully executed" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" duration=633.82µs 2024-09-21T13:50:10.346 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.19661962Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-09-21T13:50:10.346 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.197310764Z level=info msg="Migration successfully executed" id="add index in alert_instance table on def_org_id, current_state columns" duration=690.863µs 2024-09-21T13:50:10.346 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.199220377Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-09-21T13:50:10.346 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.202591466Z level=info msg="Migration successfully executed" id="add column current_state_end to alert_instance" duration=3.370872ms 2024-09-21T13:50:10.346 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.202956627Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-09-21T13:50:10.346 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.203534069Z level=info msg="Migration successfully executed" id="remove index def_org_id, def_uid, current_state on alert_instance" duration=577.338µs 2024-09-21T13:50:10.346 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.203893358Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-09-21T13:50:10.347 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.204476045Z level=info msg="Migration successfully executed" id="remove index def_org_id, current_state on alert_instance" duration=583.102µs 2024-09-21T13:50:10.347 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.204832029Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-09-21T13:50:10.347 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.222076226Z level=info msg="Migration successfully executed" id="rename def_org_id to rule_org_id in alert_instance" duration=17.243897ms 2024-09-21T13:50:10.347 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.222460243Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-09-21T13:50:10.347 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.238451893Z level=info msg="Migration successfully executed" id="rename def_uid to rule_uid in alert_instance" duration=15.990798ms 2024-09-21T13:50:10.347 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.238827649Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-09-21T13:50:10.347 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.239421858Z level=info msg="Migration successfully executed" id="add index rule_org_id, rule_uid, current_state on alert_instance" duration=593.909µs 2024-09-21T13:50:10.347 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.239865897Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-09-21T13:50:10.347 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.240461347Z level=info msg="Migration successfully executed" id="add index rule_org_id, current_state on alert_instance" duration=595.15µs 2024-09-21T13:50:10.347 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.240812147Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-09-21T13:50:10.347 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.244022011Z level=info msg="Migration successfully executed" id="add current_reason column related to current_state" duration=3.210911ms 2024-09-21T13:50:10.347 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.244355605Z level=info msg="Executing migration" id="add result_fingerprint column to alert_instance" 2024-09-21T13:50:10.347 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.247529383Z level=info msg="Migration successfully executed" id="add result_fingerprint column to alert_instance" duration=3.173587ms 2024-09-21T13:50:10.347 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.247900273Z level=info msg="Executing migration" id="create alert_rule table" 2024-09-21T13:50:10.347 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.24851239Z level=info msg="Migration successfully executed" id="create alert_rule table" duration=612µs 2024-09-21T13:50:10.348 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.248900612Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-09-21T13:50:10.348 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.249515882Z level=info msg="Migration successfully executed" id="add index in alert_rule on org_id and title columns" duration=615.011µs 2024-09-21T13:50:10.348 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.24990653Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-09-21T13:50:10.348 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.250519904Z level=info msg="Migration successfully executed" id="add index in alert_rule on org_id and uid columns" duration=613.138µs 2024-09-21T13:50:10.348 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.250899622Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-09-21T13:50:10.348 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.251566485Z level=info msg="Migration successfully executed" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" duration=666.605µs 2024-09-21T13:50:10.348 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.25193845Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-09-21T13:50:10.348 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.251976688Z level=info msg="Migration successfully executed" id="alter alert_rule table data column to mediumtext in mysql" duration=39.403µs 2024-09-21T13:50:10.348 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.252383041Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-09-21T13:50:10.348 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.255661852Z level=info msg="Migration successfully executed" id="add column for to alert_rule" duration=3.278509ms 2024-09-21T13:50:10.348 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.256021138Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-09-21T13:50:10.348 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.259350965Z level=info msg="Migration successfully executed" id="add column annotations to alert_rule" duration=3.329655ms 2024-09-21T13:50:10.348 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.259708824Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-09-21T13:50:10.348 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.262922667Z level=info msg="Migration successfully executed" id="add column labels to alert_rule" duration=3.214666ms 2024-09-21T13:50:10.349 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.26327573Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-09-21T13:50:10.349 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.263833412Z level=info msg="Migration successfully executed" id="remove unique index from alert_rule on org_id, title columns" duration=557.59µs 2024-09-21T13:50:10.349 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.264176783Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-09-21T13:50:10.349 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.264784041Z level=info msg="Migration successfully executed" id="add index in alert_rule on org_id, namespase_uid and title columns" duration=606.883µs 2024-09-21T13:50:10.349 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.265129211Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-09-21T13:50:10.349 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.268444995Z level=info msg="Migration successfully executed" id="add dashboard_uid column to alert_rule" duration=3.315549ms 2024-09-21T13:50:10.349 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.268804595Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-09-21T13:50:10.349 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.272169989Z level=info msg="Migration successfully executed" id="add panel_id column to alert_rule" duration=3.363356ms 2024-09-21T13:50:10.349 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.272531634Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-09-21T13:50:10.349 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.273155453Z level=info msg="Migration successfully executed" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" duration=623.308µs 2024-09-21T13:50:10.349 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.27351102Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-09-21T13:50:10.349 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.276821581Z level=info msg="Migration successfully executed" id="add rule_group_idx column to alert_rule" duration=3.310333ms 2024-09-21T13:50:10.349 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.277187537Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-09-21T13:50:10.349 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.280516449Z level=info msg="Migration successfully executed" id="add is_paused column to alert_rule table" duration=3.328631ms 2024-09-21T13:50:10.350 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.280889097Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-09-21T13:50:10.350 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.280928196Z level=info msg="Migration successfully executed" id="fix is_paused column for alert_rule table" duration=38.156µs 2024-09-21T13:50:10.350 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.281858149Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-09-21T13:50:10.350 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.282638472Z level=info msg="Migration successfully executed" id="create alert_rule_version table" duration=800.49µs 2024-09-21T13:50:10.350 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.283015431Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-09-21T13:50:10.350 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.283668422Z level=info msg="Migration successfully executed" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" duration=652.755µs 2024-09-21T13:50:10.350 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.2840453Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-09-21T13:50:10.350 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.284729386Z level=info msg="Migration successfully executed" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" duration=683.833µs 2024-09-21T13:50:10.350 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.285116748Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-09-21T13:50:10.350 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.285154929Z level=info msg="Migration successfully executed" id="alter alert_rule_version table data column to mediumtext in mysql" duration=38.297µs 2024-09-21T13:50:10.350 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.286012113Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-09-21T13:50:10.350 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.289576702Z level=info msg="Migration successfully executed" id="add column for to alert_rule_version" duration=3.56441ms 2024-09-21T13:50:10.350 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.290033528Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-09-21T13:50:10.350 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.293546936Z level=info msg="Migration successfully executed" id="add column annotations to alert_rule_version" duration=3.513178ms 2024-09-21T13:50:10.351 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.293907454Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-09-21T13:50:10.351 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.297415929Z level=info msg="Migration successfully executed" id="add column labels to alert_rule_version" duration=3.508244ms 2024-09-21T13:50:10.351 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.297768079Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-09-21T13:50:10.351 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.301264875Z level=info msg="Migration successfully executed" id="add rule_group_idx column to alert_rule_version" duration=3.496492ms 2024-09-21T13:50:10.351 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.301618855Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-09-21T13:50:10.351 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.305080025Z level=info msg="Migration successfully executed" id="add is_paused column to alert_rule_versions table" duration=3.460875ms 2024-09-21T13:50:10.351 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.305411325Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-09-21T13:50:10.351 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.305448746Z level=info msg="Migration successfully executed" id="fix is_paused column for alert_rule_version table" duration=37.582µs 2024-09-21T13:50:10.351 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.306292694Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-09-21T13:50:10.351 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.307006794Z level=info msg="Migration successfully executed" id=create_alert_configuration_table duration=713.711µs 2024-09-21T13:50:10.351 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.308983674Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-09-21T13:50:10.351 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.312688738Z level=info msg="Migration successfully executed" id="Add column default in alert_configuration" duration=3.704724ms 2024-09-21T13:50:10.351 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.313051388Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-09-21T13:50:10.351 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.313089406Z level=info msg="Migration successfully executed" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" duration=38.576µs 2024-09-21T13:50:10.352 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.313492812Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-09-21T13:50:10.352 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.317944697Z level=info msg="Migration successfully executed" id="add column org_id in alert_configuration" duration=4.451189ms 2024-09-21T13:50:10.352 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.31836774Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-09-21T13:50:10.352 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.319009291Z level=info msg="Migration successfully executed" id="add index in alert_configuration table on org_id column" duration=640.276µs 2024-09-21T13:50:10.352 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.321076271Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-09-21T13:50:10.352 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.327526567Z level=info msg="Migration successfully executed" id="add configuration_hash column to alert_configuration" duration=6.450585ms 2024-09-21T13:50:10.352 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.328044974Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-09-21T13:50:10.352 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.328581128Z level=info msg="Migration successfully executed" id=create_ngalert_configuration_table duration=536.635µs 2024-09-21T13:50:10.352 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.328980497Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-09-21T13:50:10.352 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.32978653Z level=info msg="Migration successfully executed" id="add index in ngalert_configuration on org_id column" duration=805.077µs 2024-09-21T13:50:10.352 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.331833607Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-09-21T13:50:10.352 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.335629149Z level=info msg="Migration successfully executed" id="add column send_alerts_to in ngalert_configuration" duration=3.796537ms 2024-09-21T13:50:10.352 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.336035277Z level=info msg="Executing migration" id="create provenance_type table" 2024-09-21T13:50:10.352 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.336545947Z level=info msg="Migration successfully executed" id="create provenance_type table" duration=509.97µs 2024-09-21T13:50:10.353 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.338792883Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-09-21T13:50:10.353 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.339449995Z level=info msg="Migration successfully executed" id="add index to uniquify (record_key, record_type, org_id) columns" duration=656.929µs 2024-09-21T13:50:10.353 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.339812191Z level=info msg="Executing migration" id="create alert_image table" 2024-09-21T13:50:10.353 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.340341796Z level=info msg="Migration successfully executed" id="create alert_image table" duration=529.505µs 2024-09-21T13:50:10.430 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:50:10.451 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:10.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:10.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:50:10.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:10 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:50:10.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:10 smithi049 ceph-mon[40489]: 3.2 scrub starts 2024-09-21T13:50:10.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:10 smithi049 ceph-mon[40489]: 3.2 scrub ok 2024-09-21T13:50:10.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:10 smithi049 ceph-mon[40489]: pgmap v34: 148 pgs: 148 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T13:50:10.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:10 smithi049 ceph-mon[40489]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:50:10.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:10 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T13:50:10.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:10 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/18041778' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T13:50:10.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:10 smithi049 ceph-mon[40489]: osdmap e86: 8 total, 8 up, 8 in 2024-09-21T13:50:10.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:10 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:10.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:10 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:10.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:10 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:10.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:10 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:10.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:10 smithi049 ceph-mon[40489]: Deploying daemon node-exporter.a on smithi049 2024-09-21T13:50:10.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:10 smithi049 ceph-mon[40489]: 6.e scrub starts 2024-09-21T13:50:10.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:10 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:50:10.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:10 smithi049 ceph-mon[30080]: 3.2 scrub starts 2024-09-21T13:50:10.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:10 smithi049 ceph-mon[30080]: 3.2 scrub ok 2024-09-21T13:50:10.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:10 smithi049 ceph-mon[30080]: pgmap v34: 148 pgs: 148 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T13:50:10.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:10 smithi049 ceph-mon[30080]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:50:10.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:10 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T13:50:10.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:10 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/18041778' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T13:50:10.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:10 smithi049 ceph-mon[30080]: osdmap e86: 8 total, 8 up, 8 in 2024-09-21T13:50:10.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:10 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:10.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:10 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:10.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:10 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:10.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:10 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:10.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:10 smithi049 ceph-mon[30080]: Deploying daemon node-exporter.a on smithi049 2024-09-21T13:50:10.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:10 smithi049 ceph-mon[30080]: 6.e scrub starts 2024-09-21T13:50:10.552 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:10.573 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:10.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:10.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:10.588 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:50:10.588 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-mon[33032]: 3.2 scrub starts 2024-09-21T13:50:10.588 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-mon[33032]: 3.2 scrub ok 2024-09-21T13:50:10.588 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-mon[33032]: pgmap v34: 148 pgs: 148 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T13:50:10.589 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-mon[33032]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:50:10.589 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T13:50:10.589 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/18041778' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T13:50:10.589 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-mon[33032]: osdmap e86: 8 total, 8 up, 8 in 2024-09-21T13:50:10.589 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:10.589 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:10.589 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:10.589 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:10.589 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-mon[33032]: Deploying daemon node-exporter.a on smithi049 2024-09-21T13:50:10.589 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-mon[33032]: 6.e scrub starts 2024-09-21T13:50:10.590 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.342585389Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-09-21T13:50:10.590 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.343186115Z level=info msg="Migration successfully executed" id="add unique index on token to alert_image table" duration=598.118µs 2024-09-21T13:50:10.590 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.345327441Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-09-21T13:50:10.590 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.345369837Z level=info msg="Migration successfully executed" id="support longer URLs in alert_image table" duration=43.176µs 2024-09-21T13:50:10.590 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.348103619Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-09-21T13:50:10.590 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.348696171Z level=info msg="Migration successfully executed" id=create_alert_configuration_history_table duration=592.146µs 2024-09-21T13:50:10.590 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.349057996Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-09-21T13:50:10.590 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.349646497Z level=info msg="Migration successfully executed" id="drop non-unique orgID index on alert_configuration" duration=588.266µs 2024-09-21T13:50:10.590 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.351705395Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-09-21T13:50:10.591 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.351935239Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-09-21T13:50:10.591 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.354504022Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-09-21T13:50:10.591 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.354776011Z level=info msg="Migration successfully executed" id="extract alertmanager configuration history to separate table" duration=270.207µs 2024-09-21T13:50:10.591 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.357331824Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-09-21T13:50:10.591 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.357953602Z level=info msg="Migration successfully executed" id="add unique index on orgID to alert_configuration" duration=621.404µs 2024-09-21T13:50:10.591 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.358285072Z level=info msg="Executing migration" id="add last_applied column to alert_configuration_history" 2024-09-21T13:50:10.591 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.362054906Z level=info msg="Migration successfully executed" id="add last_applied column to alert_configuration_history" duration=3.76949ms 2024-09-21T13:50:10.591 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.362426687Z level=info msg="Executing migration" id="create library_element table v1" 2024-09-21T13:50:10.591 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.363174442Z level=info msg="Migration successfully executed" id="create library_element table v1" duration=747.618µs 2024-09-21T13:50:10.591 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.365360795Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-09-21T13:50:10.591 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.366135755Z level=info msg="Migration successfully executed" id="add index library_element org_id-folder_id-name-kind" duration=774.958µs 2024-09-21T13:50:10.591 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.368109471Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-09-21T13:50:10.591 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.368782117Z level=info msg="Migration successfully executed" id="create library_element_connection table v1" duration=674.793µs 2024-09-21T13:50:10.592 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.369148702Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-09-21T13:50:10.592 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.369783879Z level=info msg="Migration successfully executed" id="add index library_element_connection element_id-kind-connection_id" duration=634.887µs 2024-09-21T13:50:10.592 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.372070409Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-09-21T13:50:10.592 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.372671238Z level=info msg="Migration successfully executed" id="add unique index library_element org_id_uid" duration=600.555µs 2024-09-21T13:50:10.592 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.374439503Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-09-21T13:50:10.592 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.374471597Z level=info msg="Migration successfully executed" id="increase max description length to 2048" duration=32.593µs 2024-09-21T13:50:10.592 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.377182103Z level=info msg="Executing migration" id="alter library_element model to mediumtext" 2024-09-21T13:50:10.592 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.37723556Z level=info msg="Migration successfully executed" id="alter library_element model to mediumtext" duration=53.783µs 2024-09-21T13:50:10.592 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.377645653Z level=info msg="Executing migration" id="add library_element folder uid" 2024-09-21T13:50:10.592 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.38169121Z level=info msg="Migration successfully executed" id="add library_element folder uid" duration=4.043677ms 2024-09-21T13:50:10.592 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.382046482Z level=info msg="Executing migration" id="populate library_element folder_uid" 2024-09-21T13:50:10.592 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.382279811Z level=info msg="Migration successfully executed" id="populate library_element folder_uid" duration=233.177µs 2024-09-21T13:50:10.592 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.384620448Z level=info msg="Executing migration" id="add index library_element org_id-folder_uid-name-kind" 2024-09-21T13:50:10.593 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.385313209Z level=info msg="Migration successfully executed" id="add index library_element org_id-folder_uid-name-kind" duration=692.196µs 2024-09-21T13:50:10.593 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.387525177Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-09-21T13:50:10.593 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.387699023Z level=info msg="Migration successfully executed" id="clone move dashboard alerts to unified alerting" duration=173.652µs 2024-09-21T13:50:10.593 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.388240028Z level=info msg="Executing migration" id="create data_keys table" 2024-09-21T13:50:10.593 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.388956085Z level=info msg="Migration successfully executed" id="create data_keys table" duration=715.718µs 2024-09-21T13:50:10.593 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.390956459Z level=info msg="Executing migration" id="create secrets table" 2024-09-21T13:50:10.593 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.39152812Z level=info msg="Migration successfully executed" id="create secrets table" duration=571.446µs 2024-09-21T13:50:10.593 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.393828382Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-09-21T13:50:10.593 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.416117268Z level=info msg="Migration successfully executed" id="rename data_keys name column to id" duration=22.290651ms 2024-09-21T13:50:10.593 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.416543367Z level=info msg="Executing migration" id="add name column into data_keys" 2024-09-21T13:50:10.593 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.42102734Z level=info msg="Migration successfully executed" id="add name column into data_keys" duration=4.483718ms 2024-09-21T13:50:10.593 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.4213683Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-09-21T13:50:10.593 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.421460156Z level=info msg="Migration successfully executed" id="copy data_keys id column values into name" duration=92.225µs 2024-09-21T13:50:10.593 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.42185569Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-09-21T13:50:10.594 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.443580301Z level=info msg="Migration successfully executed" id="rename data_keys name column to label" duration=21.720507ms 2024-09-21T13:50:10.594 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.444061446Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-09-21T13:50:10.594 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.46626745Z level=info msg="Migration successfully executed" id="rename data_keys id column back to name" duration=22.204411ms 2024-09-21T13:50:10.594 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.46675847Z level=info msg="Executing migration" id="create kv_store table v1" 2024-09-21T13:50:10.594 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.467355871Z level=info msg="Migration successfully executed" id="create kv_store table v1" duration=597.353µs 2024-09-21T13:50:10.594 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.467725619Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-09-21T13:50:10.594 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.468433627Z level=info msg="Migration successfully executed" id="add index kv_store.org_id-namespace-key" duration=707.705µs 2024-09-21T13:50:10.594 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.468790282Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-09-21T13:50:10.594 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.468927016Z level=info msg="Migration successfully executed" id="update dashboard_uid and panel_id from existing annotations" duration=136.707µs 2024-09-21T13:50:10.594 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.46931404Z level=info msg="Executing migration" id="create permission table" 2024-09-21T13:50:10.594 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.469852443Z level=info msg="Migration successfully executed" id="create permission table" duration=539.242µs 2024-09-21T13:50:10.594 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.470192302Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-09-21T13:50:10.594 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.47078719Z level=info msg="Migration successfully executed" id="add unique index permission.role_id" duration=594.617µs 2024-09-21T13:50:10.594 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.471141162Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-09-21T13:50:10.595 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.471777456Z level=info msg="Migration successfully executed" id="add unique index role_id_action_scope" duration=635.841µs 2024-09-21T13:50:10.595 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.472141782Z level=info msg="Executing migration" id="create role table" 2024-09-21T13:50:10.595 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.472717009Z level=info msg="Migration successfully executed" id="create role table" duration=573.314µs 2024-09-21T13:50:10.595 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.473068887Z level=info msg="Executing migration" id="add column display_name" 2024-09-21T13:50:10.595 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.477497856Z level=info msg="Migration successfully executed" id="add column display_name" duration=4.428799ms 2024-09-21T13:50:10.595 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.477866957Z level=info msg="Executing migration" id="add column group_name" 2024-09-21T13:50:10.595 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.482359758Z level=info msg="Migration successfully executed" id="add column group_name" duration=4.492565ms 2024-09-21T13:50:10.596 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.482726788Z level=info msg="Executing migration" id="add index role.org_id" 2024-09-21T13:50:10.596 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.483375537Z level=info msg="Migration successfully executed" id="add index role.org_id" duration=649.295µs 2024-09-21T13:50:10.596 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.483728399Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-09-21T13:50:10.596 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.48440286Z level=info msg="Migration successfully executed" id="add unique index role_org_id_name" duration=674.035µs 2024-09-21T13:50:10.596 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.484757493Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-09-21T13:50:10.596 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.485425427Z level=info msg="Migration successfully executed" id="add index role_org_id_uid" duration=667.623µs 2024-09-21T13:50:10.596 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.485765167Z level=info msg="Executing migration" id="create team role table" 2024-09-21T13:50:10.597 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.486381254Z level=info msg="Migration successfully executed" id="create team role table" duration=615.88µs 2024-09-21T13:50:10.597 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.486788993Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-09-21T13:50:10.597 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.487455411Z level=info msg="Migration successfully executed" id="add index team_role.org_id" duration=666.206µs 2024-09-21T13:50:10.597 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.48780065Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-09-21T13:50:10.597 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.48847413Z level=info msg="Migration successfully executed" id="add unique index team_role_org_id_team_id_role_id" duration=673.131µs 2024-09-21T13:50:10.597 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.488822745Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-09-21T13:50:10.597 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.489477982Z level=info msg="Migration successfully executed" id="add index team_role.team_id" duration=655.233µs 2024-09-21T13:50:10.597 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.489827067Z level=info msg="Executing migration" id="create user role table" 2024-09-21T13:50:10.597 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.490377359Z level=info msg="Migration successfully executed" id="create user role table" duration=550.254µs 2024-09-21T13:50:10.597 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.490722433Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-09-21T13:50:10.597 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.491363584Z level=info msg="Migration successfully executed" id="add index user_role.org_id" duration=640.991µs 2024-09-21T13:50:10.597 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.491694469Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-09-21T13:50:10.597 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.492387152Z level=info msg="Migration successfully executed" id="add unique index user_role_org_id_user_id_role_id" duration=692.401µs 2024-09-21T13:50:10.597 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.492734149Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-09-21T13:50:10.598 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.49339367Z level=info msg="Migration successfully executed" id="add index user_role.user_id" duration=659.295µs 2024-09-21T13:50:10.598 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.493744163Z level=info msg="Executing migration" id="create builtin role table" 2024-09-21T13:50:10.598 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.494312925Z level=info msg="Migration successfully executed" id="create builtin role table" duration=568.566µs 2024-09-21T13:50:10.598 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.494662717Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-09-21T13:50:10.598 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.49533841Z level=info msg="Migration successfully executed" id="add index builtin_role.role_id" duration=675.411µs 2024-09-21T13:50:10.598 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.495673625Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-09-21T13:50:10.598 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.496334858Z level=info msg="Migration successfully executed" id="add index builtin_role.name" duration=660.981µs 2024-09-21T13:50:10.598 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.496678472Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-09-21T13:50:10.598 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.501191674Z level=info msg="Migration successfully executed" id="Add column org_id to builtin_role table" duration=4.512862ms 2024-09-21T13:50:10.598 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.50153768Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-09-21T13:50:10.598 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.502180385Z level=info msg="Migration successfully executed" id="add index builtin_role.org_id" duration=642.272µs 2024-09-21T13:50:10.598 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.502528666Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-09-21T13:50:10.598 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.503186682Z level=info msg="Migration successfully executed" id="add unique index builtin_role_org_id_role_id_role" duration=657.677µs 2024-09-21T13:50:10.599 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.503533748Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-09-21T13:50:10.599 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.504190223Z level=info msg="Migration successfully executed" id="Remove unique index role_org_id_uid" duration=656.499µs 2024-09-21T13:50:10.599 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.504633293Z level=info msg="Executing migration" id="add unique index role.uid" 2024-09-21T13:50:10.599 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.505292646Z level=info msg="Migration successfully executed" id="add unique index role.uid" duration=659.132µs 2024-09-21T13:50:10.599 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.505627266Z level=info msg="Executing migration" id="create seed assignment table" 2024-09-21T13:50:10.599 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.506127353Z level=info msg="Migration successfully executed" id="create seed assignment table" duration=499.795µs 2024-09-21T13:50:10.599 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.506473996Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-09-21T13:50:10.599 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.507164399Z level=info msg="Migration successfully executed" id="add unique index builtin_role_role_name" duration=690.02µs 2024-09-21T13:50:10.599 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.50751462Z level=info msg="Executing migration" id="add column hidden to role table" 2024-09-21T13:50:10.599 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.512165953Z level=info msg="Migration successfully executed" id="add column hidden to role table" duration=4.651072ms 2024-09-21T13:50:10.599 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.512512704Z level=info msg="Executing migration" id="permission kind migration" 2024-09-21T13:50:10.599 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.517323935Z level=info msg="Migration successfully executed" id="permission kind migration" duration=4.810917ms 2024-09-21T13:50:10.599 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.517685847Z level=info msg="Executing migration" id="permission attribute migration" 2024-09-21T13:50:10.600 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.522244837Z level=info msg="Migration successfully executed" id="permission attribute migration" duration=4.558828ms 2024-09-21T13:50:10.600 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.522590958Z level=info msg="Executing migration" id="permission identifier migration" 2024-09-21T13:50:10.600 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.526974498Z level=info msg="Migration successfully executed" id="permission identifier migration" duration=4.382416ms 2024-09-21T13:50:10.600 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.527318219Z level=info msg="Executing migration" id="add permission identifier index" 2024-09-21T13:50:10.600 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.527982457Z level=info msg="Migration successfully executed" id="add permission identifier index" duration=663.759µs 2024-09-21T13:50:10.600 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.528320845Z level=info msg="Executing migration" id="add permission action scope role_id index" 2024-09-21T13:50:10.600 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.528972626Z level=info msg="Migration successfully executed" id="add permission action scope role_id index" duration=651.505µs 2024-09-21T13:50:10.600 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.529305271Z level=info msg="Executing migration" id="remove permission role_id action scope index" 2024-09-21T13:50:10.600 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.529930306Z level=info msg="Migration successfully executed" id="remove permission role_id action scope index" duration=624.973µs 2024-09-21T13:50:10.600 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.53026018Z level=info msg="Executing migration" id="create query_history table v1" 2024-09-21T13:50:10.600 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.530814424Z level=info msg="Migration successfully executed" id="create query_history table v1" duration=553.882µs 2024-09-21T13:50:10.600 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.531174818Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-09-21T13:50:10.600 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.531817079Z level=info msg="Migration successfully executed" id="add index query_history.org_id-created_by-datasource_uid" duration=642.043µs 2024-09-21T13:50:10.600 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.532171224Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-09-21T13:50:10.601 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.532209416Z level=info msg="Migration successfully executed" id="alter table query_history alter column created_by type to bigint" duration=38.598µs 2024-09-21T13:50:10.601 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.532589376Z level=info msg="Executing migration" id="rbac disabled migrator" 2024-09-21T13:50:10.601 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.532608048Z level=info msg="Migration successfully executed" id="rbac disabled migrator" duration=19.131µs 2024-09-21T13:50:10.601 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.532960372Z level=info msg="Executing migration" id="teams permissions migration" 2024-09-21T13:50:10.601 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.533205988Z level=info msg="Migration successfully executed" id="teams permissions migration" duration=245.617µs 2024-09-21T13:50:10.601 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.533572197Z level=info msg="Executing migration" id="dashboard permissions" 2024-09-21T13:50:10.601 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.533903361Z level=info msg="Migration successfully executed" id="dashboard permissions" duration=331.662µs 2024-09-21T13:50:10.601 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.534273861Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-09-21T13:50:10.601 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.534659715Z level=info msg="Migration successfully executed" id="dashboard permissions uid scopes" duration=385.72µs 2024-09-21T13:50:10.601 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.535038511Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-09-21T13:50:10.601 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.535164959Z level=info msg="Migration successfully executed" id="drop managed folder create actions" duration=126.428µs 2024-09-21T13:50:10.601 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.535523202Z level=info msg="Executing migration" id="alerting notification permissions" 2024-09-21T13:50:10.601 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.535797718Z level=info msg="Migration successfully executed" id="alerting notification permissions" duration=273.62µs 2024-09-21T13:50:10.601 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.536198604Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-09-21T13:50:10.602 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.536696271Z level=info msg="Migration successfully executed" id="create query_history_star table v1" duration=497.481µs 2024-09-21T13:50:10.602 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.537055041Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-09-21T13:50:10.602 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.537752242Z level=info msg="Migration successfully executed" id="add index query_history.user_id-query_uid" duration=696.814µs 2024-09-21T13:50:10.602 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.538109189Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-09-21T13:50:10.602 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.542704259Z level=info msg="Migration successfully executed" id="add column org_id in query_history_star" duration=4.594899ms 2024-09-21T13:50:10.602 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.543073595Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-09-21T13:50:10.602 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.543110874Z level=info msg="Migration successfully executed" id="alter table query_history_star_mig column user_id type to bigint" duration=37.554µs 2024-09-21T13:50:10.602 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.543501827Z level=info msg="Executing migration" id="create correlation table v1" 2024-09-21T13:50:10.602 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.544127499Z level=info msg="Migration successfully executed" id="create correlation table v1" duration=625.314µs 2024-09-21T13:50:10.602 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.54447518Z level=info msg="Executing migration" id="add index correlations.uid" 2024-09-21T13:50:10.602 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.54518379Z level=info msg="Migration successfully executed" id="add index correlations.uid" duration=708.248µs 2024-09-21T13:50:10.602 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.545525388Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-09-21T13:50:10.602 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.546182639Z level=info msg="Migration successfully executed" id="add index correlations.source_uid" duration=657.175µs 2024-09-21T13:50:10.602 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.547948809Z level=info msg="Executing migration" id="add correlation config column" 2024-09-21T13:50:10.603 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.552642417Z level=info msg="Migration successfully executed" id="add correlation config column" duration=4.69328ms 2024-09-21T13:50:10.603 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.552996016Z level=info msg="Executing migration" id="drop index IDX_correlation_uid - v1" 2024-09-21T13:50:10.603 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.5536332Z level=info msg="Migration successfully executed" id="drop index IDX_correlation_uid - v1" duration=639.008µs 2024-09-21T13:50:10.603 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.553978445Z level=info msg="Executing migration" id="drop index IDX_correlation_source_uid - v1" 2024-09-21T13:50:10.603 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.554601718Z level=info msg="Migration successfully executed" id="drop index IDX_correlation_source_uid - v1" duration=623.161µs 2024-09-21T13:50:10.603 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.55494005Z level=info msg="Executing migration" id="Rename table correlation to correlation_tmp_qwerty - v1" 2024-09-21T13:50:10.603 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.572166947Z level=info msg="Migration successfully executed" id="Rename table correlation to correlation_tmp_qwerty - v1" duration=17.223657ms 2024-09-21T13:50:10.603 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.572644359Z level=info msg="Executing migration" id="create correlation v2" 2024-09-21T13:50:10.603 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.573485426Z level=info msg="Migration successfully executed" id="create correlation v2" duration=840.647µs 2024-09-21T13:50:10.603 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.573903188Z level=info msg="Executing migration" id="create index IDX_correlation_uid - v2" 2024-09-21T13:50:10.603 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.574631301Z level=info msg="Migration successfully executed" id="create index IDX_correlation_uid - v2" duration=727.986µs 2024-09-21T13:50:10.603 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.576406411Z level=info msg="Executing migration" id="create index IDX_correlation_source_uid - v2" 2024-09-21T13:50:10.603 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.577090593Z level=info msg="Migration successfully executed" id="create index IDX_correlation_source_uid - v2" duration=683.977µs 2024-09-21T13:50:10.604 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.577548645Z level=info msg="Executing migration" id="create index IDX_correlation_org_id - v2" 2024-09-21T13:50:10.604 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.578243853Z level=info msg="Migration successfully executed" id="create index IDX_correlation_org_id - v2" duration=694.967µs 2024-09-21T13:50:10.604 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.580369978Z level=info msg="Executing migration" id="copy correlation v1 to v2" 2024-09-21T13:50:10.604 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.580612499Z level=info msg="Migration successfully executed" id="copy correlation v1 to v2" duration=246.037µs 2024-09-21T13:50:10.604 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.582959949Z level=info msg="Executing migration" id="drop correlation_tmp_qwerty" 2024-09-21T13:50:10.604 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.583545075Z level=info msg="Migration successfully executed" id="drop correlation_tmp_qwerty" duration=585.595µs 2024-09-21T13:50:10.604 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.583936681Z level=info msg="Executing migration" id="add provisioning column" 2024-09-21T13:50:10.604 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.588689736Z level=info msg="Migration successfully executed" id="add provisioning column" duration=4.75343ms 2024-09-21T13:50:10.604 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.589128599Z level=info msg="Executing migration" id="create entity_events table" 2024-09-21T13:50:10.604 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.589681757Z level=info msg="Migration successfully executed" id="create entity_events table" duration=553.478µs 2024-09-21T13:50:10.604 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.591715415Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-09-21T13:50:10.604 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.592371398Z level=info msg="Migration successfully executed" id="create dashboard public config v1" duration=655.741µs 2024-09-21T13:50:10.675 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:10.696 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:10.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:10.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:10.798 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:50:10.824 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:10.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:10.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:50:10.840 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.594370215Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-21T13:50:10.841 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.594632695Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-21T13:50:10.841 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.59502603Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-21T13:50:10.841 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.595397663Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-21T13:50:10.841 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.597054855Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-09-21T13:50:10.841 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.597626543Z level=info msg="Migration successfully executed" id="Drop old dashboard public config table" duration=571.421µs 2024-09-21T13:50:10.841 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.599637667Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-09-21T13:50:10.841 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.600325267Z level=info msg="Migration successfully executed" id="recreate dashboard public config v1" duration=688.149µs 2024-09-21T13:50:10.841 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.600672729Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-09-21T13:50:10.841 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.601336886Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_public_config_uid - v1" duration=663.996µs 2024-09-21T13:50:10.841 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.601708123Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-21T13:50:10.841 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.602414697Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" duration=706.618µs 2024-09-21T13:50:10.842 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.602767658Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-09-21T13:50:10.842 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.603421455Z level=info msg="Migration successfully executed" id="drop index UQE_dashboard_public_config_uid - v2" duration=653.619µs 2024-09-21T13:50:10.842 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.603751423Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-21T13:50:10.842 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.60440258Z level=info msg="Migration successfully executed" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" duration=650.986µs 2024-09-21T13:50:10.843 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.604736478Z level=info msg="Executing migration" id="Drop public config table" 2024-09-21T13:50:10.843 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.605291744Z level=info msg="Migration successfully executed" id="Drop public config table" duration=554.863µs 2024-09-21T13:50:10.843 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.605630183Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-09-21T13:50:10.843 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.606384652Z level=info msg="Migration successfully executed" id="Recreate dashboard public config v2" duration=754.262µs 2024-09-21T13:50:10.843 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.606717439Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-09-21T13:50:10.843 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.607441743Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_public_config_uid - v2" duration=723.93µs 2024-09-21T13:50:10.843 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.607812814Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-21T13:50:10.843 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.60864097Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" duration=827.139µs 2024-09-21T13:50:10.843 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.609021739Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-09-21T13:50:10.843 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.609758874Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_public_config_access_token - v2" duration=736.792µs 2024-09-21T13:50:10.844 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.610108197Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-09-21T13:50:10.844 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.624731504Z level=info msg="Migration successfully executed" id="Rename table dashboard_public_config to dashboard_public - v2" duration=14.624695ms 2024-09-21T13:50:10.844 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.625110084Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-09-21T13:50:10.844 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.630311439Z level=info msg="Migration successfully executed" id="add annotations_enabled column" duration=5.201236ms 2024-09-21T13:50:10.844 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.630665242Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-09-21T13:50:10.844 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.635581499Z level=info msg="Migration successfully executed" id="add time_selection_enabled column" duration=4.91414ms 2024-09-21T13:50:10.844 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.635939582Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-09-21T13:50:10.844 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.636078755Z level=info msg="Migration successfully executed" id="delete orphaned public dashboards" duration=138.242µs 2024-09-21T13:50:10.844 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.636912651Z level=info msg="Executing migration" id="add share column" 2024-09-21T13:50:10.844 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.641974327Z level=info msg="Migration successfully executed" id="add share column" duration=5.061341ms 2024-09-21T13:50:10.844 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.642310255Z level=info msg="Executing migration" id="backfill empty share column fields with default of public" 2024-09-21T13:50:10.844 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.642437441Z level=info msg="Migration successfully executed" id="backfill empty share column fields with default of public" duration=127.013µs 2024-09-21T13:50:10.844 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.643231239Z level=info msg="Executing migration" id="create file table" 2024-09-21T13:50:10.844 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.643882775Z level=info msg="Migration successfully executed" id="create file table" duration=651.19µs 2024-09-21T13:50:10.844 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.644229856Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-09-21T13:50:10.845 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.644932186Z level=info msg="Migration successfully executed" id="file table idx: path natural pk" duration=701.967µs 2024-09-21T13:50:10.845 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.645270661Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-09-21T13:50:10.845 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.645969289Z level=info msg="Migration successfully executed" id="file table idx: parent_folder_path_hash fast folder retrieval" duration=698.298µs 2024-09-21T13:50:10.845 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.646310472Z level=info msg="Executing migration" id="create file_meta table" 2024-09-21T13:50:10.845 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.646910215Z level=info msg="Migration successfully executed" id="create file_meta table" duration=599.394µs 2024-09-21T13:50:10.845 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.647256651Z level=info msg="Executing migration" id="file table idx: path key" 2024-09-21T13:50:10.845 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.647957616Z level=info msg="Migration successfully executed" id="file table idx: path key" duration=700.582µs 2024-09-21T13:50:10.845 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.648296152Z level=info msg="Executing migration" id="set path collation in file table" 2024-09-21T13:50:10.845 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.648340581Z level=info msg="Migration successfully executed" id="set path collation in file table" duration=44.747µs 2024-09-21T13:50:10.845 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.649289559Z level=info msg="Executing migration" id="migrate contents column to mediumblob for MySQL" 2024-09-21T13:50:10.845 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.649394221Z level=info msg="Migration successfully executed" id="migrate contents column to mediumblob for MySQL" duration=105.852µs 2024-09-21T13:50:10.845 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.650001816Z level=info msg="Executing migration" id="managed permissions migration" 2024-09-21T13:50:10.845 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.650305167Z level=info msg="Migration successfully executed" id="managed permissions migration" duration=303.069µs 2024-09-21T13:50:10.846 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.650863406Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-09-21T13:50:10.846 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.650984397Z level=info msg="Migration successfully executed" id="managed folder permissions alert actions migration" duration=121.09µs 2024-09-21T13:50:10.846 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.6517099Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-09-21T13:50:10.846 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.652513395Z level=info msg="Migration successfully executed" id="RBAC action name migrator" duration=803.232µs 2024-09-21T13:50:10.846 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.652904945Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-09-21T13:50:10.846 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.658068526Z level=info msg="Migration successfully executed" id="Add UID column to playlist" duration=5.164014ms 2024-09-21T13:50:10.846 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.658401477Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-09-21T13:50:10.846 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.658502801Z level=info msg="Migration successfully executed" id="Update uid column values in playlist" duration=101.499µs 2024-09-21T13:50:10.846 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.659423853Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-09-21T13:50:10.846 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.660235882Z level=info msg="Migration successfully executed" id="Add index for uid in playlist" duration=813.938µs 2024-09-21T13:50:10.846 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.660590569Z level=info msg="Executing migration" id="update group index for alert rules" 2024-09-21T13:50:10.846 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.66081153Z level=info msg="Migration successfully executed" id="update group index for alert rules" duration=221.407µs 2024-09-21T13:50:10.847 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.661447945Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-09-21T13:50:10.847 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.661586846Z level=info msg="Migration successfully executed" id="managed folder permissions alert actions repeated migration" duration=138.717µs 2024-09-21T13:50:10.847 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.662331059Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-09-21T13:50:10.847 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.662656695Z level=info msg="Migration successfully executed" id="admin only folder/dashboard permission" duration=323.693µs 2024-09-21T13:50:10.847 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.663229039Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-09-21T13:50:10.847 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.668226759Z level=info msg="Migration successfully executed" id="add action column to seed_assignment" duration=4.997739ms 2024-09-21T13:50:10.847 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.668584405Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-09-21T13:50:10.847 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.674040525Z level=info msg="Migration successfully executed" id="add scope column to seed_assignment" duration=5.455859ms 2024-09-21T13:50:10.847 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.674374986Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-09-21T13:50:10.847 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.675035099Z level=info msg="Migration successfully executed" id="remove unique index builtin_role_role_name before nullable update" duration=660.122µs 2024-09-21T13:50:10.847 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.675376137Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-09-21T13:50:10.848 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.734272004Z level=info msg="Migration successfully executed" id="update seed_assignment role_name column to nullable" duration=58.884508ms 2024-09-21T13:50:10.848 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.734763369Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-09-21T13:50:10.848 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.73548913Z level=info msg="Migration successfully executed" id="add unique index builtin_role_name back" duration=723.947µs 2024-09-21T13:50:10.848 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.737570541Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-09-21T13:50:10.848 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.738421943Z level=info msg="Migration successfully executed" id="add unique index builtin_role_action_scope" duration=852.932µs 2024-09-21T13:50:10.848 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.740437688Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-09-21T13:50:10.848 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.757631608Z level=info msg="Migration successfully executed" id="add primary key to seed_assigment" duration=17.194051ms 2024-09-21T13:50:10.848 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.758025138Z level=info msg="Executing migration" id="add origin column to seed_assignment" 2024-09-21T13:50:10.848 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.763113609Z level=info msg="Migration successfully executed" id="add origin column to seed_assignment" duration=5.088328ms 2024-09-21T13:50:10.848 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.763510995Z level=info msg="Executing migration" id="add origin to plugin seed_assignment" 2024-09-21T13:50:10.848 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.763705769Z level=info msg="Migration successfully executed" id="add origin to plugin seed_assignment" duration=194.871µs 2024-09-21T13:50:10.848 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.766366065Z level=info msg="Executing migration" id="prevent seeding OnCall access" 2024-09-21T13:50:10.849 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.766479048Z level=info msg="Migration successfully executed" id="prevent seeding OnCall access" duration=112.93µs 2024-09-21T13:50:10.849 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.769091186Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-09-21T13:50:10.849 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.769214103Z level=info msg="Migration successfully executed" id="managed folder permissions alert actions repeated fixed migration" duration=123.018µs 2024-09-21T13:50:10.849 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.771867022Z level=info msg="Executing migration" id="managed folder permissions library panel actions migration" 2024-09-21T13:50:10.849 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.771987521Z level=info msg="Migration successfully executed" id="managed folder permissions library panel actions migration" duration=120.555µs 2024-09-21T13:50:10.849 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.774605533Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-09-21T13:50:10.849 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.774745506Z level=info msg="Migration successfully executed" id="migrate external alertmanagers to datsourcse" duration=139.874µs 2024-09-21T13:50:10.849 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.777481557Z level=info msg="Executing migration" id="create folder table" 2024-09-21T13:50:10.849 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.778154344Z level=info msg="Migration successfully executed" id="create folder table" duration=672.506µs 2024-09-21T13:50:10.849 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.778532523Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-09-21T13:50:10.849 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.779246133Z level=info msg="Migration successfully executed" id="Add index for parent_uid" duration=711.671µs 2024-09-21T13:50:10.849 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.780396556Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-09-21T13:50:10.849 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.781159561Z level=info msg="Migration successfully executed" id="Add unique index for folder.uid and folder.org_id" duration=764.595µs 2024-09-21T13:50:10.850 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.782972185Z level=info msg="Executing migration" id="Update folder title length" 2024-09-21T13:50:10.850 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.782987331Z level=info msg="Migration successfully executed" id="Update folder title length" duration=15.65µs 2024-09-21T13:50:10.850 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.785901594Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-09-21T13:50:10.850 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.786621535Z level=info msg="Migration successfully executed" id="Add unique index for folder.title and folder.parent_uid" duration=719.508µs 2024-09-21T13:50:10.850 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.788799455Z level=info msg="Executing migration" id="Remove unique index for folder.title and folder.parent_uid" 2024-09-21T13:50:10.850 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.789505519Z level=info msg="Migration successfully executed" id="Remove unique index for folder.title and folder.parent_uid" duration=705.913µs 2024-09-21T13:50:10.850 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.791532549Z level=info msg="Executing migration" id="Add unique index for title, parent_uid, and org_id" 2024-09-21T13:50:10.850 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.792328059Z level=info msg="Migration successfully executed" id="Add unique index for title, parent_uid, and org_id" duration=798.418µs 2024-09-21T13:50:10.850 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.794195213Z level=info msg="Executing migration" id="Sync dashboard and folder table" 2024-09-21T13:50:10.850 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.794498466Z level=info msg="Migration successfully executed" id="Sync dashboard and folder table" duration=303.019µs 2024-09-21T13:50:10.850 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.796999022Z level=info msg="Executing migration" id="Remove ghost folders from the folder table" 2024-09-21T13:50:10.850 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.797169712Z level=info msg="Migration successfully executed" id="Remove ghost folders from the folder table" duration=170.65µs 2024-09-21T13:50:10.850 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.799894021Z level=info msg="Executing migration" id="Remove unique index UQE_folder_uid_org_id" 2024-09-21T13:50:10.851 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.800549607Z level=info msg="Migration successfully executed" id="Remove unique index UQE_folder_uid_org_id" duration=655.488µs 2024-09-21T13:50:10.851 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.802529249Z level=info msg="Executing migration" id="Add unique index UQE_folder_org_id_uid" 2024-09-21T13:50:10.851 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.803245497Z level=info msg="Migration successfully executed" id="Add unique index UQE_folder_org_id_uid" duration=715.958µs 2024-09-21T13:50:10.851 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.805195293Z level=info msg="Executing migration" id="Remove unique index UQE_folder_title_parent_uid_org_id" 2024-09-21T13:50:10.851 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.805874023Z level=info msg="Migration successfully executed" id="Remove unique index UQE_folder_title_parent_uid_org_id" duration=678.349µs 2024-09-21T13:50:10.851 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.807750213Z level=info msg="Executing migration" id="Add unique index UQE_folder_org_id_parent_uid_title" 2024-09-21T13:50:10.851 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:50:10 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:50:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:50:10.852 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.80846296Z level=info msg="Migration successfully executed" id="Add unique index UQE_folder_org_id_parent_uid_title" duration=712.409µs 2024-09-21T13:50:10.852 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.810365231Z level=info msg="Executing migration" id="Remove index IDX_folder_parent_uid_org_id" 2024-09-21T13:50:10.852 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.811029007Z level=info msg="Migration successfully executed" id="Remove index IDX_folder_parent_uid_org_id" duration=662.441µs 2024-09-21T13:50:10.852 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.813066945Z level=info msg="Executing migration" id="create anon_device table" 2024-09-21T13:50:10.852 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.813651869Z level=info msg="Migration successfully executed" id="create anon_device table" duration=564.723µs 2024-09-21T13:50:10.852 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.815733597Z level=info msg="Executing migration" id="add unique index anon_device.device_id" 2024-09-21T13:50:10.852 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.816553446Z level=info msg="Migration successfully executed" id="add unique index anon_device.device_id" duration=819.522µs 2024-09-21T13:50:10.852 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.818517972Z level=info msg="Executing migration" id="add index anon_device.updated_at" 2024-09-21T13:50:10.852 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.81926748Z level=info msg="Migration successfully executed" id="add index anon_device.updated_at" duration=749.759µs 2024-09-21T13:50:10.852 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.821233587Z level=info msg="Executing migration" id="create signing_key table" 2024-09-21T13:50:10.852 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.821879178Z level=info msg="Migration successfully executed" id="create signing_key table" duration=646.069µs 2024-09-21T13:50:10.852 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.823947255Z level=info msg="Executing migration" id="add unique index signing_key.key_id" 2024-09-21T13:50:10.853 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.824625031Z level=info msg="Migration successfully executed" id="add unique index signing_key.key_id" duration=677.583µs 2024-09-21T13:50:10.853 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.826601809Z level=info msg="Executing migration" id="set legacy alert migration status in kvstore" 2024-09-21T13:50:10.853 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.827346253Z level=info msg="Migration successfully executed" id="set legacy alert migration status in kvstore" duration=744.15µs 2024-09-21T13:50:10.853 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.829483496Z level=info msg="Executing migration" id="migrate record of created folders during legacy migration to kvstore" 2024-09-21T13:50:10.853 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.829708781Z level=info msg="Migration successfully executed" id="migrate record of created folders during legacy migration to kvstore" duration=227.975µs 2024-09-21T13:50:10.853 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.832063137Z level=info msg="Executing migration" id="Add folder_uid for dashboard" 2024-09-21T13:50:10.853 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.837477655Z level=info msg="Migration successfully executed" id="Add folder_uid for dashboard" duration=5.414078ms 2024-09-21T13:50:10.853 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.83783562Z level=info msg="Executing migration" id="Populate dashboard folder_uid column" 2024-09-21T13:50:10.853 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.83833709Z level=info msg="Migration successfully executed" id="Populate dashboard folder_uid column" duration=501.936µs 2024-09-21T13:50:10.853 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.840789746Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_folder_uid_title" 2024-09-21T13:50:10.853 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.840817636Z level=info msg="Migration successfully executed" id="Add unique index for dashboard_org_id_folder_uid_title" duration=29.606µs 2024-09-21T13:50:10.853 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.843187029Z level=info msg="Executing migration" id="Delete unique index for dashboard_org_id_folder_id_title" 2024-09-21T13:50:10.853 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.844015166Z level=info msg="Migration successfully executed" id="Delete unique index for dashboard_org_id_folder_id_title" duration=828.157µs 2024-09-21T13:50:10.926 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:50:10.954 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:10.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:10.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:50:11.056 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:50:11.078 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:11.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:11.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:50:11.122 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.845763742Z level=info msg="Executing migration" id="Delete unique index for dashboard_org_id_folder_uid_title" 2024-09-21T13:50:11.122 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.84579403Z level=info msg="Migration successfully executed" id="Delete unique index for dashboard_org_id_folder_uid_title" duration=31.398µs 2024-09-21T13:50:11.122 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.846264353Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_folder_uid_title_is_folder" 2024-09-21T13:50:11.123 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.847121302Z level=info msg="Migration successfully executed" id="Add unique index for dashboard_org_id_folder_uid_title_is_folder" duration=856.621µs 2024-09-21T13:50:11.123 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.848446262Z level=info msg="Executing migration" id="Restore index for dashboard_org_id_folder_id_title" 2024-09-21T13:50:11.123 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.849298296Z level=info msg="Migration successfully executed" id="Restore index for dashboard_org_id_folder_id_title" duration=851.746µs 2024-09-21T13:50:11.123 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.851182227Z level=info msg="Executing migration" id="create sso_setting table" 2024-09-21T13:50:11.123 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.851873543Z level=info msg="Migration successfully executed" id="create sso_setting table" duration=691.064µs 2024-09-21T13:50:11.123 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.852213811Z level=info msg="Executing migration" id="copy kvstore migration status to each org" 2024-09-21T13:50:11.123 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.852668104Z level=info msg="Migration successfully executed" id="copy kvstore migration status to each org" duration=454.68µs 2024-09-21T13:50:11.123 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.852998025Z level=info msg="Executing migration" id="add back entry for orgid=0 migrated status" 2024-09-21T13:50:11.123 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.853162755Z level=info msg="Migration successfully executed" id="add back entry for orgid=0 migrated status" duration=165.136µs 2024-09-21T13:50:11.123 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.853909954Z level=info msg="Executing migration" id="alter kv_store.value to longtext" 2024-09-21T13:50:11.123 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.853947464Z level=info msg="Migration successfully executed" id="alter kv_store.value to longtext" duration=38.848µs 2024-09-21T13:50:11.123 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.854740843Z level=info msg="Executing migration" id="add notification_settings column to alert_rule table" 2024-09-21T13:50:11.124 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.859875606Z level=info msg="Migration successfully executed" id="add notification_settings column to alert_rule table" duration=5.134363ms 2024-09-21T13:50:11.124 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.860244752Z level=info msg="Executing migration" id="add notification_settings column to alert_rule_version table" 2024-09-21T13:50:11.124 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.865557056Z level=info msg="Migration successfully executed" id="add notification_settings column to alert_rule_version table" duration=5.311383ms 2024-09-21T13:50:11.124 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.865926129Z level=info msg="Executing migration" id="removing scope from alert.instances:read action migration" 2024-09-21T13:50:11.124 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.866136636Z level=info msg="Migration successfully executed" id="removing scope from alert.instances:read action migration" duration=208.674µs 2024-09-21T13:50:11.124 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=migrator t=2024-09-21T13:50:10.866902808Z level=info msg="migrations completed" performed=551 skipped=0 duration=1.257217345s 2024-09-21T13:50:11.124 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=sqlstore t=2024-09-21T13:50:10.867916859Z level=info msg="Created default organization" 2024-09-21T13:50:11.125 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=secrets t=2024-09-21T13:50:10.868403955Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-09-21T13:50:11.125 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=plugin.store t=2024-09-21T13:50:10.882951201Z level=info msg="Loading plugins..." 2024-09-21T13:50:11.125 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=local.finder t=2024-09-21T13:50:10.92389993Z level=warn msg="Skipping finding plugins as directory does not exist" path=/usr/share/grafana/plugins-bundled 2024-09-21T13:50:11.125 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=plugin.store t=2024-09-21T13:50:10.923920394Z level=info msg="Plugins loaded" count=55 duration=40.969701ms 2024-09-21T13:50:11.125 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=query_data t=2024-09-21T13:50:10.925939317Z level=info msg="Query Service initialization" 2024-09-21T13:50:11.125 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=live.push_http t=2024-09-21T13:50:10.928276225Z level=info msg="Live Push Gateway initialization" 2024-09-21T13:50:11.125 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=ngalert.migration t=2024-09-21T13:50:10.929355432Z level=info msg=Starting 2024-09-21T13:50:11.125 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=ngalert.migration t=2024-09-21T13:50:10.929625478Z level=info msg="Applying transition" currentType=Legacy desiredType=UnifiedAlerting cleanOnDowngrade=false cleanOnUpgrade=false 2024-09-21T13:50:11.125 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=ngalert.migration orgID=1 t=2024-09-21T13:50:10.929893871Z level=info msg="Migrating alerts for organisation" 2024-09-21T13:50:11.125 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=ngalert.migration orgID=1 t=2024-09-21T13:50:10.930323461Z level=info msg="Alerts found to migrate" alerts=0 2024-09-21T13:50:11.125 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=ngalert.migration t=2024-09-21T13:50:10.931428424Z level=info msg="Completed alerting migration" 2024-09-21T13:50:11.125 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=ngalert.state.manager t=2024-09-21T13:50:10.942359049Z level=info msg="Running in alternative execution of Error/NoData mode" 2024-09-21T13:50:11.125 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=infra.usagestats.collector t=2024-09-21T13:50:10.943668311Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-09-21T13:50:11.126 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=provisioning.datasources t=2024-09-21T13:50:10.945232985Z level=info msg="inserting datasource from configuration" name=Dashboard1 uid=P43CA22E17D0F9596 2024-09-21T13:50:11.126 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=provisioning.datasources t=2024-09-21T13:50:10.957458743Z level=info msg="inserting datasource from configuration" name=Loki uid=P8E80F9AEF21F6940 2024-09-21T13:50:11.126 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=provisioning.alerting t=2024-09-21T13:50:10.97010171Z level=info msg="starting to provision alerting" 2024-09-21T13:50:11.126 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=provisioning.alerting t=2024-09-21T13:50:10.970112379Z level=info msg="finished to provision alerting" 2024-09-21T13:50:11.126 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=grafanaStorageLogger t=2024-09-21T13:50:10.970203491Z level=info msg="Storage starting" 2024-09-21T13:50:11.126 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=ngalert.state.manager t=2024-09-21T13:50:10.970385833Z level=info msg="Warming state cache for startup" 2024-09-21T13:50:11.126 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=ngalert.multiorg.alertmanager t=2024-09-21T13:50:10.971220772Z level=info msg="Starting MultiOrg Alertmanager" 2024-09-21T13:50:11.126 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=http.server t=2024-09-21T13:50:10.972914575Z level=info msg="HTTP Server TLS settings" MinTLSVersion=TLS1.2 configuredciphers=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA 2024-09-21T13:50:11.126 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=http.server t=2024-09-21T13:50:10.973210311Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-09-21T13:50:11.126 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=sqlstore.transactions t=2024-09-21T13:50:10.982265004Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-21T13:50:11.126 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:11 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=ngalert.state.manager t=2024-09-21T13:50:11.001517955Z level=info msg="State cache has been initialized" states=0 duration=31.131002ms 2024-09-21T13:50:11.126 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:11 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=ngalert.scheduler t=2024-09-21T13:50:11.001548507Z level=info msg="Starting scheduler" tickInterval=10s maxAttempts=1 2024-09-21T13:50:11.126 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:11 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=ticker t=2024-09-21T13:50:11.001595984Z level=info msg=starting first_tick=2024-09-21T13:50:20Z 2024-09-21T13:50:11.126 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:11 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=provisioning.dashboard t=2024-09-21T13:50:11.001764571Z level=info msg="starting to provision dashboards" 2024-09-21T13:50:11.127 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:11 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=sqlstore.transactions t=2024-09-21T13:50:11.005021983Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-21T13:50:11.127 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:11 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=sqlstore.transactions t=2024-09-21T13:50:11.016191539Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-09-21T13:50:11.127 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:11 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=sqlstore.transactions t=2024-09-21T13:50:11.027359819Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=2 code="database is locked" 2024-09-21T13:50:11.127 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:11 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=sqlstore.transactions t=2024-09-21T13:50:11.038064218Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=3 code="database is locked" 2024-09-21T13:50:11.127 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:11 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=grafana.update.checker t=2024-09-21T13:50:11.04253829Z level=info msg="Update check succeeded" duration=72.217496ms 2024-09-21T13:50:11.127 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:11 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=plugins.update.checker t=2024-09-21T13:50:11.044022153Z level=info msg="Update check succeeded" duration=73.012791ms 2024-09-21T13:50:11.127 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:11 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=sqlstore.transactions t=2024-09-21T13:50:11.048753699Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=4 code="database is locked" 2024-09-21T13:50:11.127 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:11 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=secret.migration t=2024-09-21T13:50:11.049625996Z level=error msg="Stopped secret migration service" service=*migrations.DataSourceSecretMigrationService reason="database is locked" 2024-09-21T13:50:11.127 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:11 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=sqlstore.transactions t=2024-09-21T13:50:11.06329714Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-21T13:50:11.127 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:11 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=sqlstore.transactions t=2024-09-21T13:50:11.12226237Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-21T13:50:11.180 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:50:11.202 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:11.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:11.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:50:11.255 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:11 smithi049 systemd[1]: Starting Ceph node-exporter.a for 40a3d2e8-781f-11ef-baf6-efdc52797490... 2024-09-21T13:50:11.304 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:50:11.326 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:11.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:11.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:50:11.428 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:50:11.444 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:11 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=provisioning.dashboard t=2024-09-21T13:50:11.131795097Z level=info msg="finished to provision dashboards" 2024-09-21T13:50:11.445 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:11 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=sqlstore.transactions t=2024-09-21T13:50:11.132786601Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-09-21T13:50:11.445 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:11 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=grafana-apiserver t=2024-09-21T13:50:11.369903867Z level=info msg="Adding GroupVersion playlist.grafana.app v0alpha1 to ResourceManager" 2024-09-21T13:50:11.445 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:50:11 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=grafana-apiserver t=2024-09-21T13:50:11.370290351Z level=info msg="Adding GroupVersion featuretoggle.grafana.app v0alpha1 to ResourceManager" 2024-09-21T13:50:11.450 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:11.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:11.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:50:11.551 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:11.573 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:11.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:11.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:11.674 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:11.695 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:11.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:11.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:50:11.797 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:50:11.820 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:11.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:11.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:50:11.922 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:50:11.946 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:11.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:11.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:50:12.029 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:11 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:50:12.029 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:11 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:50:12.029 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:11 smithi049 ceph-mon[30080]: osdmap e87: 8 total, 8 up, 8 in 2024-09-21T13:50:12.029 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:11 smithi049 ceph-mon[30080]: pgmap v37: 148 pgs: 148 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T13:50:12.029 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:11 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:50:12.029 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:11 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:50:12.029 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:11 smithi049 ceph-mon[40489]: osdmap e87: 8 total, 8 up, 8 in 2024-09-21T13:50:12.029 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:11 smithi049 ceph-mon[40489]: pgmap v37: 148 pgs: 148 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T13:50:12.031 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T13:50:12.031 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T13:50:12.031 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T13:50:12.045 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T13:50:12.045 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-09-21T13:50:12.045 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-09-21T13:50:12.045 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-09-21T13:50:12.045 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T13:50:12.047 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:50:12.074 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:12.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:12.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:50:12.176 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:12.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:11 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:50:12.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:11 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:50:12.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:11 smithi187 ceph-mon[33032]: osdmap e87: 8 total, 8 up, 8 in 2024-09-21T13:50:12.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:11 smithi187 ceph-mon[33032]: pgmap v37: 148 pgs: 148 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T13:50:12.199 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:12.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:12.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:50:12.301 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:12.324 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:12.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:12.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:50:12.354 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T13:50:12.425 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:50:12.448 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:12.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:12.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:50:12.550 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:12.574 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:12.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:12.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:12.676 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:50:12.700 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:12.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:12.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:50:12.716 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T13:50:12.741 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:50:12.803 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:50:12.809 DEBUG:teuthology.orchestra.run.smithi049:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-09-21T13:50:12.811 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-21T13:50:12.812 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-21T13:50:12.828 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:12.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:12.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:50:12.890 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:hints = 1 2024-09-21T13:50:12.890 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-21T13:50:12.890 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Object prefix: benchmark_data_smithi049_76798 2024-09-21T13:50:12.891 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T13:50:12.891 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 0 0 0 0 0 0 - 0 2024-09-21T13:50:12.929 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:50:12.934 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:12 smithi049 bash[76740]: Trying to pull quay.io/prometheus/node-exporter:v1.7.0... 2024-09-21T13:50:12.935 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:12 smithi049 ceph-mon[40489]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T13:50:12.935 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:12 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:12.935 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:12 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:12.935 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:12 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:50:12.936 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:12 smithi049 ceph-mon[30080]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T13:50:12.936 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:12 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:12.936 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:12 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:12.936 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:12 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:50:12.978 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:12.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:12.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:50:13.019 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T13:50:13.080 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:50:13.106 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:13.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:13.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:50:13.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:12 smithi187 ceph-mon[33032]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T13:50:13.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:12 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:13.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:12 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:13.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:12 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:50:13.207 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:50:13.222 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:13 smithi049 bash[76740]: Getting image source signatures 2024-09-21T13:50:13.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:13.263 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:13.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:50:13.364 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:50:13.389 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:13.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:13.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:50:13.471 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:50:13.475 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:50:13.490 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:13.505 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:13 smithi049 bash[76740]: Copying blob sha256:324153f2810a9927fcce320af9e4e291e0b6e805cbdd1f338386c756b9defa24 2024-09-21T13:50:13.505 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:13 smithi049 bash[76740]: Copying blob sha256:2abcce694348cd2c949c0e98a7400ebdfd8341021bcf6b541bc72033ce982510 2024-09-21T13:50:13.505 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:13 smithi049 bash[76740]: Copying blob sha256:455fd88e5221bc1e278ef2d059cd70e4df99a24e5af050ede621534276f6cf9a 2024-09-21T13:50:13.515 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:13.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:13.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:13.559 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:50:13.617 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:50:13.660 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:13.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:13.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:50:13.762 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:13.787 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:13.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:13.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:50:13.889 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:50:13.892 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 1 16 2582 2566 160.394 160.375 0.00431056 0.00620382 2024-09-21T13:50:13.927 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:13.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:13.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:50:13.944 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:13 smithi049 ceph-mon[30080]: pgmap v38: 148 pgs: 148 active+clean; 35 MiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 5.8 MiB/s wr, 94 op/s 2024-09-21T13:50:13.944 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:13 smithi049 ceph-mon[40489]: pgmap v38: 148 pgs: 148 active+clean; 35 MiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 5.8 MiB/s wr, 94 op/s 2024-09-21T13:50:14.029 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:50:14.057 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:14.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:14.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:50:14.095 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:13 smithi187 ceph-mon[33032]: pgmap v38: 148 pgs: 148 active+clean; 35 MiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 5.8 MiB/s wr, 94 op/s 2024-09-21T13:50:14.159 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:14.193 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:14.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:14.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:14.295 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:14.327 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:14.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:14.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:14.428 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:14.458 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:14.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:14.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:14.560 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:50:14.602 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:14.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:14.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:50:14.704 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:50:14.743 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:14.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:14.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:50:14.845 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:14.870 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:14.871 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-09-21T13:50:14.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:14.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:14.891 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 2 16 5247 5231 163.471 166.562 0.0202033 0.00609755 2024-09-21T13:50:14.960 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:50:14.962 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-09-21T13:50:14.973 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:15.000 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:15.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:15.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:15.102 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:15.133 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:15.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:15.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:50:15.158 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:14 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:50:15.158 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:14 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:50:15.158 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:14 smithi049 ceph-mon[40489]: osdmap e88: 8 total, 8 up, 8 in 2024-09-21T13:50:15.158 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:14 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2052226572' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:50:15.158 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:14 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:50:15.159 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:14 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:50:15.159 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:14 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:50:15.159 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:14 smithi049 ceph-mon[30080]: osdmap e88: 8 total, 8 up, 8 in 2024-09-21T13:50:15.159 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:14 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2052226572' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:50:15.159 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:14 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:50:15.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:14 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:50:15.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:14 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:50:15.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:14 smithi187 ceph-mon[33032]: osdmap e88: 8 total, 8 up, 8 in 2024-09-21T13:50:15.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:14 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2052226572' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:50:15.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:14 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:50:15.235 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:50:15.262 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:15.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:15.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:50:15.364 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:50:15.408 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:15.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:15.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:50:15.452 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:15 smithi049 bash[76740]: Copying config sha256:72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e 2024-09-21T13:50:15.510 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:15.541 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:15.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:15.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:15.642 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:50:15.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:15.686 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:15.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:50:15.754 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:15 smithi049 bash[76740]: Writing manifest to image destination 2024-09-21T13:50:15.787 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:50:15.828 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:15.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:15.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:50:15.892 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 3 16 7824 7808 162.665 161.062 0.00704709 0.00613335 2024-09-21T13:50:15.930 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:50:15.970 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:15.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:15.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:50:16.072 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:16.101 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:16.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:16.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:50:16.165 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:15 smithi049 ceph-mon[40489]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:50:16.165 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:15 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T13:50:16.165 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:15 smithi049 ceph-mon[40489]: osdmap e89: 8 total, 8 up, 8 in 2024-09-21T13:50:16.165 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:15 smithi049 ceph-mon[40489]: from='client.24670 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T13:50:16.166 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:15 smithi049 ceph-mon[40489]: pgmap v41: 148 pgs: 148 active+clean; 79 MiB data, 332 MiB used, 715 GiB / 715 GiB avail; 865 B/s rd, 13 MiB/s wr, 211 op/s 2024-09-21T13:50:16.166 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:15 smithi049 ceph-mon[30080]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:50:16.166 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:15 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T13:50:16.166 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:15 smithi049 ceph-mon[30080]: osdmap e89: 8 total, 8 up, 8 in 2024-09-21T13:50:16.167 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:15 smithi049 ceph-mon[30080]: from='client.24670 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T13:50:16.167 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:15 smithi049 ceph-mon[30080]: pgmap v41: 148 pgs: 148 active+clean; 79 MiB data, 332 MiB used, 715 GiB / 715 GiB avail; 865 B/s rd, 13 MiB/s wr, 211 op/s 2024-09-21T13:50:16.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:15 smithi187 ceph-mon[33032]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:50:16.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:15 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T13:50:16.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:15 smithi187 ceph-mon[33032]: osdmap e89: 8 total, 8 up, 8 in 2024-09-21T13:50:16.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:15 smithi187 ceph-mon[33032]: from='client.24670 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T13:50:16.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:15 smithi187 ceph-mon[33032]: pgmap v41: 148 pgs: 148 active+clean; 79 MiB data, 332 MiB used, 715 GiB / 715 GiB avail; 865 B/s rd, 13 MiB/s wr, 211 op/s 2024-09-21T13:50:16.203 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:16.230 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:16.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:16.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:16.331 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:16.359 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:16.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:16.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:16.392 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T13:50:16.393 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-09-21T13:50:16.393 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-09-21T13:50:16.393 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-09-21T13:50:16.393 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.393 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:2421 7.4 7:2ff03226:::benchmark_data_smithi049_76798_object2420:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-09-21T13:50:16.393 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:13.826562+0000", 2024-09-21T13:50:16.393 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 2.561112654, 2024-09-21T13:50:16.393 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.014884399, 2024-09-21T13:50:16.393 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:50:16.393 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:50:16.393 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:50:16.393 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:50:16.393 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:50:16.394 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:50:16.394 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 2421 2024-09-21T13:50:16.394 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.394 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:50:16.394 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.394 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:50:16.394 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:13.826562+0000", 2024-09-21T13:50:16.394 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:16.394 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.394 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.394 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:50:16.394 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:13.826562+0000", 2024-09-21T13:50:16.395 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:16.395 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.395 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.395 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:50:16.395 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:13.826562+0000", 2024-09-21T13:50:16.395 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9100000000000004e-07 2024-09-21T13:50:16.395 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.395 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.395 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:50:16.395 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:13.826585+0000", 2024-09-21T13:50:16.395 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3130999999999999e-05 2024-09-21T13:50:16.395 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.395 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.396 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:50:16.396 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:13.826594+0000", 2024-09-21T13:50:16.396 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.7450000000000001e-06 2024-09-21T13:50:16.396 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.396 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.396 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:50:16.396 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:13.826601+0000", 2024-09-21T13:50:16.396 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3860000000000004e-06 2024-09-21T13:50:16.396 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.396 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.396 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:50:16.396 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:13.831994+0000", 2024-09-21T13:50:16.396 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0053934889999999996 2024-09-21T13:50:16.396 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.397 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.397 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:50:16.397 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:13.832202+0000", 2024-09-21T13:50:16.397 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00020829600000000001 2024-09-21T13:50:16.397 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.397 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.397 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,4", 2024-09-21T13:50:16.397 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:13.832342+0000", 2024-09-21T13:50:16.397 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013968100000000001 2024-09-21T13:50:16.397 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.397 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.397 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:50:16.397 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:13.834667+0000", 2024-09-21T13:50:16.398 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002324946 2024-09-21T13:50:16.398 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.398 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.398 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:16.398 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:13.837472+0000", 2024-09-21T13:50:16.398 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002804997 2024-09-21T13:50:16.398 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.398 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.398 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:16.398 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:13.841356+0000", 2024-09-21T13:50:16.398 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0038838050000000002 2024-09-21T13:50:16.398 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.399 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.399 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:50:16.399 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:13.841408+0000", 2024-09-21T13:50:16.399 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2139999999999999e-05 2024-09-21T13:50:16.399 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.399 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.399 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:50:16.399 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:13.841446+0000", 2024-09-21T13:50:16.399 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8192000000000003e-05 2024-09-21T13:50:16.399 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:16.399 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:50:16.399 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:16.399 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.399 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.400 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:2448 7.f 7:f951d7f7:::benchmark_data_smithi049_76798_object2447:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-09-21T13:50:16.400 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:13.838607+0000", 2024-09-21T13:50:16.400 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 2.5490675710000001, 2024-09-21T13:50:16.400 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.014543553000000001, 2024-09-21T13:50:16.400 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:50:16.400 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:50:16.400 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:50:16.400 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:50:16.400 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:50:16.400 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:50:16.400 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 2448 2024-09-21T13:50:16.400 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.400 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:50:16.400 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.401 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:50:16.401 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:13.838607+0000", 2024-09-21T13:50:16.401 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:16.401 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.401 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.401 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:50:16.401 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:13.838607+0000", 2024-09-21T13:50:16.401 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:16.401 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.401 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.401 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:50:16.401 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:13.838608+0000", 2024-09-21T13:50:16.401 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1489999999999999e-06 2024-09-21T13:50:16.401 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.401 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.402 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:50:16.402 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:13.838647+0000", 2024-09-21T13:50:16.402 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8807000000000003e-05 2024-09-21T13:50:16.402 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.402 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.402 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:50:16.402 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:13.838676+0000", 2024-09-21T13:50:16.402 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9186000000000001e-05 2024-09-21T13:50:16.402 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.402 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.402 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:50:16.402 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:13.838691+0000", 2024-09-21T13:50:16.402 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5242000000000001e-05 2024-09-21T13:50:16.402 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.403 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.403 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:50:16.403 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:13.842144+0000", 2024-09-21T13:50:16.403 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0034528900000000001 2024-09-21T13:50:16.403 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.403 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.403 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:50:16.403 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:13.842239+0000", 2024-09-21T13:50:16.403 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4537999999999995e-05 2024-09-21T13:50:16.403 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.403 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.403 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,7", 2024-09-21T13:50:16.403 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:13.842376+0000", 2024-09-21T13:50:16.403 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013756799999999999 2024-09-21T13:50:16.404 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.404 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.404 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:50:16.404 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:13.849972+0000", 2024-09-21T13:50:16.404 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0075958570000000001 2024-09-21T13:50:16.404 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.404 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.404 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:16.404 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:13.850360+0000", 2024-09-21T13:50:16.404 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00038828400000000002 2024-09-21T13:50:16.404 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.404 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.404 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:16.404 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:13.853056+0000", 2024-09-21T13:50:16.404 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0026954140000000001 2024-09-21T13:50:16.405 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.405 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.405 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:50:16.405 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:13.853105+0000", 2024-09-21T13:50:16.405 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9619e-05 2024-09-21T13:50:16.405 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.405 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.405 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:50:16.405 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:13.853150+0000", 2024-09-21T13:50:16.405 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4999000000000001e-05 2024-09-21T13:50:16.405 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:16.405 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:50:16.405 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:16.405 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.406 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.406 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:2822 7.6 7:66e4e52a:::benchmark_data_smithi049_76798_object2821:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-21T13:50:16.406 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:13.985640+0000", 2024-09-21T13:50:16.406 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 2.4020341350000001, 2024-09-21T13:50:16.406 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.013987691, 2024-09-21T13:50:16.406 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:50:16.406 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:50:16.406 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:50:16.406 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:50:16.406 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:50:16.406 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:50:16.406 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 2822 2024-09-21T13:50:16.406 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.407 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:50:16.407 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.407 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:50:16.407 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:13.985640+0000", 2024-09-21T13:50:16.407 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:16.407 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.407 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.407 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:50:16.407 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:13.985640+0000", 2024-09-21T13:50:16.407 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:16.407 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.407 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.407 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:50:16.407 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:13.985641+0000", 2024-09-21T13:50:16.408 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0500000000000004e-07 2024-09-21T13:50:16.408 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.408 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.408 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:50:16.408 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:13.985662+0000", 2024-09-21T13:50:16.408 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0740000000000001e-05 2024-09-21T13:50:16.408 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.408 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.408 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:50:16.408 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:13.985669+0000", 2024-09-21T13:50:16.408 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0339999999999999e-06 2024-09-21T13:50:16.408 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.408 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.409 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:50:16.409 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:13.985674+0000", 2024-09-21T13:50:16.409 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3959999999999997e-06 2024-09-21T13:50:16.409 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.409 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.409 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:50:16.409 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:13.986148+0000", 2024-09-21T13:50:16.409 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00047410399999999999 2024-09-21T13:50:16.409 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.409 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.409 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:50:16.409 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:13.986362+0000", 2024-09-21T13:50:16.409 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00021347200000000001 2024-09-21T13:50:16.409 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.410 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.410 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,7", 2024-09-21T13:50:16.410 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:13.986498+0000", 2024-09-21T13:50:16.410 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000136367 2024-09-21T13:50:16.410 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.410 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.410 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:50:16.410 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:13.988603+0000", 2024-09-21T13:50:16.410 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0021049480000000001 2024-09-21T13:50:16.410 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.410 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.410 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:16.411 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:13.989069+0000", 2024-09-21T13:50:16.411 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00046623999999999998 2024-09-21T13:50:16.411 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.411 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.411 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:16.411 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:13.999500+0000", 2024-09-21T13:50:16.411 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.010430527 2024-09-21T13:50:16.411 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.411 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.411 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:50:16.411 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:13.999578+0000", 2024-09-21T13:50:16.411 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8168000000000001e-05 2024-09-21T13:50:16.411 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.411 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.412 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:50:16.412 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:13.999628+0000", 2024-09-21T13:50:16.412 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0189999999999999e-05 2024-09-21T13:50:16.412 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:16.412 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:50:16.412 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:16.412 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.412 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.412 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:2824 7.6 7:609832fb:::benchmark_data_smithi049_76798_object2823:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-21T13:50:16.412 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:13.985688+0000", 2024-09-21T13:50:16.412 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 2.4019868099999999, 2024-09-21T13:50:16.412 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.014549783, 2024-09-21T13:50:16.412 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:50:16.412 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:50:16.412 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:50:16.413 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:50:16.413 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:50:16.413 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:50:16.413 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 2824 2024-09-21T13:50:16.413 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.413 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:50:16.413 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.413 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:50:16.413 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:13.985688+0000", 2024-09-21T13:50:16.413 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:16.413 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.413 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.413 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:50:16.413 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:13.985688+0000", 2024-09-21T13:50:16.413 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:16.414 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.414 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.414 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:50:16.414 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:13.985688+0000", 2024-09-21T13:50:16.414 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1699999999999998e-07 2024-09-21T13:50:16.414 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.414 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.414 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:50:16.414 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:13.985710+0000", 2024-09-21T13:50:16.414 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2110999999999999e-05 2024-09-21T13:50:16.414 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.414 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.414 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:50:16.414 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:13.985718+0000", 2024-09-21T13:50:16.415 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8369999999999997e-06 2024-09-21T13:50:16.415 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.415 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.415 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:50:16.415 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:13.985723+0000", 2024-09-21T13:50:16.415 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.242e-06 2024-09-21T13:50:16.415 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.415 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.415 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:50:16.415 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:13.986808+0000", 2024-09-21T13:50:16.415 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0010846289999999999 2024-09-21T13:50:16.415 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.415 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.415 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:50:16.416 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:13.986900+0000", 2024-09-21T13:50:16.416 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2535999999999996e-05 2024-09-21T13:50:16.416 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.416 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.416 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,7", 2024-09-21T13:50:16.416 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:13.987075+0000", 2024-09-21T13:50:16.416 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00017442799999999999 2024-09-21T13:50:16.416 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.416 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.416 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:50:16.416 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:13.988610+0000", 2024-09-21T13:50:16.416 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0015346489999999999 2024-09-21T13:50:16.416 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.416 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.417 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:16.417 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:13.990791+0000", 2024-09-21T13:50:16.417 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0021814339999999999 2024-09-21T13:50:16.417 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.417 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.417 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:16.417 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.000150+0000", 2024-09-21T13:50:16.417 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0093593719999999995 2024-09-21T13:50:16.417 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.417 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.417 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:50:16.417 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.000199+0000", 2024-09-21T13:50:16.417 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8384000000000003e-05 2024-09-21T13:50:16.417 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.418 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.418 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:50:16.418 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.000237+0000", 2024-09-21T13:50:16.418 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8643999999999998e-05 2024-09-21T13:50:16.418 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:16.418 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:50:16.418 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:16.418 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.418 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.418 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:4978 7.6 7:63a56d51:::benchmark_data_smithi049_76798_object4977:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-21T13:50:16.418 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:14.746665+0000", 2024-09-21T13:50:16.418 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 1.64100925, 2024-09-21T13:50:16.418 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.013430899, 2024-09-21T13:50:16.418 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:50:16.419 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:50:16.419 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:50:16.419 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:50:16.419 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:50:16.419 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:50:16.419 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 4978 2024-09-21T13:50:16.419 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.419 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:50:16.419 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.419 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:50:16.419 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.746665+0000", 2024-09-21T13:50:16.419 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:16.419 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.419 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.419 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:50:16.420 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.746665+0000", 2024-09-21T13:50:16.420 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:16.420 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.420 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.420 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:50:16.421 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.746666+0000", 2024-09-21T13:50:16.421 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.5600000000000004e-07 2024-09-21T13:50:16.421 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.421 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.421 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:50:16.421 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.746696+0000", 2024-09-21T13:50:16.421 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9929000000000001e-05 2024-09-21T13:50:16.421 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.421 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.421 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:50:16.421 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.746708+0000", 2024-09-21T13:50:16.421 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2427e-05 2024-09-21T13:50:16.421 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.421 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.422 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:50:16.422 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.746715+0000", 2024-09-21T13:50:16.422 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0900000000000001e-06 2024-09-21T13:50:16.422 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.422 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.422 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:50:16.422 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.749020+0000", 2024-09-21T13:50:16.422 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00230554 2024-09-21T13:50:16.422 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.422 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.422 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:50:16.422 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.749136+0000", 2024-09-21T13:50:16.422 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000115605 2024-09-21T13:50:16.422 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.423 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.423 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,7", 2024-09-21T13:50:16.423 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.751144+0000", 2024-09-21T13:50:16.423 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002008134 2024-09-21T13:50:16.423 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.423 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.423 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:50:16.423 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.752721+0000", 2024-09-21T13:50:16.423 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0015767680000000001 2024-09-21T13:50:16.423 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.423 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.423 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:16.423 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.754793+0000", 2024-09-21T13:50:16.423 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0020720970000000002 2024-09-21T13:50:16.423 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.424 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.424 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:16.424 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.759993+0000", 2024-09-21T13:50:16.424 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0052007939999999999 2024-09-21T13:50:16.424 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.424 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.424 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:50:16.424 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.760051+0000", 2024-09-21T13:50:16.424 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7738000000000002e-05 2024-09-21T13:50:16.424 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.424 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.424 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:50:16.424 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.760096+0000", 2024-09-21T13:50:16.424 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4821000000000003e-05 2024-09-21T13:50:16.424 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:16.424 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:50:16.425 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:16.425 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.425 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.425 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:5053 7.4 7:2ecc906f:::benchmark_data_smithi049_76798_object5052:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-21T13:50:16.425 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:14.777968+0000", 2024-09-21T13:50:16.425 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 1.609706388, 2024-09-21T13:50:16.425 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.015635079999999999, 2024-09-21T13:50:16.425 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:50:16.425 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:50:16.425 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:50:16.425 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:50:16.425 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:50:16.425 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:50:16.425 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 5053 2024-09-21T13:50:16.425 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.426 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:50:16.426 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.426 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:50:16.426 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.777968+0000", 2024-09-21T13:50:16.426 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:16.426 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.426 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.426 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:50:16.426 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.777968+0000", 2024-09-21T13:50:16.426 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:16.426 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.426 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.426 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:50:16.426 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.777969+0000", 2024-09-21T13:50:16.427 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1399999999999997e-07 2024-09-21T13:50:16.427 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.427 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.427 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:50:16.427 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.777994+0000", 2024-09-21T13:50:16.427 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5749e-05 2024-09-21T13:50:16.427 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.427 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.427 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:50:16.427 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.778003+0000", 2024-09-21T13:50:16.427 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.8890000000000008e-06 2024-09-21T13:50:16.427 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.427 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.427 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:50:16.428 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.778009+0000", 2024-09-21T13:50:16.428 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9699999999999996e-06 2024-09-21T13:50:16.428 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.428 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.428 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:50:16.428 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.780751+0000", 2024-09-21T13:50:16.428 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0027415 2024-09-21T13:50:16.428 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.428 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.428 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:50:16.428 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.780914+0000", 2024-09-21T13:50:16.428 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000162812 2024-09-21T13:50:16.428 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.428 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.428 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,4", 2024-09-21T13:50:16.429 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.781134+0000", 2024-09-21T13:50:16.429 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00022019100000000001 2024-09-21T13:50:16.429 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.429 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.429 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:50:16.429 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.783375+0000", 2024-09-21T13:50:16.429 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002240901 2024-09-21T13:50:16.429 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.429 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.429 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:16.429 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.783598+0000", 2024-09-21T13:50:16.429 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000222885 2024-09-21T13:50:16.429 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.429 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.429 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:16.430 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.793511+0000", 2024-09-21T13:50:16.430 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0099136610000000007 2024-09-21T13:50:16.430 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.430 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.430 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:50:16.430 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.793564+0000", 2024-09-21T13:50:16.430 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2565000000000002e-05 2024-09-21T13:50:16.430 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.430 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.430 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:50:16.430 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.793603+0000", 2024-09-21T13:50:16.430 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9342999999999999e-05 2024-09-21T13:50:16.430 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:16.430 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:50:16.431 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:16.431 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.431 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.431 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:5169 7.f 7:fdc9d14c:::benchmark_data_smithi049_76798_object5168:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-21T13:50:16.431 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:14.833116+0000", 2024-09-21T13:50:16.431 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 1.5545580079999999, 2024-09-21T13:50:16.431 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.014252450999999999, 2024-09-21T13:50:16.431 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:50:16.431 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:50:16.431 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:50:16.431 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:50:16.431 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:50:16.431 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:50:16.431 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 5169 2024-09-21T13:50:16.432 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.432 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:50:16.432 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.432 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:50:16.432 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.833116+0000", 2024-09-21T13:50:16.432 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:16.432 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.432 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.432 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:50:16.432 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.833116+0000", 2024-09-21T13:50:16.432 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:16.432 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.432 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.432 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:50:16.432 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.833117+0000", 2024-09-21T13:50:16.432 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2699999999999999e-07 2024-09-21T13:50:16.433 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.433 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.433 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:50:16.433 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.833200+0000", 2024-09-21T13:50:16.433 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2929999999999994e-05 2024-09-21T13:50:16.433 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.433 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.433 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:50:16.433 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.833210+0000", 2024-09-21T13:50:16.433 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8369999999999992e-06 2024-09-21T13:50:16.433 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.433 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.433 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:50:16.433 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.833217+0000", 2024-09-21T13:50:16.433 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0740000000000004e-06 2024-09-21T13:50:16.433 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.434 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.434 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:50:16.434 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.834023+0000", 2024-09-21T13:50:16.434 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00080644500000000004 2024-09-21T13:50:16.434 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.434 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.434 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:50:16.434 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.834138+0000", 2024-09-21T13:50:16.434 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000115096 2024-09-21T13:50:16.434 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.434 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.434 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,7", 2024-09-21T13:50:16.434 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.834263+0000", 2024-09-21T13:50:16.434 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012479 2024-09-21T13:50:16.435 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.435 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.435 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:16.435 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.840665+0000", 2024-09-21T13:50:16.435 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0064020819999999999 2024-09-21T13:50:16.435 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.435 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.435 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:50:16.435 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.840690+0000", 2024-09-21T13:50:16.435 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4722000000000001e-05 2024-09-21T13:50:16.435 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.435 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.435 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:16.435 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.847264+0000", 2024-09-21T13:50:16.436 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0065735979999999999 2024-09-21T13:50:16.436 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.436 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.436 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:50:16.436 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.847325+0000", 2024-09-21T13:50:16.436 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0948e-05 2024-09-21T13:50:16.436 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.436 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.436 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:50:16.436 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.847369+0000", 2024-09-21T13:50:16.436 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4301999999999997e-05 2024-09-21T13:50:16.436 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:16.436 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:50:16.436 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:16.436 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.437 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.437 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.14817.0:5164 7.e e88/83 7:7bf71712:::benchmark_data_smithi049_76798_object5163:head v 88'344, mlcod=88'342)", 2024-09-21T13:50:16.437 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:14.834631+0000", 2024-09-21T13:50:16.437 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 1.5530438420000001, 2024-09-21T13:50:16.437 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.014926050999999999, 2024-09-21T13:50:16.437 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:50:16.437 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:50:16.437 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:50:16.437 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:50:16.437 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.437 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:50:16.437 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.834631+0000", 2024-09-21T13:50:16.437 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:16.437 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.438 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.438 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:50:16.438 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.834631+0000", 2024-09-21T13:50:16.438 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:16.438 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.438 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.438 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:50:16.438 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.834632+0000", 2024-09-21T13:50:16.438 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4600000000000003e-07 2024-09-21T13:50:16.438 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.438 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.438 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:50:16.438 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.834658+0000", 2024-09-21T13:50:16.438 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6174999999999999e-05 2024-09-21T13:50:16.439 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.439 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.439 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:50:16.439 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.834672+0000", 2024-09-21T13:50:16.439 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4808e-05 2024-09-21T13:50:16.439 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.439 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.439 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:50:16.439 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.834680+0000", 2024-09-21T13:50:16.439 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2849999999999997e-06 2024-09-21T13:50:16.439 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.439 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.439 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:50:16.440 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.835196+0000", 2024-09-21T13:50:16.440 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000515727 2024-09-21T13:50:16.440 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.440 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.440 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:50:16.440 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.835214+0000", 2024-09-21T13:50:16.440 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8484000000000001e-05 2024-09-21T13:50:16.440 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.440 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.440 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:50:16.440 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.849498+0000", 2024-09-21T13:50:16.440 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.014284336 2024-09-21T13:50:16.440 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.440 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.440 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:50:16.441 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.849557+0000", 2024-09-21T13:50:16.441 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8289999999999999e-05 2024-09-21T13:50:16.441 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:16.441 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:50:16.441 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:16.441 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.441 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.441 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:5174 7.4 7:216d9b72:::benchmark_data_smithi049_76798_object5173:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-21T13:50:16.441 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:14.838006+0000", 2024-09-21T13:50:16.441 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 1.5496687090000001, 2024-09-21T13:50:16.441 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.015932735, 2024-09-21T13:50:16.441 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:50:16.441 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:50:16.442 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:50:16.442 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:50:16.442 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:50:16.442 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:50:16.442 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 5174 2024-09-21T13:50:16.442 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.442 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:50:16.442 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.442 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:50:16.442 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.838006+0000", 2024-09-21T13:50:16.442 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:16.442 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.442 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.443 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:50:16.443 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.838006+0000", 2024-09-21T13:50:16.443 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:16.443 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.443 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.443 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:50:16.443 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.838006+0000", 2024-09-21T13:50:16.443 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3499999999999995e-07 2024-09-21T13:50:16.443 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.443 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.443 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:50:16.443 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.838034+0000", 2024-09-21T13:50:16.443 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.8055000000000002e-05 2024-09-21T13:50:16.443 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.444 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.444 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:50:16.444 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.838043+0000", 2024-09-21T13:50:16.444 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2039999999999997e-06 2024-09-21T13:50:16.444 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.444 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.444 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:50:16.444 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.838049+0000", 2024-09-21T13:50:16.444 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2099999999999998e-06 2024-09-21T13:50:16.444 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.444 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.444 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:50:16.444 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.838783+0000", 2024-09-21T13:50:16.444 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00073406000000000003 2024-09-21T13:50:16.445 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.445 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.445 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:50:16.445 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.838889+0000", 2024-09-21T13:50:16.445 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000106449 2024-09-21T13:50:16.445 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.445 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.445 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,4", 2024-09-21T13:50:16.445 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.840016+0000", 2024-09-21T13:50:16.445 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001126298 2024-09-21T13:50:16.445 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.445 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.445 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:16.445 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.846445+0000", 2024-09-21T13:50:16.445 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0064294410000000001 2024-09-21T13:50:16.445 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.446 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.446 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:50:16.446 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.848413+0000", 2024-09-21T13:50:16.446 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0019680790000000002 2024-09-21T13:50:16.446 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.446 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.446 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:16.446 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.853832+0000", 2024-09-21T13:50:16.446 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0054186060000000003 2024-09-21T13:50:16.446 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.446 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.446 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:50:16.446 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.853893+0000", 2024-09-21T13:50:16.446 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0809e-05 2024-09-21T13:50:16.446 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.447 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.447 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:16 smithi049 podman[76740]: 2024-09-21 13:50:16.113273201 +0000 UTC m=+3.381978457 image pull 72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e quay.io/prometheus/node-exporter:v1.7.0 2024-09-21T13:50:16.447 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:50:16.447 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.853938+0000", 2024-09-21T13:50:16.447 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5788999999999998e-05 2024-09-21T13:50:16.447 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:16.448 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:50:16.448 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:16.448 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.448 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.448 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:5194 7.f 7:f9b446b8:::benchmark_data_smithi049_76798_object5193:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-21T13:50:16.448 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:14.856066+0000", 2024-09-21T13:50:16.448 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 1.5316086339999999, 2024-09-21T13:50:16.448 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.016346387, 2024-09-21T13:50:16.448 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:50:16.448 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:50:16.448 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:50:16.448 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:50:16.448 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:50:16.448 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:50:16.449 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 5194 2024-09-21T13:50:16.449 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.449 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:50:16.449 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.449 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:50:16.449 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.856066+0000", 2024-09-21T13:50:16.449 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:16.449 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.449 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.449 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:50:16.449 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.856066+0000", 2024-09-21T13:50:16.449 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:16.449 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.449 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.449 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:50:16.450 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.856067+0000", 2024-09-21T13:50:16.450 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.308e-06 2024-09-21T13:50:16.450 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.450 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.450 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:50:16.450 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.856090+0000", 2024-09-21T13:50:16.450 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3286e-05 2024-09-21T13:50:16.450 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.450 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.450 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:50:16.450 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.856102+0000", 2024-09-21T13:50:16.450 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1218e-05 2024-09-21T13:50:16.450 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.450 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.451 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:50:16.451 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.856109+0000", 2024-09-21T13:50:16.451 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.097e-06 2024-09-21T13:50:16.451 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.451 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.451 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:50:16.451 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.856159+0000", 2024-09-21T13:50:16.451 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0491000000000002e-05 2024-09-21T13:50:16.451 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.451 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.451 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:50:16.451 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.856303+0000", 2024-09-21T13:50:16.451 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000143616 2024-09-21T13:50:16.452 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.452 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.452 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,7", 2024-09-21T13:50:16.452 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.856450+0000", 2024-09-21T13:50:16.452 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014688200000000001 2024-09-21T13:50:16.452 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.452 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.452 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:16.452 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.857819+0000", 2024-09-21T13:50:16.452 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001369739 2024-09-21T13:50:16.452 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.452 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.452 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:50:16.452 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.860162+0000", 2024-09-21T13:50:16.453 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0023424000000000001 2024-09-21T13:50:16.453 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.453 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.453 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:16.453 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.872312+0000", 2024-09-21T13:50:16.453 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.012150414 2024-09-21T13:50:16.453 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.453 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.453 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:50:16.453 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.872366+0000", 2024-09-21T13:50:16.453 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4208e-05 2024-09-21T13:50:16.453 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.453 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.454 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:50:16.454 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.872412+0000", 2024-09-21T13:50:16.454 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5728e-05 2024-09-21T13:50:16.454 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:16.454 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:50:16.454 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:16.454 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.454 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.454 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:5199 7.f 7:f47922f0:::benchmark_data_smithi049_76798_object5198:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-21T13:50:16.454 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:14.857076+0000", 2024-09-21T13:50:16.454 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 1.530598398, 2024-09-21T13:50:16.454 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.021309979, 2024-09-21T13:50:16.454 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:50:16.455 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:50:16.455 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:50:16.455 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:50:16.455 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:50:16.455 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:50:16.455 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 5199 2024-09-21T13:50:16.455 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.455 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:50:16.455 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.455 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:50:16.455 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.857076+0000", 2024-09-21T13:50:16.455 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:16.455 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.456 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.456 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:50:16.456 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.857076+0000", 2024-09-21T13:50:16.456 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:16.456 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.456 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.456 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:50:16.456 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.857077+0000", 2024-09-21T13:50:16.456 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7700000000000004e-07 2024-09-21T13:50:16.456 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.456 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.456 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:50:16.456 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.857098+0000", 2024-09-21T13:50:16.456 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1262000000000001e-05 2024-09-21T13:50:16.456 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.457 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.457 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:50:16.457 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.857107+0000", 2024-09-21T13:50:16.457 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9320000000000002e-06 2024-09-21T13:50:16.457 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.457 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.457 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:50:16.457 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.857113+0000", 2024-09-21T13:50:16.457 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8590000000000001e-06 2024-09-21T13:50:16.457 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.457 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.457 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:50:16.457 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.857219+0000", 2024-09-21T13:50:16.458 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000106505 2024-09-21T13:50:16.458 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.458 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.458 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:50:16.458 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.857328+0000", 2024-09-21T13:50:16.458 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000109338 2024-09-21T13:50:16.458 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.458 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.458 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,7", 2024-09-21T13:50:16.458 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.857461+0000", 2024-09-21T13:50:16.458 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013234 2024-09-21T13:50:16.458 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.458 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.459 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:50:16.459 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.860167+0000", 2024-09-21T13:50:16.459 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0027061049999999999 2024-09-21T13:50:16.459 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.459 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.459 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:16.459 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.860411+0000", 2024-09-21T13:50:16.459 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00024416299999999999 2024-09-21T13:50:16.459 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.459 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.459 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:16.459 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.878285+0000", 2024-09-21T13:50:16.459 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.017874054 2024-09-21T13:50:16.460 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.460 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.460 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:50:16.460 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.878350+0000", 2024-09-21T13:50:16.460 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5006999999999996e-05 2024-09-21T13:50:16.460 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.460 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.460 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:50:16.460 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.878386+0000", 2024-09-21T13:50:16.460 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5837000000000003e-05 2024-09-21T13:50:16.460 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:16.460 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:50:16.460 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:16.460 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.460 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.461 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:5201 7.6 7:6f874d6d:::benchmark_data_smithi049_76798_object5200:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-09-21T13:50:16.461 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:14.857914+0000", 2024-09-21T13:50:16.461 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 1.529760209, 2024-09-21T13:50:16.461 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.020448977, 2024-09-21T13:50:16.461 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:50:16.461 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:50:16.461 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:50:16.461 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:50:16.461 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:50:16.461 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:50:16.461 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 5201 2024-09-21T13:50:16.461 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.461 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:50:16.461 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.462 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:50:16.462 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.857914+0000", 2024-09-21T13:50:16.462 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:16.462 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.462 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.462 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:50:16.462 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.857914+0000", 2024-09-21T13:50:16.462 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:16.462 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.462 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.462 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:50:16.462 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.857915+0000", 2024-09-21T13:50:16.462 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.68e-07 2024-09-21T13:50:16.463 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.463 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.463 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:50:16.463 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.857936+0000", 2024-09-21T13:50:16.463 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.109e-05 2024-09-21T13:50:16.463 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.463 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.463 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:50:16.463 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.857944+0000", 2024-09-21T13:50:16.463 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0779999999999996e-06 2024-09-21T13:50:16.463 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.463 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.463 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:50:16.464 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.857950+0000", 2024-09-21T13:50:16.464 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6710000000000004e-06 2024-09-21T13:50:16.464 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.464 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.464 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:50:16.464 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.858006+0000", 2024-09-21T13:50:16.464 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6280999999999999e-05 2024-09-21T13:50:16.464 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.464 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.464 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:50:16.464 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.858842+0000", 2024-09-21T13:50:16.464 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00083596300000000001 2024-09-21T13:50:16.464 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.464 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.465 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,7", 2024-09-21T13:50:16.465 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.858968+0000", 2024-09-21T13:50:16.465 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012573700000000001 2024-09-21T13:50:16.465 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.465 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.465 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:50:16.465 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.860144+0000", 2024-09-21T13:50:16.465 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001176809 2024-09-21T13:50:16.465 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.465 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.465 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:16.465 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.863333+0000", 2024-09-21T13:50:16.465 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0031890809999999999 2024-09-21T13:50:16.465 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.465 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.466 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:16.466 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.878267+0000", 2024-09-21T13:50:16.466 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.014933533000000001 2024-09-21T13:50:16.466 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.466 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.466 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:50:16.466 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.878321+0000", 2024-09-21T13:50:16.466 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4447999999999998e-05 2024-09-21T13:50:16.466 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.466 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.466 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:50:16.466 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:14.878363+0000", 2024-09-21T13:50:16.466 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1718000000000003e-05 2024-09-21T13:50:16.467 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:16.467 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:50:16.467 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:16.467 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.467 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.467 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:5577 7.f 7:f658a734:::benchmark_data_smithi049_76798_object5576:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-21T13:50:16.467 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:15.052766+0000", 2024-09-21T13:50:16.467 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 1.334908327, 2024-09-21T13:50:16.467 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.01436634, 2024-09-21T13:50:16.467 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:50:16.467 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:50:16.467 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:50:16.467 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:50:16.468 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:50:16.468 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:50:16.468 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 5577 2024-09-21T13:50:16.468 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.468 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:50:16.468 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.468 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:50:16.468 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.052766+0000", 2024-09-21T13:50:16.468 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:16.468 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.468 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.468 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:50:16.468 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.052766+0000", 2024-09-21T13:50:16.468 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:16.469 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.469 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.469 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:50:16.469 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.052768+0000", 2024-09-21T13:50:16.469 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6139999999999999e-06 2024-09-21T13:50:16.469 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.469 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.469 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:50:16.469 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.052799+0000", 2024-09-21T13:50:16.469 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1794000000000001e-05 2024-09-21T13:50:16.469 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.469 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.469 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:50:16.469 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.052816+0000", 2024-09-21T13:50:16.469 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6402999999999999e-05 2024-09-21T13:50:16.470 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.470 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.470 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:50:16.470 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.052825+0000", 2024-09-21T13:50:16.470 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4769999999999992e-06 2024-09-21T13:50:16.470 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.470 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.470 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:50:16.470 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.053192+0000", 2024-09-21T13:50:16.470 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00036642499999999998 2024-09-21T13:50:16.470 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.470 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.470 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:50:16.470 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.053324+0000", 2024-09-21T13:50:16.471 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000131784 2024-09-21T13:50:16.471 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.471 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.471 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,7", 2024-09-21T13:50:16.471 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.053455+0000", 2024-09-21T13:50:16.471 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013097599999999999 2024-09-21T13:50:16.471 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.471 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.471 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:16.471 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.055211+0000", 2024-09-21T13:50:16.471 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001756884 2024-09-21T13:50:16.471 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.471 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.471 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:50:16.471 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.055508+0000", 2024-09-21T13:50:16.472 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00029648100000000002 2024-09-21T13:50:16.472 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.472 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.472 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:16.472 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.065201+0000", 2024-09-21T13:50:16.472 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0096933149999999992 2024-09-21T13:50:16.472 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.472 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.472 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:50:16.472 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.067094+0000", 2024-09-21T13:50:16.472 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00189296 2024-09-21T13:50:16.472 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.472 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.472 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:50:16.473 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.067132+0000", 2024-09-21T13:50:16.473 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8226999999999998e-05 2024-09-21T13:50:16.473 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:16.473 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:50:16.473 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:16.473 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.473 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.473 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:5578 7.f 7:fd8ef513:::benchmark_data_smithi049_76798_object5577:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-21T13:50:16.473 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:15.052844+0000", 2024-09-21T13:50:16.473 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 1.334830137, 2024-09-21T13:50:16.473 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.014873618, 2024-09-21T13:50:16.473 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:50:16.473 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:50:16.473 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:50:16.474 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:50:16.474 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:50:16.474 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:50:16.474 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 5578 2024-09-21T13:50:16.474 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.474 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:50:16.474 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.474 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:50:16.474 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.052844+0000", 2024-09-21T13:50:16.474 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:16.474 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.474 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.474 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:50:16.474 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.052844+0000", 2024-09-21T13:50:16.475 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:16.475 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.475 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:50:16.476 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.476 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:50:16.476 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.052845+0000", 2024-09-21T13:50:16.476 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2099999999999996e-07 2024-09-21T13:50:16.476 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.476 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.476 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:50:16.476 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.052870+0000", 2024-09-21T13:50:16.476 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5097e-05 2024-09-21T13:50:16.476 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.477 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.477 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:50:16.477 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.052880+0000", 2024-09-21T13:50:16.477 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.5529999999999998e-06 2024-09-21T13:50:16.477 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.477 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.477 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:50:16.477 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.052886+0000", 2024-09-21T13:50:16.477 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9070000000000001e-06 2024-09-21T13:50:16.477 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.477 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.477 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:50:16.477 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.053910+0000", 2024-09-21T13:50:16.477 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0010235820000000001 2024-09-21T13:50:16.477 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.478 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.478 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:50:16.478 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.054024+0000", 2024-09-21T13:50:16.478 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000114051 2024-09-21T13:50:16.478 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.478 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.478 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,7", 2024-09-21T13:50:16.478 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.054157+0000", 2024-09-21T13:50:16.478 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013244400000000001 2024-09-21T13:50:16.478 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.478 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.478 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:50:16.478 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.055513+0000", 2024-09-21T13:50:16.478 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001356267 2024-09-21T13:50:16.479 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.479 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.479 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:16.479 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.057808+0000", 2024-09-21T13:50:16.479 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0022955929999999999 2024-09-21T13:50:16.479 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.479 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.479 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:16.479 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.067172+0000", 2024-09-21T13:50:16.479 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0093638860000000001 2024-09-21T13:50:16.479 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.479 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.479 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:50:16.479 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.067667+0000", 2024-09-21T13:50:16.480 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000495193 2024-09-21T13:50:16.480 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.480 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.480 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:50:16.480 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.067718+0000", 2024-09-21T13:50:16.480 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0423999999999997e-05 2024-09-21T13:50:16.480 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:16.480 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:50:16.480 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:16.480 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.480 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.480 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:5949 7.4 7:2a5a79d6:::benchmark_data_smithi049_76798_object5948:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-21T13:50:16.480 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:15.186532+0000", 2024-09-21T13:50:16.480 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 1.2011423619999999, 2024-09-21T13:50:16.480 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.017663702, 2024-09-21T13:50:16.481 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:50:16.481 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:50:16.481 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:50:16.481 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:50:16.481 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:50:16.481 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:50:16.481 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 5949 2024-09-21T13:50:16.481 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.481 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:50:16.481 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.481 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:50:16.481 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.186532+0000", 2024-09-21T13:50:16.481 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:16.481 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.481 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.482 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:50:16.482 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.186532+0000", 2024-09-21T13:50:16.482 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:16.482 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.482 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.482 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:50:16.482 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.186533+0000", 2024-09-21T13:50:16.482 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7199999999999999e-07 2024-09-21T13:50:16.482 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.482 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.482 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:50:16.482 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.186553+0000", 2024-09-21T13:50:16.482 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0227000000000001e-05 2024-09-21T13:50:16.483 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.483 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.483 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:50:16.483 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.186561+0000", 2024-09-21T13:50:16.483 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1030000000000002e-06 2024-09-21T13:50:16.483 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.483 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.483 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:50:16.483 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.186567+0000", 2024-09-21T13:50:16.483 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5910000000000003e-06 2024-09-21T13:50:16.483 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.483 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.483 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:50:16.483 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.186617+0000", 2024-09-21T13:50:16.484 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0852e-05 2024-09-21T13:50:16.484 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.484 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.484 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:50:16.484 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.186727+0000", 2024-09-21T13:50:16.484 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010989699999999999 2024-09-21T13:50:16.484 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.484 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.484 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,4", 2024-09-21T13:50:16.484 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.186915+0000", 2024-09-21T13:50:16.484 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000187548 2024-09-21T13:50:16.484 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.484 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.485 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:16.485 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.188072+0000", 2024-09-21T13:50:16.485 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0011571520000000001 2024-09-21T13:50:16.485 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.485 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.485 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:50:16.485 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.188152+0000", 2024-09-21T13:50:16.485 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9914999999999996e-05 2024-09-21T13:50:16.485 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.485 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.485 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:16.485 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.204065+0000", 2024-09-21T13:50:16.485 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.015913594 2024-09-21T13:50:16.485 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.486 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.486 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:50:16.486 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.204140+0000", 2024-09-21T13:50:16.486 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4147999999999995e-05 2024-09-21T13:50:16.486 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.486 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.486 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:50:16.486 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.204196+0000", 2024-09-21T13:50:16.486 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6103000000000001e-05 2024-09-21T13:50:16.486 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:16.486 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:50:16.486 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:16.486 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.486 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.486 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:5950 7.9 7:96ab528b:::benchmark_data_smithi049_76798_object5949:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-21T13:50:16.487 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:15.186582+0000", 2024-09-21T13:50:16.487 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 1.201092515, 2024-09-21T13:50:16.487 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.019469470999999999, 2024-09-21T13:50:16.487 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:50:16.487 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:50:16.487 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:50:16.487 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:50:16.487 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:50:16.487 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:50:16.487 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 5950 2024-09-21T13:50:16.487 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.487 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:50:16.487 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.487 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:50:16.488 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.186582+0000", 2024-09-21T13:50:16.488 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:16.488 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.488 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.488 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:50:16.488 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.186582+0000", 2024-09-21T13:50:16.488 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:16.488 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.488 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.488 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:50:16.488 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.186582+0000", 2024-09-21T13:50:16.488 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5400000000000001e-07 2024-09-21T13:50:16.488 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.488 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.489 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:50:16.489 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.186605+0000", 2024-09-21T13:50:16.489 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2983999999999999e-05 2024-09-21T13:50:16.489 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.489 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.489 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:50:16.489 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.186613+0000", 2024-09-21T13:50:16.489 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4830000000000002e-06 2024-09-21T13:50:16.489 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.489 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.489 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:50:16.489 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.186618+0000", 2024-09-21T13:50:16.489 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5130000000000004e-06 2024-09-21T13:50:16.489 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.490 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.490 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:50:16.490 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.187742+0000", 2024-09-21T13:50:16.490 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001123485 2024-09-21T13:50:16.490 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.490 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.490 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:50:16.490 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.187869+0000", 2024-09-21T13:50:16.490 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012664800000000001 2024-09-21T13:50:16.490 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.490 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.490 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,4", 2024-09-21T13:50:16.490 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.187997+0000", 2024-09-21T13:50:16.490 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000127978 2024-09-21T13:50:16.491 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.491 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.491 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:50:16.491 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.192237+0000", 2024-09-21T13:50:16.491 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0042407920000000002 2024-09-21T13:50:16.491 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.491 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.491 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:16.491 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.192675+0000", 2024-09-21T13:50:16.491 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00043796900000000002 2024-09-21T13:50:16.491 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.491 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.491 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:16.491 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.205953+0000", 2024-09-21T13:50:16.492 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.013277643 2024-09-21T13:50:16.492 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.492 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.492 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:50:16.492 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.206013+0000", 2024-09-21T13:50:16.492 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9654999999999997e-05 2024-09-21T13:50:16.492 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.492 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.492 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:50:16.492 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.206051+0000", 2024-09-21T13:50:16.492 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8766999999999997e-05 2024-09-21T13:50:16.492 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:16.492 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:50:16.492 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:16.492 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.492 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.493 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:7783 7.9 7:928275b1:::benchmark_data_smithi049_76798_object7782:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-21T13:50:16.493 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:15.871064+0000", 2024-09-21T13:50:16.493 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.51661075599999995, 2024-09-21T13:50:16.493 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.014412359, 2024-09-21T13:50:16.493 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:50:16.493 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:50:16.493 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:50:16.493 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:50:16.493 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:50:16.493 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:50:16.493 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 7783 2024-09-21T13:50:16.493 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.493 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:50:16.493 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.494 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:50:16.494 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.871064+0000", 2024-09-21T13:50:16.494 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:16.494 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.494 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.494 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:50:16.494 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.871064+0000", 2024-09-21T13:50:16.494 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:16.494 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.494 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.494 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:50:16.494 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.871065+0000", 2024-09-21T13:50:16.494 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.398e-06 2024-09-21T13:50:16.494 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.494 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.495 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:50:16.495 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.871090+0000", 2024-09-21T13:50:16.495 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5225999999999999e-05 2024-09-21T13:50:16.495 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.495 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.495 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:50:16.495 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.871107+0000", 2024-09-21T13:50:16.495 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6619000000000001e-05 2024-09-21T13:50:16.495 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.495 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.495 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:50:16.495 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.871116+0000", 2024-09-21T13:50:16.495 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.7129999999999994e-06 2024-09-21T13:50:16.495 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.495 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.496 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:50:16.496 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.871746+0000", 2024-09-21T13:50:16.496 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00063040399999999999 2024-09-21T13:50:16.496 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.496 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.496 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:50:16.496 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.871889+0000", 2024-09-21T13:50:16.496 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014292499999999999 2024-09-21T13:50:16.496 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.496 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.496 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,4", 2024-09-21T13:50:16.496 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.872046+0000", 2024-09-21T13:50:16.496 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000156565 2024-09-21T13:50:16.496 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.497 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.497 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:50:16.497 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.874467+0000", 2024-09-21T13:50:16.497 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0024212610000000001 2024-09-21T13:50:16.497 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.497 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.497 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:16.497 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.876337+0000", 2024-09-21T13:50:16.497 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001870631 2024-09-21T13:50:16.497 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.497 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.497 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:16.497 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.884954+0000", 2024-09-21T13:50:16.497 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0086163560000000004 2024-09-21T13:50:16.497 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.497 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.498 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:50:16.498 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.885418+0000", 2024-09-21T13:50:16.498 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00046453199999999998 2024-09-21T13:50:16.498 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.498 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.498 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:50:16.498 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.885476+0000", 2024-09-21T13:50:16.498 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7729000000000002e-05 2024-09-21T13:50:16.498 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:16.498 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:50:16.498 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:16.498 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.498 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.499 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:7788 7.9 7:920778c8:::benchmark_data_smithi049_76798_object7787:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-21T13:50:16.499 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:15.872498+0000", 2024-09-21T13:50:16.499 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.51517593800000006, 2024-09-21T13:50:16.499 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.014571504000000001, 2024-09-21T13:50:16.499 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:50:16.499 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:50:16.499 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:50:16.499 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:50:16.499 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:50:16.499 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:50:16.499 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 7788 2024-09-21T13:50:16.499 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.499 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:50:16.499 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.500 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:50:16.500 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.872498+0000", 2024-09-21T13:50:16.500 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:16.500 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.500 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.500 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:50:16.500 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.872498+0000", 2024-09-21T13:50:16.500 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:16.500 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.500 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.500 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:50:16.500 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.872500+0000", 2024-09-21T13:50:16.500 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4419999999999999e-06 2024-09-21T13:50:16.500 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.500 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.500 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:50:16.501 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.872527+0000", 2024-09-21T13:50:16.501 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.7010000000000001e-05 2024-09-21T13:50:16.501 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.501 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.501 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:50:16.501 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.872538+0000", 2024-09-21T13:50:16.501 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.11e-05 2024-09-21T13:50:16.501 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.501 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.501 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:50:16.501 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.872545+0000", 2024-09-21T13:50:16.501 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1319999999999996e-06 2024-09-21T13:50:16.501 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.501 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.502 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:50:16.502 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.872605+0000", 2024-09-21T13:50:16.502 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9846000000000003e-05 2024-09-21T13:50:16.502 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.502 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.502 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:50:16.502 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.872717+0000", 2024-09-21T13:50:16.502 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011197900000000001 2024-09-21T13:50:16.503 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.503 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.503 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,4", 2024-09-21T13:50:16.503 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.873901+0000", 2024-09-21T13:50:16.503 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0011844480000000001 2024-09-21T13:50:16.503 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.503 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.503 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:50:16.503 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.875615+0000", 2024-09-21T13:50:16.503 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001713542 2024-09-21T13:50:16.503 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.503 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.503 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:16.503 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.876522+0000", 2024-09-21T13:50:16.504 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00090719200000000002 2024-09-21T13:50:16.504 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.504 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.504 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:16.504 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.885543+0000", 2024-09-21T13:50:16.504 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0090208270000000004 2024-09-21T13:50:16.504 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.504 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.504 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:50:16.504 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.886841+0000", 2024-09-21T13:50:16.504 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0012983529999999999 2024-09-21T13:50:16.504 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.504 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.504 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:50:16.505 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:15.887070+0000", 2024-09-21T13:50:16.505 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000228633 2024-09-21T13:50:16.505 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:16.505 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:50:16.505 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:16.505 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.505 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.505 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:9022 7.f 7:ff13dbca:::benchmark_data_smithi049_76798_object9021:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-21T13:50:16.505 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:16.360714+0000", 2024-09-21T13:50:16.505 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.026960767, 2024-09-21T13:50:16.505 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.013218527000000001, 2024-09-21T13:50:16.505 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:50:16.505 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:50:16.506 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:50:16.506 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:50:16.506 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:50:16.506 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:50:16.506 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 9022 2024-09-21T13:50:16.506 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.506 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:50:16.506 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.506 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:50:16.506 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:16.360714+0000", 2024-09-21T13:50:16.506 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:16.506 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.506 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.506 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:50:16.507 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:16.360714+0000", 2024-09-21T13:50:16.507 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:16.507 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.507 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.507 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:50:16.507 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:16.360714+0000", 2024-09-21T13:50:16.507 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6499999999999999e-07 2024-09-21T13:50:16.507 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.507 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.507 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:50:16.507 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:16.360750+0000", 2024-09-21T13:50:16.507 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6004000000000003e-05 2024-09-21T13:50:16.507 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.507 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.508 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:50:16.508 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:16.360761+0000", 2024-09-21T13:50:16.508 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0359e-05 2024-09-21T13:50:16.508 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.508 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.508 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:50:16.508 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:16.360767+0000", 2024-09-21T13:50:16.508 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8190000000000003e-06 2024-09-21T13:50:16.508 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.508 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.508 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:50:16.508 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:16.362109+0000", 2024-09-21T13:50:16.508 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0013413120000000001 2024-09-21T13:50:16.508 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.509 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.509 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:50:16.509 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:16.362236+0000", 2024-09-21T13:50:16.509 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012749699999999999 2024-09-21T13:50:16.509 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.509 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.509 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,7", 2024-09-21T13:50:16.509 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:16.362355+0000", 2024-09-21T13:50:16.509 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011869799999999999 2024-09-21T13:50:16.509 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.509 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.509 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:50:16.509 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:16.363441+0000", 2024-09-21T13:50:16.509 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001086212 2024-09-21T13:50:16.509 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.510 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.510 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:16.510 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:16.364689+0000", 2024-09-21T13:50:16.510 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001247651 2024-09-21T13:50:16.510 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.510 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.510 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:16.510 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:16.373833+0000", 2024-09-21T13:50:16.510 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.009143811 2024-09-21T13:50:16.510 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.510 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.510 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:50:16.510 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:16.373891+0000", 2024-09-21T13:50:16.511 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8746000000000003e-05 2024-09-21T13:50:16.511 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.511 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.511 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:50:16.511 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:16.373932+0000", 2024-09-21T13:50:16.511 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0753000000000001e-05 2024-09-21T13:50:16.511 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:16.511 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:50:16.511 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:16.511 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.511 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.511 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:9024 7.f 7:f75babfa:::benchmark_data_smithi049_76798_object9023:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-21T13:50:16.511 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:16.360786+0000", 2024-09-21T13:50:16.511 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.026888464000000001, 2024-09-21T13:50:16.511 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.013810906, 2024-09-21T13:50:16.512 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:50:16.512 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:50:16.512 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:50:16.512 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:50:16.512 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:50:16.512 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:50:16.512 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 9024 2024-09-21T13:50:16.512 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.512 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:50:16.512 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.512 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:50:16.512 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:16.360786+0000", 2024-09-21T13:50:16.512 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:16.512 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.513 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.513 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:50:16.513 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:16.360786+0000", 2024-09-21T13:50:16.513 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:16.513 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.513 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.513 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:50:16.513 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:16.360787+0000", 2024-09-21T13:50:16.513 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5899999999999996e-07 2024-09-21T13:50:16.513 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.513 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.513 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:50:16.513 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:16.360810+0000", 2024-09-21T13:50:16.513 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3921e-05 2024-09-21T13:50:16.513 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.514 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.514 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:50:16.514 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:16.360820+0000", 2024-09-21T13:50:16.514 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.594e-06 2024-09-21T13:50:16.514 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.514 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.514 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:50:16.514 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:16.360826+0000", 2024-09-21T13:50:16.514 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2099999999999998e-06 2024-09-21T13:50:16.514 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.514 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.514 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:50:16.514 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:16.362847+0000", 2024-09-21T13:50:16.514 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0020210219999999999 2024-09-21T13:50:16.515 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.515 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.515 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:50:16.515 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:16.362950+0000", 2024-09-21T13:50:16.515 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000103188 2024-09-21T13:50:16.515 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.515 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.515 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,7", 2024-09-21T13:50:16.515 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:16.363076+0000", 2024-09-21T13:50:16.515 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012606100000000001 2024-09-21T13:50:16.515 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.515 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.515 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:50:16.515 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:16.363761+0000", 2024-09-21T13:50:16.515 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00068462299999999998 2024-09-21T13:50:16.516 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.516 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.516 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:16.516 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:16.365390+0000", 2024-09-21T13:50:16.516 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0016285620000000001 2024-09-21T13:50:16.516 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.516 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.516 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:16.516 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:16.374401+0000", 2024-09-21T13:50:16.516 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0090114819999999995 2024-09-21T13:50:16.516 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.516 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.516 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:50:16.516 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:16.374558+0000", 2024-09-21T13:50:16.517 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000156632 2024-09-21T13:50:16.517 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:16.517 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:16.517 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:50:16.517 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:16.374597+0000", 2024-09-21T13:50:16.517 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9051999999999998e-05 2024-09-21T13:50:16.517 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:16.517 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:50:16.517 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:16.517 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:16.517 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:50:16.517 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T13:50:16.538 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:16.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:16.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:50:16.640 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:16.667 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:16.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:16.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:50:16.670 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:16 smithi049 podman[76740]: 2024-09-21 13:50:16.432303238 +0000 UTC m=+3.701008492 container create 556e964b68ca8f011a5e0cb8305e166ab59cd8d53c0710b605a382512d907fad (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T13:50:16.769 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:16.796 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:16.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:16.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:16.891 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 4 16 10315 10299 160.917 155.688 0.00281288 0.00620561 2024-09-21T13:50:16.897 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:16.924 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:16.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:16.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:17.026 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:50:17.068 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:17.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:17.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:50:17.170 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:50:17.200 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:17.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:17.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:50:17.302 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:17.331 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:17.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:17.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:50:17.433 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:17.461 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:17.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:17.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:50:17.564 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:17.594 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:17.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:17.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:50:17.696 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:50:17.740 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:17.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:17.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:50:17.842 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:50:17.863 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:17.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:17.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:50:17.891 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 5 16 13017 13001 162.506 168.875 0.00158549 0.00605429 2024-09-21T13:50:17.965 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:50:18.006 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:17 smithi049 podman[76740]: 2024-09-21 13:50:17.702179416 +0000 UTC m=+4.970884676 container init 556e964b68ca8f011a5e0cb8305e166ab59cd8d53c0710b605a382512d907fad (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T13:50:18.006 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:17 smithi049 podman[76740]: 2024-09-21 13:50:17.71366933 +0000 UTC m=+4.982374575 container start 556e964b68ca8f011a5e0cb8305e166ab59cd8d53c0710b605a382512d907fad (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T13:50:18.006 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:17 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a[77379]: ts=2024-09-21T13:50:17.720Z caller=node_exporter.go:192 level=info msg="Starting node_exporter" version="(version=1.7.0, branch=HEAD, revision=7333465abf9efba81876303bb57e6fadb946041b)" 2024-09-21T13:50:18.006 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:17 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a[77379]: ts=2024-09-21T13:50:17.720Z caller=node_exporter.go:193 level=info msg="Build context" build_context="(go=go1.21.4, platform=linux/amd64, user=root@35918982f6d8, date=20231112-23:53:35, tags=netgo osusergo static_build)" 2024-09-21T13:50:18.006 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:17 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a[77379]: ts=2024-09-21T13:50:17.721Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-21T13:50:18.006 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:17 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a[77379]: ts=2024-09-21T13:50:17.721Z caller=diskstats_linux.go:265 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-21T13:50:18.006 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:17 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a[77379]: ts=2024-09-21T13:50:17.722Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-21T13:50:18.006 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:17 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a[77379]: ts=2024-09-21T13:50:17.722Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-21T13:50:18.006 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:17 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a[77379]: ts=2024-09-21T13:50:17.722Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-21T13:50:18.006 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:17 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a[77379]: ts=2024-09-21T13:50:17.722Z caller=node_exporter.go:117 level=info collector=arp 2024-09-21T13:50:18.006 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:17 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a[77379]: ts=2024-09-21T13:50:17.722Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-21T13:50:18.007 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:17 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a[77379]: ts=2024-09-21T13:50:17.722Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-21T13:50:18.007 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:17 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a[77379]: ts=2024-09-21T13:50:17.722Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-21T13:50:18.007 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:17 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a[77379]: ts=2024-09-21T13:50:17.722Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-21T13:50:18.007 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:17 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a[77379]: ts=2024-09-21T13:50:17.722Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-21T13:50:18.007 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:17 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a[77379]: ts=2024-09-21T13:50:17.722Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-21T13:50:18.008 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:17 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a[77379]: ts=2024-09-21T13:50:17.722Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-21T13:50:18.008 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:17 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a[77379]: ts=2024-09-21T13:50:17.722Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-21T13:50:18.008 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:17 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a[77379]: ts=2024-09-21T13:50:17.723Z caller=node_exporter.go:117 level=info collector=edac 2024-09-21T13:50:18.008 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:17 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a[77379]: ts=2024-09-21T13:50:17.723Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-21T13:50:18.008 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:17 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a[77379]: ts=2024-09-21T13:50:17.723Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-21T13:50:18.008 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:17 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a[77379]: ts=2024-09-21T13:50:17.723Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-21T13:50:18.008 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:17 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a[77379]: ts=2024-09-21T13:50:17.723Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-21T13:50:18.008 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:17 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a[77379]: ts=2024-09-21T13:50:17.723Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-21T13:50:18.008 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:17 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a[77379]: ts=2024-09-21T13:50:17.723Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-21T13:50:18.008 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:17 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a[77379]: ts=2024-09-21T13:50:17.723Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-21T13:50:18.009 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:17 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a[77379]: ts=2024-09-21T13:50:17.723Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-21T13:50:18.009 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:17 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a[77379]: ts=2024-09-21T13:50:17.723Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-21T13:50:18.009 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:17 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a[77379]: ts=2024-09-21T13:50:17.723Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-21T13:50:18.009 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:17 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a[77379]: ts=2024-09-21T13:50:17.723Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-21T13:50:18.009 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:17 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a[77379]: ts=2024-09-21T13:50:17.723Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-21T13:50:18.009 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:17 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a[77379]: ts=2024-09-21T13:50:17.723Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-21T13:50:18.009 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:17 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a[77379]: ts=2024-09-21T13:50:17.723Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-21T13:50:18.009 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:17 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a[77379]: ts=2024-09-21T13:50:17.723Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-21T13:50:18.009 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:17 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a[77379]: ts=2024-09-21T13:50:17.723Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-21T13:50:18.009 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:17 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a[77379]: ts=2024-09-21T13:50:17.723Z caller=node_exporter.go:117 level=info collector=os 2024-09-21T13:50:18.009 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:17 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a[77379]: ts=2024-09-21T13:50:17.723Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-21T13:50:18.009 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:17 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a[77379]: ts=2024-09-21T13:50:17.723Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-21T13:50:18.009 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:17 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a[77379]: ts=2024-09-21T13:50:17.723Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-21T13:50:18.010 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:17 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a[77379]: ts=2024-09-21T13:50:17.723Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-21T13:50:18.010 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:17 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a[77379]: ts=2024-09-21T13:50:17.723Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-21T13:50:18.010 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:17 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a[77379]: ts=2024-09-21T13:50:17.723Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-21T13:50:18.010 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:17 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a[77379]: ts=2024-09-21T13:50:17.723Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-21T13:50:18.010 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:17 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a[77379]: ts=2024-09-21T13:50:17.723Z caller=node_exporter.go:117 level=info collector=stat 2024-09-21T13:50:18.010 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:17 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a[77379]: ts=2024-09-21T13:50:17.723Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-21T13:50:18.010 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:17 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a[77379]: ts=2024-09-21T13:50:17.723Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-21T13:50:18.010 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:17 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a[77379]: ts=2024-09-21T13:50:17.723Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-21T13:50:18.010 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:17 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a[77379]: ts=2024-09-21T13:50:17.723Z caller=node_exporter.go:117 level=info collector=time 2024-09-21T13:50:18.010 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:17 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a[77379]: ts=2024-09-21T13:50:17.723Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-21T13:50:18.010 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:17 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a[77379]: ts=2024-09-21T13:50:17.723Z caller=node_exporter.go:117 level=info collector=uname 2024-09-21T13:50:18.011 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:17 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a[77379]: ts=2024-09-21T13:50:17.723Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-21T13:50:18.011 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:17 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a[77379]: ts=2024-09-21T13:50:17.723Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-21T13:50:18.011 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:17 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a[77379]: ts=2024-09-21T13:50:17.723Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-21T13:50:18.011 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:17 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a[77379]: ts=2024-09-21T13:50:17.724Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9100 2024-09-21T13:50:18.011 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:17 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a[77379]: ts=2024-09-21T13:50:17.724Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-21T13:50:18.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:18.012 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:18.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:50:18.113 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:18.141 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:18.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:18.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:50:18.243 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:50:18.272 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:18.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:18.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:50:18.374 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:50:18.405 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:18.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:18.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:50:18.507 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:50:18.547 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:18.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:18.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:50:18.587 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:18 smithi049 ceph-mon[30080]: pgmap v42: 148 pgs: 148 active+clean; 79 MiB data, 332 MiB used, 715 GiB / 715 GiB avail; 784 B/s rd, 12 MiB/s wr, 192 op/s 2024-09-21T13:50:18.587 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:18 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:18.587 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:18 smithi049 ceph-mon[40489]: pgmap v42: 148 pgs: 148 active+clean; 79 MiB data, 332 MiB used, 715 GiB / 715 GiB avail; 784 B/s rd, 12 MiB/s wr, 192 op/s 2024-09-21T13:50:18.587 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:18 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:18.650 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:18.679 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:18.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:18.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:50:18.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:18 smithi187 ceph-mon[33032]: pgmap v42: 148 pgs: 148 active+clean; 79 MiB data, 332 MiB used, 715 GiB / 715 GiB avail; 784 B/s rd, 12 MiB/s wr, 192 op/s 2024-09-21T13:50:18.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:18 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:18.781 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:50:18.808 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:18.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:18.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:50:18.891 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 6 15 15574 15559 162.065 159.875 0.0111879 0.00616238 2024-09-21T13:50:18.911 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:18.941 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:18.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:18.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:19.044 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:50:19.089 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:19.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:19.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:50:19.191 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:19.216 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:19.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:19.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:19.318 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:50:19.358 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:19.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:19.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:50:19.411 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:50:18 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:50:19.461 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:50:19.498 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:19.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:19.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:50:19.601 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:50:19.641 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:19.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:19.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:50:19.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:19 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3417254797' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:50:19.743 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:50:19.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:19 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3417254797' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:50:19.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:19 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3417254797' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:50:19.783 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:19.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:19.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:50:19.885 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:19.891 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 7 16 17920 17904 159.848 146.562 0.00538342 0.00624744 2024-09-21T13:50:19.918 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:19.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:19.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:50:20.020 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:50:20.050 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:20.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:20.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:50:20.152 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:50:20.191 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:20.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:20.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:50:20.293 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:20.317 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":40,"stamp":"2024-09-21T13:50:14.863748+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":82450971,"num_objects":1448,"num_object_clones":0,"num_object_copies":4344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1448,"num_whiteouts":0,"num_read":918,"num_read_kb":775,"num_write":1798,"num_write_kb":81609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1801,"ondisk_log_size":1801,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":340292,"kb_used_data":103260,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":749620924,"statfs":{"total":767960285184,"available":767611826176,"internally_reserved":0,"allocated":105738240,"data_stored":101392873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":81854464,"num_objects":1249,"num_object_clones":0,"num_object_copies":3747,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1249,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1249,"num_write_kb":79936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.525755"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":64,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:10.820874+0000","last_change":"2024-09-21T13:48:58.061797+0000","last_active":"2024-09-21T13:50:10.820874+0000","last_peered":"2024-09-21T13:50:10.820874+0000","last_clean":"2024-09-21T13:50:10.820874+0000","last_became_active":"2024-09-21T13:48:58.061415+0000","last_became_peered":"2024-09-21T13:48:58.061415+0000","last_unstale":"2024-09-21T13:50:10.820874+0000","last_undegraded":"2024-09-21T13:50:10.820874+0000","last_fullsized":"2024-09-21T13:50:10.820874+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"68'9","reported_seq":76,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:10.819654+0000","last_change":"2024-09-21T13:49:00.055717+0000","last_active":"2024-09-21T13:50:10.819654+0000","last_peered":"2024-09-21T13:50:10.819654+0000","last_clean":"2024-09-21T13:50:10.819654+0000","last_became_active":"2024-09-21T13:49:00.055210+0000","last_became_peered":"2024-09-21T13:49:00.055210+0000","last_unstale":"2024-09-21T13:50:10.819654+0000","last_undegraded":"2024-09-21T13:50:10.819654+0000","last_fullsized":"2024-09-21T13:50:10.819654+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:13.541017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":56,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:10.819791+0000","last_change":"2024-09-21T13:49:02.061394+0000","last_active":"2024-09-21T13:50:10.819791+0000","last_peered":"2024-09-21T13:50:10.819791+0000","last_clean":"2024-09-21T13:50:10.819791+0000","last_became_active":"2024-09-21T13:49:02.060984+0000","last_became_peered":"2024-09-21T13:49:02.060984+0000","last_unstale":"2024-09-21T13:50:10.819791+0000","last_undegraded":"2024-09-21T13:50:10.819791+0000","last_fullsized":"2024-09-21T13:50:10.819791+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:18.936236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":52,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:10.819689+0000","last_change":"2024-09-21T13:49:04.072734+0000","last_active":"2024-09-21T13:50:10.819689+0000","last_peered":"2024-09-21T13:50:10.819689+0000","last_clean":"2024-09-21T13:50:10.819689+0000","last_became_active":"2024-09-21T13:49:04.072460+0000","last_became_peered":"2024-09-21T13:49:04.072460+0000","last_unstale":"2024-09-21T13:50:10.819689+0000","last_undegraded":"2024-09-21T13:50:10.819689+0000","last_fullsized":"2024-09-21T13:50:10.819689+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:30.526897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":52,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.241469+0000","last_change":"2024-09-21T13:49:04.434312+0000","last_active":"2024-09-21T13:50:12.241469+0000","last_peered":"2024-09-21T13:50:12.241469+0000","last_clean":"2024-09-21T13:50:12.241469+0000","last_became_active":"2024-09-21T13:49:04.433965+0000","last_became_peered":"2024-09-21T13:49:04.433965+0000","last_unstale":"2024-09-21T13:50:12.241469+0000","last_undegraded":"2024-09-21T13:50:12.241469+0000","last_fullsized":"2024-09-21T13:50:12.241469+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":64,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.241867+0000","last_change":"2024-09-21T13:48:58.051225+0000","last_active":"2024-09-21T13:50:12.241867+0000","last_peered":"2024-09-21T13:50:12.241867+0000","last_clean":"2024-09-21T13:50:12.241867+0000","last_became_active":"2024-09-21T13:48:58.050479+0000","last_became_peered":"2024-09-21T13:48:58.050479+0000","last_unstale":"2024-09-21T13:50:12.241867+0000","last_undegraded":"2024-09-21T13:50:12.241867+0000","last_fullsized":"2024-09-21T13:50:12.241867+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:19.515469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"68'15","reported_seq":85,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.240698+0000","last_change":"2024-09-21T13:49:00.063787+0000","last_active":"2024-09-21T13:50:12.240698+0000","last_peered":"2024-09-21T13:50:12.240698+0000","last_clean":"2024-09-21T13:50:12.240698+0000","last_became_active":"2024-09-21T13:49:00.063460+0000","last_became_peered":"2024-09-21T13:49:00.063460+0000","last_unstale":"2024-09-21T13:50:12.240698+0000","last_undegraded":"2024-09-21T13:50:12.240698+0000","last_fullsized":"2024-09-21T13:50:12.240698+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:14.007409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":56,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:10.819584+0000","last_change":"2024-09-21T13:49:02.064007+0000","last_active":"2024-09-21T13:50:10.819584+0000","last_peered":"2024-09-21T13:50:10.819584+0000","last_clean":"2024-09-21T13:50:10.819584+0000","last_became_active":"2024-09-21T13:49:02.063687+0000","last_became_peered":"2024-09-21T13:49:02.063687+0000","last_unstale":"2024-09-21T13:50:10.819584+0000","last_undegraded":"2024-09-21T13:50:10.819584+0000","last_fullsized":"2024-09-21T13:50:10.819584+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":64,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.241390+0000","last_change":"2024-09-21T13:48:58.051001+0000","last_active":"2024-09-21T13:50:12.241390+0000","last_peered":"2024-09-21T13:50:12.241390+0000","last_clean":"2024-09-21T13:50:12.241390+0000","last_became_active":"2024-09-21T13:48:58.050695+0000","last_became_peered":"2024-09-21T13:48:58.050695+0000","last_unstale":"2024-09-21T13:50:12.241390+0000","last_undegraded":"2024-09-21T13:50:12.241390+0000","last_fullsized":"2024-09-21T13:50:12.241390+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"68'9","reported_seq":76,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:10.820002+0000","last_change":"2024-09-21T13:49:00.059748+0000","last_active":"2024-09-21T13:50:10.820002+0000","last_peered":"2024-09-21T13:50:10.820002+0000","last_clean":"2024-09-21T13:50:10.820002+0000","last_became_active":"2024-09-21T13:49:00.059382+0000","last_became_peered":"2024-09-21T13:49:00.059382+0000","last_unstale":"2024-09-21T13:50:10.820002+0000","last_undegraded":"2024-09-21T13:50:10.820002+0000","last_fullsized":"2024-09-21T13:50:10.820002+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":56,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.241347+0000","last_change":"2024-09-21T13:49:02.066978+0000","last_active":"2024-09-21T13:50:12.241347+0000","last_peered":"2024-09-21T13:50:12.241347+0000","last_clean":"2024-09-21T13:50:12.241347+0000","last_became_active":"2024-09-21T13:49:02.066693+0000","last_became_peered":"2024-09-21T13:49:02.066693+0000","last_unstale":"2024-09-21T13:50:12.241347+0000","last_undegraded":"2024-09-21T13:50:12.241347+0000","last_fullsized":"2024-09-21T13:50:12.241347+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:28.710214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":52,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:10.818042+0000","last_change":"2024-09-21T13:49:04.080709+0000","last_active":"2024-09-21T13:50:10.818042+0000","last_peered":"2024-09-21T13:50:10.818042+0000","last_clean":"2024-09-21T13:50:10.818042+0000","last_became_active":"2024-09-21T13:49:04.080359+0000","last_became_peered":"2024-09-21T13:49:04.080359+0000","last_unstale":"2024-09-21T13:50:10.818042+0000","last_undegraded":"2024-09-21T13:50:10.818042+0000","last_fullsized":"2024-09-21T13:50:10.818042+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":64,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.241851+0000","last_change":"2024-09-21T13:48:58.052581+0000","last_active":"2024-09-21T13:50:12.241851+0000","last_peered":"2024-09-21T13:50:12.241851+0000","last_clean":"2024-09-21T13:50:12.241851+0000","last_became_active":"2024-09-21T13:48:58.050906+0000","last_became_peered":"2024-09-21T13:48:58.050906+0000","last_unstale":"2024-09-21T13:50:12.241851+0000","last_undegraded":"2024-09-21T13:50:12.241851+0000","last_fullsized":"2024-09-21T13:50:12.241851+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:19.977336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"68'5","reported_seq":70,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:10.819782+0000","last_change":"2024-09-21T13:49:00.060913+0000","last_active":"2024-09-21T13:50:10.819782+0000","last_peered":"2024-09-21T13:50:10.819782+0000","last_clean":"2024-09-21T13:50:10.819782+0000","last_became_active":"2024-09-21T13:49:00.060497+0000","last_became_peered":"2024-09-21T13:49:00.060497+0000","last_unstale":"2024-09-21T13:50:10.819782+0000","last_undegraded":"2024-09-21T13:50:10.819782+0000","last_fullsized":"2024-09-21T13:50:10.819782+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":56,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.241069+0000","last_change":"2024-09-21T13:49:02.060484+0000","last_active":"2024-09-21T13:50:12.241069+0000","last_peered":"2024-09-21T13:50:12.241069+0000","last_clean":"2024-09-21T13:50:12.241069+0000","last_became_active":"2024-09-21T13:49:02.060212+0000","last_became_peered":"2024-09-21T13:49:02.060212+0000","last_unstale":"2024-09-21T13:50:12.241069+0000","last_undegraded":"2024-09-21T13:50:12.241069+0000","last_fullsized":"2024-09-21T13:50:12.241069+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:17.976156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":52,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.240809+0000","last_change":"2024-09-21T13:49:04.078296+0000","last_active":"2024-09-21T13:50:12.240809+0000","last_peered":"2024-09-21T13:50:12.240809+0000","last_clean":"2024-09-21T13:50:12.240809+0000","last_became_active":"2024-09-21T13:49:04.078026+0000","last_became_peered":"2024-09-21T13:49:04.078026+0000","last_unstale":"2024-09-21T13:50:12.240809+0000","last_undegraded":"2024-09-21T13:50:12.240809+0000","last_fullsized":"2024-09-21T13:50:12.240809+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":52,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:10.820214+0000","last_change":"2024-09-21T13:49:04.432458+0000","last_active":"2024-09-21T13:50:10.820214+0000","last_peered":"2024-09-21T13:50:10.820214+0000","last_clean":"2024-09-21T13:50:10.820214+0000","last_became_active":"2024-09-21T13:49:04.431422+0000","last_became_peered":"2024-09-21T13:49:04.431422+0000","last_unstale":"2024-09-21T13:50:10.820214+0000","last_undegraded":"2024-09-21T13:50:10.820214+0000","last_fullsized":"2024-09-21T13:50:10.820214+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":64,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:10.820311+0000","last_change":"2024-09-21T13:48:58.057667+0000","last_active":"2024-09-21T13:50:10.820311+0000","last_peered":"2024-09-21T13:50:10.820311+0000","last_clean":"2024-09-21T13:50:10.820311+0000","last_became_active":"2024-09-21T13:48:58.057336+0000","last_became_peered":"2024-09-21T13:48:58.057336+0000","last_unstale":"2024-09-21T13:50:10.820311+0000","last_undegraded":"2024-09-21T13:50:10.820311+0000","last_fullsized":"2024-09-21T13:50:10.820311+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:18.555629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"68'15","reported_seq":85,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:11.784881+0000","last_change":"2024-09-21T13:49:00.059068+0000","last_active":"2024-09-21T13:50:11.784881+0000","last_peered":"2024-09-21T13:50:11.784881+0000","last_clean":"2024-09-21T13:50:11.784881+0000","last_became_active":"2024-09-21T13:49:00.058730+0000","last_became_peered":"2024-09-21T13:49:00.058730+0000","last_unstale":"2024-09-21T13:50:11.784881+0000","last_undegraded":"2024-09-21T13:50:11.784881+0000","last_fullsized":"2024-09-21T13:50:11.784881+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:18.796269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":56,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:10.818970+0000","last_change":"2024-09-21T13:49:02.067253+0000","last_active":"2024-09-21T13:50:10.818970+0000","last_peered":"2024-09-21T13:50:10.818970+0000","last_clean":"2024-09-21T13:50:10.818970+0000","last_became_active":"2024-09-21T13:49:02.066796+0000","last_became_peered":"2024-09-21T13:49:02.066796+0000","last_unstale":"2024-09-21T13:50:10.818970+0000","last_undegraded":"2024-09-21T13:50:10.818970+0000","last_fullsized":"2024-09-21T13:50:10.818970+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:24.625163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":52,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.241938+0000","last_change":"2024-09-21T13:49:04.434429+0000","last_active":"2024-09-21T13:50:12.241938+0000","last_peered":"2024-09-21T13:50:12.241938+0000","last_clean":"2024-09-21T13:50:12.241938+0000","last_became_active":"2024-09-21T13:49:04.434049+0000","last_became_peered":"2024-09-21T13:49:04.434049+0000","last_unstale":"2024-09-21T13:50:12.241938+0000","last_undegraded":"2024-09-21T13:50:12.241938+0000","last_fullsized":"2024-09-21T13:50:12.241938+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:25.759474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":64,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:10.820145+0000","last_change":"2024-09-21T13:48:58.043457+0000","last_active":"2024-09-21T13:50:10.820145+0000","last_peered":"2024-09-21T13:50:10.820145+0000","last_clean":"2024-09-21T13:50:10.820145+0000","last_became_active":"2024-09-21T13:48:58.043088+0000","last_became_peered":"2024-09-21T13:48:58.043088+0000","last_unstale":"2024-09-21T13:50:10.820145+0000","last_undegraded":"2024-09-21T13:50:10.820145+0000","last_fullsized":"2024-09-21T13:50:10.820145+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:21.622429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"68'12","reported_seq":83,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.240773+0000","last_change":"2024-09-21T13:49:00.062162+0000","last_active":"2024-09-21T13:50:12.240773+0000","last_peered":"2024-09-21T13:50:12.240773+0000","last_clean":"2024-09-21T13:50:12.240773+0000","last_became_active":"2024-09-21T13:49:00.061867+0000","last_became_peered":"2024-09-21T13:49:00.061867+0000","last_unstale":"2024-09-21T13:50:12.240773+0000","last_undegraded":"2024-09-21T13:50:12.240773+0000","last_fullsized":"2024-09-21T13:50:12.240773+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:27.206067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":56,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:10.820075+0000","last_change":"2024-09-21T13:49:02.063090+0000","last_active":"2024-09-21T13:50:10.820075+0000","last_peered":"2024-09-21T13:50:10.820075+0000","last_clean":"2024-09-21T13:50:10.820075+0000","last_became_active":"2024-09-21T13:49:02.062802+0000","last_became_peered":"2024-09-21T13:49:02.062802+0000","last_unstale":"2024-09-21T13:50:10.820075+0000","last_undegraded":"2024-09-21T13:50:10.820075+0000","last_fullsized":"2024-09-21T13:50:10.820075+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:20.369055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.241400+0000","last_change":"2024-09-21T13:49:04.076090+0000","last_active":"2024-09-21T13:50:12.241400+0000","last_peered":"2024-09-21T13:50:12.241400+0000","last_clean":"2024-09-21T13:50:12.241400+0000","last_became_active":"2024-09-21T13:49:04.075792+0000","last_became_peered":"2024-09-21T13:49:04.075792+0000","last_unstale":"2024-09-21T13:50:12.241400+0000","last_undegraded":"2024-09-21T13:50:12.241400+0000","last_fullsized":"2024-09-21T13:50:12.241400+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:51:11.144062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":65,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:10.819960+0000","last_change":"2024-09-21T13:48:58.049463+0000","last_active":"2024-09-21T13:50:10.819960+0000","last_peered":"2024-09-21T13:50:10.819960+0000","last_clean":"2024-09-21T13:50:10.819960+0000","last_became_active":"2024-09-21T13:48:58.048972+0000","last_became_peered":"2024-09-21T13:48:58.048972+0000","last_unstale":"2024-09-21T13:50:10.819960+0000","last_undegraded":"2024-09-21T13:50:10.819960+0000","last_fullsized":"2024-09-21T13:50:10.819960+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"68'10","reported_seq":75,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:10.820743+0000","last_change":"2024-09-21T13:49:00.426837+0000","last_active":"2024-09-21T13:50:10.820743+0000","last_peered":"2024-09-21T13:50:10.820743+0000","last_clean":"2024-09-21T13:50:10.820743+0000","last_became_active":"2024-09-21T13:49:00.426178+0000","last_became_peered":"2024-09-21T13:49:00.426178+0000","last_unstale":"2024-09-21T13:50:10.820743+0000","last_undegraded":"2024-09-21T13:50:10.820743+0000","last_fullsized":"2024-09-21T13:50:10.820743+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:22.029884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"68'8","reported_seq":83,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.243055+0000","last_change":"2024-09-21T13:49:02.059979+0000","last_active":"2024-09-21T13:50:12.243055+0000","last_peered":"2024-09-21T13:50:12.243055+0000","last_clean":"2024-09-21T13:50:12.243055+0000","last_became_active":"2024-09-21T13:49:02.059638+0000","last_became_peered":"2024-09-21T13:49:02.059638+0000","last_unstale":"2024-09-21T13:50:12.243055+0000","last_undegraded":"2024-09-21T13:50:12.243055+0000","last_fullsized":"2024-09-21T13:50:12.243055+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:26.974288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"87'71","reported_seq":90,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:13.324647+0000","last_change":"2024-09-21T13:50:07.423399+0000","last_active":"2024-09-21T13:50:13.324647+0000","last_peered":"2024-09-21T13:50:13.324647+0000","last_clean":"2024-09-21T13:50:13.324647+0000","last_became_active":"2024-09-21T13:50:07.422961+0000","last_became_peered":"2024-09-21T13:50:07.422961+0000","last_unstale":"2024-09-21T13:50:13.324647+0000","last_undegraded":"2024-09-21T13:50:13.324647+0000","last_fullsized":"2024-09-21T13:50:13.324647+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:51:17.021474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4653056,"num_objects":71,"num_object_clones":0,"num_object_copies":213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71,"num_write_kb":4544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"68'15","reported_seq":85,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:10.818640+0000","last_change":"2024-09-21T13:49:00.060620+0000","last_active":"2024-09-21T13:50:10.818640+0000","last_peered":"2024-09-21T13:50:10.818640+0000","last_clean":"2024-09-21T13:50:10.818640+0000","last_became_active":"2024-09-21T13:49:00.060324+0000","last_became_peered":"2024-09-21T13:49:00.060324+0000","last_unstale":"2024-09-21T13:50:10.818640+0000","last_undegraded":"2024-09-21T13:50:10.818640+0000","last_fullsized":"2024-09-21T13:50:10.818640+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:20.149960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":72,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.241317+0000","last_change":"2024-09-21T13:50:06.151943+0000","last_active":"2024-09-21T13:50:12.241317+0000","last_peered":"2024-09-21T13:50:12.241317+0000","last_clean":"2024-09-21T13:50:12.241317+0000","last_became_active":"2024-09-21T13:48:58.051911+0000","last_became_peered":"2024-09-21T13:48:58.051911+0000","last_unstale":"2024-09-21T13:50:12.241317+0000","last_undegraded":"2024-09-21T13:50:12.241317+0000","last_fullsized":"2024-09-21T13:50:12.241317+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.151888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.151888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:51:27.235477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"68'8","reported_seq":83,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:10.820452+0000","last_change":"2024-09-21T13:49:02.061966+0000","last_active":"2024-09-21T13:50:10.820452+0000","last_peered":"2024-09-21T13:50:10.820452+0000","last_clean":"2024-09-21T13:50:10.820452+0000","last_became_active":"2024-09-21T13:49:02.061679+0000","last_became_peered":"2024-09-21T13:49:02.061679+0000","last_unstale":"2024-09-21T13:50:10.820452+0000","last_undegraded":"2024-09-21T13:50:10.820452+0000","last_fullsized":"2024-09-21T13:50:10.820452+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:28.854727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":52,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.240949+0000","last_change":"2024-09-21T13:49:04.082213+0000","last_active":"2024-09-21T13:50:12.240949+0000","last_peered":"2024-09-21T13:50:12.240949+0000","last_clean":"2024-09-21T13:50:12.240949+0000","last_became_active":"2024-09-21T13:49:04.081905+0000","last_became_peered":"2024-09-21T13:49:04.081905+0000","last_unstale":"2024-09-21T13:50:12.240949+0000","last_undegraded":"2024-09-21T13:50:12.240949+0000","last_fullsized":"2024-09-21T13:50:12.240949+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:17.499009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"87'69","reported_seq":88,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:13.324341+0000","last_change":"2024-09-21T13:50:07.419256+0000","last_active":"2024-09-21T13:50:13.324341+0000","last_peered":"2024-09-21T13:50:13.324341+0000","last_clean":"2024-09-21T13:50:13.324341+0000","last_became_active":"2024-09-21T13:50:07.418837+0000","last_became_peered":"2024-09-21T13:50:07.418837+0000","last_unstale":"2024-09-21T13:50:13.324341+0000","last_undegraded":"2024-09-21T13:50:13.324341+0000","last_fullsized":"2024-09-21T13:50:13.324341+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:51:34.550162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4521984,"num_objects":69,"num_object_clones":0,"num_object_copies":207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":69,"num_write_kb":4416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"68'18","reported_seq":92,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.241106+0000","last_change":"2024-09-21T13:49:00.429686+0000","last_active":"2024-09-21T13:50:12.241106+0000","last_peered":"2024-09-21T13:50:12.241106+0000","last_clean":"2024-09-21T13:50:12.241106+0000","last_became_active":"2024-09-21T13:49:00.429314+0000","last_became_peered":"2024-09-21T13:49:00.429314+0000","last_unstale":"2024-09-21T13:50:12.241106+0000","last_undegraded":"2024-09-21T13:50:12.241106+0000","last_fullsized":"2024-09-21T13:50:12.241106+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":64,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.242086+0000","last_change":"2024-09-21T13:48:58.056669+0000","last_active":"2024-09-21T13:50:12.242086+0000","last_peered":"2024-09-21T13:50:12.242086+0000","last_clean":"2024-09-21T13:50:12.242086+0000","last_became_active":"2024-09-21T13:48:58.056375+0000","last_became_peered":"2024-09-21T13:48:58.056375+0000","last_unstale":"2024-09-21T13:50:12.242086+0000","last_undegraded":"2024-09-21T13:50:12.242086+0000","last_fullsized":"2024-09-21T13:50:12.242086+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:21.686532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":56,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:10.819771+0000","last_change":"2024-09-21T13:49:02.063237+0000","last_active":"2024-09-21T13:50:10.819771+0000","last_peered":"2024-09-21T13:50:10.819771+0000","last_clean":"2024-09-21T13:50:10.819771+0000","last_became_active":"2024-09-21T13:49:02.062955+0000","last_became_peered":"2024-09-21T13:49:02.062955+0000","last_unstale":"2024-09-21T13:50:10.819771+0000","last_undegraded":"2024-09-21T13:50:10.819771+0000","last_fullsized":"2024-09-21T13:50:10.819771+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":52,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:10.819896+0000","last_change":"2024-09-21T13:49:04.076445+0000","last_active":"2024-09-21T13:50:10.819896+0000","last_peered":"2024-09-21T13:50:10.819896+0000","last_clean":"2024-09-21T13:50:10.819896+0000","last_became_active":"2024-09-21T13:49:04.075818+0000","last_became_peered":"2024-09-21T13:49:04.075818+0000","last_unstale":"2024-09-21T13:50:10.819896+0000","last_undegraded":"2024-09-21T13:50:10.819896+0000","last_fullsized":"2024-09-21T13:50:10.819896+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:31.482751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"87'82","reported_seq":101,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:13.339749+0000","last_change":"2024-09-21T13:50:07.419199+0000","last_active":"2024-09-21T13:50:13.339749+0000","last_peered":"2024-09-21T13:50:13.339749+0000","last_clean":"2024-09-21T13:50:13.339749+0000","last_became_active":"2024-09-21T13:50:07.418879+0000","last_became_peered":"2024-09-21T13:50:07.418879+0000","last_unstale":"2024-09-21T13:50:13.339749+0000","last_undegraded":"2024-09-21T13:50:13.339749+0000","last_fullsized":"2024-09-21T13:50:13.339749+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:51:21.118740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5373952,"num_objects":82,"num_object_clones":0,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":82,"num_write_kb":5248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"68'14","reported_seq":81,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:10.819778+0000","last_change":"2024-09-21T13:49:00.055423+0000","last_active":"2024-09-21T13:50:10.819778+0000","last_peered":"2024-09-21T13:50:10.819778+0000","last_clean":"2024-09-21T13:50:10.819778+0000","last_became_active":"2024-09-21T13:49:00.055031+0000","last_became_peered":"2024-09-21T13:49:00.055031+0000","last_unstale":"2024-09-21T13:50:10.819778+0000","last_undegraded":"2024-09-21T13:50:10.819778+0000","last_fullsized":"2024-09-21T13:50:10.819778+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":73,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:10.820888+0000","last_change":"2024-09-21T13:49:59.253636+0000","last_active":"2024-09-21T13:50:10.820888+0000","last_peered":"2024-09-21T13:50:10.820888+0000","last_clean":"2024-09-21T13:50:10.820888+0000","last_became_active":"2024-09-21T13:48:58.053533+0000","last_became_peered":"2024-09-21T13:48:58.053533+0000","last_unstale":"2024-09-21T13:50:10.820888+0000","last_undegraded":"2024-09-21T13:50:10.820888+0000","last_fullsized":"2024-09-21T13:50:10.820888+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:49:59.253587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:49:59.253587+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:51:28.789661+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"68'8","reported_seq":80,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.240986+0000","last_change":"2024-09-21T13:49:02.070405+0000","last_active":"2024-09-21T13:50:12.240986+0000","last_peered":"2024-09-21T13:50:12.240986+0000","last_clean":"2024-09-21T13:50:12.240986+0000","last_became_active":"2024-09-21T13:49:02.070064+0000","last_became_peered":"2024-09-21T13:49:02.070064+0000","last_unstale":"2024-09-21T13:50:12.240986+0000","last_undegraded":"2024-09-21T13:50:12.240986+0000","last_fullsized":"2024-09-21T13:50:12.240986+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:13.746404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":52,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.242521+0000","last_change":"2024-09-21T13:49:04.434498+0000","last_active":"2024-09-21T13:50:12.242521+0000","last_peered":"2024-09-21T13:50:12.242521+0000","last_clean":"2024-09-21T13:50:12.242521+0000","last_became_active":"2024-09-21T13:49:04.433920+0000","last_became_peered":"2024-09-21T13:49:04.433920+0000","last_unstale":"2024-09-21T13:50:12.242521+0000","last_undegraded":"2024-09-21T13:50:12.242521+0000","last_fullsized":"2024-09-21T13:50:12.242521+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:22.847243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"87'80","reported_seq":99,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:13.321028+0000","last_change":"2024-09-21T13:50:07.419221+0000","last_active":"2024-09-21T13:50:13.321028+0000","last_peered":"2024-09-21T13:50:13.321028+0000","last_clean":"2024-09-21T13:50:13.321028+0000","last_became_active":"2024-09-21T13:50:07.418929+0000","last_became_peered":"2024-09-21T13:50:07.418929+0000","last_unstale":"2024-09-21T13:50:13.321028+0000","last_undegraded":"2024-09-21T13:50:13.321028+0000","last_fullsized":"2024-09-21T13:50:13.321028+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:51:34.498237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":80,"num_object_clones":0,"num_object_copies":240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80,"num_write_kb":5120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"68'10","reported_seq":75,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:10.820306+0000","last_change":"2024-09-21T13:49:00.054972+0000","last_active":"2024-09-21T13:50:10.820306+0000","last_peered":"2024-09-21T13:50:10.820306+0000","last_clean":"2024-09-21T13:50:10.820306+0000","last_became_active":"2024-09-21T13:49:00.054564+0000","last_became_peered":"2024-09-21T13:49:00.054564+0000","last_unstale":"2024-09-21T13:50:10.820306+0000","last_undegraded":"2024-09-21T13:50:10.820306+0000","last_fullsized":"2024-09-21T13:50:10.820306+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:18.647498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"61'1","reported_seq":70,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.241486+0000","last_change":"2024-09-21T13:48:58.050844+0000","last_active":"2024-09-21T13:50:12.241486+0000","last_peered":"2024-09-21T13:50:12.241486+0000","last_clean":"2024-09-21T13:50:12.241486+0000","last_became_active":"2024-09-21T13:48:58.050548+0000","last_became_peered":"2024-09-21T13:48:58.050548+0000","last_unstale":"2024-09-21T13:50:12.241486+0000","last_undegraded":"2024-09-21T13:50:12.241486+0000","last_fullsized":"2024-09-21T13:50:12.241486+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:17.587266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"68'8","reported_seq":80,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:10.819945+0000","last_change":"2024-09-21T13:49:02.068926+0000","last_active":"2024-09-21T13:50:10.819945+0000","last_peered":"2024-09-21T13:50:10.819945+0000","last_clean":"2024-09-21T13:50:10.819945+0000","last_became_active":"2024-09-21T13:49:02.067664+0000","last_became_peered":"2024-09-21T13:49:02.067664+0000","last_unstale":"2024-09-21T13:50:10.819945+0000","last_undegraded":"2024-09-21T13:50:10.819945+0000","last_fullsized":"2024-09-21T13:50:10.819945+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":52,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:10.818228+0000","last_change":"2024-09-21T13:49:04.081771+0000","last_active":"2024-09-21T13:50:10.818228+0000","last_peered":"2024-09-21T13:50:10.818228+0000","last_clean":"2024-09-21T13:50:10.818228+0000","last_became_active":"2024-09-21T13:49:04.081462+0000","last_became_peered":"2024-09-21T13:49:04.081462+0000","last_unstale":"2024-09-21T13:50:10.818228+0000","last_undegraded":"2024-09-21T13:50:10.818228+0000","last_fullsized":"2024-09-21T13:50:10.818228+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:25.632205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"87'86","reported_seq":105,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:13.329184+0000","last_change":"2024-09-21T13:50:07.424005+0000","last_active":"2024-09-21T13:50:13.329184+0000","last_peered":"2024-09-21T13:50:13.329184+0000","last_clean":"2024-09-21T13:50:13.329184+0000","last_became_active":"2024-09-21T13:50:07.423702+0000","last_became_peered":"2024-09-21T13:50:07.423702+0000","last_unstale":"2024-09-21T13:50:13.329184+0000","last_undegraded":"2024-09-21T13:50:13.329184+0000","last_fullsized":"2024-09-21T13:50:13.329184+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:51:07.704975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5636096,"num_objects":86,"num_object_clones":0,"num_object_copies":258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":86,"num_write_kb":5504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"68'19","reported_seq":96,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:10.820081+0000","last_change":"2024-09-21T13:49:00.426917+0000","last_active":"2024-09-21T13:50:10.820081+0000","last_peered":"2024-09-21T13:50:10.820081+0000","last_clean":"2024-09-21T13:50:10.820081+0000","last_became_active":"2024-09-21T13:49:00.426579+0000","last_became_peered":"2024-09-21T13:49:00.426579+0000","last_unstale":"2024-09-21T13:50:10.820081+0000","last_undegraded":"2024-09-21T13:50:10.820081+0000","last_fullsized":"2024-09-21T13:50:10.820081+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:15.913331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":64,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:10.819874+0000","last_change":"2024-09-21T13:48:58.049961+0000","last_active":"2024-09-21T13:50:10.819874+0000","last_peered":"2024-09-21T13:50:10.819874+0000","last_clean":"2024-09-21T13:50:10.819874+0000","last_became_active":"2024-09-21T13:48:58.049682+0000","last_became_peered":"2024-09-21T13:48:58.049682+0000","last_unstale":"2024-09-21T13:50:10.819874+0000","last_undegraded":"2024-09-21T13:50:10.819874+0000","last_fullsized":"2024-09-21T13:50:10.819874+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:51:10.038761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":56,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.242776+0000","last_change":"2024-09-21T13:49:02.061604+0000","last_active":"2024-09-21T13:50:12.242776+0000","last_peered":"2024-09-21T13:50:12.242776+0000","last_clean":"2024-09-21T13:50:12.242776+0000","last_became_active":"2024-09-21T13:49:02.061187+0000","last_became_peered":"2024-09-21T13:49:02.061187+0000","last_unstale":"2024-09-21T13:50:12.242776+0000","last_undegraded":"2024-09-21T13:50:12.242776+0000","last_fullsized":"2024-09-21T13:50:12.242776+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:51:10.863280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":52,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:10.818085+0000","last_change":"2024-09-21T13:49:04.431651+0000","last_active":"2024-09-21T13:50:10.818085+0000","last_peered":"2024-09-21T13:50:10.818085+0000","last_clean":"2024-09-21T13:50:10.818085+0000","last_became_active":"2024-09-21T13:49:04.431079+0000","last_became_peered":"2024-09-21T13:49:04.431079+0000","last_unstale":"2024-09-21T13:50:10.818085+0000","last_undegraded":"2024-09-21T13:50:10.818085+0000","last_fullsized":"2024-09-21T13:50:10.818085+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"87'86","reported_seq":105,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:13.320018+0000","last_change":"2024-09-21T13:50:07.421958+0000","last_active":"2024-09-21T13:50:13.320018+0000","last_peered":"2024-09-21T13:50:13.320018+0000","last_clean":"2024-09-21T13:50:13.320018+0000","last_became_active":"2024-09-21T13:50:07.421666+0000","last_became_peered":"2024-09-21T13:50:07.421666+0000","last_unstale":"2024-09-21T13:50:13.320018+0000","last_undegraded":"2024-09-21T13:50:13.320018+0000","last_fullsized":"2024-09-21T13:50:13.320018+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:51:34.296774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5636096,"num_objects":86,"num_object_clones":0,"num_object_copies":258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":86,"num_write_kb":5504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"68'28","reported_seq":124,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:10.819777+0000","last_change":"2024-09-21T13:49:00.060616+0000","last_active":"2024-09-21T13:50:10.819777+0000","last_peered":"2024-09-21T13:50:10.819777+0000","last_clean":"2024-09-21T13:50:10.819777+0000","last_became_active":"2024-09-21T13:49:00.060355+0000","last_became_peered":"2024-09-21T13:49:00.060355+0000","last_unstale":"2024-09-21T13:50:10.819777+0000","last_undegraded":"2024-09-21T13:50:10.819777+0000","last_fullsized":"2024-09-21T13:50:10.819777+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:26.214000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":64,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:10.820064+0000","last_change":"2024-09-21T13:48:58.055640+0000","last_active":"2024-09-21T13:50:10.820064+0000","last_peered":"2024-09-21T13:50:10.820064+0000","last_clean":"2024-09-21T13:50:10.820064+0000","last_became_active":"2024-09-21T13:48:58.055366+0000","last_became_peered":"2024-09-21T13:48:58.055366+0000","last_unstale":"2024-09-21T13:50:10.820064+0000","last_undegraded":"2024-09-21T13:50:10.820064+0000","last_fullsized":"2024-09-21T13:50:10.820064+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:17.442979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":74,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.242083+0000","last_change":"2024-09-21T13:49:00.053256+0000","last_active":"2024-09-21T13:50:12.242083+0000","last_peered":"2024-09-21T13:50:12.242083+0000","last_clean":"2024-09-21T13:50:12.242083+0000","last_became_active":"2024-09-21T13:48:57.046246+0000","last_became_peered":"2024-09-21T13:48:57.046246+0000","last_unstale":"2024-09-21T13:50:12.242083+0000","last_undegraded":"2024-09-21T13:50:12.242083+0000","last_fullsized":"2024-09-21T13:50:12.242083+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:48:56.029323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015946669999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":64,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:10.819057+0000","last_change":"2024-09-21T13:50:03.135744+0000","last_active":"2024-09-21T13:50:10.819057+0000","last_peered":"2024-09-21T13:50:10.819057+0000","last_clean":"2024-09-21T13:50:10.819057+0000","last_became_active":"2024-09-21T13:49:02.068611+0000","last_became_peered":"2024-09-21T13:49:02.068611+0000","last_unstale":"2024-09-21T13:50:10.819057+0000","last_undegraded":"2024-09-21T13:50:10.819057+0000","last_fullsized":"2024-09-21T13:50:10.819057+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:03.135687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:03.135687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:51:31.776506+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":52,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.241748+0000","last_change":"2024-09-21T13:49:04.080755+0000","last_active":"2024-09-21T13:50:12.241748+0000","last_peered":"2024-09-21T13:50:12.241748+0000","last_clean":"2024-09-21T13:50:12.241748+0000","last_became_active":"2024-09-21T13:49:04.080466+0000","last_became_peered":"2024-09-21T13:49:04.080466+0000","last_unstale":"2024-09-21T13:50:12.241748+0000","last_undegraded":"2024-09-21T13:50:12.241748+0000","last_fullsized":"2024-09-21T13:50:12.241748+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"87'67","reported_seq":86,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:13.317257+0000","last_change":"2024-09-21T13:50:07.422954+0000","last_active":"2024-09-21T13:50:13.317257+0000","last_peered":"2024-09-21T13:50:13.317257+0000","last_clean":"2024-09-21T13:50:13.317257+0000","last_became_active":"2024-09-21T13:50:07.422349+0000","last_became_peered":"2024-09-21T13:50:07.422349+0000","last_unstale":"2024-09-21T13:50:13.317257+0000","last_undegraded":"2024-09-21T13:50:13.317257+0000","last_fullsized":"2024-09-21T13:50:13.317257+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:51:13.743126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4390912,"num_objects":67,"num_object_clones":0,"num_object_copies":201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":67,"num_write_kb":4288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"68'13","reported_seq":95,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:10.818739+0000","last_change":"2024-09-21T13:50:06.973399+0000","last_active":"2024-09-21T13:50:10.818739+0000","last_peered":"2024-09-21T13:50:10.818739+0000","last_clean":"2024-09-21T13:50:10.818739+0000","last_became_active":"2024-09-21T13:49:00.053323+0000","last_became_peered":"2024-09-21T13:49:00.053323+0000","last_unstale":"2024-09-21T13:50:10.818739+0000","last_undegraded":"2024-09-21T13:50:10.818739+0000","last_fullsized":"2024-09-21T13:50:10.818739+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'13","last_scrub_stamp":"2024-09-21T13:50:06.973336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.973336+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:51:25.069546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":64,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:10.818050+0000","last_change":"2024-09-21T13:48:58.050687+0000","last_active":"2024-09-21T13:50:10.818050+0000","last_peered":"2024-09-21T13:50:10.818050+0000","last_clean":"2024-09-21T13:50:10.818050+0000","last_became_active":"2024-09-21T13:48:58.050371+0000","last_became_peered":"2024-09-21T13:48:58.050371+0000","last_unstale":"2024-09-21T13:50:10.818050+0000","last_undegraded":"2024-09-21T13:50:10.818050+0000","last_fullsized":"2024-09-21T13:50:10.818050+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:19.757852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":81,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:11.784530+0000","last_change":"2024-09-21T13:49:56.756325+0000","last_active":"2024-09-21T13:50:11.784530+0000","last_peered":"2024-09-21T13:50:11.784530+0000","last_clean":"2024-09-21T13:50:11.784530+0000","last_became_active":"2024-09-21T13:48:57.047067+0000","last_became_peered":"2024-09-21T13:48:57.047067+0000","last_unstale":"2024-09-21T13:50:11.784530+0000","last_undegraded":"2024-09-21T13:50:11.784530+0000","last_fullsized":"2024-09-21T13:50:11.784530+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:49:56.756268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:49:56.756268+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:51:12.311848+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00065666300000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":56,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:11.784878+0000","last_change":"2024-09-21T13:49:02.057270+0000","last_active":"2024-09-21T13:50:11.784878+0000","last_peered":"2024-09-21T13:50:11.784878+0000","last_clean":"2024-09-21T13:50:11.784878+0000","last_became_active":"2024-09-21T13:49:02.056806+0000","last_became_peered":"2024-09-21T13:49:02.056806+0000","last_unstale":"2024-09-21T13:50:11.784878+0000","last_undegraded":"2024-09-21T13:50:11.784878+0000","last_fullsized":"2024-09-21T13:50:11.784878+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:17.996686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":60,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.242734+0000","last_change":"2024-09-21T13:50:04.164889+0000","last_active":"2024-09-21T13:50:12.242734+0000","last_peered":"2024-09-21T13:50:12.242734+0000","last_clean":"2024-09-21T13:50:12.242734+0000","last_became_active":"2024-09-21T13:49:04.431035+0000","last_became_peered":"2024-09-21T13:49:04.431035+0000","last_unstale":"2024-09-21T13:50:12.242734+0000","last_undegraded":"2024-09-21T13:50:12.242734+0000","last_fullsized":"2024-09-21T13:50:12.242734+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:04.164815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:04.164815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:51:07.597183+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"87'76","reported_seq":95,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:13.334062+0000","last_change":"2024-09-21T13:50:07.425663+0000","last_active":"2024-09-21T13:50:13.334062+0000","last_peered":"2024-09-21T13:50:13.334062+0000","last_clean":"2024-09-21T13:50:13.334062+0000","last_became_active":"2024-09-21T13:50:07.425323+0000","last_became_peered":"2024-09-21T13:50:07.425323+0000","last_unstale":"2024-09-21T13:50:13.334062+0000","last_undegraded":"2024-09-21T13:50:13.334062+0000","last_fullsized":"2024-09-21T13:50:13.334062+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:51:15.706707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4980736,"num_objects":76,"num_object_clones":0,"num_object_copies":228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":76,"num_write_kb":4864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"68'12","reported_seq":78,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:10.820791+0000","last_change":"2024-09-21T13:49:00.053466+0000","last_active":"2024-09-21T13:50:10.820791+0000","last_peered":"2024-09-21T13:50:10.820791+0000","last_clean":"2024-09-21T13:50:10.820791+0000","last_became_active":"2024-09-21T13:49:00.053080+0000","last_became_peered":"2024-09-21T13:49:00.053080+0000","last_unstale":"2024-09-21T13:50:10.820791+0000","last_undegraded":"2024-09-21T13:50:10.820791+0000","last_fullsized":"2024-09-21T13:50:10.820791+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":64,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:10.819295+0000","last_change":"2024-09-21T13:48:58.051534+0000","last_active":"2024-09-21T13:50:10.819295+0000","last_peered":"2024-09-21T13:50:10.819295+0000","last_clean":"2024-09-21T13:50:10.819295+0000","last_became_active":"2024-09-21T13:48:58.051249+0000","last_became_peered":"2024-09-21T13:48:58.051249+0000","last_unstale":"2024-09-21T13:50:10.819295+0000","last_undegraded":"2024-09-21T13:50:10.819295+0000","last_fullsized":"2024-09-21T13:50:10.819295+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":145,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.241347+0000","last_change":"2024-09-21T13:49:00.426330+0000","last_active":"2024-09-21T13:50:12.241347+0000","last_peered":"2024-09-21T13:50:12.241347+0000","last_clean":"2024-09-21T13:50:12.241347+0000","last_became_active":"2024-09-21T13:48:57.046084+0000","last_became_peered":"2024-09-21T13:48:57.046084+0000","last_unstale":"2024-09-21T13:50:12.241347+0000","last_undegraded":"2024-09-21T13:50:12.241347+0000","last_fullsized":"2024-09-21T13:50:12.241347+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:48:56.029323+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047048000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":70,"num_read_kb":65,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":56,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.242203+0000","last_change":"2024-09-21T13:49:02.066942+0000","last_active":"2024-09-21T13:50:12.242203+0000","last_peered":"2024-09-21T13:50:12.242203+0000","last_clean":"2024-09-21T13:50:12.242203+0000","last_became_active":"2024-09-21T13:49:02.066641+0000","last_became_peered":"2024-09-21T13:49:02.066641+0000","last_unstale":"2024-09-21T13:50:12.242203+0000","last_undegraded":"2024-09-21T13:50:12.242203+0000","last_fullsized":"2024-09-21T13:50:12.242203+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:22.324406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":52,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:10.819402+0000","last_change":"2024-09-21T13:49:04.078686+0000","last_active":"2024-09-21T13:50:10.819402+0000","last_peered":"2024-09-21T13:50:10.819402+0000","last_clean":"2024-09-21T13:50:10.819402+0000","last_became_active":"2024-09-21T13:49:04.078372+0000","last_became_peered":"2024-09-21T13:49:04.078372+0000","last_unstale":"2024-09-21T13:50:10.819402+0000","last_undegraded":"2024-09-21T13:50:10.819402+0000","last_fullsized":"2024-09-21T13:50:10.819402+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:25.621989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"87'87","reported_seq":106,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:13.332820+0000","last_change":"2024-09-21T13:50:07.424355+0000","last_active":"2024-09-21T13:50:13.332820+0000","last_peered":"2024-09-21T13:50:13.332820+0000","last_clean":"2024-09-21T13:50:13.332820+0000","last_became_active":"2024-09-21T13:50:07.423287+0000","last_became_peered":"2024-09-21T13:50:07.423287+0000","last_unstale":"2024-09-21T13:50:13.332820+0000","last_undegraded":"2024-09-21T13:50:13.332820+0000","last_fullsized":"2024-09-21T13:50:13.332820+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:51:24.268943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5701632,"num_objects":87,"num_object_clones":0,"num_object_copies":261,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":87,"num_write_kb":5568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"68'16","reported_seq":98,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.242071+0000","last_change":"2024-09-21T13:49:00.427430+0000","last_active":"2024-09-21T13:50:12.242071+0000","last_peered":"2024-09-21T13:50:12.242071+0000","last_clean":"2024-09-21T13:50:12.242071+0000","last_became_active":"2024-09-21T13:49:00.426867+0000","last_became_peered":"2024-09-21T13:49:00.426867+0000","last_unstale":"2024-09-21T13:50:12.242071+0000","last_undegraded":"2024-09-21T13:50:12.242071+0000","last_fullsized":"2024-09-21T13:50:12.242071+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":72,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.241395+0000","last_change":"2024-09-21T13:50:09.208053+0000","last_active":"2024-09-21T13:50:12.241395+0000","last_peered":"2024-09-21T13:50:12.241395+0000","last_clean":"2024-09-21T13:50:12.241395+0000","last_became_active":"2024-09-21T13:48:58.052659+0000","last_became_peered":"2024-09-21T13:48:58.052659+0000","last_unstale":"2024-09-21T13:50:12.241395+0000","last_undegraded":"2024-09-21T13:50:12.241395+0000","last_fullsized":"2024-09-21T13:50:12.241395+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:09.208003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:09.208003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:51:25.602783+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"75'79","reported_seq":113,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.241487+0000","last_change":"2024-09-21T13:48:51.893262+0000","last_active":"2024-09-21T13:50:12.241487+0000","last_peered":"2024-09-21T13:50:12.241487+0000","last_clean":"2024-09-21T13:50:12.241487+0000","last_became_active":"2024-09-21T13:48:48.454646+0000","last_became_peered":"2024-09-21T13:48:48.454646+0000","last_unstale":"2024-09-21T13:50:12.241487+0000","last_undegraded":"2024-09-21T13:50:12.241487+0000","last_fullsized":"2024-09-21T13:50:12.241487+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'72","last_scrub_stamp":"2024-09-21T13:48:51.893178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:48:51.893178+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":56,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.241553+0000","last_change":"2024-09-21T13:49:02.061930+0000","last_active":"2024-09-21T13:50:12.241553+0000","last_peered":"2024-09-21T13:50:12.241553+0000","last_clean":"2024-09-21T13:50:12.241553+0000","last_became_active":"2024-09-21T13:49:02.061454+0000","last_became_peered":"2024-09-21T13:49:02.061454+0000","last_unstale":"2024-09-21T13:50:12.241553+0000","last_undegraded":"2024-09-21T13:50:12.241553+0000","last_fullsized":"2024-09-21T13:50:12.241553+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:23.000364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":52,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.241992+0000","last_change":"2024-09-21T13:49:04.078343+0000","last_active":"2024-09-21T13:50:12.241992+0000","last_peered":"2024-09-21T13:50:12.241992+0000","last_clean":"2024-09-21T13:50:12.241992+0000","last_became_active":"2024-09-21T13:49:04.078091+0000","last_became_peered":"2024-09-21T13:49:04.078091+0000","last_unstale":"2024-09-21T13:50:12.241992+0000","last_undegraded":"2024-09-21T13:50:12.241992+0000","last_fullsized":"2024-09-21T13:50:12.241992+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:17.101054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"87'68","reported_seq":87,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:13.308324+0000","last_change":"2024-09-21T13:50:07.422762+0000","last_active":"2024-09-21T13:50:13.308324+0000","last_peered":"2024-09-21T13:50:13.308324+0000","last_clean":"2024-09-21T13:50:13.308324+0000","last_became_active":"2024-09-21T13:50:07.422466+0000","last_became_peered":"2024-09-21T13:50:07.422466+0000","last_unstale":"2024-09-21T13:50:13.308324+0000","last_undegraded":"2024-09-21T13:50:13.308324+0000","last_fullsized":"2024-09-21T13:50:13.308324+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:51:17.664741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4456448,"num_objects":68,"num_object_clones":0,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68,"num_write_kb":4352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"68'11","reported_seq":79,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:10.820416+0000","last_change":"2024-09-21T13:49:00.054827+0000","last_active":"2024-09-21T13:50:10.820416+0000","last_peered":"2024-09-21T13:50:10.820416+0000","last_clean":"2024-09-21T13:50:10.820416+0000","last_became_active":"2024-09-21T13:49:00.053268+0000","last_became_peered":"2024-09-21T13:49:00.053268+0000","last_unstale":"2024-09-21T13:50:10.820416+0000","last_undegraded":"2024-09-21T13:50:10.820416+0000","last_fullsized":"2024-09-21T13:50:10.820416+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:28.403874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":64,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:10.820010+0000","last_change":"2024-09-21T13:48:58.052364+0000","last_active":"2024-09-21T13:50:10.820010+0000","last_peered":"2024-09-21T13:50:10.820010+0000","last_clean":"2024-09-21T13:50:10.820010+0000","last_became_active":"2024-09-21T13:48:58.052092+0000","last_became_peered":"2024-09-21T13:48:58.052092+0000","last_unstale":"2024-09-21T13:50:10.820010+0000","last_undegraded":"2024-09-21T13:50:10.820010+0000","last_fullsized":"2024-09-21T13:50:10.820010+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":56,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.241905+0000","last_change":"2024-09-21T13:49:02.062262+0000","last_active":"2024-09-21T13:50:12.241905+0000","last_peered":"2024-09-21T13:50:12.241905+0000","last_clean":"2024-09-21T13:50:12.241905+0000","last_became_active":"2024-09-21T13:49:02.061901+0000","last_became_peered":"2024-09-21T13:49:02.061901+0000","last_unstale":"2024-09-21T13:50:12.241905+0000","last_undegraded":"2024-09-21T13:50:12.241905+0000","last_fullsized":"2024-09-21T13:50:12.241905+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:20.147536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":52,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.241428+0000","last_change":"2024-09-21T13:49:04.431695+0000","last_active":"2024-09-21T13:50:12.241428+0000","last_peered":"2024-09-21T13:50:12.241428+0000","last_clean":"2024-09-21T13:50:12.241428+0000","last_became_active":"2024-09-21T13:49:04.431268+0000","last_became_peered":"2024-09-21T13:49:04.431268+0000","last_unstale":"2024-09-21T13:50:12.241428+0000","last_undegraded":"2024-09-21T13:50:12.241428+0000","last_fullsized":"2024-09-21T13:50:12.241428+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:27.118735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"87'90","reported_seq":107,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:13.327769+0000","last_change":"2024-09-21T13:50:07.424412+0000","last_active":"2024-09-21T13:50:13.327769+0000","last_peered":"2024-09-21T13:50:13.327769+0000","last_clean":"2024-09-21T13:50:13.327769+0000","last_became_active":"2024-09-21T13:50:07.424142+0000","last_became_peered":"2024-09-21T13:50:07.424142+0000","last_unstale":"2024-09-21T13:50:13.327769+0000","last_undegraded":"2024-09-21T13:50:13.327769+0000","last_fullsized":"2024-09-21T13:50:13.327769+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:51:36.054268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5898240,"num_objects":90,"num_object_clones":0,"num_object_copies":270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90,"num_write_kb":5760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"68'17","reported_seq":88,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:10.820439+0000","last_change":"2024-09-21T13:49:00.053077+0000","last_active":"2024-09-21T13:50:10.820439+0000","last_peered":"2024-09-21T13:50:10.820439+0000","last_clean":"2024-09-21T13:50:10.820439+0000","last_became_active":"2024-09-21T13:49:00.052608+0000","last_became_peered":"2024-09-21T13:49:00.052608+0000","last_unstale":"2024-09-21T13:50:10.820439+0000","last_undegraded":"2024-09-21T13:50:10.820439+0000","last_fullsized":"2024-09-21T13:50:10.820439+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:14.266475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":72,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.243126+0000","last_change":"2024-09-21T13:50:07.839026+0000","last_active":"2024-09-21T13:50:12.243126+0000","last_peered":"2024-09-21T13:50:12.243126+0000","last_clean":"2024-09-21T13:50:12.243126+0000","last_became_active":"2024-09-21T13:48:58.049142+0000","last_became_peered":"2024-09-21T13:48:58.049142+0000","last_unstale":"2024-09-21T13:50:12.243126+0000","last_undegraded":"2024-09-21T13:50:12.243126+0000","last_fullsized":"2024-09-21T13:50:12.243126+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:07.838979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:07.838979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:51:31.729732+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":56,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:10.819496+0000","last_change":"2024-09-21T13:49:02.067811+0000","last_active":"2024-09-21T13:50:10.819496+0000","last_peered":"2024-09-21T13:50:10.819496+0000","last_clean":"2024-09-21T13:50:10.819496+0000","last_became_active":"2024-09-21T13:49:02.067472+0000","last_became_peered":"2024-09-21T13:49:02.067472+0000","last_unstale":"2024-09-21T13:50:10.819496+0000","last_undegraded":"2024-09-21T13:50:10.819496+0000","last_fullsized":"2024-09-21T13:50:10.819496+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:29.225485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":52,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:11.784851+0000","last_change":"2024-09-21T13:49:04.078285+0000","last_active":"2024-09-21T13:50:11.784851+0000","last_peered":"2024-09-21T13:50:11.784851+0000","last_clean":"2024-09-21T13:50:11.784851+0000","last_became_active":"2024-09-21T13:49:04.077962+0000","last_became_peered":"2024-09-21T13:49:04.077962+0000","last_unstale":"2024-09-21T13:50:11.784851+0000","last_undegraded":"2024-09-21T13:50:11.784851+0000","last_fullsized":"2024-09-21T13:50:11.784851+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:13.659310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"87'69","reported_seq":88,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:13.327034+0000","last_change":"2024-09-21T13:50:07.425267+0000","last_active":"2024-09-21T13:50:13.327034+0000","last_peered":"2024-09-21T13:50:13.327034+0000","last_clean":"2024-09-21T13:50:13.327034+0000","last_became_active":"2024-09-21T13:50:07.424962+0000","last_became_peered":"2024-09-21T13:50:07.424962+0000","last_unstale":"2024-09-21T13:50:13.327034+0000","last_undegraded":"2024-09-21T13:50:13.327034+0000","last_fullsized":"2024-09-21T13:50:13.327034+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:51:29.772580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4521984,"num_objects":69,"num_object_clones":0,"num_object_copies":207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":69,"num_write_kb":4416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"68'10","reported_seq":75,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:10.820100+0000","last_change":"2024-09-21T13:49:00.060182+0000","last_active":"2024-09-21T13:50:10.820100+0000","last_peered":"2024-09-21T13:50:10.820100+0000","last_clean":"2024-09-21T13:50:10.820100+0000","last_became_active":"2024-09-21T13:49:00.059850+0000","last_became_peered":"2024-09-21T13:49:00.059850+0000","last_unstale":"2024-09-21T13:50:10.820100+0000","last_undegraded":"2024-09-21T13:50:10.820100+0000","last_fullsized":"2024-09-21T13:50:10.820100+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:16.066823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":64,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.241853+0000","last_change":"2024-09-21T13:48:58.058603+0000","last_active":"2024-09-21T13:50:12.241853+0000","last_peered":"2024-09-21T13:50:12.241853+0000","last_clean":"2024-09-21T13:50:12.241853+0000","last_became_active":"2024-09-21T13:48:58.058309+0000","last_became_peered":"2024-09-21T13:48:58.058309+0000","last_unstale":"2024-09-21T13:50:12.241853+0000","last_undegraded":"2024-09-21T13:50:12.241853+0000","last_fullsized":"2024-09-21T13:50:12.241853+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"68'8","reported_seq":80,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:11.784645+0000","last_change":"2024-09-21T13:49:02.065220+0000","last_active":"2024-09-21T13:50:11.784645+0000","last_peered":"2024-09-21T13:50:11.784645+0000","last_clean":"2024-09-21T13:50:11.784645+0000","last_became_active":"2024-09-21T13:49:02.064817+0000","last_became_peered":"2024-09-21T13:49:02.064817+0000","last_unstale":"2024-09-21T13:50:11.784645+0000","last_undegraded":"2024-09-21T13:50:11.784645+0000","last_fullsized":"2024-09-21T13:50:11.784645+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"68'1","reported_seq":56,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:10.820158+0000","last_change":"2024-09-21T13:50:10.820158+0000","last_active":"2024-09-21T13:50:10.820158+0000","last_peered":"2024-09-21T13:50:10.820158+0000","last_clean":"2024-09-21T13:50:10.820158+0000","last_became_active":"2024-09-21T13:49:04.071885+0000","last_became_peered":"2024-09-21T13:49:04.071885+0000","last_unstale":"2024-09-21T13:50:10.820158+0000","last_undegraded":"2024-09-21T13:50:10.820158+0000","last_fullsized":"2024-09-21T13:50:10.820158+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:39.711957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"87'74","reported_seq":93,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:13.320979+0000","last_change":"2024-09-21T13:50:07.422425+0000","last_active":"2024-09-21T13:50:13.320979+0000","last_peered":"2024-09-21T13:50:13.320979+0000","last_clean":"2024-09-21T13:50:13.320979+0000","last_became_active":"2024-09-21T13:50:07.422122+0000","last_became_peered":"2024-09-21T13:50:07.422122+0000","last_unstale":"2024-09-21T13:50:13.320979+0000","last_undegraded":"2024-09-21T13:50:13.320979+0000","last_fullsized":"2024-09-21T13:50:13.320979+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:51:07.432954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4849664,"num_objects":74,"num_object_clones":0,"num_object_copies":222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":74,"num_write_kb":4736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"68'9","reported_seq":76,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:10.819994+0000","last_change":"2024-09-21T13:49:00.053349+0000","last_active":"2024-09-21T13:50:10.819994+0000","last_peered":"2024-09-21T13:50:10.819994+0000","last_clean":"2024-09-21T13:50:10.819994+0000","last_became_active":"2024-09-21T13:49:00.053016+0000","last_became_peered":"2024-09-21T13:49:00.053016+0000","last_unstale":"2024-09-21T13:50:10.819994+0000","last_undegraded":"2024-09-21T13:50:10.819994+0000","last_fullsized":"2024-09-21T13:50:10.819994+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":64,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.241913+0000","last_change":"2024-09-21T13:48:58.050243+0000","last_active":"2024-09-21T13:50:12.241913+0000","last_peered":"2024-09-21T13:50:12.241913+0000","last_clean":"2024-09-21T13:50:12.241913+0000","last_became_active":"2024-09-21T13:48:58.049543+0000","last_became_peered":"2024-09-21T13:48:58.049543+0000","last_unstale":"2024-09-21T13:50:12.241913+0000","last_undegraded":"2024-09-21T13:50:12.241913+0000","last_fullsized":"2024-09-21T13:50:12.241913+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:17.729176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":56,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:11.784612+0000","last_change":"2024-09-21T13:49:02.063877+0000","last_active":"2024-09-21T13:50:11.784612+0000","last_peered":"2024-09-21T13:50:11.784612+0000","last_clean":"2024-09-21T13:50:11.784612+0000","last_became_active":"2024-09-21T13:49:02.063525+0000","last_became_peered":"2024-09-21T13:49:02.063525+0000","last_unstale":"2024-09-21T13:50:11.784612+0000","last_undegraded":"2024-09-21T13:50:11.784612+0000","last_fullsized":"2024-09-21T13:50:11.784612+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:30.535568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":52,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:10.819211+0000","last_change":"2024-09-21T13:49:04.081924+0000","last_active":"2024-09-21T13:50:10.819211+0000","last_peered":"2024-09-21T13:50:10.819211+0000","last_clean":"2024-09-21T13:50:10.819211+0000","last_became_active":"2024-09-21T13:49:04.081617+0000","last_became_peered":"2024-09-21T13:49:04.081617+0000","last_unstale":"2024-09-21T13:50:10.819211+0000","last_undegraded":"2024-09-21T13:50:10.819211+0000","last_fullsized":"2024-09-21T13:50:10.819211+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:25.513722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"87'77","reported_seq":96,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:13.333348+0000","last_change":"2024-09-21T13:50:07.422648+0000","last_active":"2024-09-21T13:50:13.333348+0000","last_peered":"2024-09-21T13:50:13.333348+0000","last_clean":"2024-09-21T13:50:13.333348+0000","last_became_active":"2024-09-21T13:50:07.422342+0000","last_became_peered":"2024-09-21T13:50:07.422342+0000","last_unstale":"2024-09-21T13:50:13.333348+0000","last_undegraded":"2024-09-21T13:50:13.333348+0000","last_fullsized":"2024-09-21T13:50:13.333348+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:51:12.890050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5046272,"num_objects":77,"num_object_clones":0,"num_object_copies":231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77,"num_write_kb":4928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"68'19","reported_seq":99,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.242900+0000","last_change":"2024-09-21T13:50:03.769686+0000","last_active":"2024-09-21T13:50:12.242900+0000","last_peered":"2024-09-21T13:50:12.242900+0000","last_clean":"2024-09-21T13:50:12.242900+0000","last_became_active":"2024-09-21T13:49:00.425955+0000","last_became_peered":"2024-09-21T13:49:00.425955+0000","last_unstale":"2024-09-21T13:50:12.242900+0000","last_undegraded":"2024-09-21T13:50:12.242900+0000","last_fullsized":"2024-09-21T13:50:12.242900+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-21T13:50:03.769613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:03.769613+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:51:30.226608+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":64,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.243105+0000","last_change":"2024-09-21T13:48:58.053940+0000","last_active":"2024-09-21T13:50:12.243105+0000","last_peered":"2024-09-21T13:50:12.243105+0000","last_clean":"2024-09-21T13:50:12.243105+0000","last_became_active":"2024-09-21T13:48:58.053641+0000","last_became_peered":"2024-09-21T13:48:58.053641+0000","last_unstale":"2024-09-21T13:50:12.243105+0000","last_undegraded":"2024-09-21T13:50:12.243105+0000","last_fullsized":"2024-09-21T13:50:12.243105+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:25.094666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":56,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:11.784722+0000","last_change":"2024-09-21T13:49:02.066996+0000","last_active":"2024-09-21T13:50:11.784722+0000","last_peered":"2024-09-21T13:50:11.784722+0000","last_clean":"2024-09-21T13:50:11.784722+0000","last_became_active":"2024-09-21T13:49:02.066676+0000","last_became_peered":"2024-09-21T13:49:02.066676+0000","last_unstale":"2024-09-21T13:50:11.784722+0000","last_undegraded":"2024-09-21T13:50:11.784722+0000","last_fullsized":"2024-09-21T13:50:11.784722+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:19.241359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":52,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.241393+0000","last_change":"2024-09-21T13:49:04.078412+0000","last_active":"2024-09-21T13:50:12.241393+0000","last_peered":"2024-09-21T13:50:12.241393+0000","last_clean":"2024-09-21T13:50:12.241393+0000","last_became_active":"2024-09-21T13:49:04.078084+0000","last_became_peered":"2024-09-21T13:49:04.078084+0000","last_unstale":"2024-09-21T13:50:12.241393+0000","last_undegraded":"2024-09-21T13:50:12.241393+0000","last_fullsized":"2024-09-21T13:50:12.241393+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"87'82","reported_seq":100,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:13.324597+0000","last_change":"2024-09-21T13:50:07.423687+0000","last_active":"2024-09-21T13:50:13.324597+0000","last_peered":"2024-09-21T13:50:13.324597+0000","last_clean":"2024-09-21T13:50:13.324597+0000","last_became_active":"2024-09-21T13:50:07.422834+0000","last_became_peered":"2024-09-21T13:50:07.422834+0000","last_unstale":"2024-09-21T13:50:13.324597+0000","last_undegraded":"2024-09-21T13:50:13.324597+0000","last_fullsized":"2024-09-21T13:50:13.324597+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:51:18.372283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5373952,"num_objects":82,"num_object_clones":0,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":82,"num_write_kb":5248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"68'12","reported_seq":83,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:10.819928+0000","last_change":"2024-09-21T13:49:00.060188+0000","last_active":"2024-09-21T13:50:10.819928+0000","last_peered":"2024-09-21T13:50:10.819928+0000","last_clean":"2024-09-21T13:50:10.819928+0000","last_became_active":"2024-09-21T13:49:00.059720+0000","last_became_peered":"2024-09-21T13:49:00.059720+0000","last_unstale":"2024-09-21T13:50:10.819928+0000","last_undegraded":"2024-09-21T13:50:10.819928+0000","last_fullsized":"2024-09-21T13:50:10.819928+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:17.978182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":64,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.242167+0000","last_change":"2024-09-21T13:48:58.058654+0000","last_active":"2024-09-21T13:50:12.242167+0000","last_peered":"2024-09-21T13:50:12.242167+0000","last_clean":"2024-09-21T13:50:12.242167+0000","last_became_active":"2024-09-21T13:48:58.057641+0000","last_became_peered":"2024-09-21T13:48:58.057641+0000","last_unstale":"2024-09-21T13:50:12.242167+0000","last_undegraded":"2024-09-21T13:50:12.242167+0000","last_fullsized":"2024-09-21T13:50:12.242167+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":64,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:11.784351+0000","last_change":"2024-09-21T13:50:02.761568+0000","last_active":"2024-09-21T13:50:11.784351+0000","last_peered":"2024-09-21T13:50:11.784351+0000","last_clean":"2024-09-21T13:50:11.784351+0000","last_became_active":"2024-09-21T13:49:02.057195+0000","last_became_peered":"2024-09-21T13:49:02.057195+0000","last_unstale":"2024-09-21T13:50:11.784351+0000","last_undegraded":"2024-09-21T13:50:11.784351+0000","last_fullsized":"2024-09-21T13:50:11.784351+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:02.761516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:02.761516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:51:04.258639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":52,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.241474+0000","last_change":"2024-09-21T13:49:04.078422+0000","last_active":"2024-09-21T13:50:12.241474+0000","last_peered":"2024-09-21T13:50:12.241474+0000","last_clean":"2024-09-21T13:50:12.241474+0000","last_became_active":"2024-09-21T13:49:04.078089+0000","last_became_peered":"2024-09-21T13:49:04.078089+0000","last_unstale":"2024-09-21T13:50:12.241474+0000","last_undegraded":"2024-09-21T13:50:12.241474+0000","last_fullsized":"2024-09-21T13:50:12.241474+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:18.974401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"87'85","reported_seq":104,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:13.329880+0000","last_change":"2024-09-21T13:50:07.422240+0000","last_active":"2024-09-21T13:50:13.329880+0000","last_peered":"2024-09-21T13:50:13.329880+0000","last_clean":"2024-09-21T13:50:13.329880+0000","last_became_active":"2024-09-21T13:50:07.421962+0000","last_became_peered":"2024-09-21T13:50:07.421962+0000","last_unstale":"2024-09-21T13:50:13.329880+0000","last_undegraded":"2024-09-21T13:50:13.329880+0000","last_fullsized":"2024-09-21T13:50:13.329880+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:51:25.872666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5570560,"num_objects":85,"num_object_clones":0,"num_object_copies":255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":85,"num_write_kb":5440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"68'15","reported_seq":93,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.241078+0000","last_change":"2024-09-21T13:50:04.445409+0000","last_active":"2024-09-21T13:50:12.241078+0000","last_peered":"2024-09-21T13:50:12.241078+0000","last_clean":"2024-09-21T13:50:12.241078+0000","last_became_active":"2024-09-21T13:49:00.425997+0000","last_became_peered":"2024-09-21T13:49:00.425997+0000","last_unstale":"2024-09-21T13:50:12.241078+0000","last_undegraded":"2024-09-21T13:50:12.241078+0000","last_fullsized":"2024-09-21T13:50:12.241078+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:50:04.445328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:04.445328+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:51:11.769218+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":84,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.242919+0000","last_change":"2024-09-21T13:50:08.119154+0000","last_active":"2024-09-21T13:50:12.242919+0000","last_peered":"2024-09-21T13:50:12.242919+0000","last_clean":"2024-09-21T13:50:12.242919+0000","last_became_active":"2024-09-21T13:48:58.057632+0000","last_became_peered":"2024-09-21T13:48:58.057632+0000","last_unstale":"2024-09-21T13:50:12.242919+0000","last_undegraded":"2024-09-21T13:50:12.242919+0000","last_fullsized":"2024-09-21T13:50:12.242919+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T13:50:08.119099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:08.119099+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:51:17.458280+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"68'8","reported_seq":80,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.242471+0000","last_change":"2024-09-21T13:49:02.056471+0000","last_active":"2024-09-21T13:50:12.242471+0000","last_peered":"2024-09-21T13:50:12.242471+0000","last_clean":"2024-09-21T13:50:12.242471+0000","last_became_active":"2024-09-21T13:49:02.056167+0000","last_became_peered":"2024-09-21T13:49:02.056167+0000","last_unstale":"2024-09-21T13:50:12.242471+0000","last_undegraded":"2024-09-21T13:50:12.242471+0000","last_fullsized":"2024-09-21T13:50:12.242471+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":52,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.241834+0000","last_change":"2024-09-21T13:49:04.432809+0000","last_active":"2024-09-21T13:50:12.241834+0000","last_peered":"2024-09-21T13:50:12.241834+0000","last_clean":"2024-09-21T13:50:12.241834+0000","last_became_active":"2024-09-21T13:49:04.432546+0000","last_became_peered":"2024-09-21T13:49:04.432546+0000","last_unstale":"2024-09-21T13:50:12.241834+0000","last_undegraded":"2024-09-21T13:50:12.241834+0000","last_fullsized":"2024-09-21T13:50:12.241834+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":64,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.242896+0000","last_change":"2024-09-21T13:48:58.056383+0000","last_active":"2024-09-21T13:50:12.242896+0000","last_peered":"2024-09-21T13:50:12.242896+0000","last_clean":"2024-09-21T13:50:12.242896+0000","last_became_active":"2024-09-21T13:48:58.056080+0000","last_became_peered":"2024-09-21T13:48:58.056080+0000","last_unstale":"2024-09-21T13:50:12.242896+0000","last_undegraded":"2024-09-21T13:50:12.242896+0000","last_fullsized":"2024-09-21T13:50:12.242896+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:26.306822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"68'6","reported_seq":69,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.242008+0000","last_change":"2024-09-21T13:49:00.427204+0000","last_active":"2024-09-21T13:50:12.242008+0000","last_peered":"2024-09-21T13:50:12.242008+0000","last_clean":"2024-09-21T13:50:12.242008+0000","last_became_active":"2024-09-21T13:49:00.426829+0000","last_became_peered":"2024-09-21T13:49:00.426829+0000","last_unstale":"2024-09-21T13:50:12.242008+0000","last_undegraded":"2024-09-21T13:50:12.242008+0000","last_fullsized":"2024-09-21T13:50:12.242008+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:20.014163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":56,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.241472+0000","last_change":"2024-09-21T13:49:02.062503+0000","last_active":"2024-09-21T13:50:12.241472+0000","last_peered":"2024-09-21T13:50:12.241472+0000","last_clean":"2024-09-21T13:50:12.241472+0000","last_became_active":"2024-09-21T13:49:02.062231+0000","last_became_peered":"2024-09-21T13:49:02.062231+0000","last_unstale":"2024-09-21T13:50:12.241472+0000","last_undegraded":"2024-09-21T13:50:12.241472+0000","last_fullsized":"2024-09-21T13:50:12.241472+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:21.288932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":60,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.242627+0000","last_change":"2024-09-21T13:50:06.158899+0000","last_active":"2024-09-21T13:50:12.242627+0000","last_peered":"2024-09-21T13:50:12.242627+0000","last_clean":"2024-09-21T13:50:12.242627+0000","last_became_active":"2024-09-21T13:49:04.432507+0000","last_became_peered":"2024-09-21T13:49:04.432507+0000","last_unstale":"2024-09-21T13:50:12.242627+0000","last_undegraded":"2024-09-21T13:50:12.242627+0000","last_fullsized":"2024-09-21T13:50:12.242627+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.158822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.158822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:51:17.043836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"68'9","reported_seq":76,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:10.820842+0000","last_change":"2024-09-21T13:49:00.428704+0000","last_active":"2024-09-21T13:50:10.820842+0000","last_peered":"2024-09-21T13:50:10.820842+0000","last_clean":"2024-09-21T13:50:10.820842+0000","last_became_active":"2024-09-21T13:49:00.428330+0000","last_became_peered":"2024-09-21T13:49:00.428330+0000","last_unstale":"2024-09-21T13:50:10.820842+0000","last_undegraded":"2024-09-21T13:50:10.820842+0000","last_fullsized":"2024-09-21T13:50:10.820842+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"61'1","reported_seq":70,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.242544+0000","last_change":"2024-09-21T13:48:58.050948+0000","last_active":"2024-09-21T13:50:12.242544+0000","last_peered":"2024-09-21T13:50:12.242544+0000","last_clean":"2024-09-21T13:50:12.242544+0000","last_became_active":"2024-09-21T13:48:58.050645+0000","last_became_peered":"2024-09-21T13:48:58.050645+0000","last_unstale":"2024-09-21T13:50:12.242544+0000","last_undegraded":"2024-09-21T13:50:12.242544+0000","last_fullsized":"2024-09-21T13:50:12.242544+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:24.159998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":56,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.242001+0000","last_change":"2024-09-21T13:49:02.062075+0000","last_active":"2024-09-21T13:50:12.242001+0000","last_peered":"2024-09-21T13:50:12.242001+0000","last_clean":"2024-09-21T13:50:12.242001+0000","last_became_active":"2024-09-21T13:49:02.061775+0000","last_became_peered":"2024-09-21T13:49:02.061775+0000","last_unstale":"2024-09-21T13:50:12.242001+0000","last_undegraded":"2024-09-21T13:50:12.242001+0000","last_fullsized":"2024-09-21T13:50:12.242001+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:16.428002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":52,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:11.784498+0000","last_change":"2024-09-21T13:49:04.069452+0000","last_active":"2024-09-21T13:50:11.784498+0000","last_peered":"2024-09-21T13:50:11.784498+0000","last_clean":"2024-09-21T13:50:11.784498+0000","last_became_active":"2024-09-21T13:49:04.069142+0000","last_became_peered":"2024-09-21T13:49:04.069142+0000","last_unstale":"2024-09-21T13:50:11.784498+0000","last_undegraded":"2024-09-21T13:50:11.784498+0000","last_fullsized":"2024-09-21T13:50:11.784498+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:28.894052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"68'9","reported_seq":76,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.241296+0000","last_change":"2024-09-21T13:49:00.428080+0000","last_active":"2024-09-21T13:50:12.241296+0000","last_peered":"2024-09-21T13:50:12.241296+0000","last_clean":"2024-09-21T13:50:12.241296+0000","last_became_active":"2024-09-21T13:49:00.427766+0000","last_became_peered":"2024-09-21T13:49:00.427766+0000","last_unstale":"2024-09-21T13:50:12.241296+0000","last_undegraded":"2024-09-21T13:50:12.241296+0000","last_fullsized":"2024-09-21T13:50:12.241296+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:16.384643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":64,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:10.819632+0000","last_change":"2024-09-21T13:48:58.052465+0000","last_active":"2024-09-21T13:50:10.819632+0000","last_peered":"2024-09-21T13:50:10.819632+0000","last_clean":"2024-09-21T13:50:10.819632+0000","last_became_active":"2024-09-21T13:48:58.052151+0000","last_became_peered":"2024-09-21T13:48:58.052151+0000","last_unstale":"2024-09-21T13:50:10.819632+0000","last_undegraded":"2024-09-21T13:50:10.819632+0000","last_fullsized":"2024-09-21T13:50:10.819632+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"68'8","reported_seq":80,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.241568+0000","last_change":"2024-09-21T13:49:02.066442+0000","last_active":"2024-09-21T13:50:12.241568+0000","last_peered":"2024-09-21T13:50:12.241568+0000","last_clean":"2024-09-21T13:50:12.241568+0000","last_became_active":"2024-09-21T13:49:02.065602+0000","last_became_peered":"2024-09-21T13:49:02.065602+0000","last_unstale":"2024-09-21T13:50:12.241568+0000","last_undegraded":"2024-09-21T13:50:12.241568+0000","last_fullsized":"2024-09-21T13:50:12.241568+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":52,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:10.820280+0000","last_change":"2024-09-21T13:49:04.073461+0000","last_active":"2024-09-21T13:50:10.820280+0000","last_peered":"2024-09-21T13:50:10.820280+0000","last_clean":"2024-09-21T13:50:10.820280+0000","last_became_active":"2024-09-21T13:49:04.073188+0000","last_became_peered":"2024-09-21T13:49:04.073188+0000","last_unstale":"2024-09-21T13:50:10.820280+0000","last_undegraded":"2024-09-21T13:50:10.820280+0000","last_fullsized":"2024-09-21T13:50:10.820280+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"68'10","reported_seq":75,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.241658+0000","last_change":"2024-09-21T13:49:00.427262+0000","last_active":"2024-09-21T13:50:12.241658+0000","last_peered":"2024-09-21T13:50:12.241658+0000","last_clean":"2024-09-21T13:50:12.241658+0000","last_became_active":"2024-09-21T13:49:00.426071+0000","last_became_peered":"2024-09-21T13:49:00.426071+0000","last_unstale":"2024-09-21T13:50:12.241658+0000","last_undegraded":"2024-09-21T13:50:12.241658+0000","last_fullsized":"2024-09-21T13:50:12.241658+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:51:10.226221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":64,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.242813+0000","last_change":"2024-09-21T13:48:58.058198+0000","last_active":"2024-09-21T13:50:12.242813+0000","last_peered":"2024-09-21T13:50:12.242813+0000","last_clean":"2024-09-21T13:50:12.242813+0000","last_became_active":"2024-09-21T13:48:58.057587+0000","last_became_peered":"2024-09-21T13:48:58.057587+0000","last_unstale":"2024-09-21T13:50:12.242813+0000","last_undegraded":"2024-09-21T13:50:12.242813+0000","last_fullsized":"2024-09-21T13:50:12.242813+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:19.518354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"68'8","reported_seq":80,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.241139+0000","last_change":"2024-09-21T13:49:02.067827+0000","last_active":"2024-09-21T13:50:12.241139+0000","last_peered":"2024-09-21T13:50:12.241139+0000","last_clean":"2024-09-21T13:50:12.241139+0000","last_became_active":"2024-09-21T13:49:02.067487+0000","last_became_peered":"2024-09-21T13:49:02.067487+0000","last_unstale":"2024-09-21T13:50:12.241139+0000","last_undegraded":"2024-09-21T13:50:12.241139+0000","last_fullsized":"2024-09-21T13:50:12.241139+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:23.003843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":52,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:10.820596+0000","last_change":"2024-09-21T13:49:04.081928+0000","last_active":"2024-09-21T13:50:10.820596+0000","last_peered":"2024-09-21T13:50:10.820596+0000","last_clean":"2024-09-21T13:50:10.820596+0000","last_became_active":"2024-09-21T13:49:04.081609+0000","last_became_peered":"2024-09-21T13:49:04.081609+0000","last_unstale":"2024-09-21T13:50:10.820596+0000","last_undegraded":"2024-09-21T13:50:10.820596+0000","last_fullsized":"2024-09-21T13:50:10.820596+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:31.380912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"68'11","reported_seq":87,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:10.819960+0000","last_change":"2024-09-21T13:50:02.673500+0000","last_active":"2024-09-21T13:50:10.819960+0000","last_peered":"2024-09-21T13:50:10.819960+0000","last_clean":"2024-09-21T13:50:10.819960+0000","last_became_active":"2024-09-21T13:49:00.055024+0000","last_became_peered":"2024-09-21T13:49:00.055024+0000","last_unstale":"2024-09-21T13:50:10.819960+0000","last_undegraded":"2024-09-21T13:50:10.819960+0000","last_fullsized":"2024-09-21T13:50:10.819960+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:50:02.673442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:02.673442+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:51:17.473109+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":64,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:10.820252+0000","last_change":"2024-09-21T13:48:58.051845+0000","last_active":"2024-09-21T13:50:10.820252+0000","last_peered":"2024-09-21T13:50:10.820252+0000","last_clean":"2024-09-21T13:50:10.820252+0000","last_became_active":"2024-09-21T13:48:58.051541+0000","last_became_peered":"2024-09-21T13:48:58.051541+0000","last_unstale":"2024-09-21T13:50:10.820252+0000","last_undegraded":"2024-09-21T13:50:10.820252+0000","last_fullsized":"2024-09-21T13:50:10.820252+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":56,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:10.820199+0000","last_change":"2024-09-21T13:49:02.063328+0000","last_active":"2024-09-21T13:50:10.820199+0000","last_peered":"2024-09-21T13:50:10.820199+0000","last_clean":"2024-09-21T13:50:10.820199+0000","last_became_active":"2024-09-21T13:49:02.062868+0000","last_became_peered":"2024-09-21T13:49:02.062868+0000","last_unstale":"2024-09-21T13:50:10.820199+0000","last_undegraded":"2024-09-21T13:50:10.820199+0000","last_fullsized":"2024-09-21T13:50:10.820199+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:12.568168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":52,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.241076+0000","last_change":"2024-09-21T13:49:04.083254+0000","last_active":"2024-09-21T13:50:12.241076+0000","last_peered":"2024-09-21T13:50:12.241076+0000","last_clean":"2024-09-21T13:50:12.241076+0000","last_became_active":"2024-09-21T13:49:04.082961+0000","last_became_peered":"2024-09-21T13:49:04.082961+0000","last_unstale":"2024-09-21T13:50:12.241076+0000","last_undegraded":"2024-09-21T13:50:12.241076+0000","last_fullsized":"2024-09-21T13:50:12.241076+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"68'9","reported_seq":76,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:10.820361+0000","last_change":"2024-09-21T13:49:00.057817+0000","last_active":"2024-09-21T13:50:10.820361+0000","last_peered":"2024-09-21T13:50:10.820361+0000","last_clean":"2024-09-21T13:50:10.820361+0000","last_became_active":"2024-09-21T13:49:00.057392+0000","last_became_peered":"2024-09-21T13:49:00.057392+0000","last_unstale":"2024-09-21T13:50:10.820361+0000","last_undegraded":"2024-09-21T13:50:10.820361+0000","last_fullsized":"2024-09-21T13:50:10.820361+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":64,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.242995+0000","last_change":"2024-09-21T13:48:58.058032+0000","last_active":"2024-09-21T13:50:12.242995+0000","last_peered":"2024-09-21T13:50:12.242995+0000","last_clean":"2024-09-21T13:50:12.242995+0000","last_became_active":"2024-09-21T13:48:58.057516+0000","last_became_peered":"2024-09-21T13:48:58.057516+0000","last_unstale":"2024-09-21T13:50:12.242995+0000","last_undegraded":"2024-09-21T13:50:12.242995+0000","last_fullsized":"2024-09-21T13:50:12.242995+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:20.885486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":56,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.241566+0000","last_change":"2024-09-21T13:49:02.072264+0000","last_active":"2024-09-21T13:50:12.241566+0000","last_peered":"2024-09-21T13:50:12.241566+0000","last_clean":"2024-09-21T13:50:12.241566+0000","last_became_active":"2024-09-21T13:49:02.071922+0000","last_became_peered":"2024-09-21T13:49:02.071922+0000","last_unstale":"2024-09-21T13:50:12.241566+0000","last_undegraded":"2024-09-21T13:50:12.241566+0000","last_fullsized":"2024-09-21T13:50:12.241566+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:12.773964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":52,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:10.817884+0000","last_change":"2024-09-21T13:49:04.080300+0000","last_active":"2024-09-21T13:50:10.817884+0000","last_peered":"2024-09-21T13:50:10.817884+0000","last_clean":"2024-09-21T13:50:10.817884+0000","last_became_active":"2024-09-21T13:49:04.079950+0000","last_became_peered":"2024-09-21T13:49:04.079950+0000","last_unstale":"2024-09-21T13:50:10.817884+0000","last_undegraded":"2024-09-21T13:50:10.817884+0000","last_fullsized":"2024-09-21T13:50:10.817884+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:20.474523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"68'11","reported_seq":79,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.241174+0000","last_change":"2024-09-21T13:49:00.426487+0000","last_active":"2024-09-21T13:50:12.241174+0000","last_peered":"2024-09-21T13:50:12.241174+0000","last_clean":"2024-09-21T13:50:12.241174+0000","last_became_active":"2024-09-21T13:49:00.426091+0000","last_became_peered":"2024-09-21T13:49:00.426091+0000","last_unstale":"2024-09-21T13:50:12.241174+0000","last_undegraded":"2024-09-21T13:50:12.241174+0000","last_fullsized":"2024-09-21T13:50:12.241174+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":64,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.241557+0000","last_change":"2024-09-21T13:48:58.052281+0000","last_active":"2024-09-21T13:50:12.241557+0000","last_peered":"2024-09-21T13:50:12.241557+0000","last_clean":"2024-09-21T13:50:12.241557+0000","last_became_active":"2024-09-21T13:48:58.051974+0000","last_became_peered":"2024-09-21T13:48:58.051974+0000","last_unstale":"2024-09-21T13:50:12.241557+0000","last_undegraded":"2024-09-21T13:50:12.241557+0000","last_fullsized":"2024-09-21T13:50:12.241557+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:25.614472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":56,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.240920+0000","last_change":"2024-09-21T13:49:02.056666+0000","last_active":"2024-09-21T13:50:12.240920+0000","last_peered":"2024-09-21T13:50:12.240920+0000","last_clean":"2024-09-21T13:50:12.240920+0000","last_became_active":"2024-09-21T13:49:02.056006+0000","last_became_peered":"2024-09-21T13:49:02.056006+0000","last_unstale":"2024-09-21T13:50:12.240920+0000","last_undegraded":"2024-09-21T13:50:12.240920+0000","last_fullsized":"2024-09-21T13:50:12.240920+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:17.511468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":52,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.241849+0000","last_change":"2024-09-21T13:49:04.434380+0000","last_active":"2024-09-21T13:50:12.241849+0000","last_peered":"2024-09-21T13:50:12.241849+0000","last_clean":"2024-09-21T13:50:12.241849+0000","last_became_active":"2024-09-21T13:49:04.434034+0000","last_became_peered":"2024-09-21T13:49:04.434034+0000","last_unstale":"2024-09-21T13:50:12.241849+0000","last_undegraded":"2024-09-21T13:50:12.241849+0000","last_fullsized":"2024-09-21T13:50:12.241849+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:28.547927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"68'4","reported_seq":66,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.240697+0000","last_change":"2024-09-21T13:49:00.063374+0000","last_active":"2024-09-21T13:50:12.240697+0000","last_peered":"2024-09-21T13:50:12.240697+0000","last_clean":"2024-09-21T13:50:12.240697+0000","last_became_active":"2024-09-21T13:49:00.063066+0000","last_became_peered":"2024-09-21T13:49:00.063066+0000","last_unstale":"2024-09-21T13:50:12.240697+0000","last_undegraded":"2024-09-21T13:50:12.240697+0000","last_fullsized":"2024-09-21T13:50:12.240697+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:19.893798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":72,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:10.820830+0000","last_change":"2024-09-21T13:50:04.145769+0000","last_active":"2024-09-21T13:50:10.820830+0000","last_peered":"2024-09-21T13:50:10.820830+0000","last_clean":"2024-09-21T13:50:10.820830+0000","last_became_active":"2024-09-21T13:48:58.059674+0000","last_became_peered":"2024-09-21T13:48:58.059674+0000","last_unstale":"2024-09-21T13:50:10.820830+0000","last_undegraded":"2024-09-21T13:50:10.820830+0000","last_fullsized":"2024-09-21T13:50:10.820830+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:04.145682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:04.145682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:51:25.052293+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":56,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.241179+0000","last_change":"2024-09-21T13:49:02.065043+0000","last_active":"2024-09-21T13:50:12.241179+0000","last_peered":"2024-09-21T13:50:12.241179+0000","last_clean":"2024-09-21T13:50:12.241179+0000","last_became_active":"2024-09-21T13:49:02.064789+0000","last_became_peered":"2024-09-21T13:49:02.064789+0000","last_unstale":"2024-09-21T13:50:12.241179+0000","last_undegraded":"2024-09-21T13:50:12.241179+0000","last_fullsized":"2024-09-21T13:50:12.241179+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":52,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.240963+0000","last_change":"2024-09-21T13:49:04.075170+0000","last_active":"2024-09-21T13:50:12.240963+0000","last_peered":"2024-09-21T13:50:12.240963+0000","last_clean":"2024-09-21T13:50:12.240963+0000","last_became_active":"2024-09-21T13:49:04.074857+0000","last_became_peered":"2024-09-21T13:49:04.074857+0000","last_unstale":"2024-09-21T13:50:12.240963+0000","last_undegraded":"2024-09-21T13:50:12.240963+0000","last_fullsized":"2024-09-21T13:50:12.240963+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:17.971950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":52,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:10.818725+0000","last_change":"2024-09-21T13:49:04.075860+0000","last_active":"2024-09-21T13:50:10.818725+0000","last_peered":"2024-09-21T13:50:10.818725+0000","last_clean":"2024-09-21T13:50:10.818725+0000","last_became_active":"2024-09-21T13:49:04.075562+0000","last_became_peered":"2024-09-21T13:49:04.075562+0000","last_unstale":"2024-09-21T13:50:10.818725+0000","last_undegraded":"2024-09-21T13:50:10.818725+0000","last_fullsized":"2024-09-21T13:50:10.818725+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:21.041626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":64,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.241261+0000","last_change":"2024-09-21T13:48:58.055791+0000","last_active":"2024-09-21T13:50:12.241261+0000","last_peered":"2024-09-21T13:50:12.241261+0000","last_clean":"2024-09-21T13:50:12.241261+0000","last_became_active":"2024-09-21T13:48:58.055497+0000","last_became_peered":"2024-09-21T13:48:58.055497+0000","last_unstale":"2024-09-21T13:50:12.241261+0000","last_undegraded":"2024-09-21T13:50:12.241261+0000","last_fullsized":"2024-09-21T13:50:12.241261+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:19.570880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"68'11","reported_seq":79,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:12.242474+0000","last_change":"2024-09-21T13:49:00.058623+0000","last_active":"2024-09-21T13:50:12.242474+0000","last_peered":"2024-09-21T13:50:12.242474+0000","last_clean":"2024-09-21T13:50:12.242474+0000","last_became_active":"2024-09-21T13:49:00.058259+0000","last_became_peered":"2024-09-21T13:49:00.058259+0000","last_unstale":"2024-09-21T13:50:12.242474+0000","last_undegraded":"2024-09-21T13:50:12.242474+0000","last_fullsized":"2024-09-21T13:50:12.242474+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:17.663571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":56,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-21T13:50:10.820702+0000","last_change":"2024-09-21T13:49:02.068530+0000","last_active":"2024-09-21T13:50:10.820702+0000","last_peered":"2024-09-21T13:50:10.820702+0000","last_clean":"2024-09-21T13:50:10.820702+0000","last_became_active":"2024-09-21T13:49:02.066159+0000","last_became_peered":"2024-09-21T13:49:02.066159+0000","last_unstale":"2024-09-21T13:50:10.820702+0000","last_undegraded":"2024-09-21T13:50:10.820702+0000","last_fullsized":"2024-09-21T13:50:10.820702+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:50:29.718157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":81854464,"num_objects":1249,"num_object_clones":0,"num_object_copies":3747,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1249,"num_write_kb":79936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":246218752,"data_stored":246218752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1249,"ondisk_log_size":1249,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":70,"num_read_kb":65,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234004,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":44064,"kb_used_data":12876,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93701088,"statfs":{"total":95995035648,"available":95949914112,"internally_reserved":0,"allocated":13185024,"data_stored":12641953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462937,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28556,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716596,"statfs":{"total":95995035648,"available":95965794304,"internally_reserved":0,"allocated":1564672,"data_stored":1040996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724576,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":974848,"data_stored":451544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920806,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1003520,"data_stored":451246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117035,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":74476,"kb_used_data":43288,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93670676,"statfs":{"total":95995035648,"available":95918772224,"internally_reserved":0,"allocated":44326912,"data_stored":43771372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411376,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":970752,"data_stored":451346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574904,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":41992,"kb_used_data":10804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93703160,"statfs":{"total":95995035648,"available":95952035840,"internally_reserved":0,"allocated":11063296,"data_stored":10478998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738430,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":63072,"kb_used_data":31884,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93682080,"statfs":{"total":95995035648,"available":95930449920,"internally_reserved":0,"allocated":32649216,"data_stored":32105418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40894464,"data_stored":40894464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":30408704,"data_stored":30408704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":30801920,"data_stored":30801920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":46989312,"data_stored":46989312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24248320,"data_stored":24248320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":22478848,"data_stored":22478848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":25624576,"data_stored":25624576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24772608,"data_stored":24772608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T13:50:20.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:20.324 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:20.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:50:20.334 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T13:50:20.425 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:50:20.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:20.467 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:20.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:50:20.568 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:50:20.581 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:20 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:50:20.581 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:20 smithi049 ceph-mon[30080]: pgmap v43: 148 pgs: 148 active+clean; 887 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 111 MiB/s wr, 1.77k op/s 2024-09-21T13:50:20.581 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:20 smithi049 ceph-mon[30080]: osdmap e90: 8 total, 8 up, 8 in 2024-09-21T13:50:20.581 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:20 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:50:20.581 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:20 smithi049 ceph-mon[40489]: pgmap v43: 148 pgs: 148 active+clean; 887 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 111 MiB/s wr, 1.77k op/s 2024-09-21T13:50:20.582 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:20 smithi049 ceph-mon[40489]: osdmap e90: 8 total, 8 up, 8 in 2024-09-21T13:50:20.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:20.613 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:20.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:50:20.669 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:50:20.669 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:50:20.669 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:50:20.669 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:50:20.669 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:50:20.670 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:50:20.670 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:50:20.670 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:50:20.670 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:50:20.670 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:50:20.670 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:50:20.670 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:50:20.670 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:50:20.670 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:50:20.670 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:50:20.671 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:50:20.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:20 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:50:20.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:20 smithi187 ceph-mon[33032]: pgmap v43: 148 pgs: 148 active+clean; 887 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 111 MiB/s wr, 1.77k op/s 2024-09-21T13:50:20.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:20 smithi187 ceph-mon[33032]: osdmap e90: 8 total, 8 up, 8 in 2024-09-21T13:50:20.714 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:20.744 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:20.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:20.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:20.845 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:50:20.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:20.889 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:20.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:50:20.891 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 8 16 20297 20281 158.436 148.562 0.00425235 0.00630374 2024-09-21T13:50:20.990 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:21.006 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:50:20 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:50:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:50:21.017 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:21.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:21.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:21.119 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:21.146 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:21.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:21.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:21.248 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:50:21.278 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:21.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:21.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:50:21.380 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:50:21.410 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:21.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:21.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:50:21.512 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:50:21.557 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:21.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:21.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:50:21.659 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:50:21.701 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:21.704 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:21 smithi049 bash[76740]: 556e964b68ca8f011a5e0cb8305e166ab59cd8d53c0710b605a382512d907fad 2024-09-21T13:50:21.705 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 13:50:21 smithi049 systemd[1]: Started Ceph node-exporter.a for 40a3d2e8-781f-11ef-baf6-efdc52797490. 2024-09-21T13:50:21.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:21.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:50:21.806 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:50:21.844 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:21.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:21.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:50:21.892 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 9 16 22666 22650 157.282 148.062 0.00359648 0.00635164 2024-09-21T13:50:21.945 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:50:21.986 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:21.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:21.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:50:22.038 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:50:22.071 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T13:50:22.089 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:50:22.159 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:22.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:22.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:50:22.260 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:22.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:22.288 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:22.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:22.389 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:22.417 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:22.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:22.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:22.518 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:50:22.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:22.542 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:22.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:50:22.643 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:50:22.683 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:22.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:22.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:50:22.785 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:50:22.822 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:22.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:22.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:50:22.892 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 10 16 24575 24559 153.484 119.312 0.00238025 0.00650157 2024-09-21T13:50:22.924 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:22.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:22 smithi187 ceph-mon[33032]: pgmap v45: 148 pgs: 148 active+clean; 887 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 682 B/s rd, 108 MiB/s wr, 1.72k op/s 2024-09-21T13:50:22.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:22 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:22.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:22 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:22.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:22 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:22.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:22 smithi187 ceph-mon[33032]: Deploying daemon node-exporter.b on smithi187 2024-09-21T13:50:22.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:22.947 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:22.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:23.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:22 smithi049 ceph-mon[40489]: pgmap v45: 148 pgs: 148 active+clean; 887 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 682 B/s rd, 108 MiB/s wr, 1.72k op/s 2024-09-21T13:50:23.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:22 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:23.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:22 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:23.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:22 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:23.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:22 smithi049 ceph-mon[40489]: Deploying daemon node-exporter.b on smithi187 2024-09-21T13:50:23.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:22 smithi049 ceph-mon[30080]: pgmap v45: 148 pgs: 148 active+clean; 887 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 682 B/s rd, 108 MiB/s wr, 1.72k op/s 2024-09-21T13:50:23.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:22 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:23.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:22 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:23.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:22 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:23.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:22 smithi049 ceph-mon[30080]: Deploying daemon node-exporter.b on smithi187 2024-09-21T13:50:23.048 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:50:23.087 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:23.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:23.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:50:23.189 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:50:23.214 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:23.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:23.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:50:23.316 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:50:23.341 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:23.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:23.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:50:23.354 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:50:23.443 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:23.466 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T13:50:23.470 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:23.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:23.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:23.572 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:50:23.614 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:23.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:23.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:50:23.716 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:23.745 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:23.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:23.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:50:23.847 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:50:23.879 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:23.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:23.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:50:23.892 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 11 16 26333 26317 149.518 109.875 0.00403364 0.00668239 2024-09-21T13:50:23.980 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:50:24.020 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:24.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:24.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:50:24.122 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:24.160 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:24.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:24.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:50:24.194 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:24 smithi187 systemd[1]: Starting Ceph node-exporter.b for 40a3d2e8-781f-11ef-baf6-efdc52797490... 2024-09-21T13:50:24.262 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:50:24.290 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:24.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:24.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:50:24.391 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:50:24.433 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:24.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:24.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:50:24.535 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:24.556 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:24.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:24.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:24.643 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:24 smithi187 bash[65131]: Trying to pull quay.io/prometheus/node-exporter:v1.7.0... 2024-09-21T13:50:24.658 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:24.666 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-09-21T13:50:24.685 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:24.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:24.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:50:24.786 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:50:24.828 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:24.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:24.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:50:24.892 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 12 16 28309 28293 147.349 123.5 0.0033778 0.00678055 2024-09-21T13:50:24.930 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:24.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:24 smithi187 ceph-mon[33032]: pgmap v46: 148 pgs: 148 active+clean; 1.3 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 142 MiB/s wr, 2.27k op/s 2024-09-21T13:50:24.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:24 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/4203307196' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:50:24.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:24 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:50:24.945 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:24 smithi187 bash[65131]: Getting image source signatures 2024-09-21T13:50:24.945 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:24 smithi187 bash[65131]: Copying blob sha256:455fd88e5221bc1e278ef2d059cd70e4df99a24e5af050ede621534276f6cf9a 2024-09-21T13:50:24.945 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:24 smithi187 bash[65131]: Copying blob sha256:324153f2810a9927fcce320af9e4e291e0b6e805cbdd1f338386c756b9defa24 2024-09-21T13:50:24.945 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:24 smithi187 bash[65131]: Copying blob sha256:2abcce694348cd2c949c0e98a7400ebdfd8341021bcf6b541bc72033ce982510 2024-09-21T13:50:24.948 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:24 smithi049 ceph-mon[30080]: pgmap v46: 148 pgs: 148 active+clean; 1.3 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 142 MiB/s wr, 2.27k op/s 2024-09-21T13:50:24.948 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:24 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/4203307196' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:50:24.948 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:24 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:50:24.949 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:24 smithi049 ceph-mon[40489]: pgmap v46: 148 pgs: 148 active+clean; 1.3 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 142 MiB/s wr, 2.27k op/s 2024-09-21T13:50:24.949 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:24 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/4203307196' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:50:24.949 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:24 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:50:24.961 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:24.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:24.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:25.064 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:25.092 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:25.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:25.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:50:25.194 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:50:25.234 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:25.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:25.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:50:25.324 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T13:50:25.324 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T13:50:25.324 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-21T13:50:25.324 INFO:tasks.thrashosds.thrasher:Setting osd 0 primary_affinity to 0.121611 2024-09-21T13:50:25.324 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 0 0.12161117022857315 2024-09-21T13:50:25.337 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:50:25.411 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:25.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:25.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:50:25.465 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T13:50:25.465 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:50:25.514 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:50:25.541 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:25.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:25.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:50:25.643 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:50:25.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:25.689 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:25.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:50:25.790 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:25.822 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:25.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:25.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:25.892 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 13 16 30260 30244 145.394 121.938 0.0075821 0.00687116 2024-09-21T13:50:25.923 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:50:25.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:25 smithi187 ceph-mon[33032]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:50:25.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:25 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T13:50:25.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:25 smithi187 ceph-mon[33032]: osdmap e91: 8 total, 8 up, 8 in 2024-09-21T13:50:25.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:25 smithi187 ceph-mon[33032]: 4.1c scrub starts 2024-09-21T13:50:25.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:25 smithi187 ceph-mon[33032]: 4.1c scrub ok 2024-09-21T13:50:25.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:25 smithi187 ceph-mon[33032]: osdmap e92: 8 total, 8 up, 8 in 2024-09-21T13:50:25.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:25 smithi187 ceph-mon[33032]: 5.12 scrub starts 2024-09-21T13:50:25.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:25 smithi187 ceph-mon[33032]: 5.12 scrub ok 2024-09-21T13:50:25.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:25 smithi049 ceph-mon[30080]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:50:25.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:25 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T13:50:25.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:25 smithi049 ceph-mon[30080]: osdmap e91: 8 total, 8 up, 8 in 2024-09-21T13:50:25.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:25 smithi049 ceph-mon[30080]: 4.1c scrub starts 2024-09-21T13:50:25.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:25 smithi049 ceph-mon[30080]: 4.1c scrub ok 2024-09-21T13:50:25.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:25 smithi049 ceph-mon[30080]: osdmap e92: 8 total, 8 up, 8 in 2024-09-21T13:50:25.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:25 smithi049 ceph-mon[30080]: 5.12 scrub starts 2024-09-21T13:50:25.961 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:25 smithi049 ceph-mon[30080]: 5.12 scrub ok 2024-09-21T13:50:25.961 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:25 smithi049 ceph-mon[40489]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:50:25.961 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:25 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T13:50:25.961 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:25 smithi049 ceph-mon[40489]: osdmap e91: 8 total, 8 up, 8 in 2024-09-21T13:50:25.961 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:25 smithi049 ceph-mon[40489]: 4.1c scrub starts 2024-09-21T13:50:25.961 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:25 smithi049 ceph-mon[40489]: 4.1c scrub ok 2024-09-21T13:50:25.961 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:25 smithi049 ceph-mon[40489]: osdmap e92: 8 total, 8 up, 8 in 2024-09-21T13:50:25.961 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:25 smithi049 ceph-mon[40489]: 5.12 scrub starts 2024-09-21T13:50:25.961 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:25 smithi049 ceph-mon[40489]: 5.12 scrub ok 2024-09-21T13:50:25.965 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:25.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:25.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:50:26.067 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:50:26.109 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:26.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:26.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:50:26.211 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:50:26.252 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:26.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:26.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:50:26.348 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:50:26.354 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:26.377 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:26.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:26.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:50:26.399 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T13:50:26.479 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:50:26.529 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:26.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:26.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:50:26.631 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:50:26.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:26.677 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:26.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:50:26.694 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:26 smithi187 bash[65131]: Copying config sha256:72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e 2024-09-21T13:50:26.694 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:26 smithi187 bash[65131]: Writing manifest to image destination 2024-09-21T13:50:26.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:26 smithi187 ceph-mon[33032]: pgmap v49: 148 pgs: 148 active+clean; 1.5 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 116 MiB/s wr, 1.86k op/s 2024-09-21T13:50:26.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:26 smithi187 ceph-mon[33032]: 6.7 scrub starts 2024-09-21T13:50:26.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:26 smithi187 ceph-mon[33032]: 6.7 scrub ok 2024-09-21T13:50:26.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:26 smithi187 ceph-mon[33032]: 3.1f scrub starts 2024-09-21T13:50:26.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:26 smithi187 ceph-mon[33032]: 3.1f scrub ok 2024-09-21T13:50:26.778 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:50:26.808 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:26.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:26.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:50:26.893 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:50:26.894 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 14 16 32223 32207 143.749 122.688 0.00802521 0.00694931 2024-09-21T13:50:26.910 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:50:26.959 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:26 smithi049 ceph-mon[40489]: pgmap v49: 148 pgs: 148 active+clean; 1.5 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 116 MiB/s wr, 1.86k op/s 2024-09-21T13:50:26.959 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:26 smithi049 ceph-mon[40489]: 6.7 scrub starts 2024-09-21T13:50:26.959 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:26 smithi049 ceph-mon[40489]: 6.7 scrub ok 2024-09-21T13:50:26.959 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:26 smithi049 ceph-mon[40489]: 3.1f scrub starts 2024-09-21T13:50:26.959 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:26 smithi049 ceph-mon[40489]: 3.1f scrub ok 2024-09-21T13:50:26.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:26 smithi049 ceph-mon[30080]: pgmap v49: 148 pgs: 148 active+clean; 1.5 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 116 MiB/s wr, 1.86k op/s 2024-09-21T13:50:26.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:26 smithi049 ceph-mon[30080]: 6.7 scrub starts 2024-09-21T13:50:26.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:26 smithi049 ceph-mon[30080]: 6.7 scrub ok 2024-09-21T13:50:26.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:26 smithi049 ceph-mon[30080]: 3.1f scrub starts 2024-09-21T13:50:26.960 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:26 smithi049 ceph-mon[30080]: 3.1f scrub ok 2024-09-21T13:50:26.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:26.985 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:26.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:50:27.086 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:50:27.134 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:27.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:27.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:50:27.194 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:26 smithi187 podman[65131]: 2024-09-21 13:50:26.875665774 +0000 UTC m=+2.678426043 image pull 72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e quay.io/prometheus/node-exporter:v1.7.0 2024-09-21T13:50:27.195 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:26 smithi187 podman[65131]: 2024-09-21 13:50:26.998954079 +0000 UTC m=+2.801714321 container create 5520efe30bc09115edf596ac2923e898d5bcac34c846319d1ede36be5169ff23 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T13:50:27.237 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:50:27.269 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:27.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:27.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:50:27.370 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:50:27.412 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:27.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:27.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:50:27.514 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:50:27.557 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:27.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:27.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:50:27.658 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:50:27.660 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:27 smithi187 podman[65131]: 2024-09-21 13:50:27.285413146 +0000 UTC m=+3.088173393 container init 5520efe30bc09115edf596ac2923e898d5bcac34c846319d1ede36be5169ff23 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T13:50:27.660 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:27 smithi187 podman[65131]: 2024-09-21 13:50:27.292414333 +0000 UTC m=+3.095174572 container start 5520efe30bc09115edf596ac2923e898d5bcac34c846319d1ede36be5169ff23 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T13:50:27.661 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b[65357]: ts=2024-09-21T13:50:27.297Z caller=node_exporter.go:192 level=info msg="Starting node_exporter" version="(version=1.7.0, branch=HEAD, revision=7333465abf9efba81876303bb57e6fadb946041b)" 2024-09-21T13:50:27.661 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b[65357]: ts=2024-09-21T13:50:27.297Z caller=node_exporter.go:193 level=info msg="Build context" build_context="(go=go1.21.4, platform=linux/amd64, user=root@35918982f6d8, date=20231112-23:53:35, tags=netgo osusergo static_build)" 2024-09-21T13:50:27.661 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b[65357]: ts=2024-09-21T13:50:27.298Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-21T13:50:27.661 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b[65357]: ts=2024-09-21T13:50:27.298Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-21T13:50:27.661 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b[65357]: ts=2024-09-21T13:50:27.299Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-21T13:50:27.661 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b[65357]: ts=2024-09-21T13:50:27.299Z caller=diskstats_linux.go:265 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-21T13:50:27.661 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b[65357]: ts=2024-09-21T13:50:27.299Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-21T13:50:27.661 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b[65357]: ts=2024-09-21T13:50:27.299Z caller=node_exporter.go:117 level=info collector=arp 2024-09-21T13:50:27.661 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b[65357]: ts=2024-09-21T13:50:27.299Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-21T13:50:27.661 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b[65357]: ts=2024-09-21T13:50:27.299Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-21T13:50:27.661 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b[65357]: ts=2024-09-21T13:50:27.299Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-21T13:50:27.661 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b[65357]: ts=2024-09-21T13:50:27.299Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-21T13:50:27.661 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b[65357]: ts=2024-09-21T13:50:27.299Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-21T13:50:27.662 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b[65357]: ts=2024-09-21T13:50:27.299Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-21T13:50:27.662 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b[65357]: ts=2024-09-21T13:50:27.299Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-21T13:50:27.662 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b[65357]: ts=2024-09-21T13:50:27.299Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-21T13:50:27.662 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b[65357]: ts=2024-09-21T13:50:27.299Z caller=node_exporter.go:117 level=info collector=edac 2024-09-21T13:50:27.662 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b[65357]: ts=2024-09-21T13:50:27.299Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-21T13:50:27.662 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b[65357]: ts=2024-09-21T13:50:27.299Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-21T13:50:27.662 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b[65357]: ts=2024-09-21T13:50:27.299Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-21T13:50:27.662 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b[65357]: ts=2024-09-21T13:50:27.299Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-21T13:50:27.662 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b[65357]: ts=2024-09-21T13:50:27.299Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-21T13:50:27.662 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b[65357]: ts=2024-09-21T13:50:27.299Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-21T13:50:27.662 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b[65357]: ts=2024-09-21T13:50:27.299Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-21T13:50:27.662 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b[65357]: ts=2024-09-21T13:50:27.300Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-21T13:50:27.662 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b[65357]: ts=2024-09-21T13:50:27.300Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-21T13:50:27.662 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b[65357]: ts=2024-09-21T13:50:27.300Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-21T13:50:27.663 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b[65357]: ts=2024-09-21T13:50:27.300Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-21T13:50:27.663 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b[65357]: ts=2024-09-21T13:50:27.300Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-21T13:50:27.663 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b[65357]: ts=2024-09-21T13:50:27.300Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-21T13:50:27.663 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b[65357]: ts=2024-09-21T13:50:27.300Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-21T13:50:27.663 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b[65357]: ts=2024-09-21T13:50:27.300Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-21T13:50:27.663 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b[65357]: ts=2024-09-21T13:50:27.300Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-21T13:50:27.663 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b[65357]: ts=2024-09-21T13:50:27.300Z caller=node_exporter.go:117 level=info collector=os 2024-09-21T13:50:27.663 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b[65357]: ts=2024-09-21T13:50:27.300Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-21T13:50:27.663 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b[65357]: ts=2024-09-21T13:50:27.300Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-21T13:50:27.663 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b[65357]: ts=2024-09-21T13:50:27.300Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-21T13:50:27.663 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b[65357]: ts=2024-09-21T13:50:27.300Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-21T13:50:27.663 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b[65357]: ts=2024-09-21T13:50:27.300Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-21T13:50:27.663 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b[65357]: ts=2024-09-21T13:50:27.300Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-21T13:50:27.663 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b[65357]: ts=2024-09-21T13:50:27.300Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-21T13:50:27.664 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b[65357]: ts=2024-09-21T13:50:27.300Z caller=node_exporter.go:117 level=info collector=stat 2024-09-21T13:50:27.664 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b[65357]: ts=2024-09-21T13:50:27.300Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-21T13:50:27.664 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b[65357]: ts=2024-09-21T13:50:27.300Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-21T13:50:27.664 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b[65357]: ts=2024-09-21T13:50:27.300Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-21T13:50:27.664 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b[65357]: ts=2024-09-21T13:50:27.300Z caller=node_exporter.go:117 level=info collector=time 2024-09-21T13:50:27.664 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b[65357]: ts=2024-09-21T13:50:27.300Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-21T13:50:27.664 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b[65357]: ts=2024-09-21T13:50:27.300Z caller=node_exporter.go:117 level=info collector=uname 2024-09-21T13:50:27.664 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b[65357]: ts=2024-09-21T13:50:27.300Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-21T13:50:27.664 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b[65357]: ts=2024-09-21T13:50:27.300Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-21T13:50:27.664 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b[65357]: ts=2024-09-21T13:50:27.300Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-21T13:50:27.664 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b[65357]: ts=2024-09-21T13:50:27.300Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9100 2024-09-21T13:50:27.664 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b[65357]: ts=2024-09-21T13:50:27.300Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-21T13:50:27.664 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:27 smithi187 bash[65131]: 5520efe30bc09115edf596ac2923e898d5bcac34c846319d1ede36be5169ff23 2024-09-21T13:50:27.664 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 13:50:27 smithi187 systemd[1]: Started Ceph node-exporter.b for 40a3d2e8-781f-11ef-baf6-efdc52797490. 2024-09-21T13:50:27.712 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:27.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:27.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:50:27.814 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:50:27.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:27.867 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:27.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:50:27.894 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 15 16 33942 33926 141.328 107.438 0.0130341 0.00706709 2024-09-21T13:50:27.918 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:27 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:50:27.918 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:27 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:50:27.919 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:27 smithi049 ceph-mon[30080]: 4.2 scrub starts 2024-09-21T13:50:27.919 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:27 smithi049 ceph-mon[30080]: 4.2 scrub ok 2024-09-21T13:50:27.919 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:27 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]} : dispatch 2024-09-21T13:50:27.919 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:27 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]} : dispatch 2024-09-21T13:50:27.919 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:27 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]} : dispatch 2024-09-21T13:50:27.919 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:27 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:27.919 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:27 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:27.919 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:27 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:27.919 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:27 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:27.919 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:27 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:27.919 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:27 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:50:27.920 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:27 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:50:27.920 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:27 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:50:27.920 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:27 smithi049 ceph-mon[40489]: 4.2 scrub starts 2024-09-21T13:50:27.920 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:27 smithi049 ceph-mon[40489]: 4.2 scrub ok 2024-09-21T13:50:27.920 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:27 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]} : dispatch 2024-09-21T13:50:27.920 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:27 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]} : dispatch 2024-09-21T13:50:27.920 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:27 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]} : dispatch 2024-09-21T13:50:27.920 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:27 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:27.920 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:27 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:27.920 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:27 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:27.920 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:27 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:27.920 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:27 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:27.920 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:27 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:50:27.941 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:50:27.941 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:50:27.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:50:27.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:50:27.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-mon[33032]: 4.2 scrub starts 2024-09-21T13:50:27.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-mon[33032]: 4.2 scrub ok 2024-09-21T13:50:27.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]} : dispatch 2024-09-21T13:50:27.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]} : dispatch 2024-09-21T13:50:27.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]} : dispatch 2024-09-21T13:50:27.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:27.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:27.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:27.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:27.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:27.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:27 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:50:27.960 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:50:27.960 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:50:27.968 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:27.996 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:27.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:27.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:27.998 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:50:27.999 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:50:28.037 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:50:28.037 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:50:28.073 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:50:28.073 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:50:28.098 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:28.103 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:50:28.115 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:50:28.115 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:50:28.121 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:28.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:28.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:50:28.163 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:50:28.164 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:50:28.211 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:50:28.211 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:50:28.223 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:28.249 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:28.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:28.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:28.351 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:50:28.389 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:28.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:28.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:50:28.490 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:28.520 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:28.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:28.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:28.622 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:28.651 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:28.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:28.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:28.678 INFO:teuthology.orchestra.run.smithi049.stderr:set osd.0 primary-affinity to 0.121611 (879692) 2024-09-21T13:50:28.753 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:28.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:28.774 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:28.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:28.848 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T13:50:28.848 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-09-21T13:50:28.848 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-09-21T13:50:28.848 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-09-21T13:50:28.848 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T13:50:28.875 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:50:28.894 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 16 16 35289 35273 137.756 84.1875 0.00189096 0.00716752 2024-09-21T13:50:28.897 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:28.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:28.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:50:28.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:28 smithi187 ceph-mon[33032]: 5.18 scrub starts 2024-09-21T13:50:28.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:28 smithi187 ceph-mon[33032]: 5.18 scrub ok 2024-09-21T13:50:28.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:28 smithi187 ceph-mon[33032]: 1.0 scrub starts 2024-09-21T13:50:28.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:28 smithi187 ceph-mon[33032]: 1.0 scrub ok 2024-09-21T13:50:28.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:28 smithi187 ceph-mon[33032]: 4.0 scrub starts 2024-09-21T13:50:28.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:28 smithi187 ceph-mon[33032]: 4.0 scrub ok 2024-09-21T13:50:28.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:28 smithi187 ceph-mon[33032]: pgmap v50: 148 pgs: 148 active+clean; 1.5 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 680 B/s rd, 93 MiB/s wr, 1.49k op/s 2024-09-21T13:50:28.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:28 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-09-21T13:50:28.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:28 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-09-21T13:50:28.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:28 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-09-21T13:50:28.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:28 smithi187 ceph-mon[33032]: osdmap e93: 8 total, 8 up, 8 in 2024-09-21T13:50:28.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:28 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:28.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:28 smithi187 ceph-mon[33032]: 3.10 scrub starts 2024-09-21T13:50:28.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:28 smithi187 ceph-mon[33032]: 3.10 scrub ok 2024-09-21T13:50:28.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:28 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:50:28.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:28 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/4227889356' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:50:28.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:28 smithi187 ceph-mon[33032]: 5.19 scrub starts 2024-09-21T13:50:28.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:28 smithi187 ceph-mon[33032]: 5.19 scrub ok 2024-09-21T13:50:28.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:28 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/840432554' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 0.12161117022857315} : dispatch 2024-09-21T13:50:28.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:28 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 0.12161117022857315} : dispatch 2024-09-21T13:50:28.960 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T13:50:28.983 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:28 smithi049 ceph-mon[40489]: 5.18 scrub starts 2024-09-21T13:50:28.983 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:28 smithi049 ceph-mon[40489]: 5.18 scrub ok 2024-09-21T13:50:28.983 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:28 smithi049 ceph-mon[40489]: 1.0 scrub starts 2024-09-21T13:50:28.983 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:28 smithi049 ceph-mon[40489]: 1.0 scrub ok 2024-09-21T13:50:28.983 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:28 smithi049 ceph-mon[40489]: 4.0 scrub starts 2024-09-21T13:50:28.984 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:28 smithi049 ceph-mon[40489]: 4.0 scrub ok 2024-09-21T13:50:28.984 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:28 smithi049 ceph-mon[40489]: pgmap v50: 148 pgs: 148 active+clean; 1.5 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 680 B/s rd, 93 MiB/s wr, 1.49k op/s 2024-09-21T13:50:28.984 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:28 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-09-21T13:50:28.984 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:28 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-09-21T13:50:28.984 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:28 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-09-21T13:50:28.984 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:28 smithi049 ceph-mon[40489]: osdmap e93: 8 total, 8 up, 8 in 2024-09-21T13:50:28.984 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:28 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:28.984 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:28 smithi049 ceph-mon[40489]: 3.10 scrub starts 2024-09-21T13:50:28.984 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:28 smithi049 ceph-mon[40489]: 3.10 scrub ok 2024-09-21T13:50:28.984 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:28 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:50:28.984 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:28 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/4227889356' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:50:28.984 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:28 smithi049 ceph-mon[40489]: 5.19 scrub starts 2024-09-21T13:50:28.984 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:28 smithi049 ceph-mon[40489]: 5.19 scrub ok 2024-09-21T13:50:28.985 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:28 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/840432554' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 0.12161117022857315} : dispatch 2024-09-21T13:50:28.985 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:28 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 0.12161117022857315} : dispatch 2024-09-21T13:50:28.985 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:28 smithi049 ceph-mon[30080]: 5.18 scrub starts 2024-09-21T13:50:28.985 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:28 smithi049 ceph-mon[30080]: 5.18 scrub ok 2024-09-21T13:50:28.985 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:28 smithi049 ceph-mon[30080]: 1.0 scrub starts 2024-09-21T13:50:28.985 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:28 smithi049 ceph-mon[30080]: 1.0 scrub ok 2024-09-21T13:50:28.985 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:28 smithi049 ceph-mon[30080]: 4.0 scrub starts 2024-09-21T13:50:28.985 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:28 smithi049 ceph-mon[30080]: 4.0 scrub ok 2024-09-21T13:50:28.985 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:28 smithi049 ceph-mon[30080]: pgmap v50: 148 pgs: 148 active+clean; 1.5 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 680 B/s rd, 93 MiB/s wr, 1.49k op/s 2024-09-21T13:50:28.985 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:28 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-09-21T13:50:28.985 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:28 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-09-21T13:50:28.986 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:28 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-09-21T13:50:28.986 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:28 smithi049 ceph-mon[30080]: osdmap e93: 8 total, 8 up, 8 in 2024-09-21T13:50:28.986 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:28 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:28.986 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:28 smithi049 ceph-mon[30080]: 3.10 scrub starts 2024-09-21T13:50:28.986 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:28 smithi049 ceph-mon[30080]: 3.10 scrub ok 2024-09-21T13:50:28.986 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:28 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:50:28.986 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:28 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/4227889356' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:50:28.986 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:28 smithi049 ceph-mon[30080]: 5.19 scrub starts 2024-09-21T13:50:28.986 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:28 smithi049 ceph-mon[30080]: 5.19 scrub ok 2024-09-21T13:50:28.986 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:28 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/840432554' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 0.12161117022857315} : dispatch 2024-09-21T13:50:28.986 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:28 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 0.12161117022857315} : dispatch 2024-09-21T13:50:28.999 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:29.021 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:29.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:29.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:29.122 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:50:29.144 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:29.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:29.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:50:29.246 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:29.268 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:29.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:29.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:29.369 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:50:29.392 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:29.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:29.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:50:29.444 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:50:28 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:50:29.494 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:50:29.517 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:29.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:29.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:50:29.618 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:50:29.644 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:29.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:29.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:50:29.681 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:29 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.12161117022857315}]': finished 2024-09-21T13:50:29.681 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:29 smithi049 ceph-mon[30080]: osdmap e94: 8 total, 8 up, 8 in 2024-09-21T13:50:29.681 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:29 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:50:29.681 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:29 smithi049 ceph-mon[30080]: pgmap v53: 148 pgs: 3 peering, 145 active+clean; 2.1 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 124 MiB/s wr, 1.98k op/s 2024-09-21T13:50:29.681 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:29 smithi049 ceph-mon[30080]: 4.6 scrub starts 2024-09-21T13:50:29.681 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:29 smithi049 ceph-mon[30080]: 4.6 scrub ok 2024-09-21T13:50:29.745 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:50:29.785 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:29.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:29.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:50:29.887 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:50:29.894 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 17 16 35616 35600 130.856 20.4375 0.0397684 0.00763059 2024-09-21T13:50:29.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:29.929 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:29.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:50:29.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:29 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.12161117022857315}]': finished 2024-09-21T13:50:29.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:29 smithi187 ceph-mon[33032]: osdmap e94: 8 total, 8 up, 8 in 2024-09-21T13:50:29.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:29 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:50:29.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:29 smithi187 ceph-mon[33032]: pgmap v53: 148 pgs: 3 peering, 145 active+clean; 2.1 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 124 MiB/s wr, 1.98k op/s 2024-09-21T13:50:29.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:29 smithi187 ceph-mon[33032]: 4.6 scrub starts 2024-09-21T13:50:29.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:29 smithi187 ceph-mon[33032]: 4.6 scrub ok 2024-09-21T13:50:30.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:29 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.12161117022857315}]': finished 2024-09-21T13:50:30.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:29 smithi049 ceph-mon[40489]: osdmap e94: 8 total, 8 up, 8 in 2024-09-21T13:50:30.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:29 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:50:30.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:29 smithi049 ceph-mon[40489]: pgmap v53: 148 pgs: 3 peering, 145 active+clean; 2.1 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 124 MiB/s wr, 1.98k op/s 2024-09-21T13:50:30.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:29 smithi049 ceph-mon[40489]: 4.6 scrub starts 2024-09-21T13:50:30.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:29 smithi049 ceph-mon[40489]: 4.6 scrub ok 2024-09-21T13:50:30.030 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:30.059 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:30.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:30.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:30.161 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:30.191 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:30.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:30.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:50:30.293 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:30.314 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:30.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:30.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:30.415 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:30.448 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:30.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:30.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:30.550 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:30.571 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:30.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:30.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:30.672 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:50:30.695 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:30.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:30.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:50:30.796 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:50:30.825 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:30.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:30.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:50:30.894 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 18 16 37123 37107 128.819 94.1875 0.0102956 0.0077571 2024-09-21T13:50:30.927 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:50:30.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:30 smithi187 ceph-mon[33032]: 4.1 scrub starts 2024-09-21T13:50:30.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:30 smithi187 ceph-mon[33032]: 4.1 scrub ok 2024-09-21T13:50:30.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:30 smithi187 ceph-mon[33032]: 3.e scrub starts 2024-09-21T13:50:30.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:30 smithi187 ceph-mon[33032]: 3.e scrub ok 2024-09-21T13:50:30.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:30 smithi187 ceph-mon[33032]: 5.14 scrub starts 2024-09-21T13:50:30.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:30 smithi187 ceph-mon[33032]: 5.14 scrub ok 2024-09-21T13:50:30.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:30 smithi187 ceph-mon[33032]: osdmap e95: 8 total, 8 up, 8 in 2024-09-21T13:50:30.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:30 smithi187 ceph-mon[33032]: osdmap e96: 8 total, 8 up, 8 in 2024-09-21T13:50:30.958 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:30.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:30.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:50:31.004 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:50:30 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:50:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:50:31.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:30 smithi049 ceph-mon[30080]: 4.1 scrub starts 2024-09-21T13:50:31.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:30 smithi049 ceph-mon[30080]: 4.1 scrub ok 2024-09-21T13:50:31.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:30 smithi049 ceph-mon[30080]: 3.e scrub starts 2024-09-21T13:50:31.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:30 smithi049 ceph-mon[30080]: 3.e scrub ok 2024-09-21T13:50:31.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:30 smithi049 ceph-mon[30080]: 5.14 scrub starts 2024-09-21T13:50:31.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:30 smithi049 ceph-mon[30080]: 5.14 scrub ok 2024-09-21T13:50:31.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:30 smithi049 ceph-mon[30080]: osdmap e95: 8 total, 8 up, 8 in 2024-09-21T13:50:31.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:30 smithi049 ceph-mon[30080]: osdmap e96: 8 total, 8 up, 8 in 2024-09-21T13:50:31.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:30 smithi049 ceph-mon[40489]: 4.1 scrub starts 2024-09-21T13:50:31.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:30 smithi049 ceph-mon[40489]: 4.1 scrub ok 2024-09-21T13:50:31.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:30 smithi049 ceph-mon[40489]: 3.e scrub starts 2024-09-21T13:50:31.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:30 smithi049 ceph-mon[40489]: 3.e scrub ok 2024-09-21T13:50:31.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:30 smithi049 ceph-mon[40489]: 5.14 scrub starts 2024-09-21T13:50:31.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:30 smithi049 ceph-mon[40489]: 5.14 scrub ok 2024-09-21T13:50:31.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:30 smithi049 ceph-mon[40489]: osdmap e95: 8 total, 8 up, 8 in 2024-09-21T13:50:31.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:30 smithi049 ceph-mon[40489]: osdmap e96: 8 total, 8 up, 8 in 2024-09-21T13:50:31.060 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:50:31.091 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:31.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:31.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:50:31.193 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:31.222 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:31.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:31.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:31.323 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:50:31.362 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:31.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:31.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:50:31.464 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:50:31.507 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:31.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:31.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:50:31.609 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:31.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:31.645 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:31.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:31.746 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:31.774 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:31.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:31.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:50:31.876 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:31.894 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 19 16 38957 38941 128.071 114.625 0.00781372 0.00780309 2024-09-21T13:50:31.909 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:31.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:31.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:31.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:31 smithi187 ceph-mon[33032]: 3.1 scrub starts 2024-09-21T13:50:31.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:31 smithi187 ceph-mon[33032]: 3.1 scrub ok 2024-09-21T13:50:31.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:31 smithi187 ceph-mon[33032]: 3.12 scrub starts 2024-09-21T13:50:31.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:31 smithi187 ceph-mon[33032]: 3.12 scrub ok 2024-09-21T13:50:31.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:31 smithi187 ceph-mon[33032]: 6.18 scrub starts 2024-09-21T13:50:31.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:31 smithi187 ceph-mon[33032]: 6.18 scrub ok 2024-09-21T13:50:31.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:31 smithi187 ceph-mon[33032]: 4.11 scrub starts 2024-09-21T13:50:31.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:31 smithi187 ceph-mon[33032]: 4.11 scrub ok 2024-09-21T13:50:31.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:31 smithi187 ceph-mon[33032]: pgmap v56: 148 pgs: 3 peering, 145 active+clean; 2.1 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 145 MiB/s wr, 2.31k op/s 2024-09-21T13:50:32.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:31 smithi049 ceph-mon[40489]: 3.1 scrub starts 2024-09-21T13:50:32.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:31 smithi049 ceph-mon[40489]: 3.1 scrub ok 2024-09-21T13:50:32.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:31 smithi049 ceph-mon[40489]: 3.12 scrub starts 2024-09-21T13:50:32.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:31 smithi049 ceph-mon[40489]: 3.12 scrub ok 2024-09-21T13:50:32.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:31 smithi049 ceph-mon[40489]: 6.18 scrub starts 2024-09-21T13:50:32.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:31 smithi049 ceph-mon[40489]: 6.18 scrub ok 2024-09-21T13:50:32.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:31 smithi049 ceph-mon[40489]: 4.11 scrub starts 2024-09-21T13:50:32.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:31 smithi049 ceph-mon[40489]: 4.11 scrub ok 2024-09-21T13:50:32.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:31 smithi049 ceph-mon[40489]: pgmap v56: 148 pgs: 3 peering, 145 active+clean; 2.1 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 145 MiB/s wr, 2.31k op/s 2024-09-21T13:50:32.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:31 smithi049 ceph-mon[30080]: 3.1 scrub starts 2024-09-21T13:50:32.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:31 smithi049 ceph-mon[30080]: 3.1 scrub ok 2024-09-21T13:50:32.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:31 smithi049 ceph-mon[30080]: 3.12 scrub starts 2024-09-21T13:50:32.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:31 smithi049 ceph-mon[30080]: 3.12 scrub ok 2024-09-21T13:50:32.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:31 smithi049 ceph-mon[30080]: 6.18 scrub starts 2024-09-21T13:50:32.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:31 smithi049 ceph-mon[30080]: 6.18 scrub ok 2024-09-21T13:50:32.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:31 smithi049 ceph-mon[30080]: 4.11 scrub starts 2024-09-21T13:50:32.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:31 smithi049 ceph-mon[30080]: 4.11 scrub ok 2024-09-21T13:50:32.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:31 smithi049 ceph-mon[30080]: pgmap v56: 148 pgs: 3 peering, 145 active+clean; 2.1 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 145 MiB/s wr, 2.31k op/s 2024-09-21T13:50:32.011 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:50:32.053 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:32.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:32.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:50:32.155 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:50:32.193 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:32.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:32.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:50:32.295 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:50:32.341 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:32.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:32.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:50:32.443 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:32.475 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:32.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:32.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:32.577 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:50:32.623 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:32.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:32.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:50:32.676 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T13:50:32.726 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:50:32.751 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:32.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:32.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:50:32.853 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:50:32.903 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:2024-09-21T13:50:32.903502+0000 min lat: 0.00133054 max lat: 1.02032 avg lat: 0.00784822 2024-09-21T13:50:32.903 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T13:50:32.903 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 20 16 40778 40762 127.307 113.812 0.00852569 0.00784822 2024-09-21T13:50:32.906 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:32.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:32.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:50:32.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:32 smithi187 ceph-mon[33032]: 3.0 scrub starts 2024-09-21T13:50:32.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:32 smithi187 ceph-mon[33032]: 3.0 scrub ok 2024-09-21T13:50:33.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:32 smithi049 ceph-mon[40489]: 3.0 scrub starts 2024-09-21T13:50:33.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:32 smithi049 ceph-mon[40489]: 3.0 scrub ok 2024-09-21T13:50:33.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:32 smithi049 ceph-mon[30080]: 3.0 scrub starts 2024-09-21T13:50:33.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:32 smithi049 ceph-mon[30080]: 3.0 scrub ok 2024-09-21T13:50:33.008 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:33.036 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:33.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:33.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:33.138 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:50:33.167 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:33.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:33.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:50:33.269 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:33.300 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:33.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:33.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:50:33.317 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:50:33.402 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:33.432 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:33.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:33.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:33.534 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:50:33.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:33.577 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:33.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:50:33.678 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:50:33.712 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:33.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:33.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:50:33.815 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:50:33.847 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:33.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:33.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:50:33.904 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 21 16 42497 42481 126.352 107.438 0.00837348 0.00790853 2024-09-21T13:50:33.933 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:33 smithi049 ceph-mon[30080]: 4.3 scrub starts 2024-09-21T13:50:33.933 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:33 smithi049 ceph-mon[30080]: 4.3 scrub ok 2024-09-21T13:50:33.933 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:33 smithi049 ceph-mon[30080]: 4.f scrub starts 2024-09-21T13:50:33.933 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:33 smithi049 ceph-mon[30080]: 4.f scrub ok 2024-09-21T13:50:33.933 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:33 smithi049 ceph-mon[30080]: pgmap v57: 148 pgs: 1 peering, 147 active+clean; 2.4 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1.4 KiB/s rd, 159 MiB/s wr, 2.54k op/s; 38 B/s, 1 objects/s recovering 2024-09-21T13:50:33.933 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:33 smithi049 ceph-mon[40489]: 4.3 scrub starts 2024-09-21T13:50:33.933 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:33 smithi049 ceph-mon[40489]: 4.3 scrub ok 2024-09-21T13:50:33.933 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:33 smithi049 ceph-mon[40489]: 4.f scrub starts 2024-09-21T13:50:33.934 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:33 smithi049 ceph-mon[40489]: 4.f scrub ok 2024-09-21T13:50:33.934 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:33 smithi049 ceph-mon[40489]: pgmap v57: 148 pgs: 1 peering, 147 active+clean; 2.4 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1.4 KiB/s rd, 159 MiB/s wr, 2.54k op/s; 38 B/s, 1 objects/s recovering 2024-09-21T13:50:33.948 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:33.975 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:33.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:33.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:34.061 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:33 smithi187 ceph-mon[33032]: 4.3 scrub starts 2024-09-21T13:50:34.061 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:33 smithi187 ceph-mon[33032]: 4.3 scrub ok 2024-09-21T13:50:34.061 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:33 smithi187 ceph-mon[33032]: 4.f scrub starts 2024-09-21T13:50:34.061 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:33 smithi187 ceph-mon[33032]: 4.f scrub ok 2024-09-21T13:50:34.061 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:33 smithi187 ceph-mon[33032]: pgmap v57: 148 pgs: 1 peering, 147 active+clean; 2.4 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1.4 KiB/s rd, 159 MiB/s wr, 2.54k op/s; 38 B/s, 1 objects/s recovering 2024-09-21T13:50:34.077 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:50:34.108 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:34.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:34.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:50:34.210 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:50:34.240 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:34.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:34.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:50:34.342 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:50:34.368 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:34.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:34.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:50:34.374 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T13:50:34.469 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:50:34.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:34.537 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:34.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:50:34.638 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:34.640 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:50:34.664 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:34.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:34.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:34.720 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-09-21T13:50:34.766 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:50:34.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:34.813 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:34.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:50:34.904 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 22 16 43956 43940 124.753 91.1875 0.0156945 0.00801094 2024-09-21T13:50:34.914 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:34.944 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:34.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:34.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:35.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:34 smithi049 ceph-mon[40489]: 6.16 scrub starts 2024-09-21T13:50:35.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:34 smithi049 ceph-mon[40489]: 6.16 scrub ok 2024-09-21T13:50:35.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:34 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2602686088' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:50:35.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:34 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:50:35.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:34 smithi049 ceph-mon[30080]: 6.16 scrub starts 2024-09-21T13:50:35.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:34 smithi049 ceph-mon[30080]: 6.16 scrub ok 2024-09-21T13:50:35.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:34 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2602686088' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:50:35.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:34 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:50:35.046 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:35.049 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:34 smithi187 ceph-mon[33032]: 6.16 scrub starts 2024-09-21T13:50:35.049 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:34 smithi187 ceph-mon[33032]: 6.16 scrub ok 2024-09-21T13:50:35.049 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:34 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2602686088' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:50:35.049 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:34 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:50:35.076 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:35.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:35.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:50:35.178 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:50:35.217 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:35.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:35.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:50:35.319 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:50:35.365 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:35.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:35.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:50:35.467 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:50:35.500 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:35.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:35.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:50:35.602 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:35.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:35.643 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:35.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:50:35.744 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:50:35.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:35.796 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:35.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:50:35.897 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:35.904 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 23 15 45718 45703 124.12 110.188 0.00986545 0.00805152 2024-09-21T13:50:35.934 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:35.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:35.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:36.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:35 smithi049 ceph-mon[40489]: 6.d scrub starts 2024-09-21T13:50:36.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:35 smithi049 ceph-mon[40489]: 6.d scrub ok 2024-09-21T13:50:36.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:35 smithi049 ceph-mon[40489]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T13:50:36.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:35 smithi049 ceph-mon[40489]: Cluster is now healthy 2024-09-21T13:50:36.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:35 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T13:50:36.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:35 smithi049 ceph-mon[40489]: osdmap e97: 8 total, 8 up, 8 in 2024-09-21T13:50:36.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:35 smithi049 ceph-mon[40489]: osdmap e98: 8 total, 8 up, 8 in 2024-09-21T13:50:36.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:35 smithi049 ceph-mon[40489]: 6.1d scrub starts 2024-09-21T13:50:36.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:35 smithi049 ceph-mon[40489]: 6.1d scrub ok 2024-09-21T13:50:36.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:35 smithi049 ceph-mon[40489]: pgmap v60: 148 pgs: 148 active+clean; 2.5 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 361 B/s rd, 76 MiB/s wr, 1.22k op/s; 38 B/s, 2 objects/s recovering 2024-09-21T13:50:36.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:35 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:36.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:35 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:36.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:35 smithi049 ceph-mon[30080]: 6.d scrub starts 2024-09-21T13:50:36.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:35 smithi049 ceph-mon[30080]: 6.d scrub ok 2024-09-21T13:50:36.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:35 smithi049 ceph-mon[30080]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T13:50:36.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:35 smithi049 ceph-mon[30080]: Cluster is now healthy 2024-09-21T13:50:36.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:35 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T13:50:36.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:35 smithi049 ceph-mon[30080]: osdmap e97: 8 total, 8 up, 8 in 2024-09-21T13:50:36.007 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:35 smithi049 ceph-mon[30080]: osdmap e98: 8 total, 8 up, 8 in 2024-09-21T13:50:36.007 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:35 smithi049 ceph-mon[30080]: 6.1d scrub starts 2024-09-21T13:50:36.007 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:35 smithi049 ceph-mon[30080]: 6.1d scrub ok 2024-09-21T13:50:36.007 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:35 smithi049 ceph-mon[30080]: pgmap v60: 148 pgs: 148 active+clean; 2.5 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 361 B/s rd, 76 MiB/s wr, 1.22k op/s; 38 B/s, 2 objects/s recovering 2024-09-21T13:50:36.007 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:35 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:36.007 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:35 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:36.036 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:50:36.081 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:50:36.083 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:36.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:36.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:50:36.136 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T13:50:36.136 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-09-21T13:50:36.136 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-09-21T13:50:36.136 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-09-21T13:50:36.137 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.137 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-09-21T13:50:36.137 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:46:22.930233+0000", 2024-09-21T13:50:36.137 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 253.20014108199999, 2024-09-21T13:50:36.137 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.47268379100000002, 2024-09-21T13:50:36.137 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:50:36.137 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:50:36.137 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:50:36.137 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:50:36.137 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14158", 2024-09-21T13:50:36.137 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2731793917", 2024-09-21T13:50:36.137 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 1 2024-09-21T13:50:36.137 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.137 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:50:36.137 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.138 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:50:36.138 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:46:22.930233+0000", 2024-09-21T13:50:36.138 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:36.138 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.138 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.138 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:50:36.138 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:46:22.930233+0000", 2024-09-21T13:50:36.138 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:36.138 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.138 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.138 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:50:36.138 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:46:22.930234+0000", 2024-09-21T13:50:36.138 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.325e-06 2024-09-21T13:50:36.139 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.139 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.139 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:50:36.139 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:46:22.930242+0000", 2024-09-21T13:50:36.139 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6229999999999997e-06 2024-09-21T13:50:36.139 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.139 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.139 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:50:36.139 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:46:22.930256+0000", 2024-09-21T13:50:36.139 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3757999999999999e-05 2024-09-21T13:50:36.139 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.139 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.139 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:50:36.140 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:46:22.930266+0000", 2024-09-21T13:50:36.140 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0492000000000001e-05 2024-09-21T13:50:36.140 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.140 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.140 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:50:36.140 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:46:22.930330+0000", 2024-09-21T13:50:36.140 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4306000000000004e-05 2024-09-21T13:50:36.140 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.140 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.140 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:50:36.140 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:46:22.930339+0000", 2024-09-21T13:50:36.140 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1410000000000005e-06 2024-09-21T13:50:36.140 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.140 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.141 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:50:36.141 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:46:23.401797+0000", 2024-09-21T13:50:36.141 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.47145811900000001 2024-09-21T13:50:36.141 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.141 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.141 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:50:36.141 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:46:23.401959+0000", 2024-09-21T13:50:36.141 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016216700000000001 2024-09-21T13:50:36.141 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.141 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.141 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,2", 2024-09-21T13:50:36.141 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:46:23.402066+0000", 2024-09-21T13:50:36.141 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010719 2024-09-21T13:50:36.141 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.141 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.141 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:50:36.142 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:46:23.402704+0000", 2024-09-21T13:50:36.142 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00063833100000000001 2024-09-21T13:50:36.142 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.142 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.142 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:36.142 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:46:23.402815+0000", 2024-09-21T13:50:36.142 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000110921 2024-09-21T13:50:36.142 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.142 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.142 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:36.142 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:46:23.402855+0000", 2024-09-21T13:50:36.142 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9799999999999998e-05 2024-09-21T13:50:36.142 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.142 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.143 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:50:36.143 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:46:23.402895+0000", 2024-09-21T13:50:36.143 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0167999999999999e-05 2024-09-21T13:50:36.143 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.143 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.143 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:50:36.143 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:46:23.402917+0000", 2024-09-21T13:50:36.143 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.145e-05 2024-09-21T13:50:36.143 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:36.143 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:50:36.143 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:36.143 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.143 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.144 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:35035 7.b 7:daad5729:::benchmark_data_smithi049_76798_object35034:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-21T13:50:36.144 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:28.571760+0000", 2024-09-21T13:50:36.144 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 7.5586138800000002, 2024-09-21T13:50:36.144 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.031042809000000001, 2024-09-21T13:50:36.144 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:50:36.144 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:50:36.144 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:50:36.144 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:50:36.144 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:50:36.144 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:50:36.144 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 35035 2024-09-21T13:50:36.144 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.144 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:50:36.144 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.144 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:50:36.145 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.571760+0000", 2024-09-21T13:50:36.145 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:36.145 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.145 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.145 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:50:36.145 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.571760+0000", 2024-09-21T13:50:36.145 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:36.145 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.145 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.145 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:50:36.145 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.571761+0000", 2024-09-21T13:50:36.145 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.124e-06 2024-09-21T13:50:36.145 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.145 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.145 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:50:36.146 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.571785+0000", 2024-09-21T13:50:36.146 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4153999999999999e-05 2024-09-21T13:50:36.146 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.146 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.146 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:50:36.146 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.571797+0000", 2024-09-21T13:50:36.146 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1875000000000001e-05 2024-09-21T13:50:36.146 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.146 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.146 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:50:36.146 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.571805+0000", 2024-09-21T13:50:36.146 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5499999999999997e-06 2024-09-21T13:50:36.146 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.147 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.147 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:50:36.147 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.574448+0000", 2024-09-21T13:50:36.147 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0026436319999999999 2024-09-21T13:50:36.147 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.147 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.147 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:50:36.147 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.574567+0000", 2024-09-21T13:50:36.147 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000118415 2024-09-21T13:50:36.147 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.147 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.147 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,6", 2024-09-21T13:50:36.147 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.574693+0000", 2024-09-21T13:50:36.147 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012572500000000001 2024-09-21T13:50:36.147 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.148 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.148 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:36.148 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.576453+0000", 2024-09-21T13:50:36.148 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0017605660000000001 2024-09-21T13:50:36.148 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.148 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.148 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:50:36.148 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.577936+0000", 2024-09-21T13:50:36.148 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0014831180000000001 2024-09-21T13:50:36.148 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.148 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.148 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:36.148 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.602361+0000", 2024-09-21T13:50:36.149 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.024424353999999999 2024-09-21T13:50:36.149 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.149 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.149 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:50:36.149 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.602632+0000", 2024-09-21T13:50:36.149 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00027091300000000001 2024-09-21T13:50:36.149 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.149 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.149 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:50:36.149 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.602803+0000", 2024-09-21T13:50:36.149 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000171383 2024-09-21T13:50:36.149 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:36.149 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:50:36.149 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:36.149 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.150 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.150 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:35232 7.9 7:9fe7d429:::benchmark_data_smithi049_76798_object35231:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T13:50:36.150 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:28.686371+0000", 2024-09-21T13:50:36.150 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 7.4440031009999998, 2024-09-21T13:50:36.150 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.004075987, 2024-09-21T13:50:36.150 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:50:36.150 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:50:36.150 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T13:50:36.150 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:50:36.150 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:50:36.150 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:50:36.150 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 35232 2024-09-21T13:50:36.151 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.151 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:50:36.151 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.151 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:50:36.151 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.686371+0000", 2024-09-21T13:50:36.151 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:36.151 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.151 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.151 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:50:36.151 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.686371+0000", 2024-09-21T13:50:36.151 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:36.151 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.151 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.151 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:50:36.152 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.686372+0000", 2024-09-21T13:50:36.152 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5230000000000001e-06 2024-09-21T13:50:36.152 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.152 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.152 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:50:36.152 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.686409+0000", 2024-09-21T13:50:36.152 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6964e-05 2024-09-21T13:50:36.152 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.152 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.152 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:50:36.152 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.686422+0000", 2024-09-21T13:50:36.152 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2405000000000001e-05 2024-09-21T13:50:36.152 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.152 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.153 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:50:36.153 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.686432+0000", 2024-09-21T13:50:36.153 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8130000000000007e-06 2024-09-21T13:50:36.153 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.153 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.153 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:50:36.153 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.686758+0000", 2024-09-21T13:50:36.153 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00032668200000000001 2024-09-21T13:50:36.153 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.153 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.153 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:50:36.153 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.686768+0000", 2024-09-21T13:50:36.153 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9920000000000002e-06 2024-09-21T13:50:36.154 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.154 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.154 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:50:36.154 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.689859+0000", 2024-09-21T13:50:36.154 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.003090952 2024-09-21T13:50:36.154 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.154 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.154 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:50:36.154 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.690447+0000", 2024-09-21T13:50:36.154 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00058765599999999999 2024-09-21T13:50:36.154 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:36.154 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:50:36.154 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:36.155 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.155 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.155 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:35237 7.9 7:9d5e8f11:::benchmark_data_smithi049_76798_object35236:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T13:50:36.155 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:28.686454+0000", 2024-09-21T13:50:36.155 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 7.4439195700000003, 2024-09-21T13:50:36.155 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0108801160000001, 2024-09-21T13:50:36.155 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:50:36.155 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:50:36.155 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:50:36.155 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:50:36.155 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:50:36.155 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:50:36.155 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 35237 2024-09-21T13:50:36.156 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.156 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:50:36.156 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.156 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:50:36.156 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.686454+0000", 2024-09-21T13:50:36.156 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:36.156 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.156 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.156 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:50:36.156 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.686454+0000", 2024-09-21T13:50:36.156 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:36.156 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.156 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.156 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:50:36.157 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.686455+0000", 2024-09-21T13:50:36.157 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7100000000000002e-07 2024-09-21T13:50:36.157 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.157 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.157 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:50:36.157 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.686477+0000", 2024-09-21T13:50:36.157 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2257000000000001e-05 2024-09-21T13:50:36.157 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.157 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.157 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:50:36.157 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.686486+0000", 2024-09-21T13:50:36.157 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4100000000000008e-06 2024-09-21T13:50:36.157 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.157 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.158 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:50:36.158 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.686499+0000", 2024-09-21T13:50:36.158 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3200999999999999e-05 2024-09-21T13:50:36.158 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.158 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.158 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:50:36.158 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.686795+0000", 2024-09-21T13:50:36.158 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00029598200000000003 2024-09-21T13:50:36.158 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.158 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.158 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:50:36.158 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.686802+0000", 2024-09-21T13:50:36.159 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3560000000000004e-06 2024-09-21T13:50:36.159 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.159 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.159 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:50:36.159 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.690840+0000", 2024-09-21T13:50:36.159 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0040381190000001 2024-09-21T13:50:36.159 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.159 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.159 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:50:36.159 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.691025+0000", 2024-09-21T13:50:36.159 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00018466700000000001 2024-09-21T13:50:36.159 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.159 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.159 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,4", 2024-09-21T13:50:36.159 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.691168+0000", 2024-09-21T13:50:36.160 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014334499999999999 2024-09-21T13:50:36.160 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.160 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.160 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:50:36.160 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.696996+0000", 2024-09-21T13:50:36.160 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0058274470000000004 2024-09-21T13:50:36.160 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.160 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.160 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:36.160 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.697045+0000", 2024-09-21T13:50:36.160 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9653e-05 2024-09-21T13:50:36.160 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.160 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.160 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:36.160 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.697095+0000", 2024-09-21T13:50:36.160 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9666000000000002e-05 2024-09-21T13:50:36.161 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.161 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.161 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:50:36.161 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.697224+0000", 2024-09-21T13:50:36.161 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012912700000000001 2024-09-21T13:50:36.161 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.161 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.161 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:50:36.161 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.697335+0000", 2024-09-21T13:50:36.161 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000110315 2024-09-21T13:50:36.161 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:36.161 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:50:36.161 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:36.161 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.161 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.162 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:35246 7.9 7:9e7a779d:::benchmark_data_smithi049_76798_object35245:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T13:50:36.162 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:28.690764+0000", 2024-09-21T13:50:36.162 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 7.4396101540000004, 2024-09-21T13:50:36.162 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.007016763, 2024-09-21T13:50:36.162 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:50:36.162 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:50:36.162 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:50:36.162 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:50:36.162 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:50:36.162 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:50:36.162 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 35246 2024-09-21T13:50:36.162 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.162 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:50:36.163 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.163 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:50:36.163 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.690764+0000", 2024-09-21T13:50:36.163 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:36.163 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.163 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.164 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:50:36.164 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.690764+0000", 2024-09-21T13:50:36.164 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:36.164 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.164 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.164 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:50:36.164 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.690766+0000", 2024-09-21T13:50:36.164 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.77e-06 2024-09-21T13:50:36.164 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.164 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.164 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:50:36.164 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.690790+0000", 2024-09-21T13:50:36.164 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4717999999999999e-05 2024-09-21T13:50:36.165 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.165 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.165 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:50:36.165 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.690804+0000", 2024-09-21T13:50:36.165 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3743e-05 2024-09-21T13:50:36.165 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.165 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.165 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:50:36.165 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.690813+0000", 2024-09-21T13:50:36.165 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9879999999999993e-06 2024-09-21T13:50:36.165 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.165 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.165 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:50:36.165 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.691059+0000", 2024-09-21T13:50:36.166 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000245424 2024-09-21T13:50:36.166 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.166 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.166 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:50:36.166 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.691070+0000", 2024-09-21T13:50:36.166 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1375999999999999e-05 2024-09-21T13:50:36.166 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.166 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.166 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:50:36.166 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.692262+0000", 2024-09-21T13:50:36.166 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.001191693 2024-09-21T13:50:36.166 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.166 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.167 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:50:36.167 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.692378+0000", 2024-09-21T13:50:36.167 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000115998 2024-09-21T13:50:36.167 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.167 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.167 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,4", 2024-09-21T13:50:36.167 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.692518+0000", 2024-09-21T13:50:36.167 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014013900000000001 2024-09-21T13:50:36.167 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.167 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.167 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:50:36.167 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.697001+0000", 2024-09-21T13:50:36.167 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0044833909999999998 2024-09-21T13:50:36.167 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.168 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.168 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:36.168 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.697514+0000", 2024-09-21T13:50:36.168 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00051338899999999995 2024-09-21T13:50:36.168 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.168 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.168 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:36.168 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.697690+0000", 2024-09-21T13:50:36.168 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000175377 2024-09-21T13:50:36.168 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.168 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.168 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:50:36.168 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.697744+0000", 2024-09-21T13:50:36.169 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4018999999999999e-05 2024-09-21T13:50:36.169 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.169 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.169 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:50:36.169 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.697781+0000", 2024-09-21T13:50:36.169 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6738e-05 2024-09-21T13:50:36.169 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:36.169 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:50:36.169 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:36.169 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.169 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.169 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:35255 7.9 7:9a89d0ac:::benchmark_data_smithi049_76798_object35254:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T13:50:36.169 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:28.698899+0000", 2024-09-21T13:50:36.170 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 7.4314752019999997, 2024-09-21T13:50:36.170 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.004516322, 2024-09-21T13:50:36.170 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:50:36.170 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:50:36.170 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:50:36.170 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:50:36.170 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:50:36.170 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:50:36.170 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 35255 2024-09-21T13:50:36.170 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.170 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:50:36.170 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.170 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:50:36.170 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.698899+0000", 2024-09-21T13:50:36.170 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:36.171 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.171 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.171 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:50:36.171 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.698899+0000", 2024-09-21T13:50:36.171 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:36.171 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.171 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.171 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:50:36.171 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.698899+0000", 2024-09-21T13:50:36.171 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4799999999999998e-07 2024-09-21T13:50:36.171 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.171 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.171 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:50:36.171 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.698921+0000", 2024-09-21T13:50:36.171 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1942000000000001e-05 2024-09-21T13:50:36.172 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.172 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.172 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:50:36.172 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.698935+0000", 2024-09-21T13:50:36.172 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3798000000000001e-05 2024-09-21T13:50:36.172 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.172 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.172 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:50:36.172 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.698942+0000", 2024-09-21T13:50:36.172 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4570000000000002e-06 2024-09-21T13:50:36.172 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.172 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.172 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:50:36.172 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.699176+0000", 2024-09-21T13:50:36.173 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000234446 2024-09-21T13:50:36.173 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.173 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.173 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:50:36.173 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.699185+0000", 2024-09-21T13:50:36.173 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9930000000000001e-06 2024-09-21T13:50:36.173 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.173 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.173 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:50:36.173 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.694230+0000", 2024-09-21T13:50:36.173 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.99504515500000001 2024-09-21T13:50:36.173 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.173 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.173 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:50:36.174 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.694419+0000", 2024-09-21T13:50:36.174 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00018863800000000001 2024-09-21T13:50:36.174 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.174 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.174 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,4", 2024-09-21T13:50:36.174 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.694572+0000", 2024-09-21T13:50:36.174 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000153385 2024-09-21T13:50:36.174 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.174 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.174 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:50:36.175 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.697061+0000", 2024-09-21T13:50:36.175 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0024887680000000001 2024-09-21T13:50:36.175 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.175 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.175 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:36.175 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.698026+0000", 2024-09-21T13:50:36.175 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00096500399999999999 2024-09-21T13:50:36.175 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.175 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.175 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:36.175 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.703311+0000", 2024-09-21T13:50:36.175 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.005285171 2024-09-21T13:50:36.175 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.175 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.176 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:50:36.176 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.703371+0000", 2024-09-21T13:50:36.176 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9296999999999998e-05 2024-09-21T13:50:36.176 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.176 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.176 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:50:36.176 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.703415+0000", 2024-09-21T13:50:36.176 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4620000000000003e-05 2024-09-21T13:50:36.176 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:36.176 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:50:36.176 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:36.176 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.176 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.176 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:35592 7.b 7:df560bf1:::benchmark_data_smithi049_76798_object35591:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-21T13:50:36.176 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:29.861611+0000", 2024-09-21T13:50:36.177 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 6.2687625530000002, 2024-09-21T13:50:36.177 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.029425032, 2024-09-21T13:50:36.177 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:50:36.177 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:50:36.177 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:50:36.177 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:50:36.177 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:50:36.177 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:50:36.177 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 35592 2024-09-21T13:50:36.177 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.177 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:50:36.177 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.177 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:50:36.178 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.861611+0000", 2024-09-21T13:50:36.178 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:36.178 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.178 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.178 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:50:36.178 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.861611+0000", 2024-09-21T13:50:36.178 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:36.178 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.178 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.178 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:50:36.178 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.861614+0000", 2024-09-21T13:50:36.178 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1179999999999999e-06 2024-09-21T13:50:36.178 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.178 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.179 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:50:36.179 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.861638+0000", 2024-09-21T13:50:36.179 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4474e-05 2024-09-21T13:50:36.179 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.179 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.179 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:50:36.179 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.861650+0000", 2024-09-21T13:50:36.179 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1931e-05 2024-09-21T13:50:36.179 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.179 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.179 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:50:36.179 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.861657+0000", 2024-09-21T13:50:36.179 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4850000000000003e-06 2024-09-21T13:50:36.180 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.180 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.180 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:50:36.180 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.861726+0000", 2024-09-21T13:50:36.180 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8387999999999996e-05 2024-09-21T13:50:36.180 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.180 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.180 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:50:36.180 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.861816+0000", 2024-09-21T13:50:36.180 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0383999999999994e-05 2024-09-21T13:50:36.180 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.180 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.180 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,6", 2024-09-21T13:50:36.181 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.861951+0000", 2024-09-21T13:50:36.181 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013503699999999999 2024-09-21T13:50:36.181 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.181 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.181 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:36.181 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.863596+0000", 2024-09-21T13:50:36.181 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001644235 2024-09-21T13:50:36.181 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.181 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.181 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:50:36.181 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.863700+0000", 2024-09-21T13:50:36.181 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000104808 2024-09-21T13:50:36.181 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.181 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.182 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:36.182 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.890818+0000", 2024-09-21T13:50:36.182 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.027117751999999998 2024-09-21T13:50:36.182 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.182 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.182 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:50:36.182 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.890985+0000", 2024-09-21T13:50:36.182 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016664300000000001 2024-09-21T13:50:36.182 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.182 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.182 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:50:36.182 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.891036+0000", 2024-09-21T13:50:36.182 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1777000000000002e-05 2024-09-21T13:50:36.183 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:36.183 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:50:36.183 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:36.183 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.183 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.183 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:36807 7.9 7:93224273:::benchmark_data_smithi049_76798_object36806:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-21T13:50:36.183 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:30.550197+0000", 2024-09-21T13:50:36.183 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 5.5801771379999998, 2024-09-21T13:50:36.183 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.058156514999999999, 2024-09-21T13:50:36.183 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:50:36.183 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:50:36.183 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:50:36.183 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:50:36.184 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:50:36.184 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:50:36.184 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 36807 2024-09-21T13:50:36.184 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.184 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:50:36.184 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.184 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:50:36.184 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.550197+0000", 2024-09-21T13:50:36.184 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:36.184 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.184 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.184 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:50:36.184 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.550197+0000", 2024-09-21T13:50:36.184 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:36.185 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.185 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.185 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:50:36.185 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.550197+0000", 2024-09-21T13:50:36.185 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5000000000000003e-07 2024-09-21T13:50:36.185 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.185 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.185 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:50:36.185 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.550218+0000", 2024-09-21T13:50:36.185 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0649000000000002e-05 2024-09-21T13:50:36.185 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.185 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.185 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:50:36.185 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.550225+0000", 2024-09-21T13:50:36.186 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1119999999999998e-06 2024-09-21T13:50:36.186 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.186 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.186 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:50:36.186 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.550230+0000", 2024-09-21T13:50:36.186 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.135e-06 2024-09-21T13:50:36.186 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.186 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.186 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:50:36.186 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.552608+0000", 2024-09-21T13:50:36.186 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0023779740000000001 2024-09-21T13:50:36.186 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.186 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.186 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:50:36.187 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.552731+0000", 2024-09-21T13:50:36.187 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012265600000000001 2024-09-21T13:50:36.187 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.187 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.187 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,4", 2024-09-21T13:50:36.187 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.552856+0000", 2024-09-21T13:50:36.187 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000125184 2024-09-21T13:50:36.187 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.187 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.187 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:36.187 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.554088+0000", 2024-09-21T13:50:36.187 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0012316969999999999 2024-09-21T13:50:36.187 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.187 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.188 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:50:36.188 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.554096+0000", 2024-09-21T13:50:36.188 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1049999999999995e-06 2024-09-21T13:50:36.188 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.188 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.188 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:36.188 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.554765+0000", 2024-09-21T13:50:36.188 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00066949999999999996 2024-09-21T13:50:36.188 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.188 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.188 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:50:36.188 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.554796+0000", 2024-09-21T13:50:36.188 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.0471e-05 2024-09-21T13:50:36.188 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.189 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.189 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:50:36.189 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.608353+0000", 2024-09-21T13:50:36.189 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.053557481999999997 2024-09-21T13:50:36.189 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:36.189 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:50:36.189 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:36.189 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.189 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.189 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.14817.0:36814 7.0 e96/83 7:0b36f6f1:::benchmark_data_smithi049_76798_object36813:head v 96'2264, mlcod=96'2262)", 2024-09-21T13:50:36.189 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:30.554590+0000", 2024-09-21T13:50:36.189 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 5.5757843549999997, 2024-09-21T13:50:36.189 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.055937255999999998, 2024-09-21T13:50:36.189 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:50:36.189 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:50:36.190 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:50:36.190 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:50:36.190 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.190 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:50:36.190 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.554590+0000", 2024-09-21T13:50:36.190 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:36.190 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.190 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.190 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:50:36.190 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.554590+0000", 2024-09-21T13:50:36.190 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:36.190 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.190 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.190 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:50:36.191 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.554590+0000", 2024-09-21T13:50:36.191 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8999999999999995e-07 2024-09-21T13:50:36.191 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.191 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.191 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:50:36.191 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.554699+0000", 2024-09-21T13:50:36.191 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010897 2024-09-21T13:50:36.191 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.191 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.191 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:50:36.192 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:50:36.192 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.554708+0000", 2024-09-21T13:50:36.192 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.473e-06 2024-09-21T13:50:36.193 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.193 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.193 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:50:36.193 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.554712+0000", 2024-09-21T13:50:36.193 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4970000000000003e-06 2024-09-21T13:50:36.193 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.193 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.193 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:50:36.193 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.554759+0000", 2024-09-21T13:50:36.193 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6885000000000003e-05 2024-09-21T13:50:36.193 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.193 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.193 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:50:36.194 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.554772+0000", 2024-09-21T13:50:36.194 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2422000000000001e-05 2024-09-21T13:50:36.194 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.194 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.194 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:50:36.194 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.610435+0000", 2024-09-21T13:50:36.194 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.055663231000000001 2024-09-21T13:50:36.194 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.194 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.194 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:50:36.194 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.610527+0000", 2024-09-21T13:50:36.194 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1988000000000001e-05 2024-09-21T13:50:36.195 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:36.195 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:50:36.195 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:36.195 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.195 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.195 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.14817.0:36807 7.9 e96/94 ondisk, result = 0)", 2024-09-21T13:50:36.195 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:30.554699+0000", 2024-09-21T13:50:36.195 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 5.5756753149999998, 2024-09-21T13:50:36.195 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.053663795, 2024-09-21T13:50:36.195 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:50:36.195 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:50:36.195 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T13:50:36.195 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:50:36.196 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.196 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:50:36.196 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.554699+0000", 2024-09-21T13:50:36.196 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:36.196 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.196 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.196 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:50:36.196 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.554699+0000", 2024-09-21T13:50:36.196 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:36.196 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.196 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.196 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:50:36.196 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.554701+0000", 2024-09-21T13:50:36.196 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2469999999999998e-06 2024-09-21T13:50:36.197 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.197 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.197 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:50:36.197 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.554704+0000", 2024-09-21T13:50:36.197 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2629999999999999e-06 2024-09-21T13:50:36.197 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.197 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.197 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:50:36.197 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.554712+0000", 2024-09-21T13:50:36.197 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5859999999999999e-06 2024-09-21T13:50:36.197 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.197 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.197 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:50:36.197 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.554718+0000", 2024-09-21T13:50:36.198 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0900000000000001e-06 2024-09-21T13:50:36.198 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.198 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.198 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:50:36.198 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.554759+0000", 2024-09-21T13:50:36.198 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0881999999999999e-05 2024-09-21T13:50:36.198 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.198 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.198 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:50:36.198 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.554763+0000", 2024-09-21T13:50:36.198 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8800000000000001e-06 2024-09-21T13:50:36.198 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.198 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.199 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:50:36.199 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.608362+0000", 2024-09-21T13:50:36.199 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.053599846999999999 2024-09-21T13:50:36.199 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:36.199 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:50:36.199 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:36.199 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.199 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.199 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:36816 7.9 7:9feceab0:::benchmark_data_smithi049_76798_object36815:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-21T13:50:36.199 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:30.554961+0000", 2024-09-21T13:50:36.199 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 5.575413052, 2024-09-21T13:50:36.199 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.056427244000000001, 2024-09-21T13:50:36.199 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:50:36.200 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:50:36.200 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:50:36.200 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:50:36.200 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:50:36.200 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:50:36.200 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 36816 2024-09-21T13:50:36.200 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.200 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:50:36.200 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.200 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:50:36.200 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.554961+0000", 2024-09-21T13:50:36.200 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:36.200 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.200 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.201 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:50:36.201 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.554961+0000", 2024-09-21T13:50:36.201 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:36.201 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.201 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.201 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:50:36.201 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.554962+0000", 2024-09-21T13:50:36.201 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.288e-06 2024-09-21T13:50:36.201 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.201 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.201 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:50:36.201 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.554983+0000", 2024-09-21T13:50:36.201 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0462000000000002e-05 2024-09-21T13:50:36.202 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.202 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.202 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:50:36.202 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.608317+0000", 2024-09-21T13:50:36.202 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.053333981000000003 2024-09-21T13:50:36.202 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.202 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.202 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:50:36.202 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.608333+0000", 2024-09-21T13:50:36.202 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6107e-05 2024-09-21T13:50:36.202 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.202 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.202 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:50:36.202 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.608393+0000", 2024-09-21T13:50:36.203 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.003e-05 2024-09-21T13:50:36.203 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.203 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.203 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:50:36.203 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.608544+0000", 2024-09-21T13:50:36.203 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000151285 2024-09-21T13:50:36.203 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.203 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.203 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,4", 2024-09-21T13:50:36.203 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.608681+0000", 2024-09-21T13:50:36.203 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000136981 2024-09-21T13:50:36.203 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.203 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.203 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:36.204 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.610786+0000", 2024-09-21T13:50:36.204 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0021052760000000001 2024-09-21T13:50:36.204 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.204 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.204 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:50:36.204 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.610810+0000", 2024-09-21T13:50:36.204 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3907e-05 2024-09-21T13:50:36.204 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.204 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.204 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:36.204 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.611167+0000", 2024-09-21T13:50:36.204 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00035682900000000002 2024-09-21T13:50:36.204 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.204 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.204 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:50:36.205 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.611320+0000", 2024-09-21T13:50:36.205 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00015334600000000001 2024-09-21T13:50:36.205 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.205 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.205 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:50:36.205 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.611388+0000", 2024-09-21T13:50:36.205 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7751999999999998e-05 2024-09-21T13:50:36.205 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:36.205 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:50:36.205 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:36.205 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.205 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.205 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:36851 7.b 7:d46d3307:::benchmark_data_smithi049_76798_object36850:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-21T13:50:36.205 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:30.627278+0000", 2024-09-21T13:50:36.206 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 5.5030959929999996, 2024-09-21T13:50:36.206 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.14148804500000001, 2024-09-21T13:50:36.206 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:50:36.206 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:50:36.206 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:50:36.206 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:50:36.206 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:50:36.206 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:50:36.206 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 36851 2024-09-21T13:50:36.206 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.206 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:50:36.206 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.206 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:50:36.207 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.627278+0000", 2024-09-21T13:50:36.207 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:36.207 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.207 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.207 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:50:36.207 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.627278+0000", 2024-09-21T13:50:36.207 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:36.207 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.207 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.207 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:50:36.207 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.627280+0000", 2024-09-21T13:50:36.207 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7239999999999999e-06 2024-09-21T13:50:36.207 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.208 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.208 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:50:36.208 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.627303+0000", 2024-09-21T13:50:36.208 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2844999999999999e-05 2024-09-21T13:50:36.208 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.208 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.208 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:50:36.208 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.627319+0000", 2024-09-21T13:50:36.208 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6714e-05 2024-09-21T13:50:36.208 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.208 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.208 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:50:36.208 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.627328+0000", 2024-09-21T13:50:36.208 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.7180000000000002e-06 2024-09-21T13:50:36.209 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.209 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.209 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:50:36.209 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.627997+0000", 2024-09-21T13:50:36.209 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00066938200000000005 2024-09-21T13:50:36.209 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.209 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.209 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:50:36.209 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.629243+0000", 2024-09-21T13:50:36.209 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0012453430000000001 2024-09-21T13:50:36.209 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.209 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.209 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,6", 2024-09-21T13:50:36.209 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.629398+0000", 2024-09-21T13:50:36.210 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000155179 2024-09-21T13:50:36.210 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.210 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.210 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:36.210 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.631277+0000", 2024-09-21T13:50:36.210 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001879181 2024-09-21T13:50:36.210 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.210 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.210 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:50:36.210 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.631907+0000", 2024-09-21T13:50:36.210 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00062996799999999996 2024-09-21T13:50:36.210 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.210 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.210 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:36.211 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.768520+0000", 2024-09-21T13:50:36.211 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.13661250899999999 2024-09-21T13:50:36.211 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.211 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.211 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:50:36.211 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.768705+0000", 2024-09-21T13:50:36.211 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00018564699999999999 2024-09-21T13:50:36.211 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.211 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.211 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:50:36.211 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.768766+0000", 2024-09-21T13:50:36.211 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0835000000000003e-05 2024-09-21T13:50:36.211 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:36.212 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:50:36.212 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:36.212 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.212 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.212 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:36870 7.b 7:d08d3ab5:::benchmark_data_smithi049_76798_object36869:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-21T13:50:36.212 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:30.637634+0000", 2024-09-21T13:50:36.212 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 5.4927398700000003, 2024-09-21T13:50:36.212 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.13145359500000001, 2024-09-21T13:50:36.212 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:50:36.212 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:50:36.212 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:50:36.212 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:50:36.212 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:50:36.213 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:50:36.213 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 36870 2024-09-21T13:50:36.213 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.213 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:50:36.213 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.213 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:50:36.213 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.637634+0000", 2024-09-21T13:50:36.213 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:36.213 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.213 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.213 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:50:36.213 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.637634+0000", 2024-09-21T13:50:36.213 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:36.213 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.213 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.214 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:50:36.214 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.637636+0000", 2024-09-21T13:50:36.214 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5489999999999999e-06 2024-09-21T13:50:36.214 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.214 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.214 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:50:36.214 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.637759+0000", 2024-09-21T13:50:36.214 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012323799999999999 2024-09-21T13:50:36.214 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.214 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.214 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:50:36.214 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.637772+0000", 2024-09-21T13:50:36.214 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3318e-05 2024-09-21T13:50:36.215 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.215 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.215 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:50:36.215 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.637780+0000", 2024-09-21T13:50:36.215 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3710000000000002e-06 2024-09-21T13:50:36.215 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.215 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.215 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:50:36.215 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.637841+0000", 2024-09-21T13:50:36.215 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0897999999999999e-05 2024-09-21T13:50:36.215 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.215 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.215 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:50:36.215 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.638039+0000", 2024-09-21T13:50:36.216 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00019803900000000001 2024-09-21T13:50:36.216 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.216 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.216 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,6", 2024-09-21T13:50:36.216 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.638176+0000", 2024-09-21T13:50:36.216 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013776199999999999 2024-09-21T13:50:36.216 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.216 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.216 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:36.216 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.639679+0000", 2024-09-21T13:50:36.216 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0015029450000000001 2024-09-21T13:50:36.216 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.216 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.216 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:50:36.217 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.639708+0000", 2024-09-21T13:50:36.217 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.8839e-05 2024-09-21T13:50:36.217 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.217 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.217 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:36.217 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.768901+0000", 2024-09-21T13:50:36.217 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.12919276900000001 2024-09-21T13:50:36.217 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.217 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.217 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:50:36.217 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.769000+0000", 2024-09-21T13:50:36.217 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9220000000000002e-05 2024-09-21T13:50:36.217 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.218 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.218 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:50:36.218 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.769088+0000", 2024-09-21T13:50:36.218 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.7646999999999996e-05 2024-09-21T13:50:36.218 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:36.218 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:50:36.218 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:36.218 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.218 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.218 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:36884 7.b 7:d5aed2de:::benchmark_data_smithi049_76798_object36883:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-21T13:50:36.218 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:30.647863+0000", 2024-09-21T13:50:36.218 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 5.4825106430000003, 2024-09-21T13:50:36.218 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.12384792999999999, 2024-09-21T13:50:36.219 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:50:36.219 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:50:36.219 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:50:36.219 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:50:36.219 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:50:36.219 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:50:36.219 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 36884 2024-09-21T13:50:36.219 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.219 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:50:36.219 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.219 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:50:36.219 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.647863+0000", 2024-09-21T13:50:36.219 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:36.220 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.220 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.220 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:50:36.220 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.647863+0000", 2024-09-21T13:50:36.220 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:36.220 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.220 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.220 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:50:36.220 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.647865+0000", 2024-09-21T13:50:36.220 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4339999999999999e-06 2024-09-21T13:50:36.220 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.220 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.221 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:35 smithi187 ceph-mon[33032]: 6.d scrub starts 2024-09-21T13:50:36.221 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:35 smithi187 ceph-mon[33032]: 6.d scrub ok 2024-09-21T13:50:36.221 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:35 smithi187 ceph-mon[33032]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T13:50:36.221 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:35 smithi187 ceph-mon[33032]: Cluster is now healthy 2024-09-21T13:50:36.221 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:35 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T13:50:36.221 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:35 smithi187 ceph-mon[33032]: osdmap e97: 8 total, 8 up, 8 in 2024-09-21T13:50:36.221 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:35 smithi187 ceph-mon[33032]: osdmap e98: 8 total, 8 up, 8 in 2024-09-21T13:50:36.222 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:35 smithi187 ceph-mon[33032]: 6.1d scrub starts 2024-09-21T13:50:36.222 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:35 smithi187 ceph-mon[33032]: 6.1d scrub ok 2024-09-21T13:50:36.222 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:35 smithi187 ceph-mon[33032]: pgmap v60: 148 pgs: 148 active+clean; 2.5 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 361 B/s rd, 76 MiB/s wr, 1.22k op/s; 38 B/s, 2 objects/s recovering 2024-09-21T13:50:36.222 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:35 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:36.222 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:35 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:36.222 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:50:36.222 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.647886+0000", 2024-09-21T13:50:36.222 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0919000000000001e-05 2024-09-21T13:50:36.223 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.223 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.223 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:50:36.223 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.647899+0000", 2024-09-21T13:50:36.223 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2995e-05 2024-09-21T13:50:36.223 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.223 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.223 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:50:36.223 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.647905+0000", 2024-09-21T13:50:36.223 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.212e-06 2024-09-21T13:50:36.223 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.223 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.223 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:50:36.224 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.647964+0000", 2024-09-21T13:50:36.224 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9555000000000002e-05 2024-09-21T13:50:36.224 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.224 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.224 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:50:36.224 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.648095+0000", 2024-09-21T13:50:36.224 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013020800000000001 2024-09-21T13:50:36.224 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.224 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.224 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,6", 2024-09-21T13:50:36.224 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.648210+0000", 2024-09-21T13:50:36.224 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000114827 2024-09-21T13:50:36.224 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.224 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.224 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:50:36.225 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.649302+0000", 2024-09-21T13:50:36.225 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0010921010000000001 2024-09-21T13:50:36.225 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.225 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.225 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:36.225 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.649464+0000", 2024-09-21T13:50:36.225 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016226200000000001 2024-09-21T13:50:36.225 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.225 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.225 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:36.225 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.771611+0000", 2024-09-21T13:50:36.225 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.122147371 2024-09-21T13:50:36.225 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.225 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.226 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:50:36.226 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.771668+0000", 2024-09-21T13:50:36.226 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6802000000000003e-05 2024-09-21T13:50:36.226 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.226 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.226 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:50:36.226 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.771711+0000", 2024-09-21T13:50:36.226 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3244000000000001e-05 2024-09-21T13:50:36.226 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:36.226 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:50:36.226 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:36.226 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.226 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.227 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:37302 7.b 7:d3ea8f73:::benchmark_data_smithi049_76798_object37301:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-21T13:50:36.227 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:30.982008+0000", 2024-09-21T13:50:36.227 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 5.1483664420000004, 2024-09-21T13:50:36.227 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.044150149, 2024-09-21T13:50:36.227 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:50:36.227 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:50:36.227 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:50:36.227 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:50:36.227 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:50:36.227 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:50:36.227 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 37302 2024-09-21T13:50:36.227 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.227 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:50:36.227 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.228 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:50:36.228 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.982008+0000", 2024-09-21T13:50:36.228 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:36.228 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.228 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.228 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:50:36.228 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.982008+0000", 2024-09-21T13:50:36.228 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:36.228 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.228 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.228 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:50:36.228 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.982009+0000", 2024-09-21T13:50:36.228 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1689999999999999e-06 2024-09-21T13:50:36.228 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.228 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.229 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:50:36.229 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.982029+0000", 2024-09-21T13:50:36.229 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0449999999999999e-05 2024-09-21T13:50:36.229 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.229 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.229 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:50:36.229 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.982042+0000", 2024-09-21T13:50:36.229 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2404e-05 2024-09-21T13:50:36.229 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.229 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.229 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:50:36.229 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.982064+0000", 2024-09-21T13:50:36.229 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2541e-05 2024-09-21T13:50:36.229 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.230 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.230 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:50:36.230 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.983963+0000", 2024-09-21T13:50:36.230 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0018985230000000001 2024-09-21T13:50:36.230 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.230 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.230 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:50:36.230 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.984092+0000", 2024-09-21T13:50:36.230 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012920600000000001 2024-09-21T13:50:36.230 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.230 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.230 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,6", 2024-09-21T13:50:36.230 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.984237+0000", 2024-09-21T13:50:36.231 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000144943 2024-09-21T13:50:36.231 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.231 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.231 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:50:36.231 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.985858+0000", 2024-09-21T13:50:36.231 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001621543 2024-09-21T13:50:36.231 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.231 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.231 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:36.231 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.985930+0000", 2024-09-21T13:50:36.231 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1927000000000004e-05 2024-09-21T13:50:36.231 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.231 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.231 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:36.231 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:31.025963+0000", 2024-09-21T13:50:36.232 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.040032542999999997 2024-09-21T13:50:36.232 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.232 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.232 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:50:36.232 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:31.026109+0000", 2024-09-21T13:50:36.232 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014574600000000001 2024-09-21T13:50:36.232 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.232 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.232 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:50:36.232 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:31.026158+0000", 2024-09-21T13:50:36.232 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9153999999999997e-05 2024-09-21T13:50:36.232 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:36.232 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:50:36.232 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:36.233 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.233 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.233 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:37304 7.9 7:9dbea9ab:::benchmark_data_smithi049_76798_object37303:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-21T13:50:36.233 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:30.982827+0000", 2024-09-21T13:50:36.233 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 5.1475471959999997, 2024-09-21T13:50:36.233 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.046057406000000002, 2024-09-21T13:50:36.233 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:50:36.233 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:50:36.233 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:50:36.233 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:50:36.233 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:50:36.233 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:50:36.233 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 37304 2024-09-21T13:50:36.233 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.234 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:50:36.234 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.234 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:50:36.234 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.982827+0000", 2024-09-21T13:50:36.234 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:36.234 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.234 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.234 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:50:36.234 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.982827+0000", 2024-09-21T13:50:36.234 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:36.234 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.234 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.234 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:50:36.234 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.982828+0000", 2024-09-21T13:50:36.235 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0499999999999999e-06 2024-09-21T13:50:36.235 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.235 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.235 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:50:36.235 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.982848+0000", 2024-09-21T13:50:36.235 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9947999999999999e-05 2024-09-21T13:50:36.235 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.235 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.235 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:50:36.235 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.982862+0000", 2024-09-21T13:50:36.235 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.385e-05 2024-09-21T13:50:36.235 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.235 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.235 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:50:36.236 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.982869+0000", 2024-09-21T13:50:36.236 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2640000000000002e-06 2024-09-21T13:50:36.236 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.236 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.236 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:50:36.236 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.982927+0000", 2024-09-21T13:50:36.236 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8553e-05 2024-09-21T13:50:36.236 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.236 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.236 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:50:36.236 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.983048+0000", 2024-09-21T13:50:36.236 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012084899999999999 2024-09-21T13:50:36.236 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.236 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.237 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,4", 2024-09-21T13:50:36.237 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.983178+0000", 2024-09-21T13:50:36.237 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000129546 2024-09-21T13:50:36.237 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.237 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.237 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:50:36.237 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.984732+0000", 2024-09-21T13:50:36.237 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0015539110000000001 2024-09-21T13:50:36.237 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.237 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.237 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:36.237 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.985011+0000", 2024-09-21T13:50:36.237 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00027897799999999998 2024-09-21T13:50:36.237 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.237 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.238 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:36.238 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:31.028716+0000", 2024-09-21T13:50:36.238 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.043705212 2024-09-21T13:50:36.238 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.238 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.238 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:50:36.238 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:31.028839+0000", 2024-09-21T13:50:36.238 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012271600000000001 2024-09-21T13:50:36.238 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.238 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.238 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:50:36.238 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:31.028884+0000", 2024-09-21T13:50:36.238 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5528999999999997e-05 2024-09-21T13:50:36.239 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:36.239 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:50:36.239 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:36.239 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.239 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.239 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:37306 7.b 7:d0981cac:::benchmark_data_smithi049_76798_object37305:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-21T13:50:36.239 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:30.985447+0000", 2024-09-21T13:50:36.239 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 5.1449266549999999, 2024-09-21T13:50:36.239 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.040936221000000002, 2024-09-21T13:50:36.239 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:50:36.239 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:50:36.239 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:50:36.239 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:50:36.239 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:50:36.240 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:50:36.240 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 37306 2024-09-21T13:50:36.240 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.240 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:50:36.240 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.240 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:50:36.240 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.985447+0000", 2024-09-21T13:50:36.240 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:36.240 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.240 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.240 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:50:36.240 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.985447+0000", 2024-09-21T13:50:36.240 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:36.240 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.241 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.241 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:50:36.241 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.985449+0000", 2024-09-21T13:50:36.241 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.531e-06 2024-09-21T13:50:36.241 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.241 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.241 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:50:36.241 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.985472+0000", 2024-09-21T13:50:36.241 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3510999999999999e-05 2024-09-21T13:50:36.241 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.241 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.241 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:50:36.241 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.985486+0000", 2024-09-21T13:50:36.241 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3675e-05 2024-09-21T13:50:36.242 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.242 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.242 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:50:36.242 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.985513+0000", 2024-09-21T13:50:36.242 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6665e-05 2024-09-21T13:50:36.242 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.242 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.242 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:50:36.242 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.985969+0000", 2024-09-21T13:50:36.242 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00045631799999999999 2024-09-21T13:50:36.242 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.242 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.242 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:50:36.243 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.986095+0000", 2024-09-21T13:50:36.243 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000126024 2024-09-21T13:50:36.243 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.243 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.243 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,6", 2024-09-21T13:50:36.243 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.986228+0000", 2024-09-21T13:50:36.243 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013327400000000001 2024-09-21T13:50:36.243 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.243 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.243 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:36.244 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.987380+0000", 2024-09-21T13:50:36.244 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001152134 2024-09-21T13:50:36.244 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.244 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.244 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:50:36.244 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.987483+0000", 2024-09-21T13:50:36.244 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000102427 2024-09-21T13:50:36.244 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.244 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.244 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:36.244 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:31.026276+0000", 2024-09-21T13:50:36.245 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.038793484000000003 2024-09-21T13:50:36.245 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.245 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.245 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:50:36.245 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:31.026344+0000", 2024-09-21T13:50:36.245 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7188999999999997e-05 2024-09-21T13:50:36.245 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.245 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.245 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:50:36.245 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:31.026384+0000", 2024-09-21T13:50:36.245 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9988999999999999e-05 2024-09-21T13:50:36.245 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:36.245 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:50:36.245 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:36.245 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.246 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.246 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:41235 7.b 7:d3b3f630:::benchmark_data_smithi049_76798_object41234:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-21T13:50:36.246 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:33.157364+0000", 2024-09-21T13:50:36.246 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 2.9730097249999998, 2024-09-21T13:50:36.246 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.024907629000000001, 2024-09-21T13:50:36.246 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:50:36.246 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:50:36.246 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:50:36.246 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:50:36.246 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:50:36.246 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:50:36.246 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 41235 2024-09-21T13:50:36.246 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.246 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:50:36.247 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.247 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:50:36.247 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:33.157364+0000", 2024-09-21T13:50:36.247 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:36.247 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.247 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.247 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:50:36.247 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:33.157364+0000", 2024-09-21T13:50:36.247 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:36.247 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.247 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.247 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:50:36.247 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:33.157366+0000", 2024-09-21T13:50:36.247 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0140000000000001e-06 2024-09-21T13:50:36.248 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.248 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.248 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:50:36.248 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:33.157391+0000", 2024-09-21T13:50:36.248 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5066999999999999e-05 2024-09-21T13:50:36.248 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.248 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.248 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:50:36.248 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:33.157406+0000", 2024-09-21T13:50:36.248 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4783999999999999e-05 2024-09-21T13:50:36.248 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.248 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.248 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:50:36.248 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:33.157415+0000", 2024-09-21T13:50:36.248 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.7150000000000004e-06 2024-09-21T13:50:36.249 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.249 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.249 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:50:36.249 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:33.157474+0000", 2024-09-21T13:50:36.249 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9089999999999998e-05 2024-09-21T13:50:36.249 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.249 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.249 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:50:36.249 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:33.157595+0000", 2024-09-21T13:50:36.249 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000121384 2024-09-21T13:50:36.249 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.249 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.249 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,6", 2024-09-21T13:50:36.249 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:33.157721+0000", 2024-09-21T13:50:36.250 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000126031 2024-09-21T13:50:36.250 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.250 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.250 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:36.250 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:33.179873+0000", 2024-09-21T13:50:36.250 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.022151535 2024-09-21T13:50:36.250 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.250 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.250 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:36.250 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:33.179937+0000", 2024-09-21T13:50:36.250 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4412000000000006e-05 2024-09-21T13:50:36.250 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.250 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.250 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:50:36.251 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:33.182163+0000", 2024-09-21T13:50:36.251 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0022252859999999999 2024-09-21T13:50:36.251 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.251 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.251 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:50:36.251 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:33.182224+0000", 2024-09-21T13:50:36.251 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1105000000000005e-05 2024-09-21T13:50:36.251 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.251 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.251 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:50:36.251 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:33.182272+0000", 2024-09-21T13:50:36.251 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8205999999999999e-05 2024-09-21T13:50:36.251 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:36.252 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:50:36.252 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:36.252 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.252 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.252 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:41236 7.9 7:952f4a54:::benchmark_data_smithi049_76798_object41235:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-21T13:50:36.252 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:33.157548+0000", 2024-09-21T13:50:36.252 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 2.9728255839999997, 2024-09-21T13:50:36.252 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.025328659, 2024-09-21T13:50:36.252 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:50:36.252 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:50:36.252 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:50:36.252 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:50:36.252 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:50:36.252 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:50:36.252 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 41236 2024-09-21T13:50:36.253 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.253 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:50:36.253 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.253 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:50:36.253 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:33.157548+0000", 2024-09-21T13:50:36.253 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:36.253 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.253 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.253 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:50:36.253 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:33.157548+0000", 2024-09-21T13:50:36.253 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:36.253 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.253 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.253 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:50:36.254 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:33.157549+0000", 2024-09-21T13:50:36.254 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2900000000000002e-07 2024-09-21T13:50:36.254 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.254 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.254 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:50:36.254 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:33.157574+0000", 2024-09-21T13:50:36.254 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4825999999999999e-05 2024-09-21T13:50:36.254 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.254 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.254 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:50:36.254 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:33.157585+0000", 2024-09-21T13:50:36.254 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1031e-05 2024-09-21T13:50:36.254 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.254 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.254 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:50:36.255 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:33.157592+0000", 2024-09-21T13:50:36.255 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1770000000000001e-06 2024-09-21T13:50:36.255 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.255 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.255 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:50:36.255 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:33.157870+0000", 2024-09-21T13:50:36.255 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00027776299999999999 2024-09-21T13:50:36.255 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.255 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.255 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:50:36.255 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:33.177806+0000", 2024-09-21T13:50:36.255 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.01993549 2024-09-21T13:50:36.255 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.255 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.256 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,4", 2024-09-21T13:50:36.256 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:33.177931+0000", 2024-09-21T13:50:36.256 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000125525 2024-09-21T13:50:36.256 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.256 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.256 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:36.256 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:33.179973+0000", 2024-09-21T13:50:36.256 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0020422769999999999 2024-09-21T13:50:36.256 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.256 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.256 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:50:36.256 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:33.179993+0000", 2024-09-21T13:50:36.256 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9332000000000001e-05 2024-09-21T13:50:36.256 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.257 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.257 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:36.257 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:33.182262+0000", 2024-09-21T13:50:36.257 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0022695509999999999 2024-09-21T13:50:36.257 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.257 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.257 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:50:36.257 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:33.182836+0000", 2024-09-21T13:50:36.257 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00057378300000000002 2024-09-21T13:50:36.257 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.257 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.257 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:50:36.257 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:33.182877+0000", 2024-09-21T13:50:36.257 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0975e-05 2024-09-21T13:50:36.257 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:36.258 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:50:36.258 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:36.258 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.258 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.258 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:43695 7.b 7:d076fa8f:::benchmark_data_smithi049_76798_object43694:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-21T13:50:36.258 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:34.725693+0000", 2024-09-21T13:50:36.258 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 1.4046809950000001, 2024-09-21T13:50:36.258 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.020685878000000001, 2024-09-21T13:50:36.258 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:50:36.258 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:50:36.258 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:50:36.258 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:50:36.258 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:50:36.258 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:50:36.259 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 43695 2024-09-21T13:50:36.259 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.259 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:50:36.259 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.259 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:50:36.259 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:34.725693+0000", 2024-09-21T13:50:36.259 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:36.259 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.259 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.259 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:50:36.259 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:34.725693+0000", 2024-09-21T13:50:36.259 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:50:36.259 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.259 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.259 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:50:36.260 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:34.725694+0000", 2024-09-21T13:50:36.260 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0100000000000005e-07 2024-09-21T13:50:36.260 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.260 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.260 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:50:36.260 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:34.725721+0000", 2024-09-21T13:50:36.260 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.758e-05 2024-09-21T13:50:36.260 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.260 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.260 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:50:36.260 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:34.725730+0000", 2024-09-21T13:50:36.260 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6079999999999996e-06 2024-09-21T13:50:36.260 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.260 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.261 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:50:36.261 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:34.725736+0000", 2024-09-21T13:50:36.261 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1380000000000004e-06 2024-09-21T13:50:36.261 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.261 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.261 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:50:36.261 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:34.730164+0000", 2024-09-21T13:50:36.261 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0044284579999999997 2024-09-21T13:50:36.261 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.261 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.261 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:50:36.261 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:34.730284+0000", 2024-09-21T13:50:36.261 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011986500000000001 2024-09-21T13:50:36.261 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.261 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.262 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,6", 2024-09-21T13:50:36.262 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:34.730417+0000", 2024-09-21T13:50:36.262 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013265 2024-09-21T13:50:36.262 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.262 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.262 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:50:36.262 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:34.734859+0000", 2024-09-21T13:50:36.262 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0044417190000000002 2024-09-21T13:50:36.262 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.262 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.262 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:36.262 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:34.741142+0000", 2024-09-21T13:50:36.262 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00628292 2024-09-21T13:50:36.262 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.262 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.263 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:50:36.263 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:34.746177+0000", 2024-09-21T13:50:36.263 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0050349940000000001 2024-09-21T13:50:36.264 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.264 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.264 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:50:36.264 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:34.746337+0000", 2024-09-21T13:50:36.264 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016056499999999999 2024-09-21T13:50:36.264 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:50:36.264 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:50:36.264 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:50:36.264 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:34.746379+0000", 2024-09-21T13:50:36.264 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1780000000000003e-05 2024-09-21T13:50:36.264 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:36.264 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:50:36.264 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:36.264 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:50:36.265 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:50:36.265 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T13:50:36.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:36.276 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:36.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:50:36.376 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:50:36.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:36.407 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:36.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:50:36.508 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:50:36.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:36.547 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:36.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:50:36.648 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:36.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:36.678 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:36.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:36.779 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:50:36.806 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:36.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:36.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:50:36.905 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 24 16 47318 47302 123.112 99.9375 0.00547516 0.00811755 2024-09-21T13:50:36.908 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:50:36.948 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:36.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:36.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:50:37.050 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:37.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:37.077 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:37.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:37.178 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:37.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:36 smithi187 ceph-mon[33032]: 6.8 scrub starts 2024-09-21T13:50:37.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:36 smithi187 ceph-mon[33032]: 6.8 scrub ok 2024-09-21T13:50:37.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:36 smithi187 ceph-mon[33032]: 3.5 scrub starts 2024-09-21T13:50:37.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:36 smithi187 ceph-mon[33032]: 3.5 scrub ok 2024-09-21T13:50:37.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:36 smithi187 ceph-mon[33032]: 4.1f scrub starts 2024-09-21T13:50:37.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:36 smithi187 ceph-mon[33032]: 4.1f scrub ok 2024-09-21T13:50:37.209 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:37.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:37.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:37.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:36 smithi049 ceph-mon[30080]: 6.8 scrub starts 2024-09-21T13:50:37.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:36 smithi049 ceph-mon[30080]: 6.8 scrub ok 2024-09-21T13:50:37.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:36 smithi049 ceph-mon[30080]: 3.5 scrub starts 2024-09-21T13:50:37.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:36 smithi049 ceph-mon[30080]: 3.5 scrub ok 2024-09-21T13:50:37.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:36 smithi049 ceph-mon[30080]: 4.1f scrub starts 2024-09-21T13:50:37.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:36 smithi049 ceph-mon[30080]: 4.1f scrub ok 2024-09-21T13:50:37.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:36 smithi049 ceph-mon[40489]: 6.8 scrub starts 2024-09-21T13:50:37.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:36 smithi049 ceph-mon[40489]: 6.8 scrub ok 2024-09-21T13:50:37.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:36 smithi049 ceph-mon[40489]: 3.5 scrub starts 2024-09-21T13:50:37.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:36 smithi049 ceph-mon[40489]: 3.5 scrub ok 2024-09-21T13:50:37.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:36 smithi049 ceph-mon[40489]: 4.1f scrub starts 2024-09-21T13:50:37.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:36 smithi049 ceph-mon[40489]: 4.1f scrub ok 2024-09-21T13:50:37.311 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:50:37.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:37.350 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:37.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:50:37.451 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:50:37.492 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:37.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:37.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:50:37.594 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:50:37.632 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:37.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:37.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:50:37.734 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:50:37.760 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:37.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:37.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:50:37.817 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T13:50:37.861 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:50:37.905 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 25 16 48875 48859 122.08 97.3125 0.00738115 0.008183 2024-09-21T13:50:37.906 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:37.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:37.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:50:38.007 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:38.035 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:38.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:38.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:38.137 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:50:38.138 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:37 smithi049 ceph-mon[30080]: 5.1e scrub starts 2024-09-21T13:50:38.138 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:37 smithi049 ceph-mon[30080]: 5.1e scrub ok 2024-09-21T13:50:38.138 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:37 smithi049 ceph-mon[30080]: pgmap v61: 148 pgs: 148 active+clean; 2.5 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 272 B/s rd, 57 MiB/s wr, 916 op/s; 29 B/s, 1 objects/s recovering 2024-09-21T13:50:38.138 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:37 smithi049 ceph-mon[40489]: 5.1e scrub starts 2024-09-21T13:50:38.138 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:37 smithi049 ceph-mon[40489]: 5.1e scrub ok 2024-09-21T13:50:38.138 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:37 smithi049 ceph-mon[40489]: pgmap v61: 148 pgs: 148 active+clean; 2.5 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 272 B/s rd, 57 MiB/s wr, 916 op/s; 29 B/s, 1 objects/s recovering 2024-09-21T13:50:38.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:37 smithi187 ceph-mon[33032]: 5.1e scrub starts 2024-09-21T13:50:38.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:37 smithi187 ceph-mon[33032]: 5.1e scrub ok 2024-09-21T13:50:38.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:37 smithi187 ceph-mon[33032]: pgmap v61: 148 pgs: 148 active+clean; 2.5 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 272 B/s rd, 57 MiB/s wr, 916 op/s; 29 B/s, 1 objects/s recovering 2024-09-21T13:50:38.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:38.205 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:38.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:50:38.306 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:50:38.327 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:38.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:38.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:50:38.337 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T13:50:38.337 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T13:50:38.337 INFO:tasks.thrashosds.thrasher:Killing osd 2, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T13:50:38.337 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-21T13:50:38.338 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@osd.2 2024-09-21T13:50:38.428 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:50:38.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:38.461 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:38.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:50:38.562 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:38.590 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:38.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:38.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:38.692 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:38.718 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:38.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:38.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:50:38.754 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:50:38 smithi049 systemd[1]: Stopping Ceph osd.2 for 40a3d2e8-781f-11ef-baf6-efdc52797490... 2024-09-21T13:50:38.820 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:38.844 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:38.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:38.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:38.905 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 26 16 49983 49967 120.049 69.25 0.00754805 0.00832509 2024-09-21T13:50:38.945 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:50:38.985 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:38.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:38.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:50:39.087 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:50:39.115 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:39.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:39.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:50:39.194 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:50:38 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:50:39.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:38 smithi187 ceph-mon[33032]: 5.10 scrub starts 2024-09-21T13:50:39.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:38 smithi187 ceph-mon[33032]: 5.10 scrub ok 2024-09-21T13:50:39.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:38 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:50:39.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:38 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:50:39.217 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:50:39.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:38 smithi049 ceph-mon[40489]: 5.10 scrub starts 2024-09-21T13:50:39.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:38 smithi049 ceph-mon[40489]: 5.10 scrub ok 2024-09-21T13:50:39.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:38 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:50:39.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:38 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:50:39.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:38 smithi049 ceph-mon[30080]: 5.10 scrub starts 2024-09-21T13:50:39.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:38 smithi049 ceph-mon[30080]: 5.10 scrub ok 2024-09-21T13:50:39.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:38 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:50:39.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:38 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:50:39.257 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:39.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:39.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:50:39.359 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:50:39.389 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:39.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:39.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:50:39.490 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:39.514 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:39.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:39.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:50:39.616 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:39.641 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:39.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:39.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:39.743 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:39.771 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:39.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:39.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:50:39.873 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:39.900 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:39.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:39.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:39.905 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 27 16 51563 51547 119.26 98.75 0.00755481 0.00837965 2024-09-21T13:50:40.002 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:50:40.024 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:40.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:40.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:50:40.125 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:40.150 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:40.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:40.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:40.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:39 smithi187 ceph-mon[33032]: 6.1e scrub starts 2024-09-21T13:50:40.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:39 smithi187 ceph-mon[33032]: 6.1e scrub ok 2024-09-21T13:50:40.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:39 smithi187 ceph-mon[33032]: 3.c scrub starts 2024-09-21T13:50:40.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:39 smithi187 ceph-mon[33032]: 3.c scrub ok 2024-09-21T13:50:40.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:39 smithi187 ceph-mon[33032]: osdmap e99: 8 total, 8 up, 8 in 2024-09-21T13:50:40.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:39 smithi187 ceph-mon[33032]: 5.1d scrub starts 2024-09-21T13:50:40.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:39 smithi187 ceph-mon[33032]: 5.1d scrub ok 2024-09-21T13:50:40.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:39 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:50:40.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:39 smithi187 ceph-mon[33032]: pgmap v63: 148 pgs: 148 active+clean; 3.0 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 682 B/s rd, 100 MiB/s wr, 1.59k op/s; 0 B/s, 0 objects/s recovering 2024-09-21T13:50:40.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:39 smithi187 ceph-mon[33032]: 5.13 scrub starts 2024-09-21T13:50:40.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:39 smithi187 ceph-mon[33032]: 5.13 scrub ok 2024-09-21T13:50:40.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:39 smithi187 ceph-mon[33032]: osdmap e100: 8 total, 8 up, 8 in 2024-09-21T13:50:40.252 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:40.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:39 smithi049 ceph-mon[40489]: 6.1e scrub starts 2024-09-21T13:50:40.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:39 smithi049 ceph-mon[40489]: 6.1e scrub ok 2024-09-21T13:50:40.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:39 smithi049 ceph-mon[40489]: 3.c scrub starts 2024-09-21T13:50:40.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:39 smithi049 ceph-mon[40489]: 3.c scrub ok 2024-09-21T13:50:40.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:39 smithi049 ceph-mon[40489]: osdmap e99: 8 total, 8 up, 8 in 2024-09-21T13:50:40.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:39 smithi049 ceph-mon[40489]: 5.1d scrub starts 2024-09-21T13:50:40.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:39 smithi049 ceph-mon[40489]: 5.1d scrub ok 2024-09-21T13:50:40.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:39 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:50:40.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:39 smithi049 ceph-mon[40489]: pgmap v63: 148 pgs: 148 active+clean; 3.0 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 682 B/s rd, 100 MiB/s wr, 1.59k op/s; 0 B/s, 0 objects/s recovering 2024-09-21T13:50:40.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:39 smithi049 ceph-mon[40489]: 5.13 scrub starts 2024-09-21T13:50:40.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:39 smithi049 ceph-mon[40489]: 5.13 scrub ok 2024-09-21T13:50:40.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:39 smithi049 ceph-mon[40489]: osdmap e100: 8 total, 8 up, 8 in 2024-09-21T13:50:40.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:39 smithi049 ceph-mon[30080]: 6.1e scrub starts 2024-09-21T13:50:40.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:39 smithi049 ceph-mon[30080]: 6.1e scrub ok 2024-09-21T13:50:40.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:39 smithi049 ceph-mon[30080]: 3.c scrub starts 2024-09-21T13:50:40.257 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:39 smithi049 ceph-mon[30080]: 3.c scrub ok 2024-09-21T13:50:40.257 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:39 smithi049 ceph-mon[30080]: osdmap e99: 8 total, 8 up, 8 in 2024-09-21T13:50:40.257 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:39 smithi049 ceph-mon[30080]: 5.1d scrub starts 2024-09-21T13:50:40.257 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:39 smithi049 ceph-mon[30080]: 5.1d scrub ok 2024-09-21T13:50:40.257 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:39 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:50:40.257 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:39 smithi049 ceph-mon[30080]: pgmap v63: 148 pgs: 148 active+clean; 3.0 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 682 B/s rd, 100 MiB/s wr, 1.59k op/s; 0 B/s, 0 objects/s recovering 2024-09-21T13:50:40.257 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:39 smithi049 ceph-mon[30080]: 5.13 scrub starts 2024-09-21T13:50:40.257 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:39 smithi049 ceph-mon[30080]: 5.13 scrub ok 2024-09-21T13:50:40.257 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:39 smithi049 ceph-mon[30080]: osdmap e100: 8 total, 8 up, 8 in 2024-09-21T13:50:40.281 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:40.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:40.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:50:40.383 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:50:40.411 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:40.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:40.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:50:40.513 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:40.539 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:40.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:40.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:40.641 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:50:40.683 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:40.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:40.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:50:40.785 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:40.811 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:40.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:40.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:40.872 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:50:40 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:50:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:50:40.905 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 28 16 52983 52967 118.171 88.75 0.00802422 0.00845668 2024-09-21T13:50:40.913 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:40.942 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:40.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:40.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:41.044 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:50:41.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:41.086 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:41.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:50:41.187 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:41.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:40 smithi187 ceph-mon[33032]: 6.17 scrub starts 2024-09-21T13:50:41.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:40 smithi187 ceph-mon[33032]: 6.17 scrub ok 2024-09-21T13:50:41.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:40 smithi187 ceph-mon[33032]: 6.12 scrub starts 2024-09-21T13:50:41.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:40 smithi187 ceph-mon[33032]: 6.12 scrub ok 2024-09-21T13:50:41.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:40 smithi187 ceph-mon[33032]: 3.1c scrub starts 2024-09-21T13:50:41.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:40 smithi187 ceph-mon[33032]: 3.1c scrub ok 2024-09-21T13:50:41.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:40 smithi187 ceph-mon[33032]: 5.1f scrub starts 2024-09-21T13:50:41.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:40 smithi187 ceph-mon[33032]: 5.1f scrub ok 2024-09-21T13:50:41.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:40 smithi187 ceph-mon[33032]: 6.4 scrub starts 2024-09-21T13:50:41.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:40 smithi187 ceph-mon[33032]: 6.4 scrub ok 2024-09-21T13:50:41.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:40 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3168806297' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:50:41.195 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:40 smithi049 ceph-mon[30080]: 6.17 scrub starts 2024-09-21T13:50:41.195 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:40 smithi049 ceph-mon[30080]: 6.17 scrub ok 2024-09-21T13:50:41.196 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:40 smithi049 ceph-mon[30080]: 6.12 scrub starts 2024-09-21T13:50:41.196 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:40 smithi049 ceph-mon[30080]: 6.12 scrub ok 2024-09-21T13:50:41.196 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:40 smithi049 ceph-mon[30080]: 3.1c scrub starts 2024-09-21T13:50:41.196 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:40 smithi049 ceph-mon[30080]: 3.1c scrub ok 2024-09-21T13:50:41.196 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:40 smithi049 ceph-mon[30080]: 5.1f scrub starts 2024-09-21T13:50:41.196 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:40 smithi049 ceph-mon[30080]: 5.1f scrub ok 2024-09-21T13:50:41.196 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:40 smithi049 ceph-mon[30080]: 6.4 scrub starts 2024-09-21T13:50:41.196 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:40 smithi049 ceph-mon[30080]: 6.4 scrub ok 2024-09-21T13:50:41.196 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:40 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3168806297' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:50:41.196 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:40 smithi049 ceph-mon[40489]: 6.17 scrub starts 2024-09-21T13:50:41.196 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:40 smithi049 ceph-mon[40489]: 6.17 scrub ok 2024-09-21T13:50:41.196 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:40 smithi049 ceph-mon[40489]: 6.12 scrub starts 2024-09-21T13:50:41.196 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:40 smithi049 ceph-mon[40489]: 6.12 scrub ok 2024-09-21T13:50:41.197 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:40 smithi049 ceph-mon[40489]: 3.1c scrub starts 2024-09-21T13:50:41.197 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:40 smithi049 ceph-mon[40489]: 3.1c scrub ok 2024-09-21T13:50:41.197 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:40 smithi049 ceph-mon[40489]: 5.1f scrub starts 2024-09-21T13:50:41.197 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:40 smithi049 ceph-mon[40489]: 5.1f scrub ok 2024-09-21T13:50:41.197 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:40 smithi049 ceph-mon[40489]: 6.4 scrub starts 2024-09-21T13:50:41.197 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:40 smithi049 ceph-mon[40489]: 6.4 scrub ok 2024-09-21T13:50:41.197 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:40 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3168806297' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:50:41.213 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:41.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:41.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:50:41.252 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T13:50:41.316 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:41.341 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:41.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:41.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:50:41.344 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:50:41.344 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:50:41.344 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:50:41.344 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:50:41.344 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:50:41.344 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:50:41.345 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:50:41.345 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:50:41.345 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:50:41.345 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:50:41.345 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:50:41.345 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:50:41.345 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:50:41.345 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:50:41.345 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:50:41.345 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:50:41.443 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:50:41.489 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:41.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:41.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:50:41.591 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:41.618 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:41.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:41.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:50:41.720 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:41.752 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:41.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:41.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:41.853 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:50:41.882 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:41.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:41.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:50:41.905 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 29 16 54516 54500 117.4 95.8125 0.00694646 0.00851257 2024-09-21T13:50:41.984 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:42.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:41 smithi049 ceph-mon[40489]: 5.1a scrub starts 2024-09-21T13:50:42.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:41 smithi049 ceph-mon[40489]: 5.1a scrub ok 2024-09-21T13:50:42.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:41 smithi049 ceph-mon[40489]: 5.f scrub starts 2024-09-21T13:50:42.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:41 smithi049 ceph-mon[40489]: 5.f scrub ok 2024-09-21T13:50:42.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:41 smithi049 ceph-mon[40489]: 5.b scrub starts 2024-09-21T13:50:42.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:41 smithi049 ceph-mon[40489]: 5.b scrub ok 2024-09-21T13:50:42.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:41 smithi049 ceph-mon[40489]: pgmap v65: 148 pgs: 148 active+clean; 3.0 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 784 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-09-21T13:50:42.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:41 smithi049 ceph-mon[40489]: 5.0 scrub starts 2024-09-21T13:50:42.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:41 smithi049 ceph-mon[40489]: 5.0 scrub ok 2024-09-21T13:50:42.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:41 smithi049 ceph-mon[30080]: 5.1a scrub starts 2024-09-21T13:50:42.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:41 smithi049 ceph-mon[30080]: 5.1a scrub ok 2024-09-21T13:50:42.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:41 smithi049 ceph-mon[30080]: 5.f scrub starts 2024-09-21T13:50:42.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:41 smithi049 ceph-mon[30080]: 5.f scrub ok 2024-09-21T13:50:42.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:41 smithi049 ceph-mon[30080]: 5.b scrub starts 2024-09-21T13:50:42.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:41 smithi049 ceph-mon[30080]: 5.b scrub ok 2024-09-21T13:50:42.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:41 smithi049 ceph-mon[30080]: pgmap v65: 148 pgs: 148 active+clean; 3.0 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 784 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-09-21T13:50:42.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:41 smithi049 ceph-mon[30080]: 5.0 scrub starts 2024-09-21T13:50:42.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:41 smithi049 ceph-mon[30080]: 5.0 scrub ok 2024-09-21T13:50:42.012 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:42.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:42.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:50:42.113 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:42.138 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:42.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:42.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:50:42.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:41 smithi187 ceph-mon[33032]: 5.1a scrub starts 2024-09-21T13:50:42.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:41 smithi187 ceph-mon[33032]: 5.1a scrub ok 2024-09-21T13:50:42.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:41 smithi187 ceph-mon[33032]: 5.f scrub starts 2024-09-21T13:50:42.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:41 smithi187 ceph-mon[33032]: 5.f scrub ok 2024-09-21T13:50:42.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:41 smithi187 ceph-mon[33032]: 5.b scrub starts 2024-09-21T13:50:42.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:41 smithi187 ceph-mon[33032]: 5.b scrub ok 2024-09-21T13:50:42.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:41 smithi187 ceph-mon[33032]: pgmap v65: 148 pgs: 148 active+clean; 3.0 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 784 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-09-21T13:50:42.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:41 smithi187 ceph-mon[33032]: 5.0 scrub starts 2024-09-21T13:50:42.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:41 smithi187 ceph-mon[33032]: 5.0 scrub ok 2024-09-21T13:50:42.240 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:50:42.307 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:50:42.308 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:42.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:42.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:50:42.326 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:50:42.410 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:42.437 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:42.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:42.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:42.539 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:42.570 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:42.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:42.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:42.672 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:42.699 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:42.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:42.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:42.741 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:50:42 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-2[55757]: 2024-09-21T13:50:42.461+0000 7f637077b640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T13:50:42.741 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:50:42 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-2[55757]: 2024-09-21T13:50:42.461+0000 7f637077b640 -1 osd.2 100 *** Got signal Terminated *** 2024-09-21T13:50:42.741 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:50:42 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-2[55757]: 2024-09-21T13:50:42.461+0000 7f637077b640 -1 osd.2 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T13:50:42.801 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:42.828 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:42.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:42.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:42.905 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 30 16 55950 55934 116.474 89.625 0.00519629 0.00857889 2024-09-21T13:50:42.930 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:42.950 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:42.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:42.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:43.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:42 smithi049 ceph-mon[30080]: 6.e scrub starts 2024-09-21T13:50:43.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:42 smithi049 ceph-mon[30080]: 6.e scrub ok 2024-09-21T13:50:43.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:42 smithi049 ceph-mon[30080]: 3.13 scrub starts 2024-09-21T13:50:43.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:42 smithi049 ceph-mon[30080]: 3.13 scrub ok 2024-09-21T13:50:43.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:42 smithi049 ceph-mon[30080]: 6.14 scrub starts 2024-09-21T13:50:43.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:42 smithi049 ceph-mon[30080]: 6.14 scrub ok 2024-09-21T13:50:43.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:42 smithi049 ceph-mon[30080]: osd.2 marked itself down and dead 2024-09-21T13:50:43.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:42 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:43.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:42 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:50:43.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:42 smithi049 ceph-mon[40489]: 6.e scrub starts 2024-09-21T13:50:43.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:42 smithi049 ceph-mon[40489]: 6.e scrub ok 2024-09-21T13:50:43.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:42 smithi049 ceph-mon[40489]: 3.13 scrub starts 2024-09-21T13:50:43.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:42 smithi049 ceph-mon[40489]: 3.13 scrub ok 2024-09-21T13:50:43.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:42 smithi049 ceph-mon[40489]: 6.14 scrub starts 2024-09-21T13:50:43.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:42 smithi049 ceph-mon[40489]: 6.14 scrub ok 2024-09-21T13:50:43.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:42 smithi049 ceph-mon[40489]: osd.2 marked itself down and dead 2024-09-21T13:50:43.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:42 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:43.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:42 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:50:43.052 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:50:43.074 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:43.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:43.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:50:43.176 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:43.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:42 smithi187 ceph-mon[33032]: 6.e scrub starts 2024-09-21T13:50:43.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:42 smithi187 ceph-mon[33032]: 6.e scrub ok 2024-09-21T13:50:43.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:42 smithi187 ceph-mon[33032]: 3.13 scrub starts 2024-09-21T13:50:43.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:42 smithi187 ceph-mon[33032]: 3.13 scrub ok 2024-09-21T13:50:43.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:42 smithi187 ceph-mon[33032]: 6.14 scrub starts 2024-09-21T13:50:43.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:42 smithi187 ceph-mon[33032]: 6.14 scrub ok 2024-09-21T13:50:43.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:42 smithi187 ceph-mon[33032]: osd.2 marked itself down and dead 2024-09-21T13:50:43.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:42 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:43.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:42 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:50:43.197 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:43.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:43.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:43.299 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:50:43.323 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:43.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:43.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:50:43.424 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:50:43.453 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:43.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:43.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:50:43.456 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T13:50:43.456 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:50:43.555 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:43.576 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:43.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:43.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:43.678 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:43.699 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:43.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:43.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:43.801 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:43.822 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:43.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:43.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:50:43.906 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 31 16 55978 55962 112.775 1.75 0.00419353 0.00858028 2024-09-21T13:50:43.906 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-09-21T13:50:43.924 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:43.951 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:43.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:43.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:50:44.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:43 smithi049 ceph-mon[40489]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T13:50:44.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:43 smithi049 ceph-mon[40489]: osdmap e101: 8 total, 7 up, 8 in 2024-09-21T13:50:44.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:43 smithi049 ceph-mon[40489]: pgmap v67: 148 pgs: 34 peering, 2 stale+active+clean, 112 active+clean; 3.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.5 KiB/s rd, 134 MiB/s wr, 2.14k op/s 2024-09-21T13:50:44.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:43 smithi049 ceph-mon[40489]: 5.17 scrub starts 2024-09-21T13:50:44.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:43 smithi049 ceph-mon[40489]: 5.17 scrub ok 2024-09-21T13:50:44.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:43 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/4170767682' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:50:44.005 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:50:43 smithi049 podman[80269]: 2024-09-21 13:50:43.680107517 +0000 UTC m=+1.389480051 container died 7db2c49e6a909bd73b04ae33ff37f2c06417e70c712d2b5f342c50c17ca3731d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-guits-main-2024-09-21-0842-4417849, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0) 2024-09-21T13:50:44.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:43 smithi049 ceph-mon[30080]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T13:50:44.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:43 smithi049 ceph-mon[30080]: osdmap e101: 8 total, 7 up, 8 in 2024-09-21T13:50:44.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:43 smithi049 ceph-mon[30080]: pgmap v67: 148 pgs: 34 peering, 2 stale+active+clean, 112 active+clean; 3.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.5 KiB/s rd, 134 MiB/s wr, 2.14k op/s 2024-09-21T13:50:44.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:43 smithi049 ceph-mon[30080]: 5.17 scrub starts 2024-09-21T13:50:44.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:43 smithi049 ceph-mon[30080]: 5.17 scrub ok 2024-09-21T13:50:44.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:43 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/4170767682' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:50:44.053 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:44.089 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:44.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:44.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:44.188 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T13:50:44.190 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:44.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:43 smithi187 ceph-mon[33032]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T13:50:44.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:43 smithi187 ceph-mon[33032]: osdmap e101: 8 total, 7 up, 8 in 2024-09-21T13:50:44.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:43 smithi187 ceph-mon[33032]: pgmap v67: 148 pgs: 34 peering, 2 stale+active+clean, 112 active+clean; 3.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.5 KiB/s rd, 134 MiB/s wr, 2.14k op/s 2024-09-21T13:50:44.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:43 smithi187 ceph-mon[33032]: 5.17 scrub starts 2024-09-21T13:50:44.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:43 smithi187 ceph-mon[33032]: 5.17 scrub ok 2024-09-21T13:50:44.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:43 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/4170767682' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:50:44.224 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:44.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:44.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:44.326 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:44.346 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:50:44 smithi049 podman[80269]: 2024-09-21 13:50:44.128171847 +0000 UTC m=+1.837544382 container cleanup 7db2c49e6a909bd73b04ae33ff37f2c06417e70c712d2b5f342c50c17ca3731d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.37.2, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-guits-main-2024-09-21-0842-4417849, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-21T13:50:44.347 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:50:44 smithi049 bash[80269]: ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-2 2024-09-21T13:50:44.361 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:44.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:44.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:44.462 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:50:44.514 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:44.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:44.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:50:44.616 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:50:44.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:44.688 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:44.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:50:44.744 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:50:44 smithi049 podman[80473]: 2024-09-21 13:50:44.346722757 +0000 UTC m=+0.884161517 container remove 7db2c49e6a909bd73b04ae33ff37f2c06417e70c712d2b5f342c50c17ca3731d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-2, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-21-0842-4417849) 2024-09-21T13:50:44.789 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:44.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:44.824 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:44.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:44.906 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 32 16 57705 57689 112.622 107.938 0.0107509 0.00887377 2024-09-21T13:50:44.925 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:50:44.964 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:44.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:44.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:50:45.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:44 smithi049 ceph-mon[30080]: 5.16 scrub starts 2024-09-21T13:50:45.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:44 smithi049 ceph-mon[30080]: 5.16 scrub ok 2024-09-21T13:50:45.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:44 smithi049 ceph-mon[30080]: Health check failed: Reduced data availability: 5 pgs inactive, 7 pgs peering (PG_AVAILABILITY) 2024-09-21T13:50:45.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:44 smithi049 ceph-mon[30080]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:50:45.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:44 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/4170767682' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T13:50:45.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:44 smithi049 ceph-mon[30080]: osdmap e102: 8 total, 7 up, 8 in 2024-09-21T13:50:45.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:44 smithi049 ceph-mon[40489]: 5.16 scrub starts 2024-09-21T13:50:45.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:44 smithi049 ceph-mon[40489]: 5.16 scrub ok 2024-09-21T13:50:45.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:44 smithi049 ceph-mon[40489]: Health check failed: Reduced data availability: 5 pgs inactive, 7 pgs peering (PG_AVAILABILITY) 2024-09-21T13:50:45.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:44 smithi049 ceph-mon[40489]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:50:45.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:44 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/4170767682' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T13:50:45.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:44 smithi049 ceph-mon[40489]: osdmap e102: 8 total, 7 up, 8 in 2024-09-21T13:50:45.065 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:50:45.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:45.098 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:45.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:50:45.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:44 smithi187 ceph-mon[33032]: 5.16 scrub starts 2024-09-21T13:50:45.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:44 smithi187 ceph-mon[33032]: 5.16 scrub ok 2024-09-21T13:50:45.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:44 smithi187 ceph-mon[33032]: Health check failed: Reduced data availability: 5 pgs inactive, 7 pgs peering (PG_AVAILABILITY) 2024-09-21T13:50:45.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:44 smithi187 ceph-mon[33032]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:50:45.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:44 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/4170767682' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T13:50:45.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:44 smithi187 ceph-mon[33032]: osdmap e102: 8 total, 7 up, 8 in 2024-09-21T13:50:45.199 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:50:45.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:45.234 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:45.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:50:45.335 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:50:45.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:45.375 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:45.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:50:45.476 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:50:45.477 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T13:50:45.518 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:45.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:45.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:50:45.622 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:50:45.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:45.681 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:45.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:50:45.782 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:50:45.837 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:45.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:45.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:50:45.906 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 33 16 59265 59249 112.163 97.5 0.00831157 0.00890975 2024-09-21T13:50:45.939 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:50:45.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:45.977 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:45.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:50:46.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:45 smithi049 ceph-mon[40489]: pgmap v69: 148 pgs: 1 active+clean+scrubbing, 44 peering, 103 active+clean; 3.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 682 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-09-21T13:50:46.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:45 smithi049 ceph-mon[30080]: pgmap v69: 148 pgs: 1 active+clean+scrubbing, 44 peering, 103 active+clean; 3.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 682 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-09-21T13:50:46.077 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:50:46.101 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:46.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:46.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:50:46.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:45 smithi187 ceph-mon[33032]: pgmap v69: 148 pgs: 1 active+clean+scrubbing, 44 peering, 103 active+clean; 3.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 682 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-09-21T13:50:46.203 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:46.235 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:46.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:46.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:50:46.337 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:46.367 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:46.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:46.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:50:46.469 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:46.497 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:46.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:46.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:50:46.598 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:46.628 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:46.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:46.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:46.730 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:50:46.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:46.772 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:46.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:50:46.873 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:50:46.901 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:46.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:46.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:50:46.906 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 34 16 60781 60765 111.651 94.75 0.0273385 0.00895107 2024-09-21T13:50:47.002 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:47.031 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:47.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:47.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:47.133 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:50:47.184 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:47.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:47.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:50:47.285 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:50:47.289 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:50:47.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:47.352 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:47.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:50:47.453 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:50:47.494 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:47.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:47.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:50:47.596 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:50:47.639 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:47.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:47.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:50:47.714 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:50:47.741 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:47.768 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:47.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:47.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:47.869 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:50:47.905 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:47.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:47.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:50:47.907 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 35 16 62276 62260 111.131 93.4375 0.00490714 0.00899347 2024-09-21T13:50:48.007 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:48.041 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:48.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:48.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:48.143 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:48.172 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:48.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:48.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:50:48.273 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:48.304 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:48.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:48.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:48.406 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:48.441 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:48.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:48.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:48.542 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:48.570 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:48.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:48.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:50:48.672 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:48.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:48 smithi187 ceph-mon[33032]: pgmap v70: 148 pgs: 1 active+clean+scrubbing, 44 peering, 103 active+clean; 3.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 544 B/s rd, 55 MiB/s wr, 877 op/s 2024-09-21T13:50:48.704 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:48.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:48.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:48.733 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:48 smithi049 ceph-mon[30080]: pgmap v70: 148 pgs: 1 active+clean+scrubbing, 44 peering, 103 active+clean; 3.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 544 B/s rd, 55 MiB/s wr, 877 op/s 2024-09-21T13:50:48.733 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:48 smithi049 ceph-mon[40489]: pgmap v70: 148 pgs: 1 active+clean+scrubbing, 44 peering, 103 active+clean; 3.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 544 B/s rd, 55 MiB/s wr, 877 op/s 2024-09-21T13:50:48.765 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T13:50:48.765 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:50:48.806 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:48.822 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T13:50:48.991 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 36 16 64014 63998 111.061 108.625 0.00733596 0.00899917 2024-09-21T13:50:48.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:48.992 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:48.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:49.092 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:49.122 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:49.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:49.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:50:49.224 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:49.254 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:49.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:49.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:49.356 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:50:49.406 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:49.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:49.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:50:49.433 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:50:48 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:50:49.508 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:49.539 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:49.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:49.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:49.641 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:50:49.667 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:49.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:49.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:50:49.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:49 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:50:49.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:49 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:50:49.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:49 smithi187 ceph-mon[33032]: osdmap e103: 8 total, 7 up, 8 in 2024-09-21T13:50:49.697 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:49 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:50:49.697 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:49 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:50:49.697 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:49 smithi049 ceph-mon[40489]: osdmap e103: 8 total, 7 up, 8 in 2024-09-21T13:50:49.697 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:49 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:50:49.697 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:49 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:50:49.697 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:49 smithi049 ceph-mon[30080]: osdmap e103: 8 total, 7 up, 8 in 2024-09-21T13:50:49.769 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:49.795 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:49.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:49.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:49.802 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:50:49.803 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:50:49.844 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:50:49.844 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:50:49.846 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-21T13:50:49.884 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:50:49.884 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:50:49.896 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:50:49.908 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 37 16 65520 65504 110.597 94.125 0.00593343 0.00903506 2024-09-21T13:50:49.921 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:50:49.921 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:50:49.952 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:49.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:49.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:50:49.954 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:50:49.955 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:50:49.955 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:50:49 smithi049 podman[81150]: 2024-09-21 13:50:49.748797226 +0000 UTC m=+0.578568870 image pull b415816db8b41f544cdfdc71d193ff8b1ee008bf7b0ec4f64da8eb8f71b62e85 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5 2024-09-21T13:50:49.955 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:50:49 smithi049 podman[81150]: 2024-09-21 13:50:49.899376055 +0000 UTC m=+0.729147687 container create 1257afc69f9e9c30b943eab9c07fdbe62e01f99a8c8ef16cf26d03fccb0d4256 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-2-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.build-date=20240731, ceph=True) 2024-09-21T13:50:49.989 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:50:49.990 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:50:50.027 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:50:50.027 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:50:50.054 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:50.082 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:50.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:50.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:50.183 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:50.215 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:50.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:50.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:50.317 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:50:50.359 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:50.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:50.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:50:50.461 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:50.489 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T13:50:50.494 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:50.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:50.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:50:50.596 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:50.623 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:50.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:50.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:50:50.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:50 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:50:50.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:50 smithi187 ceph-mon[33032]: pgmap v72: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 3.8 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.4 KiB/s rd, 133 MiB/s wr, 2.12k op/s; 23677/189750 objects degraded (12.478%); 30 KiB/s, 0 objects/s recovering 2024-09-21T13:50:50.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:50 smithi187 ceph-mon[33032]: Health check failed: Degraded data redundancy: 23677/189750 objects degraded (12.478%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T13:50:50.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:50 smithi187 ceph-mon[33032]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 9 pgs peering) 2024-09-21T13:50:50.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:50 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:50.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:50 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:50.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:50 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2479115456' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:50:50.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:50 smithi187 ceph-mon[33032]: osdmap e104: 8 total, 7 up, 8 in 2024-09-21T13:50:50.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:50 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:50:50.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:50 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:50:50.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:50 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:50.725 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:50:50.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:50 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:50:50.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:50 smithi049 ceph-mon[30080]: pgmap v72: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 3.8 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.4 KiB/s rd, 133 MiB/s wr, 2.12k op/s; 23677/189750 objects degraded (12.478%); 30 KiB/s, 0 objects/s recovering 2024-09-21T13:50:50.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:50 smithi049 ceph-mon[30080]: Health check failed: Degraded data redundancy: 23677/189750 objects degraded (12.478%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T13:50:50.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:50 smithi049 ceph-mon[30080]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 9 pgs peering) 2024-09-21T13:50:50.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:50 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:50.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:50 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:50.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:50 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2479115456' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:50:50.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:50 smithi049 ceph-mon[30080]: osdmap e104: 8 total, 7 up, 8 in 2024-09-21T13:50:50.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:50 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:50:50.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:50 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:50:50.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:50 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:50.756 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:50:50 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:50:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:50:50.757 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:50 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:50:50.757 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:50 smithi049 ceph-mon[40489]: pgmap v72: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 3.8 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.4 KiB/s rd, 133 MiB/s wr, 2.12k op/s; 23677/189750 objects degraded (12.478%); 30 KiB/s, 0 objects/s recovering 2024-09-21T13:50:50.757 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:50 smithi049 ceph-mon[40489]: Health check failed: Degraded data redundancy: 23677/189750 objects degraded (12.478%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T13:50:50.757 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:50 smithi049 ceph-mon[40489]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 9 pgs peering) 2024-09-21T13:50:50.757 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:50 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:50.757 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:50 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:50.757 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:50 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2479115456' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:50:50.757 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:50 smithi049 ceph-mon[40489]: osdmap e104: 8 total, 7 up, 8 in 2024-09-21T13:50:50.757 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:50 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:50:50.757 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:50 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:50:50.757 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:50 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:50.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:50.778 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:50.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:50:50.879 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:50:50.909 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 38 16 66951 66935 110.04 89.4375 0.00639019 0.0090824 2024-09-21T13:50:50.915 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:50.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:50.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:50:51.017 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:51.048 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:50:51.048 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:51.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:51.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:50:51.150 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:50:51.182 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:50:50 smithi049 podman[81150]: 2024-09-21 13:50:50.91899193 +0000 UTC m=+1.748763573 container init 1257afc69f9e9c30b943eab9c07fdbe62e01f99a8c8ef16cf26d03fccb0d4256 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-2-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-21-0842-4417849, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731) 2024-09-21T13:50:51.182 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:50:50 smithi049 podman[81150]: 2024-09-21 13:50:50.930380229 +0000 UTC m=+1.760151860 container start 1257afc69f9e9c30b943eab9c07fdbe62e01f99a8c8ef16cf26d03fccb0d4256 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-2-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-21-0842-4417849, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-09-21T13:50:51.182 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:50:51 smithi049 podman[81150]: 2024-09-21 13:50:51.035205117 +0000 UTC m=+1.864976745 container attach 1257afc69f9e9c30b943eab9c07fdbe62e01f99a8c8ef16cf26d03fccb0d4256 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-2-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T13:50:51.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:51.198 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:51.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:50:51.299 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:50:51.337 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:51.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:51.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:50:51.439 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:51.443 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:51 smithi049 ceph-mon[30080]: pgmap v74: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 3.8 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 817 B/s rd, 71 MiB/s wr, 1.13k op/s; 23677/189750 objects degraded (12.478%); 31 KiB/s, 0 objects/s recovering 2024-09-21T13:50:51.443 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:51 smithi049 ceph-mon[30080]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-21T13:50:51.443 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:51 smithi049 ceph-mon[30080]: Reconfiguring daemon alertmanager.a on smithi049 2024-09-21T13:50:51.443 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:51 smithi049 ceph-mon[30080]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-21T13:50:51.473 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:51.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:51.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:51.575 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:50:51.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:51.610 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:51.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:50:51.656 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T13:50:51.657 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T13:50:51.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:51 smithi187 ceph-mon[33032]: pgmap v74: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 3.8 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 817 B/s rd, 71 MiB/s wr, 1.13k op/s; 23677/189750 objects degraded (12.478%); 31 KiB/s, 0 objects/s recovering 2024-09-21T13:50:51.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:51 smithi187 ceph-mon[33032]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-21T13:50:51.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:51 smithi187 ceph-mon[33032]: Reconfiguring daemon alertmanager.a on smithi049 2024-09-21T13:50:51.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:51 smithi187 ceph-mon[33032]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-21T13:50:51.712 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:50:51.716 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:51 smithi049 ceph-mon[40489]: pgmap v74: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 3.8 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 817 B/s rd, 71 MiB/s wr, 1.13k op/s; 23677/189750 objects degraded (12.478%); 31 KiB/s, 0 objects/s recovering 2024-09-21T13:50:51.716 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:51 smithi049 ceph-mon[40489]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-21T13:50:51.716 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:51 smithi049 ceph-mon[40489]: Reconfiguring daemon alertmanager.a on smithi049 2024-09-21T13:50:51.716 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:51 smithi049 ceph-mon[40489]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-21T13:50:51.747 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:51.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:51.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:50:51.848 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:51.881 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:51.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:51.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:50:51.908 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 39 16 68593 68577 109.85 102.625 0.00845982 0.00909839 2024-09-21T13:50:51.983 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:52.016 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:52.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:52.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:52.117 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:52.149 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:52.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:52.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:52.251 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:50:52.294 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:52.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:52.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:50:52.395 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:50:52.430 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:52.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:52.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:50:52.533 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:50:52.567 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:52.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:52.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:50:52.669 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:50:52.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:52.708 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:52.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:50:52.809 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:50:52.849 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:52.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:52.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:50:52.909 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:2024-09-21T13:50:52.909505+0000 min lat: 0.00133054 max lat: 1.05128 avg lat: 0.00911029 2024-09-21T13:50:52.909 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T13:50:52.909 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 40 16 70262 70246 109.711 104.312 0.00653418 0.00911029 2024-09-21T13:50:52.951 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:52.982 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:52.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:52.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:53.084 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:50:53.113 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:53.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:53.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:50:53.214 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:50:53.241 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:53.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:53.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:50:53.343 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:50:53.380 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:53.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:53.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:50:53.482 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:53.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:53 smithi049 ceph-mon[30080]: pgmap v75: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 3.8 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 750 B/s rd, 65 MiB/s wr, 1.04k op/s; 23677/189750 objects degraded (12.478%); 28 KiB/s, 0 objects/s recovering 2024-09-21T13:50:53.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:53 smithi049 ceph-mon[40489]: pgmap v75: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 3.8 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 750 B/s rd, 65 MiB/s wr, 1.04k op/s; 23677/189750 objects degraded (12.478%); 28 KiB/s, 0 objects/s recovering 2024-09-21T13:50:53.514 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:53.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:53.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:53.616 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:53.651 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:53.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:53.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:53.658 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T13:50:53.752 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:53.782 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:53.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:53.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:53.884 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:53.910 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 41 16 71755 71739 109.311 93.3125 0.0115365 0.00914266 2024-09-21T13:50:53.912 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:53.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:53.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:53.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:53 smithi187 ceph-mon[33032]: pgmap v75: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 3.8 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 750 B/s rd, 65 MiB/s wr, 1.04k op/s; 23677/189750 objects degraded (12.478%); 28 KiB/s, 0 objects/s recovering 2024-09-21T13:50:54.005 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:50:53 smithi049 podman[81150]: 2024-09-21 13:50:53.6871447 +0000 UTC m=+4.516916348 container died 1257afc69f9e9c30b943eab9c07fdbe62e01f99a8c8ef16cf26d03fccb0d4256 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-2-deactivate, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T13:50:54.014 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:54.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:54.048 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:54.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:54.149 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:54.181 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:54.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:54.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:54.284 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:50:54.315 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:54.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:54.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:50:54.417 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:50:54.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:54.456 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:54.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:50:54.467 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-09-21T13:50:54.556 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:50:54.585 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:54.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:54.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:50:54.687 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:50:54.719 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:54.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:54.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:50:54.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:54 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1690509323' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:50:54.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:54 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:50:54.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:54 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1690509323' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:50:54.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:54 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:50:54.820 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:54.855 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:54.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:54.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:50:54.909 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 42 16 73192 73176 108.847 89.8125 0.00333498 0.00918132 2024-09-21T13:50:54.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:54 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1690509323' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:50:54.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:54 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:50:54.957 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:54.986 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:54.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:54.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:55.088 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:50:55.130 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:55.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:55.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:50:55.231 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:55.266 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:55.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:55.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:55.367 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:55.400 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:55.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:55.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:55.502 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:50:55.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:55 smithi049 ceph-mon[40489]: pgmap v76: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.5 KiB/s rd, 137 MiB/s wr, 2.19k op/s; 26638/213318 objects degraded (12.487%); 28 KiB/s, 0 objects/s recovering 2024-09-21T13:50:55.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:55 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 26638/213318 objects degraded (12.487%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T13:50:55.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:55 smithi049 ceph-mon[40489]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:50:55.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:55 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T13:50:55.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:55 smithi049 ceph-mon[40489]: osdmap e105: 8 total, 7 up, 8 in 2024-09-21T13:50:55.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:55 smithi049 ceph-mon[40489]: osdmap e106: 8 total, 7 up, 8 in 2024-09-21T13:50:55.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:55 smithi049 ceph-mon[30080]: pgmap v76: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.5 KiB/s rd, 137 MiB/s wr, 2.19k op/s; 26638/213318 objects degraded (12.487%); 28 KiB/s, 0 objects/s recovering 2024-09-21T13:50:55.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:55 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 26638/213318 objects degraded (12.487%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T13:50:55.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:55 smithi049 ceph-mon[30080]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:50:55.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:55 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T13:50:55.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:55 smithi049 ceph-mon[30080]: osdmap e105: 8 total, 7 up, 8 in 2024-09-21T13:50:55.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:55 smithi049 ceph-mon[30080]: osdmap e106: 8 total, 7 up, 8 in 2024-09-21T13:50:55.539 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:55.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:55.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:50:55.641 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:55.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:55.674 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:55.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:50:55.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:55 smithi187 ceph-mon[33032]: pgmap v76: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.5 KiB/s rd, 137 MiB/s wr, 2.19k op/s; 26638/213318 objects degraded (12.487%); 28 KiB/s, 0 objects/s recovering 2024-09-21T13:50:55.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:55 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 26638/213318 objects degraded (12.487%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T13:50:55.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:55 smithi187 ceph-mon[33032]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:50:55.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:55 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T13:50:55.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:55 smithi187 ceph-mon[33032]: osdmap e105: 8 total, 7 up, 8 in 2024-09-21T13:50:55.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:55 smithi187 ceph-mon[33032]: osdmap e106: 8 total, 7 up, 8 in 2024-09-21T13:50:55.776 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:55.807 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:55.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:55.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:50:55.909 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:50:55.910 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 43 16 74776 74760 108.618 99 0.0156674 0.0092021 2024-09-21T13:50:55.937 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:55.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:55.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:50:56.039 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:56.073 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:56.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:56.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:56.175 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:56.206 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:56.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:56.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:56.308 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:50:56.349 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:56.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:56.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:50:56.451 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:50:56.487 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:56.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:56.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:50:56.589 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:56.623 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:56.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:56.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:56.725 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:50:56.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:56 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:50:56.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:56 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:50:56.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:56 smithi049 ceph-mon[40489]: osdmap e107: 8 total, 7 up, 8 in 2024-09-21T13:50:56.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:56 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:50:56.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:56 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:50:56.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:56 smithi049 ceph-mon[30080]: osdmap e107: 8 total, 7 up, 8 in 2024-09-21T13:50:56.763 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:56.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:56.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:50:56.865 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:50:56.898 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:56.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:56.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:50:56.909 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 44 15 76280 76265 108.287 94.0625 0.0186996 0.00922927 2024-09-21T13:50:56.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:56 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:50:56.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:56 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:50:56.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:56 smithi187 ceph-mon[33032]: osdmap e107: 8 total, 7 up, 8 in 2024-09-21T13:50:57.000 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:50:57.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:57.040 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:57.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:50:57.138 DEBUG:teuthology.orchestra.run.smithi049:> sudo pkill -f 'journalctl -f -n 0 -u ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@osd.2.service' 2024-09-21T13:50:57.142 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:57.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:57.173 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:57.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:57.244 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T13:50:57.244 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-21T13:50:57.244 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.2 2024-09-21T13:50:57.244 INFO:tasks.thrashosds.thrasher:remote for osd 2 is ubuntu@smithi049.front.sepia.ceph.com 2024-09-21T13:50:57.244 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell -n osd.2 --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --op list-pgs 2024-09-21T13:50:57.274 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:57.303 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:57.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:57.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:57.404 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:50:57.470 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:57.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:57.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:50:57.572 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:50:57.600 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:57.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:57.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:50:57.641 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/osd.2/config 2024-09-21T13:50:57.702 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:50:57.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:57.744 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:57.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:50:57.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:57 smithi049 ceph-mon[40489]: pgmap v80: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 82 MiB/s wr, 1.31k op/s; 26638/213318 objects degraded (12.487%) 2024-09-21T13:50:57.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:57 smithi049 ceph-mon[30080]: pgmap v80: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 82 MiB/s wr, 1.31k op/s; 26638/213318 objects degraded (12.487%) 2024-09-21T13:50:57.845 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:50:57.894 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:57.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:57.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:50:57.909 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 45 16 77702 77686 107.854 88.8125 0.0120335 0.00926663 2024-09-21T13:50:57.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:57 smithi187 ceph-mon[33032]: pgmap v80: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 82 MiB/s wr, 1.31k op/s; 26638/213318 objects degraded (12.487%) 2024-09-21T13:50:57.997 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:50:58.035 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:58.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:58.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:50:58.137 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:50:58.174 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:58.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:58.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:50:58.276 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:50:58.323 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:58.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:58.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:50:58.425 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:58.448 INFO:journalctl@ceph.alertmanager.a.smithi049.stdout:Sep 21 13:50:58 smithi049 systemd[1]: Stopping Ceph alertmanager.a for 40a3d2e8-781f-11ef-baf6-efdc52797490... 2024-09-21T13:50:58.460 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:58.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:58.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:50:58.562 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:50:58.612 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:58.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:58.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:50:58.714 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:50:58.744 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:58.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:58.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:50:58.846 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:58.881 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:58.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:58.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:58.909 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 46 16 79198 79182 107.542 93.5 0.00650229 0.00929402 2024-09-21T13:50:58.982 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:59.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:58 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:59.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:58 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:50:59.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:58 smithi049 ceph-mon[40489]: pgmap v81: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 82 MiB/s wr, 1.31k op/s; 26638/213318 objects degraded (12.487%) 2024-09-21T13:50:59.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:58 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:59.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:58 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:50:59.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:58 smithi049 ceph-mon[30080]: pgmap v81: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 82 MiB/s wr, 1.31k op/s; 26638/213318 objects degraded (12.487%) 2024-09-21T13:50:59.014 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:59.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:59.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:59.090 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:50:59.116 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:59.149 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:59.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:59.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:50:59.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:58 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:50:59.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:58 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:50:59.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:58 smithi187 ceph-mon[33032]: pgmap v81: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 82 MiB/s wr, 1.31k op/s; 26638/213318 objects degraded (12.487%) 2024-09-21T13:50:59.195 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:50:59 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:50:59.252 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:50:59.295 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:59.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:59.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:50:59.396 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:50:59.427 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:59.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:59.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:50:59.529 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:50:59.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:59.572 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:59.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:50:59.673 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:50:59.726 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:50:59.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:59.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:50:59.828 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:59.860 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:59.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:59.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:50:59.909 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 47 16 80668 80652 107.208 91.875 0.0132178 0.00932218 2024-09-21T13:50:59.961 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:50:59.992 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:50:59.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:50:59.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:00.056 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:59 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:51:00.056 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:50:59 smithi049 ceph-mon[40489]: osdmap e108: 8 total, 7 up, 8 in 2024-09-21T13:51:00.057 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:59 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:51:00.057 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:50:59 smithi049 ceph-mon[30080]: osdmap e108: 8 total, 7 up, 8 in 2024-09-21T13:51:00.094 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:51:00.130 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:00.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:00.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:51:00.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:59 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:51:00.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:50:59 smithi187 ceph-mon[33032]: osdmap e108: 8 total, 7 up, 8 in 2024-09-21T13:51:00.232 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:51:00.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:00.275 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:00.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:51:00.346 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T13:51:00.346 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-09-21T13:51:00.346 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-09-21T13:51:00.347 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-09-21T13:51:00.347 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.347 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:35220 7.f 7:f1d65728:::benchmark_data_smithi049_76798_object35219:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T13:51:00.347 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:28.689446+0000", 2024-09-21T13:51:00.347 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 31.653392348000001, 2024-09-21T13:51:00.347 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0014680600000001, 2024-09-21T13:51:00.347 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:51:00.347 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:51:00.347 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T13:51:00.347 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:51:00.347 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:51:00.347 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:00.347 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 35220 2024-09-21T13:51:00.347 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.348 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:51:00.348 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.348 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:51:00.348 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.689446+0000", 2024-09-21T13:51:00.348 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:00.348 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.348 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.348 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:51:00.348 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.689446+0000", 2024-09-21T13:51:00.348 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:00.348 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.348 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.348 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:51:00.348 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.689447+0000", 2024-09-21T13:51:00.348 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9500000000000002e-07 2024-09-21T13:51:00.349 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.349 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.349 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:51:00.349 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.689473+0000", 2024-09-21T13:51:00.349 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5661e-05 2024-09-21T13:51:00.349 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.349 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.349 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:51:00.349 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.689481+0000", 2024-09-21T13:51:00.349 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2009999999999999e-06 2024-09-21T13:51:00.349 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.349 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.349 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:51:00.349 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.689487+0000", 2024-09-21T13:51:00.350 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3659999999999997e-06 2024-09-21T13:51:00.350 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.350 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.350 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:51:00.350 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.689704+0000", 2024-09-21T13:51:00.350 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00021680000000000001 2024-09-21T13:51:00.350 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.350 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.350 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:51:00.350 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.689714+0000", 2024-09-21T13:51:00.350 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0481e-05 2024-09-21T13:51:00.350 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.350 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.350 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:51:00.350 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.690368+0000", 2024-09-21T13:51:00.351 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.00065395 2024-09-21T13:51:00.351 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.351 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.351 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:51:00.351 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.690914+0000", 2024-09-21T13:51:00.351 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00054600600000000003 2024-09-21T13:51:00.351 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:00.351 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:51:00.351 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:00.351 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.351 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.351 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:35261 7.f 7:fafb65cb:::benchmark_data_smithi049_76798_object35260:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T13:51:00.351 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:28.705982+0000", 2024-09-21T13:51:00.352 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 31.636856358999999, 2024-09-21T13:51:00.352 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.99118446199999999, 2024-09-21T13:51:00.352 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:51:00.352 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:51:00.352 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:00.352 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:51:00.352 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:51:00.352 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:00.352 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 35261 2024-09-21T13:51:00.352 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.352 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:51:00.352 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.352 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:51:00.352 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.705982+0000", 2024-09-21T13:51:00.353 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:00.353 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.353 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.353 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:51:00.353 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.705982+0000", 2024-09-21T13:51:00.353 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:00.353 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.353 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.353 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:51:00.353 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.705983+0000", 2024-09-21T13:51:00.353 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7400000000000003e-07 2024-09-21T13:51:00.353 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.353 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.353 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:51:00.353 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.706009+0000", 2024-09-21T13:51:00.354 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5777000000000001e-05 2024-09-21T13:51:00.354 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.354 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.354 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:51:00.354 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.706036+0000", 2024-09-21T13:51:00.354 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.7529e-05 2024-09-21T13:51:00.354 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.354 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.354 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:51:00.354 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.706043+0000", 2024-09-21T13:51:00.354 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2280000000000001e-06 2024-09-21T13:51:00.354 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.354 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.354 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:51:00.355 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.706108+0000", 2024-09-21T13:51:00.355 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4133000000000004e-05 2024-09-21T13:51:00.355 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.355 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.355 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:51:00.355 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.706119+0000", 2024-09-21T13:51:00.355 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1105e-05 2024-09-21T13:51:00.355 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.355 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.355 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:51:00.355 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.691847+0000", 2024-09-21T13:51:00.355 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.98572784199999997 2024-09-21T13:51:00.355 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.355 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.355 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:51:00.355 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.692093+0000", 2024-09-21T13:51:00.356 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000246405 2024-09-21T13:51:00.356 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.356 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.356 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,7", 2024-09-21T13:51:00.356 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.692178+0000", 2024-09-21T13:51:00.356 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4666999999999994e-05 2024-09-21T13:51:00.356 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.356 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.356 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:51:00.356 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.696555+0000", 2024-09-21T13:51:00.356 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0043774039999999997 2024-09-21T13:51:00.356 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.356 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.356 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:00.357 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.696599+0000", 2024-09-21T13:51:00.357 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4319e-05 2024-09-21T13:51:00.357 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.357 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.357 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:00.357 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.697065+0000", 2024-09-21T13:51:00.357 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00046557800000000002 2024-09-21T13:51:00.357 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.357 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.357 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:51:00.357 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.697124+0000", 2024-09-21T13:51:00.357 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9250999999999999e-05 2024-09-21T13:51:00.357 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.358 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.358 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:51:00.358 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.697167+0000", 2024-09-21T13:51:00.358 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.265e-05 2024-09-21T13:51:00.358 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:00.358 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:51:00.358 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:00.358 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.358 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.358 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:35268 7.f 7:f0677f6b:::benchmark_data_smithi049_76798_object35267:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T13:51:00.358 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:28.710368+0000", 2024-09-21T13:51:00.358 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 31.632470571999999, 2024-09-21T13:51:00.359 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.98801466100000002, 2024-09-21T13:51:00.359 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:51:00.359 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:51:00.359 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:00.359 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:51:00.359 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:51:00.359 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:00.359 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 35268 2024-09-21T13:51:00.359 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.359 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:51:00.359 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.359 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:51:00.359 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.710368+0000", 2024-09-21T13:51:00.359 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:00.359 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.360 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.360 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:51:00.360 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.710368+0000", 2024-09-21T13:51:00.360 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:00.360 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.360 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.360 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:51:00.360 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.710369+0000", 2024-09-21T13:51:00.360 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3300000000000001e-07 2024-09-21T13:51:00.360 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.360 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.360 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:51:00.360 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.710395+0000", 2024-09-21T13:51:00.360 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5939e-05 2024-09-21T13:51:00.361 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.361 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.361 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:51:00.361 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.710406+0000", 2024-09-21T13:51:00.361 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0733999999999999e-05 2024-09-21T13:51:00.361 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.361 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.361 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:51:00.361 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.710456+0000", 2024-09-21T13:51:00.361 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0151000000000001e-05 2024-09-21T13:51:00.361 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.361 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.361 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:51:00.361 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.710532+0000", 2024-09-21T13:51:00.362 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6718000000000006e-05 2024-09-21T13:51:00.362 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.362 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.362 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:51:00.362 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.710542+0000", 2024-09-21T13:51:00.362 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.5279999999999992e-06 2024-09-21T13:51:00.362 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.362 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.362 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:51:00.362 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.693647+0000", 2024-09-21T13:51:00.362 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.98310485999999997 2024-09-21T13:51:00.362 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.362 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.362 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:51:00.363 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.693760+0000", 2024-09-21T13:51:00.363 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011317 2024-09-21T13:51:00.363 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.363 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.363 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,7", 2024-09-21T13:51:00.363 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.693901+0000", 2024-09-21T13:51:00.363 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014079300000000001 2024-09-21T13:51:00.363 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.363 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.363 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:51:00.363 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.696562+0000", 2024-09-21T13:51:00.363 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0026607789999999998 2024-09-21T13:51:00.363 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.363 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.363 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:00.364 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.696868+0000", 2024-09-21T13:51:00.364 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00030615599999999997 2024-09-21T13:51:00.364 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.364 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.364 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:00.364 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.698285+0000", 2024-09-21T13:51:00.364 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001417422 2024-09-21T13:51:00.364 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.364 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.364 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:51:00.364 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.698342+0000", 2024-09-21T13:51:00.364 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7130999999999999e-05 2024-09-21T13:51:00.364 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.364 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.364 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:51:00.365 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.698383+0000", 2024-09-21T13:51:00.365 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0547000000000003e-05 2024-09-21T13:51:00.365 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:00.365 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:51:00.365 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:00.365 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.365 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.365 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:35289 7.f 7:fb842f7b:::benchmark_data_smithi049_76798_object35288:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T13:51:00.365 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:28.735051+0000", 2024-09-21T13:51:00.365 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 31.607787941000002, 2024-09-21T13:51:00.365 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.96353024399999998, 2024-09-21T13:51:00.365 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:51:00.365 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:51:00.365 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:00.365 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:51:00.366 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:51:00.366 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:00.366 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 35289 2024-09-21T13:51:00.366 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.366 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:51:00.366 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.366 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:51:00.366 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.735051+0000", 2024-09-21T13:51:00.366 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:00.366 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.366 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.366 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:51:00.366 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.735051+0000", 2024-09-21T13:51:00.366 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:00.367 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.367 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.367 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:51:00.367 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.735052+0000", 2024-09-21T13:51:00.367 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.043e-06 2024-09-21T13:51:00.367 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.367 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.367 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:51:00.367 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.735070+0000", 2024-09-21T13:51:00.367 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7926000000000001e-05 2024-09-21T13:51:00.367 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.367 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.367 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:51:00.367 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.735080+0000", 2024-09-21T13:51:00.367 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0375e-05 2024-09-21T13:51:00.368 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.368 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.368 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:51:00.368 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.735087+0000", 2024-09-21T13:51:00.368 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3540000000000002e-06 2024-09-21T13:51:00.368 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.368 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.368 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:51:00.368 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.735144+0000", 2024-09-21T13:51:00.368 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6777999999999997e-05 2024-09-21T13:51:00.368 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.368 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.368 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:51:00.368 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.735152+0000", 2024-09-21T13:51:00.368 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6340000000000001e-06 2024-09-21T13:51:00.369 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.369 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.369 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:51:00.369 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.695195+0000", 2024-09-21T13:51:00.369 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.96004361299999996 2024-09-21T13:51:00.369 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.369 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.369 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:51:00.369 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.695290+0000", 2024-09-21T13:51:00.369 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4399000000000002e-05 2024-09-21T13:51:00.369 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.369 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.369 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,7", 2024-09-21T13:51:00.369 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.695411+0000", 2024-09-21T13:51:00.370 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012121900000000001 2024-09-21T13:51:00.370 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.370 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.370 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:51:00.370 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.696894+0000", 2024-09-21T13:51:00.370 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0014829800000000001 2024-09-21T13:51:00.370 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.370 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.370 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:00.370 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.698132+0000", 2024-09-21T13:51:00.370 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001237897 2024-09-21T13:51:00.370 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.370 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.370 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:00.370 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.698487+0000", 2024-09-21T13:51:00.371 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00035546299999999998 2024-09-21T13:51:00.371 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.371 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.371 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:51:00.371 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.698545+0000", 2024-09-21T13:51:00.371 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7883000000000001e-05 2024-09-21T13:51:00.371 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.371 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.371 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:51:00.371 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.698581+0000", 2024-09-21T13:51:00.371 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5679999999999997e-05 2024-09-21T13:51:00.371 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:00.371 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:51:00.371 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:00.371 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.372 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.372 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:36417 7.3 7:c65d1f2d:::benchmark_data_smithi049_76798_object36416:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-21T13:51:00.372 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:30.286264+0000", 2024-09-21T13:51:00.372 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 30.056574567999998, 2024-09-21T13:51:00.372 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.086645097000000004, 2024-09-21T13:51:00.372 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:51:00.372 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:51:00.372 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:00.372 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:51:00.372 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:51:00.372 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:00.372 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 36417 2024-09-21T13:51:00.372 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.373 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:51:00.373 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.373 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:51:00.373 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.286264+0000", 2024-09-21T13:51:00.373 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:00.373 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.373 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.373 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:51:00.373 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.286264+0000", 2024-09-21T13:51:00.374 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:00.374 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.374 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.374 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:51:00.374 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.286265+0000", 2024-09-21T13:51:00.374 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.24e-06 2024-09-21T13:51:00.374 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.374 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.374 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:51:00.374 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.286289+0000", 2024-09-21T13:51:00.374 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4131e-05 2024-09-21T13:51:00.374 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.374 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.374 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:51:00.375 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.286304+0000", 2024-09-21T13:51:00.375 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4479e-05 2024-09-21T13:51:00.375 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.375 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.375 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:51:00.375 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.286311+0000", 2024-09-21T13:51:00.375 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0940000000000002e-06 2024-09-21T13:51:00.375 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.375 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.375 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:51:00.375 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.286391+0000", 2024-09-21T13:51:00.375 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0210000000000006e-05 2024-09-21T13:51:00.375 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.375 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.375 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:51:00.376 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.286521+0000", 2024-09-21T13:51:00.376 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012974899999999999 2024-09-21T13:51:00.376 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.376 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.376 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,4", 2024-09-21T13:51:00.376 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.286643+0000", 2024-09-21T13:51:00.376 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000121698 2024-09-21T13:51:00.376 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.376 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.376 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:00.376 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.288737+0000", 2024-09-21T13:51:00.376 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002093925 2024-09-21T13:51:00.376 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.376 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.376 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:51:00.377 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.290162+0000", 2024-09-21T13:51:00.377 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001424993 2024-09-21T13:51:00.377 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.377 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.377 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:00.377 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.372809+0000", 2024-09-21T13:51:00.377 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.082647709999999999 2024-09-21T13:51:00.377 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.377 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.377 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:51:00.377 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.372859+0000", 2024-09-21T13:51:00.377 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9772999999999999e-05 2024-09-21T13:51:00.377 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.377 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.378 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:51:00.378 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.372909+0000", 2024-09-21T13:51:00.378 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0095000000000001e-05 2024-09-21T13:51:00.378 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:00.378 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:51:00.378 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:00.378 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.378 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.378 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:49348 7.f 7:fcf6cd05:::benchmark_data_smithi049_76798_object49347:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T13:51:00.378 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:38.246592+0000", 2024-09-21T13:51:00.378 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 22.096246333, 2024-09-21T13:51:00.378 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.16280218699999999, 2024-09-21T13:51:00.378 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:51:00.378 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:51:00.379 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:00.379 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:51:00.379 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:51:00.379 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:00.379 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 49348 2024-09-21T13:51:00.379 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.379 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:51:00.379 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.379 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:51:00.379 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.246592+0000", 2024-09-21T13:51:00.379 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:00.379 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.379 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.379 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:51:00.379 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.246592+0000", 2024-09-21T13:51:00.380 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:00.380 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.380 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.380 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:51:00.380 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.246594+0000", 2024-09-21T13:51:00.380 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9439999999999999e-06 2024-09-21T13:51:00.380 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.380 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.380 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:51:00.380 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.246628+0000", 2024-09-21T13:51:00.380 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3380000000000002e-05 2024-09-21T13:51:00.380 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.380 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.380 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:51:00.380 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.246645+0000", 2024-09-21T13:51:00.381 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7606999999999999e-05 2024-09-21T13:51:00.381 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.381 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.381 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:51:00.381 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.246653+0000", 2024-09-21T13:51:00.381 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5020000000000003e-06 2024-09-21T13:51:00.381 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.381 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.381 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:51:00.381 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.246704+0000", 2024-09-21T13:51:00.381 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0729999999999997e-05 2024-09-21T13:51:00.381 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.381 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.381 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:51:00.381 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.246890+0000", 2024-09-21T13:51:00.382 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00018683600000000001 2024-09-21T13:51:00.382 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.382 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.382 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,7", 2024-09-21T13:51:00.382 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.250341+0000", 2024-09-21T13:51:00.382 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0034508149999999999 2024-09-21T13:51:00.382 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.382 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.382 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:51:00.382 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.252556+0000", 2024-09-21T13:51:00.382 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0022145440000000001 2024-09-21T13:51:00.382 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.382 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.382 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:00.382 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.253106+0000", 2024-09-21T13:51:00.383 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00055010900000000001 2024-09-21T13:51:00.383 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.383 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.383 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:00.383 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.409254+0000", 2024-09-21T13:51:00.383 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.15614781599999999 2024-09-21T13:51:00.383 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.383 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.383 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:51:00.383 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.409313+0000", 2024-09-21T13:51:00.383 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8888999999999998e-05 2024-09-21T13:51:00.383 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.383 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.383 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:51:00.383 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.409395+0000", 2024-09-21T13:51:00.383 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2015000000000007e-05 2024-09-21T13:51:00.384 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:00.384 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:51:00.384 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:00.384 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.384 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.384 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:49355 7.f 7:f1158586:::benchmark_data_smithi049_76798_object49354:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T13:51:00.384 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:38.254426+0000", 2024-09-21T13:51:00.384 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 22.088412466000001, 2024-09-21T13:51:00.384 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.159277998, 2024-09-21T13:51:00.384 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:51:00.384 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:51:00.384 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:00.384 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:51:00.384 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:51:00.384 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:00.385 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 49355 2024-09-21T13:51:00.385 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.385 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:51:00.385 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.385 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:51:00.385 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.254426+0000", 2024-09-21T13:51:00.385 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:00.385 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.385 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.385 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:51:00.385 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.254426+0000", 2024-09-21T13:51:00.385 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:00.385 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.385 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.385 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:51:00.386 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.254428+0000", 2024-09-21T13:51:00.386 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.362e-06 2024-09-21T13:51:00.386 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.386 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.386 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:51:00.386 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.254456+0000", 2024-09-21T13:51:00.386 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.8883000000000001e-05 2024-09-21T13:51:00.386 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.386 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.386 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:51:00.386 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.254468+0000", 2024-09-21T13:51:00.386 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1695999999999999e-05 2024-09-21T13:51:00.386 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.386 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.387 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:51:00.387 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.254476+0000", 2024-09-21T13:51:00.387 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8820000000000001e-06 2024-09-21T13:51:00.387 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.387 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.387 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:51:00.387 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.254563+0000", 2024-09-21T13:51:00.387 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6483999999999994e-05 2024-09-21T13:51:00.387 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.387 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.387 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:51:00.387 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.254683+0000", 2024-09-21T13:51:00.387 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012024899999999999 2024-09-21T13:51:00.387 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.387 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.388 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,7", 2024-09-21T13:51:00.388 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.254768+0000", 2024-09-21T13:51:00.388 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4840999999999996e-05 2024-09-21T13:51:00.388 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.388 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.388 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:00.388 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257692+0000", 2024-09-21T13:51:00.388 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0029238929999999999 2024-09-21T13:51:00.388 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.388 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.388 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:51:00.388 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.259575+0000", 2024-09-21T13:51:00.388 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00188386 2024-09-21T13:51:00.388 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.388 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.389 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:00.389 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.413590+0000", 2024-09-21T13:51:00.389 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.15401436800000001 2024-09-21T13:51:00.389 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.389 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.389 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:51:00.389 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.413652+0000", 2024-09-21T13:51:00.389 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2428000000000006e-05 2024-09-21T13:51:00.389 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.389 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.389 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:51:00.389 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.413704+0000", 2024-09-21T13:51:00.389 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2052000000000002e-05 2024-09-21T13:51:00.389 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:00.389 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:51:00.390 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:00.390 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.390 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.390 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:49357 7.f 7:fffc650e:::benchmark_data_smithi049_76798_object49356:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T13:51:00.390 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:38.254508+0000", 2024-09-21T13:51:00.390 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 22.088330232000001, 2024-09-21T13:51:00.390 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.166569883, 2024-09-21T13:51:00.390 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:51:00.390 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:51:00.390 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:00.390 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:51:00.390 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:51:00.390 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:00.390 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 49357 2024-09-21T13:51:00.390 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.391 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:51:00.391 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.391 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:51:00.391 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.254508+0000", 2024-09-21T13:51:00.391 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:00.391 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.391 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.391 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:51:00.391 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.254508+0000", 2024-09-21T13:51:00.391 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:00.391 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.391 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.391 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:51:00.391 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.254509+0000", 2024-09-21T13:51:00.391 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1399999999999997e-07 2024-09-21T13:51:00.392 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.392 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.392 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:51:00.392 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.254532+0000", 2024-09-21T13:51:00.392 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3070000000000001e-05 2024-09-21T13:51:00.392 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.392 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.392 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:51:00.392 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.254542+0000", 2024-09-21T13:51:00.392 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.5659999999999996e-06 2024-09-21T13:51:00.392 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.392 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.392 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:51:00.392 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.254548+0000", 2024-09-21T13:51:00.392 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7750000000000004e-06 2024-09-21T13:51:00.393 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.393 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.393 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:51:00.393 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257723+0000", 2024-09-21T13:51:00.393 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0031742179999999999 2024-09-21T13:51:00.393 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.393 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.393 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:51:00.393 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257836+0000", 2024-09-21T13:51:00.393 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000113806 2024-09-21T13:51:00.393 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.393 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.393 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,7", 2024-09-21T13:51:00.393 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257974+0000", 2024-09-21T13:51:00.393 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013741900000000001 2024-09-21T13:51:00.394 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.394 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.394 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:51:00.394 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.259870+0000", 2024-09-21T13:51:00.394 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001895618 2024-09-21T13:51:00.394 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.394 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.394 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:00.394 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.260501+0000", 2024-09-21T13:51:00.394 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00063127599999999995 2024-09-21T13:51:00.394 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.394 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.394 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:00.394 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.420936+0000", 2024-09-21T13:51:00.394 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.16043501600000001 2024-09-21T13:51:00.395 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.395 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.395 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:51:00.395 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.421012+0000", 2024-09-21T13:51:00.395 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5928999999999998e-05 2024-09-21T13:51:00.395 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.395 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.395 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:51:00.395 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.421078+0000", 2024-09-21T13:51:00.395 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6575999999999994e-05 2024-09-21T13:51:00.395 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:00.395 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:51:00.395 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:00.395 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.395 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.396 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:49379 7.f 7:f572c33b:::benchmark_data_smithi049_76798_object49378:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T13:51:00.396 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:38.266784+0000", 2024-09-21T13:51:00.396 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 22.076054199000001, 2024-09-21T13:51:00.396 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.15800610500000001, 2024-09-21T13:51:00.396 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:51:00.396 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:51:00.396 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:00.396 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:51:00.396 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:51:00.396 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:00.396 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 49379 2024-09-21T13:51:00.396 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.396 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:51:00.396 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.396 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:51:00.396 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.266784+0000", 2024-09-21T13:51:00.397 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:00.397 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.397 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.397 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:51:00.397 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.266784+0000", 2024-09-21T13:51:00.397 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:00.397 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.397 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.397 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:51:00.397 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.266785+0000", 2024-09-21T13:51:00.397 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4499999999999996e-07 2024-09-21T13:51:00.397 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.397 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.397 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:51:00.397 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.266812+0000", 2024-09-21T13:51:00.398 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6188000000000001e-05 2024-09-21T13:51:00.398 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.398 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.398 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:51:00.398 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.266821+0000", 2024-09-21T13:51:00.398 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.55e-06 2024-09-21T13:51:00.398 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.398 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.398 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:51:00.398 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.266826+0000", 2024-09-21T13:51:00.398 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:51:00.400 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6500000000000004e-06 2024-09-21T13:51:00.400 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.400 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.400 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:51:00.400 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.266873+0000", 2024-09-21T13:51:00.400 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6860999999999997e-05 2024-09-21T13:51:00.400 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.400 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.400 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:51:00.400 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.267039+0000", 2024-09-21T13:51:00.400 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016667799999999999 2024-09-21T13:51:00.400 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.400 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.400 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,7", 2024-09-21T13:51:00.401 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.267211+0000", 2024-09-21T13:51:00.401 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00017161800000000001 2024-09-21T13:51:00.401 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.401 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.401 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:51:00.401 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.269160+0000", 2024-09-21T13:51:00.401 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001949568 2024-09-21T13:51:00.401 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.401 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.401 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:00.401 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.269376+0000", 2024-09-21T13:51:00.401 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00021547600000000001 2024-09-21T13:51:00.401 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.401 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.402 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:00.402 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.424351+0000", 2024-09-21T13:51:00.402 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.15497515100000001 2024-09-21T13:51:00.402 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.402 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.402 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:51:00.402 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.424741+0000", 2024-09-21T13:51:00.402 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00039041800000000002 2024-09-21T13:51:00.402 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.402 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.402 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:51:00.402 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.424791+0000", 2024-09-21T13:51:00.402 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9101999999999998e-05 2024-09-21T13:51:00.402 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:00.402 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:51:00.403 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:00.403 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.403 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.403 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:49382 7.f 7:ff3e454f:::benchmark_data_smithi049_76798_object49381:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T13:51:00.403 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:38.271099+0000", 2024-09-21T13:51:00.403 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 22.071740092999999, 2024-09-21T13:51:00.403 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.153799623, 2024-09-21T13:51:00.403 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:51:00.403 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:51:00.404 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:00.404 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:51:00.404 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:51:00.404 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:00.404 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 49382 2024-09-21T13:51:00.404 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.404 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:51:00.404 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.404 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:51:00.404 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.271099+0000", 2024-09-21T13:51:00.404 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:00.404 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.404 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.405 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:51:00.405 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.271099+0000", 2024-09-21T13:51:00.405 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:00.405 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.405 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.405 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:51:00.405 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.271100+0000", 2024-09-21T13:51:00.405 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.536e-06 2024-09-21T13:51:00.406 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.406 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.406 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:51:00.406 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.271118+0000", 2024-09-21T13:51:00.406 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7759999999999999e-05 2024-09-21T13:51:00.407 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.407 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.407 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:51:00.407 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.271128+0000", 2024-09-21T13:51:00.407 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0081000000000001e-05 2024-09-21T13:51:00.407 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.407 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.407 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:51:00.407 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.271133+0000", 2024-09-21T13:51:00.407 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0479999999999998e-06 2024-09-21T13:51:00.407 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.407 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.407 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:51:00.407 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.271171+0000", 2024-09-21T13:51:00.408 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8124000000000003e-05 2024-09-21T13:51:00.408 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.408 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.408 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:51:00.408 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.271248+0000", 2024-09-21T13:51:00.408 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6790000000000002e-05 2024-09-21T13:51:00.408 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.408 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.408 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,7", 2024-09-21T13:51:00.408 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.271330+0000", 2024-09-21T13:51:00.408 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1915000000000004e-05 2024-09-21T13:51:00.408 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.408 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.408 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:51:00.408 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.272432+0000", 2024-09-21T13:51:00.409 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0011020489999999999 2024-09-21T13:51:00.409 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.409 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.409 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:00.409 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.272727+0000", 2024-09-21T13:51:00.409 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00029485399999999999 2024-09-21T13:51:00.409 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.409 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.409 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:00.409 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.424826+0000", 2024-09-21T13:51:00.409 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.15209910300000001 2024-09-21T13:51:00.409 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.409 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.409 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:51:00.410 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.424869+0000", 2024-09-21T13:51:00.410 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2765000000000001e-05 2024-09-21T13:51:00.410 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.410 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.410 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:51:00.410 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.424898+0000", 2024-09-21T13:51:00.410 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9598e-05 2024-09-21T13:51:00.410 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:00.410 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:51:00.410 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:00.410 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.410 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.410 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:49438 7.2 7:46ec36bc:::benchmark_data_smithi049_76798_object49437:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T13:51:00.410 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:38.443721+0000", 2024-09-21T13:51:00.410 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 21.899117284999999, 2024-09-21T13:51:00.411 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.085477419999999998, 2024-09-21T13:51:00.411 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:51:00.411 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:51:00.411 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:00.411 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:51:00.411 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:51:00.411 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:00.411 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 49438 2024-09-21T13:51:00.411 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.411 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:51:00.411 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.411 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:51:00.411 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.443721+0000", 2024-09-21T13:51:00.411 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:00.412 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.412 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.412 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:51:00.412 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.443721+0000", 2024-09-21T13:51:00.412 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:00.412 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.412 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.412 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:51:00.412 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.443722+0000", 2024-09-21T13:51:00.412 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3900000000000004e-07 2024-09-21T13:51:00.412 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.412 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.412 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:51:00.412 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.443748+0000", 2024-09-21T13:51:00.412 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6301000000000001e-05 2024-09-21T13:51:00.413 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.413 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.413 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:51:00.413 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.443760+0000", 2024-09-21T13:51:00.413 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1299e-05 2024-09-21T13:51:00.413 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.413 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.413 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:51:00.413 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.443800+0000", 2024-09-21T13:51:00.413 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9801000000000001e-05 2024-09-21T13:51:00.413 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.413 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.413 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:51:00.413 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.515265+0000", 2024-09-21T13:51:00.413 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.071465552000000002 2024-09-21T13:51:00.414 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.414 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.414 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:51:00.414 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.515426+0000", 2024-09-21T13:51:00.414 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000160946 2024-09-21T13:51:00.414 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.414 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.414 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,2", 2024-09-21T13:51:00.414 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.515575+0000", 2024-09-21T13:51:00.414 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014882400000000001 2024-09-21T13:51:00.414 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.415 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.415 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:51:00.415 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.524056+0000", 2024-09-21T13:51:00.415 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0084811369999999997 2024-09-21T13:51:00.415 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.415 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.415 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:00.415 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.524097+0000", 2024-09-21T13:51:00.415 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1343e-05 2024-09-21T13:51:00.415 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.415 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.415 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:00.415 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.529111+0000", 2024-09-21T13:51:00.415 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0050135650000000002 2024-09-21T13:51:00.415 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.416 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.416 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:51:00.416 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.529162+0000", 2024-09-21T13:51:00.416 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0796e-05 2024-09-21T13:51:00.416 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.416 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.416 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:51:00.416 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.529199+0000", 2024-09-21T13:51:00.416 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7116999999999997e-05 2024-09-21T13:51:00.416 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:00.416 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:51:00.416 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:00.416 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.416 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.416 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:49440 7.2 7:44fcce4f:::benchmark_data_smithi049_76798_object49439:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T13:51:00.417 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:38.446168+0000", 2024-09-21T13:51:00.417 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 21.896670788000002, 2024-09-21T13:51:00.417 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.083184360999999998, 2024-09-21T13:51:00.417 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:51:00.417 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:51:00.417 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:00.417 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:51:00.417 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:51:00.417 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:00.417 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 49440 2024-09-21T13:51:00.417 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.418 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:51:00.418 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.418 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:51:00.418 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.446168+0000", 2024-09-21T13:51:00.418 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:00.418 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.418 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.418 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:51:00.418 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.446168+0000", 2024-09-21T13:51:00.418 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:00.418 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.418 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.418 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:51:00.418 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.446169+0000", 2024-09-21T13:51:00.419 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.398e-06 2024-09-21T13:51:00.419 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.419 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.419 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:51:00.419 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.446193+0000", 2024-09-21T13:51:00.419 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.349e-05 2024-09-21T13:51:00.419 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.419 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.419 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:51:00.419 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.446203+0000", 2024-09-21T13:51:00.419 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8009999999999998e-06 2024-09-21T13:51:00.419 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.419 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.420 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:51:00.420 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.446211+0000", 2024-09-21T13:51:00.420 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0530000000000007e-06 2024-09-21T13:51:00.420 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.420 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.420 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:51:00.420 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.517409+0000", 2024-09-21T13:51:00.420 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.071198537000000006 2024-09-21T13:51:00.420 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.420 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.420 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:51:00.420 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.517526+0000", 2024-09-21T13:51:00.420 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000117055 2024-09-21T13:51:00.420 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.420 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.421 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,2", 2024-09-21T13:51:00.421 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.517660+0000", 2024-09-21T13:51:00.421 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013403800000000001 2024-09-21T13:51:00.421 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.421 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.421 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:51:00.421 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.524108+0000", 2024-09-21T13:51:00.421 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0064480140000000002 2024-09-21T13:51:00.421 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.421 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.421 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:00.421 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.529235+0000", 2024-09-21T13:51:00.421 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.005127197 2024-09-21T13:51:00.421 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.422 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.422 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:00.422 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.529272+0000", 2024-09-21T13:51:00.422 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6659999999999998e-05 2024-09-21T13:51:00.422 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.422 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.422 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:51:00.422 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.529318+0000", 2024-09-21T13:51:00.422 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5818000000000001e-05 2024-09-21T13:51:00.422 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.422 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.422 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:51:00.422 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.529352+0000", 2024-09-21T13:51:00.422 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.43e-05 2024-09-21T13:51:00.423 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:00.423 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:51:00.423 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:00.423 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.423 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.423 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:49858 7.3 7:c0f3cdcd:::benchmark_data_smithi049_76798_object49857:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T13:51:00.423 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:38.748352+0000", 2024-09-21T13:51:00.423 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 21.594486454999998, 2024-09-21T13:51:00.423 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.109293347, 2024-09-21T13:51:00.423 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:51:00.423 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:51:00.423 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:00.423 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:51:00.423 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:51:00.423 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:00.424 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 49858 2024-09-21T13:51:00.424 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.424 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:51:00.424 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.424 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:51:00.424 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.748352+0000", 2024-09-21T13:51:00.424 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:00.424 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.424 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.424 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:51:00.424 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.748352+0000", 2024-09-21T13:51:00.424 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:00.424 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.424 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.424 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:51:00.425 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.748354+0000", 2024-09-21T13:51:00.425 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.691e-06 2024-09-21T13:51:00.425 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.425 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.425 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:51:00.425 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.748375+0000", 2024-09-21T13:51:00.425 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0971e-05 2024-09-21T13:51:00.425 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.425 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.425 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:51:00.425 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.748385+0000", 2024-09-21T13:51:00.425 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0216e-05 2024-09-21T13:51:00.425 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.425 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.425 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:51:00.425 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.748392+0000", 2024-09-21T13:51:00.426 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2899999999999997e-06 2024-09-21T13:51:00.426 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.426 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.426 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:51:00.426 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.748528+0000", 2024-09-21T13:51:00.426 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013609000000000001 2024-09-21T13:51:00.426 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.426 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.426 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:51:00.426 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.748630+0000", 2024-09-21T13:51:00.426 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000101103 2024-09-21T13:51:00.426 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.426 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.427 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,4", 2024-09-21T13:51:00.427 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.748751+0000", 2024-09-21T13:51:00.427 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000121266 2024-09-21T13:51:00.427 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.427 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.427 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:00.427 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.752266+0000", 2024-09-21T13:51:00.427 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0035147949999999998 2024-09-21T13:51:00.427 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.427 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.427 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:00.427 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.752405+0000", 2024-09-21T13:51:00.427 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013943199999999999 2024-09-21T13:51:00.427 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.427 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.428 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:51:00.429 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.857545+0000", 2024-09-21T13:51:00.429 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.105139435 2024-09-21T13:51:00.429 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.429 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.429 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:51:00.429 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.857599+0000", 2024-09-21T13:51:00.429 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4409e-05 2024-09-21T13:51:00.430 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.430 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.430 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:51:00.430 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.857646+0000", 2024-09-21T13:51:00.430 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6649e-05 2024-09-21T13:51:00.430 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:00.430 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:51:00.430 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:00.430 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.430 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.430 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:49863 7.3 7:c3764246:::benchmark_data_smithi049_76798_object49862:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T13:51:00.431 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:38.752183+0000", 2024-09-21T13:51:00.431 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 21.590655933000001, 2024-09-21T13:51:00.431 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.106083472, 2024-09-21T13:51:00.431 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:51:00.431 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:51:00.431 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:00.431 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:51:00.431 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:51:00.431 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:00.431 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 49863 2024-09-21T13:51:00.431 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.431 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:51:00.431 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.431 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:51:00.431 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.752183+0000", 2024-09-21T13:51:00.432 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:00.432 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.432 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.432 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:51:00.432 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.752183+0000", 2024-09-21T13:51:00.432 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:00.432 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.432 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.432 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:51:00.432 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.752184+0000", 2024-09-21T13:51:00.432 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5969999999999999e-06 2024-09-21T13:51:00.432 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.432 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.432 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:51:00.432 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.752212+0000", 2024-09-21T13:51:00.433 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.7770999999999999e-05 2024-09-21T13:51:00.433 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.433 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.433 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:51:00.433 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.752222+0000", 2024-09-21T13:51:00.433 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.5079999999999994e-06 2024-09-21T13:51:00.433 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.433 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.433 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:51:00.433 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.752229+0000", 2024-09-21T13:51:00.433 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2649999999999999e-06 2024-09-21T13:51:00.433 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.433 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.433 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:51:00.434 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.752442+0000", 2024-09-21T13:51:00.434 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00021349800000000001 2024-09-21T13:51:00.434 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.434 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.434 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:51:00.434 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.752592+0000", 2024-09-21T13:51:00.434 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014949200000000001 2024-09-21T13:51:00.434 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.434 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.434 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,4", 2024-09-21T13:51:00.434 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.752669+0000", 2024-09-21T13:51:00.434 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7012000000000001e-05 2024-09-21T13:51:00.434 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.434 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.435 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:00.435 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.836904+0000", 2024-09-21T13:51:00.435 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.084235589999999999 2024-09-21T13:51:00.435 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.435 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.435 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:00.435 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.836940+0000", 2024-09-21T13:51:00.435 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5556000000000003e-05 2024-09-21T13:51:00.435 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.435 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.435 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:51:00.435 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.857655+0000", 2024-09-21T13:51:00.435 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.020715026000000001 2024-09-21T13:51:00.435 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.436 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.436 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:51:00.436 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.858220+0000", 2024-09-21T13:51:00.436 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00056542099999999998 2024-09-21T13:51:00.436 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.436 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.436 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:51:00.436 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.858266+0000", 2024-09-21T13:51:00.436 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5735999999999997e-05 2024-09-21T13:51:00.436 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:00.436 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:51:00.436 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:00.436 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.436 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.436 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:49865 7.3 7:cd993bf8:::benchmark_data_smithi049_76798_object49864:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T13:51:00.437 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:38.752293+0000", 2024-09-21T13:51:00.437 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 21.590545518999999, 2024-09-21T13:51:00.437 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.106068513, 2024-09-21T13:51:00.437 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:51:00.437 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:51:00.437 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:00.437 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:51:00.437 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:51:00.437 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:00.437 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 49865 2024-09-21T13:51:00.437 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.437 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:51:00.438 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.438 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:51:00.438 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.752293+0000", 2024-09-21T13:51:00.438 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:00.438 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.438 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.438 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:51:00.438 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.752293+0000", 2024-09-21T13:51:00.438 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:00.438 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.438 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.438 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:51:00.438 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.752294+0000", 2024-09-21T13:51:00.438 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5199999999999996e-07 2024-09-21T13:51:00.439 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.439 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.439 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:51:00.439 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.752314+0000", 2024-09-21T13:51:00.439 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0262999999999999e-05 2024-09-21T13:51:00.439 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.439 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.439 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:51:00.439 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.752323+0000", 2024-09-21T13:51:00.439 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3600000000000002e-06 2024-09-21T13:51:00.439 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.439 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.439 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:51:00.439 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.752331+0000", 2024-09-21T13:51:00.440 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7759999999999997e-06 2024-09-21T13:51:00.440 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.440 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.440 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:51:00.440 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.754447+0000", 2024-09-21T13:51:00.440 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0021153690000000002 2024-09-21T13:51:00.440 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.440 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.440 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:51:00.440 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.754554+0000", 2024-09-21T13:51:00.440 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000107425 2024-09-21T13:51:00.440 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.440 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.440 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,4", 2024-09-21T13:51:00.440 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.754838+0000", 2024-09-21T13:51:00.441 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00028426500000000002 2024-09-21T13:51:00.441 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.441 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.441 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:00.441 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.836976+0000", 2024-09-21T13:51:00.441 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.082137817000000002 2024-09-21T13:51:00.441 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.441 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.441 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:00.441 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.837017+0000", 2024-09-21T13:51:00.441 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0580000000000001e-05 2024-09-21T13:51:00.441 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.441 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.441 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:51:00.442 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.858276+0000", 2024-09-21T13:51:00.442 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.021259699 2024-09-21T13:51:00.442 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.442 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.442 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:51:00.442 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.858327+0000", 2024-09-21T13:51:00.442 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0398000000000001e-05 2024-09-21T13:51:00.442 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.442 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.442 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:51:00.442 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.858362+0000", 2024-09-21T13:51:00.442 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4909000000000001e-05 2024-09-21T13:51:00.442 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:00.442 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:51:00.443 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:00.443 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.443 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.443 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:49866 7.3 7:cd709e42:::benchmark_data_smithi049_76798_object49865:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T13:51:00.443 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:38.752341+0000", 2024-09-21T13:51:00.443 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 21.590497676999998, 2024-09-21T13:51:00.443 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.10610172900000001, 2024-09-21T13:51:00.443 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:51:00.443 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:51:00.443 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:00.443 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:51:00.443 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:51:00.444 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:00.444 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 49866 2024-09-21T13:51:00.444 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.444 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:51:00.444 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.444 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:51:00.444 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.752341+0000", 2024-09-21T13:51:00.444 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:00.444 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.444 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.444 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:51:00.444 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.752341+0000", 2024-09-21T13:51:00.444 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:00.444 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.445 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.445 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:51:00.445 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.752342+0000", 2024-09-21T13:51:00.445 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7599999999999997e-07 2024-09-21T13:51:00.445 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.445 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.445 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:51:00.445 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.752385+0000", 2024-09-21T13:51:00.445 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3495000000000003e-05 2024-09-21T13:51:00.445 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.445 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.445 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:51:00.445 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.752393+0000", 2024-09-21T13:51:00.445 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.858e-06 2024-09-21T13:51:00.446 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.446 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.446 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:51:00.446 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.752399+0000", 2024-09-21T13:51:00.446 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9200000000000001e-06 2024-09-21T13:51:00.446 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.446 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.446 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:51:00.446 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.758212+0000", 2024-09-21T13:51:00.446 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0058135870000000003 2024-09-21T13:51:00.446 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.446 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.446 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:51:00.446 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.758312+0000", 2024-09-21T13:51:00.447 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9961000000000001e-05 2024-09-21T13:51:00.447 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.447 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.447 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,4", 2024-09-21T13:51:00.447 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.758431+0000", 2024-09-21T13:51:00.447 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000118114 2024-09-21T13:51:00.447 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.447 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.447 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:00.447 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.837052+0000", 2024-09-21T13:51:00.447 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.078621257999999999 2024-09-21T13:51:00.447 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.447 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.447 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:00.447 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.837091+0000", 2024-09-21T13:51:00.448 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9011999999999999e-05 2024-09-21T13:51:00.448 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.448 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.448 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:51:00.448 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.858369+0000", 2024-09-21T13:51:00.448 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.021277978999999999 2024-09-21T13:51:00.448 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.448 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.448 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:51:00.448 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.858409+0000", 2024-09-21T13:51:00.448 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0626e-05 2024-09-21T13:51:00.448 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.448 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.448 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:51:00.448 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.858443+0000", 2024-09-21T13:51:00.449 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3343000000000002e-05 2024-09-21T13:51:00.449 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:00.449 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:51:00.449 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:00.449 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.449 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.449 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:49877 7.f 7:fd3ea318:::benchmark_data_smithi049_76798_object49876:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T13:51:00.449 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:38.757895+0000", 2024-09-21T13:51:00.449 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 21.584944020999998, 2024-09-21T13:51:00.449 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.10146791099999999, 2024-09-21T13:51:00.449 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:51:00.449 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:51:00.449 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:00.449 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:51:00.449 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:51:00.450 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:00.450 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 49877 2024-09-21T13:51:00.450 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.450 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:51:00.450 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.450 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:51:00.450 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.757895+0000", 2024-09-21T13:51:00.450 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:00.450 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.450 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.450 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:51:00.450 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.757895+0000", 2024-09-21T13:51:00.450 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:00.450 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.451 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.451 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:51:00.451 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.757896+0000", 2024-09-21T13:51:00.451 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.091e-06 2024-09-21T13:51:00.451 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.451 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.451 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:51:00.451 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.757919+0000", 2024-09-21T13:51:00.451 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3495000000000001e-05 2024-09-21T13:51:00.451 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.451 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.451 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:51:00.451 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.757929+0000", 2024-09-21T13:51:00.452 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4040000000000001e-06 2024-09-21T13:51:00.452 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.452 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.452 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:51:00.452 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.757935+0000", 2024-09-21T13:51:00.452 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1809999999999998e-06 2024-09-21T13:51:00.452 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.452 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.452 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:51:00.452 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.833902+0000", 2024-09-21T13:51:00.452 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.075966843000000006 2024-09-21T13:51:00.452 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.452 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.452 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:51:00.453 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.834058+0000", 2024-09-21T13:51:00.453 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00015635599999999999 2024-09-21T13:51:00.453 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.453 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.453 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,7", 2024-09-21T13:51:00.453 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.834205+0000", 2024-09-21T13:51:00.453 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000147331 2024-09-21T13:51:00.453 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.453 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.453 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:00.453 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.840174+0000", 2024-09-21T13:51:00.453 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0059689529999999999 2024-09-21T13:51:00.453 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.453 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.453 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:51:00.454 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.840215+0000", 2024-09-21T13:51:00.454 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0948000000000002e-05 2024-09-21T13:51:00.454 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.454 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.454 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:00.454 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.859275+0000", 2024-09-21T13:51:00.454 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.01905975 2024-09-21T13:51:00.454 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.454 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.454 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:51:00.454 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.859325+0000", 2024-09-21T13:51:00.454 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0170000000000002e-05 2024-09-21T13:51:00.454 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.455 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.455 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:51:00.455 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.859363+0000", 2024-09-21T13:51:00.455 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7388999999999997e-05 2024-09-21T13:51:00.455 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:00.455 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:51:00.455 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:00.455 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.455 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.455 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:51700 7.2 7:4cc1fe17:::benchmark_data_smithi049_76798_object51699:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-21T13:51:00.455 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:39.987885+0000", 2024-09-21T13:51:00.455 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 20.354953403, 2024-09-21T13:51:00.455 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.085668237999999994, 2024-09-21T13:51:00.455 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:51:00.455 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:51:00.456 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:00.456 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:51:00.456 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:51:00.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:00.457 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:00.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:51:00.457 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:00.457 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 51700 2024-09-21T13:51:00.458 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.458 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:51:00.458 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.458 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:51:00.458 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:39.987885+0000", 2024-09-21T13:51:00.458 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:00.458 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.458 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.458 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:51:00.458 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:39.987885+0000", 2024-09-21T13:51:00.458 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:00.458 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.458 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.459 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:51:00.459 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:39.987886+0000", 2024-09-21T13:51:00.459 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.218e-06 2024-09-21T13:51:00.459 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.459 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.459 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:51:00.459 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:39.987909+0000", 2024-09-21T13:51:00.459 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2563000000000001e-05 2024-09-21T13:51:00.459 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.459 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.459 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:51:00.459 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:39.987918+0000", 2024-09-21T13:51:00.459 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.8210000000000008e-06 2024-09-21T13:51:00.459 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.460 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.460 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:51:00.460 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:39.987924+0000", 2024-09-21T13:51:00.460 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7250000000000002e-06 2024-09-21T13:51:00.460 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.460 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.460 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:51:00.460 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:39.988741+0000", 2024-09-21T13:51:00.460 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00081775800000000003 2024-09-21T13:51:00.460 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.460 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.460 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:51:00.460 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:39.988867+0000", 2024-09-21T13:51:00.460 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012573099999999999 2024-09-21T13:51:00.461 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.461 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.461 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,2", 2024-09-21T13:51:00.461 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:39.988990+0000", 2024-09-21T13:51:00.461 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012309899999999999 2024-09-21T13:51:00.461 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.461 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.461 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:51:00.461 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:39.992050+0000", 2024-09-21T13:51:00.461 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.003060021 2024-09-21T13:51:00.461 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.461 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.461 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:00.461 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:39.992457+0000", 2024-09-21T13:51:00.461 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00040643899999999998 2024-09-21T13:51:00.462 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.462 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.462 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:00.462 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:40.073312+0000", 2024-09-21T13:51:00.462 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.080855642000000005 2024-09-21T13:51:00.462 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.462 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.462 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:51:00.462 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:40.073370+0000", 2024-09-21T13:51:00.462 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7478000000000001e-05 2024-09-21T13:51:00.462 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.462 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.462 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:51:00.462 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:40.073554+0000", 2024-09-21T13:51:00.462 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00018374299999999999 2024-09-21T13:51:00.463 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:00.463 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:51:00.463 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:00.463 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.463 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.463 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:72624 7.3 7:cdd1aad8:::benchmark_data_smithi049_76798_object72623:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T13:51:00.463 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:54.473256+0000", 2024-09-21T13:51:00.463 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 5.8695829310000001, 2024-09-21T13:51:00.463 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.083663932999999996, 2024-09-21T13:51:00.463 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:51:00.463 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:51:00.463 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:00.463 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:51:00.463 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:51:00.464 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:00.464 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 72624 2024-09-21T13:51:00.464 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.464 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:51:00.464 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.464 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:51:00.464 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:54.473256+0000", 2024-09-21T13:51:00.464 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:00.464 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.464 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.464 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:51:00.464 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:54.473256+0000", 2024-09-21T13:51:00.464 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:00.464 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.464 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.465 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:51:00.465 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:54.473257+0000", 2024-09-21T13:51:00.465 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1399999999999996e-07 2024-09-21T13:51:00.465 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.465 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.465 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:51:00.465 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:54.473279+0000", 2024-09-21T13:51:00.465 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2935999999999999e-05 2024-09-21T13:51:00.465 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.465 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.465 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:51:00.465 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:54.473290+0000", 2024-09-21T13:51:00.466 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0858e-05 2024-09-21T13:51:00.466 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.466 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.466 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:51:00.466 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:54.473297+0000", 2024-09-21T13:51:00.466 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4250000000000003e-06 2024-09-21T13:51:00.466 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.466 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.466 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:51:00.466 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:54.477732+0000", 2024-09-21T13:51:00.466 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0044347859999999996 2024-09-21T13:51:00.466 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.466 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.466 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:51:00.466 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:54.477849+0000", 2024-09-21T13:51:00.467 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000117046 2024-09-21T13:51:00.467 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.467 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.467 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4", 2024-09-21T13:51:00.467 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:54.477974+0000", 2024-09-21T13:51:00.467 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012523600000000001 2024-09-21T13:51:00.467 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.467 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.467 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:51:00.467 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:54.480597+0000", 2024-09-21T13:51:00.467 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0026229790000000001 2024-09-21T13:51:00.467 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.467 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.467 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:00.468 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:54.556764+0000", 2024-09-21T13:51:00.468 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.076167224000000006 2024-09-21T13:51:00.468 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.468 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.468 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:51:00.468 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:54.556826+0000", 2024-09-21T13:51:00.468 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1896999999999994e-05 2024-09-21T13:51:00.468 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.468 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.468 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:51:00.468 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:54.556920+0000", 2024-09-21T13:51:00.468 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3732000000000003e-05 2024-09-21T13:51:00.468 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:00.468 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:51:00.468 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:00.469 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.469 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.469 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:72625 7.f 7:f70beccb:::benchmark_data_smithi049_76798_object72624:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T13:51:00.469 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:54.473313+0000", 2024-09-21T13:51:00.469 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 5.8695256210000002, 2024-09-21T13:51:00.469 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.091992851, 2024-09-21T13:51:00.469 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:51:00.469 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:51:00.469 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:00.469 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:51:00.469 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:51:00.469 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:00.469 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 72625 2024-09-21T13:51:00.469 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.469 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:51:00.469 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.470 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:51:00.470 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:54.473313+0000", 2024-09-21T13:51:00.470 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:00.470 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.470 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.470 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:51:00.470 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:54.473313+0000", 2024-09-21T13:51:00.470 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:00.470 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.470 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.470 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:51:00.470 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:54.473314+0000", 2024-09-21T13:51:00.470 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0999999999999998e-07 2024-09-21T13:51:00.470 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.471 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.471 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:51:00.471 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:54.473334+0000", 2024-09-21T13:51:00.471 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0460999999999999e-05 2024-09-21T13:51:00.471 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.471 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.471 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:51:00.471 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:54.473342+0000", 2024-09-21T13:51:00.471 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8709999999999997e-06 2024-09-21T13:51:00.471 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.471 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.471 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:51:00.471 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:54.473348+0000", 2024-09-21T13:51:00.471 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8699999999999997e-06 2024-09-21T13:51:00.472 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.472 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.472 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:51:00.472 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:54.473398+0000", 2024-09-21T13:51:00.472 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9676999999999999e-05 2024-09-21T13:51:00.472 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.472 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.472 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:51:00.472 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:54.473582+0000", 2024-09-21T13:51:00.472 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000184177 2024-09-21T13:51:00.472 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.473 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.473 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,7", 2024-09-21T13:51:00.473 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:54.473809+0000", 2024-09-21T13:51:00.473 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000226892 2024-09-21T13:51:00.473 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.473 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.473 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:51:00.473 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:54.476284+0000", 2024-09-21T13:51:00.473 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0024750150000000001 2024-09-21T13:51:00.473 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.473 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.473 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:00.473 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:54.480237+0000", 2024-09-21T13:51:00.473 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0039534890000000001 2024-09-21T13:51:00.473 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.474 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.474 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:00.474 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:54.565151+0000", 2024-09-21T13:51:00.474 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.084914006 2024-09-21T13:51:00.474 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.474 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.474 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:51:00.474 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:54.565256+0000", 2024-09-21T13:51:00.474 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010514300000000001 2024-09-21T13:51:00.474 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:00.474 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:00.474 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:51:00.475 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:54.565306+0000", 2024-09-21T13:51:00.475 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9639999999999999e-05 2024-09-21T13:51:00.475 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:00.475 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:51:00.475 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:00.475 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:00.475 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:51:00.475 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T13:51:00.557 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:00.598 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:00.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:00.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:00.700 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:00.743 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:00.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:00.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:00.754 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:51:00 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:51:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:51:00.846 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:51:00.876 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:00.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:00.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:51:00.909 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 48 16 82145 82129 106.898 92.3125 0.0169452 0.00934948 2024-09-21T13:51:00.978 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:51:01.015 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:01.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:01.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:51:01.117 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:01.128 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T13:51:01.150 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:01.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:01.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:01.169 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:00 smithi049 ceph-mon[40489]: pgmap v83: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 4.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 896 B/s rd, 82 MiB/s wr, 1.31k op/s; 29426/235728 objects degraded (12.483%) 2024-09-21T13:51:01.169 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:00 smithi049 ceph-mon[30080]: pgmap v83: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 4.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 896 B/s rd, 82 MiB/s wr, 1.31k op/s; 29426/235728 objects degraded (12.483%) 2024-09-21T13:51:01.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:00 smithi187 ceph-mon[33032]: pgmap v83: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 4.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 896 B/s rd, 82 MiB/s wr, 1.31k op/s; 29426/235728 objects degraded (12.483%) 2024-09-21T13:51:01.252 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:01.286 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:01.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:01.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:01.388 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:01.417 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:01.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:01.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:01.519 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:51:01.560 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:01.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:01.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:51:01.662 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:01.691 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:01.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:01.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:01.793 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:01.833 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:01.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:01.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:01.909 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 49 16 83439 83423 106.367 80.875 0.00898443 0.00939639 2024-09-21T13:51:01.935 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:01.967 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:01.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:01.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:02.069 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:51:02.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:02.113 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:02.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:51:02.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:01 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 29426/235728 objects degraded (12.483%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T13:51:02.214 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:02.243 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:02.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:02.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:02.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:01 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 29426/235728 objects degraded (12.483%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T13:51:02.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:01 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 29426/235728 objects degraded (12.483%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T13:51:02.345 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:02.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:02.367 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:02.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:02.468 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:51:02.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:02.508 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:02.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:51:02.610 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:51:02.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:02.649 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:02.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:51:02.750 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:02.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:02.787 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:02.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:02.888 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:51:02.909 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 50 16 84901 84885 106.067 91.375 0.00595772 0.00942235 2024-09-21T13:51:02.928 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:02.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:02.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:51:03.030 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:03.056 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:03.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:03.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:03.157 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:03.181 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:03.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:03.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:03.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:02 smithi187 ceph-mon[33032]: pgmap v84: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 4.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 697 B/s rd, 64 MiB/s wr, 1.02k op/s; 29426/235728 objects degraded (12.483%) 2024-09-21T13:51:03.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:02 smithi049 ceph-mon[30080]: pgmap v84: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 4.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 697 B/s rd, 64 MiB/s wr, 1.02k op/s; 29426/235728 objects degraded (12.483%) 2024-09-21T13:51:03.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:02 smithi049 ceph-mon[40489]: pgmap v84: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 4.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 697 B/s rd, 64 MiB/s wr, 1.02k op/s; 29426/235728 objects degraded (12.483%) 2024-09-21T13:51:03.283 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:03.314 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:03.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:03.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:03.415 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:51:03.443 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:03.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:03.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:51:03.545 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:51:03.581 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:03.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:03.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:51:03.682 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:51:03.714 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:03.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:03.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:51:03.721 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:51:03.793 INFO:teuthology.orchestra.run.smithi049.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-21T13:51:03.816 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:51:03.845 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:03.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:03.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:51:03.910 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 51 16 86280 86264 105.677 86.1875 0.00430137 0.00945805 2024-09-21T13:51:03.947 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:51:03.992 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:03.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:03.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:51:04.094 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:51:04.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:04.141 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:04.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:51:04.242 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:51:04.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:04.291 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:04.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:51:04.307 INFO:journalctl@ceph.alertmanager.a.smithi049.stdout:Sep 21 13:51:04 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-alertmanager-a[74283]: ts=2024-09-21T13:51:04.191Z caller=main.go:542 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-21T13:51:04.392 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:51:04.424 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:04.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:04.425 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:51:04.526 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:04.557 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:04.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:04.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:04.659 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:51:04.699 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:04.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:04.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:51:04.801 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:04.829 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:04.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:04.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:04.910 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 52 16 87677 87661 105.324 87.3125 0.00438431 0.00948979 2024-09-21T13:51:04.926 INFO:journalctl@ceph.alertmanager.a.smithi049.stdout:Sep 21 13:51:04 smithi049 podman[82709]: 2024-09-21 13:51:04.573852193 +0000 UTC m=+0.771227083 container died 3529245bd961feeb62033134ae8a2e0a6d8b0fcef0af93b943484e6b78cedb09 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T13:51:04.932 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:51:04.963 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:04.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:04.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:51:05.065 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:05.095 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:05.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:05.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:05.196 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:51:05.233 INFO:journalctl@ceph.alertmanager.a.smithi049.stdout:Sep 21 13:51:05 smithi049 podman[82709]: 2024-09-21 13:51:05.137746781 +0000 UTC m=+1.335121651 container cleanup 3529245bd961feeb62033134ae8a2e0a6d8b0fcef0af93b943484e6b78cedb09 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T13:51:05.233 INFO:journalctl@ceph.alertmanager.a.smithi049.stdout:Sep 21 13:51:05 smithi049 bash[82709]: ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-alertmanager-a 2024-09-21T13:51:05.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:05.238 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:05.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:51:05.339 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:51:05.370 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:05.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:05.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:51:05.472 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:05.502 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:05.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:05.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:05.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:05 smithi049 ceph-mon[40489]: pgmap v85: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 5.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 110 MiB/s wr, 1.75k op/s; 32062/257124 objects degraded (12.469%) 2024-09-21T13:51:05.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:05 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/4235147109' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:51:05.504 INFO:journalctl@ceph.alertmanager.a.smithi049.stdout:Sep 21 13:51:05 smithi049 podman[82795]: 2024-09-21 13:51:05.348196528 +0000 UTC m=+1.098866009 container remove 3529245bd961feeb62033134ae8a2e0a6d8b0fcef0af93b943484e6b78cedb09 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T13:51:05.505 INFO:journalctl@ceph.alertmanager.a.smithi049.stdout:Sep 21 13:51:05 smithi049 podman[82795]: 2024-09-21 13:51:05.466272948 +0000 UTC m=+1.216942430 volume remove 9b71406b8e40dec90764f447603cd1c4198bd1adbb9c5cf89fc7d850cbcf8285 2024-09-21T13:51:05.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:05 smithi049 ceph-mon[30080]: pgmap v85: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 5.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 110 MiB/s wr, 1.75k op/s; 32062/257124 objects degraded (12.469%) 2024-09-21T13:51:05.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:05 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/4235147109' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:51:05.604 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:05.631 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:05.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:05.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:05.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:05 smithi187 ceph-mon[33032]: pgmap v85: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 5.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 110 MiB/s wr, 1.75k op/s; 32062/257124 objects degraded (12.469%) 2024-09-21T13:51:05.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:05 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/4235147109' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:51:05.733 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:51:05.764 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:05.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:05.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:51:05.843 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:51:05.843 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:51:05.843 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:51:05.843 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:51:05.843 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:51:05.843 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:51:05.843 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:51:05.843 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:51:05.844 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:51:05.844 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-21T13:51:05.844 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:51:05.844 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:51:05.844 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:51:05.844 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:51:05.844 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:51:05.865 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:51:05.906 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:05.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:05.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:51:05.910 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 53 16 89161 89145 105.086 92.75 0.00560675 0.00951108 2024-09-21T13:51:06.008 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:51:06.056 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:06.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:06.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:51:06.156 INFO:teuthology.orchestra.run.smithi049.stdout:7.8 2024-09-21T13:51:06.156 INFO:teuthology.orchestra.run.smithi049.stdout:7.5 2024-09-21T13:51:06.156 INFO:teuthology.orchestra.run.smithi049.stdout:7.3 2024-09-21T13:51:06.157 INFO:teuthology.orchestra.run.smithi049.stdout:6.f 2024-09-21T13:51:06.157 INFO:teuthology.orchestra.run.smithi049.stdout:6.e 2024-09-21T13:51:06.157 INFO:teuthology.orchestra.run.smithi049.stdout:6.9 2024-09-21T13:51:06.157 INFO:teuthology.orchestra.run.smithi049.stdout:6.3 2024-09-21T13:51:06.157 INFO:teuthology.orchestra.run.smithi049.stdout:6.1c 2024-09-21T13:51:06.157 INFO:teuthology.orchestra.run.smithi049.stdout:6.1 2024-09-21T13:51:06.157 INFO:teuthology.orchestra.run.smithi049.stdout:7.2 2024-09-21T13:51:06.157 INFO:teuthology.orchestra.run.smithi049.stdout:6.17 2024-09-21T13:51:06.157 INFO:teuthology.orchestra.run.smithi049.stdout:4.19 2024-09-21T13:51:06.157 INFO:teuthology.orchestra.run.smithi049.stdout:3.1a 2024-09-21T13:51:06.157 INFO:teuthology.orchestra.run.smithi049.stdout:5.6 2024-09-21T13:51:06.157 INFO:teuthology.orchestra.run.smithi049.stdout:7.a 2024-09-21T13:51:06.157 INFO:teuthology.orchestra.run.smithi049.stdout:4.12 2024-09-21T13:51:06.157 INFO:teuthology.orchestra.run.smithi049.stdout:3.9 2024-09-21T13:51:06.157 INFO:teuthology.orchestra.run.smithi049.stdout:4.1b 2024-09-21T13:51:06.158 INFO:teuthology.orchestra.run.smithi049.stdout:3.13 2024-09-21T13:51:06.158 INFO:teuthology.orchestra.run.smithi049.stdout:3.1f 2024-09-21T13:51:06.158 INFO:teuthology.orchestra.run.smithi049.stdout:3.5 2024-09-21T13:51:06.158 INFO:teuthology.orchestra.run.smithi049.stdout:4.1c 2024-09-21T13:51:06.158 INFO:teuthology.orchestra.run.smithi049.stdout:5.d 2024-09-21T13:51:06.158 INFO:teuthology.orchestra.run.smithi049.stdout:3.0 2024-09-21T13:51:06.158 INFO:teuthology.orchestra.run.smithi049.stdout:2.1 2024-09-21T13:51:06.158 INFO:teuthology.orchestra.run.smithi049.stdout:3.4 2024-09-21T13:51:06.158 INFO:teuthology.orchestra.run.smithi049.stdout:4.1f 2024-09-21T13:51:06.158 INFO:teuthology.orchestra.run.smithi049.stdout:5.b 2024-09-21T13:51:06.158 INFO:teuthology.orchestra.run.smithi049.stdout:6.2 2024-09-21T13:51:06.158 INFO:teuthology.orchestra.run.smithi049.stdout:4.4 2024-09-21T13:51:06.158 INFO:teuthology.orchestra.run.smithi049.stdout:5.a 2024-09-21T13:51:06.159 INFO:teuthology.orchestra.run.smithi049.stdout:6.8 2024-09-21T13:51:06.159 INFO:teuthology.orchestra.run.smithi049.stdout:3.1e 2024-09-21T13:51:06.159 INFO:teuthology.orchestra.run.smithi049.stdout:4.d 2024-09-21T13:51:06.159 INFO:teuthology.orchestra.run.smithi049.stdout:5.14 2024-09-21T13:51:06.159 INFO:teuthology.orchestra.run.smithi049.stdout:5.1c 2024-09-21T13:51:06.159 INFO:teuthology.orchestra.run.smithi049.stdout:5.1e 2024-09-21T13:51:06.159 INFO:teuthology.orchestra.run.smithi049.stdout:4.6 2024-09-21T13:51:06.159 INFO:teuthology.orchestra.run.smithi049.stdout:5.4 2024-09-21T13:51:06.159 INFO:teuthology.orchestra.run.smithi049.stdout:7.e 2024-09-21T13:51:06.159 INFO:teuthology.orchestra.run.smithi049.stdout:5.8 2024-09-21T13:51:06.159 INFO:teuthology.orchestra.run.smithi049.stdout:6.12 2024-09-21T13:51:06.159 INFO:teuthology.orchestra.run.smithi049.stdout:6.0 2024-09-21T13:51:06.159 INFO:teuthology.orchestra.run.smithi049.stdout:6.14 2024-09-21T13:51:06.160 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:51:06.192 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:06.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:06.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:51:06.294 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:06.325 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:06.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:06.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:06.427 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:51:06.454 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:06.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:06.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:51:06.555 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T13:51:06.555 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T13:51:06.557 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:06.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:06.593 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:06.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:06.694 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:06.727 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:06.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:06.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:06.829 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:51:06.857 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:06.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:06.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:51:06.910 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 54 16 90603 90587 104.809 90.125 0.00892035 0.00953666 2024-09-21T13:51:06.958 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:51:06.991 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:06.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:06.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:51:07.092 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:07.127 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:07.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:07.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:07.229 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:07.259 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T13:51:07.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:07.283 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:07.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:07.384 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:51:07.418 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:07.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:07.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:51:07.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:07 smithi049 ceph-mon[40489]: pgmap v86: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 5.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 91 MiB/s wr, 1.46k op/s; 32062/257124 objects degraded (12.469%) 2024-09-21T13:51:07.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:07 smithi049 ceph-mon[30080]: pgmap v86: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 5.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 91 MiB/s wr, 1.46k op/s; 32062/257124 objects degraded (12.469%) 2024-09-21T13:51:07.520 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:07.558 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:07.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:07.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:07.660 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:07.668 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:51:07.693 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:07.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:07.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:07.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:07 smithi187 ceph-mon[33032]: pgmap v86: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 5.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 91 MiB/s wr, 1.46k op/s; 32062/257124 objects degraded (12.469%) 2024-09-21T13:51:07.712 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell -n osd.2 --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --op export --pgid 3.1f --file /var/log/ceph/exp.3.1f.2 2024-09-21T13:51:07.730 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:51:07.795 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:07.823 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:07.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:07.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:07.910 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 55 16 91993 91977 104.482 86.875 0.0097418 0.0095659 2024-09-21T13:51:07.924 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:07.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:07.954 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:07.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:08.055 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:51:08.096 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/osd.2/config 2024-09-21T13:51:08.105 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:08.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:08.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:51:08.207 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:08.240 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:08.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:08.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:08.276 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T13:51:08.276 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:51:08.342 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:51:08.392 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:08.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:08.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:51:08.494 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:08.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:08.532 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:08.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:08.556 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T13:51:08.583 INFO:journalctl@ceph.alertmanager.a.smithi049.stdout:Sep 21 13:51:08 smithi049 systemd[1]: ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@alertmanager.a.service: Deactivated successfully. 2024-09-21T13:51:08.583 INFO:journalctl@ceph.alertmanager.a.smithi049.stdout:Sep 21 13:51:08 smithi049 systemd[1]: Stopped Ceph alertmanager.a for 40a3d2e8-781f-11ef-baf6-efdc52797490. 2024-09-21T13:51:08.583 INFO:journalctl@ceph.alertmanager.a.smithi049.stdout:Sep 21 13:51:08 smithi049 systemd[1]: ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@alertmanager.a.service: Consumed 1.303s CPU time. 2024-09-21T13:51:08.583 INFO:journalctl@ceph.alertmanager.a.smithi049.stdout:Sep 21 13:51:08 smithi049 systemd[1]: Starting Ceph alertmanager.a for 40a3d2e8-781f-11ef-baf6-efdc52797490... 2024-09-21T13:51:08.633 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:08.687 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:08.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:08.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:08.790 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:51:08.848 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:08.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:08.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:51:08.911 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 56 16 93176 93160 103.936 73.9375 0.00560024 0.00961517 2024-09-21T13:51:08.950 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:51:08.966 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T13:51:08.997 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:08.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:08.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:51:09.098 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:51:09.134 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:09.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:09.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:51:09.194 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:51:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:51:09.236 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:51:09.251 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-09-21T13:51:09.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:09.284 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:09.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:51:09.385 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:09.393 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:51:09.423 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:09.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:09.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:09.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:09 smithi049 ceph-mon[40489]: pgmap v87: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 5.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 91 MiB/s wr, 1.46k op/s; 32062/257124 objects degraded (12.469%) 2024-09-21T13:51:09.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:09 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2490478645' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:51:09.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:09 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:51:09.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:09 smithi049 ceph-mon[30080]: pgmap v87: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 5.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 91 MiB/s wr, 1.46k op/s; 32062/257124 objects degraded (12.469%) 2024-09-21T13:51:09.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:09 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2490478645' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:51:09.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:09 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:51:09.525 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:09.529 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:09 smithi187 ceph-mon[33032]: pgmap v87: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 5.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 91 MiB/s wr, 1.46k op/s; 32062/257124 objects degraded (12.469%) 2024-09-21T13:51:09.529 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:09 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2490478645' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:51:09.529 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:09 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:51:09.554 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:09.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:09.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:09.656 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:51:09.693 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:09.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:09.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:51:09.795 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:51:09.823 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:09.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:09.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:51:09.832 INFO:teuthology.orchestra.run.smithi049.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-21T13:51:09.911 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 57 16 94554 94538 103.623 86.125 0.00789432 0.00964551 2024-09-21T13:51:09.925 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:51:09.964 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:09.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:09.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:51:10.003 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T13:51:10.003 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:51:10.065 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:10.097 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:10.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:10.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:10.199 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:51:10.208 INFO:journalctl@ceph.alertmanager.a.smithi049.stdout:Sep 21 13:51:09 smithi049 podman[83642]: 2024-09-21 13:51:09.781618544 +0000 UTC m=+0.565102825 image pull 11f11916f8cdfb17c1da74b20deb2075d778357b4d60474d82453303f8d461f5 quay.io/prometheus/alertmanager:v0.27.0 2024-09-21T13:51:10.208 INFO:journalctl@ceph.alertmanager.a.smithi049.stdout:Sep 21 13:51:09 smithi049 podman[83642]: 2024-09-21 13:51:09.883567774 +0000 UTC m=+0.667052046 volume create 4deb6f5be0a9fd7049fd917ef1002640663e30d11df69dbe0274465f50711c52 2024-09-21T13:51:10.208 INFO:journalctl@ceph.alertmanager.a.smithi049.stdout:Sep 21 13:51:09 smithi049 podman[83642]: 2024-09-21 13:51:09.992559585 +0000 UTC m=+0.776043859 container create 4adfef62b5cbc923096c89843940f55d787405b3944d75c91b1a3462382d1e7f (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T13:51:10.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:10.240 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:10.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:51:10.341 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:51:10.375 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:10.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:10.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:51:10.477 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:10.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:10 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:51:10.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:10 smithi049 ceph-mon[30080]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:51:10.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:10 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T13:51:10.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:10 smithi049 ceph-mon[30080]: osdmap e109: 8 total, 7 up, 8 in 2024-09-21T13:51:10.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:10 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 32062/257124 objects degraded (12.469%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T13:51:10.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:10 smithi049 ceph-mon[30080]: osdmap e110: 8 total, 7 up, 8 in 2024-09-21T13:51:10.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:10 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:51:10.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:10 smithi049 ceph-mon[40489]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:51:10.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:10 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T13:51:10.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:10 smithi049 ceph-mon[40489]: osdmap e109: 8 total, 7 up, 8 in 2024-09-21T13:51:10.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:10 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 32062/257124 objects degraded (12.469%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T13:51:10.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:10 smithi049 ceph-mon[40489]: osdmap e110: 8 total, 7 up, 8 in 2024-09-21T13:51:10.517 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:10.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:10.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:10.619 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:51:10.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:10.661 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:10.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:51:10.678 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:10 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:51:10.678 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:10 smithi187 ceph-mon[33032]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:51:10.678 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:10 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T13:51:10.678 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:10 smithi187 ceph-mon[33032]: osdmap e109: 8 total, 7 up, 8 in 2024-09-21T13:51:10.678 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:10 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 32062/257124 objects degraded (12.469%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T13:51:10.678 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:10 smithi187 ceph-mon[33032]: osdmap e110: 8 total, 7 up, 8 in 2024-09-21T13:51:10.725 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T13:51:10.762 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:51:10.809 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:10.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:10.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:51:10.899 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:51:10 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:51:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:51:10.911 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:51:10.912 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 58 16 95996 95980 103.39 90.125 0.0142257 0.00966705 2024-09-21T13:51:10.943 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:10.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:10.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:51:11.045 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:51:11.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:11.087 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:11.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:51:11.131 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:51:11.188 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:51:11.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:11.234 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:11.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:51:11.249 INFO:journalctl@ceph.alertmanager.a.smithi049.stdout:Sep 21 13:51:11 smithi049 podman[83642]: 2024-09-21 13:51:11.010947351 +0000 UTC m=+1.794431640 container init 4adfef62b5cbc923096c89843940f55d787405b3944d75c91b1a3462382d1e7f (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T13:51:11.249 INFO:journalctl@ceph.alertmanager.a.smithi049.stdout:Sep 21 13:51:11 smithi049 podman[83642]: 2024-09-21 13:51:11.020620683 +0000 UTC m=+1.804104960 container start 4adfef62b5cbc923096c89843940f55d787405b3944d75c91b1a3462382d1e7f (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T13:51:11.249 INFO:journalctl@ceph.alertmanager.a.smithi049.stdout:Sep 21 13:51:11 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-alertmanager-a[83730]: ts=2024-09-21T13:51:11.040Z caller=main.go:181 level=info msg="Starting Alertmanager" version="(version=0.27.0, branch=HEAD, revision=0aa3c2aad14cff039931923ab16b26b7481783b5)" 2024-09-21T13:51:11.249 INFO:journalctl@ceph.alertmanager.a.smithi049.stdout:Sep 21 13:51:11 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-alertmanager-a[83730]: ts=2024-09-21T13:51:11.041Z caller=main.go:182 level=info build_context="(go=go1.21.7, platform=linux/amd64, user=root@22cd11f671e9, date=20240228-11:51:20, tags=netgo)" 2024-09-21T13:51:11.250 INFO:journalctl@ceph.alertmanager.a.smithi049.stdout:Sep 21 13:51:11 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-alertmanager-a[83730]: ts=2024-09-21T13:51:11.042Z caller=cluster.go:186 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.49 port=9094 2024-09-21T13:51:11.250 INFO:journalctl@ceph.alertmanager.a.smithi049.stdout:Sep 21 13:51:11 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-alertmanager-a[83730]: ts=2024-09-21T13:51:11.046Z caller=cluster.go:683 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-21T13:51:11.250 INFO:journalctl@ceph.alertmanager.a.smithi049.stdout:Sep 21 13:51:11 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-alertmanager-a[83730]: ts=2024-09-21T13:51:11.098Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-21T13:51:11.250 INFO:journalctl@ceph.alertmanager.a.smithi049.stdout:Sep 21 13:51:11 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-alertmanager-a[83730]: ts=2024-09-21T13:51:11.098Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-21T13:51:11.250 INFO:journalctl@ceph.alertmanager.a.smithi049.stdout:Sep 21 13:51:11 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-alertmanager-a[83730]: ts=2024-09-21T13:51:11.103Z caller=tls_config.go:313 level=info msg="Listening on" address=[::]:9093 2024-09-21T13:51:11.250 INFO:journalctl@ceph.alertmanager.a.smithi049.stdout:Sep 21 13:51:11 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-alertmanager-a[83730]: ts=2024-09-21T13:51:11.103Z caller=tls_config.go:316 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-21T13:51:11.250 INFO:journalctl@ceph.alertmanager.a.smithi049.stdout:Sep 21 13:51:11 smithi049 bash[83642]: 4adfef62b5cbc923096c89843940f55d787405b3944d75c91b1a3462382d1e7f 2024-09-21T13:51:11.250 INFO:journalctl@ceph.alertmanager.a.smithi049.stdout:Sep 21 13:51:11 smithi049 systemd[1]: Started Ceph alertmanager.a for 40a3d2e8-781f-11ef-baf6-efdc52797490. 2024-09-21T13:51:11.336 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:51:11.367 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:11.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:11.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:51:11.469 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:51:11.500 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:11 smithi049 ceph-mon[30080]: pgmap v90: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 5.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 111 MiB/s wr, 1.77k op/s; 34693/278304 objects degraded (12.466%) 2024-09-21T13:51:11.500 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:11 smithi049 ceph-mon[30080]: 4.9 scrub starts 2024-09-21T13:51:11.500 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:11 smithi049 ceph-mon[30080]: 4.9 scrub ok 2024-09-21T13:51:11.500 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:11 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:51:11.500 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:11 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:51:11.500 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:11 smithi049 ceph-mon[40489]: pgmap v90: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 5.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 111 MiB/s wr, 1.77k op/s; 34693/278304 objects degraded (12.466%) 2024-09-21T13:51:11.500 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:11 smithi049 ceph-mon[40489]: 4.9 scrub starts 2024-09-21T13:51:11.500 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:11 smithi049 ceph-mon[40489]: 4.9 scrub ok 2024-09-21T13:51:11.501 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:11 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:51:11.501 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:11 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:51:11.510 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:11.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:11.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:51:11.518 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:11 smithi187 ceph-mon[33032]: pgmap v90: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 5.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 111 MiB/s wr, 1.77k op/s; 34693/278304 objects degraded (12.466%) 2024-09-21T13:51:11.518 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:11 smithi187 ceph-mon[33032]: 4.9 scrub starts 2024-09-21T13:51:11.518 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:11 smithi187 ceph-mon[33032]: 4.9 scrub ok 2024-09-21T13:51:11.518 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:11 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:51:11.518 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:11 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:51:11.612 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:51:11.651 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:11.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:11.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:51:11.753 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:51:11.774 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:11.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:11.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:51:11.776 INFO:teuthology.orchestra.run.smithi049.stderr:Exporting 3.1f info 3.1f( empty local-lis/les=94/95 n=0 ec=60/60 lis/c=94/94 les/c/f=95/95/0 sis=94) 2024-09-21T13:51:11.776 INFO:teuthology.orchestra.run.smithi049.stderr:Export successful 2024-09-21T13:51:11.876 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:51:11.906 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:11.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:11.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:51:11.911 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 59 16 97214 97198 102.928 76.125 0.0173871 0.00971074 2024-09-21T13:51:11.920 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:51:12.008 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:12.041 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:12.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:12.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:12.143 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:51:12.174 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:12.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:12.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:51:12.275 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:12.306 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:12.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:12.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:12.382 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:12 smithi049 ceph-mon[40489]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T13:51:12.382 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:12 smithi049 ceph-mon[40489]: Reconfiguring daemon prometheus.a on smithi187 2024-09-21T13:51:12.382 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:12 smithi049 ceph-mon[40489]: 4.14 scrub starts 2024-09-21T13:51:12.382 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:12 smithi049 ceph-mon[40489]: 4.14 scrub ok 2024-09-21T13:51:12.382 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:12 smithi049 ceph-mon[30080]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T13:51:12.382 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:12 smithi049 ceph-mon[30080]: Reconfiguring daemon prometheus.a on smithi187 2024-09-21T13:51:12.382 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:12 smithi049 ceph-mon[30080]: 4.14 scrub starts 2024-09-21T13:51:12.382 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:12 smithi049 ceph-mon[30080]: 4.14 scrub ok 2024-09-21T13:51:12.408 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:12.437 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:12.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:12.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:12.539 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:12.582 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:12.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:12.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:12.600 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:12 smithi187 ceph-mon[33032]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T13:51:12.600 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:12 smithi187 ceph-mon[33032]: Reconfiguring daemon prometheus.a on smithi187 2024-09-21T13:51:12.601 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:12 smithi187 ceph-mon[33032]: 4.14 scrub starts 2024-09-21T13:51:12.601 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:12 smithi187 ceph-mon[33032]: 4.14 scrub ok 2024-09-21T13:51:12.683 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:12.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:12.722 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:12.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:12.822 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:51:12.856 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:12.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:12.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:51:12.911 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:2024-09-21T13:51:12.912401+0000 min lat: 0.00133054 max lat: 1.05128 avg lat: 0.00973838 2024-09-21T13:51:12.912 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T13:51:12.912 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 60 16 98574 98558 102.63 85 0.0137526 0.00973838 2024-09-21T13:51:12.957 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:51:12.997 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T13:51:12.997 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:51:13.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:13.004 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:13.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:51:13.105 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:51:13.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:13.149 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:13.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:51:13.215 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell -n osd.2 --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --force --op remove --pgid 3.1f 2024-09-21T13:51:13.250 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:13.273 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:13.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:13.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:13.277 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:51:13.277 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:51:13.277 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:51:13.277 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:51:13.277 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:51:13.278 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-21T13:51:13.278 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:51:13.278 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:51:13.278 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:51:13.278 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:51:13.278 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:51:13.278 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:51:13.278 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:51:13.279 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:51:13.279 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:51:13.279 INFO:journalctl@ceph.alertmanager.a.smithi049.stdout:Sep 21 13:51:13 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-alertmanager-a[83730]: ts=2024-09-21T13:51:13.046Z caller=cluster.go:708 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000196828s 2024-09-21T13:51:13.375 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:13.431 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:13.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:13.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:13.533 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:13.558 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:13.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:13.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:13.660 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:51:13.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:13 smithi187 ceph-mon[33032]: pgmap v91: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 5.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 55 MiB/s wr, 882 op/s; 34693/278304 objects degraded (12.466%) 2024-09-21T13:51:13.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:13 smithi187 ceph-mon[33032]: 6.5 scrub starts 2024-09-21T13:51:13.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:13 smithi187 ceph-mon[33032]: 6.5 scrub ok 2024-09-21T13:51:13.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:13 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:51:13.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:13 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:51:13.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:13 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/4222430905' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:51:13.696 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:13 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:51:13.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:13.708 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:13.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:51:13.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:13 smithi049 ceph-mon[40489]: pgmap v91: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 5.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 55 MiB/s wr, 882 op/s; 34693/278304 objects degraded (12.466%) 2024-09-21T13:51:13.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:13 smithi049 ceph-mon[40489]: 6.5 scrub starts 2024-09-21T13:51:13.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:13 smithi049 ceph-mon[40489]: 6.5 scrub ok 2024-09-21T13:51:13.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:13 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:51:13.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:13 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:51:13.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:13 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/4222430905' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:51:13.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:13 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:51:13.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:13 smithi049 ceph-mon[30080]: pgmap v91: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 5.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 55 MiB/s wr, 882 op/s; 34693/278304 objects degraded (12.466%) 2024-09-21T13:51:13.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:13 smithi049 ceph-mon[30080]: 6.5 scrub starts 2024-09-21T13:51:13.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:13 smithi049 ceph-mon[30080]: 6.5 scrub ok 2024-09-21T13:51:13.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:13 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:51:13.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:13 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:51:13.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:13 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/4222430905' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:51:13.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:13 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:51:13.809 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:13.838 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:13.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:13.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:13.912 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 61 16 99777 99761 102.18 75.1875 0.00701998 0.00978211 2024-09-21T13:51:13.940 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:13.958 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T13:51:13.967 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:13.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:13.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:14.069 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:51:14.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:14.109 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:14.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:51:14.209 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:14.246 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:14.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:14.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:14.348 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:14.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:14.382 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:14.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:14.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:14 smithi187 ceph-mon[33032]: 5.2 scrub starts 2024-09-21T13:51:14.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:14 smithi187 ceph-mon[33032]: 5.2 scrub ok 2024-09-21T13:51:14.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:14 smithi187 ceph-mon[33032]: osdmap e111: 8 total, 7 up, 8 in 2024-09-21T13:51:14.483 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:51:14.524 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:14.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:14.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:51:14.626 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:51:14.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:14.667 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:14.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:51:14.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:14 smithi049 ceph-mon[40489]: 5.2 scrub starts 2024-09-21T13:51:14.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:14 smithi049 ceph-mon[40489]: 5.2 scrub ok 2024-09-21T13:51:14.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:14 smithi049 ceph-mon[40489]: osdmap e111: 8 total, 7 up, 8 in 2024-09-21T13:51:14.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:14 smithi049 ceph-mon[30080]: 5.2 scrub starts 2024-09-21T13:51:14.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:14 smithi049 ceph-mon[30080]: 5.2 scrub ok 2024-09-21T13:51:14.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:14 smithi049 ceph-mon[30080]: osdmap e111: 8 total, 7 up, 8 in 2024-09-21T13:51:14.769 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:14.820 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:14.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:14.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:14.853 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T13:51:14.911 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 62 16 100982 100966 101.746 75.3125 0.0172985 0.00982344 2024-09-21T13:51:14.922 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:51:14.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:14.977 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:14.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:51:15.078 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:51:15.126 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T13:51:15.126 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T13:51:15.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:15.133 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:15.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:51:15.169 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/osd.2/config 2024-09-21T13:51:15.235 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:15.269 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:15.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:15.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:15.340 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:15 smithi187 ceph-mon[33032]: pgmap v93: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 6.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.7 KiB/s rd, 141 MiB/s wr, 2.27k op/s; 37137/297876 objects degraded (12.467%) 2024-09-21T13:51:15.340 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:15 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 37137/297876 objects degraded (12.467%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T13:51:15.341 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:15 smithi187 ceph-mon[33032]: osdmap e112: 8 total, 7 up, 8 in 2024-09-21T13:51:15.353 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:15 smithi049 ceph-mon[30080]: pgmap v93: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 6.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.7 KiB/s rd, 141 MiB/s wr, 2.27k op/s; 37137/297876 objects degraded (12.467%) 2024-09-21T13:51:15.353 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:15 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 37137/297876 objects degraded (12.467%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T13:51:15.353 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:15 smithi049 ceph-mon[30080]: osdmap e112: 8 total, 7 up, 8 in 2024-09-21T13:51:15.353 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:15 smithi049 ceph-mon[40489]: pgmap v93: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 6.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.7 KiB/s rd, 141 MiB/s wr, 2.27k op/s; 37137/297876 objects degraded (12.467%) 2024-09-21T13:51:15.353 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:15 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 37137/297876 objects degraded (12.467%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T13:51:15.353 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:15 smithi049 ceph-mon[40489]: osdmap e112: 8 total, 7 up, 8 in 2024-09-21T13:51:15.371 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:51:15.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:15.413 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:15.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:51:15.514 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:15.549 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:15.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:15.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:15.651 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:15.685 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:15.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:15.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:15.787 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:51:15.817 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:15.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:15.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:51:15.876 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:51:15.912 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 63 16 102188 102172 101.328 75.375 0.0193124 0.00986383 2024-09-21T13:51:15.919 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:51:15.967 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:15.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:15.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:51:16.069 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:16.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:16.107 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:16.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:16.208 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:51:16.240 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:16.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:16.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:51:16.341 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:16.375 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:16.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:16.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:16.477 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:16.507 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:16.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:16.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:16.608 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:16.641 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:16.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:16.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:16.693 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:51:16.743 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:16.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:16.789 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:16.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:16.890 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:51:16.912 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 64 16 103422 103406 100.948 77.125 0.013105 0.0099015 2024-09-21T13:51:16.927 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:16.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:16.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:51:17.029 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:51:17.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:17.073 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:17.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:51:17.127 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T13:51:17.174 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:51:17.212 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:17.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:17.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:51:17.314 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:17.324 INFO:teuthology.orchestra.run.smithi049.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-21T13:51:17.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:17.381 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:17.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:17.401 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:17 smithi187 ceph-mon[33032]: 5.11 scrub starts 2024-09-21T13:51:17.401 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:17 smithi187 ceph-mon[33032]: 5.11 scrub ok 2024-09-21T13:51:17.401 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:17 smithi187 ceph-mon[33032]: pgmap v95: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 6.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 808 B/s rd, 64 MiB/s wr, 1.03k op/s; 37137/297876 objects degraded (12.467%) 2024-09-21T13:51:17.422 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:17 smithi049 ceph-mon[40489]: 5.11 scrub starts 2024-09-21T13:51:17.422 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:17 smithi049 ceph-mon[40489]: 5.11 scrub ok 2024-09-21T13:51:17.422 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:17 smithi049 ceph-mon[40489]: pgmap v95: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 6.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 808 B/s rd, 64 MiB/s wr, 1.03k op/s; 37137/297876 objects degraded (12.467%) 2024-09-21T13:51:17.422 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:17 smithi049 ceph-mon[30080]: 5.11 scrub starts 2024-09-21T13:51:17.423 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:17 smithi049 ceph-mon[30080]: 5.11 scrub ok 2024-09-21T13:51:17.423 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:17 smithi049 ceph-mon[30080]: pgmap v95: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 6.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 808 B/s rd, 64 MiB/s wr, 1.03k op/s; 37137/297876 objects degraded (12.467%) 2024-09-21T13:51:17.482 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:17.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:17.524 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:17.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:17.625 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:17.660 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:17.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:17.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:17.762 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:17.792 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:17.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:17.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:17.894 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:51:17.913 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 65 16 104717 104701 100.641 80.9375 0.0138913 0.00993148 2024-09-21T13:51:17.927 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:17.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:17.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:51:18.029 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:18.067 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:18.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:18.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:18.169 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:51:18.200 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:18.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:18.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:51:18.302 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:18.347 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:18.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:18.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:18.425 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T13:51:18.426 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-21T13:51:18.426 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:18.426 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:104044 7.0 7:0306660e:::benchmark_data_smithi049_76798_object104043:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-21T13:51:18.426 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:51:17.424337+0000", 2024-09-21T13:51:18.426 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.007949968, 2024-09-21T13:51:18.426 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0079810109999999997, 2024-09-21T13:51:18.426 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:51:18.426 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:51:18.426 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-21T13:51:18.426 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:51:18.426 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:51:18.427 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:18.427 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 104044 2024-09-21T13:51:18.427 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:18.427 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:51:18.427 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:18.427 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:51:18.427 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:17.424337+0000", 2024-09-21T13:51:18.427 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:18.427 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:18.427 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:18.427 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:51:18.427 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:17.424337+0000", 2024-09-21T13:51:18.427 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:18.427 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:18.428 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:18.428 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:51:18.428 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:17.424339+0000", 2024-09-21T13:51:18.428 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5090000000000001e-06 2024-09-21T13:51:18.428 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:18.428 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:18.428 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:51:18.428 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:17.424376+0000", 2024-09-21T13:51:18.428 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7166000000000003e-05 2024-09-21T13:51:18.428 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:18.428 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:18.428 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:51:18.428 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:17.424387+0000", 2024-09-21T13:51:18.428 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0468999999999999e-05 2024-09-21T13:51:18.429 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:18.429 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:18.429 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:51:18.429 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:17.424394+0000", 2024-09-21T13:51:18.429 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9929999999999998e-06 2024-09-21T13:51:18.429 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:18.429 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:18.429 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:51:18.429 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:17.424461+0000", 2024-09-21T13:51:18.429 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7638999999999994e-05 2024-09-21T13:51:18.429 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:18.429 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:18.429 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:51:18.429 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:17.424594+0000", 2024-09-21T13:51:18.430 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000133175 2024-09-21T13:51:18.430 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:18.430 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:18.430 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3", 2024-09-21T13:51:18.430 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:17.424741+0000", 2024-09-21T13:51:18.430 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014651700000000001 2024-09-21T13:51:18.430 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:18.430 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:18.430 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:51:18.430 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:17.428104+0000", 2024-09-21T13:51:18.430 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0033631450000000001 2024-09-21T13:51:18.430 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:18.430 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:51:18.431 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:18.431 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:18.431 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:18.431 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:104052 7.e 7:7a7eca34:::benchmark_data_smithi049_76798_object104051:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-21T13:51:18.431 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:51:17.432054+0000", 2024-09-21T13:51:18.431 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.000233802, 2024-09-21T13:51:18.431 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00034341999999999999, 2024-09-21T13:51:18.431 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:51:18.431 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:51:18.431 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T13:51:18.431 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:51:18.432 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:51:18.432 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:18.432 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 104052 2024-09-21T13:51:18.432 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:18.432 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:51:18.432 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:18.432 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:51:18.432 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:17.432054+0000", 2024-09-21T13:51:18.432 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:18.432 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:18.432 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:18.432 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:51:18.432 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:17.432054+0000", 2024-09-21T13:51:18.433 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:18.433 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:18.433 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:18.433 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:51:18.433 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:17.432056+0000", 2024-09-21T13:51:18.433 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.841e-06 2024-09-21T13:51:18.433 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:18.433 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:18.433 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:51:18.433 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:17.432099+0000", 2024-09-21T13:51:18.433 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2660000000000002e-05 2024-09-21T13:51:18.433 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:18.433 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:18.433 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:51:18.434 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:17.432121+0000", 2024-09-21T13:51:18.434 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2388999999999999e-05 2024-09-21T13:51:18.434 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:18.434 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:18.434 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:51:18.434 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:17.432132+0000", 2024-09-21T13:51:18.434 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0359e-05 2024-09-21T13:51:18.434 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:18.434 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:18.434 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:51:18.434 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:17.432209+0000", 2024-09-21T13:51:18.434 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6860000000000006e-05 2024-09-21T13:51:18.434 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:18.434 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:18.435 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:51:18.435 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:17.432343+0000", 2024-09-21T13:51:18.435 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000133926 2024-09-21T13:51:18.435 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:18.435 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:51:18.435 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:18.435 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:18.435 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2024-09-21T13:51:18.435 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 2 2024-09-21T13:51:18.435 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T13:51:18.448 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:18.484 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:18.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:18.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:18.586 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:18.612 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:18.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:18.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:18.694 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:51:18 smithi187 systemd[1]: Stopping Ceph prometheus.a for 40a3d2e8-781f-11ef-baf6-efdc52797490... 2024-09-21T13:51:18.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:18 smithi187 ceph-mon[33032]: 4.5 scrub starts 2024-09-21T13:51:18.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:18 smithi187 ceph-mon[33032]: 4.5 scrub ok 2024-09-21T13:51:18.713 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:51:18.734 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:18 smithi049 ceph-mon[40489]: 4.5 scrub starts 2024-09-21T13:51:18.734 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:18 smithi049 ceph-mon[40489]: 4.5 scrub ok 2024-09-21T13:51:18.734 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:18 smithi049 ceph-mon[30080]: 4.5 scrub starts 2024-09-21T13:51:18.734 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:18 smithi049 ceph-mon[30080]: 4.5 scrub ok 2024-09-21T13:51:18.745 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:18.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:18.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:51:18.847 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:18.879 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:18.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:18.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:18.912 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 66 16 106083 106067 100.409 85.375 0.00752576 0.0099537 2024-09-21T13:51:18.981 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:19.015 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:19.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:19.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:19.117 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:51:19.153 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:19.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:19.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:51:19.194 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:51:19 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:51:19.195 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 13:51:18 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=infra.usagestats t=2024-09-21T13:51:18.980305679Z level=info msg="Usage stats are ready to report" 2024-09-21T13:51:19.255 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:51:19.285 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T13:51:19.289 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:19.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:19.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:51:19.331 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:19 smithi049 ceph-mon[40489]: 3.19 scrub starts 2024-09-21T13:51:19.331 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:19 smithi049 ceph-mon[40489]: 3.19 scrub ok 2024-09-21T13:51:19.331 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:19 smithi049 ceph-mon[40489]: pgmap v96: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 6.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 51 MiB/s wr, 815 op/s; 37137/297876 objects degraded (12.467%) 2024-09-21T13:51:19.331 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:19 smithi049 ceph-mon[40489]: 6.1b scrub starts 2024-09-21T13:51:19.331 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:19 smithi049 ceph-mon[40489]: 6.1b scrub ok 2024-09-21T13:51:19.331 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:19 smithi049 ceph-mon[30080]: 3.19 scrub starts 2024-09-21T13:51:19.331 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:19 smithi049 ceph-mon[30080]: 3.19 scrub ok 2024-09-21T13:51:19.331 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:19 smithi049 ceph-mon[30080]: pgmap v96: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 6.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 51 MiB/s wr, 815 op/s; 37137/297876 objects degraded (12.467%) 2024-09-21T13:51:19.331 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:19 smithi049 ceph-mon[30080]: 6.1b scrub starts 2024-09-21T13:51:19.332 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:19 smithi049 ceph-mon[30080]: 6.1b scrub ok 2024-09-21T13:51:19.391 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:19.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:19.426 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:19.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:19.486 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:51:19.527 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:19.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:19.573 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:19.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:19.674 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:51:19.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:19 smithi187 ceph-mon[33032]: 3.19 scrub starts 2024-09-21T13:51:19.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:19 smithi187 ceph-mon[33032]: 3.19 scrub ok 2024-09-21T13:51:19.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:19 smithi187 ceph-mon[33032]: pgmap v96: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 6.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 51 MiB/s wr, 815 op/s; 37137/297876 objects degraded (12.467%) 2024-09-21T13:51:19.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:19 smithi187 ceph-mon[33032]: 6.1b scrub starts 2024-09-21T13:51:19.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:19 smithi187 ceph-mon[33032]: 6.1b scrub ok 2024-09-21T13:51:19.695 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:51:19 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[61820]: ts=2024-09-21T13:51:19.456Z caller=main.go:964 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-21T13:51:19.695 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:51:19 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[61820]: ts=2024-09-21T13:51:19.456Z caller=main.go:988 level=info msg="Stopping scrape discovery manager..." 2024-09-21T13:51:19.695 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:51:19 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[61820]: ts=2024-09-21T13:51:19.456Z caller=main.go:1002 level=info msg="Stopping notify discovery manager..." 2024-09-21T13:51:19.695 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:51:19 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[61820]: ts=2024-09-21T13:51:19.456Z caller=manager.go:177 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-21T13:51:19.695 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:51:19 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[61820]: ts=2024-09-21T13:51:19.456Z caller=main.go:984 level=info msg="Scrape discovery manager stopped" 2024-09-21T13:51:19.696 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:51:19 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[61820]: ts=2024-09-21T13:51:19.456Z caller=manager.go:187 level=info component="rule manager" msg="Rule manager stopped" 2024-09-21T13:51:19.696 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:51:19 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[61820]: ts=2024-09-21T13:51:19.456Z caller=main.go:1039 level=info msg="Stopping scrape manager..." 2024-09-21T13:51:19.696 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:51:19 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[61820]: ts=2024-09-21T13:51:19.456Z caller=main.go:998 level=info msg="Notify discovery manager stopped" 2024-09-21T13:51:19.696 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:51:19 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[61820]: ts=2024-09-21T13:51:19.457Z caller=main.go:1031 level=info msg="Scrape manager stopped" 2024-09-21T13:51:19.696 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:51:19 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[61820]: ts=2024-09-21T13:51:19.457Z caller=notifier.go:618 level=info component=notifier msg="Stopping notification manager..." 2024-09-21T13:51:19.696 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:51:19 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[61820]: ts=2024-09-21T13:51:19.457Z caller=main.go:1261 level=info msg="Notifier manager stopped" 2024-09-21T13:51:19.696 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:51:19 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[61820]: ts=2024-09-21T13:51:19.457Z caller=main.go:1273 level=info msg="See you next time!" 2024-09-21T13:51:19.696 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:51:19 smithi187 podman[70579]: 2024-09-21 13:51:19.467310003 +0000 UTC m=+0.150829952 container died f6d988baca2083ec6576667c7fc0aa84f7e284effa48c99afc12971935f1cd27 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T13:51:19.720 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:19.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:19.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:51:19.821 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:19.854 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:19.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:19.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:19.913 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 67 16 107311 107295 100.056 76.75 0.0044942 0.00998937 2024-09-21T13:51:19.956 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:51:20.013 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:20.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:20.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:51:20.116 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:20.160 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:20.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:20.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:20.170 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:51:20.171 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:51:20.194 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:51:19 smithi187 podman[70579]: 2024-09-21 13:51:19.886605793 +0000 UTC m=+0.570125734 container cleanup f6d988baca2083ec6576667c7fc0aa84f7e284effa48c99afc12971935f1cd27 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T13:51:20.194 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:51:19 smithi187 bash[70579]: ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a 2024-09-21T13:51:20.198 INFO:teuthology.orchestra.run.smithi049.stdout: marking collection for removal 2024-09-21T13:51:20.198 INFO:teuthology.orchestra.run.smithi049.stdout:setting '_remove' omap key 2024-09-21T13:51:20.198 INFO:teuthology.orchestra.run.smithi049.stdout:finish_remove_pgs 3.1f_head removing 3.1f 2024-09-21T13:51:20.199 INFO:teuthology.orchestra.run.smithi049.stdout:Remove successful 2024-09-21T13:51:20.214 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:51:20.214 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:51:20.214 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-21T13:51:20.251 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:51:20.251 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:51:20.263 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:51:20.285 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:51:20.285 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:51:20.303 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:20.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:20.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:51:20.315 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:51:20.316 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:51:20.322 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-09-21T13:51:20.352 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:51:20.357 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:51:20.357 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:51:20.391 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:51:20.392 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:51:20.405 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:51:20.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:20 smithi187 ceph-mon[33032]: 7.c scrub starts 2024-09-21T13:51:20.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:20 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:51:20.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:20 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/127259993' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:51:20.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:20 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:51:20.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:20 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/4168169280' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:51:20.445 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:51:20 smithi187 podman[70614]: 2024-09-21 13:51:20.194206544 +0000 UTC m=+0.726222760 container remove f6d988baca2083ec6576667c7fc0aa84f7e284effa48c99afc12971935f1cd27 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T13:51:20.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:20.446 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:20.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:51:20.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:20 smithi049 ceph-mon[40489]: 7.c scrub starts 2024-09-21T13:51:20.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:20 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:51:20.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:20 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/127259993' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:51:20.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:20 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:51:20.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:20 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/4168169280' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:51:20.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:20 smithi049 ceph-mon[30080]: 7.c scrub starts 2024-09-21T13:51:20.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:20 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:51:20.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:20 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/127259993' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:51:20.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:20 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:51:20.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:20 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/4168169280' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:51:20.547 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:20.589 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:20.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:20.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:20.690 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:51:20.731 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:20.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:20.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:51:20.833 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:51:20.856 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:20.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:20.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:51:20.913 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 68 16 108438 108422 99.6203 70.4375 0.00295733 0.0100261 2024-09-21T13:51:20.958 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:20.991 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:20.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:20.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:21.093 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:21.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:21.126 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:21.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:21.227 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:21.254 INFO:journalctl@ceph.alertmanager.a.smithi049.stdout:Sep 21 13:51:21 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-alertmanager-a[83730]: ts=2024-09-21T13:51:21.051Z caller=cluster.go:700 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.005044923s 2024-09-21T13:51:21.264 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:21.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:21.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:21.366 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:51:21.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:21.409 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:21.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:51:21.510 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:51:21.540 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T13:51:21.540 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T13:51:21.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:21.541 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:21.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:51:21.642 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:51:21.673 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:21.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:21.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:51:21.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:21 smithi187 ceph-mon[33032]: 7.d scrub starts 2024-09-21T13:51:21.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:21 smithi187 ceph-mon[33032]: pgmap v97: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 6.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.60k op/s; 39470/316590 objects degraded (12.467%) 2024-09-21T13:51:21.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:21 smithi187 ceph-mon[33032]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T13:51:21.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:21 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 39470/316590 objects degraded (12.467%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T13:51:21.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:21 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T13:51:21.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:21 smithi187 ceph-mon[33032]: osdmap e113: 8 total, 7 up, 8 in 2024-09-21T13:51:21.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:21 smithi187 ceph-mon[33032]: 3.14 scrub starts 2024-09-21T13:51:21.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:21 smithi187 ceph-mon[33032]: 3.14 scrub ok 2024-09-21T13:51:21.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:21 smithi049 ceph-mon[40489]: 7.d scrub starts 2024-09-21T13:51:21.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:21 smithi049 ceph-mon[40489]: pgmap v97: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 6.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.60k op/s; 39470/316590 objects degraded (12.467%) 2024-09-21T13:51:21.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:21 smithi049 ceph-mon[40489]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T13:51:21.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:21 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 39470/316590 objects degraded (12.467%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T13:51:21.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:21 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T13:51:21.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:21 smithi049 ceph-mon[40489]: osdmap e113: 8 total, 7 up, 8 in 2024-09-21T13:51:21.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:21 smithi049 ceph-mon[40489]: 3.14 scrub starts 2024-09-21T13:51:21.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:21 smithi049 ceph-mon[40489]: 3.14 scrub ok 2024-09-21T13:51:21.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:21 smithi049 ceph-mon[30080]: 7.d scrub starts 2024-09-21T13:51:21.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:21 smithi049 ceph-mon[30080]: pgmap v97: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 6.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.60k op/s; 39470/316590 objects degraded (12.467%) 2024-09-21T13:51:21.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:21 smithi049 ceph-mon[30080]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T13:51:21.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:21 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 39470/316590 objects degraded (12.467%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T13:51:21.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:21 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T13:51:21.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:21 smithi049 ceph-mon[30080]: osdmap e113: 8 total, 7 up, 8 in 2024-09-21T13:51:21.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:21 smithi049 ceph-mon[30080]: 3.14 scrub starts 2024-09-21T13:51:21.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:21 smithi049 ceph-mon[30080]: 3.14 scrub ok 2024-09-21T13:51:21.774 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:21.797 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:21.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:21.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:21.899 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:51:21.913 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 69 15 109758 109743 99.3731 82.5625 0.00553424 0.010058 2024-09-21T13:51:21.929 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:21.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:21.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:51:22.031 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:22.053 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:22.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:22.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:22.154 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:51:22.190 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:22.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:22.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:51:22.292 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:22.321 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell -n osd.2 --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.3.1f.2 2024-09-21T13:51:22.331 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:22.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:22.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:22.433 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:22.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:22.470 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:22.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:22.571 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:51:22.625 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:22.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:22.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:51:22.639 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:22 smithi187 ceph-mon[33032]: 7.c scrub ok 2024-09-21T13:51:22.640 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:22 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:51:22.640 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:22 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:51:22.640 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:22 smithi187 ceph-mon[33032]: 2.0 scrub starts 2024-09-21T13:51:22.640 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:22 smithi187 ceph-mon[33032]: 2.0 scrub ok 2024-09-21T13:51:22.640 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:22 smithi187 ceph-mon[33032]: osdmap e114: 8 total, 7 up, 8 in 2024-09-21T13:51:22.640 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:22 smithi187 ceph-mon[33032]: 2.2 scrub starts 2024-09-21T13:51:22.640 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:22 smithi187 ceph-mon[33032]: 2.2 scrub ok 2024-09-21T13:51:22.640 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:22 smithi187 ceph-mon[33032]: 4.1a scrub starts 2024-09-21T13:51:22.640 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:22 smithi187 ceph-mon[33032]: 4.1a scrub ok 2024-09-21T13:51:22.683 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T13:51:22.683 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:51:22.693 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/osd.2/config 2024-09-21T13:51:22.727 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:51:22.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:22 smithi049 ceph-mon[30080]: 7.c scrub ok 2024-09-21T13:51:22.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:22 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:51:22.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:22 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:51:22.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:22 smithi049 ceph-mon[30080]: 2.0 scrub starts 2024-09-21T13:51:22.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:22 smithi049 ceph-mon[30080]: 2.0 scrub ok 2024-09-21T13:51:22.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:22 smithi049 ceph-mon[30080]: osdmap e114: 8 total, 7 up, 8 in 2024-09-21T13:51:22.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:22 smithi049 ceph-mon[30080]: 2.2 scrub starts 2024-09-21T13:51:22.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:22 smithi049 ceph-mon[30080]: 2.2 scrub ok 2024-09-21T13:51:22.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:22 smithi049 ceph-mon[30080]: 4.1a scrub starts 2024-09-21T13:51:22.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:22 smithi049 ceph-mon[30080]: 4.1a scrub ok 2024-09-21T13:51:22.757 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:22 smithi049 ceph-mon[40489]: 7.c scrub ok 2024-09-21T13:51:22.757 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:22 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:51:22.757 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:22 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:51:22.757 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:22 smithi049 ceph-mon[40489]: 2.0 scrub starts 2024-09-21T13:51:22.758 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:22 smithi049 ceph-mon[40489]: 2.0 scrub ok 2024-09-21T13:51:22.758 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:22 smithi049 ceph-mon[40489]: osdmap e114: 8 total, 7 up, 8 in 2024-09-21T13:51:22.758 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:22 smithi049 ceph-mon[40489]: 2.2 scrub starts 2024-09-21T13:51:22.758 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:22 smithi049 ceph-mon[40489]: 2.2 scrub ok 2024-09-21T13:51:22.758 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:22 smithi049 ceph-mon[40489]: 4.1a scrub starts 2024-09-21T13:51:22.758 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:22 smithi049 ceph-mon[40489]: 4.1a scrub ok 2024-09-21T13:51:22.762 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:22.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:22.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:51:22.864 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:51:22.891 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:22.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:22.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:51:22.913 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 70 16 110999 110983 99.0606 77.5 0.00826771 0.0100902 2024-09-21T13:51:22.944 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:51:22 smithi187 systemd[1]: ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@prometheus.a.service: Deactivated successfully. 2024-09-21T13:51:22.944 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:51:22 smithi187 systemd[1]: Stopped Ceph prometheus.a for 40a3d2e8-781f-11ef-baf6-efdc52797490. 2024-09-21T13:51:22.945 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:51:22 smithi187 systemd[1]: ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@prometheus.a.service: Consumed 1.402s CPU time. 2024-09-21T13:51:22.945 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:51:22 smithi187 systemd[1]: Starting Ceph prometheus.a for 40a3d2e8-781f-11ef-baf6-efdc52797490... 2024-09-21T13:51:22.992 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:51:23.033 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:23.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:23.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:51:23.134 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:51:23.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:23.168 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:23.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:51:23.269 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:51:23.297 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:23.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:23.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:51:23.399 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:51:23.434 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:23.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:23.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:51:23.536 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:51:23.541 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T13:51:23.560 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:23.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:23.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:51:23.662 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:51:23.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:23 smithi187 ceph-mon[33032]: pgmap v100: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 6.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 697 B/s rd, 53 MiB/s wr, 850 op/s; 39470/316590 objects degraded (12.467%) 2024-09-21T13:51:23.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:23 smithi187 ceph-mon[33032]: 7.d scrub ok 2024-09-21T13:51:23.706 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:23.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:23.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:51:23.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:23 smithi049 ceph-mon[30080]: pgmap v100: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 6.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 697 B/s rd, 53 MiB/s wr, 850 op/s; 39470/316590 objects degraded (12.467%) 2024-09-21T13:51:23.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:23 smithi049 ceph-mon[30080]: 7.d scrub ok 2024-09-21T13:51:23.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:23 smithi049 ceph-mon[40489]: pgmap v100: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 6.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 697 B/s rd, 53 MiB/s wr, 850 op/s; 39470/316590 objects degraded (12.467%) 2024-09-21T13:51:23.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:23 smithi049 ceph-mon[40489]: 7.d scrub ok 2024-09-21T13:51:23.809 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:51:23.836 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:23.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:23.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:51:23.914 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 71 16 112322 112306 98.8295 82.6875 0.00475662 0.0101137 2024-09-21T13:51:23.939 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:23.977 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:23.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:23.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:24.078 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:24.101 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T13:51:24.121 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:24.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:24.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:24.186 INFO:teuthology.orchestra.run.smithi049.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-21T13:51:24.223 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:24.262 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:24.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:24.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:24.364 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:24.416 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:24.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:24.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:24.518 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:51:24.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:24.556 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:24.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:51:24.657 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:24.682 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:24 smithi049 ceph-mon[40489]: 6.19 scrub starts 2024-09-21T13:51:24.682 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:24 smithi049 ceph-mon[40489]: 6.19 scrub ok 2024-09-21T13:51:24.682 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:24 smithi049 ceph-mon[40489]: 6.6 scrub starts 2024-09-21T13:51:24.682 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:24 smithi049 ceph-mon[40489]: 6.6 scrub ok 2024-09-21T13:51:24.682 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:24 smithi049 ceph-mon[40489]: 5.1 scrub starts 2024-09-21T13:51:24.682 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:24 smithi049 ceph-mon[40489]: 5.1 scrub ok 2024-09-21T13:51:24.682 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:24 smithi049 ceph-mon[30080]: 6.19 scrub starts 2024-09-21T13:51:24.682 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:24 smithi049 ceph-mon[30080]: 6.19 scrub ok 2024-09-21T13:51:24.682 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:24 smithi049 ceph-mon[30080]: 6.6 scrub starts 2024-09-21T13:51:24.683 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:24 smithi049 ceph-mon[30080]: 6.6 scrub ok 2024-09-21T13:51:24.683 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:24 smithi049 ceph-mon[30080]: 5.1 scrub starts 2024-09-21T13:51:24.683 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:24 smithi049 ceph-mon[30080]: 5.1 scrub ok 2024-09-21T13:51:24.694 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:51:24 smithi187 podman[71143]: 2024-09-21 13:51:24.236526275 +0000 UTC m=+0.041880327 image pull 1d3b7f56885b6dd623f1785be963aa9c195f86bc256ea454e8d02a7980b79c53 quay.io/prometheus/prometheus:v2.51.0 2024-09-21T13:51:24.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:24 smithi187 ceph-mon[33032]: 6.19 scrub starts 2024-09-21T13:51:24.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:24 smithi187 ceph-mon[33032]: 6.19 scrub ok 2024-09-21T13:51:24.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:24 smithi187 ceph-mon[33032]: 6.6 scrub starts 2024-09-21T13:51:24.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:24 smithi187 ceph-mon[33032]: 6.6 scrub ok 2024-09-21T13:51:24.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:24 smithi187 ceph-mon[33032]: 5.1 scrub starts 2024-09-21T13:51:24.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:24 smithi187 ceph-mon[33032]: 5.1 scrub ok 2024-09-21T13:51:24.696 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:24.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:24.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:24.798 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:51:24.804 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:51:24.829 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:24.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:24.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:51:24.861 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T13:51:24.916 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 72 16 113696 113680 98.6469 85.875 0.0103744 0.0101325 2024-09-21T13:51:24.931 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:51:24.951 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:24.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:24.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:51:25.053 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:51:25.083 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:25.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:25.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:51:25.185 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:51:25.210 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:25.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:25.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:51:25.312 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:51:25.344 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:25.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:25.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:51:25.446 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:25.481 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:25.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:25.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:25.527 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:51:25.527 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:51:25.576 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:51:25.576 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:51:25.581 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-21T13:51:25.584 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:25.620 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:25 smithi049 ceph-mon[30080]: 5.5 scrub starts 2024-09-21T13:51:25.620 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:25 smithi049 ceph-mon[30080]: pgmap v101: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 6.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.57k op/s; 41800/335595 objects degraded (12.455%) 2024-09-21T13:51:25.621 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:25 smithi049 ceph-mon[30080]: 5.5 scrub ok 2024-09-21T13:51:25.621 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:25 smithi049 ceph-mon[30080]: 6.11 scrub starts 2024-09-21T13:51:25.621 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:25 smithi049 ceph-mon[30080]: 6.11 scrub ok 2024-09-21T13:51:25.621 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:25 smithi049 ceph-mon[40489]: 5.5 scrub starts 2024-09-21T13:51:25.621 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:25 smithi049 ceph-mon[40489]: pgmap v101: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 6.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.57k op/s; 41800/335595 objects degraded (12.455%) 2024-09-21T13:51:25.621 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:25 smithi049 ceph-mon[40489]: 5.5 scrub ok 2024-09-21T13:51:25.621 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:25 smithi049 ceph-mon[40489]: 6.11 scrub starts 2024-09-21T13:51:25.621 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:25 smithi049 ceph-mon[40489]: 6.11 scrub ok 2024-09-21T13:51:25.622 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:51:25.622 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:51:25.626 INFO:teuthology.orchestra.run.smithi049.stdout:get_pg_num_history pg_num_history pg_num_history(e100 pg_nums {1={20=1},2={59=3},3={60=32},4={62=32},5={64=32},6={66=32},7={83=16}} deleted_pools ) 2024-09-21T13:51:25.626 INFO:teuthology.orchestra.run.smithi049.stdout:Importing pgid 3.1f 2024-09-21T13:51:25.626 INFO:teuthology.orchestra.run.smithi049.stdout:write_pg epoch 95 info 3.1f( empty local-lis/les=94/95 n=0 ec=60/60 lis/c=94/94 les/c/f=95/95/0 sis=94) 2024-09-21T13:51:25.628 INFO:teuthology.orchestra.run.smithi049.stdout:Import successful 2024-09-21T13:51:25.637 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:25.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:25.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:25.642 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:25 smithi187 ceph-mon[33032]: 5.5 scrub starts 2024-09-21T13:51:25.642 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:25 smithi187 ceph-mon[33032]: pgmap v101: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 6.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.57k op/s; 41800/335595 objects degraded (12.455%) 2024-09-21T13:51:25.642 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:25 smithi187 ceph-mon[33032]: 5.5 scrub ok 2024-09-21T13:51:25.642 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:25 smithi187 ceph-mon[33032]: 6.11 scrub starts 2024-09-21T13:51:25.642 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:25 smithi187 ceph-mon[33032]: 6.11 scrub ok 2024-09-21T13:51:25.643 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:51:25.674 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:51:25.675 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:51:25.707 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:51:25.707 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:51:25.740 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:25.743 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:51:25.743 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:51:25.767 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:25.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:25.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:25.780 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:51:25.781 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:51:25.869 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:51:25.903 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:25.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:25.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:51:25.916 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 73 16 114827 114811 98.2639 70.6875 0.00539189 0.010172 2024-09-21T13:51:25.944 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:51:25 smithi187 podman[71143]: 2024-09-21 13:51:25.730657896 +0000 UTC m=+1.536011927 container create fb308ba7f84fd9687f72f2d2ce6f0d19af7167503647515356f5b8627f4798d9 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T13:51:26.005 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:26.040 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:26.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:26.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:26.142 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:26.169 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:26.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:26.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:26.270 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:51:26.307 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:26.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:26.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:51:26.409 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:51:26.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:26.447 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:26.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:51:26.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:26 smithi049 ceph-mon[40489]: 4.b scrub starts 2024-09-21T13:51:26.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:26 smithi049 ceph-mon[40489]: 4.b scrub ok 2024-09-21T13:51:26.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:26 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/598691690' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:51:26.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:26 smithi049 ceph-mon[30080]: 4.b scrub starts 2024-09-21T13:51:26.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:26 smithi049 ceph-mon[30080]: 4.b scrub ok 2024-09-21T13:51:26.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:26 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/598691690' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:51:26.524 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:26 smithi187 ceph-mon[33032]: 4.b scrub starts 2024-09-21T13:51:26.524 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:26 smithi187 ceph-mon[33032]: 4.b scrub ok 2024-09-21T13:51:26.524 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:26 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/598691690' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:51:26.524 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:51:26 smithi187 podman[71143]: 2024-09-21 13:51:26.415289187 +0000 UTC m=+2.220643239 container init fb308ba7f84fd9687f72f2d2ce6f0d19af7167503647515356f5b8627f4798d9 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T13:51:26.524 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:51:26 smithi187 podman[71143]: 2024-09-21 13:51:26.423857958 +0000 UTC m=+2.229211997 container start fb308ba7f84fd9687f72f2d2ce6f0d19af7167503647515356f5b8627f4798d9 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T13:51:26.524 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:51:26 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[71382]: ts=2024-09-21T13:51:26.471Z caller=main.go:617 level=info msg="Starting Prometheus Server" mode=server version="(version=2.51.0, branch=HEAD, revision=c05c15512acb675e3f6cd662a6727854e93fc024)" 2024-09-21T13:51:26.525 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:51:26 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[71382]: ts=2024-09-21T13:51:26.472Z caller=main.go:622 level=info build_context="(go=go1.22.1, platform=linux/amd64, user=root@b5723e458358, date=20240319-10:54:45, tags=netgo,builtinassets,stringlabels)" 2024-09-21T13:51:26.525 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:51:26 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[71382]: ts=2024-09-21T13:51:26.472Z caller=main.go:623 level=info host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi187 (none))" 2024-09-21T13:51:26.525 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:51:26 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[71382]: ts=2024-09-21T13:51:26.472Z caller=main.go:624 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-21T13:51:26.525 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:51:26 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[71382]: ts=2024-09-21T13:51:26.472Z caller=main.go:625 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-21T13:51:26.525 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:51:26 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[71382]: ts=2024-09-21T13:51:26.475Z caller=web.go:568 level=info component=web msg="Start listening for connections" address=:9095 2024-09-21T13:51:26.525 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:51:26 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[71382]: ts=2024-09-21T13:51:26.477Z caller=main.go:1129 level=info msg="Starting TSDB ..." 2024-09-21T13:51:26.525 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:51:26 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[71382]: ts=2024-09-21T13:51:26.479Z caller=tls_config.go:313 level=info component=web msg="Listening on" address=[::]:9095 2024-09-21T13:51:26.525 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:51:26 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[71382]: ts=2024-09-21T13:51:26.479Z caller=tls_config.go:316 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-21T13:51:26.525 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:51:26 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[71382]: ts=2024-09-21T13:51:26.486Z caller=head.go:616 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-21T13:51:26.525 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:51:26 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[71382]: ts=2024-09-21T13:51:26.486Z caller=head.go:698 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=4.006µs 2024-09-21T13:51:26.525 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:51:26 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[71382]: ts=2024-09-21T13:51:26.486Z caller=head.go:706 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-21T13:51:26.525 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:51:26 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[71382]: ts=2024-09-21T13:51:26.487Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-09-21T13:51:26.526 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:51:26 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[71382]: ts=2024-09-21T13:51:26.487Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-09-21T13:51:26.526 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:51:26 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[71382]: ts=2024-09-21T13:51:26.487Z caller=head.go:815 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=45.07µs wal_replay_duration=862.761µs wbl_replay_duration=126ns total_replay_duration=942.949µs 2024-09-21T13:51:26.526 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:51:26 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[71382]: ts=2024-09-21T13:51:26.490Z caller=main.go:1150 level=info fs_type=XFS_SUPER_MAGIC 2024-09-21T13:51:26.526 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:51:26 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[71382]: ts=2024-09-21T13:51:26.490Z caller=main.go:1153 level=info msg="TSDB started" 2024-09-21T13:51:26.526 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:51:26 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[71382]: ts=2024-09-21T13:51:26.490Z caller=main.go:1335 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-21T13:51:26.526 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:51:26 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[71382]: ts=2024-09-21T13:51:26.517Z caller=main.go:1372 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=26.722895ms db_storage=1.161µs remote_storage=1.522µs web_handler=504ns query_engine=828ns scrape=590.757µs scrape_sd=164.637µs notify=16.801µs notify_sd=489.444µs rules=24.872065ms tracing=9.479µs 2024-09-21T13:51:26.526 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:51:26 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[71382]: ts=2024-09-21T13:51:26.517Z caller=main.go:1114 level=info msg="Server is ready to receive web requests." 2024-09-21T13:51:26.526 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:51:26 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[71382]: ts=2024-09-21T13:51:26.517Z caller=manager.go:163 level=info component="rule manager" msg="Starting rule manager..." 2024-09-21T13:51:26.526 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:51:26 smithi187 bash[71143]: fb308ba7f84fd9687f72f2d2ce6f0d19af7167503647515356f5b8627f4798d9 2024-09-21T13:51:26.548 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:26.593 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:51:26.594 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:26.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:26.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:26.696 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:51:26.727 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:26.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:26.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:51:26.829 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:26.861 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:26.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:26.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:26.884 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 13:51:26 smithi187 systemd[1]: Started Ceph prometheus.a for 40a3d2e8-781f-11ef-baf6-efdc52797490. 2024-09-21T13:51:26.916 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 74 16 116141 116125 98.0457 82.125 0.0152975 0.0101944 2024-09-21T13:51:26.917 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:51:26 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: [21/Sep/2024:13:51:26] ENGINE Bus STOPPING 2024-09-21T13:51:26.963 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:26.969 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T13:51:26.969 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T13:51:27.006 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:27.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:27.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:27.108 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:27.142 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:27.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:27.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:27.190 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T13:51:27.190 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-21T13:51:27.191 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-21T13:51:27.191 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-21T13:51:27.191 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.191 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:36814 7.0 7:0b36f6f1:::benchmark_data_smithi049_76798_object36813:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-21T13:51:27.191 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:30.554038+0000", 2024-09-21T13:51:27.191 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 56.633158393000002, 2024-09-21T13:51:27.191 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.060013114999999999, 2024-09-21T13:51:27.191 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:51:27.191 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:51:27.191 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:27.191 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:51:27.191 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:51:27.192 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:27.192 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 36814 2024-09-21T13:51:27.192 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.192 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:51:27.192 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.192 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:51:27.192 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.554038+0000", 2024-09-21T13:51:27.192 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:27.192 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.192 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.192 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:51:27.192 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.554038+0000", 2024-09-21T13:51:27.192 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:27.192 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.193 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.193 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:51:27.193 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.554040+0000", 2024-09-21T13:51:27.193 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0499999999999999e-06 2024-09-21T13:51:27.193 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.193 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.193 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:51:27.193 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.554085+0000", 2024-09-21T13:51:27.193 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5862000000000002e-05 2024-09-21T13:51:27.193 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.193 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.193 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:51:27.193 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.554104+0000", 2024-09-21T13:51:27.194 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8538000000000001e-05 2024-09-21T13:51:27.194 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.194 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.194 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:51:27.194 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.554113+0000", 2024-09-21T13:51:27.194 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5820000000000002e-06 2024-09-21T13:51:27.194 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.194 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.194 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:51:27.194 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.554181+0000", 2024-09-21T13:51:27.194 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8225999999999994e-05 2024-09-21T13:51:27.194 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.194 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.195 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:51:27.195 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.554326+0000", 2024-09-21T13:51:27.195 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014477399999999999 2024-09-21T13:51:27.195 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.195 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.195 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3", 2024-09-21T13:51:27.195 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.554457+0000", 2024-09-21T13:51:27.195 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013148399999999999 2024-09-21T13:51:27.195 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.195 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.195 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:51:27.195 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.555732+0000", 2024-09-21T13:51:27.195 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00127516 2024-09-21T13:51:27.195 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.196 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.196 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:27.196 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.610625+0000", 2024-09-21T13:51:27.196 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.054892409000000003 2024-09-21T13:51:27.196 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.196 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.196 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:27.196 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.613925+0000", 2024-09-21T13:51:27.196 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0033000210000000002 2024-09-21T13:51:27.196 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.196 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.196 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:51:27.196 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.613996+0000", 2024-09-21T13:51:27.196 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1878000000000005e-05 2024-09-21T13:51:27.197 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.197 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.197 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:51:27.197 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.614051+0000", 2024-09-21T13:51:27.197 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4131000000000001e-05 2024-09-21T13:51:27.197 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:27.197 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:51:27.197 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:27.197 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.197 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.197 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:49869 7.0 7:07d76dad:::benchmark_data_smithi049_76798_object49868:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T13:51:27.197 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:38.752211+0000", 2024-09-21T13:51:27.197 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 48.434984722999999, 2024-09-21T13:51:27.197 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.081978307, 2024-09-21T13:51:27.197 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:51:27.198 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:51:27.198 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:27.198 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:51:27.198 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:51:27.198 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:27.198 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 49869 2024-09-21T13:51:27.198 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.198 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:51:27.198 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.198 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:51:27.198 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.752211+0000", 2024-09-21T13:51:27.198 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:27.198 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.198 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.199 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:51:27.199 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.752211+0000", 2024-09-21T13:51:27.199 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:27.199 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.199 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.199 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:51:27.199 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.752212+0000", 2024-09-21T13:51:27.199 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2839999999999999e-06 2024-09-21T13:51:27.199 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.199 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.199 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:51:27.199 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.752258+0000", 2024-09-21T13:51:27.199 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5627999999999997e-05 2024-09-21T13:51:27.199 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.200 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.200 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:51:27.200 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.752269+0000", 2024-09-21T13:51:27.200 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0808999999999999e-05 2024-09-21T13:51:27.200 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.200 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.200 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:51:27.200 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.752275+0000", 2024-09-21T13:51:27.200 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6180000000000003e-06 2024-09-21T13:51:27.200 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.200 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.200 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:51:27.200 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.752315+0000", 2024-09-21T13:51:27.200 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9999000000000001e-05 2024-09-21T13:51:27.201 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.201 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.201 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:51:27.201 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.752421+0000", 2024-09-21T13:51:27.201 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000106604 2024-09-21T13:51:27.201 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.201 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.201 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3", 2024-09-21T13:51:27.201 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.752559+0000", 2024-09-21T13:51:27.201 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000137868 2024-09-21T13:51:27.201 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.201 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.201 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:51:27.201 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.754020+0000", 2024-09-21T13:51:27.201 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001461118 2024-09-21T13:51:27.202 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.202 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.202 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:27.202 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.758622+0000", 2024-09-21T13:51:27.202 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0046015600000000002 2024-09-21T13:51:27.202 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.202 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.202 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:27.202 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.834115+0000", 2024-09-21T13:51:27.202 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.075493219 2024-09-21T13:51:27.202 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.202 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.202 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:51:27.202 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.834157+0000", 2024-09-21T13:51:27.202 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2208e-05 2024-09-21T13:51:27.203 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.203 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.203 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:51:27.203 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.834189+0000", 2024-09-21T13:51:27.203 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2391999999999998e-05 2024-09-21T13:51:27.203 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:27.203 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:51:27.203 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:27.203 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.203 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.203 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:49879 7.0 7:01f9292a:::benchmark_data_smithi049_76798_object49878:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T13:51:27.203 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:38.759652+0000", 2024-09-21T13:51:27.203 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 48.427543745999998, 2024-09-21T13:51:27.203 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.083001569999999997, 2024-09-21T13:51:27.203 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:51:27.204 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:51:27.204 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:27.204 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:51:26 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: [21/Sep/2024:13:51:26] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T13:51:27.204 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:51:26 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: [21/Sep/2024:13:51:26] ENGINE Bus STOPPED 2024-09-21T13:51:27.204 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:51:26 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: [21/Sep/2024:13:51:26] ENGINE Bus STARTING 2024-09-21T13:51:27.205 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: [21/Sep/2024:13:51:27] ENGINE Serving on http://:::9283 2024-09-21T13:51:27.205 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: [21/Sep/2024:13:51:27] ENGINE Bus STARTED 2024-09-21T13:51:27.205 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: [21/Sep/2024:13:51:27] ENGINE Bus STOPPING 2024-09-21T13:51:27.205 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: [21/Sep/2024:13:51:27] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T13:51:27.205 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: [21/Sep/2024:13:51:27] ENGINE Bus STOPPED 2024-09-21T13:51:27.205 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: [21/Sep/2024:13:51:27] ENGINE Bus STARTING 2024-09-21T13:51:27.205 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: [21/Sep/2024:13:51:27] ENGINE Serving on http://:::9283 2024-09-21T13:51:27.205 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: [21/Sep/2024:13:51:27] ENGINE Bus STARTED 2024-09-21T13:51:27.205 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:51:27.205 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:51:27.205 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:27.206 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 49879 2024-09-21T13:51:27.206 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.206 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:51:27.206 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.206 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:51:27.206 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.759652+0000", 2024-09-21T13:51:27.206 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:27.206 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.206 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.206 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:51:27.206 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.759652+0000", 2024-09-21T13:51:27.206 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:27.206 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.207 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.207 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:51:27.207 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.759653+0000", 2024-09-21T13:51:27.207 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.325e-06 2024-09-21T13:51:27.207 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.207 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.207 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:51:27.207 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.759716+0000", 2024-09-21T13:51:27.207 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2747000000000001e-05 2024-09-21T13:51:27.207 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.207 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.207 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:51:27.207 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.759730+0000", 2024-09-21T13:51:27.207 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3560999999999999e-05 2024-09-21T13:51:27.208 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.208 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.208 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:51:27.208 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.759738+0000", 2024-09-21T13:51:27.208 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8320000000000006e-06 2024-09-21T13:51:27.208 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.208 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.208 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:51:27.208 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.759790+0000", 2024-09-21T13:51:27.208 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2002000000000001e-05 2024-09-21T13:51:27.208 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.208 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.208 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:51:27.208 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.759956+0000", 2024-09-21T13:51:27.209 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00016607699999999999 2024-09-21T13:51:27.209 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.209 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.209 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3", 2024-09-21T13:51:27.209 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.760106+0000", 2024-09-21T13:51:27.209 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015017599999999999 2024-09-21T13:51:27.209 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.209 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.209 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:51:27.209 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.761441+0000", 2024-09-21T13:51:27.209 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0013353970000000001 2024-09-21T13:51:27.209 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.209 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.209 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:27.209 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.761942+0000", 2024-09-21T13:51:27.209 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00050108399999999995 2024-09-21T13:51:27.210 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.210 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.210 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:27.210 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.842549+0000", 2024-09-21T13:51:27.210 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.080607112999999994 2024-09-21T13:51:27.210 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.210 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.210 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:51:27.210 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.842611+0000", 2024-09-21T13:51:27.210 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1668000000000007e-05 2024-09-21T13:51:27.210 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.210 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.210 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:51:27.210 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.842654+0000", 2024-09-21T13:51:27.211 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2588e-05 2024-09-21T13:51:27.211 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:27.211 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:51:27.211 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:27.211 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.211 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.211 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:50937 7.0 7:0298dd21:::benchmark_data_smithi049_76798_object50936:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-21T13:51:27.211 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:39.496488+0000", 2024-09-21T13:51:27.211 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 47.690707918999998, 2024-09-21T13:51:27.211 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.057282837000000003, 2024-09-21T13:51:27.211 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:51:27.211 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:51:27.211 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:27.212 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:51:27.212 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:51:27.212 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:27.212 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 50937 2024-09-21T13:51:27.212 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.212 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:51:27.212 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.212 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:51:27.212 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.496488+0000", 2024-09-21T13:51:27.212 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:27.212 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.212 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.212 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:51:27.212 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.496488+0000", 2024-09-21T13:51:27.213 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:27.213 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.213 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.213 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:51:27.213 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.496489+0000", 2024-09-21T13:51:27.213 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2559999999999999e-06 2024-09-21T13:51:27.213 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.213 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.213 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:51:27.213 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.496566+0000", 2024-09-21T13:51:27.213 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6606000000000005e-05 2024-09-21T13:51:27.213 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.213 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.213 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:51:27.214 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.496577+0000", 2024-09-21T13:51:27.214 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0681e-05 2024-09-21T13:51:27.214 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.214 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.214 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:51:27.214 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.496584+0000", 2024-09-21T13:51:27.214 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5259999999999996e-06 2024-09-21T13:51:27.214 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.214 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.214 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:51:27.214 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.496706+0000", 2024-09-21T13:51:27.214 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012240800000000001 2024-09-21T13:51:27.214 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.214 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.215 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:51:27.215 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.496851+0000", 2024-09-21T13:51:27.215 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014474999999999999 2024-09-21T13:51:27.215 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.215 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.215 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3", 2024-09-21T13:51:27.215 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.497006+0000", 2024-09-21T13:51:27.215 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000154957 2024-09-21T13:51:27.215 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.215 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.215 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:51:27.215 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.500208+0000", 2024-09-21T13:51:27.215 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003201588 2024-09-21T13:51:27.215 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.216 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.216 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:27.216 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.502109+0000", 2024-09-21T13:51:27.216 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001901142 2024-09-21T13:51:27.216 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.216 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.216 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:27.216 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.553664+0000", 2024-09-21T13:51:27.216 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.051554836 2024-09-21T13:51:27.216 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.216 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.216 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:51:27.216 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.553728+0000", 2024-09-21T13:51:27.216 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4288999999999994e-05 2024-09-21T13:51:27.216 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.217 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.217 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:51:27.217 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.553771+0000", 2024-09-21T13:51:27.217 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2797999999999999e-05 2024-09-21T13:51:27.217 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:27.217 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:51:27.217 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:27.217 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.217 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.217 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:50938 7.0 7:09dca739:::benchmark_data_smithi049_76798_object50937:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-21T13:51:27.217 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:39.496603+0000", 2024-09-21T13:51:27.217 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 47.690592717999998, 2024-09-21T13:51:27.217 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.057305832000000001, 2024-09-21T13:51:27.217 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:51:27.218 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:51:27.218 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:27.218 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:51:27.218 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:51:27.218 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:27.218 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 50938 2024-09-21T13:51:27.218 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.218 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:51:27.218 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.218 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:51:27.218 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.496603+0000", 2024-09-21T13:51:27.218 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:27.218 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.218 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.219 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:51:27.219 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.496603+0000", 2024-09-21T13:51:27.219 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:27.219 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.219 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.219 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:51:27.219 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.496604+0000", 2024-09-21T13:51:27.219 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.44e-07 2024-09-21T13:51:27.219 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.219 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.219 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:51:27.219 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.496638+0000", 2024-09-21T13:51:27.219 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3887999999999998e-05 2024-09-21T13:51:27.219 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.219 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.220 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:51:27.220 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.496651+0000", 2024-09-21T13:51:27.220 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.377e-05 2024-09-21T13:51:27.220 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.220 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.220 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:51:27.220 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.496658+0000", 2024-09-21T13:51:27.220 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4810000000000003e-06 2024-09-21T13:51:27.220 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.220 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.220 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:51:27.220 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.498663+0000", 2024-09-21T13:51:27.220 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0020046080000000002 2024-09-21T13:51:27.220 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.220 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.221 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:51:27.221 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.498798+0000", 2024-09-21T13:51:27.221 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013565100000000001 2024-09-21T13:51:27.221 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.221 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.221 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3", 2024-09-21T13:51:27.221 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.498962+0000", 2024-09-21T13:51:27.221 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00016373700000000001 2024-09-21T13:51:27.221 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.221 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.221 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:51:27.221 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.501589+0000", 2024-09-21T13:51:27.221 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0026274269999999999 2024-09-21T13:51:27.221 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.221 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.222 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:27.222 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.503624+0000", 2024-09-21T13:51:27.222 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.002034545 2024-09-21T13:51:27.222 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.222 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.222 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:27.222 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.553826+0000", 2024-09-21T13:51:27.222 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.050202291000000003 2024-09-21T13:51:27.222 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.222 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.222 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:51:27.222 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.553869+0000", 2024-09-21T13:51:27.222 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3103999999999999e-05 2024-09-21T13:51:27.223 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.223 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.223 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:51:27.223 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.553909+0000", 2024-09-21T13:51:27.223 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9685999999999999e-05 2024-09-21T13:51:27.223 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:27.223 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:51:27.223 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:27.223 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.223 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.223 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:50940 7.0 7:02a88a25:::benchmark_data_smithi049_76798_object50939:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-21T13:51:27.223 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:39.497816+0000", 2024-09-21T13:51:27.224 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 47.689379713999998, 2024-09-21T13:51:27.224 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.056212959999999999, 2024-09-21T13:51:27.224 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:51:27.224 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:51:27.224 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:27.224 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:51:27.224 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:51:27.224 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:27.224 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 50940 2024-09-21T13:51:27.224 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.224 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:51:27.224 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.224 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:51:27.224 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.497816+0000", 2024-09-21T13:51:27.225 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:27.225 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.225 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.225 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:51:27.225 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.497816+0000", 2024-09-21T13:51:27.225 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:27.225 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.225 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.225 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:51:27.225 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.497817+0000", 2024-09-21T13:51:27.225 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0979999999999999e-06 2024-09-21T13:51:27.225 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.225 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.225 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:51:27.226 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.497856+0000", 2024-09-21T13:51:27.226 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8840000000000001e-05 2024-09-21T13:51:27.226 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.226 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.226 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:51:27.226 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.497871+0000", 2024-09-21T13:51:27.226 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4739000000000001e-05 2024-09-21T13:51:27.226 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.226 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.226 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:51:27.226 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.497878+0000", 2024-09-21T13:51:27.226 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2429999999999999e-06 2024-09-21T13:51:27.226 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.226 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.227 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:51:27.227 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.500241+0000", 2024-09-21T13:51:27.227 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0023630669999999999 2024-09-21T13:51:27.227 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.227 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.227 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:51:27.227 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.500314+0000", 2024-09-21T13:51:27.227 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2613000000000004e-05 2024-09-21T13:51:27.227 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.227 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.227 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3", 2024-09-21T13:51:27.227 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.500406+0000", 2024-09-21T13:51:27.227 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.2023999999999998e-05 2024-09-21T13:51:27.227 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.228 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.228 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:51:27.228 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.501749+0000", 2024-09-21T13:51:27.228 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00134329 2024-09-21T13:51:27.228 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.228 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.228 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:27.228 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.503660+0000", 2024-09-21T13:51:27.228 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0019112140000000001 2024-09-21T13:51:27.228 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.228 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.228 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:27.228 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.553971+0000", 2024-09-21T13:51:27.228 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.050310362999999997 2024-09-21T13:51:27.228 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.229 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.229 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:51:27.229 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.554007+0000", 2024-09-21T13:51:27.229 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6705999999999997e-05 2024-09-21T13:51:27.229 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.229 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.229 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:51:27.229 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.554029+0000", 2024-09-21T13:51:27.229 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1763000000000002e-05 2024-09-21T13:51:27.229 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:27.229 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:51:27.229 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:27.229 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.229 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.229 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:51707 7.0 7:0ca9656b:::benchmark_data_smithi049_76798_object51706:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-21T13:51:27.230 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:39.989086+0000", 2024-09-21T13:51:27.230 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 47.198109844999998, 2024-09-21T13:51:27.230 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.081225758999999995, 2024-09-21T13:51:27.230 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:51:27.230 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:51:27.230 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:27.230 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:51:27.230 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:51:27.230 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:27.230 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 51707 2024-09-21T13:51:27.230 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.230 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:51:27.230 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.230 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:51:27.231 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.989086+0000", 2024-09-21T13:51:27.231 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:27.231 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.231 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.231 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:51:27.231 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.989086+0000", 2024-09-21T13:51:27.231 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:27.231 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.231 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.231 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:51:27.231 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.989088+0000", 2024-09-21T13:51:27.231 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.592e-06 2024-09-21T13:51:27.231 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.231 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.232 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:51:27.232 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.989126+0000", 2024-09-21T13:51:27.232 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8670000000000001e-05 2024-09-21T13:51:27.232 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.232 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.232 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:51:27.232 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.989142+0000", 2024-09-21T13:51:27.232 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5704000000000001e-05 2024-09-21T13:51:27.232 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.232 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.232 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:51:27.232 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.989156+0000", 2024-09-21T13:51:27.232 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4156e-05 2024-09-21T13:51:27.232 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.233 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.233 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:51:27.233 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.989207+0000", 2024-09-21T13:51:27.233 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0905999999999997e-05 2024-09-21T13:51:27.233 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.233 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.233 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:51:27.233 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.989291+0000", 2024-09-21T13:51:27.233 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3602000000000004e-05 2024-09-21T13:51:27.233 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.233 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.233 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3", 2024-09-21T13:51:27.234 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.989372+0000", 2024-09-21T13:51:27.234 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1148000000000003e-05 2024-09-21T13:51:27.234 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.234 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.234 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:51:27.234 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.990876+0000", 2024-09-21T13:51:27.234 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001504411 2024-09-21T13:51:27.234 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.234 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.234 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:27.234 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.992325+0000", 2024-09-21T13:51:27.234 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0014486869999999999 2024-09-21T13:51:27.235 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.235 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.235 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:27.235 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:40.070209+0000", 2024-09-21T13:51:27.235 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.077884298000000005 2024-09-21T13:51:27.235 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.235 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.235 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:51:27.235 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:40.070266+0000", 2024-09-21T13:51:27.235 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7031000000000003e-05 2024-09-21T13:51:27.235 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.235 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.235 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:51:27.236 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:40.070312+0000", 2024-09-21T13:51:27.236 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5553999999999998e-05 2024-09-21T13:51:27.236 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:27.236 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:51:27.236 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:27.236 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.236 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.236 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:72607 7.e 7:7c367c4a:::benchmark_data_smithi049_76798_object72606:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-09-21T13:51:27.236 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:54.457892+0000", 2024-09-21T13:51:27.236 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 32.729303749000003, 2024-09-21T13:51:27.236 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.10128385299999999, 2024-09-21T13:51:27.236 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:51:27.236 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:51:27.237 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:27.237 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:51:27.237 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:51:27.237 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:27.237 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 72607 2024-09-21T13:51:27.237 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.237 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:51:27.237 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.237 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:51:27.237 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.457892+0000", 2024-09-21T13:51:27.237 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:27.237 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.237 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.237 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:51:27.238 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.457892+0000", 2024-09-21T13:51:27.238 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:27.238 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.238 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.238 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:51:27.238 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.457894+0000", 2024-09-21T13:51:27.238 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5090000000000001e-06 2024-09-21T13:51:27.238 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.238 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.238 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:51:27.238 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.457945+0000", 2024-09-21T13:51:27.238 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0983999999999998e-05 2024-09-21T13:51:27.238 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.239 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.239 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:51:27.239 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.457959+0000", 2024-09-21T13:51:27.239 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3987e-05 2024-09-21T13:51:27.239 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.239 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.239 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:51:27.239 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.457968+0000", 2024-09-21T13:51:27.239 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1200000000000008e-06 2024-09-21T13:51:27.239 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.239 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.239 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:51:27.239 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.458027+0000", 2024-09-21T13:51:27.240 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9599000000000003e-05 2024-09-21T13:51:27.240 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.240 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.240 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:51:27.240 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.458157+0000", 2024-09-21T13:51:27.240 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012947699999999999 2024-09-21T13:51:27.240 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.240 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.240 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0", 2024-09-21T13:51:27.240 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.458292+0000", 2024-09-21T13:51:27.240 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000135642 2024-09-21T13:51:27.240 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.241 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.241 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:51:27.241 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.462010+0000", 2024-09-21T13:51:27.241 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0037178240000000002 2024-09-21T13:51:27.241 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.241 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.241 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:27.241 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.559021+0000", 2024-09-21T13:51:27.241 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.097010420999999999 2024-09-21T13:51:27.241 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.241 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.241 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:51:27.241 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.559081+0000", 2024-09-21T13:51:27.241 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0089e-05 2024-09-21T13:51:27.241 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.242 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.242 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:51:27.242 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.559176+0000", 2024-09-21T13:51:27.242 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.5200999999999999e-05 2024-09-21T13:51:27.242 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:27.242 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:51:27.242 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:27.242 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.242 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.242 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:72635 7.e 7:7083cf4b:::benchmark_data_smithi049_76798_object72634:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T13:51:27.242 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:54.481377+0000", 2024-09-21T13:51:27.242 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 32.705819030000001, 2024-09-21T13:51:27.242 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.083056779999999997, 2024-09-21T13:51:27.243 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:51:27.243 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:51:27.243 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:27.243 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:51:27.243 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:51:27.243 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:27.243 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 72635 2024-09-21T13:51:27.243 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.243 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:51:27.243 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.243 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:51:27.243 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.481377+0000", 2024-09-21T13:51:27.243 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:27.243 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.243 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.244 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:51:27.244 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.481377+0000", 2024-09-21T13:51:27.244 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:27.244 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.244 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.244 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:51:27.244 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.481378+0000", 2024-09-21T13:51:27.244 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4300000000000001e-06 2024-09-21T13:51:27.244 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.244 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.244 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:51:27.244 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.481491+0000", 2024-09-21T13:51:27.244 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011306000000000001 2024-09-21T13:51:27.244 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.245 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.245 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:51:27.245 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.481507+0000", 2024-09-21T13:51:27.245 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5167000000000001e-05 2024-09-21T13:51:27.245 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.245 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.245 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:51:27.245 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.481514+0000", 2024-09-21T13:51:27.245 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5109999999999997e-06 2024-09-21T13:51:27.245 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.245 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.245 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:51:27.245 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.481573+0000", 2024-09-21T13:51:27.245 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9197999999999998e-05 2024-09-21T13:51:27.246 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.246 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.246 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:51:27.246 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.481707+0000", 2024-09-21T13:51:27.246 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000133996 2024-09-21T13:51:27.246 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.246 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.246 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0", 2024-09-21T13:51:27.246 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:27.248 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.481857+0000", 2024-09-21T13:51:27.248 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000150223 2024-09-21T13:51:27.248 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.248 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.248 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:51:27.248 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.484065+0000", 2024-09-21T13:51:27.248 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.002207859 2024-09-21T13:51:27.248 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.248 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.248 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:27.248 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.564256+0000", 2024-09-21T13:51:27.248 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.080190206999999999 2024-09-21T13:51:27.249 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.249 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.249 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:51:27.249 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.564332+0000", 2024-09-21T13:51:27.249 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6137999999999995e-05 2024-09-21T13:51:27.249 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.249 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.249 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:51:27.249 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.564434+0000", 2024-09-21T13:51:27.249 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010199100000000001 2024-09-21T13:51:27.249 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:27.249 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:51:27.249 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:27.249 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.250 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.250 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:73184 7.e 7:7c9d6f07:::benchmark_data_smithi049_76798_object73183:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-09-21T13:51:27.250 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:54.903333+0000", 2024-09-21T13:51:27.250 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 32.283863312999998, 2024-09-21T13:51:27.250 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.050502233000000001, 2024-09-21T13:51:27.250 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:51:27.250 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:51:27.250 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:27.250 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:51:27.250 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:51:27.250 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:27.250 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 73184 2024-09-21T13:51:27.250 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.250 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:51:27.251 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.251 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:51:27.251 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.903333+0000", 2024-09-21T13:51:27.251 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:27.251 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.251 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.251 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:51:27.251 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.903333+0000", 2024-09-21T13:51:27.251 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:27.251 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.251 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.251 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:51:27.251 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.903334+0000", 2024-09-21T13:51:27.251 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.815e-06 2024-09-21T13:51:27.252 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.252 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.252 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:51:27.252 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.903368+0000", 2024-09-21T13:51:27.252 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3732000000000002e-05 2024-09-21T13:51:27.252 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.252 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.252 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:51:27.252 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.903386+0000", 2024-09-21T13:51:27.252 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7499e-05 2024-09-21T13:51:27.252 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.252 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.252 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:51:27.252 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.903393+0000", 2024-09-21T13:51:27.253 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6899999999999992e-06 2024-09-21T13:51:27.253 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.253 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.253 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:51:27.253 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.903449+0000", 2024-09-21T13:51:27.253 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5692000000000001e-05 2024-09-21T13:51:27.253 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.253 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.253 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:51:27.253 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.903576+0000", 2024-09-21T13:51:27.253 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012695100000000001 2024-09-21T13:51:27.253 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.253 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.253 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0", 2024-09-21T13:51:27.254 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.903728+0000", 2024-09-21T13:51:27.254 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015220500000000001 2024-09-21T13:51:27.254 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.254 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.254 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:51:27.254 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.906140+0000", 2024-09-21T13:51:27.254 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0024115600000000001 2024-09-21T13:51:27.254 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.254 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.254 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:27.254 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.953728+0000", 2024-09-21T13:51:27.254 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.047588295000000003 2024-09-21T13:51:27.254 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.255 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.255 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:51:27.255 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.953768+0000", 2024-09-21T13:51:27.255 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9932999999999999e-05 2024-09-21T13:51:27.255 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.255 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.255 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:51:27.255 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.953835+0000", 2024-09-21T13:51:27.255 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6860999999999996e-05 2024-09-21T13:51:27.255 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:27.256 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:51:27.256 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:27.256 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.256 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.256 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:82865 7.0 7:0b2d2443:::benchmark_data_smithi049_76798_object82864:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-21T13:51:27.256 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:51:01.424453+0000", 2024-09-21T13:51:27.256 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 25.762742631999998, 2024-09-21T13:51:27.256 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.064463224, 2024-09-21T13:51:27.256 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:51:27.256 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:51:27.256 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:27.256 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:51:27.256 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:51:27.257 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:27.257 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 82865 2024-09-21T13:51:27.257 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.257 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:51:27.257 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.257 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:51:27.257 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:01.424453+0000", 2024-09-21T13:51:27.257 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:27.257 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.257 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.257 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:51:27.257 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:01.424453+0000", 2024-09-21T13:51:27.257 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:27.257 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.258 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.258 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:51:27.258 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:01.424461+0000", 2024-09-21T13:51:27.258 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3420000000000002e-06 2024-09-21T13:51:27.258 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.258 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.258 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:51:27.258 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:01.424502+0000", 2024-09-21T13:51:27.258 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0936000000000002e-05 2024-09-21T13:51:27.258 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.258 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.258 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:51:27.258 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:01.424514+0000", 2024-09-21T13:51:27.259 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2500999999999999e-05 2024-09-21T13:51:27.259 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.259 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.259 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:51:27.259 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:01.424521+0000", 2024-09-21T13:51:27.259 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2640000000000002e-06 2024-09-21T13:51:27.259 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.259 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.259 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:51:27.259 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:01.424574+0000", 2024-09-21T13:51:27.259 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3115999999999998e-05 2024-09-21T13:51:27.259 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.259 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.259 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:51:27.259 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:01.424692+0000", 2024-09-21T13:51:27.260 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011772700000000001 2024-09-21T13:51:27.260 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.260 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.260 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3", 2024-09-21T13:51:27.260 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:01.424812+0000", 2024-09-21T13:51:27.260 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011950700000000001 2024-09-21T13:51:27.260 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.260 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.260 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:27.260 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:01.427476+0000", 2024-09-21T13:51:27.260 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0026641619999999999 2024-09-21T13:51:27.260 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.260 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.261 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:51:27.261 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:01.433321+0000", 2024-09-21T13:51:27.261 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0058450610000000004 2024-09-21T13:51:27.261 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.261 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.261 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:27.261 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:01.488824+0000", 2024-09-21T13:51:27.261 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.055502978000000001 2024-09-21T13:51:27.261 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.261 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.261 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:51:27.261 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:01.488874+0000", 2024-09-21T13:51:27.261 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0470000000000003e-05 2024-09-21T13:51:27.261 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.262 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.262 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:51:27.262 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:01.488916+0000", 2024-09-21T13:51:27.262 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2160000000000003e-05 2024-09-21T13:51:27.262 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:27.262 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:51:27.262 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:27.262 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.262 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.262 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:82866 7.0 7:0b0b6fd8:::benchmark_data_smithi049_76798_object82865:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-21T13:51:27.262 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:51:01.427444+0000", 2024-09-21T13:51:27.262 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 25.759751886, 2024-09-21T13:51:27.262 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.066806426000000002, 2024-09-21T13:51:27.263 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:51:27.263 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:51:27.263 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:27.263 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:51:27.263 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:51:27.263 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:27.263 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 82866 2024-09-21T13:51:27.263 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.263 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:51:27.263 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.263 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:51:27.263 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:01.427444+0000", 2024-09-21T13:51:27.263 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:27.263 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.264 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.264 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:51:27.264 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:01.427444+0000", 2024-09-21T13:51:27.264 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:27.264 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.264 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.264 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:51:27.264 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:01.427446+0000", 2024-09-21T13:51:27.264 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5489999999999999e-06 2024-09-21T13:51:27.264 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.264 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.264 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:51:27.264 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:01.427477+0000", 2024-09-21T13:51:27.264 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1489999999999998e-05 2024-09-21T13:51:27.265 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.265 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.265 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:51:27.265 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:01.427488+0000", 2024-09-21T13:51:27.265 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0676000000000001e-05 2024-09-21T13:51:27.265 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.265 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.265 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:51:27.265 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:01.427494+0000", 2024-09-21T13:51:27.265 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2280000000000003e-06 2024-09-21T13:51:27.265 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.265 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.265 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:51:27.265 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:01.427566+0000", 2024-09-21T13:51:27.266 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2298000000000005e-05 2024-09-21T13:51:27.266 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.266 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.266 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:51:27.266 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:01.427693+0000", 2024-09-21T13:51:27.266 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012639699999999999 2024-09-21T13:51:27.266 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.266 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.266 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3", 2024-09-21T13:51:27.266 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:01.427833+0000", 2024-09-21T13:51:27.266 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000139937 2024-09-21T13:51:27.266 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.266 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.266 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:51:27.267 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:01.433326+0000", 2024-09-21T13:51:27.267 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0054930719999999999 2024-09-21T13:51:27.267 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.267 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.267 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:27.267 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:01.433361+0000", 2024-09-21T13:51:27.267 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5417000000000003e-05 2024-09-21T13:51:27.267 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.267 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.267 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:27.267 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:01.494153+0000", 2024-09-21T13:51:27.267 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.060791507000000002 2024-09-21T13:51:27.267 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.267 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.267 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:51:27.268 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:01.494205+0000", 2024-09-21T13:51:27.268 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2753000000000001e-05 2024-09-21T13:51:27.268 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.268 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.268 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:51:27.268 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:01.494250+0000", 2024-09-21T13:51:27.268 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5102000000000003e-05 2024-09-21T13:51:27.268 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:27.268 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:51:27.268 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:27.268 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.268 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.268 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:82867 7.0 7:0f3b154d:::benchmark_data_smithi049_76798_object82866:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-21T13:51:27.268 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:51:01.427520+0000", 2024-09-21T13:51:27.269 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 25.759676285000001, 2024-09-21T13:51:27.269 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.066892863999999996, 2024-09-21T13:51:27.269 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:51:27.269 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:51:27.269 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:27.269 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:51:27.269 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:51:27.269 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:27.269 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 82867 2024-09-21T13:51:27.269 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.269 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:51:27.269 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.269 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:51:27.269 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:01.427520+0000", 2024-09-21T13:51:27.269 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:27.270 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.270 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.270 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:51:27.270 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:01.427520+0000", 2024-09-21T13:51:27.270 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:27.270 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.270 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.270 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:51:27.270 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:01.427520+0000", 2024-09-21T13:51:27.270 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8800000000000001e-07 2024-09-21T13:51:27.270 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.270 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.270 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:51:27.270 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:01.427551+0000", 2024-09-21T13:51:27.271 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0491e-05 2024-09-21T13:51:27.271 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.271 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.271 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:51:27.271 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:01.427558+0000", 2024-09-21T13:51:27.271 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3039999999999999e-06 2024-09-21T13:51:27.271 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.271 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.271 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:51:27.271 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:01.427563+0000", 2024-09-21T13:51:27.271 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1050000000000003e-06 2024-09-21T13:51:27.271 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.271 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.271 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:51:27.271 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:01.430373+0000", 2024-09-21T13:51:27.272 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.002809509 2024-09-21T13:51:27.272 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.272 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.272 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:51:27.272 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:01.430498+0000", 2024-09-21T13:51:27.272 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012471300000000001 2024-09-21T13:51:27.272 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.272 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.272 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3", 2024-09-21T13:51:27.272 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:01.430635+0000", 2024-09-21T13:51:27.272 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000136904 2024-09-21T13:51:27.272 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.272 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.272 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:27.272 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:01.433541+0000", 2024-09-21T13:51:27.273 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0029065409999999999 2024-09-21T13:51:27.273 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.273 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.273 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:51:27.273 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:01.433786+0000", 2024-09-21T13:51:27.273 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000245324 2024-09-21T13:51:27.273 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.273 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.273 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:27.273 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:01.494320+0000", 2024-09-21T13:51:27.273 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.060533837 2024-09-21T13:51:27.273 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.273 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.273 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:51:27.273 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:01.494371+0000", 2024-09-21T13:51:27.274 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1047000000000001e-05 2024-09-21T13:51:27.274 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.274 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.274 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:51:27.274 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:01.494412+0000", 2024-09-21T13:51:27.274 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1201000000000001e-05 2024-09-21T13:51:27.274 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:27.274 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:51:27.274 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:27.274 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.274 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.274 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:84068 7.0 7:0621d917:::benchmark_data_smithi049_76798_object84067:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-21T13:51:27.274 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:51:02.324154+0000", 2024-09-21T13:51:27.274 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 24.863041892999998, 2024-09-21T13:51:27.275 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.050004989, 2024-09-21T13:51:27.275 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:51:27.275 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:51:27.275 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:27.275 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:51:27.275 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:51:27.275 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:27.275 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 84068 2024-09-21T13:51:27.275 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.275 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:51:27.275 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.275 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:51:27.275 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:02.324154+0000", 2024-09-21T13:51:27.275 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:27.275 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.276 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.276 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:51:27.276 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:02.324154+0000", 2024-09-21T13:51:27.276 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:27.276 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.276 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.276 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:51:27.276 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:02.324156+0000", 2024-09-21T13:51:27.276 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.886e-06 2024-09-21T13:51:27.276 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.276 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.276 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:51:27.276 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:02.324206+0000", 2024-09-21T13:51:27.276 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0050999999999999e-05 2024-09-21T13:51:27.277 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.277 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.277 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:51:27.277 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:02.324217+0000", 2024-09-21T13:51:27.277 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0763e-05 2024-09-21T13:51:27.277 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.277 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.277 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:51:27.277 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:02.324224+0000", 2024-09-21T13:51:27.277 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.486e-06 2024-09-21T13:51:27.277 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.277 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.277 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:51:27.277 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:02.324427+0000", 2024-09-21T13:51:27.278 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000202807 2024-09-21T13:51:27.278 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.278 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.278 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:51:27.278 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:02.324555+0000", 2024-09-21T13:51:27.278 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012768100000000001 2024-09-21T13:51:27.278 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.278 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.278 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3", 2024-09-21T13:51:27.278 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:02.324643+0000", 2024-09-21T13:51:27.278 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8195000000000005e-05 2024-09-21T13:51:27.278 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.278 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.279 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:51:27.279 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:02.329880+0000", 2024-09-21T13:51:27.279 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0052373740000000004 2024-09-21T13:51:27.279 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.279 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.279 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:27.279 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:02.332588+0000", 2024-09-21T13:51:27.279 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.002707615 2024-09-21T13:51:27.279 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.279 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.279 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:27.279 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:02.374069+0000", 2024-09-21T13:51:27.279 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.041480992000000001 2024-09-21T13:51:27.280 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.280 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.280 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:51:27.280 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:02.374113+0000", 2024-09-21T13:51:27.280 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4474999999999997e-05 2024-09-21T13:51:27.280 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.280 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.280 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:51:27.280 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:02.374159+0000", 2024-09-21T13:51:27.280 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5664000000000002e-05 2024-09-21T13:51:27.280 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:27.280 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:51:27.280 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:27.280 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.281 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.281 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:84074 7.0 7:0854da3a:::benchmark_data_smithi049_76798_object84073:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-21T13:51:27.281 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:51:02.327026+0000", 2024-09-21T13:51:27.281 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 24.860169559999999, 2024-09-21T13:51:27.281 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.047289862000000002, 2024-09-21T13:51:27.281 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:51:27.281 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:51:27.281 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:27.281 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:51:27.281 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:51:27.281 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:27.281 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 84074 2024-09-21T13:51:27.282 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.282 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:51:27.282 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.282 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:51:27.282 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:02.327026+0000", 2024-09-21T13:51:27.282 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:27.282 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.282 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.282 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:51:27.282 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:02.327026+0000", 2024-09-21T13:51:27.282 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:27.282 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.282 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.282 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:51:27.283 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:02.327028+0000", 2024-09-21T13:51:27.283 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.265e-06 2024-09-21T13:51:27.283 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.283 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.283 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:51:27.283 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:02.327064+0000", 2024-09-21T13:51:27.283 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6505999999999999e-05 2024-09-21T13:51:27.283 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.283 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.283 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:51:27.283 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:02.327077+0000", 2024-09-21T13:51:27.283 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2551000000000001e-05 2024-09-21T13:51:27.284 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.284 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.284 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:51:27.284 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:02.327083+0000", 2024-09-21T13:51:27.284 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7900000000000002e-06 2024-09-21T13:51:27.284 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.284 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.284 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:51:27.284 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:02.327148+0000", 2024-09-21T13:51:27.284 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4953e-05 2024-09-21T13:51:27.284 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.284 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.284 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:51:27.284 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:02.327259+0000", 2024-09-21T13:51:27.284 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000110215 2024-09-21T13:51:27.284 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.285 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.285 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3", 2024-09-21T13:51:27.285 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:02.327382+0000", 2024-09-21T13:51:27.285 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012318799999999999 2024-09-21T13:51:27.285 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.285 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.285 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:51:27.285 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:02.330196+0000", 2024-09-21T13:51:27.285 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0028143199999999999 2024-09-21T13:51:27.285 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.285 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.285 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:27.285 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:02.332768+0000", 2024-09-21T13:51:27.285 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.002572122 2024-09-21T13:51:27.285 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.285 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.286 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:27.286 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:02.374223+0000", 2024-09-21T13:51:27.286 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.041455082999999997 2024-09-21T13:51:27.286 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.286 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.286 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:51:27.286 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:02.374278+0000", 2024-09-21T13:51:27.286 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4548e-05 2024-09-21T13:51:27.286 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.286 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.286 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:51:27.286 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:02.374316+0000", 2024-09-21T13:51:27.286 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8321000000000001e-05 2024-09-21T13:51:27.286 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:27.287 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:51:27.287 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:27.287 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.287 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.287 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:85258 7.0 7:008db81d:::benchmark_data_smithi049_76798_object85257:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-21T13:51:27.287 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:51:03.152770+0000", 2024-09-21T13:51:27.287 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 24.03442575, 2024-09-21T13:51:27.287 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.060843383000000001, 2024-09-21T13:51:27.287 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:51:27.287 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:51:27.287 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:27.287 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:51:27.287 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:51:27.287 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:27.287 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 85258 2024-09-21T13:51:27.288 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.288 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:51:27.288 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.288 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:51:27.288 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:03.152770+0000", 2024-09-21T13:51:27.288 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:27.288 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.288 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.288 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:51:27.288 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:03.152770+0000", 2024-09-21T13:51:27.288 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:27.288 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.288 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.288 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:51:27.289 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:03.152772+0000", 2024-09-21T13:51:27.289 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4160000000000001e-06 2024-09-21T13:51:27.289 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.289 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.289 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:51:27.289 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:03.152829+0000", 2024-09-21T13:51:27.289 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7726000000000003e-05 2024-09-21T13:51:27.289 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.289 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.289 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:51:27.289 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:03.152851+0000", 2024-09-21T13:51:27.289 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2022999999999999e-05 2024-09-21T13:51:27.289 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.289 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.290 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:51:27.290 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:03.152859+0000", 2024-09-21T13:51:27.290 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0199999999999994e-06 2024-09-21T13:51:27.290 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.290 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.290 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:51:27.290 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:03.152908+0000", 2024-09-21T13:51:27.290 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8850000000000002e-05 2024-09-21T13:51:27.290 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.290 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.290 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:51:27.290 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:03.153021+0000", 2024-09-21T13:51:27.290 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000113144 2024-09-21T13:51:27.290 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.291 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.291 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3", 2024-09-21T13:51:27.291 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:03.153142+0000", 2024-09-21T13:51:27.291 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000120988 2024-09-21T13:51:27.291 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.291 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.291 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:51:27.291 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:03.157119+0000", 2024-09-21T13:51:27.291 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0039769999999999996 2024-09-21T13:51:27.291 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.291 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.291 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:27.291 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:03.158170+0000", 2024-09-21T13:51:27.291 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0010505009999999999 2024-09-21T13:51:27.291 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.292 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.292 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:27.292 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:03.213514+0000", 2024-09-21T13:51:27.292 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.055344653000000001 2024-09-21T13:51:27.292 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.292 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.292 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:51:27.292 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:03.213569+0000", 2024-09-21T13:51:27.292 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.482e-05 2024-09-21T13:51:27.292 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.292 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.292 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:51:27.292 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:03.213614+0000", 2024-09-21T13:51:27.292 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4242000000000001e-05 2024-09-21T13:51:27.292 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:27.293 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:51:27.293 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:27.293 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.293 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.293 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:85274 7.0 7:082e50df:::benchmark_data_smithi049_76798_object85273:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-21T13:51:27.293 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:51:03.165302+0000", 2024-09-21T13:51:27.293 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 24.021893411000001, 2024-09-21T13:51:27.293 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.048795023, 2024-09-21T13:51:27.293 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:51:27.293 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:51:27.293 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:27.293 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:51:27.293 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:51:27.294 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:27.294 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 85274 2024-09-21T13:51:27.294 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.294 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:51:27.294 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.294 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:51:27.294 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:03.165302+0000", 2024-09-21T13:51:27.294 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:27.294 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.294 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.294 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:51:27.294 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:03.165302+0000", 2024-09-21T13:51:27.294 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:27.294 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.295 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.295 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:51:27.295 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:03.165304+0000", 2024-09-21T13:51:27.295 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6899999999999999e-06 2024-09-21T13:51:27.295 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.295 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.295 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:51:27.295 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:03.165360+0000", 2024-09-21T13:51:27.295 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5714000000000002e-05 2024-09-21T13:51:27.295 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.295 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.295 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:51:27.295 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:03.165377+0000", 2024-09-21T13:51:27.295 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7363999999999998e-05 2024-09-21T13:51:27.295 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.296 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.296 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:51:27.296 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:03.165385+0000", 2024-09-21T13:51:27.296 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.396e-06 2024-09-21T13:51:27.296 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.296 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.296 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:51:27.296 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:03.165461+0000", 2024-09-21T13:51:27.296 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6847000000000005e-05 2024-09-21T13:51:27.296 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.296 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.296 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:51:27.296 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:03.165607+0000", 2024-09-21T13:51:27.296 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014575499999999999 2024-09-21T13:51:27.296 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.297 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.297 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3", 2024-09-21T13:51:27.297 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:03.165751+0000", 2024-09-21T13:51:27.297 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000143953 2024-09-21T13:51:27.297 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.297 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.297 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:51:27.297 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:03.167971+0000", 2024-09-21T13:51:27.297 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0022197570000000002 2024-09-21T13:51:27.297 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.297 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.297 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:27.297 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:03.168643+0000", 2024-09-21T13:51:27.297 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00067230999999999996 2024-09-21T13:51:27.297 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.298 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.298 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:27.298 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:03.213962+0000", 2024-09-21T13:51:27.298 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.045319217000000002 2024-09-21T13:51:27.298 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.298 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.298 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:51:27.298 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:03.214061+0000", 2024-09-21T13:51:27.298 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.8301000000000006e-05 2024-09-21T13:51:27.298 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.298 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.298 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:51:27.298 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:03.214098+0000", 2024-09-21T13:51:27.299 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6718999999999999e-05 2024-09-21T13:51:27.299 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:27.299 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:51:27.299 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:27.299 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.299 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.299 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:100014 7.0 7:09685671:::benchmark_data_smithi049_76798_object100013:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-09-21T13:51:27.299 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:51:14.134499+0000", 2024-09-21T13:51:27.299 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 13.052696707999999, 2024-09-21T13:51:27.299 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.065503151999999995, 2024-09-21T13:51:27.299 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:51:27.299 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:51:27.299 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:27.299 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:51:27.300 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:51:27.300 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:27.300 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 100014 2024-09-21T13:51:27.300 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.300 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:51:27.300 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.300 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:51:27.300 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:14.134499+0000", 2024-09-21T13:51:27.300 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:27.300 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.300 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.300 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:51:27.300 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:14.134499+0000", 2024-09-21T13:51:27.300 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:27.301 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.301 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.301 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:51:27.301 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:14.134502+0000", 2024-09-21T13:51:27.301 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.407e-06 2024-09-21T13:51:27.301 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.301 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.301 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:51:27.301 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:14.134560+0000", 2024-09-21T13:51:27.301 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.859e-05 2024-09-21T13:51:27.301 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.301 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.301 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:51:27.301 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:14.134572+0000", 2024-09-21T13:51:27.302 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2105e-05 2024-09-21T13:51:27.302 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.302 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.302 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:51:27.302 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:14.134580+0000", 2024-09-21T13:51:27.302 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7589999999999997e-06 2024-09-21T13:51:27.302 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.302 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.302 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:51:27.302 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:14.134637+0000", 2024-09-21T13:51:27.302 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6805000000000002e-05 2024-09-21T13:51:27.302 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.302 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.302 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:51:27.303 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:14.134756+0000", 2024-09-21T13:51:27.303 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000119638 2024-09-21T13:51:27.303 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.303 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.303 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3", 2024-09-21T13:51:27.303 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:14.134895+0000", 2024-09-21T13:51:27.303 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013888799999999999 2024-09-21T13:51:27.303 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.303 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.303 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:51:27.303 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:14.137738+0000", 2024-09-21T13:51:27.303 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0028429610000000002 2024-09-21T13:51:27.303 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.303 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.303 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:27.304 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:14.138739+0000", 2024-09-21T13:51:27.304 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0010004530000000001 2024-09-21T13:51:27.304 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.304 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.304 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:27.304 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:14.199817+0000", 2024-09-21T13:51:27.304 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.061078501 2024-09-21T13:51:27.304 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.304 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.304 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:51:27.304 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:14.199900+0000", 2024-09-21T13:51:27.304 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3007e-05 2024-09-21T13:51:27.304 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.304 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.305 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:51:27.305 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:14.200002+0000", 2024-09-21T13:51:27.305 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000102038 2024-09-21T13:51:27.305 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:27.305 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:51:27.305 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:27.305 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.305 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.305 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:104985 7.0 7:0b4cce6e:::benchmark_data_smithi049_76798_object104984:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-21T13:51:27.305 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:51:18.090881+0000", 2024-09-21T13:51:27.305 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 9.0963143950000003, 2024-09-21T13:51:27.305 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.068427587999999998, 2024-09-21T13:51:27.305 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:51:27.305 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:51:27.305 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:27.306 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:51:27.306 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:51:27.306 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:27.306 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 104985 2024-09-21T13:51:27.306 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.306 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:51:27.306 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.306 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:51:27.306 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:18.090881+0000", 2024-09-21T13:51:27.306 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:27.306 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.306 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.306 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:51:27.306 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:18.090881+0000", 2024-09-21T13:51:27.307 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:27.307 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.307 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.307 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:51:27.307 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:18.090882+0000", 2024-09-21T13:51:27.307 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6700000000000002e-07 2024-09-21T13:51:27.307 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.307 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.307 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:51:27.307 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:18.090908+0000", 2024-09-21T13:51:27.307 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5945999999999999e-05 2024-09-21T13:51:27.307 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.307 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.308 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:51:27.308 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:18.090920+0000", 2024-09-21T13:51:27.308 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1459e-05 2024-09-21T13:51:27.308 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.308 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.308 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:51:27.308 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:18.090927+0000", 2024-09-21T13:51:27.308 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9840000000000004e-06 2024-09-21T13:51:27.308 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.308 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.308 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:51:27.308 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:18.090976+0000", 2024-09-21T13:51:27.308 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9264000000000001e-05 2024-09-21T13:51:27.308 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.309 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.309 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:51:27.309 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:18.091089+0000", 2024-09-21T13:51:27.309 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000113045 2024-09-21T13:51:27.309 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.309 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.309 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3", 2024-09-21T13:51:27.309 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:18.091215+0000", 2024-09-21T13:51:27.309 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000125387 2024-09-21T13:51:27.309 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.309 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.309 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:27.309 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:18.094008+0000", 2024-09-21T13:51:27.309 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0027934840000000002 2024-09-21T13:51:27.310 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.310 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.310 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:51:27.310 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:18.094029+0000", 2024-09-21T13:51:27.310 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0865000000000001e-05 2024-09-21T13:51:27.310 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.310 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.310 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:27.310 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:18.159176+0000", 2024-09-21T13:51:27.310 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.065146974999999996 2024-09-21T13:51:27.310 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.310 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.310 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:51:27.310 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:18.159266+0000", 2024-09-21T13:51:27.310 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.9976999999999996e-05 2024-09-21T13:51:27.311 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.311 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.311 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:51:27.311 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:18.159309+0000", 2024-09-21T13:51:27.311 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3235000000000002e-05 2024-09-21T13:51:27.311 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:27.311 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:51:27.311 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:27.311 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.311 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.311 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:104992 7.0 7:04345daf:::benchmark_data_smithi049_76798_object104991:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-21T13:51:27.311 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:51:18.101240+0000", 2024-09-21T13:51:27.311 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 9.0859558160000002, 2024-09-21T13:51:27.311 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.060974961000000001, 2024-09-21T13:51:27.312 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:51:27.312 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:51:27.312 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:27.312 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:51:27.312 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:51:27.312 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:27.312 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 104992 2024-09-21T13:51:27.312 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.312 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:51:27.312 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.312 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:51:27.312 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:18.101240+0000", 2024-09-21T13:51:27.312 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:27.312 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.312 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.313 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:51:27.313 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:18.101240+0000", 2024-09-21T13:51:27.313 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:27.313 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.313 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.313 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:51:27.313 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:18.101245+0000", 2024-09-21T13:51:27.313 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7770000000000002e-06 2024-09-21T13:51:27.313 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.313 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.313 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:51:27.313 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:18.101291+0000", 2024-09-21T13:51:27.313 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5757000000000003e-05 2024-09-21T13:51:27.314 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.314 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.314 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:51:27.314 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:18.101301+0000", 2024-09-21T13:51:27.314 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0284e-05 2024-09-21T13:51:27.314 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.314 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.314 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:51:27.314 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:18.101307+0000", 2024-09-21T13:51:27.314 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8719999999999999e-06 2024-09-21T13:51:27.314 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.314 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.314 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:51:27.314 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:18.101346+0000", 2024-09-21T13:51:27.314 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9405e-05 2024-09-21T13:51:27.315 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.315 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.315 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:51:27.315 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:18.101434+0000", 2024-09-21T13:51:27.315 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7676999999999994e-05 2024-09-21T13:51:27.315 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.315 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.315 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3", 2024-09-21T13:51:27.315 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:18.101523+0000", 2024-09-21T13:51:27.315 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.9491000000000001e-05 2024-09-21T13:51:27.315 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.315 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.315 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:51:27.316 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:18.103600+0000", 2024-09-21T13:51:27.316 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0020768499999999999 2024-09-21T13:51:27.316 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.316 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.316 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:27.316 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:18.104730+0000", 2024-09-21T13:51:27.316 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001129971 2024-09-21T13:51:27.316 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.316 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.316 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:27.316 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:18.162077+0000", 2024-09-21T13:51:27.316 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.057346803000000002 2024-09-21T13:51:27.316 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.316 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.317 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:51:27.317 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:18.162168+0000", 2024-09-21T13:51:27.317 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0796000000000003e-05 2024-09-21T13:51:27.317 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:27.317 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:27.317 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:51:27.317 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:18.162215+0000", 2024-09-21T13:51:27.317 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7277999999999997e-05 2024-09-21T13:51:27.317 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:27.317 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:51:27.317 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:27.317 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:27.317 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:51:27.317 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T13:51:27.344 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:27.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:27.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:27.446 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:51:27.473 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:27.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:27.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:51:27.504 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: [21/Sep/2024:13:51:27] ENGINE Bus STOPPING 2024-09-21T13:51:27.575 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:27.576 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -f /var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/exp.3.1f.2 2024-09-21T13:51:27.610 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:27.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:27.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:27.712 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:51:27.751 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:27.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:27.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:51:27.822 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-mon[40489]: pgmap v102: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 6.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.57k op/s; 41800/335595 objects degraded (12.455%) 2024-09-21T13:51:27.823 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-mon[40489]: 5.9 scrub starts 2024-09-21T13:51:27.823 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-mon[40489]: 5.9 scrub ok 2024-09-21T13:51:27.823 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-mon[40489]: 6.a scrub starts 2024-09-21T13:51:27.823 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-mon[40489]: 6.a scrub ok 2024-09-21T13:51:27.823 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:51:27.823 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:51:27.823 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "dashboard get-alertmanager-api-host"} : dispatch 2024-09-21T13:51:27.823 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-mon[40489]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T13:51:27.823 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi049.front.sepia.ceph.com:9093"} : dispatch 2024-09-21T13:51:27.823 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-mon[40489]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi049.front.sepia.ceph.com:9093"}]: dispatch 2024-09-21T13:51:27.823 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:51:27.823 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "dashboard get-prometheus-api-host"} : dispatch 2024-09-21T13:51:27.824 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-mon[40489]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T13:51:27.824 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi187.front.sepia.ceph.com:9095"} : dispatch 2024-09-21T13:51:27.824 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-mon[40489]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi187.front.sepia.ceph.com:9095"}]: dispatch 2024-09-21T13:51:27.824 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:51:27.824 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "dashboard get-grafana-api-url"} : dispatch 2024-09-21T13:51:27.824 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-mon[40489]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T13:51:27.824 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-url", "value": "https://smithi187.front.sepia.ceph.com:3000"} : dispatch 2024-09-21T13:51:27.824 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-mon[40489]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi187.front.sepia.ceph.com:3000"}]: dispatch 2024-09-21T13:51:27.824 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:51:27.824 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T13:51:27.824 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T13:51:27.824 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:51:27.824 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-mon[30080]: pgmap v102: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 6.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.57k op/s; 41800/335595 objects degraded (12.455%) 2024-09-21T13:51:27.825 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-mon[30080]: 5.9 scrub starts 2024-09-21T13:51:27.825 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-mon[30080]: 5.9 scrub ok 2024-09-21T13:51:27.825 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-mon[30080]: 6.a scrub starts 2024-09-21T13:51:27.825 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-mon[30080]: 6.a scrub ok 2024-09-21T13:51:27.825 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:51:27.825 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:51:27.825 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "dashboard get-alertmanager-api-host"} : dispatch 2024-09-21T13:51:27.825 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-mon[30080]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T13:51:27.825 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi049.front.sepia.ceph.com:9093"} : dispatch 2024-09-21T13:51:27.825 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-mon[30080]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi049.front.sepia.ceph.com:9093"}]: dispatch 2024-09-21T13:51:27.825 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:51:27.825 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "dashboard get-prometheus-api-host"} : dispatch 2024-09-21T13:51:27.825 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-mon[30080]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T13:51:27.825 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi187.front.sepia.ceph.com:9095"} : dispatch 2024-09-21T13:51:27.826 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-mon[30080]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi187.front.sepia.ceph.com:9095"}]: dispatch 2024-09-21T13:51:27.826 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:51:27.826 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "dashboard get-grafana-api-url"} : dispatch 2024-09-21T13:51:27.826 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-mon[30080]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T13:51:27.826 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-url", "value": "https://smithi187.front.sepia.ceph.com:3000"} : dispatch 2024-09-21T13:51:27.826 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-mon[30080]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi187.front.sepia.ceph.com:3000"}]: dispatch 2024-09-21T13:51:27.826 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:51:27.826 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T13:51:27.826 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T13:51:27.826 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:51:27.826 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: [21/Sep/2024:13:51:27] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T13:51:27.826 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: [21/Sep/2024:13:51:27] ENGINE Bus STOPPED 2024-09-21T13:51:27.827 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: [21/Sep/2024:13:51:27] ENGINE Bus STARTING 2024-09-21T13:51:27.827 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: [21/Sep/2024:13:51:27] ENGINE Serving on http://:::9283 2024-09-21T13:51:27.827 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:51:27 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: [21/Sep/2024:13:51:27] ENGINE Bus STARTED 2024-09-21T13:51:27.852 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:27.884 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:27.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:27.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:27.917 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 75 16 117290 117274 97.695 71.8125 0.00423316 0.0102314 2024-09-21T13:51:27.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:27 smithi187 ceph-mon[33032]: pgmap v102: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 6.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.57k op/s; 41800/335595 objects degraded (12.455%) 2024-09-21T13:51:27.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:27 smithi187 ceph-mon[33032]: 5.9 scrub starts 2024-09-21T13:51:27.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:27 smithi187 ceph-mon[33032]: 5.9 scrub ok 2024-09-21T13:51:27.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:27 smithi187 ceph-mon[33032]: 6.a scrub starts 2024-09-21T13:51:27.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:27 smithi187 ceph-mon[33032]: 6.a scrub ok 2024-09-21T13:51:27.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:27 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:51:27.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:27 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:51:27.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:27 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "dashboard get-alertmanager-api-host"} : dispatch 2024-09-21T13:51:27.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:27 smithi187 ceph-mon[33032]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T13:51:27.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:27 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi049.front.sepia.ceph.com:9093"} : dispatch 2024-09-21T13:51:27.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:27 smithi187 ceph-mon[33032]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi049.front.sepia.ceph.com:9093"}]: dispatch 2024-09-21T13:51:27.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:27 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:51:27.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:27 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "dashboard get-prometheus-api-host"} : dispatch 2024-09-21T13:51:27.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:27 smithi187 ceph-mon[33032]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T13:51:27.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:27 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi187.front.sepia.ceph.com:9095"} : dispatch 2024-09-21T13:51:27.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:27 smithi187 ceph-mon[33032]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi187.front.sepia.ceph.com:9095"}]: dispatch 2024-09-21T13:51:27.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:27 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:51:27.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:27 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "dashboard get-grafana-api-url"} : dispatch 2024-09-21T13:51:27.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:27 smithi187 ceph-mon[33032]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T13:51:27.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:27 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "dashboard set-grafana-api-url", "value": "https://smithi187.front.sepia.ceph.com:3000"} : dispatch 2024-09-21T13:51:27.947 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:27 smithi187 ceph-mon[33032]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi187.front.sepia.ceph.com:3000"}]: dispatch 2024-09-21T13:51:27.947 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:27 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:51:27.947 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:27 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T13:51:27.947 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:27 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T13:51:27.947 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:27 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:51:27.985 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:51:28.042 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:28.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:28.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:51:28.143 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:28.173 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:28.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:28.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:28.221 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T13:51:28.276 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:51:28.308 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:28.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:28.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:51:28.410 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:51:28.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:28.451 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:28.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:51:28.552 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:51:28.595 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:28.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:28.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:51:28.604 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-09-21T13:51:28.697 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:51:28.728 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:28.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:28.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:51:28.830 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:28.864 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:28.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:28.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:28.917 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 76 16 118603 118587 97.4892 82.0625 0.00887066 0.010253 2024-09-21T13:51:28.939 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:28 smithi187 ceph-mon[33032]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T13:51:28.940 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:28 smithi187 ceph-mon[33032]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T13:51:28.940 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:28 smithi187 ceph-mon[33032]: 3.1d scrub starts 2024-09-21T13:51:28.940 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:28 smithi187 ceph-mon[33032]: 3.1d scrub ok 2024-09-21T13:51:28.940 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:28 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2912739289' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:51:28.940 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:28 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:51:28.940 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:28 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T13:51:28.940 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:28 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:51:28.940 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:28 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:51:28.966 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:51:28.970 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T13:51:29.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:28 smithi049 ceph-mon[30080]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T13:51:29.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:28 smithi049 ceph-mon[30080]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T13:51:29.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:28 smithi049 ceph-mon[30080]: 3.1d scrub starts 2024-09-21T13:51:29.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:28 smithi049 ceph-mon[30080]: 3.1d scrub ok 2024-09-21T13:51:29.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:28 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2912739289' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:51:29.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:28 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:51:29.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:28 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T13:51:29.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:28 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:51:29.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:28 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:51:29.007 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:28 smithi049 ceph-mon[40489]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T13:51:29.007 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:28 smithi049 ceph-mon[40489]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T13:51:29.007 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:28 smithi049 ceph-mon[40489]: 3.1d scrub starts 2024-09-21T13:51:29.007 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:28 smithi049 ceph-mon[40489]: 3.1d scrub ok 2024-09-21T13:51:29.007 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:28 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2912739289' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:51:29.007 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:28 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:51:29.007 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:28 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T13:51:29.007 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:28 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:51:29.007 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:28 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:51:29.008 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:29.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:29.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:51:29.113 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:51:29.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:29.167 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:29.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:51:29.194 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:51:29 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:51:29.268 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:51:29.311 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:51:29.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:29.315 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:29.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:51:29.416 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:29.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:29.454 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:29.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:29.554 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:51:29.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:29.581 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:29.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:51:29.682 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:29.713 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:29.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:29.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:29.787 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:29 smithi187 ceph-mon[33032]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T13:51:29.787 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:29 smithi187 ceph-mon[33032]: pgmap v103: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 6.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 49 MiB/s wr, 792 op/s; 41800/335595 objects degraded (12.455%) 2024-09-21T13:51:29.787 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:29 smithi187 ceph-mon[33032]: 4.15 scrub starts 2024-09-21T13:51:29.787 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:29 smithi187 ceph-mon[33032]: 4.15 scrub ok 2024-09-21T13:51:29.787 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:29 smithi187 ceph-mon[33032]: 3.b scrub starts 2024-09-21T13:51:29.788 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:29 smithi187 ceph-mon[33032]: 3.b scrub ok 2024-09-21T13:51:29.788 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:29 smithi187 ceph-mon[33032]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:51:29.788 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:29 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T13:51:29.788 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:29 smithi187 ceph-mon[33032]: osdmap e115: 8 total, 7 up, 8 in 2024-09-21T13:51:29.810 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:51:29.815 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:51:29.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:29.885 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:29.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:51:29.917 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 77 16 119855 119839 97.2393 78.25 0.0128554 0.0102791 2024-09-21T13:51:29.930 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T13:51:29.930 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-21T13:51:29.930 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:29.930 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:119858 7.c 7:317ad955:::benchmark_data_smithi049_76798_object119857:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-21T13:51:29.930 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:51:29.920679+0000", 2024-09-21T13:51:29.930 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0085489629999999997, 2024-09-21T13:51:29.930 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0085836220000000008, 2024-09-21T13:51:29.931 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:51:29.931 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:51:29.931 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-21T13:51:29.931 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:51:29.931 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:51:29.931 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:29.931 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 119858 2024-09-21T13:51:29.931 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:29.931 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:51:29.931 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:29.931 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:51:29.931 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:29.920679+0000", 2024-09-21T13:51:29.931 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:29.932 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:29.932 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:29.932 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:51:29.932 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:29.920679+0000", 2024-09-21T13:51:29.932 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:29.932 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:29.932 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:29.932 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:51:29.932 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:29.920681+0000", 2024-09-21T13:51:29.932 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8279999999999999e-06 2024-09-21T13:51:29.932 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:29.932 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:29.933 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:51:29.933 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:29.922879+0000", 2024-09-21T13:51:29.933 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0021974569999999999 2024-09-21T13:51:29.933 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:29.933 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:29.933 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:51:29.933 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:29.922895+0000", 2024-09-21T13:51:29.933 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6212999999999999e-05 2024-09-21T13:51:29.933 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:29.933 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:29.933 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:51:29.933 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:29.922903+0000", 2024-09-21T13:51:29.933 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.6389999999999998e-06 2024-09-21T13:51:29.933 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:29.933 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:29.933 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:51:29.934 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:29.922969+0000", 2024-09-21T13:51:29.934 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5067000000000006e-05 2024-09-21T13:51:29.934 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:29.934 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:29.934 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:51:29.934 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:29.923105+0000", 2024-09-21T13:51:29.934 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013653999999999999 2024-09-21T13:51:29.934 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:29.934 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:29.934 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,7", 2024-09-21T13:51:29.934 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:29.923251+0000", 2024-09-21T13:51:29.934 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014620400000000001 2024-09-21T13:51:29.934 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:29.934 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:29.935 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:29.935 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:29.926954+0000", 2024-09-21T13:51:29.935 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0037027990000000001 2024-09-21T13:51:29.935 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:29.935 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:51:29.935 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:29.935 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:29.935 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:29.935 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:119864 7.d 7:b576f0de:::benchmark_data_smithi049_76798_object119863:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-21T13:51:29.935 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:51:29.926097+0000", 2024-09-21T13:51:29.935 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0031314979999999999, 2024-09-21T13:51:29.935 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0032487269999999999, 2024-09-21T13:51:29.935 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:51:29.936 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:51:29.936 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-21T13:51:29.936 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:51:29.936 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:51:29.936 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:29.936 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 119864 2024-09-21T13:51:29.936 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:29.936 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:51:29.936 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:29.936 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:51:29.936 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:29.926097+0000", 2024-09-21T13:51:29.936 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:29.936 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:29.936 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:29.937 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:51:29.937 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:29.926097+0000", 2024-09-21T13:51:29.937 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:29.937 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:29.937 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:29.937 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:51:29.937 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:29.926099+0000", 2024-09-21T13:51:29.937 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9120000000000001e-06 2024-09-21T13:51:29.937 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:29.937 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:29.937 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:51:29.937 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:29.926154+0000", 2024-09-21T13:51:29.937 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5433999999999998e-05 2024-09-21T13:51:29.937 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:29.937 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:29.938 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:51:29.938 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:29.926172+0000", 2024-09-21T13:51:29.938 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7926000000000001e-05 2024-09-21T13:51:29.938 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:29.938 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:29.938 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:51:29.938 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:29.926180+0000", 2024-09-21T13:51:29.938 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7319999999999998e-06 2024-09-21T13:51:29.938 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:29.938 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:29.938 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:51:29.938 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:29.926235+0000", 2024-09-21T13:51:29.938 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5578999999999998e-05 2024-09-21T13:51:29.938 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:29.939 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:29.939 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:51:29.939 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:29.926350+0000", 2024-09-21T13:51:29.939 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000114756 2024-09-21T13:51:29.939 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:29.939 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:29.939 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-21T13:51:29.939 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:29.926488+0000", 2024-09-21T13:51:29.939 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000137832 2024-09-21T13:51:29.939 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:29.939 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:51:29.939 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:29.939 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:29.939 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:29.939 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:119866 7.c 7:3cf886d0:::benchmark_data_smithi049_76798_object119865:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-21T13:51:29.940 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:51:29.926883+0000", 2024-09-21T13:51:29.940 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.002344802, 2024-09-21T13:51:29.940 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0025297449999999999, 2024-09-21T13:51:29.940 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:51:29.940 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:51:29.940 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-21T13:51:29.940 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:51:29.940 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:51:29.940 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:29.940 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 119866 2024-09-21T13:51:29.940 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:29.940 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:51:29.940 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:29.940 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:51:29.940 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:29.926883+0000", 2024-09-21T13:51:29.940 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:29.940 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:29.941 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:29.941 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:51:29.941 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:29.926883+0000", 2024-09-21T13:51:29.941 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:29.941 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:29.941 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:29.941 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:51:29.941 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:29.926884+0000", 2024-09-21T13:51:29.941 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.9900000000000009e-07 2024-09-21T13:51:29.941 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:29.941 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:29.941 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:51:29.941 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:29.926912+0000", 2024-09-21T13:51:29.941 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7676000000000001e-05 2024-09-21T13:51:29.941 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:29.942 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:29.942 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:51:29.942 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:29.926934+0000", 2024-09-21T13:51:29.942 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1659999999999999e-05 2024-09-21T13:51:29.942 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:29.942 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:29.942 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:51:29.942 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:29.926940+0000", 2024-09-21T13:51:29.942 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0229999999999998e-06 2024-09-21T13:51:29.942 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:29.942 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:29.942 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:51:29.942 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:29.926982+0000", 2024-09-21T13:51:29.942 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1758999999999998e-05 2024-09-21T13:51:29.943 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:29.943 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:29.943 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:51:29.943 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:29.927065+0000", 2024-09-21T13:51:29.943 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2915000000000002e-05 2024-09-21T13:51:29.943 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:29.943 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:29.943 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,7", 2024-09-21T13:51:29.943 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:29.927192+0000", 2024-09-21T13:51:29.943 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012754899999999999 2024-09-21T13:51:29.943 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:29.943 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:51:29.943 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:29.943 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:29.943 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2024-09-21T13:51:29.944 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 3 2024-09-21T13:51:29.944 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T13:51:29.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:29 smithi049 ceph-mon[30080]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T13:51:29.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:29 smithi049 ceph-mon[30080]: pgmap v103: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 6.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 49 MiB/s wr, 792 op/s; 41800/335595 objects degraded (12.455%) 2024-09-21T13:51:29.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:29 smithi049 ceph-mon[30080]: 4.15 scrub starts 2024-09-21T13:51:29.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:29 smithi049 ceph-mon[30080]: 4.15 scrub ok 2024-09-21T13:51:29.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:29 smithi049 ceph-mon[30080]: 3.b scrub starts 2024-09-21T13:51:29.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:29 smithi049 ceph-mon[30080]: 3.b scrub ok 2024-09-21T13:51:29.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:29 smithi049 ceph-mon[30080]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:51:29.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:29 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T13:51:29.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:29 smithi049 ceph-mon[30080]: osdmap e115: 8 total, 7 up, 8 in 2024-09-21T13:51:29.959 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:29 smithi049 ceph-mon[40489]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T13:51:29.959 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:29 smithi049 ceph-mon[40489]: pgmap v103: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 6.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 49 MiB/s wr, 792 op/s; 41800/335595 objects degraded (12.455%) 2024-09-21T13:51:29.959 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:29 smithi049 ceph-mon[40489]: 4.15 scrub starts 2024-09-21T13:51:29.959 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:29 smithi049 ceph-mon[40489]: 4.15 scrub ok 2024-09-21T13:51:29.959 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:29 smithi049 ceph-mon[40489]: 3.b scrub starts 2024-09-21T13:51:29.959 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:29 smithi049 ceph-mon[40489]: 3.b scrub ok 2024-09-21T13:51:29.959 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:29 smithi049 ceph-mon[40489]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:51:29.960 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:29 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T13:51:29.960 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:29 smithi049 ceph-mon[40489]: osdmap e115: 8 total, 7 up, 8 in 2024-09-21T13:51:29.986 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:51:30.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:30.039 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:30.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:51:30.141 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:51:30.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:30.193 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:30.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:51:30.294 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:30.332 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:30.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:30.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:30.433 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:51:30.463 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:30.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:30.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:51:30.535 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:51:30.536 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:51:30.564 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:51:30.588 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:51:30.589 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:51:30.589 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-21T13:51:30.622 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:51:30.623 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:51:30.629 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:30.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:30.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:51:30.643 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T13:51:30.657 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:51:30.657 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:51:30.690 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:51:30.690 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:51:30.726 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:51:30.726 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:51:30.731 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:30.767 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:51:30.767 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:51:30.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:30.772 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:30.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:30.872 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:51:30.908 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:30.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:30.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:51:30.917 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 78 16 121113 121097 97.0006 78.625 0.0141206 0.0103048 2024-09-21T13:51:30.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:30 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:51:30.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:30 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 41800/335595 objects degraded (12.455%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T13:51:30.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:30 smithi187 ceph-mon[33032]: osdmap e116: 8 total, 7 up, 8 in 2024-09-21T13:51:30.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:30 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/19542357' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:51:31.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:30 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:51:31.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:30 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 41800/335595 objects degraded (12.455%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T13:51:31.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:30 smithi049 ceph-mon[40489]: osdmap e116: 8 total, 7 up, 8 in 2024-09-21T13:51:31.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:30 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/19542357' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:51:31.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:30 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:51:31.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:30 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 41800/335595 objects degraded (12.455%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T13:51:31.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:30 smithi049 ceph-mon[30080]: osdmap e116: 8 total, 7 up, 8 in 2024-09-21T13:51:31.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:30 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/19542357' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:51:31.010 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:31.041 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:31.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:31.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:31.142 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:31.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:31.179 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:31.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:31.280 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:31.323 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:31.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:31.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:31.425 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:31.463 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:31.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:31.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:31.561 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T13:51:31.561 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T13:51:31.564 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:31.595 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:31.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:31.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:31.697 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:31.716 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T13:51:31.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:31.746 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:31.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:31.749 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:51:31.783 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:31 smithi049 ceph-mon[40489]: pgmap v106: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 7.2 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.57k op/s; 44096/354294 objects degraded (12.446%) 2024-09-21T13:51:31.784 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:31 smithi049 ceph-mon[30080]: pgmap v106: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 7.2 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.57k op/s; 44096/354294 objects degraded (12.446%) 2024-09-21T13:51:31.846 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:31.886 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:31 smithi187 ceph-mon[33032]: pgmap v106: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 7.2 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.57k op/s; 44096/354294 objects degraded (12.446%) 2024-09-21T13:51:31.896 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:31.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:31.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:31.919 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 79 16 122447 122431 96.8252 83.375 0.0172464 0.0103233 2024-09-21T13:51:31.998 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:51:32.026 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:32.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:32.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:51:32.128 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:32.169 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:32.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:32.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:32.271 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:51:32.311 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:32.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:32.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:51:32.412 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:32.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:32.474 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:32.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:32.575 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:32.610 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:32.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:32.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:32.630 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2024-09-21T13:51:32.630 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T13:51:32.630 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-21T13:51:32.630 INFO:tasks.thrashosds.thrasher:Setting osd 2 primary_affinity to 0.826869 2024-09-21T13:51:32.630 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 2 0.8268689002638664 2024-09-21T13:51:32.648 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T13:51:32.648 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:51:32.712 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:51:32.761 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:32.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:32.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:51:32.863 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:51:32.915 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:32.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:32.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:51:32.919 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:2024-09-21T13:51:32.920358+0000 min lat: 0.00133054 max lat: 1.05128 avg lat: 0.010345 2024-09-21T13:51:32.919 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T13:51:32.920 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 80 16 123734 123718 96.6203 80.4375 0.0121743 0.010345 2024-09-21T13:51:33.017 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:51:33.064 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:33.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:33.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:51:33.102 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:51:33.120 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:51:33.166 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:33.204 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:33.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:33.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:33.306 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:51:33.345 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T13:51:33.360 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:33.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:33.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:51:33.462 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:51:33.517 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:33.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:33.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:51:33.562 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T13:51:33.618 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:51:33.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:33.654 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:33.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:51:33.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:33 smithi049 ceph-mon[30080]: pgmap v107: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 7.2 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 49 MiB/s wr, 779 op/s; 44096/354294 objects degraded (12.446%) 2024-09-21T13:51:33.719 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:33 smithi049 ceph-mon[40489]: pgmap v107: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 7.2 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 49 MiB/s wr, 779 op/s; 44096/354294 objects degraded (12.446%) 2024-09-21T13:51:33.754 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:33.799 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:33.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:33.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:33.882 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:51:33.901 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:51:33.920 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 81 16 125010 124994 96.411 79.75 0.0128869 0.0103676 2024-09-21T13:51:33.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:33.937 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:33.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:51:33.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:33 smithi187 ceph-mon[33032]: pgmap v107: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 7.2 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 49 MiB/s wr, 779 op/s; 44096/354294 objects degraded (12.446%) 2024-09-21T13:51:34.038 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:51:34.093 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:34.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:34.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:51:34.195 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:34.243 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:34.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:34.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:34.345 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:51:34.389 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:34.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:34.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:51:34.491 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:51:34.532 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:34.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:34.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:51:34.634 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:34.687 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:34.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:34.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:34.731 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T13:51:34.731 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:51:34.788 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:51:34.817 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:34.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:34.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:51:34.823 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:51:34.919 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:35.110 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 82 16 126234 126218 96.1681 76.5 0.0276485 0.0103936 2024-09-21T13:51:35.114 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-09-21T13:51:35.144 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:35.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:35.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:35.222 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:35 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:51:35.222 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:35 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:51:35.222 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:35 smithi049 ceph-mon[40489]: pgmap v108: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 7.6 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 99 MiB/s wr, 1.58k op/s; 46526/373614 objects degraded (12.453%) 2024-09-21T13:51:35.223 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:35 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 44096/354294 objects degraded (12.446%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T13:51:35.223 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:35 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1078910813' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:51:35.223 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:35 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:51:35.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:35 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:51:35.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:35 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:51:35.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:35 smithi049 ceph-mon[30080]: pgmap v108: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 7.6 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 99 MiB/s wr, 1.58k op/s; 46526/373614 objects degraded (12.453%) 2024-09-21T13:51:35.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:35 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 44096/354294 objects degraded (12.446%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T13:51:35.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:35 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1078910813' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:51:35.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:35 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:51:35.246 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:51:35.283 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:35.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:35.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:51:35.354 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:35 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:51:35.355 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:35 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:51:35.355 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:35 smithi187 ceph-mon[33032]: pgmap v108: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 7.6 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 99 MiB/s wr, 1.58k op/s; 46526/373614 objects degraded (12.453%) 2024-09-21T13:51:35.355 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:35 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 44096/354294 objects degraded (12.446%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T13:51:35.355 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:35 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1078910813' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:51:35.355 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:35 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:51:35.384 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:51:35.410 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T13:51:35.420 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:35.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:35.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:51:35.522 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:35.555 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:35.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:35.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:35.658 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:51:35.698 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:35.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:35.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:51:35.727 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:51:35.727 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:51:35.729 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:51:35.729 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:51:35.730 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-21T13:51:35.762 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:51:35.762 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:51:35.802 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:35.803 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:51:35.803 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:51:35.832 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:51:35.832 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:51:35.845 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:51:35.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:35.854 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:35.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:35.865 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:51:35.865 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:51:35.896 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:51:35.897 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:51:35.920 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 83 16 127496 127480 95.9597 78.875 0.0131123 0.0104165 2024-09-21T13:51:35.955 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:35.993 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:35.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:35.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:36.094 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:36.124 INFO:teuthology.orchestra.run.smithi049.stderr:set osd.2 primary-affinity to 0.826869 (8541892) 2024-09-21T13:51:36.145 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:36.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:36.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:36.248 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:51:36.289 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:36.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:36.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:51:36.385 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:36 smithi187 ceph-mon[33032]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:51:36.385 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:36 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T13:51:36.385 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:36 smithi187 ceph-mon[33032]: osdmap e117: 8 total, 7 up, 8 in 2024-09-21T13:51:36.385 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:36 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1317939841' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 0.8268689002638664} : dispatch 2024-09-21T13:51:36.385 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:36 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 0.8268689002638664} : dispatch 2024-09-21T13:51:36.385 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:36 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1625873503' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:51:36.391 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:51:36.414 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:36.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:36.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:51:36.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:36 smithi049 ceph-mon[40489]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:51:36.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:36 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T13:51:36.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:36 smithi049 ceph-mon[40489]: osdmap e117: 8 total, 7 up, 8 in 2024-09-21T13:51:36.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:36 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1317939841' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 0.8268689002638664} : dispatch 2024-09-21T13:51:36.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:36 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 0.8268689002638664} : dispatch 2024-09-21T13:51:36.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:36 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1625873503' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:51:36.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:36 smithi049 ceph-mon[30080]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:51:36.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:36 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T13:51:36.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:36 smithi049 ceph-mon[30080]: osdmap e117: 8 total, 7 up, 8 in 2024-09-21T13:51:36.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:36 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1317939841' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 0.8268689002638664} : dispatch 2024-09-21T13:51:36.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:36 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 0.8268689002638664} : dispatch 2024-09-21T13:51:36.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:36 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1625873503' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:51:36.515 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:51:36.553 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:36.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:36.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:51:36.655 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:51:36.690 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:36.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:36.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:51:36.766 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T13:51:36.766 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-21T13:51:36.766 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:36.766 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:128520 7.5 7:ac405758:::benchmark_data_smithi049_76798_object128519:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-21T13:51:36.766 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:51:36.757731+0000", 2024-09-21T13:51:36.766 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0070170049999999998, 2024-09-21T13:51:36.766 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0070493259999999999, 2024-09-21T13:51:36.766 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:51:36.766 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:51:36.766 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-21T13:51:36.766 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:51:36.767 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:51:36.767 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:36.767 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 128520 2024-09-21T13:51:36.767 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:36.767 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:51:36.767 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:36.767 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:51:36.767 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.757731+0000", 2024-09-21T13:51:36.767 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:36.767 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:36.767 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:36.767 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:51:36.767 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.757731+0000", 2024-09-21T13:51:36.767 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:36.768 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:36.768 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:36.768 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:51:36.768 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.757733+0000", 2024-09-21T13:51:36.768 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.283e-06 2024-09-21T13:51:36.768 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:36.768 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:36.768 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:51:36.768 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.757779+0000", 2024-09-21T13:51:36.768 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6010000000000002e-05 2024-09-21T13:51:36.768 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:36.768 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:36.768 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:51:36.768 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.757789+0000", 2024-09-21T13:51:36.769 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0392e-05 2024-09-21T13:51:36.769 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:36.769 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:36.769 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:51:36.769 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.757796+0000", 2024-09-21T13:51:36.769 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7700000000000004e-06 2024-09-21T13:51:36.769 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:36.769 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:36.769 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:51:36.769 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.757863+0000", 2024-09-21T13:51:36.769 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6650999999999996e-05 2024-09-21T13:51:36.769 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:36.769 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:36.770 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:51:36.770 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.757982+0000", 2024-09-21T13:51:36.770 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000119214 2024-09-21T13:51:36.770 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:36.770 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:36.770 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0", 2024-09-21T13:51:36.770 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.758119+0000", 2024-09-21T13:51:36.770 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000137367 2024-09-21T13:51:36.770 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:36.770 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:36.770 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:51:36.770 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.761334+0000", 2024-09-21T13:51:36.770 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0032147130000000001 2024-09-21T13:51:36.771 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:36.771 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:51:36.771 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:36.771 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:36.771 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:36.771 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:128522 7.8 7:1939a70d:::benchmark_data_smithi049_76798_object128521:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-21T13:51:36.771 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:51:36.757819+0000", 2024-09-21T13:51:36.771 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0069294769999999999, 2024-09-21T13:51:36.771 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0070454560000000003, 2024-09-21T13:51:36.771 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:51:36.771 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:51:36.771 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-21T13:51:36.771 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:51:36.772 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:51:36.772 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:36.772 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 128522 2024-09-21T13:51:36.772 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:36.772 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:51:36.772 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:36.772 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:51:36.772 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.757819+0000", 2024-09-21T13:51:36.772 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:36.772 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:36.772 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:36.772 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:51:36.772 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.757819+0000", 2024-09-21T13:51:36.772 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:36.773 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:36.773 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:36.773 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:51:36.773 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.757820+0000", 2024-09-21T13:51:36.773 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9399999999999995e-07 2024-09-21T13:51:36.773 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:36.773 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:36.773 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:51:36.773 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.757857+0000", 2024-09-21T13:51:36.773 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7471000000000001e-05 2024-09-21T13:51:36.773 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:36.773 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:36.773 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:51:36.773 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.757864+0000", 2024-09-21T13:51:36.773 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4479999999999997e-06 2024-09-21T13:51:36.774 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:36.774 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:36.774 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:51:36.774 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.757870+0000", 2024-09-21T13:51:36.774 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5419999999999996e-06 2024-09-21T13:51:36.774 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:36.774 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:36.774 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:51:36.774 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.757923+0000", 2024-09-21T13:51:36.774 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2549999999999997e-05 2024-09-21T13:51:36.774 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:36.774 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:36.774 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:51:36.774 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.758050+0000", 2024-09-21T13:51:36.775 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000127916 2024-09-21T13:51:36.775 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:36.775 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:36.775 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 4", 2024-09-21T13:51:36.775 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.758178+0000", 2024-09-21T13:51:36.775 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000127207 2024-09-21T13:51:36.775 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:36.775 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:36.775 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:51:36.775 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.761181+0000", 2024-09-21T13:51:36.775 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0030038460000000001 2024-09-21T13:51:36.775 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:36.775 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:51:36.775 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:36.775 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:36.776 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:36.776 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:128526 7.5 7:a2f71c98:::benchmark_data_smithi049_76798_object128525:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-21T13:51:36.776 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:51:36.760983+0000", 2024-09-21T13:51:36.776 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0037656149999999999, 2024-09-21T13:51:36.776 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0039609989999999998, 2024-09-21T13:51:36.776 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:51:36.776 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:51:36.776 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-21T13:51:36.776 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:51:36.776 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:51:36.776 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:36.776 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 128526 2024-09-21T13:51:36.776 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:36.776 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:51:36.777 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:36.777 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:51:36.777 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.760983+0000", 2024-09-21T13:51:36.777 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:36.777 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:36.777 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:36.777 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:51:36.777 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.760983+0000", 2024-09-21T13:51:36.777 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:36.777 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:36.777 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:36.777 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:51:36.777 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.760984+0000", 2024-09-21T13:51:36.778 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.548e-06 2024-09-21T13:51:36.778 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:36.778 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:36.778 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:51:36.778 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.761480+0000", 2024-09-21T13:51:36.778 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00049519099999999997 2024-09-21T13:51:36.778 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:36.778 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:36.778 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:51:36.778 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.761492+0000", 2024-09-21T13:51:36.778 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2221e-05 2024-09-21T13:51:36.778 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:36.778 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:36.778 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:51:36.778 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.761499+0000", 2024-09-21T13:51:36.779 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.917e-06 2024-09-21T13:51:36.779 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:36.779 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:36.779 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:51:36.779 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.761585+0000", 2024-09-21T13:51:36.779 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5890000000000006e-05 2024-09-21T13:51:36.779 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:36.779 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:36.779 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:51:36.779 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.761702+0000", 2024-09-21T13:51:36.780 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000117679 2024-09-21T13:51:36.780 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:36.780 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:36.780 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0", 2024-09-21T13:51:36.780 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.761834+0000", 2024-09-21T13:51:36.780 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013146699999999999 2024-09-21T13:51:36.780 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:36.780 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:51:36.780 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:36.780 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:36.780 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:36.780 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:128527 7.8 7:19d8e686:::benchmark_data_smithi049_76798_object128526:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-21T13:51:36.780 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:51:36.761524+0000", 2024-09-21T13:51:36.781 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0032248210000000001, 2024-09-21T13:51:36.781 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0034793659999999998, 2024-09-21T13:51:36.781 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:51:36.781 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:51:36.781 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-21T13:51:36.781 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:51:36.781 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:51:36.781 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:36.781 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 128527 2024-09-21T13:51:36.781 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:36.781 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:51:36.781 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:36.781 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:51:36.782 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.761524+0000", 2024-09-21T13:51:36.782 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:36.782 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:36.782 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:36.782 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:51:36.782 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.761524+0000", 2024-09-21T13:51:36.782 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:36.782 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:36.782 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:36.782 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:51:36.782 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.761525+0000", 2024-09-21T13:51:36.782 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.47e-07 2024-09-21T13:51:36.782 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:36.783 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:36.783 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:51:36.783 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.761552+0000", 2024-09-21T13:51:36.783 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7036e-05 2024-09-21T13:51:36.783 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:36.783 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:36.783 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:51:36.783 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.761562+0000", 2024-09-21T13:51:36.783 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0669999999999999e-05 2024-09-21T13:51:36.783 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:36.783 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:36.783 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:51:36.783 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.761568+0000", 2024-09-21T13:51:36.783 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4199999999999998e-06 2024-09-21T13:51:36.784 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:36.784 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:36.784 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:51:36.784 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.762106+0000", 2024-09-21T13:51:36.784 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000538354 2024-09-21T13:51:36.784 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:36.784 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:36.784 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:51:36.784 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.762224+0000", 2024-09-21T13:51:36.784 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000118386 2024-09-21T13:51:36.784 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:36.784 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:36.784 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 4", 2024-09-21T13:51:36.784 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.762355+0000", 2024-09-21T13:51:36.785 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013111000000000001 2024-09-21T13:51:36.785 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:36.785 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:51:36.785 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:36.785 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:36.785 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:36.785 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.14817.0:128520 7.5 e118/101 ondisk, result = 0)", 2024-09-21T13:51:36.785 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:51:36.761830+0000", 2024-09-21T13:51:36.785 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.00291829, 2024-09-21T13:51:36.785 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0032241919999999999, 2024-09-21T13:51:36.785 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:51:36.785 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:51:36.785 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T13:51:36.785 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:51:36.786 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:36.786 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:51:36.786 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.761830+0000", 2024-09-21T13:51:36.786 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:36.786 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:36.786 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:36.786 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:51:36.786 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.761830+0000", 2024-09-21T13:51:36.786 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:36.786 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:36.786 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:36.786 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:51:36.786 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.761831+0000", 2024-09-21T13:51:36.786 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.128e-06 2024-09-21T13:51:36.787 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:36.787 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:36.787 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:51:36.787 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.761836+0000", 2024-09-21T13:51:36.787 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.391e-06 2024-09-21T13:51:36.787 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:36.787 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:36.787 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:51:36.787 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.761855+0000", 2024-09-21T13:51:36.787 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9709000000000001e-05 2024-09-21T13:51:36.787 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:36.787 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:36.787 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:51:36.787 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.761862+0000", 2024-09-21T13:51:36.788 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2600000000000002e-06 2024-09-21T13:51:36.788 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:36.788 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:36.788 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:51:36.788 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.764940+0000", 2024-09-21T13:51:36.788 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0030780199999999999 2024-09-21T13:51:36.788 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:36.788 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:36.788 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:51:36.788 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.764947+0000", 2024-09-21T13:51:36.788 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1149999999999996e-06 2024-09-21T13:51:36.788 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:36.788 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:51:36.788 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:36.788 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:36.789 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:36.789 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.14817.0:128522 7.8 e118/101)", 2024-09-21T13:51:36.789 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:51:36.762278+0000", 2024-09-21T13:51:36.789 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0024705959999999998, 2024-09-21T13:51:36.789 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0028215969999999999, 2024-09-21T13:51:36.789 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:51:36.789 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:51:36.789 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "queued for pg", 2024-09-21T13:51:36.789 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:51:36.789 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:36.789 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:51:36.789 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.762278+0000", 2024-09-21T13:51:36.789 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:36.789 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:36.790 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:36.790 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:51:36.790 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.762278+0000", 2024-09-21T13:51:36.790 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:36.790 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:36.790 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:36.790 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:51:36.790 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.762279+0000", 2024-09-21T13:51:36.790 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4199999999999995e-07 2024-09-21T13:51:36.790 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:36.790 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:36.790 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:51:36.790 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.762283+0000", 2024-09-21T13:51:36.790 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.578e-06 2024-09-21T13:51:36.791 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:36.791 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:36.791 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:51:36.791 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.762292+0000", 2024-09-21T13:51:36.791 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.9530000000000005e-06 2024-09-21T13:51:36.791 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:36.791 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:36.791 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:51:36.791 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.762298+0000", 2024-09-21T13:51:36.791 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0469999999999999e-06 2024-09-21T13:51:36.791 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:36.791 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:51:36.791 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:36.791 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:36.791 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2024-09-21T13:51:36.792 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 6 2024-09-21T13:51:36.792 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T13:51:36.792 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:36.830 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:36.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:36.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:36.921 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 84 16 128764 128748 95.7608 79.25 0.00567099 0.0104379 2024-09-21T13:51:36.931 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:51:36.969 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:36.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:36.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:51:37.070 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:51:37.109 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:37.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:37.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:51:37.211 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:37.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:37.254 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:37.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:37.355 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:51:37.394 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:37.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:37.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:51:37.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:37 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.8268689002638664}]': finished 2024-09-21T13:51:37.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:37 smithi187 ceph-mon[33032]: osdmap e118: 8 total, 7 up, 8 in 2024-09-21T13:51:37.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:37 smithi187 ceph-mon[33032]: pgmap v111: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 7.6 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 808 B/s rd, 64 MiB/s wr, 1.02k op/s; 46526/373614 objects degraded (12.453%) 2024-09-21T13:51:37.496 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:37.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:37 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.8268689002638664}]': finished 2024-09-21T13:51:37.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:37 smithi049 ceph-mon[30080]: osdmap e118: 8 total, 7 up, 8 in 2024-09-21T13:51:37.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:37 smithi049 ceph-mon[30080]: pgmap v111: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 7.6 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 808 B/s rd, 64 MiB/s wr, 1.02k op/s; 46526/373614 objects degraded (12.453%) 2024-09-21T13:51:37.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:37 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.8268689002638664}]': finished 2024-09-21T13:51:37.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:37 smithi049 ceph-mon[40489]: osdmap e118: 8 total, 7 up, 8 in 2024-09-21T13:51:37.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:37 smithi049 ceph-mon[40489]: pgmap v111: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 7.6 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 808 B/s rd, 64 MiB/s wr, 1.02k op/s; 46526/373614 objects degraded (12.453%) 2024-09-21T13:51:37.528 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:37.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:37.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:37.631 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:37.680 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:37.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:37.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:37.782 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:37.815 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:37.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:37.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:37.917 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:51:37.920 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 85 16 130031 130015 95.5658 79.1875 0.00778932 0.0104594 2024-09-21T13:51:37.960 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:37.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:37.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:51:38.021 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T13:51:38.062 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:51:38.104 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:38.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:38.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:51:38.174 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T13:51:38.206 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:51:38.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:38.258 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:38.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:51:38.349 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:51:38.359 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:51:38.396 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:38.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:38.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:51:38.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:38 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:51:38.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:38 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:51:38.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:38 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:51:38.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:38 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:51:38.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:38 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:51:38.498 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:38.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:38 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:51:38.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:38 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:51:38.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:38 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:51:38.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:38 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:51:38.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:38 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:51:38.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:38 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:51:38.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:38 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:51:38.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:38 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:51:38.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:38 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:51:38.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:38 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:51:38.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:38.543 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:38.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:38.644 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:38.678 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:38.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:38.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:38.781 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:51:38.828 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:38.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:38.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:51:38.921 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 86 16 131161 131145 95.2758 70.625 0.0109989 0.0104916 2024-09-21T13:51:38.928 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T13:51:38.928 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-21T13:51:38.928 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-21T13:51:38.928 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-21T13:51:38.928 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T13:51:38.930 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:51:38.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:38.955 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:38.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:51:39.056 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:51:39.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:39.088 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:39.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:51:39.188 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:51:39.224 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:39.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:39.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:51:39.326 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:39.350 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:39 smithi187 ceph-mon[33032]: pgmap v112: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 7.6 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 680 B/s rd, 54 MiB/s wr, 857 op/s; 46526/373614 objects degraded (12.453%) 2024-09-21T13:51:39.350 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:39 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:51:39.350 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:39 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:51:39.350 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:39 smithi187 ceph-mon[33032]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-09-21T13:51:39.351 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:51:39 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:51:39.366 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:39.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:39.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:39.468 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:39.502 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:39.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:39.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:39.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:39 smithi049 ceph-mon[30080]: pgmap v112: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 7.6 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 680 B/s rd, 54 MiB/s wr, 857 op/s; 46526/373614 objects degraded (12.453%) 2024-09-21T13:51:39.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:39 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:51:39.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:39 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:51:39.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:39 smithi049 ceph-mon[30080]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-09-21T13:51:39.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:39 smithi049 ceph-mon[40489]: pgmap v112: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 7.6 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 680 B/s rd, 54 MiB/s wr, 857 op/s; 46526/373614 objects degraded (12.453%) 2024-09-21T13:51:39.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:39 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:51:39.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:39 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:51:39.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:39 smithi049 ceph-mon[40489]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-09-21T13:51:39.604 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:51:39.639 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:39.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:39.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:51:39.718 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T13:51:39.740 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:39.801 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:39.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:39.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:39.902 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:51:39.921 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 87 16 132350 132334 95.0348 74.3125 0.0169583 0.0105178 2024-09-21T13:51:39.936 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:39.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:39.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:51:40.039 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:40.077 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:40.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:40.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:40.121 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:51:40.178 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:40.214 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:40.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:40.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:40.316 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:40.351 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:40.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:40.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:40.409 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:40 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:51:40.409 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:40 smithi187 ceph-mon[33032]: osdmap e119: 8 total, 7 up, 8 in 2024-09-21T13:51:40.409 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:40 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 46526/373614 objects degraded (12.453%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T13:51:40.409 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:40 smithi187 ceph-mon[33032]: osdmap e120: 8 total, 7 up, 8 in 2024-09-21T13:51:40.453 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:51:40.493 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:40.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:40.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:51:40.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:40 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:51:40.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:40 smithi049 ceph-mon[30080]: osdmap e119: 8 total, 7 up, 8 in 2024-09-21T13:51:40.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:40 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 46526/373614 objects degraded (12.453%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T13:51:40.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:40 smithi049 ceph-mon[30080]: osdmap e120: 8 total, 7 up, 8 in 2024-09-21T13:51:40.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:40 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:51:40.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:40 smithi049 ceph-mon[40489]: osdmap e119: 8 total, 7 up, 8 in 2024-09-21T13:51:40.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:40 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 46526/373614 objects degraded (12.453%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T13:51:40.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:40 smithi049 ceph-mon[40489]: osdmap e120: 8 total, 7 up, 8 in 2024-09-21T13:51:40.595 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:40.631 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:40.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:40.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:40.720 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T13:51:40.720 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T13:51:40.731 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T13:51:40.731 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-21T13:51:40.731 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-21T13:51:40.731 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-21T13:51:40.731 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.731 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:13004 7.5 7:ad0adcbe:::benchmark_data_smithi049_76798_object13003:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-21T13:51:40.731 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:17.815450+0000", 2024-09-21T13:51:40.731 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 82.913631639000002, 2024-09-21T13:51:40.732 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.089512449999999993, 2024-09-21T13:51:40.732 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:51:40.732 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:51:40.732 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:40.732 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:51:40.732 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:51:40.732 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:40.732 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 13004 2024-09-21T13:51:40.732 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.732 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:51:40.732 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.732 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:51:40.732 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:17.815450+0000", 2024-09-21T13:51:40.733 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:40.733 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.733 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.733 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:51:40.733 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:17.815450+0000", 2024-09-21T13:51:40.733 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:40.733 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.733 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.733 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:51:40.733 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:17.815452+0000", 2024-09-21T13:51:40.733 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5340000000000001e-06 2024-09-21T13:51:40.733 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.733 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.733 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:51:40.733 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:17.815526+0000", 2024-09-21T13:51:40.734 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3779e-05 2024-09-21T13:51:40.734 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.734 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.734 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:51:40.734 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:17.815542+0000", 2024-09-21T13:51:40.734 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6375000000000002e-05 2024-09-21T13:51:40.734 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.734 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.734 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:51:40.734 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:17.815548+0000", 2024-09-21T13:51:40.734 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5520000000000001e-06 2024-09-21T13:51:40.734 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.734 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.735 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:51:40.735 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:17.815601+0000", 2024-09-21T13:51:40.735 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2416000000000001e-05 2024-09-21T13:51:40.735 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.735 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.735 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:51:40.735 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:17.815686+0000", 2024-09-21T13:51:40.735 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5573000000000002e-05 2024-09-21T13:51:40.735 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.735 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.735 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2", 2024-09-21T13:51:40.735 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:17.815801+0000", 2024-09-21T13:51:40.735 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011408600000000001 2024-09-21T13:51:40.735 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.735 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.735 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:51:40.736 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:17.816341+0000", 2024-09-21T13:51:40.736 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00054071700000000002 2024-09-21T13:51:40.736 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.736 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.736 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:40.736 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:17.817654+0000", 2024-09-21T13:51:40.736 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0013126399999999999 2024-09-21T13:51:40.736 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.736 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.736 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:40.736 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:17.904857+0000", 2024-09-21T13:51:40.736 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.087202814000000003 2024-09-21T13:51:40.736 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.736 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.737 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:51:40.737 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:17.904916+0000", 2024-09-21T13:51:40.737 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9716000000000002e-05 2024-09-21T13:51:40.737 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.737 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.737 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:51:40.737 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:17.904963+0000", 2024-09-21T13:51:40.737 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6248000000000002e-05 2024-09-21T13:51:40.737 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:40.737 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:51:40.737 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:40.737 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.737 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.737 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:35249 7.6 7:6f97b9cd:::benchmark_data_smithi049_76798_object35248:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T13:51:40.737 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:28.696132+0000", 2024-09-21T13:51:40.738 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 72.032949694999999, 2024-09-21T13:51:40.738 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0107570610000001, 2024-09-21T13:51:40.738 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:51:40.738 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:51:40.738 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:40.738 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:51:40.738 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:51:40.738 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:40.738 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 35249 2024-09-21T13:51:40.738 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.739 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:51:40.739 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.739 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:51:40.739 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.696132+0000", 2024-09-21T13:51:40.739 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:40.739 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.739 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.739 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:51:40.739 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.696132+0000", 2024-09-21T13:51:40.739 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:40.739 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.739 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.739 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:51:40.739 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.696134+0000", 2024-09-21T13:51:40.739 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6080000000000001e-06 2024-09-21T13:51:40.739 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.740 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.740 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:51:40.740 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.696185+0000", 2024-09-21T13:51:40.740 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1328000000000001e-05 2024-09-21T13:51:40.740 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.740 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.740 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:51:40.740 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.696202+0000", 2024-09-21T13:51:40.740 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7334999999999999e-05 2024-09-21T13:51:40.740 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.740 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.740 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:51:40.740 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.696212+0000", 2024-09-21T13:51:40.740 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3489999999999998e-06 2024-09-21T13:51:40.740 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.741 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.741 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:51:40.741 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.696291+0000", 2024-09-21T13:51:40.741 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9078000000000004e-05 2024-09-21T13:51:40.741 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.741 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.741 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:51:40.741 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.696301+0000", 2024-09-21T13:51:40.741 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7780000000000002e-06 2024-09-21T13:51:40.741 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.741 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.741 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:51:40.741 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.697326+0000", 2024-09-21T13:51:40.741 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0010256289999999 2024-09-21T13:51:40.741 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.742 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.742 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:51:40.742 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.698023+0000", 2024-09-21T13:51:40.742 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00069705799999999997 2024-09-21T13:51:40.742 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.742 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.742 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3", 2024-09-21T13:51:40.742 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.698222+0000", 2024-09-21T13:51:40.742 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000198937 2024-09-21T13:51:40.742 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.742 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.742 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:51:40.742 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.703079+0000", 2024-09-21T13:51:40.742 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0048563690000000001 2024-09-21T13:51:40.742 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.743 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.743 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:40.743 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.706547+0000", 2024-09-21T13:51:40.743 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0034680499999999999 2024-09-21T13:51:40.743 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.743 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.743 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:40.743 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.706817+0000", 2024-09-21T13:51:40.743 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00026996700000000003 2024-09-21T13:51:40.743 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.743 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.743 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:51:40.743 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.706866+0000", 2024-09-21T13:51:40.743 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9292999999999997e-05 2024-09-21T13:51:40.743 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.743 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.744 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:51:40.744 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.706889+0000", 2024-09-21T13:51:40.744 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3281999999999998e-05 2024-09-21T13:51:40.744 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:40.744 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:51:40.744 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:40.744 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.744 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.744 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:51:40.745 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:35258 7.6 7:6cec60f9:::benchmark_data_smithi049_76798_object35257:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T13:51:40.745 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:28.700988+0000", 2024-09-21T13:51:40.745 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 72.028093964000007, 2024-09-21T13:51:40.745 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.007550977, 2024-09-21T13:51:40.745 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:51:40.745 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:51:40.745 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:40.745 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:51:40.746 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:51:40.746 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:40.746 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 35258 2024-09-21T13:51:40.746 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.746 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:51:40.746 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.746 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:51:40.746 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.700988+0000", 2024-09-21T13:51:40.746 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:40.746 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.746 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.746 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:51:40.746 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.700988+0000", 2024-09-21T13:51:40.746 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:40.746 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.746 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.747 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:51:40.747 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.700990+0000", 2024-09-21T13:51:40.747 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.795e-06 2024-09-21T13:51:40.747 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.747 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.747 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:51:40.747 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.701031+0000", 2024-09-21T13:51:40.747 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0836e-05 2024-09-21T13:51:40.747 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.747 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.747 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:51:40.747 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.701045+0000", 2024-09-21T13:51:40.747 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4178e-05 2024-09-21T13:51:40.747 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.747 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.748 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:51:40.748 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.701053+0000", 2024-09-21T13:51:40.748 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3499999999999997e-06 2024-09-21T13:51:40.748 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.748 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.748 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:51:40.748 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.701113+0000", 2024-09-21T13:51:40.748 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9857e-05 2024-09-21T13:51:40.748 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.748 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.748 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:51:40.749 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.701123+0000", 2024-09-21T13:51:40.749 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0202e-05 2024-09-21T13:51:40.749 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.749 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.749 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:51:40.749 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.700130+0000", 2024-09-21T13:51:40.749 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.99900719299999996 2024-09-21T13:51:40.749 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.749 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.749 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:51:40.749 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.700245+0000", 2024-09-21T13:51:40.749 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000115047 2024-09-21T13:51:40.749 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.749 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.749 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3", 2024-09-21T13:51:40.749 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.700392+0000", 2024-09-21T13:51:40.750 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000146358 2024-09-21T13:51:40.750 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.750 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.750 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:51:40.750 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.704100+0000", 2024-09-21T13:51:40.750 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0037085780000000001 2024-09-21T13:51:40.750 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.750 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.750 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:40.750 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.706575+0000", 2024-09-21T13:51:40.750 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.002474642 2024-09-21T13:51:40.750 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.750 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.750 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:40.750 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.708441+0000", 2024-09-21T13:51:40.751 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00186568 2024-09-21T13:51:40.751 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.751 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.751 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:51:40.751 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.708496+0000", 2024-09-21T13:51:40.751 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5340000000000002e-05 2024-09-21T13:51:40.751 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.751 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.751 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:51:40.751 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.708539+0000", 2024-09-21T13:51:40.751 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2920999999999998e-05 2024-09-21T13:51:40.751 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:40.751 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:51:40.751 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:40.751 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.752 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.752 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:35272 7.6 7:61fa97ba:::benchmark_data_smithi049_76798_object35271:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T13:51:40.752 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:28.712676+0000", 2024-09-21T13:51:40.752 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 72.016405668999994, 2024-09-21T13:51:40.752 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.99907040300000005, 2024-09-21T13:51:40.752 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:51:40.752 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:51:40.752 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:40.752 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:51:40.752 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:51:40.752 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:40.752 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 35272 2024-09-21T13:51:40.752 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.752 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:51:40.752 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.752 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:51:40.753 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.712676+0000", 2024-09-21T13:51:40.753 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:40.753 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.753 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.753 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:51:40.753 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.712676+0000", 2024-09-21T13:51:40.753 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:40.753 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.753 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.753 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:51:40.753 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.712678+0000", 2024-09-21T13:51:40.753 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.401e-06 2024-09-21T13:51:40.753 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.753 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.754 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:51:40.754 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.712724+0000", 2024-09-21T13:51:40.754 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6576999999999998e-05 2024-09-21T13:51:40.754 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.754 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.754 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:51:40.754 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.712736+0000", 2024-09-21T13:51:40.754 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2231e-05 2024-09-21T13:51:40.754 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.754 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.754 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:51:40.754 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.712742+0000", 2024-09-21T13:51:40.754 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9649999999999996e-06 2024-09-21T13:51:40.754 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.755 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.755 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:51:40.755 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.712819+0000", 2024-09-21T13:51:40.755 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6258000000000001e-05 2024-09-21T13:51:40.755 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.755 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.755 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:51:40.755 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.712831+0000", 2024-09-21T13:51:40.755 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.275e-05 2024-09-21T13:51:40.755 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.755 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.755 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:51:40.755 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.701685+0000", 2024-09-21T13:51:40.755 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.98885339699999997 2024-09-21T13:51:40.755 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.755 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.756 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:51:40.756 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.701779+0000", 2024-09-21T13:51:40.756 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3944999999999996e-05 2024-09-21T13:51:40.756 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.756 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.756 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3", 2024-09-21T13:51:40.756 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.701912+0000", 2024-09-21T13:51:40.756 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000133156 2024-09-21T13:51:40.756 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.756 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.756 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:51:40.756 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.705423+0000", 2024-09-21T13:51:40.756 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003511587 2024-09-21T13:51:40.756 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.757 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.757 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:40.757 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.708588+0000", 2024-09-21T13:51:40.757 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0031643309999999998 2024-09-21T13:51:40.757 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.757 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.757 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:40.757 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.711670+0000", 2024-09-21T13:51:40.757 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003081748 2024-09-21T13:51:40.757 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.757 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.757 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:51:40.757 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.711714+0000", 2024-09-21T13:51:40.757 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4406999999999997e-05 2024-09-21T13:51:40.757 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.757 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.758 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:51:40.758 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.711747+0000", 2024-09-21T13:51:40.758 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2650000000000001e-05 2024-09-21T13:51:40.758 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:40.758 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:51:40.758 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:40.758 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.758 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.758 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:35278 7.6 7:6d0c33e9:::benchmark_data_smithi049_76798_object35277:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T13:51:40.758 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:28.718784+0000", 2024-09-21T13:51:40.758 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 72.010297508999997, 2024-09-21T13:51:40.758 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.993077243, 2024-09-21T13:51:40.758 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:51:40.758 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:51:40.759 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:40.759 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:51:40.759 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:51:40.759 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:40.759 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 35278 2024-09-21T13:51:40.759 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.759 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:51:40.759 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.759 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:51:40.759 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.718784+0000", 2024-09-21T13:51:40.759 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:40.760 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.760 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.760 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:51:40.760 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.718784+0000", 2024-09-21T13:51:40.760 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:40.760 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.760 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.760 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:51:40.760 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.718787+0000", 2024-09-21T13:51:40.760 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1849999999999998e-06 2024-09-21T13:51:40.760 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.760 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.760 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:51:40.760 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.718859+0000", 2024-09-21T13:51:40.760 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2344999999999999e-05 2024-09-21T13:51:40.760 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.761 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.761 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:51:40.761 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.718872+0000", 2024-09-21T13:51:40.761 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3113e-05 2024-09-21T13:51:40.761 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.761 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.761 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:51:40.761 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.718881+0000", 2024-09-21T13:51:40.761 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.448e-06 2024-09-21T13:51:40.761 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.761 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.761 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:51:40.761 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.718941+0000", 2024-09-21T13:51:40.761 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9580999999999998e-05 2024-09-21T13:51:40.761 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.762 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.762 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:51:40.762 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.718948+0000", 2024-09-21T13:51:40.762 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.391e-06 2024-09-21T13:51:40.762 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.762 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.762 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:51:40.762 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.703121+0000", 2024-09-21T13:51:40.762 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.98417250099999998 2024-09-21T13:51:40.762 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.762 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.762 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:51:40.762 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.703248+0000", 2024-09-21T13:51:40.763 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012707800000000001 2024-09-21T13:51:40.763 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.763 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.763 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3", 2024-09-21T13:51:40.763 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.703345+0000", 2024-09-21T13:51:40.763 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7287999999999994e-05 2024-09-21T13:51:40.763 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.763 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.763 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:51:40.763 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.705432+0000", 2024-09-21T13:51:40.763 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0020863499999999998 2024-09-21T13:51:40.763 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.763 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.764 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:40.764 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.708649+0000", 2024-09-21T13:51:40.764 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003217443 2024-09-21T13:51:40.764 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.764 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.764 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:40.764 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.711798+0000", 2024-09-21T13:51:40.764 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0031489809999999999 2024-09-21T13:51:40.764 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.764 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.764 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:51:40.764 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.711832+0000", 2024-09-21T13:51:40.764 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3707000000000001e-05 2024-09-21T13:51:40.764 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.765 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.765 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:51:40.765 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.711862+0000", 2024-09-21T13:51:40.765 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9832000000000002e-05 2024-09-21T13:51:40.765 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:40.765 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:51:40.765 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:40.765 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.765 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.765 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:35280 7.6 7:68737c4e:::benchmark_data_smithi049_76798_object35279:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T13:51:40.765 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:28.719589+0000", 2024-09-21T13:51:40.765 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 72.009493047000007, 2024-09-21T13:51:40.765 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.99249087499999999, 2024-09-21T13:51:40.766 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:51:40.766 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:51:40.766 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:40.766 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:51:40.766 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:51:40.766 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:40.766 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 35280 2024-09-21T13:51:40.766 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.766 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:51:40.766 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.766 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:51:40.766 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.719589+0000", 2024-09-21T13:51:40.766 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:40.766 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.766 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.767 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:51:40.767 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.719589+0000", 2024-09-21T13:51:40.767 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:40.767 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.767 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.767 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:51:40.767 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.719590+0000", 2024-09-21T13:51:40.767 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.5600000000000004e-07 2024-09-21T13:51:40.767 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.767 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.767 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:51:40.767 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.719649+0000", 2024-09-21T13:51:40.767 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8730999999999997e-05 2024-09-21T13:51:40.767 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.767 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.768 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:51:40.768 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.719659+0000", 2024-09-21T13:51:40.768 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0498e-05 2024-09-21T13:51:40.768 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.768 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.768 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:51:40.768 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.719666+0000", 2024-09-21T13:51:40.768 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9029999999999998e-06 2024-09-21T13:51:40.768 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.768 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.768 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:51:40.768 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.719710+0000", 2024-09-21T13:51:40.768 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3859000000000001e-05 2024-09-21T13:51:40.768 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.769 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.769 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:51:40.769 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.719715+0000", 2024-09-21T13:51:40.769 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2229999999999998e-06 2024-09-21T13:51:40.769 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.769 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.769 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:51:40.769 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.704132+0000", 2024-09-21T13:51:40.769 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.98441694300000004 2024-09-21T13:51:40.769 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.769 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.770 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:51:40.770 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.704260+0000", 2024-09-21T13:51:40.770 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012844899999999999 2024-09-21T13:51:40.770 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.770 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.770 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3", 2024-09-21T13:51:40.770 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.704400+0000", 2024-09-21T13:51:40.770 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013959700000000001 2024-09-21T13:51:40.770 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.770 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.770 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:51:40.770 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.706512+0000", 2024-09-21T13:51:40.770 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0021123270000000002 2024-09-21T13:51:40.770 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.771 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.771 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:40.771 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.708683+0000", 2024-09-21T13:51:40.771 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00217022 2024-09-21T13:51:40.771 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.771 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.771 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:40.771 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.712030+0000", 2024-09-21T13:51:40.771 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0033470459999999998 2024-09-21T13:51:40.771 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.771 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.771 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:51:40.771 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.712058+0000", 2024-09-21T13:51:40.771 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8586e-05 2024-09-21T13:51:40.772 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.772 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.772 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:51:40.772 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.712080+0000", 2024-09-21T13:51:40.772 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1537000000000001e-05 2024-09-21T13:51:40.772 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:40.772 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:51:40.772 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:40.772 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.772 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.772 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:36401 7.5 7:a626d22a:::benchmark_data_smithi049_76798_object36400:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-21T13:51:40.772 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:30.277263+0000", 2024-09-21T13:51:40.772 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 70.451818975999998, 2024-09-21T13:51:40.772 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.091931077, 2024-09-21T13:51:40.772 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:51:40.773 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:51:40.773 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:40.773 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:51:40.773 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:51:40.773 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:40.773 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 36401 2024-09-21T13:51:40.773 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.773 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:51:40.773 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.773 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:51:40.773 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.277263+0000", 2024-09-21T13:51:40.773 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:40.773 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.773 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.774 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:51:40.774 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.277263+0000", 2024-09-21T13:51:40.774 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:40.774 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.774 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.774 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:51:40.774 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.277264+0000", 2024-09-21T13:51:40.774 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1519999999999999e-06 2024-09-21T13:51:40.774 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.774 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.774 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:51:40.774 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.277369+0000", 2024-09-21T13:51:40.774 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010487799999999999 2024-09-21T13:51:40.774 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.775 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.775 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:51:40.775 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.277380+0000", 2024-09-21T13:51:40.775 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1418e-05 2024-09-21T13:51:40.775 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.775 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.775 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:51:40.775 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.277387+0000", 2024-09-21T13:51:40.775 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.72e-06 2024-09-21T13:51:40.775 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.775 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.775 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:51:40.775 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.277444+0000", 2024-09-21T13:51:40.775 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7065999999999999e-05 2024-09-21T13:51:40.775 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.775 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.776 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:51:40.776 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.277598+0000", 2024-09-21T13:51:40.776 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015410900000000001 2024-09-21T13:51:40.776 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.776 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.776 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2", 2024-09-21T13:51:40.776 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.277715+0000", 2024-09-21T13:51:40.776 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011702899999999999 2024-09-21T13:51:40.776 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.776 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.776 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:51:40.776 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.280204+0000", 2024-09-21T13:51:40.776 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0024888839999999998 2024-09-21T13:51:40.776 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.777 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.777 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:40.777 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.281420+0000", 2024-09-21T13:51:40.777 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0012163020000000001 2024-09-21T13:51:40.777 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.777 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.777 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:40.777 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.369071+0000", 2024-09-21T13:51:40.777 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.087650371000000005 2024-09-21T13:51:40.777 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.777 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.777 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:51:40.777 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.369153+0000", 2024-09-21T13:51:40.777 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1759000000000001e-05 2024-09-21T13:51:40.778 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.778 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.778 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:51:40.778 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.369194+0000", 2024-09-21T13:51:40.778 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1389e-05 2024-09-21T13:51:40.778 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:40.778 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:51:40.778 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:40.778 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.778 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.778 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:36402 7.5 7:a61c91df:::benchmark_data_smithi049_76798_object36401:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-21T13:51:40.778 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:30.278041+0000", 2024-09-21T13:51:40.778 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 70.451041207000003, 2024-09-21T13:51:40.778 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.091287402000000004, 2024-09-21T13:51:40.778 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:51:40.779 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:51:40.779 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:40.779 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:51:40.779 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:51:40.779 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:40.779 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 36402 2024-09-21T13:51:40.779 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.779 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:51:40.779 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.779 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:51:40.779 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.278041+0000", 2024-09-21T13:51:40.779 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:40.780 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.780 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.780 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:51:40.780 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.278041+0000", 2024-09-21T13:51:40.780 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:40.780 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.780 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.780 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:51:40.780 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.278042+0000", 2024-09-21T13:51:40.780 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7300000000000004e-07 2024-09-21T13:51:40.780 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.780 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.780 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:51:40.780 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.278081+0000", 2024-09-21T13:51:40.780 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9644999999999997e-05 2024-09-21T13:51:40.781 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.781 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.781 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:51:40.781 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.278094+0000", 2024-09-21T13:51:40.781 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2581e-05 2024-09-21T13:51:40.781 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.781 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.781 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:51:40.781 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.278101+0000", 2024-09-21T13:51:40.781 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3529999999999997e-06 2024-09-21T13:51:40.781 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.781 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.781 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:51:40.782 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.278885+0000", 2024-09-21T13:51:40.782 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00078357700000000004 2024-09-21T13:51:40.782 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.782 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.782 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:51:40.782 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.278991+0000", 2024-09-21T13:51:40.782 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000106054 2024-09-21T13:51:40.782 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.782 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.782 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2", 2024-09-21T13:51:40.782 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.279121+0000", 2024-09-21T13:51:40.782 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012970399999999999 2024-09-21T13:51:40.782 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.783 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.783 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:51:40.783 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.280341+0000", 2024-09-21T13:51:40.783 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00122002 2024-09-21T13:51:40.783 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.783 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.783 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:40.783 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.281449+0000", 2024-09-21T13:51:40.783 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0011087549999999999 2024-09-21T13:51:40.783 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.783 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.783 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:40.783 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.369229+0000", 2024-09-21T13:51:40.783 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.087779550999999997 2024-09-21T13:51:40.783 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.784 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.784 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:51:40.784 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.369297+0000", 2024-09-21T13:51:40.784 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8429999999999994e-05 2024-09-21T13:51:40.784 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.784 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.784 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:51:40.784 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.369328+0000", 2024-09-21T13:51:40.784 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0759000000000001e-05 2024-09-21T13:51:40.784 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:40.784 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:51:40.784 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:40.784 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.784 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.785 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:51:40.786 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:36405 7.6 7:6f4f0025:::benchmark_data_smithi049_76798_object36404:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-21T13:51:40.786 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:30.279475+0000", 2024-09-21T13:51:40.786 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 70.449607138999994, 2024-09-21T13:51:40.786 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.092807156000000002, 2024-09-21T13:51:40.786 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:51:40.786 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:51:40.786 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:40.786 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:51:40.786 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:51:40.786 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:40.787 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 36405 2024-09-21T13:51:40.787 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.787 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:51:40.787 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.787 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:51:40.787 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.279475+0000", 2024-09-21T13:51:40.787 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:40.787 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.787 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.787 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:51:40.787 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.279475+0000", 2024-09-21T13:51:40.787 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:40.787 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.787 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.788 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:51:40.788 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.279476+0000", 2024-09-21T13:51:40.788 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.082e-06 2024-09-21T13:51:40.788 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.788 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.788 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:51:40.788 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.279540+0000", 2024-09-21T13:51:40.788 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4145000000000003e-05 2024-09-21T13:51:40.788 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.788 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.788 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:51:40.788 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.279551+0000", 2024-09-21T13:51:40.788 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1369000000000001e-05 2024-09-21T13:51:40.788 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.789 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.789 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:51:40.789 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.279557+0000", 2024-09-21T13:51:40.789 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3800000000000002e-06 2024-09-21T13:51:40.789 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.789 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.789 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:51:40.789 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.279599+0000", 2024-09-21T13:51:40.789 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2188999999999999e-05 2024-09-21T13:51:40.789 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.789 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.789 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:51:40.789 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.279685+0000", 2024-09-21T13:51:40.789 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.6558999999999996e-05 2024-09-21T13:51:40.790 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.790 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.790 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3", 2024-09-21T13:51:40.790 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.279783+0000", 2024-09-21T13:51:40.790 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7089999999999994e-05 2024-09-21T13:51:40.790 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.790 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.790 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:51:40.790 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.280828+0000", 2024-09-21T13:51:40.790 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0010456490000000001 2024-09-21T13:51:40.791 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.791 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.791 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:40.791 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.372109+0000", 2024-09-21T13:51:40.791 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.091280426999999997 2024-09-21T13:51:40.791 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.791 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.791 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:40.791 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.372184+0000", 2024-09-21T13:51:40.791 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5692999999999995e-05 2024-09-21T13:51:40.791 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.791 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.791 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:51:40.791 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.372240+0000", 2024-09-21T13:51:40.792 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5980999999999998e-05 2024-09-21T13:51:40.792 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.792 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.792 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:51:40.792 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.372282+0000", 2024-09-21T13:51:40.792 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1591999999999998e-05 2024-09-21T13:51:40.792 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:40.792 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:51:40.792 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:40.792 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.792 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.793 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:36408 7.6 7:6aca1bd2:::benchmark_data_smithi049_76798_object36407:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-21T13:51:40.793 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:30.281461+0000", 2024-09-21T13:51:40.793 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 70.447621018999996, 2024-09-21T13:51:40.793 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.090996460000000001, 2024-09-21T13:51:40.793 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:51:40.793 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:51:40.793 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:40.793 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:51:40.793 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:51:40.793 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:40.793 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 36408 2024-09-21T13:51:40.793 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.793 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:51:40.794 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.794 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:51:40.794 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.281461+0000", 2024-09-21T13:51:40.794 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:40.794 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.794 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.794 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:51:40.794 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.281461+0000", 2024-09-21T13:51:40.794 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:40.794 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.794 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.794 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:51:40.794 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.281462+0000", 2024-09-21T13:51:40.794 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.5999999999999991e-07 2024-09-21T13:51:40.795 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.795 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.795 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:51:40.795 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.281533+0000", 2024-09-21T13:51:40.795 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1056999999999994e-05 2024-09-21T13:51:40.795 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.795 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.795 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:51:40.795 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.281540+0000", 2024-09-21T13:51:40.795 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3440000000000003e-06 2024-09-21T13:51:40.795 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.795 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.795 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:51:40.795 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.281544+0000", 2024-09-21T13:51:40.796 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2289999999999997e-06 2024-09-21T13:51:40.796 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.796 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.796 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:51:40.796 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.281580+0000", 2024-09-21T13:51:40.796 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5163000000000002e-05 2024-09-21T13:51:40.796 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.796 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.796 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:51:40.796 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.281647+0000", 2024-09-21T13:51:40.796 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7015999999999997e-05 2024-09-21T13:51:40.796 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.796 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.796 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3", 2024-09-21T13:51:40.797 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.281724+0000", 2024-09-21T13:51:40.797 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7642e-05 2024-09-21T13:51:40.797 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.797 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.797 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:51:40.797 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.282653+0000", 2024-09-21T13:51:40.797 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00092829299999999996 2024-09-21T13:51:40.797 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.797 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.797 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:40.797 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.372148+0000", 2024-09-21T13:51:40.797 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.089495692000000002 2024-09-21T13:51:40.797 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.797 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.797 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:40.798 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.372370+0000", 2024-09-21T13:51:40.798 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000221559 2024-09-21T13:51:40.798 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.798 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.798 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:51:40.798 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.372423+0000", 2024-09-21T13:51:40.798 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2930999999999999e-05 2024-09-21T13:51:40.798 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.798 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.798 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:51:40.798 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.372457+0000", 2024-09-21T13:51:40.798 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4573999999999998e-05 2024-09-21T13:51:40.798 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:40.799 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:51:40.799 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:40.799 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.799 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.799 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:36413 7.6 7:6ca15b3e:::benchmark_data_smithi049_76798_object36412:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-21T13:51:40.799 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:30.284228+0000", 2024-09-21T13:51:40.799 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 70.444853796000004, 2024-09-21T13:51:40.799 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.090214049000000004, 2024-09-21T13:51:40.799 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:51:40.799 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:51:40.799 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:40.799 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:51:40.800 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:51:40.800 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:40.800 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 36413 2024-09-21T13:51:40.800 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.800 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:51:40.800 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.800 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:51:40.800 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.284228+0000", 2024-09-21T13:51:40.800 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:40.800 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.800 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.801 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:51:40.801 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.284228+0000", 2024-09-21T13:51:40.801 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:40.801 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.801 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.801 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:51:40.801 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.284229+0000", 2024-09-21T13:51:40.801 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9699999999999995e-07 2024-09-21T13:51:40.801 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.801 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.801 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:51:40.801 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.284283+0000", 2024-09-21T13:51:40.801 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4088999999999997e-05 2024-09-21T13:51:40.801 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.802 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.802 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:51:40.802 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.284295+0000", 2024-09-21T13:51:40.802 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1982e-05 2024-09-21T13:51:40.802 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.802 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.802 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:51:40.802 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.284302+0000", 2024-09-21T13:51:40.802 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7819999999999996e-06 2024-09-21T13:51:40.802 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.802 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.802 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:51:40.802 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.284344+0000", 2024-09-21T13:51:40.802 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2095999999999998e-05 2024-09-21T13:51:40.803 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.803 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.803 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:51:40.803 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.284416+0000", 2024-09-21T13:51:40.803 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1822000000000005e-05 2024-09-21T13:51:40.803 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.803 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.803 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3", 2024-09-21T13:51:40.803 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.284528+0000", 2024-09-21T13:51:40.803 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000112044 2024-09-21T13:51:40.803 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.803 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.803 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:40.803 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.372331+0000", 2024-09-21T13:51:40.804 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.087803479000000004 2024-09-21T13:51:40.804 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.804 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.804 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:40.804 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.372509+0000", 2024-09-21T13:51:40.804 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00017788999999999999 2024-09-21T13:51:40.804 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.804 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.804 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:51:40.804 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.374367+0000", 2024-09-21T13:51:40.804 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001858176 2024-09-21T13:51:40.804 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.804 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.804 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:51:40.805 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.374405+0000", 2024-09-21T13:51:40.805 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.824e-05 2024-09-21T13:51:40.805 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.805 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.805 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:51:40.805 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.374442+0000", 2024-09-21T13:51:40.805 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6652000000000001e-05 2024-09-21T13:51:40.805 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:40.805 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:51:40.805 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:40.805 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.805 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.805 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:36418 7.6 7:62f73716:::benchmark_data_smithi049_76798_object36417:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-21T13:51:40.805 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:30.286476+0000", 2024-09-21T13:51:40.805 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 70.442605678999996, 2024-09-21T13:51:40.806 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.094207888000000004, 2024-09-21T13:51:40.806 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:51:40.806 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:51:40.806 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:40.806 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:51:40.806 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:51:40.806 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:40.806 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 36418 2024-09-21T13:51:40.806 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.806 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:51:40.806 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.806 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:51:40.806 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.286476+0000", 2024-09-21T13:51:40.806 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:40.807 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.807 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.807 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:51:40.807 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.286476+0000", 2024-09-21T13:51:40.807 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:40.807 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.807 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.807 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:51:40.807 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.286477+0000", 2024-09-21T13:51:40.807 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2389999999999999e-06 2024-09-21T13:51:40.807 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.807 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.807 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:51:40.807 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.286537+0000", 2024-09-21T13:51:40.808 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9660999999999997e-05 2024-09-21T13:51:40.808 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.808 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.808 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:51:40.808 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.368559+0000", 2024-09-21T13:51:40.808 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.082022069000000003 2024-09-21T13:51:40.808 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.808 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.808 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:51:40.808 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.368593+0000", 2024-09-21T13:51:40.808 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3742000000000003e-05 2024-09-21T13:51:40.808 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.808 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.808 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:51:40.808 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.372543+0000", 2024-09-21T13:51:40.809 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0039505180000000001 2024-09-21T13:51:40.809 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.809 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.809 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:51:40.809 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.372670+0000", 2024-09-21T13:51:40.809 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012616 2024-09-21T13:51:40.809 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.809 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.809 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3", 2024-09-21T13:51:40.809 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.372815+0000", 2024-09-21T13:51:40.809 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014502499999999999 2024-09-21T13:51:40.809 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.809 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.809 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:51:40.810 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.374509+0000", 2024-09-21T13:51:40.810 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001694231 2024-09-21T13:51:40.810 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.810 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.810 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:40.810 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.376304+0000", 2024-09-21T13:51:40.810 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0017954069999999999 2024-09-21T13:51:40.810 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.810 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.810 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:40.810 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.380572+0000", 2024-09-21T13:51:40.810 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.004267468 2024-09-21T13:51:40.810 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.811 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.811 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:51:40.811 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.380642+0000", 2024-09-21T13:51:40.811 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.0594999999999997e-05 2024-09-21T13:51:40.811 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.811 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.811 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:51:40.811 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.380684+0000", 2024-09-21T13:51:40.811 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1773000000000001e-05 2024-09-21T13:51:40.811 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:40.811 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:51:40.811 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:40.811 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.812 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.812 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:49374 7.5 7:ade03d95:::benchmark_data_smithi049_76798_object49373:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T13:51:40.812 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:38.263545+0000", 2024-09-21T13:51:40.812 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 62.465536477999997, 2024-09-21T13:51:40.812 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.14789579899999999, 2024-09-21T13:51:40.812 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:51:40.812 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:51:40.812 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:40.812 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:51:40.812 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:51:40.812 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:40.812 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 49374 2024-09-21T13:51:40.812 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.813 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:51:40.813 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.813 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:51:40.813 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.263545+0000", 2024-09-21T13:51:40.813 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:40.813 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.813 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.813 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:51:40.813 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.263545+0000", 2024-09-21T13:51:40.813 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:40.813 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.813 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.813 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:51:40.813 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.263547+0000", 2024-09-21T13:51:40.814 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.561e-06 2024-09-21T13:51:40.814 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.814 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.814 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:51:40.814 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.263633+0000", 2024-09-21T13:51:40.814 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5536000000000002e-05 2024-09-21T13:51:40.814 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.814 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.814 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:51:40.814 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.263645+0000", 2024-09-21T13:51:40.814 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.253e-05 2024-09-21T13:51:40.814 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.814 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.815 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:51:40.815 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.263653+0000", 2024-09-21T13:51:40.815 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6990000000000003e-06 2024-09-21T13:51:40.815 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.815 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.815 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:51:40.815 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.263718+0000", 2024-09-21T13:51:40.815 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5140000000000003e-05 2024-09-21T13:51:40.815 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.815 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.815 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:51:40.815 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.263854+0000", 2024-09-21T13:51:40.815 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013566199999999999 2024-09-21T13:51:40.816 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.816 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.816 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2", 2024-09-21T13:51:40.816 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.263984+0000", 2024-09-21T13:51:40.816 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013001500000000001 2024-09-21T13:51:40.816 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.816 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.816 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:51:40.816 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.265404+0000", 2024-09-21T13:51:40.816 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0014200199999999999 2024-09-21T13:51:40.816 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.816 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.816 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:40.816 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.266190+0000", 2024-09-21T13:51:40.816 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00078600999999999996 2024-09-21T13:51:40.817 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.817 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.817 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:40.817 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.411303+0000", 2024-09-21T13:51:40.817 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.145113412 2024-09-21T13:51:40.817 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.817 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.817 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:51:40.817 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.411399+0000", 2024-09-21T13:51:40.817 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.5816000000000006e-05 2024-09-21T13:51:40.817 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.817 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.817 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:51:40.817 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.411441+0000", 2024-09-21T13:51:40.818 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2398000000000003e-05 2024-09-21T13:51:40.818 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:40.818 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:51:40.818 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:40.818 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.818 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.818 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:128080 7.6 7:6129c63d:::benchmark_data_smithi049_76798_object128079:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-21T13:51:40.818 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:51:36.368029+0000", 2024-09-21T13:51:40.818 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 4.3610524530000001, 2024-09-21T13:51:40.818 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.098668359999999997, 2024-09-21T13:51:40.818 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:51:40.818 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:51:40.818 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:40.819 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:51:40.819 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:51:40.819 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:40.819 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 128080 2024-09-21T13:51:40.819 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.819 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:51:40.819 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.819 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:51:40.819 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.368029+0000", 2024-09-21T13:51:40.819 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:40.819 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.819 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.819 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:51:40.819 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.368029+0000", 2024-09-21T13:51:40.819 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:40.820 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.820 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.820 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:51:40.820 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.368030+0000", 2024-09-21T13:51:40.820 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9299999999999997e-07 2024-09-21T13:51:40.820 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.820 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.820 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:51:40.820 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.368057+0000", 2024-09-21T13:51:40.820 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.6594999999999999e-05 2024-09-21T13:51:40.820 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.820 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.820 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:51:40.820 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.368065+0000", 2024-09-21T13:51:40.820 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3839999999999997e-06 2024-09-21T13:51:40.821 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.821 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.821 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:51:40.821 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.368071+0000", 2024-09-21T13:51:40.821 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3609999999999997e-06 2024-09-21T13:51:40.821 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.821 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.821 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:51:40.821 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.371169+0000", 2024-09-21T13:51:40.821 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0030970450000000001 2024-09-21T13:51:40.821 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.822 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.822 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:51:40.822 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.371298+0000", 2024-09-21T13:51:40.822 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012910299999999999 2024-09-21T13:51:40.822 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.822 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.822 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3", 2024-09-21T13:51:40.822 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.371447+0000", 2024-09-21T13:51:40.822 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000148997 2024-09-21T13:51:40.822 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.822 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.822 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:51:40.822 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.374940+0000", 2024-09-21T13:51:40.822 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0034930360000000001 2024-09-21T13:51:40.823 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.823 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.823 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:40.823 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.377972+0000", 2024-09-21T13:51:40.823 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0030321390000000001 2024-09-21T13:51:40.823 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.823 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.823 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:40.823 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.466603+0000", 2024-09-21T13:51:40.823 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.088631571000000006 2024-09-21T13:51:40.823 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.823 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.823 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:51:40.823 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.466659+0000", 2024-09-21T13:51:40.824 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.516e-05 2024-09-21T13:51:40.824 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.824 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.824 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:51:40.824 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.466698+0000", 2024-09-21T13:51:40.824 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9276000000000001e-05 2024-09-21T13:51:40.824 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:40.824 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:51:40.824 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:40.824 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.824 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.824 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:128086 7.6 7:6ba216cb:::benchmark_data_smithi049_76798_object128085:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-21T13:51:40.824 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:51:36.373022+0000", 2024-09-21T13:51:40.824 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 4.3560596729999999, 2024-09-21T13:51:40.825 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.093846730000000003, 2024-09-21T13:51:40.825 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:51:40.825 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:51:40.825 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:40.825 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:51:40.825 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:51:40.825 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:40.825 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 128086 2024-09-21T13:51:40.825 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.825 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:51:40.825 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.825 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:51:40.825 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.373022+0000", 2024-09-21T13:51:40.825 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:40.826 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.826 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.826 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:51:40.826 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.373022+0000", 2024-09-21T13:51:40.826 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:40.826 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.826 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.826 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:51:40.826 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.373024+0000", 2024-09-21T13:51:40.826 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6449999999999999e-06 2024-09-21T13:51:40.826 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.826 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.827 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:51:40.827 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.373065+0000", 2024-09-21T13:51:40.827 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1637999999999997e-05 2024-09-21T13:51:40.827 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.827 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.827 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:51:40.827 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.373078+0000", 2024-09-21T13:51:40.827 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.235e-05 2024-09-21T13:51:40.827 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.827 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.827 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:51:40.827 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.373086+0000", 2024-09-21T13:51:40.827 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5520000000000005e-06 2024-09-21T13:51:40.827 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.827 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.828 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:51:40.828 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.374965+0000", 2024-09-21T13:51:40.828 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001878338 2024-09-21T13:51:40.828 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.828 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.828 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:51:40.828 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.375084+0000", 2024-09-21T13:51:40.828 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000119739 2024-09-21T13:51:40.828 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.828 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.828 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3", 2024-09-21T13:51:40.828 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.375222+0000", 2024-09-21T13:51:40.828 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013773600000000001 2024-09-21T13:51:40.828 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.829 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.829 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:40.829 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.378015+0000", 2024-09-21T13:51:40.829 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0027930199999999998 2024-09-21T13:51:40.829 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.829 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.829 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:51:40.829 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.378315+0000", 2024-09-21T13:51:40.829 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00029946500000000001 2024-09-21T13:51:40.829 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.829 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.829 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:40.829 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.466767+0000", 2024-09-21T13:51:40.829 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.088452006 2024-09-21T13:51:40.829 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.830 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.830 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:51:40.830 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.466821+0000", 2024-09-21T13:51:40.830 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4323000000000001e-05 2024-09-21T13:51:40.830 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.830 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.830 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:51:40.830 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.466869+0000", 2024-09-21T13:51:40.830 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7917999999999998e-05 2024-09-21T13:51:40.830 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:40.830 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:51:40.830 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:40.830 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.831 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.831 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:130274 7.6 7:6b6c7cd6:::benchmark_data_smithi049_76798_object130273:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-21T13:51:40.831 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:51:38.117519+0000", 2024-09-21T13:51:40.831 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 2.6115627799999999, 2024-09-21T13:51:40.831 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.096025074000000002, 2024-09-21T13:51:40.831 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:51:40.831 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:51:40.831 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:40.831 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:51:40.831 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:51:40.831 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:40.831 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 130274 2024-09-21T13:51:40.832 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.832 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:51:40.832 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.832 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:51:40.832 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.117519+0000", 2024-09-21T13:51:40.832 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:40.832 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.832 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.832 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:51:40.832 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.117519+0000", 2024-09-21T13:51:40.832 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:40.832 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.832 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.833 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:51:40.833 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.117523+0000", 2024-09-21T13:51:40.833 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9169999999999999e-06 2024-09-21T13:51:40.833 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.833 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.833 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:51:40.833 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.117580+0000", 2024-09-21T13:51:40.833 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7092000000000002e-05 2024-09-21T13:51:40.833 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.833 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.833 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:51:40.833 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.117594+0000", 2024-09-21T13:51:40.833 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3592e-05 2024-09-21T13:51:40.833 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.833 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.834 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:51:40.834 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.117601+0000", 2024-09-21T13:51:40.834 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3470000000000001e-06 2024-09-21T13:51:40.834 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.834 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.834 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:51:40.834 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.117658+0000", 2024-09-21T13:51:40.834 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7151999999999998e-05 2024-09-21T13:51:40.834 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.834 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.834 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:51:40.834 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.117790+0000", 2024-09-21T13:51:40.834 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013137700000000001 2024-09-21T13:51:40.834 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.834 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.835 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3", 2024-09-21T13:51:40.835 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.117940+0000", 2024-09-21T13:51:40.835 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000150113 2024-09-21T13:51:40.835 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.835 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.835 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:40.835 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.121978+0000", 2024-09-21T13:51:40.835 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0040387779999999998 2024-09-21T13:51:40.835 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.835 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.835 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:51:40.835 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.129084+0000", 2024-09-21T13:51:40.835 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0071057899999999998 2024-09-21T13:51:40.835 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.835 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.836 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:40.836 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.213338+0000", 2024-09-21T13:51:40.836 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.084253761999999996 2024-09-21T13:51:40.836 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.836 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.836 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:51:40.836 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.213490+0000", 2024-09-21T13:51:40.836 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015244100000000001 2024-09-21T13:51:40.836 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.836 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.836 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:51:40.836 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.213544+0000", 2024-09-21T13:51:40.836 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3712999999999999e-05 2024-09-21T13:51:40.836 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:40.836 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:51:40.837 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:40.837 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.837 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.837 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:130279 7.6 7:62017b0c:::benchmark_data_smithi049_76798_object130278:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-21T13:51:40.837 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:51:38.122206+0000", 2024-09-21T13:51:40.837 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 2.6068754670000001, 2024-09-21T13:51:40.837 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.091537365999999995, 2024-09-21T13:51:40.837 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:51:40.837 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:51:40.837 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:40.837 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:51:40.837 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:51:40.837 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:40.837 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 130279 2024-09-21T13:51:40.838 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.838 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:51:40.838 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.838 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:51:40.838 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.122206+0000", 2024-09-21T13:51:40.838 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:40.838 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.838 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.838 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:51:40.838 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.122206+0000", 2024-09-21T13:51:40.838 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:40.838 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.838 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.838 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:51:40.838 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.122208+0000", 2024-09-21T13:51:40.839 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.488e-06 2024-09-21T13:51:40.839 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.839 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.839 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:51:40.839 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.122239+0000", 2024-09-21T13:51:40.839 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1312000000000001e-05 2024-09-21T13:51:40.839 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.839 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.839 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:51:40.839 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.122255+0000", 2024-09-21T13:51:40.839 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5628999999999999e-05 2024-09-21T13:51:40.839 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.839 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.839 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:51:40.839 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.122263+0000", 2024-09-21T13:51:40.839 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1149999999999994e-06 2024-09-21T13:51:40.840 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.840 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.840 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:51:40.840 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.122521+0000", 2024-09-21T13:51:40.840 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00025809299999999998 2024-09-21T13:51:40.840 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.840 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.840 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:51:40.840 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.122634+0000", 2024-09-21T13:51:40.840 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000113256 2024-09-21T13:51:40.840 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.840 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.840 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3", 2024-09-21T13:51:40.840 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.122776+0000", 2024-09-21T13:51:40.842 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014201599999999999 2024-09-21T13:51:40.842 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.842 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.842 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:40.842 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.129013+0000", 2024-09-21T13:51:40.842 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0062370999999999998 2024-09-21T13:51:40.842 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.842 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.842 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:51:40.842 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.131560+0000", 2024-09-21T13:51:40.843 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0025468309999999998 2024-09-21T13:51:40.843 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.843 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.843 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:40.843 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.213590+0000", 2024-09-21T13:51:40.843 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.082029460999999998 2024-09-21T13:51:40.843 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.843 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.843 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:51:40.843 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.213670+0000", 2024-09-21T13:51:40.843 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0747000000000005e-05 2024-09-21T13:51:40.843 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.843 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.843 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:51:40.843 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.213744+0000", 2024-09-21T13:51:40.844 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3318000000000005e-05 2024-09-21T13:51:40.844 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:40.844 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:51:40.844 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:40.844 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.844 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.844 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:130284 7.6 7:64f9f54c:::benchmark_data_smithi049_76798_object130283:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-21T13:51:40.844 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:51:38.123475+0000", 2024-09-21T13:51:40.844 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 2.6056066690000002, 2024-09-21T13:51:40.844 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.094856334, 2024-09-21T13:51:40.844 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:51:40.845 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:51:40.845 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:40.845 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:51:40.845 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:51:40.845 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:40.845 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 130284 2024-09-21T13:51:40.845 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.845 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:51:40.845 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.845 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:51:40.845 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.123475+0000", 2024-09-21T13:51:40.845 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:40.845 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.845 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.845 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:51:40.846 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.123475+0000", 2024-09-21T13:51:40.846 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:40.846 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.846 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.846 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:51:40.846 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.123476+0000", 2024-09-21T13:51:40.846 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0100000000000003e-07 2024-09-21T13:51:40.846 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.846 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.846 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:51:40.846 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.123573+0000", 2024-09-21T13:51:40.846 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6545999999999993e-05 2024-09-21T13:51:40.846 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.846 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.847 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:51:40.847 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.123583+0000", 2024-09-21T13:51:40.847 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0155e-05 2024-09-21T13:51:40.847 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.847 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.847 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:51:40.847 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.123589+0000", 2024-09-21T13:51:40.847 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4339999999999997e-06 2024-09-21T13:51:40.847 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.847 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.847 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:51:40.847 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.125707+0000", 2024-09-21T13:51:40.847 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.002117673 2024-09-21T13:51:40.847 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.848 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.848 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:51:40.848 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.125980+0000", 2024-09-21T13:51:40.848 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00027341699999999997 2024-09-21T13:51:40.848 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.848 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.848 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3", 2024-09-21T13:51:40.848 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.126135+0000", 2024-09-21T13:51:40.848 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000154346 2024-09-21T13:51:40.848 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.848 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.849 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:40.849 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.129046+0000", 2024-09-21T13:51:40.849 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0029114660000000001 2024-09-21T13:51:40.849 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.849 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.849 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:51:40.849 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.131565+0000", 2024-09-21T13:51:40.849 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.002519126 2024-09-21T13:51:40.849 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.849 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.849 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:40.849 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.218233+0000", 2024-09-21T13:51:40.849 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.086667771000000005 2024-09-21T13:51:40.849 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.850 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.850 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:51:40.850 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.218292+0000", 2024-09-21T13:51:40.850 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9008999999999997e-05 2024-09-21T13:51:40.850 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.850 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.850 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:51:40.850 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.218332+0000", 2024-09-21T13:51:40.850 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9490000000000003e-05 2024-09-21T13:51:40.850 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:40.850 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:51:40.850 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:40.850 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.850 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.851 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:130287 7.6 7:6fd3f54f:::benchmark_data_smithi049_76798_object130286:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-21T13:51:40.851 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:51:38.127595+0000", 2024-09-21T13:51:40.851 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 2.601486972, 2024-09-21T13:51:40.851 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.090871489, 2024-09-21T13:51:40.851 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:51:40.851 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:51:40.851 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:40.851 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:51:40.851 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:51:40.851 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:40.851 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 130287 2024-09-21T13:51:40.851 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.851 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:51:40.851 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.851 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:51:40.852 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.127595+0000", 2024-09-21T13:51:40.852 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:40.852 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.852 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.852 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:51:40.852 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.127595+0000", 2024-09-21T13:51:40.852 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:40.852 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.852 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.852 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:51:40.852 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.127596+0000", 2024-09-21T13:51:40.853 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.102e-06 2024-09-21T13:51:40.853 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.853 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.853 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:51:40.853 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.127656+0000", 2024-09-21T13:51:40.853 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9843000000000003e-05 2024-09-21T13:51:40.853 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.853 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.853 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:51:40.853 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.127667+0000", 2024-09-21T13:51:40.853 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0997e-05 2024-09-21T13:51:40.853 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.853 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.854 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:51:40.854 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.127674+0000", 2024-09-21T13:51:40.854 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2490000000000004e-06 2024-09-21T13:51:40.854 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.854 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.854 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:51:40.854 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.129153+0000", 2024-09-21T13:51:40.854 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001479129 2024-09-21T13:51:40.854 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.854 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.854 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:51:40.854 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.129307+0000", 2024-09-21T13:51:40.854 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015345100000000001 2024-09-21T13:51:40.854 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.855 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.855 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3", 2024-09-21T13:51:40.855 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.129449+0000", 2024-09-21T13:51:40.855 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014281400000000001 2024-09-21T13:51:40.855 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.855 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.855 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:51:40.855 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.131657+0000", 2024-09-21T13:51:40.855 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0022071299999999999 2024-09-21T13:51:40.855 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.855 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.855 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:40.855 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.132318+0000", 2024-09-21T13:51:40.855 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00066174099999999998 2024-09-21T13:51:40.855 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.856 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.856 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:40.856 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.218377+0000", 2024-09-21T13:51:40.856 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.086058529999999994 2024-09-21T13:51:40.856 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.856 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.856 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:51:40.856 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.218428+0000", 2024-09-21T13:51:40.856 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1515999999999999e-05 2024-09-21T13:51:40.856 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.856 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.856 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:51:40.856 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.218466+0000", 2024-09-21T13:51:40.856 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7987000000000001e-05 2024-09-21T13:51:40.857 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:40.857 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:51:40.857 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:40.857 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.857 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.857 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:131163 7.6 7:6c697fbd:::benchmark_data_smithi049_76798_object131162:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-21T13:51:40.857 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:51:38.922491+0000", 2024-09-21T13:51:40.857 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 1.8065909449999999, 2024-09-21T13:51:40.857 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.092446456999999996, 2024-09-21T13:51:40.857 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:51:40.857 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:51:40.857 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:51:40.857 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:51:40.857 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:51:40.857 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:40.857 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 131163 2024-09-21T13:51:40.858 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.858 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:51:40.858 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.858 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:51:40.858 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.922491+0000", 2024-09-21T13:51:40.858 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:40.858 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.858 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.858 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:51:40.858 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.922491+0000", 2024-09-21T13:51:40.858 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:51:40.858 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.858 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.858 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:51:40.859 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.922492+0000", 2024-09-21T13:51:40.859 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.291e-06 2024-09-21T13:51:40.859 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.859 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.859 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:51:40.859 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.922522+0000", 2024-09-21T13:51:40.859 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9527999999999999e-05 2024-09-21T13:51:40.859 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.859 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.859 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:51:40.859 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.922537+0000", 2024-09-21T13:51:40.859 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4882000000000001e-05 2024-09-21T13:51:40.859 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.859 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.859 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:51:40.859 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.922543+0000", 2024-09-21T13:51:40.860 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7349999999999999e-06 2024-09-21T13:51:40.860 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.860 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.860 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:51:40.860 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.927301+0000", 2024-09-21T13:51:40.860 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0047571779999999999 2024-09-21T13:51:40.860 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.860 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.860 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:51:40.860 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.927410+0000", 2024-09-21T13:51:40.860 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010956299999999999 2024-09-21T13:51:40.860 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.860 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.860 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3", 2024-09-21T13:51:40.860 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.927534+0000", 2024-09-21T13:51:40.861 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000124108 2024-09-21T13:51:40.861 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.861 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.861 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:51:40.861 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.933284+0000", 2024-09-21T13:51:40.861 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0057494729999999997 2024-09-21T13:51:40.861 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.861 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.861 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:40.861 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.934943+0000", 2024-09-21T13:51:40.861 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001659766 2024-09-21T13:51:40.861 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.861 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.862 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:40.862 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:39.014866+0000", 2024-09-21T13:51:40.862 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.079922332999999998 2024-09-21T13:51:40.862 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.862 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.862 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:51:40.862 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:39.014908+0000", 2024-09-21T13:51:40.862 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2006999999999999e-05 2024-09-21T13:51:40.862 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:51:40.862 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:51:40.862 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:51:40.862 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:39.014937+0000", 2024-09-21T13:51:40.863 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9592999999999999e-05 2024-09-21T13:51:40.863 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:40.863 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:51:40.863 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:40.863 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:51:40.863 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:51:40.863 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T13:51:40.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:40.866 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:40.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:51:40.909 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:51:40 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:51:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:51:40.921 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 88 16 133733 133717 94.937 86.4375 0.0122612 0.0105287 2024-09-21T13:51:40.966 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:41.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:41.007 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:41.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:41.108 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:51:41.153 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:41.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:41.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:51:41.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:41 smithi049 ceph-mon[30080]: pgmap v114: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 8.0 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 927 B/s rd, 70 MiB/s wr, 1.11k op/s; 48826/392070 objects degraded (12.453%) 2024-09-21T13:51:41.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:41 smithi049 ceph-mon[40489]: pgmap v114: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 8.0 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 927 B/s rd, 70 MiB/s wr, 1.11k op/s; 48826/392070 objects degraded (12.453%) 2024-09-21T13:51:41.255 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:51:41.286 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:41.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:41.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:51:41.388 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:51:41.414 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:41.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:41.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:51:41.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:41 smithi187 ceph-mon[33032]: pgmap v114: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 8.0 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 927 B/s rd, 70 MiB/s wr, 1.11k op/s; 48826/392070 objects degraded (12.453%) 2024-09-21T13:51:41.516 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:51:41.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:41.550 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:41.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:51:41.651 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:51:41.682 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:41.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:41.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:51:41.784 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:41.817 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T13:51:41.824 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:41.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:41.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:41.921 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 89 16 135095 135079 94.8267 85.125 0.00969068 0.010541 2024-09-21T13:51:41.925 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:41.958 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:41.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:41.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:42.059 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:51:42.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:42.105 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:42.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:51:42.163 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-09-21T13:51:42.207 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:42.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:42.256 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:42.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:42.357 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:42.395 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:42.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:42.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:42.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:42 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/4210617901' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:51:42.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:42 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:51:42.497 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:42.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:42 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/4210617901' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:51:42.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:42 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:51:42.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:42 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/4210617901' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:51:42.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:42 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:51:42.536 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:42.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:42.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:42.638 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:51:42.669 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:42.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:42.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:51:42.721 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T13:51:42.770 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:51:42.798 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:42.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:42.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:51:42.899 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:51:42.921 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Total time run: 90.0084 2024-09-21T13:51:42.921 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Total writes made: 136417 2024-09-21T13:51:42.921 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Write size: 65536 2024-09-21T13:51:42.921 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Object size: 65536 2024-09-21T13:51:42.921 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Bandwidth (MB/sec): 94.7252 2024-09-21T13:51:42.922 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Stddev Bandwidth: 27.2591 2024-09-21T13:51:42.922 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Max bandwidth (MB/sec): 168.875 2024-09-21T13:51:42.922 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Min bandwidth (MB/sec): 1.75 2024-09-21T13:51:42.922 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Average IOPS: 1515 2024-09-21T13:51:42.922 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Stddev IOPS: 436.145 2024-09-21T13:51:42.922 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Max IOPS: 2702 2024-09-21T13:51:42.922 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Min IOPS: 28 2024-09-21T13:51:42.922 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Average Latency(s): 0.0105527 2024-09-21T13:51:42.922 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Stddev Latency(s): 0.0168827 2024-09-21T13:51:42.922 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Max latency(s): 1.05128 2024-09-21T13:51:42.922 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Min latency(s): 0.00133054 2024-09-21T13:51:42.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:42.923 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:42.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:51:42.926 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Cleaning up (deleting benchmark objects) 2024-09-21T13:51:43.024 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:51:43.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:43.117 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:43.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:51:43.130 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:51:43.175 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:51:43.218 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:43.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:43 smithi049 ceph-mon[30080]: pgmap v116: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 8.0 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 916 B/s rd, 69 MiB/s wr, 1.10k op/s; 48826/392070 objects degraded (12.453%) 2024-09-21T13:51:43.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:43 smithi049 ceph-mon[30080]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:51:43.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:43 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T13:51:43.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:43 smithi049 ceph-mon[30080]: osdmap e121: 8 total, 7 up, 8 in 2024-09-21T13:51:43.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:43 smithi049 ceph-mon[30080]: 5.1f scrub starts 2024-09-21T13:51:43.257 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:43 smithi049 ceph-mon[30080]: 5.1f scrub ok 2024-09-21T13:51:43.257 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:43 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:51:43.257 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:43 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:51:43.257 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:43 smithi049 ceph-mon[40489]: pgmap v116: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 8.0 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 916 B/s rd, 69 MiB/s wr, 1.10k op/s; 48826/392070 objects degraded (12.453%) 2024-09-21T13:51:43.257 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:43 smithi049 ceph-mon[40489]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:51:43.257 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:43 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T13:51:43.257 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:43 smithi049 ceph-mon[40489]: osdmap e121: 8 total, 7 up, 8 in 2024-09-21T13:51:43.257 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:43 smithi049 ceph-mon[40489]: 5.1f scrub starts 2024-09-21T13:51:43.257 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:43 smithi049 ceph-mon[40489]: 5.1f scrub ok 2024-09-21T13:51:43.257 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:43 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:51:43.257 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:43 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:51:43.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:43.266 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:43.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:43.367 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:51:43.399 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:43.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:43.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:51:43.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:43 smithi187 ceph-mon[33032]: pgmap v116: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 8.0 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 916 B/s rd, 69 MiB/s wr, 1.10k op/s; 48826/392070 objects degraded (12.453%) 2024-09-21T13:51:43.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:43 smithi187 ceph-mon[33032]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:51:43.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:43 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T13:51:43.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:43 smithi187 ceph-mon[33032]: osdmap e121: 8 total, 7 up, 8 in 2024-09-21T13:51:43.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:43 smithi187 ceph-mon[33032]: 5.1f scrub starts 2024-09-21T13:51:43.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:43 smithi187 ceph-mon[33032]: 5.1f scrub ok 2024-09-21T13:51:43.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:43 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:51:43.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:43 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:51:43.501 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:43.541 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:43.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:43.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:43.643 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:51:43.673 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:43.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:43.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:51:43.775 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:51:43.810 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:43.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:43.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:51:43.912 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:43.961 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:43.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:43.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:44.063 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:44.102 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:44.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:44.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:44.104 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:51:44.104 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:51:44.143 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:51:44.143 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:51:44.147 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-21T13:51:44.178 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T13:51:44.178 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-09-21T13:51:44.178 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.178 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.14817.0:137865 7.6 e121/94 7:6dcd9174:::benchmark_data_smithi049_76798_object1445:head v 121'8701, mlcod=121'8698)", 2024-09-21T13:51:44.178 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:51:43.982893+0000", 2024-09-21T13:51:44.178 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.0079665510000000005, 2024-09-21T13:51:44.178 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0079945740000000008, 2024-09-21T13:51:44.178 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:51:44.178 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:51:44.178 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T13:51:44.178 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:51:44.178 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.179 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:51:44.179 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.982893+0000", 2024-09-21T13:51:44.179 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:44.179 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.179 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.179 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:51:44.179 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.982893+0000", 2024-09-21T13:51:44.179 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:44.179 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.179 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.179 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:51:44.179 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.982895+0000", 2024-09-21T13:51:44.179 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1519999999999999e-06 2024-09-21T13:51:44.179 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.180 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.180 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:51:44.180 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.982900+0000", 2024-09-21T13:51:44.180 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5770000000000001e-06 2024-09-21T13:51:44.180 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.180 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.180 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:51:44.180 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.982913+0000", 2024-09-21T13:51:44.180 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.242e-05 2024-09-21T13:51:44.180 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.180 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.180 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:51:44.180 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.982919+0000", 2024-09-21T13:51:44.180 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4969999999999998e-06 2024-09-21T13:51:44.180 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.181 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.181 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:51:44.181 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.984030+0000", 2024-09-21T13:51:44.181 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001111372 2024-09-21T13:51:44.181 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.181 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.181 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:51:44.181 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.984050+0000", 2024-09-21T13:51:44.181 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9673e-05 2024-09-21T13:51:44.181 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:44.181 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:51:44.181 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:44.181 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.181 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.181 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.14817.0:137864 7.e e121/101 7:755a9d97:::benchmark_data_smithi049_76798_object1444:head v 121'8597, mlcod=121'8593)", 2024-09-21T13:51:44.182 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:51:43.983562+0000", 2024-09-21T13:51:44.182 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.0072978679999999999, 2024-09-21T13:51:44.182 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0073814390000000001, 2024-09-21T13:51:44.182 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:51:44.182 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:51:44.182 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T13:51:44.182 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:51:44.182 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.182 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:51:44.182 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.983562+0000", 2024-09-21T13:51:44.182 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:44.182 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.182 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.182 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:51:44.183 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.983562+0000", 2024-09-21T13:51:44.183 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:44.183 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.183 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.183 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:51:44.183 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.983563+0000", 2024-09-21T13:51:44.183 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5199999999999996e-07 2024-09-21T13:51:44.183 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.183 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.183 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:51:44.183 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.983567+0000", 2024-09-21T13:51:44.183 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4449999999999998e-06 2024-09-21T13:51:44.183 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.183 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.183 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:51:44.183 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.983577+0000", 2024-09-21T13:51:44.184 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7990000000000005e-06 2024-09-21T13:51:44.184 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.184 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.184 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:51:44.184 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.983584+0000", 2024-09-21T13:51:44.184 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9949999999999999e-06 2024-09-21T13:51:44.184 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.184 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.184 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:51:44.184 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.983682+0000", 2024-09-21T13:51:44.184 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7768000000000003e-05 2024-09-21T13:51:44.184 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.184 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.184 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:51:44.184 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.983852+0000", 2024-09-21T13:51:44.184 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016998 2024-09-21T13:51:44.185 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:44.185 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:51:44.185 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:44.185 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.185 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.185 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.14817.0:137867 7.6 e121/94 7:694c0cfe:::benchmark_data_smithi049_76798_object1447:head v 121'8702, mlcod=121'8699)", 2024-09-21T13:51:44.185 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:51:43.986821+0000", 2024-09-21T13:51:44.185 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.0040386830000000004, 2024-09-21T13:51:44.185 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0041706349999999998, 2024-09-21T13:51:44.185 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:51:44.185 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:51:44.185 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T13:51:44.185 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:51:44.185 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.185 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:51:44.186 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.986821+0000", 2024-09-21T13:51:44.186 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:44.186 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.186 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.186 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:51:44.186 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.986821+0000", 2024-09-21T13:51:44.186 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:44.186 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.186 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.186 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:51:44.186 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.986822+0000", 2024-09-21T13:51:44.186 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6799999999999999e-07 2024-09-21T13:51:44.186 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.186 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.187 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:51:44.187 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.986828+0000", 2024-09-21T13:51:44.187 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3820000000000003e-06 2024-09-21T13:51:44.187 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.187 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.187 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:51:44.187 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.986839+0000", 2024-09-21T13:51:44.187 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1046999999999999e-05 2024-09-21T13:51:44.187 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.187 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.187 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:51:44.187 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.986846+0000", 2024-09-21T13:51:44.187 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3699999999999997e-06 2024-09-21T13:51:44.187 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.187 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.187 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:51:44.188 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.989591+0000", 2024-09-21T13:51:44.188 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0027451810000000002 2024-09-21T13:51:44.188 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.188 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.188 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:51:44.188 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.989608+0000", 2024-09-21T13:51:44.188 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7282000000000002e-05 2024-09-21T13:51:44.188 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:44.188 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:51:44.188 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:44.188 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.188 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.188 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.14817.0:137870 7.7 e121/83 7:e886c3a1:::benchmark_data_smithi049_76798_object1450:head v 121'8680, mlcod=121'8678)", 2024-09-21T13:51:44.188 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:51:43.986892+0000", 2024-09-21T13:51:44.188 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.0039679540000000001, 2024-09-21T13:51:44.188 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0041469949999999997, 2024-09-21T13:51:44.189 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:51:44.189 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:51:44.189 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T13:51:44.189 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:51:44.189 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.189 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:51:44.189 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.986892+0000", 2024-09-21T13:51:44.189 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:44.189 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.189 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.189 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:51:44.189 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.986892+0000", 2024-09-21T13:51:44.189 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:44.189 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.189 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.189 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:51:44.190 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.986893+0000", 2024-09-21T13:51:44.190 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3499999999999996e-07 2024-09-21T13:51:44.190 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.190 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.190 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:51:44.190 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.986897+0000", 2024-09-21T13:51:44.190 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2799999999999997e-06 2024-09-21T13:51:44.190 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.190 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.190 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:51:44.190 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.986906+0000", 2024-09-21T13:51:44.190 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.7849999999999997e-06 2024-09-21T13:51:44.190 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.190 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.190 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:51:44.190 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.986911+0000", 2024-09-21T13:51:44.191 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7459999999999997e-06 2024-09-21T13:51:44.191 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.191 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.191 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:51:44.191 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.986989+0000", 2024-09-21T13:51:44.191 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7449999999999999e-05 2024-09-21T13:51:44.191 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.191 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.191 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:51:44.191 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.987010+0000", 2024-09-21T13:51:44.191 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1290000000000001e-05 2024-09-21T13:51:44.191 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:44.191 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:51:44.191 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:44.191 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.192 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.192 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.14817.0:137868 7.9 e121/94 7:988a1399:::benchmark_data_smithi049_76798_object1448:head v 121'8594, mlcod=121'8592)", 2024-09-21T13:51:44.192 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:51:43.988611+0000", 2024-09-21T13:51:44.192 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.0022487269999999998, 2024-09-21T13:51:44.192 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0024750140000000002, 2024-09-21T13:51:44.192 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:51:44.192 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:51:44.192 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T13:51:44.192 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:51:44.192 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.192 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:51:44.192 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.988611+0000", 2024-09-21T13:51:44.192 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:44.192 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.193 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.193 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:51:44.193 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.988611+0000", 2024-09-21T13:51:44.193 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:44.193 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.193 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.193 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:51:44.193 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.988613+0000", 2024-09-21T13:51:44.193 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.31e-06 2024-09-21T13:51:44.193 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.193 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.193 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:51:44.193 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.988620+0000", 2024-09-21T13:51:44.193 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7260000000000002e-06 2024-09-21T13:51:44.193 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.193 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.194 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:51:44.194 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.988633+0000", 2024-09-21T13:51:44.194 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3193999999999999e-05 2024-09-21T13:51:44.194 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.194 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.194 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:51:44.194 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.988641+0000", 2024-09-21T13:51:44.194 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8280000000000003e-06 2024-09-21T13:51:44.194 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.194 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.194 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:51:44.194 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.988696+0000", 2024-09-21T13:51:44.194 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4484999999999998e-05 2024-09-21T13:51:44.194 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.194 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.195 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:51:44.195 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.988717+0000", 2024-09-21T13:51:44.195 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.124e-05 2024-09-21T13:51:44.195 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:44.195 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:51:44.195 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:44.195 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.195 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.195 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.14817.0:137866 7.f e121/94 7:f3cc00c8:::benchmark_data_smithi049_76798_object1446:head v 121'8382, mlcod=121'8380)", 2024-09-21T13:51:44.195 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:51:43.988683+0000", 2024-09-21T13:51:44.195 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.002176903, 2024-09-21T13:51:44.195 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0024497680000000002, 2024-09-21T13:51:44.195 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:51:44.195 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:51:44.195 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T13:51:44.196 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:51:44.196 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.196 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:51:44.196 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.988683+0000", 2024-09-21T13:51:44.196 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:44.196 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.196 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.196 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:51:44.196 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.988683+0000", 2024-09-21T13:51:44.196 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:44.196 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.196 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.196 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:51:44.196 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.988684+0000", 2024-09-21T13:51:44.197 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1399999999999997e-07 2024-09-21T13:51:44.197 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.197 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.197 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:51:44.197 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.988688+0000", 2024-09-21T13:51:44.197 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4050000000000002e-06 2024-09-21T13:51:44.197 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.197 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.197 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:51:44.197 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.988696+0000", 2024-09-21T13:51:44.197 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2509999999999994e-06 2024-09-21T13:51:44.197 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.197 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.197 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:51:44.197 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.988702+0000", 2024-09-21T13:51:44.197 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4639999999999997e-06 2024-09-21T13:51:44.197 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.198 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.198 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:51:44.198 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.988866+0000", 2024-09-21T13:51:44.198 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000164372 2024-09-21T13:51:44.198 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.198 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.198 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:51:44.198 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.988887+0000", 2024-09-21T13:51:44.198 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1121999999999999e-05 2024-09-21T13:51:44.198 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:44.198 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:51:44.198 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:44.198 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.198 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.198 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.14817.0:137873 7.f e121/94 7:fa95340e:::benchmark_data_smithi049_76798_object1453:head v 121'8383, mlcod=121'8380)", 2024-09-21T13:51:44.199 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:51:43.988714+0000", 2024-09-21T13:51:44.199 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.002145704, 2024-09-21T13:51:44.199 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0024646870000000001, 2024-09-21T13:51:44.199 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:51:44.199 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:51:44.199 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T13:51:44.199 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:51:44.199 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.199 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:51:44.199 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.988714+0000", 2024-09-21T13:51:44.199 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:44.199 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.199 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.199 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:51:44.199 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.988714+0000", 2024-09-21T13:51:44.200 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:44.200 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.200 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.200 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:51:44.200 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.988715+0000", 2024-09-21T13:51:44.200 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9299999999999999e-07 2024-09-21T13:51:44.200 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.200 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.200 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:51:44.200 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.988718+0000", 2024-09-21T13:51:44.200 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3400000000000002e-06 2024-09-21T13:51:44.200 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.200 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.200 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:51:44.200 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.988724+0000", 2024-09-21T13:51:44.201 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7869999999999998e-06 2024-09-21T13:51:44.201 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.201 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.201 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:51:44.201 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.988728+0000", 2024-09-21T13:51:44.201 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3329999999999998e-06 2024-09-21T13:51:44.201 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.201 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.201 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:51:44.201 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.991136+0000", 2024-09-21T13:51:44.201 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0024080159999999998 2024-09-21T13:51:44.201 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.201 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.201 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:51:44.201 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.991154+0000", 2024-09-21T13:51:44.202 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8077e-05 2024-09-21T13:51:44.202 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:44.202 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:51:44.202 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:44.202 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.202 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.202 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.14817.0:137869 7.e e121/101)", 2024-09-21T13:51:44.202 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:51:43.988746+0000", 2024-09-21T13:51:44.202 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.0021143339999999998, 2024-09-21T13:51:44.202 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0024761359999999999, 2024-09-21T13:51:44.202 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:51:44.202 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:51:44.202 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "queued for pg", 2024-09-21T13:51:44.202 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:51:44.203 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.203 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:51:44.203 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.988746+0000", 2024-09-21T13:51:44.203 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:44.203 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.203 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.203 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:51:44.203 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.988746+0000", 2024-09-21T13:51:44.203 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:44.203 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.203 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.203 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:51:44.203 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.988746+0000", 2024-09-21T13:51:44.203 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9299999999999999e-07 2024-09-21T13:51:44.203 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.204 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.204 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:51:44.204 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.988750+0000", 2024-09-21T13:51:44.204 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.895e-06 2024-09-21T13:51:44.204 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.204 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.204 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:51:44.204 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.988757+0000", 2024-09-21T13:51:44.204 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6150000000000002e-06 2024-09-21T13:51:44.204 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:44.204 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:44.204 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:51:44.204 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:43.988761+0000", 2024-09-21T13:51:44.204 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3669999999999999e-06 2024-09-21T13:51:44.204 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:44.204 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:51:44.205 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:44.205 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:44.205 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T13:51:44.205 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 8 2024-09-21T13:51:44.205 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T13:51:44.205 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:44.206 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:51:44.206 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:51:44.222 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:51:44.222 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:51:44.238 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:44.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:44.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:44.257 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:51:44.258 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:51:44.296 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:51:44.296 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:51:44.336 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:51:44.336 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:51:44.340 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:51:44.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:44.377 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:44.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:51:44.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:44 smithi187 ceph-mon[33032]: 5.18 scrub starts 2024-09-21T13:51:44.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:44 smithi187 ceph-mon[33032]: 5.18 scrub ok 2024-09-21T13:51:44.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:44 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3122693722' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:51:44.478 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:51:44.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:44 smithi049 ceph-mon[40489]: 5.18 scrub starts 2024-09-21T13:51:44.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:44 smithi049 ceph-mon[40489]: 5.18 scrub ok 2024-09-21T13:51:44.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:44 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3122693722' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:51:44.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:44 smithi049 ceph-mon[30080]: 5.18 scrub starts 2024-09-21T13:51:44.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:44 smithi049 ceph-mon[30080]: 5.18 scrub ok 2024-09-21T13:51:44.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:44 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3122693722' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:51:44.513 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:44.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:44.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:51:44.615 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:44.655 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:44.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:44.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:44.757 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:51:44.795 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:44.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:44.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:51:44.897 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:44.938 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:44.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:44.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:45.026 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T13:51:45.040 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:51:45.073 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2024-09-21T13:51:45.073 INFO:tasks.thrashosds.thrasher:Reviving osd 2 2024-09-21T13:51:45.073 INFO:tasks.cephadm.osd.2:Restarting osd.2 (starting--it wasn't running)... 2024-09-21T13:51:45.073 DEBUG:teuthology.orchestra.run.smithi049:osd.2> sudo journalctl -f -n 0 -u ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@osd.2.service 2024-09-21T13:51:45.103 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:45.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:45.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:51:45.104 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl start ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@osd.2 2024-09-21T13:51:45.205 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:45.235 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:45.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:45.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:45.337 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:45.365 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:45.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:45.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:45.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:45 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:51:45.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:45 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:51:45.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:45 smithi187 ceph-mon[33032]: pgmap v118: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 8.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.8 KiB/s rd, 126 MiB/s wr, 2.12k op/s; 50860/407901 objects degraded (12.469%) 2024-09-21T13:51:45.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:45 smithi187 ceph-mon[33032]: osdmap e122: 8 total, 7 up, 8 in 2024-09-21T13:51:45.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:45 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 50860/407901 objects degraded (12.469%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T13:51:45.467 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:51:45.504 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:45.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:45 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:51:45.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:45 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:51:45.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:45 smithi049 ceph-mon[30080]: pgmap v118: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 8.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.8 KiB/s rd, 126 MiB/s wr, 2.12k op/s; 50860/407901 objects degraded (12.469%) 2024-09-21T13:51:45.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:45 smithi049 ceph-mon[30080]: osdmap e122: 8 total, 7 up, 8 in 2024-09-21T13:51:45.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:45 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 50860/407901 objects degraded (12.469%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T13:51:45.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:45 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:51:45.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:45 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:51:45.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:45 smithi049 ceph-mon[40489]: pgmap v118: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 8.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.8 KiB/s rd, 126 MiB/s wr, 2.12k op/s; 50860/407901 objects degraded (12.469%) 2024-09-21T13:51:45.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:45 smithi049 ceph-mon[40489]: osdmap e122: 8 total, 7 up, 8 in 2024-09-21T13:51:45.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:45 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 50860/407901 objects degraded (12.469%), 20 pgs degraded (PG_DEGRADED) 2024-09-21T13:51:45.506 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:51:45 smithi049 systemd[1]: Starting Ceph osd.2 for 40a3d2e8-781f-11ef-baf6-efdc52797490... 2024-09-21T13:51:45.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:45.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:51:45.607 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:45.643 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:45.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:45.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:45.745 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:45.776 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:45.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:45.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:45.878 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:45.913 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:45.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:45.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:46.015 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:51:46.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:46.049 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:46.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:51:46.149 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:46.184 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:46.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:46.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:46.286 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:51:46.316 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:46.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:46.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:51:46.418 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:46.451 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:46.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:46.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:46.553 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:51:46.569 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T13:51:46.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:46.599 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:46.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:51:46.700 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:51:46.772 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:46.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:46.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:51:46.874 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:51:46.900 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:46.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:46.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:51:47.002 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:47.025 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:47.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:47.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:47.039 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T13:51:47.040 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T13:51:47.124 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:51:47.126 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:47.139 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:51:47.154 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:47.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:47.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:47.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:47 smithi049 ceph-mon[30080]: pgmap v120: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 8.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 62 MiB/s wr, 1.10k op/s; 50860/407901 objects degraded (12.469%) 2024-09-21T13:51:47.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:47 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:51:47.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:47 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:51:47.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:47 smithi049 ceph-mon[30080]: osdmap e123: 8 total, 7 up, 8 in 2024-09-21T13:51:47.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:47 smithi049 ceph-mon[30080]: 5.3 scrub starts 2024-09-21T13:51:47.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:47 smithi049 ceph-mon[30080]: 5.3 scrub ok 2024-09-21T13:51:47.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:47 smithi049 ceph-mon[30080]: 5.f scrub starts 2024-09-21T13:51:47.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:47 smithi049 ceph-mon[30080]: 5.f scrub ok 2024-09-21T13:51:47.256 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:47.257 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:47 smithi049 ceph-mon[40489]: pgmap v120: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 8.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 62 MiB/s wr, 1.10k op/s; 50860/407901 objects degraded (12.469%) 2024-09-21T13:51:47.257 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:47 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:51:47.257 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:47 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:51:47.257 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:47 smithi049 ceph-mon[40489]: osdmap e123: 8 total, 7 up, 8 in 2024-09-21T13:51:47.258 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:47 smithi049 ceph-mon[40489]: 5.3 scrub starts 2024-09-21T13:51:47.258 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:47 smithi049 ceph-mon[40489]: 5.3 scrub ok 2024-09-21T13:51:47.258 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:47 smithi049 ceph-mon[40489]: 5.f scrub starts 2024-09-21T13:51:47.258 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:47 smithi049 ceph-mon[40489]: 5.f scrub ok 2024-09-21T13:51:47.283 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:47.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:47.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:47.385 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:51:47.412 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:47.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:47.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:51:47.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:47 smithi187 ceph-mon[33032]: pgmap v120: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 8.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 62 MiB/s wr, 1.10k op/s; 50860/407901 objects degraded (12.469%) 2024-09-21T13:51:47.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:47 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:51:47.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:47 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:51:47.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:47 smithi187 ceph-mon[33032]: osdmap e123: 8 total, 7 up, 8 in 2024-09-21T13:51:47.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:47 smithi187 ceph-mon[33032]: 5.3 scrub starts 2024-09-21T13:51:47.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:47 smithi187 ceph-mon[33032]: 5.3 scrub ok 2024-09-21T13:51:47.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:47 smithi187 ceph-mon[33032]: 5.f scrub starts 2024-09-21T13:51:47.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:47 smithi187 ceph-mon[33032]: 5.f scrub ok 2024-09-21T13:51:47.513 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:51:47.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:47.542 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:47.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:51:47.643 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:47.671 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:47.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:47.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:47.773 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:47.801 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:47.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:47.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:47.903 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:51:47.952 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:47.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:47.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:51:48.054 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:51:48.097 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:48.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:48.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:51:48.185 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T13:51:48.186 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:51:48.199 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:51:48.242 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:48.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:48.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:51:48.344 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:48.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:48.371 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:48.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:48.472 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:48.505 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:48.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:48.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:48.507 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:51:48 smithi049 podman[89079]: 2024-09-21 13:51:48.27574077 +0000 UTC m=+1.084024672 image pull b415816db8b41f544cdfdc71d193ff8b1ee008bf7b0ec4f64da8eb8f71b62e85 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5 2024-09-21T13:51:48.607 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:48.650 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:48.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:48.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:48.752 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:51:48.783 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:48.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:48.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:51:48.884 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:48.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:48.914 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:48.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:49.015 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:49.041 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T13:51:49.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:49.053 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:49.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:49.118 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T13:51:49.154 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:49.182 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:49.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:49.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:49.193 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-09-21T13:51:49.284 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:49.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:49.339 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:49.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:49.439 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:49.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:49 smithi187 ceph-mon[33032]: pgmap v122: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 8.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 62 MiB/s wr, 1.10k op/s; 50860/407901 objects degraded (12.469%) 2024-09-21T13:51:49.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:49 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3793150145' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:51:49.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:49 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:51:49.445 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:51:49 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:51:49.470 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:49.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:49.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:49.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:49 smithi049 ceph-mon[30080]: pgmap v122: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 8.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 62 MiB/s wr, 1.10k op/s; 50860/407901 objects degraded (12.469%) 2024-09-21T13:51:49.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:49 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3793150145' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:51:49.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:49 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:51:49.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:49 smithi049 ceph-mon[40489]: pgmap v122: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 8.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 62 MiB/s wr, 1.10k op/s; 50860/407901 objects degraded (12.469%) 2024-09-21T13:51:49.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:49 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3793150145' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:51:49.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:49 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:51:49.506 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:51:49 smithi049 podman[89079]: 2024-09-21 13:51:49.249536436 +0000 UTC m=+2.057820324 container create 2e17dd8c18fb3e87d17ce8a824943a01743e16a77a84111cd64ad7d78f8db2a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-2-activate, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-21-0842-4417849, io.buildah.version=1.37.2) 2024-09-21T13:51:49.572 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:49.604 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:49.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:49.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:49.705 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:51:49.741 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:49.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:49.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:51:49.844 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:51:49.876 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:49.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:49.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:51:49.978 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:51:50.003 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:50.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:50.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:51:50.105 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:51:50.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:50.142 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:50.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:51:50.243 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:51:50.276 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:50.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:50.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:51:50.378 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:50.409 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:50.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:50.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:50.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:50 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:51:50.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:50 smithi049 ceph-mon[30080]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T13:51:50.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:50 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T13:51:50.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:50 smithi049 ceph-mon[30080]: osdmap e124: 8 total, 7 up, 8 in 2024-09-21T13:51:50.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:50 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 50860/407901 objects degraded (12.469%), 20 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-09-21T13:51:50.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:50 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:51:50.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:50 smithi049 ceph-mon[40489]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T13:51:50.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:50 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T13:51:50.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:50 smithi049 ceph-mon[40489]: osdmap e124: 8 total, 7 up, 8 in 2024-09-21T13:51:50.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:50 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 50860/407901 objects degraded (12.469%), 20 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-09-21T13:51:50.511 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:51:50.547 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:50.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:50.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:51:50.649 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:50.681 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:50.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:50.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:50.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:50 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:51:50.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:50 smithi187 ceph-mon[33032]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T13:51:50.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:50 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T13:51:50.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:50 smithi187 ceph-mon[33032]: osdmap e124: 8 total, 7 up, 8 in 2024-09-21T13:51:50.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:50 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 50860/407901 objects degraded (12.469%), 20 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-09-21T13:51:50.783 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:50.813 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:50.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:50.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:50.915 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:50.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:50.950 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:50.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:51.004 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:51:50 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:51:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:51:51.050 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:51:51.109 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:51.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:51.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:51:51.211 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:51.241 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:51.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:51.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:51.296 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:51 smithi049 ceph-mon[40489]: pgmap v124: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 7.9 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.09k op/s; 48439/388344 objects degraded (12.473%) 2024-09-21T13:51:51.296 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:51 smithi049 ceph-mon[40489]: 4.2 scrub starts 2024-09-21T13:51:51.296 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:51 smithi049 ceph-mon[40489]: 4.2 scrub ok 2024-09-21T13:51:51.297 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:51 smithi049 ceph-mon[30080]: pgmap v124: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 7.9 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.09k op/s; 48439/388344 objects degraded (12.473%) 2024-09-21T13:51:51.297 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:51 smithi049 ceph-mon[30080]: 4.2 scrub starts 2024-09-21T13:51:51.297 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:51 smithi049 ceph-mon[30080]: 4.2 scrub ok 2024-09-21T13:51:51.342 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:51.379 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:51.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:51.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:51.481 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:51:51.527 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:51.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:51.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:51:51.629 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:51.630 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:51:51.653 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:51:51.656 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:51.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:51.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:51.668 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:51:51 smithi049 podman[89079]: 2024-09-21 13:51:51.402312076 +0000 UTC m=+4.210595971 container init 2e17dd8c18fb3e87d17ce8a824943a01743e16a77a84111cd64ad7d78f8db2a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-2-activate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-21-0842-4417849, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T13:51:51.668 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:51:51 smithi049 podman[89079]: 2024-09-21 13:51:51.41105537 +0000 UTC m=+4.219339257 container start 2e17dd8c18fb3e87d17ce8a824943a01743e16a77a84111cd64ad7d78f8db2a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-2-activate, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T13:51:51.669 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:51:51 smithi049 podman[89079]: 2024-09-21 13:51:51.552377533 +0000 UTC m=+4.360661433 container attach 2e17dd8c18fb3e87d17ce8a824943a01743e16a77a84111cd64ad7d78f8db2a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-2-activate, RELEASE=wip-guits-main-2024-09-21-0842-4417849, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2) 2024-09-21T13:51:51.669 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:51:51 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-2-activate[89480]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T13:51:51.669 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:51:51 smithi049 bash[89079]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T13:51:51.669 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:51:51 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-2-activate[89480]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T13:51:51.669 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:51:51 smithi049 bash[89079]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T13:51:51.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:51 smithi187 ceph-mon[33032]: pgmap v124: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 7.9 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.09k op/s; 48439/388344 objects degraded (12.473%) 2024-09-21T13:51:51.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:51 smithi187 ceph-mon[33032]: 4.2 scrub starts 2024-09-21T13:51:51.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:51 smithi187 ceph-mon[33032]: 4.2 scrub ok 2024-09-21T13:51:51.758 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:51.808 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:51.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:51.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:51.910 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:51:51.946 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:51.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:51.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:51:52.048 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:51:52.089 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:52.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:52.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:51:52.191 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:52.226 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:52.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:52.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:52.328 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:52.357 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:52.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:52.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:52.459 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:51:52.476 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T13:51:52.476 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:51:52.497 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:52.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:52.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:51:52.599 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:52.624 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:52.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:52.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:52.726 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:52.753 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:52.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:52.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:52.855 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:51:52.884 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:52.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:52.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:51:52.986 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:53.014 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:53.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:53.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:53.090 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T13:51:53.116 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:51:53.180 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:53.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:53.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:51:53.252 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:51:53.252 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:51:53.282 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:53.298 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:51:53.298 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:51:53.305 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-21T13:51:53.307 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:53.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:53.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:53.323 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:53 smithi049 ceph-mon[40489]: pgmap v125: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 7.9 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 678 B/s rd, 0 B/s wr, 864 op/s; 48439/388344 objects degraded (12.473%) 2024-09-21T13:51:53.323 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:53 smithi049 ceph-mon[40489]: 6.c scrub starts 2024-09-21T13:51:53.323 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:53 smithi049 ceph-mon[40489]: 6.c scrub ok 2024-09-21T13:51:53.323 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:53 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:51:53.323 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:53 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:51:53.323 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:53 smithi049 ceph-mon[40489]: osdmap e125: 8 total, 7 up, 8 in 2024-09-21T13:51:53.323 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:53 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2168252580' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:51:53.325 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:53 smithi049 ceph-mon[30080]: pgmap v125: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 7.9 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 678 B/s rd, 0 B/s wr, 864 op/s; 48439/388344 objects degraded (12.473%) 2024-09-21T13:51:53.325 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:53 smithi049 ceph-mon[30080]: 6.c scrub starts 2024-09-21T13:51:53.325 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:53 smithi049 ceph-mon[30080]: 6.c scrub ok 2024-09-21T13:51:53.325 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:53 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:51:53.325 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:53 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:51:53.325 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:53 smithi049 ceph-mon[30080]: osdmap e125: 8 total, 7 up, 8 in 2024-09-21T13:51:53.326 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:53 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2168252580' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:51:53.339 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:51:53.339 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:51:53.375 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:51:53.375 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:51:53.408 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:51:53.416 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:51:53.416 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:51:53.433 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:53.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:53.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:51:53.452 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:51:53.452 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:51:53.490 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:51:53.490 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:51:53.534 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:51:53.564 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:53.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:53.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:51:53.666 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:53.684 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T13:51:53.692 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:53.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:53.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:53.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:53 smithi187 ceph-mon[33032]: pgmap v125: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 7.9 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 678 B/s rd, 0 B/s wr, 864 op/s; 48439/388344 objects degraded (12.473%) 2024-09-21T13:51:53.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:53 smithi187 ceph-mon[33032]: 6.c scrub starts 2024-09-21T13:51:53.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:53 smithi187 ceph-mon[33032]: 6.c scrub ok 2024-09-21T13:51:53.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:53 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:51:53.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:53 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:51:53.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:53 smithi187 ceph-mon[33032]: osdmap e125: 8 total, 7 up, 8 in 2024-09-21T13:51:53.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:53 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2168252580' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:51:53.793 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:53.818 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:53.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:53.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:53.920 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:51:53.946 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:53.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:53.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:51:54.048 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:51:54.071 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:54.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:54.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:51:54.172 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:54.200 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:54.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:54.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:54.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:54 smithi049 ceph-mon[40489]: 4.1d scrub starts 2024-09-21T13:51:54.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:54 smithi049 ceph-mon[40489]: 4.1d scrub ok 2024-09-21T13:51:54.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:54 smithi049 ceph-mon[40489]: 5.7 scrub starts 2024-09-21T13:51:54.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:54 smithi049 ceph-mon[40489]: 5.7 scrub ok 2024-09-21T13:51:54.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:54 smithi049 ceph-mon[30080]: 4.1d scrub starts 2024-09-21T13:51:54.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:54 smithi049 ceph-mon[30080]: 4.1d scrub ok 2024-09-21T13:51:54.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:54 smithi049 ceph-mon[30080]: 5.7 scrub starts 2024-09-21T13:51:54.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:54 smithi049 ceph-mon[30080]: 5.7 scrub ok 2024-09-21T13:51:54.256 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:51:54 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-2-activate[89480]: --> Failed to activate via raw: did not find any matching OSD to activate 2024-09-21T13:51:54.256 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:51:54 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-2-activate[89480]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T13:51:54.256 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:51:54 smithi049 bash[89079]: --> Failed to activate via raw: did not find any matching OSD to activate 2024-09-21T13:51:54.256 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:51:54 smithi049 bash[89079]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T13:51:54.256 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:51:54 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-2-activate[89480]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T13:51:54.256 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:51:54 smithi049 bash[89079]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T13:51:54.256 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:51:54 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-2-activate[89480]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-21T13:51:54.256 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:51:54 smithi049 bash[89079]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-21T13:51:54.256 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:51:54 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-2-activate[89480]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_2 --path /var/lib/ceph/osd/ceph-2 --no-mon-config 2024-09-21T13:51:54.256 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:51:54 smithi049 bash[89079]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_2 --path /var/lib/ceph/osd/ceph-2 --no-mon-config 2024-09-21T13:51:54.302 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:51:54.338 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:54.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:54.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:51:54.415 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T13:51:54.416 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T13:51:54.440 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:54.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:54.483 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:54.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:54.584 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:51:54.623 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:54.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:54.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:51:54.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:54 smithi187 ceph-mon[33032]: 4.1d scrub starts 2024-09-21T13:51:54.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:54 smithi187 ceph-mon[33032]: 4.1d scrub ok 2024-09-21T13:51:54.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:54 smithi187 ceph-mon[33032]: 5.7 scrub starts 2024-09-21T13:51:54.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:54 smithi187 ceph-mon[33032]: 5.7 scrub ok 2024-09-21T13:51:54.725 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:51:54.755 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:51:54 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-2-activate[89480]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-2/block 2024-09-21T13:51:54.755 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:51:54 smithi049 bash[89079]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-2/block 2024-09-21T13:51:54.755 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:51:54 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-2-activate[89480]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block 2024-09-21T13:51:54.755 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:51:54 smithi049 bash[89079]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block 2024-09-21T13:51:54.755 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:51:54 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-2-activate[89480]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-21T13:51:54.755 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:51:54 smithi049 bash[89079]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-21T13:51:54.755 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:51:54 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-2-activate[89480]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-21T13:51:54.755 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:51:54 smithi049 bash[89079]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-21T13:51:54.755 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:51:54 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-2-activate[89480]: --> ceph-volume lvm activate successful for osd ID: 2 2024-09-21T13:51:54.756 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:51:54 smithi049 bash[89079]: --> ceph-volume lvm activate successful for osd ID: 2 2024-09-21T13:51:54.756 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:51:54 smithi049 podman[89079]: 2024-09-21 13:51:54.458076857 +0000 UTC m=+7.266360752 container died 2e17dd8c18fb3e87d17ce8a824943a01743e16a77a84111cd64ad7d78f8db2a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-2-activate, RELEASE=wip-guits-main-2024-09-21-0842-4417849, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T13:51:54.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:54.757 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:54.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:51:54.858 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:54.891 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:54.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:54.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:54.993 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:55.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:55.026 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:55.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:55.127 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:55.156 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:55.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:55.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:55.257 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:55.285 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:55.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:55.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:55.372 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:55 smithi049 ceph-mon[40489]: pgmap v127: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 7.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.79k op/s; 45808/367446 objects degraded (12.467%) 2024-09-21T13:51:55.372 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:55 smithi049 ceph-mon[40489]: 3.10 scrub starts 2024-09-21T13:51:55.372 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:55 smithi049 ceph-mon[40489]: 3.10 scrub ok 2024-09-21T13:51:55.372 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:55 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 45808/367446 objects degraded (12.467%), 20 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-09-21T13:51:55.372 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:55 smithi049 ceph-mon[40489]: osdmap e126: 8 total, 7 up, 8 in 2024-09-21T13:51:55.372 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:55 smithi049 ceph-mon[30080]: pgmap v127: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 7.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.79k op/s; 45808/367446 objects degraded (12.467%) 2024-09-21T13:51:55.373 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:55 smithi049 ceph-mon[30080]: 3.10 scrub starts 2024-09-21T13:51:55.373 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:55 smithi049 ceph-mon[30080]: 3.10 scrub ok 2024-09-21T13:51:55.373 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:55 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 45808/367446 objects degraded (12.467%), 20 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-09-21T13:51:55.373 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:55 smithi049 ceph-mon[30080]: osdmap e126: 8 total, 7 up, 8 in 2024-09-21T13:51:55.387 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:55.508 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:51:55.508 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:51:55.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:55.539 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:55.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:55.639 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:51:55.682 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:55.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:55.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:51:55.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:55 smithi187 ceph-mon[33032]: pgmap v127: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 7.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.79k op/s; 45808/367446 objects degraded (12.467%) 2024-09-21T13:51:55.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:55 smithi187 ceph-mon[33032]: 3.10 scrub starts 2024-09-21T13:51:55.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:55 smithi187 ceph-mon[33032]: 3.10 scrub ok 2024-09-21T13:51:55.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:55 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 45808/367446 objects degraded (12.467%), 20 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-09-21T13:51:55.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:55 smithi187 ceph-mon[33032]: osdmap e126: 8 total, 7 up, 8 in 2024-09-21T13:51:55.726 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:51:55 smithi049 podman[90099]: 2024-09-21 13:51:55.372116336 +0000 UTC m=+0.907988355 container remove 2e17dd8c18fb3e87d17ce8a824943a01743e16a77a84111cd64ad7d78f8db2a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-2-activate, ceph=True, RELEASE=wip-guits-main-2024-09-21-0842-4417849, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-21T13:51:55.786 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:51:55.825 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:55.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:55.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:51:55.927 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:51:55.966 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:55.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:55.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:51:56.068 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:51:56.103 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:56.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:56.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:51:56.205 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:51:56.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:56.255 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:56.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:51:56.355 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:56 smithi049 ceph-mon[40489]: 4.16 scrub starts 2024-09-21T13:51:56.355 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:56 smithi049 ceph-mon[40489]: 4.16 scrub ok 2024-09-21T13:51:56.355 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:56 smithi049 ceph-mon[40489]: 4.18 scrub starts 2024-09-21T13:51:56.355 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:56 smithi049 ceph-mon[40489]: 4.18 scrub ok 2024-09-21T13:51:56.355 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:56 smithi049 ceph-mon[40489]: 5.16 scrub starts 2024-09-21T13:51:56.356 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:56 smithi049 ceph-mon[40489]: 5.16 scrub ok 2024-09-21T13:51:56.356 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:56 smithi049 ceph-mon[30080]: 4.16 scrub starts 2024-09-21T13:51:56.356 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:56 smithi049 ceph-mon[30080]: 4.16 scrub ok 2024-09-21T13:51:56.356 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:56 smithi049 ceph-mon[30080]: 4.18 scrub starts 2024-09-21T13:51:56.356 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:56 smithi049 ceph-mon[30080]: 4.18 scrub ok 2024-09-21T13:51:56.356 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:56 smithi049 ceph-mon[30080]: 5.16 scrub starts 2024-09-21T13:51:56.356 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:56 smithi049 ceph-mon[30080]: 5.16 scrub ok 2024-09-21T13:51:56.356 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:51:56.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:56.395 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:56.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:51:56.417 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T13:51:56.432 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T13:51:56.432 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-09-21T13:51:56.432 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.432 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.14817.0:154467 7.2 e126/101 7:42b04ff7:::benchmark_data_smithi049_76798_object18047:head v 126'9530, mlcod=126'9528)", 2024-09-21T13:51:56.433 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:51:56.430012+0000", 2024-09-21T13:51:56.433 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.0010259119999999999, 2024-09-21T13:51:56.433 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0010506280000000001, 2024-09-21T13:51:56.433 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:51:56.433 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:51:56.433 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T13:51:56.433 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:51:56.433 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.433 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:51:56.433 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:56.430012+0000", 2024-09-21T13:51:56.433 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:56.433 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:56.433 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.433 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:51:56.434 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:56.430012+0000", 2024-09-21T13:51:56.434 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:56.434 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:56.434 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.434 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:51:56.434 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:56.430014+0000", 2024-09-21T13:51:56.434 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.062e-06 2024-09-21T13:51:56.434 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:56.434 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.434 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:51:56.434 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:56.430019+0000", 2024-09-21T13:51:56.434 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6960000000000002e-06 2024-09-21T13:51:56.434 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:56.434 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.435 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:51:56.435 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:56.430031+0000", 2024-09-21T13:51:56.435 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1365e-05 2024-09-21T13:51:56.435 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:56.435 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.435 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:51:56.435 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:56.430037+0000", 2024-09-21T13:51:56.435 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8900000000000001e-06 2024-09-21T13:51:56.435 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:56.435 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.435 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:51:56.435 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:56.430120+0000", 2024-09-21T13:51:56.435 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2384000000000003e-05 2024-09-21T13:51:56.435 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:56.436 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.436 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:51:56.436 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:56.430142+0000", 2024-09-21T13:51:56.436 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2396999999999999e-05 2024-09-21T13:51:56.436 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:56.436 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:51:56.436 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:56.436 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:56.436 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.436 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:154452 7.b 7:d5b9e403:::benchmark_data_smithi049_76798_object18032:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-21T13:51:56.436 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:51:56.418515+0000", 2024-09-21T13:51:56.436 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.012523276999999999, 2024-09-21T13:51:56.436 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.012607940999999999, 2024-09-21T13:51:56.436 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:51:56.437 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:51:56.437 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "waiting for sub ops", 2024-09-21T13:51:56.437 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:51:56.437 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:51:56.437 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:56.437 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 154452 2024-09-21T13:51:56.437 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:56.437 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:51:56.437 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.437 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:51:56.437 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:56.418515+0000", 2024-09-21T13:51:56.437 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:56.437 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:56.437 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.437 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:51:56.438 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:56.418515+0000", 2024-09-21T13:51:56.438 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:56.438 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:56.438 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.438 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:51:56.438 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:56.418517+0000", 2024-09-21T13:51:56.438 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1090000000000001e-06 2024-09-21T13:51:56.438 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:56.438 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.438 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:51:56.438 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:56.418524+0000", 2024-09-21T13:51:56.438 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2110000000000001e-06 2024-09-21T13:51:56.438 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:56.438 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.438 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:51:56.439 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:56.418544+0000", 2024-09-21T13:51:56.439 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9211e-05 2024-09-21T13:51:56.439 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:56.439 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.439 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:51:56.439 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:56.418552+0000", 2024-09-21T13:51:56.439 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.8389999999999996e-06 2024-09-21T13:51:56.439 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:56.439 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.439 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:51:56.439 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:56.418636+0000", 2024-09-21T13:51:56.439 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3702000000000006e-05 2024-09-21T13:51:56.439 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:56.439 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.439 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:51:56.440 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:56.418751+0000", 2024-09-21T13:51:56.440 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000115277 2024-09-21T13:51:56.440 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:56.440 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.440 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,6", 2024-09-21T13:51:56.440 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:56.418868+0000", 2024-09-21T13:51:56.440 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000116444 2024-09-21T13:51:56.440 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:56.440 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.440 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:56.440 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:56.424192+0000", 2024-09-21T13:51:56.440 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0053244720000000002 2024-09-21T13:51:56.440 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:56.440 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.440 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:56.440 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:56.424579+0000", 2024-09-21T13:51:56.441 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00038642700000000001 2024-09-21T13:51:56.441 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:56.441 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:51:56.441 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:56.441 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:56.441 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.441 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.14817.0:154456 7.0 e126/83 7:0eb3de49:::benchmark_data_smithi049_76798_object18036:head v 126'9679, mlcod=126'9673)", 2024-09-21T13:51:56.441 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:51:56.419019+0000", 2024-09-21T13:51:56.441 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.012019748, 2024-09-21T13:51:56.441 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.012176418, 2024-09-21T13:51:56.441 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:51:56.441 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:51:56.441 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T13:51:56.441 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:51:56.441 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.441 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:51:56.442 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:56.419019+0000", 2024-09-21T13:51:56.442 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:56.442 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:56.442 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.442 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:51:56.442 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:56.419019+0000", 2024-09-21T13:51:56.442 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:56.442 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:56.442 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.442 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:51:56.442 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:56.419020+0000", 2024-09-21T13:51:56.442 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2900000000000002e-07 2024-09-21T13:51:56.442 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:56.442 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.443 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:51:56.443 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:56.419025+0000", 2024-09-21T13:51:56.443 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8289999999999996e-06 2024-09-21T13:51:56.443 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:56.443 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.443 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:51:56.443 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:56.419048+0000", 2024-09-21T13:51:56.443 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2277999999999999e-05 2024-09-21T13:51:56.443 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:56.443 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.443 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:51:56.443 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:56.419055+0000", 2024-09-21T13:51:56.443 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5000000000000002e-06 2024-09-21T13:51:56.443 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:56.443 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.444 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:51:56.444 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:56.419140+0000", 2024-09-21T13:51:56.444 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5031000000000006e-05 2024-09-21T13:51:56.444 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:56.444 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.444 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:51:56.444 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:56.419161+0000", 2024-09-21T13:51:56.444 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0641000000000001e-05 2024-09-21T13:51:56.444 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:56.444 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:51:56.444 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:56.444 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:56.444 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.444 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:154464 7.9 7:9b5158a1:::benchmark_data_smithi049_76798_object18044:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-21T13:51:56.445 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:51:56.426385+0000", 2024-09-21T13:51:56.445 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.0046534100000000002, 2024-09-21T13:51:56.445 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0048590939999999996, 2024-09-21T13:51:56.445 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:51:56.445 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:51:56.445 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "waiting for sub ops", 2024-09-21T13:51:56.445 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:51:56.445 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:51:56.445 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:51:56.445 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 154464 2024-09-21T13:51:56.445 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:56.445 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:51:56.445 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.446 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:51:56.446 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:56.426385+0000", 2024-09-21T13:51:56.446 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:56.446 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:56.446 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.446 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:51:56.446 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:56.426385+0000", 2024-09-21T13:51:56.446 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:56.446 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:56.447 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.447 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:51:56.447 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:56.426387+0000", 2024-09-21T13:51:56.447 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9920000000000002e-06 2024-09-21T13:51:56.447 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:56.447 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.447 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:51:56.447 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:56.426391+0000", 2024-09-21T13:51:56.447 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.51e-06 2024-09-21T13:51:56.447 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:56.447 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.447 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:51:56.447 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:56.426404+0000", 2024-09-21T13:51:56.447 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2911000000000001e-05 2024-09-21T13:51:56.447 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:56.448 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.448 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:51:56.448 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:56.426411+0000", 2024-09-21T13:51:56.448 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9789999999999996e-06 2024-09-21T13:51:56.448 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:56.448 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.448 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:51:56.448 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:56.426663+0000", 2024-09-21T13:51:56.448 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00025145400000000002 2024-09-21T13:51:56.448 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:56.448 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.448 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:51:56.448 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:56.426780+0000", 2024-09-21T13:51:56.448 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000117539 2024-09-21T13:51:56.449 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:56.449 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.449 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,4", 2024-09-21T13:51:56.449 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:56.426894+0000", 2024-09-21T13:51:56.449 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011381400000000001 2024-09-21T13:51:56.449 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:56.449 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.449 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:51:56.449 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:56.429802+0000", 2024-09-21T13:51:56.449 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0029081469999999998 2024-09-21T13:51:56.449 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:56.449 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.449 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:51:56.449 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:56.429836+0000", 2024-09-21T13:51:56.449 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4008999999999999e-05 2024-09-21T13:51:56.450 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:56.450 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:51:56.450 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:56.450 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:56.450 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.450 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.14817.0:154463 7.0 e126/83 7:07c9a1dc:::benchmark_data_smithi049_76798_object18043:head v 126'9680, mlcod=126'9677)", 2024-09-21T13:51:56.450 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:51:56.428511+0000", 2024-09-21T13:51:56.450 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.0025273040000000002, 2024-09-21T13:51:56.450 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002800463, 2024-09-21T13:51:56.450 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:51:56.450 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:51:56.450 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T13:51:56.450 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:51:56.450 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.451 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:51:56.451 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:56.428511+0000", 2024-09-21T13:51:56.451 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:56.451 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:56.451 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.451 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:51:56.451 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:56.428511+0000", 2024-09-21T13:51:56.451 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:56.451 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:56.451 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.451 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:51:56.451 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:56.428512+0000", 2024-09-21T13:51:56.451 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1289999999999999e-06 2024-09-21T13:51:56.452 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:56.452 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.452 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:51:56.452 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:56.428519+0000", 2024-09-21T13:51:56.452 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1829999999999997e-06 2024-09-21T13:51:56.452 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:56.452 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.452 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:51:56.452 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:56.428536+0000", 2024-09-21T13:51:56.452 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6317e-05 2024-09-21T13:51:56.452 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:56.452 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.452 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:51:56.452 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:56.428543+0000", 2024-09-21T13:51:56.453 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5979999999999999e-06 2024-09-21T13:51:56.453 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:56.453 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.453 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:51:56.453 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:56.428801+0000", 2024-09-21T13:51:56.453 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00025727600000000002 2024-09-21T13:51:56.453 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:56.453 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.453 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:51:56.453 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:56.428821+0000", 2024-09-21T13:51:56.453 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0877999999999999e-05 2024-09-21T13:51:56.453 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:56.453 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:51:56.453 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:56.454 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:56.454 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.454 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.14817.0:154465 7.d e126/83 7:b30cf9b7:::benchmark_data_smithi049_76798_object18045:head v 126'9625, mlcod=126'9623)", 2024-09-21T13:51:56.454 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:51:56.429522+0000", 2024-09-21T13:51:56.454 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.001516859, 2024-09-21T13:51:56.454 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0018373230000000001, 2024-09-21T13:51:56.454 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:51:56.454 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:51:56.454 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T13:51:56.454 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:51:56.454 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.454 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:51:56.454 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:56.429522+0000", 2024-09-21T13:51:56.455 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:56.455 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:56.455 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.455 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:51:56.455 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:56.429522+0000", 2024-09-21T13:51:56.455 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:51:56.455 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:56.455 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.455 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:51:56.455 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:56.429522+0000", 2024-09-21T13:51:56.455 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3e-07 2024-09-21T13:51:56.455 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:56.455 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.455 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:51:56.455 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:56.429528+0000", 2024-09-21T13:51:56.456 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5400000000000003e-06 2024-09-21T13:51:56.456 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:56.456 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.456 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:51:56.456 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:56.429538+0000", 2024-09-21T13:51:56.456 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0514e-05 2024-09-21T13:51:56.456 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:56.456 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.456 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:51:56.456 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:56.429545+0000", 2024-09-21T13:51:56.456 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3040000000000001e-06 2024-09-21T13:51:56.456 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:56.456 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.456 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:51:56.456 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:56.430007+0000", 2024-09-21T13:51:56.457 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00046244100000000001 2024-09-21T13:51:56.457 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:51:56.457 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:51:56.457 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:51:56.457 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:56.430026+0000", 2024-09-21T13:51:56.457 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.889e-05 2024-09-21T13:51:56.457 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:56.457 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:51:56.457 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:56.457 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:51:56.457 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T13:51:56.457 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 6 2024-09-21T13:51:56.457 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T13:51:56.496 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:51:56.526 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:56.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:56.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:51:56.628 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:51:56.664 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:56.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:56.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:51:56.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:56 smithi187 ceph-mon[33032]: 4.16 scrub starts 2024-09-21T13:51:56.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:56 smithi187 ceph-mon[33032]: 4.16 scrub ok 2024-09-21T13:51:56.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:56 smithi187 ceph-mon[33032]: 4.18 scrub starts 2024-09-21T13:51:56.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:56 smithi187 ceph-mon[33032]: 4.18 scrub ok 2024-09-21T13:51:56.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:56 smithi187 ceph-mon[33032]: 5.16 scrub starts 2024-09-21T13:51:56.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:56 smithi187 ceph-mon[33032]: 5.16 scrub ok 2024-09-21T13:51:56.767 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:51:56.798 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:56.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:56.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:51:56.899 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:56.954 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:56.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:56.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:57.056 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:51:57.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:57.096 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:57.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:51:57.197 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:57.229 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:57.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:57.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:57.283 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-09-21T13:51:57.330 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:51:57.357 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:57.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:57.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:51:57.398 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T13:51:57.459 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:51:57.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:57.486 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:57.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:51:57.587 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:57.616 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:57.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:57.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:57.678 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:57 smithi049 ceph-mon[40489]: pgmap v129: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 7.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 785 B/s rd, 0 B/s wr, 1.07k op/s; 45808/367446 objects degraded (12.467%) 2024-09-21T13:51:57.679 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:57 smithi049 ceph-mon[40489]: 6.13 scrub starts 2024-09-21T13:51:57.679 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:57 smithi049 ceph-mon[40489]: 6.13 scrub ok 2024-09-21T13:51:57.679 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:57 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3039857882' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:51:57.679 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:57 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:51:57.679 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:57 smithi049 ceph-mon[30080]: pgmap v129: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 7.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 785 B/s rd, 0 B/s wr, 1.07k op/s; 45808/367446 objects degraded (12.467%) 2024-09-21T13:51:57.679 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:57 smithi049 ceph-mon[30080]: 6.13 scrub starts 2024-09-21T13:51:57.679 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:57 smithi049 ceph-mon[30080]: 6.13 scrub ok 2024-09-21T13:51:57.679 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:57 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3039857882' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:51:57.679 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:57 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:51:57.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:57 smithi187 ceph-mon[33032]: pgmap v129: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 7.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 785 B/s rd, 0 B/s wr, 1.07k op/s; 45808/367446 objects degraded (12.467%) 2024-09-21T13:51:57.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:57 smithi187 ceph-mon[33032]: 6.13 scrub starts 2024-09-21T13:51:57.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:57 smithi187 ceph-mon[33032]: 6.13 scrub ok 2024-09-21T13:51:57.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:57 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3039857882' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:51:57.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:57 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:51:57.718 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:57.750 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:57.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:57.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:57.852 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:51:57.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:57.881 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:57.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:51:57.982 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:51:58.067 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:58.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:58.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:51:58.169 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:58.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:58.200 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:58.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:58.255 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:51:58 smithi049 podman[90306]: 2024-09-21 13:51:58.007161389 +0000 UTC m=+1.984448533 image pull b415816db8b41f544cdfdc71d193ff8b1ee008bf7b0ec4f64da8eb8f71b62e85 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5 2024-09-21T13:51:58.255 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:51:58 smithi049 podman[90306]: 2024-09-21 13:51:58.154203702 +0000 UTC m=+2.131490838 container create 3237a2aba8854d698c9a5eaf24faee6e29558c9ddf9a48b90f332e76bee835c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-2, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-21-0842-4417849, GIT_CLEAN=True) 2024-09-21T13:51:58.301 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:51:58.332 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:58.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:58.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:51:58.434 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:58.465 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:58.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:58.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:58.566 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:58.575 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:58 smithi049 ceph-mon[30080]: 4.c scrub starts 2024-09-21T13:51:58.575 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:58 smithi049 ceph-mon[30080]: 4.c scrub ok 2024-09-21T13:51:58.576 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:58 smithi049 ceph-mon[30080]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:51:58.576 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:58 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T13:51:58.576 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:58 smithi049 ceph-mon[30080]: osdmap e127: 8 total, 7 up, 8 in 2024-09-21T13:51:58.576 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:58 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:51:58.576 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:58 smithi049 ceph-mon[40489]: 4.c scrub starts 2024-09-21T13:51:58.576 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:58 smithi049 ceph-mon[40489]: 4.c scrub ok 2024-09-21T13:51:58.576 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:58 smithi049 ceph-mon[40489]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:51:58.576 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:58 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T13:51:58.576 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:58 smithi049 ceph-mon[40489]: osdmap e127: 8 total, 7 up, 8 in 2024-09-21T13:51:58.576 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:58 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:51:58.578 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:51:58 smithi049 podman[90306]: 2024-09-21 13:51:58.45757492 +0000 UTC m=+2.434862060 container init 3237a2aba8854d698c9a5eaf24faee6e29558c9ddf9a48b90f332e76bee835c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T13:51:58.578 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:51:58 smithi049 podman[90306]: 2024-09-21 13:51:58.470544178 +0000 UTC m=+2.447831316 container start 3237a2aba8854d698c9a5eaf24faee6e29558c9ddf9a48b90f332e76bee835c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-21-0842-4417849, ceph=True) 2024-09-21T13:51:58.578 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:51:58 smithi049 bash[90306]: 3237a2aba8854d698c9a5eaf24faee6e29558c9ddf9a48b90f332e76bee835c3 2024-09-21T13:51:58.583 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T13:51:58.587 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:51:58.591 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:58.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:58.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:58.666 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:51:58.694 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:58.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:58 smithi187 ceph-mon[33032]: 4.c scrub starts 2024-09-21T13:51:58.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:58 smithi187 ceph-mon[33032]: 4.c scrub ok 2024-09-21T13:51:58.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:58 smithi187 ceph-mon[33032]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:51:58.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:58 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T13:51:58.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:58 smithi187 ceph-mon[33032]: osdmap e127: 8 total, 7 up, 8 in 2024-09-21T13:51:58.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:58 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:51:58.718 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:58.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:58.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:58.820 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:51:58.846 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:58.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:58.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:51:58.926 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:51:58 smithi049 systemd[1]: Started Ceph osd.2 for 40a3d2e8-781f-11ef-baf6-efdc52797490. 2024-09-21T13:51:58.948 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:58.975 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:58.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:58.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:59.077 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:59.107 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:59.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:59.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:59.209 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:51:59.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:59.255 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:59.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:51:59.356 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:51:59.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:59.395 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:51:59.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:51:59.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:51:59 smithi187 ceph-mon[33032]: pgmap v131: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 7.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.16k op/s; 45808/367446 objects degraded (12.467%) 2024-09-21T13:51:59.445 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:51:59 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:51:59.496 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:51:59.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:51:59 smithi049 ceph-mon[30080]: pgmap v131: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 7.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.16k op/s; 45808/367446 objects degraded (12.467%) 2024-09-21T13:51:59.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:51:59 smithi049 ceph-mon[40489]: pgmap v131: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 7.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.16k op/s; 45808/367446 objects degraded (12.467%) 2024-09-21T13:51:59.522 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:59.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:59.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:51:59.558 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:51:59.558 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:51:59.570 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:51:59.570 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:51:59.575 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-21T13:51:59.605 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:51:59.605 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:51:59.624 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:51:59.640 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:51:59.641 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:51:59.650 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:59.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:59.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:51:59.683 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:51:59.683 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:51:59.683 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T13:51:59.684 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:51:59.721 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:51:59.722 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:51:59.751 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:51:59.755 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:51:59.755 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:51:59.780 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:59.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:59.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:51:59.816 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:51:59 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-2[90617]: 2024-09-21T13:51:59.616+0000 7fe75feac640 -1 Falling back to public interface 2024-09-21T13:51:59.882 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:51:59.915 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:51:59.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:51:59.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:52:00.017 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:52:00.043 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T13:52:00.065 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:00.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:00.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:52:00.167 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:52:00.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:00.195 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:00.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:52:00.296 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:52:00.318 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:00.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:00.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:52:00.420 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:00.447 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:00.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:00.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:00.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:00 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:52:00.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:00 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1986103818' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:52:00.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:00 smithi049 ceph-mon[30080]: osdmap e128: 8 total, 7 up, 8 in 2024-09-21T13:52:00.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:00 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:52:00.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:00 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1986103818' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:52:00.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:00 smithi049 ceph-mon[40489]: osdmap e128: 8 total, 7 up, 8 in 2024-09-21T13:52:00.550 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:00.577 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:00.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:00.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:00.678 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:52:00.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:00 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:52:00.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:00 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1986103818' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:52:00.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:00 smithi187 ceph-mon[33032]: osdmap e128: 8 total, 7 up, 8 in 2024-09-21T13:52:00.706 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:00.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:00.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:52:00.808 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:52:00.839 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:00.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:00.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:52:00.859 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:52:00 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:52:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:52:00.941 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:52:00.977 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:00.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:00.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:52:01.079 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:01.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:01.116 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:01.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:01.217 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:52:01.229 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T13:52:01.229 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T13:52:01.242 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:01.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:01.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:52:01.344 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:01.369 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:01.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:01.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:01.471 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:52:01.500 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:01.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:01.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:52:01.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:01 smithi049 ceph-mon[40489]: pgmap v132: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 7.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.82k op/s; 43301/347466 objects degraded (12.462%) 2024-09-21T13:52:01.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:01 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 43301/347466 objects degraded (12.462%), 20 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-09-21T13:52:01.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:01 smithi049 ceph-mon[30080]: pgmap v132: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 7.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.82k op/s; 43301/347466 objects degraded (12.462%) 2024-09-21T13:52:01.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:01 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 43301/347466 objects degraded (12.462%), 20 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-09-21T13:52:01.601 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:01.633 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:01.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:01.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:01.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:01 smithi187 ceph-mon[33032]: pgmap v132: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 7.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.82k op/s; 43301/347466 objects degraded (12.462%) 2024-09-21T13:52:01.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:01 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 43301/347466 objects degraded (12.462%), 20 pgs degraded, 44 pgs undersized (PG_DEGRADED) 2024-09-21T13:52:01.735 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:52:01.753 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T13:52:01.758 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:01.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:01.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:52:01.760 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:52:01.797 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:52:01.860 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:52:01.923 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:01.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:01.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:52:02.025 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:02.057 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:02.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:02.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:02.158 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:52:02.208 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:02.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:02.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:52:02.255 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:52:02 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-2[90617]: 2024-09-21T13:52:01.999+0000 7fe75feac640 -1 osd.2 100 log_to_monitors true 2024-09-21T13:52:02.312 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:52:02.339 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:02.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:02.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:52:02.440 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:52:02.478 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:02.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:02.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:52:02.540 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:02 smithi049 ceph-mon[40489]: from='osd.2 [v2:172.21.15.49:6818/1945314741,v1:172.21.15.49:6819/1945314741]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T13:52:02.540 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:02 smithi049 ceph-mon[40489]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T13:52:02.541 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 13:52:02 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-2[90617]: 2024-09-21T13:52:02.346+0000 7fe753c4e640 -1 osd.2 100 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T13:52:02.541 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:02 smithi049 ceph-mon[30080]: from='osd.2 [v2:172.21.15.49:6818/1945314741,v1:172.21.15.49:6819/1945314741]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T13:52:02.541 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:02 smithi049 ceph-mon[30080]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T13:52:02.580 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:52:02.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:02.622 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:02.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:52:02.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:02 smithi187 ceph-mon[33032]: from='osd.2 [v2:172.21.15.49:6818/1945314741,v1:172.21.15.49:6819/1945314741]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T13:52:02.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:02 smithi187 ceph-mon[33032]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T13:52:02.723 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:52:02.748 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:02.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:02.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:52:02.780 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T13:52:02.780 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-09-21T13:52:02.780 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 0 2024-09-21T13:52:02.780 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T13:52:02.850 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:52:02.885 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:02.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:02.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:52:02.987 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:52:03.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:03.031 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:03.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:52:03.132 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:03.160 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:03.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:03.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:03.230 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T13:52:03.262 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:52:03.289 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:03.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:03.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:52:03.391 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:52:03.413 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:03.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:03.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:52:03.496 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd reweight 0 1 2024-09-21T13:52:03.516 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:52:03.561 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:03.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:03.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:52:03.597 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-09-21T13:52:03.597 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:03 smithi049 ceph-mon[40489]: pgmap v134: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 7.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 746 B/s rd, 0 B/s wr, 971 op/s; 43301/347466 objects degraded (12.462%) 2024-09-21T13:52:03.598 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:03 smithi049 ceph-mon[40489]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T13:52:03.598 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:03 smithi049 ceph-mon[40489]: osdmap e129: 8 total, 7 up, 8 in 2024-09-21T13:52:03.598 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:03 smithi049 ceph-mon[40489]: from='osd.2 [v2:172.21.15.49:6818/1945314741,v1:172.21.15.49:6819/1945314741]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi049", "root=default"]} : dispatch 2024-09-21T13:52:03.598 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:03 smithi049 ceph-mon[40489]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi049", "root=default"]} : dispatch 2024-09-21T13:52:03.598 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:03 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/642314905' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:52:03.599 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:03 smithi049 ceph-mon[30080]: pgmap v134: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 7.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 746 B/s rd, 0 B/s wr, 971 op/s; 43301/347466 objects degraded (12.462%) 2024-09-21T13:52:03.599 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:03 smithi049 ceph-mon[30080]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T13:52:03.599 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:03 smithi049 ceph-mon[30080]: osdmap e129: 8 total, 7 up, 8 in 2024-09-21T13:52:03.599 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:03 smithi049 ceph-mon[30080]: from='osd.2 [v2:172.21.15.49:6818/1945314741,v1:172.21.15.49:6819/1945314741]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi049", "root=default"]} : dispatch 2024-09-21T13:52:03.599 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:03 smithi049 ceph-mon[30080]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi049", "root=default"]} : dispatch 2024-09-21T13:52:03.600 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:03 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/642314905' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:52:03.663 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:52:03.686 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:03.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:03.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:52:03.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:03 smithi187 ceph-mon[33032]: pgmap v134: 148 pgs: 24 active+undersized, 20 active+undersized+degraded, 104 active+clean; 7.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 746 B/s rd, 0 B/s wr, 971 op/s; 43301/347466 objects degraded (12.462%) 2024-09-21T13:52:03.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:03 smithi187 ceph-mon[33032]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T13:52:03.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:03 smithi187 ceph-mon[33032]: osdmap e129: 8 total, 7 up, 8 in 2024-09-21T13:52:03.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:03 smithi187 ceph-mon[33032]: from='osd.2 [v2:172.21.15.49:6818/1945314741,v1:172.21.15.49:6819/1945314741]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi049", "root=default"]} : dispatch 2024-09-21T13:52:03.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:03 smithi187 ceph-mon[33032]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi049", "root=default"]} : dispatch 2024-09-21T13:52:03.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:03 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/642314905' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:52:03.788 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:52:03.810 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:03.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:03.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:52:03.911 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:03.935 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:03.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:03.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:04.037 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:52:04.059 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:04.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:04.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:52:04.160 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:52:04.182 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:04.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:04.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:52:04.283 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:04.312 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:04.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:04.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:04.365 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:52:04.388 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:52:04.414 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:52:04.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:04.439 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:04.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:52:04.539 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:52:04.562 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:04.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:04.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:52:04.606 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:04 smithi049 ceph-mon[30080]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:52:04.606 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:04 smithi049 ceph-mon[30080]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T13:52:04.606 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:04 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/642314905' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T13:52:04.606 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:04 smithi049 ceph-mon[30080]: osd.2 [v2:172.21.15.49:6818/1945314741,v1:172.21.15.49:6819/1945314741] boot 2024-09-21T13:52:04.607 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:04 smithi049 ceph-mon[30080]: osdmap e130: 8 total, 8 up, 8 in 2024-09-21T13:52:04.607 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:04 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T13:52:04.607 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:04 smithi049 ceph-mon[30080]: osdmap e131: 8 total, 8 up, 8 in 2024-09-21T13:52:04.607 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:04 smithi049 ceph-mon[40489]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:52:04.607 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:04 smithi049 ceph-mon[40489]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T13:52:04.607 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:04 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/642314905' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T13:52:04.607 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:04 smithi049 ceph-mon[40489]: osd.2 [v2:172.21.15.49:6818/1945314741,v1:172.21.15.49:6819/1945314741] boot 2024-09-21T13:52:04.607 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:04 smithi049 ceph-mon[40489]: osdmap e130: 8 total, 8 up, 8 in 2024-09-21T13:52:04.607 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:04 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T13:52:04.607 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:04 smithi049 ceph-mon[40489]: osdmap e131: 8 total, 8 up, 8 in 2024-09-21T13:52:04.664 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:52:04.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:04.689 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:04.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:52:04.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:04 smithi187 ceph-mon[33032]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:52:04.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:04 smithi187 ceph-mon[33032]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T13:52:04.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:04 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/642314905' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T13:52:04.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:04 smithi187 ceph-mon[33032]: osd.2 [v2:172.21.15.49:6818/1945314741,v1:172.21.15.49:6819/1945314741] boot 2024-09-21T13:52:04.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:04 smithi187 ceph-mon[33032]: osdmap e130: 8 total, 8 up, 8 in 2024-09-21T13:52:04.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:04 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T13:52:04.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:04 smithi187 ceph-mon[33032]: osdmap e131: 8 total, 8 up, 8 in 2024-09-21T13:52:04.790 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:04.814 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:04.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:04.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:04.915 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:04.938 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:04.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:04.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:05.040 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:52:05.080 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:05.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:05.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:52:05.135 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T13:52:05.135 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:52:05.182 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:52:05.238 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:05.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:05.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:52:05.339 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:52:05.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:05.394 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:05.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:52:05.398 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:52:05.399 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:52:05.399 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:05 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:52:05.399 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:05 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:52:05.399 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:05 smithi049 ceph-mon[30080]: pgmap v137: 148 pgs: 34 peering, 5 active+undersized, 5 active+undersized+degraded, 104 active+clean; 6.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2.04k op/s; 14342/330807 objects degraded (4.335%) 2024-09-21T13:52:05.399 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:05 smithi049 ceph-mon[30080]: 7.8 starting backfill to osd.2 from (0'0,100'3487] MAX to 129'10239 2024-09-21T13:52:05.399 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:05 smithi049 ceph-mon[30080]: osdmap e132: 8 total, 8 up, 8 in 2024-09-21T13:52:05.399 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:05 smithi049 ceph-mon[30080]: 7.5 starting backfill to osd.2 from (0'0,100'3480] MAX to 129'10213 2024-09-21T13:52:05.400 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:05 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:52:05.400 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:05 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:52:05.400 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:05 smithi049 ceph-mon[40489]: pgmap v137: 148 pgs: 34 peering, 5 active+undersized, 5 active+undersized+degraded, 104 active+clean; 6.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2.04k op/s; 14342/330807 objects degraded (4.335%) 2024-09-21T13:52:05.400 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:05 smithi049 ceph-mon[40489]: 7.8 starting backfill to osd.2 from (0'0,100'3487] MAX to 129'10239 2024-09-21T13:52:05.400 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:05 smithi049 ceph-mon[40489]: osdmap e132: 8 total, 8 up, 8 in 2024-09-21T13:52:05.400 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:05 smithi049 ceph-mon[40489]: 7.5 starting backfill to osd.2 from (0'0,100'3480] MAX to 129'10213 2024-09-21T13:52:05.440 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:52:05.441 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:52:05.478 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:52:05.478 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_enable_op_tracker = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-09-21T13:52:05.496 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:52:05.526 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:52:05.526 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:52:05.539 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:05.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:05.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:52:05.560 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:52:05.561 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:52:05.600 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:52:05.601 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:52:05.634 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:52:05.635 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:52:05.641 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:52:05.669 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:52:05.669 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:52:05.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:05 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:52:05.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:05 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:52:05.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:05 smithi187 ceph-mon[33032]: pgmap v137: 148 pgs: 34 peering, 5 active+undersized, 5 active+undersized+degraded, 104 active+clean; 6.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2.04k op/s; 14342/330807 objects degraded (4.335%) 2024-09-21T13:52:05.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:05 smithi187 ceph-mon[33032]: 7.8 starting backfill to osd.2 from (0'0,100'3487] MAX to 129'10239 2024-09-21T13:52:05.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:05 smithi187 ceph-mon[33032]: osdmap e132: 8 total, 8 up, 8 in 2024-09-21T13:52:05.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:05 smithi187 ceph-mon[33032]: 7.5 starting backfill to osd.2 from (0'0,100'3480] MAX to 129'10213 2024-09-21T13:52:05.698 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:05.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:05.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:52:05.800 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:52:05.829 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:05.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:05.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:52:05.930 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:52:05.955 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:05.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:05.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:52:06.057 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:52:06.098 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:06.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:06.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:52:06.200 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:52:06.245 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:06.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:06.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:52:06.347 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:52:06.352 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T13:52:06.359 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T13:52:06.384 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:06.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:06.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:52:06.486 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:52:06.510 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:06.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:06.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:52:06.612 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:52:06.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:06.676 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:06.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:52:06.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:06 smithi187 ceph-mon[33032]: 7.3 starting backfill to osd.2 from (0'0,100'3524] MAX to 129'10338 2024-09-21T13:52:06.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:06 smithi187 ceph-mon[33032]: 7.2 starting backfill to osd.2 from (0'0,100'3392] MAX to 129'10066 2024-09-21T13:52:06.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:06 smithi187 ceph-mon[33032]: 7.a starting backfill to osd.2 from (0'0,100'3496] MAX to 129'10210 2024-09-21T13:52:06.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:06 smithi187 ceph-mon[33032]: 7.e starting backfill to osd.2 from (0'0,100'3533] MAX to 129'10243 2024-09-21T13:52:06.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:06 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2976538272' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:52:06.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:06 smithi049 ceph-mon[30080]: 7.3 starting backfill to osd.2 from (0'0,100'3524] MAX to 129'10338 2024-09-21T13:52:06.709 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:06 smithi049 ceph-mon[30080]: 7.2 starting backfill to osd.2 from (0'0,100'3392] MAX to 129'10066 2024-09-21T13:52:06.709 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:06 smithi049 ceph-mon[30080]: 7.a starting backfill to osd.2 from (0'0,100'3496] MAX to 129'10210 2024-09-21T13:52:06.709 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:06 smithi049 ceph-mon[30080]: 7.e starting backfill to osd.2 from (0'0,100'3533] MAX to 129'10243 2024-09-21T13:52:06.709 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:06 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2976538272' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:52:06.709 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:06 smithi049 ceph-mon[40489]: 7.3 starting backfill to osd.2 from (0'0,100'3524] MAX to 129'10338 2024-09-21T13:52:06.709 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:06 smithi049 ceph-mon[40489]: 7.2 starting backfill to osd.2 from (0'0,100'3392] MAX to 129'10066 2024-09-21T13:52:06.709 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:06 smithi049 ceph-mon[40489]: 7.a starting backfill to osd.2 from (0'0,100'3496] MAX to 129'10210 2024-09-21T13:52:06.709 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:06 smithi049 ceph-mon[40489]: 7.e starting backfill to osd.2 from (0'0,100'3533] MAX to 129'10243 2024-09-21T13:52:06.709 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:06 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2976538272' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:52:06.777 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:06.806 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:06.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:06.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:06.908 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:52:06.936 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:06.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:06.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:52:07.037 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:52:07.086 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:52:07.098 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:07.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:07.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:52:07.200 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:52:07.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:07.262 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:07.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:52:07.363 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:07.392 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:07.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:07.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:07.493 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:52:07.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:07.519 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:07.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:52:07.620 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:52:07.649 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:07.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:07.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:52:07.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:07 smithi187 ceph-mon[33032]: pgmap v140: 148 pgs: 34 peering, 5 active+undersized, 5 active+undersized+degraded, 104 active+clean; 6.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s; 14342/330807 objects degraded (4.335%) 2024-09-21T13:52:07.751 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:52:07.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:07 smithi049 ceph-mon[30080]: pgmap v140: 148 pgs: 34 peering, 5 active+undersized, 5 active+undersized+degraded, 104 active+clean; 6.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s; 14342/330807 objects degraded (4.335%) 2024-09-21T13:52:07.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:07 smithi049 ceph-mon[40489]: pgmap v140: 148 pgs: 34 peering, 5 active+undersized, 5 active+undersized+degraded, 104 active+clean; 6.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s; 14342/330807 objects degraded (4.335%) 2024-09-21T13:52:07.790 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:07.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:07.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:52:07.892 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:52:07.932 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:07.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:07.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:52:08.033 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:52:08.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:08.071 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:08.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:52:08.172 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:08.197 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:08.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:08.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:08.299 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:52:08.338 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:08.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:08.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:52:08.440 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:52:08.482 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:08.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:08.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:52:08.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:08 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 14333/330807 objects degraded (4.333%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T13:52:08.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:08 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 14333/330807 objects degraded (4.333%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T13:52:08.525 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:52:08.525 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:52:08.583 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:52:08.610 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:08.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:08.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:52:08.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:08 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 14333/330807 objects degraded (4.333%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T13:52:08.712 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:52:08.769 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:08.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:08.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:52:08.871 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:52:08.898 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:08.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:08.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:52:09.000 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:52:09.030 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:09.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:09.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:52:09.123 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T13:52:09.133 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:52:09.161 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:09.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:09.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:52:09.263 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:52:09.309 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:09.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:09.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:52:09.411 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:52:09.420 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:52:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:52:09.438 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.0 to 1 (10000) 2024-09-21T13:52:09.476 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:09.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:09.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:52:09.514 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T13:52:09.514 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-09-21T13:52:09.514 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.514 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:168057 7.3 7:ca551071:::benchmark_data_smithi049_76798_object31637:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-09-21T13:52:09.514 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:09.458098+0000", 2024-09-21T13:52:09.514 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.031028111000000001, 2024-09-21T13:52:09.514 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.031056871999999999, 2024-09-21T13:52:09.514 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:52:09.514 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:52:09.514 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T13:52:09.514 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:52:09.515 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:52:09.515 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:52:09.515 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 168057 2024-09-21T13:52:09.515 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.515 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:52:09.515 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.515 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:52:09.515 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.458098+0000", 2024-09-21T13:52:09.515 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:52:09.515 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.515 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.515 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:52:09.515 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.458098+0000", 2024-09-21T13:52:09.516 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:52:09.516 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.516 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.516 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:52:09.516 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.458100+0000", 2024-09-21T13:52:09.516 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.646e-06 2024-09-21T13:52:09.516 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.516 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.516 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:52:09.516 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.458106+0000", 2024-09-21T13:52:09.517 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6159999999999998e-06 2024-09-21T13:52:09.517 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.517 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.517 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:52:09.517 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.458120+0000", 2024-09-21T13:52:09.517 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3319000000000001e-05 2024-09-21T13:52:09.517 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.517 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.517 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:52:09.517 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.458129+0000", 2024-09-21T13:52:09.517 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.8619999999999992e-06 2024-09-21T13:52:09.518 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.518 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.518 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:52:09.518 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.488319+0000", 2024-09-21T13:52:09.518 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.030190207 2024-09-21T13:52:09.518 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:52:09.518 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:52:09.518 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:52:09.518 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.518 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.518 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.14817.0:168058 7.7 e133/83)", 2024-09-21T13:52:09.518 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:09.463710+0000", 2024-09-21T13:52:09.518 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.025416014000000001, 2024-09-21T13:52:09.518 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.025499643999999998, 2024-09-21T13:52:09.519 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:52:09.519 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:52:09.519 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T13:52:09.519 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:52:09.519 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.519 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:52:09.519 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.463710+0000", 2024-09-21T13:52:09.519 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:52:09.519 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.519 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.519 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:52:09.519 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.463710+0000", 2024-09-21T13:52:09.519 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:52:09.519 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.520 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.520 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:52:09.520 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.463711+0000", 2024-09-21T13:52:09.520 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2700000000000001e-07 2024-09-21T13:52:09.520 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.520 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.520 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:52:09.520 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.463714+0000", 2024-09-21T13:52:09.520 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6399999999999999e-06 2024-09-21T13:52:09.520 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.520 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.520 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:52:09.520 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.463725+0000", 2024-09-21T13:52:09.520 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0382e-05 2024-09-21T13:52:09.520 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.521 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.521 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:52:09.521 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.463732+0000", 2024-09-21T13:52:09.521 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2790000000000001e-06 2024-09-21T13:52:09.521 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.521 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.521 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:52:09.521 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.482928+0000", 2024-09-21T13:52:09.521 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.019196265000000001 2024-09-21T13:52:09.521 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:52:09.521 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:52:09.521 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:52:09.521 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.521 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.521 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:168062 7.2 7:41c69180:::benchmark_data_smithi049_76798_object31642:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-09-21T13:52:09.522 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:09.471650+0000", 2024-09-21T13:52:09.522 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.017476149, 2024-09-21T13:52:09.522 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.017606668999999998, 2024-09-21T13:52:09.522 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:52:09.522 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:52:09.522 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "waiting for sub ops", 2024-09-21T13:52:09.522 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:52:09.522 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:52:09.523 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:52:09.523 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 168062 2024-09-21T13:52:09.523 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.523 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:52:09.523 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.523 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:52:09.523 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.471650+0000", 2024-09-21T13:52:09.523 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:52:09.523 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.524 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.524 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:52:09.524 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.471650+0000", 2024-09-21T13:52:09.524 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:52:09.524 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.524 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.524 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:52:09.524 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.471652+0000", 2024-09-21T13:52:09.524 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9520000000000001e-06 2024-09-21T13:52:09.524 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.524 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.524 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:52:09.525 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.471660+0000", 2024-09-21T13:52:09.525 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0350000000000001e-06 2024-09-21T13:52:09.525 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.525 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.525 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:52:09.525 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.473633+0000", 2024-09-21T13:52:09.525 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0019729700000000001 2024-09-21T13:52:09.525 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.525 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.525 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:52:09.525 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.473646+0000", 2024-09-21T13:52:09.525 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2659e-05 2024-09-21T13:52:09.525 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.526 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.526 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:52:09.526 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.487532+0000", 2024-09-21T13:52:09.526 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.013886287000000001 2024-09-21T13:52:09.526 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.526 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.526 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:52:09.526 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.487663+0000", 2024-09-21T13:52:09.526 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013045799999999999 2024-09-21T13:52:09.526 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.526 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.526 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,2", 2024-09-21T13:52:09.526 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.487791+0000", 2024-09-21T13:52:09.526 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000128162 2024-09-21T13:52:09.527 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:52:09.527 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:52:09.527 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:52:09.527 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.527 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.527 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDPGPushReply(7.a 132/131 [PushReplyOp(7:50ff1ec0:::benchmark_data_smithi049_76798_object105665:head)])", 2024-09-21T13:52:09.527 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:09.475497+0000", 2024-09-21T13:52:09.527 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.013629275999999999, 2024-09-21T13:52:09.527 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.013818301999999999, 2024-09-21T13:52:09.527 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:52:09.528 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:52:09.528 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "queued for pg", 2024-09-21T13:52:09.528 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:52:09.528 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.528 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:52:09.528 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.475497+0000", 2024-09-21T13:52:09.528 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:52:09.528 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.528 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.528 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:52:09.528 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.475497+0000", 2024-09-21T13:52:09.528 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:52:09.528 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.528 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.529 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:52:09.529 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.475498+0000", 2024-09-21T13:52:09.529 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8500000000000002e-07 2024-09-21T13:52:09.529 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.529 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.529 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:52:09.529 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.475505+0000", 2024-09-21T13:52:09.529 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4819999999999997e-06 2024-09-21T13:52:09.529 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.529 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.529 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:52:09.529 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.475515+0000", 2024-09-21T13:52:09.529 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0448999999999999e-05 2024-09-21T13:52:09.529 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.530 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.530 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:52:09.530 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.475524+0000", 2024-09-21T13:52:09.530 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1899999999999995e-06 2024-09-21T13:52:09.530 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:52:09.530 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:52:09.530 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:52:09.530 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.530 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.530 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.14817.0:168069 7.0 e133/83 7:03ab0060:::benchmark_data_smithi049_76798_object31649:head v 133'10512, mlcod=132'10510)", 2024-09-21T13:52:09.530 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:09.482332+0000", 2024-09-21T13:52:09.531 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.0067940509999999997, 2024-09-21T13:52:09.531 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0070248849999999998, 2024-09-21T13:52:09.531 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:52:09.531 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:52:09.531 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T13:52:09.531 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:52:09.531 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.532 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:52:09.532 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.482332+0000", 2024-09-21T13:52:09.532 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:52:09.532 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.532 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.532 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:52:09.533 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.482332+0000", 2024-09-21T13:52:09.533 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:52:09.533 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.533 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.533 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:52:09.533 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.482333+0000", 2024-09-21T13:52:09.534 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2399999999999997e-07 2024-09-21T13:52:09.534 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.534 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.534 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:52:09.534 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.482343+0000", 2024-09-21T13:52:09.534 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8689999999999999e-06 2024-09-21T13:52:09.535 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.535 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.535 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:52:09.535 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.482354+0000", 2024-09-21T13:52:09.535 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0763e-05 2024-09-21T13:52:09.535 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.536 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.536 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:52:09.536 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.482360+0000", 2024-09-21T13:52:09.536 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.032e-06 2024-09-21T13:52:09.536 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.536 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.537 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:52:09.537 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.484909+0000", 2024-09-21T13:52:09.537 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0025496939999999999 2024-09-21T13:52:09.537 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.537 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.537 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:52:09.538 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.484930+0000", 2024-09-21T13:52:09.538 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1109000000000001e-05 2024-09-21T13:52:09.538 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:52:09.538 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:52:09.538 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:52:09.538 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.539 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.539 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.14817.0:168068 7.c e133/83 7:317b165a:::benchmark_data_smithi049_76798_object31648:head v 133'10552, mlcod=132'10549)", 2024-09-21T13:52:09.539 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:09.483506+0000", 2024-09-21T13:52:09.539 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.005620122, 2024-09-21T13:52:09.539 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0058987349999999996, 2024-09-21T13:52:09.539 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:52:09.540 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:52:09.540 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T13:52:09.540 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:52:09.540 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.540 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:52:09.540 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.483506+0000", 2024-09-21T13:52:09.541 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:52:09.541 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.541 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.541 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:52:09.541 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.483506+0000", 2024-09-21T13:52:09.541 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:52:09.542 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.542 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.542 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:52:09.542 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.483507+0000", 2024-09-21T13:52:09.542 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0500000000000002e-07 2024-09-21T13:52:09.542 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.543 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.543 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:52:09.543 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.483513+0000", 2024-09-21T13:52:09.543 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9120000000000003e-06 2024-09-21T13:52:09.543 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.543 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.543 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:52:09.543 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.483525+0000", 2024-09-21T13:52:09.543 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1771e-05 2024-09-21T13:52:09.544 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.544 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.544 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:52:09.544 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.483532+0000", 2024-09-21T13:52:09.544 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6790000000000005e-06 2024-09-21T13:52:09.544 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.544 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.544 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:52:09.544 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.485456+0000", 2024-09-21T13:52:09.544 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0019235490000000001 2024-09-21T13:52:09.544 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.544 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.544 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:52:09.544 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.485504+0000", 2024-09-21T13:52:09.545 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8257000000000002e-05 2024-09-21T13:52:09.545 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:52:09.545 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:52:09.545 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:52:09.545 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.545 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.545 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.14817.0:168056 7.a e133/131)", 2024-09-21T13:52:09.545 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:09.488064+0000", 2024-09-21T13:52:09.545 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.001062541, 2024-09-21T13:52:09.545 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001385081, 2024-09-21T13:52:09.545 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:52:09.545 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:52:09.545 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "queued for pg", 2024-09-21T13:52:09.545 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:52:09.546 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.546 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:52:09.546 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.488064+0000", 2024-09-21T13:52:09.546 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:52:09.546 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.546 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.546 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:52:09.546 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.488064+0000", 2024-09-21T13:52:09.546 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:52:09.546 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.546 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.546 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:52:09.547 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.488064+0000", 2024-09-21T13:52:09.547 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.8999999999999998e-07 2024-09-21T13:52:09.547 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.547 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.547 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:52:09.547 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.488066+0000", 2024-09-21T13:52:09.547 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1569999999999998e-06 2024-09-21T13:52:09.547 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.547 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.547 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:52:09.547 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.488072+0000", 2024-09-21T13:52:09.548 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5509999999999999e-06 2024-09-21T13:52:09.548 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.548 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.548 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:52:09.548 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.488076+0000", 2024-09-21T13:52:09.548 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1910000000000002e-06 2024-09-21T13:52:09.548 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:52:09.548 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:52:09.548 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:52:09.548 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.548 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.548 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:168072 7.f 7.ffac26ff (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-09-21T13:52:09.548 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:09.488096+0000", 2024-09-21T13:52:09.548 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.001030719, 2024-09-21T13:52:09.548 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0013908740000000001, 2024-09-21T13:52:09.549 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:52:09.549 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:52:09.549 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "queued for pg", 2024-09-21T13:52:09.549 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:52:09.549 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:52:09.549 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:52:09.549 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 168072 2024-09-21T13:52:09.549 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.549 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:52:09.549 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.549 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:52:09.549 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.488096+0000", 2024-09-21T13:52:09.549 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:52:09.549 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.549 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.549 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:52:09.550 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.488096+0000", 2024-09-21T13:52:09.550 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:52:09.550 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.550 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.550 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:52:09.550 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.488097+0000", 2024-09-21T13:52:09.550 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.466e-06 2024-09-21T13:52:09.550 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.550 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.550 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:52:09.550 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.488102+0000", 2024-09-21T13:52:09.551 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1909999999999999e-06 2024-09-21T13:52:09.551 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.551 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.551 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:52:09.551 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.488114+0000", 2024-09-21T13:52:09.551 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1382999999999999e-05 2024-09-21T13:52:09.551 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.551 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.551 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:52:09.551 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.488122+0000", 2024-09-21T13:52:09.551 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9799999999999998e-06 2024-09-21T13:52:09.551 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:52:09.551 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:52:09.551 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:52:09.552 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.552 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.552 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.14817.0:168071 7.6 e133/94 7:69302e20:::benchmark_data_smithi049_76798_object31651:head v 133'10613, mlcod=132'10610)", 2024-09-21T13:52:09.552 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:09.488515+0000", 2024-09-21T13:52:09.552 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.00061124599999999997, 2024-09-21T13:52:09.552 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001027397, 2024-09-21T13:52:09.552 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:52:09.552 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:52:09.552 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T13:52:09.552 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:52:09.552 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.552 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:52:09.552 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.488515+0000", 2024-09-21T13:52:09.553 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:52:09.553 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.553 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.553 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:52:09.553 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.488515+0000", 2024-09-21T13:52:09.553 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:52:09.553 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.553 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.553 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:52:09.553 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.488516+0000", 2024-09-21T13:52:09.553 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1999999999999999e-07 2024-09-21T13:52:09.553 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.553 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.553 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:52:09.553 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.488523+0000", 2024-09-21T13:52:09.553 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2960000000000001e-06 2024-09-21T13:52:09.554 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.554 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.554 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:52:09.554 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.488531+0000", 2024-09-21T13:52:09.554 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2360000000000004e-06 2024-09-21T13:52:09.554 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.554 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.554 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:52:09.554 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.488537+0000", 2024-09-21T13:52:09.554 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7189999999999998e-06 2024-09-21T13:52:09.554 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.554 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.554 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:52:09.555 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.488612+0000", 2024-09-21T13:52:09.555 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4981999999999995e-05 2024-09-21T13:52:09.555 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:52:09.555 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:52:09.555 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:52:09.555 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.555 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.555 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:168048 7.f 7:f7829ba9:::benchmark_data_smithi049_76798_object31628:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-09-21T13:52:09.555 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:09.441389+0000", 2024-09-21T13:52:09.555 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.047737214, 2024-09-21T13:52:09.555 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.048198613000000001, 2024-09-21T13:52:09.555 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:52:09.555 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:52:09.555 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "waiting for sub ops", 2024-09-21T13:52:09.555 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:52:09.555 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:52:09.556 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:52:09.556 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 168048 2024-09-21T13:52:09.556 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.556 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:52:09.556 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.556 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:52:09.556 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.441389+0000", 2024-09-21T13:52:09.556 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:52:09.556 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.556 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.556 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:52:09.556 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.441389+0000", 2024-09-21T13:52:09.556 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:52:09.556 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.556 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.557 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:52:09.557 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.441391+0000", 2024-09-21T13:52:09.557 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6190000000000001e-06 2024-09-21T13:52:09.557 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.557 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.557 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:52:09.557 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.441396+0000", 2024-09-21T13:52:09.557 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.135e-06 2024-09-21T13:52:09.557 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.557 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.557 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:52:09.557 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.441406+0000", 2024-09-21T13:52:09.557 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0546e-05 2024-09-21T13:52:09.557 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.557 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.557 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:52:09.558 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.441414+0000", 2024-09-21T13:52:09.558 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9009999999999994e-06 2024-09-21T13:52:09.558 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.558 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.558 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:52:09.558 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.441603+0000", 2024-09-21T13:52:09.558 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00018850299999999999 2024-09-21T13:52:09.558 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.558 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.558 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:52:09.558 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.441746+0000", 2024-09-21T13:52:09.558 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000143597 2024-09-21T13:52:09.558 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.558 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.559 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,7", 2024-09-21T13:52:09.559 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.441867+0000", 2024-09-21T13:52:09.559 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000121031 2024-09-21T13:52:09.559 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.559 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.559 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:52:09.559 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.451271+0000", 2024-09-21T13:52:09.559 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0094031930000000007 2024-09-21T13:52:09.559 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.559 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.559 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:52:09.559 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.459565+0000", 2024-09-21T13:52:09.559 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0082941500000000001 2024-09-21T13:52:09.559 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:52:09.559 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:52:09.559 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:52:09.560 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.560 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.560 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:168052 7.f 7:fd449ea3:::benchmark_data_smithi049_76798_object31632:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-09-21T13:52:09.560 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:09.454812+0000", 2024-09-21T13:52:09.560 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.034314206, 2024-09-21T13:52:09.560 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.034844470000000002, 2024-09-21T13:52:09.560 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:52:09.560 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:52:09.560 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "waiting for sub ops", 2024-09-21T13:52:09.560 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:52:09.560 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:52:09.560 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:52:09.560 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 168052 2024-09-21T13:52:09.560 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.560 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:52:09.560 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.561 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:52:09.561 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.454812+0000", 2024-09-21T13:52:09.561 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:52:09.561 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.561 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.561 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:52:09.561 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.454812+0000", 2024-09-21T13:52:09.561 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:52:09.561 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.561 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.561 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:52:09.561 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.454814+0000", 2024-09-21T13:52:09.561 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.381e-06 2024-09-21T13:52:09.561 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.561 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.561 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:52:09.562 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.454818+0000", 2024-09-21T13:52:09.562 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4519999999999999e-06 2024-09-21T13:52:09.562 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.562 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.562 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:52:09.562 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.454829+0000", 2024-09-21T13:52:09.562 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0533000000000001e-05 2024-09-21T13:52:09.562 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.562 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.562 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:52:09.562 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.454835+0000", 2024-09-21T13:52:09.562 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4400000000000002e-06 2024-09-21T13:52:09.562 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.562 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.562 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:52:09.563 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.459602+0000", 2024-09-21T13:52:09.563 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0047667029999999997 2024-09-21T13:52:09.563 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.563 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.563 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:52:09.563 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.459743+0000", 2024-09-21T13:52:09.563 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014182999999999999 2024-09-21T13:52:09.563 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.563 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.563 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,7", 2024-09-21T13:52:09.563 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.459882+0000", 2024-09-21T13:52:09.563 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000138527 2024-09-21T13:52:09.563 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.563 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.563 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:52:09.563 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.462845+0000", 2024-09-21T13:52:09.564 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002963093 2024-09-21T13:52:09.564 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.564 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.564 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:52:09.564 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.476335+0000", 2024-09-21T13:52:09.564 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.013489597000000001 2024-09-21T13:52:09.564 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:52:09.564 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:52:09.564 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:52:09.564 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.564 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.564 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:168053 7.3 7:c7334b49:::benchmark_data_smithi049_76798_object31633:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-09-21T13:52:09.564 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:09.454845+0000", 2024-09-21T13:52:09.564 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.034281381, 2024-09-21T13:52:09.565 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.034879205000000003, 2024-09-21T13:52:09.565 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:52:09.565 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:52:09.565 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "waiting for sub ops", 2024-09-21T13:52:09.565 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:52:09.565 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:52:09.565 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:52:09.565 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 168053 2024-09-21T13:52:09.565 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.565 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:52:09.565 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.565 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:52:09.565 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.454845+0000", 2024-09-21T13:52:09.565 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:52:09.565 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.565 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.566 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:52:09.566 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.454845+0000", 2024-09-21T13:52:09.566 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:52:09.566 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.566 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.566 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:52:09.566 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.454845+0000", 2024-09-21T13:52:09.566 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4000000000000002e-07 2024-09-21T13:52:09.566 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.566 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.566 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:52:09.566 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.454848+0000", 2024-09-21T13:52:09.566 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3860000000000001e-06 2024-09-21T13:52:09.566 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.567 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.567 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:52:09.567 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.456579+0000", 2024-09-21T13:52:09.567 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0017314159999999999 2024-09-21T13:52:09.567 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.567 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.567 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:52:09.567 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.456588+0000", 2024-09-21T13:52:09.567 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1200000000000008e-06 2024-09-21T13:52:09.567 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.567 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.567 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:52:09.567 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.475634+0000", 2024-09-21T13:52:09.567 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.019045717 2024-09-21T13:52:09.568 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.568 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.568 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:52:09.568 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.475753+0000", 2024-09-21T13:52:09.568 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011892600000000001 2024-09-21T13:52:09.568 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.568 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.568 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,4", 2024-09-21T13:52:09.568 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.475884+0000", 2024-09-21T13:52:09.568 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013060799999999999 2024-09-21T13:52:09.568 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.568 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.568 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:52:09.568 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.488234+0000", 2024-09-21T13:52:09.569 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.012350089 2024-09-21T13:52:09.569 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.569 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.569 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:52:09.569 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.488286+0000", 2024-09-21T13:52:09.569 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2793000000000001e-05 2024-09-21T13:52:09.569 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:52:09.569 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:52:09.569 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:52:09.569 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.569 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.569 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:168054 7.f 7:fe95268e:::benchmark_data_smithi049_76798_object31634:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-09-21T13:52:09.569 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:09.456614+0000", 2024-09-21T13:52:09.569 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.032512381, 2024-09-21T13:52:09.569 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.033178744000000003, 2024-09-21T13:52:09.569 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:52:09.570 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:52:09.570 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "waiting for sub ops", 2024-09-21T13:52:09.570 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:52:09.570 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:52:09.570 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:52:09.570 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 168054 2024-09-21T13:52:09.570 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.570 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:52:09.570 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.570 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:52:09.570 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.456614+0000", 2024-09-21T13:52:09.570 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:52:09.570 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.571 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.571 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:52:09.571 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.456614+0000", 2024-09-21T13:52:09.571 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:52:09.571 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.571 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.571 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:52:09.571 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.456615+0000", 2024-09-21T13:52:09.571 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4470000000000001e-06 2024-09-21T13:52:09.571 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.571 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.571 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:52:09.571 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.456621+0000", 2024-09-21T13:52:09.571 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4e-06 2024-09-21T13:52:09.572 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.572 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.572 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:52:09.572 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.456629+0000", 2024-09-21T13:52:09.572 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6589999999999996e-06 2024-09-21T13:52:09.572 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.572 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.572 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:52:09.572 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.456636+0000", 2024-09-21T13:52:09.572 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3899999999999998e-06 2024-09-21T13:52:09.572 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.572 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.572 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:52:09.572 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.462876+0000", 2024-09-21T13:52:09.572 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0062405630000000002 2024-09-21T13:52:09.573 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.573 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.573 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op must wait for map", 2024-09-21T13:52:09.573 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.462883+0000", 2024-09-21T13:52:09.573 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2550000000000003e-06 2024-09-21T13:52:09.573 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.573 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.573 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:52:09.573 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.483717+0000", 2024-09-21T13:52:09.573 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.020834207 2024-09-21T13:52:09.573 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.573 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.573 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:52:09.573 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.483830+0000", 2024-09-21T13:52:09.573 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000113514 2024-09-21T13:52:09.573 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.574 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.574 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,7", 2024-09-21T13:52:09.574 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.483956+0000", 2024-09-21T13:52:09.574 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012552900000000001 2024-09-21T13:52:09.574 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:52:09.574 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:52:09.574 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:52:09.574 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.574 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.574 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:168056 7.a 7:51141609:::benchmark_data_smithi049_76798_object31636:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-09-21T13:52:09.574 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:09.456646+0000", 2024-09-21T13:52:09.574 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.032480583, 2024-09-21T13:52:09.574 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.033214981999999997, 2024-09-21T13:52:09.574 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:52:09.574 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:52:09.574 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "waiting for sub ops", 2024-09-21T13:52:09.575 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:52:09.575 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:52:09.575 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:52:09.575 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 168056 2024-09-21T13:52:09.575 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.575 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:52:09.575 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.575 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:52:09.575 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.456646+0000", 2024-09-21T13:52:09.575 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:52:09.575 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.575 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.575 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:52:09.575 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.456646+0000", 2024-09-21T13:52:09.575 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:52:09.575 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.576 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.576 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:52:09.576 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.456646+0000", 2024-09-21T13:52:09.576 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2900000000000004e-07 2024-09-21T13:52:09.576 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.576 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.576 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:52:09.576 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.456649+0000", 2024-09-21T13:52:09.576 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.7580000000000001e-06 2024-09-21T13:52:09.576 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.576 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.576 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:52:09.576 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.456656+0000", 2024-09-21T13:52:09.576 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.764e-06 2024-09-21T13:52:09.576 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.576 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.577 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:52:09.577 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.456662+0000", 2024-09-21T13:52:09.577 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5699999999999998e-06 2024-09-21T13:52:09.577 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.577 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.577 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:52:09.577 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.462706+0000", 2024-09-21T13:52:09.577 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0060432919999999996 2024-09-21T13:52:09.577 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.577 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.577 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op must wait for map", 2024-09-21T13:52:09.577 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.462715+0000", 2024-09-21T13:52:09.577 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3400000000000004e-06 2024-09-21T13:52:09.577 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.577 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.577 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:52:09.578 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.474894+0000", 2024-09-21T13:52:09.578 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.012179469 2024-09-21T13:52:09.578 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.578 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.578 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:52:09.578 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.475059+0000", 2024-09-21T13:52:09.578 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016422600000000001 2024-09-21T13:52:09.579 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.579 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.579 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,5", 2024-09-21T13:52:09.579 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.475237+0000", 2024-09-21T13:52:09.579 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00017808400000000001 2024-09-21T13:52:09.579 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:09.579 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:09.579 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:52:09.579 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:09.487877+0000", 2024-09-21T13:52:09.579 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.012640235 2024-09-21T13:52:09.579 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:52:09.579 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:52:09.579 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:52:09.579 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:52:09.579 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T13:52:09.580 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 14 2024-09-21T13:52:09.580 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T13:52:09.580 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:52:09.625 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:09.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:09.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:52:09.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:09 smithi187 ceph-mon[33032]: pgmap v141: 148 pgs: 34 peering, 2 active+undersized+degraded, 112 active+clean; 6.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 944 B/s rd, 0 B/s wr, 1.02k op/s; 14333/330807 objects degraded (4.333%) 2024-09-21T13:52:09.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:09 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3797526519' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T13:52:09.727 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:52:09.752 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:09 smithi049 ceph-mon[30080]: pgmap v141: 148 pgs: 34 peering, 2 active+undersized+degraded, 112 active+clean; 6.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 944 B/s rd, 0 B/s wr, 1.02k op/s; 14333/330807 objects degraded (4.333%) 2024-09-21T13:52:09.752 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:09 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3797526519' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T13:52:09.752 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:09 smithi049 ceph-mon[40489]: pgmap v141: 148 pgs: 34 peering, 2 active+undersized+degraded, 112 active+clean; 6.7 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 944 B/s rd, 0 B/s wr, 1.02k op/s; 14333/330807 objects degraded (4.333%) 2024-09-21T13:52:09.753 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:09 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3797526519' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T13:52:09.754 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:09.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:09.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:52:09.856 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:52:09.869 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-09-21T13:52:09.902 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:09.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:09.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:52:10.004 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:10.031 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:10.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:10.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:10.132 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:52:10.159 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:10.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:10.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:52:10.261 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:52:10.300 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:10.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:10.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:52:10.402 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:52:10.439 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:10.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:10.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:52:10.541 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:52:10.568 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:10.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:10.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:52:10.671 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:52:10.714 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:10.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:10.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:52:10.754 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:52:10 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:52:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:52:10.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:10 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:52:10.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:10 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3797526519' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T13:52:10.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:10 smithi049 ceph-mon[40489]: osdmap e133: 8 total, 8 up, 8 in 2024-09-21T13:52:10.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:10 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3364116674' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:52:10.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:10 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:52:10.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:10 smithi049 ceph-mon[40489]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:52:10.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:10 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T13:52:10.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:10 smithi049 ceph-mon[40489]: osdmap e134: 8 total, 8 up, 8 in 2024-09-21T13:52:10.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:10 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:52:10.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:10 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3797526519' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T13:52:10.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:10 smithi049 ceph-mon[30080]: osdmap e133: 8 total, 8 up, 8 in 2024-09-21T13:52:10.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:10 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3364116674' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:52:10.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:10 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:52:10.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:10 smithi049 ceph-mon[30080]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:52:10.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:10 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T13:52:10.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:10 smithi049 ceph-mon[30080]: osdmap e134: 8 total, 8 up, 8 in 2024-09-21T13:52:10.816 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:10.842 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:10.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:10.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:10.875 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T13:52:10.943 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:52:10.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:10 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:52:10.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:10 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3797526519' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T13:52:10.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:10 smithi187 ceph-mon[33032]: osdmap e133: 8 total, 8 up, 8 in 2024-09-21T13:52:10.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:10 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3364116674' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:52:10.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:10 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:52:10.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:10 smithi187 ceph-mon[33032]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:52:10.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:10 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T13:52:10.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:10 smithi187 ceph-mon[33032]: osdmap e134: 8 total, 8 up, 8 in 2024-09-21T13:52:11.023 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:11.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:11.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:52:11.125 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:52:11.156 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:11.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:11.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:52:11.258 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:11.289 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:11.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:11.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:11.390 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:52:11.417 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:11.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:11.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:52:11.518 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:52:11.545 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:11.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:11.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:52:11.647 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:11.673 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:11.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:11.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:11.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:11 smithi049 ceph-mon[40489]: pgmap v143: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 6.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 724 op/s; 37834/317781 objects degraded (11.906%); 13 MiB/s, 208 objects/s recovering 2024-09-21T13:52:11.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:11 smithi049 ceph-mon[40489]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T13:52:11.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:11 smithi049 ceph-mon[40489]: 4.12 scrub starts 2024-09-21T13:52:11.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:11 smithi049 ceph-mon[40489]: 4.12 scrub ok 2024-09-21T13:52:11.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:11 smithi049 ceph-mon[30080]: pgmap v143: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 6.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 724 op/s; 37834/317781 objects degraded (11.906%); 13 MiB/s, 208 objects/s recovering 2024-09-21T13:52:11.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:11 smithi049 ceph-mon[30080]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T13:52:11.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:11 smithi049 ceph-mon[30080]: 4.12 scrub starts 2024-09-21T13:52:11.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:11 smithi049 ceph-mon[30080]: 4.12 scrub ok 2024-09-21T13:52:11.776 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:52:11.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:11.816 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:11.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:52:11.917 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:52:11.942 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:11.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:11.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:52:11.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:11 smithi187 ceph-mon[33032]: pgmap v143: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 6.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 724 op/s; 37834/317781 objects degraded (11.906%); 13 MiB/s, 208 objects/s recovering 2024-09-21T13:52:11.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:11 smithi187 ceph-mon[33032]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T13:52:11.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:11 smithi187 ceph-mon[33032]: 4.12 scrub starts 2024-09-21T13:52:11.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:11 smithi187 ceph-mon[33032]: 4.12 scrub ok 2024-09-21T13:52:12.044 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:12.072 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:12.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:12.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:12.173 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:52:12.205 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:12.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:12.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:52:12.306 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:52:12.336 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:12.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:12.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:52:12.438 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:12.465 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:12.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:12.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:12.567 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:52:12.597 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:12.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:12.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:52:12.699 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:52:12.728 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:12.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:12.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:52:12.829 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:52:12.859 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:12.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:12.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:52:12.961 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:52:12.993 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:12.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:12.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:52:13.095 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:52:13.116 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:13.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:13.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:52:13.218 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:52:13.260 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:13.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:13.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:52:13.362 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T13:52:13.363 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:52:13.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:13.385 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:13.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:52:13.486 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:52:13.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:13.513 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:13.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:52:13.614 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:52:13.661 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:13.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:13.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:52:13.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:13 smithi049 ceph-mon[30080]: pgmap v145: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 6.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 745 B/s rd, 0 B/s wr, 633 op/s; 37834/317781 objects degraded (11.906%); 11 MiB/s, 182 objects/s recovering 2024-09-21T13:52:13.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:13 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:52:13.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:13 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:52:13.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:13 smithi049 ceph-mon[40489]: pgmap v145: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 6.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 745 B/s rd, 0 B/s wr, 633 op/s; 37834/317781 objects degraded (11.906%); 11 MiB/s, 182 objects/s recovering 2024-09-21T13:52:13.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:13 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:52:13.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:13 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:52:13.763 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:52:13.807 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:13.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:13.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:52:13.908 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:52:13.936 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:13.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:13.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:52:13.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:13 smithi187 ceph-mon[33032]: pgmap v145: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 6.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 745 B/s rd, 0 B/s wr, 633 op/s; 37834/317781 objects degraded (11.906%); 11 MiB/s, 182 objects/s recovering 2024-09-21T13:52:13.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:13 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:52:13.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:13 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:52:13.952 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd reweight 1 1 2024-09-21T13:52:13.960 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:52:14.038 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:52:14.046 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:52:14.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:14.114 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:14.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:52:14.215 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:52:14.244 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:14.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:14.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:52:14.346 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:52:14.416 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:14.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:14.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:52:14.518 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:52:14.545 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:14.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:14.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:52:14.646 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:52:14.693 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:14.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:14.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:52:14.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:14 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 33807/301881 objects degraded (11.199%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T13:52:14.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:14 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 33807/301881 objects degraded (11.199%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T13:52:14.795 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:52:14.820 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:14.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:14.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:52:14.845 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:14 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 33807/301881 objects degraded (11.199%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T13:52:14.922 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:52:14.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:14.981 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:14.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:52:15.082 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:52:15.108 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:15.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:15.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:52:15.209 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:52:15.250 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:15.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:15.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:52:15.337 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:52:15.337 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:52:15.352 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:52:15.379 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:15.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:15.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:52:15.387 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:52:15.387 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:52:15.395 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T13:52:15.395 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-09-21T13:52:15.396 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-09-21T13:52:15.396 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-09-21T13:52:15.396 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T13:52:15.427 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:52:15.427 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:52:15.481 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:52:15.482 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:52:15.482 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:52:15.517 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:52:15.517 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:52:15.537 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:15.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:15.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:52:15.552 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:52:15.553 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:52:15.595 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:52:15.595 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:52:15.633 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:52:15.633 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:52:15.640 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:52:15.657 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:15 smithi049 ceph-mon[30080]: pgmap v146: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 6.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.21k op/s; 33807/301881 objects degraded (11.199%); 23 MiB/s, 373 objects/s recovering 2024-09-21T13:52:15.657 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:15 smithi049 ceph-mon[30080]: 5.12 scrub starts 2024-09-21T13:52:15.657 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:15 smithi049 ceph-mon[30080]: 5.12 scrub ok 2024-09-21T13:52:15.657 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:15 smithi049 ceph-mon[30080]: 7.0 scrub starts 2024-09-21T13:52:15.657 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:15 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3761469619' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:52:15.657 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:15 smithi049 ceph-mon[40489]: pgmap v146: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 6.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.21k op/s; 33807/301881 objects degraded (11.199%); 23 MiB/s, 373 objects/s recovering 2024-09-21T13:52:15.658 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:15 smithi049 ceph-mon[40489]: 5.12 scrub starts 2024-09-21T13:52:15.658 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:15 smithi049 ceph-mon[40489]: 5.12 scrub ok 2024-09-21T13:52:15.658 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:15 smithi049 ceph-mon[40489]: 7.0 scrub starts 2024-09-21T13:52:15.658 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:15 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3761469619' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:52:15.665 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:15.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:15.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:52:15.767 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:52:15.804 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:15.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:15.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:52:15.905 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:15.933 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:15.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:15.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:15.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:15 smithi187 ceph-mon[33032]: pgmap v146: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 6.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.21k op/s; 33807/301881 objects degraded (11.199%); 23 MiB/s, 373 objects/s recovering 2024-09-21T13:52:15.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:15 smithi187 ceph-mon[33032]: 5.12 scrub starts 2024-09-21T13:52:15.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:15 smithi187 ceph-mon[33032]: 5.12 scrub ok 2024-09-21T13:52:15.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:15 smithi187 ceph-mon[33032]: 7.0 scrub starts 2024-09-21T13:52:15.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:15 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3761469619' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:52:16.034 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:52:16.049 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:52:16.098 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:16.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:16.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:52:16.107 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T13:52:16.200 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:16.227 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:16.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:16.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:16.329 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:52:16.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:16.369 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:16.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:52:16.470 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:52:16.497 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:16.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:16.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:52:16.555 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:52:16.598 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:52:16.659 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:16.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:16.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:52:16.760 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:52:16.799 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:16.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:16.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:52:16.901 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:52:16.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:16.943 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:16.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:52:17.044 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:52:17.071 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:17.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:17.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:52:17.173 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:52:17.202 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:17.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:17.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:52:17.304 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:17.330 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:17.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:17.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:17.433 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:17.468 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:17.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:17.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:17.569 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:52:17.597 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:17.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:17.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:52:17.622 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:17 smithi049 ceph-mon[40489]: pgmap v147: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 6.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.21k op/s; 33807/301881 objects degraded (11.199%); 23 MiB/s, 373 objects/s recovering 2024-09-21T13:52:17.622 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:17 smithi049 ceph-mon[40489]: 4.f scrub starts 2024-09-21T13:52:17.623 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:17 smithi049 ceph-mon[40489]: 4.f scrub ok 2024-09-21T13:52:17.623 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:17 smithi049 ceph-mon[40489]: 3.5 scrub starts 2024-09-21T13:52:17.623 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:17 smithi049 ceph-mon[40489]: 3.5 scrub ok 2024-09-21T13:52:17.623 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:17 smithi049 ceph-mon[40489]: 7.0 scrub ok 2024-09-21T13:52:17.623 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:17 smithi049 ceph-mon[30080]: pgmap v147: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 6.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.21k op/s; 33807/301881 objects degraded (11.199%); 23 MiB/s, 373 objects/s recovering 2024-09-21T13:52:17.623 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:17 smithi049 ceph-mon[30080]: 4.f scrub starts 2024-09-21T13:52:17.623 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:17 smithi049 ceph-mon[30080]: 4.f scrub ok 2024-09-21T13:52:17.623 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:17 smithi049 ceph-mon[30080]: 3.5 scrub starts 2024-09-21T13:52:17.623 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:17 smithi049 ceph-mon[30080]: 3.5 scrub ok 2024-09-21T13:52:17.623 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:17 smithi049 ceph-mon[30080]: 7.0 scrub ok 2024-09-21T13:52:17.699 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:52:17.725 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:17.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:17.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:52:17.827 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:52:17.852 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:17.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:17.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:52:17.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:17 smithi187 ceph-mon[33032]: pgmap v147: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 6.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.21k op/s; 33807/301881 objects degraded (11.199%); 23 MiB/s, 373 objects/s recovering 2024-09-21T13:52:17.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:17 smithi187 ceph-mon[33032]: 4.f scrub starts 2024-09-21T13:52:17.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:17 smithi187 ceph-mon[33032]: 4.f scrub ok 2024-09-21T13:52:17.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:17 smithi187 ceph-mon[33032]: 3.5 scrub starts 2024-09-21T13:52:17.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:17 smithi187 ceph-mon[33032]: 3.5 scrub ok 2024-09-21T13:52:17.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:17 smithi187 ceph-mon[33032]: 7.0 scrub ok 2024-09-21T13:52:17.954 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:52:17.985 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:17.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:17.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:52:18.087 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:52:18.112 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:18.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:18.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:52:18.214 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:52:18.228 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:52:18.264 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:18.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:18.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:52:18.366 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:52:18.392 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:18.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:18.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:52:18.494 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:52:18.523 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:18.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:18.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:52:18.526 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.1 to 1 (10000) 2024-09-21T13:52:18.625 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:52:18.651 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:18.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:18.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:52:18.752 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:18.778 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:18.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:18.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:18.880 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:52:18.926 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:18.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:18.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:52:18.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:18 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/4226801164' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T13:52:18.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:18 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T13:52:19.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:18 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/4226801164' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T13:52:19.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:18 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T13:52:19.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:18 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/4226801164' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T13:52:19.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:18 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T13:52:19.027 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T13:52:19.028 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:52:19.077 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:19.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:19.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:52:19.179 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:52:19.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:19.227 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:19.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:52:19.328 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:52:19.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:19.373 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:19.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:52:19.444 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:52:19 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:52:19.474 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:52:19.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:19.525 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:19.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:52:19.556 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-09-21T13:52:19.626 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:52:19.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:19.670 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:19.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:52:19.674 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T13:52:19.674 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:52:19.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:19 smithi049 ceph-mon[30080]: pgmap v148: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 6.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 618 B/s rd, 0 B/s wr, 640 op/s; 33807/301881 objects degraded (11.199%); 13 MiB/s, 209 objects/s recovering 2024-09-21T13:52:19.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:19 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T13:52:19.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:19 smithi049 ceph-mon[30080]: osdmap e135: 8 total, 8 up, 8 in 2024-09-21T13:52:19.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:19 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/841946398' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:52:19.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:19 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:52:19.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:19 smithi049 ceph-mon[40489]: pgmap v148: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 6.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 618 B/s rd, 0 B/s wr, 640 op/s; 33807/301881 objects degraded (11.199%); 13 MiB/s, 209 objects/s recovering 2024-09-21T13:52:19.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:19 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T13:52:19.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:19 smithi049 ceph-mon[40489]: osdmap e135: 8 total, 8 up, 8 in 2024-09-21T13:52:19.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:19 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/841946398' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:52:19.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:19 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:52:19.771 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:52:19.809 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:19.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:19.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:52:19.911 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:52:19.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:19 smithi187 ceph-mon[33032]: pgmap v148: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 6.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 618 B/s rd, 0 B/s wr, 640 op/s; 33807/301881 objects degraded (11.199%); 13 MiB/s, 209 objects/s recovering 2024-09-21T13:52:19.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:19 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T13:52:19.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:19 smithi187 ceph-mon[33032]: osdmap e135: 8 total, 8 up, 8 in 2024-09-21T13:52:19.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:19 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/841946398' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:52:19.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:19 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:52:19.948 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:19.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:19.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:52:20.050 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:52:20.077 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:20.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:20.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:52:20.179 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:20.205 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:20.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:20.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:20.307 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:52:20.349 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:20.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:20.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:52:20.450 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:52:20.488 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:20.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:20.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:52:20.593 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:52:20.621 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:20.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:20.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:52:20.723 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:52:20.751 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:20.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:20.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:52:20.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:20 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:52:20.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:20 smithi049 ceph-mon[40489]: 4.5 scrub starts 2024-09-21T13:52:20.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:20 smithi049 ceph-mon[40489]: 4.5 scrub ok 2024-09-21T13:52:20.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:20 smithi049 ceph-mon[40489]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T13:52:20.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:20 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T13:52:20.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:20 smithi049 ceph-mon[40489]: osdmap e136: 8 total, 8 up, 8 in 2024-09-21T13:52:20.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:20 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:52:20.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:20 smithi049 ceph-mon[30080]: 4.5 scrub starts 2024-09-21T13:52:20.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:20 smithi049 ceph-mon[30080]: 4.5 scrub ok 2024-09-21T13:52:20.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:20 smithi049 ceph-mon[30080]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T13:52:20.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:20 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T13:52:20.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:20 smithi049 ceph-mon[30080]: osdmap e136: 8 total, 8 up, 8 in 2024-09-21T13:52:20.756 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:52:20 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:52:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:52:20.853 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:52:20.892 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:20.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:20.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:52:20.936 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T13:52:20.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:20 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:52:20.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:20 smithi187 ceph-mon[33032]: 4.5 scrub starts 2024-09-21T13:52:20.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:20 smithi187 ceph-mon[33032]: 4.5 scrub ok 2024-09-21T13:52:20.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:20 smithi187 ceph-mon[33032]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T13:52:20.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:20 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T13:52:20.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:20 smithi187 ceph-mon[33032]: osdmap e136: 8 total, 8 up, 8 in 2024-09-21T13:52:20.994 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:52:21.031 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:21.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:21.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:52:21.133 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:52:21.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:21.171 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:21.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:52:21.272 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:21.308 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:21.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:21.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:21.310 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:52:21.410 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:52:21.453 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:21.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:21.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:52:21.555 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:52:21.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:21.594 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:21.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:52:21.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:21 smithi187 ceph-mon[33032]: pgmap v151: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 5.8 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s; 30186/287421 objects degraded (10.502%); 27 MiB/s, 426 objects/s recovering 2024-09-21T13:52:21.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:21 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 30186/287421 objects degraded (10.502%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T13:52:21.695 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:52:21.724 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:21.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:21.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:52:21.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:21 smithi049 ceph-mon[30080]: pgmap v151: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 5.8 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s; 30186/287421 objects degraded (10.502%); 27 MiB/s, 426 objects/s recovering 2024-09-21T13:52:21.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:21 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 30186/287421 objects degraded (10.502%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T13:52:21.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:21 smithi049 ceph-mon[40489]: pgmap v151: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 5.8 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s; 30186/287421 objects degraded (10.502%); 27 MiB/s, 426 objects/s recovering 2024-09-21T13:52:21.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:21 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 30186/287421 objects degraded (10.502%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T13:52:21.826 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:52:21.855 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:21.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:21.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:52:21.957 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:52:21.977 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:21.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:21.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:52:22.079 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:52:22.116 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:22.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:22.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:52:22.217 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:52:22.242 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:22.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:22.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:52:22.344 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:52:22.374 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:22.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:22.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:52:22.477 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:52:22.539 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:22.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:22.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:52:22.549 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd reweight 2 1 2024-09-21T13:52:22.571 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:52:22.642 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:22.667 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:22.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:22.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:22.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:22 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:52:22.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:22 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:52:22.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:22 smithi049 ceph-mon[30080]: 6.4 scrub starts 2024-09-21T13:52:22.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:22 smithi049 ceph-mon[30080]: 6.4 scrub ok 2024-09-21T13:52:22.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:22 smithi049 ceph-mon[30080]: osdmap e137: 8 total, 8 up, 8 in 2024-09-21T13:52:22.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:22 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:52:22.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:22 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:52:22.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:22 smithi049 ceph-mon[40489]: 6.4 scrub starts 2024-09-21T13:52:22.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:22 smithi049 ceph-mon[40489]: 6.4 scrub ok 2024-09-21T13:52:22.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:22 smithi049 ceph-mon[40489]: osdmap e137: 8 total, 8 up, 8 in 2024-09-21T13:52:22.769 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:52:22.796 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:22.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:22.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:52:22.898 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:52:22.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:22.935 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:22.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:52:22.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:22 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:52:22.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:22 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:52:22.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:22 smithi187 ceph-mon[33032]: 6.4 scrub starts 2024-09-21T13:52:22.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:22 smithi187 ceph-mon[33032]: 6.4 scrub ok 2024-09-21T13:52:22.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:22 smithi187 ceph-mon[33032]: osdmap e137: 8 total, 8 up, 8 in 2024-09-21T13:52:23.036 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:52:23.064 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:23.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:23.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:52:23.166 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:52:23.197 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:23.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:23.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:52:23.298 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:52:23.331 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:23.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:23.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:52:23.400 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T13:52:23.434 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:52:23.470 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:52:23.494 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:52:23.494 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:52:23.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:23.532 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:23.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:52:23.542 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:52:23.542 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:52:23.579 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:52:23.579 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:52:23.608 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T13:52:23.608 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:52:23.626 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:52:23.626 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:52:23.634 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:52:23.661 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:23.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:23.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:52:23.664 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:52:23.664 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:52:23.706 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:52:23.706 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:52:23.748 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:52:23.749 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:52:23.763 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:52:23.779 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:52:23.781 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:52:23.796 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:23 smithi049 ceph-mon[40489]: pgmap v153: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 5.8 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 803 op/s; 30186/287421 objects degraded (10.502%); 17 MiB/s, 279 objects/s recovering 2024-09-21T13:52:23.796 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:23 smithi049 ceph-mon[40489]: 3.0 scrub starts 2024-09-21T13:52:23.796 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:23 smithi049 ceph-mon[40489]: 3.0 scrub ok 2024-09-21T13:52:23.796 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:23 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1766193888' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:52:23.796 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:23 smithi049 ceph-mon[30080]: pgmap v153: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 5.8 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 803 op/s; 30186/287421 objects degraded (10.502%); 17 MiB/s, 279 objects/s recovering 2024-09-21T13:52:23.797 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:23 smithi049 ceph-mon[30080]: 3.0 scrub starts 2024-09-21T13:52:23.797 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:23 smithi049 ceph-mon[30080]: 3.0 scrub ok 2024-09-21T13:52:23.797 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:23 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1766193888' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:52:23.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:23.832 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:23.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:52:23.933 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:52:23.943 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:23 smithi187 ceph-mon[33032]: pgmap v153: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 5.8 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 803 op/s; 30186/287421 objects degraded (10.502%); 17 MiB/s, 279 objects/s recovering 2024-09-21T13:52:23.943 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:23 smithi187 ceph-mon[33032]: 3.0 scrub starts 2024-09-21T13:52:23.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:23 smithi187 ceph-mon[33032]: 3.0 scrub ok 2024-09-21T13:52:23.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:23 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1766193888' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:52:23.980 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:23.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:23.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:52:24.081 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:52:24.120 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:24.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:24.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:52:24.222 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:52:24.251 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:24.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:24.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:52:24.353 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:52:24.381 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T13:52:24.390 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:24.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:24.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:52:24.492 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:52:24.528 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:24.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:24.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:52:24.629 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:52:24.653 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:24.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:24.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:52:24.727 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:52:24.755 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:52:24.787 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:24.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:24.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:52:24.890 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:52:24.918 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:24.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:24.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:52:25.020 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:52:25.059 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:25.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:25.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:52:25.161 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:52:25.191 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:25.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:25.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:52:25.293 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:52:25.335 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:25.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:25.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:52:25.344 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T13:52:25.344 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-21T13:52:25.344 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-21T13:52:25.344 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-21T13:52:25.344 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T13:52:25.437 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:52:25.478 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:25.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:25.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:52:25.580 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:52:25.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:25.634 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:25.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:52:25.735 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:52:25.764 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:25.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:25.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:52:25.866 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:52:25.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:25.905 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:25.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:52:25.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:25 smithi187 ceph-mon[33032]: pgmap v154: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.47k op/s; 27956/275445 objects degraded (10.149%); 31 MiB/s, 498 objects/s recovering 2024-09-21T13:52:25.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:25 smithi187 ceph-mon[33032]: osdmap e138: 8 total, 8 up, 8 in 2024-09-21T13:52:26.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:25 smithi049 ceph-mon[30080]: pgmap v154: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.47k op/s; 27956/275445 objects degraded (10.149%); 31 MiB/s, 498 objects/s recovering 2024-09-21T13:52:26.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:25 smithi049 ceph-mon[30080]: osdmap e138: 8 total, 8 up, 8 in 2024-09-21T13:52:26.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:25 smithi049 ceph-mon[40489]: pgmap v154: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.47k op/s; 27956/275445 objects degraded (10.149%); 31 MiB/s, 498 objects/s recovering 2024-09-21T13:52:26.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:25 smithi049 ceph-mon[40489]: osdmap e138: 8 total, 8 up, 8 in 2024-09-21T13:52:26.006 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:26.035 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:26.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:26.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:26.069 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T13:52:26.137 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:52:26.188 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:26.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:26.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:52:26.290 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:52:26.330 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:26.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:26.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:52:26.432 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:52:26.449 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:52:26.473 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:26.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:26.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:52:26.574 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:26.600 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:26.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:26.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:26.702 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:52:26.727 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:26.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:26.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:52:26.735 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:26 smithi049 ceph-mon[30080]: 5.1d scrub starts 2024-09-21T13:52:26.735 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:26 smithi049 ceph-mon[30080]: 5.1d scrub ok 2024-09-21T13:52:26.735 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:26 smithi049 ceph-mon[40489]: 5.1d scrub starts 2024-09-21T13:52:26.736 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:26 smithi049 ceph-mon[40489]: 5.1d scrub ok 2024-09-21T13:52:26.829 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:52:26.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:26.876 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:26.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:52:26.926 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:26 smithi187 ceph-mon[33032]: 5.1d scrub starts 2024-09-21T13:52:26.926 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:26 smithi187 ceph-mon[33032]: 5.1d scrub ok 2024-09-21T13:52:26.926 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:26 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 26565/271587 objects degraded (9.781%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T13:52:26.977 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:52:27.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:26 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 26565/271587 objects degraded (9.781%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T13:52:27.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:26 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 26565/271587 objects degraded (9.781%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T13:52:27.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:27.011 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:27.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:52:27.112 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:52:27.140 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:27.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:27.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:52:27.241 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:52:27.285 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:27.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:27.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:52:27.377 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T13:52:27.377 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-21T13:52:27.377 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-21T13:52:27.377 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-21T13:52:27.378 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.378 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:35241 7.4 7:2f984d9b:::benchmark_data_smithi049_76798_object35240:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T13:52:27.378 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:28.687118+0000", 2024-09-21T13:52:27.378 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 118.688344469, 2024-09-21T13:52:27.378 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.011460528, 2024-09-21T13:52:27.378 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:52:27.378 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:52:27.378 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:52:27.378 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:52:27.378 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:52:27.378 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:52:27.378 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 35241 2024-09-21T13:52:27.379 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.379 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:52:27.379 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.379 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:52:27.379 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.687118+0000", 2024-09-21T13:52:27.379 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:52:27.379 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.379 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.379 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:52:27.379 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.687118+0000", 2024-09-21T13:52:27.379 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:52:27.379 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.379 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.379 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:52:27.380 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.687120+0000", 2024-09-21T13:52:27.380 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.573e-06 2024-09-21T13:52:27.380 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.380 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.380 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:52:27.380 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.687176+0000", 2024-09-21T13:52:27.380 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6502000000000002e-05 2024-09-21T13:52:27.380 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.380 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.380 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:52:27.380 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.687189+0000", 2024-09-21T13:52:27.380 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2747e-05 2024-09-21T13:52:27.380 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.380 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.381 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:52:27.381 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.687196+0000", 2024-09-21T13:52:27.381 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6050000000000003e-06 2024-09-21T13:52:27.381 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.381 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.381 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:52:27.381 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.687248+0000", 2024-09-21T13:52:27.381 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2234000000000002e-05 2024-09-21T13:52:27.381 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.381 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.381 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:52:27.381 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.687256+0000", 2024-09-21T13:52:27.381 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3299999999999999e-06 2024-09-21T13:52:27.381 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.381 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.382 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:52:27.382 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.691286+0000", 2024-09-21T13:52:27.382 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.004029828 2024-09-21T13:52:27.382 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.382 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.382 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:52:27.382 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.692244+0000", 2024-09-21T13:52:27.382 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00095738099999999999 2024-09-21T13:52:27.382 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.382 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.382 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-21T13:52:27.382 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.692408+0000", 2024-09-21T13:52:27.382 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000164041 2024-09-21T13:52:27.382 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.383 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.383 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:52:27.383 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.698261+0000", 2024-09-21T13:52:27.383 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.005852983 2024-09-21T13:52:27.383 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.383 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.383 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:52:27.383 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.698274+0000", 2024-09-21T13:52:27.383 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3485e-05 2024-09-21T13:52:27.383 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.383 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.383 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:52:27.383 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.698481+0000", 2024-09-21T13:52:27.383 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00020698099999999999 2024-09-21T13:52:27.384 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.384 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.384 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:52:27.384 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.698537+0000", 2024-09-21T13:52:27.384 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6082999999999998e-05 2024-09-21T13:52:27.384 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.384 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.384 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:52:27.384 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.698579+0000", 2024-09-21T13:52:27.384 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1755000000000003e-05 2024-09-21T13:52:27.384 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:52:27.384 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:52:27.384 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:52:27.384 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.384 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.385 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:35262 7.4 7:2f95173f:::benchmark_data_smithi049_76798_object35261:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T13:52:27.385 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:28.705575+0000", 2024-09-21T13:52:27.385 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 118.66988778699999, 2024-09-21T13:52:27.385 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.99486020900000005, 2024-09-21T13:52:27.385 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:52:27.385 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:52:27.385 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:52:27.385 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:52:27.385 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:52:27.385 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:52:27.385 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 35262 2024-09-21T13:52:27.385 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.385 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:52:27.385 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.385 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:52:27.386 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.705575+0000", 2024-09-21T13:52:27.386 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:52:27.386 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.386 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.386 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:52:27.386 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.705575+0000", 2024-09-21T13:52:27.386 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:52:27.386 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.386 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.386 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:52:27.386 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.705577+0000", 2024-09-21T13:52:27.386 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.485e-06 2024-09-21T13:52:27.386 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.386 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.386 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:52:27.387 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.705634+0000", 2024-09-21T13:52:27.387 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7954000000000001e-05 2024-09-21T13:52:27.387 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.387 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.387 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:52:27.387 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.705649+0000", 2024-09-21T13:52:27.387 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4209999999999999e-05 2024-09-21T13:52:27.387 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.387 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.387 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:52:27.387 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.705657+0000", 2024-09-21T13:52:27.387 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7910000000000005e-06 2024-09-21T13:52:27.387 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.387 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.387 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:52:27.387 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.705703+0000", 2024-09-21T13:52:27.388 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6712999999999998e-05 2024-09-21T13:52:27.388 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.388 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.388 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:52:27.388 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.705711+0000", 2024-09-21T13:52:27.388 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2050000000000002e-06 2024-09-21T13:52:27.388 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.388 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.388 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:52:27.388 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.694482+0000", 2024-09-21T13:52:27.388 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.98877102800000005 2024-09-21T13:52:27.388 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.388 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.388 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:52:27.388 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.694624+0000", 2024-09-21T13:52:27.388 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014155400000000001 2024-09-21T13:52:27.389 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.389 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.389 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-21T13:52:27.389 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.694770+0000", 2024-09-21T13:52:27.389 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014635399999999999 2024-09-21T13:52:27.389 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.389 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.389 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:52:27.389 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.698351+0000", 2024-09-21T13:52:27.389 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0035804869999999998 2024-09-21T13:52:27.389 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.389 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.389 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:52:27.389 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.698411+0000", 2024-09-21T13:52:27.389 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0132e-05 2024-09-21T13:52:27.389 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.390 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.390 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:52:27.390 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.700312+0000", 2024-09-21T13:52:27.390 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001901109 2024-09-21T13:52:27.390 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.390 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.390 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:52:27.390 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.700374+0000", 2024-09-21T13:52:27.390 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:52:27.391 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2137999999999994e-05 2024-09-21T13:52:27.391 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.391 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.392 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:52:27.392 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.700435+0000", 2024-09-21T13:52:27.392 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1049000000000005e-05 2024-09-21T13:52:27.392 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:52:27.392 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:52:27.392 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:52:27.392 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.392 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.392 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:35282 7.4 7:26bb5871:::benchmark_data_smithi049_76798_object35281:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T13:52:27.392 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:28.721297+0000", 2024-09-21T13:52:27.392 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 118.654166132, 2024-09-21T13:52:27.392 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.98226591299999999, 2024-09-21T13:52:27.392 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:52:27.393 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:52:27.393 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:52:27.393 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:52:27.393 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:52:27.393 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:52:27.393 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 35282 2024-09-21T13:52:27.393 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.393 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:52:27.393 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.393 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:52:27.393 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.721297+0000", 2024-09-21T13:52:27.393 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:52:27.393 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.393 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.393 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:52:27.394 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.721297+0000", 2024-09-21T13:52:27.394 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:52:27.394 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.394 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.394 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:52:27.394 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.721298+0000", 2024-09-21T13:52:27.394 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0920000000000001e-06 2024-09-21T13:52:27.394 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.394 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.394 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:52:27.394 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.721380+0000", 2024-09-21T13:52:27.394 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2422000000000005e-05 2024-09-21T13:52:27.394 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.394 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.394 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:52:27.395 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.721389+0000", 2024-09-21T13:52:27.395 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1789999999999997e-06 2024-09-21T13:52:27.395 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.395 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.395 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:52:27.395 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.721394+0000", 2024-09-21T13:52:27.395 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9080000000000003e-06 2024-09-21T13:52:27.395 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.395 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.395 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:52:27.395 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.721436+0000", 2024-09-21T13:52:27.395 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1887000000000001e-05 2024-09-21T13:52:27.395 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.395 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.396 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:52:27.396 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.721443+0000", 2024-09-21T13:52:27.396 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.725e-06 2024-09-21T13:52:27.396 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.396 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.396 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:52:27.396 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.696771+0000", 2024-09-21T13:52:27.396 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.97532851799999998 2024-09-21T13:52:27.396 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.396 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.396 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:52:27.396 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.696923+0000", 2024-09-21T13:52:27.396 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015162800000000001 2024-09-21T13:52:27.396 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.396 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.396 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-21T13:52:27.397 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.697094+0000", 2024-09-21T13:52:27.397 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000170884 2024-09-21T13:52:27.397 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.397 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.397 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:52:27.397 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.700495+0000", 2024-09-21T13:52:27.397 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0034005519999999998 2024-09-21T13:52:27.397 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.397 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.397 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:52:27.397 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.701924+0000", 2024-09-21T13:52:27.397 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001429303 2024-09-21T13:52:27.397 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.397 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.397 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:52:27.397 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.703461+0000", 2024-09-21T13:52:27.398 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001537395 2024-09-21T13:52:27.398 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.398 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.398 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:52:27.398 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.703519+0000", 2024-09-21T13:52:27.398 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7435999999999997e-05 2024-09-21T13:52:27.398 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.398 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.398 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:52:27.398 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.703563+0000", 2024-09-21T13:52:27.398 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3983999999999998e-05 2024-09-21T13:52:27.398 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:52:27.398 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:52:27.398 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:52:27.398 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.399 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.399 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:36414 7.4 7:22f017ab:::benchmark_data_smithi049_76798_object36413:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-21T13:52:27.399 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:30.284375+0000", 2024-09-21T13:52:27.399 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 117.091087968, 2024-09-21T13:52:27.399 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.087128891, 2024-09-21T13:52:27.399 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:52:27.399 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:52:27.399 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:52:27.399 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:52:27.399 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:52:27.399 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:52:27.399 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 36414 2024-09-21T13:52:27.399 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.399 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:52:27.399 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.400 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:52:27.400 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.284375+0000", 2024-09-21T13:52:27.400 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:52:27.400 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.400 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.400 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:52:27.400 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.284375+0000", 2024-09-21T13:52:27.400 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:52:27.400 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.400 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.400 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:52:27.400 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.284377+0000", 2024-09-21T13:52:27.400 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.784e-06 2024-09-21T13:52:27.400 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.400 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.400 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:52:27.401 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.284433+0000", 2024-09-21T13:52:27.401 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6186e-05 2024-09-21T13:52:27.401 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.401 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.401 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:52:27.401 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.284442+0000", 2024-09-21T13:52:27.401 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.6740000000000003e-06 2024-09-21T13:52:27.401 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.401 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.401 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:52:27.401 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.284458+0000", 2024-09-21T13:52:27.401 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6708e-05 2024-09-21T13:52:27.401 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.401 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.402 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:52:27.402 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.284508+0000", 2024-09-21T13:52:27.402 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0138e-05 2024-09-21T13:52:27.402 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.402 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.402 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:52:27.402 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.284621+0000", 2024-09-21T13:52:27.402 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011225700000000001 2024-09-21T13:52:27.402 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.402 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.402 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-21T13:52:27.402 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.284738+0000", 2024-09-21T13:52:27.402 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011702200000000001 2024-09-21T13:52:27.402 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.402 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.402 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:52:27.403 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.286753+0000", 2024-09-21T13:52:27.403 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0020150739999999999 2024-09-21T13:52:27.403 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.403 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.403 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:52:27.403 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.368894+0000", 2024-09-21T13:52:27.403 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.082141599999999995 2024-09-21T13:52:27.403 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.403 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.403 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:52:27.403 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.371396+0000", 2024-09-21T13:52:27.403 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0025017339999999998 2024-09-21T13:52:27.403 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.403 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.403 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:52:27.403 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.371450+0000", 2024-09-21T13:52:27.404 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4190000000000001e-05 2024-09-21T13:52:27.404 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.404 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.404 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:52:27.404 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.371504+0000", 2024-09-21T13:52:27.404 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3523999999999998e-05 2024-09-21T13:52:27.404 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:52:27.404 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:52:27.404 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:52:27.404 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.404 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.404 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:36415 7.4 7:239fcd45:::benchmark_data_smithi049_76798_object36414:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-21T13:52:27.404 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:30.284471+0000", 2024-09-21T13:52:27.404 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 117.090991558, 2024-09-21T13:52:27.404 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.089453959999999999, 2024-09-21T13:52:27.404 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:52:27.405 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:52:27.405 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:52:27.405 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:52:27.405 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:52:27.405 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:52:27.405 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 36415 2024-09-21T13:52:27.405 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.405 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:52:27.405 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.405 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:52:27.405 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.284471+0000", 2024-09-21T13:52:27.405 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:52:27.405 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.405 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.405 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:52:27.406 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.284471+0000", 2024-09-21T13:52:27.406 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:52:27.406 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.406 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.406 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:52:27.406 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.284472+0000", 2024-09-21T13:52:27.406 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7299999999999995e-07 2024-09-21T13:52:27.406 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.406 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.406 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:52:27.406 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.284499+0000", 2024-09-21T13:52:27.406 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7344000000000001e-05 2024-09-21T13:52:27.406 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.406 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.407 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:52:27.407 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.284506+0000", 2024-09-21T13:52:27.407 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5799999999999997e-06 2024-09-21T13:52:27.407 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.407 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.407 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:52:27.407 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.284511+0000", 2024-09-21T13:52:27.407 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1410000000000004e-06 2024-09-21T13:52:27.407 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.407 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.407 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:52:27.407 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.285801+0000", 2024-09-21T13:52:27.407 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001290023 2024-09-21T13:52:27.407 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.407 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.408 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:52:27.408 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.285896+0000", 2024-09-21T13:52:27.408 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.5357000000000002e-05 2024-09-21T13:52:27.408 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.408 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.408 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-21T13:52:27.408 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.285982+0000", 2024-09-21T13:52:27.408 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.53e-05 2024-09-21T13:52:27.408 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.408 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.408 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:52:27.408 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.286879+0000", 2024-09-21T13:52:27.408 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00089719500000000005 2024-09-21T13:52:27.408 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.408 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.409 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:52:27.409 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.368934+0000", 2024-09-21T13:52:27.409 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.082055115999999997 2024-09-21T13:52:27.409 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.409 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.409 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:52:27.409 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.373694+0000", 2024-09-21T13:52:27.409 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0047602670000000003 2024-09-21T13:52:27.409 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.409 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.409 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:52:27.409 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.373777+0000", 2024-09-21T13:52:27.409 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2953000000000004e-05 2024-09-21T13:52:27.409 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.409 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.409 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:52:27.410 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.373925+0000", 2024-09-21T13:52:27.410 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000148011 2024-09-21T13:52:27.410 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:52:27.410 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:52:27.410 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:52:27.410 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.410 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.410 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.14817.0:36417 7.3 e96/83 7:c65d1f2d:::benchmark_data_smithi049_76798_object36416:head v 96'2339, mlcod=96'2337)", 2024-09-21T13:52:27.410 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:30.287399+0000", 2024-09-21T13:52:27.410 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 117.088063977, 2024-09-21T13:52:27.410 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.084777056000000003, 2024-09-21T13:52:27.410 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:52:27.410 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:52:27.410 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:52:27.410 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:52:27.410 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.411 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:52:27.411 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.287399+0000", 2024-09-21T13:52:27.411 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:52:27.411 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.411 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.411 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:52:27.411 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.287399+0000", 2024-09-21T13:52:27.411 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:52:27.411 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.411 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.411 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:52:27.411 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.287400+0000", 2024-09-21T13:52:27.411 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1600000000000001e-07 2024-09-21T13:52:27.411 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.412 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.412 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:52:27.412 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.287449+0000", 2024-09-21T13:52:27.412 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9137000000000001e-05 2024-09-21T13:52:27.412 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.412 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.412 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:52:27.412 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.287457+0000", 2024-09-21T13:52:27.412 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0059999999999992e-06 2024-09-21T13:52:27.412 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.412 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.412 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:52:27.412 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.287461+0000", 2024-09-21T13:52:27.412 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4730000000000002e-06 2024-09-21T13:52:27.412 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.413 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.413 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:52:27.413 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.287499+0000", 2024-09-21T13:52:27.413 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7920000000000003e-05 2024-09-21T13:52:27.413 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.413 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.413 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:52:27.413 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.287512+0000", 2024-09-21T13:52:27.413 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2839e-05 2024-09-21T13:52:27.413 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.413 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.413 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:52:27.413 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.372141+0000", 2024-09-21T13:52:27.413 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.084629393999999997 2024-09-21T13:52:27.413 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.414 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.414 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:52:27.414 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.372176+0000", 2024-09-21T13:52:27.414 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4570999999999998e-05 2024-09-21T13:52:27.414 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:52:27.414 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:52:27.414 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:52:27.414 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.414 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.414 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.14817.0:36420 7.8 e96/83 7:19c96052:::benchmark_data_smithi049_76798_object36419:head v 96'2319, mlcod=96'2317)", 2024-09-21T13:52:27.414 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:30.288772+0000", 2024-09-21T13:52:27.414 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 117.08669083300001, 2024-09-21T13:52:27.414 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.083767610000000006, 2024-09-21T13:52:27.414 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:52:27.414 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:52:27.414 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:52:27.415 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:52:27.415 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.415 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:52:27.415 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.288772+0000", 2024-09-21T13:52:27.415 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:52:27.415 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.415 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.415 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:52:27.415 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.288772+0000", 2024-09-21T13:52:27.415 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:52:27.415 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.415 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.415 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:52:27.415 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.288773+0000", 2024-09-21T13:52:27.415 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2399999999999997e-07 2024-09-21T13:52:27.416 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.416 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.416 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:52:27.416 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.288844+0000", 2024-09-21T13:52:27.416 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1304999999999996e-05 2024-09-21T13:52:27.416 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.416 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.416 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:52:27.416 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.368671+0000", 2024-09-21T13:52:27.416 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.079826578999999995 2024-09-21T13:52:27.416 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.416 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.416 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:52:27.416 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.368682+0000", 2024-09-21T13:52:27.417 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1058e-05 2024-09-21T13:52:27.417 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.418 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.418 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:52:27.418 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.368909+0000", 2024-09-21T13:52:27.418 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00022760900000000001 2024-09-21T13:52:27.418 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.418 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.418 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:52:27.418 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.368933+0000", 2024-09-21T13:52:27.418 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3750000000000001e-05 2024-09-21T13:52:27.418 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.418 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.418 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:52:27.418 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.372387+0000", 2024-09-21T13:52:27.419 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0034542990000000001 2024-09-21T13:52:27.419 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.419 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.419 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:52:27.419 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.372540+0000", 2024-09-21T13:52:27.419 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015228600000000001 2024-09-21T13:52:27.419 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:52:27.419 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:52:27.419 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:52:27.419 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.419 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.419 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.14817.0:36414 7.4 e96/94 ondisk, result = 0)", 2024-09-21T13:52:27.419 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:30.290044+0000", 2024-09-21T13:52:27.419 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 117.08541895800001, 2024-09-21T13:52:27.419 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.078859123000000003, 2024-09-21T13:52:27.420 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:52:27.420 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:52:27.420 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T13:52:27.420 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:52:27.420 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.420 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:52:27.420 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.290044+0000", 2024-09-21T13:52:27.420 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:52:27.420 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.420 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.420 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:52:27.420 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.290044+0000", 2024-09-21T13:52:27.420 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:52:27.420 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.420 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.421 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:52:27.421 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.290045+0000", 2024-09-21T13:52:27.421 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6899999999999997e-07 2024-09-21T13:52:27.421 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.421 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.421 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:52:27.421 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.290048+0000", 2024-09-21T13:52:27.421 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.428e-06 2024-09-21T13:52:27.421 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.421 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.421 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:52:27.421 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.368619+0000", 2024-09-21T13:52:27.421 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.078571040999999994 2024-09-21T13:52:27.421 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.422 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.422 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:52:27.422 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.368646+0000", 2024-09-21T13:52:27.422 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7206e-05 2024-09-21T13:52:27.422 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.422 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.422 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:52:27.422 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.368878+0000", 2024-09-21T13:52:27.422 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000231943 2024-09-21T13:52:27.422 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.422 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.422 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:52:27.422 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.368888+0000", 2024-09-21T13:52:27.422 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.516e-06 2024-09-21T13:52:27.422 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.423 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.423 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:52:27.423 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.368903+0000", 2024-09-21T13:52:27.423 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5319999999999999e-05 2024-09-21T13:52:27.423 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:52:27.423 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:52:27.423 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:52:27.423 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.423 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.423 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.14817.0:36414 7.4 e96/94 ondisk, result = 0)", 2024-09-21T13:52:27.423 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:30.291205+0000", 2024-09-21T13:52:27.423 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 117.084257393, 2024-09-21T13:52:27.423 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.080307343000000003, 2024-09-21T13:52:27.423 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:52:27.423 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:52:27.424 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T13:52:27.424 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:52:27.424 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.424 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:52:27.424 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.291205+0000", 2024-09-21T13:52:27.424 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:52:27.424 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.424 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.424 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:52:27.424 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.291205+0000", 2024-09-21T13:52:27.424 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:52:27.424 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.424 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.424 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:52:27.424 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.291206+0000", 2024-09-21T13:52:27.424 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.0399999999999995e-07 2024-09-21T13:52:27.425 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.425 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.425 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:52:27.425 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.291210+0000", 2024-09-21T13:52:27.425 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.393e-06 2024-09-21T13:52:27.425 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.425 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.425 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:52:27.425 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.369386+0000", 2024-09-21T13:52:27.425 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.078176271000000006 2024-09-21T13:52:27.425 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.425 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.425 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:52:27.425 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.369396+0000", 2024-09-21T13:52:27.425 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0609e-05 2024-09-21T13:52:27.426 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.426 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.426 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:52:27.426 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.371383+0000", 2024-09-21T13:52:27.426 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0019864169999999999 2024-09-21T13:52:27.426 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.426 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.426 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:52:27.426 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.371391+0000", 2024-09-21T13:52:27.426 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8480000000000001e-06 2024-09-21T13:52:27.426 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.426 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.426 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:52:27.426 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.371513+0000", 2024-09-21T13:52:27.427 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000122101 2024-09-21T13:52:27.427 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:52:27.427 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:52:27.427 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:52:27.427 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.427 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.427 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:36866 7.1 7:81a03870:::benchmark_data_smithi049_76798_object36865:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-21T13:52:27.427 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:30.635343+0000", 2024-09-21T13:52:27.427 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 116.740120133, 2024-09-21T13:52:27.427 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13232190499999999, 2024-09-21T13:52:27.427 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:52:27.427 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:52:27.427 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:52:27.427 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:52:27.428 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:52:27.428 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:52:27.428 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 36866 2024-09-21T13:52:27.428 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.428 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:52:27.428 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.428 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:52:27.428 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.635343+0000", 2024-09-21T13:52:27.428 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:52:27.428 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.428 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.428 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:52:27.428 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.635343+0000", 2024-09-21T13:52:27.428 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:52:27.428 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.429 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.429 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:52:27.429 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.635344+0000", 2024-09-21T13:52:27.429 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6920000000000001e-06 2024-09-21T13:52:27.429 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.429 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.429 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:52:27.429 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.635406+0000", 2024-09-21T13:52:27.429 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1353999999999996e-05 2024-09-21T13:52:27.429 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.429 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.429 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:52:27.429 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.635418+0000", 2024-09-21T13:52:27.429 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2082000000000001e-05 2024-09-21T13:52:27.429 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.429 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.430 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:52:27.430 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.635425+0000", 2024-09-21T13:52:27.430 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1509999999999998e-06 2024-09-21T13:52:27.430 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.430 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.430 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:52:27.430 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.635486+0000", 2024-09-21T13:52:27.430 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0523999999999999e-05 2024-09-21T13:52:27.430 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.430 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.430 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:52:27.430 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.635599+0000", 2024-09-21T13:52:27.431 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000113731 2024-09-21T13:52:27.431 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.431 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.431 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,6", 2024-09-21T13:52:27.431 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.635724+0000", 2024-09-21T13:52:27.431 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000124292 2024-09-21T13:52:27.431 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.431 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.431 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:52:27.431 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.636808+0000", 2024-09-21T13:52:27.431 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001084631 2024-09-21T13:52:27.431 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.431 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.431 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:52:27.432 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.638114+0000", 2024-09-21T13:52:27.432 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0013061679999999999 2024-09-21T13:52:27.432 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.432 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.432 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:52:27.432 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.767545+0000", 2024-09-21T13:52:27.432 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.12943043400000001 2024-09-21T13:52:27.432 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.432 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.432 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:52:27.432 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.767607+0000", 2024-09-21T13:52:27.432 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2065999999999998e-05 2024-09-21T13:52:27.432 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.432 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.433 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:52:27.433 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.767665+0000", 2024-09-21T13:52:27.433 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7779999999999999e-05 2024-09-21T13:52:27.433 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:52:27.433 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:52:27.433 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:52:27.433 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.433 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.433 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:36886 7.1 7:8f0780fe:::benchmark_data_smithi049_76798_object36885:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-21T13:52:27.433 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:30.650584+0000", 2024-09-21T13:52:27.433 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 116.724878554, 2024-09-21T13:52:27.433 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.11732397999999999, 2024-09-21T13:52:27.433 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:52:27.433 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:52:27.434 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:52:27.434 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:52:27.434 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:52:27.434 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:52:27.434 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 36886 2024-09-21T13:52:27.434 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.434 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:52:27.434 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.434 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:52:27.434 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.650584+0000", 2024-09-21T13:52:27.434 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:52:27.434 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.434 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.434 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:52:27.435 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.650584+0000", 2024-09-21T13:52:27.435 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:52:27.435 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.435 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.435 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:52:27.435 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.650587+0000", 2024-09-21T13:52:27.435 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5600000000000001e-06 2024-09-21T13:52:27.435 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.435 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.435 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:52:27.435 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.650631+0000", 2024-09-21T13:52:27.435 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4499999999999997e-05 2024-09-21T13:52:27.435 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.435 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.436 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:52:27.436 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.650643+0000", 2024-09-21T13:52:27.436 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1404e-05 2024-09-21T13:52:27.436 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.436 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.436 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:52:27.436 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.650648+0000", 2024-09-21T13:52:27.436 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1800000000000004e-06 2024-09-21T13:52:27.436 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.436 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.436 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:52:27.436 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.650689+0000", 2024-09-21T13:52:27.436 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0784000000000001e-05 2024-09-21T13:52:27.436 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.437 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.437 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:52:27.437 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.650777+0000", 2024-09-21T13:52:27.437 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8154999999999999e-05 2024-09-21T13:52:27.437 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.437 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.437 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,6", 2024-09-21T13:52:27.437 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.650871+0000", 2024-09-21T13:52:27.437 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3738000000000003e-05 2024-09-21T13:52:27.437 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.437 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.437 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:52:27.437 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.651931+0000", 2024-09-21T13:52:27.437 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0010604270000000001 2024-09-21T13:52:27.438 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.438 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.438 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:52:27.438 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.652228+0000", 2024-09-21T13:52:27.438 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000296507 2024-09-21T13:52:27.438 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.438 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.438 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:52:27.438 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.767813+0000", 2024-09-21T13:52:27.438 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.115584956 2024-09-21T13:52:27.438 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.438 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.438 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:52:27.439 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.767869+0000", 2024-09-21T13:52:27.439 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6851999999999997e-05 2024-09-21T13:52:27.439 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.439 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.439 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:52:27.439 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.767908+0000", 2024-09-21T13:52:27.439 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8917e-05 2024-09-21T13:52:27.439 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:52:27.439 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:52:27.439 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:52:27.439 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.439 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.439 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:51696 7.4 7:29a9ea16:::benchmark_data_smithi049_76798_object51695:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-21T13:52:27.439 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:39.982665+0000", 2024-09-21T13:52:27.440 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 107.39279833499999, 2024-09-21T13:52:27.440 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.085001300000000002, 2024-09-21T13:52:27.440 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:52:27.440 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:52:27.440 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:52:27.440 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:52:27.440 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:52:27.440 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:52:27.440 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 51696 2024-09-21T13:52:27.440 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.440 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:52:27.440 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.440 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:52:27.440 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.982665+0000", 2024-09-21T13:52:27.440 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:52:27.441 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.441 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.441 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:52:27.441 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.982665+0000", 2024-09-21T13:52:27.441 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:52:27.441 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.441 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.441 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:52:27.441 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.982666+0000", 2024-09-21T13:52:27.441 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8250000000000001e-06 2024-09-21T13:52:27.441 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.441 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.441 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:52:27.441 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.982722+0000", 2024-09-21T13:52:27.442 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5636000000000001e-05 2024-09-21T13:52:27.442 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.442 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.442 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:52:27.442 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.982740+0000", 2024-09-21T13:52:27.442 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7643e-05 2024-09-21T13:52:27.442 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.442 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.442 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:52:27.442 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.982750+0000", 2024-09-21T13:52:27.442 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0125999999999999e-05 2024-09-21T13:52:27.442 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.442 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.442 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:52:27.443 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.982810+0000", 2024-09-21T13:52:27.443 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0420000000000001e-05 2024-09-21T13:52:27.443 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.443 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.443 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:52:27.443 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.982933+0000", 2024-09-21T13:52:27.443 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000122736 2024-09-21T13:52:27.443 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.443 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.443 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-21T13:52:27.443 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.983067+0000", 2024-09-21T13:52:27.443 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000134349 2024-09-21T13:52:27.443 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.443 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.444 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:52:27.444 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.984794+0000", 2024-09-21T13:52:27.444 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001726769 2024-09-21T13:52:27.444 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.444 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.444 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:52:27.444 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.985673+0000", 2024-09-21T13:52:27.444 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00087921199999999998 2024-09-21T13:52:27.444 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.444 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.444 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:52:27.444 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:40.067567+0000", 2024-09-21T13:52:27.444 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.081893401000000005 2024-09-21T13:52:27.444 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.445 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.445 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:52:27.445 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:40.067623+0000", 2024-09-21T13:52:27.445 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5878999999999998e-05 2024-09-21T13:52:27.445 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.445 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.445 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:52:27.445 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:40.067666+0000", 2024-09-21T13:52:27.445 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3303999999999997e-05 2024-09-21T13:52:27.445 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:52:27.445 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:52:27.445 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:52:27.445 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.445 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.445 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:51718 7.4 7:2354644a:::benchmark_data_smithi049_76798_object51717:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-21T13:52:27.446 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:39.995425+0000", 2024-09-21T13:52:27.446 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 107.38003773600001, 2024-09-21T13:52:27.446 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.077269423000000004, 2024-09-21T13:52:27.446 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:52:27.446 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:52:27.446 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:52:27.446 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:52:27.446 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:52:27.446 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:52:27.446 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 51718 2024-09-21T13:52:27.446 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.446 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:52:27.446 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.446 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:52:27.447 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.995425+0000", 2024-09-21T13:52:27.447 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:52:27.447 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.447 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.447 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:52:27.447 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.995425+0000", 2024-09-21T13:52:27.447 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:52:27.447 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.447 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.447 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:52:27.447 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.995426+0000", 2024-09-21T13:52:27.447 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.47e-07 2024-09-21T13:52:27.447 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.447 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.448 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:52:27.448 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.995472+0000", 2024-09-21T13:52:27.448 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5534000000000001e-05 2024-09-21T13:52:27.448 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.448 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.448 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:52:27.448 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.995484+0000", 2024-09-21T13:52:27.448 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2802999999999999e-05 2024-09-21T13:52:27.448 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.448 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.448 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:52:27.448 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.995492+0000", 2024-09-21T13:52:27.448 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9230000000000002e-06 2024-09-21T13:52:27.448 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.448 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.449 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:52:27.449 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.995535+0000", 2024-09-21T13:52:27.449 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2537999999999998e-05 2024-09-21T13:52:27.449 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.449 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.449 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:52:27.449 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.995622+0000", 2024-09-21T13:52:27.449 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.6759000000000001e-05 2024-09-21T13:52:27.449 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.449 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.449 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-21T13:52:27.449 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.995772+0000", 2024-09-21T13:52:27.449 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015015300000000001 2024-09-21T13:52:27.449 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.450 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.450 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:52:27.450 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.997454+0000", 2024-09-21T13:52:27.450 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001682612 2024-09-21T13:52:27.450 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.450 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.450 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:52:27.450 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.997889+0000", 2024-09-21T13:52:27.450 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00043466499999999999 2024-09-21T13:52:27.450 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.450 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.450 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:52:27.450 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:40.072593+0000", 2024-09-21T13:52:27.450 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.074703529000000005 2024-09-21T13:52:27.450 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.451 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.451 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:52:27.451 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:40.072653+0000", 2024-09-21T13:52:27.451 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9988000000000002e-05 2024-09-21T13:52:27.451 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.451 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.451 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:52:27.451 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:40.072695+0000", 2024-09-21T13:52:27.451 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1971999999999997e-05 2024-09-21T13:52:27.451 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:52:27.451 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:52:27.451 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:52:27.451 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.451 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.452 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:72620 7.4 7:21febffb:::benchmark_data_smithi049_76798_object72619:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T13:52:27.452 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:54.472955+0000", 2024-09-21T13:52:27.452 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 92.902507697000004, 2024-09-21T13:52:27.452 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.092558697999999995, 2024-09-21T13:52:27.452 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:52:27.452 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:52:27.452 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:52:27.452 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:52:27.452 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:52:27.452 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:52:27.452 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 72620 2024-09-21T13:52:27.452 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.452 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:52:27.452 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.452 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:52:27.453 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.472955+0000", 2024-09-21T13:52:27.453 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:52:27.453 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.453 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.453 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:52:27.453 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.472955+0000", 2024-09-21T13:52:27.453 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:52:27.453 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.453 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.453 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:52:27.453 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.472956+0000", 2024-09-21T13:52:27.453 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2500000000000001e-06 2024-09-21T13:52:27.453 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.453 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.453 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:52:27.454 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.473008+0000", 2024-09-21T13:52:27.454 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1379999999999999e-05 2024-09-21T13:52:27.454 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.454 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.454 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:52:27.454 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.473020+0000", 2024-09-21T13:52:27.454 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2687999999999999e-05 2024-09-21T13:52:27.454 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.454 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.454 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:52:27.454 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.473028+0000", 2024-09-21T13:52:27.454 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6850000000000001e-06 2024-09-21T13:52:27.454 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.454 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.454 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:52:27.455 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.473103+0000", 2024-09-21T13:52:27.455 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4821999999999996e-05 2024-09-21T13:52:27.455 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.455 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.455 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:52:27.455 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.473221+0000", 2024-09-21T13:52:27.455 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011758 2024-09-21T13:52:27.455 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.455 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.455 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-21T13:52:27.455 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.473351+0000", 2024-09-21T13:52:27.455 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000130359 2024-09-21T13:52:27.455 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.455 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.455 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:52:27.456 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.476613+0000", 2024-09-21T13:52:27.456 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0032616120000000001 2024-09-21T13:52:27.456 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.456 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.456 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:52:27.456 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.479113+0000", 2024-09-21T13:52:27.456 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0025009720000000002 2024-09-21T13:52:27.456 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.456 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.456 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:52:27.456 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.565355+0000", 2024-09-21T13:52:27.456 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.086241622000000004 2024-09-21T13:52:27.456 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.456 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.456 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:52:27.456 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.565419+0000", 2024-09-21T13:52:27.457 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4358999999999998e-05 2024-09-21T13:52:27.457 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.457 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.457 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:52:27.457 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.565514+0000", 2024-09-21T13:52:27.457 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.4369000000000004e-05 2024-09-21T13:52:27.457 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:52:27.457 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:52:27.457 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:52:27.457 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.457 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:27.458 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:72632 7.4 7:2125dae0:::benchmark_data_smithi049_76798_object72631:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T13:52:27.458 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:54.480545+0000", 2024-09-21T13:52:27.459 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 92.894918274000005, 2024-09-21T13:52:27.459 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.085078231000000004, 2024-09-21T13:52:27.459 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:52:27.459 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:52:27.459 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:52:27.459 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:52:27.459 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:52:27.459 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:52:27.459 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 72632 2024-09-21T13:52:27.459 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.459 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:52:27.459 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.459 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:52:27.459 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.480545+0000", 2024-09-21T13:52:27.460 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:52:27.460 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.460 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.460 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:52:27.460 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.480545+0000", 2024-09-21T13:52:27.460 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:52:27.460 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.460 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.460 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:52:27.460 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.480546+0000", 2024-09-21T13:52:27.460 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.694e-06 2024-09-21T13:52:27.460 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.460 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.461 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:52:27.461 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.480584+0000", 2024-09-21T13:52:27.461 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7435999999999999e-05 2024-09-21T13:52:27.461 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.461 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.461 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:52:27.461 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.480599+0000", 2024-09-21T13:52:27.461 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5014999999999999e-05 2024-09-21T13:52:27.461 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.461 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.461 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:52:27.461 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.480606+0000", 2024-09-21T13:52:27.461 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2849999999999997e-06 2024-09-21T13:52:27.461 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.462 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.462 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:52:27.462 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.480652+0000", 2024-09-21T13:52:27.462 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6254999999999997e-05 2024-09-21T13:52:27.462 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.462 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.462 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:52:27.462 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.480773+0000", 2024-09-21T13:52:27.462 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000121079 2024-09-21T13:52:27.462 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.462 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.462 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-21T13:52:27.462 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.480960+0000", 2024-09-21T13:52:27.462 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00018655399999999999 2024-09-21T13:52:27.463 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.463 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.463 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:52:27.463 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.485213+0000", 2024-09-21T13:52:27.463 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0042532510000000004 2024-09-21T13:52:27.463 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.463 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.463 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:52:27.463 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.485239+0000", 2024-09-21T13:52:27.463 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.6004999999999999e-05 2024-09-21T13:52:27.463 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.463 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.463 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:52:27.463 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.565549+0000", 2024-09-21T13:52:27.464 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.080310030000000004 2024-09-21T13:52:27.464 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.464 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.464 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:52:27.464 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.565593+0000", 2024-09-21T13:52:27.464 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3723999999999997e-05 2024-09-21T13:52:27.464 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.464 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.464 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:52:27.464 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.565623+0000", 2024-09-21T13:52:27.464 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9903000000000002e-05 2024-09-21T13:52:27.464 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:52:27.464 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:52:27.464 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:52:27.464 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.464 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.465 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:72633 7.4 7:2a45d345:::benchmark_data_smithi049_76798_object72632:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T13:52:27.465 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:54.480891+0000", 2024-09-21T13:52:27.465 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 92.894571987999996, 2024-09-21T13:52:27.465 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.084841954999999997, 2024-09-21T13:52:27.465 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:52:27.465 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:52:27.465 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:52:27.465 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:52:27.465 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:52:27.465 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:52:27.465 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 72633 2024-09-21T13:52:27.465 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.465 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:52:27.465 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.465 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:52:27.466 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.480891+0000", 2024-09-21T13:52:27.466 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:52:27.466 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.466 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.466 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:52:27.466 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.480891+0000", 2024-09-21T13:52:27.466 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:52:27.466 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.466 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.466 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:52:27.466 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.480892+0000", 2024-09-21T13:52:27.466 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0559999999999999e-06 2024-09-21T13:52:27.466 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.466 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.466 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:52:27.467 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.480956+0000", 2024-09-21T13:52:27.467 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4314999999999997e-05 2024-09-21T13:52:27.467 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.467 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.467 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:52:27.467 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.480968+0000", 2024-09-21T13:52:27.467 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1392e-05 2024-09-21T13:52:27.467 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.467 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.467 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:52:27.467 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.480975+0000", 2024-09-21T13:52:27.467 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7719999999999994e-06 2024-09-21T13:52:27.467 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.467 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.468 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:52:27.468 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.483108+0000", 2024-09-21T13:52:27.468 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0021330590000000001 2024-09-21T13:52:27.468 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.468 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.468 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:52:27.468 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.483222+0000", 2024-09-21T13:52:27.468 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000113217 2024-09-21T13:52:27.468 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.468 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.468 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-21T13:52:27.468 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.483351+0000", 2024-09-21T13:52:27.468 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000129147 2024-09-21T13:52:27.468 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.468 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.469 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:52:27.469 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.485328+0000", 2024-09-21T13:52:27.469 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0019775000000000001 2024-09-21T13:52:27.469 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.469 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.469 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:52:27.469 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.485447+0000", 2024-09-21T13:52:27.469 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000118727 2024-09-21T13:52:27.469 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.469 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.469 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:52:27.469 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.565661+0000", 2024-09-21T13:52:27.469 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.080213941999999996 2024-09-21T13:52:27.469 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.469 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.470 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:52:27.470 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.565705+0000", 2024-09-21T13:52:27.470 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3847999999999998e-05 2024-09-21T13:52:27.470 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.470 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.470 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:52:27.470 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.565733+0000", 2024-09-21T13:52:27.470 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.798e-05 2024-09-21T13:52:27.470 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:52:27.470 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:52:27.470 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:52:27.470 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.470 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.470 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:100027 7.4 7:2377bcea:::benchmark_data_smithi049_76798_object100026:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-09-21T13:52:27.471 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:51:14.142163+0000", 2024-09-21T13:52:27.471 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 73.233299895000002, 2024-09-21T13:52:27.471 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.082779666000000002, 2024-09-21T13:52:27.471 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:52:27.471 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:52:27.471 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:52:27.471 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:52:27.471 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:52:27.471 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:52:27.471 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 100027 2024-09-21T13:52:27.471 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.471 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:52:27.471 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.471 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:52:27.472 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:14.142163+0000", 2024-09-21T13:52:27.472 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:52:27.472 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.472 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.472 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:52:27.472 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:14.142163+0000", 2024-09-21T13:52:27.472 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:52:27.472 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.472 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.472 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:52:27.472 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:14.142164+0000", 2024-09-21T13:52:27.472 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5099999999999999e-06 2024-09-21T13:52:27.472 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.472 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.473 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:52:27.473 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:14.142192+0000", 2024-09-21T13:52:27.473 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7705e-05 2024-09-21T13:52:27.473 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.473 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.473 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:52:27.473 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:14.142208+0000", 2024-09-21T13:52:27.473 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5545000000000001e-05 2024-09-21T13:52:27.473 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.473 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.473 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:52:27.473 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:14.142215+0000", 2024-09-21T13:52:27.473 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.464e-06 2024-09-21T13:52:27.473 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.473 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.474 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:52:27.474 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:14.142263+0000", 2024-09-21T13:52:27.474 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7753999999999997e-05 2024-09-21T13:52:27.474 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.474 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.474 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:52:27.474 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:14.142372+0000", 2024-09-21T13:52:27.474 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000108953 2024-09-21T13:52:27.474 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.474 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.474 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-21T13:52:27.474 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:14.142506+0000", 2024-09-21T13:52:27.474 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000134587 2024-09-21T13:52:27.474 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.474 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.475 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:52:27.475 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:14.145116+0000", 2024-09-21T13:52:27.475 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0026092009999999998 2024-09-21T13:52:27.475 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.475 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.475 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:52:27.475 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:14.146110+0000", 2024-09-21T13:52:27.475 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00099431999999999993 2024-09-21T13:52:27.475 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.475 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.475 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:52:27.475 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:14.224882+0000", 2024-09-21T13:52:27.475 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.078772357000000001 2024-09-21T13:52:27.475 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.476 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.476 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:52:27.476 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:14.224917+0000", 2024-09-21T13:52:27.476 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4249999999999999e-05 2024-09-21T13:52:27.476 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.476 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.476 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:52:27.476 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:14.224943+0000", 2024-09-21T13:52:27.476 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.6020000000000002e-05 2024-09-21T13:52:27.476 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:52:27.476 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:52:27.476 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:52:27.476 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.477 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.477 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:166057 7.1 7:8806b5cb:::benchmark_data_smithi049_76798_object29637:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-09-21T13:52:27.477 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:07.436722+0000", 2024-09-21T13:52:27.477 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 19.938740464999999, 2024-09-21T13:52:27.477 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.081910074999999999, 2024-09-21T13:52:27.477 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:52:27.477 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:52:27.477 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:52:27.477 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:52:27.477 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:52:27.477 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:52:27.477 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 166057 2024-09-21T13:52:27.477 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.477 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:52:27.478 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.478 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:52:27.478 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:07.436722+0000", 2024-09-21T13:52:27.478 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:52:27.478 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.478 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.478 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:52:27.478 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:07.436722+0000", 2024-09-21T13:52:27.478 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:52:27.478 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.478 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.478 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:52:27.478 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:07.436724+0000", 2024-09-21T13:52:27.478 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.341e-06 2024-09-21T13:52:27.479 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.479 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.479 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:52:27.479 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:07.436728+0000", 2024-09-21T13:52:27.479 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5510000000000001e-06 2024-09-21T13:52:27.479 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.479 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.479 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:52:27.479 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:07.436741+0000", 2024-09-21T13:52:27.479 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3196999999999999e-05 2024-09-21T13:52:27.479 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.479 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.479 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:52:27.479 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:07.436749+0000", 2024-09-21T13:52:27.480 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2150000000000004e-06 2024-09-21T13:52:27.480 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.480 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.480 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:52:27.480 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:07.436813+0000", 2024-09-21T13:52:27.480 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4293000000000003e-05 2024-09-21T13:52:27.480 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.480 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.480 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:52:27.480 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:07.436963+0000", 2024-09-21T13:52:27.480 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000150032 2024-09-21T13:52:27.480 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.480 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.481 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,6", 2024-09-21T13:52:27.481 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:07.437068+0000", 2024-09-21T13:52:27.481 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000104644 2024-09-21T13:52:27.481 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.481 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.481 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:52:27.481 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:07.440186+0000", 2024-09-21T13:52:27.481 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0031186040000000001 2024-09-21T13:52:27.481 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.481 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.481 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:52:27.481 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:07.440269+0000", 2024-09-21T13:52:27.481 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2299999999999995e-05 2024-09-21T13:52:27.481 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.481 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.481 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:52:27.482 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:07.518571+0000", 2024-09-21T13:52:27.482 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.078302215999999994 2024-09-21T13:52:27.482 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.482 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.482 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:52:27.482 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:07.518607+0000", 2024-09-21T13:52:27.482 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.608e-05 2024-09-21T13:52:27.482 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.482 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.482 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:52:27.482 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:07.518632+0000", 2024-09-21T13:52:27.482 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5602e-05 2024-09-21T13:52:27.482 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:52:27.483 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:52:27.483 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:52:27.483 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.483 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.483 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:172089 7.4 7:236b8a6b:::benchmark_data_smithi049_76798_object35669:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-09-21T13:52:27.483 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:13.086882+0000", 2024-09-21T13:52:27.483 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 14.288580469999999, 2024-09-21T13:52:27.483 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.096754271000000003, 2024-09-21T13:52:27.483 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:52:27.483 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:52:27.483 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:52:27.483 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:52:27.483 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:52:27.483 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:52:27.483 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 172089 2024-09-21T13:52:27.484 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.484 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:52:27.484 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.484 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:52:27.484 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:13.086882+0000", 2024-09-21T13:52:27.484 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:52:27.484 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.484 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.484 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:52:27.484 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:13.086882+0000", 2024-09-21T13:52:27.484 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:52:27.484 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.484 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.484 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:52:27.485 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:13.086884+0000", 2024-09-21T13:52:27.485 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4300000000000001e-06 2024-09-21T13:52:27.485 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.485 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.485 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:52:27.485 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:13.086889+0000", 2024-09-21T13:52:27.485 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2669999999999997e-06 2024-09-21T13:52:27.485 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.485 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.485 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:52:27.485 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:13.086901+0000", 2024-09-21T13:52:27.485 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1793999999999999e-05 2024-09-21T13:52:27.485 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.485 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.486 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:52:27.486 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:13.086908+0000", 2024-09-21T13:52:27.486 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1799999999999999e-06 2024-09-21T13:52:27.486 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.486 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.486 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:52:27.486 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:13.086960+0000", 2024-09-21T13:52:27.486 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2367999999999997e-05 2024-09-21T13:52:27.486 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.486 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.486 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:52:27.486 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:13.087072+0000", 2024-09-21T13:52:27.486 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000111779 2024-09-21T13:52:27.486 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.487 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.487 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-21T13:52:27.487 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:13.087181+0000", 2024-09-21T13:52:27.487 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000109184 2024-09-21T13:52:27.487 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.487 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.487 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:52:27.487 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:13.089557+0000", 2024-09-21T13:52:27.487 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0023758220000000001 2024-09-21T13:52:27.487 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.487 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.487 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:52:27.487 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:13.089639+0000", 2024-09-21T13:52:27.487 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1357e-05 2024-09-21T13:52:27.487 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.488 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.488 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:52:27.488 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:13.183539+0000", 2024-09-21T13:52:27.488 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.093900288999999998 2024-09-21T13:52:27.488 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.488 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.488 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:52:27.488 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:13.183601+0000", 2024-09-21T13:52:27.488 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2244999999999998e-05 2024-09-21T13:52:27.488 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.488 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.488 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:52:27.488 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:13.183637+0000", 2024-09-21T13:52:27.488 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5556000000000003e-05 2024-09-21T13:52:27.489 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:52:27.489 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:52:27.489 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:52:27.489 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.489 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.489 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:185401 7.1 7:89febdc9:::benchmark_data_smithi049_76798_object48981:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-21T13:52:27.489 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:26.179574+0000", 2024-09-21T13:52:27.489 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 1.195889293, 2024-09-21T13:52:27.489 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.105705047, 2024-09-21T13:52:27.489 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:52:27.489 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:52:27.489 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:52:27.489 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:52:27.489 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:52:27.489 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:52:27.490 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 185401 2024-09-21T13:52:27.490 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.490 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:52:27.490 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.490 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:52:27.490 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:26.179574+0000", 2024-09-21T13:52:27.490 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:52:27.490 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.490 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.490 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:52:27.490 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:26.179574+0000", 2024-09-21T13:52:27.490 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:52:27.490 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.490 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.491 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:52:27.491 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:26.179575+0000", 2024-09-21T13:52:27.491 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3850000000000001e-06 2024-09-21T13:52:27.491 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.491 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.491 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:52:27.491 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:26.179579+0000", 2024-09-21T13:52:27.491 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.968e-06 2024-09-21T13:52:27.491 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.491 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.491 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:52:27.491 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:26.179588+0000", 2024-09-21T13:52:27.491 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8240000000000006e-06 2024-09-21T13:52:27.491 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.491 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.492 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:52:27.492 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:26.179594+0000", 2024-09-21T13:52:27.492 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1210000000000003e-06 2024-09-21T13:52:27.492 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.492 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.492 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:52:27.492 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:26.179654+0000", 2024-09-21T13:52:27.492 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0315000000000001e-05 2024-09-21T13:52:27.492 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.492 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.492 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:52:27.492 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:26.179772+0000", 2024-09-21T13:52:27.492 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000118232 2024-09-21T13:52:27.492 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.493 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.493 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,6", 2024-09-21T13:52:27.493 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:26.179956+0000", 2024-09-21T13:52:27.493 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00018340500000000001 2024-09-21T13:52:27.493 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.493 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.493 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:52:27.493 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:26.183309+0000", 2024-09-21T13:52:27.493 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003352743 2024-09-21T13:52:27.493 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.493 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.493 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:52:27.493 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:26.184074+0000", 2024-09-21T13:52:27.493 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00076550499999999998 2024-09-21T13:52:27.493 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.494 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.494 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:52:27.494 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:26.285117+0000", 2024-09-21T13:52:27.494 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.101043312 2024-09-21T13:52:27.494 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.494 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.494 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:52:27.494 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:26.285235+0000", 2024-09-21T13:52:27.494 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000118023 2024-09-21T13:52:27.494 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:27.494 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:27.494 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:52:27.494 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:26.285279+0000", 2024-09-21T13:52:27.494 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3214000000000003e-05 2024-09-21T13:52:27.495 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:52:27.495 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:52:27.495 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:52:27.495 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:52:27.495 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:52:27.495 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T13:52:27.495 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:27.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:52:27.596 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:52:27.624 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:27.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:27.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:52:27.633 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:52:27.678 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.2 to 1 (10000) 2024-09-21T13:52:27.726 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:52:27.781 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:27.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:27.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:52:27.884 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:52:27.921 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:27.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:27.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:52:27.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:27 smithi187 ceph-mon[33032]: pgmap v156: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 829 B/s rd, 0 B/s wr, 855 op/s; 26565/271587 objects degraded (9.781%); 18 MiB/s, 281 objects/s recovering 2024-09-21T13:52:27.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:27 smithi187 ceph-mon[33032]: 3.1f scrub starts 2024-09-21T13:52:27.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:27 smithi187 ceph-mon[33032]: 3.1f scrub ok 2024-09-21T13:52:27.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:27 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2637082177' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T13:52:27.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:27 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T13:52:27.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:27 smithi187 ceph-mon[33032]: 6.1d scrub starts 2024-09-21T13:52:27.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:27 smithi187 ceph-mon[33032]: 6.1d scrub ok 2024-09-21T13:52:28.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:27 smithi049 ceph-mon[30080]: pgmap v156: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 829 B/s rd, 0 B/s wr, 855 op/s; 26565/271587 objects degraded (9.781%); 18 MiB/s, 281 objects/s recovering 2024-09-21T13:52:28.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:27 smithi049 ceph-mon[30080]: 3.1f scrub starts 2024-09-21T13:52:28.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:27 smithi049 ceph-mon[30080]: 3.1f scrub ok 2024-09-21T13:52:28.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:27 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2637082177' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T13:52:28.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:27 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T13:52:28.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:27 smithi049 ceph-mon[30080]: 6.1d scrub starts 2024-09-21T13:52:28.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:27 smithi049 ceph-mon[30080]: 6.1d scrub ok 2024-09-21T13:52:28.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:27 smithi049 ceph-mon[40489]: pgmap v156: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 829 B/s rd, 0 B/s wr, 855 op/s; 26565/271587 objects degraded (9.781%); 18 MiB/s, 281 objects/s recovering 2024-09-21T13:52:28.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:27 smithi049 ceph-mon[40489]: 3.1f scrub starts 2024-09-21T13:52:28.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:27 smithi049 ceph-mon[40489]: 3.1f scrub ok 2024-09-21T13:52:28.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:27 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2637082177' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T13:52:28.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:27 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T13:52:28.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:27 smithi049 ceph-mon[40489]: 6.1d scrub starts 2024-09-21T13:52:28.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:27 smithi049 ceph-mon[40489]: 6.1d scrub ok 2024-09-21T13:52:28.023 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:52:28.051 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:28.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:28.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:52:28.153 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:52:28.181 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:28.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:28.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:52:28.283 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:52:28.318 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:28.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:28.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:52:28.420 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:52:28.448 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:28.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:28.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:52:28.461 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd reweight 3 1 2024-09-21T13:52:28.500 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T13:52:28.551 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:52:28.585 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:28.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:28.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:52:28.687 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:52:28.717 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:28.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:28.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:52:28.819 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:52:28.863 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:52:28.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:28.876 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:28.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:52:28.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:28 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T13:52:28.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:28 smithi187 ceph-mon[33032]: osdmap e139: 8 total, 8 up, 8 in 2024-09-21T13:52:28.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:28 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:52:28.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:28 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:52:28.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:28 smithi187 ceph-mon[33032]: 6.18 scrub starts 2024-09-21T13:52:28.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:28 smithi187 ceph-mon[33032]: 6.18 scrub ok 2024-09-21T13:52:28.977 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:52:29.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:28 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T13:52:29.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:28 smithi049 ceph-mon[40489]: osdmap e139: 8 total, 8 up, 8 in 2024-09-21T13:52:29.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:28 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:52:29.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:28 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:52:29.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:28 smithi049 ceph-mon[40489]: 6.18 scrub starts 2024-09-21T13:52:29.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:28 smithi049 ceph-mon[40489]: 6.18 scrub ok 2024-09-21T13:52:29.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:28 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T13:52:29.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:28 smithi049 ceph-mon[30080]: osdmap e139: 8 total, 8 up, 8 in 2024-09-21T13:52:29.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:28 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:52:29.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:28 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:52:29.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:28 smithi049 ceph-mon[30080]: 6.18 scrub starts 2024-09-21T13:52:29.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:28 smithi049 ceph-mon[30080]: 6.18 scrub ok 2024-09-21T13:52:29.022 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:29.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:29.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:52:29.110 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:52:29.125 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:29.159 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:29.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:29.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:29.261 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:52:29.295 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:29.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:29.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:52:29.318 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:52:29 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:52:29.397 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:52:29.428 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:29.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:29.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:52:29.461 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T13:52:29.461 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-21T13:52:29.461 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:29.461 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:188673 7.0 7:0bb09906:::benchmark_data_smithi049_76798_object52253:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T13:52:29.462 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:29.442528+0000", 2024-09-21T13:52:29.462 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.017886886000000001, 2024-09-21T13:52:29.462 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.017924516000000001, 2024-09-21T13:52:29.462 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:52:29.462 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:52:29.462 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-21T13:52:29.462 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:52:29.462 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:52:29.462 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:52:29.462 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 188673 2024-09-21T13:52:29.462 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:29.462 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:52:29.463 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:29.463 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:52:29.463 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:29.442528+0000", 2024-09-21T13:52:29.463 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:52:29.463 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:29.463 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:29.463 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:52:29.463 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:29.442528+0000", 2024-09-21T13:52:29.463 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:52:29.463 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:29.463 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:29.463 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:52:29.463 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:29.442530+0000", 2024-09-21T13:52:29.463 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6190000000000001e-06 2024-09-21T13:52:29.464 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:29.464 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:29.464 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:52:29.464 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:29.442535+0000", 2024-09-21T13:52:29.464 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8110000000000002e-06 2024-09-21T13:52:29.464 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:29.464 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:29.464 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:52:29.464 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:29.442549+0000", 2024-09-21T13:52:29.464 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3905e-05 2024-09-21T13:52:29.464 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:29.464 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:29.464 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:52:29.464 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:29.442557+0000", 2024-09-21T13:52:29.464 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5459999999999992e-06 2024-09-21T13:52:29.465 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:29.465 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:29.465 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:52:29.465 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:29.442663+0000", 2024-09-21T13:52:29.465 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010552600000000001 2024-09-21T13:52:29.465 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:29.465 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:29.465 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:52:29.465 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:29.442770+0000", 2024-09-21T13:52:29.465 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000106782 2024-09-21T13:52:29.465 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:29.465 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:29.465 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3", 2024-09-21T13:52:29.465 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:29.442959+0000", 2024-09-21T13:52:29.466 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00018939400000000001 2024-09-21T13:52:29.466 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:29.466 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:29.466 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:52:29.466 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:29.449570+0000", 2024-09-21T13:52:29.466 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0066112810000000001 2024-09-21T13:52:29.466 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:29.466 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:29.466 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:52:29.466 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:29.454936+0000", 2024-09-21T13:52:29.466 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0053659019999999997 2024-09-21T13:52:29.466 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:52:29.466 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:52:29.466 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:52:29.467 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:29.467 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:29.467 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:188678 7.0 7:00edd945:::benchmark_data_smithi049_76798_object52258:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T13:52:29.467 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:29.444635+0000", 2024-09-21T13:52:29.467 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.015780104999999999, 2024-09-21T13:52:29.467 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.015913173999999999, 2024-09-21T13:52:29.467 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:52:29.467 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:52:29.467 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-21T13:52:29.467 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:52:29.467 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:52:29.467 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:52:29.467 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 188678 2024-09-21T13:52:29.467 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:29.468 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:52:29.468 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:29.468 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:52:29.468 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:29.444635+0000", 2024-09-21T13:52:29.468 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:52:29.468 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:29.468 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:29.468 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:52:29.468 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:29.444635+0000", 2024-09-21T13:52:29.468 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:52:29.468 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:29.468 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:29.468 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:52:29.469 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:29.444636+0000", 2024-09-21T13:52:29.469 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.238e-06 2024-09-21T13:52:29.469 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:29.469 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:29.469 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:52:29.469 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:29.444641+0000", 2024-09-21T13:52:29.469 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6110000000000004e-06 2024-09-21T13:52:29.469 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:29.469 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:29.469 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:52:29.469 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:29.444655+0000", 2024-09-21T13:52:29.469 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3912e-05 2024-09-21T13:52:29.469 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:29.470 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:29.470 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:52:29.470 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:29.444662+0000", 2024-09-21T13:52:29.470 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4109999999999999e-06 2024-09-21T13:52:29.470 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:29.470 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:29.470 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:52:29.470 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:29.446189+0000", 2024-09-21T13:52:29.470 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0015270629999999999 2024-09-21T13:52:29.470 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:29.470 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:29.470 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:52:29.470 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:29.446899+0000", 2024-09-21T13:52:29.470 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000709799 2024-09-21T13:52:29.471 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:29.471 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:29.471 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3", 2024-09-21T13:52:29.471 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:29.447002+0000", 2024-09-21T13:52:29.471 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000103213 2024-09-21T13:52:29.471 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:29.471 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:29.471 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:52:29.471 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:29.449684+0000", 2024-09-21T13:52:29.471 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0026818739999999999 2024-09-21T13:52:29.471 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:52:29.471 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:52:29.471 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:52:29.471 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:29.472 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:29.472 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:188682 7.0 7:05ba9035:::benchmark_data_smithi049_76798_object52262:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T13:52:29.472 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:29.451732+0000", 2024-09-21T13:52:29.472 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0086828889999999992, 2024-09-21T13:52:29.472 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0088888430000000004, 2024-09-21T13:52:29.472 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:52:29.472 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:52:29.472 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-21T13:52:29.472 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:52:29.472 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:52:29.472 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:52:29.472 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 188682 2024-09-21T13:52:29.472 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:29.472 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:52:29.472 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:29.473 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:52:29.473 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:29.451732+0000", 2024-09-21T13:52:29.473 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:52:29.473 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:29.473 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:29.473 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:52:29.473 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:29.451732+0000", 2024-09-21T13:52:29.473 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:52:29.473 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:29.473 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:29.473 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:52:29.473 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:29.451734+0000", 2024-09-21T13:52:29.473 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.26e-06 2024-09-21T13:52:29.473 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:29.474 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:29.474 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:52:29.474 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:29.451740+0000", 2024-09-21T13:52:29.474 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3060000000000003e-06 2024-09-21T13:52:29.474 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:29.474 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:29.474 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:52:29.474 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:29.451752+0000", 2024-09-21T13:52:29.474 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1929e-05 2024-09-21T13:52:29.474 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:29.474 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:29.474 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:52:29.474 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:29.451758+0000", 2024-09-21T13:52:29.474 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8819999999999998e-06 2024-09-21T13:52:29.475 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:29.475 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:29.475 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:52:29.475 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:29.451819+0000", 2024-09-21T13:52:29.475 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1319e-05 2024-09-21T13:52:29.475 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:29.475 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:29.475 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:52:29.475 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:29.451894+0000", 2024-09-21T13:52:29.475 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4957999999999996e-05 2024-09-21T13:52:29.475 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:29.475 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:29.475 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3", 2024-09-21T13:52:29.475 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:29.452002+0000", 2024-09-21T13:52:29.475 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000108127 2024-09-21T13:52:29.476 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:29.476 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:29.476 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:52:29.476 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:29.457135+0000", 2024-09-21T13:52:29.476 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0051324659999999996 2024-09-21T13:52:29.476 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:52:29.476 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:52:29.476 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:52:29.476 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:29.476 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:29.476 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:188684 7.0 7:089b17eb:::benchmark_data_smithi049_76798_object52264:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-21T13:52:29.476 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:29.452712+0000", 2024-09-21T13:52:29.476 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0077028679999999999, 2024-09-21T13:52:29.476 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0079809730000000006, 2024-09-21T13:52:29.477 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:52:29.477 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:52:29.477 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-21T13:52:29.477 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:52:29.477 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:52:29.477 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:52:29.477 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 188684 2024-09-21T13:52:29.477 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:29.477 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:52:29.477 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:29.477 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:52:29.477 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:29.452712+0000", 2024-09-21T13:52:29.477 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:52:29.477 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:29.478 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:29.478 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:52:29.478 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:29.452712+0000", 2024-09-21T13:52:29.478 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:52:29.478 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:29.478 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:29.478 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:52:29.478 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:29.452713+0000", 2024-09-21T13:52:29.478 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.065e-06 2024-09-21T13:52:29.478 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:29.478 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:29.478 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:52:29.478 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:29.452718+0000", 2024-09-21T13:52:29.478 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1790000000000001e-06 2024-09-21T13:52:29.479 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:29.479 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:29.479 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:52:29.479 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:29.452729+0000", 2024-09-21T13:52:29.479 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1428e-05 2024-09-21T13:52:29.479 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:29.479 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:29.479 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:52:29.479 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:29.452737+0000", 2024-09-21T13:52:29.479 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.498e-06 2024-09-21T13:52:29.479 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:29.479 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:29.479 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:52:29.479 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:29.454970+0000", 2024-09-21T13:52:29.479 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0022330150000000001 2024-09-21T13:52:29.480 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:29.480 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:29.480 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:52:29.480 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:29.455084+0000", 2024-09-21T13:52:29.480 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000114739 2024-09-21T13:52:29.480 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:29.480 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:29.480 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3", 2024-09-21T13:52:29.480 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:29.455171+0000", 2024-09-21T13:52:29.480 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.6273999999999994e-05 2024-09-21T13:52:29.480 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:29.480 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:29.480 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:52:29.481 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:29.457272+0000", 2024-09-21T13:52:29.481 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0021013949999999998 2024-09-21T13:52:29.481 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:52:29.481 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:52:29.481 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:52:29.481 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:52:29.481 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2024-09-21T13:52:29.481 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 4 2024-09-21T13:52:29.481 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T13:52:29.505 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T13:52:29.530 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:52:29.604 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:29.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:29.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:52:29.706 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:52:29.722 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-09-21T13:52:29.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:29.785 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:29.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:52:29.886 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:29.914 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:29.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:29.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:30.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:29 smithi049 ceph-mon[40489]: pgmap v158: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 834 B/s rd, 0 B/s wr, 861 op/s; 26565/271587 objects degraded (9.781%); 18 MiB/s, 283 objects/s recovering 2024-09-21T13:52:30.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:29 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2994632407' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:52:30.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:29 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:52:30.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:29 smithi049 ceph-mon[30080]: pgmap v158: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 834 B/s rd, 0 B/s wr, 861 op/s; 26565/271587 objects degraded (9.781%); 18 MiB/s, 283 objects/s recovering 2024-09-21T13:52:30.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:29 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2994632407' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:52:30.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:29 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:52:30.016 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:52:30.041 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:30.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:30.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:52:30.143 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:52:30.170 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:30.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:30.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:52:30.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:29 smithi187 ceph-mon[33032]: pgmap v158: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 834 B/s rd, 0 B/s wr, 861 op/s; 26565/271587 objects degraded (9.781%); 18 MiB/s, 283 objects/s recovering 2024-09-21T13:52:30.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:29 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2994632407' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:52:30.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:29 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:52:30.272 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:52:30.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:30.317 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:30.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:52:30.370 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T13:52:30.418 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:52:30.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:30.487 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:30.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:52:30.583 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:52:30.588 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:30.621 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:30.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:30.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:30.722 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.3 to 1 (10000) 2024-09-21T13:52:30.723 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:52:30.725 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:52:30 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:52:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:52:30.725 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:30 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:52:30.725 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:30 smithi049 ceph-mon[40489]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:52:30.725 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:30 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T13:52:30.725 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:30 smithi049 ceph-mon[40489]: osdmap e140: 8 total, 8 up, 8 in 2024-09-21T13:52:30.725 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:30 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/30260081' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T13:52:30.729 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:30 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:52:30.729 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:30 smithi049 ceph-mon[30080]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:52:30.729 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:30 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T13:52:30.730 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:30 smithi049 ceph-mon[30080]: osdmap e140: 8 total, 8 up, 8 in 2024-09-21T13:52:30.730 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:30 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/30260081' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T13:52:30.770 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:30.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:30.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:52:30.775 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:52:30.873 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:52:30.911 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:30.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:30.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:52:31.012 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:31.042 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:31.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:31.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:31.058 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:30 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:52:31.058 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:30 smithi187 ceph-mon[33032]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:52:31.058 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:30 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T13:52:31.058 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:30 smithi187 ceph-mon[33032]: osdmap e140: 8 total, 8 up, 8 in 2024-09-21T13:52:31.058 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:30 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/30260081' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T13:52:31.144 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:52:31.187 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:31.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:31.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:52:31.288 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:31.323 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:31.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:31.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:31.390 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T13:52:31.390 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-21T13:52:31.390 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-21T13:52:31.390 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-21T13:52:31.390 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T13:52:31.425 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:31.454 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:31.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:31.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:31.556 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:31.582 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:31.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:31.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:31.683 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:31.711 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:31.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:31.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:31.756 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:52:31.756 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:52:31.756 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:52:31.756 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:52:31.756 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:52:31.757 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:52:31.757 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:52:31.757 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:52:31.757 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:52:31.757 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:52:31.757 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:52:31.757 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:52:31.777 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:52:31.777 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:52:31.809 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:52:31.809 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:52:31.809 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:31 smithi049 ceph-mon[40489]: pgmap v160: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.09k op/s; 23216/255858 objects degraded (9.074%); 22 MiB/s, 357 objects/s recovering 2024-09-21T13:52:31.810 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:31 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/30260081' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T13:52:31.810 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:31 smithi049 ceph-mon[40489]: osdmap e141: 8 total, 8 up, 8 in 2024-09-21T13:52:31.810 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:31 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/372618943' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:52:31.810 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:31 smithi049 ceph-mon[30080]: pgmap v160: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.09k op/s; 23216/255858 objects degraded (9.074%); 22 MiB/s, 357 objects/s recovering 2024-09-21T13:52:31.810 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:31 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/30260081' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T13:52:31.810 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:31 smithi049 ceph-mon[30080]: osdmap e141: 8 total, 8 up, 8 in 2024-09-21T13:52:31.810 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:31 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/372618943' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:52:31.812 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:52:31.839 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:31.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:31.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:52:31.941 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:52:31.967 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:31.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:31.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:52:32.070 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:32.099 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:32.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:32.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:32.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:31 smithi187 ceph-mon[33032]: pgmap v160: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.09k op/s; 23216/255858 objects degraded (9.074%); 22 MiB/s, 357 objects/s recovering 2024-09-21T13:52:32.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:31 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/30260081' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T13:52:32.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:31 smithi187 ceph-mon[33032]: osdmap e141: 8 total, 8 up, 8 in 2024-09-21T13:52:32.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:31 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/372618943' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:52:32.201 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:52:32.225 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:32.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:32.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:52:32.259 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T13:52:32.326 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:52:32.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:32.366 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:32.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:52:32.467 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:32.507 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:32.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:32.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:32.528 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T13:52:32.609 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:52:32.621 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:52:32.637 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:32.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:32.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:52:32.739 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:52:32.787 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:32.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:32.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:52:32.889 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:52:32.896 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:32 smithi049 ceph-mon[40489]: pgmap v162: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 874 op/s; 23216/255858 objects degraded (9.074%); 18 MiB/s, 287 objects/s recovering 2024-09-21T13:52:32.896 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:32 smithi049 ceph-mon[30080]: pgmap v162: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 874 op/s; 23216/255858 objects degraded (9.074%); 18 MiB/s, 287 objects/s recovering 2024-09-21T13:52:32.943 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:32.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:32.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:52:32.966 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd reweight 4 1 2024-09-21T13:52:33.044 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:52:33.057 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:32 smithi187 ceph-mon[33032]: pgmap v162: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 874 op/s; 23216/255858 objects degraded (9.074%); 18 MiB/s, 287 objects/s recovering 2024-09-21T13:52:33.072 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:33.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:33.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:52:33.174 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:52:33.203 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T13:52:33.203 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:52:33.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:33.206 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:33.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:52:33.307 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:52:33.335 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:33.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:33.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:52:33.437 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:52:33.463 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:33.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:33.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:52:33.502 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:52:33.516 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:52:33.564 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:52:33.608 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:33.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:33.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:52:33.709 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:52:33.736 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:33.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:33.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:52:33.838 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:52:33.887 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:33.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:33.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:52:33.903 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:33 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 21299/244239 objects degraded (8.721%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T13:52:33.903 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:33 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 21299/244239 objects degraded (8.721%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T13:52:33.989 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:52:34.003 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T13:52:34.018 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:34.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:34.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:52:34.119 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:52:34.155 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:34.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:34.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:52:34.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:33 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 21299/244239 objects degraded (8.721%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T13:52:34.257 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:52:34.287 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:34.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:34.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:52:34.384 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:52:34.389 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:52:34.424 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:34.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:34.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:52:34.526 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:52:34.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:34.577 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:34.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:52:34.678 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:52:34.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:34.718 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:34.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:52:34.819 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:52:34.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:34.865 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:34.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:52:34.937 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T13:52:34.937 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:52:34.967 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:52:35.001 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:35.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:35.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:52:35.102 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:52:35.152 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:35.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:35.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:52:35.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:34 smithi187 ceph-mon[33032]: pgmap v163: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.51k op/s; 21299/244239 objects degraded (8.721%); 32 MiB/s, 515 objects/s recovering 2024-09-21T13:52:35.254 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:52:35.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:34 smithi049 ceph-mon[30080]: pgmap v163: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.51k op/s; 21299/244239 objects degraded (8.721%); 32 MiB/s, 515 objects/s recovering 2024-09-21T13:52:35.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:34 smithi049 ceph-mon[40489]: pgmap v163: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.51k op/s; 21299/244239 objects degraded (8.721%); 32 MiB/s, 515 objects/s recovering 2024-09-21T13:52:35.293 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:35.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:35.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:52:35.395 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:52:35.429 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T13:52:35.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:35.438 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:35.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:52:35.540 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:52:35.566 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:35.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:35.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:52:35.619 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T13:52:35.668 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:52:35.723 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:35.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:35.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:52:35.824 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:52:35.866 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-09-21T13:52:35.868 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.4 to 1 (10000) 2024-09-21T13:52:35.902 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:35.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:35.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:52:35.959 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:52:36.006 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:52:36.046 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:36.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:36.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:52:36.138 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:35 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:52:36.139 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:35 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:52:36.139 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:35 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3425581220' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:52:36.139 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:35 smithi049 ceph-mon[40489]: osdmap e142: 8 total, 8 up, 8 in 2024-09-21T13:52:36.139 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:35 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:52:36.139 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:35 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/548784776' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T13:52:36.139 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:35 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T13:52:36.139 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:35 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:52:36.139 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:35 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:52:36.139 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:35 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3425581220' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:52:36.139 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:35 smithi049 ceph-mon[30080]: osdmap e142: 8 total, 8 up, 8 in 2024-09-21T13:52:36.139 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:35 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:52:36.140 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:35 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/548784776' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T13:52:36.140 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:35 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T13:52:36.148 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:52:36.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:36.179 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:36.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:52:36.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:35 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:52:36.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:35 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:52:36.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:35 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3425581220' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:52:36.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:35 smithi187 ceph-mon[33032]: osdmap e142: 8 total, 8 up, 8 in 2024-09-21T13:52:36.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:35 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:52:36.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:35 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/548784776' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T13:52:36.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:35 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T13:52:36.280 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:52:36.316 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:36.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:36.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:52:36.418 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:52:36.447 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:36.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:36.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:52:36.549 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:52:36.583 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:36.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:36.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:52:36.594 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T13:52:36.594 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:52:36.685 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:52:36.725 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:36.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:36.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:52:36.827 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:52:36.871 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:36.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:36.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:52:36.973 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:52:37.002 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:37.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:37.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:52:37.104 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:37.133 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:37.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:37.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:37.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:36 smithi187 ceph-mon[33032]: pgmap v165: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 851 B/s rd, 0 B/s wr, 849 op/s; 20111/240546 objects degraded (8.361%); 18 MiB/s, 292 objects/s recovering 2024-09-21T13:52:37.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:36 smithi187 ceph-mon[33032]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:52:37.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:36 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T13:52:37.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:36 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T13:52:37.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:36 smithi187 ceph-mon[33032]: osdmap e143: 8 total, 8 up, 8 in 2024-09-21T13:52:37.235 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:52:37.235 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:36 smithi049 ceph-mon[40489]: pgmap v165: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 851 B/s rd, 0 B/s wr, 849 op/s; 20111/240546 objects degraded (8.361%); 18 MiB/s, 292 objects/s recovering 2024-09-21T13:52:37.236 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:36 smithi049 ceph-mon[40489]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:52:37.236 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:36 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T13:52:37.236 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:36 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T13:52:37.236 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:36 smithi049 ceph-mon[40489]: osdmap e143: 8 total, 8 up, 8 in 2024-09-21T13:52:37.236 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:36 smithi049 ceph-mon[30080]: pgmap v165: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 851 B/s rd, 0 B/s wr, 849 op/s; 20111/240546 objects degraded (8.361%); 18 MiB/s, 292 objects/s recovering 2024-09-21T13:52:37.236 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:36 smithi049 ceph-mon[30080]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:52:37.237 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:36 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T13:52:37.237 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:36 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T13:52:37.237 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:36 smithi049 ceph-mon[30080]: osdmap e143: 8 total, 8 up, 8 in 2024-09-21T13:52:37.247 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:52:37.294 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd reweight 5 1 2024-09-21T13:52:37.309 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:37.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:37.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:52:37.398 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T13:52:37.411 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:52:37.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:37.473 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:37.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:52:37.573 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:52:37.600 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:37.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:37.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:52:37.702 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:52:37.743 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:52:37.749 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:37.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:37.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:52:37.851 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:52:37.880 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:37.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:37.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:52:37.982 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:52:38.011 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:38.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:38.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:52:38.113 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:52:38.148 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:38.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:38.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:52:38.250 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:52:38.272 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:38.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:38.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:52:38.374 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:52:38.405 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:38.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:38.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:52:38.507 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:38.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:38.544 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:38.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:38.645 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:52:38.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:38.717 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:38.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:52:38.818 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:38.855 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:38.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:38.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:38.894 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:38 smithi049 ceph-mon[40489]: pgmap v167: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 730 B/s rd, 0 B/s wr, 729 op/s; 20111/240546 objects degraded (8.361%); 16 MiB/s, 251 objects/s recovering 2024-09-21T13:52:38.894 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:38 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:52:38.894 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:38 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3500727708' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:52:38.894 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:38 smithi049 ceph-mon[30080]: pgmap v167: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 730 B/s rd, 0 B/s wr, 729 op/s; 20111/240546 objects degraded (8.361%); 16 MiB/s, 251 objects/s recovering 2024-09-21T13:52:38.894 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:38 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:52:38.895 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:38 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3500727708' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:52:38.913 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T13:52:38.957 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:52:38.965 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T13:52:38.965 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:52:38.994 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:52:38.994 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:52:38.994 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:52:38.994 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:52:38.994 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:52:38.994 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:52:38.995 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:52:38.995 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:52:38.995 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:52:38.995 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:52:39.000 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:52:39.001 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:52:39.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:39.021 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:39.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:52:39.038 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:52:39.039 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:52:39.068 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:52:39.069 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:52:39.110 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:38 smithi187 ceph-mon[33032]: pgmap v167: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 730 B/s rd, 0 B/s wr, 729 op/s; 20111/240546 objects degraded (8.361%); 16 MiB/s, 251 objects/s recovering 2024-09-21T13:52:39.110 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:38 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:52:39.110 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:38 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3500727708' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:52:39.122 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:52:39.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:39.171 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:39.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:52:39.272 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:52:39.329 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:39.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:39.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:52:39.366 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:52:39 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:52:39.432 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:52:39.464 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:39.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:39.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:52:39.566 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:52:39.594 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:39.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:39.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:52:39.696 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:39.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:39.725 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:39.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:39.822 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T13:52:39.826 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:52:39.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:39.888 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:39.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:52:39.953 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:52:39.989 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:52:40.016 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:52:40.043 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:40.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:40.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:52:40.116 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:39 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:52:40.116 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:39 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:52:40.145 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:52:40.147 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:39 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:52:40.191 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:40.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:40.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:52:40.293 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:52:40.336 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:40.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:40.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:52:40.438 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:52:40.468 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:40.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:40.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:52:40.570 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:52:40.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:40.599 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:40.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:52:40.624 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:52:40.700 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:52:40.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:40.750 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:40.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:52:40.851 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:52:40.886 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:52:40 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:52:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:52:40.893 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:40.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:40.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:52:40.995 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:52:41.026 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:41.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:41.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:52:41.128 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:52:41.173 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:41.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:41.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:52:41.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:40 smithi187 ceph-mon[33032]: pgmap v168: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s; 17221/224832 objects degraded (7.659%); 26 MiB/s, 417 objects/s recovering 2024-09-21T13:52:41.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:40 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 17221/224832 objects degraded (7.659%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T13:52:41.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:40 smithi187 ceph-mon[33032]: osdmap e144: 8 total, 8 up, 8 in 2024-09-21T13:52:41.213 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:40 smithi049 ceph-mon[40489]: pgmap v168: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s; 17221/224832 objects degraded (7.659%); 26 MiB/s, 417 objects/s recovering 2024-09-21T13:52:41.213 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:40 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 17221/224832 objects degraded (7.659%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T13:52:41.214 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:40 smithi049 ceph-mon[40489]: osdmap e144: 8 total, 8 up, 8 in 2024-09-21T13:52:41.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:40 smithi049 ceph-mon[30080]: pgmap v168: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s; 17221/224832 objects degraded (7.659%); 26 MiB/s, 417 objects/s recovering 2024-09-21T13:52:41.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:40 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 17221/224832 objects degraded (7.659%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T13:52:41.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:40 smithi049 ceph-mon[30080]: osdmap e144: 8 total, 8 up, 8 in 2024-09-21T13:52:41.275 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:52:41.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:41.297 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:41.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:52:41.398 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:52:41.428 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:41.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:41.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:52:41.530 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:52:41.544 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T13:52:41.544 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-21T13:52:41.544 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:41.544 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:201055 7.5 7:a8ab2ce6:::benchmark_data_smithi049_76798_object64635:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-09-21T13:52:41.545 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:41.537656+0000", 2024-09-21T13:52:41.545 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0035709050000000001, 2024-09-21T13:52:41.545 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0036014599999999999, 2024-09-21T13:52:41.545 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:52:41.545 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:52:41.545 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-21T13:52:41.545 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:52:41.545 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:52:41.545 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:52:41.545 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 201055 2024-09-21T13:52:41.545 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:41.545 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:52:41.545 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:41.546 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:52:41.546 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.537656+0000", 2024-09-21T13:52:41.546 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:52:41.546 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:41.546 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:41.546 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:52:41.546 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.537656+0000", 2024-09-21T13:52:41.546 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:52:41.546 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:41.546 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:41.546 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:52:41.546 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.537658+0000", 2024-09-21T13:52:41.546 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5689999999999999e-06 2024-09-21T13:52:41.547 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:41.547 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:41.547 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:52:41.547 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.537664+0000", 2024-09-21T13:52:41.547 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3040000000000001e-06 2024-09-21T13:52:41.547 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:41.547 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:41.547 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:52:41.547 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.537677+0000", 2024-09-21T13:52:41.547 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2734999999999999e-05 2024-09-21T13:52:41.547 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:41.547 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:41.547 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:52:41.547 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.537685+0000", 2024-09-21T13:52:41.548 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.9020000000000005e-06 2024-09-21T13:52:41.548 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:41.548 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:41.548 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:52:41.548 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.537825+0000", 2024-09-21T13:52:41.548 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013975899999999999 2024-09-21T13:52:41.548 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:41.548 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:41.548 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:52:41.548 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.537975+0000", 2024-09-21T13:52:41.548 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000149863 2024-09-21T13:52:41.548 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:41.548 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:41.548 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2", 2024-09-21T13:52:41.549 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.538136+0000", 2024-09-21T13:52:41.549 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00016056800000000001 2024-09-21T13:52:41.549 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:52:41.549 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:52:41.549 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:52:41.549 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:41.549 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:41.549 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.14817.0:201054 7.c e145/83 7:3e8dd2e4:::benchmark_data_smithi049_76798_object64634:head v 145'12581, mlcod=145'12579)", 2024-09-21T13:52:41.549 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:41.540387+0000", 2024-09-21T13:52:41.549 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.00083973499999999998, 2024-09-21T13:52:41.549 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00095155200000000004, 2024-09-21T13:52:41.549 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:52:41.549 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:52:41.549 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T13:52:41.550 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:52:41.550 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:41.550 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:52:41.550 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.540387+0000", 2024-09-21T13:52:41.550 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:52:41.550 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:41.550 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:41.550 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:52:41.550 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.540387+0000", 2024-09-21T13:52:41.550 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:52:41.550 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:41.550 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:41.550 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:52:41.550 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.540388+0000", 2024-09-21T13:52:41.551 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.22e-06 2024-09-21T13:52:41.551 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:41.551 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:41.551 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:52:41.551 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.540396+0000", 2024-09-21T13:52:41.551 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7729999999999999e-06 2024-09-21T13:52:41.551 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:41.551 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:41.551 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:52:41.551 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.540416+0000", 2024-09-21T13:52:41.551 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9993999999999999e-05 2024-09-21T13:52:41.551 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:41.551 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:41.551 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:52:41.551 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.540425+0000", 2024-09-21T13:52:41.552 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7220000000000005e-06 2024-09-21T13:52:41.552 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:41.552 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:41.552 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:52:41.552 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.540483+0000", 2024-09-21T13:52:41.552 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7827e-05 2024-09-21T13:52:41.552 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:52:41.552 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:52:41.552 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:52:41.552 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.540502+0000", 2024-09-21T13:52:41.552 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9809999999999998e-05 2024-09-21T13:52:41.552 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:52:41.552 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:52:41.552 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:52:41.553 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:52:41.553 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2024-09-21T13:52:41.553 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 2 2024-09-21T13:52:41.553 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T13:52:41.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:41.566 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:41.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:52:41.667 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:52:41.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:41.707 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:41.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:52:41.808 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:52:41.838 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:41.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:41.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:52:41.917 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T13:52:41.939 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:52:42.025 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:42.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:42.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:52:42.127 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:52:42.166 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:42.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:42.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:52:42.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:41 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:52:42.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:41 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:52:42.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:41 smithi187 ceph-mon[33032]: osdmap e145: 8 total, 8 up, 8 in 2024-09-21T13:52:42.210 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:41 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:52:42.211 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:41 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:52:42.211 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:41 smithi049 ceph-mon[40489]: osdmap e145: 8 total, 8 up, 8 in 2024-09-21T13:52:42.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:41 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:52:42.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:41 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:52:42.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:41 smithi049 ceph-mon[30080]: osdmap e145: 8 total, 8 up, 8 in 2024-09-21T13:52:42.243 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T13:52:42.268 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:52:42.299 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:42.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:42.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:52:42.401 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:42.422 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:42.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:42.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:42.524 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:52:42.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:42.558 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:42.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:52:42.561 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:52:42.660 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:52:42.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:42.682 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:42.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:52:42.782 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:52:42.804 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:42.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:42.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:52:42.905 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:52:42.926 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:42.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:42.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:52:42.929 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-09-21T13:52:42.937 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.5 to 1 (10000) 2024-09-21T13:52:42.949 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:42 smithi049 ceph-mon[30080]: pgmap v171: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 873 op/s; 17221/224832 objects degraded (7.659%); 16 MiB/s, 262 objects/s recovering 2024-09-21T13:52:42.949 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:42 smithi049 ceph-mon[30080]: osdmap e146: 8 total, 8 up, 8 in 2024-09-21T13:52:42.949 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:42 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2006497893' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T13:52:42.949 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:42 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T13:52:42.949 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:42 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/4293934870' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:52:42.949 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:42 smithi049 ceph-mon[40489]: pgmap v171: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 873 op/s; 17221/224832 objects degraded (7.659%); 16 MiB/s, 262 objects/s recovering 2024-09-21T13:52:42.950 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:42 smithi049 ceph-mon[40489]: osdmap e146: 8 total, 8 up, 8 in 2024-09-21T13:52:42.950 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:42 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2006497893' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T13:52:42.950 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:42 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T13:52:42.950 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:42 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/4293934870' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:52:43.028 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:43.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:43.064 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:43.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:43.085 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:52:43.166 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:52:43.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:42 smithi187 ceph-mon[33032]: pgmap v171: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 873 op/s; 17221/224832 objects degraded (7.659%); 16 MiB/s, 262 objects/s recovering 2024-09-21T13:52:43.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:42 smithi187 ceph-mon[33032]: osdmap e146: 8 total, 8 up, 8 in 2024-09-21T13:52:43.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:42 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2006497893' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T13:52:43.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:42 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T13:52:43.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:42 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/4293934870' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:52:43.195 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:43.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:43.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:52:43.297 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:52:43.318 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:43.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:43.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:52:43.420 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:52:43.453 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:43.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:43.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:52:43.555 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:52:43.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:43.578 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:43.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:52:43.679 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:52:43.700 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:43.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:43.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:52:43.802 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:52:43.832 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:43.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:43.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:52:43.880 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T13:52:43.880 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-21T13:52:43.880 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-21T13:52:43.880 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-21T13:52:43.880 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T13:52:43.934 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:52:43.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:43.968 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:43.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:52:44.069 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:52:44.094 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:44.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:44.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:52:44.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:43 smithi187 ceph-mon[33032]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:52:44.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:43 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T13:52:44.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:43 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/4293934870' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T13:52:44.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:43 smithi187 ceph-mon[33032]: osdmap e147: 8 total, 8 up, 8 in 2024-09-21T13:52:44.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:43 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:52:44.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:43 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:52:44.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:43 smithi187 ceph-mon[33032]: 3.19 scrub starts 2024-09-21T13:52:44.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:43 smithi187 ceph-mon[33032]: 3.19 scrub ok 2024-09-21T13:52:44.196 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:44.219 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:44.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:44.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:44.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:43 smithi049 ceph-mon[40489]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:52:44.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:43 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T13:52:44.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:43 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/4293934870' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T13:52:44.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:43 smithi049 ceph-mon[40489]: osdmap e147: 8 total, 8 up, 8 in 2024-09-21T13:52:44.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:43 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:52:44.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:43 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:52:44.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:43 smithi049 ceph-mon[40489]: 3.19 scrub starts 2024-09-21T13:52:44.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:43 smithi049 ceph-mon[40489]: 3.19 scrub ok 2024-09-21T13:52:44.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:43 smithi049 ceph-mon[30080]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:52:44.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:43 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T13:52:44.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:43 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/4293934870' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T13:52:44.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:43 smithi049 ceph-mon[30080]: osdmap e147: 8 total, 8 up, 8 in 2024-09-21T13:52:44.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:43 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:52:44.257 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:43 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:52:44.257 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:43 smithi049 ceph-mon[30080]: 3.19 scrub starts 2024-09-21T13:52:44.257 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:43 smithi049 ceph-mon[30080]: 3.19 scrub ok 2024-09-21T13:52:44.321 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:52:44.351 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:44.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:44.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:52:44.453 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:52:44.491 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:44.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:44.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:52:44.593 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:52:44.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:44.617 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:44.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:52:44.718 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:52:44.741 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:44.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:44.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:52:44.843 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:52:44.867 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:44.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:44.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:52:44.968 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:52:45.010 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:45.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:45.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:52:45.112 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:52:45.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:45.150 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:45.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:52:45.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:44 smithi187 ceph-mon[33032]: 6.19 scrub starts 2024-09-21T13:52:45.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:44 smithi187 ceph-mon[33032]: 6.19 scrub ok 2024-09-21T13:52:45.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:44 smithi187 ceph-mon[33032]: pgmap v174: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 peering, 2 active+undersized+degraded+remapped+backfilling, 142 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 722 op/s; 15628/216177 objects degraded (7.229%); 21 MiB/s, 338 objects/s recovering 2024-09-21T13:52:45.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:44 smithi187 ceph-mon[33032]: osdmap e148: 8 total, 8 up, 8 in 2024-09-21T13:52:45.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:44 smithi187 ceph-mon[33032]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T13:52:45.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:44 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/195189178' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:52:45.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:44 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 15628/216177 objects degraded (7.229%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T13:52:45.251 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:52:45.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:44 smithi049 ceph-mon[40489]: 6.19 scrub starts 2024-09-21T13:52:45.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:44 smithi049 ceph-mon[40489]: 6.19 scrub ok 2024-09-21T13:52:45.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:44 smithi049 ceph-mon[40489]: pgmap v174: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 peering, 2 active+undersized+degraded+remapped+backfilling, 142 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 722 op/s; 15628/216177 objects degraded (7.229%); 21 MiB/s, 338 objects/s recovering 2024-09-21T13:52:45.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:44 smithi049 ceph-mon[40489]: osdmap e148: 8 total, 8 up, 8 in 2024-09-21T13:52:45.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:44 smithi049 ceph-mon[40489]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T13:52:45.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:44 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/195189178' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:52:45.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:44 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 15628/216177 objects degraded (7.229%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T13:52:45.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:44 smithi049 ceph-mon[30080]: 6.19 scrub starts 2024-09-21T13:52:45.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:44 smithi049 ceph-mon[30080]: 6.19 scrub ok 2024-09-21T13:52:45.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:44 smithi049 ceph-mon[30080]: pgmap v174: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 2 peering, 2 active+undersized+degraded+remapped+backfilling, 142 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 722 op/s; 15628/216177 objects degraded (7.229%); 21 MiB/s, 338 objects/s recovering 2024-09-21T13:52:45.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:44 smithi049 ceph-mon[30080]: osdmap e148: 8 total, 8 up, 8 in 2024-09-21T13:52:45.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:44 smithi049 ceph-mon[30080]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T13:52:45.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:44 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/195189178' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:52:45.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:44 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 15628/216177 objects degraded (7.229%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T13:52:45.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:45.277 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:45.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:52:45.378 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:52:45.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:45.425 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:45.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:52:45.526 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:52:45.547 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:45.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:45.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:52:45.649 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:52:45.675 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:45.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:45.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:52:45.776 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:52:45.802 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:45.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:45.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:52:45.904 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:52:45.933 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:45.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:45.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:52:46.003 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T13:52:46.035 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:52:46.058 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:46.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:46.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:52:46.160 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:52:46.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:45 smithi187 ceph-mon[33032]: 3.a scrub starts 2024-09-21T13:52:46.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:45 smithi187 ceph-mon[33032]: 3.a scrub ok 2024-09-21T13:52:46.220 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:46.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:46.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:52:46.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:45 smithi049 ceph-mon[40489]: 3.a scrub starts 2024-09-21T13:52:46.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:45 smithi049 ceph-mon[40489]: 3.a scrub ok 2024-09-21T13:52:46.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:45 smithi049 ceph-mon[30080]: 3.a scrub starts 2024-09-21T13:52:46.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:45 smithi049 ceph-mon[30080]: 3.a scrub ok 2024-09-21T13:52:46.322 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:52:46.349 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:46.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:46.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:52:46.451 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:52:46.500 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:46.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:46.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:52:46.602 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:52:46.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:46.697 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:46.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:52:46.798 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:46.828 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:46.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:46.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:46.930 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:52:46.952 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:46.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:46.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:52:47.053 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:52:47.097 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:47.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:47.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:52:47.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:46 smithi187 ceph-mon[33032]: 4.d scrub starts 2024-09-21T13:52:47.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:46 smithi187 ceph-mon[33032]: 4.d scrub ok 2024-09-21T13:52:47.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:46 smithi187 ceph-mon[33032]: pgmap v176: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 peering, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 780 op/s; 13611/213546 objects degraded (6.374%); 37 MiB/s, 587 objects/s recovering 2024-09-21T13:52:47.199 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:52:47.227 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:47.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:47.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:52:47.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:46 smithi049 ceph-mon[40489]: 4.d scrub starts 2024-09-21T13:52:47.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:46 smithi049 ceph-mon[40489]: 4.d scrub ok 2024-09-21T13:52:47.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:46 smithi049 ceph-mon[40489]: pgmap v176: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 peering, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 780 op/s; 13611/213546 objects degraded (6.374%); 37 MiB/s, 587 objects/s recovering 2024-09-21T13:52:47.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:46 smithi049 ceph-mon[30080]: 4.d scrub starts 2024-09-21T13:52:47.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:46 smithi049 ceph-mon[30080]: 4.d scrub ok 2024-09-21T13:52:47.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:46 smithi049 ceph-mon[30080]: pgmap v176: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 peering, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 780 op/s; 13611/213546 objects degraded (6.374%); 37 MiB/s, 587 objects/s recovering 2024-09-21T13:52:47.329 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:47.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:47.351 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:47.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:47.452 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:52:47.486 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:47.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:47.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:52:47.544 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:52:47.544 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:52:47.544 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:52:47.544 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:52:47.544 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:52:47.544 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:52:47.545 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:52:47.545 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:52:47.545 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:52:47.545 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:52:47.545 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:52:47.545 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:52:47.545 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:52:47.545 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:52:47.545 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:52:47.545 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:52:47.588 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:52:47.627 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:47.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:47.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:52:47.728 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:52:47.755 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:47.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:47.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:52:47.857 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:52:47.884 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:47.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:47.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:52:47.985 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:52:48.026 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:48.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:48.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:52:48.127 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:52:48.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:48.173 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:48.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:52:48.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:47 smithi049 ceph-mon[40489]: 7.7 scrub starts 2024-09-21T13:52:48.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:47 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:52:48.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:47 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:52:48.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:47 smithi049 ceph-mon[30080]: 7.7 scrub starts 2024-09-21T13:52:48.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:47 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:52:48.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:47 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:52:48.274 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:52:48.319 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:48.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:48.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:52:48.420 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:52:48.441 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:48.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:48.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:52:48.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:47 smithi187 ceph-mon[33032]: 7.7 scrub starts 2024-09-21T13:52:48.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:47 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:52:48.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:47 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:52:48.543 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:52:48.588 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:48.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:48.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:52:48.690 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:52:48.712 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:48.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:48.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:52:48.814 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:52:48.846 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:48.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:48.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:52:48.948 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:52:48.975 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:48.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:48.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:52:49.077 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:52:49.103 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:49.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:49.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:52:49.205 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:49.213 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:48 smithi187 ceph-mon[33032]: pgmap v177: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 peering, 3 active+undersized+degraded+remapped+backfilling, 143 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 661 op/s; 13611/212940 objects degraded (6.392%); 30 MiB/s, 472 objects/s recovering 2024-09-21T13:52:49.213 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:48 smithi187 ceph-mon[33032]: osdmap e149: 8 total, 8 up, 8 in 2024-09-21T13:52:49.213 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:48 smithi187 ceph-mon[33032]: 5.11 scrub starts 2024-09-21T13:52:49.213 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:48 smithi187 ceph-mon[33032]: 5.11 scrub ok 2024-09-21T13:52:49.213 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:52:49 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:52:49.227 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:49.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:49.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:49.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:48 smithi049 ceph-mon[30080]: pgmap v177: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 peering, 3 active+undersized+degraded+remapped+backfilling, 143 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 661 op/s; 13611/212940 objects degraded (6.392%); 30 MiB/s, 472 objects/s recovering 2024-09-21T13:52:49.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:49 smithi049 ceph-mon[30080]: osdmap e149: 8 total, 8 up, 8 in 2024-09-21T13:52:49.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:49 smithi049 ceph-mon[30080]: 5.11 scrub starts 2024-09-21T13:52:49.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:49 smithi049 ceph-mon[30080]: 5.11 scrub ok 2024-09-21T13:52:49.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:48 smithi049 ceph-mon[40489]: pgmap v177: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 peering, 3 active+undersized+degraded+remapped+backfilling, 143 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 661 op/s; 13611/212940 objects degraded (6.392%); 30 MiB/s, 472 objects/s recovering 2024-09-21T13:52:49.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:49 smithi049 ceph-mon[40489]: osdmap e149: 8 total, 8 up, 8 in 2024-09-21T13:52:49.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:49 smithi049 ceph-mon[40489]: 5.11 scrub starts 2024-09-21T13:52:49.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:49 smithi049 ceph-mon[40489]: 5.11 scrub ok 2024-09-21T13:52:49.329 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:52:49.365 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:49.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:49.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:52:49.466 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:52:49.509 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:49.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:49.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:52:49.611 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:49.639 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:49.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:49.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:49.741 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:52:49.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:49.779 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:49.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:52:49.880 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:52:49.921 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:49.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:49.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:52:50.023 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:52:50.065 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:50.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:50.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:52:50.167 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:52:50.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:50.204 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:50.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:52:50.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:49 smithi049 ceph-mon[40489]: 3.9 scrub starts 2024-09-21T13:52:50.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:49 smithi049 ceph-mon[40489]: 3.9 scrub ok 2024-09-21T13:52:50.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:49 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:52:50.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:49 smithi049 ceph-mon[40489]: 7.7 scrub ok 2024-09-21T13:52:50.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:49 smithi049 ceph-mon[40489]: 7.d scrub starts 2024-09-21T13:52:50.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:49 smithi049 ceph-mon[40489]: 3.1d scrub starts 2024-09-21T13:52:50.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:49 smithi049 ceph-mon[40489]: 3.1d scrub ok 2024-09-21T13:52:50.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:49 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 13611/212940 objects degraded (6.392%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T13:52:50.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:49 smithi049 ceph-mon[40489]: osdmap e150: 8 total, 8 up, 8 in 2024-09-21T13:52:50.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:49 smithi049 ceph-mon[30080]: 3.9 scrub starts 2024-09-21T13:52:50.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:49 smithi049 ceph-mon[30080]: 3.9 scrub ok 2024-09-21T13:52:50.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:49 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:52:50.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:49 smithi049 ceph-mon[30080]: 7.7 scrub ok 2024-09-21T13:52:50.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:49 smithi049 ceph-mon[30080]: 7.d scrub starts 2024-09-21T13:52:50.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:49 smithi049 ceph-mon[30080]: 3.1d scrub starts 2024-09-21T13:52:50.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:49 smithi049 ceph-mon[30080]: 3.1d scrub ok 2024-09-21T13:52:50.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:49 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 13611/212940 objects degraded (6.392%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T13:52:50.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:49 smithi049 ceph-mon[30080]: osdmap e150: 8 total, 8 up, 8 in 2024-09-21T13:52:50.305 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:52:50.333 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:50.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:50.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:52:50.435 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:52:50.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:49 smithi187 ceph-mon[33032]: 3.9 scrub starts 2024-09-21T13:52:50.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:49 smithi187 ceph-mon[33032]: 3.9 scrub ok 2024-09-21T13:52:50.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:49 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:52:50.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:49 smithi187 ceph-mon[33032]: 7.7 scrub ok 2024-09-21T13:52:50.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:49 smithi187 ceph-mon[33032]: 7.d scrub starts 2024-09-21T13:52:50.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:49 smithi187 ceph-mon[33032]: 3.1d scrub starts 2024-09-21T13:52:50.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:49 smithi187 ceph-mon[33032]: 3.1d scrub ok 2024-09-21T13:52:50.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:49 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 13611/212940 objects degraded (6.392%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T13:52:50.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:49 smithi187 ceph-mon[33032]: osdmap e150: 8 total, 8 up, 8 in 2024-09-21T13:52:50.464 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:50.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:50.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:52:50.566 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:52:50.606 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:50.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:50.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:52:50.708 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd reweight 6 1 2024-09-21T13:52:50.709 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:52:50.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:50.763 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:50.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:52:50.845 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:52:50 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:52:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:52:50.865 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:52:50.891 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:50.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:50.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:52:50.992 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:52:51.019 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:51.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:51.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:52:51.122 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:52:51.143 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:51.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:51.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:52:51.158 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:50 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:52:51.158 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:50 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:52:51.158 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:50 smithi049 ceph-mon[40489]: pgmap v179: 148 pgs: 1 active+clean+scrubbing, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 143 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.04k op/s; 11245/202947 objects degraded (5.541%); 38 MiB/s, 605 objects/s recovering 2024-09-21T13:52:51.158 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:50 smithi049 ceph-mon[40489]: 7.c scrub starts 2024-09-21T13:52:51.158 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:50 smithi049 ceph-mon[40489]: 7.d scrub ok 2024-09-21T13:52:51.162 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:50 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:52:51.162 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:50 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:52:51.162 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:50 smithi049 ceph-mon[30080]: pgmap v179: 148 pgs: 1 active+clean+scrubbing, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 143 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.04k op/s; 11245/202947 objects degraded (5.541%); 38 MiB/s, 605 objects/s recovering 2024-09-21T13:52:51.162 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:50 smithi049 ceph-mon[30080]: 7.c scrub starts 2024-09-21T13:52:51.162 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:50 smithi049 ceph-mon[30080]: 7.d scrub ok 2024-09-21T13:52:51.244 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:52:51.305 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:52:51.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:51.310 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:51.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:52:51.347 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:52:51.411 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:52:51.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:50 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:52:51.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:50 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:52:51.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:50 smithi187 ceph-mon[33032]: pgmap v179: 148 pgs: 1 active+clean+scrubbing, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 143 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.04k op/s; 11245/202947 objects degraded (5.541%); 38 MiB/s, 605 objects/s recovering 2024-09-21T13:52:51.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:50 smithi187 ceph-mon[33032]: 7.c scrub starts 2024-09-21T13:52:51.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:50 smithi187 ceph-mon[33032]: 7.d scrub ok 2024-09-21T13:52:51.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:51.475 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:51.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:52:51.549 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T13:52:51.576 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:52:51.620 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:51.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:51.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:52:51.721 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:52:51.748 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:51.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:51.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:52:51.850 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:52:51.891 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:51.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:51.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:52:51.993 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:52:52.033 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:52.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:52.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:52:52.074 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:52:52.135 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:52:52.176 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:52.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:52.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:52:52.177 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:51 smithi049 ceph-mon[40489]: 7.c scrub ok 2024-09-21T13:52:52.177 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:51 smithi049 ceph-mon[40489]: 5.3 scrub starts 2024-09-21T13:52:52.178 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:51 smithi049 ceph-mon[40489]: 2.2 scrub starts 2024-09-21T13:52:52.178 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:51 smithi049 ceph-mon[40489]: 5.3 scrub ok 2024-09-21T13:52:52.178 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:51 smithi049 ceph-mon[40489]: 2.2 scrub ok 2024-09-21T13:52:52.178 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:51 smithi049 ceph-mon[40489]: 6.9 scrub starts 2024-09-21T13:52:52.178 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:51 smithi049 ceph-mon[40489]: 6.9 scrub ok 2024-09-21T13:52:52.178 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:51 smithi049 ceph-mon[30080]: 7.c scrub ok 2024-09-21T13:52:52.178 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:51 smithi049 ceph-mon[30080]: 5.3 scrub starts 2024-09-21T13:52:52.178 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:51 smithi049 ceph-mon[30080]: 2.2 scrub starts 2024-09-21T13:52:52.178 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:51 smithi049 ceph-mon[30080]: 5.3 scrub ok 2024-09-21T13:52:52.178 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:51 smithi049 ceph-mon[30080]: 2.2 scrub ok 2024-09-21T13:52:52.178 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:51 smithi049 ceph-mon[30080]: 6.9 scrub starts 2024-09-21T13:52:52.178 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:51 smithi049 ceph-mon[30080]: 6.9 scrub ok 2024-09-21T13:52:52.278 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:52:52.320 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:52.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:52.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:52:52.362 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:51 smithi187 ceph-mon[33032]: 7.c scrub ok 2024-09-21T13:52:52.362 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:51 smithi187 ceph-mon[33032]: 5.3 scrub starts 2024-09-21T13:52:52.362 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:51 smithi187 ceph-mon[33032]: 2.2 scrub starts 2024-09-21T13:52:52.363 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:51 smithi187 ceph-mon[33032]: 5.3 scrub ok 2024-09-21T13:52:52.363 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:51 smithi187 ceph-mon[33032]: 2.2 scrub ok 2024-09-21T13:52:52.363 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:51 smithi187 ceph-mon[33032]: 6.9 scrub starts 2024-09-21T13:52:52.363 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:51 smithi187 ceph-mon[33032]: 6.9 scrub ok 2024-09-21T13:52:52.422 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:52:52.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:52.467 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:52.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:52:52.568 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:52:52.630 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:52.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:52.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:52:52.732 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:52:52.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:52.760 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:52.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:52:52.861 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:52:52.907 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:52.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:52.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:52:53.009 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:52:53.056 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:53.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:53.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:52:53.157 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:52:53.206 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:53.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:53.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:52:53.237 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T13:52:53.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:52 smithi049 ceph-mon[30080]: pgmap v181: 148 pgs: 1 active+clean+scrubbing, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 143 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 655 B/s rd, 0 B/s wr, 453 op/s; 11245/202947 objects degraded (5.541%); 14 MiB/s, 216 objects/s recovering 2024-09-21T13:52:53.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:52 smithi049 ceph-mon[40489]: pgmap v181: 148 pgs: 1 active+clean+scrubbing, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 143 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 655 B/s rd, 0 B/s wr, 453 op/s; 11245/202947 objects degraded (5.541%); 14 MiB/s, 216 objects/s recovering 2024-09-21T13:52:53.273 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:52 smithi187 ceph-mon[33032]: pgmap v181: 148 pgs: 1 active+clean+scrubbing, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 143 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 655 B/s rd, 0 B/s wr, 453 op/s; 11245/202947 objects degraded (5.541%); 14 MiB/s, 216 objects/s recovering 2024-09-21T13:52:53.284 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T13:52:53.285 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:52:53.308 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:52:53.352 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:53.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:53.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:52:53.454 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:52:53.480 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:53.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:53.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:52:53.582 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:52:53.607 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:53.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:53.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:52:53.709 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:52:53.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:53.749 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:53.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:52:53.850 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:52:53.895 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:53.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:53.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:52:53.997 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:54.009 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-09-21T13:52:54.015 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.6 to 1 (10000) 2024-09-21T13:52:54.021 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:54.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:54.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:54.123 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:52:54.153 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:54.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:54.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:52:54.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:54 smithi049 ceph-mon[30080]: 3.1a scrub starts 2024-09-21T13:52:54.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:54 smithi049 ceph-mon[30080]: 3.1a scrub ok 2024-09-21T13:52:54.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:54 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3422421428' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:52:54.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:54 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:52:54.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:54 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1166267482' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T13:52:54.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:54 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T13:52:54.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:54 smithi049 ceph-mon[40489]: 3.1a scrub starts 2024-09-21T13:52:54.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:54 smithi049 ceph-mon[40489]: 3.1a scrub ok 2024-09-21T13:52:54.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:54 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3422421428' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:52:54.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:54 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:52:54.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:54 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1166267482' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T13:52:54.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:54 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T13:52:54.256 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:52:54.282 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:54.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:54.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:52:54.388 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:52:54.404 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:54 smithi187 ceph-mon[33032]: 3.1a scrub starts 2024-09-21T13:52:54.404 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:54 smithi187 ceph-mon[33032]: 3.1a scrub ok 2024-09-21T13:52:54.404 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:54 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3422421428' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:52:54.404 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:54 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:52:54.404 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:54 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1166267482' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T13:52:54.404 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:54 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T13:52:54.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:54.435 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:54.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:52:54.536 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:52:54.575 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:54.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:54.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:52:54.676 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:54.702 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:54.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:54.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:54.804 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:52:54.829 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:54.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:54.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:52:54.910 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T13:52:54.931 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:52:55.009 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:55.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:55.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:52:55.110 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:52:55.134 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:55.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:55.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:52:55.193 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:55 smithi187 ceph-mon[33032]: pgmap v182: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 144 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 856 op/s; 10607/193011 objects degraded (5.496%); 17 MiB/s, 268 objects/s recovering 2024-09-21T13:52:55.193 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:55 smithi187 ceph-mon[33032]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T13:52:55.193 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:55 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T13:52:55.193 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:55 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T13:52:55.193 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:55 smithi187 ceph-mon[33032]: osdmap e151: 8 total, 8 up, 8 in 2024-09-21T13:52:55.193 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:55 smithi187 ceph-mon[33032]: 6.a scrub starts 2024-09-21T13:52:55.193 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:55 smithi187 ceph-mon[33032]: 6.a scrub ok 2024-09-21T13:52:55.193 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:55 smithi187 ceph-mon[33032]: 6.1c scrub starts 2024-09-21T13:52:55.193 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:55 smithi187 ceph-mon[33032]: 6.1c scrub ok 2024-09-21T13:52:55.193 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:55 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 10607/193011 objects degraded (5.496%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T13:52:55.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:55 smithi187 ceph-mon[33032]: osdmap e152: 8 total, 8 up, 8 in 2024-09-21T13:52:55.235 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:52:55.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:55 smithi049 ceph-mon[30080]: pgmap v182: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 144 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 856 op/s; 10607/193011 objects degraded (5.496%); 17 MiB/s, 268 objects/s recovering 2024-09-21T13:52:55.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:55 smithi049 ceph-mon[30080]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T13:52:55.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:55 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T13:52:55.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:55 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T13:52:55.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:55 smithi049 ceph-mon[30080]: osdmap e151: 8 total, 8 up, 8 in 2024-09-21T13:52:55.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:55 smithi049 ceph-mon[30080]: 6.a scrub starts 2024-09-21T13:52:55.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:55 smithi049 ceph-mon[30080]: 6.a scrub ok 2024-09-21T13:52:55.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:55 smithi049 ceph-mon[30080]: 6.1c scrub starts 2024-09-21T13:52:55.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:55 smithi049 ceph-mon[30080]: 6.1c scrub ok 2024-09-21T13:52:55.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:55 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 10607/193011 objects degraded (5.496%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T13:52:55.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:55 smithi049 ceph-mon[30080]: osdmap e152: 8 total, 8 up, 8 in 2024-09-21T13:52:55.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:55 smithi049 ceph-mon[40489]: pgmap v182: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 144 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 856 op/s; 10607/193011 objects degraded (5.496%); 17 MiB/s, 268 objects/s recovering 2024-09-21T13:52:55.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:55 smithi049 ceph-mon[40489]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T13:52:55.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:55 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T13:52:55.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:55 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T13:52:55.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:55 smithi049 ceph-mon[40489]: osdmap e151: 8 total, 8 up, 8 in 2024-09-21T13:52:55.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:55 smithi049 ceph-mon[40489]: 6.a scrub starts 2024-09-21T13:52:55.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:55 smithi049 ceph-mon[40489]: 6.a scrub ok 2024-09-21T13:52:55.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:55 smithi049 ceph-mon[40489]: 6.1c scrub starts 2024-09-21T13:52:55.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:55 smithi049 ceph-mon[40489]: 6.1c scrub ok 2024-09-21T13:52:55.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:55 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 10607/193011 objects degraded (5.496%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T13:52:55.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:55 smithi049 ceph-mon[40489]: osdmap e152: 8 total, 8 up, 8 in 2024-09-21T13:52:55.286 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:55.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:55.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:52:55.388 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:52:55.413 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:55.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:55.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:52:55.515 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:52:55.551 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:55.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:55.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:52:55.653 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:55.685 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:55.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:55.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:55.787 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:52:55.827 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:55.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:55.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:52:55.929 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:52:55.968 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:55.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:55.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:52:56.070 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:52:56.096 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:56.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:56.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:52:56.123 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:52:56.198 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:52:56.241 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:56.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:56.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:52:56.343 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:52:56.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:56.387 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:56.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:52:56.488 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:56.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:56 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:52:56.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:56 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:52:56.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:56 smithi049 ceph-mon[40489]: 6.3 scrub starts 2024-09-21T13:52:56.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:56 smithi049 ceph-mon[40489]: 6.3 scrub ok 2024-09-21T13:52:56.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:56 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:52:56.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:56 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:52:56.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:56 smithi049 ceph-mon[30080]: 6.3 scrub starts 2024-09-21T13:52:56.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:56 smithi049 ceph-mon[30080]: 6.3 scrub ok 2024-09-21T13:52:56.514 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:56.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:56.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:56.616 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:52:56.624 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd reweight 7 1 2024-09-21T13:52:56.641 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:56.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:56.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:52:56.742 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:52:56.796 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:56.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:56.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:52:56.816 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:56 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:52:56.816 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:56 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:52:56.816 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:56 smithi187 ceph-mon[33032]: 6.3 scrub starts 2024-09-21T13:52:56.816 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:56 smithi187 ceph-mon[33032]: 6.3 scrub ok 2024-09-21T13:52:56.898 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:52:56.922 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:56.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:56.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:52:57.023 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:52:57.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:57.067 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:57.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:52:57.168 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:52:57.191 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:57.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:57.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:52:57.293 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:57.318 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:57.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:57.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:57.420 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:57.445 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:57.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:57.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:57.547 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:57.570 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:57.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:57.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:57.657 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:57 smithi049 ceph-mon[30080]: pgmap v185: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 144 active+clean; 3.8 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 739 op/s; 9373/189639 objects degraded (4.943%); 13 MiB/s, 200 objects/s recovering 2024-09-21T13:52:57.657 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:57 smithi049 ceph-mon[30080]: 4.1e scrub starts 2024-09-21T13:52:57.657 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:57 smithi049 ceph-mon[30080]: 4.1e scrub ok 2024-09-21T13:52:57.657 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:57 smithi049 ceph-mon[40489]: pgmap v185: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 144 active+clean; 3.8 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 739 op/s; 9373/189639 objects degraded (4.943%); 13 MiB/s, 200 objects/s recovering 2024-09-21T13:52:57.657 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:57 smithi049 ceph-mon[40489]: 4.1e scrub starts 2024-09-21T13:52:57.657 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:57 smithi049 ceph-mon[40489]: 4.1e scrub ok 2024-09-21T13:52:57.662 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:52:57.663 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:52:57.672 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:52:57.697 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:57.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:57.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:52:57.700 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:52:57.700 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:52:57.738 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:52:57.739 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:52:57.783 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:52:57.783 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:52:57.800 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:52:57.822 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:52:57.823 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:52:57.842 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:57.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:57.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:52:57.857 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:52:57.857 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:52:57.892 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:52:57.892 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:52:57.910 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T13:52:57.925 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:52:57.925 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:52:57.944 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:57.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:57 smithi187 ceph-mon[33032]: pgmap v185: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 144 active+clean; 3.8 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 739 op/s; 9373/189639 objects degraded (4.943%); 13 MiB/s, 200 objects/s recovering 2024-09-21T13:52:57.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:57 smithi187 ceph-mon[33032]: 4.1e scrub starts 2024-09-21T13:52:57.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:57 smithi187 ceph-mon[33032]: 4.1e scrub ok 2024-09-21T13:52:57.966 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:57.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:57.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:58.069 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:58.091 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:58.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:58.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:58.193 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:52:58.217 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:58.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:58.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:52:58.318 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:52:58.342 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:58.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:58.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:52:58.443 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:52:58.490 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:58.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:58.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:52:58.591 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:52:58.613 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:58 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/4143540959' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:52:58.613 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:58 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:52:58.614 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:58 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:52:58.614 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:58 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:52:58.614 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:58 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:52:58.615 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:58 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/4143540959' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:52:58.615 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:58 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:52:58.615 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:58 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:52:58.615 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:58 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:52:58.615 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:58 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:52:58.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:58.633 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:58.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:52:58.659 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:52:58.660 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:52:58.734 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:52:58.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:58.799 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:58.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:52:58.900 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:52:58.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:58 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/4143540959' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:52:58.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:58 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:52:58.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:58 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:52:58.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:58 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:52:58.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:58 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:52:58.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:58.963 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:58.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:52:59.064 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:59.088 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:59.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:59.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:59.190 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:52:59.238 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:59.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:59.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:52:59.339 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:52:59.363 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:59.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:59.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:52:59.444 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:52:59 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:52:59.465 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:52:59.489 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:59.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:59.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:52:59.590 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:52:59.612 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:52:59.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:59.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:52:59.638 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T13:52:59.638 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-09-21T13:52:59.638 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:59.638 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.14817.0:215564 7.a e152/131 7:530ed55e:::benchmark_data_smithi049_76798_object79144:head v 152'13453, mlcod=152'13451)", 2024-09-21T13:52:59.638 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:59.634021+0000", 2024-09-21T13:52:59.639 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.00367177, 2024-09-21T13:52:59.639 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0037001149999999999, 2024-09-21T13:52:59.639 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:52:59.639 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:52:59.639 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T13:52:59.639 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:52:59.639 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:59.639 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:52:59.639 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:59.634021+0000", 2024-09-21T13:52:59.639 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:52:59.639 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:59.639 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:59.640 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:52:59.640 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:59.634021+0000", 2024-09-21T13:52:59.640 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:52:59.640 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:59.640 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:59.640 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:52:59.640 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:59.634022+0000", 2024-09-21T13:52:59.640 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.198e-06 2024-09-21T13:52:59.640 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:59.640 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:59.640 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:52:59.640 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:59.634029+0000", 2024-09-21T13:52:59.641 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.593e-06 2024-09-21T13:52:59.641 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:59.641 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:59.641 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:52:59.641 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:59.634047+0000", 2024-09-21T13:52:59.641 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7731e-05 2024-09-21T13:52:59.641 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:59.641 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:59.641 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:52:59.641 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:59.634056+0000", 2024-09-21T13:52:59.641 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6549999999999993e-06 2024-09-21T13:52:59.641 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:59.641 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:59.641 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:52:59.641 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:59.636060+0000", 2024-09-21T13:52:59.642 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0020043389999999999 2024-09-21T13:52:59.642 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:52:59.642 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:52:59.642 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:52:59.642 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:59.636080+0000", 2024-09-21T13:52:59.642 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0242e-05 2024-09-21T13:52:59.642 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:52:59.642 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:52:59.642 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:52:59.642 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:52:59.642 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T13:52:59.642 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 1 2024-09-21T13:52:59.642 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T13:52:59.714 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:52:59.721 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:59 smithi049 ceph-mon[40489]: pgmap v186: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 144 active+clean; 3.8 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 651 B/s rd, 0 B/s wr, 602 op/s; 9373/188754 objects degraded (4.966%); 9.6 MiB/s, 153 objects/s recovering 2024-09-21T13:52:59.722 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:59 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:52:59.722 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:59 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:52:59.722 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:52:59 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:52:59.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:59 smithi049 ceph-mon[30080]: pgmap v186: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 144 active+clean; 3.8 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 651 B/s rd, 0 B/s wr, 602 op/s; 9373/188754 objects degraded (4.966%); 9.6 MiB/s, 153 objects/s recovering 2024-09-21T13:52:59.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:59 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:52:59.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:59 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:52:59.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:52:59 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:52:59.771 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:59.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:59.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:52:59.873 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:52:59.926 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:52:59.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:52:59.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:52:59.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:59 smithi187 ceph-mon[33032]: pgmap v186: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 144 active+clean; 3.8 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 651 B/s rd, 0 B/s wr, 602 op/s; 9373/188754 objects degraded (4.966%); 9.6 MiB/s, 153 objects/s recovering 2024-09-21T13:52:59.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:59 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:52:59.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:59 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:52:59.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:52:59 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:53:00.004 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:53:00.028 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:53:00.090 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:00.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:00.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:53:00.192 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:53:00.217 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:00.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:00.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:53:00.319 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:53:00.344 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:00.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:00.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:53:00.445 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:53:00.498 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T13:53:00.500 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:00.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:00.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:53:00.606 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:53:00.667 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T13:53:00.677 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:00.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:00.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:53:00.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:00 smithi049 ceph-mon[30080]: 5.a scrub starts 2024-09-21T13:53:00.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:00 smithi049 ceph-mon[30080]: 5.a scrub ok 2024-09-21T13:53:00.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:00 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:53:00.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:00 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 9373/188754 objects degraded (4.966%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T13:53:00.719 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:00 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1048436101' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T13:53:00.720 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:53:00 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:53:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:53:00.720 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:00 smithi049 ceph-mon[40489]: 5.a scrub starts 2024-09-21T13:53:00.720 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:00 smithi049 ceph-mon[40489]: 5.a scrub ok 2024-09-21T13:53:00.720 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:00 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:53:00.721 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:00 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 9373/188754 objects degraded (4.966%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T13:53:00.721 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:00 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1048436101' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T13:53:00.779 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:53:00.803 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:00.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:00.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:53:00.898 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.7 to 1 (10000) 2024-09-21T13:53:00.906 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:53:00.931 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:00.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:00.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:53:00.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:00 smithi187 ceph-mon[33032]: 5.a scrub starts 2024-09-21T13:53:00.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:00 smithi187 ceph-mon[33032]: 5.a scrub ok 2024-09-21T13:53:00.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:00 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:53:00.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:00 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 9373/188754 objects degraded (4.966%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T13:53:00.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:00 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1048436101' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T13:53:01.033 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:53:01.058 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:01.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:01.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:53:01.159 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:53:01.195 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:01.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:01.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:53:01.298 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:53:01.327 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:01.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:01.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:53:01.428 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:01.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:01.451 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:01.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:01.552 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:53:01.575 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:01.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:01.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:53:01.676 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:53:01.698 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:01.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:01.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:53:01.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:01 smithi049 ceph-mon[40489]: pgmap v187: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 144 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s; 7622/175344 objects degraded (4.347%); 18 MiB/s, 284 objects/s recovering 2024-09-21T13:53:01.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:01 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1048436101' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T13:53:01.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:01 smithi049 ceph-mon[40489]: osdmap e153: 8 total, 8 up, 8 in 2024-09-21T13:53:01.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:01 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3124049260' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:53:01.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:01 smithi049 ceph-mon[30080]: pgmap v187: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 144 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s; 7622/175344 objects degraded (4.347%); 18 MiB/s, 284 objects/s recovering 2024-09-21T13:53:01.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:01 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1048436101' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T13:53:01.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:01 smithi049 ceph-mon[30080]: osdmap e153: 8 total, 8 up, 8 in 2024-09-21T13:53:01.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:01 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3124049260' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:53:01.801 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:53:01.830 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:01.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:01.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:53:01.854 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-21T13:53:01.857 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T13:53:01.879 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:53:01.899 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-09-21T13:53:01.932 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:53:01.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:01 smithi187 ceph-mon[33032]: pgmap v187: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 144 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s; 7622/175344 objects degraded (4.347%); 18 MiB/s, 284 objects/s recovering 2024-09-21T13:53:01.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:01 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1048436101' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T13:53:01.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:01 smithi187 ceph-mon[33032]: osdmap e153: 8 total, 8 up, 8 in 2024-09-21T13:53:01.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:01 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3124049260' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:53:01.971 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:01.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:01.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:53:02.073 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:53:02.136 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:02.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:02.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:53:02.175 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:53:02.239 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:53:02.267 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:02.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:02.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:53:02.369 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:53:02.395 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:02.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:02.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:53:02.497 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:53:02.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:02.561 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:02.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:53:02.662 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:53:02.707 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:02.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:02.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:53:02.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:02 smithi049 ceph-mon[30080]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:53:02.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:02 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3124049260' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T13:53:02.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:02 smithi049 ceph-mon[30080]: osdmap e154: 8 total, 8 up, 8 in 2024-09-21T13:53:02.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:02 smithi049 ceph-mon[40489]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:53:02.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:02 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3124049260' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T13:53:02.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:02 smithi049 ceph-mon[40489]: osdmap e154: 8 total, 8 up, 8 in 2024-09-21T13:53:02.810 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:53:02.833 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:02.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:02.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:53:02.935 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:53:02.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:02 smithi187 ceph-mon[33032]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:53:02.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:02 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3124049260' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T13:53:02.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:02 smithi187 ceph-mon[33032]: osdmap e154: 8 total, 8 up, 8 in 2024-09-21T13:53:02.962 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:02.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:02.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:53:03.064 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:53:03.101 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:03.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:03.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:53:03.203 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:53:03.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:03.229 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:03.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:53:03.330 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:53:03.376 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:03.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:03.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:53:03.478 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:53:03.504 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:03.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:03.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:53:03.605 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:53:03.629 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:03.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:03.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:53:03.686 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T13:53:03.686 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-09-21T13:53:03.687 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-09-21T13:53:03.687 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-09-21T13:53:03.687 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T13:53:03.731 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:53:03.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:03 smithi049 ceph-mon[40489]: pgmap v189: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 144 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 662 B/s rd, 0 B/s wr, 617 op/s; 7622/175344 objects degraded (4.347%); 8.7 MiB/s, 138 objects/s recovering 2024-09-21T13:53:03.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:03 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:53:03.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:03 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:53:03.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:03 smithi049 ceph-mon[40489]: osdmap e155: 8 total, 8 up, 8 in 2024-09-21T13:53:03.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:03 smithi049 ceph-mon[30080]: pgmap v189: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 144 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 662 B/s rd, 0 B/s wr, 617 op/s; 7622/175344 objects degraded (4.347%); 8.7 MiB/s, 138 objects/s recovering 2024-09-21T13:53:03.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:03 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:53:03.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:03 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:53:03.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:03 smithi049 ceph-mon[30080]: osdmap e155: 8 total, 8 up, 8 in 2024-09-21T13:53:03.775 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:03.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:03.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:53:03.877 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:53:03.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:03.926 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:03.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:53:03.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:03 smithi187 ceph-mon[33032]: pgmap v189: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 144 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 662 B/s rd, 0 B/s wr, 617 op/s; 7622/175344 objects degraded (4.347%); 8.7 MiB/s, 138 objects/s recovering 2024-09-21T13:53:03.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:03 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:53:03.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:03 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:53:03.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:03 smithi187 ceph-mon[33032]: osdmap e155: 8 total, 8 up, 8 in 2024-09-21T13:53:04.015 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:53:04.015 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:53:04.015 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:53:04.015 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:53:04.015 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:53:04.015 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:53:04.015 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:53:04.016 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:53:04.016 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:53:04.016 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:53:04.016 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:53:04.016 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:53:04.016 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:53:04.016 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:53:04.023 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:53:04.023 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:53:04.027 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:53:04.050 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:04.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:04.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:53:04.151 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:53:04.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:04.189 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:04.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:53:04.290 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:53:04.317 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:04.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:04.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:53:04.419 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:53:04.464 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:04.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:04.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:53:04.565 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:53:04.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:04.612 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:04.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:53:04.713 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:53:04.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:04 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1507389511' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:53:04.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:04 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1507389511' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:53:04.757 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:04.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:04.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:53:04.858 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:53:04.884 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:04.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:04.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:53:04.917 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T13:53:04.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:04 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1507389511' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:53:04.986 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:53:05.011 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:05.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:05.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:53:05.113 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:53:05.138 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:05.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:05.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:53:05.240 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:53:05.241 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T13:53:05.303 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:53:05.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:05.358 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:05.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:53:05.459 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:53:05.484 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:05.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:05.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:53:05.586 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:53:05.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:05.613 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:05.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:53:05.649 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:53:05.709 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:05 smithi049 ceph-mon[40489]: pgmap v192: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 144 active+clean; 3.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.24k op/s; 6827/166533 objects degraded (4.099%); 18 MiB/s, 292 objects/s recovering 2024-09-21T13:53:05.709 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:05 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 6827/166533 objects degraded (4.099%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T13:53:05.709 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:05 smithi049 ceph-mon[40489]: osdmap e156: 8 total, 8 up, 8 in 2024-09-21T13:53:05.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:05 smithi049 ceph-mon[30080]: pgmap v192: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 144 active+clean; 3.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.24k op/s; 6827/166533 objects degraded (4.099%); 18 MiB/s, 292 objects/s recovering 2024-09-21T13:53:05.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:05 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 6827/166533 objects degraded (4.099%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T13:53:05.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:05 smithi049 ceph-mon[30080]: osdmap e156: 8 total, 8 up, 8 in 2024-09-21T13:53:05.715 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:53:05.771 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:05.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:05.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:53:05.873 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:53:05.918 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:05.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:05.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:53:05.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:05 smithi187 ceph-mon[33032]: pgmap v192: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 144 active+clean; 3.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.24k op/s; 6827/166533 objects degraded (4.099%); 18 MiB/s, 292 objects/s recovering 2024-09-21T13:53:05.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:05 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 6827/166533 objects degraded (4.099%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T13:53:05.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:05 smithi187 ceph-mon[33032]: osdmap e156: 8 total, 8 up, 8 in 2024-09-21T13:53:06.020 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:53:06.047 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:06.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:06.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:53:06.148 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:06.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:06.175 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:06.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:06.276 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:53:06.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:06.300 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:06.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:53:06.401 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:53:06.447 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:06.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:06.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:53:06.549 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:53:06.618 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:06.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:06.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:53:06.637 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:53:06.643 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:53:06.645 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-09-21T13:53:06.720 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:53:06.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:06.764 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:06.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:53:06.864 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T13:53:06.864 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:53:06.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:06.914 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:06.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:53:07.015 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:07.038 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:07.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:07.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:07.140 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:53:07.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:07.180 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:07.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:53:07.281 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:53:07.307 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:07.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:07.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:53:07.409 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:53:07.445 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:07.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:07.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:53:07.547 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:53:07.554 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-09-21T13:53:07.573 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:07.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:07.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:53:07.674 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:53:07.698 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:07.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:07.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:53:07.800 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:53:07.801 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":194,"stamp":"2024-09-21T13:53:05.744060+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3563657779,"num_objects":54568,"num_object_clones":0,"num_object_copies":163704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5682,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54568,"num_whiteouts":0,"num_read":1085,"num_read_kb":942,"num_write":219016,"num_write_kb":8732362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19768,"num_bytes_recovered":1295123196,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85660,"ondisk_log_size":85660,"up":444,"acting":441,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":11641632,"kb_used_data":10556588,"kb_used_omap":15,"kb_used_meta":1084720,"kb_avail":738319584,"statfs":{"total":767960285184,"available":756039254016,"internally_reserved":0,"allocated":10809946112,"data_stored":10806104799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15598,"internal_metadata":1110754066},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,9,9,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-254279680,"num_objects":-3880,"num_object_clones":0,"num_object_copies":-11640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1940,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3880,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":3880,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1492,"num_bytes_recovered":97779712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.839685"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":153,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.913760+0000","last_change":"2024-09-21T13:52:26.677736+0000","last_active":"2024-09-21T13:53:02.913760+0000","last_peered":"2024-09-21T13:53:02.913760+0000","last_clean":"2024-09-21T13:53:02.913760+0000","last_became_active":"2024-09-21T13:52:04.317322+0000","last_became_peered":"2024-09-21T13:52:04.317322+0000","last_unstale":"2024-09-21T13:53:02.913760+0000","last_undegraded":"2024-09-21T13:53:02.913760+0000","last_fullsized":"2024-09-21T13:53:02.913760+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:26.677689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:52:26.677689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:45.636529+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"68'9","reported_seq":221,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.899357+0000","last_change":"2024-09-21T13:51:55.225649+0000","last_active":"2024-09-21T13:53:02.899357+0000","last_peered":"2024-09-21T13:53:02.899357+0000","last_clean":"2024-09-21T13:53:02.899357+0000","last_became_active":"2024-09-21T13:49:00.055210+0000","last_became_peered":"2024-09-21T13:49:00.055210+0000","last_unstale":"2024-09-21T13:53:02.899357+0000","last_undegraded":"2024-09-21T13:53:02.899357+0000","last_fullsized":"2024-09-21T13:53:02.899357+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:51:55.225579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:55.225579+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":201,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.912986+0000","last_change":"2024-09-21T13:50:28.058990+0000","last_active":"2024-09-21T13:53:02.912986+0000","last_peered":"2024-09-21T13:53:02.912986+0000","last_clean":"2024-09-21T13:53:02.912986+0000","last_became_active":"2024-09-21T13:49:02.060984+0000","last_became_peered":"2024-09-21T13:49:02.060984+0000","last_unstale":"2024-09-21T13:53:02.912986+0000","last_undegraded":"2024-09-21T13:53:02.912986+0000","last_fullsized":"2024-09-21T13:53:02.912986+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:28.058918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:28.058918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:39.434903+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":189,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.899435+0000","last_change":"2024-09-21T13:49:04.072734+0000","last_active":"2024-09-21T13:53:02.899435+0000","last_peered":"2024-09-21T13:53:02.899435+0000","last_clean":"2024-09-21T13:53:02.899435+0000","last_became_active":"2024-09-21T13:49:04.072460+0000","last_became_peered":"2024-09-21T13:49:04.072460+0000","last_unstale":"2024-09-21T13:53:02.899435+0000","last_undegraded":"2024-09-21T13:53:02.899435+0000","last_fullsized":"2024-09-21T13:53:02.899435+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":197,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.906774+0000","last_change":"2024-09-21T13:51:18.572788+0000","last_active":"2024-09-21T13:53:02.906774+0000","last_peered":"2024-09-21T13:53:02.906774+0000","last_clean":"2024-09-21T13:53:02.906774+0000","last_became_active":"2024-09-21T13:49:04.433965+0000","last_became_peered":"2024-09-21T13:49:04.433965+0000","last_unstale":"2024-09-21T13:53:02.906774+0000","last_undegraded":"2024-09-21T13:53:02.906774+0000","last_fullsized":"2024-09-21T13:53:02.906774+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:18.572729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:51:18.572729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":213,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.918434+0000","last_change":"2024-09-21T13:52:04.317119+0000","last_active":"2024-09-21T13:53:02.918434+0000","last_peered":"2024-09-21T13:53:02.918434+0000","last_clean":"2024-09-21T13:53:02.918434+0000","last_became_active":"2024-09-21T13:52:04.316819+0000","last_became_peered":"2024-09-21T13:52:04.316819+0000","last_unstale":"2024-09-21T13:53:02.918434+0000","last_undegraded":"2024-09-21T13:53:02.918434+0000","last_fullsized":"2024-09-21T13:53:02.918434+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"68'15","reported_seq":233,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.904444+0000","last_change":"2024-09-21T13:52:04.315645+0000","last_active":"2024-09-21T13:53:02.904444+0000","last_peered":"2024-09-21T13:53:02.904444+0000","last_clean":"2024-09-21T13:53:02.904444+0000","last_became_active":"2024-09-21T13:52:04.315101+0000","last_became_peered":"2024-09-21T13:52:04.315101+0000","last_unstale":"2024-09-21T13:53:02.904444+0000","last_undegraded":"2024-09-21T13:53:02.904444+0000","last_fullsized":"2024-09-21T13:53:02.904444+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":209,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.899265+0000","last_change":"2024-09-21T13:51:43.040397+0000","last_active":"2024-09-21T13:53:02.899265+0000","last_peered":"2024-09-21T13:53:02.899265+0000","last_clean":"2024-09-21T13:53:02.899265+0000","last_became_active":"2024-09-21T13:49:02.063687+0000","last_became_peered":"2024-09-21T13:49:02.063687+0000","last_unstale":"2024-09-21T13:53:02.899265+0000","last_undegraded":"2024-09-21T13:53:02.899265+0000","last_fullsized":"2024-09-21T13:53:02.899265+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:43.040348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:43.040348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":217,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.913415+0000","last_change":"2024-09-21T13:52:49.451929+0000","last_active":"2024-09-21T13:53:02.913415+0000","last_peered":"2024-09-21T13:53:02.913415+0000","last_clean":"2024-09-21T13:53:02.913415+0000","last_became_active":"2024-09-21T13:48:58.050695+0000","last_became_peered":"2024-09-21T13:48:58.050695+0000","last_unstale":"2024-09-21T13:53:02.913415+0000","last_undegraded":"2024-09-21T13:53:02.913415+0000","last_fullsized":"2024-09-21T13:53:02.913415+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:49.451882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:52:49.451882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:54.087443+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"68'9","reported_seq":221,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.899809+0000","last_change":"2024-09-21T13:51:21.951391+0000","last_active":"2024-09-21T13:53:02.899809+0000","last_peered":"2024-09-21T13:53:02.899809+0000","last_clean":"2024-09-21T13:53:02.899809+0000","last_became_active":"2024-09-21T13:49:00.059382+0000","last_became_peered":"2024-09-21T13:49:00.059382+0000","last_unstale":"2024-09-21T13:53:02.899809+0000","last_undegraded":"2024-09-21T13:53:02.899809+0000","last_fullsized":"2024-09-21T13:53:02.899809+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:51:21.951305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:21.951305+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":193,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.912881+0000","last_change":"2024-09-21T13:49:02.066978+0000","last_active":"2024-09-21T13:53:02.912881+0000","last_peered":"2024-09-21T13:53:02.912881+0000","last_clean":"2024-09-21T13:53:02.912881+0000","last_became_active":"2024-09-21T13:49:02.066693+0000","last_became_peered":"2024-09-21T13:49:02.066693+0000","last_unstale":"2024-09-21T13:53:02.912881+0000","last_undegraded":"2024-09-21T13:53:02.912881+0000","last_fullsized":"2024-09-21T13:53:02.912881+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:08.561603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":150,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.908656+0000","last_change":"2024-09-21T13:52:28.417865+0000","last_active":"2024-09-21T13:53:02.908656+0000","last_peered":"2024-09-21T13:53:02.908656+0000","last_clean":"2024-09-21T13:53:02.908656+0000","last_became_active":"2024-09-21T13:50:29.689381+0000","last_became_peered":"2024-09-21T13:50:29.689381+0000","last_unstale":"2024-09-21T13:53:02.908656+0000","last_undegraded":"2024-09-21T13:53:02.908656+0000","last_fullsized":"2024-09-21T13:53:02.908656+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:28.417690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:28.417690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:33.915909+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":209,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.912971+0000","last_change":"2024-09-21T13:50:39.480995+0000","last_active":"2024-09-21T13:53:02.912971+0000","last_peered":"2024-09-21T13:53:02.912971+0000","last_clean":"2024-09-21T13:53:02.912971+0000","last_became_active":"2024-09-21T13:48:58.050906+0000","last_became_peered":"2024-09-21T13:48:58.050906+0000","last_unstale":"2024-09-21T13:53:02.912971+0000","last_undegraded":"2024-09-21T13:53:02.912971+0000","last_fullsized":"2024-09-21T13:53:02.912971+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:39.480930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:39.480930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:32.611461+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"68'5","reported_seq":231,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.899517+0000","last_change":"2024-09-21T13:52:04.319297+0000","last_active":"2024-09-21T13:53:02.899517+0000","last_peered":"2024-09-21T13:53:02.899517+0000","last_clean":"2024-09-21T13:53:02.899517+0000","last_became_active":"2024-09-21T13:52:04.318515+0000","last_became_peered":"2024-09-21T13:52:04.318515+0000","last_unstale":"2024-09-21T13:53:02.899517+0000","last_undegraded":"2024-09-21T13:53:02.899517+0000","last_fullsized":"2024-09-21T13:53:02.899517+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:02.067591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":201,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.908343+0000","last_change":"2024-09-21T13:50:40.188797+0000","last_active":"2024-09-21T13:53:02.908343+0000","last_peered":"2024-09-21T13:53:02.908343+0000","last_clean":"2024-09-21T13:53:02.908343+0000","last_became_active":"2024-09-21T13:49:02.060212+0000","last_became_peered":"2024-09-21T13:49:02.060212+0000","last_unstale":"2024-09-21T13:53:02.908343+0000","last_undegraded":"2024-09-21T13:53:02.908343+0000","last_fullsized":"2024-09-21T13:53:02.908343+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:40.188712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:40.188712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":205,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.912668+0000","last_change":"2024-09-21T13:52:43.516806+0000","last_active":"2024-09-21T13:53:02.912668+0000","last_peered":"2024-09-21T13:53:02.912668+0000","last_clean":"2024-09-21T13:53:02.912668+0000","last_became_active":"2024-09-21T13:49:04.078026+0000","last_became_peered":"2024-09-21T13:49:04.078026+0000","last_unstale":"2024-09-21T13:53:02.912668+0000","last_undegraded":"2024-09-21T13:53:02.912668+0000","last_fullsized":"2024-09-21T13:53:02.912668+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:43.516741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:43.516741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:08.753512+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":197,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.900217+0000","last_change":"2024-09-21T13:50:38.027288+0000","last_active":"2024-09-21T13:53:02.900217+0000","last_peered":"2024-09-21T13:53:02.900217+0000","last_clean":"2024-09-21T13:53:02.900217+0000","last_became_active":"2024-09-21T13:49:04.431422+0000","last_became_peered":"2024-09-21T13:49:04.431422+0000","last_unstale":"2024-09-21T13:53:02.900217+0000","last_undegraded":"2024-09-21T13:53:02.900217+0000","last_fullsized":"2024-09-21T13:53:02.900217+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:38.027227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:38.027227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":134,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.899799+0000","last_change":"2024-09-21T13:50:29.692621+0000","last_active":"2024-09-21T13:53:02.899799+0000","last_peered":"2024-09-21T13:53:02.899799+0000","last_clean":"2024-09-21T13:53:02.899799+0000","last_became_active":"2024-09-21T13:50:29.688282+0000","last_became_peered":"2024-09-21T13:50:29.688282+0000","last_unstale":"2024-09-21T13:53:02.899799+0000","last_undegraded":"2024-09-21T13:53:02.899799+0000","last_fullsized":"2024-09-21T13:53:02.899799+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"68'15","reported_seq":160,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-21T13:53:01.908009+0000","last_change":"2024-09-21T13:52:04.324874+0000","last_active":"2024-09-21T13:53:01.908009+0000","last_peered":"2024-09-21T13:53:01.908009+0000","last_clean":"2024-09-21T13:53:01.908009+0000","last_became_active":"2024-09-21T13:52:04.324273+0000","last_became_peered":"2024-09-21T13:52:04.324273+0000","last_unstale":"2024-09-21T13:53:01.908009+0000","last_undegraded":"2024-09-21T13:53:01.908009+0000","last_fullsized":"2024-09-21T13:53:01.908009+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:50:24.749694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:24.749694+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":209,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.912007+0000","last_change":"2024-09-21T13:52:26.372217+0000","last_active":"2024-09-21T13:53:02.912007+0000","last_peered":"2024-09-21T13:53:02.912007+0000","last_clean":"2024-09-21T13:53:02.912007+0000","last_became_active":"2024-09-21T13:49:02.066796+0000","last_became_peered":"2024-09-21T13:49:02.066796+0000","last_unstale":"2024-09-21T13:53:02.912007+0000","last_undegraded":"2024-09-21T13:53:02.912007+0000","last_fullsized":"2024-09-21T13:53:02.912007+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:26.372144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:52:26.372144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:40.709431+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":189,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.907313+0000","last_change":"2024-09-21T13:49:04.434429+0000","last_active":"2024-09-21T13:53:02.907313+0000","last_peered":"2024-09-21T13:53:02.907313+0000","last_clean":"2024-09-21T13:53:02.907313+0000","last_became_active":"2024-09-21T13:49:04.434049+0000","last_became_peered":"2024-09-21T13:49:04.434049+0000","last_unstale":"2024-09-21T13:53:02.907313+0000","last_undegraded":"2024-09-21T13:53:02.907313+0000","last_fullsized":"2024-09-21T13:53:02.907313+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":220,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.899928+0000","last_change":"2024-09-21T13:52:52.142682+0000","last_active":"2024-09-21T13:53:02.899928+0000","last_peered":"2024-09-21T13:53:02.899928+0000","last_clean":"2024-09-21T13:53:02.899928+0000","last_became_active":"2024-09-21T13:52:04.316016+0000","last_became_peered":"2024-09-21T13:52:04.316016+0000","last_unstale":"2024-09-21T13:53:02.899928+0000","last_undegraded":"2024-09-21T13:53:02.899928+0000","last_fullsized":"2024-09-21T13:53:02.899928+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:52.142638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:52:52.142638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:13.595892+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"68'12","reported_seq":228,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.905696+0000","last_change":"2024-09-21T13:51:52.819966+0000","last_active":"2024-09-21T13:53:02.905696+0000","last_peered":"2024-09-21T13:53:02.905696+0000","last_clean":"2024-09-21T13:53:02.905696+0000","last_became_active":"2024-09-21T13:49:00.061867+0000","last_became_peered":"2024-09-21T13:49:00.061867+0000","last_unstale":"2024-09-21T13:53:02.905696+0000","last_undegraded":"2024-09-21T13:53:02.905696+0000","last_fullsized":"2024-09-21T13:53:02.905696+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'12","last_scrub_stamp":"2024-09-21T13:51:52.819879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:52.819879+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:16.488906+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":204,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.900179+0000","last_change":"2024-09-21T13:52:04.321297+0000","last_active":"2024-09-21T13:53:02.900179+0000","last_peered":"2024-09-21T13:53:02.900179+0000","last_clean":"2024-09-21T13:53:02.900179+0000","last_became_active":"2024-09-21T13:52:04.320800+0000","last_became_peered":"2024-09-21T13:52:04.320800+0000","last_unstale":"2024-09-21T13:53:02.900179+0000","last_undegraded":"2024-09-21T13:53:02.900179+0000","last_fullsized":"2024-09-21T13:53:02.900179+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":209,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.908581+0000","last_change":"2024-09-21T13:52:54.580070+0000","last_active":"2024-09-21T13:53:02.908581+0000","last_peered":"2024-09-21T13:53:02.908581+0000","last_clean":"2024-09-21T13:53:02.908581+0000","last_became_active":"2024-09-21T13:52:04.321358+0000","last_became_peered":"2024-09-21T13:52:04.321358+0000","last_unstale":"2024-09-21T13:53:02.908581+0000","last_undegraded":"2024-09-21T13:53:02.908581+0000","last_fullsized":"2024-09-21T13:53:02.908581+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T13:52:54.580022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:54.580022+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:21.087079+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":218,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.913482+0000","last_change":"2024-09-21T13:52:43.516442+0000","last_active":"2024-09-21T13:53:02.913482+0000","last_peered":"2024-09-21T13:53:02.913482+0000","last_clean":"2024-09-21T13:53:02.913482+0000","last_became_active":"2024-09-21T13:48:58.048972+0000","last_became_peered":"2024-09-21T13:48:58.048972+0000","last_unstale":"2024-09-21T13:53:02.913482+0000","last_undegraded":"2024-09-21T13:53:02.913482+0000","last_fullsized":"2024-09-21T13:53:02.913482+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:52:43.516221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:52:43.516221+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:48.258159+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"68'10","reported_seq":177,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.908824+0000","last_change":"2024-09-21T13:52:56.526032+0000","last_active":"2024-09-21T13:53:02.908824+0000","last_peered":"2024-09-21T13:53:02.908824+0000","last_clean":"2024-09-21T13:53:02.908824+0000","last_became_active":"2024-09-21T13:50:29.694465+0000","last_became_peered":"2024-09-21T13:50:29.694465+0000","last_unstale":"2024-09-21T13:53:02.908824+0000","last_undegraded":"2024-09-21T13:53:02.908824+0000","last_fullsized":"2024-09-21T13:53:02.908824+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:52:56.525966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:52:56.525966+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:10.063932+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"68'8","reported_seq":270,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.914619+0000","last_change":"2024-09-21T13:51:42.174696+0000","last_active":"2024-09-21T13:53:02.914619+0000","last_peered":"2024-09-21T13:53:02.914619+0000","last_clean":"2024-09-21T13:53:02.914619+0000","last_became_active":"2024-09-21T13:49:02.059638+0000","last_became_peered":"2024-09-21T13:49:02.059638+0000","last_unstale":"2024-09-21T13:53:02.914619+0000","last_undegraded":"2024-09-21T13:53:02.914619+0000","last_fullsized":"2024-09-21T13:53:02.914619+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:51:42.174629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:42.174629+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:08.098143+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"155'13715","reported_seq":13995,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:03.369602+0000","last_change":"2024-09-21T13:52:51.407976+0000","last_active":"2024-09-21T13:53:03.369602+0000","last_peered":"2024-09-21T13:53:03.369602+0000","last_clean":"2024-09-21T13:53:03.369602+0000","last_became_active":"2024-09-21T13:50:07.422961+0000","last_became_peered":"2024-09-21T13:50:07.422961+0000","last_unstale":"2024-09-21T13:53:03.369602+0000","last_undegraded":"2024-09-21T13:53:03.369602+0000","last_fullsized":"2024-09-21T13:53:03.369602+0000","mapping_epoch":83,"log_start":"114'7400","ondisk_log_start":"114'7400","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'13038","last_scrub_stamp":"2024-09-21T13:52:51.407818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:52:51.407818+0000","objects_scrubbed":4170,"log_size":6315,"log_dups_size":0,"ondisk_log_size":6315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:10.866985+0000","scrub_duration":960,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227606528,"num_objects":3473,"num_object_clones":0,"num_object_copies":10419,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13715,"num_write_kb":550016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"68'15","reported_seq":238,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.908636+0000","last_change":"2024-09-21T13:52:16.452049+0000","last_active":"2024-09-21T13:53:02.908636+0000","last_peered":"2024-09-21T13:53:02.908636+0000","last_clean":"2024-09-21T13:53:02.908636+0000","last_became_active":"2024-09-21T13:49:00.060324+0000","last_became_peered":"2024-09-21T13:49:00.060324+0000","last_unstale":"2024-09-21T13:53:02.908636+0000","last_undegraded":"2024-09-21T13:53:02.908636+0000","last_fullsized":"2024-09-21T13:53:02.908636+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:52:16.451964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:52:16.451964+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:31.981849+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":209,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.913410+0000","last_change":"2024-09-21T13:50:06.151943+0000","last_active":"2024-09-21T13:53:02.913410+0000","last_peered":"2024-09-21T13:53:02.913410+0000","last_clean":"2024-09-21T13:53:02.913410+0000","last_became_active":"2024-09-21T13:48:58.051911+0000","last_became_peered":"2024-09-21T13:48:58.051911+0000","last_unstale":"2024-09-21T13:53:02.913410+0000","last_undegraded":"2024-09-21T13:53:02.913410+0000","last_fullsized":"2024-09-21T13:53:02.913410+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.151888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.151888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"68'8","reported_seq":254,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.900304+0000","last_change":"2024-09-21T13:49:02.061966+0000","last_active":"2024-09-21T13:53:02.900304+0000","last_peered":"2024-09-21T13:53:02.900304+0000","last_clean":"2024-09-21T13:53:02.900304+0000","last_became_active":"2024-09-21T13:49:02.061679+0000","last_became_peered":"2024-09-21T13:49:02.061679+0000","last_unstale":"2024-09-21T13:53:02.900304+0000","last_undegraded":"2024-09-21T13:53:02.900304+0000","last_fullsized":"2024-09-21T13:53:02.900304+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":197,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.912730+0000","last_change":"2024-09-21T13:50:34.481503+0000","last_active":"2024-09-21T13:53:02.912730+0000","last_peered":"2024-09-21T13:53:02.912730+0000","last_clean":"2024-09-21T13:53:02.912730+0000","last_became_active":"2024-09-21T13:49:04.081905+0000","last_became_peered":"2024-09-21T13:49:04.081905+0000","last_unstale":"2024-09-21T13:53:02.912730+0000","last_undegraded":"2024-09-21T13:53:02.912730+0000","last_fullsized":"2024-09-21T13:53:02.912730+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:34.481450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:34.481450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:42.543466+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"155'13821","reported_seq":19370,"reported_epoch":155,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-21T13:53:03.860418+0000","last_change":"2024-09-21T13:52:41.673644+0000","last_active":"2024-09-21T13:53:03.860418+0000","last_peered":"2024-09-21T13:53:03.860418+0000","last_clean":"2024-09-21T13:50:42.911378+0000","last_became_active":"2024-09-21T13:52:04.993197+0000","last_became_peered":"2024-09-21T13:52:04.993197+0000","last_unstale":"2024-09-21T13:53:03.860418+0000","last_undegraded":"2024-09-21T13:52:04.885711+0000","last_fullsized":"2024-09-21T13:52:04.863350+0000","mapping_epoch":131,"log_start":"124'9200","ondisk_log_start":"124'9200","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":4621,"log_dups_size":0,"ondisk_log_size":4621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":223936512,"num_objects":3417,"num_object_clones":0,"num_object_copies":10251,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13821,"num_write_kb":551616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2693,"num_bytes_recovered":176488448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":3417}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"68'18","reported_seq":238,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.913209+0000","last_change":"2024-09-21T13:50:27.281455+0000","last_active":"2024-09-21T13:53:02.913209+0000","last_peered":"2024-09-21T13:53:02.913209+0000","last_clean":"2024-09-21T13:53:02.913209+0000","last_became_active":"2024-09-21T13:49:00.429314+0000","last_became_peered":"2024-09-21T13:49:00.429314+0000","last_unstale":"2024-09-21T13:53:02.913209+0000","last_undegraded":"2024-09-21T13:53:02.913209+0000","last_fullsized":"2024-09-21T13:53:02.913209+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'18","last_scrub_stamp":"2024-09-21T13:50:27.281356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:27.281356+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":201,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.912713+0000","last_change":"2024-09-21T13:48:58.056669+0000","last_active":"2024-09-21T13:53:02.912713+0000","last_peered":"2024-09-21T13:53:02.912713+0000","last_clean":"2024-09-21T13:53:02.912713+0000","last_became_active":"2024-09-21T13:48:58.056375+0000","last_became_peered":"2024-09-21T13:48:58.056375+0000","last_unstale":"2024-09-21T13:53:02.912713+0000","last_undegraded":"2024-09-21T13:53:02.912713+0000","last_fullsized":"2024-09-21T13:53:02.912713+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":201,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.899567+0000","last_change":"2024-09-21T13:51:23.940809+0000","last_active":"2024-09-21T13:53:02.899567+0000","last_peered":"2024-09-21T13:53:02.899567+0000","last_clean":"2024-09-21T13:53:02.899567+0000","last_became_active":"2024-09-21T13:49:02.062955+0000","last_became_peered":"2024-09-21T13:49:02.062955+0000","last_unstale":"2024-09-21T13:53:02.899567+0000","last_undegraded":"2024-09-21T13:53:02.899567+0000","last_fullsized":"2024-09-21T13:53:02.899567+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:23.940746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:23.940746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":200,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.899709+0000","last_change":"2024-09-21T13:52:04.316586+0000","last_active":"2024-09-21T13:53:02.899709+0000","last_peered":"2024-09-21T13:53:02.899709+0000","last_clean":"2024-09-21T13:53:02.899709+0000","last_became_active":"2024-09-21T13:52:04.316121+0000","last_became_peered":"2024-09-21T13:52:04.316121+0000","last_unstale":"2024-09-21T13:53:02.899709+0000","last_undegraded":"2024-09-21T13:53:02.899709+0000","last_fullsized":"2024-09-21T13:53:02.899709+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"155'13412","reported_seq":13586,"reported_epoch":155,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2024-09-21T13:53:03.802123+0000","last_change":"2024-09-21T13:52:35.034828+0000","last_active":"2024-09-21T13:53:03.802123+0000","last_peered":"2024-09-21T13:53:03.802123+0000","last_clean":"2024-09-21T13:50:42.909755+0000","last_became_active":"2024-09-21T13:52:04.998636+0000","last_became_peered":"2024-09-21T13:52:04.998636+0000","last_unstale":"2024-09-21T13:53:03.802123+0000","last_undegraded":"2024-09-21T13:52:04.888197+0000","last_fullsized":"2024-09-21T13:52:04.863335+0000","mapping_epoch":131,"log_start":"122'8700","ondisk_log_start":"122'8700","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":4712,"log_dups_size":0,"ondisk_log_size":4712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224657408,"num_objects":3428,"num_object_clones":0,"num_object_copies":10284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3428,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13412,"num_write_kb":538880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":3428}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"68'14","reported_seq":226,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.899670+0000","last_change":"2024-09-21T13:50:29.009786+0000","last_active":"2024-09-21T13:53:02.899670+0000","last_peered":"2024-09-21T13:53:02.899670+0000","last_clean":"2024-09-21T13:53:02.899670+0000","last_became_active":"2024-09-21T13:49:00.055031+0000","last_became_peered":"2024-09-21T13:49:00.055031+0000","last_unstale":"2024-09-21T13:53:02.899670+0000","last_undegraded":"2024-09-21T13:53:02.899670+0000","last_fullsized":"2024-09-21T13:53:02.899670+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'14","last_scrub_stamp":"2024-09-21T13:50:29.009720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:29.009720+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:04.236253+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":210,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.914606+0000","last_change":"2024-09-21T13:49:59.253636+0000","last_active":"2024-09-21T13:53:02.914606+0000","last_peered":"2024-09-21T13:53:02.914606+0000","last_clean":"2024-09-21T13:53:02.914606+0000","last_became_active":"2024-09-21T13:48:58.053533+0000","last_became_peered":"2024-09-21T13:48:58.053533+0000","last_unstale":"2024-09-21T13:53:02.914606+0000","last_undegraded":"2024-09-21T13:53:02.914606+0000","last_fullsized":"2024-09-21T13:53:02.914606+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:49:59.253587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:49:59.253587+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:34.697714+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"68'8","reported_seq":259,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.916041+0000","last_change":"2024-09-21T13:50:41.458637+0000","last_active":"2024-09-21T13:53:02.916041+0000","last_peered":"2024-09-21T13:53:02.916041+0000","last_clean":"2024-09-21T13:53:02.916041+0000","last_became_active":"2024-09-21T13:49:02.070064+0000","last_became_peered":"2024-09-21T13:49:02.070064+0000","last_unstale":"2024-09-21T13:53:02.916041+0000","last_undegraded":"2024-09-21T13:53:02.916041+0000","last_fullsized":"2024-09-21T13:53:02.916041+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:50:41.458562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:41.458562+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":208,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.908219+0000","last_change":"2024-09-21T13:52:55.541288+0000","last_active":"2024-09-21T13:53:02.908219+0000","last_peered":"2024-09-21T13:53:02.908219+0000","last_clean":"2024-09-21T13:53:02.908219+0000","last_became_active":"2024-09-21T13:52:04.322185+0000","last_became_peered":"2024-09-21T13:52:04.322185+0000","last_unstale":"2024-09-21T13:53:02.908219+0000","last_undegraded":"2024-09-21T13:53:02.908219+0000","last_fullsized":"2024-09-21T13:53:02.908219+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:55.541232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:55.541232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:57.661213+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"155'13514","reported_seq":13670,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:03.373143+0000","last_change":"2024-09-21T13:50:07.419221+0000","last_active":"2024-09-21T13:53:03.373143+0000","last_peered":"2024-09-21T13:53:03.373143+0000","last_clean":"2024-09-21T13:53:03.373143+0000","last_became_active":"2024-09-21T13:50:07.418929+0000","last_became_peered":"2024-09-21T13:50:07.418929+0000","last_unstale":"2024-09-21T13:53:03.373143+0000","last_undegraded":"2024-09-21T13:53:03.373143+0000","last_fullsized":"2024-09-21T13:53:03.373143+0000","mapping_epoch":83,"log_start":"122'8700","ondisk_log_start":"122'8700","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":4814,"log_dups_size":0,"ondisk_log_size":4814,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220856320,"num_objects":3370,"num_object_clones":0,"num_object_copies":10110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13514,"num_write_kb":540288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"68'10","reported_seq":228,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.909144+0000","last_change":"2024-09-21T13:51:50.211365+0000","last_active":"2024-09-21T13:53:02.909144+0000","last_peered":"2024-09-21T13:53:02.909144+0000","last_clean":"2024-09-21T13:53:02.909144+0000","last_became_active":"2024-09-21T13:49:00.054564+0000","last_became_peered":"2024-09-21T13:49:00.054564+0000","last_unstale":"2024-09-21T13:53:02.909144+0000","last_undegraded":"2024-09-21T13:53:02.909144+0000","last_fullsized":"2024-09-21T13:53:02.909144+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:51:50.211279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:50.211279+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:03.129559+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"61'1","reported_seq":234,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.913497+0000","last_change":"2024-09-21T13:52:16.585403+0000","last_active":"2024-09-21T13:53:02.913497+0000","last_peered":"2024-09-21T13:53:02.913497+0000","last_clean":"2024-09-21T13:53:02.913497+0000","last_became_active":"2024-09-21T13:52:04.315788+0000","last_became_peered":"2024-09-21T13:52:04.315788+0000","last_unstale":"2024-09-21T13:53:02.913497+0000","last_undegraded":"2024-09-21T13:53:02.913497+0000","last_fullsized":"2024-09-21T13:53:02.913497+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:52:16.585323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:52:16.585323+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:28.935922+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"96'9","reported_seq":207,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.914797+0000","last_change":"2024-09-21T13:52:51.443040+0000","last_active":"2024-09-21T13:53:02.914797+0000","last_peered":"2024-09-21T13:53:02.914797+0000","last_clean":"2024-09-21T13:53:02.914797+0000","last_became_active":"2024-09-21T13:50:29.693557+0000","last_became_peered":"2024-09-21T13:50:29.693557+0000","last_unstale":"2024-09-21T13:53:02.914797+0000","last_undegraded":"2024-09-21T13:53:02.914797+0000","last_fullsized":"2024-09-21T13:53:02.914797+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'9","last_scrub_stamp":"2024-09-21T13:52:51.442930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:52:51.442930+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:04.534122+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":145,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.907313+0000","last_change":"2024-09-21T13:52:04.319323+0000","last_active":"2024-09-21T13:53:02.907313+0000","last_peered":"2024-09-21T13:53:02.907313+0000","last_clean":"2024-09-21T13:53:02.907313+0000","last_became_active":"2024-09-21T13:52:04.319034+0000","last_became_peered":"2024-09-21T13:52:04.319034+0000","last_unstale":"2024-09-21T13:53:02.907313+0000","last_undegraded":"2024-09-21T13:53:02.907313+0000","last_fullsized":"2024-09-21T13:53:02.907313+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"155'13657","reported_seq":13884,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:03.348111+0000","last_change":"2024-09-21T13:52:16.618637+0000","last_active":"2024-09-21T13:53:03.348111+0000","last_peered":"2024-09-21T13:53:03.348111+0000","last_clean":"2024-09-21T13:53:03.348111+0000","last_became_active":"2024-09-21T13:50:07.423702+0000","last_became_peered":"2024-09-21T13:50:07.423702+0000","last_unstale":"2024-09-21T13:53:03.348111+0000","last_undegraded":"2024-09-21T13:53:03.348111+0000","last_fullsized":"2024-09-21T13:53:03.348111+0000","mapping_epoch":83,"log_start":"116'7800","ondisk_log_start":"116'7800","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'10945","last_scrub_stamp":"2024-09-21T13:52:16.618486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:52:16.618486+0000","objects_scrubbed":6221,"log_size":5857,"log_dups_size":0,"ondisk_log_size":5857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:43.545327+0000","scrub_duration":1967,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226164736,"num_objects":3451,"num_object_clones":0,"num_object_copies":10353,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13657,"num_write_kb":547456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"68'19","reported_seq":198,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.913002+0000","last_change":"2024-09-21T13:50:32.514267+0000","last_active":"2024-09-21T13:53:02.913002+0000","last_peered":"2024-09-21T13:53:02.913002+0000","last_clean":"2024-09-21T13:53:02.913002+0000","last_became_active":"2024-09-21T13:50:29.690185+0000","last_became_peered":"2024-09-21T13:50:29.690185+0000","last_unstale":"2024-09-21T13:53:02.913002+0000","last_undegraded":"2024-09-21T13:53:02.913002+0000","last_fullsized":"2024-09-21T13:53:02.913002+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-21T13:50:32.514210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:32.514210+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:34.599292+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":212,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.910094+0000","last_change":"2024-09-21T13:52:04.322289+0000","last_active":"2024-09-21T13:53:02.910094+0000","last_peered":"2024-09-21T13:53:02.910094+0000","last_clean":"2024-09-21T13:53:02.910094+0000","last_became_active":"2024-09-21T13:52:04.321656+0000","last_became_peered":"2024-09-21T13:52:04.321656+0000","last_unstale":"2024-09-21T13:53:02.910094+0000","last_undegraded":"2024-09-21T13:53:02.910094+0000","last_fullsized":"2024-09-21T13:53:02.910094+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:04.353911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":201,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.914057+0000","last_change":"2024-09-21T13:51:12.939488+0000","last_active":"2024-09-21T13:53:02.914057+0000","last_peered":"2024-09-21T13:53:02.914057+0000","last_clean":"2024-09-21T13:53:02.914057+0000","last_became_active":"2024-09-21T13:49:02.061187+0000","last_became_peered":"2024-09-21T13:49:02.061187+0000","last_unstale":"2024-09-21T13:53:02.914057+0000","last_undegraded":"2024-09-21T13:53:02.914057+0000","last_fullsized":"2024-09-21T13:53:02.914057+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:12.939444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:12.939444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:31.497225+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":200,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.912910+0000","last_change":"2024-09-21T13:52:04.321133+0000","last_active":"2024-09-21T13:53:02.912910+0000","last_peered":"2024-09-21T13:53:02.912910+0000","last_clean":"2024-09-21T13:53:02.912910+0000","last_became_active":"2024-09-21T13:52:04.320826+0000","last_became_peered":"2024-09-21T13:52:04.320826+0000","last_unstale":"2024-09-21T13:53:02.912910+0000","last_undegraded":"2024-09-21T13:53:02.912910+0000","last_fullsized":"2024-09-21T13:53:02.912910+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:06.341004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"155'13722","reported_seq":13929,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:03.386457+0000","last_change":"2024-09-21T13:52:49.375801+0000","last_active":"2024-09-21T13:53:03.386457+0000","last_peered":"2024-09-21T13:53:03.386457+0000","last_clean":"2024-09-21T13:53:03.386457+0000","last_became_active":"2024-09-21T13:50:07.421666+0000","last_became_peered":"2024-09-21T13:50:07.421666+0000","last_unstale":"2024-09-21T13:53:03.386457+0000","last_undegraded":"2024-09-21T13:53:03.386457+0000","last_fullsized":"2024-09-21T13:53:03.386457+0000","mapping_epoch":83,"log_start":"114'7400","ondisk_log_start":"114'7400","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"149'12949","last_scrub_stamp":"2024-09-21T13:52:49.375671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:52:49.375671+0000","objects_scrubbed":4241,"log_size":6322,"log_dups_size":0,"ondisk_log_size":6322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:54.416521+0000","scrub_duration":1855,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":225312768,"num_objects":3438,"num_object_clones":0,"num_object_copies":10314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13722,"num_write_kb":549120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"132'30","reported_seq":308,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.909988+0000","last_change":"2024-09-21T13:52:04.322738+0000","last_active":"2024-09-21T13:53:02.909988+0000","last_peered":"2024-09-21T13:53:02.909988+0000","last_clean":"2024-09-21T13:53:02.909988+0000","last_became_active":"2024-09-21T13:52:04.319431+0000","last_became_peered":"2024-09-21T13:52:04.319431+0000","last_unstale":"2024-09-21T13:53:02.909988+0000","last_undegraded":"2024-09-21T13:53:02.909988+0000","last_fullsized":"2024-09-21T13:53:02.909988+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:07.363362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":201,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.899851+0000","last_change":"2024-09-21T13:48:58.055640+0000","last_active":"2024-09-21T13:53:02.899851+0000","last_peered":"2024-09-21T13:53:02.899851+0000","last_clean":"2024-09-21T13:53:02.899851+0000","last_became_active":"2024-09-21T13:48:58.055366+0000","last_became_peered":"2024-09-21T13:48:58.055366+0000","last_unstale":"2024-09-21T13:53:02.899851+0000","last_undegraded":"2024-09-21T13:53:02.899851+0000","last_fullsized":"2024-09-21T13:53:02.899851+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":227,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.913209+0000","last_change":"2024-09-21T13:52:51.445882+0000","last_active":"2024-09-21T13:53:02.913209+0000","last_peered":"2024-09-21T13:53:02.913209+0000","last_clean":"2024-09-21T13:53:02.913209+0000","last_became_active":"2024-09-21T13:48:57.046246+0000","last_became_peered":"2024-09-21T13:48:57.046246+0000","last_unstale":"2024-09-21T13:53:02.913209+0000","last_undegraded":"2024-09-21T13:53:02.913209+0000","last_fullsized":"2024-09-21T13:53:02.913209+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-21T13:52:51.445812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:52:51.445812+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:18.734189+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.0015946669999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":142,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.908338+0000","last_change":"2024-09-21T13:51:24.187154+0000","last_active":"2024-09-21T13:53:02.908338+0000","last_peered":"2024-09-21T13:53:02.908338+0000","last_clean":"2024-09-21T13:53:02.908338+0000","last_became_active":"2024-09-21T13:50:29.688249+0000","last_became_peered":"2024-09-21T13:50:29.688249+0000","last_unstale":"2024-09-21T13:53:02.908338+0000","last_undegraded":"2024-09-21T13:53:02.908338+0000","last_fullsized":"2024-09-21T13:53:02.908338+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:24.186934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:24.186934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:33.380668+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":197,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.906406+0000","last_change":"2024-09-21T13:51:23.597164+0000","last_active":"2024-09-21T13:53:02.906406+0000","last_peered":"2024-09-21T13:53:02.906406+0000","last_clean":"2024-09-21T13:53:02.906406+0000","last_became_active":"2024-09-21T13:49:04.080466+0000","last_became_peered":"2024-09-21T13:49:04.080466+0000","last_unstale":"2024-09-21T13:53:02.906406+0000","last_undegraded":"2024-09-21T13:53:02.906406+0000","last_fullsized":"2024-09-21T13:53:02.906406+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:23.596879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:51:23.596879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"155'13838","reported_seq":13999,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:03.397459+0000","last_change":"2024-09-21T13:50:29.694240+0000","last_active":"2024-09-21T13:53:03.397459+0000","last_peered":"2024-09-21T13:53:03.397459+0000","last_clean":"2024-09-21T13:53:03.397459+0000","last_became_active":"2024-09-21T13:50:29.688167+0000","last_became_peered":"2024-09-21T13:50:29.688167+0000","last_unstale":"2024-09-21T13:53:03.397459+0000","last_undegraded":"2024-09-21T13:53:03.397459+0000","last_fullsized":"2024-09-21T13:53:03.397459+0000","mapping_epoch":94,"log_start":"123'9000","ondisk_log_start":"123'9000","created":83,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":4838,"log_dups_size":0,"ondisk_log_size":4838,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":228065280,"num_objects":3480,"num_object_clones":0,"num_object_copies":10440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13838,"num_write_kb":554176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"68'13","reported_seq":232,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.908726+0000","last_change":"2024-09-21T13:50:06.973399+0000","last_active":"2024-09-21T13:53:02.908726+0000","last_peered":"2024-09-21T13:53:02.908726+0000","last_clean":"2024-09-21T13:53:02.908726+0000","last_became_active":"2024-09-21T13:49:00.053323+0000","last_became_peered":"2024-09-21T13:49:00.053323+0000","last_unstale":"2024-09-21T13:53:02.908726+0000","last_undegraded":"2024-09-21T13:53:02.908726+0000","last_fullsized":"2024-09-21T13:53:02.908726+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'13","last_scrub_stamp":"2024-09-21T13:50:06.973336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.973336+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:36.427623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":229,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.913409+0000","last_change":"2024-09-21T13:52:22.318613+0000","last_active":"2024-09-21T13:53:02.913409+0000","last_peered":"2024-09-21T13:53:02.913409+0000","last_clean":"2024-09-21T13:53:02.913409+0000","last_became_active":"2024-09-21T13:52:04.320581+0000","last_became_peered":"2024-09-21T13:52:04.320581+0000","last_unstale":"2024-09-21T13:53:02.913409+0000","last_undegraded":"2024-09-21T13:53:02.913409+0000","last_fullsized":"2024-09-21T13:53:02.913409+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:22.318545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:52:22.318545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:24.469330+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":126,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-21T13:53:01.907406+0000","last_change":"2024-09-21T13:52:04.316578+0000","last_active":"2024-09-21T13:53:01.907406+0000","last_peered":"2024-09-21T13:53:01.907406+0000","last_clean":"2024-09-21T13:53:01.907406+0000","last_became_active":"2024-09-21T13:52:04.316171+0000","last_became_peered":"2024-09-21T13:52:04.316171+0000","last_unstale":"2024-09-21T13:53:01.907406+0000","last_undegraded":"2024-09-21T13:53:01.907406+0000","last_fullsized":"2024-09-21T13:53:01.907406+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:49:56.756268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:49:56.756268+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":70,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-21T13:53:01.910160+0000","last_change":"2024-09-21T13:52:04.320997+0000","last_active":"2024-09-21T13:53:01.910160+0000","last_peered":"2024-09-21T13:53:01.910160+0000","last_clean":"2024-09-21T13:53:01.910160+0000","last_became_active":"2024-09-21T13:52:04.320701+0000","last_became_peered":"2024-09-21T13:52:04.320701+0000","last_unstale":"2024-09-21T13:53:01.910160+0000","last_undegraded":"2024-09-21T13:53:01.910160+0000","last_fullsized":"2024-09-21T13:53:01.910160+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":205,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.910589+0000","last_change":"2024-09-21T13:51:12.212558+0000","last_active":"2024-09-21T13:53:02.910589+0000","last_peered":"2024-09-21T13:53:02.910589+0000","last_clean":"2024-09-21T13:53:02.910589+0000","last_became_active":"2024-09-21T13:49:04.431035+0000","last_became_peered":"2024-09-21T13:49:04.431035+0000","last_unstale":"2024-09-21T13:53:02.910589+0000","last_undegraded":"2024-09-21T13:53:02.910589+0000","last_fullsized":"2024-09-21T13:53:02.910589+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:12.212391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:51:12.212391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"155'13662","reported_seq":23777,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:03.426331+0000","last_change":"2024-09-21T13:52:43.960019+0000","last_active":"2024-09-21T13:53:03.426331+0000","last_peered":"2024-09-21T13:53:03.426331+0000","last_clean":"2024-09-21T13:53:03.426331+0000","last_became_active":"2024-09-21T13:52:43.938223+0000","last_became_peered":"2024-09-21T13:52:43.938223+0000","last_unstale":"2024-09-21T13:53:03.426331+0000","last_undegraded":"2024-09-21T13:53:03.426331+0000","last_fullsized":"2024-09-21T13:53:03.426331+0000","mapping_epoch":147,"log_start":"120'8400","ondisk_log_start":"120'8400","created":83,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":5262,"log_dups_size":0,"ondisk_log_size":5262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222822400,"num_objects":3400,"num_object_clones":0,"num_object_copies":10200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13662,"num_write_kb":545984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4971,"num_bytes_recovered":325779456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"68'12","reported_seq":255,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.912969+0000","last_change":"2024-09-21T13:52:04.319341+0000","last_active":"2024-09-21T13:53:02.912969+0000","last_peered":"2024-09-21T13:53:02.912969+0000","last_clean":"2024-09-21T13:53:02.912969+0000","last_became_active":"2024-09-21T13:52:04.318875+0000","last_became_peered":"2024-09-21T13:52:04.318875+0000","last_unstale":"2024-09-21T13:53:02.912969+0000","last_undegraded":"2024-09-21T13:53:02.912969+0000","last_fullsized":"2024-09-21T13:53:02.912969+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'12","last_scrub_stamp":"2024-09-21T13:50:29.438468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:29.438468+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:08.675996+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":142,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.899813+0000","last_change":"2024-09-21T13:50:30.020722+0000","last_active":"2024-09-21T13:53:02.899813+0000","last_peered":"2024-09-21T13:53:02.899813+0000","last_clean":"2024-09-21T13:53:02.899813+0000","last_became_active":"2024-09-21T13:50:29.688352+0000","last_became_peered":"2024-09-21T13:50:29.688352+0000","last_unstale":"2024-09-21T13:53:02.899813+0000","last_undegraded":"2024-09-21T13:53:02.899813+0000","last_fullsized":"2024-09-21T13:53:02.899813+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:30.020656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:30.020656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":461,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.908178+0000","last_change":"2024-09-21T13:51:21.271997+0000","last_active":"2024-09-21T13:53:02.908178+0000","last_peered":"2024-09-21T13:53:02.908178+0000","last_clean":"2024-09-21T13:53:02.908178+0000","last_became_active":"2024-09-21T13:48:57.046084+0000","last_became_peered":"2024-09-21T13:48:57.046084+0000","last_unstale":"2024-09-21T13:53:02.908178+0000","last_undegraded":"2024-09-21T13:53:02.908178+0000","last_fullsized":"2024-09-21T13:53:02.908178+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-21T13:51:21.271917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:51:21.271917+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00047048000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":240,"num_read_kb":235,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":201,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.913707+0000","last_change":"2024-09-21T13:51:53.518645+0000","last_active":"2024-09-21T13:53:02.913707+0000","last_peered":"2024-09-21T13:53:02.913707+0000","last_clean":"2024-09-21T13:53:02.913707+0000","last_became_active":"2024-09-21T13:49:02.066641+0000","last_became_peered":"2024-09-21T13:49:02.066641+0000","last_unstale":"2024-09-21T13:53:02.913707+0000","last_undegraded":"2024-09-21T13:53:02.913707+0000","last_fullsized":"2024-09-21T13:53:02.913707+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:53.518567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:53.518567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":205,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.908656+0000","last_change":"2024-09-21T13:52:21.332066+0000","last_active":"2024-09-21T13:53:02.908656+0000","last_peered":"2024-09-21T13:53:02.908656+0000","last_clean":"2024-09-21T13:53:02.908656+0000","last_became_active":"2024-09-21T13:49:04.078372+0000","last_became_peered":"2024-09-21T13:49:04.078372+0000","last_unstale":"2024-09-21T13:53:02.908656+0000","last_undegraded":"2024-09-21T13:53:02.908656+0000","last_fullsized":"2024-09-21T13:53:02.908656+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:21.331792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:21.331792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:44.913330+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"155'13751","reported_seq":13911,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:03.418076+0000","last_change":"2024-09-21T13:50:29.699865+0000","last_active":"2024-09-21T13:53:03.418076+0000","last_peered":"2024-09-21T13:53:03.418076+0000","last_clean":"2024-09-21T13:53:03.418076+0000","last_became_active":"2024-09-21T13:50:29.694886+0000","last_became_peered":"2024-09-21T13:50:29.694886+0000","last_unstale":"2024-09-21T13:53:03.418076+0000","last_undegraded":"2024-09-21T13:53:03.418076+0000","last_fullsized":"2024-09-21T13:53:03.418076+0000","mapping_epoch":94,"log_start":"120'8500","ondisk_log_start":"120'8500","created":83,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":5251,"log_dups_size":0,"ondisk_log_size":5251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":225640448,"num_objects":3443,"num_object_clones":0,"num_object_copies":10329,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13751,"num_write_kb":550208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"68'16","reported_seq":285,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.914754+0000","last_change":"2024-09-21T13:52:19.452239+0000","last_active":"2024-09-21T13:53:02.914754+0000","last_peered":"2024-09-21T13:53:02.914754+0000","last_clean":"2024-09-21T13:53:02.914754+0000","last_became_active":"2024-09-21T13:49:00.426867+0000","last_became_peered":"2024-09-21T13:49:00.426867+0000","last_unstale":"2024-09-21T13:53:02.914754+0000","last_undegraded":"2024-09-21T13:53:02.914754+0000","last_fullsized":"2024-09-21T13:53:02.914754+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'16","last_scrub_stamp":"2024-09-21T13:52:19.452163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:52:19.452163+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:46.305610+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":209,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.907150+0000","last_change":"2024-09-21T13:50:09.208053+0000","last_active":"2024-09-21T13:53:02.907150+0000","last_peered":"2024-09-21T13:53:02.907150+0000","last_clean":"2024-09-21T13:53:02.907150+0000","last_became_active":"2024-09-21T13:48:58.052659+0000","last_became_peered":"2024-09-21T13:48:58.052659+0000","last_unstale":"2024-09-21T13:53:02.907150+0000","last_undegraded":"2024-09-21T13:53:02.907150+0000","last_fullsized":"2024-09-21T13:53:02.907150+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:09.208003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:09.208003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"75'79","reported_seq":259,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.908241+0000","last_change":"2024-09-21T13:50:27.096470+0000","last_active":"2024-09-21T13:53:02.908241+0000","last_peered":"2024-09-21T13:53:02.908241+0000","last_clean":"2024-09-21T13:53:02.908241+0000","last_became_active":"2024-09-21T13:48:48.454646+0000","last_became_peered":"2024-09-21T13:48:48.454646+0000","last_unstale":"2024-09-21T13:53:02.908241+0000","last_undegraded":"2024-09-21T13:53:02.908241+0000","last_fullsized":"2024-09-21T13:53:02.908241+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:50:27.096413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:50:27.096413+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":204,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.908637+0000","last_change":"2024-09-21T13:52:04.320948+0000","last_active":"2024-09-21T13:53:02.908637+0000","last_peered":"2024-09-21T13:53:02.908637+0000","last_clean":"2024-09-21T13:53:02.908637+0000","last_became_active":"2024-09-21T13:52:04.320627+0000","last_became_peered":"2024-09-21T13:52:04.320627+0000","last_unstale":"2024-09-21T13:53:02.908637+0000","last_undegraded":"2024-09-21T13:53:02.908637+0000","last_fullsized":"2024-09-21T13:53:02.908637+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:42.557761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":197,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.913563+0000","last_change":"2024-09-21T13:50:25.560928+0000","last_active":"2024-09-21T13:53:02.913563+0000","last_peered":"2024-09-21T13:53:02.913563+0000","last_clean":"2024-09-21T13:53:02.913563+0000","last_became_active":"2024-09-21T13:49:04.078091+0000","last_became_peered":"2024-09-21T13:49:04.078091+0000","last_unstale":"2024-09-21T13:53:02.913563+0000","last_undegraded":"2024-09-21T13:53:02.913563+0000","last_fullsized":"2024-09-21T13:53:02.913563+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:25.560863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:25.560863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:40.548062+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"155'13673","reported_seq":13958,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:03.321702+0000","last_change":"2024-09-21T13:52:50.770818+0000","last_active":"2024-09-21T13:53:03.321702+0000","last_peered":"2024-09-21T13:53:03.321702+0000","last_clean":"2024-09-21T13:53:03.321702+0000","last_became_active":"2024-09-21T13:50:07.422466+0000","last_became_peered":"2024-09-21T13:50:07.422466+0000","last_unstale":"2024-09-21T13:53:03.321702+0000","last_undegraded":"2024-09-21T13:53:03.321702+0000","last_fullsized":"2024-09-21T13:53:03.321702+0000","mapping_epoch":83,"log_start":"114'7400","ondisk_log_start":"114'7400","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'12965","last_scrub_stamp":"2024-09-21T13:52:50.770670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:52:50.770670+0000","objects_scrubbed":4108,"log_size":6273,"log_dups_size":0,"ondisk_log_size":6273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:56.210477+0000","scrub_duration":1333,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220200984,"num_objects":3361,"num_object_clones":0,"num_object_copies":10083,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3361,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13673,"num_write_kb":545025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"68'11","reported_seq":216,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.900364+0000","last_change":"2024-09-21T13:49:00.054827+0000","last_active":"2024-09-21T13:53:02.900364+0000","last_peered":"2024-09-21T13:53:02.900364+0000","last_clean":"2024-09-21T13:53:02.900364+0000","last_became_active":"2024-09-21T13:49:00.053268+0000","last_became_peered":"2024-09-21T13:49:00.053268+0000","last_unstale":"2024-09-21T13:53:02.900364+0000","last_undegraded":"2024-09-21T13:53:02.900364+0000","last_fullsized":"2024-09-21T13:53:02.900364+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":220,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.900038+0000","last_change":"2024-09-21T13:52:48.165210+0000","last_active":"2024-09-21T13:53:02.900038+0000","last_peered":"2024-09-21T13:53:02.900038+0000","last_clean":"2024-09-21T13:53:02.900038+0000","last_became_active":"2024-09-21T13:52:04.318345+0000","last_became_peered":"2024-09-21T13:52:04.318345+0000","last_unstale":"2024-09-21T13:53:02.900038+0000","last_undegraded":"2024-09-21T13:53:02.900038+0000","last_fullsized":"2024-09-21T13:53:02.900038+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:48.165145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:52:48.165145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:03.774580+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":209,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.913531+0000","last_change":"2024-09-21T13:51:46.503640+0000","last_active":"2024-09-21T13:53:02.913531+0000","last_peered":"2024-09-21T13:53:02.913531+0000","last_clean":"2024-09-21T13:53:02.913531+0000","last_became_active":"2024-09-21T13:49:02.061901+0000","last_became_peered":"2024-09-21T13:49:02.061901+0000","last_unstale":"2024-09-21T13:53:02.913531+0000","last_undegraded":"2024-09-21T13:53:02.913531+0000","last_fullsized":"2024-09-21T13:53:02.913531+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:46.503576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:46.503576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":197,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.904611+0000","last_change":"2024-09-21T13:51:51.814259+0000","last_active":"2024-09-21T13:53:02.904611+0000","last_peered":"2024-09-21T13:53:02.904611+0000","last_clean":"2024-09-21T13:53:02.904611+0000","last_became_active":"2024-09-21T13:49:04.431268+0000","last_became_peered":"2024-09-21T13:49:04.431268+0000","last_unstale":"2024-09-21T13:53:02.904611+0000","last_undegraded":"2024-09-21T13:53:02.904611+0000","last_fullsized":"2024-09-21T13:53:02.904611+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:51.814200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:51:51.814200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:09.956776+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"155'13662","reported_seq":19041,"reported_epoch":155,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-21T13:53:03.637681+0000","last_change":"2024-09-21T13:52:42.230831+0000","last_active":"2024-09-21T13:53:03.637681+0000","last_peered":"2024-09-21T13:53:03.637681+0000","last_clean":"2024-09-21T13:50:42.883597+0000","last_became_active":"2024-09-21T13:52:04.974475+0000","last_became_peered":"2024-09-21T13:52:04.974475+0000","last_unstale":"2024-09-21T13:53:03.637681+0000","last_undegraded":"2024-09-21T13:52:04.880075+0000","last_fullsized":"2024-09-21T13:52:04.865259+0000","mapping_epoch":131,"log_start":"118'8088","ondisk_log_start":"118'8088","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":5574,"log_dups_size":0,"ondisk_log_size":5574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218103808,"num_objects":3328,"num_object_clones":0,"num_object_copies":9984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13662,"num_write_kb":543680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2607,"num_bytes_recovered":170852352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[{"shards":"0,5","objects":3328}],"blocked_by":[],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"68'17","reported_seq":244,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.900530+0000","last_change":"2024-09-21T13:52:45.176950+0000","last_active":"2024-09-21T13:53:02.900530+0000","last_peered":"2024-09-21T13:53:02.900530+0000","last_clean":"2024-09-21T13:53:02.900530+0000","last_became_active":"2024-09-21T13:52:04.313218+0000","last_became_peered":"2024-09-21T13:52:04.313218+0000","last_unstale":"2024-09-21T13:53:02.900530+0000","last_undegraded":"2024-09-21T13:53:02.900530+0000","last_fullsized":"2024-09-21T13:53:02.900530+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'17","last_scrub_stamp":"2024-09-21T13:52:45.176873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:52:45.176873+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:03.832378+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":217,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.914367+0000","last_change":"2024-09-21T13:52:45.444445+0000","last_active":"2024-09-21T13:53:02.914367+0000","last_peered":"2024-09-21T13:53:02.914367+0000","last_clean":"2024-09-21T13:53:02.914367+0000","last_became_active":"2024-09-21T13:48:58.049142+0000","last_became_peered":"2024-09-21T13:48:58.049142+0000","last_unstale":"2024-09-21T13:53:02.914367+0000","last_undegraded":"2024-09-21T13:53:02.914367+0000","last_fullsized":"2024-09-21T13:53:02.914367+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:45.444381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:52:45.444381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:54.348542+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":193,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.908764+0000","last_change":"2024-09-21T13:49:02.067811+0000","last_active":"2024-09-21T13:53:02.908764+0000","last_peered":"2024-09-21T13:53:02.908764+0000","last_clean":"2024-09-21T13:53:02.908764+0000","last_became_active":"2024-09-21T13:49:02.067472+0000","last_became_peered":"2024-09-21T13:49:02.067472+0000","last_unstale":"2024-09-21T13:53:02.908764+0000","last_undegraded":"2024-09-21T13:53:02.908764+0000","last_fullsized":"2024-09-21T13:53:02.908764+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:02.701863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":70,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-21T13:53:01.908241+0000","last_change":"2024-09-21T13:52:04.316322+0000","last_active":"2024-09-21T13:53:01.908241+0000","last_peered":"2024-09-21T13:53:01.908241+0000","last_clean":"2024-09-21T13:53:01.908241+0000","last_became_active":"2024-09-21T13:52:04.315920+0000","last_became_peered":"2024-09-21T13:52:04.315920+0000","last_unstale":"2024-09-21T13:53:01.908241+0000","last_undegraded":"2024-09-21T13:53:01.908241+0000","last_fullsized":"2024-09-21T13:53:01.908241+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"155'13383","reported_seq":13546,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:03.845395+0000","last_change":"2024-09-21T13:50:29.691429+0000","last_active":"2024-09-21T13:53:03.845395+0000","last_peered":"2024-09-21T13:53:03.845395+0000","last_clean":"2024-09-21T13:53:03.845395+0000","last_became_active":"2024-09-21T13:50:29.687898+0000","last_became_peered":"2024-09-21T13:50:29.687898+0000","last_unstale":"2024-09-21T13:53:03.845395+0000","last_undegraded":"2024-09-21T13:53:03.845395+0000","last_fullsized":"2024-09-21T13:53:03.845395+0000","mapping_epoch":94,"log_start":"123'8700","ondisk_log_start":"123'8700","created":83,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":4683,"log_dups_size":0,"ondisk_log_size":4683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211222528,"num_objects":3223,"num_object_clones":0,"num_object_copies":9669,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13383,"num_write_kb":531392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"68'10","reported_seq":220,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.900194+0000","last_change":"2024-09-21T13:51:57.155811+0000","last_active":"2024-09-21T13:53:02.900194+0000","last_peered":"2024-09-21T13:53:02.900194+0000","last_clean":"2024-09-21T13:53:02.900194+0000","last_became_active":"2024-09-21T13:49:00.059850+0000","last_became_peered":"2024-09-21T13:49:00.059850+0000","last_unstale":"2024-09-21T13:53:02.900194+0000","last_undegraded":"2024-09-21T13:53:02.900194+0000","last_fullsized":"2024-09-21T13:53:02.900194+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:51:57.155761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:57.155761+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:12.086769+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":209,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.907209+0000","last_change":"2024-09-21T13:51:28.565842+0000","last_active":"2024-09-21T13:53:02.907209+0000","last_peered":"2024-09-21T13:53:02.907209+0000","last_clean":"2024-09-21T13:53:02.907209+0000","last_became_active":"2024-09-21T13:48:58.058309+0000","last_became_peered":"2024-09-21T13:48:58.058309+0000","last_unstale":"2024-09-21T13:53:02.907209+0000","last_undegraded":"2024-09-21T13:53:02.907209+0000","last_fullsized":"2024-09-21T13:53:02.907209+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:28.565766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:51:28.565766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"132'10","reported_seq":100,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-21T13:53:01.907626+0000","last_change":"2024-09-21T13:52:04.320834+0000","last_active":"2024-09-21T13:53:01.907626+0000","last_peered":"2024-09-21T13:53:01.907626+0000","last_clean":"2024-09-21T13:53:01.907626+0000","last_became_active":"2024-09-21T13:52:04.320441+0000","last_became_peered":"2024-09-21T13:52:04.320441+0000","last_unstale":"2024-09-21T13:53:01.907626+0000","last_undegraded":"2024-09-21T13:53:01.907626+0000","last_fullsized":"2024-09-21T13:53:01.907626+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"68'1","reported_seq":212,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.900124+0000","last_change":"2024-09-21T13:52:04.314151+0000","last_active":"2024-09-21T13:53:02.900124+0000","last_peered":"2024-09-21T13:53:02.900124+0000","last_clean":"2024-09-21T13:53:02.900124+0000","last_became_active":"2024-09-21T13:52:04.313210+0000","last_became_peered":"2024-09-21T13:52:04.313210+0000","last_unstale":"2024-09-21T13:53:02.900124+0000","last_undegraded":"2024-09-21T13:53:02.900124+0000","last_fullsized":"2024-09-21T13:53:02.900124+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T13:50:41.000201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:41.000201+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"154'13569","reported_seq":23621,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.402011+0000","last_change":"2024-09-21T13:52:42.978682+0000","last_active":"2024-09-21T13:53:02.402011+0000","last_peered":"2024-09-21T13:53:02.402011+0000","last_clean":"2024-09-21T13:53:02.402011+0000","last_became_active":"2024-09-21T13:52:42.942510+0000","last_became_peered":"2024-09-21T13:52:42.942510+0000","last_unstale":"2024-09-21T13:53:02.402011+0000","last_undegraded":"2024-09-21T13:53:02.402011+0000","last_fullsized":"2024-09-21T13:53:02.402011+0000","mapping_epoch":146,"log_start":"116'7778","ondisk_log_start":"116'7778","created":83,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":5791,"log_dups_size":0,"ondisk_log_size":5791,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224722944,"num_objects":3429,"num_object_clones":0,"num_object_copies":10287,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13569,"num_write_kb":543936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4942,"num_bytes_recovered":323878912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"68'9","reported_seq":221,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.911091+0000","last_change":"2024-09-21T13:51:24.800926+0000","last_active":"2024-09-21T13:53:02.911091+0000","last_peered":"2024-09-21T13:53:02.911091+0000","last_clean":"2024-09-21T13:53:02.911091+0000","last_became_active":"2024-09-21T13:49:00.053016+0000","last_became_peered":"2024-09-21T13:49:00.053016+0000","last_unstale":"2024-09-21T13:53:02.911091+0000","last_undegraded":"2024-09-21T13:53:02.911091+0000","last_fullsized":"2024-09-21T13:53:02.911091+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:51:24.800844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:24.800844+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":209,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.913067+0000","last_change":"2024-09-21T13:50:38.445732+0000","last_active":"2024-09-21T13:53:02.913067+0000","last_peered":"2024-09-21T13:53:02.913067+0000","last_clean":"2024-09-21T13:53:02.913067+0000","last_became_active":"2024-09-21T13:48:58.049543+0000","last_became_peered":"2024-09-21T13:48:58.049543+0000","last_unstale":"2024-09-21T13:53:02.913067+0000","last_undegraded":"2024-09-21T13:53:02.913067+0000","last_fullsized":"2024-09-21T13:53:02.913067+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:38.445667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:38.445667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:35.568002+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":132,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.899524+0000","last_change":"2024-09-21T13:52:59.112648+0000","last_active":"2024-09-21T13:53:02.899524+0000","last_peered":"2024-09-21T13:53:02.899524+0000","last_clean":"2024-09-21T13:53:02.899524+0000","last_became_active":"2024-09-21T13:52:04.317429+0000","last_became_peered":"2024-09-21T13:52:04.317429+0000","last_unstale":"2024-09-21T13:53:02.899524+0000","last_undegraded":"2024-09-21T13:53:02.899524+0000","last_fullsized":"2024-09-21T13:53:02.899524+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:59.112602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:52:59.112602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:59.206422+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":153,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.908066+0000","last_change":"2024-09-21T13:52:51.590860+0000","last_active":"2024-09-21T13:53:02.908066+0000","last_peered":"2024-09-21T13:53:02.908066+0000","last_clean":"2024-09-21T13:53:02.908066+0000","last_became_active":"2024-09-21T13:52:04.321703+0000","last_became_peered":"2024-09-21T13:52:04.321703+0000","last_unstale":"2024-09-21T13:53:02.908066+0000","last_undegraded":"2024-09-21T13:53:02.908066+0000","last_fullsized":"2024-09-21T13:53:02.908066+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:51.590800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:51.590800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:01.244154+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.9","version":"155'13668","reported_seq":13831,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:03.473153+0000","last_change":"2024-09-21T13:50:29.690666+0000","last_active":"2024-09-21T13:53:03.473153+0000","last_peered":"2024-09-21T13:53:03.473153+0000","last_clean":"2024-09-21T13:53:03.473153+0000","last_became_active":"2024-09-21T13:50:29.687815+0000","last_became_peered":"2024-09-21T13:50:29.687815+0000","last_unstale":"2024-09-21T13:53:03.473153+0000","last_undegraded":"2024-09-21T13:53:03.473153+0000","last_fullsized":"2024-09-21T13:53:03.473153+0000","mapping_epoch":94,"log_start":"121'8600","ondisk_log_start":"121'8600","created":83,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":5068,"log_dups_size":0,"ondisk_log_size":5068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:32.358031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218628096,"num_objects":3336,"num_object_clones":0,"num_object_copies":10008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13668,"num_write_kb":544128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"68'19","reported_seq":236,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.914137+0000","last_change":"2024-09-21T13:50:03.769686+0000","last_active":"2024-09-21T13:53:02.914137+0000","last_peered":"2024-09-21T13:53:02.914137+0000","last_clean":"2024-09-21T13:53:02.914137+0000","last_became_active":"2024-09-21T13:49:00.425955+0000","last_became_peered":"2024-09-21T13:49:00.425955+0000","last_unstale":"2024-09-21T13:53:02.914137+0000","last_undegraded":"2024-09-21T13:53:02.914137+0000","last_fullsized":"2024-09-21T13:53:02.914137+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-21T13:50:03.769613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:03.769613+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:32.476093+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":201,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.910706+0000","last_change":"2024-09-21T13:48:58.053940+0000","last_active":"2024-09-21T13:53:02.910706+0000","last_peered":"2024-09-21T13:53:02.910706+0000","last_clean":"2024-09-21T13:53:02.910706+0000","last_became_active":"2024-09-21T13:48:58.053641+0000","last_became_peered":"2024-09-21T13:48:58.053641+0000","last_unstale":"2024-09-21T13:53:02.910706+0000","last_undegraded":"2024-09-21T13:53:02.910706+0000","last_fullsized":"2024-09-21T13:53:02.910706+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":148,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-21T13:53:01.907780+0000","last_change":"2024-09-21T13:52:04.321471+0000","last_active":"2024-09-21T13:53:01.907780+0000","last_peered":"2024-09-21T13:53:01.907780+0000","last_clean":"2024-09-21T13:53:01.907780+0000","last_became_active":"2024-09-21T13:52:04.321146+0000","last_became_peered":"2024-09-21T13:52:04.321146+0000","last_unstale":"2024-09-21T13:53:01.907780+0000","last_undegraded":"2024-09-21T13:53:01.907780+0000","last_fullsized":"2024-09-21T13:53:01.907780+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:40.850721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:40.850721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":208,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.908141+0000","last_change":"2024-09-21T13:52:04.323179+0000","last_active":"2024-09-21T13:53:02.908141+0000","last_peered":"2024-09-21T13:53:02.908141+0000","last_clean":"2024-09-21T13:53:02.908141+0000","last_became_active":"2024-09-21T13:52:04.322213+0000","last_became_peered":"2024-09-21T13:52:04.322213+0000","last_unstale":"2024-09-21T13:53:02.908141+0000","last_undegraded":"2024-09-21T13:53:02.908141+0000","last_fullsized":"2024-09-21T13:53:02.908141+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:35.067261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:35.067261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"155'13651","reported_seq":22899,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:03.831072+0000","last_change":"2024-09-21T13:53:01.956068+0000","last_active":"2024-09-21T13:53:03.831072+0000","last_peered":"2024-09-21T13:53:03.831072+0000","last_clean":"2024-09-21T13:53:03.831072+0000","last_became_active":"2024-09-21T13:53:01.911422+0000","last_became_peered":"2024-09-21T13:53:01.911422+0000","last_unstale":"2024-09-21T13:53:03.831072+0000","last_undegraded":"2024-09-21T13:53:03.831072+0000","last_fullsized":"2024-09-21T13:53:03.831072+0000","mapping_epoch":153,"log_start":"123'9000","ondisk_log_start":"123'9000","created":83,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":4651,"log_dups_size":0,"ondisk_log_size":4651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220004352,"num_objects":3357,"num_object_clones":0,"num_object_copies":10071,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13651,"num_write_kb":544256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4537,"num_bytes_recovered":297336832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"68'12","reported_seq":228,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.899902+0000","last_change":"2024-09-21T13:51:10.942190+0000","last_active":"2024-09-21T13:53:02.899902+0000","last_peered":"2024-09-21T13:53:02.899902+0000","last_clean":"2024-09-21T13:53:02.899902+0000","last_became_active":"2024-09-21T13:49:00.059720+0000","last_became_peered":"2024-09-21T13:49:00.059720+0000","last_unstale":"2024-09-21T13:53:02.899902+0000","last_undegraded":"2024-09-21T13:53:02.899902+0000","last_fullsized":"2024-09-21T13:53:02.899902+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'12","last_scrub_stamp":"2024-09-21T13:51:10.942105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:10.942105+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":209,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.908857+0000","last_change":"2024-09-21T13:50:29.100595+0000","last_active":"2024-09-21T13:53:02.908857+0000","last_peered":"2024-09-21T13:53:02.908857+0000","last_clean":"2024-09-21T13:53:02.908857+0000","last_became_active":"2024-09-21T13:48:58.057641+0000","last_became_peered":"2024-09-21T13:48:58.057641+0000","last_unstale":"2024-09-21T13:53:02.908857+0000","last_undegraded":"2024-09-21T13:53:02.908857+0000","last_fullsized":"2024-09-21T13:53:02.908857+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:29.100547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:29.100547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":113,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-21T13:53:01.907157+0000","last_change":"2024-09-21T13:52:04.322501+0000","last_active":"2024-09-21T13:53:01.907157+0000","last_peered":"2024-09-21T13:53:01.907157+0000","last_clean":"2024-09-21T13:53:01.907157+0000","last_became_active":"2024-09-21T13:52:04.322147+0000","last_became_peered":"2024-09-21T13:52:04.322147+0000","last_unstale":"2024-09-21T13:53:01.907157+0000","last_undegraded":"2024-09-21T13:53:01.907157+0000","last_fullsized":"2024-09-21T13:53:01.907157+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:02.761516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:02.761516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":189,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.906873+0000","last_change":"2024-09-21T13:49:04.078422+0000","last_active":"2024-09-21T13:53:02.906873+0000","last_peered":"2024-09-21T13:53:02.906873+0000","last_clean":"2024-09-21T13:53:02.906873+0000","last_became_active":"2024-09-21T13:49:04.078089+0000","last_became_peered":"2024-09-21T13:49:04.078089+0000","last_unstale":"2024-09-21T13:53:02.906873+0000","last_undegraded":"2024-09-21T13:53:02.906873+0000","last_fullsized":"2024-09-21T13:53:02.906873+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"155'13769","reported_seq":13925,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:03.500415+0000","last_change":"2024-09-21T13:50:07.422240+0000","last_active":"2024-09-21T13:53:03.500415+0000","last_peered":"2024-09-21T13:53:03.500415+0000","last_clean":"2024-09-21T13:53:03.500415+0000","last_became_active":"2024-09-21T13:50:07.421962+0000","last_became_peered":"2024-09-21T13:50:07.421962+0000","last_unstale":"2024-09-21T13:53:03.500415+0000","last_undegraded":"2024-09-21T13:53:03.500415+0000","last_fullsized":"2024-09-21T13:53:03.500415+0000","mapping_epoch":83,"log_start":"121'8700","ondisk_log_start":"121'8700","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":5069,"log_dups_size":0,"ondisk_log_size":5069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:37.396702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":225116160,"num_objects":3435,"num_object_clones":0,"num_object_copies":10305,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13769,"num_write_kb":550528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"68'15","reported_seq":230,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.912684+0000","last_change":"2024-09-21T13:50:04.445409+0000","last_active":"2024-09-21T13:53:02.912684+0000","last_peered":"2024-09-21T13:53:02.912684+0000","last_clean":"2024-09-21T13:53:02.912684+0000","last_became_active":"2024-09-21T13:49:00.425997+0000","last_became_peered":"2024-09-21T13:49:00.425997+0000","last_unstale":"2024-09-21T13:53:02.912684+0000","last_undegraded":"2024-09-21T13:53:02.912684+0000","last_fullsized":"2024-09-21T13:53:02.912684+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:50:04.445328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:04.445328+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:30.619277+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":221,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.909012+0000","last_change":"2024-09-21T13:50:08.119154+0000","last_active":"2024-09-21T13:53:02.909012+0000","last_peered":"2024-09-21T13:53:02.909012+0000","last_clean":"2024-09-21T13:53:02.909012+0000","last_became_active":"2024-09-21T13:48:58.057632+0000","last_became_peered":"2024-09-21T13:48:58.057632+0000","last_unstale":"2024-09-21T13:53:02.909012+0000","last_undegraded":"2024-09-21T13:53:02.909012+0000","last_fullsized":"2024-09-21T13:53:02.909012+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T13:50:08.119099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:08.119099+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"68'8","reported_seq":259,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.908191+0000","last_change":"2024-09-21T13:51:26.297643+0000","last_active":"2024-09-21T13:53:02.908191+0000","last_peered":"2024-09-21T13:53:02.908191+0000","last_clean":"2024-09-21T13:53:02.908191+0000","last_became_active":"2024-09-21T13:49:02.056167+0000","last_became_peered":"2024-09-21T13:49:02.056167+0000","last_unstale":"2024-09-21T13:53:02.908191+0000","last_undegraded":"2024-09-21T13:53:02.908191+0000","last_fullsized":"2024-09-21T13:53:02.908191+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:51:26.297550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:26.297550+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":205,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.912831+0000","last_change":"2024-09-21T13:52:54.314087+0000","last_active":"2024-09-21T13:53:02.912831+0000","last_peered":"2024-09-21T13:53:02.912831+0000","last_clean":"2024-09-21T13:53:02.912831+0000","last_became_active":"2024-09-21T13:49:04.432546+0000","last_became_peered":"2024-09-21T13:49:04.432546+0000","last_unstale":"2024-09-21T13:53:02.912831+0000","last_undegraded":"2024-09-21T13:53:02.912831+0000","last_fullsized":"2024-09-21T13:53:02.912831+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:54.314042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:54.314042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:05.575071+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":217,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.914228+0000","last_change":"2024-09-21T13:51:54.352313+0000","last_active":"2024-09-21T13:53:02.914228+0000","last_peered":"2024-09-21T13:53:02.914228+0000","last_clean":"2024-09-21T13:53:02.914228+0000","last_became_active":"2024-09-21T13:48:58.056080+0000","last_became_peered":"2024-09-21T13:48:58.056080+0000","last_unstale":"2024-09-21T13:53:02.914228+0000","last_undegraded":"2024-09-21T13:53:02.914228+0000","last_fullsized":"2024-09-21T13:53:02.914228+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:54.352244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:51:54.352244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:08.478082+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"68'6","reported_seq":206,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.912574+0000","last_change":"2024-09-21T13:49:00.427204+0000","last_active":"2024-09-21T13:53:02.912574+0000","last_peered":"2024-09-21T13:53:02.912574+0000","last_clean":"2024-09-21T13:53:02.912574+0000","last_became_active":"2024-09-21T13:49:00.426829+0000","last_became_peered":"2024-09-21T13:49:00.426829+0000","last_unstale":"2024-09-21T13:53:02.912574+0000","last_undegraded":"2024-09-21T13:53:02.912574+0000","last_fullsized":"2024-09-21T13:53:02.912574+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":209,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.913990+0000","last_change":"2024-09-21T13:51:55.488973+0000","last_active":"2024-09-21T13:53:02.913990+0000","last_peered":"2024-09-21T13:53:02.913990+0000","last_clean":"2024-09-21T13:53:02.913990+0000","last_became_active":"2024-09-21T13:49:02.062231+0000","last_became_peered":"2024-09-21T13:49:02.062231+0000","last_unstale":"2024-09-21T13:53:02.913990+0000","last_undegraded":"2024-09-21T13:53:02.913990+0000","last_fullsized":"2024-09-21T13:53:02.913990+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:55.488908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:55.488908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:07.596252+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":197,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.910527+0000","last_change":"2024-09-21T13:50:06.158899+0000","last_active":"2024-09-21T13:53:02.910527+0000","last_peered":"2024-09-21T13:53:02.910527+0000","last_clean":"2024-09-21T13:53:02.910527+0000","last_became_active":"2024-09-21T13:49:04.432507+0000","last_became_peered":"2024-09-21T13:49:04.432507+0000","last_unstale":"2024-09-21T13:53:02.910527+0000","last_undegraded":"2024-09-21T13:53:02.910527+0000","last_fullsized":"2024-09-21T13:53:02.910527+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.158822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.158822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"68'9","reported_seq":178,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.906553+0000","last_change":"2024-09-21T13:51:54.808758+0000","last_active":"2024-09-21T13:53:02.906553+0000","last_peered":"2024-09-21T13:53:02.906553+0000","last_clean":"2024-09-21T13:53:02.906553+0000","last_became_active":"2024-09-21T13:50:29.693589+0000","last_became_peered":"2024-09-21T13:50:29.693589+0000","last_unstale":"2024-09-21T13:53:02.906553+0000","last_undegraded":"2024-09-21T13:53:02.906553+0000","last_fullsized":"2024-09-21T13:53:02.906553+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:51:54.808559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:54.808559+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:18.303211+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"61'1","reported_seq":207,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.908271+0000","last_change":"2024-09-21T13:48:58.050948+0000","last_active":"2024-09-21T13:53:02.908271+0000","last_peered":"2024-09-21T13:53:02.908271+0000","last_clean":"2024-09-21T13:53:02.908271+0000","last_became_active":"2024-09-21T13:48:58.050645+0000","last_became_peered":"2024-09-21T13:48:58.050645+0000","last_unstale":"2024-09-21T13:53:02.908271+0000","last_undegraded":"2024-09-21T13:53:02.908271+0000","last_fullsized":"2024-09-21T13:53:02.908271+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":201,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.912333+0000","last_change":"2024-09-21T13:50:43.375425+0000","last_active":"2024-09-21T13:53:02.912333+0000","last_peered":"2024-09-21T13:53:02.912333+0000","last_clean":"2024-09-21T13:53:02.912333+0000","last_became_active":"2024-09-21T13:49:02.061775+0000","last_became_peered":"2024-09-21T13:49:02.061775+0000","last_unstale":"2024-09-21T13:53:02.912333+0000","last_undegraded":"2024-09-21T13:53:02.912333+0000","last_fullsized":"2024-09-21T13:53:02.912333+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:43.375381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:43.375381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":144,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-21T13:53:01.907705+0000","last_change":"2024-09-21T13:52:04.324521+0000","last_active":"2024-09-21T13:53:01.907705+0000","last_peered":"2024-09-21T13:53:01.907705+0000","last_clean":"2024-09-21T13:53:01.907705+0000","last_became_active":"2024-09-21T13:52:04.323549+0000","last_became_peered":"2024-09-21T13:52:04.323549+0000","last_unstale":"2024-09-21T13:53:01.907705+0000","last_undegraded":"2024-09-21T13:53:01.907705+0000","last_fullsized":"2024-09-21T13:53:01.907705+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:41.808746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:41.808746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"68'9","reported_seq":221,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.913398+0000","last_change":"2024-09-21T13:51:28.547052+0000","last_active":"2024-09-21T13:53:02.913398+0000","last_peered":"2024-09-21T13:53:02.913398+0000","last_clean":"2024-09-21T13:53:02.913398+0000","last_became_active":"2024-09-21T13:49:00.427766+0000","last_became_peered":"2024-09-21T13:49:00.427766+0000","last_unstale":"2024-09-21T13:53:02.913398+0000","last_undegraded":"2024-09-21T13:53:02.913398+0000","last_fullsized":"2024-09-21T13:53:02.913398+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:51:28.546958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:28.546958+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":142,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.908423+0000","last_change":"2024-09-21T13:50:30.076191+0000","last_active":"2024-09-21T13:53:02.908423+0000","last_peered":"2024-09-21T13:53:02.908423+0000","last_clean":"2024-09-21T13:53:02.908423+0000","last_became_active":"2024-09-21T13:50:29.690634+0000","last_became_peered":"2024-09-21T13:50:29.690634+0000","last_unstale":"2024-09-21T13:53:02.908423+0000","last_undegraded":"2024-09-21T13:53:02.908423+0000","last_fullsized":"2024-09-21T13:53:02.908423+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:30.076115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:30.076115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"132'10","reported_seq":272,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.904787+0000","last_change":"2024-09-21T13:52:04.319377+0000","last_active":"2024-09-21T13:53:02.904787+0000","last_peered":"2024-09-21T13:53:02.904787+0000","last_clean":"2024-09-21T13:53:02.904787+0000","last_became_active":"2024-09-21T13:52:04.318626+0000","last_became_peered":"2024-09-21T13:52:04.318626+0000","last_unstale":"2024-09-21T13:53:02.904787+0000","last_undegraded":"2024-09-21T13:53:02.904787+0000","last_fullsized":"2024-09-21T13:53:02.904787+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:50:29.249680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:29.249680+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":208,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.900436+0000","last_change":"2024-09-21T13:52:04.316179+0000","last_active":"2024-09-21T13:53:02.900436+0000","last_peered":"2024-09-21T13:53:02.900436+0000","last_clean":"2024-09-21T13:53:02.900436+0000","last_became_active":"2024-09-21T13:52:04.315865+0000","last_became_peered":"2024-09-21T13:52:04.315865+0000","last_unstale":"2024-09-21T13:53:02.900436+0000","last_undegraded":"2024-09-21T13:53:02.900436+0000","last_fullsized":"2024-09-21T13:53:02.900436+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:38.983326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:38.983326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"68'10","reported_seq":220,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.905041+0000","last_change":"2024-09-21T13:51:11.545546+0000","last_active":"2024-09-21T13:53:02.905041+0000","last_peered":"2024-09-21T13:53:02.905041+0000","last_clean":"2024-09-21T13:53:02.905041+0000","last_became_active":"2024-09-21T13:49:00.426071+0000","last_became_peered":"2024-09-21T13:49:00.426071+0000","last_unstale":"2024-09-21T13:53:02.905041+0000","last_undegraded":"2024-09-21T13:53:02.905041+0000","last_fullsized":"2024-09-21T13:53:02.905041+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:51:11.545458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:11.545458+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":220,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.908393+0000","last_change":"2024-09-21T13:52:04.318335+0000","last_active":"2024-09-21T13:53:02.908393+0000","last_peered":"2024-09-21T13:53:02.908393+0000","last_clean":"2024-09-21T13:53:02.908393+0000","last_became_active":"2024-09-21T13:52:04.318006+0000","last_became_peered":"2024-09-21T13:52:04.318006+0000","last_unstale":"2024-09-21T13:53:02.908393+0000","last_undegraded":"2024-09-21T13:53:02.908393+0000","last_fullsized":"2024-09-21T13:53:02.908393+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:41.162315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:41.162315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"68'8","reported_seq":251,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.913351+0000","last_change":"2024-09-21T13:49:02.067827+0000","last_active":"2024-09-21T13:53:02.913351+0000","last_peered":"2024-09-21T13:53:02.913351+0000","last_clean":"2024-09-21T13:53:02.913351+0000","last_became_active":"2024-09-21T13:49:02.067487+0000","last_became_peered":"2024-09-21T13:49:02.067487+0000","last_unstale":"2024-09-21T13:53:02.913351+0000","last_undegraded":"2024-09-21T13:53:02.913351+0000","last_fullsized":"2024-09-21T13:53:02.913351+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:02.411463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":142,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.908662+0000","last_change":"2024-09-21T13:50:33.050276+0000","last_active":"2024-09-21T13:53:02.908662+0000","last_peered":"2024-09-21T13:53:02.908662+0000","last_clean":"2024-09-21T13:53:02.908662+0000","last_became_active":"2024-09-21T13:50:29.695082+0000","last_became_peered":"2024-09-21T13:50:29.695082+0000","last_unstale":"2024-09-21T13:53:02.908662+0000","last_undegraded":"2024-09-21T13:53:02.908662+0000","last_fullsized":"2024-09-21T13:53:02.908662+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:33.050227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:33.050227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"68'11","reported_seq":224,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.899546+0000","last_change":"2024-09-21T13:50:02.673500+0000","last_active":"2024-09-21T13:53:02.899546+0000","last_peered":"2024-09-21T13:53:02.899546+0000","last_clean":"2024-09-21T13:53:02.899546+0000","last_became_active":"2024-09-21T13:49:00.055024+0000","last_became_peered":"2024-09-21T13:49:00.055024+0000","last_unstale":"2024-09-21T13:53:02.899546+0000","last_undegraded":"2024-09-21T13:53:02.899546+0000","last_fullsized":"2024-09-21T13:53:02.899546+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:50:02.673442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:02.673442+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":209,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.900315+0000","last_change":"2024-09-21T13:51:20.956291+0000","last_active":"2024-09-21T13:53:02.900315+0000","last_peered":"2024-09-21T13:53:02.900315+0000","last_clean":"2024-09-21T13:53:02.900315+0000","last_became_active":"2024-09-21T13:48:58.051541+0000","last_became_peered":"2024-09-21T13:48:58.051541+0000","last_unstale":"2024-09-21T13:53:02.900315+0000","last_undegraded":"2024-09-21T13:53:02.900315+0000","last_fullsized":"2024-09-21T13:53:02.900315+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:20.956226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:51:20.956226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":209,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.909015+0000","last_change":"2024-09-21T13:52:14.465671+0000","last_active":"2024-09-21T13:53:02.909015+0000","last_peered":"2024-09-21T13:53:02.909015+0000","last_clean":"2024-09-21T13:53:02.909015+0000","last_became_active":"2024-09-21T13:49:02.062868+0000","last_became_peered":"2024-09-21T13:49:02.062868+0000","last_unstale":"2024-09-21T13:53:02.909015+0000","last_undegraded":"2024-09-21T13:53:02.909015+0000","last_fullsized":"2024-09-21T13:53:02.909015+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:14.465434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:52:14.465434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:25.424869+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":197,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.904018+0000","last_change":"2024-09-21T13:51:24.549993+0000","last_active":"2024-09-21T13:53:02.904018+0000","last_peered":"2024-09-21T13:53:02.904018+0000","last_clean":"2024-09-21T13:53:02.904018+0000","last_became_active":"2024-09-21T13:49:04.082961+0000","last_became_peered":"2024-09-21T13:49:04.082961+0000","last_unstale":"2024-09-21T13:53:02.904018+0000","last_undegraded":"2024-09-21T13:53:02.904018+0000","last_fullsized":"2024-09-21T13:53:02.904018+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:24.549706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:51:24.549706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"68'9","reported_seq":232,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.909284+0000","last_change":"2024-09-21T13:52:10.447789+0000","last_active":"2024-09-21T13:53:02.909284+0000","last_peered":"2024-09-21T13:53:02.909284+0000","last_clean":"2024-09-21T13:53:02.909284+0000","last_became_active":"2024-09-21T13:52:04.320149+0000","last_became_peered":"2024-09-21T13:52:04.320149+0000","last_unstale":"2024-09-21T13:53:02.909284+0000","last_undegraded":"2024-09-21T13:53:02.909284+0000","last_fullsized":"2024-09-21T13:53:02.909284+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:52:10.447530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:52:10.447530+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:37.291172+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":201,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.910650+0000","last_change":"2024-09-21T13:48:58.058032+0000","last_active":"2024-09-21T13:53:02.910650+0000","last_peered":"2024-09-21T13:53:02.910650+0000","last_clean":"2024-09-21T13:53:02.910650+0000","last_became_active":"2024-09-21T13:48:58.057516+0000","last_became_peered":"2024-09-21T13:48:58.057516+0000","last_unstale":"2024-09-21T13:53:02.910650+0000","last_undegraded":"2024-09-21T13:53:02.910650+0000","last_fullsized":"2024-09-21T13:53:02.910650+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":201,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.906966+0000","last_change":"2024-09-21T13:50:39.411785+0000","last_active":"2024-09-21T13:53:02.906966+0000","last_peered":"2024-09-21T13:53:02.906966+0000","last_clean":"2024-09-21T13:53:02.906966+0000","last_became_active":"2024-09-21T13:49:02.071922+0000","last_became_peered":"2024-09-21T13:49:02.071922+0000","last_unstale":"2024-09-21T13:53:02.906966+0000","last_undegraded":"2024-09-21T13:53:02.906966+0000","last_fullsized":"2024-09-21T13:53:02.906966+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:39.411718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:39.411718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":134,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.912546+0000","last_change":"2024-09-21T13:50:29.689537+0000","last_active":"2024-09-21T13:53:02.912546+0000","last_peered":"2024-09-21T13:53:02.912546+0000","last_clean":"2024-09-21T13:53:02.912546+0000","last_became_active":"2024-09-21T13:50:29.689045+0000","last_became_peered":"2024-09-21T13:50:29.689045+0000","last_unstale":"2024-09-21T13:53:02.912546+0000","last_undegraded":"2024-09-21T13:53:02.912546+0000","last_fullsized":"2024-09-21T13:53:02.912546+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:09.575797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"68'11","reported_seq":224,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.904212+0000","last_change":"2024-09-21T13:50:31.329973+0000","last_active":"2024-09-21T13:53:02.904212+0000","last_peered":"2024-09-21T13:53:02.904212+0000","last_clean":"2024-09-21T13:53:02.904212+0000","last_became_active":"2024-09-21T13:49:00.426091+0000","last_became_peered":"2024-09-21T13:49:00.426091+0000","last_unstale":"2024-09-21T13:53:02.904212+0000","last_undegraded":"2024-09-21T13:53:02.904212+0000","last_fullsized":"2024-09-21T13:53:02.904212+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:50:31.329740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:31.329740+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":201,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.914021+0000","last_change":"2024-09-21T13:48:58.052281+0000","last_active":"2024-09-21T13:53:02.914021+0000","last_peered":"2024-09-21T13:53:02.914021+0000","last_clean":"2024-09-21T13:53:02.914021+0000","last_became_active":"2024-09-21T13:48:58.051974+0000","last_became_peered":"2024-09-21T13:48:58.051974+0000","last_unstale":"2024-09-21T13:53:02.914021+0000","last_undegraded":"2024-09-21T13:53:02.914021+0000","last_fullsized":"2024-09-21T13:53:02.914021+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:03.555777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":201,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.907920+0000","last_change":"2024-09-21T13:50:37.118637+0000","last_active":"2024-09-21T13:53:02.907920+0000","last_peered":"2024-09-21T13:53:02.907920+0000","last_clean":"2024-09-21T13:53:02.907920+0000","last_became_active":"2024-09-21T13:49:02.056006+0000","last_became_peered":"2024-09-21T13:49:02.056006+0000","last_unstale":"2024-09-21T13:53:02.907920+0000","last_undegraded":"2024-09-21T13:53:02.907920+0000","last_fullsized":"2024-09-21T13:53:02.907920+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:37.118594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:37.118594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":197,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.907112+0000","last_change":"2024-09-21T13:51:55.827109+0000","last_active":"2024-09-21T13:53:02.907112+0000","last_peered":"2024-09-21T13:53:02.907112+0000","last_clean":"2024-09-21T13:53:02.907112+0000","last_became_active":"2024-09-21T13:49:04.434034+0000","last_became_peered":"2024-09-21T13:49:04.434034+0000","last_unstale":"2024-09-21T13:53:02.907112+0000","last_undegraded":"2024-09-21T13:53:02.907112+0000","last_fullsized":"2024-09-21T13:53:02.907112+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:55.826922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:51:55.826922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"68'4","reported_seq":203,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.913058+0000","last_change":"2024-09-21T13:49:00.063374+0000","last_active":"2024-09-21T13:53:02.913058+0000","last_peered":"2024-09-21T13:53:02.913058+0000","last_clean":"2024-09-21T13:53:02.913058+0000","last_became_active":"2024-09-21T13:49:00.063066+0000","last_became_peered":"2024-09-21T13:49:00.063066+0000","last_unstale":"2024-09-21T13:53:02.913058+0000","last_undegraded":"2024-09-21T13:53:02.913058+0000","last_fullsized":"2024-09-21T13:53:02.913058+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:03.763727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":134,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.913674+0000","last_change":"2024-09-21T13:50:29.690572+0000","last_active":"2024-09-21T13:53:02.913674+0000","last_peered":"2024-09-21T13:53:02.913674+0000","last_clean":"2024-09-21T13:53:02.913674+0000","last_became_active":"2024-09-21T13:50:29.690269+0000","last_became_peered":"2024-09-21T13:50:29.690269+0000","last_unstale":"2024-09-21T13:53:02.913674+0000","last_undegraded":"2024-09-21T13:53:02.913674+0000","last_fullsized":"2024-09-21T13:53:02.913674+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:04.145682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:04.145682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:31.629620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":209,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.915780+0000","last_change":"2024-09-21T13:52:48.472414+0000","last_active":"2024-09-21T13:53:02.915780+0000","last_peered":"2024-09-21T13:53:02.915780+0000","last_clean":"2024-09-21T13:53:02.915780+0000","last_became_active":"2024-09-21T13:49:02.064789+0000","last_became_peered":"2024-09-21T13:49:02.064789+0000","last_unstale":"2024-09-21T13:53:02.915780+0000","last_undegraded":"2024-09-21T13:53:02.915780+0000","last_fullsized":"2024-09-21T13:53:02.915780+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:48.472373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:52:48.472373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:51.057252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":208,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.908245+0000","last_change":"2024-09-21T13:52:04.322055+0000","last_active":"2024-09-21T13:53:02.908245+0000","last_peered":"2024-09-21T13:53:02.908245+0000","last_clean":"2024-09-21T13:53:02.908245+0000","last_became_active":"2024-09-21T13:52:04.321667+0000","last_became_peered":"2024-09-21T13:52:04.321667+0000","last_unstale":"2024-09-21T13:53:02.908245+0000","last_undegraded":"2024-09-21T13:53:02.908245+0000","last_fullsized":"2024-09-21T13:53:02.908245+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:39.151320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:39.151320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":205,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.908234+0000","last_change":"2024-09-21T13:52:27.409588+0000","last_active":"2024-09-21T13:53:02.908234+0000","last_peered":"2024-09-21T13:53:02.908234+0000","last_clean":"2024-09-21T13:53:02.908234+0000","last_became_active":"2024-09-21T13:49:04.075562+0000","last_became_peered":"2024-09-21T13:49:04.075562+0000","last_unstale":"2024-09-21T13:53:02.908234+0000","last_undegraded":"2024-09-21T13:53:02.908234+0000","last_fullsized":"2024-09-21T13:53:02.908234+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:27.409408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:27.409408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:52.946524+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":201,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.913325+0000","last_change":"2024-09-21T13:48:58.055791+0000","last_active":"2024-09-21T13:53:02.913325+0000","last_peered":"2024-09-21T13:53:02.913325+0000","last_clean":"2024-09-21T13:53:02.913325+0000","last_became_active":"2024-09-21T13:48:58.055497+0000","last_became_peered":"2024-09-21T13:48:58.055497+0000","last_unstale":"2024-09-21T13:53:02.913325+0000","last_undegraded":"2024-09-21T13:53:02.913325+0000","last_fullsized":"2024-09-21T13:53:02.913325+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"68'11","reported_seq":254,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.914664+0000","last_change":"2024-09-21T13:52:04.321178+0000","last_active":"2024-09-21T13:53:02.914664+0000","last_peered":"2024-09-21T13:53:02.914664+0000","last_clean":"2024-09-21T13:53:02.914664+0000","last_became_active":"2024-09-21T13:52:04.320647+0000","last_became_peered":"2024-09-21T13:52:04.320647+0000","last_unstale":"2024-09-21T13:53:02.914664+0000","last_undegraded":"2024-09-21T13:53:02.914664+0000","last_fullsized":"2024-09-21T13:53:02.914664+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:50:35.790090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:35.790090+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:06.446986+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":153,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T13:53:02.908576+0000","last_change":"2024-09-21T13:52:04.323345+0000","last_active":"2024-09-21T13:53:02.908576+0000","last_peered":"2024-09-21T13:53:02.908576+0000","last_clean":"2024-09-21T13:53:02.908576+0000","last_became_active":"2024-09-21T13:52:04.322880+0000","last_became_peered":"2024-09-21T13:52:04.322880+0000","last_unstale":"2024-09-21T13:53:02.908576+0000","last_undegraded":"2024-09-21T13:53:02.908576+0000","last_fullsized":"2024-09-21T13:53:02.908576+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:36.086691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:36.086691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":3563061272,"num_objects":54369,"num_object_clones":0,"num_object_copies":163107,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5682,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54369,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":218467,"num_write_kb":8730689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19753,"num_bytes_recovered":1294532608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10691162112,"data_stored":10691149896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85101,"ondisk_log_size":85101,"up":48,"acting":45,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69,"ondisk_log_size":69,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":694,"num_read_kb":451,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":395,"ondisk_log_size":395,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":240,"num_read_kb":235,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234038,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1222804,"kb_used_data":1101376,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":92522348,"statfs":{"total":95995035648,"available":94742884352,"internally_reserved":0,"allocated":1127809024,"data_stored":1127332004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":124320202},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462971,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1201808,"kb_used_data":1098428,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":92543344,"statfs":{"total":95995035648,"available":94764384256,"internally_reserved":0,"allocated":1124790272,"data_stored":1124324529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":105839046},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724610,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":989184,"kb_used_data":889964,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":92755968,"statfs":{"total":95995035648,"available":94982111232,"internally_reserved":0,"allocated":911323136,"data_stored":910865883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920840,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1216816,"kb_used_data":1095380,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":92528336,"statfs":{"total":95995035648,"available":94749016064,"internally_reserved":0,"allocated":1121669120,"data_stored":1121178851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117069,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2159532,"kb_used_data":1957528,"kb_used_omap":1,"kb_used_meta":201918,"kb_avail":91585620,"statfs":{"total":95995035648,"available":93783674880,"internally_reserved":0,"allocated":2004508672,"data_stored":2004015137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":206764487},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":130,"seq":558345748493,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1489032,"kb_used_data":1326320,"kb_used_omap":4,"kb_used_meta":162683,"kb_avail":92256120,"statfs":{"total":95995035648,"available":94470266880,"internally_reserved":0,"allocated":1358151680,"data_stored":1357690353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4441,"internal_metadata":166588071},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574938,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1449660,"kb_used_data":1332388,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":92295492,"statfs":{"total":95995035648,"available":94510583808,"internally_reserved":0,"allocated":1364365312,"data_stored":1363850965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":120060357},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738464,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1912796,"kb_used_data":1755204,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":91832356,"statfs":{"total":95995035648,"available":94036332544,"internally_reserved":0,"allocated":1797328896,"data_stored":1796847077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":161282501},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1765150720,"data_stored":1765146648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1341919232,"data_stored":1341915160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1350500352,"data_stored":1350500352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1996685312,"data_stored":1996685312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1114374144,"data_stored":1114374144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":893452288,"data_stored":893452288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1119752192,"data_stored":1119748120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1109327872,"data_stored":1109327872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T13:53:07.807 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T13:53:07.894 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:07 smithi049 ceph-mon[40489]: pgmap v194: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 145 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 802 op/s; 5682/163704 objects degraded (3.471%); 19 MiB/s, 308 objects/s recovering 2024-09-21T13:53:07.894 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:07 smithi049 ceph-mon[40489]: from='client.15057 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T13:53:07.894 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:07 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3472127786' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:53:07.894 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:07 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:53:07.895 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:07 smithi049 ceph-mon[30080]: pgmap v194: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 145 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 802 op/s; 5682/163704 objects degraded (3.471%); 19 MiB/s, 308 objects/s recovering 2024-09-21T13:53:07.895 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:07 smithi049 ceph-mon[30080]: from='client.15057 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T13:53:07.895 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:07 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3472127786' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:53:07.895 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:07 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:53:07.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:07.896 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:07.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:53:07.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:07 smithi187 ceph-mon[33032]: pgmap v194: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 145 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 802 op/s; 5682/163704 objects degraded (3.471%); 19 MiB/s, 308 objects/s recovering 2024-09-21T13:53:07.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:07 smithi187 ceph-mon[33032]: from='client.15057 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T13:53:07.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:07 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3472127786' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:53:07.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:07 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:53:07.997 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:53:08.065 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:08.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:08.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:53:08.167 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:08.191 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:08.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:08.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:08.293 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:53:08.320 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:08.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:08.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:53:08.422 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:53:08.465 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:08.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:08.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:53:08.567 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:53:08.608 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:08.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:08.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:53:08.710 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:53:08.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:08.773 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:08.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:53:08.874 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:08.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:08.899 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:08.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:08.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:08 smithi187 ceph-mon[33032]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:53:08.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:08 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T13:53:08.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:08 smithi187 ceph-mon[33032]: osdmap e157: 8 total, 8 up, 8 in 2024-09-21T13:53:09.000 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:53:09.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:08 smithi049 ceph-mon[40489]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:53:09.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:08 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T13:53:09.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:08 smithi049 ceph-mon[40489]: osdmap e157: 8 total, 8 up, 8 in 2024-09-21T13:53:09.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:08 smithi049 ceph-mon[30080]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:53:09.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:08 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T13:53:09.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:08 smithi049 ceph-mon[30080]: osdmap e157: 8 total, 8 up, 8 in 2024-09-21T13:53:09.025 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:09.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:09.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:53:09.127 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:09.153 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:09.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:09.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:09.255 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:53:09.295 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:09.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:09.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:53:09.397 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:53:09.423 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:09.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:09.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:53:09.444 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:53:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:53:09.525 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:53:09.563 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:09.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:09.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:53:09.665 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:09.687 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:09.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:09.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:09.789 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:53:09.828 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:09.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:09.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:53:09.869 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:09 smithi049 ceph-mon[40489]: pgmap v196: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 145 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 875 B/s rd, 0 B/s wr, 714 op/s; 5682/162816 objects degraded (3.490%); 16 MiB/s, 255 objects/s recovering 2024-09-21T13:53:09.869 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:09 smithi049 ceph-mon[30080]: pgmap v196: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 145 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 875 B/s rd, 0 B/s wr, 714 op/s; 5682/162816 objects degraded (3.490%); 16 MiB/s, 255 objects/s recovering 2024-09-21T13:53:09.930 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:53:09.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:09 smithi187 ceph-mon[33032]: pgmap v196: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 145 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 875 B/s rd, 0 B/s wr, 714 op/s; 5682/162816 objects degraded (3.490%); 16 MiB/s, 255 objects/s recovering 2024-09-21T13:53:09.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:09.979 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:09.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:53:10.080 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:53:10.120 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:10.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:10.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:53:10.222 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:53:10.263 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:10.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:10.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:53:10.365 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:53:10.405 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:10.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:10.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:53:10.507 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:53:10.546 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:10.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:10.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:53:10.647 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:53:10.694 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:10.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:10.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:53:10.766 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T13:53:10.778 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T13:53:10.778 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:53:10.796 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:10.821 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:10.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:10.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:10.923 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:10.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:10 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:53:10.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:10 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:53:10.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:10 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:53:10.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:10 smithi187 ceph-mon[33032]: osdmap e158: 8 total, 8 up, 8 in 2024-09-21T13:53:10.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:10 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 5682/162816 objects degraded (3.490%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T13:53:10.949 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:10.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:10.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:11.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:10 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:53:11.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:10 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:53:11.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:10 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:53:11.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:10 smithi049 ceph-mon[40489]: osdmap e158: 8 total, 8 up, 8 in 2024-09-21T13:53:11.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:10 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 5682/162816 objects degraded (3.490%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T13:53:11.005 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:53:10 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:53:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:53:11.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:10 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:53:11.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:10 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:53:11.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:10 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:53:11.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:10 smithi049 ceph-mon[30080]: osdmap e158: 8 total, 8 up, 8 in 2024-09-21T13:53:11.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:10 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 5682/162816 objects degraded (3.490%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T13:53:11.051 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:53:11.078 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:11.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:11.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:53:11.179 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:53:11.209 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:11.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:11.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:53:11.311 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:53:11.340 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:11.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:11.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:53:11.442 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:53:11.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:11.465 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:11.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:53:11.566 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:11.592 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:11.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:11.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:11.694 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:53:11.732 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:11.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:11.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:53:11.833 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:53:11.857 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:11.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:11.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:53:11.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:11 smithi187 ceph-mon[33032]: pgmap v198: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 145 active+clean; 3.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 976 op/s; 4435/148959 objects degraded (2.977%); 16 MiB/s, 261 objects/s recovering 2024-09-21T13:53:11.959 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:53:11.980 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:11.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:11.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:53:12.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:11 smithi049 ceph-mon[40489]: pgmap v198: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 145 active+clean; 3.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 976 op/s; 4435/148959 objects degraded (2.977%); 16 MiB/s, 261 objects/s recovering 2024-09-21T13:53:12.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:11 smithi049 ceph-mon[30080]: pgmap v198: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 145 active+clean; 3.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 976 op/s; 4435/148959 objects degraded (2.977%); 16 MiB/s, 261 objects/s recovering 2024-09-21T13:53:12.081 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:53:12.123 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:12.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:12.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:53:12.225 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:53:12.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:12.328 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:12.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:53:12.429 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:53:12.473 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:12.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:12.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:53:12.575 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:53:12.601 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:12.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:12.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:53:12.703 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:53:12.748 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:12.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:12.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:53:12.850 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:53:12.876 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:12.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:12.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:53:12.978 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:53:13.015 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:13.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:13.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:53:13.117 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:53:13.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:13.147 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:13.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:53:13.247 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:53:13.273 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:13.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:13.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:53:13.375 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:53:13.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:13.417 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:13.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:53:13.518 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:53:13.543 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:13.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:13.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:53:13.645 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:53:13.687 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:13.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:13.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:53:13.789 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:53:13.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:13.831 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:13.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:53:13.932 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:53:13.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:13 smithi187 ceph-mon[33032]: pgmap v199: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 145 active+clean; 3.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 746 B/s rd, 0 B/s wr, 717 op/s; 4435/148959 objects degraded (2.977%); 6.9 MiB/s, 111 objects/s recovering 2024-09-21T13:53:13.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:13 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:53:13.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:13 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:53:13.976 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:13.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:13.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:53:14.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:13 smithi049 ceph-mon[40489]: pgmap v199: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 145 active+clean; 3.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 746 B/s rd, 0 B/s wr, 717 op/s; 4435/148959 objects degraded (2.977%); 6.9 MiB/s, 111 objects/s recovering 2024-09-21T13:53:14.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:13 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:53:14.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:13 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:53:14.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:13 smithi049 ceph-mon[30080]: pgmap v199: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 145 active+clean; 3.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 746 B/s rd, 0 B/s wr, 717 op/s; 4435/148959 objects degraded (2.977%); 6.9 MiB/s, 111 objects/s recovering 2024-09-21T13:53:14.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:13 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:53:14.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:13 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:53:14.078 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:14.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:14.100 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:14.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:14.201 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:53:14.246 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:14.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:14.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:53:14.348 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:53:14.385 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:14.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:14.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:53:14.487 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:53:14.512 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:14.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:14.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:53:14.537 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T13:53:14.543 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:53:14.556 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:53:14.570 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:53:14.614 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:53:14.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:14.673 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:14.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:53:14.774 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:53:14.798 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:14.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:14.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:53:14.899 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:53:14.923 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:14.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:14.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:53:15.025 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:15.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:15.051 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:15.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:15.152 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:53:15.177 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:15.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:15.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:53:15.279 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:53:15.302 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:15.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:15.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:53:15.404 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:53:15.445 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:15.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:15.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:53:15.525 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:53:15.525 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:53:15.547 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:53:15.569 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:53:15.570 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:53:15.570 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:15.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:15.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:53:15.621 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:53:15.621 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:53:15.662 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:15 smithi049 ceph-mon[40489]: pgmap v200: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 145 active+clean; 2.8 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s; 4024/138720 objects degraded (2.901%); 8.9 MiB/s, 142 objects/s recovering 2024-09-21T13:53:15.662 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:15 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 4024/138720 objects degraded (2.901%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T13:53:15.662 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:15 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3177541826' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:53:15.664 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:15 smithi049 ceph-mon[30080]: pgmap v200: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 145 active+clean; 2.8 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s; 4024/138720 objects degraded (2.901%); 8.9 MiB/s, 142 objects/s recovering 2024-09-21T13:53:15.664 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:15 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 4024/138720 objects degraded (2.901%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T13:53:15.664 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:15 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3177541826' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:53:15.672 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:53:15.681 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:53:15.682 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:53:15.720 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:15.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:15.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:53:15.725 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:53:15.725 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:53:15.763 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:53:15.764 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:53:15.805 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:53:15.805 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:53:15.821 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:53:15.840 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:53:15.841 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:53:15.862 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:15.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:15.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:53:15.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:15 smithi187 ceph-mon[33032]: pgmap v200: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 145 active+clean; 2.8 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s; 4024/138720 objects degraded (2.901%); 8.9 MiB/s, 142 objects/s recovering 2024-09-21T13:53:15.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:15 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 4024/138720 objects degraded (2.901%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T13:53:15.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:15 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3177541826' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:53:15.964 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:53:15.986 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:15.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:15.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:53:16.051 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:53:16.053 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-09-21T13:53:16.088 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:16.109 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:16.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:16.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:16.210 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:53:16.234 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:16.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:16.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:53:16.336 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:53:16.360 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:16.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:16.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:53:16.462 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:53:16.483 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:16.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:16.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:53:16.585 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:53:16.607 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:16.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:16.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:53:16.708 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:16.737 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:16.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:16.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:16.839 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:53:16.872 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:16.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:16.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:53:16.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:16 smithi187 ceph-mon[33032]: osdmap e159: 8 total, 8 up, 8 in 2024-09-21T13:53:16.974 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:53:16.996 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:16.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:16.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:53:17.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:16 smithi049 ceph-mon[40489]: osdmap e159: 8 total, 8 up, 8 in 2024-09-21T13:53:17.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:16 smithi049 ceph-mon[30080]: osdmap e159: 8 total, 8 up, 8 in 2024-09-21T13:53:17.097 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:17.118 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:17.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:17.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:17.219 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:53:17.242 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:17.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:17.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:53:17.335 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":202,"stamp":"2024-09-21T13:53:15.750025+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2944604723,"num_objects":45122,"num_object_clones":0,"num_object_copies":135366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3242,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45122,"num_whiteouts":0,"num_read":1095,"num_read_kb":952,"num_write":228462,"num_write_kb":8732362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21341,"num_bytes_recovered":1398211324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86106,"ondisk_log_size":86106,"up":444,"acting":441,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":9872712,"kb_used_data":8737544,"kb_used_omap":15,"kb_used_meta":1134640,"kb_avail":740088504,"statfs":{"total":767960285184,"available":757850628096,"internally_reserved":0,"allocated":8947245056,"data_stored":8943285609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15598,"internal_metadata":1161872146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-599654400,"num_objects":-9150,"num_object_clones":0,"num_object_copies":-27450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2440,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9150,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":9150,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1573,"num_bytes_recovered":103088128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.005472"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":159,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.588220+0000","last_change":"2024-09-21T13:52:26.677736+0000","last_active":"2024-09-21T13:53:09.588220+0000","last_peered":"2024-09-21T13:53:09.588220+0000","last_clean":"2024-09-21T13:53:09.588220+0000","last_became_active":"2024-09-21T13:52:04.317322+0000","last_became_peered":"2024-09-21T13:52:04.317322+0000","last_unstale":"2024-09-21T13:53:09.588220+0000","last_undegraded":"2024-09-21T13:53:09.588220+0000","last_fullsized":"2024-09-21T13:53:09.588220+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:26.677689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:52:26.677689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:45.636529+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"68'9","reported_seq":227,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.595692+0000","last_change":"2024-09-21T13:51:55.225649+0000","last_active":"2024-09-21T13:53:09.595692+0000","last_peered":"2024-09-21T13:53:09.595692+0000","last_clean":"2024-09-21T13:53:09.595692+0000","last_became_active":"2024-09-21T13:49:00.055210+0000","last_became_peered":"2024-09-21T13:49:00.055210+0000","last_unstale":"2024-09-21T13:53:09.595692+0000","last_undegraded":"2024-09-21T13:53:09.595692+0000","last_fullsized":"2024-09-21T13:53:09.595692+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:51:55.225579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:55.225579+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":207,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.596645+0000","last_change":"2024-09-21T13:50:28.058990+0000","last_active":"2024-09-21T13:53:09.596645+0000","last_peered":"2024-09-21T13:53:09.596645+0000","last_clean":"2024-09-21T13:53:09.596645+0000","last_became_active":"2024-09-21T13:49:02.060984+0000","last_became_peered":"2024-09-21T13:49:02.060984+0000","last_unstale":"2024-09-21T13:53:09.596645+0000","last_undegraded":"2024-09-21T13:53:09.596645+0000","last_fullsized":"2024-09-21T13:53:09.596645+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:28.058918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:28.058918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:39.434903+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":195,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.595727+0000","last_change":"2024-09-21T13:49:04.072734+0000","last_active":"2024-09-21T13:53:09.595727+0000","last_peered":"2024-09-21T13:53:09.595727+0000","last_clean":"2024-09-21T13:53:09.595727+0000","last_became_active":"2024-09-21T13:49:04.072460+0000","last_became_peered":"2024-09-21T13:49:04.072460+0000","last_unstale":"2024-09-21T13:53:09.595727+0000","last_undegraded":"2024-09-21T13:53:09.595727+0000","last_fullsized":"2024-09-21T13:53:09.595727+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":203,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.600798+0000","last_change":"2024-09-21T13:51:18.572788+0000","last_active":"2024-09-21T13:53:09.600798+0000","last_peered":"2024-09-21T13:53:09.600798+0000","last_clean":"2024-09-21T13:53:09.600798+0000","last_became_active":"2024-09-21T13:49:04.433965+0000","last_became_peered":"2024-09-21T13:49:04.433965+0000","last_unstale":"2024-09-21T13:53:09.600798+0000","last_undegraded":"2024-09-21T13:53:09.600798+0000","last_fullsized":"2024-09-21T13:53:09.600798+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:18.572729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:51:18.572729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":219,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.591231+0000","last_change":"2024-09-21T13:52:04.317119+0000","last_active":"2024-09-21T13:53:09.591231+0000","last_peered":"2024-09-21T13:53:09.591231+0000","last_clean":"2024-09-21T13:53:09.591231+0000","last_became_active":"2024-09-21T13:52:04.316819+0000","last_became_peered":"2024-09-21T13:52:04.316819+0000","last_unstale":"2024-09-21T13:53:09.591231+0000","last_undegraded":"2024-09-21T13:53:09.591231+0000","last_fullsized":"2024-09-21T13:53:09.591231+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"68'15","reported_seq":239,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.591058+0000","last_change":"2024-09-21T13:52:04.315645+0000","last_active":"2024-09-21T13:53:09.591058+0000","last_peered":"2024-09-21T13:53:09.591058+0000","last_clean":"2024-09-21T13:53:09.591058+0000","last_became_active":"2024-09-21T13:52:04.315101+0000","last_became_peered":"2024-09-21T13:52:04.315101+0000","last_unstale":"2024-09-21T13:53:09.591058+0000","last_undegraded":"2024-09-21T13:53:09.591058+0000","last_fullsized":"2024-09-21T13:53:09.591058+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":215,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.595621+0000","last_change":"2024-09-21T13:51:43.040397+0000","last_active":"2024-09-21T13:53:09.595621+0000","last_peered":"2024-09-21T13:53:09.595621+0000","last_clean":"2024-09-21T13:53:09.595621+0000","last_became_active":"2024-09-21T13:49:02.063687+0000","last_became_peered":"2024-09-21T13:49:02.063687+0000","last_unstale":"2024-09-21T13:53:09.595621+0000","last_undegraded":"2024-09-21T13:53:09.595621+0000","last_fullsized":"2024-09-21T13:53:09.595621+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:43.040348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:43.040348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":223,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.588000+0000","last_change":"2024-09-21T13:52:49.451929+0000","last_active":"2024-09-21T13:53:09.588000+0000","last_peered":"2024-09-21T13:53:09.588000+0000","last_clean":"2024-09-21T13:53:09.588000+0000","last_became_active":"2024-09-21T13:48:58.050695+0000","last_became_peered":"2024-09-21T13:48:58.050695+0000","last_unstale":"2024-09-21T13:53:09.588000+0000","last_undegraded":"2024-09-21T13:53:09.588000+0000","last_fullsized":"2024-09-21T13:53:09.588000+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:49.451882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:52:49.451882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:54.087443+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"68'9","reported_seq":227,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.595945+0000","last_change":"2024-09-21T13:51:21.951391+0000","last_active":"2024-09-21T13:53:09.595945+0000","last_peered":"2024-09-21T13:53:09.595945+0000","last_clean":"2024-09-21T13:53:09.595945+0000","last_became_active":"2024-09-21T13:49:00.059382+0000","last_became_peered":"2024-09-21T13:49:00.059382+0000","last_unstale":"2024-09-21T13:53:09.595945+0000","last_undegraded":"2024-09-21T13:53:09.595945+0000","last_fullsized":"2024-09-21T13:53:09.595945+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:51:21.951305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:21.951305+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":199,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.587416+0000","last_change":"2024-09-21T13:49:02.066978+0000","last_active":"2024-09-21T13:53:09.587416+0000","last_peered":"2024-09-21T13:53:09.587416+0000","last_clean":"2024-09-21T13:53:09.587416+0000","last_became_active":"2024-09-21T13:49:02.066693+0000","last_became_peered":"2024-09-21T13:49:02.066693+0000","last_unstale":"2024-09-21T13:53:09.587416+0000","last_undegraded":"2024-09-21T13:53:09.587416+0000","last_fullsized":"2024-09-21T13:53:09.587416+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:09.312032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":156,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.596498+0000","last_change":"2024-09-21T13:52:28.417865+0000","last_active":"2024-09-21T13:53:09.596498+0000","last_peered":"2024-09-21T13:53:09.596498+0000","last_clean":"2024-09-21T13:53:09.596498+0000","last_became_active":"2024-09-21T13:50:29.689381+0000","last_became_peered":"2024-09-21T13:50:29.689381+0000","last_unstale":"2024-09-21T13:53:09.596498+0000","last_undegraded":"2024-09-21T13:53:09.596498+0000","last_fullsized":"2024-09-21T13:53:09.596498+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:28.417690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:28.417690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:33.915909+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":215,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.587455+0000","last_change":"2024-09-21T13:50:39.480995+0000","last_active":"2024-09-21T13:53:09.587455+0000","last_peered":"2024-09-21T13:53:09.587455+0000","last_clean":"2024-09-21T13:53:09.587455+0000","last_became_active":"2024-09-21T13:48:58.050906+0000","last_became_peered":"2024-09-21T13:48:58.050906+0000","last_unstale":"2024-09-21T13:53:09.587455+0000","last_undegraded":"2024-09-21T13:53:09.587455+0000","last_fullsized":"2024-09-21T13:53:09.587455+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:39.480930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:39.480930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:32.611461+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"68'5","reported_seq":237,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.597270+0000","last_change":"2024-09-21T13:52:04.319297+0000","last_active":"2024-09-21T13:53:09.597270+0000","last_peered":"2024-09-21T13:53:09.597270+0000","last_clean":"2024-09-21T13:53:09.597270+0000","last_became_active":"2024-09-21T13:52:04.318515+0000","last_became_peered":"2024-09-21T13:52:04.318515+0000","last_unstale":"2024-09-21T13:53:09.597270+0000","last_undegraded":"2024-09-21T13:53:09.597270+0000","last_fullsized":"2024-09-21T13:53:09.597270+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:02.067591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":207,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.599492+0000","last_change":"2024-09-21T13:50:40.188797+0000","last_active":"2024-09-21T13:53:09.599492+0000","last_peered":"2024-09-21T13:53:09.599492+0000","last_clean":"2024-09-21T13:53:09.599492+0000","last_became_active":"2024-09-21T13:49:02.060212+0000","last_became_peered":"2024-09-21T13:49:02.060212+0000","last_unstale":"2024-09-21T13:53:09.599492+0000","last_undegraded":"2024-09-21T13:53:09.599492+0000","last_fullsized":"2024-09-21T13:53:09.599492+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:40.188712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:40.188712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":211,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.586693+0000","last_change":"2024-09-21T13:52:43.516806+0000","last_active":"2024-09-21T13:53:09.586693+0000","last_peered":"2024-09-21T13:53:09.586693+0000","last_clean":"2024-09-21T13:53:09.586693+0000","last_became_active":"2024-09-21T13:49:04.078026+0000","last_became_peered":"2024-09-21T13:49:04.078026+0000","last_unstale":"2024-09-21T13:53:09.586693+0000","last_undegraded":"2024-09-21T13:53:09.586693+0000","last_fullsized":"2024-09-21T13:53:09.586693+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:43.516741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:43.516741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:08.753512+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":204,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.596172+0000","last_change":"2024-09-21T13:50:38.027288+0000","last_active":"2024-09-21T13:53:09.596172+0000","last_peered":"2024-09-21T13:53:09.596172+0000","last_clean":"2024-09-21T13:53:09.596172+0000","last_became_active":"2024-09-21T13:49:04.431422+0000","last_became_peered":"2024-09-21T13:49:04.431422+0000","last_unstale":"2024-09-21T13:53:09.596172+0000","last_undegraded":"2024-09-21T13:53:09.596172+0000","last_fullsized":"2024-09-21T13:53:09.596172+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:38.027227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:38.027227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":140,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.597448+0000","last_change":"2024-09-21T13:50:29.692621+0000","last_active":"2024-09-21T13:53:09.597448+0000","last_peered":"2024-09-21T13:53:09.597448+0000","last_clean":"2024-09-21T13:53:09.597448+0000","last_became_active":"2024-09-21T13:50:29.688282+0000","last_became_peered":"2024-09-21T13:50:29.688282+0000","last_unstale":"2024-09-21T13:53:09.597448+0000","last_undegraded":"2024-09-21T13:53:09.597448+0000","last_fullsized":"2024-09-21T13:53:09.597448+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:04.102443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"68'15","reported_seq":168,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.596816+0000","last_change":"2024-09-21T13:52:04.324874+0000","last_active":"2024-09-21T13:53:09.596816+0000","last_peered":"2024-09-21T13:53:09.596816+0000","last_clean":"2024-09-21T13:53:09.596816+0000","last_became_active":"2024-09-21T13:52:04.324273+0000","last_became_peered":"2024-09-21T13:52:04.324273+0000","last_unstale":"2024-09-21T13:53:09.596816+0000","last_undegraded":"2024-09-21T13:53:09.596816+0000","last_fullsized":"2024-09-21T13:53:09.596816+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:50:24.749694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:24.749694+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":215,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.602240+0000","last_change":"2024-09-21T13:52:26.372217+0000","last_active":"2024-09-21T13:53:09.602240+0000","last_peered":"2024-09-21T13:53:09.602240+0000","last_clean":"2024-09-21T13:53:09.602240+0000","last_became_active":"2024-09-21T13:49:02.066796+0000","last_became_peered":"2024-09-21T13:49:02.066796+0000","last_unstale":"2024-09-21T13:53:09.602240+0000","last_undegraded":"2024-09-21T13:53:09.602240+0000","last_fullsized":"2024-09-21T13:53:09.602240+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:26.372144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:52:26.372144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:40.709431+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":196,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.612738+0000","last_change":"2024-09-21T13:49:04.434429+0000","last_active":"2024-09-21T13:53:09.612738+0000","last_peered":"2024-09-21T13:53:09.612738+0000","last_clean":"2024-09-21T13:53:09.612738+0000","last_became_active":"2024-09-21T13:49:04.434049+0000","last_became_peered":"2024-09-21T13:49:04.434049+0000","last_unstale":"2024-09-21T13:53:09.612738+0000","last_undegraded":"2024-09-21T13:53:09.612738+0000","last_fullsized":"2024-09-21T13:53:09.612738+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":226,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.596038+0000","last_change":"2024-09-21T13:52:52.142682+0000","last_active":"2024-09-21T13:53:09.596038+0000","last_peered":"2024-09-21T13:53:09.596038+0000","last_clean":"2024-09-21T13:53:09.596038+0000","last_became_active":"2024-09-21T13:52:04.316016+0000","last_became_peered":"2024-09-21T13:52:04.316016+0000","last_unstale":"2024-09-21T13:53:09.596038+0000","last_undegraded":"2024-09-21T13:53:09.596038+0000","last_fullsized":"2024-09-21T13:53:09.596038+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:52.142638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:52:52.142638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:13.595892+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"68'12","reported_seq":234,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.590846+0000","last_change":"2024-09-21T13:51:52.819966+0000","last_active":"2024-09-21T13:53:09.590846+0000","last_peered":"2024-09-21T13:53:09.590846+0000","last_clean":"2024-09-21T13:53:09.590846+0000","last_became_active":"2024-09-21T13:49:00.061867+0000","last_became_peered":"2024-09-21T13:49:00.061867+0000","last_unstale":"2024-09-21T13:53:09.590846+0000","last_undegraded":"2024-09-21T13:53:09.590846+0000","last_fullsized":"2024-09-21T13:53:09.590846+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'12","last_scrub_stamp":"2024-09-21T13:51:52.819879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:52.819879+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:16.488906+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":210,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.596483+0000","last_change":"2024-09-21T13:52:04.321297+0000","last_active":"2024-09-21T13:53:09.596483+0000","last_peered":"2024-09-21T13:53:09.596483+0000","last_clean":"2024-09-21T13:53:09.596483+0000","last_became_active":"2024-09-21T13:52:04.320800+0000","last_became_peered":"2024-09-21T13:52:04.320800+0000","last_unstale":"2024-09-21T13:53:09.596483+0000","last_undegraded":"2024-09-21T13:53:09.596483+0000","last_fullsized":"2024-09-21T13:53:09.596483+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":215,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.600082+0000","last_change":"2024-09-21T13:52:54.580070+0000","last_active":"2024-09-21T13:53:09.600082+0000","last_peered":"2024-09-21T13:53:09.600082+0000","last_clean":"2024-09-21T13:53:09.600082+0000","last_became_active":"2024-09-21T13:52:04.321358+0000","last_became_peered":"2024-09-21T13:52:04.321358+0000","last_unstale":"2024-09-21T13:53:09.600082+0000","last_undegraded":"2024-09-21T13:53:09.600082+0000","last_fullsized":"2024-09-21T13:53:09.600082+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T13:52:54.580022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:54.580022+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:21.087079+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":224,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.596803+0000","last_change":"2024-09-21T13:52:43.516442+0000","last_active":"2024-09-21T13:53:09.596803+0000","last_peered":"2024-09-21T13:53:09.596803+0000","last_clean":"2024-09-21T13:53:09.596803+0000","last_became_active":"2024-09-21T13:48:58.048972+0000","last_became_peered":"2024-09-21T13:48:58.048972+0000","last_unstale":"2024-09-21T13:53:09.596803+0000","last_undegraded":"2024-09-21T13:53:09.596803+0000","last_fullsized":"2024-09-21T13:53:09.596803+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:52:43.516221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:52:43.516221+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:48.258159+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"68'10","reported_seq":183,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.600942+0000","last_change":"2024-09-21T13:52:56.526032+0000","last_active":"2024-09-21T13:53:09.600942+0000","last_peered":"2024-09-21T13:53:09.600942+0000","last_clean":"2024-09-21T13:53:09.600942+0000","last_became_active":"2024-09-21T13:50:29.694465+0000","last_became_peered":"2024-09-21T13:50:29.694465+0000","last_unstale":"2024-09-21T13:53:09.600942+0000","last_undegraded":"2024-09-21T13:53:09.600942+0000","last_fullsized":"2024-09-21T13:53:09.600942+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:52:56.525966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:52:56.525966+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:10.063932+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"68'8","reported_seq":279,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:13.161254+0000","last_change":"2024-09-21T13:51:42.174696+0000","last_active":"2024-09-21T13:53:13.161254+0000","last_peered":"2024-09-21T13:53:13.161254+0000","last_clean":"2024-09-21T13:53:13.161254+0000","last_became_active":"2024-09-21T13:49:02.059638+0000","last_became_peered":"2024-09-21T13:49:02.059638+0000","last_unstale":"2024-09-21T13:53:13.161254+0000","last_undegraded":"2024-09-21T13:53:13.161254+0000","last_fullsized":"2024-09-21T13:53:13.161254+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:51:42.174629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:42.174629+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:08.238915+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"158'14319","reported_seq":14602,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:13.378901+0000","last_change":"2024-09-21T13:52:51.407976+0000","last_active":"2024-09-21T13:53:13.378901+0000","last_peered":"2024-09-21T13:53:13.378901+0000","last_clean":"2024-09-21T13:53:13.378901+0000","last_became_active":"2024-09-21T13:50:07.422961+0000","last_became_peered":"2024-09-21T13:50:07.422961+0000","last_unstale":"2024-09-21T13:53:13.378901+0000","last_undegraded":"2024-09-21T13:53:13.378901+0000","last_fullsized":"2024-09-21T13:53:13.378901+0000","mapping_epoch":83,"log_start":"117'8000","ondisk_log_start":"117'8000","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'13038","last_scrub_stamp":"2024-09-21T13:52:51.407818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:52:51.407818+0000","objects_scrubbed":4170,"log_size":6319,"log_dups_size":0,"ondisk_log_size":6319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:10.866985+0000","scrub_duration":960,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188022784,"num_objects":2869,"num_object_clones":0,"num_object_copies":8607,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14319,"num_write_kb":550016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"68'15","reported_seq":244,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.598389+0000","last_change":"2024-09-21T13:52:16.452049+0000","last_active":"2024-09-21T13:53:09.598389+0000","last_peered":"2024-09-21T13:53:09.598389+0000","last_clean":"2024-09-21T13:53:09.598389+0000","last_became_active":"2024-09-21T13:49:00.060324+0000","last_became_peered":"2024-09-21T13:49:00.060324+0000","last_unstale":"2024-09-21T13:53:09.598389+0000","last_undegraded":"2024-09-21T13:53:09.598389+0000","last_fullsized":"2024-09-21T13:53:09.598389+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:52:16.451964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:52:16.451964+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:31.981849+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":215,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.595251+0000","last_change":"2024-09-21T13:50:06.151943+0000","last_active":"2024-09-21T13:53:09.595251+0000","last_peered":"2024-09-21T13:53:09.595251+0000","last_clean":"2024-09-21T13:53:09.595251+0000","last_became_active":"2024-09-21T13:48:58.051911+0000","last_became_peered":"2024-09-21T13:48:58.051911+0000","last_unstale":"2024-09-21T13:53:09.595251+0000","last_undegraded":"2024-09-21T13:53:09.595251+0000","last_fullsized":"2024-09-21T13:53:09.595251+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.151888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.151888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"68'8","reported_seq":262,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.596256+0000","last_change":"2024-09-21T13:49:02.061966+0000","last_active":"2024-09-21T13:53:09.596256+0000","last_peered":"2024-09-21T13:53:09.596256+0000","last_clean":"2024-09-21T13:53:09.596256+0000","last_became_active":"2024-09-21T13:49:02.061679+0000","last_became_peered":"2024-09-21T13:49:02.061679+0000","last_unstale":"2024-09-21T13:53:09.596256+0000","last_undegraded":"2024-09-21T13:53:09.596256+0000","last_fullsized":"2024-09-21T13:53:09.596256+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":203,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.587598+0000","last_change":"2024-09-21T13:50:34.481503+0000","last_active":"2024-09-21T13:53:09.587598+0000","last_peered":"2024-09-21T13:53:09.587598+0000","last_clean":"2024-09-21T13:53:09.587598+0000","last_became_active":"2024-09-21T13:49:04.081905+0000","last_became_peered":"2024-09-21T13:49:04.081905+0000","last_unstale":"2024-09-21T13:53:09.587598+0000","last_undegraded":"2024-09-21T13:53:09.587598+0000","last_fullsized":"2024-09-21T13:53:09.587598+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:34.481450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:34.481450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:42.543466+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"158'14396","reported_seq":21640,"reported_epoch":158,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-21T13:53:13.859467+0000","last_change":"2024-09-21T13:52:41.673644+0000","last_active":"2024-09-21T13:53:13.859467+0000","last_peered":"2024-09-21T13:53:13.859467+0000","last_clean":"2024-09-21T13:50:42.911378+0000","last_became_active":"2024-09-21T13:52:04.993197+0000","last_became_peered":"2024-09-21T13:52:04.993197+0000","last_unstale":"2024-09-21T13:53:13.859467+0000","last_undegraded":"2024-09-21T13:52:04.885711+0000","last_fullsized":"2024-09-21T13:52:04.863350+0000","mapping_epoch":131,"log_start":"126'9700","ondisk_log_start":"126'9700","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":4696,"log_dups_size":0,"ondisk_log_size":4696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186253312,"num_objects":2842,"num_object_clones":0,"num_object_copies":8526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":167,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14396,"num_write_kb":551616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3538,"num_bytes_recovered":231866368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":2842}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"68'18","reported_seq":244,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.595139+0000","last_change":"2024-09-21T13:50:27.281455+0000","last_active":"2024-09-21T13:53:09.595139+0000","last_peered":"2024-09-21T13:53:09.595139+0000","last_clean":"2024-09-21T13:53:09.595139+0000","last_became_active":"2024-09-21T13:49:00.429314+0000","last_became_peered":"2024-09-21T13:49:00.429314+0000","last_unstale":"2024-09-21T13:53:09.595139+0000","last_undegraded":"2024-09-21T13:53:09.595139+0000","last_fullsized":"2024-09-21T13:53:09.595139+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'18","last_scrub_stamp":"2024-09-21T13:50:27.281356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:27.281356+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":207,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.613231+0000","last_change":"2024-09-21T13:48:58.056669+0000","last_active":"2024-09-21T13:53:09.613231+0000","last_peered":"2024-09-21T13:53:09.613231+0000","last_clean":"2024-09-21T13:53:09.613231+0000","last_became_active":"2024-09-21T13:48:58.056375+0000","last_became_peered":"2024-09-21T13:48:58.056375+0000","last_unstale":"2024-09-21T13:53:09.613231+0000","last_undegraded":"2024-09-21T13:53:09.613231+0000","last_fullsized":"2024-09-21T13:53:09.613231+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":208,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.595767+0000","last_change":"2024-09-21T13:51:23.940809+0000","last_active":"2024-09-21T13:53:09.595767+0000","last_peered":"2024-09-21T13:53:09.595767+0000","last_clean":"2024-09-21T13:53:09.595767+0000","last_became_active":"2024-09-21T13:49:02.062955+0000","last_became_peered":"2024-09-21T13:49:02.062955+0000","last_unstale":"2024-09-21T13:53:09.595767+0000","last_undegraded":"2024-09-21T13:53:09.595767+0000","last_fullsized":"2024-09-21T13:53:09.595767+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:23.940746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:23.940746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":206,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.595903+0000","last_change":"2024-09-21T13:52:04.316586+0000","last_active":"2024-09-21T13:53:09.595903+0000","last_peered":"2024-09-21T13:53:09.595903+0000","last_clean":"2024-09-21T13:53:09.595903+0000","last_became_active":"2024-09-21T13:52:04.316121+0000","last_became_peered":"2024-09-21T13:52:04.316121+0000","last_unstale":"2024-09-21T13:53:09.595903+0000","last_undegraded":"2024-09-21T13:53:09.595903+0000","last_fullsized":"2024-09-21T13:53:09.595903+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"158'14042","reported_seq":14226,"reported_epoch":158,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2024-09-21T13:53:13.876626+0000","last_change":"2024-09-21T13:53:05.053228+0000","last_active":"2024-09-21T13:53:13.876626+0000","last_peered":"2024-09-21T13:53:13.876626+0000","last_clean":"2024-09-21T13:50:42.909755+0000","last_became_active":"2024-09-21T13:52:04.998636+0000","last_became_peered":"2024-09-21T13:52:04.998636+0000","last_unstale":"2024-09-21T13:53:13.876626+0000","last_undegraded":"2024-09-21T13:52:04.888197+0000","last_fullsized":"2024-09-21T13:52:04.863335+0000","mapping_epoch":131,"log_start":"126'9400","ondisk_log_start":"126'9400","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":4642,"log_dups_size":0,"ondisk_log_size":4642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183369728,"num_objects":2798,"num_object_clones":0,"num_object_copies":8394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2798,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14042,"num_write_kb":538880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":2798}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"68'14","reported_seq":233,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.595828+0000","last_change":"2024-09-21T13:50:29.009786+0000","last_active":"2024-09-21T13:53:09.595828+0000","last_peered":"2024-09-21T13:53:09.595828+0000","last_clean":"2024-09-21T13:53:09.595828+0000","last_became_active":"2024-09-21T13:49:00.055031+0000","last_became_peered":"2024-09-21T13:49:00.055031+0000","last_unstale":"2024-09-21T13:53:09.595828+0000","last_undegraded":"2024-09-21T13:53:09.595828+0000","last_fullsized":"2024-09-21T13:53:09.595828+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'14","last_scrub_stamp":"2024-09-21T13:50:29.009720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:29.009720+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":216,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.599609+0000","last_change":"2024-09-21T13:49:59.253636+0000","last_active":"2024-09-21T13:53:09.599609+0000","last_peered":"2024-09-21T13:53:09.599609+0000","last_clean":"2024-09-21T13:53:09.599609+0000","last_became_active":"2024-09-21T13:48:58.053533+0000","last_became_peered":"2024-09-21T13:48:58.053533+0000","last_unstale":"2024-09-21T13:53:09.599609+0000","last_undegraded":"2024-09-21T13:53:09.599609+0000","last_fullsized":"2024-09-21T13:53:09.599609+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:49:59.253587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:49:59.253587+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:34.697714+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"68'8","reported_seq":268,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.593088+0000","last_change":"2024-09-21T13:50:41.458637+0000","last_active":"2024-09-21T13:53:09.593088+0000","last_peered":"2024-09-21T13:53:09.593088+0000","last_clean":"2024-09-21T13:53:09.593088+0000","last_became_active":"2024-09-21T13:49:02.070064+0000","last_became_peered":"2024-09-21T13:49:02.070064+0000","last_unstale":"2024-09-21T13:53:09.593088+0000","last_undegraded":"2024-09-21T13:53:09.593088+0000","last_fullsized":"2024-09-21T13:53:09.593088+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:50:41.458562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:41.458562+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":214,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.599701+0000","last_change":"2024-09-21T13:52:55.541288+0000","last_active":"2024-09-21T13:53:09.599701+0000","last_peered":"2024-09-21T13:53:09.599701+0000","last_clean":"2024-09-21T13:53:09.599701+0000","last_became_active":"2024-09-21T13:52:04.322185+0000","last_became_peered":"2024-09-21T13:52:04.322185+0000","last_unstale":"2024-09-21T13:53:09.599701+0000","last_undegraded":"2024-09-21T13:53:09.599701+0000","last_fullsized":"2024-09-21T13:53:09.599701+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:55.541232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:55.541232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:57.661213+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"158'14108","reported_seq":14270,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:13.356769+0000","last_change":"2024-09-21T13:50:07.419221+0000","last_active":"2024-09-21T13:53:13.356769+0000","last_peered":"2024-09-21T13:53:13.356769+0000","last_clean":"2024-09-21T13:53:13.356769+0000","last_became_active":"2024-09-21T13:50:07.418929+0000","last_became_peered":"2024-09-21T13:50:07.418929+0000","last_unstale":"2024-09-21T13:53:13.356769+0000","last_undegraded":"2024-09-21T13:53:13.356769+0000","last_fullsized":"2024-09-21T13:53:13.356769+0000","mapping_epoch":83,"log_start":"125'9300","ondisk_log_start":"125'9300","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":4808,"log_dups_size":0,"ondisk_log_size":4808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181927936,"num_objects":2776,"num_object_clones":0,"num_object_copies":8328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14108,"num_write_kb":540288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"68'10","reported_seq":235,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.596373+0000","last_change":"2024-09-21T13:51:50.211365+0000","last_active":"2024-09-21T13:53:09.596373+0000","last_peered":"2024-09-21T13:53:09.596373+0000","last_clean":"2024-09-21T13:53:09.596373+0000","last_became_active":"2024-09-21T13:49:00.054564+0000","last_became_peered":"2024-09-21T13:49:00.054564+0000","last_unstale":"2024-09-21T13:53:09.596373+0000","last_undegraded":"2024-09-21T13:53:09.596373+0000","last_fullsized":"2024-09-21T13:53:09.596373+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:51:50.211279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:50.211279+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"61'1","reported_seq":241,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.588109+0000","last_change":"2024-09-21T13:52:16.585403+0000","last_active":"2024-09-21T13:53:09.588109+0000","last_peered":"2024-09-21T13:53:09.588109+0000","last_clean":"2024-09-21T13:53:09.588109+0000","last_became_active":"2024-09-21T13:52:04.315788+0000","last_became_peered":"2024-09-21T13:52:04.315788+0000","last_unstale":"2024-09-21T13:53:09.588109+0000","last_undegraded":"2024-09-21T13:53:09.588109+0000","last_fullsized":"2024-09-21T13:53:09.588109+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:52:16.585323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:52:16.585323+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:28.935922+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"96'9","reported_seq":215,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.591714+0000","last_change":"2024-09-21T13:52:51.443040+0000","last_active":"2024-09-21T13:53:09.591714+0000","last_peered":"2024-09-21T13:53:09.591714+0000","last_clean":"2024-09-21T13:53:09.591714+0000","last_became_active":"2024-09-21T13:50:29.693557+0000","last_became_peered":"2024-09-21T13:50:29.693557+0000","last_unstale":"2024-09-21T13:53:09.591714+0000","last_undegraded":"2024-09-21T13:53:09.591714+0000","last_fullsized":"2024-09-21T13:53:09.591714+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'9","last_scrub_stamp":"2024-09-21T13:52:51.442930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:52:51.442930+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:04.534122+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":151,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.592938+0000","last_change":"2024-09-21T13:52:04.319323+0000","last_active":"2024-09-21T13:53:09.592938+0000","last_peered":"2024-09-21T13:53:09.592938+0000","last_clean":"2024-09-21T13:53:09.592938+0000","last_became_active":"2024-09-21T13:52:04.319034+0000","last_became_peered":"2024-09-21T13:52:04.319034+0000","last_unstale":"2024-09-21T13:53:09.592938+0000","last_undegraded":"2024-09-21T13:53:09.592938+0000","last_fullsized":"2024-09-21T13:53:09.592938+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"158'14288","reported_seq":14521,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:13.380381+0000","last_change":"2024-09-21T13:52:16.618637+0000","last_active":"2024-09-21T13:53:13.380381+0000","last_peered":"2024-09-21T13:53:13.380381+0000","last_clean":"2024-09-21T13:53:13.380381+0000","last_became_active":"2024-09-21T13:50:07.423702+0000","last_became_peered":"2024-09-21T13:50:07.423702+0000","last_unstale":"2024-09-21T13:53:13.380381+0000","last_undegraded":"2024-09-21T13:53:13.380381+0000","last_fullsized":"2024-09-21T13:53:13.380381+0000","mapping_epoch":83,"log_start":"121'8500","ondisk_log_start":"121'8500","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'10945","last_scrub_stamp":"2024-09-21T13:52:16.618486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:52:16.618486+0000","objects_scrubbed":6221,"log_size":5788,"log_dups_size":0,"ondisk_log_size":5788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:43.545327+0000","scrub_duration":1967,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184811520,"num_objects":2820,"num_object_clones":0,"num_object_copies":8460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14288,"num_write_kb":547456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"68'19","reported_seq":204,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.587496+0000","last_change":"2024-09-21T13:50:32.514267+0000","last_active":"2024-09-21T13:53:09.587496+0000","last_peered":"2024-09-21T13:53:09.587496+0000","last_clean":"2024-09-21T13:53:09.587496+0000","last_became_active":"2024-09-21T13:50:29.690185+0000","last_became_peered":"2024-09-21T13:50:29.690185+0000","last_unstale":"2024-09-21T13:53:09.587496+0000","last_undegraded":"2024-09-21T13:53:09.587496+0000","last_fullsized":"2024-09-21T13:53:09.587496+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-21T13:50:32.514210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:32.514210+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:34.599292+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":219,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.598134+0000","last_change":"2024-09-21T13:52:04.322289+0000","last_active":"2024-09-21T13:53:09.598134+0000","last_peered":"2024-09-21T13:53:09.598134+0000","last_clean":"2024-09-21T13:53:09.598134+0000","last_became_active":"2024-09-21T13:52:04.321656+0000","last_became_peered":"2024-09-21T13:52:04.321656+0000","last_unstale":"2024-09-21T13:53:09.598134+0000","last_undegraded":"2024-09-21T13:53:09.598134+0000","last_fullsized":"2024-09-21T13:53:09.598134+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:08.762218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":208,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.591509+0000","last_change":"2024-09-21T13:51:12.939488+0000","last_active":"2024-09-21T13:53:09.591509+0000","last_peered":"2024-09-21T13:53:09.591509+0000","last_clean":"2024-09-21T13:53:09.591509+0000","last_became_active":"2024-09-21T13:49:02.061187+0000","last_became_peered":"2024-09-21T13:49:02.061187+0000","last_unstale":"2024-09-21T13:53:09.591509+0000","last_undegraded":"2024-09-21T13:53:09.591509+0000","last_fullsized":"2024-09-21T13:53:09.591509+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:12.939444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:12.939444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:31.497225+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":207,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.596122+0000","last_change":"2024-09-21T13:52:04.321133+0000","last_active":"2024-09-21T13:53:09.596122+0000","last_peered":"2024-09-21T13:53:09.596122+0000","last_clean":"2024-09-21T13:53:09.596122+0000","last_became_active":"2024-09-21T13:52:04.320826+0000","last_became_peered":"2024-09-21T13:52:04.320826+0000","last_unstale":"2024-09-21T13:53:09.596122+0000","last_undegraded":"2024-09-21T13:53:09.596122+0000","last_fullsized":"2024-09-21T13:53:09.596122+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:09.756023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"158'14312","reported_seq":14526,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:13.329154+0000","last_change":"2024-09-21T13:52:49.375801+0000","last_active":"2024-09-21T13:53:13.329154+0000","last_peered":"2024-09-21T13:53:13.329154+0000","last_clean":"2024-09-21T13:53:13.329154+0000","last_became_active":"2024-09-21T13:50:07.421666+0000","last_became_peered":"2024-09-21T13:50:07.421666+0000","last_unstale":"2024-09-21T13:53:13.329154+0000","last_undegraded":"2024-09-21T13:53:13.329154+0000","last_fullsized":"2024-09-21T13:53:13.329154+0000","mapping_epoch":83,"log_start":"117'8000","ondisk_log_start":"117'8000","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"149'12949","last_scrub_stamp":"2024-09-21T13:52:49.375671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:52:49.375671+0000","objects_scrubbed":4241,"log_size":6312,"log_dups_size":0,"ondisk_log_size":6312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:54.416521+0000","scrub_duration":1855,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186646528,"num_objects":2848,"num_object_clones":0,"num_object_copies":8544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14312,"num_write_kb":549120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"132'30","reported_seq":317,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:13.160985+0000","last_change":"2024-09-21T13:52:04.322738+0000","last_active":"2024-09-21T13:53:13.160985+0000","last_peered":"2024-09-21T13:53:13.160985+0000","last_clean":"2024-09-21T13:53:13.160985+0000","last_became_active":"2024-09-21T13:52:04.319431+0000","last_became_peered":"2024-09-21T13:52:04.319431+0000","last_unstale":"2024-09-21T13:53:13.160985+0000","last_undegraded":"2024-09-21T13:53:13.160985+0000","last_fullsized":"2024-09-21T13:53:13.160985+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:12.782459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":208,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.597532+0000","last_change":"2024-09-21T13:48:58.055640+0000","last_active":"2024-09-21T13:53:09.597532+0000","last_peered":"2024-09-21T13:53:09.597532+0000","last_clean":"2024-09-21T13:53:09.597532+0000","last_became_active":"2024-09-21T13:48:58.055366+0000","last_became_peered":"2024-09-21T13:48:58.055366+0000","last_unstale":"2024-09-21T13:53:09.597532+0000","last_undegraded":"2024-09-21T13:53:09.597532+0000","last_fullsized":"2024-09-21T13:53:09.597532+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:09.147679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":234,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.587184+0000","last_change":"2024-09-21T13:52:51.445882+0000","last_active":"2024-09-21T13:53:09.587184+0000","last_peered":"2024-09-21T13:53:09.587184+0000","last_clean":"2024-09-21T13:53:09.587184+0000","last_became_active":"2024-09-21T13:48:57.046246+0000","last_became_peered":"2024-09-21T13:48:57.046246+0000","last_unstale":"2024-09-21T13:53:09.587184+0000","last_undegraded":"2024-09-21T13:53:09.587184+0000","last_fullsized":"2024-09-21T13:53:09.587184+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-21T13:52:51.445812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:52:51.445812+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:18.734189+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.0015946669999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":148,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.602067+0000","last_change":"2024-09-21T13:51:24.187154+0000","last_active":"2024-09-21T13:53:09.602067+0000","last_peered":"2024-09-21T13:53:09.602067+0000","last_clean":"2024-09-21T13:53:09.602067+0000","last_became_active":"2024-09-21T13:50:29.688249+0000","last_became_peered":"2024-09-21T13:50:29.688249+0000","last_unstale":"2024-09-21T13:53:09.602067+0000","last_undegraded":"2024-09-21T13:53:09.602067+0000","last_fullsized":"2024-09-21T13:53:09.602067+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:24.186934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:24.186934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:33.380668+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":203,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.590945+0000","last_change":"2024-09-21T13:51:23.597164+0000","last_active":"2024-09-21T13:53:09.590945+0000","last_peered":"2024-09-21T13:53:09.590945+0000","last_clean":"2024-09-21T13:53:09.590945+0000","last_became_active":"2024-09-21T13:49:04.080466+0000","last_became_peered":"2024-09-21T13:49:04.080466+0000","last_unstale":"2024-09-21T13:53:09.590945+0000","last_undegraded":"2024-09-21T13:53:09.590945+0000","last_fullsized":"2024-09-21T13:53:09.590945+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:23.596879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:51:23.596879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"158'14421","reported_seq":14588,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:13.371692+0000","last_change":"2024-09-21T13:50:29.694240+0000","last_active":"2024-09-21T13:53:13.371692+0000","last_peered":"2024-09-21T13:53:13.371692+0000","last_clean":"2024-09-21T13:53:13.371692+0000","last_became_active":"2024-09-21T13:50:29.688167+0000","last_became_peered":"2024-09-21T13:50:29.688167+0000","last_unstale":"2024-09-21T13:53:13.371692+0000","last_undegraded":"2024-09-21T13:53:13.371692+0000","last_fullsized":"2024-09-21T13:53:13.371692+0000","mapping_epoch":94,"log_start":"125'9600","ondisk_log_start":"125'9600","created":83,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":4821,"log_dups_size":0,"ondisk_log_size":4821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189857792,"num_objects":2897,"num_object_clones":0,"num_object_copies":8691,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14421,"num_write_kb":554176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"68'13","reported_seq":238,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.598482+0000","last_change":"2024-09-21T13:50:06.973399+0000","last_active":"2024-09-21T13:53:09.598482+0000","last_peered":"2024-09-21T13:53:09.598482+0000","last_clean":"2024-09-21T13:53:09.598482+0000","last_became_active":"2024-09-21T13:49:00.053323+0000","last_became_peered":"2024-09-21T13:49:00.053323+0000","last_unstale":"2024-09-21T13:53:09.598482+0000","last_undegraded":"2024-09-21T13:53:09.598482+0000","last_fullsized":"2024-09-21T13:53:09.598482+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'13","last_scrub_stamp":"2024-09-21T13:50:06.973336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.973336+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:36.427623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":235,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.601660+0000","last_change":"2024-09-21T13:52:22.318613+0000","last_active":"2024-09-21T13:53:09.601660+0000","last_peered":"2024-09-21T13:53:09.601660+0000","last_clean":"2024-09-21T13:53:09.601660+0000","last_became_active":"2024-09-21T13:52:04.320581+0000","last_became_peered":"2024-09-21T13:52:04.320581+0000","last_unstale":"2024-09-21T13:53:09.601660+0000","last_undegraded":"2024-09-21T13:53:09.601660+0000","last_fullsized":"2024-09-21T13:53:09.601660+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:22.318545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:52:22.318545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:24.469330+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":134,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.597578+0000","last_change":"2024-09-21T13:52:04.316578+0000","last_active":"2024-09-21T13:53:09.597578+0000","last_peered":"2024-09-21T13:53:09.597578+0000","last_clean":"2024-09-21T13:53:09.597578+0000","last_became_active":"2024-09-21T13:52:04.316171+0000","last_became_peered":"2024-09-21T13:52:04.316171+0000","last_unstale":"2024-09-21T13:53:09.597578+0000","last_undegraded":"2024-09-21T13:53:09.597578+0000","last_fullsized":"2024-09-21T13:53:09.597578+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:49:56.756268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:49:56.756268+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":78,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.598027+0000","last_change":"2024-09-21T13:52:04.320997+0000","last_active":"2024-09-21T13:53:09.598027+0000","last_peered":"2024-09-21T13:53:09.598027+0000","last_clean":"2024-09-21T13:53:09.598027+0000","last_became_active":"2024-09-21T13:52:04.320701+0000","last_became_peered":"2024-09-21T13:52:04.320701+0000","last_unstale":"2024-09-21T13:53:09.598027+0000","last_undegraded":"2024-09-21T13:53:09.598027+0000","last_fullsized":"2024-09-21T13:53:09.598027+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":211,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.600350+0000","last_change":"2024-09-21T13:51:12.212558+0000","last_active":"2024-09-21T13:53:09.600350+0000","last_peered":"2024-09-21T13:53:09.600350+0000","last_clean":"2024-09-21T13:53:09.600350+0000","last_became_active":"2024-09-21T13:49:04.431035+0000","last_became_peered":"2024-09-21T13:49:04.431035+0000","last_unstale":"2024-09-21T13:53:09.600350+0000","last_undegraded":"2024-09-21T13:53:09.600350+0000","last_fullsized":"2024-09-21T13:53:09.600350+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:12.212391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:51:12.212391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"158'14240","reported_seq":24362,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:13.385309+0000","last_change":"2024-09-21T13:52:43.960019+0000","last_active":"2024-09-21T13:53:13.385309+0000","last_peered":"2024-09-21T13:53:13.385309+0000","last_clean":"2024-09-21T13:53:13.385309+0000","last_became_active":"2024-09-21T13:52:43.938223+0000","last_became_peered":"2024-09-21T13:52:43.938223+0000","last_unstale":"2024-09-21T13:53:13.385309+0000","last_undegraded":"2024-09-21T13:53:13.385309+0000","last_fullsized":"2024-09-21T13:53:13.385309+0000","mapping_epoch":147,"log_start":"123'9000","ondisk_log_start":"123'9000","created":83,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":5240,"log_dups_size":0,"ondisk_log_size":5240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184942592,"num_objects":2822,"num_object_clones":0,"num_object_copies":8466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14240,"num_write_kb":545984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4971,"num_bytes_recovered":325779456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"68'12","reported_seq":261,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.599454+0000","last_change":"2024-09-21T13:52:04.319341+0000","last_active":"2024-09-21T13:53:09.599454+0000","last_peered":"2024-09-21T13:53:09.599454+0000","last_clean":"2024-09-21T13:53:09.599454+0000","last_became_active":"2024-09-21T13:52:04.318875+0000","last_became_peered":"2024-09-21T13:52:04.318875+0000","last_unstale":"2024-09-21T13:53:09.599454+0000","last_undegraded":"2024-09-21T13:53:09.599454+0000","last_fullsized":"2024-09-21T13:53:09.599454+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'12","last_scrub_stamp":"2024-09-21T13:50:29.438468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:29.438468+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":148,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.595912+0000","last_change":"2024-09-21T13:50:30.020722+0000","last_active":"2024-09-21T13:53:09.595912+0000","last_peered":"2024-09-21T13:53:09.595912+0000","last_clean":"2024-09-21T13:53:09.595912+0000","last_became_active":"2024-09-21T13:50:29.688352+0000","last_became_peered":"2024-09-21T13:50:29.688352+0000","last_unstale":"2024-09-21T13:53:09.595912+0000","last_undegraded":"2024-09-21T13:53:09.595912+0000","last_fullsized":"2024-09-21T13:53:09.595912+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:30.020656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:30.020656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":477,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:12.252617+0000","last_change":"2024-09-21T13:51:21.271997+0000","last_active":"2024-09-21T13:53:12.252617+0000","last_peered":"2024-09-21T13:53:12.252617+0000","last_clean":"2024-09-21T13:53:12.252617+0000","last_became_active":"2024-09-21T13:48:57.046084+0000","last_became_peered":"2024-09-21T13:48:57.046084+0000","last_unstale":"2024-09-21T13:53:12.252617+0000","last_undegraded":"2024-09-21T13:53:12.252617+0000","last_fullsized":"2024-09-21T13:53:12.252617+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-21T13:51:21.271917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:51:21.271917+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00047048000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":250,"num_read_kb":245,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":207,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.588143+0000","last_change":"2024-09-21T13:51:53.518645+0000","last_active":"2024-09-21T13:53:09.588143+0000","last_peered":"2024-09-21T13:53:09.588143+0000","last_clean":"2024-09-21T13:53:09.588143+0000","last_became_active":"2024-09-21T13:49:02.066641+0000","last_became_peered":"2024-09-21T13:49:02.066641+0000","last_unstale":"2024-09-21T13:53:09.588143+0000","last_undegraded":"2024-09-21T13:53:09.588143+0000","last_fullsized":"2024-09-21T13:53:09.588143+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:53.518567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:53.518567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":212,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.596237+0000","last_change":"2024-09-21T13:52:21.332066+0000","last_active":"2024-09-21T13:53:09.596237+0000","last_peered":"2024-09-21T13:53:09.596237+0000","last_clean":"2024-09-21T13:53:09.596237+0000","last_became_active":"2024-09-21T13:49:04.078372+0000","last_became_peered":"2024-09-21T13:49:04.078372+0000","last_unstale":"2024-09-21T13:53:09.596237+0000","last_undegraded":"2024-09-21T13:53:09.596237+0000","last_fullsized":"2024-09-21T13:53:09.596237+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:21.331792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:21.331792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:44.913330+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"158'14342","reported_seq":14510,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:13.383552+0000","last_change":"2024-09-21T13:50:29.699865+0000","last_active":"2024-09-21T13:53:13.383552+0000","last_peered":"2024-09-21T13:53:13.383552+0000","last_clean":"2024-09-21T13:53:13.383552+0000","last_became_active":"2024-09-21T13:50:29.694886+0000","last_became_peered":"2024-09-21T13:50:29.694886+0000","last_unstale":"2024-09-21T13:53:13.383552+0000","last_undegraded":"2024-09-21T13:53:13.383552+0000","last_fullsized":"2024-09-21T13:53:13.383552+0000","mapping_epoch":94,"log_start":"123'9100","ondisk_log_start":"123'9100","created":83,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":5242,"log_dups_size":0,"ondisk_log_size":5242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186908672,"num_objects":2852,"num_object_clones":0,"num_object_copies":8556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14342,"num_write_kb":550208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"68'16","reported_seq":294,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:13.161346+0000","last_change":"2024-09-21T13:52:19.452239+0000","last_active":"2024-09-21T13:53:13.161346+0000","last_peered":"2024-09-21T13:53:13.161346+0000","last_clean":"2024-09-21T13:53:13.161346+0000","last_became_active":"2024-09-21T13:49:00.426867+0000","last_became_peered":"2024-09-21T13:49:00.426867+0000","last_unstale":"2024-09-21T13:53:13.161346+0000","last_undegraded":"2024-09-21T13:53:13.161346+0000","last_fullsized":"2024-09-21T13:53:13.161346+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'16","last_scrub_stamp":"2024-09-21T13:52:19.452163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:52:19.452163+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:46.305610+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":216,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.594153+0000","last_change":"2024-09-21T13:50:09.208053+0000","last_active":"2024-09-21T13:53:09.594153+0000","last_peered":"2024-09-21T13:53:09.594153+0000","last_clean":"2024-09-21T13:53:09.594153+0000","last_became_active":"2024-09-21T13:48:58.052659+0000","last_became_peered":"2024-09-21T13:48:58.052659+0000","last_unstale":"2024-09-21T13:53:09.594153+0000","last_undegraded":"2024-09-21T13:53:09.594153+0000","last_fullsized":"2024-09-21T13:53:09.594153+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:09.208003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:09.208003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"75'79","reported_seq":265,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.598327+0000","last_change":"2024-09-21T13:50:27.096470+0000","last_active":"2024-09-21T13:53:09.598327+0000","last_peered":"2024-09-21T13:53:09.598327+0000","last_clean":"2024-09-21T13:53:09.598327+0000","last_became_active":"2024-09-21T13:48:48.454646+0000","last_became_peered":"2024-09-21T13:48:48.454646+0000","last_unstale":"2024-09-21T13:53:09.598327+0000","last_undegraded":"2024-09-21T13:53:09.598327+0000","last_fullsized":"2024-09-21T13:53:09.598327+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:50:27.096413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:50:27.096413+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":211,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.600267+0000","last_change":"2024-09-21T13:52:04.320948+0000","last_active":"2024-09-21T13:53:09.600267+0000","last_peered":"2024-09-21T13:53:09.600267+0000","last_clean":"2024-09-21T13:53:09.600267+0000","last_became_active":"2024-09-21T13:52:04.320627+0000","last_became_peered":"2024-09-21T13:52:04.320627+0000","last_unstale":"2024-09-21T13:53:09.600267+0000","last_undegraded":"2024-09-21T13:53:09.600267+0000","last_fullsized":"2024-09-21T13:53:09.600267+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:42.557761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":203,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.587996+0000","last_change":"2024-09-21T13:50:25.560928+0000","last_active":"2024-09-21T13:53:09.587996+0000","last_peered":"2024-09-21T13:53:09.587996+0000","last_clean":"2024-09-21T13:53:09.587996+0000","last_became_active":"2024-09-21T13:49:04.078091+0000","last_became_peered":"2024-09-21T13:49:04.078091+0000","last_unstale":"2024-09-21T13:53:09.587996+0000","last_undegraded":"2024-09-21T13:53:09.587996+0000","last_fullsized":"2024-09-21T13:53:09.587996+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:25.560863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:25.560863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:40.548062+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"158'14250","reported_seq":14540,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:13.380667+0000","last_change":"2024-09-21T13:52:50.770818+0000","last_active":"2024-09-21T13:53:13.380667+0000","last_peered":"2024-09-21T13:53:13.380667+0000","last_clean":"2024-09-21T13:53:13.380667+0000","last_became_active":"2024-09-21T13:50:07.422466+0000","last_became_peered":"2024-09-21T13:50:07.422466+0000","last_unstale":"2024-09-21T13:53:13.380667+0000","last_undegraded":"2024-09-21T13:53:13.380667+0000","last_fullsized":"2024-09-21T13:53:13.380667+0000","mapping_epoch":83,"log_start":"116'7900","ondisk_log_start":"116'7900","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'12965","last_scrub_stamp":"2024-09-21T13:52:50.770670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:52:50.770670+0000","objects_scrubbed":4108,"log_size":6350,"log_dups_size":0,"ondisk_log_size":6350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:56.210477+0000","scrub_duration":1333,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182386712,"num_objects":2784,"num_object_clones":0,"num_object_copies":8352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2784,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14250,"num_write_kb":545025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"68'11","reported_seq":222,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.596263+0000","last_change":"2024-09-21T13:49:00.054827+0000","last_active":"2024-09-21T13:53:09.596263+0000","last_peered":"2024-09-21T13:53:09.596263+0000","last_clean":"2024-09-21T13:53:09.596263+0000","last_became_active":"2024-09-21T13:49:00.053268+0000","last_became_peered":"2024-09-21T13:49:00.053268+0000","last_unstale":"2024-09-21T13:53:09.596263+0000","last_undegraded":"2024-09-21T13:53:09.596263+0000","last_fullsized":"2024-09-21T13:53:09.596263+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":226,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.596115+0000","last_change":"2024-09-21T13:52:48.165210+0000","last_active":"2024-09-21T13:53:09.596115+0000","last_peered":"2024-09-21T13:53:09.596115+0000","last_clean":"2024-09-21T13:53:09.596115+0000","last_became_active":"2024-09-21T13:52:04.318345+0000","last_became_peered":"2024-09-21T13:52:04.318345+0000","last_unstale":"2024-09-21T13:53:09.596115+0000","last_undegraded":"2024-09-21T13:53:09.596115+0000","last_fullsized":"2024-09-21T13:53:09.596115+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:48.165145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:52:48.165145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:03.774580+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":215,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.587978+0000","last_change":"2024-09-21T13:51:46.503640+0000","last_active":"2024-09-21T13:53:09.587978+0000","last_peered":"2024-09-21T13:53:09.587978+0000","last_clean":"2024-09-21T13:53:09.587978+0000","last_became_active":"2024-09-21T13:49:02.061901+0000","last_became_peered":"2024-09-21T13:49:02.061901+0000","last_unstale":"2024-09-21T13:53:09.587978+0000","last_undegraded":"2024-09-21T13:53:09.587978+0000","last_fullsized":"2024-09-21T13:53:09.587978+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:46.503576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:46.503576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":203,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.591328+0000","last_change":"2024-09-21T13:51:51.814259+0000","last_active":"2024-09-21T13:53:09.591328+0000","last_peered":"2024-09-21T13:53:09.591328+0000","last_clean":"2024-09-21T13:53:09.591328+0000","last_became_active":"2024-09-21T13:49:04.431268+0000","last_became_peered":"2024-09-21T13:49:04.431268+0000","last_unstale":"2024-09-21T13:53:09.591328+0000","last_undegraded":"2024-09-21T13:53:09.591328+0000","last_fullsized":"2024-09-21T13:53:09.591328+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:51.814200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:51:51.814200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:09.956776+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"158'14260","reported_seq":21102,"reported_epoch":158,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-21T13:53:13.670131+0000","last_change":"2024-09-21T13:52:42.230831+0000","last_active":"2024-09-21T13:53:13.670131+0000","last_peered":"2024-09-21T13:53:13.670131+0000","last_clean":"2024-09-21T13:50:42.883597+0000","last_became_active":"2024-09-21T13:52:04.974475+0000","last_became_peered":"2024-09-21T13:52:04.974475+0000","last_unstale":"2024-09-21T13:53:13.670131+0000","last_undegraded":"2024-09-21T13:52:04.880075+0000","last_fullsized":"2024-09-21T13:52:04.865259+0000","mapping_epoch":131,"log_start":"122'8688","ondisk_log_start":"122'8688","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":5572,"log_dups_size":0,"ondisk_log_size":5572,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178913280,"num_objects":2730,"num_object_clones":0,"num_object_copies":8190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":277,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14260,"num_write_kb":543680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3335,"num_bytes_recovered":218562560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[{"shards":"0,5","objects":2730}],"blocked_by":[],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"68'17","reported_seq":250,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.596393+0000","last_change":"2024-09-21T13:52:45.176950+0000","last_active":"2024-09-21T13:53:09.596393+0000","last_peered":"2024-09-21T13:53:09.596393+0000","last_clean":"2024-09-21T13:53:09.596393+0000","last_became_active":"2024-09-21T13:52:04.313218+0000","last_became_peered":"2024-09-21T13:52:04.313218+0000","last_unstale":"2024-09-21T13:53:09.596393+0000","last_undegraded":"2024-09-21T13:53:09.596393+0000","last_fullsized":"2024-09-21T13:53:09.596393+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'17","last_scrub_stamp":"2024-09-21T13:52:45.176873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:52:45.176873+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:03.832378+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":223,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.591720+0000","last_change":"2024-09-21T13:52:45.444445+0000","last_active":"2024-09-21T13:53:09.591720+0000","last_peered":"2024-09-21T13:53:09.591720+0000","last_clean":"2024-09-21T13:53:09.591720+0000","last_became_active":"2024-09-21T13:48:58.049142+0000","last_became_peered":"2024-09-21T13:48:58.049142+0000","last_unstale":"2024-09-21T13:53:09.591720+0000","last_undegraded":"2024-09-21T13:53:09.591720+0000","last_fullsized":"2024-09-21T13:53:09.591720+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:45.444381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:52:45.444381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:54.348542+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":199,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.596283+0000","last_change":"2024-09-21T13:49:02.067811+0000","last_active":"2024-09-21T13:53:09.596283+0000","last_peered":"2024-09-21T13:53:09.596283+0000","last_clean":"2024-09-21T13:53:09.596283+0000","last_became_active":"2024-09-21T13:49:02.067472+0000","last_became_peered":"2024-09-21T13:49:02.067472+0000","last_unstale":"2024-09-21T13:53:09.596283+0000","last_undegraded":"2024-09-21T13:53:09.596283+0000","last_fullsized":"2024-09-21T13:53:09.596283+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:02.701863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":78,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.596713+0000","last_change":"2024-09-21T13:52:04.316322+0000","last_active":"2024-09-21T13:53:09.596713+0000","last_peered":"2024-09-21T13:53:09.596713+0000","last_clean":"2024-09-21T13:53:09.596713+0000","last_became_active":"2024-09-21T13:52:04.315920+0000","last_became_peered":"2024-09-21T13:52:04.315920+0000","last_unstale":"2024-09-21T13:53:09.596713+0000","last_undegraded":"2024-09-21T13:53:09.596713+0000","last_fullsized":"2024-09-21T13:53:09.596713+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"158'13931","reported_seq":14100,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:13.881727+0000","last_change":"2024-09-21T13:50:29.691429+0000","last_active":"2024-09-21T13:53:13.881727+0000","last_peered":"2024-09-21T13:53:13.881727+0000","last_clean":"2024-09-21T13:53:13.881727+0000","last_became_active":"2024-09-21T13:50:29.687898+0000","last_became_peered":"2024-09-21T13:50:29.687898+0000","last_unstale":"2024-09-21T13:53:13.881727+0000","last_undegraded":"2024-09-21T13:53:13.881727+0000","last_fullsized":"2024-09-21T13:53:13.881727+0000","mapping_epoch":94,"log_start":"126'9300","ondisk_log_start":"126'9300","created":83,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":4631,"log_dups_size":0,"ondisk_log_size":4631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175308800,"num_objects":2675,"num_object_clones":0,"num_object_copies":8025,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13931,"num_write_kb":531392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"68'10","reported_seq":226,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.596474+0000","last_change":"2024-09-21T13:51:57.155811+0000","last_active":"2024-09-21T13:53:09.596474+0000","last_peered":"2024-09-21T13:53:09.596474+0000","last_clean":"2024-09-21T13:53:09.596474+0000","last_became_active":"2024-09-21T13:49:00.059850+0000","last_became_peered":"2024-09-21T13:49:00.059850+0000","last_unstale":"2024-09-21T13:53:09.596474+0000","last_undegraded":"2024-09-21T13:53:09.596474+0000","last_fullsized":"2024-09-21T13:53:09.596474+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:51:57.155761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:57.155761+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:12.086769+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":215,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.601231+0000","last_change":"2024-09-21T13:51:28.565842+0000","last_active":"2024-09-21T13:53:09.601231+0000","last_peered":"2024-09-21T13:53:09.601231+0000","last_clean":"2024-09-21T13:53:09.601231+0000","last_became_active":"2024-09-21T13:48:58.058309+0000","last_became_peered":"2024-09-21T13:48:58.058309+0000","last_unstale":"2024-09-21T13:53:09.601231+0000","last_undegraded":"2024-09-21T13:53:09.601231+0000","last_fullsized":"2024-09-21T13:53:09.601231+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:28.565766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:51:28.565766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"132'10","reported_seq":110,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.597713+0000","last_change":"2024-09-21T13:52:04.320834+0000","last_active":"2024-09-21T13:53:09.597713+0000","last_peered":"2024-09-21T13:53:09.597713+0000","last_clean":"2024-09-21T13:53:09.597713+0000","last_became_active":"2024-09-21T13:52:04.320441+0000","last_became_peered":"2024-09-21T13:52:04.320441+0000","last_unstale":"2024-09-21T13:53:09.597713+0000","last_undegraded":"2024-09-21T13:53:09.597713+0000","last_fullsized":"2024-09-21T13:53:09.597713+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"68'1","reported_seq":218,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.596106+0000","last_change":"2024-09-21T13:52:04.314151+0000","last_active":"2024-09-21T13:53:09.596106+0000","last_peered":"2024-09-21T13:53:09.596106+0000","last_clean":"2024-09-21T13:53:09.596106+0000","last_became_active":"2024-09-21T13:52:04.313210+0000","last_became_peered":"2024-09-21T13:52:04.313210+0000","last_unstale":"2024-09-21T13:53:09.596106+0000","last_undegraded":"2024-09-21T13:53:09.596106+0000","last_fullsized":"2024-09-21T13:53:09.596106+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T13:50:41.000201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:41.000201+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"158'14136","reported_seq":24197,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:12.424170+0000","last_change":"2024-09-21T13:52:42.978682+0000","last_active":"2024-09-21T13:53:12.424170+0000","last_peered":"2024-09-21T13:53:12.424170+0000","last_clean":"2024-09-21T13:53:12.424170+0000","last_became_active":"2024-09-21T13:52:42.942510+0000","last_became_peered":"2024-09-21T13:52:42.942510+0000","last_unstale":"2024-09-21T13:53:12.424170+0000","last_undegraded":"2024-09-21T13:53:12.424170+0000","last_fullsized":"2024-09-21T13:53:12.424170+0000","mapping_epoch":146,"log_start":"116'7778","ondisk_log_start":"116'7778","created":83,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":6358,"log_dups_size":0,"ondisk_log_size":6358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187564032,"num_objects":2862,"num_object_clones":0,"num_object_copies":8586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14136,"num_write_kb":543936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4942,"num_bytes_recovered":323878912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"68'9","reported_seq":227,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.588682+0000","last_change":"2024-09-21T13:51:24.800926+0000","last_active":"2024-09-21T13:53:09.588682+0000","last_peered":"2024-09-21T13:53:09.588682+0000","last_clean":"2024-09-21T13:53:09.588682+0000","last_became_active":"2024-09-21T13:49:00.053016+0000","last_became_peered":"2024-09-21T13:49:00.053016+0000","last_unstale":"2024-09-21T13:53:09.588682+0000","last_undegraded":"2024-09-21T13:53:09.588682+0000","last_fullsized":"2024-09-21T13:53:09.588682+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:51:24.800844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:24.800844+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":215,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.587503+0000","last_change":"2024-09-21T13:50:38.445732+0000","last_active":"2024-09-21T13:53:09.587503+0000","last_peered":"2024-09-21T13:53:09.587503+0000","last_clean":"2024-09-21T13:53:09.587503+0000","last_became_active":"2024-09-21T13:48:58.049543+0000","last_became_peered":"2024-09-21T13:48:58.049543+0000","last_unstale":"2024-09-21T13:53:09.587503+0000","last_undegraded":"2024-09-21T13:53:09.587503+0000","last_fullsized":"2024-09-21T13:53:09.587503+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:38.445667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:38.445667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:35.568002+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":138,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.595781+0000","last_change":"2024-09-21T13:52:59.112648+0000","last_active":"2024-09-21T13:53:09.595781+0000","last_peered":"2024-09-21T13:53:09.595781+0000","last_clean":"2024-09-21T13:53:09.595781+0000","last_became_active":"2024-09-21T13:52:04.317429+0000","last_became_peered":"2024-09-21T13:52:04.317429+0000","last_unstale":"2024-09-21T13:53:09.595781+0000","last_undegraded":"2024-09-21T13:53:09.595781+0000","last_fullsized":"2024-09-21T13:53:09.595781+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:59.112602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:52:59.112602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:59.206422+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":159,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.598815+0000","last_change":"2024-09-21T13:52:51.590860+0000","last_active":"2024-09-21T13:53:09.598815+0000","last_peered":"2024-09-21T13:53:09.598815+0000","last_clean":"2024-09-21T13:53:09.598815+0000","last_became_active":"2024-09-21T13:52:04.321703+0000","last_became_peered":"2024-09-21T13:52:04.321703+0000","last_unstale":"2024-09-21T13:53:09.598815+0000","last_undegraded":"2024-09-21T13:53:09.598815+0000","last_fullsized":"2024-09-21T13:53:09.598815+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:51.590800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:51.590800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:01.244154+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.9","version":"158'14240","reported_seq":14409,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:13.494452+0000","last_change":"2024-09-21T13:50:29.690666+0000","last_active":"2024-09-21T13:53:13.494452+0000","last_peered":"2024-09-21T13:53:13.494452+0000","last_clean":"2024-09-21T13:53:13.494452+0000","last_became_active":"2024-09-21T13:50:29.687815+0000","last_became_peered":"2024-09-21T13:50:29.687815+0000","last_unstale":"2024-09-21T13:53:13.494452+0000","last_undegraded":"2024-09-21T13:53:13.494452+0000","last_fullsized":"2024-09-21T13:53:13.494452+0000","mapping_epoch":94,"log_start":"124'9200","ondisk_log_start":"124'9200","created":83,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":5040,"log_dups_size":0,"ondisk_log_size":5040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:32.358031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181141504,"num_objects":2764,"num_object_clones":0,"num_object_copies":8292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14240,"num_write_kb":544128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"68'19","reported_seq":242,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.591540+0000","last_change":"2024-09-21T13:50:03.769686+0000","last_active":"2024-09-21T13:53:09.591540+0000","last_peered":"2024-09-21T13:53:09.591540+0000","last_clean":"2024-09-21T13:53:09.591540+0000","last_became_active":"2024-09-21T13:49:00.425955+0000","last_became_peered":"2024-09-21T13:49:00.425955+0000","last_unstale":"2024-09-21T13:53:09.591540+0000","last_undegraded":"2024-09-21T13:53:09.591540+0000","last_fullsized":"2024-09-21T13:53:09.591540+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-21T13:50:03.769613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:03.769613+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:32.476093+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":207,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.600555+0000","last_change":"2024-09-21T13:48:58.053940+0000","last_active":"2024-09-21T13:53:09.600555+0000","last_peered":"2024-09-21T13:53:09.600555+0000","last_clean":"2024-09-21T13:53:09.600555+0000","last_became_active":"2024-09-21T13:48:58.053641+0000","last_became_peered":"2024-09-21T13:48:58.053641+0000","last_unstale":"2024-09-21T13:53:09.600555+0000","last_undegraded":"2024-09-21T13:53:09.600555+0000","last_fullsized":"2024-09-21T13:53:09.600555+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":156,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.597620+0000","last_change":"2024-09-21T13:52:04.321471+0000","last_active":"2024-09-21T13:53:09.597620+0000","last_peered":"2024-09-21T13:53:09.597620+0000","last_clean":"2024-09-21T13:53:09.597620+0000","last_became_active":"2024-09-21T13:52:04.321146+0000","last_became_peered":"2024-09-21T13:52:04.321146+0000","last_unstale":"2024-09-21T13:53:09.597620+0000","last_undegraded":"2024-09-21T13:53:09.597620+0000","last_fullsized":"2024-09-21T13:53:09.597620+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:40.850721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:40.850721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":214,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.598251+0000","last_change":"2024-09-21T13:52:04.323179+0000","last_active":"2024-09-21T13:53:09.598251+0000","last_peered":"2024-09-21T13:53:09.598251+0000","last_clean":"2024-09-21T13:53:09.598251+0000","last_became_active":"2024-09-21T13:52:04.322213+0000","last_became_peered":"2024-09-21T13:52:04.322213+0000","last_unstale":"2024-09-21T13:53:09.598251+0000","last_undegraded":"2024-09-21T13:53:09.598251+0000","last_fullsized":"2024-09-21T13:53:09.598251+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:35.067261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:35.067261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"158'14244","reported_seq":23498,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:13.883387+0000","last_change":"2024-09-21T13:53:01.956068+0000","last_active":"2024-09-21T13:53:13.883387+0000","last_peered":"2024-09-21T13:53:13.883387+0000","last_clean":"2024-09-21T13:53:13.883387+0000","last_became_active":"2024-09-21T13:53:01.911422+0000","last_became_peered":"2024-09-21T13:53:01.911422+0000","last_unstale":"2024-09-21T13:53:13.883387+0000","last_undegraded":"2024-09-21T13:53:13.883387+0000","last_fullsized":"2024-09-21T13:53:13.883387+0000","mapping_epoch":153,"log_start":"126'9600","ondisk_log_start":"126'9600","created":83,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":4644,"log_dups_size":0,"ondisk_log_size":4644,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181141504,"num_objects":2764,"num_object_clones":0,"num_object_copies":8292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14244,"num_write_kb":544256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4537,"num_bytes_recovered":297336832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"68'12","reported_seq":234,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.596014+0000","last_change":"2024-09-21T13:51:10.942190+0000","last_active":"2024-09-21T13:53:09.596014+0000","last_peered":"2024-09-21T13:53:09.596014+0000","last_clean":"2024-09-21T13:53:09.596014+0000","last_became_active":"2024-09-21T13:49:00.059720+0000","last_became_peered":"2024-09-21T13:49:00.059720+0000","last_unstale":"2024-09-21T13:53:09.596014+0000","last_undegraded":"2024-09-21T13:53:09.596014+0000","last_fullsized":"2024-09-21T13:53:09.596014+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'12","last_scrub_stamp":"2024-09-21T13:51:10.942105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:10.942105+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":215,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.600986+0000","last_change":"2024-09-21T13:50:29.100595+0000","last_active":"2024-09-21T13:53:09.600986+0000","last_peered":"2024-09-21T13:53:09.600986+0000","last_clean":"2024-09-21T13:53:09.600986+0000","last_became_active":"2024-09-21T13:48:58.057641+0000","last_became_peered":"2024-09-21T13:48:58.057641+0000","last_unstale":"2024-09-21T13:53:09.600986+0000","last_undegraded":"2024-09-21T13:53:09.600986+0000","last_fullsized":"2024-09-21T13:53:09.600986+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:29.100547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:29.100547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":121,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.598393+0000","last_change":"2024-09-21T13:52:04.322501+0000","last_active":"2024-09-21T13:53:09.598393+0000","last_peered":"2024-09-21T13:53:09.598393+0000","last_clean":"2024-09-21T13:53:09.598393+0000","last_became_active":"2024-09-21T13:52:04.322147+0000","last_became_peered":"2024-09-21T13:52:04.322147+0000","last_unstale":"2024-09-21T13:53:09.598393+0000","last_undegraded":"2024-09-21T13:53:09.598393+0000","last_fullsized":"2024-09-21T13:53:09.598393+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:02.761516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:02.761516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":195,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.600903+0000","last_change":"2024-09-21T13:49:04.078422+0000","last_active":"2024-09-21T13:53:09.600903+0000","last_peered":"2024-09-21T13:53:09.600903+0000","last_clean":"2024-09-21T13:53:09.600903+0000","last_became_active":"2024-09-21T13:49:04.078089+0000","last_became_peered":"2024-09-21T13:49:04.078089+0000","last_unstale":"2024-09-21T13:53:09.600903+0000","last_undegraded":"2024-09-21T13:53:09.600903+0000","last_fullsized":"2024-09-21T13:53:09.600903+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"158'14384","reported_seq":14546,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:13.509295+0000","last_change":"2024-09-21T13:50:07.422240+0000","last_active":"2024-09-21T13:53:13.509295+0000","last_peered":"2024-09-21T13:53:13.509295+0000","last_clean":"2024-09-21T13:53:13.509295+0000","last_became_active":"2024-09-21T13:50:07.421962+0000","last_became_peered":"2024-09-21T13:50:07.421962+0000","last_unstale":"2024-09-21T13:53:13.509295+0000","last_undegraded":"2024-09-21T13:53:13.509295+0000","last_fullsized":"2024-09-21T13:53:13.509295+0000","mapping_epoch":83,"log_start":"124'9300","ondisk_log_start":"124'9300","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":5084,"log_dups_size":0,"ondisk_log_size":5084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:37.396702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184811520,"num_objects":2820,"num_object_clones":0,"num_object_copies":8460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14384,"num_write_kb":550528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"68'15","reported_seq":236,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.586719+0000","last_change":"2024-09-21T13:50:04.445409+0000","last_active":"2024-09-21T13:53:09.586719+0000","last_peered":"2024-09-21T13:53:09.586719+0000","last_clean":"2024-09-21T13:53:09.586719+0000","last_became_active":"2024-09-21T13:49:00.425997+0000","last_became_peered":"2024-09-21T13:49:00.425997+0000","last_unstale":"2024-09-21T13:53:09.586719+0000","last_undegraded":"2024-09-21T13:53:09.586719+0000","last_fullsized":"2024-09-21T13:53:09.586719+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:50:04.445328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:04.445328+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:30.619277+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":227,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.600467+0000","last_change":"2024-09-21T13:50:08.119154+0000","last_active":"2024-09-21T13:53:09.600467+0000","last_peered":"2024-09-21T13:53:09.600467+0000","last_clean":"2024-09-21T13:53:09.600467+0000","last_became_active":"2024-09-21T13:48:58.057632+0000","last_became_peered":"2024-09-21T13:48:58.057632+0000","last_unstale":"2024-09-21T13:53:09.600467+0000","last_undegraded":"2024-09-21T13:53:09.600467+0000","last_fullsized":"2024-09-21T13:53:09.600467+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T13:50:08.119099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:08.119099+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"68'8","reported_seq":267,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.598981+0000","last_change":"2024-09-21T13:51:26.297643+0000","last_active":"2024-09-21T13:53:09.598981+0000","last_peered":"2024-09-21T13:53:09.598981+0000","last_clean":"2024-09-21T13:53:09.598981+0000","last_became_active":"2024-09-21T13:49:02.056167+0000","last_became_peered":"2024-09-21T13:49:02.056167+0000","last_unstale":"2024-09-21T13:53:09.598981+0000","last_undegraded":"2024-09-21T13:53:09.598981+0000","last_fullsized":"2024-09-21T13:53:09.598981+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:51:26.297550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:26.297550+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":211,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.586631+0000","last_change":"2024-09-21T13:52:54.314087+0000","last_active":"2024-09-21T13:53:09.586631+0000","last_peered":"2024-09-21T13:53:09.586631+0000","last_clean":"2024-09-21T13:53:09.586631+0000","last_became_active":"2024-09-21T13:49:04.432546+0000","last_became_peered":"2024-09-21T13:49:04.432546+0000","last_unstale":"2024-09-21T13:53:09.586631+0000","last_undegraded":"2024-09-21T13:53:09.586631+0000","last_fullsized":"2024-09-21T13:53:09.586631+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:54.314042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:54.314042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:05.575071+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":223,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.591450+0000","last_change":"2024-09-21T13:51:54.352313+0000","last_active":"2024-09-21T13:53:09.591450+0000","last_peered":"2024-09-21T13:53:09.591450+0000","last_clean":"2024-09-21T13:53:09.591450+0000","last_became_active":"2024-09-21T13:48:58.056080+0000","last_became_peered":"2024-09-21T13:48:58.056080+0000","last_unstale":"2024-09-21T13:53:09.591450+0000","last_undegraded":"2024-09-21T13:53:09.591450+0000","last_fullsized":"2024-09-21T13:53:09.591450+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:54.352244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:51:54.352244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"68'6","reported_seq":212,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.612975+0000","last_change":"2024-09-21T13:49:00.427204+0000","last_active":"2024-09-21T13:53:09.612975+0000","last_peered":"2024-09-21T13:53:09.612975+0000","last_clean":"2024-09-21T13:53:09.612975+0000","last_became_active":"2024-09-21T13:49:00.426829+0000","last_became_peered":"2024-09-21T13:49:00.426829+0000","last_unstale":"2024-09-21T13:53:09.612975+0000","last_undegraded":"2024-09-21T13:53:09.612975+0000","last_fullsized":"2024-09-21T13:53:09.612975+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":215,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.587761+0000","last_change":"2024-09-21T13:51:55.488973+0000","last_active":"2024-09-21T13:53:09.587761+0000","last_peered":"2024-09-21T13:53:09.587761+0000","last_clean":"2024-09-21T13:53:09.587761+0000","last_became_active":"2024-09-21T13:49:02.062231+0000","last_became_peered":"2024-09-21T13:49:02.062231+0000","last_unstale":"2024-09-21T13:53:09.587761+0000","last_undegraded":"2024-09-21T13:53:09.587761+0000","last_fullsized":"2024-09-21T13:53:09.587761+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:55.488908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:55.488908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":203,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.600265+0000","last_change":"2024-09-21T13:50:06.158899+0000","last_active":"2024-09-21T13:53:09.600265+0000","last_peered":"2024-09-21T13:53:09.600265+0000","last_clean":"2024-09-21T13:53:09.600265+0000","last_became_active":"2024-09-21T13:49:04.432507+0000","last_became_peered":"2024-09-21T13:49:04.432507+0000","last_unstale":"2024-09-21T13:53:09.600265+0000","last_undegraded":"2024-09-21T13:53:09.600265+0000","last_fullsized":"2024-09-21T13:53:09.600265+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.158822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.158822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"68'9","reported_seq":184,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.591092+0000","last_change":"2024-09-21T13:51:54.808758+0000","last_active":"2024-09-21T13:53:09.591092+0000","last_peered":"2024-09-21T13:53:09.591092+0000","last_clean":"2024-09-21T13:53:09.591092+0000","last_became_active":"2024-09-21T13:50:29.693589+0000","last_became_peered":"2024-09-21T13:50:29.693589+0000","last_unstale":"2024-09-21T13:53:09.591092+0000","last_undegraded":"2024-09-21T13:53:09.591092+0000","last_fullsized":"2024-09-21T13:53:09.591092+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:51:54.808559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:54.808559+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:18.303211+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"61'1","reported_seq":213,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.599105+0000","last_change":"2024-09-21T13:48:58.050948+0000","last_active":"2024-09-21T13:53:09.599105+0000","last_peered":"2024-09-21T13:53:09.599105+0000","last_clean":"2024-09-21T13:53:09.599105+0000","last_became_active":"2024-09-21T13:48:58.050645+0000","last_became_peered":"2024-09-21T13:48:58.050645+0000","last_unstale":"2024-09-21T13:53:09.599105+0000","last_undegraded":"2024-09-21T13:53:09.599105+0000","last_fullsized":"2024-09-21T13:53:09.599105+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":207,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.612833+0000","last_change":"2024-09-21T13:50:43.375425+0000","last_active":"2024-09-21T13:53:09.612833+0000","last_peered":"2024-09-21T13:53:09.612833+0000","last_clean":"2024-09-21T13:53:09.612833+0000","last_became_active":"2024-09-21T13:49:02.061775+0000","last_became_peered":"2024-09-21T13:49:02.061775+0000","last_unstale":"2024-09-21T13:53:09.612833+0000","last_undegraded":"2024-09-21T13:53:09.612833+0000","last_fullsized":"2024-09-21T13:53:09.612833+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:43.375381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:43.375381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":152,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.597074+0000","last_change":"2024-09-21T13:52:04.324521+0000","last_active":"2024-09-21T13:53:09.597074+0000","last_peered":"2024-09-21T13:53:09.597074+0000","last_clean":"2024-09-21T13:53:09.597074+0000","last_became_active":"2024-09-21T13:52:04.323549+0000","last_became_peered":"2024-09-21T13:52:04.323549+0000","last_unstale":"2024-09-21T13:53:09.597074+0000","last_undegraded":"2024-09-21T13:53:09.597074+0000","last_fullsized":"2024-09-21T13:53:09.597074+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:41.808746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:41.808746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"68'9","reported_seq":227,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.587919+0000","last_change":"2024-09-21T13:51:28.547052+0000","last_active":"2024-09-21T13:53:09.587919+0000","last_peered":"2024-09-21T13:53:09.587919+0000","last_clean":"2024-09-21T13:53:09.587919+0000","last_became_active":"2024-09-21T13:49:00.427766+0000","last_became_peered":"2024-09-21T13:49:00.427766+0000","last_unstale":"2024-09-21T13:53:09.587919+0000","last_undegraded":"2024-09-21T13:53:09.587919+0000","last_fullsized":"2024-09-21T13:53:09.587919+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:51:28.546958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:28.546958+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:08.329203+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":148,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.599568+0000","last_change":"2024-09-21T13:50:30.076191+0000","last_active":"2024-09-21T13:53:09.599568+0000","last_peered":"2024-09-21T13:53:09.599568+0000","last_clean":"2024-09-21T13:53:09.599568+0000","last_became_active":"2024-09-21T13:50:29.690634+0000","last_became_peered":"2024-09-21T13:50:29.690634+0000","last_unstale":"2024-09-21T13:53:09.599568+0000","last_undegraded":"2024-09-21T13:53:09.599568+0000","last_fullsized":"2024-09-21T13:53:09.599568+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:30.076115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:30.076115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"132'10","reported_seq":280,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.591500+0000","last_change":"2024-09-21T13:52:04.319377+0000","last_active":"2024-09-21T13:53:09.591500+0000","last_peered":"2024-09-21T13:53:09.591500+0000","last_clean":"2024-09-21T13:53:09.591500+0000","last_became_active":"2024-09-21T13:52:04.318626+0000","last_became_peered":"2024-09-21T13:52:04.318626+0000","last_unstale":"2024-09-21T13:53:09.591500+0000","last_undegraded":"2024-09-21T13:53:09.591500+0000","last_fullsized":"2024-09-21T13:53:09.591500+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:50:29.249680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:29.249680+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":214,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.596260+0000","last_change":"2024-09-21T13:52:04.316179+0000","last_active":"2024-09-21T13:53:09.596260+0000","last_peered":"2024-09-21T13:53:09.596260+0000","last_clean":"2024-09-21T13:53:09.596260+0000","last_became_active":"2024-09-21T13:52:04.315865+0000","last_became_peered":"2024-09-21T13:52:04.315865+0000","last_unstale":"2024-09-21T13:53:09.596260+0000","last_undegraded":"2024-09-21T13:53:09.596260+0000","last_fullsized":"2024-09-21T13:53:09.596260+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:38.983326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:38.983326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"68'10","reported_seq":226,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.595737+0000","last_change":"2024-09-21T13:51:11.545546+0000","last_active":"2024-09-21T13:53:09.595737+0000","last_peered":"2024-09-21T13:53:09.595737+0000","last_clean":"2024-09-21T13:53:09.595737+0000","last_became_active":"2024-09-21T13:49:00.426071+0000","last_became_peered":"2024-09-21T13:49:00.426071+0000","last_unstale":"2024-09-21T13:53:09.595737+0000","last_undegraded":"2024-09-21T13:53:09.595737+0000","last_fullsized":"2024-09-21T13:53:09.595737+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:51:11.545458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:11.545458+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":226,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.599962+0000","last_change":"2024-09-21T13:52:04.318335+0000","last_active":"2024-09-21T13:53:09.599962+0000","last_peered":"2024-09-21T13:53:09.599962+0000","last_clean":"2024-09-21T13:53:09.599962+0000","last_became_active":"2024-09-21T13:52:04.318006+0000","last_became_peered":"2024-09-21T13:52:04.318006+0000","last_unstale":"2024-09-21T13:53:09.599962+0000","last_undegraded":"2024-09-21T13:53:09.599962+0000","last_fullsized":"2024-09-21T13:53:09.599962+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:41.162315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:41.162315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"68'8","reported_seq":259,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.587773+0000","last_change":"2024-09-21T13:49:02.067827+0000","last_active":"2024-09-21T13:53:09.587773+0000","last_peered":"2024-09-21T13:53:09.587773+0000","last_clean":"2024-09-21T13:53:09.587773+0000","last_became_active":"2024-09-21T13:49:02.067487+0000","last_became_peered":"2024-09-21T13:49:02.067487+0000","last_unstale":"2024-09-21T13:53:09.587773+0000","last_undegraded":"2024-09-21T13:53:09.587773+0000","last_fullsized":"2024-09-21T13:53:09.587773+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:02.411463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":148,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.600729+0000","last_change":"2024-09-21T13:50:33.050276+0000","last_active":"2024-09-21T13:53:09.600729+0000","last_peered":"2024-09-21T13:53:09.600729+0000","last_clean":"2024-09-21T13:53:09.600729+0000","last_became_active":"2024-09-21T13:50:29.695082+0000","last_became_peered":"2024-09-21T13:50:29.695082+0000","last_unstale":"2024-09-21T13:53:09.600729+0000","last_undegraded":"2024-09-21T13:53:09.600729+0000","last_fullsized":"2024-09-21T13:53:09.600729+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:33.050227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:33.050227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"68'11","reported_seq":230,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.597314+0000","last_change":"2024-09-21T13:50:02.673500+0000","last_active":"2024-09-21T13:53:09.597314+0000","last_peered":"2024-09-21T13:53:09.597314+0000","last_clean":"2024-09-21T13:53:09.597314+0000","last_became_active":"2024-09-21T13:49:00.055024+0000","last_became_peered":"2024-09-21T13:49:00.055024+0000","last_unstale":"2024-09-21T13:53:09.597314+0000","last_undegraded":"2024-09-21T13:53:09.597314+0000","last_fullsized":"2024-09-21T13:53:09.597314+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:50:02.673442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:02.673442+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":215,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.596638+0000","last_change":"2024-09-21T13:51:20.956291+0000","last_active":"2024-09-21T13:53:09.596638+0000","last_peered":"2024-09-21T13:53:09.596638+0000","last_clean":"2024-09-21T13:53:09.596638+0000","last_became_active":"2024-09-21T13:48:58.051541+0000","last_became_peered":"2024-09-21T13:48:58.051541+0000","last_unstale":"2024-09-21T13:53:09.596638+0000","last_undegraded":"2024-09-21T13:53:09.596638+0000","last_fullsized":"2024-09-21T13:53:09.596638+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:20.956226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:51:20.956226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":215,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.596177+0000","last_change":"2024-09-21T13:52:14.465671+0000","last_active":"2024-09-21T13:53:09.596177+0000","last_peered":"2024-09-21T13:53:09.596177+0000","last_clean":"2024-09-21T13:53:09.596177+0000","last_became_active":"2024-09-21T13:49:02.062868+0000","last_became_peered":"2024-09-21T13:49:02.062868+0000","last_unstale":"2024-09-21T13:53:09.596177+0000","last_undegraded":"2024-09-21T13:53:09.596177+0000","last_fullsized":"2024-09-21T13:53:09.596177+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:14.465434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:52:14.465434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:25.424869+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":203,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.590605+0000","last_change":"2024-09-21T13:51:24.549993+0000","last_active":"2024-09-21T13:53:09.590605+0000","last_peered":"2024-09-21T13:53:09.590605+0000","last_clean":"2024-09-21T13:53:09.590605+0000","last_became_active":"2024-09-21T13:49:04.082961+0000","last_became_peered":"2024-09-21T13:49:04.082961+0000","last_unstale":"2024-09-21T13:53:09.590605+0000","last_undegraded":"2024-09-21T13:53:09.590605+0000","last_fullsized":"2024-09-21T13:53:09.590605+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:24.549706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:51:24.549706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"68'9","reported_seq":238,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.596422+0000","last_change":"2024-09-21T13:52:10.447789+0000","last_active":"2024-09-21T13:53:09.596422+0000","last_peered":"2024-09-21T13:53:09.596422+0000","last_clean":"2024-09-21T13:53:09.596422+0000","last_became_active":"2024-09-21T13:52:04.320149+0000","last_became_peered":"2024-09-21T13:52:04.320149+0000","last_unstale":"2024-09-21T13:53:09.596422+0000","last_undegraded":"2024-09-21T13:53:09.596422+0000","last_fullsized":"2024-09-21T13:53:09.596422+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:52:10.447530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:52:10.447530+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:37.291172+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":207,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.600474+0000","last_change":"2024-09-21T13:48:58.058032+0000","last_active":"2024-09-21T13:53:09.600474+0000","last_peered":"2024-09-21T13:53:09.600474+0000","last_clean":"2024-09-21T13:53:09.600474+0000","last_became_active":"2024-09-21T13:48:58.057516+0000","last_became_peered":"2024-09-21T13:48:58.057516+0000","last_unstale":"2024-09-21T13:53:09.600474+0000","last_undegraded":"2024-09-21T13:53:09.600474+0000","last_fullsized":"2024-09-21T13:53:09.600474+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:06.571008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":207,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.600993+0000","last_change":"2024-09-21T13:50:39.411785+0000","last_active":"2024-09-21T13:53:09.600993+0000","last_peered":"2024-09-21T13:53:09.600993+0000","last_clean":"2024-09-21T13:53:09.600993+0000","last_became_active":"2024-09-21T13:49:02.071922+0000","last_became_peered":"2024-09-21T13:49:02.071922+0000","last_unstale":"2024-09-21T13:53:09.600993+0000","last_undegraded":"2024-09-21T13:53:09.600993+0000","last_fullsized":"2024-09-21T13:53:09.600993+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:39.411718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:39.411718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":140,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.586628+0000","last_change":"2024-09-21T13:50:29.689537+0000","last_active":"2024-09-21T13:53:09.586628+0000","last_peered":"2024-09-21T13:53:09.586628+0000","last_clean":"2024-09-21T13:53:09.586628+0000","last_became_active":"2024-09-21T13:50:29.689045+0000","last_became_peered":"2024-09-21T13:50:29.689045+0000","last_unstale":"2024-09-21T13:53:09.586628+0000","last_undegraded":"2024-09-21T13:53:09.586628+0000","last_fullsized":"2024-09-21T13:53:09.586628+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"68'11","reported_seq":230,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.590806+0000","last_change":"2024-09-21T13:50:31.329973+0000","last_active":"2024-09-21T13:53:09.590806+0000","last_peered":"2024-09-21T13:53:09.590806+0000","last_clean":"2024-09-21T13:53:09.590806+0000","last_became_active":"2024-09-21T13:49:00.426091+0000","last_became_peered":"2024-09-21T13:49:00.426091+0000","last_unstale":"2024-09-21T13:53:09.590806+0000","last_undegraded":"2024-09-21T13:53:09.590806+0000","last_fullsized":"2024-09-21T13:53:09.590806+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:50:31.329740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:31.329740+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":207,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.587857+0000","last_change":"2024-09-21T13:48:58.052281+0000","last_active":"2024-09-21T13:53:09.587857+0000","last_peered":"2024-09-21T13:53:09.587857+0000","last_clean":"2024-09-21T13:53:09.587857+0000","last_became_active":"2024-09-21T13:48:58.051974+0000","last_became_peered":"2024-09-21T13:48:58.051974+0000","last_unstale":"2024-09-21T13:53:09.587857+0000","last_undegraded":"2024-09-21T13:53:09.587857+0000","last_fullsized":"2024-09-21T13:53:09.587857+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:07.361374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":207,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.597911+0000","last_change":"2024-09-21T13:50:37.118637+0000","last_active":"2024-09-21T13:53:09.597911+0000","last_peered":"2024-09-21T13:53:09.597911+0000","last_clean":"2024-09-21T13:53:09.597911+0000","last_became_active":"2024-09-21T13:49:02.056006+0000","last_became_peered":"2024-09-21T13:49:02.056006+0000","last_unstale":"2024-09-21T13:53:09.597911+0000","last_undegraded":"2024-09-21T13:53:09.597911+0000","last_fullsized":"2024-09-21T13:53:09.597911+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:37.118594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:37.118594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":203,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.601142+0000","last_change":"2024-09-21T13:51:55.827109+0000","last_active":"2024-09-21T13:53:09.601142+0000","last_peered":"2024-09-21T13:53:09.601142+0000","last_clean":"2024-09-21T13:53:09.601142+0000","last_became_active":"2024-09-21T13:49:04.434034+0000","last_became_peered":"2024-09-21T13:49:04.434034+0000","last_unstale":"2024-09-21T13:53:09.601142+0000","last_undegraded":"2024-09-21T13:53:09.601142+0000","last_fullsized":"2024-09-21T13:53:09.601142+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:55.826922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:51:55.826922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"68'4","reported_seq":209,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.595007+0000","last_change":"2024-09-21T13:49:00.063374+0000","last_active":"2024-09-21T13:53:09.595007+0000","last_peered":"2024-09-21T13:53:09.595007+0000","last_clean":"2024-09-21T13:53:09.595007+0000","last_became_active":"2024-09-21T13:49:00.063066+0000","last_became_peered":"2024-09-21T13:49:00.063066+0000","last_unstale":"2024-09-21T13:53:09.595007+0000","last_undegraded":"2024-09-21T13:53:09.595007+0000","last_fullsized":"2024-09-21T13:53:09.595007+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":140,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.588136+0000","last_change":"2024-09-21T13:50:29.690572+0000","last_active":"2024-09-21T13:53:09.588136+0000","last_peered":"2024-09-21T13:53:09.588136+0000","last_clean":"2024-09-21T13:53:09.588136+0000","last_became_active":"2024-09-21T13:50:29.690269+0000","last_became_peered":"2024-09-21T13:50:29.690269+0000","last_unstale":"2024-09-21T13:53:09.588136+0000","last_undegraded":"2024-09-21T13:53:09.588136+0000","last_fullsized":"2024-09-21T13:53:09.588136+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:04.145682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:04.145682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:31.629620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":215,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.591460+0000","last_change":"2024-09-21T13:52:48.472414+0000","last_active":"2024-09-21T13:53:09.591460+0000","last_peered":"2024-09-21T13:53:09.591460+0000","last_clean":"2024-09-21T13:53:09.591460+0000","last_became_active":"2024-09-21T13:49:02.064789+0000","last_became_peered":"2024-09-21T13:49:02.064789+0000","last_unstale":"2024-09-21T13:53:09.591460+0000","last_undegraded":"2024-09-21T13:53:09.591460+0000","last_fullsized":"2024-09-21T13:53:09.591460+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:48.472373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:52:48.472373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:51.057252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":214,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.599442+0000","last_change":"2024-09-21T13:52:04.322055+0000","last_active":"2024-09-21T13:53:09.599442+0000","last_peered":"2024-09-21T13:53:09.599442+0000","last_clean":"2024-09-21T13:53:09.599442+0000","last_became_active":"2024-09-21T13:52:04.321667+0000","last_became_peered":"2024-09-21T13:52:04.321667+0000","last_unstale":"2024-09-21T13:53:09.599442+0000","last_undegraded":"2024-09-21T13:53:09.599442+0000","last_fullsized":"2024-09-21T13:53:09.599442+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:39.151320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:39.151320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":211,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.601975+0000","last_change":"2024-09-21T13:52:27.409588+0000","last_active":"2024-09-21T13:53:09.601975+0000","last_peered":"2024-09-21T13:53:09.601975+0000","last_clean":"2024-09-21T13:53:09.601975+0000","last_became_active":"2024-09-21T13:49:04.075562+0000","last_became_peered":"2024-09-21T13:49:04.075562+0000","last_unstale":"2024-09-21T13:53:09.601975+0000","last_undegraded":"2024-09-21T13:53:09.601975+0000","last_fullsized":"2024-09-21T13:53:09.601975+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:27.409408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:27.409408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:52.946524+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":207,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.595166+0000","last_change":"2024-09-21T13:48:58.055791+0000","last_active":"2024-09-21T13:53:09.595166+0000","last_peered":"2024-09-21T13:53:09.595166+0000","last_clean":"2024-09-21T13:53:09.595166+0000","last_became_active":"2024-09-21T13:48:58.055497+0000","last_became_peered":"2024-09-21T13:48:58.055497+0000","last_unstale":"2024-09-21T13:53:09.595166+0000","last_undegraded":"2024-09-21T13:53:09.595166+0000","last_fullsized":"2024-09-21T13:53:09.595166+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"68'11","reported_seq":261,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.592042+0000","last_change":"2024-09-21T13:52:04.321178+0000","last_active":"2024-09-21T13:53:09.592042+0000","last_peered":"2024-09-21T13:53:09.592042+0000","last_clean":"2024-09-21T13:53:09.592042+0000","last_became_active":"2024-09-21T13:52:04.320647+0000","last_became_peered":"2024-09-21T13:52:04.320647+0000","last_unstale":"2024-09-21T13:53:09.592042+0000","last_undegraded":"2024-09-21T13:53:09.592042+0000","last_fullsized":"2024-09-21T13:53:09.592042+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:50:35.790090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:35.790090+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:07.227949+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":159,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-21T13:53:09.600651+0000","last_change":"2024-09-21T13:52:04.323345+0000","last_active":"2024-09-21T13:53:09.600651+0000","last_peered":"2024-09-21T13:53:09.600651+0000","last_clean":"2024-09-21T13:53:09.600651+0000","last_became_active":"2024-09-21T13:52:04.322880+0000","last_became_peered":"2024-09-21T13:52:04.322880+0000","last_unstale":"2024-09-21T13:53:09.600651+0000","last_undegraded":"2024-09-21T13:53:09.600651+0000","last_fullsized":"2024-09-21T13:53:09.600651+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:36.086691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:36.086691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":2944008216,"num_objects":44923,"num_object_clones":0,"num_object_copies":134769,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3242,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44923,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":227913,"num_write_kb":8730689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21326,"num_bytes_recovered":1397620736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8885907456,"data_stored":8885895240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85547,"ondisk_log_size":85547,"up":48,"acting":45,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69,"ondisk_log_size":69,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":694,"num_read_kb":451,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":395,"ondisk_log_size":395,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":250,"num_read_kb":245,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234040,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1050944,"kb_used_data":917040,"kb_used_omap":1,"kb_used_meta":133886,"kb_avail":92694208,"statfs":{"total":95995035648,"available":94918868992,"internally_reserved":0,"allocated":939048960,"data_stored":938555656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":137099722},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462973,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1019504,"kb_used_data":907808,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":92725648,"statfs":{"total":95995035648,"available":94951063552,"internally_reserved":0,"allocated":929595392,"data_stored":929117411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":114358726},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724612,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":829740,"kb_used_data":718044,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":92915412,"statfs":{"total":95995035648,"available":95145381888,"internally_reserved":0,"allocated":735277056,"data_stored":734803519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920842,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1026376,"kb_used_data":909112,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":92718776,"statfs":{"total":95995035648,"available":94944026624,"internally_reserved":0,"allocated":930930688,"data_stored":930428181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":120060357},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117071,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1813132,"kb_used_data":1610940,"kb_used_omap":1,"kb_used_meta":201918,"kb_avail":91932020,"statfs":{"total":95995035648,"available":94138388480,"internally_reserved":0,"allocated":1649602560,"data_stored":1649096787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":206764487},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":130,"seq":558345748495,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1319920,"kb_used_data":1148832,"kb_used_omap":4,"kb_used_meta":171003,"kb_avail":92425232,"statfs":{"total":95995035648,"available":94643437568,"internally_reserved":0,"allocated":1176403968,"data_stored":1175926357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4441,"internal_metadata":175107751},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574940,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1226724,"kb_used_data":1096980,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":92518428,"statfs":{"total":95995035648,"available":94738870272,"internally_reserved":0,"allocated":1123307520,"data_stored":1122776889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738466,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1586372,"kb_used_data":1428788,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":92158780,"statfs":{"total":95995035648,"available":94370590720,"internally_reserved":0,"allocated":1463078912,"data_stored":1462580809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":161282501},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1460080640,"data_stored":1460076568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1106644992,"data_stored":1106640920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1169358848,"data_stored":1169358848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1645871104,"data_stored":1645871104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":925302784,"data_stored":925302784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":733806592,"data_stored":733806592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":924454912,"data_stored":924450840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":920387584,"data_stored":920387584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T13:53:17.340 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph status --format=json 2024-09-21T13:53:17.344 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:53:17.405 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:17.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:17.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:53:17.507 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:53:17.532 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:17.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:17.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:53:17.634 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:53:17.642 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-09-21T13:53:17.661 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:17.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:17.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:53:17.763 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:53:17.797 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:17.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:17.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:53:17.899 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:17.929 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:17.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:17.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:17.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:17 smithi187 ceph-mon[33032]: pgmap v202: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 145 active+clean; 2.7 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s; 3242/135366 objects degraded (2.395%); 12 MiB/s, 196 objects/s recovering 2024-09-21T13:53:17.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:17 smithi187 ceph-mon[33032]: from='client.15069 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T13:53:17.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:17 smithi187 ceph-mon[33032]: osdmap e160: 8 total, 8 up, 8 in 2024-09-21T13:53:17.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:17 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3190765245' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:53:17.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:17 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:53:18.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:17 smithi049 ceph-mon[40489]: pgmap v202: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 145 active+clean; 2.7 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s; 3242/135366 objects degraded (2.395%); 12 MiB/s, 196 objects/s recovering 2024-09-21T13:53:18.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:17 smithi049 ceph-mon[40489]: from='client.15069 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T13:53:18.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:17 smithi049 ceph-mon[40489]: osdmap e160: 8 total, 8 up, 8 in 2024-09-21T13:53:18.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:17 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3190765245' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:53:18.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:17 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:53:18.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:17 smithi049 ceph-mon[30080]: pgmap v202: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 145 active+clean; 2.7 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s; 3242/135366 objects degraded (2.395%); 12 MiB/s, 196 objects/s recovering 2024-09-21T13:53:18.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:17 smithi049 ceph-mon[30080]: from='client.15069 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T13:53:18.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:17 smithi049 ceph-mon[30080]: osdmap e160: 8 total, 8 up, 8 in 2024-09-21T13:53:18.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:17 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3190765245' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:53:18.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:17 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:53:18.032 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:53:18.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:18.069 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:18.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:53:18.170 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:18.196 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:18.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:18.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:18.297 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:53:18.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:18.338 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:18.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:53:18.439 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:18.460 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:18.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:18.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:18.562 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:53:18.588 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:18.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:18.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:53:18.689 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:53:18.716 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:18.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:18.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:53:18.818 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:53:18.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:18.863 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:18.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:53:18.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:18 smithi187 ceph-mon[33032]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:53:18.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:18 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T13:53:18.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:18 smithi187 ceph-mon[33032]: osdmap e161: 8 total, 8 up, 8 in 2024-09-21T13:53:18.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:18 smithi187 ceph-mon[33032]: 4.6 scrub starts 2024-09-21T13:53:18.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:18 smithi187 ceph-mon[33032]: 4.6 scrub ok 2024-09-21T13:53:18.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:18 smithi187 ceph-mon[33032]: 6.f scrub starts 2024-09-21T13:53:18.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:18 smithi187 ceph-mon[33032]: 6.f scrub ok 2024-09-21T13:53:18.964 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:53:19.004 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:19.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:18 smithi049 ceph-mon[40489]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:53:19.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:18 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T13:53:19.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:18 smithi049 ceph-mon[40489]: osdmap e161: 8 total, 8 up, 8 in 2024-09-21T13:53:19.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:18 smithi049 ceph-mon[40489]: 4.6 scrub starts 2024-09-21T13:53:19.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:18 smithi049 ceph-mon[40489]: 4.6 scrub ok 2024-09-21T13:53:19.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:18 smithi049 ceph-mon[40489]: 6.f scrub starts 2024-09-21T13:53:19.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:18 smithi049 ceph-mon[40489]: 6.f scrub ok 2024-09-21T13:53:19.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:18 smithi049 ceph-mon[30080]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:53:19.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:18 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T13:53:19.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:18 smithi049 ceph-mon[30080]: osdmap e161: 8 total, 8 up, 8 in 2024-09-21T13:53:19.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:18 smithi049 ceph-mon[30080]: 4.6 scrub starts 2024-09-21T13:53:19.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:18 smithi049 ceph-mon[30080]: 4.6 scrub ok 2024-09-21T13:53:19.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:18 smithi049 ceph-mon[30080]: 6.f scrub starts 2024-09-21T13:53:19.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:18 smithi049 ceph-mon[30080]: 6.f scrub ok 2024-09-21T13:53:19.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:19.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:53:19.107 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:53:19.143 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:19.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:19.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:53:19.245 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:53:19.285 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:19.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:19.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:53:19.387 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:53:19.414 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:19.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:19.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:53:19.444 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:53:19 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:53:19.516 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:53:19.546 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:19.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:19.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:53:19.647 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:53:19.672 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:19.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:19.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:53:19.773 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:53:19.800 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:19.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:19.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:53:19.901 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:53:19.943 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:19.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:19.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:53:19.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:19 smithi187 ceph-mon[33032]: pgmap v205: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 peering, 1 active+undersized+degraded+remapped+backfilling, 145 active+clean; 2.7 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 808 op/s; 2965/134412 objects degraded (2.206%); 11 MiB/s, 180 objects/s recovering 2024-09-21T13:53:19.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:19 smithi187 ceph-mon[33032]: 6.15 scrub starts 2024-09-21T13:53:19.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:19 smithi187 ceph-mon[33032]: 6.15 scrub ok 2024-09-21T13:53:19.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:19 smithi187 ceph-mon[33032]: 4.b scrub starts 2024-09-21T13:53:19.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:19 smithi187 ceph-mon[33032]: 4.b scrub ok 2024-09-21T13:53:19.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:19 smithi187 ceph-mon[33032]: 5.6 scrub starts 2024-09-21T13:53:19.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:19 smithi187 ceph-mon[33032]: 5.6 scrub ok 2024-09-21T13:53:20.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:19 smithi049 ceph-mon[40489]: pgmap v205: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 peering, 1 active+undersized+degraded+remapped+backfilling, 145 active+clean; 2.7 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 808 op/s; 2965/134412 objects degraded (2.206%); 11 MiB/s, 180 objects/s recovering 2024-09-21T13:53:20.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:19 smithi049 ceph-mon[40489]: 6.15 scrub starts 2024-09-21T13:53:20.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:19 smithi049 ceph-mon[40489]: 6.15 scrub ok 2024-09-21T13:53:20.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:19 smithi049 ceph-mon[40489]: 4.b scrub starts 2024-09-21T13:53:20.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:19 smithi049 ceph-mon[40489]: 4.b scrub ok 2024-09-21T13:53:20.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:19 smithi049 ceph-mon[40489]: 5.6 scrub starts 2024-09-21T13:53:20.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:19 smithi049 ceph-mon[40489]: 5.6 scrub ok 2024-09-21T13:53:20.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:19 smithi049 ceph-mon[30080]: pgmap v205: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 peering, 1 active+undersized+degraded+remapped+backfilling, 145 active+clean; 2.7 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 808 op/s; 2965/134412 objects degraded (2.206%); 11 MiB/s, 180 objects/s recovering 2024-09-21T13:53:20.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:19 smithi049 ceph-mon[30080]: 6.15 scrub starts 2024-09-21T13:53:20.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:19 smithi049 ceph-mon[30080]: 6.15 scrub ok 2024-09-21T13:53:20.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:19 smithi049 ceph-mon[30080]: 4.b scrub starts 2024-09-21T13:53:20.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:19 smithi049 ceph-mon[30080]: 4.b scrub ok 2024-09-21T13:53:20.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:19 smithi049 ceph-mon[30080]: 5.6 scrub starts 2024-09-21T13:53:20.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:19 smithi049 ceph-mon[30080]: 5.6 scrub ok 2024-09-21T13:53:20.045 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:53:20.086 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:20.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:20.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:53:20.188 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:53:20.227 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:20.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:20.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:53:20.328 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:20.356 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:20.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:20.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:20.457 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:20.491 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:20.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:20.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:20.576 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:53:20.592 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:53:20.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:20.639 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:20.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:53:20.740 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:53:20.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:20.785 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:20.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:53:20.885 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:20.912 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:20.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:20.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:20.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:20 smithi187 ceph-mon[33032]: 5.e scrub starts 2024-09-21T13:53:20.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:20 smithi187 ceph-mon[33032]: 5.e scrub ok 2024-09-21T13:53:20.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:20 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:53:20.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:20 smithi187 ceph-mon[33032]: 3.f scrub starts 2024-09-21T13:53:20.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:20 smithi187 ceph-mon[33032]: 3.f scrub ok 2024-09-21T13:53:20.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:20 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 2965/134412 objects degraded (2.206%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T13:53:20.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:20 smithi187 ceph-mon[33032]: osdmap e162: 8 total, 8 up, 8 in 2024-09-21T13:53:20.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:20 smithi187 ceph-mon[33032]: 2.1 scrub starts 2024-09-21T13:53:20.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:20 smithi187 ceph-mon[33032]: 2.1 scrub ok 2024-09-21T13:53:21.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:20 smithi049 ceph-mon[40489]: 5.e scrub starts 2024-09-21T13:53:21.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:20 smithi049 ceph-mon[40489]: 5.e scrub ok 2024-09-21T13:53:21.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:20 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:53:21.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:20 smithi049 ceph-mon[40489]: 3.f scrub starts 2024-09-21T13:53:21.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:20 smithi049 ceph-mon[40489]: 3.f scrub ok 2024-09-21T13:53:21.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:20 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 2965/134412 objects degraded (2.206%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T13:53:21.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:20 smithi049 ceph-mon[40489]: osdmap e162: 8 total, 8 up, 8 in 2024-09-21T13:53:21.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:20 smithi049 ceph-mon[40489]: 2.1 scrub starts 2024-09-21T13:53:21.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:20 smithi049 ceph-mon[40489]: 2.1 scrub ok 2024-09-21T13:53:21.006 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:53:20 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:53:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:53:21.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:20 smithi049 ceph-mon[30080]: 5.e scrub starts 2024-09-21T13:53:21.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:20 smithi049 ceph-mon[30080]: 5.e scrub ok 2024-09-21T13:53:21.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:20 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:53:21.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:20 smithi049 ceph-mon[30080]: 3.f scrub starts 2024-09-21T13:53:21.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:20 smithi049 ceph-mon[30080]: 3.f scrub ok 2024-09-21T13:53:21.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:20 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 2965/134412 objects degraded (2.206%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T13:53:21.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:20 smithi049 ceph-mon[30080]: osdmap e162: 8 total, 8 up, 8 in 2024-09-21T13:53:21.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:20 smithi049 ceph-mon[30080]: 2.1 scrub starts 2024-09-21T13:53:21.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:20 smithi049 ceph-mon[30080]: 2.1 scrub ok 2024-09-21T13:53:21.014 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:21.039 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:21.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:21.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:21.141 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:53:21.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:21.181 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:21.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:53:21.282 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:53:21.332 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:21.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:21.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:53:21.435 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:53:21.462 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:21.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:21.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:53:21.563 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:53:21.615 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:21.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:21.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:53:21.654 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:53:21.703 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T13:53:21.703 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-09-21T13:53:21.704 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:21.704 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.14817.0:234625 7.6 e162/94)", 2024-09-21T13:53:21.704 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:53:21.701244+0000", 2024-09-21T13:53:21.704 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.00036522900000000001, 2024-09-21T13:53:21.704 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000385473, 2024-09-21T13:53:21.704 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:53:21.704 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:53:21.704 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "queued for pg", 2024-09-21T13:53:21.704 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:53:21.704 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:21.704 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:53:21.704 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:21.701244+0000", 2024-09-21T13:53:21.704 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:53:21.704 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:21.704 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:21.705 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:53:21.705 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:21.701244+0000", 2024-09-21T13:53:21.705 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:53:21.705 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:21.705 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:21.705 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:53:21.705 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:21.701245+0000", 2024-09-21T13:53:21.705 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0100000000000001e-06 2024-09-21T13:53:21.705 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:21.705 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:21.705 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:53:21.705 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:21.701256+0000", 2024-09-21T13:53:21.705 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1314e-05 2024-09-21T13:53:21.705 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:21.705 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:21.706 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:53:21.706 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:21.701273+0000", 2024-09-21T13:53:21.706 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6651e-05 2024-09-21T13:53:21.706 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:21.706 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:21.706 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:53:21.706 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:21.701281+0000", 2024-09-21T13:53:21.706 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2269999999999993e-06 2024-09-21T13:53:21.706 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:53:21.706 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:53:21.706 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:53:21.706 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:53:21.706 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T13:53:21.706 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 1 2024-09-21T13:53:21.707 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T13:53:21.717 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:53:21.760 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:21.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:21.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:53:21.862 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:53:21.889 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:21.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:21.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:53:21.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:21 smithi187 ceph-mon[33032]: pgmap v206: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 peering, 146 active+clean; 2.6 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 707 op/s; 2600/125994 objects degraded (2.064%); 9.1 MiB/s, 144 objects/s recovering 2024-09-21T13:53:21.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:21 smithi187 ceph-mon[33032]: 3.1e scrub starts 2024-09-21T13:53:21.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:21 smithi187 ceph-mon[33032]: 3.1e scrub ok 2024-09-21T13:53:21.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:21 smithi187 ceph-mon[33032]: 7.5 scrub starts 2024-09-21T13:53:21.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:21 smithi187 ceph-mon[33032]: 6.b scrub starts 2024-09-21T13:53:21.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:21 smithi187 ceph-mon[33032]: 6.b scrub ok 2024-09-21T13:53:21.992 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:53:22.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:21 smithi049 ceph-mon[40489]: pgmap v206: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 peering, 146 active+clean; 2.6 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 707 op/s; 2600/125994 objects degraded (2.064%); 9.1 MiB/s, 144 objects/s recovering 2024-09-21T13:53:22.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:21 smithi049 ceph-mon[40489]: 3.1e scrub starts 2024-09-21T13:53:22.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:21 smithi049 ceph-mon[40489]: 3.1e scrub ok 2024-09-21T13:53:22.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:21 smithi049 ceph-mon[40489]: 7.5 scrub starts 2024-09-21T13:53:22.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:21 smithi049 ceph-mon[40489]: 6.b scrub starts 2024-09-21T13:53:22.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:21 smithi049 ceph-mon[40489]: 6.b scrub ok 2024-09-21T13:53:22.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:21 smithi049 ceph-mon[30080]: pgmap v206: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 peering, 146 active+clean; 2.6 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 707 op/s; 2600/125994 objects degraded (2.064%); 9.1 MiB/s, 144 objects/s recovering 2024-09-21T13:53:22.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:21 smithi049 ceph-mon[30080]: 3.1e scrub starts 2024-09-21T13:53:22.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:21 smithi049 ceph-mon[30080]: 3.1e scrub ok 2024-09-21T13:53:22.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:21 smithi049 ceph-mon[30080]: 7.5 scrub starts 2024-09-21T13:53:22.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:21 smithi049 ceph-mon[30080]: 6.b scrub starts 2024-09-21T13:53:22.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:21 smithi049 ceph-mon[30080]: 6.b scrub ok 2024-09-21T13:53:22.017 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:22.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:22.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:53:22.119 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:53:22.159 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:22.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:22.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:53:22.262 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:53:22.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:22.300 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:22.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:53:22.401 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:53:22.446 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:22.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:22.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:53:22.518 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T13:53:22.548 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:22.573 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:22.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:22.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:22.623 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T13:53:22.674 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:22.701 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:22.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:22.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:22.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:22 smithi049 ceph-mon[40489]: 6.1a scrub starts 2024-09-21T13:53:22.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:22 smithi049 ceph-mon[40489]: 6.1a scrub ok 2024-09-21T13:53:22.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:22 smithi049 ceph-mon[40489]: 7.5 scrub ok 2024-09-21T13:53:22.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:22 smithi049 ceph-mon[30080]: 6.1a scrub starts 2024-09-21T13:53:22.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:22 smithi049 ceph-mon[30080]: 6.1a scrub ok 2024-09-21T13:53:22.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:22 smithi049 ceph-mon[30080]: 7.5 scrub ok 2024-09-21T13:53:22.803 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:53:22.825 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:22.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:22.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:53:22.927 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:53:22.976 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:22.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:22.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:53:23.064 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T13:53:23.078 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:23.104 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:23.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:23.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:23.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:22 smithi187 ceph-mon[33032]: 6.1a scrub starts 2024-09-21T13:53:23.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:22 smithi187 ceph-mon[33032]: 6.1a scrub ok 2024-09-21T13:53:23.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:22 smithi187 ceph-mon[33032]: 7.5 scrub ok 2024-09-21T13:53:23.206 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:53:23.229 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:23.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:23.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:53:23.330 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:53:23.354 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:23.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:23.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:53:23.456 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:53:23.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:23.508 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:23.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:53:23.609 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:53:23.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:23.649 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:23.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:53:23.750 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:53:23.801 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:23.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:23.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:53:23.903 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:53:23.946 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:53:23.954 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:53:23.966 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:53:23.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:23.967 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:23.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:53:24.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:23 smithi049 ceph-mon[30080]: pgmap v208: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 peering, 146 active+clean; 2.6 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 521 op/s; 2600/125994 objects degraded (2.064%); 4.5 MiB/s, 72 objects/s recovering 2024-09-21T13:53:24.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:23 smithi049 ceph-mon[40489]: pgmap v208: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 peering, 146 active+clean; 2.6 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 521 op/s; 2600/125994 objects degraded (2.064%); 4.5 MiB/s, 72 objects/s recovering 2024-09-21T13:53:24.068 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:53:24.093 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:24.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:24.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:53:24.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:23 smithi187 ceph-mon[33032]: pgmap v208: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 peering, 146 active+clean; 2.6 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 521 op/s; 2600/125994 objects degraded (2.064%); 4.5 MiB/s, 72 objects/s recovering 2024-09-21T13:53:24.196 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:24.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:24.225 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:24.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:24.280 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:53:24.280 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"40a3d2e8-781f-11ef-baf6-efdc52797490","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"nodeep-scrub flag(s) set","count":12},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2600/125994 objects degraded (2.064%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":520,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":163,"num_osds":8,"num_up_osds":8,"osd_up_since":1726926723,"num_in_osds":8,"osd_in_since":1726926503,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":146},{"state_name":"peering","count":1},{"state_name":"active+undersized+degraded+remapped+backfill_toofull","count":1}],"num_pgs":148,"num_pools":7,"num_objects":41998,"data_bytes":2739870259,"bytes_used":9527390208,"bytes_avail":758432894976,"bytes_total":767960285184,"inactive_pgs_ratio":0.0067567569203674793,"degraded_objects":2600,"degraded_total":125994,"degraded_ratio":0.020635903296982398,"recovering_objects_per_sec":72,"recovering_bytes_per_sec":4760727,"recovering_keys_per_sec":0,"num_objects_recovered":436,"num_bytes_recovered":28573696,"num_keys_recovered":0,"read_bytes_sec":853,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":520},"fsmap":{"epoch":1,"btime":"2024-09-21T13:42:57:838094+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.49:8443/","prometheus":"http://172.21.15.49:9283/"}},"servicemap":{"epoch":15,"modified":"2024-09-21T13:53:03.743631+0000","services":{"rgw":{"daemons":{"summary":"","24365":{"start_epoch":12,"start_stamp":"2024-09-21T13:49:04.899719+0000","gid":24365,"addr":"172.21.15.49:0/3404764347","metadata":{"arch":"x86_64","ceph_release":"squid","ceph_version":"ceph version 19.3.0-5104-g44178494 (441784946648ff53dd310293f9dd752afeeea72a) squid (dev)","ceph_version_short":"19.3.0-5104-g44178494","container_hostname":"smithi049","container_image":"quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5","cpu":"Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz","distro":"centos","distro_description":"CentOS Stream 9","distro_version":"9","frontend_config#0":"beast port=80","frontend_type#0":"beast","hostname":"smithi049","id":"foo.a","kernel_description":"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024","kernel_version":"5.14.0-508.el9.x86_64","mem_swap_kb":"0","mem_total_kb":"32487164","num_handles":"1","os":"Linux","pid":"7","realm_id":"","realm_name":"","zone_id":"d447add1-8b31-4043-a929-4cb437e5f078","zone_name":"default","zonegroup_id":"06d4f733-b51c-4f26-9844-acd95f223f5b","zonegroup_name":"default"},"task_status":{}}}}}},"progress_events":{}} 2024-09-21T13:53:24.325 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:53:24.351 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:24.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:24.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:53:24.453 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:53:24.497 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:24.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:24.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:53:24.599 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:24.623 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:24.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:24.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:24.725 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:53:24.768 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:24.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:24.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:53:24.870 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:53:24.911 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:24.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:24.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:53:25.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:24 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:53:25.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:24 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:53:25.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:24 smithi049 ceph-mon[40489]: osdmap e163: 8 total, 8 up, 8 in 2024-09-21T13:53:25.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:24 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1396365987' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T13:53:25.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:24 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:53:25.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:24 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:53:25.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:24 smithi049 ceph-mon[30080]: osdmap e163: 8 total, 8 up, 8 in 2024-09-21T13:53:25.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:24 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1396365987' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T13:53:25.013 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:53:25.056 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:25.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:25.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:53:25.158 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:25.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:25.186 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:25.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:25.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:24 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:53:25.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:24 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:53:25.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:24 smithi187 ceph-mon[33032]: osdmap e163: 8 total, 8 up, 8 in 2024-09-21T13:53:25.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:24 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1396365987' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T13:53:25.287 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:53:25.327 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:25.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:25.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:53:25.429 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:53:25.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:25.471 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:25.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:53:25.572 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:53:25.606 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:25.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:25.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:53:25.707 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:53:25.757 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:25.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:25.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:53:25.776 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:25 smithi049 ceph-mon[40489]: pgmap v210: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 147 active+clean; 2.3 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.13k op/s; 2600/113742 objects degraded (2.286%); 1.7 MiB/s, 26 objects/s recovering 2024-09-21T13:53:25.776 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:25 smithi049 ceph-mon[40489]: 3.0 scrub starts 2024-09-21T13:53:25.776 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:25 smithi049 ceph-mon[40489]: 3.0 scrub ok 2024-09-21T13:53:25.776 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:25 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 2600/113742 objects degraded (2.286%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T13:53:25.776 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:25 smithi049 ceph-mon[40489]: osdmap e164: 8 total, 8 up, 8 in 2024-09-21T13:53:25.777 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:25 smithi049 ceph-mon[30080]: pgmap v210: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 147 active+clean; 2.3 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.13k op/s; 2600/113742 objects degraded (2.286%); 1.7 MiB/s, 26 objects/s recovering 2024-09-21T13:53:25.777 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:25 smithi049 ceph-mon[30080]: 3.0 scrub starts 2024-09-21T13:53:25.777 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:25 smithi049 ceph-mon[30080]: 3.0 scrub ok 2024-09-21T13:53:25.777 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:25 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 2600/113742 objects degraded (2.286%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T13:53:25.777 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:25 smithi049 ceph-mon[30080]: osdmap e164: 8 total, 8 up, 8 in 2024-09-21T13:53:25.859 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:25.884 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:25.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:25.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:25.947 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T13:53:25.947 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-09-21T13:53:25.947 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-09-21T13:53:25.947 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-09-21T13:53:25.947 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T13:53:25.986 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:26.010 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:26.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:26.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:26.112 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:53:26.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:26.140 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:26.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:53:26.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:25 smithi187 ceph-mon[33032]: pgmap v210: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 147 active+clean; 2.3 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.13k op/s; 2600/113742 objects degraded (2.286%); 1.7 MiB/s, 26 objects/s recovering 2024-09-21T13:53:26.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:25 smithi187 ceph-mon[33032]: 3.0 scrub starts 2024-09-21T13:53:26.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:25 smithi187 ceph-mon[33032]: 3.0 scrub ok 2024-09-21T13:53:26.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:25 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 2600/113742 objects degraded (2.286%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T13:53:26.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:25 smithi187 ceph-mon[33032]: osdmap e164: 8 total, 8 up, 8 in 2024-09-21T13:53:26.241 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:53:26.266 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:26.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:26.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:53:26.367 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:53:26.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:26.411 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:26.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:53:26.512 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:53:26.543 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:26.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:26.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:53:26.645 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:53:26.695 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:26.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:26.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:53:26.697 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-21T13:53:26.697 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T13:53:26.749 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-09-21T13:53:26.769 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:26 smithi049 ceph-mon[40489]: 3.18 scrub starts 2024-09-21T13:53:26.769 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:26 smithi049 ceph-mon[40489]: 3.18 scrub ok 2024-09-21T13:53:26.770 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:26 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3374419724' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:53:26.770 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:26 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:53:26.770 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:26 smithi049 ceph-mon[30080]: 3.18 scrub starts 2024-09-21T13:53:26.770 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:26 smithi049 ceph-mon[30080]: 3.18 scrub ok 2024-09-21T13:53:26.770 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:26 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3374419724' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:53:26.770 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:26 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:53:26.796 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:26.821 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:26.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:26.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:26.922 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:53:26.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:26 smithi187 ceph-mon[33032]: 3.18 scrub starts 2024-09-21T13:53:26.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:26 smithi187 ceph-mon[33032]: 3.18 scrub ok 2024-09-21T13:53:26.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:26 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3374419724' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:53:26.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:26 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:53:26.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:26.979 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:26.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:53:27.080 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:53:27.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:27.111 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:27.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:53:27.212 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:53:27.237 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:27.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:27.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:53:27.339 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:53:27.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:27.368 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:27.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:53:27.443 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T13:53:27.469 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:53:27.495 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:27.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:27.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:53:27.596 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:53:27.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:27.647 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:27.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:53:27.748 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:53:27.772 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:27.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:27.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:53:27.874 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:53:27.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:27.900 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:27.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:53:27.926 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:53:27.926 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:53:27.966 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:53:27.967 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:53:27.983 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:27 smithi049 ceph-mon[40489]: pgmap v212: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 147 active+clean; 2.2 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 904 op/s; 2260/109719 objects degraded (2.060%) 2024-09-21T13:53:27.983 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:27 smithi049 ceph-mon[40489]: 4.13 scrub starts 2024-09-21T13:53:27.983 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:27 smithi049 ceph-mon[40489]: 4.13 scrub ok 2024-09-21T13:53:27.983 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:27 smithi049 ceph-mon[40489]: 7.6 scrub starts 2024-09-21T13:53:27.983 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:27 smithi049 ceph-mon[40489]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T13:53:27.983 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:27 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T13:53:27.983 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:27 smithi049 ceph-mon[40489]: osdmap e165: 8 total, 8 up, 8 in 2024-09-21T13:53:27.983 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:27 smithi049 ceph-mon[30080]: pgmap v212: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 147 active+clean; 2.2 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 904 op/s; 2260/109719 objects degraded (2.060%) 2024-09-21T13:53:27.983 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:27 smithi049 ceph-mon[30080]: 4.13 scrub starts 2024-09-21T13:53:27.983 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:27 smithi049 ceph-mon[30080]: 4.13 scrub ok 2024-09-21T13:53:27.984 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:27 smithi049 ceph-mon[30080]: 7.6 scrub starts 2024-09-21T13:53:27.984 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:27 smithi049 ceph-mon[30080]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T13:53:27.984 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:27 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T13:53:27.984 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:27 smithi049 ceph-mon[30080]: osdmap e165: 8 total, 8 up, 8 in 2024-09-21T13:53:28.000 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:53:28.000 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:53:28.001 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:53:28.045 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:53:28.045 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:53:28.048 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:28.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:28.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:53:28.085 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:53:28.085 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:53:28.124 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:53:28.124 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:53:28.152 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:53:28.159 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:53:28.159 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:53:28.179 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:28.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:28.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:53:28.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:27 smithi187 ceph-mon[33032]: pgmap v212: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 147 active+clean; 2.2 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 904 op/s; 2260/109719 objects degraded (2.060%) 2024-09-21T13:53:28.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:27 smithi187 ceph-mon[33032]: 4.13 scrub starts 2024-09-21T13:53:28.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:27 smithi187 ceph-mon[33032]: 4.13 scrub ok 2024-09-21T13:53:28.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:27 smithi187 ceph-mon[33032]: 7.6 scrub starts 2024-09-21T13:53:28.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:27 smithi187 ceph-mon[33032]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T13:53:28.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:27 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T13:53:28.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:27 smithi187 ceph-mon[33032]: osdmap e165: 8 total, 8 up, 8 in 2024-09-21T13:53:28.196 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:53:28.196 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:53:28.281 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:53:28.309 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:28.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:28.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:53:28.411 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:28.442 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:28.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:28.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:28.544 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:53:28.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:28.592 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:28.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:53:28.693 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:53:28.738 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:28.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:28.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:53:28.839 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:28.867 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:28.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:28.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:28.969 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:53:29.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:28 smithi049 ceph-mon[40489]: 7.6 scrub ok 2024-09-21T13:53:29.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:28 smithi049 ceph-mon[40489]: 4.10 scrub starts 2024-09-21T13:53:29.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:28 smithi049 ceph-mon[40489]: 4.10 scrub ok 2024-09-21T13:53:29.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:28 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2667465639' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:53:29.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:28 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:53:29.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:28 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:53:29.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:28 smithi049 ceph-mon[30080]: 7.6 scrub ok 2024-09-21T13:53:29.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:28 smithi049 ceph-mon[30080]: 4.10 scrub starts 2024-09-21T13:53:29.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:28 smithi049 ceph-mon[30080]: 4.10 scrub ok 2024-09-21T13:53:29.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:28 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2667465639' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:53:29.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:28 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:53:29.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:28 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:53:29.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:29.013 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:29.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:53:29.114 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:29.142 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:29.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:29.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:29.145 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:53:29.167 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:28 smithi187 ceph-mon[33032]: 7.6 scrub ok 2024-09-21T13:53:29.167 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:28 smithi187 ceph-mon[33032]: 4.10 scrub starts 2024-09-21T13:53:29.167 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:28 smithi187 ceph-mon[33032]: 4.10 scrub ok 2024-09-21T13:53:29.168 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:28 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2667465639' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:53:29.168 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:28 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:53:29.168 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:28 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:53:29.244 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:29.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:29.274 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:29.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:29.375 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:53:29.400 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:29.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:29.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:53:29.444 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:53:29 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:53:29.502 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:53:29.543 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:29.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:29.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:53:29.645 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:53:29.672 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:29.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:29.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:53:29.774 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:53:29.799 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:29.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:29.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:53:29.900 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:53:29.926 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:29.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:29.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:53:30.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:29 smithi049 ceph-mon[40489]: pgmap v214: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 147 active+clean; 2.2 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.00k op/s; 2260/107907 objects degraded (2.094%) 2024-09-21T13:53:30.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:29 smithi049 ceph-mon[40489]: 7.e scrub starts 2024-09-21T13:53:30.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:29 smithi049 ceph-mon[30080]: pgmap v214: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 147 active+clean; 2.2 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.00k op/s; 2260/107907 objects degraded (2.094%) 2024-09-21T13:53:30.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:29 smithi049 ceph-mon[30080]: 7.e scrub starts 2024-09-21T13:53:30.028 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:53:30.050 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:30.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:30.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:53:30.132 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T13:53:30.133 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:53:30.152 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:53:30.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:29 smithi187 ceph-mon[33032]: pgmap v214: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 147 active+clean; 2.2 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.00k op/s; 2260/107907 objects degraded (2.094%) 2024-09-21T13:53:30.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:29 smithi187 ceph-mon[33032]: 7.e scrub starts 2024-09-21T13:53:30.196 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:30.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:30.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:53:30.298 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:53:30.337 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:30.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:30.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:53:30.352 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:53:30.374 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T13:53:30.438 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:53:30.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:30.501 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:30.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:53:30.602 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:53:30.627 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:30.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:30.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:53:30.729 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:30.755 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:53:30 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:53:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:53:30.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:30.756 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:30.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:30.843 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T13:53:30.857 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:53:30.884 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:30.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:30.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:53:30.986 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:53:31.043 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:31.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:31.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:53:31.118 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T13:53:31.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:30 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:53:31.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:30 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:53:31.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:30 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:53:31.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:30 smithi049 ceph-mon[30080]: 7.e scrub ok 2024-09-21T13:53:31.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:30 smithi049 ceph-mon[30080]: pgmap v215: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 147 active+clean; 1.9 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 846 B/s rd, 0 B/s wr, 1.08k op/s; 1940/94107 objects degraded (2.061%) 2024-09-21T13:53:31.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:30 smithi049 ceph-mon[30080]: osdmap e166: 8 total, 8 up, 8 in 2024-09-21T13:53:31.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:30 smithi049 ceph-mon[30080]: 5.12 scrub starts 2024-09-21T13:53:31.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:30 smithi049 ceph-mon[30080]: 5.12 scrub ok 2024-09-21T13:53:31.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:30 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 1940/94107 objects degraded (2.061%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T13:53:31.126 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:30 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:53:31.126 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:30 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:53:31.126 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:30 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:53:31.126 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:30 smithi049 ceph-mon[40489]: 7.e scrub ok 2024-09-21T13:53:31.126 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:30 smithi049 ceph-mon[40489]: pgmap v215: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 147 active+clean; 1.9 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 846 B/s rd, 0 B/s wr, 1.08k op/s; 1940/94107 objects degraded (2.061%) 2024-09-21T13:53:31.126 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:30 smithi049 ceph-mon[40489]: osdmap e166: 8 total, 8 up, 8 in 2024-09-21T13:53:31.126 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:30 smithi049 ceph-mon[40489]: 5.12 scrub starts 2024-09-21T13:53:31.127 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:30 smithi049 ceph-mon[40489]: 5.12 scrub ok 2024-09-21T13:53:31.127 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:30 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 1940/94107 objects degraded (2.061%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T13:53:31.145 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:53:31.169 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:31.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:31.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:53:31.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:30 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:53:31.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:30 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:53:31.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:30 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:53:31.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:30 smithi187 ceph-mon[33032]: 7.e scrub ok 2024-09-21T13:53:31.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:30 smithi187 ceph-mon[33032]: pgmap v215: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 147 active+clean; 1.9 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 846 B/s rd, 0 B/s wr, 1.08k op/s; 1940/94107 objects degraded (2.061%) 2024-09-21T13:53:31.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:30 smithi187 ceph-mon[33032]: osdmap e166: 8 total, 8 up, 8 in 2024-09-21T13:53:31.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:30 smithi187 ceph-mon[33032]: 5.12 scrub starts 2024-09-21T13:53:31.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:30 smithi187 ceph-mon[33032]: 5.12 scrub ok 2024-09-21T13:53:31.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:30 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 1940/94107 objects degraded (2.061%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T13:53:31.271 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:53:31.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:31.316 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:31.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:53:31.417 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:53:31.444 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:31.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:31.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:53:31.521 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:53:31.545 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:53:31.573 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:53:31.602 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:31.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:31.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:53:31.704 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:53:31.728 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:31.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:31.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:53:31.830 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:53:31.895 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:53:31.897 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:31.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:31.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:53:31.900 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-09-21T13:53:32.000 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:53:32.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:31 smithi049 ceph-mon[30080]: 3.5 scrub starts 2024-09-21T13:53:32.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:31 smithi049 ceph-mon[30080]: 3.5 scrub ok 2024-09-21T13:53:32.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:31 smithi049 ceph-mon[30080]: 7.8 scrub starts 2024-09-21T13:53:32.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:31 smithi049 ceph-mon[40489]: 3.5 scrub starts 2024-09-21T13:53:32.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:31 smithi049 ceph-mon[40489]: 3.5 scrub ok 2024-09-21T13:53:32.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:31 smithi049 ceph-mon[40489]: 7.8 scrub starts 2024-09-21T13:53:32.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:32.028 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:32.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:53:32.129 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:53:32.176 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:32.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:32.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:53:32.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:31 smithi187 ceph-mon[33032]: 3.5 scrub starts 2024-09-21T13:53:32.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:31 smithi187 ceph-mon[33032]: 3.5 scrub ok 2024-09-21T13:53:32.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:31 smithi187 ceph-mon[33032]: 7.8 scrub starts 2024-09-21T13:53:32.278 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:53:32.318 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:32.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:32.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:53:32.420 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:53:32.446 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:32.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:32.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:53:32.548 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:53:32.593 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:32.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:32.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:53:32.694 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:53:32.883 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:32.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:32.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:53:32.985 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:33.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:33.011 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:33.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:33.092 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:32 smithi049 ceph-mon[40489]: 4.8 scrub starts 2024-09-21T13:53:33.092 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:32 smithi049 ceph-mon[40489]: 4.8 scrub ok 2024-09-21T13:53:33.092 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:32 smithi049 ceph-mon[40489]: 7.8 scrub ok 2024-09-21T13:53:33.092 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:32 smithi049 ceph-mon[40489]: pgmap v217: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 147 active+clean; 1.9 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 747 B/s rd, 0 B/s wr, 760 op/s; 1940/94107 objects degraded (2.061%) 2024-09-21T13:53:33.092 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:32 smithi049 ceph-mon[40489]: 4.17 scrub starts 2024-09-21T13:53:33.092 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:32 smithi049 ceph-mon[40489]: 4.17 scrub ok 2024-09-21T13:53:33.092 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:32 smithi049 ceph-mon[40489]: from='client.24989 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T13:53:33.093 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:32 smithi049 ceph-mon[40489]: 5.8 scrub starts 2024-09-21T13:53:33.093 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:32 smithi049 ceph-mon[40489]: 5.8 scrub ok 2024-09-21T13:53:33.093 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:32 smithi049 ceph-mon[30080]: 4.8 scrub starts 2024-09-21T13:53:33.093 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:32 smithi049 ceph-mon[30080]: 4.8 scrub ok 2024-09-21T13:53:33.093 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:32 smithi049 ceph-mon[30080]: 7.8 scrub ok 2024-09-21T13:53:33.093 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:32 smithi049 ceph-mon[30080]: pgmap v217: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 147 active+clean; 1.9 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 747 B/s rd, 0 B/s wr, 760 op/s; 1940/94107 objects degraded (2.061%) 2024-09-21T13:53:33.093 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:32 smithi049 ceph-mon[30080]: 4.17 scrub starts 2024-09-21T13:53:33.093 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:32 smithi049 ceph-mon[30080]: 4.17 scrub ok 2024-09-21T13:53:33.093 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:32 smithi049 ceph-mon[30080]: from='client.24989 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T13:53:33.093 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:32 smithi049 ceph-mon[30080]: 5.8 scrub starts 2024-09-21T13:53:33.093 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:32 smithi049 ceph-mon[30080]: 5.8 scrub ok 2024-09-21T13:53:33.113 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:53:33.139 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:33.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:33.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:53:33.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:32 smithi187 ceph-mon[33032]: 4.8 scrub starts 2024-09-21T13:53:33.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:32 smithi187 ceph-mon[33032]: 4.8 scrub ok 2024-09-21T13:53:33.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:32 smithi187 ceph-mon[33032]: 7.8 scrub ok 2024-09-21T13:53:33.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:32 smithi187 ceph-mon[33032]: pgmap v217: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 147 active+clean; 1.9 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 747 B/s rd, 0 B/s wr, 760 op/s; 1940/94107 objects degraded (2.061%) 2024-09-21T13:53:33.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:32 smithi187 ceph-mon[33032]: 4.17 scrub starts 2024-09-21T13:53:33.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:32 smithi187 ceph-mon[33032]: 4.17 scrub ok 2024-09-21T13:53:33.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:32 smithi187 ceph-mon[33032]: from='client.24989 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T13:53:33.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:32 smithi187 ceph-mon[33032]: 5.8 scrub starts 2024-09-21T13:53:33.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:32 smithi187 ceph-mon[33032]: 5.8 scrub ok 2024-09-21T13:53:33.242 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:53:33.271 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:33.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:33.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:53:33.373 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:53:33.416 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":217,"stamp":"2024-09-21T13:53:31.757096+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2043288115,"num_objects":31369,"num_object_clones":0,"num_object_copies":94107,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1940,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31369,"num_whiteouts":0,"num_read":1110,"num_read_kb":967,"num_write":242215,"num_write_kb":8732362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21777,"num_bytes_recovered":1426785020,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87259,"ondisk_log_size":87259,"up":444,"acting":443,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":7295804,"kb_used_data":6162280,"kb_used_omap":15,"kb_used_meta":1133296,"kb_avail":742665412,"statfs":{"total":767960285184,"available":760489381888,"internally_reserved":0,"allocated":6310174720,"data_stored":6306896817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15598,"internal_metadata":1160495890},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,11,10,8,7],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-341049344,"num_objects":-5204,"num_object_clones":0,"num_object_copies":-15612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5204,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":5204,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.851959"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":173,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.744051+0000","last_change":"2024-09-21T13:52:26.677736+0000","last_active":"2024-09-21T13:53:26.744051+0000","last_peered":"2024-09-21T13:53:26.744051+0000","last_clean":"2024-09-21T13:53:26.744051+0000","last_became_active":"2024-09-21T13:52:04.317322+0000","last_became_peered":"2024-09-21T13:52:04.317322+0000","last_unstale":"2024-09-21T13:53:26.744051+0000","last_undegraded":"2024-09-21T13:53:26.744051+0000","last_fullsized":"2024-09-21T13:53:26.744051+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:26.677689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:52:26.677689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:45.636529+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"68'9","reported_seq":241,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.755027+0000","last_change":"2024-09-21T13:51:55.225649+0000","last_active":"2024-09-21T13:53:26.755027+0000","last_peered":"2024-09-21T13:53:26.755027+0000","last_clean":"2024-09-21T13:53:26.755027+0000","last_became_active":"2024-09-21T13:49:00.055210+0000","last_became_peered":"2024-09-21T13:49:00.055210+0000","last_unstale":"2024-09-21T13:53:26.755027+0000","last_undegraded":"2024-09-21T13:53:26.755027+0000","last_fullsized":"2024-09-21T13:53:26.755027+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:51:55.225579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:55.225579+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":221,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.746412+0000","last_change":"2024-09-21T13:50:28.058990+0000","last_active":"2024-09-21T13:53:26.746412+0000","last_peered":"2024-09-21T13:53:26.746412+0000","last_clean":"2024-09-21T13:53:26.746412+0000","last_became_active":"2024-09-21T13:49:02.060984+0000","last_became_peered":"2024-09-21T13:49:02.060984+0000","last_unstale":"2024-09-21T13:53:26.746412+0000","last_undegraded":"2024-09-21T13:53:26.746412+0000","last_fullsized":"2024-09-21T13:53:26.746412+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:28.058918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:28.058918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:39.434903+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":217,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.755141+0000","last_change":"2024-09-21T13:53:21.005444+0000","last_active":"2024-09-21T13:53:26.755141+0000","last_peered":"2024-09-21T13:53:26.755141+0000","last_clean":"2024-09-21T13:53:26.755141+0000","last_became_active":"2024-09-21T13:49:04.072460+0000","last_became_peered":"2024-09-21T13:49:04.072460+0000","last_unstale":"2024-09-21T13:53:26.755141+0000","last_undegraded":"2024-09-21T13:53:26.755141+0000","last_fullsized":"2024-09-21T13:53:26.755141+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:21.005352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:21.005352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:43.773187+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":217,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.748029+0000","last_change":"2024-09-21T13:51:18.572788+0000","last_active":"2024-09-21T13:53:26.748029+0000","last_peered":"2024-09-21T13:53:26.748029+0000","last_clean":"2024-09-21T13:53:26.748029+0000","last_became_active":"2024-09-21T13:49:04.433965+0000","last_became_peered":"2024-09-21T13:49:04.433965+0000","last_unstale":"2024-09-21T13:53:26.748029+0000","last_undegraded":"2024-09-21T13:53:26.748029+0000","last_fullsized":"2024-09-21T13:53:26.748029+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:18.572729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:51:18.572729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":241,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.752385+0000","last_change":"2024-09-21T13:53:19.910708+0000","last_active":"2024-09-21T13:53:26.752385+0000","last_peered":"2024-09-21T13:53:26.752385+0000","last_clean":"2024-09-21T13:53:26.752385+0000","last_became_active":"2024-09-21T13:52:04.316819+0000","last_became_peered":"2024-09-21T13:52:04.316819+0000","last_unstale":"2024-09-21T13:53:26.752385+0000","last_undegraded":"2024-09-21T13:53:26.752385+0000","last_fullsized":"2024-09-21T13:53:26.752385+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:19.910638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:53:19.910638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:27.797531+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"68'15","reported_seq":253,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.746658+0000","last_change":"2024-09-21T13:52:04.315645+0000","last_active":"2024-09-21T13:53:26.746658+0000","last_peered":"2024-09-21T13:53:26.746658+0000","last_clean":"2024-09-21T13:53:26.746658+0000","last_became_active":"2024-09-21T13:52:04.315101+0000","last_became_peered":"2024-09-21T13:52:04.315101+0000","last_unstale":"2024-09-21T13:53:26.746658+0000","last_undegraded":"2024-09-21T13:53:26.746658+0000","last_fullsized":"2024-09-21T13:53:26.746658+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:15.792034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":229,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.754878+0000","last_change":"2024-09-21T13:51:43.040397+0000","last_active":"2024-09-21T13:53:26.754878+0000","last_peered":"2024-09-21T13:53:26.754878+0000","last_clean":"2024-09-21T13:53:26.754878+0000","last_became_active":"2024-09-21T13:49:02.063687+0000","last_became_peered":"2024-09-21T13:49:02.063687+0000","last_unstale":"2024-09-21T13:53:26.754878+0000","last_undegraded":"2024-09-21T13:53:26.754878+0000","last_fullsized":"2024-09-21T13:53:26.754878+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:43.040348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:43.040348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":237,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.743119+0000","last_change":"2024-09-21T13:52:49.451929+0000","last_active":"2024-09-21T13:53:26.743119+0000","last_peered":"2024-09-21T13:53:26.743119+0000","last_clean":"2024-09-21T13:53:26.743119+0000","last_became_active":"2024-09-21T13:48:58.050695+0000","last_became_peered":"2024-09-21T13:48:58.050695+0000","last_unstale":"2024-09-21T13:53:26.743119+0000","last_undegraded":"2024-09-21T13:53:26.743119+0000","last_fullsized":"2024-09-21T13:53:26.743119+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:49.451882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:52:49.451882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:54.087443+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"68'9","reported_seq":241,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.755508+0000","last_change":"2024-09-21T13:51:21.951391+0000","last_active":"2024-09-21T13:53:26.755508+0000","last_peered":"2024-09-21T13:53:26.755508+0000","last_clean":"2024-09-21T13:53:26.755508+0000","last_became_active":"2024-09-21T13:49:00.059382+0000","last_became_peered":"2024-09-21T13:49:00.059382+0000","last_unstale":"2024-09-21T13:53:26.755508+0000","last_undegraded":"2024-09-21T13:53:26.755508+0000","last_fullsized":"2024-09-21T13:53:26.755508+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:51:21.951305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:21.951305+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":213,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.742678+0000","last_change":"2024-09-21T13:49:02.066978+0000","last_active":"2024-09-21T13:53:26.742678+0000","last_peered":"2024-09-21T13:53:26.742678+0000","last_clean":"2024-09-21T13:53:26.742678+0000","last_became_active":"2024-09-21T13:49:02.066693+0000","last_became_peered":"2024-09-21T13:49:02.066693+0000","last_unstale":"2024-09-21T13:53:26.742678+0000","last_undegraded":"2024-09-21T13:53:26.742678+0000","last_fullsized":"2024-09-21T13:53:26.742678+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:09.312032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":170,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.748662+0000","last_change":"2024-09-21T13:52:28.417865+0000","last_active":"2024-09-21T13:53:26.748662+0000","last_peered":"2024-09-21T13:53:26.748662+0000","last_clean":"2024-09-21T13:53:26.748662+0000","last_became_active":"2024-09-21T13:50:29.689381+0000","last_became_peered":"2024-09-21T13:50:29.689381+0000","last_unstale":"2024-09-21T13:53:26.748662+0000","last_undegraded":"2024-09-21T13:53:26.748662+0000","last_fullsized":"2024-09-21T13:53:26.748662+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:28.417690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:28.417690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:33.915909+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":229,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.742792+0000","last_change":"2024-09-21T13:50:39.480995+0000","last_active":"2024-09-21T13:53:26.742792+0000","last_peered":"2024-09-21T13:53:26.742792+0000","last_clean":"2024-09-21T13:53:26.742792+0000","last_became_active":"2024-09-21T13:48:58.050906+0000","last_became_peered":"2024-09-21T13:48:58.050906+0000","last_unstale":"2024-09-21T13:53:26.742792+0000","last_undegraded":"2024-09-21T13:53:26.742792+0000","last_fullsized":"2024-09-21T13:53:26.742792+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:39.480930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:39.480930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:32.611461+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"68'5","reported_seq":251,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.755360+0000","last_change":"2024-09-21T13:52:04.319297+0000","last_active":"2024-09-21T13:53:26.755360+0000","last_peered":"2024-09-21T13:53:26.755360+0000","last_clean":"2024-09-21T13:53:26.755360+0000","last_became_active":"2024-09-21T13:52:04.318515+0000","last_became_peered":"2024-09-21T13:52:04.318515+0000","last_unstale":"2024-09-21T13:53:26.755360+0000","last_undegraded":"2024-09-21T13:53:26.755360+0000","last_fullsized":"2024-09-21T13:53:26.755360+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:02.067591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":221,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.753555+0000","last_change":"2024-09-21T13:50:40.188797+0000","last_active":"2024-09-21T13:53:26.753555+0000","last_peered":"2024-09-21T13:53:26.753555+0000","last_clean":"2024-09-21T13:53:26.753555+0000","last_became_active":"2024-09-21T13:49:02.060212+0000","last_became_peered":"2024-09-21T13:49:02.060212+0000","last_unstale":"2024-09-21T13:53:26.753555+0000","last_undegraded":"2024-09-21T13:53:26.753555+0000","last_fullsized":"2024-09-21T13:53:26.753555+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:40.188712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:40.188712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":225,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.742423+0000","last_change":"2024-09-21T13:52:43.516806+0000","last_active":"2024-09-21T13:53:26.742423+0000","last_peered":"2024-09-21T13:53:26.742423+0000","last_clean":"2024-09-21T13:53:26.742423+0000","last_became_active":"2024-09-21T13:49:04.078026+0000","last_became_peered":"2024-09-21T13:49:04.078026+0000","last_unstale":"2024-09-21T13:53:26.742423+0000","last_undegraded":"2024-09-21T13:53:26.742423+0000","last_fullsized":"2024-09-21T13:53:26.742423+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:43.516741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:43.516741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:08.753512+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":218,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.757071+0000","last_change":"2024-09-21T13:50:38.027288+0000","last_active":"2024-09-21T13:53:26.757071+0000","last_peered":"2024-09-21T13:53:26.757071+0000","last_clean":"2024-09-21T13:53:26.757071+0000","last_became_active":"2024-09-21T13:49:04.431422+0000","last_became_peered":"2024-09-21T13:49:04.431422+0000","last_unstale":"2024-09-21T13:53:26.757071+0000","last_undegraded":"2024-09-21T13:53:26.757071+0000","last_fullsized":"2024-09-21T13:53:26.757071+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:38.027227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:38.027227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":154,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.755643+0000","last_change":"2024-09-21T13:50:29.692621+0000","last_active":"2024-09-21T13:53:26.755643+0000","last_peered":"2024-09-21T13:53:26.755643+0000","last_clean":"2024-09-21T13:53:26.755643+0000","last_became_active":"2024-09-21T13:50:29.688282+0000","last_became_peered":"2024-09-21T13:50:29.688282+0000","last_unstale":"2024-09-21T13:53:26.755643+0000","last_undegraded":"2024-09-21T13:53:26.755643+0000","last_fullsized":"2024-09-21T13:53:26.755643+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:04.102443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"68'15","reported_seq":182,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.763264+0000","last_change":"2024-09-21T13:52:04.324874+0000","last_active":"2024-09-21T13:53:26.763264+0000","last_peered":"2024-09-21T13:53:26.763264+0000","last_clean":"2024-09-21T13:53:26.763264+0000","last_became_active":"2024-09-21T13:52:04.324273+0000","last_became_peered":"2024-09-21T13:52:04.324273+0000","last_unstale":"2024-09-21T13:53:26.763264+0000","last_undegraded":"2024-09-21T13:53:26.763264+0000","last_fullsized":"2024-09-21T13:53:26.763264+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:50:24.749694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:24.749694+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":229,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.748359+0000","last_change":"2024-09-21T13:52:26.372217+0000","last_active":"2024-09-21T13:53:26.748359+0000","last_peered":"2024-09-21T13:53:26.748359+0000","last_clean":"2024-09-21T13:53:26.748359+0000","last_became_active":"2024-09-21T13:49:02.066796+0000","last_became_peered":"2024-09-21T13:49:02.066796+0000","last_unstale":"2024-09-21T13:53:26.748359+0000","last_undegraded":"2024-09-21T13:53:26.748359+0000","last_fullsized":"2024-09-21T13:53:26.748359+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:26.372144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:52:26.372144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:40.709431+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":210,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.750392+0000","last_change":"2024-09-21T13:49:04.434429+0000","last_active":"2024-09-21T13:53:26.750392+0000","last_peered":"2024-09-21T13:53:26.750392+0000","last_clean":"2024-09-21T13:53:26.750392+0000","last_became_active":"2024-09-21T13:49:04.434049+0000","last_became_peered":"2024-09-21T13:49:04.434049+0000","last_unstale":"2024-09-21T13:53:26.750392+0000","last_undegraded":"2024-09-21T13:53:26.750392+0000","last_fullsized":"2024-09-21T13:53:26.750392+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":240,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.755631+0000","last_change":"2024-09-21T13:52:52.142682+0000","last_active":"2024-09-21T13:53:26.755631+0000","last_peered":"2024-09-21T13:53:26.755631+0000","last_clean":"2024-09-21T13:53:26.755631+0000","last_became_active":"2024-09-21T13:52:04.316016+0000","last_became_peered":"2024-09-21T13:52:04.316016+0000","last_unstale":"2024-09-21T13:53:26.755631+0000","last_undegraded":"2024-09-21T13:53:26.755631+0000","last_fullsized":"2024-09-21T13:53:26.755631+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:52.142638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:52:52.142638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:13.595892+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"68'12","reported_seq":248,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.747700+0000","last_change":"2024-09-21T13:51:52.819966+0000","last_active":"2024-09-21T13:53:26.747700+0000","last_peered":"2024-09-21T13:53:26.747700+0000","last_clean":"2024-09-21T13:53:26.747700+0000","last_became_active":"2024-09-21T13:49:00.061867+0000","last_became_peered":"2024-09-21T13:49:00.061867+0000","last_unstale":"2024-09-21T13:53:26.747700+0000","last_undegraded":"2024-09-21T13:53:26.747700+0000","last_fullsized":"2024-09-21T13:53:26.747700+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'12","last_scrub_stamp":"2024-09-21T13:51:52.819879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:52.819879+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":224,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.758026+0000","last_change":"2024-09-21T13:52:04.321297+0000","last_active":"2024-09-21T13:53:26.758026+0000","last_peered":"2024-09-21T13:53:26.758026+0000","last_clean":"2024-09-21T13:53:26.758026+0000","last_became_active":"2024-09-21T13:52:04.320800+0000","last_became_peered":"2024-09-21T13:52:04.320800+0000","last_unstale":"2024-09-21T13:53:26.758026+0000","last_undegraded":"2024-09-21T13:53:26.758026+0000","last_fullsized":"2024-09-21T13:53:26.758026+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:14.081205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":229,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.754458+0000","last_change":"2024-09-21T13:52:54.580070+0000","last_active":"2024-09-21T13:53:26.754458+0000","last_peered":"2024-09-21T13:53:26.754458+0000","last_clean":"2024-09-21T13:53:26.754458+0000","last_became_active":"2024-09-21T13:52:04.321358+0000","last_became_peered":"2024-09-21T13:52:04.321358+0000","last_unstale":"2024-09-21T13:53:26.754458+0000","last_undegraded":"2024-09-21T13:53:26.754458+0000","last_fullsized":"2024-09-21T13:53:26.754458+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T13:52:54.580022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:54.580022+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:21.087079+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":238,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.746547+0000","last_change":"2024-09-21T13:52:43.516442+0000","last_active":"2024-09-21T13:53:26.746547+0000","last_peered":"2024-09-21T13:53:26.746547+0000","last_clean":"2024-09-21T13:53:26.746547+0000","last_became_active":"2024-09-21T13:48:58.048972+0000","last_became_peered":"2024-09-21T13:48:58.048972+0000","last_unstale":"2024-09-21T13:53:26.746547+0000","last_undegraded":"2024-09-21T13:53:26.746547+0000","last_fullsized":"2024-09-21T13:53:26.746547+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:52:43.516221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:52:43.516221+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:48.258159+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"68'10","reported_seq":197,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.757715+0000","last_change":"2024-09-21T13:52:56.526032+0000","last_active":"2024-09-21T13:53:26.757715+0000","last_peered":"2024-09-21T13:53:26.757715+0000","last_clean":"2024-09-21T13:53:26.757715+0000","last_became_active":"2024-09-21T13:50:29.694465+0000","last_became_peered":"2024-09-21T13:50:29.694465+0000","last_unstale":"2024-09-21T13:53:26.757715+0000","last_undegraded":"2024-09-21T13:53:26.757715+0000","last_fullsized":"2024-09-21T13:53:26.757715+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:52:56.525966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:52:56.525966+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:10.063932+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"68'8","reported_seq":296,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.759381+0000","last_change":"2024-09-21T13:51:42.174696+0000","last_active":"2024-09-21T13:53:26.759381+0000","last_peered":"2024-09-21T13:53:26.759381+0000","last_clean":"2024-09-21T13:53:26.759381+0000","last_became_active":"2024-09-21T13:49:02.059638+0000","last_became_peered":"2024-09-21T13:49:02.059638+0000","last_unstale":"2024-09-21T13:53:26.759381+0000","last_undegraded":"2024-09-21T13:53:26.759381+0000","last_fullsized":"2024-09-21T13:53:26.759381+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:51:42.174629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:42.174629+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:08.238915+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"165'15196","reported_seq":15496,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:28.376755+0000","last_change":"2024-09-21T13:52:51.407976+0000","last_active":"2024-09-21T13:53:28.376755+0000","last_peered":"2024-09-21T13:53:28.376755+0000","last_clean":"2024-09-21T13:53:28.376755+0000","last_became_active":"2024-09-21T13:50:07.422961+0000","last_became_peered":"2024-09-21T13:50:07.422961+0000","last_unstale":"2024-09-21T13:53:28.376755+0000","last_undegraded":"2024-09-21T13:53:28.376755+0000","last_fullsized":"2024-09-21T13:53:28.376755+0000","mapping_epoch":83,"log_start":"123'8900","ondisk_log_start":"123'8900","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'13038","last_scrub_stamp":"2024-09-21T13:52:51.407818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:52:51.407818+0000","objects_scrubbed":4170,"log_size":6296,"log_dups_size":0,"ondisk_log_size":6296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:10.866985+0000","scrub_duration":960,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130547712,"num_objects":1992,"num_object_clones":0,"num_object_copies":5976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15196,"num_write_kb":550016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"68'15","reported_seq":258,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.745580+0000","last_change":"2024-09-21T13:52:16.452049+0000","last_active":"2024-09-21T13:53:26.745580+0000","last_peered":"2024-09-21T13:53:26.745580+0000","last_clean":"2024-09-21T13:53:26.745580+0000","last_became_active":"2024-09-21T13:49:00.060324+0000","last_became_peered":"2024-09-21T13:49:00.060324+0000","last_unstale":"2024-09-21T13:53:26.745580+0000","last_undegraded":"2024-09-21T13:53:26.745580+0000","last_fullsized":"2024-09-21T13:53:26.745580+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:52:16.451964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:52:16.451964+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:31.981849+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":230,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.758864+0000","last_change":"2024-09-21T13:50:06.151943+0000","last_active":"2024-09-21T13:53:26.758864+0000","last_peered":"2024-09-21T13:53:26.758864+0000","last_clean":"2024-09-21T13:53:26.758864+0000","last_became_active":"2024-09-21T13:48:58.051911+0000","last_became_peered":"2024-09-21T13:48:58.051911+0000","last_unstale":"2024-09-21T13:53:26.758864+0000","last_undegraded":"2024-09-21T13:53:26.758864+0000","last_fullsized":"2024-09-21T13:53:26.758864+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.151888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.151888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"68'8","reported_seq":287,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.757291+0000","last_change":"2024-09-21T13:53:19.021881+0000","last_active":"2024-09-21T13:53:26.757291+0000","last_peered":"2024-09-21T13:53:26.757291+0000","last_clean":"2024-09-21T13:53:26.757291+0000","last_became_active":"2024-09-21T13:49:02.061679+0000","last_became_peered":"2024-09-21T13:49:02.061679+0000","last_unstale":"2024-09-21T13:53:26.757291+0000","last_undegraded":"2024-09-21T13:53:26.757291+0000","last_fullsized":"2024-09-21T13:53:26.757291+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:53:19.021810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:53:19.021810+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:35.961578+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":217,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.742584+0000","last_change":"2024-09-21T13:50:34.481503+0000","last_active":"2024-09-21T13:53:26.742584+0000","last_peered":"2024-09-21T13:53:26.742584+0000","last_clean":"2024-09-21T13:53:26.742584+0000","last_became_active":"2024-09-21T13:49:04.081905+0000","last_became_peered":"2024-09-21T13:49:04.081905+0000","last_unstale":"2024-09-21T13:53:26.742584+0000","last_undegraded":"2024-09-21T13:53:26.742584+0000","last_fullsized":"2024-09-21T13:53:26.742584+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:34.481450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:34.481450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:42.543466+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"165'15331","reported_seq":22929,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:28.927731+0000","last_change":"2024-09-21T13:53:16.667415+0000","last_active":"2024-09-21T13:53:28.927731+0000","last_peered":"2024-09-21T13:53:28.927731+0000","last_clean":"2024-09-21T13:53:28.927731+0000","last_became_active":"2024-09-21T13:53:16.646871+0000","last_became_peered":"2024-09-21T13:53:16.646871+0000","last_unstale":"2024-09-21T13:53:28.927731+0000","last_undegraded":"2024-09-21T13:53:28.927731+0000","last_fullsized":"2024-09-21T13:53:28.927731+0000","mapping_epoch":159,"log_start":"134'10700","ondisk_log_start":"134'10700","created":83,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":4631,"log_dups_size":0,"ondisk_log_size":4631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124977152,"num_objects":1907,"num_object_clones":0,"num_object_copies":5721,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15331,"num_write_kb":551616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3703,"num_bytes_recovered":242679808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"68'18","reported_seq":258,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.758661+0000","last_change":"2024-09-21T13:50:27.281455+0000","last_active":"2024-09-21T13:53:26.758661+0000","last_peered":"2024-09-21T13:53:26.758661+0000","last_clean":"2024-09-21T13:53:26.758661+0000","last_became_active":"2024-09-21T13:49:00.429314+0000","last_became_peered":"2024-09-21T13:49:00.429314+0000","last_unstale":"2024-09-21T13:53:26.758661+0000","last_undegraded":"2024-09-21T13:53:26.758661+0000","last_fullsized":"2024-09-21T13:53:26.758661+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'18","last_scrub_stamp":"2024-09-21T13:50:27.281356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:27.281356+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":222,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.752318+0000","last_change":"2024-09-21T13:48:58.056669+0000","last_active":"2024-09-21T13:53:26.752318+0000","last_peered":"2024-09-21T13:53:26.752318+0000","last_clean":"2024-09-21T13:53:26.752318+0000","last_became_active":"2024-09-21T13:48:58.056375+0000","last_became_peered":"2024-09-21T13:48:58.056375+0000","last_unstale":"2024-09-21T13:53:26.752318+0000","last_undegraded":"2024-09-21T13:53:26.752318+0000","last_fullsized":"2024-09-21T13:53:26.752318+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":222,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.757621+0000","last_change":"2024-09-21T13:51:23.940809+0000","last_active":"2024-09-21T13:53:26.757621+0000","last_peered":"2024-09-21T13:53:26.757621+0000","last_clean":"2024-09-21T13:53:26.757621+0000","last_became_active":"2024-09-21T13:49:02.062955+0000","last_became_peered":"2024-09-21T13:49:02.062955+0000","last_unstale":"2024-09-21T13:53:26.757621+0000","last_undegraded":"2024-09-21T13:53:26.757621+0000","last_fullsized":"2024-09-21T13:53:26.757621+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:23.940746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:23.940746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":221,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.755429+0000","last_change":"2024-09-21T13:52:04.316586+0000","last_active":"2024-09-21T13:53:26.755429+0000","last_peered":"2024-09-21T13:53:26.755429+0000","last_clean":"2024-09-21T13:53:26.755429+0000","last_became_active":"2024-09-21T13:52:04.316121+0000","last_became_peered":"2024-09-21T13:52:04.316121+0000","last_unstale":"2024-09-21T13:53:26.755429+0000","last_undegraded":"2024-09-21T13:53:26.755429+0000","last_fullsized":"2024-09-21T13:53:26.755429+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:10.130707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"165'14900","reported_seq":15098,"reported_epoch":165,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2024-09-21T13:53:28.925155+0000","last_change":"2024-09-21T13:53:05.053228+0000","last_active":"2024-09-21T13:53:28.925155+0000","last_peered":"2024-09-21T13:53:28.925155+0000","last_clean":"2024-09-21T13:50:42.909755+0000","last_became_active":"2024-09-21T13:52:04.998636+0000","last_became_peered":"2024-09-21T13:52:04.998636+0000","last_unstale":"2024-09-21T13:53:28.925155+0000","last_undegraded":"2024-09-21T13:52:04.888197+0000","last_fullsized":"2024-09-21T13:52:04.863335+0000","mapping_epoch":131,"log_start":"132'10200","ondisk_log_start":"132'10200","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":4700,"log_dups_size":0,"ondisk_log_size":4700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127139840,"num_objects":1940,"num_object_clones":0,"num_object_copies":5820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1940,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14900,"num_write_kb":538880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":1940}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"68'14","reported_seq":247,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.757696+0000","last_change":"2024-09-21T13:50:29.009786+0000","last_active":"2024-09-21T13:53:26.757696+0000","last_peered":"2024-09-21T13:53:26.757696+0000","last_clean":"2024-09-21T13:53:26.757696+0000","last_became_active":"2024-09-21T13:49:00.055031+0000","last_became_peered":"2024-09-21T13:49:00.055031+0000","last_unstale":"2024-09-21T13:53:26.757696+0000","last_undegraded":"2024-09-21T13:53:26.757696+0000","last_fullsized":"2024-09-21T13:53:26.757696+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'14","last_scrub_stamp":"2024-09-21T13:50:29.009720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:29.009720+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":231,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.773958+0000","last_change":"2024-09-21T13:49:59.253636+0000","last_active":"2024-09-21T13:53:26.773958+0000","last_peered":"2024-09-21T13:53:26.773958+0000","last_clean":"2024-09-21T13:53:26.773958+0000","last_became_active":"2024-09-21T13:48:58.053533+0000","last_became_peered":"2024-09-21T13:48:58.053533+0000","last_unstale":"2024-09-21T13:53:26.773958+0000","last_undegraded":"2024-09-21T13:53:26.773958+0000","last_fullsized":"2024-09-21T13:53:26.773958+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:49:59.253587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:49:59.253587+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:34.697714+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"68'8","reported_seq":285,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.758317+0000","last_change":"2024-09-21T13:50:41.458637+0000","last_active":"2024-09-21T13:53:26.758317+0000","last_peered":"2024-09-21T13:53:26.758317+0000","last_clean":"2024-09-21T13:53:26.758317+0000","last_became_active":"2024-09-21T13:49:02.070064+0000","last_became_peered":"2024-09-21T13:49:02.070064+0000","last_unstale":"2024-09-21T13:53:26.758317+0000","last_undegraded":"2024-09-21T13:53:26.758317+0000","last_fullsized":"2024-09-21T13:53:26.758317+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:50:41.458562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:41.458562+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":229,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.754287+0000","last_change":"2024-09-21T13:52:55.541288+0000","last_active":"2024-09-21T13:53:26.754287+0000","last_peered":"2024-09-21T13:53:26.754287+0000","last_clean":"2024-09-21T13:53:26.754287+0000","last_became_active":"2024-09-21T13:52:04.322185+0000","last_became_peered":"2024-09-21T13:52:04.322185+0000","last_unstale":"2024-09-21T13:53:26.754287+0000","last_undegraded":"2024-09-21T13:53:26.754287+0000","last_fullsized":"2024-09-21T13:53:26.754287+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:55.541232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:55.541232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:57.661213+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"165'14927","reported_seq":15103,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:28.411597+0000","last_change":"2024-09-21T13:50:07.419221+0000","last_active":"2024-09-21T13:53:28.411597+0000","last_peered":"2024-09-21T13:53:28.411597+0000","last_clean":"2024-09-21T13:53:28.411597+0000","last_became_active":"2024-09-21T13:50:07.418929+0000","last_became_peered":"2024-09-21T13:50:07.418929+0000","last_unstale":"2024-09-21T13:53:28.411597+0000","last_undegraded":"2024-09-21T13:53:28.411597+0000","last_fullsized":"2024-09-21T13:53:28.411597+0000","mapping_epoch":83,"log_start":"129'10100","ondisk_log_start":"129'10100","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":4827,"log_dups_size":0,"ondisk_log_size":4827,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128253952,"num_objects":1957,"num_object_clones":0,"num_object_copies":5871,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14927,"num_write_kb":540288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"68'10","reported_seq":249,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.746209+0000","last_change":"2024-09-21T13:51:50.211365+0000","last_active":"2024-09-21T13:53:26.746209+0000","last_peered":"2024-09-21T13:53:26.746209+0000","last_clean":"2024-09-21T13:53:26.746209+0000","last_became_active":"2024-09-21T13:49:00.054564+0000","last_became_peered":"2024-09-21T13:49:00.054564+0000","last_unstale":"2024-09-21T13:53:26.746209+0000","last_undegraded":"2024-09-21T13:53:26.746209+0000","last_fullsized":"2024-09-21T13:53:26.746209+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:51:50.211279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:50.211279+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:15.800803+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"61'1","reported_seq":255,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.743197+0000","last_change":"2024-09-21T13:52:16.585403+0000","last_active":"2024-09-21T13:53:26.743197+0000","last_peered":"2024-09-21T13:53:26.743197+0000","last_clean":"2024-09-21T13:53:26.743197+0000","last_became_active":"2024-09-21T13:52:04.315788+0000","last_became_peered":"2024-09-21T13:52:04.315788+0000","last_unstale":"2024-09-21T13:53:26.743197+0000","last_undegraded":"2024-09-21T13:53:26.743197+0000","last_fullsized":"2024-09-21T13:53:26.743197+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:52:16.585323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:52:16.585323+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:28.935922+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"96'9","reported_seq":232,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.756440+0000","last_change":"2024-09-21T13:52:51.443040+0000","last_active":"2024-09-21T13:53:26.756440+0000","last_peered":"2024-09-21T13:53:26.756440+0000","last_clean":"2024-09-21T13:53:26.756440+0000","last_became_active":"2024-09-21T13:50:29.693557+0000","last_became_peered":"2024-09-21T13:50:29.693557+0000","last_unstale":"2024-09-21T13:53:26.756440+0000","last_undegraded":"2024-09-21T13:53:26.756440+0000","last_fullsized":"2024-09-21T13:53:26.756440+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'9","last_scrub_stamp":"2024-09-21T13:52:51.442930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:52:51.442930+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:04.534122+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":165,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.754250+0000","last_change":"2024-09-21T13:52:04.319323+0000","last_active":"2024-09-21T13:53:26.754250+0000","last_peered":"2024-09-21T13:53:26.754250+0000","last_clean":"2024-09-21T13:53:26.754250+0000","last_became_active":"2024-09-21T13:52:04.319034+0000","last_became_peered":"2024-09-21T13:52:04.319034+0000","last_unstale":"2024-09-21T13:53:26.754250+0000","last_undegraded":"2024-09-21T13:53:26.754250+0000","last_fullsized":"2024-09-21T13:53:26.754250+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"165'15156","reported_seq":15403,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:28.370160+0000","last_change":"2024-09-21T13:52:16.618637+0000","last_active":"2024-09-21T13:53:28.370160+0000","last_peered":"2024-09-21T13:53:28.370160+0000","last_clean":"2024-09-21T13:53:28.370160+0000","last_became_active":"2024-09-21T13:50:07.423702+0000","last_became_peered":"2024-09-21T13:50:07.423702+0000","last_unstale":"2024-09-21T13:53:28.370160+0000","last_undegraded":"2024-09-21T13:53:28.370160+0000","last_fullsized":"2024-09-21T13:53:28.370160+0000","mapping_epoch":83,"log_start":"124'9300","ondisk_log_start":"124'9300","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'10945","last_scrub_stamp":"2024-09-21T13:52:16.618486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:52:16.618486+0000","objects_scrubbed":6221,"log_size":5856,"log_dups_size":0,"ondisk_log_size":5856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:43.545327+0000","scrub_duration":1967,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127926272,"num_objects":1952,"num_object_clones":0,"num_object_copies":5856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15156,"num_write_kb":547456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"68'19","reported_seq":218,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.742746+0000","last_change":"2024-09-21T13:50:32.514267+0000","last_active":"2024-09-21T13:53:26.742746+0000","last_peered":"2024-09-21T13:53:26.742746+0000","last_clean":"2024-09-21T13:53:26.742746+0000","last_became_active":"2024-09-21T13:50:29.690185+0000","last_became_peered":"2024-09-21T13:50:29.690185+0000","last_unstale":"2024-09-21T13:53:26.742746+0000","last_undegraded":"2024-09-21T13:53:26.742746+0000","last_fullsized":"2024-09-21T13:53:26.742746+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-21T13:50:32.514210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:32.514210+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:34.599292+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":233,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.745524+0000","last_change":"2024-09-21T13:52:04.322289+0000","last_active":"2024-09-21T13:53:26.745524+0000","last_peered":"2024-09-21T13:53:26.745524+0000","last_clean":"2024-09-21T13:53:26.745524+0000","last_became_active":"2024-09-21T13:52:04.321656+0000","last_became_peered":"2024-09-21T13:52:04.321656+0000","last_unstale":"2024-09-21T13:53:26.745524+0000","last_undegraded":"2024-09-21T13:53:26.745524+0000","last_fullsized":"2024-09-21T13:53:26.745524+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:08.762218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":222,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.755179+0000","last_change":"2024-09-21T13:51:12.939488+0000","last_active":"2024-09-21T13:53:26.755179+0000","last_peered":"2024-09-21T13:53:26.755179+0000","last_clean":"2024-09-21T13:53:26.755179+0000","last_became_active":"2024-09-21T13:49:02.061187+0000","last_became_peered":"2024-09-21T13:49:02.061187+0000","last_unstale":"2024-09-21T13:53:26.755179+0000","last_undegraded":"2024-09-21T13:53:26.755179+0000","last_fullsized":"2024-09-21T13:53:26.755179+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:12.939444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:12.939444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:31.497225+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":221,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.746375+0000","last_change":"2024-09-21T13:52:04.321133+0000","last_active":"2024-09-21T13:53:26.746375+0000","last_peered":"2024-09-21T13:53:26.746375+0000","last_clean":"2024-09-21T13:53:26.746375+0000","last_became_active":"2024-09-21T13:52:04.320826+0000","last_became_peered":"2024-09-21T13:52:04.320826+0000","last_unstale":"2024-09-21T13:53:26.746375+0000","last_undegraded":"2024-09-21T13:53:26.746375+0000","last_fullsized":"2024-09-21T13:53:26.746375+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:09.756023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"165'15198","reported_seq":15426,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:28.320290+0000","last_change":"2024-09-21T13:52:49.375801+0000","last_active":"2024-09-21T13:53:28.320290+0000","last_peered":"2024-09-21T13:53:28.320290+0000","last_clean":"2024-09-21T13:53:28.320290+0000","last_became_active":"2024-09-21T13:50:07.421666+0000","last_became_peered":"2024-09-21T13:50:07.421666+0000","last_unstale":"2024-09-21T13:53:28.320290+0000","last_undegraded":"2024-09-21T13:53:28.320290+0000","last_fullsized":"2024-09-21T13:53:28.320290+0000","mapping_epoch":83,"log_start":"123'8900","ondisk_log_start":"123'8900","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"149'12949","last_scrub_stamp":"2024-09-21T13:52:49.375671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:52:49.375671+0000","objects_scrubbed":4241,"log_size":6298,"log_dups_size":0,"ondisk_log_size":6298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:54.416521+0000","scrub_duration":1855,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128581632,"num_objects":1962,"num_object_clones":0,"num_object_copies":5886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15198,"num_write_kb":549120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"132'30","reported_seq":334,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.745418+0000","last_change":"2024-09-21T13:52:04.322738+0000","last_active":"2024-09-21T13:53:26.745418+0000","last_peered":"2024-09-21T13:53:26.745418+0000","last_clean":"2024-09-21T13:53:26.745418+0000","last_became_active":"2024-09-21T13:52:04.319431+0000","last_became_peered":"2024-09-21T13:52:04.319431+0000","last_unstale":"2024-09-21T13:53:26.745418+0000","last_undegraded":"2024-09-21T13:53:26.745418+0000","last_fullsized":"2024-09-21T13:53:26.745418+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:12.782459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":222,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.755726+0000","last_change":"2024-09-21T13:48:58.055640+0000","last_active":"2024-09-21T13:53:26.755726+0000","last_peered":"2024-09-21T13:53:26.755726+0000","last_clean":"2024-09-21T13:53:26.755726+0000","last_became_active":"2024-09-21T13:48:58.055366+0000","last_became_peered":"2024-09-21T13:48:58.055366+0000","last_unstale":"2024-09-21T13:53:26.755726+0000","last_undegraded":"2024-09-21T13:53:26.755726+0000","last_fullsized":"2024-09-21T13:53:26.755726+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:09.147679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":248,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.742656+0000","last_change":"2024-09-21T13:52:51.445882+0000","last_active":"2024-09-21T13:53:26.742656+0000","last_peered":"2024-09-21T13:53:26.742656+0000","last_clean":"2024-09-21T13:53:26.742656+0000","last_became_active":"2024-09-21T13:48:57.046246+0000","last_became_peered":"2024-09-21T13:48:57.046246+0000","last_unstale":"2024-09-21T13:53:26.742656+0000","last_undegraded":"2024-09-21T13:53:26.742656+0000","last_fullsized":"2024-09-21T13:53:26.742656+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-21T13:52:51.445812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:52:51.445812+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:18.734189+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.0015946669999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":162,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.748140+0000","last_change":"2024-09-21T13:51:24.187154+0000","last_active":"2024-09-21T13:53:26.748140+0000","last_peered":"2024-09-21T13:53:26.748140+0000","last_clean":"2024-09-21T13:53:26.748140+0000","last_became_active":"2024-09-21T13:50:29.688249+0000","last_became_peered":"2024-09-21T13:50:29.688249+0000","last_unstale":"2024-09-21T13:53:26.748140+0000","last_undegraded":"2024-09-21T13:53:26.748140+0000","last_fullsized":"2024-09-21T13:53:26.748140+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:24.186934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:24.186934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:33.380668+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":218,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.751653+0000","last_change":"2024-09-21T13:51:23.597164+0000","last_active":"2024-09-21T13:53:26.751653+0000","last_peered":"2024-09-21T13:53:26.751653+0000","last_clean":"2024-09-21T13:53:26.751653+0000","last_became_active":"2024-09-21T13:49:04.080466+0000","last_became_peered":"2024-09-21T13:49:04.080466+0000","last_unstale":"2024-09-21T13:53:26.751653+0000","last_undegraded":"2024-09-21T13:53:26.751653+0000","last_fullsized":"2024-09-21T13:53:26.751653+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:23.596879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:51:23.596879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"165'15293","reported_seq":15474,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:28.406200+0000","last_change":"2024-09-21T13:50:29.694240+0000","last_active":"2024-09-21T13:53:28.406200+0000","last_peered":"2024-09-21T13:53:28.406200+0000","last_clean":"2024-09-21T13:53:28.406200+0000","last_became_active":"2024-09-21T13:50:29.688167+0000","last_became_peered":"2024-09-21T13:50:29.688167+0000","last_unstale":"2024-09-21T13:53:28.406200+0000","last_undegraded":"2024-09-21T13:53:28.406200+0000","last_fullsized":"2024-09-21T13:53:28.406200+0000","mapping_epoch":94,"log_start":"132'10500","ondisk_log_start":"132'10500","created":83,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":4793,"log_dups_size":0,"ondisk_log_size":4793,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":132710400,"num_objects":2025,"num_object_clones":0,"num_object_copies":6075,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15293,"num_write_kb":554176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"68'13","reported_seq":253,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.745669+0000","last_change":"2024-09-21T13:50:06.973399+0000","last_active":"2024-09-21T13:53:26.745669+0000","last_peered":"2024-09-21T13:53:26.745669+0000","last_clean":"2024-09-21T13:53:26.745669+0000","last_became_active":"2024-09-21T13:49:00.053323+0000","last_became_peered":"2024-09-21T13:49:00.053323+0000","last_unstale":"2024-09-21T13:53:26.745669+0000","last_undegraded":"2024-09-21T13:53:26.745669+0000","last_fullsized":"2024-09-21T13:53:26.745669+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'13","last_scrub_stamp":"2024-09-21T13:50:06.973336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.973336+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:36.427623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":257,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.749416+0000","last_change":"2024-09-21T13:53:24.885266+0000","last_active":"2024-09-21T13:53:26.749416+0000","last_peered":"2024-09-21T13:53:26.749416+0000","last_clean":"2024-09-21T13:53:26.749416+0000","last_became_active":"2024-09-21T13:52:04.320581+0000","last_became_peered":"2024-09-21T13:52:04.320581+0000","last_unstale":"2024-09-21T13:53:26.749416+0000","last_undegraded":"2024-09-21T13:53:26.749416+0000","last_fullsized":"2024-09-21T13:53:26.749416+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:24.884932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:53:24.884932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:31.375453+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":156,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.761850+0000","last_change":"2024-09-21T13:53:19.985044+0000","last_active":"2024-09-21T13:53:26.761850+0000","last_peered":"2024-09-21T13:53:26.761850+0000","last_clean":"2024-09-21T13:53:26.761850+0000","last_became_active":"2024-09-21T13:52:04.316171+0000","last_became_peered":"2024-09-21T13:52:04.316171+0000","last_unstale":"2024-09-21T13:53:26.761850+0000","last_undegraded":"2024-09-21T13:53:26.761850+0000","last_fullsized":"2024-09-21T13:53:26.761850+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:53:19.984966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:53:19.984966+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:47.985977+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":100,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.763733+0000","last_change":"2024-09-21T13:53:18.988030+0000","last_active":"2024-09-21T13:53:26.763733+0000","last_peered":"2024-09-21T13:53:26.763733+0000","last_clean":"2024-09-21T13:53:26.763733+0000","last_became_active":"2024-09-21T13:52:04.320701+0000","last_became_peered":"2024-09-21T13:52:04.320701+0000","last_unstale":"2024-09-21T13:53:26.763733+0000","last_undegraded":"2024-09-21T13:53:26.763733+0000","last_fullsized":"2024-09-21T13:53:26.763733+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:18.987829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:53:18.987829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:21.866149+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":226,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.754752+0000","last_change":"2024-09-21T13:51:12.212558+0000","last_active":"2024-09-21T13:53:26.754752+0000","last_peered":"2024-09-21T13:53:26.754752+0000","last_clean":"2024-09-21T13:53:26.754752+0000","last_became_active":"2024-09-21T13:49:04.431035+0000","last_became_peered":"2024-09-21T13:49:04.431035+0000","last_unstale":"2024-09-21T13:53:26.754752+0000","last_undegraded":"2024-09-21T13:53:26.754752+0000","last_fullsized":"2024-09-21T13:53:26.754752+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:12.212391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:51:12.212391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"165'15126","reported_seq":25294,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:28.432818+0000","last_change":"2024-09-21T13:53:21.165869+0000","last_active":"2024-09-21T13:53:28.432818+0000","last_peered":"2024-09-21T13:53:28.432818+0000","last_clean":"2024-09-21T13:53:28.432818+0000","last_became_active":"2024-09-21T13:52:43.938223+0000","last_became_peered":"2024-09-21T13:52:43.938223+0000","last_unstale":"2024-09-21T13:53:28.432818+0000","last_undegraded":"2024-09-21T13:53:28.432818+0000","last_fullsized":"2024-09-21T13:53:28.432818+0000","mapping_epoch":147,"log_start":"127'9900","ondisk_log_start":"127'9900","created":83,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'14639","last_scrub_stamp":"2024-09-21T13:53:21.165793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:53:21.165793+0000","objects_scrubbed":2453,"log_size":5226,"log_dups_size":0,"ondisk_log_size":5226,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:29.496429+0000","scrub_duration":804,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":126877696,"num_objects":1936,"num_object_clones":0,"num_object_copies":5808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15126,"num_write_kb":545984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4971,"num_bytes_recovered":325779456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"68'12","reported_seq":284,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.773147+0000","last_change":"2024-09-21T13:53:17.740156+0000","last_active":"2024-09-21T13:53:26.773147+0000","last_peered":"2024-09-21T13:53:26.773147+0000","last_clean":"2024-09-21T13:53:26.773147+0000","last_became_active":"2024-09-21T13:52:04.318875+0000","last_became_peered":"2024-09-21T13:52:04.318875+0000","last_unstale":"2024-09-21T13:53:26.773147+0000","last_undegraded":"2024-09-21T13:53:26.773147+0000","last_fullsized":"2024-09-21T13:53:26.773147+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'12","last_scrub_stamp":"2024-09-21T13:53:17.740080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:17.740080+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:30.638014+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":162,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.757825+0000","last_change":"2024-09-21T13:50:30.020722+0000","last_active":"2024-09-21T13:53:26.757825+0000","last_peered":"2024-09-21T13:53:26.757825+0000","last_clean":"2024-09-21T13:53:26.757825+0000","last_became_active":"2024-09-21T13:50:29.688352+0000","last_became_peered":"2024-09-21T13:50:29.688352+0000","last_unstale":"2024-09-21T13:53:26.757825+0000","last_undegraded":"2024-09-21T13:53:26.757825+0000","last_fullsized":"2024-09-21T13:53:26.757825+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:30.020656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:30.020656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":506,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:27.268149+0000","last_change":"2024-09-21T13:51:21.271997+0000","last_active":"2024-09-21T13:53:27.268149+0000","last_peered":"2024-09-21T13:53:27.268149+0000","last_clean":"2024-09-21T13:53:27.268149+0000","last_became_active":"2024-09-21T13:48:57.046084+0000","last_became_peered":"2024-09-21T13:48:57.046084+0000","last_unstale":"2024-09-21T13:53:27.268149+0000","last_undegraded":"2024-09-21T13:53:27.268149+0000","last_fullsized":"2024-09-21T13:53:27.268149+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-21T13:51:21.271917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:51:21.271917+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00047048000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":265,"num_read_kb":260,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":222,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.743959+0000","last_change":"2024-09-21T13:51:53.518645+0000","last_active":"2024-09-21T13:53:26.743959+0000","last_peered":"2024-09-21T13:53:26.743959+0000","last_clean":"2024-09-21T13:53:26.743959+0000","last_became_active":"2024-09-21T13:49:02.066641+0000","last_became_peered":"2024-09-21T13:49:02.066641+0000","last_unstale":"2024-09-21T13:53:26.743959+0000","last_undegraded":"2024-09-21T13:53:26.743959+0000","last_fullsized":"2024-09-21T13:53:26.743959+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:53.518567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:53.518567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:14.335443+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":226,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.745005+0000","last_change":"2024-09-21T13:52:21.332066+0000","last_active":"2024-09-21T13:53:26.745005+0000","last_peered":"2024-09-21T13:53:26.745005+0000","last_clean":"2024-09-21T13:53:26.745005+0000","last_became_active":"2024-09-21T13:49:04.078372+0000","last_became_peered":"2024-09-21T13:49:04.078372+0000","last_unstale":"2024-09-21T13:53:26.745005+0000","last_undegraded":"2024-09-21T13:53:26.745005+0000","last_fullsized":"2024-09-21T13:53:26.745005+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:21.331792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:21.331792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:44.913330+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"165'15205","reported_seq":15415,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:28.414054+0000","last_change":"2024-09-21T13:53:27.064451+0000","last_active":"2024-09-21T13:53:28.414054+0000","last_peered":"2024-09-21T13:53:28.414054+0000","last_clean":"2024-09-21T13:53:28.414054+0000","last_became_active":"2024-09-21T13:50:29.694886+0000","last_became_peered":"2024-09-21T13:50:29.694886+0000","last_unstale":"2024-09-21T13:53:28.414054+0000","last_undegraded":"2024-09-21T13:53:28.414054+0000","last_fullsized":"2024-09-21T13:53:28.414054+0000","mapping_epoch":94,"log_start":"127'10000","ondisk_log_start":"127'10000","created":83,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'15120","last_scrub_stamp":"2024-09-21T13:53:27.064254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:53:27.064254+0000","objects_scrubbed":2095,"log_size":5205,"log_dups_size":0,"ondisk_log_size":5205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:50.816602+0000","scrub_duration":619,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130351104,"num_objects":1989,"num_object_clones":0,"num_object_copies":5967,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15205,"num_write_kb":550208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"68'16","reported_seq":311,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.755390+0000","last_change":"2024-09-21T13:52:19.452239+0000","last_active":"2024-09-21T13:53:26.755390+0000","last_peered":"2024-09-21T13:53:26.755390+0000","last_clean":"2024-09-21T13:53:26.755390+0000","last_became_active":"2024-09-21T13:49:00.426867+0000","last_became_peered":"2024-09-21T13:49:00.426867+0000","last_unstale":"2024-09-21T13:53:26.755390+0000","last_undegraded":"2024-09-21T13:53:26.755390+0000","last_fullsized":"2024-09-21T13:53:26.755390+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'16","last_scrub_stamp":"2024-09-21T13:52:19.452163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:52:19.452163+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:46.305610+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":230,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.749525+0000","last_change":"2024-09-21T13:50:09.208053+0000","last_active":"2024-09-21T13:53:26.749525+0000","last_peered":"2024-09-21T13:53:26.749525+0000","last_clean":"2024-09-21T13:53:26.749525+0000","last_became_active":"2024-09-21T13:48:58.052659+0000","last_became_peered":"2024-09-21T13:48:58.052659+0000","last_unstale":"2024-09-21T13:53:26.749525+0000","last_undegraded":"2024-09-21T13:53:26.749525+0000","last_fullsized":"2024-09-21T13:53:26.749525+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:09.208003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:09.208003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"75'79","reported_seq":279,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.753477+0000","last_change":"2024-09-21T13:50:27.096470+0000","last_active":"2024-09-21T13:53:26.753477+0000","last_peered":"2024-09-21T13:53:26.753477+0000","last_clean":"2024-09-21T13:53:26.753477+0000","last_became_active":"2024-09-21T13:48:48.454646+0000","last_became_peered":"2024-09-21T13:48:48.454646+0000","last_unstale":"2024-09-21T13:53:26.753477+0000","last_undegraded":"2024-09-21T13:53:26.753477+0000","last_fullsized":"2024-09-21T13:53:26.753477+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:50:27.096413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:50:27.096413+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":225,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.754608+0000","last_change":"2024-09-21T13:52:04.320948+0000","last_active":"2024-09-21T13:53:26.754608+0000","last_peered":"2024-09-21T13:53:26.754608+0000","last_clean":"2024-09-21T13:53:26.754608+0000","last_became_active":"2024-09-21T13:52:04.320627+0000","last_became_peered":"2024-09-21T13:52:04.320627+0000","last_unstale":"2024-09-21T13:53:26.754608+0000","last_undegraded":"2024-09-21T13:53:26.754608+0000","last_fullsized":"2024-09-21T13:53:26.754608+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:42.557761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":218,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.743618+0000","last_change":"2024-09-21T13:50:25.560928+0000","last_active":"2024-09-21T13:53:26.743618+0000","last_peered":"2024-09-21T13:53:26.743618+0000","last_clean":"2024-09-21T13:53:26.743618+0000","last_became_active":"2024-09-21T13:49:04.078091+0000","last_became_peered":"2024-09-21T13:49:04.078091+0000","last_unstale":"2024-09-21T13:53:26.743618+0000","last_undegraded":"2024-09-21T13:53:26.743618+0000","last_fullsized":"2024-09-21T13:53:26.743618+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:25.560863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:25.560863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:40.548062+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"165'15125","reported_seq":15430,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:28.376776+0000","last_change":"2024-09-21T13:52:50.770818+0000","last_active":"2024-09-21T13:53:28.376776+0000","last_peered":"2024-09-21T13:53:28.376776+0000","last_clean":"2024-09-21T13:53:28.376776+0000","last_became_active":"2024-09-21T13:50:07.422466+0000","last_became_peered":"2024-09-21T13:50:07.422466+0000","last_unstale":"2024-09-21T13:53:28.376776+0000","last_undegraded":"2024-09-21T13:53:28.376776+0000","last_fullsized":"2024-09-21T13:53:28.376776+0000","mapping_epoch":83,"log_start":"122'8800","ondisk_log_start":"122'8800","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'12965","last_scrub_stamp":"2024-09-21T13:52:50.770670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:52:50.770670+0000","objects_scrubbed":4108,"log_size":6325,"log_dups_size":0,"ondisk_log_size":6325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:56.210477+0000","scrub_duration":1333,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125042712,"num_objects":1909,"num_object_clones":0,"num_object_copies":5727,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1909,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15125,"num_write_kb":545025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"68'11","reported_seq":236,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.757341+0000","last_change":"2024-09-21T13:49:00.054827+0000","last_active":"2024-09-21T13:53:26.757341+0000","last_peered":"2024-09-21T13:53:26.757341+0000","last_clean":"2024-09-21T13:53:26.757341+0000","last_became_active":"2024-09-21T13:49:00.053268+0000","last_became_peered":"2024-09-21T13:49:00.053268+0000","last_unstale":"2024-09-21T13:53:26.757341+0000","last_undegraded":"2024-09-21T13:53:26.757341+0000","last_fullsized":"2024-09-21T13:53:26.757341+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:16.039942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":241,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.761281+0000","last_change":"2024-09-21T13:52:48.165210+0000","last_active":"2024-09-21T13:53:26.761281+0000","last_peered":"2024-09-21T13:53:26.761281+0000","last_clean":"2024-09-21T13:53:26.761281+0000","last_became_active":"2024-09-21T13:52:04.318345+0000","last_became_peered":"2024-09-21T13:52:04.318345+0000","last_unstale":"2024-09-21T13:53:26.761281+0000","last_undegraded":"2024-09-21T13:53:26.761281+0000","last_fullsized":"2024-09-21T13:53:26.761281+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:48.165145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:52:48.165145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:03.774580+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":229,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.743539+0000","last_change":"2024-09-21T13:51:46.503640+0000","last_active":"2024-09-21T13:53:26.743539+0000","last_peered":"2024-09-21T13:53:26.743539+0000","last_clean":"2024-09-21T13:53:26.743539+0000","last_became_active":"2024-09-21T13:49:02.061901+0000","last_became_peered":"2024-09-21T13:49:02.061901+0000","last_unstale":"2024-09-21T13:53:26.743539+0000","last_undegraded":"2024-09-21T13:53:26.743539+0000","last_fullsized":"2024-09-21T13:53:26.743539+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:46.503576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:46.503576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:11.300124+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":217,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.749638+0000","last_change":"2024-09-21T13:51:51.814259+0000","last_active":"2024-09-21T13:53:26.749638+0000","last_peered":"2024-09-21T13:53:26.749638+0000","last_clean":"2024-09-21T13:53:26.749638+0000","last_became_active":"2024-09-21T13:49:04.431268+0000","last_became_peered":"2024-09-21T13:49:04.431268+0000","last_unstale":"2024-09-21T13:53:26.749638+0000","last_undegraded":"2024-09-21T13:53:26.749638+0000","last_fullsized":"2024-09-21T13:53:26.749638+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:51.814200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:51:51.814200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"165'15012","reported_seq":22417,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:27.389786+0000","last_change":"2024-09-21T13:53:17.689354+0000","last_active":"2024-09-21T13:53:27.389786+0000","last_peered":"2024-09-21T13:53:27.389786+0000","last_clean":"2024-09-21T13:53:27.389786+0000","last_became_active":"2024-09-21T13:53:17.659357+0000","last_became_peered":"2024-09-21T13:53:17.659357+0000","last_unstale":"2024-09-21T13:53:27.389786+0000","last_undegraded":"2024-09-21T13:53:27.389786+0000","last_fullsized":"2024-09-21T13:53:27.389786+0000","mapping_epoch":160,"log_start":"122'8788","ondisk_log_start":"122'8788","created":83,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":6224,"log_dups_size":0,"ondisk_log_size":6224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129630208,"num_objects":1978,"num_object_clones":0,"num_object_copies":5934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15012,"num_write_kb":543680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3606,"num_bytes_recovered":236322816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"68'17","reported_seq":264,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.756037+0000","last_change":"2024-09-21T13:52:45.176950+0000","last_active":"2024-09-21T13:53:26.756037+0000","last_peered":"2024-09-21T13:53:26.756037+0000","last_clean":"2024-09-21T13:53:26.756037+0000","last_became_active":"2024-09-21T13:52:04.313218+0000","last_became_peered":"2024-09-21T13:52:04.313218+0000","last_unstale":"2024-09-21T13:53:26.756037+0000","last_undegraded":"2024-09-21T13:53:26.756037+0000","last_fullsized":"2024-09-21T13:53:26.756037+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'17","last_scrub_stamp":"2024-09-21T13:52:45.176873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:52:45.176873+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:03.832378+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":238,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.755518+0000","last_change":"2024-09-21T13:52:45.444445+0000","last_active":"2024-09-21T13:53:26.755518+0000","last_peered":"2024-09-21T13:53:26.755518+0000","last_clean":"2024-09-21T13:53:26.755518+0000","last_became_active":"2024-09-21T13:48:58.049142+0000","last_became_peered":"2024-09-21T13:48:58.049142+0000","last_unstale":"2024-09-21T13:53:26.755518+0000","last_undegraded":"2024-09-21T13:53:26.755518+0000","last_fullsized":"2024-09-21T13:53:26.755518+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:45.444381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:52:45.444381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:54.348542+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":213,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.745117+0000","last_change":"2024-09-21T13:49:02.067811+0000","last_active":"2024-09-21T13:53:26.745117+0000","last_peered":"2024-09-21T13:53:26.745117+0000","last_clean":"2024-09-21T13:53:26.745117+0000","last_became_active":"2024-09-21T13:49:02.067472+0000","last_became_peered":"2024-09-21T13:49:02.067472+0000","last_unstale":"2024-09-21T13:53:26.745117+0000","last_undegraded":"2024-09-21T13:53:26.745117+0000","last_fullsized":"2024-09-21T13:53:26.745117+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:02.701863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":100,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.763684+0000","last_change":"2024-09-21T13:53:18.028013+0000","last_active":"2024-09-21T13:53:26.763684+0000","last_peered":"2024-09-21T13:53:26.763684+0000","last_clean":"2024-09-21T13:53:26.763684+0000","last_became_active":"2024-09-21T13:52:04.315920+0000","last_became_peered":"2024-09-21T13:52:04.315920+0000","last_unstale":"2024-09-21T13:53:26.763684+0000","last_undegraded":"2024-09-21T13:53:26.763684+0000","last_fullsized":"2024-09-21T13:53:26.763684+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:18.027944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:18.027944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:26.231349+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"165'14806","reported_seq":14988,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:28.930525+0000","last_change":"2024-09-21T13:50:29.691429+0000","last_active":"2024-09-21T13:53:28.930525+0000","last_peered":"2024-09-21T13:53:28.930525+0000","last_clean":"2024-09-21T13:53:28.930525+0000","last_became_active":"2024-09-21T13:50:29.687898+0000","last_became_peered":"2024-09-21T13:50:29.687898+0000","last_unstale":"2024-09-21T13:53:28.930525+0000","last_undegraded":"2024-09-21T13:53:28.930525+0000","last_fullsized":"2024-09-21T13:53:28.930525+0000","mapping_epoch":94,"log_start":"132'10100","ondisk_log_start":"132'10100","created":83,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":4706,"log_dups_size":0,"ondisk_log_size":4706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117964800,"num_objects":1800,"num_object_clones":0,"num_object_copies":5400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14806,"num_write_kb":531392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"68'10","reported_seq":240,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.758097+0000","last_change":"2024-09-21T13:51:57.155811+0000","last_active":"2024-09-21T13:53:26.758097+0000","last_peered":"2024-09-21T13:53:26.758097+0000","last_clean":"2024-09-21T13:53:26.758097+0000","last_became_active":"2024-09-21T13:49:00.059850+0000","last_became_peered":"2024-09-21T13:49:00.059850+0000","last_unstale":"2024-09-21T13:53:26.758097+0000","last_undegraded":"2024-09-21T13:53:26.758097+0000","last_fullsized":"2024-09-21T13:53:26.758097+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:51:57.155761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:57.155761+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":230,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.748455+0000","last_change":"2024-09-21T13:51:28.565842+0000","last_active":"2024-09-21T13:53:26.748455+0000","last_peered":"2024-09-21T13:53:26.748455+0000","last_clean":"2024-09-21T13:53:26.748455+0000","last_became_active":"2024-09-21T13:48:58.058309+0000","last_became_peered":"2024-09-21T13:48:58.058309+0000","last_unstale":"2024-09-21T13:53:26.748455+0000","last_undegraded":"2024-09-21T13:53:26.748455+0000","last_fullsized":"2024-09-21T13:53:26.748455+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:28.565766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:51:28.565766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"132'10","reported_seq":127,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.768473+0000","last_change":"2024-09-21T13:52:04.320834+0000","last_active":"2024-09-21T13:53:26.768473+0000","last_peered":"2024-09-21T13:53:26.768473+0000","last_clean":"2024-09-21T13:53:26.768473+0000","last_became_active":"2024-09-21T13:52:04.320441+0000","last_became_peered":"2024-09-21T13:52:04.320441+0000","last_unstale":"2024-09-21T13:53:26.768473+0000","last_undegraded":"2024-09-21T13:53:26.768473+0000","last_fullsized":"2024-09-21T13:53:26.768473+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:16.073559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"68'1","reported_seq":232,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.755220+0000","last_change":"2024-09-21T13:52:04.314151+0000","last_active":"2024-09-21T13:53:26.755220+0000","last_peered":"2024-09-21T13:53:26.755220+0000","last_clean":"2024-09-21T13:53:26.755220+0000","last_became_active":"2024-09-21T13:52:04.313210+0000","last_became_peered":"2024-09-21T13:52:04.313210+0000","last_unstale":"2024-09-21T13:53:26.755220+0000","last_undegraded":"2024-09-21T13:53:26.755220+0000","last_fullsized":"2024-09-21T13:53:26.755220+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T13:50:41.000201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:41.000201+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"165'14961","reported_seq":25036,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:27.371312+0000","last_change":"2024-09-21T13:52:42.978682+0000","last_active":"2024-09-21T13:53:27.371312+0000","last_peered":"2024-09-21T13:53:27.371312+0000","last_clean":"2024-09-21T13:53:27.371312+0000","last_became_active":"2024-09-21T13:52:42.942510+0000","last_became_peered":"2024-09-21T13:52:42.942510+0000","last_unstale":"2024-09-21T13:53:27.371312+0000","last_undegraded":"2024-09-21T13:53:27.371312+0000","last_fullsized":"2024-09-21T13:53:27.371312+0000","mapping_epoch":146,"log_start":"118'8078","ondisk_log_start":"118'8078","created":83,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":6883,"log_dups_size":0,"ondisk_log_size":6883,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":133496832,"num_objects":2037,"num_object_clones":0,"num_object_copies":6111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14961,"num_write_kb":543936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4942,"num_bytes_recovered":323878912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"68'9","reported_seq":250,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.757004+0000","last_change":"2024-09-21T13:53:18.740202+0000","last_active":"2024-09-21T13:53:26.757004+0000","last_peered":"2024-09-21T13:53:26.757004+0000","last_clean":"2024-09-21T13:53:26.757004+0000","last_became_active":"2024-09-21T13:49:00.053016+0000","last_became_peered":"2024-09-21T13:49:00.053016+0000","last_unstale":"2024-09-21T13:53:26.757004+0000","last_undegraded":"2024-09-21T13:53:26.757004+0000","last_fullsized":"2024-09-21T13:53:26.757004+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:53:18.740111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:18.740111+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:34.752035+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":230,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.742995+0000","last_change":"2024-09-21T13:50:38.445732+0000","last_active":"2024-09-21T13:53:26.742995+0000","last_peered":"2024-09-21T13:53:26.742995+0000","last_clean":"2024-09-21T13:53:26.742995+0000","last_became_active":"2024-09-21T13:48:58.049543+0000","last_became_peered":"2024-09-21T13:48:58.049543+0000","last_unstale":"2024-09-21T13:53:26.742995+0000","last_undegraded":"2024-09-21T13:53:26.742995+0000","last_fullsized":"2024-09-21T13:53:26.742995+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:38.445667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:38.445667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:35.568002+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":152,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.755219+0000","last_change":"2024-09-21T13:52:59.112648+0000","last_active":"2024-09-21T13:53:26.755219+0000","last_peered":"2024-09-21T13:53:26.755219+0000","last_clean":"2024-09-21T13:53:26.755219+0000","last_became_active":"2024-09-21T13:52:04.317429+0000","last_became_peered":"2024-09-21T13:52:04.317429+0000","last_unstale":"2024-09-21T13:53:26.755219+0000","last_undegraded":"2024-09-21T13:53:26.755219+0000","last_fullsized":"2024-09-21T13:53:26.755219+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:59.112602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:52:59.112602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:59.206422+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":173,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.753311+0000","last_change":"2024-09-21T13:52:51.590860+0000","last_active":"2024-09-21T13:53:26.753311+0000","last_peered":"2024-09-21T13:53:26.753311+0000","last_clean":"2024-09-21T13:53:26.753311+0000","last_became_active":"2024-09-21T13:52:04.321703+0000","last_became_peered":"2024-09-21T13:52:04.321703+0000","last_unstale":"2024-09-21T13:53:26.753311+0000","last_undegraded":"2024-09-21T13:53:26.753311+0000","last_fullsized":"2024-09-21T13:53:26.753311+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:51.590800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:51.590800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:01.244154+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.9","version":"165'15108","reported_seq":15290,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:28.535187+0000","last_change":"2024-09-21T13:50:29.690666+0000","last_active":"2024-09-21T13:53:28.535187+0000","last_peered":"2024-09-21T13:53:28.535187+0000","last_clean":"2024-09-21T13:53:28.535187+0000","last_became_active":"2024-09-21T13:50:29.687815+0000","last_became_peered":"2024-09-21T13:50:29.687815+0000","last_unstale":"2024-09-21T13:53:28.535187+0000","last_undegraded":"2024-09-21T13:53:28.535187+0000","last_fullsized":"2024-09-21T13:53:28.535187+0000","mapping_epoch":94,"log_start":"129'10100","ondisk_log_start":"129'10100","created":83,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":5008,"log_dups_size":0,"ondisk_log_size":5008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:32.358031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124256256,"num_objects":1896,"num_object_clones":0,"num_object_copies":5688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15108,"num_write_kb":544128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"68'19","reported_seq":257,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.755270+0000","last_change":"2024-09-21T13:50:03.769686+0000","last_active":"2024-09-21T13:53:26.755270+0000","last_peered":"2024-09-21T13:53:26.755270+0000","last_clean":"2024-09-21T13:53:26.755270+0000","last_became_active":"2024-09-21T13:49:00.425955+0000","last_became_peered":"2024-09-21T13:49:00.425955+0000","last_unstale":"2024-09-21T13:53:26.755270+0000","last_undegraded":"2024-09-21T13:53:26.755270+0000","last_fullsized":"2024-09-21T13:53:26.755270+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-21T13:50:03.769613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:03.769613+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:32.476093+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":221,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.756748+0000","last_change":"2024-09-21T13:48:58.053940+0000","last_active":"2024-09-21T13:53:26.756748+0000","last_peered":"2024-09-21T13:53:26.756748+0000","last_clean":"2024-09-21T13:53:26.756748+0000","last_became_active":"2024-09-21T13:48:58.053641+0000","last_became_peered":"2024-09-21T13:48:58.053641+0000","last_unstale":"2024-09-21T13:53:26.756748+0000","last_undegraded":"2024-09-21T13:53:26.756748+0000","last_fullsized":"2024-09-21T13:53:26.756748+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:17.391569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":170,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.764833+0000","last_change":"2024-09-21T13:52:04.321471+0000","last_active":"2024-09-21T13:53:26.764833+0000","last_peered":"2024-09-21T13:53:26.764833+0000","last_clean":"2024-09-21T13:53:26.764833+0000","last_became_active":"2024-09-21T13:52:04.321146+0000","last_became_peered":"2024-09-21T13:52:04.321146+0000","last_unstale":"2024-09-21T13:53:26.764833+0000","last_undegraded":"2024-09-21T13:53:26.764833+0000","last_fullsized":"2024-09-21T13:53:26.764833+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:40.850721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:40.850721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":229,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.753459+0000","last_change":"2024-09-21T13:52:04.323179+0000","last_active":"2024-09-21T13:53:26.753459+0000","last_peered":"2024-09-21T13:53:26.753459+0000","last_clean":"2024-09-21T13:53:26.753459+0000","last_became_active":"2024-09-21T13:52:04.322213+0000","last_became_peered":"2024-09-21T13:52:04.322213+0000","last_unstale":"2024-09-21T13:53:26.753459+0000","last_undegraded":"2024-09-21T13:53:26.753459+0000","last_fullsized":"2024-09-21T13:53:26.753459+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:35.067261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:35.067261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"165'15066","reported_seq":24334,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:28.892163+0000","last_change":"2024-09-21T13:53:01.956068+0000","last_active":"2024-09-21T13:53:28.892163+0000","last_peered":"2024-09-21T13:53:28.892163+0000","last_clean":"2024-09-21T13:53:28.892163+0000","last_became_active":"2024-09-21T13:53:01.911422+0000","last_became_peered":"2024-09-21T13:53:01.911422+0000","last_unstale":"2024-09-21T13:53:28.892163+0000","last_undegraded":"2024-09-21T13:53:28.892163+0000","last_fullsized":"2024-09-21T13:53:28.892163+0000","mapping_epoch":153,"log_start":"132'10400","ondisk_log_start":"132'10400","created":83,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":4666,"log_dups_size":0,"ondisk_log_size":4666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127270912,"num_objects":1942,"num_object_clones":0,"num_object_copies":5826,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15066,"num_write_kb":544256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4537,"num_bytes_recovered":297336832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"68'12","reported_seq":249,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.761235+0000","last_change":"2024-09-21T13:51:10.942190+0000","last_active":"2024-09-21T13:53:26.761235+0000","last_peered":"2024-09-21T13:53:26.761235+0000","last_clean":"2024-09-21T13:53:26.761235+0000","last_became_active":"2024-09-21T13:49:00.059720+0000","last_became_peered":"2024-09-21T13:49:00.059720+0000","last_unstale":"2024-09-21T13:53:26.761235+0000","last_undegraded":"2024-09-21T13:53:26.761235+0000","last_fullsized":"2024-09-21T13:53:26.761235+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'12","last_scrub_stamp":"2024-09-21T13:51:10.942105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:10.942105+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":229,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.757801+0000","last_change":"2024-09-21T13:50:29.100595+0000","last_active":"2024-09-21T13:53:26.757801+0000","last_peered":"2024-09-21T13:53:26.757801+0000","last_clean":"2024-09-21T13:53:26.757801+0000","last_became_active":"2024-09-21T13:48:58.057641+0000","last_became_peered":"2024-09-21T13:48:58.057641+0000","last_unstale":"2024-09-21T13:53:26.757801+0000","last_undegraded":"2024-09-21T13:53:26.757801+0000","last_fullsized":"2024-09-21T13:53:26.757801+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:29.100547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:29.100547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":135,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.761815+0000","last_change":"2024-09-21T13:52:04.322501+0000","last_active":"2024-09-21T13:53:26.761815+0000","last_peered":"2024-09-21T13:53:26.761815+0000","last_clean":"2024-09-21T13:53:26.761815+0000","last_became_active":"2024-09-21T13:52:04.322147+0000","last_became_peered":"2024-09-21T13:52:04.322147+0000","last_unstale":"2024-09-21T13:53:26.761815+0000","last_undegraded":"2024-09-21T13:53:26.761815+0000","last_fullsized":"2024-09-21T13:53:26.761815+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:02.761516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:02.761516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":217,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.748113+0000","last_change":"2024-09-21T13:53:20.888048+0000","last_active":"2024-09-21T13:53:26.748113+0000","last_peered":"2024-09-21T13:53:26.748113+0000","last_clean":"2024-09-21T13:53:26.748113+0000","last_became_active":"2024-09-21T13:49:04.078089+0000","last_became_peered":"2024-09-21T13:49:04.078089+0000","last_unstale":"2024-09-21T13:53:26.748113+0000","last_undegraded":"2024-09-21T13:53:26.748113+0000","last_fullsized":"2024-09-21T13:53:26.748113+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:20.882449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:20.882449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:45.947326+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"165'15256","reported_seq":15432,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:28.517101+0000","last_change":"2024-09-21T13:50:07.422240+0000","last_active":"2024-09-21T13:53:28.517101+0000","last_peered":"2024-09-21T13:53:28.517101+0000","last_clean":"2024-09-21T13:53:28.517101+0000","last_became_active":"2024-09-21T13:50:07.421962+0000","last_became_peered":"2024-09-21T13:50:07.421962+0000","last_unstale":"2024-09-21T13:53:28.517101+0000","last_undegraded":"2024-09-21T13:53:28.517101+0000","last_fullsized":"2024-09-21T13:53:28.517101+0000","mapping_epoch":83,"log_start":"128'10200","ondisk_log_start":"128'10200","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":5056,"log_dups_size":0,"ondisk_log_size":5056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:37.396702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127664128,"num_objects":1948,"num_object_clones":0,"num_object_copies":5844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15256,"num_write_kb":550528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"68'15","reported_seq":251,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.743638+0000","last_change":"2024-09-21T13:50:04.445409+0000","last_active":"2024-09-21T13:53:26.743638+0000","last_peered":"2024-09-21T13:53:26.743638+0000","last_clean":"2024-09-21T13:53:26.743638+0000","last_became_active":"2024-09-21T13:49:00.425997+0000","last_became_peered":"2024-09-21T13:49:00.425997+0000","last_unstale":"2024-09-21T13:53:26.743638+0000","last_undegraded":"2024-09-21T13:53:26.743638+0000","last_fullsized":"2024-09-21T13:53:26.743638+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:50:04.445328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:04.445328+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:30.619277+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":249,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.755101+0000","last_change":"2024-09-21T13:53:19.396344+0000","last_active":"2024-09-21T13:53:26.755101+0000","last_peered":"2024-09-21T13:53:26.755101+0000","last_clean":"2024-09-21T13:53:26.755101+0000","last_became_active":"2024-09-21T13:48:58.057632+0000","last_became_peered":"2024-09-21T13:48:58.057632+0000","last_unstale":"2024-09-21T13:53:26.755101+0000","last_undegraded":"2024-09-21T13:53:26.755101+0000","last_fullsized":"2024-09-21T13:53:26.755101+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T13:53:19.396277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:53:19.396277+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:47.867330+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"68'8","reported_seq":285,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.753408+0000","last_change":"2024-09-21T13:51:26.297643+0000","last_active":"2024-09-21T13:53:26.753408+0000","last_peered":"2024-09-21T13:53:26.753408+0000","last_clean":"2024-09-21T13:53:26.753408+0000","last_became_active":"2024-09-21T13:49:02.056167+0000","last_became_peered":"2024-09-21T13:49:02.056167+0000","last_unstale":"2024-09-21T13:53:26.753408+0000","last_undegraded":"2024-09-21T13:53:26.753408+0000","last_fullsized":"2024-09-21T13:53:26.753408+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:51:26.297550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:26.297550+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":226,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.742502+0000","last_change":"2024-09-21T13:52:54.314087+0000","last_active":"2024-09-21T13:53:26.742502+0000","last_peered":"2024-09-21T13:53:26.742502+0000","last_clean":"2024-09-21T13:53:26.742502+0000","last_became_active":"2024-09-21T13:49:04.432546+0000","last_became_peered":"2024-09-21T13:49:04.432546+0000","last_unstale":"2024-09-21T13:53:26.742502+0000","last_undegraded":"2024-09-21T13:53:26.742502+0000","last_fullsized":"2024-09-21T13:53:26.742502+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:54.314042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:54.314042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:05.575071+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":237,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.755068+0000","last_change":"2024-09-21T13:51:54.352313+0000","last_active":"2024-09-21T13:53:26.755068+0000","last_peered":"2024-09-21T13:53:26.755068+0000","last_clean":"2024-09-21T13:53:26.755068+0000","last_became_active":"2024-09-21T13:48:58.056080+0000","last_became_peered":"2024-09-21T13:48:58.056080+0000","last_unstale":"2024-09-21T13:53:26.755068+0000","last_undegraded":"2024-09-21T13:53:26.755068+0000","last_fullsized":"2024-09-21T13:53:26.755068+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:54.352244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:51:54.352244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:14.235456+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"68'6","reported_seq":226,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.752140+0000","last_change":"2024-09-21T13:49:00.427204+0000","last_active":"2024-09-21T13:53:26.752140+0000","last_peered":"2024-09-21T13:53:26.752140+0000","last_clean":"2024-09-21T13:53:26.752140+0000","last_became_active":"2024-09-21T13:49:00.426829+0000","last_became_peered":"2024-09-21T13:49:00.426829+0000","last_unstale":"2024-09-21T13:53:26.752140+0000","last_undegraded":"2024-09-21T13:53:26.752140+0000","last_fullsized":"2024-09-21T13:53:26.752140+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":229,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.743673+0000","last_change":"2024-09-21T13:51:55.488973+0000","last_active":"2024-09-21T13:53:26.743673+0000","last_peered":"2024-09-21T13:53:26.743673+0000","last_clean":"2024-09-21T13:53:26.743673+0000","last_became_active":"2024-09-21T13:49:02.062231+0000","last_became_peered":"2024-09-21T13:49:02.062231+0000","last_unstale":"2024-09-21T13:53:26.743673+0000","last_undegraded":"2024-09-21T13:53:26.743673+0000","last_fullsized":"2024-09-21T13:53:26.743673+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:55.488908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:55.488908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:16.351786+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":225,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.756891+0000","last_change":"2024-09-21T13:53:18.373418+0000","last_active":"2024-09-21T13:53:26.756891+0000","last_peered":"2024-09-21T13:53:26.756891+0000","last_clean":"2024-09-21T13:53:26.756891+0000","last_became_active":"2024-09-21T13:49:04.432507+0000","last_became_peered":"2024-09-21T13:49:04.432507+0000","last_unstale":"2024-09-21T13:53:26.756891+0000","last_undegraded":"2024-09-21T13:53:26.756891+0000","last_fullsized":"2024-09-21T13:53:26.756891+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:18.373356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:18.373356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:38.812691+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"68'9","reported_seq":198,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.752231+0000","last_change":"2024-09-21T13:51:54.808758+0000","last_active":"2024-09-21T13:53:26.752231+0000","last_peered":"2024-09-21T13:53:26.752231+0000","last_clean":"2024-09-21T13:53:26.752231+0000","last_became_active":"2024-09-21T13:50:29.693589+0000","last_became_peered":"2024-09-21T13:50:29.693589+0000","last_unstale":"2024-09-21T13:53:26.752231+0000","last_undegraded":"2024-09-21T13:53:26.752231+0000","last_fullsized":"2024-09-21T13:53:26.752231+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:51:54.808559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:54.808559+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"61'1","reported_seq":227,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.753522+0000","last_change":"2024-09-21T13:48:58.050948+0000","last_active":"2024-09-21T13:53:26.753522+0000","last_peered":"2024-09-21T13:53:26.753522+0000","last_clean":"2024-09-21T13:53:26.753522+0000","last_became_active":"2024-09-21T13:48:58.050645+0000","last_became_peered":"2024-09-21T13:48:58.050645+0000","last_unstale":"2024-09-21T13:53:26.753522+0000","last_undegraded":"2024-09-21T13:53:26.753522+0000","last_fullsized":"2024-09-21T13:53:26.753522+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:13.439561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":221,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.750470+0000","last_change":"2024-09-21T13:50:43.375425+0000","last_active":"2024-09-21T13:53:26.750470+0000","last_peered":"2024-09-21T13:53:26.750470+0000","last_clean":"2024-09-21T13:53:26.750470+0000","last_became_active":"2024-09-21T13:49:02.061775+0000","last_became_peered":"2024-09-21T13:49:02.061775+0000","last_unstale":"2024-09-21T13:53:26.750470+0000","last_undegraded":"2024-09-21T13:53:26.750470+0000","last_fullsized":"2024-09-21T13:53:26.750470+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:43.375381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:43.375381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":166,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.762987+0000","last_change":"2024-09-21T13:52:04.324521+0000","last_active":"2024-09-21T13:53:26.762987+0000","last_peered":"2024-09-21T13:53:26.762987+0000","last_clean":"2024-09-21T13:53:26.762987+0000","last_became_active":"2024-09-21T13:52:04.323549+0000","last_became_peered":"2024-09-21T13:52:04.323549+0000","last_unstale":"2024-09-21T13:53:26.762987+0000","last_undegraded":"2024-09-21T13:53:26.762987+0000","last_fullsized":"2024-09-21T13:53:26.762987+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:41.808746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:41.808746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"68'9","reported_seq":241,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.743039+0000","last_change":"2024-09-21T13:51:28.547052+0000","last_active":"2024-09-21T13:53:26.743039+0000","last_peered":"2024-09-21T13:53:26.743039+0000","last_clean":"2024-09-21T13:53:26.743039+0000","last_became_active":"2024-09-21T13:49:00.427766+0000","last_became_peered":"2024-09-21T13:49:00.427766+0000","last_unstale":"2024-09-21T13:53:26.743039+0000","last_undegraded":"2024-09-21T13:53:26.743039+0000","last_fullsized":"2024-09-21T13:53:26.743039+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:51:28.546958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:28.546958+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:08.329203+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":162,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.753717+0000","last_change":"2024-09-21T13:50:30.076191+0000","last_active":"2024-09-21T13:53:26.753717+0000","last_peered":"2024-09-21T13:53:26.753717+0000","last_clean":"2024-09-21T13:53:26.753717+0000","last_became_active":"2024-09-21T13:50:29.690634+0000","last_became_peered":"2024-09-21T13:50:29.690634+0000","last_unstale":"2024-09-21T13:53:26.753717+0000","last_undegraded":"2024-09-21T13:53:26.753717+0000","last_fullsized":"2024-09-21T13:53:26.753717+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:30.076115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:30.076115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"132'10","reported_seq":297,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.749865+0000","last_change":"2024-09-21T13:52:04.319377+0000","last_active":"2024-09-21T13:53:26.749865+0000","last_peered":"2024-09-21T13:53:26.749865+0000","last_clean":"2024-09-21T13:53:26.749865+0000","last_became_active":"2024-09-21T13:52:04.318626+0000","last_became_peered":"2024-09-21T13:52:04.318626+0000","last_unstale":"2024-09-21T13:53:26.749865+0000","last_undegraded":"2024-09-21T13:53:26.749865+0000","last_fullsized":"2024-09-21T13:53:26.749865+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:50:29.249680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:29.249680+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":228,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.757715+0000","last_change":"2024-09-21T13:52:04.316179+0000","last_active":"2024-09-21T13:53:26.757715+0000","last_peered":"2024-09-21T13:53:26.757715+0000","last_clean":"2024-09-21T13:53:26.757715+0000","last_became_active":"2024-09-21T13:52:04.315865+0000","last_became_peered":"2024-09-21T13:52:04.315865+0000","last_unstale":"2024-09-21T13:53:26.757715+0000","last_undegraded":"2024-09-21T13:53:26.757715+0000","last_fullsized":"2024-09-21T13:53:26.757715+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:38.983326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:38.983326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"68'10","reported_seq":240,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.750147+0000","last_change":"2024-09-21T13:51:11.545546+0000","last_active":"2024-09-21T13:53:26.750147+0000","last_peered":"2024-09-21T13:53:26.750147+0000","last_clean":"2024-09-21T13:53:26.750147+0000","last_became_active":"2024-09-21T13:49:00.426071+0000","last_became_peered":"2024-09-21T13:49:00.426071+0000","last_unstale":"2024-09-21T13:53:26.750147+0000","last_undegraded":"2024-09-21T13:53:26.750147+0000","last_fullsized":"2024-09-21T13:53:26.750147+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:51:11.545458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:11.545458+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":240,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.754218+0000","last_change":"2024-09-21T13:52:04.318335+0000","last_active":"2024-09-21T13:53:26.754218+0000","last_peered":"2024-09-21T13:53:26.754218+0000","last_clean":"2024-09-21T13:53:26.754218+0000","last_became_active":"2024-09-21T13:52:04.318006+0000","last_became_peered":"2024-09-21T13:52:04.318006+0000","last_unstale":"2024-09-21T13:53:26.754218+0000","last_undegraded":"2024-09-21T13:53:26.754218+0000","last_fullsized":"2024-09-21T13:53:26.754218+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:41.162315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:41.162315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"68'8","reported_seq":276,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.742892+0000","last_change":"2024-09-21T13:49:02.067827+0000","last_active":"2024-09-21T13:53:26.742892+0000","last_peered":"2024-09-21T13:53:26.742892+0000","last_clean":"2024-09-21T13:53:26.742892+0000","last_became_active":"2024-09-21T13:49:02.067487+0000","last_became_peered":"2024-09-21T13:49:02.067487+0000","last_unstale":"2024-09-21T13:53:26.742892+0000","last_undegraded":"2024-09-21T13:53:26.742892+0000","last_fullsized":"2024-09-21T13:53:26.742892+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:02.411463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":162,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.757439+0000","last_change":"2024-09-21T13:50:33.050276+0000","last_active":"2024-09-21T13:53:26.757439+0000","last_peered":"2024-09-21T13:53:26.757439+0000","last_clean":"2024-09-21T13:53:26.757439+0000","last_became_active":"2024-09-21T13:50:29.695082+0000","last_became_peered":"2024-09-21T13:50:29.695082+0000","last_unstale":"2024-09-21T13:53:26.757439+0000","last_undegraded":"2024-09-21T13:53:26.757439+0000","last_fullsized":"2024-09-21T13:53:26.757439+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:33.050227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:33.050227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"68'11","reported_seq":252,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.755453+0000","last_change":"2024-09-21T13:53:26.146937+0000","last_active":"2024-09-21T13:53:26.755453+0000","last_peered":"2024-09-21T13:53:26.755453+0000","last_clean":"2024-09-21T13:53:26.755453+0000","last_became_active":"2024-09-21T13:49:00.055024+0000","last_became_peered":"2024-09-21T13:49:00.055024+0000","last_unstale":"2024-09-21T13:53:26.755453+0000","last_undegraded":"2024-09-21T13:53:26.755453+0000","last_fullsized":"2024-09-21T13:53:26.755453+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:53:26.146725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:26.146725+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:44.146330+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":229,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.758260+0000","last_change":"2024-09-21T13:51:20.956291+0000","last_active":"2024-09-21T13:53:26.758260+0000","last_peered":"2024-09-21T13:53:26.758260+0000","last_clean":"2024-09-21T13:53:26.758260+0000","last_became_active":"2024-09-21T13:48:58.051541+0000","last_became_peered":"2024-09-21T13:48:58.051541+0000","last_unstale":"2024-09-21T13:53:26.758260+0000","last_undegraded":"2024-09-21T13:53:26.758260+0000","last_fullsized":"2024-09-21T13:53:26.758260+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:20.956226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:51:20.956226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":229,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.746060+0000","last_change":"2024-09-21T13:52:14.465671+0000","last_active":"2024-09-21T13:53:26.746060+0000","last_peered":"2024-09-21T13:53:26.746060+0000","last_clean":"2024-09-21T13:53:26.746060+0000","last_became_active":"2024-09-21T13:49:02.062868+0000","last_became_peered":"2024-09-21T13:49:02.062868+0000","last_unstale":"2024-09-21T13:53:26.746060+0000","last_undegraded":"2024-09-21T13:53:26.746060+0000","last_fullsized":"2024-09-21T13:53:26.746060+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:14.465434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:52:14.465434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":217,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.746296+0000","last_change":"2024-09-21T13:51:24.549993+0000","last_active":"2024-09-21T13:53:26.746296+0000","last_peered":"2024-09-21T13:53:26.746296+0000","last_clean":"2024-09-21T13:53:26.746296+0000","last_became_active":"2024-09-21T13:49:04.082961+0000","last_became_peered":"2024-09-21T13:49:04.082961+0000","last_unstale":"2024-09-21T13:53:26.746296+0000","last_undegraded":"2024-09-21T13:53:26.746296+0000","last_fullsized":"2024-09-21T13:53:26.746296+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:24.549706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:51:24.549706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"68'9","reported_seq":252,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.747514+0000","last_change":"2024-09-21T13:52:10.447789+0000","last_active":"2024-09-21T13:53:26.747514+0000","last_peered":"2024-09-21T13:53:26.747514+0000","last_clean":"2024-09-21T13:53:26.747514+0000","last_became_active":"2024-09-21T13:52:04.320149+0000","last_became_peered":"2024-09-21T13:52:04.320149+0000","last_unstale":"2024-09-21T13:53:26.747514+0000","last_undegraded":"2024-09-21T13:53:26.747514+0000","last_fullsized":"2024-09-21T13:53:26.747514+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:52:10.447530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:52:10.447530+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:37.291172+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":221,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.756576+0000","last_change":"2024-09-21T13:48:58.058032+0000","last_active":"2024-09-21T13:53:26.756576+0000","last_peered":"2024-09-21T13:53:26.756576+0000","last_clean":"2024-09-21T13:53:26.756576+0000","last_became_active":"2024-09-21T13:48:58.057516+0000","last_became_peered":"2024-09-21T13:48:58.057516+0000","last_unstale":"2024-09-21T13:53:26.756576+0000","last_undegraded":"2024-09-21T13:53:26.756576+0000","last_fullsized":"2024-09-21T13:53:26.756576+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:06.571008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":221,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.748193+0000","last_change":"2024-09-21T13:50:39.411785+0000","last_active":"2024-09-21T13:53:26.748193+0000","last_peered":"2024-09-21T13:53:26.748193+0000","last_clean":"2024-09-21T13:53:26.748193+0000","last_became_active":"2024-09-21T13:49:02.071922+0000","last_became_peered":"2024-09-21T13:49:02.071922+0000","last_unstale":"2024-09-21T13:53:26.748193+0000","last_undegraded":"2024-09-21T13:53:26.748193+0000","last_fullsized":"2024-09-21T13:53:26.748193+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:39.411718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:39.411718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":154,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.743554+0000","last_change":"2024-09-21T13:50:29.689537+0000","last_active":"2024-09-21T13:53:26.743554+0000","last_peered":"2024-09-21T13:53:26.743554+0000","last_clean":"2024-09-21T13:53:26.743554+0000","last_became_active":"2024-09-21T13:50:29.689045+0000","last_became_peered":"2024-09-21T13:50:29.689045+0000","last_unstale":"2024-09-21T13:53:26.743554+0000","last_undegraded":"2024-09-21T13:53:26.743554+0000","last_fullsized":"2024-09-21T13:53:26.743554+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:10.347114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"68'11","reported_seq":244,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.746425+0000","last_change":"2024-09-21T13:50:31.329973+0000","last_active":"2024-09-21T13:53:26.746425+0000","last_peered":"2024-09-21T13:53:26.746425+0000","last_clean":"2024-09-21T13:53:26.746425+0000","last_became_active":"2024-09-21T13:49:00.426091+0000","last_became_peered":"2024-09-21T13:49:00.426091+0000","last_unstale":"2024-09-21T13:53:26.746425+0000","last_undegraded":"2024-09-21T13:53:26.746425+0000","last_fullsized":"2024-09-21T13:53:26.746425+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:50:31.329740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:31.329740+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":221,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.743773+0000","last_change":"2024-09-21T13:48:58.052281+0000","last_active":"2024-09-21T13:53:26.743773+0000","last_peered":"2024-09-21T13:53:26.743773+0000","last_clean":"2024-09-21T13:53:26.743773+0000","last_became_active":"2024-09-21T13:48:58.051974+0000","last_became_peered":"2024-09-21T13:48:58.051974+0000","last_unstale":"2024-09-21T13:53:26.743773+0000","last_undegraded":"2024-09-21T13:53:26.743773+0000","last_fullsized":"2024-09-21T13:53:26.743773+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:07.361374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":221,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.753122+0000","last_change":"2024-09-21T13:50:37.118637+0000","last_active":"2024-09-21T13:53:26.753122+0000","last_peered":"2024-09-21T13:53:26.753122+0000","last_clean":"2024-09-21T13:53:26.753122+0000","last_became_active":"2024-09-21T13:49:02.056006+0000","last_became_peered":"2024-09-21T13:49:02.056006+0000","last_unstale":"2024-09-21T13:53:26.753122+0000","last_undegraded":"2024-09-21T13:53:26.753122+0000","last_fullsized":"2024-09-21T13:53:26.753122+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:37.118594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:37.118594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":217,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.748327+0000","last_change":"2024-09-21T13:51:55.827109+0000","last_active":"2024-09-21T13:53:26.748327+0000","last_peered":"2024-09-21T13:53:26.748327+0000","last_clean":"2024-09-21T13:53:26.748327+0000","last_became_active":"2024-09-21T13:49:04.434034+0000","last_became_peered":"2024-09-21T13:49:04.434034+0000","last_unstale":"2024-09-21T13:53:26.748327+0000","last_undegraded":"2024-09-21T13:53:26.748327+0000","last_fullsized":"2024-09-21T13:53:26.748327+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:55.826922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:51:55.826922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"68'4","reported_seq":231,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:27.826804+0000","last_change":"2024-09-21T13:53:27.826804+0000","last_active":"2024-09-21T13:53:27.826804+0000","last_peered":"2024-09-21T13:53:27.826804+0000","last_clean":"2024-09-21T13:53:27.826804+0000","last_became_active":"2024-09-21T13:49:00.063066+0000","last_became_peered":"2024-09-21T13:49:00.063066+0000","last_unstale":"2024-09-21T13:53:27.826804+0000","last_undegraded":"2024-09-21T13:53:27.826804+0000","last_fullsized":"2024-09-21T13:53:27.826804+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4","last_scrub_stamp":"2024-09-21T13:53:27.826729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:27.826729+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:54.701343+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":154,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.743910+0000","last_change":"2024-09-21T13:50:29.690572+0000","last_active":"2024-09-21T13:53:26.743910+0000","last_peered":"2024-09-21T13:53:26.743910+0000","last_clean":"2024-09-21T13:53:26.743910+0000","last_became_active":"2024-09-21T13:50:29.690269+0000","last_became_peered":"2024-09-21T13:50:29.690269+0000","last_unstale":"2024-09-21T13:53:26.743910+0000","last_undegraded":"2024-09-21T13:53:26.743910+0000","last_fullsized":"2024-09-21T13:53:26.743910+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:04.145682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:04.145682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:31.629620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":229,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.755952+0000","last_change":"2024-09-21T13:52:48.472414+0000","last_active":"2024-09-21T13:53:26.755952+0000","last_peered":"2024-09-21T13:53:26.755952+0000","last_clean":"2024-09-21T13:53:26.755952+0000","last_became_active":"2024-09-21T13:49:02.064789+0000","last_became_peered":"2024-09-21T13:49:02.064789+0000","last_unstale":"2024-09-21T13:53:26.755952+0000","last_undegraded":"2024-09-21T13:53:26.755952+0000","last_fullsized":"2024-09-21T13:53:26.755952+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:48.472373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:52:48.472373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:51.057252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":228,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.753363+0000","last_change":"2024-09-21T13:52:04.322055+0000","last_active":"2024-09-21T13:53:26.753363+0000","last_peered":"2024-09-21T13:53:26.753363+0000","last_clean":"2024-09-21T13:53:26.753363+0000","last_became_active":"2024-09-21T13:52:04.321667+0000","last_became_peered":"2024-09-21T13:52:04.321667+0000","last_unstale":"2024-09-21T13:53:26.753363+0000","last_undegraded":"2024-09-21T13:53:26.753363+0000","last_fullsized":"2024-09-21T13:53:26.753363+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:39.151320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:39.151320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":225,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.748042+0000","last_change":"2024-09-21T13:52:27.409588+0000","last_active":"2024-09-21T13:53:26.748042+0000","last_peered":"2024-09-21T13:53:26.748042+0000","last_clean":"2024-09-21T13:53:26.748042+0000","last_became_active":"2024-09-21T13:49:04.075562+0000","last_became_peered":"2024-09-21T13:49:04.075562+0000","last_unstale":"2024-09-21T13:53:26.748042+0000","last_undegraded":"2024-09-21T13:53:26.748042+0000","last_fullsized":"2024-09-21T13:53:26.748042+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:27.409408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:27.409408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:52.946524+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":229,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.758793+0000","last_change":"2024-09-21T13:53:25.873998+0000","last_active":"2024-09-21T13:53:26.758793+0000","last_peered":"2024-09-21T13:53:26.758793+0000","last_clean":"2024-09-21T13:53:26.758793+0000","last_became_active":"2024-09-21T13:48:58.055497+0000","last_became_peered":"2024-09-21T13:48:58.055497+0000","last_unstale":"2024-09-21T13:53:26.758793+0000","last_undegraded":"2024-09-21T13:53:26.758793+0000","last_fullsized":"2024-09-21T13:53:26.758793+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:25.865470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:53:25.865470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:51.548858+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"68'11","reported_seq":275,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.759428+0000","last_change":"2024-09-21T13:52:04.321178+0000","last_active":"2024-09-21T13:53:26.759428+0000","last_peered":"2024-09-21T13:53:26.759428+0000","last_clean":"2024-09-21T13:53:26.759428+0000","last_became_active":"2024-09-21T13:52:04.320647+0000","last_became_peered":"2024-09-21T13:52:04.320647+0000","last_unstale":"2024-09-21T13:53:26.759428+0000","last_undegraded":"2024-09-21T13:53:26.759428+0000","last_fullsized":"2024-09-21T13:53:26.759428+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:50:35.790090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:35.790090+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:07.227949+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":173,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-09-21T13:53:26.757356+0000","last_change":"2024-09-21T13:52:04.323345+0000","last_active":"2024-09-21T13:53:26.757356+0000","last_peered":"2024-09-21T13:53:26.757356+0000","last_clean":"2024-09-21T13:53:26.757356+0000","last_became_active":"2024-09-21T13:52:04.322880+0000","last_became_peered":"2024-09-21T13:52:04.322880+0000","last_unstale":"2024-09-21T13:53:26.757356+0000","last_undegraded":"2024-09-21T13:53:26.757356+0000","last_fullsized":"2024-09-21T13:53:26.757356+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:36.086691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:36.086691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":2042691608,"num_objects":31170,"num_object_clones":0,"num_object_copies":93510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1940,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31170,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":241666,"num_write_kb":8730689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21762,"num_bytes_recovered":1426194432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6210596864,"data_stored":6210584648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86700,"ondisk_log_size":86700,"up":48,"acting":47,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69,"ondisk_log_size":69,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":694,"num_read_kb":451,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":395,"ondisk_log_size":395,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":265,"num_read_kb":260,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234043,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":741944,"kb_used_data":624676,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":93003208,"statfs":{"total":95995035648,"available":95235284992,"internally_reserved":0,"allocated":639668224,"data_stored":639260113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":120060362},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462976,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":746212,"kb_used_data":630356,"kb_used_omap":1,"kb_used_meta":115838,"kb_avail":92998940,"statfs":{"total":95995035648,"available":95230914560,"internally_reserved":0,"allocated":645484544,"data_stored":645091756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":118618566},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724615,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":611940,"kb_used_data":504400,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93133212,"statfs":{"total":95995035648,"available":95368409088,"internally_reserved":0,"allocated":516505600,"data_stored":516117256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920845,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":751100,"kb_used_data":633836,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":92994052,"statfs":{"total":95995035648,"available":95225909248,"internally_reserved":0,"allocated":649048064,"data_stored":648630750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":120060357},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117074,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1305092,"kb_used_data":1111408,"kb_used_omap":1,"kb_used_meta":193598,"kb_avail":92440060,"statfs":{"total":95995035648,"available":94658621440,"internally_reserved":0,"allocated":1138081792,"data_stored":1137661212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":198244807},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,6,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":130,"seq":558345748498,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1075176,"kb_used_data":876372,"kb_used_omap":4,"kb_used_meta":198779,"kb_avail":92669976,"statfs":{"total":95995035648,"available":94894055424,"internally_reserved":0,"allocated":897404928,"data_stored":897012510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4441,"internal_metadata":203550375},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,0,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574943,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":882840,"kb_used_data":765576,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":92862312,"statfs":{"total":95995035648,"available":95091007488,"internally_reserved":0,"allocated":783949824,"data_stored":783504386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":120060357},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738469,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1181500,"kb_used_data":1015656,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":92563652,"statfs":{"total":95995035648,"available":94785179648,"internally_reserved":0,"allocated":1040031744,"data_stored":1039618834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":169802181},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1003098112,"data_stored":1003094040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":763629568,"data_stored":763625496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":882966528,"data_stored":882966528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1129578496,"data_stored":1129578496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":642580480,"data_stored":642580480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":511377408,"data_stored":511377408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":640749568,"data_stored":640745496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":636616704,"data_stored":636616704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T13:53:33.438 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:33.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:33.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:53:33.539 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:53:33.542 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T13:53:33.542 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:53:33.564 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:33.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:33.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:53:33.665 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:53:33.699 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:33.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:33.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:53:33.756 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:53:33.801 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:53:33.814 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-09-21T13:53:33.828 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:33.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:33.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:53:33.929 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:33.957 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:33.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:33.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:34.059 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:53:34.089 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:34.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:34.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:53:34.184 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:33 smithi049 ceph-mon[40489]: 3.17 scrub starts 2024-09-21T13:53:34.184 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:33 smithi049 ceph-mon[40489]: 3.17 scrub ok 2024-09-21T13:53:34.185 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:33 smithi049 ceph-mon[40489]: 5.2 scrub starts 2024-09-21T13:53:34.185 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:33 smithi049 ceph-mon[40489]: 5.2 scrub ok 2024-09-21T13:53:34.185 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:33 smithi049 ceph-mon[40489]: 4.1c scrub starts 2024-09-21T13:53:34.185 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:33 smithi049 ceph-mon[40489]: 4.1c scrub ok 2024-09-21T13:53:34.185 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:33 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2496875124' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:53:34.185 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:33 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:53:34.185 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:33 smithi049 ceph-mon[30080]: 3.17 scrub starts 2024-09-21T13:53:34.185 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:33 smithi049 ceph-mon[30080]: 3.17 scrub ok 2024-09-21T13:53:34.185 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:33 smithi049 ceph-mon[30080]: 5.2 scrub starts 2024-09-21T13:53:34.185 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:33 smithi049 ceph-mon[30080]: 5.2 scrub ok 2024-09-21T13:53:34.186 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:33 smithi049 ceph-mon[30080]: 4.1c scrub starts 2024-09-21T13:53:34.186 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:33 smithi049 ceph-mon[30080]: 4.1c scrub ok 2024-09-21T13:53:34.186 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:33 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2496875124' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:53:34.186 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:33 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:53:34.190 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:53:34.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:33 smithi187 ceph-mon[33032]: 3.17 scrub starts 2024-09-21T13:53:34.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:33 smithi187 ceph-mon[33032]: 3.17 scrub ok 2024-09-21T13:53:34.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:33 smithi187 ceph-mon[33032]: 5.2 scrub starts 2024-09-21T13:53:34.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:33 smithi187 ceph-mon[33032]: 5.2 scrub ok 2024-09-21T13:53:34.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:33 smithi187 ceph-mon[33032]: 4.1c scrub starts 2024-09-21T13:53:34.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:33 smithi187 ceph-mon[33032]: 4.1c scrub ok 2024-09-21T13:53:34.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:33 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2496875124' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:53:34.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:33 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:53:34.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:34.215 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:34.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:53:34.236 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T13:53:34.316 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:53:34.341 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:34.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:34.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:53:34.442 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:53:34.498 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:34.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:34.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:53:34.600 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:53:34.643 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:34.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:34.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:53:34.745 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:53:34.771 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:34.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:34.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:53:34.873 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:53:34.899 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:34.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:34.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:53:34.910 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:34 smithi049 ceph-mon[40489]: pgmap v218: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 147 active+clean; 1.7 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.13k op/s; 1940/82626 objects degraded (2.348%) 2024-09-21T13:53:34.910 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:34 smithi049 ceph-mon[40489]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:53:34.910 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:34 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T13:53:34.910 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:34 smithi049 ceph-mon[40489]: osdmap e167: 8 total, 8 up, 8 in 2024-09-21T13:53:34.911 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:34 smithi049 ceph-mon[30080]: pgmap v218: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 147 active+clean; 1.7 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.13k op/s; 1940/82626 objects degraded (2.348%) 2024-09-21T13:53:34.911 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:34 smithi049 ceph-mon[30080]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:53:34.911 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:34 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T13:53:34.911 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:34 smithi049 ceph-mon[30080]: osdmap e167: 8 total, 8 up, 8 in 2024-09-21T13:53:35.001 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:53:35.042 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:35.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:35.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:53:35.144 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:53:35.176 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:35.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:35.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:53:35.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:34 smithi187 ceph-mon[33032]: pgmap v218: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 147 active+clean; 1.7 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.13k op/s; 1940/82626 objects degraded (2.348%) 2024-09-21T13:53:35.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:34 smithi187 ceph-mon[33032]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:53:35.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:34 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T13:53:35.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:34 smithi187 ceph-mon[33032]: osdmap e167: 8 total, 8 up, 8 in 2024-09-21T13:53:35.278 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:53:35.306 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:35.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:35.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:53:35.408 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:35.433 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:35.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:35.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:35.534 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:53:35.556 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:35.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:35.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:53:35.658 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:53:35.696 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:35.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:35.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:53:35.798 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:35.824 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:35.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:35.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:35.926 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:35.952 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:35.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:35.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:36.054 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:36.079 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:36.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:36.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:36.181 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:53:36.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:35 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 1940/82626 objects degraded (2.348%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T13:53:36.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:35 smithi187 ceph-mon[33032]: osdmap e168: 8 total, 8 up, 8 in 2024-09-21T13:53:36.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:35 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/563179272' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:53:36.209 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:36.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:36.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:53:36.246 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:53:36.246 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:53:36.246 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:53:36.246 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:53:36.246 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:53:36.246 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:53:36.246 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:53:36.246 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:53:36.247 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:53:36.247 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:53:36.247 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:53:36.247 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:53:36.247 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:53:36.247 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:53:36.247 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:53:36.247 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:53:36.248 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:35 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 1940/82626 objects degraded (2.348%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T13:53:36.248 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:35 smithi049 ceph-mon[40489]: osdmap e168: 8 total, 8 up, 8 in 2024-09-21T13:53:36.248 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:35 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/563179272' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:53:36.248 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:35 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 1940/82626 objects degraded (2.348%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T13:53:36.248 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:35 smithi049 ceph-mon[30080]: osdmap e168: 8 total, 8 up, 8 in 2024-09-21T13:53:36.248 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:35 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/563179272' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:53:36.310 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:53:36.333 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:36.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:36.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:53:36.422 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T13:53:36.435 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:53:36.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:36.515 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:36.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:53:36.616 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:53:36.640 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:36.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:36.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:53:36.741 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:53:36.799 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:36.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:36.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:53:36.901 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:53:36.925 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:36.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:36.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:53:37.027 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:53:37.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:37.074 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:37.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:53:37.156 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:36 smithi049 ceph-mon[40489]: pgmap v221: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 147 active+clean; 1.6 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 846 op/s; 1608/78867 objects degraded (2.039%) 2024-09-21T13:53:37.156 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:36 smithi049 ceph-mon[30080]: pgmap v221: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 147 active+clean; 1.6 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 846 op/s; 1608/78867 objects degraded (2.039%) 2024-09-21T13:53:37.175 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:53:37.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:36 smithi187 ceph-mon[33032]: pgmap v221: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 147 active+clean; 1.6 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 846 op/s; 1608/78867 objects degraded (2.039%) 2024-09-21T13:53:37.224 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:37.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:37.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:53:37.289 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T13:53:37.313 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:53:37.326 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:53:37.380 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:53:37.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:37.400 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:37.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:53:37.501 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:53:37.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:37.581 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:37.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:53:37.682 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:53:37.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:37.742 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:37.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:53:37.843 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:53:37.869 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:37.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:37.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:53:37.971 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:53:38.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:38.018 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:38.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:53:38.119 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:53:38.144 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:38.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:38.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:53:38.245 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:53:38.271 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:38.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:38.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:53:38.276 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T13:53:38.277 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-09-21T13:53:38.277 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-09-21T13:53:38.277 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-09-21T13:53:38.277 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T13:53:38.373 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:53:38.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:38.410 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:38.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:53:38.511 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:53:38.553 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:38.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:38.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:53:38.655 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:53:38.695 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:38.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:38.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:53:38.797 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:53:38.839 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:38.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:38.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:53:38.894 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:53:38.896 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-09-21T13:53:38.941 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:53:38.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:38.980 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:38.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:53:39.081 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:53:39.121 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:39.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:39.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:53:39.178 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:38 smithi187 ceph-mon[33032]: pgmap v222: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 147 active+clean; 1.6 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 640 B/s rd, 0 B/s wr, 715 op/s; 1608/76950 objects degraded (2.090%) 2024-09-21T13:53:39.205 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T13:53:39.223 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:53:39.248 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:38 smithi049 ceph-mon[40489]: pgmap v222: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 147 active+clean; 1.6 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 640 B/s rd, 0 B/s wr, 715 op/s; 1608/76950 objects degraded (2.090%) 2024-09-21T13:53:39.248 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:38 smithi049 ceph-mon[30080]: pgmap v222: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 147 active+clean; 1.6 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 640 B/s rd, 0 B/s wr, 715 op/s; 1608/76950 objects degraded (2.090%) 2024-09-21T13:53:39.293 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:39.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:39.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:53:39.396 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:53:39.419 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:39.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:39.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:53:39.444 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:53:39 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:53:39.521 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:53:39.573 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:39.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:39.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:53:39.675 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:53:39.718 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:39.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:39.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:53:39.820 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:53:39.845 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:39.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:39.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:53:39.870 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T13:53:39.889 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:53:39.947 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:39.973 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:39.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:39.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:40.075 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:53:40.102 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:40.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:40.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:53:40.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:39 smithi187 ceph-mon[33032]: from='client.24934 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T13:53:40.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:39 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:53:40.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:39 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 1608/76950 objects degraded (2.090%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T13:53:40.204 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:53:40.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:39 smithi049 ceph-mon[30080]: from='client.24934 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T13:53:40.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:39 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:53:40.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:39 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 1608/76950 objects degraded (2.090%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T13:53:40.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:39 smithi049 ceph-mon[40489]: from='client.24934 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T13:53:40.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:39 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:53:40.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:39 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 1608/76950 objects degraded (2.090%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T13:53:40.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:40.262 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:40.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:53:40.363 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:53:40.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:40.411 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:40.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:53:40.433 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":222,"stamp":"2024-09-21T13:53:37.761456+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1668487731,"num_objects":25650,"num_object_clones":0,"num_object_copies":76950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1608,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25650,"num_whiteouts":0,"num_read":1115,"num_read_kb":972,"num_write":247934,"num_write_kb":8732362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21777,"num_bytes_recovered":1426785020,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87978,"ondisk_log_size":87978,"up":444,"acting":443,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6281056,"kb_used_data":5094496,"kb_used_omap":15,"kb_used_meta":1186032,"kb_avail":743680160,"statfs":{"total":767960285184,"available":761528483840,"internally_reserved":0,"allocated":5216763904,"data_stored":5213444861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15598,"internal_metadata":1214497554},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-374800384,"num_objects":-5719,"num_object_clones":0,"num_object_copies":-17157,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-332,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5719,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":5719,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.999394"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":175,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.765673+0000","last_change":"2024-09-21T13:52:26.677736+0000","last_active":"2024-09-21T13:53:29.765673+0000","last_peered":"2024-09-21T13:53:29.765673+0000","last_clean":"2024-09-21T13:53:29.765673+0000","last_became_active":"2024-09-21T13:52:04.317322+0000","last_became_peered":"2024-09-21T13:52:04.317322+0000","last_unstale":"2024-09-21T13:53:29.765673+0000","last_undegraded":"2024-09-21T13:53:29.765673+0000","last_fullsized":"2024-09-21T13:53:29.765673+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:26.677689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:52:26.677689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:45.636529+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"68'9","reported_seq":243,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.761345+0000","last_change":"2024-09-21T13:51:55.225649+0000","last_active":"2024-09-21T13:53:29.761345+0000","last_peered":"2024-09-21T13:53:29.761345+0000","last_clean":"2024-09-21T13:53:29.761345+0000","last_became_active":"2024-09-21T13:49:00.055210+0000","last_became_peered":"2024-09-21T13:49:00.055210+0000","last_unstale":"2024-09-21T13:53:29.761345+0000","last_undegraded":"2024-09-21T13:53:29.761345+0000","last_fullsized":"2024-09-21T13:53:29.761345+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:51:55.225579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:55.225579+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":223,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.769689+0000","last_change":"2024-09-21T13:50:28.058990+0000","last_active":"2024-09-21T13:53:29.769689+0000","last_peered":"2024-09-21T13:53:29.769689+0000","last_clean":"2024-09-21T13:53:29.769689+0000","last_became_active":"2024-09-21T13:49:02.060984+0000","last_became_peered":"2024-09-21T13:49:02.060984+0000","last_unstale":"2024-09-21T13:53:29.769689+0000","last_undegraded":"2024-09-21T13:53:29.769689+0000","last_fullsized":"2024-09-21T13:53:29.769689+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:28.058918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:28.058918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:39.434903+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":219,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.758166+0000","last_change":"2024-09-21T13:53:21.005444+0000","last_active":"2024-09-21T13:53:29.758166+0000","last_peered":"2024-09-21T13:53:29.758166+0000","last_clean":"2024-09-21T13:53:29.758166+0000","last_became_active":"2024-09-21T13:49:04.072460+0000","last_became_peered":"2024-09-21T13:49:04.072460+0000","last_unstale":"2024-09-21T13:53:29.758166+0000","last_undegraded":"2024-09-21T13:53:29.758166+0000","last_fullsized":"2024-09-21T13:53:29.758166+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:21.005352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:21.005352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:43.773187+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":221,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T13:53:33.819030+0000","last_change":"2024-09-21T13:51:18.572788+0000","last_active":"2024-09-21T13:53:33.819030+0000","last_peered":"2024-09-21T13:53:33.819030+0000","last_clean":"2024-09-21T13:53:33.819030+0000","last_became_active":"2024-09-21T13:49:04.433965+0000","last_became_peered":"2024-09-21T13:49:04.433965+0000","last_unstale":"2024-09-21T13:53:33.819030+0000","last_undegraded":"2024-09-21T13:53:33.819030+0000","last_fullsized":"2024-09-21T13:53:33.819030+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:18.572729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:51:18.572729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":245,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T13:53:33.820043+0000","last_change":"2024-09-21T13:53:19.910708+0000","last_active":"2024-09-21T13:53:33.820043+0000","last_peered":"2024-09-21T13:53:33.820043+0000","last_clean":"2024-09-21T13:53:33.820043+0000","last_became_active":"2024-09-21T13:52:04.316819+0000","last_became_peered":"2024-09-21T13:52:04.316819+0000","last_unstale":"2024-09-21T13:53:33.820043+0000","last_undegraded":"2024-09-21T13:53:33.820043+0000","last_fullsized":"2024-09-21T13:53:33.820043+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:19.910638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:53:19.910638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:27.797531+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"68'15","reported_seq":257,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T13:53:33.824806+0000","last_change":"2024-09-21T13:52:04.315645+0000","last_active":"2024-09-21T13:53:33.824806+0000","last_peered":"2024-09-21T13:53:33.824806+0000","last_clean":"2024-09-21T13:53:33.824806+0000","last_became_active":"2024-09-21T13:52:04.315101+0000","last_became_peered":"2024-09-21T13:52:04.315101+0000","last_unstale":"2024-09-21T13:53:33.824806+0000","last_undegraded":"2024-09-21T13:53:33.824806+0000","last_fullsized":"2024-09-21T13:53:33.824806+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:15.792034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":231,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.758774+0000","last_change":"2024-09-21T13:51:43.040397+0000","last_active":"2024-09-21T13:53:29.758774+0000","last_peered":"2024-09-21T13:53:29.758774+0000","last_clean":"2024-09-21T13:53:29.758774+0000","last_became_active":"2024-09-21T13:49:02.063687+0000","last_became_peered":"2024-09-21T13:49:02.063687+0000","last_unstale":"2024-09-21T13:53:29.758774+0000","last_undegraded":"2024-09-21T13:53:29.758774+0000","last_fullsized":"2024-09-21T13:53:29.758774+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:43.040348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:43.040348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":239,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.765974+0000","last_change":"2024-09-21T13:52:49.451929+0000","last_active":"2024-09-21T13:53:29.765974+0000","last_peered":"2024-09-21T13:53:29.765974+0000","last_clean":"2024-09-21T13:53:29.765974+0000","last_became_active":"2024-09-21T13:48:58.050695+0000","last_became_peered":"2024-09-21T13:48:58.050695+0000","last_unstale":"2024-09-21T13:53:29.765974+0000","last_undegraded":"2024-09-21T13:53:29.765974+0000","last_fullsized":"2024-09-21T13:53:29.765974+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:49.451882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:52:49.451882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:54.087443+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"68'9","reported_seq":243,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.758476+0000","last_change":"2024-09-21T13:51:21.951391+0000","last_active":"2024-09-21T13:53:29.758476+0000","last_peered":"2024-09-21T13:53:29.758476+0000","last_clean":"2024-09-21T13:53:29.758476+0000","last_became_active":"2024-09-21T13:49:00.059382+0000","last_became_peered":"2024-09-21T13:49:00.059382+0000","last_unstale":"2024-09-21T13:53:29.758476+0000","last_undegraded":"2024-09-21T13:53:29.758476+0000","last_fullsized":"2024-09-21T13:53:29.758476+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:51:21.951305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:21.951305+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":215,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.764722+0000","last_change":"2024-09-21T13:49:02.066978+0000","last_active":"2024-09-21T13:53:29.764722+0000","last_peered":"2024-09-21T13:53:29.764722+0000","last_clean":"2024-09-21T13:53:29.764722+0000","last_became_active":"2024-09-21T13:49:02.066693+0000","last_became_peered":"2024-09-21T13:49:02.066693+0000","last_unstale":"2024-09-21T13:53:29.764722+0000","last_undegraded":"2024-09-21T13:53:29.764722+0000","last_fullsized":"2024-09-21T13:53:29.764722+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:09.312032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":172,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.768552+0000","last_change":"2024-09-21T13:52:28.417865+0000","last_active":"2024-09-21T13:53:29.768552+0000","last_peered":"2024-09-21T13:53:29.768552+0000","last_clean":"2024-09-21T13:53:29.768552+0000","last_became_active":"2024-09-21T13:50:29.689381+0000","last_became_peered":"2024-09-21T13:50:29.689381+0000","last_unstale":"2024-09-21T13:53:29.768552+0000","last_undegraded":"2024-09-21T13:53:29.768552+0000","last_fullsized":"2024-09-21T13:53:29.768552+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:28.417690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:28.417690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:33.915909+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":231,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.764979+0000","last_change":"2024-09-21T13:50:39.480995+0000","last_active":"2024-09-21T13:53:29.764979+0000","last_peered":"2024-09-21T13:53:29.764979+0000","last_clean":"2024-09-21T13:53:29.764979+0000","last_became_active":"2024-09-21T13:48:58.050906+0000","last_became_peered":"2024-09-21T13:48:58.050906+0000","last_unstale":"2024-09-21T13:53:29.764979+0000","last_undegraded":"2024-09-21T13:53:29.764979+0000","last_fullsized":"2024-09-21T13:53:29.764979+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:39.480930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:39.480930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:32.611461+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"68'5","reported_seq":253,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.759982+0000","last_change":"2024-09-21T13:52:04.319297+0000","last_active":"2024-09-21T13:53:29.759982+0000","last_peered":"2024-09-21T13:53:29.759982+0000","last_clean":"2024-09-21T13:53:29.759982+0000","last_became_active":"2024-09-21T13:52:04.318515+0000","last_became_peered":"2024-09-21T13:52:04.318515+0000","last_unstale":"2024-09-21T13:53:29.759982+0000","last_undegraded":"2024-09-21T13:53:29.759982+0000","last_fullsized":"2024-09-21T13:53:29.759982+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:02.067591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":223,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.766377+0000","last_change":"2024-09-21T13:50:40.188797+0000","last_active":"2024-09-21T13:53:29.766377+0000","last_peered":"2024-09-21T13:53:29.766377+0000","last_clean":"2024-09-21T13:53:29.766377+0000","last_became_active":"2024-09-21T13:49:02.060212+0000","last_became_peered":"2024-09-21T13:49:02.060212+0000","last_unstale":"2024-09-21T13:53:29.766377+0000","last_undegraded":"2024-09-21T13:53:29.766377+0000","last_fullsized":"2024-09-21T13:53:29.766377+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:40.188712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:40.188712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":227,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.764492+0000","last_change":"2024-09-21T13:52:43.516806+0000","last_active":"2024-09-21T13:53:29.764492+0000","last_peered":"2024-09-21T13:53:29.764492+0000","last_clean":"2024-09-21T13:53:29.764492+0000","last_became_active":"2024-09-21T13:49:04.078026+0000","last_became_peered":"2024-09-21T13:49:04.078026+0000","last_unstale":"2024-09-21T13:53:29.764492+0000","last_undegraded":"2024-09-21T13:53:29.764492+0000","last_fullsized":"2024-09-21T13:53:29.764492+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:43.516741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:43.516741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:08.753512+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":220,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.760316+0000","last_change":"2024-09-21T13:50:38.027288+0000","last_active":"2024-09-21T13:53:29.760316+0000","last_peered":"2024-09-21T13:53:29.760316+0000","last_clean":"2024-09-21T13:53:29.760316+0000","last_became_active":"2024-09-21T13:49:04.431422+0000","last_became_peered":"2024-09-21T13:49:04.431422+0000","last_unstale":"2024-09-21T13:53:29.760316+0000","last_undegraded":"2024-09-21T13:53:29.760316+0000","last_fullsized":"2024-09-21T13:53:29.760316+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:38.027227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:38.027227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":156,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.760138+0000","last_change":"2024-09-21T13:50:29.692621+0000","last_active":"2024-09-21T13:53:29.760138+0000","last_peered":"2024-09-21T13:53:29.760138+0000","last_clean":"2024-09-21T13:53:29.760138+0000","last_became_active":"2024-09-21T13:50:29.688282+0000","last_became_peered":"2024-09-21T13:50:29.688282+0000","last_unstale":"2024-09-21T13:53:29.760138+0000","last_undegraded":"2024-09-21T13:53:29.760138+0000","last_fullsized":"2024-09-21T13:53:29.760138+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:04.102443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"68'15","reported_seq":196,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T13:53:34.917936+0000","last_change":"2024-09-21T13:53:32.929362+0000","last_active":"2024-09-21T13:53:34.917936+0000","last_peered":"2024-09-21T13:53:34.917936+0000","last_clean":"2024-09-21T13:53:34.917936+0000","last_became_active":"2024-09-21T13:52:04.324273+0000","last_became_peered":"2024-09-21T13:52:04.324273+0000","last_unstale":"2024-09-21T13:53:34.917936+0000","last_undegraded":"2024-09-21T13:53:34.917936+0000","last_fullsized":"2024-09-21T13:53:34.917936+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:53:32.929017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:32.929017+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:37.476556+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":231,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.773825+0000","last_change":"2024-09-21T13:52:26.372217+0000","last_active":"2024-09-21T13:53:29.773825+0000","last_peered":"2024-09-21T13:53:29.773825+0000","last_clean":"2024-09-21T13:53:29.773825+0000","last_became_active":"2024-09-21T13:49:02.066796+0000","last_became_peered":"2024-09-21T13:49:02.066796+0000","last_unstale":"2024-09-21T13:53:29.773825+0000","last_undegraded":"2024-09-21T13:53:29.773825+0000","last_fullsized":"2024-09-21T13:53:29.773825+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:26.372144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:52:26.372144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:40.709431+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":214,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T13:53:33.823008+0000","last_change":"2024-09-21T13:49:04.434429+0000","last_active":"2024-09-21T13:53:33.823008+0000","last_peered":"2024-09-21T13:53:33.823008+0000","last_clean":"2024-09-21T13:53:33.823008+0000","last_became_active":"2024-09-21T13:49:04.434049+0000","last_became_peered":"2024-09-21T13:49:04.434049+0000","last_unstale":"2024-09-21T13:53:33.823008+0000","last_undegraded":"2024-09-21T13:53:33.823008+0000","last_fullsized":"2024-09-21T13:53:33.823008+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":242,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.758589+0000","last_change":"2024-09-21T13:52:52.142682+0000","last_active":"2024-09-21T13:53:29.758589+0000","last_peered":"2024-09-21T13:53:29.758589+0000","last_clean":"2024-09-21T13:53:29.758589+0000","last_became_active":"2024-09-21T13:52:04.316016+0000","last_became_peered":"2024-09-21T13:52:04.316016+0000","last_unstale":"2024-09-21T13:53:29.758589+0000","last_undegraded":"2024-09-21T13:53:29.758589+0000","last_fullsized":"2024-09-21T13:53:29.758589+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:52.142638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:52:52.142638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:13.595892+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"68'12","reported_seq":252,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T13:53:33.818961+0000","last_change":"2024-09-21T13:51:52.819966+0000","last_active":"2024-09-21T13:53:33.818961+0000","last_peered":"2024-09-21T13:53:33.818961+0000","last_clean":"2024-09-21T13:53:33.818961+0000","last_became_active":"2024-09-21T13:49:00.061867+0000","last_became_peered":"2024-09-21T13:49:00.061867+0000","last_unstale":"2024-09-21T13:53:33.818961+0000","last_undegraded":"2024-09-21T13:53:33.818961+0000","last_fullsized":"2024-09-21T13:53:33.818961+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'12","last_scrub_stamp":"2024-09-21T13:51:52.819879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:52.819879+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":226,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.760718+0000","last_change":"2024-09-21T13:52:04.321297+0000","last_active":"2024-09-21T13:53:29.760718+0000","last_peered":"2024-09-21T13:53:29.760718+0000","last_clean":"2024-09-21T13:53:29.760718+0000","last_became_active":"2024-09-21T13:52:04.320800+0000","last_became_peered":"2024-09-21T13:52:04.320800+0000","last_unstale":"2024-09-21T13:53:29.760718+0000","last_undegraded":"2024-09-21T13:53:29.760718+0000","last_fullsized":"2024-09-21T13:53:29.760718+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:14.081205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":231,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.766643+0000","last_change":"2024-09-21T13:52:54.580070+0000","last_active":"2024-09-21T13:53:29.766643+0000","last_peered":"2024-09-21T13:53:29.766643+0000","last_clean":"2024-09-21T13:53:29.766643+0000","last_became_active":"2024-09-21T13:52:04.321358+0000","last_became_peered":"2024-09-21T13:52:04.321358+0000","last_unstale":"2024-09-21T13:53:29.766643+0000","last_undegraded":"2024-09-21T13:53:29.766643+0000","last_fullsized":"2024-09-21T13:53:29.766643+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T13:52:54.580022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:54.580022+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:21.087079+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":240,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.769840+0000","last_change":"2024-09-21T13:52:43.516442+0000","last_active":"2024-09-21T13:53:29.769840+0000","last_peered":"2024-09-21T13:53:29.769840+0000","last_clean":"2024-09-21T13:53:29.769840+0000","last_became_active":"2024-09-21T13:48:58.048972+0000","last_became_peered":"2024-09-21T13:48:58.048972+0000","last_unstale":"2024-09-21T13:53:29.769840+0000","last_undegraded":"2024-09-21T13:53:29.769840+0000","last_fullsized":"2024-09-21T13:53:29.769840+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:52:43.516221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:52:43.516221+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:48.258159+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"68'10","reported_seq":199,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.767530+0000","last_change":"2024-09-21T13:52:56.526032+0000","last_active":"2024-09-21T13:53:29.767530+0000","last_peered":"2024-09-21T13:53:29.767530+0000","last_clean":"2024-09-21T13:53:29.767530+0000","last_became_active":"2024-09-21T13:50:29.694465+0000","last_became_peered":"2024-09-21T13:50:29.694465+0000","last_unstale":"2024-09-21T13:53:29.767530+0000","last_undegraded":"2024-09-21T13:53:29.767530+0000","last_fullsized":"2024-09-21T13:53:29.767530+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:52:56.525966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:52:56.525966+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:10.063932+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"68'8","reported_seq":299,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.762743+0000","last_change":"2024-09-21T13:51:42.174696+0000","last_active":"2024-09-21T13:53:29.762743+0000","last_peered":"2024-09-21T13:53:29.762743+0000","last_clean":"2024-09-21T13:53:29.762743+0000","last_became_active":"2024-09-21T13:49:02.059638+0000","last_became_peered":"2024-09-21T13:49:02.059638+0000","last_unstale":"2024-09-21T13:53:29.762743+0000","last_undegraded":"2024-09-21T13:53:29.762743+0000","last_fullsized":"2024-09-21T13:53:29.762743+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:51:42.174629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:42.174629+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:08.238915+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"166'15520","reported_seq":15822,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:33.308053+0000","last_change":"2024-09-21T13:52:51.407976+0000","last_active":"2024-09-21T13:53:33.308053+0000","last_peered":"2024-09-21T13:53:33.308053+0000","last_clean":"2024-09-21T13:53:33.308053+0000","last_became_active":"2024-09-21T13:50:07.422961+0000","last_became_peered":"2024-09-21T13:50:07.422961+0000","last_unstale":"2024-09-21T13:53:33.308053+0000","last_undegraded":"2024-09-21T13:53:33.308053+0000","last_fullsized":"2024-09-21T13:53:33.308053+0000","mapping_epoch":83,"log_start":"124'9200","ondisk_log_start":"124'9200","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'13038","last_scrub_stamp":"2024-09-21T13:52:51.407818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:52:51.407818+0000","objects_scrubbed":4170,"log_size":6320,"log_dups_size":0,"ondisk_log_size":6320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:10.866985+0000","scrub_duration":960,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109314048,"num_objects":1668,"num_object_clones":0,"num_object_copies":5004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15520,"num_write_kb":550016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"68'15","reported_seq":260,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.767027+0000","last_change":"2024-09-21T13:52:16.452049+0000","last_active":"2024-09-21T13:53:29.767027+0000","last_peered":"2024-09-21T13:53:29.767027+0000","last_clean":"2024-09-21T13:53:29.767027+0000","last_became_active":"2024-09-21T13:49:00.060324+0000","last_became_peered":"2024-09-21T13:49:00.060324+0000","last_unstale":"2024-09-21T13:53:29.767027+0000","last_undegraded":"2024-09-21T13:53:29.767027+0000","last_fullsized":"2024-09-21T13:53:29.767027+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:52:16.451964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:52:16.451964+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:31.981849+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":234,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T13:53:33.821702+0000","last_change":"2024-09-21T13:50:06.151943+0000","last_active":"2024-09-21T13:53:33.821702+0000","last_peered":"2024-09-21T13:53:33.821702+0000","last_clean":"2024-09-21T13:53:33.821702+0000","last_became_active":"2024-09-21T13:48:58.051911+0000","last_became_peered":"2024-09-21T13:48:58.051911+0000","last_unstale":"2024-09-21T13:53:33.821702+0000","last_undegraded":"2024-09-21T13:53:33.821702+0000","last_fullsized":"2024-09-21T13:53:33.821702+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.151888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.151888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"68'8","reported_seq":291,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:33.169697+0000","last_change":"2024-09-21T13:53:19.021881+0000","last_active":"2024-09-21T13:53:33.169697+0000","last_peered":"2024-09-21T13:53:33.169697+0000","last_clean":"2024-09-21T13:53:33.169697+0000","last_became_active":"2024-09-21T13:49:02.061679+0000","last_became_peered":"2024-09-21T13:49:02.061679+0000","last_unstale":"2024-09-21T13:53:33.169697+0000","last_undegraded":"2024-09-21T13:53:33.169697+0000","last_fullsized":"2024-09-21T13:53:33.169697+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:53:19.021810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:53:19.021810+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:35.961578+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":219,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.764979+0000","last_change":"2024-09-21T13:50:34.481503+0000","last_active":"2024-09-21T13:53:29.764979+0000","last_peered":"2024-09-21T13:53:29.764979+0000","last_clean":"2024-09-21T13:53:29.764979+0000","last_became_active":"2024-09-21T13:49:04.081905+0000","last_became_peered":"2024-09-21T13:49:04.081905+0000","last_unstale":"2024-09-21T13:53:29.764979+0000","last_undegraded":"2024-09-21T13:53:29.764979+0000","last_fullsized":"2024-09-21T13:53:29.764979+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:34.481450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:34.481450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:42.543466+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"167'15654","reported_seq":23256,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T13:53:33.894677+0000","last_change":"2024-09-21T13:53:16.667415+0000","last_active":"2024-09-21T13:53:33.894677+0000","last_peered":"2024-09-21T13:53:33.894677+0000","last_clean":"2024-09-21T13:53:33.894677+0000","last_became_active":"2024-09-21T13:53:16.646871+0000","last_became_peered":"2024-09-21T13:53:16.646871+0000","last_unstale":"2024-09-21T13:53:33.894677+0000","last_undegraded":"2024-09-21T13:53:33.894677+0000","last_fullsized":"2024-09-21T13:53:33.894677+0000","mapping_epoch":159,"log_start":"134'11000","ondisk_log_start":"134'11000","created":83,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":4654,"log_dups_size":0,"ondisk_log_size":4654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103809024,"num_objects":1584,"num_object_clones":0,"num_object_copies":4752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15654,"num_write_kb":551616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3703,"num_bytes_recovered":242679808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"68'18","reported_seq":262,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T13:53:33.821421+0000","last_change":"2024-09-21T13:50:27.281455+0000","last_active":"2024-09-21T13:53:33.821421+0000","last_peered":"2024-09-21T13:53:33.821421+0000","last_clean":"2024-09-21T13:53:33.821421+0000","last_became_active":"2024-09-21T13:49:00.429314+0000","last_became_peered":"2024-09-21T13:49:00.429314+0000","last_unstale":"2024-09-21T13:53:33.821421+0000","last_undegraded":"2024-09-21T13:53:33.821421+0000","last_fullsized":"2024-09-21T13:53:33.821421+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'18","last_scrub_stamp":"2024-09-21T13:50:27.281356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:27.281356+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":226,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T13:53:33.823334+0000","last_change":"2024-09-21T13:48:58.056669+0000","last_active":"2024-09-21T13:53:33.823334+0000","last_peered":"2024-09-21T13:53:33.823334+0000","last_clean":"2024-09-21T13:53:33.823334+0000","last_became_active":"2024-09-21T13:48:58.056375+0000","last_became_peered":"2024-09-21T13:48:58.056375+0000","last_unstale":"2024-09-21T13:53:33.823334+0000","last_undegraded":"2024-09-21T13:53:33.823334+0000","last_fullsized":"2024-09-21T13:53:33.823334+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":224,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.759250+0000","last_change":"2024-09-21T13:51:23.940809+0000","last_active":"2024-09-21T13:53:29.759250+0000","last_peered":"2024-09-21T13:53:29.759250+0000","last_clean":"2024-09-21T13:53:29.759250+0000","last_became_active":"2024-09-21T13:49:02.062955+0000","last_became_peered":"2024-09-21T13:49:02.062955+0000","last_unstale":"2024-09-21T13:53:29.759250+0000","last_undegraded":"2024-09-21T13:53:29.759250+0000","last_fullsized":"2024-09-21T13:53:29.759250+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:23.940746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:23.940746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":223,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.758404+0000","last_change":"2024-09-21T13:52:04.316586+0000","last_active":"2024-09-21T13:53:29.758404+0000","last_peered":"2024-09-21T13:53:29.758404+0000","last_clean":"2024-09-21T13:53:29.758404+0000","last_became_active":"2024-09-21T13:52:04.316121+0000","last_became_peered":"2024-09-21T13:52:04.316121+0000","last_unstale":"2024-09-21T13:53:29.758404+0000","last_undegraded":"2024-09-21T13:53:29.758404+0000","last_fullsized":"2024-09-21T13:53:29.758404+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:10.130707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"167'15232","reported_seq":15434,"reported_epoch":167,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2024-09-21T13:53:33.938431+0000","last_change":"2024-09-21T13:53:05.053228+0000","last_active":"2024-09-21T13:53:33.938431+0000","last_peered":"2024-09-21T13:53:33.938431+0000","last_clean":"2024-09-21T13:50:42.909755+0000","last_became_active":"2024-09-21T13:52:04.998636+0000","last_became_peered":"2024-09-21T13:52:04.998636+0000","last_unstale":"2024-09-21T13:53:33.938431+0000","last_undegraded":"2024-09-21T13:52:04.888197+0000","last_fullsized":"2024-09-21T13:52:04.863335+0000","mapping_epoch":131,"log_start":"134'10600","ondisk_log_start":"134'10600","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":4632,"log_dups_size":0,"ondisk_log_size":4632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105381888,"num_objects":1608,"num_object_clones":0,"num_object_copies":4824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1608,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15232,"num_write_kb":538880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":1608}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"68'14","reported_seq":249,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.759319+0000","last_change":"2024-09-21T13:50:29.009786+0000","last_active":"2024-09-21T13:53:29.759319+0000","last_peered":"2024-09-21T13:53:29.759319+0000","last_clean":"2024-09-21T13:53:29.759319+0000","last_became_active":"2024-09-21T13:49:00.055031+0000","last_became_peered":"2024-09-21T13:49:00.055031+0000","last_unstale":"2024-09-21T13:53:29.759319+0000","last_undegraded":"2024-09-21T13:53:29.759319+0000","last_fullsized":"2024-09-21T13:53:29.759319+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'14","last_scrub_stamp":"2024-09-21T13:50:29.009720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:29.009720+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":233,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.784412+0000","last_change":"2024-09-21T13:49:59.253636+0000","last_active":"2024-09-21T13:53:29.784412+0000","last_peered":"2024-09-21T13:53:29.784412+0000","last_clean":"2024-09-21T13:53:29.784412+0000","last_became_active":"2024-09-21T13:48:58.053533+0000","last_became_peered":"2024-09-21T13:48:58.053533+0000","last_unstale":"2024-09-21T13:53:29.784412+0000","last_undegraded":"2024-09-21T13:53:29.784412+0000","last_fullsized":"2024-09-21T13:53:29.784412+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:49:59.253587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:49:59.253587+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:34.697714+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"68'8","reported_seq":290,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T13:53:33.820633+0000","last_change":"2024-09-21T13:50:41.458637+0000","last_active":"2024-09-21T13:53:33.820633+0000","last_peered":"2024-09-21T13:53:33.820633+0000","last_clean":"2024-09-21T13:53:33.820633+0000","last_became_active":"2024-09-21T13:49:02.070064+0000","last_became_peered":"2024-09-21T13:49:02.070064+0000","last_unstale":"2024-09-21T13:53:33.820633+0000","last_undegraded":"2024-09-21T13:53:33.820633+0000","last_fullsized":"2024-09-21T13:53:33.820633+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:50:41.458562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:41.458562+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":231,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.766718+0000","last_change":"2024-09-21T13:52:55.541288+0000","last_active":"2024-09-21T13:53:29.766718+0000","last_peered":"2024-09-21T13:53:29.766718+0000","last_clean":"2024-09-21T13:53:29.766718+0000","last_became_active":"2024-09-21T13:52:04.322185+0000","last_became_peered":"2024-09-21T13:52:04.322185+0000","last_unstale":"2024-09-21T13:53:29.766718+0000","last_undegraded":"2024-09-21T13:53:29.766718+0000","last_fullsized":"2024-09-21T13:53:29.766718+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:55.541232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:55.541232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:57.661213+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"166'15227","reported_seq":15405,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:33.320527+0000","last_change":"2024-09-21T13:50:07.419221+0000","last_active":"2024-09-21T13:53:33.320527+0000","last_peered":"2024-09-21T13:53:33.320527+0000","last_clean":"2024-09-21T13:53:33.320527+0000","last_became_active":"2024-09-21T13:50:07.418929+0000","last_became_peered":"2024-09-21T13:50:07.418929+0000","last_unstale":"2024-09-21T13:53:33.320527+0000","last_undegraded":"2024-09-21T13:53:33.320527+0000","last_fullsized":"2024-09-21T13:53:33.320527+0000","mapping_epoch":83,"log_start":"132'10400","ondisk_log_start":"132'10400","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":4827,"log_dups_size":0,"ondisk_log_size":4827,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108593152,"num_objects":1657,"num_object_clones":0,"num_object_copies":4971,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15227,"num_write_kb":540288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"68'10","reported_seq":251,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.767749+0000","last_change":"2024-09-21T13:51:50.211365+0000","last_active":"2024-09-21T13:53:29.767749+0000","last_peered":"2024-09-21T13:53:29.767749+0000","last_clean":"2024-09-21T13:53:29.767749+0000","last_became_active":"2024-09-21T13:49:00.054564+0000","last_became_peered":"2024-09-21T13:49:00.054564+0000","last_unstale":"2024-09-21T13:53:29.767749+0000","last_undegraded":"2024-09-21T13:53:29.767749+0000","last_fullsized":"2024-09-21T13:53:29.767749+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:51:50.211279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:50.211279+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:15.800803+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"61'1","reported_seq":265,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:30.179679+0000","last_change":"2024-09-21T13:53:30.179679+0000","last_active":"2024-09-21T13:53:30.179679+0000","last_peered":"2024-09-21T13:53:30.179679+0000","last_clean":"2024-09-21T13:53:30.179679+0000","last_became_active":"2024-09-21T13:52:04.315788+0000","last_became_peered":"2024-09-21T13:52:04.315788+0000","last_unstale":"2024-09-21T13:53:30.179679+0000","last_undegraded":"2024-09-21T13:53:30.179679+0000","last_fullsized":"2024-09-21T13:53:30.179679+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:53:30.179605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:53:30.179605+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:43.052251+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"96'9","reported_seq":235,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.762422+0000","last_change":"2024-09-21T13:52:51.443040+0000","last_active":"2024-09-21T13:53:29.762422+0000","last_peered":"2024-09-21T13:53:29.762422+0000","last_clean":"2024-09-21T13:53:29.762422+0000","last_became_active":"2024-09-21T13:50:29.693557+0000","last_became_peered":"2024-09-21T13:50:29.693557+0000","last_unstale":"2024-09-21T13:53:29.762422+0000","last_undegraded":"2024-09-21T13:53:29.762422+0000","last_fullsized":"2024-09-21T13:53:29.762422+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'9","last_scrub_stamp":"2024-09-21T13:52:51.442930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:52:51.442930+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:04.534122+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":169,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T13:53:33.820081+0000","last_change":"2024-09-21T13:52:04.319323+0000","last_active":"2024-09-21T13:53:33.820081+0000","last_peered":"2024-09-21T13:53:33.820081+0000","last_clean":"2024-09-21T13:53:33.820081+0000","last_became_active":"2024-09-21T13:52:04.319034+0000","last_became_peered":"2024-09-21T13:52:04.319034+0000","last_unstale":"2024-09-21T13:53:33.820081+0000","last_undegraded":"2024-09-21T13:53:33.820081+0000","last_fullsized":"2024-09-21T13:53:33.820081+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"166'15451","reported_seq":15700,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:33.347573+0000","last_change":"2024-09-21T13:52:16.618637+0000","last_active":"2024-09-21T13:53:33.347573+0000","last_peered":"2024-09-21T13:53:33.347573+0000","last_clean":"2024-09-21T13:53:33.347573+0000","last_became_active":"2024-09-21T13:50:07.423702+0000","last_became_peered":"2024-09-21T13:50:07.423702+0000","last_unstale":"2024-09-21T13:53:33.347573+0000","last_undegraded":"2024-09-21T13:53:33.347573+0000","last_fullsized":"2024-09-21T13:53:33.347573+0000","mapping_epoch":83,"log_start":"126'9600","ondisk_log_start":"126'9600","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'10945","last_scrub_stamp":"2024-09-21T13:52:16.618486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:52:16.618486+0000","objects_scrubbed":6221,"log_size":5851,"log_dups_size":0,"ondisk_log_size":5851,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:43.545327+0000","scrub_duration":1967,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108593152,"num_objects":1657,"num_object_clones":0,"num_object_copies":4971,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15451,"num_write_kb":547456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"68'19","reported_seq":220,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.764801+0000","last_change":"2024-09-21T13:50:32.514267+0000","last_active":"2024-09-21T13:53:29.764801+0000","last_peered":"2024-09-21T13:53:29.764801+0000","last_clean":"2024-09-21T13:53:29.764801+0000","last_became_active":"2024-09-21T13:50:29.690185+0000","last_became_peered":"2024-09-21T13:50:29.690185+0000","last_unstale":"2024-09-21T13:53:29.764801+0000","last_undegraded":"2024-09-21T13:53:29.764801+0000","last_fullsized":"2024-09-21T13:53:29.764801+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-21T13:50:32.514210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:32.514210+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:34.599292+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":235,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.772049+0000","last_change":"2024-09-21T13:52:04.322289+0000","last_active":"2024-09-21T13:53:29.772049+0000","last_peered":"2024-09-21T13:53:29.772049+0000","last_clean":"2024-09-21T13:53:29.772049+0000","last_became_active":"2024-09-21T13:52:04.321656+0000","last_became_peered":"2024-09-21T13:52:04.321656+0000","last_unstale":"2024-09-21T13:53:29.772049+0000","last_undegraded":"2024-09-21T13:53:29.772049+0000","last_fullsized":"2024-09-21T13:53:29.772049+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:08.762218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":232,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:32.295930+0000","last_change":"2024-09-21T13:53:32.295930+0000","last_active":"2024-09-21T13:53:32.295930+0000","last_peered":"2024-09-21T13:53:32.295930+0000","last_clean":"2024-09-21T13:53:32.295930+0000","last_became_active":"2024-09-21T13:49:02.061187+0000","last_became_peered":"2024-09-21T13:49:02.061187+0000","last_unstale":"2024-09-21T13:53:32.295930+0000","last_undegraded":"2024-09-21T13:53:32.295930+0000","last_fullsized":"2024-09-21T13:53:32.295930+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:32.295869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:53:32.295869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:32.443805+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":223,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.769595+0000","last_change":"2024-09-21T13:52:04.321133+0000","last_active":"2024-09-21T13:53:29.769595+0000","last_peered":"2024-09-21T13:53:29.769595+0000","last_clean":"2024-09-21T13:53:29.769595+0000","last_became_active":"2024-09-21T13:52:04.320826+0000","last_became_peered":"2024-09-21T13:52:04.320826+0000","last_unstale":"2024-09-21T13:53:29.769595+0000","last_undegraded":"2024-09-21T13:53:29.769595+0000","last_fullsized":"2024-09-21T13:53:29.769595+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:09.756023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"166'15544","reported_seq":15772,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:33.384217+0000","last_change":"2024-09-21T13:52:49.375801+0000","last_active":"2024-09-21T13:53:33.384217+0000","last_peered":"2024-09-21T13:53:33.384217+0000","last_clean":"2024-09-21T13:53:33.384217+0000","last_became_active":"2024-09-21T13:50:07.421666+0000","last_became_peered":"2024-09-21T13:50:07.421666+0000","last_unstale":"2024-09-21T13:53:33.384217+0000","last_undegraded":"2024-09-21T13:53:33.384217+0000","last_fullsized":"2024-09-21T13:53:33.384217+0000","mapping_epoch":83,"log_start":"124'9200","ondisk_log_start":"124'9200","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"149'12949","last_scrub_stamp":"2024-09-21T13:52:49.375671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:52:49.375671+0000","objects_scrubbed":4241,"log_size":6344,"log_dups_size":0,"ondisk_log_size":6344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:54.416521+0000","scrub_duration":1855,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105906176,"num_objects":1616,"num_object_clones":0,"num_object_copies":4848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15544,"num_write_kb":549120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"132'30","reported_seq":337,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.771962+0000","last_change":"2024-09-21T13:52:04.322738+0000","last_active":"2024-09-21T13:53:29.771962+0000","last_peered":"2024-09-21T13:53:29.771962+0000","last_clean":"2024-09-21T13:53:29.771962+0000","last_became_active":"2024-09-21T13:52:04.319431+0000","last_became_peered":"2024-09-21T13:52:04.319431+0000","last_unstale":"2024-09-21T13:53:29.771962+0000","last_undegraded":"2024-09-21T13:53:29.771962+0000","last_fullsized":"2024-09-21T13:53:29.771962+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:12.782459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":224,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.760218+0000","last_change":"2024-09-21T13:48:58.055640+0000","last_active":"2024-09-21T13:53:29.760218+0000","last_peered":"2024-09-21T13:53:29.760218+0000","last_clean":"2024-09-21T13:53:29.760218+0000","last_became_active":"2024-09-21T13:48:58.055366+0000","last_became_peered":"2024-09-21T13:48:58.055366+0000","last_unstale":"2024-09-21T13:53:29.760218+0000","last_undegraded":"2024-09-21T13:53:29.760218+0000","last_fullsized":"2024-09-21T13:53:29.760218+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:09.147679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":250,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.764713+0000","last_change":"2024-09-21T13:52:51.445882+0000","last_active":"2024-09-21T13:53:29.764713+0000","last_peered":"2024-09-21T13:53:29.764713+0000","last_clean":"2024-09-21T13:53:29.764713+0000","last_became_active":"2024-09-21T13:48:57.046246+0000","last_became_peered":"2024-09-21T13:48:57.046246+0000","last_unstale":"2024-09-21T13:53:29.764713+0000","last_undegraded":"2024-09-21T13:53:29.764713+0000","last_fullsized":"2024-09-21T13:53:29.764713+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-21T13:52:51.445812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:52:51.445812+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:18.734189+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.0015946669999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":164,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.773429+0000","last_change":"2024-09-21T13:51:24.187154+0000","last_active":"2024-09-21T13:53:29.773429+0000","last_peered":"2024-09-21T13:53:29.773429+0000","last_clean":"2024-09-21T13:53:29.773429+0000","last_became_active":"2024-09-21T13:50:29.688249+0000","last_became_peered":"2024-09-21T13:50:29.688249+0000","last_unstale":"2024-09-21T13:53:29.773429+0000","last_undegraded":"2024-09-21T13:53:29.773429+0000","last_fullsized":"2024-09-21T13:53:29.773429+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:24.186934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:24.186934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:33.380668+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":222,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T13:53:33.819758+0000","last_change":"2024-09-21T13:51:23.597164+0000","last_active":"2024-09-21T13:53:33.819758+0000","last_peered":"2024-09-21T13:53:33.819758+0000","last_clean":"2024-09-21T13:53:33.819758+0000","last_became_active":"2024-09-21T13:49:04.080466+0000","last_became_peered":"2024-09-21T13:49:04.080466+0000","last_unstale":"2024-09-21T13:53:33.819758+0000","last_undegraded":"2024-09-21T13:53:33.819758+0000","last_fullsized":"2024-09-21T13:53:33.819758+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:23.596879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:51:23.596879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"166'15635","reported_seq":15818,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:33.356736+0000","last_change":"2024-09-21T13:50:29.694240+0000","last_active":"2024-09-21T13:53:33.356736+0000","last_peered":"2024-09-21T13:53:33.356736+0000","last_clean":"2024-09-21T13:53:33.356736+0000","last_became_active":"2024-09-21T13:50:29.688167+0000","last_became_peered":"2024-09-21T13:50:29.688167+0000","last_unstale":"2024-09-21T13:53:33.356736+0000","last_undegraded":"2024-09-21T13:53:33.356736+0000","last_fullsized":"2024-09-21T13:53:33.356736+0000","mapping_epoch":94,"log_start":"134'10800","ondisk_log_start":"134'10800","created":83,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":4835,"log_dups_size":0,"ondisk_log_size":4835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110297088,"num_objects":1683,"num_object_clones":0,"num_object_copies":5049,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15635,"num_write_kb":554176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"68'13","reported_seq":255,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.767120+0000","last_change":"2024-09-21T13:50:06.973399+0000","last_active":"2024-09-21T13:53:29.767120+0000","last_peered":"2024-09-21T13:53:29.767120+0000","last_clean":"2024-09-21T13:53:29.767120+0000","last_became_active":"2024-09-21T13:49:00.053323+0000","last_became_peered":"2024-09-21T13:49:00.053323+0000","last_unstale":"2024-09-21T13:53:29.767120+0000","last_undegraded":"2024-09-21T13:53:29.767120+0000","last_fullsized":"2024-09-21T13:53:29.767120+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'13","last_scrub_stamp":"2024-09-21T13:50:06.973336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.973336+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:36.427623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":259,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.769227+0000","last_change":"2024-09-21T13:53:24.885266+0000","last_active":"2024-09-21T13:53:29.769227+0000","last_peered":"2024-09-21T13:53:29.769227+0000","last_clean":"2024-09-21T13:53:29.769227+0000","last_became_active":"2024-09-21T13:52:04.320581+0000","last_became_peered":"2024-09-21T13:52:04.320581+0000","last_unstale":"2024-09-21T13:53:29.769227+0000","last_undegraded":"2024-09-21T13:53:29.769227+0000","last_fullsized":"2024-09-21T13:53:29.769227+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:24.884932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:53:24.884932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:31.375453+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":162,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T13:53:34.916543+0000","last_change":"2024-09-21T13:53:19.985044+0000","last_active":"2024-09-21T13:53:34.916543+0000","last_peered":"2024-09-21T13:53:34.916543+0000","last_clean":"2024-09-21T13:53:34.916543+0000","last_became_active":"2024-09-21T13:52:04.316171+0000","last_became_peered":"2024-09-21T13:52:04.316171+0000","last_unstale":"2024-09-21T13:53:34.916543+0000","last_undegraded":"2024-09-21T13:53:34.916543+0000","last_fullsized":"2024-09-21T13:53:34.916543+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:53:19.984966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:53:19.984966+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:47.985977+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":106,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T13:53:34.918105+0000","last_change":"2024-09-21T13:53:18.988030+0000","last_active":"2024-09-21T13:53:34.918105+0000","last_peered":"2024-09-21T13:53:34.918105+0000","last_clean":"2024-09-21T13:53:34.918105+0000","last_became_active":"2024-09-21T13:52:04.320701+0000","last_became_peered":"2024-09-21T13:52:04.320701+0000","last_unstale":"2024-09-21T13:53:34.918105+0000","last_undegraded":"2024-09-21T13:53:34.918105+0000","last_fullsized":"2024-09-21T13:53:34.918105+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:18.987829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:53:18.987829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:21.866149+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":228,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.766646+0000","last_change":"2024-09-21T13:51:12.212558+0000","last_active":"2024-09-21T13:53:29.766646+0000","last_peered":"2024-09-21T13:53:29.766646+0000","last_clean":"2024-09-21T13:53:29.766646+0000","last_became_active":"2024-09-21T13:49:04.431035+0000","last_became_peered":"2024-09-21T13:49:04.431035+0000","last_unstale":"2024-09-21T13:53:29.766646+0000","last_undegraded":"2024-09-21T13:53:29.766646+0000","last_fullsized":"2024-09-21T13:53:29.766646+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:12.212391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:51:12.212391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"166'15433","reported_seq":25603,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:33.434147+0000","last_change":"2024-09-21T13:53:21.165869+0000","last_active":"2024-09-21T13:53:33.434147+0000","last_peered":"2024-09-21T13:53:33.434147+0000","last_clean":"2024-09-21T13:53:33.434147+0000","last_became_active":"2024-09-21T13:52:43.938223+0000","last_became_peered":"2024-09-21T13:52:43.938223+0000","last_unstale":"2024-09-21T13:53:33.434147+0000","last_undegraded":"2024-09-21T13:53:33.434147+0000","last_fullsized":"2024-09-21T13:53:33.434147+0000","mapping_epoch":147,"log_start":"129'10200","ondisk_log_start":"129'10200","created":83,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'14639","last_scrub_stamp":"2024-09-21T13:53:21.165793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:53:21.165793+0000","objects_scrubbed":2453,"log_size":5233,"log_dups_size":0,"ondisk_log_size":5233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:29.496429+0000","scrub_duration":804,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106758144,"num_objects":1629,"num_object_clones":0,"num_object_copies":4887,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15433,"num_write_kb":545984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4971,"num_bytes_recovered":325779456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"68'12","reported_seq":286,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.784245+0000","last_change":"2024-09-21T13:53:17.740156+0000","last_active":"2024-09-21T13:53:29.784245+0000","last_peered":"2024-09-21T13:53:29.784245+0000","last_clean":"2024-09-21T13:53:29.784245+0000","last_became_active":"2024-09-21T13:52:04.318875+0000","last_became_peered":"2024-09-21T13:52:04.318875+0000","last_unstale":"2024-09-21T13:53:29.784245+0000","last_undegraded":"2024-09-21T13:53:29.784245+0000","last_fullsized":"2024-09-21T13:53:29.784245+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'12","last_scrub_stamp":"2024-09-21T13:53:17.740080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:17.740080+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:30.638014+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":164,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.759440+0000","last_change":"2024-09-21T13:50:30.020722+0000","last_active":"2024-09-21T13:53:29.759440+0000","last_peered":"2024-09-21T13:53:29.759440+0000","last_clean":"2024-09-21T13:53:29.759440+0000","last_became_active":"2024-09-21T13:50:29.688352+0000","last_became_peered":"2024-09-21T13:50:29.688352+0000","last_unstale":"2024-09-21T13:53:29.759440+0000","last_undegraded":"2024-09-21T13:53:29.759440+0000","last_fullsized":"2024-09-21T13:53:29.759440+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:30.020656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:30.020656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":513,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:32.273168+0000","last_change":"2024-09-21T13:51:21.271997+0000","last_active":"2024-09-21T13:53:32.273168+0000","last_peered":"2024-09-21T13:53:32.273168+0000","last_clean":"2024-09-21T13:53:32.273168+0000","last_became_active":"2024-09-21T13:48:57.046084+0000","last_became_peered":"2024-09-21T13:48:57.046084+0000","last_unstale":"2024-09-21T13:53:32.273168+0000","last_undegraded":"2024-09-21T13:53:32.273168+0000","last_fullsized":"2024-09-21T13:53:32.273168+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-21T13:51:21.271917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:51:21.271917+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00047048000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":270,"num_read_kb":265,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":224,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.765628+0000","last_change":"2024-09-21T13:51:53.518645+0000","last_active":"2024-09-21T13:53:29.765628+0000","last_peered":"2024-09-21T13:53:29.765628+0000","last_clean":"2024-09-21T13:53:29.765628+0000","last_became_active":"2024-09-21T13:49:02.066641+0000","last_became_peered":"2024-09-21T13:49:02.066641+0000","last_unstale":"2024-09-21T13:53:29.765628+0000","last_undegraded":"2024-09-21T13:53:29.765628+0000","last_fullsized":"2024-09-21T13:53:29.765628+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:53.518567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:53.518567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:14.335443+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":228,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.769449+0000","last_change":"2024-09-21T13:52:21.332066+0000","last_active":"2024-09-21T13:53:29.769449+0000","last_peered":"2024-09-21T13:53:29.769449+0000","last_clean":"2024-09-21T13:53:29.769449+0000","last_became_active":"2024-09-21T13:49:04.078372+0000","last_became_peered":"2024-09-21T13:49:04.078372+0000","last_unstale":"2024-09-21T13:53:29.769449+0000","last_undegraded":"2024-09-21T13:53:29.769449+0000","last_fullsized":"2024-09-21T13:53:29.769449+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:21.331792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:21.331792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:44.913330+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"166'15554","reported_seq":15766,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:33.416050+0000","last_change":"2024-09-21T13:53:27.064451+0000","last_active":"2024-09-21T13:53:33.416050+0000","last_peered":"2024-09-21T13:53:33.416050+0000","last_clean":"2024-09-21T13:53:33.416050+0000","last_became_active":"2024-09-21T13:50:29.694886+0000","last_became_peered":"2024-09-21T13:50:29.694886+0000","last_unstale":"2024-09-21T13:53:33.416050+0000","last_undegraded":"2024-09-21T13:53:33.416050+0000","last_fullsized":"2024-09-21T13:53:33.416050+0000","mapping_epoch":94,"log_start":"129'10300","ondisk_log_start":"129'10300","created":83,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'15120","last_scrub_stamp":"2024-09-21T13:53:27.064254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:53:27.064254+0000","objects_scrubbed":2095,"log_size":5254,"log_dups_size":0,"ondisk_log_size":5254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:50.816602+0000","scrub_duration":619,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107479040,"num_objects":1640,"num_object_clones":0,"num_object_copies":4920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15554,"num_write_kb":550208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"68'16","reported_seq":314,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.762679+0000","last_change":"2024-09-21T13:52:19.452239+0000","last_active":"2024-09-21T13:53:29.762679+0000","last_peered":"2024-09-21T13:53:29.762679+0000","last_clean":"2024-09-21T13:53:29.762679+0000","last_became_active":"2024-09-21T13:49:00.426867+0000","last_became_peered":"2024-09-21T13:49:00.426867+0000","last_unstale":"2024-09-21T13:53:29.762679+0000","last_undegraded":"2024-09-21T13:53:29.762679+0000","last_fullsized":"2024-09-21T13:53:29.762679+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'16","last_scrub_stamp":"2024-09-21T13:52:19.452163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:52:19.452163+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:46.305610+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":234,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T13:53:33.825106+0000","last_change":"2024-09-21T13:50:09.208053+0000","last_active":"2024-09-21T13:53:33.825106+0000","last_peered":"2024-09-21T13:53:33.825106+0000","last_clean":"2024-09-21T13:53:33.825106+0000","last_became_active":"2024-09-21T13:48:58.052659+0000","last_became_peered":"2024-09-21T13:48:58.052659+0000","last_unstale":"2024-09-21T13:53:33.825106+0000","last_undegraded":"2024-09-21T13:53:33.825106+0000","last_fullsized":"2024-09-21T13:53:33.825106+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:09.208003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:09.208003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"75'79","reported_seq":281,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.767450+0000","last_change":"2024-09-21T13:50:27.096470+0000","last_active":"2024-09-21T13:53:29.767450+0000","last_peered":"2024-09-21T13:53:29.767450+0000","last_clean":"2024-09-21T13:53:29.767450+0000","last_became_active":"2024-09-21T13:48:48.454646+0000","last_became_peered":"2024-09-21T13:48:48.454646+0000","last_unstale":"2024-09-21T13:53:29.767450+0000","last_undegraded":"2024-09-21T13:53:29.767450+0000","last_fullsized":"2024-09-21T13:53:29.767450+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:50:27.096413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:50:27.096413+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":227,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.766851+0000","last_change":"2024-09-21T13:52:04.320948+0000","last_active":"2024-09-21T13:53:29.766851+0000","last_peered":"2024-09-21T13:53:29.766851+0000","last_clean":"2024-09-21T13:53:29.766851+0000","last_became_active":"2024-09-21T13:52:04.320627+0000","last_became_peered":"2024-09-21T13:52:04.320627+0000","last_unstale":"2024-09-21T13:53:29.766851+0000","last_undegraded":"2024-09-21T13:53:29.766851+0000","last_fullsized":"2024-09-21T13:53:29.766851+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:42.557761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":220,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.765366+0000","last_change":"2024-09-21T13:50:25.560928+0000","last_active":"2024-09-21T13:53:29.765366+0000","last_peered":"2024-09-21T13:53:29.765366+0000","last_clean":"2024-09-21T13:53:29.765366+0000","last_became_active":"2024-09-21T13:49:04.078091+0000","last_became_peered":"2024-09-21T13:49:04.078091+0000","last_unstale":"2024-09-21T13:53:29.765366+0000","last_undegraded":"2024-09-21T13:53:29.765366+0000","last_fullsized":"2024-09-21T13:53:29.765366+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:25.560863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:25.560863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:40.548062+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"166'15448","reported_seq":15755,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:33.391862+0000","last_change":"2024-09-21T13:52:50.770818+0000","last_active":"2024-09-21T13:53:33.391862+0000","last_peered":"2024-09-21T13:53:33.391862+0000","last_clean":"2024-09-21T13:53:33.391862+0000","last_became_active":"2024-09-21T13:50:07.422466+0000","last_became_peered":"2024-09-21T13:50:07.422466+0000","last_unstale":"2024-09-21T13:53:33.391862+0000","last_undegraded":"2024-09-21T13:53:33.391862+0000","last_fullsized":"2024-09-21T13:53:33.391862+0000","mapping_epoch":83,"log_start":"124'9100","ondisk_log_start":"124'9100","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'12965","last_scrub_stamp":"2024-09-21T13:52:50.770670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:52:50.770670+0000","objects_scrubbed":4108,"log_size":6348,"log_dups_size":0,"ondisk_log_size":6348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:56.210477+0000","scrub_duration":1333,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103874584,"num_objects":1586,"num_object_clones":0,"num_object_copies":4758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1586,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15448,"num_write_kb":545025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"68'11","reported_seq":239,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.760721+0000","last_change":"2024-09-21T13:49:00.054827+0000","last_active":"2024-09-21T13:53:29.760721+0000","last_peered":"2024-09-21T13:53:29.760721+0000","last_clean":"2024-09-21T13:53:29.760721+0000","last_became_active":"2024-09-21T13:49:00.053268+0000","last_became_peered":"2024-09-21T13:49:00.053268+0000","last_unstale":"2024-09-21T13:53:29.760721+0000","last_undegraded":"2024-09-21T13:53:29.760721+0000","last_fullsized":"2024-09-21T13:53:29.760721+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:16.039942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":243,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.759627+0000","last_change":"2024-09-21T13:52:48.165210+0000","last_active":"2024-09-21T13:53:29.759627+0000","last_peered":"2024-09-21T13:53:29.759627+0000","last_clean":"2024-09-21T13:53:29.759627+0000","last_became_active":"2024-09-21T13:52:04.318345+0000","last_became_peered":"2024-09-21T13:52:04.318345+0000","last_unstale":"2024-09-21T13:53:29.759627+0000","last_undegraded":"2024-09-21T13:53:29.759627+0000","last_fullsized":"2024-09-21T13:53:29.759627+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:48.165145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:52:48.165145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:03.774580+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":231,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.765294+0000","last_change":"2024-09-21T13:51:46.503640+0000","last_active":"2024-09-21T13:53:29.765294+0000","last_peered":"2024-09-21T13:53:29.765294+0000","last_clean":"2024-09-21T13:53:29.765294+0000","last_became_active":"2024-09-21T13:49:02.061901+0000","last_became_peered":"2024-09-21T13:49:02.061901+0000","last_unstale":"2024-09-21T13:53:29.765294+0000","last_undegraded":"2024-09-21T13:53:29.765294+0000","last_fullsized":"2024-09-21T13:53:29.765294+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:46.503576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:46.503576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:11.300124+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":222,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T13:53:33.822287+0000","last_change":"2024-09-21T13:51:51.814259+0000","last_active":"2024-09-21T13:53:33.822287+0000","last_peered":"2024-09-21T13:53:33.822287+0000","last_clean":"2024-09-21T13:53:33.822287+0000","last_became_active":"2024-09-21T13:49:04.431268+0000","last_became_peered":"2024-09-21T13:49:04.431268+0000","last_unstale":"2024-09-21T13:53:33.822287+0000","last_undegraded":"2024-09-21T13:53:33.822287+0000","last_fullsized":"2024-09-21T13:53:33.822287+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:51.814200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:51:51.814200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"168'15622","reported_seq":23058,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T13:53:37.418200+0000","last_change":"2024-09-21T13:53:29.631333+0000","last_active":"2024-09-21T13:53:37.418200+0000","last_peered":"2024-09-21T13:53:37.418200+0000","last_clean":"2024-09-21T13:53:37.418200+0000","last_became_active":"2024-09-21T13:53:17.659357+0000","last_became_peered":"2024-09-21T13:53:17.659357+0000","last_unstale":"2024-09-21T13:53:37.418200+0000","last_undegraded":"2024-09-21T13:53:37.418200+0000","last_fullsized":"2024-09-21T13:53:37.418200+0000","mapping_epoch":160,"log_start":"122'8788","ondisk_log_start":"122'8788","created":83,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'15154","last_scrub_stamp":"2024-09-21T13:53:29.631248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:53:29.631248+0000","objects_scrubbed":1863,"log_size":6834,"log_dups_size":0,"ondisk_log_size":6834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:49.019026+0000","scrub_duration":768,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89653248,"num_objects":1368,"num_object_clones":0,"num_object_copies":4104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15622,"num_write_kb":543680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3606,"num_bytes_recovered":236322816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"68'17","reported_seq":267,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.760484+0000","last_change":"2024-09-21T13:52:45.176950+0000","last_active":"2024-09-21T13:53:29.760484+0000","last_peered":"2024-09-21T13:53:29.760484+0000","last_clean":"2024-09-21T13:53:29.760484+0000","last_became_active":"2024-09-21T13:52:04.313218+0000","last_became_peered":"2024-09-21T13:52:04.313218+0000","last_unstale":"2024-09-21T13:53:29.760484+0000","last_undegraded":"2024-09-21T13:53:29.760484+0000","last_fullsized":"2024-09-21T13:53:29.760484+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'17","last_scrub_stamp":"2024-09-21T13:52:45.176873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:52:45.176873+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:03.832378+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":240,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.762252+0000","last_change":"2024-09-21T13:52:45.444445+0000","last_active":"2024-09-21T13:53:29.762252+0000","last_peered":"2024-09-21T13:53:29.762252+0000","last_clean":"2024-09-21T13:53:29.762252+0000","last_became_active":"2024-09-21T13:48:58.049142+0000","last_became_peered":"2024-09-21T13:48:58.049142+0000","last_unstale":"2024-09-21T13:53:29.762252+0000","last_undegraded":"2024-09-21T13:53:29.762252+0000","last_fullsized":"2024-09-21T13:53:29.762252+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:45.444381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:52:45.444381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:54.348542+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":216,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.769627+0000","last_change":"2024-09-21T13:49:02.067811+0000","last_active":"2024-09-21T13:53:29.769627+0000","last_peered":"2024-09-21T13:53:29.769627+0000","last_clean":"2024-09-21T13:53:29.769627+0000","last_became_active":"2024-09-21T13:49:02.067472+0000","last_became_peered":"2024-09-21T13:49:02.067472+0000","last_unstale":"2024-09-21T13:53:29.769627+0000","last_undegraded":"2024-09-21T13:53:29.769627+0000","last_fullsized":"2024-09-21T13:53:29.769627+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:02.701863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":106,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T13:53:34.917504+0000","last_change":"2024-09-21T13:53:18.028013+0000","last_active":"2024-09-21T13:53:34.917504+0000","last_peered":"2024-09-21T13:53:34.917504+0000","last_clean":"2024-09-21T13:53:34.917504+0000","last_became_active":"2024-09-21T13:52:04.315920+0000","last_became_peered":"2024-09-21T13:52:04.315920+0000","last_unstale":"2024-09-21T13:53:34.917504+0000","last_undegraded":"2024-09-21T13:53:34.917504+0000","last_fullsized":"2024-09-21T13:53:34.917504+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:18.027944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:18.027944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:26.231349+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"167'15099","reported_seq":15286,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T13:53:33.944683+0000","last_change":"2024-09-21T13:50:29.691429+0000","last_active":"2024-09-21T13:53:33.944683+0000","last_peered":"2024-09-21T13:53:33.944683+0000","last_clean":"2024-09-21T13:53:33.944683+0000","last_became_active":"2024-09-21T13:50:29.687898+0000","last_became_peered":"2024-09-21T13:50:29.687898+0000","last_unstale":"2024-09-21T13:53:33.944683+0000","last_undegraded":"2024-09-21T13:53:33.944683+0000","last_fullsized":"2024-09-21T13:53:33.944683+0000","mapping_epoch":94,"log_start":"134'10400","ondisk_log_start":"134'10400","created":83,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":4699,"log_dups_size":0,"ondisk_log_size":4699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98762752,"num_objects":1507,"num_object_clones":0,"num_object_copies":4521,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15099,"num_write_kb":531392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"68'10","reported_seq":243,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.760424+0000","last_change":"2024-09-21T13:51:57.155811+0000","last_active":"2024-09-21T13:53:29.760424+0000","last_peered":"2024-09-21T13:53:29.760424+0000","last_clean":"2024-09-21T13:53:29.760424+0000","last_became_active":"2024-09-21T13:49:00.059850+0000","last_became_peered":"2024-09-21T13:49:00.059850+0000","last_unstale":"2024-09-21T13:53:29.760424+0000","last_undegraded":"2024-09-21T13:53:29.760424+0000","last_fullsized":"2024-09-21T13:53:29.760424+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:51:57.155761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:57.155761+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":234,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T13:53:33.819522+0000","last_change":"2024-09-21T13:51:28.565842+0000","last_active":"2024-09-21T13:53:33.819522+0000","last_peered":"2024-09-21T13:53:33.819522+0000","last_clean":"2024-09-21T13:53:33.819522+0000","last_became_active":"2024-09-21T13:48:58.058309+0000","last_became_peered":"2024-09-21T13:48:58.058309+0000","last_unstale":"2024-09-21T13:53:33.819522+0000","last_undegraded":"2024-09-21T13:53:33.819522+0000","last_fullsized":"2024-09-21T13:53:33.819522+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:28.565766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:51:28.565766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"132'10","reported_seq":135,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T13:53:34.918453+0000","last_change":"2024-09-21T13:52:04.320834+0000","last_active":"2024-09-21T13:53:34.918453+0000","last_peered":"2024-09-21T13:53:34.918453+0000","last_clean":"2024-09-21T13:53:34.918453+0000","last_became_active":"2024-09-21T13:52:04.320441+0000","last_became_peered":"2024-09-21T13:52:04.320441+0000","last_unstale":"2024-09-21T13:53:34.918453+0000","last_undegraded":"2024-09-21T13:53:34.918453+0000","last_fullsized":"2024-09-21T13:53:34.918453+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:16.073559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"68'1","reported_seq":234,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.760259+0000","last_change":"2024-09-21T13:52:04.314151+0000","last_active":"2024-09-21T13:53:29.760259+0000","last_peered":"2024-09-21T13:53:29.760259+0000","last_clean":"2024-09-21T13:53:29.760259+0000","last_became_active":"2024-09-21T13:52:04.313210+0000","last_became_peered":"2024-09-21T13:52:04.313210+0000","last_unstale":"2024-09-21T13:53:29.760259+0000","last_undegraded":"2024-09-21T13:53:29.760259+0000","last_fullsized":"2024-09-21T13:53:29.760259+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T13:50:41.000201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:41.000201+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"168'15602","reported_seq":25709,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T13:53:37.373430+0000","last_change":"2024-09-21T13:53:31.637036+0000","last_active":"2024-09-21T13:53:37.373430+0000","last_peered":"2024-09-21T13:53:37.373430+0000","last_clean":"2024-09-21T13:53:37.373430+0000","last_became_active":"2024-09-21T13:52:42.942510+0000","last_became_peered":"2024-09-21T13:52:42.942510+0000","last_unstale":"2024-09-21T13:53:37.373430+0000","last_undegraded":"2024-09-21T13:53:37.373430+0000","last_fullsized":"2024-09-21T13:53:37.373430+0000","mapping_epoch":146,"log_start":"122'8778","ondisk_log_start":"122'8778","created":83,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'15232","last_scrub_stamp":"2024-09-21T13:53:31.636880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:53:31.636880+0000","objects_scrubbed":1782,"log_size":6824,"log_dups_size":0,"ondisk_log_size":6824,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:43.661832+0000","scrub_duration":726,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91488256,"num_objects":1396,"num_object_clones":0,"num_object_copies":4188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15602,"num_write_kb":543936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4942,"num_bytes_recovered":323878912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"68'9","reported_seq":252,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.781326+0000","last_change":"2024-09-21T13:53:18.740202+0000","last_active":"2024-09-21T13:53:29.781326+0000","last_peered":"2024-09-21T13:53:29.781326+0000","last_clean":"2024-09-21T13:53:29.781326+0000","last_became_active":"2024-09-21T13:49:00.053016+0000","last_became_peered":"2024-09-21T13:49:00.053016+0000","last_unstale":"2024-09-21T13:53:29.781326+0000","last_undegraded":"2024-09-21T13:53:29.781326+0000","last_fullsized":"2024-09-21T13:53:29.781326+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:53:18.740111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:18.740111+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:34.752035+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":232,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.765035+0000","last_change":"2024-09-21T13:50:38.445732+0000","last_active":"2024-09-21T13:53:29.765035+0000","last_peered":"2024-09-21T13:53:29.765035+0000","last_clean":"2024-09-21T13:53:29.765035+0000","last_became_active":"2024-09-21T13:48:58.049543+0000","last_became_peered":"2024-09-21T13:48:58.049543+0000","last_unstale":"2024-09-21T13:53:29.765035+0000","last_undegraded":"2024-09-21T13:53:29.765035+0000","last_fullsized":"2024-09-21T13:53:29.765035+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:38.445667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:38.445667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:35.568002+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":154,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.758248+0000","last_change":"2024-09-21T13:52:59.112648+0000","last_active":"2024-09-21T13:53:29.758248+0000","last_peered":"2024-09-21T13:53:29.758248+0000","last_clean":"2024-09-21T13:53:29.758248+0000","last_became_active":"2024-09-21T13:52:04.317429+0000","last_became_peered":"2024-09-21T13:52:04.317429+0000","last_unstale":"2024-09-21T13:53:29.758248+0000","last_undegraded":"2024-09-21T13:53:29.758248+0000","last_fullsized":"2024-09-21T13:53:29.758248+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:59.112602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:52:59.112602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:59.206422+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":175,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.766503+0000","last_change":"2024-09-21T13:52:51.590860+0000","last_active":"2024-09-21T13:53:29.766503+0000","last_peered":"2024-09-21T13:53:29.766503+0000","last_clean":"2024-09-21T13:53:29.766503+0000","last_became_active":"2024-09-21T13:52:04.321703+0000","last_became_peered":"2024-09-21T13:52:04.321703+0000","last_unstale":"2024-09-21T13:53:29.766503+0000","last_undegraded":"2024-09-21T13:53:29.766503+0000","last_fullsized":"2024-09-21T13:53:29.766503+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:51.590800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:51.590800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:01.244154+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.9","version":"166'15417","reported_seq":15602,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:33.553475+0000","last_change":"2024-09-21T13:50:29.690666+0000","last_active":"2024-09-21T13:53:33.553475+0000","last_peered":"2024-09-21T13:53:33.553475+0000","last_clean":"2024-09-21T13:53:33.553475+0000","last_became_active":"2024-09-21T13:50:29.687815+0000","last_became_peered":"2024-09-21T13:50:29.687815+0000","last_unstale":"2024-09-21T13:53:33.553475+0000","last_undegraded":"2024-09-21T13:53:33.553475+0000","last_fullsized":"2024-09-21T13:53:33.553475+0000","mapping_epoch":94,"log_start":"132'10400","ondisk_log_start":"132'10400","created":83,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":5017,"log_dups_size":0,"ondisk_log_size":5017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104005632,"num_objects":1587,"num_object_clones":0,"num_object_copies":4761,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15417,"num_write_kb":544128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"68'19","reported_seq":259,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.762113+0000","last_change":"2024-09-21T13:50:03.769686+0000","last_active":"2024-09-21T13:53:29.762113+0000","last_peered":"2024-09-21T13:53:29.762113+0000","last_clean":"2024-09-21T13:53:29.762113+0000","last_became_active":"2024-09-21T13:49:00.425955+0000","last_became_peered":"2024-09-21T13:49:00.425955+0000","last_unstale":"2024-09-21T13:53:29.762113+0000","last_undegraded":"2024-09-21T13:53:29.762113+0000","last_fullsized":"2024-09-21T13:53:29.762113+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-21T13:50:03.769613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:03.769613+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:32.476093+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":224,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.767320+0000","last_change":"2024-09-21T13:48:58.053940+0000","last_active":"2024-09-21T13:53:29.767320+0000","last_peered":"2024-09-21T13:53:29.767320+0000","last_clean":"2024-09-21T13:53:29.767320+0000","last_became_active":"2024-09-21T13:48:58.053641+0000","last_became_peered":"2024-09-21T13:48:58.053641+0000","last_unstale":"2024-09-21T13:53:29.767320+0000","last_undegraded":"2024-09-21T13:53:29.767320+0000","last_fullsized":"2024-09-21T13:53:29.767320+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:17.391569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":176,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T13:53:34.918230+0000","last_change":"2024-09-21T13:52:04.321471+0000","last_active":"2024-09-21T13:53:34.918230+0000","last_peered":"2024-09-21T13:53:34.918230+0000","last_clean":"2024-09-21T13:53:34.918230+0000","last_became_active":"2024-09-21T13:52:04.321146+0000","last_became_peered":"2024-09-21T13:52:04.321146+0000","last_unstale":"2024-09-21T13:53:34.918230+0000","last_undegraded":"2024-09-21T13:53:34.918230+0000","last_fullsized":"2024-09-21T13:53:34.918230+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:40.850721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:40.850721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":231,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.767397+0000","last_change":"2024-09-21T13:52:04.323179+0000","last_active":"2024-09-21T13:53:29.767397+0000","last_peered":"2024-09-21T13:53:29.767397+0000","last_clean":"2024-09-21T13:53:29.767397+0000","last_became_active":"2024-09-21T13:52:04.322213+0000","last_became_peered":"2024-09-21T13:52:04.322213+0000","last_unstale":"2024-09-21T13:53:29.767397+0000","last_undegraded":"2024-09-21T13:53:29.767397+0000","last_fullsized":"2024-09-21T13:53:29.767397+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:35.067261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:35.067261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"167'15371","reported_seq":24643,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T13:53:33.952069+0000","last_change":"2024-09-21T13:53:01.956068+0000","last_active":"2024-09-21T13:53:33.952069+0000","last_peered":"2024-09-21T13:53:33.952069+0000","last_clean":"2024-09-21T13:53:33.952069+0000","last_became_active":"2024-09-21T13:53:01.911422+0000","last_became_peered":"2024-09-21T13:53:01.911422+0000","last_unstale":"2024-09-21T13:53:33.952069+0000","last_undegraded":"2024-09-21T13:53:33.952069+0000","last_fullsized":"2024-09-21T13:53:33.952069+0000","mapping_epoch":153,"log_start":"134'10700","ondisk_log_start":"134'10700","created":83,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":4671,"log_dups_size":0,"ondisk_log_size":4671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107282432,"num_objects":1637,"num_object_clones":0,"num_object_copies":4911,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15371,"num_write_kb":544256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4537,"num_bytes_recovered":297336832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"68'12","reported_seq":251,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.759508+0000","last_change":"2024-09-21T13:51:10.942190+0000","last_active":"2024-09-21T13:53:29.759508+0000","last_peered":"2024-09-21T13:53:29.759508+0000","last_clean":"2024-09-21T13:53:29.759508+0000","last_became_active":"2024-09-21T13:49:00.059720+0000","last_became_peered":"2024-09-21T13:49:00.059720+0000","last_unstale":"2024-09-21T13:53:29.759508+0000","last_undegraded":"2024-09-21T13:53:29.759508+0000","last_fullsized":"2024-09-21T13:53:29.759508+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'12","last_scrub_stamp":"2024-09-21T13:51:10.942105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:10.942105+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":232,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.767617+0000","last_change":"2024-09-21T13:50:29.100595+0000","last_active":"2024-09-21T13:53:29.767617+0000","last_peered":"2024-09-21T13:53:29.767617+0000","last_clean":"2024-09-21T13:53:29.767617+0000","last_became_active":"2024-09-21T13:48:58.057641+0000","last_became_peered":"2024-09-21T13:48:58.057641+0000","last_unstale":"2024-09-21T13:53:29.767617+0000","last_undegraded":"2024-09-21T13:53:29.767617+0000","last_fullsized":"2024-09-21T13:53:29.767617+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:29.100547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:29.100547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":149,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T13:53:34.914865+0000","last_change":"2024-09-21T13:53:31.964122+0000","last_active":"2024-09-21T13:53:34.914865+0000","last_peered":"2024-09-21T13:53:34.914865+0000","last_clean":"2024-09-21T13:53:34.914865+0000","last_became_active":"2024-09-21T13:52:04.322147+0000","last_became_peered":"2024-09-21T13:52:04.322147+0000","last_unstale":"2024-09-21T13:53:34.914865+0000","last_undegraded":"2024-09-21T13:53:34.914865+0000","last_fullsized":"2024-09-21T13:53:34.914865+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:31.958946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:53:31.958946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:51.369874+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":222,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T13:53:33.819174+0000","last_change":"2024-09-21T13:53:20.888048+0000","last_active":"2024-09-21T13:53:33.819174+0000","last_peered":"2024-09-21T13:53:33.819174+0000","last_clean":"2024-09-21T13:53:33.819174+0000","last_became_active":"2024-09-21T13:49:04.078089+0000","last_became_peered":"2024-09-21T13:49:04.078089+0000","last_unstale":"2024-09-21T13:53:33.819174+0000","last_undegraded":"2024-09-21T13:53:33.819174+0000","last_fullsized":"2024-09-21T13:53:33.819174+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:20.882449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:20.882449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:45.947326+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"166'15576","reported_seq":15754,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:33.549557+0000","last_change":"2024-09-21T13:50:07.422240+0000","last_active":"2024-09-21T13:53:33.549557+0000","last_peered":"2024-09-21T13:53:33.549557+0000","last_clean":"2024-09-21T13:53:33.549557+0000","last_became_active":"2024-09-21T13:50:07.421962+0000","last_became_peered":"2024-09-21T13:50:07.421962+0000","last_unstale":"2024-09-21T13:53:33.549557+0000","last_undegraded":"2024-09-21T13:53:33.549557+0000","last_fullsized":"2024-09-21T13:53:33.549557+0000","mapping_epoch":83,"log_start":"132'10500","ondisk_log_start":"132'10500","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":5076,"log_dups_size":0,"ondisk_log_size":5076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:37.396702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106692608,"num_objects":1628,"num_object_clones":0,"num_object_copies":4884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15576,"num_write_kb":550528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"68'15","reported_seq":261,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:31.204731+0000","last_change":"2024-09-21T13:53:31.204731+0000","last_active":"2024-09-21T13:53:31.204731+0000","last_peered":"2024-09-21T13:53:31.204731+0000","last_clean":"2024-09-21T13:53:31.204731+0000","last_became_active":"2024-09-21T13:49:00.425997+0000","last_became_peered":"2024-09-21T13:49:00.425997+0000","last_unstale":"2024-09-21T13:53:31.204731+0000","last_undegraded":"2024-09-21T13:53:31.204731+0000","last_fullsized":"2024-09-21T13:53:31.204731+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:53:31.204637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:31.204637+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:41.385554+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":252,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.767122+0000","last_change":"2024-09-21T13:53:19.396344+0000","last_active":"2024-09-21T13:53:29.767122+0000","last_peered":"2024-09-21T13:53:29.767122+0000","last_clean":"2024-09-21T13:53:29.767122+0000","last_became_active":"2024-09-21T13:48:58.057632+0000","last_became_peered":"2024-09-21T13:48:58.057632+0000","last_unstale":"2024-09-21T13:53:29.767122+0000","last_undegraded":"2024-09-21T13:53:29.767122+0000","last_fullsized":"2024-09-21T13:53:29.767122+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T13:53:19.396277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:53:19.396277+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:47.867330+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"68'8","reported_seq":288,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.766698+0000","last_change":"2024-09-21T13:51:26.297643+0000","last_active":"2024-09-21T13:53:29.766698+0000","last_peered":"2024-09-21T13:53:29.766698+0000","last_clean":"2024-09-21T13:53:29.766698+0000","last_became_active":"2024-09-21T13:49:02.056167+0000","last_became_peered":"2024-09-21T13:49:02.056167+0000","last_unstale":"2024-09-21T13:53:29.766698+0000","last_undegraded":"2024-09-21T13:53:29.766698+0000","last_fullsized":"2024-09-21T13:53:29.766698+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:51:26.297550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:26.297550+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":228,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.764514+0000","last_change":"2024-09-21T13:52:54.314087+0000","last_active":"2024-09-21T13:53:29.764514+0000","last_peered":"2024-09-21T13:53:29.764514+0000","last_clean":"2024-09-21T13:53:29.764514+0000","last_became_active":"2024-09-21T13:49:04.432546+0000","last_became_peered":"2024-09-21T13:49:04.432546+0000","last_unstale":"2024-09-21T13:53:29.764514+0000","last_undegraded":"2024-09-21T13:53:29.764514+0000","last_fullsized":"2024-09-21T13:53:29.764514+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:54.314042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:54.314042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:05.575071+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":239,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.762029+0000","last_change":"2024-09-21T13:51:54.352313+0000","last_active":"2024-09-21T13:53:29.762029+0000","last_peered":"2024-09-21T13:53:29.762029+0000","last_clean":"2024-09-21T13:53:29.762029+0000","last_became_active":"2024-09-21T13:48:58.056080+0000","last_became_peered":"2024-09-21T13:48:58.056080+0000","last_unstale":"2024-09-21T13:53:29.762029+0000","last_undegraded":"2024-09-21T13:53:29.762029+0000","last_fullsized":"2024-09-21T13:53:29.762029+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:54.352244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:51:54.352244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:14.235456+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"68'6","reported_seq":238,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T13:53:33.823212+0000","last_change":"2024-09-21T13:53:31.807215+0000","last_active":"2024-09-21T13:53:33.823212+0000","last_peered":"2024-09-21T13:53:33.823212+0000","last_clean":"2024-09-21T13:53:33.823212+0000","last_became_active":"2024-09-21T13:49:00.426829+0000","last_became_peered":"2024-09-21T13:49:00.426829+0000","last_unstale":"2024-09-21T13:53:33.823212+0000","last_undegraded":"2024-09-21T13:53:33.823212+0000","last_fullsized":"2024-09-21T13:53:33.823212+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'6","last_scrub_stamp":"2024-09-21T13:53:31.807028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:31.807028+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:49.612755+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":231,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.765386+0000","last_change":"2024-09-21T13:51:55.488973+0000","last_active":"2024-09-21T13:53:29.765386+0000","last_peered":"2024-09-21T13:53:29.765386+0000","last_clean":"2024-09-21T13:53:29.765386+0000","last_became_active":"2024-09-21T13:49:02.062231+0000","last_became_peered":"2024-09-21T13:49:02.062231+0000","last_unstale":"2024-09-21T13:53:29.765386+0000","last_undegraded":"2024-09-21T13:53:29.765386+0000","last_fullsized":"2024-09-21T13:53:29.765386+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:55.488908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:55.488908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:16.351786+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":227,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.766564+0000","last_change":"2024-09-21T13:53:18.373418+0000","last_active":"2024-09-21T13:53:29.766564+0000","last_peered":"2024-09-21T13:53:29.766564+0000","last_clean":"2024-09-21T13:53:29.766564+0000","last_became_active":"2024-09-21T13:49:04.432507+0000","last_became_peered":"2024-09-21T13:49:04.432507+0000","last_unstale":"2024-09-21T13:53:29.766564+0000","last_undegraded":"2024-09-21T13:53:29.766564+0000","last_fullsized":"2024-09-21T13:53:29.766564+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:18.373356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:18.373356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:38.812691+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"68'9","reported_seq":202,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T13:53:33.819903+0000","last_change":"2024-09-21T13:51:54.808758+0000","last_active":"2024-09-21T13:53:33.819903+0000","last_peered":"2024-09-21T13:53:33.819903+0000","last_clean":"2024-09-21T13:53:33.819903+0000","last_became_active":"2024-09-21T13:50:29.693589+0000","last_became_peered":"2024-09-21T13:50:29.693589+0000","last_unstale":"2024-09-21T13:53:33.819903+0000","last_undegraded":"2024-09-21T13:53:33.819903+0000","last_fullsized":"2024-09-21T13:53:33.819903+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:51:54.808559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:54.808559+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"61'1","reported_seq":229,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.766830+0000","last_change":"2024-09-21T13:48:58.050948+0000","last_active":"2024-09-21T13:53:29.766830+0000","last_peered":"2024-09-21T13:53:29.766830+0000","last_clean":"2024-09-21T13:53:29.766830+0000","last_became_active":"2024-09-21T13:48:58.050645+0000","last_became_peered":"2024-09-21T13:48:58.050645+0000","last_unstale":"2024-09-21T13:53:29.766830+0000","last_undegraded":"2024-09-21T13:53:29.766830+0000","last_fullsized":"2024-09-21T13:53:29.766830+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:13.439561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":225,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T13:53:33.823083+0000","last_change":"2024-09-21T13:50:43.375425+0000","last_active":"2024-09-21T13:53:33.823083+0000","last_peered":"2024-09-21T13:53:33.823083+0000","last_clean":"2024-09-21T13:53:33.823083+0000","last_became_active":"2024-09-21T13:49:02.061775+0000","last_became_peered":"2024-09-21T13:49:02.061775+0000","last_unstale":"2024-09-21T13:53:33.823083+0000","last_undegraded":"2024-09-21T13:53:33.823083+0000","last_fullsized":"2024-09-21T13:53:33.823083+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:43.375381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:43.375381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":172,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T13:53:34.917619+0000","last_change":"2024-09-21T13:52:04.324521+0000","last_active":"2024-09-21T13:53:34.917619+0000","last_peered":"2024-09-21T13:53:34.917619+0000","last_clean":"2024-09-21T13:53:34.917619+0000","last_became_active":"2024-09-21T13:52:04.323549+0000","last_became_peered":"2024-09-21T13:52:04.323549+0000","last_unstale":"2024-09-21T13:53:34.917619+0000","last_undegraded":"2024-09-21T13:53:34.917619+0000","last_fullsized":"2024-09-21T13:53:34.917619+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:41.808746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:41.808746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"68'9","reported_seq":243,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.765867+0000","last_change":"2024-09-21T13:51:28.547052+0000","last_active":"2024-09-21T13:53:29.765867+0000","last_peered":"2024-09-21T13:53:29.765867+0000","last_clean":"2024-09-21T13:53:29.765867+0000","last_became_active":"2024-09-21T13:49:00.427766+0000","last_became_peered":"2024-09-21T13:49:00.427766+0000","last_unstale":"2024-09-21T13:53:29.765867+0000","last_undegraded":"2024-09-21T13:53:29.765867+0000","last_fullsized":"2024-09-21T13:53:29.765867+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:51:28.546958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:28.546958+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:08.329203+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":164,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.766508+0000","last_change":"2024-09-21T13:50:30.076191+0000","last_active":"2024-09-21T13:53:29.766508+0000","last_peered":"2024-09-21T13:53:29.766508+0000","last_clean":"2024-09-21T13:53:29.766508+0000","last_became_active":"2024-09-21T13:50:29.690634+0000","last_became_peered":"2024-09-21T13:50:29.690634+0000","last_unstale":"2024-09-21T13:53:29.766508+0000","last_undegraded":"2024-09-21T13:53:29.766508+0000","last_fullsized":"2024-09-21T13:53:29.766508+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:30.076115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:30.076115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"132'10","reported_seq":302,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T13:53:33.822385+0000","last_change":"2024-09-21T13:52:04.319377+0000","last_active":"2024-09-21T13:53:33.822385+0000","last_peered":"2024-09-21T13:53:33.822385+0000","last_clean":"2024-09-21T13:53:33.822385+0000","last_became_active":"2024-09-21T13:52:04.318626+0000","last_became_peered":"2024-09-21T13:52:04.318626+0000","last_unstale":"2024-09-21T13:53:33.822385+0000","last_undegraded":"2024-09-21T13:53:33.822385+0000","last_fullsized":"2024-09-21T13:53:33.822385+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:50:29.249680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:29.249680+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":230,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.760365+0000","last_change":"2024-09-21T13:52:04.316179+0000","last_active":"2024-09-21T13:53:29.760365+0000","last_peered":"2024-09-21T13:53:29.760365+0000","last_clean":"2024-09-21T13:53:29.760365+0000","last_became_active":"2024-09-21T13:52:04.315865+0000","last_became_peered":"2024-09-21T13:52:04.315865+0000","last_unstale":"2024-09-21T13:53:29.760365+0000","last_undegraded":"2024-09-21T13:53:29.760365+0000","last_fullsized":"2024-09-21T13:53:29.760365+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:38.983326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:38.983326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"68'10","reported_seq":244,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T13:53:33.824898+0000","last_change":"2024-09-21T13:51:11.545546+0000","last_active":"2024-09-21T13:53:33.824898+0000","last_peered":"2024-09-21T13:53:33.824898+0000","last_clean":"2024-09-21T13:53:33.824898+0000","last_became_active":"2024-09-21T13:49:00.426071+0000","last_became_peered":"2024-09-21T13:49:00.426071+0000","last_unstale":"2024-09-21T13:53:33.824898+0000","last_undegraded":"2024-09-21T13:53:33.824898+0000","last_fullsized":"2024-09-21T13:53:33.824898+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:51:11.545458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:11.545458+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":242,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.766983+0000","last_change":"2024-09-21T13:52:04.318335+0000","last_active":"2024-09-21T13:53:29.766983+0000","last_peered":"2024-09-21T13:53:29.766983+0000","last_clean":"2024-09-21T13:53:29.766983+0000","last_became_active":"2024-09-21T13:52:04.318006+0000","last_became_peered":"2024-09-21T13:52:04.318006+0000","last_unstale":"2024-09-21T13:53:29.766983+0000","last_undegraded":"2024-09-21T13:53:29.766983+0000","last_fullsized":"2024-09-21T13:53:29.766983+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:41.162315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:41.162315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"68'8","reported_seq":279,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.765129+0000","last_change":"2024-09-21T13:49:02.067827+0000","last_active":"2024-09-21T13:53:29.765129+0000","last_peered":"2024-09-21T13:53:29.765129+0000","last_clean":"2024-09-21T13:53:29.765129+0000","last_became_active":"2024-09-21T13:49:02.067487+0000","last_became_peered":"2024-09-21T13:49:02.067487+0000","last_unstale":"2024-09-21T13:53:29.765129+0000","last_undegraded":"2024-09-21T13:53:29.765129+0000","last_fullsized":"2024-09-21T13:53:29.765129+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:02.411463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":164,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.767177+0000","last_change":"2024-09-21T13:50:33.050276+0000","last_active":"2024-09-21T13:53:29.767177+0000","last_peered":"2024-09-21T13:53:29.767177+0000","last_clean":"2024-09-21T13:53:29.767177+0000","last_became_active":"2024-09-21T13:50:29.695082+0000","last_became_peered":"2024-09-21T13:50:29.695082+0000","last_unstale":"2024-09-21T13:53:29.767177+0000","last_undegraded":"2024-09-21T13:53:29.767177+0000","last_fullsized":"2024-09-21T13:53:29.767177+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:33.050227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:33.050227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"68'11","reported_seq":254,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.759999+0000","last_change":"2024-09-21T13:53:26.146937+0000","last_active":"2024-09-21T13:53:29.759999+0000","last_peered":"2024-09-21T13:53:29.759999+0000","last_clean":"2024-09-21T13:53:29.759999+0000","last_became_active":"2024-09-21T13:49:00.055024+0000","last_became_peered":"2024-09-21T13:49:00.055024+0000","last_unstale":"2024-09-21T13:53:29.759999+0000","last_undegraded":"2024-09-21T13:53:29.759999+0000","last_fullsized":"2024-09-21T13:53:29.759999+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:53:26.146725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:26.146725+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:44.146330+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":231,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.760574+0000","last_change":"2024-09-21T13:51:20.956291+0000","last_active":"2024-09-21T13:53:29.760574+0000","last_peered":"2024-09-21T13:53:29.760574+0000","last_clean":"2024-09-21T13:53:29.760574+0000","last_became_active":"2024-09-21T13:48:58.051541+0000","last_became_peered":"2024-09-21T13:48:58.051541+0000","last_unstale":"2024-09-21T13:53:29.760574+0000","last_undegraded":"2024-09-21T13:53:29.760574+0000","last_fullsized":"2024-09-21T13:53:29.760574+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:20.956226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:51:20.956226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":239,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.868061+0000","last_change":"2024-09-21T13:53:29.868061+0000","last_active":"2024-09-21T13:53:29.868061+0000","last_peered":"2024-09-21T13:53:29.868061+0000","last_clean":"2024-09-21T13:53:29.868061+0000","last_became_active":"2024-09-21T13:49:02.062868+0000","last_became_peered":"2024-09-21T13:49:02.062868+0000","last_unstale":"2024-09-21T13:53:29.868061+0000","last_undegraded":"2024-09-21T13:53:29.868061+0000","last_fullsized":"2024-09-21T13:53:29.868061+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:29.867278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:53:29.867278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:49.405014+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":221,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T13:53:33.820048+0000","last_change":"2024-09-21T13:51:24.549993+0000","last_active":"2024-09-21T13:53:33.820048+0000","last_peered":"2024-09-21T13:53:33.820048+0000","last_clean":"2024-09-21T13:53:33.820048+0000","last_became_active":"2024-09-21T13:49:04.082961+0000","last_became_peered":"2024-09-21T13:49:04.082961+0000","last_unstale":"2024-09-21T13:53:33.820048+0000","last_undegraded":"2024-09-21T13:53:33.820048+0000","last_fullsized":"2024-09-21T13:53:33.820048+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:24.549706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:51:24.549706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"68'9","reported_seq":254,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.767838+0000","last_change":"2024-09-21T13:52:10.447789+0000","last_active":"2024-09-21T13:53:29.767838+0000","last_peered":"2024-09-21T13:53:29.767838+0000","last_clean":"2024-09-21T13:53:29.767838+0000","last_became_active":"2024-09-21T13:52:04.320149+0000","last_became_peered":"2024-09-21T13:52:04.320149+0000","last_unstale":"2024-09-21T13:53:29.767838+0000","last_undegraded":"2024-09-21T13:53:29.767838+0000","last_fullsized":"2024-09-21T13:53:29.767838+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:52:10.447530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:52:10.447530+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:37.291172+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":223,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.767151+0000","last_change":"2024-09-21T13:48:58.058032+0000","last_active":"2024-09-21T13:53:29.767151+0000","last_peered":"2024-09-21T13:53:29.767151+0000","last_clean":"2024-09-21T13:53:29.767151+0000","last_became_active":"2024-09-21T13:48:58.057516+0000","last_became_peered":"2024-09-21T13:48:58.057516+0000","last_unstale":"2024-09-21T13:53:29.767151+0000","last_undegraded":"2024-09-21T13:53:29.767151+0000","last_fullsized":"2024-09-21T13:53:29.767151+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:06.571008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":225,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T13:53:33.819224+0000","last_change":"2024-09-21T13:50:39.411785+0000","last_active":"2024-09-21T13:53:33.819224+0000","last_peered":"2024-09-21T13:53:33.819224+0000","last_clean":"2024-09-21T13:53:33.819224+0000","last_became_active":"2024-09-21T13:49:02.071922+0000","last_became_peered":"2024-09-21T13:49:02.071922+0000","last_unstale":"2024-09-21T13:53:33.819224+0000","last_undegraded":"2024-09-21T13:53:33.819224+0000","last_fullsized":"2024-09-21T13:53:33.819224+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:39.411718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:39.411718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":156,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.764388+0000","last_change":"2024-09-21T13:50:29.689537+0000","last_active":"2024-09-21T13:53:29.764388+0000","last_peered":"2024-09-21T13:53:29.764388+0000","last_clean":"2024-09-21T13:53:29.764388+0000","last_became_active":"2024-09-21T13:50:29.689045+0000","last_became_peered":"2024-09-21T13:50:29.689045+0000","last_unstale":"2024-09-21T13:53:29.764388+0000","last_undegraded":"2024-09-21T13:53:29.764388+0000","last_fullsized":"2024-09-21T13:53:29.764388+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:10.347114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"68'11","reported_seq":248,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T13:53:33.821658+0000","last_change":"2024-09-21T13:50:31.329973+0000","last_active":"2024-09-21T13:53:33.821658+0000","last_peered":"2024-09-21T13:53:33.821658+0000","last_clean":"2024-09-21T13:53:33.821658+0000","last_became_active":"2024-09-21T13:49:00.426091+0000","last_became_peered":"2024-09-21T13:49:00.426091+0000","last_unstale":"2024-09-21T13:53:33.821658+0000","last_undegraded":"2024-09-21T13:53:33.821658+0000","last_fullsized":"2024-09-21T13:53:33.821658+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:50:31.329740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:31.329740+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":223,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.765458+0000","last_change":"2024-09-21T13:48:58.052281+0000","last_active":"2024-09-21T13:53:29.765458+0000","last_peered":"2024-09-21T13:53:29.765458+0000","last_clean":"2024-09-21T13:53:29.765458+0000","last_became_active":"2024-09-21T13:48:58.051974+0000","last_became_peered":"2024-09-21T13:48:58.051974+0000","last_unstale":"2024-09-21T13:53:29.765458+0000","last_undegraded":"2024-09-21T13:53:29.765458+0000","last_fullsized":"2024-09-21T13:53:29.765458+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:07.361374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":223,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.767097+0000","last_change":"2024-09-21T13:50:37.118637+0000","last_active":"2024-09-21T13:53:29.767097+0000","last_peered":"2024-09-21T13:53:29.767097+0000","last_clean":"2024-09-21T13:53:29.767097+0000","last_became_active":"2024-09-21T13:49:02.056006+0000","last_became_peered":"2024-09-21T13:49:02.056006+0000","last_unstale":"2024-09-21T13:53:29.767097+0000","last_undegraded":"2024-09-21T13:53:29.767097+0000","last_fullsized":"2024-09-21T13:53:29.767097+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:37.118594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:37.118594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":221,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T13:53:33.819373+0000","last_change":"2024-09-21T13:51:55.827109+0000","last_active":"2024-09-21T13:53:33.819373+0000","last_peered":"2024-09-21T13:53:33.819373+0000","last_clean":"2024-09-21T13:53:33.819373+0000","last_became_active":"2024-09-21T13:49:04.434034+0000","last_became_peered":"2024-09-21T13:49:04.434034+0000","last_unstale":"2024-09-21T13:53:33.819373+0000","last_undegraded":"2024-09-21T13:53:33.819373+0000","last_fullsized":"2024-09-21T13:53:33.819373+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:55.826922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:51:55.826922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"68'4","reported_seq":235,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T13:53:33.821246+0000","last_change":"2024-09-21T13:53:27.826804+0000","last_active":"2024-09-21T13:53:33.821246+0000","last_peered":"2024-09-21T13:53:33.821246+0000","last_clean":"2024-09-21T13:53:33.821246+0000","last_became_active":"2024-09-21T13:49:00.063066+0000","last_became_peered":"2024-09-21T13:49:00.063066+0000","last_unstale":"2024-09-21T13:53:33.821246+0000","last_undegraded":"2024-09-21T13:53:33.821246+0000","last_fullsized":"2024-09-21T13:53:33.821246+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4","last_scrub_stamp":"2024-09-21T13:53:27.826729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:27.826729+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:54.701343+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":164,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:32.253481+0000","last_change":"2024-09-21T13:53:32.253481+0000","last_active":"2024-09-21T13:53:32.253481+0000","last_peered":"2024-09-21T13:53:32.253481+0000","last_clean":"2024-09-21T13:53:32.253481+0000","last_became_active":"2024-09-21T13:50:29.690269+0000","last_became_peered":"2024-09-21T13:50:29.690269+0000","last_unstale":"2024-09-21T13:53:32.253481+0000","last_undegraded":"2024-09-21T13:53:32.253481+0000","last_fullsized":"2024-09-21T13:53:32.253481+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:32.253408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:53:32.253408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:51.887076+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":231,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.763280+0000","last_change":"2024-09-21T13:52:48.472414+0000","last_active":"2024-09-21T13:53:29.763280+0000","last_peered":"2024-09-21T13:53:29.763280+0000","last_clean":"2024-09-21T13:53:29.763280+0000","last_became_active":"2024-09-21T13:49:02.064789+0000","last_became_peered":"2024-09-21T13:49:02.064789+0000","last_unstale":"2024-09-21T13:53:29.763280+0000","last_undegraded":"2024-09-21T13:53:29.763280+0000","last_fullsized":"2024-09-21T13:53:29.763280+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:48.472373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:52:48.472373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:51.057252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":230,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.766274+0000","last_change":"2024-09-21T13:52:04.322055+0000","last_active":"2024-09-21T13:53:29.766274+0000","last_peered":"2024-09-21T13:53:29.766274+0000","last_clean":"2024-09-21T13:53:29.766274+0000","last_became_active":"2024-09-21T13:52:04.321667+0000","last_became_peered":"2024-09-21T13:52:04.321667+0000","last_unstale":"2024-09-21T13:53:29.766274+0000","last_undegraded":"2024-09-21T13:53:29.766274+0000","last_fullsized":"2024-09-21T13:53:29.766274+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:39.151320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:39.151320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":227,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.773343+0000","last_change":"2024-09-21T13:52:27.409588+0000","last_active":"2024-09-21T13:53:29.773343+0000","last_peered":"2024-09-21T13:53:29.773343+0000","last_clean":"2024-09-21T13:53:29.773343+0000","last_became_active":"2024-09-21T13:49:04.075562+0000","last_became_peered":"2024-09-21T13:49:04.075562+0000","last_unstale":"2024-09-21T13:53:29.773343+0000","last_undegraded":"2024-09-21T13:53:29.773343+0000","last_fullsized":"2024-09-21T13:53:29.773343+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:27.409408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:27.409408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:52.946524+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":233,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T13:53:33.821612+0000","last_change":"2024-09-21T13:53:25.873998+0000","last_active":"2024-09-21T13:53:33.821612+0000","last_peered":"2024-09-21T13:53:33.821612+0000","last_clean":"2024-09-21T13:53:33.821612+0000","last_became_active":"2024-09-21T13:48:58.055497+0000","last_became_peered":"2024-09-21T13:48:58.055497+0000","last_unstale":"2024-09-21T13:53:33.821612+0000","last_undegraded":"2024-09-21T13:53:33.821612+0000","last_fullsized":"2024-09-21T13:53:33.821612+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:25.865470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:53:25.865470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:51.548858+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"68'11","reported_seq":277,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.762789+0000","last_change":"2024-09-21T13:52:04.321178+0000","last_active":"2024-09-21T13:53:29.762789+0000","last_peered":"2024-09-21T13:53:29.762789+0000","last_clean":"2024-09-21T13:53:29.762789+0000","last_became_active":"2024-09-21T13:52:04.320647+0000","last_became_peered":"2024-09-21T13:52:04.320647+0000","last_unstale":"2024-09-21T13:53:29.762789+0000","last_undegraded":"2024-09-21T13:53:29.762789+0000","last_fullsized":"2024-09-21T13:53:29.762789+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:50:35.790090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:35.790090+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:07.227949+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":175,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T13:53:29.767105+0000","last_change":"2024-09-21T13:52:04.323345+0000","last_active":"2024-09-21T13:53:29.767105+0000","last_peered":"2024-09-21T13:53:29.767105+0000","last_clean":"2024-09-21T13:53:29.767105+0000","last_became_active":"2024-09-21T13:52:04.322880+0000","last_became_peered":"2024-09-21T13:52:04.322880+0000","last_unstale":"2024-09-21T13:53:29.767105+0000","last_undegraded":"2024-09-21T13:53:29.767105+0000","last_fullsized":"2024-09-21T13:53:29.767105+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:36.086691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:36.086691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":1667891224,"num_objects":25451,"num_object_clones":0,"num_object_copies":76353,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1608,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25451,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":247385,"num_write_kb":8730689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21762,"num_bytes_recovered":1426194432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5118636032,"data_stored":5118623816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87419,"ondisk_log_size":87419,"up":48,"acting":47,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69,"ondisk_log_size":69,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":694,"num_read_kb":451,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":395,"ondisk_log_size":395,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":270,"num_read_kb":265,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234044,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":649728,"kb_used_data":524144,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":93095424,"statfs":{"total":95995035648,"available":95329714176,"internally_reserved":0,"allocated":536723456,"data_stored":536311241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":128580042},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462977,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":629360,"kb_used_data":525984,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93115792,"statfs":{"total":95995035648,"available":95350571008,"internally_reserved":0,"allocated":538607616,"data_stored":538210724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":105839046},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724616,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":541484,"kb_used_data":425628,"kb_used_omap":1,"kb_used_meta":115838,"kb_avail":93203668,"statfs":{"total":95995035648,"available":95440556032,"internally_reserved":0,"allocated":435843072,"data_stored":435450624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":118618565},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920846,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":655816,"kb_used_data":530168,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":93089336,"statfs":{"total":95995035648,"available":95323480064,"internally_reserved":0,"allocated":542892032,"data_stored":542470614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117075,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1145996,"kb_used_data":925884,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":92599156,"statfs":{"total":95995035648,"available":94821535744,"internally_reserved":0,"allocated":948105216,"data_stored":947680532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":225245639},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":130,"seq":558345748500,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":859848,"kb_used_data":665208,"kb_used_omap":4,"kb_used_meta":194619,"kb_avail":92885304,"statfs":{"total":95995035648,"available":95114551296,"internally_reserved":0,"allocated":681172992,"data_stored":680768162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4441,"internal_metadata":199290535},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574944,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":770916,"kb_used_data":645332,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":92974236,"statfs":{"total":95995035648,"available":95205617664,"internally_reserved":0,"allocated":660819968,"data_stored":660370426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738470,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1027908,"kb_used_data":852148,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":92717244,"statfs":{"total":95995035648,"available":94942457856,"internally_reserved":0,"allocated":872599552,"data_stored":872182538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":835653632,"data_stored":835649560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":638652416,"data_stored":638648344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":664862720,"data_stored":664862720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":941228032,"data_stored":941228032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":538378240,"data_stored":538378240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":432275456,"data_stored":432275456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":534908928,"data_stored":534904856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532676608,"data_stored":532676608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T13:53:40.439 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-21T13:53:40.439 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T13:53:40.439 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-21T13:53:40.440 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd dump -f json-pretty 2024-09-21T13:53:40.512 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:53:40.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:40.537 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:40.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:53:40.637 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:53:40.709 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:40.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:40.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:53:40.753 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:53:40 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:53:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:53:40.811 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:53:40.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:40.869 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:40.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:53:40.970 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:40.993 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:40.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:40.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:41.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:40 smithi049 ceph-mon[40489]: pgmap v223: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 147 active+clean; 1.3 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.28k op/s; 1097/63387 objects degraded (1.731%); 1.7 MiB/s, 26 objects/s recovering 2024-09-21T13:53:41.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:40 smithi049 ceph-mon[40489]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-21T13:53:41.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:40 smithi049 ceph-mon[30080]: pgmap v223: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 147 active+clean; 1.3 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.28k op/s; 1097/63387 objects degraded (1.731%); 1.7 MiB/s, 26 objects/s recovering 2024-09-21T13:53:41.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:40 smithi049 ceph-mon[30080]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-21T13:53:41.095 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:53:41.121 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:41.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:41.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:53:41.136 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:53:41.137 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:53:41.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:40 smithi187 ceph-mon[33032]: pgmap v223: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 147 active+clean; 1.3 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.28k op/s; 1097/63387 objects degraded (1.731%); 1.7 MiB/s, 26 objects/s recovering 2024-09-21T13:53:41.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:40 smithi187 ceph-mon[33032]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-21T13:53:41.223 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:53:41.271 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:41.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:41.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:53:41.373 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:53:41.398 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:41.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:41.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:53:41.499 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:53:41.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:41.541 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:41.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:53:41.642 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:53:41.684 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:41.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:41.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:53:41.786 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:53:41.814 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:41.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:41.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:53:41.916 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:53:41.929 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-09-21T13:53:41.973 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:41.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:41.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:53:42.074 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:53:42.113 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:42.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:42.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:53:42.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:41 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/239009246' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:53:42.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:41 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:53:42.215 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:53:42.240 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:42.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:42.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:53:42.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:41 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/239009246' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:53:42.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:41 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:53:42.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:41 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/239009246' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:53:42.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:41 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:53:42.341 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:53:42.380 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:42.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:42.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:53:42.481 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:53:42.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:42.522 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:42.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:53:42.623 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:53:42.661 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:42.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:42.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:53:42.762 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:53:42.788 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:42.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:42.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:53:42.890 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:53:42.919 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:42.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:42.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:53:43.020 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:53:43.050 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:43.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:43.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:53:43.152 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:53:43.177 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:43.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:43.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:53:43.190 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:42 smithi049 ceph-mon[40489]: pgmap v224: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 147 active+clean; 1.3 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 801 op/s; 1097/63387 objects degraded (1.731%); 1.7 MiB/s, 26 objects/s recovering 2024-09-21T13:53:43.190 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:43 smithi049 ceph-mon[40489]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:53:43.190 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:43 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T13:53:43.190 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:43 smithi049 ceph-mon[40489]: osdmap e169: 8 total, 8 up, 8 in 2024-09-21T13:53:43.191 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:42 smithi049 ceph-mon[30080]: pgmap v224: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 147 active+clean; 1.3 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 801 op/s; 1097/63387 objects degraded (1.731%); 1.7 MiB/s, 26 objects/s recovering 2024-09-21T13:53:43.191 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:43 smithi049 ceph-mon[30080]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:53:43.191 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:43 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T13:53:43.191 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:43 smithi049 ceph-mon[30080]: osdmap e169: 8 total, 8 up, 8 in 2024-09-21T13:53:43.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:42 smithi187 ceph-mon[33032]: pgmap v224: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 147 active+clean; 1.3 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 801 op/s; 1097/63387 objects degraded (1.731%); 1.7 MiB/s, 26 objects/s recovering 2024-09-21T13:53:43.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:42 smithi187 ceph-mon[33032]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:53:43.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:42 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T13:53:43.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:42 smithi187 ceph-mon[33032]: osdmap e169: 8 total, 8 up, 8 in 2024-09-21T13:53:43.278 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:53:43.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:43.320 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:43.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:53:43.421 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:53:43.448 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:43.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:43.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:53:43.550 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:53:43.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:43.594 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:43.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:53:43.695 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:53:43.717 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:43.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:43.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:53:43.819 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:53:43.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:43.858 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:43.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:53:43.958 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:53:43.985 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:43.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:43.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:53:44.086 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:53:44.123 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:44.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:44.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:53:44.225 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:53:44.266 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:44.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:44.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:53:44.368 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:53:44.393 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:44.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:44.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:53:44.405 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:44 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:53:44.405 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:44 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:53:44.405 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:44 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:53:44.406 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:44 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:53:44.495 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:53:44.520 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:44.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:44.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:53:44.622 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:53:44.645 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:44.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:44.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:53:44.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:44 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:53:44.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:44 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:53:44.746 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:44.771 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:44.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:44.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:44.873 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:53:44.915 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:44.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:44.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:53:45.017 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:53:45.059 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:45.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:45.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:53:45.114 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:53:45.114 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:53:45.114 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:53:45.114 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:53:45.114 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:53:45.115 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:53:45.115 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:53:45.115 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:53:45.115 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:53:45.115 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:53:45.115 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:53:45.115 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:53:45.115 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:53:45.115 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:53:45.115 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:53:45.115 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:53:45.161 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:45.186 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:45.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:45.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:45.287 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:45.311 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:45.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:45.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:45.413 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:53:45.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:45.455 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:45.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:53:45.459 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:45 smithi049 ceph-mon[40489]: pgmap v226: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 147 active+clean; 1.1 GiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1k op/s; 1097/52302 objects degraded (2.097%); 1.5 MiB/s, 24 objects/s recovering 2024-09-21T13:53:45.459 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:45 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2829527405' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:53:45.459 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:45 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 1097/52302 objects degraded (2.097%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T13:53:45.459 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:45 smithi049 ceph-mon[40489]: osdmap e170: 8 total, 8 up, 8 in 2024-09-21T13:53:45.459 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:45 smithi049 ceph-mon[30080]: pgmap v226: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 147 active+clean; 1.1 GiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1k op/s; 1097/52302 objects degraded (2.097%); 1.5 MiB/s, 24 objects/s recovering 2024-09-21T13:53:45.459 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:45 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2829527405' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:53:45.459 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:45 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 1097/52302 objects degraded (2.097%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T13:53:45.459 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:45 smithi049 ceph-mon[30080]: osdmap e170: 8 total, 8 up, 8 in 2024-09-21T13:53:45.557 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:53:45.579 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:45.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:45.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:53:45.595 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:45 smithi187 ceph-mon[33032]: pgmap v226: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 147 active+clean; 1.1 GiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1k op/s; 1097/52302 objects degraded (2.097%); 1.5 MiB/s, 24 objects/s recovering 2024-09-21T13:53:45.595 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:45 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2829527405' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:53:45.595 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:45 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 1097/52302 objects degraded (2.097%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T13:53:45.595 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:45 smithi187 ceph-mon[33032]: osdmap e170: 8 total, 8 up, 8 in 2024-09-21T13:53:45.672 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:53:45.681 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:53:45.704 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:45.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:45.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:53:45.705 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T13:53:45.705 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:53:45.805 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:53:45.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:45.846 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:45.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:53:45.947 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:53:45.998 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:45.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:45.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:53:46.100 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:53:46.137 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:46.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:46.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:53:46.238 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:53:46.279 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:46.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:46.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:53:46.381 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:53:46.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:46.424 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:46.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:53:46.526 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:53:46.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:46.553 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:46.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:53:46.595 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T13:53:46.654 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:53:46.705 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:46.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:46.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:53:46.807 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:53:46.852 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:46.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:46.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:53:46.954 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:53:47.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:47.003 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:47.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:53:47.104 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:53:47.129 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:47.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:47.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:53:47.231 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:53:47.270 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:47.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:47.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:53:47.372 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:53:47.396 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:47.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:47.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:53:47.460 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T13:53:47.498 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:47.500 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:47 smithi049 ceph-mon[40489]: pgmap v228: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 147 active+clean; 1001 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.18k op/s; 606/48642 objects degraded (1.246%); 3.8 MiB/s, 61 objects/s recovering 2024-09-21T13:53:47.501 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:47 smithi049 ceph-mon[30080]: pgmap v228: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 147 active+clean; 1001 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.18k op/s; 606/48642 objects degraded (1.246%); 3.8 MiB/s, 61 objects/s recovering 2024-09-21T13:53:47.524 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:47.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:47.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:47.625 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:53:47.667 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:47.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:47.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:53:47.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:47 smithi187 ceph-mon[33032]: pgmap v228: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 147 active+clean; 1001 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.18k op/s; 606/48642 objects degraded (1.246%); 3.8 MiB/s, 61 objects/s recovering 2024-09-21T13:53:47.769 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:53:47.805 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:47.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:47.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:53:47.907 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:53:47.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:47.934 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:47.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:53:48.035 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:53:48.070 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:48.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:48.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:53:48.172 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:53:48.210 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:48.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:48.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:53:48.312 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:53:48.336 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:48.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:48.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:53:48.438 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:53:48.481 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:48.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:48.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:53:48.583 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:53:48.607 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:48.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:48.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:53:48.665 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:53:48.669 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:53:48.708 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:53:48.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:48.783 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:48.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:53:48.883 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:53:48.908 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:48.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:48.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:53:49.010 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:53:49.034 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:53:49.034 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T13:53:49.034 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 170, 2024-09-21T13:53:49.034 INFO:teuthology.orchestra.run.smithi049.stdout: "fsid": "40a3d2e8-781f-11ef-baf6-efdc52797490", 2024-09-21T13:53:49.034 INFO:teuthology.orchestra.run.smithi049.stdout: "created": "2024-09-21T13:42:57.838275+0000", 2024-09-21T13:53:49.034 INFO:teuthology.orchestra.run.smithi049.stdout: "modified": "2024-09-21T13:53:44.901484+0000", 2024-09-21T13:53:49.034 INFO:teuthology.orchestra.run.smithi049.stdout: "last_up_change": "2024-09-21T13:52:03.297364+0000", 2024-09-21T13:53:49.034 INFO:teuthology.orchestra.run.smithi049.stdout: "last_in_change": "2024-09-21T13:48:23.514355+0000", 2024-09-21T13:53:49.035 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-21T13:53:49.035 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_num": 5806080, 2024-09-21T13:53:49.035 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_set": [ 2024-09-21T13:53:49.035 INFO:teuthology.orchestra.run.smithi049.stdout: "nodeep-scrub", 2024-09-21T13:53:49.035 INFO:teuthology.orchestra.run.smithi049.stdout: "noscrub", 2024-09-21T13:53:49.035 INFO:teuthology.orchestra.run.smithi049.stdout: "pglog_hardlimit", 2024-09-21T13:53:49.035 INFO:teuthology.orchestra.run.smithi049.stdout: "purged_snapdirs", 2024-09-21T13:53:49.035 INFO:teuthology.orchestra.run.smithi049.stdout: "recovery_deletes", 2024-09-21T13:53:49.035 INFO:teuthology.orchestra.run.smithi049.stdout: "sortbitwise" 2024-09-21T13:53:49.035 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T13:53:49.035 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_version": 19, 2024-09-21T13:53:49.035 INFO:teuthology.orchestra.run.smithi049.stdout: "full_ratio": 0.94999998807907104, 2024-09-21T13:53:49.035 INFO:teuthology.orchestra.run.smithi049.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-21T13:53:49.035 INFO:teuthology.orchestra.run.smithi049.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-21T13:53:49.035 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_snapshot": "", 2024-09-21T13:53:49.036 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_max": 7, 2024-09-21T13:53:49.036 INFO:teuthology.orchestra.run.smithi049.stdout: "max_osd": 8, 2024-09-21T13:53:49.036 INFO:teuthology.orchestra.run.smithi049.stdout: "require_min_compat_client": "luminous", 2024-09-21T13:53:49.036 INFO:teuthology.orchestra.run.smithi049.stdout: "min_compat_client": "luminous", 2024-09-21T13:53:49.036 INFO:teuthology.orchestra.run.smithi049.stdout: "require_osd_release": "squid", 2024-09-21T13:53:49.036 INFO:teuthology.orchestra.run.smithi049.stdout: "allow_crimson": false, 2024-09-21T13:53:49.036 INFO:teuthology.orchestra.run.smithi049.stdout: "pools": [ 2024-09-21T13:53:49.036 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.036 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 1, 2024-09-21T13:53:49.036 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": ".mgr", 2024-09-21T13:53:49.036 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:46:21.096244+0000", 2024-09-21T13:53:49.036 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T13:53:49.036 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T13:53:49.036 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T13:53:49.036 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T13:53:49.037 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T13:53:49.037 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T13:53:49.037 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T13:53:49.037 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T13:53:49.037 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T13:53:49.037 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T13:53:49.037 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T13:53:49.037 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T13:53:49.037 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T13:53:49.037 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 1, 2024-09-21T13:53:49.037 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 1, 2024-09-21T13:53:49.037 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 1, 2024-09-21T13:53:49.037 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 1, 2024-09-21T13:53:49.037 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 1, 2024-09-21T13:53:49.038 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T13:53:49.038 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T13:53:49.038 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T13:53:49.038 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T13:53:49.038 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T13:53:49.038 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T13:53:49.038 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T13:53:49.038 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.038 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "22", 2024-09-21T13:53:49.038 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T13:53:49.038 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T13:53:49.038 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T13:53:49.038 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T13:53:49.038 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T13:53:49.039 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T13:53:49.039 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T13:53:49.039 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T13:53:49.039 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T13:53:49.039 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T13:53:49.039 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T13:53:49.039 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T13:53:49.039 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T13:53:49.039 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T13:53:49.039 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T13:53:49.039 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T13:53:49.039 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T13:53:49.039 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T13:53:49.040 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T13:53:49.040 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T13:53:49.040 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T13:53:49.040 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T13:53:49.040 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T13:53:49.040 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T13:53:49.040 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T13:53:49.040 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T13:53:49.040 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.040 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T13:53:49.040 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T13:53:49.040 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T13:53:49.040 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T13:53:49.040 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T13:53:49.041 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T13:53:49.041 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T13:53:49.041 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T13:53:49.041 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T13:53:49.041 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T13:53:49.041 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T13:53:49.041 INFO:teuthology.orchestra.run.smithi049.stdout: "options": { 2024-09-21T13:53:49.041 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_max": 32, 2024-09-21T13:53:49.041 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_min": 1 2024-09-21T13:53:49.041 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.041 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T13:53:49.041 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr": {} 2024-09-21T13:53:49.041 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.041 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T13:53:49.041 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T13:53:49.042 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 7.8899998664855957, 2024-09-21T13:53:49.042 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 7.8899998664855957, 2024-09-21T13:53:49.042 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 0.37999999523162842, 2024-09-21T13:53:49.042 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 3, 2024-09-21T13:53:49.042 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 3, 2024-09-21T13:53:49.042 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.86855697631835938, 2024-09-21T13:53:49.042 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.87000000476837158, 2024-09-21T13:53:49.042 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.87000000476837158 2024-09-21T13:53:49.042 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:53:49.042 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.042 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.042 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 2, 2024-09-21T13:53:49.042 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "datapool", 2024-09-21T13:53:49.043 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:48:55.540712+0000", 2024-09-21T13:53:49.043 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 8193, 2024-09-21T13:53:49.043 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T13:53:49.043 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T13:53:49.043 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T13:53:49.043 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T13:53:49.043 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T13:53:49.043 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T13:53:49.043 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T13:53:49.043 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T13:53:49.043 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T13:53:49.043 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T13:53:49.043 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T13:53:49.043 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T13:53:49.043 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 3, 2024-09-21T13:53:49.044 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 3, 2024-09-21T13:53:49.044 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 3, 2024-09-21T13:53:49.044 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 3, 2024-09-21T13:53:49.044 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 3, 2024-09-21T13:53:49.044 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T13:53:49.044 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T13:53:49.044 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T13:53:49.044 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T13:53:49.044 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T13:53:49.044 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T13:53:49.044 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T13:53:49.044 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.044 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "63", 2024-09-21T13:53:49.044 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T13:53:49.044 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T13:53:49.045 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T13:53:49.045 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T13:53:49.045 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T13:53:49.045 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 2, 2024-09-21T13:53:49.045 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 63, 2024-09-21T13:53:49.045 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T13:53:49.045 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T13:53:49.045 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T13:53:49.045 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T13:53:49.045 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T13:53:49.045 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T13:53:49.045 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T13:53:49.045 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T13:53:49.045 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T13:53:49.045 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T13:53:49.046 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T13:53:49.046 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T13:53:49.046 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T13:53:49.046 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T13:53:49.046 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T13:53:49.046 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T13:53:49.046 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T13:53:49.046 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T13:53:49.046 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T13:53:49.046 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.046 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T13:53:49.046 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T13:53:49.046 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T13:53:49.046 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T13:53:49.047 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T13:53:49.047 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T13:53:49.047 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T13:53:49.047 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T13:53:49.047 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T13:53:49.047 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T13:53:49.047 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T13:53:49.047 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T13:53:49.047 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T13:53:49.047 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd": {} 2024-09-21T13:53:49.047 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.047 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T13:53:49.047 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T13:53:49.047 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2.6500000953674316, 2024-09-21T13:53:49.048 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2.6500000953674316, 2024-09-21T13:53:49.048 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 0.87999999523162842, 2024-09-21T13:53:49.048 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2.3299999237060547, 2024-09-21T13:53:49.048 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2.3299999237060547, 2024-09-21T13:53:49.048 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.86855697631835938, 2024-09-21T13:53:49.048 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.87000000476837158, 2024-09-21T13:53:49.048 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.87000000476837158 2024-09-21T13:53:49.048 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:53:49.048 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.048 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.048 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 3, 2024-09-21T13:53:49.048 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": ".rgw.root", 2024-09-21T13:53:49.048 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:48:56.417456+0000", 2024-09-21T13:53:49.048 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T13:53:49.048 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T13:53:49.049 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T13:53:49.049 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T13:53:49.049 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T13:53:49.049 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T13:53:49.049 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T13:53:49.049 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T13:53:49.049 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T13:53:49.049 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T13:53:49.049 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T13:53:49.049 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T13:53:49.049 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T13:53:49.049 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 32, 2024-09-21T13:53:49.049 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 32, 2024-09-21T13:53:49.049 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 32, 2024-09-21T13:53:49.050 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 32, 2024-09-21T13:53:49.050 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 32, 2024-09-21T13:53:49.050 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T13:53:49.050 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T13:53:49.050 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T13:53:49.050 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T13:53:49.050 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T13:53:49.050 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T13:53:49.050 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T13:53:49.050 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.050 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "62", 2024-09-21T13:53:49.050 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T13:53:49.050 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T13:53:49.050 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T13:53:49.051 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T13:53:49.051 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T13:53:49.051 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T13:53:49.051 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T13:53:49.051 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T13:53:49.051 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T13:53:49.051 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T13:53:49.051 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T13:53:49.051 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T13:53:49.051 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T13:53:49.051 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T13:53:49.051 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T13:53:49.051 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T13:53:49.051 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T13:53:49.051 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T13:53:49.052 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T13:53:49.052 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T13:53:49.052 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T13:53:49.052 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T13:53:49.052 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T13:53:49.052 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T13:53:49.052 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T13:53:49.052 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T13:53:49.052 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.052 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T13:53:49.052 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T13:53:49.052 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T13:53:49.052 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T13:53:49.052 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T13:53:49.053 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T13:53:49.053 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T13:53:49.053 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T13:53:49.053 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T13:53:49.053 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T13:53:49.053 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T13:53:49.053 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T13:53:49.053 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T13:53:49.053 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw": {} 2024-09-21T13:53:49.053 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.053 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T13:53:49.053 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T13:53:49.053 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 1.75, 2024-09-21T13:53:49.053 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 1.75, 2024-09-21T13:53:49.054 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1, 2024-09-21T13:53:49.054 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 1.75, 2024-09-21T13:53:49.054 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 1.75, 2024-09-21T13:53:49.054 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.86855697631835938, 2024-09-21T13:53:49.054 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.87000000476837158, 2024-09-21T13:53:49.054 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.87000000476837158 2024-09-21T13:53:49.054 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:53:49.054 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.054 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.054 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 4, 2024-09-21T13:53:49.054 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "default.rgw.log", 2024-09-21T13:53:49.054 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:48:58.128455+0000", 2024-09-21T13:53:49.054 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T13:53:49.054 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T13:53:49.055 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T13:53:49.055 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T13:53:49.055 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T13:53:49.055 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T13:53:49.055 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T13:53:49.055 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T13:53:49.055 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T13:53:49.055 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T13:53:49.055 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T13:53:49.055 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T13:53:49.055 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T13:53:49.055 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 32, 2024-09-21T13:53:49.055 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 32, 2024-09-21T13:53:49.055 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 32, 2024-09-21T13:53:49.055 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 32, 2024-09-21T13:53:49.056 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 32, 2024-09-21T13:53:49.056 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T13:53:49.056 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T13:53:49.056 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T13:53:49.056 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T13:53:49.056 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T13:53:49.056 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T13:53:49.056 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T13:53:49.056 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.056 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "64", 2024-09-21T13:53:49.056 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T13:53:49.056 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T13:53:49.056 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T13:53:49.056 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T13:53:49.057 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T13:53:49.057 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T13:53:49.057 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T13:53:49.057 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T13:53:49.057 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T13:53:49.057 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T13:53:49.057 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T13:53:49.057 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T13:53:49.057 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T13:53:49.057 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T13:53:49.057 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T13:53:49.057 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T13:53:49.057 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T13:53:49.057 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T13:53:49.057 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T13:53:49.058 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T13:53:49.058 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T13:53:49.058 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T13:53:49.058 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T13:53:49.058 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T13:53:49.058 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T13:53:49.058 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T13:53:49.058 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.058 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T13:53:49.058 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T13:53:49.058 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T13:53:49.058 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T13:53:49.058 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T13:53:49.059 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T13:53:49.059 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T13:53:49.059 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T13:53:49.059 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T13:53:49.059 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T13:53:49.059 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T13:53:49.059 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T13:53:49.059 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T13:53:49.059 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw": {} 2024-09-21T13:53:49.059 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.059 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T13:53:49.059 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T13:53:49.059 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2.25, 2024-09-21T13:53:49.059 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2.25, 2024-09-21T13:53:49.059 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1, 2024-09-21T13:53:49.060 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2.25, 2024-09-21T13:53:49.060 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2.25, 2024-09-21T13:53:49.060 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.86855697631835938, 2024-09-21T13:53:49.060 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.87000000476837158, 2024-09-21T13:53:49.060 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.87000000476837158 2024-09-21T13:53:49.060 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:53:49.060 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.060 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.060 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 5, 2024-09-21T13:53:49.060 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "default.rgw.control", 2024-09-21T13:53:49.060 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:49:00.069361+0000", 2024-09-21T13:53:49.060 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T13:53:49.060 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T13:53:49.060 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T13:53:49.061 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T13:53:49.061 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T13:53:49.061 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T13:53:49.061 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T13:53:49.061 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T13:53:49.061 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T13:53:49.061 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T13:53:49.061 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T13:53:49.061 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T13:53:49.061 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T13:53:49.061 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 32, 2024-09-21T13:53:49.061 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 32, 2024-09-21T13:53:49.061 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 32, 2024-09-21T13:53:49.061 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 32, 2024-09-21T13:53:49.061 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 32, 2024-09-21T13:53:49.062 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T13:53:49.062 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T13:53:49.062 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T13:53:49.062 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T13:53:49.062 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T13:53:49.062 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T13:53:49.062 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T13:53:49.062 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.062 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "66", 2024-09-21T13:53:49.062 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T13:53:49.062 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T13:53:49.062 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T13:53:49.062 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T13:53:49.063 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T13:53:49.063 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T13:53:49.063 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T13:53:49.063 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T13:53:49.063 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T13:53:49.063 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T13:53:49.063 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T13:53:49.063 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T13:53:49.063 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T13:53:49.063 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T13:53:49.063 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T13:53:49.063 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T13:53:49.063 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T13:53:49.063 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T13:53:49.063 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T13:53:49.064 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T13:53:49.064 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T13:53:49.064 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T13:53:49.064 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T13:53:49.064 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T13:53:49.064 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T13:53:49.064 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T13:53:49.064 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.064 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T13:53:49.064 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T13:53:49.064 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T13:53:49.064 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T13:53:49.064 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T13:53:49.064 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T13:53:49.064 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T13:53:49.065 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T13:53:49.065 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T13:53:49.065 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T13:53:49.065 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T13:53:49.065 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T13:53:49.065 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T13:53:49.065 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw": {} 2024-09-21T13:53:49.065 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.065 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T13:53:49.065 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T13:53:49.065 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 1.25, 2024-09-21T13:53:49.065 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 1.25, 2024-09-21T13:53:49.065 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1, 2024-09-21T13:53:49.066 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 1.25, 2024-09-21T13:53:49.066 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 1.25, 2024-09-21T13:53:49.066 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.86855697631835938, 2024-09-21T13:53:49.066 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.87000000476837158, 2024-09-21T13:53:49.066 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.87000000476837158 2024-09-21T13:53:49.066 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:53:49.066 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.066 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.066 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 6, 2024-09-21T13:53:49.067 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "default.rgw.meta", 2024-09-21T13:53:49.067 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:49:02.127399+0000", 2024-09-21T13:53:49.067 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T13:53:49.067 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T13:53:49.067 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T13:53:49.067 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T13:53:49.067 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T13:53:49.067 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T13:53:49.067 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T13:53:49.067 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T13:53:49.067 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T13:53:49.067 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T13:53:49.067 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T13:53:49.067 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T13:53:49.068 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T13:53:49.068 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 32, 2024-09-21T13:53:49.068 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 32, 2024-09-21T13:53:49.068 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 32, 2024-09-21T13:53:49.068 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 32, 2024-09-21T13:53:49.068 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 32, 2024-09-21T13:53:49.068 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T13:53:49.068 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T13:53:49.068 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T13:53:49.068 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T13:53:49.068 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T13:53:49.068 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T13:53:49.068 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T13:53:49.068 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.068 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "68", 2024-09-21T13:53:49.069 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T13:53:49.069 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T13:53:49.069 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T13:53:49.069 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T13:53:49.069 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T13:53:49.069 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T13:53:49.069 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T13:53:49.069 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T13:53:49.069 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T13:53:49.069 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T13:53:49.069 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T13:53:49.069 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T13:53:49.069 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T13:53:49.069 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T13:53:49.070 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T13:53:49.070 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T13:53:49.070 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T13:53:49.070 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T13:53:49.070 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T13:53:49.070 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T13:53:49.070 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T13:53:49.070 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T13:53:49.070 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T13:53:49.070 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T13:53:49.070 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T13:53:49.070 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T13:53:49.070 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.071 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T13:53:49.071 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T13:53:49.071 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T13:53:49.071 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T13:53:49.071 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T13:53:49.071 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T13:53:49.071 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T13:53:49.071 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T13:53:49.071 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T13:53:49.071 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T13:53:49.071 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T13:53:49.071 INFO:teuthology.orchestra.run.smithi049.stdout: "options": { 2024-09-21T13:53:49.071 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_bias": 4 2024-09-21T13:53:49.071 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.072 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T13:53:49.072 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw": {} 2024-09-21T13:53:49.072 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.072 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T13:53:49.072 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T13:53:49.072 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2, 2024-09-21T13:53:49.072 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2, 2024-09-21T13:53:49.072 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1, 2024-09-21T13:53:49.072 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2, 2024-09-21T13:53:49.072 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2, 2024-09-21T13:53:49.072 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.86855697631835938, 2024-09-21T13:53:49.072 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.87000000476837158, 2024-09-21T13:53:49.072 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.87000000476837158 2024-09-21T13:53:49.072 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:53:49.073 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.073 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.073 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 7, 2024-09-21T13:53:49.073 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "unique_pool_0", 2024-09-21T13:53:49.073 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:50:05.454919+0000", 2024-09-21T13:53:49.073 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T13:53:49.073 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T13:53:49.073 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T13:53:49.073 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T13:53:49.073 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T13:53:49.073 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T13:53:49.073 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T13:53:49.073 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T13:53:49.073 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T13:53:49.074 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T13:53:49.074 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T13:53:49.074 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T13:53:49.074 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T13:53:49.074 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 16, 2024-09-21T13:53:49.074 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 16, 2024-09-21T13:53:49.074 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 16, 2024-09-21T13:53:49.074 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 16, 2024-09-21T13:53:49.074 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 16, 2024-09-21T13:53:49.074 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T13:53:49.074 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T13:53:49.074 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T13:53:49.074 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T13:53:49.074 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T13:53:49.075 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T13:53:49.075 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T13:53:49.075 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.075 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "86", 2024-09-21T13:53:49.075 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T13:53:49.075 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T13:53:49.075 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T13:53:49.075 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T13:53:49.075 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T13:53:49.075 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T13:53:49.075 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T13:53:49.075 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T13:53:49.075 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T13:53:49.075 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T13:53:49.075 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T13:53:49.076 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T13:53:49.076 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T13:53:49.076 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T13:53:49.076 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T13:53:49.076 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T13:53:49.076 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T13:53:49.076 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T13:53:49.076 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T13:53:49.076 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T13:53:49.076 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T13:53:49.076 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T13:53:49.076 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T13:53:49.076 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T13:53:49.076 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T13:53:49.077 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T13:53:49.077 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.077 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T13:53:49.077 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T13:53:49.077 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T13:53:49.077 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T13:53:49.077 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T13:53:49.077 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T13:53:49.077 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T13:53:49.077 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T13:53:49.077 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T13:53:49.077 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T13:53:49.077 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T13:53:49.077 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T13:53:49.078 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T13:53:49.078 INFO:teuthology.orchestra.run.smithi049.stdout: "rados": {} 2024-09-21T13:53:49.078 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.078 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T13:53:49.078 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T13:53:49.078 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2, 2024-09-21T13:53:49.078 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2, 2024-09-21T13:53:49.078 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1, 2024-09-21T13:53:49.078 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2, 2024-09-21T13:53:49.078 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2, 2024-09-21T13:53:49.078 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.86855697631835938, 2024-09-21T13:53:49.078 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.87000000476837158, 2024-09-21T13:53:49.078 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.87000000476837158 2024-09-21T13:53:49.078 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:53:49.078 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:53:49.079 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T13:53:49.079 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-09-21T13:53:49.079 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.079 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 0, 2024-09-21T13:53:49.079 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "2595dca9-6219-4e0c-8728-751eb102e73a", 2024-09-21T13:53:49.079 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T13:53:49.079 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T13:53:49.079 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T13:53:49.079 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 0.1215972900390625, 2024-09-21T13:53:49.079 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T13:53:49.079 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T13:53:49.079 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 8, 2024-09-21T13:53:49.079 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 159, 2024-09-21T13:53:49.079 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T13:53:49.079 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T13:53:49.080 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T13:53:49.080 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:53:49.080 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.080 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:53:49.080 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6802", 2024-09-21T13:53:49.080 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T13:53:49.080 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.080 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.080 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:53:49.080 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6803", 2024-09-21T13:53:49.080 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T13:53:49.080 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:53:49.080 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:53:49.080 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.080 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T13:53:49.081 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:53:49.081 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.081 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:53:49.081 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6804", 2024-09-21T13:53:49.081 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T13:53:49.081 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.081 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.081 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:53:49.081 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6805", 2024-09-21T13:53:49.081 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T13:53:49.081 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:53:49.081 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:53:49.081 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.081 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T13:53:49.081 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:53:49.082 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.082 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:53:49.082 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6808", 2024-09-21T13:53:49.082 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T13:53:49.082 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.082 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.082 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:53:49.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:49.083 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:49.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:53:49.084 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6809", 2024-09-21T13:53:49.084 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T13:53:49.084 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:53:49.084 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:53:49.084 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.084 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T13:53:49.084 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:53:49.084 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.084 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:53:49.085 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6806", 2024-09-21T13:53:49.085 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T13:53:49.085 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.085 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.085 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:53:49.085 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6807", 2024-09-21T13:53:49.085 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T13:53:49.085 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:53:49.085 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:53:49.085 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.085 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6803/2095561412", 2024-09-21T13:53:49.085 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6805/2095561412", 2024-09-21T13:53:49.085 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6809/2095561412", 2024-09-21T13:53:49.085 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6807/2095561412", 2024-09-21T13:53:49.085 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T13:53:49.086 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T13:53:49.086 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T13:53:49.086 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:53:49.086 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.086 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.086 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 1, 2024-09-21T13:53:49.086 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "0ba327f9-5524-4a40-abcc-729ded0b1917", 2024-09-21T13:53:49.086 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T13:53:49.086 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T13:53:49.086 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T13:53:49.086 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T13:53:49.086 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T13:53:49.086 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T13:53:49.087 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 13, 2024-09-21T13:53:49.087 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 130, 2024-09-21T13:53:49.087 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T13:53:49.087 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T13:53:49.087 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T13:53:49.087 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:53:49.087 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.087 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:53:49.087 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6810", 2024-09-21T13:53:49.087 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T13:53:49.087 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.087 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.087 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:53:49.087 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6811", 2024-09-21T13:53:49.087 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T13:53:49.088 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:53:49.088 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:53:49.088 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.088 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T13:53:49.088 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:53:49.088 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.088 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:53:49.088 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6812", 2024-09-21T13:53:49.088 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T13:53:49.088 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.088 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.088 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:53:49.088 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6813", 2024-09-21T13:53:49.088 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T13:53:49.089 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:53:49.089 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:53:49.089 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.089 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T13:53:49.089 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:53:49.089 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.089 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:53:49.089 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6816", 2024-09-21T13:53:49.089 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T13:53:49.089 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.089 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.089 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:53:49.089 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6817", 2024-09-21T13:53:49.089 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T13:53:49.089 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:53:49.090 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:53:49.090 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.090 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T13:53:49.090 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:53:49.090 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.090 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:53:49.090 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6814", 2024-09-21T13:53:49.090 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T13:53:49.090 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.090 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.090 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:53:49.090 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6815", 2024-09-21T13:53:49.090 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T13:53:49.090 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:53:49.091 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:53:49.091 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.091 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6811/1382840527", 2024-09-21T13:53:49.091 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6813/1382840527", 2024-09-21T13:53:49.091 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6817/1382840527", 2024-09-21T13:53:49.091 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6815/1382840527", 2024-09-21T13:53:49.091 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T13:53:49.091 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T13:53:49.091 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T13:53:49.091 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:53:49.091 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.091 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.091 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 2, 2024-09-21T13:53:49.091 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "27a45750-204d-48fd-bb8f-ebb01273343d", 2024-09-21T13:53:49.092 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T13:53:49.092 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T13:53:49.092 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T13:53:49.092 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 0.8268585205078125, 2024-09-21T13:53:49.092 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 18, 2024-09-21T13:53:49.092 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 100, 2024-09-21T13:53:49.092 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 130, 2024-09-21T13:53:49.092 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 160, 2024-09-21T13:53:49.092 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 101, 2024-09-21T13:53:49.092 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T13:53:49.092 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T13:53:49.092 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:53:49.092 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.092 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:53:49.092 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6818", 2024-09-21T13:53:49.093 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T13:53:49.093 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.093 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.093 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:53:49.093 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6819", 2024-09-21T13:53:49.093 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T13:53:49.093 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:53:49.093 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:53:49.093 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.093 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T13:53:49.093 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:53:49.093 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.093 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:53:49.093 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6820", 2024-09-21T13:53:49.094 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T13:53:49.094 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.094 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.094 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:53:49.094 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6821", 2024-09-21T13:53:49.094 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T13:53:49.094 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:53:49.094 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:53:49.094 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.094 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T13:53:49.094 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:53:49.094 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.094 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:53:49.094 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6824", 2024-09-21T13:53:49.094 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T13:53:49.095 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.095 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.095 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:53:49.095 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6825", 2024-09-21T13:53:49.095 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T13:53:49.095 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:53:49.095 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:53:49.095 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.095 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T13:53:49.095 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:53:49.095 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.095 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:53:49.095 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6822", 2024-09-21T13:53:49.095 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T13:53:49.096 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.096 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.096 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:53:49.096 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6823", 2024-09-21T13:53:49.096 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T13:53:49.096 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:53:49.096 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:53:49.096 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.096 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6819/1945314741", 2024-09-21T13:53:49.096 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6821/1945314741", 2024-09-21T13:53:49.096 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6825/1945314741", 2024-09-21T13:53:49.096 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6823/1945314741", 2024-09-21T13:53:49.096 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T13:53:49.096 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T13:53:49.096 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T13:53:49.097 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:53:49.097 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.097 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.097 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 3, 2024-09-21T13:53:49.097 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "528bc390-3dc9-4f3b-ab75-d7412c9625db", 2024-09-21T13:53:49.097 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T13:53:49.097 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T13:53:49.097 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T13:53:49.097 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T13:53:49.097 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T13:53:49.097 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T13:53:49.097 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 27, 2024-09-21T13:53:49.097 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 159, 2024-09-21T13:53:49.097 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T13:53:49.097 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T13:53:49.098 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T13:53:49.098 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:53:49.098 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.098 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:53:49.098 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6826", 2024-09-21T13:53:49.098 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1981113123 2024-09-21T13:53:49.098 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.098 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.098 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:53:49.098 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6827", 2024-09-21T13:53:49.098 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1981113123 2024-09-21T13:53:49.098 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:53:49.098 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:53:49.098 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.099 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T13:53:49.099 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:53:49.099 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.099 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:53:49.099 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6828", 2024-09-21T13:53:49.099 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1981113123 2024-09-21T13:53:49.099 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.099 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.099 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:53:49.099 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6829", 2024-09-21T13:53:49.099 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1981113123 2024-09-21T13:53:49.099 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:53:49.099 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:53:49.099 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.099 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T13:53:49.100 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:53:49.100 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.100 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:53:49.100 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6832", 2024-09-21T13:53:49.100 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1981113123 2024-09-21T13:53:49.100 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.100 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.100 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:53:49.100 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6833", 2024-09-21T13:53:49.100 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1981113123 2024-09-21T13:53:49.100 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:53:49.100 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:53:49.100 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.100 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T13:53:49.100 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:53:49.101 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.101 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:53:49.101 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6830", 2024-09-21T13:53:49.101 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1981113123 2024-09-21T13:53:49.101 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.101 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.101 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:53:49.101 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6831", 2024-09-21T13:53:49.101 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1981113123 2024-09-21T13:53:49.101 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:53:49.101 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:53:49.101 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.101 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6827/1981113123", 2024-09-21T13:53:49.101 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6829/1981113123", 2024-09-21T13:53:49.102 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6833/1981113123", 2024-09-21T13:53:49.102 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6831/1981113123", 2024-09-21T13:53:49.102 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T13:53:49.102 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T13:53:49.102 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T13:53:49.102 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:53:49.102 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.102 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.102 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 4, 2024-09-21T13:53:49.102 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "6fc64698-2d37-4db0-bf3d-c7e612c45163", 2024-09-21T13:53:49.102 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T13:53:49.102 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T13:53:49.102 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T13:53:49.102 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T13:53:49.103 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T13:53:49.103 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T13:53:49.103 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 33, 2024-09-21T13:53:49.103 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 145, 2024-09-21T13:53:49.103 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T13:53:49.103 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T13:53:49.103 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T13:53:49.103 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:53:49.103 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.103 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:53:49.103 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6800", 2024-09-21T13:53:49.103 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T13:53:49.103 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.103 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.103 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:53:49.104 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6801", 2024-09-21T13:53:49.104 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T13:53:49.104 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:53:49.104 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:53:49.104 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.104 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T13:53:49.104 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:53:49.104 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.104 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:53:49.104 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6802", 2024-09-21T13:53:49.104 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T13:53:49.104 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.104 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.104 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:53:49.105 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6803", 2024-09-21T13:53:49.105 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T13:53:49.105 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:53:49.105 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:53:49.105 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.105 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T13:53:49.105 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:53:49.105 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.105 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:53:49.105 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6806", 2024-09-21T13:53:49.105 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T13:53:49.105 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.105 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.105 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:53:49.106 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6807", 2024-09-21T13:53:49.106 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T13:53:49.106 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:53:49.106 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:53:49.106 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.106 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T13:53:49.106 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:53:49.106 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.106 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:53:49.106 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6804", 2024-09-21T13:53:49.106 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T13:53:49.106 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.106 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.107 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:53:49.107 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6805", 2024-09-21T13:53:49.107 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T13:53:49.107 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:53:49.107 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:53:49.107 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.107 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.187:6801/1702447776", 2024-09-21T13:53:49.107 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.187:6803/1702447776", 2024-09-21T13:53:49.107 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.187:6807/1702447776", 2024-09-21T13:53:49.107 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.187:6805/1702447776", 2024-09-21T13:53:49.107 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T13:53:49.107 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T13:53:49.107 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T13:53:49.107 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:53:49.107 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.108 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.108 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 5, 2024-09-21T13:53:49.108 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "8d7f62a1-7e92-4e06-9cad-86b96be514d8", 2024-09-21T13:53:49.108 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T13:53:49.108 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T13:53:49.108 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T13:53:49.108 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T13:53:49.108 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T13:53:49.108 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T13:53:49.108 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 39, 2024-09-21T13:53:49.108 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 130, 2024-09-21T13:53:49.108 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T13:53:49.108 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T13:53:49.108 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T13:53:49.109 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:53:49.109 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.109 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:53:49.109 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6808", 2024-09-21T13:53:49.109 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3659050776 2024-09-21T13:53:49.109 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.109 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.109 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:53:49.109 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6809", 2024-09-21T13:53:49.109 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3659050776 2024-09-21T13:53:49.109 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:53:49.109 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:53:49.109 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.109 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T13:53:49.110 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:53:49.110 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.110 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:53:49.110 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6810", 2024-09-21T13:53:49.110 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3659050776 2024-09-21T13:53:49.110 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.110 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.110 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:53:49.110 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6811", 2024-09-21T13:53:49.110 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3659050776 2024-09-21T13:53:49.110 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:53:49.110 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:53:49.110 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.110 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T13:53:49.111 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:53:49.111 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.111 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:53:49.111 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6814", 2024-09-21T13:53:49.111 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3659050776 2024-09-21T13:53:49.111 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.111 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.111 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:53:49.111 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6815", 2024-09-21T13:53:49.111 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3659050776 2024-09-21T13:53:49.111 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:53:49.111 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:53:49.111 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.111 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T13:53:49.112 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:53:49.112 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.112 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:53:49.112 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6812", 2024-09-21T13:53:49.112 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3659050776 2024-09-21T13:53:49.112 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.112 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.112 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:53:49.112 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6813", 2024-09-21T13:53:49.112 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3659050776 2024-09-21T13:53:49.112 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:53:49.112 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:53:49.112 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.112 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.187:6809/3659050776", 2024-09-21T13:53:49.112 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.187:6811/3659050776", 2024-09-21T13:53:49.113 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.187:6815/3659050776", 2024-09-21T13:53:49.113 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.187:6813/3659050776", 2024-09-21T13:53:49.113 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T13:53:49.113 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T13:53:49.113 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T13:53:49.113 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:53:49.113 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.113 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.113 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 6, 2024-09-21T13:53:49.113 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "3a7afd13-26ce-444c-9cc0-81ce2eab6ce5", 2024-09-21T13:53:49.113 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T13:53:49.113 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T13:53:49.113 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T13:53:49.113 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T13:53:49.114 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T13:53:49.114 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T13:53:49.114 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 47, 2024-09-21T13:53:49.114 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 130, 2024-09-21T13:53:49.114 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T13:53:49.114 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T13:53:49.114 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T13:53:49.114 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:53:49.114 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.114 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:53:49.114 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6816", 2024-09-21T13:53:49.114 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T13:53:49.114 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.115 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.115 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:53:49.115 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6817", 2024-09-21T13:53:49.115 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T13:53:49.115 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:53:49.115 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:53:49.115 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.115 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T13:53:49.115 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:53:49.115 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.115 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:53:49.115 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6818", 2024-09-21T13:53:49.115 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T13:53:49.115 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.115 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.116 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:53:49.116 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6819", 2024-09-21T13:53:49.116 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T13:53:49.116 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:53:49.116 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:53:49.116 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.116 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T13:53:49.116 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:53:49.116 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.116 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:53:49.116 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6822", 2024-09-21T13:53:49.116 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T13:53:49.116 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.117 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.117 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:53:49.117 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6823", 2024-09-21T13:53:49.117 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T13:53:49.117 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:53:49.117 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:53:49.117 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.117 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T13:53:49.117 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:53:49.117 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.117 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:53:49.117 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6820", 2024-09-21T13:53:49.117 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T13:53:49.117 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.118 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.118 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:53:49.118 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6821", 2024-09-21T13:53:49.118 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T13:53:49.118 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:53:49.118 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:53:49.118 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.118 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.187:6817/2728426525", 2024-09-21T13:53:49.118 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.187:6819/2728426525", 2024-09-21T13:53:49.118 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.187:6823/2728426525", 2024-09-21T13:53:49.118 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.187:6821/2728426525", 2024-09-21T13:53:49.118 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T13:53:49.118 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T13:53:49.118 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T13:53:49.118 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:53:49.119 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.119 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.119 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 7, 2024-09-21T13:53:49.119 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "1e08d891-8747-402e-b4c2-aa3802ce8537", 2024-09-21T13:53:49.119 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T13:53:49.119 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T13:53:49.119 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T13:53:49.120 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T13:53:49.120 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T13:53:49.120 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T13:53:49.120 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 54, 2024-09-21T13:53:49.120 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 147, 2024-09-21T13:53:49.120 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T13:53:49.120 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T13:53:49.120 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T13:53:49.120 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:53:49.120 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.120 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:53:49.120 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6824", 2024-09-21T13:53:49.120 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3680850037 2024-09-21T13:53:49.120 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.121 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.121 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:53:49.121 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6825", 2024-09-21T13:53:49.121 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3680850037 2024-09-21T13:53:49.121 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:53:49.121 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:53:49.121 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.121 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T13:53:49.121 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:53:49.121 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.121 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:53:49.121 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6826", 2024-09-21T13:53:49.121 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3680850037 2024-09-21T13:53:49.121 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.121 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.122 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:53:49.122 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6827", 2024-09-21T13:53:49.122 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3680850037 2024-09-21T13:53:49.122 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:53:49.122 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:53:49.122 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.122 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T13:53:49.122 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:53:49.122 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.122 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:53:49.122 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6830", 2024-09-21T13:53:49.122 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3680850037 2024-09-21T13:53:49.122 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.122 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.123 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:53:49.123 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6831", 2024-09-21T13:53:49.123 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3680850037 2024-09-21T13:53:49.123 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:53:49.123 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:53:49.123 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.123 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T13:53:49.123 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:53:49.123 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.123 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:53:49.123 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6828", 2024-09-21T13:53:49.123 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3680850037 2024-09-21T13:53:49.123 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.123 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.123 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:53:49.124 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6829", 2024-09-21T13:53:49.124 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3680850037 2024-09-21T13:53:49.124 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:53:49.124 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:53:49.124 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.124 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.187:6825/3680850037", 2024-09-21T13:53:49.124 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.187:6827/3680850037", 2024-09-21T13:53:49.124 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.187:6831/3680850037", 2024-09-21T13:53:49.125 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.187:6829/3680850037", 2024-09-21T13:53:49.125 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T13:53:49.125 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T13:53:49.125 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T13:53:49.125 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:53:49.125 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:53:49.125 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T13:53:49.126 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_xinfo": [ 2024-09-21T13:53:49.126 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.126 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 0, 2024-09-21T13:53:49.126 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T13:53:49.126 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T13:53:49.126 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T13:53:49.126 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T13:53:49.126 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T13:53:49.127 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T13:52:47.585725+0000", 2024-09-21T13:53:49.127 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T13:53:49.127 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.127 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.127 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 1, 2024-09-21T13:53:49.127 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T13:53:49.127 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T13:53:49.127 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T13:53:49.128 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T13:53:49.128 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T13:53:49.128 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T13:52:49.702557+0000", 2024-09-21T13:53:49.128 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T13:53:49.128 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.128 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.128 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 2, 2024-09-21T13:53:49.128 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T13:53:49.129 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T13:53:49.129 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T13:53:49.129 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T13:53:49.129 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T13:53:49.129 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T13:53:29.356825+0000", 2024-09-21T13:53:49.129 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 100 2024-09-21T13:53:49.129 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.129 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.130 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 3, 2024-09-21T13:53:49.130 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T13:53:49.130 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T13:53:49.130 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T13:53:49.130 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T13:53:49.130 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T13:53:49.130 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T13:53:23.160665+0000", 2024-09-21T13:53:49.131 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T13:53:49.131 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.131 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.131 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 4, 2024-09-21T13:53:49.131 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T13:53:49.131 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T13:53:49.131 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T13:53:49.132 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T13:53:49.132 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T13:53:49.132 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T13:52:34.646916+0000", 2024-09-21T13:53:49.132 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T13:53:49.132 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.132 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.132 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 5, 2024-09-21T13:53:49.132 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T13:53:49.132 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T13:53:49.132 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T13:53:49.132 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T13:53:49.132 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T13:53:49.132 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T13:52:40.414515+0000", 2024-09-21T13:53:49.132 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T13:53:49.133 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.133 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.133 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 6, 2024-09-21T13:53:49.133 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T13:53:49.133 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T13:53:49.133 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T13:53:49.133 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T13:53:49.133 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T13:53:49.133 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T13:53:09.477653+0000", 2024-09-21T13:53:49.133 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T13:53:49.133 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.133 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.133 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 7, 2024-09-21T13:53:49.133 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T13:53:49.133 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T13:53:49.134 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T13:53:49.134 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T13:53:49.134 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T13:53:49.134 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T13:53:02.305094+0000", 2024-09-21T13:53:49.134 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T13:53:49.134 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:53:49.134 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T13:53:49.134 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap": [], 2024-09-21T13:53:49.134 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap_items": [ 2024-09-21T13:53:49.134 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.134 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "4.6", 2024-09-21T13:53:49.134 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-09-21T13:53:49.134 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.134 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 1, 2024-09-21T13:53:49.135 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 5 2024-09-21T13:53:49.135 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:53:49.135 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:53:49.135 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.135 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.135 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "4.1b", 2024-09-21T13:53:49.135 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-09-21T13:53:49.135 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.135 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 1, 2024-09-21T13:53:49.135 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 2 2024-09-21T13:53:49.135 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:53:49.135 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:53:49.135 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.135 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.136 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "4.1f", 2024-09-21T13:53:49.136 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-09-21T13:53:49.136 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.136 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 1, 2024-09-21T13:53:49.136 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 2 2024-09-21T13:53:49.136 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:53:49.136 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:53:49.136 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:53:49.136 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T13:53:49.136 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap_primaries": [], 2024-09-21T13:53:49.136 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_temp": [ 2024-09-21T13:53:49.136 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:53:49.136 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "7.2", 2024-09-21T13:53:49.136 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-09-21T13:53:49.136 INFO:teuthology.orchestra.run.smithi049.stdout: 3, 2024-09-21T13:53:49.137 INFO:teuthology.orchestra.run.smithi049.stdout: 1 2024-09-21T13:53:49.137 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:53:49.137 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:53:49.137 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T13:53:49.137 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_temp": [], 2024-09-21T13:53:49.137 INFO:teuthology.orchestra.run.smithi049.stdout: "blocklist": { 2024-09-21T13:53:49.137 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/2731793917": "2024-09-22T13:49:27.311313+0000", 2024-09-21T13:53:49.137 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/799831425": "2024-09-22T13:49:27.311313+0000", 2024-09-21T13:53:49.137 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/2518197280": "2024-09-22T13:43:28.110499+0000", 2024-09-21T13:53:49.137 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3556829486": "2024-09-22T13:43:28.110499+0000", 2024-09-21T13:53:49.137 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/1249926115": "2024-09-22T13:49:27.311313+0000", 2024-09-21T13:53:49.137 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/231530138": "2024-09-22T13:43:28.110499+0000", 2024-09-21T13:53:49.137 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/4220781134": "2024-09-22T13:49:27.311313+0000", 2024-09-21T13:53:49.137 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/202325455": "2024-09-22T13:43:28.110499+0000", 2024-09-21T13:53:49.137 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/4129636473": "2024-09-22T13:43:56.633723+0000", 2024-09-21T13:53:49.138 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/2518197280": "2024-09-22T13:43:28.110499+0000", 2024-09-21T13:53:49.138 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3514741273": "2024-09-22T13:43:56.633723+0000", 2024-09-21T13:53:49.138 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/4129636473": "2024-09-22T13:43:56.633723+0000", 2024-09-21T13:53:49.138 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3125743776": "2024-09-22T13:43:56.633723+0000", 2024-09-21T13:53:49.138 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3600977363": "2024-09-22T13:43:56.633723+0000", 2024-09-21T13:53:49.138 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/3453571598": "2024-09-22T13:49:27.311313+0000", 2024-09-21T13:53:49.138 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/3453571598": "2024-09-22T13:49:27.311313+0000", 2024-09-21T13:53:49.138 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3553734432": "2024-09-22T13:49:27.311313+0000" 2024-09-21T13:53:49.138 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.138 INFO:teuthology.orchestra.run.smithi049.stdout: "range_blocklist": {}, 2024-09-21T13:53:49.138 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profiles": { 2024-09-21T13:53:49.138 INFO:teuthology.orchestra.run.smithi049.stdout: "default": { 2024-09-21T13:53:49.138 INFO:teuthology.orchestra.run.smithi049.stdout: "crush-failure-domain": "osd", 2024-09-21T13:53:49.138 INFO:teuthology.orchestra.run.smithi049.stdout: "k": "2", 2024-09-21T13:53:49.139 INFO:teuthology.orchestra.run.smithi049.stdout: "m": "1", 2024-09-21T13:53:49.139 INFO:teuthology.orchestra.run.smithi049.stdout: "plugin": "jerasure", 2024-09-21T13:53:49.139 INFO:teuthology.orchestra.run.smithi049.stdout: "technique": "reed_sol_van" 2024-09-21T13:53:49.139 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:53:49.139 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:53:49.139 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps_queue": [], 2024-09-21T13:53:49.139 INFO:teuthology.orchestra.run.smithi049.stdout: "new_removed_snaps": [], 2024-09-21T13:53:49.139 INFO:teuthology.orchestra.run.smithi049.stdout: "new_purged_snaps": [], 2024-09-21T13:53:49.139 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_node_flags": {}, 2024-09-21T13:53:49.139 INFO:teuthology.orchestra.run.smithi049.stdout: "device_class_flags": {}, 2024-09-21T13:53:49.139 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode": { 2024-09-21T13:53:49.139 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode_enabled": false, 2024-09-21T13:53:49.139 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_bucket_count": 0, 2024-09-21T13:53:49.139 INFO:teuthology.orchestra.run.smithi049.stdout: "degraded_stretch_mode": 0, 2024-09-21T13:53:49.139 INFO:teuthology.orchestra.run.smithi049.stdout: "recovering_stretch_mode": 0, 2024-09-21T13:53:49.140 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode_bucket": 0 2024-09-21T13:53:49.140 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:53:49.140 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T13:53:49.184 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:53:49.209 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:49.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:49.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:53:49.311 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:53:49.337 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:49 smithi049 ceph-mon[40489]: pgmap v229: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 147 active+clean; 966 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 685 op/s; 606/46929 objects degraded (1.291%); 2.2 MiB/s, 34 objects/s recovering 2024-09-21T13:53:49.337 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:49 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3227857106' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T13:53:49.338 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:49 smithi049 ceph-mon[30080]: pgmap v229: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 147 active+clean; 966 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 685 op/s; 606/46929 objects degraded (1.291%); 2.2 MiB/s, 34 objects/s recovering 2024-09-21T13:53:49.338 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:49 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3227857106' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T13:53:49.355 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:49.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:49.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:53:49.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:49 smithi187 ceph-mon[33032]: pgmap v229: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 147 active+clean; 966 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 685 op/s; 606/46929 objects degraded (1.291%); 2.2 MiB/s, 34 objects/s recovering 2024-09-21T13:53:49.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:49 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3227857106' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T13:53:49.445 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:53:49 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:53:49.457 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:53:49.507 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:49.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:49.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:53:49.610 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:53:49.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:49.654 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:49.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:53:49.754 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:53:49.793 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:49.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:49.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:53:49.894 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:49.921 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:49.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:49.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:50.023 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:50.048 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:50.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:50.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:50.150 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:53:50.176 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:50.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:50.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:53:50.278 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:53:50.283 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T13:53:50.284 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:53:50.302 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:50.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:50.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:53:50.368 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T13:53:50.404 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:53:50.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:50.429 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:50.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:53:50.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:50 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:53:50.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:50 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 606/46929 objects degraded (1.291%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T13:53:50.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:50 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:53:50.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:50 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 606/46929 objects degraded (1.291%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T13:53:50.531 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:53:50.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:50.557 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:50.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:53:50.659 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:53:50.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:50 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:53:50.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:50 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 606/46929 objects degraded (1.291%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T13:53:50.718 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:50.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:50.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:53:50.820 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:50.846 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:50.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:50.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:50.949 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:53:50.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:50.989 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:50.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:53:51.004 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:53:50 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:53:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:53:51.090 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:53:51.095 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T13:53:51.115 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:51.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:51.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:53:51.217 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:53:51.229 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-09-21T13:53:51.242 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:51.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:51.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:53:51.344 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:51.367 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:51.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:51.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:51.468 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:51.492 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:51.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:51.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:51.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:51 smithi049 ceph-mon[30080]: pgmap v230: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 147 active+clean; 702 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.21k op/s; 229/34284 objects degraded (0.668%); 4.0 MiB/s, 64 objects/s recovering 2024-09-21T13:53:51.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:51 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3015367534' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:53:51.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:51 smithi049 ceph-mon[40489]: pgmap v230: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 147 active+clean; 702 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.21k op/s; 229/34284 objects degraded (0.668%); 4.0 MiB/s, 64 objects/s recovering 2024-09-21T13:53:51.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:51 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3015367534' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:53:51.595 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:53:51.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:51.649 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:51.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:53:51.663 INFO:tasks.thrashosds.thrasher:j is {'epoch': 170, 'fsid': '40a3d2e8-781f-11ef-baf6-efdc52797490', 'created': '2024-09-21T13:42:57.838275+0000', 'modified': '2024-09-21T13:53:44.901484+0000', 'last_up_change': '2024-09-21T13:52:03.297364+0000', 'last_in_change': '2024-09-21T13:48:23.514355+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 19, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T13:46:21.096244+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.8685569763183594, 'average_primary_affinity': 0.8700000047683716, 'average_primary_affinity_weighted': 0.8700000047683716}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-21T13:48:55.540712+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 63, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 0.8685569763183594, 'average_primary_affinity': 0.8700000047683716, 'average_primary_affinity_weighted': 0.8700000047683716}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-21T13:48:56.417456+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8685569763183594, 'average_primary_affinity': 0.8700000047683716, 'average_primary_affinity_weighted': 0.8700000047683716}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-21T13:48:58.128455+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.8685569763183594, 'average_primary_affinity': 0.8700000047683716, 'average_primary_affinity_weighted': 0.8700000047683716}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-21T13:49:00.069361+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 0.8685569763183594, 'average_primary_affinity': 0.8700000047683716, 'average_primary_affinity_weighted': 0.8700000047683716}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-21T13:49:02.127399+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8685569763183594, 'average_primary_affinity': 0.8700000047683716, 'average_primary_affinity_weighted': 0.8700000047683716}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-21T13:50:05.454919+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8685569763183594, 'average_primary_affinity': 0.8700000047683716, 'average_primary_affinity_weighted': 0.8700000047683716}}], 'osds': [{'osd': 0, 'uuid': '2595dca9-6219-4e0c-8728-751eb102e73a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.1215972900390625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 159, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6802', 'nonce': 2095561412}, {'type': 'v1', 'addr': '172.21.15.49:6803', 'nonce': 2095561412}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6804', 'nonce': 2095561412}, {'type': 'v1', 'addr': '172.21.15.49:6805', 'nonce': 2095561412}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6808', 'nonce': 2095561412}, {'type': 'v1', 'addr': '172.21.15.49:6809', 'nonce': 2095561412}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6806', 'nonce': 2095561412}, {'type': 'v1', 'addr': '172.21.15.49:6807', 'nonce': 2095561412}]}, 'public_addr': '172.21.15.49:6803/2095561412', 'cluster_addr': '172.21.15.49:6805/2095561412', 'heartbeat_back_addr': '172.21.15.49:6809/2095561412', 'heartbeat_front_addr': '172.21.15.49:6807/2095561412', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '0ba327f9-5524-4a40-abcc-729ded0b1917', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 130, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6810', 'nonce': 1382840527}, {'type': 'v1', 'addr': '172.21.15.49:6811', 'nonce': 1382840527}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6812', 'nonce': 1382840527}, {'type': 'v1', 'addr': '172.21.15.49:6813', 'nonce': 1382840527}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6816', 'nonce': 1382840527}, {'type': 'v1', 'addr': '172.21.15.49:6817', 'nonce': 1382840527}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6814', 'nonce': 1382840527}, {'type': 'v1', 'addr': '172.21.15.49:6815', 'nonce': 1382840527}]}, 'public_addr': '172.21.15.49:6811/1382840527', 'cluster_addr': '172.21.15.49:6813/1382840527', 'heartbeat_back_addr': '172.21.15.49:6817/1382840527', 'heartbeat_front_addr': '172.21.15.49:6815/1382840527', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '27a45750-204d-48fd-bb8f-ebb01273343d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.8268585205078125, 'last_clean_begin': 18, 'last_clean_end': 100, 'up_from': 130, 'up_thru': 160, 'down_at': 101, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6818', 'nonce': 1945314741}, {'type': 'v1', 'addr': '172.21.15.49:6819', 'nonce': 1945314741}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6820', 'nonce': 1945314741}, {'type': 'v1', 'addr': '172.21.15.49:6821', 'nonce': 1945314741}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6824', 'nonce': 1945314741}, {'type': 'v1', 'addr': '172.21.15.49:6825', 'nonce': 1945314741}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6822', 'nonce': 1945314741}, {'type': 'v1', 'addr': '172.21.15.49:6823', 'nonce': 1945314741}]}, 'public_addr': '172.21.15.49:6819/1945314741', 'cluster_addr': '172.21.15.49:6821/1945314741', 'heartbeat_back_addr': '172.21.15.49:6825/1945314741', 'heartbeat_front_addr': '172.21.15.49:6823/1945314741', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '528bc390-3dc9-4f3b-ab75-d7412c9625db', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 159, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6826', 'nonce': 1981113123}, {'type': 'v1', 'addr': '172.21.15.49:6827', 'nonce': 1981113123}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6828', 'nonce': 1981113123}, {'type': 'v1', 'addr': '172.21.15.49:6829', 'nonce': 1981113123}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6832', 'nonce': 1981113123}, {'type': 'v1', 'addr': '172.21.15.49:6833', 'nonce': 1981113123}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6830', 'nonce': 1981113123}, {'type': 'v1', 'addr': '172.21.15.49:6831', 'nonce': 1981113123}]}, 'public_addr': '172.21.15.49:6827/1981113123', 'cluster_addr': '172.21.15.49:6829/1981113123', 'heartbeat_back_addr': '172.21.15.49:6833/1981113123', 'heartbeat_front_addr': '172.21.15.49:6831/1981113123', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '6fc64698-2d37-4db0-bf3d-c7e612c45163', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 145, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6800', 'nonce': 1702447776}, {'type': 'v1', 'addr': '172.21.15.187:6801', 'nonce': 1702447776}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6802', 'nonce': 1702447776}, {'type': 'v1', 'addr': '172.21.15.187:6803', 'nonce': 1702447776}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6806', 'nonce': 1702447776}, {'type': 'v1', 'addr': '172.21.15.187:6807', 'nonce': 1702447776}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6804', 'nonce': 1702447776}, {'type': 'v1', 'addr': '172.21.15.187:6805', 'nonce': 1702447776}]}, 'public_addr': '172.21.15.187:6801/1702447776', 'cluster_addr': '172.21.15.187:6803/1702447776', 'heartbeat_back_addr': '172.21.15.187:6807/1702447776', 'heartbeat_front_addr': '172.21.15.187:6805/1702447776', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '8d7f62a1-7e92-4e06-9cad-86b96be514d8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 130, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6808', 'nonce': 3659050776}, {'type': 'v1', 'addr': '172.21.15.187:6809', 'nonce': 3659050776}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6810', 'nonce': 3659050776}, {'type': 'v1', 'addr': '172.21.15.187:6811', 'nonce': 3659050776}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6814', 'nonce': 3659050776}, {'type': 'v1', 'addr': '172.21.15.187:6815', 'nonce': 3659050776}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6812', 'nonce': 3659050776}, {'type': 'v1', 'addr': '172.21.15.187:6813', 'nonce': 3659050776}]}, 'public_addr': '172.21.15.187:6809/3659050776', 'cluster_addr': '172.21.15.187:6811/3659050776', 'heartbeat_back_addr': '172.21.15.187:6815/3659050776', 'heartbeat_front_addr': '172.21.15.187:6813/3659050776', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '3a7afd13-26ce-444c-9cc0-81ce2eab6ce5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 130, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6816', 'nonce': 2728426525}, {'type': 'v1', 'addr': '172.21.15.187:6817', 'nonce': 2728426525}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6818', 'nonce': 2728426525}, {'type': 'v1', 'addr': '172.21.15.187:6819', 'nonce': 2728426525}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6822', 'nonce': 2728426525}, {'type': 'v1', 'addr': '172.21.15.187:6823', 'nonce': 2728426525}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6820', 'nonce': 2728426525}, {'type': 'v1', 'addr': '172.21.15.187:6821', 'nonce': 2728426525}]}, 'public_addr': '172.21.15.187:6817/2728426525', 'cluster_addr': '172.21.15.187:6819/2728426525', 'heartbeat_back_addr': '172.21.15.187:6823/2728426525', 'heartbeat_front_addr': '172.21.15.187:6821/2728426525', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1e08d891-8747-402e-b4c2-aa3802ce8537', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 147, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 3680850037}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 3680850037}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 3680850037}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 3680850037}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6830', 'nonce': 3680850037}, {'type': 'v1', 'addr': '172.21.15.187:6831', 'nonce': 3680850037}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6828', 'nonce': 3680850037}, {'type': 'v1', 'addr': '172.21.15.187:6829', 'nonce': 3680850037}]}, 'public_addr': '172.21.15.187:6825/3680850037', 'cluster_addr': '172.21.15.187:6827/3680850037', 'heartbeat_back_addr': '172.21.15.187:6831/3680850037', 'heartbeat_front_addr': '172.21.15.187:6829/3680850037', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T13:52:47.585725+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T13:52:49.702557+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T13:53:29.356825+0000', 'dead_epoch': 100}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T13:53:23.160665+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T13:52:34.646916+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T13:52:40.414515+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T13:53:09.477653+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T13:53:02.305094+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.6', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.1b', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1f', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '7.2', 'osds': [3, 1]}], 'primary_temp': [], 'blocklist': {'172.21.15.49:0/2731793917': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:0/799831425': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:6801/2518197280': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:0/3556829486': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:0/1249926115': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:0/231530138': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:0/4220781134': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:0/202325455': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:6800/4129636473': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:6800/2518197280': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:0/3514741273': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:6801/4129636473': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:0/3125743776': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:0/3600977363': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:6801/3453571598': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:6800/3453571598': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:0/3553734432': '2024-09-22T13:49:27.311313+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-21T13:53:51.663 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T13:53:51.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:51 smithi187 ceph-mon[33032]: pgmap v230: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 147 active+clean; 702 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.21k op/s; 229/34284 objects degraded (0.668%); 4.0 MiB/s, 64 objects/s recovering 2024-09-21T13:53:51.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:51 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3015367534' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:53:51.750 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:53:51.808 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:51.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:51.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:53:51.910 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:53:51.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:51.932 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:51.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:53:52.033 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:53:52.063 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:52.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:52.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:53:52.166 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:53:52.200 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:52.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:52.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:53:52.302 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:53:52.329 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:52.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:52.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:53:52.431 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:53:52.460 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:52.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:52.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:53:52.562 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:53:52.588 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:52.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:52.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:53:52.690 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:53:52.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:52 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:53:52.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:52 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:53:52.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:52 smithi187 ceph-mon[33032]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:53:52.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:52 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3015367534' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T13:53:52.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:52 smithi187 ceph-mon[33032]: osdmap e171: 8 total, 8 up, 8 in 2024-09-21T13:53:52.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:52 smithi187 ceph-mon[33032]: 5.11 scrub starts 2024-09-21T13:53:52.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:52 smithi187 ceph-mon[33032]: 5.11 scrub ok 2024-09-21T13:53:52.696 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:52 smithi187 ceph-mon[33032]: 6.17 scrub starts 2024-09-21T13:53:52.696 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:52 smithi187 ceph-mon[33032]: 6.17 scrub ok 2024-09-21T13:53:52.719 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:52.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:52.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:53:52.752 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:52 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:53:52.752 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:52 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:53:52.752 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:52 smithi049 ceph-mon[40489]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:53:52.752 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:52 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3015367534' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T13:53:52.752 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:52 smithi049 ceph-mon[40489]: osdmap e171: 8 total, 8 up, 8 in 2024-09-21T13:53:52.752 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:52 smithi049 ceph-mon[40489]: 5.11 scrub starts 2024-09-21T13:53:52.752 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:52 smithi049 ceph-mon[40489]: 5.11 scrub ok 2024-09-21T13:53:52.753 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:52 smithi049 ceph-mon[40489]: 6.17 scrub starts 2024-09-21T13:53:52.753 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:52 smithi049 ceph-mon[40489]: 6.17 scrub ok 2024-09-21T13:53:52.753 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:52 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:53:52.753 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:52 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:53:52.753 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:52 smithi049 ceph-mon[30080]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:53:52.753 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:52 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3015367534' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T13:53:52.753 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:52 smithi049 ceph-mon[30080]: osdmap e171: 8 total, 8 up, 8 in 2024-09-21T13:53:52.753 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:52 smithi049 ceph-mon[30080]: 5.11 scrub starts 2024-09-21T13:53:52.753 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:52 smithi049 ceph-mon[30080]: 5.11 scrub ok 2024-09-21T13:53:52.753 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:52 smithi049 ceph-mon[30080]: 6.17 scrub starts 2024-09-21T13:53:52.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:52 smithi049 ceph-mon[30080]: 6.17 scrub ok 2024-09-21T13:53:52.820 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:52.824 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:53:52.840 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:53:52.844 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:52.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:52.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:52.946 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:53:52.974 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:52.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:52.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:53:53.076 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:53:53.099 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:53.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:53.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:53:53.201 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:53:53.226 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:53.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:53.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:53:53.328 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:53:53.367 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:53.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:53.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:53:53.390 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:53 smithi049 ceph-mon[30080]: pgmap v232: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 147 active+clean; 702 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 750 op/s; 229/34284 objects degraded (0.668%); 4.0 MiB/s, 64 objects/s recovering 2024-09-21T13:53:53.390 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:53 smithi049 ceph-mon[40489]: pgmap v232: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 147 active+clean; 702 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 750 op/s; 229/34284 objects degraded (0.668%); 4.0 MiB/s, 64 objects/s recovering 2024-09-21T13:53:53.468 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:53.492 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:53.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:53.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:53.558 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:53:53.594 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:53:53.616 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:53.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:53.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:53:53.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:53 smithi187 ceph-mon[33032]: pgmap v232: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 147 active+clean; 702 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 750 op/s; 229/34284 objects degraded (0.668%); 4.0 MiB/s, 64 objects/s recovering 2024-09-21T13:53:53.718 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:53:53.739 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:53.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:53.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:53:53.772 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:53:53.772 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:53:53.789 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:53:53.789 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:53:53.805 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:53:53.805 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:53:53.821 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:53:53.821 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:53:53.838 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:53:53.838 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:53:53.840 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:53:53.855 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:53:53.855 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:53:53.862 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:53.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:53.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:53:53.872 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:53:53.872 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:53:53.888 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:53:53.889 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:53:53.963 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:53:53.986 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:53.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:53.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:53:54.087 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:53:54.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:54.110 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:54.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:53:54.211 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:54.228 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T13:53:54.229 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-09-21T13:53:54.229 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-09-21T13:53:54.229 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-09-21T13:53:54.229 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T13:53:54.233 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:54.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:54.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:54.334 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:53:54.358 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:54.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:54.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:53:54.460 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:53:54.482 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:54.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:54.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:53:54.583 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:53:54.611 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:54.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:54.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:53:54.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:54 smithi187 ceph-mon[33032]: 5.b scrub starts 2024-09-21T13:53:54.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:54 smithi187 ceph-mon[33032]: 5.b scrub ok 2024-09-21T13:53:54.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:54 smithi187 ceph-mon[33032]: osdmap e172: 8 total, 8 up, 8 in 2024-09-21T13:53:54.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:54 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2084698009' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:53:54.713 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:53:54.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:54 smithi049 ceph-mon[40489]: 5.b scrub starts 2024-09-21T13:53:54.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:54 smithi049 ceph-mon[40489]: 5.b scrub ok 2024-09-21T13:53:54.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:54 smithi049 ceph-mon[40489]: osdmap e172: 8 total, 8 up, 8 in 2024-09-21T13:53:54.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:54 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2084698009' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:53:54.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:54 smithi049 ceph-mon[30080]: 5.b scrub starts 2024-09-21T13:53:54.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:54 smithi049 ceph-mon[30080]: 5.b scrub ok 2024-09-21T13:53:54.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:54 smithi049 ceph-mon[30080]: osdmap e172: 8 total, 8 up, 8 in 2024-09-21T13:53:54.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:54 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2084698009' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:53:54.758 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:54.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:54.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:53:54.817 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T13:53:54.860 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:53:54.882 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:54.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:54.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:53:54.984 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:53:55.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:55.027 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:55.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:53:55.127 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:53:55.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:55.158 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:55.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:53:55.259 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:53:55.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:55.289 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:55.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:53:55.390 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:53:55.411 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:55.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:55.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:53:55.513 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:53:55.549 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:55.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:55.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:53:55.641 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:55 smithi049 ceph-mon[40489]: pgmap v234: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 147 active+clean; 465 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.07k op/s; 229/22881 objects degraded (1.001%); 1.9 MiB/s, 29 objects/s recovering 2024-09-21T13:53:55.641 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:55 smithi049 ceph-mon[40489]: osdmap e173: 8 total, 8 up, 8 in 2024-09-21T13:53:55.641 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:55 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 229/22881 objects degraded (1.001%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T13:53:55.641 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:55 smithi049 ceph-mon[40489]: osdmap e174: 8 total, 8 up, 8 in 2024-09-21T13:53:55.641 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:55 smithi049 ceph-mon[40489]: 6.0 scrub starts 2024-09-21T13:53:55.641 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:55 smithi049 ceph-mon[40489]: 6.0 scrub ok 2024-09-21T13:53:55.641 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:55 smithi049 ceph-mon[30080]: pgmap v234: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 147 active+clean; 465 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.07k op/s; 229/22881 objects degraded (1.001%); 1.9 MiB/s, 29 objects/s recovering 2024-09-21T13:53:55.641 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:55 smithi049 ceph-mon[30080]: osdmap e173: 8 total, 8 up, 8 in 2024-09-21T13:53:55.642 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:55 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 229/22881 objects degraded (1.001%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T13:53:55.642 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:55 smithi049 ceph-mon[30080]: osdmap e174: 8 total, 8 up, 8 in 2024-09-21T13:53:55.642 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:55 smithi049 ceph-mon[30080]: 6.0 scrub starts 2024-09-21T13:53:55.642 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:55 smithi049 ceph-mon[30080]: 6.0 scrub ok 2024-09-21T13:53:55.651 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:53:55.694 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:55.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:55 smithi187 ceph-mon[33032]: pgmap v234: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 147 active+clean; 465 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.07k op/s; 229/22881 objects degraded (1.001%); 1.9 MiB/s, 29 objects/s recovering 2024-09-21T13:53:55.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:55 smithi187 ceph-mon[33032]: osdmap e173: 8 total, 8 up, 8 in 2024-09-21T13:53:55.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:55 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 229/22881 objects degraded (1.001%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T13:53:55.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:55 smithi187 ceph-mon[33032]: osdmap e174: 8 total, 8 up, 8 in 2024-09-21T13:53:55.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:55 smithi187 ceph-mon[33032]: 6.0 scrub starts 2024-09-21T13:53:55.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:55 smithi187 ceph-mon[33032]: 6.0 scrub ok 2024-09-21T13:53:55.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:55.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:53:55.797 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:53:55.836 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:55.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:55.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:53:55.937 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:53:55.976 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:55.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:55.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:53:56.078 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:53:56.121 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:56.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:56.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:53:56.223 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:53:56.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:56.267 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:56.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:53:56.368 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:56.397 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:56.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:56.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:56.499 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:53:56.545 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:56.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:56.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:53:56.609 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:56 smithi049 ceph-mon[40489]: 3.a scrub starts 2024-09-21T13:53:56.609 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:56 smithi049 ceph-mon[40489]: 3.a scrub ok 2024-09-21T13:53:56.610 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:56 smithi049 ceph-mon[30080]: 3.a scrub starts 2024-09-21T13:53:56.610 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:56 smithi049 ceph-mon[30080]: 3.a scrub ok 2024-09-21T13:53:56.647 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:53:56.686 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:56.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:56.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:53:56.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:56 smithi187 ceph-mon[33032]: 3.a scrub starts 2024-09-21T13:53:56.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:56 smithi187 ceph-mon[33032]: 3.a scrub ok 2024-09-21T13:53:56.789 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:53:56.832 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:56.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:56.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:53:56.934 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:53:56.972 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:56.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:56.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:53:57.073 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:53:57.117 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T13:53:57.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:57.139 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:57.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:53:57.241 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:53:57.268 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:57.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:57.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:53:57.370 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:53:57.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:57.411 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:57.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:53:57.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:57 smithi049 ceph-mon[30080]: pgmap v237: 148 pgs: 1 peering, 147 active+clean; 393 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.09k op/s; 2.8 MiB/s, 44 objects/s recovering 2024-09-21T13:53:57.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:57 smithi049 ceph-mon[30080]: 7.7 scrub starts 2024-09-21T13:53:57.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:57 smithi049 ceph-mon[30080]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 229/22881 objects degraded (1.001%), 1 pg degraded, 1 pg undersized) 2024-09-21T13:53:57.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:57 smithi049 ceph-mon[30080]: 7.7 scrub ok 2024-09-21T13:53:57.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:57 smithi049 ceph-mon[30080]: 5.5 scrub starts 2024-09-21T13:53:57.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:57 smithi049 ceph-mon[30080]: 5.5 scrub ok 2024-09-21T13:53:57.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:57 smithi049 ceph-mon[40489]: pgmap v237: 148 pgs: 1 peering, 147 active+clean; 393 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.09k op/s; 2.8 MiB/s, 44 objects/s recovering 2024-09-21T13:53:57.507 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:57 smithi049 ceph-mon[40489]: 7.7 scrub starts 2024-09-21T13:53:57.507 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:57 smithi049 ceph-mon[40489]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 229/22881 objects degraded (1.001%), 1 pg degraded, 1 pg undersized) 2024-09-21T13:53:57.507 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:57 smithi049 ceph-mon[40489]: 7.7 scrub ok 2024-09-21T13:53:57.507 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:57 smithi049 ceph-mon[40489]: 5.5 scrub starts 2024-09-21T13:53:57.507 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:57 smithi049 ceph-mon[40489]: 5.5 scrub ok 2024-09-21T13:53:57.512 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:53:57.538 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:57.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:57.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:53:57.640 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:57.666 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:57.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:57.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:57.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:57 smithi187 ceph-mon[33032]: pgmap v237: 148 pgs: 1 peering, 147 active+clean; 393 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.09k op/s; 2.8 MiB/s, 44 objects/s recovering 2024-09-21T13:53:57.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:57 smithi187 ceph-mon[33032]: 7.7 scrub starts 2024-09-21T13:53:57.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:57 smithi187 ceph-mon[33032]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 229/22881 objects degraded (1.001%), 1 pg degraded, 1 pg undersized) 2024-09-21T13:53:57.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:57 smithi187 ceph-mon[33032]: 7.7 scrub ok 2024-09-21T13:53:57.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:57 smithi187 ceph-mon[33032]: 5.5 scrub starts 2024-09-21T13:53:57.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:57 smithi187 ceph-mon[33032]: 5.5 scrub ok 2024-09-21T13:53:57.767 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:53:57.792 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:57.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:57.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:53:57.894 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:53:57.922 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:57.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:57.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:53:58.024 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:53:58.045 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:58.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:58.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:53:58.148 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:53:58.174 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:58.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:58.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:53:58.276 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:53:58.314 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:58.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:58.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:53:58.415 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:53:58.444 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:58.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:58.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:53:58.546 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:53:58.575 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:58.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:58.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:53:58.677 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:53:58.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:58 smithi187 ceph-mon[33032]: 7.d scrub starts 2024-09-21T13:53:58.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:58 smithi187 ceph-mon[33032]: 7.d scrub ok 2024-09-21T13:53:58.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:58 smithi187 ceph-mon[33032]: 4.9 scrub starts 2024-09-21T13:53:58.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:58 smithi187 ceph-mon[33032]: 4.9 scrub ok 2024-09-21T13:53:58.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:58 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:53:58.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:58 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:53:58.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:58 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:53:58.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:58 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:53:58.718 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:58.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:58.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:53:58.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:58 smithi049 ceph-mon[40489]: 7.d scrub starts 2024-09-21T13:53:58.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:58 smithi049 ceph-mon[40489]: 7.d scrub ok 2024-09-21T13:53:58.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:58 smithi049 ceph-mon[40489]: 4.9 scrub starts 2024-09-21T13:53:58.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:58 smithi049 ceph-mon[40489]: 4.9 scrub ok 2024-09-21T13:53:58.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:58 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:53:58.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:58 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:53:58.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:58 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:53:58.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:58 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:53:58.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:58 smithi049 ceph-mon[30080]: 7.d scrub starts 2024-09-21T13:53:58.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:58 smithi049 ceph-mon[30080]: 7.d scrub ok 2024-09-21T13:53:58.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:58 smithi049 ceph-mon[30080]: 4.9 scrub starts 2024-09-21T13:53:58.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:58 smithi049 ceph-mon[30080]: 4.9 scrub ok 2024-09-21T13:53:58.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:58 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:53:58.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:58 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:53:58.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:58 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:53:58.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:58 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:53:58.820 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:53:58.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:58.862 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:58.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:53:58.963 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:53:59.022 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:59.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:59.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:53:59.124 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:53:59.149 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:59.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:59.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:53:59.250 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:53:59.309 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:53:59.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:59.316 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:59.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:53:59.417 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:53:59.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:59.443 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:59.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:53:59.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:59 smithi049 ceph-mon[40489]: pgmap v238: 148 pgs: 1 peering, 147 active+clean; 358 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 917 op/s; 2.1 MiB/s, 33 objects/s recovering 2024-09-21T13:53:59.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:59 smithi049 ceph-mon[40489]: 6.1f scrub starts 2024-09-21T13:53:59.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:59 smithi049 ceph-mon[40489]: 6.1f scrub ok 2024-09-21T13:53:59.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:59 smithi049 ceph-mon[40489]: 4.1a scrub starts 2024-09-21T13:53:59.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:59 smithi049 ceph-mon[40489]: 4.1a scrub ok 2024-09-21T13:53:59.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:59 smithi049 ceph-mon[40489]: osdmap e175: 8 total, 8 up, 8 in 2024-09-21T13:53:59.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:59 smithi049 ceph-mon[40489]: 4.f scrub starts 2024-09-21T13:53:59.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:59 smithi049 ceph-mon[40489]: 4.f scrub ok 2024-09-21T13:53:59.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:53:59 smithi049 ceph-mon[40489]: 7.a scrub starts 2024-09-21T13:53:59.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:59 smithi049 ceph-mon[30080]: pgmap v238: 148 pgs: 1 peering, 147 active+clean; 358 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 917 op/s; 2.1 MiB/s, 33 objects/s recovering 2024-09-21T13:53:59.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:59 smithi049 ceph-mon[30080]: 6.1f scrub starts 2024-09-21T13:53:59.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:59 smithi049 ceph-mon[30080]: 6.1f scrub ok 2024-09-21T13:53:59.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:59 smithi049 ceph-mon[30080]: 4.1a scrub starts 2024-09-21T13:53:59.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:59 smithi049 ceph-mon[30080]: 4.1a scrub ok 2024-09-21T13:53:59.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:59 smithi049 ceph-mon[30080]: osdmap e175: 8 total, 8 up, 8 in 2024-09-21T13:53:59.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:59 smithi049 ceph-mon[30080]: 4.f scrub starts 2024-09-21T13:53:59.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:59 smithi049 ceph-mon[30080]: 4.f scrub ok 2024-09-21T13:53:59.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:53:59 smithi049 ceph-mon[30080]: 7.a scrub starts 2024-09-21T13:53:59.544 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:53:59.568 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:59.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:59.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:53:59.632 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:53:59.638 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-09-21T13:53:59.670 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:53:59.694 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:53:59 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:53:59.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:59 smithi187 ceph-mon[33032]: pgmap v238: 148 pgs: 1 peering, 147 active+clean; 358 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 917 op/s; 2.1 MiB/s, 33 objects/s recovering 2024-09-21T13:53:59.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:59 smithi187 ceph-mon[33032]: 6.1f scrub starts 2024-09-21T13:53:59.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:59 smithi187 ceph-mon[33032]: 6.1f scrub ok 2024-09-21T13:53:59.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:59 smithi187 ceph-mon[33032]: 4.1a scrub starts 2024-09-21T13:53:59.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:59 smithi187 ceph-mon[33032]: 4.1a scrub ok 2024-09-21T13:53:59.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:59 smithi187 ceph-mon[33032]: osdmap e175: 8 total, 8 up, 8 in 2024-09-21T13:53:59.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:59 smithi187 ceph-mon[33032]: 4.f scrub starts 2024-09-21T13:53:59.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:59 smithi187 ceph-mon[33032]: 4.f scrub ok 2024-09-21T13:53:59.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:53:59 smithi187 ceph-mon[33032]: 7.a scrub starts 2024-09-21T13:53:59.709 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:59.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:59.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:53:59.811 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:53:59.861 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:53:59.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:59.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:53:59.962 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:53:59.988 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:53:59.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:53:59.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:00.090 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:54:00.119 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:00.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:00.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:54:00.138 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:54:00.221 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:00.268 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:00.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:00.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:00.287 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Removed 136417 objects 2024-09-21T13:54:00.287 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Clean up completed and total clean up time :137.361 2024-09-21T13:54:00.370 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:00.392 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:00.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:00.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:00.494 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:00.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:00 smithi049 ceph-mon[40489]: 3.1d scrub starts 2024-09-21T13:54:00.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:00 smithi049 ceph-mon[40489]: 3.1d scrub ok 2024-09-21T13:54:00.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:00 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:54:00.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:00 smithi049 ceph-mon[40489]: 7.a scrub ok 2024-09-21T13:54:00.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:00 smithi049 ceph-mon[40489]: from='client.24964 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T13:54:00.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:00 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:54:00.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:00 smithi049 ceph-mon[40489]: osdmap e176: 8 total, 8 up, 8 in 2024-09-21T13:54:00.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:00 smithi049 ceph-mon[30080]: 3.1d scrub starts 2024-09-21T13:54:00.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:00 smithi049 ceph-mon[30080]: 3.1d scrub ok 2024-09-21T13:54:00.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:00 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:54:00.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:00 smithi049 ceph-mon[30080]: 7.a scrub ok 2024-09-21T13:54:00.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:00 smithi049 ceph-mon[30080]: from='client.24964 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T13:54:00.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:00 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:54:00.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:00 smithi049 ceph-mon[30080]: osdmap e176: 8 total, 8 up, 8 in 2024-09-21T13:54:00.515 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:00.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:00.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:00.617 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:54:00.624 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":239,"stamp":"2024-09-21T13:53:58.310817+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":375724595,"num_objects":5924,"num_object_clones":0,"num_object_copies":17772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5924,"num_whiteouts":0,"num_read":1135,"num_read_kb":992,"num_write":267660,"num_write_kb":8732362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22713,"num_bytes_recovered":1488126716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87704,"ondisk_log_size":87704,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2366616,"kb_used_data":1169200,"kb_used_omap":15,"kb_used_meta":1197168,"kb_avail":747594600,"statfs":{"total":767960285184,"available":765536870400,"internally_reserved":0,"allocated":1197260800,"data_stored":1193759331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15598,"internal_metadata":1225900818},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,0,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-319750144,"num_objects":-4879,"num_object_clones":0,"num_object_copies":-14637,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4879,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4879,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":204,"num_bytes_recovered":13369344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.031420"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":187,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.289030+0000","last_change":"2024-09-21T13:52:26.677736+0000","last_active":"2024-09-21T13:53:53.289030+0000","last_peered":"2024-09-21T13:53:53.289030+0000","last_clean":"2024-09-21T13:53:53.289030+0000","last_became_active":"2024-09-21T13:52:04.317322+0000","last_became_peered":"2024-09-21T13:52:04.317322+0000","last_unstale":"2024-09-21T13:53:53.289030+0000","last_undegraded":"2024-09-21T13:53:53.289030+0000","last_fullsized":"2024-09-21T13:53:53.289030+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:26.677689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:52:26.677689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:46.286464+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"68'9","reported_seq":256,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.273727+0000","last_change":"2024-09-21T13:51:55.225649+0000","last_active":"2024-09-21T13:53:53.273727+0000","last_peered":"2024-09-21T13:53:53.273727+0000","last_clean":"2024-09-21T13:53:53.273727+0000","last_became_active":"2024-09-21T13:49:00.055210+0000","last_became_peered":"2024-09-21T13:49:00.055210+0000","last_unstale":"2024-09-21T13:53:53.273727+0000","last_undegraded":"2024-09-21T13:53:53.273727+0000","last_fullsized":"2024-09-21T13:53:53.273727+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:51:55.225579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:55.225579+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":235,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.292643+0000","last_change":"2024-09-21T13:50:28.058990+0000","last_active":"2024-09-21T13:53:53.292643+0000","last_peered":"2024-09-21T13:53:53.292643+0000","last_clean":"2024-09-21T13:53:53.292643+0000","last_became_active":"2024-09-21T13:49:02.060984+0000","last_became_peered":"2024-09-21T13:49:02.060984+0000","last_unstale":"2024-09-21T13:53:53.292643+0000","last_undegraded":"2024-09-21T13:53:53.292643+0000","last_fullsized":"2024-09-21T13:53:53.292643+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:28.058918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:28.058918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:48.620137+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":231,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.273601+0000","last_change":"2024-09-21T13:53:21.005444+0000","last_active":"2024-09-21T13:53:53.273601+0000","last_peered":"2024-09-21T13:53:53.273601+0000","last_clean":"2024-09-21T13:53:53.273601+0000","last_became_active":"2024-09-21T13:49:04.072460+0000","last_became_peered":"2024-09-21T13:49:04.072460+0000","last_unstale":"2024-09-21T13:53:53.273601+0000","last_undegraded":"2024-09-21T13:53:53.273601+0000","last_fullsized":"2024-09-21T13:53:53.273601+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:21.005352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:21.005352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:43.773187+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":231,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.285860+0000","last_change":"2024-09-21T13:51:18.572788+0000","last_active":"2024-09-21T13:53:53.285860+0000","last_peered":"2024-09-21T13:53:53.285860+0000","last_clean":"2024-09-21T13:53:53.285860+0000","last_became_active":"2024-09-21T13:49:04.433965+0000","last_became_peered":"2024-09-21T13:49:04.433965+0000","last_unstale":"2024-09-21T13:53:53.285860+0000","last_undegraded":"2024-09-21T13:53:53.285860+0000","last_fullsized":"2024-09-21T13:53:53.285860+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:18.572729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:51:18.572729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":255,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.283711+0000","last_change":"2024-09-21T13:53:19.910708+0000","last_active":"2024-09-21T13:53:53.283711+0000","last_peered":"2024-09-21T13:53:53.283711+0000","last_clean":"2024-09-21T13:53:53.283711+0000","last_became_active":"2024-09-21T13:52:04.316819+0000","last_became_peered":"2024-09-21T13:52:04.316819+0000","last_unstale":"2024-09-21T13:53:53.283711+0000","last_undegraded":"2024-09-21T13:53:53.283711+0000","last_fullsized":"2024-09-21T13:53:53.283711+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:19.910638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:53:19.910638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:27.797531+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"68'15","reported_seq":268,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.285284+0000","last_change":"2024-09-21T13:52:04.315645+0000","last_active":"2024-09-21T13:53:53.285284+0000","last_peered":"2024-09-21T13:53:53.285284+0000","last_clean":"2024-09-21T13:53:53.285284+0000","last_became_active":"2024-09-21T13:52:04.315101+0000","last_became_peered":"2024-09-21T13:52:04.315101+0000","last_unstale":"2024-09-21T13:53:53.285284+0000","last_undegraded":"2024-09-21T13:53:53.285284+0000","last_fullsized":"2024-09-21T13:53:53.285284+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:15.792034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":244,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.273577+0000","last_change":"2024-09-21T13:51:43.040397+0000","last_active":"2024-09-21T13:53:53.273577+0000","last_peered":"2024-09-21T13:53:53.273577+0000","last_clean":"2024-09-21T13:53:53.273577+0000","last_became_active":"2024-09-21T13:49:02.063687+0000","last_became_peered":"2024-09-21T13:49:02.063687+0000","last_unstale":"2024-09-21T13:53:53.273577+0000","last_undegraded":"2024-09-21T13:53:53.273577+0000","last_fullsized":"2024-09-21T13:53:53.273577+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:43.040348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:43.040348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":251,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.288875+0000","last_change":"2024-09-21T13:52:49.451929+0000","last_active":"2024-09-21T13:53:53.288875+0000","last_peered":"2024-09-21T13:53:53.288875+0000","last_clean":"2024-09-21T13:53:53.288875+0000","last_became_active":"2024-09-21T13:48:58.050695+0000","last_became_peered":"2024-09-21T13:48:58.050695+0000","last_unstale":"2024-09-21T13:53:53.288875+0000","last_undegraded":"2024-09-21T13:53:53.288875+0000","last_fullsized":"2024-09-21T13:53:53.288875+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:49.451882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:52:49.451882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:54.087443+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"68'9","reported_seq":255,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.273921+0000","last_change":"2024-09-21T13:51:21.951391+0000","last_active":"2024-09-21T13:53:53.273921+0000","last_peered":"2024-09-21T13:53:53.273921+0000","last_clean":"2024-09-21T13:53:53.273921+0000","last_became_active":"2024-09-21T13:49:00.059382+0000","last_became_peered":"2024-09-21T13:49:00.059382+0000","last_unstale":"2024-09-21T13:53:53.273921+0000","last_undegraded":"2024-09-21T13:53:53.273921+0000","last_fullsized":"2024-09-21T13:53:53.273921+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:51:21.951305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:21.951305+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":227,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.288330+0000","last_change":"2024-09-21T13:49:02.066978+0000","last_active":"2024-09-21T13:53:53.288330+0000","last_peered":"2024-09-21T13:53:53.288330+0000","last_clean":"2024-09-21T13:53:53.288330+0000","last_became_active":"2024-09-21T13:49:02.066693+0000","last_became_peered":"2024-09-21T13:49:02.066693+0000","last_unstale":"2024-09-21T13:53:53.288330+0000","last_undegraded":"2024-09-21T13:53:53.288330+0000","last_fullsized":"2024-09-21T13:53:53.288330+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:09.312032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":184,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.282809+0000","last_change":"2024-09-21T13:52:28.417865+0000","last_active":"2024-09-21T13:53:53.282809+0000","last_peered":"2024-09-21T13:53:53.282809+0000","last_clean":"2024-09-21T13:53:53.282809+0000","last_became_active":"2024-09-21T13:50:29.689381+0000","last_became_peered":"2024-09-21T13:50:29.689381+0000","last_unstale":"2024-09-21T13:53:53.282809+0000","last_undegraded":"2024-09-21T13:53:53.282809+0000","last_fullsized":"2024-09-21T13:53:53.282809+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:28.417690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:28.417690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":243,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.288469+0000","last_change":"2024-09-21T13:50:39.480995+0000","last_active":"2024-09-21T13:53:53.288469+0000","last_peered":"2024-09-21T13:53:53.288469+0000","last_clean":"2024-09-21T13:53:53.288469+0000","last_became_active":"2024-09-21T13:48:58.050906+0000","last_became_peered":"2024-09-21T13:48:58.050906+0000","last_unstale":"2024-09-21T13:53:53.288469+0000","last_undegraded":"2024-09-21T13:53:53.288469+0000","last_fullsized":"2024-09-21T13:53:53.288469+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:39.480930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:39.480930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:34.214082+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"68'5","reported_seq":265,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.273434+0000","last_change":"2024-09-21T13:52:04.319297+0000","last_active":"2024-09-21T13:53:53.273434+0000","last_peered":"2024-09-21T13:53:53.273434+0000","last_clean":"2024-09-21T13:53:53.273434+0000","last_became_active":"2024-09-21T13:52:04.318515+0000","last_became_peered":"2024-09-21T13:52:04.318515+0000","last_unstale":"2024-09-21T13:53:53.273434+0000","last_undegraded":"2024-09-21T13:53:53.273434+0000","last_fullsized":"2024-09-21T13:53:53.273434+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:02.067591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":235,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.281465+0000","last_change":"2024-09-21T13:50:40.188797+0000","last_active":"2024-09-21T13:53:53.281465+0000","last_peered":"2024-09-21T13:53:53.281465+0000","last_clean":"2024-09-21T13:53:53.281465+0000","last_became_active":"2024-09-21T13:49:02.060212+0000","last_became_peered":"2024-09-21T13:49:02.060212+0000","last_unstale":"2024-09-21T13:53:53.281465+0000","last_undegraded":"2024-09-21T13:53:53.281465+0000","last_fullsized":"2024-09-21T13:53:53.281465+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:40.188712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:40.188712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":239,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.287963+0000","last_change":"2024-09-21T13:52:43.516806+0000","last_active":"2024-09-21T13:53:53.287963+0000","last_peered":"2024-09-21T13:53:53.287963+0000","last_clean":"2024-09-21T13:53:53.287963+0000","last_became_active":"2024-09-21T13:49:04.078026+0000","last_became_peered":"2024-09-21T13:49:04.078026+0000","last_unstale":"2024-09-21T13:53:53.287963+0000","last_undegraded":"2024-09-21T13:53:53.287963+0000","last_fullsized":"2024-09-21T13:53:53.287963+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:43.516741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:43.516741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:08.753512+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":232,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.274648+0000","last_change":"2024-09-21T13:50:38.027288+0000","last_active":"2024-09-21T13:53:53.274648+0000","last_peered":"2024-09-21T13:53:53.274648+0000","last_clean":"2024-09-21T13:53:53.274648+0000","last_became_active":"2024-09-21T13:49:04.431422+0000","last_became_peered":"2024-09-21T13:49:04.431422+0000","last_unstale":"2024-09-21T13:53:53.274648+0000","last_undegraded":"2024-09-21T13:53:53.274648+0000","last_fullsized":"2024-09-21T13:53:53.274648+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:38.027227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:38.027227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:49.236040+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":168,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.273787+0000","last_change":"2024-09-21T13:50:29.692621+0000","last_active":"2024-09-21T13:53:53.273787+0000","last_peered":"2024-09-21T13:53:53.273787+0000","last_clean":"2024-09-21T13:53:53.273787+0000","last_became_active":"2024-09-21T13:50:29.688282+0000","last_became_peered":"2024-09-21T13:50:29.688282+0000","last_unstale":"2024-09-21T13:53:53.273787+0000","last_undegraded":"2024-09-21T13:53:53.273787+0000","last_fullsized":"2024-09-21T13:53:53.273787+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:04.102443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"68'15","reported_seq":208,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-21T13:53:54.927098+0000","last_change":"2024-09-21T13:53:32.929362+0000","last_active":"2024-09-21T13:53:54.927098+0000","last_peered":"2024-09-21T13:53:54.927098+0000","last_clean":"2024-09-21T13:53:54.927098+0000","last_became_active":"2024-09-21T13:52:04.324273+0000","last_became_peered":"2024-09-21T13:52:04.324273+0000","last_unstale":"2024-09-21T13:53:54.927098+0000","last_undegraded":"2024-09-21T13:53:54.927098+0000","last_fullsized":"2024-09-21T13:53:54.927098+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:53:32.929017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:32.929017+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:37.476556+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":243,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.291515+0000","last_change":"2024-09-21T13:52:26.372217+0000","last_active":"2024-09-21T13:53:53.291515+0000","last_peered":"2024-09-21T13:53:53.291515+0000","last_clean":"2024-09-21T13:53:53.291515+0000","last_became_active":"2024-09-21T13:49:02.066796+0000","last_became_peered":"2024-09-21T13:49:02.066796+0000","last_unstale":"2024-09-21T13:53:53.291515+0000","last_undegraded":"2024-09-21T13:53:53.291515+0000","last_fullsized":"2024-09-21T13:53:53.291515+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:26.372144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:52:26.372144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":224,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.284348+0000","last_change":"2024-09-21T13:49:04.434429+0000","last_active":"2024-09-21T13:53:53.284348+0000","last_peered":"2024-09-21T13:53:53.284348+0000","last_clean":"2024-09-21T13:53:53.284348+0000","last_became_active":"2024-09-21T13:49:04.434049+0000","last_became_peered":"2024-09-21T13:49:04.434049+0000","last_unstale":"2024-09-21T13:53:53.284348+0000","last_undegraded":"2024-09-21T13:53:53.284348+0000","last_fullsized":"2024-09-21T13:53:53.284348+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":254,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.274052+0000","last_change":"2024-09-21T13:52:52.142682+0000","last_active":"2024-09-21T13:53:53.274052+0000","last_peered":"2024-09-21T13:53:53.274052+0000","last_clean":"2024-09-21T13:53:53.274052+0000","last_became_active":"2024-09-21T13:52:04.316016+0000","last_became_peered":"2024-09-21T13:52:04.316016+0000","last_unstale":"2024-09-21T13:53:53.274052+0000","last_undegraded":"2024-09-21T13:53:53.274052+0000","last_fullsized":"2024-09-21T13:53:53.274052+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:52.142638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:52:52.142638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:13.595892+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"68'12","reported_seq":262,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.283465+0000","last_change":"2024-09-21T13:51:52.819966+0000","last_active":"2024-09-21T13:53:53.283465+0000","last_peered":"2024-09-21T13:53:53.283465+0000","last_clean":"2024-09-21T13:53:53.283465+0000","last_became_active":"2024-09-21T13:49:00.061867+0000","last_became_peered":"2024-09-21T13:49:00.061867+0000","last_unstale":"2024-09-21T13:53:53.283465+0000","last_undegraded":"2024-09-21T13:53:53.283465+0000","last_fullsized":"2024-09-21T13:53:53.283465+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'12","last_scrub_stamp":"2024-09-21T13:51:52.819879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:52.819879+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":238,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.275801+0000","last_change":"2024-09-21T13:52:04.321297+0000","last_active":"2024-09-21T13:53:53.275801+0000","last_peered":"2024-09-21T13:53:53.275801+0000","last_clean":"2024-09-21T13:53:53.275801+0000","last_became_active":"2024-09-21T13:52:04.320800+0000","last_became_peered":"2024-09-21T13:52:04.320800+0000","last_unstale":"2024-09-21T13:53:53.275801+0000","last_undegraded":"2024-09-21T13:53:53.275801+0000","last_fullsized":"2024-09-21T13:53:53.275801+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:14.081205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":243,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.283167+0000","last_change":"2024-09-21T13:52:54.580070+0000","last_active":"2024-09-21T13:53:53.283167+0000","last_peered":"2024-09-21T13:53:53.283167+0000","last_clean":"2024-09-21T13:53:53.283167+0000","last_became_active":"2024-09-21T13:52:04.321358+0000","last_became_peered":"2024-09-21T13:52:04.321358+0000","last_unstale":"2024-09-21T13:53:53.283167+0000","last_undegraded":"2024-09-21T13:53:53.283167+0000","last_fullsized":"2024-09-21T13:53:53.283167+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T13:52:54.580022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:54.580022+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:21.087079+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":253,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.292331+0000","last_change":"2024-09-21T13:52:43.516442+0000","last_active":"2024-09-21T13:53:53.292331+0000","last_peered":"2024-09-21T13:53:53.292331+0000","last_clean":"2024-09-21T13:53:53.292331+0000","last_became_active":"2024-09-21T13:48:58.048972+0000","last_became_peered":"2024-09-21T13:48:58.048972+0000","last_unstale":"2024-09-21T13:53:53.292331+0000","last_undegraded":"2024-09-21T13:53:53.292331+0000","last_fullsized":"2024-09-21T13:53:53.292331+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:52:43.516221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:52:43.516221+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"68'10","reported_seq":211,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.282373+0000","last_change":"2024-09-21T13:52:56.526032+0000","last_active":"2024-09-21T13:53:53.282373+0000","last_peered":"2024-09-21T13:53:53.282373+0000","last_clean":"2024-09-21T13:53:53.282373+0000","last_became_active":"2024-09-21T13:50:29.694465+0000","last_became_peered":"2024-09-21T13:50:29.694465+0000","last_unstale":"2024-09-21T13:53:53.282373+0000","last_undegraded":"2024-09-21T13:53:53.282373+0000","last_fullsized":"2024-09-21T13:53:53.282373+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:52:56.525966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:52:56.525966+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:10.063932+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"68'8","reported_seq":315,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.278749+0000","last_change":"2024-09-21T13:51:42.174696+0000","last_active":"2024-09-21T13:53:53.278749+0000","last_peered":"2024-09-21T13:53:53.278749+0000","last_clean":"2024-09-21T13:53:53.278749+0000","last_became_active":"2024-09-21T13:49:02.059638+0000","last_became_peered":"2024-09-21T13:49:02.059638+0000","last_unstale":"2024-09-21T13:53:53.278749+0000","last_undegraded":"2024-09-21T13:53:53.278749+0000","last_fullsized":"2024-09-21T13:53:53.278749+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:51:42.174629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:42.174629+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:08.238915+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"172'16784","reported_seq":17098,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.412501+0000","last_change":"2024-09-21T13:52:51.407976+0000","last_active":"2024-09-21T13:53:53.412501+0000","last_peered":"2024-09-21T13:53:53.412501+0000","last_clean":"2024-09-21T13:53:53.412501+0000","last_became_active":"2024-09-21T13:50:07.422961+0000","last_became_peered":"2024-09-21T13:50:07.422961+0000","last_unstale":"2024-09-21T13:53:53.412501+0000","last_undegraded":"2024-09-21T13:53:53.412501+0000","last_fullsized":"2024-09-21T13:53:53.412501+0000","mapping_epoch":83,"log_start":"132'10500","ondisk_log_start":"132'10500","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'13038","last_scrub_stamp":"2024-09-21T13:52:51.407818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:52:51.407818+0000","objects_scrubbed":4170,"log_size":6284,"log_dups_size":0,"ondisk_log_size":6284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:10.866985+0000","scrub_duration":960,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26476544,"num_objects":404,"num_object_clones":0,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16784,"num_write_kb":550016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"68'15","reported_seq":273,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.283075+0000","last_change":"2024-09-21T13:52:16.452049+0000","last_active":"2024-09-21T13:53:53.283075+0000","last_peered":"2024-09-21T13:53:53.283075+0000","last_clean":"2024-09-21T13:53:53.283075+0000","last_became_active":"2024-09-21T13:49:00.060324+0000","last_became_peered":"2024-09-21T13:49:00.060324+0000","last_unstale":"2024-09-21T13:53:53.283075+0000","last_undegraded":"2024-09-21T13:53:53.283075+0000","last_fullsized":"2024-09-21T13:53:53.283075+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:52:16.451964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:52:16.451964+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":244,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.283937+0000","last_change":"2024-09-21T13:50:06.151943+0000","last_active":"2024-09-21T13:53:53.283937+0000","last_peered":"2024-09-21T13:53:53.283937+0000","last_clean":"2024-09-21T13:53:53.283937+0000","last_became_active":"2024-09-21T13:48:58.051911+0000","last_became_peered":"2024-09-21T13:48:58.051911+0000","last_unstale":"2024-09-21T13:53:53.283937+0000","last_undegraded":"2024-09-21T13:53:53.283937+0000","last_fullsized":"2024-09-21T13:53:53.283937+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.151888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.151888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"68'8","reported_seq":307,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.274725+0000","last_change":"2024-09-21T13:53:19.021881+0000","last_active":"2024-09-21T13:53:53.274725+0000","last_peered":"2024-09-21T13:53:53.274725+0000","last_clean":"2024-09-21T13:53:53.274725+0000","last_became_active":"2024-09-21T13:49:02.061679+0000","last_became_peered":"2024-09-21T13:49:02.061679+0000","last_unstale":"2024-09-21T13:53:53.274725+0000","last_undegraded":"2024-09-21T13:53:53.274725+0000","last_fullsized":"2024-09-21T13:53:53.274725+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:53:19.021810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:53:19.021810+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:35.961578+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":232,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.288383+0000","last_change":"2024-09-21T13:50:34.481503+0000","last_active":"2024-09-21T13:53:53.288383+0000","last_peered":"2024-09-21T13:53:53.288383+0000","last_clean":"2024-09-21T13:53:53.288383+0000","last_became_active":"2024-09-21T13:49:04.081905+0000","last_became_peered":"2024-09-21T13:49:04.081905+0000","last_unstale":"2024-09-21T13:53:53.288383+0000","last_undegraded":"2024-09-21T13:53:53.288383+0000","last_fullsized":"2024-09-21T13:53:53.288383+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:34.481450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:34.481450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:43.214157+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"172'16880","reported_seq":24492,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.557994+0000","last_change":"2024-09-21T13:53:16.667415+0000","last_active":"2024-09-21T13:53:53.557994+0000","last_peered":"2024-09-21T13:53:53.557994+0000","last_clean":"2024-09-21T13:53:53.557994+0000","last_became_active":"2024-09-21T13:53:16.646871+0000","last_became_peered":"2024-09-21T13:53:16.646871+0000","last_unstale":"2024-09-21T13:53:53.557994+0000","last_undegraded":"2024-09-21T13:53:53.557994+0000","last_fullsized":"2024-09-21T13:53:53.557994+0000","mapping_epoch":159,"log_start":"141'12200","ondisk_log_start":"141'12200","created":83,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":4680,"log_dups_size":0,"ondisk_log_size":4680,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23461888,"num_objects":358,"num_object_clones":0,"num_object_copies":1074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16880,"num_write_kb":551616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3703,"num_bytes_recovered":242679808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"68'18","reported_seq":272,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.283646+0000","last_change":"2024-09-21T13:50:27.281455+0000","last_active":"2024-09-21T13:53:53.283646+0000","last_peered":"2024-09-21T13:53:53.283646+0000","last_clean":"2024-09-21T13:53:53.283646+0000","last_became_active":"2024-09-21T13:49:00.429314+0000","last_became_peered":"2024-09-21T13:49:00.429314+0000","last_unstale":"2024-09-21T13:53:53.283646+0000","last_undegraded":"2024-09-21T13:53:53.283646+0000","last_fullsized":"2024-09-21T13:53:53.283646+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'18","last_scrub_stamp":"2024-09-21T13:50:27.281356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:27.281356+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":236,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.284702+0000","last_change":"2024-09-21T13:48:58.056669+0000","last_active":"2024-09-21T13:53:53.284702+0000","last_peered":"2024-09-21T13:53:53.284702+0000","last_clean":"2024-09-21T13:53:53.284702+0000","last_became_active":"2024-09-21T13:48:58.056375+0000","last_became_peered":"2024-09-21T13:48:58.056375+0000","last_unstale":"2024-09-21T13:53:53.284702+0000","last_undegraded":"2024-09-21T13:53:53.284702+0000","last_fullsized":"2024-09-21T13:53:53.284702+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:34.834115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":236,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.275046+0000","last_change":"2024-09-21T13:51:23.940809+0000","last_active":"2024-09-21T13:53:53.275046+0000","last_peered":"2024-09-21T13:53:53.275046+0000","last_clean":"2024-09-21T13:53:53.275046+0000","last_became_active":"2024-09-21T13:49:02.062955+0000","last_became_peered":"2024-09-21T13:49:02.062955+0000","last_unstale":"2024-09-21T13:53:53.275046+0000","last_undegraded":"2024-09-21T13:53:53.275046+0000","last_fullsized":"2024-09-21T13:53:53.275046+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:23.940746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:23.940746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":235,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.273799+0000","last_change":"2024-09-21T13:52:04.316586+0000","last_active":"2024-09-21T13:53:53.273799+0000","last_peered":"2024-09-21T13:53:53.273799+0000","last_clean":"2024-09-21T13:53:53.273799+0000","last_became_active":"2024-09-21T13:52:04.316121+0000","last_became_peered":"2024-09-21T13:52:04.316121+0000","last_unstale":"2024-09-21T13:53:53.273799+0000","last_undegraded":"2024-09-21T13:53:53.273799+0000","last_fullsized":"2024-09-21T13:53:53.273799+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:10.130707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"171'16417","reported_seq":18506,"reported_epoch":171,"state":"peering","last_fresh":"2024-09-21T13:53:53.271304+0000","last_change":"2024-09-21T13:53:53.288163+0000","last_active":"2024-09-21T13:53:53.288163+0000","last_peered":"2024-09-21T13:53:53.271304+0000","last_clean":"2024-09-21T13:50:42.909755+0000","last_became_active":"2024-09-21T13:52:04.998636+0000","last_became_peered":"2024-09-21T13:52:04.998636+0000","last_unstale":"2024-09-21T13:53:53.271304+0000","last_undegraded":"2024-09-21T13:53:53.271304+0000","last_fullsized":"2024-09-21T13:52:04.863335+0000","mapping_epoch":172,"log_start":"141'11800","ondisk_log_start":"141'11800","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":4617,"log_dups_size":0,"ondisk_log_size":4617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27721728,"num_objects":423,"num_object_clones":0,"num_object_copies":1269,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16417,"num_write_kb":538880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":939,"num_bytes_recovered":61538304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"68'14","reported_seq":261,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.275186+0000","last_change":"2024-09-21T13:50:29.009786+0000","last_active":"2024-09-21T13:53:53.275186+0000","last_peered":"2024-09-21T13:53:53.275186+0000","last_clean":"2024-09-21T13:53:53.275186+0000","last_became_active":"2024-09-21T13:49:00.055031+0000","last_became_peered":"2024-09-21T13:49:00.055031+0000","last_unstale":"2024-09-21T13:53:53.275186+0000","last_undegraded":"2024-09-21T13:53:53.275186+0000","last_fullsized":"2024-09-21T13:53:53.275186+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'14","last_scrub_stamp":"2024-09-21T13:50:29.009720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:29.009720+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:41.285061+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":245,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.292936+0000","last_change":"2024-09-21T13:49:59.253636+0000","last_active":"2024-09-21T13:53:53.292936+0000","last_peered":"2024-09-21T13:53:53.292936+0000","last_clean":"2024-09-21T13:53:53.292936+0000","last_became_active":"2024-09-21T13:48:58.053533+0000","last_became_peered":"2024-09-21T13:48:58.053533+0000","last_unstale":"2024-09-21T13:53:53.292936+0000","last_undegraded":"2024-09-21T13:53:53.292936+0000","last_fullsized":"2024-09-21T13:53:53.292936+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:49:59.253587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:49:59.253587+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:36.768506+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"68'8","reported_seq":304,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.283373+0000","last_change":"2024-09-21T13:50:41.458637+0000","last_active":"2024-09-21T13:53:53.283373+0000","last_peered":"2024-09-21T13:53:53.283373+0000","last_clean":"2024-09-21T13:53:53.283373+0000","last_became_active":"2024-09-21T13:49:02.070064+0000","last_became_peered":"2024-09-21T13:49:02.070064+0000","last_unstale":"2024-09-21T13:53:53.283373+0000","last_undegraded":"2024-09-21T13:53:53.283373+0000","last_fullsized":"2024-09-21T13:53:53.283373+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:50:41.458562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:41.458562+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":243,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.281732+0000","last_change":"2024-09-21T13:52:55.541288+0000","last_active":"2024-09-21T13:53:53.281732+0000","last_peered":"2024-09-21T13:53:53.281732+0000","last_clean":"2024-09-21T13:53:53.281732+0000","last_became_active":"2024-09-21T13:52:04.322185+0000","last_became_peered":"2024-09-21T13:52:04.322185+0000","last_unstale":"2024-09-21T13:53:53.281732+0000","last_undegraded":"2024-09-21T13:53:53.281732+0000","last_fullsized":"2024-09-21T13:53:53.281732+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:55.541232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:55.541232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:57.661213+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"172'16538","reported_seq":16728,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.407352+0000","last_change":"2024-09-21T13:50:07.419221+0000","last_active":"2024-09-21T13:53:53.407352+0000","last_peered":"2024-09-21T13:53:53.407352+0000","last_clean":"2024-09-21T13:53:53.407352+0000","last_became_active":"2024-09-21T13:50:07.418929+0000","last_became_peered":"2024-09-21T13:50:07.418929+0000","last_unstale":"2024-09-21T13:53:53.407352+0000","last_undegraded":"2024-09-21T13:53:53.407352+0000","last_fullsized":"2024-09-21T13:53:53.407352+0000","mapping_epoch":83,"log_start":"139'11700","ondisk_log_start":"139'11700","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":4838,"log_dups_size":0,"ondisk_log_size":4838,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:40.293070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22675456,"num_objects":346,"num_object_clones":0,"num_object_copies":1038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16538,"num_write_kb":540288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"68'10","reported_seq":263,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.289594+0000","last_change":"2024-09-21T13:51:50.211365+0000","last_active":"2024-09-21T13:53:53.289594+0000","last_peered":"2024-09-21T13:53:53.289594+0000","last_clean":"2024-09-21T13:53:53.289594+0000","last_became_active":"2024-09-21T13:49:00.054564+0000","last_became_peered":"2024-09-21T13:49:00.054564+0000","last_unstale":"2024-09-21T13:53:53.289594+0000","last_undegraded":"2024-09-21T13:53:53.289594+0000","last_fullsized":"2024-09-21T13:53:53.289594+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:51:50.211279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:50.211279+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:15.800803+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"61'1","reported_seq":277,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.288961+0000","last_change":"2024-09-21T13:53:30.179679+0000","last_active":"2024-09-21T13:53:53.288961+0000","last_peered":"2024-09-21T13:53:53.288961+0000","last_clean":"2024-09-21T13:53:53.288961+0000","last_became_active":"2024-09-21T13:52:04.315788+0000","last_became_peered":"2024-09-21T13:52:04.315788+0000","last_unstale":"2024-09-21T13:53:53.288961+0000","last_undegraded":"2024-09-21T13:53:53.288961+0000","last_fullsized":"2024-09-21T13:53:53.288961+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:53:30.179605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:53:30.179605+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:43.052251+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"96'9","reported_seq":251,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.279235+0000","last_change":"2024-09-21T13:52:51.443040+0000","last_active":"2024-09-21T13:53:53.279235+0000","last_peered":"2024-09-21T13:53:53.279235+0000","last_clean":"2024-09-21T13:53:53.279235+0000","last_became_active":"2024-09-21T13:50:29.693557+0000","last_became_peered":"2024-09-21T13:50:29.693557+0000","last_unstale":"2024-09-21T13:53:53.279235+0000","last_undegraded":"2024-09-21T13:53:53.279235+0000","last_fullsized":"2024-09-21T13:53:53.279235+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'9","last_scrub_stamp":"2024-09-21T13:52:51.442930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:52:51.442930+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:04.534122+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":179,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.283251+0000","last_change":"2024-09-21T13:52:04.319323+0000","last_active":"2024-09-21T13:53:53.283251+0000","last_peered":"2024-09-21T13:53:53.283251+0000","last_clean":"2024-09-21T13:53:53.283251+0000","last_became_active":"2024-09-21T13:52:04.319034+0000","last_became_peered":"2024-09-21T13:52:04.319034+0000","last_unstale":"2024-09-21T13:53:53.283251+0000","last_undegraded":"2024-09-21T13:53:53.283251+0000","last_fullsized":"2024-09-21T13:53:53.283251+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"172'16722","reported_seq":16983,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.386580+0000","last_change":"2024-09-21T13:52:16.618637+0000","last_active":"2024-09-21T13:53:53.386580+0000","last_peered":"2024-09-21T13:53:53.386580+0000","last_clean":"2024-09-21T13:53:53.386580+0000","last_became_active":"2024-09-21T13:50:07.423702+0000","last_became_peered":"2024-09-21T13:50:07.423702+0000","last_unstale":"2024-09-21T13:53:53.386580+0000","last_undegraded":"2024-09-21T13:53:53.386580+0000","last_fullsized":"2024-09-21T13:53:53.386580+0000","mapping_epoch":83,"log_start":"134'10900","ondisk_log_start":"134'10900","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'10945","last_scrub_stamp":"2024-09-21T13:52:16.618486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:52:16.618486+0000","objects_scrubbed":6221,"log_size":5822,"log_dups_size":0,"ondisk_log_size":5822,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:44.257271+0000","scrub_duration":1967,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25296896,"num_objects":386,"num_object_clones":0,"num_object_copies":1158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16722,"num_write_kb":547456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"68'19","reported_seq":232,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.288745+0000","last_change":"2024-09-21T13:50:32.514267+0000","last_active":"2024-09-21T13:53:53.288745+0000","last_peered":"2024-09-21T13:53:53.288745+0000","last_clean":"2024-09-21T13:53:53.288745+0000","last_became_active":"2024-09-21T13:50:29.690185+0000","last_became_peered":"2024-09-21T13:50:29.690185+0000","last_unstale":"2024-09-21T13:53:53.288745+0000","last_undegraded":"2024-09-21T13:53:53.288745+0000","last_fullsized":"2024-09-21T13:53:53.288745+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-21T13:50:32.514210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:32.514210+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:35.217090+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":247,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.288899+0000","last_change":"2024-09-21T13:52:04.322289+0000","last_active":"2024-09-21T13:53:53.288899+0000","last_peered":"2024-09-21T13:53:53.288899+0000","last_clean":"2024-09-21T13:53:53.288899+0000","last_became_active":"2024-09-21T13:52:04.321656+0000","last_became_peered":"2024-09-21T13:52:04.321656+0000","last_unstale":"2024-09-21T13:53:53.288899+0000","last_undegraded":"2024-09-21T13:53:53.288899+0000","last_fullsized":"2024-09-21T13:53:53.288899+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:08.762218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":244,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.277701+0000","last_change":"2024-09-21T13:53:32.295930+0000","last_active":"2024-09-21T13:53:53.277701+0000","last_peered":"2024-09-21T13:53:53.277701+0000","last_clean":"2024-09-21T13:53:53.277701+0000","last_became_active":"2024-09-21T13:49:02.061187+0000","last_became_peered":"2024-09-21T13:49:02.061187+0000","last_unstale":"2024-09-21T13:53:53.277701+0000","last_undegraded":"2024-09-21T13:53:53.277701+0000","last_fullsized":"2024-09-21T13:53:53.277701+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:32.295869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:53:32.295869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:32.443805+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":235,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.292023+0000","last_change":"2024-09-21T13:52:04.321133+0000","last_active":"2024-09-21T13:53:53.292023+0000","last_peered":"2024-09-21T13:53:53.292023+0000","last_clean":"2024-09-21T13:53:53.292023+0000","last_became_active":"2024-09-21T13:52:04.320826+0000","last_became_peered":"2024-09-21T13:52:04.320826+0000","last_unstale":"2024-09-21T13:53:53.292023+0000","last_undegraded":"2024-09-21T13:53:53.292023+0000","last_fullsized":"2024-09-21T13:53:53.292023+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:09.756023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"172'16776","reported_seq":17018,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.414498+0000","last_change":"2024-09-21T13:52:49.375801+0000","last_active":"2024-09-21T13:53:53.414498+0000","last_peered":"2024-09-21T13:53:53.414498+0000","last_clean":"2024-09-21T13:53:53.414498+0000","last_became_active":"2024-09-21T13:50:07.421666+0000","last_became_peered":"2024-09-21T13:50:07.421666+0000","last_unstale":"2024-09-21T13:53:53.414498+0000","last_undegraded":"2024-09-21T13:53:53.414498+0000","last_fullsized":"2024-09-21T13:53:53.414498+0000","mapping_epoch":83,"log_start":"132'10500","ondisk_log_start":"132'10500","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"149'12949","last_scrub_stamp":"2024-09-21T13:52:49.375671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:52:49.375671+0000","objects_scrubbed":4241,"log_size":6276,"log_dups_size":0,"ondisk_log_size":6276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:54.416521+0000","scrub_duration":1855,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":384,"num_object_clones":0,"num_object_copies":1152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16776,"num_write_kb":549120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"132'30","reported_seq":353,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.288738+0000","last_change":"2024-09-21T13:52:04.322738+0000","last_active":"2024-09-21T13:53:53.288738+0000","last_peered":"2024-09-21T13:53:53.288738+0000","last_clean":"2024-09-21T13:53:53.288738+0000","last_became_active":"2024-09-21T13:52:04.319431+0000","last_became_peered":"2024-09-21T13:52:04.319431+0000","last_unstale":"2024-09-21T13:53:53.288738+0000","last_undegraded":"2024-09-21T13:53:53.288738+0000","last_fullsized":"2024-09-21T13:53:53.288738+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:12.782459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":236,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.273868+0000","last_change":"2024-09-21T13:48:58.055640+0000","last_active":"2024-09-21T13:53:53.273868+0000","last_peered":"2024-09-21T13:53:53.273868+0000","last_clean":"2024-09-21T13:53:53.273868+0000","last_became_active":"2024-09-21T13:48:58.055366+0000","last_became_peered":"2024-09-21T13:48:58.055366+0000","last_unstale":"2024-09-21T13:53:53.273868+0000","last_undegraded":"2024-09-21T13:53:53.273868+0000","last_fullsized":"2024-09-21T13:53:53.273868+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:09.147679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":262,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.288366+0000","last_change":"2024-09-21T13:52:51.445882+0000","last_active":"2024-09-21T13:53:53.288366+0000","last_peered":"2024-09-21T13:53:53.288366+0000","last_clean":"2024-09-21T13:53:53.288366+0000","last_became_active":"2024-09-21T13:48:57.046246+0000","last_became_peered":"2024-09-21T13:48:57.046246+0000","last_unstale":"2024-09-21T13:53:53.288366+0000","last_undegraded":"2024-09-21T13:53:53.288366+0000","last_fullsized":"2024-09-21T13:53:53.288366+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-21T13:52:51.445812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:52:51.445812+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:18.734189+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.0015946669999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":176,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.283094+0000","last_change":"2024-09-21T13:51:24.187154+0000","last_active":"2024-09-21T13:53:53.283094+0000","last_peered":"2024-09-21T13:53:53.283094+0000","last_clean":"2024-09-21T13:53:53.283094+0000","last_became_active":"2024-09-21T13:50:29.688249+0000","last_became_peered":"2024-09-21T13:50:29.688249+0000","last_unstale":"2024-09-21T13:53:53.283094+0000","last_undegraded":"2024-09-21T13:53:53.283094+0000","last_fullsized":"2024-09-21T13:53:53.283094+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:24.186934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:24.186934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":232,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.283380+0000","last_change":"2024-09-21T13:51:23.597164+0000","last_active":"2024-09-21T13:53:53.283380+0000","last_peered":"2024-09-21T13:53:53.283380+0000","last_clean":"2024-09-21T13:53:53.283380+0000","last_became_active":"2024-09-21T13:49:04.080466+0000","last_became_peered":"2024-09-21T13:49:04.080466+0000","last_unstale":"2024-09-21T13:53:53.283380+0000","last_undegraded":"2024-09-21T13:53:53.283380+0000","last_fullsized":"2024-09-21T13:53:53.283380+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:23.596879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:51:23.596879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"172'16907","reported_seq":17102,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.354779+0000","last_change":"2024-09-21T13:50:29.694240+0000","last_active":"2024-09-21T13:53:53.354779+0000","last_peered":"2024-09-21T13:53:53.354779+0000","last_clean":"2024-09-21T13:53:53.354779+0000","last_became_active":"2024-09-21T13:50:29.688167+0000","last_became_peered":"2024-09-21T13:50:29.688167+0000","last_unstale":"2024-09-21T13:53:53.354779+0000","last_undegraded":"2024-09-21T13:53:53.354779+0000","last_fullsized":"2024-09-21T13:53:53.354779+0000","mapping_epoch":94,"log_start":"141'12100","ondisk_log_start":"141'12100","created":83,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":4807,"log_dups_size":0,"ondisk_log_size":4807,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:37.270741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26935296,"num_objects":411,"num_object_clones":0,"num_object_copies":1233,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16907,"num_write_kb":554176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"68'13","reported_seq":267,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.282832+0000","last_change":"2024-09-21T13:50:06.973399+0000","last_active":"2024-09-21T13:53:53.282832+0000","last_peered":"2024-09-21T13:53:53.282832+0000","last_clean":"2024-09-21T13:53:53.282832+0000","last_became_active":"2024-09-21T13:49:00.053323+0000","last_became_peered":"2024-09-21T13:49:00.053323+0000","last_unstale":"2024-09-21T13:53:53.282832+0000","last_undegraded":"2024-09-21T13:53:53.282832+0000","last_fullsized":"2024-09-21T13:53:53.282832+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'13","last_scrub_stamp":"2024-09-21T13:50:06.973336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.973336+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:41.649253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":271,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.290352+0000","last_change":"2024-09-21T13:53:24.885266+0000","last_active":"2024-09-21T13:53:53.290352+0000","last_peered":"2024-09-21T13:53:53.290352+0000","last_clean":"2024-09-21T13:53:53.290352+0000","last_became_active":"2024-09-21T13:52:04.320581+0000","last_became_peered":"2024-09-21T13:52:04.320581+0000","last_unstale":"2024-09-21T13:53:53.290352+0000","last_undegraded":"2024-09-21T13:53:53.290352+0000","last_fullsized":"2024-09-21T13:53:53.290352+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:24.884932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:53:24.884932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:31.375453+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":174,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-21T13:53:54.925776+0000","last_change":"2024-09-21T13:53:19.985044+0000","last_active":"2024-09-21T13:53:54.925776+0000","last_peered":"2024-09-21T13:53:54.925776+0000","last_clean":"2024-09-21T13:53:54.925776+0000","last_became_active":"2024-09-21T13:52:04.316171+0000","last_became_peered":"2024-09-21T13:52:04.316171+0000","last_unstale":"2024-09-21T13:53:54.925776+0000","last_undegraded":"2024-09-21T13:53:54.925776+0000","last_fullsized":"2024-09-21T13:53:54.925776+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:53:19.984966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:53:19.984966+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:47.985977+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":118,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-21T13:53:54.930260+0000","last_change":"2024-09-21T13:53:18.988030+0000","last_active":"2024-09-21T13:53:54.930260+0000","last_peered":"2024-09-21T13:53:54.930260+0000","last_clean":"2024-09-21T13:53:54.930260+0000","last_became_active":"2024-09-21T13:52:04.320701+0000","last_became_peered":"2024-09-21T13:52:04.320701+0000","last_unstale":"2024-09-21T13:53:54.930260+0000","last_undegraded":"2024-09-21T13:53:54.930260+0000","last_fullsized":"2024-09-21T13:53:54.930260+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:18.987829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:53:18.987829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:21.866149+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":240,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.282489+0000","last_change":"2024-09-21T13:51:12.212558+0000","last_active":"2024-09-21T13:53:53.282489+0000","last_peered":"2024-09-21T13:53:53.282489+0000","last_clean":"2024-09-21T13:53:53.282489+0000","last_became_active":"2024-09-21T13:49:04.431035+0000","last_became_peered":"2024-09-21T13:49:04.431035+0000","last_unstale":"2024-09-21T13:53:53.282489+0000","last_undegraded":"2024-09-21T13:53:53.282489+0000","last_fullsized":"2024-09-21T13:53:53.282489+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:12.212391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:51:12.212391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"172'16687","reported_seq":26870,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.433600+0000","last_change":"2024-09-21T13:53:21.165869+0000","last_active":"2024-09-21T13:53:53.433600+0000","last_peered":"2024-09-21T13:53:53.433600+0000","last_clean":"2024-09-21T13:53:53.433600+0000","last_became_active":"2024-09-21T13:52:43.938223+0000","last_became_peered":"2024-09-21T13:52:43.938223+0000","last_unstale":"2024-09-21T13:53:53.433600+0000","last_undegraded":"2024-09-21T13:53:53.433600+0000","last_fullsized":"2024-09-21T13:53:53.433600+0000","mapping_epoch":147,"log_start":"138'11500","ondisk_log_start":"138'11500","created":83,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"162'14639","last_scrub_stamp":"2024-09-21T13:53:21.165793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:53:21.165793+0000","objects_scrubbed":2453,"log_size":5187,"log_dups_size":0,"ondisk_log_size":5187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:29.496429+0000","scrub_duration":804,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24576000,"num_objects":375,"num_object_clones":0,"num_object_copies":1125,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16687,"num_write_kb":545984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4971,"num_bytes_recovered":325779456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"68'12","reported_seq":298,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.292849+0000","last_change":"2024-09-21T13:53:17.740156+0000","last_active":"2024-09-21T13:53:53.292849+0000","last_peered":"2024-09-21T13:53:53.292849+0000","last_clean":"2024-09-21T13:53:53.292849+0000","last_became_active":"2024-09-21T13:52:04.318875+0000","last_became_peered":"2024-09-21T13:52:04.318875+0000","last_unstale":"2024-09-21T13:53:53.292849+0000","last_undegraded":"2024-09-21T13:53:53.292849+0000","last_fullsized":"2024-09-21T13:53:53.292849+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'12","last_scrub_stamp":"2024-09-21T13:53:17.740080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:17.740080+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:30.638014+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":176,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.275336+0000","last_change":"2024-09-21T13:50:30.020722+0000","last_active":"2024-09-21T13:53:53.275336+0000","last_peered":"2024-09-21T13:53:53.275336+0000","last_clean":"2024-09-21T13:53:53.275336+0000","last_became_active":"2024-09-21T13:50:29.688352+0000","last_became_peered":"2024-09-21T13:50:29.688352+0000","last_unstale":"2024-09-21T13:53:53.275336+0000","last_undegraded":"2024-09-21T13:53:53.275336+0000","last_fullsized":"2024-09-21T13:53:53.275336+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:30.020656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:30.020656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:38.234245+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":545,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.283287+0000","last_change":"2024-09-21T13:51:21.271997+0000","last_active":"2024-09-21T13:53:53.283287+0000","last_peered":"2024-09-21T13:53:53.283287+0000","last_clean":"2024-09-21T13:53:53.283287+0000","last_became_active":"2024-09-21T13:48:57.046084+0000","last_became_peered":"2024-09-21T13:48:57.046084+0000","last_unstale":"2024-09-21T13:53:53.283287+0000","last_undegraded":"2024-09-21T13:53:53.283287+0000","last_fullsized":"2024-09-21T13:53:53.283287+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-21T13:51:21.271917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:51:21.271917+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00047048000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":290,"num_read_kb":285,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":236,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.288953+0000","last_change":"2024-09-21T13:51:53.518645+0000","last_active":"2024-09-21T13:53:53.288953+0000","last_peered":"2024-09-21T13:53:53.288953+0000","last_clean":"2024-09-21T13:53:53.288953+0000","last_became_active":"2024-09-21T13:49:02.066641+0000","last_became_peered":"2024-09-21T13:49:02.066641+0000","last_unstale":"2024-09-21T13:53:53.288953+0000","last_undegraded":"2024-09-21T13:53:53.288953+0000","last_fullsized":"2024-09-21T13:53:53.288953+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:53.518567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:53.518567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:14.335443+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":240,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.282566+0000","last_change":"2024-09-21T13:52:21.332066+0000","last_active":"2024-09-21T13:53:53.282566+0000","last_peered":"2024-09-21T13:53:53.282566+0000","last_clean":"2024-09-21T13:53:53.282566+0000","last_became_active":"2024-09-21T13:49:04.078372+0000","last_became_peered":"2024-09-21T13:49:04.078372+0000","last_unstale":"2024-09-21T13:53:53.282566+0000","last_undegraded":"2024-09-21T13:53:53.282566+0000","last_fullsized":"2024-09-21T13:53:53.282566+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:21.331792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:21.331792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"172'16797","reported_seq":17021,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.430519+0000","last_change":"2024-09-21T13:53:27.064451+0000","last_active":"2024-09-21T13:53:53.430519+0000","last_peered":"2024-09-21T13:53:53.430519+0000","last_clean":"2024-09-21T13:53:53.430519+0000","last_became_active":"2024-09-21T13:50:29.694886+0000","last_became_peered":"2024-09-21T13:50:29.694886+0000","last_unstale":"2024-09-21T13:53:53.430519+0000","last_undegraded":"2024-09-21T13:53:53.430519+0000","last_fullsized":"2024-09-21T13:53:53.430519+0000","mapping_epoch":94,"log_start":"137'11600","ondisk_log_start":"137'11600","created":83,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'15120","last_scrub_stamp":"2024-09-21T13:53:27.064254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:53:27.064254+0000","objects_scrubbed":2095,"log_size":5197,"log_dups_size":0,"ondisk_log_size":5197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:50.816602+0000","scrub_duration":619,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26017792,"num_objects":397,"num_object_clones":0,"num_object_copies":1191,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16797,"num_write_kb":550208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"68'16","reported_seq":330,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.278470+0000","last_change":"2024-09-21T13:52:19.452239+0000","last_active":"2024-09-21T13:53:53.278470+0000","last_peered":"2024-09-21T13:53:53.278470+0000","last_clean":"2024-09-21T13:53:53.278470+0000","last_became_active":"2024-09-21T13:49:00.426867+0000","last_became_peered":"2024-09-21T13:49:00.426867+0000","last_unstale":"2024-09-21T13:53:53.278470+0000","last_undegraded":"2024-09-21T13:53:53.278470+0000","last_fullsized":"2024-09-21T13:53:53.278470+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'16","last_scrub_stamp":"2024-09-21T13:52:19.452163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:52:19.452163+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:49.265943+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":244,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.287458+0000","last_change":"2024-09-21T13:50:09.208053+0000","last_active":"2024-09-21T13:53:53.287458+0000","last_peered":"2024-09-21T13:53:53.287458+0000","last_clean":"2024-09-21T13:53:53.287458+0000","last_became_active":"2024-09-21T13:48:58.052659+0000","last_became_peered":"2024-09-21T13:48:58.052659+0000","last_unstale":"2024-09-21T13:53:53.287458+0000","last_undegraded":"2024-09-21T13:53:53.287458+0000","last_fullsized":"2024-09-21T13:53:53.287458+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:09.208003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:09.208003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"75'79","reported_seq":293,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.283414+0000","last_change":"2024-09-21T13:50:27.096470+0000","last_active":"2024-09-21T13:53:53.283414+0000","last_peered":"2024-09-21T13:53:53.283414+0000","last_clean":"2024-09-21T13:53:53.283414+0000","last_became_active":"2024-09-21T13:48:48.454646+0000","last_became_peered":"2024-09-21T13:48:48.454646+0000","last_unstale":"2024-09-21T13:53:53.283414+0000","last_undegraded":"2024-09-21T13:53:53.283414+0000","last_fullsized":"2024-09-21T13:53:53.283414+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:50:27.096413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:50:27.096413+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:37.381057+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":239,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.283320+0000","last_change":"2024-09-21T13:52:04.320948+0000","last_active":"2024-09-21T13:53:53.283320+0000","last_peered":"2024-09-21T13:53:53.283320+0000","last_clean":"2024-09-21T13:53:53.283320+0000","last_became_active":"2024-09-21T13:52:04.320627+0000","last_became_peered":"2024-09-21T13:52:04.320627+0000","last_unstale":"2024-09-21T13:53:53.283320+0000","last_undegraded":"2024-09-21T13:53:53.283320+0000","last_fullsized":"2024-09-21T13:53:53.283320+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:50.544725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":232,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.288638+0000","last_change":"2024-09-21T13:50:25.560928+0000","last_active":"2024-09-21T13:53:53.288638+0000","last_peered":"2024-09-21T13:53:53.288638+0000","last_clean":"2024-09-21T13:53:53.288638+0000","last_became_active":"2024-09-21T13:49:04.078091+0000","last_became_peered":"2024-09-21T13:49:04.078091+0000","last_unstale":"2024-09-21T13:53:53.288638+0000","last_undegraded":"2024-09-21T13:53:53.288638+0000","last_fullsized":"2024-09-21T13:53:53.288638+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:25.560863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:25.560863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:41.214571+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"172'16689","reported_seq":17008,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.401989+0000","last_change":"2024-09-21T13:52:50.770818+0000","last_active":"2024-09-21T13:53:53.401989+0000","last_peered":"2024-09-21T13:53:53.401989+0000","last_clean":"2024-09-21T13:53:53.401989+0000","last_became_active":"2024-09-21T13:50:07.422466+0000","last_became_peered":"2024-09-21T13:50:07.422466+0000","last_unstale":"2024-09-21T13:53:53.401989+0000","last_undegraded":"2024-09-21T13:53:53.401989+0000","last_fullsized":"2024-09-21T13:53:53.401989+0000","mapping_epoch":83,"log_start":"132'10400","ondisk_log_start":"132'10400","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'12965","last_scrub_stamp":"2024-09-21T13:52:50.770670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:52:50.770670+0000","objects_scrubbed":4108,"log_size":6289,"log_dups_size":0,"ondisk_log_size":6289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:56.210477+0000","scrub_duration":1333,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22544408,"num_objects":345,"num_object_clones":0,"num_object_copies":1035,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":345,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16689,"num_write_kb":545025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"68'11","reported_seq":251,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.274792+0000","last_change":"2024-09-21T13:49:00.054827+0000","last_active":"2024-09-21T13:53:53.274792+0000","last_peered":"2024-09-21T13:53:53.274792+0000","last_clean":"2024-09-21T13:53:53.274792+0000","last_became_active":"2024-09-21T13:49:00.053268+0000","last_became_peered":"2024-09-21T13:49:00.053268+0000","last_unstale":"2024-09-21T13:53:53.274792+0000","last_undegraded":"2024-09-21T13:53:53.274792+0000","last_fullsized":"2024-09-21T13:53:53.274792+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:48:59.038826+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:16.039942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":255,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.275536+0000","last_change":"2024-09-21T13:52:48.165210+0000","last_active":"2024-09-21T13:53:53.275536+0000","last_peered":"2024-09-21T13:53:53.275536+0000","last_clean":"2024-09-21T13:53:53.275536+0000","last_became_active":"2024-09-21T13:52:04.318345+0000","last_became_peered":"2024-09-21T13:52:04.318345+0000","last_unstale":"2024-09-21T13:53:53.275536+0000","last_undegraded":"2024-09-21T13:53:53.275536+0000","last_fullsized":"2024-09-21T13:53:53.275536+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:48.165145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:52:48.165145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:03.774580+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":244,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.289264+0000","last_change":"2024-09-21T13:51:46.503640+0000","last_active":"2024-09-21T13:53:53.289264+0000","last_peered":"2024-09-21T13:53:53.289264+0000","last_clean":"2024-09-21T13:53:53.289264+0000","last_became_active":"2024-09-21T13:49:02.061901+0000","last_became_peered":"2024-09-21T13:49:02.061901+0000","last_unstale":"2024-09-21T13:53:53.289264+0000","last_undegraded":"2024-09-21T13:53:53.289264+0000","last_fullsized":"2024-09-21T13:53:53.289264+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:46.503576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:46.503576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:11.300124+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":232,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.283264+0000","last_change":"2024-09-21T13:51:51.814259+0000","last_active":"2024-09-21T13:53:53.283264+0000","last_peered":"2024-09-21T13:53:53.283264+0000","last_clean":"2024-09-21T13:53:53.283264+0000","last_became_active":"2024-09-21T13:49:04.431268+0000","last_became_peered":"2024-09-21T13:49:04.431268+0000","last_unstale":"2024-09-21T13:53:53.283264+0000","last_undegraded":"2024-09-21T13:53:53.283264+0000","last_fullsized":"2024-09-21T13:53:53.283264+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:51.814200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:51:51.814200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"174'16809","reported_seq":24258,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-21T13:53:57.435914+0000","last_change":"2024-09-21T13:53:29.631333+0000","last_active":"2024-09-21T13:53:57.435914+0000","last_peered":"2024-09-21T13:53:57.435914+0000","last_clean":"2024-09-21T13:53:57.435914+0000","last_became_active":"2024-09-21T13:53:17.659357+0000","last_became_peered":"2024-09-21T13:53:17.659357+0000","last_unstale":"2024-09-21T13:53:57.435914+0000","last_undegraded":"2024-09-21T13:53:57.435914+0000","last_fullsized":"2024-09-21T13:53:57.435914+0000","mapping_epoch":160,"log_start":"128'9988","ondisk_log_start":"128'9988","created":83,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'15154","last_scrub_stamp":"2024-09-21T13:53:29.631248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:53:29.631248+0000","objects_scrubbed":1863,"log_size":6821,"log_dups_size":0,"ondisk_log_size":6821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:49.019026+0000","scrub_duration":768,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11862016,"num_objects":181,"num_object_clones":0,"num_object_copies":543,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16809,"num_write_kb":543680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3606,"num_bytes_recovered":236322816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"68'17","reported_seq":279,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.275145+0000","last_change":"2024-09-21T13:52:45.176950+0000","last_active":"2024-09-21T13:53:53.275145+0000","last_peered":"2024-09-21T13:53:53.275145+0000","last_clean":"2024-09-21T13:53:53.275145+0000","last_became_active":"2024-09-21T13:52:04.313218+0000","last_became_peered":"2024-09-21T13:52:04.313218+0000","last_unstale":"2024-09-21T13:53:53.275145+0000","last_undegraded":"2024-09-21T13:53:53.275145+0000","last_fullsized":"2024-09-21T13:53:53.275145+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'17","last_scrub_stamp":"2024-09-21T13:52:45.176873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:52:45.176873+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:03.832378+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":252,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.278091+0000","last_change":"2024-09-21T13:52:45.444445+0000","last_active":"2024-09-21T13:53:53.278091+0000","last_peered":"2024-09-21T13:53:53.278091+0000","last_clean":"2024-09-21T13:53:53.278091+0000","last_became_active":"2024-09-21T13:48:58.049142+0000","last_became_peered":"2024-09-21T13:48:58.049142+0000","last_unstale":"2024-09-21T13:53:53.278091+0000","last_undegraded":"2024-09-21T13:53:53.278091+0000","last_fullsized":"2024-09-21T13:53:53.278091+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:45.444381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:52:45.444381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:54.348542+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":228,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.282679+0000","last_change":"2024-09-21T13:49:02.067811+0000","last_active":"2024-09-21T13:53:53.282679+0000","last_peered":"2024-09-21T13:53:53.282679+0000","last_clean":"2024-09-21T13:53:53.282679+0000","last_became_active":"2024-09-21T13:49:02.067472+0000","last_became_peered":"2024-09-21T13:49:02.067472+0000","last_unstale":"2024-09-21T13:53:53.282679+0000","last_undegraded":"2024-09-21T13:53:53.282679+0000","last_fullsized":"2024-09-21T13:53:53.282679+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:02.701863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":118,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-21T13:53:54.926870+0000","last_change":"2024-09-21T13:53:18.028013+0000","last_active":"2024-09-21T13:53:54.926870+0000","last_peered":"2024-09-21T13:53:54.926870+0000","last_clean":"2024-09-21T13:53:54.926870+0000","last_became_active":"2024-09-21T13:52:04.315920+0000","last_became_peered":"2024-09-21T13:52:04.315920+0000","last_unstale":"2024-09-21T13:53:54.926870+0000","last_undegraded":"2024-09-21T13:53:54.926870+0000","last_fullsized":"2024-09-21T13:53:54.926870+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:18.027944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:18.027944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:26.231349+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"172'16269","reported_seq":16466,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.531574+0000","last_change":"2024-09-21T13:50:29.691429+0000","last_active":"2024-09-21T13:53:53.531574+0000","last_peered":"2024-09-21T13:53:53.531574+0000","last_clean":"2024-09-21T13:53:53.531574+0000","last_became_active":"2024-09-21T13:50:29.687898+0000","last_became_peered":"2024-09-21T13:50:29.687898+0000","last_unstale":"2024-09-21T13:53:53.531574+0000","last_undegraded":"2024-09-21T13:53:53.531574+0000","last_fullsized":"2024-09-21T13:53:53.531574+0000","mapping_epoch":94,"log_start":"141'11600","ondisk_log_start":"141'11600","created":83,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":4669,"log_dups_size":0,"ondisk_log_size":4669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22085632,"num_objects":337,"num_object_clones":0,"num_object_copies":1011,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16269,"num_write_kb":531392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"68'10","reported_seq":255,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.275896+0000","last_change":"2024-09-21T13:51:57.155811+0000","last_active":"2024-09-21T13:53:53.275896+0000","last_peered":"2024-09-21T13:53:53.275896+0000","last_clean":"2024-09-21T13:53:53.275896+0000","last_became_active":"2024-09-21T13:49:00.059850+0000","last_became_peered":"2024-09-21T13:49:00.059850+0000","last_unstale":"2024-09-21T13:53:53.275896+0000","last_undegraded":"2024-09-21T13:53:53.275896+0000","last_fullsized":"2024-09-21T13:53:53.275896+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:51:57.155761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:57.155761+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":244,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.286320+0000","last_change":"2024-09-21T13:51:28.565842+0000","last_active":"2024-09-21T13:53:53.286320+0000","last_peered":"2024-09-21T13:53:53.286320+0000","last_clean":"2024-09-21T13:53:53.286320+0000","last_became_active":"2024-09-21T13:48:58.058309+0000","last_became_peered":"2024-09-21T13:48:58.058309+0000","last_unstale":"2024-09-21T13:53:53.286320+0000","last_undegraded":"2024-09-21T13:53:53.286320+0000","last_fullsized":"2024-09-21T13:53:53.286320+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:28.565766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:51:28.565766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"132'10","reported_seq":151,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-21T13:53:54.927987+0000","last_change":"2024-09-21T13:52:04.320834+0000","last_active":"2024-09-21T13:53:54.927987+0000","last_peered":"2024-09-21T13:53:54.927987+0000","last_clean":"2024-09-21T13:53:54.927987+0000","last_became_active":"2024-09-21T13:52:04.320441+0000","last_became_peered":"2024-09-21T13:52:04.320441+0000","last_unstale":"2024-09-21T13:53:54.927987+0000","last_undegraded":"2024-09-21T13:53:54.927987+0000","last_fullsized":"2024-09-21T13:53:54.927987+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:16.073559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"68'1","reported_seq":247,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.274525+0000","last_change":"2024-09-21T13:52:04.314151+0000","last_active":"2024-09-21T13:53:53.274525+0000","last_peered":"2024-09-21T13:53:53.274525+0000","last_clean":"2024-09-21T13:53:53.274525+0000","last_became_active":"2024-09-21T13:52:04.313210+0000","last_became_peered":"2024-09-21T13:52:04.313210+0000","last_unstale":"2024-09-21T13:53:53.274525+0000","last_undegraded":"2024-09-21T13:53:53.274525+0000","last_fullsized":"2024-09-21T13:53:53.274525+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T13:50:41.000201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:41.000201+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:43.256736+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"174'16776","reported_seq":26895,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-21T13:53:57.394626+0000","last_change":"2024-09-21T13:53:31.637036+0000","last_active":"2024-09-21T13:53:57.394626+0000","last_peered":"2024-09-21T13:53:57.394626+0000","last_clean":"2024-09-21T13:53:57.394626+0000","last_became_active":"2024-09-21T13:52:42.942510+0000","last_became_peered":"2024-09-21T13:52:42.942510+0000","last_unstale":"2024-09-21T13:53:57.394626+0000","last_undegraded":"2024-09-21T13:53:57.394626+0000","last_fullsized":"2024-09-21T13:53:57.394626+0000","mapping_epoch":146,"log_start":"127'9878","ondisk_log_start":"127'9878","created":83,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'15232","last_scrub_stamp":"2024-09-21T13:53:31.636880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:53:31.636880+0000","objects_scrubbed":1782,"log_size":6898,"log_dups_size":0,"ondisk_log_size":6898,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:43.661832+0000","scrub_duration":726,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14548992,"num_objects":222,"num_object_clones":0,"num_object_copies":666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16776,"num_write_kb":543936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4942,"num_bytes_recovered":323878912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"68'9","reported_seq":264,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.285459+0000","last_change":"2024-09-21T13:53:18.740202+0000","last_active":"2024-09-21T13:53:53.285459+0000","last_peered":"2024-09-21T13:53:53.285459+0000","last_clean":"2024-09-21T13:53:53.285459+0000","last_became_active":"2024-09-21T13:49:00.053016+0000","last_became_peered":"2024-09-21T13:49:00.053016+0000","last_unstale":"2024-09-21T13:53:53.285459+0000","last_undegraded":"2024-09-21T13:53:53.285459+0000","last_fullsized":"2024-09-21T13:53:53.285459+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:53:18.740111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:18.740111+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:34.752035+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":244,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.289202+0000","last_change":"2024-09-21T13:50:38.445732+0000","last_active":"2024-09-21T13:53:53.289202+0000","last_peered":"2024-09-21T13:53:53.289202+0000","last_clean":"2024-09-21T13:53:53.289202+0000","last_became_active":"2024-09-21T13:48:58.049543+0000","last_became_peered":"2024-09-21T13:48:58.049543+0000","last_unstale":"2024-09-21T13:53:53.289202+0000","last_undegraded":"2024-09-21T13:53:53.289202+0000","last_fullsized":"2024-09-21T13:53:53.289202+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:38.445667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:38.445667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:36.189674+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":166,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.273554+0000","last_change":"2024-09-21T13:52:59.112648+0000","last_active":"2024-09-21T13:53:53.273554+0000","last_peered":"2024-09-21T13:53:53.273554+0000","last_clean":"2024-09-21T13:53:53.273554+0000","last_became_active":"2024-09-21T13:52:04.317429+0000","last_became_peered":"2024-09-21T13:52:04.317429+0000","last_unstale":"2024-09-21T13:53:53.273554+0000","last_undegraded":"2024-09-21T13:53:53.273554+0000","last_fullsized":"2024-09-21T13:53:53.273554+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:59.112602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:52:59.112602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:53:59.206422+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":187,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.280945+0000","last_change":"2024-09-21T13:52:51.590860+0000","last_active":"2024-09-21T13:53:53.280945+0000","last_peered":"2024-09-21T13:53:53.280945+0000","last_clean":"2024-09-21T13:53:53.280945+0000","last_became_active":"2024-09-21T13:52:04.321703+0000","last_became_peered":"2024-09-21T13:52:04.321703+0000","last_unstale":"2024-09-21T13:53:53.280945+0000","last_undegraded":"2024-09-21T13:53:53.280945+0000","last_fullsized":"2024-09-21T13:53:53.280945+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:51.590800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:51.590800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:01.244154+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.9","version":"172'16642","reported_seq":16839,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.547272+0000","last_change":"2024-09-21T13:50:29.690666+0000","last_active":"2024-09-21T13:53:53.547272+0000","last_peered":"2024-09-21T13:53:53.547272+0000","last_clean":"2024-09-21T13:53:53.547272+0000","last_became_active":"2024-09-21T13:50:29.687815+0000","last_became_peered":"2024-09-21T13:50:29.687815+0000","last_unstale":"2024-09-21T13:53:53.547272+0000","last_undegraded":"2024-09-21T13:53:53.547272+0000","last_fullsized":"2024-09-21T13:53:53.547272+0000","mapping_epoch":94,"log_start":"138'11600","ondisk_log_start":"138'11600","created":83,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":5042,"log_dups_size":0,"ondisk_log_size":5042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:38.682969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23724032,"num_objects":362,"num_object_clones":0,"num_object_copies":1086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16642,"num_write_kb":544128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"68'19","reported_seq":271,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.277738+0000","last_change":"2024-09-21T13:50:03.769686+0000","last_active":"2024-09-21T13:53:53.277738+0000","last_peered":"2024-09-21T13:53:53.277738+0000","last_clean":"2024-09-21T13:53:53.277738+0000","last_became_active":"2024-09-21T13:49:00.425955+0000","last_became_peered":"2024-09-21T13:49:00.425955+0000","last_unstale":"2024-09-21T13:53:53.277738+0000","last_undegraded":"2024-09-21T13:53:53.277738+0000","last_fullsized":"2024-09-21T13:53:53.277738+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-21T13:50:03.769613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:03.769613+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:36.254505+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":236,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.282767+0000","last_change":"2024-09-21T13:48:58.053940+0000","last_active":"2024-09-21T13:53:53.282767+0000","last_peered":"2024-09-21T13:53:53.282767+0000","last_clean":"2024-09-21T13:53:53.282767+0000","last_became_active":"2024-09-21T13:48:58.053641+0000","last_became_peered":"2024-09-21T13:48:58.053641+0000","last_unstale":"2024-09-21T13:53:53.282767+0000","last_undegraded":"2024-09-21T13:53:53.282767+0000","last_fullsized":"2024-09-21T13:53:53.282767+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:17.391569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":196,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-21T13:53:54.925715+0000","last_change":"2024-09-21T13:53:53.244317+0000","last_active":"2024-09-21T13:53:54.925715+0000","last_peered":"2024-09-21T13:53:54.925715+0000","last_clean":"2024-09-21T13:53:54.925715+0000","last_became_active":"2024-09-21T13:52:04.321146+0000","last_became_peered":"2024-09-21T13:52:04.321146+0000","last_unstale":"2024-09-21T13:53:54.925715+0000","last_undegraded":"2024-09-21T13:53:54.925715+0000","last_fullsized":"2024-09-21T13:53:54.925715+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:53.244131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:53:53.244131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:13.809484+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":243,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.283338+0000","last_change":"2024-09-21T13:52:04.323179+0000","last_active":"2024-09-21T13:53:53.283338+0000","last_peered":"2024-09-21T13:53:53.283338+0000","last_clean":"2024-09-21T13:53:53.283338+0000","last_became_active":"2024-09-21T13:52:04.322213+0000","last_became_peered":"2024-09-21T13:52:04.322213+0000","last_unstale":"2024-09-21T13:53:53.283338+0000","last_undegraded":"2024-09-21T13:53:53.283338+0000","last_fullsized":"2024-09-21T13:53:53.283338+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:35.067261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:35.067261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"172'16603","reported_seq":25885,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.567868+0000","last_change":"2024-09-21T13:53:01.956068+0000","last_active":"2024-09-21T13:53:53.567868+0000","last_peered":"2024-09-21T13:53:53.567868+0000","last_clean":"2024-09-21T13:53:53.567868+0000","last_became_active":"2024-09-21T13:53:01.911422+0000","last_became_peered":"2024-09-21T13:53:01.911422+0000","last_unstale":"2024-09-21T13:53:53.567868+0000","last_undegraded":"2024-09-21T13:53:53.567868+0000","last_fullsized":"2024-09-21T13:53:53.567868+0000","mapping_epoch":153,"log_start":"141'11900","ondisk_log_start":"141'11900","created":83,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":4703,"log_dups_size":0,"ondisk_log_size":4703,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26542080,"num_objects":405,"num_object_clones":0,"num_object_copies":1215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16603,"num_write_kb":544256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4537,"num_bytes_recovered":297336832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"68'12","reported_seq":263,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.275407+0000","last_change":"2024-09-21T13:51:10.942190+0000","last_active":"2024-09-21T13:53:53.275407+0000","last_peered":"2024-09-21T13:53:53.275407+0000","last_clean":"2024-09-21T13:53:53.275407+0000","last_became_active":"2024-09-21T13:49:00.059720+0000","last_became_peered":"2024-09-21T13:49:00.059720+0000","last_unstale":"2024-09-21T13:53:53.275407+0000","last_undegraded":"2024-09-21T13:53:53.275407+0000","last_fullsized":"2024-09-21T13:53:53.275407+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'12","last_scrub_stamp":"2024-09-21T13:51:10.942105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:10.942105+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":244,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.282452+0000","last_change":"2024-09-21T13:50:29.100595+0000","last_active":"2024-09-21T13:53:53.282452+0000","last_peered":"2024-09-21T13:53:53.282452+0000","last_clean":"2024-09-21T13:53:53.282452+0000","last_became_active":"2024-09-21T13:48:58.057641+0000","last_became_peered":"2024-09-21T13:48:58.057641+0000","last_unstale":"2024-09-21T13:53:53.282452+0000","last_undegraded":"2024-09-21T13:53:53.282452+0000","last_fullsized":"2024-09-21T13:53:53.282452+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:29.100547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:29.100547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:34.353972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":161,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-21T13:53:54.927475+0000","last_change":"2024-09-21T13:53:31.964122+0000","last_active":"2024-09-21T13:53:54.927475+0000","last_peered":"2024-09-21T13:53:54.927475+0000","last_clean":"2024-09-21T13:53:54.927475+0000","last_became_active":"2024-09-21T13:52:04.322147+0000","last_became_peered":"2024-09-21T13:52:04.322147+0000","last_unstale":"2024-09-21T13:53:54.927475+0000","last_undegraded":"2024-09-21T13:53:54.927475+0000","last_fullsized":"2024-09-21T13:53:54.927475+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:31.958946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:53:31.958946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:51.369874+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":232,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.285952+0000","last_change":"2024-09-21T13:53:20.888048+0000","last_active":"2024-09-21T13:53:53.285952+0000","last_peered":"2024-09-21T13:53:53.285952+0000","last_clean":"2024-09-21T13:53:53.285952+0000","last_became_active":"2024-09-21T13:49:04.078089+0000","last_became_peered":"2024-09-21T13:49:04.078089+0000","last_unstale":"2024-09-21T13:53:53.285952+0000","last_undegraded":"2024-09-21T13:53:53.285952+0000","last_fullsized":"2024-09-21T13:53:53.285952+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:20.882449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:20.882449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:45.947326+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"172'16815","reported_seq":17005,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.537918+0000","last_change":"2024-09-21T13:50:07.422240+0000","last_active":"2024-09-21T13:53:53.537918+0000","last_peered":"2024-09-21T13:53:53.537918+0000","last_clean":"2024-09-21T13:53:53.537918+0000","last_became_active":"2024-09-21T13:50:07.421962+0000","last_became_peered":"2024-09-21T13:50:07.421962+0000","last_unstale":"2024-09-21T13:53:53.537918+0000","last_undegraded":"2024-09-21T13:53:53.537918+0000","last_fullsized":"2024-09-21T13:53:53.537918+0000","mapping_epoch":83,"log_start":"139'11800","ondisk_log_start":"139'11800","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:50:06.399399+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.399399+0000","objects_scrubbed":0,"log_size":5015,"log_dups_size":0,"ondisk_log_size":5015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:44.583978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25493504,"num_objects":389,"num_object_clones":0,"num_object_copies":1167,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16815,"num_write_kb":550528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"68'15","reported_seq":273,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.287991+0000","last_change":"2024-09-21T13:53:31.204731+0000","last_active":"2024-09-21T13:53:53.287991+0000","last_peered":"2024-09-21T13:53:53.287991+0000","last_clean":"2024-09-21T13:53:53.287991+0000","last_became_active":"2024-09-21T13:49:00.425997+0000","last_became_peered":"2024-09-21T13:49:00.425997+0000","last_unstale":"2024-09-21T13:53:53.287991+0000","last_undegraded":"2024-09-21T13:53:53.287991+0000","last_fullsized":"2024-09-21T13:53:53.287991+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:53:31.204637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:31.204637+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:41.385554+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":264,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.282182+0000","last_change":"2024-09-21T13:53:19.396344+0000","last_active":"2024-09-21T13:53:53.282182+0000","last_peered":"2024-09-21T13:53:53.282182+0000","last_clean":"2024-09-21T13:53:53.282182+0000","last_became_active":"2024-09-21T13:48:58.057632+0000","last_became_peered":"2024-09-21T13:48:58.057632+0000","last_unstale":"2024-09-21T13:53:53.282182+0000","last_undegraded":"2024-09-21T13:53:53.282182+0000","last_fullsized":"2024-09-21T13:53:53.282182+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T13:53:19.396277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:53:19.396277+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:47.867330+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"68'8","reported_seq":304,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.281078+0000","last_change":"2024-09-21T13:51:26.297643+0000","last_active":"2024-09-21T13:53:53.281078+0000","last_peered":"2024-09-21T13:53:53.281078+0000","last_clean":"2024-09-21T13:53:53.281078+0000","last_became_active":"2024-09-21T13:49:02.056167+0000","last_became_peered":"2024-09-21T13:49:02.056167+0000","last_unstale":"2024-09-21T13:53:53.281078+0000","last_undegraded":"2024-09-21T13:53:53.281078+0000","last_fullsized":"2024-09-21T13:53:53.281078+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:51:26.297550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:26.297550+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":240,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.287919+0000","last_change":"2024-09-21T13:52:54.314087+0000","last_active":"2024-09-21T13:53:53.287919+0000","last_peered":"2024-09-21T13:53:53.287919+0000","last_clean":"2024-09-21T13:53:53.287919+0000","last_became_active":"2024-09-21T13:49:04.432546+0000","last_became_peered":"2024-09-21T13:49:04.432546+0000","last_unstale":"2024-09-21T13:53:53.287919+0000","last_undegraded":"2024-09-21T13:53:53.287919+0000","last_fullsized":"2024-09-21T13:53:53.287919+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:54.314042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:54.314042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:05.575071+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":252,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.277561+0000","last_change":"2024-09-21T13:51:54.352313+0000","last_active":"2024-09-21T13:53:53.277561+0000","last_peered":"2024-09-21T13:53:53.277561+0000","last_clean":"2024-09-21T13:53:53.277561+0000","last_became_active":"2024-09-21T13:48:58.056080+0000","last_became_peered":"2024-09-21T13:48:58.056080+0000","last_unstale":"2024-09-21T13:53:53.277561+0000","last_undegraded":"2024-09-21T13:53:53.277561+0000","last_fullsized":"2024-09-21T13:53:53.277561+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:54.352244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:51:54.352244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:14.235456+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"68'6","reported_seq":249,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.284564+0000","last_change":"2024-09-21T13:53:31.807215+0000","last_active":"2024-09-21T13:53:53.284564+0000","last_peered":"2024-09-21T13:53:53.284564+0000","last_clean":"2024-09-21T13:53:53.284564+0000","last_became_active":"2024-09-21T13:49:00.426829+0000","last_became_peered":"2024-09-21T13:49:00.426829+0000","last_unstale":"2024-09-21T13:53:53.284564+0000","last_undegraded":"2024-09-21T13:53:53.284564+0000","last_fullsized":"2024-09-21T13:53:53.284564+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'6","last_scrub_stamp":"2024-09-21T13:53:31.807028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:31.807028+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:49.612755+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":244,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.290798+0000","last_change":"2024-09-21T13:51:55.488973+0000","last_active":"2024-09-21T13:53:53.290798+0000","last_peered":"2024-09-21T13:53:53.290798+0000","last_clean":"2024-09-21T13:53:53.290798+0000","last_became_active":"2024-09-21T13:49:02.062231+0000","last_became_peered":"2024-09-21T13:49:02.062231+0000","last_unstale":"2024-09-21T13:53:53.290798+0000","last_undegraded":"2024-09-21T13:53:53.290798+0000","last_fullsized":"2024-09-21T13:53:53.290798+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:55.488908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:55.488908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:16.351786+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":240,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.282449+0000","last_change":"2024-09-21T13:53:18.373418+0000","last_active":"2024-09-21T13:53:53.282449+0000","last_peered":"2024-09-21T13:53:53.282449+0000","last_clean":"2024-09-21T13:53:53.282449+0000","last_became_active":"2024-09-21T13:49:04.432507+0000","last_became_peered":"2024-09-21T13:49:04.432507+0000","last_unstale":"2024-09-21T13:53:53.282449+0000","last_undegraded":"2024-09-21T13:53:53.282449+0000","last_fullsized":"2024-09-21T13:53:53.282449+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:18.373356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:18.373356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:38.812691+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"68'9","reported_seq":212,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.283550+0000","last_change":"2024-09-21T13:51:54.808758+0000","last_active":"2024-09-21T13:53:53.283550+0000","last_peered":"2024-09-21T13:53:53.283550+0000","last_clean":"2024-09-21T13:53:53.283550+0000","last_became_active":"2024-09-21T13:50:29.693589+0000","last_became_peered":"2024-09-21T13:50:29.693589+0000","last_unstale":"2024-09-21T13:53:53.283550+0000","last_undegraded":"2024-09-21T13:53:53.283550+0000","last_fullsized":"2024-09-21T13:53:53.283550+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:51:54.808559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:54.808559+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"61'1","reported_seq":242,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.281254+0000","last_change":"2024-09-21T13:48:58.050948+0000","last_active":"2024-09-21T13:53:53.281254+0000","last_peered":"2024-09-21T13:53:53.281254+0000","last_clean":"2024-09-21T13:53:53.281254+0000","last_became_active":"2024-09-21T13:48:58.050645+0000","last_became_peered":"2024-09-21T13:48:58.050645+0000","last_unstale":"2024-09-21T13:53:53.281254+0000","last_undegraded":"2024-09-21T13:53:53.281254+0000","last_fullsized":"2024-09-21T13:53:53.281254+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:13.439561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":236,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.284459+0000","last_change":"2024-09-21T13:50:43.375425+0000","last_active":"2024-09-21T13:53:53.284459+0000","last_peered":"2024-09-21T13:53:53.284459+0000","last_clean":"2024-09-21T13:53:53.284459+0000","last_became_active":"2024-09-21T13:49:02.061775+0000","last_became_peered":"2024-09-21T13:49:02.061775+0000","last_unstale":"2024-09-21T13:53:53.284459+0000","last_undegraded":"2024-09-21T13:53:53.284459+0000","last_fullsized":"2024-09-21T13:53:53.284459+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:43.375381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:43.375381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":184,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-21T13:53:54.926443+0000","last_change":"2024-09-21T13:52:04.324521+0000","last_active":"2024-09-21T13:53:54.926443+0000","last_peered":"2024-09-21T13:53:54.926443+0000","last_clean":"2024-09-21T13:53:54.926443+0000","last_became_active":"2024-09-21T13:52:04.323549+0000","last_became_peered":"2024-09-21T13:52:04.323549+0000","last_unstale":"2024-09-21T13:53:54.926443+0000","last_undegraded":"2024-09-21T13:53:54.926443+0000","last_fullsized":"2024-09-21T13:53:54.926443+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:41.808746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:41.808746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:34.950671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"68'9","reported_seq":256,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.288776+0000","last_change":"2024-09-21T13:51:28.547052+0000","last_active":"2024-09-21T13:53:53.288776+0000","last_peered":"2024-09-21T13:53:53.288776+0000","last_clean":"2024-09-21T13:53:53.288776+0000","last_became_active":"2024-09-21T13:49:00.427766+0000","last_became_peered":"2024-09-21T13:49:00.427766+0000","last_unstale":"2024-09-21T13:53:53.288776+0000","last_undegraded":"2024-09-21T13:53:53.288776+0000","last_fullsized":"2024-09-21T13:53:53.288776+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:51:28.546958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:28.546958+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:08.329203+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":176,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.281604+0000","last_change":"2024-09-21T13:50:30.076191+0000","last_active":"2024-09-21T13:53:53.281604+0000","last_peered":"2024-09-21T13:53:53.281604+0000","last_clean":"2024-09-21T13:53:53.281604+0000","last_became_active":"2024-09-21T13:50:29.690634+0000","last_became_peered":"2024-09-21T13:50:29.690634+0000","last_unstale":"2024-09-21T13:53:53.281604+0000","last_undegraded":"2024-09-21T13:53:53.281604+0000","last_fullsized":"2024-09-21T13:53:53.281604+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:30.076115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:30.076115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"132'10","reported_seq":317,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.283500+0000","last_change":"2024-09-21T13:52:04.319377+0000","last_active":"2024-09-21T13:53:53.283500+0000","last_peered":"2024-09-21T13:53:53.283500+0000","last_clean":"2024-09-21T13:53:53.283500+0000","last_became_active":"2024-09-21T13:52:04.318626+0000","last_became_peered":"2024-09-21T13:52:04.318626+0000","last_unstale":"2024-09-21T13:53:53.283500+0000","last_undegraded":"2024-09-21T13:53:53.283500+0000","last_fullsized":"2024-09-21T13:53:53.283500+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:50:29.249680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:29.249680+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":250,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.275817+0000","last_change":"2024-09-21T13:53:51.285122+0000","last_active":"2024-09-21T13:53:53.275817+0000","last_peered":"2024-09-21T13:53:53.275817+0000","last_clean":"2024-09-21T13:53:53.275817+0000","last_became_active":"2024-09-21T13:52:04.315865+0000","last_became_peered":"2024-09-21T13:52:04.315865+0000","last_unstale":"2024-09-21T13:53:53.275817+0000","last_undegraded":"2024-09-21T13:53:53.275817+0000","last_fullsized":"2024-09-21T13:53:53.275817+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:51.285077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:51.285077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:16.987394+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"68'10","reported_seq":255,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.286513+0000","last_change":"2024-09-21T13:51:11.545546+0000","last_active":"2024-09-21T13:53:53.286513+0000","last_peered":"2024-09-21T13:53:53.286513+0000","last_clean":"2024-09-21T13:53:53.286513+0000","last_became_active":"2024-09-21T13:49:00.426071+0000","last_became_peered":"2024-09-21T13:49:00.426071+0000","last_unstale":"2024-09-21T13:53:53.286513+0000","last_undegraded":"2024-09-21T13:53:53.286513+0000","last_fullsized":"2024-09-21T13:53:53.286513+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:51:11.545458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:11.545458+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":255,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.281994+0000","last_change":"2024-09-21T13:52:04.318335+0000","last_active":"2024-09-21T13:53:53.281994+0000","last_peered":"2024-09-21T13:53:53.281994+0000","last_clean":"2024-09-21T13:53:53.281994+0000","last_became_active":"2024-09-21T13:52:04.318006+0000","last_became_peered":"2024-09-21T13:52:04.318006+0000","last_unstale":"2024-09-21T13:53:53.281994+0000","last_undegraded":"2024-09-21T13:53:53.281994+0000","last_fullsized":"2024-09-21T13:53:53.281994+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:41.162315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:41.162315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:42.458106+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"68'8","reported_seq":296,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.288569+0000","last_change":"2024-09-21T13:49:02.067827+0000","last_active":"2024-09-21T13:53:53.288569+0000","last_peered":"2024-09-21T13:53:53.288569+0000","last_clean":"2024-09-21T13:53:53.288569+0000","last_became_active":"2024-09-21T13:49:02.067487+0000","last_became_peered":"2024-09-21T13:49:02.067487+0000","last_unstale":"2024-09-21T13:53:53.288569+0000","last_undegraded":"2024-09-21T13:53:53.288569+0000","last_fullsized":"2024-09-21T13:53:53.288569+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:02.411463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":176,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.282108+0000","last_change":"2024-09-21T13:50:33.050276+0000","last_active":"2024-09-21T13:53:53.282108+0000","last_peered":"2024-09-21T13:53:53.282108+0000","last_clean":"2024-09-21T13:53:53.282108+0000","last_became_active":"2024-09-21T13:50:29.695082+0000","last_became_peered":"2024-09-21T13:50:29.695082+0000","last_unstale":"2024-09-21T13:53:53.282108+0000","last_undegraded":"2024-09-21T13:53:53.282108+0000","last_fullsized":"2024-09-21T13:53:53.282108+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:33.050227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:33.050227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"68'11","reported_seq":267,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.273564+0000","last_change":"2024-09-21T13:53:26.146937+0000","last_active":"2024-09-21T13:53:53.273564+0000","last_peered":"2024-09-21T13:53:53.273564+0000","last_clean":"2024-09-21T13:53:53.273564+0000","last_became_active":"2024-09-21T13:49:00.055024+0000","last_became_peered":"2024-09-21T13:49:00.055024+0000","last_unstale":"2024-09-21T13:53:53.273564+0000","last_undegraded":"2024-09-21T13:53:53.273564+0000","last_fullsized":"2024-09-21T13:53:53.273564+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:53:26.146725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:26.146725+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:44.146330+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":244,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.276097+0000","last_change":"2024-09-21T13:51:20.956291+0000","last_active":"2024-09-21T13:53:53.276097+0000","last_peered":"2024-09-21T13:53:53.276097+0000","last_clean":"2024-09-21T13:53:53.276097+0000","last_became_active":"2024-09-21T13:48:58.051541+0000","last_became_peered":"2024-09-21T13:48:58.051541+0000","last_unstale":"2024-09-21T13:53:53.276097+0000","last_undegraded":"2024-09-21T13:53:53.276097+0000","last_fullsized":"2024-09-21T13:53:53.276097+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:20.956226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:51:20.956226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":252,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.289462+0000","last_change":"2024-09-21T13:53:29.868061+0000","last_active":"2024-09-21T13:53:53.289462+0000","last_peered":"2024-09-21T13:53:53.289462+0000","last_clean":"2024-09-21T13:53:53.289462+0000","last_became_active":"2024-09-21T13:49:02.062868+0000","last_became_peered":"2024-09-21T13:49:02.062868+0000","last_unstale":"2024-09-21T13:53:53.289462+0000","last_undegraded":"2024-09-21T13:53:53.289462+0000","last_fullsized":"2024-09-21T13:53:53.289462+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:29.867278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:53:29.867278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:49.405014+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":232,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.284783+0000","last_change":"2024-09-21T13:51:24.549993+0000","last_active":"2024-09-21T13:53:53.284783+0000","last_peered":"2024-09-21T13:53:53.284783+0000","last_clean":"2024-09-21T13:53:53.284783+0000","last_became_active":"2024-09-21T13:49:04.082961+0000","last_became_peered":"2024-09-21T13:49:04.082961+0000","last_unstale":"2024-09-21T13:53:53.284783+0000","last_undegraded":"2024-09-21T13:53:53.284783+0000","last_fullsized":"2024-09-21T13:53:53.284783+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:24.549706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:51:24.549706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"68'9","reported_seq":267,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.289679+0000","last_change":"2024-09-21T13:52:10.447789+0000","last_active":"2024-09-21T13:53:53.289679+0000","last_peered":"2024-09-21T13:53:53.289679+0000","last_clean":"2024-09-21T13:53:53.289679+0000","last_became_active":"2024-09-21T13:52:04.320149+0000","last_became_peered":"2024-09-21T13:52:04.320149+0000","last_unstale":"2024-09-21T13:53:53.289679+0000","last_undegraded":"2024-09-21T13:53:53.289679+0000","last_fullsized":"2024-09-21T13:53:53.289679+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:52:10.447530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:52:10.447530+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":236,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.282652+0000","last_change":"2024-09-21T13:48:58.058032+0000","last_active":"2024-09-21T13:53:53.282652+0000","last_peered":"2024-09-21T13:53:53.282652+0000","last_clean":"2024-09-21T13:53:53.282652+0000","last_became_active":"2024-09-21T13:48:58.057516+0000","last_became_peered":"2024-09-21T13:48:58.057516+0000","last_unstale":"2024-09-21T13:53:53.282652+0000","last_undegraded":"2024-09-21T13:53:53.282652+0000","last_fullsized":"2024-09-21T13:53:53.282652+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:06.571008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":236,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.286037+0000","last_change":"2024-09-21T13:50:39.411785+0000","last_active":"2024-09-21T13:53:53.286037+0000","last_peered":"2024-09-21T13:53:53.286037+0000","last_clean":"2024-09-21T13:53:53.286037+0000","last_became_active":"2024-09-21T13:49:02.071922+0000","last_became_peered":"2024-09-21T13:49:02.071922+0000","last_unstale":"2024-09-21T13:53:53.286037+0000","last_undegraded":"2024-09-21T13:53:53.286037+0000","last_fullsized":"2024-09-21T13:53:53.286037+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:39.411718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:39.411718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":168,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.287907+0000","last_change":"2024-09-21T13:50:29.689537+0000","last_active":"2024-09-21T13:53:53.287907+0000","last_peered":"2024-09-21T13:53:53.287907+0000","last_clean":"2024-09-21T13:53:53.287907+0000","last_became_active":"2024-09-21T13:50:29.689045+0000","last_became_peered":"2024-09-21T13:50:29.689045+0000","last_unstale":"2024-09-21T13:53:53.287907+0000","last_undegraded":"2024-09-21T13:53:53.287907+0000","last_fullsized":"2024-09-21T13:53:53.287907+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:10.347114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"68'11","reported_seq":259,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.284952+0000","last_change":"2024-09-21T13:50:31.329973+0000","last_active":"2024-09-21T13:53:53.284952+0000","last_peered":"2024-09-21T13:53:53.284952+0000","last_clean":"2024-09-21T13:53:53.284952+0000","last_became_active":"2024-09-21T13:49:00.426091+0000","last_became_peered":"2024-09-21T13:49:00.426091+0000","last_unstale":"2024-09-21T13:53:53.284952+0000","last_undegraded":"2024-09-21T13:53:53.284952+0000","last_fullsized":"2024-09-21T13:53:53.284952+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:50:31.329740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:31.329740+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":236,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.290953+0000","last_change":"2024-09-21T13:48:58.052281+0000","last_active":"2024-09-21T13:53:53.290953+0000","last_peered":"2024-09-21T13:53:53.290953+0000","last_clean":"2024-09-21T13:53:53.290953+0000","last_became_active":"2024-09-21T13:48:58.051974+0000","last_became_peered":"2024-09-21T13:48:58.051974+0000","last_unstale":"2024-09-21T13:53:53.290953+0000","last_undegraded":"2024-09-21T13:53:53.290953+0000","last_fullsized":"2024-09-21T13:53:53.290953+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:07.361374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":236,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.281240+0000","last_change":"2024-09-21T13:50:37.118637+0000","last_active":"2024-09-21T13:53:53.281240+0000","last_peered":"2024-09-21T13:53:53.281240+0000","last_clean":"2024-09-21T13:53:53.281240+0000","last_became_active":"2024-09-21T13:49:02.056006+0000","last_became_peered":"2024-09-21T13:49:02.056006+0000","last_unstale":"2024-09-21T13:53:53.281240+0000","last_undegraded":"2024-09-21T13:53:53.281240+0000","last_fullsized":"2024-09-21T13:53:53.281240+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:37.118594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:37.118594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":232,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.286227+0000","last_change":"2024-09-21T13:51:55.827109+0000","last_active":"2024-09-21T13:53:53.286227+0000","last_peered":"2024-09-21T13:53:53.286227+0000","last_clean":"2024-09-21T13:53:53.286227+0000","last_became_active":"2024-09-21T13:49:04.434034+0000","last_became_peered":"2024-09-21T13:49:04.434034+0000","last_unstale":"2024-09-21T13:53:53.286227+0000","last_undegraded":"2024-09-21T13:53:53.286227+0000","last_fullsized":"2024-09-21T13:53:53.286227+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:55.826922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:51:55.826922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"68'4","reported_seq":246,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.283540+0000","last_change":"2024-09-21T13:53:27.826804+0000","last_active":"2024-09-21T13:53:53.283540+0000","last_peered":"2024-09-21T13:53:53.283540+0000","last_clean":"2024-09-21T13:53:53.283540+0000","last_became_active":"2024-09-21T13:49:00.063066+0000","last_became_peered":"2024-09-21T13:49:00.063066+0000","last_unstale":"2024-09-21T13:53:53.283540+0000","last_undegraded":"2024-09-21T13:53:53.283540+0000","last_fullsized":"2024-09-21T13:53:53.283540+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4","last_scrub_stamp":"2024-09-21T13:53:27.826729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:27.826729+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:54.701343+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":176,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.288868+0000","last_change":"2024-09-21T13:53:32.253481+0000","last_active":"2024-09-21T13:53:53.288868+0000","last_peered":"2024-09-21T13:53:53.288868+0000","last_clean":"2024-09-21T13:53:53.288868+0000","last_became_active":"2024-09-21T13:50:29.690269+0000","last_became_peered":"2024-09-21T13:50:29.690269+0000","last_unstale":"2024-09-21T13:53:53.288868+0000","last_undegraded":"2024-09-21T13:53:53.288868+0000","last_fullsized":"2024-09-21T13:53:53.288868+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:32.253408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:53:32.253408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:51.887076+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":252,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.278456+0000","last_change":"2024-09-21T13:53:51.254522+0000","last_active":"2024-09-21T13:53:53.278456+0000","last_peered":"2024-09-21T13:53:53.278456+0000","last_clean":"2024-09-21T13:53:53.278456+0000","last_became_active":"2024-09-21T13:49:02.064789+0000","last_became_peered":"2024-09-21T13:49:02.064789+0000","last_unstale":"2024-09-21T13:53:53.278456+0000","last_undegraded":"2024-09-21T13:53:53.278456+0000","last_fullsized":"2024-09-21T13:53:53.278456+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:51.254456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:53:51.254456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:51.405316+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":243,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.281344+0000","last_change":"2024-09-21T13:52:04.322055+0000","last_active":"2024-09-21T13:53:53.281344+0000","last_peered":"2024-09-21T13:53:53.281344+0000","last_clean":"2024-09-21T13:53:53.281344+0000","last_became_active":"2024-09-21T13:52:04.321667+0000","last_became_peered":"2024-09-21T13:52:04.321667+0000","last_unstale":"2024-09-21T13:53:53.281344+0000","last_undegraded":"2024-09-21T13:53:53.281344+0000","last_fullsized":"2024-09-21T13:53:53.281344+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:39.151320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:39.151320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":239,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.283001+0000","last_change":"2024-09-21T13:52:27.409588+0000","last_active":"2024-09-21T13:53:53.283001+0000","last_peered":"2024-09-21T13:53:53.283001+0000","last_clean":"2024-09-21T13:53:53.283001+0000","last_became_active":"2024-09-21T13:49:04.075562+0000","last_became_peered":"2024-09-21T13:49:04.075562+0000","last_unstale":"2024-09-21T13:53:53.283001+0000","last_undegraded":"2024-09-21T13:53:53.283001+0000","last_fullsized":"2024-09-21T13:53:53.283001+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:27.409408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:27.409408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":244,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.283889+0000","last_change":"2024-09-21T13:53:25.873998+0000","last_active":"2024-09-21T13:53:53.283889+0000","last_peered":"2024-09-21T13:53:53.283889+0000","last_clean":"2024-09-21T13:53:53.283889+0000","last_became_active":"2024-09-21T13:48:58.055497+0000","last_became_peered":"2024-09-21T13:48:58.055497+0000","last_unstale":"2024-09-21T13:53:53.283889+0000","last_undegraded":"2024-09-21T13:53:53.283889+0000","last_fullsized":"2024-09-21T13:53:53.283889+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:25.865470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:53:25.865470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:51.548858+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"68'11","reported_seq":289,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.278812+0000","last_change":"2024-09-21T13:52:04.321178+0000","last_active":"2024-09-21T13:53:53.278812+0000","last_peered":"2024-09-21T13:53:53.278812+0000","last_clean":"2024-09-21T13:53:53.278812+0000","last_became_active":"2024-09-21T13:52:04.320647+0000","last_became_peered":"2024-09-21T13:52:04.320647+0000","last_unstale":"2024-09-21T13:53:53.278812+0000","last_undegraded":"2024-09-21T13:53:53.278812+0000","last_fullsized":"2024-09-21T13:53:53.278812+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:50:35.790090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:35.790090+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:07.227949+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":187,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T13:53:53.282001+0000","last_change":"2024-09-21T13:52:04.323345+0000","last_active":"2024-09-21T13:53:53.282001+0000","last_peered":"2024-09-21T13:53:53.282001+0000","last_clean":"2024-09-21T13:53:53.282001+0000","last_became_active":"2024-09-21T13:52:04.322880+0000","last_became_peered":"2024-09-21T13:52:04.322880+0000","last_unstale":"2024-09-21T13:53:53.282001+0000","last_undegraded":"2024-09-21T13:53:53.282001+0000","last_fullsized":"2024-09-21T13:53:53.282001+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:36.086691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:36.086691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":375128088,"num_objects":5725,"num_object_clones":0,"num_object_copies":17175,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5725,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":267111,"num_write_kb":8730689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22698,"num_bytes_recovered":1487536128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1121464320,"data_stored":1121452104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87145,"ondisk_log_size":87145,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69,"ondisk_log_size":69,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":694,"num_read_kb":451,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":395,"ondisk_log_size":395,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":290,"num_read_kb":285,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234048,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":276736,"kb_used_data":146988,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":93468416,"statfs":{"total":95995035648,"available":95711657984,"internally_reserved":0,"allocated":150515712,"data_stored":150082777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":132839882},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462981,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":237808,"kb_used_data":126108,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93507344,"statfs":{"total":95995035648,"available":95751520256,"internally_reserved":0,"allocated":129134592,"data_stored":128716980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":114358726},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724620,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":228392,"kb_used_data":106968,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93516760,"statfs":{"total":95995035648,"available":95761162240,"internally_reserved":0,"allocated":109535232,"data_stored":109122064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920850,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":244824,"kb_used_data":127552,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":93500328,"statfs":{"total":95995035648,"available":95744335872,"internally_reserved":0,"allocated":130613248,"data_stored":130166956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":120060357},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117079,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":425564,"kb_used_data":218052,"kb_used_omap":1,"kb_used_meta":207486,"kb_avail":93319588,"statfs":{"total":95995035648,"available":95559258112,"internally_reserved":0,"allocated":223285248,"data_stored":222835690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":212466119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":130,"seq":558345748504,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":272664,"kb_used_data":77952,"kb_used_omap":4,"kb_used_meta":194619,"kb_avail":93472488,"statfs":{"total":95995035648,"available":95715827712,"internally_reserved":0,"allocated":79822848,"data_stored":79393094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4441,"internal_metadata":199290535},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,0,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574948,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":288048,"kb_used_data":148572,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":93457104,"statfs":{"total":95995035648,"available":95700074496,"internally_reserved":0,"allocated":152137728,"data_stored":151663312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738474,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":392580,"kb_used_data":217008,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":93352572,"statfs":{"total":95995035648,"available":95593033728,"internally_reserved":0,"allocated":222216192,"data_stored":221778458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":193269760,"data_stored":193265688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":150147072,"data_stored":150143000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":72089600,"data_stored":72089600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":222101504,"data_stored":222101504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":127205376,"data_stored":127205376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":104792064,"data_stored":104792064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":123080704,"data_stored":123076632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":128778240,"data_stored":128778240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T13:54:00.630 INFO:tasks.thrashosds.thrasher:Setting 3.14 to [7, 6, 0] 2024-09-21T13:54:00.631 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '3.14', '7', '6', '0'] 2024-09-21T13:54:00.631 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd pg-upmap 3.14 7 6 0 2024-09-21T13:54:00.639 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:00.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:00.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:54:00.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:00 smithi187 ceph-mon[33032]: 3.1d scrub starts 2024-09-21T13:54:00.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:00 smithi187 ceph-mon[33032]: 3.1d scrub ok 2024-09-21T13:54:00.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:00 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:54:00.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:00 smithi187 ceph-mon[33032]: 7.a scrub ok 2024-09-21T13:54:00.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:00 smithi187 ceph-mon[33032]: from='client.24964 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T13:54:00.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:00 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:54:00.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:00 smithi187 ceph-mon[33032]: osdmap e176: 8 total, 8 up, 8 in 2024-09-21T13:54:00.741 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:54:00.764 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:00.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:00.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:54:00.865 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:54:00 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:54:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:54:00.865 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:54:00.892 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:00.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:00.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:54:00.994 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:54:01.014 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T13:54:01.016 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:01.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:01.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:54:01.118 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:01.138 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:01.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:01.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:01.163 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T13:54:01.163 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-09-21T13:54:01.163 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-09-21T13:54:01.163 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-09-21T13:54:01.163 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.163 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:35220 7.f 7:f1d65728:::benchmark_data_smithi049_76798_object35219:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T13:54:01.163 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:28.689446+0000", 2024-09-21T13:54:01.164 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 212.46452325800001, 2024-09-21T13:54:01.164 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0014680600000001, 2024-09-21T13:54:01.164 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:54:01.164 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:54:01.164 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T13:54:01.164 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:54:01.164 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:54:01.164 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:01.164 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 35220 2024-09-21T13:54:01.164 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.164 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:54:01.164 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.164 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:54:01.164 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.689446+0000", 2024-09-21T13:54:01.164 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:01.165 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.165 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.165 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:54:01.165 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.689446+0000", 2024-09-21T13:54:01.165 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:01.165 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.165 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.165 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:54:01.165 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.689447+0000", 2024-09-21T13:54:01.165 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9500000000000002e-07 2024-09-21T13:54:01.165 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.165 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.165 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:54:01.165 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.689473+0000", 2024-09-21T13:54:01.166 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5661e-05 2024-09-21T13:54:01.166 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.166 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.166 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:54:01.166 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.689481+0000", 2024-09-21T13:54:01.166 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2009999999999999e-06 2024-09-21T13:54:01.166 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.166 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.166 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:54:01.166 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.689487+0000", 2024-09-21T13:54:01.166 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3659999999999997e-06 2024-09-21T13:54:01.166 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.166 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.167 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:01.167 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.689704+0000", 2024-09-21T13:54:01.167 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00021680000000000001 2024-09-21T13:54:01.167 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.167 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.167 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:54:01.167 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.689714+0000", 2024-09-21T13:54:01.167 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0481e-05 2024-09-21T13:54:01.167 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.167 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.167 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:01.167 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.690368+0000", 2024-09-21T13:54:01.167 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.00065395 2024-09-21T13:54:01.167 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.167 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.168 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:54:01.168 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.690914+0000", 2024-09-21T13:54:01.168 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00054600600000000003 2024-09-21T13:54:01.168 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:01.168 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:54:01.168 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:01.168 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.168 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.168 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:35261 7.f 7:fafb65cb:::benchmark_data_smithi049_76798_object35260:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T13:54:01.168 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:28.705982+0000", 2024-09-21T13:54:01.168 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 212.44798726900001, 2024-09-21T13:54:01.168 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.99118446199999999, 2024-09-21T13:54:01.168 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:54:01.168 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:54:01.168 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:01.168 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:54:01.169 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:54:01.169 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:01.169 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 35261 2024-09-21T13:54:01.169 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.169 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:54:01.169 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.169 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:54:01.169 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.705982+0000", 2024-09-21T13:54:01.169 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:01.169 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.169 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.169 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:54:01.169 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.705982+0000", 2024-09-21T13:54:01.169 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:01.169 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.169 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.169 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:54:01.170 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.705983+0000", 2024-09-21T13:54:01.170 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7400000000000003e-07 2024-09-21T13:54:01.170 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.170 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.170 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:54:01.170 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.706009+0000", 2024-09-21T13:54:01.170 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5777000000000001e-05 2024-09-21T13:54:01.170 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.170 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.170 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:54:01.170 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.706036+0000", 2024-09-21T13:54:01.170 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.7529e-05 2024-09-21T13:54:01.170 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.170 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.170 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:54:01.170 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.706043+0000", 2024-09-21T13:54:01.171 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2280000000000001e-06 2024-09-21T13:54:01.171 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.171 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.171 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:01.171 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.706108+0000", 2024-09-21T13:54:01.171 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4133000000000004e-05 2024-09-21T13:54:01.171 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.171 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.171 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:54:01.171 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.706119+0000", 2024-09-21T13:54:01.171 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1105e-05 2024-09-21T13:54:01.171 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.171 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.171 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:01.171 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.691847+0000", 2024-09-21T13:54:01.172 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.98572784199999997 2024-09-21T13:54:01.172 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.172 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.172 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:54:01.172 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.692093+0000", 2024-09-21T13:54:01.172 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000246405 2024-09-21T13:54:01.172 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.172 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.172 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,7", 2024-09-21T13:54:01.172 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.692178+0000", 2024-09-21T13:54:01.172 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4666999999999994e-05 2024-09-21T13:54:01.172 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.172 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.172 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:54:01.172 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.696555+0000", 2024-09-21T13:54:01.172 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0043774039999999997 2024-09-21T13:54:01.173 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.173 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.173 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:01.173 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.696599+0000", 2024-09-21T13:54:01.173 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4319e-05 2024-09-21T13:54:01.173 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.173 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.173 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:01.173 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.697065+0000", 2024-09-21T13:54:01.173 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00046557800000000002 2024-09-21T13:54:01.173 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.173 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.173 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:54:01.173 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.697124+0000", 2024-09-21T13:54:01.173 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9250999999999999e-05 2024-09-21T13:54:01.174 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.174 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.174 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:54:01.174 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.697167+0000", 2024-09-21T13:54:01.174 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.265e-05 2024-09-21T13:54:01.174 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:01.174 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:54:01.174 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:01.174 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.174 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.174 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:35268 7.f 7:f0677f6b:::benchmark_data_smithi049_76798_object35267:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T13:54:01.174 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:28.710368+0000", 2024-09-21T13:54:01.174 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 212.44360148199999, 2024-09-21T13:54:01.174 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.98801466100000002, 2024-09-21T13:54:01.175 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:54:01.175 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:54:01.175 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:01.175 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:54:01.175 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:54:01.175 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:01.175 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 35268 2024-09-21T13:54:01.175 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.175 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:54:01.175 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.175 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:54:01.175 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.710368+0000", 2024-09-21T13:54:01.175 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:01.175 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.175 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.175 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:54:01.176 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.710368+0000", 2024-09-21T13:54:01.176 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:01.176 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.176 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.176 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:54:01.176 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.710369+0000", 2024-09-21T13:54:01.176 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3300000000000001e-07 2024-09-21T13:54:01.176 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.176 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.176 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:54:01.176 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.710395+0000", 2024-09-21T13:54:01.176 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5939e-05 2024-09-21T13:54:01.176 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.176 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.176 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:54:01.177 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.710406+0000", 2024-09-21T13:54:01.177 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0733999999999999e-05 2024-09-21T13:54:01.177 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.177 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.177 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:54:01.177 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.710456+0000", 2024-09-21T13:54:01.177 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0151000000000001e-05 2024-09-21T13:54:01.177 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.177 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.177 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:01.177 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.710532+0000", 2024-09-21T13:54:01.177 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6718000000000006e-05 2024-09-21T13:54:01.177 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.177 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.177 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:54:01.178 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.710542+0000", 2024-09-21T13:54:01.178 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.5279999999999992e-06 2024-09-21T13:54:01.178 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.178 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.178 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:01.178 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.693647+0000", 2024-09-21T13:54:01.178 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.98310485999999997 2024-09-21T13:54:01.178 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.178 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.178 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:54:01.178 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.693760+0000", 2024-09-21T13:54:01.178 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011317 2024-09-21T13:54:01.178 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.178 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.178 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,7", 2024-09-21T13:54:01.179 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.693901+0000", 2024-09-21T13:54:01.179 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014079300000000001 2024-09-21T13:54:01.179 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.179 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.179 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:54:01.179 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.696562+0000", 2024-09-21T13:54:01.179 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0026607789999999998 2024-09-21T13:54:01.179 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.179 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.179 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:01.179 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.696868+0000", 2024-09-21T13:54:01.179 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00030615599999999997 2024-09-21T13:54:01.179 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.179 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.179 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:01.179 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.698285+0000", 2024-09-21T13:54:01.180 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001417422 2024-09-21T13:54:01.180 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.180 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.180 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:54:01.180 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.698342+0000", 2024-09-21T13:54:01.180 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7130999999999999e-05 2024-09-21T13:54:01.180 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.180 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.180 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:54:01.180 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.698383+0000", 2024-09-21T13:54:01.180 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0547000000000003e-05 2024-09-21T13:54:01.180 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:01.180 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:54:01.180 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:01.180 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.181 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.181 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:35289 7.f 7:fb842f7b:::benchmark_data_smithi049_76798_object35288:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T13:54:01.181 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:28.735051+0000", 2024-09-21T13:54:01.181 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 212.418918851, 2024-09-21T13:54:01.181 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.96353024399999998, 2024-09-21T13:54:01.181 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:54:01.181 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:54:01.181 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:01.181 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:54:01.181 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:54:01.181 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:01.181 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 35289 2024-09-21T13:54:01.181 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.181 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:54:01.181 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.182 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:54:01.182 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.735051+0000", 2024-09-21T13:54:01.182 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:01.182 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.182 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.182 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:54:01.182 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.735051+0000", 2024-09-21T13:54:01.182 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:01.182 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.182 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.182 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:54:01.182 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.735052+0000", 2024-09-21T13:54:01.182 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.043e-06 2024-09-21T13:54:01.182 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.183 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.183 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:54:01.183 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.735070+0000", 2024-09-21T13:54:01.183 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7926000000000001e-05 2024-09-21T13:54:01.183 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.183 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.183 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:54:01.183 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.735080+0000", 2024-09-21T13:54:01.183 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0375e-05 2024-09-21T13:54:01.183 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.183 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.183 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:54:01.183 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.735087+0000", 2024-09-21T13:54:01.183 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3540000000000002e-06 2024-09-21T13:54:01.183 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.184 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.184 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:01.184 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.735144+0000", 2024-09-21T13:54:01.184 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6777999999999997e-05 2024-09-21T13:54:01.184 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.184 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.184 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:54:01.184 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.735152+0000", 2024-09-21T13:54:01.184 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6340000000000001e-06 2024-09-21T13:54:01.184 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.184 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.184 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:01.184 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.695195+0000", 2024-09-21T13:54:01.184 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.96004361299999996 2024-09-21T13:54:01.184 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.184 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.184 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:54:01.185 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.695290+0000", 2024-09-21T13:54:01.185 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4399000000000002e-05 2024-09-21T13:54:01.185 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.185 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.185 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,7", 2024-09-21T13:54:01.185 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.695411+0000", 2024-09-21T13:54:01.185 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012121900000000001 2024-09-21T13:54:01.185 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.185 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.185 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:54:01.185 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.696894+0000", 2024-09-21T13:54:01.185 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0014829800000000001 2024-09-21T13:54:01.185 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.185 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.185 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:01.185 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.698132+0000", 2024-09-21T13:54:01.185 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001237897 2024-09-21T13:54:01.186 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.186 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.186 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:01.186 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.698487+0000", 2024-09-21T13:54:01.186 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00035546299999999998 2024-09-21T13:54:01.186 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.186 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.186 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:54:01.186 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.698545+0000", 2024-09-21T13:54:01.186 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7883000000000001e-05 2024-09-21T13:54:01.186 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.186 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.186 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:54:01.186 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.698581+0000", 2024-09-21T13:54:01.186 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5679999999999997e-05 2024-09-21T13:54:01.186 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:01.187 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:54:01.187 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:01.187 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.187 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.187 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:216727 7.a 7:5a6c5e40:::benchmark_data_smithi049_76798_object80307:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-09-21T13:54:01.187 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:53:00.900577+0000", 2024-09-21T13:54:01.187 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 60.253392152000004, 2024-09-21T13:54:01.187 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.061257892, 2024-09-21T13:54:01.187 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:54:01.187 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:54:01.187 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:01.187 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:54:01.187 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:54:01.187 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:01.187 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 216727 2024-09-21T13:54:01.187 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.188 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:54:01.188 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.188 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:54:01.188 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:00.900577+0000", 2024-09-21T13:54:01.188 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:01.188 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.188 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.188 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:54:01.188 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:00.900577+0000", 2024-09-21T13:54:01.188 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:01.188 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.188 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.188 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:54:01.188 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:00.900579+0000", 2024-09-21T13:54:01.188 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.889e-06 2024-09-21T13:54:01.188 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.189 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.189 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:54:01.189 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:00.900584+0000", 2024-09-21T13:54:01.189 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9849999999999997e-06 2024-09-21T13:54:01.189 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.189 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.189 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:54:01.189 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:00.900597+0000", 2024-09-21T13:54:01.189 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.234e-05 2024-09-21T13:54:01.189 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.189 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.189 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:54:01.189 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:00.900606+0000", 2024-09-21T13:54:01.189 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.8570000000000001e-06 2024-09-21T13:54:01.189 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.189 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.189 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:01.190 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:00.913883+0000", 2024-09-21T13:54:01.190 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.013277106 2024-09-21T13:54:01.190 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.190 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.190 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:54:01.190 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:00.913894+0000", 2024-09-21T13:54:01.190 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1805e-05 2024-09-21T13:54:01.190 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.190 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.190 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:01.190 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.951390+0000", 2024-09-21T13:54:01.190 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0374955379999999 2024-09-21T13:54:01.190 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.190 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.190 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:54:01.191 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.951528+0000", 2024-09-21T13:54:01.191 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013781700000000001 2024-09-21T13:54:01.191 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.191 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.191 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,5", 2024-09-21T13:54:01.191 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.951643+0000", 2024-09-21T13:54:01.191 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000115473 2024-09-21T13:54:01.191 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.191 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.191 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:01.191 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.956431+0000", 2024-09-21T13:54:01.191 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0047878269999999997 2024-09-21T13:54:01.191 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.191 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.191 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:01.192 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.956476+0000", 2024-09-21T13:54:01.192 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4922999999999997e-05 2024-09-21T13:54:01.192 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.192 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.192 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:54:01.192 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.961751+0000", 2024-09-21T13:54:01.192 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0052752789999999999 2024-09-21T13:54:01.192 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.192 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.192 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:54:01.192 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.961793+0000", 2024-09-21T13:54:01.192 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1925000000000003e-05 2024-09-21T13:54:01.192 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.192 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.192 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:54:01.192 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.961835+0000", 2024-09-21T13:54:01.193 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2128000000000001e-05 2024-09-21T13:54:01.193 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:01.193 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:54:01.193 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:01.193 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.193 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.193 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:216746 7.a 7:5f3f3675:::benchmark_data_smithi049_76798_object80326:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-09-21T13:54:01.193 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:53:00.932374+0000", 2024-09-21T13:54:01.193 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 60.221595815000001, 2024-09-21T13:54:01.193 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.029708748, 2024-09-21T13:54:01.193 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:54:01.193 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:54:01.193 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:01.193 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:54:01.193 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:54:01.194 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:01.194 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 216746 2024-09-21T13:54:01.194 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.194 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:54:01.194 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.194 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:54:01.194 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:00.932374+0000", 2024-09-21T13:54:01.194 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:01.194 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.194 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.194 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:54:01.194 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:00.932374+0000", 2024-09-21T13:54:01.194 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:01.194 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.194 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.195 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:54:01.195 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:00.932376+0000", 2024-09-21T13:54:01.195 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9810000000000002e-06 2024-09-21T13:54:01.195 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.195 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.195 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:54:01.195 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:00.932382+0000", 2024-09-21T13:54:01.195 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1580000000000002e-06 2024-09-21T13:54:01.195 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.195 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.195 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:54:01.195 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:00.932396+0000", 2024-09-21T13:54:01.195 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3602e-05 2024-09-21T13:54:01.195 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.195 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.196 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:54:01.196 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:00.932404+0000", 2024-09-21T13:54:01.196 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2930000000000001e-06 2024-09-21T13:54:01.196 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.196 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.196 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:01.196 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:00.932822+0000", 2024-09-21T13:54:01.196 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00041849399999999999 2024-09-21T13:54:01.196 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.196 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.196 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:54:01.196 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:00.932833+0000", 2024-09-21T13:54:01.196 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0336999999999999e-05 2024-09-21T13:54:01.196 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.196 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.197 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:01.197 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.956537+0000", 2024-09-21T13:54:01.197 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0237041520000001 2024-09-21T13:54:01.197 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.197 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.197 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:54:01.197 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.956656+0000", 2024-09-21T13:54:01.197 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011948800000000001 2024-09-21T13:54:01.197 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.197 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.197 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,5", 2024-09-21T13:54:01.197 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.956772+0000", 2024-09-21T13:54:01.197 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011541599999999999 2024-09-21T13:54:01.197 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.198 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.198 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:01.198 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.961743+0000", 2024-09-21T13:54:01.198 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0049709819999999997 2024-09-21T13:54:01.198 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.198 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.198 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:54:01.198 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.961883+0000", 2024-09-21T13:54:01.198 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000140233 2024-09-21T13:54:01.198 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.198 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.198 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:01.198 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.961983+0000", 2024-09-21T13:54:01.198 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010023599999999999 2024-09-21T13:54:01.198 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.198 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.199 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:54:01.199 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.962044+0000", 2024-09-21T13:54:01.199 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0989000000000002e-05 2024-09-21T13:54:01.199 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.199 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.199 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:54:01.199 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.962083+0000", 2024-09-21T13:54:01.199 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8386999999999997e-05 2024-09-21T13:54:01.199 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:01.199 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:54:01.199 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:01.199 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.199 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.199 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:216757 7.a 7:51a3b337:::benchmark_data_smithi049_76798_object80337:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-09-21T13:54:01.200 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:53:00.944656+0000", 2024-09-21T13:54:01.200 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 60.209313680000001, 2024-09-21T13:54:01.200 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0309007539999999, 2024-09-21T13:54:01.200 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:54:01.200 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:54:01.200 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:01.200 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:54:01.200 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:54:01.201 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:01.201 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 216757 2024-09-21T13:54:01.201 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.201 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:54:01.201 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.201 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:54:01.201 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:00.944656+0000", 2024-09-21T13:54:01.201 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:01.201 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.201 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.201 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:54:01.202 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:00.944656+0000", 2024-09-21T13:54:01.202 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:01.202 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.202 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.202 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:54:01.202 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:00.944656+0000", 2024-09-21T13:54:01.202 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6899999999999998e-07 2024-09-21T13:54:01.202 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.202 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.202 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:54:01.202 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:00.944659+0000", 2024-09-21T13:54:01.202 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.295e-06 2024-09-21T13:54:01.202 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.202 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.202 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:54:01.203 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:00.944665+0000", 2024-09-21T13:54:01.203 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.404e-06 2024-09-21T13:54:01.203 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.203 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.203 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:54:01.203 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:00.944670+0000", 2024-09-21T13:54:01.203 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3329999999999996e-06 2024-09-21T13:54:01.203 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.203 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.203 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:01.203 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:00.944737+0000", 2024-09-21T13:54:01.203 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6761999999999995e-05 2024-09-21T13:54:01.203 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.203 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.203 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:54:01.203 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:00.944747+0000", 2024-09-21T13:54:01.204 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0036e-05 2024-09-21T13:54:01.204 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.204 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.204 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:01.204 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.959224+0000", 2024-09-21T13:54:01.204 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.014477106 2024-09-21T13:54:01.204 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.204 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.204 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:54:01.204 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.959292+0000", 2024-09-21T13:54:01.204 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7531000000000001e-05 2024-09-21T13:54:01.204 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.204 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.204 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,5", 2024-09-21T13:54:01.204 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.959363+0000", 2024-09-21T13:54:01.204 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0735000000000006e-05 2024-09-21T13:54:01.205 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.205 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.205 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:01.205 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.961929+0000", 2024-09-21T13:54:01.205 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0025660750000000001 2024-09-21T13:54:01.205 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.205 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.205 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:01.205 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.967833+0000", 2024-09-21T13:54:01.205 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0059041780000000004 2024-09-21T13:54:01.205 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.205 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.205 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:54:01.205 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.975438+0000", 2024-09-21T13:54:01.205 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.007604759 2024-09-21T13:54:01.205 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.206 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.206 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:54:01.206 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.975509+0000", 2024-09-21T13:54:01.206 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1522999999999999e-05 2024-09-21T13:54:01.206 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.206 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.206 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:54:01.206 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.975557+0000", 2024-09-21T13:54:01.206 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7547999999999999e-05 2024-09-21T13:54:01.206 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:01.206 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:54:01.206 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:01.206 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.206 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.207 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:216789 7.a 7:5a27fb65:::benchmark_data_smithi049_76798_object80369:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-09-21T13:54:01.207 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:53:00.977986+0000", 2024-09-21T13:54:01.207 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 60.175983217999999, 2024-09-21T13:54:01.207 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.99775057, 2024-09-21T13:54:01.207 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:54:01.207 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:54:01.207 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:01.207 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:54:01.207 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:54:01.207 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:01.207 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 216789 2024-09-21T13:54:01.207 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.207 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:54:01.207 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.208 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:54:01.208 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:00.977986+0000", 2024-09-21T13:54:01.208 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:01.208 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.208 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.208 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:54:01.208 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:00.977986+0000", 2024-09-21T13:54:01.208 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:01.208 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.208 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.208 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:54:01.208 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:00.977988+0000", 2024-09-21T13:54:01.208 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9489999999999999e-06 2024-09-21T13:54:01.208 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.208 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.209 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:54:01.209 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:00.977993+0000", 2024-09-21T13:54:01.209 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7419999999999997e-06 2024-09-21T13:54:01.209 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.209 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.209 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:54:01.209 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:00.978004+0000", 2024-09-21T13:54:01.209 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1398e-05 2024-09-21T13:54:01.209 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.209 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.209 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:54:01.209 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:00.978012+0000", 2024-09-21T13:54:01.209 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1600000000000001e-06 2024-09-21T13:54:01.209 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.209 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.210 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:01.210 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:00.978062+0000", 2024-09-21T13:54:01.210 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9891000000000001e-05 2024-09-21T13:54:01.210 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.210 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.210 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:54:01.210 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:00.978069+0000", 2024-09-21T13:54:01.210 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5900000000000002e-06 2024-09-21T13:54:01.210 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.210 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.210 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:01.210 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.962120+0000", 2024-09-21T13:54:01.210 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.98405099799999995 2024-09-21T13:54:01.210 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.211 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.211 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:54:01.211 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.962233+0000", 2024-09-21T13:54:01.211 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011304 2024-09-21T13:54:01.211 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.211 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.211 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,5", 2024-09-21T13:54:01.211 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.962358+0000", 2024-09-21T13:54:01.211 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012471800000000001 2024-09-21T13:54:01.211 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.211 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.211 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:01.211 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.967874+0000", 2024-09-21T13:54:01.211 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0055161369999999999 2024-09-21T13:54:01.211 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.212 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.212 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:54:01.212 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.975617+0000", 2024-09-21T13:54:01.212 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0077429209999999998 2024-09-21T13:54:01.212 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.212 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.212 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:01.212 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.975647+0000", 2024-09-21T13:54:01.212 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.0031000000000001e-05 2024-09-21T13:54:01.212 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.212 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.212 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:54:01.212 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.975709+0000", 2024-09-21T13:54:01.212 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1573000000000002e-05 2024-09-21T13:54:01.213 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.213 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.213 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:54:01.213 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.975737+0000", 2024-09-21T13:54:01.213 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.8422e-05 2024-09-21T13:54:01.213 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:01.213 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:54:01.213 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:01.213 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.213 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.213 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:216812 7.a 7:5430eb8f:::benchmark_data_smithi049_76798_object80392:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-09-21T13:54:01.213 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:53:01.005287+0000", 2024-09-21T13:54:01.213 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 60.148682981999997, 2024-09-21T13:54:01.213 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97064463899999998, 2024-09-21T13:54:01.214 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:54:01.214 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:54:01.214 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:01.214 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:54:01.214 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:54:01.214 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:01.214 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 216812 2024-09-21T13:54:01.214 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.214 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:54:01.214 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.214 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:54:01.214 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.005287+0000", 2024-09-21T13:54:01.214 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:01.214 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.214 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.215 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:54:01.215 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.005287+0000", 2024-09-21T13:54:01.215 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:01.215 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.215 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.215 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:54:01.215 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.005288+0000", 2024-09-21T13:54:01.215 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.421e-06 2024-09-21T13:54:01.215 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.215 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.215 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:54:01.215 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.005294+0000", 2024-09-21T13:54:01.215 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.524e-06 2024-09-21T13:54:01.215 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.215 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.215 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:54:01.216 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.005306+0000", 2024-09-21T13:54:01.216 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2045000000000001e-05 2024-09-21T13:54:01.216 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.216 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.216 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:54:01.216 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.005313+0000", 2024-09-21T13:54:01.216 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3520000000000001e-06 2024-09-21T13:54:01.216 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.216 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.216 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:01.216 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.005394+0000", 2024-09-21T13:54:01.216 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1081999999999994e-05 2024-09-21T13:54:01.216 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.216 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.216 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:54:01.216 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.005403+0000", 2024-09-21T13:54:01.217 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9029999999999993e-06 2024-09-21T13:54:01.217 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.217 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.217 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:01.217 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.965120+0000", 2024-09-21T13:54:01.217 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.95971668200000004 2024-09-21T13:54:01.217 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.217 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.217 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:54:01.217 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.965242+0000", 2024-09-21T13:54:01.217 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000122693 2024-09-21T13:54:01.217 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.217 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.218 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,5", 2024-09-21T13:54:01.218 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.965367+0000", 2024-09-21T13:54:01.218 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012494000000000001 2024-09-21T13:54:01.218 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.218 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.218 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:01.218 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.975603+0000", 2024-09-21T13:54:01.218 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.010236198 2024-09-21T13:54:01.218 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.218 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.218 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:54:01.218 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.975620+0000", 2024-09-21T13:54:01.218 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6373999999999999e-05 2024-09-21T13:54:01.218 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.218 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.219 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:01.219 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.975841+0000", 2024-09-21T13:54:01.219 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00022164400000000001 2024-09-21T13:54:01.219 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.219 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.219 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:54:01.219 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.975895+0000", 2024-09-21T13:54:01.219 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3059000000000001e-05 2024-09-21T13:54:01.219 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.219 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.219 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:54:01.219 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.975931+0000", 2024-09-21T13:54:01.219 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6721999999999998e-05 2024-09-21T13:54:01.219 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:01.219 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:54:01.220 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:01.220 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.220 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.220 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:216829 7.a 7:5026c22e:::benchmark_data_smithi049_76798_object80409:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-09-21T13:54:01.220 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:53:01.026115+0000", 2024-09-21T13:54:01.220 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 60.127854933999998, 2024-09-21T13:54:01.220 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.95070566899999998, 2024-09-21T13:54:01.220 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:54:01.220 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:54:01.220 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:01.220 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:54:01.220 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:54:01.220 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:01.220 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 216829 2024-09-21T13:54:01.220 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.221 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:54:01.221 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.221 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:54:01.221 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.026115+0000", 2024-09-21T13:54:01.221 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:01.221 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.221 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.221 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:54:01.221 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.026115+0000", 2024-09-21T13:54:01.221 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:01.221 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.221 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.221 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:54:01.221 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.026115+0000", 2024-09-21T13:54:01.221 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5999999999999999e-07 2024-09-21T13:54:01.222 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.222 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.222 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:54:01.222 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.026118+0000", 2024-09-21T13:54:01.222 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6419999999999999e-06 2024-09-21T13:54:01.222 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.222 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.222 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:54:01.222 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.026125+0000", 2024-09-21T13:54:01.222 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1849999999999998e-06 2024-09-21T13:54:01.222 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.222 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.222 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:54:01.222 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.026144+0000", 2024-09-21T13:54:01.223 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9357000000000001e-05 2024-09-21T13:54:01.223 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.223 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.223 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:01.223 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.026321+0000", 2024-09-21T13:54:01.223 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00017695800000000001 2024-09-21T13:54:01.223 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.223 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.223 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:54:01.223 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.026329+0000", 2024-09-21T13:54:01.223 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1710000000000002e-06 2024-09-21T13:54:01.223 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.223 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.223 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:01.224 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.967905+0000", 2024-09-21T13:54:01.224 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.94157559599999996 2024-09-21T13:54:01.224 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.224 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.224 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:54:01.224 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.968049+0000", 2024-09-21T13:54:01.224 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014368000000000001 2024-09-21T13:54:01.224 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.224 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.224 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,5", 2024-09-21T13:54:01.224 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.968170+0000", 2024-09-21T13:54:01.224 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000121142 2024-09-21T13:54:01.224 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.224 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.224 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:54:01.225 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.975623+0000", 2024-09-21T13:54:01.225 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0074527669999999999 2024-09-21T13:54:01.225 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.225 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.225 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:01.225 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.975791+0000", 2024-09-21T13:54:01.225 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016814999999999999 2024-09-21T13:54:01.225 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.225 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.225 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:01.225 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.976735+0000", 2024-09-21T13:54:01.225 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00094394100000000001 2024-09-21T13:54:01.225 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.225 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.225 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:54:01.226 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.976781+0000", 2024-09-21T13:54:01.226 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5911000000000001e-05 2024-09-21T13:54:01.226 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.226 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.226 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:54:01.226 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.976820+0000", 2024-09-21T13:54:01.226 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9709000000000002e-05 2024-09-21T13:54:01.226 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:01.226 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:54:01.226 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:01.226 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.226 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.226 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:216850 7.a 7:53c6cd2b:::benchmark_data_smithi049_76798_object80430:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-09-21T13:54:01.227 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:53:01.049880+0000", 2024-09-21T13:54:01.227 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 60.104089737000002, 2024-09-21T13:54:01.227 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.92707560899999997, 2024-09-21T13:54:01.227 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:54:01.227 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:54:01.227 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:01.227 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:54:01.227 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:54:01.227 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:01.227 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 216850 2024-09-21T13:54:01.227 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.227 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:54:01.227 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.227 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:54:01.228 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.049880+0000", 2024-09-21T13:54:01.228 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:01.228 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.228 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.228 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:54:01.228 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.049880+0000", 2024-09-21T13:54:01.228 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:01.228 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.228 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.228 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:54:01.228 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.049882+0000", 2024-09-21T13:54:01.228 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6810000000000001e-06 2024-09-21T13:54:01.228 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.228 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.228 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:54:01.229 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.049888+0000", 2024-09-21T13:54:01.229 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6490000000000002e-06 2024-09-21T13:54:01.229 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.229 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.229 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:54:01.229 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.049905+0000", 2024-09-21T13:54:01.229 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.698e-05 2024-09-21T13:54:01.229 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.229 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.229 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:54:01.229 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.049913+0000", 2024-09-21T13:54:01.229 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.259e-06 2024-09-21T13:54:01.229 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.229 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.230 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:01.230 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.049987+0000", 2024-09-21T13:54:01.230 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3724000000000001e-05 2024-09-21T13:54:01.230 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.230 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.230 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:54:01.230 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.049994+0000", 2024-09-21T13:54:01.230 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2309999999999999e-06 2024-09-21T13:54:01.230 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.230 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.230 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:01.230 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.970569+0000", 2024-09-21T13:54:01.230 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.92057444099999997 2024-09-21T13:54:01.230 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.230 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.231 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:54:01.231 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.970679+0000", 2024-09-21T13:54:01.231 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010977200000000001 2024-09-21T13:54:01.231 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.231 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.231 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,5", 2024-09-21T13:54:01.231 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.970823+0000", 2024-09-21T13:54:01.231 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000144029 2024-09-21T13:54:01.231 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.231 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.231 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:54:01.231 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.975626+0000", 2024-09-21T13:54:01.231 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0048029090000000002 2024-09-21T13:54:01.231 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.231 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.232 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:01.232 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.975987+0000", 2024-09-21T13:54:01.232 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00036170499999999998 2024-09-21T13:54:01.232 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.232 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.232 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:01.232 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.976873+0000", 2024-09-21T13:54:01.232 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00088570900000000004 2024-09-21T13:54:01.232 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.232 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.232 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:54:01.232 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.976921+0000", 2024-09-21T13:54:01.232 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8037000000000001e-05 2024-09-21T13:54:01.232 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.232 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.233 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:54:01.233 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.976955+0000", 2024-09-21T13:54:01.233 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4483000000000002e-05 2024-09-21T13:54:01.233 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:01.233 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:54:01.233 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:01.233 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.233 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.233 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:216890 7.a 7:52b58da3:::benchmark_data_smithi049_76798_object80470:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-09-21T13:54:01.233 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:53:01.095525+0000", 2024-09-21T13:54:01.233 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 60.058444942999998, 2024-09-21T13:54:01.233 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.91254773099999997, 2024-09-21T13:54:01.233 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:54:01.233 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:54:01.234 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:01.234 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:54:01.234 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:54:01.234 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:01.234 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 216890 2024-09-21T13:54:01.234 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.234 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:54:01.234 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.234 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:54:01.234 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.095525+0000", 2024-09-21T13:54:01.234 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:01.234 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.234 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.234 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:54:01.235 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.095525+0000", 2024-09-21T13:54:01.235 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:01.235 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.235 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.235 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:54:01.235 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.095525+0000", 2024-09-21T13:54:01.235 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5700000000000002e-07 2024-09-21T13:54:01.235 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.235 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.235 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:54:01.235 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.095529+0000", 2024-09-21T13:54:01.235 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.253e-06 2024-09-21T13:54:01.235 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.235 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.236 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:54:01.236 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.095539+0000", 2024-09-21T13:54:01.236 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0485e-05 2024-09-21T13:54:01.236 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.236 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.236 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:54:01.236 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.095545+0000", 2024-09-21T13:54:01.236 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1999999999999999e-06 2024-09-21T13:54:01.236 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.236 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.236 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:01.236 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.095647+0000", 2024-09-21T13:54:01.236 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010145 2024-09-21T13:54:01.236 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.237 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.237 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:54:01.237 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.095664+0000", 2024-09-21T13:54:01.237 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7292999999999999e-05 2024-09-21T13:54:01.237 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.237 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.237 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:01.237 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.986877+0000", 2024-09-21T13:54:01.237 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.89121302099999999 2024-09-21T13:54:01.237 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.237 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.237 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:54:01.237 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.986981+0000", 2024-09-21T13:54:01.237 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000103646 2024-09-21T13:54:01.237 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.238 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.238 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,5", 2024-09-21T13:54:01.238 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.987093+0000", 2024-09-21T13:54:01.238 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000112756 2024-09-21T13:54:01.238 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.238 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.238 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:54:01.238 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.989655+0000", 2024-09-21T13:54:01.238 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0025619560000000002 2024-09-21T13:54:01.238 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.238 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.238 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:01.238 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:02.004729+0000", 2024-09-21T13:54:01.238 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.015073822000000001 2024-09-21T13:54:01.239 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.239 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.239 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:01.239 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:02.007968+0000", 2024-09-21T13:54:01.239 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0032389609999999998 2024-09-21T13:54:01.239 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.239 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.239 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:54:01.239 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:02.008029+0000", 2024-09-21T13:54:01.239 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0723000000000001e-05 2024-09-21T13:54:01.239 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.239 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.239 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:54:01.239 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:02.008072+0000", 2024-09-21T13:54:01.240 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3507999999999997e-05 2024-09-21T13:54:01.240 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:01.240 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:54:01.240 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:01.240 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.240 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.240 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:216892 7.a 7:5c1d49ad:::benchmark_data_smithi049_76798_object80472:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-09-21T13:54:01.240 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:53:01.095559+0000", 2024-09-21T13:54:01.240 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 60.058410477000002, 2024-09-21T13:54:01.240 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.91266443500000005, 2024-09-21T13:54:01.240 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:54:01.240 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:54:01.240 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:01.240 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:54:01.241 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:54:01.241 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:01.241 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 216892 2024-09-21T13:54:01.241 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.241 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:54:01.241 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.241 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:54:01.241 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.095559+0000", 2024-09-21T13:54:01.241 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:01.241 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.241 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.241 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:54:01.241 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.095559+0000", 2024-09-21T13:54:01.241 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:01.242 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.242 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.242 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:54:01.242 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.095560+0000", 2024-09-21T13:54:01.242 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0399999999999996e-07 2024-09-21T13:54:01.242 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.242 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.242 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:54:01.242 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.095563+0000", 2024-09-21T13:54:01.242 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.0149999999999999e-06 2024-09-21T13:54:01.242 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.242 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.242 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:54:01.242 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.095574+0000", 2024-09-21T13:54:01.242 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1704e-05 2024-09-21T13:54:01.243 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.243 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.243 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:54:01.243 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.095581+0000", 2024-09-21T13:54:01.243 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7859999999999999e-06 2024-09-21T13:54:01.243 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.243 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.243 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:01.243 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.095694+0000", 2024-09-21T13:54:01.243 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000113341 2024-09-21T13:54:01.243 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.243 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.243 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:54:01.243 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.095701+0000", 2024-09-21T13:54:01.243 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1890000000000004e-06 2024-09-21T13:54:01.244 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.244 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.244 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:01.244 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.990057+0000", 2024-09-21T13:54:01.244 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.89435641099999996 2024-09-21T13:54:01.244 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.244 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.244 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:54:01.244 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.990167+0000", 2024-09-21T13:54:01.244 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000109536 2024-09-21T13:54:01.244 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.244 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.244 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,5", 2024-09-21T13:54:01.244 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.990284+0000", 2024-09-21T13:54:01.244 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011695 2024-09-21T13:54:01.245 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.245 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.245 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:54:01.245 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.999354+0000", 2024-09-21T13:54:01.245 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0090706829999999995 2024-09-21T13:54:01.245 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.245 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.245 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:01.245 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:02.004766+0000", 2024-09-21T13:54:01.245 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0054113850000000003 2024-09-21T13:54:01.245 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.245 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.245 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:01.245 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:02.008151+0000", 2024-09-21T13:54:01.245 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0033852130000000002 2024-09-21T13:54:01.246 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.246 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.246 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:54:01.246 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:02.008191+0000", 2024-09-21T13:54:01.246 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0209000000000001e-05 2024-09-21T13:54:01.246 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.246 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.246 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:54:01.246 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:02.008224+0000", 2024-09-21T13:54:01.246 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2509000000000003e-05 2024-09-21T13:54:01.246 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:01.246 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:54:01.246 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:01.247 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.247 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.247 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:216899 7.a 7:52979337:::benchmark_data_smithi049_76798_object80479:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-09-21T13:54:01.247 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:53:01.106350+0000", 2024-09-21T13:54:01.247 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 60.047619623999999, 2024-09-21T13:54:01.247 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.91232380300000004, 2024-09-21T13:54:01.247 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:54:01.247 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:54:01.247 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:01.247 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:54:01.247 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:54:01.247 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:01.247 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 216899 2024-09-21T13:54:01.247 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.248 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:54:01.248 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.248 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:54:01.248 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.106350+0000", 2024-09-21T13:54:01.248 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:01.248 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.248 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.248 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:54:01.248 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.106350+0000", 2024-09-21T13:54:01.248 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:01.248 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.248 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.248 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:54:01.248 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.106351+0000", 2024-09-21T13:54:01.248 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4479999999999999e-06 2024-09-21T13:54:01.248 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.249 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.249 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:54:01.249 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.106358+0000", 2024-09-21T13:54:01.249 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3860000000000004e-06 2024-09-21T13:54:01.249 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.249 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.249 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:54:01.249 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.106381+0000", 2024-09-21T13:54:01.249 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2963999999999999e-05 2024-09-21T13:54:01.249 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.249 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.249 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:54:01.249 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.106389+0000", 2024-09-21T13:54:01.249 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9959999999999993e-06 2024-09-21T13:54:01.249 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.250 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.250 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:01.250 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.106443+0000", 2024-09-21T13:54:01.250 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4455e-05 2024-09-21T13:54:01.250 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.250 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.250 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:54:01.250 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.106451+0000", 2024-09-21T13:54:01.250 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8399999999999995e-06 2024-09-21T13:54:01.250 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.250 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.250 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:01.250 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.992852+0000", 2024-09-21T13:54:01.250 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.88640085800000001 2024-09-21T13:54:01.251 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.251 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.251 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:54:01.251 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.992977+0000", 2024-09-21T13:54:01.251 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000125363 2024-09-21T13:54:01.251 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.251 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.251 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,5", 2024-09-21T13:54:01.251 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:01.993115+0000", 2024-09-21T13:54:01.251 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013804900000000001 2024-09-21T13:54:01.251 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.251 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.251 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:54:01.251 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:02.002003+0000", 2024-09-21T13:54:01.252 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0088878810000000003 2024-09-21T13:54:01.252 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.252 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.252 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:01.252 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:02.004798+0000", 2024-09-21T13:54:01.252 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0027944210000000001 2024-09-21T13:54:01.252 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.252 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.252 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:01.252 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:02.018529+0000", 2024-09-21T13:54:01.252 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.013731228999999999 2024-09-21T13:54:01.252 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.252 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.252 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:54:01.252 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:02.018628+0000", 2024-09-21T13:54:01.253 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9256e-05 2024-09-21T13:54:01.253 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.253 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.253 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:54:01.253 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:02.018674+0000", 2024-09-21T13:54:01.253 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5657e-05 2024-09-21T13:54:01.253 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:01.253 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:54:01.253 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:01.253 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.253 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.253 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:229752 7.3 7:c55d7a46:::benchmark_data_smithi049_76798_object93332:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e159)", 2024-09-21T13:54:01.253 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:53:15.678599+0000", 2024-09-21T13:54:01.253 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 45.475370724000001, 2024-09-21T13:54:01.254 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.99959303700000002, 2024-09-21T13:54:01.254 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:54:01.254 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:54:01.254 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:01.254 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:54:01.254 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:54:01.254 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:01.254 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 229752 2024-09-21T13:54:01.254 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.254 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:54:01.254 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.254 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:54:01.254 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:15.678599+0000", 2024-09-21T13:54:01.254 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:01.255 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.255 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.255 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:54:01.255 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:15.678599+0000", 2024-09-21T13:54:01.255 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:01.255 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.255 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.255 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:54:01.255 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:15.678600+0000", 2024-09-21T13:54:01.255 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0559999999999999e-06 2024-09-21T13:54:01.255 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.255 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.255 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:54:01.256 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:15.678604+0000", 2024-09-21T13:54:01.256 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9339999999999999e-06 2024-09-21T13:54:01.256 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.256 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.256 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:54:01.256 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:15.678616+0000", 2024-09-21T13:54:01.256 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2029e-05 2024-09-21T13:54:01.256 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.256 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.256 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:54:01.256 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:15.678622+0000", 2024-09-21T13:54:01.256 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0909999999999998e-06 2024-09-21T13:54:01.256 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.256 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.257 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:01.257 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:15.678889+0000", 2024-09-21T13:54:01.257 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000267459 2024-09-21T13:54:01.257 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.257 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.257 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:54:01.257 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:15.678898+0000", 2024-09-21T13:54:01.257 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6810000000000004e-06 2024-09-21T13:54:01.257 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.257 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.257 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:01.257 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.667771+0000", 2024-09-21T13:54:01.257 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.98887278899999997 2024-09-21T13:54:01.257 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.257 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.257 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:54:01.258 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.667865+0000", 2024-09-21T13:54:01.258 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4455000000000003e-05 2024-09-21T13:54:01.258 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.258 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.258 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,4", 2024-09-21T13:54:01.258 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.667950+0000", 2024-09-21T13:54:01.258 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4398000000000001e-05 2024-09-21T13:54:01.258 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.258 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.258 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:01.258 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.678051+0000", 2024-09-21T13:54:01.258 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.010101433 2024-09-21T13:54:01.258 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.259 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.259 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:01.259 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.678093+0000", 2024-09-21T13:54:01.259 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2284999999999999e-05 2024-09-21T13:54:01.259 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.259 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.259 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:54:01.259 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.678106+0000", 2024-09-21T13:54:01.259 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2992e-05 2024-09-21T13:54:01.259 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.259 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.259 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:54:01.259 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.678152+0000", 2024-09-21T13:54:01.259 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5995999999999998e-05 2024-09-21T13:54:01.259 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.260 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.260 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:54:01.260 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.678192+0000", 2024-09-21T13:54:01.260 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9439e-05 2024-09-21T13:54:01.260 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:01.260 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:54:01.260 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:01.260 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.260 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.260 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:229757 7.3 7:c8df107a:::benchmark_data_smithi049_76798_object93337:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e159)", 2024-09-21T13:54:01.260 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:53:15.688215+0000", 2024-09-21T13:54:01.260 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 45.465755010999999, 2024-09-21T13:54:01.260 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.99018317300000003, 2024-09-21T13:54:01.260 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:54:01.260 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:54:01.261 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:01.261 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:54:01.261 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:54:01.261 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:01.261 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 229757 2024-09-21T13:54:01.261 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.261 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:54:01.261 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.261 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:54:01.261 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:15.688215+0000", 2024-09-21T13:54:01.261 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:01.261 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.261 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.261 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:54:01.261 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:15.688215+0000", 2024-09-21T13:54:01.261 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:01.262 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.262 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.262 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:54:01.262 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:15.688218+0000", 2024-09-21T13:54:01.262 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9579999999999999e-06 2024-09-21T13:54:01.262 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.262 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.262 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:54:01.262 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:15.688227+0000", 2024-09-21T13:54:01.262 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6220000000000004e-06 2024-09-21T13:54:01.262 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.262 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.262 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:54:01.262 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:15.688251+0000", 2024-09-21T13:54:01.262 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3845e-05 2024-09-21T13:54:01.262 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.263 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.263 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:54:01.263 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:15.688261+0000", 2024-09-21T13:54:01.263 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0029e-05 2024-09-21T13:54:01.263 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.263 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.263 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:01.263 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:15.688336+0000", 2024-09-21T13:54:01.263 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4846999999999997e-05 2024-09-21T13:54:01.263 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.263 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.263 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:54:01.263 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:15.688348+0000", 2024-09-21T13:54:01.263 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2435e-05 2024-09-21T13:54:01.264 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.264 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.264 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:01.264 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.669817+0000", 2024-09-21T13:54:01.264 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.98146878299999996 2024-09-21T13:54:01.264 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.264 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.264 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:54:01.264 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.669936+0000", 2024-09-21T13:54:01.264 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011935 2024-09-21T13:54:01.264 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.264 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.264 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,4", 2024-09-21T13:54:01.264 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.670008+0000", 2024-09-21T13:54:01.265 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1585e-05 2024-09-21T13:54:01.265 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.265 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.265 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:54:01.265 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.678198+0000", 2024-09-21T13:54:01.265 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0081895189999999993 2024-09-21T13:54:01.265 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.265 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.265 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:01.265 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.678248+0000", 2024-09-21T13:54:01.265 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0923e-05 2024-09-21T13:54:01.265 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.265 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.265 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:01.265 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.678308+0000", 2024-09-21T13:54:01.265 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9706000000000001e-05 2024-09-21T13:54:01.266 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.266 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.266 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:54:01.266 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.678357+0000", 2024-09-21T13:54:01.266 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8528e-05 2024-09-21T13:54:01.266 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.266 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.266 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:54:01.266 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.678398+0000", 2024-09-21T13:54:01.266 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:01.267 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1043e-05 2024-09-21T13:54:01.267 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:01.267 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:54:01.267 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:01.267 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.267 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.268 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:229771 7.3 7:cdb024c5:::benchmark_data_smithi049_76798_object93351:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e159)", 2024-09-21T13:54:01.268 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:53:15.707807+0000", 2024-09-21T13:54:01.268 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 45.446162092999998, 2024-09-21T13:54:01.268 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97081205699999995, 2024-09-21T13:54:01.268 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:54:01.268 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:54:01.268 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:01.268 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:54:01.268 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:54:01.268 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:01.268 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 229771 2024-09-21T13:54:01.268 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.268 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:54:01.268 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.268 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:54:01.268 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:15.707807+0000", 2024-09-21T13:54:01.269 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:01.269 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.269 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.269 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:54:01.269 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:15.707807+0000", 2024-09-21T13:54:01.269 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:01.269 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.269 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.269 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:54:01.269 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:15.707810+0000", 2024-09-21T13:54:01.269 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.215e-06 2024-09-21T13:54:01.269 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.269 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.269 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:54:01.270 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:15.707816+0000", 2024-09-21T13:54:01.270 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4829999999999996e-06 2024-09-21T13:54:01.270 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.270 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.270 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:54:01.270 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:15.707837+0000", 2024-09-21T13:54:01.270 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0395999999999999e-05 2024-09-21T13:54:01.270 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.270 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.270 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:54:01.270 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:15.707847+0000", 2024-09-21T13:54:01.270 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9539999999999999e-06 2024-09-21T13:54:01.270 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.270 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.270 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:01.270 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:15.709036+0000", 2024-09-21T13:54:01.271 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001189098 2024-09-21T13:54:01.271 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.271 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.271 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:54:01.271 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:15.709046+0000", 2024-09-21T13:54:01.271 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0416e-05 2024-09-21T13:54:01.271 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.271 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.271 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:01.271 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.672456+0000", 2024-09-21T13:54:01.271 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.963409666 2024-09-21T13:54:01.271 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.271 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.271 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:54:01.271 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.672603+0000", 2024-09-21T13:54:01.272 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014728199999999999 2024-09-21T13:54:01.272 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.272 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.272 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,4", 2024-09-21T13:54:01.272 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.672811+0000", 2024-09-21T13:54:01.272 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00020784699999999999 2024-09-21T13:54:01.272 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.272 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.272 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:54:01.272 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.678202+0000", 2024-09-21T13:54:01.272 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.005390794 2024-09-21T13:54:01.272 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.272 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.272 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:01.273 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.678460+0000", 2024-09-21T13:54:01.273 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00025832999999999997 2024-09-21T13:54:01.273 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.273 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.273 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:01.273 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.678530+0000", 2024-09-21T13:54:01.273 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0178000000000005e-05 2024-09-21T13:54:01.273 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.273 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.273 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:54:01.273 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.678584+0000", 2024-09-21T13:54:01.273 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3667e-05 2024-09-21T13:54:01.273 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.273 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.273 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:54:01.273 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.678620+0000", 2024-09-21T13:54:01.274 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5731000000000001e-05 2024-09-21T13:54:01.274 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:01.274 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:54:01.274 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:01.274 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.274 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.274 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:229774 7.3 7:c563f8c1:::benchmark_data_smithi049_76798_object93354:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e159)", 2024-09-21T13:54:01.274 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:53:15.718455+0000", 2024-09-21T13:54:01.274 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 45.435514101000003, 2024-09-21T13:54:01.274 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.96031567200000001, 2024-09-21T13:54:01.274 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:54:01.274 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:54:01.274 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:01.274 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:54:01.275 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:54:01.275 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:01.275 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 229774 2024-09-21T13:54:01.275 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.275 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:54:01.275 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.275 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:54:01.275 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:15.718455+0000", 2024-09-21T13:54:01.275 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:01.275 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.275 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.275 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:54:01.275 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:15.718455+0000", 2024-09-21T13:54:01.275 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:01.275 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.276 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.276 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:54:01.276 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:15.718457+0000", 2024-09-21T13:54:01.276 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7579999999999999e-06 2024-09-21T13:54:01.276 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.276 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.276 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:54:01.276 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:15.718461+0000", 2024-09-21T13:54:01.276 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.49e-06 2024-09-21T13:54:01.276 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.276 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.276 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:54:01.276 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:15.718471+0000", 2024-09-21T13:54:01.277 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8439999999999993e-06 2024-09-21T13:54:01.277 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.277 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.277 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:54:01.277 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:15.718477+0000", 2024-09-21T13:54:01.277 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.438e-06 2024-09-21T13:54:01.277 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.277 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.277 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:01.277 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:15.718609+0000", 2024-09-21T13:54:01.277 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013200399999999999 2024-09-21T13:54:01.277 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.277 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.277 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:54:01.278 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:15.718619+0000", 2024-09-21T13:54:01.278 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0196e-05 2024-09-21T13:54:01.278 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.278 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.278 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:01.278 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.675525+0000", 2024-09-21T13:54:01.278 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.95690540800000001 2024-09-21T13:54:01.278 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.278 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.278 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:54:01.278 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.675630+0000", 2024-09-21T13:54:01.278 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000105831 2024-09-21T13:54:01.278 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.278 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.278 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,4", 2024-09-21T13:54:01.279 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.675741+0000", 2024-09-21T13:54:01.279 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011034700000000001 2024-09-21T13:54:01.279 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.279 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.279 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:54:01.279 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.678410+0000", 2024-09-21T13:54:01.279 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002668934 2024-09-21T13:54:01.279 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.279 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.279 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:01.279 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.678656+0000", 2024-09-21T13:54:01.279 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00024622100000000003 2024-09-21T13:54:01.279 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.279 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.280 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:01.280 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.678691+0000", 2024-09-21T13:54:01.280 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4949e-05 2024-09-21T13:54:01.280 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.280 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.280 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:54:01.280 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.678739+0000", 2024-09-21T13:54:01.280 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8084999999999998e-05 2024-09-21T13:54:01.280 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.280 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.280 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:54:01.281 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:54:01.281 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.678771+0000", 2024-09-21T13:54:01.281 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2166999999999999e-05 2024-09-21T13:54:01.281 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:01.281 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:54:01.281 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:01.281 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.281 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.281 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:229776 7.3 7:cb2e5c70:::benchmark_data_smithi049_76798_object93356:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e159)", 2024-09-21T13:54:01.281 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:53:15.718505+0000", 2024-09-21T13:54:01.281 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 45.435465078999997, 2024-09-21T13:54:01.281 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.96966926899999994, 2024-09-21T13:54:01.282 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:54:01.282 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:54:01.282 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:01.282 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:54:01.282 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:54:01.282 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:01.282 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 229776 2024-09-21T13:54:01.282 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.282 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:54:01.282 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.282 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:54:01.282 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:15.718505+0000", 2024-09-21T13:54:01.282 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:01.282 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.282 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.283 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:54:01.283 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:15.718505+0000", 2024-09-21T13:54:01.283 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:01.283 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.283 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.283 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:54:01.283 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:15.718505+0000", 2024-09-21T13:54:01.283 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9800000000000004e-07 2024-09-21T13:54:01.283 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.283 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.283 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:54:01.283 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:15.718508+0000", 2024-09-21T13:54:01.283 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.52e-06 2024-09-21T13:54:01.284 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.284 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.284 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:54:01.284 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:15.718514+0000", 2024-09-21T13:54:01.284 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8630000000000002e-06 2024-09-21T13:54:01.284 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.284 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.284 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:54:01.284 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:15.718519+0000", 2024-09-21T13:54:01.284 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1080000000000001e-06 2024-09-21T13:54:01.284 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.284 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.284 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:01.284 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:15.718640+0000", 2024-09-21T13:54:01.284 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000120326 2024-09-21T13:54:01.285 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.285 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.285 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:54:01.285 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:15.718645+0000", 2024-09-21T13:54:01.285 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6080000000000003e-06 2024-09-21T13:54:01.285 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.285 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.285 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:01.285 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.678795+0000", 2024-09-21T13:54:01.285 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.96014993500000001 2024-09-21T13:54:01.285 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.285 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.285 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:54:01.285 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.678884+0000", 2024-09-21T13:54:01.285 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.8729999999999999e-05 2024-09-21T13:54:01.286 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.286 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.286 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,4", 2024-09-21T13:54:01.286 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.679023+0000", 2024-09-21T13:54:01.286 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013920100000000001 2024-09-21T13:54:01.286 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.286 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.286 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:01.286 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.687990+0000", 2024-09-21T13:54:01.286 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.008967081 2024-09-21T13:54:01.286 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.286 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.286 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:01.286 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.688036+0000", 2024-09-21T13:54:01.287 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5972e-05 2024-09-21T13:54:01.287 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.287 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.287 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:54:01.287 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.688047+0000", 2024-09-21T13:54:01.287 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0412e-05 2024-09-21T13:54:01.287 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.287 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.287 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:54:01.287 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.688134+0000", 2024-09-21T13:54:01.287 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6927000000000003e-05 2024-09-21T13:54:01.287 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.287 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.287 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:54:01.288 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.688174+0000", 2024-09-21T13:54:01.288 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0088e-05 2024-09-21T13:54:01.288 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:01.288 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:54:01.288 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:01.288 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.288 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.288 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:229823 7.3 7:c04fa0ac:::benchmark_data_smithi049_76798_object93403:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e159)", 2024-09-21T13:54:01.288 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:53:15.789559+0000", 2024-09-21T13:54:01.288 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 45.364410102999997, 2024-09-21T13:54:01.288 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.91220963399999999, 2024-09-21T13:54:01.288 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:54:01.288 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:54:01.288 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:01.289 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:54:01.289 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:54:01.289 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:01.289 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 229823 2024-09-21T13:54:01.289 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.289 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:54:01.289 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.289 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:54:01.289 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:15.789559+0000", 2024-09-21T13:54:01.289 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:01.289 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.289 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.289 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:54:01.289 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:15.789559+0000", 2024-09-21T13:54:01.290 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:01.290 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.290 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.290 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:54:01.290 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:15.789561+0000", 2024-09-21T13:54:01.290 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.46e-06 2024-09-21T13:54:01.290 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.290 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.290 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:54:01.290 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:15.789567+0000", 2024-09-21T13:54:01.290 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0689999999999998e-06 2024-09-21T13:54:01.290 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.290 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.290 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:54:01.291 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:15.789578+0000", 2024-09-21T13:54:01.291 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1408e-05 2024-09-21T13:54:01.291 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.291 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.291 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:54:01.291 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:15.789585+0000", 2024-09-21T13:54:01.291 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7039999999999997e-06 2024-09-21T13:54:01.291 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.291 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.291 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:01.291 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:15.789641+0000", 2024-09-21T13:54:01.291 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5992999999999997e-05 2024-09-21T13:54:01.291 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.291 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.291 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:54:01.292 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:15.789651+0000", 2024-09-21T13:54:01.292 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4939999999999992e-06 2024-09-21T13:54:01.292 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.292 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.292 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:01.292 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.688525+0000", 2024-09-21T13:54:01.292 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.89887427200000003 2024-09-21T13:54:01.292 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.292 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.292 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:54:01.292 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.688615+0000", 2024-09-21T13:54:01.292 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0424e-05 2024-09-21T13:54:01.292 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.292 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.293 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,4", 2024-09-21T13:54:01.293 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.688714+0000", 2024-09-21T13:54:01.293 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8970999999999998e-05 2024-09-21T13:54:01.293 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.293 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.293 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:01.293 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.694524+0000", 2024-09-21T13:54:01.293 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0058098760000000003 2024-09-21T13:54:01.293 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.293 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.293 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:01.293 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.697439+0000", 2024-09-21T13:54:01.293 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0029144209999999999 2024-09-21T13:54:01.293 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.293 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.293 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:54:01.294 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.701685+0000", 2024-09-21T13:54:01.294 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0042459330000000003 2024-09-21T13:54:01.294 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.294 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.294 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:54:01.294 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.701730+0000", 2024-09-21T13:54:01.294 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5306999999999998e-05 2024-09-21T13:54:01.294 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:01.294 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:01.294 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:54:01.294 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.701769+0000", 2024-09-21T13:54:01.294 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9301999999999997e-05 2024-09-21T13:54:01.294 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:01.294 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:54:01.294 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:01.294 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:01.295 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:54:01.295 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T13:54:01.318 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:01.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:01.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:01.324 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-09-21T13:54:01.420 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:54:01.441 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:01.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:01.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:54:01.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:01 smithi049 ceph-mon[30080]: pgmap v240: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 114 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 935 op/s; 2.1 MiB/s, 33 objects/s recovering 2024-09-21T13:54:01.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:01 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2477544974' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:54:01.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:01 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:54:01.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:01 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:54:01.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:01 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:54:01.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:01 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:54:01.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:01 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:54:01.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:01 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:54:01.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:01 smithi049 ceph-mon[40489]: pgmap v240: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 114 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 935 op/s; 2.1 MiB/s, 33 objects/s recovering 2024-09-21T13:54:01.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:01 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2477544974' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:54:01.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:01 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:54:01.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:01 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:54:01.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:01 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:54:01.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:01 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:54:01.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:01 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:54:01.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:01 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:54:01.543 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:54:01.565 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:01.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:01.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:54:01.666 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:01.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:01.690 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:01.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:01.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:01 smithi187 ceph-mon[33032]: pgmap v240: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 114 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 935 op/s; 2.1 MiB/s, 33 objects/s recovering 2024-09-21T13:54:01.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:01 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2477544974' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:54:01.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:01 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:54:01.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:01 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:54:01.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:01 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:54:01.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:01 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:54:01.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:01 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:54:01.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:01 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:54:01.792 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:01.813 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:01.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:01.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:01.914 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:54:01.935 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:01.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:01.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:54:02.037 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:54:02.062 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:02.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:02.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:54:02.163 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:54:02.187 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:02.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:02.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:54:02.289 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:02.312 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:02.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:02.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:02.414 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:54:02.437 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:02.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:02.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:54:02.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:02 smithi049 ceph-mon[30080]: 7.3 scrub starts 2024-09-21T13:54:02.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:02 smithi049 ceph-mon[30080]: 7.3 scrub ok 2024-09-21T13:54:02.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:02 smithi049 ceph-mon[30080]: 3.14 scrub starts 2024-09-21T13:54:02.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:02 smithi049 ceph-mon[30080]: 3.14 scrub ok 2024-09-21T13:54:02.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:02 smithi049 ceph-mon[30080]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T13:54:02.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:02 smithi049 ceph-mon[30080]: Cluster is now healthy 2024-09-21T13:54:02.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:02 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T13:54:02.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:02 smithi049 ceph-mon[30080]: osdmap e177: 8 total, 8 up, 8 in 2024-09-21T13:54:02.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:02 smithi049 ceph-mon[30080]: 5.1e scrub starts 2024-09-21T13:54:02.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:02 smithi049 ceph-mon[30080]: 5.1e scrub ok 2024-09-21T13:54:02.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:02 smithi049 ceph-mon[40489]: 7.3 scrub starts 2024-09-21T13:54:02.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:02 smithi049 ceph-mon[40489]: 7.3 scrub ok 2024-09-21T13:54:02.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:02 smithi049 ceph-mon[40489]: 3.14 scrub starts 2024-09-21T13:54:02.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:02 smithi049 ceph-mon[40489]: 3.14 scrub ok 2024-09-21T13:54:02.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:02 smithi049 ceph-mon[40489]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T13:54:02.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:02 smithi049 ceph-mon[40489]: Cluster is now healthy 2024-09-21T13:54:02.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:02 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T13:54:02.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:02 smithi049 ceph-mon[40489]: osdmap e177: 8 total, 8 up, 8 in 2024-09-21T13:54:02.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:02 smithi049 ceph-mon[40489]: 5.1e scrub starts 2024-09-21T13:54:02.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:02 smithi049 ceph-mon[40489]: 5.1e scrub ok 2024-09-21T13:54:02.538 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:02.561 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:02.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:02.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:02.663 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:02.686 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:02.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:02.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:02.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:02 smithi187 ceph-mon[33032]: 7.3 scrub starts 2024-09-21T13:54:02.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:02 smithi187 ceph-mon[33032]: 7.3 scrub ok 2024-09-21T13:54:02.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:02 smithi187 ceph-mon[33032]: 3.14 scrub starts 2024-09-21T13:54:02.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:02 smithi187 ceph-mon[33032]: 3.14 scrub ok 2024-09-21T13:54:02.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:02 smithi187 ceph-mon[33032]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T13:54:02.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:02 smithi187 ceph-mon[33032]: Cluster is now healthy 2024-09-21T13:54:02.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:02 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T13:54:02.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:02 smithi187 ceph-mon[33032]: osdmap e177: 8 total, 8 up, 8 in 2024-09-21T13:54:02.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:02 smithi187 ceph-mon[33032]: 5.1e scrub starts 2024-09-21T13:54:02.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:02 smithi187 ceph-mon[33032]: 5.1e scrub ok 2024-09-21T13:54:02.787 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:02.810 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:02.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:02.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:02.911 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:02.937 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:02.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:02.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:03.038 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:03.063 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:03.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:03.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:03.127 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T13:54:03.157 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:54:03.165 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:03.193 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:03.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:03.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:03.294 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:03.316 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:03.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:03.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:03.327 INFO:teuthology.orchestra.run.smithi049.stderr:set 3.14 pg_upmap mapping to [7,6,0] 2024-09-21T13:54:03.400 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:54:03.478 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:54:03.479 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:03 smithi187 ceph-mon[33032]: pgmap v243: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 114 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 743 op/s 2024-09-21T13:54:03.479 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:03 smithi187 ceph-mon[33032]: 3.12 scrub starts 2024-09-21T13:54:03.479 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:03 smithi187 ceph-mon[33032]: 3.12 scrub ok 2024-09-21T13:54:03.479 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:03 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3034271898' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "3.14", "id": [7, 6, 0]} : dispatch 2024-09-21T13:54:03.479 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:03 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "3.14", "id": [7, 6, 0]} : dispatch 2024-09-21T13:54:03.500 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:03.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:03.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:54:03.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:03 smithi049 ceph-mon[40489]: pgmap v243: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 114 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 743 op/s 2024-09-21T13:54:03.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:03 smithi049 ceph-mon[40489]: 3.12 scrub starts 2024-09-21T13:54:03.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:03 smithi049 ceph-mon[40489]: 3.12 scrub ok 2024-09-21T13:54:03.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:03 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3034271898' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "3.14", "id": [7, 6, 0]} : dispatch 2024-09-21T13:54:03.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:03 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "3.14", "id": [7, 6, 0]} : dispatch 2024-09-21T13:54:03.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:03 smithi049 ceph-mon[30080]: pgmap v243: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 114 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 743 op/s 2024-09-21T13:54:03.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:03 smithi049 ceph-mon[30080]: 3.12 scrub starts 2024-09-21T13:54:03.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:03 smithi049 ceph-mon[30080]: 3.12 scrub ok 2024-09-21T13:54:03.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:03 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3034271898' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "3.14", "id": [7, 6, 0]} : dispatch 2024-09-21T13:54:03.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:03 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "3.14", "id": [7, 6, 0]} : dispatch 2024-09-21T13:54:03.602 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:54:03.651 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:03.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:03.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:54:03.753 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:54:03.775 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:03.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:03.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:54:03.874 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T13:54:03.875 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-21T13:54:03.875 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 0 2024-09-21T13:54:03.875 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T13:54:03.876 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:54:03.898 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:03.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:03.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:54:04.000 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:54:04.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:04.023 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:04.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:54:04.124 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:04.146 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:04.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:04.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:04.247 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:04.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:04.270 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:04.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:04.371 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:04.393 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:04.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:04.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:04.495 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:04.496 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:04 smithi049 ceph-mon[40489]: 5.15 scrub starts 2024-09-21T13:54:04.496 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:04 smithi049 ceph-mon[40489]: 5.15 scrub ok 2024-09-21T13:54:04.496 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:04 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "3.14", "id": [7, 6, 0]}]': finished 2024-09-21T13:54:04.496 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:04 smithi049 ceph-mon[40489]: osdmap e178: 8 total, 8 up, 8 in 2024-09-21T13:54:04.496 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:04 smithi049 ceph-mon[40489]: 6.8 scrub starts 2024-09-21T13:54:04.496 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:04 smithi049 ceph-mon[40489]: 6.8 scrub ok 2024-09-21T13:54:04.497 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:04 smithi049 ceph-mon[30080]: 5.15 scrub starts 2024-09-21T13:54:04.497 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:04 smithi049 ceph-mon[30080]: 5.15 scrub ok 2024-09-21T13:54:04.497 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:04 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "3.14", "id": [7, 6, 0]}]': finished 2024-09-21T13:54:04.497 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:04 smithi049 ceph-mon[30080]: osdmap e178: 8 total, 8 up, 8 in 2024-09-21T13:54:04.497 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:04 smithi049 ceph-mon[30080]: 6.8 scrub starts 2024-09-21T13:54:04.497 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:04 smithi049 ceph-mon[30080]: 6.8 scrub ok 2024-09-21T13:54:04.517 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:04.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:04.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:04.619 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:04.640 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:04.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:04.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:04.661 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-09-21T13:54:04.661 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-09-21T13:54:04.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:04 smithi187 ceph-mon[33032]: 5.15 scrub starts 2024-09-21T13:54:04.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:04 smithi187 ceph-mon[33032]: 5.15 scrub ok 2024-09-21T13:54:04.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:04 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "3.14", "id": [7, 6, 0]}]': finished 2024-09-21T13:54:04.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:04 smithi187 ceph-mon[33032]: osdmap e178: 8 total, 8 up, 8 in 2024-09-21T13:54:04.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:04 smithi187 ceph-mon[33032]: 6.8 scrub starts 2024-09-21T13:54:04.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:04 smithi187 ceph-mon[33032]: 6.8 scrub ok 2024-09-21T13:54:04.741 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:54:04.763 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:04.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:04.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:54:04.779 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:54:04.779 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:54:04.782 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T13:54:04.785 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:54:04.786 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:54:04.818 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:54:04.818 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:54:04.851 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:54:04.851 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:54:04.864 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:04.880 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:54:04.881 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:54:04.886 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:04.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:04.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:04.906 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:54:04.907 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:54:04.934 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:54:04.934 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:54:04.951 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:54:04.951 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:54:04.988 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:54:05.010 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:05.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:05.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:54:05.073 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:54:05.112 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:54:05.134 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:05.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:05.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:54:05.235 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:54:05.257 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:05.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:05.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:54:05.359 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:54:05.381 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:05.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:05.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:54:05.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:05 smithi187 ceph-mon[33032]: pgmap v245: 148 pgs: 1 peering, 1 active+clean+scrubbing, 146 active+clean; 19 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 996 op/s 2024-09-21T13:54:05.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:05 smithi187 ceph-mon[33032]: 7.f scrub starts 2024-09-21T13:54:05.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:05 smithi187 ceph-mon[33032]: 7.f scrub ok 2024-09-21T13:54:05.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:05 smithi187 ceph-mon[33032]: 4.1b scrub starts 2024-09-21T13:54:05.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:05 smithi187 ceph-mon[33032]: 4.1b scrub ok 2024-09-21T13:54:05.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:05 smithi187 ceph-mon[33032]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-21T13:54:05.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:05 smithi187 ceph-mon[33032]: osdmap e179: 8 total, 8 up, 8 in 2024-09-21T13:54:05.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:05 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2453220554' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:54:05.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:05 smithi187 ceph-mon[33032]: osdmap e180: 8 total, 8 up, 8 in 2024-09-21T13:54:05.483 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:05.505 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:05.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:05.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:05.590 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T13:54:05.593 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T13:54:05.593 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:54:05.607 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:05.608 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:05 smithi049 ceph-mon[40489]: pgmap v245: 148 pgs: 1 peering, 1 active+clean+scrubbing, 146 active+clean; 19 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 996 op/s 2024-09-21T13:54:05.609 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:05 smithi049 ceph-mon[40489]: 7.f scrub starts 2024-09-21T13:54:05.609 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:05 smithi049 ceph-mon[40489]: 7.f scrub ok 2024-09-21T13:54:05.609 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:05 smithi049 ceph-mon[40489]: 4.1b scrub starts 2024-09-21T13:54:05.609 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:05 smithi049 ceph-mon[40489]: 4.1b scrub ok 2024-09-21T13:54:05.609 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:05 smithi049 ceph-mon[40489]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-21T13:54:05.609 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:05 smithi049 ceph-mon[40489]: osdmap e179: 8 total, 8 up, 8 in 2024-09-21T13:54:05.609 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:05 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2453220554' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:54:05.609 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:05 smithi049 ceph-mon[40489]: osdmap e180: 8 total, 8 up, 8 in 2024-09-21T13:54:05.610 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:05 smithi049 ceph-mon[30080]: pgmap v245: 148 pgs: 1 peering, 1 active+clean+scrubbing, 146 active+clean; 19 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 996 op/s 2024-09-21T13:54:05.610 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:05 smithi049 ceph-mon[30080]: 7.f scrub starts 2024-09-21T13:54:05.610 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:05 smithi049 ceph-mon[30080]: 7.f scrub ok 2024-09-21T13:54:05.610 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:05 smithi049 ceph-mon[30080]: 4.1b scrub starts 2024-09-21T13:54:05.610 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:05 smithi049 ceph-mon[30080]: 4.1b scrub ok 2024-09-21T13:54:05.610 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:05 smithi049 ceph-mon[30080]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-21T13:54:05.610 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:05 smithi049 ceph-mon[30080]: osdmap e179: 8 total, 8 up, 8 in 2024-09-21T13:54:05.610 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:05 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2453220554' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:54:05.610 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:05 smithi049 ceph-mon[30080]: osdmap e180: 8 total, 8 up, 8 in 2024-09-21T13:54:05.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:05.629 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:05.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:05.703 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:54:05.729 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:54:05.750 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:05.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:05.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:54:05.852 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:54:05.875 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:05.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:05.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:54:05.977 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:06.002 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:06.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:06.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:06.103 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:54:06.124 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:06.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:06.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:54:06.226 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:54:06.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:06.247 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:06.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:54:06.269 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:54:06.348 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:06.370 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:06.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:06.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:06.389 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T13:54:06.472 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:54:06.493 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:06.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:06.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:54:06.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:06 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:54:06.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:06 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:54:06.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:06 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:54:06.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:06 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:54:06.595 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:54:06.616 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:06.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:06.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:54:06.637 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:54:06.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:06 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:54:06.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:06 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:54:06.717 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:54:06.739 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:06.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:06.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:54:06.841 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:06.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:06.863 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:06.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:06.964 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:54:06.986 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:06.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:06.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:54:07.075 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T13:54:07.076 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:54:07.088 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:54:07.110 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:07.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:07.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:54:07.212 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:07.234 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:07.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:07.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:07.335 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:54:07.349 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-09-21T13:54:07.355 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'unique_pool_0' removed 2024-09-21T13:54:07.358 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:07.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:07.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:54:07.460 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:54:07.482 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:07.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:07.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:54:07.584 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:54:07.605 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:07.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:07.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:54:07.641 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T13:54:07.676 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:07 smithi049 ceph-mon[40489]: pgmap v248: 148 pgs: 1 active+clean+scrubbing, 1 peering, 146 active+clean; 583 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 408 op/s 2024-09-21T13:54:07.676 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:07 smithi049 ceph-mon[40489]: 5.3 scrub starts 2024-09-21T13:54:07.677 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:07 smithi049 ceph-mon[40489]: 3.1b scrub starts 2024-09-21T13:54:07.677 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:07 smithi049 ceph-mon[40489]: 5.3 scrub ok 2024-09-21T13:54:07.677 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:07 smithi049 ceph-mon[40489]: 3.1b scrub ok 2024-09-21T13:54:07.677 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:07 smithi049 ceph-mon[40489]: osdmap e181: 8 total, 8 up, 8 in 2024-09-21T13:54:07.677 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:07 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1378708144' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T13:54:07.677 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:07 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T13:54:07.677 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:07 smithi049 ceph-mon[40489]: 5.c scrub starts 2024-09-21T13:54:07.677 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:07 smithi049 ceph-mon[40489]: 5.c scrub ok 2024-09-21T13:54:07.677 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:07 smithi049 ceph-mon[40489]: 3.15 scrub starts 2024-09-21T13:54:07.677 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:07 smithi049 ceph-mon[40489]: 3.15 scrub ok 2024-09-21T13:54:07.677 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:07 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/407876136' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:54:07.678 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:07 smithi049 ceph-mon[30080]: pgmap v248: 148 pgs: 1 active+clean+scrubbing, 1 peering, 146 active+clean; 583 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 408 op/s 2024-09-21T13:54:07.678 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:07 smithi049 ceph-mon[30080]: 5.3 scrub starts 2024-09-21T13:54:07.678 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:07 smithi049 ceph-mon[30080]: 3.1b scrub starts 2024-09-21T13:54:07.678 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:07 smithi049 ceph-mon[30080]: 5.3 scrub ok 2024-09-21T13:54:07.678 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:07 smithi049 ceph-mon[30080]: 3.1b scrub ok 2024-09-21T13:54:07.678 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:07 smithi049 ceph-mon[30080]: osdmap e181: 8 total, 8 up, 8 in 2024-09-21T13:54:07.678 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:07 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1378708144' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T13:54:07.678 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:07 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T13:54:07.678 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:07 smithi049 ceph-mon[30080]: 5.c scrub starts 2024-09-21T13:54:07.678 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:07 smithi049 ceph-mon[30080]: 5.c scrub ok 2024-09-21T13:54:07.678 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:07 smithi049 ceph-mon[30080]: 3.15 scrub starts 2024-09-21T13:54:07.678 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:07 smithi049 ceph-mon[30080]: 3.15 scrub ok 2024-09-21T13:54:07.678 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:07 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/407876136' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:54:07.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:07 smithi187 ceph-mon[33032]: pgmap v248: 148 pgs: 1 active+clean+scrubbing, 1 peering, 146 active+clean; 583 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 408 op/s 2024-09-21T13:54:07.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:07 smithi187 ceph-mon[33032]: 5.3 scrub starts 2024-09-21T13:54:07.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:07 smithi187 ceph-mon[33032]: 3.1b scrub starts 2024-09-21T13:54:07.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:07 smithi187 ceph-mon[33032]: 5.3 scrub ok 2024-09-21T13:54:07.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:07 smithi187 ceph-mon[33032]: 3.1b scrub ok 2024-09-21T13:54:07.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:07 smithi187 ceph-mon[33032]: osdmap e181: 8 total, 8 up, 8 in 2024-09-21T13:54:07.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:07 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1378708144' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T13:54:07.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:07 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T13:54:07.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:07 smithi187 ceph-mon[33032]: 5.c scrub starts 2024-09-21T13:54:07.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:07 smithi187 ceph-mon[33032]: 5.c scrub ok 2024-09-21T13:54:07.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:07 smithi187 ceph-mon[33032]: 3.15 scrub starts 2024-09-21T13:54:07.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:07 smithi187 ceph-mon[33032]: 3.15 scrub ok 2024-09-21T13:54:07.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:07 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/407876136' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:54:07.707 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:07.728 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:07.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:07.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:07.829 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:54:07.852 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:07.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:07.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:54:07.954 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:54:07.976 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:07.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:07.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:54:08.000 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T13:54:08.078 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:54:08.103 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:08.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:08.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:54:08.205 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:08.228 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:08.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:08.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:08.291 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:54:08.330 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:54:08.345 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:08 smithi049 ceph-mon[30080]: 3.2 scrub starts 2024-09-21T13:54:08.345 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:08 smithi049 ceph-mon[30080]: 3.2 scrub ok 2024-09-21T13:54:08.357 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:08.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:08.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:54:08.458 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:08.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:08.481 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:08.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:08.582 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:08.603 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:08.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:08.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:08.693 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:08 smithi187 ceph-mon[33032]: 3.2 scrub starts 2024-09-21T13:54:08.693 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:08 smithi187 ceph-mon[33032]: 3.2 scrub ok 2024-09-21T13:54:08.693 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:08 smithi187 ceph-mon[33032]: 5.1 scrub starts 2024-09-21T13:54:08.693 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:08 smithi187 ceph-mon[33032]: 5.1 scrub ok 2024-09-21T13:54:08.693 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:08 smithi187 ceph-mon[33032]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:54:08.693 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:08 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T13:54:08.693 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:08 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/407876136' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T13:54:08.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:08 smithi187 ceph-mon[33032]: osdmap e182: 8 total, 8 up, 8 in 2024-09-21T13:54:08.705 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:08.725 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:08 smithi049 ceph-mon[40489]: 3.2 scrub starts 2024-09-21T13:54:08.726 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:08 smithi049 ceph-mon[40489]: 3.2 scrub ok 2024-09-21T13:54:08.726 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:08 smithi049 ceph-mon[40489]: 5.1 scrub starts 2024-09-21T13:54:08.726 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:08 smithi049 ceph-mon[40489]: 5.1 scrub ok 2024-09-21T13:54:08.726 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:08 smithi049 ceph-mon[40489]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:54:08.726 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:08 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T13:54:08.726 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:08 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/407876136' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T13:54:08.726 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:08 smithi049 ceph-mon[40489]: osdmap e182: 8 total, 8 up, 8 in 2024-09-21T13:54:08.726 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:08 smithi049 ceph-mon[30080]: 5.1 scrub starts 2024-09-21T13:54:08.726 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:08 smithi049 ceph-mon[30080]: 5.1 scrub ok 2024-09-21T13:54:08.726 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:08 smithi049 ceph-mon[30080]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:54:08.726 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:08 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T13:54:08.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:08 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/407876136' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T13:54:08.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:08 smithi049 ceph-mon[30080]: osdmap e182: 8 total, 8 up, 8 in 2024-09-21T13:54:08.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:08.727 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:08.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:08.753 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-21T13:54:08.753 INFO:tasks.radosbench:Beginning radosbench... 2024-09-21T13:54:08.753 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-09-21T13:54:08.754 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd pool create unique_pool_1 16 2024-09-21T13:54:08.802 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T13:54:08.802 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:54:08.813 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:54:08.828 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:08.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:08.850 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:08.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:08.951 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:08.975 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:08.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:08.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:09.077 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:09.099 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:09.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:09.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:09.200 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:09.211 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 2] 2024-09-21T13:54:09.211 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T13:54:09.223 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:09.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:09.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:09.324 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:54:09.325 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:54:09.326 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:54:09.343 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:54:09.345 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:54:09.346 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:54:09.350 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:54:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:54:09.351 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:09.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:09.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:54:09.365 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:54:09.365 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:54:09.383 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:54:09.383 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:54:09.399 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:54:09.400 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:54:09.416 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:54:09.416 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:54:09.433 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T13:54:09.434 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:54:09.434 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:54:09.450 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:54:09.450 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:54:09.453 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:09.498 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:09.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:09.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:09.588 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:09 smithi049 ceph-mon[40489]: pgmap v251: 132 pgs: 1 peering, 131 active+clean; 583 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-09-21T13:54:09.588 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:09 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/672120918' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:54:09.588 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:09 smithi049 ceph-mon[30080]: pgmap v251: 132 pgs: 1 peering, 131 active+clean; 583 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-09-21T13:54:09.589 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:09 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/672120918' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:54:09.600 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:09.622 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:09.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:09.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:09.681 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:54:09.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:09 smithi187 ceph-mon[33032]: pgmap v251: 132 pgs: 1 peering, 131 active+clean; 583 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-09-21T13:54:09.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:09 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/672120918' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:54:09.724 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:54:09.746 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:09.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:09.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:54:09.848 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:54:09.869 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:09.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:09.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:54:09.972 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:09.993 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:09.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:09.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:10.093 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T13:54:10.095 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:54:10.116 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:10.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:10.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:54:10.210 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T13:54:10.210 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-21T13:54:10.210 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-21T13:54:10.210 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-21T13:54:10.210 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T13:54:10.218 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:10.241 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:10.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:10.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:10.343 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:10.364 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:10.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:10.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:10.465 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:10.489 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:10.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:10.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:10.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:10 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:54:10.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:10 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:54:10.535 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:54:10.591 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:10.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:10.613 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:10.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:10.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:10 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:54:10.714 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:54:10.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:10.735 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:10.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:54:10.830 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:54:10 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:54:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:54:10.836 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:10.859 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:10.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:10.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:10.943 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T13:54:10.960 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:54:10.982 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:10.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:10.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:54:11.084 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:54:11.105 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:11.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:11.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:54:11.187 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:54:11.207 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:54:11.229 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:11.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:11.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:54:11.330 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:54:11.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:11.352 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:11.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:54:11.362 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-09-21T13:54:11.366 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'unique_pool_1' created 2024-09-21T13:54:11.453 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:11.474 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:11.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:11.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:11.569 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:11 smithi187 ceph-mon[33032]: pgmap v252: 132 pgs: 132 active+clean; 583 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 752 B/s rd, 0 op/s 2024-09-21T13:54:11.569 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:11 smithi187 ceph-mon[33032]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-21T13:54:11.569 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:11 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3546747537' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-21T13:54:11.569 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:11 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-21T13:54:11.569 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:11 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3649122924' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:54:11.575 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:54:11.596 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:11.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:11.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:54:11.694 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T13:54:11.694 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-21T13:54:11.695 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-21T13:54:11.695 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-21T13:54:11.695 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.695 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:49869 7.0 7:07d76dad:::benchmark_data_smithi049_76798_object49868:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T13:54:11.695 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:38.752211+0000", 2024-09-21T13:54:11.695 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 212.918670268, 2024-09-21T13:54:11.695 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.081978307, 2024-09-21T13:54:11.695 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:54:11.695 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:54:11.695 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:11.695 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:54:11.695 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:54:11.695 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:11.696 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 49869 2024-09-21T13:54:11.696 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.696 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:54:11.696 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.696 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:54:11.696 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.752211+0000", 2024-09-21T13:54:11.696 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:11.696 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.696 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.696 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:54:11.696 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.752211+0000", 2024-09-21T13:54:11.696 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:11.696 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.696 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.696 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:54:11.697 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.752212+0000", 2024-09-21T13:54:11.697 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2839999999999999e-06 2024-09-21T13:54:11.697 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.697 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.697 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:54:11.697 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.752258+0000", 2024-09-21T13:54:11.697 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5627999999999997e-05 2024-09-21T13:54:11.697 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.697 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.697 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:54:11.697 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.752269+0000", 2024-09-21T13:54:11.697 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0808999999999999e-05 2024-09-21T13:54:11.697 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.698 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.698 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:54:11.698 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.752275+0000", 2024-09-21T13:54:11.698 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6180000000000003e-06 2024-09-21T13:54:11.698 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.698 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.698 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:11.698 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.752315+0000", 2024-09-21T13:54:11.698 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9999000000000001e-05 2024-09-21T13:54:11.698 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.698 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.698 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:54:11.698 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.752421+0000", 2024-09-21T13:54:11.698 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000106604 2024-09-21T13:54:11.699 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.699 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.699 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3", 2024-09-21T13:54:11.699 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.752559+0000", 2024-09-21T13:54:11.699 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000137868 2024-09-21T13:54:11.699 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.699 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.699 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:54:11.699 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.754020+0000", 2024-09-21T13:54:11.699 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001461118 2024-09-21T13:54:11.699 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.699 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.699 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:11.699 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.758622+0000", 2024-09-21T13:54:11.700 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0046015600000000002 2024-09-21T13:54:11.700 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.700 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.700 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:11.700 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.834115+0000", 2024-09-21T13:54:11.700 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.075493219 2024-09-21T13:54:11.700 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.700 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.700 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:54:11.700 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.834157+0000", 2024-09-21T13:54:11.700 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2208e-05 2024-09-21T13:54:11.700 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.700 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.700 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:54:11.701 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.834189+0000", 2024-09-21T13:54:11.701 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2391999999999998e-05 2024-09-21T13:54:11.701 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:11.701 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:54:11.701 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:11.701 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.701 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.701 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:49879 7.0 7:01f9292a:::benchmark_data_smithi049_76798_object49878:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T13:54:11.701 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:38.759652+0000", 2024-09-21T13:54:11.701 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 212.91122929100001, 2024-09-21T13:54:11.701 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.083001569999999997, 2024-09-21T13:54:11.701 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:54:11.701 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:54:11.701 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:11.702 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:54:11.702 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:54:11.702 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:11.702 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 49879 2024-09-21T13:54:11.702 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.702 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:54:11.702 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.702 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:54:11.702 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.759652+0000", 2024-09-21T13:54:11.702 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:11.703 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.703 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.703 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:54:11.703 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.759652+0000", 2024-09-21T13:54:11.703 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:11.703 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.703 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.703 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:54:11.703 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.759653+0000", 2024-09-21T13:54:11.703 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.325e-06 2024-09-21T13:54:11.703 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.703 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.703 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:54:11.703 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.759716+0000", 2024-09-21T13:54:11.704 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2747000000000001e-05 2024-09-21T13:54:11.704 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.704 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.704 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:54:11.704 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.759730+0000", 2024-09-21T13:54:11.704 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3560999999999999e-05 2024-09-21T13:54:11.704 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.704 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.704 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:54:11.704 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.759738+0000", 2024-09-21T13:54:11.704 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8320000000000006e-06 2024-09-21T13:54:11.704 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.704 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.704 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:11.704 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.759790+0000", 2024-09-21T13:54:11.705 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2002000000000001e-05 2024-09-21T13:54:11.705 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.705 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.705 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:54:11.705 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.759956+0000", 2024-09-21T13:54:11.705 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00016607699999999999 2024-09-21T13:54:11.705 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.705 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.705 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3", 2024-09-21T13:54:11.705 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.760106+0000", 2024-09-21T13:54:11.705 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015017599999999999 2024-09-21T13:54:11.705 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.705 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.705 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:54:11.706 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.761441+0000", 2024-09-21T13:54:11.706 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0013353970000000001 2024-09-21T13:54:11.706 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.706 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.706 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:11.706 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.761942+0000", 2024-09-21T13:54:11.706 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00050108399999999995 2024-09-21T13:54:11.706 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.706 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.706 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:11.706 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.842549+0000", 2024-09-21T13:54:11.706 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.080607112999999994 2024-09-21T13:54:11.706 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.707 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.707 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:54:11.707 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.842611+0000", 2024-09-21T13:54:11.707 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1668000000000007e-05 2024-09-21T13:54:11.707 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.707 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.707 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:54:11.707 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.842654+0000", 2024-09-21T13:54:11.707 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2588e-05 2024-09-21T13:54:11.707 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:11.707 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:54:11.707 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:11.707 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.707 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.707 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:51707 7.0 7:0ca9656b:::benchmark_data_smithi049_76798_object51706:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-09-21T13:54:11.708 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:39.989086+0000", 2024-09-21T13:54:11.708 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 211.68179538999999, 2024-09-21T13:54:11.708 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.081225758999999995, 2024-09-21T13:54:11.708 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:54:11.708 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:54:11.708 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:11.708 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:54:11.708 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:54:11.708 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:11.708 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 51707 2024-09-21T13:54:11.708 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.708 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:54:11.708 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.709 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:54:11.709 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.989086+0000", 2024-09-21T13:54:11.709 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:11.709 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.709 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.709 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:54:11.709 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.989086+0000", 2024-09-21T13:54:11.709 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:11.709 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.709 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.709 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:54:11.709 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.989088+0000", 2024-09-21T13:54:11.709 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.592e-06 2024-09-21T13:54:11.709 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.710 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.710 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:54:11.710 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.989126+0000", 2024-09-21T13:54:11.710 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8670000000000001e-05 2024-09-21T13:54:11.710 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.710 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.710 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:54:11.710 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.989142+0000", 2024-09-21T13:54:11.710 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5704000000000001e-05 2024-09-21T13:54:11.710 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.710 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.710 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:54:11.711 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.989156+0000", 2024-09-21T13:54:11.711 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4156e-05 2024-09-21T13:54:11.711 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.711 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.711 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:11.711 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.989207+0000", 2024-09-21T13:54:11.711 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0905999999999997e-05 2024-09-21T13:54:11.711 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.711 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.711 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:54:11.711 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.989291+0000", 2024-09-21T13:54:11.711 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3602000000000004e-05 2024-09-21T13:54:11.711 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.712 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.712 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3", 2024-09-21T13:54:11.712 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.989372+0000", 2024-09-21T13:54:11.712 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1148000000000003e-05 2024-09-21T13:54:11.712 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.712 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.712 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:54:11.712 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.990876+0000", 2024-09-21T13:54:11.712 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001504411 2024-09-21T13:54:11.712 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.712 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.712 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:11.712 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:39.992325+0000", 2024-09-21T13:54:11.712 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0014486869999999999 2024-09-21T13:54:11.713 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.713 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.713 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:11.713 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:40.070209+0000", 2024-09-21T13:54:11.713 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.077884298000000005 2024-09-21T13:54:11.713 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.713 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.713 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:54:11.713 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:40.070266+0000", 2024-09-21T13:54:11.713 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7031000000000003e-05 2024-09-21T13:54:11.713 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.713 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.713 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:54:11.713 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:40.070312+0000", 2024-09-21T13:54:11.714 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5553999999999998e-05 2024-09-21T13:54:11.714 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:11.714 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:54:11.714 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:11.714 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.714 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.714 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:72607 7.e 7:7c367c4a:::benchmark_data_smithi049_76798_object72606:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-09-21T13:54:11.714 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:54.457892+0000", 2024-09-21T13:54:11.714 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 197.21298929400001, 2024-09-21T13:54:11.714 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.10128385299999999, 2024-09-21T13:54:11.714 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:54:11.714 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:54:11.714 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:11.714 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:54:11.715 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:54:11.715 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:11.715 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 72607 2024-09-21T13:54:11.715 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.715 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:54:11.715 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.715 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:54:11.715 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.457892+0000", 2024-09-21T13:54:11.715 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:11.715 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.715 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.715 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:54:11.715 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.457892+0000", 2024-09-21T13:54:11.716 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:11.716 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.716 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.716 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:54:11.716 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.457894+0000", 2024-09-21T13:54:11.716 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5090000000000001e-06 2024-09-21T13:54:11.716 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.716 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.716 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:54:11.716 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.457945+0000", 2024-09-21T13:54:11.716 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0983999999999998e-05 2024-09-21T13:54:11.716 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.716 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.716 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:54:11.716 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.457959+0000", 2024-09-21T13:54:11.717 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3987e-05 2024-09-21T13:54:11.717 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.717 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.717 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:54:11.717 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.457968+0000", 2024-09-21T13:54:11.717 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1200000000000008e-06 2024-09-21T13:54:11.717 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.717 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.717 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:11.717 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.458027+0000", 2024-09-21T13:54:11.717 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9599000000000003e-05 2024-09-21T13:54:11.717 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.717 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.717 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:54:11.718 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.458157+0000", 2024-09-21T13:54:11.718 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012947699999999999 2024-09-21T13:54:11.718 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.718 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.718 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0", 2024-09-21T13:54:11.718 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.458292+0000", 2024-09-21T13:54:11.718 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000135642 2024-09-21T13:54:11.718 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.718 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.718 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:54:11.718 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.462010+0000", 2024-09-21T13:54:11.718 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0037178240000000002 2024-09-21T13:54:11.718 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.718 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.719 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:11.719 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.559021+0000", 2024-09-21T13:54:11.719 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.097010420999999999 2024-09-21T13:54:11.719 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.719 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.719 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:54:11.719 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.559081+0000", 2024-09-21T13:54:11.719 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0089e-05 2024-09-21T13:54:11.719 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.719 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.719 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:54:11.719 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.559176+0000", 2024-09-21T13:54:11.719 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.5200999999999999e-05 2024-09-21T13:54:11.720 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:11.720 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:54:11.720 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:11.720 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.720 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.720 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:72635 7.e 7:7083cf4b:::benchmark_data_smithi049_76798_object72634:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T13:54:11.720 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:54.481377+0000", 2024-09-21T13:54:11.720 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 197.189504575, 2024-09-21T13:54:11.720 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.083056779999999997, 2024-09-21T13:54:11.720 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:54:11.720 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:54:11.720 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:11.720 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:54:11.720 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:54:11.720 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:11.721 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 72635 2024-09-21T13:54:11.721 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.721 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:54:11.721 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.721 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:54:11.721 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.481377+0000", 2024-09-21T13:54:11.721 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:11.721 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.721 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.721 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:54:11.721 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.481377+0000", 2024-09-21T13:54:11.721 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:11.721 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.721 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.721 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:54:11.722 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.481378+0000", 2024-09-21T13:54:11.722 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4300000000000001e-06 2024-09-21T13:54:11.722 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.722 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.722 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:54:11.722 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.481491+0000", 2024-09-21T13:54:11.722 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011306000000000001 2024-09-21T13:54:11.722 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.722 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.722 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:54:11.722 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.481507+0000", 2024-09-21T13:54:11.722 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5167000000000001e-05 2024-09-21T13:54:11.722 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.722 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.723 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:54:11.723 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.481514+0000", 2024-09-21T13:54:11.723 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5109999999999997e-06 2024-09-21T13:54:11.723 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.723 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.723 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:11.723 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.481573+0000", 2024-09-21T13:54:11.723 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9197999999999998e-05 2024-09-21T13:54:11.723 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.723 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.723 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:54:11.723 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.481707+0000", 2024-09-21T13:54:11.723 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000133996 2024-09-21T13:54:11.723 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.724 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.724 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0", 2024-09-21T13:54:11.724 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.481857+0000", 2024-09-21T13:54:11.724 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000150223 2024-09-21T13:54:11.724 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.724 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.724 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:54:11.724 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.484065+0000", 2024-09-21T13:54:11.724 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.002207859 2024-09-21T13:54:11.724 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.724 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.724 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:11.724 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.564256+0000", 2024-09-21T13:54:11.724 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.080190206999999999 2024-09-21T13:54:11.725 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.725 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.725 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:54:11.725 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.564332+0000", 2024-09-21T13:54:11.725 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6137999999999995e-05 2024-09-21T13:54:11.725 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.725 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.725 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:54:11.725 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.564434+0000", 2024-09-21T13:54:11.725 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010199100000000001 2024-09-21T13:54:11.725 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:11.725 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:54:11.725 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:11.725 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.726 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.726 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:128074 7.0 7:09ef3a19:::benchmark_data_smithi049_76798_object128073:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-21T13:54:11.726 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:51:36.366911+0000", 2024-09-21T13:54:11.726 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 155.30397077699999, 2024-09-21T13:54:11.726 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.092114021000000004, 2024-09-21T13:54:11.726 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:54:11.726 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:54:11.726 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:11.726 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:54:11.726 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:54:11.726 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:11.726 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 128074 2024-09-21T13:54:11.726 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.727 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:54:11.727 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.727 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:54:11.727 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.366911+0000", 2024-09-21T13:54:11.727 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:11.727 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.727 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.727 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:54:11.727 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.366911+0000", 2024-09-21T13:54:11.727 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:11.727 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.727 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.727 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:54:11.727 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.366912+0000", 2024-09-21T13:54:11.727 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.452e-06 2024-09-21T13:54:11.728 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.728 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.728 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:54:11.728 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.366967+0000", 2024-09-21T13:54:11.728 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5108999999999997e-05 2024-09-21T13:54:11.728 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.728 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.728 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:54:11.728 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.366981+0000", 2024-09-21T13:54:11.728 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4127999999999999e-05 2024-09-21T13:54:11.728 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.728 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.728 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:54:11.728 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.366988+0000", 2024-09-21T13:54:11.728 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3500000000000002e-06 2024-09-21T13:54:11.729 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.729 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.729 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:11.729 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.367057+0000", 2024-09-21T13:54:11.729 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9187e-05 2024-09-21T13:54:11.729 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.729 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.729 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:54:11.729 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.367208+0000", 2024-09-21T13:54:11.729 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015113 2024-09-21T13:54:11.729 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.729 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.729 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3", 2024-09-21T13:54:11.729 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.367364+0000", 2024-09-21T13:54:11.729 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000156439 2024-09-21T13:54:11.730 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.730 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.730 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:11.730 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.375412+0000", 2024-09-21T13:54:11.730 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0080475389999999994 2024-09-21T13:54:11.730 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.730 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.730 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:54:11.730 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.375509+0000", 2024-09-21T13:54:11.730 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6904999999999995e-05 2024-09-21T13:54:11.730 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.730 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.730 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:11.730 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.458910+0000", 2024-09-21T13:54:11.730 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.083401574000000006 2024-09-21T13:54:11.731 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.731 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.731 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:54:11.731 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.458973+0000", 2024-09-21T13:54:11.731 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2145999999999998e-05 2024-09-21T13:54:11.731 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.731 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.731 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:54:11.731 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.459025+0000", 2024-09-21T13:54:11.731 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2061999999999997e-05 2024-09-21T13:54:11.731 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:11.731 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:54:11.731 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:11.731 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.731 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.732 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:128084 7.0 7:09ff6e39:::benchmark_data_smithi049_76798_object128083:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-21T13:54:11.732 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:51:36.368607+0000", 2024-09-21T13:54:11.732 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 155.30227471500001, 2024-09-21T13:54:11.732 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.098058588000000002, 2024-09-21T13:54:11.732 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:54:11.732 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:54:11.732 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:11.732 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:54:11.732 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:54:11.732 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:11.732 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 128084 2024-09-21T13:54:11.732 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.732 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:54:11.732 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.732 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:54:11.733 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.368607+0000", 2024-09-21T13:54:11.733 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:11.733 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.733 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.733 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:54:11.733 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.368607+0000", 2024-09-21T13:54:11.733 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:11.733 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.733 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.733 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:54:11.733 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.368608+0000", 2024-09-21T13:54:11.733 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.71e-07 2024-09-21T13:54:11.733 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.733 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.733 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:54:11.733 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.368640+0000", 2024-09-21T13:54:11.734 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2517000000000001e-05 2024-09-21T13:54:11.734 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.734 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.734 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:54:11.734 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.368651+0000", 2024-09-21T13:54:11.734 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1165000000000001e-05 2024-09-21T13:54:11.734 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.734 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.734 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:54:11.734 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.368659+0000", 2024-09-21T13:54:11.734 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4390000000000003e-06 2024-09-21T13:54:11.734 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.734 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.734 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:11.735 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.371393+0000", 2024-09-21T13:54:11.735 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0027345059999999998 2024-09-21T13:54:11.735 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.735 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.735 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:54:11.735 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.371495+0000", 2024-09-21T13:54:11.735 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000101483 2024-09-21T13:54:11.735 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.735 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.735 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3", 2024-09-21T13:54:11.735 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.371620+0000", 2024-09-21T13:54:11.735 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012551399999999999 2024-09-21T13:54:11.735 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.735 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.735 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:11.736 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.375447+0000", 2024-09-21T13:54:11.736 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0038265040000000001 2024-09-21T13:54:11.736 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.736 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.736 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:54:11.736 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.375514+0000", 2024-09-21T13:54:11.736 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6884000000000005e-05 2024-09-21T13:54:11.736 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.736 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.736 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:11.736 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.466540+0000", 2024-09-21T13:54:11.736 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.091026222000000004 2024-09-21T13:54:11.736 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.736 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.736 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:54:11.737 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.466625+0000", 2024-09-21T13:54:11.737 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5352999999999994e-05 2024-09-21T13:54:11.737 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.737 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.737 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:54:11.737 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:36.466665+0000", 2024-09-21T13:54:11.737 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0129999999999997e-05 2024-09-21T13:54:11.737 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:11.737 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:54:11.737 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:11.737 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.737 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.737 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:130278 7.0 7:0067dc28:::benchmark_data_smithi049_76798_object130277:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-21T13:54:11.737 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:51:38.122067+0000", 2024-09-21T13:54:11.737 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 153.548814916, 2024-09-21T13:54:11.738 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.090663242000000005, 2024-09-21T13:54:11.738 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:54:11.738 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:54:11.738 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:11.738 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:54:11.738 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:54:11.738 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:11.738 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 130278 2024-09-21T13:54:11.738 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.738 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:54:11.738 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.738 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:54:11.738 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.122067+0000", 2024-09-21T13:54:11.738 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:11.738 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.739 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.739 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:54:11.739 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.122067+0000", 2024-09-21T13:54:11.739 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:11.739 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.739 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.739 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:54:11.739 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.122068+0000", 2024-09-21T13:54:11.739 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8330000000000001e-06 2024-09-21T13:54:11.739 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.739 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.739 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:54:11.739 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.122139+0000", 2024-09-21T13:54:11.739 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1093999999999994e-05 2024-09-21T13:54:11.740 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.740 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.740 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:54:11.740 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.122153+0000", 2024-09-21T13:54:11.740 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3951000000000001e-05 2024-09-21T13:54:11.740 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.740 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.740 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:54:11.740 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.122163+0000", 2024-09-21T13:54:11.740 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.9529999999999994e-06 2024-09-21T13:54:11.740 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.740 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.740 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:11.740 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.122229+0000", 2024-09-21T13:54:11.741 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5977999999999997e-05 2024-09-21T13:54:11.741 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.741 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.741 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:54:11.741 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.122370+0000", 2024-09-21T13:54:11.741 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014081099999999999 2024-09-21T13:54:11.741 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.741 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.741 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3", 2024-09-21T13:54:11.741 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.122513+0000", 2024-09-21T13:54:11.741 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000143168 2024-09-21T13:54:11.741 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.741 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.741 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:11.741 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.130151+0000", 2024-09-21T13:54:11.741 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0076378230000000002 2024-09-21T13:54:11.742 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.742 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.742 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:54:11.742 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.130203+0000", 2024-09-21T13:54:11.742 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1817000000000002e-05 2024-09-21T13:54:11.742 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.742 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.742 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:11.742 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.212544+0000", 2024-09-21T13:54:11.742 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.082341356000000004 2024-09-21T13:54:11.742 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.742 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.742 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:54:11.742 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.212676+0000", 2024-09-21T13:54:11.742 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013132399999999999 2024-09-21T13:54:11.742 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.743 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.743 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:54:11.743 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.212730+0000", 2024-09-21T13:54:11.743 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4134e-05 2024-09-21T13:54:11.743 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:11.743 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:54:11.743 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:11.743 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.743 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.743 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:130282 7.0 7:0db29cef:::benchmark_data_smithi049_76798_object130281:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-21T13:54:11.743 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:51:38.122970+0000", 2024-09-21T13:54:11.743 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 153.547911339, 2024-09-21T13:54:11.743 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.089923892000000005, 2024-09-21T13:54:11.743 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:54:11.743 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:54:11.743 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:11.744 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:54:11.744 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:54:11.744 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:11.744 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 130282 2024-09-21T13:54:11.744 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.744 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:54:11.744 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.744 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:54:11.744 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.122970+0000", 2024-09-21T13:54:11.744 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:11.744 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.744 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.744 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:54:11.744 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.122970+0000", 2024-09-21T13:54:11.744 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:11.745 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.745 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.745 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:54:11.745 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.122985+0000", 2024-09-21T13:54:11.745 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4579999999999999e-05 2024-09-21T13:54:11.745 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.745 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.745 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:54:11.745 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.123038+0000", 2024-09-21T13:54:11.745 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3792000000000003e-05 2024-09-21T13:54:11.745 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.745 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.745 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:54:11.745 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.123050+0000", 2024-09-21T13:54:11.745 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1711e-05 2024-09-21T13:54:11.746 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.746 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.746 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:54:11.746 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.123059+0000", 2024-09-21T13:54:11.746 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4149999999999999e-06 2024-09-21T13:54:11.746 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.746 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.746 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:11.746 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.126699+0000", 2024-09-21T13:54:11.746 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003640285 2024-09-21T13:54:11.746 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.746 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.746 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:54:11.746 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.126807+0000", 2024-09-21T13:54:11.746 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000108418 2024-09-21T13:54:11.746 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.747 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.747 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3", 2024-09-21T13:54:11.747 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.127095+0000", 2024-09-21T13:54:11.747 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00028747100000000003 2024-09-21T13:54:11.747 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.747 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.747 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:11.747 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.130192+0000", 2024-09-21T13:54:11.747 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0030972450000000002 2024-09-21T13:54:11.747 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.747 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.747 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:54:11.747 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.130209+0000", 2024-09-21T13:54:11.747 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7238e-05 2024-09-21T13:54:11.747 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.747 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.748 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:11.748 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.212787+0000", 2024-09-21T13:54:11.748 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.082577958000000007 2024-09-21T13:54:11.748 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.748 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.748 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:54:11.748 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.212853+0000", 2024-09-21T13:54:11.748 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6209000000000003e-05 2024-09-21T13:54:11.748 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.748 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.748 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:54:11.748 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.212894+0000", 2024-09-21T13:54:11.748 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.057e-05 2024-09-21T13:54:11.748 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:11.749 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:54:11.749 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:11.749 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.749 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.749 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:131159 7.0 7:02ff7d1c:::benchmark_data_smithi049_76798_object131158:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-21T13:54:11.749 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:51:38.920953+0000", 2024-09-21T13:54:11.749 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 152.749928692, 2024-09-21T13:54:11.749 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.080217627999999999, 2024-09-21T13:54:11.749 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:54:11.749 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:54:11.749 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:11.749 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:54:11.749 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:54:11.749 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:11.750 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 131159 2024-09-21T13:54:11.750 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.750 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:54:11.750 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.750 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:54:11.750 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.920953+0000", 2024-09-21T13:54:11.750 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:11.750 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.750 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.750 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:54:11.750 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.920953+0000", 2024-09-21T13:54:11.750 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:11.750 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.751 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.751 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:54:11.751 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.920957+0000", 2024-09-21T13:54:11.751 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2069999999999997e-06 2024-09-21T13:54:11.751 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.751 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.751 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:54:11.751 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.921010+0000", 2024-09-21T13:54:11.751 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3084000000000002e-05 2024-09-21T13:54:11.751 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.751 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.751 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:54:11.751 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.921026+0000", 2024-09-21T13:54:11.751 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5707999999999999e-05 2024-09-21T13:54:11.751 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.752 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.752 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:54:11.752 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.921033+0000", 2024-09-21T13:54:11.752 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2370000000000003e-06 2024-09-21T13:54:11.752 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.752 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.752 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:11.752 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.921086+0000", 2024-09-21T13:54:11.752 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2911000000000002e-05 2024-09-21T13:54:11.752 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.752 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.752 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:54:11.752 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.921218+0000", 2024-09-21T13:54:11.752 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013251100000000001 2024-09-21T13:54:11.752 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.753 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.753 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3", 2024-09-21T13:54:11.753 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.921367+0000", 2024-09-21T13:54:11.753 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014847300000000001 2024-09-21T13:54:11.753 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.753 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.753 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:11.753 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.924827+0000", 2024-09-21T13:54:11.753 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0034603049999999999 2024-09-21T13:54:11.753 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.753 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.753 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:54:11.753 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:38.924842+0000", 2024-09-21T13:54:11.753 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5226e-05 2024-09-21T13:54:11.753 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.753 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.754 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:11.754 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:39.001082+0000", 2024-09-21T13:54:11.754 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.076239336000000005 2024-09-21T13:54:11.754 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.754 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.754 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:54:11.754 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:39.001136+0000", 2024-09-21T13:54:11.754 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3764999999999998e-05 2024-09-21T13:54:11.754 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.754 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.754 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:54:11.754 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:39.001170+0000", 2024-09-21T13:54:11.754 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4864999999999999e-05 2024-09-21T13:54:11.754 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:11.754 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:54:11.755 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:11.755 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.755 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.755 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:151180 7.0 7:032b5732:::benchmark_data_smithi049_76798_object14760:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-21T13:54:11.755 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:51:53.937762+0000", 2024-09-21T13:54:11.755 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 137.733119655, 2024-09-21T13:54:11.755 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21988037999999999, 2024-09-21T13:54:11.755 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:54:11.755 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:54:11.755 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:11.755 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:54:11.755 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:54:11.755 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:11.755 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 151180 2024-09-21T13:54:11.755 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.756 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:54:11.756 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.756 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:54:11.756 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:53.937762+0000", 2024-09-21T13:54:11.756 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:11.756 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.756 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.756 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:54:11.756 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:53.937762+0000", 2024-09-21T13:54:11.756 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:11.756 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.756 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.756 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:54:11.756 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:53.937763+0000", 2024-09-21T13:54:11.756 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6139999999999999e-06 2024-09-21T13:54:11.756 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.757 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.757 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:54:11.757 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:53.937769+0000", 2024-09-21T13:54:11.757 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.558e-06 2024-09-21T13:54:11.757 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.757 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.757 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:54:11.757 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:53.937781+0000", 2024-09-21T13:54:11.757 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.238e-05 2024-09-21T13:54:11.757 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.757 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.757 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:54:11.757 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:53.937789+0000", 2024-09-21T13:54:11.757 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5650000000000004e-06 2024-09-21T13:54:11.757 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.757 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.758 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:11.758 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:53.937849+0000", 2024-09-21T13:54:11.758 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9722000000000002e-05 2024-09-21T13:54:11.758 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.758 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.758 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:54:11.758 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:53.937950+0000", 2024-09-21T13:54:11.758 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000101439 2024-09-21T13:54:11.758 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.758 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.758 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3", 2024-09-21T13:54:11.758 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:53.938051+0000", 2024-09-21T13:54:11.758 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000101199 2024-09-21T13:54:11.758 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.759 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.759 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:54:11.759 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:53.940747+0000", 2024-09-21T13:54:11.759 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0026956390000000001 2024-09-21T13:54:11.759 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.759 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.759 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:11.759 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:53.951037+0000", 2024-09-21T13:54:11.759 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.01029005 2024-09-21T13:54:11.759 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.759 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.759 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:11.759 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:54.157534+0000", 2024-09-21T13:54:11.759 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20649731299999999 2024-09-21T13:54:11.759 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.759 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.760 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:54:11.760 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:54.157598+0000", 2024-09-21T13:54:11.760 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3962999999999997e-05 2024-09-21T13:54:11.760 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.760 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.760 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:54:11.760 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:54.157642+0000", 2024-09-21T13:54:11.760 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3937999999999998e-05 2024-09-21T13:54:11.760 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:11.760 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:54:11.760 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:11.760 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.760 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.760 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:151183 7.0 7:0c6c7120:::benchmark_data_smithi049_76798_object14763:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-21T13:54:11.761 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:51:53.944491+0000", 2024-09-21T13:54:11.761 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 137.72639090300001, 2024-09-21T13:54:11.761 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.213263598, 2024-09-21T13:54:11.761 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:54:11.761 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:54:11.761 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:11.761 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:54:11.761 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:54:11.761 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:11.761 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 151183 2024-09-21T13:54:11.761 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.761 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:54:11.761 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.761 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:54:11.761 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:53.944491+0000", 2024-09-21T13:54:11.761 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:11.762 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.762 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.762 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:54:11.762 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:53.944491+0000", 2024-09-21T13:54:11.762 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:11.762 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.762 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.762 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:54:11.762 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:53.944492+0000", 2024-09-21T13:54:11.762 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1009999999999999e-06 2024-09-21T13:54:11.762 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.762 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.762 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:54:11.762 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:53.944496+0000", 2024-09-21T13:54:11.762 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5709999999999999e-06 2024-09-21T13:54:11.763 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.763 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.763 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:54:11.763 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:53.944505+0000", 2024-09-21T13:54:11.763 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1519999999999998e-06 2024-09-21T13:54:11.763 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.763 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.763 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:54:11.763 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:53.944510+0000", 2024-09-21T13:54:11.763 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1080000000000001e-06 2024-09-21T13:54:11.763 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.763 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.763 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:11.763 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:53.944558+0000", 2024-09-21T13:54:11.764 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7432000000000003e-05 2024-09-21T13:54:11.764 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.764 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.764 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:54:11.764 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:53.944666+0000", 2024-09-21T13:54:11.764 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010791800000000001 2024-09-21T13:54:11.764 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.764 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.764 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3", 2024-09-21T13:54:11.764 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:53.945050+0000", 2024-09-21T13:54:11.764 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00038448300000000003 2024-09-21T13:54:11.764 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.764 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.764 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:54:11.765 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:53.948660+0000", 2024-09-21T13:54:11.765 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003609733 2024-09-21T13:54:11.765 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.765 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.765 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:11.765 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:53.953758+0000", 2024-09-21T13:54:11.765 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0050984250000000002 2024-09-21T13:54:11.765 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.765 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.765 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:11.765 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:54.157697+0000", 2024-09-21T13:54:11.765 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20393897999999999 2024-09-21T13:54:11.765 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.765 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.765 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:54:11.766 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:54.157724+0000", 2024-09-21T13:54:11.766 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.6444e-05 2024-09-21T13:54:11.766 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.766 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.766 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:54:11.766 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:54.157754+0000", 2024-09-21T13:54:11.766 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0250999999999999e-05 2024-09-21T13:54:11.766 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:11.766 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:54:11.766 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:11.766 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.766 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.766 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:151196 7.e 7:7fc82d2e:::benchmark_data_smithi049_76798_object14776:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-21T13:54:11.766 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:51:53.955337+0000", 2024-09-21T13:54:11.767 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 137.71554476099999, 2024-09-21T13:54:11.767 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20238076699999999, 2024-09-21T13:54:11.767 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:54:11.767 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:54:11.767 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:11.767 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:54:11.767 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:54:11.767 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:11.767 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 151196 2024-09-21T13:54:11.767 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.767 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:54:11.767 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.767 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:54:11.767 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:53.955337+0000", 2024-09-21T13:54:11.767 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:11.768 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.768 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.768 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:54:11.768 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:53.955337+0000", 2024-09-21T13:54:11.768 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:11.768 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.768 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.768 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:54:11.768 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:53.955338+0000", 2024-09-21T13:54:11.768 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6390000000000001e-06 2024-09-21T13:54:11.768 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.768 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.768 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:54:11.768 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:53.955343+0000", 2024-09-21T13:54:11.769 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1189999999999996e-06 2024-09-21T13:54:11.769 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.769 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.769 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:54:11.769 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:53.955355+0000", 2024-09-21T13:54:11.769 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1984000000000001e-05 2024-09-21T13:54:11.769 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.769 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.769 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:54:11.769 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:53.955363+0000", 2024-09-21T13:54:11.769 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1679999999999998e-06 2024-09-21T13:54:11.769 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.769 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.769 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:11.769 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:53.955408+0000", 2024-09-21T13:54:11.770 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5552e-05 2024-09-21T13:54:11.770 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.770 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.770 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:54:11.770 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:53.955494+0000", 2024-09-21T13:54:11.770 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5859000000000006e-05 2024-09-21T13:54:11.770 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.770 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.770 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0", 2024-09-21T13:54:11.770 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:53.955559+0000", 2024-09-21T13:54:11.770 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4763000000000003e-05 2024-09-21T13:54:11.770 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.770 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.770 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:54:11.771 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:54.154818+0000", 2024-09-21T13:54:11.771 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.19925954900000001 2024-09-21T13:54:11.771 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.771 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.771 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:11.771 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:54.157539+0000", 2024-09-21T13:54:11.771 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0027207870000000001 2024-09-21T13:54:11.771 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.771 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.771 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:54:11.771 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:54.157604+0000", 2024-09-21T13:54:11.771 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4406999999999995e-05 2024-09-21T13:54:11.771 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.772 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.772 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:54:11.772 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:54.157717+0000", 2024-09-21T13:54:11.772 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011394 2024-09-21T13:54:11.772 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:11.772 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:54:11.772 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:11.772 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.772 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.772 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:151200 7.e 7:721e0085:::benchmark_data_smithi049_76798_object14780:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-21T13:54:11.772 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:51:53.956084+0000", 2024-09-21T13:54:11.772 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 137.71479722300001, 2024-09-21T13:54:11.772 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.206651112, 2024-09-21T13:54:11.772 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:54:11.772 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:54:11.772 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:11.772 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:54:11.773 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:54:11.773 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:11.773 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:11.774 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:11 smithi049 ceph-mon[40489]: pgmap v252: 132 pgs: 132 active+clean; 583 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 752 B/s rd, 0 op/s 2024-09-21T13:54:11.774 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:11 smithi049 ceph-mon[40489]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-21T13:54:11.774 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:11 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3546747537' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-21T13:54:11.774 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:11 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-21T13:54:11.774 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:11 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3649122924' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:54:11.774 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:11 smithi049 ceph-mon[30080]: pgmap v252: 132 pgs: 132 active+clean; 583 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 752 B/s rd, 0 op/s 2024-09-21T13:54:11.774 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:11 smithi049 ceph-mon[30080]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-21T13:54:11.775 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:11 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3546747537' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-21T13:54:11.775 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:11 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-21T13:54:11.775 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:11 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3649122924' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:54:11.775 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 151200 2024-09-21T13:54:11.775 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.775 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:54:11.775 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.775 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:54:11.775 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:53.956084+0000", 2024-09-21T13:54:11.775 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:11.775 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.776 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.776 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:54:11.776 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:53.956084+0000", 2024-09-21T13:54:11.776 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:11.776 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.776 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.776 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:54:11.776 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:53.956085+0000", 2024-09-21T13:54:11.776 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.64e-07 2024-09-21T13:54:11.776 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.776 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.776 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:54:11.776 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:53.956092+0000", 2024-09-21T13:54:11.776 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5749999999999997e-06 2024-09-21T13:54:11.776 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.777 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.777 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:54:11.777 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:53.956102+0000", 2024-09-21T13:54:11.777 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0651e-05 2024-09-21T13:54:11.777 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.777 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.777 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:54:11.777 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:53.956109+0000", 2024-09-21T13:54:11.777 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9469999999999997e-06 2024-09-21T13:54:11.777 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.777 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.777 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:11.777 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:54.154999+0000", 2024-09-21T13:54:11.777 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.19888961799999999 2024-09-21T13:54:11.777 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.778 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.778 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:54:11.778 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:54.155116+0000", 2024-09-21T13:54:11.778 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000117303 2024-09-21T13:54:11.778 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.778 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.778 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0", 2024-09-21T13:54:11.778 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:54.155237+0000", 2024-09-21T13:54:11.778 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000120411 2024-09-21T13:54:11.778 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.778 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.778 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:54:11.778 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:54.162589+0000", 2024-09-21T13:54:11.778 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0073523219999999997 2024-09-21T13:54:11.779 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.779 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.779 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:11.779 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:54.162627+0000", 2024-09-21T13:54:11.779 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7499999999999997e-05 2024-09-21T13:54:11.779 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.779 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.779 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:54:11.779 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:54.162659+0000", 2024-09-21T13:54:11.779 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2008999999999998e-05 2024-09-21T13:54:11.779 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.779 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.779 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:54:11.779 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:54.162735+0000", 2024-09-21T13:54:11.780 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6811999999999996e-05 2024-09-21T13:54:11.780 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:11.780 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:54:11.780 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:11.780 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.780 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.780 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.14817.0:151198 7.b e125/83 7:d833e411:::benchmark_data_smithi049_76798_object14778:head v 125'9529, mlcod=125'9527)", 2024-09-21T13:54:11.780 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:51:53.956165+0000", 2024-09-21T13:54:11.780 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 137.71471687499999, 2024-09-21T13:54:11.780 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.204920361, 2024-09-21T13:54:11.780 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:54:11.780 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:54:11.780 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:11.780 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:54:11.781 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.781 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:54:11.781 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:53.956165+0000", 2024-09-21T13:54:11.781 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:11.781 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.781 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.781 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:54:11.781 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:53.956165+0000", 2024-09-21T13:54:11.781 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:11.781 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.781 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.781 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:54:11.781 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:53.956165+0000", 2024-09-21T13:54:11.781 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2900000000000003e-07 2024-09-21T13:54:11.781 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.782 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.782 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:54:11.782 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:53.956170+0000", 2024-09-21T13:54:11.782 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7260000000000002e-06 2024-09-21T13:54:11.782 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.782 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.782 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:54:11.782 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:53.956178+0000", 2024-09-21T13:54:11.782 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0860000000000002e-06 2024-09-21T13:54:11.782 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.782 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.782 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:54:11.782 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:53.956184+0000", 2024-09-21T13:54:11.783 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.48e-06 2024-09-21T13:54:11.783 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.783 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.783 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:11.783 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:54.154800+0000", 2024-09-21T13:54:11.783 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.19861606900000001 2024-09-21T13:54:11.783 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.783 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.783 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:54:11.783 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:54.154822+0000", 2024-09-21T13:54:11.783 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2518999999999999e-05 2024-09-21T13:54:11.783 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.783 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.783 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:54:11.783 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:54.161047+0000", 2024-09-21T13:54:11.784 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0062249499999999999 2024-09-21T13:54:11.784 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.784 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.784 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:54:11.784 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:54.161085+0000", 2024-09-21T13:54:11.784 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7802000000000002e-05 2024-09-21T13:54:11.784 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:11.784 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:54:11.784 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:11.784 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.784 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.784 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:151204 7.0 7:05bf3c7b:::benchmark_data_smithi049_76798_object14784:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-21T13:54:11.784 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:51:53.956874+0000", 2024-09-21T13:54:11.784 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 137.714007751, 2024-09-21T13:54:11.785 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20645238399999999, 2024-09-21T13:54:11.785 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:54:11.785 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:54:11.785 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:11.785 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:54:11.785 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:54:11.785 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:11.785 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 151204 2024-09-21T13:54:11.785 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.785 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:54:11.785 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.785 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:54:11.785 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:53.956874+0000", 2024-09-21T13:54:11.785 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:11.785 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.786 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.786 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:54:11.786 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:53.956874+0000", 2024-09-21T13:54:11.786 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:11.786 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.786 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.786 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:54:11.786 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:53.956875+0000", 2024-09-21T13:54:11.786 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.64e-07 2024-09-21T13:54:11.786 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.786 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.786 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:54:11.786 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:53.956879+0000", 2024-09-21T13:54:11.786 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6650000000000002e-06 2024-09-21T13:54:11.786 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.787 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.787 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:54:11.787 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:54.154784+0000", 2024-09-21T13:54:11.787 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.19790424700000001 2024-09-21T13:54:11.787 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.787 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.787 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:54:11.787 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:54.154797+0000", 2024-09-21T13:54:11.787 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3072e-05 2024-09-21T13:54:11.787 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.787 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.787 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:11.787 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:54.155084+0000", 2024-09-21T13:54:11.787 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00028739199999999998 2024-09-21T13:54:11.788 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.788 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.788 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:54:11.788 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:54.155195+0000", 2024-09-21T13:54:11.788 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011092899999999999 2024-09-21T13:54:11.788 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.788 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.788 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3", 2024-09-21T13:54:11.788 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:54.155301+0000", 2024-09-21T13:54:11.788 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010595 2024-09-21T13:54:11.788 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.788 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.788 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:54:11.788 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:54.163157+0000", 2024-09-21T13:54:11.789 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0078565749999999993 2024-09-21T13:54:11.789 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.789 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.789 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:11.789 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:54.163220+0000", 2024-09-21T13:54:11.789 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2465000000000006e-05 2024-09-21T13:54:11.789 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.789 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.789 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:11.789 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:54.163264+0000", 2024-09-21T13:54:11.789 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3591999999999999e-05 2024-09-21T13:54:11.789 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.789 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.789 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:54:11.790 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:54.163292+0000", 2024-09-21T13:54:11.790 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8418000000000001e-05 2024-09-21T13:54:11.790 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.790 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.790 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:54:11.790 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:54.163326+0000", 2024-09-21T13:54:11.790 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4115000000000001e-05 2024-09-21T13:54:11.790 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:11.790 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:54:11.790 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:11.790 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.790 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.790 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.14817.0:151202 7.a e125/101 7:5b4e3deb:::benchmark_data_smithi049_76798_object14782:head v 125'9425, mlcod=125'9422)", 2024-09-21T13:54:11.790 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:51:53.957145+0000", 2024-09-21T13:54:11.791 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 137.713736657, 2024-09-21T13:54:11.791 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.202219179, 2024-09-21T13:54:11.791 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:54:11.791 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:54:11.791 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:11.791 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:54:11.791 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.791 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:54:11.791 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:53.957145+0000", 2024-09-21T13:54:11.791 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:11.791 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.791 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.791 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:54:11.791 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:53.957145+0000", 2024-09-21T13:54:11.792 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:11.792 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.792 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.792 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:54:11.792 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:53.957146+0000", 2024-09-21T13:54:11.792 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4099999999999998e-07 2024-09-21T13:54:11.792 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.792 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.792 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:54:11.792 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:53.957150+0000", 2024-09-21T13:54:11.792 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6619999999999996e-06 2024-09-21T13:54:11.792 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.792 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.792 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:54:11.793 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:54.154763+0000", 2024-09-21T13:54:11.793 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.19761256799999999 2024-09-21T13:54:11.793 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.793 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.793 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:54:11.793 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:54.154774+0000", 2024-09-21T13:54:11.793 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1532000000000001e-05 2024-09-21T13:54:11.793 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.793 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.793 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:11.793 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:54.154966+0000", 2024-09-21T13:54:11.793 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00019161499999999999 2024-09-21T13:54:11.793 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.793 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.794 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:54:11.794 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:54.154986+0000", 2024-09-21T13:54:11.794 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0102000000000002e-05 2024-09-21T13:54:11.794 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.794 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.794 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:54:11.794 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:54.159328+0000", 2024-09-21T13:54:11.794 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.004342383 2024-09-21T13:54:11.794 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.794 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.794 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:54:11.794 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:51:54.159364+0000", 2024-09-21T13:54:11.794 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5576e-05 2024-09-21T13:54:11.794 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:11.795 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:54:11.795 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:11.795 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.795 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.795 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:166052 7.0 7:037403d1:::benchmark_data_smithi049_76798_object29632:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-09-21T13:54:11.795 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:07.431034+0000", 2024-09-21T13:54:11.795 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 124.23984739300001, 2024-09-21T13:54:11.795 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.093141650000000006, 2024-09-21T13:54:11.795 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:54:11.795 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:54:11.795 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:11.795 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:54:11.795 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:54:11.795 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:11.796 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 166052 2024-09-21T13:54:11.796 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.796 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:54:11.796 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.796 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:54:11.796 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:07.431034+0000", 2024-09-21T13:54:11.796 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:11.796 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.796 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.796 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:54:11.796 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:07.431034+0000", 2024-09-21T13:54:11.796 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:11.796 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.797 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.797 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:54:11.797 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:07.431035+0000", 2024-09-21T13:54:11.797 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.238e-06 2024-09-21T13:54:11.797 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.797 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.797 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:54:11.797 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:07.431039+0000", 2024-09-21T13:54:11.797 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9550000000000002e-06 2024-09-21T13:54:11.797 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.797 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.797 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:54:11.797 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:07.431049+0000", 2024-09-21T13:54:11.797 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6700000000000006e-06 2024-09-21T13:54:11.797 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.798 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.798 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:54:11.798 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:07.431054+0000", 2024-09-21T13:54:11.798 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1580000000000004e-06 2024-09-21T13:54:11.798 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.798 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.798 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:11.798 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:07.431101+0000", 2024-09-21T13:54:11.798 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6526000000000001e-05 2024-09-21T13:54:11.798 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.798 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.798 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:54:11.798 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:07.431209+0000", 2024-09-21T13:54:11.798 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000108742 2024-09-21T13:54:11.799 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.799 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.799 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3", 2024-09-21T13:54:11.799 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:07.431316+0000", 2024-09-21T13:54:11.799 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010619 2024-09-21T13:54:11.799 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.799 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.799 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:54:11.799 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:07.434142+0000", 2024-09-21T13:54:11.799 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.002826088 2024-09-21T13:54:11.799 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.799 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.799 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:11.799 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:07.437807+0000", 2024-09-21T13:54:11.800 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003665775 2024-09-21T13:54:11.800 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.800 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.800 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:11.800 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:07.523974+0000", 2024-09-21T13:54:11.800 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.086166968999999996 2024-09-21T13:54:11.800 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.800 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.800 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:54:11.800 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:07.524123+0000", 2024-09-21T13:54:11.800 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014856900000000001 2024-09-21T13:54:11.800 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.800 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.800 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:54:11.801 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:07.524176+0000", 2024-09-21T13:54:11.801 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2769999999999998e-05 2024-09-21T13:54:11.801 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:11.801 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:54:11.801 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:11.801 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.801 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.801 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:166055 7.0 7:08223a14:::benchmark_data_smithi049_76798_object29635:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-09-21T13:54:11.801 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:07.434929+0000", 2024-09-21T13:54:11.801 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 124.235952846, 2024-09-21T13:54:11.801 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.089373284999999997, 2024-09-21T13:54:11.801 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:54:11.801 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:54:11.801 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:11.802 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:54:11.802 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:54:11.802 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:11.802 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 166055 2024-09-21T13:54:11.802 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.802 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:54:11.802 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.802 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:54:11.802 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:07.434929+0000", 2024-09-21T13:54:11.802 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:11.802 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.802 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.802 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:54:11.803 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:07.434929+0000", 2024-09-21T13:54:11.803 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:11.803 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.803 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.803 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:54:11.803 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:07.434931+0000", 2024-09-21T13:54:11.803 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.057e-06 2024-09-21T13:54:11.803 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.803 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.803 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:54:11.803 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:07.434935+0000", 2024-09-21T13:54:11.803 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6460000000000001e-06 2024-09-21T13:54:11.803 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.803 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.804 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:54:11.804 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:07.434948+0000", 2024-09-21T13:54:11.804 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2561000000000001e-05 2024-09-21T13:54:11.804 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.804 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.804 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:54:11.804 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:07.434955+0000", 2024-09-21T13:54:11.804 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.029e-06 2024-09-21T13:54:11.804 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.804 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.804 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:11.804 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:07.435005+0000", 2024-09-21T13:54:11.804 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0463000000000001e-05 2024-09-21T13:54:11.804 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.805 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.805 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:54:11.805 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:07.435097+0000", 2024-09-21T13:54:11.805 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.2075000000000002e-05 2024-09-21T13:54:11.805 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.805 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.805 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3", 2024-09-21T13:54:11.805 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:07.435185+0000", 2024-09-21T13:54:11.805 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7280000000000005e-05 2024-09-21T13:54:11.805 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.805 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.805 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:11.805 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:07.438298+0000", 2024-09-21T13:54:11.805 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0031137449999999998 2024-09-21T13:54:11.806 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.806 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.806 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:54:11.806 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:07.438799+0000", 2024-09-21T13:54:11.806 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00050047700000000004 2024-09-21T13:54:11.806 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.806 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.806 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:11.806 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:07.524221+0000", 2024-09-21T13:54:11.806 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.085421625000000001 2024-09-21T13:54:11.806 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.806 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.806 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:54:11.806 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:07.524266+0000", 2024-09-21T13:54:11.807 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5417999999999998e-05 2024-09-21T13:54:11.807 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.807 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.807 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:54:11.807 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:07.524302+0000", 2024-09-21T13:54:11.807 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5908999999999998e-05 2024-09-21T13:54:11.807 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:11.807 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:54:11.807 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:11.807 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.807 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.807 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:249863 7.0 7:096084cf:::benchmark_data_smithi049_76798_object113443:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e168)", 2024-09-21T13:54:11.807 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:53:36.300385+0000", 2024-09-21T13:54:11.807 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 35.370496510000002, 2024-09-21T13:54:11.808 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.108484917, 2024-09-21T13:54:11.808 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:54:11.808 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:54:11.808 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:11.808 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:54:11.808 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:54:11.808 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:11.808 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 249863 2024-09-21T13:54:11.808 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.808 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:54:11.808 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.808 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:54:11.808 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:53:36.300385+0000", 2024-09-21T13:54:11.808 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:11.809 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.809 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.809 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:54:11.809 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:53:36.300385+0000", 2024-09-21T13:54:11.809 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:11.809 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.809 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.809 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:54:11.809 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:53:36.300386+0000", 2024-09-21T13:54:11.809 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.533e-06 2024-09-21T13:54:11.809 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.809 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.809 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:54:11.809 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:53:36.300392+0000", 2024-09-21T13:54:11.809 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4589999999999997e-06 2024-09-21T13:54:11.810 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.810 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.810 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:54:11.810 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:53:36.300406+0000", 2024-09-21T13:54:11.810 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3730999999999999e-05 2024-09-21T13:54:11.810 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.810 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.810 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:54:11.810 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:53:36.300414+0000", 2024-09-21T13:54:11.810 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1710000000000002e-06 2024-09-21T13:54:11.810 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.810 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.810 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:11.810 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:53:36.300477+0000", 2024-09-21T13:54:11.811 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3534999999999994e-05 2024-09-21T13:54:11.811 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.811 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.811 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:54:11.811 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:53:36.300588+0000", 2024-09-21T13:54:11.811 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000110567 2024-09-21T13:54:11.811 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.811 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.811 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3", 2024-09-21T13:54:11.811 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:53:36.300652+0000", 2024-09-21T13:54:11.811 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4033000000000002e-05 2024-09-21T13:54:11.811 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.811 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.811 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:54:11.811 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:53:36.303355+0000", 2024-09-21T13:54:11.812 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0027026160000000001 2024-09-21T13:54:11.812 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.812 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.812 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:11.812 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:53:36.304090+0000", 2024-09-21T13:54:11.812 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00073510000000000003 2024-09-21T13:54:11.812 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.812 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.812 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:11.812 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:53:36.408758+0000", 2024-09-21T13:54:11.812 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.104668784 2024-09-21T13:54:11.812 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.812 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.812 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:54:11.813 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:53:36.408820+0000", 2024-09-21T13:54:11.813 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1710000000000004e-05 2024-09-21T13:54:11.813 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:11.813 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:11.813 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:54:11.813 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:53:36.408870+0000", 2024-09-21T13:54:11.813 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9678000000000001e-05 2024-09-21T13:54:11.813 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:11.813 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:54:11.813 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:11.813 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:11.813 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:54:11.813 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T13:54:11.834 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:11.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:11.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:11.936 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:54:11.957 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:11.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:11.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:54:12.059 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:54:12.080 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:12.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:12.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:54:12.182 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:54:12.203 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:12.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:12.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:54:12.267 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T13:54:12.304 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:12.327 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:12.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:12.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:12.428 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:54:12.452 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:12.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:12.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:54:12.549 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:12 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-21T13:54:12.550 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:12 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3649122924' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T13:54:12.550 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:12 smithi049 ceph-mon[40489]: osdmap e183: 8 total, 8 up, 8 in 2024-09-21T13:54:12.550 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:12 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-21T13:54:12.550 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:12 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3649122924' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T13:54:12.550 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:12 smithi049 ceph-mon[30080]: osdmap e183: 8 total, 8 up, 8 in 2024-09-21T13:54:12.554 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:12.577 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:12.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:12.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:12.582 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T13:54:12.644 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:12 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-21T13:54:12.644 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:12 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3649122924' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T13:54:12.644 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:12 smithi187 ceph-mon[33032]: osdmap e183: 8 total, 8 up, 8 in 2024-09-21T13:54:12.679 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:12.700 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:12.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:12.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:12.801 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:12.825 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:12.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:12.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:12.872 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:54:12.912 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-09-21T13:54:12.926 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:54:12.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:12.949 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:12.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:54:12.984 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:54:13.050 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:13.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:13.071 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:13.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:13.172 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:13.193 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:13.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:13.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:13.295 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:54:13.316 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:13.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:13.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:54:13.391 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T13:54:13.391 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-21T13:54:13.391 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 0 2024-09-21T13:54:13.391 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T13:54:13.418 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:13.439 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:13.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:13.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:13.454 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:13 smithi049 ceph-mon[30080]: pgmap v254: 148 pgs: 16 unknown, 132 active+clean; 583 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-21T13:54:13.454 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:13 smithi049 ceph-mon[30080]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T13:54:13.454 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:13 smithi049 ceph-mon[30080]: osdmap e184: 8 total, 8 up, 8 in 2024-09-21T13:54:13.454 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:13 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:54:13.455 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:13 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:54:13.455 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:13 smithi049 ceph-mon[40489]: pgmap v254: 148 pgs: 16 unknown, 132 active+clean; 583 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-21T13:54:13.455 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:13 smithi049 ceph-mon[40489]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T13:54:13.455 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:13 smithi049 ceph-mon[40489]: osdmap e184: 8 total, 8 up, 8 in 2024-09-21T13:54:13.455 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:13 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:54:13.455 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:13 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:54:13.540 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:13.561 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:13.562 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:54:13.562 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:54:13.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:13.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:13.573 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:54:13.573 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:54:13.589 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:54:13.589 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:54:13.605 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:54:13.606 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:54:13.623 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:54:13.623 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:54:13.640 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:54:13.640 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:54:13.657 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:54:13.657 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:54:13.663 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:13.674 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:54:13.674 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:54:13.686 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:13.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:13.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:13.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:13 smithi187 ceph-mon[33032]: pgmap v254: 148 pgs: 16 unknown, 132 active+clean; 583 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-21T13:54:13.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:13 smithi187 ceph-mon[33032]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T13:54:13.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:13 smithi187 ceph-mon[33032]: osdmap e184: 8 total, 8 up, 8 in 2024-09-21T13:54:13.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:13 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:54:13.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:13 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:54:13.787 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:54:13.809 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:13.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:13.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:54:13.869 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:54:13.911 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:13.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:13.935 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:13.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:14.036 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:14.055 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T13:54:14.059 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:14.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:14.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:14.161 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:54:14.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:14.184 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:14.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:54:14.285 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:14.302 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:54:14.308 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:14.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:14.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:14.410 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:54:14.432 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:14.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:14.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:54:14.446 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T13:54:14.533 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:54:14.558 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:14.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:14.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:54:14.579 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:14 smithi049 ceph-mon[40489]: osdmap e185: 8 total, 8 up, 8 in 2024-09-21T13:54:14.579 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:14 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1791929504' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:54:14.579 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:14 smithi049 ceph-mon[30080]: osdmap e185: 8 total, 8 up, 8 in 2024-09-21T13:54:14.579 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:14 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1791929504' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:54:14.647 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:14 smithi187 ceph-mon[33032]: osdmap e185: 8 total, 8 up, 8 in 2024-09-21T13:54:14.648 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:14 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1791929504' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:54:14.660 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:14.682 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:14.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:14.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:14.772 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T13:54:14.772 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:54:14.784 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:54:14.805 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:14.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:14.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:54:14.907 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:14.934 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:14.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:14.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:14.968 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:54:15.036 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:15.058 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:15.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:15.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:15.160 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:15.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:15.182 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:15.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:15.283 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:54:15.306 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:15.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:15.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:54:15.378 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T13:54:15.407 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:54:15.424 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:15 smithi049 ceph-mon[40489]: pgmap v257: 148 pgs: 3 unknown, 145 active+clean; 583 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T13:54:15.424 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:15 smithi049 ceph-mon[40489]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:54:15.424 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:15 smithi049 ceph-mon[40489]: osdmap e186: 8 total, 8 up, 8 in 2024-09-21T13:54:15.424 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:15 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3356408603' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T13:54:15.424 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:15 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T13:54:15.424 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:15 smithi049 ceph-mon[30080]: pgmap v257: 148 pgs: 3 unknown, 145 active+clean; 583 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T13:54:15.424 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:15 smithi049 ceph-mon[30080]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:54:15.424 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:15 smithi049 ceph-mon[30080]: osdmap e186: 8 total, 8 up, 8 in 2024-09-21T13:54:15.425 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:15 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3356408603' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T13:54:15.425 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:15 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T13:54:15.430 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:15.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:15.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:54:15.531 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:54:15.553 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:15.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:15.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:54:15.623 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:54:15.655 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:54:15.677 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:15.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:15.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:54:15.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:15 smithi187 ceph-mon[33032]: pgmap v257: 148 pgs: 3 unknown, 145 active+clean; 583 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T13:54:15.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:15 smithi187 ceph-mon[33032]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:54:15.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:15 smithi187 ceph-mon[33032]: osdmap e186: 8 total, 8 up, 8 in 2024-09-21T13:54:15.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:15 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3356408603' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T13:54:15.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:15 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T13:54:15.778 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:15.801 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:15.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:15.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:15.902 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:54:15.916 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-09-21T13:54:15.918 INFO:teuthology.orchestra.run.smithi049.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-09-21T13:54:15.928 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:15.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:15.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:54:16.030 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:16.051 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:16.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:16.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:16.129 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T13:54:16.130 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:54:16.153 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:16.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:16.175 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:16.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:16.276 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:54:16.299 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:16.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:16.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:54:16.331 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T13:54:16.401 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:54:16.423 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:16.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:16.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:54:16.525 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:54:16.547 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:16.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:16.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:54:16.649 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:54:16.671 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:16.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:16.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:54:16.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:16 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1552242343' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:54:16.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:16 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:54:16.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:16 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T13:54:16.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:16 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T13:54:16.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:16 smithi187 ceph-mon[33032]: osdmap e187: 8 total, 8 up, 8 in 2024-09-21T13:54:16.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:16 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1552242343' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:54:16.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:16 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:54:16.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:16 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T13:54:16.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:16 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T13:54:16.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:16 smithi049 ceph-mon[40489]: osdmap e187: 8 total, 8 up, 8 in 2024-09-21T13:54:16.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:16 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1552242343' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:54:16.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:16 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:54:16.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:16 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T13:54:16.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:16 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T13:54:16.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:16 smithi049 ceph-mon[30080]: osdmap e187: 8 total, 8 up, 8 in 2024-09-21T13:54:16.773 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:16.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:16.796 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:16.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:16.897 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:54:16.905 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T13:54:16.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:16.919 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:16.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:54:17.020 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:54:17.042 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:17.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:17.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:54:17.144 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:54:17.167 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:17.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:17.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:54:17.180 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:54:17.269 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:17.293 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:17.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:17.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:17.335 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:54:17.394 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:54:17.417 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:17.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:17.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:54:17.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:17 smithi049 ceph-mon[30080]: pgmap v259: 148 pgs: 148 active+clean; 583 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T13:54:17.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:17 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:54:17.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:17 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:54:17.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:17 smithi049 ceph-mon[30080]: 4.e scrub starts 2024-09-21T13:54:17.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:17 smithi049 ceph-mon[30080]: 4.e scrub ok 2024-09-21T13:54:17.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:17 smithi049 ceph-mon[30080]: 4.4 scrub starts 2024-09-21T13:54:17.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:17 smithi049 ceph-mon[30080]: 4.4 scrub ok 2024-09-21T13:54:17.457 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:17 smithi049 ceph-mon[40489]: pgmap v259: 148 pgs: 148 active+clean; 583 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T13:54:17.457 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:17 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:54:17.457 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:17 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:54:17.457 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:17 smithi049 ceph-mon[40489]: 4.e scrub starts 2024-09-21T13:54:17.457 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:17 smithi049 ceph-mon[40489]: 4.e scrub ok 2024-09-21T13:54:17.457 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:17 smithi049 ceph-mon[40489]: 4.4 scrub starts 2024-09-21T13:54:17.457 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:17 smithi049 ceph-mon[40489]: 4.4 scrub ok 2024-09-21T13:54:17.519 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:54:17.541 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:17.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:17.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:54:17.564 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:17 smithi187 ceph-mon[33032]: pgmap v259: 148 pgs: 148 active+clean; 583 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T13:54:17.564 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:17 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:54:17.564 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:17 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:54:17.564 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:17 smithi187 ceph-mon[33032]: 4.e scrub starts 2024-09-21T13:54:17.564 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:17 smithi187 ceph-mon[33032]: 4.e scrub ok 2024-09-21T13:54:17.564 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:17 smithi187 ceph-mon[33032]: 4.4 scrub starts 2024-09-21T13:54:17.564 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:17 smithi187 ceph-mon[33032]: 4.4 scrub ok 2024-09-21T13:54:17.642 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:54:17.664 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:17.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:17.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:54:17.668 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T13:54:17.668 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:54:17.765 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:17.787 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:17.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:17.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:17.841 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:54:17.842 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:54:17.858 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:54:17.858 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:54:17.874 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:54:17.875 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:54:17.888 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:17.891 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:54:17.891 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:54:17.908 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:54:17.908 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:54:17.910 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:17.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:17.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:17.925 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:54:17.925 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:54:17.942 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:54:17.942 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:54:17.959 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:54:17.959 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:54:18.013 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:18.034 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:18.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:18.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:18.135 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:54:18.157 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:18.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:18.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:54:18.228 DEBUG:teuthology.orchestra.run.smithi049:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-09-21T13:54:18.229 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-09-21T13:54:18.229 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-09-21T13:54:18.229 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T13:54:18.231 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-21T13:54:18.231 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-21T13:54:18.260 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:54:18.281 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:18.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:18.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:54:18.284 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:hints = 1 2024-09-21T13:54:18.284 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-21T13:54:18.284 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Object prefix: benchmark_data_smithi049_109311 2024-09-21T13:54:18.285 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T13:54:18.285 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 0 0 0 0 0 0 - 0 2024-09-21T13:54:18.383 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:18.414 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:18.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:18.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:18.468 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T13:54:18.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:18 smithi049 ceph-mon[30080]: 4.19 scrub starts 2024-09-21T13:54:18.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:18 smithi049 ceph-mon[30080]: 4.19 scrub ok 2024-09-21T13:54:18.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:18 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3664483101' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:54:18.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:18 smithi049 ceph-mon[40489]: 4.19 scrub starts 2024-09-21T13:54:18.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:18 smithi049 ceph-mon[40489]: 4.19 scrub ok 2024-09-21T13:54:18.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:18 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3664483101' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:54:18.516 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:54:18.547 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:18.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:18.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:54:18.649 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:54:18.682 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:18.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:18.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:54:18.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:18 smithi187 ceph-mon[33032]: 4.19 scrub starts 2024-09-21T13:54:18.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:18 smithi187 ceph-mon[33032]: 4.19 scrub ok 2024-09-21T13:54:18.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:18 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3664483101' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:54:18.784 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:54:18.832 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T13:54:18.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:18.834 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:18.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:54:18.883 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:54:18.936 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:54:18.995 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:18.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:18.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:54:19.011 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:54:19.098 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:19.131 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:19.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:19.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:19.232 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:19.266 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:19.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:19.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:19.285 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 1 16 3360 3344 208.969 209 0.00398097 0.00477007 2024-09-21T13:54:19.342 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:54:19 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:54:19.368 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:19.398 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:19 smithi049 ceph-mon[30080]: pgmap v261: 148 pgs: 148 active+clean; 583 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 946 B/s rd, 0 op/s 2024-09-21T13:54:19.399 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:19 smithi049 ceph-mon[30080]: 5.16 scrub starts 2024-09-21T13:54:19.401 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:19.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:19.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:19.501 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T13:54:19.502 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-21T13:54:19.502 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-21T13:54:19.502 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-21T13:54:19.502 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T13:54:19.503 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:54:19.540 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:19.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:19.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:54:19.641 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:19.694 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:19.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:19 smithi187 ceph-mon[33032]: pgmap v261: 148 pgs: 148 active+clean; 583 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 946 B/s rd, 0 op/s 2024-09-21T13:54:19.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:19 smithi187 ceph-mon[33032]: 5.16 scrub starts 2024-09-21T13:54:19.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:19 smithi187 ceph-mon[33032]: 5.16 scrub ok 2024-09-21T13:54:19.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:19 smithi187 ceph-mon[33032]: 3.10 scrub starts 2024-09-21T13:54:19.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:19 smithi187 ceph-mon[33032]: 3.10 scrub ok 2024-09-21T13:54:19.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:19 smithi187 ceph-mon[33032]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T13:54:19.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:19 smithi187 ceph-mon[33032]: 4.18 scrub starts 2024-09-21T13:54:19.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:19 smithi187 ceph-mon[33032]: 4.18 scrub ok 2024-09-21T13:54:19.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:19 smithi187 ceph-mon[33032]: 3.11 scrub starts 2024-09-21T13:54:19.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:19 smithi187 ceph-mon[33032]: 3.11 scrub ok 2024-09-21T13:54:19.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:19.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:19.707 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:54:19.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:19 smithi049 ceph-mon[40489]: pgmap v261: 148 pgs: 148 active+clean; 583 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 946 B/s rd, 0 op/s 2024-09-21T13:54:19.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:19 smithi049 ceph-mon[40489]: 5.16 scrub starts 2024-09-21T13:54:19.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:19 smithi049 ceph-mon[40489]: 5.16 scrub ok 2024-09-21T13:54:19.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:19 smithi049 ceph-mon[40489]: 3.10 scrub starts 2024-09-21T13:54:19.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:19 smithi049 ceph-mon[40489]: 3.10 scrub ok 2024-09-21T13:54:19.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:19 smithi049 ceph-mon[40489]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T13:54:19.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:19 smithi049 ceph-mon[40489]: 4.18 scrub starts 2024-09-21T13:54:19.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:19 smithi049 ceph-mon[40489]: 4.18 scrub ok 2024-09-21T13:54:19.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:19 smithi049 ceph-mon[40489]: 3.11 scrub starts 2024-09-21T13:54:19.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:19 smithi049 ceph-mon[40489]: 3.11 scrub ok 2024-09-21T13:54:19.757 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:19 smithi049 ceph-mon[30080]: 5.16 scrub ok 2024-09-21T13:54:19.758 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:19 smithi049 ceph-mon[30080]: 3.10 scrub starts 2024-09-21T13:54:19.758 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:19 smithi049 ceph-mon[30080]: 3.10 scrub ok 2024-09-21T13:54:19.758 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:19 smithi049 ceph-mon[30080]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T13:54:19.758 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:19 smithi049 ceph-mon[30080]: 4.18 scrub starts 2024-09-21T13:54:19.758 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:19 smithi049 ceph-mon[30080]: 4.18 scrub ok 2024-09-21T13:54:19.758 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:19 smithi049 ceph-mon[30080]: 3.11 scrub starts 2024-09-21T13:54:19.758 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:19 smithi049 ceph-mon[30080]: 3.11 scrub ok 2024-09-21T13:54:19.797 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:19.837 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:19.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:19.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:19.939 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:19.971 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:19.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:19.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:20.073 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:54:20.105 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:20.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:20.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:54:20.110 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:54:20.112 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-09-21T13:54:20.207 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:54:20.246 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T13:54:20.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:20.247 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:20.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:54:20.285 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 2 16 6351 6335 197.945 186.938 0.00287958 0.00504101 2024-09-21T13:54:20.348 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:54:20.394 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:20.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:20.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:54:20.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:20 smithi187 ceph-mon[33032]: 4.0 scrub starts 2024-09-21T13:54:20.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:20 smithi187 ceph-mon[33032]: 4.0 scrub ok 2024-09-21T13:54:20.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:20 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:54:20.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:20 smithi187 ceph-mon[33032]: 4.2 scrub starts 2024-09-21T13:54:20.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:20 smithi187 ceph-mon[33032]: 4.2 scrub ok 2024-09-21T13:54:20.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:20 smithi187 ceph-mon[33032]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:54:20.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:20 smithi187 ceph-mon[33032]: osdmap e188: 8 total, 8 up, 8 in 2024-09-21T13:54:20.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:20 smithi187 ceph-mon[33032]: 5.d scrub starts 2024-09-21T13:54:20.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:20 smithi187 ceph-mon[33032]: 5.d scrub ok 2024-09-21T13:54:20.495 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:54:20.526 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:20.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:20.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:54:20.628 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:20.667 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:20.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:20.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:20.682 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:54:20.703 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:20 smithi049 ceph-mon[40489]: 4.0 scrub starts 2024-09-21T13:54:20.703 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:20 smithi049 ceph-mon[40489]: 4.0 scrub ok 2024-09-21T13:54:20.703 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:20 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:54:20.703 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:20 smithi049 ceph-mon[40489]: 4.2 scrub starts 2024-09-21T13:54:20.703 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:20 smithi049 ceph-mon[40489]: 4.2 scrub ok 2024-09-21T13:54:20.703 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:20 smithi049 ceph-mon[40489]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:54:20.703 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:20 smithi049 ceph-mon[40489]: osdmap e188: 8 total, 8 up, 8 in 2024-09-21T13:54:20.703 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:20 smithi049 ceph-mon[40489]: 5.d scrub starts 2024-09-21T13:54:20.703 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:20 smithi049 ceph-mon[40489]: 5.d scrub ok 2024-09-21T13:54:20.704 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:54:20 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:54:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:54:20.704 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:20 smithi049 ceph-mon[30080]: 4.0 scrub starts 2024-09-21T13:54:20.704 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:20 smithi049 ceph-mon[30080]: 4.0 scrub ok 2024-09-21T13:54:20.704 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:20 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:54:20.704 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:20 smithi049 ceph-mon[30080]: 4.2 scrub starts 2024-09-21T13:54:20.704 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:20 smithi049 ceph-mon[30080]: 4.2 scrub ok 2024-09-21T13:54:20.704 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:20 smithi049 ceph-mon[30080]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:54:20.704 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:20 smithi049 ceph-mon[30080]: osdmap e188: 8 total, 8 up, 8 in 2024-09-21T13:54:20.704 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:20 smithi049 ceph-mon[30080]: 5.d scrub starts 2024-09-21T13:54:20.704 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:20 smithi049 ceph-mon[30080]: 5.d scrub ok 2024-09-21T13:54:20.770 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:20.801 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:20.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:20.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:20.856 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T13:54:20.903 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:54:20.947 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:20.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:20.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:54:21.049 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:54:21.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:21.094 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:21.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:54:21.195 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:21.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:21.229 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:21.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:21.247 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":263,"stamp":"2024-09-21T13:54:19.919657+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":72358427,"num_objects":1294,"num_object_clones":0,"num_object_copies":3882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1294,"num_whiteouts":0,"num_read":1159,"num_read_kb":1016,"num_write":1644,"num_write_kb":71753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":590588,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1654,"ondisk_log_size":1654,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1371160,"kb_used_data":112764,"kb_used_omap":15,"kb_used_meta":1258224,"kb_avail":748590056,"statfs":{"total":767960285184,"available":766556217344,"internally_reserved":0,"allocated":115470336,"data_stored":111450673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15598,"internal_metadata":1288422162},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":71761920,"num_objects":1095,"num_object_clones":0,"num_object_copies":3285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1095,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1095,"num_write_kb":70080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.141448"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":217,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:17.026106+0000","last_change":"2024-09-21T13:52:26.677736+0000","last_active":"2024-09-21T13:54:17.026106+0000","last_peered":"2024-09-21T13:54:17.026106+0000","last_clean":"2024-09-21T13:54:17.026106+0000","last_became_active":"2024-09-21T13:52:04.317322+0000","last_became_peered":"2024-09-21T13:52:04.317322+0000","last_unstale":"2024-09-21T13:54:17.026106+0000","last_undegraded":"2024-09-21T13:54:17.026106+0000","last_fullsized":"2024-09-21T13:54:17.026106+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:26.677689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:52:26.677689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:46.286464+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"68'9","reported_seq":293,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:18.411247+0000","last_change":"2024-09-21T13:54:18.411247+0000","last_active":"2024-09-21T13:54:18.411247+0000","last_peered":"2024-09-21T13:54:18.411247+0000","last_clean":"2024-09-21T13:54:18.411247+0000","last_became_active":"2024-09-21T13:49:00.055210+0000","last_became_peered":"2024-09-21T13:49:00.055210+0000","last_unstale":"2024-09-21T13:54:18.411247+0000","last_undegraded":"2024-09-21T13:54:18.411247+0000","last_fullsized":"2024-09-21T13:54:18.411247+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:54:18.410821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:18.410821+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:40.272886+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":266,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.915105+0000","last_change":"2024-09-21T13:50:28.058990+0000","last_active":"2024-09-21T13:54:15.915105+0000","last_peered":"2024-09-21T13:54:15.915105+0000","last_clean":"2024-09-21T13:54:15.915105+0000","last_became_active":"2024-09-21T13:49:02.060984+0000","last_became_peered":"2024-09-21T13:49:02.060984+0000","last_unstale":"2024-09-21T13:54:15.915105+0000","last_undegraded":"2024-09-21T13:54:15.915105+0000","last_fullsized":"2024-09-21T13:54:15.915105+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:28.058918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:28.058918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:48.620137+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":261,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.915107+0000","last_change":"2024-09-21T13:53:21.005444+0000","last_active":"2024-09-21T13:54:15.915107+0000","last_peered":"2024-09-21T13:54:15.915107+0000","last_clean":"2024-09-21T13:54:15.915107+0000","last_became_active":"2024-09-21T13:49:04.072460+0000","last_became_peered":"2024-09-21T13:49:04.072460+0000","last_unstale":"2024-09-21T13:54:15.915107+0000","last_undegraded":"2024-09-21T13:54:15.915107+0000","last_fullsized":"2024-09-21T13:54:15.915107+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:21.005352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:21.005352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:43.773187+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":261,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.916450+0000","last_change":"2024-09-21T13:51:18.572788+0000","last_active":"2024-09-21T13:54:15.916450+0000","last_peered":"2024-09-21T13:54:15.916450+0000","last_clean":"2024-09-21T13:54:15.916450+0000","last_became_active":"2024-09-21T13:49:04.433965+0000","last_became_peered":"2024-09-21T13:49:04.433965+0000","last_unstale":"2024-09-21T13:54:15.916450+0000","last_undegraded":"2024-09-21T13:54:15.916450+0000","last_fullsized":"2024-09-21T13:54:15.916450+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:18.572729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:51:18.572729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":284,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.918265+0000","last_change":"2024-09-21T13:53:19.910708+0000","last_active":"2024-09-21T13:54:15.918265+0000","last_peered":"2024-09-21T13:54:15.918265+0000","last_clean":"2024-09-21T13:54:15.918265+0000","last_became_active":"2024-09-21T13:52:04.316819+0000","last_became_peered":"2024-09-21T13:52:04.316819+0000","last_unstale":"2024-09-21T13:54:15.918265+0000","last_undegraded":"2024-09-21T13:54:15.918265+0000","last_fullsized":"2024-09-21T13:54:15.918265+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:19.910638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:53:19.910638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:27.797531+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"68'15","reported_seq":305,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:17.024113+0000","last_change":"2024-09-21T13:54:17.024113+0000","last_active":"2024-09-21T13:54:17.024113+0000","last_peered":"2024-09-21T13:54:17.024113+0000","last_clean":"2024-09-21T13:54:17.024113+0000","last_became_active":"2024-09-21T13:52:04.315101+0000","last_became_peered":"2024-09-21T13:52:04.315101+0000","last_unstale":"2024-09-21T13:54:17.024113+0000","last_undegraded":"2024-09-21T13:54:17.024113+0000","last_fullsized":"2024-09-21T13:54:17.024113+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:54:17.024053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:17.024053+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:34.778666+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":273,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.914652+0000","last_change":"2024-09-21T13:51:43.040397+0000","last_active":"2024-09-21T13:54:15.914652+0000","last_peered":"2024-09-21T13:54:15.914652+0000","last_clean":"2024-09-21T13:54:15.914652+0000","last_became_active":"2024-09-21T13:49:02.063687+0000","last_became_peered":"2024-09-21T13:49:02.063687+0000","last_unstale":"2024-09-21T13:54:15.914652+0000","last_undegraded":"2024-09-21T13:54:15.914652+0000","last_fullsized":"2024-09-21T13:54:15.914652+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:43.040348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:43.040348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:08.325545+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":290,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:17.025583+0000","last_change":"2024-09-21T13:53:59.104109+0000","last_active":"2024-09-21T13:54:17.025583+0000","last_peered":"2024-09-21T13:54:17.025583+0000","last_clean":"2024-09-21T13:54:17.025583+0000","last_became_active":"2024-09-21T13:48:58.050695+0000","last_became_peered":"2024-09-21T13:48:58.050695+0000","last_unstale":"2024-09-21T13:54:17.025583+0000","last_undegraded":"2024-09-21T13:54:17.025583+0000","last_fullsized":"2024-09-21T13:54:17.025583+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:59.104062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:53:59.104062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:28.421324+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"68'9","reported_seq":293,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.915754+0000","last_change":"2024-09-21T13:53:58.308240+0000","last_active":"2024-09-21T13:54:15.915754+0000","last_peered":"2024-09-21T13:54:15.915754+0000","last_clean":"2024-09-21T13:54:15.915754+0000","last_became_active":"2024-09-21T13:49:00.059382+0000","last_became_peered":"2024-09-21T13:49:00.059382+0000","last_unstale":"2024-09-21T13:54:15.915754+0000","last_undegraded":"2024-09-21T13:54:15.915754+0000","last_fullsized":"2024-09-21T13:54:15.915754+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:53:58.308144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:58.308144+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:27.655659+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":258,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:17.024967+0000","last_change":"2024-09-21T13:49:02.066978+0000","last_active":"2024-09-21T13:54:17.024967+0000","last_peered":"2024-09-21T13:54:17.024967+0000","last_clean":"2024-09-21T13:54:17.024967+0000","last_became_active":"2024-09-21T13:49:02.066693+0000","last_became_peered":"2024-09-21T13:49:02.066693+0000","last_unstale":"2024-09-21T13:54:17.024967+0000","last_undegraded":"2024-09-21T13:54:17.024967+0000","last_fullsized":"2024-09-21T13:54:17.024967+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:12.027057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":214,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.914354+0000","last_change":"2024-09-21T13:52:28.417865+0000","last_active":"2024-09-21T13:54:15.914354+0000","last_peered":"2024-09-21T13:54:15.914354+0000","last_clean":"2024-09-21T13:54:15.914354+0000","last_became_active":"2024-09-21T13:50:29.689381+0000","last_became_peered":"2024-09-21T13:50:29.689381+0000","last_unstale":"2024-09-21T13:54:15.914354+0000","last_undegraded":"2024-09-21T13:54:15.914354+0000","last_fullsized":"2024-09-21T13:54:15.914354+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:28.417690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:28.417690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":274,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:17.025290+0000","last_change":"2024-09-21T13:50:39.480995+0000","last_active":"2024-09-21T13:54:17.025290+0000","last_peered":"2024-09-21T13:54:17.025290+0000","last_clean":"2024-09-21T13:54:17.025290+0000","last_became_active":"2024-09-21T13:48:58.050906+0000","last_became_peered":"2024-09-21T13:48:58.050906+0000","last_unstale":"2024-09-21T13:54:17.025290+0000","last_undegraded":"2024-09-21T13:54:17.025290+0000","last_fullsized":"2024-09-21T13:54:17.025290+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:39.480930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:39.480930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:34.214082+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"68'5","reported_seq":303,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.915185+0000","last_change":"2024-09-21T13:54:04.271336+0000","last_active":"2024-09-21T13:54:15.915185+0000","last_peered":"2024-09-21T13:54:15.915185+0000","last_clean":"2024-09-21T13:54:15.915185+0000","last_became_active":"2024-09-21T13:52:04.318515+0000","last_became_peered":"2024-09-21T13:52:04.318515+0000","last_unstale":"2024-09-21T13:54:15.915185+0000","last_undegraded":"2024-09-21T13:54:15.915185+0000","last_fullsized":"2024-09-21T13:54:15.915185+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-21T13:54:04.271280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:04.271280+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:18.236498+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":265,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:16.024690+0000","last_change":"2024-09-21T13:50:40.188797+0000","last_active":"2024-09-21T13:54:16.024690+0000","last_peered":"2024-09-21T13:54:16.024690+0000","last_clean":"2024-09-21T13:54:16.024690+0000","last_became_active":"2024-09-21T13:49:02.060212+0000","last_became_peered":"2024-09-21T13:49:02.060212+0000","last_unstale":"2024-09-21T13:54:16.024690+0000","last_undegraded":"2024-09-21T13:54:16.024690+0000","last_fullsized":"2024-09-21T13:54:16.024690+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:40.188712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:40.188712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":270,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:17.024273+0000","last_change":"2024-09-21T13:52:43.516806+0000","last_active":"2024-09-21T13:54:17.024273+0000","last_peered":"2024-09-21T13:54:17.024273+0000","last_clean":"2024-09-21T13:54:17.024273+0000","last_became_active":"2024-09-21T13:49:04.078026+0000","last_became_peered":"2024-09-21T13:49:04.078026+0000","last_unstale":"2024-09-21T13:54:17.024273+0000","last_undegraded":"2024-09-21T13:54:17.024273+0000","last_fullsized":"2024-09-21T13:54:17.024273+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:43.516741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:43.516741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":261,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.915588+0000","last_change":"2024-09-21T13:50:38.027288+0000","last_active":"2024-09-21T13:54:15.915588+0000","last_peered":"2024-09-21T13:54:15.915588+0000","last_clean":"2024-09-21T13:54:15.915588+0000","last_became_active":"2024-09-21T13:49:04.431422+0000","last_became_peered":"2024-09-21T13:49:04.431422+0000","last_unstale":"2024-09-21T13:54:15.915588+0000","last_undegraded":"2024-09-21T13:54:15.915588+0000","last_fullsized":"2024-09-21T13:54:15.915588+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:38.027227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:38.027227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:49.236040+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":205,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.915591+0000","last_change":"2024-09-21T13:54:06.280941+0000","last_active":"2024-09-21T13:54:15.915591+0000","last_peered":"2024-09-21T13:54:15.915591+0000","last_clean":"2024-09-21T13:54:15.915591+0000","last_became_active":"2024-09-21T13:50:29.688282+0000","last_became_peered":"2024-09-21T13:50:29.688282+0000","last_unstale":"2024-09-21T13:54:15.915591+0000","last_undegraded":"2024-09-21T13:54:15.915591+0000","last_fullsized":"2024-09-21T13:54:15.915591+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:06.280894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:06.280894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:07.185880+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"68'15","reported_seq":234,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:16.678017+0000","last_change":"2024-09-21T13:53:32.929362+0000","last_active":"2024-09-21T13:54:16.678017+0000","last_peered":"2024-09-21T13:54:16.678017+0000","last_clean":"2024-09-21T13:54:16.678017+0000","last_became_active":"2024-09-21T13:52:04.324273+0000","last_became_peered":"2024-09-21T13:52:04.324273+0000","last_unstale":"2024-09-21T13:54:16.678017+0000","last_undegraded":"2024-09-21T13:54:16.678017+0000","last_fullsized":"2024-09-21T13:54:16.678017+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:53:32.929017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:32.929017+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:37.476556+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":274,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.916255+0000","last_change":"2024-09-21T13:52:26.372217+0000","last_active":"2024-09-21T13:54:15.916255+0000","last_peered":"2024-09-21T13:54:15.916255+0000","last_clean":"2024-09-21T13:54:15.916255+0000","last_became_active":"2024-09-21T13:49:02.066796+0000","last_became_peered":"2024-09-21T13:49:02.066796+0000","last_unstale":"2024-09-21T13:54:15.916255+0000","last_undegraded":"2024-09-21T13:54:15.916255+0000","last_fullsized":"2024-09-21T13:54:15.916255+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:26.372144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:52:26.372144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":261,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.918431+0000","last_change":"2024-09-21T13:53:57.962327+0000","last_active":"2024-09-21T13:54:15.918431+0000","last_peered":"2024-09-21T13:54:15.918431+0000","last_clean":"2024-09-21T13:54:15.918431+0000","last_became_active":"2024-09-21T13:49:04.434049+0000","last_became_peered":"2024-09-21T13:49:04.434049+0000","last_unstale":"2024-09-21T13:54:15.918431+0000","last_undegraded":"2024-09-21T13:54:15.918431+0000","last_fullsized":"2024-09-21T13:54:15.918431+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:57.962266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:57.962266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:07.827936+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":284,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.915873+0000","last_change":"2024-09-21T13:52:52.142682+0000","last_active":"2024-09-21T13:54:15.915873+0000","last_peered":"2024-09-21T13:54:15.915873+0000","last_clean":"2024-09-21T13:54:15.915873+0000","last_became_active":"2024-09-21T13:52:04.316016+0000","last_became_peered":"2024-09-21T13:52:04.316016+0000","last_unstale":"2024-09-21T13:54:15.915873+0000","last_undegraded":"2024-09-21T13:54:15.915873+0000","last_fullsized":"2024-09-21T13:54:15.915873+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:52.142638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:52:52.142638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"68'12","reported_seq":292,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.917645+0000","last_change":"2024-09-21T13:51:52.819966+0000","last_active":"2024-09-21T13:54:15.917645+0000","last_peered":"2024-09-21T13:54:15.917645+0000","last_clean":"2024-09-21T13:54:15.917645+0000","last_became_active":"2024-09-21T13:49:00.061867+0000","last_became_peered":"2024-09-21T13:49:00.061867+0000","last_unstale":"2024-09-21T13:54:15.917645+0000","last_undegraded":"2024-09-21T13:54:15.917645+0000","last_fullsized":"2024-09-21T13:54:15.917645+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'12","last_scrub_stamp":"2024-09-21T13:51:52.819879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:52.819879+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":268,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.915115+0000","last_change":"2024-09-21T13:52:04.321297+0000","last_active":"2024-09-21T13:54:15.915115+0000","last_peered":"2024-09-21T13:54:15.915115+0000","last_clean":"2024-09-21T13:54:15.915115+0000","last_became_active":"2024-09-21T13:52:04.320800+0000","last_became_peered":"2024-09-21T13:52:04.320800+0000","last_unstale":"2024-09-21T13:54:15.915115+0000","last_undegraded":"2024-09-21T13:54:15.915115+0000","last_fullsized":"2024-09-21T13:54:15.915115+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:15.423635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":273,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:16.025293+0000","last_change":"2024-09-21T13:52:54.580070+0000","last_active":"2024-09-21T13:54:16.025293+0000","last_peered":"2024-09-21T13:54:16.025293+0000","last_clean":"2024-09-21T13:54:16.025293+0000","last_became_active":"2024-09-21T13:52:04.321358+0000","last_became_peered":"2024-09-21T13:52:04.321358+0000","last_unstale":"2024-09-21T13:54:16.025293+0000","last_undegraded":"2024-09-21T13:54:16.025293+0000","last_fullsized":"2024-09-21T13:54:16.025293+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T13:52:54.580022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:54.580022+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:21.087079+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":283,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.915262+0000","last_change":"2024-09-21T13:52:43.516442+0000","last_active":"2024-09-21T13:54:15.915262+0000","last_peered":"2024-09-21T13:54:15.915262+0000","last_clean":"2024-09-21T13:54:15.915262+0000","last_became_active":"2024-09-21T13:48:58.048972+0000","last_became_peered":"2024-09-21T13:48:58.048972+0000","last_unstale":"2024-09-21T13:54:15.915262+0000","last_undegraded":"2024-09-21T13:54:15.915262+0000","last_fullsized":"2024-09-21T13:54:15.915262+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:52:43.516221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:52:43.516221+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"68'10","reported_seq":240,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:16.025805+0000","last_change":"2024-09-21T13:52:56.526032+0000","last_active":"2024-09-21T13:54:16.025805+0000","last_peered":"2024-09-21T13:54:16.025805+0000","last_clean":"2024-09-21T13:54:16.025805+0000","last_became_active":"2024-09-21T13:50:29.694465+0000","last_became_peered":"2024-09-21T13:50:29.694465+0000","last_unstale":"2024-09-21T13:54:16.025805+0000","last_undegraded":"2024-09-21T13:54:16.025805+0000","last_fullsized":"2024-09-21T13:54:16.025805+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:52:56.525966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:52:56.525966+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"68'8","reported_seq":350,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:16.449970+0000","last_change":"2024-09-21T13:51:42.174696+0000","last_active":"2024-09-21T13:54:16.449970+0000","last_peered":"2024-09-21T13:54:16.449970+0000","last_clean":"2024-09-21T13:54:16.449970+0000","last_became_active":"2024-09-21T13:49:02.059638+0000","last_became_peered":"2024-09-21T13:49:02.059638+0000","last_unstale":"2024-09-21T13:54:16.449970+0000","last_undegraded":"2024-09-21T13:54:16.449970+0000","last_fullsized":"2024-09-21T13:54:16.449970+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:51:42.174629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:42.174629+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:14.205512+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"187'54","reported_seq":73,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:18.574524+0000","last_change":"2024-09-21T13:54:12.377270+0000","last_active":"2024-09-21T13:54:18.574524+0000","last_peered":"2024-09-21T13:54:18.574524+0000","last_clean":"2024-09-21T13:54:18.574524+0000","last_became_active":"2024-09-21T13:54:12.376466+0000","last_became_peered":"2024-09-21T13:54:12.376466+0000","last_unstale":"2024-09-21T13:54:18.574524+0000","last_undegraded":"2024-09-21T13:54:18.574524+0000","last_fullsized":"2024-09-21T13:54:18.574524+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:30.045611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3538944,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54,"num_write_kb":3456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"68'15","reported_seq":311,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.918973+0000","last_change":"2024-09-21T13:53:58.658230+0000","last_active":"2024-09-21T13:54:15.918973+0000","last_peered":"2024-09-21T13:54:15.918973+0000","last_clean":"2024-09-21T13:54:15.918973+0000","last_became_active":"2024-09-21T13:49:00.060324+0000","last_became_peered":"2024-09-21T13:49:00.060324+0000","last_unstale":"2024-09-21T13:54:15.918973+0000","last_undegraded":"2024-09-21T13:54:15.918973+0000","last_fullsized":"2024-09-21T13:54:15.918973+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:53:58.658150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:58.658150+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:14.840395+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":273,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.915348+0000","last_change":"2024-09-21T13:50:06.151943+0000","last_active":"2024-09-21T13:54:15.915348+0000","last_peered":"2024-09-21T13:54:15.915348+0000","last_clean":"2024-09-21T13:54:15.915348+0000","last_became_active":"2024-09-21T13:48:58.051911+0000","last_became_peered":"2024-09-21T13:48:58.051911+0000","last_unstale":"2024-09-21T13:54:15.915348+0000","last_undegraded":"2024-09-21T13:54:15.915348+0000","last_fullsized":"2024-09-21T13:54:15.915348+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.151888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.151888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:12.984236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"68'8","reported_seq":341,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.915784+0000","last_change":"2024-09-21T13:53:19.021881+0000","last_active":"2024-09-21T13:54:15.915784+0000","last_peered":"2024-09-21T13:54:15.915784+0000","last_clean":"2024-09-21T13:54:15.915784+0000","last_became_active":"2024-09-21T13:49:02.061679+0000","last_became_peered":"2024-09-21T13:49:02.061679+0000","last_unstale":"2024-09-21T13:54:15.915784+0000","last_undegraded":"2024-09-21T13:54:15.915784+0000","last_fullsized":"2024-09-21T13:54:15.915784+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:53:19.021810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:53:19.021810+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:35.961578+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":262,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:17.024708+0000","last_change":"2024-09-21T13:50:34.481503+0000","last_active":"2024-09-21T13:54:17.024708+0000","last_peered":"2024-09-21T13:54:17.024708+0000","last_clean":"2024-09-21T13:54:17.024708+0000","last_became_active":"2024-09-21T13:49:04.081905+0000","last_became_peered":"2024-09-21T13:49:04.081905+0000","last_unstale":"2024-09-21T13:54:17.024708+0000","last_undegraded":"2024-09-21T13:54:17.024708+0000","last_fullsized":"2024-09-21T13:54:17.024708+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:34.481450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:34.481450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:43.214157+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.c","version":"187'116","reported_seq":135,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:18.738679+0000","last_change":"2024-09-21T13:54:12.479015+0000","last_active":"2024-09-21T13:54:18.738679+0000","last_peered":"2024-09-21T13:54:18.738679+0000","last_clean":"2024-09-21T13:54:18.738679+0000","last_became_active":"2024-09-21T13:54:12.478687+0000","last_became_peered":"2024-09-21T13:54:12.478687+0000","last_unstale":"2024-09-21T13:54:18.738679+0000","last_undegraded":"2024-09-21T13:54:18.738679+0000","last_fullsized":"2024-09-21T13:54:18.738679+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:33.300965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"68'18","reported_seq":309,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:19.001120+0000","last_change":"2024-09-21T13:54:19.001120+0000","last_active":"2024-09-21T13:54:19.001120+0000","last_peered":"2024-09-21T13:54:19.001120+0000","last_clean":"2024-09-21T13:54:19.001120+0000","last_became_active":"2024-09-21T13:49:00.429314+0000","last_became_peered":"2024-09-21T13:49:00.429314+0000","last_unstale":"2024-09-21T13:54:19.001120+0000","last_undegraded":"2024-09-21T13:54:19.001120+0000","last_fullsized":"2024-09-21T13:54:19.001120+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'18","last_scrub_stamp":"2024-09-21T13:54:19.001030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:19.001030+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:34.825871+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":265,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.918977+0000","last_change":"2024-09-21T13:48:58.056669+0000","last_active":"2024-09-21T13:54:15.918977+0000","last_peered":"2024-09-21T13:54:15.918977+0000","last_clean":"2024-09-21T13:54:15.918977+0000","last_became_active":"2024-09-21T13:48:58.056375+0000","last_became_peered":"2024-09-21T13:48:58.056375+0000","last_unstale":"2024-09-21T13:54:15.918977+0000","last_undegraded":"2024-09-21T13:54:15.918977+0000","last_fullsized":"2024-09-21T13:54:15.918977+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:34.834115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":273,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.914664+0000","last_change":"2024-09-21T13:54:07.310089+0000","last_active":"2024-09-21T13:54:15.914664+0000","last_peered":"2024-09-21T13:54:15.914664+0000","last_clean":"2024-09-21T13:54:15.914664+0000","last_became_active":"2024-09-21T13:49:02.062955+0000","last_became_peered":"2024-09-21T13:49:02.062955+0000","last_unstale":"2024-09-21T13:54:15.914664+0000","last_undegraded":"2024-09-21T13:54:15.914664+0000","last_fullsized":"2024-09-21T13:54:15.914664+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:07.310044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:07.310044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:23.878204+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":264,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.915525+0000","last_change":"2024-09-21T13:52:04.316586+0000","last_active":"2024-09-21T13:54:15.915525+0000","last_peered":"2024-09-21T13:54:15.915525+0000","last_clean":"2024-09-21T13:54:15.915525+0000","last_became_active":"2024-09-21T13:52:04.316121+0000","last_became_peered":"2024-09-21T13:52:04.316121+0000","last_unstale":"2024-09-21T13:54:15.915525+0000","last_undegraded":"2024-09-21T13:54:15.915525+0000","last_fullsized":"2024-09-21T13:54:15.915525+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:10.346375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"187'20","reported_seq":39,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:18.377357+0000","last_change":"2024-09-21T13:54:12.479429+0000","last_active":"2024-09-21T13:54:18.377357+0000","last_peered":"2024-09-21T13:54:18.377357+0000","last_clean":"2024-09-21T13:54:18.377357+0000","last_became_active":"2024-09-21T13:54:12.479146+0000","last_became_peered":"2024-09-21T13:54:12.479146+0000","last_unstale":"2024-09-21T13:54:18.377357+0000","last_undegraded":"2024-09-21T13:54:18.377357+0000","last_fullsized":"2024-09-21T13:54:18.377357+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:37.886798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":1280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"68'14","reported_seq":290,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.914715+0000","last_change":"2024-09-21T13:50:29.009786+0000","last_active":"2024-09-21T13:54:15.914715+0000","last_peered":"2024-09-21T13:54:15.914715+0000","last_clean":"2024-09-21T13:54:15.914715+0000","last_became_active":"2024-09-21T13:49:00.055031+0000","last_became_peered":"2024-09-21T13:49:00.055031+0000","last_unstale":"2024-09-21T13:54:15.914715+0000","last_undegraded":"2024-09-21T13:54:15.914715+0000","last_fullsized":"2024-09-21T13:54:15.914715+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'14","last_scrub_stamp":"2024-09-21T13:50:29.009720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:29.009720+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:41.285061+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":275,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:16.583915+0000","last_change":"2024-09-21T13:49:59.253636+0000","last_active":"2024-09-21T13:54:16.583915+0000","last_peered":"2024-09-21T13:54:16.583915+0000","last_clean":"2024-09-21T13:54:16.583915+0000","last_became_active":"2024-09-21T13:48:58.053533+0000","last_became_peered":"2024-09-21T13:48:58.053533+0000","last_unstale":"2024-09-21T13:54:16.583915+0000","last_undegraded":"2024-09-21T13:54:16.583915+0000","last_fullsized":"2024-09-21T13:54:16.583915+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:49:59.253587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:49:59.253587+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:36.768506+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"68'8","reported_seq":338,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.915081+0000","last_change":"2024-09-21T13:50:41.458637+0000","last_active":"2024-09-21T13:54:15.915081+0000","last_peered":"2024-09-21T13:54:15.915081+0000","last_clean":"2024-09-21T13:54:15.915081+0000","last_became_active":"2024-09-21T13:49:02.070064+0000","last_became_peered":"2024-09-21T13:49:02.070064+0000","last_unstale":"2024-09-21T13:54:15.915081+0000","last_undegraded":"2024-09-21T13:54:15.915081+0000","last_fullsized":"2024-09-21T13:54:15.915081+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:50:41.458562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:41.458562+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":272,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:16.024822+0000","last_change":"2024-09-21T13:52:55.541288+0000","last_active":"2024-09-21T13:54:16.024822+0000","last_peered":"2024-09-21T13:54:16.024822+0000","last_clean":"2024-09-21T13:54:16.024822+0000","last_became_active":"2024-09-21T13:52:04.322185+0000","last_became_peered":"2024-09-21T13:52:04.322185+0000","last_unstale":"2024-09-21T13:54:16.024822+0000","last_undegraded":"2024-09-21T13:54:16.024822+0000","last_fullsized":"2024-09-21T13:54:16.024822+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:55.541232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:55.541232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.e","version":"187'88","reported_seq":107,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:18.737479+0000","last_change":"2024-09-21T13:54:12.480264+0000","last_active":"2024-09-21T13:54:18.737479+0000","last_peered":"2024-09-21T13:54:18.737479+0000","last_clean":"2024-09-21T13:54:18.737479+0000","last_became_active":"2024-09-21T13:54:12.479901+0000","last_became_peered":"2024-09-21T13:54:12.479901+0000","last_unstale":"2024-09-21T13:54:18.737479+0000","last_undegraded":"2024-09-21T13:54:18.737479+0000","last_fullsized":"2024-09-21T13:54:18.737479+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:35.816508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5767168,"num_objects":88,"num_object_clones":0,"num_object_copies":264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88,"num_write_kb":5632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"68'10","reported_seq":293,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.915562+0000","last_change":"2024-09-21T13:51:50.211365+0000","last_active":"2024-09-21T13:54:15.915562+0000","last_peered":"2024-09-21T13:54:15.915562+0000","last_clean":"2024-09-21T13:54:15.915562+0000","last_became_active":"2024-09-21T13:49:00.054564+0000","last_became_peered":"2024-09-21T13:49:00.054564+0000","last_unstale":"2024-09-21T13:54:15.915562+0000","last_undegraded":"2024-09-21T13:54:15.915562+0000","last_fullsized":"2024-09-21T13:54:15.915562+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:51:50.211279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:50.211279+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"61'1","reported_seq":307,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:17.025672+0000","last_change":"2024-09-21T13:53:30.179679+0000","last_active":"2024-09-21T13:54:17.025672+0000","last_peered":"2024-09-21T13:54:17.025672+0000","last_clean":"2024-09-21T13:54:17.025672+0000","last_became_active":"2024-09-21T13:52:04.315788+0000","last_became_peered":"2024-09-21T13:52:04.315788+0000","last_unstale":"2024-09-21T13:54:17.025672+0000","last_undegraded":"2024-09-21T13:54:17.025672+0000","last_fullsized":"2024-09-21T13:54:17.025672+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:53:30.179605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:53:30.179605+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:43.052251+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"96'9","reported_seq":294,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:16.449863+0000","last_change":"2024-09-21T13:54:06.280517+0000","last_active":"2024-09-21T13:54:16.449863+0000","last_peered":"2024-09-21T13:54:16.449863+0000","last_clean":"2024-09-21T13:54:16.449863+0000","last_became_active":"2024-09-21T13:50:29.693557+0000","last_became_peered":"2024-09-21T13:50:29.693557+0000","last_unstale":"2024-09-21T13:54:16.449863+0000","last_undegraded":"2024-09-21T13:54:16.449863+0000","last_fullsized":"2024-09-21T13:54:16.449863+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'9","last_scrub_stamp":"2024-09-21T13:54:06.280457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:06.280457+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:23.993139+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":216,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.914865+0000","last_change":"2024-09-21T13:53:54.948771+0000","last_active":"2024-09-21T13:54:15.914865+0000","last_peered":"2024-09-21T13:54:15.914865+0000","last_clean":"2024-09-21T13:54:15.914865+0000","last_became_active":"2024-09-21T13:52:04.319034+0000","last_became_peered":"2024-09-21T13:52:04.319034+0000","last_unstale":"2024-09-21T13:54:15.914865+0000","last_undegraded":"2024-09-21T13:54:15.914865+0000","last_fullsized":"2024-09-21T13:54:15.914865+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:54.948705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:54.948705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:05.874612+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.f","version":"187'167","reported_seq":185,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:19.032273+0000","last_change":"2024-09-21T13:54:12.478557+0000","last_active":"2024-09-21T13:54:19.032273+0000","last_peered":"2024-09-21T13:54:19.032273+0000","last_clean":"2024-09-21T13:54:19.032273+0000","last_became_active":"2024-09-21T13:54:12.477589+0000","last_became_peered":"2024-09-21T13:54:12.477589+0000","last_unstale":"2024-09-21T13:54:19.032273+0000","last_undegraded":"2024-09-21T13:54:19.032273+0000","last_fullsized":"2024-09-21T13:54:19.032273+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:29.462737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10944512,"num_objects":167,"num_object_clones":0,"num_object_copies":501,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":167,"num_write_kb":10688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"68'19","reported_seq":262,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:17.025024+0000","last_change":"2024-09-21T13:50:32.514267+0000","last_active":"2024-09-21T13:54:17.025024+0000","last_peered":"2024-09-21T13:54:17.025024+0000","last_clean":"2024-09-21T13:54:17.025024+0000","last_became_active":"2024-09-21T13:50:29.690185+0000","last_became_peered":"2024-09-21T13:50:29.690185+0000","last_unstale":"2024-09-21T13:54:17.025024+0000","last_undegraded":"2024-09-21T13:54:17.025024+0000","last_fullsized":"2024-09-21T13:54:17.025024+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-21T13:50:32.514210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:32.514210+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:35.217090+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":277,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.916330+0000","last_change":"2024-09-21T13:52:04.322289+0000","last_active":"2024-09-21T13:54:15.916330+0000","last_peered":"2024-09-21T13:54:15.916330+0000","last_clean":"2024-09-21T13:54:15.916330+0000","last_became_active":"2024-09-21T13:52:04.321656+0000","last_became_peered":"2024-09-21T13:52:04.321656+0000","last_unstale":"2024-09-21T13:54:15.916330+0000","last_undegraded":"2024-09-21T13:54:15.916330+0000","last_fullsized":"2024-09-21T13:54:15.916330+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:12.655427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":274,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:16.449472+0000","last_change":"2024-09-21T13:53:32.295930+0000","last_active":"2024-09-21T13:54:16.449472+0000","last_peered":"2024-09-21T13:54:16.449472+0000","last_clean":"2024-09-21T13:54:16.449472+0000","last_became_active":"2024-09-21T13:49:02.061187+0000","last_became_peered":"2024-09-21T13:49:02.061187+0000","last_unstale":"2024-09-21T13:54:16.449472+0000","last_undegraded":"2024-09-21T13:54:16.449472+0000","last_fullsized":"2024-09-21T13:54:16.449472+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:32.295869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:53:32.295869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:32.443805+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":265,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.915103+0000","last_change":"2024-09-21T13:52:04.321133+0000","last_active":"2024-09-21T13:54:15.915103+0000","last_peered":"2024-09-21T13:54:15.915103+0000","last_clean":"2024-09-21T13:54:15.915103+0000","last_became_active":"2024-09-21T13:52:04.320826+0000","last_became_peered":"2024-09-21T13:52:04.320826+0000","last_unstale":"2024-09-21T13:54:15.915103+0000","last_undegraded":"2024-09-21T13:54:15.915103+0000","last_fullsized":"2024-09-21T13:54:15.915103+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:14.698955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"187'19","reported_seq":38,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:18.380544+0000","last_change":"2024-09-21T13:54:12.477847+0000","last_active":"2024-09-21T13:54:18.380544+0000","last_peered":"2024-09-21T13:54:18.380544+0000","last_clean":"2024-09-21T13:54:18.380544+0000","last_became_active":"2024-09-21T13:54:12.477306+0000","last_became_peered":"2024-09-21T13:54:12.477306+0000","last_unstale":"2024-09-21T13:54:18.380544+0000","last_undegraded":"2024-09-21T13:54:18.380544+0000","last_fullsized":"2024-09-21T13:54:18.380544+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:26.986694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1245184,"num_objects":19,"num_object_clones":0,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19,"num_write_kb":1216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"132'30","reported_seq":396,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:16.681069+0000","last_change":"2024-09-21T13:54:16.681017+0000","last_active":"2024-09-21T13:54:16.681069+0000","last_peered":"2024-09-21T13:54:16.681069+0000","last_clean":"2024-09-21T13:54:16.681069+0000","last_became_active":"2024-09-21T13:52:04.319431+0000","last_became_peered":"2024-09-21T13:52:04.319431+0000","last_unstale":"2024-09-21T13:54:16.681069+0000","last_undegraded":"2024-09-21T13:54:16.681069+0000","last_fullsized":"2024-09-21T13:54:16.681069+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'30","last_scrub_stamp":"2024-09-21T13:54:16.680953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:16.680953+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:39.761349+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":265,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.915610+0000","last_change":"2024-09-21T13:48:58.055640+0000","last_active":"2024-09-21T13:54:15.915610+0000","last_peered":"2024-09-21T13:54:15.915610+0000","last_clean":"2024-09-21T13:54:15.915610+0000","last_became_active":"2024-09-21T13:48:58.055366+0000","last_became_peered":"2024-09-21T13:48:58.055366+0000","last_unstale":"2024-09-21T13:54:15.915610+0000","last_undegraded":"2024-09-21T13:54:15.915610+0000","last_fullsized":"2024-09-21T13:54:15.915610+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:09.329706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":292,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:17.025725+0000","last_change":"2024-09-21T13:52:51.445882+0000","last_active":"2024-09-21T13:54:17.025725+0000","last_peered":"2024-09-21T13:54:17.025725+0000","last_clean":"2024-09-21T13:54:17.025725+0000","last_became_active":"2024-09-21T13:48:57.046246+0000","last_became_peered":"2024-09-21T13:48:57.046246+0000","last_unstale":"2024-09-21T13:54:17.025725+0000","last_undegraded":"2024-09-21T13:54:17.025725+0000","last_fullsized":"2024-09-21T13:54:17.025725+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-21T13:52:51.445812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:52:51.445812+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:18.734189+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.0015946669999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":214,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.916046+0000","last_change":"2024-09-21T13:53:56.670634+0000","last_active":"2024-09-21T13:54:15.916046+0000","last_peered":"2024-09-21T13:54:15.916046+0000","last_clean":"2024-09-21T13:54:15.916046+0000","last_became_active":"2024-09-21T13:50:29.688249+0000","last_became_peered":"2024-09-21T13:50:29.688249+0000","last_unstale":"2024-09-21T13:54:15.916046+0000","last_undegraded":"2024-09-21T13:54:15.916046+0000","last_fullsized":"2024-09-21T13:54:15.916046+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:56.670555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:53:56.670555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:05.626771+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":261,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.918690+0000","last_change":"2024-09-21T13:51:23.597164+0000","last_active":"2024-09-21T13:54:15.918690+0000","last_peered":"2024-09-21T13:54:15.918690+0000","last_clean":"2024-09-21T13:54:15.918690+0000","last_became_active":"2024-09-21T13:49:04.080466+0000","last_became_peered":"2024-09-21T13:49:04.080466+0000","last_unstale":"2024-09-21T13:54:15.918690+0000","last_undegraded":"2024-09-21T13:54:15.918690+0000","last_fullsized":"2024-09-21T13:54:15.918690+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:23.596879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:51:23.596879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"187'150","reported_seq":168,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:19.026964+0000","last_change":"2024-09-21T13:54:12.373760+0000","last_active":"2024-09-21T13:54:19.026964+0000","last_peered":"2024-09-21T13:54:19.026964+0000","last_clean":"2024-09-21T13:54:19.026964+0000","last_became_active":"2024-09-21T13:54:12.373395+0000","last_became_peered":"2024-09-21T13:54:12.373395+0000","last_unstale":"2024-09-21T13:54:19.026964+0000","last_undegraded":"2024-09-21T13:54:19.026964+0000","last_fullsized":"2024-09-21T13:54:19.026964+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:36.149952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9830400,"num_objects":150,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":150,"num_write_kb":9600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"68'13","reported_seq":297,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.918973+0000","last_change":"2024-09-21T13:50:06.973399+0000","last_active":"2024-09-21T13:54:15.918973+0000","last_peered":"2024-09-21T13:54:15.918973+0000","last_clean":"2024-09-21T13:54:15.918973+0000","last_became_active":"2024-09-21T13:49:00.053323+0000","last_became_peered":"2024-09-21T13:49:00.053323+0000","last_unstale":"2024-09-21T13:54:15.918973+0000","last_undegraded":"2024-09-21T13:54:15.918973+0000","last_fullsized":"2024-09-21T13:54:15.918973+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'13","last_scrub_stamp":"2024-09-21T13:50:06.973336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.973336+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:41.649253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":301,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.914994+0000","last_change":"2024-09-21T13:53:24.885266+0000","last_active":"2024-09-21T13:54:15.914994+0000","last_peered":"2024-09-21T13:54:15.914994+0000","last_clean":"2024-09-21T13:54:15.914994+0000","last_became_active":"2024-09-21T13:52:04.320581+0000","last_became_peered":"2024-09-21T13:52:04.320581+0000","last_unstale":"2024-09-21T13:54:15.914994+0000","last_undegraded":"2024-09-21T13:54:15.914994+0000","last_fullsized":"2024-09-21T13:54:15.914994+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:24.884932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:53:24.884932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:31.375453+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":201,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:16.677652+0000","last_change":"2024-09-21T13:53:19.985044+0000","last_active":"2024-09-21T13:54:16.677652+0000","last_peered":"2024-09-21T13:54:16.677652+0000","last_clean":"2024-09-21T13:54:16.677652+0000","last_became_active":"2024-09-21T13:52:04.316171+0000","last_became_peered":"2024-09-21T13:52:04.316171+0000","last_unstale":"2024-09-21T13:54:16.677652+0000","last_undegraded":"2024-09-21T13:54:16.677652+0000","last_fullsized":"2024-09-21T13:54:16.677652+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:53:19.984966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:53:19.984966+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:47.985977+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":145,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:16.678144+0000","last_change":"2024-09-21T13:53:18.988030+0000","last_active":"2024-09-21T13:54:16.678144+0000","last_peered":"2024-09-21T13:54:16.678144+0000","last_clean":"2024-09-21T13:54:16.678144+0000","last_became_active":"2024-09-21T13:52:04.320701+0000","last_became_peered":"2024-09-21T13:52:04.320701+0000","last_unstale":"2024-09-21T13:54:16.678144+0000","last_undegraded":"2024-09-21T13:54:16.678144+0000","last_fullsized":"2024-09-21T13:54:16.678144+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:18.987829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:53:18.987829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:21.866149+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":269,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:16.025159+0000","last_change":"2024-09-21T13:51:12.212558+0000","last_active":"2024-09-21T13:54:16.025159+0000","last_peered":"2024-09-21T13:54:16.025159+0000","last_clean":"2024-09-21T13:54:16.025159+0000","last_became_active":"2024-09-21T13:49:04.431035+0000","last_became_peered":"2024-09-21T13:49:04.431035+0000","last_unstale":"2024-09-21T13:54:16.025159+0000","last_undegraded":"2024-09-21T13:54:16.025159+0000","last_fullsized":"2024-09-21T13:54:16.025159+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:12.212391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:51:12.212391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"187'71","reported_seq":90,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:18.581196+0000","last_change":"2024-09-21T13:54:12.477912+0000","last_active":"2024-09-21T13:54:18.581196+0000","last_peered":"2024-09-21T13:54:18.581196+0000","last_clean":"2024-09-21T13:54:18.581196+0000","last_became_active":"2024-09-21T13:54:12.477596+0000","last_became_peered":"2024-09-21T13:54:12.477596+0000","last_unstale":"2024-09-21T13:54:18.581196+0000","last_undegraded":"2024-09-21T13:54:18.581196+0000","last_fullsized":"2024-09-21T13:54:18.581196+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:33.789453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4653056,"num_objects":71,"num_object_clones":0,"num_object_copies":213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71,"num_write_kb":4544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"68'12","reported_seq":328,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:16.583881+0000","last_change":"2024-09-21T13:53:17.740156+0000","last_active":"2024-09-21T13:54:16.583881+0000","last_peered":"2024-09-21T13:54:16.583881+0000","last_clean":"2024-09-21T13:54:16.583881+0000","last_became_active":"2024-09-21T13:52:04.318875+0000","last_became_peered":"2024-09-21T13:52:04.318875+0000","last_unstale":"2024-09-21T13:54:16.583881+0000","last_undegraded":"2024-09-21T13:54:16.583881+0000","last_fullsized":"2024-09-21T13:54:16.583881+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'12","last_scrub_stamp":"2024-09-21T13:53:17.740080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:17.740080+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:30.638014+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":205,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.914907+0000","last_change":"2024-09-21T13:50:30.020722+0000","last_active":"2024-09-21T13:54:15.914907+0000","last_peered":"2024-09-21T13:54:15.914907+0000","last_clean":"2024-09-21T13:54:15.914907+0000","last_became_active":"2024-09-21T13:50:29.688352+0000","last_became_peered":"2024-09-21T13:50:29.688352+0000","last_unstale":"2024-09-21T13:54:15.914907+0000","last_undegraded":"2024-09-21T13:54:15.914907+0000","last_fullsized":"2024-09-21T13:54:15.914907+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:30.020656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:30.020656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:38.234245+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":599,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:17.324785+0000","last_change":"2024-09-21T13:51:21.271997+0000","last_active":"2024-09-21T13:54:17.324785+0000","last_peered":"2024-09-21T13:54:17.324785+0000","last_clean":"2024-09-21T13:54:17.324785+0000","last_became_active":"2024-09-21T13:48:57.046084+0000","last_became_peered":"2024-09-21T13:48:57.046084+0000","last_unstale":"2024-09-21T13:54:17.324785+0000","last_undegraded":"2024-09-21T13:54:17.324785+0000","last_fullsized":"2024-09-21T13:54:17.324785+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-21T13:51:21.271917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:51:21.271917+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00047048000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":315,"num_read_kb":310,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":266,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:17.026034+0000","last_change":"2024-09-21T13:51:53.518645+0000","last_active":"2024-09-21T13:54:17.026034+0000","last_peered":"2024-09-21T13:54:17.026034+0000","last_clean":"2024-09-21T13:54:17.026034+0000","last_became_active":"2024-09-21T13:49:02.066641+0000","last_became_peered":"2024-09-21T13:49:02.066641+0000","last_unstale":"2024-09-21T13:54:17.026034+0000","last_undegraded":"2024-09-21T13:54:17.026034+0000","last_fullsized":"2024-09-21T13:54:17.026034+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:53.518567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:53.518567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:16.069137+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":270,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.916596+0000","last_change":"2024-09-21T13:52:21.332066+0000","last_active":"2024-09-21T13:54:15.916596+0000","last_peered":"2024-09-21T13:54:15.916596+0000","last_clean":"2024-09-21T13:54:15.916596+0000","last_became_active":"2024-09-21T13:49:04.078372+0000","last_became_peered":"2024-09-21T13:49:04.078372+0000","last_unstale":"2024-09-21T13:54:15.916596+0000","last_undegraded":"2024-09-21T13:54:15.916596+0000","last_fullsized":"2024-09-21T13:54:15.916596+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:21.331792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:21.331792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"187'32","reported_seq":51,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:18.417650+0000","last_change":"2024-09-21T13:54:12.378930+0000","last_active":"2024-09-21T13:54:18.417650+0000","last_peered":"2024-09-21T13:54:18.417650+0000","last_clean":"2024-09-21T13:54:18.417650+0000","last_became_active":"2024-09-21T13:54:12.378536+0000","last_became_peered":"2024-09-21T13:54:12.378536+0000","last_unstale":"2024-09-21T13:54:18.417650+0000","last_undegraded":"2024-09-21T13:54:18.417650+0000","last_fullsized":"2024-09-21T13:54:18.417650+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:32.246204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2097152,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32,"num_write_kb":2048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"68'16","reported_seq":365,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:16.449949+0000","last_change":"2024-09-21T13:52:19.452239+0000","last_active":"2024-09-21T13:54:16.449949+0000","last_peered":"2024-09-21T13:54:16.449949+0000","last_clean":"2024-09-21T13:54:16.449949+0000","last_became_active":"2024-09-21T13:49:00.426867+0000","last_became_peered":"2024-09-21T13:49:00.426867+0000","last_unstale":"2024-09-21T13:54:16.449949+0000","last_undegraded":"2024-09-21T13:54:16.449949+0000","last_fullsized":"2024-09-21T13:54:16.449949+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'16","last_scrub_stamp":"2024-09-21T13:52:19.452163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:52:19.452163+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:49.265943+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":281,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.915875+0000","last_change":"2024-09-21T13:54:07.057258+0000","last_active":"2024-09-21T13:54:15.915875+0000","last_peered":"2024-09-21T13:54:15.915875+0000","last_clean":"2024-09-21T13:54:15.915875+0000","last_became_active":"2024-09-21T13:48:58.052659+0000","last_became_peered":"2024-09-21T13:48:58.052659+0000","last_unstale":"2024-09-21T13:54:15.915875+0000","last_undegraded":"2024-09-21T13:54:15.915875+0000","last_fullsized":"2024-09-21T13:54:15.915875+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:07.057210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:07.057210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:14.084334+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"75'79","reported_seq":322,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:16.025352+0000","last_change":"2024-09-21T13:50:27.096470+0000","last_active":"2024-09-21T13:54:16.025352+0000","last_peered":"2024-09-21T13:54:16.025352+0000","last_clean":"2024-09-21T13:54:16.025352+0000","last_became_active":"2024-09-21T13:48:48.454646+0000","last_became_peered":"2024-09-21T13:48:48.454646+0000","last_unstale":"2024-09-21T13:54:16.025352+0000","last_undegraded":"2024-09-21T13:54:16.025352+0000","last_fullsized":"2024-09-21T13:54:16.025352+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:50:27.096413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:50:27.096413+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:37.381057+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":268,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:16.025597+0000","last_change":"2024-09-21T13:52:04.320948+0000","last_active":"2024-09-21T13:54:16.025597+0000","last_peered":"2024-09-21T13:54:16.025597+0000","last_clean":"2024-09-21T13:54:16.025597+0000","last_became_active":"2024-09-21T13:52:04.320627+0000","last_became_peered":"2024-09-21T13:52:04.320627+0000","last_unstale":"2024-09-21T13:54:16.025597+0000","last_undegraded":"2024-09-21T13:54:16.025597+0000","last_fullsized":"2024-09-21T13:54:16.025597+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:50.544725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":262,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:17.024471+0000","last_change":"2024-09-21T13:50:25.560928+0000","last_active":"2024-09-21T13:54:17.024471+0000","last_peered":"2024-09-21T13:54:17.024471+0000","last_clean":"2024-09-21T13:54:17.024471+0000","last_became_active":"2024-09-21T13:49:04.078091+0000","last_became_peered":"2024-09-21T13:49:04.078091+0000","last_unstale":"2024-09-21T13:54:17.024471+0000","last_undegraded":"2024-09-21T13:54:17.024471+0000","last_fullsized":"2024-09-21T13:54:17.024471+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:25.560863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:25.560863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:41.214571+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"187'22","reported_seq":40,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:18.412534+0000","last_change":"2024-09-21T13:54:12.378230+0000","last_active":"2024-09-21T13:54:18.412534+0000","last_peered":"2024-09-21T13:54:18.412534+0000","last_clean":"2024-09-21T13:54:18.412534+0000","last_became_active":"2024-09-21T13:54:12.377874+0000","last_became_peered":"2024-09-21T13:54:12.377874+0000","last_unstale":"2024-09-21T13:54:18.412534+0000","last_undegraded":"2024-09-21T13:54:18.412534+0000","last_fullsized":"2024-09-21T13:54:18.412534+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:39.278391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"68'11","reported_seq":288,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:16.451648+0000","last_change":"2024-09-21T13:54:16.451648+0000","last_active":"2024-09-21T13:54:16.451648+0000","last_peered":"2024-09-21T13:54:16.451648+0000","last_clean":"2024-09-21T13:54:16.451648+0000","last_became_active":"2024-09-21T13:49:00.053268+0000","last_became_peered":"2024-09-21T13:49:00.053268+0000","last_unstale":"2024-09-21T13:54:16.451648+0000","last_undegraded":"2024-09-21T13:54:16.451648+0000","last_fullsized":"2024-09-21T13:54:16.451648+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:54:16.451588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:16.451588+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:41.356489+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":284,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.915095+0000","last_change":"2024-09-21T13:52:48.165210+0000","last_active":"2024-09-21T13:54:15.915095+0000","last_peered":"2024-09-21T13:54:15.915095+0000","last_clean":"2024-09-21T13:54:15.915095+0000","last_became_active":"2024-09-21T13:52:04.318345+0000","last_became_peered":"2024-09-21T13:52:04.318345+0000","last_unstale":"2024-09-21T13:54:15.915095+0000","last_undegraded":"2024-09-21T13:54:15.915095+0000","last_fullsized":"2024-09-21T13:54:15.915095+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:48.165145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:52:48.165145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":274,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:17.025903+0000","last_change":"2024-09-21T13:51:46.503640+0000","last_active":"2024-09-21T13:54:17.025903+0000","last_peered":"2024-09-21T13:54:17.025903+0000","last_clean":"2024-09-21T13:54:17.025903+0000","last_became_active":"2024-09-21T13:49:02.061901+0000","last_became_peered":"2024-09-21T13:49:02.061901+0000","last_unstale":"2024-09-21T13:54:17.025903+0000","last_undegraded":"2024-09-21T13:54:17.025903+0000","last_fullsized":"2024-09-21T13:54:17.025903+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:46.503576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:46.503576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:15.052988+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":261,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.915606+0000","last_change":"2024-09-21T13:51:51.814259+0000","last_active":"2024-09-21T13:54:15.915606+0000","last_peered":"2024-09-21T13:54:15.915606+0000","last_clean":"2024-09-21T13:54:15.915606+0000","last_became_active":"2024-09-21T13:49:04.431268+0000","last_became_peered":"2024-09-21T13:49:04.431268+0000","last_unstale":"2024-09-21T13:54:15.915606+0000","last_undegraded":"2024-09-21T13:54:15.915606+0000","last_fullsized":"2024-09-21T13:54:15.915606+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:51.814200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:51:51.814200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1","version":"187'66","reported_seq":85,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:18.577038+0000","last_change":"2024-09-21T13:54:12.478397+0000","last_active":"2024-09-21T13:54:18.577038+0000","last_peered":"2024-09-21T13:54:18.577038+0000","last_clean":"2024-09-21T13:54:18.577038+0000","last_became_active":"2024-09-21T13:54:12.477588+0000","last_became_peered":"2024-09-21T13:54:12.477588+0000","last_unstale":"2024-09-21T13:54:18.577038+0000","last_undegraded":"2024-09-21T13:54:18.577038+0000","last_fullsized":"2024-09-21T13:54:18.577038+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:37.981662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4325376,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":66,"num_write_kb":4224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"68'17","reported_seq":308,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.915490+0000","last_change":"2024-09-21T13:52:45.176950+0000","last_active":"2024-09-21T13:54:15.915490+0000","last_peered":"2024-09-21T13:54:15.915490+0000","last_clean":"2024-09-21T13:54:15.915490+0000","last_became_active":"2024-09-21T13:52:04.313218+0000","last_became_peered":"2024-09-21T13:52:04.313218+0000","last_unstale":"2024-09-21T13:54:15.915490+0000","last_undegraded":"2024-09-21T13:54:15.915490+0000","last_fullsized":"2024-09-21T13:54:15.915490+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'17","last_scrub_stamp":"2024-09-21T13:52:45.176873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:52:45.176873+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":290,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:16.449693+0000","last_change":"2024-09-21T13:53:55.202100+0000","last_active":"2024-09-21T13:54:16.449693+0000","last_peered":"2024-09-21T13:54:16.449693+0000","last_clean":"2024-09-21T13:54:16.449693+0000","last_became_active":"2024-09-21T13:48:58.049142+0000","last_became_peered":"2024-09-21T13:48:58.049142+0000","last_unstale":"2024-09-21T13:54:16.449693+0000","last_undegraded":"2024-09-21T13:54:16.449693+0000","last_fullsized":"2024-09-21T13:54:16.449693+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:55.202057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:53:55.202057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:00.441490+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":266,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.915968+0000","last_change":"2024-09-21T13:54:06.629867+0000","last_active":"2024-09-21T13:54:15.915968+0000","last_peered":"2024-09-21T13:54:15.915968+0000","last_clean":"2024-09-21T13:54:15.915968+0000","last_became_active":"2024-09-21T13:49:02.067472+0000","last_became_peered":"2024-09-21T13:49:02.067472+0000","last_unstale":"2024-09-21T13:54:15.915968+0000","last_undegraded":"2024-09-21T13:54:15.915968+0000","last_fullsized":"2024-09-21T13:54:15.915968+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:06.629822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:06.629822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:14.040598+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":145,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:16.678254+0000","last_change":"2024-09-21T13:53:18.028013+0000","last_active":"2024-09-21T13:54:16.678254+0000","last_peered":"2024-09-21T13:54:16.678254+0000","last_clean":"2024-09-21T13:54:16.678254+0000","last_became_active":"2024-09-21T13:52:04.315920+0000","last_became_peered":"2024-09-21T13:52:04.315920+0000","last_unstale":"2024-09-21T13:54:16.678254+0000","last_undegraded":"2024-09-21T13:54:16.678254+0000","last_fullsized":"2024-09-21T13:54:16.678254+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:18.027944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:18.027944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:26.231349+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.0","version":"187'147","reported_seq":165,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:19.027910+0000","last_change":"2024-09-21T13:54:12.375805+0000","last_active":"2024-09-21T13:54:19.027910+0000","last_peered":"2024-09-21T13:54:19.027910+0000","last_clean":"2024-09-21T13:54:19.027910+0000","last_became_active":"2024-09-21T13:54:12.375197+0000","last_became_peered":"2024-09-21T13:54:12.375197+0000","last_unstale":"2024-09-21T13:54:19.027910+0000","last_undegraded":"2024-09-21T13:54:19.027910+0000","last_fullsized":"2024-09-21T13:54:19.027910+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:17.781582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9633792,"num_objects":147,"num_object_clones":0,"num_object_copies":441,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":147,"num_write_kb":9408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"68'10","reported_seq":284,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.915249+0000","last_change":"2024-09-21T13:51:57.155811+0000","last_active":"2024-09-21T13:54:15.915249+0000","last_peered":"2024-09-21T13:54:15.915249+0000","last_clean":"2024-09-21T13:54:15.915249+0000","last_became_active":"2024-09-21T13:49:00.059850+0000","last_became_peered":"2024-09-21T13:49:00.059850+0000","last_unstale":"2024-09-21T13:54:15.915249+0000","last_undegraded":"2024-09-21T13:54:15.915249+0000","last_fullsized":"2024-09-21T13:54:15.915249+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:51:57.155761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:57.155761+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":273,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.917038+0000","last_change":"2024-09-21T13:51:28.565842+0000","last_active":"2024-09-21T13:54:15.917038+0000","last_peered":"2024-09-21T13:54:15.917038+0000","last_clean":"2024-09-21T13:54:15.917038+0000","last_became_active":"2024-09-21T13:48:58.058309+0000","last_became_peered":"2024-09-21T13:48:58.058309+0000","last_unstale":"2024-09-21T13:54:15.917038+0000","last_undegraded":"2024-09-21T13:54:15.917038+0000","last_fullsized":"2024-09-21T13:54:15.917038+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:28.565766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:51:28.565766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"132'10","reported_seq":182,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:16.678044+0000","last_change":"2024-09-21T13:52:04.320834+0000","last_active":"2024-09-21T13:54:16.678044+0000","last_peered":"2024-09-21T13:54:16.678044+0000","last_clean":"2024-09-21T13:54:16.678044+0000","last_became_active":"2024-09-21T13:52:04.320441+0000","last_became_peered":"2024-09-21T13:52:04.320441+0000","last_unstale":"2024-09-21T13:54:16.678044+0000","last_undegraded":"2024-09-21T13:54:16.678044+0000","last_fullsized":"2024-09-21T13:54:16.678044+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"68'1","reported_seq":276,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.915460+0000","last_change":"2024-09-21T13:52:04.314151+0000","last_active":"2024-09-21T13:54:15.915460+0000","last_peered":"2024-09-21T13:54:15.915460+0000","last_clean":"2024-09-21T13:54:15.915460+0000","last_became_active":"2024-09-21T13:52:04.313210+0000","last_became_peered":"2024-09-21T13:52:04.313210+0000","last_unstale":"2024-09-21T13:54:15.915460+0000","last_undegraded":"2024-09-21T13:54:15.915460+0000","last_fullsized":"2024-09-21T13:54:15.915460+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T13:50:41.000201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:41.000201+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:43.256736+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.7","version":"187'58","reported_seq":77,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:18.589520+0000","last_change":"2024-09-21T13:54:12.376676+0000","last_active":"2024-09-21T13:54:18.589520+0000","last_peered":"2024-09-21T13:54:18.589520+0000","last_clean":"2024-09-21T13:54:18.589520+0000","last_became_active":"2024-09-21T13:54:12.376302+0000","last_became_peered":"2024-09-21T13:54:12.376302+0000","last_unstale":"2024-09-21T13:54:18.589520+0000","last_undegraded":"2024-09-21T13:54:18.589520+0000","last_fullsized":"2024-09-21T13:54:18.589520+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:33.120123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3801088,"num_objects":58,"num_object_clones":0,"num_object_copies":174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58,"num_write_kb":3712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"68'9","reported_seq":294,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:16.582755+0000","last_change":"2024-09-21T13:53:18.740202+0000","last_active":"2024-09-21T13:54:16.582755+0000","last_peered":"2024-09-21T13:54:16.582755+0000","last_clean":"2024-09-21T13:54:16.582755+0000","last_became_active":"2024-09-21T13:49:00.053016+0000","last_became_peered":"2024-09-21T13:49:00.053016+0000","last_unstale":"2024-09-21T13:54:16.582755+0000","last_undegraded":"2024-09-21T13:54:16.582755+0000","last_fullsized":"2024-09-21T13:54:16.582755+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:53:18.740111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:18.740111+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:34.752035+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":274,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:17.025324+0000","last_change":"2024-09-21T13:50:38.445732+0000","last_active":"2024-09-21T13:54:17.025324+0000","last_peered":"2024-09-21T13:54:17.025324+0000","last_clean":"2024-09-21T13:54:17.025324+0000","last_became_active":"2024-09-21T13:48:58.049543+0000","last_became_peered":"2024-09-21T13:48:58.049543+0000","last_unstale":"2024-09-21T13:54:17.025324+0000","last_undegraded":"2024-09-21T13:54:17.025324+0000","last_fullsized":"2024-09-21T13:54:17.025324+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:38.445667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:38.445667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:36.189674+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":195,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.915237+0000","last_change":"2024-09-21T13:52:59.112648+0000","last_active":"2024-09-21T13:54:15.915237+0000","last_peered":"2024-09-21T13:54:15.915237+0000","last_clean":"2024-09-21T13:54:15.915237+0000","last_became_active":"2024-09-21T13:52:04.317429+0000","last_became_peered":"2024-09-21T13:52:04.317429+0000","last_unstale":"2024-09-21T13:54:15.915237+0000","last_undegraded":"2024-09-21T13:54:15.915237+0000","last_fullsized":"2024-09-21T13:54:15.915237+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:59.112602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:52:59.112602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":216,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:16.024673+0000","last_change":"2024-09-21T13:52:51.590860+0000","last_active":"2024-09-21T13:54:16.024673+0000","last_peered":"2024-09-21T13:54:16.024673+0000","last_clean":"2024-09-21T13:54:16.024673+0000","last_became_active":"2024-09-21T13:52:04.321703+0000","last_became_peered":"2024-09-21T13:52:04.321703+0000","last_unstale":"2024-09-21T13:54:16.024673+0000","last_undegraded":"2024-09-21T13:54:16.024673+0000","last_fullsized":"2024-09-21T13:54:16.024673+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:51.590800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:51.590800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.6","version":"187'34","reported_seq":53,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:18.424818+0000","last_change":"2024-09-21T13:54:12.478610+0000","last_active":"2024-09-21T13:54:18.424818+0000","last_peered":"2024-09-21T13:54:18.424818+0000","last_clean":"2024-09-21T13:54:18.424818+0000","last_became_active":"2024-09-21T13:54:12.478187+0000","last_became_peered":"2024-09-21T13:54:12.478187+0000","last_unstale":"2024-09-21T13:54:18.424818+0000","last_undegraded":"2024-09-21T13:54:18.424818+0000","last_fullsized":"2024-09-21T13:54:18.424818+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:21.444672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2228224,"num_objects":34,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34,"num_write_kb":2176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"68'19","reported_seq":301,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:16.449519+0000","last_change":"2024-09-21T13:50:03.769686+0000","last_active":"2024-09-21T13:54:16.449519+0000","last_peered":"2024-09-21T13:54:16.449519+0000","last_clean":"2024-09-21T13:54:16.449519+0000","last_became_active":"2024-09-21T13:49:00.425955+0000","last_became_peered":"2024-09-21T13:49:00.425955+0000","last_unstale":"2024-09-21T13:54:16.449519+0000","last_undegraded":"2024-09-21T13:54:16.449519+0000","last_fullsized":"2024-09-21T13:54:16.449519+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-21T13:50:03.769613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:03.769613+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:36.254505+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":265,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:16.025601+0000","last_change":"2024-09-21T13:48:58.053940+0000","last_active":"2024-09-21T13:54:16.025601+0000","last_peered":"2024-09-21T13:54:16.025601+0000","last_clean":"2024-09-21T13:54:16.025601+0000","last_became_active":"2024-09-21T13:48:58.053641+0000","last_became_peered":"2024-09-21T13:48:58.053641+0000","last_unstale":"2024-09-21T13:54:16.025601+0000","last_undegraded":"2024-09-21T13:54:16.025601+0000","last_fullsized":"2024-09-21T13:54:16.025601+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:17.391569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":223,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:16.677767+0000","last_change":"2024-09-21T13:53:53.244317+0000","last_active":"2024-09-21T13:54:16.677767+0000","last_peered":"2024-09-21T13:54:16.677767+0000","last_clean":"2024-09-21T13:54:16.677767+0000","last_became_active":"2024-09-21T13:52:04.321146+0000","last_became_peered":"2024-09-21T13:52:04.321146+0000","last_unstale":"2024-09-21T13:54:16.677767+0000","last_undegraded":"2024-09-21T13:54:16.677767+0000","last_fullsized":"2024-09-21T13:54:16.677767+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:53.244131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:53:53.244131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:13.809484+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":280,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:16.025192+0000","last_change":"2024-09-21T13:54:03.688017+0000","last_active":"2024-09-21T13:54:16.025192+0000","last_peered":"2024-09-21T13:54:16.025192+0000","last_clean":"2024-09-21T13:54:16.025192+0000","last_became_active":"2024-09-21T13:52:04.322213+0000","last_became_peered":"2024-09-21T13:52:04.322213+0000","last_unstale":"2024-09-21T13:54:16.025192+0000","last_undegraded":"2024-09-21T13:54:16.025192+0000","last_fullsized":"2024-09-21T13:54:16.025192+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:03.687971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:54:03.687971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:28.966237+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"187'25","reported_seq":44,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:18.418358+0000","last_change":"2024-09-21T13:54:12.478639+0000","last_active":"2024-09-21T13:54:18.418358+0000","last_peered":"2024-09-21T13:54:18.418358+0000","last_clean":"2024-09-21T13:54:18.418358+0000","last_became_active":"2024-09-21T13:54:12.478330+0000","last_became_peered":"2024-09-21T13:54:12.478330+0000","last_unstale":"2024-09-21T13:54:18.418358+0000","last_undegraded":"2024-09-21T13:54:18.418358+0000","last_fullsized":"2024-09-21T13:54:18.418358+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:36.845419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"68'12","reported_seq":300,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.914920+0000","last_change":"2024-09-21T13:53:57.288526+0000","last_active":"2024-09-21T13:54:15.914920+0000","last_peered":"2024-09-21T13:54:15.914920+0000","last_clean":"2024-09-21T13:54:15.914920+0000","last_became_active":"2024-09-21T13:49:00.059720+0000","last_became_peered":"2024-09-21T13:49:00.059720+0000","last_unstale":"2024-09-21T13:54:15.914920+0000","last_undegraded":"2024-09-21T13:54:15.914920+0000","last_fullsized":"2024-09-21T13:54:15.914920+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'12","last_scrub_stamp":"2024-09-21T13:53:57.288451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:57.288451+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:08.532403+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":273,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:16.025807+0000","last_change":"2024-09-21T13:50:29.100595+0000","last_active":"2024-09-21T13:54:16.025807+0000","last_peered":"2024-09-21T13:54:16.025807+0000","last_clean":"2024-09-21T13:54:16.025807+0000","last_became_active":"2024-09-21T13:48:58.057641+0000","last_became_peered":"2024-09-21T13:48:58.057641+0000","last_unstale":"2024-09-21T13:54:16.025807+0000","last_undegraded":"2024-09-21T13:54:16.025807+0000","last_fullsized":"2024-09-21T13:54:16.025807+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:29.100547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:29.100547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:34.353972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":188,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:16.677523+0000","last_change":"2024-09-21T13:53:31.964122+0000","last_active":"2024-09-21T13:54:16.677523+0000","last_peered":"2024-09-21T13:54:16.677523+0000","last_clean":"2024-09-21T13:54:16.677523+0000","last_became_active":"2024-09-21T13:52:04.322147+0000","last_became_peered":"2024-09-21T13:52:04.322147+0000","last_unstale":"2024-09-21T13:54:16.677523+0000","last_undegraded":"2024-09-21T13:54:16.677523+0000","last_fullsized":"2024-09-21T13:54:16.677523+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:31.958946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:53:31.958946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:51.369874+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":261,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.916600+0000","last_change":"2024-09-21T13:53:20.888048+0000","last_active":"2024-09-21T13:54:15.916600+0000","last_peered":"2024-09-21T13:54:15.916600+0000","last_clean":"2024-09-21T13:54:15.916600+0000","last_became_active":"2024-09-21T13:49:04.078089+0000","last_became_peered":"2024-09-21T13:49:04.078089+0000","last_unstale":"2024-09-21T13:54:15.916600+0000","last_undegraded":"2024-09-21T13:54:15.916600+0000","last_fullsized":"2024-09-21T13:54:15.916600+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:20.882449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:20.882449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:45.947326+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"187'26","reported_seq":45,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:18.395859+0000","last_change":"2024-09-21T13:54:12.477912+0000","last_active":"2024-09-21T13:54:18.395859+0000","last_peered":"2024-09-21T13:54:18.395859+0000","last_clean":"2024-09-21T13:54:18.395859+0000","last_became_active":"2024-09-21T13:54:12.477455+0000","last_became_peered":"2024-09-21T13:54:12.477455+0000","last_unstale":"2024-09-21T13:54:18.395859+0000","last_undegraded":"2024-09-21T13:54:18.395859+0000","last_fullsized":"2024-09-21T13:54:18.395859+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:26.894106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1703936,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26,"num_write_kb":1664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"68'15","reported_seq":303,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:17.025093+0000","last_change":"2024-09-21T13:53:31.204731+0000","last_active":"2024-09-21T13:54:17.025093+0000","last_peered":"2024-09-21T13:54:17.025093+0000","last_clean":"2024-09-21T13:54:17.025093+0000","last_became_active":"2024-09-21T13:49:00.425997+0000","last_became_peered":"2024-09-21T13:49:00.425997+0000","last_unstale":"2024-09-21T13:54:17.025093+0000","last_undegraded":"2024-09-21T13:54:17.025093+0000","last_fullsized":"2024-09-21T13:54:17.025093+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:53:31.204637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:31.204637+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:41.385554+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":293,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:16.025516+0000","last_change":"2024-09-21T13:53:19.396344+0000","last_active":"2024-09-21T13:54:16.025516+0000","last_peered":"2024-09-21T13:54:16.025516+0000","last_clean":"2024-09-21T13:54:16.025516+0000","last_became_active":"2024-09-21T13:48:58.057632+0000","last_became_peered":"2024-09-21T13:48:58.057632+0000","last_unstale":"2024-09-21T13:54:16.025516+0000","last_undegraded":"2024-09-21T13:54:16.025516+0000","last_fullsized":"2024-09-21T13:54:16.025516+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T13:53:19.396277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:53:19.396277+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:47.867330+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"68'8","reported_seq":338,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:16.024844+0000","last_change":"2024-09-21T13:51:26.297643+0000","last_active":"2024-09-21T13:54:16.024844+0000","last_peered":"2024-09-21T13:54:16.024844+0000","last_clean":"2024-09-21T13:54:16.024844+0000","last_became_active":"2024-09-21T13:49:02.056167+0000","last_became_peered":"2024-09-21T13:49:02.056167+0000","last_unstale":"2024-09-21T13:54:16.024844+0000","last_undegraded":"2024-09-21T13:54:16.024844+0000","last_fullsized":"2024-09-21T13:54:16.024844+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:51:26.297550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:26.297550+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":270,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:17.025473+0000","last_change":"2024-09-21T13:52:54.314087+0000","last_active":"2024-09-21T13:54:17.025473+0000","last_peered":"2024-09-21T13:54:17.025473+0000","last_clean":"2024-09-21T13:54:17.025473+0000","last_became_active":"2024-09-21T13:49:04.432546+0000","last_became_peered":"2024-09-21T13:49:04.432546+0000","last_unstale":"2024-09-21T13:54:17.025473+0000","last_undegraded":"2024-09-21T13:54:17.025473+0000","last_fullsized":"2024-09-21T13:54:17.025473+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:54.314042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:54.314042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":290,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:18.131012+0000","last_change":"2024-09-21T13:54:18.131012+0000","last_active":"2024-09-21T13:54:18.131012+0000","last_peered":"2024-09-21T13:54:18.131012+0000","last_clean":"2024-09-21T13:54:18.131012+0000","last_became_active":"2024-09-21T13:48:58.056080+0000","last_became_peered":"2024-09-21T13:48:58.056080+0000","last_unstale":"2024-09-21T13:54:18.131012+0000","last_undegraded":"2024-09-21T13:54:18.131012+0000","last_fullsized":"2024-09-21T13:54:18.131012+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:18.130969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:18.130969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:34.242253+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"68'6","reported_seq":278,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.918815+0000","last_change":"2024-09-21T13:53:31.807215+0000","last_active":"2024-09-21T13:54:15.918815+0000","last_peered":"2024-09-21T13:54:15.918815+0000","last_clean":"2024-09-21T13:54:15.918815+0000","last_became_active":"2024-09-21T13:49:00.426829+0000","last_became_peered":"2024-09-21T13:49:00.426829+0000","last_unstale":"2024-09-21T13:54:15.918815+0000","last_undegraded":"2024-09-21T13:54:15.918815+0000","last_fullsized":"2024-09-21T13:54:15.918815+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'6","last_scrub_stamp":"2024-09-21T13:53:31.807028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:31.807028+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:49.612755+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":282,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:18.124591+0000","last_change":"2024-09-21T13:54:18.124591+0000","last_active":"2024-09-21T13:54:18.124591+0000","last_peered":"2024-09-21T13:54:18.124591+0000","last_clean":"2024-09-21T13:54:18.124591+0000","last_became_active":"2024-09-21T13:49:02.062231+0000","last_became_peered":"2024-09-21T13:49:02.062231+0000","last_unstale":"2024-09-21T13:54:18.124591+0000","last_undegraded":"2024-09-21T13:54:18.124591+0000","last_fullsized":"2024-09-21T13:54:18.124591+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:18.124544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:18.124544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:39.080321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":269,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:16.025014+0000","last_change":"2024-09-21T13:53:18.373418+0000","last_active":"2024-09-21T13:54:16.025014+0000","last_peered":"2024-09-21T13:54:16.025014+0000","last_clean":"2024-09-21T13:54:16.025014+0000","last_became_active":"2024-09-21T13:49:04.432507+0000","last_became_peered":"2024-09-21T13:49:04.432507+0000","last_unstale":"2024-09-21T13:54:16.025014+0000","last_undegraded":"2024-09-21T13:54:16.025014+0000","last_fullsized":"2024-09-21T13:54:16.025014+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:18.373356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:18.373356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:38.812691+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"68'9","reported_seq":241,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.918058+0000","last_change":"2024-09-21T13:51:54.808758+0000","last_active":"2024-09-21T13:54:15.918058+0000","last_peered":"2024-09-21T13:54:15.918058+0000","last_clean":"2024-09-21T13:54:15.918058+0000","last_became_active":"2024-09-21T13:50:29.693589+0000","last_became_peered":"2024-09-21T13:50:29.693589+0000","last_unstale":"2024-09-21T13:54:15.918058+0000","last_undegraded":"2024-09-21T13:54:15.918058+0000","last_fullsized":"2024-09-21T13:54:15.918058+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:51:54.808559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:54.808559+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"61'1","reported_seq":271,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:16.025366+0000","last_change":"2024-09-21T13:48:58.050948+0000","last_active":"2024-09-21T13:54:16.025366+0000","last_peered":"2024-09-21T13:54:16.025366+0000","last_clean":"2024-09-21T13:54:16.025366+0000","last_became_active":"2024-09-21T13:48:58.050645+0000","last_became_peered":"2024-09-21T13:48:58.050645+0000","last_unstale":"2024-09-21T13:54:16.025366+0000","last_undegraded":"2024-09-21T13:54:16.025366+0000","last_fullsized":"2024-09-21T13:54:16.025366+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":265,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.918541+0000","last_change":"2024-09-21T13:50:43.375425+0000","last_active":"2024-09-21T13:54:15.918541+0000","last_peered":"2024-09-21T13:54:15.918541+0000","last_clean":"2024-09-21T13:54:15.918541+0000","last_became_active":"2024-09-21T13:49:02.061775+0000","last_became_peered":"2024-09-21T13:49:02.061775+0000","last_unstale":"2024-09-21T13:54:15.918541+0000","last_undegraded":"2024-09-21T13:54:15.918541+0000","last_fullsized":"2024-09-21T13:54:15.918541+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:43.375381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:43.375381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":210,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:16.677792+0000","last_change":"2024-09-21T13:52:04.324521+0000","last_active":"2024-09-21T13:54:16.677792+0000","last_peered":"2024-09-21T13:54:16.677792+0000","last_clean":"2024-09-21T13:54:16.677792+0000","last_became_active":"2024-09-21T13:52:04.323549+0000","last_became_peered":"2024-09-21T13:52:04.323549+0000","last_unstale":"2024-09-21T13:54:16.677792+0000","last_undegraded":"2024-09-21T13:54:16.677792+0000","last_fullsized":"2024-09-21T13:54:16.677792+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:41.808746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:41.808746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:34.950671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"68'9","reported_seq":286,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:17.025723+0000","last_change":"2024-09-21T13:51:28.547052+0000","last_active":"2024-09-21T13:54:17.025723+0000","last_peered":"2024-09-21T13:54:17.025723+0000","last_clean":"2024-09-21T13:54:17.025723+0000","last_became_active":"2024-09-21T13:49:00.427766+0000","last_became_peered":"2024-09-21T13:49:00.427766+0000","last_unstale":"2024-09-21T13:54:17.025723+0000","last_undegraded":"2024-09-21T13:54:17.025723+0000","last_fullsized":"2024-09-21T13:54:17.025723+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:51:28.546958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:28.546958+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:09.056462+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":213,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:16.025101+0000","last_change":"2024-09-21T13:54:02.683225+0000","last_active":"2024-09-21T13:54:16.025101+0000","last_peered":"2024-09-21T13:54:16.025101+0000","last_clean":"2024-09-21T13:54:16.025101+0000","last_became_active":"2024-09-21T13:50:29.690634+0000","last_became_peered":"2024-09-21T13:50:29.690634+0000","last_unstale":"2024-09-21T13:54:16.025101+0000","last_undegraded":"2024-09-21T13:54:16.025101+0000","last_fullsized":"2024-09-21T13:54:16.025101+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:02.683172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:02.683172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:13.538040+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"132'10","reported_seq":351,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.916175+0000","last_change":"2024-09-21T13:52:04.319377+0000","last_active":"2024-09-21T13:54:15.916175+0000","last_peered":"2024-09-21T13:54:15.916175+0000","last_clean":"2024-09-21T13:54:15.916175+0000","last_became_active":"2024-09-21T13:52:04.318626+0000","last_became_peered":"2024-09-21T13:52:04.318626+0000","last_unstale":"2024-09-21T13:54:15.916175+0000","last_undegraded":"2024-09-21T13:54:15.916175+0000","last_fullsized":"2024-09-21T13:54:15.916175+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:50:29.249680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:29.249680+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":280,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.915734+0000","last_change":"2024-09-21T13:53:51.285122+0000","last_active":"2024-09-21T13:54:15.915734+0000","last_peered":"2024-09-21T13:54:15.915734+0000","last_clean":"2024-09-21T13:54:15.915734+0000","last_became_active":"2024-09-21T13:52:04.315865+0000","last_became_peered":"2024-09-21T13:52:04.315865+0000","last_unstale":"2024-09-21T13:54:15.915734+0000","last_undegraded":"2024-09-21T13:54:15.915734+0000","last_fullsized":"2024-09-21T13:54:15.915734+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:51.285077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:51.285077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:16.987394+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"68'10","reported_seq":284,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.916351+0000","last_change":"2024-09-21T13:51:11.545546+0000","last_active":"2024-09-21T13:54:15.916351+0000","last_peered":"2024-09-21T13:54:15.916351+0000","last_clean":"2024-09-21T13:54:15.916351+0000","last_became_active":"2024-09-21T13:49:00.426071+0000","last_became_peered":"2024-09-21T13:49:00.426071+0000","last_unstale":"2024-09-21T13:54:15.916351+0000","last_undegraded":"2024-09-21T13:54:15.916351+0000","last_fullsized":"2024-09-21T13:54:15.916351+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:51:11.545458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:11.545458+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":284,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:16.025157+0000","last_change":"2024-09-21T13:52:04.318335+0000","last_active":"2024-09-21T13:54:16.025157+0000","last_peered":"2024-09-21T13:54:16.025157+0000","last_clean":"2024-09-21T13:54:16.025157+0000","last_became_active":"2024-09-21T13:52:04.318006+0000","last_became_peered":"2024-09-21T13:52:04.318006+0000","last_unstale":"2024-09-21T13:54:16.025157+0000","last_undegraded":"2024-09-21T13:54:16.025157+0000","last_fullsized":"2024-09-21T13:54:16.025157+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:41.162315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:41.162315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:42.458106+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"68'8","reported_seq":339,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:17.025421+0000","last_change":"2024-09-21T13:54:03.065465+0000","last_active":"2024-09-21T13:54:17.025421+0000","last_peered":"2024-09-21T13:54:17.025421+0000","last_clean":"2024-09-21T13:54:17.025421+0000","last_became_active":"2024-09-21T13:49:02.067487+0000","last_became_peered":"2024-09-21T13:49:02.067487+0000","last_unstale":"2024-09-21T13:54:17.025421+0000","last_undegraded":"2024-09-21T13:54:17.025421+0000","last_fullsized":"2024-09-21T13:54:17.025421+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:54:03.065395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:03.065395+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:30.943315+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":205,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:16.025408+0000","last_change":"2024-09-21T13:50:33.050276+0000","last_active":"2024-09-21T13:54:16.025408+0000","last_peered":"2024-09-21T13:54:16.025408+0000","last_clean":"2024-09-21T13:54:16.025408+0000","last_became_active":"2024-09-21T13:50:29.695082+0000","last_became_peered":"2024-09-21T13:50:29.695082+0000","last_unstale":"2024-09-21T13:54:16.025408+0000","last_undegraded":"2024-09-21T13:54:16.025408+0000","last_fullsized":"2024-09-21T13:54:16.025408+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:33.050227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:33.050227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"68'11","reported_seq":296,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.914909+0000","last_change":"2024-09-21T13:53:26.146937+0000","last_active":"2024-09-21T13:54:15.914909+0000","last_peered":"2024-09-21T13:54:15.914909+0000","last_clean":"2024-09-21T13:54:15.914909+0000","last_became_active":"2024-09-21T13:49:00.055024+0000","last_became_peered":"2024-09-21T13:49:00.055024+0000","last_unstale":"2024-09-21T13:54:15.914909+0000","last_undegraded":"2024-09-21T13:54:15.914909+0000","last_fullsized":"2024-09-21T13:54:15.914909+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:53:26.146725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:26.146725+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:44.146330+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":28,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:16.025715+0000","last_change":"2024-09-21T13:54:04.377034+0000","last_active":"2024-09-21T13:54:16.025715+0000","last_peered":"2024-09-21T13:54:16.025715+0000","last_clean":"2024-09-21T13:54:16.025715+0000","last_became_active":"2024-09-21T13:54:04.376614+0000","last_became_peered":"2024-09-21T13:54:04.376614+0000","last_unstale":"2024-09-21T13:54:16.025715+0000","last_undegraded":"2024-09-21T13:54:16.025715+0000","last_fullsized":"2024-09-21T13:54:16.025715+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:01.228213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:01.228213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:14.625514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":282,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.915446+0000","last_change":"2024-09-21T13:53:29.868061+0000","last_active":"2024-09-21T13:54:15.915446+0000","last_peered":"2024-09-21T13:54:15.915446+0000","last_clean":"2024-09-21T13:54:15.915446+0000","last_became_active":"2024-09-21T13:49:02.062868+0000","last_became_peered":"2024-09-21T13:49:02.062868+0000","last_unstale":"2024-09-21T13:54:15.915446+0000","last_undegraded":"2024-09-21T13:54:15.915446+0000","last_fullsized":"2024-09-21T13:54:15.915446+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:29.867278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:53:29.867278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:49.405014+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":261,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.914812+0000","last_change":"2024-09-21T13:51:24.549993+0000","last_active":"2024-09-21T13:54:15.914812+0000","last_peered":"2024-09-21T13:54:15.914812+0000","last_clean":"2024-09-21T13:54:15.914812+0000","last_became_active":"2024-09-21T13:49:04.082961+0000","last_became_peered":"2024-09-21T13:49:04.082961+0000","last_unstale":"2024-09-21T13:54:15.914812+0000","last_undegraded":"2024-09-21T13:54:15.914812+0000","last_fullsized":"2024-09-21T13:54:15.914812+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:24.549706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:51:24.549706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"68'9","reported_seq":297,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.915653+0000","last_change":"2024-09-21T13:52:10.447789+0000","last_active":"2024-09-21T13:54:15.915653+0000","last_peered":"2024-09-21T13:54:15.915653+0000","last_clean":"2024-09-21T13:54:15.915653+0000","last_became_active":"2024-09-21T13:52:04.320149+0000","last_became_peered":"2024-09-21T13:52:04.320149+0000","last_unstale":"2024-09-21T13:54:15.915653+0000","last_undegraded":"2024-09-21T13:54:15.915653+0000","last_fullsized":"2024-09-21T13:54:15.915653+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:52:10.447530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:52:10.447530+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":273,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:16.025432+0000","last_change":"2024-09-21T13:54:06.687343+0000","last_active":"2024-09-21T13:54:16.025432+0000","last_peered":"2024-09-21T13:54:16.025432+0000","last_clean":"2024-09-21T13:54:16.025432+0000","last_became_active":"2024-09-21T13:48:58.057516+0000","last_became_peered":"2024-09-21T13:48:58.057516+0000","last_unstale":"2024-09-21T13:54:16.025432+0000","last_undegraded":"2024-09-21T13:54:16.025432+0000","last_fullsized":"2024-09-21T13:54:16.025432+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:06.687302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:06.687302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:28.415240+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":265,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.916712+0000","last_change":"2024-09-21T13:50:39.411785+0000","last_active":"2024-09-21T13:54:15.916712+0000","last_peered":"2024-09-21T13:54:15.916712+0000","last_clean":"2024-09-21T13:54:15.916712+0000","last_became_active":"2024-09-21T13:49:02.071922+0000","last_became_peered":"2024-09-21T13:49:02.071922+0000","last_unstale":"2024-09-21T13:54:15.916712+0000","last_undegraded":"2024-09-21T13:54:15.916712+0000","last_fullsized":"2024-09-21T13:54:15.916712+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:39.411718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:39.411718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":198,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:17.025002+0000","last_change":"2024-09-21T13:50:29.689537+0000","last_active":"2024-09-21T13:54:17.025002+0000","last_peered":"2024-09-21T13:54:17.025002+0000","last_clean":"2024-09-21T13:54:17.025002+0000","last_became_active":"2024-09-21T13:50:29.689045+0000","last_became_peered":"2024-09-21T13:50:29.689045+0000","last_unstale":"2024-09-21T13:54:17.025002+0000","last_undegraded":"2024-09-21T13:54:17.025002+0000","last_fullsized":"2024-09-21T13:54:17.025002+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:14.052362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"68'11","reported_seq":288,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.914828+0000","last_change":"2024-09-21T13:50:31.329973+0000","last_active":"2024-09-21T13:54:15.914828+0000","last_peered":"2024-09-21T13:54:15.914828+0000","last_clean":"2024-09-21T13:54:15.914828+0000","last_became_active":"2024-09-21T13:49:00.426091+0000","last_became_peered":"2024-09-21T13:49:00.426091+0000","last_unstale":"2024-09-21T13:54:15.914828+0000","last_undegraded":"2024-09-21T13:54:15.914828+0000","last_fullsized":"2024-09-21T13:54:15.914828+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:50:31.329740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:31.329740+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":266,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:17.025799+0000","last_change":"2024-09-21T13:48:58.052281+0000","last_active":"2024-09-21T13:54:17.025799+0000","last_peered":"2024-09-21T13:54:17.025799+0000","last_clean":"2024-09-21T13:54:17.025799+0000","last_became_active":"2024-09-21T13:48:58.051974+0000","last_became_peered":"2024-09-21T13:48:58.051974+0000","last_unstale":"2024-09-21T13:54:17.025799+0000","last_undegraded":"2024-09-21T13:54:17.025799+0000","last_fullsized":"2024-09-21T13:54:17.025799+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:08.067332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":265,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:16.024568+0000","last_change":"2024-09-21T13:50:37.118637+0000","last_active":"2024-09-21T13:54:16.024568+0000","last_peered":"2024-09-21T13:54:16.024568+0000","last_clean":"2024-09-21T13:54:16.024568+0000","last_became_active":"2024-09-21T13:49:02.056006+0000","last_became_peered":"2024-09-21T13:49:02.056006+0000","last_unstale":"2024-09-21T13:54:16.024568+0000","last_undegraded":"2024-09-21T13:54:16.024568+0000","last_fullsized":"2024-09-21T13:54:16.024568+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:37.118594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:37.118594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:12.668029+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":261,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.916926+0000","last_change":"2024-09-21T13:51:55.827109+0000","last_active":"2024-09-21T13:54:15.916926+0000","last_peered":"2024-09-21T13:54:15.916926+0000","last_clean":"2024-09-21T13:54:15.916926+0000","last_became_active":"2024-09-21T13:49:04.434034+0000","last_became_peered":"2024-09-21T13:49:04.434034+0000","last_unstale":"2024-09-21T13:54:15.916926+0000","last_undegraded":"2024-09-21T13:54:15.916926+0000","last_fullsized":"2024-09-21T13:54:15.916926+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:55.826922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:51:55.826922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"68'4","reported_seq":275,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.915150+0000","last_change":"2024-09-21T13:53:27.826804+0000","last_active":"2024-09-21T13:54:15.915150+0000","last_peered":"2024-09-21T13:54:15.915150+0000","last_clean":"2024-09-21T13:54:15.915150+0000","last_became_active":"2024-09-21T13:49:00.063066+0000","last_became_peered":"2024-09-21T13:49:00.063066+0000","last_unstale":"2024-09-21T13:54:15.915150+0000","last_undegraded":"2024-09-21T13:54:15.915150+0000","last_fullsized":"2024-09-21T13:54:15.915150+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4","last_scrub_stamp":"2024-09-21T13:53:27.826729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:27.826729+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:54.701343+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":206,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:17.026028+0000","last_change":"2024-09-21T13:53:32.253481+0000","last_active":"2024-09-21T13:54:17.026028+0000","last_peered":"2024-09-21T13:54:17.026028+0000","last_clean":"2024-09-21T13:54:17.026028+0000","last_became_active":"2024-09-21T13:50:29.690269+0000","last_became_peered":"2024-09-21T13:50:29.690269+0000","last_unstale":"2024-09-21T13:54:17.026028+0000","last_undegraded":"2024-09-21T13:54:17.026028+0000","last_fullsized":"2024-09-21T13:54:17.026028+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:32.253408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:53:32.253408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:51.887076+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":282,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:16.449444+0000","last_change":"2024-09-21T13:53:51.254522+0000","last_active":"2024-09-21T13:54:16.449444+0000","last_peered":"2024-09-21T13:54:16.449444+0000","last_clean":"2024-09-21T13:54:16.449444+0000","last_became_active":"2024-09-21T13:49:02.064789+0000","last_became_peered":"2024-09-21T13:49:02.064789+0000","last_unstale":"2024-09-21T13:54:16.449444+0000","last_undegraded":"2024-09-21T13:54:16.449444+0000","last_fullsized":"2024-09-21T13:54:16.449444+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:51.254456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:53:51.254456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:51.405316+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":272,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:16.024955+0000","last_change":"2024-09-21T13:52:04.322055+0000","last_active":"2024-09-21T13:54:16.024955+0000","last_peered":"2024-09-21T13:54:16.024955+0000","last_clean":"2024-09-21T13:54:16.024955+0000","last_became_active":"2024-09-21T13:52:04.321667+0000","last_became_peered":"2024-09-21T13:52:04.321667+0000","last_unstale":"2024-09-21T13:54:16.024955+0000","last_undegraded":"2024-09-21T13:54:16.024955+0000","last_fullsized":"2024-09-21T13:54:16.024955+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:39.151320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:39.151320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:07.711986+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":270,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.916039+0000","last_change":"2024-09-21T13:52:27.409588+0000","last_active":"2024-09-21T13:54:15.916039+0000","last_peered":"2024-09-21T13:54:15.916039+0000","last_clean":"2024-09-21T13:54:15.916039+0000","last_became_active":"2024-09-21T13:49:04.075562+0000","last_became_peered":"2024-09-21T13:49:04.075562+0000","last_unstale":"2024-09-21T13:54:15.916039+0000","last_undegraded":"2024-09-21T13:54:15.916039+0000","last_fullsized":"2024-09-21T13:54:15.916039+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:27.409408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:27.409408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":273,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:15.915362+0000","last_change":"2024-09-21T13:53:25.873998+0000","last_active":"2024-09-21T13:54:15.915362+0000","last_peered":"2024-09-21T13:54:15.915362+0000","last_clean":"2024-09-21T13:54:15.915362+0000","last_became_active":"2024-09-21T13:48:58.055497+0000","last_became_peered":"2024-09-21T13:48:58.055497+0000","last_unstale":"2024-09-21T13:54:15.915362+0000","last_undegraded":"2024-09-21T13:54:15.915362+0000","last_fullsized":"2024-09-21T13:54:15.915362+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:25.865470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:53:25.865470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:51.548858+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"68'11","reported_seq":319,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:16.449967+0000","last_change":"2024-09-21T13:52:04.321178+0000","last_active":"2024-09-21T13:54:16.449967+0000","last_peered":"2024-09-21T13:54:16.449967+0000","last_clean":"2024-09-21T13:54:16.449967+0000","last_became_active":"2024-09-21T13:52:04.320647+0000","last_became_peered":"2024-09-21T13:52:04.320647+0000","last_unstale":"2024-09-21T13:54:16.449967+0000","last_undegraded":"2024-09-21T13:54:16.449967+0000","last_fullsized":"2024-09-21T13:54:16.449967+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:50:35.790090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:35.790090+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:11.179000+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":224,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T13:54:16.025702+0000","last_change":"2024-09-21T13:54:01.727728+0000","last_active":"2024-09-21T13:54:16.025702+0000","last_peered":"2024-09-21T13:54:16.025702+0000","last_clean":"2024-09-21T13:54:16.025702+0000","last_became_active":"2024-09-21T13:52:04.322880+0000","last_became_peered":"2024-09-21T13:52:04.322880+0000","last_unstale":"2024-09-21T13:54:16.025702+0000","last_undegraded":"2024-09-21T13:54:16.025702+0000","last_fullsized":"2024-09-21T13:54:16.025702+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:01.727680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:01.727680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:16.479686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":71761920,"num_objects":1095,"num_object_clones":0,"num_object_copies":3285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1095,"num_write_kb":70080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174063616,"data_stored":174063616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1095,"ondisk_log_size":1095,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69,"ondisk_log_size":69,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":694,"num_read_kb":451,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":395,"ondisk_log_size":395,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":315,"num_read_kb":310,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234053,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":127492,"kb_used_data":1904,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":93617660,"statfs":{"total":95995035648,"available":95864483840,"internally_reserved":0,"allocated":1949696,"data_stored":1447338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":128580042},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462986,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":123316,"kb_used_data":1884,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93621836,"statfs":{"total":95995035648,"available":95868760064,"internally_reserved":0,"allocated":1929216,"data_stored":1446253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":124320198},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724625,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":126900,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":93618252,"statfs":{"total":95995035648,"available":95865090048,"internally_reserved":0,"allocated":1339392,"data_stored":856801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920855,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":134408,"kb_used_data":8824,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":93610744,"statfs":{"total":95995035648,"available":95857401856,"internally_reserved":0,"allocated":9035776,"data_stored":8524215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117084,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":282444,"kb_used_data":58300,"kb_used_omap":1,"kb_used_meta":224126,"kb_avail":93462708,"statfs":{"total":95995035648,"available":95705812992,"internally_reserved":0,"allocated":59699200,"data_stored":59184373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":229505479},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":130,"seq":558345748508,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":218168,"kb_used_data":1312,"kb_used_omap":4,"kb_used_meta":216827,"kb_avail":93526984,"statfs":{"total":95995035648,"available":95771631616,"internally_reserved":0,"allocated":1343488,"data_stored":856823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4441,"internal_metadata":222031527},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574953,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":149152,"kb_used_data":1360,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":93596000,"statfs":{"total":95995035648,"available":95842304000,"internally_reserved":0,"allocated":1392640,"data_stored":857027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738479,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":209280,"kb_used_data":37872,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":93535872,"statfs":{"total":95995035648,"available":95780732928,"internally_reserved":0,"allocated":38780928,"data_stored":38277843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":175503813},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40828928,"data_stored":40828928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":30081024,"data_stored":30081024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61669376,"data_stored":61669376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8585216,"data_stored":8585216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":11665408,"data_stored":11665408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":9502720,"data_stored":9502720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":11730944,"data_stored":11730944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T13:54:21.253 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj -9, currently 1 2024-09-21T13:54:21.286 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 3 16 9262 9246 192.584 181.938 0.00330224 0.00518253 2024-09-21T13:54:21.286 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T13:54:21.287 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-21T13:54:21.287 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-21T13:54:21.287 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-21T13:54:21.287 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.287 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:35249 7.6 7:6f97b9cd:::benchmark_data_smithi049_76798_object35248:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T13:54:21.287 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:28.696132+0000", 2024-09-21T13:54:21.287 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 232.58713615100001, 2024-09-21T13:54:21.287 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0107570610000001, 2024-09-21T13:54:21.287 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:54:21.287 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:54:21.287 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:21.287 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:54:21.287 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:54:21.288 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:21.288 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 35249 2024-09-21T13:54:21.288 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.288 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:54:21.288 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.288 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:54:21.288 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.696132+0000", 2024-09-21T13:54:21.288 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:21.288 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.288 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.288 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:54:21.288 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.696132+0000", 2024-09-21T13:54:21.288 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:21.288 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.289 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.289 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:54:21.289 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.696134+0000", 2024-09-21T13:54:21.289 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6080000000000001e-06 2024-09-21T13:54:21.289 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.289 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.289 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:54:21.289 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.696185+0000", 2024-09-21T13:54:21.289 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1328000000000001e-05 2024-09-21T13:54:21.289 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.290 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.290 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:54:21.290 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.696202+0000", 2024-09-21T13:54:21.290 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7334999999999999e-05 2024-09-21T13:54:21.290 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.290 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.291 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:54:21.291 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.696212+0000", 2024-09-21T13:54:21.291 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3489999999999998e-06 2024-09-21T13:54:21.291 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.291 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.291 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:21.291 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.696291+0000", 2024-09-21T13:54:21.292 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9078000000000004e-05 2024-09-21T13:54:21.292 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.292 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.292 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:54:21.292 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.696301+0000", 2024-09-21T13:54:21.292 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7780000000000002e-06 2024-09-21T13:54:21.293 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.293 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.293 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:21.293 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.697326+0000", 2024-09-21T13:54:21.293 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0010256289999999 2024-09-21T13:54:21.293 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.294 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.294 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:54:21.294 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.698023+0000", 2024-09-21T13:54:21.294 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00069705799999999997 2024-09-21T13:54:21.294 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.295 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.295 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3", 2024-09-21T13:54:21.295 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.698222+0000", 2024-09-21T13:54:21.295 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000198937 2024-09-21T13:54:21.295 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.295 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.295 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:54:21.296 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.703079+0000", 2024-09-21T13:54:21.296 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0048563690000000001 2024-09-21T13:54:21.297 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.297 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.298 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:21.298 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.706547+0000", 2024-09-21T13:54:21.298 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0034680499999999999 2024-09-21T13:54:21.298 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.298 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.299 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:21.299 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.706817+0000", 2024-09-21T13:54:21.299 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00026996700000000003 2024-09-21T13:54:21.299 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.299 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.299 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:54:21.299 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.706866+0000", 2024-09-21T13:54:21.300 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9292999999999997e-05 2024-09-21T13:54:21.300 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.300 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.300 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:54:21.300 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.706889+0000", 2024-09-21T13:54:21.300 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3281999999999998e-05 2024-09-21T13:54:21.301 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:21.301 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:54:21.301 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:21.301 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.301 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.301 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:35258 7.6 7:6cec60f9:::benchmark_data_smithi049_76798_object35257:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T13:54:21.302 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:28.700988+0000", 2024-09-21T13:54:21.302 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 232.58228041999999, 2024-09-21T13:54:21.302 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.007550977, 2024-09-21T13:54:21.302 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:54:21.302 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:54:21.302 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:21.303 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:54:21.303 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:54:21.303 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:21.303 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 35258 2024-09-21T13:54:21.303 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.303 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:54:21.304 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.304 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:54:21.304 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.700988+0000", 2024-09-21T13:54:21.304 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:21.304 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.304 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.304 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:54:21.305 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.700988+0000", 2024-09-21T13:54:21.305 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:21.305 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.305 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.305 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:54:21.305 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.700990+0000", 2024-09-21T13:54:21.305 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.795e-06 2024-09-21T13:54:21.306 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.306 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.306 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:54:21.306 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.701031+0000", 2024-09-21T13:54:21.306 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0836e-05 2024-09-21T13:54:21.306 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.307 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.307 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:54:21.307 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.701045+0000", 2024-09-21T13:54:21.307 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4178e-05 2024-09-21T13:54:21.307 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.307 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.308 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:54:21.308 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.701053+0000", 2024-09-21T13:54:21.308 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3499999999999997e-06 2024-09-21T13:54:21.308 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.308 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.308 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:21.309 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.701113+0000", 2024-09-21T13:54:21.309 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9857e-05 2024-09-21T13:54:21.309 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.309 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.309 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:54:21.309 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.701123+0000", 2024-09-21T13:54:21.310 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0202e-05 2024-09-21T13:54:21.310 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.310 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.310 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:21.310 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.700130+0000", 2024-09-21T13:54:21.310 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.99900719299999996 2024-09-21T13:54:21.311 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.311 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.311 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:54:21.311 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.700245+0000", 2024-09-21T13:54:21.311 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000115047 2024-09-21T13:54:21.311 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.312 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.312 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3", 2024-09-21T13:54:21.312 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.700392+0000", 2024-09-21T13:54:21.312 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000146358 2024-09-21T13:54:21.312 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.312 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.313 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:54:21.313 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.704100+0000", 2024-09-21T13:54:21.313 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0037085780000000001 2024-09-21T13:54:21.313 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.313 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.313 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:21.314 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.706575+0000", 2024-09-21T13:54:21.314 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.002474642 2024-09-21T13:54:21.314 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.314 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.314 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:21.315 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.708441+0000", 2024-09-21T13:54:21.315 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00186568 2024-09-21T13:54:21.315 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.315 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.315 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:54:21.315 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.708496+0000", 2024-09-21T13:54:21.316 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5340000000000002e-05 2024-09-21T13:54:21.317 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.317 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.317 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:54:21.317 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.708539+0000", 2024-09-21T13:54:21.317 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2920999999999998e-05 2024-09-21T13:54:21.317 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:21.317 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:54:21.317 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:21.317 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.317 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.317 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:35272 7.6 7:61fa97ba:::benchmark_data_smithi049_76798_object35271:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T13:54:21.318 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:28.712676+0000", 2024-09-21T13:54:21.318 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 232.57059212499999, 2024-09-21T13:54:21.318 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.99907040300000005, 2024-09-21T13:54:21.318 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:54:21.318 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:54:21.318 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:21.318 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:54:21.318 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:54:21.318 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:21.319 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 35272 2024-09-21T13:54:21.319 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.319 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:54:21.319 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.319 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:54:21.319 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.712676+0000", 2024-09-21T13:54:21.319 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:21.319 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.319 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.319 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:54:21.319 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.712676+0000", 2024-09-21T13:54:21.319 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:21.320 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.320 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.320 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:54:21.320 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.712678+0000", 2024-09-21T13:54:21.320 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.401e-06 2024-09-21T13:54:21.320 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.320 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.320 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:54:21.320 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.712724+0000", 2024-09-21T13:54:21.320 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6576999999999998e-05 2024-09-21T13:54:21.320 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.320 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.320 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:54:21.321 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.712736+0000", 2024-09-21T13:54:21.321 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2231e-05 2024-09-21T13:54:21.321 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.321 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.321 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:54:21.321 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.712742+0000", 2024-09-21T13:54:21.321 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9649999999999996e-06 2024-09-21T13:54:21.321 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.321 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.321 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:21.321 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.712819+0000", 2024-09-21T13:54:21.321 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6258000000000001e-05 2024-09-21T13:54:21.321 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.321 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.321 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:54:21.322 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.712831+0000", 2024-09-21T13:54:21.322 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.275e-05 2024-09-21T13:54:21.322 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.322 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.322 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:21.322 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.701685+0000", 2024-09-21T13:54:21.322 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.98885339699999997 2024-09-21T13:54:21.322 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.322 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.322 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:54:21.322 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.701779+0000", 2024-09-21T13:54:21.322 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3944999999999996e-05 2024-09-21T13:54:21.323 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.323 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.323 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3", 2024-09-21T13:54:21.323 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.701912+0000", 2024-09-21T13:54:21.323 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000133156 2024-09-21T13:54:21.323 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.323 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.323 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:54:21.323 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.705423+0000", 2024-09-21T13:54:21.323 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003511587 2024-09-21T13:54:21.323 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.323 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.323 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:21.324 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.708588+0000", 2024-09-21T13:54:21.324 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0031643309999999998 2024-09-21T13:54:21.324 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.324 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.324 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:21.324 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.711670+0000", 2024-09-21T13:54:21.324 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003081748 2024-09-21T13:54:21.324 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.324 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.324 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:54:21.324 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.711714+0000", 2024-09-21T13:54:21.324 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4406999999999997e-05 2024-09-21T13:54:21.324 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.324 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.325 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:54:21.325 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.711747+0000", 2024-09-21T13:54:21.325 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2650000000000001e-05 2024-09-21T13:54:21.325 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:21.325 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:54:21.325 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:21.325 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.325 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.325 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:35278 7.6 7:6d0c33e9:::benchmark_data_smithi049_76798_object35277:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T13:54:21.325 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:28.718784+0000", 2024-09-21T13:54:21.325 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 232.56448396499999, 2024-09-21T13:54:21.325 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.993077243, 2024-09-21T13:54:21.325 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:54:21.325 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:54:21.325 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:21.326 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:54:21.326 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:54:21.326 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:21.326 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 35278 2024-09-21T13:54:21.326 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.326 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:54:21.326 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.326 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:54:21.326 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.718784+0000", 2024-09-21T13:54:21.326 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:21.326 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.326 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.327 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:54:21.327 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.718784+0000", 2024-09-21T13:54:21.327 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:21.327 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.327 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.327 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:54:21.327 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.718787+0000", 2024-09-21T13:54:21.327 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1849999999999998e-06 2024-09-21T13:54:21.327 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.327 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.327 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:54:21.327 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.718859+0000", 2024-09-21T13:54:21.327 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2344999999999999e-05 2024-09-21T13:54:21.327 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.328 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.328 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:54:21.328 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.718872+0000", 2024-09-21T13:54:21.328 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3113e-05 2024-09-21T13:54:21.328 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.328 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.328 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:54:21.328 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.718881+0000", 2024-09-21T13:54:21.328 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.448e-06 2024-09-21T13:54:21.328 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.328 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.328 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:21.328 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.718941+0000", 2024-09-21T13:54:21.328 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9580999999999998e-05 2024-09-21T13:54:21.328 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.329 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.329 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:54:21.329 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.718948+0000", 2024-09-21T13:54:21.329 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.391e-06 2024-09-21T13:54:21.329 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.329 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.329 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:21.329 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.703121+0000", 2024-09-21T13:54:21.329 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.98417250099999998 2024-09-21T13:54:21.329 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.329 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.329 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:54:21.329 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.703248+0000", 2024-09-21T13:54:21.329 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012707800000000001 2024-09-21T13:54:21.330 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.330 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.330 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3", 2024-09-21T13:54:21.330 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.703345+0000", 2024-09-21T13:54:21.330 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7287999999999994e-05 2024-09-21T13:54:21.330 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.330 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.330 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:54:21.330 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.705432+0000", 2024-09-21T13:54:21.330 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0020863499999999998 2024-09-21T13:54:21.330 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.330 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.330 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:21.331 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.708649+0000", 2024-09-21T13:54:21.331 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003217443 2024-09-21T13:54:21.331 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.331 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.331 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:21.331 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.711798+0000", 2024-09-21T13:54:21.331 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0031489809999999999 2024-09-21T13:54:21.331 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.331 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.331 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:54:21.331 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.711832+0000", 2024-09-21T13:54:21.331 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3707000000000001e-05 2024-09-21T13:54:21.331 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.331 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.331 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:54:21.331 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.711862+0000", 2024-09-21T13:54:21.332 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9832000000000002e-05 2024-09-21T13:54:21.332 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:21.332 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:54:21.332 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:21.332 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.332 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.332 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:35280 7.6 7:68737c4e:::benchmark_data_smithi049_76798_object35279:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T13:54:21.332 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:28.719589+0000", 2024-09-21T13:54:21.332 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 232.563679503, 2024-09-21T13:54:21.332 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.99249087499999999, 2024-09-21T13:54:21.332 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:54:21.332 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:54:21.332 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:21.332 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:54:21.333 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:54:21.333 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:21.333 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 35280 2024-09-21T13:54:21.333 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.333 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:54:21.333 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.333 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:54:21.333 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.719589+0000", 2024-09-21T13:54:21.333 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:21.333 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.334 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.334 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:54:21.334 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.719589+0000", 2024-09-21T13:54:21.334 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:21.334 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.334 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.334 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:54:21.334 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.719590+0000", 2024-09-21T13:54:21.334 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.5600000000000004e-07 2024-09-21T13:54:21.334 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.335 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.335 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:54:21.335 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.719649+0000", 2024-09-21T13:54:21.335 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8730999999999997e-05 2024-09-21T13:54:21.335 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.335 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.335 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:54:21.335 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.719659+0000", 2024-09-21T13:54:21.335 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0498e-05 2024-09-21T13:54:21.335 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.335 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.335 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:54:21.335 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.719666+0000", 2024-09-21T13:54:21.336 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9029999999999998e-06 2024-09-21T13:54:21.336 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.336 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.336 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:21.336 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.719710+0000", 2024-09-21T13:54:21.336 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3859000000000001e-05 2024-09-21T13:54:21.336 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.336 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.336 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:54:21.336 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.719715+0000", 2024-09-21T13:54:21.336 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2229999999999998e-06 2024-09-21T13:54:21.336 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.336 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.336 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:21.337 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.704132+0000", 2024-09-21T13:54:21.337 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.98441694300000004 2024-09-21T13:54:21.337 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.337 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.337 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:54:21.337 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.704260+0000", 2024-09-21T13:54:21.337 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012844899999999999 2024-09-21T13:54:21.337 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.337 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.337 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3", 2024-09-21T13:54:21.337 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.704400+0000", 2024-09-21T13:54:21.337 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013959700000000001 2024-09-21T13:54:21.337 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.337 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.337 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:54:21.337 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.706512+0000", 2024-09-21T13:54:21.338 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0021123270000000002 2024-09-21T13:54:21.338 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.338 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.338 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:21.338 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.708683+0000", 2024-09-21T13:54:21.338 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00217022 2024-09-21T13:54:21.338 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.338 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.338 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:21.338 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.712030+0000", 2024-09-21T13:54:21.338 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0033470459999999998 2024-09-21T13:54:21.338 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.339 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.339 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:54:21.339 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.712058+0000", 2024-09-21T13:54:21.339 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8586e-05 2024-09-21T13:54:21.339 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.339 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.339 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:54:21.339 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.712080+0000", 2024-09-21T13:54:21.339 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1537000000000001e-05 2024-09-21T13:54:21.339 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:21.339 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:54:21.339 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:21.339 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.339 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.340 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:201635 7.5 7:a1b75091:::benchmark_data_smithi049_76798_object65215:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T13:54:21.340 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:43.011858+0000", 2024-09-21T13:54:21.340 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 98.271410829999994, 2024-09-21T13:54:21.340 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.96906032200000003, 2024-09-21T13:54:21.340 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:54:21.340 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:54:21.340 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:21.340 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:54:21.340 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:54:21.340 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:21.340 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 201635 2024-09-21T13:54:21.340 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.340 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:54:21.340 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.341 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:54:21.341 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.011858+0000", 2024-09-21T13:54:21.341 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:21.341 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.341 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.341 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:54:21.341 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.011858+0000", 2024-09-21T13:54:21.341 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:21.341 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.341 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.341 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:54:21.341 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.011859+0000", 2024-09-21T13:54:21.341 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.609e-06 2024-09-21T13:54:21.342 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.342 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.342 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:54:21.342 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.011865+0000", 2024-09-21T13:54:21.342 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2489999999999998e-06 2024-09-21T13:54:21.342 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.342 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.342 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:54:21.342 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.011880+0000", 2024-09-21T13:54:21.342 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4141999999999999e-05 2024-09-21T13:54:21.342 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.342 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.343 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:54:21.343 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.011887+0000", 2024-09-21T13:54:21.343 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4830000000000002e-06 2024-09-21T13:54:21.343 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.343 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.343 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:21.343 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.011963+0000", 2024-09-21T13:54:21.343 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5624999999999995e-05 2024-09-21T13:54:21.343 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.343 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.343 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:54:21.343 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.011971+0000", 2024-09-21T13:54:21.343 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.541e-06 2024-09-21T13:54:21.344 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.344 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.344 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:21.344 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.957601+0000", 2024-09-21T13:54:21.344 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.94562941099999998 2024-09-21T13:54:21.344 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.344 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.344 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:54:21.344 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.957730+0000", 2024-09-21T13:54:21.344 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000129016 2024-09-21T13:54:21.345 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.345 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.345 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2", 2024-09-21T13:54:21.345 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.957877+0000", 2024-09-21T13:54:21.345 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000147563 2024-09-21T13:54:21.345 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.345 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.345 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:54:21.345 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.962710+0000", 2024-09-21T13:54:21.345 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.004832355 2024-09-21T13:54:21.345 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.345 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.345 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:21.345 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.965643+0000", 2024-09-21T13:54:21.345 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.002933017 2024-09-21T13:54:21.346 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.346 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.346 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:21.346 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.980821+0000", 2024-09-21T13:54:21.346 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.015178552999999999 2024-09-21T13:54:21.346 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.346 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.346 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:54:21.346 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.980880+0000", 2024-09-21T13:54:21.346 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8646000000000001e-05 2024-09-21T13:54:21.346 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.346 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.346 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:54:21.346 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.980918+0000", 2024-09-21T13:54:21.347 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8111999999999997e-05 2024-09-21T13:54:21.347 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:21.347 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:54:21.347 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:21.347 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.347 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.347 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:201680 7.5 7:ac2e5a95:::benchmark_data_smithi049_76798_object65260:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T13:54:21.347 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:43.053233+0000", 2024-09-21T13:54:21.347 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 98.230035375, 2024-09-21T13:54:21.347 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.92777444499999995, 2024-09-21T13:54:21.347 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:54:21.347 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:54:21.347 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:21.347 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:54:21.347 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:54:21.348 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:21.348 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 201680 2024-09-21T13:54:21.348 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.348 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:54:21.348 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.348 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:54:21.348 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.053233+0000", 2024-09-21T13:54:21.348 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:21.348 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.348 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.348 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:54:21.348 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.053233+0000", 2024-09-21T13:54:21.348 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:21.349 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.349 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.349 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:54:21.349 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.053249+0000", 2024-09-21T13:54:21.349 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.596e-05 2024-09-21T13:54:21.349 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.349 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.349 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:54:21.349 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.053257+0000", 2024-09-21T13:54:21.349 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9980000000000003e-06 2024-09-21T13:54:21.349 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.349 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.349 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:54:21.349 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.053278+0000", 2024-09-21T13:54:21.350 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0749000000000001e-05 2024-09-21T13:54:21.350 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.350 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.350 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:54:21.350 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.053289+0000", 2024-09-21T13:54:21.350 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0864e-05 2024-09-21T13:54:21.350 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.350 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.350 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:21.350 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.053366+0000", 2024-09-21T13:54:21.350 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7082999999999994e-05 2024-09-21T13:54:21.350 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.351 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.351 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:54:21.351 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.053375+0000", 2024-09-21T13:54:21.351 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3230000000000004e-06 2024-09-21T13:54:21.351 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.351 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.351 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:21.351 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.960197+0000", 2024-09-21T13:54:21.351 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.90682242700000004 2024-09-21T13:54:21.351 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.351 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.351 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:54:21.351 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.960297+0000", 2024-09-21T13:54:21.351 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.9817999999999999e-05 2024-09-21T13:54:21.352 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.352 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.352 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2", 2024-09-21T13:54:21.352 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.960410+0000", 2024-09-21T13:54:21.352 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000113315 2024-09-21T13:54:21.352 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.352 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.352 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:54:21.352 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.965602+0000", 2024-09-21T13:54:21.352 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0051915390000000002 2024-09-21T13:54:21.352 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.352 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.352 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:21.352 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.965712+0000", 2024-09-21T13:54:21.353 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010966700000000001 2024-09-21T13:54:21.353 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.353 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.353 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:21.353 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.980954+0000", 2024-09-21T13:54:21.353 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.015242551999999999 2024-09-21T13:54:21.353 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.353 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.353 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:54:21.353 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.980985+0000", 2024-09-21T13:54:21.353 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0686999999999999e-05 2024-09-21T13:54:21.353 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.353 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.353 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:54:21.353 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.981007+0000", 2024-09-21T13:54:21.354 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2463000000000002e-05 2024-09-21T13:54:21.354 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:21.354 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:54:21.354 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:21.354 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.354 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.354 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:201691 7.5 7:a0c84d4e:::benchmark_data_smithi049_76798_object65271:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T13:54:21.354 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:43.063956+0000", 2024-09-21T13:54:21.354 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 98.219311884999996, 2024-09-21T13:54:21.354 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.91993442199999997, 2024-09-21T13:54:21.355 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:54:21.355 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:54:21.355 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:21.355 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:54:21.355 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:54:21.355 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:21.355 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 201691 2024-09-21T13:54:21.355 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.355 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:54:21.355 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.355 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:54:21.355 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.063956+0000", 2024-09-21T13:54:21.355 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:21.355 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.356 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.356 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:54:21.356 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.063956+0000", 2024-09-21T13:54:21.356 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:21.356 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.356 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.356 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:54:21.356 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.063959+0000", 2024-09-21T13:54:21.356 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.401e-06 2024-09-21T13:54:21.356 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.356 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.356 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:54:21.356 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.063966+0000", 2024-09-21T13:54:21.356 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2479999999999999e-06 2024-09-21T13:54:21.356 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.357 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.357 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:54:21.357 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.063984+0000", 2024-09-21T13:54:21.357 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7530999999999999e-05 2024-09-21T13:54:21.357 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.357 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.357 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:54:21.357 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.063992+0000", 2024-09-21T13:54:21.357 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2509999999999994e-06 2024-09-21T13:54:21.357 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.357 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.357 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:21.357 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.064060+0000", 2024-09-21T13:54:21.357 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8535999999999996e-05 2024-09-21T13:54:21.357 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.358 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.358 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:54:21.358 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.064070+0000", 2024-09-21T13:54:21.358 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6830000000000003e-06 2024-09-21T13:54:21.358 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.358 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.358 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:21.358 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.961470+0000", 2024-09-21T13:54:21.358 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.89740008599999999 2024-09-21T13:54:21.358 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.358 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.358 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:54:21.358 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.961570+0000", 2024-09-21T13:54:21.359 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000100237 2024-09-21T13:54:21.359 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.359 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.359 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2", 2024-09-21T13:54:21.359 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.961689+0000", 2024-09-21T13:54:21.359 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000118347 2024-09-21T13:54:21.359 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.359 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.359 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:54:21.359 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.965656+0000", 2024-09-21T13:54:21.359 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0039668029999999996 2024-09-21T13:54:21.359 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.360 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.360 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:21.360 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.979772+0000", 2024-09-21T13:54:21.360 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.014116717000000001 2024-09-21T13:54:21.360 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.360 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.360 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:21.360 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.983830+0000", 2024-09-21T13:54:21.360 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0040575189999999999 2024-09-21T13:54:21.360 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.360 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.360 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:54:21.360 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.983868+0000", 2024-09-21T13:54:21.361 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8586e-05 2024-09-21T13:54:21.361 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.361 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.361 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:54:21.361 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.983891+0000", 2024-09-21T13:54:21.361 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2476999999999998e-05 2024-09-21T13:54:21.361 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:21.361 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:54:21.361 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:21.362 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.362 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.362 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:201695 7.5 7:a9f46a76:::benchmark_data_smithi049_76798_object65275:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T13:54:21.362 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:43.066285+0000", 2024-09-21T13:54:21.362 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 98.216983757999998, 2024-09-21T13:54:21.362 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.92214624000000001, 2024-09-21T13:54:21.362 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:54:21.362 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:54:21.362 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:21.362 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:54:21.363 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:54:21.363 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:21.363 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 201695 2024-09-21T13:54:21.363 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.363 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:54:21.363 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.363 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:54:21.363 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.066285+0000", 2024-09-21T13:54:21.363 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:21.363 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.363 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.363 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:54:21.363 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.066285+0000", 2024-09-21T13:54:21.364 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:21.364 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.364 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.364 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:54:21.364 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.066287+0000", 2024-09-21T13:54:21.364 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0219999999999999e-06 2024-09-21T13:54:21.364 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.364 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.364 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:54:21.364 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.066295+0000", 2024-09-21T13:54:21.364 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1999999999999994e-06 2024-09-21T13:54:21.364 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.365 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.365 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:54:21.365 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.066316+0000", 2024-09-21T13:54:21.365 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1508000000000001e-05 2024-09-21T13:54:21.365 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.365 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.365 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:54:21.365 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.066326+0000", 2024-09-21T13:54:21.366 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.4290000000000007e-06 2024-09-21T13:54:21.366 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.366 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.366 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:21.366 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.066400+0000", 2024-09-21T13:54:21.366 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4339999999999996e-05 2024-09-21T13:54:21.366 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.366 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.366 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:54:21.366 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.066410+0000", 2024-09-21T13:54:21.366 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.5799999999999998e-06 2024-09-21T13:54:21.366 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.367 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.367 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:21.367 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.962805+0000", 2024-09-21T13:54:21.367 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.89639528199999996 2024-09-21T13:54:21.367 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.367 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.367 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:54:21.367 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.962880+0000", 2024-09-21T13:54:21.367 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5139e-05 2024-09-21T13:54:21.367 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.367 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.367 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2", 2024-09-21T13:54:21.367 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.962949+0000", 2024-09-21T13:54:21.367 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9098999999999997e-05 2024-09-21T13:54:21.367 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.368 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.368 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:54:21.368 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.965662+0000", 2024-09-21T13:54:21.368 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0027127259999999999 2024-09-21T13:54:21.368 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.368 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.368 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:21.368 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.983932+0000", 2024-09-21T13:54:21.368 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.018270237000000002 2024-09-21T13:54:21.368 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.368 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.368 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:21.369 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.988374+0000", 2024-09-21T13:54:21.369 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0044419239999999999 2024-09-21T13:54:21.369 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.369 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.369 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:54:21.369 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.988406+0000", 2024-09-21T13:54:21.369 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2172000000000003e-05 2024-09-21T13:54:21.369 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.369 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.369 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:54:21.369 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.988431+0000", 2024-09-21T13:54:21.369 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4581999999999999e-05 2024-09-21T13:54:21.369 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:21.369 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:54:21.369 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:21.369 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.370 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.370 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:201698 7.5 7:ae6dee09:::benchmark_data_smithi049_76798_object65278:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T13:54:21.370 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:43.068553+0000", 2024-09-21T13:54:21.370 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 98.214715286000001, 2024-09-21T13:54:21.370 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.91996031099999998, 2024-09-21T13:54:21.370 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:54:21.370 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:54:21.370 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:21.370 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:54:21.370 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:54:21.370 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:21.370 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 201698 2024-09-21T13:54:21.371 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.371 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:54:21.371 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.371 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:54:21.371 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.068553+0000", 2024-09-21T13:54:21.371 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:21.371 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.371 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.371 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:54:21.371 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.068553+0000", 2024-09-21T13:54:21.371 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:21.371 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.371 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.372 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:54:21.372 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.068555+0000", 2024-09-21T13:54:21.372 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2840000000000001e-06 2024-09-21T13:54:21.372 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.372 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.372 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:54:21.372 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.068563+0000", 2024-09-21T13:54:21.372 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6899999999999992e-06 2024-09-21T13:54:21.372 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.372 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.372 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:54:21.372 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.068582+0000", 2024-09-21T13:54:21.372 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8902e-05 2024-09-21T13:54:21.372 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.373 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.373 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:54:21.373 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.068591+0000", 2024-09-21T13:54:21.373 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.6440000000000006e-06 2024-09-21T13:54:21.373 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.373 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.373 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:21.373 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.068654+0000", 2024-09-21T13:54:21.373 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.347e-05 2024-09-21T13:54:21.373 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.373 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.373 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:54:21.373 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.068662+0000", 2024-09-21T13:54:21.374 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1829999999999994e-06 2024-09-21T13:54:21.374 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.374 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.374 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:21.374 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.964312+0000", 2024-09-21T13:54:21.374 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.89564967699999998 2024-09-21T13:54:21.374 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.374 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.374 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:54:21.374 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.964413+0000", 2024-09-21T13:54:21.374 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010090399999999999 2024-09-21T13:54:21.374 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.374 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.375 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2", 2024-09-21T13:54:21.375 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.964529+0000", 2024-09-21T13:54:21.375 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000116579 2024-09-21T13:54:21.375 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.375 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.375 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:54:21.375 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:54:21.376 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:54:21.377 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.970760+0000", 2024-09-21T13:54:21.378 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0062309820000000004 2024-09-21T13:54:21.378 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.378 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.378 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:21.378 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.983968+0000", 2024-09-21T13:54:21.378 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.013207727000000001 2024-09-21T13:54:21.378 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.378 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.379 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:21.379 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.988459+0000", 2024-09-21T13:54:21.379 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0044908889999999996 2024-09-21T13:54:21.379 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.379 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.379 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:54:21.379 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.988489+0000", 2024-09-21T13:54:21.379 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0063e-05 2024-09-21T13:54:21.379 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.379 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.379 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:54:21.380 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.988513+0000", 2024-09-21T13:54:21.380 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4317000000000001e-05 2024-09-21T13:54:21.380 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:21.380 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:54:21.380 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:21.380 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.380 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.380 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:201701 7.5 7:a7fade2d:::benchmark_data_smithi049_76798_object65281:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T13:54:21.380 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:43.071918+0000", 2024-09-21T13:54:21.380 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 98.211350019999998, 2024-09-21T13:54:21.380 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.91666014399999995, 2024-09-21T13:54:21.380 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:54:21.380 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:54:21.381 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:21.381 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:54:21.381 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:54:21.381 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:21.381 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 201701 2024-09-21T13:54:21.381 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.381 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:54:21.381 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.381 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:54:21.381 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.071918+0000", 2024-09-21T13:54:21.381 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:21.381 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.381 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.381 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:54:21.381 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.071918+0000", 2024-09-21T13:54:21.381 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:21.382 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.382 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.382 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:54:21.382 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.071920+0000", 2024-09-21T13:54:21.382 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3340000000000001e-06 2024-09-21T13:54:21.382 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.382 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.382 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:54:21.382 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.071926+0000", 2024-09-21T13:54:21.382 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8719999999999999e-06 2024-09-21T13:54:21.382 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.382 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.382 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:54:21.383 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.071941+0000", 2024-09-21T13:54:21.383 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5529999999999999e-05 2024-09-21T13:54:21.383 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.383 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.383 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:54:21.383 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.071949+0000", 2024-09-21T13:54:21.383 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1000000000000004e-06 2024-09-21T13:54:21.383 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.383 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.383 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:21.383 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.072014+0000", 2024-09-21T13:54:21.383 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4912000000000005e-05 2024-09-21T13:54:21.383 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.383 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.383 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:54:21.384 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.072023+0000", 2024-09-21T13:54:21.384 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7069999999999998e-06 2024-09-21T13:54:21.384 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.384 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.384 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:21.384 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.965753+0000", 2024-09-21T13:54:21.384 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.89372980199999996 2024-09-21T13:54:21.384 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.384 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.384 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:54:21.384 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.965886+0000", 2024-09-21T13:54:21.384 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000133669 2024-09-21T13:54:21.384 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.384 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.384 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2", 2024-09-21T13:54:21.384 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.966006+0000", 2024-09-21T13:54:21.385 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000119407 2024-09-21T13:54:21.385 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.385 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.385 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:54:21.385 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.973164+0000", 2024-09-21T13:54:21.385 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.007158569 2024-09-21T13:54:21.385 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.385 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.385 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:21.385 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.984364+0000", 2024-09-21T13:54:21.385 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.011199944 2024-09-21T13:54:21.385 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.385 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.385 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:21.386 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.988535+0000", 2024-09-21T13:54:21.386 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0041708800000000001 2024-09-21T13:54:21.386 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.386 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.386 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:54:21.386 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.988560+0000", 2024-09-21T13:54:21.386 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4879e-05 2024-09-21T13:54:21.386 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.387 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.387 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:54:21.387 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.988578+0000", 2024-09-21T13:54:21.387 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8539e-05 2024-09-21T13:54:21.387 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:21.387 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:54:21.387 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:21.387 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.387 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.387 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:201772 7.5 7:a814840e:::benchmark_data_smithi049_76798_object65352:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T13:54:21.387 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:43.156794+0000", 2024-09-21T13:54:21.387 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 98.126474611999996, 2024-09-21T13:54:21.387 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.83184825500000004, 2024-09-21T13:54:21.387 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:54:21.387 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:54:21.388 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:21.388 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:54:21.388 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:54:21.388 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:21.388 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 201772 2024-09-21T13:54:21.388 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.388 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:54:21.388 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.388 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:54:21.388 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.156794+0000", 2024-09-21T13:54:21.388 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:21.388 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.388 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.388 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:54:21.389 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.156794+0000", 2024-09-21T13:54:21.389 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:21.389 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.389 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.389 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:54:21.389 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.156795+0000", 2024-09-21T13:54:21.389 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4839999999999999e-06 2024-09-21T13:54:21.389 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.389 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.389 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:54:21.389 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.156801+0000", 2024-09-21T13:54:21.389 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5709999999999997e-06 2024-09-21T13:54:21.389 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.390 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.390 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:54:21.390 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.156817+0000", 2024-09-21T13:54:21.390 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6378000000000001e-05 2024-09-21T13:54:21.390 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.390 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.390 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:54:21.390 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.156826+0000", 2024-09-21T13:54:21.390 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4060000000000005e-06 2024-09-21T13:54:21.390 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.390 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.390 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:21.391 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.156889+0000", 2024-09-21T13:54:21.391 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3396000000000001e-05 2024-09-21T13:54:21.391 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.391 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.391 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:54:21.391 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.156895+0000", 2024-09-21T13:54:21.391 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7699999999999998e-06 2024-09-21T13:54:21.391 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.391 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.391 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:21.391 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.967038+0000", 2024-09-21T13:54:21.391 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.81014350400000001 2024-09-21T13:54:21.391 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.391 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.392 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:54:21.392 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.967105+0000", 2024-09-21T13:54:21.392 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6561000000000002e-05 2024-09-21T13:54:21.392 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.392 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.392 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2", 2024-09-21T13:54:21.392 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.967172+0000", 2024-09-21T13:54:21.392 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7336999999999996e-05 2024-09-21T13:54:21.392 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.392 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.392 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:54:21.392 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.973169+0000", 2024-09-21T13:54:21.392 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.005996927 2024-09-21T13:54:21.392 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.393 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.393 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:21.393 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.984408+0000", 2024-09-21T13:54:21.393 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.011238823 2024-09-21T13:54:21.393 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.393 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.393 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:21.393 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.988599+0000", 2024-09-21T13:54:21.393 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0041907630000000001 2024-09-21T13:54:21.393 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.393 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.393 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:54:21.394 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.988624+0000", 2024-09-21T13:54:21.394 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5148999999999999e-05 2024-09-21T13:54:21.394 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.394 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.394 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:54:21.394 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.988642+0000", 2024-09-21T13:54:21.394 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8185999999999998e-05 2024-09-21T13:54:21.394 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:21.394 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:54:21.394 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:21.395 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.395 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.395 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:201787 7.5 7:a2e71a94:::benchmark_data_smithi049_76798_object65367:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T13:54:21.395 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:43.173880+0000", 2024-09-21T13:54:21.395 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 98.109388636000006, 2024-09-21T13:54:21.395 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.81499015100000005, 2024-09-21T13:54:21.395 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:54:21.395 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:54:21.395 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:21.395 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:54:21.395 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:54:21.395 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:21.395 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 201787 2024-09-21T13:54:21.396 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.396 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:54:21.396 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.396 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:54:21.396 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.173880+0000", 2024-09-21T13:54:21.396 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:21.396 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.396 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.396 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:54:21.396 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.173880+0000", 2024-09-21T13:54:21.396 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:21.396 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.397 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.397 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:54:21.397 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.173882+0000", 2024-09-21T13:54:21.397 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1500000000000002e-06 2024-09-21T13:54:21.397 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.397 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.397 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:54:21.397 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.173888+0000", 2024-09-21T13:54:21.397 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3799999999999999e-06 2024-09-21T13:54:21.397 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.397 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.397 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:54:21.397 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.173902+0000", 2024-09-21T13:54:21.397 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4225e-05 2024-09-21T13:54:21.398 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.398 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.398 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:54:21.398 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.173911+0000", 2024-09-21T13:54:21.398 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0649999999999999e-06 2024-09-21T13:54:21.398 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.398 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.398 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:21.398 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.173972+0000", 2024-09-21T13:54:21.398 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1168999999999997e-05 2024-09-21T13:54:21.398 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.398 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.398 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:54:21.399 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.173980+0000", 2024-09-21T13:54:21.399 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2549999999999997e-06 2024-09-21T13:54:21.399 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.399 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.399 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:21.399 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.968330+0000", 2024-09-21T13:54:21.399 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.79435025800000003 2024-09-21T13:54:21.399 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.399 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.399 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:54:21.399 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.968390+0000", 2024-09-21T13:54:21.400 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0149999999999998e-05 2024-09-21T13:54:21.400 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.400 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.400 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2", 2024-09-21T13:54:21.400 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.968453+0000", 2024-09-21T13:54:21.400 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2427000000000004e-05 2024-09-21T13:54:21.400 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.400 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.400 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:54:21.400 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.973174+0000", 2024-09-21T13:54:21.400 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0047213740000000004 2024-09-21T13:54:21.400 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.401 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.401 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:21.401 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.984437+0000", 2024-09-21T13:54:21.401 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.011262507 2024-09-21T13:54:21.401 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.401 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.401 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:21.401 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.988823+0000", 2024-09-21T13:54:21.401 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0043867799999999998 2024-09-21T13:54:21.401 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.401 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.401 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:54:21.402 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.988853+0000", 2024-09-21T13:54:21.402 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9116e-05 2024-09-21T13:54:21.402 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.402 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.402 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:54:21.402 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.988870+0000", 2024-09-21T13:54:21.402 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7295e-05 2024-09-21T13:54:21.402 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:21.402 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:54:21.402 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:21.402 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.402 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.402 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:201826 7.5 7:a523bf31:::benchmark_data_smithi049_76798_object65406:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T13:54:21.403 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:43.212124+0000", 2024-09-21T13:54:21.403 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 98.071144507, 2024-09-21T13:54:21.403 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.77941543700000004, 2024-09-21T13:54:21.403 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:54:21.403 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:54:21.403 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:21.403 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:54:21.403 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:54:21.403 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:21.403 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 201826 2024-09-21T13:54:21.403 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.403 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:54:21.403 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.403 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:54:21.403 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.212124+0000", 2024-09-21T13:54:21.404 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:21.404 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.404 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.404 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:54:21.404 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.212124+0000", 2024-09-21T13:54:21.404 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:21.404 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.404 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.404 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:54:21.404 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.212125+0000", 2024-09-21T13:54:21.404 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3200000000000001e-06 2024-09-21T13:54:21.404 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.404 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.404 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:54:21.405 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.212130+0000", 2024-09-21T13:54:21.405 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5210000000000004e-06 2024-09-21T13:54:21.405 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.405 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.405 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:54:21.405 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.212142+0000", 2024-09-21T13:54:21.405 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1878e-05 2024-09-21T13:54:21.405 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.405 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.405 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:54:21.405 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.212147+0000", 2024-09-21T13:54:21.405 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3410000000000002e-06 2024-09-21T13:54:21.405 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.405 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.405 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:21.405 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.212194+0000", 2024-09-21T13:54:21.406 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7091e-05 2024-09-21T13:54:21.406 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.406 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.406 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:54:21.406 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.212200+0000", 2024-09-21T13:54:21.406 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0440000000000001e-06 2024-09-21T13:54:21.406 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.406 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.406 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:21.406 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.969460+0000", 2024-09-21T13:54:21.406 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.75725952399999996 2024-09-21T13:54:21.406 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.406 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.407 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:54:21.407 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.969570+0000", 2024-09-21T13:54:21.407 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000110876 2024-09-21T13:54:21.407 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.407 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.407 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2", 2024-09-21T13:54:21.407 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.969687+0000", 2024-09-21T13:54:21.407 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011697 2024-09-21T13:54:21.407 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.407 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.408 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:54:21.408 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.974455+0000", 2024-09-21T13:54:21.408 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0047673259999999997 2024-09-21T13:54:21.408 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.408 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.408 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:21.408 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.990260+0000", 2024-09-21T13:54:21.408 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.015805567999999999 2024-09-21T13:54:21.408 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.408 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.408 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:21.408 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.991462+0000", 2024-09-21T13:54:21.408 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0012021340000000001 2024-09-21T13:54:21.408 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.408 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.408 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:54:21.409 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.991504+0000", 2024-09-21T13:54:21.409 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1715000000000003e-05 2024-09-21T13:54:21.409 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.409 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.409 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:54:21.409 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.991539+0000", 2024-09-21T13:54:21.409 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5129000000000002e-05 2024-09-21T13:54:21.409 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:21.409 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:54:21.409 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:21.409 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.409 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.409 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:201840 7.5 7:abeb3043:::benchmark_data_smithi049_76798_object65420:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T13:54:21.409 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:43.224752+0000", 2024-09-21T13:54:21.409 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 98.058516206999997, 2024-09-21T13:54:21.409 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.76862761999999996, 2024-09-21T13:54:21.410 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:54:21.410 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:54:21.410 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:21.410 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:54:21.410 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:54:21.410 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:21.410 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 201840 2024-09-21T13:54:21.410 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.410 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:54:21.410 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.410 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:54:21.411 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.224752+0000", 2024-09-21T13:54:21.411 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:21.411 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.411 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.411 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:54:21.411 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.224752+0000", 2024-09-21T13:54:21.411 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:21.411 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.411 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.412 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:54:21.412 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.224753+0000", 2024-09-21T13:54:21.412 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.8299999999999995e-07 2024-09-21T13:54:21.412 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.412 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.412 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:54:21.412 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.224757+0000", 2024-09-21T13:54:21.412 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2379999999999999e-06 2024-09-21T13:54:21.412 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.412 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.412 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:54:21.413 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.224768+0000", 2024-09-21T13:54:21.413 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0390999999999999e-05 2024-09-21T13:54:21.413 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.413 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.413 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:54:21.413 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.224775+0000", 2024-09-21T13:54:21.413 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9999999999999999e-06 2024-09-21T13:54:21.413 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.413 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.413 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:21.413 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.224823+0000", 2024-09-21T13:54:21.414 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8186999999999998e-05 2024-09-21T13:54:21.414 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.414 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.414 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:54:21.414 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.224830+0000", 2024-09-21T13:54:21.414 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6100000000000002e-06 2024-09-21T13:54:21.414 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.414 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.414 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:21.414 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.970787+0000", 2024-09-21T13:54:21.414 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.74595736499999998 2024-09-21T13:54:21.414 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.414 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.415 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:54:21.415 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.970866+0000", 2024-09-21T13:54:21.415 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8731000000000002e-05 2024-09-21T13:54:21.415 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.415 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.415 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2", 2024-09-21T13:54:21.415 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.970932+0000", 2024-09-21T13:54:21.415 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6031000000000005e-05 2024-09-21T13:54:21.415 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.415 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.415 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:54:21.415 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.974460+0000", 2024-09-21T13:54:21.415 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0035279069999999998 2024-09-21T13:54:21.416 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.416 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.416 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:21.416 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.991576+0000", 2024-09-21T13:54:21.416 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.017116833000000001 2024-09-21T13:54:21.416 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.416 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.416 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:21.416 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.993291+0000", 2024-09-21T13:54:21.416 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0017143690000000001 2024-09-21T13:54:21.416 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.417 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.417 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:54:21.417 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.993342+0000", 2024-09-21T13:54:21.417 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1316000000000001e-05 2024-09-21T13:54:21.417 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.417 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.417 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:54:21.417 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.993380+0000", 2024-09-21T13:54:21.417 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7659e-05 2024-09-21T13:54:21.418 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:21.418 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:54:21.418 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:21.418 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.418 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.418 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:201850 7.5 7:adb529a4:::benchmark_data_smithi049_76798_object65430:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T13:54:21.418 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:43.237139+0000", 2024-09-21T13:54:21.418 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 98.046129500000006, 2024-09-21T13:54:21.418 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.76209282499999997, 2024-09-21T13:54:21.418 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:54:21.419 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:54:21.419 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:21.419 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:54:21.419 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:54:21.419 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:21.419 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 201850 2024-09-21T13:54:21.419 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.419 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:54:21.419 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.419 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:54:21.419 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.237139+0000", 2024-09-21T13:54:21.419 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:21.419 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.419 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.420 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:54:21.420 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.237139+0000", 2024-09-21T13:54:21.420 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:21.420 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.420 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.420 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:54:21.420 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.237139+0000", 2024-09-21T13:54:21.420 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5799999999999999e-07 2024-09-21T13:54:21.420 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.421 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.421 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:54:21.421 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.237143+0000", 2024-09-21T13:54:21.421 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5310000000000002e-06 2024-09-21T13:54:21.421 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.421 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.421 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:54:21.421 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.237152+0000", 2024-09-21T13:54:21.421 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.653e-06 2024-09-21T13:54:21.421 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.422 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.422 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:54:21.422 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.237157+0000", 2024-09-21T13:54:21.422 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8239999999999999e-06 2024-09-21T13:54:21.422 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.422 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.422 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:21.422 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.237205+0000", 2024-09-21T13:54:21.422 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8341999999999999e-05 2024-09-21T13:54:21.422 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.422 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.422 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:54:21.422 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.237212+0000", 2024-09-21T13:54:21.422 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6410000000000002e-06 2024-09-21T13:54:21.423 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.423 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.423 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:21.423 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.971939+0000", 2024-09-21T13:54:21.423 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.73472659200000001 2024-09-21T13:54:21.423 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.423 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.423 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:54:21.423 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.972042+0000", 2024-09-21T13:54:21.423 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000103049 2024-09-21T13:54:21.423 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.423 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.423 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2", 2024-09-21T13:54:21.424 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.972158+0000", 2024-09-21T13:54:21.424 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011618 2024-09-21T13:54:21.424 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.424 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.424 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:54:21.424 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.975735+0000", 2024-09-21T13:54:21.424 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0035764249999999998 2024-09-21T13:54:21.424 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.424 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.424 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:21.424 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.991609+0000", 2024-09-21T13:54:21.424 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.015874307000000001 2024-09-21T13:54:21.424 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.424 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.425 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:21.425 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.999171+0000", 2024-09-21T13:54:21.425 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0075622440000000001 2024-09-21T13:54:21.425 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.425 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.425 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:54:21.425 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.999201+0000", 2024-09-21T13:54:21.425 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9495000000000001e-05 2024-09-21T13:54:21.425 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.425 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.425 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:54:21.425 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.999232+0000", 2024-09-21T13:54:21.425 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0883999999999998e-05 2024-09-21T13:54:21.425 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:21.426 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:54:21.426 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:21.426 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.426 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.426 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:201852 7.5 7:a4c0c63b:::benchmark_data_smithi049_76798_object65432:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T13:54:21.426 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:43.239197+0000", 2024-09-21T13:54:21.426 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 98.044071209999998, 2024-09-21T13:54:21.426 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.76675081499999997, 2024-09-21T13:54:21.426 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:54:21.426 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:54:21.426 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:21.426 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:54:21.426 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:54:21.426 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:21.427 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 201852 2024-09-21T13:54:21.427 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.427 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:54:21.427 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.427 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:54:21.427 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.239197+0000", 2024-09-21T13:54:21.427 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:21.427 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.427 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.427 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:54:21.427 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.239197+0000", 2024-09-21T13:54:21.427 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:21.427 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.427 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.428 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:54:21.428 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.239198+0000", 2024-09-21T13:54:21.428 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6899999999999996e-07 2024-09-21T13:54:21.428 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.428 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.428 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:54:21.428 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.239202+0000", 2024-09-21T13:54:21.428 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8340000000000001e-06 2024-09-21T13:54:21.428 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.428 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.428 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:54:21.429 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.239212+0000", 2024-09-21T13:54:21.429 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0563e-05 2024-09-21T13:54:21.429 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.429 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.429 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:54:21.429 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.239219+0000", 2024-09-21T13:54:21.429 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.241e-06 2024-09-21T13:54:21.429 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.429 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.429 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:21.429 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.239260+0000", 2024-09-21T13:54:21.429 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1421999999999997e-05 2024-09-21T13:54:21.429 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.430 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.430 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:54:21.430 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.239265+0000", 2024-09-21T13:54:21.430 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5399999999999997e-06 2024-09-21T13:54:21.430 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.430 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.430 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:21.430 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.973203+0000", 2024-09-21T13:54:21.430 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.73393844600000002 2024-09-21T13:54:21.430 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.430 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.430 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:54:21.431 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.973305+0000", 2024-09-21T13:54:21.431 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010141699999999999 2024-09-21T13:54:21.431 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.431 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.431 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2", 2024-09-21T13:54:21.431 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.973424+0000", 2024-09-21T13:54:21.431 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011916700000000001 2024-09-21T13:54:21.431 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.431 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.431 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:54:21.431 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.977041+0000", 2024-09-21T13:54:21.431 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0036168989999999998 2024-09-21T13:54:21.431 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.432 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.432 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:21.432 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.996582+0000", 2024-09-21T13:54:21.432 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.019541097 2024-09-21T13:54:21.432 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.432 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.432 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:21.432 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.005893+0000", 2024-09-21T13:54:21.432 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0093108410000000003 2024-09-21T13:54:21.432 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.432 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.432 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:54:21.432 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.005925+0000", 2024-09-21T13:54:21.433 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2490000000000002e-05 2024-09-21T13:54:21.433 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.433 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.433 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:54:21.433 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.005948+0000", 2024-09-21T13:54:21.433 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2889000000000001e-05 2024-09-21T13:54:21.433 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:21.433 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:54:21.433 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:21.433 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.433 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.433 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:201856 7.5 7:a20bd2e0:::benchmark_data_smithi049_76798_object65436:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T13:54:21.433 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:43.241762+0000", 2024-09-21T13:54:21.433 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 98.041505888000003, 2024-09-21T13:54:21.434 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.76427266500000002, 2024-09-21T13:54:21.434 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:54:21.434 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:54:21.434 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:21.434 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:54:21.434 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:54:21.434 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:21.434 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 201856 2024-09-21T13:54:21.434 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.434 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:54:21.434 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.434 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:54:21.434 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.241762+0000", 2024-09-21T13:54:21.434 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:21.434 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.435 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.435 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:54:21.435 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.241762+0000", 2024-09-21T13:54:21.435 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:21.435 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.435 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.435 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:54:21.435 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.241763+0000", 2024-09-21T13:54:21.435 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.2200000000000002e-07 2024-09-21T13:54:21.435 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.435 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.435 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:54:21.435 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.241768+0000", 2024-09-21T13:54:21.436 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.357e-06 2024-09-21T13:54:21.436 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.436 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.436 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:54:21.436 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.241780+0000", 2024-09-21T13:54:21.436 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.224e-05 2024-09-21T13:54:21.436 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.436 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.436 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:54:21.436 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.241787+0000", 2024-09-21T13:54:21.436 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4710000000000001e-06 2024-09-21T13:54:21.436 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.436 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.437 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:21.437 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.241830+0000", 2024-09-21T13:54:21.437 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2960000000000002e-05 2024-09-21T13:54:21.437 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.437 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.437 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:54:21.437 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.241835+0000", 2024-09-21T13:54:21.437 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8110000000000002e-06 2024-09-21T13:54:21.437 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.437 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.437 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:21.437 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.974488+0000", 2024-09-21T13:54:21.437 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.73265256999999995 2024-09-21T13:54:21.437 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.438 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.438 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:54:21.438 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.974586+0000", 2024-09-21T13:54:21.438 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.8669e-05 2024-09-21T13:54:21.438 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.438 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.438 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2", 2024-09-21T13:54:21.438 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.974702+0000", 2024-09-21T13:54:21.438 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000115183 2024-09-21T13:54:21.438 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.438 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.439 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:54:21.439 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.979708+0000", 2024-09-21T13:54:21.439 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0050061339999999998 2024-09-21T13:54:21.439 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.439 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.439 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:21.439 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.997392+0000", 2024-09-21T13:54:21.439 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.017684388999999998 2024-09-21T13:54:21.439 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.439 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.439 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:21.439 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.005982+0000", 2024-09-21T13:54:21.439 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.008590192 2024-09-21T13:54:21.439 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.439 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.439 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:54:21.440 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.006012+0000", 2024-09-21T13:54:21.440 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9714e-05 2024-09-21T13:54:21.440 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.440 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.440 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:54:21.440 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.006035+0000", 2024-09-21T13:54:21.440 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3053000000000001e-05 2024-09-21T13:54:21.440 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:21.440 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:54:21.440 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:21.440 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.440 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.440 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:201866 7.5 7:ab0bbb06:::benchmark_data_smithi049_76798_object65446:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T13:54:21.440 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:43.261703+0000", 2024-09-21T13:54:21.440 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 98.021565332999998, 2024-09-21T13:54:21.440 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.744403649, 2024-09-21T13:54:21.441 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:54:21.441 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:54:21.441 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:21.441 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:54:21.441 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:54:21.441 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:21.441 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 201866 2024-09-21T13:54:21.441 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.441 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:54:21.441 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.441 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:54:21.441 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.261703+0000", 2024-09-21T13:54:21.441 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:21.441 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.441 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.441 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:54:21.441 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.261703+0000", 2024-09-21T13:54:21.442 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:21.442 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.442 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.442 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:54:21.442 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.261705+0000", 2024-09-21T13:54:21.442 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7290000000000001e-06 2024-09-21T13:54:21.442 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.442 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.442 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:54:21.442 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.261711+0000", 2024-09-21T13:54:21.442 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2319999999999998e-06 2024-09-21T13:54:21.442 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.442 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.442 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:54:21.443 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.261724+0000", 2024-09-21T13:54:21.443 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3375e-05 2024-09-21T13:54:21.443 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.443 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.443 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:54:21.443 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.261731+0000", 2024-09-21T13:54:21.443 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3319999999999997e-06 2024-09-21T13:54:21.443 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.443 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.443 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:21.443 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.261790+0000", 2024-09-21T13:54:21.443 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9110999999999997e-05 2024-09-21T13:54:21.443 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.443 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.444 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:54:21.444 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.261797+0000", 2024-09-21T13:54:21.444 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9739999999999996e-06 2024-09-21T13:54:21.444 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.444 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.444 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:21.444 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.975764+0000", 2024-09-21T13:54:21.444 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.71396768300000002 2024-09-21T13:54:21.444 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.444 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.444 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:54:21.444 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.975882+0000", 2024-09-21T13:54:21.444 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011723899999999999 2024-09-21T13:54:21.444 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.444 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.444 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2", 2024-09-21T13:54:21.445 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.976017+0000", 2024-09-21T13:54:21.445 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013532799999999999 2024-09-21T13:54:21.445 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.445 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.445 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:54:21.445 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.979731+0000", 2024-09-21T13:54:21.445 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0037143509999999999 2024-09-21T13:54:21.445 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.445 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.445 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:21.445 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.997419+0000", 2024-09-21T13:54:21.445 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.017688021000000002 2024-09-21T13:54:21.445 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.445 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.445 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:21.446 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.006063+0000", 2024-09-21T13:54:21.446 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0086435980000000006 2024-09-21T13:54:21.446 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.446 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.446 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:54:21.446 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.006088+0000", 2024-09-21T13:54:21.446 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5437000000000001e-05 2024-09-21T13:54:21.446 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.446 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.446 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:54:21.446 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.006107+0000", 2024-09-21T13:54:21.446 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8238999999999999e-05 2024-09-21T13:54:21.447 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:21.447 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:54:21.447 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:21.447 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.447 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.447 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:201885 7.5 7:a6d798b1:::benchmark_data_smithi049_76798_object65465:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T13:54:21.447 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:43.302360+0000", 2024-09-21T13:54:21.447 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 97.980908759000002, 2024-09-21T13:54:21.447 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.70381638199999996, 2024-09-21T13:54:21.447 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:54:21.447 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:54:21.447 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:21.447 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:54:21.447 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:54:21.448 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:21.448 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 201885 2024-09-21T13:54:21.448 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.448 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:54:21.448 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.448 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:54:21.448 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.302360+0000", 2024-09-21T13:54:21.448 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:21.448 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.448 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.448 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:54:21.448 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.302360+0000", 2024-09-21T13:54:21.448 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:54:21.448 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.449 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.449 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:54:21.449 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.302361+0000", 2024-09-21T13:54:21.449 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.291e-06 2024-09-21T13:54:21.449 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.449 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.449 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:54:21.449 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.302366+0000", 2024-09-21T13:54:21.449 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1619999999999999e-06 2024-09-21T13:54:21.449 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.449 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.449 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:54:21.449 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.302379+0000", 2024-09-21T13:54:21.449 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2824e-05 2024-09-21T13:54:21.450 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.450 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.450 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:54:21.450 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.302385+0000", 2024-09-21T13:54:21.450 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8799999999999996e-06 2024-09-21T13:54:21.450 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.450 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.450 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:21.450 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.302431+0000", 2024-09-21T13:54:21.450 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6733000000000001e-05 2024-09-21T13:54:21.450 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.450 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.450 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:54:21.450 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.302437+0000", 2024-09-21T13:54:21.451 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0170000000000002e-06 2024-09-21T13:54:21.451 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.451 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.451 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:54:21.451 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.977079+0000", 2024-09-21T13:54:21.451 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.67464170999999995 2024-09-21T13:54:21.451 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.451 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.451 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:54:21.451 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.977147+0000", 2024-09-21T13:54:21.451 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8099e-05 2024-09-21T13:54:21.451 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.451 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.451 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2", 2024-09-21T13:54:21.451 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.977215+0000", 2024-09-21T13:54:21.452 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8151000000000005e-05 2024-09-21T13:54:21.452 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.452 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.452 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:54:21.452 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.979736+0000", 2024-09-21T13:54:21.452 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.002520361 2024-09-21T13:54:21.452 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.452 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.452 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:21.452 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.000408+0000", 2024-09-21T13:54:21.452 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.020671948999999998 2024-09-21T13:54:21.452 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.452 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.452 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:21.452 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.006133+0000", 2024-09-21T13:54:21.452 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0057253149999999999 2024-09-21T13:54:21.453 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.453 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.453 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:54:21.453 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.006158+0000", 2024-09-21T13:54:21.453 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4943000000000002e-05 2024-09-21T13:54:21.453 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:54:21.453 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:54:21.453 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:54:21.453 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.006176+0000", 2024-09-21T13:54:21.453 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7946999999999999e-05 2024-09-21T13:54:21.453 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:21.453 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:54:21.453 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:21.453 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:54:21.453 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:54:21.454 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T13:54:21.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:21.455 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:21.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:54:21.456 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-09-21T13:54:21.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:21 smithi049 ceph-mon[30080]: pgmap v262: 148 pgs: 148 active+clean; 69 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.6 KiB/s rd, 11 MiB/s wr, 172 op/s 2024-09-21T13:54:21.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:21 smithi049 ceph-mon[30080]: 5.13 scrub starts 2024-09-21T13:54:21.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:21 smithi049 ceph-mon[30080]: 5.13 scrub ok 2024-09-21T13:54:21.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:21 smithi049 ceph-mon[30080]: from='client.15210 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T13:54:21.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:21 smithi049 ceph-mon[30080]: 4.c scrub starts 2024-09-21T13:54:21.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:21 smithi049 ceph-mon[30080]: 4.c scrub ok 2024-09-21T13:54:21.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:21 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2295127474' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:54:21.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:21 smithi049 ceph-mon[30080]: 5.14 scrub starts 2024-09-21T13:54:21.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:21 smithi049 ceph-mon[30080]: 5.14 scrub ok 2024-09-21T13:54:21.457 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:21 smithi049 ceph-mon[40489]: pgmap v262: 148 pgs: 148 active+clean; 69 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.6 KiB/s rd, 11 MiB/s wr, 172 op/s 2024-09-21T13:54:21.457 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:21 smithi049 ceph-mon[40489]: 5.13 scrub starts 2024-09-21T13:54:21.457 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:21 smithi049 ceph-mon[40489]: 5.13 scrub ok 2024-09-21T13:54:21.457 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:21 smithi049 ceph-mon[40489]: from='client.15210 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T13:54:21.457 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:21 smithi049 ceph-mon[40489]: 4.c scrub starts 2024-09-21T13:54:21.457 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:21 smithi049 ceph-mon[40489]: 4.c scrub ok 2024-09-21T13:54:21.457 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:21 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2295127474' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:54:21.457 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:21 smithi049 ceph-mon[40489]: 5.14 scrub starts 2024-09-21T13:54:21.457 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:21 smithi049 ceph-mon[40489]: 5.14 scrub ok 2024-09-21T13:54:21.557 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:54:21.594 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:21.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:21.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:54:21.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:21 smithi187 ceph-mon[33032]: pgmap v262: 148 pgs: 148 active+clean; 69 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.6 KiB/s rd, 11 MiB/s wr, 172 op/s 2024-09-21T13:54:21.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:21 smithi187 ceph-mon[33032]: 5.13 scrub starts 2024-09-21T13:54:21.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:21 smithi187 ceph-mon[33032]: 5.13 scrub ok 2024-09-21T13:54:21.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:21 smithi187 ceph-mon[33032]: from='client.15210 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T13:54:21.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:21 smithi187 ceph-mon[33032]: 4.c scrub starts 2024-09-21T13:54:21.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:21 smithi187 ceph-mon[33032]: 4.c scrub ok 2024-09-21T13:54:21.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:21 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2295127474' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:54:21.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:21 smithi187 ceph-mon[33032]: 5.14 scrub starts 2024-09-21T13:54:21.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:21 smithi187 ceph-mon[33032]: 5.14 scrub ok 2024-09-21T13:54:21.695 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:54:21.734 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:21.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:21.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:54:21.836 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:54:21.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:21.878 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:21.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:54:21.979 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:54:22.021 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:22.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:22.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:54:22.123 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:54:22.162 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:22.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:22.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:54:22.264 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:22.286 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 4 16 12231 12215 190.789 185.562 0.00474837 0.00523353 2024-09-21T13:54:22.302 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:22.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:22.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:22.404 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:54:22.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:22.447 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:22.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:54:22.548 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:54:22.578 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:22.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:22.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:54:22.605 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:22 smithi187 ceph-mon[33032]: 6.a scrub starts 2024-09-21T13:54:22.605 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:22 smithi187 ceph-mon[33032]: 6.a scrub ok 2024-09-21T13:54:22.605 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:22 smithi187 ceph-mon[33032]: 5.a scrub starts 2024-09-21T13:54:22.606 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:22 smithi187 ceph-mon[33032]: 5.a scrub ok 2024-09-21T13:54:22.606 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:22 smithi187 ceph-mon[33032]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T13:54:22.606 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:22 smithi187 ceph-mon[33032]: Cluster is now healthy 2024-09-21T13:54:22.606 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:22 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2295127474' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T13:54:22.606 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:22 smithi187 ceph-mon[33032]: osdmap e189: 8 total, 8 up, 8 in 2024-09-21T13:54:22.632 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T13:54:22.680 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:54:22.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:22.726 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:22.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:54:22.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:22 smithi049 ceph-mon[40489]: 6.a scrub starts 2024-09-21T13:54:22.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:22 smithi049 ceph-mon[40489]: 6.a scrub ok 2024-09-21T13:54:22.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:22 smithi049 ceph-mon[40489]: 5.a scrub starts 2024-09-21T13:54:22.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:22 smithi049 ceph-mon[40489]: 5.a scrub ok 2024-09-21T13:54:22.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:22 smithi049 ceph-mon[40489]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T13:54:22.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:22 smithi049 ceph-mon[40489]: Cluster is now healthy 2024-09-21T13:54:22.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:22 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2295127474' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T13:54:22.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:22 smithi049 ceph-mon[40489]: osdmap e189: 8 total, 8 up, 8 in 2024-09-21T13:54:22.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:22 smithi049 ceph-mon[30080]: 6.a scrub starts 2024-09-21T13:54:22.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:22 smithi049 ceph-mon[30080]: 6.a scrub ok 2024-09-21T13:54:22.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:22 smithi049 ceph-mon[30080]: 5.a scrub starts 2024-09-21T13:54:22.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:22 smithi049 ceph-mon[30080]: 5.a scrub ok 2024-09-21T13:54:22.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:22 smithi049 ceph-mon[30080]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T13:54:22.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:22 smithi049 ceph-mon[30080]: Cluster is now healthy 2024-09-21T13:54:22.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:22 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2295127474' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T13:54:22.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:22 smithi049 ceph-mon[30080]: osdmap e189: 8 total, 8 up, 8 in 2024-09-21T13:54:22.827 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:22.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:22.854 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:22.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:22.955 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:54:22.987 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:22.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:22.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:54:23.088 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:23.123 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:23.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:23.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:23.224 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:23.250 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:23.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:23.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:23.286 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 5 16 14357 14341 179.206 132.875 0.00897574 0.00557157 2024-09-21T13:54:23.352 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:54:23.392 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:23.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:23.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:54:23.487 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:54:23.487 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:54:23.494 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:23.525 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:54:23.525 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:54:23.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:23.526 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:23.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:23.535 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:54:23.561 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:23 smithi049 ceph-mon[30080]: pgmap v265: 148 pgs: 148 active+clean; 69 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 11 MiB/s wr, 183 op/s 2024-09-21T13:54:23.561 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:23 smithi049 ceph-mon[30080]: 3.d scrub starts 2024-09-21T13:54:23.561 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:23 smithi049 ceph-mon[30080]: 3.d scrub ok 2024-09-21T13:54:23.561 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:23 smithi049 ceph-mon[40489]: pgmap v265: 148 pgs: 148 active+clean; 69 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 11 MiB/s wr, 183 op/s 2024-09-21T13:54:23.561 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:23 smithi049 ceph-mon[40489]: 3.d scrub starts 2024-09-21T13:54:23.561 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:23 smithi049 ceph-mon[40489]: 3.d scrub ok 2024-09-21T13:54:23.561 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:54:23.562 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:54:23.598 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:54:23.599 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:54:23.627 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:23.633 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:54:23.633 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:54:23.656 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:23.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:23.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:23.671 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:54:23.671 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:54:23.692 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:54:23.692 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:54:23.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:23 smithi187 ceph-mon[33032]: pgmap v265: 148 pgs: 148 active+clean; 69 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 11 MiB/s wr, 183 op/s 2024-09-21T13:54:23.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:23 smithi187 ceph-mon[33032]: 3.d scrub starts 2024-09-21T13:54:23.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:23 smithi187 ceph-mon[33032]: 3.d scrub ok 2024-09-21T13:54:23.710 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:54:23.710 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:54:23.758 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:54:23.781 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:23.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:23.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:54:23.882 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:54:23.916 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:23.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:23.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:54:24.018 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:54:24.050 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:24.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:24.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:54:24.152 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:24.183 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:24.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:24.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:24.285 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:54:24.286 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 6 16 16761 16745 174.379 150.25 0.00344993 0.00572848 2024-09-21T13:54:24.316 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:24.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:24.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:54:24.418 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:54:24.449 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:24.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:24.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:54:24.508 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T13:54:24.551 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:54:24.596 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:24.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:24.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:54:24.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:24 smithi187 ceph-mon[33032]: 4.11 scrub starts 2024-09-21T13:54:24.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:24 smithi187 ceph-mon[33032]: 4.11 scrub ok 2024-09-21T13:54:24.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:24 smithi187 ceph-mon[33032]: 6.19 scrub starts 2024-09-21T13:54:24.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:24 smithi187 ceph-mon[33032]: 6.19 scrub ok 2024-09-21T13:54:24.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:24 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3109832156' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:54:24.699 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:54:24.745 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:24.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:24.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:54:24.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:24 smithi049 ceph-mon[30080]: 4.11 scrub starts 2024-09-21T13:54:24.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:24 smithi049 ceph-mon[30080]: 4.11 scrub ok 2024-09-21T13:54:24.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:24 smithi049 ceph-mon[30080]: 6.19 scrub starts 2024-09-21T13:54:24.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:24 smithi049 ceph-mon[30080]: 6.19 scrub ok 2024-09-21T13:54:24.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:24 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3109832156' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:54:24.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:24 smithi049 ceph-mon[40489]: 4.11 scrub starts 2024-09-21T13:54:24.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:24 smithi049 ceph-mon[40489]: 4.11 scrub ok 2024-09-21T13:54:24.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:24 smithi049 ceph-mon[40489]: 6.19 scrub starts 2024-09-21T13:54:24.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:24 smithi049 ceph-mon[40489]: 6.19 scrub ok 2024-09-21T13:54:24.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:24 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3109832156' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:54:24.847 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:24.877 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:24.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:24.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:24.978 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:54:25.018 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:25.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:25.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:54:25.120 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:25.154 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:25.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:25.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:25.255 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:25.287 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 7 16 19403 19387 173.055 165.125 0.00525277 0.00577272 2024-09-21T13:54:25.287 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:25.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:25.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:25.386 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T13:54:25.386 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:54:25.389 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:25.431 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:25.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:25.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:25.457 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:25 smithi049 ceph-mon[40489]: pgmap v266: 148 pgs: 148 active+clean; 787 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-09-21T13:54:25.457 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:25 smithi049 ceph-mon[40489]: osdmap e190: 8 total, 8 up, 8 in 2024-09-21T13:54:25.457 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:25 smithi049 ceph-mon[30080]: pgmap v266: 148 pgs: 148 active+clean; 787 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-09-21T13:54:25.457 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:25 smithi049 ceph-mon[30080]: osdmap e190: 8 total, 8 up, 8 in 2024-09-21T13:54:25.533 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:54:25.571 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:25.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:25.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:54:25.673 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:25.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:25 smithi187 ceph-mon[33032]: pgmap v266: 148 pgs: 148 active+clean; 787 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-09-21T13:54:25.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:25 smithi187 ceph-mon[33032]: osdmap e190: 8 total, 8 up, 8 in 2024-09-21T13:54:25.705 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:25.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:25.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:25.807 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:54:25.839 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:25.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:25.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:54:25.941 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:25.974 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:25.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:25.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:26.075 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:54:26.113 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:26.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:26.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:54:26.215 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:54:26.228 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T13:54:26.254 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 2] 2024-09-21T13:54:26.254 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T13:54:26.254 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-21T13:54:26.254 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T13:54:26.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:26.267 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:26.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:54:26.287 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 8 16 21935 21919 171.203 158.25 0.0035368 0.00583473 2024-09-21T13:54:26.307 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:54:26.368 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:54:26.422 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:26.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:26.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:54:26.524 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:54:26.552 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:26.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:26.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:54:26.653 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:54:26.709 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:26 smithi049 ceph-mon[30080]: 5.0 scrub starts 2024-09-21T13:54:26.709 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:26 smithi049 ceph-mon[30080]: 5.0 scrub ok 2024-09-21T13:54:26.709 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:26 smithi049 ceph-mon[30080]: 2.2 scrub starts 2024-09-21T13:54:26.710 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:26 smithi049 ceph-mon[30080]: 2.2 scrub ok 2024-09-21T13:54:26.710 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:26 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:54:26.710 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:26 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:54:26.710 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:26 smithi049 ceph-mon[30080]: 5.6 scrub starts 2024-09-21T13:54:26.710 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:26 smithi049 ceph-mon[30080]: 5.6 scrub ok 2024-09-21T13:54:26.710 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:26 smithi049 ceph-mon[40489]: 5.0 scrub starts 2024-09-21T13:54:26.710 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:26 smithi049 ceph-mon[40489]: 5.0 scrub ok 2024-09-21T13:54:26.710 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:26 smithi049 ceph-mon[40489]: 2.2 scrub starts 2024-09-21T13:54:26.710 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:26 smithi049 ceph-mon[40489]: 2.2 scrub ok 2024-09-21T13:54:26.710 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:26 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:54:26.710 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:26 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:54:26.711 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:26 smithi049 ceph-mon[40489]: 5.6 scrub starts 2024-09-21T13:54:26.711 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:26 smithi049 ceph-mon[40489]: 5.6 scrub ok 2024-09-21T13:54:26.713 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:26.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:26.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:54:26.788 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:26 smithi187 ceph-mon[33032]: 5.0 scrub starts 2024-09-21T13:54:26.788 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:26 smithi187 ceph-mon[33032]: 5.0 scrub ok 2024-09-21T13:54:26.788 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:26 smithi187 ceph-mon[33032]: 2.2 scrub starts 2024-09-21T13:54:26.788 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:26 smithi187 ceph-mon[33032]: 2.2 scrub ok 2024-09-21T13:54:26.789 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:26 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:54:26.789 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:26 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:54:26.789 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:26 smithi187 ceph-mon[33032]: 5.6 scrub starts 2024-09-21T13:54:26.789 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:26 smithi187 ceph-mon[33032]: 5.6 scrub ok 2024-09-21T13:54:26.815 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:54:26.848 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:26.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:26.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:54:26.949 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:54:26.980 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:26.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:26.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:54:27.082 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:54:27.099 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:54:27.114 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:54:27.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:27.141 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:27.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:54:27.241 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:27.270 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:27.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:27.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:27.287 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 9 16 24005 23989 166.538 129.375 0.00392671 0.00599573 2024-09-21T13:54:27.372 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:27.410 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:27.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:27.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:27.494 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T13:54:27.512 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:54:27.543 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:27 smithi049 ceph-mon[30080]: pgmap v268: 148 pgs: 148 active+clean; 947 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 146 MiB/s wr, 2.34k op/s 2024-09-21T13:54:27.543 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:27 smithi049 ceph-mon[30080]: 6.18 scrub starts 2024-09-21T13:54:27.543 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:27 smithi049 ceph-mon[30080]: 6.18 scrub ok 2024-09-21T13:54:27.543 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:27 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:54:27.543 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:27 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:54:27.543 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:27 smithi049 ceph-mon[30080]: osdmap e191: 8 total, 8 up, 8 in 2024-09-21T13:54:27.543 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:27 smithi049 ceph-mon[30080]: 5.1a scrub starts 2024-09-21T13:54:27.543 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:27 smithi049 ceph-mon[30080]: 5.1a scrub ok 2024-09-21T13:54:27.543 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:27 smithi049 ceph-mon[40489]: pgmap v268: 148 pgs: 148 active+clean; 947 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 146 MiB/s wr, 2.34k op/s 2024-09-21T13:54:27.543 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:27 smithi049 ceph-mon[40489]: 6.18 scrub starts 2024-09-21T13:54:27.544 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:27 smithi049 ceph-mon[40489]: 6.18 scrub ok 2024-09-21T13:54:27.544 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:27 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:54:27.544 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:27 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:54:27.544 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:27 smithi049 ceph-mon[40489]: osdmap e191: 8 total, 8 up, 8 in 2024-09-21T13:54:27.544 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:27 smithi049 ceph-mon[40489]: 5.1a scrub starts 2024-09-21T13:54:27.544 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:27 smithi049 ceph-mon[40489]: 5.1a scrub ok 2024-09-21T13:54:27.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:27.582 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:27.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:54:27.683 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:54:27.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:27 smithi187 ceph-mon[33032]: pgmap v268: 148 pgs: 148 active+clean; 947 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 146 MiB/s wr, 2.34k op/s 2024-09-21T13:54:27.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:27 smithi187 ceph-mon[33032]: 6.18 scrub starts 2024-09-21T13:54:27.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:27 smithi187 ceph-mon[33032]: 6.18 scrub ok 2024-09-21T13:54:27.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:27 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:54:27.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:27 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:54:27.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:27 smithi187 ceph-mon[33032]: osdmap e191: 8 total, 8 up, 8 in 2024-09-21T13:54:27.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:27 smithi187 ceph-mon[33032]: 5.1a scrub starts 2024-09-21T13:54:27.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:27 smithi187 ceph-mon[33032]: 5.1a scrub ok 2024-09-21T13:54:27.740 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:27.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:27.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:54:27.842 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:54:27.871 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:27.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:27.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:54:27.973 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:54:28.006 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:28.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:28.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:54:28.108 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:54:28.151 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:28.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:28.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:54:28.204 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:54:28.207 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-09-21T13:54:28.253 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:54:28.283 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T13:54:28.283 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:54:28.288 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 10 16 26028 26012 162.526 126.438 0.0105996 0.00614499 2024-09-21T13:54:28.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:28.309 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:28.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:54:28.410 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:54:28.447 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:28.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:28.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:54:28.474 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-09-21T13:54:28.549 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:28.587 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:28.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:28.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:28.689 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:54:28.727 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:28.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:28.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:54:28.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:28 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1825679263' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:54:28.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:28 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:54:28.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:28 smithi049 ceph-mon[40489]: 6.16 scrub starts 2024-09-21T13:54:28.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:28 smithi049 ceph-mon[40489]: 6.16 scrub ok 2024-09-21T13:54:28.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:28 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:54:28.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:28 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:54:28.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:28 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1825679263' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:54:28.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:28 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:54:28.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:28 smithi049 ceph-mon[30080]: 6.16 scrub starts 2024-09-21T13:54:28.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:28 smithi049 ceph-mon[30080]: 6.16 scrub ok 2024-09-21T13:54:28.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:28 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:54:28.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:28 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:54:28.830 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:54:28.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:28.858 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:28.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:54:28.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:28 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1825679263' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:54:28.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:28 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:54:28.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:28 smithi187 ceph-mon[33032]: 6.16 scrub starts 2024-09-21T13:54:28.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:28 smithi187 ceph-mon[33032]: 6.16 scrub ok 2024-09-21T13:54:28.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:28 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:54:28.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:28 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:54:28.959 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:28.994 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:28.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:28.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:29.095 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:54:29.116 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:29.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:29.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:54:29.217 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:29.257 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:29.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:29.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:29.288 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 11 16 28029 28013 159.12 125.062 0.00504103 0.00627935 2024-09-21T13:54:29.359 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:29.398 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:29.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:29.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:29.488 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:54:29 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:54:29.499 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:29.532 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:29.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:29.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:29.634 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:29.667 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:29.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:29.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:29.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:29 smithi049 ceph-mon[40489]: pgmap v270: 148 pgs: 148 active+clean; 947 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 805 B/s rd, 138 MiB/s wr, 2.21k op/s 2024-09-21T13:54:29.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:29 smithi049 ceph-mon[40489]: 5.17 scrub starts 2024-09-21T13:54:29.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:29 smithi049 ceph-mon[40489]: 5.17 scrub ok 2024-09-21T13:54:29.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:29 smithi049 ceph-mon[40489]: from='client.25133 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T13:54:29.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:29 smithi049 ceph-mon[40489]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:54:29.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:29 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T13:54:29.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:29 smithi049 ceph-mon[40489]: osdmap e192: 8 total, 8 up, 8 in 2024-09-21T13:54:29.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:29 smithi049 ceph-mon[30080]: pgmap v270: 148 pgs: 148 active+clean; 947 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 805 B/s rd, 138 MiB/s wr, 2.21k op/s 2024-09-21T13:54:29.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:29 smithi049 ceph-mon[30080]: 5.17 scrub starts 2024-09-21T13:54:29.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:29 smithi049 ceph-mon[30080]: 5.17 scrub ok 2024-09-21T13:54:29.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:29 smithi049 ceph-mon[30080]: from='client.25133 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T13:54:29.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:29 smithi049 ceph-mon[30080]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:54:29.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:29 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T13:54:29.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:29 smithi049 ceph-mon[30080]: osdmap e192: 8 total, 8 up, 8 in 2024-09-21T13:54:29.768 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:54:29.802 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:29.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:29.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:54:29.904 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:54:29.934 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:29.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:29.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:54:29.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:29 smithi187 ceph-mon[33032]: pgmap v270: 148 pgs: 148 active+clean; 947 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 805 B/s rd, 138 MiB/s wr, 2.21k op/s 2024-09-21T13:54:29.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:29 smithi187 ceph-mon[33032]: 5.17 scrub starts 2024-09-21T13:54:29.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:29 smithi187 ceph-mon[33032]: 5.17 scrub ok 2024-09-21T13:54:29.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:29 smithi187 ceph-mon[33032]: from='client.25133 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T13:54:29.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:29 smithi187 ceph-mon[33032]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:54:29.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:29 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T13:54:29.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:29 smithi187 ceph-mon[33032]: osdmap e192: 8 total, 8 up, 8 in 2024-09-21T13:54:30.036 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:30.066 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:30.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:30.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:30.168 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:54:30.208 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:30.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:30.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:54:30.288 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 12 16 29998 29982 156.114 123.062 0.00333214 0.00637579 2024-09-21T13:54:30.310 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:54:30.343 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:30.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:30.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:54:30.444 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:30.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:30.475 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:30.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:30.576 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:30.597 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:30.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:30.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:30.698 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:54:30.735 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:30.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:30.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:54:30.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:30 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:54:30.755 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:54:30 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:54:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:54:30.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:30 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:54:30.836 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:54:30.871 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:30.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:30.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:54:30.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:30 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:54:30.972 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:31.013 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:31.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:31.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:31.115 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:54:31.155 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:31.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:31.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:54:31.257 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:54:31.277 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:31.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:31.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:54:31.288 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 13 16 31702 31686 152.297 106.5 0.00191158 0.00652773 2024-09-21T13:54:31.379 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:54:31.418 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:31.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:31.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:54:31.520 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:31.549 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:31.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:31.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:31.650 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:31.672 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:31.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:31.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:31.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:31 smithi049 ceph-mon[30080]: pgmap v272: 148 pgs: 148 active+clean; 1.6 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 147 MiB/s wr, 2.36k op/s 2024-09-21T13:54:31.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:31 smithi049 ceph-mon[40489]: pgmap v272: 148 pgs: 148 active+clean; 1.6 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 147 MiB/s wr, 2.36k op/s 2024-09-21T13:54:31.774 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:54:31.809 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:31.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:31.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:54:31.911 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:54:31.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:31 smithi187 ceph-mon[33032]: pgmap v272: 148 pgs: 148 active+clean; 1.6 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 147 MiB/s wr, 2.36k op/s 2024-09-21T13:54:31.948 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:31.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:31.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:54:32.050 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:32.082 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:32.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:32.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:32.184 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:54:32.206 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:32.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:32.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:54:32.288 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 14 16 33310 33294 148.597 100.5 0.00271417 0.0067238 2024-09-21T13:54:32.308 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:54:32.342 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:32.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:32.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:54:32.446 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:54:32.479 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:32.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:32.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:54:32.582 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:32.603 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:32.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:32.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:32.705 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:54:32.743 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:32.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:32.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:54:32.845 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:54:32.886 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:32.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:32.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:54:32.988 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:33.008 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:33.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:33.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:33.110 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:54:33.134 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:33.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:33.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:54:33.237 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:33.274 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:33.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:33.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:33.288 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 15 15 34822 34807 144.995 94.5625 0.00505904 0.00689068 2024-09-21T13:54:33.376 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:33.408 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:33.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:33.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:33.510 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:33.546 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:33.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:33.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:33.647 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:54:33.670 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:33.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:33.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:54:33.772 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:33.803 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:33.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:33.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:33.905 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:54:33.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:33.936 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:33.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:54:33.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:33 smithi187 ceph-mon[33032]: pgmap v273: 148 pgs: 148 active+clean; 1.6 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 746 B/s rd, 106 MiB/s wr, 1.69k op/s 2024-09-21T13:54:34.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:33 smithi049 ceph-mon[40489]: pgmap v273: 148 pgs: 148 active+clean; 1.6 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 746 B/s rd, 106 MiB/s wr, 1.69k op/s 2024-09-21T13:54:34.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:33 smithi049 ceph-mon[30080]: pgmap v273: 148 pgs: 148 active+clean; 1.6 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 746 B/s rd, 106 MiB/s wr, 1.69k op/s 2024-09-21T13:54:34.037 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:54:34.067 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:34.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:34.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:54:34.169 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:54:34.227 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:34.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:34.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:54:34.288 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 16 16 36661 36645 143.112 114.875 0.00604281 0.00698258 2024-09-21T13:54:34.329 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:54:34.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:34.370 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:34.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:54:34.471 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:54:34.492 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:34.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:34.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:54:34.594 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:54:34.635 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:34.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:34.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:54:34.736 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:34.768 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:34.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:34.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:34.870 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:54:34.894 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:34.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:34.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:54:34.996 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:35.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:35.037 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:35.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:35.138 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:54:35.175 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:35.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:35.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:54:35.277 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:54:35.289 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 17 16 38255 38239 140.551 99.625 0.00778934 0.00710989 2024-09-21T13:54:35.312 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:35.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:35.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:54:35.414 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:54:35.438 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:35.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:35.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:54:35.540 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:35.562 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:35.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:35.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:35.664 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:35.693 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:35.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:35.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:35.795 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:35.827 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:35.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:35.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:35.928 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:35.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:35 smithi187 ceph-mon[33032]: pgmap v274: 148 pgs: 148 active+clean; 2.1 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 146 MiB/s wr, 2.34k op/s 2024-09-21T13:54:35.966 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:35.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:35.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:36.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:35 smithi049 ceph-mon[40489]: pgmap v274: 148 pgs: 148 active+clean; 2.1 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 146 MiB/s wr, 2.34k op/s 2024-09-21T13:54:36.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:35 smithi049 ceph-mon[30080]: pgmap v274: 148 pgs: 148 active+clean; 2.1 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 146 MiB/s wr, 2.34k op/s 2024-09-21T13:54:36.068 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:54:36.101 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:36.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:36.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:54:36.203 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:54:36.237 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:36.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:36.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:54:36.289 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 18 15 39940 39925 138.596 105.375 0.00712115 0.00721015 2024-09-21T13:54:36.338 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:54:36.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:36.375 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:36.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:54:36.414 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T13:54:36.423 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:54:36.476 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:54:36.507 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:36.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:36.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:54:36.608 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:54:36.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:36.660 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:36.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:54:36.672 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:36 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:54:36.672 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:36 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:54:36.672 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:36 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:54:36.672 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:36 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:54:36.761 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:36.794 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:36.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:36.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:36.897 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:36.927 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:36.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:36.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:36.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:36 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:54:36.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:36 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:54:37.029 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:37.068 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:37.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:37.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:37.169 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:37.199 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:37.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:37.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:37.289 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 19 16 41852 41836 137.588 119.438 0.00394193 0.00726233 2024-09-21T13:54:37.300 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:54:37.320 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":270,"stamp":"2024-09-21T13:54:27.782562+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":992877083,"num_objects":15340,"num_object_clones":0,"num_object_copies":46020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15340,"num_whiteouts":0,"num_read":1164,"num_read_kb":1021,"num_write":15690,"num_write_kb":970697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":590588,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15700,"ondisk_log_size":15700,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":4257780,"kb_used_data":2934036,"kb_used_omap":15,"kb_used_meta":1323504,"kb_avail":745703436,"statfs":{"total":767960285184,"available":763600318464,"internally_reserved":0,"allocated":3004452864,"data_stored":3000358601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15598,"internal_metadata":1355268882},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,9,5,6],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":920518656,"num_objects":14046,"num_object_clones":0,"num_object_copies":42138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14046,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":14046,"num_write_kb":898944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.354766"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":221,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.440262+0000","last_change":"2024-09-21T13:52:26.677736+0000","last_active":"2024-09-21T13:54:21.440262+0000","last_peered":"2024-09-21T13:54:21.440262+0000","last_clean":"2024-09-21T13:54:21.440262+0000","last_became_active":"2024-09-21T13:52:04.317322+0000","last_became_peered":"2024-09-21T13:52:04.317322+0000","last_unstale":"2024-09-21T13:54:21.440262+0000","last_undegraded":"2024-09-21T13:54:21.440262+0000","last_fullsized":"2024-09-21T13:54:21.440262+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:26.677689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:52:26.677689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:46.286464+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"68'9","reported_seq":297,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.431624+0000","last_change":"2024-09-21T13:54:18.411247+0000","last_active":"2024-09-21T13:54:21.431624+0000","last_peered":"2024-09-21T13:54:21.431624+0000","last_clean":"2024-09-21T13:54:21.431624+0000","last_became_active":"2024-09-21T13:49:00.055210+0000","last_became_peered":"2024-09-21T13:49:00.055210+0000","last_unstale":"2024-09-21T13:54:21.431624+0000","last_undegraded":"2024-09-21T13:54:21.431624+0000","last_fullsized":"2024-09-21T13:54:21.431624+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:54:18.410821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:18.410821+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:40.272886+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":270,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.429335+0000","last_change":"2024-09-21T13:50:28.058990+0000","last_active":"2024-09-21T13:54:21.429335+0000","last_peered":"2024-09-21T13:54:21.429335+0000","last_clean":"2024-09-21T13:54:21.429335+0000","last_became_active":"2024-09-21T13:49:02.060984+0000","last_became_peered":"2024-09-21T13:49:02.060984+0000","last_unstale":"2024-09-21T13:54:21.429335+0000","last_undegraded":"2024-09-21T13:54:21.429335+0000","last_fullsized":"2024-09-21T13:54:21.429335+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:28.058918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:28.058918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:48.620137+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":265,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.431860+0000","last_change":"2024-09-21T13:53:21.005444+0000","last_active":"2024-09-21T13:54:21.431860+0000","last_peered":"2024-09-21T13:54:21.431860+0000","last_clean":"2024-09-21T13:54:21.431860+0000","last_became_active":"2024-09-21T13:49:04.072460+0000","last_became_peered":"2024-09-21T13:49:04.072460+0000","last_unstale":"2024-09-21T13:54:21.431860+0000","last_undegraded":"2024-09-21T13:54:21.431860+0000","last_fullsized":"2024-09-21T13:54:21.431860+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:21.005352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:21.005352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:43.773187+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":265,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.440483+0000","last_change":"2024-09-21T13:51:18.572788+0000","last_active":"2024-09-21T13:54:21.440483+0000","last_peered":"2024-09-21T13:54:21.440483+0000","last_clean":"2024-09-21T13:54:21.440483+0000","last_became_active":"2024-09-21T13:49:04.433965+0000","last_became_peered":"2024-09-21T13:49:04.433965+0000","last_unstale":"2024-09-21T13:54:21.440483+0000","last_undegraded":"2024-09-21T13:54:21.440483+0000","last_fullsized":"2024-09-21T13:54:21.440483+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:18.572729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:51:18.572729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":288,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.441426+0000","last_change":"2024-09-21T13:53:19.910708+0000","last_active":"2024-09-21T13:54:21.441426+0000","last_peered":"2024-09-21T13:54:21.441426+0000","last_clean":"2024-09-21T13:54:21.441426+0000","last_became_active":"2024-09-21T13:52:04.316819+0000","last_became_peered":"2024-09-21T13:52:04.316819+0000","last_unstale":"2024-09-21T13:54:21.441426+0000","last_undegraded":"2024-09-21T13:54:21.441426+0000","last_fullsized":"2024-09-21T13:54:21.441426+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:19.910638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:53:19.910638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:27.797531+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"68'15","reported_seq":309,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.439957+0000","last_change":"2024-09-21T13:54:17.024113+0000","last_active":"2024-09-21T13:54:21.439957+0000","last_peered":"2024-09-21T13:54:21.439957+0000","last_clean":"2024-09-21T13:54:21.439957+0000","last_became_active":"2024-09-21T13:52:04.315101+0000","last_became_peered":"2024-09-21T13:52:04.315101+0000","last_unstale":"2024-09-21T13:54:21.439957+0000","last_undegraded":"2024-09-21T13:54:21.439957+0000","last_fullsized":"2024-09-21T13:54:21.439957+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:54:17.024053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:17.024053+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:34.778666+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":277,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.431618+0000","last_change":"2024-09-21T13:51:43.040397+0000","last_active":"2024-09-21T13:54:21.431618+0000","last_peered":"2024-09-21T13:54:21.431618+0000","last_clean":"2024-09-21T13:54:21.431618+0000","last_became_active":"2024-09-21T13:49:02.063687+0000","last_became_peered":"2024-09-21T13:49:02.063687+0000","last_unstale":"2024-09-21T13:54:21.431618+0000","last_undegraded":"2024-09-21T13:54:21.431618+0000","last_fullsized":"2024-09-21T13:54:21.431618+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:43.040348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:43.040348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:08.325545+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":294,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.438465+0000","last_change":"2024-09-21T13:53:59.104109+0000","last_active":"2024-09-21T13:54:21.438465+0000","last_peered":"2024-09-21T13:54:21.438465+0000","last_clean":"2024-09-21T13:54:21.438465+0000","last_became_active":"2024-09-21T13:48:58.050695+0000","last_became_peered":"2024-09-21T13:48:58.050695+0000","last_unstale":"2024-09-21T13:54:21.438465+0000","last_undegraded":"2024-09-21T13:54:21.438465+0000","last_fullsized":"2024-09-21T13:54:21.438465+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:59.104062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:53:59.104062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:28.421324+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"68'9","reported_seq":297,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.437364+0000","last_change":"2024-09-21T13:53:58.308240+0000","last_active":"2024-09-21T13:54:21.437364+0000","last_peered":"2024-09-21T13:54:21.437364+0000","last_clean":"2024-09-21T13:54:21.437364+0000","last_became_active":"2024-09-21T13:49:00.059382+0000","last_became_peered":"2024-09-21T13:49:00.059382+0000","last_unstale":"2024-09-21T13:54:21.437364+0000","last_undegraded":"2024-09-21T13:54:21.437364+0000","last_fullsized":"2024-09-21T13:54:21.437364+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:53:58.308144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:58.308144+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:27.655659+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":262,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.433437+0000","last_change":"2024-09-21T13:49:02.066978+0000","last_active":"2024-09-21T13:54:21.433437+0000","last_peered":"2024-09-21T13:54:21.433437+0000","last_clean":"2024-09-21T13:54:21.433437+0000","last_became_active":"2024-09-21T13:49:02.066693+0000","last_became_peered":"2024-09-21T13:49:02.066693+0000","last_unstale":"2024-09-21T13:54:21.433437+0000","last_undegraded":"2024-09-21T13:54:21.433437+0000","last_fullsized":"2024-09-21T13:54:21.433437+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:12.027057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":218,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.429122+0000","last_change":"2024-09-21T13:52:28.417865+0000","last_active":"2024-09-21T13:54:21.429122+0000","last_peered":"2024-09-21T13:54:21.429122+0000","last_clean":"2024-09-21T13:54:21.429122+0000","last_became_active":"2024-09-21T13:50:29.689381+0000","last_became_peered":"2024-09-21T13:50:29.689381+0000","last_unstale":"2024-09-21T13:54:21.429122+0000","last_undegraded":"2024-09-21T13:54:21.429122+0000","last_fullsized":"2024-09-21T13:54:21.429122+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:28.417690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:28.417690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":278,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.438398+0000","last_change":"2024-09-21T13:50:39.480995+0000","last_active":"2024-09-21T13:54:21.438398+0000","last_peered":"2024-09-21T13:54:21.438398+0000","last_clean":"2024-09-21T13:54:21.438398+0000","last_became_active":"2024-09-21T13:48:58.050906+0000","last_became_peered":"2024-09-21T13:48:58.050906+0000","last_unstale":"2024-09-21T13:54:21.438398+0000","last_undegraded":"2024-09-21T13:54:21.438398+0000","last_fullsized":"2024-09-21T13:54:21.438398+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:39.480930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:39.480930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:34.214082+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"68'5","reported_seq":307,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.431916+0000","last_change":"2024-09-21T13:54:04.271336+0000","last_active":"2024-09-21T13:54:21.431916+0000","last_peered":"2024-09-21T13:54:21.431916+0000","last_clean":"2024-09-21T13:54:21.431916+0000","last_became_active":"2024-09-21T13:52:04.318515+0000","last_became_peered":"2024-09-21T13:52:04.318515+0000","last_unstale":"2024-09-21T13:54:21.431916+0000","last_undegraded":"2024-09-21T13:54:21.431916+0000","last_fullsized":"2024-09-21T13:54:21.431916+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-21T13:54:04.271280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:04.271280+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:18.236498+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":269,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.442785+0000","last_change":"2024-09-21T13:50:40.188797+0000","last_active":"2024-09-21T13:54:21.442785+0000","last_peered":"2024-09-21T13:54:21.442785+0000","last_clean":"2024-09-21T13:54:21.442785+0000","last_became_active":"2024-09-21T13:49:02.060212+0000","last_became_peered":"2024-09-21T13:49:02.060212+0000","last_unstale":"2024-09-21T13:54:21.442785+0000","last_undegraded":"2024-09-21T13:54:21.442785+0000","last_fullsized":"2024-09-21T13:54:21.442785+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:40.188712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:40.188712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":282,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:23.043040+0000","last_change":"2024-09-21T13:54:23.043040+0000","last_active":"2024-09-21T13:54:23.043040+0000","last_peered":"2024-09-21T13:54:23.043040+0000","last_clean":"2024-09-21T13:54:23.043040+0000","last_became_active":"2024-09-21T13:49:04.078026+0000","last_became_peered":"2024-09-21T13:49:04.078026+0000","last_unstale":"2024-09-21T13:54:23.043040+0000","last_undegraded":"2024-09-21T13:54:23.043040+0000","last_fullsized":"2024-09-21T13:54:23.043040+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:23.042969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:54:23.042969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:46.405005+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":265,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.437459+0000","last_change":"2024-09-21T13:50:38.027288+0000","last_active":"2024-09-21T13:54:21.437459+0000","last_peered":"2024-09-21T13:54:21.437459+0000","last_clean":"2024-09-21T13:54:21.437459+0000","last_became_active":"2024-09-21T13:49:04.431422+0000","last_became_peered":"2024-09-21T13:49:04.431422+0000","last_unstale":"2024-09-21T13:54:21.437459+0000","last_undegraded":"2024-09-21T13:54:21.437459+0000","last_fullsized":"2024-09-21T13:54:21.437459+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:38.027227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:38.027227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:49.236040+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":209,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.436955+0000","last_change":"2024-09-21T13:54:06.280941+0000","last_active":"2024-09-21T13:54:21.436955+0000","last_peered":"2024-09-21T13:54:21.436955+0000","last_clean":"2024-09-21T13:54:21.436955+0000","last_became_active":"2024-09-21T13:50:29.688282+0000","last_became_peered":"2024-09-21T13:50:29.688282+0000","last_unstale":"2024-09-21T13:54:21.436955+0000","last_undegraded":"2024-09-21T13:54:21.436955+0000","last_fullsized":"2024-09-21T13:54:21.436955+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:06.280894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:06.280894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:07.185880+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"68'15","reported_seq":242,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T13:54:26.476007+0000","last_change":"2024-09-21T13:53:32.929362+0000","last_active":"2024-09-21T13:54:26.476007+0000","last_peered":"2024-09-21T13:54:26.476007+0000","last_clean":"2024-09-21T13:54:26.476007+0000","last_became_active":"2024-09-21T13:52:04.324273+0000","last_became_peered":"2024-09-21T13:52:04.324273+0000","last_unstale":"2024-09-21T13:54:26.476007+0000","last_undegraded":"2024-09-21T13:54:26.476007+0000","last_fullsized":"2024-09-21T13:54:26.476007+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:53:32.929017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:32.929017+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:37.476556+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":278,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.431155+0000","last_change":"2024-09-21T13:52:26.372217+0000","last_active":"2024-09-21T13:54:21.431155+0000","last_peered":"2024-09-21T13:54:21.431155+0000","last_clean":"2024-09-21T13:54:21.431155+0000","last_became_active":"2024-09-21T13:49:02.066796+0000","last_became_peered":"2024-09-21T13:49:02.066796+0000","last_unstale":"2024-09-21T13:54:21.431155+0000","last_undegraded":"2024-09-21T13:54:21.431155+0000","last_fullsized":"2024-09-21T13:54:21.431155+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:26.372144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:52:26.372144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":265,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.441141+0000","last_change":"2024-09-21T13:53:57.962327+0000","last_active":"2024-09-21T13:54:21.441141+0000","last_peered":"2024-09-21T13:54:21.441141+0000","last_clean":"2024-09-21T13:54:21.441141+0000","last_became_active":"2024-09-21T13:49:04.434049+0000","last_became_peered":"2024-09-21T13:49:04.434049+0000","last_unstale":"2024-09-21T13:54:21.441141+0000","last_undegraded":"2024-09-21T13:54:21.441141+0000","last_fullsized":"2024-09-21T13:54:21.441141+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:57.962266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:57.962266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:07.827936+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":288,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.437459+0000","last_change":"2024-09-21T13:52:52.142682+0000","last_active":"2024-09-21T13:54:21.437459+0000","last_peered":"2024-09-21T13:54:21.437459+0000","last_clean":"2024-09-21T13:54:21.437459+0000","last_became_active":"2024-09-21T13:52:04.316016+0000","last_became_peered":"2024-09-21T13:52:04.316016+0000","last_unstale":"2024-09-21T13:54:21.437459+0000","last_undegraded":"2024-09-21T13:54:21.437459+0000","last_fullsized":"2024-09-21T13:54:21.437459+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:52.142638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:52:52.142638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"68'12","reported_seq":296,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.440847+0000","last_change":"2024-09-21T13:51:52.819966+0000","last_active":"2024-09-21T13:54:21.440847+0000","last_peered":"2024-09-21T13:54:21.440847+0000","last_clean":"2024-09-21T13:54:21.440847+0000","last_became_active":"2024-09-21T13:49:00.061867+0000","last_became_peered":"2024-09-21T13:49:00.061867+0000","last_unstale":"2024-09-21T13:54:21.440847+0000","last_undegraded":"2024-09-21T13:54:21.440847+0000","last_fullsized":"2024-09-21T13:54:21.440847+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'12","last_scrub_stamp":"2024-09-21T13:51:52.819879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:52.819879+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":272,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.437578+0000","last_change":"2024-09-21T13:52:04.321297+0000","last_active":"2024-09-21T13:54:21.437578+0000","last_peered":"2024-09-21T13:54:21.437578+0000","last_clean":"2024-09-21T13:54:21.437578+0000","last_became_active":"2024-09-21T13:52:04.320800+0000","last_became_peered":"2024-09-21T13:52:04.320800+0000","last_unstale":"2024-09-21T13:54:21.437578+0000","last_undegraded":"2024-09-21T13:54:21.437578+0000","last_fullsized":"2024-09-21T13:54:21.437578+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:15.423635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":277,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.442283+0000","last_change":"2024-09-21T13:52:54.580070+0000","last_active":"2024-09-21T13:54:21.442283+0000","last_peered":"2024-09-21T13:54:21.442283+0000","last_clean":"2024-09-21T13:54:21.442283+0000","last_became_active":"2024-09-21T13:52:04.321358+0000","last_became_peered":"2024-09-21T13:52:04.321358+0000","last_unstale":"2024-09-21T13:54:21.442283+0000","last_undegraded":"2024-09-21T13:54:21.442283+0000","last_fullsized":"2024-09-21T13:54:21.442283+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T13:52:54.580022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:54.580022+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":287,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.429475+0000","last_change":"2024-09-21T13:52:43.516442+0000","last_active":"2024-09-21T13:54:21.429475+0000","last_peered":"2024-09-21T13:54:21.429475+0000","last_clean":"2024-09-21T13:54:21.429475+0000","last_became_active":"2024-09-21T13:48:58.048972+0000","last_became_peered":"2024-09-21T13:48:58.048972+0000","last_unstale":"2024-09-21T13:54:21.429475+0000","last_undegraded":"2024-09-21T13:54:21.429475+0000","last_fullsized":"2024-09-21T13:54:21.429475+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:52:43.516221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:52:43.516221+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"68'10","reported_seq":244,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.443739+0000","last_change":"2024-09-21T13:52:56.526032+0000","last_active":"2024-09-21T13:54:21.443739+0000","last_peered":"2024-09-21T13:54:21.443739+0000","last_clean":"2024-09-21T13:54:21.443739+0000","last_became_active":"2024-09-21T13:50:29.694465+0000","last_became_peered":"2024-09-21T13:50:29.694465+0000","last_unstale":"2024-09-21T13:54:21.443739+0000","last_undegraded":"2024-09-21T13:54:21.443739+0000","last_fullsized":"2024-09-21T13:54:21.443739+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:52:56.525966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:52:56.525966+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"68'8","reported_seq":355,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.438194+0000","last_change":"2024-09-21T13:51:42.174696+0000","last_active":"2024-09-21T13:54:21.438194+0000","last_peered":"2024-09-21T13:54:21.438194+0000","last_clean":"2024-09-21T13:54:21.438194+0000","last_became_active":"2024-09-21T13:49:02.059638+0000","last_became_peered":"2024-09-21T13:49:02.059638+0000","last_unstale":"2024-09-21T13:54:21.438194+0000","last_undegraded":"2024-09-21T13:54:21.438194+0000","last_fullsized":"2024-09-21T13:54:21.438194+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:51:42.174629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:42.174629+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:14.205512+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"189'931","reported_seq":954,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:23.582928+0000","last_change":"2024-09-21T13:54:12.377270+0000","last_active":"2024-09-21T13:54:23.582928+0000","last_peered":"2024-09-21T13:54:23.582928+0000","last_clean":"2024-09-21T13:54:23.582928+0000","last_became_active":"2024-09-21T13:54:12.376466+0000","last_became_peered":"2024-09-21T13:54:12.376466+0000","last_unstale":"2024-09-21T13:54:23.582928+0000","last_undegraded":"2024-09-21T13:54:23.582928+0000","last_fullsized":"2024-09-21T13:54:23.582928+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":931,"log_dups_size":0,"ondisk_log_size":931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:30.045611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61014016,"num_objects":931,"num_object_clones":0,"num_object_copies":2793,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":931,"num_write_kb":59584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"68'15","reported_seq":315,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.433761+0000","last_change":"2024-09-21T13:53:58.658230+0000","last_active":"2024-09-21T13:54:21.433761+0000","last_peered":"2024-09-21T13:54:21.433761+0000","last_clean":"2024-09-21T13:54:21.433761+0000","last_became_active":"2024-09-21T13:49:00.060324+0000","last_became_peered":"2024-09-21T13:49:00.060324+0000","last_unstale":"2024-09-21T13:54:21.433761+0000","last_undegraded":"2024-09-21T13:54:21.433761+0000","last_fullsized":"2024-09-21T13:54:21.433761+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:53:58.658150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:58.658150+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:14.840395+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":277,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.439887+0000","last_change":"2024-09-21T13:50:06.151943+0000","last_active":"2024-09-21T13:54:21.439887+0000","last_peered":"2024-09-21T13:54:21.439887+0000","last_clean":"2024-09-21T13:54:21.439887+0000","last_became_active":"2024-09-21T13:48:58.051911+0000","last_became_peered":"2024-09-21T13:48:58.051911+0000","last_unstale":"2024-09-21T13:54:21.439887+0000","last_undegraded":"2024-09-21T13:54:21.439887+0000","last_fullsized":"2024-09-21T13:54:21.439887+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.151888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.151888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:12.984236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"68'8","reported_seq":346,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.437546+0000","last_change":"2024-09-21T13:53:19.021881+0000","last_active":"2024-09-21T13:54:21.437546+0000","last_peered":"2024-09-21T13:54:21.437546+0000","last_clean":"2024-09-21T13:54:21.437546+0000","last_became_active":"2024-09-21T13:49:02.061679+0000","last_became_peered":"2024-09-21T13:49:02.061679+0000","last_unstale":"2024-09-21T13:54:21.437546+0000","last_undegraded":"2024-09-21T13:54:21.437546+0000","last_fullsized":"2024-09-21T13:54:21.437546+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:53:19.021810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:53:19.021810+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:35.961578+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":266,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.438010+0000","last_change":"2024-09-21T13:50:34.481503+0000","last_active":"2024-09-21T13:54:21.438010+0000","last_peered":"2024-09-21T13:54:21.438010+0000","last_clean":"2024-09-21T13:54:21.438010+0000","last_became_active":"2024-09-21T13:49:04.081905+0000","last_became_peered":"2024-09-21T13:49:04.081905+0000","last_unstale":"2024-09-21T13:54:21.438010+0000","last_undegraded":"2024-09-21T13:54:21.438010+0000","last_fullsized":"2024-09-21T13:54:21.438010+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:34.481450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:34.481450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:43.214157+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.c","version":"189'987","reported_seq":1010,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:23.658520+0000","last_change":"2024-09-21T13:54:12.479015+0000","last_active":"2024-09-21T13:54:23.658520+0000","last_peered":"2024-09-21T13:54:23.658520+0000","last_clean":"2024-09-21T13:54:23.658520+0000","last_became_active":"2024-09-21T13:54:12.478687+0000","last_became_peered":"2024-09-21T13:54:12.478687+0000","last_unstale":"2024-09-21T13:54:23.658520+0000","last_undegraded":"2024-09-21T13:54:23.658520+0000","last_fullsized":"2024-09-21T13:54:23.658520+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":987,"log_dups_size":0,"ondisk_log_size":987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:33.300965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64684032,"num_objects":987,"num_object_clones":0,"num_object_copies":2961,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":987,"num_write_kb":63168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"68'18","reported_seq":313,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.439777+0000","last_change":"2024-09-21T13:54:19.001120+0000","last_active":"2024-09-21T13:54:21.439777+0000","last_peered":"2024-09-21T13:54:21.439777+0000","last_clean":"2024-09-21T13:54:21.439777+0000","last_became_active":"2024-09-21T13:49:00.429314+0000","last_became_peered":"2024-09-21T13:49:00.429314+0000","last_unstale":"2024-09-21T13:54:21.439777+0000","last_undegraded":"2024-09-21T13:54:21.439777+0000","last_fullsized":"2024-09-21T13:54:21.439777+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'18","last_scrub_stamp":"2024-09-21T13:54:19.001030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:19.001030+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:34.825871+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":269,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.441395+0000","last_change":"2024-09-21T13:48:58.056669+0000","last_active":"2024-09-21T13:54:21.441395+0000","last_peered":"2024-09-21T13:54:21.441395+0000","last_clean":"2024-09-21T13:54:21.441395+0000","last_became_active":"2024-09-21T13:48:58.056375+0000","last_became_peered":"2024-09-21T13:48:58.056375+0000","last_unstale":"2024-09-21T13:54:21.441395+0000","last_undegraded":"2024-09-21T13:54:21.441395+0000","last_fullsized":"2024-09-21T13:54:21.441395+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:34.834115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":277,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.431675+0000","last_change":"2024-09-21T13:54:07.310089+0000","last_active":"2024-09-21T13:54:21.431675+0000","last_peered":"2024-09-21T13:54:21.431675+0000","last_clean":"2024-09-21T13:54:21.431675+0000","last_became_active":"2024-09-21T13:49:02.062955+0000","last_became_peered":"2024-09-21T13:49:02.062955+0000","last_unstale":"2024-09-21T13:54:21.431675+0000","last_undegraded":"2024-09-21T13:54:21.431675+0000","last_fullsized":"2024-09-21T13:54:21.431675+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:07.310044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:07.310044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:23.878204+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":268,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.437261+0000","last_change":"2024-09-21T13:52:04.316586+0000","last_active":"2024-09-21T13:54:21.437261+0000","last_peered":"2024-09-21T13:54:21.437261+0000","last_clean":"2024-09-21T13:54:21.437261+0000","last_became_active":"2024-09-21T13:52:04.316121+0000","last_became_peered":"2024-09-21T13:52:04.316121+0000","last_unstale":"2024-09-21T13:54:21.437261+0000","last_undegraded":"2024-09-21T13:54:21.437261+0000","last_fullsized":"2024-09-21T13:54:21.437261+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:10.346375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"189'900","reported_seq":923,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:23.393422+0000","last_change":"2024-09-21T13:54:12.479429+0000","last_active":"2024-09-21T13:54:23.393422+0000","last_peered":"2024-09-21T13:54:23.393422+0000","last_clean":"2024-09-21T13:54:23.393422+0000","last_became_active":"2024-09-21T13:54:12.479146+0000","last_became_peered":"2024-09-21T13:54:12.479146+0000","last_unstale":"2024-09-21T13:54:23.393422+0000","last_undegraded":"2024-09-21T13:54:23.393422+0000","last_fullsized":"2024-09-21T13:54:23.393422+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":900,"log_dups_size":0,"ondisk_log_size":900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:37.886798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58982400,"num_objects":900,"num_object_clones":0,"num_object_copies":2700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":900,"num_write_kb":57600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"68'14","reported_seq":294,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.431729+0000","last_change":"2024-09-21T13:50:29.009786+0000","last_active":"2024-09-21T13:54:21.431729+0000","last_peered":"2024-09-21T13:54:21.431729+0000","last_clean":"2024-09-21T13:54:21.431729+0000","last_became_active":"2024-09-21T13:49:00.055031+0000","last_became_peered":"2024-09-21T13:49:00.055031+0000","last_unstale":"2024-09-21T13:54:21.431729+0000","last_undegraded":"2024-09-21T13:54:21.431729+0000","last_fullsized":"2024-09-21T13:54:21.431729+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'14","last_scrub_stamp":"2024-09-21T13:50:29.009720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:29.009720+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:41.285061+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":279,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.437929+0000","last_change":"2024-09-21T13:49:59.253636+0000","last_active":"2024-09-21T13:54:21.437929+0000","last_peered":"2024-09-21T13:54:21.437929+0000","last_clean":"2024-09-21T13:54:21.437929+0000","last_became_active":"2024-09-21T13:48:58.053533+0000","last_became_peered":"2024-09-21T13:48:58.053533+0000","last_unstale":"2024-09-21T13:54:21.437929+0000","last_undegraded":"2024-09-21T13:54:21.437929+0000","last_fullsized":"2024-09-21T13:54:21.437929+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:49:59.253587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:49:59.253587+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:36.768506+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"68'8","reported_seq":343,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.439725+0000","last_change":"2024-09-21T13:50:41.458637+0000","last_active":"2024-09-21T13:54:21.439725+0000","last_peered":"2024-09-21T13:54:21.439725+0000","last_clean":"2024-09-21T13:54:21.439725+0000","last_became_active":"2024-09-21T13:49:02.070064+0000","last_became_peered":"2024-09-21T13:49:02.070064+0000","last_unstale":"2024-09-21T13:54:21.439725+0000","last_undegraded":"2024-09-21T13:54:21.439725+0000","last_fullsized":"2024-09-21T13:54:21.439725+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:50:41.458562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:41.458562+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":276,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.443047+0000","last_change":"2024-09-21T13:52:55.541288+0000","last_active":"2024-09-21T13:54:21.443047+0000","last_peered":"2024-09-21T13:54:21.443047+0000","last_clean":"2024-09-21T13:54:21.443047+0000","last_became_active":"2024-09-21T13:52:04.322185+0000","last_became_peered":"2024-09-21T13:52:04.322185+0000","last_unstale":"2024-09-21T13:54:21.443047+0000","last_undegraded":"2024-09-21T13:54:21.443047+0000","last_fullsized":"2024-09-21T13:54:21.443047+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:55.541232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:55.541232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.e","version":"189'976","reported_seq":999,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:23.679454+0000","last_change":"2024-09-21T13:54:12.480264+0000","last_active":"2024-09-21T13:54:23.679454+0000","last_peered":"2024-09-21T13:54:23.679454+0000","last_clean":"2024-09-21T13:54:23.679454+0000","last_became_active":"2024-09-21T13:54:12.479901+0000","last_became_peered":"2024-09-21T13:54:12.479901+0000","last_unstale":"2024-09-21T13:54:23.679454+0000","last_undegraded":"2024-09-21T13:54:23.679454+0000","last_fullsized":"2024-09-21T13:54:23.679454+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":976,"log_dups_size":0,"ondisk_log_size":976,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:35.816508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63963136,"num_objects":976,"num_object_clones":0,"num_object_copies":2928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":976,"num_write_kb":62464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"68'10","reported_seq":305,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.431369+0000","last_change":"2024-09-21T13:54:19.593696+0000","last_active":"2024-09-21T13:54:21.431369+0000","last_peered":"2024-09-21T13:54:21.431369+0000","last_clean":"2024-09-21T13:54:21.431369+0000","last_became_active":"2024-09-21T13:49:00.054564+0000","last_became_peered":"2024-09-21T13:49:00.054564+0000","last_unstale":"2024-09-21T13:54:21.431369+0000","last_undegraded":"2024-09-21T13:54:21.431369+0000","last_fullsized":"2024-09-21T13:54:21.431369+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:54:19.593611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:19.593611+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:40.845382+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"61'1","reported_seq":311,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.438541+0000","last_change":"2024-09-21T13:53:30.179679+0000","last_active":"2024-09-21T13:54:21.438541+0000","last_peered":"2024-09-21T13:54:21.438541+0000","last_clean":"2024-09-21T13:54:21.438541+0000","last_became_active":"2024-09-21T13:52:04.315788+0000","last_became_peered":"2024-09-21T13:52:04.315788+0000","last_unstale":"2024-09-21T13:54:21.438541+0000","last_undegraded":"2024-09-21T13:54:21.438541+0000","last_fullsized":"2024-09-21T13:54:21.438541+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:53:30.179605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:53:30.179605+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:43.052251+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"96'9","reported_seq":299,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.436790+0000","last_change":"2024-09-21T13:54:06.280517+0000","last_active":"2024-09-21T13:54:21.436790+0000","last_peered":"2024-09-21T13:54:21.436790+0000","last_clean":"2024-09-21T13:54:21.436790+0000","last_became_active":"2024-09-21T13:50:29.693557+0000","last_became_peered":"2024-09-21T13:50:29.693557+0000","last_unstale":"2024-09-21T13:54:21.436790+0000","last_undegraded":"2024-09-21T13:54:21.436790+0000","last_fullsized":"2024-09-21T13:54:21.436790+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'9","last_scrub_stamp":"2024-09-21T13:54:06.280457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:06.280457+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:23.993139+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":220,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.439640+0000","last_change":"2024-09-21T13:53:54.948771+0000","last_active":"2024-09-21T13:54:21.439640+0000","last_peered":"2024-09-21T13:54:21.439640+0000","last_clean":"2024-09-21T13:54:21.439640+0000","last_became_active":"2024-09-21T13:52:04.319034+0000","last_became_peered":"2024-09-21T13:52:04.319034+0000","last_unstale":"2024-09-21T13:54:21.439640+0000","last_undegraded":"2024-09-21T13:54:21.439640+0000","last_fullsized":"2024-09-21T13:54:21.439640+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:54.948705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:54.948705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:05.874612+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.f","version":"189'1022","reported_seq":1044,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:24.029746+0000","last_change":"2024-09-21T13:54:12.478557+0000","last_active":"2024-09-21T13:54:24.029746+0000","last_peered":"2024-09-21T13:54:24.029746+0000","last_clean":"2024-09-21T13:54:24.029746+0000","last_became_active":"2024-09-21T13:54:12.477589+0000","last_became_peered":"2024-09-21T13:54:12.477589+0000","last_unstale":"2024-09-21T13:54:24.029746+0000","last_undegraded":"2024-09-21T13:54:24.029746+0000","last_fullsized":"2024-09-21T13:54:24.029746+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":1022,"log_dups_size":0,"ondisk_log_size":1022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:29.462737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66977792,"num_objects":1022,"num_object_clones":0,"num_object_copies":3066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1022,"num_write_kb":65408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"68'19","reported_seq":266,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.433445+0000","last_change":"2024-09-21T13:50:32.514267+0000","last_active":"2024-09-21T13:54:21.433445+0000","last_peered":"2024-09-21T13:54:21.433445+0000","last_clean":"2024-09-21T13:54:21.433445+0000","last_became_active":"2024-09-21T13:50:29.690185+0000","last_became_peered":"2024-09-21T13:50:29.690185+0000","last_unstale":"2024-09-21T13:54:21.433445+0000","last_undegraded":"2024-09-21T13:54:21.433445+0000","last_fullsized":"2024-09-21T13:54:21.433445+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-21T13:50:32.514210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:32.514210+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:35.217090+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":281,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.431795+0000","last_change":"2024-09-21T13:52:04.322289+0000","last_active":"2024-09-21T13:54:21.431795+0000","last_peered":"2024-09-21T13:54:21.431795+0000","last_clean":"2024-09-21T13:54:21.431795+0000","last_became_active":"2024-09-21T13:52:04.321656+0000","last_became_peered":"2024-09-21T13:52:04.321656+0000","last_unstale":"2024-09-21T13:54:21.431795+0000","last_undegraded":"2024-09-21T13:54:21.431795+0000","last_fullsized":"2024-09-21T13:54:21.431795+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:12.655427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":278,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.432688+0000","last_change":"2024-09-21T13:53:32.295930+0000","last_active":"2024-09-21T13:54:21.432688+0000","last_peered":"2024-09-21T13:54:21.432688+0000","last_clean":"2024-09-21T13:54:21.432688+0000","last_became_active":"2024-09-21T13:49:02.061187+0000","last_became_peered":"2024-09-21T13:49:02.061187+0000","last_unstale":"2024-09-21T13:54:21.432688+0000","last_undegraded":"2024-09-21T13:54:21.432688+0000","last_fullsized":"2024-09-21T13:54:21.432688+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:32.295869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:53:32.295869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:32.443805+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":269,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.429248+0000","last_change":"2024-09-21T13:52:04.321133+0000","last_active":"2024-09-21T13:54:21.429248+0000","last_peered":"2024-09-21T13:54:21.429248+0000","last_clean":"2024-09-21T13:54:21.429248+0000","last_became_active":"2024-09-21T13:52:04.320826+0000","last_became_peered":"2024-09-21T13:52:04.320826+0000","last_unstale":"2024-09-21T13:54:21.429248+0000","last_undegraded":"2024-09-21T13:54:21.429248+0000","last_fullsized":"2024-09-21T13:54:21.429248+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:14.698955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"189'866","reported_seq":889,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:23.380625+0000","last_change":"2024-09-21T13:54:12.477847+0000","last_active":"2024-09-21T13:54:23.380625+0000","last_peered":"2024-09-21T13:54:23.380625+0000","last_clean":"2024-09-21T13:54:23.380625+0000","last_became_active":"2024-09-21T13:54:12.477306+0000","last_became_peered":"2024-09-21T13:54:12.477306+0000","last_unstale":"2024-09-21T13:54:23.380625+0000","last_undegraded":"2024-09-21T13:54:23.380625+0000","last_fullsized":"2024-09-21T13:54:23.380625+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":866,"log_dups_size":0,"ondisk_log_size":866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:26.986694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56754176,"num_objects":866,"num_object_clones":0,"num_object_copies":2598,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":866,"num_write_kb":55424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"132'30","reported_seq":401,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.431730+0000","last_change":"2024-09-21T13:54:16.681017+0000","last_active":"2024-09-21T13:54:21.431730+0000","last_peered":"2024-09-21T13:54:21.431730+0000","last_clean":"2024-09-21T13:54:21.431730+0000","last_became_active":"2024-09-21T13:52:04.319431+0000","last_became_peered":"2024-09-21T13:52:04.319431+0000","last_unstale":"2024-09-21T13:54:21.431730+0000","last_undegraded":"2024-09-21T13:54:21.431730+0000","last_fullsized":"2024-09-21T13:54:21.431730+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'30","last_scrub_stamp":"2024-09-21T13:54:16.680953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:16.680953+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:39.761349+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":269,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.437033+0000","last_change":"2024-09-21T13:48:58.055640+0000","last_active":"2024-09-21T13:54:21.437033+0000","last_peered":"2024-09-21T13:54:21.437033+0000","last_clean":"2024-09-21T13:54:21.437033+0000","last_became_active":"2024-09-21T13:48:58.055366+0000","last_became_peered":"2024-09-21T13:48:58.055366+0000","last_unstale":"2024-09-21T13:54:21.437033+0000","last_undegraded":"2024-09-21T13:54:21.437033+0000","last_fullsized":"2024-09-21T13:54:21.437033+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:09.329706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":296,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.439587+0000","last_change":"2024-09-21T13:52:51.445882+0000","last_active":"2024-09-21T13:54:21.439587+0000","last_peered":"2024-09-21T13:54:21.439587+0000","last_clean":"2024-09-21T13:54:21.439587+0000","last_became_active":"2024-09-21T13:48:57.046246+0000","last_became_peered":"2024-09-21T13:48:57.046246+0000","last_unstale":"2024-09-21T13:54:21.439587+0000","last_undegraded":"2024-09-21T13:54:21.439587+0000","last_fullsized":"2024-09-21T13:54:21.439587+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-21T13:52:51.445812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:52:51.445812+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.0015946669999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":218,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.430918+0000","last_change":"2024-09-21T13:53:56.670634+0000","last_active":"2024-09-21T13:54:21.430918+0000","last_peered":"2024-09-21T13:54:21.430918+0000","last_clean":"2024-09-21T13:54:21.430918+0000","last_became_active":"2024-09-21T13:50:29.688249+0000","last_became_peered":"2024-09-21T13:50:29.688249+0000","last_unstale":"2024-09-21T13:54:21.430918+0000","last_undegraded":"2024-09-21T13:54:21.430918+0000","last_fullsized":"2024-09-21T13:54:21.430918+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:56.670555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:53:56.670555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:05.626771+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":265,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.441096+0000","last_change":"2024-09-21T13:51:23.597164+0000","last_active":"2024-09-21T13:54:21.441096+0000","last_peered":"2024-09-21T13:54:21.441096+0000","last_clean":"2024-09-21T13:54:21.441096+0000","last_became_active":"2024-09-21T13:49:04.080466+0000","last_became_peered":"2024-09-21T13:49:04.080466+0000","last_unstale":"2024-09-21T13:54:21.441096+0000","last_undegraded":"2024-09-21T13:54:21.441096+0000","last_fullsized":"2024-09-21T13:54:21.441096+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:23.596879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:51:23.596879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"189'999","reported_seq":1021,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:24.033679+0000","last_change":"2024-09-21T13:54:12.373760+0000","last_active":"2024-09-21T13:54:24.033679+0000","last_peered":"2024-09-21T13:54:24.033679+0000","last_clean":"2024-09-21T13:54:24.033679+0000","last_became_active":"2024-09-21T13:54:12.373395+0000","last_became_peered":"2024-09-21T13:54:12.373395+0000","last_unstale":"2024-09-21T13:54:24.033679+0000","last_undegraded":"2024-09-21T13:54:24.033679+0000","last_fullsized":"2024-09-21T13:54:24.033679+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":999,"log_dups_size":0,"ondisk_log_size":999,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:36.149952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65470464,"num_objects":999,"num_object_clones":0,"num_object_copies":2997,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":999,"num_write_kb":63936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"68'13","reported_seq":301,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.433842+0000","last_change":"2024-09-21T13:50:06.973399+0000","last_active":"2024-09-21T13:54:21.433842+0000","last_peered":"2024-09-21T13:54:21.433842+0000","last_clean":"2024-09-21T13:54:21.433842+0000","last_became_active":"2024-09-21T13:49:00.053323+0000","last_became_peered":"2024-09-21T13:49:00.053323+0000","last_unstale":"2024-09-21T13:54:21.433842+0000","last_undegraded":"2024-09-21T13:54:21.433842+0000","last_fullsized":"2024-09-21T13:54:21.433842+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'13","last_scrub_stamp":"2024-09-21T13:50:06.973336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.973336+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:41.649253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":305,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.430832+0000","last_change":"2024-09-21T13:53:24.885266+0000","last_active":"2024-09-21T13:54:21.430832+0000","last_peered":"2024-09-21T13:54:21.430832+0000","last_clean":"2024-09-21T13:54:21.430832+0000","last_became_active":"2024-09-21T13:52:04.320581+0000","last_became_peered":"2024-09-21T13:52:04.320581+0000","last_unstale":"2024-09-21T13:54:21.430832+0000","last_undegraded":"2024-09-21T13:54:21.430832+0000","last_fullsized":"2024-09-21T13:54:21.430832+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:24.884932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:53:24.884932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:31.375453+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":209,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T13:54:26.475474+0000","last_change":"2024-09-21T13:53:19.985044+0000","last_active":"2024-09-21T13:54:26.475474+0000","last_peered":"2024-09-21T13:54:26.475474+0000","last_clean":"2024-09-21T13:54:26.475474+0000","last_became_active":"2024-09-21T13:52:04.316171+0000","last_became_peered":"2024-09-21T13:52:04.316171+0000","last_unstale":"2024-09-21T13:54:26.475474+0000","last_undegraded":"2024-09-21T13:54:26.475474+0000","last_fullsized":"2024-09-21T13:54:26.475474+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:53:19.984966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:53:19.984966+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:47.985977+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":161,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T13:54:26.476448+0000","last_change":"2024-09-21T13:54:26.009416+0000","last_active":"2024-09-21T13:54:26.476448+0000","last_peered":"2024-09-21T13:54:26.476448+0000","last_clean":"2024-09-21T13:54:26.476448+0000","last_became_active":"2024-09-21T13:52:04.320701+0000","last_became_peered":"2024-09-21T13:52:04.320701+0000","last_unstale":"2024-09-21T13:54:26.476448+0000","last_undegraded":"2024-09-21T13:54:26.476448+0000","last_fullsized":"2024-09-21T13:54:26.476448+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:26.009352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:26.009352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:55.894955+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":273,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.442700+0000","last_change":"2024-09-21T13:51:12.212558+0000","last_active":"2024-09-21T13:54:21.442700+0000","last_peered":"2024-09-21T13:54:21.442700+0000","last_clean":"2024-09-21T13:54:21.442700+0000","last_became_active":"2024-09-21T13:49:04.431035+0000","last_became_peered":"2024-09-21T13:49:04.431035+0000","last_unstale":"2024-09-21T13:54:21.442700+0000","last_undegraded":"2024-09-21T13:54:21.442700+0000","last_fullsized":"2024-09-21T13:54:21.442700+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:12.212391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:51:12.212391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"189'962","reported_seq":985,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:23.582187+0000","last_change":"2024-09-21T13:54:12.477912+0000","last_active":"2024-09-21T13:54:23.582187+0000","last_peered":"2024-09-21T13:54:23.582187+0000","last_clean":"2024-09-21T13:54:23.582187+0000","last_became_active":"2024-09-21T13:54:12.477596+0000","last_became_peered":"2024-09-21T13:54:12.477596+0000","last_unstale":"2024-09-21T13:54:23.582187+0000","last_undegraded":"2024-09-21T13:54:23.582187+0000","last_fullsized":"2024-09-21T13:54:23.582187+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":962,"log_dups_size":0,"ondisk_log_size":962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:33.789453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63045632,"num_objects":962,"num_object_clones":0,"num_object_copies":2886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":962,"num_write_kb":61568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"68'12","reported_seq":332,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.437775+0000","last_change":"2024-09-21T13:53:17.740156+0000","last_active":"2024-09-21T13:54:21.437775+0000","last_peered":"2024-09-21T13:54:21.437775+0000","last_clean":"2024-09-21T13:54:21.437775+0000","last_became_active":"2024-09-21T13:52:04.318875+0000","last_became_peered":"2024-09-21T13:52:04.318875+0000","last_unstale":"2024-09-21T13:54:21.437775+0000","last_undegraded":"2024-09-21T13:54:21.437775+0000","last_fullsized":"2024-09-21T13:54:21.437775+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'12","last_scrub_stamp":"2024-09-21T13:53:17.740080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:17.740080+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:30.638014+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":209,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.431782+0000","last_change":"2024-09-21T13:50:30.020722+0000","last_active":"2024-09-21T13:54:21.431782+0000","last_peered":"2024-09-21T13:54:21.431782+0000","last_clean":"2024-09-21T13:54:21.431782+0000","last_became_active":"2024-09-21T13:50:29.688352+0000","last_became_peered":"2024-09-21T13:50:29.688352+0000","last_unstale":"2024-09-21T13:54:21.431782+0000","last_undegraded":"2024-09-21T13:54:21.431782+0000","last_fullsized":"2024-09-21T13:54:21.431782+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:30.020656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:30.020656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:38.234245+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":608,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:22.331225+0000","last_change":"2024-09-21T13:51:21.271997+0000","last_active":"2024-09-21T13:54:22.331225+0000","last_peered":"2024-09-21T13:54:22.331225+0000","last_clean":"2024-09-21T13:54:22.331225+0000","last_became_active":"2024-09-21T13:48:57.046084+0000","last_became_peered":"2024-09-21T13:48:57.046084+0000","last_unstale":"2024-09-21T13:54:22.331225+0000","last_undegraded":"2024-09-21T13:54:22.331225+0000","last_fullsized":"2024-09-21T13:54:22.331225+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-21T13:51:21.271917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:51:21.271917+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00047048000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":320,"num_read_kb":315,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":270,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.440181+0000","last_change":"2024-09-21T13:51:53.518645+0000","last_active":"2024-09-21T13:54:21.440181+0000","last_peered":"2024-09-21T13:54:21.440181+0000","last_clean":"2024-09-21T13:54:21.440181+0000","last_became_active":"2024-09-21T13:49:02.066641+0000","last_became_peered":"2024-09-21T13:49:02.066641+0000","last_unstale":"2024-09-21T13:54:21.440181+0000","last_undegraded":"2024-09-21T13:54:21.440181+0000","last_fullsized":"2024-09-21T13:54:21.440181+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:53.518567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:53.518567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:16.069137+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":274,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.428927+0000","last_change":"2024-09-21T13:52:21.332066+0000","last_active":"2024-09-21T13:54:21.428927+0000","last_peered":"2024-09-21T13:54:21.428927+0000","last_clean":"2024-09-21T13:54:21.428927+0000","last_became_active":"2024-09-21T13:49:04.078372+0000","last_became_peered":"2024-09-21T13:49:04.078372+0000","last_unstale":"2024-09-21T13:54:21.428927+0000","last_undegraded":"2024-09-21T13:54:21.428927+0000","last_fullsized":"2024-09-21T13:54:21.428927+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:21.331792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:21.331792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"189'983","reported_seq":1006,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:23.418852+0000","last_change":"2024-09-21T13:54:12.378930+0000","last_active":"2024-09-21T13:54:23.418852+0000","last_peered":"2024-09-21T13:54:23.418852+0000","last_clean":"2024-09-21T13:54:23.418852+0000","last_became_active":"2024-09-21T13:54:12.378536+0000","last_became_peered":"2024-09-21T13:54:12.378536+0000","last_unstale":"2024-09-21T13:54:23.418852+0000","last_undegraded":"2024-09-21T13:54:23.418852+0000","last_fullsized":"2024-09-21T13:54:23.418852+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":983,"log_dups_size":0,"ondisk_log_size":983,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:32.246204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"68'16","reported_seq":370,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.439165+0000","last_change":"2024-09-21T13:52:19.452239+0000","last_active":"2024-09-21T13:54:21.439165+0000","last_peered":"2024-09-21T13:54:21.439165+0000","last_clean":"2024-09-21T13:54:21.439165+0000","last_became_active":"2024-09-21T13:49:00.426867+0000","last_became_peered":"2024-09-21T13:49:00.426867+0000","last_unstale":"2024-09-21T13:54:21.439165+0000","last_undegraded":"2024-09-21T13:54:21.439165+0000","last_fullsized":"2024-09-21T13:54:21.439165+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'16","last_scrub_stamp":"2024-09-21T13:52:19.452163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:52:19.452163+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:49.265943+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":285,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.440534+0000","last_change":"2024-09-21T13:54:07.057258+0000","last_active":"2024-09-21T13:54:21.440534+0000","last_peered":"2024-09-21T13:54:21.440534+0000","last_clean":"2024-09-21T13:54:21.440534+0000","last_became_active":"2024-09-21T13:48:58.052659+0000","last_became_peered":"2024-09-21T13:48:58.052659+0000","last_unstale":"2024-09-21T13:54:21.440534+0000","last_undegraded":"2024-09-21T13:54:21.440534+0000","last_fullsized":"2024-09-21T13:54:21.440534+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:07.057210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:07.057210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:14.084334+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"75'79","reported_seq":326,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.442297+0000","last_change":"2024-09-21T13:50:27.096470+0000","last_active":"2024-09-21T13:54:21.442297+0000","last_peered":"2024-09-21T13:54:21.442297+0000","last_clean":"2024-09-21T13:54:21.442297+0000","last_became_active":"2024-09-21T13:48:48.454646+0000","last_became_peered":"2024-09-21T13:48:48.454646+0000","last_unstale":"2024-09-21T13:54:21.442297+0000","last_undegraded":"2024-09-21T13:54:21.442297+0000","last_fullsized":"2024-09-21T13:54:21.442297+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:50:27.096413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:50:27.096413+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:37.381057+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":272,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.442507+0000","last_change":"2024-09-21T13:52:04.320948+0000","last_active":"2024-09-21T13:54:21.442507+0000","last_peered":"2024-09-21T13:54:21.442507+0000","last_clean":"2024-09-21T13:54:21.442507+0000","last_became_active":"2024-09-21T13:52:04.320627+0000","last_became_peered":"2024-09-21T13:52:04.320627+0000","last_unstale":"2024-09-21T13:54:21.442507+0000","last_undegraded":"2024-09-21T13:54:21.442507+0000","last_fullsized":"2024-09-21T13:54:21.442507+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:50.544725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":266,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.439901+0000","last_change":"2024-09-21T13:50:25.560928+0000","last_active":"2024-09-21T13:54:21.439901+0000","last_peered":"2024-09-21T13:54:21.439901+0000","last_clean":"2024-09-21T13:54:21.439901+0000","last_became_active":"2024-09-21T13:49:04.078091+0000","last_became_peered":"2024-09-21T13:49:04.078091+0000","last_unstale":"2024-09-21T13:54:21.439901+0000","last_undegraded":"2024-09-21T13:54:21.439901+0000","last_fullsized":"2024-09-21T13:54:21.439901+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:25.560863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:25.560863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:41.214571+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"189'915","reported_seq":937,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:23.413673+0000","last_change":"2024-09-21T13:54:12.378230+0000","last_active":"2024-09-21T13:54:23.413673+0000","last_peered":"2024-09-21T13:54:23.413673+0000","last_clean":"2024-09-21T13:54:23.413673+0000","last_became_active":"2024-09-21T13:54:12.377874+0000","last_became_peered":"2024-09-21T13:54:12.377874+0000","last_unstale":"2024-09-21T13:54:23.413673+0000","last_undegraded":"2024-09-21T13:54:23.413673+0000","last_fullsized":"2024-09-21T13:54:23.413673+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":915,"log_dups_size":0,"ondisk_log_size":915,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:39.278391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59965440,"num_objects":915,"num_object_clones":0,"num_object_copies":2745,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":915,"num_write_kb":58560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"68'11","reported_seq":292,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.438947+0000","last_change":"2024-09-21T13:54:16.451648+0000","last_active":"2024-09-21T13:54:21.438947+0000","last_peered":"2024-09-21T13:54:21.438947+0000","last_clean":"2024-09-21T13:54:21.438947+0000","last_became_active":"2024-09-21T13:49:00.053268+0000","last_became_peered":"2024-09-21T13:49:00.053268+0000","last_unstale":"2024-09-21T13:54:21.438947+0000","last_undegraded":"2024-09-21T13:54:21.438947+0000","last_fullsized":"2024-09-21T13:54:21.438947+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:54:16.451588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:16.451588+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:41.356489+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":288,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.431947+0000","last_change":"2024-09-21T13:52:48.165210+0000","last_active":"2024-09-21T13:54:21.431947+0000","last_peered":"2024-09-21T13:54:21.431947+0000","last_clean":"2024-09-21T13:54:21.431947+0000","last_became_active":"2024-09-21T13:52:04.318345+0000","last_became_peered":"2024-09-21T13:52:04.318345+0000","last_unstale":"2024-09-21T13:54:21.431947+0000","last_undegraded":"2024-09-21T13:54:21.431947+0000","last_fullsized":"2024-09-21T13:54:21.431947+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:48.165145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:52:48.165145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":278,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.439867+0000","last_change":"2024-09-21T13:51:46.503640+0000","last_active":"2024-09-21T13:54:21.439867+0000","last_peered":"2024-09-21T13:54:21.439867+0000","last_clean":"2024-09-21T13:54:21.439867+0000","last_became_active":"2024-09-21T13:49:02.061901+0000","last_became_peered":"2024-09-21T13:49:02.061901+0000","last_unstale":"2024-09-21T13:54:21.439867+0000","last_undegraded":"2024-09-21T13:54:21.439867+0000","last_fullsized":"2024-09-21T13:54:21.439867+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:46.503576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:46.503576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:15.052988+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":265,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.440275+0000","last_change":"2024-09-21T13:51:51.814259+0000","last_active":"2024-09-21T13:54:21.440275+0000","last_peered":"2024-09-21T13:54:21.440275+0000","last_clean":"2024-09-21T13:54:21.440275+0000","last_became_active":"2024-09-21T13:49:04.431268+0000","last_became_peered":"2024-09-21T13:49:04.431268+0000","last_unstale":"2024-09-21T13:54:21.440275+0000","last_undegraded":"2024-09-21T13:54:21.440275+0000","last_fullsized":"2024-09-21T13:54:21.440275+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:51.814200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:51:51.814200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1","version":"189'916","reported_seq":939,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:23.581896+0000","last_change":"2024-09-21T13:54:12.478397+0000","last_active":"2024-09-21T13:54:23.581896+0000","last_peered":"2024-09-21T13:54:23.581896+0000","last_clean":"2024-09-21T13:54:23.581896+0000","last_became_active":"2024-09-21T13:54:12.477588+0000","last_became_peered":"2024-09-21T13:54:12.477588+0000","last_unstale":"2024-09-21T13:54:23.581896+0000","last_undegraded":"2024-09-21T13:54:23.581896+0000","last_fullsized":"2024-09-21T13:54:23.581896+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":916,"log_dups_size":0,"ondisk_log_size":916,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:37.981662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60030976,"num_objects":916,"num_object_clones":0,"num_object_copies":2748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":916,"num_write_kb":58624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"68'17","reported_seq":312,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.437096+0000","last_change":"2024-09-21T13:52:45.176950+0000","last_active":"2024-09-21T13:54:21.437096+0000","last_peered":"2024-09-21T13:54:21.437096+0000","last_clean":"2024-09-21T13:54:21.437096+0000","last_became_active":"2024-09-21T13:52:04.313218+0000","last_became_peered":"2024-09-21T13:52:04.313218+0000","last_unstale":"2024-09-21T13:54:21.437096+0000","last_undegraded":"2024-09-21T13:54:21.437096+0000","last_fullsized":"2024-09-21T13:54:21.437096+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'17","last_scrub_stamp":"2024-09-21T13:52:45.176873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:52:45.176873+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":294,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.432915+0000","last_change":"2024-09-21T13:53:55.202100+0000","last_active":"2024-09-21T13:54:21.432915+0000","last_peered":"2024-09-21T13:54:21.432915+0000","last_clean":"2024-09-21T13:54:21.432915+0000","last_became_active":"2024-09-21T13:48:58.049142+0000","last_became_peered":"2024-09-21T13:48:58.049142+0000","last_unstale":"2024-09-21T13:54:21.432915+0000","last_undegraded":"2024-09-21T13:54:21.432915+0000","last_fullsized":"2024-09-21T13:54:21.432915+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:55.202057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:53:55.202057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:00.441490+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":270,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.431463+0000","last_change":"2024-09-21T13:54:06.629867+0000","last_active":"2024-09-21T13:54:21.431463+0000","last_peered":"2024-09-21T13:54:21.431463+0000","last_clean":"2024-09-21T13:54:21.431463+0000","last_became_active":"2024-09-21T13:49:02.067472+0000","last_became_peered":"2024-09-21T13:49:02.067472+0000","last_unstale":"2024-09-21T13:54:21.431463+0000","last_undegraded":"2024-09-21T13:54:21.431463+0000","last_fullsized":"2024-09-21T13:54:21.431463+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:06.629822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:06.629822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:14.040598+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":153,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T13:54:26.476803+0000","last_change":"2024-09-21T13:53:18.028013+0000","last_active":"2024-09-21T13:54:26.476803+0000","last_peered":"2024-09-21T13:54:26.476803+0000","last_clean":"2024-09-21T13:54:26.476803+0000","last_became_active":"2024-09-21T13:52:04.315920+0000","last_became_peered":"2024-09-21T13:52:04.315920+0000","last_unstale":"2024-09-21T13:54:26.476803+0000","last_undegraded":"2024-09-21T13:54:26.476803+0000","last_fullsized":"2024-09-21T13:54:26.476803+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:18.027944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:18.027944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.0","version":"189'1006","reported_seq":1028,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:24.024940+0000","last_change":"2024-09-21T13:54:12.375805+0000","last_active":"2024-09-21T13:54:24.024940+0000","last_peered":"2024-09-21T13:54:24.024940+0000","last_clean":"2024-09-21T13:54:24.024940+0000","last_became_active":"2024-09-21T13:54:12.375197+0000","last_became_peered":"2024-09-21T13:54:12.375197+0000","last_unstale":"2024-09-21T13:54:24.024940+0000","last_undegraded":"2024-09-21T13:54:24.024940+0000","last_fullsized":"2024-09-21T13:54:24.024940+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":1006,"log_dups_size":0,"ondisk_log_size":1006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:17.781582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65929216,"num_objects":1006,"num_object_clones":0,"num_object_copies":3018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1006,"num_write_kb":64384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"68'10","reported_seq":296,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.437283+0000","last_change":"2024-09-21T13:54:20.426028+0000","last_active":"2024-09-21T13:54:21.437283+0000","last_peered":"2024-09-21T13:54:21.437283+0000","last_clean":"2024-09-21T13:54:21.437283+0000","last_became_active":"2024-09-21T13:49:00.059850+0000","last_became_peered":"2024-09-21T13:49:00.059850+0000","last_unstale":"2024-09-21T13:54:21.437283+0000","last_undegraded":"2024-09-21T13:54:21.437283+0000","last_fullsized":"2024-09-21T13:54:21.437283+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:54:20.425889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:20.425889+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:32.475251+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":277,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.440917+0000","last_change":"2024-09-21T13:51:28.565842+0000","last_active":"2024-09-21T13:54:21.440917+0000","last_peered":"2024-09-21T13:54:21.440917+0000","last_clean":"2024-09-21T13:54:21.440917+0000","last_became_active":"2024-09-21T13:48:58.058309+0000","last_became_peered":"2024-09-21T13:48:58.058309+0000","last_unstale":"2024-09-21T13:54:21.440917+0000","last_undegraded":"2024-09-21T13:54:21.440917+0000","last_fullsized":"2024-09-21T13:54:21.440917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:28.565766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:51:28.565766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"132'10","reported_seq":200,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T13:54:26.476079+0000","last_change":"2024-09-21T13:54:19.934832+0000","last_active":"2024-09-21T13:54:26.476079+0000","last_peered":"2024-09-21T13:54:26.476079+0000","last_clean":"2024-09-21T13:54:26.476079+0000","last_became_active":"2024-09-21T13:52:04.320441+0000","last_became_peered":"2024-09-21T13:52:04.320441+0000","last_unstale":"2024-09-21T13:54:26.476079+0000","last_undegraded":"2024-09-21T13:54:26.476079+0000","last_fullsized":"2024-09-21T13:54:26.476079+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'10","last_scrub_stamp":"2024-09-21T13:54:19.934756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:19.934756+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:29.532492+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"68'1","reported_seq":280,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.437205+0000","last_change":"2024-09-21T13:52:04.314151+0000","last_active":"2024-09-21T13:54:21.437205+0000","last_peered":"2024-09-21T13:54:21.437205+0000","last_clean":"2024-09-21T13:54:21.437205+0000","last_became_active":"2024-09-21T13:52:04.313210+0000","last_became_peered":"2024-09-21T13:52:04.313210+0000","last_unstale":"2024-09-21T13:54:21.437205+0000","last_undegraded":"2024-09-21T13:54:21.437205+0000","last_fullsized":"2024-09-21T13:54:21.437205+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T13:50:41.000201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:41.000201+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:43.256736+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.7","version":"189'898","reported_seq":921,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:23.579177+0000","last_change":"2024-09-21T13:54:12.376676+0000","last_active":"2024-09-21T13:54:23.579177+0000","last_peered":"2024-09-21T13:54:23.579177+0000","last_clean":"2024-09-21T13:54:23.579177+0000","last_became_active":"2024-09-21T13:54:12.376302+0000","last_became_peered":"2024-09-21T13:54:12.376302+0000","last_unstale":"2024-09-21T13:54:23.579177+0000","last_undegraded":"2024-09-21T13:54:23.579177+0000","last_fullsized":"2024-09-21T13:54:23.579177+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":898,"log_dups_size":0,"ondisk_log_size":898,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:33.120123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58851328,"num_objects":898,"num_object_clones":0,"num_object_copies":2694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":898,"num_write_kb":57472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"68'9","reported_seq":298,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.436374+0000","last_change":"2024-09-21T13:53:18.740202+0000","last_active":"2024-09-21T13:54:21.436374+0000","last_peered":"2024-09-21T13:54:21.436374+0000","last_clean":"2024-09-21T13:54:21.436374+0000","last_became_active":"2024-09-21T13:49:00.053016+0000","last_became_peered":"2024-09-21T13:49:00.053016+0000","last_unstale":"2024-09-21T13:54:21.436374+0000","last_undegraded":"2024-09-21T13:54:21.436374+0000","last_fullsized":"2024-09-21T13:54:21.436374+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:53:18.740111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:18.740111+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:34.752035+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":278,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.438541+0000","last_change":"2024-09-21T13:50:38.445732+0000","last_active":"2024-09-21T13:54:21.438541+0000","last_peered":"2024-09-21T13:54:21.438541+0000","last_clean":"2024-09-21T13:54:21.438541+0000","last_became_active":"2024-09-21T13:48:58.049543+0000","last_became_peered":"2024-09-21T13:48:58.049543+0000","last_unstale":"2024-09-21T13:54:21.438541+0000","last_undegraded":"2024-09-21T13:54:21.438541+0000","last_fullsized":"2024-09-21T13:54:21.438541+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:38.445667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:38.445667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:36.189674+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":207,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.431863+0000","last_change":"2024-09-21T13:54:21.403013+0000","last_active":"2024-09-21T13:54:21.431863+0000","last_peered":"2024-09-21T13:54:21.431863+0000","last_clean":"2024-09-21T13:54:21.431863+0000","last_became_active":"2024-09-21T13:52:04.317429+0000","last_became_peered":"2024-09-21T13:52:04.317429+0000","last_unstale":"2024-09-21T13:54:21.431863+0000","last_undegraded":"2024-09-21T13:54:21.431863+0000","last_fullsized":"2024-09-21T13:54:21.431863+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:21.402946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:21.402946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:26.710441+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":220,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.442243+0000","last_change":"2024-09-21T13:52:51.590860+0000","last_active":"2024-09-21T13:54:21.442243+0000","last_peered":"2024-09-21T13:54:21.442243+0000","last_clean":"2024-09-21T13:54:21.442243+0000","last_became_active":"2024-09-21T13:52:04.321703+0000","last_became_peered":"2024-09-21T13:52:04.321703+0000","last_unstale":"2024-09-21T13:54:21.442243+0000","last_undegraded":"2024-09-21T13:54:21.442243+0000","last_fullsized":"2024-09-21T13:54:21.442243+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:51.590800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:51.590800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.6","version":"189'905","reported_seq":928,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:23.426610+0000","last_change":"2024-09-21T13:54:12.478610+0000","last_active":"2024-09-21T13:54:23.426610+0000","last_peered":"2024-09-21T13:54:23.426610+0000","last_clean":"2024-09-21T13:54:23.426610+0000","last_became_active":"2024-09-21T13:54:12.478187+0000","last_became_peered":"2024-09-21T13:54:12.478187+0000","last_unstale":"2024-09-21T13:54:23.426610+0000","last_undegraded":"2024-09-21T13:54:23.426610+0000","last_fullsized":"2024-09-21T13:54:23.426610+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":905,"log_dups_size":0,"ondisk_log_size":905,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:21.444672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59310080,"num_objects":905,"num_object_clones":0,"num_object_copies":2715,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":905,"num_write_kb":57920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"68'19","reported_seq":305,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.432774+0000","last_change":"2024-09-21T13:50:03.769686+0000","last_active":"2024-09-21T13:54:21.432774+0000","last_peered":"2024-09-21T13:54:21.432774+0000","last_clean":"2024-09-21T13:54:21.432774+0000","last_became_active":"2024-09-21T13:49:00.425955+0000","last_became_peered":"2024-09-21T13:49:00.425955+0000","last_unstale":"2024-09-21T13:54:21.432774+0000","last_undegraded":"2024-09-21T13:54:21.432774+0000","last_fullsized":"2024-09-21T13:54:21.432774+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-21T13:50:03.769613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:03.769613+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:36.254505+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":277,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:22.816889+0000","last_change":"2024-09-21T13:54:22.816889+0000","last_active":"2024-09-21T13:54:22.816889+0000","last_peered":"2024-09-21T13:54:22.816889+0000","last_clean":"2024-09-21T13:54:22.816889+0000","last_became_active":"2024-09-21T13:48:58.053641+0000","last_became_peered":"2024-09-21T13:48:58.053641+0000","last_unstale":"2024-09-21T13:54:22.816889+0000","last_undegraded":"2024-09-21T13:54:22.816889+0000","last_fullsized":"2024-09-21T13:54:22.816889+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:22.816815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:22.816815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:52.549192+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":231,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T13:54:26.475644+0000","last_change":"2024-09-21T13:53:53.244317+0000","last_active":"2024-09-21T13:54:26.475644+0000","last_peered":"2024-09-21T13:54:26.475644+0000","last_clean":"2024-09-21T13:54:26.475644+0000","last_became_active":"2024-09-21T13:52:04.321146+0000","last_became_peered":"2024-09-21T13:52:04.321146+0000","last_unstale":"2024-09-21T13:54:26.475644+0000","last_undegraded":"2024-09-21T13:54:26.475644+0000","last_fullsized":"2024-09-21T13:54:26.475644+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:53.244131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:53:53.244131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:13.809484+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":284,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.443787+0000","last_change":"2024-09-21T13:54:03.688017+0000","last_active":"2024-09-21T13:54:21.443787+0000","last_peered":"2024-09-21T13:54:21.443787+0000","last_clean":"2024-09-21T13:54:21.443787+0000","last_became_active":"2024-09-21T13:52:04.322213+0000","last_became_peered":"2024-09-21T13:52:04.322213+0000","last_unstale":"2024-09-21T13:54:21.443787+0000","last_undegraded":"2024-09-21T13:54:21.443787+0000","last_fullsized":"2024-09-21T13:54:21.443787+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:03.687971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:54:03.687971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:28.966237+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"189'962","reported_seq":985,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:23.426888+0000","last_change":"2024-09-21T13:54:12.478639+0000","last_active":"2024-09-21T13:54:23.426888+0000","last_peered":"2024-09-21T13:54:23.426888+0000","last_clean":"2024-09-21T13:54:23.426888+0000","last_became_active":"2024-09-21T13:54:12.478330+0000","last_became_peered":"2024-09-21T13:54:12.478330+0000","last_unstale":"2024-09-21T13:54:23.426888+0000","last_undegraded":"2024-09-21T13:54:23.426888+0000","last_fullsized":"2024-09-21T13:54:23.426888+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":962,"log_dups_size":0,"ondisk_log_size":962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:36.845419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63045632,"num_objects":962,"num_object_clones":0,"num_object_copies":2886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":962,"num_write_kb":61568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"68'12","reported_seq":304,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.431870+0000","last_change":"2024-09-21T13:53:57.288526+0000","last_active":"2024-09-21T13:54:21.431870+0000","last_peered":"2024-09-21T13:54:21.431870+0000","last_clean":"2024-09-21T13:54:21.431870+0000","last_became_active":"2024-09-21T13:49:00.059720+0000","last_became_peered":"2024-09-21T13:49:00.059720+0000","last_unstale":"2024-09-21T13:54:21.431870+0000","last_undegraded":"2024-09-21T13:54:21.431870+0000","last_fullsized":"2024-09-21T13:54:21.431870+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'12","last_scrub_stamp":"2024-09-21T13:53:57.288451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:57.288451+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:08.532403+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":277,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.443826+0000","last_change":"2024-09-21T13:50:29.100595+0000","last_active":"2024-09-21T13:54:21.443826+0000","last_peered":"2024-09-21T13:54:21.443826+0000","last_clean":"2024-09-21T13:54:21.443826+0000","last_became_active":"2024-09-21T13:48:58.057641+0000","last_became_peered":"2024-09-21T13:48:58.057641+0000","last_unstale":"2024-09-21T13:54:21.443826+0000","last_undegraded":"2024-09-21T13:54:21.443826+0000","last_fullsized":"2024-09-21T13:54:21.443826+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:29.100547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:29.100547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:34.353972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":196,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T13:54:26.475252+0000","last_change":"2024-09-21T13:53:31.964122+0000","last_active":"2024-09-21T13:54:26.475252+0000","last_peered":"2024-09-21T13:54:26.475252+0000","last_clean":"2024-09-21T13:54:26.475252+0000","last_became_active":"2024-09-21T13:52:04.322147+0000","last_became_peered":"2024-09-21T13:52:04.322147+0000","last_unstale":"2024-09-21T13:54:26.475252+0000","last_undegraded":"2024-09-21T13:54:26.475252+0000","last_fullsized":"2024-09-21T13:54:26.475252+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:31.958946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:53:31.958946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:51.369874+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":265,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.440591+0000","last_change":"2024-09-21T13:53:20.888048+0000","last_active":"2024-09-21T13:54:21.440591+0000","last_peered":"2024-09-21T13:54:21.440591+0000","last_clean":"2024-09-21T13:54:21.440591+0000","last_became_active":"2024-09-21T13:49:04.078089+0000","last_became_peered":"2024-09-21T13:49:04.078089+0000","last_unstale":"2024-09-21T13:54:21.440591+0000","last_undegraded":"2024-09-21T13:54:21.440591+0000","last_fullsized":"2024-09-21T13:54:21.440591+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:20.882449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:20.882449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:45.947326+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"189'913","reported_seq":936,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:23.365669+0000","last_change":"2024-09-21T13:54:12.477912+0000","last_active":"2024-09-21T13:54:23.365669+0000","last_peered":"2024-09-21T13:54:23.365669+0000","last_clean":"2024-09-21T13:54:23.365669+0000","last_became_active":"2024-09-21T13:54:12.477455+0000","last_became_peered":"2024-09-21T13:54:12.477455+0000","last_unstale":"2024-09-21T13:54:23.365669+0000","last_undegraded":"2024-09-21T13:54:23.365669+0000","last_fullsized":"2024-09-21T13:54:23.365669+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":913,"log_dups_size":0,"ondisk_log_size":913,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:26.894106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59834368,"num_objects":913,"num_object_clones":0,"num_object_copies":2739,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":913,"num_write_kb":58432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"68'15","reported_seq":307,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.433237+0000","last_change":"2024-09-21T13:53:31.204731+0000","last_active":"2024-09-21T13:54:21.433237+0000","last_peered":"2024-09-21T13:54:21.433237+0000","last_clean":"2024-09-21T13:54:21.433237+0000","last_became_active":"2024-09-21T13:49:00.425997+0000","last_became_peered":"2024-09-21T13:49:00.425997+0000","last_unstale":"2024-09-21T13:54:21.433237+0000","last_undegraded":"2024-09-21T13:54:21.433237+0000","last_fullsized":"2024-09-21T13:54:21.433237+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:53:31.204637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:31.204637+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:41.385554+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":297,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.443679+0000","last_change":"2024-09-21T13:53:19.396344+0000","last_active":"2024-09-21T13:54:21.443679+0000","last_peered":"2024-09-21T13:54:21.443679+0000","last_clean":"2024-09-21T13:54:21.443679+0000","last_became_active":"2024-09-21T13:48:58.057632+0000","last_became_peered":"2024-09-21T13:48:58.057632+0000","last_unstale":"2024-09-21T13:54:21.443679+0000","last_undegraded":"2024-09-21T13:54:21.443679+0000","last_fullsized":"2024-09-21T13:54:21.443679+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T13:53:19.396277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:53:19.396277+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:47.867330+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"68'8","reported_seq":343,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.442427+0000","last_change":"2024-09-21T13:51:26.297643+0000","last_active":"2024-09-21T13:54:21.442427+0000","last_peered":"2024-09-21T13:54:21.442427+0000","last_clean":"2024-09-21T13:54:21.442427+0000","last_became_active":"2024-09-21T13:49:02.056167+0000","last_became_peered":"2024-09-21T13:49:02.056167+0000","last_unstale":"2024-09-21T13:54:21.442427+0000","last_undegraded":"2024-09-21T13:54:21.442427+0000","last_fullsized":"2024-09-21T13:54:21.442427+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:51:26.297550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:26.297550+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":282,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.433266+0000","last_change":"2024-09-21T13:54:21.108175+0000","last_active":"2024-09-21T13:54:21.433266+0000","last_peered":"2024-09-21T13:54:21.433266+0000","last_clean":"2024-09-21T13:54:21.433266+0000","last_became_active":"2024-09-21T13:49:04.432546+0000","last_became_peered":"2024-09-21T13:49:04.432546+0000","last_unstale":"2024-09-21T13:54:21.433266+0000","last_undegraded":"2024-09-21T13:54:21.433266+0000","last_fullsized":"2024-09-21T13:54:21.433266+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:21.108123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:54:21.108123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:24.708631+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":294,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.432659+0000","last_change":"2024-09-21T13:54:18.131012+0000","last_active":"2024-09-21T13:54:21.432659+0000","last_peered":"2024-09-21T13:54:21.432659+0000","last_clean":"2024-09-21T13:54:21.432659+0000","last_became_active":"2024-09-21T13:48:58.056080+0000","last_became_peered":"2024-09-21T13:48:58.056080+0000","last_unstale":"2024-09-21T13:54:21.432659+0000","last_undegraded":"2024-09-21T13:54:21.432659+0000","last_fullsized":"2024-09-21T13:54:21.432659+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:18.130969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:18.130969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:34.242253+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"68'6","reported_seq":282,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.441257+0000","last_change":"2024-09-21T13:53:31.807215+0000","last_active":"2024-09-21T13:54:21.441257+0000","last_peered":"2024-09-21T13:54:21.441257+0000","last_clean":"2024-09-21T13:54:21.441257+0000","last_became_active":"2024-09-21T13:49:00.426829+0000","last_became_peered":"2024-09-21T13:49:00.426829+0000","last_unstale":"2024-09-21T13:54:21.441257+0000","last_undegraded":"2024-09-21T13:54:21.441257+0000","last_fullsized":"2024-09-21T13:54:21.441257+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'6","last_scrub_stamp":"2024-09-21T13:53:31.807028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:31.807028+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:49.612755+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":286,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.439689+0000","last_change":"2024-09-21T13:54:18.124591+0000","last_active":"2024-09-21T13:54:21.439689+0000","last_peered":"2024-09-21T13:54:21.439689+0000","last_clean":"2024-09-21T13:54:21.439689+0000","last_became_active":"2024-09-21T13:49:02.062231+0000","last_became_peered":"2024-09-21T13:49:02.062231+0000","last_unstale":"2024-09-21T13:54:21.439689+0000","last_undegraded":"2024-09-21T13:54:21.439689+0000","last_fullsized":"2024-09-21T13:54:21.439689+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:18.124544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:18.124544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:39.080321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":273,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.442610+0000","last_change":"2024-09-21T13:53:18.373418+0000","last_active":"2024-09-21T13:54:21.442610+0000","last_peered":"2024-09-21T13:54:21.442610+0000","last_clean":"2024-09-21T13:54:21.442610+0000","last_became_active":"2024-09-21T13:49:04.432507+0000","last_became_peered":"2024-09-21T13:49:04.432507+0000","last_unstale":"2024-09-21T13:54:21.442610+0000","last_undegraded":"2024-09-21T13:54:21.442610+0000","last_fullsized":"2024-09-21T13:54:21.442610+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:18.373356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:18.373356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:38.812691+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"68'9","reported_seq":245,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.441273+0000","last_change":"2024-09-21T13:51:54.808758+0000","last_active":"2024-09-21T13:54:21.441273+0000","last_peered":"2024-09-21T13:54:21.441273+0000","last_clean":"2024-09-21T13:54:21.441273+0000","last_became_active":"2024-09-21T13:50:29.693589+0000","last_became_peered":"2024-09-21T13:50:29.693589+0000","last_unstale":"2024-09-21T13:54:21.441273+0000","last_undegraded":"2024-09-21T13:54:21.441273+0000","last_fullsized":"2024-09-21T13:54:21.441273+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:51:54.808559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:54.808559+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"61'1","reported_seq":283,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.442591+0000","last_change":"2024-09-21T13:54:18.838960+0000","last_active":"2024-09-21T13:54:21.442591+0000","last_peered":"2024-09-21T13:54:21.442591+0000","last_clean":"2024-09-21T13:54:21.442591+0000","last_became_active":"2024-09-21T13:48:58.050645+0000","last_became_peered":"2024-09-21T13:48:58.050645+0000","last_unstale":"2024-09-21T13:54:21.442591+0000","last_undegraded":"2024-09-21T13:54:21.442591+0000","last_fullsized":"2024-09-21T13:54:21.442591+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:54:18.838906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:18.838906+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:26.089820+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":269,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.441175+0000","last_change":"2024-09-21T13:50:43.375425+0000","last_active":"2024-09-21T13:54:21.441175+0000","last_peered":"2024-09-21T13:54:21.441175+0000","last_clean":"2024-09-21T13:54:21.441175+0000","last_became_active":"2024-09-21T13:49:02.061775+0000","last_became_peered":"2024-09-21T13:49:02.061775+0000","last_unstale":"2024-09-21T13:54:21.441175+0000","last_undegraded":"2024-09-21T13:54:21.441175+0000","last_fullsized":"2024-09-21T13:54:21.441175+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:43.375381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:43.375381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":219,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T13:54:26.475650+0000","last_change":"2024-09-21T13:52:04.324521+0000","last_active":"2024-09-21T13:54:26.475650+0000","last_peered":"2024-09-21T13:54:26.475650+0000","last_clean":"2024-09-21T13:54:26.475650+0000","last_became_active":"2024-09-21T13:52:04.323549+0000","last_became_peered":"2024-09-21T13:52:04.323549+0000","last_unstale":"2024-09-21T13:54:26.475650+0000","last_undegraded":"2024-09-21T13:54:26.475650+0000","last_fullsized":"2024-09-21T13:54:26.475650+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:41.808746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:41.808746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:34.950671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"68'9","reported_seq":290,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.438381+0000","last_change":"2024-09-21T13:51:28.547052+0000","last_active":"2024-09-21T13:54:21.438381+0000","last_peered":"2024-09-21T13:54:21.438381+0000","last_clean":"2024-09-21T13:54:21.438381+0000","last_became_active":"2024-09-21T13:49:00.427766+0000","last_became_peered":"2024-09-21T13:49:00.427766+0000","last_unstale":"2024-09-21T13:54:21.438381+0000","last_undegraded":"2024-09-21T13:54:21.438381+0000","last_fullsized":"2024-09-21T13:54:21.438381+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:51:28.546958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:28.546958+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:09.056462+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":217,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.442897+0000","last_change":"2024-09-21T13:54:02.683225+0000","last_active":"2024-09-21T13:54:21.442897+0000","last_peered":"2024-09-21T13:54:21.442897+0000","last_clean":"2024-09-21T13:54:21.442897+0000","last_became_active":"2024-09-21T13:50:29.690634+0000","last_became_peered":"2024-09-21T13:50:29.690634+0000","last_unstale":"2024-09-21T13:54:21.442897+0000","last_undegraded":"2024-09-21T13:54:21.442897+0000","last_fullsized":"2024-09-21T13:54:21.442897+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:02.683172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:02.683172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:13.538040+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"132'10","reported_seq":364,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.440599+0000","last_change":"2024-09-21T13:54:20.946861+0000","last_active":"2024-09-21T13:54:21.440599+0000","last_peered":"2024-09-21T13:54:21.440599+0000","last_clean":"2024-09-21T13:54:21.440599+0000","last_became_active":"2024-09-21T13:52:04.318626+0000","last_became_peered":"2024-09-21T13:52:04.318626+0000","last_unstale":"2024-09-21T13:54:21.440599+0000","last_undegraded":"2024-09-21T13:54:21.440599+0000","last_fullsized":"2024-09-21T13:54:21.440599+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'10","last_scrub_stamp":"2024-09-21T13:54:20.946782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:20.946782+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:42.458501+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":284,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.437338+0000","last_change":"2024-09-21T13:53:51.285122+0000","last_active":"2024-09-21T13:54:21.437338+0000","last_peered":"2024-09-21T13:54:21.437338+0000","last_clean":"2024-09-21T13:54:21.437338+0000","last_became_active":"2024-09-21T13:52:04.315865+0000","last_became_peered":"2024-09-21T13:52:04.315865+0000","last_unstale":"2024-09-21T13:54:21.437338+0000","last_undegraded":"2024-09-21T13:54:21.437338+0000","last_fullsized":"2024-09-21T13:54:21.437338+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:51.285077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:51.285077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:16.987394+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"68'10","reported_seq":288,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.440838+0000","last_change":"2024-09-21T13:51:11.545546+0000","last_active":"2024-09-21T13:54:21.440838+0000","last_peered":"2024-09-21T13:54:21.440838+0000","last_clean":"2024-09-21T13:54:21.440838+0000","last_became_active":"2024-09-21T13:49:00.426071+0000","last_became_peered":"2024-09-21T13:49:00.426071+0000","last_unstale":"2024-09-21T13:54:21.440838+0000","last_undegraded":"2024-09-21T13:54:21.440838+0000","last_fullsized":"2024-09-21T13:54:21.440838+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:51:11.545458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:11.545458+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":288,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.443315+0000","last_change":"2024-09-21T13:52:04.318335+0000","last_active":"2024-09-21T13:54:21.443315+0000","last_peered":"2024-09-21T13:54:21.443315+0000","last_clean":"2024-09-21T13:54:21.443315+0000","last_became_active":"2024-09-21T13:52:04.318006+0000","last_became_peered":"2024-09-21T13:52:04.318006+0000","last_unstale":"2024-09-21T13:54:21.443315+0000","last_undegraded":"2024-09-21T13:54:21.443315+0000","last_fullsized":"2024-09-21T13:54:21.443315+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:41.162315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:41.162315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:42.458106+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"68'8","reported_seq":344,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.438181+0000","last_change":"2024-09-21T13:54:03.065465+0000","last_active":"2024-09-21T13:54:21.438181+0000","last_peered":"2024-09-21T13:54:21.438181+0000","last_clean":"2024-09-21T13:54:21.438181+0000","last_became_active":"2024-09-21T13:49:02.067487+0000","last_became_peered":"2024-09-21T13:49:02.067487+0000","last_unstale":"2024-09-21T13:54:21.438181+0000","last_undegraded":"2024-09-21T13:54:21.438181+0000","last_fullsized":"2024-09-21T13:54:21.438181+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:54:03.065395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:03.065395+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:30.943315+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":209,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.443450+0000","last_change":"2024-09-21T13:50:33.050276+0000","last_active":"2024-09-21T13:54:21.443450+0000","last_peered":"2024-09-21T13:54:21.443450+0000","last_clean":"2024-09-21T13:54:21.443450+0000","last_became_active":"2024-09-21T13:50:29.695082+0000","last_became_peered":"2024-09-21T13:50:29.695082+0000","last_unstale":"2024-09-21T13:54:21.443450+0000","last_undegraded":"2024-09-21T13:54:21.443450+0000","last_fullsized":"2024-09-21T13:54:21.443450+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:33.050227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:33.050227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"68'11","reported_seq":300,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.436846+0000","last_change":"2024-09-21T13:53:26.146937+0000","last_active":"2024-09-21T13:54:21.436846+0000","last_peered":"2024-09-21T13:54:21.436846+0000","last_clean":"2024-09-21T13:54:21.436846+0000","last_became_active":"2024-09-21T13:49:00.055024+0000","last_became_peered":"2024-09-21T13:49:00.055024+0000","last_unstale":"2024-09-21T13:54:21.436846+0000","last_undegraded":"2024-09-21T13:54:21.436846+0000","last_fullsized":"2024-09-21T13:54:21.436846+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:53:26.146725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:26.146725+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:44.146330+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":32,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.442652+0000","last_change":"2024-09-21T13:54:04.377034+0000","last_active":"2024-09-21T13:54:21.442652+0000","last_peered":"2024-09-21T13:54:21.442652+0000","last_clean":"2024-09-21T13:54:21.442652+0000","last_became_active":"2024-09-21T13:54:04.376614+0000","last_became_peered":"2024-09-21T13:54:04.376614+0000","last_unstale":"2024-09-21T13:54:21.442652+0000","last_undegraded":"2024-09-21T13:54:21.442652+0000","last_fullsized":"2024-09-21T13:54:21.442652+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:01.228213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:01.228213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:14.625514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":286,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.431199+0000","last_change":"2024-09-21T13:53:29.868061+0000","last_active":"2024-09-21T13:54:21.431199+0000","last_peered":"2024-09-21T13:54:21.431199+0000","last_clean":"2024-09-21T13:54:21.431199+0000","last_became_active":"2024-09-21T13:49:02.062868+0000","last_became_peered":"2024-09-21T13:49:02.062868+0000","last_unstale":"2024-09-21T13:54:21.431199+0000","last_undegraded":"2024-09-21T13:54:21.431199+0000","last_fullsized":"2024-09-21T13:54:21.431199+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:29.867278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:53:29.867278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:49.405014+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":265,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.439505+0000","last_change":"2024-09-21T13:51:24.549993+0000","last_active":"2024-09-21T13:54:21.439505+0000","last_peered":"2024-09-21T13:54:21.439505+0000","last_clean":"2024-09-21T13:54:21.439505+0000","last_became_active":"2024-09-21T13:49:04.082961+0000","last_became_peered":"2024-09-21T13:49:04.082961+0000","last_unstale":"2024-09-21T13:54:21.439505+0000","last_undegraded":"2024-09-21T13:54:21.439505+0000","last_fullsized":"2024-09-21T13:54:21.439505+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:24.549706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:51:24.549706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"68'9","reported_seq":301,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.431480+0000","last_change":"2024-09-21T13:52:10.447789+0000","last_active":"2024-09-21T13:54:21.431480+0000","last_peered":"2024-09-21T13:54:21.431480+0000","last_clean":"2024-09-21T13:54:21.431480+0000","last_became_active":"2024-09-21T13:52:04.320149+0000","last_became_peered":"2024-09-21T13:52:04.320149+0000","last_unstale":"2024-09-21T13:54:21.431480+0000","last_undegraded":"2024-09-21T13:54:21.431480+0000","last_fullsized":"2024-09-21T13:54:21.431480+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:52:10.447530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:52:10.447530+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":277,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.442905+0000","last_change":"2024-09-21T13:54:06.687343+0000","last_active":"2024-09-21T13:54:21.442905+0000","last_peered":"2024-09-21T13:54:21.442905+0000","last_clean":"2024-09-21T13:54:21.442905+0000","last_became_active":"2024-09-21T13:48:58.057516+0000","last_became_peered":"2024-09-21T13:48:58.057516+0000","last_unstale":"2024-09-21T13:54:21.442905+0000","last_undegraded":"2024-09-21T13:54:21.442905+0000","last_fullsized":"2024-09-21T13:54:21.442905+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:06.687302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:06.687302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:28.415240+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":277,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.440681+0000","last_change":"2024-09-21T13:54:19.999192+0000","last_active":"2024-09-21T13:54:21.440681+0000","last_peered":"2024-09-21T13:54:21.440681+0000","last_clean":"2024-09-21T13:54:21.440681+0000","last_became_active":"2024-09-21T13:49:02.071922+0000","last_became_peered":"2024-09-21T13:49:02.071922+0000","last_unstale":"2024-09-21T13:54:21.440681+0000","last_undegraded":"2024-09-21T13:54:21.440681+0000","last_fullsized":"2024-09-21T13:54:21.440681+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:19.999125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:19.999125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:38.368141+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":202,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.433180+0000","last_change":"2024-09-21T13:50:29.689537+0000","last_active":"2024-09-21T13:54:21.433180+0000","last_peered":"2024-09-21T13:54:21.433180+0000","last_clean":"2024-09-21T13:54:21.433180+0000","last_became_active":"2024-09-21T13:50:29.689045+0000","last_became_peered":"2024-09-21T13:50:29.689045+0000","last_unstale":"2024-09-21T13:54:21.433180+0000","last_undegraded":"2024-09-21T13:54:21.433180+0000","last_fullsized":"2024-09-21T13:54:21.433180+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:14.052362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"68'11","reported_seq":300,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:22.978354+0000","last_change":"2024-09-21T13:54:22.978354+0000","last_active":"2024-09-21T13:54:22.978354+0000","last_peered":"2024-09-21T13:54:22.978354+0000","last_clean":"2024-09-21T13:54:22.978354+0000","last_became_active":"2024-09-21T13:49:00.426091+0000","last_became_peered":"2024-09-21T13:49:00.426091+0000","last_unstale":"2024-09-21T13:54:22.978354+0000","last_undegraded":"2024-09-21T13:54:22.978354+0000","last_fullsized":"2024-09-21T13:54:22.978354+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:54:22.978267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:22.978267+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:40.972799+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":270,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.439775+0000","last_change":"2024-09-21T13:48:58.052281+0000","last_active":"2024-09-21T13:54:21.439775+0000","last_peered":"2024-09-21T13:54:21.439775+0000","last_clean":"2024-09-21T13:54:21.439775+0000","last_became_active":"2024-09-21T13:48:58.051974+0000","last_became_peered":"2024-09-21T13:48:58.051974+0000","last_unstale":"2024-09-21T13:54:21.439775+0000","last_undegraded":"2024-09-21T13:54:21.439775+0000","last_fullsized":"2024-09-21T13:54:21.439775+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:08.067332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":269,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.441888+0000","last_change":"2024-09-21T13:50:37.118637+0000","last_active":"2024-09-21T13:54:21.441888+0000","last_peered":"2024-09-21T13:54:21.441888+0000","last_clean":"2024-09-21T13:54:21.441888+0000","last_became_active":"2024-09-21T13:49:02.056006+0000","last_became_peered":"2024-09-21T13:49:02.056006+0000","last_unstale":"2024-09-21T13:54:21.441888+0000","last_undegraded":"2024-09-21T13:54:21.441888+0000","last_fullsized":"2024-09-21T13:54:21.441888+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:37.118594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:37.118594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:12.668029+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":265,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.440828+0000","last_change":"2024-09-21T13:51:55.827109+0000","last_active":"2024-09-21T13:54:21.440828+0000","last_peered":"2024-09-21T13:54:21.440828+0000","last_clean":"2024-09-21T13:54:21.440828+0000","last_became_active":"2024-09-21T13:49:04.434034+0000","last_became_peered":"2024-09-21T13:49:04.434034+0000","last_unstale":"2024-09-21T13:54:21.440828+0000","last_undegraded":"2024-09-21T13:54:21.440828+0000","last_fullsized":"2024-09-21T13:54:21.440828+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:55.826922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:51:55.826922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"68'4","reported_seq":279,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.439741+0000","last_change":"2024-09-21T13:53:27.826804+0000","last_active":"2024-09-21T13:54:21.439741+0000","last_peered":"2024-09-21T13:54:21.439741+0000","last_clean":"2024-09-21T13:54:21.439741+0000","last_became_active":"2024-09-21T13:49:00.063066+0000","last_became_peered":"2024-09-21T13:49:00.063066+0000","last_unstale":"2024-09-21T13:54:21.439741+0000","last_undegraded":"2024-09-21T13:54:21.439741+0000","last_fullsized":"2024-09-21T13:54:21.439741+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4","last_scrub_stamp":"2024-09-21T13:53:27.826729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:27.826729+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:54.701343+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":210,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.440098+0000","last_change":"2024-09-21T13:53:32.253481+0000","last_active":"2024-09-21T13:54:21.440098+0000","last_peered":"2024-09-21T13:54:21.440098+0000","last_clean":"2024-09-21T13:54:21.440098+0000","last_became_active":"2024-09-21T13:50:29.690269+0000","last_became_peered":"2024-09-21T13:50:29.690269+0000","last_unstale":"2024-09-21T13:54:21.440098+0000","last_undegraded":"2024-09-21T13:54:21.440098+0000","last_fullsized":"2024-09-21T13:54:21.440098+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:32.253408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:53:32.253408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:51.887076+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":286,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.432751+0000","last_change":"2024-09-21T13:53:51.254522+0000","last_active":"2024-09-21T13:54:21.432751+0000","last_peered":"2024-09-21T13:54:21.432751+0000","last_clean":"2024-09-21T13:54:21.432751+0000","last_became_active":"2024-09-21T13:49:02.064789+0000","last_became_peered":"2024-09-21T13:49:02.064789+0000","last_unstale":"2024-09-21T13:54:21.432751+0000","last_undegraded":"2024-09-21T13:54:21.432751+0000","last_fullsized":"2024-09-21T13:54:21.432751+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:51.254456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:53:51.254456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:51.405316+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":276,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.442693+0000","last_change":"2024-09-21T13:52:04.322055+0000","last_active":"2024-09-21T13:54:21.442693+0000","last_peered":"2024-09-21T13:54:21.442693+0000","last_clean":"2024-09-21T13:54:21.442693+0000","last_became_active":"2024-09-21T13:52:04.321667+0000","last_became_peered":"2024-09-21T13:52:04.321667+0000","last_unstale":"2024-09-21T13:54:21.442693+0000","last_undegraded":"2024-09-21T13:54:21.442693+0000","last_fullsized":"2024-09-21T13:54:21.442693+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:39.151320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:39.151320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:07.711986+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":274,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.430820+0000","last_change":"2024-09-21T13:52:27.409588+0000","last_active":"2024-09-21T13:54:21.430820+0000","last_peered":"2024-09-21T13:54:21.430820+0000","last_clean":"2024-09-21T13:54:21.430820+0000","last_became_active":"2024-09-21T13:49:04.075562+0000","last_became_peered":"2024-09-21T13:49:04.075562+0000","last_unstale":"2024-09-21T13:54:21.430820+0000","last_undegraded":"2024-09-21T13:54:21.430820+0000","last_fullsized":"2024-09-21T13:54:21.430820+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:27.409408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:27.409408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":277,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.439869+0000","last_change":"2024-09-21T13:53:25.873998+0000","last_active":"2024-09-21T13:54:21.439869+0000","last_peered":"2024-09-21T13:54:21.439869+0000","last_clean":"2024-09-21T13:54:21.439869+0000","last_became_active":"2024-09-21T13:48:58.055497+0000","last_became_peered":"2024-09-21T13:48:58.055497+0000","last_unstale":"2024-09-21T13:54:21.439869+0000","last_undegraded":"2024-09-21T13:54:21.439869+0000","last_fullsized":"2024-09-21T13:54:21.439869+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:25.865470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:53:25.865470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:51.548858+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"68'11","reported_seq":323,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.438249+0000","last_change":"2024-09-21T13:52:04.321178+0000","last_active":"2024-09-21T13:54:21.438249+0000","last_peered":"2024-09-21T13:54:21.438249+0000","last_clean":"2024-09-21T13:54:21.438249+0000","last_became_active":"2024-09-21T13:52:04.320647+0000","last_became_peered":"2024-09-21T13:52:04.320647+0000","last_unstale":"2024-09-21T13:54:21.438249+0000","last_undegraded":"2024-09-21T13:54:21.438249+0000","last_fullsized":"2024-09-21T13:54:21.438249+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:50:35.790090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:35.790090+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:11.179000+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":228,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-21T13:54:21.442190+0000","last_change":"2024-09-21T13:54:01.727728+0000","last_active":"2024-09-21T13:54:21.442190+0000","last_peered":"2024-09-21T13:54:21.442190+0000","last_clean":"2024-09-21T13:54:21.442190+0000","last_became_active":"2024-09-21T13:52:04.322880+0000","last_became_peered":"2024-09-21T13:52:04.322880+0000","last_unstale":"2024-09-21T13:54:21.442190+0000","last_undegraded":"2024-09-21T13:54:21.442190+0000","last_fullsized":"2024-09-21T13:54:21.442190+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:01.727680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:01.727680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:16.479686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":992280576,"num_objects":15141,"num_object_clones":0,"num_object_copies":45423,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15141,"num_write_kb":969024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3181707264,"data_stored":3181707264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15141,"ondisk_log_size":15141,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69,"ondisk_log_size":69,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":694,"num_read_kb":451,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":395,"ondisk_log_size":395,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":320,"num_read_kb":315,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234054,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":397212,"kb_used_data":271624,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":93347940,"statfs":{"total":95995035648,"available":95588290560,"internally_reserved":0,"allocated":278142976,"data_stored":277632318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":128580042},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462987,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":409608,"kb_used_data":288180,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93335544,"statfs":{"total":95995035648,"available":95575597056,"internally_reserved":0,"allocated":295096320,"data_stored":294605057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":124320198},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724626,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":564680,"kb_used_data":447412,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":93180472,"statfs":{"total":95995035648,"available":95416803328,"internally_reserved":0,"allocated":458149888,"data_stored":457658997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":120060357},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920856,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":440292,"kb_used_data":296656,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":93304860,"statfs":{"total":95995035648,"available":95544176640,"internally_reserved":0,"allocated":303775744,"data_stored":303255883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117085,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":657512,"kb_used_data":377812,"kb_used_omap":1,"kb_used_meta":279678,"kb_avail":93087640,"statfs":{"total":95995035648,"available":95321743360,"internally_reserved":0,"allocated":386879488,"data_stored":386356361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":286390727},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":130,"seq":558345748510,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":772904,"kb_used_data":555984,"kb_used_omap":4,"kb_used_meta":216827,"kb_avail":92972248,"statfs":{"total":95995035648,"available":95203581952,"internally_reserved":0,"allocated":569327616,"data_stored":568824451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4441,"internal_metadata":222031527},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574954,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":515320,"kb_used_data":367528,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":93229832,"statfs":{"total":95995035648,"available":95467347968,"internally_reserved":0,"allocated":376348672,"data_stored":375804759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738480,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":500252,"kb_used_data":328840,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":93244900,"statfs":{"total":95995035648,"available":95482777600,"internally_reserved":0,"allocated":336732160,"data_stored":336220775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":175503813},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":379650048,"data_stored":379650048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":428343296,"data_stored":428343296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593559552,"data_stored":593559552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":391118848,"data_stored":391118848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":303235072,"data_stored":303235072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":480444416,"data_stored":480444416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":303955968,"data_stored":303955968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":301400064,"data_stored":301400064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T13:54:37.326 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd pool set .mgr pgp_num 1 2024-09-21T13:54:37.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:37.330 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:37.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:54:37.430 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:54:37.458 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:37.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:37.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:54:37.560 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:37.591 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:37.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:37.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:37.693 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:37.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:37.723 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:37.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:37.787 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:37 smithi049 ceph-mon[30080]: pgmap v275: 148 pgs: 148 active+clean; 2.2 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 136 MiB/s wr, 2.18k op/s 2024-09-21T13:54:37.787 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:37 smithi049 ceph-mon[30080]: osdmap e193: 8 total, 8 up, 8 in 2024-09-21T13:54:37.787 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:37 smithi049 ceph-mon[40489]: pgmap v275: 148 pgs: 148 active+clean; 2.2 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 136 MiB/s wr, 2.18k op/s 2024-09-21T13:54:37.788 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:37 smithi049 ceph-mon[40489]: osdmap e193: 8 total, 8 up, 8 in 2024-09-21T13:54:37.824 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:54:37.880 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:37.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:37.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:54:37.897 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:54:37.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:37 smithi187 ceph-mon[33032]: pgmap v275: 148 pgs: 148 active+clean; 2.2 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 136 MiB/s wr, 2.18k op/s 2024-09-21T13:54:37.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:37 smithi187 ceph-mon[33032]: osdmap e193: 8 total, 8 up, 8 in 2024-09-21T13:54:37.983 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:38.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:38.017 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:38.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:38.118 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:38.162 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:38.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:38.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:38.264 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:54:38.290 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:2024-09-21T13:54:38.290233+0000 min lat: 0.00131689 max lat: 0.181286 avg lat: 0.00732702 2024-09-21T13:54:38.290 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T13:54:38.290 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 20 15 43665 43650 136.376 113.375 0.00980249 0.00732702 2024-09-21T13:54:38.305 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:38.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:38.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:54:38.408 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:54:38.444 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:38.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:38.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:54:38.545 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:54:38.573 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:38.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:38.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:54:38.674 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:54:38.715 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:38.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:38.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:54:38.816 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:54:38.852 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:38.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:38.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:54:38.954 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:54:38.983 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:38.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:38.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:54:39.085 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:54:39.114 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:39.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:39.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:54:39.216 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:54:39.241 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:39.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:39.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:54:39.289 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 21 16 45424 45408 135.114 109.875 0.00912745 0.0073957 2024-09-21T13:54:39.333 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:54:39.333 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:54:39.333 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:54:39.333 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:54:39.333 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:54:39.333 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:54:39.334 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:54:39.334 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:54:39.337 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:54:39.343 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:54:39.361 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:54:39.362 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:54:39.399 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:54:39.400 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:54:39.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:39.406 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:39.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:54:39.441 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:54:39.441 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:54:39.486 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:54:39.486 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:54:39.507 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:54:39.524 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:54:39 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:54:39.559 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:39.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:39.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:54:39.662 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:54:39.692 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:39.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:39.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:54:39.794 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:54:39.831 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:39.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:39.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:54:39.844 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T13:54:39.912 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:39 smithi049 ceph-mon[40489]: pgmap v277: 148 pgs: 148 active+clean; 2.2 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 549 B/s rd, 102 MiB/s wr, 1.64k op/s 2024-09-21T13:54:39.912 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:39 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/755013397' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:54:39.912 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:39 smithi049 ceph-mon[30080]: pgmap v277: 148 pgs: 148 active+clean; 2.2 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 549 B/s rd, 102 MiB/s wr, 1.64k op/s 2024-09-21T13:54:39.912 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:39 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/755013397' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:54:39.930 INFO:teuthology.orchestra.run.smithi049.stderr:set pool 1 pgp_num to 1 2024-09-21T13:54:39.933 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:54:39.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:39 smithi187 ceph-mon[33032]: pgmap v277: 148 pgs: 148 active+clean; 2.2 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 549 B/s rd, 102 MiB/s wr, 1.64k op/s 2024-09-21T13:54:39.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:39 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/755013397' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:54:39.978 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:39.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:39.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:54:40.080 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:54:40.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:40.139 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:40.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:54:40.240 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:54:40.289 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 22 16 46937 46921 133.27 94.5625 0.00850422 0.00749866 2024-09-21T13:54:40.296 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:40.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:40.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:54:40.400 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:40.442 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:40.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:40.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:40.473 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T13:54:40.473 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-09-21T13:54:40.473 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-09-21T13:54:40.473 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-09-21T13:54:40.473 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.473 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:163570 7.8 7.330e1638 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e130)", 2024-09-21T13:54:40.473 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:03.335126+0000", 2024-09-21T13:54:40.473 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 156.90881966200001, 2024-09-21T13:54:40.473 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97989348499999995, 2024-09-21T13:54:40.473 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:54:40.473 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:54:40.473 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T13:54:40.474 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:54:40.474 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:54:40.474 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:40.474 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 163570 2024-09-21T13:54:40.474 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.474 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:54:40.474 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.474 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:54:40.474 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.335126+0000", 2024-09-21T13:54:40.474 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:40.474 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.474 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.474 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:54:40.475 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.335126+0000", 2024-09-21T13:54:40.475 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:40.475 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.475 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.475 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:54:40.475 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.335127+0000", 2024-09-21T13:54:40.475 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5209999999999999e-06 2024-09-21T13:54:40.475 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.475 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.475 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:54:40.475 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.335132+0000", 2024-09-21T13:54:40.475 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4730000000000002e-06 2024-09-21T13:54:40.475 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.475 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.476 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:54:40.476 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.335142+0000", 2024-09-21T13:54:40.476 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8770000000000005e-06 2024-09-21T13:54:40.476 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.476 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.476 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:54:40.476 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.335151+0000", 2024-09-21T13:54:40.476 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7650000000000005e-06 2024-09-21T13:54:40.476 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.476 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.476 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:40.476 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.336965+0000", 2024-09-21T13:54:40.476 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001813907 2024-09-21T13:54:40.476 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.477 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.477 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:54:40.477 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.336978+0000", 2024-09-21T13:54:40.477 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2685e-05 2024-09-21T13:54:40.477 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.477 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.477 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:40.477 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:04.315001+0000", 2024-09-21T13:54:40.477 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97802330000000004 2024-09-21T13:54:40.477 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.477 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.477 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:54:40.477 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:04.315019+0000", 2024-09-21T13:54:40.477 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7957000000000001e-05 2024-09-21T13:54:40.478 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:40.478 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:54:40.478 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:40.478 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.478 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.478 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:163585 7.8 7.9fcafe08 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-21T13:54:40.478 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:03.339560+0000", 2024-09-21T13:54:40.478 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 156.90438548700001, 2024-09-21T13:54:40.478 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97550176099999997, 2024-09-21T13:54:40.478 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:54:40.478 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:54:40.478 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T13:54:40.478 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:54:40.478 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:54:40.479 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:40.479 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 163585 2024-09-21T13:54:40.479 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.479 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:54:40.479 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.479 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:54:40.479 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.339560+0000", 2024-09-21T13:54:40.479 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:40.479 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.479 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.479 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:54:40.479 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.339560+0000", 2024-09-21T13:54:40.479 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:40.479 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.480 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.480 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:54:40.480 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.339562+0000", 2024-09-21T13:54:40.480 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6449999999999999e-06 2024-09-21T13:54:40.480 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.480 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.480 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:54:40.480 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.339568+0000", 2024-09-21T13:54:40.480 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2169999999999999e-06 2024-09-21T13:54:40.480 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.480 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.480 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:54:40.480 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.339580+0000", 2024-09-21T13:54:40.481 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1846e-05 2024-09-21T13:54:40.481 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.481 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.481 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:54:40.481 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.339589+0000", 2024-09-21T13:54:40.481 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.798e-06 2024-09-21T13:54:40.481 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.481 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.481 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:40.481 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.342967+0000", 2024-09-21T13:54:40.481 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0033776990000000001 2024-09-21T13:54:40.481 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.481 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.482 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:54:40.482 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.342980+0000", 2024-09-21T13:54:40.482 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2758000000000001e-05 2024-09-21T13:54:40.482 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.482 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.482 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:40.482 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:04.315052+0000", 2024-09-21T13:54:40.482 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97207221799999999 2024-09-21T13:54:40.482 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.482 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.482 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:54:40.482 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:04.315062+0000", 2024-09-21T13:54:40.482 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.5799999999999998e-06 2024-09-21T13:54:40.482 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:40.483 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:54:40.483 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:40.483 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.483 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.483 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:163595 7.8 7.9f5fc9f8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-21T13:54:40.483 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:03.360541+0000", 2024-09-21T13:54:40.483 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 156.88340431699999, 2024-09-21T13:54:40.483 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.95455601800000001, 2024-09-21T13:54:40.483 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:54:40.483 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:54:40.483 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T13:54:40.483 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:54:40.483 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:54:40.483 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:40.484 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 163595 2024-09-21T13:54:40.484 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.484 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:54:40.484 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.484 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:54:40.484 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.360541+0000", 2024-09-21T13:54:40.484 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:40.484 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.484 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.484 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:54:40.484 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.360541+0000", 2024-09-21T13:54:40.484 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:40.484 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.484 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.485 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:54:40.485 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.360544+0000", 2024-09-21T13:54:40.485 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.632e-06 2024-09-21T13:54:40.485 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.485 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.485 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:54:40.485 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.360554+0000", 2024-09-21T13:54:40.485 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0210000000000001e-05 2024-09-21T13:54:40.485 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.485 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.485 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:54:40.485 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.360579+0000", 2024-09-21T13:54:40.485 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4878000000000001e-05 2024-09-21T13:54:40.485 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.486 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.486 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:54:40.486 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.360589+0000", 2024-09-21T13:54:40.486 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7960000000000007e-06 2024-09-21T13:54:40.486 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.486 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.486 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:40.486 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.360662+0000", 2024-09-21T13:54:40.486 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3514000000000001e-05 2024-09-21T13:54:40.486 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.486 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.486 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:54:40.486 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.360673+0000", 2024-09-21T13:54:40.487 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1080000000000001e-05 2024-09-21T13:54:40.487 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.487 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.487 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:40.487 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:04.315088+0000", 2024-09-21T13:54:40.487 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.954414859 2024-09-21T13:54:40.487 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.487 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.487 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:54:40.487 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:04.315097+0000", 2024-09-21T13:54:40.487 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0489999999999993e-06 2024-09-21T13:54:40.487 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:40.487 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:54:40.488 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:40.488 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.488 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.488 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:163602 7.8 7.3bec8d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-21T13:54:40.488 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:03.369662+0000", 2024-09-21T13:54:40.488 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 156.874283561, 2024-09-21T13:54:40.488 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.945473868, 2024-09-21T13:54:40.488 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:54:40.488 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:54:40.488 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T13:54:40.488 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:54:40.488 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:54:40.488 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:40.488 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 163602 2024-09-21T13:54:40.488 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.489 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:54:40.489 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.489 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:54:40.489 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.369662+0000", 2024-09-21T13:54:40.489 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:40.489 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.489 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.489 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:54:40.489 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.369662+0000", 2024-09-21T13:54:40.489 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:40.489 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.489 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.489 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:54:40.490 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.369664+0000", 2024-09-21T13:54:40.490 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.136e-06 2024-09-21T13:54:40.490 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.490 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.490 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:54:40.490 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.369671+0000", 2024-09-21T13:54:40.490 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9110000000000004e-06 2024-09-21T13:54:40.490 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.490 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.490 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:54:40.490 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.369689+0000", 2024-09-21T13:54:40.490 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7757999999999998e-05 2024-09-21T13:54:40.490 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.490 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.491 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:54:40.491 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.369700+0000", 2024-09-21T13:54:40.491 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.146e-05 2024-09-21T13:54:40.491 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.491 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.491 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:40.491 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.369779+0000", 2024-09-21T13:54:40.491 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9260999999999999e-05 2024-09-21T13:54:40.491 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.491 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.491 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:54:40.491 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.369793+0000", 2024-09-21T13:54:40.491 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3200000000000001e-05 2024-09-21T13:54:40.491 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.492 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.492 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:40.492 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:04.315126+0000", 2024-09-21T13:54:40.492 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.94533302500000005 2024-09-21T13:54:40.492 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.492 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.492 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:54:40.492 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:04.315136+0000", 2024-09-21T13:54:40.492 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0117e-05 2024-09-21T13:54:40.492 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:40.492 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:54:40.492 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:40.492 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.492 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.493 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:201459 7.8 7:1473b567:::benchmark_data_smithi049_76798_object65039:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-21T13:54:40.493 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:41.960649+0000", 2024-09-21T13:54:40.493 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 118.283296392, 2024-09-21T13:54:40.493 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.025332715, 2024-09-21T13:54:40.493 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:54:40.493 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:54:40.493 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:40.493 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:54:40.493 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:54:40.493 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:40.493 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 201459 2024-09-21T13:54:40.493 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.493 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:54:40.493 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.494 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:54:40.494 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:41.960649+0000", 2024-09-21T13:54:40.494 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:40.494 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.494 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.494 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:54:40.494 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:41.960649+0000", 2024-09-21T13:54:40.494 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:40.494 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.494 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.494 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:54:40.494 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:41.960651+0000", 2024-09-21T13:54:40.494 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7990000000000001e-06 2024-09-21T13:54:40.495 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.495 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.495 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:54:40.495 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:41.960657+0000", 2024-09-21T13:54:40.495 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2260000000000002e-06 2024-09-21T13:54:40.495 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.495 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.495 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:54:40.495 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:41.960677+0000", 2024-09-21T13:54:40.495 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9783999999999999e-05 2024-09-21T13:54:40.495 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.495 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.495 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:54:40.495 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:41.960686+0000", 2024-09-21T13:54:40.496 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0710000000000001e-06 2024-09-21T13:54:40.496 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.496 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.496 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:40.496 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:41.964807+0000", 2024-09-21T13:54:40.496 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0041212560000000002 2024-09-21T13:54:40.496 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.496 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.496 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:54:40.496 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:41.964817+0000", 2024-09-21T13:54:40.496 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0030000000000001e-05 2024-09-21T13:54:40.496 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.496 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.496 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:40.497 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.976018+0000", 2024-09-21T13:54:40.497 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.011200412 2024-09-21T13:54:40.497 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.497 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.497 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:54:40.497 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.976136+0000", 2024-09-21T13:54:40.497 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000118106 2024-09-21T13:54:40.497 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.497 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.497 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-09-21T13:54:40.497 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.976247+0000", 2024-09-21T13:54:40.497 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000111418 2024-09-21T13:54:40.497 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.497 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.498 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:40.498 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.980913+0000", 2024-09-21T13:54:40.498 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0046656850000000001 2024-09-21T13:54:40.498 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.498 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.498 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:40.498 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.980959+0000", 2024-09-21T13:54:40.498 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6402000000000001e-05 2024-09-21T13:54:40.498 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.498 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.498 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:54:40.498 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.985848+0000", 2024-09-21T13:54:40.498 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0048883210000000002 2024-09-21T13:54:40.499 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.499 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.499 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:54:40.499 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.985906+0000", 2024-09-21T13:54:40.499 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8242000000000002e-05 2024-09-21T13:54:40.499 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.499 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.499 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:54:40.499 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.985982+0000", 2024-09-21T13:54:40.499 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5963000000000004e-05 2024-09-21T13:54:40.499 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:40.499 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:54:40.499 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:40.499 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.500 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.500 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:201494 7.8 7:1c4e9ccf:::benchmark_data_smithi049_76798_object65074:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-21T13:54:40.500 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:42.006815+0000", 2024-09-21T13:54:40.500 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 118.237129986, 2024-09-21T13:54:40.500 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97934932200000002, 2024-09-21T13:54:40.500 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:54:40.500 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:54:40.500 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:40.500 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:54:40.500 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:54:40.500 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:40.500 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 201494 2024-09-21T13:54:40.500 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.500 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:54:40.501 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.501 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:54:40.501 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006815+0000", 2024-09-21T13:54:40.501 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:40.501 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.501 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.501 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:54:40.501 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006815+0000", 2024-09-21T13:54:40.501 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:40.501 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.501 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.501 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:54:40.501 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006817+0000", 2024-09-21T13:54:40.501 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.618e-06 2024-09-21T13:54:40.502 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.502 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.502 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:54:40.502 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006823+0000", 2024-09-21T13:54:40.502 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0859999999999998e-06 2024-09-21T13:54:40.502 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.502 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.502 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:54:40.502 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006840+0000", 2024-09-21T13:54:40.502 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7149000000000001e-05 2024-09-21T13:54:40.502 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.502 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.502 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:54:40.502 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006848+0000", 2024-09-21T13:54:40.503 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9340000000000006e-06 2024-09-21T13:54:40.503 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.503 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.503 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:40.503 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006949+0000", 2024-09-21T13:54:40.503 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010043399999999999 2024-09-21T13:54:40.503 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.503 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.503 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:54:40.503 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006957+0000", 2024-09-21T13:54:40.503 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6580000000000008e-06 2024-09-21T13:54:40.503 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.503 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.503 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:40.504 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.979032+0000", 2024-09-21T13:54:40.504 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97207428699999998 2024-09-21T13:54:40.504 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.504 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.504 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:54:40.504 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.979198+0000", 2024-09-21T13:54:40.504 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016622800000000001 2024-09-21T13:54:40.504 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.504 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.504 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-09-21T13:54:40.504 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.979306+0000", 2024-09-21T13:54:40.504 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000108356 2024-09-21T13:54:40.504 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.504 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.504 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:40.505 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.980997+0000", 2024-09-21T13:54:40.505 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0016909640000000001 2024-09-21T13:54:40.505 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.505 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.505 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:40.505 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.981060+0000", 2024-09-21T13:54:40.505 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2516999999999998e-05 2024-09-21T13:54:40.505 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.505 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.505 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:54:40.505 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986075+0000", 2024-09-21T13:54:40.505 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.005014971 2024-09-21T13:54:40.505 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.505 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.506 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:54:40.506 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986121+0000", 2024-09-21T13:54:40.506 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6313999999999997e-05 2024-09-21T13:54:40.506 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.506 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.506 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:54:40.506 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986165+0000", 2024-09-21T13:54:40.506 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3806e-05 2024-09-21T13:54:40.506 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:40.506 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:54:40.506 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:40.506 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.506 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.506 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:201495 7.8 7:18dd0180:::benchmark_data_smithi049_76798_object65075:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-21T13:54:40.507 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:42.006872+0000", 2024-09-21T13:54:40.507 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 118.237072907, 2024-09-21T13:54:40.507 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97938486800000002, 2024-09-21T13:54:40.507 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:54:40.507 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:54:40.507 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:40.507 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:54:40.507 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:54:40.507 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:40.507 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 201495 2024-09-21T13:54:40.507 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.507 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:54:40.507 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.507 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:54:40.508 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006872+0000", 2024-09-21T13:54:40.508 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:40.508 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.508 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.508 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:54:40.508 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006872+0000", 2024-09-21T13:54:40.508 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:40.508 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.508 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.508 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:54:40.508 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006873+0000", 2024-09-21T13:54:40.508 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4899999999999995e-07 2024-09-21T13:54:40.508 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.509 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.509 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:54:40.509 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006878+0000", 2024-09-21T13:54:40.509 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0429999999999998e-06 2024-09-21T13:54:40.509 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.509 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.509 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:54:40.509 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006886+0000", 2024-09-21T13:54:40.509 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3399999999999998e-06 2024-09-21T13:54:40.509 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.509 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.509 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:54:40.509 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006894+0000", 2024-09-21T13:54:40.509 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1359999999999999e-06 2024-09-21T13:54:40.509 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.510 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.510 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:40.510 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006982+0000", 2024-09-21T13:54:40.510 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.8022000000000005e-05 2024-09-21T13:54:40.510 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.510 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.510 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:54:40.510 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006988+0000", 2024-09-21T13:54:40.510 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3470000000000004e-06 2024-09-21T13:54:40.510 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.510 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.510 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:40.511 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.981113+0000", 2024-09-21T13:54:40.511 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97412548499999996 2024-09-21T13:54:40.511 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.511 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.511 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:54:40.511 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.981230+0000", 2024-09-21T13:54:40.511 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000116666 2024-09-21T13:54:40.511 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.511 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.511 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-09-21T13:54:40.511 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.981349+0000", 2024-09-21T13:54:40.511 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000118572 2024-09-21T13:54:40.511 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.511 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.512 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:40.512 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.985825+0000", 2024-09-21T13:54:40.512 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0044766739999999999 2024-09-21T13:54:40.512 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.512 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.512 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:40.512 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986063+0000", 2024-09-21T13:54:40.512 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00023781299999999999 2024-09-21T13:54:40.512 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.512 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.512 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:54:40.512 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986172+0000", 2024-09-21T13:54:40.512 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010923499999999999 2024-09-21T13:54:40.512 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.513 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.513 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:54:40.513 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986236+0000", 2024-09-21T13:54:40.513 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3606000000000001e-05 2024-09-21T13:54:40.513 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.513 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.513 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:54:40.513 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986257+0000", 2024-09-21T13:54:40.513 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1379999999999999e-05 2024-09-21T13:54:40.513 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:40.513 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:54:40.513 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:40.513 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.513 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.514 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:201496 7.8 7:10f836b0:::benchmark_data_smithi049_76798_object65076:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-21T13:54:40.514 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:42.012648+0000", 2024-09-21T13:54:40.514 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 118.231296937, 2024-09-21T13:54:40.514 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97380752599999998, 2024-09-21T13:54:40.514 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:54:40.514 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:54:40.514 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:40.514 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:54:40.514 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:54:40.514 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:40.514 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 201496 2024-09-21T13:54:40.514 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.514 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:54:40.515 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.515 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:54:40.515 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.012648+0000", 2024-09-21T13:54:40.515 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:40.515 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.515 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.515 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:54:40.515 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.012648+0000", 2024-09-21T13:54:40.515 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:40.515 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.515 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.515 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:54:40.515 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.012650+0000", 2024-09-21T13:54:40.515 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.513e-06 2024-09-21T13:54:40.516 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.516 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.516 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:54:40.516 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.012656+0000", 2024-09-21T13:54:40.516 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7300000000000002e-06 2024-09-21T13:54:40.516 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.516 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.516 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:54:40.516 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.012664+0000", 2024-09-21T13:54:40.516 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6090000000000001e-06 2024-09-21T13:54:40.516 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.516 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.516 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:54:40.516 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.012670+0000", 2024-09-21T13:54:40.517 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5829999999999997e-06 2024-09-21T13:54:40.517 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.517 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.517 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:40.517 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.012740+0000", 2024-09-21T13:54:40.517 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9834999999999998e-05 2024-09-21T13:54:40.517 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.517 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.517 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:54:40.517 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.012749+0000", 2024-09-21T13:54:40.517 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9889999999999998e-06 2024-09-21T13:54:40.517 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.517 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.518 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:40.518 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.983365+0000", 2024-09-21T13:54:40.518 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97061665600000002 2024-09-21T13:54:40.518 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.518 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.518 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:54:40.518 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.983473+0000", 2024-09-21T13:54:40.518 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000107507 2024-09-21T13:54:40.518 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.518 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.518 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-09-21T13:54:40.518 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.983595+0000", 2024-09-21T13:54:40.518 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000121614 2024-09-21T13:54:40.519 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.519 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.519 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:54:40.519 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986265+0000", 2024-09-21T13:54:40.519 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002670113 2024-09-21T13:54:40.519 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.519 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.519 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:40.519 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986301+0000", 2024-09-21T13:54:40.519 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6130999999999997e-05 2024-09-21T13:54:40.519 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.519 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.519 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:40.519 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986370+0000", 2024-09-21T13:54:40.520 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8994999999999999e-05 2024-09-21T13:54:40.520 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.520 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.520 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:54:40.520 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986417+0000", 2024-09-21T13:54:40.520 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7691000000000001e-05 2024-09-21T13:54:40.520 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.520 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.520 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:54:40.520 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986456+0000", 2024-09-21T13:54:40.520 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8559999999999997e-05 2024-09-21T13:54:40.520 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:40.520 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:54:40.520 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:40.521 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.521 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.521 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:201499 7.8 7:1329afa8:::benchmark_data_smithi049_76798_object65079:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-21T13:54:40.521 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:42.026955+0000", 2024-09-21T13:54:40.521 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 118.216990362, 2024-09-21T13:54:40.521 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97876118000000001, 2024-09-21T13:54:40.521 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:54:40.521 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:54:40.521 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:40.521 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:54:40.521 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:54:40.521 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:40.521 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 201499 2024-09-21T13:54:40.521 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.522 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:54:40.522 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.522 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:54:40.522 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.026955+0000", 2024-09-21T13:54:40.522 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:40.522 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.522 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.522 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:54:40.522 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.026955+0000", 2024-09-21T13:54:40.522 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:40.522 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.522 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.522 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:54:40.523 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.026957+0000", 2024-09-21T13:54:40.523 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7999999999999999e-06 2024-09-21T13:54:40.523 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.523 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.523 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:54:40.523 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.026962+0000", 2024-09-21T13:54:40.523 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9400000000000001e-06 2024-09-21T13:54:40.523 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.523 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.523 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:54:40.523 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.026977+0000", 2024-09-21T13:54:40.523 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5087e-05 2024-09-21T13:54:40.523 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.523 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.524 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:54:40.524 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.026983+0000", 2024-09-21T13:54:40.524 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0090000000000004e-06 2024-09-21T13:54:40.524 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.524 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.524 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:40.524 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027044+0000", 2024-09-21T13:54:40.524 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1543000000000004e-05 2024-09-21T13:54:40.524 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.524 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.524 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:54:40.524 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027051+0000", 2024-09-21T13:54:40.524 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4590000000000003e-06 2024-09-21T13:54:40.524 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.525 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.525 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:40.525 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986512+0000", 2024-09-21T13:54:40.525 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.95946141399999996 2024-09-21T13:54:40.525 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.525 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.525 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:54:40.525 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986631+0000", 2024-09-21T13:54:40.525 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011896300000000001 2024-09-21T13:54:40.525 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.525 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.525 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-09-21T13:54:40.525 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986738+0000", 2024-09-21T13:54:40.525 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000106766 2024-09-21T13:54:40.526 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.526 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.526 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:40.526 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.994021+0000", 2024-09-21T13:54:40.526 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0072827329999999996 2024-09-21T13:54:40.526 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.526 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.526 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:40.526 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.994059+0000", 2024-09-21T13:54:40.526 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8053999999999999e-05 2024-09-21T13:54:40.526 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.526 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.526 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:54:40.526 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.005582+0000", 2024-09-21T13:54:40.527 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.011523342000000001 2024-09-21T13:54:40.527 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.527 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.527 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:54:40.527 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.005671+0000", 2024-09-21T13:54:40.527 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.8708999999999994e-05 2024-09-21T13:54:40.527 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.527 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.527 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:54:40.527 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.005716+0000", 2024-09-21T13:54:40.527 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5361000000000002e-05 2024-09-21T13:54:40.527 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:40.527 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:54:40.527 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:40.527 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.528 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.528 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:201500 7.8 7:19fe3d31:::benchmark_data_smithi049_76798_object65080:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-21T13:54:40.528 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:42.027000+0000", 2024-09-21T13:54:40.528 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 118.216944922, 2024-09-21T13:54:40.528 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97887608500000001, 2024-09-21T13:54:40.528 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:54:40.528 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:54:40.528 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:40.528 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:54:40.528 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:54:40.528 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:40.528 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 201500 2024-09-21T13:54:40.528 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.528 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:54:40.529 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.529 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:54:40.529 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027000+0000", 2024-09-21T13:54:40.529 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:40.529 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.529 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.529 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:54:40.529 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027000+0000", 2024-09-21T13:54:40.529 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:40.529 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.529 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.529 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:54:40.529 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027001+0000", 2024-09-21T13:54:40.529 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3200000000000001e-07 2024-09-21T13:54:40.529 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.530 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.530 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:54:40.530 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027003+0000", 2024-09-21T13:54:40.530 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8980000000000001e-06 2024-09-21T13:54:40.530 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.530 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.530 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:54:40.530 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027007+0000", 2024-09-21T13:54:40.530 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4290000000000003e-06 2024-09-21T13:54:40.530 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.530 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.530 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:54:40.530 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027010+0000", 2024-09-21T13:54:40.530 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3419999999999999e-06 2024-09-21T13:54:40.531 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.531 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.531 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:40.531 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027064+0000", 2024-09-21T13:54:40.531 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3474000000000003e-05 2024-09-21T13:54:40.531 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.531 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.531 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:54:40.531 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027067+0000", 2024-09-21T13:54:40.531 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.932e-06 2024-09-21T13:54:40.531 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.531 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.531 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:40.531 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.989128+0000", 2024-09-21T13:54:40.532 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.96206106899999999 2024-09-21T13:54:40.532 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.532 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.532 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:54:40.532 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.990066+0000", 2024-09-21T13:54:40.532 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00093830900000000002 2024-09-21T13:54:40.532 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.532 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.532 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-09-21T13:54:40.532 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.990172+0000", 2024-09-21T13:54:40.532 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000105271 2024-09-21T13:54:40.532 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.532 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.533 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:40.533 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.996356+0000", 2024-09-21T13:54:40.533 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0061840100000000002 2024-09-21T13:54:40.533 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.533 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.533 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:40.533 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.996388+0000", 2024-09-21T13:54:40.533 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2539000000000001e-05 2024-09-21T13:54:40.533 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.533 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.533 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:54:40.533 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.005774+0000", 2024-09-21T13:54:40.533 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0093859679999999997 2024-09-21T13:54:40.534 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.534 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.534 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:54:40.534 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.005838+0000", 2024-09-21T13:54:40.534 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3724000000000002e-05 2024-09-21T13:54:40.534 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.534 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.534 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:54:40.534 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.005877+0000", 2024-09-21T13:54:40.534 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8788000000000002e-05 2024-09-21T13:54:40.535 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:40.535 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:54:40.535 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:40.535 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.535 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.535 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:201501 7.8 7:15f77bbd:::benchmark_data_smithi049_76798_object65081:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-21T13:54:40.535 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:42.027294+0000", 2024-09-21T13:54:40.535 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 118.216650957, 2024-09-21T13:54:40.535 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97870270599999998, 2024-09-21T13:54:40.535 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:54:40.535 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:54:40.535 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:40.535 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:54:40.535 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:54:40.535 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:40.536 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 201501 2024-09-21T13:54:40.536 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.536 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:54:40.537 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.537 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:54:40.537 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027294+0000", 2024-09-21T13:54:40.537 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:40.538 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.538 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.538 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:54:40.538 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027294+0000", 2024-09-21T13:54:40.538 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:40.538 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.538 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.538 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:54:40.538 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027295+0000", 2024-09-21T13:54:40.538 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7899999999999997e-07 2024-09-21T13:54:40.538 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.538 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.538 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:54:40.538 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027299+0000", 2024-09-21T13:54:40.539 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6449999999999998e-06 2024-09-21T13:54:40.539 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.539 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.539 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:54:40.539 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027310+0000", 2024-09-21T13:54:40.539 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0699999999999999e-05 2024-09-21T13:54:40.539 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.539 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.539 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:54:40.539 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027315+0000", 2024-09-21T13:54:40.539 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9729999999999997e-06 2024-09-21T13:54:40.539 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.539 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.539 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:40.539 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027363+0000", 2024-09-21T13:54:40.540 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8096000000000002e-05 2024-09-21T13:54:40.540 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.540 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.540 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:54:40.540 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027367+0000", 2024-09-21T13:54:40.540 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2930000000000002e-06 2024-09-21T13:54:40.540 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.540 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.540 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:40.540 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.992098+0000", 2024-09-21T13:54:40.540 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.96473078800000001 2024-09-21T13:54:40.540 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.540 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.540 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:54:40.540 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.992160+0000", 2024-09-21T13:54:40.541 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2133999999999998e-05 2024-09-21T13:54:40.541 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.541 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.541 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-09-21T13:54:40.541 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.992220+0000", 2024-09-21T13:54:40.541 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9911999999999998e-05 2024-09-21T13:54:40.541 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.541 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.541 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:40.541 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.003058+0000", 2024-09-21T13:54:40.541 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.010838441000000001 2024-09-21T13:54:40.541 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.541 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.541 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:40.541 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.003121+0000", 2024-09-21T13:54:40.542 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3118000000000001e-05 2024-09-21T13:54:40.542 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.542 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.542 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:54:40.542 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.005887+0000", 2024-09-21T13:54:40.542 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0027659500000000001 2024-09-21T13:54:40.542 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.542 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.542 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:54:40.542 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.005958+0000", 2024-09-21T13:54:40.542 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0842999999999999e-05 2024-09-21T13:54:40.542 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.542 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.542 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:54:40.542 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.005997+0000", 2024-09-21T13:54:40.542 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9033999999999999e-05 2024-09-21T13:54:40.543 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:40.543 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:54:40.543 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:40.543 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.543 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.543 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:201503 7.8 7:1bb4326b:::benchmark_data_smithi049_76798_object65083:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-21T13:54:40.543 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:42.027327+0000", 2024-09-21T13:54:40.543 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 118.216618881, 2024-09-21T13:54:40.543 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97877131900000003, 2024-09-21T13:54:40.543 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:54:40.543 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:54:40.543 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:40.543 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:54:40.543 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:54:40.543 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:40.544 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 201503 2024-09-21T13:54:40.544 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.544 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:54:40.544 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.544 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:54:40.544 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027327+0000", 2024-09-21T13:54:40.544 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:40.544 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.544 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.544 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:54:40.544 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027327+0000", 2024-09-21T13:54:40.544 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:40.544 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.544 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.544 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:54:40.545 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027327+0000", 2024-09-21T13:54:40.545 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4799999999999999e-07 2024-09-21T13:54:40.545 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.545 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.545 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:54:40.545 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027329+0000", 2024-09-21T13:54:40.545 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7400000000000001e-06 2024-09-21T13:54:40.545 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.545 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.545 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:54:40.545 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027333+0000", 2024-09-21T13:54:40.545 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0790000000000002e-06 2024-09-21T13:54:40.545 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.545 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.545 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:54:40.545 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027336+0000", 2024-09-21T13:54:40.546 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1190000000000001e-06 2024-09-21T13:54:40.546 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.546 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.546 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:40.546 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027379+0000", 2024-09-21T13:54:40.546 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3000000000000002e-05 2024-09-21T13:54:40.546 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.546 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.546 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:54:40.546 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027382+0000", 2024-09-21T13:54:40.546 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9399999999999998e-06 2024-09-21T13:54:40.546 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.546 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.546 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:40.546 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.994085+0000", 2024-09-21T13:54:40.547 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.96670302799999996 2024-09-21T13:54:40.547 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.547 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.547 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:54:40.547 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.994181+0000", 2024-09-21T13:54:40.547 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6340000000000003e-05 2024-09-21T13:54:40.547 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.547 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.547 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-09-21T13:54:40.547 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.994285+0000", 2024-09-21T13:54:40.547 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000103947 2024-09-21T13:54:40.547 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.547 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.547 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:40.547 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.003091+0000", 2024-09-21T13:54:40.548 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0088057599999999993 2024-09-21T13:54:40.548 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.548 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.548 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:40.548 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.003150+0000", 2024-09-21T13:54:40.548 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9670000000000003e-05 2024-09-21T13:54:40.548 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.548 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.548 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:54:40.548 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.006006+0000", 2024-09-21T13:54:40.548 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002855734 2024-09-21T13:54:40.548 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.548 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.549 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:54:40.549 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.006056+0000", 2024-09-21T13:54:40.549 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9619e-05 2024-09-21T13:54:40.549 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.549 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.549 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:54:40.549 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.006098+0000", 2024-09-21T13:54:40.549 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1995e-05 2024-09-21T13:54:40.549 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:40.549 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:54:40.549 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:40.549 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.549 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.549 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:201514 7.8 7:1c8e529c:::benchmark_data_smithi049_76798_object65094:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-21T13:54:40.550 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:42.044200+0000", 2024-09-21T13:54:40.550 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 118.19974577399999, 2024-09-21T13:54:40.550 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.96199924699999995, 2024-09-21T13:54:40.550 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:54:40.550 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:54:40.550 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:40.550 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:54:40.550 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:54:40.550 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:40.550 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 201514 2024-09-21T13:54:40.550 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.550 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:54:40.551 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.551 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:54:40.551 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.044200+0000", 2024-09-21T13:54:40.551 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:40.551 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.551 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.551 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:54:40.551 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.044200+0000", 2024-09-21T13:54:40.551 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:40.551 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.551 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.551 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:54:40.551 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.044201+0000", 2024-09-21T13:54:40.551 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7179999999999999e-06 2024-09-21T13:54:40.551 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.552 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.552 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:54:40.552 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.044208+0000", 2024-09-21T13:54:40.552 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1800000000000001e-06 2024-09-21T13:54:40.552 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.552 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.552 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:54:40.552 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.044226+0000", 2024-09-21T13:54:40.552 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8216999999999999e-05 2024-09-21T13:54:40.552 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.552 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.552 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:54:40.552 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.044235+0000", 2024-09-21T13:54:40.552 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1020000000000003e-06 2024-09-21T13:54:40.552 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.553 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.553 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:40.553 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.044482+0000", 2024-09-21T13:54:40.553 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00024732699999999999 2024-09-21T13:54:40.553 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.553 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.553 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:54:40.553 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.044499+0000", 2024-09-21T13:54:40.553 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7129000000000002e-05 2024-09-21T13:54:40.553 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.553 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.553 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:40.553 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.996416+0000", 2024-09-21T13:54:40.553 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.95191657200000002 2024-09-21T13:54:40.553 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.553 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.553 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:54:40.554 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.996528+0000", 2024-09-21T13:54:40.554 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000112118 2024-09-21T13:54:40.554 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.554 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.554 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-09-21T13:54:40.554 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.996652+0000", 2024-09-21T13:54:40.554 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012430500000000001 2024-09-21T13:54:40.554 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.554 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.554 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:40.554 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.003180+0000", 2024-09-21T13:54:40.554 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.006528056 2024-09-21T13:54:40.554 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.554 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.554 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:40.555 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.005760+0000", 2024-09-21T13:54:40.555 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0025797670000000002 2024-09-21T13:54:40.555 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.555 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.555 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:54:40.555 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.006108+0000", 2024-09-21T13:54:40.555 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00034834500000000001 2024-09-21T13:54:40.555 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.555 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.555 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:54:40.555 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.006155+0000", 2024-09-21T13:54:40.555 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6513e-05 2024-09-21T13:54:40.555 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.555 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.555 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:54:40.556 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.006199+0000", 2024-09-21T13:54:40.556 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3897999999999999e-05 2024-09-21T13:54:40.556 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:40.556 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:54:40.556 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:40.556 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.556 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.556 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:201539 7.8 7:1afa34ae:::benchmark_data_smithi049_76798_object65119:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-21T13:54:40.556 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:42.081266+0000", 2024-09-21T13:54:40.556 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 118.162679666, 2024-09-21T13:54:40.556 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.92917915600000001, 2024-09-21T13:54:40.556 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:54:40.556 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:54:40.556 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:40.556 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:54:40.557 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:54:40.557 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:40.557 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 201539 2024-09-21T13:54:40.557 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.557 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:54:40.557 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.557 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:54:40.557 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.081266+0000", 2024-09-21T13:54:40.557 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:40.557 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.557 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.557 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:54:40.557 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.081266+0000", 2024-09-21T13:54:40.557 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:40.557 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.558 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.558 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:54:40.558 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.081267+0000", 2024-09-21T13:54:40.558 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6559999999999999e-06 2024-09-21T13:54:40.558 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.558 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.558 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:54:40.558 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.081273+0000", 2024-09-21T13:54:40.558 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0839999999999997e-06 2024-09-21T13:54:40.558 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.558 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.558 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:54:40.558 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.081291+0000", 2024-09-21T13:54:40.558 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7827999999999999e-05 2024-09-21T13:54:40.559 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.559 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.559 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:54:40.559 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.081300+0000", 2024-09-21T13:54:40.559 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.7190000000000007e-06 2024-09-21T13:54:40.559 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.559 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.559 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:40.559 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.081360+0000", 2024-09-21T13:54:40.559 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9914000000000003e-05 2024-09-21T13:54:40.559 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.559 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.560 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:54:40.560 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.081369+0000", 2024-09-21T13:54:40.560 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1630000000000002e-06 2024-09-21T13:54:40.560 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.560 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.560 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:40.560 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.998700+0000", 2024-09-21T13:54:40.560 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.91733049300000002 2024-09-21T13:54:40.560 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.560 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.560 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:54:40.560 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.998798+0000", 2024-09-21T13:54:40.560 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8494999999999998e-05 2024-09-21T13:54:40.561 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.561 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.561 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-09-21T13:54:40.561 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.998903+0000", 2024-09-21T13:54:40.561 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000104782 2024-09-21T13:54:40.561 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.561 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.561 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:54:40.561 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.006208+0000", 2024-09-21T13:54:40.561 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0073052270000000001 2024-09-21T13:54:40.561 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.561 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.561 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:40.561 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.006259+0000", 2024-09-21T13:54:40.562 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1205999999999997e-05 2024-09-21T13:54:40.562 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.562 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.562 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:40.562 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.010350+0000", 2024-09-21T13:54:40.562 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0040908560000000004 2024-09-21T13:54:40.562 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.562 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.562 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:54:40.562 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.010400+0000", 2024-09-21T13:54:40.562 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9901000000000002e-05 2024-09-21T13:54:40.562 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.562 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.563 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:54:40.563 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.010445+0000", 2024-09-21T13:54:40.563 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4832e-05 2024-09-21T13:54:40.563 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:40.563 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:54:40.563 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:40.563 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.563 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.563 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:201541 7.8 7:135037ad:::benchmark_data_smithi049_76798_object65121:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-21T13:54:40.563 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:42.083222+0000", 2024-09-21T13:54:40.563 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 118.16072341500001, 2024-09-21T13:54:40.563 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.92733551700000005, 2024-09-21T13:54:40.564 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:54:40.564 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:54:40.564 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:40.564 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:54:40.564 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:54:40.564 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:40.564 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 201541 2024-09-21T13:54:40.564 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.564 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:54:40.564 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.564 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:54:40.564 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.083222+0000", 2024-09-21T13:54:40.564 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:40.564 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.565 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.565 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:54:40.565 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.083222+0000", 2024-09-21T13:54:40.565 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:40.565 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.565 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.565 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:54:40.565 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.083223+0000", 2024-09-21T13:54:40.565 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0020000000000001e-06 2024-09-21T13:54:40.565 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.565 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.565 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:54:40.565 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.083227+0000", 2024-09-21T13:54:40.565 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.636e-06 2024-09-21T13:54:40.565 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.566 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.566 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:54:40.566 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.083236+0000", 2024-09-21T13:54:40.566 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0180000000000007e-06 2024-09-21T13:54:40.566 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.566 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.566 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:54:40.566 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.083242+0000", 2024-09-21T13:54:40.566 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.891e-06 2024-09-21T13:54:40.566 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.566 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.566 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:40.566 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.083385+0000", 2024-09-21T13:54:40.566 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014374099999999999 2024-09-21T13:54:40.566 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.567 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.567 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:54:40.567 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.083394+0000", 2024-09-21T13:54:40.567 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0839999999999998e-06 2024-09-21T13:54:40.567 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.567 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.567 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:40.567 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.000839+0000", 2024-09-21T13:54:40.567 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.917444861 2024-09-21T13:54:40.567 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.567 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.567 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:54:40.567 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.000934+0000", 2024-09-21T13:54:40.567 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4943000000000002e-05 2024-09-21T13:54:40.567 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.567 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.568 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-09-21T13:54:40.568 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.001036+0000", 2024-09-21T13:54:40.568 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000101817 2024-09-21T13:54:40.568 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.568 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.568 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:54:40.568 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.006215+0000", 2024-09-21T13:54:40.568 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0051790020000000003 2024-09-21T13:54:40.568 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.568 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.568 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:40.568 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.006324+0000", 2024-09-21T13:54:40.568 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010858599999999999 2024-09-21T13:54:40.568 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.568 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.569 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:40.569 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.010481+0000", 2024-09-21T13:54:40.569 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0041575300000000004 2024-09-21T13:54:40.569 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.569 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.569 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:54:40.569 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.010528+0000", 2024-09-21T13:54:40.569 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6418000000000002e-05 2024-09-21T13:54:40.569 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.569 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.569 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:54:40.569 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.010558+0000", 2024-09-21T13:54:40.569 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9988000000000002e-05 2024-09-21T13:54:40.569 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:40.569 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:54:40.569 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:40.569 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.570 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.570 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:229964 7.e 7:7be4d3ef:::benchmark_data_smithi049_76798_object93544:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-21T13:54:40.570 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:53:16.693433+0000", 2024-09-21T13:54:40.570 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 83.550512850000004, 2024-09-21T13:54:40.570 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.99823727600000001, 2024-09-21T13:54:40.570 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:54:40.570 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:54:40.570 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:40.570 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:54:40.570 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:54:40.570 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:40.570 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 229964 2024-09-21T13:54:40.570 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.570 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:54:40.571 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.571 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:54:40.571 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.693433+0000", 2024-09-21T13:54:40.571 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:40.571 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.571 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.571 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:54:40.571 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.693433+0000", 2024-09-21T13:54:40.571 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:40.571 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.571 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.571 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:54:40.571 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.693434+0000", 2024-09-21T13:54:40.571 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3799999999999999e-06 2024-09-21T13:54:40.571 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.572 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.572 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:54:40.572 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.693438+0000", 2024-09-21T13:54:40.572 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.515e-06 2024-09-21T13:54:40.572 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.572 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.572 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:54:40.572 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.693455+0000", 2024-09-21T13:54:40.572 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6614000000000001e-05 2024-09-21T13:54:40.572 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.572 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.572 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:54:40.572 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.693463+0000", 2024-09-21T13:54:40.572 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9850000000000006e-06 2024-09-21T13:54:40.572 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.573 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.573 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:40.573 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.693542+0000", 2024-09-21T13:54:40.573 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8942000000000004e-05 2024-09-21T13:54:40.573 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.573 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.573 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:54:40.573 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.693550+0000", 2024-09-21T13:54:40.573 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7379999999999994e-06 2024-09-21T13:54:40.573 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.573 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.573 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:40.573 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.686321+0000", 2024-09-21T13:54:40.574 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.99277167399999999 2024-09-21T13:54:40.574 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.574 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.574 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:54:40.574 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.686413+0000", 2024-09-21T13:54:40.574 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1822000000000003e-05 2024-09-21T13:54:40.574 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.574 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.574 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,5", 2024-09-21T13:54:40.574 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.686547+0000", 2024-09-21T13:54:40.574 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000133516 2024-09-21T13:54:40.574 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.574 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.574 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:40.575 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.691512+0000", 2024-09-21T13:54:40.575 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0049651410000000002 2024-09-21T13:54:40.575 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.575 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.575 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:40.575 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.691558+0000", 2024-09-21T13:54:40.575 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6047999999999997e-05 2024-09-21T13:54:40.575 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.575 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.575 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:54:40.575 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.691610+0000", 2024-09-21T13:54:40.575 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1644000000000002e-05 2024-09-21T13:54:40.575 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.575 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.575 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:54:40.576 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.691642+0000", 2024-09-21T13:54:40.576 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2542000000000001e-05 2024-09-21T13:54:40.576 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.576 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.576 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:54:40.576 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.691670+0000", 2024-09-21T13:54:40.576 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.7715000000000001e-05 2024-09-21T13:54:40.576 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:40.576 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:54:40.576 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:40.576 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.576 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.576 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:230040 7.e 7:74df0b23:::benchmark_data_smithi049_76798_object93620:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-21T13:54:40.576 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:53:16.758488+0000", 2024-09-21T13:54:40.576 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 83.485457792999995, 2024-09-21T13:54:40.576 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.93513229499999995, 2024-09-21T13:54:40.577 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:54:40.577 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:54:40.577 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:40.577 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:54:40.577 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:54:40.577 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:40.577 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 230040 2024-09-21T13:54:40.577 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.577 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:54:40.577 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.577 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:54:40.577 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.758488+0000", 2024-09-21T13:54:40.577 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:40.577 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.577 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.578 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:54:40.578 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.758488+0000", 2024-09-21T13:54:40.578 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:40.578 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.578 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.578 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:54:40.578 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.758502+0000", 2024-09-21T13:54:40.578 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4207e-05 2024-09-21T13:54:40.578 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.578 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.578 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:54:40.578 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.758507+0000", 2024-09-21T13:54:40.578 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9429999999999999e-06 2024-09-21T13:54:40.579 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.579 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.579 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:54:40.579 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.758518+0000", 2024-09-21T13:54:40.579 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1447999999999999e-05 2024-09-21T13:54:40.579 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.579 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.579 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:54:40.579 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.758525+0000", 2024-09-21T13:54:40.579 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2389999999999996e-06 2024-09-21T13:54:40.579 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.579 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.579 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:40.579 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.758589+0000", 2024-09-21T13:54:40.579 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3262000000000005e-05 2024-09-21T13:54:40.580 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.580 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.580 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:54:40.580 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.758597+0000", 2024-09-21T13:54:40.580 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4100000000000008e-06 2024-09-21T13:54:40.580 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.580 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.580 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:40.580 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.689523+0000", 2024-09-21T13:54:40.580 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.93092584099999998 2024-09-21T13:54:40.580 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.580 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.580 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:54:40.581 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.689620+0000", 2024-09-21T13:54:40.581 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7081000000000001e-05 2024-09-21T13:54:40.581 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.581 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.581 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,5", 2024-09-21T13:54:40.581 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.689705+0000", 2024-09-21T13:54:40.581 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5074000000000006e-05 2024-09-21T13:54:40.581 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.581 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.581 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:40.581 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.691602+0000", 2024-09-21T13:54:40.581 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001896951 2024-09-21T13:54:40.581 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.581 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.581 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:40.582 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.693564+0000", 2024-09-21T13:54:40.582 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0019616630000000002 2024-09-21T13:54:40.582 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.582 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.582 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:54:40.582 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.693572+0000", 2024-09-21T13:54:40.582 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9160000000000001e-06 2024-09-21T13:54:40.582 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.582 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.582 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:54:40.582 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.693597+0000", 2024-09-21T13:54:40.582 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5812999999999998e-05 2024-09-21T13:54:40.582 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.582 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.583 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:54:40.583 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.693620+0000", 2024-09-21T13:54:40.583 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2447e-05 2024-09-21T13:54:40.583 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:40.583 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:54:40.583 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:40.583 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.583 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.583 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:230048 7.e 7:7e0d246c:::benchmark_data_smithi049_76798_object93628:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-21T13:54:40.583 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:53:16.763844+0000", 2024-09-21T13:54:40.583 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 83.480100914000005, 2024-09-21T13:54:40.583 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.93491194899999996, 2024-09-21T13:54:40.583 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:54:40.584 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:54:40.584 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:40.584 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:54:40.584 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:54:40.584 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:40.584 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 230048 2024-09-21T13:54:40.584 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.584 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:54:40.584 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.584 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:54:40.584 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.763844+0000", 2024-09-21T13:54:40.584 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:40.584 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.584 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.584 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:54:40.585 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.763844+0000", 2024-09-21T13:54:40.585 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:40.585 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.585 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.585 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:54:40.585 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.763846+0000", 2024-09-21T13:54:40.585 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6330000000000001e-06 2024-09-21T13:54:40.585 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.585 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.585 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:54:40.585 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.763852+0000", 2024-09-21T13:54:40.585 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3839999999999996e-06 2024-09-21T13:54:40.585 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.585 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.586 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:54:40.586 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.763915+0000", 2024-09-21T13:54:40.586 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3306999999999995e-05 2024-09-21T13:54:40.586 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.586 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.586 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:54:40.586 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.764047+0000", 2024-09-21T13:54:40.586 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013207599999999999 2024-09-21T13:54:40.586 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.586 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.586 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:40.586 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.764104+0000", 2024-09-21T13:54:40.587 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6758000000000001e-05 2024-09-21T13:54:40.587 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.587 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.587 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:54:40.587 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.764113+0000", 2024-09-21T13:54:40.587 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3700000000000001e-06 2024-09-21T13:54:40.587 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.587 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.587 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:40.587 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.691702+0000", 2024-09-21T13:54:40.587 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.92758885899999999 2024-09-21T13:54:40.587 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.587 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.588 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:54:40.588 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.691822+0000", 2024-09-21T13:54:40.588 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011997699999999999 2024-09-21T13:54:40.588 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.588 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.588 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,5", 2024-09-21T13:54:40.588 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.691939+0000", 2024-09-21T13:54:40.588 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011752 2024-09-21T13:54:40.588 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.588 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.588 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:54:40.588 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.698556+0000", 2024-09-21T13:54:40.588 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0066170739999999997 2024-09-21T13:54:40.588 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.589 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.589 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:40.589 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.698612+0000", 2024-09-21T13:54:40.589 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5362000000000003e-05 2024-09-21T13:54:40.589 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.589 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.589 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:40.589 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.698659+0000", 2024-09-21T13:54:40.589 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6931999999999997e-05 2024-09-21T13:54:40.589 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.589 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.589 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:54:40.589 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.698714+0000", 2024-09-21T13:54:40.590 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5402000000000003e-05 2024-09-21T13:54:40.590 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.590 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.590 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:54:40.590 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.698756+0000", 2024-09-21T13:54:40.590 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2295000000000001e-05 2024-09-21T13:54:40.590 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:40.590 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:54:40.590 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:40.590 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.590 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.590 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:230051 7.e 7:71fa5c2a:::benchmark_data_smithi049_76798_object93631:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-21T13:54:40.590 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:53:16.764458+0000", 2024-09-21T13:54:40.590 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 83.479487469000006, 2024-09-21T13:54:40.591 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.94179994899999997, 2024-09-21T13:54:40.591 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:54:40.591 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:54:40.591 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:40.591 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:54:40.591 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:54:40.591 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:40.591 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 230051 2024-09-21T13:54:40.591 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.591 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:54:40.591 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.591 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:54:40.592 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.764458+0000", 2024-09-21T13:54:40.592 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:40.592 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.592 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.592 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:54:40.592 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.764458+0000", 2024-09-21T13:54:40.592 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:40.592 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.592 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.592 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:54:40.592 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.764459+0000", 2024-09-21T13:54:40.592 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9400000000000004e-07 2024-09-21T13:54:40.592 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.592 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.593 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:54:40.593 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.764463+0000", 2024-09-21T13:54:40.593 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1960000000000001e-06 2024-09-21T13:54:40.593 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.593 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.593 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:54:40.593 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.764477+0000", 2024-09-21T13:54:40.593 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4426e-05 2024-09-21T13:54:40.593 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.593 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.593 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:54:40.593 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.764485+0000", 2024-09-21T13:54:40.593 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.148e-06 2024-09-21T13:54:40.593 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.593 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.593 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:40.594 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.764545+0000", 2024-09-21T13:54:40.594 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0278999999999997e-05 2024-09-21T13:54:40.594 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.594 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.594 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:54:40.594 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.764553+0000", 2024-09-21T13:54:40.594 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9170000000000006e-06 2024-09-21T13:54:40.594 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.594 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.594 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:40.594 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.693660+0000", 2024-09-21T13:54:40.595 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.92910747900000001 2024-09-21T13:54:40.595 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.595 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.595 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:54:40.595 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.693785+0000", 2024-09-21T13:54:40.595 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012436000000000001 2024-09-21T13:54:40.595 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.595 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.595 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,5", 2024-09-21T13:54:40.595 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.693907+0000", 2024-09-21T13:54:40.595 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012208600000000001 2024-09-21T13:54:40.595 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.595 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.595 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:40.595 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.698816+0000", 2024-09-21T13:54:40.595 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0049091279999999996 2024-09-21T13:54:40.596 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.596 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.596 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:54:40.596 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.698834+0000", 2024-09-21T13:54:40.596 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8236999999999998e-05 2024-09-21T13:54:40.596 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.596 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.596 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:40.596 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.706148+0000", 2024-09-21T13:54:40.596 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0073139709999999998 2024-09-21T13:54:40.596 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.596 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.596 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:54:40.596 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.706202+0000", 2024-09-21T13:54:40.596 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3779000000000001e-05 2024-09-21T13:54:40.597 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.597 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.597 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:54:40.597 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.706258+0000", 2024-09-21T13:54:40.597 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6048999999999998e-05 2024-09-21T13:54:40.597 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:40.597 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:54:40.597 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:40.597 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.597 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.597 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:230060 7.e 7:7289055b:::benchmark_data_smithi049_76798_object93640:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-21T13:54:40.597 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:53:16.772333+0000", 2024-09-21T13:54:40.597 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 83.471612348999997, 2024-09-21T13:54:40.597 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.93409186700000002, 2024-09-21T13:54:40.598 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:54:40.598 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:54:40.598 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:40.598 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:54:40.598 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:54:40.598 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:54:40.598 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 230060 2024-09-21T13:54:40.598 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.598 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:54:40.598 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.598 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:54:40.598 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.772333+0000", 2024-09-21T13:54:40.599 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:40.599 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.599 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.599 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:54:40.599 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.772333+0000", 2024-09-21T13:54:40.599 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:40.599 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.599 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.599 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:54:40.599 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.772334+0000", 2024-09-21T13:54:40.599 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1540000000000001e-06 2024-09-21T13:54:40.599 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.599 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.599 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:54:40.599 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.772339+0000", 2024-09-21T13:54:40.600 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8099999999999997e-06 2024-09-21T13:54:40.600 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.600 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.600 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:54:40.600 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.772351+0000", 2024-09-21T13:54:40.600 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2435999999999999e-05 2024-09-21T13:54:40.600 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.600 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.600 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:54:40.600 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.772359+0000", 2024-09-21T13:54:40.600 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7230000000000004e-06 2024-09-21T13:54:40.600 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.600 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.600 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:40.601 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.772435+0000", 2024-09-21T13:54:40.601 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5465999999999999e-05 2024-09-21T13:54:40.601 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.601 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.601 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:54:40.601 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.772443+0000", 2024-09-21T13:54:40.601 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9160000000000001e-06 2024-09-21T13:54:40.601 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.601 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.601 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:40.601 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.695975+0000", 2024-09-21T13:54:40.601 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.92353224599999995 2024-09-21T13:54:40.601 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.601 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.602 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:54:40.602 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.696104+0000", 2024-09-21T13:54:40.602 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012895499999999999 2024-09-21T13:54:40.602 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.602 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.602 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,5", 2024-09-21T13:54:40.602 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.696217+0000", 2024-09-21T13:54:40.602 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011362399999999999 2024-09-21T13:54:40.602 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.602 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.602 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:54:40.602 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.701076+0000", 2024-09-21T13:54:40.602 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0048582729999999998 2024-09-21T13:54:40.602 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.603 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.603 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:40.603 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.706310+0000", 2024-09-21T13:54:40.603 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0052347419999999997 2024-09-21T13:54:40.603 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.603 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.603 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:40.603 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.706347+0000", 2024-09-21T13:54:40.603 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6492999999999998e-05 2024-09-21T13:54:40.603 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.603 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.603 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:54:40.603 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.706392+0000", 2024-09-21T13:54:40.603 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5148999999999998e-05 2024-09-21T13:54:40.603 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:40.604 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:40.604 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:54:40.604 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.706425+0000", 2024-09-21T13:54:40.604 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2879999999999997e-05 2024-09-21T13:54:40.604 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:40.604 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:54:40.604 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:40.604 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:40.604 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:54:40.604 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T13:54:40.604 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:54:40.640 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:40.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:40.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:54:40.742 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:54:40.784 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:40.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:40.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:54:40.886 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:40.925 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:40.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:40.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:40.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:40 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:54:40.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:40 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1534027970' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-21T13:54:40.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:40 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-21T13:54:40.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:40 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-09-21T13:54:40.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:40 smithi187 ceph-mon[33032]: osdmap e194: 8 total, 8 up, 8 in 2024-09-21T13:54:41.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:40 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:54:41.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:40 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1534027970' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-21T13:54:41.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:40 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-21T13:54:41.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:40 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-09-21T13:54:41.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:40 smithi049 ceph-mon[40489]: osdmap e194: 8 total, 8 up, 8 in 2024-09-21T13:54:41.005 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:54:40 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:54:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:54:41.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:40 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:54:41.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:40 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1534027970' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-21T13:54:41.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:40 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-21T13:54:41.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:40 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-09-21T13:54:41.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:40 smithi049 ceph-mon[30080]: osdmap e194: 8 total, 8 up, 8 in 2024-09-21T13:54:41.027 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:54:41.061 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:41.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:41.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:54:41.163 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:54:41.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:41.196 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:41.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:54:41.289 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 23 16 48692 48676 132.245 109.688 0.0117166 0.0075556 2024-09-21T13:54:41.297 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:41.330 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:41.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:41.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:41.432 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:54:41.464 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:41.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:41.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:54:41.566 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:54:41.600 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:41.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:41.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:54:41.702 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:41.732 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:41.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:41.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:41.834 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:41.872 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:41.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:41.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:41.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:41 smithi187 ceph-mon[33032]: pgmap v278: 148 pgs: 148 active+clean; 2.7 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 110 MiB/s wr, 1.75k op/s 2024-09-21T13:54:41.974 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:54:42.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:41 smithi049 ceph-mon[40489]: pgmap v278: 148 pgs: 148 active+clean; 2.7 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 110 MiB/s wr, 1.75k op/s 2024-09-21T13:54:42.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:41 smithi049 ceph-mon[30080]: pgmap v278: 148 pgs: 148 active+clean; 2.7 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 110 MiB/s wr, 1.75k op/s 2024-09-21T13:54:42.005 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:42.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:42.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:54:42.107 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:42.138 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:42.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:42.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:42.239 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:42.248 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T13:54:42.279 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:42.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:42.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:42.289 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 24 15 50431 50416 131.265 108.75 0.0206355 0.00761245 2024-09-21T13:54:42.381 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:54:42.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:42.430 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:42.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:54:42.530 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:54:42.556 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:42.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:42.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:54:42.658 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:54:42.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:42.707 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:42.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:54:42.808 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:54:42.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:42.838 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:42.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:54:42.938 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:54:42.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:42.984 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:42.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:54:43.085 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:43.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:43.115 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:43.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:43.216 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:54:43.251 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:43.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:43.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:54:43.289 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 25 16 52123 52107 130.242 105.688 0.0084661 0.0076728 2024-09-21T13:54:43.353 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:54:43.391 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:43.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:43.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:54:43.475 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:54:43.480 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:54:43.493 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:54:43.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:43.538 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:43.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:54:43.638 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:43.668 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:43.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:43.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:43.670 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:43 smithi049 ceph-mon[40489]: pgmap v280: 148 pgs: 148 active+clean; 2.7 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 82 MiB/s wr, 1.30k op/s 2024-09-21T13:54:43.670 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:43 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:54:43.670 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:43 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:54:43.671 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:43 smithi049 ceph-mon[30080]: pgmap v280: 148 pgs: 148 active+clean; 2.7 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 82 MiB/s wr, 1.30k op/s 2024-09-21T13:54:43.671 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:43 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:54:43.671 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:43 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:54:43.770 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:54:43.817 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:43.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:43.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:54:43.918 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:43.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:43 smithi187 ceph-mon[33032]: pgmap v280: 148 pgs: 148 active+clean; 2.7 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 82 MiB/s wr, 1.30k op/s 2024-09-21T13:54:43.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:43 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:54:43.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:43 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:54:43.956 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:43.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:43.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:44.057 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:44.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:44.103 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:44.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:44.204 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:44.245 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:44.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:44.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:44.290 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 26 16 53721 53705 129.074 99.875 0.0120327 0.00774251 2024-09-21T13:54:44.347 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:44.386 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:44.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:44.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:44.475 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T13:54:44.475 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-09-21T13:54:44.476 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.476 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:53936 8.c 8:3a550406:::benchmark_data_smithi049_109311_object53935:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-21T13:54:44.476 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:54:44.459844+0000", 2024-09-21T13:54:44.476 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.014766415999999999, 2024-09-21T13:54:44.476 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.014798736, 2024-09-21T13:54:44.476 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:54:44.476 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:54:44.476 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "waiting for sub ops", 2024-09-21T13:54:44.476 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:54:44.476 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:54:44.476 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:54:44.476 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 53936 2024-09-21T13:54:44.476 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:44.476 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:54:44.477 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.477 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:54:44.477 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:44.459844+0000", 2024-09-21T13:54:44.477 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:44.477 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:44.477 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.477 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:54:44.477 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:44.459844+0000", 2024-09-21T13:54:44.477 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:44.477 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:44.477 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.477 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:54:44.477 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:44.459845+0000", 2024-09-21T13:54:44.477 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0440000000000001e-06 2024-09-21T13:54:44.478 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:44.478 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.478 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:54:44.478 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:44.459901+0000", 2024-09-21T13:54:44.478 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6400000000000002e-05 2024-09-21T13:54:44.478 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:44.478 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.478 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:54:44.478 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:44.459915+0000", 2024-09-21T13:54:44.478 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3621999999999999e-05 2024-09-21T13:54:44.478 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:44.478 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.478 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:54:44.478 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:44.459923+0000", 2024-09-21T13:54:44.479 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3790000000000006e-06 2024-09-21T13:54:44.479 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:44.479 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.479 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:44.479 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:44.462402+0000", 2024-09-21T13:54:44.479 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002478404 2024-09-21T13:54:44.479 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:44.479 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.479 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:54:44.479 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:44.462544+0000", 2024-09-21T13:54:44.479 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014185000000000001 2024-09-21T13:54:44.479 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:44.479 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.479 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,7", 2024-09-21T13:54:44.479 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:44.462811+0000", 2024-09-21T13:54:44.480 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00026732900000000001 2024-09-21T13:54:44.480 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:44.480 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.480 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:44.480 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:44.469902+0000", 2024-09-21T13:54:44.480 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0070907510000000002 2024-09-21T13:54:44.480 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:44.480 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.480 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:54:44.480 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:44.469916+0000", 2024-09-21T13:54:44.480 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4158999999999999e-05 2024-09-21T13:54:44.480 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:44.480 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:54:44.480 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:44.480 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:44.481 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.481 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:53939 8.c 8:3765f5b0:::benchmark_data_smithi049_109311_object53938:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-21T13:54:44.481 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:54:44.462396+0000", 2024-09-21T13:54:44.481 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.012213928000000001, 2024-09-21T13:54:44.481 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.01233157, 2024-09-21T13:54:44.481 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:54:44.481 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:54:44.481 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "waiting for sub ops", 2024-09-21T13:54:44.481 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:54:44.481 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:54:44.481 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:54:44.481 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 53939 2024-09-21T13:54:44.481 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:44.481 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:54:44.481 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.482 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:54:44.482 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:44.462396+0000", 2024-09-21T13:54:44.482 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:44.482 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:44.482 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.482 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:54:44.482 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:44.462396+0000", 2024-09-21T13:54:44.482 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:44.482 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:44.482 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.482 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:54:44.482 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:44.462398+0000", 2024-09-21T13:54:44.482 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.643e-06 2024-09-21T13:54:44.482 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:44.483 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.483 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:54:44.483 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:44.462421+0000", 2024-09-21T13:54:44.483 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2702e-05 2024-09-21T13:54:44.483 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:44.483 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.483 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:54:44.483 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:44.462441+0000", 2024-09-21T13:54:44.483 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0075999999999999e-05 2024-09-21T13:54:44.483 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:44.483 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.483 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:54:44.483 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:44.462448+0000", 2024-09-21T13:54:44.483 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4080000000000001e-06 2024-09-21T13:54:44.483 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:44.484 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.484 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:44.484 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:44.466612+0000", 2024-09-21T13:54:44.484 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0041638739999999997 2024-09-21T13:54:44.484 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:44.484 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.484 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:54:44.484 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:44.467049+0000", 2024-09-21T13:54:44.484 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00043644899999999999 2024-09-21T13:54:44.484 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:44.484 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.484 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,7", 2024-09-21T13:54:44.484 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:44.467212+0000", 2024-09-21T13:54:44.484 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000163466 2024-09-21T13:54:44.484 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:44.485 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.485 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:54:44.485 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:44.471459+0000", 2024-09-21T13:54:44.485 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0042472869999999998 2024-09-21T13:54:44.485 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:44.485 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.485 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:44.485 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:44.474132+0000", 2024-09-21T13:54:44.485 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0026722870000000002 2024-09-21T13:54:44.485 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:44.485 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:54:44.485 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:44.485 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:44.485 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.485 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:53948 8.e 8:729d7899:::benchmark_data_smithi049_109311_object53947:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-09-21T13:54:44.486 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:54:44.468785+0000", 2024-09-21T13:54:44.486 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.0058253920000000004, 2024-09-21T13:54:44.486 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.006022739, 2024-09-21T13:54:44.486 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:54:44.486 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:54:44.486 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "waiting for sub ops", 2024-09-21T13:54:44.486 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:54:44.486 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:54:44.486 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:54:44.486 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 53948 2024-09-21T13:54:44.486 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:44.486 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:54:44.486 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.486 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:54:44.486 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:44.468785+0000", 2024-09-21T13:54:44.487 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:44.487 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:44.487 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.487 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:54:44.487 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:44.468785+0000", 2024-09-21T13:54:44.487 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:44.487 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:44.487 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.487 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:54:44.487 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:44.468787+0000", 2024-09-21T13:54:44.487 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5659999999999999e-06 2024-09-21T13:54:44.487 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:44.487 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.487 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:54:44.487 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:44.468840+0000", 2024-09-21T13:54:44.488 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3625999999999998e-05 2024-09-21T13:54:44.488 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:44.488 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.488 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:54:44.488 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:44.468858+0000", 2024-09-21T13:54:44.488 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8329e-05 2024-09-21T13:54:44.488 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:44.488 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.488 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:54:44.488 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:54:44.489 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:44.468868+0000", 2024-09-21T13:54:44.489 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0440000000000001e-06 2024-09-21T13:54:44.489 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:44.489 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.489 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:44.489 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:44.468992+0000", 2024-09-21T13:54:44.489 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012472300000000001 2024-09-21T13:54:44.489 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:44.489 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.489 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:54:44.490 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:44.469128+0000", 2024-09-21T13:54:44.490 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013625000000000001 2024-09-21T13:54:44.490 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:44.490 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.490 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,4", 2024-09-21T13:54:44.490 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:44.469277+0000", 2024-09-21T13:54:44.490 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014891200000000001 2024-09-21T13:54:44.490 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:44.490 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.490 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:54:44.490 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:44.471471+0000", 2024-09-21T13:54:44.490 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0021940869999999999 2024-09-21T13:54:44.490 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:44.490 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.490 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:44.491 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:44.473809+0000", 2024-09-21T13:54:44.491 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0023378420000000001 2024-09-21T13:54:44.491 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:44.491 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:54:44.491 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:44.491 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:44.491 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.491 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.15207.0:53944 8.9 e194/183 8:966c6f87:::benchmark_data_smithi049_109311_object53943:head v 194'3446, mlcod=194'3444)", 2024-09-21T13:54:44.491 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:54:44.469993+0000", 2024-09-21T13:54:44.491 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.004617337, 2024-09-21T13:54:44.491 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0049228659999999997, 2024-09-21T13:54:44.491 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:54:44.491 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:54:44.491 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T13:54:44.491 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:54:44.492 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.492 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:54:44.492 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:44.469993+0000", 2024-09-21T13:54:44.492 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:44.492 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:44.492 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.492 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:54:44.492 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:44.469993+0000", 2024-09-21T13:54:44.492 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:44.492 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:44.492 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.492 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:54:44.492 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:44.469994+0000", 2024-09-21T13:54:44.492 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.8000000000000004e-07 2024-09-21T13:54:44.492 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:44.493 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.493 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:54:44.493 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:44.470036+0000", 2024-09-21T13:54:44.493 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1965999999999998e-05 2024-09-21T13:54:44.493 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:44.493 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.493 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:54:44.493 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:44.470049+0000", 2024-09-21T13:54:44.493 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3210000000000001e-05 2024-09-21T13:54:44.493 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:44.493 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.493 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:54:44.493 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:44.470055+0000", 2024-09-21T13:54:44.493 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3080000000000004e-06 2024-09-21T13:54:44.493 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:44.494 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.494 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:44.494 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:44.470819+0000", 2024-09-21T13:54:44.494 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000763516 2024-09-21T13:54:44.494 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:44.494 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.494 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:54:44.494 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:44.470837+0000", 2024-09-21T13:54:44.494 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7745999999999999e-05 2024-09-21T13:54:44.494 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:44.494 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:54:44.494 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:44.494 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:44.494 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.494 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.15207.0:53945 8.9 e194/183 8:9f2d9928:::benchmark_data_smithi049_109311_object53944:head v 194'3447, mlcod=194'3444)", 2024-09-21T13:54:44.495 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:54:44.470073+0000", 2024-09-21T13:54:44.495 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.0045374889999999996, 2024-09-21T13:54:44.495 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0049696840000000003, 2024-09-21T13:54:44.495 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:54:44.495 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:54:44.495 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T13:54:44.495 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:54:44.495 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.495 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:54:44.495 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:44.470073+0000", 2024-09-21T13:54:44.495 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:44.495 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:44.495 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.495 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:54:44.495 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:44.470073+0000", 2024-09-21T13:54:44.496 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:44.496 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:44.496 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.496 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:54:44.496 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:44.470073+0000", 2024-09-21T13:54:44.496 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7e-07 2024-09-21T13:54:44.496 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:44.496 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.496 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:54:44.496 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:44.470103+0000", 2024-09-21T13:54:44.496 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9675999999999998e-05 2024-09-21T13:54:44.496 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:44.496 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.496 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:54:44.496 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:44.470109+0000", 2024-09-21T13:54:44.497 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3289999999999999e-06 2024-09-21T13:54:44.497 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:44.497 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.497 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:54:44.497 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:44.470114+0000", 2024-09-21T13:54:44.497 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3250000000000001e-06 2024-09-21T13:54:44.497 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:44.497 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.497 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:44.497 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:44.473369+0000", 2024-09-21T13:54:44.497 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0032552459999999998 2024-09-21T13:54:44.497 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:44.497 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.497 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:54:44.497 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:44.473387+0000", 2024-09-21T13:54:44.497 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8521000000000001e-05 2024-09-21T13:54:44.497 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:44.498 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:54:44.498 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:44.498 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:44.498 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.498 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.15207.0:53949 8.9 e194/183)", 2024-09-21T13:54:44.498 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:54:44.471288+0000", 2024-09-21T13:54:44.498 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.0033226710000000001, 2024-09-21T13:54:44.498 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.003798049, 2024-09-21T13:54:44.498 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:54:44.498 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:54:44.498 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "queued for pg", 2024-09-21T13:54:44.498 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:54:44.498 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.498 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:54:44.499 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:44.471288+0000", 2024-09-21T13:54:44.499 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:44.499 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:44.499 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.499 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:54:44.499 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:44.471288+0000", 2024-09-21T13:54:44.499 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:44.499 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:44.499 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.499 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:54:44.499 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:44.471288+0000", 2024-09-21T13:54:44.499 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4099999999999998e-07 2024-09-21T13:54:44.499 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:44.499 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.499 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:54:44.500 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:44.471322+0000", 2024-09-21T13:54:44.500 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3877000000000001e-05 2024-09-21T13:54:44.500 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:44.500 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.500 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:54:44.500 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:44.471333+0000", 2024-09-21T13:54:44.500 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0849999999999999e-05 2024-09-21T13:54:44.500 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:44.500 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:44.500 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:54:44.500 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:44.471340+0000", 2024-09-21T13:54:44.500 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3790000000000003e-06 2024-09-21T13:54:44.500 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:44.500 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:54:44.500 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:44.500 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:44.501 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T13:54:44.501 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 6 2024-09-21T13:54:44.501 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T13:54:44.530 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:44.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:44.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:54:44.632 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:44.669 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:44.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:44.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:44.771 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:44.808 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:44.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:44.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:44.839 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T13:54:44.910 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:44.967 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:44.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:44.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:45.069 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:54:45.118 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:45.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:45.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:54:45.220 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:54:45.254 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:45.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:45.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:54:45.290 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 27 16 55171 55155 127.65 90.625 0.00482675 0.0078295 2024-09-21T13:54:45.356 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:54:45.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:45.398 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:45.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:54:45.447 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T13:54:45.499 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:54:45.524 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:45.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:45.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:54:45.556 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-09-21T13:54:45.559 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:54:45.625 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:54:45.668 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:45.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:45.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:54:45.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:45 smithi049 ceph-mon[40489]: pgmap v281: 148 pgs: 148 active+clean; 3.1 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 122 MiB/s wr, 1.95k op/s 2024-09-21T13:54:45.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:45 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2263202723' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:54:45.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:45 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:54:45.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:45 smithi049 ceph-mon[30080]: pgmap v281: 148 pgs: 148 active+clean; 3.1 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 122 MiB/s wr, 1.95k op/s 2024-09-21T13:54:45.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:45 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2263202723' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:54:45.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:45 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:54:45.770 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:54:45.823 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:45.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:45.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:54:45.924 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:54:45.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:45 smithi187 ceph-mon[33032]: pgmap v281: 148 pgs: 148 active+clean; 3.1 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 122 MiB/s wr, 1.95k op/s 2024-09-21T13:54:45.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:45 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2263202723' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:54:45.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:45 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:54:45.972 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:45.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:45.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:54:46.074 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:46.110 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:46.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:46.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:46.212 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:46.246 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:46.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:46.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:46.290 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 28 15 56601 56586 126.285 89.4375 0.0112509 0.00791422 2024-09-21T13:54:46.348 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:54:46.385 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:46.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:46.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:54:46.487 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:46.530 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:46.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:46.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:46.632 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:46.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:46.670 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:46.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:46.727 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:54:46.728 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:54:46.768 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:54:46.769 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:54:46.771 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:54:46.805 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:46 smithi049 ceph-mon[40489]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:54:46.805 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:46 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T13:54:46.805 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:46 smithi049 ceph-mon[40489]: osdmap e195: 8 total, 8 up, 8 in 2024-09-21T13:54:46.805 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:46 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:54:46.805 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:46 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:54:46.805 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:54:46.806 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:54:46.806 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:46 smithi049 ceph-mon[30080]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:54:46.806 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:46 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T13:54:46.806 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:46 smithi049 ceph-mon[30080]: osdmap e195: 8 total, 8 up, 8 in 2024-09-21T13:54:46.806 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:46 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:54:46.806 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:46 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:54:46.818 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:46.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:46.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:54:46.840 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:54:46.840 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:54:46.874 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:54:46.875 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:54:46.891 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:54:46.919 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:54:46.919 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:54:46.921 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:54:46.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:46 smithi187 ceph-mon[33032]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:54:46.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:46 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T13:54:46.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:46 smithi187 ceph-mon[33032]: osdmap e195: 8 total, 8 up, 8 in 2024-09-21T13:54:46.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:46 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:54:46.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:46 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:54:46.950 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:46.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:46.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:54:46.953 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:54:46.953 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:54:46.986 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:54:46.986 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:54:47.052 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:47.087 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:47.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:47.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:47.189 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:47.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:47.234 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:47.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:47.290 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 29 16 58079 58063 125.113 92.3125 0.0113926 0.00798755 2024-09-21T13:54:47.335 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:47.374 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:47.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:47.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:47.475 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:47.512 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:47.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:47.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:47.614 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:54:47.659 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:47.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:47.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:54:47.739 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:47 smithi049 ceph-mon[40489]: pgmap v283: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 134 MiB/s wr, 2.14k op/s 2024-09-21T13:54:47.740 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:47 smithi049 ceph-mon[40489]: osdmap e196: 8 total, 8 up, 8 in 2024-09-21T13:54:47.740 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:47 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2186110618' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:54:47.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:47 smithi049 ceph-mon[30080]: pgmap v283: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 134 MiB/s wr, 2.14k op/s 2024-09-21T13:54:47.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:47 smithi049 ceph-mon[30080]: osdmap e196: 8 total, 8 up, 8 in 2024-09-21T13:54:47.740 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:47 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2186110618' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:54:47.761 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:54:47.789 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:47.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:47.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:54:47.809 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T13:54:47.809 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:54:47.891 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:54:47.932 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:47.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:47.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:54:47.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:47 smithi187 ceph-mon[33032]: pgmap v283: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 134 MiB/s wr, 2.14k op/s 2024-09-21T13:54:47.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:47 smithi187 ceph-mon[33032]: osdmap e196: 8 total, 8 up, 8 in 2024-09-21T13:54:47.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:47 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2186110618' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:54:48.033 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:54:48.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:48.071 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:48.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:54:48.172 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:54:48.205 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:48.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:48.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:54:48.290 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 30 16 59650 59634 124.216 98.1875 0.0037322 0.00804515 2024-09-21T13:54:48.308 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:48.337 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:48.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:48.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:48.439 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:48.455 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 2] 2024-09-21T13:54:48.456 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T13:54:48.456 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-21T13:54:48.456 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T13:54:48.476 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:48.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:48.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:48.578 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:54:48.624 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:48.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:48.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:54:48.725 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:48.781 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T13:54:48.792 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:48.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:48.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:48.894 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:54:48.938 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T13:54:48.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:48.950 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:48.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:54:49.050 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:54:49.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:49.094 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:49.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:54:49.195 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:54:49.222 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:49.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:49.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:54:49.290 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 31 16 61034 61018 122.997 86.5 0.00595687 0.00812492 2024-09-21T13:54:49.324 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:54:49.363 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:49.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:49.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:54:49.464 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:54:49.497 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:54:49.522 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:49.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:49.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:54:49.528 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:54:49.549 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:54:49.578 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:54:49 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:54:49.625 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:54:49.636 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:49 smithi049 ceph-mon[30080]: pgmap v285: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 651 B/s rd, 66 MiB/s wr, 1.06k op/s 2024-09-21T13:54:49.637 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:49 smithi049 ceph-mon[40489]: pgmap v285: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 651 B/s rd, 66 MiB/s wr, 1.06k op/s 2024-09-21T13:54:49.662 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:49.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:49.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:54:49.764 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:54:49.805 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:49.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:49.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:54:49.906 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:54:49.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:49 smithi187 ceph-mon[33032]: pgmap v285: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 651 B/s rd, 66 MiB/s wr, 1.06k op/s 2024-09-21T13:54:49.948 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:49.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:49.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:54:50.051 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:50.083 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:50.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:50.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:50.185 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:54:50.212 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:50.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:50.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:54:50.291 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 32 15 62420 62405 121.86 86.6875 0.0120989 0.0082009 2024-09-21T13:54:50.314 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:50.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:50.358 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:50.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:50.459 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:50.496 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:50.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:50.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:50.575 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:54:50.578 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-09-21T13:54:50.599 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:54:50.649 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:50.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:50.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:54:50.751 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:50.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:50.783 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:50.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:50.884 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:50.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:50.914 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:50.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:50.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:50 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:54:50.957 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:50 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:54:50.957 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:54:50 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:54:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:54:50.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:50 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:54:51.016 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:54:51.056 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:51.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:51.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:54:51.158 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:54:51.200 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:51.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:51.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:54:51.291 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 33 16 63864 63848 120.9 90.1875 0.00415659 0.00826614 2024-09-21T13:54:51.302 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:51.328 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:51.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:51.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:51.401 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T13:54:51.430 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:54:51.441 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":286,"stamp":"2024-09-21T13:54:49.790121+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3943504411,"num_objects":60363,"num_object_clones":0,"num_object_copies":181089,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60363,"num_whiteouts":0,"num_read":1189,"num_read_kb":1046,"num_write":60713,"num_write_kb":3852169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":590588,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60723,"ondisk_log_size":60723,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":12914800,"kb_used_data":11331548,"kb_used_omap":15,"kb_used_meta":1583088,"kb_avail":737046416,"statfs":{"total":767960285184,"available":754735529984,"internally_reserved":0,"allocated":11603505152,"data_stored":11599191249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15598,"internal_metadata":1621082898},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,6,4,6,7,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1040908288,"num_objects":15883,"num_object_clones":0,"num_object_copies":47649,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15883,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":15883,"num_write_kb":1016512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003115"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":236,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.579656+0000","last_change":"2024-09-21T13:52:26.677736+0000","last_active":"2024-09-21T13:54:46.579656+0000","last_peered":"2024-09-21T13:54:46.579656+0000","last_clean":"2024-09-21T13:54:46.579656+0000","last_became_active":"2024-09-21T13:52:04.317322+0000","last_became_peered":"2024-09-21T13:52:04.317322+0000","last_unstale":"2024-09-21T13:54:46.579656+0000","last_undegraded":"2024-09-21T13:54:46.579656+0000","last_fullsized":"2024-09-21T13:54:46.579656+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:26.677689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:52:26.677689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"68'9","reported_seq":311,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.574533+0000","last_change":"2024-09-21T13:54:18.411247+0000","last_active":"2024-09-21T13:54:46.574533+0000","last_peered":"2024-09-21T13:54:46.574533+0000","last_clean":"2024-09-21T13:54:46.574533+0000","last_became_active":"2024-09-21T13:49:00.055210+0000","last_became_peered":"2024-09-21T13:49:00.055210+0000","last_unstale":"2024-09-21T13:54:46.574533+0000","last_undegraded":"2024-09-21T13:54:46.574533+0000","last_fullsized":"2024-09-21T13:54:46.574533+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:54:18.410821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:18.410821+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:40.272886+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":284,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.580025+0000","last_change":"2024-09-21T13:50:28.058990+0000","last_active":"2024-09-21T13:54:46.580025+0000","last_peered":"2024-09-21T13:54:46.580025+0000","last_clean":"2024-09-21T13:54:46.580025+0000","last_became_active":"2024-09-21T13:49:02.060984+0000","last_became_peered":"2024-09-21T13:49:02.060984+0000","last_unstale":"2024-09-21T13:54:46.580025+0000","last_undegraded":"2024-09-21T13:54:46.580025+0000","last_fullsized":"2024-09-21T13:54:46.580025+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:28.058918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:28.058918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:48.620137+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":279,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.576614+0000","last_change":"2024-09-21T13:53:21.005444+0000","last_active":"2024-09-21T13:54:46.576614+0000","last_peered":"2024-09-21T13:54:46.576614+0000","last_clean":"2024-09-21T13:54:46.576614+0000","last_became_active":"2024-09-21T13:49:04.072460+0000","last_became_peered":"2024-09-21T13:49:04.072460+0000","last_unstale":"2024-09-21T13:54:46.576614+0000","last_undegraded":"2024-09-21T13:54:46.576614+0000","last_fullsized":"2024-09-21T13:54:46.576614+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:21.005352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:21.005352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":279,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.593065+0000","last_change":"2024-09-21T13:51:18.572788+0000","last_active":"2024-09-21T13:54:46.593065+0000","last_peered":"2024-09-21T13:54:46.593065+0000","last_clean":"2024-09-21T13:54:46.593065+0000","last_became_active":"2024-09-21T13:49:04.433965+0000","last_became_peered":"2024-09-21T13:49:04.433965+0000","last_unstale":"2024-09-21T13:54:46.593065+0000","last_undegraded":"2024-09-21T13:54:46.593065+0000","last_fullsized":"2024-09-21T13:54:46.593065+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:18.572729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:51:18.572729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:28.969697+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":302,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.589712+0000","last_change":"2024-09-21T13:53:19.910708+0000","last_active":"2024-09-21T13:54:46.589712+0000","last_peered":"2024-09-21T13:54:46.589712+0000","last_clean":"2024-09-21T13:54:46.589712+0000","last_became_active":"2024-09-21T13:52:04.316819+0000","last_became_peered":"2024-09-21T13:52:04.316819+0000","last_unstale":"2024-09-21T13:54:46.589712+0000","last_undegraded":"2024-09-21T13:54:46.589712+0000","last_fullsized":"2024-09-21T13:54:46.589712+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:19.910638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:53:19.910638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"68'15","reported_seq":323,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.588975+0000","last_change":"2024-09-21T13:54:17.024113+0000","last_active":"2024-09-21T13:54:46.588975+0000","last_peered":"2024-09-21T13:54:46.588975+0000","last_clean":"2024-09-21T13:54:46.588975+0000","last_became_active":"2024-09-21T13:52:04.315101+0000","last_became_peered":"2024-09-21T13:52:04.315101+0000","last_unstale":"2024-09-21T13:54:46.588975+0000","last_undegraded":"2024-09-21T13:54:46.588975+0000","last_fullsized":"2024-09-21T13:54:46.588975+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:54:17.024053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:17.024053+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:34.778666+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":291,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.575202+0000","last_change":"2024-09-21T13:51:43.040397+0000","last_active":"2024-09-21T13:54:46.575202+0000","last_peered":"2024-09-21T13:54:46.575202+0000","last_clean":"2024-09-21T13:54:46.575202+0000","last_became_active":"2024-09-21T13:49:02.063687+0000","last_became_peered":"2024-09-21T13:49:02.063687+0000","last_unstale":"2024-09-21T13:54:46.575202+0000","last_undegraded":"2024-09-21T13:54:46.575202+0000","last_fullsized":"2024-09-21T13:54:46.575202+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:43.040348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:43.040348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:08.325545+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":308,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.580135+0000","last_change":"2024-09-21T13:53:59.104109+0000","last_active":"2024-09-21T13:54:46.580135+0000","last_peered":"2024-09-21T13:54:46.580135+0000","last_clean":"2024-09-21T13:54:46.580135+0000","last_became_active":"2024-09-21T13:48:58.050695+0000","last_became_peered":"2024-09-21T13:48:58.050695+0000","last_unstale":"2024-09-21T13:54:46.580135+0000","last_undegraded":"2024-09-21T13:54:46.580135+0000","last_fullsized":"2024-09-21T13:54:46.580135+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:59.104062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:53:59.104062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:28.421324+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"68'9","reported_seq":311,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.576466+0000","last_change":"2024-09-21T13:53:58.308240+0000","last_active":"2024-09-21T13:54:46.576466+0000","last_peered":"2024-09-21T13:54:46.576466+0000","last_clean":"2024-09-21T13:54:46.576466+0000","last_became_active":"2024-09-21T13:49:00.059382+0000","last_became_peered":"2024-09-21T13:49:00.059382+0000","last_unstale":"2024-09-21T13:54:46.576466+0000","last_undegraded":"2024-09-21T13:54:46.576466+0000","last_fullsized":"2024-09-21T13:54:46.576466+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:53:58.308144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:58.308144+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:27.655659+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":276,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.578990+0000","last_change":"2024-09-21T13:49:02.066978+0000","last_active":"2024-09-21T13:54:46.578990+0000","last_peered":"2024-09-21T13:54:46.578990+0000","last_clean":"2024-09-21T13:54:46.578990+0000","last_became_active":"2024-09-21T13:49:02.066693+0000","last_became_peered":"2024-09-21T13:49:02.066693+0000","last_unstale":"2024-09-21T13:54:46.578990+0000","last_undegraded":"2024-09-21T13:54:46.578990+0000","last_fullsized":"2024-09-21T13:54:46.578990+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:12.027057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":240,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.578931+0000","last_change":"2024-09-21T13:54:25.793052+0000","last_active":"2024-09-21T13:54:46.578931+0000","last_peered":"2024-09-21T13:54:46.578931+0000","last_clean":"2024-09-21T13:54:46.578931+0000","last_became_active":"2024-09-21T13:50:29.689381+0000","last_became_peered":"2024-09-21T13:50:29.689381+0000","last_unstale":"2024-09-21T13:54:46.578931+0000","last_undegraded":"2024-09-21T13:54:46.578931+0000","last_fullsized":"2024-09-21T13:54:46.578931+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:25.792985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:54:25.792985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:31.251922+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":292,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.579183+0000","last_change":"2024-09-21T13:50:39.480995+0000","last_active":"2024-09-21T13:54:46.579183+0000","last_peered":"2024-09-21T13:54:46.579183+0000","last_clean":"2024-09-21T13:54:46.579183+0000","last_became_active":"2024-09-21T13:48:58.050906+0000","last_became_peered":"2024-09-21T13:48:58.050906+0000","last_unstale":"2024-09-21T13:54:46.579183+0000","last_undegraded":"2024-09-21T13:54:46.579183+0000","last_fullsized":"2024-09-21T13:54:46.579183+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:39.480930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:39.480930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:34.949015+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"68'5","reported_seq":321,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.576937+0000","last_change":"2024-09-21T13:54:04.271336+0000","last_active":"2024-09-21T13:54:46.576937+0000","last_peered":"2024-09-21T13:54:46.576937+0000","last_clean":"2024-09-21T13:54:46.576937+0000","last_became_active":"2024-09-21T13:52:04.318515+0000","last_became_peered":"2024-09-21T13:52:04.318515+0000","last_unstale":"2024-09-21T13:54:46.576937+0000","last_undegraded":"2024-09-21T13:54:46.576937+0000","last_fullsized":"2024-09-21T13:54:46.576937+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-21T13:54:04.271280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:04.271280+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:18.236498+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":291,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.584664+0000","last_change":"2024-09-21T13:54:26.857307+0000","last_active":"2024-09-21T13:54:46.584664+0000","last_peered":"2024-09-21T13:54:46.584664+0000","last_clean":"2024-09-21T13:54:46.584664+0000","last_became_active":"2024-09-21T13:49:02.060212+0000","last_became_peered":"2024-09-21T13:49:02.060212+0000","last_unstale":"2024-09-21T13:54:46.584664+0000","last_undegraded":"2024-09-21T13:54:46.584664+0000","last_fullsized":"2024-09-21T13:54:46.584664+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:26.857231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:26.857231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:42.878634+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":296,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.580471+0000","last_change":"2024-09-21T13:54:23.043040+0000","last_active":"2024-09-21T13:54:46.580471+0000","last_peered":"2024-09-21T13:54:46.580471+0000","last_clean":"2024-09-21T13:54:46.580471+0000","last_became_active":"2024-09-21T13:49:04.078026+0000","last_became_peered":"2024-09-21T13:49:04.078026+0000","last_unstale":"2024-09-21T13:54:46.580471+0000","last_undegraded":"2024-09-21T13:54:46.580471+0000","last_fullsized":"2024-09-21T13:54:46.580471+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:23.042969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:54:23.042969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:46.405005+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":279,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.576407+0000","last_change":"2024-09-21T13:50:38.027288+0000","last_active":"2024-09-21T13:54:46.576407+0000","last_peered":"2024-09-21T13:54:46.576407+0000","last_clean":"2024-09-21T13:54:46.576407+0000","last_became_active":"2024-09-21T13:49:04.431422+0000","last_became_peered":"2024-09-21T13:49:04.431422+0000","last_unstale":"2024-09-21T13:54:46.576407+0000","last_undegraded":"2024-09-21T13:54:46.576407+0000","last_fullsized":"2024-09-21T13:54:46.576407+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:38.027227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:38.027227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:49.236040+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":223,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.577867+0000","last_change":"2024-09-21T13:54:06.280941+0000","last_active":"2024-09-21T13:54:46.577867+0000","last_peered":"2024-09-21T13:54:46.577867+0000","last_clean":"2024-09-21T13:54:46.577867+0000","last_became_active":"2024-09-21T13:50:29.688282+0000","last_became_peered":"2024-09-21T13:50:29.688282+0000","last_unstale":"2024-09-21T13:54:46.577867+0000","last_undegraded":"2024-09-21T13:54:46.577867+0000","last_fullsized":"2024-09-21T13:54:46.577867+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:06.280894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:06.280894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:07.185880+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"68'15","reported_seq":253,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.576690+0000","last_change":"2024-09-21T13:53:32.929362+0000","last_active":"2024-09-21T13:54:46.576690+0000","last_peered":"2024-09-21T13:54:46.576690+0000","last_clean":"2024-09-21T13:54:46.576690+0000","last_became_active":"2024-09-21T13:52:04.324273+0000","last_became_peered":"2024-09-21T13:52:04.324273+0000","last_unstale":"2024-09-21T13:54:46.576690+0000","last_undegraded":"2024-09-21T13:54:46.576690+0000","last_fullsized":"2024-09-21T13:54:46.576690+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:53:32.929017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:32.929017+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:38.870894+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":292,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.580703+0000","last_change":"2024-09-21T13:52:26.372217+0000","last_active":"2024-09-21T13:54:46.580703+0000","last_peered":"2024-09-21T13:54:46.580703+0000","last_clean":"2024-09-21T13:54:46.580703+0000","last_became_active":"2024-09-21T13:49:02.066796+0000","last_became_peered":"2024-09-21T13:49:02.066796+0000","last_unstale":"2024-09-21T13:54:46.580703+0000","last_undegraded":"2024-09-21T13:54:46.580703+0000","last_fullsized":"2024-09-21T13:54:46.580703+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:26.372144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:52:26.372144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:29.891331+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":279,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.593498+0000","last_change":"2024-09-21T13:53:57.962327+0000","last_active":"2024-09-21T13:54:46.593498+0000","last_peered":"2024-09-21T13:54:46.593498+0000","last_clean":"2024-09-21T13:54:46.593498+0000","last_became_active":"2024-09-21T13:49:04.434049+0000","last_became_peered":"2024-09-21T13:49:04.434049+0000","last_unstale":"2024-09-21T13:54:46.593498+0000","last_undegraded":"2024-09-21T13:54:46.593498+0000","last_fullsized":"2024-09-21T13:54:46.593498+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:57.962266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:57.962266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:07.827936+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":302,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.576598+0000","last_change":"2024-09-21T13:52:52.142682+0000","last_active":"2024-09-21T13:54:46.576598+0000","last_peered":"2024-09-21T13:54:46.576598+0000","last_clean":"2024-09-21T13:54:46.576598+0000","last_became_active":"2024-09-21T13:52:04.316016+0000","last_became_peered":"2024-09-21T13:52:04.316016+0000","last_unstale":"2024-09-21T13:54:46.576598+0000","last_undegraded":"2024-09-21T13:54:46.576598+0000","last_fullsized":"2024-09-21T13:54:46.576598+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:52.142638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:52:52.142638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:36.373450+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"68'12","reported_seq":310,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.594871+0000","last_change":"2024-09-21T13:51:52.819966+0000","last_active":"2024-09-21T13:54:46.594871+0000","last_peered":"2024-09-21T13:54:46.594871+0000","last_clean":"2024-09-21T13:54:46.594871+0000","last_became_active":"2024-09-21T13:49:00.061867+0000","last_became_peered":"2024-09-21T13:49:00.061867+0000","last_unstale":"2024-09-21T13:54:46.594871+0000","last_undegraded":"2024-09-21T13:54:46.594871+0000","last_fullsized":"2024-09-21T13:54:46.594871+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'12","last_scrub_stamp":"2024-09-21T13:51:52.819879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:52.819879+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":286,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.576309+0000","last_change":"2024-09-21T13:52:04.321297+0000","last_active":"2024-09-21T13:54:46.576309+0000","last_peered":"2024-09-21T13:54:46.576309+0000","last_clean":"2024-09-21T13:54:46.576309+0000","last_became_active":"2024-09-21T13:52:04.320800+0000","last_became_peered":"2024-09-21T13:52:04.320800+0000","last_unstale":"2024-09-21T13:54:46.576309+0000","last_undegraded":"2024-09-21T13:54:46.576309+0000","last_fullsized":"2024-09-21T13:54:46.576309+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:15.423635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":291,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.585212+0000","last_change":"2024-09-21T13:52:54.580070+0000","last_active":"2024-09-21T13:54:46.585212+0000","last_peered":"2024-09-21T13:54:46.585212+0000","last_clean":"2024-09-21T13:54:46.585212+0000","last_became_active":"2024-09-21T13:52:04.321358+0000","last_became_peered":"2024-09-21T13:52:04.321358+0000","last_unstale":"2024-09-21T13:54:46.585212+0000","last_undegraded":"2024-09-21T13:54:46.585212+0000","last_fullsized":"2024-09-21T13:54:46.585212+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T13:52:54.580022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:54.580022+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":301,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.580243+0000","last_change":"2024-09-21T13:52:43.516442+0000","last_active":"2024-09-21T13:54:46.580243+0000","last_peered":"2024-09-21T13:54:46.580243+0000","last_clean":"2024-09-21T13:54:46.580243+0000","last_became_active":"2024-09-21T13:48:58.048972+0000","last_became_peered":"2024-09-21T13:48:58.048972+0000","last_unstale":"2024-09-21T13:54:46.580243+0000","last_undegraded":"2024-09-21T13:54:46.580243+0000","last_fullsized":"2024-09-21T13:54:46.580243+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:52:43.516221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:52:43.516221+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:34.940975+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"68'10","reported_seq":259,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.586216+0000","last_change":"2024-09-21T13:52:56.526032+0000","last_active":"2024-09-21T13:54:46.586216+0000","last_peered":"2024-09-21T13:54:46.586216+0000","last_clean":"2024-09-21T13:54:46.586216+0000","last_became_active":"2024-09-21T13:50:29.694465+0000","last_became_peered":"2024-09-21T13:50:29.694465+0000","last_unstale":"2024-09-21T13:54:46.586216+0000","last_undegraded":"2024-09-21T13:54:46.586216+0000","last_fullsized":"2024-09-21T13:54:46.586216+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:52:56.525966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:52:56.525966+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"68'8","reported_seq":374,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.582886+0000","last_change":"2024-09-21T13:51:42.174696+0000","last_active":"2024-09-21T13:54:46.582886+0000","last_peered":"2024-09-21T13:54:46.582886+0000","last_clean":"2024-09-21T13:54:46.582886+0000","last_became_active":"2024-09-21T13:49:02.059638+0000","last_became_peered":"2024-09-21T13:49:02.059638+0000","last_unstale":"2024-09-21T13:54:46.582886+0000","last_undegraded":"2024-09-21T13:54:46.582886+0000","last_fullsized":"2024-09-21T13:54:46.582886+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:51:42.174629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:42.174629+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:14.205512+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"196'3786","reported_seq":3823,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:48.608929+0000","last_change":"2024-09-21T13:54:12.377270+0000","last_active":"2024-09-21T13:54:48.608929+0000","last_peered":"2024-09-21T13:54:48.608929+0000","last_clean":"2024-09-21T13:54:48.608929+0000","last_became_active":"2024-09-21T13:54:12.376466+0000","last_became_peered":"2024-09-21T13:54:12.376466+0000","last_unstale":"2024-09-21T13:54:48.608929+0000","last_undegraded":"2024-09-21T13:54:48.608929+0000","last_fullsized":"2024-09-21T13:54:48.608929+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":3786,"log_dups_size":0,"ondisk_log_size":3786,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:30.045611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":248119296,"num_objects":3786,"num_object_clones":0,"num_object_copies":11358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3786,"num_write_kb":242304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"68'15","reported_seq":329,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.582288+0000","last_change":"2024-09-21T13:53:58.658230+0000","last_active":"2024-09-21T13:54:46.582288+0000","last_peered":"2024-09-21T13:54:46.582288+0000","last_clean":"2024-09-21T13:54:46.582288+0000","last_became_active":"2024-09-21T13:49:00.060324+0000","last_became_peered":"2024-09-21T13:49:00.060324+0000","last_unstale":"2024-09-21T13:54:46.582288+0000","last_undegraded":"2024-09-21T13:54:46.582288+0000","last_fullsized":"2024-09-21T13:54:46.582288+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:53:58.658150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:58.658150+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:14.840395+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":291,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.590853+0000","last_change":"2024-09-21T13:50:06.151943+0000","last_active":"2024-09-21T13:54:46.590853+0000","last_peered":"2024-09-21T13:54:46.590853+0000","last_clean":"2024-09-21T13:54:46.590853+0000","last_became_active":"2024-09-21T13:48:58.051911+0000","last_became_peered":"2024-09-21T13:48:58.051911+0000","last_unstale":"2024-09-21T13:54:46.590853+0000","last_undegraded":"2024-09-21T13:54:46.590853+0000","last_fullsized":"2024-09-21T13:54:46.590853+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.151888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.151888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:12.984236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"68'8","reported_seq":365,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.576525+0000","last_change":"2024-09-21T13:53:19.021881+0000","last_active":"2024-09-21T13:54:46.576525+0000","last_peered":"2024-09-21T13:54:46.576525+0000","last_clean":"2024-09-21T13:54:46.576525+0000","last_became_active":"2024-09-21T13:49:02.061679+0000","last_became_peered":"2024-09-21T13:49:02.061679+0000","last_unstale":"2024-09-21T13:54:46.576525+0000","last_undegraded":"2024-09-21T13:54:46.576525+0000","last_fullsized":"2024-09-21T13:54:46.576525+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:53:19.021810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:53:19.021810+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":280,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.579499+0000","last_change":"2024-09-21T13:50:34.481503+0000","last_active":"2024-09-21T13:54:46.579499+0000","last_peered":"2024-09-21T13:54:46.579499+0000","last_clean":"2024-09-21T13:54:46.579499+0000","last_became_active":"2024-09-21T13:49:04.081905+0000","last_became_peered":"2024-09-21T13:49:04.081905+0000","last_unstale":"2024-09-21T13:54:46.579499+0000","last_undegraded":"2024-09-21T13:54:46.579499+0000","last_fullsized":"2024-09-21T13:54:46.579499+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:34.481450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:34.481450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:44.964465+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.c","version":"196'3824","reported_seq":3860,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:48.746783+0000","last_change":"2024-09-21T13:54:12.479015+0000","last_active":"2024-09-21T13:54:48.746783+0000","last_peered":"2024-09-21T13:54:48.746783+0000","last_clean":"2024-09-21T13:54:48.746783+0000","last_became_active":"2024-09-21T13:54:12.478687+0000","last_became_peered":"2024-09-21T13:54:12.478687+0000","last_unstale":"2024-09-21T13:54:48.746783+0000","last_undegraded":"2024-09-21T13:54:48.746783+0000","last_fullsized":"2024-09-21T13:54:48.746783+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":3824,"log_dups_size":0,"ondisk_log_size":3824,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:33.300965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250609664,"num_objects":3824,"num_object_clones":0,"num_object_copies":11472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3824,"num_write_kb":244736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"68'18","reported_seq":327,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.590630+0000","last_change":"2024-09-21T13:54:19.001120+0000","last_active":"2024-09-21T13:54:46.590630+0000","last_peered":"2024-09-21T13:54:46.590630+0000","last_clean":"2024-09-21T13:54:46.590630+0000","last_became_active":"2024-09-21T13:49:00.429314+0000","last_became_peered":"2024-09-21T13:49:00.429314+0000","last_unstale":"2024-09-21T13:54:46.590630+0000","last_undegraded":"2024-09-21T13:54:46.590630+0000","last_fullsized":"2024-09-21T13:54:46.590630+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'18","last_scrub_stamp":"2024-09-21T13:54:19.001030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:19.001030+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:34.825871+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":283,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.593973+0000","last_change":"2024-09-21T13:48:58.056669+0000","last_active":"2024-09-21T13:54:46.593973+0000","last_peered":"2024-09-21T13:54:46.593973+0000","last_clean":"2024-09-21T13:54:46.593973+0000","last_became_active":"2024-09-21T13:48:58.056375+0000","last_became_peered":"2024-09-21T13:48:58.056375+0000","last_unstale":"2024-09-21T13:54:46.593973+0000","last_undegraded":"2024-09-21T13:54:46.593973+0000","last_fullsized":"2024-09-21T13:54:46.593973+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:36.988185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":291,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.574738+0000","last_change":"2024-09-21T13:54:07.310089+0000","last_active":"2024-09-21T13:54:46.574738+0000","last_peered":"2024-09-21T13:54:46.574738+0000","last_clean":"2024-09-21T13:54:46.574738+0000","last_became_active":"2024-09-21T13:49:02.062955+0000","last_became_peered":"2024-09-21T13:49:02.062955+0000","last_unstale":"2024-09-21T13:54:46.574738+0000","last_undegraded":"2024-09-21T13:54:46.574738+0000","last_fullsized":"2024-09-21T13:54:46.574738+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:07.310044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:07.310044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:23.878204+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":282,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.576380+0000","last_change":"2024-09-21T13:52:04.316586+0000","last_active":"2024-09-21T13:54:46.576380+0000","last_peered":"2024-09-21T13:54:46.576380+0000","last_clean":"2024-09-21T13:54:46.576380+0000","last_became_active":"2024-09-21T13:52:04.316121+0000","last_became_peered":"2024-09-21T13:52:04.316121+0000","last_unstale":"2024-09-21T13:54:46.576380+0000","last_undegraded":"2024-09-21T13:54:46.576380+0000","last_fullsized":"2024-09-21T13:54:46.576380+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:10.346375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"196'3740","reported_seq":3777,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:48.393668+0000","last_change":"2024-09-21T13:54:12.479429+0000","last_active":"2024-09-21T13:54:48.393668+0000","last_peered":"2024-09-21T13:54:48.393668+0000","last_clean":"2024-09-21T13:54:48.393668+0000","last_became_active":"2024-09-21T13:54:12.479146+0000","last_became_peered":"2024-09-21T13:54:12.479146+0000","last_unstale":"2024-09-21T13:54:48.393668+0000","last_undegraded":"2024-09-21T13:54:48.393668+0000","last_fullsized":"2024-09-21T13:54:48.393668+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":3740,"log_dups_size":0,"ondisk_log_size":3740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:37.886798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":245104640,"num_objects":3740,"num_object_clones":0,"num_object_copies":11220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3740,"num_write_kb":239360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"68'14","reported_seq":308,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.574820+0000","last_change":"2024-09-21T13:50:29.009786+0000","last_active":"2024-09-21T13:54:46.574820+0000","last_peered":"2024-09-21T13:54:46.574820+0000","last_clean":"2024-09-21T13:54:46.574820+0000","last_became_active":"2024-09-21T13:49:00.055031+0000","last_became_peered":"2024-09-21T13:49:00.055031+0000","last_unstale":"2024-09-21T13:54:46.574820+0000","last_undegraded":"2024-09-21T13:54:46.574820+0000","last_fullsized":"2024-09-21T13:54:46.574820+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'14","last_scrub_stamp":"2024-09-21T13:50:29.009720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:29.009720+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:44.453910+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":293,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.589139+0000","last_change":"2024-09-21T13:49:59.253636+0000","last_active":"2024-09-21T13:54:46.589139+0000","last_peered":"2024-09-21T13:54:46.589139+0000","last_clean":"2024-09-21T13:54:46.589139+0000","last_became_active":"2024-09-21T13:48:58.053533+0000","last_became_peered":"2024-09-21T13:48:58.053533+0000","last_unstale":"2024-09-21T13:54:46.589139+0000","last_undegraded":"2024-09-21T13:54:46.589139+0000","last_fullsized":"2024-09-21T13:54:46.589139+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:49:59.253587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:49:59.253587+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:36.902694+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"68'8","reported_seq":370,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.590470+0000","last_change":"2024-09-21T13:54:25.020479+0000","last_active":"2024-09-21T13:54:46.590470+0000","last_peered":"2024-09-21T13:54:46.590470+0000","last_clean":"2024-09-21T13:54:46.590470+0000","last_became_active":"2024-09-21T13:49:02.070064+0000","last_became_peered":"2024-09-21T13:49:02.070064+0000","last_unstale":"2024-09-21T13:54:46.590470+0000","last_undegraded":"2024-09-21T13:54:46.590470+0000","last_fullsized":"2024-09-21T13:54:46.590470+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:54:25.020230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:25.020230+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:50.633173+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":290,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.584660+0000","last_change":"2024-09-21T13:52:55.541288+0000","last_active":"2024-09-21T13:54:46.584660+0000","last_peered":"2024-09-21T13:54:46.584660+0000","last_clean":"2024-09-21T13:54:46.584660+0000","last_became_active":"2024-09-21T13:52:04.322185+0000","last_became_peered":"2024-09-21T13:52:04.322185+0000","last_unstale":"2024-09-21T13:54:46.584660+0000","last_undegraded":"2024-09-21T13:54:46.584660+0000","last_fullsized":"2024-09-21T13:54:46.584660+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:55.541232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:55.541232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.e","version":"196'3782","reported_seq":3819,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:48.739523+0000","last_change":"2024-09-21T13:54:12.480264+0000","last_active":"2024-09-21T13:54:48.739523+0000","last_peered":"2024-09-21T13:54:48.739523+0000","last_clean":"2024-09-21T13:54:48.739523+0000","last_became_active":"2024-09-21T13:54:12.479901+0000","last_became_peered":"2024-09-21T13:54:12.479901+0000","last_unstale":"2024-09-21T13:54:48.739523+0000","last_undegraded":"2024-09-21T13:54:48.739523+0000","last_fullsized":"2024-09-21T13:54:48.739523+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":3782,"log_dups_size":0,"ondisk_log_size":3782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:35.816508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247857152,"num_objects":3782,"num_object_clones":0,"num_object_copies":11346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3782,"num_write_kb":242048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"68'10","reported_seq":319,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.581654+0000","last_change":"2024-09-21T13:54:19.593696+0000","last_active":"2024-09-21T13:54:46.581654+0000","last_peered":"2024-09-21T13:54:46.581654+0000","last_clean":"2024-09-21T13:54:46.581654+0000","last_became_active":"2024-09-21T13:49:00.054564+0000","last_became_peered":"2024-09-21T13:49:00.054564+0000","last_unstale":"2024-09-21T13:54:46.581654+0000","last_undegraded":"2024-09-21T13:54:46.581654+0000","last_fullsized":"2024-09-21T13:54:46.581654+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:54:19.593611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:19.593611+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:40.845382+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"61'1","reported_seq":325,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.580210+0000","last_change":"2024-09-21T13:53:30.179679+0000","last_active":"2024-09-21T13:54:46.580210+0000","last_peered":"2024-09-21T13:54:46.580210+0000","last_clean":"2024-09-21T13:54:46.580210+0000","last_became_active":"2024-09-21T13:52:04.315788+0000","last_became_peered":"2024-09-21T13:52:04.315788+0000","last_unstale":"2024-09-21T13:54:46.580210+0000","last_undegraded":"2024-09-21T13:54:46.580210+0000","last_fullsized":"2024-09-21T13:54:46.580210+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:53:30.179605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:53:30.179605+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"96'9","reported_seq":319,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.582746+0000","last_change":"2024-09-21T13:54:06.280517+0000","last_active":"2024-09-21T13:54:46.582746+0000","last_peered":"2024-09-21T13:54:46.582746+0000","last_clean":"2024-09-21T13:54:46.582746+0000","last_became_active":"2024-09-21T13:50:29.693557+0000","last_became_peered":"2024-09-21T13:50:29.693557+0000","last_unstale":"2024-09-21T13:54:46.582746+0000","last_undegraded":"2024-09-21T13:54:46.582746+0000","last_fullsized":"2024-09-21T13:54:46.582746+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'9","last_scrub_stamp":"2024-09-21T13:54:06.280457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:06.280457+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:23.993139+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":235,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.590345+0000","last_change":"2024-09-21T13:53:54.948771+0000","last_active":"2024-09-21T13:54:46.590345+0000","last_peered":"2024-09-21T13:54:46.590345+0000","last_clean":"2024-09-21T13:54:46.590345+0000","last_became_active":"2024-09-21T13:52:04.319034+0000","last_became_peered":"2024-09-21T13:52:04.319034+0000","last_unstale":"2024-09-21T13:54:46.590345+0000","last_undegraded":"2024-09-21T13:54:46.590345+0000","last_fullsized":"2024-09-21T13:54:46.590345+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:54.948705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:54.948705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:05.874612+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.f","version":"196'3871","reported_seq":3906,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:49.058652+0000","last_change":"2024-09-21T13:54:12.478557+0000","last_active":"2024-09-21T13:54:49.058652+0000","last_peered":"2024-09-21T13:54:49.058652+0000","last_clean":"2024-09-21T13:54:49.058652+0000","last_became_active":"2024-09-21T13:54:12.477589+0000","last_became_peered":"2024-09-21T13:54:12.477589+0000","last_unstale":"2024-09-21T13:54:49.058652+0000","last_undegraded":"2024-09-21T13:54:49.058652+0000","last_fullsized":"2024-09-21T13:54:49.058652+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":3871,"log_dups_size":0,"ondisk_log_size":3871,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:29.462737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":253689856,"num_objects":3871,"num_object_clones":0,"num_object_copies":11613,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3871,"num_write_kb":247744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"68'19","reported_seq":281,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.579074+0000","last_change":"2024-09-21T13:50:32.514267+0000","last_active":"2024-09-21T13:54:46.579074+0000","last_peered":"2024-09-21T13:54:46.579074+0000","last_clean":"2024-09-21T13:54:46.579074+0000","last_became_active":"2024-09-21T13:50:29.690185+0000","last_became_peered":"2024-09-21T13:50:29.690185+0000","last_unstale":"2024-09-21T13:54:46.579074+0000","last_undegraded":"2024-09-21T13:54:46.579074+0000","last_fullsized":"2024-09-21T13:54:46.579074+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-21T13:50:32.514210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:32.514210+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":295,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.580438+0000","last_change":"2024-09-21T13:52:04.322289+0000","last_active":"2024-09-21T13:54:46.580438+0000","last_peered":"2024-09-21T13:54:46.580438+0000","last_clean":"2024-09-21T13:54:46.580438+0000","last_became_active":"2024-09-21T13:52:04.321656+0000","last_became_peered":"2024-09-21T13:52:04.321656+0000","last_unstale":"2024-09-21T13:54:46.580438+0000","last_undegraded":"2024-09-21T13:54:46.580438+0000","last_fullsized":"2024-09-21T13:54:46.580438+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:12.655427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":292,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.582106+0000","last_change":"2024-09-21T13:53:32.295930+0000","last_active":"2024-09-21T13:54:46.582106+0000","last_peered":"2024-09-21T13:54:46.582106+0000","last_clean":"2024-09-21T13:54:46.582106+0000","last_became_active":"2024-09-21T13:49:02.061187+0000","last_became_peered":"2024-09-21T13:49:02.061187+0000","last_unstale":"2024-09-21T13:54:46.582106+0000","last_undegraded":"2024-09-21T13:54:46.582106+0000","last_fullsized":"2024-09-21T13:54:46.582106+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:32.295869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:53:32.295869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:33.093729+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":283,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.579932+0000","last_change":"2024-09-21T13:52:04.321133+0000","last_active":"2024-09-21T13:54:46.579932+0000","last_peered":"2024-09-21T13:54:46.579932+0000","last_clean":"2024-09-21T13:54:46.579932+0000","last_became_active":"2024-09-21T13:52:04.320826+0000","last_became_peered":"2024-09-21T13:52:04.320826+0000","last_unstale":"2024-09-21T13:54:46.579932+0000","last_undegraded":"2024-09-21T13:54:46.579932+0000","last_fullsized":"2024-09-21T13:54:46.579932+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:14.698955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"196'3766","reported_seq":3800,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:48.401700+0000","last_change":"2024-09-21T13:54:12.477847+0000","last_active":"2024-09-21T13:54:48.401700+0000","last_peered":"2024-09-21T13:54:48.401700+0000","last_clean":"2024-09-21T13:54:48.401700+0000","last_became_active":"2024-09-21T13:54:12.477306+0000","last_became_peered":"2024-09-21T13:54:12.477306+0000","last_unstale":"2024-09-21T13:54:48.401700+0000","last_undegraded":"2024-09-21T13:54:48.401700+0000","last_fullsized":"2024-09-21T13:54:48.401700+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":3766,"log_dups_size":0,"ondisk_log_size":3766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:26.986694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":246808576,"num_objects":3766,"num_object_clones":0,"num_object_copies":11298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3766,"num_write_kb":241024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"132'30","reported_seq":420,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.581062+0000","last_change":"2024-09-21T13:54:16.681017+0000","last_active":"2024-09-21T13:54:46.581062+0000","last_peered":"2024-09-21T13:54:46.581062+0000","last_clean":"2024-09-21T13:54:46.581062+0000","last_became_active":"2024-09-21T13:52:04.319431+0000","last_became_peered":"2024-09-21T13:52:04.319431+0000","last_unstale":"2024-09-21T13:54:46.581062+0000","last_undegraded":"2024-09-21T13:54:46.581062+0000","last_fullsized":"2024-09-21T13:54:46.581062+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'30","last_scrub_stamp":"2024-09-21T13:54:16.680953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:16.680953+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:39.761349+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":283,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.577250+0000","last_change":"2024-09-21T13:48:58.055640+0000","last_active":"2024-09-21T13:54:46.577250+0000","last_peered":"2024-09-21T13:54:46.577250+0000","last_clean":"2024-09-21T13:54:46.577250+0000","last_became_active":"2024-09-21T13:48:58.055366+0000","last_became_peered":"2024-09-21T13:48:58.055366+0000","last_unstale":"2024-09-21T13:54:46.577250+0000","last_undegraded":"2024-09-21T13:54:46.577250+0000","last_fullsized":"2024-09-21T13:54:46.577250+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:09.329706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":318,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.580048+0000","last_change":"2024-09-21T13:54:25.029827+0000","last_active":"2024-09-21T13:54:46.580048+0000","last_peered":"2024-09-21T13:54:46.580048+0000","last_clean":"2024-09-21T13:54:46.580048+0000","last_became_active":"2024-09-21T13:48:57.046246+0000","last_became_peered":"2024-09-21T13:48:57.046246+0000","last_unstale":"2024-09-21T13:54:46.580048+0000","last_undegraded":"2024-09-21T13:54:46.580048+0000","last_fullsized":"2024-09-21T13:54:46.580048+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-21T13:54:25.029757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:54:25.029757+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:27.260800+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0015946669999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":233,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.580525+0000","last_change":"2024-09-21T13:53:56.670634+0000","last_active":"2024-09-21T13:54:46.580525+0000","last_peered":"2024-09-21T13:54:46.580525+0000","last_clean":"2024-09-21T13:54:46.580525+0000","last_became_active":"2024-09-21T13:50:29.688249+0000","last_became_peered":"2024-09-21T13:50:29.688249+0000","last_unstale":"2024-09-21T13:54:46.580525+0000","last_undegraded":"2024-09-21T13:54:46.580525+0000","last_fullsized":"2024-09-21T13:54:46.580525+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:56.670555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:53:56.670555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:05.626771+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":279,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.589349+0000","last_change":"2024-09-21T13:51:23.597164+0000","last_active":"2024-09-21T13:54:46.589349+0000","last_peered":"2024-09-21T13:54:46.589349+0000","last_clean":"2024-09-21T13:54:46.589349+0000","last_became_active":"2024-09-21T13:49:04.080466+0000","last_became_peered":"2024-09-21T13:49:04.080466+0000","last_unstale":"2024-09-21T13:54:46.589349+0000","last_undegraded":"2024-09-21T13:54:46.589349+0000","last_fullsized":"2024-09-21T13:54:46.589349+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:23.596879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:51:23.596879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"196'3734","reported_seq":3770,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:49.047741+0000","last_change":"2024-09-21T13:54:12.373760+0000","last_active":"2024-09-21T13:54:49.047741+0000","last_peered":"2024-09-21T13:54:49.047741+0000","last_clean":"2024-09-21T13:54:49.047741+0000","last_became_active":"2024-09-21T13:54:12.373395+0000","last_became_peered":"2024-09-21T13:54:12.373395+0000","last_unstale":"2024-09-21T13:54:49.047741+0000","last_undegraded":"2024-09-21T13:54:49.047741+0000","last_fullsized":"2024-09-21T13:54:49.047741+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":3734,"log_dups_size":0,"ondisk_log_size":3734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:36.149952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244711424,"num_objects":3734,"num_object_clones":0,"num_object_copies":11202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3734,"num_write_kb":238976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"68'13","reported_seq":315,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.582294+0000","last_change":"2024-09-21T13:50:06.973399+0000","last_active":"2024-09-21T13:54:46.582294+0000","last_peered":"2024-09-21T13:54:46.582294+0000","last_clean":"2024-09-21T13:54:46.582294+0000","last_became_active":"2024-09-21T13:49:00.053323+0000","last_became_peered":"2024-09-21T13:49:00.053323+0000","last_unstale":"2024-09-21T13:54:46.582294+0000","last_undegraded":"2024-09-21T13:54:46.582294+0000","last_fullsized":"2024-09-21T13:54:46.582294+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'13","last_scrub_stamp":"2024-09-21T13:50:06.973336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.973336+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:41.970680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":319,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.579582+0000","last_change":"2024-09-21T13:53:24.885266+0000","last_active":"2024-09-21T13:54:46.579582+0000","last_peered":"2024-09-21T13:54:46.579582+0000","last_clean":"2024-09-21T13:54:46.579582+0000","last_became_active":"2024-09-21T13:52:04.320581+0000","last_became_peered":"2024-09-21T13:52:04.320581+0000","last_unstale":"2024-09-21T13:54:46.579582+0000","last_undegraded":"2024-09-21T13:54:46.579582+0000","last_fullsized":"2024-09-21T13:54:46.579582+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:24.884932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:53:24.884932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:40.993958+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":219,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.577713+0000","last_change":"2024-09-21T13:53:19.985044+0000","last_active":"2024-09-21T13:54:46.577713+0000","last_peered":"2024-09-21T13:54:46.577713+0000","last_clean":"2024-09-21T13:54:46.577713+0000","last_became_active":"2024-09-21T13:52:04.316171+0000","last_became_peered":"2024-09-21T13:52:04.316171+0000","last_unstale":"2024-09-21T13:54:46.577713+0000","last_undegraded":"2024-09-21T13:54:46.577713+0000","last_fullsized":"2024-09-21T13:54:46.577713+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:53:19.984966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:53:19.984966+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:47.985977+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":171,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.579293+0000","last_change":"2024-09-21T13:54:26.009416+0000","last_active":"2024-09-21T13:54:46.579293+0000","last_peered":"2024-09-21T13:54:46.579293+0000","last_clean":"2024-09-21T13:54:46.579293+0000","last_became_active":"2024-09-21T13:52:04.320701+0000","last_became_peered":"2024-09-21T13:52:04.320701+0000","last_unstale":"2024-09-21T13:54:46.579293+0000","last_undegraded":"2024-09-21T13:54:46.579293+0000","last_fullsized":"2024-09-21T13:54:46.579293+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:26.009352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:26.009352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:55.894955+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":287,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.585516+0000","last_change":"2024-09-21T13:51:12.212558+0000","last_active":"2024-09-21T13:54:46.585516+0000","last_peered":"2024-09-21T13:54:46.585516+0000","last_clean":"2024-09-21T13:54:46.585516+0000","last_became_active":"2024-09-21T13:49:04.431035+0000","last_became_peered":"2024-09-21T13:49:04.431035+0000","last_unstale":"2024-09-21T13:54:46.585516+0000","last_undegraded":"2024-09-21T13:54:46.585516+0000","last_fullsized":"2024-09-21T13:54:46.585516+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:12.212391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:51:12.212391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:31.820181+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"196'3719","reported_seq":3755,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:48.619321+0000","last_change":"2024-09-21T13:54:12.477912+0000","last_active":"2024-09-21T13:54:48.619321+0000","last_peered":"2024-09-21T13:54:48.619321+0000","last_clean":"2024-09-21T13:54:48.619321+0000","last_became_active":"2024-09-21T13:54:12.477596+0000","last_became_peered":"2024-09-21T13:54:12.477596+0000","last_unstale":"2024-09-21T13:54:48.619321+0000","last_undegraded":"2024-09-21T13:54:48.619321+0000","last_fullsized":"2024-09-21T13:54:48.619321+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":3719,"log_dups_size":0,"ondisk_log_size":3719,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:33.789453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243728384,"num_objects":3719,"num_object_clones":0,"num_object_copies":11157,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3719,"num_write_kb":238016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"68'12","reported_seq":346,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.588984+0000","last_change":"2024-09-21T13:53:17.740156+0000","last_active":"2024-09-21T13:54:46.588984+0000","last_peered":"2024-09-21T13:54:46.588984+0000","last_clean":"2024-09-21T13:54:46.588984+0000","last_became_active":"2024-09-21T13:52:04.318875+0000","last_became_peered":"2024-09-21T13:52:04.318875+0000","last_unstale":"2024-09-21T13:54:46.588984+0000","last_undegraded":"2024-09-21T13:54:46.588984+0000","last_fullsized":"2024-09-21T13:54:46.588984+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'12","last_scrub_stamp":"2024-09-21T13:53:17.740080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:17.740080+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:33.880649+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":224,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.575335+0000","last_change":"2024-09-21T13:50:30.020722+0000","last_active":"2024-09-21T13:54:46.575335+0000","last_peered":"2024-09-21T13:54:46.575335+0000","last_clean":"2024-09-21T13:54:46.575335+0000","last_became_active":"2024-09-21T13:50:29.688352+0000","last_became_peered":"2024-09-21T13:50:29.688352+0000","last_unstale":"2024-09-21T13:54:46.575335+0000","last_undegraded":"2024-09-21T13:54:46.575335+0000","last_fullsized":"2024-09-21T13:54:46.575335+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:30.020656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:30.020656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:42.379271+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":647,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:47.369300+0000","last_change":"2024-09-21T13:51:21.271997+0000","last_active":"2024-09-21T13:54:47.369300+0000","last_peered":"2024-09-21T13:54:47.369300+0000","last_clean":"2024-09-21T13:54:47.369300+0000","last_became_active":"2024-09-21T13:48:57.046084+0000","last_became_peered":"2024-09-21T13:48:57.046084+0000","last_unstale":"2024-09-21T13:54:47.369300+0000","last_undegraded":"2024-09-21T13:54:47.369300+0000","last_fullsized":"2024-09-21T13:54:47.369300+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-21T13:51:21.271917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:51:21.271917+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00047048000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":345,"num_read_kb":340,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":284,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.579579+0000","last_change":"2024-09-21T13:51:53.518645+0000","last_active":"2024-09-21T13:54:46.579579+0000","last_peered":"2024-09-21T13:54:46.579579+0000","last_clean":"2024-09-21T13:54:46.579579+0000","last_became_active":"2024-09-21T13:49:02.066641+0000","last_became_peered":"2024-09-21T13:49:02.066641+0000","last_unstale":"2024-09-21T13:54:46.579579+0000","last_undegraded":"2024-09-21T13:54:46.579579+0000","last_fullsized":"2024-09-21T13:54:46.579579+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:53.518567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:53.518567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:16.069137+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":288,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.579936+0000","last_change":"2024-09-21T13:52:21.332066+0000","last_active":"2024-09-21T13:54:46.579936+0000","last_peered":"2024-09-21T13:54:46.579936+0000","last_clean":"2024-09-21T13:54:46.579936+0000","last_became_active":"2024-09-21T13:49:04.078372+0000","last_became_peered":"2024-09-21T13:49:04.078372+0000","last_unstale":"2024-09-21T13:54:46.579936+0000","last_undegraded":"2024-09-21T13:54:46.579936+0000","last_fullsized":"2024-09-21T13:54:46.579936+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:21.331792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:21.331792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:32.973257+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"196'3825","reported_seq":3862,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:48.420795+0000","last_change":"2024-09-21T13:54:12.378930+0000","last_active":"2024-09-21T13:54:48.420795+0000","last_peered":"2024-09-21T13:54:48.420795+0000","last_clean":"2024-09-21T13:54:48.420795+0000","last_became_active":"2024-09-21T13:54:12.378536+0000","last_became_peered":"2024-09-21T13:54:12.378536+0000","last_unstale":"2024-09-21T13:54:48.420795+0000","last_undegraded":"2024-09-21T13:54:48.420795+0000","last_fullsized":"2024-09-21T13:54:48.420795+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":3825,"log_dups_size":0,"ondisk_log_size":3825,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:32.246204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250675200,"num_objects":3825,"num_object_clones":0,"num_object_copies":11475,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3825,"num_write_kb":244800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"68'16","reported_seq":389,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.582866+0000","last_change":"2024-09-21T13:52:19.452239+0000","last_active":"2024-09-21T13:54:46.582866+0000","last_peered":"2024-09-21T13:54:46.582866+0000","last_clean":"2024-09-21T13:54:46.582866+0000","last_became_active":"2024-09-21T13:49:00.426867+0000","last_became_peered":"2024-09-21T13:49:00.426867+0000","last_unstale":"2024-09-21T13:54:46.582866+0000","last_undegraded":"2024-09-21T13:54:46.582866+0000","last_fullsized":"2024-09-21T13:54:46.582866+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'16","last_scrub_stamp":"2024-09-21T13:52:19.452163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:52:19.452163+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:49.265943+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":299,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.588868+0000","last_change":"2024-09-21T13:54:07.057258+0000","last_active":"2024-09-21T13:54:46.588868+0000","last_peered":"2024-09-21T13:54:46.588868+0000","last_clean":"2024-09-21T13:54:46.588868+0000","last_became_active":"2024-09-21T13:48:58.052659+0000","last_became_peered":"2024-09-21T13:48:58.052659+0000","last_unstale":"2024-09-21T13:54:46.588868+0000","last_undegraded":"2024-09-21T13:54:46.588868+0000","last_fullsized":"2024-09-21T13:54:46.588868+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:07.057210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:07.057210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:14.084334+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"75'79","reported_seq":341,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.584821+0000","last_change":"2024-09-21T13:50:27.096470+0000","last_active":"2024-09-21T13:54:46.584821+0000","last_peered":"2024-09-21T13:54:46.584821+0000","last_clean":"2024-09-21T13:54:46.584821+0000","last_became_active":"2024-09-21T13:48:48.454646+0000","last_became_peered":"2024-09-21T13:48:48.454646+0000","last_unstale":"2024-09-21T13:54:46.584821+0000","last_undegraded":"2024-09-21T13:54:46.584821+0000","last_fullsized":"2024-09-21T13:54:46.584821+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:50:27.096413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:50:27.096413+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:41.842457+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":286,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.586432+0000","last_change":"2024-09-21T13:52:04.320948+0000","last_active":"2024-09-21T13:54:46.586432+0000","last_peered":"2024-09-21T13:54:46.586432+0000","last_clean":"2024-09-21T13:54:46.586432+0000","last_became_active":"2024-09-21T13:52:04.320627+0000","last_became_peered":"2024-09-21T13:52:04.320627+0000","last_unstale":"2024-09-21T13:54:46.586432+0000","last_undegraded":"2024-09-21T13:54:46.586432+0000","last_fullsized":"2024-09-21T13:54:46.586432+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:50.544725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":280,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.579303+0000","last_change":"2024-09-21T13:50:25.560928+0000","last_active":"2024-09-21T13:54:46.579303+0000","last_peered":"2024-09-21T13:54:46.579303+0000","last_clean":"2024-09-21T13:54:46.579303+0000","last_became_active":"2024-09-21T13:49:04.078091+0000","last_became_peered":"2024-09-21T13:49:04.078091+0000","last_unstale":"2024-09-21T13:54:46.579303+0000","last_undegraded":"2024-09-21T13:54:46.579303+0000","last_fullsized":"2024-09-21T13:54:46.579303+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:25.560863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:25.560863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:41.830570+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"196'3705","reported_seq":3741,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:48.403784+0000","last_change":"2024-09-21T13:54:12.378230+0000","last_active":"2024-09-21T13:54:48.403784+0000","last_peered":"2024-09-21T13:54:48.403784+0000","last_clean":"2024-09-21T13:54:48.403784+0000","last_became_active":"2024-09-21T13:54:12.377874+0000","last_became_peered":"2024-09-21T13:54:12.377874+0000","last_unstale":"2024-09-21T13:54:48.403784+0000","last_undegraded":"2024-09-21T13:54:48.403784+0000","last_fullsized":"2024-09-21T13:54:48.403784+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":3705,"log_dups_size":0,"ondisk_log_size":3705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:39.278391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":242810880,"num_objects":3705,"num_object_clones":0,"num_object_copies":11115,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3705,"num_write_kb":237120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"68'11","reported_seq":306,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.576571+0000","last_change":"2024-09-21T13:54:16.451648+0000","last_active":"2024-09-21T13:54:46.576571+0000","last_peered":"2024-09-21T13:54:46.576571+0000","last_clean":"2024-09-21T13:54:46.576571+0000","last_became_active":"2024-09-21T13:49:00.053268+0000","last_became_peered":"2024-09-21T13:49:00.053268+0000","last_unstale":"2024-09-21T13:54:46.576571+0000","last_undegraded":"2024-09-21T13:54:46.576571+0000","last_fullsized":"2024-09-21T13:54:46.576571+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:54:16.451588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:16.451588+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:41.356489+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":302,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.575552+0000","last_change":"2024-09-21T13:52:48.165210+0000","last_active":"2024-09-21T13:54:46.575552+0000","last_peered":"2024-09-21T13:54:46.575552+0000","last_clean":"2024-09-21T13:54:46.575552+0000","last_became_active":"2024-09-21T13:52:04.318345+0000","last_became_peered":"2024-09-21T13:52:04.318345+0000","last_unstale":"2024-09-21T13:54:46.575552+0000","last_undegraded":"2024-09-21T13:54:46.575552+0000","last_fullsized":"2024-09-21T13:54:46.575552+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:48.165145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:52:48.165145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:32.368867+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":292,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.579148+0000","last_change":"2024-09-21T13:51:46.503640+0000","last_active":"2024-09-21T13:54:46.579148+0000","last_peered":"2024-09-21T13:54:46.579148+0000","last_clean":"2024-09-21T13:54:46.579148+0000","last_became_active":"2024-09-21T13:49:02.061901+0000","last_became_peered":"2024-09-21T13:49:02.061901+0000","last_unstale":"2024-09-21T13:54:46.579148+0000","last_undegraded":"2024-09-21T13:54:46.579148+0000","last_fullsized":"2024-09-21T13:54:46.579148+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:46.503576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:46.503576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:15.052988+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":279,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.589597+0000","last_change":"2024-09-21T13:51:51.814259+0000","last_active":"2024-09-21T13:54:46.589597+0000","last_peered":"2024-09-21T13:54:46.589597+0000","last_clean":"2024-09-21T13:54:46.589597+0000","last_became_active":"2024-09-21T13:49:04.431268+0000","last_became_peered":"2024-09-21T13:49:04.431268+0000","last_unstale":"2024-09-21T13:54:46.589597+0000","last_undegraded":"2024-09-21T13:54:46.589597+0000","last_fullsized":"2024-09-21T13:54:46.589597+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:51.814200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:51:51.814200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1","version":"196'3610","reported_seq":3647,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:48.604453+0000","last_change":"2024-09-21T13:54:12.478397+0000","last_active":"2024-09-21T13:54:48.604453+0000","last_peered":"2024-09-21T13:54:48.604453+0000","last_clean":"2024-09-21T13:54:48.604453+0000","last_became_active":"2024-09-21T13:54:12.477588+0000","last_became_peered":"2024-09-21T13:54:12.477588+0000","last_unstale":"2024-09-21T13:54:48.604453+0000","last_undegraded":"2024-09-21T13:54:48.604453+0000","last_fullsized":"2024-09-21T13:54:48.604453+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":3610,"log_dups_size":0,"ondisk_log_size":3610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:37.981662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236584960,"num_objects":3610,"num_object_clones":0,"num_object_copies":10830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3610,"num_write_kb":231040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"68'17","reported_seq":326,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.578076+0000","last_change":"2024-09-21T13:52:45.176950+0000","last_active":"2024-09-21T13:54:46.578076+0000","last_peered":"2024-09-21T13:54:46.578076+0000","last_clean":"2024-09-21T13:54:46.578076+0000","last_became_active":"2024-09-21T13:52:04.313218+0000","last_became_peered":"2024-09-21T13:52:04.313218+0000","last_unstale":"2024-09-21T13:54:46.578076+0000","last_undegraded":"2024-09-21T13:54:46.578076+0000","last_fullsized":"2024-09-21T13:54:46.578076+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'17","last_scrub_stamp":"2024-09-21T13:52:45.176873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:52:45.176873+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:35.397286+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":308,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.582464+0000","last_change":"2024-09-21T13:53:55.202100+0000","last_active":"2024-09-21T13:54:46.582464+0000","last_peered":"2024-09-21T13:54:46.582464+0000","last_clean":"2024-09-21T13:54:46.582464+0000","last_became_active":"2024-09-21T13:48:58.049142+0000","last_became_peered":"2024-09-21T13:48:58.049142+0000","last_unstale":"2024-09-21T13:54:46.582464+0000","last_undegraded":"2024-09-21T13:54:46.582464+0000","last_fullsized":"2024-09-21T13:54:46.582464+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:55.202057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:53:55.202057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:00.441490+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":284,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.580028+0000","last_change":"2024-09-21T13:54:06.629867+0000","last_active":"2024-09-21T13:54:46.580028+0000","last_peered":"2024-09-21T13:54:46.580028+0000","last_clean":"2024-09-21T13:54:46.580028+0000","last_became_active":"2024-09-21T13:49:02.067472+0000","last_became_peered":"2024-09-21T13:49:02.067472+0000","last_unstale":"2024-09-21T13:54:46.580028+0000","last_undegraded":"2024-09-21T13:54:46.580028+0000","last_fullsized":"2024-09-21T13:54:46.580028+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:06.629822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:06.629822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:14.040598+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":163,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.578628+0000","last_change":"2024-09-21T13:53:18.028013+0000","last_active":"2024-09-21T13:54:46.578628+0000","last_peered":"2024-09-21T13:54:46.578628+0000","last_clean":"2024-09-21T13:54:46.578628+0000","last_became_active":"2024-09-21T13:52:04.315920+0000","last_became_peered":"2024-09-21T13:52:04.315920+0000","last_unstale":"2024-09-21T13:54:46.578628+0000","last_undegraded":"2024-09-21T13:54:46.578628+0000","last_fullsized":"2024-09-21T13:54:46.578628+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:18.027944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:18.027944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:30.919961+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.0","version":"196'3799","reported_seq":3835,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:49.059635+0000","last_change":"2024-09-21T13:54:12.375805+0000","last_active":"2024-09-21T13:54:49.059635+0000","last_peered":"2024-09-21T13:54:49.059635+0000","last_clean":"2024-09-21T13:54:49.059635+0000","last_became_active":"2024-09-21T13:54:12.375197+0000","last_became_peered":"2024-09-21T13:54:12.375197+0000","last_unstale":"2024-09-21T13:54:49.059635+0000","last_undegraded":"2024-09-21T13:54:49.059635+0000","last_fullsized":"2024-09-21T13:54:49.059635+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":3799,"log_dups_size":0,"ondisk_log_size":3799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:17.781582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":248971264,"num_objects":3799,"num_object_clones":0,"num_object_copies":11397,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3799,"num_write_kb":243136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"68'10","reported_seq":310,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.576391+0000","last_change":"2024-09-21T13:54:20.426028+0000","last_active":"2024-09-21T13:54:46.576391+0000","last_peered":"2024-09-21T13:54:46.576391+0000","last_clean":"2024-09-21T13:54:46.576391+0000","last_became_active":"2024-09-21T13:49:00.059850+0000","last_became_peered":"2024-09-21T13:49:00.059850+0000","last_unstale":"2024-09-21T13:54:46.576391+0000","last_undegraded":"2024-09-21T13:54:46.576391+0000","last_fullsized":"2024-09-21T13:54:46.576391+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:54:20.425889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:20.425889+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:32.475251+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":291,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.593951+0000","last_change":"2024-09-21T13:51:28.565842+0000","last_active":"2024-09-21T13:54:46.593951+0000","last_peered":"2024-09-21T13:54:46.593951+0000","last_clean":"2024-09-21T13:54:46.593951+0000","last_became_active":"2024-09-21T13:48:58.058309+0000","last_became_peered":"2024-09-21T13:48:58.058309+0000","last_unstale":"2024-09-21T13:54:46.593951+0000","last_undegraded":"2024-09-21T13:54:46.593951+0000","last_fullsized":"2024-09-21T13:54:46.593951+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:28.565766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:51:28.565766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"132'10","reported_seq":214,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.578185+0000","last_change":"2024-09-21T13:54:19.934832+0000","last_active":"2024-09-21T13:54:46.578185+0000","last_peered":"2024-09-21T13:54:46.578185+0000","last_clean":"2024-09-21T13:54:46.578185+0000","last_became_active":"2024-09-21T13:52:04.320441+0000","last_became_peered":"2024-09-21T13:52:04.320441+0000","last_unstale":"2024-09-21T13:54:46.578185+0000","last_undegraded":"2024-09-21T13:54:46.578185+0000","last_fullsized":"2024-09-21T13:54:46.578185+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'10","last_scrub_stamp":"2024-09-21T13:54:19.934756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:19.934756+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:29.532492+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"68'1","reported_seq":294,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.576313+0000","last_change":"2024-09-21T13:52:04.314151+0000","last_active":"2024-09-21T13:54:46.576313+0000","last_peered":"2024-09-21T13:54:46.576313+0000","last_clean":"2024-09-21T13:54:46.576313+0000","last_became_active":"2024-09-21T13:52:04.313210+0000","last_became_peered":"2024-09-21T13:52:04.313210+0000","last_unstale":"2024-09-21T13:54:46.576313+0000","last_undegraded":"2024-09-21T13:54:46.576313+0000","last_fullsized":"2024-09-21T13:54:46.576313+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T13:50:41.000201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:41.000201+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.7","version":"196'3669","reported_seq":3706,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:48.591315+0000","last_change":"2024-09-21T13:54:12.376676+0000","last_active":"2024-09-21T13:54:48.591315+0000","last_peered":"2024-09-21T13:54:48.591315+0000","last_clean":"2024-09-21T13:54:48.591315+0000","last_became_active":"2024-09-21T13:54:12.376302+0000","last_became_peered":"2024-09-21T13:54:12.376302+0000","last_unstale":"2024-09-21T13:54:48.591315+0000","last_undegraded":"2024-09-21T13:54:48.591315+0000","last_fullsized":"2024-09-21T13:54:48.591315+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":3669,"log_dups_size":0,"ondisk_log_size":3669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:33.120123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":240451584,"num_objects":3669,"num_object_clones":0,"num_object_copies":11007,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3669,"num_write_kb":234816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"68'9","reported_seq":312,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.585891+0000","last_change":"2024-09-21T13:53:18.740202+0000","last_active":"2024-09-21T13:54:46.585891+0000","last_peered":"2024-09-21T13:54:46.585891+0000","last_clean":"2024-09-21T13:54:46.585891+0000","last_became_active":"2024-09-21T13:49:00.053016+0000","last_became_peered":"2024-09-21T13:49:00.053016+0000","last_unstale":"2024-09-21T13:54:46.585891+0000","last_undegraded":"2024-09-21T13:54:46.585891+0000","last_fullsized":"2024-09-21T13:54:46.585891+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:53:18.740111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:18.740111+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:34.930684+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":292,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.579313+0000","last_change":"2024-09-21T13:50:38.445732+0000","last_active":"2024-09-21T13:54:46.579313+0000","last_peered":"2024-09-21T13:54:46.579313+0000","last_clean":"2024-09-21T13:54:46.579313+0000","last_became_active":"2024-09-21T13:48:58.049543+0000","last_became_peered":"2024-09-21T13:48:58.049543+0000","last_unstale":"2024-09-21T13:54:46.579313+0000","last_undegraded":"2024-09-21T13:54:46.579313+0000","last_fullsized":"2024-09-21T13:54:46.579313+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:38.445667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:38.445667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":221,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.576202+0000","last_change":"2024-09-21T13:54:21.403013+0000","last_active":"2024-09-21T13:54:46.576202+0000","last_peered":"2024-09-21T13:54:46.576202+0000","last_clean":"2024-09-21T13:54:46.576202+0000","last_became_active":"2024-09-21T13:52:04.317429+0000","last_became_peered":"2024-09-21T13:52:04.317429+0000","last_unstale":"2024-09-21T13:54:46.576202+0000","last_undegraded":"2024-09-21T13:54:46.576202+0000","last_fullsized":"2024-09-21T13:54:46.576202+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:21.402946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:21.402946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:26.710441+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":234,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.584769+0000","last_change":"2024-09-21T13:52:51.590860+0000","last_active":"2024-09-21T13:54:46.584769+0000","last_peered":"2024-09-21T13:54:46.584769+0000","last_clean":"2024-09-21T13:54:46.584769+0000","last_became_active":"2024-09-21T13:52:04.321703+0000","last_became_peered":"2024-09-21T13:52:04.321703+0000","last_unstale":"2024-09-21T13:54:46.584769+0000","last_undegraded":"2024-09-21T13:54:46.584769+0000","last_fullsized":"2024-09-21T13:54:46.584769+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:51.590800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:51.590800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.6","version":"196'3701","reported_seq":3738,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:48.376967+0000","last_change":"2024-09-21T13:54:12.478610+0000","last_active":"2024-09-21T13:54:48.376967+0000","last_peered":"2024-09-21T13:54:48.376967+0000","last_clean":"2024-09-21T13:54:48.376967+0000","last_became_active":"2024-09-21T13:54:12.478187+0000","last_became_peered":"2024-09-21T13:54:12.478187+0000","last_unstale":"2024-09-21T13:54:48.376967+0000","last_undegraded":"2024-09-21T13:54:48.376967+0000","last_fullsized":"2024-09-21T13:54:48.376967+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":3701,"log_dups_size":0,"ondisk_log_size":3701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:21.444672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":242548736,"num_objects":3701,"num_object_clones":0,"num_object_copies":11103,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3701,"num_write_kb":236864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"68'19","reported_seq":319,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.582194+0000","last_change":"2024-09-21T13:50:03.769686+0000","last_active":"2024-09-21T13:54:46.582194+0000","last_peered":"2024-09-21T13:54:46.582194+0000","last_clean":"2024-09-21T13:54:46.582194+0000","last_became_active":"2024-09-21T13:49:00.425955+0000","last_became_peered":"2024-09-21T13:49:00.425955+0000","last_unstale":"2024-09-21T13:54:46.582194+0000","last_undegraded":"2024-09-21T13:54:46.582194+0000","last_fullsized":"2024-09-21T13:54:46.582194+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-21T13:50:03.769613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:03.769613+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:37.036196+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":291,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.585710+0000","last_change":"2024-09-21T13:54:22.816889+0000","last_active":"2024-09-21T13:54:46.585710+0000","last_peered":"2024-09-21T13:54:46.585710+0000","last_clean":"2024-09-21T13:54:46.585710+0000","last_became_active":"2024-09-21T13:48:58.053641+0000","last_became_peered":"2024-09-21T13:48:58.053641+0000","last_unstale":"2024-09-21T13:54:46.585710+0000","last_undegraded":"2024-09-21T13:54:46.585710+0000","last_fullsized":"2024-09-21T13:54:46.585710+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:22.816815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:22.816815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:52.549192+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":241,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.576115+0000","last_change":"2024-09-21T13:53:53.244317+0000","last_active":"2024-09-21T13:54:46.576115+0000","last_peered":"2024-09-21T13:54:46.576115+0000","last_clean":"2024-09-21T13:54:46.576115+0000","last_became_active":"2024-09-21T13:52:04.321146+0000","last_became_peered":"2024-09-21T13:52:04.321146+0000","last_unstale":"2024-09-21T13:54:46.576115+0000","last_undegraded":"2024-09-21T13:54:46.576115+0000","last_fullsized":"2024-09-21T13:54:46.576115+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:53.244131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:53:53.244131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:13.809484+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":298,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.584743+0000","last_change":"2024-09-21T13:54:03.688017+0000","last_active":"2024-09-21T13:54:46.584743+0000","last_peered":"2024-09-21T13:54:46.584743+0000","last_clean":"2024-09-21T13:54:46.584743+0000","last_became_active":"2024-09-21T13:52:04.322213+0000","last_became_peered":"2024-09-21T13:52:04.322213+0000","last_unstale":"2024-09-21T13:54:46.584743+0000","last_undegraded":"2024-09-21T13:54:46.584743+0000","last_fullsized":"2024-09-21T13:54:46.584743+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:03.687971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:54:03.687971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:28.966237+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"196'3844","reported_seq":3881,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:48.427670+0000","last_change":"2024-09-21T13:54:12.478639+0000","last_active":"2024-09-21T13:54:48.427670+0000","last_peered":"2024-09-21T13:54:48.427670+0000","last_clean":"2024-09-21T13:54:48.427670+0000","last_became_active":"2024-09-21T13:54:12.478330+0000","last_became_peered":"2024-09-21T13:54:12.478330+0000","last_unstale":"2024-09-21T13:54:48.427670+0000","last_undegraded":"2024-09-21T13:54:48.427670+0000","last_fullsized":"2024-09-21T13:54:48.427670+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":3844,"log_dups_size":0,"ondisk_log_size":3844,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:36.845419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251920384,"num_objects":3844,"num_object_clones":0,"num_object_copies":11532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3844,"num_write_kb":246016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"68'12","reported_seq":318,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.575419+0000","last_change":"2024-09-21T13:53:57.288526+0000","last_active":"2024-09-21T13:54:46.575419+0000","last_peered":"2024-09-21T13:54:46.575419+0000","last_clean":"2024-09-21T13:54:46.575419+0000","last_became_active":"2024-09-21T13:49:00.059720+0000","last_became_peered":"2024-09-21T13:49:00.059720+0000","last_unstale":"2024-09-21T13:54:46.575419+0000","last_undegraded":"2024-09-21T13:54:46.575419+0000","last_fullsized":"2024-09-21T13:54:46.575419+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'12","last_scrub_stamp":"2024-09-21T13:53:57.288451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:57.288451+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:08.532403+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":291,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.586309+0000","last_change":"2024-09-21T13:50:29.100595+0000","last_active":"2024-09-21T13:54:46.586309+0000","last_peered":"2024-09-21T13:54:46.586309+0000","last_clean":"2024-09-21T13:54:46.586309+0000","last_became_active":"2024-09-21T13:48:58.057641+0000","last_became_peered":"2024-09-21T13:48:58.057641+0000","last_unstale":"2024-09-21T13:54:46.586309+0000","last_undegraded":"2024-09-21T13:54:46.586309+0000","last_fullsized":"2024-09-21T13:54:46.586309+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:29.100547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:29.100547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:35.899528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":206,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.575896+0000","last_change":"2024-09-21T13:53:31.964122+0000","last_active":"2024-09-21T13:54:46.575896+0000","last_peered":"2024-09-21T13:54:46.575896+0000","last_clean":"2024-09-21T13:54:46.575896+0000","last_became_active":"2024-09-21T13:52:04.322147+0000","last_became_peered":"2024-09-21T13:52:04.322147+0000","last_unstale":"2024-09-21T13:54:46.575896+0000","last_undegraded":"2024-09-21T13:54:46.575896+0000","last_fullsized":"2024-09-21T13:54:46.575896+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:31.958946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:53:31.958946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:51.369874+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":279,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.593200+0000","last_change":"2024-09-21T13:53:20.888048+0000","last_active":"2024-09-21T13:54:46.593200+0000","last_peered":"2024-09-21T13:54:46.593200+0000","last_clean":"2024-09-21T13:54:46.593200+0000","last_became_active":"2024-09-21T13:49:04.078089+0000","last_became_peered":"2024-09-21T13:49:04.078089+0000","last_unstale":"2024-09-21T13:54:46.593200+0000","last_undegraded":"2024-09-21T13:54:46.593200+0000","last_fullsized":"2024-09-21T13:54:46.593200+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:20.882449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:20.882449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"196'3789","reported_seq":3826,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:48.411187+0000","last_change":"2024-09-21T13:54:12.477912+0000","last_active":"2024-09-21T13:54:48.411187+0000","last_peered":"2024-09-21T13:54:48.411187+0000","last_clean":"2024-09-21T13:54:48.411187+0000","last_became_active":"2024-09-21T13:54:12.477455+0000","last_became_peered":"2024-09-21T13:54:12.477455+0000","last_unstale":"2024-09-21T13:54:48.411187+0000","last_undegraded":"2024-09-21T13:54:48.411187+0000","last_fullsized":"2024-09-21T13:54:48.411187+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":3789,"log_dups_size":0,"ondisk_log_size":3789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:26.894106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":248315904,"num_objects":3789,"num_object_clones":0,"num_object_copies":11367,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3789,"num_write_kb":242496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"68'15","reported_seq":321,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.580040+0000","last_change":"2024-09-21T13:53:31.204731+0000","last_active":"2024-09-21T13:54:46.580040+0000","last_peered":"2024-09-21T13:54:46.580040+0000","last_clean":"2024-09-21T13:54:46.580040+0000","last_became_active":"2024-09-21T13:49:00.425997+0000","last_became_peered":"2024-09-21T13:49:00.425997+0000","last_unstale":"2024-09-21T13:54:46.580040+0000","last_undegraded":"2024-09-21T13:54:46.580040+0000","last_fullsized":"2024-09-21T13:54:46.580040+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:53:31.204637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:31.204637+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":311,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.585563+0000","last_change":"2024-09-21T13:53:19.396344+0000","last_active":"2024-09-21T13:54:46.585563+0000","last_peered":"2024-09-21T13:54:46.585563+0000","last_clean":"2024-09-21T13:54:46.585563+0000","last_became_active":"2024-09-21T13:48:58.057632+0000","last_became_peered":"2024-09-21T13:48:58.057632+0000","last_unstale":"2024-09-21T13:54:46.585563+0000","last_undegraded":"2024-09-21T13:54:46.585563+0000","last_fullsized":"2024-09-21T13:54:46.585563+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T13:53:19.396277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:53:19.396277+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:47.867330+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"68'8","reported_seq":362,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.584946+0000","last_change":"2024-09-21T13:51:26.297643+0000","last_active":"2024-09-21T13:54:46.584946+0000","last_peered":"2024-09-21T13:54:46.584946+0000","last_clean":"2024-09-21T13:54:46.584946+0000","last_became_active":"2024-09-21T13:49:02.056167+0000","last_became_peered":"2024-09-21T13:49:02.056167+0000","last_unstale":"2024-09-21T13:54:46.584946+0000","last_undegraded":"2024-09-21T13:54:46.584946+0000","last_fullsized":"2024-09-21T13:54:46.584946+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:51:26.297550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:26.297550+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:33.879499+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":296,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.579635+0000","last_change":"2024-09-21T13:54:21.108175+0000","last_active":"2024-09-21T13:54:46.579635+0000","last_peered":"2024-09-21T13:54:46.579635+0000","last_clean":"2024-09-21T13:54:46.579635+0000","last_became_active":"2024-09-21T13:49:04.432546+0000","last_became_peered":"2024-09-21T13:49:04.432546+0000","last_unstale":"2024-09-21T13:54:46.579635+0000","last_undegraded":"2024-09-21T13:54:46.579635+0000","last_fullsized":"2024-09-21T13:54:46.579635+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:21.108123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:54:21.108123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:24.708631+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":308,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.581937+0000","last_change":"2024-09-21T13:54:18.131012+0000","last_active":"2024-09-21T13:54:46.581937+0000","last_peered":"2024-09-21T13:54:46.581937+0000","last_clean":"2024-09-21T13:54:46.581937+0000","last_became_active":"2024-09-21T13:48:58.056080+0000","last_became_peered":"2024-09-21T13:48:58.056080+0000","last_unstale":"2024-09-21T13:54:46.581937+0000","last_undegraded":"2024-09-21T13:54:46.581937+0000","last_fullsized":"2024-09-21T13:54:46.581937+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:18.130969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:18.130969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:34.242253+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"68'6","reported_seq":296,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.593744+0000","last_change":"2024-09-21T13:53:31.807215+0000","last_active":"2024-09-21T13:54:46.593744+0000","last_peered":"2024-09-21T13:54:46.593744+0000","last_clean":"2024-09-21T13:54:46.593744+0000","last_became_active":"2024-09-21T13:49:00.426829+0000","last_became_peered":"2024-09-21T13:49:00.426829+0000","last_unstale":"2024-09-21T13:54:46.593744+0000","last_undegraded":"2024-09-21T13:54:46.593744+0000","last_fullsized":"2024-09-21T13:54:46.593744+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'6","last_scrub_stamp":"2024-09-21T13:53:31.807028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:31.807028+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:49.612755+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":300,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.579447+0000","last_change":"2024-09-21T13:54:18.124591+0000","last_active":"2024-09-21T13:54:46.579447+0000","last_peered":"2024-09-21T13:54:46.579447+0000","last_clean":"2024-09-21T13:54:46.579447+0000","last_became_active":"2024-09-21T13:49:02.062231+0000","last_became_peered":"2024-09-21T13:49:02.062231+0000","last_unstale":"2024-09-21T13:54:46.579447+0000","last_undegraded":"2024-09-21T13:54:46.579447+0000","last_fullsized":"2024-09-21T13:54:46.579447+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:18.124544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:18.124544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:39.080321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":287,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.585435+0000","last_change":"2024-09-21T13:53:18.373418+0000","last_active":"2024-09-21T13:54:46.585435+0000","last_peered":"2024-09-21T13:54:46.585435+0000","last_clean":"2024-09-21T13:54:46.585435+0000","last_became_active":"2024-09-21T13:49:04.432507+0000","last_became_peered":"2024-09-21T13:49:04.432507+0000","last_unstale":"2024-09-21T13:54:46.585435+0000","last_undegraded":"2024-09-21T13:54:46.585435+0000","last_fullsized":"2024-09-21T13:54:46.585435+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:18.373356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:18.373356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"68'9","reported_seq":259,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.589549+0000","last_change":"2024-09-21T13:51:54.808758+0000","last_active":"2024-09-21T13:54:46.589549+0000","last_peered":"2024-09-21T13:54:46.589549+0000","last_clean":"2024-09-21T13:54:46.589549+0000","last_became_active":"2024-09-21T13:50:29.693589+0000","last_became_peered":"2024-09-21T13:50:29.693589+0000","last_unstale":"2024-09-21T13:54:46.589549+0000","last_undegraded":"2024-09-21T13:54:46.589549+0000","last_fullsized":"2024-09-21T13:54:46.589549+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:51:54.808559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:54.808559+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"61'1","reported_seq":297,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.585126+0000","last_change":"2024-09-21T13:54:18.838960+0000","last_active":"2024-09-21T13:54:46.585126+0000","last_peered":"2024-09-21T13:54:46.585126+0000","last_clean":"2024-09-21T13:54:46.585126+0000","last_became_active":"2024-09-21T13:48:58.050645+0000","last_became_peered":"2024-09-21T13:48:58.050645+0000","last_unstale":"2024-09-21T13:54:46.585126+0000","last_undegraded":"2024-09-21T13:54:46.585126+0000","last_fullsized":"2024-09-21T13:54:46.585126+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:54:18.838906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:18.838906+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:26.089820+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":291,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.593597+0000","last_change":"2024-09-21T13:54:28.016449+0000","last_active":"2024-09-21T13:54:46.593597+0000","last_peered":"2024-09-21T13:54:46.593597+0000","last_clean":"2024-09-21T13:54:46.593597+0000","last_became_active":"2024-09-21T13:49:02.061775+0000","last_became_peered":"2024-09-21T13:49:02.061775+0000","last_unstale":"2024-09-21T13:54:46.593597+0000","last_undegraded":"2024-09-21T13:54:46.593597+0000","last_fullsized":"2024-09-21T13:54:46.593597+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:28.014760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:28.014760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:36.849367+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":229,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.576181+0000","last_change":"2024-09-21T13:52:04.324521+0000","last_active":"2024-09-21T13:54:46.576181+0000","last_peered":"2024-09-21T13:54:46.576181+0000","last_clean":"2024-09-21T13:54:46.576181+0000","last_became_active":"2024-09-21T13:52:04.323549+0000","last_became_peered":"2024-09-21T13:52:04.323549+0000","last_unstale":"2024-09-21T13:54:46.576181+0000","last_undegraded":"2024-09-21T13:54:46.576181+0000","last_fullsized":"2024-09-21T13:54:46.576181+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:41.808746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:41.808746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:35.892198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"68'9","reported_seq":304,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.579797+0000","last_change":"2024-09-21T13:51:28.547052+0000","last_active":"2024-09-21T13:54:46.579797+0000","last_peered":"2024-09-21T13:54:46.579797+0000","last_clean":"2024-09-21T13:54:46.579797+0000","last_became_active":"2024-09-21T13:49:00.427766+0000","last_became_peered":"2024-09-21T13:49:00.427766+0000","last_unstale":"2024-09-21T13:54:46.579797+0000","last_undegraded":"2024-09-21T13:54:46.579797+0000","last_fullsized":"2024-09-21T13:54:46.579797+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:51:28.546958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:28.546958+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:09.056462+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":231,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.585032+0000","last_change":"2024-09-21T13:54:02.683225+0000","last_active":"2024-09-21T13:54:46.585032+0000","last_peered":"2024-09-21T13:54:46.585032+0000","last_clean":"2024-09-21T13:54:46.585032+0000","last_became_active":"2024-09-21T13:50:29.690634+0000","last_became_peered":"2024-09-21T13:50:29.690634+0000","last_unstale":"2024-09-21T13:54:46.585032+0000","last_undegraded":"2024-09-21T13:54:46.585032+0000","last_fullsized":"2024-09-21T13:54:46.585032+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:02.683172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:02.683172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:13.538040+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"132'10","reported_seq":383,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.589872+0000","last_change":"2024-09-21T13:54:20.946861+0000","last_active":"2024-09-21T13:54:46.589872+0000","last_peered":"2024-09-21T13:54:46.589872+0000","last_clean":"2024-09-21T13:54:46.589872+0000","last_became_active":"2024-09-21T13:52:04.318626+0000","last_became_peered":"2024-09-21T13:52:04.318626+0000","last_unstale":"2024-09-21T13:54:46.589872+0000","last_undegraded":"2024-09-21T13:54:46.589872+0000","last_fullsized":"2024-09-21T13:54:46.589872+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'10","last_scrub_stamp":"2024-09-21T13:54:20.946782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:20.946782+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:42.458501+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":298,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.577508+0000","last_change":"2024-09-21T13:53:51.285122+0000","last_active":"2024-09-21T13:54:46.577508+0000","last_peered":"2024-09-21T13:54:46.577508+0000","last_clean":"2024-09-21T13:54:46.577508+0000","last_became_active":"2024-09-21T13:52:04.315865+0000","last_became_peered":"2024-09-21T13:52:04.315865+0000","last_unstale":"2024-09-21T13:54:46.577508+0000","last_undegraded":"2024-09-21T13:54:46.577508+0000","last_fullsized":"2024-09-21T13:54:46.577508+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:51.285077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:51.285077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:16.987394+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"68'10","reported_seq":302,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.590085+0000","last_change":"2024-09-21T13:51:11.545546+0000","last_active":"2024-09-21T13:54:46.590085+0000","last_peered":"2024-09-21T13:54:46.590085+0000","last_clean":"2024-09-21T13:54:46.590085+0000","last_became_active":"2024-09-21T13:49:00.426071+0000","last_became_peered":"2024-09-21T13:49:00.426071+0000","last_unstale":"2024-09-21T13:54:46.590085+0000","last_undegraded":"2024-09-21T13:54:46.590085+0000","last_fullsized":"2024-09-21T13:54:46.590085+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:51:11.545458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:11.545458+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:42.030147+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":302,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.585145+0000","last_change":"2024-09-21T13:52:04.318335+0000","last_active":"2024-09-21T13:54:46.585145+0000","last_peered":"2024-09-21T13:54:46.585145+0000","last_clean":"2024-09-21T13:54:46.585145+0000","last_became_active":"2024-09-21T13:52:04.318006+0000","last_became_peered":"2024-09-21T13:52:04.318006+0000","last_unstale":"2024-09-21T13:54:46.585145+0000","last_undegraded":"2024-09-21T13:54:46.585145+0000","last_fullsized":"2024-09-21T13:54:46.585145+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:41.162315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:41.162315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:42.880310+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"68'8","reported_seq":363,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.579664+0000","last_change":"2024-09-21T13:54:03.065465+0000","last_active":"2024-09-21T13:54:46.579664+0000","last_peered":"2024-09-21T13:54:46.579664+0000","last_clean":"2024-09-21T13:54:46.579664+0000","last_became_active":"2024-09-21T13:49:02.067487+0000","last_became_peered":"2024-09-21T13:49:02.067487+0000","last_unstale":"2024-09-21T13:54:46.579664+0000","last_undegraded":"2024-09-21T13:54:46.579664+0000","last_fullsized":"2024-09-21T13:54:46.579664+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:54:03.065395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:03.065395+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:30.943315+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":231,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.585834+0000","last_change":"2024-09-21T13:54:27.842220+0000","last_active":"2024-09-21T13:54:46.585834+0000","last_peered":"2024-09-21T13:54:46.585834+0000","last_clean":"2024-09-21T13:54:46.585834+0000","last_became_active":"2024-09-21T13:50:29.695082+0000","last_became_peered":"2024-09-21T13:50:29.695082+0000","last_unstale":"2024-09-21T13:54:46.585834+0000","last_undegraded":"2024-09-21T13:54:46.585834+0000","last_fullsized":"2024-09-21T13:54:46.585834+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:27.842150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:54:27.842150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:35.545905+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"68'11","reported_seq":314,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.577064+0000","last_change":"2024-09-21T13:53:26.146937+0000","last_active":"2024-09-21T13:54:46.577064+0000","last_peered":"2024-09-21T13:54:46.577064+0000","last_clean":"2024-09-21T13:54:46.577064+0000","last_became_active":"2024-09-21T13:49:00.055024+0000","last_became_peered":"2024-09-21T13:49:00.055024+0000","last_unstale":"2024-09-21T13:54:46.577064+0000","last_undegraded":"2024-09-21T13:54:46.577064+0000","last_fullsized":"2024-09-21T13:54:46.577064+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:53:26.146725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:26.146725+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":46,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.586608+0000","last_change":"2024-09-21T13:54:04.377034+0000","last_active":"2024-09-21T13:54:46.586608+0000","last_peered":"2024-09-21T13:54:46.586608+0000","last_clean":"2024-09-21T13:54:46.586608+0000","last_became_active":"2024-09-21T13:54:04.376614+0000","last_became_peered":"2024-09-21T13:54:04.376614+0000","last_unstale":"2024-09-21T13:54:46.586608+0000","last_undegraded":"2024-09-21T13:54:46.586608+0000","last_fullsized":"2024-09-21T13:54:46.586608+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:01.228213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:01.228213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:14.625514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":300,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.581522+0000","last_change":"2024-09-21T13:53:29.868061+0000","last_active":"2024-09-21T13:54:46.581522+0000","last_peered":"2024-09-21T13:54:46.581522+0000","last_clean":"2024-09-21T13:54:46.581522+0000","last_became_active":"2024-09-21T13:49:02.062868+0000","last_became_peered":"2024-09-21T13:49:02.062868+0000","last_unstale":"2024-09-21T13:54:46.581522+0000","last_undegraded":"2024-09-21T13:54:46.581522+0000","last_fullsized":"2024-09-21T13:54:46.581522+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:29.867278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:53:29.867278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:49.405014+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":279,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.588033+0000","last_change":"2024-09-21T13:51:24.549993+0000","last_active":"2024-09-21T13:54:46.588033+0000","last_peered":"2024-09-21T13:54:46.588033+0000","last_clean":"2024-09-21T13:54:46.588033+0000","last_became_active":"2024-09-21T13:49:04.082961+0000","last_became_peered":"2024-09-21T13:49:04.082961+0000","last_unstale":"2024-09-21T13:54:46.588033+0000","last_undegraded":"2024-09-21T13:54:46.588033+0000","last_fullsized":"2024-09-21T13:54:46.588033+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:24.549706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:51:24.549706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"68'9","reported_seq":315,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.581735+0000","last_change":"2024-09-21T13:52:10.447789+0000","last_active":"2024-09-21T13:54:46.581735+0000","last_peered":"2024-09-21T13:54:46.581735+0000","last_clean":"2024-09-21T13:54:46.581735+0000","last_became_active":"2024-09-21T13:52:04.320149+0000","last_became_peered":"2024-09-21T13:52:04.320149+0000","last_unstale":"2024-09-21T13:54:46.581735+0000","last_undegraded":"2024-09-21T13:54:46.581735+0000","last_fullsized":"2024-09-21T13:54:46.581735+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:52:10.447530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:52:10.447530+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:28.845521+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":291,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.585630+0000","last_change":"2024-09-21T13:54:06.687343+0000","last_active":"2024-09-21T13:54:46.585630+0000","last_peered":"2024-09-21T13:54:46.585630+0000","last_clean":"2024-09-21T13:54:46.585630+0000","last_became_active":"2024-09-21T13:48:58.057516+0000","last_became_peered":"2024-09-21T13:48:58.057516+0000","last_unstale":"2024-09-21T13:54:46.585630+0000","last_undegraded":"2024-09-21T13:54:46.585630+0000","last_fullsized":"2024-09-21T13:54:46.585630+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:06.687302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:06.687302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:28.415240+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":291,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.593730+0000","last_change":"2024-09-21T13:54:19.999192+0000","last_active":"2024-09-21T13:54:46.593730+0000","last_peered":"2024-09-21T13:54:46.593730+0000","last_clean":"2024-09-21T13:54:46.593730+0000","last_became_active":"2024-09-21T13:49:02.071922+0000","last_became_peered":"2024-09-21T13:49:02.071922+0000","last_unstale":"2024-09-21T13:54:46.593730+0000","last_undegraded":"2024-09-21T13:54:46.593730+0000","last_fullsized":"2024-09-21T13:54:46.593730+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:19.999125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:19.999125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:38.368141+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":216,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.579950+0000","last_change":"2024-09-21T13:50:29.689537+0000","last_active":"2024-09-21T13:54:46.579950+0000","last_peered":"2024-09-21T13:54:46.579950+0000","last_clean":"2024-09-21T13:54:46.579950+0000","last_became_active":"2024-09-21T13:50:29.689045+0000","last_became_peered":"2024-09-21T13:50:29.689045+0000","last_unstale":"2024-09-21T13:54:46.579950+0000","last_undegraded":"2024-09-21T13:54:46.579950+0000","last_fullsized":"2024-09-21T13:54:46.579950+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:14.052362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"68'11","reported_seq":314,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.588233+0000","last_change":"2024-09-21T13:54:22.978354+0000","last_active":"2024-09-21T13:54:46.588233+0000","last_peered":"2024-09-21T13:54:46.588233+0000","last_clean":"2024-09-21T13:54:46.588233+0000","last_became_active":"2024-09-21T13:49:00.426091+0000","last_became_peered":"2024-09-21T13:49:00.426091+0000","last_unstale":"2024-09-21T13:54:46.588233+0000","last_undegraded":"2024-09-21T13:54:46.588233+0000","last_fullsized":"2024-09-21T13:54:46.588233+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:54:22.978267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:22.978267+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:40.972799+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":284,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.579388+0000","last_change":"2024-09-21T13:48:58.052281+0000","last_active":"2024-09-21T13:54:46.579388+0000","last_peered":"2024-09-21T13:54:46.579388+0000","last_clean":"2024-09-21T13:54:46.579388+0000","last_became_active":"2024-09-21T13:48:58.051974+0000","last_became_peered":"2024-09-21T13:48:58.051974+0000","last_unstale":"2024-09-21T13:54:46.579388+0000","last_undegraded":"2024-09-21T13:54:46.579388+0000","last_fullsized":"2024-09-21T13:54:46.579388+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:08.067332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":283,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.584372+0000","last_change":"2024-09-21T13:50:37.118637+0000","last_active":"2024-09-21T13:54:46.584372+0000","last_peered":"2024-09-21T13:54:46.584372+0000","last_clean":"2024-09-21T13:54:46.584372+0000","last_became_active":"2024-09-21T13:49:02.056006+0000","last_became_peered":"2024-09-21T13:49:02.056006+0000","last_unstale":"2024-09-21T13:54:46.584372+0000","last_undegraded":"2024-09-21T13:54:46.584372+0000","last_fullsized":"2024-09-21T13:54:46.584372+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:37.118594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:37.118594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:12.668029+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":279,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.593867+0000","last_change":"2024-09-21T13:51:55.827109+0000","last_active":"2024-09-21T13:54:46.593867+0000","last_peered":"2024-09-21T13:54:46.593867+0000","last_clean":"2024-09-21T13:54:46.593867+0000","last_became_active":"2024-09-21T13:49:04.434034+0000","last_became_peered":"2024-09-21T13:49:04.434034+0000","last_unstale":"2024-09-21T13:54:46.593867+0000","last_undegraded":"2024-09-21T13:54:46.593867+0000","last_fullsized":"2024-09-21T13:54:46.593867+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:55.826922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:51:55.826922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"68'4","reported_seq":293,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.590534+0000","last_change":"2024-09-21T13:53:27.826804+0000","last_active":"2024-09-21T13:54:46.590534+0000","last_peered":"2024-09-21T13:54:46.590534+0000","last_clean":"2024-09-21T13:54:46.590534+0000","last_became_active":"2024-09-21T13:49:00.063066+0000","last_became_peered":"2024-09-21T13:49:00.063066+0000","last_unstale":"2024-09-21T13:54:46.590534+0000","last_undegraded":"2024-09-21T13:54:46.590534+0000","last_fullsized":"2024-09-21T13:54:46.590534+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4","last_scrub_stamp":"2024-09-21T13:53:27.826729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:27.826729+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:54.701343+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":224,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.579501+0000","last_change":"2024-09-21T13:53:32.253481+0000","last_active":"2024-09-21T13:54:46.579501+0000","last_peered":"2024-09-21T13:54:46.579501+0000","last_clean":"2024-09-21T13:54:46.579501+0000","last_became_active":"2024-09-21T13:50:29.690269+0000","last_became_peered":"2024-09-21T13:50:29.690269+0000","last_unstale":"2024-09-21T13:54:46.579501+0000","last_undegraded":"2024-09-21T13:54:46.579501+0000","last_fullsized":"2024-09-21T13:54:46.579501+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:32.253408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:53:32.253408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:51.887076+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":300,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.582602+0000","last_change":"2024-09-21T13:53:51.254522+0000","last_active":"2024-09-21T13:54:46.582602+0000","last_peered":"2024-09-21T13:54:46.582602+0000","last_clean":"2024-09-21T13:54:46.582602+0000","last_became_active":"2024-09-21T13:49:02.064789+0000","last_became_peered":"2024-09-21T13:49:02.064789+0000","last_unstale":"2024-09-21T13:54:46.582602+0000","last_undegraded":"2024-09-21T13:54:46.582602+0000","last_fullsized":"2024-09-21T13:54:46.582602+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:51.254456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:53:51.254456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:51.405316+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":290,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.584580+0000","last_change":"2024-09-21T13:52:04.322055+0000","last_active":"2024-09-21T13:54:46.584580+0000","last_peered":"2024-09-21T13:54:46.584580+0000","last_clean":"2024-09-21T13:54:46.584580+0000","last_became_active":"2024-09-21T13:52:04.321667+0000","last_became_peered":"2024-09-21T13:52:04.321667+0000","last_unstale":"2024-09-21T13:54:46.584580+0000","last_undegraded":"2024-09-21T13:54:46.584580+0000","last_fullsized":"2024-09-21T13:54:46.584580+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:39.151320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:39.151320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:07.711986+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":288,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.580893+0000","last_change":"2024-09-21T13:52:27.409588+0000","last_active":"2024-09-21T13:54:46.580893+0000","last_peered":"2024-09-21T13:54:46.580893+0000","last_clean":"2024-09-21T13:54:46.580893+0000","last_became_active":"2024-09-21T13:49:04.075562+0000","last_became_peered":"2024-09-21T13:49:04.075562+0000","last_unstale":"2024-09-21T13:54:46.580893+0000","last_undegraded":"2024-09-21T13:54:46.580893+0000","last_fullsized":"2024-09-21T13:54:46.580893+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:27.409408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:27.409408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:39.987239+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":291,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.590777+0000","last_change":"2024-09-21T13:53:25.873998+0000","last_active":"2024-09-21T13:54:46.590777+0000","last_peered":"2024-09-21T13:54:46.590777+0000","last_clean":"2024-09-21T13:54:46.590777+0000","last_became_active":"2024-09-21T13:48:58.055497+0000","last_became_peered":"2024-09-21T13:48:58.055497+0000","last_unstale":"2024-09-21T13:54:46.590777+0000","last_undegraded":"2024-09-21T13:54:46.590777+0000","last_fullsized":"2024-09-21T13:54:46.590777+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:25.865470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:53:25.865470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:54:51.548858+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"68'11","reported_seq":337,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.582929+0000","last_change":"2024-09-21T13:52:04.321178+0000","last_active":"2024-09-21T13:54:46.582929+0000","last_peered":"2024-09-21T13:54:46.582929+0000","last_clean":"2024-09-21T13:54:46.582929+0000","last_became_active":"2024-09-21T13:52:04.320647+0000","last_became_peered":"2024-09-21T13:52:04.320647+0000","last_unstale":"2024-09-21T13:54:46.582929+0000","last_undegraded":"2024-09-21T13:54:46.582929+0000","last_fullsized":"2024-09-21T13:54:46.582929+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:50:35.790090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:35.790090+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:11.179000+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":243,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-09-21T13:54:46.585721+0000","last_change":"2024-09-21T13:54:01.727728+0000","last_active":"2024-09-21T13:54:46.585721+0000","last_peered":"2024-09-21T13:54:46.585721+0000","last_clean":"2024-09-21T13:54:46.585721+0000","last_became_active":"2024-09-21T13:52:04.322880+0000","last_became_peered":"2024-09-21T13:52:04.322880+0000","last_unstale":"2024-09-21T13:54:46.585721+0000","last_undegraded":"2024-09-21T13:54:46.585721+0000","last_fullsized":"2024-09-21T13:54:46.585721+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:01.727680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:01.727680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:16.479686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":3942907904,"num_objects":60164,"num_object_clones":0,"num_object_copies":180492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60164,"num_write_kb":3850496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11775967232,"data_stored":11775967232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60164,"ondisk_log_size":60164,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69,"ondisk_log_size":69,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":694,"num_read_kb":451,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":395,"ondisk_log_size":395,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":345,"num_read_kb":340,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234059,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1352556,"kb_used_data":1186716,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":92392596,"statfs":{"total":95995035648,"available":94610018304,"internally_reserved":0,"allocated":1215197184,"data_stored":1214658046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":169802186},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462992,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1334304,"kb_used_data":1190664,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":92410848,"statfs":{"total":95995035648,"available":94628708352,"internally_reserved":0,"allocated":1219239936,"data_stored":1218720193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":147061190},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724631,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2051928,"kb_used_data":1898568,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":91693224,"statfs":{"total":95995035648,"available":93893861376,"internally_reserved":0,"allocated":1944133632,"data_stored":1943614261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":157022661},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920861,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1316476,"kb_used_data":1168676,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":92428676,"statfs":{"total":95995035648,"available":94646964224,"internally_reserved":0,"allocated":1196724224,"data_stored":1196175883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117090,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1745720,"kb_used_data":1457704,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":91999432,"statfs":{"total":95995035648,"available":94207418368,"internally_reserved":0,"allocated":1492688896,"data_stored":1492137289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":294910407},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,5,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":130,"seq":558345748514,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1602912,"kb_used_data":1372172,"kb_used_omap":4,"kb_used_meta":230715,"kb_avail":92142240,"statfs":{"total":95995035648,"available":94353653760,"internally_reserved":0,"allocated":1405104128,"data_stored":1404580683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4441,"internal_metadata":236252839},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574959,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1828492,"kb_used_data":1644604,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":91916660,"statfs":{"total":95995035648,"available":94122659840,"internally_reserved":0,"allocated":1684074496,"data_stored":1683502103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188283333},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738485,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1682412,"kb_used_data":1412444,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":92062740,"statfs":{"total":95995035648,"available":94272245760,"internally_reserved":0,"allocated":1446342656,"data_stored":1445802791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":276429253},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1477640192,"data_stored":1477640192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1707999232,"data_stored":1707999232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1430388736,"data_stored":1430388736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1492975616,"data_stored":1492975616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1230569472,"data_stored":1230569472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1968046080,"data_stored":1968046080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1232273408,"data_stored":1232273408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1236074496,"data_stored":1236074496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T13:54:51.447 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd pool set .mgr pgp_num 1 2024-09-21T13:54:51.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:51.529 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:51.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:54:51.629 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:51.638 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:51 smithi049 ceph-mon[30080]: pgmap v286: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 124 MiB/s wr, 1.99k op/s 2024-09-21T13:54:51.638 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:51 smithi049 ceph-mon[30080]: from='client.25072 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T13:54:51.638 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:51 smithi049 ceph-mon[40489]: pgmap v286: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 124 MiB/s wr, 1.99k op/s 2024-09-21T13:54:51.638 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:51 smithi049 ceph-mon[40489]: from='client.25072 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T13:54:51.700 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:51.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:51.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:51.802 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:51.833 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:51.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:51.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:51.935 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:54:51.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:51 smithi187 ceph-mon[33032]: pgmap v286: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 124 MiB/s wr, 1.99k op/s 2024-09-21T13:54:51.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:51 smithi187 ceph-mon[33032]: from='client.25072 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T13:54:51.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:51.970 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:51.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:54:52.071 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:52.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:52.114 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:52.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:52.215 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:54:52.255 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:52.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:52.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:54:52.292 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 34 16 65097 65081 119.608 77.0625 0.0137943 0.00835599 2024-09-21T13:54:52.356 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:54:52.402 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:52.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:52.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:54:52.501 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T13:54:52.501 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:54:52.504 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:52.533 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:52.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:52.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:52.619 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-09-21T13:54:52.634 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:52.662 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:52.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:52.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:52.725 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:52 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/403153377' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:54:52.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:52 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/403153377' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:54:52.764 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:54:52.791 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:52.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:52.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:54:52.893 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:54:52.930 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:52.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:52.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:54:52.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:52 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/403153377' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:54:52.983 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:54:52.999 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:54:53.031 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:54:53.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:53.074 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:53.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:54:53.175 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:54:53.224 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:53.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:53.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:54:53.292 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 35 15 66472 66457 118.648 86 0.00347859 0.00842334 2024-09-21T13:54:53.326 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:53.363 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:53.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:53.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:53.464 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:54:53.501 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:53.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:53.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:54:53.603 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:54:53.651 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T13:54:53.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:53.652 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:53.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:54:53.753 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:54:53.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:53.814 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:53.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:54:53.915 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:54:53.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:53 smithi187 ceph-mon[33032]: pgmap v287: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-09-21T13:54:53.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:53 smithi187 ceph-mon[33032]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:54:53.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:53 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/403153377' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T13:54:53.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:53 smithi187 ceph-mon[33032]: osdmap e197: 8 total, 8 up, 8 in 2024-09-21T13:54:53.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:53 smithi187 ceph-mon[33032]: 5.4 scrub starts 2024-09-21T13:54:53.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:53 smithi187 ceph-mon[33032]: 5.4 scrub ok 2024-09-21T13:54:53.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:53.986 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:53.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:54:54.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:53 smithi049 ceph-mon[40489]: pgmap v287: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-09-21T13:54:54.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:53 smithi049 ceph-mon[40489]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:54:54.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:53 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/403153377' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T13:54:54.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:53 smithi049 ceph-mon[40489]: osdmap e197: 8 total, 8 up, 8 in 2024-09-21T13:54:54.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:53 smithi049 ceph-mon[40489]: 5.4 scrub starts 2024-09-21T13:54:54.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:53 smithi049 ceph-mon[40489]: 5.4 scrub ok 2024-09-21T13:54:54.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:53 smithi049 ceph-mon[30080]: pgmap v287: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-09-21T13:54:54.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:53 smithi049 ceph-mon[30080]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:54:54.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:53 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/403153377' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T13:54:54.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:53 smithi049 ceph-mon[30080]: osdmap e197: 8 total, 8 up, 8 in 2024-09-21T13:54:54.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:53 smithi049 ceph-mon[30080]: 5.4 scrub starts 2024-09-21T13:54:54.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:53 smithi049 ceph-mon[30080]: 5.4 scrub ok 2024-09-21T13:54:54.087 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:54.118 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:54.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:54.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:54.220 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:54:54.250 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:54.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:54.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:54:54.292 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 36 16 67840 67824 117.725 85.4375 0.0145877 0.00848922 2024-09-21T13:54:54.352 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:54:54.388 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:54.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:54.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:54:54.490 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:54:54.524 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:54.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:54.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:54:54.625 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:54.660 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:54.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:54.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:54.763 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:54:54.796 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:54.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:54.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:54:54.898 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:54:54.928 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:54.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:54.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:54:54.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:54 smithi187 ceph-mon[33032]: 3.c scrub starts 2024-09-21T13:54:54.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:54 smithi187 ceph-mon[33032]: 3.c scrub ok 2024-09-21T13:54:54.954 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:54 smithi049 ceph-mon[40489]: 3.c scrub starts 2024-09-21T13:54:54.954 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:54 smithi049 ceph-mon[40489]: 3.c scrub ok 2024-09-21T13:54:54.954 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:54 smithi049 ceph-mon[30080]: 3.c scrub starts 2024-09-21T13:54:54.955 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:54 smithi049 ceph-mon[30080]: 3.c scrub ok 2024-09-21T13:54:55.030 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:54:55.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:55.067 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:55.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:54:55.168 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:54:55.197 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:55.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:55.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:54:55.292 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 37 16 69267 69251 116.954 89.1875 0.0133776 0.00854523 2024-09-21T13:54:55.299 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:55.327 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:55.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:55.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:55.429 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:54:55.459 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:55.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:55.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:54:55.561 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:55.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:55.591 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:55.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:55.692 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:54:55.721 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:55.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:55.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:54:55.822 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:55.862 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:55.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:55.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:55.868 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:55 smithi049 ceph-mon[40489]: pgmap v289: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.63k op/s 2024-09-21T13:54:55.868 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:55 smithi049 ceph-mon[40489]: 6.6 scrub starts 2024-09-21T13:54:55.868 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:55 smithi049 ceph-mon[40489]: 6.6 scrub ok 2024-09-21T13:54:55.868 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:55 smithi049 ceph-mon[40489]: osdmap e198: 8 total, 8 up, 8 in 2024-09-21T13:54:55.873 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:55 smithi049 ceph-mon[30080]: pgmap v289: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.63k op/s 2024-09-21T13:54:55.873 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:55 smithi049 ceph-mon[30080]: 6.6 scrub starts 2024-09-21T13:54:55.873 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:55 smithi049 ceph-mon[30080]: 6.6 scrub ok 2024-09-21T13:54:55.873 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:55 smithi049 ceph-mon[30080]: osdmap e198: 8 total, 8 up, 8 in 2024-09-21T13:54:55.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:55 smithi187 ceph-mon[33032]: pgmap v289: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.63k op/s 2024-09-21T13:54:55.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:55 smithi187 ceph-mon[33032]: 6.6 scrub starts 2024-09-21T13:54:55.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:55 smithi187 ceph-mon[33032]: 6.6 scrub ok 2024-09-21T13:54:55.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:55 smithi187 ceph-mon[33032]: osdmap e198: 8 total, 8 up, 8 in 2024-09-21T13:54:55.949 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:54:55.949 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:54:55.964 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:54:55.987 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:54:55.987 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:54:55.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:55.993 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:55.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:54:56.021 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:54:56.022 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:54:56.061 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:54:56.061 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:54:56.093 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:56.104 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:54:56.104 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:54:56.122 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:56.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:56.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:56.136 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:54:56.148 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:54:56.148 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:54:56.183 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:54:56.184 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:54:56.199 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T13:54:56.224 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:56.225 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:54:56.225 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:54:56.294 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 38 16 70549 70533 115.985 80.125 0.00706695 0.00861525 2024-09-21T13:54:56.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:56.295 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:56.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:56.396 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:54:56.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:56.455 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:56.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:54:56.556 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:54:56.604 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:56.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:56.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:54:56.641 INFO:teuthology.orchestra.run.smithi049.stderr:set pool 1 pgp_num to 1 2024-09-21T13:54:56.706 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:56.737 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:56.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:56.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:56.839 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:56.871 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:56.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:56.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:56.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:56 smithi187 ceph-mon[33032]: 6.9 scrub starts 2024-09-21T13:54:56.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:56 smithi187 ceph-mon[33032]: 6.9 scrub ok 2024-09-21T13:54:56.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:56 smithi187 ceph-mon[33032]: 3.1f scrub starts 2024-09-21T13:54:56.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:56 smithi187 ceph-mon[33032]: 3.1f scrub ok 2024-09-21T13:54:56.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:56 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3563619947' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:54:56.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:56 smithi187 ceph-mon[33032]: 5.8 scrub starts 2024-09-21T13:54:56.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:56 smithi187 ceph-mon[33032]: 5.8 scrub ok 2024-09-21T13:54:56.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:56 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1847741787' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-21T13:54:56.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:56 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-21T13:54:56.974 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:54:57.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:56 smithi049 ceph-mon[30080]: 6.9 scrub starts 2024-09-21T13:54:57.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:56 smithi049 ceph-mon[30080]: 6.9 scrub ok 2024-09-21T13:54:57.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:56 smithi049 ceph-mon[30080]: 3.1f scrub starts 2024-09-21T13:54:57.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:56 smithi049 ceph-mon[30080]: 3.1f scrub ok 2024-09-21T13:54:57.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:56 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3563619947' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:54:57.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:56 smithi049 ceph-mon[30080]: 5.8 scrub starts 2024-09-21T13:54:57.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:56 smithi049 ceph-mon[30080]: 5.8 scrub ok 2024-09-21T13:54:57.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:56 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1847741787' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-21T13:54:57.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:56 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-21T13:54:57.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:56 smithi049 ceph-mon[40489]: 6.9 scrub starts 2024-09-21T13:54:57.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:56 smithi049 ceph-mon[40489]: 6.9 scrub ok 2024-09-21T13:54:57.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:56 smithi049 ceph-mon[40489]: 3.1f scrub starts 2024-09-21T13:54:57.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:56 smithi049 ceph-mon[40489]: 3.1f scrub ok 2024-09-21T13:54:57.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:56 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3563619947' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:54:57.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:56 smithi049 ceph-mon[40489]: 5.8 scrub starts 2024-09-21T13:54:57.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:56 smithi049 ceph-mon[40489]: 5.8 scrub ok 2024-09-21T13:54:57.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:56 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1847741787' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-21T13:54:57.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:56 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-09-21T13:54:57.011 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:57.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:57.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:54:57.113 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:57.146 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:57.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:57.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:57.247 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:54:57.280 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:57.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:57.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:54:57.292 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 39 15 71790 71775 115.001 77.625 0.0210417 0.00869052 2024-09-21T13:54:57.382 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:54:57.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:57.420 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:57.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:54:57.520 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:54:57.557 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:57.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:57.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:54:57.631 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:57 smithi049 ceph-mon[30080]: pgmap v291: 148 pgs: 148 active+clean; 4.1 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 112 MiB/s wr, 1.80k op/s 2024-09-21T13:54:57.658 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:57.693 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:57.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:57.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:57.795 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:54:57.842 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:57.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:57.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:54:57.943 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:57.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:57 smithi187 ceph-mon[33032]: pgmap v291: 148 pgs: 148 active+clean; 4.1 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 112 MiB/s wr, 1.80k op/s 2024-09-21T13:54:57.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:57 smithi187 ceph-mon[33032]: 4.13 scrub starts 2024-09-21T13:54:57.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:57 smithi187 ceph-mon[33032]: 4.13 scrub ok 2024-09-21T13:54:57.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:57 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-09-21T13:54:57.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:57 smithi187 ceph-mon[33032]: osdmap e199: 8 total, 8 up, 8 in 2024-09-21T13:54:57.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:57 smithi187 ceph-mon[33032]: 1.0 scrub starts 2024-09-21T13:54:57.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:57 smithi187 ceph-mon[33032]: 1.0 scrub ok 2024-09-21T13:54:57.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:57.983 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:57.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:58.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:57 smithi049 ceph-mon[40489]: pgmap v291: 148 pgs: 148 active+clean; 4.1 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 112 MiB/s wr, 1.80k op/s 2024-09-21T13:54:58.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:57 smithi049 ceph-mon[40489]: 4.13 scrub starts 2024-09-21T13:54:58.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:57 smithi049 ceph-mon[40489]: 4.13 scrub ok 2024-09-21T13:54:58.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:57 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-09-21T13:54:58.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:57 smithi049 ceph-mon[40489]: osdmap e199: 8 total, 8 up, 8 in 2024-09-21T13:54:58.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:57 smithi049 ceph-mon[40489]: 1.0 scrub starts 2024-09-21T13:54:58.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:57 smithi049 ceph-mon[40489]: 1.0 scrub ok 2024-09-21T13:54:58.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:57 smithi049 ceph-mon[30080]: 4.13 scrub starts 2024-09-21T13:54:58.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:57 smithi049 ceph-mon[30080]: 4.13 scrub ok 2024-09-21T13:54:58.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:57 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-09-21T13:54:58.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:57 smithi049 ceph-mon[30080]: osdmap e199: 8 total, 8 up, 8 in 2024-09-21T13:54:58.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:57 smithi049 ceph-mon[30080]: 1.0 scrub starts 2024-09-21T13:54:58.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:57 smithi049 ceph-mon[30080]: 1.0 scrub ok 2024-09-21T13:54:58.084 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:54:58.116 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T13:54:58.117 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-09-21T13:54:58.117 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:58.117 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:71956 8.7 8:e05b3ce8:::benchmark_data_smithi049_109311_object71955:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-09-21T13:54:58.117 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:54:57.427283+0000", 2024-09-21T13:54:58.117 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.036816686000000001, 2024-09-21T13:54:58.117 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.036844399, 2024-09-21T13:54:58.117 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:54:58.117 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:54:58.117 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "waiting for sub ops", 2024-09-21T13:54:58.117 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:54:58.118 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:54:58.118 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:54:58.118 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 71956 2024-09-21T13:54:58.118 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:58.118 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:54:58.118 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:58.118 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:54:58.118 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:57.427283+0000", 2024-09-21T13:54:58.118 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:58.118 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:58.118 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:58.118 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:54:58.118 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:57.427283+0000", 2024-09-21T13:54:58.118 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:58.119 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:58.119 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:58.119 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:54:58.119 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:57.427284+0000", 2024-09-21T13:54:58.119 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.891e-06 2024-09-21T13:54:58.119 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:58.119 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:58.119 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:54:58.119 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:57.427307+0000", 2024-09-21T13:54:58.119 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2891000000000002e-05 2024-09-21T13:54:58.119 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:58.119 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:58.119 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:54:58.120 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:57.427320+0000", 2024-09-21T13:54:58.120 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2432999999999999e-05 2024-09-21T13:54:58.120 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:58.120 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:58.120 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:54:58.120 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:57.427327+0000", 2024-09-21T13:54:58.120 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2960000000000001e-06 2024-09-21T13:54:58.120 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:58.120 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:58.120 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:58.120 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:57.427825+0000", 2024-09-21T13:54:58.120 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00049787500000000005 2024-09-21T13:54:58.120 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:58.120 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:58.121 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:54:58.121 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:57.427972+0000", 2024-09-21T13:54:58.121 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014731299999999999 2024-09-21T13:54:58.121 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:58.121 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:58.121 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,6", 2024-09-21T13:54:58.121 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:57.428113+0000", 2024-09-21T13:54:58.121 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014040999999999999 2024-09-21T13:54:58.121 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:58.121 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:58.121 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:54:58.121 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:57.430889+0000", 2024-09-21T13:54:58.121 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0027761790000000001 2024-09-21T13:54:58.121 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:58.121 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:58.122 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:58.122 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:57.431325+0000", 2024-09-21T13:54:58.122 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00043652799999999999 2024-09-21T13:54:58.122 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:58.122 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:54:58.122 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:58.122 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:58.122 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:58.122 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:71966 8.3 8:c7064237:::benchmark_data_smithi049_109311_object71965:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-09-21T13:54:58.122 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:54:57.434890+0000", 2024-09-21T13:54:58.122 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.029208854999999999, 2024-09-21T13:54:58.122 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.029285276999999998, 2024-09-21T13:54:58.122 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:54:58.123 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:54:58.123 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:54:58.123 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:54:58.123 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:54:58.123 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:54:58.123 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 71966 2024-09-21T13:54:58.123 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:58.123 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:54:58.123 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:58.123 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:54:58.123 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:57.434890+0000", 2024-09-21T13:54:58.123 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:58.123 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:58.123 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:58.124 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:54:58.124 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:57.434890+0000", 2024-09-21T13:54:58.124 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:58.124 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:58.124 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:58.124 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:54:58.124 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:57.434891+0000", 2024-09-21T13:54:58.124 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2399999999999996e-07 2024-09-21T13:54:58.124 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:58.124 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:58.124 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:54:58.124 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:57.434917+0000", 2024-09-21T13:54:58.124 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.561e-05 2024-09-21T13:54:58.124 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:58.125 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:58.125 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:54:58.125 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:57.434935+0000", 2024-09-21T13:54:58.125 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8238999999999999e-05 2024-09-21T13:54:58.125 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:58.125 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:58.125 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:54:58.125 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:57.434943+0000", 2024-09-21T13:54:58.125 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5290000000000003e-06 2024-09-21T13:54:58.125 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:58.125 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:58.125 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:58.125 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:57.439310+0000", 2024-09-21T13:54:58.125 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0043674389999999999 2024-09-21T13:54:58.126 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:58.126 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:58.126 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:54:58.126 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:57.439441+0000", 2024-09-21T13:54:58.126 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013075599999999999 2024-09-21T13:54:58.126 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:58.126 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:58.126 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,6", 2024-09-21T13:54:58.126 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:57.439663+0000", 2024-09-21T13:54:58.126 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00022171300000000001 2024-09-21T13:54:58.126 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:58.127 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:58.127 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:58.127 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:57.442098+0000", 2024-09-21T13:54:58.127 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0024356030000000002 2024-09-21T13:54:58.127 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:58.127 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:58.127 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:54:58.127 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:57.442230+0000", 2024-09-21T13:54:58.127 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013154599999999999 2024-09-21T13:54:58.127 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:58.127 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:58.127 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:54:58.127 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:57.442381+0000", 2024-09-21T13:54:58.127 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000151717 2024-09-21T13:54:58.127 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:58.128 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:58.128 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:54:58.128 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:57.442486+0000", 2024-09-21T13:54:58.128 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000104503 2024-09-21T13:54:58.128 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:58.128 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:54:58.128 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:58.128 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:58.128 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:58.129 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:71970 8.1 8:825b048a:::benchmark_data_smithi049_109311_object71969:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-09-21T13:54:58.129 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:54:57.442171+0000", 2024-09-21T13:54:58.129 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.021928551000000001, 2024-09-21T13:54:58.129 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.022064640999999999, 2024-09-21T13:54:58.129 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:54:58.129 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:54:58.129 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T13:54:58.129 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:54:58.129 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:54:58.129 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:54:58.129 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 71970 2024-09-21T13:54:58.129 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:58.129 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:54:58.130 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:58.130 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:54:58.130 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:57.442171+0000", 2024-09-21T13:54:58.130 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:58.130 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:58.130 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:58.130 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:54:58.130 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:57.442171+0000", 2024-09-21T13:54:58.130 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:58.130 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:58.130 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:58.130 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:54:58.131 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:57.442172+0000", 2024-09-21T13:54:58.131 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6330000000000001e-06 2024-09-21T13:54:58.131 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:58.131 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:58.131 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:54:58.131 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:57.442194+0000", 2024-09-21T13:54:58.131 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2152999999999999e-05 2024-09-21T13:54:58.131 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:58.131 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:58.131 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:54:58.131 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:57.442204+0000", 2024-09-21T13:54:58.131 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4930000000000004e-06 2024-09-21T13:54:58.131 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:58.131 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:58.132 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:54:58.132 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:57.442211+0000", 2024-09-21T13:54:58.132 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3470000000000001e-06 2024-09-21T13:54:58.132 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:58.132 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:58.132 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:58.132 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:57.442264+0000", 2024-09-21T13:54:58.132 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3047000000000002e-05 2024-09-21T13:54:58.132 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:58.132 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:58.132 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:54:58.132 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:57.442390+0000", 2024-09-21T13:54:58.132 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012547599999999999 2024-09-21T13:54:58.132 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:58.132 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:54:58.133 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:58.133 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:58.133 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:58.133 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.15207.0:71966 8.3 e199/183 ondisk, result = 0)", 2024-09-21T13:54:58.133 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:54:57.442305+0000", 2024-09-21T13:54:58.133 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.021794173, 2024-09-21T13:54:58.133 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.021961535000000001, 2024-09-21T13:54:58.133 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:54:58.133 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:54:58.133 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T13:54:58.133 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:54:58.133 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:58.133 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:54:58.133 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:57.442305+0000", 2024-09-21T13:54:58.133 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:58.133 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:58.134 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:58.134 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:54:58.134 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:57.442305+0000", 2024-09-21T13:54:58.134 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:54:58.134 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:58.134 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:58.134 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:54:58.134 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:57.442306+0000", 2024-09-21T13:54:58.134 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.8999999999999995e-07 2024-09-21T13:54:58.134 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:58.134 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:58.134 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:54:58.134 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:57.442310+0000", 2024-09-21T13:54:58.135 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2409999999999997e-06 2024-09-21T13:54:58.135 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:58.135 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:58.135 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:54:58.135 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:57.442320+0000", 2024-09-21T13:54:58.135 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7319999999999993e-06 2024-09-21T13:54:58.135 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:58.135 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:58.135 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:54:58.135 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:57.442327+0000", 2024-09-21T13:54:58.135 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9970000000000001e-06 2024-09-21T13:54:58.135 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:58.135 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:58.135 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:54:58.136 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:57.442369+0000", 2024-09-21T13:54:58.136 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2342000000000002e-05 2024-09-21T13:54:58.136 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:54:58.136 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:54:58.136 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:54:58.136 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:57.442376+0000", 2024-09-21T13:54:58.136 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9990000000000002e-06 2024-09-21T13:54:58.136 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:58.136 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:54:58.136 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:58.136 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:54:58.136 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T13:54:58.136 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 4 2024-09-21T13:54:58.137 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T13:54:58.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:58.138 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:58.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:54:58.238 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:54:58.275 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:58.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:58.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:54:58.293 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:2024-09-21T13:54:58.293675+0000 min lat: 0.00131689 max lat: 0.181286 avg lat: 0.00875822 2024-09-21T13:54:58.293 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T13:54:58.293 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 40 16 73063 73047 114.114 79.5 0.00838383 0.00875822 2024-09-21T13:54:58.377 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:54:58.412 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:58.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:58.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:54:58.513 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:58.548 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:58.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:58.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:58.650 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:54:58.679 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:58.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:58.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:54:58.780 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:54:58.810 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:58.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:58.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:54:58.911 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:54:58.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:58 smithi187 ceph-mon[33032]: 6.13 scrub starts 2024-09-21T13:54:58.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:58 smithi187 ceph-mon[33032]: 6.13 scrub ok 2024-09-21T13:54:58.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:58 smithi187 ceph-mon[33032]: 4.1e scrub starts 2024-09-21T13:54:58.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:58 smithi187 ceph-mon[33032]: 4.1e scrub ok 2024-09-21T13:54:58.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:58 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:54:58.947 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:58.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:58.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:54:59.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:58 smithi049 ceph-mon[40489]: 6.13 scrub starts 2024-09-21T13:54:59.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:58 smithi049 ceph-mon[40489]: 6.13 scrub ok 2024-09-21T13:54:59.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:58 smithi049 ceph-mon[40489]: 4.1e scrub starts 2024-09-21T13:54:59.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:58 smithi049 ceph-mon[40489]: 4.1e scrub ok 2024-09-21T13:54:59.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:58 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:54:59.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:58 smithi049 ceph-mon[30080]: 6.13 scrub starts 2024-09-21T13:54:59.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:58 smithi049 ceph-mon[30080]: 6.13 scrub ok 2024-09-21T13:54:59.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:58 smithi049 ceph-mon[30080]: 4.1e scrub starts 2024-09-21T13:54:59.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:58 smithi049 ceph-mon[30080]: 4.1e scrub ok 2024-09-21T13:54:59.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:58 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:54:59.050 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:54:59.081 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:59.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:59.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:54:59.182 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:54:59.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:59.222 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:59.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:54:59.293 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 41 16 74482 74466 113.493 88.6875 0.0113648 0.00880593 2024-09-21T13:54:59.323 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:59.359 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:59.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:59.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:59.461 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:54:59.492 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:59.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:59.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:54:59.593 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:54:59.631 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:54:59.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:59.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:54:59.648 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:54:59 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:54:59.733 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:54:59.759 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:59.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:59.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:54:59.861 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:54:59.892 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:54:59.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:54:59.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:54:59.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:59 smithi187 ceph-mon[33032]: pgmap v293: 148 pgs: 148 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-09-21T13:54:59.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:59 smithi187 ceph-mon[33032]: 6.c scrub starts 2024-09-21T13:54:59.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:59 smithi187 ceph-mon[33032]: 4.8 scrub starts 2024-09-21T13:54:59.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:59 smithi187 ceph-mon[33032]: 4.8 scrub ok 2024-09-21T13:54:59.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:54:59 smithi187 ceph-mon[33032]: 6.c scrub ok 2024-09-21T13:54:59.994 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:55:00.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:59 smithi049 ceph-mon[40489]: pgmap v293: 148 pgs: 148 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-09-21T13:55:00.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:59 smithi049 ceph-mon[40489]: 6.c scrub starts 2024-09-21T13:55:00.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:59 smithi049 ceph-mon[40489]: 4.8 scrub starts 2024-09-21T13:55:00.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:59 smithi049 ceph-mon[40489]: 4.8 scrub ok 2024-09-21T13:55:00.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:54:59 smithi049 ceph-mon[40489]: 6.c scrub ok 2024-09-21T13:55:00.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:59 smithi049 ceph-mon[30080]: pgmap v293: 148 pgs: 148 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-09-21T13:55:00.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:59 smithi049 ceph-mon[30080]: 6.c scrub starts 2024-09-21T13:55:00.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:59 smithi049 ceph-mon[30080]: 4.8 scrub starts 2024-09-21T13:55:00.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:59 smithi049 ceph-mon[30080]: 4.8 scrub ok 2024-09-21T13:55:00.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:54:59 smithi049 ceph-mon[30080]: 6.c scrub ok 2024-09-21T13:55:00.021 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:00.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:00.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:55:00.123 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:55:00.167 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:00.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:00.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:55:00.269 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:55:00.293 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 42 16 75727 75711 112.644 77.8125 0.010189 0.00887192 2024-09-21T13:55:00.306 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:00.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:00.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:55:00.408 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:55:00.442 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:00.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:00.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:55:00.544 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:00.570 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:00.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:00.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:00.672 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:00.703 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:00.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:00.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:00.805 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:00.833 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:00.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:00.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:00.934 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:55:00.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:00 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:55:00.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:00 smithi187 ceph-mon[33032]: 6.1c scrub starts 2024-09-21T13:55:00.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:00 smithi187 ceph-mon[33032]: 6.1c scrub ok 2024-09-21T13:55:00.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:00 smithi187 ceph-mon[33032]: 3.5 scrub starts 2024-09-21T13:55:00.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:00 smithi187 ceph-mon[33032]: 3.5 scrub ok 2024-09-21T13:55:00.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:00 smithi187 ceph-mon[33032]: osdmap e200: 8 total, 8 up, 8 in 2024-09-21T13:55:00.972 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:00.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:00.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:55:01.004 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:55:00 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:55:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:55:01.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:00 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:55:01.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:00 smithi049 ceph-mon[30080]: 6.1c scrub starts 2024-09-21T13:55:01.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:00 smithi049 ceph-mon[30080]: 6.1c scrub ok 2024-09-21T13:55:01.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:00 smithi049 ceph-mon[30080]: 3.5 scrub starts 2024-09-21T13:55:01.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:00 smithi049 ceph-mon[30080]: 3.5 scrub ok 2024-09-21T13:55:01.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:00 smithi049 ceph-mon[30080]: osdmap e200: 8 total, 8 up, 8 in 2024-09-21T13:55:01.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:00 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:55:01.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:00 smithi049 ceph-mon[40489]: 6.1c scrub starts 2024-09-21T13:55:01.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:00 smithi049 ceph-mon[40489]: 6.1c scrub ok 2024-09-21T13:55:01.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:00 smithi049 ceph-mon[40489]: 3.5 scrub starts 2024-09-21T13:55:01.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:00 smithi049 ceph-mon[40489]: 3.5 scrub ok 2024-09-21T13:55:01.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:00 smithi049 ceph-mon[40489]: osdmap e200: 8 total, 8 up, 8 in 2024-09-21T13:55:01.074 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:55:01.117 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:01.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:01.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:55:01.219 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:55:01.266 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:01.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:01.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:55:01.293 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 43 16 77033 77017 111.922 81.625 0.012163 0.00893005 2024-09-21T13:55:01.368 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:55:01.403 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:01.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:01.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:55:01.504 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:01.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:01.533 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:01.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:01.634 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:01.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:01.665 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:01.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:01.765 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:55:01.802 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:01.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:01.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:55:01.904 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:55:01.932 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:01 smithi049 ceph-mon[40489]: pgmap v294: 148 pgs: 148 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.4 KiB/s rd, 116 MiB/s wr, 1.86k op/s 2024-09-21T13:55:01.932 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:01 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:55:01.932 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:01 smithi049 ceph-mon[30080]: pgmap v294: 148 pgs: 148 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.4 KiB/s rd, 116 MiB/s wr, 1.86k op/s 2024-09-21T13:55:01.932 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:01 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:55:01.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:01 smithi187 ceph-mon[33032]: pgmap v294: 148 pgs: 148 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.4 KiB/s rd, 116 MiB/s wr, 1.86k op/s 2024-09-21T13:55:01.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:01 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:55:01.945 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:55:01.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:01.956 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:01.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:55:01.991 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T13:55:02.057 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:55:02.115 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:02.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:02.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:55:02.216 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:02.248 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:02.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:02.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:02.294 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 44 16 78161 78145 110.979 70.5 0.0192126 0.00900452 2024-09-21T13:55:02.350 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:55:02.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:02.417 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:02.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:55:02.518 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:55:02.568 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:02.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:02.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:55:02.670 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:55:02.709 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:02.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:02.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:55:02.811 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:02.832 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:02.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:02.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:02.933 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:55:02.961 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:02.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:02.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:55:03.063 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:55:03.089 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:03.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:03.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:55:03.191 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:55:03.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:02 smithi187 ceph-mon[33032]: pgmap v296: 148 pgs: 148 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 745 B/s rd, 59 MiB/s wr, 952 op/s 2024-09-21T13:55:03.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:02 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:55:03.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:02 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:55:03.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:02 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:55:03.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:02 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:55:03.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:02 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:55:03.223 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:03.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:03.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:55:03.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:02 smithi049 ceph-mon[40489]: pgmap v296: 148 pgs: 148 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 745 B/s rd, 59 MiB/s wr, 952 op/s 2024-09-21T13:55:03.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:02 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:55:03.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:02 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:55:03.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:02 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:55:03.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:02 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:55:03.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:02 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:55:03.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:02 smithi049 ceph-mon[30080]: pgmap v296: 148 pgs: 148 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 745 B/s rd, 59 MiB/s wr, 952 op/s 2024-09-21T13:55:03.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:02 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:55:03.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:02 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:55:03.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:02 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:55:03.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:02 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:55:03.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:02 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:55:03.294 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 45 16 79266 79250 110.047 69.0625 0.0168977 0.00908138 2024-09-21T13:55:03.325 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:03.346 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:03.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:03.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:03.448 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:55:03.493 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:03.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:03.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:55:03.595 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:55:03.628 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:03.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:03.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:55:03.729 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:55:03.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:03.770 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:03.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:55:03.870 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:55:03.900 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:03.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:03.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:55:04.002 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:55:04.040 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:04.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:04.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:55:04.142 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:55:04.188 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:04.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:04.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:55:04.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:03 smithi187 ceph-mon[33032]: 4.1d scrub starts 2024-09-21T13:55:04.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:03 smithi187 ceph-mon[33032]: 4.1d scrub ok 2024-09-21T13:55:04.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:03 smithi187 ceph-mon[33032]: 6.15 scrub starts 2024-09-21T13:55:04.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:03 smithi187 ceph-mon[33032]: 6.15 scrub ok 2024-09-21T13:55:04.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:03 smithi049 ceph-mon[40489]: 4.1d scrub starts 2024-09-21T13:55:04.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:03 smithi049 ceph-mon[40489]: 4.1d scrub ok 2024-09-21T13:55:04.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:03 smithi049 ceph-mon[40489]: 6.15 scrub starts 2024-09-21T13:55:04.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:03 smithi049 ceph-mon[40489]: 6.15 scrub ok 2024-09-21T13:55:04.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:03 smithi049 ceph-mon[30080]: 4.1d scrub starts 2024-09-21T13:55:04.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:03 smithi049 ceph-mon[30080]: 4.1d scrub ok 2024-09-21T13:55:04.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:03 smithi049 ceph-mon[30080]: 6.15 scrub starts 2024-09-21T13:55:04.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:03 smithi049 ceph-mon[30080]: 6.15 scrub ok 2024-09-21T13:55:04.289 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:04.294 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 46 16 80430 80414 109.237 72.75 0.020756 0.00914839 2024-09-21T13:55:04.315 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:04.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:04.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:04.386 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:55:04.417 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:55:04.470 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:04.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:04.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:55:04.572 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:04.604 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:04.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:04.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:04.706 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:55:04.745 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:04.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:04.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:55:04.847 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:55:04.935 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-09-21T13:55:04.966 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:04.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:04.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:55:05.048 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T13:55:05.048 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-09-21T13:55:05.048 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-09-21T13:55:05.048 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-09-21T13:55:05.048 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T13:55:05.068 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:05.097 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:05.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:05.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:05.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:04 smithi187 ceph-mon[33032]: 3.17 scrub starts 2024-09-21T13:55:05.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:04 smithi187 ceph-mon[33032]: 3.17 scrub ok 2024-09-21T13:55:05.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:04 smithi187 ceph-mon[33032]: pgmap v297: 148 pgs: 148 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 90 MiB/s wr, 1.44k op/s 2024-09-21T13:55:05.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:04 smithi187 ceph-mon[33032]: 3.f scrub starts 2024-09-21T13:55:05.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:04 smithi187 ceph-mon[33032]: 3.f scrub ok 2024-09-21T13:55:05.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:04 smithi187 ceph-mon[33032]: 3.a scrub starts 2024-09-21T13:55:05.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:04 smithi187 ceph-mon[33032]: 3.a scrub ok 2024-09-21T13:55:05.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:04 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1075059439' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:55:05.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:04 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:55:05.200 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:05.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:05.229 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:05.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:05.243 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:04 smithi049 ceph-mon[40489]: 3.17 scrub starts 2024-09-21T13:55:05.243 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:04 smithi049 ceph-mon[40489]: 3.17 scrub ok 2024-09-21T13:55:05.243 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:04 smithi049 ceph-mon[40489]: pgmap v297: 148 pgs: 148 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 90 MiB/s wr, 1.44k op/s 2024-09-21T13:55:05.243 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:04 smithi049 ceph-mon[40489]: 3.f scrub starts 2024-09-21T13:55:05.243 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:04 smithi049 ceph-mon[40489]: 3.f scrub ok 2024-09-21T13:55:05.244 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:04 smithi049 ceph-mon[40489]: 3.a scrub starts 2024-09-21T13:55:05.244 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:04 smithi049 ceph-mon[40489]: 3.a scrub ok 2024-09-21T13:55:05.244 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:04 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1075059439' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:55:05.244 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:04 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:55:05.244 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:04 smithi049 ceph-mon[30080]: 3.17 scrub starts 2024-09-21T13:55:05.244 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:04 smithi049 ceph-mon[30080]: 3.17 scrub ok 2024-09-21T13:55:05.244 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:04 smithi049 ceph-mon[30080]: pgmap v297: 148 pgs: 148 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 90 MiB/s wr, 1.44k op/s 2024-09-21T13:55:05.244 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:04 smithi049 ceph-mon[30080]: 3.f scrub starts 2024-09-21T13:55:05.244 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:04 smithi049 ceph-mon[30080]: 3.f scrub ok 2024-09-21T13:55:05.244 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:04 smithi049 ceph-mon[30080]: 3.a scrub starts 2024-09-21T13:55:05.244 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:04 smithi049 ceph-mon[30080]: 3.a scrub ok 2024-09-21T13:55:05.244 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:04 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1075059439' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:55:05.245 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:04 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:55:05.294 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 47 16 81562 81546 108.418 70.75 0.0082771 0.00921875 2024-09-21T13:55:05.330 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:55:05.374 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:05.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:05.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:55:05.476 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:55:05.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:05.510 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:05.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:55:05.611 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:55:05.646 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:05.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:05.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:55:05.747 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:05.783 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:05.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:05.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:05.885 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:05.918 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:05.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:05.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:06.020 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:55:06.067 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:06.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:06.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:55:06.075 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T13:55:06.161 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:05 smithi049 ceph-mon[40489]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T13:55:06.161 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:05 smithi049 ceph-mon[40489]: Cluster is now healthy 2024-09-21T13:55:06.161 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:05 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T13:55:06.161 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:05 smithi049 ceph-mon[40489]: osdmap e201: 8 total, 8 up, 8 in 2024-09-21T13:55:06.161 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:05 smithi049 ceph-mon[40489]: osdmap e202: 8 total, 8 up, 8 in 2024-09-21T13:55:06.161 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:05 smithi049 ceph-mon[40489]: 4.16 scrub starts 2024-09-21T13:55:06.161 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:05 smithi049 ceph-mon[30080]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T13:55:06.161 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:05 smithi049 ceph-mon[30080]: Cluster is now healthy 2024-09-21T13:55:06.162 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:05 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T13:55:06.162 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:05 smithi049 ceph-mon[30080]: osdmap e201: 8 total, 8 up, 8 in 2024-09-21T13:55:06.162 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:05 smithi049 ceph-mon[30080]: osdmap e202: 8 total, 8 up, 8 in 2024-09-21T13:55:06.162 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:05 smithi049 ceph-mon[30080]: 4.16 scrub starts 2024-09-21T13:55:06.169 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:06.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:05 smithi187 ceph-mon[33032]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T13:55:06.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:05 smithi187 ceph-mon[33032]: Cluster is now healthy 2024-09-21T13:55:06.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:05 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T13:55:06.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:05 smithi187 ceph-mon[33032]: osdmap e201: 8 total, 8 up, 8 in 2024-09-21T13:55:06.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:05 smithi187 ceph-mon[33032]: osdmap e202: 8 total, 8 up, 8 in 2024-09-21T13:55:06.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:05 smithi187 ceph-mon[33032]: 4.16 scrub starts 2024-09-21T13:55:06.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:06.203 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:06.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:06.294 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 48 15 82845 82830 107.831 80.25 0.00491741 0.00926858 2024-09-21T13:55:06.305 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:55:06.317 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T13:55:06.331 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:06.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:06.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:55:06.386 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 2] 2024-09-21T13:55:06.386 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T13:55:06.386 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-21T13:55:06.387 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd dump -f json-pretty 2024-09-21T13:55:06.433 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:55:06.459 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:06.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:06.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:55:06.561 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:55:06.588 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:06.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:06.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:55:06.690 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:55:06.718 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:06.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:06.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:55:06.819 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:55:06.836 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:55:06.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:06.879 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:06.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:55:06.885 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:55:06.980 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:06.985 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:06 smithi049 ceph-mon[40489]: 4.16 scrub ok 2024-09-21T13:55:06.985 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:06 smithi049 ceph-mon[40489]: pgmap v300: 148 pgs: 148 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-09-21T13:55:06.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:06 smithi049 ceph-mon[30080]: 4.16 scrub ok 2024-09-21T13:55:06.989 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:06 smithi049 ceph-mon[30080]: pgmap v300: 148 pgs: 148 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-09-21T13:55:07.010 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:07.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:07.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:07.112 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:55:07.144 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:07.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:07.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:55:07.245 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:55:07.289 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:07.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:07.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:55:07.294 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 49 16 83926 83910 107.008 67.5 0.0265082 0.0093399 2024-09-21T13:55:07.392 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:55:07.426 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:07.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:07.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:55:07.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:06 smithi187 ceph-mon[33032]: 4.16 scrub ok 2024-09-21T13:55:07.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:06 smithi187 ceph-mon[33032]: pgmap v300: 148 pgs: 148 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-09-21T13:55:07.528 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:07.562 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:07.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:07.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:07.666 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:07.691 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:07.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:07.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:07.695 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:55:07.767 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:55:07.768 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:55:07.793 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:55:07.806 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:55:07.807 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:55:07.849 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:07.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:07.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:55:07.851 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:55:07.851 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:55:07.890 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:55:07.890 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:55:07.928 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:55:07.928 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:55:07.952 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:07.972 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:55:07.972 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:55:07.985 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:07.985 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T13:55:07.985 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:55:07.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:07.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:08.012 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:55:08.012 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:55:08.046 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:55:08.046 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:55:08.057 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:07 smithi049 ceph-mon[40489]: 3.1e scrub starts 2024-09-21T13:55:08.057 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:07 smithi049 ceph-mon[40489]: 5.5 scrub starts 2024-09-21T13:55:08.057 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:07 smithi049 ceph-mon[40489]: 5.5 scrub ok 2024-09-21T13:55:08.057 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:07 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2610766931' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:55:08.058 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:07 smithi049 ceph-mon[30080]: 3.1e scrub starts 2024-09-21T13:55:08.058 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:07 smithi049 ceph-mon[30080]: 5.5 scrub starts 2024-09-21T13:55:08.058 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:07 smithi049 ceph-mon[30080]: 5.5 scrub ok 2024-09-21T13:55:08.058 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:07 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2610766931' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:55:08.086 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:55:08.135 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:08.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:08.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:55:08.237 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:55:08.276 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:08.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:08.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:55:08.294 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 50 16 85087 85071 106.319 72.5625 0.00607772 0.00940122 2024-09-21T13:55:08.378 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:55:08.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:08.419 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:08.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:55:08.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:07 smithi187 ceph-mon[33032]: 3.1e scrub starts 2024-09-21T13:55:08.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:07 smithi187 ceph-mon[33032]: 5.5 scrub starts 2024-09-21T13:55:08.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:07 smithi187 ceph-mon[33032]: 5.5 scrub ok 2024-09-21T13:55:08.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:07 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2610766931' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:55:08.520 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:55:08.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:08.548 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:08.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:55:08.649 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:08.683 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:08.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:08.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:08.738 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T13:55:08.785 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:55:08.839 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:08.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:08.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:55:08.941 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:55:08.978 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:08 smithi049 ceph-mon[30080]: 3.1e scrub ok 2024-09-21T13:55:08.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:08.994 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:08.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:55:09.095 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:09.127 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:09.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:09.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:09.228 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:55:09.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:08 smithi049 ceph-mon[40489]: 3.1e scrub ok 2024-09-21T13:55:09.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:08 smithi049 ceph-mon[40489]: 3.1b scrub starts 2024-09-21T13:55:09.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:08 smithi049 ceph-mon[40489]: 3.1b scrub ok 2024-09-21T13:55:09.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:08 smithi049 ceph-mon[40489]: pgmap v301: 148 pgs: 148 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 651 B/s rd, 49 MiB/s wr, 782 op/s 2024-09-21T13:55:09.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:08 smithi049 ceph-mon[30080]: 3.1b scrub starts 2024-09-21T13:55:09.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:08 smithi049 ceph-mon[30080]: 3.1b scrub ok 2024-09-21T13:55:09.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:08 smithi049 ceph-mon[30080]: pgmap v301: 148 pgs: 148 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 651 B/s rd, 49 MiB/s wr, 782 op/s 2024-09-21T13:55:09.260 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:55:09.260 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T13:55:09.260 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 202, 2024-09-21T13:55:09.260 INFO:teuthology.orchestra.run.smithi049.stdout: "fsid": "40a3d2e8-781f-11ef-baf6-efdc52797490", 2024-09-21T13:55:09.260 INFO:teuthology.orchestra.run.smithi049.stdout: "created": "2024-09-21T13:42:57.838275+0000", 2024-09-21T13:55:09.260 INFO:teuthology.orchestra.run.smithi049.stdout: "modified": "2024-09-21T13:55:04.933637+0000", 2024-09-21T13:55:09.260 INFO:teuthology.orchestra.run.smithi049.stdout: "last_up_change": "2024-09-21T13:52:03.297364+0000", 2024-09-21T13:55:09.260 INFO:teuthology.orchestra.run.smithi049.stdout: "last_in_change": "2024-09-21T13:48:23.514355+0000", 2024-09-21T13:55:09.261 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-21T13:55:09.261 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_num": 5799936, 2024-09-21T13:55:09.261 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_set": [ 2024-09-21T13:55:09.261 INFO:teuthology.orchestra.run.smithi049.stdout: "pglog_hardlimit", 2024-09-21T13:55:09.261 INFO:teuthology.orchestra.run.smithi049.stdout: "purged_snapdirs", 2024-09-21T13:55:09.261 INFO:teuthology.orchestra.run.smithi049.stdout: "recovery_deletes", 2024-09-21T13:55:09.261 INFO:teuthology.orchestra.run.smithi049.stdout: "sortbitwise" 2024-09-21T13:55:09.261 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T13:55:09.261 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_version": 19, 2024-09-21T13:55:09.261 INFO:teuthology.orchestra.run.smithi049.stdout: "full_ratio": 0.94999998807907104, 2024-09-21T13:55:09.261 INFO:teuthology.orchestra.run.smithi049.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-21T13:55:09.261 INFO:teuthology.orchestra.run.smithi049.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-21T13:55:09.262 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_snapshot": "", 2024-09-21T13:55:09.262 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_max": 8, 2024-09-21T13:55:09.262 INFO:teuthology.orchestra.run.smithi049.stdout: "max_osd": 8, 2024-09-21T13:55:09.262 INFO:teuthology.orchestra.run.smithi049.stdout: "require_min_compat_client": "luminous", 2024-09-21T13:55:09.262 INFO:teuthology.orchestra.run.smithi049.stdout: "min_compat_client": "luminous", 2024-09-21T13:55:09.262 INFO:teuthology.orchestra.run.smithi049.stdout: "require_osd_release": "squid", 2024-09-21T13:55:09.262 INFO:teuthology.orchestra.run.smithi049.stdout: "allow_crimson": false, 2024-09-21T13:55:09.262 INFO:teuthology.orchestra.run.smithi049.stdout: "pools": [ 2024-09-21T13:55:09.262 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.262 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 1, 2024-09-21T13:55:09.262 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": ".mgr", 2024-09-21T13:55:09.263 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:46:21.096244+0000", 2024-09-21T13:55:09.263 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T13:55:09.263 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T13:55:09.263 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T13:55:09.263 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T13:55:09.263 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T13:55:09.263 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T13:55:09.263 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T13:55:09.263 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T13:55:09.263 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T13:55:09.263 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T13:55:09.263 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T13:55:09.264 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T13:55:09.264 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T13:55:09.264 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 1, 2024-09-21T13:55:09.264 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 1, 2024-09-21T13:55:09.264 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 1, 2024-09-21T13:55:09.264 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 1, 2024-09-21T13:55:09.264 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 1, 2024-09-21T13:55:09.264 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T13:55:09.264 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T13:55:09.264 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T13:55:09.264 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T13:55:09.264 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T13:55:09.265 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T13:55:09.265 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T13:55:09.265 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.265 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "199", 2024-09-21T13:55:09.265 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T13:55:09.265 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T13:55:09.265 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T13:55:09.265 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T13:55:09.265 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T13:55:09.265 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T13:55:09.265 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T13:55:09.266 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T13:55:09.266 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T13:55:09.266 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T13:55:09.266 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T13:55:09.266 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T13:55:09.266 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T13:55:09.266 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T13:55:09.266 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T13:55:09.266 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T13:55:09.266 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T13:55:09.267 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T13:55:09.267 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T13:55:09.267 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T13:55:09.267 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T13:55:09.267 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T13:55:09.267 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T13:55:09.267 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T13:55:09.267 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T13:55:09.267 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T13:55:09.267 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.267 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T13:55:09.267 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T13:55:09.268 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T13:55:09.268 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T13:55:09.268 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T13:55:09.268 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T13:55:09.268 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T13:55:09.268 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T13:55:09.268 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T13:55:09.268 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T13:55:09.268 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T13:55:09.268 INFO:teuthology.orchestra.run.smithi049.stdout: "options": { 2024-09-21T13:55:09.268 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_max": 32, 2024-09-21T13:55:09.268 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_min": 1 2024-09-21T13:55:09.269 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.269 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T13:55:09.269 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr": {} 2024-09-21T13:55:09.269 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.269 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T13:55:09.269 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T13:55:09.269 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 7.8899998664855957, 2024-09-21T13:55:09.269 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 7.8899998664855957, 2024-09-21T13:55:09.269 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 0.37999999523162842, 2024-09-21T13:55:09.269 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 3, 2024-09-21T13:55:09.269 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 3, 2024-09-21T13:55:09.269 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.86855697631835938, 2024-09-21T13:55:09.270 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.87000000476837158, 2024-09-21T13:55:09.270 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.87000000476837158 2024-09-21T13:55:09.270 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:09.270 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.270 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.270 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 2, 2024-09-21T13:55:09.270 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "datapool", 2024-09-21T13:55:09.270 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:48:55.540712+0000", 2024-09-21T13:55:09.270 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 8193, 2024-09-21T13:55:09.270 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T13:55:09.270 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T13:55:09.270 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T13:55:09.271 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T13:55:09.271 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T13:55:09.271 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T13:55:09.271 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T13:55:09.271 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T13:55:09.271 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T13:55:09.271 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T13:55:09.271 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T13:55:09.271 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T13:55:09.271 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 3, 2024-09-21T13:55:09.271 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 3, 2024-09-21T13:55:09.271 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 3, 2024-09-21T13:55:09.272 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 3, 2024-09-21T13:55:09.272 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 3, 2024-09-21T13:55:09.272 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T13:55:09.272 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T13:55:09.272 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T13:55:09.272 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T13:55:09.272 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T13:55:09.272 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T13:55:09.272 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T13:55:09.272 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.272 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "63", 2024-09-21T13:55:09.272 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T13:55:09.272 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T13:55:09.273 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T13:55:09.273 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T13:55:09.273 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T13:55:09.273 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 2, 2024-09-21T13:55:09.273 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 63, 2024-09-21T13:55:09.273 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T13:55:09.273 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T13:55:09.273 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T13:55:09.273 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T13:55:09.273 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T13:55:09.273 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T13:55:09.273 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T13:55:09.274 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T13:55:09.274 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T13:55:09.274 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T13:55:09.274 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T13:55:09.274 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T13:55:09.274 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T13:55:09.274 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T13:55:09.274 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T13:55:09.274 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T13:55:09.274 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T13:55:09.274 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T13:55:09.274 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T13:55:09.275 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.275 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T13:55:09.275 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T13:55:09.275 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T13:55:09.275 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T13:55:09.275 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T13:55:09.275 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T13:55:09.275 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T13:55:09.275 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T13:55:09.275 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T13:55:09.275 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T13:55:09.275 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T13:55:09.276 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T13:55:09.276 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T13:55:09.276 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd": {} 2024-09-21T13:55:09.276 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.276 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T13:55:09.276 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T13:55:09.276 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2.6500000953674316, 2024-09-21T13:55:09.276 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2.6500000953674316, 2024-09-21T13:55:09.276 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 0.87999999523162842, 2024-09-21T13:55:09.276 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2.3299999237060547, 2024-09-21T13:55:09.276 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2.3299999237060547, 2024-09-21T13:55:09.276 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.86855697631835938, 2024-09-21T13:55:09.276 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.87000000476837158, 2024-09-21T13:55:09.277 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.87000000476837158 2024-09-21T13:55:09.277 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:09.277 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.277 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.277 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 3, 2024-09-21T13:55:09.277 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": ".rgw.root", 2024-09-21T13:55:09.277 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:48:56.417456+0000", 2024-09-21T13:55:09.277 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T13:55:09.277 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T13:55:09.277 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T13:55:09.277 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T13:55:09.278 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T13:55:09.278 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T13:55:09.278 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T13:55:09.278 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T13:55:09.278 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T13:55:09.278 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T13:55:09.278 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T13:55:09.278 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T13:55:09.278 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T13:55:09.278 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 32, 2024-09-21T13:55:09.279 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 32, 2024-09-21T13:55:09.279 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 32, 2024-09-21T13:55:09.279 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 32, 2024-09-21T13:55:09.279 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 32, 2024-09-21T13:55:09.279 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T13:55:09.279 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T13:55:09.279 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T13:55:09.279 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T13:55:09.279 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T13:55:09.279 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T13:55:09.279 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T13:55:09.279 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.279 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "62", 2024-09-21T13:55:09.280 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T13:55:09.280 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T13:55:09.280 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T13:55:09.280 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T13:55:09.280 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T13:55:09.280 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T13:55:09.280 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T13:55:09.280 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T13:55:09.280 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T13:55:09.280 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T13:55:09.280 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T13:55:09.280 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T13:55:09.280 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T13:55:09.281 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T13:55:09.281 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T13:55:09.281 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T13:55:09.281 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T13:55:09.281 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T13:55:09.281 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T13:55:09.281 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T13:55:09.281 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T13:55:09.281 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T13:55:09.281 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T13:55:09.281 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T13:55:09.281 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T13:55:09.281 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T13:55:09.282 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.282 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T13:55:09.282 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T13:55:09.282 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T13:55:09.282 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T13:55:09.282 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T13:55:09.282 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T13:55:09.282 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T13:55:09.282 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T13:55:09.282 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T13:55:09.282 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T13:55:09.282 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T13:55:09.283 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T13:55:09.283 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T13:55:09.283 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw": {} 2024-09-21T13:55:09.283 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.283 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T13:55:09.283 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T13:55:09.283 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2, 2024-09-21T13:55:09.283 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2, 2024-09-21T13:55:09.283 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1, 2024-09-21T13:55:09.283 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2, 2024-09-21T13:55:09.283 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2, 2024-09-21T13:55:09.284 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.86855697631835938, 2024-09-21T13:55:09.284 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.87000000476837158, 2024-09-21T13:55:09.284 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.87000000476837158 2024-09-21T13:55:09.284 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:09.284 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.284 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.284 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 4, 2024-09-21T13:55:09.284 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "default.rgw.log", 2024-09-21T13:55:09.284 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:48:58.128455+0000", 2024-09-21T13:55:09.284 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T13:55:09.284 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T13:55:09.284 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T13:55:09.285 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T13:55:09.285 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T13:55:09.285 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T13:55:09.285 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T13:55:09.285 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T13:55:09.285 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T13:55:09.285 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T13:55:09.285 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T13:55:09.285 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T13:55:09.285 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T13:55:09.285 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 32, 2024-09-21T13:55:09.285 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 32, 2024-09-21T13:55:09.286 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 32, 2024-09-21T13:55:09.286 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 32, 2024-09-21T13:55:09.286 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 32, 2024-09-21T13:55:09.286 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T13:55:09.286 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T13:55:09.286 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T13:55:09.286 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T13:55:09.286 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T13:55:09.286 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T13:55:09.286 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T13:55:09.286 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.286 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "64", 2024-09-21T13:55:09.287 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T13:55:09.287 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T13:55:09.287 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T13:55:09.287 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T13:55:09.287 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T13:55:09.287 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T13:55:09.287 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T13:55:09.287 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T13:55:09.287 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T13:55:09.287 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T13:55:09.287 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T13:55:09.287 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T13:55:09.288 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T13:55:09.288 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T13:55:09.288 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T13:55:09.288 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T13:55:09.288 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T13:55:09.288 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T13:55:09.288 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T13:55:09.288 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T13:55:09.288 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T13:55:09.288 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T13:55:09.288 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T13:55:09.288 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T13:55:09.289 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T13:55:09.289 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T13:55:09.289 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.289 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T13:55:09.289 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T13:55:09.289 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T13:55:09.289 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T13:55:09.289 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T13:55:09.289 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T13:55:09.289 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T13:55:09.289 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T13:55:09.289 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T13:55:09.290 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T13:55:09.290 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T13:55:09.290 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T13:55:09.290 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T13:55:09.290 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw": {} 2024-09-21T13:55:09.290 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.290 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T13:55:09.290 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T13:55:09.290 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2.25, 2024-09-21T13:55:09.290 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2.25, 2024-09-21T13:55:09.290 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1, 2024-09-21T13:55:09.291 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2.25, 2024-09-21T13:55:09.291 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2.25, 2024-09-21T13:55:09.291 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.86855697631835938, 2024-09-21T13:55:09.291 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.87000000476837158, 2024-09-21T13:55:09.291 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.87000000476837158 2024-09-21T13:55:09.291 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:09.291 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.291 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.291 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 5, 2024-09-21T13:55:09.291 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "default.rgw.control", 2024-09-21T13:55:09.291 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:49:00.069361+0000", 2024-09-21T13:55:09.292 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T13:55:09.292 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T13:55:09.292 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T13:55:09.292 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T13:55:09.292 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T13:55:09.292 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T13:55:09.292 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T13:55:09.292 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T13:55:09.292 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T13:55:09.292 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T13:55:09.292 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T13:55:09.292 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T13:55:09.292 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T13:55:09.293 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 32, 2024-09-21T13:55:09.293 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 32, 2024-09-21T13:55:09.293 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 32, 2024-09-21T13:55:09.293 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 32, 2024-09-21T13:55:09.293 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 32, 2024-09-21T13:55:09.293 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T13:55:09.293 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T13:55:09.293 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T13:55:09.293 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T13:55:09.293 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T13:55:09.293 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T13:55:09.294 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T13:55:09.294 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.294 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "66", 2024-09-21T13:55:09.294 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T13:55:09.294 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T13:55:09.294 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T13:55:09.294 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T13:55:09.294 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T13:55:09.294 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T13:55:09.294 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T13:55:09.294 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T13:55:09.294 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T13:55:09.295 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T13:55:09.295 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T13:55:09.295 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T13:55:09.295 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T13:55:09.295 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T13:55:09.295 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T13:55:09.295 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T13:55:09.295 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T13:55:09.295 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T13:55:09.295 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T13:55:09.295 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T13:55:09.295 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T13:55:09.296 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T13:55:09.296 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T13:55:09.296 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T13:55:09.296 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T13:55:09.296 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T13:55:09.296 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.296 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T13:55:09.296 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T13:55:09.296 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T13:55:09.296 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T13:55:09.296 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T13:55:09.296 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T13:55:09.297 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T13:55:09.297 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T13:55:09.297 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T13:55:09.297 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T13:55:09.297 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T13:55:09.297 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T13:55:09.297 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T13:55:09.297 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw": {} 2024-09-21T13:55:09.297 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.297 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T13:55:09.297 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T13:55:09.298 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 1.25, 2024-09-21T13:55:09.298 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 1.25, 2024-09-21T13:55:09.298 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1, 2024-09-21T13:55:09.298 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 1.25, 2024-09-21T13:55:09.299 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 1.25, 2024-09-21T13:55:09.299 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.86855697631835938, 2024-09-21T13:55:09.299 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.87000000476837158, 2024-09-21T13:55:09.299 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.87000000476837158 2024-09-21T13:55:09.299 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:09.299 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.299 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.299 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 6, 2024-09-21T13:55:09.300 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "default.rgw.meta", 2024-09-21T13:55:09.300 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:49:02.127399+0000", 2024-09-21T13:55:09.300 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T13:55:09.300 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T13:55:09.300 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T13:55:09.300 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T13:55:09.300 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T13:55:09.300 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T13:55:09.300 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T13:55:09.300 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T13:55:09.300 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T13:55:09.300 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T13:55:09.301 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T13:55:09.301 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T13:55:09.301 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T13:55:09.301 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 32, 2024-09-21T13:55:09.301 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 32, 2024-09-21T13:55:09.301 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 32, 2024-09-21T13:55:09.301 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 32, 2024-09-21T13:55:09.301 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 32, 2024-09-21T13:55:09.301 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T13:55:09.301 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T13:55:09.301 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T13:55:09.302 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T13:55:09.302 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T13:55:09.302 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T13:55:09.302 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T13:55:09.302 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.302 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "68", 2024-09-21T13:55:09.302 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T13:55:09.302 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T13:55:09.302 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T13:55:09.302 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T13:55:09.302 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T13:55:09.302 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T13:55:09.303 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T13:55:09.303 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T13:55:09.303 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T13:55:09.303 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T13:55:09.303 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T13:55:09.303 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T13:55:09.303 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T13:55:09.303 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T13:55:09.303 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T13:55:09.303 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T13:55:09.303 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T13:55:09.303 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T13:55:09.304 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T13:55:09.304 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T13:55:09.304 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T13:55:09.304 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T13:55:09.304 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T13:55:09.304 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T13:55:09.304 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T13:55:09.304 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T13:55:09.304 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.304 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T13:55:09.304 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T13:55:09.304 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T13:55:09.305 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T13:55:09.305 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T13:55:09.305 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T13:55:09.305 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T13:55:09.305 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T13:55:09.305 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T13:55:09.305 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T13:55:09.305 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T13:55:09.305 INFO:teuthology.orchestra.run.smithi049.stdout: "options": { 2024-09-21T13:55:09.305 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_bias": 4 2024-09-21T13:55:09.305 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.305 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T13:55:09.305 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw": {} 2024-09-21T13:55:09.306 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.306 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T13:55:09.306 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T13:55:09.306 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2, 2024-09-21T13:55:09.306 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2, 2024-09-21T13:55:09.306 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1, 2024-09-21T13:55:09.306 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2, 2024-09-21T13:55:09.306 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2, 2024-09-21T13:55:09.306 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.86855697631835938, 2024-09-21T13:55:09.306 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.87000000476837158, 2024-09-21T13:55:09.306 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.87000000476837158 2024-09-21T13:55:09.307 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:09.307 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.307 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.307 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 8, 2024-09-21T13:55:09.307 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "unique_pool_1", 2024-09-21T13:55:09.307 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:54:10.659855+0000", 2024-09-21T13:55:09.307 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T13:55:09.307 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T13:55:09.307 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T13:55:09.307 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T13:55:09.307 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T13:55:09.307 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T13:55:09.308 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T13:55:09.308 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T13:55:09.308 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T13:55:09.308 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T13:55:09.308 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T13:55:09.308 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T13:55:09.308 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T13:55:09.308 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 16, 2024-09-21T13:55:09.308 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 16, 2024-09-21T13:55:09.308 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 16, 2024-09-21T13:55:09.308 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 16, 2024-09-21T13:55:09.308 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 16, 2024-09-21T13:55:09.308 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T13:55:09.309 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T13:55:09.309 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T13:55:09.309 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T13:55:09.309 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T13:55:09.309 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T13:55:09.309 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T13:55:09.309 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.309 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "187", 2024-09-21T13:55:09.309 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T13:55:09.309 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T13:55:09.309 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T13:55:09.309 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T13:55:09.309 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T13:55:09.310 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T13:55:09.310 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T13:55:09.310 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T13:55:09.310 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T13:55:09.310 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T13:55:09.310 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T13:55:09.310 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T13:55:09.310 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T13:55:09.310 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T13:55:09.310 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T13:55:09.310 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T13:55:09.310 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T13:55:09.311 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T13:55:09.311 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T13:55:09.311 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T13:55:09.311 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T13:55:09.311 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T13:55:09.311 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T13:55:09.311 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T13:55:09.311 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T13:55:09.311 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T13:55:09.311 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.311 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T13:55:09.311 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T13:55:09.312 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T13:55:09.312 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T13:55:09.312 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T13:55:09.312 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T13:55:09.312 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T13:55:09.312 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T13:55:09.312 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T13:55:09.312 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T13:55:09.312 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T13:55:09.312 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T13:55:09.312 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T13:55:09.313 INFO:teuthology.orchestra.run.smithi049.stdout: "rados": {} 2024-09-21T13:55:09.313 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.313 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T13:55:09.313 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T13:55:09.313 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2, 2024-09-21T13:55:09.313 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2, 2024-09-21T13:55:09.313 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1, 2024-09-21T13:55:09.313 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2, 2024-09-21T13:55:09.313 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2, 2024-09-21T13:55:09.313 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.86855697631835938, 2024-09-21T13:55:09.313 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.87000000476837158, 2024-09-21T13:55:09.313 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.87000000476837158 2024-09-21T13:55:09.314 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:09.314 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:09.314 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T13:55:09.314 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-09-21T13:55:09.314 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.314 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 0, 2024-09-21T13:55:09.314 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "2595dca9-6219-4e0c-8728-751eb102e73a", 2024-09-21T13:55:09.314 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T13:55:09.314 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T13:55:09.314 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T13:55:09.314 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 0.1215972900390625, 2024-09-21T13:55:09.314 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T13:55:09.314 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T13:55:09.315 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 8, 2024-09-21T13:55:09.315 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 183, 2024-09-21T13:55:09.315 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T13:55:09.315 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T13:55:09.315 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T13:55:09.315 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:55:09.315 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.315 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:55:09.315 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6802", 2024-09-21T13:55:09.315 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T13:55:09.315 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.315 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.315 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:55:09.316 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6803", 2024-09-21T13:55:09.316 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T13:55:09.316 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:09.316 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:55:09.316 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.316 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T13:55:09.316 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:55:09.316 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.316 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:55:09.316 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6804", 2024-09-21T13:55:09.316 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T13:55:09.316 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.316 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.317 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:55:09.317 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6805", 2024-09-21T13:55:09.317 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T13:55:09.317 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:09.317 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:55:09.317 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.317 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T13:55:09.317 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:55:09.317 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.317 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:55:09.317 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6808", 2024-09-21T13:55:09.317 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T13:55:09.317 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.318 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.318 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:55:09.318 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6809", 2024-09-21T13:55:09.318 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T13:55:09.318 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:09.318 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:55:09.318 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.318 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T13:55:09.318 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:55:09.318 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.318 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:55:09.318 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6806", 2024-09-21T13:55:09.319 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T13:55:09.319 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.319 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.319 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:55:09.319 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6807", 2024-09-21T13:55:09.319 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T13:55:09.319 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:09.319 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:55:09.319 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.319 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6803/2095561412", 2024-09-21T13:55:09.319 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6805/2095561412", 2024-09-21T13:55:09.319 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6809/2095561412", 2024-09-21T13:55:09.320 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6807/2095561412", 2024-09-21T13:55:09.320 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T13:55:09.320 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T13:55:09.320 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T13:55:09.320 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:55:09.320 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.320 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.320 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 1, 2024-09-21T13:55:09.320 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "0ba327f9-5524-4a40-abcc-729ded0b1917", 2024-09-21T13:55:09.320 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T13:55:09.320 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T13:55:09.321 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T13:55:09.321 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T13:55:09.321 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T13:55:09.321 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T13:55:09.321 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 13, 2024-09-21T13:55:09.321 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 183, 2024-09-21T13:55:09.321 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T13:55:09.321 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T13:55:09.321 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T13:55:09.321 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:55:09.321 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.321 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:55:09.322 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6810", 2024-09-21T13:55:09.322 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T13:55:09.322 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.322 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.322 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:55:09.322 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6811", 2024-09-21T13:55:09.322 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T13:55:09.322 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:09.322 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:55:09.322 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.322 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T13:55:09.322 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:55:09.323 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.323 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:55:09.323 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6812", 2024-09-21T13:55:09.323 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T13:55:09.323 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.323 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.323 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:55:09.323 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6813", 2024-09-21T13:55:09.323 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T13:55:09.323 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:09.323 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:55:09.323 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.324 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T13:55:09.324 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:55:09.324 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.324 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:55:09.324 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6816", 2024-09-21T13:55:09.324 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T13:55:09.324 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.324 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.324 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:55:09.324 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6817", 2024-09-21T13:55:09.324 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T13:55:09.324 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:09.324 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:55:09.325 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.325 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T13:55:09.325 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:55:09.325 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.325 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:55:09.325 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6814", 2024-09-21T13:55:09.325 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T13:55:09.325 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.325 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.325 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:55:09.325 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6815", 2024-09-21T13:55:09.325 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T13:55:09.325 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:09.326 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:55:09.326 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.326 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6811/1382840527", 2024-09-21T13:55:09.326 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6813/1382840527", 2024-09-21T13:55:09.326 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6817/1382840527", 2024-09-21T13:55:09.326 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6815/1382840527", 2024-09-21T13:55:09.326 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T13:55:09.326 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T13:55:09.326 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T13:55:09.326 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:55:09.326 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.326 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.326 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 2, 2024-09-21T13:55:09.327 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "27a45750-204d-48fd-bb8f-ebb01273343d", 2024-09-21T13:55:09.327 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T13:55:09.327 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T13:55:09.327 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T13:55:09.327 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 0.8268585205078125, 2024-09-21T13:55:09.327 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 18, 2024-09-21T13:55:09.327 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 100, 2024-09-21T13:55:09.327 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 130, 2024-09-21T13:55:09.327 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 160, 2024-09-21T13:55:09.327 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 101, 2024-09-21T13:55:09.327 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T13:55:09.327 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T13:55:09.328 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:55:09.328 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.328 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:55:09.328 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6818", 2024-09-21T13:55:09.328 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T13:55:09.328 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.328 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.328 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:55:09.328 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6819", 2024-09-21T13:55:09.328 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T13:55:09.328 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:09.328 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:55:09.328 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.329 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T13:55:09.329 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:55:09.329 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.329 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:55:09.329 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6820", 2024-09-21T13:55:09.329 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T13:55:09.329 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.329 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.329 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:55:09.329 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6821", 2024-09-21T13:55:09.329 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T13:55:09.329 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:09.329 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:55:09.330 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.330 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T13:55:09.330 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:55:09.330 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.330 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:55:09.330 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6824", 2024-09-21T13:55:09.330 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T13:55:09.330 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.330 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.330 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:55:09.330 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6825", 2024-09-21T13:55:09.330 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T13:55:09.330 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:09.331 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:55:09.331 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.331 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T13:55:09.331 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:55:09.331 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.331 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:55:09.331 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6822", 2024-09-21T13:55:09.331 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T13:55:09.331 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.331 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.331 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:55:09.331 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6823", 2024-09-21T13:55:09.331 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T13:55:09.332 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:09.332 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:55:09.332 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.332 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6819/1945314741", 2024-09-21T13:55:09.332 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6821/1945314741", 2024-09-21T13:55:09.332 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6825/1945314741", 2024-09-21T13:55:09.332 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6823/1945314741", 2024-09-21T13:55:09.332 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T13:55:09.332 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T13:55:09.332 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T13:55:09.332 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:55:09.332 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.333 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.333 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 3, 2024-09-21T13:55:09.333 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "528bc390-3dc9-4f3b-ab75-d7412c9625db", 2024-09-21T13:55:09.333 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T13:55:09.333 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T13:55:09.333 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T13:55:09.333 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T13:55:09.333 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T13:55:09.333 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T13:55:09.333 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 27, 2024-09-21T13:55:09.333 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 183, 2024-09-21T13:55:09.333 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T13:55:09.334 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T13:55:09.334 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T13:55:09.334 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:55:09.334 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.334 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:55:09.334 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6826", 2024-09-21T13:55:09.334 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1981113123 2024-09-21T13:55:09.334 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.334 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.334 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:55:09.334 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6827", 2024-09-21T13:55:09.335 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1981113123 2024-09-21T13:55:09.335 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:09.335 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:55:09.335 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.335 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T13:55:09.335 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:55:09.335 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.335 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:55:09.335 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6828", 2024-09-21T13:55:09.335 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1981113123 2024-09-21T13:55:09.335 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.335 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.336 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:55:09.336 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6829", 2024-09-21T13:55:09.336 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1981113123 2024-09-21T13:55:09.336 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:09.336 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:55:09.336 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.336 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T13:55:09.336 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:55:09.336 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.336 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:55:09.336 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6832", 2024-09-21T13:55:09.336 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1981113123 2024-09-21T13:55:09.337 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.337 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.337 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:55:09.337 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6833", 2024-09-21T13:55:09.337 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1981113123 2024-09-21T13:55:09.337 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:09.337 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:55:09.337 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.337 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T13:55:09.337 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:55:09.337 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.337 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:55:09.337 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6830", 2024-09-21T13:55:09.338 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1981113123 2024-09-21T13:55:09.338 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.338 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.338 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:55:09.338 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6831", 2024-09-21T13:55:09.338 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1981113123 2024-09-21T13:55:09.338 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:09.338 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:55:09.338 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.339 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 51 16 86218 86202 105.62 70.6875 0.0198012 0.00946226 2024-09-21T13:55:09.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:09.340 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:09.340 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6827/1981113123", 2024-09-21T13:55:09.340 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6829/1981113123", 2024-09-21T13:55:09.340 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6833/1981113123", 2024-09-21T13:55:09.340 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6831/1981113123", 2024-09-21T13:55:09.340 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T13:55:09.340 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T13:55:09.340 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T13:55:09.341 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:55:09.341 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.341 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.341 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 4, 2024-09-21T13:55:09.341 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "6fc64698-2d37-4db0-bf3d-c7e612c45163", 2024-09-21T13:55:09.341 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T13:55:09.341 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T13:55:09.341 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T13:55:09.341 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T13:55:09.341 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T13:55:09.341 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T13:55:09.341 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 33, 2024-09-21T13:55:09.342 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 183, 2024-09-21T13:55:09.342 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T13:55:09.342 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T13:55:09.342 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T13:55:09.342 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:55:09.342 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.342 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:55:09.342 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6800", 2024-09-21T13:55:09.342 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T13:55:09.342 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.342 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.343 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:55:09.343 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6801", 2024-09-21T13:55:09.343 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T13:55:09.343 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:09.343 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:55:09.343 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.343 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T13:55:09.343 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:55:09.343 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.343 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:55:09.343 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6802", 2024-09-21T13:55:09.343 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T13:55:09.344 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.344 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.344 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:55:09.344 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6803", 2024-09-21T13:55:09.344 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T13:55:09.344 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:09.344 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:55:09.344 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.344 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T13:55:09.344 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:55:09.344 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.344 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:55:09.345 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6806", 2024-09-21T13:55:09.345 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T13:55:09.345 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.345 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.345 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:55:09.345 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6807", 2024-09-21T13:55:09.345 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T13:55:09.345 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:09.345 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:55:09.345 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.345 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T13:55:09.346 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:55:09.346 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.346 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:55:09.346 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6804", 2024-09-21T13:55:09.346 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T13:55:09.346 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.346 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.346 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:55:09.346 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6805", 2024-09-21T13:55:09.346 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T13:55:09.346 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:09.346 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:55:09.347 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.347 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.187:6801/1702447776", 2024-09-21T13:55:09.347 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.187:6803/1702447776", 2024-09-21T13:55:09.347 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.187:6807/1702447776", 2024-09-21T13:55:09.347 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.187:6805/1702447776", 2024-09-21T13:55:09.347 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T13:55:09.347 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T13:55:09.347 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T13:55:09.347 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:55:09.347 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.347 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.347 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 5, 2024-09-21T13:55:09.348 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "8d7f62a1-7e92-4e06-9cad-86b96be514d8", 2024-09-21T13:55:09.348 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T13:55:09.348 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T13:55:09.348 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T13:55:09.348 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T13:55:09.348 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T13:55:09.348 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T13:55:09.348 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 39, 2024-09-21T13:55:09.348 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 183, 2024-09-21T13:55:09.348 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T13:55:09.348 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T13:55:09.348 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T13:55:09.349 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:55:09.349 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.349 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:55:09.349 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6808", 2024-09-21T13:55:09.349 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3659050776 2024-09-21T13:55:09.349 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.349 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.349 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:55:09.349 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6809", 2024-09-21T13:55:09.349 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3659050776 2024-09-21T13:55:09.349 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:09.349 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:55:09.350 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.350 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T13:55:09.350 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:55:09.350 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.350 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:55:09.350 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6810", 2024-09-21T13:55:09.350 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3659050776 2024-09-21T13:55:09.350 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.350 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.350 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:55:09.350 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6811", 2024-09-21T13:55:09.350 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3659050776 2024-09-21T13:55:09.351 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:09.351 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:55:09.351 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.351 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T13:55:09.351 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:55:09.351 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.351 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:55:09.351 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6814", 2024-09-21T13:55:09.351 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3659050776 2024-09-21T13:55:09.351 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.351 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.351 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:55:09.351 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6815", 2024-09-21T13:55:09.352 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3659050776 2024-09-21T13:55:09.352 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:09.352 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:55:09.352 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.352 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T13:55:09.352 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:55:09.352 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.352 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:55:09.352 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6812", 2024-09-21T13:55:09.352 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3659050776 2024-09-21T13:55:09.352 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.352 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.353 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:55:09.353 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6813", 2024-09-21T13:55:09.353 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3659050776 2024-09-21T13:55:09.353 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:09.353 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:55:09.353 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.353 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.187:6809/3659050776", 2024-09-21T13:55:09.353 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.187:6811/3659050776", 2024-09-21T13:55:09.353 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.187:6815/3659050776", 2024-09-21T13:55:09.353 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.187:6813/3659050776", 2024-09-21T13:55:09.353 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T13:55:09.354 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T13:55:09.354 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T13:55:09.354 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:55:09.354 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.354 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.354 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 6, 2024-09-21T13:55:09.354 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "3a7afd13-26ce-444c-9cc0-81ce2eab6ce5", 2024-09-21T13:55:09.354 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T13:55:09.354 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T13:55:09.354 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T13:55:09.354 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T13:55:09.354 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T13:55:09.355 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T13:55:09.355 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 47, 2024-09-21T13:55:09.355 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 183, 2024-09-21T13:55:09.355 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T13:55:09.355 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T13:55:09.355 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T13:55:09.355 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:55:09.355 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.355 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:55:09.355 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6816", 2024-09-21T13:55:09.355 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T13:55:09.355 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.356 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.356 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:55:09.356 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6817", 2024-09-21T13:55:09.356 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T13:55:09.356 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:09.356 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:55:09.356 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.356 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T13:55:09.356 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:55:09.356 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.356 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:55:09.356 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6818", 2024-09-21T13:55:09.357 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T13:55:09.357 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.357 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.357 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:55:09.357 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6819", 2024-09-21T13:55:09.357 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T13:55:09.357 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:09.357 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:55:09.357 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.357 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T13:55:09.357 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:55:09.357 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.357 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:55:09.358 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6822", 2024-09-21T13:55:09.358 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T13:55:09.358 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.358 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.358 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:55:09.358 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6823", 2024-09-21T13:55:09.358 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T13:55:09.358 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:09.358 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:55:09.358 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.358 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T13:55:09.359 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:55:09.359 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.359 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:55:09.359 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6820", 2024-09-21T13:55:09.359 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T13:55:09.359 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.359 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.359 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:55:09.359 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6821", 2024-09-21T13:55:09.359 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T13:55:09.359 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:09.359 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:55:09.359 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.360 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.187:6817/2728426525", 2024-09-21T13:55:09.360 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.187:6819/2728426525", 2024-09-21T13:55:09.360 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.187:6823/2728426525", 2024-09-21T13:55:09.360 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.187:6821/2728426525", 2024-09-21T13:55:09.360 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T13:55:09.360 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T13:55:09.360 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T13:55:09.360 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:55:09.360 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.360 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.360 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 7, 2024-09-21T13:55:09.360 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "1e08d891-8747-402e-b4c2-aa3802ce8537", 2024-09-21T13:55:09.361 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T13:55:09.361 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T13:55:09.361 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T13:55:09.361 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T13:55:09.361 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T13:55:09.361 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T13:55:09.361 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 54, 2024-09-21T13:55:09.361 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 178, 2024-09-21T13:55:09.361 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T13:55:09.361 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T13:55:09.361 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T13:55:09.362 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:55:09.362 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.362 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:55:09.362 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6824", 2024-09-21T13:55:09.362 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3680850037 2024-09-21T13:55:09.362 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.362 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.362 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:55:09.362 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6825", 2024-09-21T13:55:09.362 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3680850037 2024-09-21T13:55:09.362 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:09.362 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:55:09.363 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.363 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T13:55:09.363 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:55:09.363 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.363 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:55:09.363 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6826", 2024-09-21T13:55:09.363 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3680850037 2024-09-21T13:55:09.363 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.363 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.363 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:55:09.363 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6827", 2024-09-21T13:55:09.364 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3680850037 2024-09-21T13:55:09.364 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:09.364 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:55:09.364 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.364 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T13:55:09.364 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:55:09.364 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.364 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:55:09.364 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6830", 2024-09-21T13:55:09.364 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3680850037 2024-09-21T13:55:09.364 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.365 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.365 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:55:09.365 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6831", 2024-09-21T13:55:09.365 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3680850037 2024-09-21T13:55:09.365 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:09.365 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:55:09.365 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.365 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T13:55:09.365 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:55:09.365 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.365 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:55:09.365 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6828", 2024-09-21T13:55:09.366 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3680850037 2024-09-21T13:55:09.366 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.366 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.366 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:55:09.366 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6829", 2024-09-21T13:55:09.366 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3680850037 2024-09-21T13:55:09.366 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:09.366 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:55:09.366 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.366 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.187:6825/3680850037", 2024-09-21T13:55:09.366 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.187:6827/3680850037", 2024-09-21T13:55:09.366 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.187:6831/3680850037", 2024-09-21T13:55:09.367 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.187:6829/3680850037", 2024-09-21T13:55:09.367 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T13:55:09.367 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T13:55:09.367 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T13:55:09.367 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:55:09.367 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:09.367 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T13:55:09.367 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_xinfo": [ 2024-09-21T13:55:09.367 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.367 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 0, 2024-09-21T13:55:09.367 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T13:55:09.367 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T13:55:09.368 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T13:55:09.368 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T13:55:09.368 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T13:55:09.368 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T13:54:05.775310+0000", 2024-09-21T13:55:09.368 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T13:55:09.368 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.368 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.368 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 1, 2024-09-21T13:55:09.368 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T13:55:09.368 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T13:55:09.369 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T13:55:09.369 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T13:55:09.369 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T13:55:09.369 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T13:54:15.859507+0000", 2024-09-21T13:55:09.369 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T13:55:09.369 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.369 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.369 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 2, 2024-09-21T13:55:09.369 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T13:55:09.369 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T13:55:09.369 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T13:55:09.370 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T13:55:09.370 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T13:55:09.370 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T13:54:46.037848+0000", 2024-09-21T13:55:09.370 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 100 2024-09-21T13:55:09.370 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.370 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.370 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 3, 2024-09-21T13:55:09.370 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T13:55:09.370 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T13:55:09.370 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T13:55:09.370 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T13:55:09.371 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T13:55:09.371 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T13:54:35.626926+0000", 2024-09-21T13:55:09.371 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T13:55:09.371 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.371 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.371 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 4, 2024-09-21T13:55:09.371 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T13:55:09.371 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T13:55:09.371 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T13:55:09.371 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T13:55:09.371 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T13:55:09.372 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T13:53:57.366601+0000", 2024-09-21T13:55:09.372 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T13:55:09.372 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.372 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.372 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 5, 2024-09-21T13:55:09.372 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T13:55:09.372 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T13:55:09.372 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T13:55:09.372 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T13:55:09.372 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T13:55:09.372 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T13:53:51.162637+0000", 2024-09-21T13:55:09.372 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T13:55:09.373 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.373 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.373 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 6, 2024-09-21T13:55:09.373 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T13:55:09.373 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T13:55:09.373 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T13:55:09.373 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T13:55:09.373 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T13:55:09.373 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T13:54:26.057508+0000", 2024-09-21T13:55:09.373 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T13:55:09.373 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.373 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.374 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 7, 2024-09-21T13:55:09.374 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T13:55:09.374 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T13:55:09.374 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T13:55:09.374 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T13:55:09.374 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T13:55:09.374 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T13:54:25.511045+0000", 2024-09-21T13:55:09.374 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T13:55:09.374 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:09.374 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T13:55:09.374 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap": [ 2024-09-21T13:55:09.374 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.375 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "3.14", 2024-09-21T13:55:09.375 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-09-21T13:55:09.375 INFO:teuthology.orchestra.run.smithi049.stdout: 7, 2024-09-21T13:55:09.375 INFO:teuthology.orchestra.run.smithi049.stdout: 6, 2024-09-21T13:55:09.375 INFO:teuthology.orchestra.run.smithi049.stdout: 0 2024-09-21T13:55:09.375 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:55:09.375 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:09.375 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T13:55:09.375 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap_items": [ 2024-09-21T13:55:09.375 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.375 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "4.6", 2024-09-21T13:55:09.376 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-09-21T13:55:09.376 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.376 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 1, 2024-09-21T13:55:09.376 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 5 2024-09-21T13:55:09.376 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:09.376 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:55:09.376 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.376 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.376 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "4.1b", 2024-09-21T13:55:09.376 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-09-21T13:55:09.376 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.376 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 1, 2024-09-21T13:55:09.376 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 2 2024-09-21T13:55:09.377 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:09.377 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:55:09.377 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.377 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.377 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "4.1f", 2024-09-21T13:55:09.377 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-09-21T13:55:09.377 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:09.377 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 1, 2024-09-21T13:55:09.377 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 2 2024-09-21T13:55:09.377 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:09.377 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:55:09.377 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:09.377 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T13:55:09.378 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap_primaries": [], 2024-09-21T13:55:09.378 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_temp": [], 2024-09-21T13:55:09.378 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_temp": [], 2024-09-21T13:55:09.378 INFO:teuthology.orchestra.run.smithi049.stdout: "blocklist": { 2024-09-21T13:55:09.378 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/2731793917": "2024-09-22T13:49:27.311313+0000", 2024-09-21T13:55:09.378 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/799831425": "2024-09-22T13:49:27.311313+0000", 2024-09-21T13:55:09.378 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/2518197280": "2024-09-22T13:43:28.110499+0000", 2024-09-21T13:55:09.378 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3556829486": "2024-09-22T13:43:28.110499+0000", 2024-09-21T13:55:09.378 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/1249926115": "2024-09-22T13:49:27.311313+0000", 2024-09-21T13:55:09.378 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/231530138": "2024-09-22T13:43:28.110499+0000", 2024-09-21T13:55:09.378 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/4220781134": "2024-09-22T13:49:27.311313+0000", 2024-09-21T13:55:09.378 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/202325455": "2024-09-22T13:43:28.110499+0000", 2024-09-21T13:55:09.379 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/4129636473": "2024-09-22T13:43:56.633723+0000", 2024-09-21T13:55:09.379 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/2518197280": "2024-09-22T13:43:28.110499+0000", 2024-09-21T13:55:09.379 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3514741273": "2024-09-22T13:43:56.633723+0000", 2024-09-21T13:55:09.379 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/4129636473": "2024-09-22T13:43:56.633723+0000", 2024-09-21T13:55:09.379 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3125743776": "2024-09-22T13:43:56.633723+0000", 2024-09-21T13:55:09.379 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3600977363": "2024-09-22T13:43:56.633723+0000", 2024-09-21T13:55:09.379 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/3453571598": "2024-09-22T13:49:27.311313+0000", 2024-09-21T13:55:09.379 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/3453571598": "2024-09-22T13:49:27.311313+0000", 2024-09-21T13:55:09.379 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3553734432": "2024-09-22T13:49:27.311313+0000" 2024-09-21T13:55:09.379 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.379 INFO:teuthology.orchestra.run.smithi049.stdout: "range_blocklist": {}, 2024-09-21T13:55:09.379 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profiles": { 2024-09-21T13:55:09.380 INFO:teuthology.orchestra.run.smithi049.stdout: "default": { 2024-09-21T13:55:09.380 INFO:teuthology.orchestra.run.smithi049.stdout: "crush-failure-domain": "osd", 2024-09-21T13:55:09.380 INFO:teuthology.orchestra.run.smithi049.stdout: "k": "2", 2024-09-21T13:55:09.380 INFO:teuthology.orchestra.run.smithi049.stdout: "m": "1", 2024-09-21T13:55:09.380 INFO:teuthology.orchestra.run.smithi049.stdout: "plugin": "jerasure", 2024-09-21T13:55:09.380 INFO:teuthology.orchestra.run.smithi049.stdout: "technique": "reed_sol_van" 2024-09-21T13:55:09.380 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:09.380 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:09.380 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps_queue": [], 2024-09-21T13:55:09.380 INFO:teuthology.orchestra.run.smithi049.stdout: "new_removed_snaps": [], 2024-09-21T13:55:09.380 INFO:teuthology.orchestra.run.smithi049.stdout: "new_purged_snaps": [], 2024-09-21T13:55:09.380 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_node_flags": {}, 2024-09-21T13:55:09.381 INFO:teuthology.orchestra.run.smithi049.stdout: "device_class_flags": {}, 2024-09-21T13:55:09.381 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode": { 2024-09-21T13:55:09.381 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode_enabled": false, 2024-09-21T13:55:09.381 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_bucket_count": 0, 2024-09-21T13:55:09.381 INFO:teuthology.orchestra.run.smithi049.stdout: "degraded_stretch_mode": 0, 2024-09-21T13:55:09.381 INFO:teuthology.orchestra.run.smithi049.stdout: "recovering_stretch_mode": 0, 2024-09-21T13:55:09.381 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode_bucket": 0 2024-09-21T13:55:09.381 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:09.381 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T13:55:09.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:55:09.382 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:08 smithi187 ceph-mon[33032]: 3.1e scrub ok 2024-09-21T13:55:09.382 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:08 smithi187 ceph-mon[33032]: 3.1b scrub starts 2024-09-21T13:55:09.382 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:08 smithi187 ceph-mon[33032]: 3.1b scrub ok 2024-09-21T13:55:09.382 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:08 smithi187 ceph-mon[33032]: pgmap v301: 148 pgs: 148 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 651 B/s rd, 49 MiB/s wr, 782 op/s 2024-09-21T13:55:09.482 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:55:09.523 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:09.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:09.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:55:09.625 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:55:09.653 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:09.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:09.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:55:09.694 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:55:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:55:09.756 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:55:09.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:09.796 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:09.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:55:09.897 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:55:09.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:09.935 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:09.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:55:10.036 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:55:10.069 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:10.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:10.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:55:10.170 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:55:10.203 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:10.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:10.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:55:10.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:09 smithi049 ceph-mon[40489]: 6.12 scrub starts 2024-09-21T13:55:10.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:09 smithi049 ceph-mon[40489]: 6.12 scrub ok 2024-09-21T13:55:10.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:09 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2837963557' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T13:55:10.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:09 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:55:10.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:09 smithi049 ceph-mon[30080]: 6.12 scrub starts 2024-09-21T13:55:10.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:09 smithi049 ceph-mon[30080]: 6.12 scrub ok 2024-09-21T13:55:10.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:09 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2837963557' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T13:55:10.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:09 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:55:10.295 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 52 16 87489 87473 105.114 79.4375 0.00479166 0.00950744 2024-09-21T13:55:10.305 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:55:10.329 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:10.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:10.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:55:10.331 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T13:55:10.431 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:55:10.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:09 smithi187 ceph-mon[33032]: 6.12 scrub starts 2024-09-21T13:55:10.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:09 smithi187 ceph-mon[33032]: 6.12 scrub ok 2024-09-21T13:55:10.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:09 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2837963557' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T13:55:10.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:09 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:55:10.480 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:10.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:10.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:55:10.582 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:55:10.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:10.652 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:10.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:55:10.753 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:55:10.794 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:10.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:10.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:55:10.896 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:10.933 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:10.933 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:55:10 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:55:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:55:10.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:10.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:11.035 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:55:11.075 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:11.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:11.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:55:11.177 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:55:11.220 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:11.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:11.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:55:11.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:10 smithi049 ceph-mon[40489]: 6.b scrub starts 2024-09-21T13:55:11.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:10 smithi049 ceph-mon[40489]: 6.b scrub ok 2024-09-21T13:55:11.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:10 smithi049 ceph-mon[40489]: pgmap v302: 148 pgs: 148 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 94 MiB/s wr, 1.50k op/s 2024-09-21T13:55:11.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:10 smithi049 ceph-mon[30080]: 6.b scrub starts 2024-09-21T13:55:11.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:10 smithi049 ceph-mon[30080]: 6.b scrub ok 2024-09-21T13:55:11.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:10 smithi049 ceph-mon[30080]: pgmap v302: 148 pgs: 148 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 94 MiB/s wr, 1.50k op/s 2024-09-21T13:55:11.296 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 53 16 88705 88689 104.565 76 0.0130768 0.00955859 2024-09-21T13:55:11.322 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:55:11.360 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:11.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:11.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:55:11.421 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:55:11.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:10 smithi187 ceph-mon[33032]: 6.b scrub starts 2024-09-21T13:55:11.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:10 smithi187 ceph-mon[33032]: 6.b scrub ok 2024-09-21T13:55:11.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:10 smithi187 ceph-mon[33032]: pgmap v302: 148 pgs: 148 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 94 MiB/s wr, 1.50k op/s 2024-09-21T13:55:11.448 INFO:tasks.thrashosds.thrasher:j is {'epoch': 202, 'fsid': '40a3d2e8-781f-11ef-baf6-efdc52797490', 'created': '2024-09-21T13:42:57.838275+0000', 'modified': '2024-09-21T13:55:04.933637+0000', 'last_up_change': '2024-09-21T13:52:03.297364+0000', 'last_in_change': '2024-09-21T13:48:23.514355+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 19, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 8, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T13:46:21.096244+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '199', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.8685569763183594, 'average_primary_affinity': 0.8700000047683716, 'average_primary_affinity_weighted': 0.8700000047683716}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-21T13:48:55.540712+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 63, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 0.8685569763183594, 'average_primary_affinity': 0.8700000047683716, 'average_primary_affinity_weighted': 0.8700000047683716}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-21T13:48:56.417456+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8685569763183594, 'average_primary_affinity': 0.8700000047683716, 'average_primary_affinity_weighted': 0.8700000047683716}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-21T13:48:58.128455+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.8685569763183594, 'average_primary_affinity': 0.8700000047683716, 'average_primary_affinity_weighted': 0.8700000047683716}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-21T13:49:00.069361+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 0.8685569763183594, 'average_primary_affinity': 0.8700000047683716, 'average_primary_affinity_weighted': 0.8700000047683716}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-21T13:49:02.127399+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8685569763183594, 'average_primary_affinity': 0.8700000047683716, 'average_primary_affinity_weighted': 0.8700000047683716}}, {'pool': 8, 'pool_name': 'unique_pool_1', 'create_time': '2024-09-21T13:54:10.659855+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '187', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8685569763183594, 'average_primary_affinity': 0.8700000047683716, 'average_primary_affinity_weighted': 0.8700000047683716}}], 'osds': [{'osd': 0, 'uuid': '2595dca9-6219-4e0c-8728-751eb102e73a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.1215972900390625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 183, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6802', 'nonce': 2095561412}, {'type': 'v1', 'addr': '172.21.15.49:6803', 'nonce': 2095561412}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6804', 'nonce': 2095561412}, {'type': 'v1', 'addr': '172.21.15.49:6805', 'nonce': 2095561412}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6808', 'nonce': 2095561412}, {'type': 'v1', 'addr': '172.21.15.49:6809', 'nonce': 2095561412}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6806', 'nonce': 2095561412}, {'type': 'v1', 'addr': '172.21.15.49:6807', 'nonce': 2095561412}]}, 'public_addr': '172.21.15.49:6803/2095561412', 'cluster_addr': '172.21.15.49:6805/2095561412', 'heartbeat_back_addr': '172.21.15.49:6809/2095561412', 'heartbeat_front_addr': '172.21.15.49:6807/2095561412', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '0ba327f9-5524-4a40-abcc-729ded0b1917', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 183, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6810', 'nonce': 1382840527}, {'type': 'v1', 'addr': '172.21.15.49:6811', 'nonce': 1382840527}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6812', 'nonce': 1382840527}, {'type': 'v1', 'addr': '172.21.15.49:6813', 'nonce': 1382840527}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6816', 'nonce': 1382840527}, {'type': 'v1', 'addr': '172.21.15.49:6817', 'nonce': 1382840527}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6814', 'nonce': 1382840527}, {'type': 'v1', 'addr': '172.21.15.49:6815', 'nonce': 1382840527}]}, 'public_addr': '172.21.15.49:6811/1382840527', 'cluster_addr': '172.21.15.49:6813/1382840527', 'heartbeat_back_addr': '172.21.15.49:6817/1382840527', 'heartbeat_front_addr': '172.21.15.49:6815/1382840527', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '27a45750-204d-48fd-bb8f-ebb01273343d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.8268585205078125, 'last_clean_begin': 18, 'last_clean_end': 100, 'up_from': 130, 'up_thru': 160, 'down_at': 101, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6818', 'nonce': 1945314741}, {'type': 'v1', 'addr': '172.21.15.49:6819', 'nonce': 1945314741}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6820', 'nonce': 1945314741}, {'type': 'v1', 'addr': '172.21.15.49:6821', 'nonce': 1945314741}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6824', 'nonce': 1945314741}, {'type': 'v1', 'addr': '172.21.15.49:6825', 'nonce': 1945314741}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6822', 'nonce': 1945314741}, {'type': 'v1', 'addr': '172.21.15.49:6823', 'nonce': 1945314741}]}, 'public_addr': '172.21.15.49:6819/1945314741', 'cluster_addr': '172.21.15.49:6821/1945314741', 'heartbeat_back_addr': '172.21.15.49:6825/1945314741', 'heartbeat_front_addr': '172.21.15.49:6823/1945314741', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '528bc390-3dc9-4f3b-ab75-d7412c9625db', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 183, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6826', 'nonce': 1981113123}, {'type': 'v1', 'addr': '172.21.15.49:6827', 'nonce': 1981113123}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6828', 'nonce': 1981113123}, {'type': 'v1', 'addr': '172.21.15.49:6829', 'nonce': 1981113123}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6832', 'nonce': 1981113123}, {'type': 'v1', 'addr': '172.21.15.49:6833', 'nonce': 1981113123}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6830', 'nonce': 1981113123}, {'type': 'v1', 'addr': '172.21.15.49:6831', 'nonce': 1981113123}]}, 'public_addr': '172.21.15.49:6827/1981113123', 'cluster_addr': '172.21.15.49:6829/1981113123', 'heartbeat_back_addr': '172.21.15.49:6833/1981113123', 'heartbeat_front_addr': '172.21.15.49:6831/1981113123', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '6fc64698-2d37-4db0-bf3d-c7e612c45163', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 183, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6800', 'nonce': 1702447776}, {'type': 'v1', 'addr': '172.21.15.187:6801', 'nonce': 1702447776}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6802', 'nonce': 1702447776}, {'type': 'v1', 'addr': '172.21.15.187:6803', 'nonce': 1702447776}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6806', 'nonce': 1702447776}, {'type': 'v1', 'addr': '172.21.15.187:6807', 'nonce': 1702447776}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6804', 'nonce': 1702447776}, {'type': 'v1', 'addr': '172.21.15.187:6805', 'nonce': 1702447776}]}, 'public_addr': '172.21.15.187:6801/1702447776', 'cluster_addr': '172.21.15.187:6803/1702447776', 'heartbeat_back_addr': '172.21.15.187:6807/1702447776', 'heartbeat_front_addr': '172.21.15.187:6805/1702447776', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '8d7f62a1-7e92-4e06-9cad-86b96be514d8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 183, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6808', 'nonce': 3659050776}, {'type': 'v1', 'addr': '172.21.15.187:6809', 'nonce': 3659050776}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6810', 'nonce': 3659050776}, {'type': 'v1', 'addr': '172.21.15.187:6811', 'nonce': 3659050776}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6814', 'nonce': 3659050776}, {'type': 'v1', 'addr': '172.21.15.187:6815', 'nonce': 3659050776}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6812', 'nonce': 3659050776}, {'type': 'v1', 'addr': '172.21.15.187:6813', 'nonce': 3659050776}]}, 'public_addr': '172.21.15.187:6809/3659050776', 'cluster_addr': '172.21.15.187:6811/3659050776', 'heartbeat_back_addr': '172.21.15.187:6815/3659050776', 'heartbeat_front_addr': '172.21.15.187:6813/3659050776', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '3a7afd13-26ce-444c-9cc0-81ce2eab6ce5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 183, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6816', 'nonce': 2728426525}, {'type': 'v1', 'addr': '172.21.15.187:6817', 'nonce': 2728426525}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6818', 'nonce': 2728426525}, {'type': 'v1', 'addr': '172.21.15.187:6819', 'nonce': 2728426525}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6822', 'nonce': 2728426525}, {'type': 'v1', 'addr': '172.21.15.187:6823', 'nonce': 2728426525}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6820', 'nonce': 2728426525}, {'type': 'v1', 'addr': '172.21.15.187:6821', 'nonce': 2728426525}]}, 'public_addr': '172.21.15.187:6817/2728426525', 'cluster_addr': '172.21.15.187:6819/2728426525', 'heartbeat_back_addr': '172.21.15.187:6823/2728426525', 'heartbeat_front_addr': '172.21.15.187:6821/2728426525', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1e08d891-8747-402e-b4c2-aa3802ce8537', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 178, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 3680850037}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 3680850037}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 3680850037}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 3680850037}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6830', 'nonce': 3680850037}, {'type': 'v1', 'addr': '172.21.15.187:6831', 'nonce': 3680850037}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6828', 'nonce': 3680850037}, {'type': 'v1', 'addr': '172.21.15.187:6829', 'nonce': 3680850037}]}, 'public_addr': '172.21.15.187:6825/3680850037', 'cluster_addr': '172.21.15.187:6827/3680850037', 'heartbeat_back_addr': '172.21.15.187:6831/3680850037', 'heartbeat_front_addr': '172.21.15.187:6829/3680850037', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T13:54:05.775310+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T13:54:15.859507+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T13:54:46.037848+0000', 'dead_epoch': 100}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T13:54:35.626926+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T13:53:57.366601+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T13:53:51.162637+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T13:54:26.057508+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T13:54:25.511045+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '3.14', 'osds': [7, 6, 0]}], 'pg_upmap_items': [{'pgid': '4.6', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.1b', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1f', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.49:0/2731793917': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:0/799831425': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:6801/2518197280': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:0/3556829486': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:0/1249926115': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:0/231530138': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:0/4220781134': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:0/202325455': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:6800/4129636473': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:6800/2518197280': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:0/3514741273': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:6801/4129636473': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:0/3125743776': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:0/3600977363': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:6801/3453571598': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:6800/3453571598': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:0/3553734432': '2024-09-22T13:49:27.311313+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-21T13:55:11.448 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T13:55:11.462 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:11.490 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:11.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:11.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:11.592 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:55:11.644 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:11.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:11.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:55:11.657 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T13:55:11.746 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:55:11.833 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:11.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:11.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:55:11.935 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:55:11.991 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:11.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:11.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:55:12.093 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:55:12.139 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:12.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:12.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:55:12.145 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:55:12.232 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:55:12.241 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:55:12.268 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:12.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:12.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:55:12.297 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 54 16 89726 89710 103.807 63.8125 0.0054109 0.00962837 2024-09-21T13:55:12.370 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:55:12.401 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:12.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:12.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:55:12.502 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:55:12.570 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:12.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:12.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:55:12.671 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:55:12.720 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:12.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:12.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:55:12.822 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:55:12.849 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:12.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:12.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:55:12.947 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:55:12.951 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:55:13.001 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T13:55:13.001 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:55:13.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:13.008 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:13.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:55:13.016 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-09-21T13:55:13.110 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:13.139 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:13.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:13.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:13.191 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:13 smithi049 ceph-mon[40489]: 3.3 scrub starts 2024-09-21T13:55:13.191 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:13 smithi049 ceph-mon[40489]: 3.3 scrub ok 2024-09-21T13:55:13.191 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:13 smithi049 ceph-mon[40489]: pgmap v303: 148 pgs: 148 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 55 MiB/s wr, 873 op/s 2024-09-21T13:55:13.191 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:13 smithi049 ceph-mon[40489]: 4.1f scrub starts 2024-09-21T13:55:13.191 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:13 smithi049 ceph-mon[40489]: 4.1f scrub ok 2024-09-21T13:55:13.192 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:13 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3943003508' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:55:13.192 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:13 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:55:13.192 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:13 smithi049 ceph-mon[40489]: 5.10 scrub starts 2024-09-21T13:55:13.192 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:13 smithi049 ceph-mon[40489]: 5.10 scrub ok 2024-09-21T13:55:13.192 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:13 smithi049 ceph-mon[30080]: 3.3 scrub starts 2024-09-21T13:55:13.192 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:13 smithi049 ceph-mon[30080]: 3.3 scrub ok 2024-09-21T13:55:13.192 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:13 smithi049 ceph-mon[30080]: pgmap v303: 148 pgs: 148 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 55 MiB/s wr, 873 op/s 2024-09-21T13:55:13.192 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:13 smithi049 ceph-mon[30080]: 4.1f scrub starts 2024-09-21T13:55:13.192 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:13 smithi049 ceph-mon[30080]: 4.1f scrub ok 2024-09-21T13:55:13.192 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:13 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3943003508' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:55:13.192 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:13 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:55:13.192 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:13 smithi049 ceph-mon[30080]: 5.10 scrub starts 2024-09-21T13:55:13.193 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:13 smithi049 ceph-mon[30080]: 5.10 scrub ok 2024-09-21T13:55:13.241 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:13.268 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:13.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:13.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:13.297 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 55 16 90852 90836 103.199 70.375 0.0317078 0.0096851 2024-09-21T13:55:13.370 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:55:13.402 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:13.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:13.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:55:13.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:13 smithi187 ceph-mon[33032]: 3.3 scrub starts 2024-09-21T13:55:13.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:13 smithi187 ceph-mon[33032]: 3.3 scrub ok 2024-09-21T13:55:13.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:13 smithi187 ceph-mon[33032]: pgmap v303: 148 pgs: 148 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 55 MiB/s wr, 873 op/s 2024-09-21T13:55:13.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:13 smithi187 ceph-mon[33032]: 4.1f scrub starts 2024-09-21T13:55:13.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:13 smithi187 ceph-mon[33032]: 4.1f scrub ok 2024-09-21T13:55:13.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:13 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3943003508' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:55:13.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:13 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:55:13.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:13 smithi187 ceph-mon[33032]: 5.10 scrub starts 2024-09-21T13:55:13.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:13 smithi187 ceph-mon[33032]: 5.10 scrub ok 2024-09-21T13:55:13.505 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:55:13.543 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:13.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:13.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:55:13.645 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:55:13.681 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:13.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:13.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:55:13.783 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:13.815 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:13.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:13.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:13.917 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:55:13.949 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:13.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:13.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:55:14.051 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:14.079 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:14.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:14.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:14.181 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:14.213 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:14.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:14.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:14.297 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 56 16 92103 92087 102.752 78.1875 0.0102854 0.00972746 2024-09-21T13:55:14.314 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:55:14.358 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:14.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:14.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:55:14.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:14 smithi187 ceph-mon[33032]: 4.17 scrub starts 2024-09-21T13:55:14.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:14 smithi187 ceph-mon[33032]: 4.17 scrub ok 2024-09-21T13:55:14.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:14 smithi187 ceph-mon[33032]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:55:14.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:14 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T13:55:14.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:14 smithi187 ceph-mon[33032]: osdmap e203: 8 total, 8 up, 8 in 2024-09-21T13:55:14.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:14 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:55:14.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:14 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:55:14.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:14 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:55:14.460 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:55:14.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:14.489 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:14.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:55:14.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:14 smithi049 ceph-mon[40489]: 4.17 scrub starts 2024-09-21T13:55:14.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:14 smithi049 ceph-mon[40489]: 4.17 scrub ok 2024-09-21T13:55:14.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:14 smithi049 ceph-mon[40489]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:55:14.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:14 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T13:55:14.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:14 smithi049 ceph-mon[40489]: osdmap e203: 8 total, 8 up, 8 in 2024-09-21T13:55:14.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:14 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:55:14.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:14 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:55:14.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:14 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:55:14.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:14 smithi049 ceph-mon[30080]: 4.17 scrub starts 2024-09-21T13:55:14.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:14 smithi049 ceph-mon[30080]: 4.17 scrub ok 2024-09-21T13:55:14.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:14 smithi049 ceph-mon[30080]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:55:14.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:14 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T13:55:14.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:14 smithi049 ceph-mon[30080]: osdmap e203: 8 total, 8 up, 8 in 2024-09-21T13:55:14.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:14 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:55:14.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:14 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:55:14.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:14 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:55:14.590 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:55:14.617 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:14.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:14.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:55:14.719 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:55:14.757 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:14.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:14.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:55:14.859 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:55:14.895 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:14.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:14.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:55:14.997 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:55:15.032 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:15.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:15.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:55:15.133 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:55:15.157 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:15.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:15.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:55:15.258 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:55:15.293 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:15.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:15.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:55:15.297 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 57 16 93102 93086 102.045 62.4375 0.00914912 0.00979458 2024-09-21T13:55:15.345 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:15 smithi187 ceph-mon[33032]: pgmap v305: 148 pgs: 148 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-09-21T13:55:15.345 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:15 smithi187 ceph-mon[33032]: osdmap e204: 8 total, 8 up, 8 in 2024-09-21T13:55:15.345 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:15 smithi187 ceph-mon[33032]: from='client.25181 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T13:55:15.395 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:55:15.423 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:15.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:15.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:55:15.495 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:55:15.499 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:15 smithi049 ceph-mon[40489]: pgmap v305: 148 pgs: 148 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-09-21T13:55:15.499 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:15 smithi049 ceph-mon[40489]: osdmap e204: 8 total, 8 up, 8 in 2024-09-21T13:55:15.499 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:15 smithi049 ceph-mon[40489]: from='client.25181 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T13:55:15.500 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:15 smithi049 ceph-mon[30080]: pgmap v305: 148 pgs: 148 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-09-21T13:55:15.500 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:15 smithi049 ceph-mon[30080]: osdmap e204: 8 total, 8 up, 8 in 2024-09-21T13:55:15.500 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:15 smithi049 ceph-mon[30080]: from='client.25181 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T13:55:15.505 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-09-21T13:55:15.528 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:55:15.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:15.567 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:15.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:55:15.668 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:55:15.714 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:15.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:15.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:55:15.816 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:55:15.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:15.858 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:15.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:55:15.959 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:55:16.014 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:16.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:16.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:55:16.097 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T13:55:16.116 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:55:16.160 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:55:16.160 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:55:16.193 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:16.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:16.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:55:16.212 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:55:16.213 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:55:16.259 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:55:16.259 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:55:16.294 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:55:16.294 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:55:16.295 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:16.298 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 58 16 94261 94245 101.534 72.4375 0.01073 0.00984311 2024-09-21T13:55:16.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:16.325 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:16.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:16.331 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:55:16.331 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:55:16.365 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:55:16.365 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:55:16.397 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:55:16.398 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:55:16.426 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:55:16.431 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:55:16.431 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:55:16.455 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:16.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:16.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:55:16.557 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:16.586 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:16.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:16.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:16.688 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:55:16.728 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:16.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:16.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:55:16.829 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:16.863 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:16.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:16.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:16.965 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:55:17.006 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:17.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:17.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:55:17.108 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:55:17.152 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:17.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:17.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:55:17.254 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:55:17.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:17.288 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:17.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:55:17.298 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 59 16 95435 95419 101.057 73.375 0.00884867 0.00989062 2024-09-21T13:55:17.390 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:55:17.391 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:17 smithi049 ceph-mon[30080]: pgmap v307: 148 pgs: 148 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-09-21T13:55:17.391 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:17 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2701679214' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:55:17.391 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:17 smithi049 ceph-mon[40489]: pgmap v307: 148 pgs: 148 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-09-21T13:55:17.391 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:17 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2701679214' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:55:17.441 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":306,"stamp":"2024-09-21T13:55:14.051058+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5912992283,"num_objects":90415,"num_object_clones":0,"num_object_copies":271245,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90415,"num_whiteouts":0,"num_read":1214,"num_read_kb":1071,"num_write":90765,"num_write_kb":5775497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":590588,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85075,"ondisk_log_size":85075,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":19023260,"kb_used_data":17276172,"kb_used_omap":15,"kb_used_meta":1746928,"kb_avail":730937956,"statfs":{"total":767960285184,"available":748480466944,"internally_reserved":0,"allocated":17690800128,"data_stored":17686273385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15598,"internal_metadata":1788855058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":690946048,"num_objects":10543,"num_object_clones":0,"num_object_copies":31629,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10543,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10543,"num_write_kb":674752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.255003"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":258,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.015171+0000","last_change":"2024-09-21T13:54:55.873382+0000","last_active":"2024-09-21T13:55:13.015171+0000","last_peered":"2024-09-21T13:55:13.015171+0000","last_clean":"2024-09-21T13:55:13.015171+0000","last_became_active":"2024-09-21T13:52:04.317322+0000","last_became_peered":"2024-09-21T13:52:04.317322+0000","last_unstale":"2024-09-21T13:55:13.015171+0000","last_undegraded":"2024-09-21T13:55:13.015171+0000","last_fullsized":"2024-09-21T13:55:13.015171+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:55.873301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:55.873301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:56:15.283294+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"68'9","reported_seq":325,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.012080+0000","last_change":"2024-09-21T13:54:18.411247+0000","last_active":"2024-09-21T13:55:13.012080+0000","last_peered":"2024-09-21T13:55:13.012080+0000","last_clean":"2024-09-21T13:55:13.012080+0000","last_became_active":"2024-09-21T13:49:00.055210+0000","last_became_peered":"2024-09-21T13:49:00.055210+0000","last_unstale":"2024-09-21T13:55:13.012080+0000","last_undegraded":"2024-09-21T13:55:13.012080+0000","last_fullsized":"2024-09-21T13:55:13.012080+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:54:18.410821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:18.410821+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:40.272886+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":297,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.024460+0000","last_change":"2024-09-21T13:50:28.058990+0000","last_active":"2024-09-21T13:55:13.024460+0000","last_peered":"2024-09-21T13:55:13.024460+0000","last_clean":"2024-09-21T13:55:13.024460+0000","last_became_active":"2024-09-21T13:49:02.060984+0000","last_became_peered":"2024-09-21T13:49:02.060984+0000","last_unstale":"2024-09-21T13:55:13.024460+0000","last_undegraded":"2024-09-21T13:55:13.024460+0000","last_fullsized":"2024-09-21T13:55:13.024460+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:28.058918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:28.058918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:49.006644+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":293,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.012418+0000","last_change":"2024-09-21T13:53:21.005444+0000","last_active":"2024-09-21T13:55:13.012418+0000","last_peered":"2024-09-21T13:55:13.012418+0000","last_clean":"2024-09-21T13:55:13.012418+0000","last_became_active":"2024-09-21T13:49:04.072460+0000","last_became_peered":"2024-09-21T13:49:04.072460+0000","last_unstale":"2024-09-21T13:55:13.012418+0000","last_undegraded":"2024-09-21T13:55:13.012418+0000","last_fullsized":"2024-09-21T13:55:13.012418+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:21.005352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:21.005352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:50.469513+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":291,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-21T13:55:04.949040+0000","last_change":"2024-09-21T13:51:18.572788+0000","last_active":"2024-09-21T13:55:04.949040+0000","last_peered":"2024-09-21T13:55:04.949040+0000","last_clean":"2024-09-21T13:55:04.949040+0000","last_became_active":"2024-09-21T13:49:04.433965+0000","last_became_peered":"2024-09-21T13:49:04.433965+0000","last_unstale":"2024-09-21T13:55:04.949040+0000","last_undegraded":"2024-09-21T13:55:04.949040+0000","last_fullsized":"2024-09-21T13:55:04.949040+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:18.572729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:51:18.572729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:28.969697+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":322,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-21T13:55:07.001213+0000","last_change":"2024-09-21T13:55:07.001213+0000","last_active":"2024-09-21T13:55:07.001213+0000","last_peered":"2024-09-21T13:55:07.001213+0000","last_clean":"2024-09-21T13:55:07.001213+0000","last_became_active":"2024-09-21T13:52:04.316819+0000","last_became_peered":"2024-09-21T13:52:04.316819+0000","last_unstale":"2024-09-21T13:55:07.001213+0000","last_undegraded":"2024-09-21T13:55:07.001213+0000","last_fullsized":"2024-09-21T13:55:07.001213+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:07.001135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:55:07.001135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:56:09.415449+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"68'15","reported_seq":335,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-21T13:55:04.949025+0000","last_change":"2024-09-21T13:54:17.024113+0000","last_active":"2024-09-21T13:55:04.949025+0000","last_peered":"2024-09-21T13:55:04.949025+0000","last_clean":"2024-09-21T13:55:04.949025+0000","last_became_active":"2024-09-21T13:52:04.315101+0000","last_became_peered":"2024-09-21T13:52:04.315101+0000","last_unstale":"2024-09-21T13:55:04.949025+0000","last_undegraded":"2024-09-21T13:55:04.949025+0000","last_fullsized":"2024-09-21T13:55:04.949025+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:54:17.024053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:17.024053+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:34.778666+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":305,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.011978+0000","last_change":"2024-09-21T13:51:43.040397+0000","last_active":"2024-09-21T13:55:13.011978+0000","last_peered":"2024-09-21T13:55:13.011978+0000","last_clean":"2024-09-21T13:55:13.011978+0000","last_became_active":"2024-09-21T13:49:02.063687+0000","last_became_peered":"2024-09-21T13:49:02.063687+0000","last_unstale":"2024-09-21T13:55:13.011978+0000","last_undegraded":"2024-09-21T13:55:13.011978+0000","last_fullsized":"2024-09-21T13:55:13.011978+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:43.040348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:43.040348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":322,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.015022+0000","last_change":"2024-09-21T13:53:59.104109+0000","last_active":"2024-09-21T13:55:13.015022+0000","last_peered":"2024-09-21T13:55:13.015022+0000","last_clean":"2024-09-21T13:55:13.015022+0000","last_became_active":"2024-09-21T13:48:58.050695+0000","last_became_peered":"2024-09-21T13:48:58.050695+0000","last_unstale":"2024-09-21T13:55:13.015022+0000","last_undegraded":"2024-09-21T13:55:13.015022+0000","last_fullsized":"2024-09-21T13:55:13.015022+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:59.104062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:53:59.104062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:28.421324+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"68'9","reported_seq":325,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.012864+0000","last_change":"2024-09-21T13:53:58.308240+0000","last_active":"2024-09-21T13:55:13.012864+0000","last_peered":"2024-09-21T13:55:13.012864+0000","last_clean":"2024-09-21T13:55:13.012864+0000","last_became_active":"2024-09-21T13:49:00.059382+0000","last_became_peered":"2024-09-21T13:49:00.059382+0000","last_unstale":"2024-09-21T13:55:13.012864+0000","last_undegraded":"2024-09-21T13:55:13.012864+0000","last_fullsized":"2024-09-21T13:55:13.012864+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:53:58.308144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:58.308144+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:27.655659+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":290,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.014544+0000","last_change":"2024-09-21T13:49:02.066978+0000","last_active":"2024-09-21T13:55:13.014544+0000","last_peered":"2024-09-21T13:55:13.014544+0000","last_clean":"2024-09-21T13:55:13.014544+0000","last_became_active":"2024-09-21T13:49:02.066693+0000","last_became_peered":"2024-09-21T13:49:02.066693+0000","last_unstale":"2024-09-21T13:55:13.014544+0000","last_undegraded":"2024-09-21T13:55:13.014544+0000","last_fullsized":"2024-09-21T13:55:13.014544+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":253,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.021556+0000","last_change":"2024-09-21T13:54:25.793052+0000","last_active":"2024-09-21T13:55:13.021556+0000","last_peered":"2024-09-21T13:55:13.021556+0000","last_clean":"2024-09-21T13:55:13.021556+0000","last_became_active":"2024-09-21T13:50:29.689381+0000","last_became_peered":"2024-09-21T13:50:29.689381+0000","last_unstale":"2024-09-21T13:55:13.021556+0000","last_undegraded":"2024-09-21T13:55:13.021556+0000","last_fullsized":"2024-09-21T13:55:13.021556+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:25.792985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:54:25.792985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:31.251922+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":306,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.014714+0000","last_change":"2024-09-21T13:50:39.480995+0000","last_active":"2024-09-21T13:55:13.014714+0000","last_peered":"2024-09-21T13:55:13.014714+0000","last_clean":"2024-09-21T13:55:13.014714+0000","last_became_active":"2024-09-21T13:48:58.050906+0000","last_became_peered":"2024-09-21T13:48:58.050906+0000","last_unstale":"2024-09-21T13:55:13.014714+0000","last_undegraded":"2024-09-21T13:55:13.014714+0000","last_fullsized":"2024-09-21T13:55:13.014714+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:39.480930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:39.480930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:34.949015+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"68'5","reported_seq":335,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.012200+0000","last_change":"2024-09-21T13:54:04.271336+0000","last_active":"2024-09-21T13:55:13.012200+0000","last_peered":"2024-09-21T13:55:13.012200+0000","last_clean":"2024-09-21T13:55:13.012200+0000","last_became_active":"2024-09-21T13:52:04.318515+0000","last_became_peered":"2024-09-21T13:52:04.318515+0000","last_unstale":"2024-09-21T13:55:13.012200+0000","last_undegraded":"2024-09-21T13:55:13.012200+0000","last_fullsized":"2024-09-21T13:55:13.012200+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-21T13:54:04.271280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:04.271280+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:18.236498+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":305,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.033125+0000","last_change":"2024-09-21T13:54:26.857307+0000","last_active":"2024-09-21T13:55:13.033125+0000","last_peered":"2024-09-21T13:55:13.033125+0000","last_clean":"2024-09-21T13:55:13.033125+0000","last_became_active":"2024-09-21T13:49:02.060212+0000","last_became_peered":"2024-09-21T13:49:02.060212+0000","last_unstale":"2024-09-21T13:55:13.033125+0000","last_undegraded":"2024-09-21T13:55:13.033125+0000","last_fullsized":"2024-09-21T13:55:13.033125+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:26.857231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:26.857231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:42.878634+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":310,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.014369+0000","last_change":"2024-09-21T13:54:23.043040+0000","last_active":"2024-09-21T13:55:13.014369+0000","last_peered":"2024-09-21T13:55:13.014369+0000","last_clean":"2024-09-21T13:55:13.014369+0000","last_became_active":"2024-09-21T13:49:04.078026+0000","last_became_peered":"2024-09-21T13:49:04.078026+0000","last_unstale":"2024-09-21T13:55:13.014369+0000","last_undegraded":"2024-09-21T13:55:13.014369+0000","last_fullsized":"2024-09-21T13:55:13.014369+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:23.042969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:54:23.042969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:46.405005+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":294,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.012902+0000","last_change":"2024-09-21T13:50:38.027288+0000","last_active":"2024-09-21T13:55:13.012902+0000","last_peered":"2024-09-21T13:55:13.012902+0000","last_clean":"2024-09-21T13:55:13.012902+0000","last_became_active":"2024-09-21T13:49:04.431422+0000","last_became_peered":"2024-09-21T13:49:04.431422+0000","last_unstale":"2024-09-21T13:55:13.012902+0000","last_undegraded":"2024-09-21T13:55:13.012902+0000","last_fullsized":"2024-09-21T13:55:13.012902+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:38.027227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:38.027227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:49.475642+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":245,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.012531+0000","last_change":"2024-09-21T13:55:07.560698+0000","last_active":"2024-09-21T13:55:13.012531+0000","last_peered":"2024-09-21T13:55:13.012531+0000","last_clean":"2024-09-21T13:55:13.012531+0000","last_became_active":"2024-09-21T13:50:29.688282+0000","last_became_peered":"2024-09-21T13:50:29.688282+0000","last_unstale":"2024-09-21T13:55:13.012531+0000","last_undegraded":"2024-09-21T13:55:13.012531+0000","last_fullsized":"2024-09-21T13:55:13.012531+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:07.560623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:55:07.560623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:56:21.527830+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"68'15","reported_seq":265,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-21T13:55:04.968640+0000","last_change":"2024-09-21T13:53:32.929362+0000","last_active":"2024-09-21T13:55:04.968640+0000","last_peered":"2024-09-21T13:55:04.968640+0000","last_clean":"2024-09-21T13:55:04.968640+0000","last_became_active":"2024-09-21T13:52:04.324273+0000","last_became_peered":"2024-09-21T13:52:04.324273+0000","last_unstale":"2024-09-21T13:55:04.968640+0000","last_undegraded":"2024-09-21T13:55:04.968640+0000","last_fullsized":"2024-09-21T13:55:04.968640+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:53:32.929017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:32.929017+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:38.870894+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":305,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.024979+0000","last_change":"2024-09-21T13:52:26.372217+0000","last_active":"2024-09-21T13:55:13.024979+0000","last_peered":"2024-09-21T13:55:13.024979+0000","last_clean":"2024-09-21T13:55:13.024979+0000","last_became_active":"2024-09-21T13:49:02.066796+0000","last_became_peered":"2024-09-21T13:49:02.066796+0000","last_unstale":"2024-09-21T13:55:13.024979+0000","last_undegraded":"2024-09-21T13:55:13.024979+0000","last_fullsized":"2024-09-21T13:55:13.024979+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:26.372144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:52:26.372144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:29.891331+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":290,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-21T13:55:04.954179+0000","last_change":"2024-09-21T13:53:57.962327+0000","last_active":"2024-09-21T13:55:04.954179+0000","last_peered":"2024-09-21T13:55:04.954179+0000","last_clean":"2024-09-21T13:55:04.954179+0000","last_became_active":"2024-09-21T13:49:04.434049+0000","last_became_peered":"2024-09-21T13:49:04.434049+0000","last_unstale":"2024-09-21T13:55:04.954179+0000","last_undegraded":"2024-09-21T13:55:04.954179+0000","last_fullsized":"2024-09-21T13:55:04.954179+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:57.962266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:57.962266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:07.827936+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":316,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.012958+0000","last_change":"2024-09-21T13:52:52.142682+0000","last_active":"2024-09-21T13:55:13.012958+0000","last_peered":"2024-09-21T13:55:13.012958+0000","last_clean":"2024-09-21T13:55:13.012958+0000","last_became_active":"2024-09-21T13:52:04.316016+0000","last_became_peered":"2024-09-21T13:52:04.316016+0000","last_unstale":"2024-09-21T13:55:13.012958+0000","last_undegraded":"2024-09-21T13:55:13.012958+0000","last_fullsized":"2024-09-21T13:55:13.012958+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:52.142638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:52:52.142638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:36.373450+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"68'12","reported_seq":329,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-21T13:55:04.946909+0000","last_change":"2024-09-21T13:55:02.031726+0000","last_active":"2024-09-21T13:55:04.946909+0000","last_peered":"2024-09-21T13:55:04.946909+0000","last_clean":"2024-09-21T13:55:04.946909+0000","last_became_active":"2024-09-21T13:49:00.061867+0000","last_became_peered":"2024-09-21T13:49:00.061867+0000","last_unstale":"2024-09-21T13:55:04.946909+0000","last_undegraded":"2024-09-21T13:55:04.946909+0000","last_fullsized":"2024-09-21T13:55:04.946909+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'12","last_scrub_stamp":"2024-09-21T13:55:02.031642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:55:02.031642+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:56:28.143685+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":300,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.012314+0000","last_change":"2024-09-21T13:52:04.321297+0000","last_active":"2024-09-21T13:55:13.012314+0000","last_peered":"2024-09-21T13:55:13.012314+0000","last_clean":"2024-09-21T13:55:13.012314+0000","last_became_active":"2024-09-21T13:52:04.320800+0000","last_became_peered":"2024-09-21T13:52:04.320800+0000","last_unstale":"2024-09-21T13:55:13.012314+0000","last_undegraded":"2024-09-21T13:55:13.012314+0000","last_fullsized":"2024-09-21T13:55:13.012314+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:15.423635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":313,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.033091+0000","last_change":"2024-09-21T13:54:59.736018+0000","last_active":"2024-09-21T13:55:13.033091+0000","last_peered":"2024-09-21T13:55:13.033091+0000","last_clean":"2024-09-21T13:55:13.033091+0000","last_became_active":"2024-09-21T13:52:04.321358+0000","last_became_peered":"2024-09-21T13:52:04.321358+0000","last_unstale":"2024-09-21T13:55:13.033091+0000","last_undegraded":"2024-09-21T13:55:13.033091+0000","last_fullsized":"2024-09-21T13:55:13.033091+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T13:54:59.735887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:54:59.735887+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:56:04.997270+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":314,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.024600+0000","last_change":"2024-09-21T13:52:43.516442+0000","last_active":"2024-09-21T13:55:13.024600+0000","last_peered":"2024-09-21T13:55:13.024600+0000","last_clean":"2024-09-21T13:55:13.024600+0000","last_became_active":"2024-09-21T13:48:58.048972+0000","last_became_peered":"2024-09-21T13:48:58.048972+0000","last_unstale":"2024-09-21T13:55:13.024600+0000","last_undegraded":"2024-09-21T13:55:13.024600+0000","last_fullsized":"2024-09-21T13:55:13.024600+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:52:43.516221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:52:43.516221+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:34.940975+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"68'10","reported_seq":281,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.033598+0000","last_change":"2024-09-21T13:54:57.749374+0000","last_active":"2024-09-21T13:55:13.033598+0000","last_peered":"2024-09-21T13:55:13.033598+0000","last_clean":"2024-09-21T13:55:13.033598+0000","last_became_active":"2024-09-21T13:50:29.694465+0000","last_became_peered":"2024-09-21T13:50:29.694465+0000","last_unstale":"2024-09-21T13:55:13.033598+0000","last_undegraded":"2024-09-21T13:55:13.033598+0000","last_fullsized":"2024-09-21T13:55:13.033598+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:54:57.749288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:57.749288+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:56:04.361356+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"68'8","reported_seq":393,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.026094+0000","last_change":"2024-09-21T13:51:42.174696+0000","last_active":"2024-09-21T13:55:13.026094+0000","last_peered":"2024-09-21T13:55:13.026094+0000","last_clean":"2024-09-21T13:55:13.026094+0000","last_became_active":"2024-09-21T13:49:02.059638+0000","last_became_peered":"2024-09-21T13:49:02.059638+0000","last_unstale":"2024-09-21T13:55:13.026094+0000","last_undegraded":"2024-09-21T13:55:13.026094+0000","last_fullsized":"2024-09-21T13:55:13.026094+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:51:42.174629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:42.174629+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:14.205512+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"203'5709","reported_seq":5759,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.620892+0000","last_change":"2024-09-21T13:54:12.377270+0000","last_active":"2024-09-21T13:55:13.620892+0000","last_peered":"2024-09-21T13:55:13.620892+0000","last_clean":"2024-09-21T13:55:13.620892+0000","last_became_active":"2024-09-21T13:54:12.376466+0000","last_became_peered":"2024-09-21T13:54:12.376466+0000","last_unstale":"2024-09-21T13:55:13.620892+0000","last_undegraded":"2024-09-21T13:55:13.620892+0000","last_fullsized":"2024-09-21T13:55:13.620892+0000","mapping_epoch":183,"log_start":"189'700","ondisk_log_start":"189'700","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":5009,"log_dups_size":0,"ondisk_log_size":5009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:30.045611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":374145024,"num_objects":5709,"num_object_clones":0,"num_object_copies":17127,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5709,"num_write_kb":365376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"68'15","reported_seq":342,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.039149+0000","last_change":"2024-09-21T13:53:58.658230+0000","last_active":"2024-09-21T13:55:13.039149+0000","last_peered":"2024-09-21T13:55:13.039149+0000","last_clean":"2024-09-21T13:55:13.039149+0000","last_became_active":"2024-09-21T13:49:00.060324+0000","last_became_peered":"2024-09-21T13:49:00.060324+0000","last_unstale":"2024-09-21T13:55:13.039149+0000","last_undegraded":"2024-09-21T13:55:13.039149+0000","last_fullsized":"2024-09-21T13:55:13.039149+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:53:58.658150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:58.658150+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:14.840395+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":302,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-21T13:55:04.950315+0000","last_change":"2024-09-21T13:50:06.151943+0000","last_active":"2024-09-21T13:55:04.950315+0000","last_peered":"2024-09-21T13:55:04.950315+0000","last_clean":"2024-09-21T13:55:04.950315+0000","last_became_active":"2024-09-21T13:48:58.051911+0000","last_became_peered":"2024-09-21T13:48:58.051911+0000","last_unstale":"2024-09-21T13:55:04.950315+0000","last_undegraded":"2024-09-21T13:55:04.950315+0000","last_fullsized":"2024-09-21T13:55:04.950315+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:06.151888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.151888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:12.984236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"68'8","reported_seq":384,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.013015+0000","last_change":"2024-09-21T13:53:19.021881+0000","last_active":"2024-09-21T13:55:13.013015+0000","last_peered":"2024-09-21T13:55:13.013015+0000","last_clean":"2024-09-21T13:55:13.013015+0000","last_became_active":"2024-09-21T13:49:02.061679+0000","last_became_peered":"2024-09-21T13:49:02.061679+0000","last_unstale":"2024-09-21T13:55:13.013015+0000","last_undegraded":"2024-09-21T13:55:13.013015+0000","last_fullsized":"2024-09-21T13:55:13.013015+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:53:19.021810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:53:19.021810+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:48.451879+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":294,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.014687+0000","last_change":"2024-09-21T13:50:34.481503+0000","last_active":"2024-09-21T13:55:13.014687+0000","last_peered":"2024-09-21T13:55:13.014687+0000","last_clean":"2024-09-21T13:55:13.014687+0000","last_became_active":"2024-09-21T13:49:04.081905+0000","last_became_peered":"2024-09-21T13:49:04.081905+0000","last_unstale":"2024-09-21T13:55:13.014687+0000","last_undegraded":"2024-09-21T13:55:13.014687+0000","last_fullsized":"2024-09-21T13:55:13.014687+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:34.481450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:34.481450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:44.964465+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.c","version":"203'5739","reported_seq":5790,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.763219+0000","last_change":"2024-09-21T13:54:12.479015+0000","last_active":"2024-09-21T13:55:13.763219+0000","last_peered":"2024-09-21T13:55:13.763219+0000","last_clean":"2024-09-21T13:55:13.763219+0000","last_became_active":"2024-09-21T13:54:12.478687+0000","last_became_peered":"2024-09-21T13:54:12.478687+0000","last_unstale":"2024-09-21T13:55:13.763219+0000","last_undegraded":"2024-09-21T13:55:13.763219+0000","last_fullsized":"2024-09-21T13:55:13.763219+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":5739,"log_dups_size":0,"ondisk_log_size":5739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:33.300965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376111104,"num_objects":5739,"num_object_clones":0,"num_object_copies":17217,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5739,"num_write_kb":367296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"68'18","reported_seq":339,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-21T13:55:04.949979+0000","last_change":"2024-09-21T13:54:19.001120+0000","last_active":"2024-09-21T13:55:04.949979+0000","last_peered":"2024-09-21T13:55:04.949979+0000","last_clean":"2024-09-21T13:55:04.949979+0000","last_became_active":"2024-09-21T13:49:00.429314+0000","last_became_peered":"2024-09-21T13:49:00.429314+0000","last_unstale":"2024-09-21T13:55:04.949979+0000","last_undegraded":"2024-09-21T13:55:04.949979+0000","last_fullsized":"2024-09-21T13:55:04.949979+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'18","last_scrub_stamp":"2024-09-21T13:54:19.001030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:19.001030+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:34.825871+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":294,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-21T13:55:04.955212+0000","last_change":"2024-09-21T13:48:58.056669+0000","last_active":"2024-09-21T13:55:04.955212+0000","last_peered":"2024-09-21T13:55:04.955212+0000","last_clean":"2024-09-21T13:55:04.955212+0000","last_became_active":"2024-09-21T13:48:58.056375+0000","last_became_peered":"2024-09-21T13:48:58.056375+0000","last_unstale":"2024-09-21T13:55:04.955212+0000","last_undegraded":"2024-09-21T13:55:04.955212+0000","last_fullsized":"2024-09-21T13:55:04.955212+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:36.988185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":306,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.012143+0000","last_change":"2024-09-21T13:54:07.310089+0000","last_active":"2024-09-21T13:55:13.012143+0000","last_peered":"2024-09-21T13:55:13.012143+0000","last_clean":"2024-09-21T13:55:13.012143+0000","last_became_active":"2024-09-21T13:49:02.062955+0000","last_became_peered":"2024-09-21T13:49:02.062955+0000","last_unstale":"2024-09-21T13:55:13.012143+0000","last_undegraded":"2024-09-21T13:55:13.012143+0000","last_fullsized":"2024-09-21T13:55:13.012143+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:07.310044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:07.310044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:23.878204+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":296,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.012823+0000","last_change":"2024-09-21T13:52:04.316586+0000","last_active":"2024-09-21T13:55:13.012823+0000","last_peered":"2024-09-21T13:55:13.012823+0000","last_clean":"2024-09-21T13:55:13.012823+0000","last_became_active":"2024-09-21T13:52:04.316121+0000","last_became_peered":"2024-09-21T13:52:04.316121+0000","last_unstale":"2024-09-21T13:55:13.012823+0000","last_undegraded":"2024-09-21T13:55:13.012823+0000","last_fullsized":"2024-09-21T13:55:13.012823+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"203'5700","reported_seq":5750,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.433426+0000","last_change":"2024-09-21T13:54:12.479429+0000","last_active":"2024-09-21T13:55:13.433426+0000","last_peered":"2024-09-21T13:55:13.433426+0000","last_clean":"2024-09-21T13:55:13.433426+0000","last_became_active":"2024-09-21T13:54:12.479146+0000","last_became_peered":"2024-09-21T13:54:12.479146+0000","last_unstale":"2024-09-21T13:55:13.433426+0000","last_undegraded":"2024-09-21T13:55:13.433426+0000","last_fullsized":"2024-09-21T13:55:13.433426+0000","mapping_epoch":183,"log_start":"187'300","ondisk_log_start":"187'300","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":5400,"log_dups_size":0,"ondisk_log_size":5400,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:37.886798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":373555200,"num_objects":5700,"num_object_clones":0,"num_object_copies":17100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5700,"num_write_kb":364800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"68'14","reported_seq":323,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.012299+0000","last_change":"2024-09-21T13:50:29.009786+0000","last_active":"2024-09-21T13:55:13.012299+0000","last_peered":"2024-09-21T13:55:13.012299+0000","last_clean":"2024-09-21T13:55:13.012299+0000","last_became_active":"2024-09-21T13:49:00.055031+0000","last_became_peered":"2024-09-21T13:49:00.055031+0000","last_unstale":"2024-09-21T13:55:13.012299+0000","last_undegraded":"2024-09-21T13:55:13.012299+0000","last_fullsized":"2024-09-21T13:55:13.012299+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'14","last_scrub_stamp":"2024-09-21T13:50:29.009720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:29.009720+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:44.453910+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":307,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.026628+0000","last_change":"2024-09-21T13:49:59.253636+0000","last_active":"2024-09-21T13:55:13.026628+0000","last_peered":"2024-09-21T13:55:13.026628+0000","last_clean":"2024-09-21T13:55:13.026628+0000","last_became_active":"2024-09-21T13:48:58.053533+0000","last_became_peered":"2024-09-21T13:48:58.053533+0000","last_unstale":"2024-09-21T13:55:13.026628+0000","last_undegraded":"2024-09-21T13:55:13.026628+0000","last_fullsized":"2024-09-21T13:55:13.026628+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:49:59.253587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:49:59.253587+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:36.902694+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"68'8","reported_seq":386,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-21T13:55:08.187402+0000","last_change":"2024-09-21T13:54:25.020479+0000","last_active":"2024-09-21T13:55:08.187402+0000","last_peered":"2024-09-21T13:55:08.187402+0000","last_clean":"2024-09-21T13:55:08.187402+0000","last_became_active":"2024-09-21T13:49:02.070064+0000","last_became_peered":"2024-09-21T13:49:02.070064+0000","last_unstale":"2024-09-21T13:55:08.187402+0000","last_undegraded":"2024-09-21T13:55:08.187402+0000","last_fullsized":"2024-09-21T13:55:08.187402+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:54:25.020230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:25.020230+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:50.633173+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":304,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.032670+0000","last_change":"2024-09-21T13:52:55.541288+0000","last_active":"2024-09-21T13:55:13.032670+0000","last_peered":"2024-09-21T13:55:13.032670+0000","last_clean":"2024-09-21T13:55:13.032670+0000","last_became_active":"2024-09-21T13:52:04.322185+0000","last_became_peered":"2024-09-21T13:52:04.322185+0000","last_unstale":"2024-09-21T13:55:13.032670+0000","last_undegraded":"2024-09-21T13:55:13.032670+0000","last_fullsized":"2024-09-21T13:55:13.032670+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:55.541232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:55.541232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:49.814778+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.e","version":"203'5741","reported_seq":5791,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.774826+0000","last_change":"2024-09-21T13:54:12.480264+0000","last_active":"2024-09-21T13:55:13.774826+0000","last_peered":"2024-09-21T13:55:13.774826+0000","last_clean":"2024-09-21T13:55:13.774826+0000","last_became_active":"2024-09-21T13:54:12.479901+0000","last_became_peered":"2024-09-21T13:54:12.479901+0000","last_unstale":"2024-09-21T13:55:13.774826+0000","last_undegraded":"2024-09-21T13:55:13.774826+0000","last_fullsized":"2024-09-21T13:55:13.774826+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":5741,"log_dups_size":0,"ondisk_log_size":5741,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:35.816508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376242176,"num_objects":5741,"num_object_clones":0,"num_object_copies":17223,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5741,"num_write_kb":367424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"68'10","reported_seq":332,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.022862+0000","last_change":"2024-09-21T13:54:19.593696+0000","last_active":"2024-09-21T13:55:13.022862+0000","last_peered":"2024-09-21T13:55:13.022862+0000","last_clean":"2024-09-21T13:55:13.022862+0000","last_became_active":"2024-09-21T13:49:00.054564+0000","last_became_peered":"2024-09-21T13:49:00.054564+0000","last_unstale":"2024-09-21T13:55:13.022862+0000","last_undegraded":"2024-09-21T13:55:13.022862+0000","last_fullsized":"2024-09-21T13:55:13.022862+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:54:19.593611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:19.593611+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:40.845382+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"61'1","reported_seq":347,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.015088+0000","last_change":"2024-09-21T13:54:59.927244+0000","last_active":"2024-09-21T13:55:13.015088+0000","last_peered":"2024-09-21T13:55:13.015088+0000","last_clean":"2024-09-21T13:55:13.015088+0000","last_became_active":"2024-09-21T13:52:04.315788+0000","last_became_peered":"2024-09-21T13:52:04.315788+0000","last_unstale":"2024-09-21T13:55:13.015088+0000","last_undegraded":"2024-09-21T13:55:13.015088+0000","last_fullsized":"2024-09-21T13:55:13.015088+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:54:59.927133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:59.927133+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:56:14.336344+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"96'9","reported_seq":338,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.025758+0000","last_change":"2024-09-21T13:54:06.280517+0000","last_active":"2024-09-21T13:55:13.025758+0000","last_peered":"2024-09-21T13:55:13.025758+0000","last_clean":"2024-09-21T13:55:13.025758+0000","last_became_active":"2024-09-21T13:50:29.693557+0000","last_became_peered":"2024-09-21T13:50:29.693557+0000","last_unstale":"2024-09-21T13:55:13.025758+0000","last_undegraded":"2024-09-21T13:55:13.025758+0000","last_fullsized":"2024-09-21T13:55:13.025758+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'9","last_scrub_stamp":"2024-09-21T13:54:06.280457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:06.280457+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:23.993139+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":246,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-21T13:55:04.949589+0000","last_change":"2024-09-21T13:53:54.948771+0000","last_active":"2024-09-21T13:55:04.949589+0000","last_peered":"2024-09-21T13:55:04.949589+0000","last_clean":"2024-09-21T13:55:04.949589+0000","last_became_active":"2024-09-21T13:52:04.319034+0000","last_became_peered":"2024-09-21T13:52:04.319034+0000","last_unstale":"2024-09-21T13:55:04.949589+0000","last_undegraded":"2024-09-21T13:55:04.949589+0000","last_fullsized":"2024-09-21T13:55:04.949589+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:54.948705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:54.948705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:05.874612+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.f","version":"202'5500","reported_seq":5547,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-21T13:55:09.046890+0000","last_change":"2024-09-21T13:54:12.478557+0000","last_active":"2024-09-21T13:55:09.046890+0000","last_peered":"2024-09-21T13:55:09.046890+0000","last_clean":"2024-09-21T13:55:09.046890+0000","last_became_active":"2024-09-21T13:54:12.477589+0000","last_became_peered":"2024-09-21T13:54:12.477589+0000","last_unstale":"2024-09-21T13:55:09.046890+0000","last_undegraded":"2024-09-21T13:55:09.046890+0000","last_fullsized":"2024-09-21T13:55:09.046890+0000","mapping_epoch":183,"log_start":"188'600","ondisk_log_start":"188'600","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":4900,"log_dups_size":0,"ondisk_log_size":4900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:29.462737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":360448000,"num_objects":5500,"num_object_clones":0,"num_object_copies":16500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5500,"num_write_kb":352000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"68'19","reported_seq":295,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.014620+0000","last_change":"2024-09-21T13:50:32.514267+0000","last_active":"2024-09-21T13:55:13.014620+0000","last_peered":"2024-09-21T13:55:13.014620+0000","last_clean":"2024-09-21T13:55:13.014620+0000","last_became_active":"2024-09-21T13:50:29.690185+0000","last_became_peered":"2024-09-21T13:50:29.690185+0000","last_unstale":"2024-09-21T13:55:13.014620+0000","last_undegraded":"2024-09-21T13:55:13.014620+0000","last_fullsized":"2024-09-21T13:55:13.014620+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-21T13:50:32.514210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:32.514210+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:51.911483+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":309,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.024231+0000","last_change":"2024-09-21T13:52:04.322289+0000","last_active":"2024-09-21T13:55:13.024231+0000","last_peered":"2024-09-21T13:55:13.024231+0000","last_clean":"2024-09-21T13:55:13.024231+0000","last_became_active":"2024-09-21T13:52:04.321656+0000","last_became_peered":"2024-09-21T13:52:04.321656+0000","last_unstale":"2024-09-21T13:55:13.024231+0000","last_undegraded":"2024-09-21T13:55:13.024231+0000","last_fullsized":"2024-09-21T13:55:13.024231+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":306,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.025425+0000","last_change":"2024-09-21T13:53:32.295930+0000","last_active":"2024-09-21T13:55:13.025425+0000","last_peered":"2024-09-21T13:55:13.025425+0000","last_clean":"2024-09-21T13:55:13.025425+0000","last_became_active":"2024-09-21T13:49:02.061187+0000","last_became_peered":"2024-09-21T13:49:02.061187+0000","last_unstale":"2024-09-21T13:55:13.025425+0000","last_undegraded":"2024-09-21T13:55:13.025425+0000","last_fullsized":"2024-09-21T13:55:13.025425+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:32.295869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:53:32.295869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:33.093729+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":296,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.024366+0000","last_change":"2024-09-21T13:52:04.321133+0000","last_active":"2024-09-21T13:55:13.024366+0000","last_peered":"2024-09-21T13:55:13.024366+0000","last_clean":"2024-09-21T13:55:13.024366+0000","last_became_active":"2024-09-21T13:52:04.320826+0000","last_became_peered":"2024-09-21T13:52:04.320826+0000","last_unstale":"2024-09-21T13:55:13.024366+0000","last_undegraded":"2024-09-21T13:55:13.024366+0000","last_fullsized":"2024-09-21T13:55:13.024366+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:14.698955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"203'5725","reported_seq":5776,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.411583+0000","last_change":"2024-09-21T13:54:12.477847+0000","last_active":"2024-09-21T13:55:13.411583+0000","last_peered":"2024-09-21T13:55:13.411583+0000","last_clean":"2024-09-21T13:55:13.411583+0000","last_became_active":"2024-09-21T13:54:12.477306+0000","last_became_peered":"2024-09-21T13:54:12.477306+0000","last_unstale":"2024-09-21T13:55:13.411583+0000","last_undegraded":"2024-09-21T13:55:13.411583+0000","last_fullsized":"2024-09-21T13:55:13.411583+0000","mapping_epoch":183,"log_start":"187'300","ondisk_log_start":"187'300","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":5425,"log_dups_size":0,"ondisk_log_size":5425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:26.986694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":375193600,"num_objects":5725,"num_object_clones":0,"num_object_copies":17175,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5725,"num_write_kb":366400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"132'30","reported_seq":438,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.024086+0000","last_change":"2024-09-21T13:54:16.681017+0000","last_active":"2024-09-21T13:55:13.024086+0000","last_peered":"2024-09-21T13:55:13.024086+0000","last_clean":"2024-09-21T13:55:13.024086+0000","last_became_active":"2024-09-21T13:52:04.319431+0000","last_became_peered":"2024-09-21T13:52:04.319431+0000","last_unstale":"2024-09-21T13:55:13.024086+0000","last_undegraded":"2024-09-21T13:55:13.024086+0000","last_fullsized":"2024-09-21T13:55:13.024086+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'30","last_scrub_stamp":"2024-09-21T13:54:16.680953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:16.680953+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:39.761349+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":305,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.012623+0000","last_change":"2024-09-21T13:55:11.464897+0000","last_active":"2024-09-21T13:55:13.012623+0000","last_peered":"2024-09-21T13:55:13.012623+0000","last_clean":"2024-09-21T13:55:13.012623+0000","last_became_active":"2024-09-21T13:48:58.055366+0000","last_became_peered":"2024-09-21T13:48:58.055366+0000","last_unstale":"2024-09-21T13:55:13.012623+0000","last_undegraded":"2024-09-21T13:55:13.012623+0000","last_fullsized":"2024-09-21T13:55:13.012623+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:11.464842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:55:11.464842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:56:23.354512+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":333,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.015212+0000","last_change":"2024-09-21T13:54:25.029827+0000","last_active":"2024-09-21T13:55:13.015212+0000","last_peered":"2024-09-21T13:55:13.015212+0000","last_clean":"2024-09-21T13:55:13.015212+0000","last_became_active":"2024-09-21T13:48:57.046246+0000","last_became_peered":"2024-09-21T13:48:57.046246+0000","last_unstale":"2024-09-21T13:55:13.015212+0000","last_undegraded":"2024-09-21T13:55:13.015212+0000","last_fullsized":"2024-09-21T13:55:13.015212+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-21T13:54:25.029757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:54:25.029757+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:27.260800+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0015946669999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":254,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.024812+0000","last_change":"2024-09-21T13:55:07.021128+0000","last_active":"2024-09-21T13:55:13.024812+0000","last_peered":"2024-09-21T13:55:13.024812+0000","last_clean":"2024-09-21T13:55:13.024812+0000","last_became_active":"2024-09-21T13:50:29.688249+0000","last_became_peered":"2024-09-21T13:50:29.688249+0000","last_unstale":"2024-09-21T13:55:13.024812+0000","last_undegraded":"2024-09-21T13:55:13.024812+0000","last_fullsized":"2024-09-21T13:55:13.024812+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:07.021059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:55:07.021059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:56:23.728546+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":298,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-21T13:55:04.947374+0000","last_change":"2024-09-21T13:54:54.036763+0000","last_active":"2024-09-21T13:55:04.947374+0000","last_peered":"2024-09-21T13:55:04.947374+0000","last_clean":"2024-09-21T13:55:04.947374+0000","last_became_active":"2024-09-21T13:49:04.080466+0000","last_became_peered":"2024-09-21T13:49:04.080466+0000","last_unstale":"2024-09-21T13:55:04.947374+0000","last_undegraded":"2024-09-21T13:55:04.947374+0000","last_fullsized":"2024-09-21T13:55:04.947374+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:54.036699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:54:54.036699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:59.066704+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"202'5308","reported_seq":5356,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-21T13:55:09.039749+0000","last_change":"2024-09-21T13:54:12.373760+0000","last_active":"2024-09-21T13:55:09.039749+0000","last_peered":"2024-09-21T13:55:09.039749+0000","last_clean":"2024-09-21T13:55:09.039749+0000","last_became_active":"2024-09-21T13:54:12.373395+0000","last_became_peered":"2024-09-21T13:54:12.373395+0000","last_unstale":"2024-09-21T13:55:09.039749+0000","last_undegraded":"2024-09-21T13:55:09.039749+0000","last_fullsized":"2024-09-21T13:55:09.039749+0000","mapping_epoch":183,"log_start":"188'400","ondisk_log_start":"188'400","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":4908,"log_dups_size":0,"ondisk_log_size":4908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:36.149952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":347865088,"num_objects":5308,"num_object_clones":0,"num_object_copies":15924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5308,"num_write_kb":339712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"68'13","reported_seq":328,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.038826+0000","last_change":"2024-09-21T13:50:06.973399+0000","last_active":"2024-09-21T13:55:13.038826+0000","last_peered":"2024-09-21T13:55:13.038826+0000","last_clean":"2024-09-21T13:55:13.038826+0000","last_became_active":"2024-09-21T13:49:00.053323+0000","last_became_peered":"2024-09-21T13:49:00.053323+0000","last_unstale":"2024-09-21T13:55:13.038826+0000","last_undegraded":"2024-09-21T13:55:13.038826+0000","last_fullsized":"2024-09-21T13:55:13.038826+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'13","last_scrub_stamp":"2024-09-21T13:50:06.973336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:06.973336+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:41.970680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":333,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.022139+0000","last_change":"2024-09-21T13:53:24.885266+0000","last_active":"2024-09-21T13:55:13.022139+0000","last_peered":"2024-09-21T13:55:13.022139+0000","last_clean":"2024-09-21T13:55:13.022139+0000","last_became_active":"2024-09-21T13:52:04.320581+0000","last_became_peered":"2024-09-21T13:52:04.320581+0000","last_unstale":"2024-09-21T13:55:13.022139+0000","last_undegraded":"2024-09-21T13:55:13.022139+0000","last_fullsized":"2024-09-21T13:55:13.022139+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:24.884932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:53:24.884932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:40.993958+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":231,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-21T13:55:04.969434+0000","last_change":"2024-09-21T13:53:19.985044+0000","last_active":"2024-09-21T13:55:04.969434+0000","last_peered":"2024-09-21T13:55:04.969434+0000","last_clean":"2024-09-21T13:55:04.969434+0000","last_became_active":"2024-09-21T13:52:04.316171+0000","last_became_peered":"2024-09-21T13:52:04.316171+0000","last_unstale":"2024-09-21T13:55:04.969434+0000","last_undegraded":"2024-09-21T13:55:04.969434+0000","last_fullsized":"2024-09-21T13:55:04.969434+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:53:19.984966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:53:19.984966+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:49.880275+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":183,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-21T13:55:04.970138+0000","last_change":"2024-09-21T13:54:26.009416+0000","last_active":"2024-09-21T13:55:04.970138+0000","last_peered":"2024-09-21T13:55:04.970138+0000","last_clean":"2024-09-21T13:55:04.970138+0000","last_became_active":"2024-09-21T13:52:04.320701+0000","last_became_peered":"2024-09-21T13:52:04.320701+0000","last_unstale":"2024-09-21T13:55:04.970138+0000","last_undegraded":"2024-09-21T13:55:04.970138+0000","last_fullsized":"2024-09-21T13:55:04.970138+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:26.009352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:26.009352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:55.894955+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":301,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.034009+0000","last_change":"2024-09-21T13:51:12.212558+0000","last_active":"2024-09-21T13:55:13.034009+0000","last_peered":"2024-09-21T13:55:13.034009+0000","last_clean":"2024-09-21T13:55:13.034009+0000","last_became_active":"2024-09-21T13:49:04.431035+0000","last_became_peered":"2024-09-21T13:49:04.431035+0000","last_unstale":"2024-09-21T13:55:13.034009+0000","last_undegraded":"2024-09-21T13:55:13.034009+0000","last_fullsized":"2024-09-21T13:55:13.034009+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:12.212391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:51:12.212391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:31.820181+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"203'5718","reported_seq":5768,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.616691+0000","last_change":"2024-09-21T13:54:12.477912+0000","last_active":"2024-09-21T13:55:13.616691+0000","last_peered":"2024-09-21T13:55:13.616691+0000","last_clean":"2024-09-21T13:55:13.616691+0000","last_became_active":"2024-09-21T13:54:12.477596+0000","last_became_peered":"2024-09-21T13:54:12.477596+0000","last_unstale":"2024-09-21T13:55:13.616691+0000","last_undegraded":"2024-09-21T13:55:13.616691+0000","last_fullsized":"2024-09-21T13:55:13.616691+0000","mapping_epoch":183,"log_start":"189'700","ondisk_log_start":"189'700","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":5018,"log_dups_size":0,"ondisk_log_size":5018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:33.789453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":374734848,"num_objects":5718,"num_object_clones":0,"num_object_copies":17154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5718,"num_write_kb":365952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"68'12","reported_seq":360,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.026469+0000","last_change":"2024-09-21T13:53:17.740156+0000","last_active":"2024-09-21T13:55:13.026469+0000","last_peered":"2024-09-21T13:55:13.026469+0000","last_clean":"2024-09-21T13:55:13.026469+0000","last_became_active":"2024-09-21T13:52:04.318875+0000","last_became_peered":"2024-09-21T13:52:04.318875+0000","last_unstale":"2024-09-21T13:55:13.026469+0000","last_undegraded":"2024-09-21T13:55:13.026469+0000","last_fullsized":"2024-09-21T13:55:13.026469+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'12","last_scrub_stamp":"2024-09-21T13:53:17.740080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:17.740080+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:33.880649+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":238,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.012410+0000","last_change":"2024-09-21T13:50:30.020722+0000","last_active":"2024-09-21T13:55:13.012410+0000","last_peered":"2024-09-21T13:55:13.012410+0000","last_clean":"2024-09-21T13:55:13.012410+0000","last_became_active":"2024-09-21T13:50:29.688352+0000","last_became_peered":"2024-09-21T13:50:29.688352+0000","last_unstale":"2024-09-21T13:55:13.012410+0000","last_undegraded":"2024-09-21T13:55:13.012410+0000","last_fullsized":"2024-09-21T13:55:13.012410+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:30.020656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:30.020656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:42.379271+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":687,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.033764+0000","last_change":"2024-09-21T13:51:21.271997+0000","last_active":"2024-09-21T13:55:13.033764+0000","last_peered":"2024-09-21T13:55:13.033764+0000","last_clean":"2024-09-21T13:55:13.033764+0000","last_became_active":"2024-09-21T13:48:57.046084+0000","last_became_peered":"2024-09-21T13:48:57.046084+0000","last_unstale":"2024-09-21T13:55:13.033764+0000","last_undegraded":"2024-09-21T13:55:13.033764+0000","last_fullsized":"2024-09-21T13:55:13.033764+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-21T13:51:21.271917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:51:21.271917+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:48.857715+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00047048000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":370,"num_read_kb":365,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":298,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.015122+0000","last_change":"2024-09-21T13:51:53.518645+0000","last_active":"2024-09-21T13:55:13.015122+0000","last_peered":"2024-09-21T13:55:13.015122+0000","last_clean":"2024-09-21T13:55:13.015122+0000","last_became_active":"2024-09-21T13:49:02.066641+0000","last_became_peered":"2024-09-21T13:49:02.066641+0000","last_unstale":"2024-09-21T13:55:13.015122+0000","last_undegraded":"2024-09-21T13:55:13.015122+0000","last_fullsized":"2024-09-21T13:55:13.015122+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:53.518567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:53.518567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:16.069137+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":301,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.022535+0000","last_change":"2024-09-21T13:52:21.332066+0000","last_active":"2024-09-21T13:55:13.022535+0000","last_peered":"2024-09-21T13:55:13.022535+0000","last_clean":"2024-09-21T13:55:13.022535+0000","last_became_active":"2024-09-21T13:49:04.078372+0000","last_became_peered":"2024-09-21T13:49:04.078372+0000","last_unstale":"2024-09-21T13:55:13.022535+0000","last_undegraded":"2024-09-21T13:55:13.022535+0000","last_fullsized":"2024-09-21T13:55:13.022535+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:21.331792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:21.331792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:32.973257+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"203'5752","reported_seq":5803,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.450557+0000","last_change":"2024-09-21T13:54:12.378930+0000","last_active":"2024-09-21T13:55:13.450557+0000","last_peered":"2024-09-21T13:55:13.450557+0000","last_clean":"2024-09-21T13:55:13.450557+0000","last_became_active":"2024-09-21T13:54:12.378536+0000","last_became_peered":"2024-09-21T13:54:12.378536+0000","last_unstale":"2024-09-21T13:55:13.450557+0000","last_undegraded":"2024-09-21T13:55:13.450557+0000","last_fullsized":"2024-09-21T13:55:13.450557+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":5752,"log_dups_size":0,"ondisk_log_size":5752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:32.246204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376963072,"num_objects":5752,"num_object_clones":0,"num_object_copies":17256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5752,"num_write_kb":368128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"68'16","reported_seq":408,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.025999+0000","last_change":"2024-09-21T13:52:19.452239+0000","last_active":"2024-09-21T13:55:13.025999+0000","last_peered":"2024-09-21T13:55:13.025999+0000","last_clean":"2024-09-21T13:55:13.025999+0000","last_became_active":"2024-09-21T13:49:00.426867+0000","last_became_peered":"2024-09-21T13:49:00.426867+0000","last_unstale":"2024-09-21T13:55:13.025999+0000","last_undegraded":"2024-09-21T13:55:13.025999+0000","last_fullsized":"2024-09-21T13:55:13.025999+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'16","last_scrub_stamp":"2024-09-21T13:52:19.452163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:52:19.452163+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:50.008981+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":311,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-21T13:55:04.948763+0000","last_change":"2024-09-21T13:54:07.057258+0000","last_active":"2024-09-21T13:55:04.948763+0000","last_peered":"2024-09-21T13:55:04.948763+0000","last_clean":"2024-09-21T13:55:04.948763+0000","last_became_active":"2024-09-21T13:48:58.052659+0000","last_became_peered":"2024-09-21T13:48:58.052659+0000","last_unstale":"2024-09-21T13:55:04.948763+0000","last_undegraded":"2024-09-21T13:55:04.948763+0000","last_fullsized":"2024-09-21T13:55:04.948763+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:07.057210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:07.057210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:14.084334+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"75'79","reported_seq":365,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.033716+0000","last_change":"2024-09-21T13:54:56.703310+0000","last_active":"2024-09-21T13:55:13.033716+0000","last_peered":"2024-09-21T13:55:13.033716+0000","last_clean":"2024-09-21T13:55:13.033716+0000","last_became_active":"2024-09-21T13:48:48.454646+0000","last_became_peered":"2024-09-21T13:48:48.454646+0000","last_unstale":"2024-09-21T13:55:13.033716+0000","last_undegraded":"2024-09-21T13:55:13.033716+0000","last_fullsized":"2024-09-21T13:55:13.033716+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:54:56.703227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:54:56.703227+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:56:15.868800+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":308,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.033328+0000","last_change":"2024-09-21T13:54:52.829280+0000","last_active":"2024-09-21T13:55:13.033328+0000","last_peered":"2024-09-21T13:55:13.033328+0000","last_clean":"2024-09-21T13:55:13.033328+0000","last_became_active":"2024-09-21T13:52:04.320627+0000","last_became_peered":"2024-09-21T13:52:04.320627+0000","last_unstale":"2024-09-21T13:55:13.033328+0000","last_undegraded":"2024-09-21T13:55:13.033328+0000","last_fullsized":"2024-09-21T13:55:13.033328+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:52.829205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:52.829205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:56:00.052121+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":294,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.014962+0000","last_change":"2024-09-21T13:50:25.560928+0000","last_active":"2024-09-21T13:55:13.014962+0000","last_peered":"2024-09-21T13:55:13.014962+0000","last_clean":"2024-09-21T13:55:13.014962+0000","last_became_active":"2024-09-21T13:49:04.078091+0000","last_became_peered":"2024-09-21T13:49:04.078091+0000","last_unstale":"2024-09-21T13:55:13.014962+0000","last_undegraded":"2024-09-21T13:55:13.014962+0000","last_fullsized":"2024-09-21T13:55:13.014962+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:25.560863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:25.560863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:41.830570+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"203'5564","reported_seq":5614,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.444974+0000","last_change":"2024-09-21T13:54:12.378230+0000","last_active":"2024-09-21T13:55:13.444974+0000","last_peered":"2024-09-21T13:55:13.444974+0000","last_clean":"2024-09-21T13:55:13.444974+0000","last_became_active":"2024-09-21T13:54:12.377874+0000","last_became_peered":"2024-09-21T13:54:12.377874+0000","last_unstale":"2024-09-21T13:55:13.444974+0000","last_undegraded":"2024-09-21T13:55:13.444974+0000","last_fullsized":"2024-09-21T13:55:13.444974+0000","mapping_epoch":183,"log_start":"189'700","ondisk_log_start":"189'700","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":4864,"log_dups_size":0,"ondisk_log_size":4864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:39.278391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":364642304,"num_objects":5564,"num_object_clones":0,"num_object_copies":16692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5564,"num_write_kb":356096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"68'11","reported_seq":320,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.012986+0000","last_change":"2024-09-21T13:54:16.451648+0000","last_active":"2024-09-21T13:55:13.012986+0000","last_peered":"2024-09-21T13:55:13.012986+0000","last_clean":"2024-09-21T13:55:13.012986+0000","last_became_active":"2024-09-21T13:49:00.053268+0000","last_became_peered":"2024-09-21T13:49:00.053268+0000","last_unstale":"2024-09-21T13:55:13.012986+0000","last_undegraded":"2024-09-21T13:55:13.012986+0000","last_fullsized":"2024-09-21T13:55:13.012986+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:54:16.451588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:16.451588+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:41.356489+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":316,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.012885+0000","last_change":"2024-09-21T13:52:48.165210+0000","last_active":"2024-09-21T13:55:13.012885+0000","last_peered":"2024-09-21T13:55:13.012885+0000","last_clean":"2024-09-21T13:55:13.012885+0000","last_became_active":"2024-09-21T13:52:04.318345+0000","last_became_peered":"2024-09-21T13:52:04.318345+0000","last_unstale":"2024-09-21T13:55:13.012885+0000","last_undegraded":"2024-09-21T13:55:13.012885+0000","last_fullsized":"2024-09-21T13:55:13.012885+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:48.165145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:52:48.165145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:32.368867+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":306,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.014964+0000","last_change":"2024-09-21T13:51:46.503640+0000","last_active":"2024-09-21T13:55:13.014964+0000","last_peered":"2024-09-21T13:55:13.014964+0000","last_clean":"2024-09-21T13:55:13.014964+0000","last_became_active":"2024-09-21T13:49:02.061901+0000","last_became_peered":"2024-09-21T13:49:02.061901+0000","last_unstale":"2024-09-21T13:55:13.014964+0000","last_undegraded":"2024-09-21T13:55:13.014964+0000","last_fullsized":"2024-09-21T13:55:13.014964+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:46.503576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:46.503576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:15.052988+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":298,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-21T13:55:04.949605+0000","last_change":"2024-09-21T13:54:58.952280+0000","last_active":"2024-09-21T13:55:04.949605+0000","last_peered":"2024-09-21T13:55:04.949605+0000","last_clean":"2024-09-21T13:55:04.949605+0000","last_became_active":"2024-09-21T13:49:04.431268+0000","last_became_peered":"2024-09-21T13:49:04.431268+0000","last_unstale":"2024-09-21T13:55:04.949605+0000","last_undegraded":"2024-09-21T13:55:04.949605+0000","last_fullsized":"2024-09-21T13:55:04.949605+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:58.952218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:54:58.952218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:56:19.067947+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1","version":"203'5558","reported_seq":5608,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.612149+0000","last_change":"2024-09-21T13:54:12.478397+0000","last_active":"2024-09-21T13:55:13.612149+0000","last_peered":"2024-09-21T13:55:13.612149+0000","last_clean":"2024-09-21T13:55:13.612149+0000","last_became_active":"2024-09-21T13:54:12.477588+0000","last_became_peered":"2024-09-21T13:54:12.477588+0000","last_unstale":"2024-09-21T13:55:13.612149+0000","last_undegraded":"2024-09-21T13:55:13.612149+0000","last_fullsized":"2024-09-21T13:55:13.612149+0000","mapping_epoch":183,"log_start":"189'600","ondisk_log_start":"189'600","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":4958,"log_dups_size":0,"ondisk_log_size":4958,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:37.981662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":364249088,"num_objects":5558,"num_object_clones":0,"num_object_copies":16674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5558,"num_write_kb":355712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"68'17","reported_seq":340,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.012679+0000","last_change":"2024-09-21T13:52:45.176950+0000","last_active":"2024-09-21T13:55:13.012679+0000","last_peered":"2024-09-21T13:55:13.012679+0000","last_clean":"2024-09-21T13:55:13.012679+0000","last_became_active":"2024-09-21T13:52:04.313218+0000","last_became_peered":"2024-09-21T13:52:04.313218+0000","last_unstale":"2024-09-21T13:55:13.012679+0000","last_undegraded":"2024-09-21T13:55:13.012679+0000","last_fullsized":"2024-09-21T13:55:13.012679+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'17","last_scrub_stamp":"2024-09-21T13:52:45.176873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:52:45.176873+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:35.397286+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":330,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.025630+0000","last_change":"2024-09-21T13:55:04.216806+0000","last_active":"2024-09-21T13:55:13.025630+0000","last_peered":"2024-09-21T13:55:13.025630+0000","last_clean":"2024-09-21T13:55:13.025630+0000","last_became_active":"2024-09-21T13:48:58.049142+0000","last_became_peered":"2024-09-21T13:48:58.049142+0000","last_unstale":"2024-09-21T13:55:13.025630+0000","last_undegraded":"2024-09-21T13:55:13.025630+0000","last_fullsized":"2024-09-21T13:55:13.025630+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:04.216734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:55:04.216734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:56:26.712676+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":297,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.022637+0000","last_change":"2024-09-21T13:54:06.629867+0000","last_active":"2024-09-21T13:55:13.022637+0000","last_peered":"2024-09-21T13:55:13.022637+0000","last_clean":"2024-09-21T13:55:13.022637+0000","last_became_active":"2024-09-21T13:49:02.067472+0000","last_became_peered":"2024-09-21T13:49:02.067472+0000","last_unstale":"2024-09-21T13:55:13.022637+0000","last_undegraded":"2024-09-21T13:55:13.022637+0000","last_fullsized":"2024-09-21T13:55:13.022637+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:06.629822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:06.629822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:14.040598+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":175,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-21T13:55:04.970336+0000","last_change":"2024-09-21T13:53:18.028013+0000","last_active":"2024-09-21T13:55:04.970336+0000","last_peered":"2024-09-21T13:55:04.970336+0000","last_clean":"2024-09-21T13:55:04.970336+0000","last_became_active":"2024-09-21T13:52:04.315920+0000","last_became_peered":"2024-09-21T13:52:04.315920+0000","last_unstale":"2024-09-21T13:55:04.970336+0000","last_undegraded":"2024-09-21T13:55:04.970336+0000","last_fullsized":"2024-09-21T13:55:04.970336+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:18.027944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:18.027944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:30.919961+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.0","version":"202'5376","reported_seq":5423,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-21T13:55:09.044792+0000","last_change":"2024-09-21T13:54:12.375805+0000","last_active":"2024-09-21T13:55:09.044792+0000","last_peered":"2024-09-21T13:55:09.044792+0000","last_clean":"2024-09-21T13:55:09.044792+0000","last_became_active":"2024-09-21T13:54:12.375197+0000","last_became_peered":"2024-09-21T13:54:12.375197+0000","last_unstale":"2024-09-21T13:55:09.044792+0000","last_undegraded":"2024-09-21T13:55:09.044792+0000","last_fullsized":"2024-09-21T13:55:09.044792+0000","mapping_epoch":183,"log_start":"188'500","ondisk_log_start":"188'500","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":4876,"log_dups_size":0,"ondisk_log_size":4876,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:17.781582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352321536,"num_objects":5376,"num_object_clones":0,"num_object_copies":16128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5376,"num_write_kb":344064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"68'10","reported_seq":324,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.012355+0000","last_change":"2024-09-21T13:54:20.426028+0000","last_active":"2024-09-21T13:55:13.012355+0000","last_peered":"2024-09-21T13:55:13.012355+0000","last_clean":"2024-09-21T13:55:13.012355+0000","last_became_active":"2024-09-21T13:49:00.059850+0000","last_became_peered":"2024-09-21T13:49:00.059850+0000","last_unstale":"2024-09-21T13:55:13.012355+0000","last_undegraded":"2024-09-21T13:55:13.012355+0000","last_fullsized":"2024-09-21T13:55:13.012355+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:54:20.425889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:20.425889+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:32.475251+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":302,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-21T13:55:04.948890+0000","last_change":"2024-09-21T13:51:28.565842+0000","last_active":"2024-09-21T13:55:04.948890+0000","last_peered":"2024-09-21T13:55:04.948890+0000","last_clean":"2024-09-21T13:55:04.948890+0000","last_became_active":"2024-09-21T13:48:58.058309+0000","last_became_peered":"2024-09-21T13:48:58.058309+0000","last_unstale":"2024-09-21T13:55:04.948890+0000","last_undegraded":"2024-09-21T13:55:04.948890+0000","last_fullsized":"2024-09-21T13:55:04.948890+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:28.565766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:51:28.565766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:51.015992+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"132'10","reported_seq":231,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-21T13:55:04.977885+0000","last_change":"2024-09-21T13:54:19.934832+0000","last_active":"2024-09-21T13:55:04.977885+0000","last_peered":"2024-09-21T13:55:04.977885+0000","last_clean":"2024-09-21T13:55:04.977885+0000","last_became_active":"2024-09-21T13:52:04.320441+0000","last_became_peered":"2024-09-21T13:52:04.320441+0000","last_unstale":"2024-09-21T13:55:04.977885+0000","last_undegraded":"2024-09-21T13:55:04.977885+0000","last_fullsized":"2024-09-21T13:55:04.977885+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'10","last_scrub_stamp":"2024-09-21T13:54:19.934756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:19.934756+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:29.532492+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"68'1","reported_seq":308,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.012795+0000","last_change":"2024-09-21T13:52:04.314151+0000","last_active":"2024-09-21T13:55:13.012795+0000","last_peered":"2024-09-21T13:55:13.012795+0000","last_clean":"2024-09-21T13:55:13.012795+0000","last_became_active":"2024-09-21T13:52:04.313210+0000","last_became_peered":"2024-09-21T13:52:04.313210+0000","last_unstale":"2024-09-21T13:55:13.012795+0000","last_undegraded":"2024-09-21T13:55:13.012795+0000","last_fullsized":"2024-09-21T13:55:13.012795+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T13:50:41.000201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:41.000201+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:47.402459+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.7","version":"203'5606","reported_seq":5656,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.645649+0000","last_change":"2024-09-21T13:54:12.376676+0000","last_active":"2024-09-21T13:55:13.645649+0000","last_peered":"2024-09-21T13:55:13.645649+0000","last_clean":"2024-09-21T13:55:13.645649+0000","last_became_active":"2024-09-21T13:54:12.376302+0000","last_became_peered":"2024-09-21T13:54:12.376302+0000","last_unstale":"2024-09-21T13:55:13.645649+0000","last_undegraded":"2024-09-21T13:55:13.645649+0000","last_fullsized":"2024-09-21T13:55:13.645649+0000","mapping_epoch":183,"log_start":"189'600","ondisk_log_start":"189'600","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":5006,"log_dups_size":0,"ondisk_log_size":5006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:33.120123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":367394816,"num_objects":5606,"num_object_clones":0,"num_object_copies":16818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5606,"num_write_kb":358784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"68'9","reported_seq":326,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.020563+0000","last_change":"2024-09-21T13:53:18.740202+0000","last_active":"2024-09-21T13:55:13.020563+0000","last_peered":"2024-09-21T13:55:13.020563+0000","last_clean":"2024-09-21T13:55:13.020563+0000","last_became_active":"2024-09-21T13:49:00.053016+0000","last_became_peered":"2024-09-21T13:49:00.053016+0000","last_unstale":"2024-09-21T13:55:13.020563+0000","last_undegraded":"2024-09-21T13:55:13.020563+0000","last_fullsized":"2024-09-21T13:55:13.020563+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:53:18.740111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:18.740111+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:34.930684+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":314,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.014808+0000","last_change":"2024-09-21T13:54:53.894575+0000","last_active":"2024-09-21T13:55:13.014808+0000","last_peered":"2024-09-21T13:55:13.014808+0000","last_clean":"2024-09-21T13:55:13.014808+0000","last_became_active":"2024-09-21T13:48:58.049543+0000","last_became_peered":"2024-09-21T13:48:58.049543+0000","last_unstale":"2024-09-21T13:55:13.014808+0000","last_undegraded":"2024-09-21T13:55:13.014808+0000","last_fullsized":"2024-09-21T13:55:13.014808+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:53.894506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:53.894506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:56:14.921537+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":236,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.012626+0000","last_change":"2024-09-21T13:54:21.403013+0000","last_active":"2024-09-21T13:55:13.012626+0000","last_peered":"2024-09-21T13:55:13.012626+0000","last_clean":"2024-09-21T13:55:13.012626+0000","last_became_active":"2024-09-21T13:52:04.317429+0000","last_became_peered":"2024-09-21T13:52:04.317429+0000","last_unstale":"2024-09-21T13:55:13.012626+0000","last_undegraded":"2024-09-21T13:55:13.012626+0000","last_fullsized":"2024-09-21T13:55:13.012626+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:21.402946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:21.402946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:26.710441+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":257,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.032586+0000","last_change":"2024-09-21T13:54:55.720381+0000","last_active":"2024-09-21T13:55:13.032586+0000","last_peered":"2024-09-21T13:55:13.032586+0000","last_clean":"2024-09-21T13:55:13.032586+0000","last_became_active":"2024-09-21T13:52:04.321703+0000","last_became_peered":"2024-09-21T13:52:04.321703+0000","last_unstale":"2024-09-21T13:55:13.032586+0000","last_undegraded":"2024-09-21T13:55:13.032586+0000","last_fullsized":"2024-09-21T13:55:13.032586+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:55.720307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:54:55.720307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:56.692542+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.6","version":"203'5646","reported_seq":5697,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.430267+0000","last_change":"2024-09-21T13:54:12.478610+0000","last_active":"2024-09-21T13:55:13.430267+0000","last_peered":"2024-09-21T13:55:13.430267+0000","last_clean":"2024-09-21T13:55:13.430267+0000","last_became_active":"2024-09-21T13:54:12.478187+0000","last_became_peered":"2024-09-21T13:54:12.478187+0000","last_unstale":"2024-09-21T13:55:13.430267+0000","last_undegraded":"2024-09-21T13:55:13.430267+0000","last_fullsized":"2024-09-21T13:55:13.430267+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":5646,"log_dups_size":0,"ondisk_log_size":5646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:21.444672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370016256,"num_objects":5646,"num_object_clones":0,"num_object_copies":16938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5646,"num_write_kb":361344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"68'19","reported_seq":333,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.025481+0000","last_change":"2024-09-21T13:50:03.769686+0000","last_active":"2024-09-21T13:55:13.025481+0000","last_peered":"2024-09-21T13:55:13.025481+0000","last_clean":"2024-09-21T13:55:13.025481+0000","last_became_active":"2024-09-21T13:49:00.425955+0000","last_became_peered":"2024-09-21T13:49:00.425955+0000","last_unstale":"2024-09-21T13:55:13.025481+0000","last_undegraded":"2024-09-21T13:55:13.025481+0000","last_fullsized":"2024-09-21T13:55:13.025481+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-21T13:50:03.769613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:50:03.769613+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:37.036196+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":305,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.034321+0000","last_change":"2024-09-21T13:54:22.816889+0000","last_active":"2024-09-21T13:55:13.034321+0000","last_peered":"2024-09-21T13:55:13.034321+0000","last_clean":"2024-09-21T13:55:13.034321+0000","last_became_active":"2024-09-21T13:48:58.053641+0000","last_became_peered":"2024-09-21T13:48:58.053641+0000","last_unstale":"2024-09-21T13:55:13.034321+0000","last_undegraded":"2024-09-21T13:55:13.034321+0000","last_fullsized":"2024-09-21T13:55:13.034321+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:22.816815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:22.816815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:52.549192+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":253,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-21T13:55:04.967939+0000","last_change":"2024-09-21T13:53:53.244317+0000","last_active":"2024-09-21T13:55:04.967939+0000","last_peered":"2024-09-21T13:55:04.967939+0000","last_clean":"2024-09-21T13:55:04.967939+0000","last_became_active":"2024-09-21T13:52:04.321146+0000","last_became_peered":"2024-09-21T13:52:04.321146+0000","last_unstale":"2024-09-21T13:55:04.967939+0000","last_undegraded":"2024-09-21T13:55:04.967939+0000","last_fullsized":"2024-09-21T13:55:04.967939+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:53.244131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:53:53.244131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:13.809484+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":312,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.033624+0000","last_change":"2024-09-21T13:54:03.688017+0000","last_active":"2024-09-21T13:55:13.033624+0000","last_peered":"2024-09-21T13:55:13.033624+0000","last_clean":"2024-09-21T13:55:13.033624+0000","last_became_active":"2024-09-21T13:52:04.322213+0000","last_became_peered":"2024-09-21T13:52:04.322213+0000","last_unstale":"2024-09-21T13:55:13.033624+0000","last_undegraded":"2024-09-21T13:55:13.033624+0000","last_fullsized":"2024-09-21T13:55:13.033624+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:03.687971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:54:03.687971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:28.966237+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"203'5829","reported_seq":5880,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.449361+0000","last_change":"2024-09-21T13:54:12.478639+0000","last_active":"2024-09-21T13:55:13.449361+0000","last_peered":"2024-09-21T13:55:13.449361+0000","last_clean":"2024-09-21T13:55:13.449361+0000","last_became_active":"2024-09-21T13:54:12.478330+0000","last_became_peered":"2024-09-21T13:54:12.478330+0000","last_unstale":"2024-09-21T13:55:13.449361+0000","last_undegraded":"2024-09-21T13:55:13.449361+0000","last_fullsized":"2024-09-21T13:55:13.449361+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":5829,"log_dups_size":0,"ondisk_log_size":5829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:36.845419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382009344,"num_objects":5829,"num_object_clones":0,"num_object_copies":17487,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5829,"num_write_kb":373056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"68'12","reported_seq":332,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.012505+0000","last_change":"2024-09-21T13:53:57.288526+0000","last_active":"2024-09-21T13:55:13.012505+0000","last_peered":"2024-09-21T13:55:13.012505+0000","last_clean":"2024-09-21T13:55:13.012505+0000","last_became_active":"2024-09-21T13:49:00.059720+0000","last_became_peered":"2024-09-21T13:49:00.059720+0000","last_unstale":"2024-09-21T13:55:13.012505+0000","last_undegraded":"2024-09-21T13:55:13.012505+0000","last_fullsized":"2024-09-21T13:55:13.012505+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'12","last_scrub_stamp":"2024-09-21T13:53:57.288451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:57.288451+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":305,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.033678+0000","last_change":"2024-09-21T13:50:29.100595+0000","last_active":"2024-09-21T13:55:13.033678+0000","last_peered":"2024-09-21T13:55:13.033678+0000","last_clean":"2024-09-21T13:55:13.033678+0000","last_became_active":"2024-09-21T13:48:58.057641+0000","last_became_peered":"2024-09-21T13:48:58.057641+0000","last_unstale":"2024-09-21T13:55:13.033678+0000","last_undegraded":"2024-09-21T13:55:13.033678+0000","last_fullsized":"2024-09-21T13:55:13.033678+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:29.100547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:29.100547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:35.899528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":226,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-21T13:55:04.967586+0000","last_change":"2024-09-21T13:54:55.924559+0000","last_active":"2024-09-21T13:55:04.967586+0000","last_peered":"2024-09-21T13:55:04.967586+0000","last_clean":"2024-09-21T13:55:04.967586+0000","last_became_active":"2024-09-21T13:52:04.322147+0000","last_became_peered":"2024-09-21T13:52:04.322147+0000","last_unstale":"2024-09-21T13:55:04.967586+0000","last_undegraded":"2024-09-21T13:55:04.967586+0000","last_fullsized":"2024-09-21T13:55:04.967586+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:55.924499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:55.924499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:58.811467+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":298,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-21T13:55:09.040138+0000","last_change":"2024-09-21T13:55:09.040138+0000","last_active":"2024-09-21T13:55:09.040138+0000","last_peered":"2024-09-21T13:55:09.040138+0000","last_clean":"2024-09-21T13:55:09.040138+0000","last_became_active":"2024-09-21T13:49:04.078089+0000","last_became_peered":"2024-09-21T13:49:04.078089+0000","last_unstale":"2024-09-21T13:55:09.040138+0000","last_undegraded":"2024-09-21T13:55:09.040138+0000","last_fullsized":"2024-09-21T13:55:09.040138+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:09.040062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:55:09.040062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:56:17.791090+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"203'5745","reported_seq":5796,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.411564+0000","last_change":"2024-09-21T13:54:12.477912+0000","last_active":"2024-09-21T13:55:13.411564+0000","last_peered":"2024-09-21T13:55:13.411564+0000","last_clean":"2024-09-21T13:55:13.411564+0000","last_became_active":"2024-09-21T13:54:12.477455+0000","last_became_peered":"2024-09-21T13:54:12.477455+0000","last_unstale":"2024-09-21T13:55:13.411564+0000","last_undegraded":"2024-09-21T13:55:13.411564+0000","last_fullsized":"2024-09-21T13:55:13.411564+0000","mapping_epoch":183,"log_start":"187'300","ondisk_log_start":"187'300","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":5445,"log_dups_size":0,"ondisk_log_size":5445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:26.894106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376504320,"num_objects":5745,"num_object_clones":0,"num_object_copies":17235,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5745,"num_write_kb":367680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"68'15","reported_seq":343,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.014328+0000","last_change":"2024-09-21T13:54:58.947910+0000","last_active":"2024-09-21T13:55:13.014328+0000","last_peered":"2024-09-21T13:55:13.014328+0000","last_clean":"2024-09-21T13:55:13.014328+0000","last_became_active":"2024-09-21T13:49:00.425997+0000","last_became_peered":"2024-09-21T13:49:00.425997+0000","last_unstale":"2024-09-21T13:55:13.014328+0000","last_undegraded":"2024-09-21T13:55:13.014328+0000","last_fullsized":"2024-09-21T13:55:13.014328+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:54:58.947824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:58.947824+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:56:13.701891+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":333,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.032991+0000","last_change":"2024-09-21T13:55:03.879782+0000","last_active":"2024-09-21T13:55:13.032991+0000","last_peered":"2024-09-21T13:55:13.032991+0000","last_clean":"2024-09-21T13:55:13.032991+0000","last_became_active":"2024-09-21T13:48:58.057632+0000","last_became_peered":"2024-09-21T13:48:58.057632+0000","last_unstale":"2024-09-21T13:55:13.032991+0000","last_undegraded":"2024-09-21T13:55:13.032991+0000","last_fullsized":"2024-09-21T13:55:13.032991+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T13:55:03.879706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:55:03.879706+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:56:28.070727+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"68'8","reported_seq":381,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.032772+0000","last_change":"2024-09-21T13:51:26.297643+0000","last_active":"2024-09-21T13:55:13.032772+0000","last_peered":"2024-09-21T13:55:13.032772+0000","last_clean":"2024-09-21T13:55:13.032772+0000","last_became_active":"2024-09-21T13:49:02.056167+0000","last_became_peered":"2024-09-21T13:49:02.056167+0000","last_unstale":"2024-09-21T13:55:13.032772+0000","last_undegraded":"2024-09-21T13:55:13.032772+0000","last_fullsized":"2024-09-21T13:55:13.032772+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:51:26.297550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:26.297550+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:33.879499+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":310,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.014423+0000","last_change":"2024-09-21T13:54:21.108175+0000","last_active":"2024-09-21T13:55:13.014423+0000","last_peered":"2024-09-21T13:55:13.014423+0000","last_clean":"2024-09-21T13:55:13.014423+0000","last_became_active":"2024-09-21T13:49:04.432546+0000","last_became_peered":"2024-09-21T13:49:04.432546+0000","last_unstale":"2024-09-21T13:55:13.014423+0000","last_undegraded":"2024-09-21T13:55:13.014423+0000","last_fullsized":"2024-09-21T13:55:13.014423+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:21.108123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:54:21.108123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:24.708631+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":322,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.025651+0000","last_change":"2024-09-21T13:54:18.131012+0000","last_active":"2024-09-21T13:55:13.025651+0000","last_peered":"2024-09-21T13:55:13.025651+0000","last_clean":"2024-09-21T13:55:13.025651+0000","last_became_active":"2024-09-21T13:48:58.056080+0000","last_became_peered":"2024-09-21T13:48:58.056080+0000","last_unstale":"2024-09-21T13:55:13.025651+0000","last_undegraded":"2024-09-21T13:55:13.025651+0000","last_fullsized":"2024-09-21T13:55:13.025651+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:18.130969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:18.130969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:34.242253+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"68'6","reported_seq":307,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-21T13:55:04.954548+0000","last_change":"2024-09-21T13:53:31.807215+0000","last_active":"2024-09-21T13:55:04.954548+0000","last_peered":"2024-09-21T13:55:04.954548+0000","last_clean":"2024-09-21T13:55:04.954548+0000","last_became_active":"2024-09-21T13:49:00.426829+0000","last_became_peered":"2024-09-21T13:49:00.426829+0000","last_unstale":"2024-09-21T13:55:04.954548+0000","last_undegraded":"2024-09-21T13:55:04.954548+0000","last_fullsized":"2024-09-21T13:55:04.954548+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'6","last_scrub_stamp":"2024-09-21T13:53:31.807028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:31.807028+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":314,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.014821+0000","last_change":"2024-09-21T13:54:18.124591+0000","last_active":"2024-09-21T13:55:13.014821+0000","last_peered":"2024-09-21T13:55:13.014821+0000","last_clean":"2024-09-21T13:55:13.014821+0000","last_became_active":"2024-09-21T13:49:02.062231+0000","last_became_peered":"2024-09-21T13:49:02.062231+0000","last_unstale":"2024-09-21T13:55:13.014821+0000","last_undegraded":"2024-09-21T13:55:13.014821+0000","last_fullsized":"2024-09-21T13:55:13.014821+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:18.124544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:18.124544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:39.080321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":309,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.033927+0000","last_change":"2024-09-21T13:55:02.838227+0000","last_active":"2024-09-21T13:55:13.033927+0000","last_peered":"2024-09-21T13:55:13.033927+0000","last_clean":"2024-09-21T13:55:13.033927+0000","last_became_active":"2024-09-21T13:49:04.432507+0000","last_became_peered":"2024-09-21T13:49:04.432507+0000","last_unstale":"2024-09-21T13:55:13.033927+0000","last_undegraded":"2024-09-21T13:55:13.033927+0000","last_fullsized":"2024-09-21T13:55:13.033927+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:02.838181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:55:02.838181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:56:13.397863+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"68'9","reported_seq":278,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-21T13:55:04.990924+0000","last_change":"2024-09-21T13:55:04.990924+0000","last_active":"2024-09-21T13:55:04.990924+0000","last_peered":"2024-09-21T13:55:04.990924+0000","last_clean":"2024-09-21T13:55:04.990924+0000","last_became_active":"2024-09-21T13:50:29.693589+0000","last_became_peered":"2024-09-21T13:50:29.693589+0000","last_unstale":"2024-09-21T13:55:04.990924+0000","last_undegraded":"2024-09-21T13:55:04.990924+0000","last_fullsized":"2024-09-21T13:55:04.990924+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:55:04.990842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:55:04.990842+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:56:34.158483+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"61'1","reported_seq":311,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.032933+0000","last_change":"2024-09-21T13:54:18.838960+0000","last_active":"2024-09-21T13:55:13.032933+0000","last_peered":"2024-09-21T13:55:13.032933+0000","last_clean":"2024-09-21T13:55:13.032933+0000","last_became_active":"2024-09-21T13:48:58.050645+0000","last_became_peered":"2024-09-21T13:48:58.050645+0000","last_unstale":"2024-09-21T13:55:13.032933+0000","last_undegraded":"2024-09-21T13:55:13.032933+0000","last_fullsized":"2024-09-21T13:55:13.032933+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:54:18.838906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:18.838906+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:26.089820+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":302,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-21T13:55:04.954308+0000","last_change":"2024-09-21T13:54:28.016449+0000","last_active":"2024-09-21T13:55:04.954308+0000","last_peered":"2024-09-21T13:55:04.954308+0000","last_clean":"2024-09-21T13:55:04.954308+0000","last_became_active":"2024-09-21T13:49:02.061775+0000","last_became_peered":"2024-09-21T13:49:02.061775+0000","last_unstale":"2024-09-21T13:55:04.954308+0000","last_undegraded":"2024-09-21T13:55:04.954308+0000","last_fullsized":"2024-09-21T13:55:04.954308+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:28.014760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:28.014760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:36.849367+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":241,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-21T13:55:04.968278+0000","last_change":"2024-09-21T13:52:04.324521+0000","last_active":"2024-09-21T13:55:04.968278+0000","last_peered":"2024-09-21T13:55:04.968278+0000","last_clean":"2024-09-21T13:55:04.968278+0000","last_became_active":"2024-09-21T13:52:04.323549+0000","last_became_peered":"2024-09-21T13:52:04.323549+0000","last_unstale":"2024-09-21T13:55:04.968278+0000","last_undegraded":"2024-09-21T13:55:04.968278+0000","last_fullsized":"2024-09-21T13:55:04.968278+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:41.808746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:41.808746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:35.892198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"68'9","reported_seq":318,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.015023+0000","last_change":"2024-09-21T13:51:28.547052+0000","last_active":"2024-09-21T13:55:13.015023+0000","last_peered":"2024-09-21T13:55:13.015023+0000","last_clean":"2024-09-21T13:55:13.015023+0000","last_became_active":"2024-09-21T13:49:00.427766+0000","last_became_peered":"2024-09-21T13:49:00.427766+0000","last_unstale":"2024-09-21T13:55:13.015023+0000","last_undegraded":"2024-09-21T13:55:13.015023+0000","last_fullsized":"2024-09-21T13:55:13.015023+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:51:28.546958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:28.546958+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":246,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.033358+0000","last_change":"2024-09-21T13:54:02.683225+0000","last_active":"2024-09-21T13:55:13.033358+0000","last_peered":"2024-09-21T13:55:13.033358+0000","last_clean":"2024-09-21T13:55:13.033358+0000","last_became_active":"2024-09-21T13:50:29.690634+0000","last_became_peered":"2024-09-21T13:50:29.690634+0000","last_unstale":"2024-09-21T13:55:13.033358+0000","last_undegraded":"2024-09-21T13:55:13.033358+0000","last_fullsized":"2024-09-21T13:55:13.033358+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:02.683172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:02.683172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:13.538040+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"132'10","reported_seq":398,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-21T13:55:04.950348+0000","last_change":"2024-09-21T13:54:20.946861+0000","last_active":"2024-09-21T13:55:04.950348+0000","last_peered":"2024-09-21T13:55:04.950348+0000","last_clean":"2024-09-21T13:55:04.950348+0000","last_became_active":"2024-09-21T13:52:04.318626+0000","last_became_peered":"2024-09-21T13:52:04.318626+0000","last_unstale":"2024-09-21T13:55:04.950348+0000","last_undegraded":"2024-09-21T13:55:04.950348+0000","last_fullsized":"2024-09-21T13:55:04.950348+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'10","last_scrub_stamp":"2024-09-21T13:54:20.946782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:20.946782+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:42.458501+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":312,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.012707+0000","last_change":"2024-09-21T13:53:51.285122+0000","last_active":"2024-09-21T13:55:13.012707+0000","last_peered":"2024-09-21T13:55:13.012707+0000","last_clean":"2024-09-21T13:55:13.012707+0000","last_became_active":"2024-09-21T13:52:04.315865+0000","last_became_peered":"2024-09-21T13:52:04.315865+0000","last_unstale":"2024-09-21T13:55:13.012707+0000","last_undegraded":"2024-09-21T13:55:13.012707+0000","last_fullsized":"2024-09-21T13:55:13.012707+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:51.285077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:51.285077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:16.987394+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"68'10","reported_seq":313,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-21T13:55:04.950686+0000","last_change":"2024-09-21T13:51:11.545546+0000","last_active":"2024-09-21T13:55:04.950686+0000","last_peered":"2024-09-21T13:55:04.950686+0000","last_clean":"2024-09-21T13:55:04.950686+0000","last_became_active":"2024-09-21T13:49:00.426071+0000","last_became_peered":"2024-09-21T13:49:00.426071+0000","last_unstale":"2024-09-21T13:55:04.950686+0000","last_undegraded":"2024-09-21T13:55:04.950686+0000","last_fullsized":"2024-09-21T13:55:04.950686+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:51:11.545458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:11.545458+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:42.030147+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":316,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.033092+0000","last_change":"2024-09-21T13:52:04.318335+0000","last_active":"2024-09-21T13:55:13.033092+0000","last_peered":"2024-09-21T13:55:13.033092+0000","last_clean":"2024-09-21T13:55:13.033092+0000","last_became_active":"2024-09-21T13:52:04.318006+0000","last_became_peered":"2024-09-21T13:52:04.318006+0000","last_unstale":"2024-09-21T13:55:13.033092+0000","last_undegraded":"2024-09-21T13:55:13.033092+0000","last_fullsized":"2024-09-21T13:55:13.033092+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:41.162315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:50:41.162315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:42.880310+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"68'8","reported_seq":382,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.014899+0000","last_change":"2024-09-21T13:54:03.065465+0000","last_active":"2024-09-21T13:55:13.014899+0000","last_peered":"2024-09-21T13:55:13.014899+0000","last_clean":"2024-09-21T13:55:13.014899+0000","last_became_active":"2024-09-21T13:49:02.067487+0000","last_became_peered":"2024-09-21T13:49:02.067487+0000","last_unstale":"2024-09-21T13:55:13.014899+0000","last_undegraded":"2024-09-21T13:55:13.014899+0000","last_fullsized":"2024-09-21T13:55:13.014899+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:54:03.065395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:03.065395+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:30.943315+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":245,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.033224+0000","last_change":"2024-09-21T13:54:27.842220+0000","last_active":"2024-09-21T13:55:13.033224+0000","last_peered":"2024-09-21T13:55:13.033224+0000","last_clean":"2024-09-21T13:55:13.033224+0000","last_became_active":"2024-09-21T13:50:29.695082+0000","last_became_peered":"2024-09-21T13:50:29.695082+0000","last_unstale":"2024-09-21T13:55:13.033224+0000","last_undegraded":"2024-09-21T13:55:13.033224+0000","last_fullsized":"2024-09-21T13:55:13.033224+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:27.842150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:54:27.842150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:35.545905+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"68'11","reported_seq":336,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.012248+0000","last_change":"2024-09-21T13:54:56.539441+0000","last_active":"2024-09-21T13:55:13.012248+0000","last_peered":"2024-09-21T13:55:13.012248+0000","last_clean":"2024-09-21T13:55:13.012248+0000","last_became_active":"2024-09-21T13:49:00.055024+0000","last_became_peered":"2024-09-21T13:49:00.055024+0000","last_unstale":"2024-09-21T13:55:13.012248+0000","last_undegraded":"2024-09-21T13:55:13.012248+0000","last_fullsized":"2024-09-21T13:55:13.012248+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:54:56.539374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:56.539374+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:56:07.801920+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":60,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.033412+0000","last_change":"2024-09-21T13:54:04.377034+0000","last_active":"2024-09-21T13:55:13.033412+0000","last_peered":"2024-09-21T13:55:13.033412+0000","last_clean":"2024-09-21T13:55:13.033412+0000","last_became_active":"2024-09-21T13:54:04.376614+0000","last_became_peered":"2024-09-21T13:54:04.376614+0000","last_unstale":"2024-09-21T13:55:13.033412+0000","last_undegraded":"2024-09-21T13:55:13.033412+0000","last_fullsized":"2024-09-21T13:55:13.033412+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:01.228213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:01.228213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:14.625514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":313,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.022675+0000","last_change":"2024-09-21T13:53:29.868061+0000","last_active":"2024-09-21T13:55:13.022675+0000","last_peered":"2024-09-21T13:55:13.022675+0000","last_clean":"2024-09-21T13:55:13.022675+0000","last_became_active":"2024-09-21T13:49:02.062868+0000","last_became_peered":"2024-09-21T13:49:02.062868+0000","last_unstale":"2024-09-21T13:55:13.022675+0000","last_undegraded":"2024-09-21T13:55:13.022675+0000","last_fullsized":"2024-09-21T13:55:13.022675+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:29.867278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:53:29.867278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:51.970105+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":291,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-21T13:55:04.948043+0000","last_change":"2024-09-21T13:51:24.549993+0000","last_active":"2024-09-21T13:55:04.948043+0000","last_peered":"2024-09-21T13:55:04.948043+0000","last_clean":"2024-09-21T13:55:04.948043+0000","last_became_active":"2024-09-21T13:49:04.082961+0000","last_became_peered":"2024-09-21T13:49:04.082961+0000","last_unstale":"2024-09-21T13:55:04.948043+0000","last_undegraded":"2024-09-21T13:55:04.948043+0000","last_fullsized":"2024-09-21T13:55:04.948043+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:24.549706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:51:24.549706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:48.045901+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"68'9","reported_seq":328,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.023655+0000","last_change":"2024-09-21T13:52:10.447789+0000","last_active":"2024-09-21T13:55:13.023655+0000","last_peered":"2024-09-21T13:55:13.023655+0000","last_clean":"2024-09-21T13:55:13.023655+0000","last_became_active":"2024-09-21T13:52:04.320149+0000","last_became_peered":"2024-09-21T13:52:04.320149+0000","last_unstale":"2024-09-21T13:55:13.023655+0000","last_undegraded":"2024-09-21T13:55:13.023655+0000","last_fullsized":"2024-09-21T13:55:13.023655+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:52:10.447530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:52:10.447530+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:28.845521+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":305,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.034216+0000","last_change":"2024-09-21T13:54:06.687343+0000","last_active":"2024-09-21T13:55:13.034216+0000","last_peered":"2024-09-21T13:55:13.034216+0000","last_clean":"2024-09-21T13:55:13.034216+0000","last_became_active":"2024-09-21T13:48:58.057516+0000","last_became_peered":"2024-09-21T13:48:58.057516+0000","last_unstale":"2024-09-21T13:55:13.034216+0000","last_undegraded":"2024-09-21T13:55:13.034216+0000","last_fullsized":"2024-09-21T13:55:13.034216+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:06.687302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:06.687302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:28.415240+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":302,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-21T13:55:04.948560+0000","last_change":"2024-09-21T13:54:19.999192+0000","last_active":"2024-09-21T13:55:04.948560+0000","last_peered":"2024-09-21T13:55:04.948560+0000","last_clean":"2024-09-21T13:55:04.948560+0000","last_became_active":"2024-09-21T13:49:02.071922+0000","last_became_peered":"2024-09-21T13:49:02.071922+0000","last_unstale":"2024-09-21T13:55:04.948560+0000","last_undegraded":"2024-09-21T13:55:04.948560+0000","last_fullsized":"2024-09-21T13:55:04.948560+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:19.999125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:19.999125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:38.368141+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":231,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.014340+0000","last_change":"2024-09-21T13:50:29.689537+0000","last_active":"2024-09-21T13:55:13.014340+0000","last_peered":"2024-09-21T13:55:13.014340+0000","last_clean":"2024-09-21T13:55:13.014340+0000","last_became_active":"2024-09-21T13:50:29.689045+0000","last_became_peered":"2024-09-21T13:50:29.689045+0000","last_unstale":"2024-09-21T13:55:13.014340+0000","last_undegraded":"2024-09-21T13:55:13.014340+0000","last_fullsized":"2024-09-21T13:55:13.014340+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:14.052362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"68'11","reported_seq":326,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-21T13:55:04.948179+0000","last_change":"2024-09-21T13:54:22.978354+0000","last_active":"2024-09-21T13:55:04.948179+0000","last_peered":"2024-09-21T13:55:04.948179+0000","last_clean":"2024-09-21T13:55:04.948179+0000","last_became_active":"2024-09-21T13:49:00.426091+0000","last_became_peered":"2024-09-21T13:49:00.426091+0000","last_unstale":"2024-09-21T13:55:04.948179+0000","last_undegraded":"2024-09-21T13:55:04.948179+0000","last_fullsized":"2024-09-21T13:55:04.948179+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:54:22.978267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:22.978267+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:40.972799+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":298,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.014897+0000","last_change":"2024-09-21T13:48:58.052281+0000","last_active":"2024-09-21T13:55:13.014897+0000","last_peered":"2024-09-21T13:55:13.014897+0000","last_clean":"2024-09-21T13:55:13.014897+0000","last_became_active":"2024-09-21T13:48:58.051974+0000","last_became_peered":"2024-09-21T13:48:58.051974+0000","last_unstale":"2024-09-21T13:55:13.014897+0000","last_undegraded":"2024-09-21T13:55:13.014897+0000","last_fullsized":"2024-09-21T13:55:13.014897+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:48:57.035903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":305,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.033441+0000","last_change":"2024-09-21T13:55:12.792015+0000","last_active":"2024-09-21T13:55:13.033441+0000","last_peered":"2024-09-21T13:55:13.033441+0000","last_clean":"2024-09-21T13:55:13.033441+0000","last_became_active":"2024-09-21T13:49:02.056006+0000","last_became_peered":"2024-09-21T13:49:02.056006+0000","last_unstale":"2024-09-21T13:55:13.033441+0000","last_undegraded":"2024-09-21T13:55:13.033441+0000","last_fullsized":"2024-09-21T13:55:13.033441+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:12.791948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:55:12.791948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:56:27.095950+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":298,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-21T13:55:04.948775+0000","last_change":"2024-09-21T13:54:57.057734+0000","last_active":"2024-09-21T13:55:04.948775+0000","last_peered":"2024-09-21T13:55:04.948775+0000","last_clean":"2024-09-21T13:55:04.948775+0000","last_became_active":"2024-09-21T13:49:04.434034+0000","last_became_peered":"2024-09-21T13:49:04.434034+0000","last_unstale":"2024-09-21T13:55:04.948775+0000","last_undegraded":"2024-09-21T13:55:04.948775+0000","last_fullsized":"2024-09-21T13:55:04.948775+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:57.057668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:54:57.057668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:56:01.348556+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"68'4","reported_seq":304,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-21T13:55:04.949851+0000","last_change":"2024-09-21T13:53:27.826804+0000","last_active":"2024-09-21T13:55:04.949851+0000","last_peered":"2024-09-21T13:55:04.949851+0000","last_clean":"2024-09-21T13:55:04.949851+0000","last_became_active":"2024-09-21T13:49:00.063066+0000","last_became_peered":"2024-09-21T13:49:00.063066+0000","last_unstale":"2024-09-21T13:55:04.949851+0000","last_undegraded":"2024-09-21T13:55:04.949851+0000","last_fullsized":"2024-09-21T13:55:04.949851+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4","last_scrub_stamp":"2024-09-21T13:53:27.826729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:27.826729+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":246,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.015123+0000","last_change":"2024-09-21T13:55:02.998703+0000","last_active":"2024-09-21T13:55:13.015123+0000","last_peered":"2024-09-21T13:55:13.015123+0000","last_clean":"2024-09-21T13:55:13.015123+0000","last_became_active":"2024-09-21T13:50:29.690269+0000","last_became_peered":"2024-09-21T13:50:29.690269+0000","last_unstale":"2024-09-21T13:55:13.015123+0000","last_undegraded":"2024-09-21T13:55:13.015123+0000","last_fullsized":"2024-09-21T13:55:13.015123+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:02.998650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:55:02.998650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:56:09.914764+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":314,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.025395+0000","last_change":"2024-09-21T13:53:51.254522+0000","last_active":"2024-09-21T13:55:13.025395+0000","last_peered":"2024-09-21T13:55:13.025395+0000","last_clean":"2024-09-21T13:55:13.025395+0000","last_became_active":"2024-09-21T13:49:02.064789+0000","last_became_peered":"2024-09-21T13:49:02.064789+0000","last_unstale":"2024-09-21T13:55:13.025395+0000","last_undegraded":"2024-09-21T13:55:13.025395+0000","last_fullsized":"2024-09-21T13:55:13.025395+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:51.254456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:53:51.254456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:51.965028+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":312,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.033034+0000","last_change":"2024-09-21T13:55:08.852877+0000","last_active":"2024-09-21T13:55:13.033034+0000","last_peered":"2024-09-21T13:55:13.033034+0000","last_clean":"2024-09-21T13:55:13.033034+0000","last_became_active":"2024-09-21T13:52:04.321667+0000","last_became_peered":"2024-09-21T13:52:04.321667+0000","last_unstale":"2024-09-21T13:55:13.033034+0000","last_undegraded":"2024-09-21T13:55:13.033034+0000","last_fullsized":"2024-09-21T13:55:13.033034+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:08.852789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:55:08.852789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:56:31.654561+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":301,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.024737+0000","last_change":"2024-09-21T13:52:27.409588+0000","last_active":"2024-09-21T13:55:13.024737+0000","last_peered":"2024-09-21T13:55:13.024737+0000","last_clean":"2024-09-21T13:55:13.024737+0000","last_became_active":"2024-09-21T13:49:04.075562+0000","last_became_peered":"2024-09-21T13:49:04.075562+0000","last_unstale":"2024-09-21T13:55:13.024737+0000","last_undegraded":"2024-09-21T13:55:13.024737+0000","last_fullsized":"2024-09-21T13:55:13.024737+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:27.409408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:27.409408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:39.987239+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":302,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-09-21T13:55:04.950180+0000","last_change":"2024-09-21T13:53:25.873998+0000","last_active":"2024-09-21T13:55:04.950180+0000","last_peered":"2024-09-21T13:55:04.950180+0000","last_clean":"2024-09-21T13:55:04.950180+0000","last_became_active":"2024-09-21T13:48:58.055497+0000","last_became_peered":"2024-09-21T13:48:58.055497+0000","last_unstale":"2024-09-21T13:55:04.950180+0000","last_undegraded":"2024-09-21T13:55:04.950180+0000","last_fullsized":"2024-09-21T13:55:04.950180+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:25.865470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:53:25.865470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"68'11","reported_seq":359,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.026093+0000","last_change":"2024-09-21T13:55:12.198622+0000","last_active":"2024-09-21T13:55:13.026093+0000","last_peered":"2024-09-21T13:55:13.026093+0000","last_clean":"2024-09-21T13:55:13.026093+0000","last_became_active":"2024-09-21T13:52:04.320647+0000","last_became_peered":"2024-09-21T13:52:04.320647+0000","last_unstale":"2024-09-21T13:55:13.026093+0000","last_undegraded":"2024-09-21T13:55:13.026093+0000","last_fullsized":"2024-09-21T13:55:13.026093+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:55:12.198562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:55:12.198562+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:56:27.099327+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":257,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-21T13:55:13.033152+0000","last_change":"2024-09-21T13:54:01.727728+0000","last_active":"2024-09-21T13:55:13.033152+0000","last_peered":"2024-09-21T13:55:13.033152+0000","last_clean":"2024-09-21T13:55:13.033152+0000","last_became_active":"2024-09-21T13:52:04.322880+0000","last_became_peered":"2024-09-21T13:52:04.322880+0000","last_unstale":"2024-09-21T13:55:13.033152+0000","last_undegraded":"2024-09-21T13:55:13.033152+0000","last_fullsized":"2024-09-21T13:55:13.033152+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:01.727680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:01.727680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:55:16.479686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":5912395776,"num_objects":90216,"num_object_clones":0,"num_object_copies":270648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90216,"num_write_kb":5773824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17770283008,"data_stored":17770283008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84516,"ondisk_log_size":84516,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69,"ondisk_log_size":69,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":694,"num_read_kb":451,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":395,"ondisk_log_size":395,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":370,"num_read_kb":365,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234064,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1989428,"kb_used_data":1818020,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":91755724,"statfs":{"total":95995035648,"available":93957861376,"internally_reserved":0,"allocated":1861652480,"data_stored":1861088812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":175503818},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462997,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1986096,"kb_used_data":1820252,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":91759056,"statfs":{"total":95995035648,"available":93961273344,"internally_reserved":0,"allocated":1863938048,"data_stored":1863389625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":169802182},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724636,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3085872,"kb_used_data":2906140,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":90659280,"statfs":{"total":95995035648,"available":92835102720,"internally_reserved":0,"allocated":2975887360,"data_stored":2975339309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":184023493},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920866,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1976396,"kb_used_data":1823032,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":91768756,"statfs":{"total":95995035648,"available":93971206144,"internally_reserved":0,"allocated":1866784768,"data_stored":1866207747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":157022661},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117094,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2371968,"kb_used_data":2065904,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":91373184,"statfs":{"total":95995035648,"available":93566140416,"internally_reserved":0,"allocated":2115485696,"data_stored":2114909559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":313391559},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":130,"seq":558345748519,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2414500,"kb_used_data":2157396,"kb_used_omap":4,"kb_used_meta":257083,"kb_avail":91330652,"statfs":{"total":95995035648,"available":93522587648,"internally_reserved":0,"allocated":2209173504,"data_stored":2208625529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4441,"internal_metadata":263253671},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574964,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2756756,"kb_used_data":2522884,"kb_used_omap":1,"kb_used_meta":233854,"kb_avail":90988396,"statfs":{"total":95995035648,"available":93172117504,"internally_reserved":0,"allocated":2583433216,"data_stored":2582836293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":239466949},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738490,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2442244,"kb_used_data":2162544,"kb_used_omap":1,"kb_used_meta":279678,"kb_avail":91302908,"statfs":{"total":95995035648,"available":93494177792,"internally_reserved":0,"allocated":2214445056,"data_stored":2213876511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":286390725},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2234515456,"data_stored":2234515456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2606170112,"data_stored":2606170112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2218065920,"data_stored":2218065920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2115502080,"data_stored":2115502080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1866530816,"data_stored":1866530816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2984640512,"data_stored":2984640512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1869086720,"data_stored":1869086720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1875771392,"data_stored":1875771392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T13:55:17.447 INFO:tasks.thrashosds.thrasher:Setting 8.c to [4, 5, 6, 3, 2, 1] 2024-09-21T13:55:17.447 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '8.c', '4', '5', '6', '3', '2', '1'] 2024-09-21T13:55:17.448 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd pg-upmap-items 8.c 4 5 6 3 2 1 2024-09-21T13:55:17.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:17.448 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:17.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:55:17.449 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:17 smithi187 ceph-mon[33032]: pgmap v307: 148 pgs: 148 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-09-21T13:55:17.449 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:17 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2701679214' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:55:17.550 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:55:17.606 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:17.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:17.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:55:17.707 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:17.739 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:17.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:17.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:17.783 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T13:55:17.841 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:55:17.868 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:17.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:17.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:55:17.969 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:55:18.012 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:18.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:18.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:55:18.114 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:18.141 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:18.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:18.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:18.242 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:55:18.282 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:18.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:18.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:55:18.298 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:2024-09-21T13:55:18.298930+0000 min lat: 0.00131689 max lat: 0.181286 avg lat: 0.00994068 2024-09-21T13:55:18.298 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T13:55:18.298 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 60 16 96564 96548 100.549 70.5625 0.00703747 0.00994068 2024-09-21T13:55:18.384 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:18.415 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:18.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:18.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:18.516 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:18.541 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:18.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:18.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:18.608 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:55:18.609 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:55:18.630 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:55:18.643 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:18.669 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:18.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:18.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:18.770 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:18.798 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:18.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:18.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:18.900 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:55:18.938 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:18.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:18.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:55:19.039 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:55:19.079 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:19.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:19.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:55:19.181 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:55:19.231 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:19.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:19.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:55:19.298 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 61 16 97679 97663 100.043 69.6875 0.0126644 0.00999031 2024-09-21T13:55:19.334 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:55:19.376 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:19.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:19.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:55:19.411 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:19 smithi049 ceph-mon[40489]: pgmap v308: 148 pgs: 148 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 45 MiB/s wr, 725 op/s 2024-09-21T13:55:19.411 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:19 smithi049 ceph-mon[30080]: pgmap v308: 148 pgs: 148 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 45 MiB/s wr, 725 op/s 2024-09-21T13:55:19.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:19 smithi187 ceph-mon[33032]: pgmap v308: 148 pgs: 148 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 45 MiB/s wr, 725 op/s 2024-09-21T13:55:19.444 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:55:19 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:55:19.478 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:19.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:19.510 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:19.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:19.611 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:19.644 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:19.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:19.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:19.745 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:55:19.786 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:19.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:19.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:55:19.887 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:55:19.916 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:19.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:19.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:55:19.969 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T13:55:19.970 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-09-21T13:55:19.970 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-09-21T13:55:19.970 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-09-21T13:55:19.970 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T13:55:20.017 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:55:20.059 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:20.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:20.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:55:20.084 INFO:teuthology.orchestra.run.smithi049.stderr:set 8.c pg_upmap_items mapping to [4->5,6->3,2->1] 2024-09-21T13:55:20.161 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:20.190 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:20.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:20.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:20.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:20 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:55:20.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:20 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2311972848' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "8.c", "id": [4, 5, 6, 3, 2, 1]} : dispatch 2024-09-21T13:55:20.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:20 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "8.c", "id": [4, 5, 6, 3, 2, 1]} : dispatch 2024-09-21T13:55:20.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:20 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:55:20.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:20 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2311972848' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "8.c", "id": [4, 5, 6, 3, 2, 1]} : dispatch 2024-09-21T13:55:20.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:20 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "8.c", "id": [4, 5, 6, 3, 2, 1]} : dispatch 2024-09-21T13:55:20.292 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:55:20.298 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 62 16 98734 98718 99.4927 65.9375 0.0132588 0.0100452 2024-09-21T13:55:20.320 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:20.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:20.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:55:20.422 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:20.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:20 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:55:20.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:20 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2311972848' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "8.c", "id": [4, 5, 6, 3, 2, 1]} : dispatch 2024-09-21T13:55:20.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:20 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "8.c", "id": [4, 5, 6, 3, 2, 1]} : dispatch 2024-09-21T13:55:20.452 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:20.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:20.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:20.544 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T13:55:20.553 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:55:20.581 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:20.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:20.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:55:20.682 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:55:20.715 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:20.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:20.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:55:20.817 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:55:20.843 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:20.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:20.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:55:20.945 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:55:20.977 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:20.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:20.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:55:21.004 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:55:20 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:55:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:55:21.079 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:55:21.128 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:21.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:21.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:55:21.230 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:21.253 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:21.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:21.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:21.298 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 63 15 99759 99744 98.9313 64.125 0.0149261 0.0101035 2024-09-21T13:55:21.355 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:55:21.388 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:21.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:21.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:55:21.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:21 smithi187 ceph-mon[33032]: pgmap v309: 148 pgs: 148 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 89 MiB/s wr, 1.43k op/s 2024-09-21T13:55:21.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:21 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "8.c", "id": [4, 5, 6, 3, 2, 1]}]': finished 2024-09-21T13:55:21.446 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:21 smithi187 ceph-mon[33032]: osdmap e205: 8 total, 8 up, 8 in 2024-09-21T13:55:21.446 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:21 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1960019131' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:55:21.490 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:21.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:21 smithi049 ceph-mon[40489]: pgmap v309: 148 pgs: 148 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 89 MiB/s wr, 1.43k op/s 2024-09-21T13:55:21.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:21 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "8.c", "id": [4, 5, 6, 3, 2, 1]}]': finished 2024-09-21T13:55:21.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:21 smithi049 ceph-mon[40489]: osdmap e205: 8 total, 8 up, 8 in 2024-09-21T13:55:21.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:21 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1960019131' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:55:21.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:21 smithi049 ceph-mon[30080]: pgmap v309: 148 pgs: 148 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 89 MiB/s wr, 1.43k op/s 2024-09-21T13:55:21.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:21 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "8.c", "id": [4, 5, 6, 3, 2, 1]}]': finished 2024-09-21T13:55:21.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:21 smithi049 ceph-mon[30080]: osdmap e205: 8 total, 8 up, 8 in 2024-09-21T13:55:21.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:21 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1960019131' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:55:21.523 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:21.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:21.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:21.625 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:55:21.661 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:21.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:21.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:55:21.762 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:55:21.800 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:21.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:21.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:55:21.901 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:55:21.940 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:21.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:21.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:55:22.042 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:22.063 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:22.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:22.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:22.164 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:55:22.201 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:22.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:22.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:55:22.298 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 64 16 101007 100991 98.6032 77.9375 0.0030815 0.0101355 2024-09-21T13:55:22.303 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:22.325 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:22.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:22.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:22.427 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:55:22.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:22 smithi187 ceph-mon[33032]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:55:22.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:22 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1960019131' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T13:55:22.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:22 smithi187 ceph-mon[33032]: osdmap e206: 8 total, 8 up, 8 in 2024-09-21T13:55:22.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:22.468 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:22.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:55:22.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:22 smithi049 ceph-mon[40489]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:55:22.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:22 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1960019131' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T13:55:22.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:22 smithi049 ceph-mon[40489]: osdmap e206: 8 total, 8 up, 8 in 2024-09-21T13:55:22.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:22 smithi049 ceph-mon[30080]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:55:22.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:22 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1960019131' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T13:55:22.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:22 smithi049 ceph-mon[30080]: osdmap e206: 8 total, 8 up, 8 in 2024-09-21T13:55:22.569 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:55:22.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:22.602 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:22.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:55:22.703 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:55:22.714 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-09-21T13:55:22.738 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:22.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:22.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:55:22.840 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:55:22.875 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:22.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:22.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:55:22.976 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:55:22.998 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:22.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:22.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:55:23.100 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:23.126 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:23.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:23.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:23.228 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:55:23.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:23.267 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:23.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:55:23.298 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 65 16 102175 102159 98.2092 73 0.00612297 0.0101775 2024-09-21T13:55:23.368 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:55:23.405 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:23.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:23.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:55:23.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:23 smithi187 ceph-mon[33032]: pgmap v312: 148 pgs: 148 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 660 B/s rd, 54 MiB/s wr, 869 op/s 2024-09-21T13:55:23.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:23 smithi049 ceph-mon[30080]: pgmap v312: 148 pgs: 148 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 660 B/s rd, 54 MiB/s wr, 869 op/s 2024-09-21T13:55:23.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:23 smithi049 ceph-mon[40489]: pgmap v312: 148 pgs: 148 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 660 B/s rd, 54 MiB/s wr, 869 op/s 2024-09-21T13:55:23.507 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:23.538 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:23.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:23.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:23.640 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:23.676 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:23.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:23.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:23.777 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:23.804 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:23.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:23.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:23.906 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:55:23.942 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:23.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:23.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:55:24.044 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:24.076 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:24.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:24.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:24.177 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:55:24.215 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:24.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:24.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:55:24.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:24 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:55:24.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:24 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:55:24.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:24 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:55:24.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:24 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:55:24.298 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 66 16 103444 103428 97.9229 79.3125 0.0061696 0.010207 2024-09-21T13:55:24.317 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:24.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:24.357 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:24.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:24.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:24 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:55:24.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:24 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:55:24.458 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:55:24.488 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:24.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:24.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:55:24.590 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:55:24.627 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T13:55:24.656 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:24.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:24.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:55:24.758 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:55:24.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:24.812 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:24.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:55:24.913 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:55:24.977 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:24.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:24.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:55:25.079 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:55:25.099 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:55:25.105 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:25.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:25.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:55:25.207 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:55:25.228 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:25.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:25.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:55:25.299 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 67 16 104558 104542 97.5005 69.625 0.0173258 0.0102514 2024-09-21T13:55:25.330 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:25.362 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:25.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:25.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:25.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:25 smithi187 ceph-mon[33032]: pgmap v313: 148 pgs: 148 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-09-21T13:55:25.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:25 smithi187 ceph-mon[33032]: osdmap e207: 8 total, 8 up, 8 in 2024-09-21T13:55:25.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:25 smithi187 ceph-mon[33032]: osdmap e208: 8 total, 8 up, 8 in 2024-09-21T13:55:25.464 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:55:25.470 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:25 smithi049 ceph-mon[40489]: pgmap v313: 148 pgs: 148 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-09-21T13:55:25.470 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:25 smithi049 ceph-mon[40489]: osdmap e207: 8 total, 8 up, 8 in 2024-09-21T13:55:25.470 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:25 smithi049 ceph-mon[40489]: osdmap e208: 8 total, 8 up, 8 in 2024-09-21T13:55:25.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:25 smithi049 ceph-mon[30080]: pgmap v313: 148 pgs: 148 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-09-21T13:55:25.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:25 smithi049 ceph-mon[30080]: osdmap e207: 8 total, 8 up, 8 in 2024-09-21T13:55:25.471 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:25 smithi049 ceph-mon[30080]: osdmap e208: 8 total, 8 up, 8 in 2024-09-21T13:55:25.514 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:25.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:25.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:55:25.617 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:55:25.658 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:25.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:25.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:55:25.760 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:55:25.761 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:55:25.785 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:25.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:25.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:55:25.844 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:55:25.844 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:55:25.887 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:55:25.889 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:55:25.890 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:55:25.920 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:55:25.920 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:55:25.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:25.935 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:25.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:55:25.959 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:55:25.959 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:55:25.990 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:55:25.991 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:55:26.030 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:55:26.030 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:55:26.036 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:26.049 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T13:55:26.062 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:55:26.062 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:55:26.063 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:26.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:26.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:26.098 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:55:26.099 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:55:26.165 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:55:26.205 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:26.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:26.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:55:26.299 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 68 16 105679 105663 97.0969 70.0625 0.0216974 0.0102936 2024-09-21T13:55:26.307 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:55:26.355 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:26.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:26.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:55:26.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:26 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3953659540' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:55:26.457 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:55:26.487 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:26.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:26.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:55:26.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:26 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3953659540' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:55:26.491 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:26 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3953659540' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:55:26.589 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:55:26.634 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:26.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:26.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:55:26.736 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:55:26.769 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:26.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:26.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:55:26.870 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:55:26.908 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:26.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:26.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:55:27.010 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:55:27.055 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:27.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:27.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:55:27.157 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:55:27.186 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:27.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:27.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:55:27.192 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:27 smithi049 ceph-mon[40489]: pgmap v316: 148 pgs: 148 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 894 B/s rd, 62 MiB/s wr, 993 op/s 2024-09-21T13:55:27.192 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:27 smithi049 ceph-mon[30080]: pgmap v316: 148 pgs: 148 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 894 B/s rd, 62 MiB/s wr, 993 op/s 2024-09-21T13:55:27.288 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:55:27.299 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 69 16 106903 106887 96.7984 76.5 0.0139418 0.0103249 2024-09-21T13:55:27.331 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:27.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:27.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:55:27.372 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T13:55:27.373 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:55:27.434 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:55:27.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:27 smithi187 ceph-mon[33032]: pgmap v316: 148 pgs: 148 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 894 B/s rd, 62 MiB/s wr, 993 op/s 2024-09-21T13:55:27.471 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:27.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:27.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:55:27.573 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:27.597 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:27.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:27.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:27.644 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:55:27.699 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:55:27.743 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:27.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:27.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:55:27.845 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:27.877 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:27.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:27.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:27.979 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:55:28.018 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:28.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:28.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:55:28.119 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:55:28.152 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:28.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:28.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:55:28.187 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T13:55:28.255 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:28.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:28.289 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:28.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:28.299 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 70 16 108094 108078 96.4789 74.4375 0.00741239 0.0103602 2024-09-21T13:55:28.390 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:28.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:28.427 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:28.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:28.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:28 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:55:28.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:28 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:55:28.494 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:28 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:55:28.494 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:28 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:55:28.495 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:28 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:55:28.495 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:28 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:55:28.528 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:55:28.570 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:28.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:28.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:55:28.596 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:55:28.671 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:55:28.701 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:28.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:28.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:55:28.803 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:55:28.846 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:28.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:28.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:55:28.948 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:55:28.975 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T13:55:28.988 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:28.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:28.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:55:29.089 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:55:29.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:29.146 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:29.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:55:29.208 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T13:55:29.208 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:55:29.247 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:55:29.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:29.286 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:29.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:55:29.299 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 71 16 109183 109167 96.0786 68.0625 0.00522183 0.0104026 2024-09-21T13:55:29.388 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:29.421 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:29.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:29.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:29.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:29 smithi187 ceph-mon[33032]: pgmap v317: 148 pgs: 148 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 763 B/s rd, 53 MiB/s wr, 847 op/s 2024-09-21T13:55:29.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:29 smithi187 ceph-mon[33032]: osdmap e209: 8 total, 8 up, 8 in 2024-09-21T13:55:29.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:29 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:55:29.445 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:55:29 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:55:29.453 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:55:29.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:29 smithi049 ceph-mon[40489]: pgmap v317: 148 pgs: 148 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 763 B/s rd, 53 MiB/s wr, 847 op/s 2024-09-21T13:55:29.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:29 smithi049 ceph-mon[40489]: osdmap e209: 8 total, 8 up, 8 in 2024-09-21T13:55:29.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:29 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:55:29.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:29 smithi049 ceph-mon[30080]: pgmap v317: 148 pgs: 148 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 763 B/s rd, 53 MiB/s wr, 847 op/s 2024-09-21T13:55:29.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:29 smithi049 ceph-mon[30080]: osdmap e209: 8 total, 8 up, 8 in 2024-09-21T13:55:29.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:29 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:55:29.523 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:55:29.573 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:29.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:29.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:55:29.674 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:55:29.717 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:29.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:29.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:55:29.819 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:55:29.862 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:29.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:29.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:55:29.955 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-09-21T13:55:29.964 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:55:29.991 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:29.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:29.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:55:30.089 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T13:55:30.093 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:55:30.133 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:55:30.134 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:55:30.150 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 2] 2024-09-21T13:55:30.150 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T13:55:30.150 INFO:tasks.thrashosds.thrasher:Removing osd 6, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T13:55:30.151 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd out 6 2024-09-21T13:55:30.169 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:55:30.169 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:55:30.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:30.173 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:30.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:55:30.201 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:30 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3891704501' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:55:30.201 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:30 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:55:30.201 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:30 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:55:30.201 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:30 smithi049 ceph-mon[30080]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:55:30.201 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:30 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T13:55:30.201 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:30 smithi049 ceph-mon[30080]: osdmap e210: 8 total, 8 up, 8 in 2024-09-21T13:55:30.201 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:30 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1132583202' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:55:30.201 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:30 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3891704501' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:55:30.201 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:30 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:55:30.201 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:30 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:55:30.202 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:30 smithi049 ceph-mon[40489]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:55:30.202 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:30 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T13:55:30.202 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:30 smithi049 ceph-mon[40489]: osdmap e210: 8 total, 8 up, 8 in 2024-09-21T13:55:30.202 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:30 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1132583202' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:55:30.207 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:55:30.207 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:55:30.245 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:55:30.245 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:55:30.274 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:55:30.281 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:55:30.282 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:55:30.299 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 72 16 110317 110301 95.7285 70.875 0.0166711 0.0104412 2024-09-21T13:55:30.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:30.310 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:30.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:55:30.327 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:55:30.327 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:55:30.370 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:55:30.370 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:55:30.402 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:55:30.402 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:55:30.411 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:55:30.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:30 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3891704501' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:55:30.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:30 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:55:30.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:30 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:55:30.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:30 smithi187 ceph-mon[33032]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:55:30.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:30 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T13:55:30.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:30 smithi187 ceph-mon[33032]: osdmap e210: 8 total, 8 up, 8 in 2024-09-21T13:55:30.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:30 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1132583202' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:55:30.452 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:30.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:30.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:55:30.475 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:55:30.554 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:30.597 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:30.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:30.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:30.699 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:30.735 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:30.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:30.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:30.755 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:55:30 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:55:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:55:30.837 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:55:30.880 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:30.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:30.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:55:30.982 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:55:31.028 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:31.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:31.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:55:31.130 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:31.170 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:31.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:31.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:31.224 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T13:55:31.224 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-21T13:55:31.225 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-21T13:55:31.225 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-21T13:55:31.225 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T13:55:31.272 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:55:31.299 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 73 16 111481 111465 95.4137 72.75 0.0127505 0.010476 2024-09-21T13:55:31.310 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:31.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:31.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:55:31.412 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:31.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:31 smithi187 ceph-mon[33032]: pgmap v319: 148 pgs: 148 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 81 MiB/s wr, 1.29k op/s 2024-09-21T13:55:31.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:31 smithi187 ceph-mon[33032]: 4.12 scrub starts 2024-09-21T13:55:31.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:31 smithi187 ceph-mon[33032]: 4.12 scrub ok 2024-09-21T13:55:31.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:31.446 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:31.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:31.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:31 smithi049 ceph-mon[40489]: pgmap v319: 148 pgs: 148 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 81 MiB/s wr, 1.29k op/s 2024-09-21T13:55:31.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:31 smithi049 ceph-mon[40489]: 4.12 scrub starts 2024-09-21T13:55:31.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:31 smithi049 ceph-mon[40489]: 4.12 scrub ok 2024-09-21T13:55:31.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:31 smithi049 ceph-mon[30080]: pgmap v319: 148 pgs: 148 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 81 MiB/s wr, 1.29k op/s 2024-09-21T13:55:31.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:31 smithi049 ceph-mon[30080]: 4.12 scrub starts 2024-09-21T13:55:31.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:31 smithi049 ceph-mon[30080]: 4.12 scrub ok 2024-09-21T13:55:31.546 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:55:31.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:31.584 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:31.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:55:31.685 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:55:31.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:31.740 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:31.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:55:31.787 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:55:31.841 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:55:31.874 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:31.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:31.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:55:31.948 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T13:55:31.975 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:55:32.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:32.023 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:32.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:55:32.124 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:32.189 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:32.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:32.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:32.291 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:55:32.299 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 74 16 112644 112628 95.1065 72.6875 0.0139069 0.0105096 2024-09-21T13:55:32.331 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:32.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:32.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:55:32.367 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:55:32.433 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:32.446 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:32 smithi187 ceph-mon[33032]: 6.a scrub starts 2024-09-21T13:55:32.446 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:32 smithi187 ceph-mon[33032]: 6.a scrub ok 2024-09-21T13:55:32.446 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:32 smithi187 ceph-mon[33032]: 4.9 scrub starts 2024-09-21T13:55:32.446 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:32 smithi187 ceph-mon[33032]: 4.9 scrub ok 2024-09-21T13:55:32.446 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:32 smithi187 ceph-mon[33032]: 5.1d scrub starts 2024-09-21T13:55:32.446 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:32 smithi187 ceph-mon[33032]: 5.1d scrub ok 2024-09-21T13:55:32.466 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:32.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:32.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:32.479 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:32 smithi049 ceph-mon[40489]: 6.a scrub starts 2024-09-21T13:55:32.479 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:32 smithi049 ceph-mon[40489]: 6.a scrub ok 2024-09-21T13:55:32.480 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:32 smithi049 ceph-mon[40489]: 4.9 scrub starts 2024-09-21T13:55:32.480 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:32 smithi049 ceph-mon[40489]: 4.9 scrub ok 2024-09-21T13:55:32.480 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:32 smithi049 ceph-mon[40489]: 5.1d scrub starts 2024-09-21T13:55:32.480 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:32 smithi049 ceph-mon[40489]: 5.1d scrub ok 2024-09-21T13:55:32.481 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:32 smithi049 ceph-mon[30080]: 6.a scrub starts 2024-09-21T13:55:32.481 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:32 smithi049 ceph-mon[30080]: 6.a scrub ok 2024-09-21T13:55:32.481 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:32 smithi049 ceph-mon[30080]: 4.9 scrub starts 2024-09-21T13:55:32.481 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:32 smithi049 ceph-mon[30080]: 4.9 scrub ok 2024-09-21T13:55:32.481 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:32 smithi049 ceph-mon[30080]: 5.1d scrub starts 2024-09-21T13:55:32.481 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:32 smithi049 ceph-mon[30080]: 5.1d scrub ok 2024-09-21T13:55:32.568 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:32.605 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:32.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:32.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:32.707 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:55:32.735 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:32.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:32.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:55:32.837 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:55:32.869 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:32.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:32.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:55:32.954 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T13:55:32.970 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:55:33.054 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:33.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:33.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:55:33.156 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:55:33.201 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:33.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:33.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:55:33.208 INFO:teuthology.orchestra.run.smithi049.stderr:marked out osd.6. 2024-09-21T13:55:33.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:33 smithi049 ceph-mon[40489]: pgmap v321: 148 pgs: 148 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 746 B/s rd, 54 MiB/s wr, 859 op/s 2024-09-21T13:55:33.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:33 smithi049 ceph-mon[40489]: 8.0 scrub starts 2024-09-21T13:55:33.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:33 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1608865991' entity='client.admin' cmd={"prefix": "osd out", "ids": ["6"]} : dispatch 2024-09-21T13:55:33.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:33 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["6"]} : dispatch 2024-09-21T13:55:33.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:33 smithi049 ceph-mon[40489]: Client client.admin marked osd.6 out, while it was still marked up 2024-09-21T13:55:33.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:33 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mon-a[30057]: 2024-09-21T13:55:33.047+0000 7fc8b704c640 -1 mon.a@0(leader).osd e210 definitely_dead 0 2024-09-21T13:55:33.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:33 smithi049 ceph-mon[30080]: pgmap v321: 148 pgs: 148 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 746 B/s rd, 54 MiB/s wr, 859 op/s 2024-09-21T13:55:33.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:33 smithi049 ceph-mon[30080]: 8.0 scrub starts 2024-09-21T13:55:33.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:33 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1608865991' entity='client.admin' cmd={"prefix": "osd out", "ids": ["6"]} : dispatch 2024-09-21T13:55:33.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:33 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["6"]} : dispatch 2024-09-21T13:55:33.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:33 smithi049 ceph-mon[30080]: Client client.admin marked osd.6 out, while it was still marked up 2024-09-21T13:55:33.299 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 75 16 113626 113610 94.6567 61.375 0.00355116 0.0105534 2024-09-21T13:55:33.305 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:55:33.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:33.328 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:33.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:55:33.429 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:55:33.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:33 smithi187 ceph-mon[33032]: pgmap v321: 148 pgs: 148 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 746 B/s rd, 54 MiB/s wr, 859 op/s 2024-09-21T13:55:33.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:33 smithi187 ceph-mon[33032]: 8.0 scrub starts 2024-09-21T13:55:33.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:33 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1608865991' entity='client.admin' cmd={"prefix": "osd out", "ids": ["6"]} : dispatch 2024-09-21T13:55:33.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:33 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["6"]} : dispatch 2024-09-21T13:55:33.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:33 smithi187 ceph-mon[33032]: Client client.admin marked osd.6 out, while it was still marked up 2024-09-21T13:55:33.451 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:33.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:33.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:55:33.553 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:55:33.579 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:33.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:33.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:55:33.680 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:55:33.705 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:33.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:33.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:55:33.707 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T13:55:33.807 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:55:33.828 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:33.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:33.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:55:33.929 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:55:33.951 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:33.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:33.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:55:33.989 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:55:34.054 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:34.076 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:34.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:34.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:34.095 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T13:55:34.095 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-21T13:55:34.095 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-21T13:55:34.095 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-21T13:55:34.095 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.095 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:35241 7.4 7:2f984d9b:::benchmark_data_smithi049_76798_object35240:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T13:55:34.095 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:28.687118+0000", 2024-09-21T13:55:34.095 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 304.53545713099999, 2024-09-21T13:55:34.096 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.011460528, 2024-09-21T13:55:34.096 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:55:34.096 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:55:34.096 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:55:34.096 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:55:34.096 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:55:34.096 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:55:34.096 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 35241 2024-09-21T13:55:34.096 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.096 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:55:34.096 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.097 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:55:34.097 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.687118+0000", 2024-09-21T13:55:34.097 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:34.097 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.097 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.097 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:55:34.097 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.687118+0000", 2024-09-21T13:55:34.097 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:34.097 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.097 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.097 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:55:34.098 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.687120+0000", 2024-09-21T13:55:34.098 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.573e-06 2024-09-21T13:55:34.098 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.098 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.098 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:55:34.098 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.687176+0000", 2024-09-21T13:55:34.098 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6502000000000002e-05 2024-09-21T13:55:34.098 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.098 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.098 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:55:34.098 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.687189+0000", 2024-09-21T13:55:34.099 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2747e-05 2024-09-21T13:55:34.099 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.099 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.099 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:55:34.099 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.687196+0000", 2024-09-21T13:55:34.099 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6050000000000003e-06 2024-09-21T13:55:34.099 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.099 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.099 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:34.099 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.687248+0000", 2024-09-21T13:55:34.099 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2234000000000002e-05 2024-09-21T13:55:34.100 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.100 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.100 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:55:34.100 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.687256+0000", 2024-09-21T13:55:34.100 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3299999999999999e-06 2024-09-21T13:55:34.100 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.100 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.100 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:34.100 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.691286+0000", 2024-09-21T13:55:34.100 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.004029828 2024-09-21T13:55:34.100 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.100 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.101 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:55:34.101 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.692244+0000", 2024-09-21T13:55:34.101 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00095738099999999999 2024-09-21T13:55:34.101 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.101 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.101 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-21T13:55:34.101 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.692408+0000", 2024-09-21T13:55:34.101 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000164041 2024-09-21T13:55:34.101 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.101 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.101 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:34.101 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.698261+0000", 2024-09-21T13:55:34.102 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.005852983 2024-09-21T13:55:34.102 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.102 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.102 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:55:34.102 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.698274+0000", 2024-09-21T13:55:34.102 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3485e-05 2024-09-21T13:55:34.102 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.102 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.102 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:34.102 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.698481+0000", 2024-09-21T13:55:34.103 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00020698099999999999 2024-09-21T13:55:34.103 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.103 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.103 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:55:34.103 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.698537+0000", 2024-09-21T13:55:34.103 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6082999999999998e-05 2024-09-21T13:55:34.103 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.103 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.103 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:55:34.103 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.698579+0000", 2024-09-21T13:55:34.103 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1755000000000003e-05 2024-09-21T13:55:34.103 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:34.104 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:55:34.104 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:34.104 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.104 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.104 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:35262 7.4 7:2f95173f:::benchmark_data_smithi049_76798_object35261:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T13:55:34.104 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:28.705575+0000", 2024-09-21T13:55:34.104 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 304.51700044900002, 2024-09-21T13:55:34.104 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.99486020900000005, 2024-09-21T13:55:34.104 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:55:34.104 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:55:34.104 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:55:34.105 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:55:34.105 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:55:34.105 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:55:34.105 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 35262 2024-09-21T13:55:34.105 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.105 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:55:34.105 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.105 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:55:34.105 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.705575+0000", 2024-09-21T13:55:34.105 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:34.105 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.105 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.105 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:55:34.106 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.705575+0000", 2024-09-21T13:55:34.106 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:34.106 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.106 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.106 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:55:34.106 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.705577+0000", 2024-09-21T13:55:34.106 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.485e-06 2024-09-21T13:55:34.106 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.106 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.106 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:55:34.106 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.705634+0000", 2024-09-21T13:55:34.107 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7954000000000001e-05 2024-09-21T13:55:34.107 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.107 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.107 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:55:34.107 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.705649+0000", 2024-09-21T13:55:34.107 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4209999999999999e-05 2024-09-21T13:55:34.107 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.107 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.107 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:55:34.107 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.705657+0000", 2024-09-21T13:55:34.107 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7910000000000005e-06 2024-09-21T13:55:34.107 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.108 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.108 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:34.108 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.705703+0000", 2024-09-21T13:55:34.108 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6712999999999998e-05 2024-09-21T13:55:34.108 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.108 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.108 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:55:34.108 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.705711+0000", 2024-09-21T13:55:34.108 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2050000000000002e-06 2024-09-21T13:55:34.108 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.108 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.108 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:34.109 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.694482+0000", 2024-09-21T13:55:34.109 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.98877102800000005 2024-09-21T13:55:34.109 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.109 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.109 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:55:34.109 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.694624+0000", 2024-09-21T13:55:34.109 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014155400000000001 2024-09-21T13:55:34.109 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.109 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.109 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-21T13:55:34.109 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.694770+0000", 2024-09-21T13:55:34.109 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014635399999999999 2024-09-21T13:55:34.110 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.110 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.110 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:55:34.110 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.698351+0000", 2024-09-21T13:55:34.110 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0035804869999999998 2024-09-21T13:55:34.110 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.110 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.110 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:34.110 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.698411+0000", 2024-09-21T13:55:34.110 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0132e-05 2024-09-21T13:55:34.110 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.110 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.111 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:34.111 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.700312+0000", 2024-09-21T13:55:34.111 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001901109 2024-09-21T13:55:34.111 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.111 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.111 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:55:34.111 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.700374+0000", 2024-09-21T13:55:34.111 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2137999999999994e-05 2024-09-21T13:55:34.111 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.111 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.111 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:55:34.112 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.700435+0000", 2024-09-21T13:55:34.112 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1049000000000005e-05 2024-09-21T13:55:34.112 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:34.112 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:55:34.112 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:34.112 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.112 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.112 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:35282 7.4 7:26bb5871:::benchmark_data_smithi049_76798_object35281:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T13:55:34.112 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:28.721297+0000", 2024-09-21T13:55:34.112 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 304.50127879399997, 2024-09-21T13:55:34.112 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.98226591299999999, 2024-09-21T13:55:34.112 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:55:34.113 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:55:34.113 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:55:34.113 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:55:34.113 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:55:34.113 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:55:34.113 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 35282 2024-09-21T13:55:34.113 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.113 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:55:34.113 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.113 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:55:34.113 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.721297+0000", 2024-09-21T13:55:34.113 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:34.114 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.114 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.114 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:55:34.114 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.721297+0000", 2024-09-21T13:55:34.114 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:34.114 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.114 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.114 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:55:34.114 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.721298+0000", 2024-09-21T13:55:34.114 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0920000000000001e-06 2024-09-21T13:55:34.114 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.114 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.115 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:55:34.115 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.721380+0000", 2024-09-21T13:55:34.115 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2422000000000005e-05 2024-09-21T13:55:34.115 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.115 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.115 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:55:34.115 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.721389+0000", 2024-09-21T13:55:34.115 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1789999999999997e-06 2024-09-21T13:55:34.115 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.115 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.115 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:55:34.115 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.721394+0000", 2024-09-21T13:55:34.116 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9080000000000003e-06 2024-09-21T13:55:34.116 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.116 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.116 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:34.116 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.721436+0000", 2024-09-21T13:55:34.116 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1887000000000001e-05 2024-09-21T13:55:34.116 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.116 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.116 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:55:34.116 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.721443+0000", 2024-09-21T13:55:34.116 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.725e-06 2024-09-21T13:55:34.116 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.116 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.116 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:34.117 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.696771+0000", 2024-09-21T13:55:34.117 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.97532851799999998 2024-09-21T13:55:34.117 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.117 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.117 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:55:34.117 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.696923+0000", 2024-09-21T13:55:34.117 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015162800000000001 2024-09-21T13:55:34.117 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.117 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.117 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-21T13:55:34.117 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.697094+0000", 2024-09-21T13:55:34.117 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000170884 2024-09-21T13:55:34.117 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.118 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.118 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:34.118 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.700495+0000", 2024-09-21T13:55:34.118 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0034005519999999998 2024-09-21T13:55:34.118 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.118 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.118 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:55:34.118 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.701924+0000", 2024-09-21T13:55:34.118 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001429303 2024-09-21T13:55:34.118 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.118 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.119 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:34.119 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.703461+0000", 2024-09-21T13:55:34.119 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001537395 2024-09-21T13:55:34.119 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.119 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.119 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:55:34.119 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.703519+0000", 2024-09-21T13:55:34.119 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7435999999999997e-05 2024-09-21T13:55:34.119 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.119 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.119 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:55:34.119 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.703563+0000", 2024-09-21T13:55:34.119 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3983999999999998e-05 2024-09-21T13:55:34.119 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:34.120 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:55:34.120 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:34.120 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.120 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.120 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:36415 7.4 7:239fcd45:::benchmark_data_smithi049_76798_object36414:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-21T13:55:34.120 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:30.284471+0000", 2024-09-21T13:55:34.120 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 302.93810422000001, 2024-09-21T13:55:34.120 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.089453959999999999, 2024-09-21T13:55:34.120 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:55:34.120 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:55:34.120 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:55:34.120 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:55:34.120 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:55:34.121 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:55:34.121 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 36415 2024-09-21T13:55:34.121 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.121 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:55:34.121 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.121 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:55:34.121 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.284471+0000", 2024-09-21T13:55:34.121 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:34.121 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.121 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.121 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:55:34.121 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.284471+0000", 2024-09-21T13:55:34.122 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:34.122 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.122 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.122 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:55:34.122 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.284472+0000", 2024-09-21T13:55:34.122 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7299999999999995e-07 2024-09-21T13:55:34.122 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.122 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.122 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:55:34.122 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.284499+0000", 2024-09-21T13:55:34.122 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7344000000000001e-05 2024-09-21T13:55:34.122 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.123 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.123 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:55:34.123 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.284506+0000", 2024-09-21T13:55:34.123 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5799999999999997e-06 2024-09-21T13:55:34.123 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.123 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.123 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:55:34.123 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.284511+0000", 2024-09-21T13:55:34.123 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1410000000000004e-06 2024-09-21T13:55:34.123 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.123 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.123 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:34.124 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.285801+0000", 2024-09-21T13:55:34.124 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001290023 2024-09-21T13:55:34.124 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.124 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.124 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:55:34.124 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.285896+0000", 2024-09-21T13:55:34.124 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.5357000000000002e-05 2024-09-21T13:55:34.124 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.124 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.124 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-21T13:55:34.124 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.285982+0000", 2024-09-21T13:55:34.124 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.53e-05 2024-09-21T13:55:34.124 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.125 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.125 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:55:34.125 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.286879+0000", 2024-09-21T13:55:34.125 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00089719500000000005 2024-09-21T13:55:34.125 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.125 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.125 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:34.125 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.368934+0000", 2024-09-21T13:55:34.125 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.082055115999999997 2024-09-21T13:55:34.125 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.125 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.125 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:34.125 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.373694+0000", 2024-09-21T13:55:34.126 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0047602670000000003 2024-09-21T13:55:34.126 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.126 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.126 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:55:34.126 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.373777+0000", 2024-09-21T13:55:34.126 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2953000000000004e-05 2024-09-21T13:55:34.126 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.126 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.126 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:55:34.126 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.373925+0000", 2024-09-21T13:55:34.126 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000148011 2024-09-21T13:55:34.126 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:34.127 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:55:34.127 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:34.127 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.127 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.127 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:36866 7.1 7:81a03870:::benchmark_data_smithi049_76798_object36865:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-21T13:55:34.127 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:30.635343+0000", 2024-09-21T13:55:34.127 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 302.58723279499998, 2024-09-21T13:55:34.127 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13232190499999999, 2024-09-21T13:55:34.127 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:55:34.127 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:55:34.127 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:55:34.127 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:55:34.128 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:55:34.128 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:55:34.128 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 36866 2024-09-21T13:55:34.128 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.128 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:55:34.128 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.128 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:55:34.128 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.635343+0000", 2024-09-21T13:55:34.128 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:34.128 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.128 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.128 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:55:34.128 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.635343+0000", 2024-09-21T13:55:34.129 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:34.129 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.129 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.129 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:55:34.129 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.635344+0000", 2024-09-21T13:55:34.129 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6920000000000001e-06 2024-09-21T13:55:34.129 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.129 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.129 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:55:34.129 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.635406+0000", 2024-09-21T13:55:34.129 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1353999999999996e-05 2024-09-21T13:55:34.129 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.130 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.130 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:55:34.130 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.635418+0000", 2024-09-21T13:55:34.130 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2082000000000001e-05 2024-09-21T13:55:34.130 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.130 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.130 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:55:34.130 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.635425+0000", 2024-09-21T13:55:34.130 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1509999999999998e-06 2024-09-21T13:55:34.130 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.130 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.130 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:34.130 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.635486+0000", 2024-09-21T13:55:34.131 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0523999999999999e-05 2024-09-21T13:55:34.131 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.131 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.131 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:55:34.131 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.635599+0000", 2024-09-21T13:55:34.131 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000113731 2024-09-21T13:55:34.131 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.131 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.131 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,6", 2024-09-21T13:55:34.131 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.635724+0000", 2024-09-21T13:55:34.131 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000124292 2024-09-21T13:55:34.131 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.132 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.132 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:55:34.132 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.636808+0000", 2024-09-21T13:55:34.132 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001084631 2024-09-21T13:55:34.132 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.132 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.132 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:34.132 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.638114+0000", 2024-09-21T13:55:34.132 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0013061679999999999 2024-09-21T13:55:34.132 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.132 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.132 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:34.132 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.767545+0000", 2024-09-21T13:55:34.133 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.12943043400000001 2024-09-21T13:55:34.133 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.133 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.133 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:55:34.133 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.767607+0000", 2024-09-21T13:55:34.133 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2065999999999998e-05 2024-09-21T13:55:34.133 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.133 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.133 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:55:34.133 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.767665+0000", 2024-09-21T13:55:34.133 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7779999999999999e-05 2024-09-21T13:55:34.133 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:34.133 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:55:34.134 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:34.134 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.134 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.134 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:36886 7.1 7:8f0780fe:::benchmark_data_smithi049_76798_object36885:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-21T13:55:34.134 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:30.650584+0000", 2024-09-21T13:55:34.134 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 302.57199121600001, 2024-09-21T13:55:34.134 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.11732397999999999, 2024-09-21T13:55:34.134 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:55:34.134 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:55:34.134 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:55:34.134 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:55:34.134 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:55:34.135 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:55:34.135 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 36886 2024-09-21T13:55:34.135 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.135 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:55:34.135 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.135 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:55:34.135 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.650584+0000", 2024-09-21T13:55:34.135 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:34.135 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.135 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.135 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:55:34.135 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.650584+0000", 2024-09-21T13:55:34.136 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:34.136 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.136 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.136 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:55:34.136 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.650587+0000", 2024-09-21T13:55:34.136 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5600000000000001e-06 2024-09-21T13:55:34.136 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.136 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.136 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:55:34.136 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.650631+0000", 2024-09-21T13:55:34.136 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4499999999999997e-05 2024-09-21T13:55:34.137 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.137 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.137 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:55:34.137 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.650643+0000", 2024-09-21T13:55:34.137 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1404e-05 2024-09-21T13:55:34.137 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.137 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.137 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:55:34.137 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.650648+0000", 2024-09-21T13:55:34.137 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1800000000000004e-06 2024-09-21T13:55:34.137 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.137 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.137 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:34.138 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.650689+0000", 2024-09-21T13:55:34.138 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0784000000000001e-05 2024-09-21T13:55:34.138 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.138 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.138 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:55:34.138 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.650777+0000", 2024-09-21T13:55:34.138 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8154999999999999e-05 2024-09-21T13:55:34.138 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.138 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.138 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,6", 2024-09-21T13:55:34.138 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.650871+0000", 2024-09-21T13:55:34.138 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3738000000000003e-05 2024-09-21T13:55:34.139 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.139 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.139 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:55:34.139 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.651931+0000", 2024-09-21T13:55:34.139 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0010604270000000001 2024-09-21T13:55:34.139 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.139 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.139 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:34.139 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.652228+0000", 2024-09-21T13:55:34.139 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000296507 2024-09-21T13:55:34.139 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.139 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.139 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:34.140 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.767813+0000", 2024-09-21T13:55:34.140 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.115584956 2024-09-21T13:55:34.140 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.140 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.140 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:55:34.140 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.767869+0000", 2024-09-21T13:55:34.140 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6851999999999997e-05 2024-09-21T13:55:34.140 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.140 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.140 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:55:34.140 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.767908+0000", 2024-09-21T13:55:34.140 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8917e-05 2024-09-21T13:55:34.141 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:34.141 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:55:34.141 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:34.141 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.141 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.141 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:72620 7.4 7:21febffb:::benchmark_data_smithi049_76798_object72619:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T13:55:34.141 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:54.472955+0000", 2024-09-21T13:55:34.141 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 278.74962035900001, 2024-09-21T13:55:34.141 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.092558697999999995, 2024-09-21T13:55:34.141 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:55:34.141 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:55:34.141 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:55:34.141 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:55:34.142 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:55:34.142 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:55:34.142 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 72620 2024-09-21T13:55:34.142 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.142 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:55:34.142 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.142 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:55:34.142 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.472955+0000", 2024-09-21T13:55:34.142 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:34.142 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.142 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.142 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:55:34.142 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.472955+0000", 2024-09-21T13:55:34.143 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:34.143 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.143 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.143 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:55:34.143 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.472956+0000", 2024-09-21T13:55:34.143 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2500000000000001e-06 2024-09-21T13:55:34.143 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.143 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.143 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:55:34.143 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.473008+0000", 2024-09-21T13:55:34.143 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1379999999999999e-05 2024-09-21T13:55:34.143 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.143 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.144 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:55:34.144 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.473020+0000", 2024-09-21T13:55:34.144 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2687999999999999e-05 2024-09-21T13:55:34.144 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.144 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.144 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:55:34.144 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.473028+0000", 2024-09-21T13:55:34.144 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6850000000000001e-06 2024-09-21T13:55:34.144 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.144 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.144 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:34.144 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.473103+0000", 2024-09-21T13:55:34.145 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4821999999999996e-05 2024-09-21T13:55:34.145 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.145 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.145 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:55:34.145 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.473221+0000", 2024-09-21T13:55:34.145 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011758 2024-09-21T13:55:34.145 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.145 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.145 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-21T13:55:34.145 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.473351+0000", 2024-09-21T13:55:34.145 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000130359 2024-09-21T13:55:34.145 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.145 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.146 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:55:34.146 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.476613+0000", 2024-09-21T13:55:34.146 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0032616120000000001 2024-09-21T13:55:34.146 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.146 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.146 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:34.146 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.479113+0000", 2024-09-21T13:55:34.146 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0025009720000000002 2024-09-21T13:55:34.146 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.146 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.146 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:34.147 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.565355+0000", 2024-09-21T13:55:34.147 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.086241622000000004 2024-09-21T13:55:34.147 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.147 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.147 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:55:34.147 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.565419+0000", 2024-09-21T13:55:34.147 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4358999999999998e-05 2024-09-21T13:55:34.147 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.147 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.147 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:55:34.147 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:54.565514+0000", 2024-09-21T13:55:34.147 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.4369000000000004e-05 2024-09-21T13:55:34.147 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:34.148 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:55:34.148 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:34.148 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.148 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.148 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:172089 7.4 7:236b8a6b:::benchmark_data_smithi049_76798_object35669:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-09-21T13:55:34.148 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:13.086882+0000", 2024-09-21T13:55:34.148 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 200.135693132, 2024-09-21T13:55:34.148 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.096754271000000003, 2024-09-21T13:55:34.148 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:55:34.148 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:55:34.148 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:55:34.148 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:55:34.149 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:55:34.149 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:55:34.149 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 172089 2024-09-21T13:55:34.149 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.149 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:55:34.149 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.149 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:55:34.149 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:13.086882+0000", 2024-09-21T13:55:34.149 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:34.149 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.149 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.150 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:55:34.150 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:13.086882+0000", 2024-09-21T13:55:34.150 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:34.150 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.150 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.150 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:55:34.150 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:13.086884+0000", 2024-09-21T13:55:34.150 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4300000000000001e-06 2024-09-21T13:55:34.150 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.150 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.150 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:55:34.151 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:13.086889+0000", 2024-09-21T13:55:34.151 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2669999999999997e-06 2024-09-21T13:55:34.151 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.151 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.151 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:55:34.151 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:13.086901+0000", 2024-09-21T13:55:34.151 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1793999999999999e-05 2024-09-21T13:55:34.151 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.151 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.151 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:55:34.151 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:13.086908+0000", 2024-09-21T13:55:34.151 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1799999999999999e-06 2024-09-21T13:55:34.152 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.152 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.152 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:34.152 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:13.086960+0000", 2024-09-21T13:55:34.152 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2367999999999997e-05 2024-09-21T13:55:34.152 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.152 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.152 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:55:34.152 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:13.087072+0000", 2024-09-21T13:55:34.152 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000111779 2024-09-21T13:55:34.152 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.152 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.153 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-21T13:55:34.153 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:13.087181+0000", 2024-09-21T13:55:34.153 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000109184 2024-09-21T13:55:34.153 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.153 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.153 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:34.153 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:13.089557+0000", 2024-09-21T13:55:34.153 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0023758220000000001 2024-09-21T13:55:34.153 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.153 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.153 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:55:34.153 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:13.089639+0000", 2024-09-21T13:55:34.154 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1357e-05 2024-09-21T13:55:34.154 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.154 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.154 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:34.154 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:13.183539+0000", 2024-09-21T13:55:34.154 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.093900288999999998 2024-09-21T13:55:34.154 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.154 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.154 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:55:34.154 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:13.183601+0000", 2024-09-21T13:55:34.154 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2244999999999998e-05 2024-09-21T13:55:34.154 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.155 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.155 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:55:34.155 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:13.183637+0000", 2024-09-21T13:55:34.155 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5556000000000003e-05 2024-09-21T13:55:34.155 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:34.155 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:55:34.155 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:34.155 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.155 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.155 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:185401 7.1 7:89febdc9:::benchmark_data_smithi049_76798_object48981:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-09-21T13:55:34.155 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:26.179574+0000", 2024-09-21T13:55:34.155 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 187.04300195499999, 2024-09-21T13:55:34.155 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.105705047, 2024-09-21T13:55:34.156 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:55:34.156 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:55:34.156 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:55:34.156 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:55:34.156 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:55:34.156 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:55:34.156 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 185401 2024-09-21T13:55:34.156 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.156 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:55:34.156 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.156 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:55:34.156 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:26.179574+0000", 2024-09-21T13:55:34.156 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:34.156 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.157 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.157 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:55:34.157 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:26.179574+0000", 2024-09-21T13:55:34.157 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:34.157 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.157 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.157 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:55:34.157 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:26.179575+0000", 2024-09-21T13:55:34.157 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3850000000000001e-06 2024-09-21T13:55:34.157 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.157 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.157 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:55:34.157 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:26.179579+0000", 2024-09-21T13:55:34.158 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.968e-06 2024-09-21T13:55:34.158 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.158 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.158 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:55:34.158 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:26.179588+0000", 2024-09-21T13:55:34.158 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8240000000000006e-06 2024-09-21T13:55:34.158 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.158 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.158 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:55:34.158 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:26.179594+0000", 2024-09-21T13:55:34.158 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1210000000000003e-06 2024-09-21T13:55:34.158 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.159 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.159 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:34.159 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:26.179654+0000", 2024-09-21T13:55:34.159 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0315000000000001e-05 2024-09-21T13:55:34.159 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.159 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.159 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:55:34.159 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:26.179772+0000", 2024-09-21T13:55:34.159 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000118232 2024-09-21T13:55:34.159 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.159 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.159 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,6", 2024-09-21T13:55:34.159 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:26.179956+0000", 2024-09-21T13:55:34.159 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00018340500000000001 2024-09-21T13:55:34.160 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.160 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.160 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:55:34.160 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:26.183309+0000", 2024-09-21T13:55:34.160 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003352743 2024-09-21T13:55:34.160 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.160 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.160 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:34.160 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:26.184074+0000", 2024-09-21T13:55:34.160 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00076550499999999998 2024-09-21T13:55:34.160 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.160 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.161 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:34.161 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:26.285117+0000", 2024-09-21T13:55:34.161 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.101043312 2024-09-21T13:55:34.161 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.161 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.161 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:55:34.161 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:26.285235+0000", 2024-09-21T13:55:34.161 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000118023 2024-09-21T13:55:34.161 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.161 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.161 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:55:34.161 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:26.285279+0000", 2024-09-21T13:55:34.161 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3214000000000003e-05 2024-09-21T13:55:34.162 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:34.162 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:55:34.162 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:34.162 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.162 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.162 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:200754 7.8 7:13380118:::benchmark_data_smithi049_76798_object64334:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-09-21T13:55:34.162 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:41.210906+0000", 2024-09-21T13:55:34.162 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 172.01166907000001, 2024-09-21T13:55:34.162 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.089889414000000001, 2024-09-21T13:55:34.162 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:55:34.162 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:55:34.163 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:55:34.163 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:55:34.163 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:55:34.163 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:55:34.163 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 200754 2024-09-21T13:55:34.163 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.163 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:55:34.163 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.163 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:55:34.163 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.210906+0000", 2024-09-21T13:55:34.163 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:34.163 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.163 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.164 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:55:34.164 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.210906+0000", 2024-09-21T13:55:34.164 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:34.164 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.164 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.164 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:55:34.164 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.210908+0000", 2024-09-21T13:55:34.164 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3489999999999999e-06 2024-09-21T13:55:34.164 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.164 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.164 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:55:34.165 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.210913+0000", 2024-09-21T13:55:34.165 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3210000000000004e-06 2024-09-21T13:55:34.165 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.165 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.165 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:55:34.165 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.210924+0000", 2024-09-21T13:55:34.165 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1344e-05 2024-09-21T13:55:34.165 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.165 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.165 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:55:34.165 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.210931+0000", 2024-09-21T13:55:34.165 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7700000000000004e-06 2024-09-21T13:55:34.165 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.165 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.166 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:34.166 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.210976+0000", 2024-09-21T13:55:34.166 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5164999999999999e-05 2024-09-21T13:55:34.166 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.166 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.166 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:55:34.166 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.211051+0000", 2024-09-21T13:55:34.166 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4579999999999994e-05 2024-09-21T13:55:34.166 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.166 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.166 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,7", 2024-09-21T13:55:34.166 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.211362+0000", 2024-09-21T13:55:34.166 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000311405 2024-09-21T13:55:34.167 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.167 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.167 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:34.167 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.218665+0000", 2024-09-21T13:55:34.167 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0073028520000000003 2024-09-21T13:55:34.167 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.167 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.167 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:55:34.167 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.218783+0000", 2024-09-21T13:55:34.167 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000117673 2024-09-21T13:55:34.167 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.167 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.167 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:34.168 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.300582+0000", 2024-09-21T13:55:34.168 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.081798917999999998 2024-09-21T13:55:34.168 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.168 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.168 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:55:34.168 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.300684+0000", 2024-09-21T13:55:34.168 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000101817 2024-09-21T13:55:34.168 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.168 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.168 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:55:34.168 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.300796+0000", 2024-09-21T13:55:34.168 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011222000000000001 2024-09-21T13:55:34.168 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:34.168 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:55:34.168 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:34.169 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.169 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.169 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:200755 7.8 7:15b7eb40:::benchmark_data_smithi049_76798_object64335:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-09-21T13:55:34.169 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:41.212646+0000", 2024-09-21T13:55:34.169 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 172.00992956499999, 2024-09-21T13:55:34.169 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.088362590000000005, 2024-09-21T13:55:34.169 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:55:34.169 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:55:34.169 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:55:34.169 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:55:34.169 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:55:34.169 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:55:34.169 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 200755 2024-09-21T13:55:34.170 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.170 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:55:34.170 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.170 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:55:34.170 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.212646+0000", 2024-09-21T13:55:34.170 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:34.170 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.170 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.170 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:55:34.170 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.212646+0000", 2024-09-21T13:55:34.170 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:34.170 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.171 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.171 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:55:34.171 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.212647+0000", 2024-09-21T13:55:34.171 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6800000000000009e-07 2024-09-21T13:55:34.171 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.171 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.171 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:55:34.171 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.212651+0000", 2024-09-21T13:55:34.171 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2610000000000004e-06 2024-09-21T13:55:34.171 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.171 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.171 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:55:34.172 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.212662+0000", 2024-09-21T13:55:34.172 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0689e-05 2024-09-21T13:55:34.172 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.172 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.172 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:55:34.172 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.212673+0000", 2024-09-21T13:55:34.172 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0837e-05 2024-09-21T13:55:34.172 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.172 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.172 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:34.172 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.213420+0000", 2024-09-21T13:55:34.172 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000746811 2024-09-21T13:55:34.172 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.173 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.173 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:55:34.173 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.213541+0000", 2024-09-21T13:55:34.173 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012134900000000001 2024-09-21T13:55:34.173 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.173 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.173 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,7", 2024-09-21T13:55:34.173 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.213658+0000", 2024-09-21T13:55:34.173 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000117404 2024-09-21T13:55:34.173 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.173 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.173 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:34.173 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.218768+0000", 2024-09-21T13:55:34.173 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0051096789999999998 2024-09-21T13:55:34.174 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.174 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.174 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:55:34.174 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.218788+0000", 2024-09-21T13:55:34.174 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0205000000000001e-05 2024-09-21T13:55:34.174 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.174 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.174 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:34.174 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.300851+0000", 2024-09-21T13:55:34.174 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.082063285999999999 2024-09-21T13:55:34.174 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.174 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.175 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:55:34.175 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.300909+0000", 2024-09-21T13:55:34.175 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7973999999999998e-05 2024-09-21T13:55:34.175 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.175 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.175 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:55:34.175 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.301009+0000", 2024-09-21T13:55:34.175 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.9127000000000001e-05 2024-09-21T13:55:34.175 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:34.175 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:55:34.175 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:34.175 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.175 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.176 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:200758 7.8 7:14b47203:::benchmark_data_smithi049_76798_object64338:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-09-21T13:55:34.176 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:41.213083+0000", 2024-09-21T13:55:34.176 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 172.009492295, 2024-09-21T13:55:34.176 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.088159201000000006, 2024-09-21T13:55:34.176 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:55:34.176 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:55:34.176 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:55:34.176 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:55:34.176 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:55:34.176 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:55:34.176 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 200758 2024-09-21T13:55:34.176 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.176 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:55:34.176 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.177 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:55:34.177 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.213083+0000", 2024-09-21T13:55:34.177 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:34.177 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.177 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.177 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:55:34.177 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.213083+0000", 2024-09-21T13:55:34.177 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:34.177 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.177 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.177 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:55:34.177 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.213084+0000", 2024-09-21T13:55:34.177 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.0200000000000001e-07 2024-09-21T13:55:34.178 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.178 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.178 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:55:34.178 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.213087+0000", 2024-09-21T13:55:34.178 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.32e-06 2024-09-21T13:55:34.178 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.178 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.178 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:55:34.178 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.213095+0000", 2024-09-21T13:55:34.178 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8320000000000006e-06 2024-09-21T13:55:34.178 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.178 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.178 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:55:34.179 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.213100+0000", 2024-09-21T13:55:34.179 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5539999999999999e-06 2024-09-21T13:55:34.179 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.179 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.179 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:34.179 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.216276+0000", 2024-09-21T13:55:34.179 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0031763360000000001 2024-09-21T13:55:34.179 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.179 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.179 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:55:34.179 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.216393+0000", 2024-09-21T13:55:34.179 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011679 2024-09-21T13:55:34.179 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.180 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.180 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,7", 2024-09-21T13:55:34.180 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.216486+0000", 2024-09-21T13:55:34.180 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3737000000000001e-05 2024-09-21T13:55:34.180 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.180 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.180 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:55:34.180 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.218793+0000", 2024-09-21T13:55:34.180 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0023062880000000001 2024-09-21T13:55:34.180 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.180 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.180 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:34.180 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.219352+0000", 2024-09-21T13:55:34.180 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00055905799999999997 2024-09-21T13:55:34.181 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.181 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.181 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:34.181 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.301059+0000", 2024-09-21T13:55:34.181 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.081707641999999997 2024-09-21T13:55:34.181 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.181 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.181 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:55:34.181 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.301143+0000", 2024-09-21T13:55:34.181 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3610999999999996e-05 2024-09-21T13:55:34.181 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.181 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.181 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:55:34.182 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.301242+0000", 2024-09-21T13:55:34.182 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.9331000000000002e-05 2024-09-21T13:55:34.182 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:34.182 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:55:34.182 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:34.182 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.182 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.182 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:200759 7.4 7:2f331afb:::benchmark_data_smithi049_76798_object64339:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-09-21T13:55:34.182 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:41.213107+0000", 2024-09-21T13:55:34.182 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 172.00946838600001, 2024-09-21T13:55:34.182 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.087580281999999995, 2024-09-21T13:55:34.182 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:55:34.183 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:55:34.183 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:55:34.183 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:55:34.183 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:55:34.183 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:55:34.183 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 200759 2024-09-21T13:55:34.183 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.183 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:55:34.183 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.183 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:55:34.183 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.213107+0000", 2024-09-21T13:55:34.183 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:34.183 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.183 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.184 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:55:34.184 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.213107+0000", 2024-09-21T13:55:34.184 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:34.184 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.184 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.184 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:55:34.184 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.213107+0000", 2024-09-21T13:55:34.184 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8799999999999998e-07 2024-09-21T13:55:34.184 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.184 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.184 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:55:34.184 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.213109+0000", 2024-09-21T13:55:34.184 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.424e-06 2024-09-21T13:55:34.185 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.185 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.185 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:55:34.185 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.213112+0000", 2024-09-21T13:55:34.185 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4300000000000002e-06 2024-09-21T13:55:34.185 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.185 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.185 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:55:34.185 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.213115+0000", 2024-09-21T13:55:34.185 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.6299999999999998e-06 2024-09-21T13:55:34.185 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.186 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.186 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:34.186 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.213154+0000", 2024-09-21T13:55:34.186 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9295999999999998e-05 2024-09-21T13:55:34.186 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.186 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.186 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:55:34.186 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.213253+0000", 2024-09-21T13:55:34.186 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.8808000000000007e-05 2024-09-21T13:55:34.186 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.186 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.186 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-21T13:55:34.186 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.213327+0000", 2024-09-21T13:55:34.187 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3708999999999995e-05 2024-09-21T13:55:34.187 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.187 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.187 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:55:34.187 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.220472+0000", 2024-09-21T13:55:34.187 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0071453590000000004 2024-09-21T13:55:34.187 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.187 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.187 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:34.187 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.220527+0000", 2024-09-21T13:55:34.187 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5408999999999998e-05 2024-09-21T13:55:34.187 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.188 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.188 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:34.188 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.300575+0000", 2024-09-21T13:55:34.188 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.080047786999999995 2024-09-21T13:55:34.188 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.188 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.188 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:55:34.188 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.300644+0000", 2024-09-21T13:55:34.188 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9049999999999998e-05 2024-09-21T13:55:34.188 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.188 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.188 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:55:34.188 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.300687+0000", 2024-09-21T13:55:34.188 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3092e-05 2024-09-21T13:55:34.189 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:34.189 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:55:34.189 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:34.189 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.189 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.189 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:200760 7.4 7:2e2160c6:::benchmark_data_smithi049_76798_object64340:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-09-21T13:55:34.189 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:41.213121+0000", 2024-09-21T13:55:34.189 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 172.00945440699999, 2024-09-21T13:55:34.189 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.087695120000000001, 2024-09-21T13:55:34.189 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:55:34.189 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:55:34.189 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:55:34.189 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:55:34.189 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:55:34.190 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:55:34.190 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 200760 2024-09-21T13:55:34.190 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.190 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:55:34.190 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.190 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:55:34.190 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.213121+0000", 2024-09-21T13:55:34.190 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:34.190 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.190 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.190 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:55:34.190 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.213121+0000", 2024-09-21T13:55:34.190 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:34.190 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.191 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.191 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:55:34.191 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.213121+0000", 2024-09-21T13:55:34.191 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8299999999999998e-07 2024-09-21T13:55:34.191 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.191 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.191 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:55:34.191 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.213123+0000", 2024-09-21T13:55:34.191 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5120000000000001e-06 2024-09-21T13:55:34.191 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.191 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.191 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:55:34.191 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.213132+0000", 2024-09-21T13:55:34.192 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7050000000000005e-06 2024-09-21T13:55:34.192 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.192 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.192 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:55:34.192 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.213136+0000", 2024-09-21T13:55:34.192 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7190000000000001e-06 2024-09-21T13:55:34.192 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.192 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.192 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:34.192 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.215739+0000", 2024-09-21T13:55:34.192 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0026025520000000002 2024-09-21T13:55:34.192 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.192 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.193 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:55:34.193 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.215873+0000", 2024-09-21T13:55:34.193 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013393699999999999 2024-09-21T13:55:34.193 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.193 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.193 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-21T13:55:34.193 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.215971+0000", 2024-09-21T13:55:34.193 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.8191999999999998e-05 2024-09-21T13:55:34.193 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.193 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.193 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:55:34.193 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.220479+0000", 2024-09-21T13:55:34.193 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.004507914 2024-09-21T13:55:34.194 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.194 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.194 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:34.194 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.297617+0000", 2024-09-21T13:55:34.194 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.077137898999999996 2024-09-21T13:55:34.194 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.194 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.194 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:34.194 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.300731+0000", 2024-09-21T13:55:34.194 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0031139399999999999 2024-09-21T13:55:34.194 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.195 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.195 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:55:34.195 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.300785+0000", 2024-09-21T13:55:34.195 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4586999999999998e-05 2024-09-21T13:55:34.195 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.195 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.195 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:55:34.195 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.300816+0000", 2024-09-21T13:55:34.195 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0880000000000002e-05 2024-09-21T13:55:34.195 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:34.195 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:55:34.195 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:34.195 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.196 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.196 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:200761 7.4 7:2244d806:::benchmark_data_smithi049_76798_object64341:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-09-21T13:55:34.196 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:41.213153+0000", 2024-09-21T13:55:34.196 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 172.00942213799999, 2024-09-21T13:55:34.196 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.087775588000000002, 2024-09-21T13:55:34.196 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:55:34.196 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:55:34.196 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:55:34.196 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:55:34.196 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:55:34.196 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:55:34.196 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 200761 2024-09-21T13:55:34.196 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.197 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:55:34.197 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.197 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:55:34.197 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.213153+0000", 2024-09-21T13:55:34.197 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:34.197 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.197 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.197 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:55:34.197 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.213153+0000", 2024-09-21T13:55:34.197 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:34.197 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.197 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.197 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:55:34.197 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.213154+0000", 2024-09-21T13:55:34.198 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1899999999999998e-07 2024-09-21T13:55:34.198 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.198 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.198 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:55:34.198 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.213156+0000", 2024-09-21T13:55:34.198 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9240000000000001e-06 2024-09-21T13:55:34.198 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.198 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.198 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:55:34.198 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.213160+0000", 2024-09-21T13:55:34.198 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9500000000000003e-06 2024-09-21T13:55:34.198 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.199 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.199 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:55:34.199 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.213164+0000", 2024-09-21T13:55:34.199 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4909999999999999e-06 2024-09-21T13:55:34.199 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.199 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.199 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:34.199 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.218329+0000", 2024-09-21T13:55:34.199 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0051646749999999996 2024-09-21T13:55:34.199 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.199 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.199 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:55:34.200 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.218418+0000", 2024-09-21T13:55:34.200 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.9097000000000004e-05 2024-09-21T13:55:34.200 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.200 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.200 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-21T13:55:34.200 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.218524+0000", 2024-09-21T13:55:34.200 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000106258 2024-09-21T13:55:34.200 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.200 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.200 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:55:34.200 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.220599+0000", 2024-09-21T13:55:34.200 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0020744869999999999 2024-09-21T13:55:34.200 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.200 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.201 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:34.201 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.297655+0000", 2024-09-21T13:55:34.201 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.077056300999999994 2024-09-21T13:55:34.201 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.201 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.201 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:34.201 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.300858+0000", 2024-09-21T13:55:34.201 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0032033550000000002 2024-09-21T13:55:34.201 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.201 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.201 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:55:34.201 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.300900+0000", 2024-09-21T13:55:34.202 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1273000000000003e-05 2024-09-21T13:55:34.202 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.202 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.202 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:55:34.202 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:41.300929+0000", 2024-09-21T13:55:34.202 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9458000000000002e-05 2024-09-21T13:55:34.202 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:34.202 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:55:34.202 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:34.202 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.202 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.202 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:202117 7.1 7:8d183654:::benchmark_data_smithi049_76798_object65697:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-21T13:55:34.202 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:44.231638+0000", 2024-09-21T13:55:34.203 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 168.99093773300001, 2024-09-21T13:55:34.203 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.111972288, 2024-09-21T13:55:34.203 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:55:34.203 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:55:34.203 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:55:34.203 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:55:34.203 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:55:34.203 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:55:34.203 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 202117 2024-09-21T13:55:34.203 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.203 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:55:34.204 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.204 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:55:34.204 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.231638+0000", 2024-09-21T13:55:34.204 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:34.204 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.204 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.204 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:55:34.204 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.231638+0000", 2024-09-21T13:55:34.204 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:34.204 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.204 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.204 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:55:34.205 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.231640+0000", 2024-09-21T13:55:34.205 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3329999999999999e-06 2024-09-21T13:55:34.205 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.205 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.205 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:55:34.205 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.231649+0000", 2024-09-21T13:55:34.205 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0399999999999998e-06 2024-09-21T13:55:34.205 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.205 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.205 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:55:34.205 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.231674+0000", 2024-09-21T13:55:34.205 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.461e-05 2024-09-21T13:55:34.205 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.206 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.206 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:55:34.206 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.231688+0000", 2024-09-21T13:55:34.206 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4063000000000001e-05 2024-09-21T13:55:34.206 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.206 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.206 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:34.206 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.231803+0000", 2024-09-21T13:55:34.206 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000115295 2024-09-21T13:55:34.206 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.206 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.206 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:55:34.206 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.231950+0000", 2024-09-21T13:55:34.207 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014642 2024-09-21T13:55:34.207 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.207 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.207 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,6", 2024-09-21T13:55:34.207 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.232017+0000", 2024-09-21T13:55:34.207 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7754000000000002e-05 2024-09-21T13:55:34.207 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.207 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.207 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:55:34.207 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.233984+0000", 2024-09-21T13:55:34.207 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0019670320000000001 2024-09-21T13:55:34.207 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.208 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.208 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:34.208 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.236712+0000", 2024-09-21T13:55:34.208 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0027279040000000002 2024-09-21T13:55:34.208 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.208 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.208 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:34.208 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.343530+0000", 2024-09-21T13:55:34.208 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.106818107 2024-09-21T13:55:34.208 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.208 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.208 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:55:34.209 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.343568+0000", 2024-09-21T13:55:34.209 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7317000000000002e-05 2024-09-21T13:55:34.209 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.209 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.209 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:55:34.209 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.343610+0000", 2024-09-21T13:55:34.209 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2413000000000002e-05 2024-09-21T13:55:34.209 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:34.209 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:55:34.209 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:34.209 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.209 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.209 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:249864 7.1 7:8a44b6f3:::benchmark_data_smithi049_76798_object113444:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e168)", 2024-09-21T13:55:34.210 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:53:36.300462+0000", 2024-09-21T13:55:34.210 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 116.922113831, 2024-09-21T13:55:34.210 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.111981706, 2024-09-21T13:55:34.210 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:55:34.210 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:55:34.210 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:55:34.210 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:55:34.210 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:55:34.210 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:55:34.210 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 249864 2024-09-21T13:55:34.210 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.210 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:55:34.211 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.211 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:55:34.211 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:53:36.300462+0000", 2024-09-21T13:55:34.211 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:34.211 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.211 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.211 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:55:34.211 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:53:36.300462+0000", 2024-09-21T13:55:34.211 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:34.211 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.211 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.211 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:55:34.212 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:53:36.300463+0000", 2024-09-21T13:55:34.212 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7100000000000011e-07 2024-09-21T13:55:34.212 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.212 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.212 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:55:34.212 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:53:36.300467+0000", 2024-09-21T13:55:34.212 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3830000000000002e-06 2024-09-21T13:55:34.212 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.212 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.212 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:55:34.212 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:53:36.300478+0000", 2024-09-21T13:55:34.212 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.078e-05 2024-09-21T13:55:34.212 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.213 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.213 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:55:34.213 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:53:36.300486+0000", 2024-09-21T13:55:34.213 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6960000000000005e-06 2024-09-21T13:55:34.213 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.213 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.213 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:34.213 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:53:36.300533+0000", 2024-09-21T13:55:34.213 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7756999999999997e-05 2024-09-21T13:55:34.213 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.213 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.213 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:55:34.213 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:53:36.300635+0000", 2024-09-21T13:55:34.214 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000101618 2024-09-21T13:55:34.214 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.214 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.214 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,6", 2024-09-21T13:55:34.214 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:53:36.300736+0000", 2024-09-21T13:55:34.214 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000101516 2024-09-21T13:55:34.214 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.214 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.214 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:55:34.214 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:53:36.303427+0000", 2024-09-21T13:55:34.214 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.002690909 2024-09-21T13:55:34.214 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.215 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.215 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:34.215 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:53:36.303790+0000", 2024-09-21T13:55:34.215 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00036281099999999999 2024-09-21T13:55:34.215 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.215 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.215 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:34.215 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:53:36.412339+0000", 2024-09-21T13:55:34.215 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.108549004 2024-09-21T13:55:34.215 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.215 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.215 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:55:34.215 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:53:36.412400+0000", 2024-09-21T13:55:34.216 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0581000000000002e-05 2024-09-21T13:55:34.216 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.216 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.216 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:55:34.216 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:53:36.412443+0000", 2024-09-21T13:55:34.216 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3680000000000002e-05 2024-09-21T13:55:34.216 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:34.216 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:55:34.216 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:34.216 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.216 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.216 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.15207.0:13662 8.2 8:47bf13f7:::benchmark_data_smithi049_109311_object13661:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-09-21T13:55:34.216 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:54:22.840535+0000", 2024-09-21T13:55:34.216 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 70.382040418000003, 2024-09-21T13:55:34.217 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.105874989, 2024-09-21T13:55:34.217 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:55:34.217 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:55:34.217 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:55:34.217 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:55:34.217 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.15207", 2024-09-21T13:55:34.217 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:55:34.217 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 13662 2024-09-21T13:55:34.217 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.217 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:55:34.217 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.217 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:55:34.218 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:22.840535+0000", 2024-09-21T13:55:34.218 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:34.218 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.218 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.218 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:55:34.218 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:22.840535+0000", 2024-09-21T13:55:34.218 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:34.218 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.218 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.218 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:55:34.218 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:22.840537+0000", 2024-09-21T13:55:34.218 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5579999999999999e-06 2024-09-21T13:55:34.218 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.219 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.219 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:55:34.219 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:22.840652+0000", 2024-09-21T13:55:34.219 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000115439 2024-09-21T13:55:34.219 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.219 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.219 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:55:34.219 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:22.840668+0000", 2024-09-21T13:55:34.219 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5468000000000002e-05 2024-09-21T13:55:34.219 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.219 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.219 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:55:34.219 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:22.840681+0000", 2024-09-21T13:55:34.220 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3937999999999999e-05 2024-09-21T13:55:34.220 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.220 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.220 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:34.220 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:22.840742+0000", 2024-09-21T13:55:34.220 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0349000000000001e-05 2024-09-21T13:55:34.220 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.220 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.220 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:55:34.220 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:22.840878+0000", 2024-09-21T13:55:34.220 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013608399999999999 2024-09-21T13:55:34.221 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.221 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.221 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,5", 2024-09-21T13:55:34.221 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:22.841023+0000", 2024-09-21T13:55:34.221 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000145363 2024-09-21T13:55:34.221 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.221 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.221 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:55:34.221 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:22.842158+0000", 2024-09-21T13:55:34.221 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0011342360000000001 2024-09-21T13:55:34.221 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.221 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.222 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:34.222 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:22.844357+0000", 2024-09-21T13:55:34.222 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0021993910000000002 2024-09-21T13:55:34.222 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.222 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.222 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:34.222 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:22.946329+0000", 2024-09-21T13:55:34.222 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.101972518 2024-09-21T13:55:34.222 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.222 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.222 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:55:34.222 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:22.946365+0000", 2024-09-21T13:55:34.223 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5917999999999997e-05 2024-09-21T13:55:34.223 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.223 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.223 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:55:34.223 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:22.946410+0000", 2024-09-21T13:55:34.223 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4727000000000001e-05 2024-09-21T13:55:34.223 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:34.223 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:55:34.223 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:34.223 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.223 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.224 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.15207.0:14214 8.2 8:46d3a502:::benchmark_data_smithi049_109311_object14213:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-09-21T13:55:34.224 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:54:23.117083+0000", 2024-09-21T13:55:34.224 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 70.105492330999994, 2024-09-21T13:55:34.224 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.12033872, 2024-09-21T13:55:34.224 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:55:34.224 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:55:34.224 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:55:34.224 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:55:34.224 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.15207", 2024-09-21T13:55:34.224 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:55:34.224 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 14214 2024-09-21T13:55:34.225 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.225 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:55:34.225 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.225 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:55:34.225 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.117083+0000", 2024-09-21T13:55:34.225 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:34.225 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.225 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.225 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:55:34.225 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.117083+0000", 2024-09-21T13:55:34.225 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:34.226 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.226 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.226 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:55:34.226 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.117094+0000", 2024-09-21T13:55:34.226 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0915e-05 2024-09-21T13:55:34.226 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.226 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.226 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:55:34.226 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.117127+0000", 2024-09-21T13:55:34.226 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3222999999999997e-05 2024-09-21T13:55:34.226 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.227 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.227 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:55:34.227 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.117141+0000", 2024-09-21T13:55:34.227 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3604999999999999e-05 2024-09-21T13:55:34.227 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.227 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.227 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:55:34.227 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.117148+0000", 2024-09-21T13:55:34.227 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5739999999999998e-06 2024-09-21T13:55:34.227 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.227 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.227 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:34.228 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.117215+0000", 2024-09-21T13:55:34.228 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6672000000000001e-05 2024-09-21T13:55:34.228 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.228 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.228 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:55:34.228 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.117316+0000", 2024-09-21T13:55:34.228 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000100831 2024-09-21T13:55:34.228 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.228 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.228 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,5", 2024-09-21T13:55:34.228 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.117464+0000", 2024-09-21T13:55:34.229 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000148035 2024-09-21T13:55:34.229 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.229 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.229 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:55:34.229 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.118494+0000", 2024-09-21T13:55:34.229 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001029587 2024-09-21T13:55:34.229 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.229 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.229 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:34.229 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.118598+0000", 2024-09-21T13:55:34.229 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000104079 2024-09-21T13:55:34.230 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.230 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.230 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:34.230 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.237344+0000", 2024-09-21T13:55:34.230 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.118746166 2024-09-21T13:55:34.230 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.230 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.230 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:55:34.230 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.237385+0000", 2024-09-21T13:55:34.230 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1495000000000002e-05 2024-09-21T13:55:34.230 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.231 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.231 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:55:34.231 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.237422+0000", 2024-09-21T13:55:34.231 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6538000000000002e-05 2024-09-21T13:55:34.231 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:34.231 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:55:34.231 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:34.231 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.231 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.231 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.15207.0:14217 8.2 8:42c0ac0e:::benchmark_data_smithi049_109311_object14216:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-09-21T13:55:34.231 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:54:23.118628+0000", 2024-09-21T13:55:34.231 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 70.103947945000002, 2024-09-21T13:55:34.232 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.120069462, 2024-09-21T13:55:34.232 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:55:34.232 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:55:34.232 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:55:34.232 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:55:34.232 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.15207", 2024-09-21T13:55:34.232 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:55:34.232 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 14217 2024-09-21T13:55:34.232 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.232 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:55:34.232 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.232 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:55:34.233 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.118628+0000", 2024-09-21T13:55:34.233 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:34.233 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.233 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.233 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:55:34.233 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.118628+0000", 2024-09-21T13:55:34.233 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:34.233 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.233 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.233 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:55:34.233 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.118629+0000", 2024-09-21T13:55:34.233 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0499999999999999e-06 2024-09-21T13:55:34.234 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.234 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.234 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:55:34.234 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.118673+0000", 2024-09-21T13:55:34.234 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4274000000000003e-05 2024-09-21T13:55:34.234 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.234 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.234 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:55:34.234 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.118686+0000", 2024-09-21T13:55:34.234 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3189e-05 2024-09-21T13:55:34.234 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.234 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.235 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:55:34.235 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.118695+0000", 2024-09-21T13:55:34.235 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0690000000000008e-06 2024-09-21T13:55:34.235 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.235 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.235 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:34.235 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.118761+0000", 2024-09-21T13:55:34.235 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6203000000000003e-05 2024-09-21T13:55:34.235 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.235 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.235 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:55:34.235 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.118897+0000", 2024-09-21T13:55:34.235 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013574400000000001 2024-09-21T13:55:34.236 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.236 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.236 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,5", 2024-09-21T13:55:34.236 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.119033+0000", 2024-09-21T13:55:34.236 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013564499999999999 2024-09-21T13:55:34.236 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.236 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.236 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:55:34.236 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.119556+0000", 2024-09-21T13:55:34.236 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00052297800000000001 2024-09-21T13:55:34.236 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.236 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.236 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:34.237 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.120042+0000", 2024-09-21T13:55:34.237 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00048602399999999999 2024-09-21T13:55:34.237 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.237 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.237 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:34.237 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.238612+0000", 2024-09-21T13:55:34.237 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.118570721 2024-09-21T13:55:34.237 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.237 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.237 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:55:34.237 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.238657+0000", 2024-09-21T13:55:34.237 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4935000000000003e-05 2024-09-21T13:55:34.238 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:34.238 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:34.238 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:55:34.238 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.238697+0000", 2024-09-21T13:55:34.238 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9629999999999998e-05 2024-09-21T13:55:34.238 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:34.238 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:55:34.238 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:34.238 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:34.238 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:55:34.238 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T13:55:34.239 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:34.264 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:34.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:34.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:34.299 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 76 16 113626 113610 93.4114 0 - 0.0105534 2024-09-21T13:55:34.366 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:55:34.388 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:34.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:34.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:55:34.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:34 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-09-21T13:55:34.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:34 smithi187 ceph-mon[33032]: osdmap e211: 8 total, 8 up, 7 in 2024-09-21T13:55:34.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:34 smithi187 ceph-mon[33032]: 8.f scrub starts 2024-09-21T13:55:34.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:34 smithi187 ceph-mon[33032]: 8.0 scrub ok 2024-09-21T13:55:34.478 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:34 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-09-21T13:55:34.479 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:34 smithi049 ceph-mon[40489]: osdmap e211: 8 total, 8 up, 7 in 2024-09-21T13:55:34.479 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:34 smithi049 ceph-mon[40489]: 8.f scrub starts 2024-09-21T13:55:34.479 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:34 smithi049 ceph-mon[40489]: 8.0 scrub ok 2024-09-21T13:55:34.479 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:34 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-09-21T13:55:34.479 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:34 smithi049 ceph-mon[30080]: osdmap e211: 8 total, 8 up, 7 in 2024-09-21T13:55:34.479 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:34 smithi049 ceph-mon[30080]: 8.f scrub starts 2024-09-21T13:55:34.479 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:34 smithi049 ceph-mon[30080]: 8.0 scrub ok 2024-09-21T13:55:34.490 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:34.511 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:34.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:34.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:34.576 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:55:34.613 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:55:34.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:34.636 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:34.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:55:34.737 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:55:34.758 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:34.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:34.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:55:34.860 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:34.881 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:34.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:34.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:34.982 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:55:35.005 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:35.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:35.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:55:35.085 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:55:35.085 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:55:35.102 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:55:35.102 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:55:35.106 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:35.118 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:55:35.119 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:55:35.127 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:35.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:35.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:35.135 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:55:35.135 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:55:35.152 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:55:35.152 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:55:35.169 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:55:35.169 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:55:35.185 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:55:35.186 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:55:35.202 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:55:35.202 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:55:35.221 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-09-21T13:55:35.228 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:55:35.238 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:35 smithi049 ceph-mon[30080]: pgmap v323: 148 pgs: 3 unknown, 36 peering, 109 active+clean; 6.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 99 MiB/s wr, 1.59k op/s 2024-09-21T13:55:35.238 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:35 smithi049 ceph-mon[30080]: 6.4 scrub starts 2024-09-21T13:55:35.238 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:35 smithi049 ceph-mon[30080]: 6.4 scrub ok 2024-09-21T13:55:35.238 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:35 smithi049 ceph-mon[30080]: 8.f scrub ok 2024-09-21T13:55:35.238 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:35 smithi049 ceph-mon[30080]: Health check failed: Reduced data availability: 4 pgs inactive, 9 pgs peering (PG_AVAILABILITY) 2024-09-21T13:55:35.238 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:35 smithi049 ceph-mon[30080]: osdmap e212: 8 total, 8 up, 7 in 2024-09-21T13:55:35.238 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:35 smithi049 ceph-mon[30080]: 5.f scrub starts 2024-09-21T13:55:35.239 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:35 smithi049 ceph-mon[30080]: 5.f scrub ok 2024-09-21T13:55:35.239 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:35 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3676505925' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:55:35.239 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:35 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:55:35.239 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:35 smithi049 ceph-mon[30080]: 6.1 scrub starts 2024-09-21T13:55:35.239 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:35 smithi049 ceph-mon[30080]: 6.1 scrub ok 2024-09-21T13:55:35.239 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:35 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2705022933' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:55:35.239 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:35 smithi049 ceph-mon[40489]: pgmap v323: 148 pgs: 3 unknown, 36 peering, 109 active+clean; 6.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 99 MiB/s wr, 1.59k op/s 2024-09-21T13:55:35.239 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:35 smithi049 ceph-mon[40489]: 6.4 scrub starts 2024-09-21T13:55:35.239 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:35 smithi049 ceph-mon[40489]: 6.4 scrub ok 2024-09-21T13:55:35.239 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:35 smithi049 ceph-mon[40489]: 8.f scrub ok 2024-09-21T13:55:35.239 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:35 smithi049 ceph-mon[40489]: Health check failed: Reduced data availability: 4 pgs inactive, 9 pgs peering (PG_AVAILABILITY) 2024-09-21T13:55:35.239 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:35 smithi049 ceph-mon[40489]: osdmap e212: 8 total, 8 up, 7 in 2024-09-21T13:55:35.239 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:35 smithi049 ceph-mon[40489]: 5.f scrub starts 2024-09-21T13:55:35.240 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:35 smithi049 ceph-mon[40489]: 5.f scrub ok 2024-09-21T13:55:35.240 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:35 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3676505925' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:55:35.240 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:35 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:55:35.240 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:35 smithi049 ceph-mon[40489]: 6.1 scrub starts 2024-09-21T13:55:35.240 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:35 smithi049 ceph-mon[40489]: 6.1 scrub ok 2024-09-21T13:55:35.240 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:35 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2705022933' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:55:35.253 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:35.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:35.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:55:35.299 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 77 16 113626 113610 92.1984 0 - 0.0105534 2024-09-21T13:55:35.355 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:55:35.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:35.386 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:35.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:55:35.405 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T13:55:35.479 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:35 smithi187 ceph-mon[33032]: pgmap v323: 148 pgs: 3 unknown, 36 peering, 109 active+clean; 6.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 99 MiB/s wr, 1.59k op/s 2024-09-21T13:55:35.479 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:35 smithi187 ceph-mon[33032]: 6.4 scrub starts 2024-09-21T13:55:35.479 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:35 smithi187 ceph-mon[33032]: 6.4 scrub ok 2024-09-21T13:55:35.479 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:35 smithi187 ceph-mon[33032]: 8.f scrub ok 2024-09-21T13:55:35.479 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:35 smithi187 ceph-mon[33032]: Health check failed: Reduced data availability: 4 pgs inactive, 9 pgs peering (PG_AVAILABILITY) 2024-09-21T13:55:35.479 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:35 smithi187 ceph-mon[33032]: osdmap e212: 8 total, 8 up, 7 in 2024-09-21T13:55:35.479 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:35 smithi187 ceph-mon[33032]: 5.f scrub starts 2024-09-21T13:55:35.479 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:35 smithi187 ceph-mon[33032]: 5.f scrub ok 2024-09-21T13:55:35.479 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:35 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3676505925' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:55:35.479 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:35 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:55:35.480 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:35 smithi187 ceph-mon[33032]: 6.1 scrub starts 2024-09-21T13:55:35.480 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:35 smithi187 ceph-mon[33032]: 6.1 scrub ok 2024-09-21T13:55:35.480 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:35 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2705022933' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:55:35.487 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:55:35.532 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:35.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:35.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:55:35.634 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:55:35.670 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:35.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:35.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:55:35.772 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:55:35.811 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:35.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:35.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:55:35.862 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:55:35.913 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:55:35.949 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:35.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:35.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:55:36.050 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:55:36.084 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:36.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:36.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:55:36.186 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:36.214 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:36.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:36.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:36.270 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:36 smithi187 ceph-mon[33032]: 4.6 scrub starts 2024-09-21T13:55:36.270 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:36 smithi187 ceph-mon[33032]: 4.6 scrub ok 2024-09-21T13:55:36.270 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:36 smithi187 ceph-mon[33032]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T13:55:36.270 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:36 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T13:55:36.270 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:36 smithi187 ceph-mon[33032]: 8.3 starting backfill to osd.2 from (0'0,0'0] MAX to 210'7092 2024-09-21T13:55:36.270 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:36 smithi187 ceph-mon[33032]: osdmap e213: 8 total, 8 up, 7 in 2024-09-21T13:55:36.270 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:36 smithi187 ceph-mon[33032]: 8.7 starting backfill to osd.2 from (0'0,0'0] MAX to 210'7064 2024-09-21T13:55:36.271 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:36 smithi187 ceph-mon[33032]: 8.9 starting backfill to osd.3 from (0'0,0'0] MAX to 210'7197 2024-09-21T13:55:36.271 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:36 smithi187 ceph-mon[33032]: 8.6 starting backfill to osd.2 from (0'0,0'0] MAX to 210'7079 2024-09-21T13:55:36.271 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:36 smithi187 ceph-mon[33032]: 8.5 starting backfill to osd.0 from (0'0,0'0] MAX to 210'7228 2024-09-21T13:55:36.299 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 78 16 114680 114664 91.8609 21.9583 0.0183145 0.0108815 2024-09-21T13:55:36.316 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:55:36.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:36.363 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:36.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:55:36.464 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:36.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:36 smithi049 ceph-mon[40489]: 4.6 scrub starts 2024-09-21T13:55:36.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:36 smithi049 ceph-mon[40489]: 4.6 scrub ok 2024-09-21T13:55:36.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:36 smithi049 ceph-mon[40489]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T13:55:36.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:36 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T13:55:36.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:36 smithi049 ceph-mon[40489]: 8.3 starting backfill to osd.2 from (0'0,0'0] MAX to 210'7092 2024-09-21T13:55:36.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:36 smithi049 ceph-mon[40489]: osdmap e213: 8 total, 8 up, 7 in 2024-09-21T13:55:36.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:36 smithi049 ceph-mon[40489]: 8.7 starting backfill to osd.2 from (0'0,0'0] MAX to 210'7064 2024-09-21T13:55:36.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:36 smithi049 ceph-mon[40489]: 8.9 starting backfill to osd.3 from (0'0,0'0] MAX to 210'7197 2024-09-21T13:55:36.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:36 smithi049 ceph-mon[40489]: 8.6 starting backfill to osd.2 from (0'0,0'0] MAX to 210'7079 2024-09-21T13:55:36.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:36 smithi049 ceph-mon[40489]: 8.5 starting backfill to osd.0 from (0'0,0'0] MAX to 210'7228 2024-09-21T13:55:36.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:36 smithi049 ceph-mon[30080]: 4.6 scrub starts 2024-09-21T13:55:36.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:36 smithi049 ceph-mon[30080]: 4.6 scrub ok 2024-09-21T13:55:36.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:36 smithi049 ceph-mon[30080]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T13:55:36.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:36 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T13:55:36.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:36 smithi049 ceph-mon[30080]: 8.3 starting backfill to osd.2 from (0'0,0'0] MAX to 210'7092 2024-09-21T13:55:36.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:36 smithi049 ceph-mon[30080]: osdmap e213: 8 total, 8 up, 7 in 2024-09-21T13:55:36.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:36 smithi049 ceph-mon[30080]: 8.7 starting backfill to osd.2 from (0'0,0'0] MAX to 210'7064 2024-09-21T13:55:36.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:36 smithi049 ceph-mon[30080]: 8.9 starting backfill to osd.3 from (0'0,0'0] MAX to 210'7197 2024-09-21T13:55:36.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:36 smithi049 ceph-mon[30080]: 8.6 starting backfill to osd.2 from (0'0,0'0] MAX to 210'7079 2024-09-21T13:55:36.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:36 smithi049 ceph-mon[30080]: 8.5 starting backfill to osd.0 from (0'0,0'0] MAX to 210'7228 2024-09-21T13:55:36.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:36.507 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:36.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:36.591 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T13:55:36.591 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:55:36.608 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:55:36.660 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:36.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:36.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:55:36.762 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:36.798 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:36.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:36.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:36.900 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:55:36.939 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:36.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:36.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:55:37.040 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:55:37.072 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:37.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:37.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:55:37.174 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:55:37.219 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:37.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:37.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:55:37.293 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T13:55:37.300 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 79 16 115758 115742 91.5509 67.375 0.0103502 0.0109181 2024-09-21T13:55:37.321 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:55:37.363 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:37.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:37.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:55:37.465 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:37.474 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:37 smithi049 ceph-mon[40489]: pgmap v326: 148 pgs: 3 unknown, 45 peering, 100 active+clean; 6.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 350 B/s rd, 54 MiB/s wr, 869 op/s 2024-09-21T13:55:37.474 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:37 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:55:37.474 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:37 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:55:37.474 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:37 smithi049 ceph-mon[40489]: osdmap e214: 8 total, 8 up, 7 in 2024-09-21T13:55:37.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:37 smithi049 ceph-mon[30080]: pgmap v326: 148 pgs: 3 unknown, 45 peering, 100 active+clean; 6.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 350 B/s rd, 54 MiB/s wr, 869 op/s 2024-09-21T13:55:37.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:37 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:55:37.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:37 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:55:37.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:37 smithi049 ceph-mon[30080]: osdmap e214: 8 total, 8 up, 7 in 2024-09-21T13:55:37.492 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:37.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:37.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:37.594 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:55:37.637 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:37.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:37.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:55:37.667 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:55:37.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:37 smithi187 ceph-mon[33032]: pgmap v326: 148 pgs: 3 unknown, 45 peering, 100 active+clean; 6.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 350 B/s rd, 54 MiB/s wr, 869 op/s 2024-09-21T13:55:37.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:37 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:55:37.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:37 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:55:37.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:37 smithi187 ceph-mon[33032]: osdmap e214: 8 total, 8 up, 7 in 2024-09-21T13:55:37.739 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:55:37.781 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:37.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:37.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:55:37.883 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:37.918 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:37.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:37.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:38.020 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:55:38.057 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:38.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:38.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:55:38.159 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:38.192 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:38.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:38.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:38.277 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T13:55:38.277 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:55:38.294 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:55:38.299 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:2024-09-21T13:55:38.300676+0000 min lat: 0.00131689 max lat: 2.22092 avg lat: 0.010961 2024-09-21T13:55:38.300 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T13:55:38.300 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 80 16 116760 116744 91.1893 62.625 0.00902369 0.010961 2024-09-21T13:55:38.323 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:38.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:38.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:55:38.425 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:55:38.472 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:38.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:38.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:55:38.504 INFO:journalctl@ceph.rgw.foo.a.smithi049.stdout:Sep 21 13:55:38 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-rgw-foo-a[66367]: 2024-09-21T13:55:38.200+0000 7fe0d330e640 -1 rgw watcher librados: RGWWatcher::handle_error cookie 94025139210368 err (110) Connection timed out 2024-09-21T13:55:38.574 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:55:38.624 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:38.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:38.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:55:38.640 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T13:55:38.726 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:55:38.753 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:38.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:38.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:55:38.855 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:55:38.907 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:38.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:38.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:55:38.978 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 6, 7, 2] 2024-09-21T13:55:38.978 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T13:55:38.978 INFO:tasks.thrashosds.thrasher:Killing osd 3, live_osds are [0, 1, 3, 4, 5, 6, 7, 2] 2024-09-21T13:55:38.978 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-09-21T13:55:38.979 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@osd.3 2024-09-21T13:55:39.009 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:55:39.036 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:39.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:39.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:55:39.048 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T13:55:39.138 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:39.155 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:55:39.166 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:39.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:39.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:39.181 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T13:55:39.268 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:39.298 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:39.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:39.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:39.300 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 81 16 117654 117638 90.7525 55.875 0.0165 0.0110143 2024-09-21T13:55:39.308 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:39 smithi049 ceph-mon[40489]: pgmap v328: 148 pgs: 3 unknown, 45 peering, 100 active+clean; 6.9 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 444 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-09-21T13:55:39.308 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:39 smithi049 ceph-mon[30080]: pgmap v328: 148 pgs: 3 unknown, 45 peering, 100 active+clean; 6.9 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 444 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-09-21T13:55:39.309 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 13:55:39 smithi049 systemd[1]: Stopping Ceph osd.3 for 40a3d2e8-781f-11ef-baf6-efdc52797490... 2024-09-21T13:55:39.312 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:39 smithi187 ceph-mon[33032]: pgmap v328: 148 pgs: 3 unknown, 45 peering, 100 active+clean; 6.9 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 444 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-09-21T13:55:39.399 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:55:39.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:39.447 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:39.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:55:39.458 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:55:39.548 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:55:39.576 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:39.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:39.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:55:39.590 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:55:39 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:55:39.678 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:39.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:39.715 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:39.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:39.816 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:55:39.849 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:39.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:39.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:55:39.951 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:55:39.980 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:55:40.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:40.012 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:40.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:55:40.043 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T13:55:40.043 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:55:40.113 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:40.142 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:40.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:40.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:40.244 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:55:40.288 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:40.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:40.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:55:40.300 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 82 16 118560 118544 90.3363 56.625 0.0403701 0.0110594 2024-09-21T13:55:40.390 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:55:40.401 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:40 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:55:40.401 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:40 smithi049 ceph-mon[40489]: Health check update: Reduced data availability: 5 pgs inactive, 10 pgs peering (PG_AVAILABILITY) 2024-09-21T13:55:40.401 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:40 smithi049 ceph-mon[40489]: osd.3 marked itself down and dead 2024-09-21T13:55:40.402 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:40 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:55:40.402 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:40 smithi049 ceph-mon[30080]: Health check update: Reduced data availability: 5 pgs inactive, 10 pgs peering (PG_AVAILABILITY) 2024-09-21T13:55:40.402 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:40 smithi049 ceph-mon[30080]: osd.3 marked itself down and dead 2024-09-21T13:55:40.402 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 13:55:40 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-3[61768]: 2024-09-21T13:55:40.145+0000 7f5b4bc7f640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T13:55:40.402 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 13:55:40 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-3[61768]: 2024-09-21T13:55:40.145+0000 7f5b4bc7f640 -1 osd.3 214 *** Got signal Terminated *** 2024-09-21T13:55:40.402 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 13:55:40 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-3[61768]: 2024-09-21T13:55:40.145+0000 7f5b4bc7f640 -1 osd.3 214 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T13:55:40.413 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:40.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:40.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:55:40.515 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:55:40.543 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:40.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:40.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:55:40.645 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:40.667 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:40.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:40.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:40.677 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:55:40 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:55:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:55:40.679 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:40 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:55:40.679 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:40 smithi187 ceph-mon[33032]: Health check update: Reduced data availability: 5 pgs inactive, 10 pgs peering (PG_AVAILABILITY) 2024-09-21T13:55:40.679 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:40 smithi187 ceph-mon[33032]: osd.3 marked itself down and dead 2024-09-21T13:55:40.700 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:55:40.700 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:55:40.711 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T13:55:40.724 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:55:40.724 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:55:40.747 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:55:40.748 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:55:40.748 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-21T13:55:40.770 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:40.772 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:55:40.772 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:55:40.799 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:55:40.799 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:55:40.817 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:40.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:40.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:40.826 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:55:40.826 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:55:40.853 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:55:40.854 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:55:40.918 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:40.942 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:40.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:40.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:41.043 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:41.044 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:55:41.071 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:41.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:41.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:41.173 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:55:41.205 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:41.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:41.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:55:41.256 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-09-21T13:55:41.301 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 83 16 118587 118571 89.2683 1.6875 1.05202 0.011163 2024-09-21T13:55:41.307 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:41.348 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:41.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:41.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:41.426 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:41 smithi049 ceph-mon[30080]: pgmap v329: 148 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 143 active+clean; 7.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 853 B/s rd, 52 MiB/s wr, 826 op/s; 35195/351975 objects misplaced (9.999%); 16 MiB/s, 269 objects/s recovering 2024-09-21T13:55:41.426 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:41 smithi049 ceph-mon[30080]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T13:55:41.427 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:41 smithi049 ceph-mon[30080]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 10 pgs peering) 2024-09-21T13:55:41.427 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:41 smithi049 ceph-mon[30080]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T13:55:41.427 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:41 smithi049 ceph-mon[30080]: osdmap e215: 8 total, 7 up, 7 in 2024-09-21T13:55:41.427 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:41 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1215313648' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:55:41.427 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:41 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:55:41.427 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:41 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/31343619' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:55:41.427 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:41 smithi049 ceph-mon[40489]: pgmap v329: 148 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 143 active+clean; 7.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 853 B/s rd, 52 MiB/s wr, 826 op/s; 35195/351975 objects misplaced (9.999%); 16 MiB/s, 269 objects/s recovering 2024-09-21T13:55:41.427 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:41 smithi049 ceph-mon[40489]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T13:55:41.427 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:41 smithi049 ceph-mon[40489]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 10 pgs peering) 2024-09-21T13:55:41.428 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:41 smithi049 ceph-mon[40489]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T13:55:41.428 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:41 smithi049 ceph-mon[40489]: osdmap e215: 8 total, 7 up, 7 in 2024-09-21T13:55:41.428 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:41 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1215313648' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:55:41.428 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:41 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:55:41.428 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:41 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/31343619' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:55:41.428 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 13:55:41 smithi049 podman[119101]: 2024-09-21 13:55:41.114145941 +0000 UTC m=+1.138074413 container died 5d9252adab1cf824768c0ad96e6db22be42f13e760db1d3f94c01bfe7413d803 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.37.2, RELEASE=wip-guits-main-2024-09-21-0842-4417849, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-09-21T13:55:41.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:41 smithi187 ceph-mon[33032]: pgmap v329: 148 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 143 active+clean; 7.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 853 B/s rd, 52 MiB/s wr, 826 op/s; 35195/351975 objects misplaced (9.999%); 16 MiB/s, 269 objects/s recovering 2024-09-21T13:55:41.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:41 smithi187 ceph-mon[33032]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T13:55:41.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:41 smithi187 ceph-mon[33032]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 10 pgs peering) 2024-09-21T13:55:41.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:41 smithi187 ceph-mon[33032]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T13:55:41.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:41 smithi187 ceph-mon[33032]: osdmap e215: 8 total, 7 up, 7 in 2024-09-21T13:55:41.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:41 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1215313648' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:55:41.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:41 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:55:41.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:41 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/31343619' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:55:41.450 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:55:41.490 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:41.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:41.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:55:41.592 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:55:41.629 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:41.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:41.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:55:41.731 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:55:41.754 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 13:55:41 smithi049 podman[119101]: 2024-09-21 13:55:41.641463739 +0000 UTC m=+1.665392206 container cleanup 5d9252adab1cf824768c0ad96e6db22be42f13e760db1d3f94c01bfe7413d803 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-3, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, RELEASE=wip-guits-main-2024-09-21-0842-4417849, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T13:55:41.754 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 13:55:41 smithi049 bash[119101]: ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-3 2024-09-21T13:55:41.761 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:41.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:41.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:55:41.863 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:41.903 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:41.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:41.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:42.004 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:42.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:42.041 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:42.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:42.142 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:42.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:42.179 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:42.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:42.254 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 13:55:41 smithi049 podman[119257]: 2024-09-21 13:55:41.860550174 +0000 UTC m=+0.968626162 container remove 5d9252adab1cf824768c0ad96e6db22be42f13e760db1d3f94c01bfe7413d803 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-3, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-guits-main-2024-09-21-0842-4417849, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T13:55:42.280 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:55:42.300 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 84 16 119637 119621 88.9868 65.625 0.0119836 0.0112331 2024-09-21T13:55:42.323 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:42.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:42.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:55:42.426 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:55:42.461 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:42.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:42.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:55:42.564 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:42.598 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:42.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:42.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:42.643 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:42 smithi187 ceph-mon[33032]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:55:42.643 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:42 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T13:55:42.643 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:42 smithi187 ceph-mon[33032]: osdmap e216: 8 total, 7 up, 7 in 2024-09-21T13:55:42.643 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:42 smithi187 ceph-mon[33032]: 8.7 continuing backfill to osd.2 from (194'3139,214'7375] MIN to 214'7375 2024-09-21T13:55:42.700 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:55:42.735 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:42.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:42.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:55:42.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:42 smithi049 ceph-mon[40489]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:55:42.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:42 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T13:55:42.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:42 smithi049 ceph-mon[40489]: osdmap e216: 8 total, 7 up, 7 in 2024-09-21T13:55:42.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:42 smithi049 ceph-mon[40489]: 8.7 continuing backfill to osd.2 from (194'3139,214'7375] MIN to 214'7375 2024-09-21T13:55:42.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:42 smithi049 ceph-mon[30080]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:55:42.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:42 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T13:55:42.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:42 smithi049 ceph-mon[30080]: osdmap e216: 8 total, 7 up, 7 in 2024-09-21T13:55:42.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:42 smithi049 ceph-mon[30080]: 8.7 continuing backfill to osd.2 from (194'3139,214'7375] MIN to 214'7375 2024-09-21T13:55:42.816 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T13:55:42.816 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-21T13:55:42.816 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:42.816 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.15207.0:120279 8.5 e216/212 8:a63f7d9e:::benchmark_data_smithi049_109311_object120278:head v 216'7598, mlcod=216'7595)", 2024-09-21T13:55:42.816 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:55:42.806951+0000", 2024-09-21T13:55:42.816 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0081278070000000008, 2024-09-21T13:55:42.816 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.008155944, 2024-09-21T13:55:42.816 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:55:42.816 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:55:42.816 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T13:55:42.816 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:55:42.817 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:42.817 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:55:42.817 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:55:42.806951+0000", 2024-09-21T13:55:42.817 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:42.817 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:42.817 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:42.817 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:55:42.817 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:55:42.806951+0000", 2024-09-21T13:55:42.817 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:42.817 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:42.817 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:42.817 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:55:42.817 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:55:42.806952+0000", 2024-09-21T13:55:42.817 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.02e-06 2024-09-21T13:55:42.818 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:42.818 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:42.818 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:55:42.818 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:55:42.806977+0000", 2024-09-21T13:55:42.818 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4919999999999999e-05 2024-09-21T13:55:42.818 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:42.818 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:42.818 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:55:42.818 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:55:42.806992+0000", 2024-09-21T13:55:42.818 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5048999999999999e-05 2024-09-21T13:55:42.818 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:42.818 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:42.818 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:55:42.818 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:55:42.807000+0000", 2024-09-21T13:55:42.819 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3089999999999996e-06 2024-09-21T13:55:42.819 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:42.819 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:42.819 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:42.819 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:55:42.807379+0000", 2024-09-21T13:55:42.819 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00037903500000000001 2024-09-21T13:55:42.819 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:42.819 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:42.819 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:55:42.819 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:55:42.807404+0000", 2024-09-21T13:55:42.819 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4491e-05 2024-09-21T13:55:42.819 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:42.819 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:55:42.819 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:42.820 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:42.820 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:42.820 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.15207.0:120280 8.5 e216/212 8:ae6205d5:::benchmark_data_smithi049_109311_object120279:head v 216'7599, mlcod=216'7596)", 2024-09-21T13:55:42.820 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:55:42.810268+0000", 2024-09-21T13:55:42.820 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0048107489999999996, 2024-09-21T13:55:42.820 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0048972460000000001, 2024-09-21T13:55:42.820 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:55:42.820 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:55:42.820 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T13:55:42.820 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:55:42.820 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:42.820 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:55:42.820 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:55:42.810268+0000", 2024-09-21T13:55:42.820 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:42.820 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:42.821 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:42.821 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:55:42.821 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:55:42.810268+0000", 2024-09-21T13:55:42.821 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:42.821 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:42.821 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:42.821 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:55:42.821 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:55:42.810269+0000", 2024-09-21T13:55:42.821 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3399999999999998e-07 2024-09-21T13:55:42.821 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:42.821 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:42.821 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:55:42.821 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:55:42.810293+0000", 2024-09-21T13:55:42.821 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3986e-05 2024-09-21T13:55:42.821 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:42.821 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:42.822 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:55:42.822 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:55:42.810303+0000", 2024-09-21T13:55:42.822 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0443e-05 2024-09-21T13:55:42.822 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:42.822 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:42.822 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:55:42.822 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:55:42.810310+0000", 2024-09-21T13:55:42.822 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8809999999999998e-06 2024-09-21T13:55:42.822 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:42.822 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:42.822 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:42.822 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:55:42.810488+0000", 2024-09-21T13:55:42.822 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000178457 2024-09-21T13:55:42.822 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:42.823 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:42.823 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:55:42.823 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:55:42.810515+0000", 2024-09-21T13:55:42.823 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.6499000000000002e-05 2024-09-21T13:55:42.823 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:42.823 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:55:42.823 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:42.823 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:42.823 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:42.823 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.15207.0:120282 8.5 e216/212 8:ad53116e:::benchmark_data_smithi049_109311_object120281:head v 216'7600, mlcod=216'7596)", 2024-09-21T13:55:42.823 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:55:42.812859+0000", 2024-09-21T13:55:42.823 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0022193740000000001, 2024-09-21T13:55:42.823 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.002354919, 2024-09-21T13:55:42.823 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:55:42.824 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:55:42.824 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T13:55:42.824 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:55:42.824 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:42.824 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:55:42.824 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:55:42.812859+0000", 2024-09-21T13:55:42.824 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:42.824 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:42.824 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:42.824 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:55:42.824 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:55:42.812859+0000", 2024-09-21T13:55:42.824 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:42.824 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:42.824 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:42.825 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:55:42.825 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:55:42.812879+0000", 2024-09-21T13:55:42.825 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9459000000000001e-05 2024-09-21T13:55:42.825 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:42.825 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:42.825 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:55:42.825 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:55:42.812903+0000", 2024-09-21T13:55:42.825 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4094999999999999e-05 2024-09-21T13:55:42.825 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:42.825 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:42.825 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:55:42.825 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:55:42.812920+0000", 2024-09-21T13:55:42.825 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7146000000000002e-05 2024-09-21T13:55:42.825 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:42.825 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:42.825 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:55:42.826 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:55:42.812927+0000", 2024-09-21T13:55:42.826 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.379e-06 2024-09-21T13:55:42.826 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:42.826 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:42.826 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:42.826 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:55:42.813006+0000", 2024-09-21T13:55:42.826 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8770000000000006e-05 2024-09-21T13:55:42.826 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:42.826 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:42.826 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:55:42.826 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:55:42.813027+0000", 2024-09-21T13:55:42.826 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1138999999999999e-05 2024-09-21T13:55:42.826 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:42.826 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:55:42.827 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:42.827 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:42.827 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2024-09-21T13:55:42.827 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 3 2024-09-21T13:55:42.827 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T13:55:42.837 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:55:42.863 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:42.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:42.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:55:42.965 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:55:43.002 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:43.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:43.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:55:43.103 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:55:43.125 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:43.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:43.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:55:43.227 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:43.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:43.270 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:43.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:43.301 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 85 16 120878 120862 88.8523 77.5625 0.0101982 0.0112501 2024-09-21T13:55:43.370 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:43.392 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:43.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:43.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:43.493 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:55:43.529 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:43.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:43.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:55:43.630 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:43.650 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:43 smithi187 ceph-mon[33032]: pgmap v332: 148 pgs: 29 stale+active+clean, 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 114 active+clean; 7.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 853 B/s rd, 37 MiB/s wr, 586 op/s; 35195/351975 objects misplaced (9.999%); 16 MiB/s, 269 objects/s recovering 2024-09-21T13:55:43.663 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:43.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:43.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:43.706 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T13:55:43.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:43 smithi049 ceph-mon[40489]: pgmap v332: 148 pgs: 29 stale+active+clean, 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 114 active+clean; 7.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 853 B/s rd, 37 MiB/s wr, 586 op/s; 35195/351975 objects misplaced (9.999%); 16 MiB/s, 269 objects/s recovering 2024-09-21T13:55:43.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:43 smithi049 ceph-mon[30080]: pgmap v332: 148 pgs: 29 stale+active+clean, 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 114 active+clean; 7.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 853 B/s rd, 37 MiB/s wr, 586 op/s; 35195/351975 objects misplaced (9.999%); 16 MiB/s, 269 objects/s recovering 2024-09-21T13:55:43.765 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:55:43.802 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:43.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:43.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:55:43.904 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:43.925 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:43.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:43.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:44.026 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:55:44.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:44.085 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:44.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:55:44.134 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:55:44.186 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:44.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:44.238 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:44.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:44.301 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 86 16 122022 122006 88.6505 71.5 0.015071 0.0112758 2024-09-21T13:55:44.338 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:55:44.373 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:44.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:44.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:55:44.475 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:55:44.516 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:44.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:44.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:55:44.617 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:44.639 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:44 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:55:44.639 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:44 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:55:44.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:44.653 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:44.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:44.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:44 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:55:44.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:44 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:55:44.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:44 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:55:44.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:44 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:55:44.755 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:55:44.780 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:44.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:44.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:55:44.792 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T13:55:44.792 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-21T13:55:44.792 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-21T13:55:44.792 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-21T13:55:44.792 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T13:55:44.882 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:55:44.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:44.925 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:44.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:55:45.026 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:45.062 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:45.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:45.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:45.164 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:55:45.192 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:45.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:45.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:55:45.293 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:45.301 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 87 16 123072 123056 88.3858 65.625 0.01079 0.0113095 2024-09-21T13:55:45.329 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:45.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:45.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:45.431 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:55:45.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:45 smithi187 ceph-mon[33032]: pgmap v333: 148 pgs: 1 active+clean+remapped, 36 active+undersized, 1 stale+active+clean, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 29 active+undersized+degraded, 77 active+clean; 7.4 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 405 B/s rd, 58 MiB/s wr, 928 op/s; 37955/362520 objects degraded (10.470%); 33156/362520 objects misplaced (9.146%); 44 MiB/s, 716 objects/s recovering 2024-09-21T13:55:45.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:45 smithi187 ceph-mon[33032]: Health check failed: Degraded data redundancy: 37955/362520 objects degraded (10.470%), 30 pgs degraded (PG_DEGRADED) 2024-09-21T13:55:45.460 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:45.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:45.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:55:45.541 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T13:55:45.562 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:55:45.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:45.625 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:45.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:55:45.725 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:45.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:45 smithi049 ceph-mon[30080]: pgmap v333: 148 pgs: 1 active+clean+remapped, 36 active+undersized, 1 stale+active+clean, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 29 active+undersized+degraded, 77 active+clean; 7.4 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 405 B/s rd, 58 MiB/s wr, 928 op/s; 37955/362520 objects degraded (10.470%); 33156/362520 objects misplaced (9.146%); 44 MiB/s, 716 objects/s recovering 2024-09-21T13:55:45.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:45 smithi049 ceph-mon[30080]: Health check failed: Degraded data redundancy: 37955/362520 objects degraded (10.470%), 30 pgs degraded (PG_DEGRADED) 2024-09-21T13:55:45.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:45 smithi049 ceph-mon[40489]: pgmap v333: 148 pgs: 1 active+clean+remapped, 36 active+undersized, 1 stale+active+clean, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 29 active+undersized+degraded, 77 active+clean; 7.4 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 405 B/s rd, 58 MiB/s wr, 928 op/s; 37955/362520 objects degraded (10.470%); 33156/362520 objects misplaced (9.146%); 44 MiB/s, 716 objects/s recovering 2024-09-21T13:55:45.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:45 smithi049 ceph-mon[40489]: Health check failed: Degraded data redundancy: 37955/362520 objects degraded (10.470%), 30 pgs degraded (PG_DEGRADED) 2024-09-21T13:55:45.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:45.773 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:45.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:45.874 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:45.898 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:45.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:45.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:45.941 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:55:46.000 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:55:46.035 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:46.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:46.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:55:46.137 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:55:46.170 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:46.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:46.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:55:46.272 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:55:46.301 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 88 16 124249 124233 88.2173 73.5625 0.0191212 0.01133 2024-09-21T13:55:46.314 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:46.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:46.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:55:46.416 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:55:46.460 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:46.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:46.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:55:46.562 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:55:46.583 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T13:55:46.583 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-21T13:55:46.583 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-21T13:55:46.583 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-21T13:55:46.583 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.583 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:35249 7.6 7:6f97b9cd:::benchmark_data_smithi049_76798_object35248:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T13:55:46.584 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:28.696132+0000", 2024-09-21T13:55:46.584 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 317.88172398799998, 2024-09-21T13:55:46.584 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0107570610000001, 2024-09-21T13:55:46.584 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:55:46.584 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:55:46.584 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:55:46.584 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:55:46.584 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:55:46.584 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:55:46.584 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 35249 2024-09-21T13:55:46.584 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.584 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:55:46.584 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.584 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:55:46.585 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.696132+0000", 2024-09-21T13:55:46.585 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:46.585 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.585 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.585 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:55:46.585 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.696132+0000", 2024-09-21T13:55:46.585 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:46.585 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.585 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.585 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:55:46.585 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.696134+0000", 2024-09-21T13:55:46.585 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6080000000000001e-06 2024-09-21T13:55:46.585 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.585 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.585 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:55:46.586 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.696185+0000", 2024-09-21T13:55:46.586 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1328000000000001e-05 2024-09-21T13:55:46.586 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.586 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.586 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:55:46.586 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.696202+0000", 2024-09-21T13:55:46.586 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7334999999999999e-05 2024-09-21T13:55:46.586 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.586 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.586 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:55:46.586 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.696212+0000", 2024-09-21T13:55:46.586 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3489999999999998e-06 2024-09-21T13:55:46.586 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.586 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.586 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:46.587 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.696291+0000", 2024-09-21T13:55:46.587 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9078000000000004e-05 2024-09-21T13:55:46.587 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.587 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.587 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:55:46.587 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.696301+0000", 2024-09-21T13:55:46.587 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7780000000000002e-06 2024-09-21T13:55:46.587 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.587 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.587 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:46.587 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.697326+0000", 2024-09-21T13:55:46.587 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0010256289999999 2024-09-21T13:55:46.587 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.587 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.587 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:55:46.588 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.698023+0000", 2024-09-21T13:55:46.588 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00069705799999999997 2024-09-21T13:55:46.588 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.588 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.588 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3", 2024-09-21T13:55:46.588 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.698222+0000", 2024-09-21T13:55:46.588 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000198937 2024-09-21T13:55:46.588 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.588 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.588 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:55:46.588 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.703079+0000", 2024-09-21T13:55:46.588 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0048563690000000001 2024-09-21T13:55:46.588 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.588 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.588 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:46.588 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.706547+0000", 2024-09-21T13:55:46.589 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0034680499999999999 2024-09-21T13:55:46.589 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.589 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.589 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:46.589 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.706817+0000", 2024-09-21T13:55:46.589 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00026996700000000003 2024-09-21T13:55:46.589 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.589 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.589 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:55:46.589 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.706866+0000", 2024-09-21T13:55:46.589 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9292999999999997e-05 2024-09-21T13:55:46.589 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.589 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.589 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:55:46.589 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.706889+0000", 2024-09-21T13:55:46.589 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3281999999999998e-05 2024-09-21T13:55:46.590 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:46.590 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:55:46.590 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:46.590 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.590 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.590 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:35258 7.6 7:6cec60f9:::benchmark_data_smithi049_76798_object35257:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T13:55:46.590 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:28.700988+0000", 2024-09-21T13:55:46.590 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 317.87686825700001, 2024-09-21T13:55:46.590 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.007550977, 2024-09-21T13:55:46.590 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:55:46.590 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:55:46.590 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:55:46.590 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:55:46.591 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:55:46.591 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:55:46.591 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 35258 2024-09-21T13:55:46.591 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.591 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:55:46.591 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.591 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:55:46.591 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.700988+0000", 2024-09-21T13:55:46.591 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:46.591 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.591 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.591 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:55:46.591 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.700988+0000", 2024-09-21T13:55:46.591 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:46.591 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.592 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.592 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:55:46.592 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.700990+0000", 2024-09-21T13:55:46.592 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.795e-06 2024-09-21T13:55:46.592 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.592 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.592 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:55:46.592 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.701031+0000", 2024-09-21T13:55:46.592 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0836e-05 2024-09-21T13:55:46.592 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.592 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.592 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:55:46.592 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.701045+0000", 2024-09-21T13:55:46.592 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4178e-05 2024-09-21T13:55:46.592 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.593 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.593 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:55:46.593 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.701053+0000", 2024-09-21T13:55:46.593 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3499999999999997e-06 2024-09-21T13:55:46.593 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.593 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.593 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:46.593 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.701113+0000", 2024-09-21T13:55:46.593 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9857e-05 2024-09-21T13:55:46.593 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.593 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.593 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:55:46.593 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.701123+0000", 2024-09-21T13:55:46.593 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0202e-05 2024-09-21T13:55:46.593 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.593 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.594 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:46.594 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.700130+0000", 2024-09-21T13:55:46.594 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.99900719299999996 2024-09-21T13:55:46.594 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.594 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.594 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:55:46.594 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.700245+0000", 2024-09-21T13:55:46.594 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000115047 2024-09-21T13:55:46.594 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.594 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.594 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3", 2024-09-21T13:55:46.594 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.700392+0000", 2024-09-21T13:55:46.594 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000146358 2024-09-21T13:55:46.594 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.595 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.595 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:55:46.595 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.704100+0000", 2024-09-21T13:55:46.595 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0037085780000000001 2024-09-21T13:55:46.595 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.595 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.595 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:46.595 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.706575+0000", 2024-09-21T13:55:46.595 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.002474642 2024-09-21T13:55:46.595 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.595 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.595 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:46.595 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.708441+0000", 2024-09-21T13:55:46.595 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00186568 2024-09-21T13:55:46.596 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.596 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.596 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:55:46.596 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.708496+0000", 2024-09-21T13:55:46.596 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5340000000000002e-05 2024-09-21T13:55:46.596 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.596 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.596 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:55:46.596 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.708539+0000", 2024-09-21T13:55:46.596 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2920999999999998e-05 2024-09-21T13:55:46.596 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:46.596 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:55:46.596 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:46.596 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.597 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.597 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:35272 7.6 7:61fa97ba:::benchmark_data_smithi049_76798_object35271:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T13:55:46.597 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:28.712676+0000", 2024-09-21T13:55:46.597 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 317.86517996200001, 2024-09-21T13:55:46.597 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.99907040300000005, 2024-09-21T13:55:46.597 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:55:46.597 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:55:46.597 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:55:46.597 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:55:46.597 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:55:46.597 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:55:46.597 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 35272 2024-09-21T13:55:46.597 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.597 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:55:46.597 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.597 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:55:46.598 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.712676+0000", 2024-09-21T13:55:46.598 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:46.598 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.598 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.598 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:55:46.598 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.712676+0000", 2024-09-21T13:55:46.598 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:46.598 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.598 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.598 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:55:46.598 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.712678+0000", 2024-09-21T13:55:46.598 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.401e-06 2024-09-21T13:55:46.598 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.598 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.598 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:55:46.599 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.712724+0000", 2024-09-21T13:55:46.599 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6576999999999998e-05 2024-09-21T13:55:46.599 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.599 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.599 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:55:46.599 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.712736+0000", 2024-09-21T13:55:46.599 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2231e-05 2024-09-21T13:55:46.599 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.599 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.599 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:55:46.599 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.712742+0000", 2024-09-21T13:55:46.599 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9649999999999996e-06 2024-09-21T13:55:46.599 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.599 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.599 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:46.600 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.712819+0000", 2024-09-21T13:55:46.600 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6258000000000001e-05 2024-09-21T13:55:46.600 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.600 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.600 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:55:46.600 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.712831+0000", 2024-09-21T13:55:46.600 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.275e-05 2024-09-21T13:55:46.600 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.600 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.600 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:46.600 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.701685+0000", 2024-09-21T13:55:46.600 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.98885339699999997 2024-09-21T13:55:46.600 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.600 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.600 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:55:46.601 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.701779+0000", 2024-09-21T13:55:46.601 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3944999999999996e-05 2024-09-21T13:55:46.601 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.601 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.601 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3", 2024-09-21T13:55:46.601 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.701912+0000", 2024-09-21T13:55:46.601 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000133156 2024-09-21T13:55:46.601 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.601 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.601 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:55:46.601 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.705423+0000", 2024-09-21T13:55:46.601 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003511587 2024-09-21T13:55:46.601 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.601 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.601 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:46.601 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.708588+0000", 2024-09-21T13:55:46.602 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0031643309999999998 2024-09-21T13:55:46.602 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.602 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.602 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:46.602 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.711670+0000", 2024-09-21T13:55:46.602 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003081748 2024-09-21T13:55:46.602 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.602 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.602 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:55:46.602 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.711714+0000", 2024-09-21T13:55:46.602 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4406999999999997e-05 2024-09-21T13:55:46.602 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.602 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.602 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:55:46.602 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.711747+0000", 2024-09-21T13:55:46.603 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2650000000000001e-05 2024-09-21T13:55:46.603 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:46.603 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:55:46.603 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:46.603 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.603 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.603 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:35278 7.6 7:6d0c33e9:::benchmark_data_smithi049_76798_object35277:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T13:55:46.603 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:28.718784+0000", 2024-09-21T13:55:46.603 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 317.85907180200002, 2024-09-21T13:55:46.603 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.993077243, 2024-09-21T13:55:46.603 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:55:46.603 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:55:46.603 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:55:46.603 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:55:46.603 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:55:46.604 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:55:46.604 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 35278 2024-09-21T13:55:46.604 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.604 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:55:46.604 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.604 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:55:46.604 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.718784+0000", 2024-09-21T13:55:46.604 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:46.604 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.604 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.604 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:55:46.604 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.718784+0000", 2024-09-21T13:55:46.604 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:46.604 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.605 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.605 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:55:46.605 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.718787+0000", 2024-09-21T13:55:46.605 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1849999999999998e-06 2024-09-21T13:55:46.605 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.605 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.605 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:55:46.605 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.718859+0000", 2024-09-21T13:55:46.605 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2344999999999999e-05 2024-09-21T13:55:46.605 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.605 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.605 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:55:46.605 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.718872+0000", 2024-09-21T13:55:46.605 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3113e-05 2024-09-21T13:55:46.605 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.605 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.606 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:55:46.606 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.718881+0000", 2024-09-21T13:55:46.606 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.448e-06 2024-09-21T13:55:46.606 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.606 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.606 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:46.606 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.718941+0000", 2024-09-21T13:55:46.606 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9580999999999998e-05 2024-09-21T13:55:46.606 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.606 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.606 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:55:46.606 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.718948+0000", 2024-09-21T13:55:46.606 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.391e-06 2024-09-21T13:55:46.606 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.606 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.607 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:46.607 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.703121+0000", 2024-09-21T13:55:46.607 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.98417250099999998 2024-09-21T13:55:46.607 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.607 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.607 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:55:46.607 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.703248+0000", 2024-09-21T13:55:46.607 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012707800000000001 2024-09-21T13:55:46.607 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.607 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.607 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3", 2024-09-21T13:55:46.607 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.703345+0000", 2024-09-21T13:55:46.607 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7287999999999994e-05 2024-09-21T13:55:46.607 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.607 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.608 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:55:46.608 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.705432+0000", 2024-09-21T13:55:46.608 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0020863499999999998 2024-09-21T13:55:46.608 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.608 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.608 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:46.608 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.708649+0000", 2024-09-21T13:55:46.608 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003217443 2024-09-21T13:55:46.608 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.608 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.608 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:46.608 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.711798+0000", 2024-09-21T13:55:46.609 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0031489809999999999 2024-09-21T13:55:46.609 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.609 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.609 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:55:46.609 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.711832+0000", 2024-09-21T13:55:46.609 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3707000000000001e-05 2024-09-21T13:55:46.609 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.609 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.609 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:55:46.610 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.711862+0000", 2024-09-21T13:55:46.610 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9832000000000002e-05 2024-09-21T13:55:46.610 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:46.610 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:55:46.610 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:46.610 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.610 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.610 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:35280 7.6 7:68737c4e:::benchmark_data_smithi049_76798_object35279:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T13:55:46.610 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:28.719589+0000", 2024-09-21T13:55:46.610 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 317.85826734, 2024-09-21T13:55:46.610 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.99249087499999999, 2024-09-21T13:55:46.610 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:55:46.610 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:55:46.610 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:55:46.611 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:55:46.611 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:55:46.611 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:55:46.611 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 35280 2024-09-21T13:55:46.611 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.611 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:55:46.611 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.611 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:55:46.611 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.719589+0000", 2024-09-21T13:55:46.611 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:46.611 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.611 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.611 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:55:46.611 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.719589+0000", 2024-09-21T13:55:46.611 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:46.611 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.612 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.612 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:55:46.612 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.719590+0000", 2024-09-21T13:55:46.612 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.5600000000000004e-07 2024-09-21T13:55:46.612 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.612 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.612 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:55:46.612 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.719649+0000", 2024-09-21T13:55:46.612 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8730999999999997e-05 2024-09-21T13:55:46.612 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.612 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.612 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:55:46.612 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.719659+0000", 2024-09-21T13:55:46.612 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0498e-05 2024-09-21T13:55:46.612 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.613 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.613 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:55:46.613 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.719666+0000", 2024-09-21T13:55:46.613 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9029999999999998e-06 2024-09-21T13:55:46.613 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.613 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.613 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:46.613 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.719710+0000", 2024-09-21T13:55:46.613 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3859000000000001e-05 2024-09-21T13:55:46.613 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.613 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.613 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:55:46.613 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.719715+0000", 2024-09-21T13:55:46.613 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2229999999999998e-06 2024-09-21T13:55:46.613 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.614 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.614 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:46.614 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.704132+0000", 2024-09-21T13:55:46.614 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.98441694300000004 2024-09-21T13:55:46.614 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.614 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.614 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:55:46.614 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.704260+0000", 2024-09-21T13:55:46.614 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012844899999999999 2024-09-21T13:55:46.614 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.614 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.614 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3", 2024-09-21T13:55:46.614 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.704400+0000", 2024-09-21T13:55:46.614 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013959700000000001 2024-09-21T13:55:46.615 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.615 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.615 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:55:46.615 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.706512+0000", 2024-09-21T13:55:46.615 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0021123270000000002 2024-09-21T13:55:46.615 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.615 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.615 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:46.615 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.708683+0000", 2024-09-21T13:55:46.615 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00217022 2024-09-21T13:55:46.615 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.615 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.615 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:46.615 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.712030+0000", 2024-09-21T13:55:46.616 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0033470459999999998 2024-09-21T13:55:46.616 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.616 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.616 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:55:46.616 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.712058+0000", 2024-09-21T13:55:46.616 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8586e-05 2024-09-21T13:55:46.616 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.616 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.616 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:55:46.616 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.712080+0000", 2024-09-21T13:55:46.616 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1537000000000001e-05 2024-09-21T13:55:46.616 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:46.616 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:55:46.616 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:46.616 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.617 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.617 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:201635 7.5 7:a1b75091:::benchmark_data_smithi049_76798_object65215:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T13:55:46.617 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:43.011858+0000", 2024-09-21T13:55:46.617 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 183.565998667, 2024-09-21T13:55:46.617 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.96906032200000003, 2024-09-21T13:55:46.617 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:55:46.617 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:55:46.617 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:55:46.617 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:55:46.617 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:55:46.617 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:55:46.617 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 201635 2024-09-21T13:55:46.617 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.617 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:55:46.617 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.617 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:55:46.617 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.011858+0000", 2024-09-21T13:55:46.618 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:46.618 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.618 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.618 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:55:46.618 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.011858+0000", 2024-09-21T13:55:46.618 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:46.618 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.618 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.618 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:55:46.618 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.011859+0000", 2024-09-21T13:55:46.618 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.609e-06 2024-09-21T13:55:46.618 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.618 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.618 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:55:46.618 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.011865+0000", 2024-09-21T13:55:46.618 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2489999999999998e-06 2024-09-21T13:55:46.618 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.619 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.619 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:55:46.619 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.011880+0000", 2024-09-21T13:55:46.619 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4141999999999999e-05 2024-09-21T13:55:46.619 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.619 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.619 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:55:46.619 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.011887+0000", 2024-09-21T13:55:46.619 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4830000000000002e-06 2024-09-21T13:55:46.619 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.619 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.619 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:46.619 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.011963+0000", 2024-09-21T13:55:46.619 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5624999999999995e-05 2024-09-21T13:55:46.619 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.619 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.620 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:55:46.620 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.011971+0000", 2024-09-21T13:55:46.620 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.541e-06 2024-09-21T13:55:46.620 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.620 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.620 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:46.620 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.957601+0000", 2024-09-21T13:55:46.620 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.94562941099999998 2024-09-21T13:55:46.620 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.620 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.620 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:55:46.620 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.957730+0000", 2024-09-21T13:55:46.620 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000129016 2024-09-21T13:55:46.620 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.620 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.621 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2", 2024-09-21T13:55:46.621 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.957877+0000", 2024-09-21T13:55:46.621 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000147563 2024-09-21T13:55:46.621 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.621 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.621 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:55:46.621 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.962710+0000", 2024-09-21T13:55:46.621 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.004832355 2024-09-21T13:55:46.621 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.621 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.621 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:46.621 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.965643+0000", 2024-09-21T13:55:46.621 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.002933017 2024-09-21T13:55:46.621 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.621 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.622 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:46.622 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.980821+0000", 2024-09-21T13:55:46.622 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.015178552999999999 2024-09-21T13:55:46.622 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.622 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.622 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:55:46.622 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.980880+0000", 2024-09-21T13:55:46.622 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8646000000000001e-05 2024-09-21T13:55:46.622 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.622 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.622 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:55:46.622 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.980918+0000", 2024-09-21T13:55:46.622 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8111999999999997e-05 2024-09-21T13:55:46.622 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:46.622 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:55:46.623 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:46.623 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.623 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.623 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:201680 7.5 7:ac2e5a95:::benchmark_data_smithi049_76798_object65260:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T13:55:46.623 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:43.053233+0000", 2024-09-21T13:55:46.623 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 183.52462321199999, 2024-09-21T13:55:46.623 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.92777444499999995, 2024-09-21T13:55:46.623 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:55:46.623 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:55:46.623 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:55:46.623 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:55:46.623 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:55:46.623 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:55:46.623 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 201680 2024-09-21T13:55:46.623 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.623 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:55:46.624 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.624 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:55:46.624 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.053233+0000", 2024-09-21T13:55:46.624 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:46.624 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.624 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.624 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:55:46.624 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.053233+0000", 2024-09-21T13:55:46.624 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:46.624 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.624 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.624 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:55:46.624 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.053249+0000", 2024-09-21T13:55:46.624 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.596e-05 2024-09-21T13:55:46.624 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.625 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.625 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:55:46.625 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.053257+0000", 2024-09-21T13:55:46.625 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9980000000000003e-06 2024-09-21T13:55:46.625 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.625 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.625 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:55:46.625 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.053278+0000", 2024-09-21T13:55:46.625 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0749000000000001e-05 2024-09-21T13:55:46.625 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.625 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.625 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:55:46.625 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.053289+0000", 2024-09-21T13:55:46.625 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0864e-05 2024-09-21T13:55:46.625 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.626 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.626 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:46.626 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.053366+0000", 2024-09-21T13:55:46.626 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7082999999999994e-05 2024-09-21T13:55:46.626 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.626 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.626 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:55:46.626 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.053375+0000", 2024-09-21T13:55:46.626 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3230000000000004e-06 2024-09-21T13:55:46.626 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.626 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.626 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:46.626 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.960197+0000", 2024-09-21T13:55:46.626 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.90682242700000004 2024-09-21T13:55:46.626 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.627 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.627 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:55:46.627 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.960297+0000", 2024-09-21T13:55:46.627 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.9817999999999999e-05 2024-09-21T13:55:46.627 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.627 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.627 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2", 2024-09-21T13:55:46.627 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.960410+0000", 2024-09-21T13:55:46.627 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000113315 2024-09-21T13:55:46.627 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.627 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.627 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:55:46.627 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.965602+0000", 2024-09-21T13:55:46.627 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0051915390000000002 2024-09-21T13:55:46.628 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.628 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.628 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:46.628 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.965712+0000", 2024-09-21T13:55:46.628 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010966700000000001 2024-09-21T13:55:46.628 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.628 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.628 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:46.628 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.980954+0000", 2024-09-21T13:55:46.628 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.015242551999999999 2024-09-21T13:55:46.628 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.628 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.628 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:55:46.628 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.980985+0000", 2024-09-21T13:55:46.628 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0686999999999999e-05 2024-09-21T13:55:46.628 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.629 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.629 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:55:46.629 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.981007+0000", 2024-09-21T13:55:46.629 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2463000000000002e-05 2024-09-21T13:55:46.629 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:46.629 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:55:46.629 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:46.629 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.629 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.629 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:201691 7.5 7:a0c84d4e:::benchmark_data_smithi049_76798_object65271:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T13:55:46.629 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:43.063956+0000", 2024-09-21T13:55:46.629 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 183.51389972199999, 2024-09-21T13:55:46.629 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.91993442199999997, 2024-09-21T13:55:46.629 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:55:46.630 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:55:46.630 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:55:46.630 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:55:46.630 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:55:46.630 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:55:46.630 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 201691 2024-09-21T13:55:46.630 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.630 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:55:46.630 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.630 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:55:46.630 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.063956+0000", 2024-09-21T13:55:46.630 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:46.630 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.630 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.630 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:55:46.631 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.063956+0000", 2024-09-21T13:55:46.631 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:46.631 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.631 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.631 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:55:46.631 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.063959+0000", 2024-09-21T13:55:46.631 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.401e-06 2024-09-21T13:55:46.631 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.631 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.631 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:55:46.631 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.063966+0000", 2024-09-21T13:55:46.631 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2479999999999999e-06 2024-09-21T13:55:46.631 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.631 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.631 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:55:46.632 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.063984+0000", 2024-09-21T13:55:46.632 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7530999999999999e-05 2024-09-21T13:55:46.632 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.632 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.632 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:55:46.632 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.063992+0000", 2024-09-21T13:55:46.632 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2509999999999994e-06 2024-09-21T13:55:46.632 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.632 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.632 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:46.632 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.064060+0000", 2024-09-21T13:55:46.632 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8535999999999996e-05 2024-09-21T13:55:46.632 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.632 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.632 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:55:46.633 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.064070+0000", 2024-09-21T13:55:46.633 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6830000000000003e-06 2024-09-21T13:55:46.633 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.633 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.633 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:46.633 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.961470+0000", 2024-09-21T13:55:46.633 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.89740008599999999 2024-09-21T13:55:46.633 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.633 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.633 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:55:46.633 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.961570+0000", 2024-09-21T13:55:46.633 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000100237 2024-09-21T13:55:46.633 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.633 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.633 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2", 2024-09-21T13:55:46.634 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.961689+0000", 2024-09-21T13:55:46.634 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000118347 2024-09-21T13:55:46.634 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.634 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.634 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:55:46.634 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.965656+0000", 2024-09-21T13:55:46.634 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0039668029999999996 2024-09-21T13:55:46.634 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.634 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:46.635 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:46.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:55:46.636 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:46.636 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.979772+0000", 2024-09-21T13:55:46.636 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.014116717000000001 2024-09-21T13:55:46.636 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.636 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.636 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:46.636 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.983830+0000", 2024-09-21T13:55:46.636 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0040575189999999999 2024-09-21T13:55:46.636 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.636 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.636 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:55:46.636 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.983868+0000", 2024-09-21T13:55:46.637 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8586e-05 2024-09-21T13:55:46.637 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.637 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.637 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:55:46.637 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.983891+0000", 2024-09-21T13:55:46.637 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2476999999999998e-05 2024-09-21T13:55:46.637 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:46.637 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:55:46.637 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:46.637 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.637 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.637 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:201695 7.5 7:a9f46a76:::benchmark_data_smithi049_76798_object65275:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T13:55:46.637 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:43.066285+0000", 2024-09-21T13:55:46.637 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 183.51157159499999, 2024-09-21T13:55:46.638 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.92214624000000001, 2024-09-21T13:55:46.638 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:55:46.638 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:55:46.638 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:55:46.638 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:55:46.638 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:55:46.638 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:55:46.638 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 201695 2024-09-21T13:55:46.638 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.638 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:55:46.638 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.638 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:55:46.638 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.066285+0000", 2024-09-21T13:55:46.638 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:46.638 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.639 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.639 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:55:46.639 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.066285+0000", 2024-09-21T13:55:46.639 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:46.639 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.639 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.639 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:55:46.639 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.066287+0000", 2024-09-21T13:55:46.639 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0219999999999999e-06 2024-09-21T13:55:46.639 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.639 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.639 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:55:46.639 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.066295+0000", 2024-09-21T13:55:46.640 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1999999999999994e-06 2024-09-21T13:55:46.640 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.640 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.640 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:55:46.640 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.066316+0000", 2024-09-21T13:55:46.640 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1508000000000001e-05 2024-09-21T13:55:46.640 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.640 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.640 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:55:46.640 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.066326+0000", 2024-09-21T13:55:46.640 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.4290000000000007e-06 2024-09-21T13:55:46.640 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.640 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.640 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:46.640 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.066400+0000", 2024-09-21T13:55:46.641 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4339999999999996e-05 2024-09-21T13:55:46.641 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.641 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.641 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:55:46.641 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.066410+0000", 2024-09-21T13:55:46.641 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.5799999999999998e-06 2024-09-21T13:55:46.641 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.641 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.641 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:46.641 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.962805+0000", 2024-09-21T13:55:46.641 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.89639528199999996 2024-09-21T13:55:46.641 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.641 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.641 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:55:46.641 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.962880+0000", 2024-09-21T13:55:46.642 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5139e-05 2024-09-21T13:55:46.642 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.642 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.642 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2", 2024-09-21T13:55:46.642 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.962949+0000", 2024-09-21T13:55:46.642 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9098999999999997e-05 2024-09-21T13:55:46.642 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.642 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.642 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:55:46.642 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.965662+0000", 2024-09-21T13:55:46.642 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0027127259999999999 2024-09-21T13:55:46.642 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.642 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.643 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:46.643 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.983932+0000", 2024-09-21T13:55:46.643 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.018270237000000002 2024-09-21T13:55:46.643 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.643 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.643 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:46.643 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.988374+0000", 2024-09-21T13:55:46.643 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0044419239999999999 2024-09-21T13:55:46.643 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.643 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.643 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:55:46.643 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.988406+0000", 2024-09-21T13:55:46.643 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2172000000000003e-05 2024-09-21T13:55:46.643 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.644 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.644 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:55:46.644 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.988431+0000", 2024-09-21T13:55:46.644 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4581999999999999e-05 2024-09-21T13:55:46.644 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:46.644 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:55:46.644 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:46.644 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.644 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.644 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:201698 7.5 7:ae6dee09:::benchmark_data_smithi049_76798_object65278:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T13:55:46.644 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:43.068553+0000", 2024-09-21T13:55:46.644 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 183.509303123, 2024-09-21T13:55:46.644 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.91996031099999998, 2024-09-21T13:55:46.644 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:55:46.644 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:55:46.645 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:55:46.645 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:55:46.645 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:55:46.645 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:55:46.645 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 201698 2024-09-21T13:55:46.645 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.645 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:55:46.645 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.645 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:55:46.645 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.068553+0000", 2024-09-21T13:55:46.645 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:46.645 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.645 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.646 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:55:46.646 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.068553+0000", 2024-09-21T13:55:46.646 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:46.646 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.646 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.646 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:55:46.646 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.068555+0000", 2024-09-21T13:55:46.646 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2840000000000001e-06 2024-09-21T13:55:46.646 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.646 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.646 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:55:46.646 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.068563+0000", 2024-09-21T13:55:46.646 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6899999999999992e-06 2024-09-21T13:55:46.646 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.647 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.647 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:55:46.647 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.068582+0000", 2024-09-21T13:55:46.647 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8902e-05 2024-09-21T13:55:46.647 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.647 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.647 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:55:46.647 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.068591+0000", 2024-09-21T13:55:46.647 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.6440000000000006e-06 2024-09-21T13:55:46.647 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.647 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.647 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:46.647 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.068654+0000", 2024-09-21T13:55:46.647 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.347e-05 2024-09-21T13:55:46.648 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.648 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.648 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:55:46.648 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.068662+0000", 2024-09-21T13:55:46.648 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1829999999999994e-06 2024-09-21T13:55:46.648 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.648 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.648 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:46.648 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.964312+0000", 2024-09-21T13:55:46.648 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.89564967699999998 2024-09-21T13:55:46.648 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.648 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.648 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:55:46.648 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.964413+0000", 2024-09-21T13:55:46.648 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010090399999999999 2024-09-21T13:55:46.648 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.648 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.649 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2", 2024-09-21T13:55:46.649 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.964529+0000", 2024-09-21T13:55:46.649 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000116579 2024-09-21T13:55:46.649 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.649 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.649 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:55:46.649 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.970760+0000", 2024-09-21T13:55:46.649 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0062309820000000004 2024-09-21T13:55:46.649 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.649 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.649 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:46.649 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.983968+0000", 2024-09-21T13:55:46.649 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.013207727000000001 2024-09-21T13:55:46.649 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.650 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.650 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:46.650 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.988459+0000", 2024-09-21T13:55:46.650 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0044908889999999996 2024-09-21T13:55:46.650 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.650 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.650 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:55:46.650 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.988489+0000", 2024-09-21T13:55:46.650 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0063e-05 2024-09-21T13:55:46.650 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.650 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.650 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:55:46.650 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.988513+0000", 2024-09-21T13:55:46.651 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4317000000000001e-05 2024-09-21T13:55:46.651 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:46.651 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:55:46.651 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:46.651 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.651 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.651 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:201701 7.5 7:a7fade2d:::benchmark_data_smithi049_76798_object65281:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T13:55:46.651 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:43.071918+0000", 2024-09-21T13:55:46.651 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 183.50593785699999, 2024-09-21T13:55:46.651 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.91666014399999995, 2024-09-21T13:55:46.651 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:55:46.651 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:55:46.651 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:55:46.651 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:55:46.652 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:55:46.652 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:55:46.652 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 201701 2024-09-21T13:55:46.652 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.652 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:55:46.652 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.652 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:55:46.652 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.071918+0000", 2024-09-21T13:55:46.652 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:46.652 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.652 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.652 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:55:46.652 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.071918+0000", 2024-09-21T13:55:46.652 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:46.652 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.653 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.653 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:55:46.653 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.071920+0000", 2024-09-21T13:55:46.653 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3340000000000001e-06 2024-09-21T13:55:46.653 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.653 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.653 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:55:46.653 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.071926+0000", 2024-09-21T13:55:46.653 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8719999999999999e-06 2024-09-21T13:55:46.653 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.653 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.653 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:55:46.653 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.071941+0000", 2024-09-21T13:55:46.653 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5529999999999999e-05 2024-09-21T13:55:46.653 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.653 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.654 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:55:46.654 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.071949+0000", 2024-09-21T13:55:46.654 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1000000000000004e-06 2024-09-21T13:55:46.654 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.654 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.654 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:46.654 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.072014+0000", 2024-09-21T13:55:46.654 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4912000000000005e-05 2024-09-21T13:55:46.654 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.654 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.654 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:55:46.654 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.072023+0000", 2024-09-21T13:55:46.654 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7069999999999998e-06 2024-09-21T13:55:46.654 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.655 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.655 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:46.655 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.965753+0000", 2024-09-21T13:55:46.655 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.89372980199999996 2024-09-21T13:55:46.655 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.655 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.655 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:55:46.655 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.965886+0000", 2024-09-21T13:55:46.655 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000133669 2024-09-21T13:55:46.655 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.655 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.655 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2", 2024-09-21T13:55:46.655 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.966006+0000", 2024-09-21T13:55:46.655 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000119407 2024-09-21T13:55:46.656 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.656 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.656 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:55:46.656 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.973164+0000", 2024-09-21T13:55:46.656 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.007158569 2024-09-21T13:55:46.656 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.656 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.656 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:46.656 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.984364+0000", 2024-09-21T13:55:46.656 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.011199944 2024-09-21T13:55:46.656 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.656 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.656 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:46.656 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.988535+0000", 2024-09-21T13:55:46.656 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0041708800000000001 2024-09-21T13:55:46.657 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.657 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.657 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:55:46.657 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.988560+0000", 2024-09-21T13:55:46.657 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4879e-05 2024-09-21T13:55:46.657 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.657 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.657 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:55:46.657 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.988578+0000", 2024-09-21T13:55:46.657 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8539e-05 2024-09-21T13:55:46.657 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:46.657 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:55:46.657 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:46.657 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.658 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.658 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:201772 7.5 7:a814840e:::benchmark_data_smithi049_76798_object65352:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T13:55:46.658 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:43.156794+0000", 2024-09-21T13:55:46.658 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 183.421062449, 2024-09-21T13:55:46.658 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.83184825500000004, 2024-09-21T13:55:46.658 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:55:46.658 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:55:46.658 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:55:46.658 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:55:46.658 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:55:46.658 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:55:46.658 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 201772 2024-09-21T13:55:46.658 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.658 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:55:46.658 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.659 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:55:46.659 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.156794+0000", 2024-09-21T13:55:46.659 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:46.659 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.659 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.659 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:55:46.659 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.156794+0000", 2024-09-21T13:55:46.659 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:46.659 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.659 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.659 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:55:46.659 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.156795+0000", 2024-09-21T13:55:46.659 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4839999999999999e-06 2024-09-21T13:55:46.659 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.660 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.660 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:55:46.660 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.156801+0000", 2024-09-21T13:55:46.660 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5709999999999997e-06 2024-09-21T13:55:46.660 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.660 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.660 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:55:46.660 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.156817+0000", 2024-09-21T13:55:46.660 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6378000000000001e-05 2024-09-21T13:55:46.660 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.660 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.660 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:55:46.660 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.156826+0000", 2024-09-21T13:55:46.660 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4060000000000005e-06 2024-09-21T13:55:46.660 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.660 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.661 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:46.661 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.156889+0000", 2024-09-21T13:55:46.661 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3396000000000001e-05 2024-09-21T13:55:46.661 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.661 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.661 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:55:46.661 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.156895+0000", 2024-09-21T13:55:46.661 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7699999999999998e-06 2024-09-21T13:55:46.661 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.661 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.661 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:46.661 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.967038+0000", 2024-09-21T13:55:46.661 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.81014350400000001 2024-09-21T13:55:46.662 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.662 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.662 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:55:46.662 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.967105+0000", 2024-09-21T13:55:46.662 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6561000000000002e-05 2024-09-21T13:55:46.662 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.662 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.662 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2", 2024-09-21T13:55:46.662 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.967172+0000", 2024-09-21T13:55:46.663 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7336999999999996e-05 2024-09-21T13:55:46.663 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.663 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.663 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:55:46.663 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.973169+0000", 2024-09-21T13:55:46.663 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.005996927 2024-09-21T13:55:46.663 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.663 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.663 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:46.663 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.984408+0000", 2024-09-21T13:55:46.663 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.011238823 2024-09-21T13:55:46.663 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.664 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.664 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:46.664 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.988599+0000", 2024-09-21T13:55:46.664 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0041907630000000001 2024-09-21T13:55:46.664 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.664 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.664 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:55:46.664 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.988624+0000", 2024-09-21T13:55:46.664 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5148999999999999e-05 2024-09-21T13:55:46.664 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.664 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.664 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:55:46.665 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.988642+0000", 2024-09-21T13:55:46.665 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8185999999999998e-05 2024-09-21T13:55:46.665 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:46.665 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:55:46.665 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:46.665 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.665 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.665 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:201787 7.5 7:a2e71a94:::benchmark_data_smithi049_76798_object65367:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T13:55:46.665 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:43.173880+0000", 2024-09-21T13:55:46.665 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 183.403976473, 2024-09-21T13:55:46.665 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.81499015100000005, 2024-09-21T13:55:46.665 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:55:46.665 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:55:46.665 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:55:46.665 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:55:46.666 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:55:46.666 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:55:46.666 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 201787 2024-09-21T13:55:46.666 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.666 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:55:46.666 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.666 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:55:46.666 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.173880+0000", 2024-09-21T13:55:46.666 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:46.666 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.666 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.666 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:55:46.666 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.173880+0000", 2024-09-21T13:55:46.666 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:46.666 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.667 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.667 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:55:46.667 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.173882+0000", 2024-09-21T13:55:46.667 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1500000000000002e-06 2024-09-21T13:55:46.667 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.667 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.667 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:55:46.667 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.173888+0000", 2024-09-21T13:55:46.667 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3799999999999999e-06 2024-09-21T13:55:46.667 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.667 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.667 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:55:46.667 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.173902+0000", 2024-09-21T13:55:46.667 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4225e-05 2024-09-21T13:55:46.667 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.668 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.668 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:55:46.668 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.173911+0000", 2024-09-21T13:55:46.668 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0649999999999999e-06 2024-09-21T13:55:46.668 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.668 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.668 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:46.668 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.173972+0000", 2024-09-21T13:55:46.668 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1168999999999997e-05 2024-09-21T13:55:46.668 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.668 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.668 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:55:46.668 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.173980+0000", 2024-09-21T13:55:46.668 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2549999999999997e-06 2024-09-21T13:55:46.668 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.669 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.669 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:46.669 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.968330+0000", 2024-09-21T13:55:46.669 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.79435025800000003 2024-09-21T13:55:46.669 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.669 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.669 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:55:46.669 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.968390+0000", 2024-09-21T13:55:46.669 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0149999999999998e-05 2024-09-21T13:55:46.669 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.669 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.669 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2", 2024-09-21T13:55:46.669 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.968453+0000", 2024-09-21T13:55:46.669 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2427000000000004e-05 2024-09-21T13:55:46.669 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.669 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.670 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:55:46.670 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.973174+0000", 2024-09-21T13:55:46.670 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0047213740000000004 2024-09-21T13:55:46.670 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.670 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.670 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:46.670 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.984437+0000", 2024-09-21T13:55:46.670 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.011262507 2024-09-21T13:55:46.670 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.670 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.670 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:46.670 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.988823+0000", 2024-09-21T13:55:46.670 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0043867799999999998 2024-09-21T13:55:46.670 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.670 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.671 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:55:46.671 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.988853+0000", 2024-09-21T13:55:46.671 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9116e-05 2024-09-21T13:55:46.671 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.671 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.671 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:55:46.671 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.988870+0000", 2024-09-21T13:55:46.671 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7295e-05 2024-09-21T13:55:46.671 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:46.671 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:55:46.671 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:46.671 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.671 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.672 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:201826 7.5 7:a523bf31:::benchmark_data_smithi049_76798_object65406:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T13:55:46.672 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:43.212124+0000", 2024-09-21T13:55:46.672 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 183.36573234400001, 2024-09-21T13:55:46.672 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.77941543700000004, 2024-09-21T13:55:46.672 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:55:46.672 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:55:46.672 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:55:46.672 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:55:46.672 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:55:46.672 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:55:46.672 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 201826 2024-09-21T13:55:46.672 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.672 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:55:46.673 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.673 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:55:46.673 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.212124+0000", 2024-09-21T13:55:46.673 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:46.673 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.673 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.673 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:55:46.673 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.212124+0000", 2024-09-21T13:55:46.673 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:46.673 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.673 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.673 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:55:46.673 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.212125+0000", 2024-09-21T13:55:46.673 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3200000000000001e-06 2024-09-21T13:55:46.673 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.674 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.674 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:55:46.674 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.212130+0000", 2024-09-21T13:55:46.674 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5210000000000004e-06 2024-09-21T13:55:46.674 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.674 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.674 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:55:46.674 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.212142+0000", 2024-09-21T13:55:46.674 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1878e-05 2024-09-21T13:55:46.674 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.674 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.674 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:55:46.674 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.212147+0000", 2024-09-21T13:55:46.675 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3410000000000002e-06 2024-09-21T13:55:46.675 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.675 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.675 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:46.675 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.212194+0000", 2024-09-21T13:55:46.675 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7091e-05 2024-09-21T13:55:46.675 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.675 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.675 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:55:46.675 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.212200+0000", 2024-09-21T13:55:46.675 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0440000000000001e-06 2024-09-21T13:55:46.675 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.675 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.675 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:46.676 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.969460+0000", 2024-09-21T13:55:46.676 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.75725952399999996 2024-09-21T13:55:46.676 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.676 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.676 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:55:46.676 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.969570+0000", 2024-09-21T13:55:46.676 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000110876 2024-09-21T13:55:46.676 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.676 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.676 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2", 2024-09-21T13:55:46.676 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.969687+0000", 2024-09-21T13:55:46.676 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011697 2024-09-21T13:55:46.676 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.677 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.677 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:55:46.677 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.974455+0000", 2024-09-21T13:55:46.677 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0047673259999999997 2024-09-21T13:55:46.677 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.677 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.677 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:46.677 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.990260+0000", 2024-09-21T13:55:46.677 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.015805567999999999 2024-09-21T13:55:46.677 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.677 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.677 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:46.677 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.991462+0000", 2024-09-21T13:55:46.678 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0012021340000000001 2024-09-21T13:55:46.678 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.678 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.678 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:55:46.678 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.991504+0000", 2024-09-21T13:55:46.678 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1715000000000003e-05 2024-09-21T13:55:46.678 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.678 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.678 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:55:46.678 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.991539+0000", 2024-09-21T13:55:46.678 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5129000000000002e-05 2024-09-21T13:55:46.678 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:46.678 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:55:46.678 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:46.679 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.679 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.679 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:201840 7.5 7:abeb3043:::benchmark_data_smithi049_76798_object65420:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T13:55:46.679 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:43.224752+0000", 2024-09-21T13:55:46.679 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 183.35310404399999, 2024-09-21T13:55:46.679 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.76862761999999996, 2024-09-21T13:55:46.679 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:55:46.679 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:55:46.679 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:55:46.679 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:55:46.679 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:55:46.679 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:55:46.680 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 201840 2024-09-21T13:55:46.680 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.680 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:55:46.680 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.680 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:55:46.680 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.224752+0000", 2024-09-21T13:55:46.680 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:46.680 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.680 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.680 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:55:46.680 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.224752+0000", 2024-09-21T13:55:46.680 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:46.680 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.681 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.681 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:55:46.681 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.224753+0000", 2024-09-21T13:55:46.681 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.8299999999999995e-07 2024-09-21T13:55:46.681 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.681 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.681 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:55:46.681 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.224757+0000", 2024-09-21T13:55:46.681 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2379999999999999e-06 2024-09-21T13:55:46.681 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.681 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.681 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:55:46.681 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.224768+0000", 2024-09-21T13:55:46.681 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0390999999999999e-05 2024-09-21T13:55:46.681 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.682 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.682 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:55:46.682 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.224775+0000", 2024-09-21T13:55:46.682 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9999999999999999e-06 2024-09-21T13:55:46.682 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.682 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.682 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:46.682 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.224823+0000", 2024-09-21T13:55:46.682 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8186999999999998e-05 2024-09-21T13:55:46.682 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.682 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.682 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:55:46.682 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.224830+0000", 2024-09-21T13:55:46.683 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6100000000000002e-06 2024-09-21T13:55:46.683 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.683 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.683 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:46.683 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.970787+0000", 2024-09-21T13:55:46.683 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.74595736499999998 2024-09-21T13:55:46.683 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.683 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.683 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:55:46.683 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.970866+0000", 2024-09-21T13:55:46.683 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8731000000000002e-05 2024-09-21T13:55:46.683 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.683 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.683 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2", 2024-09-21T13:55:46.684 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.970932+0000", 2024-09-21T13:55:46.684 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6031000000000005e-05 2024-09-21T13:55:46.684 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.684 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.684 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:55:46.684 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.974460+0000", 2024-09-21T13:55:46.684 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0035279069999999998 2024-09-21T13:55:46.684 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.684 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.684 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:46.684 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.991576+0000", 2024-09-21T13:55:46.684 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.017116833000000001 2024-09-21T13:55:46.684 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.684 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.684 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:46.685 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.993291+0000", 2024-09-21T13:55:46.685 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0017143690000000001 2024-09-21T13:55:46.685 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.685 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.685 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:55:46.685 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.993342+0000", 2024-09-21T13:55:46.685 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1316000000000001e-05 2024-09-21T13:55:46.685 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.685 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.685 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:55:46.685 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.993380+0000", 2024-09-21T13:55:46.685 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7659e-05 2024-09-21T13:55:46.685 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:46.686 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:55:46.686 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:46.686 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.686 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.686 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:201850 7.5 7:adb529a4:::benchmark_data_smithi049_76798_object65430:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T13:55:46.686 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:43.237139+0000", 2024-09-21T13:55:46.686 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 183.340717337, 2024-09-21T13:55:46.686 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.76209282499999997, 2024-09-21T13:55:46.686 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:55:46.686 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:55:46.686 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:55:46.686 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:55:46.686 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:55:46.686 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:55:46.687 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 201850 2024-09-21T13:55:46.687 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.687 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:55:46.687 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.687 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:55:46.687 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.237139+0000", 2024-09-21T13:55:46.687 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:46.687 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.687 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.687 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:55:46.687 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.237139+0000", 2024-09-21T13:55:46.687 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:46.687 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.687 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.688 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:55:46.688 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.237139+0000", 2024-09-21T13:55:46.688 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5799999999999999e-07 2024-09-21T13:55:46.688 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.688 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.688 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:55:46.688 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.237143+0000", 2024-09-21T13:55:46.688 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5310000000000002e-06 2024-09-21T13:55:46.688 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.688 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.688 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:55:46.688 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.237152+0000", 2024-09-21T13:55:46.688 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.653e-06 2024-09-21T13:55:46.688 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.689 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.689 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:55:46.689 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.237157+0000", 2024-09-21T13:55:46.689 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8239999999999999e-06 2024-09-21T13:55:46.689 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.689 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.689 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:46.689 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.237205+0000", 2024-09-21T13:55:46.689 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8341999999999999e-05 2024-09-21T13:55:46.689 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.689 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.689 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:55:46.689 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.237212+0000", 2024-09-21T13:55:46.689 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6410000000000002e-06 2024-09-21T13:55:46.689 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.690 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.690 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:46.690 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.971939+0000", 2024-09-21T13:55:46.690 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.73472659200000001 2024-09-21T13:55:46.690 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.690 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.690 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:55:46.690 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.972042+0000", 2024-09-21T13:55:46.690 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000103049 2024-09-21T13:55:46.690 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.690 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.690 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2", 2024-09-21T13:55:46.690 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.972158+0000", 2024-09-21T13:55:46.690 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011618 2024-09-21T13:55:46.691 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.691 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.691 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:55:46.691 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.975735+0000", 2024-09-21T13:55:46.691 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0035764249999999998 2024-09-21T13:55:46.691 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.691 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.691 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:46.691 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.991609+0000", 2024-09-21T13:55:46.691 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.015874307000000001 2024-09-21T13:55:46.691 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.691 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.691 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:46.691 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.999171+0000", 2024-09-21T13:55:46.691 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0075622440000000001 2024-09-21T13:55:46.692 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.692 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.692 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:55:46.692 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.999201+0000", 2024-09-21T13:55:46.692 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9495000000000001e-05 2024-09-21T13:55:46.692 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.692 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.692 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:55:46.692 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.999232+0000", 2024-09-21T13:55:46.692 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0883999999999998e-05 2024-09-21T13:55:46.692 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:46.692 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:55:46.692 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:46.692 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.693 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.693 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:201852 7.5 7:a4c0c63b:::benchmark_data_smithi049_76798_object65432:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T13:55:46.693 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:43.239197+0000", 2024-09-21T13:55:46.693 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 183.33865904699999, 2024-09-21T13:55:46.693 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.76675081499999997, 2024-09-21T13:55:46.693 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:55:46.693 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:55:46.693 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:55:46.693 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:55:46.693 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:55:46.693 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:55:46.693 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 201852 2024-09-21T13:55:46.693 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.694 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:55:46.694 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.694 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:55:46.694 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.239197+0000", 2024-09-21T13:55:46.694 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:46.694 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.694 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.694 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:55:46.694 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.239197+0000", 2024-09-21T13:55:46.694 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:46.694 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.694 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.694 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:55:46.695 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.239198+0000", 2024-09-21T13:55:46.695 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6899999999999996e-07 2024-09-21T13:55:46.695 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.695 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.695 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:55:46.695 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.239202+0000", 2024-09-21T13:55:46.695 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8340000000000001e-06 2024-09-21T13:55:46.695 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.695 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.695 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:55:46.695 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.239212+0000", 2024-09-21T13:55:46.695 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0563e-05 2024-09-21T13:55:46.695 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.695 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.696 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:55:46.696 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.239219+0000", 2024-09-21T13:55:46.696 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.241e-06 2024-09-21T13:55:46.696 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.696 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.696 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:46.696 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.239260+0000", 2024-09-21T13:55:46.696 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1421999999999997e-05 2024-09-21T13:55:46.696 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.696 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.696 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:55:46.696 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.239265+0000", 2024-09-21T13:55:46.696 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5399999999999997e-06 2024-09-21T13:55:46.696 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.697 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.697 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:46.697 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.973203+0000", 2024-09-21T13:55:46.697 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.73393844600000002 2024-09-21T13:55:46.697 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.697 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.697 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:55:46.697 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.973305+0000", 2024-09-21T13:55:46.697 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010141699999999999 2024-09-21T13:55:46.697 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.697 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.697 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2", 2024-09-21T13:55:46.697 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.973424+0000", 2024-09-21T13:55:46.697 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011916700000000001 2024-09-21T13:55:46.697 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.698 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.698 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:55:46.698 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.977041+0000", 2024-09-21T13:55:46.698 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0036168989999999998 2024-09-21T13:55:46.698 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.698 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.698 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:46.698 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.996582+0000", 2024-09-21T13:55:46.698 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.019541097 2024-09-21T13:55:46.698 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.698 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.698 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:46.698 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.005893+0000", 2024-09-21T13:55:46.698 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0093108410000000003 2024-09-21T13:55:46.699 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.699 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.699 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:55:46.699 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.005925+0000", 2024-09-21T13:55:46.699 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2490000000000002e-05 2024-09-21T13:55:46.699 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.699 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.699 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:55:46.699 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.005948+0000", 2024-09-21T13:55:46.699 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2889000000000001e-05 2024-09-21T13:55:46.699 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:46.699 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:55:46.699 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:46.699 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.699 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.700 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:201856 7.5 7:a20bd2e0:::benchmark_data_smithi049_76798_object65436:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T13:55:46.700 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:43.241762+0000", 2024-09-21T13:55:46.700 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 183.33609372500001, 2024-09-21T13:55:46.700 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.76427266500000002, 2024-09-21T13:55:46.700 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:55:46.700 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:55:46.700 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:55:46.700 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:55:46.700 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:55:46.700 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:55:46.700 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 201856 2024-09-21T13:55:46.700 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.700 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:55:46.700 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.700 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:55:46.701 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.241762+0000", 2024-09-21T13:55:46.701 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:46.701 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.701 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.701 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:55:46.701 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.241762+0000", 2024-09-21T13:55:46.701 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:46.701 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.701 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.701 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:55:46.701 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.241763+0000", 2024-09-21T13:55:46.701 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.2200000000000002e-07 2024-09-21T13:55:46.701 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.701 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.701 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:55:46.702 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.241768+0000", 2024-09-21T13:55:46.702 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.357e-06 2024-09-21T13:55:46.702 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.702 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.702 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:55:46.702 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.241780+0000", 2024-09-21T13:55:46.702 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.224e-05 2024-09-21T13:55:46.702 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.702 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.702 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:55:46.702 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.241787+0000", 2024-09-21T13:55:46.702 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4710000000000001e-06 2024-09-21T13:55:46.702 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.702 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.703 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:46.703 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.241830+0000", 2024-09-21T13:55:46.703 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2960000000000002e-05 2024-09-21T13:55:46.703 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.703 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.703 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:55:46.703 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.241835+0000", 2024-09-21T13:55:46.703 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8110000000000002e-06 2024-09-21T13:55:46.703 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.703 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.703 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:46.704 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.974488+0000", 2024-09-21T13:55:46.704 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.73265256999999995 2024-09-21T13:55:46.704 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.704 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.704 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:55:46.704 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.974586+0000", 2024-09-21T13:55:46.704 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.8669e-05 2024-09-21T13:55:46.704 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.704 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.704 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2", 2024-09-21T13:55:46.704 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.974702+0000", 2024-09-21T13:55:46.704 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000115183 2024-09-21T13:55:46.705 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.705 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.705 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:55:46.705 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.979708+0000", 2024-09-21T13:55:46.705 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0050061339999999998 2024-09-21T13:55:46.705 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.705 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.705 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:46.705 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.997392+0000", 2024-09-21T13:55:46.705 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.017684388999999998 2024-09-21T13:55:46.705 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.705 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.705 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:46.705 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.005982+0000", 2024-09-21T13:55:46.706 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.008590192 2024-09-21T13:55:46.706 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.706 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.706 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:55:46.706 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.006012+0000", 2024-09-21T13:55:46.706 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9714e-05 2024-09-21T13:55:46.706 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.706 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.706 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:55:46.706 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.006035+0000", 2024-09-21T13:55:46.706 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3053000000000001e-05 2024-09-21T13:55:46.706 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:46.706 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:55:46.706 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:46.706 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.707 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.707 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:201866 7.5 7:ab0bbb06:::benchmark_data_smithi049_76798_object65446:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T13:55:46.707 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:43.261703+0000", 2024-09-21T13:55:46.707 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 183.31615317000001, 2024-09-21T13:55:46.707 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.744403649, 2024-09-21T13:55:46.707 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:55:46.707 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:55:46.707 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:55:46.707 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:55:46.707 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:55:46.707 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:55:46.707 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 201866 2024-09-21T13:55:46.707 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.707 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:55:46.708 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.708 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:55:46.708 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.261703+0000", 2024-09-21T13:55:46.708 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:46.708 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.708 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.708 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:55:46.708 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.261703+0000", 2024-09-21T13:55:46.708 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:46.708 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.708 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.708 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:55:46.708 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.261705+0000", 2024-09-21T13:55:46.708 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7290000000000001e-06 2024-09-21T13:55:46.708 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.709 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.709 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:55:46.709 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.261711+0000", 2024-09-21T13:55:46.709 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2319999999999998e-06 2024-09-21T13:55:46.709 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.709 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.709 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:55:46.709 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.261724+0000", 2024-09-21T13:55:46.709 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3375e-05 2024-09-21T13:55:46.709 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.709 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.709 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:55:46.709 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.261731+0000", 2024-09-21T13:55:46.709 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3319999999999997e-06 2024-09-21T13:55:46.709 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.710 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.710 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:46.710 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.261790+0000", 2024-09-21T13:55:46.710 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9110999999999997e-05 2024-09-21T13:55:46.710 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.710 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.710 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:55:46.710 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.261797+0000", 2024-09-21T13:55:46.710 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9739999999999996e-06 2024-09-21T13:55:46.710 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.710 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.710 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:46.710 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.975764+0000", 2024-09-21T13:55:46.710 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.71396768300000002 2024-09-21T13:55:46.710 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.711 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.711 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:55:46.711 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.975882+0000", 2024-09-21T13:55:46.711 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011723899999999999 2024-09-21T13:55:46.711 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.711 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.711 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2", 2024-09-21T13:55:46.711 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.976017+0000", 2024-09-21T13:55:46.711 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013532799999999999 2024-09-21T13:55:46.711 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.711 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.711 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:55:46.711 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.979731+0000", 2024-09-21T13:55:46.711 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0037143509999999999 2024-09-21T13:55:46.711 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.712 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.712 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:46.712 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.997419+0000", 2024-09-21T13:55:46.712 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.017688021000000002 2024-09-21T13:55:46.712 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.712 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.712 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:46.712 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.006063+0000", 2024-09-21T13:55:46.712 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0086435980000000006 2024-09-21T13:55:46.712 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.712 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.712 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:55:46.712 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.006088+0000", 2024-09-21T13:55:46.712 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5437000000000001e-05 2024-09-21T13:55:46.712 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.713 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.713 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:55:46.713 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.006107+0000", 2024-09-21T13:55:46.713 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8238999999999999e-05 2024-09-21T13:55:46.713 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:46.713 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:55:46.713 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:46.713 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.713 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.713 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:201885 7.5 7:a6d798b1:::benchmark_data_smithi049_76798_object65465:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T13:55:46.713 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:43.302360+0000", 2024-09-21T13:55:46.713 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 183.27549659600001, 2024-09-21T13:55:46.713 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.70381638199999996, 2024-09-21T13:55:46.713 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:55:46.714 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:55:46.714 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:55:46.714 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:55:46.714 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:55:46.714 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:55:46.714 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 201885 2024-09-21T13:55:46.714 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.714 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:55:46.714 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.714 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:55:46.714 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.302360+0000", 2024-09-21T13:55:46.714 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:46.714 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.715 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.715 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:55:46.715 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.302360+0000", 2024-09-21T13:55:46.715 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:55:46.715 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.715 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.715 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:55:46.715 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.302361+0000", 2024-09-21T13:55:46.715 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.291e-06 2024-09-21T13:55:46.715 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.715 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.715 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:55:46.715 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.302366+0000", 2024-09-21T13:55:46.715 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1619999999999999e-06 2024-09-21T13:55:46.715 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.716 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.716 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:55:46.716 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.302379+0000", 2024-09-21T13:55:46.716 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2824e-05 2024-09-21T13:55:46.716 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.716 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.716 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:55:46.716 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.302385+0000", 2024-09-21T13:55:46.716 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8799999999999996e-06 2024-09-21T13:55:46.716 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.716 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.716 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:46.716 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.302431+0000", 2024-09-21T13:55:46.716 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6733000000000001e-05 2024-09-21T13:55:46.716 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.717 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.717 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:55:46.717 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.302437+0000", 2024-09-21T13:55:46.717 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0170000000000002e-06 2024-09-21T13:55:46.717 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.717 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.717 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:55:46.717 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.977079+0000", 2024-09-21T13:55:46.717 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.67464170999999995 2024-09-21T13:55:46.717 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.717 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.717 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:55:46.717 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.977147+0000", 2024-09-21T13:55:46.717 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8099e-05 2024-09-21T13:55:46.718 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.718 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.718 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2", 2024-09-21T13:55:46.718 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.977215+0000", 2024-09-21T13:55:46.718 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8151000000000005e-05 2024-09-21T13:55:46.718 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.718 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.718 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:55:46.718 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.979736+0000", 2024-09-21T13:55:46.718 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.002520361 2024-09-21T13:55:46.718 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.718 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.718 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:46.718 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.000408+0000", 2024-09-21T13:55:46.718 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.020671948999999998 2024-09-21T13:55:46.719 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.719 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.719 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:55:46.719 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.006133+0000", 2024-09-21T13:55:46.719 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0057253149999999999 2024-09-21T13:55:46.719 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.719 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.719 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:55:46.719 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.006158+0000", 2024-09-21T13:55:46.719 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4943000000000002e-05 2024-09-21T13:55:46.719 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:55:46.719 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:55:46.719 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:55:46.719 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.006176+0000", 2024-09-21T13:55:46.719 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7946999999999999e-05 2024-09-21T13:55:46.720 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:46.720 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:55:46.720 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:46.720 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:55:46.720 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:55:46.720 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T13:55:46.736 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:55:46.772 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:46.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:46.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:55:46.874 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:55:46.909 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:46.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:46.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:55:47.011 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:55:47.046 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:47.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:47.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:55:47.148 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:55:47.177 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:47.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:47.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:55:47.279 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:47.301 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 89 16 125340 125324 87.9922 68.1875 0.00530839 0.0113599 2024-09-21T13:55:47.313 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:47.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:47.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:47.415 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:55:47.448 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:47.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:47.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:55:47.550 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:47.589 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:47.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:47.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:47.691 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:55:47.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:47 smithi187 ceph-mon[33032]: pgmap v334: 148 pgs: 1 active+clean+remapped, 38 active+undersized, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 30 active+undersized+degraded, 75 active+clean; 7.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 383 B/s rd, 59 MiB/s wr, 948 op/s; 45642/364185 objects degraded (12.533%); 33156/364185 objects misplaced (9.104%); 42 MiB/s, 678 objects/s recovering 2024-09-21T13:55:47.728 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:47.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:47.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:55:47.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:47 smithi049 ceph-mon[40489]: pgmap v334: 148 pgs: 1 active+clean+remapped, 38 active+undersized, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 30 active+undersized+degraded, 75 active+clean; 7.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 383 B/s rd, 59 MiB/s wr, 948 op/s; 45642/364185 objects degraded (12.533%); 33156/364185 objects misplaced (9.104%); 42 MiB/s, 678 objects/s recovering 2024-09-21T13:55:47.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:47 smithi049 ceph-mon[30080]: pgmap v334: 148 pgs: 1 active+clean+remapped, 38 active+undersized, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 30 active+undersized+degraded, 75 active+clean; 7.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 383 B/s rd, 59 MiB/s wr, 948 op/s; 45642/364185 objects degraded (12.533%); 33156/364185 objects misplaced (9.104%); 42 MiB/s, 678 objects/s recovering 2024-09-21T13:55:47.830 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:55:47.866 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:47.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:47.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:55:47.968 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:55:48.004 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:48.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:48.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:55:48.106 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:55:48.151 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:48.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:48.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:55:48.252 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:48.280 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:48.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:48.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:48.301 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 90 5 126516 126511 87.8388 74.1875 0.0298822 0.0113787 2024-09-21T13:55:48.382 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:55:48.407 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:48.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:48.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:55:48.509 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:48.532 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:48.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:48.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:48.634 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:55:48.657 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:48.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:48.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:55:48.759 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:55:48.783 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:48.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:48.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:55:48.885 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:48.909 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:48.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:48.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:49.011 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:49.035 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:49.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:49.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:49.137 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:49.160 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:49.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:49.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:49.256 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T13:55:49.256 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T13:55:49.261 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:55:49.290 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:49.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:49.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:55:49.301 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Total time run: 90.0413 2024-09-21T13:55:49.302 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Total writes made: 126516 2024-09-21T13:55:49.302 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Write size: 65536 2024-09-21T13:55:49.302 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Object size: 65536 2024-09-21T13:55:49.302 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Bandwidth (MB/sec): 87.8181 2024-09-21T13:55:49.302 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Stddev Bandwidth: 35.6639 2024-09-21T13:55:49.302 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Max bandwidth (MB/sec): 209 2024-09-21T13:55:49.302 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Min bandwidth (MB/sec): 0 2024-09-21T13:55:49.302 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Average IOPS: 1405 2024-09-21T13:55:49.302 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Stddev IOPS: 570.629 2024-09-21T13:55:49.302 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Max IOPS: 3344 2024-09-21T13:55:49.302 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Min IOPS: 0 2024-09-21T13:55:49.302 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Average Latency(s): 0.0113802 2024-09-21T13:55:49.302 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Stddev Latency(s): 0.0279556 2024-09-21T13:55:49.303 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Max latency(s): 2.22092 2024-09-21T13:55:49.303 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Min latency(s): 0.00131689 2024-09-21T13:55:49.308 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Cleaning up (deleting benchmark objects) 2024-09-21T13:55:49.392 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:55:49.437 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:49.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:49.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:55:49.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:49 smithi049 ceph-mon[30080]: pgmap v335: 148 pgs: 1 active+clean+remapped, 38 active+undersized, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 30 active+undersized+degraded, 75 active+clean; 7.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 32 MiB/s wr, 508 op/s; 45642/364185 objects degraded (12.533%); 33156/364185 objects misplaced (9.104%); 30 MiB/s, 476 objects/s recovering 2024-09-21T13:55:49.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:49 smithi049 ceph-mon[40489]: pgmap v335: 148 pgs: 1 active+clean+remapped, 38 active+undersized, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 30 active+undersized+degraded, 75 active+clean; 7.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 32 MiB/s wr, 508 op/s; 45642/364185 objects degraded (12.533%); 33156/364185 objects misplaced (9.104%); 30 MiB/s, 476 objects/s recovering 2024-09-21T13:55:49.540 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:55:49.576 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:49.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:49.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:55:49.619 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:49 smithi187 ceph-mon[33032]: pgmap v335: 148 pgs: 1 active+clean+remapped, 38 active+undersized, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 30 active+undersized+degraded, 75 active+clean; 7.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 32 MiB/s wr, 508 op/s; 45642/364185 objects degraded (12.533%); 33156/364185 objects misplaced (9.104%); 30 MiB/s, 476 objects/s recovering 2024-09-21T13:55:49.619 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:55:49 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:55:49.668 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T13:55:49.677 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:55:49.775 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:49.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:49.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:55:49.877 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:49.907 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:49.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:49.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:50.009 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:50.041 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:50.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:50.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:50.143 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:50.174 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:50.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:50.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:50.255 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 13:55:49 smithi049 podman[119835]: 2024-09-21 13:55:49.867555553 +0000 UTC m=+0.041233634 image pull b415816db8b41f544cdfdc71d193ff8b1ee008bf7b0ec4f64da8eb8f71b62e85 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5 2024-09-21T13:55:50.255 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 13:55:50 smithi049 podman[119835]: 2024-09-21 13:55:50.002514838 +0000 UTC m=+0.176192911 container create faff379d03108d89e46bee7e089b995b0f1b92ef9a264b006b4a0c637ec59b66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-3-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, RELEASE=wip-guits-main-2024-09-21-0842-4417849, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=) 2024-09-21T13:55:50.276 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:55:50.325 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:50.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:50.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:55:50.427 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:55:50.459 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:50.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:50.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:55:50.503 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:55:50.560 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:55:50.581 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:50 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:55:50.581 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:50 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 45642/364185 objects degraded (12.533%), 31 pgs degraded (PG_DEGRADED) 2024-09-21T13:55:50.581 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 13:55:50 smithi049 podman[119835]: 2024-09-21 13:55:50.305938725 +0000 UTC m=+0.479616811 container init faff379d03108d89e46bee7e089b995b0f1b92ef9a264b006b4a0c637ec59b66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-3-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-guits-main-2024-09-21-0842-4417849, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True) 2024-09-21T13:55:50.581 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 13:55:50 smithi049 podman[119835]: 2024-09-21 13:55:50.314962846 +0000 UTC m=+0.488640916 container start faff379d03108d89e46bee7e089b995b0f1b92ef9a264b006b4a0c637ec59b66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-3-deactivate, io.buildah.version=1.37.2, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-guits-main-2024-09-21-0842-4417849, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-09-21T13:55:50.581 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 13:55:50 smithi049 podman[119835]: 2024-09-21 13:55:50.414592837 +0000 UTC m=+0.588270907 container attach faff379d03108d89e46bee7e089b995b0f1b92ef9a264b006b4a0c637ec59b66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-3-deactivate, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-21T13:55:50.581 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:50 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:55:50.581 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:50 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 45642/364185 objects degraded (12.533%), 31 pgs degraded (PG_DEGRADED) 2024-09-21T13:55:50.583 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:55:50 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:55:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:55:50.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:50.617 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:50.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:55:50.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:50 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:55:50.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:50 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 45642/364185 objects degraded (12.533%), 31 pgs degraded (PG_DEGRADED) 2024-09-21T13:55:50.718 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:55:50.757 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:50.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:50.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:55:50.859 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:50.887 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:50.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:50.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:50.989 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:51.025 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:51.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:51.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:51.126 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:55:51.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:51.166 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:51.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:55:51.200 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 13:55:50 smithi049 podman[119835]: 2024-09-21 13:55:50.861285117 +0000 UTC m=+1.034963194 container died faff379d03108d89e46bee7e089b995b0f1b92ef9a264b006b4a0c637ec59b66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-3-deactivate, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-09-21T13:55:51.256 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T13:55:51.266 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:55:51.301 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:51.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:51.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:55:51.403 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:55:51.414 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T13:55:51.459 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:51.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:51.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:55:51.490 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:51 smithi049 ceph-mon[30080]: pgmap v336: 148 pgs: 1 active+clean+remapped, 38 active+undersized, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 30 active+undersized+degraded, 75 active+clean; 7.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 321 B/s rd, 61 MiB/s wr, 982 op/s; 47607/380145 objects degraded (12.523%); 31768/380145 objects misplaced (8.357%); 43 MiB/s, 692 objects/s recovering 2024-09-21T13:55:51.491 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:51 smithi049 ceph-mon[40489]: pgmap v336: 148 pgs: 1 active+clean+remapped, 38 active+undersized, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 30 active+undersized+degraded, 75 active+clean; 7.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 321 B/s rd, 61 MiB/s wr, 982 op/s; 47607/380145 objects degraded (12.523%); 31768/380145 objects misplaced (8.357%); 43 MiB/s, 692 objects/s recovering 2024-09-21T13:55:51.561 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:55:51.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:51.631 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:51.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:55:51.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:51 smithi187 ceph-mon[33032]: pgmap v336: 148 pgs: 1 active+clean+remapped, 38 active+undersized, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 30 active+undersized+degraded, 75 active+clean; 7.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 321 B/s rd, 61 MiB/s wr, 982 op/s; 47607/380145 objects degraded (12.523%); 31768/380145 objects misplaced (8.357%); 43 MiB/s, 692 objects/s recovering 2024-09-21T13:55:51.732 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:55:51.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:51.781 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:51.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:55:51.882 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:55:51.931 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:51.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:51.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:55:51.974 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T13:55:51.974 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-09-21T13:55:51.974 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.974 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.15207.0:130052 8.1 e216/183 8:85d45f32:::benchmark_data_smithi049_109311_object3533:head v 216'7944, mlcod=216'7942)", 2024-09-21T13:55:51.974 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:55:51.960979+0000", 2024-09-21T13:55:51.974 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.0095776569999999998, 2024-09-21T13:55:51.974 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0096042809999999992, 2024-09-21T13:55:51.974 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:55:51.974 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:55:51.974 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T13:55:51.974 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:55:51.975 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.975 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:55:51.975 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:51.960979+0000", 2024-09-21T13:55:51.975 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:55:51.975 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:51.975 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.975 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:55:51.975 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:51.960979+0000", 2024-09-21T13:55:51.975 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:55:51.975 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:51.975 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.975 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:55:51.975 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:51.960979+0000", 2024-09-21T13:55:51.975 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.7099999999999998e-07 2024-09-21T13:55:51.975 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:51.976 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.976 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:55:51.976 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:51.960983+0000", 2024-09-21T13:55:51.976 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3739999999999998e-06 2024-09-21T13:55:51.976 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:51.976 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.976 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:55:51.976 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:51.960989+0000", 2024-09-21T13:55:51.976 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.003e-06 2024-09-21T13:55:51.976 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:51.976 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.976 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:55:51.976 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:51.960994+0000", 2024-09-21T13:55:51.976 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3070000000000002e-06 2024-09-21T13:55:51.977 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:51.977 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.977 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:55:51.977 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:51.961044+0000", 2024-09-21T13:55:51.977 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9429999999999999e-05 2024-09-21T13:55:51.977 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:51.977 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.977 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:55:51.977 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:51.961063+0000", 2024-09-21T13:55:51.977 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9825000000000001e-05 2024-09-21T13:55:51.977 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:51.977 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:55:51.977 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:51.977 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:51.977 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.977 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.15207.0:130059 8.1 e216/183 8:8da259ce:::benchmark_data_smithi049_109311_object3540:head v 216'7945, mlcod=216'7942)", 2024-09-21T13:55:51.978 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:55:51.961051+0000", 2024-09-21T13:55:51.978 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.0095059879999999999, 2024-09-21T13:55:51.978 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0095884149999999994, 2024-09-21T13:55:51.978 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:55:51.978 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:55:51.978 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T13:55:51.978 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:55:51.978 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.978 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:55:51.978 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:51.961051+0000", 2024-09-21T13:55:51.978 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:55:51.978 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:51.978 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.978 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:55:51.979 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:51.961051+0000", 2024-09-21T13:55:51.979 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:55:51.979 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:51.979 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.979 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:55:51.979 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:51.961051+0000", 2024-09-21T13:55:51.979 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.35e-07 2024-09-21T13:55:51.979 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:51.979 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.979 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:55:51.979 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:51.961054+0000", 2024-09-21T13:55:51.979 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.8660000000000002e-06 2024-09-21T13:55:51.979 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:51.979 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.979 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:55:51.980 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:51.961059+0000", 2024-09-21T13:55:51.980 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3940000000000004e-06 2024-09-21T13:55:51.980 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:51.980 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.980 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:55:51.980 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:51.961064+0000", 2024-09-21T13:55:51.980 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8509999999999998e-06 2024-09-21T13:55:51.980 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:51.980 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.980 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:55:51.980 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:51.963412+0000", 2024-09-21T13:55:51.980 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0023482030000000001 2024-09-21T13:55:51.980 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:51.980 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.980 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:55:51.980 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:51.963430+0000", 2024-09-21T13:55:51.980 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7969e-05 2024-09-21T13:55:51.981 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:51.981 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:55:51.981 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:51.981 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:51.981 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.981 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.15207.0:130062 8.d e216/183 8:bb0db186:::benchmark_data_smithi049_109311_object3543:head v 216'8121, mlcod=216'8115)", 2024-09-21T13:55:51.981 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:55:51.964350+0000", 2024-09-21T13:55:51.981 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.0062067210000000001, 2024-09-21T13:55:51.981 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0063390520000000004, 2024-09-21T13:55:51.981 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:55:51.981 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:55:51.981 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T13:55:51.981 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:55:51.981 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.982 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:55:51.982 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:51.964350+0000", 2024-09-21T13:55:51.982 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:55:51.982 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:51.982 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.982 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:55:51.982 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:51.964350+0000", 2024-09-21T13:55:51.982 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:55:51.982 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:51.982 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.982 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:55:51.982 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:51.964351+0000", 2024-09-21T13:55:51.982 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.04e-06 2024-09-21T13:55:51.983 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:51.983 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.983 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:55:51.983 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:51.964358+0000", 2024-09-21T13:55:51.983 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5189999999999998e-06 2024-09-21T13:55:51.983 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:51.983 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.983 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:55:51.983 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:51.964373+0000", 2024-09-21T13:55:51.983 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5483999999999999e-05 2024-09-21T13:55:51.983 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:51.983 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.983 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:55:51.984 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:51.964380+0000", 2024-09-21T13:55:51.984 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8299999999999998e-06 2024-09-21T13:55:51.984 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:51.984 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.984 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:55:51.984 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:51.967004+0000", 2024-09-21T13:55:51.984 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0026243989999999999 2024-09-21T13:55:51.984 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:51.984 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.984 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:55:51.984 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:51.967023+0000", 2024-09-21T13:55:51.984 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8417999999999999e-05 2024-09-21T13:55:51.984 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:51.984 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:55:51.985 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:51.985 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:51.985 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.985 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.15207.0:130064 8.6 e216/212 8:6c070d7b:::benchmark_data_smithi049_109311_object3545:head v 216'8125, mlcod=216'8123)", 2024-09-21T13:55:51.985 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:55:51.964795+0000", 2024-09-21T13:55:51.985 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.0057618979999999997, 2024-09-21T13:55:51.985 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.005943121, 2024-09-21T13:55:51.985 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:55:51.985 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:55:51.985 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T13:55:51.985 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:55:51.985 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.985 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:55:51.985 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:51.964795+0000", 2024-09-21T13:55:51.986 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:55:51.986 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:51.986 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.986 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:55:51.986 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:51.964795+0000", 2024-09-21T13:55:51.986 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:55:51.986 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:51.986 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.986 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:55:51.986 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:51.964796+0000", 2024-09-21T13:55:51.986 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4000000000000001e-07 2024-09-21T13:55:51.986 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:51.986 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.986 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:55:51.987 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:51.964801+0000", 2024-09-21T13:55:51.987 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0030000000000002e-06 2024-09-21T13:55:51.987 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:51.987 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.987 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:55:51.987 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:51.964810+0000", 2024-09-21T13:55:51.987 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3510000000000008e-06 2024-09-21T13:55:51.987 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:51.987 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.987 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:55:51.987 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:51.964817+0000", 2024-09-21T13:55:51.987 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8369999999999999e-06 2024-09-21T13:55:51.987 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:51.987 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.988 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:55:51.988 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:51.965771+0000", 2024-09-21T13:55:51.988 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00095387399999999995 2024-09-21T13:55:51.988 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:51.988 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.988 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:55:51.988 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:51.965790+0000", 2024-09-21T13:55:51.988 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9701e-05 2024-09-21T13:55:51.988 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:51.988 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:55:51.988 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:51.988 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:51.988 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.988 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.15207.0:130061 8.e e216/183 8:72aa1beb:::benchmark_data_smithi049_109311_object3542:head v 216'8101, mlcod=216'8098)", 2024-09-21T13:55:51.988 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:55:51.965525+0000", 2024-09-21T13:55:51.989 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.0050314050000000001, 2024-09-21T13:55:51.989 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0052626369999999997, 2024-09-21T13:55:51.989 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:55:51.989 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:55:51.989 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T13:55:51.989 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:55:51.989 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.989 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:55:51.989 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:51.965525+0000", 2024-09-21T13:55:51.989 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:55:51.989 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:51.989 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.989 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:55:51.989 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:51.965525+0000", 2024-09-21T13:55:51.989 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:55:51.990 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:51.990 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.990 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:55:51.990 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:51.965526+0000", 2024-09-21T13:55:51.990 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.7300000000000005e-07 2024-09-21T13:55:51.990 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:51.990 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.990 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:55:51.990 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:51.965533+0000", 2024-09-21T13:55:51.990 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1409999999999999e-06 2024-09-21T13:55:51.990 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:51.990 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.990 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:55:51.991 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:51.965545+0000", 2024-09-21T13:55:51.991 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1455e-05 2024-09-21T13:55:51.991 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:51.991 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.991 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:55:51.991 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:51.965551+0000", 2024-09-21T13:55:51.991 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4930000000000003e-06 2024-09-21T13:55:51.991 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:51.991 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.991 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:55:51.991 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:51.967343+0000", 2024-09-21T13:55:51.991 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001791459 2024-09-21T13:55:51.991 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:51.991 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.992 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:55:51.992 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:51.967361+0000", 2024-09-21T13:55:51.992 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7965000000000002e-05 2024-09-21T13:55:51.992 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:51.992 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:55:51.992 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:51.992 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:51.992 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.992 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.15207.0:130066 8.3 e216/212 8:c6fb71c9:::benchmark_data_smithi049_109311_object3547:head v 216'8155, mlcod=216'8153)", 2024-09-21T13:55:51.992 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:55:51.967343+0000", 2024-09-21T13:55:51.992 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.003213722, 2024-09-21T13:55:51.992 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.003493052, 2024-09-21T13:55:51.992 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:55:51.992 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:55:51.992 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T13:55:51.993 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:55:51.993 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.993 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:55:51.993 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:51.967343+0000", 2024-09-21T13:55:51.993 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:55:51.993 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:51.993 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.993 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:55:51.993 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:51.967343+0000", 2024-09-21T13:55:51.993 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:55:51.993 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:51.993 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.993 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:55:51.993 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:51.967343+0000", 2024-09-21T13:55:51.994 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7099999999999997e-07 2024-09-21T13:55:51.994 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:51.994 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.994 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:55:51.994 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:51.967348+0000", 2024-09-21T13:55:51.994 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5430000000000004e-06 2024-09-21T13:55:51.994 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:51.994 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.994 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:55:51.994 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:51.967355+0000", 2024-09-21T13:55:51.994 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0330000000000003e-06 2024-09-21T13:55:51.994 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:51.994 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.994 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:55:51.995 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:51.967360+0000", 2024-09-21T13:55:51.995 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8520000000000003e-06 2024-09-21T13:55:51.995 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:51.995 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.995 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:55:51.995 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:51.967553+0000", 2024-09-21T13:55:51.995 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00019303899999999999 2024-09-21T13:55:51.995 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:51.995 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.995 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:55:51.995 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:51.967574+0000", 2024-09-21T13:55:51.995 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0954999999999998e-05 2024-09-21T13:55:51.995 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:51.995 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:55:51.995 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:51.996 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:51.996 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.996 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.15207.0:130065 8.1 e216/183 8:87b25119:::benchmark_data_smithi049_109311_object3546:head v 216'7946, mlcod=216'7942)", 2024-09-21T13:55:51.996 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:55:51.967596+0000", 2024-09-21T13:55:51.996 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.0029604470000000002, 2024-09-21T13:55:51.996 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0032873260000000001, 2024-09-21T13:55:51.996 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:55:51.996 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:55:51.996 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T13:55:51.996 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:55:51.996 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.996 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:55:51.996 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:51.967596+0000", 2024-09-21T13:55:51.996 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:55:51.996 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:51.997 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.997 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:55:51.997 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:51.967596+0000", 2024-09-21T13:55:51.997 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:55:51.997 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:51.997 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.997 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:55:51.997 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:51.967597+0000", 2024-09-21T13:55:51.997 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1100000000000005e-07 2024-09-21T13:55:51.997 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:51.997 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.997 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:55:51.997 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:51.967603+0000", 2024-09-21T13:55:51.997 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.772e-06 2024-09-21T13:55:51.997 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:51.998 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.998 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:55:51.998 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:51.967613+0000", 2024-09-21T13:55:51.998 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0074e-05 2024-09-21T13:55:51.998 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:51.998 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.998 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:55:51.998 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:51.967620+0000", 2024-09-21T13:55:51.998 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6089999999999997e-06 2024-09-21T13:55:51.998 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:51.998 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.998 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:55:51.998 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:51.969631+0000", 2024-09-21T13:55:51.998 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0020111740000000001 2024-09-21T13:55:51.999 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:55:51.999 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:55:51.999 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:55:51.999 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:51.969650+0000", 2024-09-21T13:55:51.999 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9689000000000001e-05 2024-09-21T13:55:51.999 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:51.999 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:55:51.999 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:51.999 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:55:51.999 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T13:55:51.999 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 7 2024-09-21T13:55:51.999 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T13:55:52.033 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:52.066 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:52.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:52.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:52.168 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:55:52.206 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:52.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:52.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:55:52.226 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 13:55:51 smithi049 podman[119954]: 2024-09-21 13:55:51.646877748 +0000 UTC m=+1.004731961 container remove faff379d03108d89e46bee7e089b995b0f1b92ef9a264b006b4a0c637ec59b66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-3-deactivate, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-21-0842-4417849) 2024-09-21T13:55:52.245 DEBUG:teuthology.orchestra.run.smithi049:> sudo pkill -f 'journalctl -f -n 0 -u ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@osd.3.service' 2024-09-21T13:55:52.308 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:52.339 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:52.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:52.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:52.351 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T13:55:52.351 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-09-21T13:55:52.351 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.3 2024-09-21T13:55:52.351 INFO:tasks.thrashosds.thrasher:remote for osd 3 is ubuntu@smithi049.front.sepia.ceph.com 2024-09-21T13:55:52.351 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell -n osd.3 --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --op list-pgs 2024-09-21T13:55:52.440 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:52.465 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:52.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:52.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:52.567 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:55:52.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:52.608 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:52.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:55:52.709 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:55:52.757 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:52.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:52.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:55:52.769 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/osd.3/config 2024-09-21T13:55:52.860 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:55:52.887 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:55:52.888 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:55:52.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:52.923 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:52.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:55:52.943 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T13:55:53.024 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:53.057 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:53.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:53.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:53.159 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:55:53.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:53.243 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:53.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:55:53.344 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:53.382 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:53.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:53.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:53.484 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:53.515 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:53.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:53.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:53.617 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:53.648 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:53.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:53.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:53.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:53 smithi187 ceph-mon[33032]: pgmap v337: 148 pgs: 1 active+clean+remapped, 38 active+undersized, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 30 active+undersized+degraded, 75 active+clean; 7.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 290 B/s rd, 56 MiB/s wr, 889 op/s; 47607/380145 objects degraded (12.523%); 31768/380145 objects misplaced (8.357%); 39 MiB/s, 626 objects/s recovering 2024-09-21T13:55:53.709 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:53 smithi049 ceph-mon[40489]: pgmap v337: 148 pgs: 1 active+clean+remapped, 38 active+undersized, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 30 active+undersized+degraded, 75 active+clean; 7.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 290 B/s rd, 56 MiB/s wr, 889 op/s; 47607/380145 objects degraded (12.523%); 31768/380145 objects misplaced (8.357%); 39 MiB/s, 626 objects/s recovering 2024-09-21T13:55:53.709 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:53 smithi049 ceph-mon[30080]: pgmap v337: 148 pgs: 1 active+clean+remapped, 38 active+undersized, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 30 active+undersized+degraded, 75 active+clean; 7.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 290 B/s rd, 56 MiB/s wr, 889 op/s; 47607/380145 objects degraded (12.523%); 31768/380145 objects misplaced (8.357%); 39 MiB/s, 626 objects/s recovering 2024-09-21T13:55:53.749 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:53.791 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:53.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:53.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:53.893 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:55:53.934 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:53.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:53.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:55:54.036 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:55:54.074 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:54.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:54.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:55:54.175 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:54.211 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:54.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:54.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:54.313 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:54.333 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:54.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:54.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:54.435 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:54.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:54.474 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:54.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:54.575 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:55:54.610 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:54.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:54.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:55:54.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:54 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1280854785' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:55:54.712 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:54.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:54.749 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:54.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:54.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:54 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1280854785' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:55:54.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:54 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1280854785' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:55:54.850 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:55:54.883 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:54.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:54.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:55:54.985 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:55:55.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:55.022 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:55.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:55:55.063 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:55:55.063 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:55:55.063 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:55:55.064 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:55:55.064 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:55:55.064 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:55:55.064 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:55:55.064 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:55:55.064 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-21T13:55:55.064 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:55:55.064 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:55:55.065 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:55:55.065 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:55:55.065 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:55:55.065 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:55:55.123 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:55:55.160 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:55.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:55.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:55:55.262 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:55:55.291 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:55.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:55.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:55:55.393 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:55:55.434 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:55.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:55.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:55:55.535 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:55.569 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:55.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:55.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:55.673 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:55:55.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:55 smithi187 ceph-mon[33032]: pgmap v338: 148 pgs: 1 active+clean+remapped, 38 active+undersized, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 30 active+undersized+degraded, 75 active+clean; 7.4 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 767 B/s rd, 49 MiB/s wr, 1.20k op/s; 45844/365238 objects degraded (12.552%); 27134/365238 objects misplaced (7.429%); 52 MiB/s, 824 objects/s recovering 2024-09-21T13:55:55.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:55 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 45844/365238 objects degraded (12.552%), 31 pgs degraded (PG_DEGRADED) 2024-09-21T13:55:55.709 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:55.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:55.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:55:55.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:55 smithi049 ceph-mon[40489]: pgmap v338: 148 pgs: 1 active+clean+remapped, 38 active+undersized, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 30 active+undersized+degraded, 75 active+clean; 7.4 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 767 B/s rd, 49 MiB/s wr, 1.20k op/s; 45844/365238 objects degraded (12.552%); 27134/365238 objects misplaced (7.429%); 52 MiB/s, 824 objects/s recovering 2024-09-21T13:55:55.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:55 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 45844/365238 objects degraded (12.552%), 31 pgs degraded (PG_DEGRADED) 2024-09-21T13:55:55.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:55 smithi049 ceph-mon[30080]: pgmap v338: 148 pgs: 1 active+clean+remapped, 38 active+undersized, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 30 active+undersized+degraded, 75 active+clean; 7.4 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 767 B/s rd, 49 MiB/s wr, 1.20k op/s; 45844/365238 objects degraded (12.552%); 27134/365238 objects misplaced (7.429%); 52 MiB/s, 824 objects/s recovering 2024-09-21T13:55:55.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:55 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 45844/365238 objects degraded (12.552%), 31 pgs degraded (PG_DEGRADED) 2024-09-21T13:55:55.811 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:55:55.846 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:55.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:55.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:55:55.948 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:55:55.990 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:55.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:55.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:55:56.092 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:55:56.130 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:56.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:56.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:55:56.232 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:55:56.270 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:56.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:56.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:55:56.372 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:55:56.409 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:56.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:56.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:55:56.511 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:55:56.549 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:56.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:56.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:55:56.651 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:55:56.688 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:56.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:56.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:55:56.704 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T13:55:56.705 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T13:55:56.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:56 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:55:56.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:56 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:55:56.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:56 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:55:56.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:56 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:55:56.789 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:55:56.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:56.834 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:56.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:55:56.935 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:55:56.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:56 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:55:56.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:56 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:55:56.972 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:56.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:56.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:55:57.074 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:55:57.110 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:57.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:57.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:55:57.212 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:55:57.231 INFO:teuthology.orchestra.run.smithi049.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-21T13:55:57.246 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:57.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:57.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:55:57.348 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:55:57.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:57.392 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:57.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:55:57.438 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:55:57.493 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:55:57.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:57.544 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:57.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:55:57.645 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:55:57.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:57.682 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:57.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:55:57.748 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:57 smithi049 ceph-mon[30080]: pgmap v339: 148 pgs: 1 active+clean+remapped, 38 active+undersized, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 30 active+undersized+degraded, 75 active+clean; 7.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 938 B/s rd, 31 MiB/s wr, 970 op/s; 45442/362853 objects degraded (12.524%); 27134/362853 objects misplaced (7.478%); 32 MiB/s, 507 objects/s recovering 2024-09-21T13:55:57.748 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:57 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:55:57.748 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:57 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:55:57.748 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:57 smithi049 ceph-mon[30080]: osdmap e217: 8 total, 7 up, 7 in 2024-09-21T13:55:57.749 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:57 smithi049 ceph-mon[40489]: pgmap v339: 148 pgs: 1 active+clean+remapped, 38 active+undersized, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 30 active+undersized+degraded, 75 active+clean; 7.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 938 B/s rd, 31 MiB/s wr, 970 op/s; 45442/362853 objects degraded (12.524%); 27134/362853 objects misplaced (7.478%); 32 MiB/s, 507 objects/s recovering 2024-09-21T13:55:57.749 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:57 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:55:57.749 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:57 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:55:57.749 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:57 smithi049 ceph-mon[40489]: osdmap e217: 8 total, 7 up, 7 in 2024-09-21T13:55:57.783 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:57.820 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:57.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:57.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:57.922 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:55:57.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:57 smithi187 ceph-mon[33032]: pgmap v339: 148 pgs: 1 active+clean+remapped, 38 active+undersized, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 30 active+undersized+degraded, 75 active+clean; 7.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 938 B/s rd, 31 MiB/s wr, 970 op/s; 45442/362853 objects degraded (12.524%); 27134/362853 objects misplaced (7.478%); 32 MiB/s, 507 objects/s recovering 2024-09-21T13:55:57.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:57 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:55:57.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:57 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:55:57.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:57 smithi187 ceph-mon[33032]: osdmap e217: 8 total, 7 up, 7 in 2024-09-21T13:55:57.955 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:57.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:57.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:55:58.057 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:55:58.102 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:58.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:58.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:55:58.203 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:58.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:58.238 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:58.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:55:58.339 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:55:58.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:58.385 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:58.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:55:58.484 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-09-21T13:55:58.486 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:55:58.527 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T13:55:58.527 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:55:58.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:58.532 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:58.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:55:58.633 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:55:58.671 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:58.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:58.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:55:58.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:58 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3430117892' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:55:58.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:58 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:55:58.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:58 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:55:58.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:58 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:55:58.706 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T13:55:58.726 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:58 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3430117892' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:55:58.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:58 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:55:58.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:58 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:55:58.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:58 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:55:58.727 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:58 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3430117892' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:55:58.727 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:58 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:55:58.727 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:58 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:55:58.727 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:58 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:55:58.773 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:55:58.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:58.806 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:58.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:55:58.907 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:55:58.933 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:58.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:58.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:55:59.035 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:55:59.062 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:59.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:59.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:55:59.165 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:55:59.202 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:59.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:59.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:55:59.304 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:55:59.344 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:59.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:59.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:55:59.446 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:55:59.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:59.489 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:59.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:55:59.590 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:55:59.631 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:59.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:59.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:55:59.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:59 smithi187 ceph-mon[33032]: pgmap v341: 148 pgs: 1 active+clean+remapped, 38 active+undersized, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 30 active+undersized+degraded, 75 active+clean; 7.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.1 KiB/s rd, 33 MiB/s wr, 1.11k op/s; 45442/362853 objects degraded (12.524%); 27134/362853 objects misplaced (7.478%); 38 MiB/s, 608 objects/s recovering 2024-09-21T13:55:59.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:59 smithi187 ceph-mon[33032]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:55:59.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:59 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T13:55:59.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:55:59 smithi187 ceph-mon[33032]: osdmap e218: 8 total, 7 up, 7 in 2024-09-21T13:55:59.695 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:55:59 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:55:59.733 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:55:59.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:59 smithi049 ceph-mon[40489]: pgmap v341: 148 pgs: 1 active+clean+remapped, 38 active+undersized, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 30 active+undersized+degraded, 75 active+clean; 7.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.1 KiB/s rd, 33 MiB/s wr, 1.11k op/s; 45442/362853 objects degraded (12.524%); 27134/362853 objects misplaced (7.478%); 38 MiB/s, 608 objects/s recovering 2024-09-21T13:55:59.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:59 smithi049 ceph-mon[40489]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:55:59.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:59 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T13:55:59.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:55:59 smithi049 ceph-mon[40489]: osdmap e218: 8 total, 7 up, 7 in 2024-09-21T13:55:59.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:59 smithi049 ceph-mon[30080]: pgmap v341: 148 pgs: 1 active+clean+remapped, 38 active+undersized, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 30 active+undersized+degraded, 75 active+clean; 7.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.1 KiB/s rd, 33 MiB/s wr, 1.11k op/s; 45442/362853 objects degraded (12.524%); 27134/362853 objects misplaced (7.478%); 38 MiB/s, 608 objects/s recovering 2024-09-21T13:55:59.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:59 smithi049 ceph-mon[30080]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:55:59.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:59 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T13:55:59.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:55:59 smithi049 ceph-mon[30080]: osdmap e218: 8 total, 7 up, 7 in 2024-09-21T13:55:59.773 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:55:59.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:59.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:55:59.875 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:55:59.907 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:55:59.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:55:59.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:56:00.009 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:56:00.043 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:00.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:00.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:56:00.079 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T13:56:00.145 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:00.173 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:00.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:00.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:00.244 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:56:00.275 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:56:00.302 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:00.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:00.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:56:00.403 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:00.434 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:00.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:00.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:00.536 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:00.586 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:00.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:00.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:00.607 INFO:teuthology.orchestra.run.smithi049.stdout:8.c 2024-09-21T13:56:00.607 INFO:teuthology.orchestra.run.smithi049.stdout:8.9 2024-09-21T13:56:00.607 INFO:teuthology.orchestra.run.smithi049.stdout:8.7 2024-09-21T13:56:00.607 INFO:teuthology.orchestra.run.smithi049.stdout:8.0 2024-09-21T13:56:00.608 INFO:teuthology.orchestra.run.smithi049.stdout:6.f 2024-09-21T13:56:00.608 INFO:teuthology.orchestra.run.smithi049.stdout:6.c 2024-09-21T13:56:00.608 INFO:teuthology.orchestra.run.smithi049.stdout:6.b 2024-09-21T13:56:00.608 INFO:teuthology.orchestra.run.smithi049.stdout:6.8 2024-09-21T13:56:00.608 INFO:teuthology.orchestra.run.smithi049.stdout:4.10 2024-09-21T13:56:00.608 INFO:teuthology.orchestra.run.smithi049.stdout:6.19 2024-09-21T13:56:00.608 INFO:teuthology.orchestra.run.smithi049.stdout:4.17 2024-09-21T13:56:00.608 INFO:teuthology.orchestra.run.smithi049.stdout:5.0 2024-09-21T13:56:00.608 INFO:teuthology.orchestra.run.smithi049.stdout:3.d 2024-09-21T13:56:00.609 INFO:teuthology.orchestra.run.smithi049.stdout:4.14 2024-09-21T13:56:00.609 INFO:teuthology.orchestra.run.smithi049.stdout:4.1b 2024-09-21T13:56:00.609 INFO:teuthology.orchestra.run.smithi049.stdout:3.a 2024-09-21T13:56:00.609 INFO:teuthology.orchestra.run.smithi049.stdout:3.10 2024-09-21T13:56:00.609 INFO:teuthology.orchestra.run.smithi049.stdout:6.0 2024-09-21T13:56:00.609 INFO:teuthology.orchestra.run.smithi049.stdout:8.f 2024-09-21T13:56:00.609 INFO:teuthology.orchestra.run.smithi049.stdout:6.6 2024-09-21T13:56:00.609 INFO:teuthology.orchestra.run.smithi049.stdout:4.18 2024-09-21T13:56:00.609 INFO:teuthology.orchestra.run.smithi049.stdout:2.1 2024-09-21T13:56:00.609 INFO:teuthology.orchestra.run.smithi049.stdout:4.0 2024-09-21T13:56:00.609 INFO:teuthology.orchestra.run.smithi049.stdout:3.12 2024-09-21T13:56:00.609 INFO:teuthology.orchestra.run.smithi049.stdout:3.17 2024-09-21T13:56:00.609 INFO:teuthology.orchestra.run.smithi049.stdout:6.5 2024-09-21T13:56:00.609 INFO:teuthology.orchestra.run.smithi049.stdout:3.15 2024-09-21T13:56:00.609 INFO:teuthology.orchestra.run.smithi049.stdout:3.18 2024-09-21T13:56:00.610 INFO:teuthology.orchestra.run.smithi049.stdout:8.b 2024-09-21T13:56:00.610 INFO:teuthology.orchestra.run.smithi049.stdout:3.5 2024-09-21T13:56:00.610 INFO:teuthology.orchestra.run.smithi049.stdout:4.1c 2024-09-21T13:56:00.610 INFO:teuthology.orchestra.run.smithi049.stdout:6.13 2024-09-21T13:56:00.610 INFO:teuthology.orchestra.run.smithi049.stdout:3.19 2024-09-21T13:56:00.610 INFO:teuthology.orchestra.run.smithi049.stdout:4.19 2024-09-21T13:56:00.610 INFO:teuthology.orchestra.run.smithi049.stdout:4.f 2024-09-21T13:56:00.610 INFO:teuthology.orchestra.run.smithi049.stdout:3.8 2024-09-21T13:56:00.610 INFO:teuthology.orchestra.run.smithi049.stdout:3.c 2024-09-21T13:56:00.610 INFO:teuthology.orchestra.run.smithi049.stdout:4.c 2024-09-21T13:56:00.610 INFO:teuthology.orchestra.run.smithi049.stdout:3.2 2024-09-21T13:56:00.610 INFO:teuthology.orchestra.run.smithi049.stdout:5.16 2024-09-21T13:56:00.610 INFO:teuthology.orchestra.run.smithi049.stdout:3.7 2024-09-21T13:56:00.611 INFO:teuthology.orchestra.run.smithi049.stdout:4.11 2024-09-21T13:56:00.611 INFO:teuthology.orchestra.run.smithi049.stdout:4.1d 2024-09-21T13:56:00.611 INFO:teuthology.orchestra.run.smithi049.stdout:4.1e 2024-09-21T13:56:00.611 INFO:teuthology.orchestra.run.smithi049.stdout:4.4 2024-09-21T13:56:00.611 INFO:teuthology.orchestra.run.smithi049.stdout:6.2 2024-09-21T13:56:00.611 INFO:teuthology.orchestra.run.smithi049.stdout:4.15 2024-09-21T13:56:00.611 INFO:teuthology.orchestra.run.smithi049.stdout:3.1 2024-09-21T13:56:00.611 INFO:teuthology.orchestra.run.smithi049.stdout:5.a 2024-09-21T13:56:00.611 INFO:teuthology.orchestra.run.smithi049.stdout:4.9 2024-09-21T13:56:00.611 INFO:teuthology.orchestra.run.smithi049.stdout:4.e 2024-09-21T13:56:00.611 INFO:teuthology.orchestra.run.smithi049.stdout:4.16 2024-09-21T13:56:00.611 INFO:teuthology.orchestra.run.smithi049.stdout:4.1a 2024-09-21T13:56:00.611 INFO:teuthology.orchestra.run.smithi049.stdout:6.7 2024-09-21T13:56:00.611 INFO:teuthology.orchestra.run.smithi049.stdout:5.12 2024-09-21T13:56:00.611 INFO:teuthology.orchestra.run.smithi049.stdout:5.13 2024-09-21T13:56:00.612 INFO:teuthology.orchestra.run.smithi049.stdout:3.b 2024-09-21T13:56:00.612 INFO:teuthology.orchestra.run.smithi049.stdout:5.14 2024-09-21T13:56:00.612 INFO:teuthology.orchestra.run.smithi049.stdout:6.11 2024-09-21T13:56:00.612 INFO:teuthology.orchestra.run.smithi049.stdout:5.17 2024-09-21T13:56:00.612 INFO:teuthology.orchestra.run.smithi049.stdout:5.1 2024-09-21T13:56:00.612 INFO:teuthology.orchestra.run.smithi049.stdout:5.3 2024-09-21T13:56:00.612 INFO:teuthology.orchestra.run.smithi049.stdout:5.1c 2024-09-21T13:56:00.612 INFO:teuthology.orchestra.run.smithi049.stdout:6.16 2024-09-21T13:56:00.612 INFO:teuthology.orchestra.run.smithi049.stdout:6.1b 2024-09-21T13:56:00.612 INFO:teuthology.orchestra.run.smithi049.stdout:6.1e 2024-09-21T13:56:00.612 INFO:teuthology.orchestra.run.smithi049.stdout:3.1e 2024-09-21T13:56:00.612 INFO:teuthology.orchestra.run.smithi049.stdout:6.4 2024-09-21T13:56:00.612 INFO:teuthology.orchestra.run.smithi049.stdout:8.8 2024-09-21T13:56:00.612 INFO:teuthology.orchestra.run.smithi049.stdout:6.1f 2024-09-21T13:56:00.688 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:00.723 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:00 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:56:00.723 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:00 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 45442/362853 objects degraded (12.524%), 31 pgs degraded (PG_DEGRADED) 2024-09-21T13:56:00.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:00 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:56:00.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:00 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 45442/362853 objects degraded (12.524%), 31 pgs degraded (PG_DEGRADED) 2024-09-21T13:56:00.724 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:56:00 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:56:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:56:00.732 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:00.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:00.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:00.834 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:56:00.865 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:00.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:00.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:56:00.929 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:56:00.929 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:56:00.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:00 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:56:00.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:00 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 45442/362853 objects degraded (12.524%), 31 pgs degraded (PG_DEGRADED) 2024-09-21T13:56:00.968 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:56:00.976 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:56:00.976 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:56:01.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:01.009 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:01.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:56:01.010 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:56:01.010 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:56:01.012 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-21T13:56:01.048 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:56:01.049 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:56:01.085 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:56:01.085 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:56:01.110 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:01.125 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:56:01.126 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:56:01.143 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:01.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:01.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:01.162 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:56:01.162 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:56:01.245 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:01.290 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:01.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:01.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:01.391 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:01.429 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:01.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:01.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:01.531 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:01.569 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:01.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:01.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:01.578 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T13:56:01.580 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:01 smithi049 ceph-mon[40489]: pgmap v343: 148 pgs: 1 active+clean+remapped, 38 active+undersized, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 30 active+undersized+degraded, 75 active+clean; 7.0 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 1.4 KiB/s rd, 127 B/s wr, 1.57k op/s; 42970/342549 objects degraded (12.544%); 23440/342549 objects misplaced (6.843%); 43 MiB/s, 690 objects/s recovering 2024-09-21T13:56:01.580 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:01 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/376224276' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:56:01.583 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:01 smithi049 ceph-mon[30080]: pgmap v343: 148 pgs: 1 active+clean+remapped, 38 active+undersized, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 30 active+undersized+degraded, 75 active+clean; 7.0 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 1.4 KiB/s rd, 127 B/s wr, 1.57k op/s; 42970/342549 objects degraded (12.544%); 23440/342549 objects misplaced (6.843%); 43 MiB/s, 690 objects/s recovering 2024-09-21T13:56:01.583 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:01 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/376224276' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:56:01.630 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell -n osd.3 --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --op export --pgid 6.13 --file /var/log/ceph/exp.6.13.3 2024-09-21T13:56:01.672 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:01.735 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:01.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:01.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:01.837 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:01.869 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:01.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:01.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:01.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:01 smithi187 ceph-mon[33032]: pgmap v343: 148 pgs: 1 active+clean+remapped, 38 active+undersized, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 30 active+undersized+degraded, 75 active+clean; 7.0 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 1.4 KiB/s rd, 127 B/s wr, 1.57k op/s; 42970/342549 objects degraded (12.544%); 23440/342549 objects misplaced (6.843%); 43 MiB/s, 690 objects/s recovering 2024-09-21T13:56:01.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:01 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/376224276' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:56:01.971 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:56:02.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:02.004 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:02.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:56:02.105 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:56:02.139 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:02.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:02.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:56:02.240 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:02.299 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:02.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:02.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:02.307 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T13:56:02.307 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T13:56:02.329 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/osd.3/config 2024-09-21T13:56:02.335 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:56:02.401 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:02.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:02.451 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:02.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:02.552 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:56:02.588 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:02.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:02.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:56:02.690 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:02.732 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:02.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:02.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:02.834 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:56:02.873 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:02.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:02.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:56:02.974 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:56:03.023 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:03.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:03.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:56:03.125 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:56:03.154 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:03.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:03.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:56:03.255 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T13:56:03.255 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-09-21T13:56:03.255 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-09-21T13:56:03.255 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-09-21T13:56:03.255 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.255 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:163570 7.8 7.330e1638 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e130)", 2024-09-21T13:56:03.255 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:03.335126+0000", 2024-09-21T13:56:03.255 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 239.91730048700001, 2024-09-21T13:56:03.256 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97989348499999995, 2024-09-21T13:56:03.256 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:03.256 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:03.256 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T13:56:03.256 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:56:03.256 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:56:03.256 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:56:03.256 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 163570 2024-09-21T13:56:03.256 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.256 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:03.256 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.256 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:03.256 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.335126+0000", 2024-09-21T13:56:03.257 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:03.257 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.257 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.257 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:03.257 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.335126+0000", 2024-09-21T13:56:03.257 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:03.257 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.257 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.257 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:03.257 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.335127+0000", 2024-09-21T13:56:03.257 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5209999999999999e-06 2024-09-21T13:56:03.257 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.257 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.257 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:03.258 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.335132+0000", 2024-09-21T13:56:03.258 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4730000000000002e-06 2024-09-21T13:56:03.258 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.258 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.258 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:03.258 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.335142+0000", 2024-09-21T13:56:03.258 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8770000000000005e-06 2024-09-21T13:56:03.258 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.258 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.258 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:03.258 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.335151+0000", 2024-09-21T13:56:03.258 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7650000000000005e-06 2024-09-21T13:56:03.259 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.259 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.259 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:03.259 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.336965+0000", 2024-09-21T13:56:03.259 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001813907 2024-09-21T13:56:03.259 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.259 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.259 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:56:03.259 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.336978+0000", 2024-09-21T13:56:03.259 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2685e-05 2024-09-21T13:56:03.259 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.259 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.259 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:03.260 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:04.315001+0000", 2024-09-21T13:56:03.260 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97802330000000004 2024-09-21T13:56:03.260 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.260 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.260 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:03.260 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:04.315019+0000", 2024-09-21T13:56:03.260 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7957000000000001e-05 2024-09-21T13:56:03.260 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:03.260 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:03.260 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:03.260 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.260 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.260 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:163585 7.8 7.9fcafe08 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-21T13:56:03.261 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:03.339560+0000", 2024-09-21T13:56:03.261 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 239.91286631200001, 2024-09-21T13:56:03.261 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97550176099999997, 2024-09-21T13:56:03.261 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:03.261 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:03.261 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T13:56:03.261 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:56:03.261 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:56:03.261 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:56:03.261 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 163585 2024-09-21T13:56:03.261 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.261 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:03.261 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.261 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:03.261 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.339560+0000", 2024-09-21T13:56:03.261 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:03.262 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.262 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.262 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:03.262 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.339560+0000", 2024-09-21T13:56:03.262 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:03.262 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.262 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.262 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:03.262 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.339562+0000", 2024-09-21T13:56:03.262 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6449999999999999e-06 2024-09-21T13:56:03.262 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.262 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.262 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:03.262 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.339568+0000", 2024-09-21T13:56:03.263 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2169999999999999e-06 2024-09-21T13:56:03.263 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.263 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.263 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:03.263 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.339580+0000", 2024-09-21T13:56:03.263 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1846e-05 2024-09-21T13:56:03.263 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.263 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.263 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:03.263 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.339589+0000", 2024-09-21T13:56:03.263 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.798e-06 2024-09-21T13:56:03.263 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.263 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.263 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:03.264 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.342967+0000", 2024-09-21T13:56:03.264 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0033776990000000001 2024-09-21T13:56:03.264 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.264 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.264 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:56:03.264 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.342980+0000", 2024-09-21T13:56:03.264 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2758000000000001e-05 2024-09-21T13:56:03.264 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.264 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.264 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:03.264 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:04.315052+0000", 2024-09-21T13:56:03.264 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97207221799999999 2024-09-21T13:56:03.264 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.264 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.265 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:03.265 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:04.315062+0000", 2024-09-21T13:56:03.265 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.5799999999999998e-06 2024-09-21T13:56:03.265 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:03.265 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:03.265 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:03.265 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.265 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.265 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:163595 7.8 7.9f5fc9f8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-21T13:56:03.265 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:03.360541+0000", 2024-09-21T13:56:03.265 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 239.89188514200001, 2024-09-21T13:56:03.265 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.95455601800000001, 2024-09-21T13:56:03.265 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:03.265 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:03.265 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T13:56:03.265 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:56:03.266 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:56:03.266 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:56:03.266 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 163595 2024-09-21T13:56:03.266 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.266 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:03.266 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.266 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:03.266 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.360541+0000", 2024-09-21T13:56:03.266 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:03.266 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.266 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.266 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:03.266 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.360541+0000", 2024-09-21T13:56:03.266 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:03.266 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.267 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.267 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:03.267 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.360544+0000", 2024-09-21T13:56:03.267 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.632e-06 2024-09-21T13:56:03.267 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.267 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.267 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:03.267 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.360554+0000", 2024-09-21T13:56:03.267 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0210000000000001e-05 2024-09-21T13:56:03.267 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.267 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.267 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:03.267 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.360579+0000", 2024-09-21T13:56:03.268 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4878000000000001e-05 2024-09-21T13:56:03.268 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.268 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.268 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:03.268 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.360589+0000", 2024-09-21T13:56:03.268 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7960000000000007e-06 2024-09-21T13:56:03.268 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.268 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.268 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:03.268 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.360662+0000", 2024-09-21T13:56:03.268 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3514000000000001e-05 2024-09-21T13:56:03.268 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.268 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.268 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:56:03.269 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.360673+0000", 2024-09-21T13:56:03.269 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1080000000000001e-05 2024-09-21T13:56:03.269 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.269 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.269 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:03.269 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:04.315088+0000", 2024-09-21T13:56:03.269 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.954414859 2024-09-21T13:56:03.269 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.269 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.269 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:03.270 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:03.270 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:04.315097+0000", 2024-09-21T13:56:03.270 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0489999999999993e-06 2024-09-21T13:56:03.270 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:03.271 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:03.271 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:03.271 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.271 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.271 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:163602 7.8 7.3bec8d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-21T13:56:03.271 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:03.369662+0000", 2024-09-21T13:56:03.271 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 239.88276438599999, 2024-09-21T13:56:03.271 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.945473868, 2024-09-21T13:56:03.271 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:03.271 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:03.271 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T13:56:03.271 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:56:03.271 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:56:03.271 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:56:03.272 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 163602 2024-09-21T13:56:03.272 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.272 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:03.272 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.272 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:03.272 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.369662+0000", 2024-09-21T13:56:03.272 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:03.272 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.272 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.272 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:03.272 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.369662+0000", 2024-09-21T13:56:03.272 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:03.272 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.272 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.273 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:03.273 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.369664+0000", 2024-09-21T13:56:03.273 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.136e-06 2024-09-21T13:56:03.273 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.273 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.273 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:03.273 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.369671+0000", 2024-09-21T13:56:03.273 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9110000000000004e-06 2024-09-21T13:56:03.273 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.273 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.273 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:03.273 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.369689+0000", 2024-09-21T13:56:03.273 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7757999999999998e-05 2024-09-21T13:56:03.273 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.273 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.273 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:03.274 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.369700+0000", 2024-09-21T13:56:03.274 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.146e-05 2024-09-21T13:56:03.274 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.274 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.274 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:03.274 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.369779+0000", 2024-09-21T13:56:03.274 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9260999999999999e-05 2024-09-21T13:56:03.274 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.274 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.274 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:56:03.274 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.369793+0000", 2024-09-21T13:56:03.274 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3200000000000001e-05 2024-09-21T13:56:03.274 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.274 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.274 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:03.275 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:04.315126+0000", 2024-09-21T13:56:03.275 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.94533302500000005 2024-09-21T13:56:03.275 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.275 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.275 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:03.275 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:04.315136+0000", 2024-09-21T13:56:03.275 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0117e-05 2024-09-21T13:56:03.275 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:03.275 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:03.275 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:03.275 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.275 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.275 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:201459 7.8 7:1473b567:::benchmark_data_smithi049_76798_object65039:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-21T13:56:03.275 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:41.960649+0000", 2024-09-21T13:56:03.275 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 201.291777217, 2024-09-21T13:56:03.275 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.025332715, 2024-09-21T13:56:03.276 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:03.276 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:03.276 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:03.276 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:56:03.276 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:56:03.276 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:56:03.276 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 201459 2024-09-21T13:56:03.276 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.276 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:03.276 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.276 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:03.276 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:41.960649+0000", 2024-09-21T13:56:03.276 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:03.276 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.276 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.276 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:03.277 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:41.960649+0000", 2024-09-21T13:56:03.277 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:03.277 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.277 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.277 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:03.277 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:41.960651+0000", 2024-09-21T13:56:03.277 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7990000000000001e-06 2024-09-21T13:56:03.277 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.277 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.277 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:03.277 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:41.960657+0000", 2024-09-21T13:56:03.277 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2260000000000002e-06 2024-09-21T13:56:03.277 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.277 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.277 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:03.277 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:41.960677+0000", 2024-09-21T13:56:03.278 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9783999999999999e-05 2024-09-21T13:56:03.278 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.278 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.278 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:03.278 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:41.960686+0000", 2024-09-21T13:56:03.278 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0710000000000001e-06 2024-09-21T13:56:03.278 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.278 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.278 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:03.278 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:41.964807+0000", 2024-09-21T13:56:03.278 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0041212560000000002 2024-09-21T13:56:03.278 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.278 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.279 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:56:03.279 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:41.964817+0000", 2024-09-21T13:56:03.279 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0030000000000001e-05 2024-09-21T13:56:03.279 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.279 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.279 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:03.279 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.976018+0000", 2024-09-21T13:56:03.279 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.011200412 2024-09-21T13:56:03.279 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.279 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.279 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:56:03.279 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.976136+0000", 2024-09-21T13:56:03.279 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000118106 2024-09-21T13:56:03.279 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.279 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.279 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-09-21T13:56:03.280 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.976247+0000", 2024-09-21T13:56:03.280 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000111418 2024-09-21T13:56:03.280 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.280 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.280 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:03.280 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.980913+0000", 2024-09-21T13:56:03.280 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0046656850000000001 2024-09-21T13:56:03.280 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.280 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.280 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:03.280 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.980959+0000", 2024-09-21T13:56:03.280 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6402000000000001e-05 2024-09-21T13:56:03.280 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.280 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.280 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:56:03.281 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.985848+0000", 2024-09-21T13:56:03.281 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0048883210000000002 2024-09-21T13:56:03.281 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.281 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.281 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:56:03.281 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.985906+0000", 2024-09-21T13:56:03.281 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8242000000000002e-05 2024-09-21T13:56:03.281 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.281 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.281 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:03.281 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.985982+0000", 2024-09-21T13:56:03.281 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5963000000000004e-05 2024-09-21T13:56:03.281 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:03.281 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:03.282 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:03.282 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.282 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.282 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:201494 7.8 7:1c4e9ccf:::benchmark_data_smithi049_76798_object65074:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-21T13:56:03.282 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:42.006815+0000", 2024-09-21T13:56:03.282 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 201.24561081100001, 2024-09-21T13:56:03.282 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97934932200000002, 2024-09-21T13:56:03.282 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:03.282 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:03.282 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:03.282 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:56:03.282 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:56:03.282 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:56:03.282 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 201494 2024-09-21T13:56:03.283 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.283 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:03.283 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.283 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:03.283 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006815+0000", 2024-09-21T13:56:03.283 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:03.283 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.283 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.283 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:03.283 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006815+0000", 2024-09-21T13:56:03.283 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:03.283 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.283 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.283 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:03.284 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006817+0000", 2024-09-21T13:56:03.284 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.618e-06 2024-09-21T13:56:03.284 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.284 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.284 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:03.284 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006823+0000", 2024-09-21T13:56:03.284 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0859999999999998e-06 2024-09-21T13:56:03.284 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.284 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.284 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:03.284 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006840+0000", 2024-09-21T13:56:03.284 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7149000000000001e-05 2024-09-21T13:56:03.284 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.284 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.285 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:03.285 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006848+0000", 2024-09-21T13:56:03.285 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9340000000000006e-06 2024-09-21T13:56:03.285 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.285 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.285 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:03.285 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006949+0000", 2024-09-21T13:56:03.285 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010043399999999999 2024-09-21T13:56:03.285 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.285 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.285 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:56:03.285 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006957+0000", 2024-09-21T13:56:03.286 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6580000000000008e-06 2024-09-21T13:56:03.286 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.286 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.286 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:03.286 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.979032+0000", 2024-09-21T13:56:03.286 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97207428699999998 2024-09-21T13:56:03.286 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.286 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.286 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:56:03.286 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.979198+0000", 2024-09-21T13:56:03.286 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016622800000000001 2024-09-21T13:56:03.286 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.286 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.286 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-09-21T13:56:03.287 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.979306+0000", 2024-09-21T13:56:03.287 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000108356 2024-09-21T13:56:03.287 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.287 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.287 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:03.287 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.980997+0000", 2024-09-21T13:56:03.287 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0016909640000000001 2024-09-21T13:56:03.287 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.287 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.287 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:03.287 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.981060+0000", 2024-09-21T13:56:03.287 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2516999999999998e-05 2024-09-21T13:56:03.287 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.287 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.288 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:56:03.288 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986075+0000", 2024-09-21T13:56:03.288 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.005014971 2024-09-21T13:56:03.288 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.288 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.288 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:56:03.288 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986121+0000", 2024-09-21T13:56:03.288 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6313999999999997e-05 2024-09-21T13:56:03.288 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.288 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.288 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:03.288 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986165+0000", 2024-09-21T13:56:03.288 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3806e-05 2024-09-21T13:56:03.289 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:03.289 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:03.289 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:03.289 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.289 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.289 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:201495 7.8 7:18dd0180:::benchmark_data_smithi049_76798_object65075:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-21T13:56:03.289 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:42.006872+0000", 2024-09-21T13:56:03.289 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 201.24555373199999, 2024-09-21T13:56:03.289 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97938486800000002, 2024-09-21T13:56:03.289 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:03.289 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:03.289 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:03.289 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:56:03.290 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:56:03.290 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:56:03.290 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 201495 2024-09-21T13:56:03.290 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.290 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:03.290 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.290 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:03.290 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006872+0000", 2024-09-21T13:56:03.290 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:03.290 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.290 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.290 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:03.290 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006872+0000", 2024-09-21T13:56:03.290 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:03.291 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.291 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.291 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:03.291 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006873+0000", 2024-09-21T13:56:03.291 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4899999999999995e-07 2024-09-21T13:56:03.291 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.291 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.291 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:03.291 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006878+0000", 2024-09-21T13:56:03.291 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0429999999999998e-06 2024-09-21T13:56:03.291 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.291 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.291 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:03.292 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006886+0000", 2024-09-21T13:56:03.292 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3399999999999998e-06 2024-09-21T13:56:03.292 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.292 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.292 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:03.292 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006894+0000", 2024-09-21T13:56:03.292 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1359999999999999e-06 2024-09-21T13:56:03.292 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.292 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.292 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:03.292 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006982+0000", 2024-09-21T13:56:03.292 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.8022000000000005e-05 2024-09-21T13:56:03.292 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.293 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.293 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:56:03.293 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006988+0000", 2024-09-21T13:56:03.293 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3470000000000004e-06 2024-09-21T13:56:03.293 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.293 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.293 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:03.293 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.981113+0000", 2024-09-21T13:56:03.293 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97412548499999996 2024-09-21T13:56:03.293 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.293 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.293 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:56:03.293 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.981230+0000", 2024-09-21T13:56:03.293 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000116666 2024-09-21T13:56:03.293 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.294 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.294 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-09-21T13:56:03.294 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.981349+0000", 2024-09-21T13:56:03.294 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000118572 2024-09-21T13:56:03.294 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.294 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.294 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:03.294 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.985825+0000", 2024-09-21T13:56:03.294 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0044766739999999999 2024-09-21T13:56:03.294 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.294 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.294 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:03.294 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986063+0000", 2024-09-21T13:56:03.295 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00023781299999999999 2024-09-21T13:56:03.295 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.295 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.295 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:56:03.295 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986172+0000", 2024-09-21T13:56:03.295 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010923499999999999 2024-09-21T13:56:03.295 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.295 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.295 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:56:03.295 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986236+0000", 2024-09-21T13:56:03.295 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3606000000000001e-05 2024-09-21T13:56:03.295 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.295 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.295 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:03.296 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986257+0000", 2024-09-21T13:56:03.296 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1379999999999999e-05 2024-09-21T13:56:03.296 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:03.296 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:03.296 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:03.296 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.296 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.296 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:201496 7.8 7:10f836b0:::benchmark_data_smithi049_76798_object65076:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-21T13:56:03.296 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:42.012648+0000", 2024-09-21T13:56:03.296 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 201.23977776199999, 2024-09-21T13:56:03.296 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97380752599999998, 2024-09-21T13:56:03.296 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:03.296 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:03.297 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:03.297 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:56:03.297 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:56:03.297 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:56:03.297 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 201496 2024-09-21T13:56:03.297 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.297 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:03.297 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.297 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:03.297 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.012648+0000", 2024-09-21T13:56:03.298 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:03.298 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.298 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.298 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:03.298 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.012648+0000", 2024-09-21T13:56:03.298 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:03.298 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.298 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.298 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:03.298 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.012650+0000", 2024-09-21T13:56:03.299 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.513e-06 2024-09-21T13:56:03.299 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.299 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.299 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:03.299 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.012656+0000", 2024-09-21T13:56:03.299 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7300000000000002e-06 2024-09-21T13:56:03.299 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.299 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.299 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:03.299 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.012664+0000", 2024-09-21T13:56:03.299 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6090000000000001e-06 2024-09-21T13:56:03.299 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.300 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.300 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:03.300 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.012670+0000", 2024-09-21T13:56:03.300 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5829999999999997e-06 2024-09-21T13:56:03.300 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.300 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.300 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:03.300 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.012740+0000", 2024-09-21T13:56:03.300 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9834999999999998e-05 2024-09-21T13:56:03.300 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.300 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.300 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:56:03.300 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.012749+0000", 2024-09-21T13:56:03.300 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9889999999999998e-06 2024-09-21T13:56:03.301 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.301 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.301 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:03.301 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.983365+0000", 2024-09-21T13:56:03.301 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97061665600000002 2024-09-21T13:56:03.301 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.301 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.301 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:56:03.301 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.983473+0000", 2024-09-21T13:56:03.301 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000107507 2024-09-21T13:56:03.301 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.301 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.301 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-09-21T13:56:03.302 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.983595+0000", 2024-09-21T13:56:03.302 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000121614 2024-09-21T13:56:03.302 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.302 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.302 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:56:03.302 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986265+0000", 2024-09-21T13:56:03.302 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002670113 2024-09-21T13:56:03.302 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.302 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.302 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:03.302 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986301+0000", 2024-09-21T13:56:03.302 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6130999999999997e-05 2024-09-21T13:56:03.302 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.303 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.303 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:03.303 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986370+0000", 2024-09-21T13:56:03.303 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8994999999999999e-05 2024-09-21T13:56:03.303 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.303 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.303 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:56:03.303 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986417+0000", 2024-09-21T13:56:03.303 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7691000000000001e-05 2024-09-21T13:56:03.303 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.303 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.303 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:03.303 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986456+0000", 2024-09-21T13:56:03.303 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8559999999999997e-05 2024-09-21T13:56:03.303 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:03.304 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:03.304 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:03.304 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.304 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.304 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:201499 7.8 7:1329afa8:::benchmark_data_smithi049_76798_object65079:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-21T13:56:03.304 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:42.026955+0000", 2024-09-21T13:56:03.304 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 201.22547118700001, 2024-09-21T13:56:03.304 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97876118000000001, 2024-09-21T13:56:03.304 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:03.304 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:03.304 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:03.304 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:56:03.304 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:56:03.304 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:56:03.304 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 201499 2024-09-21T13:56:03.305 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.305 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:03.305 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.305 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:03.305 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.026955+0000", 2024-09-21T13:56:03.305 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:03.305 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.305 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.305 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:03.305 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.026955+0000", 2024-09-21T13:56:03.305 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:03.305 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.305 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.305 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:03.305 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.026957+0000", 2024-09-21T13:56:03.306 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7999999999999999e-06 2024-09-21T13:56:03.306 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.306 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.306 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:03.306 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.026962+0000", 2024-09-21T13:56:03.306 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9400000000000001e-06 2024-09-21T13:56:03.306 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.306 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.306 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:03.306 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.026977+0000", 2024-09-21T13:56:03.306 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5087e-05 2024-09-21T13:56:03.306 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.306 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.306 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:03.307 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.026983+0000", 2024-09-21T13:56:03.307 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0090000000000004e-06 2024-09-21T13:56:03.307 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.307 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.307 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:03.307 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027044+0000", 2024-09-21T13:56:03.307 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1543000000000004e-05 2024-09-21T13:56:03.307 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.307 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.307 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:56:03.307 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027051+0000", 2024-09-21T13:56:03.307 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4590000000000003e-06 2024-09-21T13:56:03.307 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.308 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.308 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:03.308 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986512+0000", 2024-09-21T13:56:03.308 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.95946141399999996 2024-09-21T13:56:03.308 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.308 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.308 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:56:03.308 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986631+0000", 2024-09-21T13:56:03.308 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011896300000000001 2024-09-21T13:56:03.308 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.308 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.308 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-09-21T13:56:03.308 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986738+0000", 2024-09-21T13:56:03.308 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000106766 2024-09-21T13:56:03.308 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.309 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.309 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:03.309 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.994021+0000", 2024-09-21T13:56:03.309 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0072827329999999996 2024-09-21T13:56:03.309 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.309 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.309 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:03.309 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.994059+0000", 2024-09-21T13:56:03.309 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8053999999999999e-05 2024-09-21T13:56:03.309 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.309 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.309 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:56:03.309 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.005582+0000", 2024-09-21T13:56:03.309 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.011523342000000001 2024-09-21T13:56:03.310 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.310 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.310 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:56:03.310 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.005671+0000", 2024-09-21T13:56:03.310 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.8708999999999994e-05 2024-09-21T13:56:03.310 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.310 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.310 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:03.310 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.005716+0000", 2024-09-21T13:56:03.310 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5361000000000002e-05 2024-09-21T13:56:03.310 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:03.310 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:03.310 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:03.311 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.311 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.311 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:201500 7.8 7:19fe3d31:::benchmark_data_smithi049_76798_object65080:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-21T13:56:03.311 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:42.027000+0000", 2024-09-21T13:56:03.311 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 201.225425747, 2024-09-21T13:56:03.311 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97887608500000001, 2024-09-21T13:56:03.311 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:03.311 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:03.311 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:03.311 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:56:03.311 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:56:03.311 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:56:03.311 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 201500 2024-09-21T13:56:03.311 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.312 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:03.312 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.312 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:03.312 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027000+0000", 2024-09-21T13:56:03.312 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:03.312 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.312 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.312 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:03.312 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027000+0000", 2024-09-21T13:56:03.312 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:03.312 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.312 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.312 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:03.312 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027001+0000", 2024-09-21T13:56:03.313 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3200000000000001e-07 2024-09-21T13:56:03.313 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.313 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.313 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:03.313 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027003+0000", 2024-09-21T13:56:03.313 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8980000000000001e-06 2024-09-21T13:56:03.313 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.313 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.313 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:03.313 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027007+0000", 2024-09-21T13:56:03.313 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4290000000000003e-06 2024-09-21T13:56:03.313 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.313 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.313 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:03.314 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027010+0000", 2024-09-21T13:56:03.314 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3419999999999999e-06 2024-09-21T13:56:03.314 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.314 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.314 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:03.314 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027064+0000", 2024-09-21T13:56:03.314 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3474000000000003e-05 2024-09-21T13:56:03.314 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.314 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.314 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:56:03.314 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027067+0000", 2024-09-21T13:56:03.314 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.932e-06 2024-09-21T13:56:03.314 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.315 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.315 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:03.315 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.989128+0000", 2024-09-21T13:56:03.315 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.96206106899999999 2024-09-21T13:56:03.315 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.315 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.315 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:56:03.315 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.990066+0000", 2024-09-21T13:56:03.315 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00093830900000000002 2024-09-21T13:56:03.315 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.315 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.315 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-09-21T13:56:03.316 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.990172+0000", 2024-09-21T13:56:03.316 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000105271 2024-09-21T13:56:03.316 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.316 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.316 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:03.316 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.996356+0000", 2024-09-21T13:56:03.316 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0061840100000000002 2024-09-21T13:56:03.316 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.316 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.316 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:03.316 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.996388+0000", 2024-09-21T13:56:03.316 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2539000000000001e-05 2024-09-21T13:56:03.316 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.317 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.317 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:56:03.317 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.005774+0000", 2024-09-21T13:56:03.317 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0093859679999999997 2024-09-21T13:56:03.317 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.317 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.317 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:56:03.317 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.005838+0000", 2024-09-21T13:56:03.317 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3724000000000002e-05 2024-09-21T13:56:03.317 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.317 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.317 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:03.317 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.005877+0000", 2024-09-21T13:56:03.317 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8788000000000002e-05 2024-09-21T13:56:03.318 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:03.318 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:03.318 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:03.318 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.318 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.318 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:201501 7.8 7:15f77bbd:::benchmark_data_smithi049_76798_object65081:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-21T13:56:03.318 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:42.027294+0000", 2024-09-21T13:56:03.318 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 201.22513178200001, 2024-09-21T13:56:03.318 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97870270599999998, 2024-09-21T13:56:03.318 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:03.318 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:03.318 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:03.318 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:56:03.318 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:56:03.319 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:56:03.319 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 201501 2024-09-21T13:56:03.319 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.319 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:03.319 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.319 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:03.319 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027294+0000", 2024-09-21T13:56:03.319 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:03.319 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.319 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.319 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:03.319 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027294+0000", 2024-09-21T13:56:03.319 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:03.319 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.319 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.320 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:03.320 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027295+0000", 2024-09-21T13:56:03.320 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7899999999999997e-07 2024-09-21T13:56:03.320 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.320 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.320 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:03.320 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027299+0000", 2024-09-21T13:56:03.320 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6449999999999998e-06 2024-09-21T13:56:03.320 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.320 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.320 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:03.320 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027310+0000", 2024-09-21T13:56:03.320 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0699999999999999e-05 2024-09-21T13:56:03.320 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.321 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.321 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:03.321 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027315+0000", 2024-09-21T13:56:03.321 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9729999999999997e-06 2024-09-21T13:56:03.321 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.321 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.321 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:03.321 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027363+0000", 2024-09-21T13:56:03.321 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8096000000000002e-05 2024-09-21T13:56:03.321 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.321 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.321 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:56:03.321 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027367+0000", 2024-09-21T13:56:03.321 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2930000000000002e-06 2024-09-21T13:56:03.321 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.322 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.322 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:03.322 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.992098+0000", 2024-09-21T13:56:03.322 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.96473078800000001 2024-09-21T13:56:03.322 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.322 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.322 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:56:03.322 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.992160+0000", 2024-09-21T13:56:03.322 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2133999999999998e-05 2024-09-21T13:56:03.322 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.322 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.322 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-09-21T13:56:03.322 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.992220+0000", 2024-09-21T13:56:03.322 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9911999999999998e-05 2024-09-21T13:56:03.323 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.323 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.323 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:03.323 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.003058+0000", 2024-09-21T13:56:03.323 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.010838441000000001 2024-09-21T13:56:03.323 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.323 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.323 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:03.323 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.003121+0000", 2024-09-21T13:56:03.323 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3118000000000001e-05 2024-09-21T13:56:03.323 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.323 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.323 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:56:03.323 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.005887+0000", 2024-09-21T13:56:03.324 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0027659500000000001 2024-09-21T13:56:03.324 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.324 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.324 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:56:03.324 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.005958+0000", 2024-09-21T13:56:03.324 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0842999999999999e-05 2024-09-21T13:56:03.324 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.324 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.324 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:03.324 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.005997+0000", 2024-09-21T13:56:03.324 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9033999999999999e-05 2024-09-21T13:56:03.324 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:03.324 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:03.324 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:03.324 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.325 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.325 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:201503 7.8 7:1bb4326b:::benchmark_data_smithi049_76798_object65083:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-21T13:56:03.325 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:42.027327+0000", 2024-09-21T13:56:03.325 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 201.22509970600001, 2024-09-21T13:56:03.325 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97877131900000003, 2024-09-21T13:56:03.325 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:03.325 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:03.326 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:03.326 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:56:03.326 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:56:03.326 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:56:03.326 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 201503 2024-09-21T13:56:03.326 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.326 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:03.327 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.327 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:03.327 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027327+0000", 2024-09-21T13:56:03.327 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:03.327 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.327 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.327 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:03.327 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027327+0000", 2024-09-21T13:56:03.327 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:03.327 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.327 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.327 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:03.327 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027327+0000", 2024-09-21T13:56:03.327 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4799999999999999e-07 2024-09-21T13:56:03.327 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.328 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.328 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:03.328 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027329+0000", 2024-09-21T13:56:03.328 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7400000000000001e-06 2024-09-21T13:56:03.328 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.328 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.328 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:03.328 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027333+0000", 2024-09-21T13:56:03.328 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0790000000000002e-06 2024-09-21T13:56:03.328 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.328 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.328 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:03.328 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027336+0000", 2024-09-21T13:56:03.329 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1190000000000001e-06 2024-09-21T13:56:03.329 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.329 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.329 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:03.329 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027379+0000", 2024-09-21T13:56:03.329 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3000000000000002e-05 2024-09-21T13:56:03.329 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.329 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.329 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:56:03.329 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027382+0000", 2024-09-21T13:56:03.329 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9399999999999998e-06 2024-09-21T13:56:03.329 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.329 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.329 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:03.329 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.994085+0000", 2024-09-21T13:56:03.330 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.96670302799999996 2024-09-21T13:56:03.330 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.330 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.330 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:56:03.330 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.994181+0000", 2024-09-21T13:56:03.330 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6340000000000003e-05 2024-09-21T13:56:03.330 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.330 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.330 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-09-21T13:56:03.330 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.994285+0000", 2024-09-21T13:56:03.330 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000103947 2024-09-21T13:56:03.330 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.330 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.331 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:03.331 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.003091+0000", 2024-09-21T13:56:03.331 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0088057599999999993 2024-09-21T13:56:03.331 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.331 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.331 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:03.331 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.003150+0000", 2024-09-21T13:56:03.331 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9670000000000003e-05 2024-09-21T13:56:03.331 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.331 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.331 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:56:03.331 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.006006+0000", 2024-09-21T13:56:03.331 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002855734 2024-09-21T13:56:03.331 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.332 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.332 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:56:03.332 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.006056+0000", 2024-09-21T13:56:03.332 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9619e-05 2024-09-21T13:56:03.332 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.332 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.332 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:03.332 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.006098+0000", 2024-09-21T13:56:03.332 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1995e-05 2024-09-21T13:56:03.332 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:03.332 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:03.332 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:03.332 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.332 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.333 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:201514 7.8 7:1c8e529c:::benchmark_data_smithi049_76798_object65094:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-21T13:56:03.333 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:42.044200+0000", 2024-09-21T13:56:03.333 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 201.208226599, 2024-09-21T13:56:03.333 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.96199924699999995, 2024-09-21T13:56:03.333 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:03.333 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:03.333 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:03.333 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:56:03.333 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:56:03.333 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:56:03.333 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 201514 2024-09-21T13:56:03.333 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.333 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:03.334 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.334 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:03.334 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.044200+0000", 2024-09-21T13:56:03.334 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:03.334 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.334 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.334 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:03.334 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.044200+0000", 2024-09-21T13:56:03.334 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:03.334 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.335 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.335 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:03.335 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.044201+0000", 2024-09-21T13:56:03.335 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7179999999999999e-06 2024-09-21T13:56:03.335 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.335 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.335 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:03.335 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.044208+0000", 2024-09-21T13:56:03.335 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1800000000000001e-06 2024-09-21T13:56:03.335 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.335 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.335 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:03.335 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.044226+0000", 2024-09-21T13:56:03.336 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8216999999999999e-05 2024-09-21T13:56:03.336 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.336 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.336 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:03.336 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.044235+0000", 2024-09-21T13:56:03.336 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1020000000000003e-06 2024-09-21T13:56:03.336 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.336 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.336 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:03.336 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.044482+0000", 2024-09-21T13:56:03.336 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00024732699999999999 2024-09-21T13:56:03.336 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.336 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.336 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:56:03.337 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.044499+0000", 2024-09-21T13:56:03.337 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7129000000000002e-05 2024-09-21T13:56:03.337 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.337 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.337 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:03.337 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.996416+0000", 2024-09-21T13:56:03.337 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.95191657200000002 2024-09-21T13:56:03.337 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.337 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.337 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:56:03.337 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.996528+0000", 2024-09-21T13:56:03.337 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000112118 2024-09-21T13:56:03.337 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.337 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.338 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-09-21T13:56:03.338 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.996652+0000", 2024-09-21T13:56:03.338 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012430500000000001 2024-09-21T13:56:03.338 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.338 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.338 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:03.338 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.003180+0000", 2024-09-21T13:56:03.338 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.006528056 2024-09-21T13:56:03.338 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.338 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.338 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:03.338 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.005760+0000", 2024-09-21T13:56:03.338 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0025797670000000002 2024-09-21T13:56:03.339 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.339 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.339 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:56:03.339 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.006108+0000", 2024-09-21T13:56:03.339 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00034834500000000001 2024-09-21T13:56:03.339 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.339 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.339 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:56:03.339 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.006155+0000", 2024-09-21T13:56:03.339 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6513e-05 2024-09-21T13:56:03.339 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.339 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.339 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:03.339 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.006199+0000", 2024-09-21T13:56:03.340 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3897999999999999e-05 2024-09-21T13:56:03.340 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:03.340 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:03.340 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:03.340 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.340 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.340 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:201539 7.8 7:1afa34ae:::benchmark_data_smithi049_76798_object65119:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-21T13:56:03.340 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:42.081266+0000", 2024-09-21T13:56:03.340 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 201.17116049099999, 2024-09-21T13:56:03.340 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.92917915600000001, 2024-09-21T13:56:03.340 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:03.340 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:03.340 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:03.340 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:56:03.341 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:56:03.341 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:56:03.341 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 201539 2024-09-21T13:56:03.341 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.341 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:03.341 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.341 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:03.341 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.081266+0000", 2024-09-21T13:56:03.341 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:03.341 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.341 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.341 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:03.341 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.081266+0000", 2024-09-21T13:56:03.342 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:03.342 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.342 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.342 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:03.342 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.081267+0000", 2024-09-21T13:56:03.342 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6559999999999999e-06 2024-09-21T13:56:03.342 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.342 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.342 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:03.342 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.081273+0000", 2024-09-21T13:56:03.342 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0839999999999997e-06 2024-09-21T13:56:03.342 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.342 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.343 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:03.343 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.081291+0000", 2024-09-21T13:56:03.343 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7827999999999999e-05 2024-09-21T13:56:03.343 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.343 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.343 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:03.343 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.081300+0000", 2024-09-21T13:56:03.343 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.7190000000000007e-06 2024-09-21T13:56:03.343 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.343 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.343 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:03.343 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.081360+0000", 2024-09-21T13:56:03.343 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9914000000000003e-05 2024-09-21T13:56:03.343 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.343 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.343 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:56:03.344 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.081369+0000", 2024-09-21T13:56:03.344 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1630000000000002e-06 2024-09-21T13:56:03.344 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.344 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.344 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:03.344 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.998700+0000", 2024-09-21T13:56:03.344 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.91733049300000002 2024-09-21T13:56:03.344 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.344 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.344 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:56:03.344 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.998798+0000", 2024-09-21T13:56:03.344 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8494999999999998e-05 2024-09-21T13:56:03.344 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.345 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.345 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-09-21T13:56:03.345 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.998903+0000", 2024-09-21T13:56:03.345 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000104782 2024-09-21T13:56:03.345 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.345 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.345 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:56:03.345 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.006208+0000", 2024-09-21T13:56:03.345 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0073052270000000001 2024-09-21T13:56:03.345 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.345 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.345 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:03.345 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.006259+0000", 2024-09-21T13:56:03.345 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1205999999999997e-05 2024-09-21T13:56:03.345 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.345 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.345 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:03.346 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.010350+0000", 2024-09-21T13:56:03.346 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0040908560000000004 2024-09-21T13:56:03.346 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.346 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.346 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:56:03.346 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.010400+0000", 2024-09-21T13:56:03.346 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9901000000000002e-05 2024-09-21T13:56:03.346 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.346 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.346 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:03.346 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.010445+0000", 2024-09-21T13:56:03.346 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4832e-05 2024-09-21T13:56:03.346 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:03.346 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:03.347 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:03.347 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.347 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.347 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:201541 7.8 7:135037ad:::benchmark_data_smithi049_76798_object65121:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-21T13:56:03.347 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:42.083222+0000", 2024-09-21T13:56:03.347 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 201.16920424, 2024-09-21T13:56:03.347 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.92733551700000005, 2024-09-21T13:56:03.347 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:03.347 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:03.347 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:03.347 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:56:03.347 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:56:03.347 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:56:03.347 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 201541 2024-09-21T13:56:03.348 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.348 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:03.348 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.348 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:03.348 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.083222+0000", 2024-09-21T13:56:03.348 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:03.348 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.348 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.348 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:03.348 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.083222+0000", 2024-09-21T13:56:03.348 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:03.348 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.348 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.348 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:03.348 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.083223+0000", 2024-09-21T13:56:03.349 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0020000000000001e-06 2024-09-21T13:56:03.349 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.349 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.349 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:03.349 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.083227+0000", 2024-09-21T13:56:03.349 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.636e-06 2024-09-21T13:56:03.349 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.349 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.349 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:03.349 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.083236+0000", 2024-09-21T13:56:03.349 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0180000000000007e-06 2024-09-21T13:56:03.349 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.349 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.349 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:03.350 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.083242+0000", 2024-09-21T13:56:03.350 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.891e-06 2024-09-21T13:56:03.350 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.350 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.350 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:03.350 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.083385+0000", 2024-09-21T13:56:03.350 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014374099999999999 2024-09-21T13:56:03.350 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.350 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.350 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:56:03.350 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.083394+0000", 2024-09-21T13:56:03.350 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0839999999999998e-06 2024-09-21T13:56:03.350 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.350 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.350 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:03.350 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.000839+0000", 2024-09-21T13:56:03.351 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.917444861 2024-09-21T13:56:03.351 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.351 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.351 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:56:03.351 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.000934+0000", 2024-09-21T13:56:03.351 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4943000000000002e-05 2024-09-21T13:56:03.351 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.351 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.351 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-09-21T13:56:03.351 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.001036+0000", 2024-09-21T13:56:03.351 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000101817 2024-09-21T13:56:03.351 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.351 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.351 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:56:03.351 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.006215+0000", 2024-09-21T13:56:03.352 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0051790020000000003 2024-09-21T13:56:03.352 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.352 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.352 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:03.352 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.006324+0000", 2024-09-21T13:56:03.352 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010858599999999999 2024-09-21T13:56:03.352 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.352 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.352 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:03.352 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.010481+0000", 2024-09-21T13:56:03.352 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0041575300000000004 2024-09-21T13:56:03.352 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.352 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.352 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:56:03.352 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.010528+0000", 2024-09-21T13:56:03.352 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6418000000000002e-05 2024-09-21T13:56:03.353 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.353 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.353 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:03.353 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.010558+0000", 2024-09-21T13:56:03.353 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9988000000000002e-05 2024-09-21T13:56:03.353 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:03.353 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:03.353 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:03.353 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.353 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.354 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:229964 7.e 7:7be4d3ef:::benchmark_data_smithi049_76798_object93544:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-21T13:56:03.354 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:53:16.693433+0000", 2024-09-21T13:56:03.354 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 166.55899367500001, 2024-09-21T13:56:03.354 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.99823727600000001, 2024-09-21T13:56:03.354 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:03.354 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:03.354 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:03.354 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:56:03.354 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:56:03.354 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:56:03.354 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 229964 2024-09-21T13:56:03.354 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.355 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:03.355 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.355 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:03.355 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.693433+0000", 2024-09-21T13:56:03.355 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:03.355 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.355 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.355 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:03.355 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.693433+0000", 2024-09-21T13:56:03.355 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:03.355 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.355 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.355 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:03.355 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.693434+0000", 2024-09-21T13:56:03.356 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3799999999999999e-06 2024-09-21T13:56:03.356 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.356 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.356 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:03.356 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.693438+0000", 2024-09-21T13:56:03.356 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.515e-06 2024-09-21T13:56:03.356 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.356 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.356 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:03.356 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.693455+0000", 2024-09-21T13:56:03.356 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6614000000000001e-05 2024-09-21T13:56:03.356 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.356 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.356 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:03.356 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.693463+0000", 2024-09-21T13:56:03.357 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9850000000000006e-06 2024-09-21T13:56:03.357 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.357 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.357 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:03.357 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.693542+0000", 2024-09-21T13:56:03.357 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8942000000000004e-05 2024-09-21T13:56:03.357 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.357 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.357 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:56:03.357 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.693550+0000", 2024-09-21T13:56:03.357 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7379999999999994e-06 2024-09-21T13:56:03.357 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.357 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.357 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:03.357 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.686321+0000", 2024-09-21T13:56:03.357 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.99277167399999999 2024-09-21T13:56:03.358 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.358 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.358 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:56:03.358 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.686413+0000", 2024-09-21T13:56:03.358 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1822000000000003e-05 2024-09-21T13:56:03.358 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.358 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.358 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,5", 2024-09-21T13:56:03.358 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.686547+0000", 2024-09-21T13:56:03.358 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000133516 2024-09-21T13:56:03.358 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.358 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.358 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:03.358 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.691512+0000", 2024-09-21T13:56:03.358 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0049651410000000002 2024-09-21T13:56:03.358 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.359 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.359 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:03.359 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.691558+0000", 2024-09-21T13:56:03.359 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6047999999999997e-05 2024-09-21T13:56:03.359 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.359 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.359 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:56:03.359 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.691610+0000", 2024-09-21T13:56:03.359 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1644000000000002e-05 2024-09-21T13:56:03.359 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.359 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.359 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:56:03.359 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.691642+0000", 2024-09-21T13:56:03.359 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2542000000000001e-05 2024-09-21T13:56:03.359 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.359 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.360 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:03.360 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.691670+0000", 2024-09-21T13:56:03.360 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.7715000000000001e-05 2024-09-21T13:56:03.360 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:03.360 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:03.360 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:03.360 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.360 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.360 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:230040 7.e 7:74df0b23:::benchmark_data_smithi049_76798_object93620:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-21T13:56:03.360 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:53:16.758488+0000", 2024-09-21T13:56:03.360 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 166.49393861799999, 2024-09-21T13:56:03.360 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.93513229499999995, 2024-09-21T13:56:03.360 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:03.360 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:03.361 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:03.361 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:56:03.361 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:56:03.361 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:56:03.361 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 230040 2024-09-21T13:56:03.361 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.361 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:03.361 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.361 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:03.361 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.758488+0000", 2024-09-21T13:56:03.361 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:03.361 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.361 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.361 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:03.361 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.758488+0000", 2024-09-21T13:56:03.361 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:03.361 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.362 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.362 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:03.362 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.758502+0000", 2024-09-21T13:56:03.362 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4207e-05 2024-09-21T13:56:03.362 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.362 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.362 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:03.362 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.758507+0000", 2024-09-21T13:56:03.362 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9429999999999999e-06 2024-09-21T13:56:03.362 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.362 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.362 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:03.362 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.758518+0000", 2024-09-21T13:56:03.362 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1447999999999999e-05 2024-09-21T13:56:03.362 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.363 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.363 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:03.363 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.758525+0000", 2024-09-21T13:56:03.363 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2389999999999996e-06 2024-09-21T13:56:03.363 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.363 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.363 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:03.363 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.758589+0000", 2024-09-21T13:56:03.363 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3262000000000005e-05 2024-09-21T13:56:03.363 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.363 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.363 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:56:03.363 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.758597+0000", 2024-09-21T13:56:03.363 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4100000000000008e-06 2024-09-21T13:56:03.363 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.364 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.364 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:03.364 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.689523+0000", 2024-09-21T13:56:03.364 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.93092584099999998 2024-09-21T13:56:03.364 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.364 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.364 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:56:03.364 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.689620+0000", 2024-09-21T13:56:03.364 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7081000000000001e-05 2024-09-21T13:56:03.364 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.364 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.364 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,5", 2024-09-21T13:56:03.364 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.689705+0000", 2024-09-21T13:56:03.365 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5074000000000006e-05 2024-09-21T13:56:03.365 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.365 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.365 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:03.365 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.691602+0000", 2024-09-21T13:56:03.365 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001896951 2024-09-21T13:56:03.365 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.365 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.365 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:03.365 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.693564+0000", 2024-09-21T13:56:03.365 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0019616630000000002 2024-09-21T13:56:03.365 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.365 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.365 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:56:03.366 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.693572+0000", 2024-09-21T13:56:03.366 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9160000000000001e-06 2024-09-21T13:56:03.366 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.366 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.366 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:56:03.366 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.693597+0000", 2024-09-21T13:56:03.366 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5812999999999998e-05 2024-09-21T13:56:03.366 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.366 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.366 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:03.366 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.693620+0000", 2024-09-21T13:56:03.366 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2447e-05 2024-09-21T13:56:03.366 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:03.366 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:03.366 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:03.367 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.367 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.367 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:230048 7.e 7:7e0d246c:::benchmark_data_smithi049_76798_object93628:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-21T13:56:03.367 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:53:16.763844+0000", 2024-09-21T13:56:03.367 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 166.48858173900001, 2024-09-21T13:56:03.367 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.93491194899999996, 2024-09-21T13:56:03.367 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:03.367 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:03.367 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:03.367 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:56:03.367 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:56:03.367 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:56:03.367 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 230048 2024-09-21T13:56:03.367 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.367 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:03.368 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.368 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:03.368 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.763844+0000", 2024-09-21T13:56:03.368 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:03.368 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.368 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.368 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:03.368 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.763844+0000", 2024-09-21T13:56:03.368 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:03.368 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.368 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.368 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:03.368 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.763846+0000", 2024-09-21T13:56:03.368 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6330000000000001e-06 2024-09-21T13:56:03.368 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.368 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.368 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:03.369 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.763852+0000", 2024-09-21T13:56:03.369 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3839999999999996e-06 2024-09-21T13:56:03.369 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.369 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.369 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:03.369 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.763915+0000", 2024-09-21T13:56:03.369 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3306999999999995e-05 2024-09-21T13:56:03.369 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.369 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.369 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:03.369 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.764047+0000", 2024-09-21T13:56:03.369 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013207599999999999 2024-09-21T13:56:03.369 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.369 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.369 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:03.369 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.764104+0000", 2024-09-21T13:56:03.370 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6758000000000001e-05 2024-09-21T13:56:03.370 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.370 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.370 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:56:03.370 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.764113+0000", 2024-09-21T13:56:03.370 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3700000000000001e-06 2024-09-21T13:56:03.370 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.370 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.370 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:03.370 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.691702+0000", 2024-09-21T13:56:03.370 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.92758885899999999 2024-09-21T13:56:03.370 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.370 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.370 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:56:03.370 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.691822+0000", 2024-09-21T13:56:03.370 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011997699999999999 2024-09-21T13:56:03.371 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.371 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.371 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,5", 2024-09-21T13:56:03.371 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.691939+0000", 2024-09-21T13:56:03.371 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011752 2024-09-21T13:56:03.371 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.371 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.371 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:56:03.371 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.698556+0000", 2024-09-21T13:56:03.371 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0066170739999999997 2024-09-21T13:56:03.371 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.371 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.371 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:03.371 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.698612+0000", 2024-09-21T13:56:03.371 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5362000000000003e-05 2024-09-21T13:56:03.371 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.372 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.372 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:03.372 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.698659+0000", 2024-09-21T13:56:03.372 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6931999999999997e-05 2024-09-21T13:56:03.372 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.372 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.372 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:56:03.372 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.698714+0000", 2024-09-21T13:56:03.372 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5402000000000003e-05 2024-09-21T13:56:03.372 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.372 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.372 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:03.372 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.698756+0000", 2024-09-21T13:56:03.372 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2295000000000001e-05 2024-09-21T13:56:03.372 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:03.372 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:03.372 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:03.373 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.373 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.373 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:230051 7.e 7:71fa5c2a:::benchmark_data_smithi049_76798_object93631:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-21T13:56:03.373 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:53:16.764458+0000", 2024-09-21T13:56:03.373 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 166.48796829400001, 2024-09-21T13:56:03.373 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.94179994899999997, 2024-09-21T13:56:03.373 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:03.373 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:03.373 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:03.373 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:56:03.373 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:56:03.373 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:56:03.373 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 230051 2024-09-21T13:56:03.373 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.373 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:03.373 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.374 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:03.374 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.764458+0000", 2024-09-21T13:56:03.374 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:03.374 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.374 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.374 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:03.374 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.764458+0000", 2024-09-21T13:56:03.374 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:03.374 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.374 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.374 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:03.374 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.764459+0000", 2024-09-21T13:56:03.374 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9400000000000004e-07 2024-09-21T13:56:03.374 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.374 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.375 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:03.375 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.764463+0000", 2024-09-21T13:56:03.375 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1960000000000001e-06 2024-09-21T13:56:03.375 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.375 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.375 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:03.375 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.764477+0000", 2024-09-21T13:56:03.375 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4426e-05 2024-09-21T13:56:03.375 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.375 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.375 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:03.375 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.764485+0000", 2024-09-21T13:56:03.375 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.148e-06 2024-09-21T13:56:03.375 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.375 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.375 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:03.375 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.764545+0000", 2024-09-21T13:56:03.376 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0278999999999997e-05 2024-09-21T13:56:03.376 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.376 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.376 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:56:03.376 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.764553+0000", 2024-09-21T13:56:03.376 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9170000000000006e-06 2024-09-21T13:56:03.376 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.376 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.376 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:03.376 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.693660+0000", 2024-09-21T13:56:03.376 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.92910747900000001 2024-09-21T13:56:03.376 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.376 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.376 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:56:03.376 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.693785+0000", 2024-09-21T13:56:03.376 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012436000000000001 2024-09-21T13:56:03.377 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.377 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.377 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,5", 2024-09-21T13:56:03.377 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.693907+0000", 2024-09-21T13:56:03.377 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012208600000000001 2024-09-21T13:56:03.377 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.377 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.377 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:03.377 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.698816+0000", 2024-09-21T13:56:03.377 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0049091279999999996 2024-09-21T13:56:03.377 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.377 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.377 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:56:03.377 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.698834+0000", 2024-09-21T13:56:03.377 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8236999999999998e-05 2024-09-21T13:56:03.377 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.378 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.378 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:03.378 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.706148+0000", 2024-09-21T13:56:03.378 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0073139709999999998 2024-09-21T13:56:03.378 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.378 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.378 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:56:03.378 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.706202+0000", 2024-09-21T13:56:03.378 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3779000000000001e-05 2024-09-21T13:56:03.378 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.378 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.378 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:03.378 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.706258+0000", 2024-09-21T13:56:03.378 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6048999999999998e-05 2024-09-21T13:56:03.379 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:03.379 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:03.379 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:03.379 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.379 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.379 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:230060 7.e 7:7289055b:::benchmark_data_smithi049_76798_object93640:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-21T13:56:03.379 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:53:16.772333+0000", 2024-09-21T13:56:03.379 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 166.48009317399999, 2024-09-21T13:56:03.379 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.93409186700000002, 2024-09-21T13:56:03.379 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:03.379 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:03.379 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:03.379 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:56:03.379 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:56:03.379 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:56:03.380 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 230060 2024-09-21T13:56:03.380 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.380 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:03.380 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.380 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:03.380 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.772333+0000", 2024-09-21T13:56:03.380 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:03.380 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.380 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.380 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:03.380 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.772333+0000", 2024-09-21T13:56:03.380 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:03.380 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.380 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.380 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:03.381 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.772334+0000", 2024-09-21T13:56:03.381 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1540000000000001e-06 2024-09-21T13:56:03.381 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.381 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.381 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:03.381 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.772339+0000", 2024-09-21T13:56:03.381 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8099999999999997e-06 2024-09-21T13:56:03.381 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.381 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.381 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:03.381 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.772351+0000", 2024-09-21T13:56:03.381 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2435999999999999e-05 2024-09-21T13:56:03.381 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.381 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.382 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:03.382 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.772359+0000", 2024-09-21T13:56:03.382 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7230000000000004e-06 2024-09-21T13:56:03.382 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.382 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.382 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:03.382 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.772435+0000", 2024-09-21T13:56:03.382 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5465999999999999e-05 2024-09-21T13:56:03.382 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.382 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.382 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:56:03.382 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.772443+0000", 2024-09-21T13:56:03.382 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9160000000000001e-06 2024-09-21T13:56:03.382 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.383 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.383 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:03.383 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.695975+0000", 2024-09-21T13:56:03.383 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.92353224599999995 2024-09-21T13:56:03.383 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.383 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.383 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:56:03.383 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.696104+0000", 2024-09-21T13:56:03.383 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012895499999999999 2024-09-21T13:56:03.383 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.383 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.383 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,5", 2024-09-21T13:56:03.383 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.696217+0000", 2024-09-21T13:56:03.383 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011362399999999999 2024-09-21T13:56:03.384 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.384 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.384 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:56:03.384 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.701076+0000", 2024-09-21T13:56:03.384 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0048582729999999998 2024-09-21T13:56:03.384 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.384 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.384 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:03.384 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.706310+0000", 2024-09-21T13:56:03.384 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0052347419999999997 2024-09-21T13:56:03.384 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.384 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.384 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:03.384 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.706347+0000", 2024-09-21T13:56:03.385 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6492999999999998e-05 2024-09-21T13:56:03.385 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.385 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.385 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:56:03.385 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.706392+0000", 2024-09-21T13:56:03.385 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5148999999999998e-05 2024-09-21T13:56:03.385 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:03.385 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:03.385 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:03.385 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.706425+0000", 2024-09-21T13:56:03.385 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2879999999999997e-05 2024-09-21T13:56:03.385 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:03.385 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:03.385 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:03.386 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:03.386 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:03.386 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T13:56:03.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:03.387 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:03.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:03.488 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:03.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:03.523 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:03.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:03.624 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:03.657 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:03.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:03.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:03.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:03 smithi049 ceph-mon[30080]: pgmap v344: 148 pgs: 1 active+clean+remapped, 38 active+undersized, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 30 active+undersized+degraded, 75 active+clean; 7.0 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 945 op/s; 42970/342549 objects degraded (12.544%); 23440/342549 objects misplaced (6.843%); 18 MiB/s, 280 objects/s recovering 2024-09-21T13:56:03.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:03 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:56:03.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:03 smithi049 ceph-mon[40489]: pgmap v344: 148 pgs: 1 active+clean+remapped, 38 active+undersized, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 30 active+undersized+degraded, 75 active+clean; 7.0 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 945 op/s; 42970/342549 objects degraded (12.544%); 23440/342549 objects misplaced (6.843%); 18 MiB/s, 280 objects/s recovering 2024-09-21T13:56:03.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:03 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:56:03.758 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:03.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:03.800 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:03.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:03.901 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:56:03.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:03.926 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:03.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:56:03.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:03 smithi187 ceph-mon[33032]: pgmap v344: 148 pgs: 1 active+clean+remapped, 38 active+undersized, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 30 active+undersized+degraded, 75 active+clean; 7.0 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 945 op/s; 42970/342549 objects degraded (12.544%); 23440/342549 objects misplaced (6.843%); 18 MiB/s, 280 objects/s recovering 2024-09-21T13:56:03.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:03 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:56:04.027 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:56:04.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:04.063 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:04.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:56:04.164 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:56:04.200 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:04.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:04.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:56:04.302 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:04.308 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T13:56:04.334 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:04.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:04.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:04.436 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:56:04.461 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:04.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:04.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:56:04.562 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:04.603 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:04.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:04.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:04.705 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:04.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:04.743 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:04.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:04.844 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:04.882 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:04.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:04.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:04.984 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:05.027 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:05.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:05.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:05.129 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:05.164 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:05.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:05.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:05.255 INFO:teuthology.orchestra.run.smithi049.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-21T13:56:05.266 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:56:05.301 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:05.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:05.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:56:05.403 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:05.440 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:05.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:05.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:05.542 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:05.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:05.575 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:05.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:05.676 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:05.716 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:05.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:05.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:05.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:05 smithi049 ceph-mon[40489]: pgmap v345: 148 pgs: 1 active+clean+remapped, 38 active+undersized, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 30 active+undersized+degraded, 75 active+clean; 6.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.56k op/s; 40914/325464 objects degraded (12.571%); 20288/325464 objects misplaced (6.234%); 32 MiB/s, 518 objects/s recovering 2024-09-21T13:56:05.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:05 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 40914/325464 objects degraded (12.571%), 31 pgs degraded (PG_DEGRADED) 2024-09-21T13:56:05.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:05 smithi049 ceph-mon[30080]: pgmap v345: 148 pgs: 1 active+clean+remapped, 38 active+undersized, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 30 active+undersized+degraded, 75 active+clean; 6.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.56k op/s; 40914/325464 objects degraded (12.571%); 20288/325464 objects misplaced (6.234%); 32 MiB/s, 518 objects/s recovering 2024-09-21T13:56:05.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:05 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 40914/325464 objects degraded (12.571%), 31 pgs degraded (PG_DEGRADED) 2024-09-21T13:56:05.818 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:56:05.854 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:05.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:05.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:56:05.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:05 smithi187 ceph-mon[33032]: pgmap v345: 148 pgs: 1 active+clean+remapped, 38 active+undersized, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 30 active+undersized+degraded, 75 active+clean; 6.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.56k op/s; 40914/325464 objects degraded (12.571%); 20288/325464 objects misplaced (6.234%); 32 MiB/s, 518 objects/s recovering 2024-09-21T13:56:05.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:05 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 40914/325464 objects degraded (12.571%), 31 pgs degraded (PG_DEGRADED) 2024-09-21T13:56:05.956 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:06.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:06.012 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:06.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:06.113 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:56:06.128 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:56:06.129 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T13:56:06.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:06.139 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:06.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:56:06.240 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:56:06.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:06.348 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:06.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:56:06.449 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:06.503 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:06.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:06.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:06.607 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:06.649 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:06.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:06.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:06.679 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:06 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:56:06.679 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:06 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:56:06.680 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:06 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:56:06.680 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:06 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:56:06.680 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:06 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:56:06.680 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:06 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:56:06.750 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:06.792 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:06.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:06.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:06.894 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:56:06.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:06.933 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:06.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:56:06.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:06 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:56:06.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:06 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:56:06.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:06 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:56:07.033 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:56:07.034 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:07.090 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:07.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:07.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:07.192 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:07.255 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:07.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:07.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:07.357 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:07.408 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:07.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:07.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:07.511 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:56:07.538 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:07.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:07.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:56:07.639 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:56:07.668 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:07.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:07.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:56:07.770 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:56:07.795 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:07.796 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:56:07.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:07.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:56:07.871 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:56:07.871 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:56:07.897 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:56:07.913 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:56:07.913 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:56:07.940 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:07 smithi049 ceph-mon[30080]: pgmap v346: 148 pgs: 1 active+clean+remapped, 38 active+undersized, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 30 active+undersized+degraded, 75 active+clean; 6.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.42k op/s; 40459/322965 objects degraded (12.527%); 20288/322965 objects misplaced (6.282%); 28 MiB/s, 442 objects/s recovering 2024-09-21T13:56:07.940 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:07 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:56:07.940 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:07 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:56:07.940 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:07 smithi049 ceph-mon[30080]: osdmap e219: 8 total, 7 up, 7 in 2024-09-21T13:56:07.940 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:07 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/4051060753' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:56:07.940 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:07 smithi049 ceph-mon[40489]: pgmap v346: 148 pgs: 1 active+clean+remapped, 38 active+undersized, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 30 active+undersized+degraded, 75 active+clean; 6.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.42k op/s; 40459/322965 objects degraded (12.527%); 20288/322965 objects misplaced (6.282%); 28 MiB/s, 442 objects/s recovering 2024-09-21T13:56:07.940 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:07 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:56:07.940 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:07 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:56:07.941 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:07 smithi049 ceph-mon[40489]: osdmap e219: 8 total, 7 up, 7 in 2024-09-21T13:56:07.941 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:07 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/4051060753' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:56:07.943 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:07.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:07.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:56:07.966 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:56:07.966 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:56:07.967 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-21T13:56:08.000 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:56:08.000 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:56:08.033 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:56:08.033 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:56:08.045 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:56:08.071 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:08.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:08.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:56:08.077 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:56:08.077 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:56:08.113 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:56:08.113 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:56:08.173 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:56:08.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:07 smithi187 ceph-mon[33032]: pgmap v346: 148 pgs: 1 active+clean+remapped, 38 active+undersized, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 30 active+undersized+degraded, 75 active+clean; 6.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.42k op/s; 40459/322965 objects degraded (12.527%); 20288/322965 objects misplaced (6.282%); 28 MiB/s, 442 objects/s recovering 2024-09-21T13:56:08.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:07 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:56:08.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:07 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:56:08.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:07 smithi187 ceph-mon[33032]: osdmap e219: 8 total, 7 up, 7 in 2024-09-21T13:56:08.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:07 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/4051060753' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:56:08.215 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:08.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:08.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:56:08.318 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:08.365 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:08.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:08.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:08.467 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:08.497 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-09-21T13:56:08.507 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:08.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:08.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:08.609 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:56:08.639 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:08.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:08.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:56:08.674 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T13:56:08.674 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:56:08.741 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:08.770 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:08.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:08.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:08.872 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:08.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:08.909 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:08.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:09.010 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:56:09.040 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:09.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:09.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:56:09.142 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:09.178 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:09.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:09.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:09.189 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:08 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3360545810' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:56:09.189 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:08 smithi049 ceph-mon[40489]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:56:09.189 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:08 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/4051060753' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T13:56:09.189 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:08 smithi049 ceph-mon[40489]: osdmap e220: 8 total, 7 up, 7 in 2024-09-21T13:56:09.190 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:08 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3360545810' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:56:09.190 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:08 smithi049 ceph-mon[30080]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:56:09.190 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:08 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/4051060753' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T13:56:09.190 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:08 smithi049 ceph-mon[30080]: osdmap e220: 8 total, 7 up, 7 in 2024-09-21T13:56:09.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:08 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3360545810' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:56:09.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:08 smithi187 ceph-mon[33032]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:56:09.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:08 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/4051060753' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T13:56:09.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:08 smithi187 ceph-mon[33032]: osdmap e220: 8 total, 7 up, 7 in 2024-09-21T13:56:09.281 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:56:09.309 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:09.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:09.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:56:09.398 INFO:teuthology.orchestra.run.smithi049.stderr:Exporting 6.13 info 6.13( empty local-lis/les=211/212 n=0 ec=66/66 lis/c=211/211 les/c/f=212/212/0 sis=211) 2024-09-21T13:56:09.399 INFO:teuthology.orchestra.run.smithi049.stderr:Export successful 2024-09-21T13:56:09.410 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:09.443 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:09.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:09.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:09.544 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:09.565 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:09.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:09.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:09.666 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:56:09.695 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:56:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:56:09.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:09.696 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:09.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:56:09.797 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:09.836 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:09.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:09.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:09.937 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:56:09.969 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:09.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:09.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:56:10.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:09 smithi049 ceph-mon[40489]: pgmap v348: 148 pgs: 1 active+clean+remapped, 38 active+undersized, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 30 active+undersized+degraded, 75 active+clean; 6.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.42k op/s; 40459/322965 objects degraded (12.527%); 20288/322965 objects misplaced (6.282%); 28 MiB/s, 443 objects/s recovering 2024-09-21T13:56:10.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:09 smithi049 ceph-mon[30080]: pgmap v348: 148 pgs: 1 active+clean+remapped, 38 active+undersized, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 30 active+undersized+degraded, 75 active+clean; 6.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.42k op/s; 40459/322965 objects degraded (12.527%); 20288/322965 objects misplaced (6.282%); 28 MiB/s, 443 objects/s recovering 2024-09-21T13:56:10.071 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:10.093 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:10.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:10.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:10.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:09 smithi187 ceph-mon[33032]: pgmap v348: 148 pgs: 1 active+clean+remapped, 38 active+undersized, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 30 active+undersized+degraded, 75 active+clean; 6.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.42k op/s; 40459/322965 objects degraded (12.527%); 20288/322965 objects misplaced (6.282%); 28 MiB/s, 443 objects/s recovering 2024-09-21T13:56:10.196 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:56:10.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:10.229 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:10.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:56:10.330 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:10.364 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:10.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:10.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:10.465 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:56:10.503 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:10.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:10.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:56:10.605 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:56:10.635 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:10.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:10.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:56:10.737 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:10.770 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:10.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:10.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:10.872 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:56:10.893 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:10.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:10.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:56:10.995 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:11.004 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:56:10 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:56:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:56:11.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:10 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:56:11.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:10 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 40459/322965 objects degraded (12.527%), 31 pgs degraded (PG_DEGRADED) 2024-09-21T13:56:11.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:10 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:56:11.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:10 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 40459/322965 objects degraded (12.527%), 31 pgs degraded (PG_DEGRADED) 2024-09-21T13:56:11.033 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:11.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:11.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:11.135 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:56:11.172 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:11.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:11.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:56:11.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:10 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:56:11.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:10 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 40459/322965 objects degraded (12.527%), 31 pgs degraded (PG_DEGRADED) 2024-09-21T13:56:11.274 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:11.315 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:11.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:11.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:11.417 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:11.438 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:11.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:11.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:11.539 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:56:11.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:11.577 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:11.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:56:11.679 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:11.716 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:11.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:11.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:11.818 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:11.841 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:11.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:11.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:11.942 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:56:11.981 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:11.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:11.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:56:12.083 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:56:12.122 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:12.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:12.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:56:12.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:11 smithi187 ceph-mon[33032]: pgmap v350: 148 pgs: 1 active+clean+remapped, 38 active+undersized, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 30 active+undersized+degraded, 75 active+clean; 6.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.57k op/s; 38174/304818 objects degraded (12.524%); 17251/304818 objects misplaced (5.659%); 30 MiB/s, 476 objects/s recovering 2024-09-21T13:56:12.224 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:12.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:11 smithi049 ceph-mon[40489]: pgmap v350: 148 pgs: 1 active+clean+remapped, 38 active+undersized, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 30 active+undersized+degraded, 75 active+clean; 6.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.57k op/s; 38174/304818 objects degraded (12.524%); 17251/304818 objects misplaced (5.659%); 30 MiB/s, 476 objects/s recovering 2024-09-21T13:56:12.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:11 smithi049 ceph-mon[30080]: pgmap v350: 148 pgs: 1 active+clean+remapped, 38 active+undersized, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 30 active+undersized+degraded, 75 active+clean; 6.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.57k op/s; 38174/304818 objects degraded (12.524%); 17251/304818 objects misplaced (5.659%); 30 MiB/s, 476 objects/s recovering 2024-09-21T13:56:12.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:12.265 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:12.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:12.366 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:56:12.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:12.403 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:12.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:56:12.503 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:12.540 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:12.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:12.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:12.642 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:56:12.672 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:12.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:12.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:56:12.774 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:56:12.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:12.810 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:12.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:56:12.911 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:12.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:12.945 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:12.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:13.045 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:56:13.083 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:13.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:13.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:56:13.185 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:56:13.214 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:13.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:13.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:56:13.316 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:56:13.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:13.352 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:13.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:56:13.453 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:13.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:13.493 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:13.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:13.594 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:56:13.623 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:13.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:13.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:56:13.725 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:13.763 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:13.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:13.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:13.865 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:13.901 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:13.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:13.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:14.002 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:14.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:14.041 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:14.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:14.142 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:14.177 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:14.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:14.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:14.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:13 smithi187 ceph-mon[33032]: pgmap v351: 148 pgs: 1 active+clean+remapped, 38 active+undersized, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 30 active+undersized+degraded, 75 active+clean; 6.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 860 op/s; 38174/304818 objects degraded (12.524%); 17251/304818 objects misplaced (5.659%); 15 MiB/s, 239 objects/s recovering 2024-09-21T13:56:14.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:13 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:56:14.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:13 smithi049 ceph-mon[40489]: pgmap v351: 148 pgs: 1 active+clean+remapped, 38 active+undersized, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 30 active+undersized+degraded, 75 active+clean; 6.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 860 op/s; 38174/304818 objects degraded (12.524%); 17251/304818 objects misplaced (5.659%); 15 MiB/s, 239 objects/s recovering 2024-09-21T13:56:14.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:13 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:56:14.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:13 smithi049 ceph-mon[30080]: pgmap v351: 148 pgs: 1 active+clean+remapped, 38 active+undersized, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 30 active+undersized+degraded, 75 active+clean; 6.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 860 op/s; 38174/304818 objects degraded (12.524%); 17251/304818 objects misplaced (5.659%); 15 MiB/s, 239 objects/s recovering 2024-09-21T13:56:14.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:13 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:56:14.279 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:14.314 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:14.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:14.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:14.416 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:14.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:14.448 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:14.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:14.549 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:14.571 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:14.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:14.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:14.673 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:56:14.705 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:14.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:14.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:56:14.807 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:56:14.846 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:14.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:14.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:56:14.948 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:14.993 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:14.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:14.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:15.095 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:56:15.131 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:15.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:15.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:56:15.233 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:56:15.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:15.266 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:15.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:56:15.367 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:56:15.399 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:15.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:15.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:56:15.501 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:15.536 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:15.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:15.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:15.638 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:56:15.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:15.674 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:15.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:56:15.775 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:56:15.812 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:15.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:15.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:56:15.914 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:56:15.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:15.948 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:15.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:56:16.049 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:16.088 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:16.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:16.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:16.189 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:16.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:15 smithi187 ceph-mon[33032]: pgmap v352: 148 pgs: 1 active+clean+remapped, 38 active+undersized, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 30 active+undersized+degraded, 75 active+clean; 5.9 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.41k op/s; 36275/289110 objects degraded (12.547%); 14243/289110 objects misplaced (4.926%); 31 MiB/s, 498 objects/s recovering 2024-09-21T13:56:16.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:15 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 36275/289110 objects degraded (12.547%), 31 pgs degraded (PG_DEGRADED) 2024-09-21T13:56:16.227 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:16.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:16.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:16.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:15 smithi049 ceph-mon[40489]: pgmap v352: 148 pgs: 1 active+clean+remapped, 38 active+undersized, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 30 active+undersized+degraded, 75 active+clean; 5.9 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.41k op/s; 36275/289110 objects degraded (12.547%); 14243/289110 objects misplaced (4.926%); 31 MiB/s, 498 objects/s recovering 2024-09-21T13:56:16.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:15 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 36275/289110 objects degraded (12.547%), 31 pgs degraded (PG_DEGRADED) 2024-09-21T13:56:16.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:15 smithi049 ceph-mon[30080]: pgmap v352: 148 pgs: 1 active+clean+remapped, 38 active+undersized, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 30 active+undersized+degraded, 75 active+clean; 5.9 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.41k op/s; 36275/289110 objects degraded (12.547%); 14243/289110 objects misplaced (4.926%); 31 MiB/s, 498 objects/s recovering 2024-09-21T13:56:16.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:15 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 36275/289110 objects degraded (12.547%), 31 pgs degraded (PG_DEGRADED) 2024-09-21T13:56:16.329 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:16.363 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:16.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:16.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:16.465 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:16.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:16.502 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:16.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:16.603 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:16.635 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:16.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:16.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:16.737 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:16.768 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:16.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:16.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:16.857 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T13:56:16.869 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:16.928 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:16.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:16.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:17.015 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T13:56:17.016 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T13:56:17.030 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:17.053 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:17.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:17.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:17.155 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:17.179 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:17.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:17.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:17.280 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:17.303 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:17.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:17.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:17.405 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:56:17.426 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:17.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:17.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:56:17.527 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:56:17.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:17.551 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:17.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:56:17.652 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:17.675 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:17.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:17.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:17.776 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:56:17.797 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:17.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:17.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:56:17.899 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:56:17.937 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:17.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:17.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:56:17.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:17 smithi187 ceph-mon[33032]: pgmap v353: 148 pgs: 1 active+clean+remapped, 38 active+undersized, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 30 active+undersized+degraded, 75 active+clean; 5.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.45k op/s; 35897/286872 objects degraded (12.513%); 14243/286872 objects misplaced (4.965%); 30 MiB/s, 478 objects/s recovering 2024-09-21T13:56:17.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:17 smithi187 ceph-mon[33032]: osdmap e221: 8 total, 7 up, 7 in 2024-09-21T13:56:18.039 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:56:18.076 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:18.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:18.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:56:18.177 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:56:18.212 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:18.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:18.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:56:18.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:17 smithi049 ceph-mon[40489]: pgmap v353: 148 pgs: 1 active+clean+remapped, 38 active+undersized, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 30 active+undersized+degraded, 75 active+clean; 5.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.45k op/s; 35897/286872 objects degraded (12.513%); 14243/286872 objects misplaced (4.965%); 30 MiB/s, 478 objects/s recovering 2024-09-21T13:56:18.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:17 smithi049 ceph-mon[40489]: osdmap e221: 8 total, 7 up, 7 in 2024-09-21T13:56:18.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:17 smithi049 ceph-mon[30080]: pgmap v353: 148 pgs: 1 active+clean+remapped, 38 active+undersized, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 30 active+undersized+degraded, 75 active+clean; 5.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.45k op/s; 35897/286872 objects degraded (12.513%); 14243/286872 objects misplaced (4.965%); 30 MiB/s, 478 objects/s recovering 2024-09-21T13:56:18.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:17 smithi049 ceph-mon[30080]: osdmap e221: 8 total, 7 up, 7 in 2024-09-21T13:56:18.314 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:56:18.342 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:18.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:18.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:56:18.444 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:18.482 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:18.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:18.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:18.584 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:18.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:18.618 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:18.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:18.719 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:56:18.753 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:18.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:18.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:56:18.855 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:56:18.883 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:18.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:18.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:56:18.984 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell -n osd.3 --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --force --op remove --pgid 6.13 2024-09-21T13:56:18.985 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:56:19.005 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:19.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:19.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:56:19.017 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T13:56:19.107 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:56:19.130 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:19.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:19.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:56:19.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:18 smithi187 ceph-mon[33032]: 3.16 scrub starts 2024-09-21T13:56:19.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:18 smithi187 ceph-mon[33032]: 3.16 scrub ok 2024-09-21T13:56:19.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:18 smithi187 ceph-mon[33032]: 3.6 scrub starts 2024-09-21T13:56:19.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:18 smithi187 ceph-mon[33032]: 3.6 scrub ok 2024-09-21T13:56:19.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:18 smithi187 ceph-mon[33032]: osdmap e222: 8 total, 7 up, 7 in 2024-09-21T13:56:19.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:18 smithi187 ceph-mon[33032]: pgmap v356: 148 pgs: 1 active+clean+remapped, 38 active+undersized, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 30 active+undersized+degraded, 75 active+clean; 5.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 748 op/s; 35897/286872 objects degraded (12.513%); 14243/286872 objects misplaced (4.965%); 16 MiB/s, 258 objects/s recovering 2024-09-21T13:56:19.231 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:56:19.254 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:19.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:18 smithi049 ceph-mon[30080]: 3.16 scrub starts 2024-09-21T13:56:19.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:18 smithi049 ceph-mon[30080]: 3.16 scrub ok 2024-09-21T13:56:19.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:18 smithi049 ceph-mon[30080]: 3.6 scrub starts 2024-09-21T13:56:19.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:18 smithi049 ceph-mon[30080]: 3.6 scrub ok 2024-09-21T13:56:19.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:18 smithi049 ceph-mon[30080]: osdmap e222: 8 total, 7 up, 7 in 2024-09-21T13:56:19.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:18 smithi049 ceph-mon[30080]: pgmap v356: 148 pgs: 1 active+clean+remapped, 38 active+undersized, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 30 active+undersized+degraded, 75 active+clean; 5.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 748 op/s; 35897/286872 objects degraded (12.513%); 14243/286872 objects misplaced (4.965%); 16 MiB/s, 258 objects/s recovering 2024-09-21T13:56:19.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:18 smithi049 ceph-mon[40489]: 3.16 scrub starts 2024-09-21T13:56:19.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:18 smithi049 ceph-mon[40489]: 3.16 scrub ok 2024-09-21T13:56:19.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:18 smithi049 ceph-mon[40489]: 3.6 scrub starts 2024-09-21T13:56:19.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:18 smithi049 ceph-mon[40489]: 3.6 scrub ok 2024-09-21T13:56:19.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:18 smithi049 ceph-mon[40489]: osdmap e222: 8 total, 7 up, 7 in 2024-09-21T13:56:19.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:18 smithi049 ceph-mon[40489]: pgmap v356: 148 pgs: 1 active+clean+remapped, 38 active+undersized, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+remapped+backfilling, 30 active+undersized+degraded, 75 active+clean; 5.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 748 op/s; 35897/286872 objects degraded (12.513%); 14243/286872 objects misplaced (4.965%); 16 MiB/s, 258 objects/s recovering 2024-09-21T13:56:19.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:19.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:56:19.263 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/osd.3/config 2024-09-21T13:56:19.357 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:19.383 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:19.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:19.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:19.485 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:19.508 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:19.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:19.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:19.610 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:19.632 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:19.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:19.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:19.694 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:56:19 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:56:19.734 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:19.756 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:19.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:19.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:19.858 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:19.889 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:19.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:19.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:19.991 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:20.006 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:56:20.018 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:20.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:20.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:20.029 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:56:20.119 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:56:20.149 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:19 smithi049 ceph-mon[40489]: osdmap e223: 8 total, 7 up, 7 in 2024-09-21T13:56:20.150 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:19 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:56:20.150 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:19 smithi049 ceph-mon[30080]: osdmap e223: 8 total, 7 up, 7 in 2024-09-21T13:56:20.150 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:19 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:56:20.151 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:20.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:20.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:56:20.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:19 smithi187 ceph-mon[33032]: osdmap e223: 8 total, 7 up, 7 in 2024-09-21T13:56:20.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:19 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:56:20.253 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:20.288 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:20.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:20.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:20.390 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:20.426 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:20.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:20.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:20.528 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:20.556 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:20.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:20.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:20.633 INFO:teuthology.orchestra.run.smithi049.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-21T13:56:20.657 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:20.690 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:20.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:20.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:20.792 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:20.831 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:20.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:20.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:20.836 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:56:20 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:56:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:56:20.932 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:56:20.971 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:20.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:20.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:56:21.073 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:21.098 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:20 smithi049 ceph-mon[30080]: pgmap v358: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+clean+remapped, 38 active+undersized, 30 active+undersized+degraded, 77 active+clean; 5.5 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.07k op/s; 33783/269832 objects degraded (12.520%); 11328/269832 objects misplaced (4.198%); 23 MiB/s, 362 objects/s recovering 2024-09-21T13:56:21.098 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:20 smithi049 ceph-mon[30080]: osdmap e224: 8 total, 7 up, 7 in 2024-09-21T13:56:21.098 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:20 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 33783/269832 objects degraded (12.520%), 31 pgs degraded (PG_DEGRADED) 2024-09-21T13:56:21.099 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:20 smithi049 ceph-mon[40489]: pgmap v358: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+clean+remapped, 38 active+undersized, 30 active+undersized+degraded, 77 active+clean; 5.5 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.07k op/s; 33783/269832 objects degraded (12.520%); 11328/269832 objects misplaced (4.198%); 23 MiB/s, 362 objects/s recovering 2024-09-21T13:56:21.099 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:20 smithi049 ceph-mon[40489]: osdmap e224: 8 total, 7 up, 7 in 2024-09-21T13:56:21.099 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:20 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 33783/269832 objects degraded (12.520%), 31 pgs degraded (PG_DEGRADED) 2024-09-21T13:56:21.101 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:21.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:21.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:21.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:20 smithi187 ceph-mon[33032]: pgmap v358: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+clean+remapped, 38 active+undersized, 30 active+undersized+degraded, 77 active+clean; 5.5 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.07k op/s; 33783/269832 objects degraded (12.520%); 11328/269832 objects misplaced (4.198%); 23 MiB/s, 362 objects/s recovering 2024-09-21T13:56:21.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:20 smithi187 ceph-mon[33032]: osdmap e224: 8 total, 7 up, 7 in 2024-09-21T13:56:21.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:20 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 33783/269832 objects degraded (12.520%), 31 pgs degraded (PG_DEGRADED) 2024-09-21T13:56:21.204 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:56:21.252 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:21.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:21.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:56:21.294 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T13:56:21.354 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:56:21.382 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:21.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:21.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:56:21.483 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:21.525 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:21.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:21.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:21.627 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:21.670 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:21.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:21.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:21.771 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:56:21.804 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:21.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:21.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:56:21.865 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:56:21.865 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:56:21.865 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:56:21.865 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:56:21.865 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:56:21.865 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:56:21.866 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:56:21.866 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:56:21.866 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:56:21.866 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:56:21.866 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-21T13:56:21.866 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:56:21.866 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:56:21.866 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:56:21.866 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:56:21.907 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:21.942 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:21.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:21.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:22.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:21 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3939977888' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:56:22.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:21 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3939977888' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:56:22.044 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:22.075 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:22.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:22.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:22.177 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:56:22.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:21 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3939977888' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:56:22.208 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:22.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:22.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:56:22.310 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:22.355 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:22.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:22.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:22.457 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:56:22.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:22.495 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:22.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:56:22.596 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:56:22.628 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T13:56:22.628 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-09-21T13:56:22.628 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-09-21T13:56:22.628 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-09-21T13:56:22.628 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T13:56:22.631 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:22.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:22.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:56:22.733 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:22.771 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:22.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:22.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:22.873 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:56:22.900 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:22.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:22.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:56:23.002 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:23.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:22 smithi049 ceph-mon[30080]: 5.1b scrub starts 2024-09-21T13:56:23.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:22 smithi049 ceph-mon[30080]: 5.1b scrub ok 2024-09-21T13:56:23.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:22 smithi049 ceph-mon[30080]: 3.e scrub starts 2024-09-21T13:56:23.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:22 smithi049 ceph-mon[30080]: 3.e scrub ok 2024-09-21T13:56:23.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:22 smithi049 ceph-mon[30080]: pgmap v360: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+clean+remapped, 38 active+undersized, 30 active+undersized+degraded, 77 active+clean; 5.5 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s; 33783/269832 objects degraded (12.520%); 11328/269832 objects misplaced (4.198%); 27 MiB/s, 434 objects/s recovering 2024-09-21T13:56:23.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:22 smithi049 ceph-mon[40489]: 5.1b scrub starts 2024-09-21T13:56:23.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:22 smithi049 ceph-mon[40489]: 5.1b scrub ok 2024-09-21T13:56:23.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:22 smithi049 ceph-mon[40489]: 3.e scrub starts 2024-09-21T13:56:23.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:22 smithi049 ceph-mon[40489]: 3.e scrub ok 2024-09-21T13:56:23.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:22 smithi049 ceph-mon[40489]: pgmap v360: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+clean+remapped, 38 active+undersized, 30 active+undersized+degraded, 77 active+clean; 5.5 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s; 33783/269832 objects degraded (12.520%); 11328/269832 objects misplaced (4.198%); 27 MiB/s, 434 objects/s recovering 2024-09-21T13:56:23.030 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:23.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:23.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:23.132 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:23.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:23.164 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:23.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:23.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:22 smithi187 ceph-mon[33032]: 5.1b scrub starts 2024-09-21T13:56:23.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:22 smithi187 ceph-mon[33032]: 5.1b scrub ok 2024-09-21T13:56:23.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:22 smithi187 ceph-mon[33032]: 3.e scrub starts 2024-09-21T13:56:23.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:22 smithi187 ceph-mon[33032]: 3.e scrub ok 2024-09-21T13:56:23.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:22 smithi187 ceph-mon[33032]: pgmap v360: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+clean+remapped, 38 active+undersized, 30 active+undersized+degraded, 77 active+clean; 5.5 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s; 33783/269832 objects degraded (12.520%); 11328/269832 objects misplaced (4.198%); 27 MiB/s, 434 objects/s recovering 2024-09-21T13:56:23.265 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:23.307 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:23.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:23.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:23.409 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:56:23.450 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:23.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:23.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:56:23.552 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:56:23.581 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:23.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:23.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:56:23.683 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:23.719 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:23.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:23.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:23.820 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:23.863 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:23.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:23.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:23.964 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:56:23.991 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:23.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:23.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:56:24.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:23 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:56:24.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:23 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:56:24.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:23 smithi049 ceph-mon[40489]: 4.b scrub starts 2024-09-21T13:56:24.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:23 smithi049 ceph-mon[40489]: 4.b scrub ok 2024-09-21T13:56:24.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:23 smithi049 ceph-mon[40489]: osdmap e225: 8 total, 7 up, 7 in 2024-09-21T13:56:24.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:23 smithi049 ceph-mon[40489]: 3.4 scrub starts 2024-09-21T13:56:24.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:23 smithi049 ceph-mon[40489]: 3.4 scrub ok 2024-09-21T13:56:24.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:23 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:56:24.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:23 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:56:24.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:23 smithi049 ceph-mon[30080]: 4.b scrub starts 2024-09-21T13:56:24.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:23 smithi049 ceph-mon[30080]: 4.b scrub ok 2024-09-21T13:56:24.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:23 smithi049 ceph-mon[30080]: osdmap e225: 8 total, 7 up, 7 in 2024-09-21T13:56:24.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:23 smithi049 ceph-mon[30080]: 3.4 scrub starts 2024-09-21T13:56:24.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:23 smithi049 ceph-mon[30080]: 3.4 scrub ok 2024-09-21T13:56:24.093 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:56:24.124 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:24.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:24.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:56:24.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:23 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:56:24.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:23 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:56:24.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:23 smithi187 ceph-mon[33032]: 4.b scrub starts 2024-09-21T13:56:24.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:23 smithi187 ceph-mon[33032]: 4.b scrub ok 2024-09-21T13:56:24.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:23 smithi187 ceph-mon[33032]: osdmap e225: 8 total, 7 up, 7 in 2024-09-21T13:56:24.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:23 smithi187 ceph-mon[33032]: 3.4 scrub starts 2024-09-21T13:56:24.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:23 smithi187 ceph-mon[33032]: 3.4 scrub ok 2024-09-21T13:56:24.208 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T13:56:24.227 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:56:24.328 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:24.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:24.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:56:24.430 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:24.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:24.495 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:24.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:24.596 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:56:24.625 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:24.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:24.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:56:24.726 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:24.778 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T13:56:24.778 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T13:56:24.790 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:24.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:24.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:24.892 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:24.946 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:24.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:24.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:25.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:24 smithi049 ceph-mon[40489]: pgmap v362: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.84k op/s; 31845/253734 objects degraded (12.551%); 10515/253734 objects misplaced (4.144%); 23 MiB/s, 362 objects/s recovering 2024-09-21T13:56:25.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:24 smithi049 ceph-mon[30080]: pgmap v362: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.84k op/s; 31845/253734 objects degraded (12.551%); 10515/253734 objects misplaced (4.144%); 23 MiB/s, 362 objects/s recovering 2024-09-21T13:56:25.035 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:56:25.047 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:25.075 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:25.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:25.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:25.099 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:56:25.177 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:25.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:24 smithi187 ceph-mon[33032]: pgmap v362: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.84k op/s; 31845/253734 objects degraded (12.551%); 10515/253734 objects misplaced (4.144%); 23 MiB/s, 362 objects/s recovering 2024-09-21T13:56:25.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:25.224 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:25.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:25.325 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:25.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:25.370 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:25.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:25.470 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:56:25.509 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:25.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:25.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:56:25.611 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:25.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:25.644 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:25.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:25.745 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:56:25.775 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:25.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:25.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:56:25.877 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:25.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:25.936 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:25.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:25.939 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T13:56:25.939 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-09-21T13:56:25.939 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-09-21T13:56:25.939 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-09-21T13:56:25.939 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.939 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.14817.0:49348 7.f e98/94 7:fcf6cd05:::benchmark_data_smithi049_76798_object49347:head v 98'3012, mlcod=98'3010)", 2024-09-21T13:56:25.939 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:38.253256+0000", 2024-09-21T13:56:25.939 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 347.64165695299999, 2024-09-21T13:56:25.939 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.155802151, 2024-09-21T13:56:25.939 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:25.939 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:25.939 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:25.940 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:25.940 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.940 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:25.940 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.253256+0000", 2024-09-21T13:56:25.940 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:25.940 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.940 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.940 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:25.940 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.253256+0000", 2024-09-21T13:56:25.940 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:25.940 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.940 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.940 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:25.940 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.253257+0000", 2024-09-21T13:56:25.940 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0599999999999999e-07 2024-09-21T13:56:25.941 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.941 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.941 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:25.941 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.253283+0000", 2024-09-21T13:56:25.941 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.584e-05 2024-09-21T13:56:25.941 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.941 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.941 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:25.941 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.253293+0000", 2024-09-21T13:56:25.941 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0341e-05 2024-09-21T13:56:25.941 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.941 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.941 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:25.942 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.253300+0000", 2024-09-21T13:56:25.942 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9249999999999998e-06 2024-09-21T13:56:25.942 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.942 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.942 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:25.942 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.253596+0000", 2024-09-21T13:56:25.942 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00029586800000000002 2024-09-21T13:56:25.942 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.942 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.943 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:56:25.943 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.253614+0000", 2024-09-21T13:56:25.943 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7682000000000001e-05 2024-09-21T13:56:25.943 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.943 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.943 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:56:25.943 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.409015+0000", 2024-09-21T13:56:25.943 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.155401716 2024-09-21T13:56:25.943 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.943 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.943 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:25.943 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.409058+0000", 2024-09-21T13:56:25.943 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2873000000000001e-05 2024-09-21T13:56:25.943 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:25.944 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:25.944 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:25.944 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.944 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.944 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.14817.0:49359 7.e e98/83 7:7ae75bfc:::benchmark_data_smithi049_76798_object49358:head v 98'3124, mlcod=98'3122)", 2024-09-21T13:56:25.944 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:38.254961+0000", 2024-09-21T13:56:25.944 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 347.639951736, 2024-09-21T13:56:25.944 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.152662036, 2024-09-21T13:56:25.944 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:25.944 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:25.944 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:25.944 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:25.944 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.944 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:25.944 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.254961+0000", 2024-09-21T13:56:25.945 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:25.945 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.945 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.945 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:25.945 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.254961+0000", 2024-09-21T13:56:25.945 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:25.945 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.945 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.945 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:25.945 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.254962+0000", 2024-09-21T13:56:25.945 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1969999999999999e-06 2024-09-21T13:56:25.945 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.945 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.945 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:25.946 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.254987+0000", 2024-09-21T13:56:25.946 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4510000000000001e-05 2024-09-21T13:56:25.946 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.946 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.946 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:25.946 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.255000+0000", 2024-09-21T13:56:25.946 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2543e-05 2024-09-21T13:56:25.946 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.946 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.946 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:25.946 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.255007+0000", 2024-09-21T13:56:25.946 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1840000000000002e-06 2024-09-21T13:56:25.946 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.946 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.946 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:25.947 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257538+0000", 2024-09-21T13:56:25.947 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0025315950000000002 2024-09-21T13:56:25.947 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.947 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.947 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:56:25.947 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257559+0000", 2024-09-21T13:56:25.947 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0812999999999999e-05 2024-09-21T13:56:25.947 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.947 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.947 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:56:25.947 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.259702+0000", 2024-09-21T13:56:25.947 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0021427920000000001 2024-09-21T13:56:25.947 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.947 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.948 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:25.948 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.407623+0000", 2024-09-21T13:56:25.948 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.14792140200000001 2024-09-21T13:56:25.948 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:25.948 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:25.948 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:25.948 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.948 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.948 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.14817.0:49358 7.d e98/83 7:bafe9a80:::benchmark_data_smithi049_76798_object49357:head v 98'3118, mlcod=98'3116)", 2024-09-21T13:56:25.948 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:38.257159+0000", 2024-09-21T13:56:25.948 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 347.63775396199998, 2024-09-21T13:56:25.948 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.14991167599999999, 2024-09-21T13:56:25.948 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:25.948 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:25.948 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:25.949 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:25.949 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.949 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:25.949 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257159+0000", 2024-09-21T13:56:25.949 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:25.949 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.949 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.949 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:25.949 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257159+0000", 2024-09-21T13:56:25.949 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:25.949 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.949 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.949 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:25.949 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257160+0000", 2024-09-21T13:56:25.949 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.198e-06 2024-09-21T13:56:25.949 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.950 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.950 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:25.950 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257209+0000", 2024-09-21T13:56:25.950 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9098000000000003e-05 2024-09-21T13:56:25.950 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.950 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.950 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:25.950 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257226+0000", 2024-09-21T13:56:25.950 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6657e-05 2024-09-21T13:56:25.950 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.950 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.950 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:25.950 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257236+0000", 2024-09-21T13:56:25.950 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6900000000000004e-06 2024-09-21T13:56:25.951 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.951 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.951 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:25.951 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257650+0000", 2024-09-21T13:56:25.951 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000414693 2024-09-21T13:56:25.951 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.951 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.951 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:56:25.951 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257669+0000", 2024-09-21T13:56:25.951 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8938999999999999e-05 2024-09-21T13:56:25.951 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.951 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.951 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:56:25.951 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.407019+0000", 2024-09-21T13:56:25.952 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.149349182 2024-09-21T13:56:25.952 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.952 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.952 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:25.952 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.407071+0000", 2024-09-21T13:56:25.952 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2219000000000003e-05 2024-09-21T13:56:25.952 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:25.952 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:25.952 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:25.952 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.952 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.952 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.14817.0:49364 7.d e98/83 7:ba37e2e5:::benchmark_data_smithi049_76798_object49363:head v 98'3119, mlcod=98'3116)", 2024-09-21T13:56:25.952 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:38.257264+0000", 2024-09-21T13:56:25.952 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 347.63764939599997, 2024-09-21T13:56:25.953 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.15315973299999999, 2024-09-21T13:56:25.953 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:25.953 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:25.953 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:25.953 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:25.953 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.953 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:25.953 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257264+0000", 2024-09-21T13:56:25.953 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:25.953 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.953 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.953 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:25.953 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257264+0000", 2024-09-21T13:56:25.953 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:25.954 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.954 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.954 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:25.954 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257264+0000", 2024-09-21T13:56:25.954 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9000000000000002e-07 2024-09-21T13:56:25.954 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.954 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.954 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:25.954 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257298+0000", 2024-09-21T13:56:25.954 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3547999999999998e-05 2024-09-21T13:56:25.954 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.954 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.954 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:25.954 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257306+0000", 2024-09-21T13:56:25.954 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0809999999999994e-06 2024-09-21T13:56:25.955 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.955 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.955 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:25.955 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257311+0000", 2024-09-21T13:56:25.955 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7950000000000004e-06 2024-09-21T13:56:25.955 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.955 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.955 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:25.955 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.407103+0000", 2024-09-21T13:56:25.955 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.14979118299999999 2024-09-21T13:56:25.955 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.955 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.955 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:56:25.956 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.407123+0000", 2024-09-21T13:56:25.956 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9993e-05 2024-09-21T13:56:25.956 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.956 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.956 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:56:25.956 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.410375+0000", 2024-09-21T13:56:25.956 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0032521910000000002 2024-09-21T13:56:25.956 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.956 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.956 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:25.956 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.410423+0000", 2024-09-21T13:56:25.956 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8551999999999999e-05 2024-09-21T13:56:25.956 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:25.956 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:25.957 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:25.957 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.957 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.957 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.14817.0:49355 7.f e98/94 7:f1158586:::benchmark_data_smithi049_76798_object49354:head v 98'3013, mlcod=98'3010)", 2024-09-21T13:56:25.957 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:38.257372+0000", 2024-09-21T13:56:25.957 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 347.63754137699999, 2024-09-21T13:56:25.957 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.15605590799999999, 2024-09-21T13:56:25.957 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:25.957 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:25.957 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:25.957 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:25.957 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.957 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:25.957 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257372+0000", 2024-09-21T13:56:25.957 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:25.958 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.958 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.958 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:25.958 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257372+0000", 2024-09-21T13:56:25.958 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:25.958 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.958 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.958 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:25.958 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257372+0000", 2024-09-21T13:56:25.958 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7599999999999997e-07 2024-09-21T13:56:25.958 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.958 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.958 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:25.959 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257400+0000", 2024-09-21T13:56:25.959 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.7801e-05 2024-09-21T13:56:25.959 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.959 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.959 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:25.959 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257410+0000", 2024-09-21T13:56:25.959 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.978e-06 2024-09-21T13:56:25.960 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.960 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.960 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:25.960 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257415+0000", 2024-09-21T13:56:25.960 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3009999999999997e-06 2024-09-21T13:56:25.960 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.960 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.960 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:25.960 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.407104+0000", 2024-09-21T13:56:25.960 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.14968894099999999 2024-09-21T13:56:25.960 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.960 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.960 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:56:25.960 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.407125+0000", 2024-09-21T13:56:25.961 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1132999999999999e-05 2024-09-21T13:56:25.961 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.961 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.961 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:56:25.961 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.413375+0000", 2024-09-21T13:56:25.961 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0062498959999999996 2024-09-21T13:56:25.961 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.961 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.961 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:25.961 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.413428+0000", 2024-09-21T13:56:25.961 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2182000000000003e-05 2024-09-21T13:56:25.961 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:25.961 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:25.961 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:25.961 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.962 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.962 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.14817.0:49357 7.f e98/94 7:fffc650e:::benchmark_data_smithi049_76798_object49356:head v 98'3014, mlcod=98'3010)", 2024-09-21T13:56:25.962 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:38.259358+0000", 2024-09-21T13:56:25.962 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 347.63555475099997, 2024-09-21T13:56:25.962 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.160711944, 2024-09-21T13:56:25.962 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:25.962 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:25.962 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:25.962 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:25.962 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.962 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:25.962 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.259358+0000", 2024-09-21T13:56:25.962 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:25.962 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.963 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.963 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:25.963 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.259358+0000", 2024-09-21T13:56:25.963 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:25.963 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.963 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.963 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:25.963 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.259359+0000", 2024-09-21T13:56:25.963 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4099999999999998e-07 2024-09-21T13:56:25.963 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.963 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.963 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:25.963 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.259382+0000", 2024-09-21T13:56:25.963 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3357e-05 2024-09-21T13:56:25.963 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.964 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.964 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:25.964 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.259394+0000", 2024-09-21T13:56:25.964 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.183e-05 2024-09-21T13:56:25.964 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.964 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.964 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:25.964 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.259401+0000", 2024-09-21T13:56:25.964 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1930000000000004e-06 2024-09-21T13:56:25.964 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.964 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.964 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:25.964 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.409117+0000", 2024-09-21T13:56:25.964 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.149715182 2024-09-21T13:56:25.965 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.965 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.965 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:56:25.965 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.409151+0000", 2024-09-21T13:56:25.965 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4832999999999997e-05 2024-09-21T13:56:25.965 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.965 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.965 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:56:25.965 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.419333+0000", 2024-09-21T13:56:25.965 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.010181913000000001 2024-09-21T13:56:25.965 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.965 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.965 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:25.965 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.420070+0000", 2024-09-21T13:56:25.965 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000736895 2024-09-21T13:56:25.966 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:25.966 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:25.966 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:25.966 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.966 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.966 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.14817.0:49366 7.9 e98/94 7:956216ae:::benchmark_data_smithi049_76798_object49365:head v 98'3074, mlcod=98'3072)", 2024-09-21T13:56:25.966 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:38.259484+0000", 2024-09-21T13:56:25.966 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 347.63542880699998, 2024-09-21T13:56:25.966 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.15026700000000001, 2024-09-21T13:56:25.966 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:25.966 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:25.966 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:25.967 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:25.967 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.967 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:25.967 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.259484+0000", 2024-09-21T13:56:25.967 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:25.967 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.967 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.967 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:25.967 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.259484+0000", 2024-09-21T13:56:25.967 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:25.967 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.967 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.967 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:25.968 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.259485+0000", 2024-09-21T13:56:25.968 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0299999999999998e-07 2024-09-21T13:56:25.968 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.968 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.968 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:25.968 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.259543+0000", 2024-09-21T13:56:25.968 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7998000000000003e-05 2024-09-21T13:56:25.968 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.968 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.968 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:25.968 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.259554+0000", 2024-09-21T13:56:25.968 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0842000000000001e-05 2024-09-21T13:56:25.968 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.969 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.969 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:25.969 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.259561+0000", 2024-09-21T13:56:25.969 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6680000000000004e-06 2024-09-21T13:56:25.969 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.969 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.969 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:25.969 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.259639+0000", 2024-09-21T13:56:25.969 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7986999999999997e-05 2024-09-21T13:56:25.969 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.969 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.969 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:56:25.969 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.407005+0000", 2024-09-21T13:56:25.969 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.14736633399999999 2024-09-21T13:56:25.969 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.970 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.970 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:56:25.970 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.409267+0000", 2024-09-21T13:56:25.970 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0022621690000000001 2024-09-21T13:56:25.970 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.970 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.970 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:25.970 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.409751+0000", 2024-09-21T13:56:25.970 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00048419899999999999 2024-09-21T13:56:25.970 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:25.970 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:25.970 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:25.970 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.970 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.971 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.14817.0:49368 7.d e98/83 7:bfd0cd80:::benchmark_data_smithi049_76798_object49367:head v 98'3120, mlcod=98'3116)", 2024-09-21T13:56:25.971 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:38.260130+0000", 2024-09-21T13:56:25.971 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 347.63478337200002, 2024-09-21T13:56:25.971 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.151202637, 2024-09-21T13:56:25.971 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:25.971 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:25.971 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:25.971 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:25.971 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.971 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:25.971 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.260130+0000", 2024-09-21T13:56:25.971 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:25.971 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.971 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.972 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:25.972 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.260130+0000", 2024-09-21T13:56:25.972 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:25.972 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.972 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.972 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:25.972 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.260130+0000", 2024-09-21T13:56:25.972 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1100000000000005e-07 2024-09-21T13:56:25.972 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.972 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.972 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:25.972 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.260158+0000", 2024-09-21T13:56:25.972 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.7708e-05 2024-09-21T13:56:25.972 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.972 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.973 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:25.973 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.407103+0000", 2024-09-21T13:56:25.973 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.146944514 2024-09-21T13:56:25.973 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.973 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.973 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:25.973 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.407112+0000", 2024-09-21T13:56:25.973 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2569999999999996e-06 2024-09-21T13:56:25.973 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.973 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.973 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:25.973 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.408861+0000", 2024-09-21T13:56:25.973 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0017486590000000001 2024-09-21T13:56:25.973 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.974 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.974 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:56:25.974 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.408881+0000", 2024-09-21T13:56:25.974 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0038999999999999e-05 2024-09-21T13:56:25.974 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.974 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.974 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:56:25.974 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.411202+0000", 2024-09-21T13:56:25.974 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00232153 2024-09-21T13:56:25.974 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.974 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.974 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:25.974 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.411332+0000", 2024-09-21T13:56:25.974 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013011899999999999 2024-09-21T13:56:25.975 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:25.975 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:25.975 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:25.975 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.975 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.975 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.14817.0:49369 7.e e98/83 7:7b8e094d:::benchmark_data_smithi049_76798_object49368:head v 98'3125, mlcod=98'3122)", 2024-09-21T13:56:25.975 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:38.260639+0000", 2024-09-21T13:56:25.975 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 347.634274377, 2024-09-21T13:56:25.975 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.14984267100000001, 2024-09-21T13:56:25.975 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:25.975 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:25.975 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:25.975 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:25.975 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.976 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:25.976 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.260639+0000", 2024-09-21T13:56:25.976 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:25.976 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.976 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.976 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:25.976 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.260639+0000", 2024-09-21T13:56:25.976 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:25.976 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.976 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.976 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:25.976 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.260640+0000", 2024-09-21T13:56:25.976 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0400000000000005e-07 2024-09-21T13:56:25.976 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.976 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.977 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:25.977 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.260663+0000", 2024-09-21T13:56:25.977 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3859e-05 2024-09-21T13:56:25.977 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.977 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.977 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:25.977 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.406927+0000", 2024-09-21T13:56:25.977 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.146263533 2024-09-21T13:56:25.977 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.977 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.977 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:25.977 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.406950+0000", 2024-09-21T13:56:25.977 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3014e-05 2024-09-21T13:56:25.977 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.977 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.977 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:25.978 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.407706+0000", 2024-09-21T13:56:25.978 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00075584600000000001 2024-09-21T13:56:25.978 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.978 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.978 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:56:25.978 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.407723+0000", 2024-09-21T13:56:25.978 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7085999999999999e-05 2024-09-21T13:56:25.978 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.978 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.978 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:56:25.978 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.410449+0000", 2024-09-21T13:56:25.978 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0027258680000000002 2024-09-21T13:56:25.978 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.978 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.979 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:25.979 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.410481+0000", 2024-09-21T13:56:25.979 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2561000000000002e-05 2024-09-21T13:56:25.979 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:25.979 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:25.979 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:25.979 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.979 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.979 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.14817.0:72617 7.9 e105/94 7:9bb0aada:::benchmark_data_smithi049_76798_object72616:head v 105'4593, mlcod=104'4591)", 2024-09-21T13:56:25.979 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:54.476199+0000", 2024-09-21T13:56:25.979 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 331.41871365999998, 2024-09-21T13:56:25.979 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.096502150999999994, 2024-09-21T13:56:25.979 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:25.979 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:25.979 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:25.980 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:25.980 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.980 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:25.980 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:54.476199+0000", 2024-09-21T13:56:25.980 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:25.980 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.980 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.980 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:25.980 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:54.476199+0000", 2024-09-21T13:56:25.980 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:25.980 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.980 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.980 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:25.980 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:54.476201+0000", 2024-09-21T13:56:25.981 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4619999999999999e-06 2024-09-21T13:56:25.981 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.981 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.981 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:25.981 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:54.476234+0000", 2024-09-21T13:56:25.981 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2767e-05 2024-09-21T13:56:25.981 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.981 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.981 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:25.981 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:54.558715+0000", 2024-09-21T13:56:25.981 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.082481305000000005 2024-09-21T13:56:25.981 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.981 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.981 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:25.981 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:54.558723+0000", 2024-09-21T13:56:25.982 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0569999999999993e-06 2024-09-21T13:56:25.982 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.982 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.982 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:25.982 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:54.559118+0000", 2024-09-21T13:56:25.982 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00039494900000000002 2024-09-21T13:56:25.982 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.982 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.982 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:56:25.982 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:54.559136+0000", 2024-09-21T13:56:25.982 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8168e-05 2024-09-21T13:56:25.982 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.982 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.983 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:56:25.983 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:54.572578+0000", 2024-09-21T13:56:25.983 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.013442347 2024-09-21T13:56:25.983 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.983 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.983 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:25.983 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:54.572702+0000", 2024-09-21T13:56:25.983 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000123096 2024-09-21T13:56:25.983 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:25.983 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:25.983 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:25.983 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.983 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.983 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:13660 8.e 8:7e25e821:::benchmark_data_smithi049_109311_object13659:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-09-21T13:56:25.983 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:54:22.839402+0000", 2024-09-21T13:56:25.984 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 123.055511267, 2024-09-21T13:56:25.984 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.106316493, 2024-09-21T13:56:25.984 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:25.984 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:25.984 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:25.984 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:56:25.984 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:56:25.984 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:56:25.984 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 13660 2024-09-21T13:56:25.984 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.984 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:25.984 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.984 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:25.984 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.839402+0000", 2024-09-21T13:56:25.984 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:25.985 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.985 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.985 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:25.985 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.839402+0000", 2024-09-21T13:56:25.985 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:25.985 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.985 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.985 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:25.985 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.839403+0000", 2024-09-21T13:56:25.985 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.573e-06 2024-09-21T13:56:25.985 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.985 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.985 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:25.985 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.839426+0000", 2024-09-21T13:56:25.985 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3059000000000001e-05 2024-09-21T13:56:25.985 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.986 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.986 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:25.986 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.839438+0000", 2024-09-21T13:56:25.986 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1851000000000001e-05 2024-09-21T13:56:25.986 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.986 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.986 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:25.986 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.839446+0000", 2024-09-21T13:56:25.986 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8609999999999998e-06 2024-09-21T13:56:25.986 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.986 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.986 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:25.986 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.839994+0000", 2024-09-21T13:56:25.987 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00054770399999999999 2024-09-21T13:56:25.987 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.987 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.987 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:56:25.987 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.840234+0000", 2024-09-21T13:56:25.987 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00023994700000000001 2024-09-21T13:56:25.987 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.987 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.987 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,4", 2024-09-21T13:56:25.987 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.840537+0000", 2024-09-21T13:56:25.987 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000303004 2024-09-21T13:56:25.987 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.987 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.987 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:25.987 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.842148+0000", 2024-09-21T13:56:25.988 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0016112570000000001 2024-09-21T13:56:25.988 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.988 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.988 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:25.988 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.842528+0000", 2024-09-21T13:56:25.988 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00038031000000000002 2024-09-21T13:56:25.988 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.988 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.988 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:56:25.988 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.842792+0000", 2024-09-21T13:56:25.988 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00026319199999999998 2024-09-21T13:56:25.988 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.988 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.989 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:56:25.989 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.945668+0000", 2024-09-21T13:56:25.989 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.10287652 2024-09-21T13:56:25.989 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.989 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.989 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:25.989 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.945718+0000", 2024-09-21T13:56:25.989 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0214999999999999e-05 2024-09-21T13:56:25.989 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:25.989 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:25.989 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:25.989 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.989 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.989 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.15207.0:13662 8.2 e189/183 8:47bf13f7:::benchmark_data_smithi049_109311_object13661:head v 189'853, mlcod=189'851)", 2024-09-21T13:56:25.989 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:54:22.841979+0000", 2024-09-21T13:56:25.989 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 123.05293388699999, 2024-09-21T13:56:25.990 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.10434858599999999, 2024-09-21T13:56:25.990 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:25.990 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:25.990 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:25.990 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:25.990 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.990 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:25.990 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.841979+0000", 2024-09-21T13:56:25.990 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:25.990 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.990 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.990 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:25.990 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.841979+0000", 2024-09-21T13:56:25.991 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:25.991 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.991 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.991 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:25.991 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.841980+0000", 2024-09-21T13:56:25.991 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.92e-07 2024-09-21T13:56:25.991 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.991 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.991 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:25.991 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.842010+0000", 2024-09-21T13:56:25.991 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.0277000000000002e-05 2024-09-21T13:56:25.991 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.991 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.991 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:25.992 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.842021+0000", 2024-09-21T13:56:25.992 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1204e-05 2024-09-21T13:56:25.992 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.992 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.992 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:25.992 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.842029+0000", 2024-09-21T13:56:25.992 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.729e-06 2024-09-21T13:56:25.992 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.992 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.992 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:25.992 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.842391+0000", 2024-09-21T13:56:25.992 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00036184900000000002 2024-09-21T13:56:25.992 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.992 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.992 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:56:25.993 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.842408+0000", 2024-09-21T13:56:25.993 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6949e-05 2024-09-21T13:56:25.993 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.993 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.993 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:56:25.993 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.946278+0000", 2024-09-21T13:56:25.993 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.103869735 2024-09-21T13:56:25.993 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.993 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.993 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:25.993 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.946328+0000", 2024-09-21T13:56:25.993 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0151000000000001e-05 2024-09-21T13:56:25.993 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:25.993 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:25.994 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:25.994 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.994 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.994 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.15207.0:13663 8.6 e189/183 8:6d3c6ce8:::benchmark_data_smithi049_109311_object13662:head v 189'841, mlcod=189'838)", 2024-09-21T13:56:25.994 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:54:22.842094+0000", 2024-09-21T13:56:25.994 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 123.05281926000001, 2024-09-21T13:56:25.994 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.103723198, 2024-09-21T13:56:25.994 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:25.994 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:25.994 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:25.994 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:25.994 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.994 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:25.995 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.842094+0000", 2024-09-21T13:56:25.995 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:25.995 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.995 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.995 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:25.995 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.842094+0000", 2024-09-21T13:56:25.995 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:25.995 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.995 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.995 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:25.995 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.842094+0000", 2024-09-21T13:56:25.995 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2799999999999996e-07 2024-09-21T13:56:25.995 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.995 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.996 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:25.996 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.842130+0000", 2024-09-21T13:56:25.996 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5799000000000001e-05 2024-09-21T13:56:25.996 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.996 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.996 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:25.996 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.842138+0000", 2024-09-21T13:56:25.996 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7759999999999997e-06 2024-09-21T13:56:25.996 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.996 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.996 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:25.996 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.842143+0000", 2024-09-21T13:56:25.996 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2580000000000003e-06 2024-09-21T13:56:25.996 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.997 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.997 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:25.997 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.842184+0000", 2024-09-21T13:56:25.997 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0776999999999999e-05 2024-09-21T13:56:25.997 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.997 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.997 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:56:25.997 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.842202+0000", 2024-09-21T13:56:25.997 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7646999999999999e-05 2024-09-21T13:56:25.997 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.997 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.997 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:56:25.997 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.945779+0000", 2024-09-21T13:56:25.997 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.103577342 2024-09-21T13:56:25.998 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.998 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.998 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:25.998 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.945817+0000", 2024-09-21T13:56:25.998 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8071000000000002e-05 2024-09-21T13:56:25.998 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:25.998 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:25.998 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:25.998 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.998 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.998 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.15207.0:13672 8.6 e189/183 8:6c8a1d62:::benchmark_data_smithi049_109311_object13671:head v 189'842, mlcod=189'839)", 2024-09-21T13:56:25.998 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:54:22.843361+0000", 2024-09-21T13:56:25.998 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 123.051551993, 2024-09-21T13:56:25.998 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.10624172599999999, 2024-09-21T13:56:25.999 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:25.999 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:25.999 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:25.999 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:25.999 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.999 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:25.999 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.843361+0000", 2024-09-21T13:56:25.999 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:25.999 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.999 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:25.999 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:25.999 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.843361+0000", 2024-09-21T13:56:25.999 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:25.999 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:25.999 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.000 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:26.000 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.843362+0000", 2024-09-21T13:56:26.000 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6899999999999997e-07 2024-09-21T13:56:26.000 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.000 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.000 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:26.000 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.843413+0000", 2024-09-21T13:56:26.000 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0912000000000003e-05 2024-09-21T13:56:26.000 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.000 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.000 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:26.000 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.945785+0000", 2024-09-21T13:56:26.000 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.10237275999999999 2024-09-21T13:56:26.000 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.001 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.001 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:26.001 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.945792+0000", 2024-09-21T13:56:26.001 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7630000000000003e-06 2024-09-21T13:56:26.001 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.001 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.001 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:26.001 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.945840+0000", 2024-09-21T13:56:26.001 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7636000000000003e-05 2024-09-21T13:56:26.001 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.001 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.001 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:56:26.001 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.945858+0000", 2024-09-21T13:56:26.001 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7968000000000001e-05 2024-09-21T13:56:26.001 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.002 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.002 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:56:26.002 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.949564+0000", 2024-09-21T13:56:26.002 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0037061920000000001 2024-09-21T13:56:26.002 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.002 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.002 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:26.002 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.949603+0000", 2024-09-21T13:56:26.002 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8825999999999997e-05 2024-09-21T13:56:26.002 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:26.002 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:26.002 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:26.002 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.002 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.003 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:13675 8.c 8:3b0d0c74:::benchmark_data_smithi049_109311_object13674:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-09-21T13:56:26.003 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:54:22.845236+0000", 2024-09-21T13:56:26.003 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 123.04967665300001, 2024-09-21T13:56:26.003 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.105649625, 2024-09-21T13:56:26.003 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:26.003 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:26.003 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:26.003 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:56:26.003 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:56:26.003 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:56:26.003 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 13675 2024-09-21T13:56:26.003 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.003 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:26.003 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.004 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:26.004 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.845236+0000", 2024-09-21T13:56:26.004 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:26.004 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.004 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.004 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:26.004 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.845236+0000", 2024-09-21T13:56:26.004 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:26.004 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.004 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.004 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:26.004 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.845238+0000", 2024-09-21T13:56:26.004 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.206e-06 2024-09-21T13:56:26.004 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.005 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.005 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:26.005 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.845254+0000", 2024-09-21T13:56:26.005 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6436e-05 2024-09-21T13:56:26.005 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.005 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.005 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:26.005 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.945638+0000", 2024-09-21T13:56:26.005 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.10038441200000001 2024-09-21T13:56:26.005 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.005 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.005 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:26.005 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.945659+0000", 2024-09-21T13:56:26.005 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0615000000000002e-05 2024-09-21T13:56:26.005 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.006 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.006 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:26.006 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.945789+0000", 2024-09-21T13:56:26.006 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013036699999999999 2024-09-21T13:56:26.006 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.006 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.006 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:56:26.006 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.946167+0000", 2024-09-21T13:56:26.006 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00037719400000000002 2024-09-21T13:56:26.006 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.006 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.006 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,7", 2024-09-21T13:56:26.006 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.946339+0000", 2024-09-21T13:56:26.006 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00017191999999999999 2024-09-21T13:56:26.007 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.007 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.007 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:56:26.007 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.947759+0000", 2024-09-21T13:56:26.007 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00142015 2024-09-21T13:56:26.007 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.007 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.007 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:26.007 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.950219+0000", 2024-09-21T13:56:26.007 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0024608289999999999 2024-09-21T13:56:26.007 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.007 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.007 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:26.007 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.950793+0000", 2024-09-21T13:56:26.008 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00057326800000000004 2024-09-21T13:56:26.008 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.008 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.009 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:56:26.009 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.950846+0000", 2024-09-21T13:56:26.009 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2997000000000001e-05 2024-09-21T13:56:26.009 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.009 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.009 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:26.009 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.950886+0000", 2024-09-21T13:56:26.009 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0231000000000002e-05 2024-09-21T13:56:26.009 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:26.009 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:26.009 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:26.009 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.009 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.010 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:14213 8.c 8:38837bbf:::benchmark_data_smithi049_109311_object14212:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-09-21T13:56:26.010 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:54:23.116628+0000", 2024-09-21T13:56:26.010 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 122.778285236, 2024-09-21T13:56:26.010 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.120616289, 2024-09-21T13:56:26.010 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:26.010 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:26.010 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:26.010 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:56:26.010 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:56:26.010 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:56:26.010 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 14213 2024-09-21T13:56:26.010 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.010 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:26.010 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.011 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:26.011 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.116628+0000", 2024-09-21T13:56:26.011 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:26.011 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.011 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.011 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:26.011 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.116628+0000", 2024-09-21T13:56:26.011 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:26.011 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.011 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.011 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:26.011 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.116629+0000", 2024-09-21T13:56:26.011 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8299999999999995e-07 2024-09-21T13:56:26.011 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.012 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.012 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:26.012 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.116651+0000", 2024-09-21T13:56:26.012 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2005e-05 2024-09-21T13:56:26.012 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.012 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.012 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:26.012 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.116661+0000", 2024-09-21T13:56:26.012 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0533999999999999e-05 2024-09-21T13:56:26.012 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.012 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.012 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:26.012 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.116668+0000", 2024-09-21T13:56:26.013 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9349999999999997e-06 2024-09-21T13:56:26.013 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.013 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.013 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:26.013 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.116996+0000", 2024-09-21T13:56:26.013 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00032820400000000002 2024-09-21T13:56:26.013 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.013 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.013 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:56:26.013 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.117103+0000", 2024-09-21T13:56:26.013 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010673100000000001 2024-09-21T13:56:26.013 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.013 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.013 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,7", 2024-09-21T13:56:26.014 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.117264+0000", 2024-09-21T13:56:26.014 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016054200000000001 2024-09-21T13:56:26.014 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.014 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.014 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:56:26.014 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.118050+0000", 2024-09-21T13:56:26.014 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00078586699999999999 2024-09-21T13:56:26.014 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.014 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.014 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:26.014 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.118727+0000", 2024-09-21T13:56:26.014 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00067759600000000004 2024-09-21T13:56:26.014 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.015 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.015 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:26.015 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.119082+0000", 2024-09-21T13:56:26.015 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00035450800000000001 2024-09-21T13:56:26.015 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.015 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.015 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:56:26.015 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.119119+0000", 2024-09-21T13:56:26.015 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7568999999999999e-05 2024-09-21T13:56:26.015 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.015 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.015 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:26.015 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.237244+0000", 2024-09-21T13:56:26.015 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.11812481499999999 2024-09-21T13:56:26.015 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:26.016 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:26.016 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:26.016 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.016 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.016 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.15207.0:14214 8.2 e189/183 8:46d3a502:::benchmark_data_smithi049_109311_object14213:head v 189'885, mlcod=189'883)", 2024-09-21T13:56:26.016 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:54:23.118548+0000", 2024-09-21T13:56:26.016 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 122.776365181, 2024-09-21T13:56:26.016 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.118761609, 2024-09-21T13:56:26.016 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:26.016 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:26.016 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:26.016 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:26.016 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.016 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:26.017 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.118548+0000", 2024-09-21T13:56:26.017 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:26.017 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.017 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.017 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:26.017 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.118548+0000", 2024-09-21T13:56:26.017 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:26.017 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.017 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.017 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:26.017 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.118548+0000", 2024-09-21T13:56:26.017 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3e-07 2024-09-21T13:56:26.017 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.017 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.017 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:26.018 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.118593+0000", 2024-09-21T13:56:26.018 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4131999999999997e-05 2024-09-21T13:56:26.018 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.018 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.018 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:26.018 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.118604+0000", 2024-09-21T13:56:26.018 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.134e-05 2024-09-21T13:56:26.018 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.018 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.018 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:26.018 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.118611+0000", 2024-09-21T13:56:26.018 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4619999999999999e-06 2024-09-21T13:56:26.018 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.018 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.018 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:26.019 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.118669+0000", 2024-09-21T13:56:26.019 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7828999999999998e-05 2024-09-21T13:56:26.019 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.019 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.019 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:56:26.019 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.118690+0000", 2024-09-21T13:56:26.019 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0596999999999999e-05 2024-09-21T13:56:26.019 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.019 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.019 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:56:26.019 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.237258+0000", 2024-09-21T13:56:26.019 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.11856844599999999 2024-09-21T13:56:26.019 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.019 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.020 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:26.020 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.237309+0000", 2024-09-21T13:56:26.020 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1172999999999999e-05 2024-09-21T13:56:26.020 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:26.020 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:26.020 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:26.020 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.020 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.020 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.15207.0:14213 8.c e189/183 ondisk, result = 0)", 2024-09-21T13:56:26.020 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:54:23.118845+0000", 2024-09-21T13:56:26.020 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 122.776067743, 2024-09-21T13:56:26.020 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.11840756199999999, 2024-09-21T13:56:26.020 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:26.020 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:26.020 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T13:56:26.021 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:26.021 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.021 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:26.021 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.118845+0000", 2024-09-21T13:56:26.021 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:26.021 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.021 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.021 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:26.021 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.118845+0000", 2024-09-21T13:56:26.021 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:26.021 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.021 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.021 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:26.021 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.118846+0000", 2024-09-21T13:56:26.021 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3e-07 2024-09-21T13:56:26.022 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.022 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.022 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:26.022 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.118850+0000", 2024-09-21T13:56:26.022 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0219999999999998e-06 2024-09-21T13:56:26.022 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.022 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.022 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:26.022 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.118860+0000", 2024-09-21T13:56:26.022 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0162e-05 2024-09-21T13:56:26.022 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.022 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.022 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:26.022 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.118866+0000", 2024-09-21T13:56:26.023 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8980000000000001e-06 2024-09-21T13:56:26.023 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.023 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.023 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:26.023 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.119070+0000", 2024-09-21T13:56:26.023 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000204286 2024-09-21T13:56:26.023 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.023 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.023 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:56:26.023 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.119077+0000", 2024-09-21T13:56:26.023 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9909999999999996e-06 2024-09-21T13:56:26.023 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.023 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.023 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:26.023 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.237253+0000", 2024-09-21T13:56:26.024 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.118175473 2024-09-21T13:56:26.024 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:26.024 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:26.024 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:26.024 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.024 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.024 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.15207.0:14217 8.2 e189/183 8:42c0ac0e:::benchmark_data_smithi049_109311_object14216:head v 189'886, mlcod=189'883)", 2024-09-21T13:56:26.024 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:54:23.119299+0000", 2024-09-21T13:56:26.024 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 122.7756139, 2024-09-21T13:56:26.024 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.119049031, 2024-09-21T13:56:26.024 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:26.024 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:26.024 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:26.025 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:26.025 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.025 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:26.025 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.119299+0000", 2024-09-21T13:56:26.025 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:26.025 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.025 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.025 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:26.025 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.119299+0000", 2024-09-21T13:56:26.025 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:26.025 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.025 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.025 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:26.025 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.119304+0000", 2024-09-21T13:56:26.025 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9790000000000001e-06 2024-09-21T13:56:26.026 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.026 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.026 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:26.026 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.119344+0000", 2024-09-21T13:56:26.026 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9943e-05 2024-09-21T13:56:26.026 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.026 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.026 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:26.026 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.237161+0000", 2024-09-21T13:56:26.026 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.11781683 2024-09-21T13:56:26.026 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.026 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.026 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:26.026 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.237177+0000", 2024-09-21T13:56:26.027 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5709000000000002e-05 2024-09-21T13:56:26.027 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.027 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.027 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:26.027 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.237338+0000", 2024-09-21T13:56:26.027 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016102600000000001 2024-09-21T13:56:26.027 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.027 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.027 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:56:26.027 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.237359+0000", 2024-09-21T13:56:26.027 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1659999999999999e-05 2024-09-21T13:56:26.027 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.027 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.028 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:56:26.028 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.238317+0000", 2024-09-21T13:56:26.028 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00095745599999999998 2024-09-21T13:56:26.028 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.028 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.028 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:26.028 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.238348+0000", 2024-09-21T13:56:26.028 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1427999999999998e-05 2024-09-21T13:56:26.028 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:26.028 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:26.028 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:26.028 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.028 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.028 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:14223 8.e 8:753b28d9:::benchmark_data_smithi049_109311_object14222:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-09-21T13:56:26.029 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:54:23.121191+0000", 2024-09-21T13:56:26.029 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 122.773721999, 2024-09-21T13:56:26.029 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.119117285, 2024-09-21T13:56:26.029 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:26.029 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:26.029 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:26.029 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:56:26.029 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:56:26.029 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:56:26.029 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 14223 2024-09-21T13:56:26.029 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.029 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:26.029 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.029 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:26.029 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.121191+0000", 2024-09-21T13:56:26.030 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:26.030 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.030 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.030 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:26.030 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.121191+0000", 2024-09-21T13:56:26.030 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:26.030 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.030 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.030 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:26.030 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.121193+0000", 2024-09-21T13:56:26.030 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.486e-06 2024-09-21T13:56:26.030 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.030 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.030 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:26.031 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.121215+0000", 2024-09-21T13:56:26.031 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2371e-05 2024-09-21T13:56:26.031 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.031 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.031 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:26.031 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.237516+0000", 2024-09-21T13:56:26.031 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.116301107 2024-09-21T13:56:26.031 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.031 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.031 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:26.031 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.237527+0000", 2024-09-21T13:56:26.032 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1479e-05 2024-09-21T13:56:26.032 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.032 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.032 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:26.032 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.237593+0000", 2024-09-21T13:56:26.032 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5359999999999998e-05 2024-09-21T13:56:26.032 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.032 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.032 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:56:26.032 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.237741+0000", 2024-09-21T13:56:26.032 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014798799999999999 2024-09-21T13:56:26.032 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.032 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.032 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,4", 2024-09-21T13:56:26.032 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.237862+0000", 2024-09-21T13:56:26.033 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000121057 2024-09-21T13:56:26.033 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.033 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.033 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:56:26.033 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.238729+0000", 2024-09-21T13:56:26.033 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00086727799999999997 2024-09-21T13:56:26.033 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.033 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.033 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:26.033 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.239905+0000", 2024-09-21T13:56:26.033 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0011758280000000001 2024-09-21T13:56:26.033 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.033 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.034 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:26.034 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.240185+0000", 2024-09-21T13:56:26.034 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00028034499999999999 2024-09-21T13:56:26.034 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.034 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.034 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:56:26.034 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.240266+0000", 2024-09-21T13:56:26.034 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0969000000000004e-05 2024-09-21T13:56:26.034 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:26.034 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:26.034 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:26.034 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.240308+0000", 2024-09-21T13:56:26.034 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2017000000000001e-05 2024-09-21T13:56:26.034 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:26.035 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:26.035 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:26.035 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:26.035 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:26.035 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T13:56:26.035 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:25 smithi049 ceph-mon[40489]: 6.10 scrub starts 2024-09-21T13:56:26.035 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:25 smithi049 ceph-mon[40489]: 6.10 scrub ok 2024-09-21T13:56:26.035 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:25 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 31845/253734 objects degraded (12.551%), 31 pgs degraded (PG_DEGRADED) 2024-09-21T13:56:26.036 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:25 smithi049 ceph-mon[30080]: 6.10 scrub starts 2024-09-21T13:56:26.036 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:25 smithi049 ceph-mon[30080]: 6.10 scrub ok 2024-09-21T13:56:26.036 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:25 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 31845/253734 objects degraded (12.551%), 31 pgs degraded (PG_DEGRADED) 2024-09-21T13:56:26.037 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:56:26.066 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:26.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:26.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:56:26.168 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:26.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:25 smithi187 ceph-mon[33032]: 6.10 scrub starts 2024-09-21T13:56:26.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:25 smithi187 ceph-mon[33032]: 6.10 scrub ok 2024-09-21T13:56:26.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:25 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 31845/253734 objects degraded (12.551%), 31 pgs degraded (PG_DEGRADED) 2024-09-21T13:56:26.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:26.196 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:26.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:26.297 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:26.343 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:26.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:26.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:26.444 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:26.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:26.474 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:26.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:26.575 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:26.619 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:26.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:26.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:26.721 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:56:26.754 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:26.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:26.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:56:26.779 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T13:56:26.855 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:26.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:26.897 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:26.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:26.977 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-09-21T13:56:26.998 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:56:27.060 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:27.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:27.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:56:27.162 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:56:27.163 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T13:56:27.194 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:27.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:27.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:56:27.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:26 smithi049 ceph-mon[30080]: 6.14 scrub starts 2024-09-21T13:56:27.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:26 smithi049 ceph-mon[30080]: 6.14 scrub ok 2024-09-21T13:56:27.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:26 smithi049 ceph-mon[30080]: 3.13 scrub starts 2024-09-21T13:56:27.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:26 smithi049 ceph-mon[30080]: 3.13 scrub ok 2024-09-21T13:56:27.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:26 smithi049 ceph-mon[30080]: pgmap v363: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 731 B/s rd, 0 B/s wr, 974 op/s; 31480/251508 objects degraded (12.517%); 10515/251508 objects misplaced (4.181%) 2024-09-21T13:56:27.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:26 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3161897212' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:56:27.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:26 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:56:27.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:26 smithi049 ceph-mon[40489]: 6.14 scrub starts 2024-09-21T13:56:27.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:26 smithi049 ceph-mon[40489]: 6.14 scrub ok 2024-09-21T13:56:27.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:26 smithi049 ceph-mon[40489]: 3.13 scrub starts 2024-09-21T13:56:27.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:26 smithi049 ceph-mon[40489]: 3.13 scrub ok 2024-09-21T13:56:27.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:26 smithi049 ceph-mon[40489]: pgmap v363: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 731 B/s rd, 0 B/s wr, 974 op/s; 31480/251508 objects degraded (12.517%); 10515/251508 objects misplaced (4.181%) 2024-09-21T13:56:27.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:26 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3161897212' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:56:27.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:26 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:56:27.297 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:56:27.362 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:27.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:27.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:56:27.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:26 smithi187 ceph-mon[33032]: 6.14 scrub starts 2024-09-21T13:56:27.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:26 smithi187 ceph-mon[33032]: 6.14 scrub ok 2024-09-21T13:56:27.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:26 smithi187 ceph-mon[33032]: 3.13 scrub starts 2024-09-21T13:56:27.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:26 smithi187 ceph-mon[33032]: 3.13 scrub ok 2024-09-21T13:56:27.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:26 smithi187 ceph-mon[33032]: pgmap v363: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 731 B/s rd, 0 B/s wr, 974 op/s; 31480/251508 objects degraded (12.517%); 10515/251508 objects misplaced (4.181%) 2024-09-21T13:56:27.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:26 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3161897212' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:56:27.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:26 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:56:27.464 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:27.520 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:27.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:27.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:27.622 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:27.648 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:27.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:27.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:27.664 INFO:teuthology.orchestra.run.smithi049.stdout: marking collection for removal 2024-09-21T13:56:27.664 INFO:teuthology.orchestra.run.smithi049.stdout:setting '_remove' omap key 2024-09-21T13:56:27.664 INFO:teuthology.orchestra.run.smithi049.stdout:finish_remove_pgs 6.13_head removing 6.13 2024-09-21T13:56:27.666 INFO:teuthology.orchestra.run.smithi049.stdout:Remove successful 2024-09-21T13:56:27.750 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:27.789 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:27.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:27.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:27.891 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:27.919 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:27.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:27.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:27.978 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:56:28.021 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:28.067 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:28.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:28.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:28.158 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:28 smithi049 ceph-mon[40489]: 8.4 scrub starts 2024-09-21T13:56:28.158 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:28 smithi049 ceph-mon[40489]: 4.7 scrub starts 2024-09-21T13:56:28.158 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:28 smithi049 ceph-mon[40489]: 4.7 scrub ok 2024-09-21T13:56:28.158 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:28 smithi049 ceph-mon[40489]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T13:56:28.158 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:28 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T13:56:28.158 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:28 smithi049 ceph-mon[40489]: osdmap e226: 8 total, 7 up, 7 in 2024-09-21T13:56:28.158 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:28 smithi049 ceph-mon[30080]: 8.4 scrub starts 2024-09-21T13:56:28.159 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:28 smithi049 ceph-mon[30080]: 4.7 scrub starts 2024-09-21T13:56:28.159 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:28 smithi049 ceph-mon[30080]: 4.7 scrub ok 2024-09-21T13:56:28.159 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:28 smithi049 ceph-mon[30080]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T13:56:28.159 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:28 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T13:56:28.159 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:28 smithi049 ceph-mon[30080]: osdmap e226: 8 total, 7 up, 7 in 2024-09-21T13:56:28.169 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:28.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:28 smithi187 ceph-mon[33032]: 8.4 scrub starts 2024-09-21T13:56:28.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:28 smithi187 ceph-mon[33032]: 4.7 scrub starts 2024-09-21T13:56:28.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:28 smithi187 ceph-mon[33032]: 4.7 scrub ok 2024-09-21T13:56:28.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:28 smithi187 ceph-mon[33032]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T13:56:28.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:28 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T13:56:28.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:28 smithi187 ceph-mon[33032]: osdmap e226: 8 total, 7 up, 7 in 2024-09-21T13:56:28.200 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:28.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:28.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:28.302 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:28.335 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:28.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:28.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:28.437 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:56:28.468 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:28.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:28.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:56:28.570 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:56:28.608 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:28.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:28.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:56:28.710 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:56:28.740 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:28.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:28.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:56:28.810 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:56:28.842 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:28.866 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:56:28.866 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:56:28.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:28.867 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:28.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:28.905 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:56:28.905 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:56:28.937 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:56:28.938 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:56:28.940 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-21T13:56:28.968 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:28.973 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:56:28.974 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:56:29.001 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:29.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:29.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:29.007 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:56:29.007 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:56:29.040 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:56:29.041 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:56:29.073 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:56:29.074 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:56:29.103 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:29.142 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:29.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:29.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:29.244 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:56:29.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:29 smithi049 ceph-mon[40489]: 8.4 scrub ok 2024-09-21T13:56:29.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:29 smithi049 ceph-mon[40489]: pgmap v365: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 640 B/s rd, 0 B/s wr, 764 op/s; 31480/251508 objects degraded (12.517%); 10515/251508 objects misplaced (4.181%) 2024-09-21T13:56:29.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:29 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:56:29.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:29 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:56:29.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:29 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3724896664' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:56:29.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:29 smithi049 ceph-mon[30080]: 8.4 scrub ok 2024-09-21T13:56:29.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:29 smithi049 ceph-mon[30080]: pgmap v365: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 640 B/s rd, 0 B/s wr, 764 op/s; 31480/251508 objects degraded (12.517%); 10515/251508 objects misplaced (4.181%) 2024-09-21T13:56:29.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:29 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:56:29.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:29 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:56:29.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:29 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3724896664' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:56:29.275 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:29.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:29.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:56:29.366 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:29 smithi187 ceph-mon[33032]: 8.4 scrub ok 2024-09-21T13:56:29.366 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:29 smithi187 ceph-mon[33032]: pgmap v365: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 640 B/s rd, 0 B/s wr, 764 op/s; 31480/251508 objects degraded (12.517%); 10515/251508 objects misplaced (4.181%) 2024-09-21T13:56:29.366 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:29 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:56:29.366 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:29 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:56:29.366 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:29 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3724896664' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:56:29.377 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:29.416 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:29.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:29.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:29.518 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:56:29.548 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:29.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:29.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:56:29.649 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:56:29.685 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:29.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:29.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:56:29.694 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:56:29 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:56:29.787 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:56:29.818 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:29.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:29.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:56:29.920 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:29.956 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:29.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:29.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:29.962 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T13:56:30.045 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell -n osd.3 --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.6.13.3 2024-09-21T13:56:30.058 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:56:30.086 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:30.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:30.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:56:30.188 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:30.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:30.247 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:30.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:30.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:30 smithi049 ceph-mon[40489]: 6.e scrub starts 2024-09-21T13:56:30.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:30 smithi049 ceph-mon[40489]: 6.e scrub ok 2024-09-21T13:56:30.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:30 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:56:30.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:30 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 31480/251508 objects degraded (12.517%), 31 pgs degraded (PG_DEGRADED) 2024-09-21T13:56:30.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:30 smithi049 ceph-mon[30080]: 6.e scrub starts 2024-09-21T13:56:30.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:30 smithi049 ceph-mon[30080]: 6.e scrub ok 2024-09-21T13:56:30.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:30 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:56:30.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:30 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 31480/251508 objects degraded (12.517%), 31 pgs degraded (PG_DEGRADED) 2024-09-21T13:56:30.349 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:56:30.377 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:30.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:30.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:56:30.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:30 smithi187 ceph-mon[33032]: 6.e scrub starts 2024-09-21T13:56:30.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:30 smithi187 ceph-mon[33032]: 6.e scrub ok 2024-09-21T13:56:30.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:30 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:56:30.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:30 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 31480/251508 objects degraded (12.517%), 31 pgs degraded (PG_DEGRADED) 2024-09-21T13:56:30.479 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:56:30.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:30.506 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:30.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:56:30.607 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:30.649 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T13:56:30.649 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:56:30.663 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:30.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:30.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:30.720 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/osd.3/config 2024-09-21T13:56:30.754 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:56:30 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:56:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:56:30.767 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:56:30.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:30.798 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:30.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:56:30.899 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:56:30.934 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:30.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:30.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:56:31.036 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:31.076 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:31.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:31.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:31.178 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:56:31.197 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T13:56:31.197 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T13:56:31.207 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:31.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:31.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:56:31.308 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:56:31.341 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:31.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:31.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:56:31.427 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:56:31.443 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:56:31.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:31 smithi187 ceph-mon[33032]: 2.0 scrub starts 2024-09-21T13:56:31.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:31 smithi187 ceph-mon[33032]: 2.0 scrub ok 2024-09-21T13:56:31.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:31 smithi187 ceph-mon[33032]: pgmap v366: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 4.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.59k op/s; 29048/231726 objects degraded (12.535%); 9684/231726 objects misplaced (4.179%) 2024-09-21T13:56:31.486 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:31.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:31.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:56:31.488 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:31 smithi049 ceph-mon[30080]: 2.0 scrub starts 2024-09-21T13:56:31.488 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:31 smithi049 ceph-mon[30080]: 2.0 scrub ok 2024-09-21T13:56:31.489 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:31 smithi049 ceph-mon[30080]: pgmap v366: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 4.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.59k op/s; 29048/231726 objects degraded (12.535%); 9684/231726 objects misplaced (4.179%) 2024-09-21T13:56:31.489 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:31 smithi049 ceph-mon[40489]: 2.0 scrub starts 2024-09-21T13:56:31.489 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:31 smithi049 ceph-mon[40489]: 2.0 scrub ok 2024-09-21T13:56:31.489 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:31 smithi049 ceph-mon[40489]: pgmap v366: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 4.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.59k op/s; 29048/231726 objects degraded (12.535%); 9684/231726 objects misplaced (4.179%) 2024-09-21T13:56:31.588 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:56:31.616 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:31.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:31.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:56:31.718 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:56:31.745 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:31.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:31.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:56:31.847 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:56:31.877 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:31.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:31.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:56:31.979 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:56:32.015 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:32.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:32.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:56:32.117 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:32.154 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:32.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:32.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:32.256 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:56:32.297 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:32.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:32.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:56:32.397 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:32 smithi049 ceph-mon[40489]: 8.e scrub starts 2024-09-21T13:56:32.397 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:32 smithi049 ceph-mon[40489]: 5.9 scrub starts 2024-09-21T13:56:32.397 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:32 smithi049 ceph-mon[40489]: 5.9 scrub ok 2024-09-21T13:56:32.398 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:32 smithi049 ceph-mon[30080]: 8.e scrub starts 2024-09-21T13:56:32.398 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:32 smithi049 ceph-mon[30080]: 5.9 scrub starts 2024-09-21T13:56:32.398 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:32 smithi049 ceph-mon[30080]: 5.9 scrub ok 2024-09-21T13:56:32.399 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:32.441 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:32.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:32.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:32.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:32 smithi187 ceph-mon[33032]: 8.e scrub starts 2024-09-21T13:56:32.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:32 smithi187 ceph-mon[33032]: 5.9 scrub starts 2024-09-21T13:56:32.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:32 smithi187 ceph-mon[33032]: 5.9 scrub ok 2024-09-21T13:56:32.543 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:56:32.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:32.577 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:32.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:56:32.678 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:32.717 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:32.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:32.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:32.819 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:32.849 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:32.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:32.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:32.862 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T13:56:32.950 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:33.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:33.024 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:33.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:33.105 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:33 smithi049 ceph-mon[30080]: pgmap v367: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 4.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.42k op/s; 29048/231726 objects degraded (12.535%); 9684/231726 objects misplaced (4.179%) 2024-09-21T13:56:33.105 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:33 smithi049 ceph-mon[30080]: 8.e scrub ok 2024-09-21T13:56:33.105 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:33 smithi049 ceph-mon[40489]: pgmap v367: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 4.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.42k op/s; 29048/231726 objects degraded (12.535%); 9684/231726 objects misplaced (4.179%) 2024-09-21T13:56:33.105 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:33 smithi049 ceph-mon[40489]: 8.e scrub ok 2024-09-21T13:56:33.125 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:56:33.152 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:33.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:33.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:56:33.198 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T13:56:33.253 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:33.297 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:33.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:33.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:33.401 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:56:33.429 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:33.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:33.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:56:33.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:33 smithi187 ceph-mon[33032]: pgmap v367: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 4.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.42k op/s; 29048/231726 objects degraded (12.535%); 9684/231726 objects misplaced (4.179%) 2024-09-21T13:56:33.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:33 smithi187 ceph-mon[33032]: 8.e scrub ok 2024-09-21T13:56:33.531 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:33.559 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:33.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:33.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:33.634 INFO:teuthology.orchestra.run.smithi049.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-21T13:56:33.660 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:56:33.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:33.688 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:33.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:56:33.707 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:56:33.789 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:33.821 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:33.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:33.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:33.923 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:56:33.953 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:33.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:33.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:56:33.963 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:56:34.055 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:34.057 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-09-21T13:56:34.099 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:34.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:34.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:34.103 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:34 smithi049 ceph-mon[40489]: 4.1 scrub starts 2024-09-21T13:56:34.103 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:34 smithi049 ceph-mon[40489]: 4.1 scrub ok 2024-09-21T13:56:34.103 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:34 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:56:34.103 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:34 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:56:34.103 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:34 smithi049 ceph-mon[40489]: osdmap e227: 8 total, 7 up, 7 in 2024-09-21T13:56:34.103 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:34 smithi049 ceph-mon[40489]: 5.7 scrub starts 2024-09-21T13:56:34.103 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:34 smithi049 ceph-mon[40489]: 5.7 scrub ok 2024-09-21T13:56:34.104 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:34 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3852719865' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:56:34.104 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:34 smithi049 ceph-mon[30080]: 4.1 scrub starts 2024-09-21T13:56:34.104 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:34 smithi049 ceph-mon[30080]: 4.1 scrub ok 2024-09-21T13:56:34.104 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:34 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:56:34.104 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:34 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:56:34.105 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:34 smithi049 ceph-mon[30080]: osdmap e227: 8 total, 7 up, 7 in 2024-09-21T13:56:34.105 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:34 smithi049 ceph-mon[30080]: 5.7 scrub starts 2024-09-21T13:56:34.105 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:34 smithi049 ceph-mon[30080]: 5.7 scrub ok 2024-09-21T13:56:34.105 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:34 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3852719865' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:56:34.201 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:34.228 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:34.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:34.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:34.330 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:56:34.357 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:34.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:34.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:56:34.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:34 smithi187 ceph-mon[33032]: 4.1 scrub starts 2024-09-21T13:56:34.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:34 smithi187 ceph-mon[33032]: 4.1 scrub ok 2024-09-21T13:56:34.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:34 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:56:34.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:34 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:56:34.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:34 smithi187 ceph-mon[33032]: osdmap e227: 8 total, 7 up, 7 in 2024-09-21T13:56:34.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:34 smithi187 ceph-mon[33032]: 5.7 scrub starts 2024-09-21T13:56:34.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:34 smithi187 ceph-mon[33032]: 5.7 scrub ok 2024-09-21T13:56:34.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:34 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3852719865' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:56:34.460 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:56:34.487 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:34.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:34.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:56:34.588 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:34.625 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:34.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:34.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:34.727 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:34.756 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:34.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:34.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:34.858 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:56:34.889 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:34.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:34.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:56:34.991 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:35.035 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:35.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:35.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:35.137 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:35.168 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:35.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:35.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:35.270 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:35.311 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:35.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:35.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:35.316 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:35 smithi049 ceph-mon[40489]: pgmap v369: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 4.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.60k op/s; 26792/213012 objects degraded (12.578%); 8811/213012 objects misplaced (4.136%) 2024-09-21T13:56:35.316 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:35 smithi049 ceph-mon[40489]: 4.a scrub starts 2024-09-21T13:56:35.316 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:35 smithi049 ceph-mon[40489]: 4.a scrub ok 2024-09-21T13:56:35.316 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:35 smithi049 ceph-mon[40489]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:56:35.317 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:35 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3852719865' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T13:56:35.317 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:35 smithi049 ceph-mon[40489]: osdmap e228: 8 total, 7 up, 7 in 2024-09-21T13:56:35.317 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:35 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 26792/213012 objects degraded (12.578%), 31 pgs degraded (PG_DEGRADED) 2024-09-21T13:56:35.317 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:35 smithi049 ceph-mon[30080]: pgmap v369: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 4.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.60k op/s; 26792/213012 objects degraded (12.578%); 8811/213012 objects misplaced (4.136%) 2024-09-21T13:56:35.317 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:35 smithi049 ceph-mon[30080]: 4.a scrub starts 2024-09-21T13:56:35.317 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:35 smithi049 ceph-mon[30080]: 4.a scrub ok 2024-09-21T13:56:35.317 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:35 smithi049 ceph-mon[30080]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:56:35.317 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:35 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3852719865' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T13:56:35.317 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:35 smithi049 ceph-mon[30080]: osdmap e228: 8 total, 7 up, 7 in 2024-09-21T13:56:35.317 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:35 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 26792/213012 objects degraded (12.578%), 31 pgs degraded (PG_DEGRADED) 2024-09-21T13:56:35.390 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T13:56:35.390 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:56:35.404 INFO:teuthology.orchestra.run.smithi049.stdout:get_pg_num_history pg_num_history pg_num_history(e214 pg_nums {1={20=1},2={59=3},3={60=32},4={62=32},5={64=32},6={66=32},7={83=16},8={183=16}} deleted_pools 182,7) 2024-09-21T13:56:35.405 INFO:teuthology.orchestra.run.smithi049.stdout:Importing pgid 6.13 2024-09-21T13:56:35.405 INFO:teuthology.orchestra.run.smithi049.stdout:write_pg epoch 212 info 6.13( empty local-lis/les=211/212 n=0 ec=66/66 lis/c=211/211 les/c/f=212/212/0 sis=211) 2024-09-21T13:56:35.411 INFO:teuthology.orchestra.run.smithi049.stdout:Import successful 2024-09-21T13:56:35.413 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:56:35.442 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:35.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:35.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:56:35.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:35 smithi187 ceph-mon[33032]: pgmap v369: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 4.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.60k op/s; 26792/213012 objects degraded (12.578%); 8811/213012 objects misplaced (4.136%) 2024-09-21T13:56:35.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:35 smithi187 ceph-mon[33032]: 4.a scrub starts 2024-09-21T13:56:35.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:35 smithi187 ceph-mon[33032]: 4.a scrub ok 2024-09-21T13:56:35.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:35 smithi187 ceph-mon[33032]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:56:35.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:35 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3852719865' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T13:56:35.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:35 smithi187 ceph-mon[33032]: osdmap e228: 8 total, 7 up, 7 in 2024-09-21T13:56:35.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:35 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 26792/213012 objects degraded (12.578%), 31 pgs degraded (PG_DEGRADED) 2024-09-21T13:56:35.544 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:56:35.573 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:35.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:35.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:56:35.635 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:56:35.635 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:56:35.671 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:56:35.672 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:56:35.675 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:56:35.701 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:35.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:35.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:56:35.707 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:56:35.708 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:56:35.709 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-21T13:56:35.747 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:56:35.747 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:56:35.777 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:56:35.777 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:56:35.803 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:35.813 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:56:35.813 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:56:35.841 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:35.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:35.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:35.848 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:56:35.848 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:56:35.944 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:35.977 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:35.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:35.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:36.079 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:36.110 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:36.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:36.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:36.212 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:56:36.253 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:36.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:36.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:56:36.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:36 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3659320531' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:56:36.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:36 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3659320531' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:56:36.354 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:36.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:36.395 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:36.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:36.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:36 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3659320531' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:56:36.497 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:36.534 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:36.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:36.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:36.636 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:56:36.675 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:36.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:36.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:56:36.776 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:56:36.811 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:36.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:36.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:56:36.913 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:36.950 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:36.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:36.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:37.052 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:56:37.088 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:37.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:37.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:56:37.190 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:56:37.233 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T13:56:37.249 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:37.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:37.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:56:37.291 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:37 smithi049 ceph-mon[40489]: pgmap v371: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 4.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.72k op/s; 26331/210303 objects degraded (12.521%); 8811/210303 objects misplaced (4.190%) 2024-09-21T13:56:37.291 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:37 smithi049 ceph-mon[30080]: pgmap v371: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 4.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.72k op/s; 26331/210303 objects degraded (12.521%); 8811/210303 objects misplaced (4.190%) 2024-09-21T13:56:37.352 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:37.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:37.409 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:37.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:37.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:37 smithi187 ceph-mon[33032]: pgmap v371: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 4.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.72k op/s; 26331/210303 objects degraded (12.521%); 8811/210303 objects misplaced (4.190%) 2024-09-21T13:56:37.510 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:56:37.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:37.539 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:37.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:56:37.640 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:37.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:37.668 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:37.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:37.769 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:37.813 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:37.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:37.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:37.915 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:56:37.941 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -f /var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/exp.6.13.3 2024-09-21T13:56:37.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:37.945 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:37.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:56:38.046 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:38.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:38.077 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:38.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:38.178 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:56:38.214 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:38.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:38.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:56:38.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:38 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:56:38.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:38 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:56:38.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:38 smithi049 ceph-mon[40489]: osdmap e229: 8 total, 7 up, 7 in 2024-09-21T13:56:38.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:38 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:56:38.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:38 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:56:38.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:38 smithi049 ceph-mon[30080]: osdmap e229: 8 total, 7 up, 7 in 2024-09-21T13:56:38.315 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:56:38.345 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:38.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:38.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:56:38.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:38 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:56:38.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:38 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:56:38.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:38 smithi187 ceph-mon[33032]: osdmap e229: 8 total, 7 up, 7 in 2024-09-21T13:56:38.447 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:56:38.475 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:38.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:38.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:56:38.574 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T13:56:38.575 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T13:56:38.576 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:56:38.586 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:56:38.601 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:38.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:38.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:56:38.610 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T13:56:38.703 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:56:38.730 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:38.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:38.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:56:38.832 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:56:38.857 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:38.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:38.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:56:38.958 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:56:39.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:39.014 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:39.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:56:39.114 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:56:39.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:39.162 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:39.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:56:39.263 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:56:39.300 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:39.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:39.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:56:39.341 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:39 smithi049 ceph-mon[40489]: pgmap v373: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 4.3 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.19k op/s; 26331/210303 objects degraded (12.521%); 8811/210303 objects misplaced (4.190%) 2024-09-21T13:56:39.341 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:39 smithi049 ceph-mon[30080]: pgmap v373: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 4.3 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.19k op/s; 26331/210303 objects degraded (12.521%); 8811/210303 objects misplaced (4.190%) 2024-09-21T13:56:39.376 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:39 smithi187 ceph-mon[33032]: pgmap v373: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 4.3 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.19k op/s; 26331/210303 objects degraded (12.521%); 8811/210303 objects misplaced (4.190%) 2024-09-21T13:56:39.401 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:56:39.435 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:39.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:39.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:56:39.535 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T13:56:39.535 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-09-21T13:56:39.535 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-09-21T13:56:39.535 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-09-21T13:56:39.535 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.535 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:35232 7.9 7:9fe7d429:::benchmark_data_smithi049_76798_object35231:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T13:56:39.535 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:28.686371+0000", 2024-09-21T13:56:39.536 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 370.621962994, 2024-09-21T13:56:39.536 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.004075987, 2024-09-21T13:56:39.536 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:39.536 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:39.536 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T13:56:39.536 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:56:39.536 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:56:39.536 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:56:39.536 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 35232 2024-09-21T13:56:39.536 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.536 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:39.536 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.536 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:39.537 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.686371+0000", 2024-09-21T13:56:39.537 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:39.537 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.537 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.537 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:39.537 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.686371+0000", 2024-09-21T13:56:39.537 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:39.537 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.537 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.537 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:39.537 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.686372+0000", 2024-09-21T13:56:39.537 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5230000000000001e-06 2024-09-21T13:56:39.537 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.537 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.537 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:39.537 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.686409+0000", 2024-09-21T13:56:39.538 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6964e-05 2024-09-21T13:56:39.538 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.538 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.538 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:39.538 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.686422+0000", 2024-09-21T13:56:39.538 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2405000000000001e-05 2024-09-21T13:56:39.538 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.538 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.538 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:39.538 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.686432+0000", 2024-09-21T13:56:39.538 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8130000000000007e-06 2024-09-21T13:56:39.538 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.539 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.539 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:39.539 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.686758+0000", 2024-09-21T13:56:39.539 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00032668200000000001 2024-09-21T13:56:39.539 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.539 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.539 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:56:39.539 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.686768+0000", 2024-09-21T13:56:39.539 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9920000000000002e-06 2024-09-21T13:56:39.539 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.539 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.539 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:39.539 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.689859+0000", 2024-09-21T13:56:39.539 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.003090952 2024-09-21T13:56:39.540 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.540 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.540 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:39.540 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.690447+0000", 2024-09-21T13:56:39.540 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00058765599999999999 2024-09-21T13:56:39.540 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:39.540 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:39.540 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:39.540 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.540 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.540 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:35237 7.9 7:9d5e8f11:::benchmark_data_smithi049_76798_object35236:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T13:56:39.540 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:28.686454+0000", 2024-09-21T13:56:39.540 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 370.62187946300003, 2024-09-21T13:56:39.540 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0108801160000001, 2024-09-21T13:56:39.541 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:39.541 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:39.541 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:39.541 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:56:39.541 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:56:39.541 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:56:39.541 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 35237 2024-09-21T13:56:39.541 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.541 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:39.541 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.541 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:39.541 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.686454+0000", 2024-09-21T13:56:39.541 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:39.541 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.541 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.542 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:39.542 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.686454+0000", 2024-09-21T13:56:39.542 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:39.542 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.542 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.542 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:39.542 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.686455+0000", 2024-09-21T13:56:39.542 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7100000000000002e-07 2024-09-21T13:56:39.542 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.542 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.542 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:39.542 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.686477+0000", 2024-09-21T13:56:39.542 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2257000000000001e-05 2024-09-21T13:56:39.542 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.543 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.543 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:39.543 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.686486+0000", 2024-09-21T13:56:39.543 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4100000000000008e-06 2024-09-21T13:56:39.543 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.543 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.543 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:39.543 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.686499+0000", 2024-09-21T13:56:39.543 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3200999999999999e-05 2024-09-21T13:56:39.543 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.543 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.543 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:39.543 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.686795+0000", 2024-09-21T13:56:39.543 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00029598200000000003 2024-09-21T13:56:39.544 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.544 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.544 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:56:39.544 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.686802+0000", 2024-09-21T13:56:39.544 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3560000000000004e-06 2024-09-21T13:56:39.544 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.544 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.544 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:39.544 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.690840+0000", 2024-09-21T13:56:39.544 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0040381190000001 2024-09-21T13:56:39.544 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.544 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.544 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:56:39.545 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.691025+0000", 2024-09-21T13:56:39.545 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00018466700000000001 2024-09-21T13:56:39.545 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.545 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.545 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,4", 2024-09-21T13:56:39.545 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.691168+0000", 2024-09-21T13:56:39.545 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014334499999999999 2024-09-21T13:56:39.545 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.545 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.545 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:56:39.545 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.696996+0000", 2024-09-21T13:56:39.545 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0058274470000000004 2024-09-21T13:56:39.545 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.545 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.545 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:39.546 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.697045+0000", 2024-09-21T13:56:39.546 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9653e-05 2024-09-21T13:56:39.546 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.546 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.546 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:39.546 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.697095+0000", 2024-09-21T13:56:39.546 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9666000000000002e-05 2024-09-21T13:56:39.546 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.546 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.546 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:56:39.546 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.697224+0000", 2024-09-21T13:56:39.546 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012912700000000001 2024-09-21T13:56:39.546 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.546 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.547 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:39.547 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.697335+0000", 2024-09-21T13:56:39.547 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000110315 2024-09-21T13:56:39.547 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:39.547 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:39.547 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:39.547 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.547 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.547 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:35246 7.9 7:9e7a779d:::benchmark_data_smithi049_76798_object35245:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T13:56:39.547 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:28.690764+0000", 2024-09-21T13:56:39.547 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 370.61757004700002, 2024-09-21T13:56:39.547 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.007016763, 2024-09-21T13:56:39.547 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:39.547 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:39.548 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:39.548 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:56:39.548 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:56:39.548 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:56:39.548 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 35246 2024-09-21T13:56:39.548 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.548 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:39.548 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.548 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:39.548 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.690764+0000", 2024-09-21T13:56:39.548 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:39.548 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.548 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.548 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:39.548 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.690764+0000", 2024-09-21T13:56:39.549 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:39.549 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.549 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.549 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:39.549 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.690766+0000", 2024-09-21T13:56:39.549 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.77e-06 2024-09-21T13:56:39.549 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.549 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.549 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:39.549 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.690790+0000", 2024-09-21T13:56:39.549 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4717999999999999e-05 2024-09-21T13:56:39.549 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.549 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.549 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:39.550 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.690804+0000", 2024-09-21T13:56:39.550 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3743e-05 2024-09-21T13:56:39.550 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.550 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.550 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:39.550 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.690813+0000", 2024-09-21T13:56:39.550 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9879999999999993e-06 2024-09-21T13:56:39.550 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.550 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.550 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:39.550 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.691059+0000", 2024-09-21T13:56:39.550 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000245424 2024-09-21T13:56:39.550 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.550 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.551 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:56:39.551 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.691070+0000", 2024-09-21T13:56:39.551 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1375999999999999e-05 2024-09-21T13:56:39.551 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.551 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.551 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:39.551 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.692262+0000", 2024-09-21T13:56:39.551 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.001191693 2024-09-21T13:56:39.551 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.551 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.551 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:56:39.551 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.692378+0000", 2024-09-21T13:56:39.551 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000115998 2024-09-21T13:56:39.551 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.552 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.552 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,4", 2024-09-21T13:56:39.552 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.692518+0000", 2024-09-21T13:56:39.552 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014013900000000001 2024-09-21T13:56:39.552 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.552 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.552 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:56:39.552 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.697001+0000", 2024-09-21T13:56:39.552 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0044833909999999998 2024-09-21T13:56:39.552 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.552 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.552 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:39.552 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.697514+0000", 2024-09-21T13:56:39.552 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00051338899999999995 2024-09-21T13:56:39.552 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.552 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.553 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:39.553 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.697690+0000", 2024-09-21T13:56:39.553 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000175377 2024-09-21T13:56:39.553 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.553 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.553 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:56:39.553 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.697744+0000", 2024-09-21T13:56:39.553 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4018999999999999e-05 2024-09-21T13:56:39.553 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.553 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.553 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:39.553 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.697781+0000", 2024-09-21T13:56:39.553 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6738e-05 2024-09-21T13:56:39.553 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:39.554 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:39.554 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:39.554 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.554 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.554 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:35255 7.9 7:9a89d0ac:::benchmark_data_smithi049_76798_object35254:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T13:56:39.554 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:28.698899+0000", 2024-09-21T13:56:39.554 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 370.60943509499998, 2024-09-21T13:56:39.554 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.004516322, 2024-09-21T13:56:39.554 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:39.554 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:39.554 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:39.554 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:56:39.554 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:56:39.554 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:56:39.555 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 35255 2024-09-21T13:56:39.555 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.555 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:39.555 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.555 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:39.555 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.698899+0000", 2024-09-21T13:56:39.555 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:39.555 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.555 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.555 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:39.555 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.698899+0000", 2024-09-21T13:56:39.555 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:39.555 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.555 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.555 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:39.556 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.698899+0000", 2024-09-21T13:56:39.556 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4799999999999998e-07 2024-09-21T13:56:39.556 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.556 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.556 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:39.556 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.698921+0000", 2024-09-21T13:56:39.556 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1942000000000001e-05 2024-09-21T13:56:39.556 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.556 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.556 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:39.556 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.698935+0000", 2024-09-21T13:56:39.556 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3798000000000001e-05 2024-09-21T13:56:39.556 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.556 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.557 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:39.557 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.698942+0000", 2024-09-21T13:56:39.557 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4570000000000002e-06 2024-09-21T13:56:39.557 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.557 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.557 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:39.557 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.699176+0000", 2024-09-21T13:56:39.557 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000234446 2024-09-21T13:56:39.557 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.557 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.557 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:56:39.557 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:28.699185+0000", 2024-09-21T13:56:39.557 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9930000000000001e-06 2024-09-21T13:56:39.557 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.557 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.558 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:39.558 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.694230+0000", 2024-09-21T13:56:39.558 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.99504515500000001 2024-09-21T13:56:39.558 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.558 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.558 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:56:39.558 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.694419+0000", 2024-09-21T13:56:39.558 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00018863800000000001 2024-09-21T13:56:39.558 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.558 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.558 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,4", 2024-09-21T13:56:39.558 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.694572+0000", 2024-09-21T13:56:39.558 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000153385 2024-09-21T13:56:39.558 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.559 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.559 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:56:39.559 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.697061+0000", 2024-09-21T13:56:39.559 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0024887680000000001 2024-09-21T13:56:39.559 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.559 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.559 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:39.559 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.698026+0000", 2024-09-21T13:56:39.559 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00096500399999999999 2024-09-21T13:56:39.559 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.559 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.559 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:39.559 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.703311+0000", 2024-09-21T13:56:39.559 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.005285171 2024-09-21T13:56:39.560 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.560 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.560 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:56:39.560 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.703371+0000", 2024-09-21T13:56:39.560 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9296999999999998e-05 2024-09-21T13:56:39.560 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.560 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.560 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:39.560 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:29.703415+0000", 2024-09-21T13:56:39.560 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4620000000000003e-05 2024-09-21T13:56:39.560 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:39.560 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:39.560 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:39.560 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.560 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.561 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:36851 7.b 7:d46d3307:::benchmark_data_smithi049_76798_object36850:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-21T13:56:39.561 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:30.627278+0000", 2024-09-21T13:56:39.561 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 368.68105588600002, 2024-09-21T13:56:39.561 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.14148804500000001, 2024-09-21T13:56:39.561 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:39.561 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:39.561 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:39.561 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:56:39.561 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:56:39.561 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:56:39.561 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 36851 2024-09-21T13:56:39.561 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.561 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:39.561 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.561 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:39.562 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.627278+0000", 2024-09-21T13:56:39.562 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:39.562 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.562 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.562 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:39.562 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.627278+0000", 2024-09-21T13:56:39.562 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:39.562 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.562 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.562 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:39.562 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.627280+0000", 2024-09-21T13:56:39.562 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7239999999999999e-06 2024-09-21T13:56:39.562 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.563 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.563 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:39.563 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.627303+0000", 2024-09-21T13:56:39.563 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2844999999999999e-05 2024-09-21T13:56:39.563 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.563 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.563 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:39.563 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.627319+0000", 2024-09-21T13:56:39.563 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6714e-05 2024-09-21T13:56:39.563 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.563 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.563 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:39.563 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.627328+0000", 2024-09-21T13:56:39.563 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.7180000000000002e-06 2024-09-21T13:56:39.563 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.564 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.564 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:39.564 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.627997+0000", 2024-09-21T13:56:39.564 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00066938200000000005 2024-09-21T13:56:39.564 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.564 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.564 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:56:39.564 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.629243+0000", 2024-09-21T13:56:39.564 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0012453430000000001 2024-09-21T13:56:39.564 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.564 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.564 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,6", 2024-09-21T13:56:39.564 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.629398+0000", 2024-09-21T13:56:39.564 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000155179 2024-09-21T13:56:39.565 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.565 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.565 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:39.565 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.631277+0000", 2024-09-21T13:56:39.565 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001879181 2024-09-21T13:56:39.565 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.565 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.565 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:56:39.565 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.631907+0000", 2024-09-21T13:56:39.565 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00062996799999999996 2024-09-21T13:56:39.565 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.565 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.565 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:39.565 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.768520+0000", 2024-09-21T13:56:39.565 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.13661250899999999 2024-09-21T13:56:39.566 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.566 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.566 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:56:39.566 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.768705+0000", 2024-09-21T13:56:39.566 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00018564699999999999 2024-09-21T13:56:39.566 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.566 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.566 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:39.566 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.768766+0000", 2024-09-21T13:56:39.566 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0835000000000003e-05 2024-09-21T13:56:39.566 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:39.566 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:39.566 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:39.567 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.567 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.567 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:36870 7.b 7:d08d3ab5:::benchmark_data_smithi049_76798_object36869:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-21T13:56:39.567 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:30.637634+0000", 2024-09-21T13:56:39.567 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 368.67069976300002, 2024-09-21T13:56:39.567 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.13145359500000001, 2024-09-21T13:56:39.567 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:39.567 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:39.567 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:39.567 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:56:39.567 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:56:39.567 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:56:39.567 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 36870 2024-09-21T13:56:39.567 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.568 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:39.568 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.568 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:39.568 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.637634+0000", 2024-09-21T13:56:39.568 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:39.568 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.568 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.568 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:39.568 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.637634+0000", 2024-09-21T13:56:39.568 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:39.568 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.568 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.568 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:39.568 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.637636+0000", 2024-09-21T13:56:39.569 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5489999999999999e-06 2024-09-21T13:56:39.569 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.569 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.569 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:39.569 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.637759+0000", 2024-09-21T13:56:39.569 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012323799999999999 2024-09-21T13:56:39.569 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.569 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.569 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:39.569 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.637772+0000", 2024-09-21T13:56:39.569 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3318e-05 2024-09-21T13:56:39.569 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.569 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.570 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:39.570 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.637780+0000", 2024-09-21T13:56:39.570 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3710000000000002e-06 2024-09-21T13:56:39.570 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.570 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.570 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:39.570 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.637841+0000", 2024-09-21T13:56:39.570 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0897999999999999e-05 2024-09-21T13:56:39.570 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.570 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.570 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:56:39.570 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.638039+0000", 2024-09-21T13:56:39.570 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00019803900000000001 2024-09-21T13:56:39.570 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.571 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.571 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,6", 2024-09-21T13:56:39.571 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.638176+0000", 2024-09-21T13:56:39.571 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013776199999999999 2024-09-21T13:56:39.571 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.571 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.571 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:39.571 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.639679+0000", 2024-09-21T13:56:39.571 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0015029450000000001 2024-09-21T13:56:39.571 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.571 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.571 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:56:39.571 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.639708+0000", 2024-09-21T13:56:39.571 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.8839e-05 2024-09-21T13:56:39.572 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.572 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.572 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:39.572 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.768901+0000", 2024-09-21T13:56:39.572 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.12919276900000001 2024-09-21T13:56:39.572 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.572 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.572 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:56:39.572 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.769000+0000", 2024-09-21T13:56:39.572 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9220000000000002e-05 2024-09-21T13:56:39.572 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.572 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.572 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:39.572 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.769088+0000", 2024-09-21T13:56:39.573 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.7646999999999996e-05 2024-09-21T13:56:39.573 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:39.573 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:39.573 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:39.573 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.573 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.573 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:36884 7.b 7:d5aed2de:::benchmark_data_smithi049_76798_object36883:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-21T13:56:39.573 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:30.647863+0000", 2024-09-21T13:56:39.573 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 368.66047053599999, 2024-09-21T13:56:39.573 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.12384792999999999, 2024-09-21T13:56:39.573 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:39.573 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:39.573 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:39.573 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:56:39.574 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:56:39.574 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:56:39.574 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 36884 2024-09-21T13:56:39.574 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.574 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:39.574 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.574 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:39.574 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.647863+0000", 2024-09-21T13:56:39.574 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:39.574 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.574 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.574 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:39.574 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.647863+0000", 2024-09-21T13:56:39.574 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:39.575 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.575 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.575 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:39.575 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.647865+0000", 2024-09-21T13:56:39.575 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4339999999999999e-06 2024-09-21T13:56:39.575 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.575 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.575 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:39.575 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.647886+0000", 2024-09-21T13:56:39.575 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0919000000000001e-05 2024-09-21T13:56:39.575 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.575 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.575 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:39.575 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.647899+0000", 2024-09-21T13:56:39.575 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2995e-05 2024-09-21T13:56:39.576 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.576 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.576 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:39.576 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.647905+0000", 2024-09-21T13:56:39.576 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.212e-06 2024-09-21T13:56:39.576 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.576 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.576 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:39.576 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.647964+0000", 2024-09-21T13:56:39.576 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9555000000000002e-05 2024-09-21T13:56:39.576 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.576 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.576 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:56:39.577 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.648095+0000", 2024-09-21T13:56:39.577 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013020800000000001 2024-09-21T13:56:39.577 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.577 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.577 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,6", 2024-09-21T13:56:39.577 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.648210+0000", 2024-09-21T13:56:39.577 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000114827 2024-09-21T13:56:39.577 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.577 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.577 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:56:39.577 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.649302+0000", 2024-09-21T13:56:39.577 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0010921010000000001 2024-09-21T13:56:39.577 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.577 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.577 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:39.578 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.649464+0000", 2024-09-21T13:56:39.578 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016226200000000001 2024-09-21T13:56:39.578 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.578 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.578 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:39.578 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.771611+0000", 2024-09-21T13:56:39.578 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.122147371 2024-09-21T13:56:39.578 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.578 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.578 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:56:39.578 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.771668+0000", 2024-09-21T13:56:39.578 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6802000000000003e-05 2024-09-21T13:56:39.578 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.578 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.579 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:39.579 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:30.771711+0000", 2024-09-21T13:56:39.579 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3244000000000001e-05 2024-09-21T13:56:39.579 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:39.579 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:39.579 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:39.579 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.579 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.579 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:49366 7.9 7:956216ae:::benchmark_data_smithi049_76798_object49365:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T13:56:39.579 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:38.258962+0000", 2024-09-21T13:56:39.579 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 361.04937221900002, 2024-09-21T13:56:39.579 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.150737654, 2024-09-21T13:56:39.579 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:39.579 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:39.580 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:39.580 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:56:39.580 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:56:39.580 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:56:39.580 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 49366 2024-09-21T13:56:39.580 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.580 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:39.580 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.580 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:39.580 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.258962+0000", 2024-09-21T13:56:39.580 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:39.580 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.580 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.580 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:39.581 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.258962+0000", 2024-09-21T13:56:39.581 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:39.581 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.581 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.581 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:39.581 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.258964+0000", 2024-09-21T13:56:39.581 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.176e-06 2024-09-21T13:56:39.581 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.581 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.581 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:39.581 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.258988+0000", 2024-09-21T13:56:39.581 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4298e-05 2024-09-21T13:56:39.581 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.581 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.582 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:39.582 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.259001+0000", 2024-09-21T13:56:39.582 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2724999999999999e-05 2024-09-21T13:56:39.582 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.582 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.582 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:39.582 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.259008+0000", 2024-09-21T13:56:39.582 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5900000000000002e-06 2024-09-21T13:56:39.582 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.582 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.582 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:39.582 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.259093+0000", 2024-09-21T13:56:39.582 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4044999999999999e-05 2024-09-21T13:56:39.582 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.583 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.583 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:56:39.583 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.259223+0000", 2024-09-21T13:56:39.583 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000130264 2024-09-21T13:56:39.583 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.583 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.583 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,4", 2024-09-21T13:56:39.583 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.259375+0000", 2024-09-21T13:56:39.583 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00015262899999999999 2024-09-21T13:56:39.583 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.583 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.583 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:56:39.583 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.261242+0000", 2024-09-21T13:56:39.583 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0018661019999999999 2024-09-21T13:56:39.584 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.584 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.584 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:39.584 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.261949+0000", 2024-09-21T13:56:39.584 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000707513 2024-09-21T13:56:39.584 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.584 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.584 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:39.584 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.409559+0000", 2024-09-21T13:56:39.584 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.14761001100000001 2024-09-21T13:56:39.584 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.584 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.584 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:56:39.584 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.409654+0000", 2024-09-21T13:56:39.585 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.5081000000000007e-05 2024-09-21T13:56:39.585 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.585 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.585 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:39.585 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.409699+0000", 2024-09-21T13:56:39.585 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5219999999999997e-05 2024-09-21T13:56:39.585 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:39.585 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:39.585 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:39.585 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.585 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.585 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:151198 7.b 7:d833e411:::benchmark_data_smithi049_76798_object14778:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-21T13:56:39.585 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:51:53.955747+0000", 2024-09-21T13:56:39.585 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 285.35258660300002, 2024-09-21T13:56:39.585 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.205769173, 2024-09-21T13:56:39.586 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:39.586 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:39.586 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:39.586 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:56:39.586 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:56:39.586 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:56:39.586 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 151198 2024-09-21T13:56:39.586 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.586 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:39.586 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.586 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:39.586 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:53.955747+0000", 2024-09-21T13:56:39.586 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:39.586 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.587 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.587 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:39.587 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:53.955747+0000", 2024-09-21T13:56:39.587 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:39.587 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.587 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.587 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:39.587 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:53.955749+0000", 2024-09-21T13:56:39.587 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7570000000000001e-06 2024-09-21T13:56:39.587 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.587 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.587 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:39.587 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:53.955756+0000", 2024-09-21T13:56:39.587 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5939999999999999e-06 2024-09-21T13:56:39.588 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.588 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.588 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:39.588 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:53.955775+0000", 2024-09-21T13:56:39.588 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9083e-05 2024-09-21T13:56:39.588 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.588 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.588 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:39.588 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:53.955784+0000", 2024-09-21T13:56:39.588 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2380000000000003e-06 2024-09-21T13:56:39.588 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.588 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.588 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:39.588 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:53.955852+0000", 2024-09-21T13:56:39.589 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7799000000000006e-05 2024-09-21T13:56:39.589 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.589 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.589 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:56:39.589 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:53.955976+0000", 2024-09-21T13:56:39.589 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000123919 2024-09-21T13:56:39.589 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.589 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.589 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,6", 2024-09-21T13:56:39.589 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:53.956096+0000", 2024-09-21T13:56:39.589 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012077400000000001 2024-09-21T13:56:39.589 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.589 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.590 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:56:39.590 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:53.959710+0000", 2024-09-21T13:56:39.590 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00361356 2024-09-21T13:56:39.590 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.590 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.590 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:39.590 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:53.960238+0000", 2024-09-21T13:56:39.590 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00052753399999999997 2024-09-21T13:56:39.590 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.590 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.590 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:39.590 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:54.161354+0000", 2024-09-21T13:56:39.590 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.20111604899999999 2024-09-21T13:56:39.590 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.591 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.591 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:56:39.591 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:54.161458+0000", 2024-09-21T13:56:39.591 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000104277 2024-09-21T13:56:39.591 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.591 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.591 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:39.591 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:54.161516+0000", 2024-09-21T13:56:39.591 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8588999999999998e-05 2024-09-21T13:56:39.591 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:39.591 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:39.591 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:39.591 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.591 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.592 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:151221 7.b 7:d2e7434d:::benchmark_data_smithi049_76798_object14801:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-21T13:56:39.592 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:51:53.970971+0000", 2024-09-21T13:56:39.592 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 285.33736308300001, 2024-09-21T13:56:39.592 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.19072438799999999, 2024-09-21T13:56:39.592 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:39.592 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:39.592 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:39.592 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:56:39.592 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:56:39.592 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:56:39.592 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 151221 2024-09-21T13:56:39.592 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.592 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:39.593 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.593 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:39.593 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:53.970971+0000", 2024-09-21T13:56:39.593 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:39.593 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.593 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.593 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:39.593 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:53.970971+0000", 2024-09-21T13:56:39.593 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:39.593 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.593 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.593 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:39.593 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:53.970973+0000", 2024-09-21T13:56:39.593 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0789999999999999e-06 2024-09-21T13:56:39.593 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.594 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.594 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:39.594 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:53.970979+0000", 2024-09-21T13:56:39.594 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0460000000000002e-06 2024-09-21T13:56:39.594 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.594 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.594 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:39.594 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:53.970997+0000", 2024-09-21T13:56:39.594 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8447000000000001e-05 2024-09-21T13:56:39.594 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.594 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.594 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:39.594 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:53.971006+0000", 2024-09-21T13:56:39.595 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0019999999999995e-06 2024-09-21T13:56:39.595 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.595 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.595 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:39.595 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:53.971118+0000", 2024-09-21T13:56:39.595 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011206599999999999 2024-09-21T13:56:39.595 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.595 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.595 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:56:39.595 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:53.971245+0000", 2024-09-21T13:56:39.595 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000126191 2024-09-21T13:56:39.595 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.595 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.595 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,6", 2024-09-21T13:56:39.596 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:53.971359+0000", 2024-09-21T13:56:39.596 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000114763 2024-09-21T13:56:39.596 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.596 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.596 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:39.596 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:53.978199+0000", 2024-09-21T13:56:39.596 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0068397889999999998 2024-09-21T13:56:39.596 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.596 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.596 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:56:39.596 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:53.978286+0000", 2024-09-21T13:56:39.596 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6990999999999994e-05 2024-09-21T13:56:39.597 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.597 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.597 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:39.597 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:54.161567+0000", 2024-09-21T13:56:39.597 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.183280319 2024-09-21T13:56:39.597 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.597 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.597 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:56:39.598 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:54.161653+0000", 2024-09-21T13:56:39.598 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6495000000000004e-05 2024-09-21T13:56:39.598 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.598 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.598 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:39.598 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:54.161695+0000", 2024-09-21T13:56:39.598 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2200000000000003e-05 2024-09-21T13:56:39.598 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:39.598 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:39.598 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:39.598 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.598 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.598 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:151223 7.b 7:d20c811f:::benchmark_data_smithi049_76798_object14803:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-21T13:56:39.599 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:51:53.971043+0000", 2024-09-21T13:56:39.599 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 285.33729102900003, 2024-09-21T13:56:39.599 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.192804422, 2024-09-21T13:56:39.599 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:39.599 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:39.599 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:39.599 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:56:39.599 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:56:39.599 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:56:39.599 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 151223 2024-09-21T13:56:39.599 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.599 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:39.599 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.599 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:39.600 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:53.971043+0000", 2024-09-21T13:56:39.600 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:39.600 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.600 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.600 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:39.600 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:53.971043+0000", 2024-09-21T13:56:39.600 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:39.600 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.600 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.600 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:39.600 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:53.971044+0000", 2024-09-21T13:56:39.600 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7800000000000001e-07 2024-09-21T13:56:39.600 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.600 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.601 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:39.601 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:53.971048+0000", 2024-09-21T13:56:39.601 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8970000000000001e-06 2024-09-21T13:56:39.601 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.601 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.601 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:39.601 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:53.971057+0000", 2024-09-21T13:56:39.601 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4180000000000003e-06 2024-09-21T13:56:39.601 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.601 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.601 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:39.601 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:53.971064+0000", 2024-09-21T13:56:39.601 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7739999999999999e-06 2024-09-21T13:56:39.601 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.601 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.602 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:39.602 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:53.974514+0000", 2024-09-21T13:56:39.602 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0034507230000000002 2024-09-21T13:56:39.602 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.602 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.602 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:56:39.602 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:53.974651+0000", 2024-09-21T13:56:39.602 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013623500000000001 2024-09-21T13:56:39.602 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.602 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.602 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,6", 2024-09-21T13:56:39.602 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:53.974777+0000", 2024-09-21T13:56:39.602 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000126754 2024-09-21T13:56:39.602 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.603 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.603 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:39.603 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:53.978262+0000", 2024-09-21T13:56:39.603 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0034844780000000001 2024-09-21T13:56:39.603 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.603 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.603 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:56:39.603 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:53.978293+0000", 2024-09-21T13:56:39.603 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1443999999999999e-05 2024-09-21T13:56:39.603 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.603 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.603 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:39.603 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:54.163471+0000", 2024-09-21T13:56:39.604 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.18517719999999999 2024-09-21T13:56:39.604 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.604 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.604 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:56:39.604 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:54.163783+0000", 2024-09-21T13:56:39.604 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00031266300000000003 2024-09-21T13:56:39.604 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.604 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.604 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:39.604 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:54.163847+0000", 2024-09-21T13:56:39.604 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4058000000000002e-05 2024-09-21T13:56:39.604 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:39.604 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:39.604 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:39.605 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.605 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.605 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:151227 7.b 7:d41a8d35:::benchmark_data_smithi049_76798_object14807:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-21T13:56:39.605 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:51:53.975632+0000", 2024-09-21T13:56:39.605 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 285.332701574, 2024-09-21T13:56:39.605 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.18847303800000001, 2024-09-21T13:56:39.605 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:39.605 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:39.605 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:39.605 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:56:39.605 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:56:39.605 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:56:39.605 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 151227 2024-09-21T13:56:39.605 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.606 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:39.606 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.606 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:39.606 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:53.975632+0000", 2024-09-21T13:56:39.606 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:39.606 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.606 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.606 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:39.606 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:53.975632+0000", 2024-09-21T13:56:39.606 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:39.606 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.606 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.606 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:39.606 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:53.975634+0000", 2024-09-21T13:56:39.607 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5239999999999999e-06 2024-09-21T13:56:39.607 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.607 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.607 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:39.607 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:53.975640+0000", 2024-09-21T13:56:39.607 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3820000000000001e-06 2024-09-21T13:56:39.607 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.607 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.607 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:39.607 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:53.975656+0000", 2024-09-21T13:56:39.607 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5342999999999999e-05 2024-09-21T13:56:39.607 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.607 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.607 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:39.608 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:53.975665+0000", 2024-09-21T13:56:39.608 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0850000000000003e-06 2024-09-21T13:56:39.608 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.608 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.608 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:39.608 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:53.978323+0000", 2024-09-21T13:56:39.608 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002658292 2024-09-21T13:56:39.608 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.608 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.608 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:56:39.608 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:53.978438+0000", 2024-09-21T13:56:39.608 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000115287 2024-09-21T13:56:39.608 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.609 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.609 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,6", 2024-09-21T13:56:39.609 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:53.978559+0000", 2024-09-21T13:56:39.609 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000121193 2024-09-21T13:56:39.609 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.609 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.609 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:39.609 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:53.981404+0000", 2024-09-21T13:56:39.609 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0028443069999999999 2024-09-21T13:56:39.609 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.609 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.609 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:56:39.609 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:53.981414+0000", 2024-09-21T13:56:39.609 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9750000000000002e-06 2024-09-21T13:56:39.610 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.610 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.610 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:39.610 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:54.163907+0000", 2024-09-21T13:56:39.610 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.182493085 2024-09-21T13:56:39.610 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.610 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.610 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:56:39.610 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:54.164066+0000", 2024-09-21T13:56:39.610 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00015967600000000001 2024-09-21T13:56:39.610 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.610 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.610 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:39.610 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:51:54.164105+0000", 2024-09-21T13:56:39.611 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8889e-05 2024-09-21T13:56:39.611 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:39.611 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:39.611 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:39.611 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.611 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.611 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:14194 8.a 8:5d4d4bd0:::benchmark_data_smithi049_109311_object14193:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-09-21T13:56:39.611 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:54:23.112840+0000", 2024-09-21T13:56:39.611 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 136.195494118, 2024-09-21T13:56:39.611 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.12678025600000001, 2024-09-21T13:56:39.611 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:39.611 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:39.611 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:39.611 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:56:39.611 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:56:39.612 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:56:39.612 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 14194 2024-09-21T13:56:39.612 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.612 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:39.612 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.612 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:39.612 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.112840+0000", 2024-09-21T13:56:39.612 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:39.612 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.612 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.612 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:39.612 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.112840+0000", 2024-09-21T13:56:39.612 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:39.612 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.613 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.613 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:39.613 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.112841+0000", 2024-09-21T13:56:39.613 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.212e-06 2024-09-21T13:56:39.613 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.613 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.613 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:39.613 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.112862+0000", 2024-09-21T13:56:39.613 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0718000000000001e-05 2024-09-21T13:56:39.613 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.613 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.613 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:39.613 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.112872+0000", 2024-09-21T13:56:39.613 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0575e-05 2024-09-21T13:56:39.613 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.614 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.614 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:39.614 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.112879+0000", 2024-09-21T13:56:39.614 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8820000000000003e-06 2024-09-21T13:56:39.614 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.614 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.614 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:39.614 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.112926+0000", 2024-09-21T13:56:39.614 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7275999999999999e-05 2024-09-21T13:56:39.614 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.614 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.614 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:56:39.614 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.113031+0000", 2024-09-21T13:56:39.615 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010412999999999999 2024-09-21T13:56:39.615 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.615 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.615 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,5", 2024-09-21T13:56:39.615 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.113158+0000", 2024-09-21T13:56:39.615 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012715 2024-09-21T13:56:39.615 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.615 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.615 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:56:39.615 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.114101+0000", 2024-09-21T13:56:39.615 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00094301300000000003 2024-09-21T13:56:39.615 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.615 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.615 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:39.615 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.115304+0000", 2024-09-21T13:56:39.616 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0012028430000000001 2024-09-21T13:56:39.616 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.616 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.616 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:39.616 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.239283+0000", 2024-09-21T13:56:39.616 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.12397904899999999 2024-09-21T13:56:39.616 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.616 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.616 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:56:39.616 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.239569+0000", 2024-09-21T13:56:39.616 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00028588499999999999 2024-09-21T13:56:39.616 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.616 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.616 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:39.617 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.239620+0000", 2024-09-21T13:56:39.617 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1523000000000001e-05 2024-09-21T13:56:39.617 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:39.617 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:39.617 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:39.617 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.617 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.617 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:14201 8.1 8:8e5d31ee:::benchmark_data_smithi049_109311_object14200:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-09-21T13:56:39.617 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:54:23.113457+0000", 2024-09-21T13:56:39.617 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 136.19487689799999, 2024-09-21T13:56:39.617 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.12558186399999999, 2024-09-21T13:56:39.617 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:39.617 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:39.617 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:39.618 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:56:39.618 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:56:39.618 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:56:39.618 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 14201 2024-09-21T13:56:39.618 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.618 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:39.618 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.618 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:39.618 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.113457+0000", 2024-09-21T13:56:39.618 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:39.618 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.618 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.618 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:39.618 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.113457+0000", 2024-09-21T13:56:39.619 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:39.619 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.619 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.619 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:39.619 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.113458+0000", 2024-09-21T13:56:39.619 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1200000000000001e-07 2024-09-21T13:56:39.619 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.619 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.619 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:39.619 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.113507+0000", 2024-09-21T13:56:39.619 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8871e-05 2024-09-21T13:56:39.619 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.619 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.620 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:39.620 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.113520+0000", 2024-09-21T13:56:39.620 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2965999999999999e-05 2024-09-21T13:56:39.620 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.620 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.620 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:39.620 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.113530+0000", 2024-09-21T13:56:39.620 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.063e-05 2024-09-21T13:56:39.620 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.620 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.620 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:39.620 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.113607+0000", 2024-09-21T13:56:39.620 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6957000000000002e-05 2024-09-21T13:56:39.620 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.621 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.621 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:56:39.621 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.113711+0000", 2024-09-21T13:56:39.621 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000103754 2024-09-21T13:56:39.621 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.621 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.621 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,2", 2024-09-21T13:56:39.621 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.113828+0000", 2024-09-21T13:56:39.621 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000116673 2024-09-21T13:56:39.621 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.621 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.621 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:39.621 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.115357+0000", 2024-09-21T13:56:39.621 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001529508 2024-09-21T13:56:39.621 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.622 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.622 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:56:39.622 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.115368+0000", 2024-09-21T13:56:39.622 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1092e-05 2024-09-21T13:56:39.622 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.622 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.622 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:39.622 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.238880+0000", 2024-09-21T13:56:39.622 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.123511857 2024-09-21T13:56:39.622 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.622 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.622 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:56:39.622 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.238996+0000", 2024-09-21T13:56:39.622 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011578699999999999 2024-09-21T13:56:39.623 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.623 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.623 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:39.623 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.239039+0000", 2024-09-21T13:56:39.623 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2857e-05 2024-09-21T13:56:39.623 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:39.623 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:39.623 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:39.623 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.623 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.623 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:14203 8.7 8:e625aa2d:::benchmark_data_smithi049_109311_object14202:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-09-21T13:56:39.623 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:54:23.115305+0000", 2024-09-21T13:56:39.623 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 136.19302844200001, 2024-09-21T13:56:39.623 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.124634751, 2024-09-21T13:56:39.624 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:39.624 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:39.624 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:39.624 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:56:39.624 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:56:39.624 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:56:39.624 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 14203 2024-09-21T13:56:39.624 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.624 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:39.624 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.624 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:39.624 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.115305+0000", 2024-09-21T13:56:39.624 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:39.624 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.625 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.625 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:39.625 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.115305+0000", 2024-09-21T13:56:39.625 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:39.625 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.625 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.625 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:39.625 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.115307+0000", 2024-09-21T13:56:39.625 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4759999999999999e-06 2024-09-21T13:56:39.625 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.625 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.625 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:39.625 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.115332+0000", 2024-09-21T13:56:39.625 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4630999999999998e-05 2024-09-21T13:56:39.626 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.626 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.626 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:39.626 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.115345+0000", 2024-09-21T13:56:39.626 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3577999999999999e-05 2024-09-21T13:56:39.626 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.626 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.626 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:39.626 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.115353+0000", 2024-09-21T13:56:39.626 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1729999999999995e-06 2024-09-21T13:56:39.626 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.626 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.626 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:39.626 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.115469+0000", 2024-09-21T13:56:39.627 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011524 2024-09-21T13:56:39.627 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.627 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.627 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:56:39.627 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.115566+0000", 2024-09-21T13:56:39.627 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7862000000000006e-05 2024-09-21T13:56:39.627 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.627 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.627 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,6", 2024-09-21T13:56:39.627 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.115646+0000", 2024-09-21T13:56:39.627 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9612999999999998e-05 2024-09-21T13:56:39.627 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.627 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.627 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:39.627 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.239158+0000", 2024-09-21T13:56:39.628 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.12351160799999999 2024-09-21T13:56:39.628 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.628 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.628 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:39.628 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.239849+0000", 2024-09-21T13:56:39.628 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00069135799999999999 2024-09-21T13:56:39.628 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.628 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.628 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:56:39.628 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.239860+0000", 2024-09-21T13:56:39.628 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1389000000000001e-05 2024-09-21T13:56:39.628 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.628 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.628 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:56:39.628 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.239905+0000", 2024-09-21T13:56:39.629 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4363999999999997e-05 2024-09-21T13:56:39.629 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.629 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.629 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:39.629 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.239940+0000", 2024-09-21T13:56:39.629 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5459000000000001e-05 2024-09-21T13:56:39.629 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:39.629 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:39.629 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:39.629 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.629 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.629 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:14208 8.3 8:c3b3997b:::benchmark_data_smithi049_109311_object14207:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-09-21T13:56:39.629 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:54:23.115795+0000", 2024-09-21T13:56:39.629 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 136.19253898700001, 2024-09-21T13:56:39.630 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.12543985399999999, 2024-09-21T13:56:39.630 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:39.630 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:39.630 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:39.630 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:56:39.630 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:56:39.630 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:56:39.630 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 14208 2024-09-21T13:56:39.630 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.630 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:39.630 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.630 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:39.630 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.115795+0000", 2024-09-21T13:56:39.631 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:39.631 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.631 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.631 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:39.631 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.115795+0000", 2024-09-21T13:56:39.631 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:39.631 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.631 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.631 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:39.631 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.115796+0000", 2024-09-21T13:56:39.631 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0449999999999999e-06 2024-09-21T13:56:39.631 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.631 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.631 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:39.631 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.115821+0000", 2024-09-21T13:56:39.632 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5092999999999998e-05 2024-09-21T13:56:39.632 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.632 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.632 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:39.632 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.237205+0000", 2024-09-21T13:56:39.632 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.121383782 2024-09-21T13:56:39.632 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.632 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.632 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:39.632 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.237223+0000", 2024-09-21T13:56:39.632 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7728999999999999e-05 2024-09-21T13:56:39.632 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.632 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.633 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:39.633 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.237292+0000", 2024-09-21T13:56:39.633 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9070000000000001e-05 2024-09-21T13:56:39.633 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.633 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.633 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:56:39.633 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.237432+0000", 2024-09-21T13:56:39.633 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000139975 2024-09-21T13:56:39.633 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.633 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.633 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,6", 2024-09-21T13:56:39.633 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.237602+0000", 2024-09-21T13:56:39.633 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00017061299999999999 2024-09-21T13:56:39.633 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.633 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.634 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:39.634 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.240778+0000", 2024-09-21T13:56:39.634 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.003175673 2024-09-21T13:56:39.634 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.634 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.634 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:56:39.634 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.240789+0000", 2024-09-21T13:56:39.634 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1408e-05 2024-09-21T13:56:39.634 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.634 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.634 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:39.634 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.241144+0000", 2024-09-21T13:56:39.634 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00035457000000000002 2024-09-21T13:56:39.634 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.635 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.635 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:56:39.635 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.241194+0000", 2024-09-21T13:56:39.635 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0318000000000002e-05 2024-09-21T13:56:39.635 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.635 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.635 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:39.635 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.241235+0000", 2024-09-21T13:56:39.635 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0577999999999997e-05 2024-09-21T13:56:39.635 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:39.635 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:39.635 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:39.635 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.635 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.636 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.15207.0:14203 8.7 e189/183 ondisk, result = 0)", 2024-09-21T13:56:39.636 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:54:23.117028+0000", 2024-09-21T13:56:39.636 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 136.191305903, 2024-09-21T13:56:39.636 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.122915047, 2024-09-21T13:56:39.636 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:39.636 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:39.636 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T13:56:39.636 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:39.636 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.636 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:39.636 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.117028+0000", 2024-09-21T13:56:39.636 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:39.636 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.636 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.637 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:39.637 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.117028+0000", 2024-09-21T13:56:39.637 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:39.637 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.637 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.637 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:39.637 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.117029+0000", 2024-09-21T13:56:39.637 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8899999999999998e-07 2024-09-21T13:56:39.637 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.637 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.637 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:39.637 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.117036+0000", 2024-09-21T13:56:39.637 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1099999999999997e-06 2024-09-21T13:56:39.637 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.638 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.638 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:39.638 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.238624+0000", 2024-09-21T13:56:39.638 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.121588179 2024-09-21T13:56:39.638 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.638 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.638 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:39.638 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.238636+0000", 2024-09-21T13:56:39.638 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1922e-05 2024-09-21T13:56:39.638 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.638 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.638 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:39.638 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.239836+0000", 2024-09-21T13:56:39.638 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0011996260000000001 2024-09-21T13:56:39.639 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.639 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.639 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:56:39.639 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.239843+0000", 2024-09-21T13:56:39.639 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5739999999999998e-06 2024-09-21T13:56:39.639 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.639 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.639 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:39.639 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.239943+0000", 2024-09-21T13:56:39.639 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9647000000000003e-05 2024-09-21T13:56:39.639 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:39.639 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:39.639 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:39.639 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.640 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.640 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24365.0:1780 4.4 4:22d26bf9:::data_loggenerations_metadata:head [watch reconnect cookie 94025139211520 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-09-21T13:56:39.640 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:55:40.878748+0000", 2024-09-21T13:56:39.640 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 58.429586334, 2024-09-21T13:56:39.640 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.39793440499999999, 2024-09-21T13:56:39.640 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:39.640 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:39.640 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T13:56:39.640 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:56:39.640 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24365", 2024-09-21T13:56:39.640 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3404764347", 2024-09-21T13:56:39.640 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 1780 2024-09-21T13:56:39.640 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.640 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:39.640 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.641 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:39.641 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:40.878748+0000", 2024-09-21T13:56:39.641 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:39.641 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.641 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.641 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:39.641 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:40.878748+0000", 2024-09-21T13:56:39.641 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:39.641 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.641 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.641 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:39.641 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:40.878750+0000", 2024-09-21T13:56:39.641 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.962e-06 2024-09-21T13:56:39.641 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.642 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.642 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:39.642 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:40.878756+0000", 2024-09-21T13:56:39.642 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3080000000000004e-06 2024-09-21T13:56:39.642 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.642 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.642 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:39.642 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:40.878768+0000", 2024-09-21T13:56:39.642 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.203e-05 2024-09-21T13:56:39.642 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.642 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.642 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:39.642 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:40.878776+0000", 2024-09-21T13:56:39.642 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4849999999999992e-06 2024-09-21T13:56:39.643 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.643 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.643 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:39.643 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:40.878849+0000", 2024-09-21T13:56:39.643 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2670999999999996e-05 2024-09-21T13:56:39.643 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.643 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.643 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:56:39.643 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:40.878859+0000", 2024-09-21T13:56:39.643 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6099999999999995e-06 2024-09-21T13:56:39.643 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.643 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.643 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:39.643 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:41.271074+0000", 2024-09-21T13:56:39.644 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.39221556499999999 2024-09-21T13:56:39.644 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.644 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.644 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:56:39.644 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:41.271224+0000", 2024-09-21T13:56:39.644 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00015018400000000001 2024-09-21T13:56:39.644 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.644 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.644 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:39.644 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:41.276682+0000", 2024-09-21T13:56:39.644 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00545759 2024-09-21T13:56:39.644 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:39.644 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:39.644 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:39.644 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.645 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.645 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24365.0:1781 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 94025139203456 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-09-21T13:56:39.645 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:55:40.878794+0000", 2024-09-21T13:56:39.645 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 58.429539878, 2024-09-21T13:56:39.645 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.39819017699999998, 2024-09-21T13:56:39.645 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:39.645 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:39.645 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T13:56:39.645 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:56:39.645 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24365", 2024-09-21T13:56:39.645 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3404764347", 2024-09-21T13:56:39.645 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 1781 2024-09-21T13:56:39.645 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.645 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:39.646 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.646 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:39.646 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:40.878794+0000", 2024-09-21T13:56:39.646 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:39.646 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.646 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.646 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:39.646 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:40.878794+0000", 2024-09-21T13:56:39.646 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:39.646 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.646 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.646 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:39.646 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:40.878795+0000", 2024-09-21T13:56:39.647 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9299999999999998e-07 2024-09-21T13:56:39.647 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.647 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.647 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:39.647 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:40.878798+0000", 2024-09-21T13:56:39.647 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.0120000000000001e-06 2024-09-21T13:56:39.647 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.647 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.647 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:39.647 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:40.878805+0000", 2024-09-21T13:56:39.647 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6830000000000008e-06 2024-09-21T13:56:39.647 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.647 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.647 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:39.648 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:40.878811+0000", 2024-09-21T13:56:39.648 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9009999999999999e-06 2024-09-21T13:56:39.648 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.648 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.648 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:39.648 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:40.878873+0000", 2024-09-21T13:56:39.648 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1412000000000001e-05 2024-09-21T13:56:39.648 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.648 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.648 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:56:39.648 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:40.878882+0000", 2024-09-21T13:56:39.648 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2289999999999992e-06 2024-09-21T13:56:39.648 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.648 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.648 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:39.649 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:41.274347+0000", 2024-09-21T13:56:39.649 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.39546476200000003 2024-09-21T13:56:39.649 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.649 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.649 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:56:39.649 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:41.274501+0000", 2024-09-21T13:56:39.649 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00015485499999999999 2024-09-21T13:56:39.649 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.649 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.649 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:39.649 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:41.276984+0000", 2024-09-21T13:56:39.649 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0024828300000000001 2024-09-21T13:56:39.649 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:39.649 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:39.650 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:39.650 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.650 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.650 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:168449 8.3 8:c495aa3f:::benchmark_data_smithi049_109311_object41930:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e225)", 2024-09-21T13:56:39.650 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:56:22.933239+0000", 2024-09-21T13:56:39.650 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 16.375094833999999, 2024-09-21T13:56:39.650 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.097925527999999998, 2024-09-21T13:56:39.650 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:56:39.650 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:56:39.650 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:39.650 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:56:39.650 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:56:39.650 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:56:39.650 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 168449 2024-09-21T13:56:39.651 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.651 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:56:39.651 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.651 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:56:39.651 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:56:22.933239+0000", 2024-09-21T13:56:39.651 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:39.651 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.651 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.651 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:56:39.651 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:56:22.933239+0000", 2024-09-21T13:56:39.651 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:56:39.651 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.651 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.651 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:56:39.652 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:56:22.933241+0000", 2024-09-21T13:56:39.652 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6160000000000001e-06 2024-09-21T13:56:39.652 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.652 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.652 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:56:39.652 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:56:22.933246+0000", 2024-09-21T13:56:39.652 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9479999999999999e-06 2024-09-21T13:56:39.652 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.652 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.652 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:56:39.652 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:56:22.933258+0000", 2024-09-21T13:56:39.652 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1908e-05 2024-09-21T13:56:39.652 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.652 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.652 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:56:39.653 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:56:22.933265+0000", 2024-09-21T13:56:39.653 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3030000000000002e-06 2024-09-21T13:56:39.653 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.653 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.653 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:56:39.653 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:56:22.933317+0000", 2024-09-21T13:56:39.653 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1907000000000003e-05 2024-09-21T13:56:39.653 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.653 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.653 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:56:39.653 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:56:22.933422+0000", 2024-09-21T13:56:39.653 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010534799999999999 2024-09-21T13:56:39.653 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.653 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.654 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,5", 2024-09-21T13:56:39.654 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:56:22.933541+0000", 2024-09-21T13:56:39.654 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000119084 2024-09-21T13:56:39.654 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.654 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.654 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:39.654 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:56:23.020298+0000", 2024-09-21T13:56:39.654 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.086756729000000005 2024-09-21T13:56:39.654 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.654 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.654 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:56:39.654 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:56:23.021162+0000", 2024-09-21T13:56:39.654 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00086432000000000002 2024-09-21T13:56:39.654 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.655 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.655 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:39.655 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:56:23.031081+0000", 2024-09-21T13:56:39.655 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0099186139999999992 2024-09-21T13:56:39.655 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.655 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.655 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:56:39.655 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:56:23.031127+0000", 2024-09-21T13:56:39.655 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6131000000000003e-05 2024-09-21T13:56:39.655 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:56:39.655 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:56:39.655 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:56:39.655 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:56:23.031165+0000", 2024-09-21T13:56:39.655 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7620000000000002e-05 2024-09-21T13:56:39.655 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:39.656 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:39.656 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:39.656 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:56:39.656 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:56:39.656 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T13:56:39.656 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:56:39.685 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:39.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:39.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:56:39.694 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:56:39 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:56:39.787 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:39.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:39.826 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:39.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:39.927 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:39.969 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:39.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:39.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:40.070 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:56:40.096 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:40.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:40.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:56:40.177 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:56:40.197 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:56:40.233 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:40.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:40.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:56:40.334 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:40.370 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:40.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:40.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:40.378 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:40 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:56:40.378 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:40 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 26331/210303 objects degraded (12.521%), 31 pgs degraded (PG_DEGRADED) 2024-09-21T13:56:40.378 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:40 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:56:40.378 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:40 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 26331/210303 objects degraded (12.521%), 31 pgs degraded (PG_DEGRADED) 2024-09-21T13:56:40.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:40 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:56:40.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:40 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 26331/210303 objects degraded (12.521%), 31 pgs degraded (PG_DEGRADED) 2024-09-21T13:56:40.473 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:40.508 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:40.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:40.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:40.576 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T13:56:40.610 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:40.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:40.668 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:40.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:40.754 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:56:40 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:56:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:56:40.770 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:56:40.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:40.820 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:40.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:56:40.910 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T13:56:40.921 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:40.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:40.968 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:40.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:41.069 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:41.116 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:41.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:41.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:41.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:41 smithi187 ceph-mon[33032]: pgmap v374: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 3.8 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 2.12k op/s; 23612/188352 objects degraded (12.536%); 7926/188352 objects misplaced (4.208%) 2024-09-21T13:56:41.218 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:56:41.233 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:41 smithi049 ceph-mon[30080]: pgmap v374: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 3.8 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 2.12k op/s; 23612/188352 objects degraded (12.536%); 7926/188352 objects misplaced (4.208%) 2024-09-21T13:56:41.233 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:41 smithi049 ceph-mon[40489]: pgmap v374: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 3.8 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 2.12k op/s; 23612/188352 objects degraded (12.536%); 7926/188352 objects misplaced (4.208%) 2024-09-21T13:56:41.251 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:41.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:41.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:56:41.289 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:56:41.353 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:56:41.392 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:41.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:41.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:56:41.494 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:41.536 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:41.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:41.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:41.636 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:56:41.638 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:56:41.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:41.695 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:41.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:56:41.796 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:56:41.834 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:41.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:41.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:56:41.903 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T13:56:41.903 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-21T13:56:41.903 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:41.903 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.15207.0:194927 8.e e229/183 8:752c7660:::benchmark_data_smithi049_109311_object68408:head v 229'12178, mlcod=229'12175)", 2024-09-21T13:56:41.903 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:56:41.894766+0000", 2024-09-21T13:56:41.903 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0066105310000000002, 2024-09-21T13:56:41.903 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0066372879999999999, 2024-09-21T13:56:41.903 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:56:41.903 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:56:41.903 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T13:56:41.903 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:56:41.904 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:41.904 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:56:41.904 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:41.894766+0000", 2024-09-21T13:56:41.904 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:56:41.904 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:41.904 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:41.904 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:56:41.904 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:41.894766+0000", 2024-09-21T13:56:41.904 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:56:41.904 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:41.904 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:41.904 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:56:41.904 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:41.894767+0000", 2024-09-21T13:56:41.904 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0690000000000001e-06 2024-09-21T13:56:41.905 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:41.905 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:41.905 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:56:41.905 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:41.894774+0000", 2024-09-21T13:56:41.905 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1420000000000004e-06 2024-09-21T13:56:41.905 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:41.905 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:41.905 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:56:41.905 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:41.894788+0000", 2024-09-21T13:56:41.905 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.403e-05 2024-09-21T13:56:41.905 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:41.905 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:41.905 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:56:41.906 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:41.894795+0000", 2024-09-21T13:56:41.906 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6939999999999998e-06 2024-09-21T13:56:41.906 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:41.906 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:41.906 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:56:41.906 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:41.894852+0000", 2024-09-21T13:56:41.906 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7816999999999999e-05 2024-09-21T13:56:41.906 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:41.906 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:41.906 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:56:41.906 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:41.894873+0000", 2024-09-21T13:56:41.906 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0465000000000001e-05 2024-09-21T13:56:41.906 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:56:41.907 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:56:41.907 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:56:41.907 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:41.907 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:41.907 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.15207.0:194936 8.e e229/183 8:7eebd1b0:::benchmark_data_smithi049_109311_object68417:head v 229'12179, mlcod=229'12176)", 2024-09-21T13:56:41.907 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:56:41.898408+0000", 2024-09-21T13:56:41.907 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.002967889, 2024-09-21T13:56:41.907 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0030506159999999999, 2024-09-21T13:56:41.907 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:56:41.907 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:56:41.907 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T13:56:41.907 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:56:41.907 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:41.907 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:56:41.908 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:41.898408+0000", 2024-09-21T13:56:41.908 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:56:41.908 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:41.908 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:41.908 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:56:41.908 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:41.898408+0000", 2024-09-21T13:56:41.908 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:56:41.908 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:41.908 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:41.908 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:56:41.908 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:41.898409+0000", 2024-09-21T13:56:41.908 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2e-07 2024-09-21T13:56:41.908 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:41.908 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:41.909 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:56:41.909 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:41.898413+0000", 2024-09-21T13:56:41.909 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7099999999999998e-06 2024-09-21T13:56:41.909 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:41.909 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:41.909 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:56:41.909 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:41.898421+0000", 2024-09-21T13:56:41.909 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0229999999999992e-06 2024-09-21T13:56:41.909 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:41.909 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:41.909 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:56:41.909 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:41.898427+0000", 2024-09-21T13:56:41.909 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4160000000000003e-06 2024-09-21T13:56:41.909 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:41.909 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:41.909 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:56:41.910 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:41.899574+0000", 2024-09-21T13:56:41.910 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0011466930000000001 2024-09-21T13:56:41.910 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:41.910 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:41.910 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:56:41.910 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:41.899593+0000", 2024-09-21T13:56:41.910 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9514e-05 2024-09-21T13:56:41.910 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:56:41.910 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:56:41.910 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:56:41.910 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:41.910 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:41.910 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.15207.0:194949 8.2 8:4b341ed7:::benchmark_data_smithi049_109311_object68430:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e229)", 2024-09-21T13:56:41.910 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:56:41.900959+0000", 2024-09-21T13:56:41.910 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.00041713600000000003, 2024-09-21T13:56:41.910 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00055011700000000003, 2024-09-21T13:56:41.911 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:56:41.911 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:56:41.911 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-21T13:56:41.911 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:56:41.911 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.15207", 2024-09-21T13:56:41.911 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:56:41.911 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 194949 2024-09-21T13:56:41.911 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:41.911 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:56:41.911 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:41.911 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:56:41.911 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:41.900959+0000", 2024-09-21T13:56:41.911 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:56:41.911 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:41.911 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:41.912 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:56:41.912 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:41.900959+0000", 2024-09-21T13:56:41.912 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:56:41.912 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:41.912 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:41.912 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:56:41.912 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:41.900960+0000", 2024-09-21T13:56:41.912 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2139999999999999e-06 2024-09-21T13:56:41.912 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:41.912 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:41.912 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:56:41.912 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:41.900965+0000", 2024-09-21T13:56:41.912 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0749999999999997e-06 2024-09-21T13:56:41.912 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:41.912 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:41.912 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:56:41.913 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:41.900979+0000", 2024-09-21T13:56:41.913 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3716e-05 2024-09-21T13:56:41.913 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:41.913 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:41.913 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:56:41.913 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:41.900986+0000", 2024-09-21T13:56:41.913 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1779999999999997e-06 2024-09-21T13:56:41.913 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:41.913 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:41.913 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:56:41.913 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:41.901060+0000", 2024-09-21T13:56:41.913 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3725000000000003e-05 2024-09-21T13:56:41.913 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:41.913 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:41.913 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:56:41.913 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:41.901176+0000", 2024-09-21T13:56:41.914 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000116582 2024-09-21T13:56:41.914 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:41.914 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:41.914 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,5", 2024-09-21T13:56:41.914 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:41.901271+0000", 2024-09-21T13:56:41.914 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.4980000000000002e-05 2024-09-21T13:56:41.914 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:56:41.914 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:56:41.914 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:56:41.914 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:56:41.914 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2024-09-21T13:56:41.914 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 3 2024-09-21T13:56:41.914 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T13:56:41.936 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:56:41.967 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:41.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:41.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:56:42.068 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:56:42.110 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:42.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:42.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:56:42.118 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-09-21T13:56:42.212 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:42.242 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:42.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:42.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:42.344 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:56:42.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:42.375 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:42.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:56:42.392 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:42 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2584357183' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:56:42.392 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:42 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:56:42.392 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:42 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2584357183' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:56:42.392 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:42 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:56:42.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:42 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2584357183' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:56:42.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:42 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:56:42.477 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:56:42.497 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:56:42.497 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:56:42.498 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:56:42.498 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:56:42.498 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:56:42.498 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:56:42.498 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:56:42.498 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-21T13:56:42.498 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:56:42.510 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:56:42.511 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:56:42.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:42.527 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:42.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:56:42.544 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:56:42.544 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:56:42.575 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:56:42.575 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:56:42.628 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:56:42.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:42.671 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:42.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:56:42.738 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T13:56:42.772 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:56:42.820 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:42.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:42.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:56:42.922 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:42.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:42.959 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:42.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:42.997 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [3] live_osds: [0, 1, 4, 5, 6, 7, 2] 2024-09-21T13:56:42.997 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T13:56:42.997 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-09-21T13:56:42.997 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T13:56:42.997 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T13:56:42.997 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-09-21T13:56:43.061 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:56:43.105 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:43.108 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:56:43.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:43.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:56:43.209 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:43.234 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:43.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:43.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:43.336 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:43.375 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:43.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:43.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:43.398 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:43 smithi187 ceph-mon[33032]: pgmap v375: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 3.8 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.03k op/s; 23612/188352 objects degraded (12.536%); 7926/188352 objects misplaced (4.208%) 2024-09-21T13:56:43.398 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:43 smithi187 ceph-mon[33032]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:56:43.398 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:43 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T13:56:43.398 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:43 smithi187 ceph-mon[33032]: osdmap e230: 8 total, 7 up, 7 in 2024-09-21T13:56:43.398 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:43 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1039489246' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:56:43.477 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:43.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:43 smithi049 ceph-mon[40489]: pgmap v375: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 3.8 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.03k op/s; 23612/188352 objects degraded (12.536%); 7926/188352 objects misplaced (4.208%) 2024-09-21T13:56:43.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:43 smithi049 ceph-mon[40489]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:56:43.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:43 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T13:56:43.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:43 smithi049 ceph-mon[40489]: osdmap e230: 8 total, 7 up, 7 in 2024-09-21T13:56:43.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:43 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1039489246' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:56:43.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:43 smithi049 ceph-mon[30080]: pgmap v375: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 3.8 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.03k op/s; 23612/188352 objects degraded (12.536%); 7926/188352 objects misplaced (4.208%) 2024-09-21T13:56:43.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:43 smithi049 ceph-mon[30080]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:56:43.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:43 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T13:56:43.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:43 smithi049 ceph-mon[30080]: osdmap e230: 8 total, 7 up, 7 in 2024-09-21T13:56:43.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:43 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1039489246' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:56:43.518 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:43.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:43.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:43.621 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:43.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:43.658 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:43.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:43.712 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T13:56:43.712 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:56:43.759 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:56:43.792 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:43.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:43.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:56:43.894 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:56:43.926 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:43.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:43.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:56:44.028 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:56:44.057 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:44.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:44.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:56:44.158 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:44.191 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:44.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:44.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:44.293 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:44.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:44.338 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:44.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:44.439 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:44.477 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:44.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:44.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:44.578 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:44.607 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:44.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:44.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:44.708 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:56:44.741 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:44.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:44.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:56:44.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:44 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:56:44.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:44 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:56:44.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:44 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:56:44.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:44 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:56:44.842 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:56:44.875 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:44.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:44.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:56:44.910 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:44 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:56:44.910 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:44 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:56:44.954 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T13:56:44.976 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:56:45.030 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:45.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:45.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:56:45.132 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:56:45.153 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:45.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:45.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:56:45.254 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:56:45.289 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:45.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:45.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:56:45.297 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:56:45.391 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:56:45.425 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:45.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:45.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:56:45.527 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:45.567 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:45.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:45.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:45.669 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:56:45.695 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:45.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:45.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:56:45.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:45 smithi049 ceph-mon[30080]: pgmap v377: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfilling, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 3.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.67k op/s; 21247/170343 objects degraded (12.473%); 7106/170343 objects misplaced (4.172%); 1.9 MiB/s, 30 objects/s recovering 2024-09-21T13:56:45.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:45 smithi049 ceph-mon[30080]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-21T13:56:45.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:45 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 21247/170343 objects degraded (12.473%), 31 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-09-21T13:56:45.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:45 smithi049 ceph-mon[40489]: pgmap v377: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfilling, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 3.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.67k op/s; 21247/170343 objects degraded (12.473%); 7106/170343 objects misplaced (4.172%); 1.9 MiB/s, 30 objects/s recovering 2024-09-21T13:56:45.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:45 smithi049 ceph-mon[40489]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-21T13:56:45.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:45 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 21247/170343 objects degraded (12.473%), 31 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-09-21T13:56:45.765 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:45 smithi187 ceph-mon[33032]: pgmap v377: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfilling, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 3.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.67k op/s; 21247/170343 objects degraded (12.473%); 7106/170343 objects misplaced (4.172%); 1.9 MiB/s, 30 objects/s recovering 2024-09-21T13:56:45.766 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:45 smithi187 ceph-mon[33032]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-21T13:56:45.766 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:45 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 21247/170343 objects degraded (12.473%), 31 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-09-21T13:56:45.797 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:45.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:45.834 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:45.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:45.931 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T13:56:45.931 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:56:45.935 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:56:45.968 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:45.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:45.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:56:46.070 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:46.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:46.093 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:46.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:46.194 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:56:46.229 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:46.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:46.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:56:46.331 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:56:46.365 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:46.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:46.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:56:46.467 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:56:46.488 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:46.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:46.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:56:46.590 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:56:46.625 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:46.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:46.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:56:46.679 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T13:56:46.726 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:46.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:46.762 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:46.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:46.862 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:56:46.902 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:46.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:46.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:56:47.004 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:56:47.046 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:56:47.049 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:47.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:47.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:56:47.151 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:56:47.183 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:46 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:56:47.183 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:46 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:56:47.183 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:46 smithi187 ceph-mon[33032]: pgmap v378: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfilling, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 3.4 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.62k op/s; 20798/167655 objects degraded (12.405%); 7106/167655 objects misplaced (4.238%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T13:56:47.183 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:46 smithi187 ceph-mon[33032]: osdmap e231: 8 total, 7 up, 7 in 2024-09-21T13:56:47.190 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:47.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:47.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:56:47.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:46 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:56:47.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:46 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:56:47.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:46 smithi049 ceph-mon[40489]: pgmap v378: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfilling, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 3.4 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.62k op/s; 20798/167655 objects degraded (12.405%); 7106/167655 objects misplaced (4.238%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T13:56:47.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:46 smithi049 ceph-mon[40489]: osdmap e231: 8 total, 7 up, 7 in 2024-09-21T13:56:47.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:46 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:56:47.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:46 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:56:47.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:46 smithi049 ceph-mon[30080]: pgmap v378: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfilling, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 3.4 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.62k op/s; 20798/167655 objects degraded (12.405%); 7106/167655 objects misplaced (4.238%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T13:56:47.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:46 smithi049 ceph-mon[30080]: osdmap e231: 8 total, 7 up, 7 in 2024-09-21T13:56:47.293 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:47.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:47.325 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:47.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:47.426 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:56:47.463 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:47.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:47.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:56:47.565 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:56:47.606 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:47.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:47.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:56:47.634 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T13:56:47.634 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:56:47.708 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:56:47.740 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:47.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:47.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:56:47.841 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:47.864 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:47.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:47.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:47.966 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:48.006 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:48.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:48.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:48.108 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:56:48.129 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:48.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:48.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:56:48.231 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:56:48.267 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:48.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:48.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:56:48.314 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T13:56:48.371 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:56:48.393 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:48.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:48.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:56:48.495 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:48.517 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:48.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:48.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:48.580 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:56:48.620 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:48.642 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:48.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:48.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:48.744 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:56:48.776 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:48.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:48.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:56:48.878 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:48.905 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:48.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:48.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:49.007 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:49.042 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:49.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:49.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:49.143 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:49.165 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:49.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:49.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:49.169 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T13:56:49.169 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:56:49.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:48 smithi049 ceph-mon[30080]: pgmap v380: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfilling, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 3.4 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 511 B/s rd, 0 B/s wr, 862 op/s; 20798/167655 objects degraded (12.405%); 7106/167655 objects misplaced (4.238%); 1.9 MiB/s, 30 objects/s recovering 2024-09-21T13:56:49.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:48 smithi049 ceph-mon[40489]: pgmap v380: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfilling, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 3.4 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 511 B/s rd, 0 B/s wr, 862 op/s; 20798/167655 objects degraded (12.405%); 7106/167655 objects misplaced (4.238%); 1.9 MiB/s, 30 objects/s recovering 2024-09-21T13:56:49.266 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:56:49.301 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:49.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:49.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:56:49.388 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:48 smithi187 ceph-mon[33032]: pgmap v380: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfilling, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 3.4 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 511 B/s rd, 0 B/s wr, 862 op/s; 20798/167655 objects degraded (12.405%); 7106/167655 objects misplaced (4.238%); 1.9 MiB/s, 30 objects/s recovering 2024-09-21T13:56:49.403 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:49.433 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:49.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:49.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:49.535 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:56:49.570 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:49.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:49.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:56:49.659 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T13:56:49.672 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:56:49.694 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:56:49 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:56:49.702 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:49.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:49.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:56:49.804 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:49.839 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:49.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:49.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:49.940 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:56:49.943 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T13:56:49.960 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:56:49.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:49.966 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:49.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:56:50.002 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T13:56:50.003 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T13:56:50.067 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:56:50.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:50.105 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:50.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:56:50.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:49 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:56:50.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:49 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 20798/167655 objects degraded (12.405%), 31 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-09-21T13:56:50.206 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:50.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:50.244 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:50.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:50.257 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:49 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:56:50.257 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:49 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 20798/167655 objects degraded (12.405%), 31 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-09-21T13:56:50.257 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:49 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:56:50.257 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:49 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 20798/167655 objects degraded (12.405%), 31 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-09-21T13:56:50.333 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:56:50.345 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:50.392 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:50.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:50.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:50.494 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:56:50.519 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:50.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:50.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:56:50.621 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:50.629 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T13:56:50.630 INFO:teuthology.orchestra.run.smithi049.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-21T13:56:50.630 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T13:56:50.633 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:56:50.653 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:50.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:50.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:50.755 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:56:50.783 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:50.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:50.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:56:50.885 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:50.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:50.924 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:50.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:50.939 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T13:56:50.940 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:56:50.963 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:56:50 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:56:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:56:51.026 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:56:51.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:51.057 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:51.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:56:51.157 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:56:51.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:50 smithi187 ceph-mon[33032]: pgmap v381: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfilling, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 3.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.58k op/s; 18097/150402 objects degraded (12.032%); 6413/150402 objects misplaced (4.264%); 6.3 MiB/s, 101 objects/s recovering 2024-09-21T13:56:51.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:51.195 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:51.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:56:51.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:50 smithi049 ceph-mon[40489]: pgmap v381: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfilling, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 3.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.58k op/s; 18097/150402 objects degraded (12.032%); 6413/150402 objects misplaced (4.264%); 6.3 MiB/s, 101 objects/s recovering 2024-09-21T13:56:51.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:50 smithi049 ceph-mon[30080]: pgmap v381: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfilling, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 3.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.58k op/s; 18097/150402 objects degraded (12.032%); 6413/150402 objects misplaced (4.264%); 6.3 MiB/s, 101 objects/s recovering 2024-09-21T13:56:51.297 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:51.334 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:51.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:51.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:51.436 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:56:51.467 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:51.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:51.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:56:51.568 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:56:51.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:51.605 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:51.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:56:51.683 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T13:56:51.707 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:51.746 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:51.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:51.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:51.848 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:56:51.876 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:51.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:51.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:56:51.978 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:56:52.004 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T13:56:52.013 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:52.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:52.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:56:52.056 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:56:52.116 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:56:52.147 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:52.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:52.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:56:52.248 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:52.274 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:52.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:52.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:52.376 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:52.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:52.421 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:52.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:52.459 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:56:52.522 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:52.557 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:52.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:52.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:52.658 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:56:52.666 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T13:56:52.667 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:56:52.691 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:52.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:52.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:56:52.793 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:52.834 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:52.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:52.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:52.935 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:52.967 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:52.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:52.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:53.016 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-09-21T13:56:53.069 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:53.108 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:53.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:53.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:53.142 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:56:53.142 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:56:53.178 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:56:53.178 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:56:53.211 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:53.217 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:56:53.217 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:56:53.217 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:53 smithi049 ceph-mon[40489]: pgmap v382: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfilling, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 3.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 946 B/s rd, 0 B/s wr, 1.30k op/s; 18097/150402 objects degraded (12.032%); 6413/150402 objects misplaced (4.264%); 5.2 MiB/s, 83 objects/s recovering 2024-09-21T13:56:53.217 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:53 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1717298015' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:56:53.218 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:53 smithi049 ceph-mon[30080]: pgmap v382: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfilling, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 3.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 946 B/s rd, 0 B/s wr, 1.30k op/s; 18097/150402 objects degraded (12.032%); 6413/150402 objects misplaced (4.264%); 5.2 MiB/s, 83 objects/s recovering 2024-09-21T13:56:53.218 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:53 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1717298015' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:56:53.220 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-21T13:56:53.252 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:53.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:53.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:53.256 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:56:53.256 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:56:53.295 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:56:53.296 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:56:53.331 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:56:53.331 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:56:53.354 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:56:53.363 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:53 smithi187 ceph-mon[33032]: pgmap v382: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfilling, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 3.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 946 B/s rd, 0 B/s wr, 1.30k op/s; 18097/150402 objects degraded (12.032%); 6413/150402 objects misplaced (4.264%); 5.2 MiB/s, 83 objects/s recovering 2024-09-21T13:56:53.363 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:53 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1717298015' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:56:53.366 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:56:53.367 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:56:53.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:53.394 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:53.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:56:53.411 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T13:56:53.495 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:56:53.550 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:53.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:53.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:56:53.652 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:53.687 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:53.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:53.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:53.789 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:56:53.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:53.839 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:53.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:56:53.863 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:56:53.940 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:56:53.975 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:53.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:53.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:56:54.077 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:56:54.108 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:54.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:54.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:56:54.209 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:56:54.230 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:54.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:54.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:56:54.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:54 smithi049 ceph-mon[40489]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:56:54.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:54 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1717298015' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T13:56:54.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:54 smithi049 ceph-mon[40489]: osdmap e232: 8 total, 7 up, 7 in 2024-09-21T13:56:54.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:54 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1153391062' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:56:54.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:54 smithi049 ceph-mon[30080]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:56:54.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:54 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1717298015' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T13:56:54.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:54 smithi049 ceph-mon[30080]: osdmap e232: 8 total, 7 up, 7 in 2024-09-21T13:56:54.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:54 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1153391062' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:56:54.331 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:54.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:54.366 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:54.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:54.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:54 smithi187 ceph-mon[33032]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:56:54.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:54 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1717298015' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T13:56:54.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:54 smithi187 ceph-mon[33032]: osdmap e232: 8 total, 7 up, 7 in 2024-09-21T13:56:54.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:54 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1153391062' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:56:54.468 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:54.497 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:54.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:54.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:54.599 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:54.638 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:54.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:54.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:54.740 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:54.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:54.768 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:54.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:54.869 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:56:54.895 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:54.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:54.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:56:54.996 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:55.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:55.035 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:55.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:55.136 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:55.170 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:55.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:55.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:55.272 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:55.306 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:55.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:55.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:55.408 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:55.443 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:55.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:55.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:55.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:55 smithi187 ceph-mon[33032]: 3.9 scrub starts 2024-09-21T13:56:55.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:55 smithi187 ceph-mon[33032]: 3.9 scrub ok 2024-09-21T13:56:55.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:55 smithi187 ceph-mon[33032]: pgmap v384: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfilling, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 2.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.45k op/s; 15510/132846 objects degraded (11.675%); 5556/132846 objects misplaced (4.182%); 9.3 MiB/s, 149 objects/s recovering 2024-09-21T13:56:55.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:55 smithi187 ceph-mon[33032]: 3.1c scrub starts 2024-09-21T13:56:55.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:55 smithi187 ceph-mon[33032]: 3.1c scrub ok 2024-09-21T13:56:55.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:55 smithi187 ceph-mon[33032]: 8.6 scrub starts 2024-09-21T13:56:55.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:55 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 15510/132846 objects degraded (11.675%), 31 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-09-21T13:56:55.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:55 smithi049 ceph-mon[30080]: 3.9 scrub starts 2024-09-21T13:56:55.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:55 smithi049 ceph-mon[30080]: 3.9 scrub ok 2024-09-21T13:56:55.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:55 smithi049 ceph-mon[30080]: pgmap v384: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfilling, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 2.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.45k op/s; 15510/132846 objects degraded (11.675%); 5556/132846 objects misplaced (4.182%); 9.3 MiB/s, 149 objects/s recovering 2024-09-21T13:56:55.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:55 smithi049 ceph-mon[30080]: 3.1c scrub starts 2024-09-21T13:56:55.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:55 smithi049 ceph-mon[30080]: 3.1c scrub ok 2024-09-21T13:56:55.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:55 smithi049 ceph-mon[30080]: 8.6 scrub starts 2024-09-21T13:56:55.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:55 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 15510/132846 objects degraded (11.675%), 31 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-09-21T13:56:55.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:55 smithi049 ceph-mon[40489]: 3.9 scrub starts 2024-09-21T13:56:55.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:55 smithi049 ceph-mon[40489]: 3.9 scrub ok 2024-09-21T13:56:55.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:55 smithi049 ceph-mon[40489]: pgmap v384: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfilling, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 2.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.45k op/s; 15510/132846 objects degraded (11.675%); 5556/132846 objects misplaced (4.182%); 9.3 MiB/s, 149 objects/s recovering 2024-09-21T13:56:55.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:55 smithi049 ceph-mon[40489]: 3.1c scrub starts 2024-09-21T13:56:55.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:55 smithi049 ceph-mon[40489]: 3.1c scrub ok 2024-09-21T13:56:55.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:55 smithi049 ceph-mon[40489]: 8.6 scrub starts 2024-09-21T13:56:55.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:55 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 15510/132846 objects degraded (11.675%), 31 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-09-21T13:56:55.545 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:55.570 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:55.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:55.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:55.672 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:56:55.693 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:55.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:55.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:56:55.794 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:55.816 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:55.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:55.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:55.918 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:55.939 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T13:56:55.952 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:55.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:55.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:56.054 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:56:56.075 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:56.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:56.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:56:56.177 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:56.217 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:56.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:56.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:56.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:56 smithi049 ceph-mon[40489]: 4.3 scrub starts 2024-09-21T13:56:56.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:56 smithi049 ceph-mon[40489]: 4.3 scrub ok 2024-09-21T13:56:56.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:56 smithi049 ceph-mon[40489]: 8.6 scrub ok 2024-09-21T13:56:56.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:56 smithi049 ceph-mon[30080]: 4.3 scrub starts 2024-09-21T13:56:56.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:56 smithi049 ceph-mon[30080]: 4.3 scrub ok 2024-09-21T13:56:56.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:56 smithi049 ceph-mon[30080]: 8.6 scrub ok 2024-09-21T13:56:56.294 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:56 smithi187 ceph-mon[33032]: 4.3 scrub starts 2024-09-21T13:56:56.294 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:56 smithi187 ceph-mon[33032]: 4.3 scrub ok 2024-09-21T13:56:56.294 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:56 smithi187 ceph-mon[33032]: 8.6 scrub ok 2024-09-21T13:56:56.320 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:56:56.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:56.357 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:56.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:56:56.400 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [3] live_osds: [0, 1, 4, 5, 6, 7, 2] 2024-09-21T13:56:56.400 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T13:56:56.400 INFO:tasks.thrashosds.thrasher:Reviving osd 3 2024-09-21T13:56:56.400 INFO:tasks.cephadm.osd.3:Restarting osd.3 (starting--it wasn't running)... 2024-09-21T13:56:56.400 DEBUG:teuthology.orchestra.run.smithi049:osd.3> sudo journalctl -f -n 0 -u ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@osd.3.service 2024-09-21T13:56:56.403 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl start ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@osd.3 2024-09-21T13:56:56.457 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:56.486 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T13:56:56.486 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T13:56:56.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:56.584 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:56.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:56.609 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:56:56.686 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:56.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:56.710 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:56.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:56.746 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 13:56:56 smithi049 systemd[1]: Starting Ceph osd.3 for 40a3d2e8-781f-11ef-baf6-efdc52797490... 2024-09-21T13:56:56.811 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:56.851 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:56.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:56.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:56.953 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:56.983 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:56.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:56.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:57.085 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:56:57.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:57.130 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:57.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:56:57.231 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:56:57.274 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:57.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:57.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:56:57.321 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:57 smithi049 ceph-mon[40489]: 4.d scrub starts 2024-09-21T13:56:57.321 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:57 smithi049 ceph-mon[40489]: 4.d scrub ok 2024-09-21T13:56:57.322 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:57 smithi049 ceph-mon[40489]: pgmap v385: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfilling, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.26k op/s; 15104/130236 objects degraded (11.597%); 5556/130236 objects misplaced (4.266%); 7.6 MiB/s, 120 objects/s recovering 2024-09-21T13:56:57.322 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:57 smithi049 ceph-mon[30080]: 4.d scrub starts 2024-09-21T13:56:57.322 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:57 smithi049 ceph-mon[30080]: 4.d scrub ok 2024-09-21T13:56:57.322 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:57 smithi049 ceph-mon[30080]: pgmap v385: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfilling, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.26k op/s; 15104/130236 objects degraded (11.597%); 5556/130236 objects misplaced (4.266%); 7.6 MiB/s, 120 objects/s recovering 2024-09-21T13:56:57.377 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:56:57.406 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:57.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:57.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:56:57.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:57 smithi187 ceph-mon[33032]: 4.d scrub starts 2024-09-21T13:56:57.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:57 smithi187 ceph-mon[33032]: 4.d scrub ok 2024-09-21T13:56:57.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:57 smithi187 ceph-mon[33032]: pgmap v385: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfilling, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.26k op/s; 15104/130236 objects degraded (11.597%); 5556/130236 objects misplaced (4.266%); 7.6 MiB/s, 120 objects/s recovering 2024-09-21T13:56:57.508 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:57.553 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:57.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:57.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:57.655 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:56:57.665 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 13:56:57 smithi049 podman[128795]: 2024-09-21 13:56:57.228740005 +0000 UTC m=+0.265216515 image pull b415816db8b41f544cdfdc71d193ff8b1ee008bf7b0ec4f64da8eb8f71b62e85 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5 2024-09-21T13:56:57.665 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 13:56:57 smithi049 podman[128795]: 2024-09-21 13:56:57.358096156 +0000 UTC m=+0.394572665 container create b8470d59e297f480487e40c8f9b0d7adcc9a87eb3a2bfc12f48adcefb1237169 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-3-activate, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-21-0842-4417849, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-09-21T13:56:57.665 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 13:56:57 smithi049 podman[128795]: 2024-09-21 13:56:57.66512817 +0000 UTC m=+0.701604696 container init b8470d59e297f480487e40c8f9b0d7adcc9a87eb3a2bfc12f48adcefb1237169 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-3-activate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-21-0842-4417849, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T13:56:57.690 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:57.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:57.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:56:57.792 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:57.819 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:57.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:57.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:57.920 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:56:57.925 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 13:56:57 smithi049 podman[128795]: 2024-09-21 13:56:57.682784923 +0000 UTC m=+0.719261433 container start b8470d59e297f480487e40c8f9b0d7adcc9a87eb3a2bfc12f48adcefb1237169 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-3-activate, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T13:56:57.925 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 13:56:57 smithi049 podman[128795]: 2024-09-21 13:56:57.781652619 +0000 UTC m=+0.818129129 container attach b8470d59e297f480487e40c8f9b0d7adcc9a87eb3a2bfc12f48adcefb1237169 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-3-activate, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-21T13:56:57.925 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 13:56:57 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-3-activate[128867]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T13:56:57.925 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 13:56:57 smithi049 bash[128795]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T13:56:57.925 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 13:56:57 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-3-activate[128867]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T13:56:57.925 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 13:56:57 smithi049 bash[128795]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T13:56:57.953 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:57.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:57.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:56:58.048 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-09-21T13:56:58.054 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:58.093 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:58.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:58.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:58.127 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T13:56:58.128 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-21T13:56:58.128 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-21T13:56:58.128 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-21T13:56:58.128 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T13:56:58.195 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:56:58.240 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:58.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:58.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:56:58.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:58 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2678091291' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:56:58.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:58 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2678091291' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:56:58.342 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:56:58.372 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:58.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:58.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:56:58.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:58 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2678091291' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:56:58.473 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:56:58.487 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T13:56:58.499 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:58.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:58.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:56:58.601 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:58.654 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:58.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:58.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:58.757 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:56:58.782 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:58.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:58.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:56:58.883 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T13:56:58.884 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:56:58.937 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:58.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:58.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:56:58.982 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:56:59.039 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:59.076 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:59.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:59.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:56:59.118 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:59 smithi049 ceph-mon[30080]: pgmap v386: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfilling, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s; 15104/130236 objects degraded (11.597%); 5556/130236 objects misplaced (4.266%); 7.5 MiB/s, 119 objects/s recovering 2024-09-21T13:56:59.118 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:59 smithi049 ceph-mon[30080]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T13:56:59.118 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:59 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2678091291' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T13:56:59.118 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:59 smithi049 ceph-mon[30080]: osdmap e233: 8 total, 7 up, 7 in 2024-09-21T13:56:59.118 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:59 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:56:59.118 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:56:59 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:56:59.118 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:59 smithi049 ceph-mon[40489]: pgmap v386: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfilling, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s; 15104/130236 objects degraded (11.597%); 5556/130236 objects misplaced (4.266%); 7.5 MiB/s, 119 objects/s recovering 2024-09-21T13:56:59.119 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:59 smithi049 ceph-mon[40489]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T13:56:59.119 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:59 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2678091291' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T13:56:59.119 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:59 smithi049 ceph-mon[40489]: osdmap e233: 8 total, 7 up, 7 in 2024-09-21T13:56:59.119 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:59 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:56:59.119 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:56:59 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:56:59.178 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:56:59.197 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:59 smithi187 ceph-mon[33032]: pgmap v386: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfilling, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s; 15104/130236 objects degraded (11.597%); 5556/130236 objects misplaced (4.266%); 7.5 MiB/s, 119 objects/s recovering 2024-09-21T13:56:59.197 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:59 smithi187 ceph-mon[33032]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T13:56:59.197 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:59 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2678091291' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T13:56:59.197 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:59 smithi187 ceph-mon[33032]: osdmap e233: 8 total, 7 up, 7 in 2024-09-21T13:56:59.197 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:59 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:56:59.197 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:56:59 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:56:59.217 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:59.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:59.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:56:59.268 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:56:59.320 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:56:59.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:59.351 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:59.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:56:59.452 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:56:59.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:59.484 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:59.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:56:59.585 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:56:59.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:59.624 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:59.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:56:59.655 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:56:59.656 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:56:59.679 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:56:59 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:56:59.695 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:56:59.695 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:56:59.725 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:56:59.732 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:56:59.732 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:56:59.735 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-21T13:56:59.757 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:56:59.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:59.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:56:59.768 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:56:59.768 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:56:59.802 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:56:59.802 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:56:59.832 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T13:56:59.832 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-21T13:56:59.832 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-21T13:56:59.832 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-21T13:56:59.832 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.832 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:36843 7.d 7:b9f54a05:::benchmark_data_smithi049_76798_object36842:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-21T13:56:59.832 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:30.623325+0000", 2024-09-21T13:56:59.832 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 389.20661797499997, 2024-09-21T13:56:59.832 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.14164070300000001, 2024-09-21T13:56:59.832 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:56:59.833 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:56:59.833 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:59.833 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:56:59.833 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:56:59.833 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:56:59.833 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 36843 2024-09-21T13:56:59.833 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.833 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:56:59.833 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.833 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:56:59.833 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.623325+0000", 2024-09-21T13:56:59.833 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:56:59.833 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.833 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.834 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:56:59.834 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.623325+0000", 2024-09-21T13:56:59.834 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:56:59.834 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.834 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.834 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:56:59.834 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.623328+0000", 2024-09-21T13:56:59.834 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8120000000000001e-06 2024-09-21T13:56:59.834 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.834 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.834 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:56:59.834 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.623434+0000", 2024-09-21T13:56:59.834 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010542799999999999 2024-09-21T13:56:59.835 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.835 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.835 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:56:59.835 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.623443+0000", 2024-09-21T13:56:59.835 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1200000000000008e-06 2024-09-21T13:56:59.835 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.835 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.835 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:56:59.835 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.623448+0000", 2024-09-21T13:56:59.835 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2079999999999999e-06 2024-09-21T13:56:59.835 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.835 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.835 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:56:59.836 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.623483+0000", 2024-09-21T13:56:59.836 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5048000000000001e-05 2024-09-21T13:56:59.836 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.836 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.836 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:56:59.836 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.623592+0000", 2024-09-21T13:56:59.836 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000108585 2024-09-21T13:56:59.836 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.836 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.836 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-21T13:56:59.836 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.623673+0000", 2024-09-21T13:56:59.836 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1079000000000001e-05 2024-09-21T13:56:59.836 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.836 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.837 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:56:59.837 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.624761+0000", 2024-09-21T13:56:59.837 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001088449 2024-09-21T13:56:59.837 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.837 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.837 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:59.837 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.625971+0000", 2024-09-21T13:56:59.837 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0012098949999999999 2024-09-21T13:56:59.837 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.837 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.837 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:59.837 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.764850+0000", 2024-09-21T13:56:59.837 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13887893000000001 2024-09-21T13:56:59.837 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.838 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.838 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:56:59.838 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.764907+0000", 2024-09-21T13:56:59.838 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6901999999999998e-05 2024-09-21T13:56:59.838 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.838 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.838 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:56:59.838 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.764965+0000", 2024-09-21T13:56:59.838 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8247e-05 2024-09-21T13:56:59.838 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:56:59.839 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:56:59.839 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:56:59.839 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.839 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.839 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:36844 7.c 7:3245d1c7:::benchmark_data_smithi049_76798_object36843:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-21T13:56:59.839 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:30.625096+0000", 2024-09-21T13:56:59.839 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 389.20484629200001, 2024-09-21T13:56:59.839 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.140563562, 2024-09-21T13:56:59.839 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:56:59.839 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:56:59.839 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:59.839 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:56:59.839 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:56:59.839 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:56:59.840 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 36844 2024-09-21T13:56:59.840 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.840 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:56:59.840 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.840 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:56:59.840 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.625096+0000", 2024-09-21T13:56:59.840 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:56:59.840 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.840 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.840 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:56:59.840 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.625096+0000", 2024-09-21T13:56:59.840 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:56:59.840 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.840 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.840 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:56:59.841 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.625097+0000", 2024-09-21T13:56:59.841 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.116e-06 2024-09-21T13:56:59.841 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.841 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.841 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:56:59.841 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.625152+0000", 2024-09-21T13:56:59.841 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4682000000000003e-05 2024-09-21T13:56:59.841 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.841 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.841 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:56:59.841 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.625161+0000", 2024-09-21T13:56:59.841 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1670000000000005e-06 2024-09-21T13:56:59.841 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.841 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.841 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:56:59.842 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.625168+0000", 2024-09-21T13:56:59.842 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7880000000000001e-06 2024-09-21T13:56:59.842 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.842 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.842 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:56:59.842 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.625207+0000", 2024-09-21T13:56:59.842 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9251000000000001e-05 2024-09-21T13:56:59.842 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.842 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.842 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:56:59.842 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.625278+0000", 2024-09-21T13:56:59.842 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.0659000000000002e-05 2024-09-21T13:56:59.842 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.842 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.843 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,7", 2024-09-21T13:56:59.843 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.625357+0000", 2024-09-21T13:56:59.843 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9015000000000002e-05 2024-09-21T13:56:59.843 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.843 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.843 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:56:59.843 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.626576+0000", 2024-09-21T13:56:59.843 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001219363 2024-09-21T13:56:59.843 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.843 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.843 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:59.843 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.626862+0000", 2024-09-21T13:56:59.843 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00028596300000000003 2024-09-21T13:56:59.843 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.843 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.844 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:59.844 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.765560+0000", 2024-09-21T13:56:59.844 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.138697767 2024-09-21T13:56:59.844 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.844 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.845 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:56:59.845 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.765620+0000", 2024-09-21T13:56:59.845 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0099999999999997e-05 2024-09-21T13:56:59.845 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.845 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.845 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:56:59.845 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.765660+0000", 2024-09-21T13:56:59.845 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9691000000000003e-05 2024-09-21T13:56:59.845 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:56:59.845 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:56:59.845 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:56:59.845 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.845 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.845 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:36848 7.7 7:e1dc98c1:::benchmark_data_smithi049_76798_object36847:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-21T13:56:59.846 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:30.625310+0000", 2024-09-21T13:56:59.846 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 389.20463209000002, 2024-09-21T13:56:59.846 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.141300645, 2024-09-21T13:56:59.846 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:56:59.846 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:56:59.846 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:59.847 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:56:59.847 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:56:59.847 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:56:59.847 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:56:59.847 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:56:59.847 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 36848 2024-09-21T13:56:59.848 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.848 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:56:59.848 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.848 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:56:59.848 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.625310+0000", 2024-09-21T13:56:59.848 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:56:59.848 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.848 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.848 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:56:59.848 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.625310+0000", 2024-09-21T13:56:59.848 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:56:59.848 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.848 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.849 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:56:59.849 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.625311+0000", 2024-09-21T13:56:59.849 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1399999999999996e-07 2024-09-21T13:56:59.849 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.849 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.849 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:56:59.849 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.625453+0000", 2024-09-21T13:56:59.849 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014165200000000001 2024-09-21T13:56:59.849 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.849 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.849 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:56:59.849 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.625461+0000", 2024-09-21T13:56:59.849 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7579999999999992e-06 2024-09-21T13:56:59.849 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.850 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.850 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:56:59.850 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.625465+0000", 2024-09-21T13:56:59.850 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3989999999999997e-06 2024-09-21T13:56:59.850 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.850 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.850 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:56:59.850 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.625514+0000", 2024-09-21T13:56:59.850 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8742000000000002e-05 2024-09-21T13:56:59.850 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.850 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.850 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:56:59.850 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.625598+0000", 2024-09-21T13:56:59.850 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4091999999999994e-05 2024-09-21T13:56:59.850 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.851 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.851 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3", 2024-09-21T13:56:59.851 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.625671+0000", 2024-09-21T13:56:59.851 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3566999999999995e-05 2024-09-21T13:56:59.851 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.851 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.851 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:56:59.851 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.626848+0000", 2024-09-21T13:56:59.851 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0011770109999999999 2024-09-21T13:56:59.851 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.851 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.851 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:59.851 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.766471+0000", 2024-09-21T13:56:59.851 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13962287100000001 2024-09-21T13:56:59.851 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.852 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.852 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:59.852 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.766504+0000", 2024-09-21T13:56:59.852 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2790000000000003e-05 2024-09-21T13:56:59.852 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.852 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.852 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:56:59.852 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.766575+0000", 2024-09-21T13:56:59.852 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1348999999999997e-05 2024-09-21T13:56:59.852 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.852 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.852 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:56:59.852 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.766611+0000", 2024-09-21T13:56:59.852 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.57e-05 2024-09-21T13:56:59.852 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:56:59.853 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:56:59.853 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:56:59.853 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.853 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.853 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:36852 7.d 7:bc07fd14:::benchmark_data_smithi049_76798_object36851:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-21T13:56:59.853 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:30.628155+0000", 2024-09-21T13:56:59.853 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 389.201787104, 2024-09-21T13:56:59.853 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.136985101, 2024-09-21T13:56:59.853 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:56:59.853 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:56:59.853 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:59.853 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:56:59.853 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:56:59.853 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:56:59.853 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 36852 2024-09-21T13:56:59.854 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.854 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:56:59.854 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.854 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:56:59.854 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.628155+0000", 2024-09-21T13:56:59.854 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:56:59.854 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.854 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.854 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:56:59.854 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.628155+0000", 2024-09-21T13:56:59.854 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:56:59.854 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.854 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.854 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:56:59.855 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.628156+0000", 2024-09-21T13:56:59.855 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6299999999999993e-07 2024-09-21T13:56:59.855 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.855 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.855 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:56:59.855 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.628210+0000", 2024-09-21T13:56:59.855 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3278e-05 2024-09-21T13:56:59.855 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.855 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.855 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:56:59.855 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.628219+0000", 2024-09-21T13:56:59.855 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.4660000000000005e-06 2024-09-21T13:56:59.855 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.855 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.856 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:56:59.856 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.628226+0000", 2024-09-21T13:56:59.856 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9700000000000002e-06 2024-09-21T13:56:59.856 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.856 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.856 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:56:59.856 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.628275+0000", 2024-09-21T13:56:59.856 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9143e-05 2024-09-21T13:56:59.856 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.856 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.856 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:56:59.856 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.628380+0000", 2024-09-21T13:56:59.856 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010433900000000001 2024-09-21T13:56:59.856 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.856 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.856 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-21T13:56:59.857 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.628495+0000", 2024-09-21T13:56:59.857 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000115246 2024-09-21T13:56:59.857 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.857 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.857 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:56:59.857 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.764977+0000", 2024-09-21T13:56:59.857 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13648207500000001 2024-09-21T13:56:59.857 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.857 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.857 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:59.857 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.765013+0000", 2024-09-21T13:56:59.857 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6074000000000001e-05 2024-09-21T13:56:59.857 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.857 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.858 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:59.858 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.765047+0000", 2024-09-21T13:56:59.858 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3880000000000001e-05 2024-09-21T13:56:59.858 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.858 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.858 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:56:59.858 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.765108+0000", 2024-09-21T13:56:59.858 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1008000000000003e-05 2024-09-21T13:56:59.858 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.858 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.858 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:56:59.858 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.765141+0000", 2024-09-21T13:56:59.858 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2659e-05 2024-09-21T13:56:59.858 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:56:59.859 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:56:59.859 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:56:59.859 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.859 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.859 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:36854 7.7 7:eea57e62:::benchmark_data_smithi049_76798_object36853:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-21T13:56:59.859 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:30.628335+0000", 2024-09-21T13:56:59.859 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 389.20160740900002, 2024-09-21T13:56:59.859 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.140131803, 2024-09-21T13:56:59.859 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:56:59.859 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:56:59.859 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:59.859 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:56:59.859 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:56:59.859 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:56:59.860 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 36854 2024-09-21T13:56:59.860 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.860 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:56:59.860 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.860 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:56:59.860 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.628335+0000", 2024-09-21T13:56:59.860 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:56:59.860 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.860 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.860 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:56:59.860 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.628335+0000", 2024-09-21T13:56:59.860 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:56:59.860 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.860 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.860 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:56:59.861 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.628336+0000", 2024-09-21T13:56:59.861 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8899999999999999e-07 2024-09-21T13:56:59.861 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.861 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.861 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:56:59.861 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.628403+0000", 2024-09-21T13:56:59.861 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6880999999999999e-05 2024-09-21T13:56:59.861 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.861 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.861 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:56:59.861 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.628409+0000", 2024-09-21T13:56:59.861 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9329999999999998e-06 2024-09-21T13:56:59.861 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.861 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.862 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:56:59.862 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.628413+0000", 2024-09-21T13:56:59.862 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0860000000000004e-06 2024-09-21T13:56:59.862 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.862 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.862 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:56:59.862 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.628458+0000", 2024-09-21T13:56:59.862 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5111000000000002e-05 2024-09-21T13:56:59.862 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.862 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.862 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:56:59.862 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.628571+0000", 2024-09-21T13:56:59.862 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000113565 2024-09-21T13:56:59.862 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.863 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.863 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3", 2024-09-21T13:56:59.863 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.628701+0000", 2024-09-21T13:56:59.863 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012919400000000001 2024-09-21T13:56:59.863 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.863 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.863 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:59.863 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.766702+0000", 2024-09-21T13:56:59.863 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13800140499999999 2024-09-21T13:56:59.863 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.863 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.863 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:56:59.863 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.766725+0000", 2024-09-21T13:56:59.863 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2986999999999999e-05 2024-09-21T13:56:59.864 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.864 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.864 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:59.864 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.768379+0000", 2024-09-21T13:56:59.864 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0016544870000000001 2024-09-21T13:56:59.864 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.864 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.864 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:56:59.864 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.768432+0000", 2024-09-21T13:56:59.864 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2771e-05 2024-09-21T13:56:59.864 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.864 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.864 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:56:59.864 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.768467+0000", 2024-09-21T13:56:59.865 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4693999999999997e-05 2024-09-21T13:56:59.865 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:56:59.865 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:56:59.865 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:56:59.865 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.865 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.865 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.14817.0:36851 7.b e96/83 7:d46d3307:::benchmark_data_smithi049_76798_object36850:head v 96'2302, mlcod=96'2299)", 2024-09-21T13:56:59.865 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:30.629763+0000", 2024-09-21T13:56:59.865 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 389.20017958800003, 2024-09-21T13:56:59.865 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13851205, 2024-09-21T13:56:59.865 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:56:59.865 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:56:59.865 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:59.865 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:56:59.865 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.866 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:56:59.866 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.629763+0000", 2024-09-21T13:56:59.866 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:56:59.866 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.866 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.866 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:56:59.866 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.629763+0000", 2024-09-21T13:56:59.866 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:56:59.866 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.866 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.866 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:56:59.866 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.629764+0000", 2024-09-21T13:56:59.866 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6899999999999996e-07 2024-09-21T13:56:59.866 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.867 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.867 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:56:59.867 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.629814+0000", 2024-09-21T13:56:59.867 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9891000000000001e-05 2024-09-21T13:56:59.867 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.867 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.867 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:56:59.867 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.764759+0000", 2024-09-21T13:56:59.867 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13494542600000001 2024-09-21T13:56:59.867 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.867 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.867 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:56:59.867 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.764771+0000", 2024-09-21T13:56:59.867 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1764e-05 2024-09-21T13:56:59.868 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.868 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.868 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:56:59.868 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.764824+0000", 2024-09-21T13:56:59.868 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3378999999999998e-05 2024-09-21T13:56:59.868 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.868 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.868 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:56:59.868 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.764858+0000", 2024-09-21T13:56:59.868 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3714999999999998e-05 2024-09-21T13:56:59.868 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.868 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.868 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:56:59.868 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.768222+0000", 2024-09-21T13:56:59.869 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0033640749999999998 2024-09-21T13:56:59.869 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.869 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.869 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:56:59.869 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.768275+0000", 2024-09-21T13:56:59.869 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2831000000000003e-05 2024-09-21T13:56:59.869 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:56:59.869 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:56:59.869 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:56:59.869 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.869 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.869 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.14817.0:36848 7.7 e96/83 ondisk, result = 0)", 2024-09-21T13:56:59.869 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:30.630905+0000", 2024-09-21T13:56:59.869 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 389.19903718400002, 2024-09-21T13:56:59.869 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13557181099999999, 2024-09-21T13:56:59.870 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:56:59.870 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:56:59.870 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T13:56:59.870 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:56:59.870 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.870 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:56:59.870 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.630905+0000", 2024-09-21T13:56:59.870 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:56:59.870 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.870 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.870 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:56:59.870 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.630905+0000", 2024-09-21T13:56:59.870 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:56:59.871 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.871 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.871 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:56:59.871 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.630906+0000", 2024-09-21T13:56:59.871 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7000000000000004e-07 2024-09-21T13:56:59.871 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.871 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.871 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:56:59.871 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.630910+0000", 2024-09-21T13:56:59.871 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0090000000000001e-06 2024-09-21T13:56:59.871 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.871 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.871 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:56:59.871 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.764755+0000", 2024-09-21T13:56:59.872 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.133845082 2024-09-21T13:56:59.872 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.872 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.872 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:56:59.872 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.764768+0000", 2024-09-21T13:56:59.872 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2944e-05 2024-09-21T13:56:59.872 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.872 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.872 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:56:59.872 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.766460+0000", 2024-09-21T13:56:59.872 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0016917239999999999 2024-09-21T13:56:59.872 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.872 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.872 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:56:59.873 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.766466+0000", 2024-09-21T13:56:59.873 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3570000000000003e-06 2024-09-21T13:56:59.873 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.873 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.873 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:56:59.873 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.766477+0000", 2024-09-21T13:56:59.873 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0925e-05 2024-09-21T13:56:59.873 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:56:59.873 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:56:59.873 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:56:59.873 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.873 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.873 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.14817.0:36843 7.d e96/83 ondisk, result = 0)", 2024-09-21T13:56:59.873 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:30.631991+0000", 2024-09-21T13:56:59.873 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 389.19795128700002, 2024-09-21T13:56:59.874 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13298670100000001, 2024-09-21T13:56:59.874 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:56:59.874 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:56:59.874 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T13:56:59.874 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:56:59.874 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.874 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:56:59.874 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.631991+0000", 2024-09-21T13:56:59.874 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:56:59.874 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.874 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.874 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:56:59.874 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.631991+0000", 2024-09-21T13:56:59.874 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:56:59.875 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.875 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.875 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:56:59.875 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.631992+0000", 2024-09-21T13:56:59.875 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.0299999999999998e-07 2024-09-21T13:56:59.875 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.875 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.875 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:56:59.875 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.631995+0000", 2024-09-21T13:56:59.875 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3330000000000001e-06 2024-09-21T13:56:59.875 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.875 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.875 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:56:59.875 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.764776+0000", 2024-09-21T13:56:59.876 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13278029999999999 2024-09-21T13:56:59.876 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.876 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.876 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:56:59.876 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.764785+0000", 2024-09-21T13:56:59.876 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0710000000000001e-06 2024-09-21T13:56:59.876 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.876 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.876 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:56:59.876 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.764826+0000", 2024-09-21T13:56:59.876 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0859000000000003e-05 2024-09-21T13:56:59.876 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.876 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.876 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:56:59.876 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.764834+0000", 2024-09-21T13:56:59.877 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.439e-06 2024-09-21T13:56:59.877 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.877 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.877 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:56:59.877 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.764978+0000", 2024-09-21T13:56:59.877 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014399599999999999 2024-09-21T13:56:59.877 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:56:59.877 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:56:59.877 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:56:59.877 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.877 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.877 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:49358 7.d 7:bafe9a80:::benchmark_data_smithi049_76798_object49357:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T13:56:59.877 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:38.254356+0000", 2024-09-21T13:56:59.877 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 381.57558652300003, 2024-09-21T13:56:59.877 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.153550043, 2024-09-21T13:56:59.878 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:56:59.878 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:56:59.878 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:59.878 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:56:59.878 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:56:59.878 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:56:59.878 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 49358 2024-09-21T13:56:59.878 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.878 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:56:59.878 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.878 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:56:59.878 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.254356+0000", 2024-09-21T13:56:59.878 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:56:59.878 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.879 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.879 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:56:59.879 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.254356+0000", 2024-09-21T13:56:59.879 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:56:59.879 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.879 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.879 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:56:59.879 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.254357+0000", 2024-09-21T13:56:59.879 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4600000000000004e-07 2024-09-21T13:56:59.879 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.879 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.879 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:56:59.879 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.254488+0000", 2024-09-21T13:56:59.879 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013163199999999999 2024-09-21T13:56:59.880 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.880 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.880 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:56:59.880 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.254503+0000", 2024-09-21T13:56:59.880 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.452e-05 2024-09-21T13:56:59.880 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.880 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.880 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:56:59.880 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.254510+0000", 2024-09-21T13:56:59.880 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.554e-06 2024-09-21T13:56:59.880 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.880 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.880 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:56:59.880 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.254578+0000", 2024-09-21T13:56:59.881 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7968999999999999e-05 2024-09-21T13:56:59.881 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.881 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.881 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:56:59.881 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.254736+0000", 2024-09-21T13:56:59.881 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000157219 2024-09-21T13:56:59.881 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.881 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.881 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-21T13:56:59.881 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.254886+0000", 2024-09-21T13:56:59.881 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000150667 2024-09-21T13:56:59.881 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.881 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.881 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:56:59.881 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.257811+0000", 2024-09-21T13:56:59.882 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00292436 2024-09-21T13:56:59.882 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.882 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.882 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:59.882 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.261523+0000", 2024-09-21T13:56:59.882 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0037119639999999999 2024-09-21T13:56:59.882 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.882 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.882 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:59.882 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.407826+0000", 2024-09-21T13:56:59.882 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.146303813 2024-09-21T13:56:59.882 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.882 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.882 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:56:59.883 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.407877+0000", 2024-09-21T13:56:59.883 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0603999999999999e-05 2024-09-21T13:56:59.883 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.883 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.883 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:56:59.883 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.407906+0000", 2024-09-21T13:56:59.883 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8994999999999999e-05 2024-09-21T13:56:59.883 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:56:59.883 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:56:59.883 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:56:59.883 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.883 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.883 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:49364 7.d 7:ba37e2e5:::benchmark_data_smithi049_76798_object49363:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T13:56:59.883 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:38.254570+0000", 2024-09-21T13:56:59.884 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 381.57537275200002, 2024-09-21T13:56:59.884 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.156026475, 2024-09-21T13:56:59.884 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:56:59.884 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:56:59.884 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:59.884 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:56:59.884 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:56:59.884 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:56:59.884 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 49364 2024-09-21T13:56:59.884 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.884 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:56:59.884 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.884 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:56:59.884 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.254570+0000", 2024-09-21T13:56:59.884 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:56:59.885 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.885 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.885 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:56:59.885 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.254570+0000", 2024-09-21T13:56:59.885 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:56:59.885 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.885 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.885 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:56:59.885 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.254571+0000", 2024-09-21T13:56:59.885 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.8700000000000004e-07 2024-09-21T13:56:59.885 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.885 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.885 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:56:59.885 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.254637+0000", 2024-09-21T13:56:59.885 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6704000000000004e-05 2024-09-21T13:56:59.886 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.886 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.886 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:56:59.886 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.254652+0000", 2024-09-21T13:56:59.886 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.438e-05 2024-09-21T13:56:59.886 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.886 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.886 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:56:59.886 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.254659+0000", 2024-09-21T13:56:59.886 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9589999999999998e-06 2024-09-21T13:56:59.886 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.886 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.886 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:56:59.886 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.256288+0000", 2024-09-21T13:56:59.887 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001629669 2024-09-21T13:56:59.887 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.887 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.887 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:56:59.887 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.256409+0000", 2024-09-21T13:56:59.887 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012085400000000001 2024-09-21T13:56:59.887 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.887 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.887 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-21T13:56:59.887 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.256520+0000", 2024-09-21T13:56:59.887 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000111141 2024-09-21T13:56:59.887 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.887 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.887 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:56:59.888 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:56:59.889 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:56:59.889 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:56:59.890 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.257926+0000", 2024-09-21T13:56:59.890 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001405959 2024-09-21T13:56:59.890 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.890 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.890 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:59.890 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.263043+0000", 2024-09-21T13:56:59.890 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0051160989999999998 2024-09-21T13:56:59.890 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.890 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.890 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:59.890 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.410522+0000", 2024-09-21T13:56:59.890 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.14747992600000001 2024-09-21T13:56:59.890 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.891 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.891 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:56:59.891 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.410571+0000", 2024-09-21T13:56:59.891 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8538000000000002e-05 2024-09-21T13:56:59.891 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.891 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.891 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:56:59.891 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.410596+0000", 2024-09-21T13:56:59.891 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5259e-05 2024-09-21T13:56:59.891 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:56:59.891 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:56:59.891 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:56:59.891 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.891 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.892 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:49368 7.d 7:bfd0cd80:::benchmark_data_smithi049_76798_object49367:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T13:56:59.892 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:38.259454+0000", 2024-09-21T13:56:59.892 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 381.57048854200002, 2024-09-21T13:56:59.892 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.15201478299999999, 2024-09-21T13:56:59.892 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:56:59.892 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:56:59.892 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:59.892 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:56:59.892 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:56:59.892 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:56:59.892 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 49368 2024-09-21T13:56:59.892 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.892 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:56:59.892 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.893 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:56:59.893 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.259454+0000", 2024-09-21T13:56:59.893 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:56:59.893 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.893 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.893 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:56:59.893 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.259454+0000", 2024-09-21T13:56:59.893 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:56:59.893 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.893 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.893 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:56:59.893 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.259455+0000", 2024-09-21T13:56:59.893 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3260000000000001e-06 2024-09-21T13:56:59.893 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.893 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.894 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:56:59.894 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.259505+0000", 2024-09-21T13:56:59.894 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9941999999999997e-05 2024-09-21T13:56:59.894 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.894 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.894 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:56:59.894 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.259516+0000", 2024-09-21T13:56:59.894 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0943e-05 2024-09-21T13:56:59.894 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.894 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.894 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:56:59.894 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.259523+0000", 2024-09-21T13:56:59.895 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2200000000000003e-06 2024-09-21T13:56:59.895 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.895 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.895 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:56:59.895 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.259582+0000", 2024-09-21T13:56:59.895 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8789000000000002e-05 2024-09-21T13:56:59.895 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.895 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.895 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:56:59.895 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.259708+0000", 2024-09-21T13:56:59.895 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012536700000000001 2024-09-21T13:56:59.895 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.895 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.896 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-21T13:56:59.896 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.259861+0000", 2024-09-21T13:56:59.896 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015389900000000001 2024-09-21T13:56:59.896 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.896 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.896 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:56:59.896 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.261617+0000", 2024-09-21T13:56:59.896 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0017559260000000001 2024-09-21T13:56:59.896 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.896 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.896 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:59.896 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.263374+0000", 2024-09-21T13:56:59.896 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0017561440000000001 2024-09-21T13:56:59.897 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.897 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.897 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:59.897 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.411384+0000", 2024-09-21T13:56:59.897 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.148010004 2024-09-21T13:56:59.897 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.897 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.897 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:56:59.897 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.411437+0000", 2024-09-21T13:56:59.897 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3600999999999997e-05 2024-09-21T13:56:59.897 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.897 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.897 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:56:59.897 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.411469+0000", 2024-09-21T13:56:59.897 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1622000000000003e-05 2024-09-21T13:56:59.898 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:56:59.898 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:56:59.898 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:56:59.898 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.898 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.898 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:49375 7.7 7:ecf0f90a:::benchmark_data_smithi049_76798_object49374:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T13:56:59.898 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:38.263906+0000", 2024-09-21T13:56:59.898 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 381.56603650699998, 2024-09-21T13:56:59.898 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.161950177, 2024-09-21T13:56:59.898 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:56:59.898 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:56:59.898 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:59.898 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:56:59.898 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:56:59.899 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:56:59.899 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 49375 2024-09-21T13:56:59.899 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.899 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:56:59.899 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.899 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:56:59.899 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.263906+0000", 2024-09-21T13:56:59.899 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:56:59.899 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.899 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.899 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:56:59.899 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.263906+0000", 2024-09-21T13:56:59.899 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:56:59.900 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.900 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.900 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:56:59.900 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.263907+0000", 2024-09-21T13:56:59.900 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.61e-07 2024-09-21T13:56:59.900 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.900 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.900 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:56:59.900 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.263963+0000", 2024-09-21T13:56:59.900 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5772000000000001e-05 2024-09-21T13:56:59.900 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.900 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.900 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:56:59.900 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.263969+0000", 2024-09-21T13:56:59.900 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4200000000000004e-06 2024-09-21T13:56:59.900 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.901 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.901 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:56:59.901 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.263973+0000", 2024-09-21T13:56:59.901 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1869999999999999e-06 2024-09-21T13:56:59.901 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.901 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.901 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:56:59.901 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.264028+0000", 2024-09-21T13:56:59.901 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4657999999999997e-05 2024-09-21T13:56:59.901 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.901 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.901 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:56:59.901 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.264143+0000", 2024-09-21T13:56:59.901 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000115223 2024-09-21T13:56:59.901 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.901 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.902 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3", 2024-09-21T13:56:59.902 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.264277+0000", 2024-09-21T13:56:59.902 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000133993 2024-09-21T13:56:59.902 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.902 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.902 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:56:59.902 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.265777+0000", 2024-09-21T13:56:59.902 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0014999740000000001 2024-09-21T13:56:59.902 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.902 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.902 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:59.902 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.268782+0000", 2024-09-21T13:56:59.902 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0030045929999999998 2024-09-21T13:56:59.902 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.902 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.903 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:59.903 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.425737+0000", 2024-09-21T13:56:59.903 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.156955024 2024-09-21T13:56:59.903 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.903 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.903 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:56:59.903 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.425795+0000", 2024-09-21T13:56:59.903 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8443000000000003e-05 2024-09-21T13:56:59.903 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.903 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.903 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:56:59.903 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.425856+0000", 2024-09-21T13:56:59.903 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1129000000000004e-05 2024-09-21T13:56:59.903 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:56:59.904 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:56:59.904 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:56:59.904 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.904 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.904 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:49376 7.7 7:e5b9203c:::benchmark_data_smithi049_76798_object49375:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T13:56:59.904 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:38.265943+0000", 2024-09-21T13:56:59.904 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 381.56399954400001, 2024-09-21T13:56:59.904 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.16002640000000001, 2024-09-21T13:56:59.904 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:56:59.904 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:56:59.904 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:59.904 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:56:59.904 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:56:59.904 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:56:59.904 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 49376 2024-09-21T13:56:59.904 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.904 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:56:59.905 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.905 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:56:59.905 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.265943+0000", 2024-09-21T13:56:59.905 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:56:59.905 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.905 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.905 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:56:59.905 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.265943+0000", 2024-09-21T13:56:59.905 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:56:59.905 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.905 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.905 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:56:59.905 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.265944+0000", 2024-09-21T13:56:59.905 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.237e-06 2024-09-21T13:56:59.905 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.906 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.906 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:56:59.906 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.266008+0000", 2024-09-21T13:56:59.906 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3382999999999999e-05 2024-09-21T13:56:59.906 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.906 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.906 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:56:59.906 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.266018+0000", 2024-09-21T13:56:59.906 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0225e-05 2024-09-21T13:56:59.906 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.906 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.906 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:56:59.906 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.266023+0000", 2024-09-21T13:56:59.906 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0810000000000001e-06 2024-09-21T13:56:59.906 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.907 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.907 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:56:59.907 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.266066+0000", 2024-09-21T13:56:59.907 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2939999999999999e-05 2024-09-21T13:56:59.907 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.907 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.907 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:56:59.907 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.266173+0000", 2024-09-21T13:56:59.907 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000107089 2024-09-21T13:56:59.907 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.907 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.907 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3", 2024-09-21T13:56:59.907 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.266305+0000", 2024-09-21T13:56:59.907 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013210699999999999 2024-09-21T13:56:59.908 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.908 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.908 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:56:59.908 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.268746+0000", 2024-09-21T13:56:59.908 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00244072 2024-09-21T13:56:59.908 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.908 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.908 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:59.908 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.269249+0000", 2024-09-21T13:56:59.908 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00050357600000000005 2024-09-21T13:56:59.908 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.908 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.908 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:59.908 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.425891+0000", 2024-09-21T13:56:59.909 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.15664171700000001 2024-09-21T13:56:59.909 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.909 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.909 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:56:59.909 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.425934+0000", 2024-09-21T13:56:59.909 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3136999999999997e-05 2024-09-21T13:56:59.909 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.909 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.909 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:56:59.909 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.425969+0000", 2024-09-21T13:56:59.909 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5188000000000003e-05 2024-09-21T13:56:59.909 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:56:59.909 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:56:59.909 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:56:59.910 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.910 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.910 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:49377 7.7 7:e5eecd6c:::benchmark_data_smithi049_76798_object49376:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T13:56:59.910 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:38.266042+0000", 2024-09-21T13:56:59.910 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 381.56390019299999, 2024-09-21T13:56:59.910 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.160051258, 2024-09-21T13:56:59.910 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:56:59.910 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:56:59.910 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:59.910 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:56:59.910 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:56:59.910 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:56:59.910 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 49377 2024-09-21T13:56:59.910 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.911 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:56:59.911 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.911 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:56:59.911 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.266042+0000", 2024-09-21T13:56:59.911 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:56:59.911 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.911 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.911 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:56:59.911 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.266042+0000", 2024-09-21T13:56:59.911 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:56:59.911 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.911 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.911 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:56:59.911 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.266043+0000", 2024-09-21T13:56:59.911 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2599999999999998e-07 2024-09-21T13:56:59.912 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.912 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.912 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:56:59.912 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.266116+0000", 2024-09-21T13:56:59.912 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3120999999999993e-05 2024-09-21T13:56:59.912 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.912 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.912 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:56:59.912 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.266126+0000", 2024-09-21T13:56:59.912 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6809999999999993e-06 2024-09-21T13:56:59.912 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.912 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.912 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:56:59.912 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.266133+0000", 2024-09-21T13:56:59.913 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1590000000000004e-06 2024-09-21T13:56:59.913 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.913 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.913 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:56:59.913 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.267417+0000", 2024-09-21T13:56:59.913 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00128472 2024-09-21T13:56:59.913 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.913 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.913 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:56:59.913 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.267503+0000", 2024-09-21T13:56:59.913 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5261999999999998e-05 2024-09-21T13:56:59.913 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.913 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.913 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3", 2024-09-21T13:56:59.913 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.267626+0000", 2024-09-21T13:56:59.914 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000123435 2024-09-21T13:56:59.914 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.914 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.914 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:56:59.914 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.268883+0000", 2024-09-21T13:56:59.914 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00125657 2024-09-21T13:56:59.914 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.914 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.914 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:59.914 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.269389+0000", 2024-09-21T13:56:59.914 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00050581200000000002 2024-09-21T13:56:59.914 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.914 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.914 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:59.915 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.426010+0000", 2024-09-21T13:56:59.915 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.156621173 2024-09-21T13:56:59.915 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.915 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.915 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:56:59.915 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.426059+0000", 2024-09-21T13:56:59.915 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8891000000000003e-05 2024-09-21T13:56:59.915 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.915 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.915 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:56:59.915 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.426094+0000", 2024-09-21T13:56:59.915 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5008000000000001e-05 2024-09-21T13:56:59.915 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:56:59.915 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:56:59.915 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:56:59.916 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.916 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.916 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:202091 7.7 7:ebd82d39:::benchmark_data_smithi049_76798_object65671:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-21T13:56:59.916 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:44.203006+0000", 2024-09-21T13:56:59.916 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 255.626936277, 2024-09-21T13:56:59.916 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.14159556300000001, 2024-09-21T13:56:59.916 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:56:59.916 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:56:59.916 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:59.916 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:56:59.916 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:56:59.916 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:56:59.916 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 202091 2024-09-21T13:56:59.916 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.916 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:56:59.917 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.917 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:56:59.917 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.203006+0000", 2024-09-21T13:56:59.917 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:56:59.917 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.917 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.917 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:56:59.917 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.203006+0000", 2024-09-21T13:56:59.917 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:56:59.917 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.917 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.917 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:56:59.917 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.203007+0000", 2024-09-21T13:56:59.917 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1790000000000001e-06 2024-09-21T13:56:59.918 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.918 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.918 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:56:59.918 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.203012+0000", 2024-09-21T13:56:59.918 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6040000000000003e-06 2024-09-21T13:56:59.918 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.918 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.918 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:56:59.918 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.203021+0000", 2024-09-21T13:56:59.918 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.2539999999999998e-06 2024-09-21T13:56:59.918 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.918 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.918 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:56:59.918 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.203026+0000", 2024-09-21T13:56:59.919 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9830000000000004e-06 2024-09-21T13:56:59.919 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.919 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.919 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:56:59.919 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.203078+0000", 2024-09-21T13:56:59.919 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1421000000000001e-05 2024-09-21T13:56:59.919 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.919 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.919 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:56:59.919 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.203188+0000", 2024-09-21T13:56:59.919 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000110599 2024-09-21T13:56:59.919 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.919 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.919 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3", 2024-09-21T13:56:59.920 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.203294+0000", 2024-09-21T13:56:59.920 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000105986 2024-09-21T13:56:59.920 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.920 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.920 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:56:59.920 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.206024+0000", 2024-09-21T13:56:59.920 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0027295980000000002 2024-09-21T13:56:59.920 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.920 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.920 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:59.920 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.211989+0000", 2024-09-21T13:56:59.920 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0059648949999999996 2024-09-21T13:56:59.920 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.920 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.920 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:59.921 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.344508+0000", 2024-09-21T13:56:59.921 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13251928399999999 2024-09-21T13:56:59.921 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.921 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.921 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:56:59.921 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.344562+0000", 2024-09-21T13:56:59.921 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3653999999999998e-05 2024-09-21T13:56:59.921 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.921 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.921 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:56:59.921 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.344602+0000", 2024-09-21T13:56:59.921 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0105999999999999e-05 2024-09-21T13:56:59.921 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:56:59.921 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:56:59.921 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:56:59.922 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.922 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.922 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:202095 7.7 7:e91fc271:::benchmark_data_smithi049_76798_object65675:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-21T13:56:59.922 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:44.206857+0000", 2024-09-21T13:56:59.922 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 255.62308600200001, 2024-09-21T13:56:59.922 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13794869500000001, 2024-09-21T13:56:59.922 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:56:59.922 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:56:59.922 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:59.922 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:56:59.922 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:56:59.922 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:56:59.922 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 202095 2024-09-21T13:56:59.923 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.923 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:56:59.923 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.923 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:56:59.923 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.206857+0000", 2024-09-21T13:56:59.923 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:56:59.923 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.923 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.923 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:56:59.923 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.206857+0000", 2024-09-21T13:56:59.923 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:56:59.923 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.923 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.923 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:56:59.923 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.206858+0000", 2024-09-21T13:56:59.924 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.576e-06 2024-09-21T13:56:59.924 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.924 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.924 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:56:59.924 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.206863+0000", 2024-09-21T13:56:59.924 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3680000000000001e-06 2024-09-21T13:56:59.924 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.924 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.924 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:56:59.924 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.206875+0000", 2024-09-21T13:56:59.924 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.155e-05 2024-09-21T13:56:59.924 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.924 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.924 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:56:59.925 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.206882+0000", 2024-09-21T13:56:59.925 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1509999999999998e-06 2024-09-21T13:56:59.925 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.925 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.925 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:56:59.925 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.206924+0000", 2024-09-21T13:56:59.925 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1381999999999998e-05 2024-09-21T13:56:59.925 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.925 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.925 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:56:59.925 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.207007+0000", 2024-09-21T13:56:59.925 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3189000000000006e-05 2024-09-21T13:56:59.925 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.925 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.926 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3", 2024-09-21T13:56:59.926 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.207119+0000", 2024-09-21T13:56:59.926 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000112176 2024-09-21T13:56:59.926 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.926 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.926 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:56:59.926 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.211963+0000", 2024-09-21T13:56:59.926 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0048439690000000001 2024-09-21T13:56:59.926 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.926 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.926 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:59.926 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.344431+0000", 2024-09-21T13:56:59.926 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.132468163 2024-09-21T13:56:59.926 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.927 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.927 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:59.927 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.344717+0000", 2024-09-21T13:56:59.927 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00028593299999999999 2024-09-21T13:56:59.927 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.927 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.927 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:56:59.927 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.344765+0000", 2024-09-21T13:56:59.927 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8211000000000003e-05 2024-09-21T13:56:59.927 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.927 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.927 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:56:59.927 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.344805+0000", 2024-09-21T13:56:59.927 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0027000000000001e-05 2024-09-21T13:56:59.927 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:56:59.928 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:56:59.928 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:56:59.928 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.928 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.928 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:202096 7.7 7:ea9b4a34:::benchmark_data_smithi049_76798_object65676:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-21T13:56:59.928 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:44.207290+0000", 2024-09-21T13:56:59.928 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 255.62265290799999, 2024-09-21T13:56:59.928 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.137682887, 2024-09-21T13:56:59.928 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:56:59.928 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:56:59.928 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:59.928 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:56:59.928 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:56:59.928 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:56:59.928 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 202096 2024-09-21T13:56:59.929 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.929 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:56:59.929 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.929 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:56:59.929 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.207290+0000", 2024-09-21T13:56:59.929 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:56:59.929 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.929 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.930 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:56:59.930 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.207290+0000", 2024-09-21T13:56:59.931 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:56:59.931 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.931 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.931 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:56:59.931 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.207290+0000", 2024-09-21T13:56:59.931 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6499999999999998e-07 2024-09-21T13:56:59.931 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.931 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.931 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:56:59.931 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.207293+0000", 2024-09-21T13:56:59.931 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0189999999999998e-06 2024-09-21T13:56:59.931 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.931 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.931 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:56:59.931 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.207301+0000", 2024-09-21T13:56:59.932 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4229999999999999e-06 2024-09-21T13:56:59.932 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.932 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.932 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:56:59.932 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.207306+0000", 2024-09-21T13:56:59.932 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2970000000000003e-06 2024-09-21T13:56:59.932 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.932 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.932 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:56:59.932 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.209527+0000", 2024-09-21T13:56:59.932 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.002220662 2024-09-21T13:56:59.932 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.932 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.932 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:56:59.933 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.209636+0000", 2024-09-21T13:56:59.933 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000109008 2024-09-21T13:56:59.933 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.933 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.933 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3", 2024-09-21T13:56:59.933 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.209742+0000", 2024-09-21T13:56:59.933 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000106 2024-09-21T13:56:59.933 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.933 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.933 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:56:59.933 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.212108+0000", 2024-09-21T13:56:59.933 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0023659229999999998 2024-09-21T13:56:59.933 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.933 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.934 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:59.934 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.344647+0000", 2024-09-21T13:56:59.934 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.132539724 2024-09-21T13:56:59.934 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.934 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.934 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:59.934 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.344872+0000", 2024-09-21T13:56:59.934 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00022439899999999999 2024-09-21T13:56:59.934 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.934 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.934 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:56:59.934 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.344925+0000", 2024-09-21T13:56:59.934 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3557999999999998e-05 2024-09-21T13:56:59.934 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.935 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.935 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:56:59.935 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.344973+0000", 2024-09-21T13:56:59.935 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7108999999999999e-05 2024-09-21T13:56:59.935 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:56:59.935 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:56:59.935 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:56:59.935 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.935 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.935 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:202103 7.d 7:b2f8fda1:::benchmark_data_smithi049_76798_object65683:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-21T13:56:59.935 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:44.214267+0000", 2024-09-21T13:56:59.935 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 255.615676052, 2024-09-21T13:56:59.935 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.125722836, 2024-09-21T13:56:59.935 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:56:59.936 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:56:59.936 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:59.936 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:56:59.936 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:56:59.936 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:56:59.936 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 202103 2024-09-21T13:56:59.936 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.936 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:56:59.936 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.936 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:56:59.936 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.214267+0000", 2024-09-21T13:56:59.936 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:56:59.936 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.936 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.937 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:56:59.937 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.214267+0000", 2024-09-21T13:56:59.937 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:56:59.937 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.937 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.937 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:56:59.937 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.214268+0000", 2024-09-21T13:56:59.937 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9400000000000001e-06 2024-09-21T13:56:59.937 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.937 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.937 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:56:59.937 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.214278+0000", 2024-09-21T13:56:59.937 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3500000000000003e-06 2024-09-21T13:56:59.937 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.938 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.938 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:56:59.938 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.214292+0000", 2024-09-21T13:56:59.938 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4157e-05 2024-09-21T13:56:59.938 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.938 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.938 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:56:59.938 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.214301+0000", 2024-09-21T13:56:59.938 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8899999999999996e-06 2024-09-21T13:56:59.938 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.938 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.938 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:56:59.938 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.214358+0000", 2024-09-21T13:56:59.939 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7361000000000002e-05 2024-09-21T13:56:59.939 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.939 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.939 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:56:59.939 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.214475+0000", 2024-09-21T13:56:59.939 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000116654 2024-09-21T13:56:59.939 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.939 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.939 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-21T13:56:59.939 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.214586+0000", 2024-09-21T13:56:59.939 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011076 2024-09-21T13:56:59.939 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.939 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.939 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:56:59.939 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.217180+0000", 2024-09-21T13:56:59.940 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0025942140000000001 2024-09-21T13:56:59.940 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.940 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.940 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:59.940 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.339864+0000", 2024-09-21T13:56:59.940 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.122684511 2024-09-21T13:56:59.940 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.940 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.940 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:59.940 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.339900+0000", 2024-09-21T13:56:59.940 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6075999999999998e-05 2024-09-21T13:56:59.940 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.940 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.940 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:56:59.941 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.339949+0000", 2024-09-21T13:56:59.941 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8548999999999999e-05 2024-09-21T13:56:59.941 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.941 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.941 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:56:59.941 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.339989+0000", 2024-09-21T13:56:59.941 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0373999999999997e-05 2024-09-21T13:56:59.941 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:56:59.941 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:56:59.941 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:56:59.941 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.941 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.941 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:202104 7.7 7:e253bf76:::benchmark_data_smithi049_76798_object65684:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-21T13:56:59.941 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:44.217360+0000", 2024-09-21T13:56:59.941 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 255.61258237300001, 2024-09-21T13:56:59.942 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.15782971300000001, 2024-09-21T13:56:59.942 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:56:59.942 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:56:59.942 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:56:59.942 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:56:59.942 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:56:59.942 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:56:59.942 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 202104 2024-09-21T13:56:59.942 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.942 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:56:59.942 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.942 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:56:59.942 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.217360+0000", 2024-09-21T13:56:59.942 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:56:59.942 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.943 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.943 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:56:59.943 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.217360+0000", 2024-09-21T13:56:59.943 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:56:59.943 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.943 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.943 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:56:59.943 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.217361+0000", 2024-09-21T13:56:59.943 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.279e-06 2024-09-21T13:56:59.943 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.943 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.943 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:56:59.943 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.217367+0000", 2024-09-21T13:56:59.943 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2120000000000002e-06 2024-09-21T13:56:59.943 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.944 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.944 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:56:59.944 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.217379+0000", 2024-09-21T13:56:59.944 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2612000000000001e-05 2024-09-21T13:56:59.944 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.944 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.944 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:56:59.944 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.217387+0000", 2024-09-21T13:56:59.944 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8909999999999995e-06 2024-09-21T13:56:59.944 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.944 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.944 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:56:59.944 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.217429+0000", 2024-09-21T13:56:59.945 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1785000000000001e-05 2024-09-21T13:56:59.945 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.945 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.945 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:56:59.945 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.217506+0000", 2024-09-21T13:56:59.945 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7194000000000007e-05 2024-09-21T13:56:59.945 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.945 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.945 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3", 2024-09-21T13:56:59.945 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.340772+0000", 2024-09-21T13:56:59.945 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.12326546100000001 2024-09-21T13:56:59.945 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.945 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.945 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:56:59.945 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.344667+0000", 2024-09-21T13:56:59.946 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003894907 2024-09-21T13:56:59.946 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.946 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.946 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:59.946 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.369243+0000", 2024-09-21T13:56:59.946 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.024576745000000001 2024-09-21T13:56:59.946 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.946 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.946 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:56:59.946 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.375131+0000", 2024-09-21T13:56:59.946 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0058877240000000004 2024-09-21T13:56:59.946 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.946 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.946 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:56:59.947 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.375162+0000", 2024-09-21T13:56:59.947 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1198999999999997e-05 2024-09-21T13:56:59.947 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.947 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.947 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:56:59.947 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.375190+0000", 2024-09-21T13:56:59.947 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7704000000000001e-05 2024-09-21T13:56:59.947 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:56:59.947 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:56:59.947 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:56:59.947 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.947 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.947 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.14817.0:202091 7.7 e148/83 ondisk, result = 0)", 2024-09-21T13:56:59.948 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:44.218874+0000", 2024-09-21T13:56:59.948 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 255.611068507, 2024-09-21T13:56:59.948 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.125736712, 2024-09-21T13:56:59.948 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:56:59.948 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:56:59.948 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T13:56:59.948 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:56:59.948 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.948 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:56:59.948 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.218874+0000", 2024-09-21T13:56:59.948 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:56:59.948 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.948 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.948 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:56:59.949 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.218874+0000", 2024-09-21T13:56:59.949 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:56:59.949 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.949 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.949 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:56:59.949 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.218875+0000", 2024-09-21T13:56:59.949 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0699999999999996e-07 2024-09-21T13:56:59.949 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.949 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.949 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:56:59.949 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.218879+0000", 2024-09-21T13:56:59.949 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1280000000000001e-06 2024-09-21T13:56:59.949 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.949 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.950 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:56:59.950 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.340853+0000", 2024-09-21T13:56:59.950 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.121973442 2024-09-21T13:56:59.950 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.950 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.950 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:56:59.950 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.340860+0000", 2024-09-21T13:56:59.950 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1919999999999999e-06 2024-09-21T13:56:59.950 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.950 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.950 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:56:59.950 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.344495+0000", 2024-09-21T13:56:59.950 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0036355979999999999 2024-09-21T13:56:59.950 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.950 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.951 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:56:59.951 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.344502+0000", 2024-09-21T13:56:59.951 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4110000000000001e-06 2024-09-21T13:56:59.951 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:56:59.951 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:56:59.951 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:56:59.951 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.344611+0000", 2024-09-21T13:56:59.951 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000109034 2024-09-21T13:56:59.951 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:56:59.951 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:56:59.951 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:56:59.951 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:56:59.951 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:56:59.951 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T13:56:59.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:56:59.952 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:56:59.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:00.053 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:00.086 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:00.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:00.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:00.188 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:00.225 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:00.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:00.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:00.326 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:00.368 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:00.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:00.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:00.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:00 smithi187 ceph-mon[33032]: 3.1a scrub starts 2024-09-21T13:57:00.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:00 smithi187 ceph-mon[33032]: 3.1a scrub ok 2024-09-21T13:57:00.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:00 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:57:00.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:00 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2262568875' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:57:00.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:00 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:57:00.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:00 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:57:00.446 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:00 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 15104/130236 objects degraded (11.597%), 31 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-09-21T13:57:00.470 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:00.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:00 smithi049 ceph-mon[30080]: 3.1a scrub starts 2024-09-21T13:57:00.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:00 smithi049 ceph-mon[30080]: 3.1a scrub ok 2024-09-21T13:57:00.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:00 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:57:00.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:00 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2262568875' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:57:00.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:00 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:57:00.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:00 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:57:00.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:00 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 15104/130236 objects degraded (11.597%), 31 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-09-21T13:57:00.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:00 smithi049 ceph-mon[40489]: 3.1a scrub starts 2024-09-21T13:57:00.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:00 smithi049 ceph-mon[40489]: 3.1a scrub ok 2024-09-21T13:57:00.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:00 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:57:00.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:00 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2262568875' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:57:00.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:00 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:57:00.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:00 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:57:00.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:00 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 15104/130236 objects degraded (11.597%), 31 pgs degraded, 69 pgs undersized (PG_DEGRADED) 2024-09-21T13:57:00.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:00.507 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:00.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:00.608 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:00.639 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:00.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:00.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:00.729 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T13:57:00.729 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T13:57:00.742 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:00.769 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:00.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:00.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:00.871 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:00.878 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T13:57:00.881 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:57:00 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:57:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:57:00.882 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 13:57:00 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-3-activate[128867]: --> Failed to activate via raw: did not find any matching OSD to activate 2024-09-21T13:57:00.882 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 13:57:00 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-3-activate[128867]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T13:57:00.882 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 13:57:00 smithi049 bash[128795]: --> Failed to activate via raw: did not find any matching OSD to activate 2024-09-21T13:57:00.882 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 13:57:00 smithi049 bash[128795]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T13:57:00.882 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 13:57:00 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-3-activate[128867]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T13:57:00.882 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 13:57:00 smithi049 bash[128795]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T13:57:00.888 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T13:57:00.913 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:00.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:00.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:01.015 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:57:01.050 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:01.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:01.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:57:01.152 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:01.184 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:01 smithi049 ceph-mon[40489]: 5.e scrub starts 2024-09-21T13:57:01.185 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:01 smithi049 ceph-mon[40489]: 5.e scrub ok 2024-09-21T13:57:01.185 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:01 smithi049 ceph-mon[40489]: pgmap v388: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfilling, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 2.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.59k op/s; 12206/112119 objects degraded (10.887%); 4769/112119 objects misplaced (4.254%); 12 MiB/s, 192 objects/s recovering 2024-09-21T13:57:01.185 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:01 smithi049 ceph-mon[40489]: osdmap e234: 8 total, 7 up, 7 in 2024-09-21T13:57:01.185 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:01 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:57:01.185 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:01 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:57:01.185 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 13:57:00 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-3-activate[128867]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-21T13:57:01.185 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 13:57:00 smithi049 bash[128795]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-21T13:57:01.185 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 13:57:00 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-3-activate[128867]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_1 --path /var/lib/ceph/osd/ceph-3 --no-mon-config 2024-09-21T13:57:01.186 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 13:57:00 smithi049 bash[128795]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_1 --path /var/lib/ceph/osd/ceph-3 --no-mon-config 2024-09-21T13:57:01.188 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:01 smithi049 ceph-mon[30080]: 5.e scrub starts 2024-09-21T13:57:01.188 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:01 smithi049 ceph-mon[30080]: 5.e scrub ok 2024-09-21T13:57:01.188 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:01 smithi049 ceph-mon[30080]: pgmap v388: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfilling, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 2.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.59k op/s; 12206/112119 objects degraded (10.887%); 4769/112119 objects misplaced (4.254%); 12 MiB/s, 192 objects/s recovering 2024-09-21T13:57:01.188 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:01 smithi049 ceph-mon[30080]: osdmap e234: 8 total, 7 up, 7 in 2024-09-21T13:57:01.189 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:01 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:57:01.189 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:01 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:57:01.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:01.194 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:01.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:01.197 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:01 smithi187 ceph-mon[33032]: 5.e scrub starts 2024-09-21T13:57:01.198 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:01 smithi187 ceph-mon[33032]: 5.e scrub ok 2024-09-21T13:57:01.198 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:01 smithi187 ceph-mon[33032]: pgmap v388: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfilling, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 2.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.59k op/s; 12206/112119 objects degraded (10.887%); 4769/112119 objects misplaced (4.254%); 12 MiB/s, 192 objects/s recovering 2024-09-21T13:57:01.198 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:01 smithi187 ceph-mon[33032]: osdmap e234: 8 total, 7 up, 7 in 2024-09-21T13:57:01.198 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:01 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:57:01.198 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:01 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:57:01.263 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:57:01.295 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:01.336 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:01.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:01.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:01.437 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:01.470 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:01.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:01.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:01.505 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 13:57:01 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-3-activate[128867]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3/block 2024-09-21T13:57:01.505 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 13:57:01 smithi049 bash[128795]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3/block 2024-09-21T13:57:01.505 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 13:57:01 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-3-activate[128867]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-3/block 2024-09-21T13:57:01.505 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 13:57:01 smithi049 bash[128795]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-3/block 2024-09-21T13:57:01.505 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 13:57:01 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-3-activate[128867]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-21T13:57:01.505 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 13:57:01 smithi049 bash[128795]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-21T13:57:01.505 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 13:57:01 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-3-activate[128867]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-21T13:57:01.505 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 13:57:01 smithi049 bash[128795]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-21T13:57:01.506 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 13:57:01 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-3-activate[128867]: --> ceph-volume lvm activate successful for osd ID: 3 2024-09-21T13:57:01.506 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 13:57:01 smithi049 bash[128795]: --> ceph-volume lvm activate successful for osd ID: 3 2024-09-21T13:57:01.506 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 13:57:01 smithi049 conmon[128867]: conmon b8470d59e297f480487e : Failed to open cgroups file: /sys/fs/cgroup/machine.slice/libpod-b8470d59e297f480487e40c8f9b0d7adcc9a87eb3a2bfc12f48adcefb1237169.scope/memory.events 2024-09-21T13:57:01.506 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 13:57:01 smithi049 podman[128795]: 2024-09-21 13:57:01.243290118 +0000 UTC m=+4.279766636 container died b8470d59e297f480487e40c8f9b0d7adcc9a87eb3a2bfc12f48adcefb1237169 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-3-activate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T13:57:01.572 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:57:01.613 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:01.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:01.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:57:01.715 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:01.744 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:01.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:01.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:01.845 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:01.883 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:01.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:01.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:01.919 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T13:57:01.919 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:57:01.979 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:57:01.984 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:02.034 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:02.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:02.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:02.136 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:02.170 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:02.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:02.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:02.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:02 smithi049 ceph-mon[30080]: 6.1a scrub starts 2024-09-21T13:57:02.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:02 smithi049 ceph-mon[30080]: 6.1a scrub ok 2024-09-21T13:57:02.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:02 smithi049 ceph-mon[30080]: osdmap e235: 8 total, 7 up, 7 in 2024-09-21T13:57:02.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:02 smithi049 ceph-mon[40489]: 6.1a scrub starts 2024-09-21T13:57:02.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:02 smithi049 ceph-mon[40489]: 6.1a scrub ok 2024-09-21T13:57:02.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:02 smithi049 ceph-mon[40489]: osdmap e235: 8 total, 7 up, 7 in 2024-09-21T13:57:02.256 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 13:57:01 smithi049 podman[129489]: 2024-09-21 13:57:01.922935735 +0000 UTC m=+0.665271265 container remove b8470d59e297f480487e40c8f9b0d7adcc9a87eb3a2bfc12f48adcefb1237169 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-3-activate, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-guits-main-2024-09-21-0842-4417849, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-09-21T13:57:02.272 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:02.328 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:02.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:02.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:02.429 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:02.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:02 smithi187 ceph-mon[33032]: 6.1a scrub starts 2024-09-21T13:57:02.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:02 smithi187 ceph-mon[33032]: 6.1a scrub ok 2024-09-21T13:57:02.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:02 smithi187 ceph-mon[33032]: osdmap e235: 8 total, 7 up, 7 in 2024-09-21T13:57:02.459 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:02.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:02.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:02.561 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:57:02.602 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:02.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:02.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:57:02.704 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:02.731 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T13:57:02.736 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:02.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:02.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:02.838 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:02.869 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:02.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:02.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:02.924 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 13:57:02 smithi049 podman[129707]: 2024-09-21 13:57:02.631718486 +0000 UTC m=+0.043304437 image pull b415816db8b41f544cdfdc71d193ff8b1ee008bf7b0ec4f64da8eb8f71b62e85 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5 2024-09-21T13:57:02.924 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 13:57:02 smithi049 podman[129707]: 2024-09-21 13:57:02.770406048 +0000 UTC m=+0.181991984 container create 48889efcabc233953851259bbae7c1dfdd5a57b89411580d7df6cbdf2137b3f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-3, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-main-2024-09-21-0842-4417849) 2024-09-21T13:57:02.971 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:02.999 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:03.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:03.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:03.089 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-09-21T13:57:03.101 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:03.126 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:03.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:03.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:03.182 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:03 smithi049 ceph-mon[40489]: pgmap v391: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfilling, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 2.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.01k op/s; 12206/112119 objects degraded (10.887%); 4769/112119 objects misplaced (4.254%); 9.5 MiB/s, 152 objects/s recovering 2024-09-21T13:57:03.182 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:03 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3385867470' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:57:03.183 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:03 smithi049 ceph-mon[30080]: pgmap v391: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfilling, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 2.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.01k op/s; 12206/112119 objects degraded (10.887%); 4769/112119 objects misplaced (4.254%); 9.5 MiB/s, 152 objects/s recovering 2024-09-21T13:57:03.183 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:03 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3385867470' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:57:03.183 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 13:57:03 smithi049 podman[129707]: 2024-09-21 13:57:03.075530157 +0000 UTC m=+0.487116094 container init 48889efcabc233953851259bbae7c1dfdd5a57b89411580d7df6cbdf2137b3f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-3, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-21-0842-4417849, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T13:57:03.183 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 13:57:03 smithi049 podman[129707]: 2024-09-21 13:57:03.088460382 +0000 UTC m=+0.500046320 container start 48889efcabc233953851259bbae7c1dfdd5a57b89411580d7df6cbdf2137b3f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-3, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-09-21T13:57:03.189 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T13:57:03.228 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:03.260 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:03.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:03.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:03.362 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:57:03.387 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:03.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:03.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:57:03.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:03 smithi187 ceph-mon[33032]: pgmap v391: 148 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfilling, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 2.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.01k op/s; 12206/112119 objects degraded (10.887%); 4769/112119 objects misplaced (4.254%); 9.5 MiB/s, 152 objects/s recovering 2024-09-21T13:57:03.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:03 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3385867470' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:57:03.473 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:57:03.489 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:03.504 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 13:57:03 smithi049 bash[129707]: 48889efcabc233953851259bbae7c1dfdd5a57b89411580d7df6cbdf2137b3f4 2024-09-21T13:57:03.504 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 13:57:03 smithi049 systemd[1]: Started Ceph osd.3 for 40a3d2e8-781f-11ef-baf6-efdc52797490. 2024-09-21T13:57:03.510 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:03.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:03.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:03.612 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:03.633 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:03.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:03.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:03.734 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:03.757 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:03.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:03.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:03.862 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T13:57:03.862 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:57:03.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:03.884 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:03.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:57:03.985 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:04.008 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:04.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:04.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:04.108 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:57:04.109 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:57:04.137 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:04.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:04.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:57:04.151 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:57:04.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:04 smithi187 ceph-mon[33032]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:57:04.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:04 smithi187 ceph-mon[33032]: 6.1d scrub starts 2024-09-21T13:57:04.196 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:04 smithi187 ceph-mon[33032]: 6.1d scrub ok 2024-09-21T13:57:04.196 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:04 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3385867470' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T13:57:04.196 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:04 smithi187 ceph-mon[33032]: osdmap e236: 8 total, 7 up, 7 in 2024-09-21T13:57:04.239 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:04.282 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:04.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:04.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:04.384 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:04.421 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:04 smithi049 ceph-mon[40489]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:57:04.421 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:04 smithi049 ceph-mon[40489]: 6.1d scrub starts 2024-09-21T13:57:04.421 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:04 smithi049 ceph-mon[40489]: 6.1d scrub ok 2024-09-21T13:57:04.421 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:04 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3385867470' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T13:57:04.421 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:04 smithi049 ceph-mon[40489]: osdmap e236: 8 total, 7 up, 7 in 2024-09-21T13:57:04.421 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:04 smithi049 ceph-mon[30080]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:57:04.422 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:04 smithi049 ceph-mon[30080]: 6.1d scrub starts 2024-09-21T13:57:04.422 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:04 smithi049 ceph-mon[30080]: 6.1d scrub ok 2024-09-21T13:57:04.422 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:04 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3385867470' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T13:57:04.422 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:04 smithi049 ceph-mon[30080]: osdmap e236: 8 total, 7 up, 7 in 2024-09-21T13:57:04.425 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:04.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:04.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:04.527 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:57:04.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:04.562 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:04.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:57:04.662 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:04.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:04.706 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:04.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:04.716 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T13:57:04.716 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:57:04.807 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:04.850 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:04.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:04.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:04.952 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:04.983 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:04.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:04.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:05.085 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:05.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:05.126 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:05.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:05.195 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:57:05.195 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:57:05.195 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:57:05.195 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:57:05.195 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:57:05.195 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:57:05.195 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:57:05.195 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:57:05.196 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-21T13:57:05.196 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:57:05.196 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:57:05.209 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:57:05.209 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:57:05.227 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:05.244 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:57:05.245 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:57:05.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:05 smithi049 ceph-mon[40489]: pgmap v393: 148 pgs: 1 active+clean+remapped, 1 remapped+peering, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 1.9 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.99k op/s; 10111/95517 objects degraded (10.586%); 2001/95517 objects misplaced (2.095%); 14 MiB/s, 231 objects/s recovering 2024-09-21T13:57:05.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:05 smithi049 ceph-mon[40489]: osdmap e237: 8 total, 7 up, 7 in 2024-09-21T13:57:05.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:05 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/323117738' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:57:05.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:05 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 10111/95517 objects degraded (10.586%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-09-21T13:57:05.255 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 13:57:05 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-3[129752]: 2024-09-21T13:57:05.000+0000 7ff0da45c640 -1 Falling back to public interface 2024-09-21T13:57:05.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:05 smithi049 ceph-mon[30080]: pgmap v393: 148 pgs: 1 active+clean+remapped, 1 remapped+peering, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 1.9 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.99k op/s; 10111/95517 objects degraded (10.586%); 2001/95517 objects misplaced (2.095%); 14 MiB/s, 231 objects/s recovering 2024-09-21T13:57:05.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:05 smithi049 ceph-mon[30080]: osdmap e237: 8 total, 7 up, 7 in 2024-09-21T13:57:05.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:05 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/323117738' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:57:05.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:05 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 10111/95517 objects degraded (10.586%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-09-21T13:57:05.269 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:05.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:05.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:05.371 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:05.376 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:05 smithi187 ceph-mon[33032]: pgmap v393: 148 pgs: 1 active+clean+remapped, 1 remapped+peering, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 1.9 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.99k op/s; 10111/95517 objects degraded (10.586%); 2001/95517 objects misplaced (2.095%); 14 MiB/s, 231 objects/s recovering 2024-09-21T13:57:05.376 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:05 smithi187 ceph-mon[33032]: osdmap e237: 8 total, 7 up, 7 in 2024-09-21T13:57:05.376 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:05 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/323117738' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:57:05.376 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:05 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 10111/95517 objects degraded (10.586%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-09-21T13:57:05.408 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:05.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:05.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:05.420 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T13:57:05.510 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:05.542 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:05.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:05.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:05.644 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:05.687 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:05.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:05.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:05.788 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:05.827 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:57:05.829 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:05.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:05.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:05.930 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:05.964 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:05.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:05.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:06.066 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:06.072 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T13:57:06.109 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T13:57:06.110 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T13:57:06.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:06.114 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:06.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:06.215 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:57:06.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:06.264 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:06.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:57:06.356 INFO:teuthology.orchestra.run.smithi049.stderr:no valid command found; 10 closest matches: 2024-09-21T13:57:06.356 INFO:teuthology.orchestra.run.smithi049.stderr:0 2024-09-21T13:57:06.356 INFO:teuthology.orchestra.run.smithi049.stderr:1 2024-09-21T13:57:06.356 INFO:teuthology.orchestra.run.smithi049.stderr:2 2024-09-21T13:57:06.356 INFO:teuthology.orchestra.run.smithi049.stderr:abort 2024-09-21T13:57:06.356 INFO:teuthology.orchestra.run.smithi049.stderr:assert 2024-09-21T13:57:06.356 INFO:teuthology.orchestra.run.smithi049.stderr:config diff 2024-09-21T13:57:06.356 INFO:teuthology.orchestra.run.smithi049.stderr:config diff get 2024-09-21T13:57:06.356 INFO:teuthology.orchestra.run.smithi049.stderr:config get 2024-09-21T13:57:06.356 INFO:teuthology.orchestra.run.smithi049.stderr:config help [] 2024-09-21T13:57:06.357 INFO:teuthology.orchestra.run.smithi049.stderr:config set ... 2024-09-21T13:57:06.357 INFO:teuthology.orchestra.run.smithi049.stderr:admin_socket: invalid command 2024-09-21T13:57:06.365 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:06.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:06.401 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:06.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:06.435 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T13:57:06.435 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-21T13:57:06.435 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-21T13:57:06.435 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-21T13:57:06.436 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.436 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:35249 7.6 7:6f97b9cd:::benchmark_data_smithi049_76798_object35248:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T13:57:06.436 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:28.696132+0000", 2024-09-21T13:57:06.436 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 397.73746684000002, 2024-09-21T13:57:06.436 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0107570610000001, 2024-09-21T13:57:06.436 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:57:06.436 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:57:06.436 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:06.436 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:57:06.436 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:57:06.436 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:57:06.436 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 35249 2024-09-21T13:57:06.437 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.437 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:57:06.437 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.437 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:57:06.437 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.696132+0000", 2024-09-21T13:57:06.437 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:57:06.437 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.437 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.437 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:57:06.437 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.696132+0000", 2024-09-21T13:57:06.437 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:57:06.437 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.437 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.438 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:57:06.438 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.696134+0000", 2024-09-21T13:57:06.438 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6080000000000001e-06 2024-09-21T13:57:06.438 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.438 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.438 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:57:06.438 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.696185+0000", 2024-09-21T13:57:06.438 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1328000000000001e-05 2024-09-21T13:57:06.438 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.438 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.438 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:57:06.438 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.696202+0000", 2024-09-21T13:57:06.438 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7334999999999999e-05 2024-09-21T13:57:06.438 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.439 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.439 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:57:06.439 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.696212+0000", 2024-09-21T13:57:06.439 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3489999999999998e-06 2024-09-21T13:57:06.439 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.439 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.439 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:57:06.439 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.696291+0000", 2024-09-21T13:57:06.439 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9078000000000004e-05 2024-09-21T13:57:06.439 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.439 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.439 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:57:06.439 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.696301+0000", 2024-09-21T13:57:06.440 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7780000000000002e-06 2024-09-21T13:57:06.440 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.440 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.440 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:57:06.440 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.697326+0000", 2024-09-21T13:57:06.440 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0010256289999999 2024-09-21T13:57:06.440 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.440 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.440 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:57:06.440 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.698023+0000", 2024-09-21T13:57:06.440 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00069705799999999997 2024-09-21T13:57:06.440 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.440 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.441 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3", 2024-09-21T13:57:06.441 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.698222+0000", 2024-09-21T13:57:06.441 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000198937 2024-09-21T13:57:06.441 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.441 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.441 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:57:06.441 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.703079+0000", 2024-09-21T13:57:06.441 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0048563690000000001 2024-09-21T13:57:06.441 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.441 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.441 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:06.441 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.706547+0000", 2024-09-21T13:57:06.441 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0034680499999999999 2024-09-21T13:57:06.441 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.441 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.442 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:06.442 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.706817+0000", 2024-09-21T13:57:06.442 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00026996700000000003 2024-09-21T13:57:06.442 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.442 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.442 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:57:06.442 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.706866+0000", 2024-09-21T13:57:06.442 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9292999999999997e-05 2024-09-21T13:57:06.442 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.442 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.442 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:57:06.442 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.706889+0000", 2024-09-21T13:57:06.442 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3281999999999998e-05 2024-09-21T13:57:06.442 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:57:06.442 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:57:06.443 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:57:06.443 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.443 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.443 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:35258 7.6 7:6cec60f9:::benchmark_data_smithi049_76798_object35257:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T13:57:06.443 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:28.700988+0000", 2024-09-21T13:57:06.443 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 397.732611109, 2024-09-21T13:57:06.443 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.007550977, 2024-09-21T13:57:06.443 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:57:06.443 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:57:06.443 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:06.443 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:57:06.443 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:57:06.443 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:57:06.443 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 35258 2024-09-21T13:57:06.444 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.444 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:57:06.444 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.444 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:57:06.444 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.700988+0000", 2024-09-21T13:57:06.444 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:57:06.444 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.444 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.444 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:57:06.444 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.700988+0000", 2024-09-21T13:57:06.444 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:57:06.444 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.444 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.444 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:57:06.445 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.700990+0000", 2024-09-21T13:57:06.445 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.795e-06 2024-09-21T13:57:06.445 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.445 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.445 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:57:06.445 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.701031+0000", 2024-09-21T13:57:06.445 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0836e-05 2024-09-21T13:57:06.445 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.445 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.445 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:57:06.445 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.701045+0000", 2024-09-21T13:57:06.445 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4178e-05 2024-09-21T13:57:06.445 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.445 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.446 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:57:06.446 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.701053+0000", 2024-09-21T13:57:06.446 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3499999999999997e-06 2024-09-21T13:57:06.446 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.446 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.446 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:57:06.446 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.701113+0000", 2024-09-21T13:57:06.446 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9857e-05 2024-09-21T13:57:06.446 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.446 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.446 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:57:06.446 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.701123+0000", 2024-09-21T13:57:06.446 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0202e-05 2024-09-21T13:57:06.447 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.447 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.447 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:57:06.447 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.700130+0000", 2024-09-21T13:57:06.447 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.99900719299999996 2024-09-21T13:57:06.447 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.447 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.447 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:57:06.447 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.700245+0000", 2024-09-21T13:57:06.447 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000115047 2024-09-21T13:57:06.447 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.447 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.447 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3", 2024-09-21T13:57:06.447 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.700392+0000", 2024-09-21T13:57:06.448 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000146358 2024-09-21T13:57:06.448 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.448 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.448 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:57:06.448 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.704100+0000", 2024-09-21T13:57:06.448 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0037085780000000001 2024-09-21T13:57:06.448 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.448 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.448 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:06.448 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.706575+0000", 2024-09-21T13:57:06.448 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.002474642 2024-09-21T13:57:06.448 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.449 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.449 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:06.449 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.708441+0000", 2024-09-21T13:57:06.449 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00186568 2024-09-21T13:57:06.449 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.449 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.449 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:57:06.449 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.708496+0000", 2024-09-21T13:57:06.449 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5340000000000002e-05 2024-09-21T13:57:06.450 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.450 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.450 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:57:06.450 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.708539+0000", 2024-09-21T13:57:06.450 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2920999999999998e-05 2024-09-21T13:57:06.451 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:57:06.451 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:57:06.451 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:57:06.451 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.451 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.451 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:35272 7.6 7:61fa97ba:::benchmark_data_smithi049_76798_object35271:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T13:57:06.451 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:28.712676+0000", 2024-09-21T13:57:06.451 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 397.72092281400001, 2024-09-21T13:57:06.451 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.99907040300000005, 2024-09-21T13:57:06.451 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:57:06.451 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:57:06.451 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:06.451 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:57:06.451 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:57:06.452 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:57:06.452 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 35272 2024-09-21T13:57:06.452 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.452 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:57:06.452 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.452 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:57:06.452 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.712676+0000", 2024-09-21T13:57:06.452 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:57:06.452 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.452 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.452 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:57:06.452 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.712676+0000", 2024-09-21T13:57:06.452 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:57:06.452 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.453 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.453 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:57:06.453 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.712678+0000", 2024-09-21T13:57:06.453 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.401e-06 2024-09-21T13:57:06.453 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.453 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.453 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:57:06.453 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.712724+0000", 2024-09-21T13:57:06.453 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6576999999999998e-05 2024-09-21T13:57:06.453 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.453 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.453 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:57:06.453 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.712736+0000", 2024-09-21T13:57:06.453 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2231e-05 2024-09-21T13:57:06.454 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.454 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.454 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:57:06.454 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.712742+0000", 2024-09-21T13:57:06.454 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9649999999999996e-06 2024-09-21T13:57:06.454 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.454 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.454 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:57:06.454 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.712819+0000", 2024-09-21T13:57:06.454 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6258000000000001e-05 2024-09-21T13:57:06.454 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.454 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.455 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:57:06.455 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.712831+0000", 2024-09-21T13:57:06.455 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.275e-05 2024-09-21T13:57:06.455 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.455 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.455 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:57:06.455 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.701685+0000", 2024-09-21T13:57:06.455 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.98885339699999997 2024-09-21T13:57:06.455 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.455 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.455 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:57:06.455 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.701779+0000", 2024-09-21T13:57:06.456 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3944999999999996e-05 2024-09-21T13:57:06.456 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.456 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.456 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3", 2024-09-21T13:57:06.456 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.701912+0000", 2024-09-21T13:57:06.456 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000133156 2024-09-21T13:57:06.456 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.456 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.456 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:57:06.456 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.705423+0000", 2024-09-21T13:57:06.456 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003511587 2024-09-21T13:57:06.457 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.457 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.457 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:06.457 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.708588+0000", 2024-09-21T13:57:06.457 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0031643309999999998 2024-09-21T13:57:06.457 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.457 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.457 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:06.457 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.711670+0000", 2024-09-21T13:57:06.457 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003081748 2024-09-21T13:57:06.457 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.458 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.458 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:57:06.458 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.711714+0000", 2024-09-21T13:57:06.458 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4406999999999997e-05 2024-09-21T13:57:06.458 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.458 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.458 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:57:06.458 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.711747+0000", 2024-09-21T13:57:06.458 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2650000000000001e-05 2024-09-21T13:57:06.458 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:57:06.458 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:57:06.458 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:57:06.459 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.459 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.459 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:35278 7.6 7:6d0c33e9:::benchmark_data_smithi049_76798_object35277:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T13:57:06.459 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:28.718784+0000", 2024-09-21T13:57:06.459 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 397.71481465400001, 2024-09-21T13:57:06.459 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.993077243, 2024-09-21T13:57:06.459 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:57:06.459 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:57:06.459 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:06.459 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:57:06.459 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:57:06.459 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:57:06.459 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 35278 2024-09-21T13:57:06.459 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.459 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:57:06.460 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.460 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:57:06.460 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.718784+0000", 2024-09-21T13:57:06.460 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:57:06.460 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.460 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.460 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:57:06.460 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.718784+0000", 2024-09-21T13:57:06.460 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:57:06.460 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.460 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.460 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:57:06.460 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.718787+0000", 2024-09-21T13:57:06.460 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1849999999999998e-06 2024-09-21T13:57:06.461 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.461 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.461 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:57:06.461 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.718859+0000", 2024-09-21T13:57:06.461 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2344999999999999e-05 2024-09-21T13:57:06.461 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.461 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.461 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:57:06.461 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.718872+0000", 2024-09-21T13:57:06.461 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3113e-05 2024-09-21T13:57:06.461 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.461 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.461 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:57:06.461 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.718881+0000", 2024-09-21T13:57:06.462 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.448e-06 2024-09-21T13:57:06.462 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.462 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.462 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:57:06.462 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.718941+0000", 2024-09-21T13:57:06.462 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9580999999999998e-05 2024-09-21T13:57:06.462 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.462 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.462 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:57:06.462 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.718948+0000", 2024-09-21T13:57:06.462 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.391e-06 2024-09-21T13:57:06.462 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.462 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.462 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:57:06.463 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.703121+0000", 2024-09-21T13:57:06.463 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.98417250099999998 2024-09-21T13:57:06.463 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.463 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.463 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:57:06.463 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.703248+0000", 2024-09-21T13:57:06.463 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012707800000000001 2024-09-21T13:57:06.463 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.463 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.463 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3", 2024-09-21T13:57:06.463 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.703345+0000", 2024-09-21T13:57:06.463 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7287999999999994e-05 2024-09-21T13:57:06.463 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.463 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.464 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:57:06.464 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.705432+0000", 2024-09-21T13:57:06.464 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0020863499999999998 2024-09-21T13:57:06.464 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.464 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.464 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:06.464 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.708649+0000", 2024-09-21T13:57:06.464 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003217443 2024-09-21T13:57:06.464 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.464 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.464 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:06.464 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.711798+0000", 2024-09-21T13:57:06.464 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0031489809999999999 2024-09-21T13:57:06.464 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.464 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.465 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:57:06.465 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.711832+0000", 2024-09-21T13:57:06.465 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3707000000000001e-05 2024-09-21T13:57:06.465 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.465 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.465 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:57:06.465 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.711862+0000", 2024-09-21T13:57:06.465 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9832000000000002e-05 2024-09-21T13:57:06.465 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:57:06.465 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:57:06.465 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:57:06.465 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.465 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.465 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:35280 7.6 7:68737c4e:::benchmark_data_smithi049_76798_object35279:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T13:57:06.466 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:28.719589+0000", 2024-09-21T13:57:06.466 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 397.71401019199999, 2024-09-21T13:57:06.466 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.99249087499999999, 2024-09-21T13:57:06.466 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:57:06.466 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:57:06.466 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:06.466 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:57:06.466 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:57:06.466 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:57:06.466 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 35280 2024-09-21T13:57:06.467 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.467 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:57:06.467 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.467 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:57:06.467 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.719589+0000", 2024-09-21T13:57:06.467 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:57:06.467 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.467 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.467 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:57:06.467 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.719589+0000", 2024-09-21T13:57:06.467 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:57:06.467 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.467 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.468 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:57:06.468 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.719590+0000", 2024-09-21T13:57:06.468 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.5600000000000004e-07 2024-09-21T13:57:06.468 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.468 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.468 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:57:06.468 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.719649+0000", 2024-09-21T13:57:06.468 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8730999999999997e-05 2024-09-21T13:57:06.468 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.468 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.468 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:57:06.468 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.719659+0000", 2024-09-21T13:57:06.468 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0498e-05 2024-09-21T13:57:06.468 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.469 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.469 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:57:06.469 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.719666+0000", 2024-09-21T13:57:06.469 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9029999999999998e-06 2024-09-21T13:57:06.469 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.469 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.469 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:57:06.469 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.719710+0000", 2024-09-21T13:57:06.469 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3859000000000001e-05 2024-09-21T13:57:06.469 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.469 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.469 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:57:06.469 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.719715+0000", 2024-09-21T13:57:06.469 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2229999999999998e-06 2024-09-21T13:57:06.470 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.470 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.470 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:57:06.470 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.704132+0000", 2024-09-21T13:57:06.470 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.98441694300000004 2024-09-21T13:57:06.470 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.470 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.470 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:57:06.470 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.704260+0000", 2024-09-21T13:57:06.470 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012844899999999999 2024-09-21T13:57:06.470 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.470 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.470 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3", 2024-09-21T13:57:06.471 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.704400+0000", 2024-09-21T13:57:06.471 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013959700000000001 2024-09-21T13:57:06.471 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.471 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.471 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:57:06.471 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.706512+0000", 2024-09-21T13:57:06.471 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0021123270000000002 2024-09-21T13:57:06.471 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.471 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.471 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:06.471 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.708683+0000", 2024-09-21T13:57:06.471 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00217022 2024-09-21T13:57:06.471 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.471 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.472 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:06.472 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.712030+0000", 2024-09-21T13:57:06.472 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0033470459999999998 2024-09-21T13:57:06.472 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.472 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.472 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:57:06.472 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.712058+0000", 2024-09-21T13:57:06.472 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8586e-05 2024-09-21T13:57:06.472 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.472 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.472 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:57:06.472 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.712080+0000", 2024-09-21T13:57:06.472 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1537000000000001e-05 2024-09-21T13:57:06.472 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:57:06.472 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:57:06.473 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:57:06.473 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.473 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.473 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:201635 7.5 7:a1b75091:::benchmark_data_smithi049_76798_object65215:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T13:57:06.473 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:43.011858+0000", 2024-09-21T13:57:06.473 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 263.42174151900002, 2024-09-21T13:57:06.473 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.96906032200000003, 2024-09-21T13:57:06.473 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:57:06.473 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:57:06.473 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:06.473 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:57:06.473 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:57:06.473 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:57:06.473 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 201635 2024-09-21T13:57:06.474 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.474 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:57:06.474 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.474 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:57:06.474 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.011858+0000", 2024-09-21T13:57:06.474 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:57:06.474 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.474 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.474 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:57:06.474 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.011858+0000", 2024-09-21T13:57:06.474 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:57:06.474 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.474 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.474 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:57:06.475 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.011859+0000", 2024-09-21T13:57:06.475 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.609e-06 2024-09-21T13:57:06.475 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.475 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.475 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:57:06.475 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.011865+0000", 2024-09-21T13:57:06.475 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2489999999999998e-06 2024-09-21T13:57:06.475 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.475 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.475 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:57:06.475 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.011880+0000", 2024-09-21T13:57:06.475 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4141999999999999e-05 2024-09-21T13:57:06.475 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.475 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.475 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:57:06.476 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.011887+0000", 2024-09-21T13:57:06.476 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4830000000000002e-06 2024-09-21T13:57:06.476 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.476 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.476 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:57:06.476 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.011963+0000", 2024-09-21T13:57:06.476 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5624999999999995e-05 2024-09-21T13:57:06.476 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.476 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.476 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:57:06.476 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.011971+0000", 2024-09-21T13:57:06.476 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.541e-06 2024-09-21T13:57:06.476 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.476 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.477 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:57:06.477 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.957601+0000", 2024-09-21T13:57:06.477 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.94562941099999998 2024-09-21T13:57:06.477 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.477 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.477 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:57:06.477 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.957730+0000", 2024-09-21T13:57:06.477 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000129016 2024-09-21T13:57:06.477 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.477 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.478 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2", 2024-09-21T13:57:06.478 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.957877+0000", 2024-09-21T13:57:06.478 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000147563 2024-09-21T13:57:06.478 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.478 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.478 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:57:06.478 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.962710+0000", 2024-09-21T13:57:06.478 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.004832355 2024-09-21T13:57:06.478 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.478 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.478 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:06.478 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.965643+0000", 2024-09-21T13:57:06.478 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.002933017 2024-09-21T13:57:06.478 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.479 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.479 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:06.479 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.980821+0000", 2024-09-21T13:57:06.479 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.015178552999999999 2024-09-21T13:57:06.479 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.479 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.479 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:57:06.479 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.980880+0000", 2024-09-21T13:57:06.479 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8646000000000001e-05 2024-09-21T13:57:06.479 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.479 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.479 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:57:06.479 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.980918+0000", 2024-09-21T13:57:06.479 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8111999999999997e-05 2024-09-21T13:57:06.480 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:57:06.480 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:57:06.480 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:57:06.480 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.480 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.480 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:201680 7.5 7:ac2e5a95:::benchmark_data_smithi049_76798_object65260:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T13:57:06.480 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:43.053233+0000", 2024-09-21T13:57:06.480 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 263.38036606399999, 2024-09-21T13:57:06.480 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.92777444499999995, 2024-09-21T13:57:06.480 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:57:06.480 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:57:06.480 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:06.480 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:57:06.480 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:57:06.481 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:57:06.481 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 201680 2024-09-21T13:57:06.481 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.481 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:57:06.481 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.481 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:57:06.481 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.053233+0000", 2024-09-21T13:57:06.481 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:57:06.481 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.481 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.481 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:57:06.481 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.053233+0000", 2024-09-21T13:57:06.481 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:57:06.481 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.482 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.482 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:57:06.482 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.053249+0000", 2024-09-21T13:57:06.482 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.596e-05 2024-09-21T13:57:06.482 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.482 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.482 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:57:06.482 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.053257+0000", 2024-09-21T13:57:06.482 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9980000000000003e-06 2024-09-21T13:57:06.482 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.482 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.482 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:57:06.482 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.053278+0000", 2024-09-21T13:57:06.482 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0749000000000001e-05 2024-09-21T13:57:06.483 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.483 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.483 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:57:06.483 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.053289+0000", 2024-09-21T13:57:06.483 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0864e-05 2024-09-21T13:57:06.483 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.483 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.483 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:57:06.483 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.053366+0000", 2024-09-21T13:57:06.483 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7082999999999994e-05 2024-09-21T13:57:06.483 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.484 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.484 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:57:06.484 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.053375+0000", 2024-09-21T13:57:06.484 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3230000000000004e-06 2024-09-21T13:57:06.484 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.484 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.484 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:57:06.484 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.960197+0000", 2024-09-21T13:57:06.484 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.90682242700000004 2024-09-21T13:57:06.484 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.484 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.484 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:57:06.484 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.960297+0000", 2024-09-21T13:57:06.484 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.9817999999999999e-05 2024-09-21T13:57:06.485 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.485 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.485 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2", 2024-09-21T13:57:06.485 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.960410+0000", 2024-09-21T13:57:06.485 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000113315 2024-09-21T13:57:06.485 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.485 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.485 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:57:06.485 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.965602+0000", 2024-09-21T13:57:06.485 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0051915390000000002 2024-09-21T13:57:06.485 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.485 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.485 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:06.486 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.965712+0000", 2024-09-21T13:57:06.486 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010966700000000001 2024-09-21T13:57:06.486 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.486 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.486 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:06.486 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.980954+0000", 2024-09-21T13:57:06.486 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.015242551999999999 2024-09-21T13:57:06.486 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.486 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.486 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:57:06.486 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.980985+0000", 2024-09-21T13:57:06.486 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0686999999999999e-05 2024-09-21T13:57:06.486 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.487 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.487 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:57:06.487 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.981007+0000", 2024-09-21T13:57:06.487 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2463000000000002e-05 2024-09-21T13:57:06.487 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:57:06.487 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:57:06.487 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:57:06.487 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.487 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.487 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:201691 7.5 7:a0c84d4e:::benchmark_data_smithi049_76798_object65271:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T13:57:06.487 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:43.063956+0000", 2024-09-21T13:57:06.487 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 263.36964257400001, 2024-09-21T13:57:06.488 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.91993442199999997, 2024-09-21T13:57:06.488 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:57:06.488 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:57:06.488 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:06.488 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:57:06.488 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:57:06.488 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:57:06.488 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 201691 2024-09-21T13:57:06.488 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.488 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:57:06.489 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.489 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:57:06.489 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.063956+0000", 2024-09-21T13:57:06.489 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:57:06.489 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.489 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.489 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:57:06.489 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.063956+0000", 2024-09-21T13:57:06.489 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:57:06.489 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.489 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.489 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:57:06.489 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.063959+0000", 2024-09-21T13:57:06.489 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.401e-06 2024-09-21T13:57:06.489 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.490 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.490 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:57:06.490 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.063966+0000", 2024-09-21T13:57:06.490 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2479999999999999e-06 2024-09-21T13:57:06.490 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.490 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.490 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:57:06.490 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.063984+0000", 2024-09-21T13:57:06.490 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7530999999999999e-05 2024-09-21T13:57:06.490 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.490 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.490 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:57:06.490 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.063992+0000", 2024-09-21T13:57:06.491 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2509999999999994e-06 2024-09-21T13:57:06.491 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.491 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.491 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:57:06.491 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.064060+0000", 2024-09-21T13:57:06.491 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8535999999999996e-05 2024-09-21T13:57:06.491 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.491 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.491 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:57:06.491 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.064070+0000", 2024-09-21T13:57:06.491 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6830000000000003e-06 2024-09-21T13:57:06.491 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.491 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.491 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:57:06.492 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.961470+0000", 2024-09-21T13:57:06.492 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.89740008599999999 2024-09-21T13:57:06.492 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.492 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.492 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:57:06.492 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.961570+0000", 2024-09-21T13:57:06.492 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000100237 2024-09-21T13:57:06.492 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.492 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.492 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2", 2024-09-21T13:57:06.492 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.961689+0000", 2024-09-21T13:57:06.492 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000118347 2024-09-21T13:57:06.492 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.492 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.493 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:57:06.493 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.965656+0000", 2024-09-21T13:57:06.493 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0039668029999999996 2024-09-21T13:57:06.493 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.493 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.494 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:06.494 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.979772+0000", 2024-09-21T13:57:06.494 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.014116717000000001 2024-09-21T13:57:06.494 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.494 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.494 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:06.495 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.983830+0000", 2024-09-21T13:57:06.495 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0040575189999999999 2024-09-21T13:57:06.495 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.495 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.495 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:57:06.495 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.983868+0000", 2024-09-21T13:57:06.495 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8586e-05 2024-09-21T13:57:06.495 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.495 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.495 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:57:06.495 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.983891+0000", 2024-09-21T13:57:06.495 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2476999999999998e-05 2024-09-21T13:57:06.495 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:57:06.495 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:57:06.495 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:57:06.496 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.496 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.496 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:201695 7.5 7:a9f46a76:::benchmark_data_smithi049_76798_object65275:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T13:57:06.496 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:43.066285+0000", 2024-09-21T13:57:06.496 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 263.36731444700001, 2024-09-21T13:57:06.496 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.92214624000000001, 2024-09-21T13:57:06.496 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:57:06.496 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:57:06.496 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:06.496 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:57:06.496 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:57:06.496 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:57:06.496 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 201695 2024-09-21T13:57:06.496 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.497 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:57:06.497 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.497 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:57:06.497 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.066285+0000", 2024-09-21T13:57:06.497 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:57:06.497 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.497 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.497 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:57:06.497 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.066285+0000", 2024-09-21T13:57:06.497 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:57:06.497 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.497 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.497 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:57:06.497 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.066287+0000", 2024-09-21T13:57:06.497 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0219999999999999e-06 2024-09-21T13:57:06.498 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.498 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.498 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:57:06.498 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.066295+0000", 2024-09-21T13:57:06.498 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1999999999999994e-06 2024-09-21T13:57:06.498 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.498 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.498 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:57:06.498 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.066316+0000", 2024-09-21T13:57:06.499 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1508000000000001e-05 2024-09-21T13:57:06.499 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.499 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.499 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:57:06.499 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.066326+0000", 2024-09-21T13:57:06.499 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.4290000000000007e-06 2024-09-21T13:57:06.499 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.499 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.499 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:57:06.499 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.066400+0000", 2024-09-21T13:57:06.499 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4339999999999996e-05 2024-09-21T13:57:06.499 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.499 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.500 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:57:06.500 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.066410+0000", 2024-09-21T13:57:06.500 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.5799999999999998e-06 2024-09-21T13:57:06.500 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.500 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.500 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:57:06.500 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.962805+0000", 2024-09-21T13:57:06.500 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.89639528199999996 2024-09-21T13:57:06.500 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.500 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.500 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:57:06.500 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.962880+0000", 2024-09-21T13:57:06.500 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5139e-05 2024-09-21T13:57:06.500 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.501 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.501 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2", 2024-09-21T13:57:06.501 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.962949+0000", 2024-09-21T13:57:06.501 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9098999999999997e-05 2024-09-21T13:57:06.501 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.501 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.501 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:57:06.501 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.965662+0000", 2024-09-21T13:57:06.501 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0027127259999999999 2024-09-21T13:57:06.501 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.501 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.501 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:06.501 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.983932+0000", 2024-09-21T13:57:06.501 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.018270237000000002 2024-09-21T13:57:06.501 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.502 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.502 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:06.502 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.988374+0000", 2024-09-21T13:57:06.502 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0044419239999999999 2024-09-21T13:57:06.502 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.502 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.502 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:57:06.502 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.988406+0000", 2024-09-21T13:57:06.502 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2172000000000003e-05 2024-09-21T13:57:06.502 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.502 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.502 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:57:06.502 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.988431+0000", 2024-09-21T13:57:06.503 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4581999999999999e-05 2024-09-21T13:57:06.503 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:57:06.503 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:57:06.503 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:57:06.503 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.503 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.503 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:201698 7.5 7:ae6dee09:::benchmark_data_smithi049_76798_object65278:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T13:57:06.503 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:43.068553+0000", 2024-09-21T13:57:06.503 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 263.36504597499999, 2024-09-21T13:57:06.503 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.91996031099999998, 2024-09-21T13:57:06.503 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:57:06.503 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:57:06.503 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:06.503 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:57:06.503 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:57:06.504 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:57:06.504 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 201698 2024-09-21T13:57:06.504 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.504 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:57:06.504 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.504 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:57:06.504 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.068553+0000", 2024-09-21T13:57:06.504 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:57:06.504 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.504 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.504 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:57:06.504 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.068553+0000", 2024-09-21T13:57:06.504 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:57:06.504 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.505 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.505 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:57:06.505 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.068555+0000", 2024-09-21T13:57:06.505 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2840000000000001e-06 2024-09-21T13:57:06.505 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.505 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.505 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:57:06.505 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.068563+0000", 2024-09-21T13:57:06.505 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6899999999999992e-06 2024-09-21T13:57:06.505 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.505 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.505 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:57:06.505 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.068582+0000", 2024-09-21T13:57:06.505 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8902e-05 2024-09-21T13:57:06.506 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.506 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.506 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:57:06.506 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.068591+0000", 2024-09-21T13:57:06.506 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.6440000000000006e-06 2024-09-21T13:57:06.506 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.506 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.506 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:57:06.506 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.068654+0000", 2024-09-21T13:57:06.506 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.347e-05 2024-09-21T13:57:06.506 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.506 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.506 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:57:06.507 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.068662+0000", 2024-09-21T13:57:06.507 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1829999999999994e-06 2024-09-21T13:57:06.507 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.507 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.507 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:57:06.507 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.964312+0000", 2024-09-21T13:57:06.507 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.89564967699999998 2024-09-21T13:57:06.507 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.507 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.507 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:57:06.507 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.964413+0000", 2024-09-21T13:57:06.507 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010090399999999999 2024-09-21T13:57:06.507 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.507 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.508 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2", 2024-09-21T13:57:06.508 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.964529+0000", 2024-09-21T13:57:06.508 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000116579 2024-09-21T13:57:06.508 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.508 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.508 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:57:06.508 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.970760+0000", 2024-09-21T13:57:06.508 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0062309820000000004 2024-09-21T13:57:06.508 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.508 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.508 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:06.508 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.983968+0000", 2024-09-21T13:57:06.508 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.013207727000000001 2024-09-21T13:57:06.509 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.509 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.509 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:06.509 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.988459+0000", 2024-09-21T13:57:06.509 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0044908889999999996 2024-09-21T13:57:06.509 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.509 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.509 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:57:06.509 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.988489+0000", 2024-09-21T13:57:06.509 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0063e-05 2024-09-21T13:57:06.510 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.510 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.510 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:57:06.510 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.988513+0000", 2024-09-21T13:57:06.510 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4317000000000001e-05 2024-09-21T13:57:06.510 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:57:06.510 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:57:06.510 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:57:06.510 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.510 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.511 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:201701 7.5 7:a7fade2d:::benchmark_data_smithi049_76798_object65281:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T13:57:06.511 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:43.071918+0000", 2024-09-21T13:57:06.511 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 263.36168070899998, 2024-09-21T13:57:06.511 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.91666014399999995, 2024-09-21T13:57:06.511 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:57:06.511 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:57:06.511 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:06.512 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:57:06.512 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:57:06.512 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:57:06.512 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 201701 2024-09-21T13:57:06.512 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.512 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:57:06.512 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.512 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:57:06.512 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.071918+0000", 2024-09-21T13:57:06.512 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:57:06.512 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.513 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.513 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:57:06.513 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.071918+0000", 2024-09-21T13:57:06.513 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:57:06.513 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.513 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.513 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:57:06.513 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.071920+0000", 2024-09-21T13:57:06.513 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3340000000000001e-06 2024-09-21T13:57:06.513 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.513 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.513 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:57:06.513 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.071926+0000", 2024-09-21T13:57:06.513 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8719999999999999e-06 2024-09-21T13:57:06.514 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.514 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.514 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:57:06.514 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.071941+0000", 2024-09-21T13:57:06.514 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5529999999999999e-05 2024-09-21T13:57:06.514 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.514 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.514 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:57:06.514 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.071949+0000", 2024-09-21T13:57:06.514 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1000000000000004e-06 2024-09-21T13:57:06.514 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.514 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.515 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:57:06.515 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.072014+0000", 2024-09-21T13:57:06.515 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4912000000000005e-05 2024-09-21T13:57:06.515 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.515 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.515 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:57:06.515 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.072023+0000", 2024-09-21T13:57:06.515 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7069999999999998e-06 2024-09-21T13:57:06.515 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.515 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.515 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:57:06.515 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.965753+0000", 2024-09-21T13:57:06.515 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.89372980199999996 2024-09-21T13:57:06.515 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.516 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.516 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:57:06.516 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.965886+0000", 2024-09-21T13:57:06.516 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000133669 2024-09-21T13:57:06.516 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.516 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.516 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2", 2024-09-21T13:57:06.516 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.966006+0000", 2024-09-21T13:57:06.516 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000119407 2024-09-21T13:57:06.516 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.516 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.516 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:57:06.516 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.973164+0000", 2024-09-21T13:57:06.516 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.007158569 2024-09-21T13:57:06.516 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.517 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.517 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:06.517 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.984364+0000", 2024-09-21T13:57:06.517 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.011199944 2024-09-21T13:57:06.517 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.517 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.517 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:06.517 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.988535+0000", 2024-09-21T13:57:06.517 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0041708800000000001 2024-09-21T13:57:06.517 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.517 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.517 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:57:06.517 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.988560+0000", 2024-09-21T13:57:06.517 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4879e-05 2024-09-21T13:57:06.518 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.518 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.518 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:57:06.518 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.988578+0000", 2024-09-21T13:57:06.518 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8539e-05 2024-09-21T13:57:06.518 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:57:06.518 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:57:06.518 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:57:06.518 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.518 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.518 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:201772 7.5 7:a814840e:::benchmark_data_smithi049_76798_object65352:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T13:57:06.518 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:43.156794+0000", 2024-09-21T13:57:06.518 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 263.27680530100002, 2024-09-21T13:57:06.519 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.83184825500000004, 2024-09-21T13:57:06.519 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:57:06.519 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:57:06.519 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:06.519 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:57:06.519 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:57:06.519 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:57:06.519 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 201772 2024-09-21T13:57:06.519 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.519 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:57:06.519 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.521 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:57:06.521 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.156794+0000", 2024-09-21T13:57:06.521 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:57:06.521 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.522 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.522 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:57:06.522 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.156794+0000", 2024-09-21T13:57:06.522 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:57:06.522 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.522 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.522 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:57:06.522 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.156795+0000", 2024-09-21T13:57:06.522 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4839999999999999e-06 2024-09-21T13:57:06.522 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.522 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.522 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:57:06.522 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.156801+0000", 2024-09-21T13:57:06.523 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5709999999999997e-06 2024-09-21T13:57:06.523 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.523 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.523 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:57:06.523 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.156817+0000", 2024-09-21T13:57:06.523 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6378000000000001e-05 2024-09-21T13:57:06.523 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.523 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.523 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:57:06.524 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.156826+0000", 2024-09-21T13:57:06.524 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4060000000000005e-06 2024-09-21T13:57:06.524 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.524 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.524 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:57:06.524 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.156889+0000", 2024-09-21T13:57:06.524 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3396000000000001e-05 2024-09-21T13:57:06.524 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.524 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.525 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:57:06.525 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.156895+0000", 2024-09-21T13:57:06.525 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7699999999999998e-06 2024-09-21T13:57:06.525 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.525 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.525 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:57:06.525 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.967038+0000", 2024-09-21T13:57:06.525 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.81014350400000001 2024-09-21T13:57:06.525 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.525 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.525 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:57:06.525 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.967105+0000", 2024-09-21T13:57:06.525 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6561000000000002e-05 2024-09-21T13:57:06.526 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.526 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.526 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2", 2024-09-21T13:57:06.526 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.967172+0000", 2024-09-21T13:57:06.526 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7336999999999996e-05 2024-09-21T13:57:06.526 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.526 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.526 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:57:06.526 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.973169+0000", 2024-09-21T13:57:06.526 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.005996927 2024-09-21T13:57:06.526 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.526 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.527 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:06.527 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.984408+0000", 2024-09-21T13:57:06.527 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.011238823 2024-09-21T13:57:06.527 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.527 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.527 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:06.527 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.988599+0000", 2024-09-21T13:57:06.527 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0041907630000000001 2024-09-21T13:57:06.527 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.527 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.527 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:57:06.527 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.988624+0000", 2024-09-21T13:57:06.528 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5148999999999999e-05 2024-09-21T13:57:06.528 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.528 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.528 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:57:06.528 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.988642+0000", 2024-09-21T13:57:06.528 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8185999999999998e-05 2024-09-21T13:57:06.528 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:57:06.528 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:57:06.528 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:57:06.528 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.528 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.528 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:201787 7.5 7:a2e71a94:::benchmark_data_smithi049_76798_object65367:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T13:57:06.528 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:43.173880+0000", 2024-09-21T13:57:06.528 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 263.25971932499999, 2024-09-21T13:57:06.529 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.81499015100000005, 2024-09-21T13:57:06.529 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:57:06.529 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:57:06.529 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:06.529 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:57:06.529 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:57:06.529 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:57:06.529 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 201787 2024-09-21T13:57:06.529 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.529 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:57:06.529 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.529 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:57:06.529 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.173880+0000", 2024-09-21T13:57:06.530 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:57:06.530 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.530 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.530 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:57:06.530 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.173880+0000", 2024-09-21T13:57:06.530 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:57:06.530 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.530 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.530 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:57:06.530 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.173882+0000", 2024-09-21T13:57:06.530 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1500000000000002e-06 2024-09-21T13:57:06.530 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.530 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.531 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:57:06.531 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.173888+0000", 2024-09-21T13:57:06.531 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3799999999999999e-06 2024-09-21T13:57:06.531 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.531 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.531 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:57:06.531 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.173902+0000", 2024-09-21T13:57:06.531 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4225e-05 2024-09-21T13:57:06.531 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.531 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.531 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:57:06.531 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.173911+0000", 2024-09-21T13:57:06.531 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0649999999999999e-06 2024-09-21T13:57:06.532 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.532 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.532 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:57:06.532 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.173972+0000", 2024-09-21T13:57:06.532 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1168999999999997e-05 2024-09-21T13:57:06.532 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.532 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.532 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:57:06.532 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.173980+0000", 2024-09-21T13:57:06.532 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2549999999999997e-06 2024-09-21T13:57:06.532 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.532 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.532 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:57:06.533 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.968330+0000", 2024-09-21T13:57:06.533 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.79435025800000003 2024-09-21T13:57:06.533 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.533 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.533 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:57:06.533 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.968390+0000", 2024-09-21T13:57:06.533 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0149999999999998e-05 2024-09-21T13:57:06.533 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.533 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.533 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2", 2024-09-21T13:57:06.533 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.968453+0000", 2024-09-21T13:57:06.533 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2427000000000004e-05 2024-09-21T13:57:06.534 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.534 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.534 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:57:06.534 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.973174+0000", 2024-09-21T13:57:06.534 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0047213740000000004 2024-09-21T13:57:06.534 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.534 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.534 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:06.534 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.984437+0000", 2024-09-21T13:57:06.534 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.011262507 2024-09-21T13:57:06.534 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.534 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.534 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:06.535 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.988823+0000", 2024-09-21T13:57:06.535 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0043867799999999998 2024-09-21T13:57:06.535 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.535 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.535 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:57:06.535 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.988853+0000", 2024-09-21T13:57:06.535 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9116e-05 2024-09-21T13:57:06.535 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.535 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.535 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:57:06.535 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.988870+0000", 2024-09-21T13:57:06.535 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7295e-05 2024-09-21T13:57:06.535 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:57:06.535 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:57:06.536 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:57:06.536 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.536 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.536 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:201826 7.5 7:a523bf31:::benchmark_data_smithi049_76798_object65406:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T13:57:06.536 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:43.212124+0000", 2024-09-21T13:57:06.536 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 263.22147519599997, 2024-09-21T13:57:06.536 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.77941543700000004, 2024-09-21T13:57:06.536 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:57:06.536 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:57:06.536 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:06.536 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:57:06.536 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:57:06.536 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:57:06.536 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 201826 2024-09-21T13:57:06.537 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.537 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:57:06.537 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.537 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:57:06.537 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.212124+0000", 2024-09-21T13:57:06.537 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:57:06.537 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.537 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.537 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:57:06.537 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.212124+0000", 2024-09-21T13:57:06.537 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:57:06.537 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.537 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.537 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:57:06.537 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.212125+0000", 2024-09-21T13:57:06.538 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3200000000000001e-06 2024-09-21T13:57:06.538 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.538 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.538 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:57:06.538 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.212130+0000", 2024-09-21T13:57:06.538 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5210000000000004e-06 2024-09-21T13:57:06.538 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.538 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.538 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:57:06.538 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.212142+0000", 2024-09-21T13:57:06.538 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1878e-05 2024-09-21T13:57:06.538 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.538 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.539 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:57:06.539 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.212147+0000", 2024-09-21T13:57:06.539 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3410000000000002e-06 2024-09-21T13:57:06.539 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.539 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.539 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:57:06.539 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.212194+0000", 2024-09-21T13:57:06.539 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7091e-05 2024-09-21T13:57:06.539 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.539 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.539 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:57:06.539 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.212200+0000", 2024-09-21T13:57:06.539 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0440000000000001e-06 2024-09-21T13:57:06.540 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.540 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.540 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:57:06.540 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.969460+0000", 2024-09-21T13:57:06.540 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.75725952399999996 2024-09-21T13:57:06.540 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.540 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.540 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:57:06.540 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.969570+0000", 2024-09-21T13:57:06.540 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000110876 2024-09-21T13:57:06.540 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.540 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.540 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2", 2024-09-21T13:57:06.540 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.969687+0000", 2024-09-21T13:57:06.541 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011697 2024-09-21T13:57:06.541 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.541 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.541 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:57:06.541 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.974455+0000", 2024-09-21T13:57:06.541 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0047673259999999997 2024-09-21T13:57:06.541 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.541 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.541 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:06.541 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.990260+0000", 2024-09-21T13:57:06.541 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.015805567999999999 2024-09-21T13:57:06.541 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.541 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.541 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:06.542 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.991462+0000", 2024-09-21T13:57:06.542 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0012021340000000001 2024-09-21T13:57:06.542 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.542 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.542 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:57:06.542 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.991504+0000", 2024-09-21T13:57:06.542 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1715000000000003e-05 2024-09-21T13:57:06.542 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.542 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.542 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:57:06.542 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.991539+0000", 2024-09-21T13:57:06.542 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5129000000000002e-05 2024-09-21T13:57:06.543 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:57:06.543 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:57:06.543 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:57:06.543 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.543 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.543 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:201840 7.5 7:abeb3043:::benchmark_data_smithi049_76798_object65420:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T13:57:06.543 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:43.224752+0000", 2024-09-21T13:57:06.543 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 263.20884689600001, 2024-09-21T13:57:06.543 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.76862761999999996, 2024-09-21T13:57:06.543 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:57:06.543 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:57:06.543 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:06.543 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:57:06.544 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:57:06.544 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:57:06.544 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 201840 2024-09-21T13:57:06.544 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.544 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:57:06.544 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.544 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:57:06.544 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.224752+0000", 2024-09-21T13:57:06.544 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:57:06.544 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.544 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.544 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:57:06.544 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.224752+0000", 2024-09-21T13:57:06.544 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:57:06.545 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.545 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.545 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:57:06.545 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.224753+0000", 2024-09-21T13:57:06.545 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.8299999999999995e-07 2024-09-21T13:57:06.545 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.545 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.545 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:57:06.545 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.224757+0000", 2024-09-21T13:57:06.545 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2379999999999999e-06 2024-09-21T13:57:06.545 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.545 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.545 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:57:06.545 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.224768+0000", 2024-09-21T13:57:06.545 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0390999999999999e-05 2024-09-21T13:57:06.545 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.546 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.546 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:57:06.546 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.224775+0000", 2024-09-21T13:57:06.546 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9999999999999999e-06 2024-09-21T13:57:06.546 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.546 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.546 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:57:06.546 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.224823+0000", 2024-09-21T13:57:06.546 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8186999999999998e-05 2024-09-21T13:57:06.546 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.546 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.546 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:57:06.546 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.224830+0000", 2024-09-21T13:57:06.546 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6100000000000002e-06 2024-09-21T13:57:06.547 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.547 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.547 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:57:06.547 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.970787+0000", 2024-09-21T13:57:06.547 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.74595736499999998 2024-09-21T13:57:06.547 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.547 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.547 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:57:06.547 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.970866+0000", 2024-09-21T13:57:06.547 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8731000000000002e-05 2024-09-21T13:57:06.547 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.547 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.547 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2", 2024-09-21T13:57:06.547 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.970932+0000", 2024-09-21T13:57:06.548 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6031000000000005e-05 2024-09-21T13:57:06.548 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.548 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.548 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:57:06.548 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.974460+0000", 2024-09-21T13:57:06.548 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0035279069999999998 2024-09-21T13:57:06.548 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.548 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.548 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:06.548 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.991576+0000", 2024-09-21T13:57:06.548 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.017116833000000001 2024-09-21T13:57:06.548 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.548 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.548 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:06.548 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.993291+0000", 2024-09-21T13:57:06.549 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0017143690000000001 2024-09-21T13:57:06.549 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.549 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.549 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:57:06.549 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.993342+0000", 2024-09-21T13:57:06.549 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1316000000000001e-05 2024-09-21T13:57:06.549 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.549 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.549 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:57:06.549 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.993380+0000", 2024-09-21T13:57:06.549 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7659e-05 2024-09-21T13:57:06.549 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:57:06.549 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:57:06.549 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:57:06.550 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.550 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.550 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:201850 7.5 7:adb529a4:::benchmark_data_smithi049_76798_object65430:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T13:57:06.550 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:43.237139+0000", 2024-09-21T13:57:06.550 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 263.19646018899999, 2024-09-21T13:57:06.550 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.76209282499999997, 2024-09-21T13:57:06.550 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:57:06.550 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:57:06.550 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:06.550 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:57:06.550 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:57:06.550 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:57:06.550 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 201850 2024-09-21T13:57:06.551 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.551 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:57:06.551 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.551 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:57:06.551 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.237139+0000", 2024-09-21T13:57:06.551 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:57:06.551 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.551 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.551 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:57:06.551 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.237139+0000", 2024-09-21T13:57:06.551 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:57:06.551 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.551 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.551 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:57:06.552 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.237139+0000", 2024-09-21T13:57:06.552 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5799999999999999e-07 2024-09-21T13:57:06.552 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.552 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.552 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:57:06.552 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.237143+0000", 2024-09-21T13:57:06.552 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5310000000000002e-06 2024-09-21T13:57:06.552 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.552 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.552 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:57:06.552 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.237152+0000", 2024-09-21T13:57:06.552 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.653e-06 2024-09-21T13:57:06.552 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.553 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.553 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:57:06.553 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.237157+0000", 2024-09-21T13:57:06.553 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8239999999999999e-06 2024-09-21T13:57:06.553 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.553 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.553 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:57:06.553 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.237205+0000", 2024-09-21T13:57:06.553 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8341999999999999e-05 2024-09-21T13:57:06.553 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.553 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.553 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:57:06.553 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.237212+0000", 2024-09-21T13:57:06.554 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6410000000000002e-06 2024-09-21T13:57:06.554 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.554 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.554 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:57:06.554 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.971939+0000", 2024-09-21T13:57:06.554 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.73472659200000001 2024-09-21T13:57:06.554 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.554 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.554 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:57:06.554 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.972042+0000", 2024-09-21T13:57:06.554 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000103049 2024-09-21T13:57:06.554 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.554 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.555 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2", 2024-09-21T13:57:06.555 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.972158+0000", 2024-09-21T13:57:06.555 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011618 2024-09-21T13:57:06.555 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.555 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.555 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:57:06.555 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.975735+0000", 2024-09-21T13:57:06.555 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0035764249999999998 2024-09-21T13:57:06.555 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.555 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.555 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:06.555 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.991609+0000", 2024-09-21T13:57:06.555 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.015874307000000001 2024-09-21T13:57:06.555 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.555 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.556 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:06.556 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.999171+0000", 2024-09-21T13:57:06.556 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0075622440000000001 2024-09-21T13:57:06.556 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.556 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.556 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:57:06.556 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.999201+0000", 2024-09-21T13:57:06.556 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9495000000000001e-05 2024-09-21T13:57:06.556 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.556 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.556 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:57:06.556 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.999232+0000", 2024-09-21T13:57:06.556 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0883999999999998e-05 2024-09-21T13:57:06.557 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:57:06.557 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:57:06.557 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:57:06.557 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.557 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.557 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:201852 7.5 7:a4c0c63b:::benchmark_data_smithi049_76798_object65432:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T13:57:06.557 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:43.239197+0000", 2024-09-21T13:57:06.557 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 263.19440189900001, 2024-09-21T13:57:06.557 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.76675081499999997, 2024-09-21T13:57:06.557 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:57:06.557 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:57:06.557 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:06.558 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:57:06.558 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:57:06.558 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:57:06.558 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 201852 2024-09-21T13:57:06.558 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.558 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:57:06.558 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.558 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:57:06.558 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.239197+0000", 2024-09-21T13:57:06.558 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:57:06.558 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.558 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.558 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:57:06.559 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.239197+0000", 2024-09-21T13:57:06.559 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:57:06.559 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.559 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.559 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:57:06.559 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.239198+0000", 2024-09-21T13:57:06.559 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6899999999999996e-07 2024-09-21T13:57:06.559 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.559 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.559 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:57:06.559 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.239202+0000", 2024-09-21T13:57:06.559 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8340000000000001e-06 2024-09-21T13:57:06.559 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.559 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.560 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:57:06.560 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.239212+0000", 2024-09-21T13:57:06.560 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0563e-05 2024-09-21T13:57:06.560 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.560 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.560 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:57:06.560 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.239219+0000", 2024-09-21T13:57:06.560 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.241e-06 2024-09-21T13:57:06.560 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.560 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.560 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:57:06.560 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.239260+0000", 2024-09-21T13:57:06.560 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1421999999999997e-05 2024-09-21T13:57:06.560 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.561 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.561 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:57:06.561 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.239265+0000", 2024-09-21T13:57:06.561 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5399999999999997e-06 2024-09-21T13:57:06.561 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.561 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.561 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:57:06.561 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.973203+0000", 2024-09-21T13:57:06.561 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.73393844600000002 2024-09-21T13:57:06.561 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.561 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.561 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:57:06.561 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.973305+0000", 2024-09-21T13:57:06.561 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010141699999999999 2024-09-21T13:57:06.562 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.562 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.562 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2", 2024-09-21T13:57:06.562 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.973424+0000", 2024-09-21T13:57:06.562 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011916700000000001 2024-09-21T13:57:06.562 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.562 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.562 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:57:06.562 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.977041+0000", 2024-09-21T13:57:06.562 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0036168989999999998 2024-09-21T13:57:06.562 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.562 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.562 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:06.563 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.996582+0000", 2024-09-21T13:57:06.563 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.019541097 2024-09-21T13:57:06.563 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.563 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.563 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:06.563 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.005893+0000", 2024-09-21T13:57:06.563 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0093108410000000003 2024-09-21T13:57:06.563 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.563 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.563 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:57:06.563 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.005925+0000", 2024-09-21T13:57:06.563 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2490000000000002e-05 2024-09-21T13:57:06.563 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.563 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.564 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:57:06.564 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.005948+0000", 2024-09-21T13:57:06.564 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2889000000000001e-05 2024-09-21T13:57:06.564 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:57:06.564 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:57:06.564 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:57:06.564 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.564 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.564 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:201856 7.5 7:a20bd2e0:::benchmark_data_smithi049_76798_object65436:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T13:57:06.564 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:43.241762+0000", 2024-09-21T13:57:06.564 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 263.191836577, 2024-09-21T13:57:06.564 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.76427266500000002, 2024-09-21T13:57:06.564 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:57:06.564 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:57:06.565 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:06.565 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:57:06.565 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:57:06.565 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:57:06.565 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 201856 2024-09-21T13:57:06.565 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.565 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:57:06.565 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.565 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:57:06.565 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.241762+0000", 2024-09-21T13:57:06.565 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:57:06.565 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.565 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.565 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:57:06.565 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.241762+0000", 2024-09-21T13:57:06.566 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:57:06.566 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.566 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.566 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:57:06.566 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.241763+0000", 2024-09-21T13:57:06.566 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.2200000000000002e-07 2024-09-21T13:57:06.566 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.566 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.566 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:57:06.566 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.241768+0000", 2024-09-21T13:57:06.566 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.357e-06 2024-09-21T13:57:06.566 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.566 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.566 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:57:06.567 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.241780+0000", 2024-09-21T13:57:06.567 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.224e-05 2024-09-21T13:57:06.567 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.567 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.567 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:57:06.567 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.241787+0000", 2024-09-21T13:57:06.567 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4710000000000001e-06 2024-09-21T13:57:06.567 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.567 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.567 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:57:06.567 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.241830+0000", 2024-09-21T13:57:06.567 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2960000000000002e-05 2024-09-21T13:57:06.567 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.567 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.568 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:57:06.568 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.241835+0000", 2024-09-21T13:57:06.568 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8110000000000002e-06 2024-09-21T13:57:06.568 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.568 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.568 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:57:06.568 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.974488+0000", 2024-09-21T13:57:06.568 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.73265256999999995 2024-09-21T13:57:06.568 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.568 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.568 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:57:06.568 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.974586+0000", 2024-09-21T13:57:06.568 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.8669e-05 2024-09-21T13:57:06.568 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.568 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.569 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2", 2024-09-21T13:57:06.569 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.974702+0000", 2024-09-21T13:57:06.569 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000115183 2024-09-21T13:57:06.569 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.569 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.569 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:57:06.569 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.979708+0000", 2024-09-21T13:57:06.569 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0050061339999999998 2024-09-21T13:57:06.569 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.569 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.569 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:06.569 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.997392+0000", 2024-09-21T13:57:06.569 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.017684388999999998 2024-09-21T13:57:06.569 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.570 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.570 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:06.570 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.005982+0000", 2024-09-21T13:57:06.570 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.008590192 2024-09-21T13:57:06.570 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.570 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.570 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:57:06.570 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.006012+0000", 2024-09-21T13:57:06.570 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9714e-05 2024-09-21T13:57:06.570 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.570 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.571 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:57:06.571 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.006035+0000", 2024-09-21T13:57:06.571 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3053000000000001e-05 2024-09-21T13:57:06.571 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:57:06.571 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:57:06.571 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:57:06.571 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.571 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.571 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:201866 7.5 7:ab0bbb06:::benchmark_data_smithi049_76798_object65446:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T13:57:06.571 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:43.261703+0000", 2024-09-21T13:57:06.571 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 263.171896022, 2024-09-21T13:57:06.571 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.744403649, 2024-09-21T13:57:06.571 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:57:06.571 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:57:06.572 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:06.572 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:57:06.572 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:57:06.572 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:57:06.572 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 201866 2024-09-21T13:57:06.572 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.572 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:57:06.572 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.572 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:57:06.572 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.261703+0000", 2024-09-21T13:57:06.572 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:57:06.572 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.572 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.572 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:57:06.572 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.261703+0000", 2024-09-21T13:57:06.573 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:57:06.573 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.573 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.573 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:57:06.573 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.261705+0000", 2024-09-21T13:57:06.573 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7290000000000001e-06 2024-09-21T13:57:06.573 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.573 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.573 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:57:06.573 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.261711+0000", 2024-09-21T13:57:06.573 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2319999999999998e-06 2024-09-21T13:57:06.573 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.573 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.573 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:57:06.574 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.261724+0000", 2024-09-21T13:57:06.574 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3375e-05 2024-09-21T13:57:06.574 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.574 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.574 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:57:06.574 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.261731+0000", 2024-09-21T13:57:06.574 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3319999999999997e-06 2024-09-21T13:57:06.574 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.574 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.574 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:57:06.574 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.261790+0000", 2024-09-21T13:57:06.574 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9110999999999997e-05 2024-09-21T13:57:06.574 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.574 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.575 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:57:06.575 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.261797+0000", 2024-09-21T13:57:06.575 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9739999999999996e-06 2024-09-21T13:57:06.575 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.575 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.575 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:57:06.575 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.975764+0000", 2024-09-21T13:57:06.575 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.71396768300000002 2024-09-21T13:57:06.575 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.575 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.575 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:57:06.575 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.975882+0000", 2024-09-21T13:57:06.575 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011723899999999999 2024-09-21T13:57:06.575 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.576 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.576 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2", 2024-09-21T13:57:06.576 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.976017+0000", 2024-09-21T13:57:06.576 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013532799999999999 2024-09-21T13:57:06.576 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.576 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.576 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:57:06.576 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.979731+0000", 2024-09-21T13:57:06.576 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0037143509999999999 2024-09-21T13:57:06.576 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.576 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.576 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:06.576 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.997419+0000", 2024-09-21T13:57:06.576 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.017688021000000002 2024-09-21T13:57:06.576 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.577 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.577 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:06.577 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.006063+0000", 2024-09-21T13:57:06.577 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0086435980000000006 2024-09-21T13:57:06.577 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.577 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.577 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:57:06.577 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.006088+0000", 2024-09-21T13:57:06.577 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5437000000000001e-05 2024-09-21T13:57:06.577 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.577 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.577 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:57:06.577 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.006107+0000", 2024-09-21T13:57:06.577 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8238999999999999e-05 2024-09-21T13:57:06.578 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:57:06.578 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:57:06.578 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:57:06.578 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.578 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.578 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:201885 7.5 7:a6d798b1:::benchmark_data_smithi049_76798_object65465:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T13:57:06.578 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:43.302360+0000", 2024-09-21T13:57:06.578 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 263.13123944799997, 2024-09-21T13:57:06.578 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.70381638199999996, 2024-09-21T13:57:06.578 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:57:06.578 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:57:06.578 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:06.578 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:57:06.578 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:57:06.579 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:57:06.579 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 201885 2024-09-21T13:57:06.579 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.579 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:57:06.579 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.579 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:57:06.579 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.302360+0000", 2024-09-21T13:57:06.579 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:57:06.579 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.579 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.579 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:57:06.579 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.302360+0000", 2024-09-21T13:57:06.579 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:57:06.579 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.580 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.580 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:57:06.580 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.302361+0000", 2024-09-21T13:57:06.580 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.291e-06 2024-09-21T13:57:06.580 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.580 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.580 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:57:06.580 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.302366+0000", 2024-09-21T13:57:06.580 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1619999999999999e-06 2024-09-21T13:57:06.580 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.580 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.580 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:57:06.580 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.302379+0000", 2024-09-21T13:57:06.581 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2824e-05 2024-09-21T13:57:06.581 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.581 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.581 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:57:06.581 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.302385+0000", 2024-09-21T13:57:06.581 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8799999999999996e-06 2024-09-21T13:57:06.581 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.581 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.581 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:57:06.581 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.302431+0000", 2024-09-21T13:57:06.581 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6733000000000001e-05 2024-09-21T13:57:06.581 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.581 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.582 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:57:06.582 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.302437+0000", 2024-09-21T13:57:06.582 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:06.583 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0170000000000002e-06 2024-09-21T13:57:06.583 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.583 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.583 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:57:06.583 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.977079+0000", 2024-09-21T13:57:06.583 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.67464170999999995 2024-09-21T13:57:06.583 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.583 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.583 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:57:06.583 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.977147+0000", 2024-09-21T13:57:06.583 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8099e-05 2024-09-21T13:57:06.584 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.584 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.584 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2", 2024-09-21T13:57:06.584 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.977215+0000", 2024-09-21T13:57:06.584 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8151000000000005e-05 2024-09-21T13:57:06.584 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.584 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.584 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:57:06.584 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:43.979736+0000", 2024-09-21T13:57:06.584 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.002520361 2024-09-21T13:57:06.584 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.584 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.584 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:06.584 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.000408+0000", 2024-09-21T13:57:06.585 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.020671948999999998 2024-09-21T13:57:06.585 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.585 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.585 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:06.585 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.006133+0000", 2024-09-21T13:57:06.585 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0057253149999999999 2024-09-21T13:57:06.585 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.585 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.585 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:57:06.585 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.006158+0000", 2024-09-21T13:57:06.585 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4943000000000002e-05 2024-09-21T13:57:06.585 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:57:06.585 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:57:06.585 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:57:06.585 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.006176+0000", 2024-09-21T13:57:06.586 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7946999999999999e-05 2024-09-21T13:57:06.586 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:57:06.586 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:57:06.586 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:57:06.586 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:57:06.586 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:57:06.586 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T13:57:06.620 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:06.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:06.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:06.722 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:57:06.761 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:06.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:06.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:57:06.862 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:06.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:06.906 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:06.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:07.007 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:07.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:07.057 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:07.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:07.158 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:07.188 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:07.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:07.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:07.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:07 smithi187 ceph-mon[33032]: pgmap v395: 148 pgs: 1 active+clean+remapped, 1 remapped+peering, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 1.9 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 882 B/s rd, 0 B/s wr, 1.08k op/s; 9709/93345 objects degraded (10.401%); 2001/93345 objects misplaced (2.144%); 4.7 MiB/s, 74 objects/s recovering 2024-09-21T13:57:07.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:07 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:57:07.291 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:07.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:07.327 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:07.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:07.404 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T13:57:07.428 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:57:07.438 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:07 smithi049 ceph-mon[30080]: pgmap v395: 148 pgs: 1 active+clean+remapped, 1 remapped+peering, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 1.9 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 882 B/s rd, 0 B/s wr, 1.08k op/s; 9709/93345 objects degraded (10.401%); 2001/93345 objects misplaced (2.144%); 4.7 MiB/s, 74 objects/s recovering 2024-09-21T13:57:07.438 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:07 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:57:07.438 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:07 smithi049 ceph-mon[40489]: pgmap v395: 148 pgs: 1 active+clean+remapped, 1 remapped+peering, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 1.9 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 882 B/s rd, 0 B/s wr, 1.08k op/s; 9709/93345 objects degraded (10.401%); 2001/93345 objects misplaced (2.144%); 4.7 MiB/s, 74 objects/s recovering 2024-09-21T13:57:07.438 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:07 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:57:07.479 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:07.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:07.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:57:07.537 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T13:57:07.537 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-09-21T13:57:07.563 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:57:07.581 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:07.659 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:07.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:07.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:07.761 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:07.793 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:07.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:07.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:07.894 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:07.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:07.922 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:07.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:08.023 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:08.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:08.082 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:08.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:08.111 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T13:57:08.183 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:57:08.208 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:08.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:08.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:57:08.292 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:57:08.309 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:08.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:08.441 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:08.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:08.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:08 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:57:08.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:08 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:57:08.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:08 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:57:08.446 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:08 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:57:08.447 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:08 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:57:08.447 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:08 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:57:08.447 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:08 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:57:08.447 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:08 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:57:08.447 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:08 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:57:08.541 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:08.569 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:08.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:08.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:08.671 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:57:08.740 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:08.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:08.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:57:08.842 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:08.867 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:08.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:08.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:08.968 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:09.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:09.008 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:09.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:09.086 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T13:57:09.086 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-09-21T13:57:09.086 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.086 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.15207.0:228477 8.7 e237/236)", 2024-09-21T13:57:09.086 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:09.074895+0000", 2024-09-21T13:57:09.086 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.00503448, 2024-09-21T13:57:09.086 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0050568339999999996, 2024-09-21T13:57:09.086 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:09.086 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:09.086 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "queued for pg", 2024-09-21T13:57:09.087 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:09.087 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.087 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:09.087 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:09.074895+0000", 2024-09-21T13:57:09.087 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:09.087 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:09.087 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.087 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:09.087 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:09.074895+0000", 2024-09-21T13:57:09.087 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:09.087 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:09.087 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.087 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:09.087 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:09.074895+0000", 2024-09-21T13:57:09.088 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5099999999999999e-07 2024-09-21T13:57:09.088 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:09.088 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.088 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:09.088 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:09.074900+0000", 2024-09-21T13:57:09.088 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5680000000000001e-06 2024-09-21T13:57:09.088 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:09.088 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.088 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:09.088 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:09.074918+0000", 2024-09-21T13:57:09.088 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7714999999999999e-05 2024-09-21T13:57:09.088 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:09.089 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.089 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:09.089 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:09.074926+0000", 2024-09-21T13:57:09.089 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5869999999999993e-06 2024-09-21T13:57:09.089 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:09.089 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:09.089 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:09.089 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:09.089 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.089 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.15207.0:228483 8.4 e237/183 8:240f34b2:::benchmark_data_smithi049_109311_object101964:head v 237'14371, mlcod=237'14369)", 2024-09-21T13:57:09.089 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:09.077332+0000", 2024-09-21T13:57:09.089 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.0025966829999999998, 2024-09-21T13:57:09.089 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0026702140000000002, 2024-09-21T13:57:09.089 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:09.090 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:09.090 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T13:57:09.090 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:09.090 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.090 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:09.090 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:09.077332+0000", 2024-09-21T13:57:09.090 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:09.090 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:09.090 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.090 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:09.090 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:09.077332+0000", 2024-09-21T13:57:09.090 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:09.090 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:09.091 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.091 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:09.091 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:09.077333+0000", 2024-09-21T13:57:09.091 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.5900000000000005e-07 2024-09-21T13:57:09.091 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:09.091 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.091 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:09.091 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:09.077340+0000", 2024-09-21T13:57:09.091 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3600000000000001e-06 2024-09-21T13:57:09.091 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:09.091 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.091 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:09.091 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:09.077352+0000", 2024-09-21T13:57:09.092 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2333e-05 2024-09-21T13:57:09.092 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:09.092 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.092 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:09.092 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:09.077359+0000", 2024-09-21T13:57:09.092 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1550000000000001e-06 2024-09-21T13:57:09.092 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:09.092 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.092 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:09.092 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:09.077428+0000", 2024-09-21T13:57:09.092 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8942999999999994e-05 2024-09-21T13:57:09.092 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:09.092 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.092 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:57:09.093 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:09.077447+0000", 2024-09-21T13:57:09.093 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8717999999999999e-05 2024-09-21T13:57:09.093 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:09.093 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:09.093 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:09.093 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:09.093 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.093 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.15207.0:228478 8.7 e237/236)", 2024-09-21T13:57:09.093 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:09.079697+0000", 2024-09-21T13:57:09.093 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.00023233100000000001, 2024-09-21T13:57:09.093 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00035591099999999999, 2024-09-21T13:57:09.093 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:09.093 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:09.093 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "queued for pg", 2024-09-21T13:57:09.093 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:09.093 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.094 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:09.094 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:09.079697+0000", 2024-09-21T13:57:09.094 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:09.094 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:09.094 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.094 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:09.094 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:09.079697+0000", 2024-09-21T13:57:09.094 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:09.094 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:09.094 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.094 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:09.094 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:09.079698+0000", 2024-09-21T13:57:09.094 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5099999999999999e-06 2024-09-21T13:57:09.094 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:09.094 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.095 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:09.095 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:09.079707+0000", 2024-09-21T13:57:09.095 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3440000000000001e-06 2024-09-21T13:57:09.095 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:09.095 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.095 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:09.095 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:09.079728+0000", 2024-09-21T13:57:09.095 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0832999999999999e-05 2024-09-21T13:57:09.095 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:09.095 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.095 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:09.095 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:09.079736+0000", 2024-09-21T13:57:09.095 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.602e-06 2024-09-21T13:57:09.095 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:09.096 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:09.096 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:09.096 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:09.096 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.096 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.15207.0:228479 8.6 e237/223 8:6ad2e57d:::benchmark_data_smithi049_109311_object101960:head v 237'14224, mlcod=237'14222)", 2024-09-21T13:57:09.096 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:09.069599+0000", 2024-09-21T13:57:09.096 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.01032974, 2024-09-21T13:57:09.096 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.010495305, 2024-09-21T13:57:09.096 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:09.096 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:09.096 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T13:57:09.096 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:09.096 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.096 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:09.096 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:09.069599+0000", 2024-09-21T13:57:09.096 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:09.097 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:09.097 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.097 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:09.097 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:09.069599+0000", 2024-09-21T13:57:09.097 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:09.097 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:09.097 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.097 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:09.097 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:09.069600+0000", 2024-09-21T13:57:09.097 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4600000000000004e-07 2024-09-21T13:57:09.097 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:09.097 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.097 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:09.097 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:09.069604+0000", 2024-09-21T13:57:09.097 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4190000000000004e-06 2024-09-21T13:57:09.097 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:09.098 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.098 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:09.098 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:09.069612+0000", 2024-09-21T13:57:09.098 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.763e-06 2024-09-21T13:57:09.098 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:09.098 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.098 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:09.098 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:09.069618+0000", 2024-09-21T13:57:09.098 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1709999999999999e-06 2024-09-21T13:57:09.098 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:09.098 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.099 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:09.099 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:09.071707+0000", 2024-09-21T13:57:09.099 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0020889369999999999 2024-09-21T13:57:09.099 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:09.099 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.099 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:57:09.099 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:09.071726+0000", 2024-09-21T13:57:09.099 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8544999999999999e-05 2024-09-21T13:57:09.099 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:09.099 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:09.099 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:09.099 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:09.099 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.099 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.15207.0:228472 8.a e237/183 8:5a979b3c:::benchmark_data_smithi049_109311_object101953:head v 237'14193, mlcod=237'14190)", 2024-09-21T13:57:09.099 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:09.072974+0000", 2024-09-21T13:57:09.100 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.0069555199999999998, 2024-09-21T13:57:09.100 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.007168865, 2024-09-21T13:57:09.100 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:09.100 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:09.100 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T13:57:09.100 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:09.100 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.100 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:09.100 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:09.072974+0000", 2024-09-21T13:57:09.100 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:09.100 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:09.100 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.100 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:09.100 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:09.072974+0000", 2024-09-21T13:57:09.101 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:09.101 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:09.101 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.101 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:09.101 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:09.072975+0000", 2024-09-21T13:57:09.101 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1999999999999998e-07 2024-09-21T13:57:09.101 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:09.101 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.101 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:09.101 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:09.072981+0000", 2024-09-21T13:57:09.101 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5520000000000001e-06 2024-09-21T13:57:09.101 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:09.101 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.101 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:09.101 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:09.072994+0000", 2024-09-21T13:57:09.101 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3185e-05 2024-09-21T13:57:09.102 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:09.102 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.102 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:09.102 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:09.073001+0000", 2024-09-21T13:57:09.102 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8310000000000003e-06 2024-09-21T13:57:09.102 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:09.102 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.102 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:09.102 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:09.075029+0000", 2024-09-21T13:57:09.102 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0020277279999999999 2024-09-21T13:57:09.102 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:09.102 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.102 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:57:09.102 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:09.075047+0000", 2024-09-21T13:57:09.102 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8544000000000001e-05 2024-09-21T13:57:09.103 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:09.103 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:09.103 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:09.103 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:09.103 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.103 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.15207.0:228475 8.7 e237/236 8:e75ade9e:::benchmark_data_smithi049_109311_object101956:head v 237'14154, mlcod=237'14152)", 2024-09-21T13:57:09.103 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:09.073056+0000", 2024-09-21T13:57:09.103 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.0068729300000000002, 2024-09-21T13:57:09.103 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.007137582, 2024-09-21T13:57:09.103 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:09.103 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:09.103 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T13:57:09.103 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:09.104 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.104 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:09.104 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:09.073056+0000", 2024-09-21T13:57:09.104 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:09.104 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:09.104 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.104 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:09.104 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:09.073056+0000", 2024-09-21T13:57:09.104 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:09.104 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:09.104 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.104 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:09.104 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:09.073057+0000", 2024-09-21T13:57:09.104 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.8299999999999998e-07 2024-09-21T13:57:09.105 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:09.105 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.105 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:09.105 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:09.073066+0000", 2024-09-21T13:57:09.105 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7669999999999998e-06 2024-09-21T13:57:09.105 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:09.105 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.105 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:09.105 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:09.073074+0000", 2024-09-21T13:57:09.105 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9580000000000007e-06 2024-09-21T13:57:09.105 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:09.105 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.105 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:09.105 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:09.073080+0000", 2024-09-21T13:57:09.106 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6699999999999999e-06 2024-09-21T13:57:09.106 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:09.106 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.106 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:09.106 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:09.075723+0000", 2024-09-21T13:57:09.106 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0026435149999999999 2024-09-21T13:57:09.106 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:09.106 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.106 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:57:09.106 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:09.075745+0000", 2024-09-21T13:57:09.106 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1246999999999998e-05 2024-09-21T13:57:09.106 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:09.106 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:09.106 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:09.107 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:09.107 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.107 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.15207.0:228481 8.d e237/183 8:be872ccd:::benchmark_data_smithi049_109311_object101962:head v 237'14301, mlcod=237'14299)", 2024-09-21T13:57:09.107 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:09.073140+0000", 2024-09-21T13:57:09.107 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.0067894460000000002, 2024-09-21T13:57:09.107 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0071028000000000003, 2024-09-21T13:57:09.107 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:09.107 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:09.107 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T13:57:09.107 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:09.107 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.107 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:09.107 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:09.073140+0000", 2024-09-21T13:57:09.108 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:09.108 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:09.108 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.108 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:09.108 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:09.073140+0000", 2024-09-21T13:57:09.108 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:09.108 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:09.108 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.108 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:09.108 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:09.073141+0000", 2024-09-21T13:57:09.108 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9500000000000001e-07 2024-09-21T13:57:09.108 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:09.109 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.109 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:09.109 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:09.073145+0000", 2024-09-21T13:57:09.109 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2400000000000001e-06 2024-09-21T13:57:09.109 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:09.109 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.109 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:09.109 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:09.073154+0000", 2024-09-21T13:57:09.109 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2750000000000001e-06 2024-09-21T13:57:09.109 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:09.109 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.109 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:09.110 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:09.073159+0000", 2024-09-21T13:57:09.110 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3159999999999996e-06 2024-09-21T13:57:09.110 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:09.110 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.110 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:09.110 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:09.077768+0000", 2024-09-21T13:57:09.110 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0046088070000000004 2024-09-21T13:57:09.110 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:09.110 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:09.110 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:57:09.110 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:09.077785+0000", 2024-09-21T13:57:09.110 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7119e-05 2024-09-21T13:57:09.110 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:09.110 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:09.111 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:09.111 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:09.111 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T13:57:09.111 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 7 2024-09-21T13:57:09.111 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T13:57:09.111 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:57:09.112 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:57:09.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:09.178 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:09.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:57:09.279 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:09.320 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:09.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:09.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:09.422 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:09.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:09.460 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:09.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:09.561 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:09.589 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:09.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:09.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:09.636 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-09-21T13:57:09.691 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:09.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:09 smithi187 ceph-mon[33032]: pgmap v396: 148 pgs: 1 active+clean+remapped, 1 remapped+peering, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 1.9 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 752 B/s rd, 0 B/s wr, 920 op/s; 9709/93345 objects degraded (10.401%); 2001/93345 objects misplaced (2.144%); 4.0 MiB/s, 63 objects/s recovering 2024-09-21T13:57:09.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:09 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3619004542' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:57:09.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:09 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:57:09.695 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:57:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:57:09.726 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:09.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:09.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:09.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:09 smithi049 ceph-mon[40489]: pgmap v396: 148 pgs: 1 active+clean+remapped, 1 remapped+peering, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 1.9 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 752 B/s rd, 0 B/s wr, 920 op/s; 9709/93345 objects degraded (10.401%); 2001/93345 objects misplaced (2.144%); 4.0 MiB/s, 63 objects/s recovering 2024-09-21T13:57:09.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:09 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3619004542' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:57:09.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:09 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:57:09.755 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 13:57:09 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-3[129752]: 2024-09-21T13:57:09.654+0000 7ff0da45c640 -1 osd.3 214 log_to_monitors true 2024-09-21T13:57:09.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:09 smithi049 ceph-mon[30080]: pgmap v396: 148 pgs: 1 active+clean+remapped, 1 remapped+peering, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 1.9 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 752 B/s rd, 0 B/s wr, 920 op/s; 9709/93345 objects degraded (10.401%); 2001/93345 objects misplaced (2.144%); 4.0 MiB/s, 63 objects/s recovering 2024-09-21T13:57:09.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:09 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3619004542' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:57:09.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:09 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:57:09.828 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:09.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:09.866 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:09.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:09.967 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:09.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:09.995 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:09.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:10.046 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T13:57:10.095 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:57:10.164 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:10.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:10.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:57:10.266 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:10.294 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:10.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:10.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:10.395 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:57:10.426 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:10.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:10.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:57:10.528 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:10.565 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:10.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:10.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:10.666 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:57:10.703 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:10.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:10.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:57:10.804 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:10.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:10.837 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:10.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:10.938 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:10.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:10 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:57:10.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:10 smithi187 ceph-mon[33032]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:57:10.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:10 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T13:57:10.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:10 smithi187 ceph-mon[33032]: osdmap e238: 8 total, 7 up, 7 in 2024-09-21T13:57:10.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:10 smithi187 ceph-mon[33032]: from='osd.3 [v2:172.21.15.49:6826/886735721,v1:172.21.15.49:6827/886735721]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-21T13:57:10.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:10 smithi187 ceph-mon[33032]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-21T13:57:10.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:10 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 9709/93345 objects degraded (10.401%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-09-21T13:57:10.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:10.969 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:10.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:11.004 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 13:57:10 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-3[129752]: 2024-09-21T13:57:10.705+0000 7ff0d2206640 -1 osd.3 214 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T13:57:11.005 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:57:10 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:57:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:57:11.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:10 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:57:11.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:10 smithi049 ceph-mon[40489]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:57:11.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:10 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T13:57:11.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:10 smithi049 ceph-mon[40489]: osdmap e238: 8 total, 7 up, 7 in 2024-09-21T13:57:11.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:10 smithi049 ceph-mon[40489]: from='osd.3 [v2:172.21.15.49:6826/886735721,v1:172.21.15.49:6827/886735721]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-21T13:57:11.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:10 smithi049 ceph-mon[40489]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-21T13:57:11.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:10 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 9709/93345 objects degraded (10.401%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-09-21T13:57:11.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:10 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:57:11.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:10 smithi049 ceph-mon[30080]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:57:11.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:10 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T13:57:11.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:10 smithi049 ceph-mon[30080]: osdmap e238: 8 total, 7 up, 7 in 2024-09-21T13:57:11.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:10 smithi049 ceph-mon[30080]: from='osd.3 [v2:172.21.15.49:6826/886735721,v1:172.21.15.49:6827/886735721]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-21T13:57:11.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:10 smithi049 ceph-mon[30080]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-21T13:57:11.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:10 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 9709/93345 objects degraded (10.401%), 30 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-09-21T13:57:11.069 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:11.101 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:11.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:11.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:11.203 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:11.242 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:11.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:11.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:11.343 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:11.372 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:11.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:11.373 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:11.474 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:11.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:11.505 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:11.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:11.606 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:57:11.648 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:11.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:11.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:57:11.750 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:11.772 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:11.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:11.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:11.874 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:11.904 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:11.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:11.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:11.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:11 smithi187 ceph-mon[33032]: pgmap v398: 148 pgs: 2 active+clean+remapped, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 1.5 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.81k op/s; 7813/75354 objects degraded (10.368%); 3198/75354 objects misplaced (4.244%); 4.0 MiB/s, 63 objects/s recovering 2024-09-21T13:57:11.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:11 smithi187 ceph-mon[33032]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-21T13:57:11.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:11 smithi187 ceph-mon[33032]: from='osd.3 [v2:172.21.15.49:6826/886735721,v1:172.21.15.49:6827/886735721]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi049", "root=default"]} : dispatch 2024-09-21T13:57:11.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:11 smithi187 ceph-mon[33032]: osdmap e239: 8 total, 7 up, 7 in 2024-09-21T13:57:11.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:11 smithi187 ceph-mon[33032]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi049", "root=default"]} : dispatch 2024-09-21T13:57:12.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:11 smithi049 ceph-mon[40489]: pgmap v398: 148 pgs: 2 active+clean+remapped, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 1.5 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.81k op/s; 7813/75354 objects degraded (10.368%); 3198/75354 objects misplaced (4.244%); 4.0 MiB/s, 63 objects/s recovering 2024-09-21T13:57:12.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:11 smithi049 ceph-mon[40489]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-21T13:57:12.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:11 smithi049 ceph-mon[40489]: from='osd.3 [v2:172.21.15.49:6826/886735721,v1:172.21.15.49:6827/886735721]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi049", "root=default"]} : dispatch 2024-09-21T13:57:12.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:11 smithi049 ceph-mon[40489]: osdmap e239: 8 total, 7 up, 7 in 2024-09-21T13:57:12.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:11 smithi049 ceph-mon[40489]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi049", "root=default"]} : dispatch 2024-09-21T13:57:12.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:11 smithi049 ceph-mon[30080]: pgmap v398: 148 pgs: 2 active+clean+remapped, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 1.5 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.81k op/s; 7813/75354 objects degraded (10.368%); 3198/75354 objects misplaced (4.244%); 4.0 MiB/s, 63 objects/s recovering 2024-09-21T13:57:12.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:11 smithi049 ceph-mon[30080]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-21T13:57:12.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:11 smithi049 ceph-mon[30080]: from='osd.3 [v2:172.21.15.49:6826/886735721,v1:172.21.15.49:6827/886735721]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi049", "root=default"]} : dispatch 2024-09-21T13:57:12.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:11 smithi049 ceph-mon[30080]: osdmap e239: 8 total, 7 up, 7 in 2024-09-21T13:57:12.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:11 smithi049 ceph-mon[30080]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi049", "root=default"]} : dispatch 2024-09-21T13:57:12.006 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:57:12.028 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:12.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:12.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:57:12.130 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:12.152 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:12.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:12.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:12.254 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:57:12.276 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:12.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:12.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:57:12.378 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:12.400 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:12.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:12.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:12.502 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:12.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:12.525 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:12.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:12.538 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T13:57:12.627 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:12.651 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:12.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:12.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:12.752 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:12.780 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:12.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:12.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:12.882 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:12.885 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:12 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:57:12.885 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:12 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:57:12.885 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:12 smithi049 ceph-mon[40489]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T13:57:12.885 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:12 smithi049 ceph-mon[40489]: osd.3 [v2:172.21.15.49:6826/886735721,v1:172.21.15.49:6827/886735721] boot 2024-09-21T13:57:12.885 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:12 smithi049 ceph-mon[40489]: osdmap e240: 8 total, 8 up, 7 in 2024-09-21T13:57:12.885 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:12 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T13:57:12.885 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:12 smithi049 ceph-mon[40489]: osdmap e241: 8 total, 8 up, 7 in 2024-09-21T13:57:12.886 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:12 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:57:12.886 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:12 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:57:12.886 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:12 smithi049 ceph-mon[30080]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T13:57:12.886 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:12 smithi049 ceph-mon[30080]: osd.3 [v2:172.21.15.49:6826/886735721,v1:172.21.15.49:6827/886735721] boot 2024-09-21T13:57:12.886 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:12 smithi049 ceph-mon[30080]: osdmap e240: 8 total, 8 up, 7 in 2024-09-21T13:57:12.886 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:12 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T13:57:12.886 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:12 smithi049 ceph-mon[30080]: osdmap e241: 8 total, 8 up, 7 in 2024-09-21T13:57:12.912 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:12.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:12.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:12.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:12 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:57:12.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:12 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:57:12.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:12 smithi187 ceph-mon[33032]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T13:57:12.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:12 smithi187 ceph-mon[33032]: osd.3 [v2:172.21.15.49:6826/886735721,v1:172.21.15.49:6827/886735721] boot 2024-09-21T13:57:12.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:12 smithi187 ceph-mon[33032]: osdmap e240: 8 total, 8 up, 7 in 2024-09-21T13:57:12.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:12 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T13:57:12.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:12 smithi187 ceph-mon[33032]: osdmap e241: 8 total, 8 up, 7 in 2024-09-21T13:57:13.014 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:13.034 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:13.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:13.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:13.136 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:13.157 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:13.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:13.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:13.179 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:57:13.179 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:57:13.204 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:57:13.204 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:57:13.209 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:57:13.209 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:57:13.238 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:57:13.238 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:57:13.258 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:13.261 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:57:13.261 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:57:13.279 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:57:13.279 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:57:13.280 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:13.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:13.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:13.299 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:57:13.300 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:57:13.320 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:57:13.320 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:57:13.340 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:57:13.340 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:57:13.382 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:13.407 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:13.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:13.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:13.509 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:57:13.530 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:13.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:13.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:57:13.632 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:13.653 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:13.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:13.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:13.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:13 smithi049 ceph-mon[40489]: pgmap v401: 148 pgs: 2 active+clean+remapped, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 1.5 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 999 op/s; 7813/75354 objects degraded (10.368%); 3198/75354 objects misplaced (4.244%) 2024-09-21T13:57:13.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:13 smithi049 ceph-mon[40489]: 8.7 starting backfill to osd.3 from (194'3139,214'7375] MAX to 239'14372 2024-09-21T13:57:13.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:13 smithi049 ceph-mon[40489]: 8.9 starting backfill to osd.3 from (192'2497,214'7528] 8:91709739:::benchmark_data_smithi049_109311_object48864:head to 239'14721 2024-09-21T13:57:13.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:13 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3888980943' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:57:13.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:13 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:57:13.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:13 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:57:13.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:13 smithi049 ceph-mon[40489]: osdmap e242: 8 total, 8 up, 7 in 2024-09-21T13:57:13.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:13 smithi049 ceph-mon[30080]: pgmap v401: 148 pgs: 2 active+clean+remapped, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 1.5 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 999 op/s; 7813/75354 objects degraded (10.368%); 3198/75354 objects misplaced (4.244%) 2024-09-21T13:57:13.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:13 smithi049 ceph-mon[30080]: 8.7 starting backfill to osd.3 from (194'3139,214'7375] MAX to 239'14372 2024-09-21T13:57:13.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:13 smithi049 ceph-mon[30080]: 8.9 starting backfill to osd.3 from (192'2497,214'7528] 8:91709739:::benchmark_data_smithi049_109311_object48864:head to 239'14721 2024-09-21T13:57:13.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:13 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3888980943' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:57:13.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:13 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:57:13.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:13 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:57:13.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:13 smithi049 ceph-mon[30080]: osdmap e242: 8 total, 8 up, 7 in 2024-09-21T13:57:13.756 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:57:13.777 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:13.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:13.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:57:13.879 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:13.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:13.915 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:13.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:13.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:13 smithi187 ceph-mon[33032]: pgmap v401: 148 pgs: 2 active+clean+remapped, 38 active+undersized, 30 active+undersized+degraded, 78 active+clean; 1.5 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 999 op/s; 7813/75354 objects degraded (10.368%); 3198/75354 objects misplaced (4.244%) 2024-09-21T13:57:13.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:13 smithi187 ceph-mon[33032]: 8.7 starting backfill to osd.3 from (194'3139,214'7375] MAX to 239'14372 2024-09-21T13:57:13.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:13 smithi187 ceph-mon[33032]: 8.9 starting backfill to osd.3 from (192'2497,214'7528] 8:91709739:::benchmark_data_smithi049_109311_object48864:head to 239'14721 2024-09-21T13:57:13.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:13 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3888980943' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:57:13.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:13 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:57:13.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:13 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:57:13.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:13 smithi187 ceph-mon[33032]: osdmap e242: 8 total, 8 up, 7 in 2024-09-21T13:57:14.015 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:57:14.058 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:14.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:14.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:57:14.160 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:14.188 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:14.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:14.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:14.289 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:14.317 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:14.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:14.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:14.418 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:14.500 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:14.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:14.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:14.561 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T13:57:14.603 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:57:14.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:14.668 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:14.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:57:14.769 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:14.833 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:14.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:14.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:14.885 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:14 smithi049 ceph-mon[40489]: 8.0 starting backfill to osd.3 from (194'3040,214'7352] MAX to 239'14408 2024-09-21T13:57:14.886 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:14 smithi049 ceph-mon[40489]: 8.c starting backfill to osd.3 from (193'2645,214'7497] MAX to 239'14634 2024-09-21T13:57:14.886 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:14 smithi049 ceph-mon[40489]: 8.b starting backfill to osd.3 from (194'2936,214'7315] MAX to 239'14393 2024-09-21T13:57:14.886 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:14 smithi049 ceph-mon[40489]: 8.f starting backfill to osd.3 from (194'3269,214'7556] MAX to 239'14798 2024-09-21T13:57:14.886 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:14 smithi049 ceph-mon[40489]: 8.8 starting backfill to osd.3 from (192'2346,214'7428] MAX to 239'14569 2024-09-21T13:57:14.888 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:14 smithi049 ceph-mon[30080]: 8.0 starting backfill to osd.3 from (194'3040,214'7352] MAX to 239'14408 2024-09-21T13:57:14.889 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:14 smithi049 ceph-mon[30080]: 8.c starting backfill to osd.3 from (193'2645,214'7497] MAX to 239'14634 2024-09-21T13:57:14.889 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:14 smithi049 ceph-mon[30080]: 8.b starting backfill to osd.3 from (194'2936,214'7315] MAX to 239'14393 2024-09-21T13:57:14.889 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:14 smithi049 ceph-mon[30080]: 8.f starting backfill to osd.3 from (194'3269,214'7556] MAX to 239'14798 2024-09-21T13:57:14.889 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:14 smithi049 ceph-mon[30080]: 8.8 starting backfill to osd.3 from (192'2346,214'7428] MAX to 239'14569 2024-09-21T13:57:14.934 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:14.958 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T13:57:14.958 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:57:14.965 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:14.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:14.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:15.066 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:57:15.092 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:15.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:15.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:57:15.193 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:15.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:14 smithi187 ceph-mon[33032]: 8.0 starting backfill to osd.3 from (194'3040,214'7352] MAX to 239'14408 2024-09-21T13:57:15.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:14 smithi187 ceph-mon[33032]: 8.c starting backfill to osd.3 from (193'2645,214'7497] MAX to 239'14634 2024-09-21T13:57:15.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:14 smithi187 ceph-mon[33032]: 8.b starting backfill to osd.3 from (194'2936,214'7315] MAX to 239'14393 2024-09-21T13:57:15.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:14 smithi187 ceph-mon[33032]: 8.f starting backfill to osd.3 from (194'3269,214'7556] MAX to 239'14798 2024-09-21T13:57:15.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:14 smithi187 ceph-mon[33032]: 8.8 starting backfill to osd.3 from (192'2346,214'7428] MAX to 239'14569 2024-09-21T13:57:15.239 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:15.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:15.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:15.341 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:15.370 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:15.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:15.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:15.472 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:15.511 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:15.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:15.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:15.613 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:15.638 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:15.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:15.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:15.740 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:15.779 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:15.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:15.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:15.881 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:15.885 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T13:57:15.919 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:15.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:15.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:15.923 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T13:57:15.923 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:57:16.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:15 smithi049 ceph-mon[30080]: pgmap v404: 148 pgs: 1 active+remapped+backfilling, 1 remapped+peering, 3 peering, 1 active+clean+remapped, 19 active+undersized, 17 active+undersized+degraded, 106 active+clean; 1.4 GiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 458 op/s; 6230/69870 objects degraded (8.917%); 2564/69870 objects misplaced (3.670%); 4.6 MiB/s, 73 objects/s recovering 2024-09-21T13:57:16.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:15 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 6230/69870 objects degraded (8.917%), 17 pgs degraded, 36 pgs undersized (PG_DEGRADED) 2024-09-21T13:57:16.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:15 smithi049 ceph-mon[40489]: pgmap v404: 148 pgs: 1 active+remapped+backfilling, 1 remapped+peering, 3 peering, 1 active+clean+remapped, 19 active+undersized, 17 active+undersized+degraded, 106 active+clean; 1.4 GiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 458 op/s; 6230/69870 objects degraded (8.917%); 2564/69870 objects misplaced (3.670%); 4.6 MiB/s, 73 objects/s recovering 2024-09-21T13:57:16.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:15 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 6230/69870 objects degraded (8.917%), 17 pgs degraded, 36 pgs undersized (PG_DEGRADED) 2024-09-21T13:57:16.023 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:16.047 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:16.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:16.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:16.096 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:15 smithi187 ceph-mon[33032]: pgmap v404: 148 pgs: 1 active+remapped+backfilling, 1 remapped+peering, 3 peering, 1 active+clean+remapped, 19 active+undersized, 17 active+undersized+degraded, 106 active+clean; 1.4 GiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 458 op/s; 6230/69870 objects degraded (8.917%); 2564/69870 objects misplaced (3.670%); 4.6 MiB/s, 73 objects/s recovering 2024-09-21T13:57:16.096 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:15 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 6230/69870 objects degraded (8.917%), 17 pgs degraded, 36 pgs undersized (PG_DEGRADED) 2024-09-21T13:57:16.149 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:16.190 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:16.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:16.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:16.292 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:16.330 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:16.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:16.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:16.421 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T13:57:16.432 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:16.470 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:16.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:16.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:16.572 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:57:16.601 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:16.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:16.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:57:16.703 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:16.757 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:16.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:16.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:16.824 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:57:16.860 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:16.931 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:16.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:16.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:16.977 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:16 smithi049 ceph-mon[40489]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T13:57:16.977 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:16 smithi049 ceph-mon[30080]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T13:57:17.032 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:17.054 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:16 smithi187 ceph-mon[33032]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T13:57:17.062 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:17.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:17.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:17.164 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:17.196 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:17.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:17.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:17.248 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:57:17.298 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:17.326 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:17.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:17.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:17.427 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:17.473 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:17.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:17.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:17.501 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:57:17.575 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:17.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:17.624 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:17.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:17.725 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:17.771 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:17.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:17.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:17.872 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:57:17.916 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:17.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:17.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:57:18.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:17 smithi049 ceph-mon[40489]: pgmap v405: 148 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 1 remapped+peering, 2 active+undersized+degraded+remapped+backfilling, 3 peering, 1 active+remapped+backfill_toofull, 138 active+clean; 1.3 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 777 op/s; 5163/63267 objects degraded (8.161%); 1986/63267 objects misplaced (3.139%); 4.9 MiB/s, 78 objects/s recovering 2024-09-21T13:57:18.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:17 smithi049 ceph-mon[30080]: pgmap v405: 148 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 1 remapped+peering, 2 active+undersized+degraded+remapped+backfilling, 3 peering, 1 active+remapped+backfill_toofull, 138 active+clean; 1.3 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 777 op/s; 5163/63267 objects degraded (8.161%); 1986/63267 objects misplaced (3.139%); 4.9 MiB/s, 78 objects/s recovering 2024-09-21T13:57:18.018 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:57:18.056 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:18.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:18.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:57:18.101 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:17 smithi187 ceph-mon[33032]: pgmap v405: 148 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 1 remapped+peering, 2 active+undersized+degraded+remapped+backfilling, 3 peering, 1 active+remapped+backfill_toofull, 138 active+clean; 1.3 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 777 op/s; 5163/63267 objects degraded (8.161%); 1986/63267 objects misplaced (3.139%); 4.9 MiB/s, 78 objects/s recovering 2024-09-21T13:57:18.158 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:18.180 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:18.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:18.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:18.282 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:57:18.310 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:18.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:18.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:57:18.412 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:18.445 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:18.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:18.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:18.546 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:18.573 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:18.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:18.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:18.675 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:18.708 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:18.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:18.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:18.786 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-09-21T13:57:18.811 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:18.860 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:18.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:18.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:18.962 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:57:19.005 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:19.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:19.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:57:19.107 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:19.127 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:19.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:19.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:19.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:18 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3488899704' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:57:19.229 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:57:19.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:18 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3488899704' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:57:19.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:18 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3488899704' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:57:19.263 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:19.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:19.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:57:19.365 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:19.394 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:19.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:19.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:19.496 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:19.523 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:19.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:19.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:19.625 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:19.651 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:19.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:19.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:19.694 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:57:19 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:57:19.752 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:19.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:19.793 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:19.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:19.894 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:19.929 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:19.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:19.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:20.031 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:20.065 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:20.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:20.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:20.167 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:57:20.187 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:20.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:20.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:57:20.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:19 smithi187 ceph-mon[33032]: pgmap v406: 148 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 1 remapped+peering, 2 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 141 active+clean; 1.3 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 650 op/s; 5163/63267 objects degraded (8.161%); 1986/63267 objects misplaced (3.139%); 4.1 MiB/s, 65 objects/s recovering 2024-09-21T13:57:20.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:19 smithi187 ceph-mon[33032]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:57:20.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:19 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3488899704' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T13:57:20.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:19 smithi187 ceph-mon[33032]: osdmap e243: 8 total, 8 up, 7 in 2024-09-21T13:57:20.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:19 smithi049 ceph-mon[40489]: pgmap v406: 148 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 1 remapped+peering, 2 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 141 active+clean; 1.3 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 650 op/s; 5163/63267 objects degraded (8.161%); 1986/63267 objects misplaced (3.139%); 4.1 MiB/s, 65 objects/s recovering 2024-09-21T13:57:20.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:19 smithi049 ceph-mon[40489]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:57:20.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:19 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3488899704' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T13:57:20.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:19 smithi049 ceph-mon[40489]: osdmap e243: 8 total, 8 up, 7 in 2024-09-21T13:57:20.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:19 smithi049 ceph-mon[30080]: pgmap v406: 148 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 1 remapped+peering, 2 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 141 active+clean; 1.3 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 650 op/s; 5163/63267 objects degraded (8.161%); 1986/63267 objects misplaced (3.139%); 4.1 MiB/s, 65 objects/s recovering 2024-09-21T13:57:20.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:19 smithi049 ceph-mon[30080]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:57:20.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:19 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3488899704' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T13:57:20.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:19 smithi049 ceph-mon[30080]: osdmap e243: 8 total, 8 up, 7 in 2024-09-21T13:57:20.290 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:20.315 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:20.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:20.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:20.418 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:20.453 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:20.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:20.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:20.555 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:20.578 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:20.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:20.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:20.680 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:20.705 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:20.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:20.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:20.806 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:20.845 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:20.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:20.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:20.950 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:20.978 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:20.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:20.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:21.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:20 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:57:21.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:20 smithi049 ceph-mon[40489]: 6.8 scrub starts 2024-09-21T13:57:21.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:20 smithi049 ceph-mon[40489]: 6.8 scrub ok 2024-09-21T13:57:21.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:20 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 5163/63267 objects degraded (8.161%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T13:57:21.005 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:57:20 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:57:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:57:21.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:20 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:57:21.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:20 smithi049 ceph-mon[30080]: 6.8 scrub starts 2024-09-21T13:57:21.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:20 smithi049 ceph-mon[30080]: 6.8 scrub ok 2024-09-21T13:57:21.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:20 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 5163/63267 objects degraded (8.161%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T13:57:21.079 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:57:21.102 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:21.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:21.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:57:21.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:20 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:57:21.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:20 smithi187 ceph-mon[33032]: 6.8 scrub starts 2024-09-21T13:57:21.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:20 smithi187 ceph-mon[33032]: 6.8 scrub ok 2024-09-21T13:57:21.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:20 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 5163/63267 objects degraded (8.161%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T13:57:21.204 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:57:21.225 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:21.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:21.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:57:21.327 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:57:21.352 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:21.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:21.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:57:21.454 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:57:21.477 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:21.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:21.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:57:21.579 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:21.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:21.601 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:21.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:21.702 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:21.723 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:21.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:21.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:21.751 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [] live_osds: [0, 1, 4, 5, 6, 7, 2, 3] 2024-09-21T13:57:21.752 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T13:57:21.752 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-09-21T13:57:21.752 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T13:57:21.752 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T13:57:21.752 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-09-21T13:57:21.825 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:21.847 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:21.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:21.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:21.949 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:21.971 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:21.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:21.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:22.027 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:57:22.073 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:22.095 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:22.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:22.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:22.158 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:21 smithi187 ceph-mon[33032]: pgmap v408: 148 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 141 active+clean; 1008 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.22k op/s; 3531/48954 objects degraded (7.213%); 1421/48954 objects misplaced (2.903%); 11 MiB/s, 175 objects/s recovering 2024-09-21T13:57:22.159 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:21 smithi187 ceph-mon[33032]: 3.1 scrub starts 2024-09-21T13:57:22.159 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:21 smithi187 ceph-mon[33032]: 3.1 scrub ok 2024-09-21T13:57:22.159 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:21 smithi187 ceph-mon[33032]: 3.12 scrub starts 2024-09-21T13:57:22.159 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:21 smithi187 ceph-mon[33032]: 3.12 scrub ok 2024-09-21T13:57:22.159 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:21 smithi187 ceph-mon[33032]: osdmap e244: 8 total, 8 up, 7 in 2024-09-21T13:57:22.196 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:22.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:22.224 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:22.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:22.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:21 smithi049 ceph-mon[40489]: pgmap v408: 148 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 141 active+clean; 1008 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.22k op/s; 3531/48954 objects degraded (7.213%); 1421/48954 objects misplaced (2.903%); 11 MiB/s, 175 objects/s recovering 2024-09-21T13:57:22.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:21 smithi049 ceph-mon[40489]: 3.1 scrub starts 2024-09-21T13:57:22.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:21 smithi049 ceph-mon[40489]: 3.1 scrub ok 2024-09-21T13:57:22.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:21 smithi049 ceph-mon[40489]: 3.12 scrub starts 2024-09-21T13:57:22.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:21 smithi049 ceph-mon[40489]: 3.12 scrub ok 2024-09-21T13:57:22.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:21 smithi049 ceph-mon[40489]: osdmap e244: 8 total, 8 up, 7 in 2024-09-21T13:57:22.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:21 smithi049 ceph-mon[30080]: pgmap v408: 148 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 141 active+clean; 1008 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.22k op/s; 3531/48954 objects degraded (7.213%); 1421/48954 objects misplaced (2.903%); 11 MiB/s, 175 objects/s recovering 2024-09-21T13:57:22.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:21 smithi049 ceph-mon[30080]: 3.1 scrub starts 2024-09-21T13:57:22.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:21 smithi049 ceph-mon[30080]: 3.1 scrub ok 2024-09-21T13:57:22.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:21 smithi049 ceph-mon[30080]: 3.12 scrub starts 2024-09-21T13:57:22.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:21 smithi049 ceph-mon[30080]: 3.12 scrub ok 2024-09-21T13:57:22.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:21 smithi049 ceph-mon[30080]: osdmap e244: 8 total, 8 up, 7 in 2024-09-21T13:57:22.325 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:22.366 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:22.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:22.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:22.468 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:22.494 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:22.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:22.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:22.596 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:57:22.618 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:22.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:22.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:57:22.720 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:22.733 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T13:57:22.733 INFO:teuthology.orchestra.run.smithi187.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-21T13:57:22.733 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T13:57:22.754 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:22.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:22.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:22.856 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:57:22.880 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:22.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:22.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:57:22.982 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:23.018 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:23.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:23.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:23.120 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:23.150 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:23.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:23.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:23.252 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:23.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:22 smithi049 ceph-mon[40489]: 6.7 scrub starts 2024-09-21T13:57:23.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:22 smithi049 ceph-mon[40489]: 6.7 scrub ok 2024-09-21T13:57:23.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:22 smithi049 ceph-mon[40489]: 4.1c scrub starts 2024-09-21T13:57:23.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:22 smithi049 ceph-mon[40489]: 4.1c scrub ok 2024-09-21T13:57:23.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:22 smithi049 ceph-mon[40489]: 6.1e scrub starts 2024-09-21T13:57:23.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:22 smithi049 ceph-mon[40489]: 6.1e scrub ok 2024-09-21T13:57:23.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:22 smithi049 ceph-mon[40489]: 3.d scrub starts 2024-09-21T13:57:23.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:22 smithi049 ceph-mon[40489]: 3.d scrub ok 2024-09-21T13:57:23.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:22 smithi049 ceph-mon[40489]: pgmap v410: 148 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 141 active+clean; 1008 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 895 B/s rd, 0 B/s wr, 872 op/s; 3531/48954 objects degraded (7.213%); 1421/48954 objects misplaced (2.903%); 7.6 MiB/s, 121 objects/s recovering 2024-09-21T13:57:23.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:22 smithi049 ceph-mon[40489]: osdmap e245: 8 total, 8 up, 7 in 2024-09-21T13:57:23.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:22 smithi049 ceph-mon[30080]: 6.7 scrub starts 2024-09-21T13:57:23.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:22 smithi049 ceph-mon[30080]: 6.7 scrub ok 2024-09-21T13:57:23.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:22 smithi049 ceph-mon[30080]: 4.1c scrub starts 2024-09-21T13:57:23.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:22 smithi049 ceph-mon[30080]: 4.1c scrub ok 2024-09-21T13:57:23.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:22 smithi049 ceph-mon[30080]: 6.1e scrub starts 2024-09-21T13:57:23.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:22 smithi049 ceph-mon[30080]: 6.1e scrub ok 2024-09-21T13:57:23.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:22 smithi049 ceph-mon[30080]: 3.d scrub starts 2024-09-21T13:57:23.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:22 smithi049 ceph-mon[30080]: 3.d scrub ok 2024-09-21T13:57:23.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:22 smithi049 ceph-mon[30080]: pgmap v410: 148 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 141 active+clean; 1008 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 895 B/s rd, 0 B/s wr, 872 op/s; 3531/48954 objects degraded (7.213%); 1421/48954 objects misplaced (2.903%); 7.6 MiB/s, 121 objects/s recovering 2024-09-21T13:57:23.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:22 smithi049 ceph-mon[30080]: osdmap e245: 8 total, 8 up, 7 in 2024-09-21T13:57:23.274 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:23.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:23.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:23.376 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:23.400 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:23.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:23.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:23.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:22 smithi187 ceph-mon[33032]: 6.7 scrub starts 2024-09-21T13:57:23.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:22 smithi187 ceph-mon[33032]: 6.7 scrub ok 2024-09-21T13:57:23.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:22 smithi187 ceph-mon[33032]: 4.1c scrub starts 2024-09-21T13:57:23.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:22 smithi187 ceph-mon[33032]: 4.1c scrub ok 2024-09-21T13:57:23.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:22 smithi187 ceph-mon[33032]: 6.1e scrub starts 2024-09-21T13:57:23.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:22 smithi187 ceph-mon[33032]: 6.1e scrub ok 2024-09-21T13:57:23.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:22 smithi187 ceph-mon[33032]: 3.d scrub starts 2024-09-21T13:57:23.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:22 smithi187 ceph-mon[33032]: 3.d scrub ok 2024-09-21T13:57:23.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:22 smithi187 ceph-mon[33032]: pgmap v410: 148 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 141 active+clean; 1008 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 895 B/s rd, 0 B/s wr, 872 op/s; 3531/48954 objects degraded (7.213%); 1421/48954 objects misplaced (2.903%); 7.6 MiB/s, 121 objects/s recovering 2024-09-21T13:57:23.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:22 smithi187 ceph-mon[33032]: osdmap e245: 8 total, 8 up, 7 in 2024-09-21T13:57:23.502 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:23.531 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:23.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:23.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:23.632 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:23.675 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:23.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:23.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:23.777 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:23.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:23.799 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:23.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:23.900 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:23.925 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:23.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:23.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:24.027 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:24.054 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:24.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:24.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:24.155 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:24.196 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:24.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:24.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:24.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:24 smithi049 ceph-mon[40489]: 6.16 scrub starts 2024-09-21T13:57:24.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:24 smithi049 ceph-mon[40489]: 6.16 scrub ok 2024-09-21T13:57:24.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:24 smithi049 ceph-mon[30080]: 6.16 scrub starts 2024-09-21T13:57:24.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:24 smithi049 ceph-mon[30080]: 6.16 scrub ok 2024-09-21T13:57:24.297 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:24.323 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:24.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:24.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:24.425 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:24.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:23 smithi187 ceph-mon[33032]: 6.16 scrub starts 2024-09-21T13:57:24.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:24 smithi187 ceph-mon[33032]: 6.16 scrub ok 2024-09-21T13:57:24.465 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:24.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:24.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:24.567 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:24.596 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:24.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:24.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:24.697 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:24.736 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:24.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:24.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:24.838 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:24.859 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:24.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:24.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:24.961 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:24.991 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:24.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:24.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:25.093 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:25.119 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:25.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:25.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:25.220 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:25.242 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:25.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:25.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:25.345 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:25.368 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:25.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:25.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:25.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:25 smithi187 ceph-mon[33032]: pgmap v412: 148 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 142 active+clean; 905 MiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.07k op/s; 3203/43995 objects degraded (7.280%); 1273/43995 objects misplaced (2.894%); 11 MiB/s, 168 objects/s recovering 2024-09-21T13:57:25.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:25 smithi187 ceph-mon[33032]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T13:57:25.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:25 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 3203/43995 objects degraded (7.280%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T13:57:25.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:25 smithi187 ceph-mon[33032]: osdmap e246: 8 total, 8 up, 7 in 2024-09-21T13:57:25.469 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:25.493 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:25.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:25.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:25.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:25 smithi049 ceph-mon[40489]: pgmap v412: 148 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 142 active+clean; 905 MiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.07k op/s; 3203/43995 objects degraded (7.280%); 1273/43995 objects misplaced (2.894%); 11 MiB/s, 168 objects/s recovering 2024-09-21T13:57:25.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:25 smithi049 ceph-mon[40489]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T13:57:25.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:25 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 3203/43995 objects degraded (7.280%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T13:57:25.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:25 smithi049 ceph-mon[40489]: osdmap e246: 8 total, 8 up, 7 in 2024-09-21T13:57:25.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:25 smithi049 ceph-mon[30080]: pgmap v412: 148 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 142 active+clean; 905 MiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.07k op/s; 3203/43995 objects degraded (7.280%); 1273/43995 objects misplaced (2.894%); 11 MiB/s, 168 objects/s recovering 2024-09-21T13:57:25.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:25 smithi049 ceph-mon[30080]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T13:57:25.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:25 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 3203/43995 objects degraded (7.280%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T13:57:25.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:25 smithi049 ceph-mon[30080]: osdmap e246: 8 total, 8 up, 7 in 2024-09-21T13:57:25.595 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:57:25.621 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:25.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:25.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:57:25.723 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:25.749 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:25.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:25.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:25.851 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:25.876 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:25.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:25.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:25.978 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:26.005 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:26.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:26.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:26.107 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:26.136 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:26.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:26.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:26.237 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:57:26.271 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:26.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:26.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:57:26.373 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:26.400 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:26.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:26.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:26.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:26 smithi187 ceph-mon[33032]: 4.15 scrub starts 2024-09-21T13:57:26.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:26 smithi187 ceph-mon[33032]: 4.15 scrub ok 2024-09-21T13:57:26.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:26 smithi187 ceph-mon[33032]: osdmap e247: 8 total, 8 up, 7 in 2024-09-21T13:57:26.503 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:57:26.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:26 smithi049 ceph-mon[40489]: 4.15 scrub starts 2024-09-21T13:57:26.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:26 smithi049 ceph-mon[40489]: 4.15 scrub ok 2024-09-21T13:57:26.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:26 smithi049 ceph-mon[40489]: osdmap e247: 8 total, 8 up, 7 in 2024-09-21T13:57:26.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:26 smithi049 ceph-mon[30080]: 4.15 scrub starts 2024-09-21T13:57:26.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:26 smithi049 ceph-mon[30080]: 4.15 scrub ok 2024-09-21T13:57:26.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:26 smithi049 ceph-mon[30080]: osdmap e247: 8 total, 8 up, 7 in 2024-09-21T13:57:26.539 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:26.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:26.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:57:26.641 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:26.668 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:26.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:26.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:26.769 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:57:26.808 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:26.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:26.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:57:26.910 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:26.949 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:26.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:26.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:27.051 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:27.076 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:27.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:27.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:27.178 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:27.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:27.202 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:27.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:27.302 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:57:27.330 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:27.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:27.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:57:27.425 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:27 smithi187 ceph-mon[33032]: 3.7 scrub starts 2024-09-21T13:57:27.425 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:27 smithi187 ceph-mon[33032]: 3.7 scrub ok 2024-09-21T13:57:27.425 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:27 smithi187 ceph-mon[33032]: pgmap v414: 148 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 2 peering, 2 active+remapped+backfill_toofull, 142 active+clean; 763 MiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 653 op/s; 1532/37194 objects degraded (4.119%); 1083/37194 objects misplaced (2.912%); 8.2 MiB/s, 130 objects/s recovering 2024-09-21T13:57:27.432 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:57:27.459 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:27.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:27.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:57:27.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:27 smithi049 ceph-mon[40489]: 3.7 scrub starts 2024-09-21T13:57:27.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:27 smithi049 ceph-mon[40489]: 3.7 scrub ok 2024-09-21T13:57:27.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:27 smithi049 ceph-mon[40489]: pgmap v414: 148 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 2 peering, 2 active+remapped+backfill_toofull, 142 active+clean; 763 MiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 653 op/s; 1532/37194 objects degraded (4.119%); 1083/37194 objects misplaced (2.912%); 8.2 MiB/s, 130 objects/s recovering 2024-09-21T13:57:27.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:27 smithi049 ceph-mon[30080]: 3.7 scrub starts 2024-09-21T13:57:27.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:27 smithi049 ceph-mon[30080]: 3.7 scrub ok 2024-09-21T13:57:27.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:27 smithi049 ceph-mon[30080]: pgmap v414: 148 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 2 peering, 2 active+remapped+backfill_toofull, 142 active+clean; 763 MiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 653 op/s; 1532/37194 objects degraded (4.119%); 1083/37194 objects misplaced (2.912%); 8.2 MiB/s, 130 objects/s recovering 2024-09-21T13:57:27.561 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:27.589 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:27.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:27.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:27.691 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:27.721 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:27.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:27.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:27.781 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:57:27.781 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:57:27.817 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:57:27.818 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:57:27.823 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:27.852 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:57:27.852 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:57:27.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:27.876 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:27.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:27.887 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:57:27.887 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:57:27.921 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:57:27.922 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:57:27.954 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:57:27.954 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:57:27.977 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:57:27.991 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:57:27.991 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:57:28.018 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:28.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:28.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:57:28.026 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:57:28.026 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:57:28.120 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:28.162 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:28.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:28.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:28.193 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:28 smithi049 ceph-mon[40489]: 5.2 scrub starts 2024-09-21T13:57:28.193 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:28 smithi049 ceph-mon[40489]: 5.2 scrub ok 2024-09-21T13:57:28.193 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:28 smithi049 ceph-mon[40489]: 6.1c scrub starts 2024-09-21T13:57:28.193 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:28 smithi049 ceph-mon[40489]: 6.1c scrub ok 2024-09-21T13:57:28.194 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:28 smithi049 ceph-mon[40489]: 3.19 scrub starts 2024-09-21T13:57:28.194 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:28 smithi049 ceph-mon[40489]: 3.19 scrub ok 2024-09-21T13:57:28.194 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:28 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/662021265' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:57:28.194 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:28 smithi049 ceph-mon[30080]: 5.2 scrub starts 2024-09-21T13:57:28.194 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:28 smithi049 ceph-mon[30080]: 5.2 scrub ok 2024-09-21T13:57:28.194 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:28 smithi049 ceph-mon[30080]: 6.1c scrub starts 2024-09-21T13:57:28.194 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:28 smithi049 ceph-mon[30080]: 6.1c scrub ok 2024-09-21T13:57:28.194 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:28 smithi049 ceph-mon[30080]: 3.19 scrub starts 2024-09-21T13:57:28.194 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:28 smithi049 ceph-mon[30080]: 3.19 scrub ok 2024-09-21T13:57:28.194 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:28 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/662021265' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:57:28.264 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:57:28.290 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:28.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:28.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:57:28.368 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T13:57:28.368 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-09-21T13:57:28.368 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-09-21T13:57:28.368 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-09-21T13:57:28.368 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.368 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:163570 7.8 7.330e1638 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e130)", 2024-09-21T13:57:28.368 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:03.335126+0000", 2024-09-21T13:57:28.368 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 325.02980459299999, 2024-09-21T13:57:28.369 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97989348499999995, 2024-09-21T13:57:28.369 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:28.369 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:28.369 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T13:57:28.369 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:57:28.369 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:57:28.369 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:57:28.369 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 163570 2024-09-21T13:57:28.369 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.369 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:28.369 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.369 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:28.370 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.335126+0000", 2024-09-21T13:57:28.370 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:28.370 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.370 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.370 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:28.370 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.335126+0000", 2024-09-21T13:57:28.370 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:28.370 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.370 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.370 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:28.370 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.335127+0000", 2024-09-21T13:57:28.370 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5209999999999999e-06 2024-09-21T13:57:28.370 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.371 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.371 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:28.371 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.335132+0000", 2024-09-21T13:57:28.371 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4730000000000002e-06 2024-09-21T13:57:28.371 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.371 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.371 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:28.371 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.335142+0000", 2024-09-21T13:57:28.371 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8770000000000005e-06 2024-09-21T13:57:28.371 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.371 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.371 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:28.371 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.335151+0000", 2024-09-21T13:57:28.372 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7650000000000005e-06 2024-09-21T13:57:28.372 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.372 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.372 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:28.372 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.336965+0000", 2024-09-21T13:57:28.372 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001813907 2024-09-21T13:57:28.372 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.372 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.372 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:57:28.372 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.336978+0000", 2024-09-21T13:57:28.372 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2685e-05 2024-09-21T13:57:28.372 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.372 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.373 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:28.373 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:04.315001+0000", 2024-09-21T13:57:28.373 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97802330000000004 2024-09-21T13:57:28.373 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.373 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.373 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:28.373 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:04.315019+0000", 2024-09-21T13:57:28.373 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7957000000000001e-05 2024-09-21T13:57:28.373 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:28.373 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:28.373 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:28.373 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.374 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.374 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:163585 7.8 7.9fcafe08 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-21T13:57:28.374 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:03.339560+0000", 2024-09-21T13:57:28.374 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 325.02537041800002, 2024-09-21T13:57:28.374 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97550176099999997, 2024-09-21T13:57:28.374 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:28.374 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:28.374 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T13:57:28.374 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:57:28.374 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:57:28.374 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:57:28.375 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 163585 2024-09-21T13:57:28.375 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.375 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:28.375 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.375 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:28.375 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.339560+0000", 2024-09-21T13:57:28.375 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:28.375 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.375 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.375 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:28.375 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.339560+0000", 2024-09-21T13:57:28.376 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:28.376 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.376 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.376 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:28.376 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.339562+0000", 2024-09-21T13:57:28.376 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6449999999999999e-06 2024-09-21T13:57:28.376 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.376 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.376 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:28.376 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.339568+0000", 2024-09-21T13:57:28.376 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2169999999999999e-06 2024-09-21T13:57:28.376 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.377 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.377 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:28.377 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.339580+0000", 2024-09-21T13:57:28.377 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1846e-05 2024-09-21T13:57:28.377 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.377 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.377 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:28.377 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.339589+0000", 2024-09-21T13:57:28.377 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.798e-06 2024-09-21T13:57:28.377 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.377 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.377 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:28.378 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.342967+0000", 2024-09-21T13:57:28.378 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0033776990000000001 2024-09-21T13:57:28.378 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.378 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.378 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:57:28.378 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.342980+0000", 2024-09-21T13:57:28.378 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2758000000000001e-05 2024-09-21T13:57:28.378 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.378 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.378 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:28.378 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:04.315052+0000", 2024-09-21T13:57:28.378 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97207221799999999 2024-09-21T13:57:28.379 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.379 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.379 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:28.379 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:04.315062+0000", 2024-09-21T13:57:28.379 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.5799999999999998e-06 2024-09-21T13:57:28.379 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:28.379 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:28.379 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:28.379 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.379 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.379 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:163595 7.8 7.9f5fc9f8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-21T13:57:28.379 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:03.360541+0000", 2024-09-21T13:57:28.380 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 325.004389248, 2024-09-21T13:57:28.380 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.95455601800000001, 2024-09-21T13:57:28.380 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:28.380 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:28.380 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T13:57:28.380 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:57:28.380 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:57:28.380 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:57:28.380 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 163595 2024-09-21T13:57:28.380 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.380 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:28.380 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.381 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:28.381 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.360541+0000", 2024-09-21T13:57:28.381 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:28.381 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.381 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.381 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:28.381 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.360541+0000", 2024-09-21T13:57:28.381 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:28.381 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.381 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.381 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:28.381 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.360544+0000", 2024-09-21T13:57:28.382 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.632e-06 2024-09-21T13:57:28.382 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.382 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.382 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:28.382 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.360554+0000", 2024-09-21T13:57:28.382 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0210000000000001e-05 2024-09-21T13:57:28.382 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.382 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.382 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:28.382 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.360579+0000", 2024-09-21T13:57:28.382 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4878000000000001e-05 2024-09-21T13:57:28.382 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.383 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.383 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:28.383 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.360589+0000", 2024-09-21T13:57:28.383 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7960000000000007e-06 2024-09-21T13:57:28.383 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.383 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.383 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:28.383 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.360662+0000", 2024-09-21T13:57:28.383 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3514000000000001e-05 2024-09-21T13:57:28.383 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.383 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.383 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:57:28.384 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.360673+0000", 2024-09-21T13:57:28.384 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1080000000000001e-05 2024-09-21T13:57:28.384 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.384 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.384 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:28.384 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:04.315088+0000", 2024-09-21T13:57:28.384 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.954414859 2024-09-21T13:57:28.384 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.384 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.385 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:28.385 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:04.315097+0000", 2024-09-21T13:57:28.385 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0489999999999993e-06 2024-09-21T13:57:28.385 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:28.385 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:28.385 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:28.385 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.385 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.385 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:163602 7.8 7.3bec8d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-21T13:57:28.386 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:03.369662+0000", 2024-09-21T13:57:28.386 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 324.99526849199998, 2024-09-21T13:57:28.386 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.945473868, 2024-09-21T13:57:28.386 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:28.386 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:28.386 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T13:57:28.386 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:57:28.386 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:57:28.386 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:57:28.386 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 163602 2024-09-21T13:57:28.386 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.386 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:28.387 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.387 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:28.387 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.369662+0000", 2024-09-21T13:57:28.387 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:28.387 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.387 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.387 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:28.387 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.369662+0000", 2024-09-21T13:57:28.387 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:28.388 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.388 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.388 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:28.388 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.369664+0000", 2024-09-21T13:57:28.388 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.136e-06 2024-09-21T13:57:28.388 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.388 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.388 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:28.388 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.369671+0000", 2024-09-21T13:57:28.388 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9110000000000004e-06 2024-09-21T13:57:28.388 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.388 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.389 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:28.389 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.369689+0000", 2024-09-21T13:57:28.389 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7757999999999998e-05 2024-09-21T13:57:28.389 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.389 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.389 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:28.389 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.369700+0000", 2024-09-21T13:57:28.389 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.146e-05 2024-09-21T13:57:28.389 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.389 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.390 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:28.390 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.369779+0000", 2024-09-21T13:57:28.390 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9260999999999999e-05 2024-09-21T13:57:28.390 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.390 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.390 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:57:28.390 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.369793+0000", 2024-09-21T13:57:28.390 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3200000000000001e-05 2024-09-21T13:57:28.390 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.390 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.390 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:28.391 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:04.315126+0000", 2024-09-21T13:57:28.391 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.94533302500000005 2024-09-21T13:57:28.391 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.391 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.391 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:28.391 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:04.315136+0000", 2024-09-21T13:57:28.391 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0117e-05 2024-09-21T13:57:28.391 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:28.391 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:28.391 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:28.391 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.391 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.391 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:201459 7.8 7:1473b567:::benchmark_data_smithi049_76798_object65039:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-21T13:57:28.392 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:41.960649+0000", 2024-09-21T13:57:28.392 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 286.40428132300002, 2024-09-21T13:57:28.392 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.025332715, 2024-09-21T13:57:28.392 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:28.392 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:28.392 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:28.392 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:57:28.392 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:57:28.392 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:57:28.392 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 201459 2024-09-21T13:57:28.392 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.392 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:28.393 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.393 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:28.393 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:41.960649+0000", 2024-09-21T13:57:28.393 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:28.393 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.393 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.393 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:28.393 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:41.960649+0000", 2024-09-21T13:57:28.393 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:28.393 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.393 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.393 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:28.393 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:41.960651+0000", 2024-09-21T13:57:28.394 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7990000000000001e-06 2024-09-21T13:57:28.394 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.394 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.394 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:28.394 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:41.960657+0000", 2024-09-21T13:57:28.394 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2260000000000002e-06 2024-09-21T13:57:28.394 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.394 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.394 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:28.394 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:41.960677+0000", 2024-09-21T13:57:28.394 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9783999999999999e-05 2024-09-21T13:57:28.395 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.395 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.395 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:28.395 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:41.960686+0000", 2024-09-21T13:57:28.395 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0710000000000001e-06 2024-09-21T13:57:28.395 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.395 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.395 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:28.395 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:41.964807+0000", 2024-09-21T13:57:28.395 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0041212560000000002 2024-09-21T13:57:28.395 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.395 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.395 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:57:28.396 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:41.964817+0000", 2024-09-21T13:57:28.396 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0030000000000001e-05 2024-09-21T13:57:28.396 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.396 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.396 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:28.396 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.976018+0000", 2024-09-21T13:57:28.396 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.011200412 2024-09-21T13:57:28.396 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.396 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.396 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:57:28.396 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.976136+0000", 2024-09-21T13:57:28.396 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000118106 2024-09-21T13:57:28.396 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.397 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.397 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-09-21T13:57:28.397 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.976247+0000", 2024-09-21T13:57:28.397 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000111418 2024-09-21T13:57:28.397 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.397 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.397 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:28.397 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.980913+0000", 2024-09-21T13:57:28.397 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0046656850000000001 2024-09-21T13:57:28.397 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.397 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.398 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:28.398 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.980959+0000", 2024-09-21T13:57:28.398 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6402000000000001e-05 2024-09-21T13:57:28.398 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.398 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.398 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:57:28.398 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.985848+0000", 2024-09-21T13:57:28.398 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0048883210000000002 2024-09-21T13:57:28.398 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.398 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.398 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:57:28.398 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.985906+0000", 2024-09-21T13:57:28.399 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8242000000000002e-05 2024-09-21T13:57:28.399 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.399 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.399 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:28.399 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.985982+0000", 2024-09-21T13:57:28.399 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5963000000000004e-05 2024-09-21T13:57:28.399 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:28.399 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:28.399 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:28.399 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.399 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.399 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:201494 7.8 7:1c4e9ccf:::benchmark_data_smithi049_76798_object65074:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-21T13:57:28.399 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:42.006815+0000", 2024-09-21T13:57:28.400 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 286.35811491700002, 2024-09-21T13:57:28.400 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97934932200000002, 2024-09-21T13:57:28.400 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:28.400 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:28.400 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:28.400 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:57:28.400 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:57:28.400 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:57:28.400 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 201494 2024-09-21T13:57:28.400 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.400 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:28.401 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.401 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:28.401 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006815+0000", 2024-09-21T13:57:28.401 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:28.401 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.401 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.401 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:28.401 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006815+0000", 2024-09-21T13:57:28.401 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:28.401 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.401 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.401 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:28.401 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006817+0000", 2024-09-21T13:57:28.402 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.618e-06 2024-09-21T13:57:28.402 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.402 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.402 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:28.402 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006823+0000", 2024-09-21T13:57:28.402 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0859999999999998e-06 2024-09-21T13:57:28.402 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.402 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.402 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:28.402 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006840+0000", 2024-09-21T13:57:28.402 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7149000000000001e-05 2024-09-21T13:57:28.403 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.403 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.403 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:28.403 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006848+0000", 2024-09-21T13:57:28.403 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9340000000000006e-06 2024-09-21T13:57:28.403 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.403 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.403 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:28.403 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006949+0000", 2024-09-21T13:57:28.403 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010043399999999999 2024-09-21T13:57:28.403 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.403 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.404 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:57:28.404 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006957+0000", 2024-09-21T13:57:28.404 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6580000000000008e-06 2024-09-21T13:57:28.404 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.404 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.404 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:28.404 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.979032+0000", 2024-09-21T13:57:28.404 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97207428699999998 2024-09-21T13:57:28.404 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.404 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.404 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:57:28.405 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.979198+0000", 2024-09-21T13:57:28.405 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016622800000000001 2024-09-21T13:57:28.405 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.405 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.405 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-09-21T13:57:28.405 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.979306+0000", 2024-09-21T13:57:28.405 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000108356 2024-09-21T13:57:28.405 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.405 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.405 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:28.405 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.980997+0000", 2024-09-21T13:57:28.405 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0016909640000000001 2024-09-21T13:57:28.405 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.406 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.406 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:28.406 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.981060+0000", 2024-09-21T13:57:28.406 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2516999999999998e-05 2024-09-21T13:57:28.406 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.406 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.406 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:57:28.406 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986075+0000", 2024-09-21T13:57:28.406 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.005014971 2024-09-21T13:57:28.406 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.406 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.407 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:57:28.407 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986121+0000", 2024-09-21T13:57:28.407 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6313999999999997e-05 2024-09-21T13:57:28.407 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.407 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.407 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:28.407 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986165+0000", 2024-09-21T13:57:28.407 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3806e-05 2024-09-21T13:57:28.407 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:28.407 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:28.407 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:28.407 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.407 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.408 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:201495 7.8 7:18dd0180:::benchmark_data_smithi049_76798_object65075:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-21T13:57:28.408 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:42.006872+0000", 2024-09-21T13:57:28.408 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 286.35805783799998, 2024-09-21T13:57:28.408 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97938486800000002, 2024-09-21T13:57:28.408 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:28.408 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:28.408 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:28.408 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:57:28.408 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:57:28.408 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:57:28.408 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 201495 2024-09-21T13:57:28.408 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.409 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:28.409 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.409 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:28.409 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006872+0000", 2024-09-21T13:57:28.409 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:28.409 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.409 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.409 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:28.409 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006872+0000", 2024-09-21T13:57:28.409 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:28.409 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.409 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.410 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:28.410 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006873+0000", 2024-09-21T13:57:28.410 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4899999999999995e-07 2024-09-21T13:57:28.410 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.410 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.410 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:28.410 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006878+0000", 2024-09-21T13:57:28.410 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0429999999999998e-06 2024-09-21T13:57:28.410 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.410 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.410 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:28.410 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006886+0000", 2024-09-21T13:57:28.410 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3399999999999998e-06 2024-09-21T13:57:28.411 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.411 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.411 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:28.411 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006894+0000", 2024-09-21T13:57:28.411 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1359999999999999e-06 2024-09-21T13:57:28.411 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.411 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.411 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:28.411 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006982+0000", 2024-09-21T13:57:28.411 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.8022000000000005e-05 2024-09-21T13:57:28.411 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.411 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.412 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:57:28.412 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006988+0000", 2024-09-21T13:57:28.412 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3470000000000004e-06 2024-09-21T13:57:28.412 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.412 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.412 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:28.412 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.981113+0000", 2024-09-21T13:57:28.412 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97412548499999996 2024-09-21T13:57:28.412 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.412 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.412 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:57:28.412 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.981230+0000", 2024-09-21T13:57:28.412 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000116666 2024-09-21T13:57:28.413 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.413 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.413 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-09-21T13:57:28.413 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.981349+0000", 2024-09-21T13:57:28.413 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000118572 2024-09-21T13:57:28.413 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.413 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.413 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:28.413 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.985825+0000", 2024-09-21T13:57:28.413 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0044766739999999999 2024-09-21T13:57:28.413 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.413 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.414 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:28.414 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986063+0000", 2024-09-21T13:57:28.414 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00023781299999999999 2024-09-21T13:57:28.414 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.414 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.414 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:57:28.414 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986172+0000", 2024-09-21T13:57:28.414 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010923499999999999 2024-09-21T13:57:28.414 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.414 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.414 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:57:28.414 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986236+0000", 2024-09-21T13:57:28.415 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3606000000000001e-05 2024-09-21T13:57:28.415 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.415 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.415 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:28.415 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986257+0000", 2024-09-21T13:57:28.415 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1379999999999999e-05 2024-09-21T13:57:28.415 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:28.415 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:28.415 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:28.415 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.415 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.415 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:201496 7.8 7:10f836b0:::benchmark_data_smithi049_76798_object65076:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-21T13:57:28.416 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:42.012648+0000", 2024-09-21T13:57:28.416 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 286.35228186799998, 2024-09-21T13:57:28.416 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97380752599999998, 2024-09-21T13:57:28.416 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:28.416 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:28.416 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:28.416 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:57:28.416 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:57:28.416 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:57:28.416 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 201496 2024-09-21T13:57:28.416 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.416 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:28.416 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.417 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:28.417 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.012648+0000", 2024-09-21T13:57:28.417 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:28.418 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:28.418 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.418 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.418 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:28.418 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.012648+0000", 2024-09-21T13:57:28.418 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:28.418 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.419 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.419 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:28.419 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.012650+0000", 2024-09-21T13:57:28.419 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.513e-06 2024-09-21T13:57:28.419 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.419 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.419 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:28.419 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.012656+0000", 2024-09-21T13:57:28.419 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7300000000000002e-06 2024-09-21T13:57:28.419 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.419 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.420 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:28.420 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.012664+0000", 2024-09-21T13:57:28.420 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6090000000000001e-06 2024-09-21T13:57:28.420 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.420 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.420 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:28.420 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.012670+0000", 2024-09-21T13:57:28.420 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5829999999999997e-06 2024-09-21T13:57:28.420 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.420 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.420 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:28.421 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.012740+0000", 2024-09-21T13:57:28.421 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9834999999999998e-05 2024-09-21T13:57:28.421 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.421 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.421 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:57:28.421 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.012749+0000", 2024-09-21T13:57:28.421 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9889999999999998e-06 2024-09-21T13:57:28.421 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.421 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.421 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:28.421 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.983365+0000", 2024-09-21T13:57:28.421 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97061665600000002 2024-09-21T13:57:28.422 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.422 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.422 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:57:28.422 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.983473+0000", 2024-09-21T13:57:28.422 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000107507 2024-09-21T13:57:28.422 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.422 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.422 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-09-21T13:57:28.422 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.983595+0000", 2024-09-21T13:57:28.422 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000121614 2024-09-21T13:57:28.422 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.423 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.423 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:57:28.423 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986265+0000", 2024-09-21T13:57:28.423 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002670113 2024-09-21T13:57:28.423 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.423 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.423 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:28.423 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986301+0000", 2024-09-21T13:57:28.423 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6130999999999997e-05 2024-09-21T13:57:28.423 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.423 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.424 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:28.424 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986370+0000", 2024-09-21T13:57:28.424 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8994999999999999e-05 2024-09-21T13:57:28.424 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.424 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.424 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:57:28.424 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986417+0000", 2024-09-21T13:57:28.424 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7691000000000001e-05 2024-09-21T13:57:28.424 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.424 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.424 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:28.424 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986456+0000", 2024-09-21T13:57:28.425 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8559999999999997e-05 2024-09-21T13:57:28.425 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:28.425 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:28.425 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:28.425 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.425 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.425 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:201499 7.8 7:1329afa8:::benchmark_data_smithi049_76798_object65079:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-21T13:57:28.425 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:42.026955+0000", 2024-09-21T13:57:28.425 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 286.337975293, 2024-09-21T13:57:28.425 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97876118000000001, 2024-09-21T13:57:28.425 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:28.425 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:28.426 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:28.426 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:57:28.426 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:57:28.426 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:57:28.426 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 201499 2024-09-21T13:57:28.426 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.426 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:28.426 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.426 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:28.426 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.026955+0000", 2024-09-21T13:57:28.426 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:28.427 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.427 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.427 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:28.427 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.026955+0000", 2024-09-21T13:57:28.427 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:28.427 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.427 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.427 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:28.427 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.026957+0000", 2024-09-21T13:57:28.427 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7999999999999999e-06 2024-09-21T13:57:28.427 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.427 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.428 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:28.428 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.026962+0000", 2024-09-21T13:57:28.428 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9400000000000001e-06 2024-09-21T13:57:28.428 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.428 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.428 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:28.428 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.026977+0000", 2024-09-21T13:57:28.428 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5087e-05 2024-09-21T13:57:28.428 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.428 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.428 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:28.428 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.026983+0000", 2024-09-21T13:57:28.429 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0090000000000004e-06 2024-09-21T13:57:28.429 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.429 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.429 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:28.429 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027044+0000", 2024-09-21T13:57:28.429 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1543000000000004e-05 2024-09-21T13:57:28.429 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.429 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.429 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:57:28.429 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027051+0000", 2024-09-21T13:57:28.429 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4590000000000003e-06 2024-09-21T13:57:28.429 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.430 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.430 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:28.430 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986512+0000", 2024-09-21T13:57:28.430 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.95946141399999996 2024-09-21T13:57:28.430 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.430 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.430 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:57:28.430 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986631+0000", 2024-09-21T13:57:28.430 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011896300000000001 2024-09-21T13:57:28.430 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.430 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.430 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-09-21T13:57:28.430 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986738+0000", 2024-09-21T13:57:28.431 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000106766 2024-09-21T13:57:28.431 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.431 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.431 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:28.431 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.994021+0000", 2024-09-21T13:57:28.431 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0072827329999999996 2024-09-21T13:57:28.431 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.431 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.431 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:28.431 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.994059+0000", 2024-09-21T13:57:28.431 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8053999999999999e-05 2024-09-21T13:57:28.431 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.431 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.432 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:57:28.432 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.005582+0000", 2024-09-21T13:57:28.432 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.011523342000000001 2024-09-21T13:57:28.432 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.432 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.432 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:57:28.432 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.005671+0000", 2024-09-21T13:57:28.432 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.8708999999999994e-05 2024-09-21T13:57:28.432 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.432 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.432 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:28.433 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.005716+0000", 2024-09-21T13:57:28.433 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5361000000000002e-05 2024-09-21T13:57:28.433 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:28.433 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:28.433 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:28.433 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.433 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.433 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:201500 7.8 7:19fe3d31:::benchmark_data_smithi049_76798_object65080:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-21T13:57:28.433 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:42.027000+0000", 2024-09-21T13:57:28.433 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 286.33792985299999, 2024-09-21T13:57:28.433 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97887608500000001, 2024-09-21T13:57:28.433 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:28.433 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:28.434 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:28.434 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:57:28.434 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:57:28.434 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:57:28.434 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 201500 2024-09-21T13:57:28.434 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.434 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:28.434 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.434 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:28.434 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027000+0000", 2024-09-21T13:57:28.435 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:28.435 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.435 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.435 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:28.435 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027000+0000", 2024-09-21T13:57:28.435 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:28.435 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.435 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.435 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:28.435 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027001+0000", 2024-09-21T13:57:28.435 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3200000000000001e-07 2024-09-21T13:57:28.435 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.435 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.436 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:28.436 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027003+0000", 2024-09-21T13:57:28.436 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8980000000000001e-06 2024-09-21T13:57:28.436 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.436 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.436 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:28.436 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027007+0000", 2024-09-21T13:57:28.436 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4290000000000003e-06 2024-09-21T13:57:28.436 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.436 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.436 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:28.436 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027010+0000", 2024-09-21T13:57:28.437 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3419999999999999e-06 2024-09-21T13:57:28.437 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.437 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.437 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:28.437 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027064+0000", 2024-09-21T13:57:28.437 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3474000000000003e-05 2024-09-21T13:57:28.437 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.437 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.437 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:57:28.437 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027067+0000", 2024-09-21T13:57:28.437 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.932e-06 2024-09-21T13:57:28.437 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.438 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.438 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:28.438 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.989128+0000", 2024-09-21T13:57:28.438 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.96206106899999999 2024-09-21T13:57:28.438 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.438 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.438 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:57:28.438 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.990066+0000", 2024-09-21T13:57:28.438 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00093830900000000002 2024-09-21T13:57:28.438 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.438 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.439 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-09-21T13:57:28.439 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.990172+0000", 2024-09-21T13:57:28.439 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000105271 2024-09-21T13:57:28.439 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.439 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.439 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:28.439 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.996356+0000", 2024-09-21T13:57:28.439 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0061840100000000002 2024-09-21T13:57:28.439 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.439 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.439 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:28.439 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.996388+0000", 2024-09-21T13:57:28.440 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2539000000000001e-05 2024-09-21T13:57:28.440 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.440 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.440 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:57:28.440 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.005774+0000", 2024-09-21T13:57:28.440 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0093859679999999997 2024-09-21T13:57:28.440 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.440 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.440 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:57:28.440 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.005838+0000", 2024-09-21T13:57:28.440 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3724000000000002e-05 2024-09-21T13:57:28.440 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.441 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.441 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:28.441 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.005877+0000", 2024-09-21T13:57:28.441 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8788000000000002e-05 2024-09-21T13:57:28.441 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:28.441 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:28.441 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:28.441 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.441 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.441 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:201501 7.8 7:15f77bbd:::benchmark_data_smithi049_76798_object65081:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-21T13:57:28.441 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:42.027294+0000", 2024-09-21T13:57:28.441 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 286.33763588800002, 2024-09-21T13:57:28.442 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97870270599999998, 2024-09-21T13:57:28.442 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:28.442 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:28.442 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:28.442 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:57:28.442 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:57:28.442 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:57:28.442 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 201501 2024-09-21T13:57:28.442 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.442 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:28.442 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.442 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:28.443 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027294+0000", 2024-09-21T13:57:28.443 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:28.443 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.443 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.443 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:28.443 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027294+0000", 2024-09-21T13:57:28.443 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:28.443 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.443 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.443 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:28.443 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027295+0000", 2024-09-21T13:57:28.444 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7899999999999997e-07 2024-09-21T13:57:28.444 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.444 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.444 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:28.444 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027299+0000", 2024-09-21T13:57:28.444 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6449999999999998e-06 2024-09-21T13:57:28.444 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.444 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.444 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:28.444 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027310+0000", 2024-09-21T13:57:28.444 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0699999999999999e-05 2024-09-21T13:57:28.444 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.445 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.445 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:28.445 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027315+0000", 2024-09-21T13:57:28.445 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9729999999999997e-06 2024-09-21T13:57:28.445 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.445 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.445 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:28.445 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027363+0000", 2024-09-21T13:57:28.445 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8096000000000002e-05 2024-09-21T13:57:28.445 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.445 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.445 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:57:28.445 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027367+0000", 2024-09-21T13:57:28.446 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2930000000000002e-06 2024-09-21T13:57:28.446 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.446 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.446 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:28.446 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.992098+0000", 2024-09-21T13:57:28.446 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.96473078800000001 2024-09-21T13:57:28.446 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.446 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.446 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:57:28.446 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.992160+0000", 2024-09-21T13:57:28.446 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2133999999999998e-05 2024-09-21T13:57:28.447 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.447 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.447 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-09-21T13:57:28.447 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.992220+0000", 2024-09-21T13:57:28.447 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9911999999999998e-05 2024-09-21T13:57:28.447 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.447 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.447 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:28.447 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.003058+0000", 2024-09-21T13:57:28.447 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.010838441000000001 2024-09-21T13:57:28.447 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.447 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.448 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:28.448 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.003121+0000", 2024-09-21T13:57:28.448 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3118000000000001e-05 2024-09-21T13:57:28.448 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.448 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.448 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:57:28.448 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.005887+0000", 2024-09-21T13:57:28.448 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0027659500000000001 2024-09-21T13:57:28.448 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.448 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.448 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:57:28.448 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.005958+0000", 2024-09-21T13:57:28.449 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0842999999999999e-05 2024-09-21T13:57:28.449 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.449 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.449 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:28.449 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.005997+0000", 2024-09-21T13:57:28.449 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9033999999999999e-05 2024-09-21T13:57:28.449 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:28.449 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:28.449 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:28.449 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.449 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.449 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:201503 7.8 7:1bb4326b:::benchmark_data_smithi049_76798_object65083:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-21T13:57:28.450 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:42.027327+0000", 2024-09-21T13:57:28.450 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 286.337603812, 2024-09-21T13:57:28.450 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97877131900000003, 2024-09-21T13:57:28.450 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:28.450 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:28.451 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:28 smithi187 ceph-mon[33032]: 5.2 scrub starts 2024-09-21T13:57:28.451 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:28 smithi187 ceph-mon[33032]: 5.2 scrub ok 2024-09-21T13:57:28.451 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:28 smithi187 ceph-mon[33032]: 6.1c scrub starts 2024-09-21T13:57:28.451 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:28 smithi187 ceph-mon[33032]: 6.1c scrub ok 2024-09-21T13:57:28.451 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:28 smithi187 ceph-mon[33032]: 3.19 scrub starts 2024-09-21T13:57:28.451 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:28 smithi187 ceph-mon[33032]: 3.19 scrub ok 2024-09-21T13:57:28.451 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:28 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/662021265' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:57:28.452 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:28.452 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:57:28.452 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:57:28.452 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:57:28.452 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 201503 2024-09-21T13:57:28.452 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.452 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:28.452 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.452 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:28.453 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027327+0000", 2024-09-21T13:57:28.453 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:28.453 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.453 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.453 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:28.453 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027327+0000", 2024-09-21T13:57:28.453 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:28.453 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.453 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.453 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:28.453 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027327+0000", 2024-09-21T13:57:28.453 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4799999999999999e-07 2024-09-21T13:57:28.453 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.454 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.454 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:28.454 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027329+0000", 2024-09-21T13:57:28.454 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7400000000000001e-06 2024-09-21T13:57:28.454 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.454 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.454 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:28.454 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027333+0000", 2024-09-21T13:57:28.454 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0790000000000002e-06 2024-09-21T13:57:28.454 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.454 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.454 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:28.455 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027336+0000", 2024-09-21T13:57:28.455 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1190000000000001e-06 2024-09-21T13:57:28.455 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.455 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.455 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:28.455 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027379+0000", 2024-09-21T13:57:28.455 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3000000000000002e-05 2024-09-21T13:57:28.455 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.455 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.455 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:57:28.455 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027382+0000", 2024-09-21T13:57:28.455 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9399999999999998e-06 2024-09-21T13:57:28.455 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.456 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.456 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:28.456 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.994085+0000", 2024-09-21T13:57:28.456 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.96670302799999996 2024-09-21T13:57:28.456 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.456 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.456 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:57:28.456 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.994181+0000", 2024-09-21T13:57:28.456 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6340000000000003e-05 2024-09-21T13:57:28.456 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.456 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.456 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-09-21T13:57:28.457 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.994285+0000", 2024-09-21T13:57:28.457 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000103947 2024-09-21T13:57:28.457 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.457 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.457 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:28.457 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.003091+0000", 2024-09-21T13:57:28.457 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0088057599999999993 2024-09-21T13:57:28.457 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.457 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.457 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:28.457 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.003150+0000", 2024-09-21T13:57:28.457 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9670000000000003e-05 2024-09-21T13:57:28.457 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.457 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.458 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:57:28.458 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.006006+0000", 2024-09-21T13:57:28.458 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002855734 2024-09-21T13:57:28.458 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.458 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.458 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:57:28.458 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.006056+0000", 2024-09-21T13:57:28.458 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9619e-05 2024-09-21T13:57:28.458 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.458 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.458 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:28.458 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.006098+0000", 2024-09-21T13:57:28.459 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1995e-05 2024-09-21T13:57:28.459 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:28.459 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:28.459 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:28.459 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.459 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.459 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:201514 7.8 7:1c8e529c:::benchmark_data_smithi049_76798_object65094:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-21T13:57:28.459 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:42.044200+0000", 2024-09-21T13:57:28.459 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 286.32073070500002, 2024-09-21T13:57:28.459 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.96199924699999995, 2024-09-21T13:57:28.459 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:28.459 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:28.460 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:28.460 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:57:28.460 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:57:28.460 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:57:28.460 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 201514 2024-09-21T13:57:28.460 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.460 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:28.460 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.460 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:28.460 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.044200+0000", 2024-09-21T13:57:28.460 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:28.460 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.461 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.461 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:28.461 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.044200+0000", 2024-09-21T13:57:28.461 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:28.461 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.461 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.461 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:28.461 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.044201+0000", 2024-09-21T13:57:28.461 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7179999999999999e-06 2024-09-21T13:57:28.461 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.461 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.461 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:28.461 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.044208+0000", 2024-09-21T13:57:28.461 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1800000000000001e-06 2024-09-21T13:57:28.462 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.462 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.462 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:28.462 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.044226+0000", 2024-09-21T13:57:28.462 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8216999999999999e-05 2024-09-21T13:57:28.462 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.462 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.462 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:28.462 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.044235+0000", 2024-09-21T13:57:28.462 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1020000000000003e-06 2024-09-21T13:57:28.462 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.462 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.463 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:28.463 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.044482+0000", 2024-09-21T13:57:28.463 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00024732699999999999 2024-09-21T13:57:28.463 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.463 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.463 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:57:28.463 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.044499+0000", 2024-09-21T13:57:28.463 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7129000000000002e-05 2024-09-21T13:57:28.463 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.463 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.463 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:28.463 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.996416+0000", 2024-09-21T13:57:28.464 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.95191657200000002 2024-09-21T13:57:28.464 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.464 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.464 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:57:28.464 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.996528+0000", 2024-09-21T13:57:28.464 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000112118 2024-09-21T13:57:28.464 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.464 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.464 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-09-21T13:57:28.464 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.996652+0000", 2024-09-21T13:57:28.464 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012430500000000001 2024-09-21T13:57:28.464 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.465 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.465 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:28.465 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.003180+0000", 2024-09-21T13:57:28.465 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.006528056 2024-09-21T13:57:28.465 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.465 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.465 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:28.465 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.005760+0000", 2024-09-21T13:57:28.465 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0025797670000000002 2024-09-21T13:57:28.465 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.465 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.465 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:57:28.465 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.006108+0000", 2024-09-21T13:57:28.466 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00034834500000000001 2024-09-21T13:57:28.466 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.466 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.466 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:57:28.466 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.006155+0000", 2024-09-21T13:57:28.466 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6513e-05 2024-09-21T13:57:28.466 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.466 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.466 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:28.466 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.006199+0000", 2024-09-21T13:57:28.466 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3897999999999999e-05 2024-09-21T13:57:28.466 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:28.467 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:28.467 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:28.467 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.467 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.467 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:201539 7.8 7:1afa34ae:::benchmark_data_smithi049_76798_object65119:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-21T13:57:28.467 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:42.081266+0000", 2024-09-21T13:57:28.467 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 286.28366459699998, 2024-09-21T13:57:28.467 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.92917915600000001, 2024-09-21T13:57:28.467 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:28.467 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:28.467 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:28.467 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:57:28.468 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:57:28.468 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:57:28.468 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 201539 2024-09-21T13:57:28.468 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.468 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:28.468 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.468 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:28.468 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.081266+0000", 2024-09-21T13:57:28.468 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:28.468 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.468 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.468 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:28.469 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.081266+0000", 2024-09-21T13:57:28.469 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:28.469 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.469 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.469 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:28.469 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.081267+0000", 2024-09-21T13:57:28.469 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6559999999999999e-06 2024-09-21T13:57:28.469 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.469 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.469 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:28.469 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.081273+0000", 2024-09-21T13:57:28.469 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0839999999999997e-06 2024-09-21T13:57:28.469 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.470 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.470 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:28.470 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.081291+0000", 2024-09-21T13:57:28.470 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7827999999999999e-05 2024-09-21T13:57:28.470 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.470 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.470 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:28.470 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.081300+0000", 2024-09-21T13:57:28.470 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.7190000000000007e-06 2024-09-21T13:57:28.470 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.470 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.470 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:28.471 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.081360+0000", 2024-09-21T13:57:28.471 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9914000000000003e-05 2024-09-21T13:57:28.471 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.471 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.471 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:57:28.471 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.081369+0000", 2024-09-21T13:57:28.471 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1630000000000002e-06 2024-09-21T13:57:28.471 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.471 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.471 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:28.471 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.998700+0000", 2024-09-21T13:57:28.471 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.91733049300000002 2024-09-21T13:57:28.472 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.472 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.472 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:57:28.472 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.998798+0000", 2024-09-21T13:57:28.472 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8494999999999998e-05 2024-09-21T13:57:28.472 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.472 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.472 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-09-21T13:57:28.472 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.998903+0000", 2024-09-21T13:57:28.472 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000104782 2024-09-21T13:57:28.472 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.473 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.473 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:57:28.473 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.006208+0000", 2024-09-21T13:57:28.473 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0073052270000000001 2024-09-21T13:57:28.473 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.473 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.473 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:28.473 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.006259+0000", 2024-09-21T13:57:28.473 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1205999999999997e-05 2024-09-21T13:57:28.473 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.473 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.473 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:28.474 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.010350+0000", 2024-09-21T13:57:28.474 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0040908560000000004 2024-09-21T13:57:28.474 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.474 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.474 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:57:28.474 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.010400+0000", 2024-09-21T13:57:28.474 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9901000000000002e-05 2024-09-21T13:57:28.474 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.474 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.474 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:28.474 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.010445+0000", 2024-09-21T13:57:28.474 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4832e-05 2024-09-21T13:57:28.475 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:28.475 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:28.475 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:28.475 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.475 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.475 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:201541 7.8 7:135037ad:::benchmark_data_smithi049_76798_object65121:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-21T13:57:28.475 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:42.083222+0000", 2024-09-21T13:57:28.475 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 286.28170834600002, 2024-09-21T13:57:28.475 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.92733551700000005, 2024-09-21T13:57:28.475 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:28.475 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:28.476 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:28.476 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:57:28.476 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:57:28.476 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:57:28.476 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 201541 2024-09-21T13:57:28.476 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.476 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:28.476 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.476 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:28.476 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.083222+0000", 2024-09-21T13:57:28.476 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:28.477 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.477 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.477 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:28.477 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.083222+0000", 2024-09-21T13:57:28.477 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:28.477 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.477 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.477 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:28.477 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.083223+0000", 2024-09-21T13:57:28.477 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0020000000000001e-06 2024-09-21T13:57:28.477 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.477 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.477 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:28.478 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.083227+0000", 2024-09-21T13:57:28.478 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.636e-06 2024-09-21T13:57:28.478 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.478 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.478 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:28.478 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.083236+0000", 2024-09-21T13:57:28.478 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0180000000000007e-06 2024-09-21T13:57:28.478 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.478 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.478 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:28.478 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.083242+0000", 2024-09-21T13:57:28.478 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.891e-06 2024-09-21T13:57:28.479 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.479 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.479 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:28.479 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.083385+0000", 2024-09-21T13:57:28.479 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014374099999999999 2024-09-21T13:57:28.479 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.479 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.479 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:57:28.479 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.083394+0000", 2024-09-21T13:57:28.479 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0839999999999998e-06 2024-09-21T13:57:28.479 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.479 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.479 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:28.480 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.000839+0000", 2024-09-21T13:57:28.480 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.917444861 2024-09-21T13:57:28.480 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.480 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.480 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:57:28.480 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.000934+0000", 2024-09-21T13:57:28.480 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4943000000000002e-05 2024-09-21T13:57:28.480 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.480 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.480 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-09-21T13:57:28.480 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.001036+0000", 2024-09-21T13:57:28.480 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000101817 2024-09-21T13:57:28.481 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.481 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.481 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:57:28.481 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.006215+0000", 2024-09-21T13:57:28.481 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0051790020000000003 2024-09-21T13:57:28.481 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.481 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.481 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:28.481 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.006324+0000", 2024-09-21T13:57:28.481 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010858599999999999 2024-09-21T13:57:28.481 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.481 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.481 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:28.482 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.010481+0000", 2024-09-21T13:57:28.482 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0041575300000000004 2024-09-21T13:57:28.482 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.482 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.482 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:57:28.482 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.010528+0000", 2024-09-21T13:57:28.482 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6418000000000002e-05 2024-09-21T13:57:28.482 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.482 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.482 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:28.482 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.010558+0000", 2024-09-21T13:57:28.482 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9988000000000002e-05 2024-09-21T13:57:28.483 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:28.483 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:28.483 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:28.483 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.483 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.483 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:229964 7.e 7:7be4d3ef:::benchmark_data_smithi049_76798_object93544:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-21T13:57:28.483 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:53:16.693433+0000", 2024-09-21T13:57:28.483 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 251.671497781, 2024-09-21T13:57:28.483 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.99823727600000001, 2024-09-21T13:57:28.483 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:28.483 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:28.483 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:28.484 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:57:28.484 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:57:28.484 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:57:28.484 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 229964 2024-09-21T13:57:28.484 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.484 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:28.484 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.484 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:28.484 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.693433+0000", 2024-09-21T13:57:28.484 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:28.484 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.484 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.485 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:28.485 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.693433+0000", 2024-09-21T13:57:28.485 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:28.485 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.485 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.485 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:28.485 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.693434+0000", 2024-09-21T13:57:28.485 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3799999999999999e-06 2024-09-21T13:57:28.485 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.485 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.485 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:28.485 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.693438+0000", 2024-09-21T13:57:28.486 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.515e-06 2024-09-21T13:57:28.486 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.486 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.486 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:28.486 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.693455+0000", 2024-09-21T13:57:28.486 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6614000000000001e-05 2024-09-21T13:57:28.486 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.486 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.486 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:28.486 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.693463+0000", 2024-09-21T13:57:28.486 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9850000000000006e-06 2024-09-21T13:57:28.486 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.487 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.487 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:28.487 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.693542+0000", 2024-09-21T13:57:28.487 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8942000000000004e-05 2024-09-21T13:57:28.487 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.487 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.487 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:57:28.487 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.693550+0000", 2024-09-21T13:57:28.487 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7379999999999994e-06 2024-09-21T13:57:28.487 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.488 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.488 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:28.488 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.686321+0000", 2024-09-21T13:57:28.488 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.99277167399999999 2024-09-21T13:57:28.488 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.488 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.488 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:57:28.488 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.686413+0000", 2024-09-21T13:57:28.488 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1822000000000003e-05 2024-09-21T13:57:28.488 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.488 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.488 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,5", 2024-09-21T13:57:28.489 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.686547+0000", 2024-09-21T13:57:28.489 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000133516 2024-09-21T13:57:28.489 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.489 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.489 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:28.489 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.691512+0000", 2024-09-21T13:57:28.489 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0049651410000000002 2024-09-21T13:57:28.489 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.489 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.489 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:28.489 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.691558+0000", 2024-09-21T13:57:28.490 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6047999999999997e-05 2024-09-21T13:57:28.490 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.490 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.490 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:57:28.490 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.691610+0000", 2024-09-21T13:57:28.490 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1644000000000002e-05 2024-09-21T13:57:28.490 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.490 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.490 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:57:28.490 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.691642+0000", 2024-09-21T13:57:28.490 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2542000000000001e-05 2024-09-21T13:57:28.490 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.491 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.491 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:28.491 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.691670+0000", 2024-09-21T13:57:28.491 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.7715000000000001e-05 2024-09-21T13:57:28.491 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:28.491 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:28.491 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:28.491 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.491 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.491 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:230040 7.e 7:74df0b23:::benchmark_data_smithi049_76798_object93620:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-21T13:57:28.491 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:53:16.758488+0000", 2024-09-21T13:57:28.491 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 251.606442724, 2024-09-21T13:57:28.492 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.93513229499999995, 2024-09-21T13:57:28.492 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:28.492 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:28.492 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:28.492 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:57:28.492 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:57:28.492 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:57:28.492 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 230040 2024-09-21T13:57:28.492 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.492 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:28.492 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.492 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:28.492 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.758488+0000", 2024-09-21T13:57:28.493 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:28.493 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.493 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.493 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:28.493 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.758488+0000", 2024-09-21T13:57:28.493 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:28.493 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.493 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.493 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:28.493 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.758502+0000", 2024-09-21T13:57:28.493 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4207e-05 2024-09-21T13:57:28.493 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.493 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.494 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:28.494 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.758507+0000", 2024-09-21T13:57:28.494 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9429999999999999e-06 2024-09-21T13:57:28.494 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.494 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.494 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:28.494 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.758518+0000", 2024-09-21T13:57:28.494 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1447999999999999e-05 2024-09-21T13:57:28.494 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.494 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.495 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:28.495 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.758525+0000", 2024-09-21T13:57:28.495 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2389999999999996e-06 2024-09-21T13:57:28.495 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.495 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.495 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:28.495 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.758589+0000", 2024-09-21T13:57:28.495 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3262000000000005e-05 2024-09-21T13:57:28.495 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.495 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.495 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:57:28.495 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.758597+0000", 2024-09-21T13:57:28.495 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4100000000000008e-06 2024-09-21T13:57:28.496 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.496 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.496 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:28.496 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.689523+0000", 2024-09-21T13:57:28.496 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.93092584099999998 2024-09-21T13:57:28.496 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.496 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.496 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:57:28.496 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.689620+0000", 2024-09-21T13:57:28.496 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7081000000000001e-05 2024-09-21T13:57:28.496 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.496 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.496 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,5", 2024-09-21T13:57:28.496 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.689705+0000", 2024-09-21T13:57:28.497 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5074000000000006e-05 2024-09-21T13:57:28.497 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.497 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.497 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:28.497 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.691602+0000", 2024-09-21T13:57:28.497 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001896951 2024-09-21T13:57:28.497 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.497 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.497 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:28.497 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.693564+0000", 2024-09-21T13:57:28.497 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0019616630000000002 2024-09-21T13:57:28.497 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.497 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.498 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:57:28.498 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.693572+0000", 2024-09-21T13:57:28.498 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9160000000000001e-06 2024-09-21T13:57:28.498 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.498 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.498 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:57:28.498 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.693597+0000", 2024-09-21T13:57:28.498 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5812999999999998e-05 2024-09-21T13:57:28.498 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.498 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.498 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:28.498 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.693620+0000", 2024-09-21T13:57:28.499 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2447e-05 2024-09-21T13:57:28.499 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:28.500 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:28.500 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:28.500 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.500 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.500 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:230048 7.e 7:7e0d246c:::benchmark_data_smithi049_76798_object93628:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-21T13:57:28.500 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:53:16.763844+0000", 2024-09-21T13:57:28.500 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 251.601085845, 2024-09-21T13:57:28.501 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.93491194899999996, 2024-09-21T13:57:28.501 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:28.501 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:28.501 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:28.501 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:57:28.501 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:57:28.501 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:57:28.501 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 230048 2024-09-21T13:57:28.501 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.501 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:28.501 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.501 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:28.502 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.763844+0000", 2024-09-21T13:57:28.502 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:28.502 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.502 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.502 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:28.502 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.763844+0000", 2024-09-21T13:57:28.502 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:28.502 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.502 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.502 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:28.503 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.763846+0000", 2024-09-21T13:57:28.503 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6330000000000001e-06 2024-09-21T13:57:28.503 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.503 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.503 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:28.503 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.763852+0000", 2024-09-21T13:57:28.503 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3839999999999996e-06 2024-09-21T13:57:28.503 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.503 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.503 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:28.503 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.763915+0000", 2024-09-21T13:57:28.503 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3306999999999995e-05 2024-09-21T13:57:28.504 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.504 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.504 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:28.504 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.764047+0000", 2024-09-21T13:57:28.504 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013207599999999999 2024-09-21T13:57:28.504 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.504 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.504 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:28.504 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.764104+0000", 2024-09-21T13:57:28.504 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6758000000000001e-05 2024-09-21T13:57:28.504 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.504 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.505 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:57:28.505 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.764113+0000", 2024-09-21T13:57:28.505 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3700000000000001e-06 2024-09-21T13:57:28.505 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.505 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.505 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:28.505 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.691702+0000", 2024-09-21T13:57:28.505 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.92758885899999999 2024-09-21T13:57:28.505 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.505 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.505 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:57:28.505 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.691822+0000", 2024-09-21T13:57:28.506 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011997699999999999 2024-09-21T13:57:28.506 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.506 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.506 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,5", 2024-09-21T13:57:28.506 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.691939+0000", 2024-09-21T13:57:28.506 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011752 2024-09-21T13:57:28.506 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.506 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.506 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:57:28.506 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.698556+0000", 2024-09-21T13:57:28.506 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0066170739999999997 2024-09-21T13:57:28.506 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.507 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.507 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:28.507 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.698612+0000", 2024-09-21T13:57:28.507 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5362000000000003e-05 2024-09-21T13:57:28.507 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.507 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.507 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:28.507 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.698659+0000", 2024-09-21T13:57:28.507 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6931999999999997e-05 2024-09-21T13:57:28.507 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.507 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.507 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:57:28.508 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.698714+0000", 2024-09-21T13:57:28.508 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5402000000000003e-05 2024-09-21T13:57:28.508 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.508 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.508 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:28.508 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.698756+0000", 2024-09-21T13:57:28.508 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2295000000000001e-05 2024-09-21T13:57:28.508 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:28.508 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:28.508 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:28.508 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.508 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.509 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:230051 7.e 7:71fa5c2a:::benchmark_data_smithi049_76798_object93631:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-21T13:57:28.509 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:53:16.764458+0000", 2024-09-21T13:57:28.509 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 251.6004724, 2024-09-21T13:57:28.509 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.94179994899999997, 2024-09-21T13:57:28.509 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:28.509 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:28.509 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:28.509 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:57:28.509 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:57:28.509 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:57:28.509 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 230051 2024-09-21T13:57:28.509 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.509 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:28.510 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.510 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:28.510 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.764458+0000", 2024-09-21T13:57:28.510 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:28.510 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.510 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.510 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:28.510 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.764458+0000", 2024-09-21T13:57:28.510 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:28.510 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.510 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.511 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:28.511 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.764459+0000", 2024-09-21T13:57:28.511 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9400000000000004e-07 2024-09-21T13:57:28.511 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.511 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.511 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:28.511 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.764463+0000", 2024-09-21T13:57:28.511 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1960000000000001e-06 2024-09-21T13:57:28.511 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.511 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.511 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:28.511 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.764477+0000", 2024-09-21T13:57:28.512 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4426e-05 2024-09-21T13:57:28.512 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.512 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.512 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:28.512 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.764485+0000", 2024-09-21T13:57:28.512 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.148e-06 2024-09-21T13:57:28.512 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.512 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.512 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:28.512 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.764545+0000", 2024-09-21T13:57:28.512 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0278999999999997e-05 2024-09-21T13:57:28.512 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.512 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.513 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:57:28.513 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.764553+0000", 2024-09-21T13:57:28.513 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9170000000000006e-06 2024-09-21T13:57:28.513 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.513 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.513 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:28.513 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.693660+0000", 2024-09-21T13:57:28.513 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.92910747900000001 2024-09-21T13:57:28.513 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.513 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.513 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:57:28.513 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.693785+0000", 2024-09-21T13:57:28.514 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012436000000000001 2024-09-21T13:57:28.514 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.514 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.514 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,5", 2024-09-21T13:57:28.514 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.693907+0000", 2024-09-21T13:57:28.514 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012208600000000001 2024-09-21T13:57:28.514 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.514 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.514 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:28.514 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.698816+0000", 2024-09-21T13:57:28.514 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0049091279999999996 2024-09-21T13:57:28.514 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.515 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.515 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:57:28.515 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.698834+0000", 2024-09-21T13:57:28.515 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8236999999999998e-05 2024-09-21T13:57:28.515 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.515 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.515 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:28.515 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.706148+0000", 2024-09-21T13:57:28.515 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0073139709999999998 2024-09-21T13:57:28.515 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.515 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.516 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:57:28.516 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.706202+0000", 2024-09-21T13:57:28.516 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3779000000000001e-05 2024-09-21T13:57:28.516 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.516 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.516 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:28.516 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.706258+0000", 2024-09-21T13:57:28.516 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6048999999999998e-05 2024-09-21T13:57:28.516 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:28.516 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:28.516 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:28.516 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.517 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.517 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:230060 7.e 7:7289055b:::benchmark_data_smithi049_76798_object93640:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-21T13:57:28.517 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:53:16.772333+0000", 2024-09-21T13:57:28.517 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 251.59259728000001, 2024-09-21T13:57:28.517 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.93409186700000002, 2024-09-21T13:57:28.517 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:28.517 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:28.517 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:28.517 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:57:28.517 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:57:28.517 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:57:28.517 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 230060 2024-09-21T13:57:28.518 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.518 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:28.518 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.518 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:28.518 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.772333+0000", 2024-09-21T13:57:28.518 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:28.518 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.518 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.518 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:28.518 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.772333+0000", 2024-09-21T13:57:28.518 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:28.518 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.519 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.519 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:28.519 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.772334+0000", 2024-09-21T13:57:28.519 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1540000000000001e-06 2024-09-21T13:57:28.519 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.519 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.519 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:28.519 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.772339+0000", 2024-09-21T13:57:28.519 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8099999999999997e-06 2024-09-21T13:57:28.519 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.519 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.519 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:28.519 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.772351+0000", 2024-09-21T13:57:28.520 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2435999999999999e-05 2024-09-21T13:57:28.520 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.520 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.520 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:28.520 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.772359+0000", 2024-09-21T13:57:28.520 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7230000000000004e-06 2024-09-21T13:57:28.520 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.520 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.520 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:28.520 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.772435+0000", 2024-09-21T13:57:28.520 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5465999999999999e-05 2024-09-21T13:57:28.520 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.521 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.521 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:57:28.521 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.772443+0000", 2024-09-21T13:57:28.521 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9160000000000001e-06 2024-09-21T13:57:28.521 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.521 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.521 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:28.521 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.695975+0000", 2024-09-21T13:57:28.521 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.92353224599999995 2024-09-21T13:57:28.521 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.521 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.521 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:57:28.521 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.696104+0000", 2024-09-21T13:57:28.522 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012895499999999999 2024-09-21T13:57:28.522 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.522 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.522 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,5", 2024-09-21T13:57:28.522 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.696217+0000", 2024-09-21T13:57:28.522 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011362399999999999 2024-09-21T13:57:28.522 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.522 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.522 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:57:28.522 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.701076+0000", 2024-09-21T13:57:28.522 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0048582729999999998 2024-09-21T13:57:28.522 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.523 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.523 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:28.523 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.706310+0000", 2024-09-21T13:57:28.523 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0052347419999999997 2024-09-21T13:57:28.523 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.523 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.523 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:28.523 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.706347+0000", 2024-09-21T13:57:28.523 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6492999999999998e-05 2024-09-21T13:57:28.523 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.523 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.523 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:57:28.524 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.706392+0000", 2024-09-21T13:57:28.524 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5148999999999998e-05 2024-09-21T13:57:28.524 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:28.524 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:28.524 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:28.524 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.706425+0000", 2024-09-21T13:57:28.524 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2879999999999997e-05 2024-09-21T13:57:28.524 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:28.524 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:28.524 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:28.524 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:28.524 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:28.524 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T13:57:28.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:28.526 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:28.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:28.526 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [] live_osds: [0, 1, 4, 5, 6, 7, 2, 3] 2024-09-21T13:57:28.526 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T13:57:28.526 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-21T13:57:28.526 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd dump -f json-pretty 2024-09-21T13:57:28.627 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:57:28.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:28.698 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:28.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:57:28.799 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:28.822 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:28.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:28.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:28.926 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:28.968 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:28.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:28.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:29.070 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:29.106 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:29.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:29.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:29.207 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:29.240 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:29.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:29.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:29.342 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:29.368 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:29.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:29.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:29.434 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:29 smithi187 ceph-mon[33032]: 4.1e scrub starts 2024-09-21T13:57:29.434 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:29 smithi187 ceph-mon[33032]: 4.1e scrub ok 2024-09-21T13:57:29.434 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:29 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:57:29.434 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:29 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:57:29.434 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:29 smithi187 ceph-mon[33032]: pgmap v416: 148 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 2 peering, 2 active+remapped+backfill_toofull, 142 active+clean; 763 MiB data, 5.3 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 653 op/s; 1532/37194 objects degraded (4.119%); 1083/37194 objects misplaced (2.912%); 8.2 MiB/s, 130 objects/s recovering 2024-09-21T13:57:29.434 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:29 smithi187 ceph-mon[33032]: osdmap e248: 8 total, 8 up, 7 in 2024-09-21T13:57:29.435 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:29 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:57:29.435 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:29 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:57:29.470 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:29.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:29.499 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:29.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:29.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:29 smithi049 ceph-mon[40489]: 4.1e scrub starts 2024-09-21T13:57:29.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:29 smithi049 ceph-mon[40489]: 4.1e scrub ok 2024-09-21T13:57:29.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:29 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:57:29.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:29 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:57:29.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:29 smithi049 ceph-mon[40489]: pgmap v416: 148 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 2 peering, 2 active+remapped+backfill_toofull, 142 active+clean; 763 MiB data, 5.3 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 653 op/s; 1532/37194 objects degraded (4.119%); 1083/37194 objects misplaced (2.912%); 8.2 MiB/s, 130 objects/s recovering 2024-09-21T13:57:29.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:29 smithi049 ceph-mon[40489]: osdmap e248: 8 total, 8 up, 7 in 2024-09-21T13:57:29.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:29 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:57:29.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:29 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:57:29.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:29 smithi049 ceph-mon[30080]: 4.1e scrub starts 2024-09-21T13:57:29.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:29 smithi049 ceph-mon[30080]: 4.1e scrub ok 2024-09-21T13:57:29.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:29 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:57:29.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:29 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:57:29.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:29 smithi049 ceph-mon[30080]: pgmap v416: 148 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 2 peering, 2 active+remapped+backfill_toofull, 142 active+clean; 763 MiB data, 5.3 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 653 op/s; 1532/37194 objects degraded (4.119%); 1083/37194 objects misplaced (2.912%); 8.2 MiB/s, 130 objects/s recovering 2024-09-21T13:57:29.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:29 smithi049 ceph-mon[30080]: osdmap e248: 8 total, 8 up, 7 in 2024-09-21T13:57:29.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:29 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:57:29.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:29 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:57:29.599 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:29.624 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:29.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:29.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:29.694 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:57:29 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:57:29.726 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:29.752 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:29.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:29.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:29.854 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:57:29.877 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:29.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:29.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:57:29.885 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T13:57:29.978 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:57:30.031 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:30.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:30.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:57:30.133 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:30.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:30.159 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:30.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:30.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:30 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:57:30.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:30 smithi049 ceph-mon[40489]: 1.0 scrub starts 2024-09-21T13:57:30.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:30 smithi049 ceph-mon[40489]: 1.0 scrub ok 2024-09-21T13:57:30.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:30 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:57:30.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:30 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:57:30.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:30 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 1532/37194 objects degraded (4.119%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T13:57:30.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:30 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:57:30.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:30 smithi049 ceph-mon[30080]: 1.0 scrub starts 2024-09-21T13:57:30.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:30 smithi049 ceph-mon[30080]: 1.0 scrub ok 2024-09-21T13:57:30.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:30 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:57:30.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:30 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:57:30.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:30 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 1532/37194 objects degraded (4.119%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T13:57:30.260 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:57:30.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:30.314 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:30.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:57:30.415 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:30.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:30 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:57:30.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:30 smithi187 ceph-mon[33032]: 1.0 scrub starts 2024-09-21T13:57:30.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:30 smithi187 ceph-mon[33032]: 1.0 scrub ok 2024-09-21T13:57:30.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:30 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:57:30.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:30 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:57:30.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:30 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 1532/37194 objects degraded (4.119%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T13:57:30.455 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:30.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:30.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:30.557 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:57:30.596 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:30.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:30.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:57:30.698 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:30.725 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:30.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:30.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:30.754 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:57:30 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:57:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:57:30.812 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T13:57:30.826 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:57:30.942 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:57:30.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:30.943 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:30.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:57:30.959 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:57:31.045 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:31.053 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:31 smithi049 ceph-mon[30080]: 4.1b scrub starts 2024-09-21T13:57:31.071 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:31.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:31.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:31.172 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:57:31.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:31.231 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:31.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:57:31.313 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:31 smithi049 ceph-mon[30080]: 4.1b scrub ok 2024-09-21T13:57:31.314 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:31 smithi049 ceph-mon[30080]: pgmap v418: 148 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 2 peering, 2 active+remapped+backfill_toofull, 142 active+clean; 509 MiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.05k op/s; 913/25014 objects degraded (3.650%); 687/25014 objects misplaced (2.746%); 6.5 MiB/s, 103 objects/s recovering 2024-09-21T13:57:31.314 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:31 smithi049 ceph-mon[30080]: osdmap e249: 8 total, 8 up, 7 in 2024-09-21T13:57:31.314 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:31 smithi049 ceph-mon[30080]: 5.3 scrub starts 2024-09-21T13:57:31.314 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:31 smithi049 ceph-mon[30080]: 5.3 scrub ok 2024-09-21T13:57:31.314 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:31 smithi049 ceph-mon[40489]: 4.1b scrub starts 2024-09-21T13:57:31.314 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:31 smithi049 ceph-mon[40489]: 4.1b scrub ok 2024-09-21T13:57:31.314 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:31 smithi049 ceph-mon[40489]: pgmap v418: 148 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 2 peering, 2 active+remapped+backfill_toofull, 142 active+clean; 509 MiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.05k op/s; 913/25014 objects degraded (3.650%); 687/25014 objects misplaced (2.746%); 6.5 MiB/s, 103 objects/s recovering 2024-09-21T13:57:31.314 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:31 smithi049 ceph-mon[40489]: osdmap e249: 8 total, 8 up, 7 in 2024-09-21T13:57:31.315 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:31 smithi049 ceph-mon[40489]: 5.3 scrub starts 2024-09-21T13:57:31.315 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:31 smithi049 ceph-mon[40489]: 5.3 scrub ok 2024-09-21T13:57:31.332 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:57:31.391 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:31.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:31.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:57:31.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:31 smithi187 ceph-mon[33032]: 4.1b scrub starts 2024-09-21T13:57:31.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:31 smithi187 ceph-mon[33032]: 4.1b scrub ok 2024-09-21T13:57:31.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:31 smithi187 ceph-mon[33032]: pgmap v418: 148 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 2 peering, 2 active+remapped+backfill_toofull, 142 active+clean; 509 MiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.05k op/s; 913/25014 objects degraded (3.650%); 687/25014 objects misplaced (2.746%); 6.5 MiB/s, 103 objects/s recovering 2024-09-21T13:57:31.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:31 smithi187 ceph-mon[33032]: osdmap e249: 8 total, 8 up, 7 in 2024-09-21T13:57:31.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:31 smithi187 ceph-mon[33032]: 5.3 scrub starts 2024-09-21T13:57:31.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:31 smithi187 ceph-mon[33032]: 5.3 scrub ok 2024-09-21T13:57:31.492 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:31.517 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:31.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:31.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:31.619 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:31.644 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:31.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:31.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:31.745 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:31.794 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:31.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:31.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:31.897 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:57:31.958 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:31.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:31.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:57:31.959 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T13:57:32.059 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:32.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:32.109 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:32.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:32.134 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:57:32.210 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:32.233 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:32.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:32.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:32.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:32 smithi049 ceph-mon[30080]: 3.1d scrub starts 2024-09-21T13:57:32.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:32 smithi049 ceph-mon[30080]: 3.1d scrub ok 2024-09-21T13:57:32.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:32 smithi049 ceph-mon[40489]: 3.1d scrub starts 2024-09-21T13:57:32.257 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:32 smithi049 ceph-mon[40489]: 3.1d scrub ok 2024-09-21T13:57:32.335 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:32.357 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:32.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:32.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:32.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:32 smithi187 ceph-mon[33032]: 3.1d scrub starts 2024-09-21T13:57:32.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:32 smithi187 ceph-mon[33032]: 3.1d scrub ok 2024-09-21T13:57:32.459 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:57:32.466 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:57:32.466 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T13:57:32.466 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 249, 2024-09-21T13:57:32.466 INFO:teuthology.orchestra.run.smithi049.stdout: "fsid": "40a3d2e8-781f-11ef-baf6-efdc52797490", 2024-09-21T13:57:32.466 INFO:teuthology.orchestra.run.smithi049.stdout: "created": "2024-09-21T13:42:57.838275+0000", 2024-09-21T13:57:32.466 INFO:teuthology.orchestra.run.smithi049.stdout: "modified": "2024-09-21T13:57:30.040104+0000", 2024-09-21T13:57:32.467 INFO:teuthology.orchestra.run.smithi049.stdout: "last_up_change": "2024-09-21T13:57:11.653688+0000", 2024-09-21T13:57:32.467 INFO:teuthology.orchestra.run.smithi049.stdout: "last_in_change": "2024-09-21T13:55:33.181161+0000", 2024-09-21T13:57:32.467 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-21T13:57:32.467 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_num": 5804032, 2024-09-21T13:57:32.467 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_set": [ 2024-09-21T13:57:32.467 INFO:teuthology.orchestra.run.smithi049.stdout: "nodeep-scrub", 2024-09-21T13:57:32.467 INFO:teuthology.orchestra.run.smithi049.stdout: "pglog_hardlimit", 2024-09-21T13:57:32.467 INFO:teuthology.orchestra.run.smithi049.stdout: "purged_snapdirs", 2024-09-21T13:57:32.467 INFO:teuthology.orchestra.run.smithi049.stdout: "recovery_deletes", 2024-09-21T13:57:32.467 INFO:teuthology.orchestra.run.smithi049.stdout: "sortbitwise" 2024-09-21T13:57:32.467 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T13:57:32.467 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_version": 20, 2024-09-21T13:57:32.467 INFO:teuthology.orchestra.run.smithi049.stdout: "full_ratio": 0.94999998807907104, 2024-09-21T13:57:32.468 INFO:teuthology.orchestra.run.smithi049.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-21T13:57:32.468 INFO:teuthology.orchestra.run.smithi049.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-21T13:57:32.468 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_snapshot": "", 2024-09-21T13:57:32.468 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_max": 8, 2024-09-21T13:57:32.468 INFO:teuthology.orchestra.run.smithi049.stdout: "max_osd": 8, 2024-09-21T13:57:32.468 INFO:teuthology.orchestra.run.smithi049.stdout: "require_min_compat_client": "luminous", 2024-09-21T13:57:32.468 INFO:teuthology.orchestra.run.smithi049.stdout: "min_compat_client": "luminous", 2024-09-21T13:57:32.468 INFO:teuthology.orchestra.run.smithi049.stdout: "require_osd_release": "squid", 2024-09-21T13:57:32.468 INFO:teuthology.orchestra.run.smithi049.stdout: "allow_crimson": false, 2024-09-21T13:57:32.468 INFO:teuthology.orchestra.run.smithi049.stdout: "pools": [ 2024-09-21T13:57:32.468 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.468 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 1, 2024-09-21T13:57:32.469 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": ".mgr", 2024-09-21T13:57:32.469 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:46:21.096244+0000", 2024-09-21T13:57:32.469 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T13:57:32.469 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T13:57:32.469 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T13:57:32.469 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T13:57:32.469 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T13:57:32.469 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T13:57:32.469 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T13:57:32.469 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T13:57:32.469 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T13:57:32.469 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T13:57:32.469 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T13:57:32.470 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T13:57:32.470 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T13:57:32.470 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 1, 2024-09-21T13:57:32.470 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 1, 2024-09-21T13:57:32.470 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 1, 2024-09-21T13:57:32.470 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 1, 2024-09-21T13:57:32.470 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 1, 2024-09-21T13:57:32.470 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T13:57:32.470 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T13:57:32.470 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T13:57:32.470 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T13:57:32.470 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T13:57:32.471 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T13:57:32.471 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T13:57:32.471 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.471 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "199", 2024-09-21T13:57:32.471 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T13:57:32.471 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T13:57:32.471 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T13:57:32.471 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T13:57:32.471 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T13:57:32.471 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T13:57:32.471 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T13:57:32.471 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T13:57:32.472 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T13:57:32.472 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T13:57:32.472 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T13:57:32.472 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T13:57:32.472 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T13:57:32.472 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T13:57:32.472 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T13:57:32.472 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T13:57:32.472 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T13:57:32.472 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T13:57:32.472 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T13:57:32.472 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T13:57:32.472 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T13:57:32.473 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T13:57:32.473 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T13:57:32.473 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T13:57:32.473 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T13:57:32.473 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T13:57:32.473 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.473 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T13:57:32.473 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T13:57:32.473 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T13:57:32.473 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T13:57:32.473 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T13:57:32.473 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T13:57:32.474 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T13:57:32.474 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T13:57:32.474 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T13:57:32.474 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T13:57:32.474 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T13:57:32.474 INFO:teuthology.orchestra.run.smithi049.stdout: "options": { 2024-09-21T13:57:32.474 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_max": 32, 2024-09-21T13:57:32.474 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_min": 1 2024-09-21T13:57:32.474 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.474 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T13:57:32.474 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr": {} 2024-09-21T13:57:32.474 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.475 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T13:57:32.475 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T13:57:32.475 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 7.8899998664855957, 2024-09-21T13:57:32.475 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 7.8899998664855957, 2024-09-21T13:57:32.475 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 0.37999999523162842, 2024-09-21T13:57:32.475 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 3, 2024-09-21T13:57:32.475 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 3, 2024-09-21T13:57:32.475 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.86855697631835938, 2024-09-21T13:57:32.475 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.87000000476837158, 2024-09-21T13:57:32.475 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.87000000476837158 2024-09-21T13:57:32.475 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:32.475 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.475 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.475 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 2, 2024-09-21T13:57:32.475 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "datapool", 2024-09-21T13:57:32.476 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:48:55.540712+0000", 2024-09-21T13:57:32.476 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 8193, 2024-09-21T13:57:32.476 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T13:57:32.476 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T13:57:32.476 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T13:57:32.476 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T13:57:32.476 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T13:57:32.476 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T13:57:32.476 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T13:57:32.476 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T13:57:32.476 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T13:57:32.476 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T13:57:32.476 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T13:57:32.476 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T13:57:32.477 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 3, 2024-09-21T13:57:32.477 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 3, 2024-09-21T13:57:32.477 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 3, 2024-09-21T13:57:32.477 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 3, 2024-09-21T13:57:32.477 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 3, 2024-09-21T13:57:32.477 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T13:57:32.477 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T13:57:32.477 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T13:57:32.477 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T13:57:32.477 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T13:57:32.477 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T13:57:32.477 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T13:57:32.477 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.477 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "63", 2024-09-21T13:57:32.477 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T13:57:32.478 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T13:57:32.478 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T13:57:32.478 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T13:57:32.478 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T13:57:32.478 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 2, 2024-09-21T13:57:32.478 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 63, 2024-09-21T13:57:32.478 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T13:57:32.478 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T13:57:32.478 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T13:57:32.478 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T13:57:32.478 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T13:57:32.478 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T13:57:32.478 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T13:57:32.479 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T13:57:32.479 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T13:57:32.479 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T13:57:32.479 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T13:57:32.479 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T13:57:32.479 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T13:57:32.479 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T13:57:32.479 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T13:57:32.479 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T13:57:32.479 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T13:57:32.479 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T13:57:32.479 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T13:57:32.479 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.479 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T13:57:32.480 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T13:57:32.480 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T13:57:32.480 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T13:57:32.480 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T13:57:32.480 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T13:57:32.480 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T13:57:32.480 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T13:57:32.480 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T13:57:32.480 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T13:57:32.480 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T13:57:32.480 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T13:57:32.480 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T13:57:32.480 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd": {} 2024-09-21T13:57:32.481 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.481 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T13:57:32.481 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T13:57:32.481 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2.6700000762939453, 2024-09-21T13:57:32.481 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2.6700000762939453, 2024-09-21T13:57:32.481 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 0.75, 2024-09-21T13:57:32.481 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2, 2024-09-21T13:57:32.481 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2, 2024-09-21T13:57:32.481 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.86855697631835938, 2024-09-21T13:57:32.481 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.87000000476837158, 2024-09-21T13:57:32.481 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.87000000476837158 2024-09-21T13:57:32.481 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:32.481 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.482 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.482 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 3, 2024-09-21T13:57:32.482 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": ".rgw.root", 2024-09-21T13:57:32.482 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:48:56.417456+0000", 2024-09-21T13:57:32.482 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T13:57:32.482 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T13:57:32.482 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T13:57:32.482 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T13:57:32.482 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T13:57:32.482 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T13:57:32.482 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T13:57:32.482 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T13:57:32.482 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T13:57:32.482 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T13:57:32.483 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T13:57:32.483 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T13:57:32.483 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T13:57:32.483 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 32, 2024-09-21T13:57:32.483 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 32, 2024-09-21T13:57:32.483 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 32, 2024-09-21T13:57:32.483 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 32, 2024-09-21T13:57:32.483 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 32, 2024-09-21T13:57:32.483 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T13:57:32.483 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T13:57:32.483 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T13:57:32.483 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T13:57:32.484 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T13:57:32.484 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T13:57:32.484 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T13:57:32.484 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.484 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "62", 2024-09-21T13:57:32.484 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T13:57:32.484 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T13:57:32.484 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T13:57:32.484 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T13:57:32.484 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T13:57:32.484 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T13:57:32.484 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T13:57:32.484 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T13:57:32.485 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T13:57:32.485 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T13:57:32.485 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T13:57:32.485 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T13:57:32.485 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T13:57:32.485 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T13:57:32.485 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T13:57:32.485 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T13:57:32.485 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T13:57:32.485 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T13:57:32.485 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T13:57:32.485 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T13:57:32.485 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T13:57:32.485 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T13:57:32.486 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T13:57:32.486 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T13:57:32.486 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T13:57:32.486 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T13:57:32.486 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.486 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T13:57:32.486 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T13:57:32.486 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T13:57:32.486 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T13:57:32.486 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T13:57:32.486 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T13:57:32.487 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T13:57:32.487 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T13:57:32.487 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T13:57:32.487 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T13:57:32.487 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T13:57:32.487 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T13:57:32.487 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T13:57:32.487 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw": {} 2024-09-21T13:57:32.487 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.487 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T13:57:32.487 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T13:57:32.487 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 1.7400000095367432, 2024-09-21T13:57:32.487 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 1.7400000095367432, 2024-09-21T13:57:32.487 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 0.87999999523162842, 2024-09-21T13:57:32.488 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 1.5299999713897705, 2024-09-21T13:57:32.488 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 1.5299999713897705, 2024-09-21T13:57:32.488 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.86855697631835938, 2024-09-21T13:57:32.488 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.87000000476837158, 2024-09-21T13:57:32.488 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.87000000476837158 2024-09-21T13:57:32.488 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:32.488 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.488 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.488 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 4, 2024-09-21T13:57:32.488 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "default.rgw.log", 2024-09-21T13:57:32.488 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:48:58.128455+0000", 2024-09-21T13:57:32.488 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T13:57:32.488 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T13:57:32.488 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T13:57:32.488 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T13:57:32.489 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T13:57:32.489 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T13:57:32.489 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T13:57:32.489 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T13:57:32.489 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T13:57:32.489 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T13:57:32.489 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T13:57:32.489 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T13:57:32.489 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T13:57:32.489 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 32, 2024-09-21T13:57:32.489 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 32, 2024-09-21T13:57:32.489 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 32, 2024-09-21T13:57:32.489 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 32, 2024-09-21T13:57:32.490 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 32, 2024-09-21T13:57:32.490 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T13:57:32.490 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T13:57:32.490 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T13:57:32.490 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T13:57:32.490 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T13:57:32.490 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T13:57:32.490 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T13:57:32.490 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.490 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "64", 2024-09-21T13:57:32.490 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T13:57:32.490 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T13:57:32.490 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T13:57:32.490 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T13:57:32.491 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T13:57:32.491 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T13:57:32.491 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T13:57:32.491 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T13:57:32.491 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T13:57:32.491 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T13:57:32.491 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T13:57:32.491 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T13:57:32.491 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T13:57:32.491 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T13:57:32.491 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T13:57:32.491 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T13:57:32.491 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T13:57:32.491 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T13:57:32.491 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T13:57:32.492 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T13:57:32.492 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T13:57:32.492 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T13:57:32.492 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T13:57:32.492 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T13:57:32.492 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T13:57:32.492 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T13:57:32.492 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.492 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T13:57:32.492 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T13:57:32.492 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T13:57:32.492 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T13:57:32.492 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T13:57:32.492 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T13:57:32.493 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T13:57:32.493 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T13:57:32.493 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T13:57:32.493 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T13:57:32.493 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T13:57:32.493 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T13:57:32.493 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T13:57:32.493 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw": {} 2024-09-21T13:57:32.493 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.493 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T13:57:32.493 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T13:57:32.493 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2.4900000095367432, 2024-09-21T13:57:32.493 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2.4900000095367432, 2024-09-21T13:57:32.493 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 0.87999999523162842, 2024-09-21T13:57:32.494 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2.190000057220459, 2024-09-21T13:57:32.494 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2.190000057220459, 2024-09-21T13:57:32.494 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.86855697631835938, 2024-09-21T13:57:32.494 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.87000000476837158, 2024-09-21T13:57:32.494 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.87000000476837158 2024-09-21T13:57:32.494 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:32.494 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.494 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.494 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 5, 2024-09-21T13:57:32.494 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "default.rgw.control", 2024-09-21T13:57:32.494 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:49:00.069361+0000", 2024-09-21T13:57:32.494 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T13:57:32.494 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T13:57:32.494 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T13:57:32.495 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T13:57:32.495 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T13:57:32.495 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T13:57:32.495 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T13:57:32.495 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T13:57:32.495 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T13:57:32.495 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T13:57:32.495 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T13:57:32.495 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T13:57:32.495 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T13:57:32.495 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 32, 2024-09-21T13:57:32.495 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 32, 2024-09-21T13:57:32.495 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 32, 2024-09-21T13:57:32.495 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 32, 2024-09-21T13:57:32.496 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 32, 2024-09-21T13:57:32.496 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T13:57:32.496 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T13:57:32.496 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T13:57:32.496 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T13:57:32.496 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T13:57:32.496 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T13:57:32.496 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T13:57:32.496 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.496 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "66", 2024-09-21T13:57:32.496 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T13:57:32.496 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T13:57:32.496 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T13:57:32.497 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T13:57:32.497 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T13:57:32.497 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T13:57:32.497 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T13:57:32.497 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T13:57:32.497 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T13:57:32.497 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T13:57:32.497 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T13:57:32.497 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T13:57:32.497 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T13:57:32.497 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T13:57:32.497 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T13:57:32.497 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T13:57:32.497 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T13:57:32.497 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T13:57:32.498 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T13:57:32.498 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T13:57:32.498 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T13:57:32.498 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T13:57:32.498 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T13:57:32.498 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T13:57:32.498 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T13:57:32.498 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T13:57:32.498 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.498 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T13:57:32.498 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T13:57:32.498 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T13:57:32.498 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T13:57:32.499 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T13:57:32.499 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T13:57:32.499 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T13:57:32.499 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T13:57:32.499 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T13:57:32.499 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T13:57:32.499 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T13:57:32.499 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T13:57:32.499 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T13:57:32.499 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw": {} 2024-09-21T13:57:32.499 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.499 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T13:57:32.500 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T13:57:32.501 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 1.7400000095367432, 2024-09-21T13:57:32.501 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 1.7400000095367432, 2024-09-21T13:57:32.501 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 0.87999999523162842, 2024-09-21T13:57:32.501 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 1.5299999713897705, 2024-09-21T13:57:32.501 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 1.5299999713897705, 2024-09-21T13:57:32.501 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.86855697631835938, 2024-09-21T13:57:32.501 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.87000000476837158, 2024-09-21T13:57:32.501 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.87000000476837158 2024-09-21T13:57:32.502 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:32.502 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.502 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.502 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 6, 2024-09-21T13:57:32.502 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "default.rgw.meta", 2024-09-21T13:57:32.502 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:49:02.127399+0000", 2024-09-21T13:57:32.502 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T13:57:32.502 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T13:57:32.502 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T13:57:32.502 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T13:57:32.502 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T13:57:32.502 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T13:57:32.503 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T13:57:32.503 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T13:57:32.503 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T13:57:32.503 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T13:57:32.503 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T13:57:32.503 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T13:57:32.503 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T13:57:32.503 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 32, 2024-09-21T13:57:32.503 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 32, 2024-09-21T13:57:32.503 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 32, 2024-09-21T13:57:32.503 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 32, 2024-09-21T13:57:32.503 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 32, 2024-09-21T13:57:32.503 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T13:57:32.503 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T13:57:32.503 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T13:57:32.504 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T13:57:32.504 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T13:57:32.504 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T13:57:32.504 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T13:57:32.504 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.504 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "68", 2024-09-21T13:57:32.504 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T13:57:32.504 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T13:57:32.504 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T13:57:32.504 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T13:57:32.504 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T13:57:32.504 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T13:57:32.505 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T13:57:32.505 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T13:57:32.505 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T13:57:32.505 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T13:57:32.505 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T13:57:32.505 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T13:57:32.505 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T13:57:32.505 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T13:57:32.505 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T13:57:32.505 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T13:57:32.505 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T13:57:32.505 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T13:57:32.505 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T13:57:32.505 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T13:57:32.505 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T13:57:32.506 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T13:57:32.506 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T13:57:32.506 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T13:57:32.506 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T13:57:32.506 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T13:57:32.506 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.506 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T13:57:32.506 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T13:57:32.506 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T13:57:32.506 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T13:57:32.506 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T13:57:32.506 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T13:57:32.506 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T13:57:32.507 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T13:57:32.507 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T13:57:32.507 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T13:57:32.507 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T13:57:32.507 INFO:teuthology.orchestra.run.smithi049.stdout: "options": { 2024-09-21T13:57:32.507 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_bias": 4 2024-09-21T13:57:32.507 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.507 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T13:57:32.507 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw": {} 2024-09-21T13:57:32.507 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.507 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T13:57:32.507 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T13:57:32.507 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 1.9900000095367432, 2024-09-21T13:57:32.507 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 1.9900000095367432, 2024-09-21T13:57:32.507 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 0.87999999523162842, 2024-09-21T13:57:32.508 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 1.75, 2024-09-21T13:57:32.508 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 1.75, 2024-09-21T13:57:32.508 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.86855697631835938, 2024-09-21T13:57:32.508 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.87000000476837158, 2024-09-21T13:57:32.508 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.87000000476837158 2024-09-21T13:57:32.508 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:32.508 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.508 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.508 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 8, 2024-09-21T13:57:32.508 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "unique_pool_1", 2024-09-21T13:57:32.508 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:54:10.659855+0000", 2024-09-21T13:57:32.508 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T13:57:32.508 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T13:57:32.508 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T13:57:32.508 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T13:57:32.509 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T13:57:32.509 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T13:57:32.509 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T13:57:32.509 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T13:57:32.509 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T13:57:32.509 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T13:57:32.509 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T13:57:32.509 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T13:57:32.509 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T13:57:32.509 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 16, 2024-09-21T13:57:32.509 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 16, 2024-09-21T13:57:32.509 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 16, 2024-09-21T13:57:32.509 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 16, 2024-09-21T13:57:32.509 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 16, 2024-09-21T13:57:32.509 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T13:57:32.510 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T13:57:32.510 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T13:57:32.510 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T13:57:32.510 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T13:57:32.510 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T13:57:32.510 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T13:57:32.510 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.510 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "187", 2024-09-21T13:57:32.510 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T13:57:32.510 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T13:57:32.510 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T13:57:32.510 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T13:57:32.510 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T13:57:32.510 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T13:57:32.511 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T13:57:32.511 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T13:57:32.511 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T13:57:32.511 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T13:57:32.511 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T13:57:32.511 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T13:57:32.511 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T13:57:32.511 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T13:57:32.511 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T13:57:32.511 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T13:57:32.511 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T13:57:32.511 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T13:57:32.511 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T13:57:32.511 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T13:57:32.511 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T13:57:32.512 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T13:57:32.512 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T13:57:32.512 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T13:57:32.513 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T13:57:32.513 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T13:57:32.513 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.513 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T13:57:32.513 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T13:57:32.513 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T13:57:32.514 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T13:57:32.514 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T13:57:32.514 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T13:57:32.514 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T13:57:32.514 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T13:57:32.514 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T13:57:32.514 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T13:57:32.514 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T13:57:32.514 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T13:57:32.514 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T13:57:32.514 INFO:teuthology.orchestra.run.smithi049.stdout: "rados": {} 2024-09-21T13:57:32.514 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.514 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T13:57:32.514 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T13:57:32.515 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2.4900000095367432, 2024-09-21T13:57:32.515 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 1.9900000095367432, 2024-09-21T13:57:32.515 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 0.87999999523162842, 2024-09-21T13:57:32.515 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2.190000057220459, 2024-09-21T13:57:32.515 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 1.75, 2024-09-21T13:57:32.515 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.86855697631835938, 2024-09-21T13:57:32.515 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.87000000476837158, 2024-09-21T13:57:32.515 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.87000000476837158 2024-09-21T13:57:32.515 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:32.515 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:32.515 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T13:57:32.515 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-09-21T13:57:32.515 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.515 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 0, 2024-09-21T13:57:32.516 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "2595dca9-6219-4e0c-8728-751eb102e73a", 2024-09-21T13:57:32.516 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T13:57:32.516 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T13:57:32.516 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T13:57:32.516 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 0.1215972900390625, 2024-09-21T13:57:32.516 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T13:57:32.516 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T13:57:32.516 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 8, 2024-09-21T13:57:32.516 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 241, 2024-09-21T13:57:32.516 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T13:57:32.516 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T13:57:32.516 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T13:57:32.516 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:32.516 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.516 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:32.517 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6802", 2024-09-21T13:57:32.517 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T13:57:32.517 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.517 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.517 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:32.517 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6803", 2024-09-21T13:57:32.517 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T13:57:32.517 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:32.517 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:32.517 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.517 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T13:57:32.517 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:32.517 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.517 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:32.517 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6804", 2024-09-21T13:57:32.518 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T13:57:32.518 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.518 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.518 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:32.518 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6805", 2024-09-21T13:57:32.518 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T13:57:32.518 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:32.518 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:32.518 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.518 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T13:57:32.518 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:32.518 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.518 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:32.519 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6808", 2024-09-21T13:57:32.519 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T13:57:32.519 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.519 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.519 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:32.519 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6809", 2024-09-21T13:57:32.519 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T13:57:32.519 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:32.519 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:32.519 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.519 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T13:57:32.519 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:32.519 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.519 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:32.520 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6806", 2024-09-21T13:57:32.520 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T13:57:32.520 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.520 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.520 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:32.520 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6807", 2024-09-21T13:57:32.520 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T13:57:32.520 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:32.520 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:32.520 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.520 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6803/2095561412", 2024-09-21T13:57:32.520 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6805/2095561412", 2024-09-21T13:57:32.520 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6809/2095561412", 2024-09-21T13:57:32.520 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6807/2095561412", 2024-09-21T13:57:32.521 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T13:57:32.521 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T13:57:32.521 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T13:57:32.521 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:32.521 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.521 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.521 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 1, 2024-09-21T13:57:32.521 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "0ba327f9-5524-4a40-abcc-729ded0b1917", 2024-09-21T13:57:32.521 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T13:57:32.521 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T13:57:32.521 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T13:57:32.521 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T13:57:32.521 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T13:57:32.521 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T13:57:32.522 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 13, 2024-09-21T13:57:32.522 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 245, 2024-09-21T13:57:32.522 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T13:57:32.522 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T13:57:32.522 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T13:57:32.522 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:32.522 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.522 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:32.522 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6810", 2024-09-21T13:57:32.522 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T13:57:32.522 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.522 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.522 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:32.523 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6811", 2024-09-21T13:57:32.523 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T13:57:32.523 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:32.523 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:32.523 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.523 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T13:57:32.523 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:32.523 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.523 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:32.523 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6812", 2024-09-21T13:57:32.523 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T13:57:32.523 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.523 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.524 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:32.524 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6813", 2024-09-21T13:57:32.524 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T13:57:32.524 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:32.524 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:32.524 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.524 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T13:57:32.524 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:32.524 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.524 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:32.524 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6816", 2024-09-21T13:57:32.524 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T13:57:32.524 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.525 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.525 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:32.525 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6817", 2024-09-21T13:57:32.525 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T13:57:32.525 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:32.525 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:32.525 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.525 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T13:57:32.525 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:32.525 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.525 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:32.525 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6814", 2024-09-21T13:57:32.525 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T13:57:32.525 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.526 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.526 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:32.526 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6815", 2024-09-21T13:57:32.526 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T13:57:32.526 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:32.526 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:32.526 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.526 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6811/1382840527", 2024-09-21T13:57:32.526 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6813/1382840527", 2024-09-21T13:57:32.526 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6817/1382840527", 2024-09-21T13:57:32.526 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6815/1382840527", 2024-09-21T13:57:32.526 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T13:57:32.527 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T13:57:32.527 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T13:57:32.527 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:32.527 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.527 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.527 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 2, 2024-09-21T13:57:32.527 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "27a45750-204d-48fd-bb8f-ebb01273343d", 2024-09-21T13:57:32.527 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T13:57:32.527 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T13:57:32.527 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T13:57:32.527 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 0.8268585205078125, 2024-09-21T13:57:32.527 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 18, 2024-09-21T13:57:32.527 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 100, 2024-09-21T13:57:32.528 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 130, 2024-09-21T13:57:32.528 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 240, 2024-09-21T13:57:32.528 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 101, 2024-09-21T13:57:32.528 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T13:57:32.528 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T13:57:32.528 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:32.528 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.528 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:32.528 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6818", 2024-09-21T13:57:32.528 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T13:57:32.528 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.528 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.528 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:32.528 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6819", 2024-09-21T13:57:32.529 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T13:57:32.529 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:32.529 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:32.529 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.529 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T13:57:32.529 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:32.529 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.529 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:32.529 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6820", 2024-09-21T13:57:32.529 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T13:57:32.529 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.529 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.529 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:32.530 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6821", 2024-09-21T13:57:32.530 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T13:57:32.530 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:32.530 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:32.530 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.530 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T13:57:32.530 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:32.530 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.530 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:32.530 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6824", 2024-09-21T13:57:32.530 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T13:57:32.530 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.530 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.530 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:32.531 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6825", 2024-09-21T13:57:32.531 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T13:57:32.531 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:32.531 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:32.531 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.531 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T13:57:32.531 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:32.531 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.531 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:32.531 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6822", 2024-09-21T13:57:32.531 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T13:57:32.531 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.531 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.531 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:32.532 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6823", 2024-09-21T13:57:32.532 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T13:57:32.532 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:32.532 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:32.532 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.532 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6819/1945314741", 2024-09-21T13:57:32.532 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6821/1945314741", 2024-09-21T13:57:32.532 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6825/1945314741", 2024-09-21T13:57:32.532 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6823/1945314741", 2024-09-21T13:57:32.532 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T13:57:32.532 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T13:57:32.532 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T13:57:32.532 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:32.532 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.533 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.533 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 3, 2024-09-21T13:57:32.533 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "528bc390-3dc9-4f3b-ab75-d7412c9625db", 2024-09-21T13:57:32.533 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T13:57:32.533 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T13:57:32.533 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T13:57:32.533 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T13:57:32.533 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 27, 2024-09-21T13:57:32.533 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 214, 2024-09-21T13:57:32.533 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 240, 2024-09-21T13:57:32.533 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 246, 2024-09-21T13:57:32.533 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 215, 2024-09-21T13:57:32.533 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T13:57:32.533 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T13:57:32.534 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:32.534 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.534 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:32.534 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6826", 2024-09-21T13:57:32.534 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T13:57:32.534 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.534 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.534 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:32.534 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6827", 2024-09-21T13:57:32.534 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T13:57:32.534 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:32.534 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:32.534 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.535 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T13:57:32.535 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:32.535 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.535 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:32.535 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6828", 2024-09-21T13:57:32.535 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T13:57:32.535 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.535 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.535 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:32.536 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6829", 2024-09-21T13:57:32.536 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T13:57:32.536 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:32.536 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:32.536 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.536 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T13:57:32.536 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:32.536 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.536 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:32.536 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6832", 2024-09-21T13:57:32.536 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T13:57:32.536 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.536 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.536 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:32.536 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6833", 2024-09-21T13:57:32.537 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T13:57:32.537 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:32.537 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:32.537 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.537 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T13:57:32.537 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:32.537 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.537 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:32.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:32.538 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:32.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:57:32.538 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6830", 2024-09-21T13:57:32.539 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T13:57:32.539 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.539 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.539 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:32.539 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6831", 2024-09-21T13:57:32.539 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T13:57:32.539 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:32.539 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:32.539 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.539 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6827/886735721", 2024-09-21T13:57:32.539 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6829/886735721", 2024-09-21T13:57:32.539 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6833/886735721", 2024-09-21T13:57:32.539 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6831/886735721", 2024-09-21T13:57:32.540 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T13:57:32.540 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T13:57:32.540 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T13:57:32.540 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:32.540 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.540 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.540 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 4, 2024-09-21T13:57:32.540 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "6fc64698-2d37-4db0-bf3d-c7e612c45163", 2024-09-21T13:57:32.540 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T13:57:32.540 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T13:57:32.540 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T13:57:32.540 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T13:57:32.540 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T13:57:32.540 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T13:57:32.541 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 33, 2024-09-21T13:57:32.541 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 240, 2024-09-21T13:57:32.541 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T13:57:32.541 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T13:57:32.541 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T13:57:32.541 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:32.541 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.541 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:32.541 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6800", 2024-09-21T13:57:32.541 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T13:57:32.541 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.541 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.541 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:32.541 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6801", 2024-09-21T13:57:32.541 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T13:57:32.542 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:32.542 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:32.542 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.542 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T13:57:32.542 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:32.542 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.542 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:32.542 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6802", 2024-09-21T13:57:32.542 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T13:57:32.542 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.542 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.542 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:32.542 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6803", 2024-09-21T13:57:32.543 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T13:57:32.543 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:32.543 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:32.543 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.543 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T13:57:32.543 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:32.543 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.543 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:32.543 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6806", 2024-09-21T13:57:32.543 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T13:57:32.543 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.543 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.543 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:32.543 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6807", 2024-09-21T13:57:32.543 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T13:57:32.544 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:32.544 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:32.544 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.544 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T13:57:32.544 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:32.544 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.544 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:32.544 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6804", 2024-09-21T13:57:32.544 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T13:57:32.544 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.544 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.544 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:32.544 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6805", 2024-09-21T13:57:32.544 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T13:57:32.545 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:32.545 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:32.545 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.545 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.187:6801/1702447776", 2024-09-21T13:57:32.545 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.187:6803/1702447776", 2024-09-21T13:57:32.545 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.187:6807/1702447776", 2024-09-21T13:57:32.545 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.187:6805/1702447776", 2024-09-21T13:57:32.545 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T13:57:32.545 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T13:57:32.545 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T13:57:32.545 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:32.545 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.545 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.545 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 5, 2024-09-21T13:57:32.546 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "8d7f62a1-7e92-4e06-9cad-86b96be514d8", 2024-09-21T13:57:32.546 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T13:57:32.546 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T13:57:32.546 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T13:57:32.546 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T13:57:32.546 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T13:57:32.546 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T13:57:32.546 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 39, 2024-09-21T13:57:32.546 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 244, 2024-09-21T13:57:32.546 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T13:57:32.546 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T13:57:32.546 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T13:57:32.547 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:32.547 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.547 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:32.547 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6808", 2024-09-21T13:57:32.547 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3659050776 2024-09-21T13:57:32.547 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.547 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.547 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:32.547 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6809", 2024-09-21T13:57:32.547 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3659050776 2024-09-21T13:57:32.547 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:32.547 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:32.547 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.547 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T13:57:32.548 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:32.548 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.548 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:32.548 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6810", 2024-09-21T13:57:32.548 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3659050776 2024-09-21T13:57:32.548 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.548 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.548 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:32.548 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6811", 2024-09-21T13:57:32.548 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3659050776 2024-09-21T13:57:32.548 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:32.548 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:32.548 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.548 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T13:57:32.549 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:32.549 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.549 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:32.549 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6814", 2024-09-21T13:57:32.549 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3659050776 2024-09-21T13:57:32.549 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.549 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.549 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:32.549 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6815", 2024-09-21T13:57:32.549 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3659050776 2024-09-21T13:57:32.550 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:32.550 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:32.550 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.550 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T13:57:32.550 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:32.550 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.550 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:32.550 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6812", 2024-09-21T13:57:32.550 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3659050776 2024-09-21T13:57:32.550 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.550 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.550 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:32.551 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6813", 2024-09-21T13:57:32.551 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3659050776 2024-09-21T13:57:32.551 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:32.551 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:32.551 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.551 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.187:6809/3659050776", 2024-09-21T13:57:32.551 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.187:6811/3659050776", 2024-09-21T13:57:32.551 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.187:6815/3659050776", 2024-09-21T13:57:32.551 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.187:6813/3659050776", 2024-09-21T13:57:32.551 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T13:57:32.551 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T13:57:32.551 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T13:57:32.551 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:32.551 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.552 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.552 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 6, 2024-09-21T13:57:32.552 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "3a7afd13-26ce-444c-9cc0-81ce2eab6ce5", 2024-09-21T13:57:32.552 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T13:57:32.552 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 0, 2024-09-21T13:57:32.552 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 0, 2024-09-21T13:57:32.552 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T13:57:32.552 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T13:57:32.552 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T13:57:32.552 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 47, 2024-09-21T13:57:32.552 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 183, 2024-09-21T13:57:32.552 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T13:57:32.552 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T13:57:32.553 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T13:57:32.553 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:32.553 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.553 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:32.553 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6816", 2024-09-21T13:57:32.553 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T13:57:32.553 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.553 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.553 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:32.553 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6817", 2024-09-21T13:57:32.553 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T13:57:32.553 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:32.553 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:32.553 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.553 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T13:57:32.554 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:32.554 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.554 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:32.554 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6818", 2024-09-21T13:57:32.554 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T13:57:32.554 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.554 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.554 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:32.554 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6819", 2024-09-21T13:57:32.554 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T13:57:32.554 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:32.554 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:32.554 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.555 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T13:57:32.555 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:32.555 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.555 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:32.555 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6822", 2024-09-21T13:57:32.555 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T13:57:32.555 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.555 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.555 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:32.555 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6823", 2024-09-21T13:57:32.555 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T13:57:32.555 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:32.555 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:32.556 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.556 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T13:57:32.556 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:32.556 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.556 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:32.556 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6820", 2024-09-21T13:57:32.556 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T13:57:32.556 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.556 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.556 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:32.556 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6821", 2024-09-21T13:57:32.556 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T13:57:32.556 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:32.557 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:32.557 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.557 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.187:6817/2728426525", 2024-09-21T13:57:32.557 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.187:6819/2728426525", 2024-09-21T13:57:32.557 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.187:6823/2728426525", 2024-09-21T13:57:32.557 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.187:6821/2728426525", 2024-09-21T13:57:32.557 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T13:57:32.557 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T13:57:32.557 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T13:57:32.557 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:32.557 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.557 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.557 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 7, 2024-09-21T13:57:32.557 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "1e08d891-8747-402e-b4c2-aa3802ce8537", 2024-09-21T13:57:32.558 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T13:57:32.558 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T13:57:32.558 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T13:57:32.558 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T13:57:32.558 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T13:57:32.558 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T13:57:32.558 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 54, 2024-09-21T13:57:32.558 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 240, 2024-09-21T13:57:32.558 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T13:57:32.558 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T13:57:32.558 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T13:57:32.558 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:32.558 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.559 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:32.559 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6824", 2024-09-21T13:57:32.559 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3680850037 2024-09-21T13:57:32.559 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.559 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.559 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:32.559 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6825", 2024-09-21T13:57:32.559 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3680850037 2024-09-21T13:57:32.559 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:32.559 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:32.559 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.559 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T13:57:32.559 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:32.559 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.560 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:32.560 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6826", 2024-09-21T13:57:32.560 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3680850037 2024-09-21T13:57:32.560 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.560 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.560 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:32.560 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6827", 2024-09-21T13:57:32.560 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3680850037 2024-09-21T13:57:32.560 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:32.560 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:32.560 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.560 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T13:57:32.560 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:32.560 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.561 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:32.561 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6830", 2024-09-21T13:57:32.561 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3680850037 2024-09-21T13:57:32.561 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.561 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.561 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:32.561 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6831", 2024-09-21T13:57:32.561 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3680850037 2024-09-21T13:57:32.561 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:32.561 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:32.561 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.561 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T13:57:32.561 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:32.561 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.562 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:32.562 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6828", 2024-09-21T13:57:32.562 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3680850037 2024-09-21T13:57:32.562 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.562 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.562 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:32.562 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6829", 2024-09-21T13:57:32.562 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3680850037 2024-09-21T13:57:32.562 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:32.562 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:32.562 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.562 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.187:6825/3680850037", 2024-09-21T13:57:32.562 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.187:6827/3680850037", 2024-09-21T13:57:32.562 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.187:6831/3680850037", 2024-09-21T13:57:32.563 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.187:6829/3680850037", 2024-09-21T13:57:32.563 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T13:57:32.563 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T13:57:32.563 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T13:57:32.563 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:32.563 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:32.563 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T13:57:32.563 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_xinfo": [ 2024-09-21T13:57:32.563 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.563 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 0, 2024-09-21T13:57:32.563 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T13:57:32.563 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T13:57:32.563 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T13:57:32.563 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T13:57:32.564 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T13:57:32.564 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T13:56:36.792777+0000", 2024-09-21T13:57:32.564 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T13:57:32.564 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.564 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.564 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 1, 2024-09-21T13:57:32.564 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T13:57:32.564 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T13:57:32.564 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T13:57:32.564 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T13:57:32.564 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T13:57:32.564 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T13:56:45.418792+0000", 2024-09-21T13:57:32.564 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T13:57:32.564 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.565 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.565 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 2, 2024-09-21T13:57:32.565 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T13:57:32.565 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T13:57:32.565 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T13:57:32.565 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T13:57:32.565 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T13:57:32.565 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T13:57:27.804766+0000", 2024-09-21T13:57:32.565 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 100 2024-09-21T13:57:32.565 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.565 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.565 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 3, 2024-09-21T13:57:32.565 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T13:57:32.565 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T13:57:32.566 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T13:57:32.566 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T13:57:32.566 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T13:57:32.566 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T13:57:10.678447+0000", 2024-09-21T13:57:32.566 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 214 2024-09-21T13:57:32.566 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.566 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.566 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 4, 2024-09-21T13:57:32.566 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T13:57:32.566 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T13:57:32.566 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T13:57:32.566 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T13:57:32.567 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T13:57:32.567 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T13:56:32.445570+0000", 2024-09-21T13:57:32.567 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T13:57:32.567 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.567 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.567 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 5, 2024-09-21T13:57:32.567 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T13:57:32.567 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T13:57:32.567 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T13:57:32.567 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T13:57:32.567 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T13:57:32.567 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T13:57:29.881898+0000", 2024-09-21T13:57:32.567 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T13:57:32.567 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.567 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.568 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 6, 2024-09-21T13:57:32.568 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T13:57:32.568 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T13:57:32.568 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T13:57:32.568 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T13:57:32.568 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 65536, 2024-09-21T13:57:32.568 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T13:57:00.402108+0000", 2024-09-21T13:57:32.568 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T13:57:32.568 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.568 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.568 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 7, 2024-09-21T13:57:32.568 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T13:57:32.568 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T13:57:32.568 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T13:57:32.568 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T13:57:32.569 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T13:57:32.569 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T13:56:59.703741+0000", 2024-09-21T13:57:32.569 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T13:57:32.569 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:32.569 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T13:57:32.569 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap": [ 2024-09-21T13:57:32.569 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.569 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "3.14", 2024-09-21T13:57:32.569 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-09-21T13:57:32.569 INFO:teuthology.orchestra.run.smithi049.stdout: 7, 2024-09-21T13:57:32.569 INFO:teuthology.orchestra.run.smithi049.stdout: 6, 2024-09-21T13:57:32.569 INFO:teuthology.orchestra.run.smithi049.stdout: 0 2024-09-21T13:57:32.569 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:32.569 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:32.570 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T13:57:32.570 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap_items": [ 2024-09-21T13:57:32.570 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.570 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "4.6", 2024-09-21T13:57:32.570 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-09-21T13:57:32.570 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.570 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 1, 2024-09-21T13:57:32.570 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 5 2024-09-21T13:57:32.570 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:32.570 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:32.570 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.570 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.570 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "4.1b", 2024-09-21T13:57:32.571 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-09-21T13:57:32.571 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.571 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 1, 2024-09-21T13:57:32.571 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 2 2024-09-21T13:57:32.571 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:32.571 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:32.571 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.571 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.571 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "4.1f", 2024-09-21T13:57:32.571 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-09-21T13:57:32.571 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.571 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 1, 2024-09-21T13:57:32.571 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 2 2024-09-21T13:57:32.571 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:32.572 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:32.572 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:32.572 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T13:57:32.572 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap_primaries": [], 2024-09-21T13:57:32.572 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_temp": [ 2024-09-21T13:57:32.572 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.572 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "8.0", 2024-09-21T13:57:32.572 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-09-21T13:57:32.572 INFO:teuthology.orchestra.run.smithi049.stdout: 1, 2024-09-21T13:57:32.572 INFO:teuthology.orchestra.run.smithi049.stdout: 5 2024-09-21T13:57:32.572 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:32.572 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.572 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.573 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "8.7", 2024-09-21T13:57:32.573 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-09-21T13:57:32.573 INFO:teuthology.orchestra.run.smithi049.stdout: 1, 2024-09-21T13:57:32.573 INFO:teuthology.orchestra.run.smithi049.stdout: 2, 2024-09-21T13:57:32.573 INFO:teuthology.orchestra.run.smithi049.stdout: 6 2024-09-21T13:57:32.573 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:32.573 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.573 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.573 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "8.9", 2024-09-21T13:57:32.573 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-09-21T13:57:32.573 INFO:teuthology.orchestra.run.smithi049.stdout: 5, 2024-09-21T13:57:32.573 INFO:teuthology.orchestra.run.smithi049.stdout: 0, 2024-09-21T13:57:32.573 INFO:teuthology.orchestra.run.smithi049.stdout: 6 2024-09-21T13:57:32.573 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:32.573 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.574 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:32.574 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "8.c", 2024-09-21T13:57:32.574 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-09-21T13:57:32.574 INFO:teuthology.orchestra.run.smithi049.stdout: 0, 2024-09-21T13:57:32.574 INFO:teuthology.orchestra.run.smithi049.stdout: 7 2024-09-21T13:57:32.574 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:32.574 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:32.574 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T13:57:32.574 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_temp": [], 2024-09-21T13:57:32.574 INFO:teuthology.orchestra.run.smithi049.stdout: "blocklist": { 2024-09-21T13:57:32.574 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/2731793917": "2024-09-22T13:49:27.311313+0000", 2024-09-21T13:57:32.574 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/799831425": "2024-09-22T13:49:27.311313+0000", 2024-09-21T13:57:32.574 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/2518197280": "2024-09-22T13:43:28.110499+0000", 2024-09-21T13:57:32.574 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3556829486": "2024-09-22T13:43:28.110499+0000", 2024-09-21T13:57:32.575 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/1249926115": "2024-09-22T13:49:27.311313+0000", 2024-09-21T13:57:32.575 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/231530138": "2024-09-22T13:43:28.110499+0000", 2024-09-21T13:57:32.575 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/4220781134": "2024-09-22T13:49:27.311313+0000", 2024-09-21T13:57:32.575 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/202325455": "2024-09-22T13:43:28.110499+0000", 2024-09-21T13:57:32.575 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/4129636473": "2024-09-22T13:43:56.633723+0000", 2024-09-21T13:57:32.575 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/2518197280": "2024-09-22T13:43:28.110499+0000", 2024-09-21T13:57:32.575 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3514741273": "2024-09-22T13:43:56.633723+0000", 2024-09-21T13:57:32.575 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/4129636473": "2024-09-22T13:43:56.633723+0000", 2024-09-21T13:57:32.575 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3125743776": "2024-09-22T13:43:56.633723+0000", 2024-09-21T13:57:32.575 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3600977363": "2024-09-22T13:43:56.633723+0000", 2024-09-21T13:57:32.575 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/3453571598": "2024-09-22T13:49:27.311313+0000", 2024-09-21T13:57:32.575 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/3453571598": "2024-09-22T13:49:27.311313+0000", 2024-09-21T13:57:32.575 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3553734432": "2024-09-22T13:49:27.311313+0000" 2024-09-21T13:57:32.575 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.575 INFO:teuthology.orchestra.run.smithi049.stdout: "range_blocklist": {}, 2024-09-21T13:57:32.576 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profiles": { 2024-09-21T13:57:32.576 INFO:teuthology.orchestra.run.smithi049.stdout: "default": { 2024-09-21T13:57:32.576 INFO:teuthology.orchestra.run.smithi049.stdout: "crush-failure-domain": "osd", 2024-09-21T13:57:32.576 INFO:teuthology.orchestra.run.smithi049.stdout: "k": "2", 2024-09-21T13:57:32.576 INFO:teuthology.orchestra.run.smithi049.stdout: "m": "1", 2024-09-21T13:57:32.576 INFO:teuthology.orchestra.run.smithi049.stdout: "plugin": "jerasure", 2024-09-21T13:57:32.576 INFO:teuthology.orchestra.run.smithi049.stdout: "technique": "reed_sol_van" 2024-09-21T13:57:32.576 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:32.576 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:32.576 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps_queue": [], 2024-09-21T13:57:32.576 INFO:teuthology.orchestra.run.smithi049.stdout: "new_removed_snaps": [], 2024-09-21T13:57:32.576 INFO:teuthology.orchestra.run.smithi049.stdout: "new_purged_snaps": [], 2024-09-21T13:57:32.576 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_node_flags": {}, 2024-09-21T13:57:32.576 INFO:teuthology.orchestra.run.smithi049.stdout: "device_class_flags": {}, 2024-09-21T13:57:32.576 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode": { 2024-09-21T13:57:32.577 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode_enabled": false, 2024-09-21T13:57:32.577 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_bucket_count": 0, 2024-09-21T13:57:32.577 INFO:teuthology.orchestra.run.smithi049.stdout: "degraded_stretch_mode": 0, 2024-09-21T13:57:32.577 INFO:teuthology.orchestra.run.smithi049.stdout: "recovering_stretch_mode": 0, 2024-09-21T13:57:32.577 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode_bucket": 0 2024-09-21T13:57:32.577 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:32.577 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T13:57:32.638 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:32.670 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:32.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:32.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:32.771 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:57:32.801 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:32.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:32.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:57:32.903 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:57:32.944 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:32.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:32.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:57:33.046 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:57:33.078 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-09-21T13:57:33.090 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:33.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:33.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:57:33.191 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:33.231 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:33.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:33.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:33.328 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:33 smithi049 ceph-mon[40489]: pgmap v420: 148 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 144 active+clean; 467 MiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 789 op/s; 913/22986 objects degraded (3.972%); 687/22986 objects misplaced (2.989%) 2024-09-21T13:57:33.328 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:33 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2846259174' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:57:33.328 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:33 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:57:33.328 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:33 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2339052093' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T13:57:33.328 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:33 smithi049 ceph-mon[40489]: 3.8 scrub starts 2024-09-21T13:57:33.329 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:33 smithi049 ceph-mon[30080]: pgmap v420: 148 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 144 active+clean; 467 MiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 789 op/s; 913/22986 objects degraded (3.972%); 687/22986 objects misplaced (2.989%) 2024-09-21T13:57:33.329 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:33 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2846259174' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:57:33.329 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:33 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:57:33.329 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:33 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2339052093' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T13:57:33.329 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:33 smithi049 ceph-mon[30080]: 3.8 scrub starts 2024-09-21T13:57:33.333 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:33.357 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:33.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:33.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:33.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:33 smithi187 ceph-mon[33032]: pgmap v420: 148 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 144 active+clean; 467 MiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 789 op/s; 913/22986 objects degraded (3.972%); 687/22986 objects misplaced (2.989%) 2024-09-21T13:57:33.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:33 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2846259174' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:57:33.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:33 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:57:33.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:33 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2339052093' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T13:57:33.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:33 smithi187 ceph-mon[33032]: 3.8 scrub starts 2024-09-21T13:57:33.459 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:33.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:33.488 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:33.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:33.590 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:33.631 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:33.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:33.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:33.661 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T13:57:33.661 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-09-21T13:57:33.662 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.662 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:250439 8.f 8:fa7d6c91:::benchmark_data_smithi049_109311_object123920:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-09-21T13:57:33.662 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:33.319672+0000", 2024-09-21T13:57:33.662 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.0080070639999999995, 2024-09-21T13:57:33.662 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0080366569999999991, 2024-09-21T13:57:33.662 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:33.662 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:33.662 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "waiting for sub ops", 2024-09-21T13:57:33.662 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:57:33.662 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:57:33.663 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:57:33.663 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 250439 2024-09-21T13:57:33.663 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.663 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:33.663 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.663 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:33.663 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.319672+0000", 2024-09-21T13:57:33.663 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:33.663 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.663 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.663 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:33.663 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.319672+0000", 2024-09-21T13:57:33.663 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:33.663 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.663 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.664 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:33.664 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.319673+0000", 2024-09-21T13:57:33.664 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.72e-06 2024-09-21T13:57:33.664 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.664 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.664 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:33.664 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.319680+0000", 2024-09-21T13:57:33.664 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6780000000000003e-06 2024-09-21T13:57:33.664 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.664 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.664 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:33.664 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.319698+0000", 2024-09-21T13:57:33.664 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8182999999999999e-05 2024-09-21T13:57:33.664 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.664 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.665 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:33.665 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.319707+0000", 2024-09-21T13:57:33.665 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.8040000000000007e-06 2024-09-21T13:57:33.665 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.665 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.665 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:33.665 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.319774+0000", 2024-09-21T13:57:33.665 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7002999999999995e-05 2024-09-21T13:57:33.665 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.665 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.665 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:57:33.665 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.319897+0000", 2024-09-21T13:57:33.665 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000122886 2024-09-21T13:57:33.666 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.666 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.666 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,2", 2024-09-21T13:57:33.666 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.320049+0000", 2024-09-21T13:57:33.666 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000152077 2024-09-21T13:57:33.666 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.666 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.666 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:33.666 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.326774+0000", 2024-09-21T13:57:33.666 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0067245459999999996 2024-09-21T13:57:33.666 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:33.666 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:33.666 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:33.666 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.667 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.667 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:250440 8.f 8:fc70da4a:::benchmark_data_smithi049_109311_object123921:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-09-21T13:57:33.667 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:33.319730+0000", 2024-09-21T13:57:33.667 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.0079486450000000007, 2024-09-21T13:57:33.667 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0080509780000000003, 2024-09-21T13:57:33.667 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:33.667 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:33.667 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "waiting for sub ops", 2024-09-21T13:57:33.667 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:57:33.667 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:57:33.667 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:57:33.667 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 250440 2024-09-21T13:57:33.667 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.668 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:33.668 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.668 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:33.668 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.319730+0000", 2024-09-21T13:57:33.668 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:33.668 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.668 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.668 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:33.668 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.319730+0000", 2024-09-21T13:57:33.668 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:33.668 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.669 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.669 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:33.669 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.319731+0000", 2024-09-21T13:57:33.669 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6499999999999999e-07 2024-09-21T13:57:33.669 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.669 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.669 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:33.669 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.319733+0000", 2024-09-21T13:57:33.669 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6819999999999999e-06 2024-09-21T13:57:33.669 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.669 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.669 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:33.669 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.319740+0000", 2024-09-21T13:57:33.670 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3579999999999999e-06 2024-09-21T13:57:33.670 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.670 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.670 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:33.670 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.319745+0000", 2024-09-21T13:57:33.670 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.062e-06 2024-09-21T13:57:33.670 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.670 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.670 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:33.670 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.324612+0000", 2024-09-21T13:57:33.670 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0048675059999999997 2024-09-21T13:57:33.670 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.670 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.670 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:57:33.671 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.324717+0000", 2024-09-21T13:57:33.671 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000105314 2024-09-21T13:57:33.671 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.671 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.671 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,2", 2024-09-21T13:57:33.671 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.325030+0000", 2024-09-21T13:57:33.671 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000312541 2024-09-21T13:57:33.671 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:33.671 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:33.671 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:33.671 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.671 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.671 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:250442 8.f 8:f92297bf:::benchmark_data_smithi049_109311_object123923:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-09-21T13:57:33.672 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:33.322784+0000", 2024-09-21T13:57:33.672 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.0048945179999999996, 2024-09-21T13:57:33.672 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0050571009999999996, 2024-09-21T13:57:33.672 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:33.672 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:33.672 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "waiting for sub ops", 2024-09-21T13:57:33.672 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:57:33.672 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:57:33.672 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:57:33.672 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 250442 2024-09-21T13:57:33.672 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.672 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:33.672 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.672 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:33.673 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.322784+0000", 2024-09-21T13:57:33.673 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:33.673 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.673 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.673 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:33.673 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.322784+0000", 2024-09-21T13:57:33.673 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:33.673 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.673 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.673 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:33.673 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.322786+0000", 2024-09-21T13:57:33.673 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7179999999999999e-06 2024-09-21T13:57:33.673 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.673 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.674 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:33.674 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.322821+0000", 2024-09-21T13:57:33.674 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4943000000000001e-05 2024-09-21T13:57:33.674 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.674 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.674 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:33.674 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.322842+0000", 2024-09-21T13:57:33.674 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1308999999999999e-05 2024-09-21T13:57:33.674 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.674 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.674 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:33.674 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.322852+0000", 2024-09-21T13:57:33.674 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6069999999999997e-06 2024-09-21T13:57:33.675 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.675 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.675 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:33.675 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.326813+0000", 2024-09-21T13:57:33.675 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0039608480000000003 2024-09-21T13:57:33.675 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.675 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.675 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:57:33.675 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.326927+0000", 2024-09-21T13:57:33.675 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011401200000000001 2024-09-21T13:57:33.675 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.675 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.675 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,2", 2024-09-21T13:57:33.675 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.327038+0000", 2024-09-21T13:57:33.675 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000111691 2024-09-21T13:57:33.676 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:33.676 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:33.676 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:33.676 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.676 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.676 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:250444 8.b 8:dddb8f1d:::benchmark_data_smithi049_109311_object123925:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-09-21T13:57:33.676 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:33.322868+0000", 2024-09-21T13:57:33.676 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.0048109320000000004, 2024-09-21T13:57:33.676 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0050351119999999996, 2024-09-21T13:57:33.676 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:33.676 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:33.676 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "waiting for sub ops", 2024-09-21T13:57:33.676 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:57:33.676 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:57:33.677 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:57:33.677 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 250444 2024-09-21T13:57:33.677 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.677 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:33.677 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.677 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:33.677 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.322868+0000", 2024-09-21T13:57:33.677 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:33.677 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.677 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.677 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:33.677 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.322868+0000", 2024-09-21T13:57:33.677 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:33.678 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.678 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.678 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:33.678 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.322868+0000", 2024-09-21T13:57:33.678 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9200000000000001e-07 2024-09-21T13:57:33.678 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.678 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.678 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:33.678 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.322872+0000", 2024-09-21T13:57:33.678 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8530000000000002e-06 2024-09-21T13:57:33.678 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.679 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.679 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:33.679 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.322889+0000", 2024-09-21T13:57:33.679 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6745e-05 2024-09-21T13:57:33.679 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.679 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.679 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:33.679 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.322896+0000", 2024-09-21T13:57:33.679 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0849999999999999e-06 2024-09-21T13:57:33.679 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.679 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.679 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:33.679 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.322996+0000", 2024-09-21T13:57:33.679 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000100386 2024-09-21T13:57:33.680 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.680 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.680 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:57:33.680 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.323122+0000", 2024-09-21T13:57:33.680 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012558299999999999 2024-09-21T13:57:33.680 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.680 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.680 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,4", 2024-09-21T13:57:33.680 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.323244+0000", 2024-09-21T13:57:33.680 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000121969 2024-09-21T13:57:33.680 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.680 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.680 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:33.680 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.326926+0000", 2024-09-21T13:57:33.681 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0036820249999999998 2024-09-21T13:57:33.681 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:33.681 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:33.681 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:33.681 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.681 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.681 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:250446 8.f 8.1091cc5f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-09-21T13:57:33.681 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:33.322919+0000", 2024-09-21T13:57:33.681 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.0047594300000000003, 2024-09-21T13:57:33.681 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.005043461, 2024-09-21T13:57:33.681 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:33.681 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:33.681 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "queued for pg", 2024-09-21T13:57:33.681 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:57:33.682 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:57:33.682 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:57:33.682 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 250446 2024-09-21T13:57:33.682 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.682 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:33.682 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.682 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:33.682 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.322919+0000", 2024-09-21T13:57:33.682 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:33.682 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.682 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.682 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:33.682 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.322919+0000", 2024-09-21T13:57:33.683 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:33.683 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.683 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.683 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:33.683 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.322920+0000", 2024-09-21T13:57:33.683 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2799999999999996e-07 2024-09-21T13:57:33.683 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.683 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.683 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:33.683 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.322923+0000", 2024-09-21T13:57:33.683 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.145e-06 2024-09-21T13:57:33.683 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.683 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.683 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:33.683 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.322931+0000", 2024-09-21T13:57:33.684 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0069999999999997e-06 2024-09-21T13:57:33.684 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.684 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.684 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:33.684 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.322958+0000", 2024-09-21T13:57:33.684 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.7407e-05 2024-09-21T13:57:33.684 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:33.684 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:33.684 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:33.684 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.684 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.684 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:250447 8.f 8.f89e8b9f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-09-21T13:57:33.684 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:33.323059+0000", 2024-09-21T13:57:33.684 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.0046201519999999998, 2024-09-21T13:57:33.685 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0049468130000000004, 2024-09-21T13:57:33.685 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:33.685 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:33.685 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "queued for pg", 2024-09-21T13:57:33.685 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:57:33.685 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:57:33.685 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:57:33.685 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 250447 2024-09-21T13:57:33.685 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.685 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:33.685 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.685 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:33.685 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.323059+0000", 2024-09-21T13:57:33.685 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:33.685 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.686 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.686 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:33.686 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.323059+0000", 2024-09-21T13:57:33.686 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:33.686 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.686 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.686 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:33.686 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.323059+0000", 2024-09-21T13:57:33.686 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1999999999999999e-07 2024-09-21T13:57:33.686 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.686 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.686 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:33.686 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.323063+0000", 2024-09-21T13:57:33.686 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9310000000000001e-06 2024-09-21T13:57:33.687 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.687 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.687 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:33.687 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.323073+0000", 2024-09-21T13:57:33.687 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.011e-05 2024-09-21T13:57:33.687 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.687 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.687 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:33.687 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.323080+0000", 2024-09-21T13:57:33.687 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4099999999999996e-06 2024-09-21T13:57:33.687 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:33.687 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:33.687 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:33.687 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.688 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.688 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:250448 8.b 8.575e378b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-09-21T13:57:33.688 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:33.327784+0000", 2024-09-21T13:57:33.688 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 4294967295.9998951, 2024-09-21T13:57:33.688 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00027223599999999997, 2024-09-21T13:57:33.688 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:33.688 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:33.688 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T13:57:33.688 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:57:33.688 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:57:33.688 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:57:33.688 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 250448 2024-09-21T13:57:33.688 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.689 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:33.689 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.689 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:33.689 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.327784+0000", 2024-09-21T13:57:33.689 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:33.689 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.689 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.689 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:33.689 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.327784+0000", 2024-09-21T13:57:33.689 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:33.689 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.689 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.690 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:33.690 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.327786+0000", 2024-09-21T13:57:33.690 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8819999999999999e-06 2024-09-21T13:57:33.690 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.690 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.690 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:33.690 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.327791+0000", 2024-09-21T13:57:33.690 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4809999999999997e-06 2024-09-21T13:57:33.690 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.690 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.690 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:33.690 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.327846+0000", 2024-09-21T13:57:33.690 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4932000000000002e-05 2024-09-21T13:57:33.690 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.691 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.691 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:33.691 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.327853+0000", 2024-09-21T13:57:33.691 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3760000000000002e-06 2024-09-21T13:57:33.691 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.691 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.691 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:33.691 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.328027+0000", 2024-09-21T13:57:33.691 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000173461 2024-09-21T13:57:33.691 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:33.691 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:33.691 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:33.691 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.691 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.692 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:250449 8.f 8.3637c93f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-09-21T13:57:33.692 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:33.327886+0000", 2024-09-21T13:57:33.692 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 4294967295.9997931, 2024-09-21T13:57:33.692 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00021875, 2024-09-21T13:57:33.692 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:33.692 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:33.692 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "queued for pg", 2024-09-21T13:57:33.692 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:57:33.692 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:57:33.692 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:57:33.692 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 250449 2024-09-21T13:57:33.692 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.692 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:33.693 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.693 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:33.693 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.327886+0000", 2024-09-21T13:57:33.693 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:33.693 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.693 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.693 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:33.693 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.327886+0000", 2024-09-21T13:57:33.693 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:33.693 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.693 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.693 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:33.693 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.327886+0000", 2024-09-21T13:57:33.694 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5700000000000002e-07 2024-09-21T13:57:33.694 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.694 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.694 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:33.694 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.327890+0000", 2024-09-21T13:57:33.694 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4419999999999998e-06 2024-09-21T13:57:33.694 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.694 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.694 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:33.694 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.327907+0000", 2024-09-21T13:57:33.694 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7652999999999999e-05 2024-09-21T13:57:33.694 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:33.694 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:33.695 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:33.695 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.327914+0000", 2024-09-21T13:57:33.695 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5289999999999997e-06 2024-09-21T13:57:33.695 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:33.695 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:33.695 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:33.695 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:33.695 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T13:57:33.695 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 8 2024-09-21T13:57:33.695 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T13:57:33.733 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:33.761 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:33.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:33.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:33.862 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:33.888 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:33.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:33.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:33.990 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:57:34.017 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:34.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:34.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:57:34.119 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:34.164 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:34.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:34.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:34.266 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:34.310 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:34.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:34.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:34.352 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:34 smithi049 ceph-mon[40489]: 3.8 scrub ok 2024-09-21T13:57:34.353 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:34 smithi049 ceph-mon[40489]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T13:57:34.353 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:34 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T13:57:34.353 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:34 smithi049 ceph-mon[40489]: osdmap e250: 8 total, 8 up, 7 in 2024-09-21T13:57:34.354 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:34 smithi049 ceph-mon[30080]: 3.8 scrub ok 2024-09-21T13:57:34.354 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:34 smithi049 ceph-mon[30080]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T13:57:34.354 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:34 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T13:57:34.354 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:34 smithi049 ceph-mon[30080]: osdmap e250: 8 total, 8 up, 7 in 2024-09-21T13:57:34.411 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:34.438 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:34.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:34.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:34.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:34 smithi187 ceph-mon[33032]: 3.8 scrub ok 2024-09-21T13:57:34.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:34 smithi187 ceph-mon[33032]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T13:57:34.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:34 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T13:57:34.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:34 smithi187 ceph-mon[33032]: osdmap e250: 8 total, 8 up, 7 in 2024-09-21T13:57:34.540 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:34.567 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:34.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:34.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:34.669 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:34.695 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:34.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:34.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:34.797 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:34.857 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:34.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:34.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:34.959 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:35.001 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:35.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:35.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:35.103 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:35.144 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:35.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:35.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:35.246 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:35.276 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:35.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:35.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:35.339 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Removed 126516 objects 2024-09-21T13:57:35.339 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Clean up completed and total clean up time :106.03 2024-09-21T13:57:35.373 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:35 smithi049 ceph-mon[40489]: pgmap v422: 148 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 144 active+clean; 316 MiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.19k op/s; 913/15738 objects degraded (5.801%); 453/15738 objects misplaced (2.878%) 2024-09-21T13:57:35.373 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:35 smithi049 ceph-mon[40489]: 8.8 scrub starts 2024-09-21T13:57:35.373 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:35 smithi049 ceph-mon[40489]: 8.8 scrub ok 2024-09-21T13:57:35.373 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:35 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 913/15738 objects degraded (5.801%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T13:57:35.373 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:35 smithi049 ceph-mon[40489]: osdmap e251: 8 total, 8 up, 7 in 2024-09-21T13:57:35.374 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:35 smithi049 ceph-mon[30080]: pgmap v422: 148 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 144 active+clean; 316 MiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.19k op/s; 913/15738 objects degraded (5.801%); 453/15738 objects misplaced (2.878%) 2024-09-21T13:57:35.374 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:35 smithi049 ceph-mon[30080]: 8.8 scrub starts 2024-09-21T13:57:35.374 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:35 smithi049 ceph-mon[30080]: 8.8 scrub ok 2024-09-21T13:57:35.374 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:35 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 913/15738 objects degraded (5.801%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T13:57:35.374 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:35 smithi049 ceph-mon[30080]: osdmap e251: 8 total, 8 up, 7 in 2024-09-21T13:57:35.378 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:35.400 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T13:57:35.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:35.401 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:35.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:35.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:35 smithi187 ceph-mon[33032]: pgmap v422: 148 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 144 active+clean; 316 MiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.19k op/s; 913/15738 objects degraded (5.801%); 453/15738 objects misplaced (2.878%) 2024-09-21T13:57:35.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:35 smithi187 ceph-mon[33032]: 8.8 scrub starts 2024-09-21T13:57:35.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:35 smithi187 ceph-mon[33032]: 8.8 scrub ok 2024-09-21T13:57:35.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:35 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 913/15738 objects degraded (5.801%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T13:57:35.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:35 smithi187 ceph-mon[33032]: osdmap e251: 8 total, 8 up, 7 in 2024-09-21T13:57:35.502 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:35.523 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:35.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:35.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:35.625 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:35.646 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:35.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:35.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:35.748 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:57:35.769 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:35.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:35.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:57:35.795 INFO:tasks.thrashosds.thrasher:j is {'epoch': 249, 'fsid': '40a3d2e8-781f-11ef-baf6-efdc52797490', 'created': '2024-09-21T13:42:57.838275+0000', 'modified': '2024-09-21T13:57:30.040104+0000', 'last_up_change': '2024-09-21T13:57:11.653688+0000', 'last_in_change': '2024-09-21T13:55:33.181161+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 20, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 8, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T13:46:21.096244+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '199', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.8685569763183594, 'average_primary_affinity': 0.8700000047683716, 'average_primary_affinity_weighted': 0.8700000047683716}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-21T13:48:55.540712+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 63, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6700000762939453, 'score_stable': 2.6700000762939453, 'optimal_score': 0.75, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8685569763183594, 'average_primary_affinity': 0.8700000047683716, 'average_primary_affinity_weighted': 0.8700000047683716}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-21T13:48:56.417456+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 0.8685569763183594, 'average_primary_affinity': 0.8700000047683716, 'average_primary_affinity_weighted': 0.8700000047683716}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-21T13:48:58.128455+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.490000009536743, 'score_stable': 2.490000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.190000057220459, 'raw_score_stable': 2.190000057220459, 'primary_affinity_weighted': 0.8685569763183594, 'average_primary_affinity': 0.8700000047683716, 'average_primary_affinity_weighted': 0.8700000047683716}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-21T13:49:00.069361+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 0.8685569763183594, 'average_primary_affinity': 0.8700000047683716, 'average_primary_affinity_weighted': 0.8700000047683716}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-21T13:49:02.127399+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8685569763183594, 'average_primary_affinity': 0.8700000047683716, 'average_primary_affinity_weighted': 0.8700000047683716}}, {'pool': 8, 'pool_name': 'unique_pool_1', 'create_time': '2024-09-21T13:54:10.659855+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '187', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.490000009536743, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.190000057220459, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8685569763183594, 'average_primary_affinity': 0.8700000047683716, 'average_primary_affinity_weighted': 0.8700000047683716}}], 'osds': [{'osd': 0, 'uuid': '2595dca9-6219-4e0c-8728-751eb102e73a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.1215972900390625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 241, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6802', 'nonce': 2095561412}, {'type': 'v1', 'addr': '172.21.15.49:6803', 'nonce': 2095561412}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6804', 'nonce': 2095561412}, {'type': 'v1', 'addr': '172.21.15.49:6805', 'nonce': 2095561412}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6808', 'nonce': 2095561412}, {'type': 'v1', 'addr': '172.21.15.49:6809', 'nonce': 2095561412}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6806', 'nonce': 2095561412}, {'type': 'v1', 'addr': '172.21.15.49:6807', 'nonce': 2095561412}]}, 'public_addr': '172.21.15.49:6803/2095561412', 'cluster_addr': '172.21.15.49:6805/2095561412', 'heartbeat_back_addr': '172.21.15.49:6809/2095561412', 'heartbeat_front_addr': '172.21.15.49:6807/2095561412', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '0ba327f9-5524-4a40-abcc-729ded0b1917', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 245, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6810', 'nonce': 1382840527}, {'type': 'v1', 'addr': '172.21.15.49:6811', 'nonce': 1382840527}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6812', 'nonce': 1382840527}, {'type': 'v1', 'addr': '172.21.15.49:6813', 'nonce': 1382840527}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6816', 'nonce': 1382840527}, {'type': 'v1', 'addr': '172.21.15.49:6817', 'nonce': 1382840527}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6814', 'nonce': 1382840527}, {'type': 'v1', 'addr': '172.21.15.49:6815', 'nonce': 1382840527}]}, 'public_addr': '172.21.15.49:6811/1382840527', 'cluster_addr': '172.21.15.49:6813/1382840527', 'heartbeat_back_addr': '172.21.15.49:6817/1382840527', 'heartbeat_front_addr': '172.21.15.49:6815/1382840527', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '27a45750-204d-48fd-bb8f-ebb01273343d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.8268585205078125, 'last_clean_begin': 18, 'last_clean_end': 100, 'up_from': 130, 'up_thru': 240, 'down_at': 101, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6818', 'nonce': 1945314741}, {'type': 'v1', 'addr': '172.21.15.49:6819', 'nonce': 1945314741}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6820', 'nonce': 1945314741}, {'type': 'v1', 'addr': '172.21.15.49:6821', 'nonce': 1945314741}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6824', 'nonce': 1945314741}, {'type': 'v1', 'addr': '172.21.15.49:6825', 'nonce': 1945314741}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6822', 'nonce': 1945314741}, {'type': 'v1', 'addr': '172.21.15.49:6823', 'nonce': 1945314741}]}, 'public_addr': '172.21.15.49:6819/1945314741', 'cluster_addr': '172.21.15.49:6821/1945314741', 'heartbeat_back_addr': '172.21.15.49:6825/1945314741', 'heartbeat_front_addr': '172.21.15.49:6823/1945314741', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '528bc390-3dc9-4f3b-ab75-d7412c9625db', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 27, 'last_clean_end': 214, 'up_from': 240, 'up_thru': 246, 'down_at': 215, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6826', 'nonce': 886735721}, {'type': 'v1', 'addr': '172.21.15.49:6827', 'nonce': 886735721}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6828', 'nonce': 886735721}, {'type': 'v1', 'addr': '172.21.15.49:6829', 'nonce': 886735721}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6832', 'nonce': 886735721}, {'type': 'v1', 'addr': '172.21.15.49:6833', 'nonce': 886735721}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6830', 'nonce': 886735721}, {'type': 'v1', 'addr': '172.21.15.49:6831', 'nonce': 886735721}]}, 'public_addr': '172.21.15.49:6827/886735721', 'cluster_addr': '172.21.15.49:6829/886735721', 'heartbeat_back_addr': '172.21.15.49:6833/886735721', 'heartbeat_front_addr': '172.21.15.49:6831/886735721', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '6fc64698-2d37-4db0-bf3d-c7e612c45163', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 240, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6800', 'nonce': 1702447776}, {'type': 'v1', 'addr': '172.21.15.187:6801', 'nonce': 1702447776}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6802', 'nonce': 1702447776}, {'type': 'v1', 'addr': '172.21.15.187:6803', 'nonce': 1702447776}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6806', 'nonce': 1702447776}, {'type': 'v1', 'addr': '172.21.15.187:6807', 'nonce': 1702447776}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6804', 'nonce': 1702447776}, {'type': 'v1', 'addr': '172.21.15.187:6805', 'nonce': 1702447776}]}, 'public_addr': '172.21.15.187:6801/1702447776', 'cluster_addr': '172.21.15.187:6803/1702447776', 'heartbeat_back_addr': '172.21.15.187:6807/1702447776', 'heartbeat_front_addr': '172.21.15.187:6805/1702447776', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '8d7f62a1-7e92-4e06-9cad-86b96be514d8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 244, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6808', 'nonce': 3659050776}, {'type': 'v1', 'addr': '172.21.15.187:6809', 'nonce': 3659050776}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6810', 'nonce': 3659050776}, {'type': 'v1', 'addr': '172.21.15.187:6811', 'nonce': 3659050776}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6814', 'nonce': 3659050776}, {'type': 'v1', 'addr': '172.21.15.187:6815', 'nonce': 3659050776}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6812', 'nonce': 3659050776}, {'type': 'v1', 'addr': '172.21.15.187:6813', 'nonce': 3659050776}]}, 'public_addr': '172.21.15.187:6809/3659050776', 'cluster_addr': '172.21.15.187:6811/3659050776', 'heartbeat_back_addr': '172.21.15.187:6815/3659050776', 'heartbeat_front_addr': '172.21.15.187:6813/3659050776', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '3a7afd13-26ce-444c-9cc0-81ce2eab6ce5', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 183, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6816', 'nonce': 2728426525}, {'type': 'v1', 'addr': '172.21.15.187:6817', 'nonce': 2728426525}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6818', 'nonce': 2728426525}, {'type': 'v1', 'addr': '172.21.15.187:6819', 'nonce': 2728426525}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6822', 'nonce': 2728426525}, {'type': 'v1', 'addr': '172.21.15.187:6823', 'nonce': 2728426525}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6820', 'nonce': 2728426525}, {'type': 'v1', 'addr': '172.21.15.187:6821', 'nonce': 2728426525}]}, 'public_addr': '172.21.15.187:6817/2728426525', 'cluster_addr': '172.21.15.187:6819/2728426525', 'heartbeat_back_addr': '172.21.15.187:6823/2728426525', 'heartbeat_front_addr': '172.21.15.187:6821/2728426525', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1e08d891-8747-402e-b4c2-aa3802ce8537', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 240, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 3680850037}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 3680850037}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 3680850037}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 3680850037}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6830', 'nonce': 3680850037}, {'type': 'v1', 'addr': '172.21.15.187:6831', 'nonce': 3680850037}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6828', 'nonce': 3680850037}, {'type': 'v1', 'addr': '172.21.15.187:6829', 'nonce': 3680850037}]}, 'public_addr': '172.21.15.187:6825/3680850037', 'cluster_addr': '172.21.15.187:6827/3680850037', 'heartbeat_back_addr': '172.21.15.187:6831/3680850037', 'heartbeat_front_addr': '172.21.15.187:6829/3680850037', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T13:56:36.792777+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T13:56:45.418792+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T13:57:27.804766+0000', 'dead_epoch': 100}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T13:57:10.678447+0000', 'dead_epoch': 214}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T13:56:32.445570+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T13:57:29.881898+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-21T13:57:00.402108+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T13:56:59.703741+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '3.14', 'osds': [7, 6, 0]}], 'pg_upmap_items': [{'pgid': '4.6', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.1b', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1f', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '8.0', 'osds': [1, 5]}, {'pgid': '8.7', 'osds': [1, 2, 6]}, {'pgid': '8.9', 'osds': [5, 0, 6]}, {'pgid': '8.c', 'osds': [0, 7]}], 'primary_temp': [], 'blocklist': {'172.21.15.49:0/2731793917': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:0/799831425': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:6801/2518197280': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:0/3556829486': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:0/1249926115': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:0/231530138': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:0/4220781134': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:0/202325455': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:6800/4129636473': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:6800/2518197280': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:0/3514741273': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:6801/4129636473': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:0/3125743776': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:0/3600977363': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:6801/3453571598': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:6800/3453571598': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:0/3553734432': '2024-09-22T13:49:27.311313+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-21T13:57:35.795 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T13:57:35.836 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:57:35.871 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:35.873 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:57:35.897 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:35.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:35.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:35.999 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:36.021 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:36.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:36.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:36.122 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:57:36.144 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:36.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:36.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:57:36.246 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:36.268 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:36.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:36.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:36.336 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:36 smithi049 ceph-mon[40489]: 6.1b scrub starts 2024-09-21T13:57:36.337 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:36 smithi049 ceph-mon[40489]: 6.1b scrub ok 2024-09-21T13:57:36.337 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:36 smithi049 ceph-mon[40489]: 5.15 scrub starts 2024-09-21T13:57:36.337 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:36 smithi049 ceph-mon[40489]: 5.15 scrub ok 2024-09-21T13:57:36.337 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:36 smithi049 ceph-mon[30080]: 6.1b scrub starts 2024-09-21T13:57:36.337 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:36 smithi049 ceph-mon[30080]: 6.1b scrub ok 2024-09-21T13:57:36.337 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:36 smithi049 ceph-mon[30080]: 5.15 scrub starts 2024-09-21T13:57:36.337 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:36 smithi049 ceph-mon[30080]: 5.15 scrub ok 2024-09-21T13:57:36.371 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:36.392 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:36.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:36.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:36.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:36 smithi187 ceph-mon[33032]: 6.1b scrub starts 2024-09-21T13:57:36.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:36 smithi187 ceph-mon[33032]: 6.1b scrub ok 2024-09-21T13:57:36.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:36 smithi187 ceph-mon[33032]: 5.15 scrub starts 2024-09-21T13:57:36.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:36 smithi187 ceph-mon[33032]: 5.15 scrub ok 2024-09-21T13:57:36.494 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:57:36.515 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:36.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:36.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:57:36.600 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T13:57:36.600 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-09-21T13:57:36.600 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-09-21T13:57:36.600 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-09-21T13:57:36.600 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T13:57:36.616 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:36.638 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:36.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:36.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:36.739 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:36.761 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:36.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:36.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:36.843 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T13:57:36.863 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:57:36.908 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:36.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:36.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:57:36.927 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:57:36.927 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:57:36.927 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:57:36.927 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:57:36.927 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:57:36.927 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:57:36.927 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:57:36.927 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:57:36.928 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:57:36.928 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:57:36.928 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:57:36.928 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:57:36.928 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:57:36.928 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:57:36.928 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:57:36.928 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:57:37.010 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:37.031 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:37.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:37.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:37.133 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:57:37.155 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:37.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:37.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:57:37.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:37 smithi049 ceph-mon[40489]: 5.12 scrub starts 2024-09-21T13:57:37.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:37 smithi049 ceph-mon[40489]: 5.12 scrub ok 2024-09-21T13:57:37.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:37 smithi049 ceph-mon[40489]: pgmap v424: 148 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 144 active+clean; 112 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.06k op/s; 200/5967 objects degraded (3.352%); 171/5967 objects misplaced (2.866%) 2024-09-21T13:57:37.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:37 smithi049 ceph-mon[40489]: 5.10 scrub starts 2024-09-21T13:57:37.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:37 smithi049 ceph-mon[40489]: 5.10 scrub ok 2024-09-21T13:57:37.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:37 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2723885745' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:57:37.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:37 smithi049 ceph-mon[30080]: 5.12 scrub starts 2024-09-21T13:57:37.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:37 smithi049 ceph-mon[30080]: 5.12 scrub ok 2024-09-21T13:57:37.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:37 smithi049 ceph-mon[30080]: pgmap v424: 148 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 144 active+clean; 112 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.06k op/s; 200/5967 objects degraded (3.352%); 171/5967 objects misplaced (2.866%) 2024-09-21T13:57:37.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:37 smithi049 ceph-mon[30080]: 5.10 scrub starts 2024-09-21T13:57:37.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:37 smithi049 ceph-mon[30080]: 5.10 scrub ok 2024-09-21T13:57:37.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:37 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2723885745' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:57:37.257 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:37.279 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:37.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:37.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:37.381 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:37.402 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:37.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:37.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:37.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:37 smithi187 ceph-mon[33032]: 5.12 scrub starts 2024-09-21T13:57:37.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:37 smithi187 ceph-mon[33032]: 5.12 scrub ok 2024-09-21T13:57:37.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:37 smithi187 ceph-mon[33032]: pgmap v424: 148 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 144 active+clean; 112 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.06k op/s; 200/5967 objects degraded (3.352%); 171/5967 objects misplaced (2.866%) 2024-09-21T13:57:37.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:37 smithi187 ceph-mon[33032]: 5.10 scrub starts 2024-09-21T13:57:37.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:37 smithi187 ceph-mon[33032]: 5.10 scrub ok 2024-09-21T13:57:37.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:37 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2723885745' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:57:37.504 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:57:37.529 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:37.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:37.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:57:37.631 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:37.657 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:37.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:37.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:37.759 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:37.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:37.786 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:37.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:37.887 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:37.914 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:37.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:37.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:38.016 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:38.044 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:38.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:38.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:38.145 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:38.167 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:38.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:38.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:38.269 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:38.291 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:38.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:38.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:38.323 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:38 smithi049 ceph-mon[30080]: 3.b scrub starts 2024-09-21T13:57:38.323 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:38 smithi049 ceph-mon[30080]: 3.b scrub ok 2024-09-21T13:57:38.323 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:38 smithi049 ceph-mon[30080]: 5.16 scrub starts 2024-09-21T13:57:38.323 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:38 smithi049 ceph-mon[30080]: 5.16 scrub ok 2024-09-21T13:57:38.324 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:38 smithi049 ceph-mon[30080]: 6.3 scrub starts 2024-09-21T13:57:38.324 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:38 smithi049 ceph-mon[30080]: 6.3 scrub ok 2024-09-21T13:57:38.324 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:38 smithi049 ceph-mon[40489]: 3.b scrub starts 2024-09-21T13:57:38.324 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:38 smithi049 ceph-mon[40489]: 3.b scrub ok 2024-09-21T13:57:38.324 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:38 smithi049 ceph-mon[40489]: 5.16 scrub starts 2024-09-21T13:57:38.324 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:38 smithi049 ceph-mon[40489]: 5.16 scrub ok 2024-09-21T13:57:38.324 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:38 smithi049 ceph-mon[40489]: 6.3 scrub starts 2024-09-21T13:57:38.324 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:38 smithi049 ceph-mon[40489]: 6.3 scrub ok 2024-09-21T13:57:38.393 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:38.415 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:38.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:38.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:38.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:38 smithi187 ceph-mon[33032]: 3.b scrub starts 2024-09-21T13:57:38.503 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:38 smithi187 ceph-mon[33032]: 3.b scrub ok 2024-09-21T13:57:38.503 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:38 smithi187 ceph-mon[33032]: 5.16 scrub starts 2024-09-21T13:57:38.503 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:38 smithi187 ceph-mon[33032]: 5.16 scrub ok 2024-09-21T13:57:38.503 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:38 smithi187 ceph-mon[33032]: 6.3 scrub starts 2024-09-21T13:57:38.503 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:38 smithi187 ceph-mon[33032]: 6.3 scrub ok 2024-09-21T13:57:38.517 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:57:38.538 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:38.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:38.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:57:38.598 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T13:57:38.634 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:57:38.640 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:57:38.698 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:38.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:38.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:57:38.800 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:57:38.823 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:38.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:38.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:57:38.925 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:38.946 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:38.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:38.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:39.008 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:57:39.048 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:57:39.073 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:39.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:39.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:57:39.174 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:57:39.198 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:39.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:39.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:57:39.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:39 smithi049 ceph-mon[40489]: 5.b scrub starts 2024-09-21T13:57:39.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:39 smithi049 ceph-mon[40489]: 5.b scrub ok 2024-09-21T13:57:39.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:39 smithi049 ceph-mon[40489]: 4.14 scrub starts 2024-09-21T13:57:39.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:39 smithi049 ceph-mon[40489]: 4.14 scrub ok 2024-09-21T13:57:39.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:39 smithi049 ceph-mon[40489]: pgmap v425: 148 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 144 active+clean; 112 MiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 655 B/s rd, 0 B/s wr, 813 op/s; 200/5967 objects degraded (3.352%); 171/5967 objects misplaced (2.866%) 2024-09-21T13:57:39.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:39 smithi049 ceph-mon[40489]: 3.6 scrub starts 2024-09-21T13:57:39.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:39 smithi049 ceph-mon[40489]: 3.6 scrub ok 2024-09-21T13:57:39.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:39 smithi049 ceph-mon[40489]: 8.d scrub starts 2024-09-21T13:57:39.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:39 smithi049 ceph-mon[40489]: 8.d scrub ok 2024-09-21T13:57:39.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:39 smithi049 ceph-mon[40489]: 4.1a scrub starts 2024-09-21T13:57:39.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:39 smithi049 ceph-mon[40489]: 4.1a scrub ok 2024-09-21T13:57:39.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:39 smithi049 ceph-mon[30080]: 5.b scrub starts 2024-09-21T13:57:39.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:39 smithi049 ceph-mon[30080]: 5.b scrub ok 2024-09-21T13:57:39.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:39 smithi049 ceph-mon[30080]: 4.14 scrub starts 2024-09-21T13:57:39.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:39 smithi049 ceph-mon[30080]: 4.14 scrub ok 2024-09-21T13:57:39.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:39 smithi049 ceph-mon[30080]: pgmap v425: 148 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 144 active+clean; 112 MiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 655 B/s rd, 0 B/s wr, 813 op/s; 200/5967 objects degraded (3.352%); 171/5967 objects misplaced (2.866%) 2024-09-21T13:57:39.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:39 smithi049 ceph-mon[30080]: 3.6 scrub starts 2024-09-21T13:57:39.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:39 smithi049 ceph-mon[30080]: 3.6 scrub ok 2024-09-21T13:57:39.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:39 smithi049 ceph-mon[30080]: 8.d scrub starts 2024-09-21T13:57:39.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:39 smithi049 ceph-mon[30080]: 8.d scrub ok 2024-09-21T13:57:39.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:39 smithi049 ceph-mon[30080]: 4.1a scrub starts 2024-09-21T13:57:39.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:39 smithi049 ceph-mon[30080]: 4.1a scrub ok 2024-09-21T13:57:39.299 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:39.323 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:39.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:39.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:39.424 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:39.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:39 smithi187 ceph-mon[33032]: 5.b scrub starts 2024-09-21T13:57:39.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:39 smithi187 ceph-mon[33032]: 5.b scrub ok 2024-09-21T13:57:39.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:39 smithi187 ceph-mon[33032]: 4.14 scrub starts 2024-09-21T13:57:39.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:39 smithi187 ceph-mon[33032]: 4.14 scrub ok 2024-09-21T13:57:39.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:39 smithi187 ceph-mon[33032]: pgmap v425: 148 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 144 active+clean; 112 MiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 655 B/s rd, 0 B/s wr, 813 op/s; 200/5967 objects degraded (3.352%); 171/5967 objects misplaced (2.866%) 2024-09-21T13:57:39.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:39 smithi187 ceph-mon[33032]: 3.6 scrub starts 2024-09-21T13:57:39.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:39 smithi187 ceph-mon[33032]: 3.6 scrub ok 2024-09-21T13:57:39.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:39 smithi187 ceph-mon[33032]: 8.d scrub starts 2024-09-21T13:57:39.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:39 smithi187 ceph-mon[33032]: 8.d scrub ok 2024-09-21T13:57:39.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:39 smithi187 ceph-mon[33032]: 4.1a scrub starts 2024-09-21T13:57:39.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:39 smithi187 ceph-mon[33032]: 4.1a scrub ok 2024-09-21T13:57:39.446 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:39.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:39.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:39.548 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:39.571 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:39.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:39.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:39.572 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-09-21T13:57:39.572 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-09-21T13:57:39.611 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:57:39.613 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-09-21T13:57:39.672 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:57:39.694 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:39.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:39.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:57:39.796 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:39.820 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:39.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:39.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:39.922 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:39.944 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:57:39 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:57:39.946 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:39.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:39.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:40.047 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:40.071 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:40.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:40.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:40.173 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T13:57:40.174 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:40.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:40.197 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:40.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:40.298 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:40.321 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:40.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:40.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:40.379 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:40 smithi049 ceph-mon[40489]: from='client.15459 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T13:57:40.379 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:40 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:57:40.379 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:40 smithi049 ceph-mon[40489]: 6.11 scrub starts 2024-09-21T13:57:40.379 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:40 smithi049 ceph-mon[40489]: 6.11 scrub ok 2024-09-21T13:57:40.379 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:40 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 200/5967 objects degraded (3.352%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T13:57:40.379 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:40 smithi049 ceph-mon[40489]: osdmap e252: 8 total, 8 up, 7 in 2024-09-21T13:57:40.379 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:40 smithi049 ceph-mon[30080]: from='client.15459 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T13:57:40.379 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:40 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:57:40.379 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:40 smithi049 ceph-mon[30080]: 6.11 scrub starts 2024-09-21T13:57:40.379 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:40 smithi049 ceph-mon[30080]: 6.11 scrub ok 2024-09-21T13:57:40.380 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:40 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 200/5967 objects degraded (3.352%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T13:57:40.380 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:40 smithi049 ceph-mon[30080]: osdmap e252: 8 total, 8 up, 7 in 2024-09-21T13:57:40.422 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:40.443 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:40.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:40.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:40.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:40 smithi187 ceph-mon[33032]: from='client.15459 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T13:57:40.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:40 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:57:40.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:40 smithi187 ceph-mon[33032]: 6.11 scrub starts 2024-09-21T13:57:40.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:40 smithi187 ceph-mon[33032]: 6.11 scrub ok 2024-09-21T13:57:40.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:40 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 200/5967 objects degraded (3.352%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T13:57:40.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:40 smithi187 ceph-mon[33032]: osdmap e252: 8 total, 8 up, 7 in 2024-09-21T13:57:40.545 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:40.566 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:40.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:40.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:40.667 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:40.688 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:40.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:40.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:40.754 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:57:40 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:57:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:57:40.790 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:40.814 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:40.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:40.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:40.832 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":425,"stamp":"2024-09-21T13:57:37.866551+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":117840435,"num_objects":1989,"num_object_clones":0,"num_object_copies":5967,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":200,"num_objects_misplaced":171,"num_objects_unfound":0,"num_objects_dirty":1989,"num_whiteouts":0,"num_read":1346,"num_read_kb":1203,"num_write":251793,"num_write_kb":8098698,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26671,"num_bytes_recovered":1744112145,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74952,"ondisk_log_size":74952,"up":444,"acting":442,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":3820268,"kb_used_data":1658976,"kb_used_omap":16,"kb_used_meta":2161135,"kb_avail":652395796,"statfs":{"total":671965249536,"available":668053295104,"internally_reserved":0,"allocated":1698791424,"data_stored":1696346194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16854,"internal_metadata":2213002794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,5,4,8,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-416088064,"num_objects":-6349,"num_object_clones":0,"num_object_copies":-19047,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-713,"num_objects_misplaced":-516,"num_objects_unfound":0,"num_objects_dirty":-6349,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":6349,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.813681"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":353,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.086145+0000","last_change":"2024-09-21T13:54:55.873382+0000","last_active":"2024-09-21T13:57:33.086145+0000","last_peered":"2024-09-21T13:57:33.086145+0000","last_clean":"2024-09-21T13:57:33.086145+0000","last_became_active":"2024-09-21T13:52:04.317322+0000","last_became_peered":"2024-09-21T13:52:04.317322+0000","last_unstale":"2024-09-21T13:57:33.086145+0000","last_undegraded":"2024-09-21T13:57:33.086145+0000","last_fullsized":"2024-09-21T13:57:33.086145+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:55.873301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:55.873301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"68'9","reported_seq":448,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.089152+0000","last_change":"2024-09-21T13:57:12.680771+0000","last_active":"2024-09-21T13:57:33.089152+0000","last_peered":"2024-09-21T13:57:33.089152+0000","last_clean":"2024-09-21T13:57:33.089152+0000","last_became_active":"2024-09-21T13:57:12.680389+0000","last_became_peered":"2024-09-21T13:57:12.680389+0000","last_unstale":"2024-09-21T13:57:33.089152+0000","last_undegraded":"2024-09-21T13:57:33.089152+0000","last_fullsized":"2024-09-21T13:57:33.089152+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:54:18.410821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:18.410821+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":392,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.094188+0000","last_change":"2024-09-21T13:50:28.058990+0000","last_active":"2024-09-21T13:57:33.094188+0000","last_peered":"2024-09-21T13:57:33.094188+0000","last_clean":"2024-09-21T13:57:33.094188+0000","last_became_active":"2024-09-21T13:49:02.060984+0000","last_became_peered":"2024-09-21T13:49:02.060984+0000","last_unstale":"2024-09-21T13:57:33.094188+0000","last_undegraded":"2024-09-21T13:57:33.094188+0000","last_fullsized":"2024-09-21T13:57:33.094188+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:28.058918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:50:28.058918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:57:36.939321+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":396,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.089714+0000","last_change":"2024-09-21T13:57:00.320623+0000","last_active":"2024-09-21T13:57:33.089714+0000","last_peered":"2024-09-21T13:57:33.089714+0000","last_clean":"2024-09-21T13:57:33.089714+0000","last_became_active":"2024-09-21T13:49:04.072460+0000","last_became_peered":"2024-09-21T13:49:04.072460+0000","last_unstale":"2024-09-21T13:57:33.089714+0000","last_undegraded":"2024-09-21T13:57:33.089714+0000","last_fullsized":"2024-09-21T13:57:33.089714+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:00.320510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:00.320510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:58:13.663983+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":360,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-21T13:57:35.002454+0000","last_change":"2024-09-21T13:57:34.696862+0000","last_active":"2024-09-21T13:57:35.002454+0000","last_peered":"2024-09-21T13:57:35.002454+0000","last_clean":"2024-09-21T13:57:35.002454+0000","last_became_active":"2024-09-21T13:57:12.691566+0000","last_became_peered":"2024-09-21T13:57:12.691566+0000","last_unstale":"2024-09-21T13:57:35.002454+0000","last_undegraded":"2024-09-21T13:57:35.002454+0000","last_fullsized":"2024-09-21T13:57:35.002454+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:34.696780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:34.696780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:59:02.539925+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":383,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-21T13:57:34.993987+0000","last_change":"2024-09-21T13:57:12.693116+0000","last_active":"2024-09-21T13:57:34.993987+0000","last_peered":"2024-09-21T13:57:34.993987+0000","last_clean":"2024-09-21T13:57:34.993987+0000","last_became_active":"2024-09-21T13:57:12.692835+0000","last_became_peered":"2024-09-21T13:57:12.692835+0000","last_unstale":"2024-09-21T13:57:34.993987+0000","last_undegraded":"2024-09-21T13:57:34.993987+0000","last_fullsized":"2024-09-21T13:57:34.993987+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:07.001135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:55:07.001135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"68'15","reported_seq":338,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-21T13:57:34.997557+0000","last_change":"2024-09-21T13:57:12.704165+0000","last_active":"2024-09-21T13:57:34.997557+0000","last_peered":"2024-09-21T13:57:34.997557+0000","last_clean":"2024-09-21T13:57:34.997557+0000","last_became_active":"2024-09-21T13:57:12.703469+0000","last_became_peered":"2024-09-21T13:57:12.703469+0000","last_unstale":"2024-09-21T13:57:34.997557+0000","last_undegraded":"2024-09-21T13:57:34.997557+0000","last_fullsized":"2024-09-21T13:57:34.997557+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:54:17.024053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:17.024053+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":407,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.089061+0000","last_change":"2024-09-21T13:55:34.218741+0000","last_active":"2024-09-21T13:57:33.089061+0000","last_peered":"2024-09-21T13:57:33.089061+0000","last_clean":"2024-09-21T13:57:33.089061+0000","last_became_active":"2024-09-21T13:55:34.218252+0000","last_became_peered":"2024-09-21T13:55:34.218252+0000","last_unstale":"2024-09-21T13:57:33.089061+0000","last_undegraded":"2024-09-21T13:57:33.089061+0000","last_fullsized":"2024-09-21T13:57:33.089061+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:43.040348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:43.040348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:57:42.307326+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":432,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.085689+0000","last_change":"2024-09-21T13:57:31.127971+0000","last_active":"2024-09-21T13:57:33.085689+0000","last_peered":"2024-09-21T13:57:33.085689+0000","last_clean":"2024-09-21T13:57:33.085689+0000","last_became_active":"2024-09-21T13:55:34.227297+0000","last_became_peered":"2024-09-21T13:55:34.227297+0000","last_unstale":"2024-09-21T13:57:33.085689+0000","last_undegraded":"2024-09-21T13:57:33.085689+0000","last_fullsized":"2024-09-21T13:57:33.085689+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:31.127926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:31.127926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:58:47.622028+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"68'9","reported_seq":431,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.089989+0000","last_change":"2024-09-21T13:57:12.683534+0000","last_active":"2024-09-21T13:57:33.089989+0000","last_peered":"2024-09-21T13:57:33.089989+0000","last_clean":"2024-09-21T13:57:33.089989+0000","last_became_active":"2024-09-21T13:57:12.681408+0000","last_became_peered":"2024-09-21T13:57:12.681408+0000","last_unstale":"2024-09-21T13:57:33.089989+0000","last_undegraded":"2024-09-21T13:57:33.089989+0000","last_fullsized":"2024-09-21T13:57:33.089989+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:53:58.308144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:58.308144+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":393,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.085412+0000","last_change":"2024-09-21T13:56:21.284112+0000","last_active":"2024-09-21T13:57:33.085412+0000","last_peered":"2024-09-21T13:57:33.085412+0000","last_clean":"2024-09-21T13:57:33.085412+0000","last_became_active":"2024-09-21T13:49:02.066693+0000","last_became_peered":"2024-09-21T13:49:02.066693+0000","last_unstale":"2024-09-21T13:57:33.085412+0000","last_undegraded":"2024-09-21T13:57:33.085412+0000","last_fullsized":"2024-09-21T13:57:33.085412+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:21.284048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:56:21.284048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":348,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.094195+0000","last_change":"2024-09-21T13:54:25.793052+0000","last_active":"2024-09-21T13:57:33.094195+0000","last_peered":"2024-09-21T13:57:33.094195+0000","last_clean":"2024-09-21T13:57:33.094195+0000","last_became_active":"2024-09-21T13:50:29.689381+0000","last_became_peered":"2024-09-21T13:50:29.689381+0000","last_unstale":"2024-09-21T13:57:33.094195+0000","last_undegraded":"2024-09-21T13:57:33.094195+0000","last_fullsized":"2024-09-21T13:57:33.094195+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:25.792985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:54:25.792985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":409,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.085577+0000","last_change":"2024-09-21T13:56:54.328338+0000","last_active":"2024-09-21T13:57:33.085577+0000","last_peered":"2024-09-21T13:57:33.085577+0000","last_clean":"2024-09-21T13:57:33.085577+0000","last_became_active":"2024-09-21T13:48:58.050906+0000","last_became_peered":"2024-09-21T13:48:58.050906+0000","last_unstale":"2024-09-21T13:57:33.085577+0000","last_undegraded":"2024-09-21T13:57:33.085577+0000","last_fullsized":"2024-09-21T13:57:33.085577+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:54.328275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:54.328275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:57:57.845961+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"68'5","reported_seq":449,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.090325+0000","last_change":"2024-09-21T13:57:29.549576+0000","last_active":"2024-09-21T13:57:33.090325+0000","last_peered":"2024-09-21T13:57:33.090325+0000","last_clean":"2024-09-21T13:57:33.090325+0000","last_became_active":"2024-09-21T13:57:12.682824+0000","last_became_peered":"2024-09-21T13:57:12.682824+0000","last_unstale":"2024-09-21T13:57:33.090325+0000","last_undegraded":"2024-09-21T13:57:33.090325+0000","last_fullsized":"2024-09-21T13:57:33.090325+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-21T13:57:29.549403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.549403+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:58:29.893658+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":400,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.094596+0000","last_change":"2024-09-21T13:54:26.857307+0000","last_active":"2024-09-21T13:57:33.094596+0000","last_peered":"2024-09-21T13:57:33.094596+0000","last_clean":"2024-09-21T13:57:33.094596+0000","last_became_active":"2024-09-21T13:49:02.060212+0000","last_became_peered":"2024-09-21T13:49:02.060212+0000","last_unstale":"2024-09-21T13:57:33.094596+0000","last_undegraded":"2024-09-21T13:57:33.094596+0000","last_fullsized":"2024-09-21T13:57:33.094596+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:26.857231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:26.857231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:58:03.533539+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":416,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.086208+0000","last_change":"2024-09-21T13:57:12.684804+0000","last_active":"2024-09-21T13:57:33.086208+0000","last_peered":"2024-09-21T13:57:33.086208+0000","last_clean":"2024-09-21T13:57:33.086208+0000","last_became_active":"2024-09-21T13:57:12.684385+0000","last_became_peered":"2024-09-21T13:57:12.684385+0000","last_unstale":"2024-09-21T13:57:33.086208+0000","last_undegraded":"2024-09-21T13:57:33.086208+0000","last_fullsized":"2024-09-21T13:57:33.086208+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:23.042969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:54:23.042969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":415,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.090766+0000","last_change":"2024-09-21T13:57:21.448697+0000","last_active":"2024-09-21T13:57:33.090766+0000","last_peered":"2024-09-21T13:57:33.090766+0000","last_clean":"2024-09-21T13:57:33.090766+0000","last_became_active":"2024-09-21T13:57:12.682250+0000","last_became_peered":"2024-09-21T13:57:12.682250+0000","last_unstale":"2024-09-21T13:57:33.090766+0000","last_undegraded":"2024-09-21T13:57:33.090766+0000","last_fullsized":"2024-09-21T13:57:33.090766+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:21.448653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:21.448653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:58:33.310155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":340,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.090562+0000","last_change":"2024-09-21T13:55:07.560698+0000","last_active":"2024-09-21T13:57:33.090562+0000","last_peered":"2024-09-21T13:57:33.090562+0000","last_clean":"2024-09-21T13:57:33.090562+0000","last_became_active":"2024-09-21T13:50:29.688282+0000","last_became_peered":"2024-09-21T13:50:29.688282+0000","last_unstale":"2024-09-21T13:57:33.090562+0000","last_undegraded":"2024-09-21T13:57:33.090562+0000","last_fullsized":"2024-09-21T13:57:33.090562+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:07.560623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:55:07.560623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"68'15","reported_seq":383,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-21T13:57:34.996575+0000","last_change":"2024-09-21T13:57:21.265288+0000","last_active":"2024-09-21T13:57:34.996575+0000","last_peered":"2024-09-21T13:57:34.996575+0000","last_clean":"2024-09-21T13:57:34.996575+0000","last_became_active":"2024-09-21T13:57:12.680783+0000","last_became_peered":"2024-09-21T13:57:12.680783+0000","last_unstale":"2024-09-21T13:57:34.996575+0000","last_undegraded":"2024-09-21T13:57:34.996575+0000","last_fullsized":"2024-09-21T13:57:34.996575+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:57:21.265191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:57:21.265191+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:58:38.399711+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":408,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.095599+0000","last_change":"2024-09-21T13:55:31.988001+0000","last_active":"2024-09-21T13:57:33.095599+0000","last_peered":"2024-09-21T13:57:33.095599+0000","last_clean":"2024-09-21T13:57:33.095599+0000","last_became_active":"2024-09-21T13:49:02.066796+0000","last_became_peered":"2024-09-21T13:49:02.066796+0000","last_unstale":"2024-09-21T13:57:33.095599+0000","last_undegraded":"2024-09-21T13:57:33.095599+0000","last_fullsized":"2024-09-21T13:57:33.095599+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:31.987854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:55:31.987854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":352,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-21T13:57:34.998177+0000","last_change":"2024-09-21T13:57:12.707696+0000","last_active":"2024-09-21T13:57:34.998177+0000","last_peered":"2024-09-21T13:57:34.998177+0000","last_clean":"2024-09-21T13:57:34.998177+0000","last_became_active":"2024-09-21T13:57:12.703020+0000","last_became_peered":"2024-09-21T13:57:12.703020+0000","last_unstale":"2024-09-21T13:57:34.998177+0000","last_undegraded":"2024-09-21T13:57:34.998177+0000","last_fullsized":"2024-09-21T13:57:34.998177+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:57.962266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:57.962266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":419,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.090165+0000","last_change":"2024-09-21T13:56:58.309928+0000","last_active":"2024-09-21T13:57:33.090165+0000","last_peered":"2024-09-21T13:57:33.090165+0000","last_clean":"2024-09-21T13:57:33.090165+0000","last_became_active":"2024-09-21T13:52:04.316016+0000","last_became_peered":"2024-09-21T13:52:04.316016+0000","last_unstale":"2024-09-21T13:57:33.090165+0000","last_undegraded":"2024-09-21T13:57:33.090165+0000","last_fullsized":"2024-09-21T13:57:33.090165+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:58.309863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:58.309863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:57:58.826937+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"68'12","reported_seq":359,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-21T13:57:34.993605+0000","last_change":"2024-09-21T13:57:12.684797+0000","last_active":"2024-09-21T13:57:34.993605+0000","last_peered":"2024-09-21T13:57:34.993605+0000","last_clean":"2024-09-21T13:57:34.993605+0000","last_became_active":"2024-09-21T13:57:12.684454+0000","last_became_peered":"2024-09-21T13:57:12.684454+0000","last_unstale":"2024-09-21T13:57:34.993605+0000","last_undegraded":"2024-09-21T13:57:34.993605+0000","last_fullsized":"2024-09-21T13:57:34.993605+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'12","last_scrub_stamp":"2024-09-21T13:55:02.031642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:55:02.031642+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":406,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.090090+0000","last_change":"2024-09-21T13:57:12.682412+0000","last_active":"2024-09-21T13:57:33.090090+0000","last_peered":"2024-09-21T13:57:33.090090+0000","last_clean":"2024-09-21T13:57:33.090090+0000","last_became_active":"2024-09-21T13:57:12.681897+0000","last_became_peered":"2024-09-21T13:57:12.681897+0000","last_unstale":"2024-09-21T13:57:33.090090+0000","last_undegraded":"2024-09-21T13:57:33.090090+0000","last_fullsized":"2024-09-21T13:57:33.090090+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:49:01.042371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:58:14.474874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":416,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.094930+0000","last_change":"2024-09-21T13:57:26.536739+0000","last_active":"2024-09-21T13:57:33.094930+0000","last_peered":"2024-09-21T13:57:33.094930+0000","last_clean":"2024-09-21T13:57:33.094930+0000","last_became_active":"2024-09-21T13:52:04.321358+0000","last_became_peered":"2024-09-21T13:52:04.321358+0000","last_unstale":"2024-09-21T13:57:33.094930+0000","last_undegraded":"2024-09-21T13:57:33.094930+0000","last_fullsized":"2024-09-21T13:57:33.094930+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T13:57:26.536684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:26.536684+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:58:49.659492+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":428,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.094330+0000","last_change":"2024-09-21T13:57:27.305798+0000","last_active":"2024-09-21T13:57:33.094330+0000","last_peered":"2024-09-21T13:57:33.094330+0000","last_clean":"2024-09-21T13:57:33.094330+0000","last_became_active":"2024-09-21T13:57:12.685742+0000","last_became_peered":"2024-09-21T13:57:12.685742+0000","last_unstale":"2024-09-21T13:57:33.094330+0000","last_undegraded":"2024-09-21T13:57:33.094330+0000","last_fullsized":"2024-09-21T13:57:33.094330+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:57:27.305724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:27.305724+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:58:49.478937+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"68'10","reported_seq":395,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.095429+0000","last_change":"2024-09-21T13:57:27.578931+0000","last_active":"2024-09-21T13:57:33.095429+0000","last_peered":"2024-09-21T13:57:33.095429+0000","last_clean":"2024-09-21T13:57:33.095429+0000","last_became_active":"2024-09-21T13:57:12.678630+0000","last_became_peered":"2024-09-21T13:57:12.678630+0000","last_unstale":"2024-09-21T13:57:33.095429+0000","last_undegraded":"2024-09-21T13:57:33.095429+0000","last_fullsized":"2024-09-21T13:57:33.095429+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:57:27.578738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:57:27.578738+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:58:47.723830+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"214'9","reported_seq":146,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.090203+0000","last_change":"2024-09-21T13:55:34.228871+0000","last_active":"2024-09-21T13:57:33.090203+0000","last_peered":"2024-09-21T13:57:33.090203+0000","last_clean":"2024-09-21T13:57:33.090203+0000","last_became_active":"2024-09-21T13:55:34.220182+0000","last_became_peered":"2024-09-21T13:55:34.220182+0000","last_unstale":"2024-09-21T13:57:33.090203+0000","last_undegraded":"2024-09-21T13:57:33.090203+0000","last_fullsized":"2024-09-21T13:57:33.090203+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:51:42.174629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:51:42.174629+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:57:43.286320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"250'15735","reported_seq":30077,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.951769+0000","last_change":"2024-09-21T13:56:17.840980+0000","last_active":"2024-09-21T13:57:33.951769+0000","last_peered":"2024-09-21T13:57:33.951769+0000","last_clean":"2024-09-21T13:57:33.951769+0000","last_became_active":"2024-09-21T13:56:17.826416+0000","last_became_peered":"2024-09-21T13:56:17.826416+0000","last_unstale":"2024-09-21T13:57:33.951769+0000","last_undegraded":"2024-09-21T13:57:33.951769+0000","last_fullsized":"2024-09-21T13:57:33.951769+0000","mapping_epoch":221,"log_start":"227'11467","ondisk_log_start":"227'11467","created":183,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":4268,"log_dups_size":0,"ondisk_log_size":4268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8585216,"num_objects":131,"num_object_clones":0,"num_object_copies":393,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15735,"num_write_kb":507712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7090,"num_bytes_recovered":464650240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"68'15","reported_seq":448,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.096367+0000","last_change":"2024-09-21T13:57:12.695917+0000","last_active":"2024-09-21T13:57:33.096367+0000","last_peered":"2024-09-21T13:57:33.096367+0000","last_clean":"2024-09-21T13:57:33.096367+0000","last_became_active":"2024-09-21T13:57:12.695609+0000","last_became_peered":"2024-09-21T13:57:12.695609+0000","last_unstale":"2024-09-21T13:57:33.096367+0000","last_undegraded":"2024-09-21T13:57:33.096367+0000","last_fullsized":"2024-09-21T13:57:33.096367+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:53:58.658150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:58.658150+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":351,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-21T13:57:34.997674+0000","last_change":"2024-09-21T13:57:32.668922+0000","last_active":"2024-09-21T13:57:34.997674+0000","last_peered":"2024-09-21T13:57:34.997674+0000","last_clean":"2024-09-21T13:57:34.997674+0000","last_became_active":"2024-09-21T13:57:12.689487+0000","last_became_peered":"2024-09-21T13:57:12.689487+0000","last_unstale":"2024-09-21T13:57:34.997674+0000","last_undegraded":"2024-09-21T13:57:34.997674+0000","last_fullsized":"2024-09-21T13:57:34.997674+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:32.668851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:32.668851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:58:33.406706+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"68'8","reported_seq":515,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.090883+0000","last_change":"2024-09-21T13:56:59.335345+0000","last_active":"2024-09-21T13:57:33.090883+0000","last_peered":"2024-09-21T13:57:33.090883+0000","last_clean":"2024-09-21T13:57:33.090883+0000","last_became_active":"2024-09-21T13:49:02.061679+0000","last_became_peered":"2024-09-21T13:49:02.061679+0000","last_unstale":"2024-09-21T13:57:33.090883+0000","last_undegraded":"2024-09-21T13:57:33.090883+0000","last_fullsized":"2024-09-21T13:57:33.090883+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:56:59.335171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:56:59.335171+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:58:18.356433+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":389,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.085339+0000","last_change":"2024-09-21T13:50:34.481503+0000","last_active":"2024-09-21T13:57:33.085339+0000","last_peered":"2024-09-21T13:57:33.085339+0000","last_clean":"2024-09-21T13:57:33.085339+0000","last_became_active":"2024-09-21T13:49:04.081905+0000","last_became_peered":"2024-09-21T13:49:04.081905+0000","last_unstale":"2024-09-21T13:57:33.085339+0000","last_undegraded":"2024-09-21T13:57:33.085339+0000","last_fullsized":"2024-09-21T13:57:33.085339+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:50:34.481450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:50:34.481450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:57:48.393597+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.c","version":"250'15858","reported_seq":16014,"reported_epoch":250,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2024-09-21T13:57:34.008072+0000","last_change":"2024-09-21T13:57:13.806160+0000","last_active":"2024-09-21T13:57:34.008072+0000","last_peered":"2024-09-21T13:57:34.008072+0000","last_clean":"2024-09-21T13:55:40.231671+0000","last_became_active":"2024-09-21T13:57:13.758249+0000","last_became_peered":"2024-09-21T13:57:13.758249+0000","last_unstale":"2024-09-21T13:57:34.008072+0000","last_undegraded":"2024-09-21T13:57:13.673327+0000","last_fullsized":"2024-09-21T13:57:13.661249+0000","mapping_epoch":241,"log_start":"226'10945","ondisk_log_start":"226'10945","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":4913,"log_dups_size":0,"ondisk_log_size":4913,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6160384,"num_objects":94,"num_object_clones":0,"num_object_copies":282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":94,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15858,"num_write_kb":510464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[{"shards":"0,7","objects":94}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"68'18","reported_seq":342,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-21T13:57:34.997772+0000","last_change":"2024-09-21T13:57:12.707964+0000","last_active":"2024-09-21T13:57:34.997772+0000","last_peered":"2024-09-21T13:57:34.997772+0000","last_clean":"2024-09-21T13:57:34.997772+0000","last_became_active":"2024-09-21T13:57:12.704646+0000","last_became_peered":"2024-09-21T13:57:12.704646+0000","last_unstale":"2024-09-21T13:57:34.997772+0000","last_undegraded":"2024-09-21T13:57:34.997772+0000","last_fullsized":"2024-09-21T13:57:34.997772+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'18","last_scrub_stamp":"2024-09-21T13:54:19.001030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:19.001030+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":303,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-21T13:57:34.998383+0000","last_change":"2024-09-21T13:57:25.736531+0000","last_active":"2024-09-21T13:57:34.998383+0000","last_peered":"2024-09-21T13:57:34.998383+0000","last_clean":"2024-09-21T13:57:34.998383+0000","last_became_active":"2024-09-21T13:57:12.707095+0000","last_became_peered":"2024-09-21T13:57:12.707095+0000","last_unstale":"2024-09-21T13:57:34.998383+0000","last_undegraded":"2024-09-21T13:57:34.998383+0000","last_fullsized":"2024-09-21T13:57:34.998383+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:25.736461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:25.736461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:58:49.195997+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":412,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.089041+0000","last_change":"2024-09-21T13:57:12.680995+0000","last_active":"2024-09-21T13:57:33.089041+0000","last_peered":"2024-09-21T13:57:33.089041+0000","last_clean":"2024-09-21T13:57:33.089041+0000","last_became_active":"2024-09-21T13:57:12.680637+0000","last_became_peered":"2024-09-21T13:57:12.680637+0000","last_unstale":"2024-09-21T13:57:33.089041+0000","last_undegraded":"2024-09-21T13:57:33.089041+0000","last_fullsized":"2024-09-21T13:57:33.089041+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:07.310044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:07.310044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":403,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.089894+0000","last_change":"2024-09-21T13:57:12.683187+0000","last_active":"2024-09-21T13:57:33.089894+0000","last_peered":"2024-09-21T13:57:33.089894+0000","last_clean":"2024-09-21T13:57:33.089894+0000","last_became_active":"2024-09-21T13:57:12.680372+0000","last_became_peered":"2024-09-21T13:57:12.680372+0000","last_unstale":"2024-09-21T13:57:33.089894+0000","last_undegraded":"2024-09-21T13:57:33.089894+0000","last_fullsized":"2024-09-21T13:57:33.089894+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:49:03.054313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:58:17.473240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"250'15670","reported_seq":15815,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.586080+0000","last_change":"2024-09-21T13:54:12.479429+0000","last_active":"2024-09-21T13:57:33.586080+0000","last_peered":"2024-09-21T13:57:33.586080+0000","last_clean":"2024-09-21T13:57:33.586080+0000","last_became_active":"2024-09-21T13:54:12.479146+0000","last_became_peered":"2024-09-21T13:54:12.479146+0000","last_unstale":"2024-09-21T13:57:33.586080+0000","last_undegraded":"2024-09-21T13:57:33.586080+0000","last_fullsized":"2024-09-21T13:57:33.586080+0000","mapping_epoch":183,"log_start":"225'10634","ondisk_log_start":"225'10634","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":5036,"log_dups_size":0,"ondisk_log_size":5036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9764888,"num_objects":150,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15670,"num_write_kb":506177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"68'14","reported_seq":449,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.089142+0000","last_change":"2024-09-21T13:56:32.438517+0000","last_active":"2024-09-21T13:57:33.089142+0000","last_peered":"2024-09-21T13:57:33.089142+0000","last_clean":"2024-09-21T13:57:33.089142+0000","last_became_active":"2024-09-21T13:55:34.221811+0000","last_became_peered":"2024-09-21T13:55:34.221811+0000","last_unstale":"2024-09-21T13:57:33.089142+0000","last_undegraded":"2024-09-21T13:57:33.089142+0000","last_fullsized":"2024-09-21T13:57:33.089142+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'14","last_scrub_stamp":"2024-09-21T13:56:32.438459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:56:32.438459+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:57:49.648696+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":410,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.093976+0000","last_change":"2024-09-21T13:56:17.439237+0000","last_active":"2024-09-21T13:57:33.093976+0000","last_peered":"2024-09-21T13:57:33.093976+0000","last_clean":"2024-09-21T13:57:33.093976+0000","last_became_active":"2024-09-21T13:48:58.053533+0000","last_became_peered":"2024-09-21T13:48:58.053533+0000","last_unstale":"2024-09-21T13:57:33.093976+0000","last_undegraded":"2024-09-21T13:57:33.093976+0000","last_fullsized":"2024-09-21T13:57:33.093976+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:56:17.439188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:17.439188+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:57:36.327959+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"241'10","reported_seq":391,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-21T13:57:34.998795+0000","last_change":"2024-09-21T13:57:12.687127+0000","last_active":"2024-09-21T13:57:34.998795+0000","last_peered":"2024-09-21T13:57:34.998795+0000","last_clean":"2024-09-21T13:57:34.998795+0000","last_became_active":"2024-09-21T13:57:12.686651+0000","last_became_peered":"2024-09-21T13:57:12.686651+0000","last_unstale":"2024-09-21T13:57:34.998795+0000","last_undegraded":"2024-09-21T13:57:34.998795+0000","last_fullsized":"2024-09-21T13:57:34.998795+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:54:25.020230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:25.020230+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":407,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.094825+0000","last_change":"2024-09-21T13:55:34.219902+0000","last_active":"2024-09-21T13:57:33.094825+0000","last_peered":"2024-09-21T13:57:33.094825+0000","last_clean":"2024-09-21T13:57:33.094825+0000","last_became_active":"2024-09-21T13:55:34.219596+0000","last_became_peered":"2024-09-21T13:55:34.219596+0000","last_unstale":"2024-09-21T13:57:33.094825+0000","last_undegraded":"2024-09-21T13:57:33.094825+0000","last_fullsized":"2024-09-21T13:57:33.094825+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:52:55.541232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:52:55.541232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:57:36.625211+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.e","version":"250'15685","reported_seq":15884,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:34.030592+0000","last_change":"2024-09-21T13:56:32.159709+0000","last_active":"2024-09-21T13:57:34.030592+0000","last_peered":"2024-09-21T13:57:34.030592+0000","last_clean":"2024-09-21T13:57:34.030592+0000","last_became_active":"2024-09-21T13:54:12.479901+0000","last_became_peered":"2024-09-21T13:54:12.479901+0000","last_unstale":"2024-09-21T13:57:34.030592+0000","last_undegraded":"2024-09-21T13:57:34.030592+0000","last_fullsized":"2024-09-21T13:57:34.030592+0000","mapping_epoch":183,"log_start":"225'10748","ondisk_log_start":"225'10748","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"226'11266","last_scrub_stamp":"2024-09-21T13:56:32.159396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:56:32.159396+0000","objects_scrubbed":4589,"log_size":4937,"log_dups_size":0,"ondisk_log_size":4937,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:57:53.618975+0000","scrub_duration":1649,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6094848,"num_objects":93,"num_object_clones":0,"num_object_copies":279,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15685,"num_write_kb":504896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"68'10","reported_seq":428,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.093526+0000","last_change":"2024-09-21T13:54:19.593696+0000","last_active":"2024-09-21T13:57:33.093526+0000","last_peered":"2024-09-21T13:57:33.093526+0000","last_clean":"2024-09-21T13:57:33.093526+0000","last_became_active":"2024-09-21T13:49:00.054564+0000","last_became_peered":"2024-09-21T13:49:00.054564+0000","last_unstale":"2024-09-21T13:57:33.093526+0000","last_undegraded":"2024-09-21T13:57:33.093526+0000","last_fullsized":"2024-09-21T13:57:33.093526+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:54:19.593611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:19.593611+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"61'1","reported_seq":454,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.085797+0000","last_change":"2024-09-21T13:57:12.685265+0000","last_active":"2024-09-21T13:57:33.085797+0000","last_peered":"2024-09-21T13:57:33.085797+0000","last_clean":"2024-09-21T13:57:33.085797+0000","last_became_active":"2024-09-21T13:57:12.684939+0000","last_became_peered":"2024-09-21T13:57:12.684939+0000","last_unstale":"2024-09-21T13:57:33.085797+0000","last_undegraded":"2024-09-21T13:57:33.085797+0000","last_fullsized":"2024-09-21T13:57:33.085797+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:54:59.927133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:59.927133+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"241'12","reported_seq":164,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.085364+0000","last_change":"2024-09-21T13:57:30.133441+0000","last_active":"2024-09-21T13:57:33.085364+0000","last_peered":"2024-09-21T13:57:33.085364+0000","last_clean":"2024-09-21T13:57:33.085364+0000","last_became_active":"2024-09-21T13:57:12.684197+0000","last_became_peered":"2024-09-21T13:57:12.684197+0000","last_unstale":"2024-09-21T13:57:33.085364+0000","last_undegraded":"2024-09-21T13:57:33.085364+0000","last_fullsized":"2024-09-21T13:57:33.085364+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'12","last_scrub_stamp":"2024-09-21T13:57:30.133392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:30.133392+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:58:46.287085+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":224,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-21T13:57:34.997909+0000","last_change":"2024-09-21T13:57:12.705272+0000","last_active":"2024-09-21T13:57:34.997909+0000","last_peered":"2024-09-21T13:57:34.997909+0000","last_clean":"2024-09-21T13:57:34.997909+0000","last_became_active":"2024-09-21T13:57:12.704897+0000","last_became_peered":"2024-09-21T13:57:12.704897+0000","last_unstale":"2024-09-21T13:57:34.997909+0000","last_undegraded":"2024-09-21T13:57:34.997909+0000","last_fullsized":"2024-09-21T13:57:34.997909+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:54.948705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:54.948705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.f","version":"251'16080","reported_seq":17294,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-21T13:57:35.332935+0000","last_change":"2024-09-21T13:57:26.066022+0000","last_active":"2024-09-21T13:57:35.332935+0000","last_peered":"2024-09-21T13:57:35.332935+0000","last_clean":"2024-09-21T13:57:35.332935+0000","last_became_active":"2024-09-21T13:57:26.047090+0000","last_became_peered":"2024-09-21T13:57:26.047090+0000","last_unstale":"2024-09-21T13:57:35.332935+0000","last_undegraded":"2024-09-21T13:57:35.332935+0000","last_fullsized":"2024-09-21T13:57:35.332935+0000","mapping_epoch":246,"log_start":"228'11769","ondisk_log_start":"228'11769","created":183,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'7254","last_scrub_stamp":"2024-09-21T13:55:34.086923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:55:34.086923+0000","objects_scrubbed":7254,"log_size":4311,"log_dups_size":0,"ondisk_log_size":4311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":804,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16080,"num_write_kb":514560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":486,"num_bytes_recovered":31850496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"68'19","reported_seq":398,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.085434+0000","last_change":"2024-09-21T13:56:55.320249+0000","last_active":"2024-09-21T13:57:33.085434+0000","last_peered":"2024-09-21T13:57:33.085434+0000","last_clean":"2024-09-21T13:57:33.085434+0000","last_became_active":"2024-09-21T13:50:29.690185+0000","last_became_peered":"2024-09-21T13:50:29.690185+0000","last_unstale":"2024-09-21T13:57:33.085434+0000","last_undegraded":"2024-09-21T13:57:33.085434+0000","last_fullsized":"2024-09-21T13:57:33.085434+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-21T13:56:55.319759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:56:55.319759+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:58:12.769815+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":412,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.096280+0000","last_change":"2024-09-21T13:56:22.886727+0000","last_active":"2024-09-21T13:57:33.096280+0000","last_peered":"2024-09-21T13:57:33.096280+0000","last_clean":"2024-09-21T13:57:33.096280+0000","last_became_active":"2024-09-21T13:52:04.321656+0000","last_became_peered":"2024-09-21T13:52:04.321656+0000","last_unstale":"2024-09-21T13:57:33.096280+0000","last_undegraded":"2024-09-21T13:57:33.096280+0000","last_fullsized":"2024-09-21T13:57:33.096280+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:22.886651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:22.886651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:57:42.457892+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":97,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.085197+0000","last_change":"2024-09-21T13:57:26.064893+0000","last_active":"2024-09-21T13:57:33.085197+0000","last_peered":"2024-09-21T13:57:33.085197+0000","last_clean":"2024-09-21T13:57:33.085197+0000","last_became_active":"2024-09-21T13:55:34.231370+0000","last_became_peered":"2024-09-21T13:55:34.231370+0000","last_unstale":"2024-09-21T13:57:33.085197+0000","last_undegraded":"2024-09-21T13:57:33.085197+0000","last_fullsized":"2024-09-21T13:57:33.085197+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:26.064832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:26.064832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:58:28.706892+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":407,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.094105+0000","last_change":"2024-09-21T13:55:34.971776+0000","last_active":"2024-09-21T13:57:33.094105+0000","last_peered":"2024-09-21T13:57:33.094105+0000","last_clean":"2024-09-21T13:57:33.094105+0000","last_became_active":"2024-09-21T13:55:34.220896+0000","last_became_peered":"2024-09-21T13:55:34.220896+0000","last_unstale":"2024-09-21T13:57:33.094105+0000","last_undegraded":"2024-09-21T13:57:33.094105+0000","last_fullsized":"2024-09-21T13:57:33.094105+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:34.971729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:55:34.971729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"250'15694","reported_seq":16851,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.564168+0000","last_change":"2024-09-21T13:57:22.020954+0000","last_active":"2024-09-21T13:57:33.564168+0000","last_peered":"2024-09-21T13:57:33.564168+0000","last_clean":"2024-09-21T13:57:33.564168+0000","last_became_active":"2024-09-21T13:57:21.990635+0000","last_became_peered":"2024-09-21T13:57:21.990635+0000","last_unstale":"2024-09-21T13:57:33.564168+0000","last_undegraded":"2024-09-21T13:57:33.564168+0000","last_fullsized":"2024-09-21T13:57:33.564168+0000","mapping_epoch":244,"log_start":"225'10646","ondisk_log_start":"225'10646","created":183,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":5048,"log_dups_size":0,"ondisk_log_size":5048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9961472,"num_objects":152,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15694,"num_write_kb":507072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":496,"num_bytes_recovered":32505856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"241'32","reported_seq":575,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.096603+0000","last_change":"2024-09-21T13:57:12.686047+0000","last_active":"2024-09-21T13:57:33.096603+0000","last_peered":"2024-09-21T13:57:33.096603+0000","last_clean":"2024-09-21T13:57:33.096603+0000","last_became_active":"2024-09-21T13:57:12.684690+0000","last_became_peered":"2024-09-21T13:57:12.684690+0000","last_unstale":"2024-09-21T13:57:33.096603+0000","last_undegraded":"2024-09-21T13:57:33.096603+0000","last_fullsized":"2024-09-21T13:57:33.096603+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'30","last_scrub_stamp":"2024-09-21T13:54:16.680953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:16.680953+0000","objects_scrubbed":10,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":408,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.090612+0000","last_change":"2024-09-21T13:55:34.229487+0000","last_active":"2024-09-21T13:57:33.090612+0000","last_peered":"2024-09-21T13:57:33.090612+0000","last_clean":"2024-09-21T13:57:33.090612+0000","last_became_active":"2024-09-21T13:55:34.225952+0000","last_became_peered":"2024-09-21T13:55:34.225952+0000","last_unstale":"2024-09-21T13:57:33.090612+0000","last_undegraded":"2024-09-21T13:57:33.090612+0000","last_fullsized":"2024-09-21T13:57:33.090612+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:11.464842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:55:11.464842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":439,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.085585+0000","last_change":"2024-09-21T13:55:34.246215+0000","last_active":"2024-09-21T13:57:33.085585+0000","last_peered":"2024-09-21T13:57:33.085585+0000","last_clean":"2024-09-21T13:57:33.085585+0000","last_became_active":"2024-09-21T13:55:34.229262+0000","last_became_peered":"2024-09-21T13:55:34.229262+0000","last_unstale":"2024-09-21T13:57:33.085585+0000","last_undegraded":"2024-09-21T13:57:33.085585+0000","last_fullsized":"2024-09-21T13:57:33.085585+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-21T13:54:25.029757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:54:25.029757+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0015946669999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":349,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.094411+0000","last_change":"2024-09-21T13:55:07.021128+0000","last_active":"2024-09-21T13:57:33.094411+0000","last_peered":"2024-09-21T13:57:33.094411+0000","last_clean":"2024-09-21T13:57:33.094411+0000","last_became_active":"2024-09-21T13:50:29.688249+0000","last_became_peered":"2024-09-21T13:50:29.688249+0000","last_unstale":"2024-09-21T13:57:33.094411+0000","last_undegraded":"2024-09-21T13:57:33.094411+0000","last_fullsized":"2024-09-21T13:57:33.094411+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:07.021059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:55:07.021059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":322,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-21T13:57:34.994403+0000","last_change":"2024-09-21T13:57:12.691118+0000","last_active":"2024-09-21T13:57:34.994403+0000","last_peered":"2024-09-21T13:57:34.994403+0000","last_clean":"2024-09-21T13:57:34.994403+0000","last_became_active":"2024-09-21T13:57:12.690868+0000","last_became_peered":"2024-09-21T13:57:12.690868+0000","last_unstale":"2024-09-21T13:57:34.994403+0000","last_undegraded":"2024-09-21T13:57:34.994403+0000","last_fullsized":"2024-09-21T13:57:34.994403+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:54.036699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:54:54.036699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"251'15714","reported_seq":16961,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-21T13:57:35.339024+0000","last_change":"2024-09-21T13:57:26.079345+0000","last_active":"2024-09-21T13:57:35.339024+0000","last_peered":"2024-09-21T13:57:35.339024+0000","last_clean":"2024-09-21T13:57:35.339024+0000","last_became_active":"2024-09-21T13:57:26.045615+0000","last_became_peered":"2024-09-21T13:57:26.045615+0000","last_unstale":"2024-09-21T13:57:35.339024+0000","last_undegraded":"2024-09-21T13:57:35.339024+0000","last_fullsized":"2024-09-21T13:57:35.339024+0000","mapping_epoch":246,"log_start":"227'11368","ondisk_log_start":"227'11368","created":183,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":4346,"log_dups_size":0,"ondisk_log_size":4346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15714,"num_write_kb":502848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":543,"num_bytes_recovered":35586048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"68'13","reported_seq":431,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.096461+0000","last_change":"2024-09-21T13:56:26.899618+0000","last_active":"2024-09-21T13:57:33.096461+0000","last_peered":"2024-09-21T13:57:33.096461+0000","last_clean":"2024-09-21T13:57:33.096461+0000","last_became_active":"2024-09-21T13:49:00.053323+0000","last_became_peered":"2024-09-21T13:49:00.053323+0000","last_unstale":"2024-09-21T13:57:33.096461+0000","last_undegraded":"2024-09-21T13:57:33.096461+0000","last_fullsized":"2024-09-21T13:57:33.096461+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'13","last_scrub_stamp":"2024-09-21T13:56:26.899280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:56:26.899280+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:57:33.544442+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":435,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.099029+0000","last_change":"2024-09-21T13:55:34.219235+0000","last_active":"2024-09-21T13:57:33.099029+0000","last_peered":"2024-09-21T13:57:33.099029+0000","last_clean":"2024-09-21T13:57:33.099029+0000","last_became_active":"2024-09-21T13:55:34.218932+0000","last_became_peered":"2024-09-21T13:55:34.218932+0000","last_unstale":"2024-09-21T13:57:33.099029+0000","last_undegraded":"2024-09-21T13:57:33.099029+0000","last_fullsized":"2024-09-21T13:57:33.099029+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:24.884932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:53:24.884932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:58:07.178767+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":341,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-21T13:57:34.996899+0000","last_change":"2024-09-21T13:57:12.682426+0000","last_active":"2024-09-21T13:57:34.996899+0000","last_peered":"2024-09-21T13:57:34.996899+0000","last_clean":"2024-09-21T13:57:34.996899+0000","last_became_active":"2024-09-21T13:57:12.682147+0000","last_became_peered":"2024-09-21T13:57:12.682147+0000","last_unstale":"2024-09-21T13:57:34.996899+0000","last_undegraded":"2024-09-21T13:57:34.996899+0000","last_fullsized":"2024-09-21T13:57:34.996899+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:53:19.984966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:53:19.984966+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:58:18.233139+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":282,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-21T13:57:34.997093+0000","last_change":"2024-09-21T13:54:26.009416+0000","last_active":"2024-09-21T13:57:34.997093+0000","last_peered":"2024-09-21T13:57:34.997093+0000","last_clean":"2024-09-21T13:57:34.997093+0000","last_became_active":"2024-09-21T13:52:04.320701+0000","last_became_peered":"2024-09-21T13:52:04.320701+0000","last_unstale":"2024-09-21T13:57:34.997093+0000","last_undegraded":"2024-09-21T13:57:34.997093+0000","last_fullsized":"2024-09-21T13:57:34.997093+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:26.009352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:26.009352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:58:07.183994+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":415,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.094957+0000","last_change":"2024-09-21T13:57:12.677533+0000","last_active":"2024-09-21T13:57:33.094957+0000","last_peered":"2024-09-21T13:57:33.094957+0000","last_clean":"2024-09-21T13:57:33.094957+0000","last_became_active":"2024-09-21T13:57:12.676655+0000","last_became_peered":"2024-09-21T13:57:12.676655+0000","last_unstale":"2024-09-21T13:57:33.094957+0000","last_undegraded":"2024-09-21T13:57:33.094957+0000","last_fullsized":"2024-09-21T13:57:33.094957+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:12.212391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:51:12.212391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:58:14.501510+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"250'15668","reported_seq":15812,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.958719+0000","last_change":"2024-09-21T13:54:12.477912+0000","last_active":"2024-09-21T13:57:33.958719+0000","last_peered":"2024-09-21T13:57:33.958719+0000","last_clean":"2024-09-21T13:57:33.958719+0000","last_became_active":"2024-09-21T13:54:12.477596+0000","last_became_peered":"2024-09-21T13:54:12.477596+0000","last_unstale":"2024-09-21T13:57:33.958719+0000","last_undegraded":"2024-09-21T13:57:33.958719+0000","last_fullsized":"2024-09-21T13:57:33.958719+0000","mapping_epoch":183,"log_start":"227'11359","ondisk_log_start":"227'11359","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":4309,"log_dups_size":0,"ondisk_log_size":4309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6422528,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15668,"num_write_kb":504512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"68'12","reported_seq":463,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.093808+0000","last_change":"2024-09-21T13:55:35.174733+0000","last_active":"2024-09-21T13:57:33.093808+0000","last_peered":"2024-09-21T13:57:33.093808+0000","last_clean":"2024-09-21T13:57:33.093808+0000","last_became_active":"2024-09-21T13:52:04.318875+0000","last_became_peered":"2024-09-21T13:52:04.318875+0000","last_unstale":"2024-09-21T13:57:33.093808+0000","last_undegraded":"2024-09-21T13:57:33.093808+0000","last_fullsized":"2024-09-21T13:57:33.093808+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'12","last_scrub_stamp":"2024-09-21T13:55:35.174647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:55:35.174647+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:58:05.749794+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":352,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.089282+0000","last_change":"2024-09-21T13:57:20.447876+0000","last_active":"2024-09-21T13:57:33.089282+0000","last_peered":"2024-09-21T13:57:33.089282+0000","last_clean":"2024-09-21T13:57:33.089282+0000","last_became_active":"2024-09-21T13:57:12.681030+0000","last_became_peered":"2024-09-21T13:57:12.681030+0000","last_unstale":"2024-09-21T13:57:33.089282+0000","last_undegraded":"2024-09-21T13:57:33.089282+0000","last_fullsized":"2024-09-21T13:57:33.089282+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:20.447801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:20.447801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:58:24.770307+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":929,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.096443+0000","last_change":"2024-09-21T13:56:29.617218+0000","last_active":"2024-09-21T13:57:33.096443+0000","last_peered":"2024-09-21T13:57:33.096443+0000","last_clean":"2024-09-21T13:57:33.096443+0000","last_became_active":"2024-09-21T13:48:57.046084+0000","last_became_peered":"2024-09-21T13:48:57.046084+0000","last_unstale":"2024-09-21T13:57:33.096443+0000","last_undegraded":"2024-09-21T13:57:33.096443+0000","last_fullsized":"2024-09-21T13:57:33.096443+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-21T13:56:29.617143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:56:29.617143+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:57:48.974900+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00047048000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":509,"num_read_kb":504,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":401,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.086102+0000","last_change":"2024-09-21T13:56:33.316275+0000","last_active":"2024-09-21T13:57:33.086102+0000","last_peered":"2024-09-21T13:57:33.086102+0000","last_clean":"2024-09-21T13:57:33.086102+0000","last_became_active":"2024-09-21T13:49:02.066641+0000","last_became_peered":"2024-09-21T13:49:02.066641+0000","last_unstale":"2024-09-21T13:57:33.086102+0000","last_undegraded":"2024-09-21T13:57:33.086102+0000","last_fullsized":"2024-09-21T13:57:33.086102+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:33.316230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:56:33.316230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:58:02.226050+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":416,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.093436+0000","last_change":"2024-09-21T13:57:12.685727+0000","last_active":"2024-09-21T13:57:33.093436+0000","last_peered":"2024-09-21T13:57:33.093436+0000","last_clean":"2024-09-21T13:57:33.093436+0000","last_became_active":"2024-09-21T13:57:12.684254+0000","last_became_peered":"2024-09-21T13:57:12.684254+0000","last_unstale":"2024-09-21T13:57:33.093436+0000","last_undegraded":"2024-09-21T13:57:33.093436+0000","last_fullsized":"2024-09-21T13:57:33.093436+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:34.020297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:55:34.020297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"250'15931","reported_seq":18160,"reported_epoch":250,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-21T13:57:33.583303+0000","last_change":"2024-09-21T13:57:13.830786+0000","last_active":"2024-09-21T13:57:33.583303+0000","last_peered":"2024-09-21T13:57:33.583303+0000","last_clean":"2024-09-21T13:57:11.636044+0000","last_became_active":"2024-09-21T13:57:12.770960+0000","last_became_peered":"2024-09-21T13:57:12.770960+0000","last_unstale":"2024-09-21T13:57:33.583303+0000","last_undegraded":"2024-09-21T13:57:33.583303+0000","last_fullsized":"2024-09-21T13:57:33.583303+0000","mapping_epoch":240,"log_start":"225'10897","ondisk_log_start":"225'10897","created":183,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":5034,"log_dups_size":0,"ondisk_log_size":5034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10289152,"num_objects":157,"num_object_clones":0,"num_object_copies":471,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":102,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15931,"num_write_kb":514816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1031,"num_bytes_recovered":67567616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,6],"avail_no_missing":["5","0","6"],"object_location_counts":[{"shards":"0,5,6","objects":157}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"214'17","reported_seq":175,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.090258+0000","last_change":"2024-09-21T13:55:34.247480+0000","last_active":"2024-09-21T13:57:33.090258+0000","last_peered":"2024-09-21T13:57:33.090258+0000","last_clean":"2024-09-21T13:57:33.090258+0000","last_became_active":"2024-09-21T13:55:34.224511+0000","last_became_peered":"2024-09-21T13:55:34.224511+0000","last_unstale":"2024-09-21T13:57:33.090258+0000","last_undegraded":"2024-09-21T13:57:33.090258+0000","last_fullsized":"2024-09-21T13:57:33.090258+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'16","last_scrub_stamp":"2024-09-21T13:52:19.452163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:52:19.452163+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:57:50.441275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":373,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-21T13:57:34.993701+0000","last_change":"2024-09-21T13:57:12.701919+0000","last_active":"2024-09-21T13:57:34.993701+0000","last_peered":"2024-09-21T13:57:34.993701+0000","last_clean":"2024-09-21T13:57:34.993701+0000","last_became_active":"2024-09-21T13:57:12.701615+0000","last_became_peered":"2024-09-21T13:57:12.701615+0000","last_unstale":"2024-09-21T13:57:34.993701+0000","last_undegraded":"2024-09-21T13:57:34.993701+0000","last_fullsized":"2024-09-21T13:57:34.993701+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:07.057210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:07.057210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"75'79","reported_seq":481,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.096520+0000","last_change":"2024-09-21T13:57:29.501187+0000","last_active":"2024-09-21T13:57:33.096520+0000","last_peered":"2024-09-21T13:57:33.096520+0000","last_clean":"2024-09-21T13:57:33.096520+0000","last_became_active":"2024-09-21T13:55:34.225437+0000","last_became_peered":"2024-09-21T13:55:34.225437+0000","last_unstale":"2024-09-21T13:57:33.096520+0000","last_undegraded":"2024-09-21T13:57:33.096520+0000","last_fullsized":"2024-09-21T13:57:33.096520+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:58:46.621682+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":404,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.095041+0000","last_change":"2024-09-21T13:54:52.829280+0000","last_active":"2024-09-21T13:57:33.095041+0000","last_peered":"2024-09-21T13:57:33.095041+0000","last_clean":"2024-09-21T13:57:33.095041+0000","last_became_active":"2024-09-21T13:52:04.320627+0000","last_became_peered":"2024-09-21T13:52:04.320627+0000","last_unstale":"2024-09-21T13:57:33.095041+0000","last_undegraded":"2024-09-21T13:57:33.095041+0000","last_fullsized":"2024-09-21T13:57:33.095041+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:52.829205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:52.829205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:58:10.494581+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":408,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.085902+0000","last_change":"2024-09-21T13:57:21.153067+0000","last_active":"2024-09-21T13:57:33.085902+0000","last_peered":"2024-09-21T13:57:33.085902+0000","last_clean":"2024-09-21T13:57:33.085902+0000","last_became_active":"2024-09-21T13:57:12.684495+0000","last_became_peered":"2024-09-21T13:57:12.684495+0000","last_unstale":"2024-09-21T13:57:33.085902+0000","last_undegraded":"2024-09-21T13:57:33.085902+0000","last_fullsized":"2024-09-21T13:57:33.085902+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:21.153018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:21.153018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:58:32.005391+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"250'15419","reported_seq":15564,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.499941+0000","last_change":"2024-09-21T13:54:12.378230+0000","last_active":"2024-09-21T13:57:33.499941+0000","last_peered":"2024-09-21T13:57:33.499941+0000","last_clean":"2024-09-21T13:57:33.499941+0000","last_became_active":"2024-09-21T13:54:12.377874+0000","last_became_peered":"2024-09-21T13:54:12.377874+0000","last_unstale":"2024-09-21T13:57:33.499941+0000","last_undegraded":"2024-09-21T13:57:33.499941+0000","last_fullsized":"2024-09-21T13:57:33.499941+0000","mapping_epoch":183,"log_start":"226'10838","ondisk_log_start":"226'10838","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":4581,"log_dups_size":0,"ondisk_log_size":4581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9764864,"num_objects":149,"num_object_clones":0,"num_object_copies":447,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15419,"num_write_kb":498176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"68'11","reported_seq":445,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.090859+0000","last_change":"2024-09-21T13:57:12.682768+0000","last_active":"2024-09-21T13:57:33.090859+0000","last_peered":"2024-09-21T13:57:33.090859+0000","last_clean":"2024-09-21T13:57:33.090859+0000","last_became_active":"2024-09-21T13:57:12.682426+0000","last_became_peered":"2024-09-21T13:57:12.682426+0000","last_unstale":"2024-09-21T13:57:33.090859+0000","last_undegraded":"2024-09-21T13:57:33.090859+0000","last_fullsized":"2024-09-21T13:57:33.090859+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:54:16.451588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:16.451588+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":419,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.089495+0000","last_change":"2024-09-21T13:56:53.377180+0000","last_active":"2024-09-21T13:57:33.089495+0000","last_peered":"2024-09-21T13:57:33.089495+0000","last_clean":"2024-09-21T13:57:33.089495+0000","last_became_active":"2024-09-21T13:52:04.318345+0000","last_became_peered":"2024-09-21T13:52:04.318345+0000","last_unstale":"2024-09-21T13:57:33.089495+0000","last_undegraded":"2024-09-21T13:57:33.089495+0000","last_fullsized":"2024-09-21T13:57:33.089495+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:53.377100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:53.377100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:58:16.696525+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":416,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.085881+0000","last_change":"2024-09-21T13:55:34.299788+0000","last_active":"2024-09-21T13:57:33.085881+0000","last_peered":"2024-09-21T13:57:33.085881+0000","last_clean":"2024-09-21T13:57:33.085881+0000","last_became_active":"2024-09-21T13:55:34.232726+0000","last_became_peered":"2024-09-21T13:55:34.232726+0000","last_unstale":"2024-09-21T13:57:33.085881+0000","last_undegraded":"2024-09-21T13:57:33.085881+0000","last_fullsized":"2024-09-21T13:57:33.085881+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:34.299722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:55:34.299722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":359,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-21T13:57:34.994420+0000","last_change":"2024-09-21T13:57:12.702103+0000","last_active":"2024-09-21T13:57:34.994420+0000","last_peered":"2024-09-21T13:57:34.994420+0000","last_clean":"2024-09-21T13:57:34.994420+0000","last_became_active":"2024-09-21T13:57:12.701804+0000","last_became_peered":"2024-09-21T13:57:12.701804+0000","last_unstale":"2024-09-21T13:57:34.994420+0000","last_undegraded":"2024-09-21T13:57:34.994420+0000","last_fullsized":"2024-09-21T13:57:34.994420+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:58.952218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:54:58.952218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1","version":"250'15350","reported_seq":15495,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.951968+0000","last_change":"2024-09-21T13:54:12.478397+0000","last_active":"2024-09-21T13:57:33.951968+0000","last_peered":"2024-09-21T13:57:33.951968+0000","last_clean":"2024-09-21T13:57:33.951968+0000","last_became_active":"2024-09-21T13:54:12.477588+0000","last_became_peered":"2024-09-21T13:54:12.477588+0000","last_unstale":"2024-09-21T13:57:33.951968+0000","last_undegraded":"2024-09-21T13:57:33.951968+0000","last_fullsized":"2024-09-21T13:57:33.951968+0000","mapping_epoch":183,"log_start":"227'11081","ondisk_log_start":"227'11081","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":4269,"log_dups_size":0,"ondisk_log_size":4269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7864320,"num_objects":120,"num_object_clones":0,"num_object_copies":360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15350,"num_write_kb":495040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"68'17","reported_seq":443,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.090316+0000","last_change":"2024-09-21T13:56:55.354217+0000","last_active":"2024-09-21T13:57:33.090316+0000","last_peered":"2024-09-21T13:57:33.090316+0000","last_clean":"2024-09-21T13:57:33.090316+0000","last_became_active":"2024-09-21T13:52:04.313218+0000","last_became_peered":"2024-09-21T13:52:04.313218+0000","last_unstale":"2024-09-21T13:57:33.090316+0000","last_undegraded":"2024-09-21T13:57:33.090316+0000","last_fullsized":"2024-09-21T13:57:33.090316+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'17","last_scrub_stamp":"2024-09-21T13:56:55.354126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:56:55.354126+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:58:15.294689+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":101,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.090080+0000","last_change":"2024-09-21T13:57:12.683902+0000","last_active":"2024-09-21T13:57:33.090080+0000","last_peered":"2024-09-21T13:57:33.090080+0000","last_clean":"2024-09-21T13:57:33.090080+0000","last_became_active":"2024-09-21T13:57:12.681869+0000","last_became_peered":"2024-09-21T13:57:12.681869+0000","last_unstale":"2024-09-21T13:57:33.090080+0000","last_undegraded":"2024-09-21T13:57:33.090080+0000","last_fullsized":"2024-09-21T13:57:33.090080+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:04.216734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:55:04.216734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":392,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.093543+0000","last_change":"2024-09-21T13:54:06.629867+0000","last_active":"2024-09-21T13:57:33.093543+0000","last_peered":"2024-09-21T13:57:33.093543+0000","last_clean":"2024-09-21T13:57:33.093543+0000","last_became_active":"2024-09-21T13:49:02.067472+0000","last_became_peered":"2024-09-21T13:49:02.067472+0000","last_unstale":"2024-09-21T13:57:33.093543+0000","last_undegraded":"2024-09-21T13:57:33.093543+0000","last_fullsized":"2024-09-21T13:57:33.093543+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:06.629822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:06.629822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":285,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-21T13:57:35.003605+0000","last_change":"2024-09-21T13:57:12.683321+0000","last_active":"2024-09-21T13:57:35.003605+0000","last_peered":"2024-09-21T13:57:35.003605+0000","last_clean":"2024-09-21T13:57:35.003605+0000","last_became_active":"2024-09-21T13:57:12.682979+0000","last_became_peered":"2024-09-21T13:57:12.682979+0000","last_unstale":"2024-09-21T13:57:35.003605+0000","last_undegraded":"2024-09-21T13:57:35.003605+0000","last_fullsized":"2024-09-21T13:57:35.003605+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:18.027944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:18.027944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:58:17.238003+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.0","version":"250'15596","reported_seq":15826,"reported_epoch":250,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2024-09-21T13:57:33.964597+0000","last_change":"2024-09-21T13:57:13.846097+0000","last_active":"2024-09-21T13:57:33.964597+0000","last_peered":"2024-09-21T13:57:33.964597+0000","last_clean":"2024-09-21T13:55:40.242099+0000","last_became_active":"2024-09-21T13:57:13.785875+0000","last_became_peered":"2024-09-21T13:57:13.785875+0000","last_unstale":"2024-09-21T13:57:33.964597+0000","last_undegraded":"2024-09-21T13:57:13.686462+0000","last_fullsized":"2024-09-21T13:57:13.660424+0000","mapping_epoch":241,"log_start":"226'11283","ondisk_log_start":"226'11283","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'7025","last_scrub_stamp":"2024-09-21T13:55:33.793361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:55:33.793361+0000","objects_scrubbed":7008,"log_size":4313,"log_dups_size":0,"ondisk_log_size":4313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1506,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6946816,"num_objects":106,"num_object_clones":0,"num_object_copies":318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15596,"num_write_kb":502464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":106}],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"68'10","reported_seq":449,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.090179+0000","last_change":"2024-09-21T13:57:12.681865+0000","last_active":"2024-09-21T13:57:33.090179+0000","last_peered":"2024-09-21T13:57:33.090179+0000","last_clean":"2024-09-21T13:57:33.090179+0000","last_became_active":"2024-09-21T13:57:12.681445+0000","last_became_peered":"2024-09-21T13:57:12.681445+0000","last_unstale":"2024-09-21T13:57:33.090179+0000","last_undegraded":"2024-09-21T13:57:33.090179+0000","last_fullsized":"2024-09-21T13:57:33.090179+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:54:20.425889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:20.425889+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":326,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-21T13:57:35.002244+0000","last_change":"2024-09-21T13:57:12.704508+0000","last_active":"2024-09-21T13:57:35.002244+0000","last_peered":"2024-09-21T13:57:35.002244+0000","last_clean":"2024-09-21T13:57:35.002244+0000","last_became_active":"2024-09-21T13:57:12.704203+0000","last_became_peered":"2024-09-21T13:57:12.704203+0000","last_unstale":"2024-09-21T13:57:35.002244+0000","last_undegraded":"2024-09-21T13:57:35.002244+0000","last_fullsized":"2024-09-21T13:57:35.002244+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:28.565766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:51:28.565766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"132'10","reported_seq":359,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-21T13:57:34.994416+0000","last_change":"2024-09-21T13:54:19.934832+0000","last_active":"2024-09-21T13:57:34.994416+0000","last_peered":"2024-09-21T13:57:34.994416+0000","last_clean":"2024-09-21T13:57:34.994416+0000","last_became_active":"2024-09-21T13:52:04.320441+0000","last_became_peered":"2024-09-21T13:52:04.320441+0000","last_unstale":"2024-09-21T13:57:34.994416+0000","last_undegraded":"2024-09-21T13:57:34.994416+0000","last_fullsized":"2024-09-21T13:57:34.994416+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'10","last_scrub_stamp":"2024-09-21T13:54:19.934756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:19.934756+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:57:39.295139+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"68'1","reported_seq":411,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.090700+0000","last_change":"2024-09-21T13:56:28.533220+0000","last_active":"2024-09-21T13:57:33.090700+0000","last_peered":"2024-09-21T13:57:33.090700+0000","last_clean":"2024-09-21T13:57:33.090700+0000","last_became_active":"2024-09-21T13:52:04.313210+0000","last_became_peered":"2024-09-21T13:52:04.313210+0000","last_unstale":"2024-09-21T13:57:33.090700+0000","last_undegraded":"2024-09-21T13:57:33.090700+0000","last_fullsized":"2024-09-21T13:57:33.090700+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T13:56:28.533106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:56:28.533106+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:57:37.328966+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.7","version":"250'15626","reported_seq":21611,"reported_epoch":250,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-21T13:57:33.968084+0000","last_change":"2024-09-21T13:57:13.822896+0000","last_active":"2024-09-21T13:57:33.968084+0000","last_peered":"2024-09-21T13:57:33.968084+0000","last_clean":"2024-09-21T13:57:11.653903+0000","last_became_active":"2024-09-21T13:57:12.747767+0000","last_became_peered":"2024-09-21T13:57:12.747767+0000","last_unstale":"2024-09-21T13:57:33.968084+0000","last_undegraded":"2024-09-21T13:57:33.968084+0000","last_fullsized":"2024-09-21T13:57:33.968084+0000","mapping_epoch":240,"log_start":"227'11339","ondisk_log_start":"227'11339","created":183,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":4287,"log_dups_size":0,"ondisk_log_size":4287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5767168,"num_objects":88,"num_object_clones":0,"num_object_copies":264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":69,"num_objects_unfound":0,"num_objects_dirty":88,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15626,"num_write_kb":502848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2901,"num_bytes_recovered":190119936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,2,6],"avail_no_missing":["1","2","6"],"object_location_counts":[{"shards":"1,2,6","objects":88}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"68'9","reported_seq":429,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.087475+0000","last_change":"2024-09-21T13:56:22.525463+0000","last_active":"2024-09-21T13:57:33.087475+0000","last_peered":"2024-09-21T13:57:33.087475+0000","last_clean":"2024-09-21T13:57:33.087475+0000","last_became_active":"2024-09-21T13:49:00.053016+0000","last_became_peered":"2024-09-21T13:49:00.053016+0000","last_unstale":"2024-09-21T13:57:33.087475+0000","last_undegraded":"2024-09-21T13:57:33.087475+0000","last_fullsized":"2024-09-21T13:57:33.087475+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:56:22.522682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:56:22.522682+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:57:48.362416+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":427,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.085651+0000","last_change":"2024-09-21T13:57:12.680439+0000","last_active":"2024-09-21T13:57:33.085651+0000","last_peered":"2024-09-21T13:57:33.085651+0000","last_clean":"2024-09-21T13:57:33.085651+0000","last_became_active":"2024-09-21T13:57:12.680129+0000","last_became_peered":"2024-09-21T13:57:12.680129+0000","last_unstale":"2024-09-21T13:57:33.085651+0000","last_undegraded":"2024-09-21T13:57:33.085651+0000","last_fullsized":"2024-09-21T13:57:33.085651+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:53.894506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:53.894506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":342,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.089738+0000","last_change":"2024-09-21T13:57:12.683161+0000","last_active":"2024-09-21T13:57:33.089738+0000","last_peered":"2024-09-21T13:57:33.089738+0000","last_clean":"2024-09-21T13:57:33.089738+0000","last_became_active":"2024-09-21T13:57:12.680978+0000","last_became_peered":"2024-09-21T13:57:12.680978+0000","last_unstale":"2024-09-21T13:57:33.089738+0000","last_undegraded":"2024-09-21T13:57:33.089738+0000","last_fullsized":"2024-09-21T13:57:33.089738+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:21.402946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:21.402946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":352,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.094582+0000","last_change":"2024-09-21T13:54:55.720381+0000","last_active":"2024-09-21T13:57:33.094582+0000","last_peered":"2024-09-21T13:57:33.094582+0000","last_clean":"2024-09-21T13:57:33.094582+0000","last_became_active":"2024-09-21T13:52:04.321703+0000","last_became_peered":"2024-09-21T13:52:04.321703+0000","last_unstale":"2024-09-21T13:57:33.094582+0000","last_undegraded":"2024-09-21T13:57:33.094582+0000","last_fullsized":"2024-09-21T13:57:33.094582+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:55.720307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:54:55.720307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:58:08.535834+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.6","version":"250'15621","reported_seq":29642,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.526047+0000","last_change":"2024-09-21T13:56:55.751883+0000","last_active":"2024-09-21T13:57:33.526047+0000","last_peered":"2024-09-21T13:57:33.526047+0000","last_clean":"2024-09-21T13:57:33.526047+0000","last_became_active":"2024-09-21T13:56:19.841596+0000","last_became_peered":"2024-09-21T13:56:19.841596+0000","last_unstale":"2024-09-21T13:57:33.526047+0000","last_undegraded":"2024-09-21T13:57:33.526047+0000","last_fullsized":"2024-09-21T13:57:33.526047+0000","mapping_epoch":223,"log_start":"225'10579","ondisk_log_start":"225'10579","created":183,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"232'13188","last_scrub_stamp":"2024-09-21T13:56:55.751652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:56:55.751652+0000","objects_scrubbed":2602,"log_size":5042,"log_dups_size":0,"ondisk_log_size":5042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:58:12.336490+0000","scrub_duration":1099,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9240576,"num_objects":141,"num_object_clones":0,"num_object_copies":423,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15621,"num_write_kb":504384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6914,"num_bytes_recovered":453115904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"68'19","reported_seq":168,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.093406+0000","last_change":"2024-09-21T13:56:34.031897+0000","last_active":"2024-09-21T13:57:33.093406+0000","last_peered":"2024-09-21T13:57:33.093406+0000","last_clean":"2024-09-21T13:57:33.093406+0000","last_became_active":"2024-09-21T13:55:34.230856+0000","last_became_peered":"2024-09-21T13:55:34.230856+0000","last_unstale":"2024-09-21T13:57:33.093406+0000","last_undegraded":"2024-09-21T13:57:33.093406+0000","last_fullsized":"2024-09-21T13:57:33.093406+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-21T13:56:34.031785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:56:34.031785+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:57:38.546268+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":20,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":426,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.095240+0000","last_change":"2024-09-21T13:57:21.587561+0000","last_active":"2024-09-21T13:57:33.095240+0000","last_peered":"2024-09-21T13:57:33.095240+0000","last_clean":"2024-09-21T13:57:33.095240+0000","last_became_active":"2024-09-21T13:57:12.676809+0000","last_became_peered":"2024-09-21T13:57:12.676809+0000","last_unstale":"2024-09-21T13:57:33.095240+0000","last_undegraded":"2024-09-21T13:57:33.095240+0000","last_fullsized":"2024-09-21T13:57:33.095240+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:21.587493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:21.587493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:58:31.484066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":360,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-21T13:57:37.261328+0000","last_change":"2024-09-21T13:57:37.261328+0000","last_active":"2024-09-21T13:57:37.261328+0000","last_peered":"2024-09-21T13:57:37.261328+0000","last_clean":"2024-09-21T13:57:37.261328+0000","last_became_active":"2024-09-21T13:52:04.321146+0000","last_became_peered":"2024-09-21T13:52:04.321146+0000","last_unstale":"2024-09-21T13:57:37.261328+0000","last_undegraded":"2024-09-21T13:57:37.261328+0000","last_fullsized":"2024-09-21T13:57:37.261328+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:37.261275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:37.261275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:58:43.891222+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":426,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.096472+0000","last_change":"2024-09-21T13:57:19.537244+0000","last_active":"2024-09-21T13:57:33.096472+0000","last_peered":"2024-09-21T13:57:33.096472+0000","last_clean":"2024-09-21T13:57:33.096472+0000","last_became_active":"2024-09-21T13:57:12.677008+0000","last_became_peered":"2024-09-21T13:57:12.677008+0000","last_unstale":"2024-09-21T13:57:33.096472+0000","last_undegraded":"2024-09-21T13:57:33.096472+0000","last_fullsized":"2024-09-21T13:57:33.096472+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:19.537177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:19.537177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:58:31.011809+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"250'15826","reported_seq":30255,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.495615+0000","last_change":"2024-09-21T13:56:17.849563+0000","last_active":"2024-09-21T13:57:33.495615+0000","last_peered":"2024-09-21T13:57:33.495615+0000","last_clean":"2024-09-21T13:57:33.495615+0000","last_became_active":"2024-09-21T13:56:17.827717+0000","last_became_peered":"2024-09-21T13:56:17.827717+0000","last_unstale":"2024-09-21T13:57:33.495615+0000","last_undegraded":"2024-09-21T13:57:33.495615+0000","last_fullsized":"2024-09-21T13:57:33.495615+0000","mapping_epoch":221,"log_start":"226'11183","ondisk_log_start":"226'11183","created":183,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:54:11.355929+0000","objects_scrubbed":0,"log_size":4643,"log_dups_size":0,"ondisk_log_size":4643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10878976,"num_objects":166,"num_object_clones":0,"num_object_copies":498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15826,"num_write_kb":511744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7134,"num_bytes_recovered":467533824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"68'12","reported_seq":446,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.089362+0000","last_change":"2024-09-21T13:57:12.681331+0000","last_active":"2024-09-21T13:57:33.089362+0000","last_peered":"2024-09-21T13:57:33.089362+0000","last_clean":"2024-09-21T13:57:33.089362+0000","last_became_active":"2024-09-21T13:57:12.680627+0000","last_became_peered":"2024-09-21T13:57:12.680627+0000","last_unstale":"2024-09-21T13:57:33.089362+0000","last_undegraded":"2024-09-21T13:57:33.089362+0000","last_fullsized":"2024-09-21T13:57:33.089362+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'12","last_scrub_stamp":"2024-09-21T13:55:31.405227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:55:31.405227+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":408,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.095452+0000","last_change":"2024-09-21T13:56:21.582382+0000","last_active":"2024-09-21T13:57:33.095452+0000","last_peered":"2024-09-21T13:57:33.095452+0000","last_clean":"2024-09-21T13:57:33.095452+0000","last_became_active":"2024-09-21T13:48:58.057641+0000","last_became_peered":"2024-09-21T13:48:58.057641+0000","last_unstale":"2024-09-21T13:57:33.095452+0000","last_undegraded":"2024-09-21T13:57:33.095452+0000","last_fullsized":"2024-09-21T13:57:33.095452+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:21.582325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:21.582325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:57:33.348726+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":325,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-21T13:57:34.992792+0000","last_change":"2024-09-21T13:54:55.924559+0000","last_active":"2024-09-21T13:57:34.992792+0000","last_peered":"2024-09-21T13:57:34.992792+0000","last_clean":"2024-09-21T13:57:34.992792+0000","last_became_active":"2024-09-21T13:52:04.322147+0000","last_became_peered":"2024-09-21T13:52:04.322147+0000","last_unstale":"2024-09-21T13:57:34.992792+0000","last_undegraded":"2024-09-21T13:57:34.992792+0000","last_fullsized":"2024-09-21T13:57:34.992792+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:55.924499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:55.924499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:58:11.160510+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":331,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-21T13:57:35.002755+0000","last_change":"2024-09-21T13:57:12.691770+0000","last_active":"2024-09-21T13:57:35.002755+0000","last_peered":"2024-09-21T13:57:35.002755+0000","last_clean":"2024-09-21T13:57:35.002755+0000","last_became_active":"2024-09-21T13:57:12.691451+0000","last_became_peered":"2024-09-21T13:57:12.691451+0000","last_unstale":"2024-09-21T13:57:35.002755+0000","last_undegraded":"2024-09-21T13:57:35.002755+0000","last_fullsized":"2024-09-21T13:57:35.002755+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:09.040062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:55:09.040062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"250'15771","reported_seq":15975,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.574442+0000","last_change":"2024-09-21T13:56:27.422951+0000","last_active":"2024-09-21T13:57:33.574442+0000","last_peered":"2024-09-21T13:57:33.574442+0000","last_clean":"2024-09-21T13:57:33.574442+0000","last_became_active":"2024-09-21T13:54:12.477455+0000","last_became_peered":"2024-09-21T13:54:12.477455+0000","last_unstale":"2024-09-21T13:57:33.574442+0000","last_undegraded":"2024-09-21T13:57:33.574442+0000","last_fullsized":"2024-09-21T13:57:33.574442+0000","mapping_epoch":183,"log_start":"225'10729","ondisk_log_start":"225'10729","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"226'10980","last_scrub_stamp":"2024-09-21T13:56:27.422860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:54:11.355929+0000","last_clean_scrub_stamp":"2024-09-21T13:56:27.422860+0000","objects_scrubbed":4973,"log_size":5042,"log_dups_size":0,"ondisk_log_size":5042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9502720,"num_objects":145,"num_object_clones":0,"num_object_copies":435,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15771,"num_write_kb":509312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"68'15","reported_seq":461,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.085175+0000","last_change":"2024-09-21T13:55:34.246686+0000","last_active":"2024-09-21T13:57:33.085175+0000","last_peered":"2024-09-21T13:57:33.085175+0000","last_clean":"2024-09-21T13:57:33.085175+0000","last_became_active":"2024-09-21T13:55:34.225106+0000","last_became_peered":"2024-09-21T13:55:34.225106+0000","last_unstale":"2024-09-21T13:57:33.085175+0000","last_undegraded":"2024-09-21T13:57:33.085175+0000","last_fullsized":"2024-09-21T13:57:33.085175+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:54:58.947824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:58.947824+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":428,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.095338+0000","last_change":"2024-09-21T13:55:03.879782+0000","last_active":"2024-09-21T13:57:33.095338+0000","last_peered":"2024-09-21T13:57:33.095338+0000","last_clean":"2024-09-21T13:57:33.095338+0000","last_became_active":"2024-09-21T13:48:58.057632+0000","last_became_peered":"2024-09-21T13:48:58.057632+0000","last_unstale":"2024-09-21T13:57:33.095338+0000","last_undegraded":"2024-09-21T13:57:33.095338+0000","last_fullsized":"2024-09-21T13:57:33.095338+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T13:55:03.879706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:55:03.879706+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"214'11","reported_seq":525,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.094681+0000","last_change":"2024-09-21T13:56:31.616078+0000","last_active":"2024-09-21T13:57:33.094681+0000","last_peered":"2024-09-21T13:57:33.094681+0000","last_clean":"2024-09-21T13:57:33.094681+0000","last_became_active":"2024-09-21T13:55:34.224011+0000","last_became_peered":"2024-09-21T13:55:34.224011+0000","last_unstale":"2024-09-21T13:57:33.094681+0000","last_undegraded":"2024-09-21T13:57:33.094681+0000","last_fullsized":"2024-09-21T13:57:33.094681+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'11","last_scrub_stamp":"2024-09-21T13:56:31.616003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:56:31.616003+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:57:40.970871+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":420,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.085188+0000","last_change":"2024-09-21T13:55:34.230207+0000","last_active":"2024-09-21T13:57:33.085188+0000","last_peered":"2024-09-21T13:57:33.085188+0000","last_clean":"2024-09-21T13:57:33.085188+0000","last_became_active":"2024-09-21T13:55:34.227177+0000","last_became_peered":"2024-09-21T13:55:34.227177+0000","last_unstale":"2024-09-21T13:57:33.085188+0000","last_undegraded":"2024-09-21T13:57:33.085188+0000","last_fullsized":"2024-09-21T13:57:33.085188+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:31.298739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:55:31.298739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":101,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.086777+0000","last_change":"2024-09-21T13:57:12.687060+0000","last_active":"2024-09-21T13:57:33.086777+0000","last_peered":"2024-09-21T13:57:33.086777+0000","last_clean":"2024-09-21T13:57:33.086777+0000","last_became_active":"2024-09-21T13:57:12.686788+0000","last_became_peered":"2024-09-21T13:57:12.686788+0000","last_unstale":"2024-09-21T13:57:33.086777+0000","last_undegraded":"2024-09-21T13:57:33.086777+0000","last_fullsized":"2024-09-21T13:57:33.086777+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:18.130969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:18.130969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:58:12.837235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"68'6","reported_seq":348,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-21T13:57:35.003060+0000","last_change":"2024-09-21T13:57:12.708571+0000","last_active":"2024-09-21T13:57:35.003060+0000","last_peered":"2024-09-21T13:57:35.003060+0000","last_clean":"2024-09-21T13:57:35.003060+0000","last_became_active":"2024-09-21T13:57:12.704406+0000","last_became_peered":"2024-09-21T13:57:12.704406+0000","last_unstale":"2024-09-21T13:57:35.003060+0000","last_undegraded":"2024-09-21T13:57:35.003060+0000","last_fullsized":"2024-09-21T13:57:35.003060+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'6","last_scrub_stamp":"2024-09-21T13:55:12.066873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:55:12.066873+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":420,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.085782+0000","last_change":"2024-09-21T13:57:12.702915+0000","last_active":"2024-09-21T13:57:33.085782+0000","last_peered":"2024-09-21T13:57:33.085782+0000","last_clean":"2024-09-21T13:57:33.085782+0000","last_became_active":"2024-09-21T13:57:12.702621+0000","last_became_peered":"2024-09-21T13:57:12.702621+0000","last_unstale":"2024-09-21T13:57:33.085782+0000","last_undegraded":"2024-09-21T13:57:33.085782+0000","last_fullsized":"2024-09-21T13:57:33.085782+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:18.124544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:18.124544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":411,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.094908+0000","last_change":"2024-09-21T13:55:34.226972+0000","last_active":"2024-09-21T13:57:33.094908+0000","last_peered":"2024-09-21T13:57:33.094908+0000","last_clean":"2024-09-21T13:57:33.094908+0000","last_became_active":"2024-09-21T13:55:34.226032+0000","last_became_peered":"2024-09-21T13:55:34.226032+0000","last_unstale":"2024-09-21T13:57:33.094908+0000","last_undegraded":"2024-09-21T13:57:33.094908+0000","last_fullsized":"2024-09-21T13:57:33.094908+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:02.838181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:55:02.838181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"68'9","reported_seq":311,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-21T13:57:34.994118+0000","last_change":"2024-09-21T13:57:12.703848+0000","last_active":"2024-09-21T13:57:34.994118+0000","last_peered":"2024-09-21T13:57:34.994118+0000","last_clean":"2024-09-21T13:57:34.994118+0000","last_became_active":"2024-09-21T13:57:12.703505+0000","last_became_peered":"2024-09-21T13:57:12.703505+0000","last_unstale":"2024-09-21T13:57:34.994118+0000","last_undegraded":"2024-09-21T13:57:34.994118+0000","last_fullsized":"2024-09-21T13:57:34.994118+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:55:04.990842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:55:04.990842+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"61'1","reported_seq":417,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.094741+0000","last_change":"2024-09-21T13:55:34.232616+0000","last_active":"2024-09-21T13:57:33.094741+0000","last_peered":"2024-09-21T13:57:33.094741+0000","last_clean":"2024-09-21T13:57:33.094741+0000","last_became_active":"2024-09-21T13:55:34.224478+0000","last_became_peered":"2024-09-21T13:55:34.224478+0000","last_unstale":"2024-09-21T13:57:33.094741+0000","last_undegraded":"2024-09-21T13:57:33.094741+0000","last_fullsized":"2024-09-21T13:57:33.094741+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:54:18.838906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:18.838906+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:58:04.536589+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":314,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-21T13:57:34.998263+0000","last_change":"2024-09-21T13:57:12.709006+0000","last_active":"2024-09-21T13:57:34.998263+0000","last_peered":"2024-09-21T13:57:34.998263+0000","last_clean":"2024-09-21T13:57:34.998263+0000","last_became_active":"2024-09-21T13:57:12.704920+0000","last_became_peered":"2024-09-21T13:57:12.704920+0000","last_unstale":"2024-09-21T13:57:34.998263+0000","last_undegraded":"2024-09-21T13:57:34.998263+0000","last_fullsized":"2024-09-21T13:57:34.998263+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:28.014760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:28.014760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":348,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-21T13:57:34.996196+0000","last_change":"2024-09-21T13:56:25.254421+0000","last_active":"2024-09-21T13:57:34.996196+0000","last_peered":"2024-09-21T13:57:34.996196+0000","last_clean":"2024-09-21T13:57:34.996196+0000","last_became_active":"2024-09-21T13:52:04.323549+0000","last_became_peered":"2024-09-21T13:52:04.323549+0000","last_unstale":"2024-09-21T13:57:34.996196+0000","last_undegraded":"2024-09-21T13:57:34.996196+0000","last_fullsized":"2024-09-21T13:57:34.996196+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:25.254251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:56:25.254251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:57:42.115577+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"68'9","reported_seq":432,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.085691+0000","last_change":"2024-09-21T13:57:24.124666+0000","last_active":"2024-09-21T13:57:33.085691+0000","last_peered":"2024-09-21T13:57:33.085691+0000","last_clean":"2024-09-21T13:57:33.085691+0000","last_became_active":"2024-09-21T13:57:12.684294+0000","last_became_peered":"2024-09-21T13:57:12.684294+0000","last_unstale":"2024-09-21T13:57:33.085691+0000","last_undegraded":"2024-09-21T13:57:33.085691+0000","last_fullsized":"2024-09-21T13:57:33.085691+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:57:24.124604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:57:24.124604+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:58:38.626948+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":360,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.094807+0000","last_change":"2024-09-21T13:57:20.574811+0000","last_active":"2024-09-21T13:57:33.094807+0000","last_peered":"2024-09-21T13:57:33.094807+0000","last_clean":"2024-09-21T13:57:33.094807+0000","last_became_active":"2024-09-21T13:57:12.676506+0000","last_became_peered":"2024-09-21T13:57:12.676506+0000","last_unstale":"2024-09-21T13:57:33.094807+0000","last_undegraded":"2024-09-21T13:57:33.094807+0000","last_fullsized":"2024-09-21T13:57:33.094807+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:20.574745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:20.574745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:58:32.032551+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"241'12","reported_seq":399,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-21T13:57:34.994518+0000","last_change":"2024-09-21T13:57:12.694268+0000","last_active":"2024-09-21T13:57:34.994518+0000","last_peered":"2024-09-21T13:57:34.994518+0000","last_clean":"2024-09-21T13:57:34.994518+0000","last_became_active":"2024-09-21T13:57:12.693714+0000","last_became_peered":"2024-09-21T13:57:12.693714+0000","last_unstale":"2024-09-21T13:57:34.994518+0000","last_undegraded":"2024-09-21T13:57:34.994518+0000","last_fullsized":"2024-09-21T13:57:34.994518+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'10","last_scrub_stamp":"2024-09-21T13:54:20.946782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:20.946782+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":407,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.090362+0000","last_change":"2024-09-21T13:53:51.285122+0000","last_active":"2024-09-21T13:57:33.090362+0000","last_peered":"2024-09-21T13:57:33.090362+0000","last_clean":"2024-09-21T13:57:33.090362+0000","last_became_active":"2024-09-21T13:52:04.315865+0000","last_became_peered":"2024-09-21T13:52:04.315865+0000","last_unstale":"2024-09-21T13:57:33.090362+0000","last_undegraded":"2024-09-21T13:57:33.090362+0000","last_fullsized":"2024-09-21T13:57:33.090362+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:51.285077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:51.285077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"68'10","reported_seq":329,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-21T13:57:34.994330+0000","last_change":"2024-09-21T13:57:12.677164+0000","last_active":"2024-09-21T13:57:34.994330+0000","last_peered":"2024-09-21T13:57:34.994330+0000","last_clean":"2024-09-21T13:57:34.994330+0000","last_became_active":"2024-09-21T13:57:12.676584+0000","last_became_peered":"2024-09-21T13:57:12.676584+0000","last_unstale":"2024-09-21T13:57:34.994330+0000","last_undegraded":"2024-09-21T13:57:34.994330+0000","last_fullsized":"2024-09-21T13:57:34.994330+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:51:11.545458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:51:11.545458+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":419,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.095013+0000","last_change":"2024-09-21T13:56:25.571313+0000","last_active":"2024-09-21T13:57:33.095013+0000","last_peered":"2024-09-21T13:57:33.095013+0000","last_clean":"2024-09-21T13:57:33.095013+0000","last_became_active":"2024-09-21T13:52:04.318006+0000","last_became_peered":"2024-09-21T13:52:04.318006+0000","last_unstale":"2024-09-21T13:57:33.095013+0000","last_undegraded":"2024-09-21T13:57:33.095013+0000","last_fullsized":"2024-09-21T13:57:33.095013+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:25.571256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:25.571256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:57:35.144451+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"68'8","reported_seq":505,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.085579+0000","last_change":"2024-09-21T13:54:03.065465+0000","last_active":"2024-09-21T13:57:33.085579+0000","last_peered":"2024-09-21T13:57:33.085579+0000","last_clean":"2024-09-21T13:57:33.085579+0000","last_became_active":"2024-09-21T13:49:02.067487+0000","last_became_peered":"2024-09-21T13:49:02.067487+0000","last_unstale":"2024-09-21T13:57:33.085579+0000","last_undegraded":"2024-09-21T13:57:33.085579+0000","last_fullsized":"2024-09-21T13:57:33.085579+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:54:03.065395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:03.065395+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":359,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.095238+0000","last_change":"2024-09-21T13:57:22.647259+0000","last_active":"2024-09-21T13:57:33.095238+0000","last_peered":"2024-09-21T13:57:33.095238+0000","last_clean":"2024-09-21T13:57:33.095238+0000","last_became_active":"2024-09-21T13:57:12.676965+0000","last_became_peered":"2024-09-21T13:57:12.676965+0000","last_unstale":"2024-09-21T13:57:33.095238+0000","last_undegraded":"2024-09-21T13:57:33.095238+0000","last_fullsized":"2024-09-21T13:57:33.095238+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:22.647191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:22.647191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:58:27.085856+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"68'11","reported_seq":450,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.090425+0000","last_change":"2024-09-21T13:55:34.251243+0000","last_active":"2024-09-21T13:57:33.090425+0000","last_peered":"2024-09-21T13:57:33.090425+0000","last_clean":"2024-09-21T13:57:33.090425+0000","last_became_active":"2024-09-21T13:55:34.228875+0000","last_became_peered":"2024-09-21T13:55:34.228875+0000","last_unstale":"2024-09-21T13:57:33.090425+0000","last_undegraded":"2024-09-21T13:57:33.090425+0000","last_fullsized":"2024-09-21T13:57:33.090425+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:54:56.539374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:56.539374+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":88,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.089904+0000","last_change":"2024-09-21T13:55:34.229228+0000","last_active":"2024-09-21T13:57:33.089904+0000","last_peered":"2024-09-21T13:57:33.089904+0000","last_clean":"2024-09-21T13:57:33.089904+0000","last_became_active":"2024-09-21T13:55:34.223763+0000","last_became_peered":"2024-09-21T13:55:34.223763+0000","last_unstale":"2024-09-21T13:57:33.089904+0000","last_undegraded":"2024-09-21T13:57:33.089904+0000","last_fullsized":"2024-09-21T13:57:33.089904+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:01.228213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:01.228213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":419,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.093311+0000","last_change":"2024-09-21T13:57:12.684450+0000","last_active":"2024-09-21T13:57:33.093311+0000","last_peered":"2024-09-21T13:57:33.093311+0000","last_clean":"2024-09-21T13:57:33.093311+0000","last_became_active":"2024-09-21T13:57:12.683646+0000","last_became_peered":"2024-09-21T13:57:12.683646+0000","last_unstale":"2024-09-21T13:57:33.093311+0000","last_undegraded":"2024-09-21T13:57:33.093311+0000","last_fullsized":"2024-09-21T13:57:33.093311+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:29.867278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:53:29.867278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":314,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-21T13:57:34.997180+0000","last_change":"2024-09-21T13:57:12.703833+0000","last_active":"2024-09-21T13:57:34.997180+0000","last_peered":"2024-09-21T13:57:34.997180+0000","last_clean":"2024-09-21T13:57:34.997180+0000","last_became_active":"2024-09-21T13:57:12.703037+0000","last_became_peered":"2024-09-21T13:57:12.703037+0000","last_unstale":"2024-09-21T13:57:34.997180+0000","last_undegraded":"2024-09-21T13:57:34.997180+0000","last_fullsized":"2024-09-21T13:57:34.997180+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:51:24.549706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:51:24.549706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"68'9","reported_seq":448,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.093625+0000","last_change":"2024-09-21T13:55:34.230877+0000","last_active":"2024-09-21T13:57:33.093625+0000","last_peered":"2024-09-21T13:57:33.093625+0000","last_clean":"2024-09-21T13:57:33.093625+0000","last_became_active":"2024-09-21T13:55:34.222314+0000","last_became_peered":"2024-09-21T13:55:34.222314+0000","last_unstale":"2024-09-21T13:57:33.093625+0000","last_undegraded":"2024-09-21T13:57:33.093625+0000","last_fullsized":"2024-09-21T13:57:33.093625+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:55:30.946032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:55:30.946032+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":411,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.095154+0000","last_change":"2024-09-21T13:57:12.678320+0000","last_active":"2024-09-21T13:57:33.095154+0000","last_peered":"2024-09-21T13:57:33.095154+0000","last_clean":"2024-09-21T13:57:33.095154+0000","last_became_active":"2024-09-21T13:57:12.677745+0000","last_became_peered":"2024-09-21T13:57:12.677745+0000","last_unstale":"2024-09-21T13:57:33.095154+0000","last_undegraded":"2024-09-21T13:57:33.095154+0000","last_fullsized":"2024-09-21T13:57:33.095154+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:06.687302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:06.687302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:58:16.496258+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":308,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-21T13:57:35.002158+0000","last_change":"2024-09-21T13:57:12.703746+0000","last_active":"2024-09-21T13:57:35.002158+0000","last_peered":"2024-09-21T13:57:35.002158+0000","last_clean":"2024-09-21T13:57:35.002158+0000","last_became_active":"2024-09-21T13:57:12.703441+0000","last_became_peered":"2024-09-21T13:57:12.703441+0000","last_unstale":"2024-09-21T13:57:35.002158+0000","last_undegraded":"2024-09-21T13:57:35.002158+0000","last_fullsized":"2024-09-21T13:57:35.002158+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:19.999125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:19.999125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":334,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.085181+0000","last_change":"2024-09-21T13:56:24.285096+0000","last_active":"2024-09-21T13:57:33.085181+0000","last_peered":"2024-09-21T13:57:33.085181+0000","last_clean":"2024-09-21T13:57:33.085181+0000","last_became_active":"2024-09-21T13:50:29.689045+0000","last_became_peered":"2024-09-21T13:50:29.689045+0000","last_unstale":"2024-09-21T13:57:33.085181+0000","last_undegraded":"2024-09-21T13:57:33.085181+0000","last_fullsized":"2024-09-21T13:57:33.085181+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:24.285035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:56:24.285035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"68'11","reported_seq":399,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-21T13:57:34.997330+0000","last_change":"2024-09-21T13:57:12.701888+0000","last_active":"2024-09-21T13:57:34.997330+0000","last_peered":"2024-09-21T13:57:34.997330+0000","last_clean":"2024-09-21T13:57:34.997330+0000","last_became_active":"2024-09-21T13:57:12.701403+0000","last_became_peered":"2024-09-21T13:57:12.701403+0000","last_unstale":"2024-09-21T13:57:34.997330+0000","last_undegraded":"2024-09-21T13:57:34.997330+0000","last_fullsized":"2024-09-21T13:57:34.997330+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:54:22.978267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:22.978267+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":401,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.085803+0000","last_change":"2024-09-21T13:56:17.305190+0000","last_active":"2024-09-21T13:57:33.085803+0000","last_peered":"2024-09-21T13:57:33.085803+0000","last_clean":"2024-09-21T13:57:33.085803+0000","last_became_active":"2024-09-21T13:48:58.051974+0000","last_became_peered":"2024-09-21T13:48:58.051974+0000","last_unstale":"2024-09-21T13:57:33.085803+0000","last_undegraded":"2024-09-21T13:57:33.085803+0000","last_fullsized":"2024-09-21T13:57:33.085803+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:17.305144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:17.305144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:57:43.818583+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":407,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.096273+0000","last_change":"2024-09-21T13:55:34.226519+0000","last_active":"2024-09-21T13:57:33.096273+0000","last_peered":"2024-09-21T13:57:33.096273+0000","last_clean":"2024-09-21T13:57:33.096273+0000","last_became_active":"2024-09-21T13:55:34.226187+0000","last_became_peered":"2024-09-21T13:55:34.226187+0000","last_unstale":"2024-09-21T13:57:33.096273+0000","last_undegraded":"2024-09-21T13:57:33.096273+0000","last_fullsized":"2024-09-21T13:57:33.096273+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:12.791948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:55:12.791948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":359,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-21T13:57:35.002338+0000","last_change":"2024-09-21T13:57:12.705237+0000","last_active":"2024-09-21T13:57:35.002338+0000","last_peered":"2024-09-21T13:57:35.002338+0000","last_clean":"2024-09-21T13:57:35.002338+0000","last_became_active":"2024-09-21T13:57:12.704942+0000","last_became_peered":"2024-09-21T13:57:12.704942+0000","last_unstale":"2024-09-21T13:57:35.002338+0000","last_undegraded":"2024-09-21T13:57:35.002338+0000","last_fullsized":"2024-09-21T13:57:35.002338+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:57.057668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:54:57.057668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"68'4","reported_seq":371,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-21T13:57:34.998896+0000","last_change":"2024-09-21T13:57:12.690593+0000","last_active":"2024-09-21T13:57:34.998896+0000","last_peered":"2024-09-21T13:57:34.998896+0000","last_clean":"2024-09-21T13:57:34.998896+0000","last_became_active":"2024-09-21T13:57:12.689428+0000","last_became_peered":"2024-09-21T13:57:12.689428+0000","last_unstale":"2024-09-21T13:57:34.998896+0000","last_undegraded":"2024-09-21T13:57:34.998896+0000","last_fullsized":"2024-09-21T13:57:34.998896+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4","last_scrub_stamp":"2024-09-21T13:53:27.826729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:53:27.826729+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":352,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.086025+0000","last_change":"2024-09-21T13:57:12.684681+0000","last_active":"2024-09-21T13:57:33.086025+0000","last_peered":"2024-09-21T13:57:33.086025+0000","last_clean":"2024-09-21T13:57:33.086025+0000","last_became_active":"2024-09-21T13:57:12.684372+0000","last_became_peered":"2024-09-21T13:57:12.684372+0000","last_unstale":"2024-09-21T13:57:33.086025+0000","last_undegraded":"2024-09-21T13:57:33.086025+0000","last_fullsized":"2024-09-21T13:57:33.086025+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:02.998650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:55:02.998650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":89,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.089040+0000","last_change":"2024-09-21T13:55:34.220495+0000","last_active":"2024-09-21T13:57:33.089040+0000","last_peered":"2024-09-21T13:57:33.089040+0000","last_clean":"2024-09-21T13:57:33.089040+0000","last_became_active":"2024-09-21T13:55:34.220162+0000","last_became_peered":"2024-09-21T13:55:34.220162+0000","last_unstale":"2024-09-21T13:57:33.089040+0000","last_undegraded":"2024-09-21T13:57:33.089040+0000","last_fullsized":"2024-09-21T13:57:33.089040+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:51.254456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:53:51.254456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:58:03.355996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":407,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.094599+0000","last_change":"2024-09-21T13:55:08.852877+0000","last_active":"2024-09-21T13:57:33.094599+0000","last_peered":"2024-09-21T13:57:33.094599+0000","last_clean":"2024-09-21T13:57:33.094599+0000","last_became_active":"2024-09-21T13:52:04.321667+0000","last_became_peered":"2024-09-21T13:52:04.321667+0000","last_unstale":"2024-09-21T13:57:33.094599+0000","last_undegraded":"2024-09-21T13:57:33.094599+0000","last_fullsized":"2024-09-21T13:57:33.094599+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:08.852789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:55:08.852789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":404,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.094320+0000","last_change":"2024-09-21T13:57:03.085160+0000","last_active":"2024-09-21T13:57:33.094320+0000","last_peered":"2024-09-21T13:57:33.094320+0000","last_clean":"2024-09-21T13:57:33.094320+0000","last_became_active":"2024-09-21T13:49:04.075562+0000","last_became_peered":"2024-09-21T13:49:04.075562+0000","last_unstale":"2024-09-21T13:57:33.094320+0000","last_undegraded":"2024-09-21T13:57:33.094320+0000","last_fullsized":"2024-09-21T13:57:33.094320+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:03.085079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:03.085079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:58:14.788719+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":343,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-09-21T13:57:34.998953+0000","last_change":"2024-09-21T13:57:12.704977+0000","last_active":"2024-09-21T13:57:34.998953+0000","last_peered":"2024-09-21T13:57:34.998953+0000","last_clean":"2024-09-21T13:57:34.998953+0000","last_became_active":"2024-09-21T13:57:12.704042+0000","last_became_peered":"2024-09-21T13:57:12.704042+0000","last_unstale":"2024-09-21T13:57:34.998953+0000","last_undegraded":"2024-09-21T13:57:34.998953+0000","last_fullsized":"2024-09-21T13:57:34.998953+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:25.865470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:53:25.865470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"68'11","reported_seq":140,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.085973+0000","last_change":"2024-09-21T13:55:34.251495+0000","last_active":"2024-09-21T13:57:33.085973+0000","last_peered":"2024-09-21T13:57:33.085973+0000","last_clean":"2024-09-21T13:57:33.085973+0000","last_became_active":"2024-09-21T13:55:34.233209+0000","last_became_peered":"2024-09-21T13:55:34.233209+0000","last_unstale":"2024-09-21T13:57:33.085973+0000","last_undegraded":"2024-09-21T13:57:33.085973+0000","last_fullsized":"2024-09-21T13:57:33.085973+0000","mapping_epoch":211,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:55:12.198562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:55:12.198562+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":352,"reported_epoch":250,"state":"active+clean","last_fresh":"2024-09-21T13:57:33.095236+0000","last_change":"2024-09-21T13:54:01.727728+0000","last_active":"2024-09-21T13:57:33.095236+0000","last_peered":"2024-09-21T13:57:33.095236+0000","last_clean":"2024-09-21T13:57:33.095236+0000","last_became_active":"2024-09-21T13:52:04.322880+0000","last_became_peered":"2024-09-21T13:52:04.322880+0000","last_unstale":"2024-09-21T13:57:33.095236+0000","last_undegraded":"2024-09-21T13:57:33.095236+0000","last_fullsized":"2024-09-21T13:57:33.095236+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:01.727680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:01.727680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:57:40.634369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":117243928,"num_objects":1790,"num_object_clones":0,"num_object_copies":5370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":200,"num_objects_misplaced":171,"num_objects_unfound":0,"num_objects_dirty":1790,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":251244,"num_write_kb":8097025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26595,"num_bytes_recovered":1742929920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2306744320,"data_stored":2306736176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74379,"ondisk_log_size":74379,"up":48,"acting":46,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80,"ondisk_log_size":80,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":686,"num_read_kb":443,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":1034,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":454656,"data_stored":12140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":398,"ondisk_log_size":398,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":5275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":509,"num_read_kb":504,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234092,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":329728,"kb_used_data":50028,"kb_used_omap":1,"kb_used_meta":279678,"kb_avail":93415424,"statfs":{"total":95995035648,"available":95657394176,"internally_reserved":0,"allocated":51228672,"data_stored":50886061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":286390730},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":201863463025,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724664,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":421304,"kb_used_data":93028,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":93323848,"statfs":{"total":95995035648,"available":95563620352,"internally_reserved":0,"allocated":95260672,"data_stored":94925870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":336132549},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920894,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":363640,"kb_used_data":71464,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":93381512,"statfs":{"total":95995035648,"available":95622668288,"internally_reserved":0,"allocated":73179136,"data_stored":72831434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":299170245},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":240,"seq":1030792151046,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1391952,"kb_used_data":1322172,"kb_used_omap":4,"kb_used_meta":69755,"kb_avail":92353200,"statfs":{"total":95995035648,"available":94569676800,"internally_reserved":0,"allocated":1353904128,"data_stored":1353545148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4443,"internal_metadata":71429797},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":130,"seq":558345748548,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":466236,"kb_used_data":868,"kb_used_omap":4,"kb_used_meta":465339,"kb_avail":93278916,"statfs":{"total":95995035648,"available":95517609984,"internally_reserved":0,"allocated":888832,"data_stored":561960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4441,"internal_metadata":476507815},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574992,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":409632,"kb_used_data":71628,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":93335520,"statfs":{"total":95995035648,"available":95575572480,"internally_reserved":0,"allocated":73347072,"data_stored":72971726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":346094021},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,3,0,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738518,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":437776,"kb_used_data":49788,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":93307376,"statfs":{"total":95995035648,"available":95546753024,"internally_reserved":0,"allocated":50982912,"data_stored":50623995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":397277637},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49020928,"data_stored":49020928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49020928,"data_stored":49020928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1341849600,"data_stored":1341849600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":49807360,"data_stored":49807360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":76550144,"data_stored":76546072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":691929088,"data_stored":691929088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":48566272,"data_stored":48562200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T13:57:40.838 INFO:tasks.thrashosds.thrasher:Setting 8.b to [3, 0, 4] 2024-09-21T13:57:40.838 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '8.b', '3', '0', '4'] 2024-09-21T13:57:40.838 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd pg-upmap 8.b 3 0 4 2024-09-21T13:57:40.874 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:57:40.915 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:57:40.938 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:40.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:40.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:57:41.040 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:57:41.063 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:41.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:41.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:57:41.165 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:57:41.188 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:41.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:41.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:57:41.290 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:41.311 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:41.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:41.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:41.334 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:41 smithi049 ceph-mon[40489]: pgmap v426: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 144 active+clean; 583 KiB data, 3.3 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 933 op/s 2024-09-21T13:57:41.334 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:41 smithi049 ceph-mon[40489]: 5.1e scrub starts 2024-09-21T13:57:41.334 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:41 smithi049 ceph-mon[40489]: 5.1e scrub ok 2024-09-21T13:57:41.335 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:41 smithi049 ceph-mon[30080]: pgmap v426: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 144 active+clean; 583 KiB data, 3.3 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 933 op/s 2024-09-21T13:57:41.335 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:41 smithi049 ceph-mon[30080]: 5.1e scrub starts 2024-09-21T13:57:41.335 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:41 smithi049 ceph-mon[30080]: 5.1e scrub ok 2024-09-21T13:57:41.413 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:41.436 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:41.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:41.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:41.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:41 smithi187 ceph-mon[33032]: pgmap v426: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 144 active+clean; 583 KiB data, 3.3 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 933 op/s 2024-09-21T13:57:41.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:41 smithi187 ceph-mon[33032]: 5.1e scrub starts 2024-09-21T13:57:41.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:41 smithi187 ceph-mon[33032]: 5.1e scrub ok 2024-09-21T13:57:41.538 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:41.559 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:41.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:41.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:41.661 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:41.685 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:41.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:41.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:41.788 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:41.810 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:41.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:41.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:41.812 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:57:41.812 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:57:41.812 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:57:41.812 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:57:41.814 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:57:41.815 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:57:41.831 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:57:41.832 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:57:41.849 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:57:41.849 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:57:41.866 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:57:41.867 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:57:41.883 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:57:41.884 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:57:41.900 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:57:41.901 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:57:41.911 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:41.912 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:57:41.939 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:41.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:41.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:42.040 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:57:42.062 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:42.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:42.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:57:42.109 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-09-21T13:57:42.165 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:57:42.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:42.187 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:42.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:57:42.288 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:42.310 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:42.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:42.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:42.412 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:42.433 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:42.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:42.434 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:42.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:42 smithi187 ceph-mon[33032]: 5.d scrub starts 2024-09-21T13:57:42.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:42 smithi187 ceph-mon[33032]: 5.d scrub ok 2024-09-21T13:57:42.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:42 smithi187 ceph-mon[33032]: 5.19 scrub starts 2024-09-21T13:57:42.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:42 smithi187 ceph-mon[33032]: 5.19 scrub ok 2024-09-21T13:57:42.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:42 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/887698732' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:57:42.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:42 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:57:42.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:42 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1885950846' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:57:42.451 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:42 smithi049 ceph-mon[40489]: 5.d scrub starts 2024-09-21T13:57:42.451 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:42 smithi049 ceph-mon[40489]: 5.d scrub ok 2024-09-21T13:57:42.451 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:42 smithi049 ceph-mon[40489]: 5.19 scrub starts 2024-09-21T13:57:42.451 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:42 smithi049 ceph-mon[40489]: 5.19 scrub ok 2024-09-21T13:57:42.451 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:42 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/887698732' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:57:42.451 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:42 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:57:42.451 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:42 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1885950846' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:57:42.452 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:42 smithi049 ceph-mon[30080]: 5.d scrub starts 2024-09-21T13:57:42.452 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:42 smithi049 ceph-mon[30080]: 5.d scrub ok 2024-09-21T13:57:42.452 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:42 smithi049 ceph-mon[30080]: 5.19 scrub starts 2024-09-21T13:57:42.452 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:42 smithi049 ceph-mon[30080]: 5.19 scrub ok 2024-09-21T13:57:42.452 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:42 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/887698732' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:57:42.452 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:42 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:57:42.452 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:42 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1885950846' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:57:42.535 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:57:42.558 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:42.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:42.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:57:42.660 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:42.683 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:42.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:42.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:42.785 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:42.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:42.807 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:42.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:42.908 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:57:42.929 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:42.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:42.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:57:43.031 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:57:43.054 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:43.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:43.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:57:43.156 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:43.178 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:43.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:43.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:43.280 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:43.303 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:43.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:43.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:43.405 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:57:43.428 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:43.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:43.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:57:43.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:43 smithi187 ceph-mon[33032]: pgmap v428: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 144 active+clean; 583 KiB data, 3.3 GiB used, 623 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 631 op/s 2024-09-21T13:57:43.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:43 smithi187 ceph-mon[33032]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:57:43.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:43 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T13:57:43.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:43 smithi187 ceph-mon[33032]: osdmap e253: 8 total, 8 up, 7 in 2024-09-21T13:57:43.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:43 smithi187 ceph-mon[33032]: 3.18 scrub starts 2024-09-21T13:57:43.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:43 smithi049 ceph-mon[40489]: pgmap v428: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 144 active+clean; 583 KiB data, 3.3 GiB used, 623 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 631 op/s 2024-09-21T13:57:43.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:43 smithi049 ceph-mon[40489]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:57:43.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:43 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T13:57:43.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:43 smithi049 ceph-mon[40489]: osdmap e253: 8 total, 8 up, 7 in 2024-09-21T13:57:43.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:43 smithi049 ceph-mon[40489]: 3.18 scrub starts 2024-09-21T13:57:43.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:43 smithi049 ceph-mon[30080]: pgmap v428: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 144 active+clean; 583 KiB data, 3.3 GiB used, 623 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 631 op/s 2024-09-21T13:57:43.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:43 smithi049 ceph-mon[30080]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:57:43.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:43 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T13:57:43.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:43 smithi049 ceph-mon[30080]: osdmap e253: 8 total, 8 up, 7 in 2024-09-21T13:57:43.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:43 smithi049 ceph-mon[30080]: 3.18 scrub starts 2024-09-21T13:57:43.530 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:43.553 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:43.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:43.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:43.585 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:57:43.612 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:57:43.655 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:43.679 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:43.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:43.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:43.781 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:43.803 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:43.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:43.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:43.905 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:43.930 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:43.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:43.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:44.031 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:57:44.058 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:44.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:44.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:57:44.160 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:57:44.186 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:44.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:44.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:57:44.288 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:44.314 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:44.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:44.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:44.405 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:44 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:57:44.406 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:44 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:57:44.416 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:44.442 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:44.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:44.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:44.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:44 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:57:44.544 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:44.570 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:44.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:44.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:44.672 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:57:44.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:44.703 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:44.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:57:44.804 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:44.807 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T13:57:44.830 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:44.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:44.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:44.932 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:57:44.966 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:44.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:44.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:57:44.988 INFO:teuthology.orchestra.run.smithi049.stderr:set 8.b pg_upmap mapping to [3,0,4] 2024-09-21T13:57:45.000 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T13:57:45.001 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-09-21T13:57:45.001 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-09-21T13:57:45.001 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-09-21T13:57:45.001 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.001 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232022 8.f 8.d7b7a40f (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T13:57:45.001 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.700289+0000", 2024-09-21T13:57:45.001 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 33.297920148999999, 2024-09-21T13:57:45.001 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97395471300000003, 2024-09-21T13:57:45.001 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:45.001 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:45.001 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T13:57:45.001 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:57:45.002 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:57:45.002 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:57:45.002 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232022 2024-09-21T13:57:45.002 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.002 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:45.002 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.002 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:45.002 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700289+0000", 2024-09-21T13:57:45.002 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:45.002 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.002 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.002 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:45.002 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700289+0000", 2024-09-21T13:57:45.002 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:45.003 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.003 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.003 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:45.003 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700291+0000", 2024-09-21T13:57:45.003 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5850000000000001e-06 2024-09-21T13:57:45.003 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.003 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.003 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:45.003 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700296+0000", 2024-09-21T13:57:45.003 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9749999999999998e-06 2024-09-21T13:57:45.003 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.003 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.003 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:45.003 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700306+0000", 2024-09-21T13:57:45.003 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9310000000000003e-06 2024-09-21T13:57:45.004 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.004 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.004 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:45.004 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700317+0000", 2024-09-21T13:57:45.004 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1199000000000001e-05 2024-09-21T13:57:45.004 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.004 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.004 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:45.004 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.711626+0000", 2024-09-21T13:57:45.004 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.011309009 2024-09-21T13:57:45.004 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.004 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.004 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:57:45.004 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.711637+0000", 2024-09-21T13:57:45.005 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1182000000000001e-05 2024-09-21T13:57:45.005 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.005 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.005 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:45.005 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674230+0000", 2024-09-21T13:57:45.005 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.962593535 2024-09-21T13:57:45.005 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.005 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.005 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:45.005 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674244+0000", 2024-09-21T13:57:45.005 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3297e-05 2024-09-21T13:57:45.005 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:45.005 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:45.005 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:45.005 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.006 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.006 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232025 8.f 8.4e5a668f (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T13:57:45.006 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.700329+0000", 2024-09-21T13:57:45.006 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 33.297880351000003, 2024-09-21T13:57:45.006 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97395089300000004, 2024-09-21T13:57:45.006 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:45.006 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:45.006 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T13:57:45.006 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:57:45.006 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:57:45.006 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:57:45.006 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232025 2024-09-21T13:57:45.006 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.006 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:45.007 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.007 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:45.007 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700329+0000", 2024-09-21T13:57:45.007 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:45.007 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.007 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.007 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:45.007 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700329+0000", 2024-09-21T13:57:45.007 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:45.007 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.007 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.007 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:45.007 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700329+0000", 2024-09-21T13:57:45.007 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7100000000000002e-07 2024-09-21T13:57:45.007 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.008 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.008 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:45.008 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700332+0000", 2024-09-21T13:57:45.008 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6589999999999999e-06 2024-09-21T13:57:45.008 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.008 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.008 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:45.008 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700339+0000", 2024-09-21T13:57:45.008 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8940000000000004e-06 2024-09-21T13:57:45.008 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.008 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.008 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:45.008 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700345+0000", 2024-09-21T13:57:45.008 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6219999999999997e-06 2024-09-21T13:57:45.009 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.009 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.009 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:45.009 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.711662+0000", 2024-09-21T13:57:45.009 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.011317522 2024-09-21T13:57:45.009 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.009 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.009 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:57:45.009 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.711669+0000", 2024-09-21T13:57:45.009 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2570000000000001e-06 2024-09-21T13:57:45.009 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.009 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.009 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:45.009 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674270+0000", 2024-09-21T13:57:45.009 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.96260044300000003 2024-09-21T13:57:45.010 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.010 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.010 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:45.010 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674280+0000", 2024-09-21T13:57:45.010 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0025e-05 2024-09-21T13:57:45.010 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:45.010 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:45.010 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:45.010 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.010 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.010 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232033 8.f 8.d949c02f (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T13:57:45.010 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.700352+0000", 2024-09-21T13:57:45.010 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 33.297856951, 2024-09-21T13:57:45.010 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97396006099999999, 2024-09-21T13:57:45.011 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:45.011 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:45.011 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T13:57:45.011 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:57:45.011 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:57:45.011 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:57:45.011 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232033 2024-09-21T13:57:45.011 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.011 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:45.011 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.011 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:45.011 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700352+0000", 2024-09-21T13:57:45.011 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:45.011 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.012 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.012 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:45.012 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700352+0000", 2024-09-21T13:57:45.012 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:45.012 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.012 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.012 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:45.012 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700353+0000", 2024-09-21T13:57:45.012 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.89e-07 2024-09-21T13:57:45.012 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.012 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.012 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:45.012 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700355+0000", 2024-09-21T13:57:45.012 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1840000000000002e-06 2024-09-21T13:57:45.012 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.013 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.013 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:45.013 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700364+0000", 2024-09-21T13:57:45.013 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4919999999999999e-06 2024-09-21T13:57:45.013 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.013 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.013 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:45.013 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700372+0000", 2024-09-21T13:57:45.013 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9659999999999996e-06 2024-09-21T13:57:45.013 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.013 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.013 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:45.013 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.711693+0000", 2024-09-21T13:57:45.013 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.011321059 2024-09-21T13:57:45.013 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.014 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.014 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:57:45.014 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.711701+0000", 2024-09-21T13:57:45.014 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3640000000000001e-06 2024-09-21T13:57:45.014 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.014 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.014 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:45.014 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674303+0000", 2024-09-21T13:57:45.014 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.96260253100000004 2024-09-21T13:57:45.014 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.014 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.015 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:45.015 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674312+0000", 2024-09-21T13:57:45.015 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0759999999999992e-06 2024-09-21T13:57:45.015 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:45.015 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:45.016 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:45.016 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.016 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.016 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232051 8.0 8.85c8f300 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T13:57:45.016 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.700380+0000", 2024-09-21T13:57:45.016 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 33.297829247999999, 2024-09-21T13:57:45.016 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97451152799999996, 2024-09-21T13:57:45.016 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:45.016 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:45.016 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T13:57:45.016 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:57:45.016 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:57:45.016 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:57:45.016 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232051 2024-09-21T13:57:45.016 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.017 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:45.017 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.017 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:45.017 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700380+0000", 2024-09-21T13:57:45.017 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:45.017 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.017 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.017 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:45.017 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700380+0000", 2024-09-21T13:57:45.017 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:45.017 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.017 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.017 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:45.017 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700380+0000", 2024-09-21T13:57:45.018 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1699999999999999e-07 2024-09-21T13:57:45.018 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.018 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.018 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:45.018 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700383+0000", 2024-09-21T13:57:45.018 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1600000000000001e-06 2024-09-21T13:57:45.018 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.018 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.018 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:45.018 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700389+0000", 2024-09-21T13:57:45.018 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6080000000000001e-06 2024-09-21T13:57:45.018 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.018 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.018 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:45.019 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700394+0000", 2024-09-21T13:57:45.019 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9810000000000003e-06 2024-09-21T13:57:45.019 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.019 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.019 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:45.019 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.715050+0000", 2024-09-21T13:57:45.019 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.014656269 2024-09-21T13:57:45.019 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.019 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.019 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:57:45.019 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.715056+0000", 2024-09-21T13:57:45.019 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0129999999999999e-06 2024-09-21T13:57:45.019 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.019 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.019 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:45.020 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674876+0000", 2024-09-21T13:57:45.020 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.95981962700000001 2024-09-21T13:57:45.020 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.020 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.020 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:45.020 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674892+0000", 2024-09-21T13:57:45.020 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5452999999999999e-05 2024-09-21T13:57:45.020 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:45.020 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:45.020 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:45.020 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.020 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.020 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232059 8.f 8.b757f3bf (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T13:57:45.020 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.889250+0000", 2024-09-21T13:57:45.021 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 33.108959435000003, 2024-09-21T13:57:45.021 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.78509475900000003, 2024-09-21T13:57:45.021 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:45.021 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:45.021 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T13:57:45.021 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:57:45.021 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:57:45.021 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:57:45.021 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232059 2024-09-21T13:57:45.021 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.021 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:45.021 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.021 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:45.022 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889250+0000", 2024-09-21T13:57:45.022 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:45.022 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.022 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.022 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:45.022 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889250+0000", 2024-09-21T13:57:45.022 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:45.022 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.022 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.022 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:45.022 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889252+0000", 2024-09-21T13:57:45.022 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9130000000000001e-06 2024-09-21T13:57:45.022 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.022 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.022 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:45.023 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889258+0000", 2024-09-21T13:57:45.023 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7410000000000003e-06 2024-09-21T13:57:45.023 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.023 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.023 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:45.023 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889360+0000", 2024-09-21T13:57:45.023 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000101382 2024-09-21T13:57:45.023 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.023 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.023 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:45.023 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889391+0000", 2024-09-21T13:57:45.023 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1032999999999999e-05 2024-09-21T13:57:45.023 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.023 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.024 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:45.024 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889500+0000", 2024-09-21T13:57:45.024 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000109194 2024-09-21T13:57:45.024 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.024 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.024 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:57:45.024 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889512+0000", 2024-09-21T13:57:45.024 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2368000000000001e-05 2024-09-21T13:57:45.024 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.024 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.024 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:45.024 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674335+0000", 2024-09-21T13:57:45.024 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.78482304000000003 2024-09-21T13:57:45.024 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.024 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.025 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:45.025 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674345+0000", 2024-09-21T13:57:45.025 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0880000000000001e-06 2024-09-21T13:57:45.025 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:45.025 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:45.025 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:45.025 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.025 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.025 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232060 8.b 8.6c14963b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T13:57:45.025 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.889442+0000", 2024-09-21T13:57:45.025 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 33.108767247000003, 2024-09-21T13:57:45.025 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.78518956100000004, 2024-09-21T13:57:45.025 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:45.025 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:45.026 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T13:57:45.026 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:57:45.026 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:57:45.026 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:57:45.026 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232060 2024-09-21T13:57:45.026 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.026 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:45.026 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.026 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:45.026 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889442+0000", 2024-09-21T13:57:45.026 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:45.026 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.027 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.027 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:45.027 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889442+0000", 2024-09-21T13:57:45.027 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:45.027 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.027 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.027 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:45.027 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889443+0000", 2024-09-21T13:57:45.027 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9100000000000003e-07 2024-09-21T13:57:45.027 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.027 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.027 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:45.027 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889447+0000", 2024-09-21T13:57:45.027 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4939999999999997e-06 2024-09-21T13:57:45.027 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.028 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.028 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:45.028 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889457+0000", 2024-09-21T13:57:45.028 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0145e-05 2024-09-21T13:57:45.028 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.028 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.028 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:45.028 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889465+0000", 2024-09-21T13:57:45.028 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2749999999999998e-06 2024-09-21T13:57:45.028 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.028 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.028 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:45.028 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889550+0000", 2024-09-21T13:57:45.028 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5000999999999995e-05 2024-09-21T13:57:45.028 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.029 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.029 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:57:45.029 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889561+0000", 2024-09-21T13:57:45.029 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.129e-05 2024-09-21T13:57:45.029 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.029 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.029 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:45.029 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674619+0000", 2024-09-21T13:57:45.029 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.785057703 2024-09-21T13:57:45.029 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.029 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.029 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:45.029 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674632+0000", 2024-09-21T13:57:45.030 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2962000000000001e-05 2024-09-21T13:57:45.030 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:45.030 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:45.030 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:45.030 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.030 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.030 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232068 8.f 8.d5b3687f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T13:57:45.030 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.894747+0000", 2024-09-21T13:57:45.030 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 33.103462559, 2024-09-21T13:57:45.030 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.77963143999999995, 2024-09-21T13:57:45.030 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:45.030 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:45.030 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T13:57:45.031 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:57:45.031 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:57:45.031 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:57:45.031 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232068 2024-09-21T13:57:45.031 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.031 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:45.031 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.031 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:45.031 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.894747+0000", 2024-09-21T13:57:45.031 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:45.031 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.031 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.031 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:45.031 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.894747+0000", 2024-09-21T13:57:45.032 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:45.032 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.032 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.032 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:45.032 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.894749+0000", 2024-09-21T13:57:45.032 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4849999999999999e-06 2024-09-21T13:57:45.032 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.032 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.032 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:45.032 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.894761+0000", 2024-09-21T13:57:45.032 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1809e-05 2024-09-21T13:57:45.032 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.032 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.032 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:45.033 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.894780+0000", 2024-09-21T13:57:45.033 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8593999999999998e-05 2024-09-21T13:57:45.033 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.033 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.033 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:45.033 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.894789+0000", 2024-09-21T13:57:45.033 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2839999999999995e-06 2024-09-21T13:57:45.033 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.033 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.033 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:45.033 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.894865+0000", 2024-09-21T13:57:45.033 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6150999999999996e-05 2024-09-21T13:57:45.033 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.033 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.034 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:57:45.034 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.894874+0000", 2024-09-21T13:57:45.034 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2450000000000004e-06 2024-09-21T13:57:45.034 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.034 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.034 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:45.034 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674368+0000", 2024-09-21T13:57:45.034 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.77949425999999999 2024-09-21T13:57:45.034 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.034 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.034 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:45.034 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674378+0000", 2024-09-21T13:57:45.034 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6120000000000005e-06 2024-09-21T13:57:45.034 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:45.035 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:45.035 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:45.035 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.035 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.035 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232074 8.0 8.57810 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T13:57:45.035 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.901181+0000", 2024-09-21T13:57:45.035 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 33.097028045999998, 2024-09-21T13:57:45.035 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.77382758200000001, 2024-09-21T13:57:45.035 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:45.035 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:45.035 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T13:57:45.035 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:57:45.035 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:57:45.035 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:57:45.035 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232074 2024-09-21T13:57:45.036 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.036 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:45.036 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.036 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:45.036 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.901181+0000", 2024-09-21T13:57:45.036 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:45.036 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.036 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.036 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:45.036 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.901181+0000", 2024-09-21T13:57:45.036 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:45.036 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.036 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.036 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:45.037 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.901184+0000", 2024-09-21T13:57:45.037 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.8660000000000002e-06 2024-09-21T13:57:45.037 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.037 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.037 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:45.037 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.901193+0000", 2024-09-21T13:57:45.037 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1549999999999996e-06 2024-09-21T13:57:45.037 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.037 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.037 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:45.037 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.901215+0000", 2024-09-21T13:57:45.037 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2272e-05 2024-09-21T13:57:45.037 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.037 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.037 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:45.038 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.901227+0000", 2024-09-21T13:57:45.038 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1116e-05 2024-09-21T13:57:45.038 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.038 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.038 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:45.038 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.901351+0000", 2024-09-21T13:57:45.038 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000124172 2024-09-21T13:57:45.038 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.038 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.038 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:57:45.038 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.901368+0000", 2024-09-21T13:57:45.038 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7008999999999999e-05 2024-09-21T13:57:45.038 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.038 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.039 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:45.039 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674970+0000", 2024-09-21T13:57:45.039 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.77360225199999999 2024-09-21T13:57:45.039 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.039 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.039 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:45.039 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.675009+0000", 2024-09-21T13:57:45.039 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8739999999999998e-05 2024-09-21T13:57:45.039 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:45.039 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:45.039 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:45.039 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.039 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.039 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232078 8.b 8.7f49789b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T13:57:45.039 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.904662+0000", 2024-09-21T13:57:45.040 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 33.093547641000001, 2024-09-21T13:57:45.040 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.77001060399999999, 2024-09-21T13:57:45.040 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:45.040 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:45.040 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T13:57:45.040 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:57:45.040 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:57:45.040 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:57:45.040 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232078 2024-09-21T13:57:45.040 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.040 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:45.040 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.040 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:45.040 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904662+0000", 2024-09-21T13:57:45.040 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:45.041 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.041 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.041 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:45.041 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904662+0000", 2024-09-21T13:57:45.041 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:45.041 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.041 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.041 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:45.041 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904663+0000", 2024-09-21T13:57:45.041 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.548e-06 2024-09-21T13:57:45.041 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.041 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.041 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:45.041 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904669+0000", 2024-09-21T13:57:45.041 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9619999999999998e-06 2024-09-21T13:57:45.042 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.042 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.042 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:45.042 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904684+0000", 2024-09-21T13:57:45.042 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4718e-05 2024-09-21T13:57:45.042 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.042 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.042 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:45.042 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904691+0000", 2024-09-21T13:57:45.042 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.861e-06 2024-09-21T13:57:45.042 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.042 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.043 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:45.043 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904750+0000", 2024-09-21T13:57:45.043 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9533000000000001e-05 2024-09-21T13:57:45.043 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.043 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.043 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:57:45.043 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904757+0000", 2024-09-21T13:57:45.044 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0369999999999997e-06 2024-09-21T13:57:45.044 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.044 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.044 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:45.044 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674661+0000", 2024-09-21T13:57:45.044 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.769903754 2024-09-21T13:57:45.044 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.044 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.044 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:45.044 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674672+0000", 2024-09-21T13:57:45.045 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1191e-05 2024-09-21T13:57:45.045 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:45.045 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:45.045 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:45.045 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.045 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.045 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232079 8.b 8.5c3e90b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T13:57:45.045 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.904710+0000", 2024-09-21T13:57:45.045 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 33.093499285999997, 2024-09-21T13:57:45.045 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.76999876, 2024-09-21T13:57:45.045 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:45.045 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:45.045 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T13:57:45.045 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:57:45.046 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:57:45.046 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:57:45.046 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232079 2024-09-21T13:57:45.046 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.046 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:45.046 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.046 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:45.046 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904710+0000", 2024-09-21T13:57:45.046 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:45.046 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.046 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.046 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:45.046 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904710+0000", 2024-09-21T13:57:45.046 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:45.047 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.047 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.047 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:45.047 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904711+0000", 2024-09-21T13:57:45.047 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8199999999999999e-07 2024-09-21T13:57:45.047 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.047 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.047 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:45.047 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904714+0000", 2024-09-21T13:57:45.047 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9079999999999999e-06 2024-09-21T13:57:45.047 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.047 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.047 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:45.047 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904726+0000", 2024-09-21T13:57:45.048 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2514000000000001e-05 2024-09-21T13:57:45.048 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.048 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.048 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:45.048 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904733+0000", 2024-09-21T13:57:45.048 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2690000000000002e-06 2024-09-21T13:57:45.048 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.048 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.048 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:45.048 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904784+0000", 2024-09-21T13:57:45.048 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1007000000000001e-05 2024-09-21T13:57:45.048 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.048 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.048 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:57:45.048 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904794+0000", 2024-09-21T13:57:45.049 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7419999999999992e-06 2024-09-21T13:57:45.049 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.049 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.049 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:45.049 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674699+0000", 2024-09-21T13:57:45.049 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.76990482900000001 2024-09-21T13:57:45.049 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.049 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.049 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:45.049 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674709+0000", 2024-09-21T13:57:45.049 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8090000000000004e-06 2024-09-21T13:57:45.049 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:45.049 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:45.049 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:45.050 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.050 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.050 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:245292 8.f 8:f1e08d30:::benchmark_data_smithi049_109311_object118773:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e248)", 2024-09-21T13:57:45.050 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:28.780036+0000", 2024-09-21T13:57:45.050 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 16.218173562, 2024-09-21T13:57:45.050 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.057689470999999999, 2024-09-21T13:57:45.050 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:45.050 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:45.050 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:45.050 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:57:45.050 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:57:45.050 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:57:45.050 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 245292 2024-09-21T13:57:45.050 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.051 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:45.051 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.051 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:45.051 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.780036+0000", 2024-09-21T13:57:45.051 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:45.051 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.051 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.051 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:45.051 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.780036+0000", 2024-09-21T13:57:45.051 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:45.051 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.051 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.051 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:45.051 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.780037+0000", 2024-09-21T13:57:45.052 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7570000000000001e-06 2024-09-21T13:57:45.052 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.052 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.052 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:45.052 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.780044+0000", 2024-09-21T13:57:45.052 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.263e-06 2024-09-21T13:57:45.052 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.052 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.052 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:45.052 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.780056+0000", 2024-09-21T13:57:45.052 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2503999999999999e-05 2024-09-21T13:57:45.052 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.052 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.052 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:45.052 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.780064+0000", 2024-09-21T13:57:45.053 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9319999999999996e-06 2024-09-21T13:57:45.053 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.053 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.053 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:45.053 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.780406+0000", 2024-09-21T13:57:45.053 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00034155999999999998 2024-09-21T13:57:45.053 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.053 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.053 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:57:45.053 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.780525+0000", 2024-09-21T13:57:45.053 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000119339 2024-09-21T13:57:45.053 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.053 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.053 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,2", 2024-09-21T13:57:45.053 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.780638+0000", 2024-09-21T13:57:45.054 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000112835 2024-09-21T13:57:45.054 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.054 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.054 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:45.054 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.788466+0000", 2024-09-21T13:57:45.054 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0078284519999999996 2024-09-21T13:57:45.054 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.054 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.054 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:57:45.054 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.788484+0000", 2024-09-21T13:57:45.054 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8054999999999999e-05 2024-09-21T13:57:45.054 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.054 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.054 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:45.055 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.837564+0000", 2024-09-21T13:57:45.055 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.049079629999999999 2024-09-21T13:57:45.055 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.055 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.055 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:57:45.055 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.837676+0000", 2024-09-21T13:57:45.055 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000111838 2024-09-21T13:57:45.055 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.055 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.055 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:45.055 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.837725+0000", 2024-09-21T13:57:45.055 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9305999999999999e-05 2024-09-21T13:57:45.055 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:45.055 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:45.055 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:45.056 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.056 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.056 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:245296 8.f 8:f3c50252:::benchmark_data_smithi049_109311_object118777:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e248)", 2024-09-21T13:57:45.056 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:28.782162+0000", 2024-09-21T13:57:45.056 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 16.216047296999999, 2024-09-21T13:57:45.056 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.055830284000000001, 2024-09-21T13:57:45.056 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:45.056 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:45.056 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:45.056 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:57:45.056 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:57:45.056 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:57:45.056 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 245296 2024-09-21T13:57:45.056 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.056 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:45.057 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.057 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:45.057 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.782162+0000", 2024-09-21T13:57:45.057 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:45.057 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.057 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.057 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:45.057 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.782162+0000", 2024-09-21T13:57:45.057 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:45.058 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.058 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.058 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:45.058 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.782163+0000", 2024-09-21T13:57:45.058 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.401e-06 2024-09-21T13:57:45.058 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.058 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.058 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:45.058 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.782169+0000", 2024-09-21T13:57:45.058 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.609e-06 2024-09-21T13:57:45.058 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.058 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.058 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:45.059 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.782182+0000", 2024-09-21T13:57:45.059 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2788e-05 2024-09-21T13:57:45.059 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.059 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.059 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:45.059 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.782190+0000", 2024-09-21T13:57:45.059 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1710000000000002e-06 2024-09-21T13:57:45.059 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.059 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.059 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:45.059 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.783313+0000", 2024-09-21T13:57:45.059 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0011229619999999999 2024-09-21T13:57:45.059 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.059 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.060 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:57:45.060 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.783415+0000", 2024-09-21T13:57:45.060 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010203499999999999 2024-09-21T13:57:45.060 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.060 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.060 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,2", 2024-09-21T13:57:45.060 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.783529+0000", 2024-09-21T13:57:45.060 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000114549 2024-09-21T13:57:45.060 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.060 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.060 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:57:45.060 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.788495+0000", 2024-09-21T13:57:45.060 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0049655649999999999 2024-09-21T13:57:45.060 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.061 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.061 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:45.061 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.837764+0000", 2024-09-21T13:57:45.061 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.049269391000000003 2024-09-21T13:57:45.061 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.061 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.061 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:45.061 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.837882+0000", 2024-09-21T13:57:45.061 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000117113 2024-09-21T13:57:45.061 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.061 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.061 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:57:45.061 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.837956+0000", 2024-09-21T13:57:45.061 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4977999999999999e-05 2024-09-21T13:57:45.062 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.062 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.062 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:45.062 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.837992+0000", 2024-09-21T13:57:45.062 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5722000000000001e-05 2024-09-21T13:57:45.062 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:45.062 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:45.062 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:45.062 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.062 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.062 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:245298 8.f 8:f2c69ce1:::benchmark_data_smithi049_109311_object118779:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e248)", 2024-09-21T13:57:45.062 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:28.783169+0000", 2024-09-21T13:57:45.062 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 16.215039875999999, 2024-09-21T13:57:45.062 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.054981025000000003, 2024-09-21T13:57:45.063 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:45.063 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:45.063 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:45.063 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:57:45.063 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:57:45.063 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:57:45.063 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 245298 2024-09-21T13:57:45.063 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.063 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:45.063 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.063 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:45.063 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.783169+0000", 2024-09-21T13:57:45.063 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:45.063 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.063 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.064 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:45.064 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.783169+0000", 2024-09-21T13:57:45.064 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:45.064 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.064 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.064 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:45.064 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.783170+0000", 2024-09-21T13:57:45.064 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1100000000000004e-07 2024-09-21T13:57:45.064 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.064 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.064 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:45.064 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.783175+0000", 2024-09-21T13:57:45.064 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2560000000000004e-06 2024-09-21T13:57:45.064 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.065 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.065 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:45.065 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.783185+0000", 2024-09-21T13:57:45.065 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0772e-05 2024-09-21T13:57:45.065 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.065 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.065 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:45.065 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.783192+0000", 2024-09-21T13:57:45.065 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0690000000000004e-06 2024-09-21T13:57:45.065 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.065 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.065 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:45.065 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.785911+0000", 2024-09-21T13:57:45.065 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002718448 2024-09-21T13:57:45.066 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.066 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.066 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:57:45.066 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.786021+0000", 2024-09-21T13:57:45.066 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011027399999999999 2024-09-21T13:57:45.066 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.066 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.066 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,2", 2024-09-21T13:57:45.066 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.786133+0000", 2024-09-21T13:57:45.066 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000112002 2024-09-21T13:57:45.066 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.066 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.066 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:57:45.066 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.788500+0000", 2024-09-21T13:57:45.067 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0023672300000000001 2024-09-21T13:57:45.067 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.067 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.067 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:45.067 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.837844+0000", 2024-09-21T13:57:45.067 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.049343698999999998 2024-09-21T13:57:45.067 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.067 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.067 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:45.067 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.838042+0000", 2024-09-21T13:57:45.067 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00019806500000000001 2024-09-21T13:57:45.067 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.067 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.068 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:57:45.068 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.838116+0000", 2024-09-21T13:57:45.068 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3653999999999996e-05 2024-09-21T13:57:45.068 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.068 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.068 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:45.068 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.838150+0000", 2024-09-21T13:57:45.068 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4644999999999998e-05 2024-09-21T13:57:45.068 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:45.068 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:45.068 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:45.068 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.068 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.068 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:245300 8.f 8:fd639d80:::benchmark_data_smithi049_109311_object118781:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e248)", 2024-09-21T13:57:45.069 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:28.783949+0000", 2024-09-21T13:57:45.069 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 16.214259943999998, 2024-09-21T13:57:45.069 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.065086589, 2024-09-21T13:57:45.069 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:45.069 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:45.069 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:45.069 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:57:45.069 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:57:45.069 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:57:45.069 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 245300 2024-09-21T13:57:45.069 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.069 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:45.069 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.069 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:45.070 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.783949+0000", 2024-09-21T13:57:45.070 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:45.070 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.070 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.070 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:45.070 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.783949+0000", 2024-09-21T13:57:45.070 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:45.070 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.070 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.070 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:45.070 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.783950+0000", 2024-09-21T13:57:45.070 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7899999999999997e-07 2024-09-21T13:57:45.070 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.070 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.071 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:45.071 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.783964+0000", 2024-09-21T13:57:45.071 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.358e-05 2024-09-21T13:57:45.071 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.071 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.071 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:45.071 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.783975+0000", 2024-09-21T13:57:45.071 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1446e-05 2024-09-21T13:57:45.071 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.071 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.071 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:45.071 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.783982+0000", 2024-09-21T13:57:45.071 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:45.072 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0450000000000003e-06 2024-09-21T13:57:45.073 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.073 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.073 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:45.073 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.788526+0000", 2024-09-21T13:57:45.073 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0045441609999999997 2024-09-21T13:57:45.073 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.073 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.073 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:57:45.073 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.788753+0000", 2024-09-21T13:57:45.073 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00022679300000000001 2024-09-21T13:57:45.073 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.073 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.073 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,2", 2024-09-21T13:57:45.074 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.835636+0000", 2024-09-21T13:57:45.074 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.046883086999999997 2024-09-21T13:57:45.074 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.074 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.074 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:57:45.074 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.837854+0000", 2024-09-21T13:57:45.074 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0022180709999999998 2024-09-21T13:57:45.074 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.074 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.074 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:45.074 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.845509+0000", 2024-09-21T13:57:45.075 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.007654269 2024-09-21T13:57:45.075 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.075 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.075 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:45.075 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.848887+0000", 2024-09-21T13:57:45.075 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0033788939999999999 2024-09-21T13:57:45.075 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.075 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.075 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:57:45.075 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.848989+0000", 2024-09-21T13:57:45.075 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000101328 2024-09-21T13:57:45.075 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.075 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.075 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:45.076 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.849036+0000", 2024-09-21T13:57:45.076 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7135999999999997e-05 2024-09-21T13:57:45.076 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:45.076 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:45.076 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:45.076 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.076 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.076 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.15207.0:245305 8.c e248/241 8:3ff84c5f:::benchmark_data_smithi049_109311_object118786:head v 248'15487, mlcod=248'15485)", 2024-09-21T13:57:45.076 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:28.788457+0000", 2024-09-21T13:57:45.076 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 16.209752648999999, 2024-09-21T13:57:45.076 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.053768046, 2024-09-21T13:57:45.076 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:45.076 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:45.076 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:45.076 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:45.077 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.077 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:45.077 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.788457+0000", 2024-09-21T13:57:45.077 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:45.077 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.077 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.077 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:45.077 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.788457+0000", 2024-09-21T13:57:45.077 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:45.077 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.077 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.077 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:45.077 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.788458+0000", 2024-09-21T13:57:45.077 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.5000000000000001e-07 2024-09-21T13:57:45.077 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.078 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.078 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:45.078 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.788469+0000", 2024-09-21T13:57:45.078 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1725e-05 2024-09-21T13:57:45.078 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.078 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.078 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:45.078 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.788480+0000", 2024-09-21T13:57:45.078 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.112e-05 2024-09-21T13:57:45.078 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.078 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.078 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:45.078 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.788487+0000", 2024-09-21T13:57:45.078 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9619999999999996e-06 2024-09-21T13:57:45.079 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.079 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.079 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:45.079 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.788538+0000", 2024-09-21T13:57:45.079 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0436000000000003e-05 2024-09-21T13:57:45.079 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.079 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.079 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:57:45.079 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.788551+0000", 2024-09-21T13:57:45.079 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3390999999999999e-05 2024-09-21T13:57:45.079 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.079 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.079 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:57:45.079 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.842185+0000", 2024-09-21T13:57:45.080 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.053633642000000002 2024-09-21T13:57:45.080 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.080 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.080 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:45.080 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.842225+0000", 2024-09-21T13:57:45.080 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9820000000000002e-05 2024-09-21T13:57:45.080 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:45.080 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:45.080 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:45.080 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.080 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.080 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.15207.0:245308 8.8 e248/244 8:1b187c2c:::benchmark_data_smithi049_109311_object118789:head v 248'15369, mlcod=248'15367)", 2024-09-21T13:57:45.080 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:28.789218+0000", 2024-09-21T13:57:45.080 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 16.208991088000001, 2024-09-21T13:57:45.080 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.050776959000000003, 2024-09-21T13:57:45.081 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:45.081 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:45.081 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:45.081 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:45.081 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.081 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:45.081 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.789218+0000", 2024-09-21T13:57:45.081 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:45.081 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.081 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.081 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:45.081 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.789218+0000", 2024-09-21T13:57:45.081 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:45.081 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.082 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.082 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:45.082 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.789219+0000", 2024-09-21T13:57:45.082 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8599999999999997e-07 2024-09-21T13:57:45.082 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.082 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.082 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:45.082 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.789224+0000", 2024-09-21T13:57:45.082 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.967e-06 2024-09-21T13:57:45.082 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.082 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.082 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:45.082 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.837562+0000", 2024-09-21T13:57:45.082 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.048338550000000001 2024-09-21T13:57:45.082 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.083 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.083 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:45.083 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.837575+0000", 2024-09-21T13:57:45.083 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2228999999999999e-05 2024-09-21T13:57:45.083 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.083 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.083 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:45.083 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.837633+0000", 2024-09-21T13:57:45.083 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8578000000000001e-05 2024-09-21T13:57:45.083 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.083 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.083 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:57:45.083 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.837652+0000", 2024-09-21T13:57:45.083 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8700000000000001e-05 2024-09-21T13:57:45.084 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.084 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.084 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:57:45.084 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.839958+0000", 2024-09-21T13:57:45.084 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0023059489999999998 2024-09-21T13:57:45.084 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.084 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.084 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:45.084 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:28.839995+0000", 2024-09-21T13:57:45.084 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7200000000000003e-05 2024-09-21T13:57:45.084 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:45.084 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:45.084 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:45.084 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.084 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.085 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:249888 8.f 8:f647b81d:::benchmark_data_smithi049_109311_object123369:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e249)", 2024-09-21T13:57:45.085 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:32.836671+0000", 2024-09-21T13:57:45.085 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 12.161538301, 2024-09-21T13:57:45.085 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.043844849999999998, 2024-09-21T13:57:45.085 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:45.085 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:45.085 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:45.085 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:57:45.085 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:57:45.085 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:57:45.085 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 249888 2024-09-21T13:57:45.085 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.085 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:45.085 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.085 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:45.086 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:32.836671+0000", 2024-09-21T13:57:45.086 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:45.086 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.086 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.086 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:45.086 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:32.836671+0000", 2024-09-21T13:57:45.086 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:45.086 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.086 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.086 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:45.086 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:32.836671+0000", 2024-09-21T13:57:45.086 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4700000000000002e-07 2024-09-21T13:57:45.086 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.087 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.087 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:45.087 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:32.836673+0000", 2024-09-21T13:57:45.087 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.069e-06 2024-09-21T13:57:45.087 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.087 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.087 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:45.087 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:32.836679+0000", 2024-09-21T13:57:45.087 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7710000000000003e-06 2024-09-21T13:57:45.087 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.087 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.087 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:45.087 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:32.836684+0000", 2024-09-21T13:57:45.087 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9529999999999998e-06 2024-09-21T13:57:45.088 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.088 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.088 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:45.088 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:32.841622+0000", 2024-09-21T13:57:45.088 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0049379430000000002 2024-09-21T13:57:45.088 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.088 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.088 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:57:45.088 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:32.841723+0000", 2024-09-21T13:57:45.088 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000100997 2024-09-21T13:57:45.088 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.088 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.088 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,2", 2024-09-21T13:57:45.088 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:32.841826+0000", 2024-09-21T13:57:45.088 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000102901 2024-09-21T13:57:45.089 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.089 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.089 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:57:45.089 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:32.846299+0000", 2024-09-21T13:57:45.089 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0044727070000000002 2024-09-21T13:57:45.089 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.089 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.089 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:45.089 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:32.849203+0000", 2024-09-21T13:57:45.089 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0029047729999999998 2024-09-21T13:57:45.089 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.089 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.089 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:45.089 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:32.880155+0000", 2024-09-21T13:57:45.090 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.030951889999999999 2024-09-21T13:57:45.090 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.090 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.090 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:57:45.090 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:32.880372+0000", 2024-09-21T13:57:45.090 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00021634399999999999 2024-09-21T13:57:45.090 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.090 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.090 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:45.090 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:32.880516+0000", 2024-09-21T13:57:45.090 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000144155 2024-09-21T13:57:45.090 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:45.090 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:45.090 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:45.091 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.091 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.091 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:250446 8.f 8:fa338908:::benchmark_data_smithi049_109311_object123927:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-09-21T13:57:45.091 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:33.322919+0000", 2024-09-21T13:57:45.091 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 11.675289963999999, 2024-09-21T13:57:45.091 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.044297269, 2024-09-21T13:57:45.091 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:45.091 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:45.091 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:45.091 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:57:45.091 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:57:45.091 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:57:45.091 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 250446 2024-09-21T13:57:45.091 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.091 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:45.092 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.092 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:45.092 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.322919+0000", 2024-09-21T13:57:45.092 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:45.092 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.092 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.092 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:45.092 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.322919+0000", 2024-09-21T13:57:45.092 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:45.092 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.092 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.092 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:45.092 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.322920+0000", 2024-09-21T13:57:45.092 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2799999999999996e-07 2024-09-21T13:57:45.093 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.093 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.093 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:45.093 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.322923+0000", 2024-09-21T13:57:45.093 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.145e-06 2024-09-21T13:57:45.093 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.093 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.093 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:45.093 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.322931+0000", 2024-09-21T13:57:45.093 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0069999999999997e-06 2024-09-21T13:57:45.093 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.093 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.093 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:45.093 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.322958+0000", 2024-09-21T13:57:45.093 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.7407e-05 2024-09-21T13:57:45.094 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.094 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.094 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:45.094 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.332752+0000", 2024-09-21T13:57:45.094 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0097931849999999994 2024-09-21T13:57:45.094 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.094 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.094 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:57:45.094 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.332860+0000", 2024-09-21T13:57:45.094 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010846800000000001 2024-09-21T13:57:45.094 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.094 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.094 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,2", 2024-09-21T13:57:45.094 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.332971+0000", 2024-09-21T13:57:45.095 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011056 2024-09-21T13:57:45.095 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.095 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.095 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:57:45.095 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.335965+0000", 2024-09-21T13:57:45.095 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002994914 2024-09-21T13:57:45.095 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.095 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.095 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:45.095 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.346372+0000", 2024-09-21T13:57:45.095 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.01040686 2024-09-21T13:57:45.095 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.095 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.095 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:45.096 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.367063+0000", 2024-09-21T13:57:45.096 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.020690930999999999 2024-09-21T13:57:45.096 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.096 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.096 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:57:45.096 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.367168+0000", 2024-09-21T13:57:45.096 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000105031 2024-09-21T13:57:45.096 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.096 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.096 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:45.096 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.367217+0000", 2024-09-21T13:57:45.096 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8232999999999997e-05 2024-09-21T13:57:45.096 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:45.096 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:45.097 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:45.097 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.097 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.097 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:250447 8.f 8:f9d1791f:::benchmark_data_smithi049_109311_object123928:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-09-21T13:57:45.097 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:33.323059+0000", 2024-09-21T13:57:45.097 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 11.675150686, 2024-09-21T13:57:45.097 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.046075090999999999, 2024-09-21T13:57:45.097 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:45.097 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:45.097 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:45.097 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:57:45.097 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:57:45.097 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:57:45.097 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 250447 2024-09-21T13:57:45.097 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.098 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:45.098 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.098 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:45.098 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.323059+0000", 2024-09-21T13:57:45.098 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:45.098 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.098 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.098 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:45.098 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.323059+0000", 2024-09-21T13:57:45.098 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:45.098 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.098 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.098 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:45.098 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.323059+0000", 2024-09-21T13:57:45.099 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1999999999999999e-07 2024-09-21T13:57:45.099 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.099 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.099 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:45.099 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.323063+0000", 2024-09-21T13:57:45.099 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9310000000000001e-06 2024-09-21T13:57:45.099 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.099 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.099 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:45.099 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.323073+0000", 2024-09-21T13:57:45.099 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.011e-05 2024-09-21T13:57:45.099 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.099 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.099 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:45.099 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.323080+0000", 2024-09-21T13:57:45.100 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4099999999999996e-06 2024-09-21T13:57:45.100 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.100 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.100 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:45.100 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.335994+0000", 2024-09-21T13:57:45.100 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.012914079 2024-09-21T13:57:45.100 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.100 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.100 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:57:45.100 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.336102+0000", 2024-09-21T13:57:45.100 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000108439 2024-09-21T13:57:45.100 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.100 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.100 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,2", 2024-09-21T13:57:45.101 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.336209+0000", 2024-09-21T13:57:45.101 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000107001 2024-09-21T13:57:45.101 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.101 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.101 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:57:45.101 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.342295+0000", 2024-09-21T13:57:45.101 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0060853089999999997 2024-09-21T13:57:45.101 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.101 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.101 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:45.101 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.349498+0000", 2024-09-21T13:57:45.101 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0072034029999999997 2024-09-21T13:57:45.101 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.101 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.101 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:45.102 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.368741+0000", 2024-09-21T13:57:45.102 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.019242849999999999 2024-09-21T13:57:45.102 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.102 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.102 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:57:45.102 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.369085+0000", 2024-09-21T13:57:45.102 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00034449099999999999 2024-09-21T13:57:45.102 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.102 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.102 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:45.102 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.369134+0000", 2024-09-21T13:57:45.102 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8347999999999999e-05 2024-09-21T13:57:45.102 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:45.102 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:45.103 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:45.103 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.103 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.103 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:250449 8.f 8:fc93ec6c:::benchmark_data_smithi049_109311_object123930:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-09-21T13:57:45.103 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:33.327886+0000", 2024-09-21T13:57:45.103 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 11.670323482000001, 2024-09-21T13:57:45.103 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.041520184000000002, 2024-09-21T13:57:45.103 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:45.103 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:45.103 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:45.103 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:57:45.103 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:57:45.103 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:57:45.103 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 250449 2024-09-21T13:57:45.103 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.104 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:45.104 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.104 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:45.104 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.327886+0000", 2024-09-21T13:57:45.104 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:45.104 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.104 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.104 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:45.104 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.327886+0000", 2024-09-21T13:57:45.104 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:45.104 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.104 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.104 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:45.104 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.327886+0000", 2024-09-21T13:57:45.104 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5700000000000002e-07 2024-09-21T13:57:45.105 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.105 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.105 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:45.105 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.327890+0000", 2024-09-21T13:57:45.105 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4419999999999998e-06 2024-09-21T13:57:45.105 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.105 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.105 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:45.105 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.327907+0000", 2024-09-21T13:57:45.105 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7652999999999999e-05 2024-09-21T13:57:45.105 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.105 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.105 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:45.105 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.327914+0000", 2024-09-21T13:57:45.106 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5289999999999997e-06 2024-09-21T13:57:45.106 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.106 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.106 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:45.106 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.338414+0000", 2024-09-21T13:57:45.106 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.010499845000000001 2024-09-21T13:57:45.106 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.106 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.106 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:57:45.106 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.338554+0000", 2024-09-21T13:57:45.106 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014023 2024-09-21T13:57:45.106 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.106 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.106 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,2", 2024-09-21T13:57:45.106 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.339062+0000", 2024-09-21T13:57:45.107 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00050783600000000001 2024-09-21T13:57:45.107 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.107 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.107 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:57:45.107 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.344249+0000", 2024-09-21T13:57:45.107 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0051872059999999998 2024-09-21T13:57:45.107 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.107 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.107 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:45.107 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.349535+0000", 2024-09-21T13:57:45.107 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0052860019999999997 2024-09-21T13:57:45.107 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.107 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.107 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:45.107 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.369251+0000", 2024-09-21T13:57:45.108 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.019715766999999999 2024-09-21T13:57:45.108 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.108 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.108 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:57:45.108 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.369368+0000", 2024-09-21T13:57:45.108 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011738700000000001 2024-09-21T13:57:45.108 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:45.108 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:45.108 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:45.108 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:33.369406+0000", 2024-09-21T13:57:45.108 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7629999999999997e-05 2024-09-21T13:57:45.108 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:45.108 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:45.108 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:45.108 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:45.109 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:45.109 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T13:57:45.132 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:45.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:45.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:45.234 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:57:45.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:45 smithi049 ceph-mon[40489]: pgmap v430: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 144 active+clean; 583 KiB data, 3.3 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 224 op/s 2024-09-21T13:57:45.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:45 smithi049 ceph-mon[40489]: osdmap e254: 8 total, 8 up, 7 in 2024-09-21T13:57:45.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:45 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1444762519' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "8.b", "id": [3, 0, 4]} : dispatch 2024-09-21T13:57:45.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:45 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 1 pg degraded (PG_DEGRADED) 2024-09-21T13:57:45.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:45 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1444762519' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "8.b", "id": [3, 0, 4]}]': finished 2024-09-21T13:57:45.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:45 smithi049 ceph-mon[40489]: osdmap e255: 8 total, 8 up, 7 in 2024-09-21T13:57:45.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:45 smithi049 ceph-mon[30080]: pgmap v430: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 144 active+clean; 583 KiB data, 3.3 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 224 op/s 2024-09-21T13:57:45.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:45 smithi049 ceph-mon[30080]: osdmap e254: 8 total, 8 up, 7 in 2024-09-21T13:57:45.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:45 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1444762519' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "8.b", "id": [3, 0, 4]} : dispatch 2024-09-21T13:57:45.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:45 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 1 pg degraded (PG_DEGRADED) 2024-09-21T13:57:45.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:45 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1444762519' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "8.b", "id": [3, 0, 4]}]': finished 2024-09-21T13:57:45.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:45 smithi049 ceph-mon[30080]: osdmap e255: 8 total, 8 up, 7 in 2024-09-21T13:57:45.258 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:45.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:45.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:57:45.359 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:45.382 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:45.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:45.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:45.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:45 smithi187 ceph-mon[33032]: pgmap v430: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 144 active+clean; 583 KiB data, 3.3 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 224 op/s 2024-09-21T13:57:45.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:45 smithi187 ceph-mon[33032]: osdmap e254: 8 total, 8 up, 7 in 2024-09-21T13:57:45.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:45 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1444762519' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "8.b", "id": [3, 0, 4]} : dispatch 2024-09-21T13:57:45.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:45 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 1 pg degraded (PG_DEGRADED) 2024-09-21T13:57:45.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:45 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1444762519' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "8.b", "id": [3, 0, 4]}]': finished 2024-09-21T13:57:45.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:45 smithi187 ceph-mon[33032]: osdmap e255: 8 total, 8 up, 7 in 2024-09-21T13:57:45.484 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:45.508 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:45.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:45.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:45.556 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T13:57:45.610 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:45.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:45.638 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:45.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:45.739 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:57:45.761 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:45.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:45.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:57:45.862 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:45.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:45.886 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:45.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:45.988 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:46.003 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'unique_pool_1' removed 2024-09-21T13:57:46.015 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:46.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:46.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:46.117 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:57:46.145 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:46.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:46.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:57:46.156 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T13:57:46.248 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:46.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:46 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2582682256' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T13:57:46.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:46 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2582682256' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T13:57:46.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:46 smithi049 ceph-mon[40489]: osdmap e256: 8 total, 8 up, 7 in 2024-09-21T13:57:46.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:46 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2582682256' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T13:57:46.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:46 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2582682256' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T13:57:46.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:46 smithi049 ceph-mon[30080]: osdmap e256: 8 total, 8 up, 7 in 2024-09-21T13:57:46.276 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:46.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:46.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:46.377 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:46.402 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:46.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:46.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:46.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:46 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2582682256' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T13:57:46.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:46 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2582682256' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T13:57:46.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:46 smithi187 ceph-mon[33032]: osdmap e256: 8 total, 8 up, 7 in 2024-09-21T13:57:46.504 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:46.526 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:46.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:46.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:46.628 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:46.649 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:46.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:46.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:46.751 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:46.773 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:46.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:46.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:46.806 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-21T13:57:46.807 INFO:tasks.radosbench:Beginning radosbench... 2024-09-21T13:57:46.807 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-09-21T13:57:46.807 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd pool create unique_pool_2 16 2024-09-21T13:57:46.874 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:57:46.911 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:46.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:46.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:57:47.013 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:47.039 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:47.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:47.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:47.141 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:47.164 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:47.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:47.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:47.266 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:57:47.287 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:47.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:47.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:57:47.389 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:57:47.410 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:47.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:47.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:57:47.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:47 smithi187 ceph-mon[33032]: pgmap v433: 148 pgs: 1 peering, 1 active+remapped+backfilling, 2 active+undersized+remapped+backfilling, 1 active+remapped+backfill_toofull, 143 active+clean; 583 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 870 B/s rd, 0 op/s 2024-09-21T13:57:47.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:47 smithi187 ceph-mon[33032]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T13:57:47.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:47 smithi187 ceph-mon[33032]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg degraded) 2024-09-21T13:57:47.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:47 smithi049 ceph-mon[40489]: pgmap v433: 148 pgs: 1 peering, 1 active+remapped+backfilling, 2 active+undersized+remapped+backfilling, 1 active+remapped+backfill_toofull, 143 active+clean; 583 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 870 B/s rd, 0 op/s 2024-09-21T13:57:47.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:47 smithi049 ceph-mon[40489]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T13:57:47.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:47 smithi049 ceph-mon[40489]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg degraded) 2024-09-21T13:57:47.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:47 smithi049 ceph-mon[30080]: pgmap v433: 148 pgs: 1 peering, 1 active+remapped+backfilling, 2 active+undersized+remapped+backfilling, 1 active+remapped+backfill_toofull, 143 active+clean; 583 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 870 B/s rd, 0 op/s 2024-09-21T13:57:47.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:47 smithi049 ceph-mon[30080]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T13:57:47.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:47 smithi049 ceph-mon[30080]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg degraded) 2024-09-21T13:57:47.511 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:47.535 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:47.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:47.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:47.555 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:57:47.637 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:47.658 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:47.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:47.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:47.760 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:57:47.780 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:47.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:47.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:57:47.784 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:57:47.882 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:47.903 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:47.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:47.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:48.005 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:48.028 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:48.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:48.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:48.130 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:48.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:48.153 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:48.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:48.254 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:48.276 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:48.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:48.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:48.377 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:48.399 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:48.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:48.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:48.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:48 smithi187 ceph-mon[33032]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-21T13:57:48.475 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:48 smithi049 ceph-mon[40489]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-21T13:57:48.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:48 smithi049 ceph-mon[30080]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-21T13:57:48.501 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:57:48.523 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:48.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:48.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:57:48.584 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:57:48.584 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:57:48.598 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:57:48.598 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:57:48.614 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:57:48.614 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:57:48.625 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:57:48.630 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:57:48.631 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:57:48.647 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:48.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:48.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:57:48.649 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:57:48.649 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:57:48.665 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:57:48.665 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:57:48.682 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:57:48.683 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:57:48.699 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:57:48.699 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:57:48.749 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:48.771 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:48.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:48.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:48.869 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:57:48.873 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:48.887 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T13:57:48.887 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:57:48.899 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:48.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:48.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:49.000 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:57:49.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:49.024 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:49.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:57:49.125 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:49.148 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:49.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:49.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:49.249 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:49.276 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:49.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:49.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:49.377 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:57:49.400 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:49.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:49.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:57:49.435 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:57:49.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:49 smithi187 ceph-mon[33032]: pgmap v435: 132 pgs: 132 active+clean; 583 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 710 B/s rd, 0 op/s 2024-09-21T13:57:49.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:49 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/125845495' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:57:49.483 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:49 smithi049 ceph-mon[40489]: pgmap v435: 132 pgs: 132 active+clean; 583 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 710 B/s rd, 0 op/s 2024-09-21T13:57:49.483 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:49 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/125845495' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:57:49.483 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:49 smithi049 ceph-mon[30080]: pgmap v435: 132 pgs: 132 active+clean; 583 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 710 B/s rd, 0 op/s 2024-09-21T13:57:49.483 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:49 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/125845495' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:57:49.502 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:49.523 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:49.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:49.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:49.625 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:57:49.647 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:49.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:49.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:57:49.749 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:49.775 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:49.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:49.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:49.876 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:49.888 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T13:57:49.901 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:49.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:49.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:49.944 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:57:49 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:57:50.002 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:57:50.024 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:50.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:50.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:57:50.126 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:50.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:50.156 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:50.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:50.257 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:50.280 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:50.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:50.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:50.382 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:50.404 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:50.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:50.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:50.412 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:50 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:57:50.412 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:50 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:57:50.413 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:50 smithi049 ceph-mon[40489]: osdmap e257: 8 total, 8 up, 7 in 2024-09-21T13:57:50.413 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:50 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:57:50.413 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:50 smithi049 ceph-mon[40489]: osdmap e258: 8 total, 8 up, 7 in 2024-09-21T13:57:50.413 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:50 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:57:50.413 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:50 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:57:50.413 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:50 smithi049 ceph-mon[30080]: osdmap e257: 8 total, 8 up, 7 in 2024-09-21T13:57:50.413 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:50 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:57:50.413 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:50 smithi049 ceph-mon[30080]: osdmap e258: 8 total, 8 up, 7 in 2024-09-21T13:57:50.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:50 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:57:50.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:50 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:57:50.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:50 smithi187 ceph-mon[33032]: osdmap e257: 8 total, 8 up, 7 in 2024-09-21T13:57:50.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:50 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:57:50.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:50 smithi187 ceph-mon[33032]: osdmap e258: 8 total, 8 up, 7 in 2024-09-21T13:57:50.506 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:50.528 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:50.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:50.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:50.630 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:50.652 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:50.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:50.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:50.754 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:50.755 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:57:50 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:57:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:57:50.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:50.779 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:50.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:50.880 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:50.902 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:50.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:50.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:51.004 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:57:51.026 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:51.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:51.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:57:51.128 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:57:51.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:51.151 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:51.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:57:51.194 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-09-21T13:57:51.252 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:51.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:51.276 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:51.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:51.377 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:51.403 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:51.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:51.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:51.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:51 smithi187 ceph-mon[33032]: pgmap v437: 132 pgs: 132 active+clean; 583 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 711 B/s rd, 0 op/s 2024-09-21T13:57:51.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:51 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/550848042' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:57:51.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:51 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:57:51.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:51 smithi049 ceph-mon[40489]: pgmap v437: 132 pgs: 132 active+clean; 583 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 711 B/s rd, 0 op/s 2024-09-21T13:57:51.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:51 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/550848042' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:57:51.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:51 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:57:51.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:51 smithi049 ceph-mon[30080]: pgmap v437: 132 pgs: 132 active+clean; 583 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 711 B/s rd, 0 op/s 2024-09-21T13:57:51.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:51 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/550848042' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:57:51.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:51 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:57:51.505 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:57:51.527 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:51.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:51.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:57:51.629 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:51.654 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:51.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:51.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:51.756 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:51.781 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:51.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:51.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:51.883 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:51.908 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:51.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:51.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:52.010 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:52.033 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:52.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:52.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:52.135 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:52.137 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:57:52.158 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:52.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:52.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:52.199 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'unique_pool_2' created 2024-09-21T13:57:52.261 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:52.285 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:52.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:52.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:52.387 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:57:52.411 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:52.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:52.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:57:52.443 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:52 smithi049 ceph-mon[40489]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:57:52.443 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:52 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T13:57:52.443 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:52 smithi049 ceph-mon[40489]: osdmap e259: 8 total, 8 up, 7 in 2024-09-21T13:57:52.443 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:52 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2535019339' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-21T13:57:52.443 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:52 smithi049 ceph-mon[30080]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:57:52.443 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:52 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T13:57:52.443 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:52 smithi049 ceph-mon[30080]: osdmap e259: 8 total, 8 up, 7 in 2024-09-21T13:57:52.443 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:52 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2535019339' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-21T13:57:52.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:52 smithi187 ceph-mon[33032]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:57:52.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:52 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T13:57:52.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:52 smithi187 ceph-mon[33032]: osdmap e259: 8 total, 8 up, 7 in 2024-09-21T13:57:52.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:52 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2535019339' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-21T13:57:52.514 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:52.537 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:52.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:52.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:52.552 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [] live_osds: [0, 1, 4, 5, 6, 7, 2, 3] 2024-09-21T13:57:52.553 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T13:57:52.553 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-21T13:57:52.553 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd dump -f json-pretty 2024-09-21T13:57:52.639 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:57:52.661 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:52.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:52.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:57:52.762 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:52.786 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:52.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:52.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:52.829 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T13:57:52.829 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:57:52.888 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:57:52.915 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:52.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:52.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:57:53.016 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:53.041 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:53.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:53.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:53.095 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T13:57:53.143 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:53.168 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:53.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:53.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:53.270 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:53.293 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:53.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:53.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:53.395 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:57:53.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:53.418 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:53.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:57:53.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:53 smithi049 ceph-mon[40489]: pgmap v440: 132 pgs: 132 active+clean; 583 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 872 B/s rd, 0 op/s 2024-09-21T13:57:53.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:53 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2535019339' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-21T13:57:53.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:53 smithi049 ceph-mon[40489]: osdmap e260: 8 total, 8 up, 7 in 2024-09-21T13:57:53.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:53 smithi049 ceph-mon[30080]: pgmap v440: 132 pgs: 132 active+clean; 583 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 872 B/s rd, 0 op/s 2024-09-21T13:57:53.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:53 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2535019339' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-21T13:57:53.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:53 smithi049 ceph-mon[30080]: osdmap e260: 8 total, 8 up, 7 in 2024-09-21T13:57:53.519 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:53.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:53.541 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:53.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:53.642 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:53.667 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:53.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:53.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:53.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:53 smithi187 ceph-mon[33032]: pgmap v440: 132 pgs: 132 active+clean; 583 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 872 B/s rd, 0 op/s 2024-09-21T13:57:53.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:53 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2535019339' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-21T13:57:53.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:53 smithi187 ceph-mon[33032]: osdmap e260: 8 total, 8 up, 7 in 2024-09-21T13:57:53.768 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:57:53.795 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:53.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:53.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:57:53.802 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T13:57:53.807 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:57:53.897 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:53.922 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:53.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:53.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:53.951 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T13:57:54.024 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:57:54.049 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:54.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:54.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:57:54.151 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:54.173 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:54.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:54.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:54.275 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:54.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:54.298 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:54.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:54.399 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:57:54.421 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:54.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:54.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:57:54.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:54 smithi049 ceph-mon[40489]: osdmap e261: 8 total, 8 up, 7 in 2024-09-21T13:57:54.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:54 smithi049 ceph-mon[30080]: osdmap e261: 8 total, 8 up, 7 in 2024-09-21T13:57:54.524 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:54.550 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:54.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:54.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:54.652 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:54.677 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:54.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:54.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:54.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:54 smithi187 ceph-mon[33032]: osdmap e261: 8 total, 8 up, 7 in 2024-09-21T13:57:54.779 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:54.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:54.805 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:54.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:54.906 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:57:54.932 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:54.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:54.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:57:55.033 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:57:55.058 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:55.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:55.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:57:55.160 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:55.186 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:55.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:55.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:55.287 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:55.312 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:55.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:55.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:55.414 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:57:55.441 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:55.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:55.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:57:55.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:55 smithi049 ceph-mon[40489]: pgmap v443: 148 pgs: 3 creating+activating, 6 unknown, 139 active+clean; 583 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T13:57:55.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:55 smithi049 ceph-mon[40489]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T13:57:55.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:55 smithi049 ceph-mon[40489]: osdmap e262: 8 total, 8 up, 7 in 2024-09-21T13:57:55.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:55 smithi049 ceph-mon[30080]: pgmap v443: 148 pgs: 3 creating+activating, 6 unknown, 139 active+clean; 583 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T13:57:55.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:55 smithi049 ceph-mon[30080]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T13:57:55.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:55 smithi049 ceph-mon[30080]: osdmap e262: 8 total, 8 up, 7 in 2024-09-21T13:57:55.543 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:55.567 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:55.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:55.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:55.669 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:55.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:55 smithi187 ceph-mon[33032]: pgmap v443: 148 pgs: 3 creating+activating, 6 unknown, 139 active+clean; 583 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T13:57:55.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:55 smithi187 ceph-mon[33032]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T13:57:55.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:55 smithi187 ceph-mon[33032]: osdmap e262: 8 total, 8 up, 7 in 2024-09-21T13:57:55.695 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:55.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:55.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:55.797 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:55.823 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:55.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:55.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:55.925 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:55.950 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:55.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:55.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:56.052 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:56.080 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:56.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:56.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:56.182 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:56.206 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:56.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:56.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:56.308 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:56.334 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:56.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:56.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:56.435 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:57:56.461 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:56.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:56.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:57:56.562 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:56.589 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:56.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:56.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:56.691 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:56.717 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:56.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:56.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:56.819 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:56.845 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:56.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:56.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:56.946 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:57:56.973 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:56.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:56.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:57:57.075 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:57:57.098 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:57.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:57.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:57:57.200 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:57.224 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:57.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:57.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:57.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:57 smithi049 ceph-mon[40489]: pgmap v445: 148 pgs: 3 creating+activating, 145 active+clean; 583 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T13:57:57.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:57 smithi049 ceph-mon[30080]: pgmap v445: 148 pgs: 3 creating+activating, 145 active+clean; 583 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T13:57:57.267 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:57:57.268 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T13:57:57.268 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 262, 2024-09-21T13:57:57.268 INFO:teuthology.orchestra.run.smithi049.stdout: "fsid": "40a3d2e8-781f-11ef-baf6-efdc52797490", 2024-09-21T13:57:57.268 INFO:teuthology.orchestra.run.smithi049.stdout: "created": "2024-09-21T13:42:57.838275+0000", 2024-09-21T13:57:57.268 INFO:teuthology.orchestra.run.smithi049.stdout: "modified": "2024-09-21T13:57:54.209561+0000", 2024-09-21T13:57:57.268 INFO:teuthology.orchestra.run.smithi049.stdout: "last_up_change": "2024-09-21T13:57:11.653688+0000", 2024-09-21T13:57:57.268 INFO:teuthology.orchestra.run.smithi049.stdout: "last_in_change": "2024-09-21T13:55:33.181161+0000", 2024-09-21T13:57:57.268 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-21T13:57:57.268 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_num": 5806080, 2024-09-21T13:57:57.268 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_set": [ 2024-09-21T13:57:57.268 INFO:teuthology.orchestra.run.smithi049.stdout: "nodeep-scrub", 2024-09-21T13:57:57.268 INFO:teuthology.orchestra.run.smithi049.stdout: "noscrub", 2024-09-21T13:57:57.268 INFO:teuthology.orchestra.run.smithi049.stdout: "pglog_hardlimit", 2024-09-21T13:57:57.269 INFO:teuthology.orchestra.run.smithi049.stdout: "purged_snapdirs", 2024-09-21T13:57:57.269 INFO:teuthology.orchestra.run.smithi049.stdout: "recovery_deletes", 2024-09-21T13:57:57.269 INFO:teuthology.orchestra.run.smithi049.stdout: "sortbitwise" 2024-09-21T13:57:57.269 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T13:57:57.269 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_version": 20, 2024-09-21T13:57:57.269 INFO:teuthology.orchestra.run.smithi049.stdout: "full_ratio": 0.94999998807907104, 2024-09-21T13:57:57.269 INFO:teuthology.orchestra.run.smithi049.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-21T13:57:57.269 INFO:teuthology.orchestra.run.smithi049.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-21T13:57:57.269 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_snapshot": "", 2024-09-21T13:57:57.269 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_max": 9, 2024-09-21T13:57:57.269 INFO:teuthology.orchestra.run.smithi049.stdout: "max_osd": 8, 2024-09-21T13:57:57.269 INFO:teuthology.orchestra.run.smithi049.stdout: "require_min_compat_client": "luminous", 2024-09-21T13:57:57.269 INFO:teuthology.orchestra.run.smithi049.stdout: "min_compat_client": "luminous", 2024-09-21T13:57:57.269 INFO:teuthology.orchestra.run.smithi049.stdout: "require_osd_release": "squid", 2024-09-21T13:57:57.270 INFO:teuthology.orchestra.run.smithi049.stdout: "allow_crimson": false, 2024-09-21T13:57:57.270 INFO:teuthology.orchestra.run.smithi049.stdout: "pools": [ 2024-09-21T13:57:57.270 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.270 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 1, 2024-09-21T13:57:57.270 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": ".mgr", 2024-09-21T13:57:57.270 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:46:21.096244+0000", 2024-09-21T13:57:57.270 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T13:57:57.270 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T13:57:57.270 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T13:57:57.270 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T13:57:57.270 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T13:57:57.271 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T13:57:57.271 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T13:57:57.271 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T13:57:57.271 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T13:57:57.271 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T13:57:57.271 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T13:57:57.271 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T13:57:57.271 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T13:57:57.271 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 1, 2024-09-21T13:57:57.271 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 1, 2024-09-21T13:57:57.271 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 1, 2024-09-21T13:57:57.271 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 1, 2024-09-21T13:57:57.271 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 1, 2024-09-21T13:57:57.271 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T13:57:57.272 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T13:57:57.272 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T13:57:57.272 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T13:57:57.272 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T13:57:57.272 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T13:57:57.272 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T13:57:57.272 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.272 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "199", 2024-09-21T13:57:57.272 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T13:57:57.272 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T13:57:57.272 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T13:57:57.272 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T13:57:57.272 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T13:57:57.273 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T13:57:57.273 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T13:57:57.273 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T13:57:57.273 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T13:57:57.273 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T13:57:57.273 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T13:57:57.273 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T13:57:57.273 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T13:57:57.273 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T13:57:57.273 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T13:57:57.273 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T13:57:57.273 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T13:57:57.273 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T13:57:57.273 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T13:57:57.274 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T13:57:57.274 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T13:57:57.274 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T13:57:57.274 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T13:57:57.274 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T13:57:57.274 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T13:57:57.274 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T13:57:57.274 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.274 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T13:57:57.274 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T13:57:57.274 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T13:57:57.274 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T13:57:57.274 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T13:57:57.274 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T13:57:57.275 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T13:57:57.275 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T13:57:57.275 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T13:57:57.275 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T13:57:57.275 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T13:57:57.275 INFO:teuthology.orchestra.run.smithi049.stdout: "options": { 2024-09-21T13:57:57.275 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_max": 32, 2024-09-21T13:57:57.275 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_min": 1 2024-09-21T13:57:57.275 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.275 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T13:57:57.275 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr": {} 2024-09-21T13:57:57.275 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.275 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T13:57:57.275 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T13:57:57.276 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 7.8899998664855957, 2024-09-21T13:57:57.276 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 7.8899998664855957, 2024-09-21T13:57:57.276 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 0.37999999523162842, 2024-09-21T13:57:57.276 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 3, 2024-09-21T13:57:57.276 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 3, 2024-09-21T13:57:57.276 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.86855697631835938, 2024-09-21T13:57:57.276 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.87000000476837158, 2024-09-21T13:57:57.276 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.87000000476837158 2024-09-21T13:57:57.276 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:57.276 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.276 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.276 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 2, 2024-09-21T13:57:57.276 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "datapool", 2024-09-21T13:57:57.276 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:48:55.540712+0000", 2024-09-21T13:57:57.277 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 8193, 2024-09-21T13:57:57.277 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T13:57:57.277 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T13:57:57.277 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T13:57:57.277 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T13:57:57.277 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T13:57:57.277 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T13:57:57.277 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T13:57:57.277 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T13:57:57.277 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T13:57:57.277 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T13:57:57.277 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T13:57:57.277 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T13:57:57.277 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 3, 2024-09-21T13:57:57.277 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 3, 2024-09-21T13:57:57.278 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 3, 2024-09-21T13:57:57.278 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 3, 2024-09-21T13:57:57.278 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 3, 2024-09-21T13:57:57.278 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T13:57:57.278 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T13:57:57.278 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T13:57:57.278 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T13:57:57.278 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T13:57:57.278 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T13:57:57.278 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T13:57:57.278 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.278 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "63", 2024-09-21T13:57:57.278 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T13:57:57.278 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T13:57:57.279 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T13:57:57.279 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T13:57:57.279 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T13:57:57.279 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 2, 2024-09-21T13:57:57.279 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 63, 2024-09-21T13:57:57.279 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T13:57:57.279 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T13:57:57.279 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T13:57:57.279 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T13:57:57.279 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T13:57:57.279 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T13:57:57.279 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T13:57:57.279 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T13:57:57.279 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T13:57:57.280 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T13:57:57.280 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T13:57:57.280 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T13:57:57.280 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T13:57:57.280 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T13:57:57.280 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T13:57:57.280 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T13:57:57.280 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T13:57:57.280 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T13:57:57.280 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T13:57:57.280 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.280 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T13:57:57.280 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T13:57:57.280 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T13:57:57.281 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T13:57:57.281 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T13:57:57.281 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T13:57:57.281 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T13:57:57.281 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T13:57:57.281 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T13:57:57.281 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T13:57:57.281 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T13:57:57.281 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T13:57:57.281 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T13:57:57.281 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd": {} 2024-09-21T13:57:57.281 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.281 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T13:57:57.281 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T13:57:57.281 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2.6700000762939453, 2024-09-21T13:57:57.282 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2.6700000762939453, 2024-09-21T13:57:57.282 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 0.75, 2024-09-21T13:57:57.282 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2, 2024-09-21T13:57:57.282 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2, 2024-09-21T13:57:57.282 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.86855697631835938, 2024-09-21T13:57:57.282 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.87000000476837158, 2024-09-21T13:57:57.282 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.87000000476837158 2024-09-21T13:57:57.282 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:57.282 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.282 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.282 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 3, 2024-09-21T13:57:57.282 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": ".rgw.root", 2024-09-21T13:57:57.282 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:48:56.417456+0000", 2024-09-21T13:57:57.282 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T13:57:57.283 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T13:57:57.283 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T13:57:57.283 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T13:57:57.283 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T13:57:57.283 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T13:57:57.283 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T13:57:57.283 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T13:57:57.283 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T13:57:57.283 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T13:57:57.283 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T13:57:57.283 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T13:57:57.283 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T13:57:57.283 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 32, 2024-09-21T13:57:57.283 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 32, 2024-09-21T13:57:57.284 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 32, 2024-09-21T13:57:57.284 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 32, 2024-09-21T13:57:57.284 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 32, 2024-09-21T13:57:57.284 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T13:57:57.284 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T13:57:57.285 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T13:57:57.285 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T13:57:57.285 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T13:57:57.285 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T13:57:57.285 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T13:57:57.285 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.285 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "62", 2024-09-21T13:57:57.285 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T13:57:57.285 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T13:57:57.286 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T13:57:57.286 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T13:57:57.286 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T13:57:57.286 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T13:57:57.286 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T13:57:57.286 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T13:57:57.286 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T13:57:57.286 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T13:57:57.286 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T13:57:57.286 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T13:57:57.286 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T13:57:57.286 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T13:57:57.286 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T13:57:57.287 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T13:57:57.287 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T13:57:57.287 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T13:57:57.287 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T13:57:57.287 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T13:57:57.287 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T13:57:57.287 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T13:57:57.287 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T13:57:57.287 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T13:57:57.287 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T13:57:57.287 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T13:57:57.287 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.287 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T13:57:57.287 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T13:57:57.288 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T13:57:57.288 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T13:57:57.288 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T13:57:57.288 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T13:57:57.288 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T13:57:57.288 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T13:57:57.288 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T13:57:57.288 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T13:57:57.288 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T13:57:57.288 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T13:57:57.288 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T13:57:57.288 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw": {} 2024-09-21T13:57:57.288 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.288 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T13:57:57.289 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T13:57:57.289 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 1.7400000095367432, 2024-09-21T13:57:57.289 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 1.7400000095367432, 2024-09-21T13:57:57.289 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 0.87999999523162842, 2024-09-21T13:57:57.289 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 1.5299999713897705, 2024-09-21T13:57:57.289 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 1.5299999713897705, 2024-09-21T13:57:57.289 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.86855697631835938, 2024-09-21T13:57:57.289 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.87000000476837158, 2024-09-21T13:57:57.289 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.87000000476837158 2024-09-21T13:57:57.289 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:57.289 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.289 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.289 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 4, 2024-09-21T13:57:57.289 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "default.rgw.log", 2024-09-21T13:57:57.289 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:48:58.128455+0000", 2024-09-21T13:57:57.290 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T13:57:57.290 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T13:57:57.290 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T13:57:57.290 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T13:57:57.290 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T13:57:57.290 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T13:57:57.290 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T13:57:57.290 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T13:57:57.290 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T13:57:57.290 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T13:57:57.290 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T13:57:57.290 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T13:57:57.290 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T13:57:57.290 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 32, 2024-09-21T13:57:57.291 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 32, 2024-09-21T13:57:57.291 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 32, 2024-09-21T13:57:57.291 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 32, 2024-09-21T13:57:57.291 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 32, 2024-09-21T13:57:57.291 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T13:57:57.291 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T13:57:57.291 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T13:57:57.291 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T13:57:57.291 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T13:57:57.291 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T13:57:57.291 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T13:57:57.291 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.291 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "64", 2024-09-21T13:57:57.291 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T13:57:57.292 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T13:57:57.292 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T13:57:57.292 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T13:57:57.292 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T13:57:57.292 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T13:57:57.292 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T13:57:57.292 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T13:57:57.292 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T13:57:57.292 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T13:57:57.292 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T13:57:57.292 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T13:57:57.292 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T13:57:57.292 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T13:57:57.292 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T13:57:57.293 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T13:57:57.293 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T13:57:57.293 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T13:57:57.293 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T13:57:57.293 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T13:57:57.293 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T13:57:57.293 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T13:57:57.293 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T13:57:57.293 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T13:57:57.293 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T13:57:57.293 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T13:57:57.293 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.293 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T13:57:57.293 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T13:57:57.293 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T13:57:57.294 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T13:57:57.294 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T13:57:57.294 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T13:57:57.294 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T13:57:57.294 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T13:57:57.294 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T13:57:57.294 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T13:57:57.294 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T13:57:57.294 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T13:57:57.294 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T13:57:57.294 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw": {} 2024-09-21T13:57:57.294 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.294 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T13:57:57.294 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T13:57:57.295 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2.4900000095367432, 2024-09-21T13:57:57.295 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2.4900000095367432, 2024-09-21T13:57:57.295 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 0.87999999523162842, 2024-09-21T13:57:57.295 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2.190000057220459, 2024-09-21T13:57:57.295 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2.190000057220459, 2024-09-21T13:57:57.295 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.86855697631835938, 2024-09-21T13:57:57.295 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.87000000476837158, 2024-09-21T13:57:57.295 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.87000000476837158 2024-09-21T13:57:57.295 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:57.295 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.295 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.295 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 5, 2024-09-21T13:57:57.295 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "default.rgw.control", 2024-09-21T13:57:57.295 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:49:00.069361+0000", 2024-09-21T13:57:57.296 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T13:57:57.296 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T13:57:57.296 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T13:57:57.296 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T13:57:57.296 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T13:57:57.296 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T13:57:57.296 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T13:57:57.296 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T13:57:57.296 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T13:57:57.296 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T13:57:57.296 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T13:57:57.296 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T13:57:57.296 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T13:57:57.296 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 32, 2024-09-21T13:57:57.296 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 32, 2024-09-21T13:57:57.296 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 32, 2024-09-21T13:57:57.297 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 32, 2024-09-21T13:57:57.297 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 32, 2024-09-21T13:57:57.297 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T13:57:57.297 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T13:57:57.297 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T13:57:57.297 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T13:57:57.297 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T13:57:57.297 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T13:57:57.297 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T13:57:57.297 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.297 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "66", 2024-09-21T13:57:57.297 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T13:57:57.297 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T13:57:57.297 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T13:57:57.297 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T13:57:57.297 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T13:57:57.298 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T13:57:57.298 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T13:57:57.298 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T13:57:57.298 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T13:57:57.298 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T13:57:57.298 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T13:57:57.298 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T13:57:57.298 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T13:57:57.298 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T13:57:57.298 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T13:57:57.298 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T13:57:57.298 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T13:57:57.298 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T13:57:57.298 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T13:57:57.298 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T13:57:57.299 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T13:57:57.299 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T13:57:57.299 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T13:57:57.299 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T13:57:57.299 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T13:57:57.299 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T13:57:57.299 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.299 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T13:57:57.299 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T13:57:57.299 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T13:57:57.299 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T13:57:57.299 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T13:57:57.299 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T13:57:57.299 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T13:57:57.299 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T13:57:57.299 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T13:57:57.300 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T13:57:57.300 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T13:57:57.300 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T13:57:57.300 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T13:57:57.300 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw": {} 2024-09-21T13:57:57.300 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.300 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T13:57:57.300 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T13:57:57.300 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 1.7400000095367432, 2024-09-21T13:57:57.300 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 1.7400000095367432, 2024-09-21T13:57:57.300 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 0.87999999523162842, 2024-09-21T13:57:57.300 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 1.5299999713897705, 2024-09-21T13:57:57.300 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 1.5299999713897705, 2024-09-21T13:57:57.300 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.86855697631835938, 2024-09-21T13:57:57.300 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.87000000476837158, 2024-09-21T13:57:57.301 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.87000000476837158 2024-09-21T13:57:57.301 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:57.301 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.301 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.301 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 6, 2024-09-21T13:57:57.301 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "default.rgw.meta", 2024-09-21T13:57:57.301 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:49:02.127399+0000", 2024-09-21T13:57:57.301 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T13:57:57.301 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T13:57:57.301 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T13:57:57.301 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T13:57:57.301 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T13:57:57.301 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T13:57:57.301 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T13:57:57.302 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T13:57:57.302 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T13:57:57.302 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T13:57:57.302 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T13:57:57.302 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T13:57:57.302 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T13:57:57.302 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 32, 2024-09-21T13:57:57.302 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 32, 2024-09-21T13:57:57.302 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 32, 2024-09-21T13:57:57.302 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 32, 2024-09-21T13:57:57.302 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 32, 2024-09-21T13:57:57.302 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T13:57:57.302 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T13:57:57.303 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T13:57:57.303 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T13:57:57.303 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T13:57:57.303 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T13:57:57.303 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T13:57:57.303 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.303 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "68", 2024-09-21T13:57:57.303 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T13:57:57.303 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T13:57:57.303 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T13:57:57.303 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T13:57:57.303 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T13:57:57.303 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T13:57:57.303 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T13:57:57.303 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T13:57:57.304 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T13:57:57.304 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T13:57:57.304 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T13:57:57.304 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T13:57:57.304 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T13:57:57.304 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T13:57:57.304 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T13:57:57.304 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T13:57:57.304 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T13:57:57.304 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T13:57:57.304 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T13:57:57.304 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T13:57:57.304 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T13:57:57.305 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T13:57:57.305 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T13:57:57.305 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T13:57:57.305 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T13:57:57.305 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T13:57:57.305 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.305 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T13:57:57.305 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T13:57:57.305 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T13:57:57.305 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T13:57:57.305 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T13:57:57.305 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T13:57:57.305 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T13:57:57.305 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T13:57:57.306 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T13:57:57.306 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T13:57:57.306 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T13:57:57.306 INFO:teuthology.orchestra.run.smithi049.stdout: "options": { 2024-09-21T13:57:57.306 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_bias": 4 2024-09-21T13:57:57.306 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.306 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T13:57:57.306 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw": {} 2024-09-21T13:57:57.306 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.306 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T13:57:57.306 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T13:57:57.306 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 1.9900000095367432, 2024-09-21T13:57:57.306 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 1.9900000095367432, 2024-09-21T13:57:57.307 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 0.87999999523162842, 2024-09-21T13:57:57.307 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 1.75, 2024-09-21T13:57:57.307 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 1.75, 2024-09-21T13:57:57.307 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.86855697631835938, 2024-09-21T13:57:57.307 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.87000000476837158, 2024-09-21T13:57:57.307 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.87000000476837158 2024-09-21T13:57:57.307 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:57.307 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.307 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.307 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2024-09-21T13:57:57.307 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "unique_pool_2", 2024-09-21T13:57:57.307 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:57:51.528024+0000", 2024-09-21T13:57:57.307 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T13:57:57.307 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T13:57:57.308 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T13:57:57.308 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T13:57:57.308 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T13:57:57.308 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T13:57:57.308 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T13:57:57.308 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T13:57:57.308 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T13:57:57.308 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T13:57:57.308 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T13:57:57.308 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T13:57:57.308 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T13:57:57.308 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 16, 2024-09-21T13:57:57.308 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 16, 2024-09-21T13:57:57.308 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 16, 2024-09-21T13:57:57.309 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 16, 2024-09-21T13:57:57.309 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 16, 2024-09-21T13:57:57.309 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T13:57:57.309 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T13:57:57.309 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T13:57:57.309 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T13:57:57.309 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T13:57:57.309 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T13:57:57.309 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T13:57:57.309 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.309 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "262", 2024-09-21T13:57:57.309 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T13:57:57.309 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T13:57:57.309 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T13:57:57.310 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T13:57:57.310 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T13:57:57.310 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T13:57:57.310 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T13:57:57.310 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T13:57:57.310 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T13:57:57.310 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T13:57:57.310 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T13:57:57.310 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T13:57:57.310 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T13:57:57.310 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T13:57:57.310 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T13:57:57.310 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T13:57:57.310 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T13:57:57.311 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T13:57:57.311 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T13:57:57.311 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T13:57:57.311 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T13:57:57.311 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T13:57:57.311 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T13:57:57.311 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T13:57:57.311 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T13:57:57.311 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T13:57:57.311 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.311 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T13:57:57.311 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T13:57:57.311 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T13:57:57.312 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T13:57:57.312 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T13:57:57.312 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T13:57:57.312 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T13:57:57.312 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T13:57:57.312 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T13:57:57.312 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T13:57:57.312 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T13:57:57.312 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T13:57:57.312 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": {}, 2024-09-21T13:57:57.312 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T13:57:57.312 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T13:57:57.312 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2.9900000095367432, 2024-09-21T13:57:57.312 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2.9900000095367432, 2024-09-21T13:57:57.312 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 0.87999999523162842, 2024-09-21T13:57:57.313 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2.630000114440918, 2024-09-21T13:57:57.313 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2.630000114440918, 2024-09-21T13:57:57.313 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.86855697631835938, 2024-09-21T13:57:57.313 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.87000000476837158, 2024-09-21T13:57:57.313 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.87000000476837158 2024-09-21T13:57:57.313 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:57.313 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:57.313 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T13:57:57.313 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-09-21T13:57:57.313 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.313 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 0, 2024-09-21T13:57:57.313 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "2595dca9-6219-4e0c-8728-751eb102e73a", 2024-09-21T13:57:57.313 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T13:57:57.313 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T13:57:57.314 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T13:57:57.314 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 0.1215972900390625, 2024-09-21T13:57:57.314 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T13:57:57.314 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T13:57:57.314 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 8, 2024-09-21T13:57:57.314 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 255, 2024-09-21T13:57:57.314 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T13:57:57.314 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T13:57:57.314 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T13:57:57.314 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:57.314 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.314 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:57.314 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6802", 2024-09-21T13:57:57.314 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T13:57:57.315 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.315 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.315 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:57.315 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6803", 2024-09-21T13:57:57.315 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T13:57:57.315 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:57.315 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:57.315 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.315 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T13:57:57.315 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:57.315 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.315 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:57.315 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6804", 2024-09-21T13:57:57.315 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T13:57:57.315 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.316 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.316 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:57.316 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6805", 2024-09-21T13:57:57.316 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T13:57:57.316 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:57.316 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:57.316 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.316 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T13:57:57.316 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:57.316 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.316 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:57.316 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6808", 2024-09-21T13:57:57.316 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T13:57:57.317 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.317 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.317 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:57.317 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6809", 2024-09-21T13:57:57.317 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T13:57:57.317 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:57.317 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:57.317 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.317 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T13:57:57.317 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:57.317 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.317 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:57.317 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6806", 2024-09-21T13:57:57.317 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T13:57:57.317 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.317 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.318 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:57.318 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6807", 2024-09-21T13:57:57.318 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T13:57:57.318 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:57.318 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:57.318 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.318 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6803/2095561412", 2024-09-21T13:57:57.318 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6805/2095561412", 2024-09-21T13:57:57.318 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6809/2095561412", 2024-09-21T13:57:57.318 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6807/2095561412", 2024-09-21T13:57:57.318 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T13:57:57.318 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T13:57:57.318 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T13:57:57.318 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:57.319 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.319 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.319 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 1, 2024-09-21T13:57:57.319 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "0ba327f9-5524-4a40-abcc-729ded0b1917", 2024-09-21T13:57:57.319 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T13:57:57.319 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T13:57:57.319 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T13:57:57.319 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T13:57:57.319 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T13:57:57.319 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T13:57:57.319 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 13, 2024-09-21T13:57:57.319 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 260, 2024-09-21T13:57:57.319 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T13:57:57.319 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T13:57:57.319 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T13:57:57.320 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:57.320 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.320 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:57.320 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6810", 2024-09-21T13:57:57.320 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T13:57:57.320 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.320 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.320 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:57.320 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6811", 2024-09-21T13:57:57.320 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T13:57:57.320 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:57.320 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:57.320 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.320 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T13:57:57.320 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:57.321 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.321 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:57.321 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6812", 2024-09-21T13:57:57.321 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T13:57:57.321 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.321 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.321 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:57.321 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6813", 2024-09-21T13:57:57.321 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T13:57:57.321 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:57.321 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:57.321 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.321 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T13:57:57.322 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:57.322 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.322 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:57.322 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6816", 2024-09-21T13:57:57.322 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T13:57:57.322 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.322 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.322 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:57.322 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6817", 2024-09-21T13:57:57.322 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T13:57:57.322 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:57.322 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:57.322 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.322 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T13:57:57.323 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:57.323 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.323 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:57.323 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6814", 2024-09-21T13:57:57.323 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T13:57:57.323 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.323 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.323 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:57.323 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6815", 2024-09-21T13:57:57.323 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T13:57:57.323 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:57.323 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:57.323 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.323 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6811/1382840527", 2024-09-21T13:57:57.323 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6813/1382840527", 2024-09-21T13:57:57.324 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6817/1382840527", 2024-09-21T13:57:57.324 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6815/1382840527", 2024-09-21T13:57:57.324 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T13:57:57.324 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T13:57:57.324 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T13:57:57.324 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:57.324 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.324 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.324 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 2, 2024-09-21T13:57:57.324 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "27a45750-204d-48fd-bb8f-ebb01273343d", 2024-09-21T13:57:57.324 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T13:57:57.324 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T13:57:57.324 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T13:57:57.325 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 0.8268585205078125, 2024-09-21T13:57:57.325 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 18, 2024-09-21T13:57:57.325 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 100, 2024-09-21T13:57:57.325 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 130, 2024-09-21T13:57:57.325 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 240, 2024-09-21T13:57:57.325 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 101, 2024-09-21T13:57:57.325 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T13:57:57.325 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T13:57:57.325 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:57.325 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.325 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:57.325 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6818", 2024-09-21T13:57:57.325 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T13:57:57.325 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.325 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.326 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:57.326 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6819", 2024-09-21T13:57:57.326 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T13:57:57.326 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:57.326 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:57.326 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.326 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T13:57:57.326 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:57.326 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.326 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:57.326 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6820", 2024-09-21T13:57:57.326 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T13:57:57.326 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.326 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.327 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:57.327 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6821", 2024-09-21T13:57:57.327 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T13:57:57.327 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:57.327 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:57.327 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.327 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T13:57:57.327 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:57.327 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.327 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:57.327 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6824", 2024-09-21T13:57:57.327 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T13:57:57.327 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.328 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.328 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:57.328 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6825", 2024-09-21T13:57:57.328 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T13:57:57.328 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:57.328 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:57.328 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.328 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T13:57:57.328 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:57.328 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.328 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:57.328 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6822", 2024-09-21T13:57:57.328 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T13:57:57.328 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.328 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.329 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:57.329 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6823", 2024-09-21T13:57:57.329 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T13:57:57.329 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:57.329 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:57.329 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.329 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6819/1945314741", 2024-09-21T13:57:57.329 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6821/1945314741", 2024-09-21T13:57:57.329 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6825/1945314741", 2024-09-21T13:57:57.329 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6823/1945314741", 2024-09-21T13:57:57.329 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T13:57:57.329 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T13:57:57.329 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T13:57:57.329 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:57.329 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.329 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.330 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 3, 2024-09-21T13:57:57.330 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "528bc390-3dc9-4f3b-ab75-d7412c9625db", 2024-09-21T13:57:57.330 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T13:57:57.330 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T13:57:57.330 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T13:57:57.330 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T13:57:57.330 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 27, 2024-09-21T13:57:57.330 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 214, 2024-09-21T13:57:57.330 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 240, 2024-09-21T13:57:57.330 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 260, 2024-09-21T13:57:57.330 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 215, 2024-09-21T13:57:57.330 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T13:57:57.330 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T13:57:57.331 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:57.331 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.331 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:57.331 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6826", 2024-09-21T13:57:57.331 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T13:57:57.331 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.331 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.331 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:57.331 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6827", 2024-09-21T13:57:57.331 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T13:57:57.331 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:57.331 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:57.331 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.331 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T13:57:57.331 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:57.332 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.332 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:57.332 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6828", 2024-09-21T13:57:57.332 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T13:57:57.332 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.332 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.332 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:57.332 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6829", 2024-09-21T13:57:57.332 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T13:57:57.332 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:57.332 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:57.332 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.332 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T13:57:57.332 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:57.332 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.333 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:57.333 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6832", 2024-09-21T13:57:57.333 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T13:57:57.333 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.333 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.333 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:57.333 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6833", 2024-09-21T13:57:57.333 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T13:57:57.333 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:57.333 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:57.333 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.333 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T13:57:57.333 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:57.333 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.334 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:57.334 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:57.335 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6830", 2024-09-21T13:57:57.335 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T13:57:57.335 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.335 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.335 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:57.335 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6831", 2024-09-21T13:57:57.335 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T13:57:57.335 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:57.335 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:57.336 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.336 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6827/886735721", 2024-09-21T13:57:57.336 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6829/886735721", 2024-09-21T13:57:57.336 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6833/886735721", 2024-09-21T13:57:57.336 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6831/886735721", 2024-09-21T13:57:57.336 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T13:57:57.336 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T13:57:57.336 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T13:57:57.336 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:57.336 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.336 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.336 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 4, 2024-09-21T13:57:57.336 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "6fc64698-2d37-4db0-bf3d-c7e612c45163", 2024-09-21T13:57:57.337 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T13:57:57.337 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T13:57:57.337 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T13:57:57.337 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T13:57:57.337 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T13:57:57.337 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T13:57:57.337 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 33, 2024-09-21T13:57:57.337 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 240, 2024-09-21T13:57:57.337 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T13:57:57.337 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T13:57:57.337 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T13:57:57.337 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:57.337 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.337 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:57.338 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6800", 2024-09-21T13:57:57.338 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T13:57:57.338 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.338 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.338 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:57.338 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6801", 2024-09-21T13:57:57.338 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T13:57:57.338 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:57.338 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:57.338 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.338 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T13:57:57.338 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:57.338 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.339 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:57.339 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6802", 2024-09-21T13:57:57.339 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T13:57:57.339 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.339 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.339 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:57.339 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6803", 2024-09-21T13:57:57.339 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T13:57:57.339 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:57.339 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:57.339 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.339 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T13:57:57.339 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:57.339 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.340 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:57.340 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6806", 2024-09-21T13:57:57.340 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T13:57:57.340 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.340 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.340 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:57.340 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6807", 2024-09-21T13:57:57.340 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T13:57:57.340 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:57.340 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:57.340 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.340 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T13:57:57.340 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:57.340 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.341 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:57.341 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6804", 2024-09-21T13:57:57.341 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T13:57:57.341 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.341 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.341 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:57.341 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6805", 2024-09-21T13:57:57.341 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T13:57:57.341 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:57.341 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:57.341 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.341 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.187:6801/1702447776", 2024-09-21T13:57:57.341 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.187:6803/1702447776", 2024-09-21T13:57:57.341 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.187:6807/1702447776", 2024-09-21T13:57:57.341 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.187:6805/1702447776", 2024-09-21T13:57:57.341 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T13:57:57.342 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T13:57:57.342 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T13:57:57.342 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:57.342 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.342 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.342 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 5, 2024-09-21T13:57:57.342 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "8d7f62a1-7e92-4e06-9cad-86b96be514d8", 2024-09-21T13:57:57.342 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T13:57:57.342 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T13:57:57.342 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T13:57:57.342 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T13:57:57.342 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T13:57:57.342 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T13:57:57.343 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 39, 2024-09-21T13:57:57.343 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 260, 2024-09-21T13:57:57.343 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T13:57:57.343 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T13:57:57.343 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T13:57:57.343 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:57.343 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.343 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:57.343 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6808", 2024-09-21T13:57:57.343 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3659050776 2024-09-21T13:57:57.343 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.343 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.343 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:57.343 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6809", 2024-09-21T13:57:57.344 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3659050776 2024-09-21T13:57:57.344 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:57.344 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:57.344 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.344 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T13:57:57.344 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:57.344 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.344 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:57.344 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6810", 2024-09-21T13:57:57.344 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3659050776 2024-09-21T13:57:57.344 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.344 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.344 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:57.344 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6811", 2024-09-21T13:57:57.345 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3659050776 2024-09-21T13:57:57.345 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:57.345 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:57.345 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.345 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T13:57:57.345 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:57.345 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.345 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:57.345 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6814", 2024-09-21T13:57:57.345 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3659050776 2024-09-21T13:57:57.345 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.345 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.345 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:57.345 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6815", 2024-09-21T13:57:57.346 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3659050776 2024-09-21T13:57:57.346 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:57.346 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:57.346 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.346 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T13:57:57.346 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:57.346 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.346 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:57.346 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6812", 2024-09-21T13:57:57.346 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3659050776 2024-09-21T13:57:57.346 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.346 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.346 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:57.347 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6813", 2024-09-21T13:57:57.347 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3659050776 2024-09-21T13:57:57.347 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:57.347 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:57.347 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.347 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.187:6809/3659050776", 2024-09-21T13:57:57.347 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.187:6811/3659050776", 2024-09-21T13:57:57.347 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.187:6815/3659050776", 2024-09-21T13:57:57.347 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.187:6813/3659050776", 2024-09-21T13:57:57.347 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T13:57:57.347 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T13:57:57.347 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T13:57:57.347 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:57.347 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.348 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.348 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 6, 2024-09-21T13:57:57.348 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "3a7afd13-26ce-444c-9cc0-81ce2eab6ce5", 2024-09-21T13:57:57.348 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T13:57:57.348 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 0, 2024-09-21T13:57:57.348 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 0, 2024-09-21T13:57:57.348 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T13:57:57.348 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T13:57:57.348 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T13:57:57.348 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 47, 2024-09-21T13:57:57.348 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 183, 2024-09-21T13:57:57.348 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T13:57:57.348 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T13:57:57.348 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T13:57:57.348 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:57.349 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.349 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:57.349 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6816", 2024-09-21T13:57:57.349 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T13:57:57.349 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.349 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.349 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:57.349 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6817", 2024-09-21T13:57:57.349 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T13:57:57.349 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:57.349 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:57.349 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.349 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T13:57:57.349 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:57.350 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.350 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:57.350 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6818", 2024-09-21T13:57:57.350 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T13:57:57.350 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.350 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.350 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:57.350 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6819", 2024-09-21T13:57:57.350 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T13:57:57.350 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:57.350 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:57.350 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.350 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T13:57:57.350 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:57.351 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.351 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:57.351 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6822", 2024-09-21T13:57:57.351 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T13:57:57.351 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.351 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.351 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:57.351 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6823", 2024-09-21T13:57:57.351 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T13:57:57.351 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:57.351 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:57.351 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.351 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T13:57:57.351 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:57.352 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.352 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:57.352 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6820", 2024-09-21T13:57:57.352 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T13:57:57.352 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.352 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.352 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:57.352 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6821", 2024-09-21T13:57:57.352 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T13:57:57.352 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:57.352 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:57.352 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.352 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.187:6817/2728426525", 2024-09-21T13:57:57.352 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.187:6819/2728426525", 2024-09-21T13:57:57.353 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.187:6823/2728426525", 2024-09-21T13:57:57.353 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.187:6821/2728426525", 2024-09-21T13:57:57.353 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T13:57:57.353 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T13:57:57.353 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T13:57:57.353 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:57.353 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.353 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.353 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 7, 2024-09-21T13:57:57.353 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "1e08d891-8747-402e-b4c2-aa3802ce8537", 2024-09-21T13:57:57.353 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T13:57:57.353 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T13:57:57.353 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T13:57:57.353 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T13:57:57.354 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T13:57:57.354 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T13:57:57.354 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 54, 2024-09-21T13:57:57.354 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 260, 2024-09-21T13:57:57.354 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T13:57:57.354 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T13:57:57.354 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T13:57:57.354 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:57.354 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.354 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:57.354 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6824", 2024-09-21T13:57:57.354 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3680850037 2024-09-21T13:57:57.354 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.354 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.355 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:57.355 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6825", 2024-09-21T13:57:57.355 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3680850037 2024-09-21T13:57:57.355 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:57.355 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:57.355 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.355 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T13:57:57.355 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:57.355 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.355 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:57.355 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6826", 2024-09-21T13:57:57.355 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3680850037 2024-09-21T13:57:57.355 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.355 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.356 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:57.356 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6827", 2024-09-21T13:57:57.356 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3680850037 2024-09-21T13:57:57.356 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:57.356 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:57.356 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.356 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T13:57:57.356 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:57.356 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.356 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:57.356 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6830", 2024-09-21T13:57:57.356 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3680850037 2024-09-21T13:57:57.356 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.356 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.356 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:57.357 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6831", 2024-09-21T13:57:57.357 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3680850037 2024-09-21T13:57:57.357 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:57.357 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:57.357 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.357 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T13:57:57.357 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:57:57.357 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.357 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:57:57.357 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6828", 2024-09-21T13:57:57.357 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3680850037 2024-09-21T13:57:57.357 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.357 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.357 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:57:57.358 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6829", 2024-09-21T13:57:57.358 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3680850037 2024-09-21T13:57:57.358 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:57.358 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:57.358 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.358 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.187:6825/3680850037", 2024-09-21T13:57:57.358 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.187:6827/3680850037", 2024-09-21T13:57:57.358 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.187:6831/3680850037", 2024-09-21T13:57:57.358 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.187:6829/3680850037", 2024-09-21T13:57:57.358 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T13:57:57.358 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T13:57:57.358 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T13:57:57.358 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:57.358 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:57.359 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T13:57:57.359 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_xinfo": [ 2024-09-21T13:57:57.359 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.359 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 0, 2024-09-21T13:57:57.359 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T13:57:57.359 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T13:57:57.359 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T13:57:57.359 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T13:57:57.359 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T13:57:57.359 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T13:56:36.792777+0000", 2024-09-21T13:57:57.359 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T13:57:57.359 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.359 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.359 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 1, 2024-09-21T13:57:57.360 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T13:57:57.360 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T13:57:57.360 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T13:57:57.360 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T13:57:57.360 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T13:57:57.360 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T13:57:48.533160+0000", 2024-09-21T13:57:57.360 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T13:57:57.360 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.360 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.360 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 2, 2024-09-21T13:57:57.360 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T13:57:57.360 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T13:57:57.360 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T13:57:57.361 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T13:57:57.361 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T13:57:57.361 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T13:57:27.804766+0000", 2024-09-21T13:57:57.361 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 100 2024-09-21T13:57:57.361 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.361 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.361 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 3, 2024-09-21T13:57:57.361 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T13:57:57.361 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T13:57:57.361 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T13:57:57.361 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T13:57:57.361 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T13:57:57.361 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T13:57:10.678447+0000", 2024-09-21T13:57:57.361 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 214 2024-09-21T13:57:57.362 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.362 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.362 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 4, 2024-09-21T13:57:57.362 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T13:57:57.362 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T13:57:57.362 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T13:57:57.362 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T13:57:57.362 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T13:57:57.362 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T13:56:32.445570+0000", 2024-09-21T13:57:57.362 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T13:57:57.362 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.362 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.362 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 5, 2024-09-21T13:57:57.362 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T13:57:57.363 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T13:57:57.363 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T13:57:57.363 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T13:57:57.363 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T13:57:57.363 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T13:57:29.881898+0000", 2024-09-21T13:57:57.363 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T13:57:57.363 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.363 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.363 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 6, 2024-09-21T13:57:57.363 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T13:57:57.363 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T13:57:57.363 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T13:57:57.363 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T13:57:57.363 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 65536, 2024-09-21T13:57:57.364 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T13:57:00.402108+0000", 2024-09-21T13:57:57.364 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T13:57:57.364 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.364 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.364 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 7, 2024-09-21T13:57:57.364 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T13:57:57.364 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T13:57:57.364 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T13:57:57.364 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T13:57:57.364 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T13:57:57.364 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T13:56:59.703741+0000", 2024-09-21T13:57:57.364 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T13:57:57.364 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:57.364 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T13:57:57.365 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap": [ 2024-09-21T13:57:57.365 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.365 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "3.14", 2024-09-21T13:57:57.365 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-09-21T13:57:57.365 INFO:teuthology.orchestra.run.smithi049.stdout: 7, 2024-09-21T13:57:57.365 INFO:teuthology.orchestra.run.smithi049.stdout: 6, 2024-09-21T13:57:57.365 INFO:teuthology.orchestra.run.smithi049.stdout: 0 2024-09-21T13:57:57.365 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:57.365 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:57.365 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T13:57:57.365 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap_items": [ 2024-09-21T13:57:57.365 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.365 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "4.6", 2024-09-21T13:57:57.365 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-09-21T13:57:57.365 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.366 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 1, 2024-09-21T13:57:57.366 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 5 2024-09-21T13:57:57.366 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:57.366 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:57.366 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.366 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.366 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "4.1b", 2024-09-21T13:57:57.366 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-09-21T13:57:57.366 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.366 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 1, 2024-09-21T13:57:57.366 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 2 2024-09-21T13:57:57.366 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:57.366 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:57.367 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.367 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.367 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "4.1f", 2024-09-21T13:57:57.367 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-09-21T13:57:57.367 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:57.367 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 1, 2024-09-21T13:57:57.367 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 2 2024-09-21T13:57:57.367 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:57.367 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:57.367 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:57.367 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T13:57:57.367 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap_primaries": [], 2024-09-21T13:57:57.367 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_temp": [], 2024-09-21T13:57:57.367 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_temp": [], 2024-09-21T13:57:57.368 INFO:teuthology.orchestra.run.smithi049.stdout: "blocklist": { 2024-09-21T13:57:57.368 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/2731793917": "2024-09-22T13:49:27.311313+0000", 2024-09-21T13:57:57.368 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/799831425": "2024-09-22T13:49:27.311313+0000", 2024-09-21T13:57:57.368 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/2518197280": "2024-09-22T13:43:28.110499+0000", 2024-09-21T13:57:57.368 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3556829486": "2024-09-22T13:43:28.110499+0000", 2024-09-21T13:57:57.368 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/1249926115": "2024-09-22T13:49:27.311313+0000", 2024-09-21T13:57:57.368 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/231530138": "2024-09-22T13:43:28.110499+0000", 2024-09-21T13:57:57.368 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/4220781134": "2024-09-22T13:49:27.311313+0000", 2024-09-21T13:57:57.368 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/202325455": "2024-09-22T13:43:28.110499+0000", 2024-09-21T13:57:57.368 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/4129636473": "2024-09-22T13:43:56.633723+0000", 2024-09-21T13:57:57.368 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/2518197280": "2024-09-22T13:43:28.110499+0000", 2024-09-21T13:57:57.368 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3514741273": "2024-09-22T13:43:56.633723+0000", 2024-09-21T13:57:57.368 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/4129636473": "2024-09-22T13:43:56.633723+0000", 2024-09-21T13:57:57.368 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3125743776": "2024-09-22T13:43:56.633723+0000", 2024-09-21T13:57:57.369 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3600977363": "2024-09-22T13:43:56.633723+0000", 2024-09-21T13:57:57.369 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/3453571598": "2024-09-22T13:49:27.311313+0000", 2024-09-21T13:57:57.369 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/3453571598": "2024-09-22T13:49:27.311313+0000", 2024-09-21T13:57:57.369 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3553734432": "2024-09-22T13:49:27.311313+0000" 2024-09-21T13:57:57.369 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.369 INFO:teuthology.orchestra.run.smithi049.stdout: "range_blocklist": {}, 2024-09-21T13:57:57.369 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profiles": { 2024-09-21T13:57:57.369 INFO:teuthology.orchestra.run.smithi049.stdout: "default": { 2024-09-21T13:57:57.369 INFO:teuthology.orchestra.run.smithi049.stdout: "crush-failure-domain": "osd", 2024-09-21T13:57:57.369 INFO:teuthology.orchestra.run.smithi049.stdout: "k": "2", 2024-09-21T13:57:57.369 INFO:teuthology.orchestra.run.smithi049.stdout: "m": "1", 2024-09-21T13:57:57.369 INFO:teuthology.orchestra.run.smithi049.stdout: "plugin": "jerasure", 2024-09-21T13:57:57.369 INFO:teuthology.orchestra.run.smithi049.stdout: "technique": "reed_sol_van" 2024-09-21T13:57:57.369 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:57.369 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:57.370 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps_queue": [], 2024-09-21T13:57:57.370 INFO:teuthology.orchestra.run.smithi049.stdout: "new_removed_snaps": [], 2024-09-21T13:57:57.370 INFO:teuthology.orchestra.run.smithi049.stdout: "new_purged_snaps": [], 2024-09-21T13:57:57.370 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_node_flags": {}, 2024-09-21T13:57:57.370 INFO:teuthology.orchestra.run.smithi049.stdout: "device_class_flags": {}, 2024-09-21T13:57:57.370 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode": { 2024-09-21T13:57:57.370 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode_enabled": false, 2024-09-21T13:57:57.370 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_bucket_count": 0, 2024-09-21T13:57:57.370 INFO:teuthology.orchestra.run.smithi049.stdout: "degraded_stretch_mode": 0, 2024-09-21T13:57:57.370 INFO:teuthology.orchestra.run.smithi049.stdout: "recovering_stretch_mode": 0, 2024-09-21T13:57:57.370 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode_bucket": 0 2024-09-21T13:57:57.370 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:57.370 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T13:57:57.392 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:57.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:57.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:57.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:57 smithi187 ceph-mon[33032]: pgmap v445: 148 pgs: 3 creating+activating, 145 active+clean; 583 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-21T13:57:57.494 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:57.518 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:57.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:57.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:57.620 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:57:57.622 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-09-21T13:57:57.645 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:57.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:57.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:57:57.746 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:57:57.769 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:57.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:57.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:57:57.871 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:57.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:57.895 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:57.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:57.963 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:57:57.995 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:57:58.007 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:57:58.022 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:58.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:58.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:57:58.123 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:57:58.147 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:58.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:58.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:57:58.251 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:57:58.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:58.274 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:58.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:57:58.362 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:58 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/341272278' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T13:57:58.362 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:58 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/341272278' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T13:57:58.376 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:58.398 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:58.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:58.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:57:58.500 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:58.522 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:58.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:58.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:58.624 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:58.646 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:58.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:58.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:58.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:58 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/341272278' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T13:57:58.748 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:58.771 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:58.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:58.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:58.872 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:58.895 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:58.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:58.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:58.966 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T13:57:58.966 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-09-21T13:57:58.966 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-09-21T13:57:58.967 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-09-21T13:57:58.967 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.967 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.14817.0:49348 7.f e98/94 7:fcf6cd05:::benchmark_data_smithi049_76798_object49347:head v 98'3012, mlcod=98'3010)", 2024-09-21T13:57:58.967 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:38.253256+0000", 2024-09-21T13:57:58.967 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 440.69063920899998, 2024-09-21T13:57:58.967 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.155802151, 2024-09-21T13:57:58.967 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:58.967 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:58.967 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:58.967 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:58.967 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.967 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:58.967 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.253256+0000", 2024-09-21T13:57:58.967 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:58.968 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.968 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.968 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:58.968 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.253256+0000", 2024-09-21T13:57:58.968 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:58.968 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.968 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.968 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:58.968 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.253257+0000", 2024-09-21T13:57:58.968 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0599999999999999e-07 2024-09-21T13:57:58.968 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.968 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.968 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:58.968 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.253283+0000", 2024-09-21T13:57:58.969 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.584e-05 2024-09-21T13:57:58.969 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.969 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.969 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:58.969 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.253293+0000", 2024-09-21T13:57:58.969 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0341e-05 2024-09-21T13:57:58.969 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.969 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.969 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:58.969 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.253300+0000", 2024-09-21T13:57:58.969 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9249999999999998e-06 2024-09-21T13:57:58.969 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.969 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.969 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:58.969 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.253596+0000", 2024-09-21T13:57:58.970 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00029586800000000002 2024-09-21T13:57:58.970 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.970 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.970 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:57:58.970 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.253614+0000", 2024-09-21T13:57:58.970 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7682000000000001e-05 2024-09-21T13:57:58.970 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.970 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.970 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:57:58.970 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.409015+0000", 2024-09-21T13:57:58.970 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.155401716 2024-09-21T13:57:58.970 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.970 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.970 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:58.971 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.409058+0000", 2024-09-21T13:57:58.971 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2873000000000001e-05 2024-09-21T13:57:58.971 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:58.971 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:58.971 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:58.971 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.971 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.971 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.14817.0:49359 7.e e98/83 7:7ae75bfc:::benchmark_data_smithi049_76798_object49358:head v 98'3124, mlcod=98'3122)", 2024-09-21T13:57:58.971 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:38.254961+0000", 2024-09-21T13:57:58.971 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 440.68893399199999, 2024-09-21T13:57:58.971 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.152662036, 2024-09-21T13:57:58.971 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:58.971 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:58.971 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:58.972 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:58.972 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.972 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:58.972 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.254961+0000", 2024-09-21T13:57:58.972 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:58.972 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.972 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.972 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:58.972 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.254961+0000", 2024-09-21T13:57:58.973 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:58.973 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.973 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.973 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:58.973 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.254962+0000", 2024-09-21T13:57:58.973 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1969999999999999e-06 2024-09-21T13:57:58.973 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.973 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.973 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:58.973 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.254987+0000", 2024-09-21T13:57:58.973 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4510000000000001e-05 2024-09-21T13:57:58.973 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.973 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.973 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:58.974 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.255000+0000", 2024-09-21T13:57:58.974 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2543e-05 2024-09-21T13:57:58.974 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.974 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.974 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:58.974 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.255007+0000", 2024-09-21T13:57:58.974 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1840000000000002e-06 2024-09-21T13:57:58.974 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.974 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.974 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:58.975 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257538+0000", 2024-09-21T13:57:58.975 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0025315950000000002 2024-09-21T13:57:58.975 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.975 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.975 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:57:58.975 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257559+0000", 2024-09-21T13:57:58.975 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0812999999999999e-05 2024-09-21T13:57:58.975 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.975 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.975 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:57:58.975 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.259702+0000", 2024-09-21T13:57:58.975 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0021427920000000001 2024-09-21T13:57:58.975 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.975 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.976 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:58.976 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.407623+0000", 2024-09-21T13:57:58.976 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.14792140200000001 2024-09-21T13:57:58.976 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:58.976 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:58.976 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:58.976 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.976 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.976 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.14817.0:49358 7.d e98/83 7:bafe9a80:::benchmark_data_smithi049_76798_object49357:head v 98'3118, mlcod=98'3116)", 2024-09-21T13:57:58.976 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:38.257159+0000", 2024-09-21T13:57:58.976 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 440.68673621800002, 2024-09-21T13:57:58.976 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.14991167599999999, 2024-09-21T13:57:58.976 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:58.976 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:58.976 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:58.977 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:58.977 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.977 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:58.977 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257159+0000", 2024-09-21T13:57:58.977 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:58.977 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.977 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.977 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:58.977 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257159+0000", 2024-09-21T13:57:58.977 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:58.977 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.977 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.977 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:58.977 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257160+0000", 2024-09-21T13:57:58.978 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.198e-06 2024-09-21T13:57:58.978 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.978 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.978 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:58.978 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257209+0000", 2024-09-21T13:57:58.978 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9098000000000003e-05 2024-09-21T13:57:58.978 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.978 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.978 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:58.978 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257226+0000", 2024-09-21T13:57:58.978 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6657e-05 2024-09-21T13:57:58.978 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.978 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.978 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:58.978 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257236+0000", 2024-09-21T13:57:58.979 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6900000000000004e-06 2024-09-21T13:57:58.979 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.979 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.979 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:58.979 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257650+0000", 2024-09-21T13:57:58.979 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000414693 2024-09-21T13:57:58.979 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.979 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.979 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:57:58.979 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257669+0000", 2024-09-21T13:57:58.979 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8938999999999999e-05 2024-09-21T13:57:58.979 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.979 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.979 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:57:58.979 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.407019+0000", 2024-09-21T13:57:58.980 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.149349182 2024-09-21T13:57:58.980 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.980 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.980 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:58.980 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.407071+0000", 2024-09-21T13:57:58.980 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2219000000000003e-05 2024-09-21T13:57:58.980 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:58.980 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:58.980 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:58.980 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.980 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.980 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.14817.0:49364 7.d e98/83 7:ba37e2e5:::benchmark_data_smithi049_76798_object49363:head v 98'3119, mlcod=98'3116)", 2024-09-21T13:57:58.980 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:38.257264+0000", 2024-09-21T13:57:58.980 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 440.68663165200002, 2024-09-21T13:57:58.980 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.15315973299999999, 2024-09-21T13:57:58.981 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:58.981 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:58.981 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:58.981 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:58.981 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.981 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:58.981 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257264+0000", 2024-09-21T13:57:58.981 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:58.981 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.981 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.981 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:58.981 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257264+0000", 2024-09-21T13:57:58.982 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:58.982 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.982 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.982 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:58.982 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257264+0000", 2024-09-21T13:57:58.982 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9000000000000002e-07 2024-09-21T13:57:58.982 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.982 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.982 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:58.982 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257298+0000", 2024-09-21T13:57:58.982 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3547999999999998e-05 2024-09-21T13:57:58.982 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.982 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.983 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:58.983 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257306+0000", 2024-09-21T13:57:58.983 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0809999999999994e-06 2024-09-21T13:57:58.983 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.983 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.983 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:58.983 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257311+0000", 2024-09-21T13:57:58.983 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7950000000000004e-06 2024-09-21T13:57:58.983 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.983 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.983 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:58.983 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.407103+0000", 2024-09-21T13:57:58.983 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.14979118299999999 2024-09-21T13:57:58.983 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.983 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.984 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:57:58.984 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.407123+0000", 2024-09-21T13:57:58.984 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9993e-05 2024-09-21T13:57:58.984 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.984 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.984 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:57:58.984 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.410375+0000", 2024-09-21T13:57:58.984 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0032521910000000002 2024-09-21T13:57:58.984 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.984 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.984 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:58.984 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.410423+0000", 2024-09-21T13:57:58.984 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8551999999999999e-05 2024-09-21T13:57:58.984 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:58.985 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:58.985 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:58.985 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.985 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.985 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.14817.0:49355 7.f e98/94 7:f1158586:::benchmark_data_smithi049_76798_object49354:head v 98'3013, mlcod=98'3010)", 2024-09-21T13:57:58.985 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:38.257372+0000", 2024-09-21T13:57:58.985 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 440.68652363299998, 2024-09-21T13:57:58.985 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.15605590799999999, 2024-09-21T13:57:58.985 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:58.985 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:58.985 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:58.985 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:58.985 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.985 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:58.985 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257372+0000", 2024-09-21T13:57:58.986 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:58.986 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.986 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.986 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:58.986 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257372+0000", 2024-09-21T13:57:58.986 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:58.986 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.986 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.986 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:58.986 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257372+0000", 2024-09-21T13:57:58.986 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7599999999999997e-07 2024-09-21T13:57:58.986 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.986 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.987 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:58.987 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257400+0000", 2024-09-21T13:57:58.987 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.7801e-05 2024-09-21T13:57:58.987 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.987 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.987 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:58.987 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257410+0000", 2024-09-21T13:57:58.987 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.978e-06 2024-09-21T13:57:58.987 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.987 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.987 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:58.987 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257415+0000", 2024-09-21T13:57:58.987 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3009999999999997e-06 2024-09-21T13:57:58.987 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.987 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.988 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:58.988 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.407104+0000", 2024-09-21T13:57:58.988 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.14968894099999999 2024-09-21T13:57:58.988 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.988 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.988 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:57:58.988 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.407125+0000", 2024-09-21T13:57:58.988 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1132999999999999e-05 2024-09-21T13:57:58.988 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.988 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.988 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:57:58.988 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.413375+0000", 2024-09-21T13:57:58.988 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0062498959999999996 2024-09-21T13:57:58.989 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.989 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.989 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:58.989 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.413428+0000", 2024-09-21T13:57:58.989 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2182000000000003e-05 2024-09-21T13:57:58.989 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:58.989 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:58.989 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:58.989 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.989 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.989 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.14817.0:49357 7.f e98/94 7:fffc650e:::benchmark_data_smithi049_76798_object49356:head v 98'3014, mlcod=98'3010)", 2024-09-21T13:57:58.989 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:38.259358+0000", 2024-09-21T13:57:58.989 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 440.68453700700002, 2024-09-21T13:57:58.989 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.160711944, 2024-09-21T13:57:58.989 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:58.990 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:58.990 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:58.990 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:58.990 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.990 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:58.990 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.259358+0000", 2024-09-21T13:57:58.990 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:58.990 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.990 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.990 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:58.990 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.259358+0000", 2024-09-21T13:57:58.990 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:58.990 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.990 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.991 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:58.991 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.259359+0000", 2024-09-21T13:57:58.991 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4099999999999998e-07 2024-09-21T13:57:58.991 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.991 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.991 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:58.991 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.259382+0000", 2024-09-21T13:57:58.991 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3357e-05 2024-09-21T13:57:58.991 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.991 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.991 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:58.991 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.259394+0000", 2024-09-21T13:57:58.991 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.183e-05 2024-09-21T13:57:58.991 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.992 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.992 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:58.992 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.259401+0000", 2024-09-21T13:57:58.992 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1930000000000004e-06 2024-09-21T13:57:58.992 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.992 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.992 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:58.992 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.409117+0000", 2024-09-21T13:57:58.992 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.149715182 2024-09-21T13:57:58.992 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.992 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.992 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:57:58.992 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.409151+0000", 2024-09-21T13:57:58.992 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4832999999999997e-05 2024-09-21T13:57:58.993 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.993 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.993 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:57:58.993 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.419333+0000", 2024-09-21T13:57:58.993 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.010181913000000001 2024-09-21T13:57:58.993 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.993 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.993 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:58.993 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.420070+0000", 2024-09-21T13:57:58.993 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000736895 2024-09-21T13:57:58.993 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:58.993 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:58.993 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:58.993 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.994 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.994 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.14817.0:49366 7.9 e98/94 7:956216ae:::benchmark_data_smithi049_76798_object49365:head v 98'3074, mlcod=98'3072)", 2024-09-21T13:57:58.994 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:38.259484+0000", 2024-09-21T13:57:58.994 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 440.68441106300003, 2024-09-21T13:57:58.994 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.15026700000000001, 2024-09-21T13:57:58.994 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:58.994 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:58.994 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:58.994 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:58.994 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.994 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:58.994 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.259484+0000", 2024-09-21T13:57:58.994 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:58.994 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.995 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.995 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:58.995 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.259484+0000", 2024-09-21T13:57:58.995 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:58.995 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.995 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.995 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:58.995 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.259485+0000", 2024-09-21T13:57:58.995 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0299999999999998e-07 2024-09-21T13:57:58.995 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.995 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.996 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:58.996 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.259543+0000", 2024-09-21T13:57:58.996 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7998000000000003e-05 2024-09-21T13:57:58.996 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.996 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.996 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:58.996 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.259554+0000", 2024-09-21T13:57:58.996 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0842000000000001e-05 2024-09-21T13:57:58.996 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.996 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.996 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:58.996 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.259561+0000", 2024-09-21T13:57:58.996 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6680000000000004e-06 2024-09-21T13:57:58.996 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.996 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.997 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:58.997 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.259639+0000", 2024-09-21T13:57:58.997 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7986999999999997e-05 2024-09-21T13:57:58.997 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.997 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.997 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:57:58.997 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.407005+0000", 2024-09-21T13:57:58.997 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.14736633399999999 2024-09-21T13:57:58.997 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.997 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.997 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:57:58.997 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.409267+0000", 2024-09-21T13:57:58.997 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0022621690000000001 2024-09-21T13:57:58.997 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.998 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.998 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:58.998 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.409751+0000", 2024-09-21T13:57:58.998 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00048419899999999999 2024-09-21T13:57:58.998 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:58.998 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:58.998 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:58.998 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.998 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.998 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.14817.0:49368 7.d e98/83 7:bfd0cd80:::benchmark_data_smithi049_76798_object49367:head v 98'3120, mlcod=98'3116)", 2024-09-21T13:57:58.998 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:38.260130+0000", 2024-09-21T13:57:58.998 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 440.683765628, 2024-09-21T13:57:58.998 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.151202637, 2024-09-21T13:57:58.998 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:58.998 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:58.999 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:58.999 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:58.999 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.999 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:58.999 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.260130+0000", 2024-09-21T13:57:58.999 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:58.999 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.999 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.999 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:58.999 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.260130+0000", 2024-09-21T13:57:58.999 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:58.999 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:58.999 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:58.999 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:59.000 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.260130+0000", 2024-09-21T13:57:59.000 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1100000000000005e-07 2024-09-21T13:57:59.000 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.000 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.000 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:59.000 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.260158+0000", 2024-09-21T13:57:59.000 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.7708e-05 2024-09-21T13:57:59.000 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.000 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.000 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:59.000 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.407103+0000", 2024-09-21T13:57:59.000 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.146944514 2024-09-21T13:57:59.000 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.000 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.001 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:59.001 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.407112+0000", 2024-09-21T13:57:59.001 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2569999999999996e-06 2024-09-21T13:57:59.001 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.001 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.001 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:59.001 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.408861+0000", 2024-09-21T13:57:59.001 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0017486590000000001 2024-09-21T13:57:59.001 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.001 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.001 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:57:59.001 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.408881+0000", 2024-09-21T13:57:59.001 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0038999999999999e-05 2024-09-21T13:57:59.001 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.001 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.002 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:57:59.002 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.411202+0000", 2024-09-21T13:57:59.002 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00232153 2024-09-21T13:57:59.002 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.002 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.002 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:59.002 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.411332+0000", 2024-09-21T13:57:59.002 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013011899999999999 2024-09-21T13:57:59.002 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:59.002 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:59.002 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:59.002 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.002 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.003 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.14817.0:49369 7.e e98/83 7:7b8e094d:::benchmark_data_smithi049_76798_object49368:head v 98'3125, mlcod=98'3122)", 2024-09-21T13:57:59.003 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:38.260639+0000", 2024-09-21T13:57:59.003 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 440.68325663299998, 2024-09-21T13:57:59.003 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.14984267100000001, 2024-09-21T13:57:59.003 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:59.003 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:59.003 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:59.003 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:59.003 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.003 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:59.003 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.260639+0000", 2024-09-21T13:57:59.003 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:59.003 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.003 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.004 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:59.004 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.260639+0000", 2024-09-21T13:57:59.004 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:59.004 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.004 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.004 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:59.004 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.260640+0000", 2024-09-21T13:57:59.004 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0400000000000005e-07 2024-09-21T13:57:59.004 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.004 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.004 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:59.004 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.260663+0000", 2024-09-21T13:57:59.004 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3859e-05 2024-09-21T13:57:59.004 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.005 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.005 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:59.005 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.406927+0000", 2024-09-21T13:57:59.005 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.146263533 2024-09-21T13:57:59.005 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.005 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.005 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:59.005 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.406950+0000", 2024-09-21T13:57:59.005 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3014e-05 2024-09-21T13:57:59.005 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.005 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.005 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:59.005 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.407706+0000", 2024-09-21T13:57:59.005 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00075584600000000001 2024-09-21T13:57:59.006 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.006 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.006 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:57:59.006 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.407723+0000", 2024-09-21T13:57:59.006 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7085999999999999e-05 2024-09-21T13:57:59.006 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.006 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.006 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:57:59.006 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.410449+0000", 2024-09-21T13:57:59.006 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0027258680000000002 2024-09-21T13:57:59.006 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.006 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.007 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:59.007 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.410481+0000", 2024-09-21T13:57:59.007 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2561000000000002e-05 2024-09-21T13:57:59.007 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:59.007 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:59.007 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:59.007 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.007 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.007 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:13660 8.e 8:7e25e821:::benchmark_data_smithi049_109311_object13659:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-09-21T13:57:59.007 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:54:22.839402+0000", 2024-09-21T13:57:59.007 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 216.104493523, 2024-09-21T13:57:59.007 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.106316493, 2024-09-21T13:57:59.007 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:59.007 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:59.008 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:59.008 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:57:59.008 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:57:59.008 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:57:59.008 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 13660 2024-09-21T13:57:59.008 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.008 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:59.008 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.008 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:59.008 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.839402+0000", 2024-09-21T13:57:59.008 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:59.008 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.008 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.008 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:59.009 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.839402+0000", 2024-09-21T13:57:59.009 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:59.009 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.009 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.009 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:59.009 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.839403+0000", 2024-09-21T13:57:59.009 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.573e-06 2024-09-21T13:57:59.009 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.009 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.009 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:59.009 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.839426+0000", 2024-09-21T13:57:59.009 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3059000000000001e-05 2024-09-21T13:57:59.009 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.009 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.010 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:59.010 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.839438+0000", 2024-09-21T13:57:59.010 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1851000000000001e-05 2024-09-21T13:57:59.010 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.010 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.010 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:59.010 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.839446+0000", 2024-09-21T13:57:59.010 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8609999999999998e-06 2024-09-21T13:57:59.010 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.010 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.010 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:59.010 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.839994+0000", 2024-09-21T13:57:59.010 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00054770399999999999 2024-09-21T13:57:59.010 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.011 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.011 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:57:59.011 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.840234+0000", 2024-09-21T13:57:59.011 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00023994700000000001 2024-09-21T13:57:59.011 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.011 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.011 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,4", 2024-09-21T13:57:59.011 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.840537+0000", 2024-09-21T13:57:59.011 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000303004 2024-09-21T13:57:59.011 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.011 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.011 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:59.011 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.842148+0000", 2024-09-21T13:57:59.011 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0016112570000000001 2024-09-21T13:57:59.011 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.012 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.012 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:59.012 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.842528+0000", 2024-09-21T13:57:59.012 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00038031000000000002 2024-09-21T13:57:59.012 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.012 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.012 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:57:59.012 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.842792+0000", 2024-09-21T13:57:59.012 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00026319199999999998 2024-09-21T13:57:59.012 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.012 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.012 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:57:59.012 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.945668+0000", 2024-09-21T13:57:59.012 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.10287652 2024-09-21T13:57:59.013 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.013 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.013 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:59.013 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.945718+0000", 2024-09-21T13:57:59.013 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0214999999999999e-05 2024-09-21T13:57:59.013 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:59.013 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:59.013 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:59.013 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.013 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.013 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:14213 8.c 8:38837bbf:::benchmark_data_smithi049_109311_object14212:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-09-21T13:57:59.013 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:54:23.116628+0000", 2024-09-21T13:57:59.013 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 215.827267492, 2024-09-21T13:57:59.013 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.120616289, 2024-09-21T13:57:59.013 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:59.014 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:59.014 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:59.014 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:57:59.014 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:57:59.014 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:57:59.014 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 14213 2024-09-21T13:57:59.014 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.014 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:59.014 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.014 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:59.014 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.116628+0000", 2024-09-21T13:57:59.014 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:59.014 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.014 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.015 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:59.015 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.116628+0000", 2024-09-21T13:57:59.015 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:59.015 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.015 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.015 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:59.015 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.116629+0000", 2024-09-21T13:57:59.015 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8299999999999995e-07 2024-09-21T13:57:59.015 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.015 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.015 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:59.015 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.116651+0000", 2024-09-21T13:57:59.015 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2005e-05 2024-09-21T13:57:59.015 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.016 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.016 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:59.016 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.116661+0000", 2024-09-21T13:57:59.016 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0533999999999999e-05 2024-09-21T13:57:59.016 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.016 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.016 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:59.016 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.116668+0000", 2024-09-21T13:57:59.016 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9349999999999997e-06 2024-09-21T13:57:59.016 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.016 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.016 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:59.016 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.116996+0000", 2024-09-21T13:57:59.016 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00032820400000000002 2024-09-21T13:57:59.016 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.017 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.017 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:57:59.017 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.117103+0000", 2024-09-21T13:57:59.017 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010673100000000001 2024-09-21T13:57:59.017 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.017 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.017 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,7", 2024-09-21T13:57:59.017 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.117264+0000", 2024-09-21T13:57:59.017 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016054200000000001 2024-09-21T13:57:59.017 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.017 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.017 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:57:59.017 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.118050+0000", 2024-09-21T13:57:59.018 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00078586699999999999 2024-09-21T13:57:59.018 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.018 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.018 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:59.018 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.118727+0000", 2024-09-21T13:57:59.018 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00067759600000000004 2024-09-21T13:57:59.018 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.018 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.018 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:59.018 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.119082+0000", 2024-09-21T13:57:59.018 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00035450800000000001 2024-09-21T13:57:59.018 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.018 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.018 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:57:59.018 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.119119+0000", 2024-09-21T13:57:59.019 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7568999999999999e-05 2024-09-21T13:57:59.019 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.019 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.019 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:59.019 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.237244+0000", 2024-09-21T13:57:59.019 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.11812481499999999 2024-09-21T13:57:59.019 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:59.019 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:59.019 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:59.019 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.019 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.019 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.15207.0:14214 8.2 e189/183 8:46d3a502:::benchmark_data_smithi049_109311_object14213:head v 189'885, mlcod=189'883)", 2024-09-21T13:57:59.019 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:54:23.118548+0000", 2024-09-21T13:57:59.019 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 215.825347437, 2024-09-21T13:57:59.020 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.118761609, 2024-09-21T13:57:59.020 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:59.020 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:59.020 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:59.020 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:59.020 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.020 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:59.020 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.118548+0000", 2024-09-21T13:57:59.020 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:59.020 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.020 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.020 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:59.020 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.118548+0000", 2024-09-21T13:57:59.020 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:59.020 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.021 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.021 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:59.021 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.118548+0000", 2024-09-21T13:57:59.021 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3e-07 2024-09-21T13:57:59.021 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.021 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.021 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:59.021 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.118593+0000", 2024-09-21T13:57:59.021 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4131999999999997e-05 2024-09-21T13:57:59.021 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.021 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.021 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:59.021 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.118604+0000", 2024-09-21T13:57:59.021 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.134e-05 2024-09-21T13:57:59.021 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.022 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.022 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:59.022 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.118611+0000", 2024-09-21T13:57:59.022 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4619999999999999e-06 2024-09-21T13:57:59.022 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.022 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.022 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:59.022 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.118669+0000", 2024-09-21T13:57:59.022 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7828999999999998e-05 2024-09-21T13:57:59.022 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.022 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.022 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:57:59.022 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.118690+0000", 2024-09-21T13:57:59.022 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0596999999999999e-05 2024-09-21T13:57:59.023 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.023 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.023 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:57:59.023 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.237258+0000", 2024-09-21T13:57:59.023 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.11856844599999999 2024-09-21T13:57:59.023 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.023 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.023 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:59.023 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.237309+0000", 2024-09-21T13:57:59.023 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1172999999999999e-05 2024-09-21T13:57:59.023 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:59.023 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:59.023 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:59.023 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.023 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.024 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.15207.0:14213 8.c e189/183 ondisk, result = 0)", 2024-09-21T13:57:59.024 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:54:23.118845+0000", 2024-09-21T13:57:59.024 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 215.82504999899999, 2024-09-21T13:57:59.024 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.11840756199999999, 2024-09-21T13:57:59.024 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:59.024 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:59.024 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T13:57:59.024 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:59.024 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.024 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:59.024 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.118845+0000", 2024-09-21T13:57:59.024 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:59.024 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.024 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.024 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:59.025 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.118845+0000", 2024-09-21T13:57:59.025 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:59.025 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.025 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.025 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:59.025 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.118846+0000", 2024-09-21T13:57:59.025 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3e-07 2024-09-21T13:57:59.025 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.025 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.025 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:59.025 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.118850+0000", 2024-09-21T13:57:59.025 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0219999999999998e-06 2024-09-21T13:57:59.025 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.025 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.026 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:59.026 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.118860+0000", 2024-09-21T13:57:59.026 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0162e-05 2024-09-21T13:57:59.026 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.026 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.026 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:59.026 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.118866+0000", 2024-09-21T13:57:59.026 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8980000000000001e-06 2024-09-21T13:57:59.026 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.026 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.026 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:59.026 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.119070+0000", 2024-09-21T13:57:59.026 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000204286 2024-09-21T13:57:59.026 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.026 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.027 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:57:59.027 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.119077+0000", 2024-09-21T13:57:59.027 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9909999999999996e-06 2024-09-21T13:57:59.027 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.027 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.027 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:59.027 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.237253+0000", 2024-09-21T13:57:59.027 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.118175473 2024-09-21T13:57:59.027 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:59.027 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:59.027 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:59.027 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.027 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.027 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.15207.0:14217 8.2 e189/183 8:42c0ac0e:::benchmark_data_smithi049_109311_object14216:head v 189'886, mlcod=189'883)", 2024-09-21T13:57:59.028 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:54:23.119299+0000", 2024-09-21T13:57:59.028 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 215.82459615600001, 2024-09-21T13:57:59.028 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.119049031, 2024-09-21T13:57:59.028 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:59.028 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:59.028 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:59.028 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:59.028 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.028 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:59.028 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.119299+0000", 2024-09-21T13:57:59.028 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:59.028 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.028 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.028 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:59.029 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.119299+0000", 2024-09-21T13:57:59.029 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:59.029 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.029 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.029 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:59.029 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.119304+0000", 2024-09-21T13:57:59.029 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9790000000000001e-06 2024-09-21T13:57:59.029 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.029 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.029 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:59.029 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.119344+0000", 2024-09-21T13:57:59.029 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9943e-05 2024-09-21T13:57:59.029 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.029 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.029 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:59.030 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.237161+0000", 2024-09-21T13:57:59.030 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.11781683 2024-09-21T13:57:59.030 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.030 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.030 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:59.030 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.237177+0000", 2024-09-21T13:57:59.030 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5709000000000002e-05 2024-09-21T13:57:59.030 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.030 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.030 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:59.030 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.237338+0000", 2024-09-21T13:57:59.030 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016102600000000001 2024-09-21T13:57:59.030 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.030 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.031 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:57:59.031 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.237359+0000", 2024-09-21T13:57:59.031 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1659999999999999e-05 2024-09-21T13:57:59.031 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.031 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.031 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:57:59.031 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.238317+0000", 2024-09-21T13:57:59.031 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00095745599999999998 2024-09-21T13:57:59.031 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.031 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.031 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:59.031 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.238348+0000", 2024-09-21T13:57:59.031 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1427999999999998e-05 2024-09-21T13:57:59.031 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:59.031 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:59.032 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:59.032 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.032 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.032 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:14223 8.e 8:753b28d9:::benchmark_data_smithi049_109311_object14222:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-09-21T13:57:59.032 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:54:23.121191+0000", 2024-09-21T13:57:59.032 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 215.82270425499999, 2024-09-21T13:57:59.032 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.119117285, 2024-09-21T13:57:59.032 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:59.032 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:59.032 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:59.032 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:57:59.032 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:57:59.032 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:57:59.032 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 14223 2024-09-21T13:57:59.032 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.033 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:59.033 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.033 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:59.033 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.121191+0000", 2024-09-21T13:57:59.033 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:59.033 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.033 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.033 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:59.033 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.121191+0000", 2024-09-21T13:57:59.033 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:59.033 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.033 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.033 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:59.033 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.121193+0000", 2024-09-21T13:57:59.034 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.486e-06 2024-09-21T13:57:59.034 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.034 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.034 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:59.034 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.121215+0000", 2024-09-21T13:57:59.034 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2371e-05 2024-09-21T13:57:59.034 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.034 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.034 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:59.034 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.237516+0000", 2024-09-21T13:57:59.034 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.116301107 2024-09-21T13:57:59.034 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.034 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.034 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:59.034 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.237527+0000", 2024-09-21T13:57:59.035 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1479e-05 2024-09-21T13:57:59.035 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.035 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.035 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:59.035 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.237593+0000", 2024-09-21T13:57:59.035 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5359999999999998e-05 2024-09-21T13:57:59.035 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.035 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.035 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:57:59.035 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.237741+0000", 2024-09-21T13:57:59.035 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014798799999999999 2024-09-21T13:57:59.035 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.035 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.035 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,4", 2024-09-21T13:57:59.035 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.237862+0000", 2024-09-21T13:57:59.036 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000121057 2024-09-21T13:57:59.036 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.036 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.036 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:57:59.036 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.238729+0000", 2024-09-21T13:57:59.036 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00086727799999999997 2024-09-21T13:57:59.036 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.036 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.036 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:59.036 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.239905+0000", 2024-09-21T13:57:59.036 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0011758280000000001 2024-09-21T13:57:59.036 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.036 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.036 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:59.037 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.240185+0000", 2024-09-21T13:57:59.037 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00028034499999999999 2024-09-21T13:57:59.037 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.037 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.037 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:57:59.037 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.240266+0000", 2024-09-21T13:57:59.037 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0969000000000004e-05 2024-09-21T13:57:59.037 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.037 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.037 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:59.037 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.240308+0000", 2024-09-21T13:57:59.037 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2017000000000001e-05 2024-09-21T13:57:59.037 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:59.037 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:59.037 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:59.038 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.038 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.038 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:212087 8.c 8:37868233:::benchmark_data_smithi049_109311_object85568:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-09-21T13:57:59.038 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:56:55.643930+0000", 2024-09-21T13:57:59.038 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 63.299964969999998, 2024-09-21T13:57:59.038 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.107147015, 2024-09-21T13:57:59.038 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:59.038 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:59.038 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:59.038 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:57:59.038 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:57:59.038 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:57:59.038 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 212087 2024-09-21T13:57:59.038 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.038 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:59.039 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.039 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:59.039 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:56:55.643930+0000", 2024-09-21T13:57:59.039 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:59.039 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.039 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.039 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:59.039 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:56:55.643930+0000", 2024-09-21T13:57:59.039 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:59.039 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.039 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.039 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:59.039 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:56:55.643932+0000", 2024-09-21T13:57:59.039 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.418e-06 2024-09-21T13:57:59.039 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.040 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.040 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:59.040 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:56:55.643938+0000", 2024-09-21T13:57:59.040 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9349999999999999e-06 2024-09-21T13:57:59.040 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.040 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.040 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:59.040 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:56:55.643953+0000", 2024-09-21T13:57:59.040 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5277999999999998e-05 2024-09-21T13:57:59.040 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.040 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.040 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:59.040 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:56:55.643961+0000", 2024-09-21T13:57:59.040 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6129999999999998e-06 2024-09-21T13:57:59.040 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.041 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.041 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:59.041 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:56:55.644034+0000", 2024-09-21T13:57:59.041 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3248000000000001e-05 2024-09-21T13:57:59.041 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.041 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.041 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:57:59.041 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:56:55.644172+0000", 2024-09-21T13:57:59.041 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013812500000000001 2024-09-21T13:57:59.041 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.041 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.041 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 7", 2024-09-21T13:57:59.041 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:56:55.644305+0000", 2024-09-21T13:57:59.041 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013281400000000001 2024-09-21T13:57:59.041 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.042 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.042 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:57:59.042 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:56:55.648266+0000", 2024-09-21T13:57:59.042 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.003961366 2024-09-21T13:57:59.042 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.042 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.042 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:59.042 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:56:55.750852+0000", 2024-09-21T13:57:59.042 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.102585553 2024-09-21T13:57:59.042 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.042 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.042 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:57:59.042 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:56:55.750985+0000", 2024-09-21T13:57:59.042 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013286999999999999 2024-09-21T13:57:59.042 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.043 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.043 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:59.043 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:56:55.751077+0000", 2024-09-21T13:57:59.043 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2794999999999995e-05 2024-09-21T13:57:59.043 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:59.043 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:59.043 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:59.043 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.043 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.043 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.15207.0:232030 8.2 e239/183 8:49ac6d95:::benchmark_data_smithi049_109311_object105511:head v 239'14230, mlcod=239'14227)", 2024-09-21T13:57:59.043 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.654181+0000", 2024-09-21T13:57:59.043 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 47.289714644999997, 2024-09-21T13:57:59.043 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.227698072, 2024-09-21T13:57:59.043 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:59.043 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:59.044 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:59.044 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:59.044 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.044 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:59.044 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.654181+0000", 2024-09-21T13:57:59.044 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:59.044 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.044 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.044 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:59.044 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.654181+0000", 2024-09-21T13:57:59.044 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:59.044 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.044 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.044 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:59.044 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.654181+0000", 2024-09-21T13:57:59.045 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.44e-07 2024-09-21T13:57:59.045 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.045 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.045 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:59.045 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.654188+0000", 2024-09-21T13:57:59.045 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1080000000000004e-06 2024-09-21T13:57:59.045 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.045 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.045 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:59.045 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.654202+0000", 2024-09-21T13:57:59.045 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3283e-05 2024-09-21T13:57:59.045 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.045 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.045 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:59.045 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.654208+0000", 2024-09-21T13:57:59.046 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1179999999999997e-06 2024-09-21T13:57:59.046 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.046 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.046 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:59.046 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.654644+0000", 2024-09-21T13:57:59.046 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000436191 2024-09-21T13:57:59.046 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.046 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.046 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:57:59.046 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.654661+0000", 2024-09-21T13:57:59.046 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6623e-05 2024-09-21T13:57:59.046 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.046 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.046 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:57:59.047 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.881820+0000", 2024-09-21T13:57:59.047 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22715939499999999 2024-09-21T13:57:59.047 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.047 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.047 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:59.047 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.881879+0000", 2024-09-21T13:57:59.047 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8709999999999999e-05 2024-09-21T13:57:59.047 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:59.047 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:59.047 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:59.047 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.047 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.047 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232032 8.e 8:755f5137:::benchmark_data_smithi049_109311_object105513:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T13:57:59.047 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.656185+0000", 2024-09-21T13:57:59.048 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 47.287710591, 2024-09-21T13:57:59.048 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22610913499999999, 2024-09-21T13:57:59.048 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:59.048 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:59.048 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:59.048 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:57:59.048 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:57:59.048 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:57:59.048 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232032 2024-09-21T13:57:59.048 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.048 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:59.048 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:57:59.049 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.049 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:59.049 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.656185+0000", 2024-09-21T13:57:59.050 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:59.050 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.050 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.050 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:59.050 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.656185+0000", 2024-09-21T13:57:59.050 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:59.050 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.050 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.050 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:59.050 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.656187+0000", 2024-09-21T13:57:59.050 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.215e-06 2024-09-21T13:57:59.050 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.050 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.050 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:59.051 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.656193+0000", 2024-09-21T13:57:59.051 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8900000000000004e-06 2024-09-21T13:57:59.051 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.051 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.051 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:59.051 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.656207+0000", 2024-09-21T13:57:59.051 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3886999999999999e-05 2024-09-21T13:57:59.051 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.051 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.051 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:59.051 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.656214+0000", 2024-09-21T13:57:59.051 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6210000000000004e-06 2024-09-21T13:57:59.051 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.051 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.052 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:59.052 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.658622+0000", 2024-09-21T13:57:59.052 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0024080999999999998 2024-09-21T13:57:59.052 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.052 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.052 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:57:59.052 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.658725+0000", 2024-09-21T13:57:59.052 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000102497 2024-09-21T13:57:59.052 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.052 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.052 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,4", 2024-09-21T13:57:59.052 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.658827+0000", 2024-09-21T13:57:59.052 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000101846 2024-09-21T13:57:59.052 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.052 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.053 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:59.053 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.666078+0000", 2024-09-21T13:57:59.053 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0072517420000000003 2024-09-21T13:57:59.053 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.053 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.053 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:57:59.053 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.666142+0000", 2024-09-21T13:57:59.053 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3538000000000001e-05 2024-09-21T13:57:59.053 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.053 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.053 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:57:59.053 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.882193+0000", 2024-09-21T13:57:59.053 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21605080600000001 2024-09-21T13:57:59.053 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.053 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.054 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:57:59.054 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.882250+0000", 2024-09-21T13:57:59.054 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6773e-05 2024-09-21T13:57:59.054 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.054 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.054 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:59.054 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.882294+0000", 2024-09-21T13:57:59.054 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.422e-05 2024-09-21T13:57:59.054 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:59.054 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:59.054 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:59.054 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.054 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.054 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.15207.0:232036 8.2 e239/183 8:4507ec93:::benchmark_data_smithi049_109311_object105517:head v 239'14231, mlcod=239'14227)", 2024-09-21T13:57:59.055 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.656873+0000", 2024-09-21T13:57:59.055 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 47.287022069000002, 2024-09-21T13:57:59.055 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22504153599999999, 2024-09-21T13:57:59.055 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:59.055 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:59.055 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:57:59.055 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:59.055 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.055 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:59.055 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.656873+0000", 2024-09-21T13:57:59.055 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:59.055 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.055 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.055 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:59.055 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.656873+0000", 2024-09-21T13:57:59.056 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:59.056 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.056 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.056 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:59.056 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.656874+0000", 2024-09-21T13:57:59.056 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7e-07 2024-09-21T13:57:59.056 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.056 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.056 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:59.056 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.656877+0000", 2024-09-21T13:57:59.056 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4489999999999999e-06 2024-09-21T13:57:59.056 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.056 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.056 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:59.057 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.656887+0000", 2024-09-21T13:57:59.057 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9550000000000004e-06 2024-09-21T13:57:59.057 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.057 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.057 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:59.057 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.656892+0000", 2024-09-21T13:57:59.057 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6630000000000001e-06 2024-09-21T13:57:59.057 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.057 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.057 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:59.057 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.670143+0000", 2024-09-21T13:57:59.057 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.013251447 2024-09-21T13:57:59.057 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.057 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.057 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:57:59.058 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.670163+0000", 2024-09-21T13:57:59.058 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0185000000000001e-05 2024-09-21T13:57:59.058 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.058 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.058 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:57:59.058 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.881884+0000", 2024-09-21T13:57:59.058 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21172070400000001 2024-09-21T13:57:59.058 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.058 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.058 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:59.058 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.881915+0000", 2024-09-21T13:57:59.058 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.0762999999999997e-05 2024-09-21T13:57:59.058 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:59.058 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:59.059 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:59.059 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.059 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.059 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.15207.0:232032 8.e e240/183 ondisk, result = 0)", 2024-09-21T13:57:59.059 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.685964+0000", 2024-09-21T13:57:59.059 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 47.257931124000002, 2024-09-21T13:57:59.059 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.19633603699999999, 2024-09-21T13:57:59.059 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:57:59.059 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:57:59.059 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T13:57:59.059 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:57:59.059 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.059 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:57:59.059 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.685964+0000", 2024-09-21T13:57:59.059 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:59.060 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.060 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.060 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:57:59.060 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.685964+0000", 2024-09-21T13:57:59.060 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:57:59.060 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.060 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.060 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:57:59.060 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.685965+0000", 2024-09-21T13:57:59.060 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2890000000000001e-06 2024-09-21T13:57:59.060 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.060 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.060 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:57:59.060 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.685971+0000", 2024-09-21T13:57:59.061 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4280000000000004e-06 2024-09-21T13:57:59.061 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.061 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.061 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:57:59.061 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.881522+0000", 2024-09-21T13:57:59.061 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.19555102499999999 2024-09-21T13:57:59.061 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.061 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.061 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:57:59.061 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.881535+0000", 2024-09-21T13:57:59.061 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3157e-05 2024-09-21T13:57:59.061 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.061 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.061 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:57:59.061 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.882178+0000", 2024-09-21T13:57:59.062 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00064299300000000002 2024-09-21T13:57:59.062 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.062 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.062 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:57:59.062 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.882187+0000", 2024-09-21T13:57:59.062 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2790000000000004e-06 2024-09-21T13:57:59.062 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:57:59.062 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:57:59.062 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:57:59.062 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.882300+0000", 2024-09-21T13:57:59.062 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000112866 2024-09-21T13:57:59.062 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:59.062 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:59.062 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:59.062 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:57:59.063 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:57:59.063 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T13:57:59.085 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:59.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:59.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:57:59.187 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:59.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:59.209 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:59.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:59.310 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:57:59.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:59.333 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:59.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:57:59.434 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:57:59.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:59.457 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:59.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:57:59.558 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:57:59.579 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:59.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:59.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:57:59.681 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:57:59.704 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:59.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:59.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:57:59.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:59 smithi049 ceph-mon[30080]: pgmap v446: 148 pgs: 3 creating+activating, 145 active+clean; 583 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T13:57:59.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:59 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:57:59.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:59 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:57:59.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:57:59 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2549622329' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:57:59.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:59 smithi049 ceph-mon[40489]: pgmap v446: 148 pgs: 3 creating+activating, 145 active+clean; 583 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T13:57:59.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:59 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:57:59.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:59 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:57:59.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:57:59 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2549622329' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:57:59.806 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:57:59.828 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:57:59.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:59.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:57:59.930 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:57:59.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:59 smithi187 ceph-mon[33032]: pgmap v446: 148 pgs: 3 creating+activating, 145 active+clean; 583 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T13:57:59.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:59 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:57:59.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:59 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:57:59.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:57:59 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2549622329' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:57:59.945 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:57:59 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:57:59.953 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:57:59.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:57:59.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:58:00.054 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:00.077 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T13:58:00.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:00.078 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:00.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:00.179 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:00.201 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:00.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:00.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:00.265 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:58:00.265 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:58:00.265 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:58:00.265 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:58:00.265 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:58:00.266 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:58:00.266 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:58:00.266 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:58:00.266 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:58:00.266 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:58:00.266 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:58:00.266 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:58:00.266 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:58:00.267 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:58:00.267 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:58:00.267 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:58:00.302 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:58:00.325 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:00.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:00.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:58:00.427 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:58:00.450 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:00.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:00.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:58:00.552 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:00.576 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:00.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:00.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:00.678 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:58:00.702 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:00.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:00.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:58:00.804 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:58:00.826 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:00.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:00.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:58:00.927 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:58:00.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:00 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:58:00.950 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:00.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:00.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:58:00.997 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:00 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:58:00.997 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:00 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:58:00.998 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:58:00 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:58:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:58:01.052 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:01.078 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:01.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:01.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:01.180 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:58:01.206 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:01.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:01.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:58:01.307 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:01.330 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:01.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:01.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:01.432 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:01.454 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:01.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:01.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:01.556 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:01.579 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:01.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:01.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:01.680 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:01.684 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:58:01.685 INFO:tasks.thrashosds.thrasher:j is {'epoch': 262, 'fsid': '40a3d2e8-781f-11ef-baf6-efdc52797490', 'created': '2024-09-21T13:42:57.838275+0000', 'modified': '2024-09-21T13:57:54.209561+0000', 'last_up_change': '2024-09-21T13:57:11.653688+0000', 'last_in_change': '2024-09-21T13:55:33.181161+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 20, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 9, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T13:46:21.096244+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '199', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.8685569763183594, 'average_primary_affinity': 0.8700000047683716, 'average_primary_affinity_weighted': 0.8700000047683716}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-21T13:48:55.540712+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 63, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6700000762939453, 'score_stable': 2.6700000762939453, 'optimal_score': 0.75, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8685569763183594, 'average_primary_affinity': 0.8700000047683716, 'average_primary_affinity_weighted': 0.8700000047683716}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-21T13:48:56.417456+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 0.8685569763183594, 'average_primary_affinity': 0.8700000047683716, 'average_primary_affinity_weighted': 0.8700000047683716}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-21T13:48:58.128455+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.490000009536743, 'score_stable': 2.490000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.190000057220459, 'raw_score_stable': 2.190000057220459, 'primary_affinity_weighted': 0.8685569763183594, 'average_primary_affinity': 0.8700000047683716, 'average_primary_affinity_weighted': 0.8700000047683716}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-21T13:49:00.069361+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 0.8685569763183594, 'average_primary_affinity': 0.8700000047683716, 'average_primary_affinity_weighted': 0.8700000047683716}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-21T13:49:02.127399+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8685569763183594, 'average_primary_affinity': 0.8700000047683716, 'average_primary_affinity_weighted': 0.8700000047683716}}, {'pool': 9, 'pool_name': 'unique_pool_2', 'create_time': '2024-09-21T13:57:51.528024+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '262', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.990000009536743, 'score_stable': 2.990000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.630000114440918, 'raw_score_stable': 2.630000114440918, 'primary_affinity_weighted': 0.8685569763183594, 'average_primary_affinity': 0.8700000047683716, 'average_primary_affinity_weighted': 0.8700000047683716}}], 'osds': [{'osd': 0, 'uuid': '2595dca9-6219-4e0c-8728-751eb102e73a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.1215972900390625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 255, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6802', 'nonce': 2095561412}, {'type': 'v1', 'addr': '172.21.15.49:6803', 'nonce': 2095561412}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6804', 'nonce': 2095561412}, {'type': 'v1', 'addr': '172.21.15.49:6805', 'nonce': 2095561412}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6808', 'nonce': 2095561412}, {'type': 'v1', 'addr': '172.21.15.49:6809', 'nonce': 2095561412}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6806', 'nonce': 2095561412}, {'type': 'v1', 'addr': '172.21.15.49:6807', 'nonce': 2095561412}]}, 'public_addr': '172.21.15.49:6803/2095561412', 'cluster_addr': '172.21.15.49:6805/2095561412', 'heartbeat_back_addr': '172.21.15.49:6809/2095561412', 'heartbeat_front_addr': '172.21.15.49:6807/2095561412', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '0ba327f9-5524-4a40-abcc-729ded0b1917', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 260, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6810', 'nonce': 1382840527}, {'type': 'v1', 'addr': '172.21.15.49:6811', 'nonce': 1382840527}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6812', 'nonce': 1382840527}, {'type': 'v1', 'addr': '172.21.15.49:6813', 'nonce': 1382840527}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6816', 'nonce': 1382840527}, {'type': 'v1', 'addr': '172.21.15.49:6817', 'nonce': 1382840527}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6814', 'nonce': 1382840527}, {'type': 'v1', 'addr': '172.21.15.49:6815', 'nonce': 1382840527}]}, 'public_addr': '172.21.15.49:6811/1382840527', 'cluster_addr': '172.21.15.49:6813/1382840527', 'heartbeat_back_addr': '172.21.15.49:6817/1382840527', 'heartbeat_front_addr': '172.21.15.49:6815/1382840527', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '27a45750-204d-48fd-bb8f-ebb01273343d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.8268585205078125, 'last_clean_begin': 18, 'last_clean_end': 100, 'up_from': 130, 'up_thru': 240, 'down_at': 101, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6818', 'nonce': 1945314741}, {'type': 'v1', 'addr': '172.21.15.49:6819', 'nonce': 1945314741}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6820', 'nonce': 1945314741}, {'type': 'v1', 'addr': '172.21.15.49:6821', 'nonce': 1945314741}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6824', 'nonce': 1945314741}, {'type': 'v1', 'addr': '172.21.15.49:6825', 'nonce': 1945314741}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6822', 'nonce': 1945314741}, {'type': 'v1', 'addr': '172.21.15.49:6823', 'nonce': 1945314741}]}, 'public_addr': '172.21.15.49:6819/1945314741', 'cluster_addr': '172.21.15.49:6821/1945314741', 'heartbeat_back_addr': '172.21.15.49:6825/1945314741', 'heartbeat_front_addr': '172.21.15.49:6823/1945314741', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '528bc390-3dc9-4f3b-ab75-d7412c9625db', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 27, 'last_clean_end': 214, 'up_from': 240, 'up_thru': 260, 'down_at': 215, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6826', 'nonce': 886735721}, {'type': 'v1', 'addr': '172.21.15.49:6827', 'nonce': 886735721}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6828', 'nonce': 886735721}, {'type': 'v1', 'addr': '172.21.15.49:6829', 'nonce': 886735721}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6832', 'nonce': 886735721}, {'type': 'v1', 'addr': '172.21.15.49:6833', 'nonce': 886735721}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6830', 'nonce': 886735721}, {'type': 'v1', 'addr': '172.21.15.49:6831', 'nonce': 886735721}]}, 'public_addr': '172.21.15.49:6827/886735721', 'cluster_addr': '172.21.15.49:6829/886735721', 'heartbeat_back_addr': '172.21.15.49:6833/886735721', 'heartbeat_front_addr': '172.21.15.49:6831/886735721', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '6fc64698-2d37-4db0-bf3d-c7e612c45163', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 240, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6800', 'nonce': 1702447776}, {'type': 'v1', 'addr': '172.21.15.187:6801', 'nonce': 1702447776}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6802', 'nonce': 1702447776}, {'type': 'v1', 'addr': '172.21.15.187:6803', 'nonce': 1702447776}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6806', 'nonce': 1702447776}, {'type': 'v1', 'addr': '172.21.15.187:6807', 'nonce': 1702447776}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6804', 'nonce': 1702447776}, {'type': 'v1', 'addr': '172.21.15.187:6805', 'nonce': 1702447776}]}, 'public_addr': '172.21.15.187:6801/1702447776', 'cluster_addr': '172.21.15.187:6803/1702447776', 'heartbeat_back_addr': '172.21.15.187:6807/1702447776', 'heartbeat_front_addr': '172.21.15.187:6805/1702447776', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '8d7f62a1-7e92-4e06-9cad-86b96be514d8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 260, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6808', 'nonce': 3659050776}, {'type': 'v1', 'addr': '172.21.15.187:6809', 'nonce': 3659050776}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6810', 'nonce': 3659050776}, {'type': 'v1', 'addr': '172.21.15.187:6811', 'nonce': 3659050776}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6814', 'nonce': 3659050776}, {'type': 'v1', 'addr': '172.21.15.187:6815', 'nonce': 3659050776}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6812', 'nonce': 3659050776}, {'type': 'v1', 'addr': '172.21.15.187:6813', 'nonce': 3659050776}]}, 'public_addr': '172.21.15.187:6809/3659050776', 'cluster_addr': '172.21.15.187:6811/3659050776', 'heartbeat_back_addr': '172.21.15.187:6815/3659050776', 'heartbeat_front_addr': '172.21.15.187:6813/3659050776', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '3a7afd13-26ce-444c-9cc0-81ce2eab6ce5', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 183, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6816', 'nonce': 2728426525}, {'type': 'v1', 'addr': '172.21.15.187:6817', 'nonce': 2728426525}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6818', 'nonce': 2728426525}, {'type': 'v1', 'addr': '172.21.15.187:6819', 'nonce': 2728426525}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6822', 'nonce': 2728426525}, {'type': 'v1', 'addr': '172.21.15.187:6823', 'nonce': 2728426525}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6820', 'nonce': 2728426525}, {'type': 'v1', 'addr': '172.21.15.187:6821', 'nonce': 2728426525}]}, 'public_addr': '172.21.15.187:6817/2728426525', 'cluster_addr': '172.21.15.187:6819/2728426525', 'heartbeat_back_addr': '172.21.15.187:6823/2728426525', 'heartbeat_front_addr': '172.21.15.187:6821/2728426525', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1e08d891-8747-402e-b4c2-aa3802ce8537', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 260, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 3680850037}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 3680850037}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 3680850037}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 3680850037}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6830', 'nonce': 3680850037}, {'type': 'v1', 'addr': '172.21.15.187:6831', 'nonce': 3680850037}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6828', 'nonce': 3680850037}, {'type': 'v1', 'addr': '172.21.15.187:6829', 'nonce': 3680850037}]}, 'public_addr': '172.21.15.187:6825/3680850037', 'cluster_addr': '172.21.15.187:6827/3680850037', 'heartbeat_back_addr': '172.21.15.187:6831/3680850037', 'heartbeat_front_addr': '172.21.15.187:6829/3680850037', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T13:56:36.792777+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T13:57:48.533160+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T13:57:27.804766+0000', 'dead_epoch': 100}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T13:57:10.678447+0000', 'dead_epoch': 214}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T13:56:32.445570+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T13:57:29.881898+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-21T13:57:00.402108+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T13:56:59.703741+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '3.14', 'osds': [7, 6, 0]}], 'pg_upmap_items': [{'pgid': '4.6', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.1b', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1f', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.49:0/2731793917': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:0/799831425': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:6801/2518197280': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:0/3556829486': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:0/1249926115': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:0/231530138': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:0/4220781134': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:0/202325455': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:6800/4129636473': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:6800/2518197280': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:0/3514741273': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:6801/4129636473': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:0/3125743776': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:0/3600977363': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:6801/3453571598': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:6800/3453571598': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:0/3553734432': '2024-09-22T13:49:27.311313+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-21T13:58:01.685 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 4.1f 2024-09-21T13:58:01.686 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd rm-pg-upmap-items 4.1f 2024-09-21T13:58:01.686 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:58:01.709 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:01.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:01.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:01.717 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:58:01.811 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:01.832 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:01.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:01.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:01.877 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:01 smithi049 ceph-mon[40489]: pgmap v447: 148 pgs: 148 active+clean; 583 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T13:58:01.877 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:01 smithi049 ceph-mon[30080]: pgmap v447: 148 pgs: 148 active+clean; 583 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T13:58:01.934 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:01.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:01 smithi187 ceph-mon[33032]: pgmap v447: 148 pgs: 148 active+clean; 583 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-21T13:58:01.956 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:01.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:01.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:02.058 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:58:02.084 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:02.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:02.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:58:02.186 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:02.211 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:02.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:02.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:02.313 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:02.337 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:02.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:02.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:02.439 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:02.462 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:02.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:02.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:02.564 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:58:02.589 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:02.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:02.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:58:02.613 INFO:teuthology.orchestra.run.smithi049.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-09-21T13:58:02.691 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:02.715 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:02.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:02.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:02.736 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:02 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/238932007' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T13:58:02.736 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:02 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T13:58:02.736 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:02 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/238932007' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T13:58:02.736 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:02 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T13:58:02.818 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:58:02.843 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:02.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:02.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:58:02.944 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:02.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:02 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/238932007' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T13:58:02.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:02 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T13:58:02.969 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:02.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:02.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:03.031 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T13:58:03.071 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:58:03.096 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:03.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:03.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:58:03.198 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:58:03.223 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:03.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:03.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:58:03.324 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:03.349 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:03.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:03.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:03.451 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:03.476 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:03.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:03.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:03.577 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:58:03.601 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:03.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:03.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:58:03.703 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:58:03.729 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:03.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:03.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:58:03.793 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:58:03.796 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T13:58:03.797 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-09-21T13:58:03.797 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 0 2024-09-21T13:58:03.797 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T13:58:03.830 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:03.853 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:03.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:03.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:03.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:03 smithi187 ceph-mon[33032]: pgmap v448: 148 pgs: 148 active+clean; 583 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T13:58:03.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:03 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:58:03.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:03 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:58:03.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:03 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T13:58:03.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:03 smithi187 ceph-mon[33032]: osdmap e263: 8 total, 8 up, 7 in 2024-09-21T13:58:03.954 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:58:03.980 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:03.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:03.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:58:04.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:03 smithi049 ceph-mon[30080]: pgmap v448: 148 pgs: 148 active+clean; 583 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T13:58:04.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:03 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:58:04.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:03 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:58:04.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:03 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T13:58:04.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:03 smithi049 ceph-mon[30080]: osdmap e263: 8 total, 8 up, 7 in 2024-09-21T13:58:04.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:03 smithi049 ceph-mon[40489]: pgmap v448: 148 pgs: 148 active+clean; 583 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T13:58:04.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:03 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:58:04.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:03 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:58:04.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:03 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T13:58:04.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:03 smithi049 ceph-mon[40489]: osdmap e263: 8 total, 8 up, 7 in 2024-09-21T13:58:04.082 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:58:04.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:04.105 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:04.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:58:04.206 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:58:04.232 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:04.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:04.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:58:04.334 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:04.357 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:04.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:04.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:04.458 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:04.484 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:04.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:04.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:04.497 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T13:58:04.585 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:58:04.604 DEBUG:teuthology.orchestra.run.smithi049:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-09-21T13:58:04.606 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-21T13:58:04.606 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-21T13:58:04.611 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:04.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:04.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:58:04.663 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:hints = 1 2024-09-21T13:58:04.663 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-21T13:58:04.663 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Object prefix: benchmark_data_smithi049_138504 2024-09-21T13:58:04.664 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T13:58:04.664 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 0 0 0 0 0 0 - 0 2024-09-21T13:58:04.712 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:04.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:04 smithi049 ceph-mon[40489]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T13:58:04.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:04 smithi049 ceph-mon[30080]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T13:58:04.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:04.757 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:04.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:04.858 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:58:04.884 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:04.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:04.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:58:04.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:04 smithi187 ceph-mon[33032]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T13:58:04.986 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:05.043 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:05.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:05.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:05.145 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:58:05.188 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:05.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:05.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:58:05.289 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:58:05.324 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:05.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:05.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:58:05.426 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:58:05.457 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:05.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:05.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:58:05.505 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:58:05.559 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:05.580 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:58:05.620 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:05.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:05.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:05.664 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 1 16 3135 3119 194.848 194.938 0.00605327 0.00511165 2024-09-21T13:58:05.723 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:05.768 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:05.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:05.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:05.870 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:05.913 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:05.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:05.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:05.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:05 smithi187 ceph-mon[33032]: pgmap v450: 148 pgs: 148 active+clean; 583 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-21T13:58:05.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:05 smithi187 ceph-mon[33032]: osdmap e264: 8 total, 8 up, 7 in 2024-09-21T13:58:05.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:05 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/615858149' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:58:05.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:05 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:58:05.982 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:05 smithi049 ceph-mon[40489]: pgmap v450: 148 pgs: 148 active+clean; 583 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-21T13:58:05.982 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:05 smithi049 ceph-mon[40489]: osdmap e264: 8 total, 8 up, 7 in 2024-09-21T13:58:05.982 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:05 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/615858149' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:58:05.982 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:05 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:58:05.983 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:05 smithi049 ceph-mon[30080]: pgmap v450: 148 pgs: 148 active+clean; 583 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-21T13:58:05.983 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:05 smithi049 ceph-mon[30080]: osdmap e264: 8 total, 8 up, 7 in 2024-09-21T13:58:05.983 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:05 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/615858149' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:58:05.983 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:05 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:58:06.010 INFO:teuthology.orchestra.run.smithi049.stderr:clear 4.1f pg_upmap_items mapping 2024-09-21T13:58:06.015 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:06.020 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-09-21T13:58:06.054 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:06.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:06.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:06.157 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:58:06.189 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:06.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:06.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:58:06.291 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:58:06.324 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:06.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:06.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:58:06.425 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:06.448 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:06.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:06.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:06.550 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:58:06.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:06.591 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:06.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:58:06.605 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T13:58:06.605 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-09-21T13:58:06.605 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-09-21T13:58:06.605 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-09-21T13:58:06.605 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T13:58:06.664 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 2 16 6133 6117 191.106 187.375 0.00784565 0.00521403 2024-09-21T13:58:06.692 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:58:06.724 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:06.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:06.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:58:06.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:06 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/797643515' entity='client.admin' cmd={"prefix": "osd rm-pg-upmap-items", "pgid": "4.1f"} : dispatch 2024-09-21T13:58:06.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:06 smithi049 ceph-mon[40489]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:58:06.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:06 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T13:58:06.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:06 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/797643515' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.1f"}]': finished 2024-09-21T13:58:06.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:06 smithi049 ceph-mon[40489]: osdmap e265: 8 total, 8 up, 7 in 2024-09-21T13:58:06.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:06 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/797643515' entity='client.admin' cmd={"prefix": "osd rm-pg-upmap-items", "pgid": "4.1f"} : dispatch 2024-09-21T13:58:06.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:06 smithi049 ceph-mon[30080]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:58:06.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:06 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T13:58:06.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:06 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/797643515' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.1f"}]': finished 2024-09-21T13:58:06.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:06 smithi049 ceph-mon[30080]: osdmap e265: 8 total, 8 up, 7 in 2024-09-21T13:58:06.826 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:58:06.858 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:06.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:06.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:58:06.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:06 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/797643515' entity='client.admin' cmd={"prefix": "osd rm-pg-upmap-items", "pgid": "4.1f"} : dispatch 2024-09-21T13:58:06.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:06 smithi187 ceph-mon[33032]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:58:06.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:06 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T13:58:06.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:06 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/797643515' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.1f"}]': finished 2024-09-21T13:58:06.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:06 smithi187 ceph-mon[33032]: osdmap e265: 8 total, 8 up, 7 in 2024-09-21T13:58:06.959 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:06.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:06.999 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:06.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:07.100 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:58:07.131 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:07.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:07.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:58:07.233 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:58:07.234 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:58:07.235 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:58:07.235 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:58:07.235 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:58:07.235 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:58:07.235 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:58:07.235 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:58:07.235 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:58:07.251 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:58:07.251 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:58:07.264 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:07.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:07.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:58:07.288 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:58:07.288 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:58:07.322 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:58:07.322 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:58:07.357 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:58:07.357 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:58:07.366 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:58:07.395 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:07.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:07.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:58:07.496 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:07.530 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:07.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:07.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:07.632 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:07.665 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:07.665 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 3 15 9365 9350 194.751 202.062 0.00318337 0.00512706 2024-09-21T13:58:07.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:07.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:07.766 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:07.799 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:07.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:07.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:07.901 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:07.939 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:07.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:07.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:07.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:07 smithi187 ceph-mon[33032]: pgmap v452: 148 pgs: 148 active+clean; 53 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 6.6 MiB/s wr, 106 op/s 2024-09-21T13:58:07.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:07 smithi187 ceph-mon[33032]: 5.1a scrub starts 2024-09-21T13:58:07.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:07 smithi187 ceph-mon[33032]: 5.1a scrub ok 2024-09-21T13:58:07.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:07 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:58:07.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:07 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:58:07.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:07 smithi187 ceph-mon[33032]: osdmap e266: 8 total, 8 up, 7 in 2024-09-21T13:58:07.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:07 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3545425488' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:58:07.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:07 smithi187 ceph-mon[33032]: 4.6 scrub starts 2024-09-21T13:58:07.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:07 smithi187 ceph-mon[33032]: 4.6 scrub ok 2024-09-21T13:58:08.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:07 smithi049 ceph-mon[40489]: pgmap v452: 148 pgs: 148 active+clean; 53 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 6.6 MiB/s wr, 106 op/s 2024-09-21T13:58:08.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:07 smithi049 ceph-mon[40489]: 5.1a scrub starts 2024-09-21T13:58:08.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:07 smithi049 ceph-mon[40489]: 5.1a scrub ok 2024-09-21T13:58:08.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:07 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:58:08.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:07 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:58:08.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:07 smithi049 ceph-mon[40489]: osdmap e266: 8 total, 8 up, 7 in 2024-09-21T13:58:08.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:07 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3545425488' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:58:08.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:07 smithi049 ceph-mon[40489]: 4.6 scrub starts 2024-09-21T13:58:08.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:07 smithi049 ceph-mon[40489]: 4.6 scrub ok 2024-09-21T13:58:08.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:07 smithi049 ceph-mon[30080]: pgmap v452: 148 pgs: 148 active+clean; 53 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 6.6 MiB/s wr, 106 op/s 2024-09-21T13:58:08.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:07 smithi049 ceph-mon[30080]: 5.1a scrub starts 2024-09-21T13:58:08.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:07 smithi049 ceph-mon[30080]: 5.1a scrub ok 2024-09-21T13:58:08.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:07 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:58:08.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:07 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:58:08.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:07 smithi049 ceph-mon[30080]: osdmap e266: 8 total, 8 up, 7 in 2024-09-21T13:58:08.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:07 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3545425488' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:58:08.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:07 smithi049 ceph-mon[30080]: 4.6 scrub starts 2024-09-21T13:58:08.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:07 smithi049 ceph-mon[30080]: 4.6 scrub ok 2024-09-21T13:58:08.042 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:58:08.074 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:08.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:08.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:58:08.166 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T13:58:08.175 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:58:08.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:08.257 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:08.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:58:08.358 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:58:08.387 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:08.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:08.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:58:08.488 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:58:08.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:08.519 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:08.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:58:08.621 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:58:08.651 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:08.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:08.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:58:08.665 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 4 16 12106 12090 188.873 171.25 0.00662257 0.0052853 2024-09-21T13:58:08.752 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:58:08.783 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:08.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:08.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:58:08.886 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:58:08.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:08.941 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:08.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:58:08.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:08 smithi187 ceph-mon[33032]: 5.0 scrub starts 2024-09-21T13:58:08.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:08 smithi187 ceph-mon[33032]: 5.0 scrub ok 2024-09-21T13:58:08.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:08 smithi187 ceph-mon[33032]: 3.11 scrub starts 2024-09-21T13:58:08.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:08 smithi187 ceph-mon[33032]: 3.11 scrub ok 2024-09-21T13:58:08.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:08 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:58:08.998 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:08 smithi049 ceph-mon[40489]: 5.0 scrub starts 2024-09-21T13:58:08.998 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:08 smithi049 ceph-mon[40489]: 5.0 scrub ok 2024-09-21T13:58:08.998 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:08 smithi049 ceph-mon[40489]: 3.11 scrub starts 2024-09-21T13:58:08.998 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:08 smithi049 ceph-mon[40489]: 3.11 scrub ok 2024-09-21T13:58:08.998 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:08 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:58:08.998 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:08 smithi049 ceph-mon[30080]: 5.0 scrub starts 2024-09-21T13:58:08.998 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:08 smithi049 ceph-mon[30080]: 5.0 scrub ok 2024-09-21T13:58:08.999 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:08 smithi049 ceph-mon[30080]: 3.11 scrub starts 2024-09-21T13:58:08.999 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:08 smithi049 ceph-mon[30080]: 3.11 scrub ok 2024-09-21T13:58:08.999 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:08 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:58:09.042 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:09.070 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:09.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:09.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:09.172 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:09.282 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:09.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:09.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:09.385 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:58:09.414 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:09.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:09.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:58:09.516 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:58:09.548 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:09.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:09.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:58:09.650 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:58:09.666 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 5 16 14436 14420 180.19 145.625 0.0103126 0.00553559 2024-09-21T13:58:09.675 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:09.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:09.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:58:09.704 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:58:09.778 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:58:09.808 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:09.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:09.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:58:09.891 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:09 smithi049 ceph-mon[40489]: pgmap v455: 148 pgs: 148 active+clean; 53 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 971 B/s rd, 10 MiB/s wr, 160 op/s 2024-09-21T13:58:09.892 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:09 smithi049 ceph-mon[40489]: 3.0 scrub starts 2024-09-21T13:58:09.892 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:09 smithi049 ceph-mon[40489]: 3.0 scrub ok 2024-09-21T13:58:09.892 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:09 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:58:09.892 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:09 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:58:09.892 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:09 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:58:09.892 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:09 smithi049 ceph-mon[30080]: pgmap v455: 148 pgs: 148 active+clean; 53 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 971 B/s rd, 10 MiB/s wr, 160 op/s 2024-09-21T13:58:09.893 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:09 smithi049 ceph-mon[30080]: 3.0 scrub starts 2024-09-21T13:58:09.893 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:09 smithi049 ceph-mon[30080]: 3.0 scrub ok 2024-09-21T13:58:09.893 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:09 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:58:09.893 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:09 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:58:09.893 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:09 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:58:09.910 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:09.944 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:58:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:58:09.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:09 smithi187 ceph-mon[33032]: pgmap v455: 148 pgs: 148 active+clean; 53 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 971 B/s rd, 10 MiB/s wr, 160 op/s 2024-09-21T13:58:09.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:09 smithi187 ceph-mon[33032]: 3.0 scrub starts 2024-09-21T13:58:09.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:09 smithi187 ceph-mon[33032]: 3.0 scrub ok 2024-09-21T13:58:09.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:09 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:58:09.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:09 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:58:09.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:09 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:58:09.952 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:09.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:09.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:10.054 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:58:10.102 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:10.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:10.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:58:10.204 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:58:10.238 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:10.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:10.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:58:10.340 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:58:10.368 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:10.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:10.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:58:10.424 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T13:58:10.424 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:58:10.470 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:58:10.502 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:10.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:10.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:58:10.604 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:58:10.638 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:10.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:10.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:58:10.649 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T13:58:10.666 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 6 16 16842 16826 175.218 150.375 0.00745944 0.00569883 2024-09-21T13:58:10.740 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:10.771 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:10.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:10.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:10.872 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:58:10.902 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:10.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:10.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:58:11.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:10 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:58:11.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:10 smithi049 ceph-mon[40489]: 6.4 scrub starts 2024-09-21T13:58:11.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:10 smithi049 ceph-mon[40489]: 6.4 scrub ok 2024-09-21T13:58:11.005 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:58:10 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:58:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:58:11.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:10 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:58:11.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:10 smithi049 ceph-mon[30080]: 6.4 scrub starts 2024-09-21T13:58:11.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:10 smithi049 ceph-mon[30080]: 6.4 scrub ok 2024-09-21T13:58:11.006 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:11.054 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:11.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:11.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:11.120 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T13:58:11.156 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:11.186 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:11.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:11.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:11.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:10 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:58:11.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:10 smithi187 ceph-mon[33032]: 6.4 scrub starts 2024-09-21T13:58:11.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:10 smithi187 ceph-mon[33032]: 6.4 scrub ok 2024-09-21T13:58:11.288 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:58:11.317 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:11.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:11.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:58:11.335 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T13:58:11.419 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:58:11.453 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:11.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:11.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:58:11.470 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:58:11.555 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:58:11.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:11.593 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:11.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:58:11.666 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 7 16 19306 19290 172.186 154 0.00655834 0.0058015 2024-09-21T13:58:11.694 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:58:11.727 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:11.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:11.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:58:11.762 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:58:11.829 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:11.872 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:11.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:11.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:11.974 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:11.990 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:11 smithi049 ceph-mon[40489]: pgmap v456: 148 pgs: 148 active+clean; 627 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 852 B/s rd, 104 MiB/s wr, 1.67k op/s; 18 B/s, 0 objects/s recovering 2024-09-21T13:58:11.990 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:11 smithi049 ceph-mon[40489]: 5.6 scrub starts 2024-09-21T13:58:11.990 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:11 smithi049 ceph-mon[40489]: 5.6 scrub ok 2024-09-21T13:58:11.990 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:11 smithi049 ceph-mon[30080]: pgmap v456: 148 pgs: 148 active+clean; 627 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 852 B/s rd, 104 MiB/s wr, 1.67k op/s; 18 B/s, 0 objects/s recovering 2024-09-21T13:58:11.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:11 smithi049 ceph-mon[30080]: 5.6 scrub starts 2024-09-21T13:58:11.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:11 smithi049 ceph-mon[30080]: 5.6 scrub ok 2024-09-21T13:58:12.016 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:12.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:12.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:12.118 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:12.157 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:12.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:12.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:12.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:11 smithi187 ceph-mon[33032]: pgmap v456: 148 pgs: 148 active+clean; 627 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 852 B/s rd, 104 MiB/s wr, 1.67k op/s; 18 B/s, 0 objects/s recovering 2024-09-21T13:58:12.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:11 smithi187 ceph-mon[33032]: 5.6 scrub starts 2024-09-21T13:58:12.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:11 smithi187 ceph-mon[33032]: 5.6 scrub ok 2024-09-21T13:58:12.237 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:58:12.259 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:58:12.289 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:12.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:12.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:58:12.390 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:58:12.411 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:12.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:12.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:58:12.513 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:58:12.544 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:12.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:12.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:58:12.646 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:58:12.666 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 8 16 21402 21386 167.037 131 0.0055422 0.00598101 2024-09-21T13:58:12.679 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:12.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:12.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:58:12.732 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-09-21T13:58:12.781 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:58:12.820 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:12.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:12.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:58:12.898 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:58:12.898 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:58:12.923 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:58:12.941 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:58:12.941 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:58:12.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:12.955 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:12.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:58:12.974 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:58:12.974 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:58:12.975 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:12 smithi049 ceph-mon[40489]: 5.8 scrub starts 2024-09-21T13:58:12.975 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:12 smithi049 ceph-mon[40489]: 5.8 scrub ok 2024-09-21T13:58:12.975 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:12 smithi049 ceph-mon[40489]: 4.7 scrub starts 2024-09-21T13:58:12.975 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:12 smithi049 ceph-mon[40489]: 4.7 scrub ok 2024-09-21T13:58:12.975 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:12 smithi049 ceph-mon[40489]: 4.19 scrub starts 2024-09-21T13:58:12.975 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:12 smithi049 ceph-mon[40489]: 4.19 scrub ok 2024-09-21T13:58:12.975 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:12 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3508093566' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:58:12.975 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:12 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:58:12.975 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:12 smithi049 ceph-mon[30080]: 5.8 scrub starts 2024-09-21T13:58:12.975 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:12 smithi049 ceph-mon[30080]: 5.8 scrub ok 2024-09-21T13:58:12.975 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:12 smithi049 ceph-mon[30080]: 4.7 scrub starts 2024-09-21T13:58:12.976 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:12 smithi049 ceph-mon[30080]: 4.7 scrub ok 2024-09-21T13:58:12.976 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:12 smithi049 ceph-mon[30080]: 4.19 scrub starts 2024-09-21T13:58:12.976 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:12 smithi049 ceph-mon[30080]: 4.19 scrub ok 2024-09-21T13:58:12.976 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:12 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3508093566' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:58:12.976 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:12 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:58:13.001 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:12 smithi187 ceph-mon[33032]: 5.8 scrub starts 2024-09-21T13:58:13.001 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:12 smithi187 ceph-mon[33032]: 5.8 scrub ok 2024-09-21T13:58:13.001 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:12 smithi187 ceph-mon[33032]: 4.7 scrub starts 2024-09-21T13:58:13.001 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:12 smithi187 ceph-mon[33032]: 4.7 scrub ok 2024-09-21T13:58:13.002 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:12 smithi187 ceph-mon[33032]: 4.19 scrub starts 2024-09-21T13:58:13.002 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:12 smithi187 ceph-mon[33032]: 4.19 scrub ok 2024-09-21T13:58:13.002 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:12 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3508093566' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:58:13.002 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:12 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:58:13.011 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:58:13.012 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:58:13.046 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:58:13.046 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:58:13.056 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:13.082 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:58:13.082 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:58:13.085 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:13.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:13.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:13.118 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:58:13.118 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:58:13.153 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:58:13.153 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:58:13.187 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:13.227 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:13.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:13.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:13.329 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:13.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:13.366 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:13.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:13.467 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:13.505 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:13.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:13.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:13.607 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:58:13.635 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:13.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:13.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:58:13.666 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 9 15 23584 23569 163.636 136.438 0.00790746 0.00610635 2024-09-21T13:58:13.737 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:58:13.768 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:13.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:13.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:58:13.870 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:13.906 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:13.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:13.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:13.974 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:13 smithi187 ceph-mon[33032]: pgmap v457: 148 pgs: 148 active+clean; 839 MiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 743 B/s rd, 122 MiB/s wr, 1.95k op/s; 15 B/s, 0 objects/s recovering 2024-09-21T13:58:13.974 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:13 smithi187 ceph-mon[33032]: 4.a scrub starts 2024-09-21T13:58:13.975 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:13 smithi187 ceph-mon[33032]: 4.a scrub ok 2024-09-21T13:58:13.975 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:13 smithi187 ceph-mon[33032]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T13:58:13.975 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:13 smithi187 ceph-mon[33032]: Cluster is now healthy 2024-09-21T13:58:13.975 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:13 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T13:58:13.975 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:13 smithi187 ceph-mon[33032]: osdmap e267: 8 total, 8 up, 7 in 2024-09-21T13:58:13.975 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:13 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2566295328' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:58:13.975 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:13 smithi187 ceph-mon[33032]: 3.10 scrub starts 2024-09-21T13:58:13.975 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:13 smithi187 ceph-mon[33032]: 3.10 scrub ok 2024-09-21T13:58:13.975 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:13 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:58:13.975 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:13 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:58:14.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:13 smithi049 ceph-mon[40489]: pgmap v457: 148 pgs: 148 active+clean; 839 MiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 743 B/s rd, 122 MiB/s wr, 1.95k op/s; 15 B/s, 0 objects/s recovering 2024-09-21T13:58:14.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:13 smithi049 ceph-mon[40489]: 4.a scrub starts 2024-09-21T13:58:14.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:13 smithi049 ceph-mon[40489]: 4.a scrub ok 2024-09-21T13:58:14.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:13 smithi049 ceph-mon[40489]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T13:58:14.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:13 smithi049 ceph-mon[40489]: Cluster is now healthy 2024-09-21T13:58:14.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:13 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T13:58:14.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:13 smithi049 ceph-mon[40489]: osdmap e267: 8 total, 8 up, 7 in 2024-09-21T13:58:14.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:13 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2566295328' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:58:14.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:13 smithi049 ceph-mon[40489]: 3.10 scrub starts 2024-09-21T13:58:14.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:13 smithi049 ceph-mon[40489]: 3.10 scrub ok 2024-09-21T13:58:14.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:13 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:58:14.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:13 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:58:14.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:13 smithi049 ceph-mon[30080]: pgmap v457: 148 pgs: 148 active+clean; 839 MiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 743 B/s rd, 122 MiB/s wr, 1.95k op/s; 15 B/s, 0 objects/s recovering 2024-09-21T13:58:14.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:13 smithi049 ceph-mon[30080]: 4.a scrub starts 2024-09-21T13:58:14.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:13 smithi049 ceph-mon[30080]: 4.a scrub ok 2024-09-21T13:58:14.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:13 smithi049 ceph-mon[30080]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T13:58:14.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:13 smithi049 ceph-mon[30080]: Cluster is now healthy 2024-09-21T13:58:14.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:13 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T13:58:14.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:13 smithi049 ceph-mon[30080]: osdmap e267: 8 total, 8 up, 7 in 2024-09-21T13:58:14.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:13 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2566295328' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:58:14.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:13 smithi049 ceph-mon[30080]: 3.10 scrub starts 2024-09-21T13:58:14.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:13 smithi049 ceph-mon[30080]: 3.10 scrub ok 2024-09-21T13:58:14.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:13 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:58:14.007 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:13 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:58:14.008 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:14.045 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:14.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:14.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:14.147 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:14.184 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:14.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:14.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:14.286 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:58:14.294 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T13:58:14.295 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-21T13:58:14.295 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:14.295 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25421.0:24953 9.1 e267/260 9:89be3145:::benchmark_data_smithi049_138504_object24952:head v 267'1534, mlcod=267'1531)", 2024-09-21T13:58:14.295 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:58:14.288527+0000", 2024-09-21T13:58:14.295 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.00065541199999999997, 2024-09-21T13:58:14.295 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0006824, 2024-09-21T13:58:14.295 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:58:14.295 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:58:14.295 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T13:58:14.295 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:58:14.296 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:14.296 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:58:14.296 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:14.288527+0000", 2024-09-21T13:58:14.296 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:58:14.296 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:14.296 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:14.296 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:58:14.296 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:14.288527+0000", 2024-09-21T13:58:14.296 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:58:14.296 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:14.296 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:14.296 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:58:14.296 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:14.288527+0000", 2024-09-21T13:58:14.297 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9999999999999996e-07 2024-09-21T13:58:14.297 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:14.297 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:14.297 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:58:14.297 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:14.288559+0000", 2024-09-21T13:58:14.297 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1998000000000001e-05 2024-09-21T13:58:14.297 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:14.297 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:14.297 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:58:14.297 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:14.288577+0000", 2024-09-21T13:58:14.297 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7941e-05 2024-09-21T13:58:14.297 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:14.297 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:14.297 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:58:14.297 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:14.288585+0000", 2024-09-21T13:58:14.298 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2599999999999999e-06 2024-09-21T13:58:14.298 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:14.298 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:14.298 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:58:14.298 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:14.288633+0000", 2024-09-21T13:58:14.298 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8513000000000001e-05 2024-09-21T13:58:14.298 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:14.298 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:14.298 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:58:14.298 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:14.288653+0000", 2024-09-21T13:58:14.298 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0367999999999999e-05 2024-09-21T13:58:14.298 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:58:14.298 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:58:14.298 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:58:14.299 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:58:14.299 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2024-09-21T13:58:14.299 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 1 2024-09-21T13:58:14.299 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T13:58:14.327 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:14.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:14.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:58:14.430 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:58:14.462 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:14.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:14.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:58:14.564 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:58:14.596 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:14.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:14.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:58:14.666 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 10 15 25888 25873 161.671 144 0.00555311 0.0061794 2024-09-21T13:58:14.673 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [] live_osds: [0, 1, 4, 5, 6, 7, 2, 3] 2024-09-21T13:58:14.674 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T13:58:14.674 INFO:tasks.thrashosds.thrasher:Adding osd 6 2024-09-21T13:58:14.674 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd in 6 2024-09-21T13:58:14.698 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:14.761 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:14.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:14.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:14.862 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:14.903 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:14.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:14.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:14.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:14 smithi187 ceph-mon[33032]: 4.c scrub starts 2024-09-21T13:58:14.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:14 smithi187 ceph-mon[33032]: 4.c scrub ok 2024-09-21T13:58:14.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:14 smithi187 ceph-mon[33032]: 6.9 scrub starts 2024-09-21T13:58:14.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:14 smithi187 ceph-mon[33032]: 6.9 scrub ok 2024-09-21T13:58:14.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:14 smithi187 ceph-mon[33032]: 3.18 scrub starts 2024-09-21T13:58:14.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:14 smithi187 ceph-mon[33032]: 3.18 scrub ok 2024-09-21T13:58:15.005 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:58:15.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:14 smithi049 ceph-mon[30080]: 4.c scrub starts 2024-09-21T13:58:15.007 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:14 smithi049 ceph-mon[30080]: 4.c scrub ok 2024-09-21T13:58:15.007 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:14 smithi049 ceph-mon[30080]: 6.9 scrub starts 2024-09-21T13:58:15.007 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:14 smithi049 ceph-mon[30080]: 6.9 scrub ok 2024-09-21T13:58:15.007 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:14 smithi049 ceph-mon[30080]: 3.18 scrub starts 2024-09-21T13:58:15.007 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:14 smithi049 ceph-mon[30080]: 3.18 scrub ok 2024-09-21T13:58:15.007 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:14 smithi049 ceph-mon[40489]: 4.c scrub starts 2024-09-21T13:58:15.007 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:14 smithi049 ceph-mon[40489]: 4.c scrub ok 2024-09-21T13:58:15.007 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:14 smithi049 ceph-mon[40489]: 6.9 scrub starts 2024-09-21T13:58:15.007 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:14 smithi049 ceph-mon[40489]: 6.9 scrub ok 2024-09-21T13:58:15.007 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:14 smithi049 ceph-mon[40489]: 3.18 scrub starts 2024-09-21T13:58:15.008 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:14 smithi049 ceph-mon[40489]: 3.18 scrub ok 2024-09-21T13:58:15.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:15.040 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:15.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:58:15.129 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:58:15.141 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:15.143 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T13:58:15.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:15.189 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:15.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:15.290 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:15.327 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:15.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:15.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:15.429 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:15.477 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:15.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:15.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:15.544 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:58:15.578 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:58:15.624 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:15.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:15.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:58:15.666 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 11 16 28062 28046 159.32 135.812 0.00397846 0.00627078 2024-09-21T13:58:15.726 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:15.762 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:15.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:15.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:15.786 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T13:58:15.864 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:58:15.895 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:15.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:15.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:58:15.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:15 smithi187 ceph-mon[33032]: pgmap v459: 148 pgs: 148 active+clean; 1.3 GiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 1.3 KiB/s rd, 156 MiB/s wr, 2.50k op/s; 13 B/s, 0 objects/s recovering 2024-09-21T13:58:15.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:15 smithi187 ceph-mon[33032]: 6.5 scrub starts 2024-09-21T13:58:15.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:15 smithi187 ceph-mon[33032]: 6.5 scrub ok 2024-09-21T13:58:15.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:15 smithi187 ceph-mon[33032]: osdmap e268: 8 total, 8 up, 7 in 2024-09-21T13:58:15.997 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:16.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:15 smithi049 ceph-mon[30080]: pgmap v459: 148 pgs: 148 active+clean; 1.3 GiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 1.3 KiB/s rd, 156 MiB/s wr, 2.50k op/s; 13 B/s, 0 objects/s recovering 2024-09-21T13:58:16.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:15 smithi049 ceph-mon[30080]: 6.5 scrub starts 2024-09-21T13:58:16.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:15 smithi049 ceph-mon[30080]: 6.5 scrub ok 2024-09-21T13:58:16.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:15 smithi049 ceph-mon[30080]: osdmap e268: 8 total, 8 up, 7 in 2024-09-21T13:58:16.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:15 smithi049 ceph-mon[40489]: pgmap v459: 148 pgs: 148 active+clean; 1.3 GiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 1.3 KiB/s rd, 156 MiB/s wr, 2.50k op/s; 13 B/s, 0 objects/s recovering 2024-09-21T13:58:16.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:15 smithi049 ceph-mon[40489]: 6.5 scrub starts 2024-09-21T13:58:16.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:15 smithi049 ceph-mon[40489]: 6.5 scrub ok 2024-09-21T13:58:16.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:15 smithi049 ceph-mon[40489]: osdmap e268: 8 total, 8 up, 7 in 2024-09-21T13:58:16.051 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:16.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:16.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:16.153 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:58:16.179 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:16.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:16.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:58:16.210 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:58:16.281 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:58:16.308 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:16.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:16.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:58:16.410 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:58:16.444 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:16.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:16.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:58:16.546 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:16.596 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:16.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:16.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:16.613 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T13:58:16.651 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:16 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mon-a[30057]: 2024-09-21T13:58:16.492+0000 7fc8b704c640 -1 mon.a@0(leader).osd e268 definitely_dead 0 2024-09-21T13:58:16.667 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 12 16 29833 29817 155.267 110.688 0.00535193 0.00643461 2024-09-21T13:58:16.697 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:58:16.726 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:16.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:16.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:58:16.747 INFO:teuthology.orchestra.run.smithi049.stderr:marked in osd.6. 2024-09-21T13:58:16.828 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:58:16.854 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:16.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:16.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:58:16.956 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:16.978 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:16.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:16.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:17.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:16 smithi049 ceph-mon[30080]: 5.4 scrub starts 2024-09-21T13:58:17.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:16 smithi049 ceph-mon[30080]: 5.4 scrub ok 2024-09-21T13:58:17.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:16 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1703948717' entity='client.admin' cmd={"prefix": "osd in", "ids": ["6"]} : dispatch 2024-09-21T13:58:17.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:16 smithi049 ceph-mon[40489]: 5.4 scrub starts 2024-09-21T13:58:17.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:16 smithi049 ceph-mon[40489]: 5.4 scrub ok 2024-09-21T13:58:17.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:16 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1703948717' entity='client.admin' cmd={"prefix": "osd in", "ids": ["6"]} : dispatch 2024-09-21T13:58:17.079 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:17.101 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:17.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:17.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:17.116 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:16 smithi187 ceph-mon[33032]: 5.4 scrub starts 2024-09-21T13:58:17.116 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:16 smithi187 ceph-mon[33032]: 5.4 scrub ok 2024-09-21T13:58:17.116 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:16 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1703948717' entity='client.admin' cmd={"prefix": "osd in", "ids": ["6"]} : dispatch 2024-09-21T13:58:17.204 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:17.226 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:17.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:17.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:17.328 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:17.351 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:17.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:17.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:17.452 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:58:17.473 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:17.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:17.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:58:17.575 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:58:17.597 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:17.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:17.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:58:17.645 INFO:tasks.thrashosds.thrasher:Added osd 6 2024-09-21T13:58:17.666 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 13 16 30055 30039 144.391 13.875 0.00251924 0.00645698 2024-09-21T13:58:17.698 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:58:17.706 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:58:17.719 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:17.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:17.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:58:17.747 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-09-21T13:58:17.821 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:58:17.842 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:17.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:17.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:58:17.943 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:58:17.958 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T13:58:17.959 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-21T13:58:17.959 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-21T13:58:17.959 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-21T13:58:17.959 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T13:58:17.965 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:17.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:17.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:58:18.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:17 smithi049 ceph-mon[40489]: pgmap v461: 148 pgs: 148 active+clean; 1.5 GiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 187 MiB/s wr, 2.99k op/s; 13 B/s, 0 objects/s recovering 2024-09-21T13:58:18.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:17 smithi049 ceph-mon[40489]: 5.1c scrub starts 2024-09-21T13:58:18.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:17 smithi049 ceph-mon[40489]: 5.1c scrub ok 2024-09-21T13:58:18.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:17 smithi049 ceph-mon[40489]: 3.17 scrub starts 2024-09-21T13:58:18.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:17 smithi049 ceph-mon[40489]: 3.17 scrub ok 2024-09-21T13:58:18.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:17 smithi049 ceph-mon[40489]: 3.4 scrub starts 2024-09-21T13:58:18.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:17 smithi049 ceph-mon[40489]: 3.4 scrub ok 2024-09-21T13:58:18.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:17 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1703948717' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-09-21T13:58:18.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:17 smithi049 ceph-mon[40489]: osdmap e269: 8 total, 8 up, 8 in 2024-09-21T13:58:18.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:17 smithi049 ceph-mon[40489]: 6.6 scrub starts 2024-09-21T13:58:18.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:17 smithi049 ceph-mon[40489]: 6.6 scrub ok 2024-09-21T13:58:18.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:17 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1662138536' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:58:18.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:17 smithi049 ceph-mon[30080]: pgmap v461: 148 pgs: 148 active+clean; 1.5 GiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 187 MiB/s wr, 2.99k op/s; 13 B/s, 0 objects/s recovering 2024-09-21T13:58:18.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:17 smithi049 ceph-mon[30080]: 5.1c scrub starts 2024-09-21T13:58:18.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:17 smithi049 ceph-mon[30080]: 5.1c scrub ok 2024-09-21T13:58:18.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:17 smithi049 ceph-mon[30080]: 3.17 scrub starts 2024-09-21T13:58:18.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:17 smithi049 ceph-mon[30080]: 3.17 scrub ok 2024-09-21T13:58:18.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:17 smithi049 ceph-mon[30080]: 3.4 scrub starts 2024-09-21T13:58:18.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:17 smithi049 ceph-mon[30080]: 3.4 scrub ok 2024-09-21T13:58:18.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:17 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1703948717' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-09-21T13:58:18.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:17 smithi049 ceph-mon[30080]: osdmap e269: 8 total, 8 up, 8 in 2024-09-21T13:58:18.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:17 smithi049 ceph-mon[30080]: 6.6 scrub starts 2024-09-21T13:58:18.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:17 smithi049 ceph-mon[30080]: 6.6 scrub ok 2024-09-21T13:58:18.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:17 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1662138536' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:58:18.067 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:58:18.088 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:18.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:18.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:58:18.190 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:58:18.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:17 smithi187 ceph-mon[33032]: pgmap v461: 148 pgs: 148 active+clean; 1.5 GiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 187 MiB/s wr, 2.99k op/s; 13 B/s, 0 objects/s recovering 2024-09-21T13:58:18.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:17 smithi187 ceph-mon[33032]: 5.1c scrub starts 2024-09-21T13:58:18.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:17 smithi187 ceph-mon[33032]: 5.1c scrub ok 2024-09-21T13:58:18.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:17 smithi187 ceph-mon[33032]: 3.17 scrub starts 2024-09-21T13:58:18.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:17 smithi187 ceph-mon[33032]: 3.17 scrub ok 2024-09-21T13:58:18.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:17 smithi187 ceph-mon[33032]: 3.4 scrub starts 2024-09-21T13:58:18.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:17 smithi187 ceph-mon[33032]: 3.4 scrub ok 2024-09-21T13:58:18.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:17 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1703948717' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-09-21T13:58:18.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:17 smithi187 ceph-mon[33032]: osdmap e269: 8 total, 8 up, 8 in 2024-09-21T13:58:18.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:17 smithi187 ceph-mon[33032]: 6.6 scrub starts 2024-09-21T13:58:18.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:17 smithi187 ceph-mon[33032]: 6.6 scrub ok 2024-09-21T13:58:18.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:17 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1662138536' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:58:18.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:18.212 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:18.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:58:18.313 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:18.336 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:18.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:18.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:18.438 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:58:18.460 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:18.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:18.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:58:18.562 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:18.583 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:18.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:18.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:18.667 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 14 16 30055 30039 134.078 0 - 0.00645698 2024-09-21T13:58:18.685 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:18.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:18.707 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:18.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:18.808 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:18.832 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:18.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:18.833 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T13:58:18.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:18.934 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:58:18.969 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:18.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:18.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:58:19.029 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:18 smithi049 ceph-mon[40489]: 3.1b scrub starts 2024-09-21T13:58:19.029 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:18 smithi049 ceph-mon[40489]: 3.1b scrub ok 2024-09-21T13:58:19.030 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:18 smithi049 ceph-mon[40489]: 3.15 scrub starts 2024-09-21T13:58:19.030 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:18 smithi049 ceph-mon[40489]: 3.15 scrub ok 2024-09-21T13:58:19.030 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:18 smithi049 ceph-mon[40489]: 6.1d scrub starts 2024-09-21T13:58:19.030 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:18 smithi049 ceph-mon[40489]: 6.1d scrub ok 2024-09-21T13:58:19.030 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:18 smithi049 ceph-mon[40489]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:58:19.030 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:18 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1662138536' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T13:58:19.030 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:18 smithi049 ceph-mon[40489]: osdmap e270: 8 total, 8 up, 8 in 2024-09-21T13:58:19.031 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:18 smithi049 ceph-mon[40489]: osdmap e271: 8 total, 8 up, 8 in 2024-09-21T13:58:19.031 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:18 smithi049 ceph-mon[30080]: 3.1b scrub starts 2024-09-21T13:58:19.031 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:18 smithi049 ceph-mon[30080]: 3.1b scrub ok 2024-09-21T13:58:19.031 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:18 smithi049 ceph-mon[30080]: 3.15 scrub starts 2024-09-21T13:58:19.031 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:18 smithi049 ceph-mon[30080]: 3.15 scrub ok 2024-09-21T13:58:19.032 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:18 smithi049 ceph-mon[30080]: 6.1d scrub starts 2024-09-21T13:58:19.032 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:18 smithi049 ceph-mon[30080]: 6.1d scrub ok 2024-09-21T13:58:19.032 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:18 smithi049 ceph-mon[30080]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:58:19.032 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:18 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1662138536' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T13:58:19.032 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:18 smithi049 ceph-mon[30080]: osdmap e270: 8 total, 8 up, 8 in 2024-09-21T13:58:19.032 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:18 smithi049 ceph-mon[30080]: osdmap e271: 8 total, 8 up, 8 in 2024-09-21T13:58:19.071 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:19.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:19.110 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:19.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:19.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:18 smithi187 ceph-mon[33032]: 3.1b scrub starts 2024-09-21T13:58:19.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:18 smithi187 ceph-mon[33032]: 3.1b scrub ok 2024-09-21T13:58:19.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:18 smithi187 ceph-mon[33032]: 3.15 scrub starts 2024-09-21T13:58:19.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:18 smithi187 ceph-mon[33032]: 3.15 scrub ok 2024-09-21T13:58:19.196 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:18 smithi187 ceph-mon[33032]: 6.1d scrub starts 2024-09-21T13:58:19.196 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:18 smithi187 ceph-mon[33032]: 6.1d scrub ok 2024-09-21T13:58:19.196 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:18 smithi187 ceph-mon[33032]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:58:19.196 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:18 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1662138536' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T13:58:19.196 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:18 smithi187 ceph-mon[33032]: osdmap e270: 8 total, 8 up, 8 in 2024-09-21T13:58:19.196 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:18 smithi187 ceph-mon[33032]: osdmap e271: 8 total, 8 up, 8 in 2024-09-21T13:58:19.204 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:58:19.211 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:58:19.255 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:19.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:19.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:58:19.357 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:19.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:19.386 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:19.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:19.488 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:19.519 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:19.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:19.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:19.620 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:58:19.654 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:19.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:19.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:58:19.667 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 15 16 31102 31086 129.502 32.7188 0.0102963 0.00771651 2024-09-21T13:58:19.694 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:58:19 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:58:19.756 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:19.791 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:19.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:19.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:19.893 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:19.932 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:19.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:19.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:20.001 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:19 smithi187 ceph-mon[33032]: pgmap v464: 148 pgs: 148 active+clean; 1.5 GiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 992 B/s rd, 138 MiB/s wr, 2.21k op/s 2024-09-21T13:58:20.002 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:19 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2191246212' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:58:20.034 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:20.070 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:20.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:20.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:20.132 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:58:20.133 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:58:20.133 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:58:20.133 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:58:20.133 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:58:20.133 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:58:20.133 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:58:20.134 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:58:20.134 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:58:20.134 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:58:20.134 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:58:20.134 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:58:20.135 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:58:20.135 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:58:20.135 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:58:20.135 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:58:20.136 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:19 smithi049 ceph-mon[30080]: pgmap v464: 148 pgs: 148 active+clean; 1.5 GiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 992 B/s rd, 138 MiB/s wr, 2.21k op/s 2024-09-21T13:58:20.136 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:19 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2191246212' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:58:20.136 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:19 smithi049 ceph-mon[40489]: pgmap v464: 148 pgs: 148 active+clean; 1.5 GiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 992 B/s rd, 138 MiB/s wr, 2.21k op/s 2024-09-21T13:58:20.136 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:19 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2191246212' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:58:20.161 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T13:58:20.161 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:58:20.172 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:20.209 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:20.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:20.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:20.311 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:20.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:20.350 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:20.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:20.452 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:20.487 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:20.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:20.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:20.589 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:58:20.621 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:20.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:20.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:58:20.667 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 16 16 32772 32756 127.931 104.375 0.0140394 0.00781054 2024-09-21T13:58:20.724 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:58:20.762 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:20.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:20.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:58:20.864 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:58:20.867 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:20 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:58:20.867 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:20 smithi187 ceph-mon[33032]: osdmap e272: 8 total, 8 up, 8 in 2024-09-21T13:58:20.902 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:20.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:20.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:58:20.923 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T13:58:21.004 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:58:21.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:20 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:58:21.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:20 smithi049 ceph-mon[40489]: osdmap e272: 8 total, 8 up, 8 in 2024-09-21T13:58:21.005 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:58:20 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:58:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:58:21.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:20 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:58:21.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:20 smithi049 ceph-mon[30080]: osdmap e272: 8 total, 8 up, 8 in 2024-09-21T13:58:21.044 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:21.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:21.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:58:21.146 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:21.180 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:21.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:21.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:21.282 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:21.320 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:21.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:21.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:21.342 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:58:21.422 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:21.454 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:21.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:21.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:21.556 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:58:21.596 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:21.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:21.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:58:21.667 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 17 16 34361 34345 126.247 99.3125 0.00672998 0.00791368 2024-09-21T13:58:21.698 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:58:21.733 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:21.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:21.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:58:21.834 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:58:21.868 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:21.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:21.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:58:21.970 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:21.980 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T13:58:21.980 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:58:22.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:22.011 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:22.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:22.112 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:22.149 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:22.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:22.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:22.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:21 smithi187 ceph-mon[33032]: pgmap v466: 148 pgs: 2 active+recovering+undersized+remapped, 146 active+clean; 1.8 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 99 MiB/s wr, 1.59k op/s; 3787/89685 objects misplaced (4.223%); 498 KiB/s, 15 objects/s recovering 2024-09-21T13:58:22.206 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:21 smithi049 ceph-mon[40489]: pgmap v466: 148 pgs: 2 active+recovering+undersized+remapped, 146 active+clean; 1.8 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 99 MiB/s wr, 1.59k op/s; 3787/89685 objects misplaced (4.223%); 498 KiB/s, 15 objects/s recovering 2024-09-21T13:58:22.206 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:21 smithi049 ceph-mon[30080]: pgmap v466: 148 pgs: 2 active+recovering+undersized+remapped, 146 active+clean; 1.8 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 99 MiB/s wr, 1.59k op/s; 3787/89685 objects misplaced (4.223%); 498 KiB/s, 15 objects/s recovering 2024-09-21T13:58:22.251 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:22.295 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:22.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:22.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:22.397 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:58:22.432 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:22.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:22.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:58:22.534 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:22.571 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:22.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:22.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:22.646 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 4, 5, 6, 7, 2, 3] 2024-09-21T13:58:22.647 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T13:58:22.649 INFO:tasks.thrashosds.thrasher:Killing osd 7, live_osds are [0, 1, 4, 5, 6, 7, 2, 3] 2024-09-21T13:58:22.649 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-21T13:58:22.650 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@osd.7 2024-09-21T13:58:22.667 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 18 15 36003 35988 124.938 102.688 0.00896907 0.00799714 2024-09-21T13:58:22.673 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:58:22.714 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:22.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:22.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:58:22.817 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:22.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:22.854 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:22.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:22.919 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T13:58:22.944 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 13:58:22 smithi187 systemd[1]: Stopping Ceph osd.7 for 40a3d2e8-781f-11ef-baf6-efdc52797490... 2024-09-21T13:58:22.955 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:58:23.001 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:23.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:23.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:58:23.103 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:23.144 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:23.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:23.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:23.246 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:23.281 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 13:58:23 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[55522]: 2024-09-21T13:58:23.278+0000 7fadfe938640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T13:58:23.281 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 13:58:23 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[55522]: 2024-09-21T13:58:23.278+0000 7fadfe938640 -1 osd.7 272 *** Got signal Terminated *** 2024-09-21T13:58:23.281 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 13:58:23 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[55522]: 2024-09-21T13:58:23.278+0000 7fadfe938640 -1 osd.7 272 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T13:58:23.288 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:23.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:23.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:23.389 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:23.392 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:58:23.436 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:23.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:23.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:23.538 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:58:23.568 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:23.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:23.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:58:23.670 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 19 15 37596 37581 123.602 99.5625 0.00363976 0.00808392 2024-09-21T13:58:23.671 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:58:23.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:23.706 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:23.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:58:23.807 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:58:23.832 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:23.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:23.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:58:23.933 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:58:23.957 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:23.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:23.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:58:23.989 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T13:58:23.989 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:58:24.058 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:58:24.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:24.083 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:24.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:58:24.183 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:24.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:23 smithi187 ceph-mon[33032]: pgmap v468: 148 pgs: 3 active+recovering+undersized+remapped, 145 active+clean; 1.9 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 997 B/s rd, 74 MiB/s wr, 1.18k op/s; 5509/93111 objects misplaced (5.917%); 1.9 MiB/s, 38 objects/s recovering 2024-09-21T13:58:24.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:23 smithi187 ceph-mon[33032]: osd.7 marked itself down and dead 2024-09-21T13:58:24.207 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:24.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:24.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:24.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:23 smithi049 ceph-mon[40489]: pgmap v468: 148 pgs: 3 active+recovering+undersized+remapped, 145 active+clean; 1.9 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 997 B/s rd, 74 MiB/s wr, 1.18k op/s; 5509/93111 objects misplaced (5.917%); 1.9 MiB/s, 38 objects/s recovering 2024-09-21T13:58:24.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:23 smithi049 ceph-mon[40489]: osd.7 marked itself down and dead 2024-09-21T13:58:24.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:23 smithi049 ceph-mon[30080]: pgmap v468: 148 pgs: 3 active+recovering+undersized+remapped, 145 active+clean; 1.9 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 997 B/s rd, 74 MiB/s wr, 1.18k op/s; 5509/93111 objects misplaced (5.917%); 1.9 MiB/s, 38 objects/s recovering 2024-09-21T13:58:24.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:23 smithi049 ceph-mon[30080]: osd.7 marked itself down and dead 2024-09-21T13:58:24.309 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:58:24.333 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:24.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:24.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:58:24.434 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:24.458 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:24.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:24.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:24.560 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:58:24.582 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:24.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:24.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:58:24.666 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T13:58:24.667 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:2024-09-21T13:58:24.668081+0000 min lat: 0.00125866 max lat: 2.20809 avg lat: 0.00809926 2024-09-21T13:58:24.667 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T13:58:24.667 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 20 16 37807 37791 118.079 13.125 0.00317902 0.00809926 2024-09-21T13:58:24.683 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:58:24.734 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:24.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:24.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:58:24.836 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:24.867 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T13:58:24.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:24.888 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:24.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:24.989 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:25.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:24 smithi049 ceph-mon[30080]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T13:58:25.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:24 smithi049 ceph-mon[30080]: osdmap e273: 8 total, 7 up, 8 in 2024-09-21T13:58:25.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:24 smithi049 ceph-mon[30080]: osdmap e274: 8 total, 7 up, 8 in 2024-09-21T13:58:25.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:24 smithi049 ceph-mon[40489]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T13:58:25.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:24 smithi049 ceph-mon[40489]: osdmap e273: 8 total, 7 up, 8 in 2024-09-21T13:58:25.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:24 smithi049 ceph-mon[40489]: osdmap e274: 8 total, 7 up, 8 in 2024-09-21T13:58:25.014 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:25.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:25.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:25.048 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:58:25.116 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:25.147 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:25.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:25.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:25.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:24 smithi187 ceph-mon[33032]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T13:58:25.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:24 smithi187 ceph-mon[33032]: osdmap e273: 8 total, 7 up, 8 in 2024-09-21T13:58:25.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:24 smithi187 ceph-mon[33032]: osdmap e274: 8 total, 7 up, 8 in 2024-09-21T13:58:25.195 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 13:58:24 smithi187 podman[107101]: 2024-09-21 13:58:24.717921166 +0000 UTC m=+1.582112246 container died f8c61e8297eded67def0b81403249a9ec46710bf70d5364296ef2e1456600eb7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, RELEASE=wip-guits-main-2024-09-21-0842-4417849, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-09-21T13:58:25.248 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:25.272 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:25.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:25.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:25.373 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:25.397 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:25.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:25.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:25.499 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:58:25.523 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:25.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:25.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:58:25.625 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:25.649 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:25.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:25.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:25.667 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 21 16 38055 38039 113.194 15.5 0.00270234 0.00847303 2024-09-21T13:58:25.751 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:58:25.764 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 13:58:25 smithi187 podman[107101]: 2024-09-21 13:58:25.605601359 +0000 UTC m=+2.469792427 container cleanup f8c61e8297eded67def0b81403249a9ec46710bf70d5364296ef2e1456600eb7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, ceph=True) 2024-09-21T13:58:25.764 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 13:58:25 smithi187 bash[107101]: ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7 2024-09-21T13:58:25.775 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:25.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:25.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:58:25.802 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T13:58:25.877 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:25.900 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:25.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:25.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:26.002 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:26.027 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:26.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:26.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:26.076 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:58:26.129 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:26.165 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:26.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:26.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:26.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:25 smithi187 ceph-mon[33032]: pgmap v470: 148 pgs: 28 stale+active+clean, 3 active+recovering+undersized+remapped, 117 active+clean; 2.1 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.7 KiB/s rd, 106 MiB/s wr, 1.70k op/s; 5509/105594 objects misplaced (5.217%); 1.6 MiB/s, 33 objects/s recovering 2024-09-21T13:58:26.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:25 smithi187 ceph-mon[33032]: osdmap e275: 8 total, 7 up, 8 in 2024-09-21T13:58:26.195 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 13:58:26 smithi187 podman[107313]: 2024-09-21 13:58:26.008144326 +0000 UTC m=+1.612736986 container remove f8c61e8297eded67def0b81403249a9ec46710bf70d5364296ef2e1456600eb7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-guits-main-2024-09-21-0842-4417849, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T13:58:26.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:25 smithi049 ceph-mon[40489]: pgmap v470: 148 pgs: 28 stale+active+clean, 3 active+recovering+undersized+remapped, 117 active+clean; 2.1 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.7 KiB/s rd, 106 MiB/s wr, 1.70k op/s; 5509/105594 objects misplaced (5.217%); 1.6 MiB/s, 33 objects/s recovering 2024-09-21T13:58:26.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:25 smithi049 ceph-mon[40489]: osdmap e275: 8 total, 7 up, 8 in 2024-09-21T13:58:26.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:25 smithi049 ceph-mon[30080]: pgmap v470: 148 pgs: 28 stale+active+clean, 3 active+recovering+undersized+remapped, 117 active+clean; 2.1 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.7 KiB/s rd, 106 MiB/s wr, 1.70k op/s; 5509/105594 objects misplaced (5.217%); 1.6 MiB/s, 33 objects/s recovering 2024-09-21T13:58:26.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:25 smithi049 ceph-mon[30080]: osdmap e275: 8 total, 7 up, 8 in 2024-09-21T13:58:26.267 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:26.289 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:26.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:26.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:26.390 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:58:26.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:26.434 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:26.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:58:26.535 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:58:26.573 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:26.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:26.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:58:26.653 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T13:58:26.653 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:58:26.667 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 22 16 38824 38808 110.234 48.0625 0.0167266 0.00906547 2024-09-21T13:58:26.676 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:58:26.714 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:26.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:26.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:58:26.815 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:58:26.839 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:26.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:26.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:58:26.941 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:26.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:26.964 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:26.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:27.065 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:58:27.103 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:27.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:27.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:58:27.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:26 smithi187 ceph-mon[33032]: pgmap v473: 148 pgs: 1 remapped+peering, 9 active+undersized, 4 peering, 21 stale+active+clean, 2 active+recovering+undersized+remapped, 9 active+undersized+degraded, 102 active+clean; 2.3 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s; 4691/113700 objects degraded (4.126%); 2996/113700 objects misplaced (2.635%); 12 MiB/s, 190 objects/s recovering 2024-09-21T13:58:27.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:26 smithi187 ceph-mon[33032]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-21T13:58:27.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:26 smithi187 ceph-mon[33032]: Health check failed: Degraded data redundancy: 4691/113700 objects degraded (4.126%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T13:58:27.204 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:27.233 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:27.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:27.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:27.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:26 smithi049 ceph-mon[40489]: pgmap v473: 148 pgs: 1 remapped+peering, 9 active+undersized, 4 peering, 21 stale+active+clean, 2 active+recovering+undersized+remapped, 9 active+undersized+degraded, 102 active+clean; 2.3 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s; 4691/113700 objects degraded (4.126%); 2996/113700 objects misplaced (2.635%); 12 MiB/s, 190 objects/s recovering 2024-09-21T13:58:27.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:26 smithi049 ceph-mon[40489]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-21T13:58:27.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:26 smithi049 ceph-mon[40489]: Health check failed: Degraded data redundancy: 4691/113700 objects degraded (4.126%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T13:58:27.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:26 smithi049 ceph-mon[30080]: pgmap v473: 148 pgs: 1 remapped+peering, 9 active+undersized, 4 peering, 21 stale+active+clean, 2 active+recovering+undersized+remapped, 9 active+undersized+degraded, 102 active+clean; 2.3 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s; 4691/113700 objects degraded (4.126%); 2996/113700 objects misplaced (2.635%); 12 MiB/s, 190 objects/s recovering 2024-09-21T13:58:27.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:26 smithi049 ceph-mon[30080]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-21T13:58:27.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:26 smithi049 ceph-mon[30080]: Health check failed: Degraded data redundancy: 4691/113700 objects degraded (4.126%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T13:58:27.335 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:27.371 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T13:58:27.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:27.372 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:27.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:27.473 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:58:27.501 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:27.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:27.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:58:27.603 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:58:27.638 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:27.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:27.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:58:27.667 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 23 16 40334 40318 109.544 94.375 0.0174001 0.00912262 2024-09-21T13:58:27.695 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 13:58:27 smithi187 podman[107647]: 2024-09-21 13:58:27.421127948 +0000 UTC m=+0.027651520 image pull b415816db8b41f544cdfdc71d193ff8b1ee008bf7b0ec4f64da8eb8f71b62e85 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5 2024-09-21T13:58:27.695 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 13:58:27 smithi187 podman[107647]: 2024-09-21 13:58:27.560677251 +0000 UTC m=+0.167200807 container create 0ff93a1e38929096b163aca66a5de3338baaed5a0e672fb5987d44de2c9c95dd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T13:58:27.740 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:58:27.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:27.786 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:27.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:58:27.887 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:58:27.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:27.914 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:27.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:58:28.015 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:28.055 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:28.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:28.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:28.156 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:58:28.186 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:28.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:28.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:58:28.195 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 13:58:28 smithi187 podman[107647]: 2024-09-21 13:58:28.005929385 +0000 UTC m=+0.612452949 container init 0ff93a1e38929096b163aca66a5de3338baaed5a0e672fb5987d44de2c9c95dd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7-deactivate, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=) 2024-09-21T13:58:28.195 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 13:58:28 smithi187 podman[107647]: 2024-09-21 13:58:28.013136064 +0000 UTC m=+0.619659620 container start 0ff93a1e38929096b163aca66a5de3338baaed5a0e672fb5987d44de2c9c95dd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7-deactivate, RELEASE=wip-guits-main-2024-09-21-0842-4417849, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2) 2024-09-21T13:58:28.268 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:58:28.287 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:28.326 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:28.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:28.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:28.428 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:58:28.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:28.462 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:28.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:58:28.557 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 13:58:28 smithi187 podman[107647]: 2024-09-21 13:58:28.205815338 +0000 UTC m=+0.812338902 container attach 0ff93a1e38929096b163aca66a5de3338baaed5a0e672fb5987d44de2c9c95dd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-09-21T13:58:28.557 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 13:58:28 smithi187 podman[107647]: 2024-09-21 13:58:28.290388549 +0000 UTC m=+0.896912116 container died 0ff93a1e38929096b163aca66a5de3338baaed5a0e672fb5987d44de2c9c95dd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T13:58:28.563 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:28.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:28.595 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:28.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:28.667 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 24 16 41717 41701 108.581 86.4375 0.0112134 0.00920332 2024-09-21T13:58:28.696 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:28.737 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:28.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:28.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:28.839 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:58:28.869 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:28.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:28.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:58:28.971 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:29.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:29.009 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:29.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:29.110 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:29.132 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:29.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:29.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:29.234 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:58:29.277 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:29.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:29.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:58:29.379 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:29.406 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:29.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:29.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:29.494 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:58:29.495 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:58:29.508 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:58:29.539 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:58:29.539 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:58:29.555 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:58:29.578 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:58:29.579 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:29.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:29.579 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:58:29.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:58:29.617 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:58:29.617 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:58:29.644 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:58:29 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:58:29.650 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:58:29.651 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:58:29.668 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 25 16 43187 43171 107.912 91.875 0.0106846 0.00926042 2024-09-21T13:58:29.680 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:29.691 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:58:29.691 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:58:29.725 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:58:29.725 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:58:29.728 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T13:58:29.730 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:29.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:29.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:29.767 DEBUG:teuthology.orchestra.run.smithi187:> sudo pkill -f 'journalctl -f -n 0 -u ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@osd.7.service' 2024-09-21T13:58:29.831 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:58:29.858 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:29.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:29.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:58:29.862 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:29 smithi049 ceph-mon[30080]: pgmap v474: 148 pgs: 1 remapped+peering, 15 active+undersized, 4 peering, 17 stale+active+clean, 2 active+recovering+undersized+remapped, 10 active+undersized+degraded, 99 active+clean; 2.3 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 71 MiB/s wr, 1.14k op/s; 4692/113700 objects degraded (4.127%); 2996/113700 objects misplaced (2.635%); 10 MiB/s, 165 objects/s recovering 2024-09-21T13:58:29.862 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:29 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:58:29.862 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:29 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:58:29.862 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:29 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:58:29.862 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:29 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:58:29.862 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:29 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3181407732' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:58:29.863 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:29 smithi049 ceph-mon[40489]: pgmap v474: 148 pgs: 1 remapped+peering, 15 active+undersized, 4 peering, 17 stale+active+clean, 2 active+recovering+undersized+remapped, 10 active+undersized+degraded, 99 active+clean; 2.3 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 71 MiB/s wr, 1.14k op/s; 4692/113700 objects degraded (4.127%); 2996/113700 objects misplaced (2.635%); 10 MiB/s, 165 objects/s recovering 2024-09-21T13:58:29.863 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:29 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:58:29.863 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:29 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:58:29.863 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:29 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:58:29.863 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:29 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:58:29.863 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:29 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3181407732' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:58:29.920 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:29 smithi187 ceph-mon[33032]: pgmap v474: 148 pgs: 1 remapped+peering, 15 active+undersized, 4 peering, 17 stale+active+clean, 2 active+recovering+undersized+remapped, 10 active+undersized+degraded, 99 active+clean; 2.3 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 71 MiB/s wr, 1.14k op/s; 4692/113700 objects degraded (4.127%); 2996/113700 objects misplaced (2.635%); 10 MiB/s, 165 objects/s recovering 2024-09-21T13:58:29.921 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:29 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:58:29.921 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:29 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:58:29.921 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:29 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:58:29.921 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:29 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:58:29.921 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:29 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3181407732' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:58:29.924 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T13:58:29.925 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-21T13:58:29.925 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.7 2024-09-21T13:58:29.925 INFO:tasks.thrashosds.thrasher:remote for osd 7 is ubuntu@smithi187.front.sepia.ceph.com 2024-09-21T13:58:29.925 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell -n osd.7 --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --op list-pgs 2024-09-21T13:58:29.960 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:30.007 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:30.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:30.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:30.109 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:30.148 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:30.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:30.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:30.231 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/osd.7/config 2024-09-21T13:58:30.252 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:58:30.261 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T13:58:30.261 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-21T13:58:30.261 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 0 2024-09-21T13:58:30.261 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T13:58:30.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:30.298 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:30.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:58:30.399 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:58:30.425 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:30.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:30.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:58:30.527 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:30.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:30.558 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:30.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:30.659 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:58:30.667 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 26 16 44668 44652 107.322 92.5625 0.00462855 0.00931239 2024-09-21T13:58:30.687 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:30.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:30.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:58:30.789 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:30.833 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:30.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:30.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:30.905 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:30 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:58:30.905 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:30 smithi187 ceph-mon[33032]: osdmap e276: 8 total, 7 up, 8 in 2024-09-21T13:58:30.935 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:30.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:30.975 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:30.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:31.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:30 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:58:31.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:30 smithi049 ceph-mon[30080]: osdmap e276: 8 total, 7 up, 8 in 2024-09-21T13:58:31.005 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:58:30 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:58:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:58:31.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:30 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:58:31.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:30 smithi049 ceph-mon[40489]: osdmap e276: 8 total, 7 up, 8 in 2024-09-21T13:58:31.076 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:31.120 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:31.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:31.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:31.215 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T13:58:31.216 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T13:58:31.221 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:31.262 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:31.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:31.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:31.364 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:58:31.392 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:31.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:31.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:58:31.494 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:31.536 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:31.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:31.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:31.638 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:31.668 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 27 15 46249 46234 107.007 98.875 0.00624528 0.00933852 2024-09-21T13:58:31.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:31.678 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:31.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:31.683 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-09-21T13:58:31.685 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T13:58:31.779 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:58:31.812 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:31.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:31.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:58:31.913 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:31.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:31 smithi187 ceph-mon[33032]: pgmap v476: 148 pgs: 1 remapped+peering, 33 active+undersized, 2 active+recovering+undersized+remapped, 28 active+undersized+degraded, 84 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 59 MiB/s wr, 949 op/s; 20412/122688 objects degraded (16.637%); 1591/122688 objects misplaced (1.297%); 25 MiB/s, 399 objects/s recovering 2024-09-21T13:58:31.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:31 smithi187 ceph-mon[33032]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-21T13:58:31.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:31 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3751732532' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:58:31.951 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:31.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:31.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:32.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:31 smithi049 ceph-mon[40489]: pgmap v476: 148 pgs: 1 remapped+peering, 33 active+undersized, 2 active+recovering+undersized+remapped, 28 active+undersized+degraded, 84 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 59 MiB/s wr, 949 op/s; 20412/122688 objects degraded (16.637%); 1591/122688 objects misplaced (1.297%); 25 MiB/s, 399 objects/s recovering 2024-09-21T13:58:32.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:31 smithi049 ceph-mon[40489]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-21T13:58:32.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:31 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3751732532' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:58:32.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:31 smithi049 ceph-mon[30080]: pgmap v476: 148 pgs: 1 remapped+peering, 33 active+undersized, 2 active+recovering+undersized+remapped, 28 active+undersized+degraded, 84 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 59 MiB/s wr, 949 op/s; 20412/122688 objects degraded (16.637%); 1591/122688 objects misplaced (1.297%); 25 MiB/s, 399 objects/s recovering 2024-09-21T13:58:32.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:31 smithi049 ceph-mon[30080]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-21T13:58:32.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:31 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3751732532' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:58:32.053 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:32.094 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:32.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:32.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:32.196 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:58:32.208 INFO:teuthology.orchestra.run.smithi187.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-21T13:58:32.241 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:32.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:32.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:58:32.259 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:58:32.343 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:32.385 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:32.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:32.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:32.487 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:32.532 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:32.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:32.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:32.635 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:32.668 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 28 15 47808 47793 106.665 97.4375 0.00888051 0.00937032 2024-09-21T13:58:32.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:32.676 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:32.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:32.777 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:58:32.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:32 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:58:32.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:32 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:58:32.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:32 smithi187 ceph-mon[33032]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:58:32.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:32 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3751732532' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T13:58:32.806 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:32 smithi187 ceph-mon[33032]: osdmap e277: 8 total, 7 up, 8 in 2024-09-21T13:58:32.826 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:32.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:32.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:58:32.927 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:32.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:32.971 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:32.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:33.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:32 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:58:33.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:32 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:58:33.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:32 smithi049 ceph-mon[30080]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:58:33.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:32 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3751732532' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T13:58:33.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:32 smithi049 ceph-mon[30080]: osdmap e277: 8 total, 7 up, 8 in 2024-09-21T13:58:33.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:32 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:58:33.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:32 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:58:33.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:32 smithi049 ceph-mon[40489]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:58:33.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:32 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3751732532' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T13:58:33.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:32 smithi049 ceph-mon[40489]: osdmap e277: 8 total, 7 up, 8 in 2024-09-21T13:58:33.072 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:33.110 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T13:58:33.111 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-21T13:58:33.111 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-21T13:58:33.111 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-21T13:58:33.111 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T13:58:33.116 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:33.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:33.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:33.217 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T13:58:33.218 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:33.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:33.265 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:33.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:33.366 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:33.411 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:33.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:33.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:33.515 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:58:33.550 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:33.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:33.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:58:33.651 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:58:33.668 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 29 16 49272 49256 106.139 91.4375 0.0102457 0.00941513 2024-09-21T13:58:33.700 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:33 smithi049 ceph-mon[30080]: pgmap v478: 148 pgs: 1 active+recovering+degraded+remapped, 33 active+undersized, 2 active+recovering+undersized+remapped, 28 active+undersized+degraded, 84 active+clean; 2.6 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 53 MiB/s wr, 845 op/s; 24020/129126 objects degraded (18.602%); 3110/129126 objects misplaced (2.409%); 17 MiB/s, 269 objects/s recovering 2024-09-21T13:58:33.704 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:33.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:33.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:58:33.716 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:58:33.807 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:33.851 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:33.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:33.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:33.954 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:33.985 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:33.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:33.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:34.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:33 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 24020/129126 objects degraded (18.602%), 29 pgs degraded (PG_DEGRADED) 2024-09-21T13:58:34.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:33 smithi049 ceph-mon[40489]: pgmap v478: 148 pgs: 1 active+recovering+degraded+remapped, 33 active+undersized, 2 active+recovering+undersized+remapped, 28 active+undersized+degraded, 84 active+clean; 2.6 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 53 MiB/s wr, 845 op/s; 24020/129126 objects degraded (18.602%); 3110/129126 objects misplaced (2.409%); 17 MiB/s, 269 objects/s recovering 2024-09-21T13:58:34.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:33 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 24020/129126 objects degraded (18.602%), 29 pgs degraded (PG_DEGRADED) 2024-09-21T13:58:34.087 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:34.097 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:33 smithi187 ceph-mon[33032]: pgmap v478: 148 pgs: 1 active+recovering+degraded+remapped, 33 active+undersized, 2 active+recovering+undersized+remapped, 28 active+undersized+degraded, 84 active+clean; 2.6 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 53 MiB/s wr, 845 op/s; 24020/129126 objects degraded (18.602%); 3110/129126 objects misplaced (2.409%); 17 MiB/s, 269 objects/s recovering 2024-09-21T13:58:34.097 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:33 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 24020/129126 objects degraded (18.602%), 29 pgs degraded (PG_DEGRADED) 2024-09-21T13:58:34.126 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:34.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:34.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:34.143 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T13:58:34.227 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:34.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:34.268 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:34.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:34.369 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:58:34.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:34.405 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:34.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:58:34.506 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:34.511 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:58:34.535 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:34.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:34.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:34.638 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:58:34.669 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 30 16 50723 50707 105.623 90.6875 0.0108702 0.00946155 2024-09-21T13:58:34.677 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:34.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:34.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:58:34.780 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:58:34.796 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T13:58:34.814 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:34.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:34.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:58:34.916 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:58:34.942 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:34.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:34.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:58:35.011 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:58:35.011 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:58:35.039 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:58:35.039 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:58:35.044 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:58:35.065 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:58:35.065 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:58:35.072 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:35.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:35.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:58:35.089 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:58:35.090 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:58:35.111 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:58:35.111 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:58:35.124 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T13:58:35.124 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-21T13:58:35.124 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-21T13:58:35.125 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-21T13:58:35.125 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.125 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:36843 7.d 7:b9f54a05:::benchmark_data_smithi049_76798_object36842:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-21T13:58:35.125 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:30.623325+0000", 2024-09-21T13:58:35.125 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 484.49965371299999, 2024-09-21T13:58:35.125 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.14164070300000001, 2024-09-21T13:58:35.125 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:58:35.125 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:58:35.125 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:58:35.125 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:58:35.125 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:58:35.125 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:58:35.125 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 36843 2024-09-21T13:58:35.125 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.126 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:58:35.126 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.126 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:58:35.126 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.623325+0000", 2024-09-21T13:58:35.126 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:58:35.126 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.126 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.126 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:58:35.126 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.623325+0000", 2024-09-21T13:58:35.126 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:58:35.126 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.126 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.126 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:58:35.127 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.623328+0000", 2024-09-21T13:58:35.127 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8120000000000001e-06 2024-09-21T13:58:35.127 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.127 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.127 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:58:35.127 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.623434+0000", 2024-09-21T13:58:35.127 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010542799999999999 2024-09-21T13:58:35.127 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.127 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.127 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:58:35.127 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.623443+0000", 2024-09-21T13:58:35.127 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1200000000000008e-06 2024-09-21T13:58:35.127 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.127 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.127 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:58:35.128 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.623448+0000", 2024-09-21T13:58:35.128 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2079999999999999e-06 2024-09-21T13:58:35.128 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.128 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.128 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:58:35.128 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.623483+0000", 2024-09-21T13:58:35.128 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5048000000000001e-05 2024-09-21T13:58:35.128 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.128 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.128 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:58:35.128 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.623592+0000", 2024-09-21T13:58:35.129 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000108585 2024-09-21T13:58:35.129 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.129 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.129 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-21T13:58:35.130 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.623673+0000", 2024-09-21T13:58:35.130 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1079000000000001e-05 2024-09-21T13:58:35.131 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.131 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.131 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:58:35.131 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.624761+0000", 2024-09-21T13:58:35.131 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001088449 2024-09-21T13:58:35.131 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.131 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.131 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:58:35.131 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.625971+0000", 2024-09-21T13:58:35.131 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0012098949999999999 2024-09-21T13:58:35.131 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.131 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.131 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:58:35.131 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.764850+0000", 2024-09-21T13:58:35.132 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13887893000000001 2024-09-21T13:58:35.132 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.132 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.132 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:58:35.132 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.764907+0000", 2024-09-21T13:58:35.132 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6901999999999998e-05 2024-09-21T13:58:35.132 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.132 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.132 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:58:35.132 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.764965+0000", 2024-09-21T13:58:35.132 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8247e-05 2024-09-21T13:58:35.132 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:58:35.132 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:58:35.132 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:58:35.133 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.133 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.133 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:36844 7.c 7:3245d1c7:::benchmark_data_smithi049_76798_object36843:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-21T13:58:35.133 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:30.625096+0000", 2024-09-21T13:58:35.133 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 484.49788203000003, 2024-09-21T13:58:35.133 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.140563562, 2024-09-21T13:58:35.133 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:58:35.133 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:58:35.133 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:58:35.133 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:58:35.133 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:58:35.133 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:58:35.133 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 36844 2024-09-21T13:58:35.133 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.133 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:58:35.134 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.134 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:58:35.134 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.625096+0000", 2024-09-21T13:58:35.134 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:58:35.134 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.134 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.134 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:58:35.134 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.625096+0000", 2024-09-21T13:58:35.134 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:58:35.134 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.134 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.134 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:58:35.134 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.625097+0000", 2024-09-21T13:58:35.134 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.116e-06 2024-09-21T13:58:35.134 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.135 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.135 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:58:35.135 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.625152+0000", 2024-09-21T13:58:35.135 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4682000000000003e-05 2024-09-21T13:58:35.135 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.135 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.135 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:58:35.135 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.625161+0000", 2024-09-21T13:58:35.135 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1670000000000005e-06 2024-09-21T13:58:35.135 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.135 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.135 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:58:35.135 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.625168+0000", 2024-09-21T13:58:35.135 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7880000000000001e-06 2024-09-21T13:58:35.135 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.136 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.136 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:58:35.136 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.625207+0000", 2024-09-21T13:58:35.136 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9251000000000001e-05 2024-09-21T13:58:35.136 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.136 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.136 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:58:35.136 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.625278+0000", 2024-09-21T13:58:35.136 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.0659000000000002e-05 2024-09-21T13:58:35.136 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.136 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.136 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,7", 2024-09-21T13:58:35.136 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.625357+0000", 2024-09-21T13:58:35.136 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9015000000000002e-05 2024-09-21T13:58:35.136 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.137 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.137 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:58:35.137 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.626576+0000", 2024-09-21T13:58:35.137 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001219363 2024-09-21T13:58:35.137 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.137 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.137 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:58:35.137 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.626862+0000", 2024-09-21T13:58:35.137 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00028596300000000003 2024-09-21T13:58:35.137 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.137 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.137 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:58:35.137 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.765560+0000", 2024-09-21T13:58:35.137 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.138697767 2024-09-21T13:58:35.137 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.138 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.138 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:58:35.138 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.765620+0000", 2024-09-21T13:58:35.138 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0099999999999997e-05 2024-09-21T13:58:35.138 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.138 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.138 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:58:35.138 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.765660+0000", 2024-09-21T13:58:35.138 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9691000000000003e-05 2024-09-21T13:58:35.138 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:58:35.138 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:58:35.138 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:58:35.138 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.138 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.138 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:36848 7.7 7:e1dc98c1:::benchmark_data_smithi049_76798_object36847:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-21T13:58:35.139 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:30.625310+0000", 2024-09-21T13:58:35.139 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 484.49766782799998, 2024-09-21T13:58:35.139 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.141300645, 2024-09-21T13:58:35.139 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:58:35.139 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:58:35.139 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:58:35.139 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:58:35.139 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:58:35.139 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:58:35.139 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 36848 2024-09-21T13:58:35.139 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.139 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:58:35.139 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.139 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:58:35.140 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.625310+0000", 2024-09-21T13:58:35.140 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:58:35.140 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.140 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.140 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:58:35.140 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.625310+0000", 2024-09-21T13:58:35.140 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:58:35.140 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.140 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.140 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:58:35.140 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.625311+0000", 2024-09-21T13:58:35.140 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1399999999999996e-07 2024-09-21T13:58:35.140 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.140 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.141 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:58:35.141 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.625453+0000", 2024-09-21T13:58:35.141 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014165200000000001 2024-09-21T13:58:35.141 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.141 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.141 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:58:35.141 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.625461+0000", 2024-09-21T13:58:35.141 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7579999999999992e-06 2024-09-21T13:58:35.141 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.141 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.141 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:58:35.141 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.625465+0000", 2024-09-21T13:58:35.141 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3989999999999997e-06 2024-09-21T13:58:35.141 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.141 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.142 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:58:35.142 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.625514+0000", 2024-09-21T13:58:35.142 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8742000000000002e-05 2024-09-21T13:58:35.142 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.142 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.142 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:58:35.142 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.625598+0000", 2024-09-21T13:58:35.142 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4091999999999994e-05 2024-09-21T13:58:35.142 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.142 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.142 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3", 2024-09-21T13:58:35.142 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.625671+0000", 2024-09-21T13:58:35.142 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3566999999999995e-05 2024-09-21T13:58:35.142 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.142 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.143 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:58:35.143 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.626848+0000", 2024-09-21T13:58:35.143 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0011770109999999999 2024-09-21T13:58:35.143 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.143 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.143 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:58:35.143 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.766471+0000", 2024-09-21T13:58:35.143 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13962287100000001 2024-09-21T13:58:35.143 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.143 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.143 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:58:35.144 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.766504+0000", 2024-09-21T13:58:35.144 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2790000000000003e-05 2024-09-21T13:58:35.144 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.144 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.144 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:58:35.144 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.766575+0000", 2024-09-21T13:58:35.144 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1348999999999997e-05 2024-09-21T13:58:35.144 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.144 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.144 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:58:35.144 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.766611+0000", 2024-09-21T13:58:35.144 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.57e-05 2024-09-21T13:58:35.144 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:58:35.144 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:58:35.144 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:58:35.145 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.145 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.145 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:36854 7.7 7:eea57e62:::benchmark_data_smithi049_76798_object36853:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-21T13:58:35.145 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:30.628335+0000", 2024-09-21T13:58:35.145 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 484.49464314699998, 2024-09-21T13:58:35.145 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.140131803, 2024-09-21T13:58:35.145 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:58:35.145 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:58:35.145 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:58:35.145 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:58:35.145 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:58:35.145 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:58:35.145 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 36854 2024-09-21T13:58:35.145 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.146 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:58:35.146 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.146 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:58:35.146 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.628335+0000", 2024-09-21T13:58:35.146 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:58:35.146 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.146 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.146 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:58:35.146 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.628335+0000", 2024-09-21T13:58:35.146 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:58:35.146 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.146 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.146 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:58:35.146 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.628336+0000", 2024-09-21T13:58:35.147 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8899999999999999e-07 2024-09-21T13:58:35.147 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.147 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.147 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:58:35.147 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.628403+0000", 2024-09-21T13:58:35.147 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6880999999999999e-05 2024-09-21T13:58:35.147 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.147 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.147 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:58:35.147 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.628409+0000", 2024-09-21T13:58:35.147 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9329999999999998e-06 2024-09-21T13:58:35.147 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.147 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.147 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:58:35.147 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.628413+0000", 2024-09-21T13:58:35.148 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0860000000000004e-06 2024-09-21T13:58:35.148 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.148 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.148 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:58:35.148 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.628458+0000", 2024-09-21T13:58:35.148 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5111000000000002e-05 2024-09-21T13:58:35.148 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.148 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.148 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:58:35.148 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.628571+0000", 2024-09-21T13:58:35.148 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000113565 2024-09-21T13:58:35.148 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.148 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.148 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3", 2024-09-21T13:58:35.148 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.628701+0000", 2024-09-21T13:58:35.149 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012919400000000001 2024-09-21T13:58:35.149 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.149 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.149 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:58:35.149 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.766702+0000", 2024-09-21T13:58:35.149 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13800140499999999 2024-09-21T13:58:35.149 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.149 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.149 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:58:35.149 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.766725+0000", 2024-09-21T13:58:35.149 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2986999999999999e-05 2024-09-21T13:58:35.149 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.149 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.149 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:58:35.150 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.768379+0000", 2024-09-21T13:58:35.150 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0016544870000000001 2024-09-21T13:58:35.150 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.150 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.150 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:58:35.150 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.768432+0000", 2024-09-21T13:58:35.150 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2771e-05 2024-09-21T13:58:35.150 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.150 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.150 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:58:35.150 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.768467+0000", 2024-09-21T13:58:35.150 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4693999999999997e-05 2024-09-21T13:58:35.150 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:58:35.150 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:58:35.150 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:58:35.151 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.151 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.151 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.14817.0:36851 7.b e96/83 7:d46d3307:::benchmark_data_smithi049_76798_object36850:head v 96'2302, mlcod=96'2299)", 2024-09-21T13:58:35.151 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:30.629763+0000", 2024-09-21T13:58:35.151 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 484.49321532599998, 2024-09-21T13:58:35.151 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13851205, 2024-09-21T13:58:35.151 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:58:35.151 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:58:35.151 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:58:35.151 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:58:35.151 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.151 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:58:35.151 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.629763+0000", 2024-09-21T13:58:35.151 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:58:35.152 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.152 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.152 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:58:35.152 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.629763+0000", 2024-09-21T13:58:35.152 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:58:35.152 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.152 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.152 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:58:35.152 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.629764+0000", 2024-09-21T13:58:35.152 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6899999999999996e-07 2024-09-21T13:58:35.152 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.152 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.152 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:58:35.153 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:58:35.153 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:58:35.154 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.629814+0000", 2024-09-21T13:58:35.154 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9891000000000001e-05 2024-09-21T13:58:35.154 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.154 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.154 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:58:35.154 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.764759+0000", 2024-09-21T13:58:35.154 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13494542600000001 2024-09-21T13:58:35.154 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.154 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.154 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:58:35.155 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.764771+0000", 2024-09-21T13:58:35.155 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1764e-05 2024-09-21T13:58:35.155 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.155 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.155 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:58:35.155 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.764824+0000", 2024-09-21T13:58:35.155 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3378999999999998e-05 2024-09-21T13:58:35.155 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.155 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.155 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:58:35.155 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.764858+0000", 2024-09-21T13:58:35.155 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3714999999999998e-05 2024-09-21T13:58:35.155 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.155 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.156 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:58:35.156 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.768222+0000", 2024-09-21T13:58:35.156 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0033640749999999998 2024-09-21T13:58:35.156 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.156 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.156 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:58:35.156 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.768275+0000", 2024-09-21T13:58:35.156 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2831000000000003e-05 2024-09-21T13:58:35.156 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:58:35.156 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:58:35.156 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:58:35.156 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.156 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.156 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:49358 7.d 7:bafe9a80:::benchmark_data_smithi049_76798_object49357:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T13:58:35.156 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:38.254356+0000", 2024-09-21T13:58:35.157 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 476.86862226099998, 2024-09-21T13:58:35.157 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.153550043, 2024-09-21T13:58:35.157 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:58:35.157 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:58:35.157 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:58:35.157 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:58:35.157 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:58:35.157 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:58:35.157 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 49358 2024-09-21T13:58:35.157 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.157 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:58:35.157 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.157 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:58:35.157 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.254356+0000", 2024-09-21T13:58:35.157 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:58:35.158 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.158 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.158 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:58:35.158 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.254356+0000", 2024-09-21T13:58:35.158 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:58:35.158 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.158 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.158 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:58:35.158 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.254357+0000", 2024-09-21T13:58:35.158 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4600000000000004e-07 2024-09-21T13:58:35.158 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.158 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.159 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:58:35.159 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.254488+0000", 2024-09-21T13:58:35.159 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013163199999999999 2024-09-21T13:58:35.159 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.159 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.159 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:58:35.159 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.254503+0000", 2024-09-21T13:58:35.159 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.452e-05 2024-09-21T13:58:35.159 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.159 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.159 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:58:35.159 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.254510+0000", 2024-09-21T13:58:35.159 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.554e-06 2024-09-21T13:58:35.159 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.160 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.160 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:58:35.160 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.254578+0000", 2024-09-21T13:58:35.160 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7968999999999999e-05 2024-09-21T13:58:35.160 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.160 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.160 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:58:35.160 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.254736+0000", 2024-09-21T13:58:35.160 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000157219 2024-09-21T13:58:35.160 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.160 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.160 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-21T13:58:35.160 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.254886+0000", 2024-09-21T13:58:35.160 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000150667 2024-09-21T13:58:35.161 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.161 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.161 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:58:35.161 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.257811+0000", 2024-09-21T13:58:35.161 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00292436 2024-09-21T13:58:35.161 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.161 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.161 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:58:35.161 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.261523+0000", 2024-09-21T13:58:35.161 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0037119639999999999 2024-09-21T13:58:35.161 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.161 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.161 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:58:35.161 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.407826+0000", 2024-09-21T13:58:35.161 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.146303813 2024-09-21T13:58:35.162 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.162 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.162 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:58:35.162 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.407877+0000", 2024-09-21T13:58:35.162 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0603999999999999e-05 2024-09-21T13:58:35.162 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.162 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.162 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:58:35.162 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.407906+0000", 2024-09-21T13:58:35.162 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8994999999999999e-05 2024-09-21T13:58:35.162 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:58:35.162 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:58:35.162 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:58:35.162 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.163 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.163 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:49364 7.d 7:ba37e2e5:::benchmark_data_smithi049_76798_object49363:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T13:58:35.163 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:38.254570+0000", 2024-09-21T13:58:35.163 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 476.86840848999998, 2024-09-21T13:58:35.163 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.156026475, 2024-09-21T13:58:35.163 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:58:35.163 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:58:35.163 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:58:35.163 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:58:35.163 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:58:35.163 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:58:35.164 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 49364 2024-09-21T13:58:35.164 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.164 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:58:35.164 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.164 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:58:35.164 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.254570+0000", 2024-09-21T13:58:35.164 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:58:35.164 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.164 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.164 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:58:35.164 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.254570+0000", 2024-09-21T13:58:35.164 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:58:35.164 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.164 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.164 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:58:35.165 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.254571+0000", 2024-09-21T13:58:35.165 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.8700000000000004e-07 2024-09-21T13:58:35.165 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.165 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.165 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:58:35.165 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.254637+0000", 2024-09-21T13:58:35.165 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6704000000000004e-05 2024-09-21T13:58:35.165 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.165 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.165 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:58:35.165 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.254652+0000", 2024-09-21T13:58:35.165 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.438e-05 2024-09-21T13:58:35.165 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.165 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.165 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:58:35.166 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.254659+0000", 2024-09-21T13:58:35.166 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9589999999999998e-06 2024-09-21T13:58:35.166 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.166 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.166 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:58:35.166 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.256288+0000", 2024-09-21T13:58:35.166 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001629669 2024-09-21T13:58:35.166 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.166 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.166 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:58:35.166 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.256409+0000", 2024-09-21T13:58:35.166 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012085400000000001 2024-09-21T13:58:35.166 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.166 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.166 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-21T13:58:35.167 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.256520+0000", 2024-09-21T13:58:35.167 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000111141 2024-09-21T13:58:35.167 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.167 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.167 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:58:35.167 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.257926+0000", 2024-09-21T13:58:35.167 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001405959 2024-09-21T13:58:35.167 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.167 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.167 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:58:35.167 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.263043+0000", 2024-09-21T13:58:35.167 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0051160989999999998 2024-09-21T13:58:35.167 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.167 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.168 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:58:35.168 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.410522+0000", 2024-09-21T13:58:35.168 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.14747992600000001 2024-09-21T13:58:35.168 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.168 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.168 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:58:35.168 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.410571+0000", 2024-09-21T13:58:35.168 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8538000000000002e-05 2024-09-21T13:58:35.168 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.168 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.168 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:58:35.168 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.410596+0000", 2024-09-21T13:58:35.168 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5259e-05 2024-09-21T13:58:35.168 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:58:35.169 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:58:35.169 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:58:35.169 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.169 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.169 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:49368 7.d 7:bfd0cd80:::benchmark_data_smithi049_76798_object49367:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T13:58:35.169 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:38.259454+0000", 2024-09-21T13:58:35.169 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 476.86352427999998, 2024-09-21T13:58:35.169 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.15201478299999999, 2024-09-21T13:58:35.169 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:58:35.169 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:58:35.169 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:58:35.169 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:58:35.169 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:58:35.169 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:58:35.170 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 49368 2024-09-21T13:58:35.170 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.170 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:58:35.170 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.170 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:58:35.170 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.259454+0000", 2024-09-21T13:58:35.170 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:58:35.170 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.170 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.170 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:58:35.170 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.259454+0000", 2024-09-21T13:58:35.170 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:58:35.170 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.170 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.171 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:58:35.171 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.259455+0000", 2024-09-21T13:58:35.171 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3260000000000001e-06 2024-09-21T13:58:35.171 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.171 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.171 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:58:35.171 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.259505+0000", 2024-09-21T13:58:35.171 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9941999999999997e-05 2024-09-21T13:58:35.171 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.171 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.171 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:58:35.171 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.259516+0000", 2024-09-21T13:58:35.171 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0943e-05 2024-09-21T13:58:35.171 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.171 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.172 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:58:35.172 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.259523+0000", 2024-09-21T13:58:35.172 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2200000000000003e-06 2024-09-21T13:58:35.172 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.172 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.172 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:58:35.172 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.259582+0000", 2024-09-21T13:58:35.172 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8789000000000002e-05 2024-09-21T13:58:35.172 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.172 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.172 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:58:35.172 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.259708+0000", 2024-09-21T13:58:35.172 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012536700000000001 2024-09-21T13:58:35.172 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.173 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.173 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-21T13:58:35.173 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.259861+0000", 2024-09-21T13:58:35.173 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015389900000000001 2024-09-21T13:58:35.173 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.173 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.173 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:58:35.173 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.261617+0000", 2024-09-21T13:58:35.173 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0017559260000000001 2024-09-21T13:58:35.173 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.173 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.173 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:58:35.173 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.263374+0000", 2024-09-21T13:58:35.173 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0017561440000000001 2024-09-21T13:58:35.173 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.174 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.174 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:58:35.174 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.411384+0000", 2024-09-21T13:58:35.174 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.148010004 2024-09-21T13:58:35.174 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.174 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.174 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:58:35.174 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.411437+0000", 2024-09-21T13:58:35.174 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3600999999999997e-05 2024-09-21T13:58:35.174 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.174 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.174 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:58:35.174 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.411469+0000", 2024-09-21T13:58:35.174 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1622000000000003e-05 2024-09-21T13:58:35.175 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:58:35.175 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:58:35.175 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:58:35.175 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.175 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.175 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:49375 7.7 7:ecf0f90a:::benchmark_data_smithi049_76798_object49374:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T13:58:35.175 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:38.263906+0000", 2024-09-21T13:58:35.175 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 476.85907224499999, 2024-09-21T13:58:35.175 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.161950177, 2024-09-21T13:58:35.175 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:58:35.175 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:58:35.175 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:58:35.175 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:58:35.175 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:58:35.175 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:58:35.176 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 49375 2024-09-21T13:58:35.176 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.176 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:58:35.176 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.176 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:58:35.176 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.263906+0000", 2024-09-21T13:58:35.176 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:58:35.176 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.176 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.176 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:58:35.176 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.263906+0000", 2024-09-21T13:58:35.176 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:58:35.176 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.176 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.176 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:58:35.177 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.263907+0000", 2024-09-21T13:58:35.177 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.61e-07 2024-09-21T13:58:35.177 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.177 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.177 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:58:35.177 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.263963+0000", 2024-09-21T13:58:35.177 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5772000000000001e-05 2024-09-21T13:58:35.177 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.177 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.177 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:58:35.177 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.263969+0000", 2024-09-21T13:58:35.177 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4200000000000004e-06 2024-09-21T13:58:35.177 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.177 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.177 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:58:35.178 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.263973+0000", 2024-09-21T13:58:35.178 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1869999999999999e-06 2024-09-21T13:58:35.178 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.178 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.178 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:58:35.178 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.264028+0000", 2024-09-21T13:58:35.178 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4657999999999997e-05 2024-09-21T13:58:35.178 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.178 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.178 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:58:35.178 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.264143+0000", 2024-09-21T13:58:35.178 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000115223 2024-09-21T13:58:35.178 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.178 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.178 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3", 2024-09-21T13:58:35.179 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.264277+0000", 2024-09-21T13:58:35.179 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000133993 2024-09-21T13:58:35.179 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.179 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.179 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:58:35.179 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.265777+0000", 2024-09-21T13:58:35.179 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0014999740000000001 2024-09-21T13:58:35.179 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.179 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.179 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:58:35.179 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.268782+0000", 2024-09-21T13:58:35.179 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0030045929999999998 2024-09-21T13:58:35.179 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.179 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.179 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:58:35.180 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.425737+0000", 2024-09-21T13:58:35.180 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.156955024 2024-09-21T13:58:35.180 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.180 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.180 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:58:35.180 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.425795+0000", 2024-09-21T13:58:35.180 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8443000000000003e-05 2024-09-21T13:58:35.180 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.180 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.180 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:58:35.180 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.425856+0000", 2024-09-21T13:58:35.180 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1129000000000004e-05 2024-09-21T13:58:35.180 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:58:35.180 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:58:35.181 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:58:35.181 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.181 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.181 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:49376 7.7 7:e5b9203c:::benchmark_data_smithi049_76798_object49375:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T13:58:35.181 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:38.265943+0000", 2024-09-21T13:58:35.181 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 476.85703528200003, 2024-09-21T13:58:35.181 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.16002640000000001, 2024-09-21T13:58:35.181 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:58:35.181 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:58:35.181 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:58:35.181 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:58:35.181 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:58:35.181 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:58:35.181 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 49376 2024-09-21T13:58:35.181 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.182 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:58:35.182 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.182 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:58:35.182 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.265943+0000", 2024-09-21T13:58:35.182 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:58:35.182 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.182 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.182 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:58:35.182 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.265943+0000", 2024-09-21T13:58:35.182 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:58:35.182 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.182 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.182 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:58:35.182 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.265944+0000", 2024-09-21T13:58:35.183 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.237e-06 2024-09-21T13:58:35.183 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.183 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.183 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:58:35.183 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.266008+0000", 2024-09-21T13:58:35.183 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3382999999999999e-05 2024-09-21T13:58:35.183 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.183 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.183 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:58:35.183 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.266018+0000", 2024-09-21T13:58:35.183 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0225e-05 2024-09-21T13:58:35.183 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.183 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.183 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:58:35.183 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.266023+0000", 2024-09-21T13:58:35.184 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0810000000000001e-06 2024-09-21T13:58:35.184 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.184 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.184 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:58:35.184 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.266066+0000", 2024-09-21T13:58:35.184 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2939999999999999e-05 2024-09-21T13:58:35.184 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.184 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.184 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:58:35.184 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.266173+0000", 2024-09-21T13:58:35.184 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000107089 2024-09-21T13:58:35.184 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.184 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.184 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3", 2024-09-21T13:58:35.185 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.266305+0000", 2024-09-21T13:58:35.185 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013210699999999999 2024-09-21T13:58:35.185 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.185 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.185 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:58:35.185 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.268746+0000", 2024-09-21T13:58:35.185 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00244072 2024-09-21T13:58:35.185 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.185 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.185 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:58:35.185 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.269249+0000", 2024-09-21T13:58:35.185 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00050357600000000005 2024-09-21T13:58:35.186 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.186 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.186 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:58:35.186 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.425891+0000", 2024-09-21T13:58:35.186 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.15664171700000001 2024-09-21T13:58:35.186 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.186 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.186 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:58:35.186 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.425934+0000", 2024-09-21T13:58:35.186 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3136999999999997e-05 2024-09-21T13:58:35.186 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.186 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.186 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:58:35.186 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.425969+0000", 2024-09-21T13:58:35.187 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5188000000000003e-05 2024-09-21T13:58:35.187 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:58:35.187 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:58:35.187 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:58:35.187 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.187 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.187 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:49377 7.7 7:e5eecd6c:::benchmark_data_smithi049_76798_object49376:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T13:58:35.187 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:38.266042+0000", 2024-09-21T13:58:35.187 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 476.85693593100001, 2024-09-21T13:58:35.187 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.160051258, 2024-09-21T13:58:35.187 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:58:35.187 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:58:35.187 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:58:35.187 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:58:35.188 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:58:35.188 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:58:35.188 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 49377 2024-09-21T13:58:35.188 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.188 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:58:35.188 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.188 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:58:35.188 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.266042+0000", 2024-09-21T13:58:35.188 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:58:35.188 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.188 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.188 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:58:35.188 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.266042+0000", 2024-09-21T13:58:35.188 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:58:35.188 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.188 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.189 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:58:35.189 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.266043+0000", 2024-09-21T13:58:35.189 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2599999999999998e-07 2024-09-21T13:58:35.189 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.189 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.189 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:58:35.189 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.266116+0000", 2024-09-21T13:58:35.189 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3120999999999993e-05 2024-09-21T13:58:35.189 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.189 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.189 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:58:35.189 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.266126+0000", 2024-09-21T13:58:35.189 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6809999999999993e-06 2024-09-21T13:58:35.189 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.189 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.190 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:58:35.190 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.266133+0000", 2024-09-21T13:58:35.190 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1590000000000004e-06 2024-09-21T13:58:35.190 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.190 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.190 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:58:35.190 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.267417+0000", 2024-09-21T13:58:35.190 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00128472 2024-09-21T13:58:35.190 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.190 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.190 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:58:35.190 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.267503+0000", 2024-09-21T13:58:35.190 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5261999999999998e-05 2024-09-21T13:58:35.190 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.191 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.191 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3", 2024-09-21T13:58:35.191 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.267626+0000", 2024-09-21T13:58:35.191 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000123435 2024-09-21T13:58:35.191 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.191 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.191 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:58:35.191 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.268883+0000", 2024-09-21T13:58:35.191 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00125657 2024-09-21T13:58:35.191 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.191 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.191 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:58:35.191 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.269389+0000", 2024-09-21T13:58:35.191 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00050581200000000002 2024-09-21T13:58:35.191 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.192 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.192 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:58:35.192 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.426010+0000", 2024-09-21T13:58:35.192 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.156621173 2024-09-21T13:58:35.192 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.192 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.192 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:58:35.192 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.426059+0000", 2024-09-21T13:58:35.192 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8891000000000003e-05 2024-09-21T13:58:35.192 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.192 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.192 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:58:35.192 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.426094+0000", 2024-09-21T13:58:35.192 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5008000000000001e-05 2024-09-21T13:58:35.192 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:58:35.193 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:58:35.193 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:58:35.193 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.193 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.193 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:202091 7.7 7:ebd82d39:::benchmark_data_smithi049_76798_object65671:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-21T13:58:35.193 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:44.203006+0000", 2024-09-21T13:58:35.193 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 350.91997201499998, 2024-09-21T13:58:35.193 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.14159556300000001, 2024-09-21T13:58:35.193 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:58:35.193 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:58:35.193 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:58:35.193 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:58:35.193 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:58:35.193 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:58:35.193 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 202091 2024-09-21T13:58:35.194 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.194 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:58:35.194 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.194 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:58:35.194 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.203006+0000", 2024-09-21T13:58:35.194 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:58:35.194 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.194 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.194 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:58:35.194 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.203006+0000", 2024-09-21T13:58:35.194 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:58:35.194 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.194 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.194 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:58:35.195 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.203007+0000", 2024-09-21T13:58:35.195 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1790000000000001e-06 2024-09-21T13:58:35.195 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.195 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.195 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:58:35.195 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.203012+0000", 2024-09-21T13:58:35.195 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6040000000000003e-06 2024-09-21T13:58:35.195 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.195 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.195 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:58:35.195 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.203021+0000", 2024-09-21T13:58:35.195 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.2539999999999998e-06 2024-09-21T13:58:35.195 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.195 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.196 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:58:35.196 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.203026+0000", 2024-09-21T13:58:35.196 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9830000000000004e-06 2024-09-21T13:58:35.196 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.196 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.196 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:58:35.196 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.203078+0000", 2024-09-21T13:58:35.196 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1421000000000001e-05 2024-09-21T13:58:35.196 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.196 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.196 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:58:35.196 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.203188+0000", 2024-09-21T13:58:35.196 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000110599 2024-09-21T13:58:35.196 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.196 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.197 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3", 2024-09-21T13:58:35.197 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.203294+0000", 2024-09-21T13:58:35.197 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000105986 2024-09-21T13:58:35.197 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.197 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.197 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:58:35.197 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.206024+0000", 2024-09-21T13:58:35.197 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0027295980000000002 2024-09-21T13:58:35.197 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.197 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.197 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:58:35.197 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.211989+0000", 2024-09-21T13:58:35.197 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0059648949999999996 2024-09-21T13:58:35.197 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.197 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.198 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:58:35.198 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.344508+0000", 2024-09-21T13:58:35.198 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13251928399999999 2024-09-21T13:58:35.198 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.198 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.198 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:58:35.198 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.344562+0000", 2024-09-21T13:58:35.198 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3653999999999998e-05 2024-09-21T13:58:35.198 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.198 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.198 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:58:35.198 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.344602+0000", 2024-09-21T13:58:35.198 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0105999999999999e-05 2024-09-21T13:58:35.198 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:58:35.199 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:58:35.199 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:58:35.199 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.199 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.199 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:202095 7.7 7:e91fc271:::benchmark_data_smithi049_76798_object65675:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-21T13:58:35.199 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:44.206857+0000", 2024-09-21T13:58:35.199 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 350.91612173999999, 2024-09-21T13:58:35.199 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13794869500000001, 2024-09-21T13:58:35.199 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:58:35.199 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:58:35.199 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:58:35.199 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:58:35.199 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:58:35.199 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:58:35.199 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 202095 2024-09-21T13:58:35.200 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.200 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:58:35.200 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.200 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:58:35.200 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.206857+0000", 2024-09-21T13:58:35.200 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:58:35.200 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.200 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.200 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:58:35.200 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.206857+0000", 2024-09-21T13:58:35.200 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:58:35.200 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.200 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.200 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:58:35.200 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.206858+0000", 2024-09-21T13:58:35.201 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.576e-06 2024-09-21T13:58:35.201 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.201 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.201 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:58:35.201 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.206863+0000", 2024-09-21T13:58:35.201 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3680000000000001e-06 2024-09-21T13:58:35.201 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.201 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.201 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:58:35.201 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.206875+0000", 2024-09-21T13:58:35.201 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.155e-05 2024-09-21T13:58:35.201 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.201 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.201 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:58:35.201 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.206882+0000", 2024-09-21T13:58:35.202 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1509999999999998e-06 2024-09-21T13:58:35.202 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.202 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.202 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:58:35.202 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.206924+0000", 2024-09-21T13:58:35.202 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1381999999999998e-05 2024-09-21T13:58:35.202 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.202 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.202 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:58:35.202 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.207007+0000", 2024-09-21T13:58:35.202 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3189000000000006e-05 2024-09-21T13:58:35.202 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.202 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.202 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3", 2024-09-21T13:58:35.202 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.207119+0000", 2024-09-21T13:58:35.203 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000112176 2024-09-21T13:58:35.203 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.203 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.203 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:58:35.203 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.211963+0000", 2024-09-21T13:58:35.203 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0048439690000000001 2024-09-21T13:58:35.203 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.203 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.203 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:58:35.203 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.344431+0000", 2024-09-21T13:58:35.203 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.132468163 2024-09-21T13:58:35.203 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.203 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.203 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:58:35.203 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.344717+0000", 2024-09-21T13:58:35.204 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00028593299999999999 2024-09-21T13:58:35.204 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.204 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.204 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:58:35.204 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.344765+0000", 2024-09-21T13:58:35.204 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8211000000000003e-05 2024-09-21T13:58:35.204 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.204 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.204 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:58:35.204 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.344805+0000", 2024-09-21T13:58:35.204 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0027000000000001e-05 2024-09-21T13:58:35.204 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:58:35.204 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:58:35.204 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:58:35.204 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.205 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.205 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:202104 7.7 7:e253bf76:::benchmark_data_smithi049_76798_object65684:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-21T13:58:35.205 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:44.217360+0000", 2024-09-21T13:58:35.205 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 350.90561811100002, 2024-09-21T13:58:35.205 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.15782971300000001, 2024-09-21T13:58:35.205 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:58:35.205 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:58:35.205 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:58:35.205 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:58:35.205 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:58:35.205 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:58:35.205 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 202104 2024-09-21T13:58:35.205 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.205 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:58:35.205 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.206 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:58:35.206 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.217360+0000", 2024-09-21T13:58:35.206 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:58:35.206 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.206 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.206 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:58:35.206 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.217360+0000", 2024-09-21T13:58:35.206 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:58:35.206 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.206 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.206 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:58:35.207 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:58:35.208 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:58:35.208 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T13:58:35.208 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:58:35.208 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.217361+0000", 2024-09-21T13:58:35.208 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.279e-06 2024-09-21T13:58:35.208 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.208 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.208 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:58:35.209 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.217367+0000", 2024-09-21T13:58:35.209 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2120000000000002e-06 2024-09-21T13:58:35.209 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.209 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.209 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:58:35.209 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.217379+0000", 2024-09-21T13:58:35.209 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2612000000000001e-05 2024-09-21T13:58:35.209 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.209 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.209 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:58:35.209 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.217387+0000", 2024-09-21T13:58:35.209 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8909999999999995e-06 2024-09-21T13:58:35.209 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.209 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.209 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:58:35.210 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.217429+0000", 2024-09-21T13:58:35.210 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1785000000000001e-05 2024-09-21T13:58:35.210 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.210 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.210 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:58:35.210 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.217506+0000", 2024-09-21T13:58:35.210 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7194000000000007e-05 2024-09-21T13:58:35.210 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.210 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.210 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3", 2024-09-21T13:58:35.210 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.340772+0000", 2024-09-21T13:58:35.210 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.12326546100000001 2024-09-21T13:58:35.210 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.211 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.211 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:58:35.211 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.344667+0000", 2024-09-21T13:58:35.211 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003894907 2024-09-21T13:58:35.211 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.211 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.211 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:58:35.211 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.369243+0000", 2024-09-21T13:58:35.211 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.024576745000000001 2024-09-21T13:58:35.211 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.211 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.211 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:58:35.211 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.375131+0000", 2024-09-21T13:58:35.211 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0058877240000000004 2024-09-21T13:58:35.212 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.212 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.212 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:58:35.212 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.375162+0000", 2024-09-21T13:58:35.212 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1198999999999997e-05 2024-09-21T13:58:35.212 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.212 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.212 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:58:35.212 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.375190+0000", 2024-09-21T13:58:35.212 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7704000000000001e-05 2024-09-21T13:58:35.212 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:58:35.212 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:58:35.212 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:58:35.212 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.212 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.213 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:29977 9.b 9.ed124ccb (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e269)", 2024-09-21T13:58:35.213 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:58:16.771091+0000", 2024-09-21T13:58:35.213 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 18.351887290000001, 2024-09-21T13:58:35.213 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.97926761900000003, 2024-09-21T13:58:35.213 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:58:35.213 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:58:35.213 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-21T13:58:35.213 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:58:35.213 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T13:58:35.213 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T13:58:35.213 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 29977 2024-09-21T13:58:35.213 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.213 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:58:35.213 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.213 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:58:35.214 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.771091+0000", 2024-09-21T13:58:35.214 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:58:35.214 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.214 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.214 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:58:35.214 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.771091+0000", 2024-09-21T13:58:35.214 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:58:35.214 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.214 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.214 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:58:35.214 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.771093+0000", 2024-09-21T13:58:35.214 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.606e-06 2024-09-21T13:58:35.214 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.214 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.215 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:58:35.215 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.771152+0000", 2024-09-21T13:58:35.215 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9827000000000002e-05 2024-09-21T13:58:35.215 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.215 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.215 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:58:35.215 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.771162+0000", 2024-09-21T13:58:35.215 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0010000000000001e-05 2024-09-21T13:58:35.215 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.215 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.215 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:58:35.215 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.771171+0000", 2024-09-21T13:58:35.215 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2970000000000004e-06 2024-09-21T13:58:35.215 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.216 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.216 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:58:35.216 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.771209+0000", 2024-09-21T13:58:35.216 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8182000000000001e-05 2024-09-21T13:58:35.216 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.216 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.216 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:58:35.216 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.771215+0000", 2024-09-21T13:58:35.216 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9599999999999997e-06 2024-09-21T13:58:35.216 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.216 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.216 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:58:35.216 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:17.750348+0000", 2024-09-21T13:58:35.216 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.97913300400000003 2024-09-21T13:58:35.216 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.216 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.217 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:58:35.217 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:17.750359+0000", 2024-09-21T13:58:35.217 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0733e-05 2024-09-21T13:58:35.217 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:58:35.217 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:58:35.217 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:58:35.217 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.217 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.217 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:29992 9.b 9.6daaec5b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-09-21T13:58:35.217 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:58:16.780680+0000", 2024-09-21T13:58:35.217 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 18.342298579000001, 2024-09-21T13:58:35.217 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.96971144600000003, 2024-09-21T13:58:35.217 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:58:35.217 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:58:35.218 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-21T13:58:35.218 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:58:35.218 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T13:58:35.218 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T13:58:35.218 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 29992 2024-09-21T13:58:35.218 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.218 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:58:35.218 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.218 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:58:35.218 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.780680+0000", 2024-09-21T13:58:35.218 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:58:35.218 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.218 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.218 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:58:35.218 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.780680+0000", 2024-09-21T13:58:35.219 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:58:35.219 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.219 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.219 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:58:35.219 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.780681+0000", 2024-09-21T13:58:35.219 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6509999999999999e-06 2024-09-21T13:58:35.219 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.219 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.219 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:58:35.219 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.780750+0000", 2024-09-21T13:58:35.219 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8570000000000002e-05 2024-09-21T13:58:35.219 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.219 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.219 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:58:35.219 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.780762+0000", 2024-09-21T13:58:35.220 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2109999999999999e-05 2024-09-21T13:58:35.220 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.220 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.220 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:58:35.220 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.780770+0000", 2024-09-21T13:58:35.220 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3690000000000007e-06 2024-09-21T13:58:35.220 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.220 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.220 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:58:35.220 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.780828+0000", 2024-09-21T13:58:35.220 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.749e-05 2024-09-21T13:58:35.220 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.220 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.220 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:58:35.220 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.780836+0000", 2024-09-21T13:58:35.220 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1219999999999995e-06 2024-09-21T13:58:35.221 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.221 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.221 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:58:35.221 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:17.750382+0000", 2024-09-21T13:58:35.221 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.96954600400000002 2024-09-21T13:58:35.221 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.221 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.221 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:58:35.221 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:17.750391+0000", 2024-09-21T13:58:35.221 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1300000000000007e-06 2024-09-21T13:58:35.221 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:58:35.221 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:58:35.221 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:58:35.221 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.221 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.222 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:29996 9.b 9.d0040c7b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-09-21T13:58:35.222 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:58:16.785408+0000", 2024-09-21T13:58:35.222 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 18.337570249999999, 2024-09-21T13:58:35.222 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.96553488099999996, 2024-09-21T13:58:35.222 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:58:35.222 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:58:35.222 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-21T13:58:35.222 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:58:35.222 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T13:58:35.222 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T13:58:35.222 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 29996 2024-09-21T13:58:35.222 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.222 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:58:35.222 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.223 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:58:35.223 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.785408+0000", 2024-09-21T13:58:35.223 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:58:35.223 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.223 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.223 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:58:35.223 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.785408+0000", 2024-09-21T13:58:35.223 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:58:35.223 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.223 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.223 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:58:35.223 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.785410+0000", 2024-09-21T13:58:35.223 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5030000000000001e-06 2024-09-21T13:58:35.223 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.223 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.224 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:58:35.224 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.785466+0000", 2024-09-21T13:58:35.224 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6938999999999998e-05 2024-09-21T13:58:35.224 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.224 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.224 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:58:35.224 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.785477+0000", 2024-09-21T13:58:35.224 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0543000000000001e-05 2024-09-21T13:58:35.224 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.224 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.224 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:58:35.224 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.785483+0000", 2024-09-21T13:58:35.224 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5999999999999997e-06 2024-09-21T13:58:35.224 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.224 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.225 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:58:35.225 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.785528+0000", 2024-09-21T13:58:35.225 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5286e-05 2024-09-21T13:58:35.225 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.225 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.225 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:58:35.225 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.785537+0000", 2024-09-21T13:58:35.225 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3600000000000002e-06 2024-09-21T13:58:35.225 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.225 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.225 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:58:35.225 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:17.750935+0000", 2024-09-21T13:58:35.225 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.96539792099999999 2024-09-21T13:58:35.225 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.225 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.226 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:58:35.226 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:17.750943+0000", 2024-09-21T13:58:35.226 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.729e-06 2024-09-21T13:58:35.226 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:58:35.226 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:58:35.226 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:58:35.226 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.226 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.226 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:30003 9.b 9.b7b5b7bb (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-09-21T13:58:35.226 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:58:16.789071+0000", 2024-09-21T13:58:35.226 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 18.333907530000001, 2024-09-21T13:58:35.226 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.96190145699999996, 2024-09-21T13:58:35.226 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:58:35.226 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:58:35.227 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-21T13:58:35.227 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:58:35.227 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T13:58:35.227 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T13:58:35.227 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 30003 2024-09-21T13:58:35.227 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.227 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:58:35.227 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.227 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:58:35.227 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.789071+0000", 2024-09-21T13:58:35.227 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:58:35.227 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.227 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.227 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:58:35.227 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.789071+0000", 2024-09-21T13:58:35.228 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:58:35.228 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.228 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.228 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:58:35.228 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.789074+0000", 2024-09-21T13:58:35.228 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8430000000000001e-06 2024-09-21T13:58:35.228 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.228 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.228 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:58:35.228 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.789182+0000", 2024-09-21T13:58:35.228 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000108019 2024-09-21T13:58:35.228 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.228 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.228 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:58:35.228 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.789210+0000", 2024-09-21T13:58:35.228 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8503000000000001e-05 2024-09-21T13:58:35.229 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.229 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.229 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:58:35.229 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.789226+0000", 2024-09-21T13:58:35.229 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5403000000000001e-05 2024-09-21T13:58:35.229 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.229 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.229 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:58:35.229 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.789287+0000", 2024-09-21T13:58:35.229 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1326000000000002e-05 2024-09-21T13:58:35.229 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.229 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.229 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:58:35.229 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.789293+0000", 2024-09-21T13:58:35.229 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3509999999999998e-06 2024-09-21T13:58:35.230 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.230 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.230 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:58:35.230 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:17.750965+0000", 2024-09-21T13:58:35.230 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.96167175199999999 2024-09-21T13:58:35.230 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.230 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.230 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:58:35.230 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:17.750972+0000", 2024-09-21T13:58:35.230 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2599999999999999e-06 2024-09-21T13:58:35.230 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:58:35.230 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:58:35.230 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:58:35.230 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.230 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.231 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:30009 9.b 9.13ae813b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-09-21T13:58:35.231 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:58:16.793372+0000", 2024-09-21T13:58:35.231 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 18.329606016, 2024-09-21T13:58:35.231 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.95778215200000005, 2024-09-21T13:58:35.231 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:58:35.231 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:58:35.231 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-21T13:58:35.231 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:58:35.231 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T13:58:35.231 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T13:58:35.231 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 30009 2024-09-21T13:58:35.231 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.231 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:58:35.231 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.231 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:58:35.232 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.793372+0000", 2024-09-21T13:58:35.232 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:58:35.232 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.232 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.232 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:58:35.232 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.793372+0000", 2024-09-21T13:58:35.232 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:58:35.232 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.232 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.232 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:58:35.232 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.793374+0000", 2024-09-21T13:58:35.232 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0190000000000001e-06 2024-09-21T13:58:35.232 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.232 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.232 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:58:35.233 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.793447+0000", 2024-09-21T13:58:35.233 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2768000000000005e-05 2024-09-21T13:58:35.233 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.233 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.233 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:58:35.233 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.793464+0000", 2024-09-21T13:58:35.233 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6612e-05 2024-09-21T13:58:35.233 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.233 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.233 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:58:35.233 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.793472+0000", 2024-09-21T13:58:35.233 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.276e-06 2024-09-21T13:58:35.233 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.233 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.233 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:58:35.234 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.793527+0000", 2024-09-21T13:58:35.234 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4752e-05 2024-09-21T13:58:35.234 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.234 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.234 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:58:35.234 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.793535+0000", 2024-09-21T13:58:35.234 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3890000000000005e-06 2024-09-21T13:58:35.234 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.234 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.234 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:58:35.234 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:17.751138+0000", 2024-09-21T13:58:35.234 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.95760284799999995 2024-09-21T13:58:35.234 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.234 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.234 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:58:35.235 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:17.751154+0000", 2024-09-21T13:58:35.235 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6487999999999999e-05 2024-09-21T13:58:35.235 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:58:35.235 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:58:35.235 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:58:35.235 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.235 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.235 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:30021 9.b 9.9be4c56b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-09-21T13:58:35.235 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:58:16.803723+0000", 2024-09-21T13:58:35.235 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 18.319254941000001, 2024-09-21T13:58:35.235 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.94748764900000004, 2024-09-21T13:58:35.235 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:58:35.235 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:58:35.235 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-21T13:58:35.236 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:58:35.236 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T13:58:35.236 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T13:58:35.236 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 30021 2024-09-21T13:58:35.236 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.236 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:58:35.236 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.236 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:58:35.236 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.803723+0000", 2024-09-21T13:58:35.236 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:58:35.236 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.236 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.236 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:58:35.236 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.803723+0000", 2024-09-21T13:58:35.236 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:58:35.236 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.237 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.237 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:58:35.237 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.803725+0000", 2024-09-21T13:58:35.237 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.973e-06 2024-09-21T13:58:35.237 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.237 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.237 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:58:35.237 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.803794+0000", 2024-09-21T13:58:35.237 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9092999999999997e-05 2024-09-21T13:58:35.237 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.237 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.237 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:58:35.237 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.803814+0000", 2024-09-21T13:58:35.237 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9618000000000001e-05 2024-09-21T13:58:35.237 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.237 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.238 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:58:35.238 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.803825+0000", 2024-09-21T13:58:35.238 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0788000000000001e-05 2024-09-21T13:58:35.238 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.238 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.238 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:58:35.238 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.803887+0000", 2024-09-21T13:58:35.238 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2618000000000003e-05 2024-09-21T13:58:35.238 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.238 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.238 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:58:35.238 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.803893+0000", 2024-09-21T13:58:35.238 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.874e-06 2024-09-21T13:58:35.238 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.239 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.239 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:58:35.239 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:17.751204+0000", 2024-09-21T13:58:35.239 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.94731062300000002 2024-09-21T13:58:35.239 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:58:35.239 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:58:35.239 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:58:35.239 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:17.751211+0000", 2024-09-21T13:58:35.239 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.0620000000000003e-06 2024-09-21T13:58:35.239 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:58:35.239 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:58:35.239 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:58:35.239 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:58:35.239 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:58:35.239 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T13:58:35.261 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:35.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:35.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:58:35.334 INFO:teuthology.orchestra.run.smithi187.stdout:9.e 2024-09-21T13:58:35.334 INFO:teuthology.orchestra.run.smithi187.stdout:9.c 2024-09-21T13:58:35.334 INFO:teuthology.orchestra.run.smithi187.stdout:5.10 2024-09-21T13:58:35.334 INFO:teuthology.orchestra.run.smithi187.stdout:4.a 2024-09-21T13:58:35.334 INFO:teuthology.orchestra.run.smithi187.stdout:9.9 2024-09-21T13:58:35.334 INFO:teuthology.orchestra.run.smithi187.stdout:4.5 2024-09-21T13:58:35.334 INFO:teuthology.orchestra.run.smithi187.stdout:4.11 2024-09-21T13:58:35.335 INFO:teuthology.orchestra.run.smithi187.stdout:4.1e 2024-09-21T13:58:35.335 INFO:teuthology.orchestra.run.smithi187.stdout:5.6 2024-09-21T13:58:35.335 INFO:teuthology.orchestra.run.smithi187.stdout:6.3 2024-09-21T13:58:35.335 INFO:teuthology.orchestra.run.smithi187.stdout:5.14 2024-09-21T13:58:35.335 INFO:teuthology.orchestra.run.smithi187.stdout:4.17 2024-09-21T13:58:35.335 INFO:teuthology.orchestra.run.smithi187.stdout:1.0 2024-09-21T13:58:35.335 INFO:teuthology.orchestra.run.smithi187.stdout:4.14 2024-09-21T13:58:35.335 INFO:teuthology.orchestra.run.smithi187.stdout:2.0 2024-09-21T13:58:35.335 INFO:teuthology.orchestra.run.smithi187.stdout:3.1b 2024-09-21T13:58:35.335 INFO:teuthology.orchestra.run.smithi187.stdout:3.11 2024-09-21T13:58:35.335 INFO:teuthology.orchestra.run.smithi187.stdout:3.d 2024-09-21T13:58:35.335 INFO:teuthology.orchestra.run.smithi187.stdout:5.11 2024-09-21T13:58:35.335 INFO:teuthology.orchestra.run.smithi187.stdout:5.1e 2024-09-21T13:58:35.335 INFO:teuthology.orchestra.run.smithi187.stdout:3.f 2024-09-21T13:58:35.335 INFO:teuthology.orchestra.run.smithi187.stdout:3.e 2024-09-21T13:58:35.336 INFO:teuthology.orchestra.run.smithi187.stdout:4.0 2024-09-21T13:58:35.336 INFO:teuthology.orchestra.run.smithi187.stdout:3.12 2024-09-21T13:58:35.336 INFO:teuthology.orchestra.run.smithi187.stdout:3.13 2024-09-21T13:58:35.336 INFO:teuthology.orchestra.run.smithi187.stdout:4.3 2024-09-21T13:58:35.336 INFO:teuthology.orchestra.run.smithi187.stdout:5.1f 2024-09-21T13:58:35.336 INFO:teuthology.orchestra.run.smithi187.stdout:3.16 2024-09-21T13:58:35.336 INFO:teuthology.orchestra.run.smithi187.stdout:9.d 2024-09-21T13:58:35.336 INFO:teuthology.orchestra.run.smithi187.stdout:3.9 2024-09-21T13:58:35.336 INFO:teuthology.orchestra.run.smithi187.stdout:3.8 2024-09-21T13:58:35.336 INFO:teuthology.orchestra.run.smithi187.stdout:4.8 2024-09-21T13:58:35.336 INFO:teuthology.orchestra.run.smithi187.stdout:6.15 2024-09-21T13:58:35.336 INFO:teuthology.orchestra.run.smithi187.stdout:3.14 2024-09-21T13:58:35.336 INFO:teuthology.orchestra.run.smithi187.stdout:5.17 2024-09-21T13:58:35.336 INFO:teuthology.orchestra.run.smithi187.stdout:5.19 2024-09-21T13:58:35.337 INFO:teuthology.orchestra.run.smithi187.stdout:5.1 2024-09-21T13:58:35.337 INFO:teuthology.orchestra.run.smithi187.stdout:5.4 2024-09-21T13:58:35.337 INFO:teuthology.orchestra.run.smithi187.stdout:5.9 2024-09-21T13:58:35.337 INFO:teuthology.orchestra.run.smithi187.stdout:5.d 2024-09-21T13:58:35.337 INFO:teuthology.orchestra.run.smithi187.stdout:6.14 2024-09-21T13:58:35.337 INFO:teuthology.orchestra.run.smithi187.stdout:6.6 2024-09-21T13:58:35.337 INFO:teuthology.orchestra.run.smithi187.stdout:6.16 2024-09-21T13:58:35.337 INFO:teuthology.orchestra.run.smithi187.stdout:6.18 2024-09-21T13:58:35.337 INFO:teuthology.orchestra.run.smithi187.stdout:6.1b 2024-09-21T13:58:35.337 INFO:teuthology.orchestra.run.smithi187.stdout:4.16 2024-09-21T13:58:35.337 INFO:teuthology.orchestra.run.smithi187.stdout:6.1c 2024-09-21T13:58:35.337 INFO:teuthology.orchestra.run.smithi187.stdout:6.12 2024-09-21T13:58:35.337 INFO:teuthology.orchestra.run.smithi187.stdout:9.1 2024-09-21T13:58:35.337 INFO:teuthology.orchestra.run.smithi187.stdout:3.15 2024-09-21T13:58:35.337 INFO:teuthology.orchestra.run.smithi187.stdout:6.5 2024-09-21T13:58:35.338 INFO:teuthology.orchestra.run.smithi187.stdout:5.1a 2024-09-21T13:58:35.338 INFO:teuthology.orchestra.run.smithi187.stdout:6.8 2024-09-21T13:58:35.338 INFO:teuthology.orchestra.run.smithi187.stdout:6.9 2024-09-21T13:58:35.338 INFO:teuthology.orchestra.run.smithi187.stdout:6.b 2024-09-21T13:58:35.338 INFO:teuthology.orchestra.run.smithi187.stdout:4.15 2024-09-21T13:58:35.338 INFO:teuthology.orchestra.run.smithi187.stdout:5.1b 2024-09-21T13:58:35.338 INFO:teuthology.orchestra.run.smithi187.stdout:9.3 2024-09-21T13:58:35.338 INFO:teuthology.orchestra.run.smithi187.stdout:3.7 2024-09-21T13:58:35.338 INFO:teuthology.orchestra.run.smithi187.stdout:9.5 2024-09-21T13:58:35.338 INFO:teuthology.orchestra.run.smithi187.stdout:9.7 2024-09-21T13:58:35.338 INFO:teuthology.orchestra.run.smithi187.stdout:9.a 2024-09-21T13:58:35.364 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:35.385 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:35.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:35.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:35.411 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:58:35.487 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:35.510 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:35.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:35.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:35.612 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:35.636 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:35.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:35.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:35.668 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 31 16 51164 51148 103.105 27.5625 0.00372344 0.00944944 2024-09-21T13:58:35.737 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:58:35.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:35.774 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:35.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:58:35.876 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:35.902 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:35.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:35.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:35.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:35 smithi187 ceph-mon[33032]: pgmap v479: 148 pgs: 1 active+recovering+degraded+remapped, 33 active+undersized, 2 active+recovering+undersized+remapped, 28 active+undersized+degraded, 84 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s; 26420/139101 objects degraded (18.993%); 3110/139101 objects misplaced (2.236%); 13 MiB/s, 204 objects/s recovering 2024-09-21T13:58:35.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:35 smithi187 ceph-mon[33032]: osdmap e278: 8 total, 7 up, 8 in 2024-09-21T13:58:35.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:35 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/558459966' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:58:36.004 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:58:36.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:35 smithi049 ceph-mon[30080]: pgmap v479: 148 pgs: 1 active+recovering+degraded+remapped, 33 active+undersized, 2 active+recovering+undersized+remapped, 28 active+undersized+degraded, 84 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s; 26420/139101 objects degraded (18.993%); 3110/139101 objects misplaced (2.236%); 13 MiB/s, 204 objects/s recovering 2024-09-21T13:58:36.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:35 smithi049 ceph-mon[30080]: osdmap e278: 8 total, 7 up, 8 in 2024-09-21T13:58:36.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:35 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/558459966' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:58:36.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:35 smithi049 ceph-mon[40489]: pgmap v479: 148 pgs: 1 active+recovering+degraded+remapped, 33 active+undersized, 2 active+recovering+undersized+remapped, 28 active+undersized+degraded, 84 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s; 26420/139101 objects degraded (18.993%); 3110/139101 objects misplaced (2.236%); 13 MiB/s, 204 objects/s recovering 2024-09-21T13:58:36.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:35 smithi049 ceph-mon[40489]: osdmap e278: 8 total, 7 up, 8 in 2024-09-21T13:58:36.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:35 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/558459966' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:58:36.027 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:36.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:36.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:58:36.082 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T13:58:36.083 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T13:58:36.128 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:58:36.152 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:36.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:36.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:58:36.253 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:36.276 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:36.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:36.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:36.378 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:58:36.400 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:36.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:36.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:58:36.404 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T13:58:36.502 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:36.525 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:36.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:36.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:36.627 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:58:36.649 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:36.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:36.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:58:36.669 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 32 16 51383 51367 100.311 13.6875 0.00282004 0.0097011 2024-09-21T13:58:36.698 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:58:36.751 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:36.795 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:36.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:36.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:36.869 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:36 smithi049 ceph-mon[30080]: osdmap e279: 8 total, 7 up, 8 in 2024-09-21T13:58:36.869 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:36 smithi049 ceph-mon[30080]: osdmap e280: 8 total, 7 up, 8 in 2024-09-21T13:58:36.870 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:36 smithi049 ceph-mon[40489]: osdmap e279: 8 total, 7 up, 8 in 2024-09-21T13:58:36.870 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:36 smithi049 ceph-mon[40489]: osdmap e280: 8 total, 7 up, 8 in 2024-09-21T13:58:36.896 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:36.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:36.934 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:36.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:37.035 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:37.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:37.064 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:37.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:37.077 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:36 smithi187 ceph-mon[33032]: osdmap e279: 8 total, 7 up, 8 in 2024-09-21T13:58:37.078 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:36 smithi187 ceph-mon[33032]: osdmap e280: 8 total, 7 up, 8 in 2024-09-21T13:58:37.122 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell -n osd.7 --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --op export --pgid 3.14 --file /var/log/ceph/exp.3.14.7 2024-09-21T13:58:37.165 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:58:37.233 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:37.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:37.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:58:37.335 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:58:37.382 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:37.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:37.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:58:37.399 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T13:58:37.399 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:58:37.421 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/osd.7/config 2024-09-21T13:58:37.485 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:37.531 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:37.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:37.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:37.633 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:37.669 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 33 16 52736 52720 99.8338 84.5625 0.0099895 0.0100102 2024-09-21T13:58:37.678 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:37.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:37.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:37.760 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-09-21T13:58:37.780 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:37.821 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:37.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:37.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:37.873 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:37 smithi187 ceph-mon[33032]: pgmap v482: 148 pgs: 1 active+recovering+degraded+remapped, 1 active+undersized+remapped, 33 active+undersized, 1 active+recovering+undersized+remapped, 28 active+undersized+degraded, 84 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 97 MiB/s wr, 1.56k op/s; 28130/150768 objects degraded (18.658%); 1737/150768 objects misplaced (1.152%); 19 MiB/s, 307 objects/s recovering 2024-09-21T13:58:37.873 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:37 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3200916073' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:58:37.873 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:37 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:58:37.923 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:58:37.968 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:37.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:37.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:58:38.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:37 smithi049 ceph-mon[40489]: pgmap v482: 148 pgs: 1 active+recovering+degraded+remapped, 1 active+undersized+remapped, 33 active+undersized, 1 active+recovering+undersized+remapped, 28 active+undersized+degraded, 84 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 97 MiB/s wr, 1.56k op/s; 28130/150768 objects degraded (18.658%); 1737/150768 objects misplaced (1.152%); 19 MiB/s, 307 objects/s recovering 2024-09-21T13:58:38.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:37 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3200916073' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:58:38.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:37 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:58:38.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:37 smithi049 ceph-mon[30080]: pgmap v482: 148 pgs: 1 active+recovering+degraded+remapped, 1 active+undersized+remapped, 33 active+undersized, 1 active+recovering+undersized+remapped, 28 active+undersized+degraded, 84 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 97 MiB/s wr, 1.56k op/s; 28130/150768 objects degraded (18.658%); 1737/150768 objects misplaced (1.152%); 19 MiB/s, 307 objects/s recovering 2024-09-21T13:58:38.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:37 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3200916073' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:58:38.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:37 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:58:38.070 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:38.083 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T13:58:38.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:38.118 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:38.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:38.219 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:38.275 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:38.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:38.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:38.378 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:58:38.404 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:38.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:38.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:58:38.506 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:38.549 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:38.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:38.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:38.651 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:38.669 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 34 16 54103 54087 99.4101 85.4375 0.00681254 0.0100531 2024-09-21T13:58:38.689 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:38.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:38.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:38.714 INFO:teuthology.orchestra.run.smithi187.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-21T13:58:38.791 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:38.818 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:38.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:38.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:38.920 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:38.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:38 smithi187 ceph-mon[33032]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:58:38.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:38 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T13:58:38.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:38 smithi187 ceph-mon[33032]: osdmap e281: 8 total, 7 up, 8 in 2024-09-21T13:58:38.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:38.966 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:38.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:39.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:38 smithi049 ceph-mon[40489]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:58:39.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:38 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T13:58:39.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:38 smithi049 ceph-mon[40489]: osdmap e281: 8 total, 7 up, 8 in 2024-09-21T13:58:39.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:38 smithi049 ceph-mon[30080]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:58:39.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:38 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T13:58:39.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:38 smithi049 ceph-mon[30080]: osdmap e281: 8 total, 7 up, 8 in 2024-09-21T13:58:39.067 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:58:39.092 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:39.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:39.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:58:39.194 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:39.223 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:39.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:39.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:39.325 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:39.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:39.366 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:39.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:39.467 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:58:39.506 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:39.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:39.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:58:39.608 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:39.650 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:39.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:39.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:39.669 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 35 15 55531 55516 99.1214 89.3125 0.00744957 0.0100834 2024-09-21T13:58:39.695 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:58:39 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:58:39.752 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:39.773 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:39.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:39.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:39.875 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:39.918 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:39.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:39.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:40.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:39 smithi049 ceph-mon[40489]: pgmap v485: 148 pgs: 1 active+recovering+degraded+remapped, 1 active+undersized+remapped, 33 active+undersized, 1 active+recovering+undersized+remapped, 28 active+undersized+degraded, 84 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 973 op/s; 28130/150768 objects degraded (18.658%); 1737/150768 objects misplaced (1.152%); 25 MiB/s, 402 objects/s recovering 2024-09-21T13:58:40.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:39 smithi049 ceph-mon[30080]: pgmap v485: 148 pgs: 1 active+recovering+degraded+remapped, 1 active+undersized+remapped, 33 active+undersized, 1 active+recovering+undersized+remapped, 28 active+undersized+degraded, 84 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 973 op/s; 28130/150768 objects degraded (18.658%); 1737/150768 objects misplaced (1.152%); 25 MiB/s, 402 objects/s recovering 2024-09-21T13:58:40.021 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:40.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:40.057 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:40.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:40.158 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:40.187 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:40.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:40.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:40.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:39 smithi187 ceph-mon[33032]: pgmap v485: 148 pgs: 1 active+recovering+degraded+remapped, 1 active+undersized+remapped, 33 active+undersized, 1 active+recovering+undersized+remapped, 28 active+undersized+degraded, 84 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 973 op/s; 28130/150768 objects degraded (18.658%); 1737/150768 objects misplaced (1.152%); 25 MiB/s, 402 objects/s recovering 2024-09-21T13:58:40.289 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:40.319 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:40.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:40.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:40.421 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:58:40.462 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:40.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:40.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:58:40.565 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:58:40.593 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:40.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:40.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:58:40.669 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 36 15 57043 57028 98.9928 94.5 0.00606532 0.0100964 2024-09-21T13:58:40.695 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:40.725 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:40.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:40.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:40.827 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:40.864 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:40.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:40.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:40.966 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:40.988 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:40.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:40.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:41.004 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:58:40 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:58:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:58:41.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:40 smithi049 ceph-mon[40489]: 6.a scrub starts 2024-09-21T13:58:41.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:40 smithi049 ceph-mon[40489]: 6.a scrub ok 2024-09-21T13:58:41.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:40 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:58:41.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:40 smithi049 ceph-mon[40489]: 4.12 scrub starts 2024-09-21T13:58:41.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:40 smithi049 ceph-mon[40489]: 4.12 scrub ok 2024-09-21T13:58:41.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:40 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 28130/150768 objects degraded (18.658%), 29 pgs degraded (PG_DEGRADED) 2024-09-21T13:58:41.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:40 smithi049 ceph-mon[40489]: osdmap e282: 8 total, 7 up, 8 in 2024-09-21T13:58:41.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:40 smithi049 ceph-mon[30080]: 6.a scrub starts 2024-09-21T13:58:41.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:40 smithi049 ceph-mon[30080]: 6.a scrub ok 2024-09-21T13:58:41.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:40 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:58:41.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:40 smithi049 ceph-mon[30080]: 4.12 scrub starts 2024-09-21T13:58:41.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:40 smithi049 ceph-mon[30080]: 4.12 scrub ok 2024-09-21T13:58:41.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:40 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 28130/150768 objects degraded (18.658%), 29 pgs degraded (PG_DEGRADED) 2024-09-21T13:58:41.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:40 smithi049 ceph-mon[30080]: osdmap e282: 8 total, 7 up, 8 in 2024-09-21T13:58:41.089 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:41.118 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:41.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:41.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:41.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:40 smithi187 ceph-mon[33032]: 6.a scrub starts 2024-09-21T13:58:41.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:40 smithi187 ceph-mon[33032]: 6.a scrub ok 2024-09-21T13:58:41.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:40 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:58:41.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:40 smithi187 ceph-mon[33032]: 4.12 scrub starts 2024-09-21T13:58:41.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:40 smithi187 ceph-mon[33032]: 4.12 scrub ok 2024-09-21T13:58:41.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:40 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 28130/150768 objects degraded (18.658%), 29 pgs degraded (PG_DEGRADED) 2024-09-21T13:58:41.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:40 smithi187 ceph-mon[33032]: osdmap e282: 8 total, 7 up, 8 in 2024-09-21T13:58:41.220 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:41.241 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:41.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:41.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:41.260 INFO:teuthology.orchestra.run.smithi187.stderr:Exporting 3.14 info 3.14( empty local-lis/les=269/270 n=0 ec=60/60 lis/c=269/269 les/c/f=270/270/0 sis=269) 2024-09-21T13:58:41.261 INFO:teuthology.orchestra.run.smithi187.stderr:Export successful 2024-09-21T13:58:41.343 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:58:41.370 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:41.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:41.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:58:41.472 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:41.511 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:41.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:41.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:41.613 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:41.653 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:41.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:41.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:41.669 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 37 16 58372 58356 98.5604 83 0.0132135 0.0101407 2024-09-21T13:58:41.756 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:41.799 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:41.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:41.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:41.880 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:41 smithi187 ceph-mon[33032]: pgmap v486: 148 pgs: 1 active+recovering+degraded+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 989 B/s rd, 64 MiB/s wr, 1.03k op/s; 29986/160683 objects degraded (18.662%); 1339/160683 objects misplaced (0.833%); 9.9 MiB/s, 157 objects/s recovering 2024-09-21T13:58:41.900 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:41.939 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:41.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:41.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:42.041 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:42.081 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:42.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:42.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:42.183 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:58:42.214 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:42.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:42.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:58:42.254 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell -n osd.7 --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --force --op remove --pgid 3.14 2024-09-21T13:58:42.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:41 smithi049 ceph-mon[40489]: pgmap v486: 148 pgs: 1 active+recovering+degraded+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 989 B/s rd, 64 MiB/s wr, 1.03k op/s; 29986/160683 objects degraded (18.662%); 1339/160683 objects misplaced (0.833%); 9.9 MiB/s, 157 objects/s recovering 2024-09-21T13:58:42.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:41 smithi049 ceph-mon[30080]: pgmap v486: 148 pgs: 1 active+recovering+degraded+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 989 B/s rd, 64 MiB/s wr, 1.03k op/s; 29986/160683 objects degraded (18.662%); 1339/160683 objects misplaced (0.833%); 9.9 MiB/s, 157 objects/s recovering 2024-09-21T13:58:42.316 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:42.359 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:42.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:42.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:42.460 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:42.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:42.501 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:42.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:42.571 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/osd.7/config 2024-09-21T13:58:42.602 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:42.642 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:42.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:42.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:42.669 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 38 15 59999 59984 98.6441 101.75 0.00655224 0.0101322 2024-09-21T13:58:42.744 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:42.824 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:42.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:42.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:42.925 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:58:42.938 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T13:58:42.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:42.967 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:42.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:58:42.992 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:42 smithi187 ceph-mon[33032]: 2.1 scrub starts 2024-09-21T13:58:42.992 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:42 smithi187 ceph-mon[33032]: 2.1 scrub ok 2024-09-21T13:58:43.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:42 smithi049 ceph-mon[30080]: 2.1 scrub starts 2024-09-21T13:58:43.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:42 smithi049 ceph-mon[30080]: 2.1 scrub ok 2024-09-21T13:58:43.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:42 smithi049 ceph-mon[40489]: 2.1 scrub starts 2024-09-21T13:58:43.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:42 smithi049 ceph-mon[40489]: 2.1 scrub ok 2024-09-21T13:58:43.068 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:43.073 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:58:43.124 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:43.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:43.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:43.226 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:43.277 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:43.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:43.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:43.379 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:58:43.403 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:43.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:43.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:58:43.505 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:58:43.532 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:43.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:43.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:58:43.634 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:58:43.662 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:43.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:43.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:58:43.669 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 39 16 61399 61383 98.3566 87.4375 0.00747546 0.0101622 2024-09-21T13:58:43.764 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:43.769 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:58:43.782 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:58:43.783 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:58:43.819 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:58:43.819 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:58:43.835 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:43.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:43.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:43.854 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:58:43.855 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:58:43.894 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:58:43.894 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:58:43.930 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:58:43.930 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:58:43.937 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:43.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:43 smithi187 ceph-mon[33032]: pgmap v488: 148 pgs: 1 active+recovering+degraded+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 53 MiB/s wr, 852 op/s; 31038/166104 objects degraded (18.686%); 798/166104 objects misplaced (0.480%); 9.8 MiB/s, 156 objects/s recovering 2024-09-21T13:58:43.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:43 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:58:43.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:43 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:58:43.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:43 smithi187 ceph-mon[33032]: osdmap e283: 8 total, 7 up, 8 in 2024-09-21T13:58:43.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:43 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:58:43.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:43 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:58:43.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:43 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2783964306' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:58:43.954 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:43 smithi049 ceph-mon[40489]: pgmap v488: 148 pgs: 1 active+recovering+degraded+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 53 MiB/s wr, 852 op/s; 31038/166104 objects degraded (18.686%); 798/166104 objects misplaced (0.480%); 9.8 MiB/s, 156 objects/s recovering 2024-09-21T13:58:43.954 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:43 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:58:43.955 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:43 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:58:43.955 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:43 smithi049 ceph-mon[40489]: osdmap e283: 8 total, 7 up, 8 in 2024-09-21T13:58:43.955 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:43 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:58:43.955 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:43 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:58:43.955 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:43 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2783964306' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:58:43.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:43 smithi049 ceph-mon[30080]: pgmap v488: 148 pgs: 1 active+recovering+degraded+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 53 MiB/s wr, 852 op/s; 31038/166104 objects degraded (18.686%); 798/166104 objects misplaced (0.480%); 9.8 MiB/s, 156 objects/s recovering 2024-09-21T13:58:43.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:43 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:58:43.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:43 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:58:43.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:43 smithi049 ceph-mon[30080]: osdmap e283: 8 total, 7 up, 8 in 2024-09-21T13:58:43.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:43 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:58:43.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:43 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:58:43.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:43 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2783964306' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:58:43.964 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:58:43.964 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:58:43.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:43.982 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:43.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:43.996 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:58:43.996 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:58:43.997 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T13:58:44.009 INFO:teuthology.orchestra.run.smithi187.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-21T13:58:44.083 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:58:44.132 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:44.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:44.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:58:44.234 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:44.273 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:44.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:44.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:44.375 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:44.414 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:44.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:44.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:44.516 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:44.555 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:44.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:44.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:44.641 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T13:58:44.641 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-09-21T13:58:44.641 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-09-21T13:58:44.641 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-09-21T13:58:44.641 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T13:58:44.658 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:44.670 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:2024-09-21T13:58:44.670537+0000 min lat: 0.00125866 max lat: 2.32137 avg lat: 0.0101959 2024-09-21T13:58:44.670 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T13:58:44.670 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 40 15 62753 62738 98.0145 84.6875 0.0084922 0.0101959 2024-09-21T13:58:44.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:44.702 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:44.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:44.745 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T13:58:44.803 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:44.830 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:44.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:44.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:44.931 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:44.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:44 smithi187 ceph-mon[33032]: 6.1 scrub starts 2024-09-21T13:58:44.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:44 smithi187 ceph-mon[33032]: 6.1 scrub ok 2024-09-21T13:58:44.972 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:44.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:44.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:45.073 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:45.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:45.118 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:45.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:45.218 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:45.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:44 smithi049 ceph-mon[30080]: 6.1 scrub starts 2024-09-21T13:58:45.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:44 smithi049 ceph-mon[30080]: 6.1 scrub ok 2024-09-21T13:58:45.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:44 smithi049 ceph-mon[40489]: 6.1 scrub starts 2024-09-21T13:58:45.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:44 smithi049 ceph-mon[40489]: 6.1 scrub ok 2024-09-21T13:58:45.257 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:45.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:45.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:45.359 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:45.400 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:45.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:45.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:45.502 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:58:45.511 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T13:58:45.511 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T13:58:45.533 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:45.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:45.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:58:45.635 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:58:45.669 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 41 16 64178 64162 97.7945 89 0.00851592 0.0102201 2024-09-21T13:58:45.678 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:45.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:45.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:58:45.780 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:45.820 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:45.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:45.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:45.922 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:45.964 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:45.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:45.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:46.066 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:46.109 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:46.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:46.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:46.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:45 smithi187 ceph-mon[33032]: pgmap v490: 148 pgs: 1 active+recovering+degraded+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 833 B/s rd, 90 MiB/s wr, 1.43k op/s; 33353/177165 objects degraded (18.826%); 798/177165 objects misplaced (0.450%); 9.6 MiB/s, 152 objects/s recovering 2024-09-21T13:58:46.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:45 smithi187 ceph-mon[33032]: 6.19 scrub starts 2024-09-21T13:58:46.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:45 smithi187 ceph-mon[33032]: 6.19 scrub ok 2024-09-21T13:58:46.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:45 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 33353/177165 objects degraded (18.826%), 29 pgs degraded (PG_DEGRADED) 2024-09-21T13:58:46.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:45 smithi187 ceph-mon[33032]: osdmap e284: 8 total, 7 up, 8 in 2024-09-21T13:58:46.211 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:46.250 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:46.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:46.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:46.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:45 smithi049 ceph-mon[40489]: pgmap v490: 148 pgs: 1 active+recovering+degraded+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 833 B/s rd, 90 MiB/s wr, 1.43k op/s; 33353/177165 objects degraded (18.826%); 798/177165 objects misplaced (0.450%); 9.6 MiB/s, 152 objects/s recovering 2024-09-21T13:58:46.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:45 smithi049 ceph-mon[40489]: 6.19 scrub starts 2024-09-21T13:58:46.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:45 smithi049 ceph-mon[40489]: 6.19 scrub ok 2024-09-21T13:58:46.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:45 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 33353/177165 objects degraded (18.826%), 29 pgs degraded (PG_DEGRADED) 2024-09-21T13:58:46.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:45 smithi049 ceph-mon[40489]: osdmap e284: 8 total, 7 up, 8 in 2024-09-21T13:58:46.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:45 smithi049 ceph-mon[30080]: pgmap v490: 148 pgs: 1 active+recovering+degraded+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 833 B/s rd, 90 MiB/s wr, 1.43k op/s; 33353/177165 objects degraded (18.826%); 798/177165 objects misplaced (0.450%); 9.6 MiB/s, 152 objects/s recovering 2024-09-21T13:58:46.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:45 smithi049 ceph-mon[30080]: 6.19 scrub starts 2024-09-21T13:58:46.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:45 smithi049 ceph-mon[30080]: 6.19 scrub ok 2024-09-21T13:58:46.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:45 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 33353/177165 objects degraded (18.826%), 29 pgs degraded (PG_DEGRADED) 2024-09-21T13:58:46.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:45 smithi049 ceph-mon[30080]: osdmap e284: 8 total, 7 up, 8 in 2024-09-21T13:58:46.352 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:46.389 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:46.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:46.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:46.450 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T13:58:46.478 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:58:46.491 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:46.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:46.547 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:46.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:46.648 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:46.670 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 42 16 65601 65585 97.5834 88.9375 0.0085297 0.0102426 2024-09-21T13:58:46.676 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:46.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:46.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:46.778 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:46.828 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:46.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:46.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:46.930 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:46.972 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:46.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:46.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:47.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:46 smithi049 ceph-mon[30080]: 4.f scrub starts 2024-09-21T13:58:47.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:46 smithi049 ceph-mon[30080]: 4.f scrub ok 2024-09-21T13:58:47.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:46 smithi049 ceph-mon[40489]: 4.f scrub starts 2024-09-21T13:58:47.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:46 smithi049 ceph-mon[40489]: 4.f scrub ok 2024-09-21T13:58:47.073 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:47.095 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:46 smithi187 ceph-mon[33032]: 4.f scrub starts 2024-09-21T13:58:47.095 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:46 smithi187 ceph-mon[33032]: 4.f scrub ok 2024-09-21T13:58:47.112 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:47.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:47.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:47.199 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:58:47.214 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:47.267 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:47.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:47.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:47.369 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:58:47.401 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:47.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:47.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:58:47.503 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:58:47.512 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T13:58:47.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:47.544 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:47.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:58:47.645 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:58:47.670 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 43 16 66845 66829 97.122 77.75 0.00690528 0.0102911 2024-09-21T13:58:47.673 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:47.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:47.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:58:47.775 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:58:47.816 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-09-21T13:58:47.820 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:47.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:47.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:58:47.922 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:58:47.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:47.960 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:47.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:58:48.061 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:58:48.065 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:47 smithi187 ceph-mon[33032]: pgmap v492: 148 pgs: 1 active+recovering+degraded+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 95 MiB/s wr, 1.52k op/s; 35164/188037 objects degraded (18.701%); 326/188037 objects misplaced (0.173%); 11 MiB/s, 168 objects/s recovering 2024-09-21T13:58:48.065 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:47 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3808904622' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:58:48.066 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:47 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:58:48.095 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:48.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:48.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:58:48.197 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:58:48.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:48.230 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:48.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:58:48.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:47 smithi049 ceph-mon[30080]: pgmap v492: 148 pgs: 1 active+recovering+degraded+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 95 MiB/s wr, 1.52k op/s; 35164/188037 objects degraded (18.701%); 326/188037 objects misplaced (0.173%); 11 MiB/s, 168 objects/s recovering 2024-09-21T13:58:48.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:47 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3808904622' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:58:48.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:47 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:58:48.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:47 smithi049 ceph-mon[40489]: pgmap v492: 148 pgs: 1 active+recovering+degraded+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 95 MiB/s wr, 1.52k op/s; 35164/188037 objects degraded (18.701%); 326/188037 objects misplaced (0.173%); 11 MiB/s, 168 objects/s recovering 2024-09-21T13:58:48.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:47 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3808904622' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:58:48.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:47 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:58:48.330 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:48.367 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:48.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:48.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:48.468 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:58:48.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:48.505 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:48.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:58:48.606 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:48.656 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:48.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:48.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:48.660 INFO:teuthology.orchestra.run.smithi187.stdout: marking collection for removal 2024-09-21T13:58:48.660 INFO:teuthology.orchestra.run.smithi187.stdout:setting '_remove' omap key 2024-09-21T13:58:48.660 INFO:teuthology.orchestra.run.smithi187.stdout:finish_remove_pgs 3.14_head removing 3.14 2024-09-21T13:58:48.661 INFO:teuthology.orchestra.run.smithi187.stdout:Remove successful 2024-09-21T13:58:48.670 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 44 16 68160 68144 96.7819 82.1875 0.0130299 0.0103275 2024-09-21T13:58:48.758 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:48.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:48.807 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:48.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:48.845 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:48 smithi049 ceph-mon[40489]: 6.10 scrub starts 2024-09-21T13:58:48.846 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:48 smithi049 ceph-mon[30080]: 6.10 scrub starts 2024-09-21T13:58:48.846 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:48 smithi049 ceph-mon[30080]: 6.10 scrub ok 2024-09-21T13:58:48.846 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:48 smithi049 ceph-mon[30080]: 4.1c scrub starts 2024-09-21T13:58:48.846 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:48 smithi049 ceph-mon[30080]: 4.1c scrub ok 2024-09-21T13:58:48.846 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:48 smithi049 ceph-mon[30080]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T13:58:48.846 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:48 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T13:58:48.846 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:48 smithi049 ceph-mon[30080]: osdmap e285: 8 total, 7 up, 8 in 2024-09-21T13:58:48.908 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:48.937 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T13:58:48.937 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-09-21T13:58:48.937 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-09-21T13:58:48.937 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-09-21T13:58:48.937 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.937 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:163570 7.8 7.330e1638 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e130)", 2024-09-21T13:58:48.938 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:03.335126+0000", 2024-09-21T13:58:48.938 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 405.59951809199998, 2024-09-21T13:58:48.938 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97989348499999995, 2024-09-21T13:58:48.938 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:58:48.938 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:58:48.938 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T13:58:48.938 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:58:48.938 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:58:48.938 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:58:48.938 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 163570 2024-09-21T13:58:48.938 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.938 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:58:48.938 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.938 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:58:48.939 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.335126+0000", 2024-09-21T13:58:48.939 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:58:48.939 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.939 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.939 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:58:48.939 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.335126+0000", 2024-09-21T13:58:48.939 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:58:48.939 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.939 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.939 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:58:48.939 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.335127+0000", 2024-09-21T13:58:48.939 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5209999999999999e-06 2024-09-21T13:58:48.939 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.939 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.940 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:58:48.940 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.335132+0000", 2024-09-21T13:58:48.940 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4730000000000002e-06 2024-09-21T13:58:48.940 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.940 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.940 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:58:48.940 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.335142+0000", 2024-09-21T13:58:48.940 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8770000000000005e-06 2024-09-21T13:58:48.940 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.940 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.940 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:58:48.940 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.335151+0000", 2024-09-21T13:58:48.940 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7650000000000005e-06 2024-09-21T13:58:48.940 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.941 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.941 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:58:48.941 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.336965+0000", 2024-09-21T13:58:48.941 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001813907 2024-09-21T13:58:48.941 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.941 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.941 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:58:48.941 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.336978+0000", 2024-09-21T13:58:48.941 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2685e-05 2024-09-21T13:58:48.941 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.941 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.941 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:58:48.941 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:04.315001+0000", 2024-09-21T13:58:48.941 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97802330000000004 2024-09-21T13:58:48.941 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.942 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.942 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:58:48.942 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:04.315019+0000", 2024-09-21T13:58:48.942 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7957000000000001e-05 2024-09-21T13:58:48.942 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:58:48.942 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:58:48.942 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:58:48.942 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.942 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.942 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:163585 7.8 7.9fcafe08 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-21T13:58:48.942 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:03.339560+0000", 2024-09-21T13:58:48.942 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 405.59508391700001, 2024-09-21T13:58:48.942 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97550176099999997, 2024-09-21T13:58:48.943 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:58:48.943 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:58:48.943 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T13:58:48.943 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:58:48.943 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:58:48.943 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:58:48.943 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 163585 2024-09-21T13:58:48.943 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.943 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:58:48.943 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.943 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:58:48.943 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.339560+0000", 2024-09-21T13:58:48.943 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:58:48.943 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.944 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.944 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:58:48.944 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.339560+0000", 2024-09-21T13:58:48.944 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:58:48.944 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.944 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.944 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:58:48.944 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.339562+0000", 2024-09-21T13:58:48.944 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6449999999999999e-06 2024-09-21T13:58:48.944 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.944 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.944 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:58:48.944 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.339568+0000", 2024-09-21T13:58:48.945 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2169999999999999e-06 2024-09-21T13:58:48.945 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.945 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.945 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:58:48.945 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.339580+0000", 2024-09-21T13:58:48.945 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1846e-05 2024-09-21T13:58:48.945 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.945 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.945 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:58:48.945 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.339589+0000", 2024-09-21T13:58:48.945 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.798e-06 2024-09-21T13:58:48.945 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.945 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.945 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:58:48.946 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.342967+0000", 2024-09-21T13:58:48.946 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0033776990000000001 2024-09-21T13:58:48.946 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.946 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.946 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:58:48.946 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.342980+0000", 2024-09-21T13:58:48.946 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2758000000000001e-05 2024-09-21T13:58:48.946 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.946 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.946 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:58:48.946 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:04.315052+0000", 2024-09-21T13:58:48.946 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97207221799999999 2024-09-21T13:58:48.946 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.946 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.947 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:58:48.947 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:04.315062+0000", 2024-09-21T13:58:48.947 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.5799999999999998e-06 2024-09-21T13:58:48.947 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:58:48.947 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:58:48.947 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:58:48.947 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.947 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.947 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:163595 7.8 7.9f5fc9f8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-21T13:58:48.947 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:03.360541+0000", 2024-09-21T13:58:48.947 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 405.57410274699998, 2024-09-21T13:58:48.947 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.95455601800000001, 2024-09-21T13:58:48.947 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:58:48.947 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:58:48.947 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T13:58:48.948 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:58:48.948 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:58:48.948 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:58:48.948 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 163595 2024-09-21T13:58:48.948 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.948 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:58:48.948 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.948 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:58:48.948 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.360541+0000", 2024-09-21T13:58:48.948 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:58:48.948 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.948 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.948 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:58:48.948 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.360541+0000", 2024-09-21T13:58:48.948 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:58:48.949 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.949 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.949 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:58:48.949 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.360544+0000", 2024-09-21T13:58:48.949 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.632e-06 2024-09-21T13:58:48.949 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.949 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.949 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:58:48.949 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.360554+0000", 2024-09-21T13:58:48.949 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0210000000000001e-05 2024-09-21T13:58:48.949 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.949 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.949 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:58:48.949 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.360579+0000", 2024-09-21T13:58:48.949 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4878000000000001e-05 2024-09-21T13:58:48.950 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.950 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.950 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:58:48.950 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.360589+0000", 2024-09-21T13:58:48.950 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7960000000000007e-06 2024-09-21T13:58:48.950 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.950 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.950 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:58:48.950 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.360662+0000", 2024-09-21T13:58:48.950 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3514000000000001e-05 2024-09-21T13:58:48.950 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.950 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.950 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:58:48.950 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.360673+0000", 2024-09-21T13:58:48.951 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1080000000000001e-05 2024-09-21T13:58:48.951 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.951 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.951 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:58:48.951 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:04.315088+0000", 2024-09-21T13:58:48.951 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.954414859 2024-09-21T13:58:48.951 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.951 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.952 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:48 smithi187 ceph-mon[33032]: 6.10 scrub starts 2024-09-21T13:58:48.952 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:48 smithi187 ceph-mon[33032]: 6.10 scrub ok 2024-09-21T13:58:48.952 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:48 smithi187 ceph-mon[33032]: 4.1c scrub starts 2024-09-21T13:58:48.952 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:48 smithi187 ceph-mon[33032]: 4.1c scrub ok 2024-09-21T13:58:48.952 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:48 smithi187 ceph-mon[33032]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T13:58:48.952 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:48 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T13:58:48.952 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:48 smithi187 ceph-mon[33032]: osdmap e285: 8 total, 7 up, 8 in 2024-09-21T13:58:48.952 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:58:48.952 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:04.315097+0000", 2024-09-21T13:58:48.953 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0489999999999993e-06 2024-09-21T13:58:48.953 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:58:48.953 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:58:48.953 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:58:48.953 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.953 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.953 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:163602 7.8 7.3bec8d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-21T13:58:48.953 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:03.369662+0000", 2024-09-21T13:58:48.953 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 405.56498199100002, 2024-09-21T13:58:48.953 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.945473868, 2024-09-21T13:58:48.953 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:58:48.953 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:58:48.953 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T13:58:48.954 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:58:48.954 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:58:48.954 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:58:48.954 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 163602 2024-09-21T13:58:48.954 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.954 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:58:48.954 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.954 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:58:48.954 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.369662+0000", 2024-09-21T13:58:48.954 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:58:48.954 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.954 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.954 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:58:48.955 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.369662+0000", 2024-09-21T13:58:48.955 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:58:48.955 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.955 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.955 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:58:48.955 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.369664+0000", 2024-09-21T13:58:48.955 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.136e-06 2024-09-21T13:58:48.955 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.955 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.955 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:58:48.955 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.369671+0000", 2024-09-21T13:58:48.955 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9110000000000004e-06 2024-09-21T13:58:48.955 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.956 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.956 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:58:48.956 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.369689+0000", 2024-09-21T13:58:48.956 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7757999999999998e-05 2024-09-21T13:58:48.956 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.956 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.956 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:58:48.956 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.369700+0000", 2024-09-21T13:58:48.956 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.146e-05 2024-09-21T13:58:48.956 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.956 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.956 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:58:48.956 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.369779+0000", 2024-09-21T13:58:48.956 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9260999999999999e-05 2024-09-21T13:58:48.957 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.957 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.957 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:58:48.957 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.369793+0000", 2024-09-21T13:58:48.957 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3200000000000001e-05 2024-09-21T13:58:48.957 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.957 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.957 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:58:48.957 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:04.315126+0000", 2024-09-21T13:58:48.957 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.94533302500000005 2024-09-21T13:58:48.957 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.957 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.957 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:58:48.957 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:04.315136+0000", 2024-09-21T13:58:48.958 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0117e-05 2024-09-21T13:58:48.958 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:58:48.958 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:58:48.958 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:58:48.958 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.958 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.958 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:201459 7.8 7:1473b567:::benchmark_data_smithi049_76798_object65039:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-21T13:58:48.958 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:41.960649+0000", 2024-09-21T13:58:48.958 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 366.97399482200001, 2024-09-21T13:58:48.958 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.025332715, 2024-09-21T13:58:48.958 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:58:48.958 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:58:48.958 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:58:48.959 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:58:48.959 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:58:48.959 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:58:48.959 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 201459 2024-09-21T13:58:48.959 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.959 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:58:48.959 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.959 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:58:48.959 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:41.960649+0000", 2024-09-21T13:58:48.959 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:58:48.959 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.959 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.959 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:58:48.959 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:41.960649+0000", 2024-09-21T13:58:48.959 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:58:48.960 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.960 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.960 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:58:48.960 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:41.960651+0000", 2024-09-21T13:58:48.960 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7990000000000001e-06 2024-09-21T13:58:48.960 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.960 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.960 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:58:48.960 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:41.960657+0000", 2024-09-21T13:58:48.960 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2260000000000002e-06 2024-09-21T13:58:48.960 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.960 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.960 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:58:48.960 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:41.960677+0000", 2024-09-21T13:58:48.961 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9783999999999999e-05 2024-09-21T13:58:48.961 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.961 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.961 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:58:48.961 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:41.960686+0000", 2024-09-21T13:58:48.961 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0710000000000001e-06 2024-09-21T13:58:48.961 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.961 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.961 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:58:48.961 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:41.964807+0000", 2024-09-21T13:58:48.961 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0041212560000000002 2024-09-21T13:58:48.961 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.961 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.961 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:58:48.961 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:41.964817+0000", 2024-09-21T13:58:48.962 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0030000000000001e-05 2024-09-21T13:58:48.962 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.962 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.962 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:58:48.962 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.976018+0000", 2024-09-21T13:58:48.962 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.011200412 2024-09-21T13:58:48.962 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.962 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.962 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:58:48.962 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.976136+0000", 2024-09-21T13:58:48.962 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000118106 2024-09-21T13:58:48.962 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.962 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.963 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-09-21T13:58:48.963 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.976247+0000", 2024-09-21T13:58:48.963 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000111418 2024-09-21T13:58:48.963 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.963 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.963 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:58:48.963 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.980913+0000", 2024-09-21T13:58:48.963 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0046656850000000001 2024-09-21T13:58:48.963 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.963 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.963 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:58:48.963 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.980959+0000", 2024-09-21T13:58:48.963 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6402000000000001e-05 2024-09-21T13:58:48.963 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.963 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.964 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:58:48.964 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.985848+0000", 2024-09-21T13:58:48.964 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0048883210000000002 2024-09-21T13:58:48.964 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.964 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.964 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:58:48.964 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.985906+0000", 2024-09-21T13:58:48.964 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8242000000000002e-05 2024-09-21T13:58:48.964 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.964 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.964 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:58:48.964 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.985982+0000", 2024-09-21T13:58:48.964 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5963000000000004e-05 2024-09-21T13:58:48.964 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:58:48.965 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:58:48.965 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:58:48.965 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.965 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.965 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:201494 7.8 7:1c4e9ccf:::benchmark_data_smithi049_76798_object65074:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-21T13:58:48.965 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:42.006815+0000", 2024-09-21T13:58:48.965 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 366.92782841600001, 2024-09-21T13:58:48.965 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97934932200000002, 2024-09-21T13:58:48.965 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:58:48.965 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:58:48.965 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:58:48.965 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:58:48.965 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:58:48.965 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:58:48.966 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 201494 2024-09-21T13:58:48.966 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.966 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:58:48.966 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.966 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:58:48.966 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006815+0000", 2024-09-21T13:58:48.966 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:58:48.966 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.966 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.966 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:58:48.966 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006815+0000", 2024-09-21T13:58:48.966 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:58:48.966 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.966 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.966 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:58:48.967 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006817+0000", 2024-09-21T13:58:48.967 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.618e-06 2024-09-21T13:58:48.967 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.967 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.967 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:58:48.967 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006823+0000", 2024-09-21T13:58:48.967 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0859999999999998e-06 2024-09-21T13:58:48.967 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.967 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.967 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:58:48.967 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006840+0000", 2024-09-21T13:58:48.967 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7149000000000001e-05 2024-09-21T13:58:48.967 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.967 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.968 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:58:48.968 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006848+0000", 2024-09-21T13:58:48.968 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9340000000000006e-06 2024-09-21T13:58:48.968 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.968 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.968 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:58:48.968 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006949+0000", 2024-09-21T13:58:48.968 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010043399999999999 2024-09-21T13:58:48.968 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.968 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.968 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:58:48.968 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006957+0000", 2024-09-21T13:58:48.968 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6580000000000008e-06 2024-09-21T13:58:48.968 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.968 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.969 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:58:48.969 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.979032+0000", 2024-09-21T13:58:48.969 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97207428699999998 2024-09-21T13:58:48.969 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.969 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.969 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:58:48.969 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.979198+0000", 2024-09-21T13:58:48.969 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016622800000000001 2024-09-21T13:58:48.969 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.969 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.969 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-09-21T13:58:48.969 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.979306+0000", 2024-09-21T13:58:48.969 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000108356 2024-09-21T13:58:48.969 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.970 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.970 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:58:48.970 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.980997+0000", 2024-09-21T13:58:48.970 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0016909640000000001 2024-09-21T13:58:48.970 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.970 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.970 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:58:48.970 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.981060+0000", 2024-09-21T13:58:48.970 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2516999999999998e-05 2024-09-21T13:58:48.970 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.970 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.970 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:58:48.970 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986075+0000", 2024-09-21T13:58:48.970 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.005014971 2024-09-21T13:58:48.971 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.971 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.971 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:58:48.971 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986121+0000", 2024-09-21T13:58:48.971 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6313999999999997e-05 2024-09-21T13:58:48.971 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.971 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.971 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:58:48.971 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986165+0000", 2024-09-21T13:58:48.971 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3806e-05 2024-09-21T13:58:48.971 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:58:48.971 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:58:48.971 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:58:48.971 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.971 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.972 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:201495 7.8 7:18dd0180:::benchmark_data_smithi049_76798_object65075:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-21T13:58:48.972 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:42.006872+0000", 2024-09-21T13:58:48.972 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 366.92777133700002, 2024-09-21T13:58:48.972 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97938486800000002, 2024-09-21T13:58:48.972 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:58:48.972 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:58:48.972 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:58:48.972 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:58:48.972 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:58:48.972 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:58:48.972 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 201495 2024-09-21T13:58:48.972 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.972 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:58:48.973 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.973 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:58:48.973 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006872+0000", 2024-09-21T13:58:48.973 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:58:48.973 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.973 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.973 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:58:48.973 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006872+0000", 2024-09-21T13:58:48.973 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:58:48.973 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.973 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.973 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:58:48.973 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006873+0000", 2024-09-21T13:58:48.973 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4899999999999995e-07 2024-09-21T13:58:48.974 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.974 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.974 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:58:48.974 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006878+0000", 2024-09-21T13:58:48.974 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0429999999999998e-06 2024-09-21T13:58:48.974 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.974 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.974 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:58:48.974 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006886+0000", 2024-09-21T13:58:48.974 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3399999999999998e-06 2024-09-21T13:58:48.974 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.974 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.974 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:58:48.974 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006894+0000", 2024-09-21T13:58:48.974 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1359999999999999e-06 2024-09-21T13:58:48.975 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.975 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.975 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:58:48.975 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006982+0000", 2024-09-21T13:58:48.975 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.8022000000000005e-05 2024-09-21T13:58:48.975 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.975 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.975 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:58:48.975 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006988+0000", 2024-09-21T13:58:48.975 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3470000000000004e-06 2024-09-21T13:58:48.975 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.975 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.975 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:58:48.975 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.981113+0000", 2024-09-21T13:58:48.975 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97412548499999996 2024-09-21T13:58:48.976 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.976 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.976 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:58:48.976 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.981230+0000", 2024-09-21T13:58:48.976 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000116666 2024-09-21T13:58:48.976 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.976 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.976 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-09-21T13:58:48.976 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.981349+0000", 2024-09-21T13:58:48.976 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000118572 2024-09-21T13:58:48.976 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.976 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.976 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:58:48.977 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.985825+0000", 2024-09-21T13:58:48.977 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0044766739999999999 2024-09-21T13:58:48.977 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.977 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.977 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:58:48.977 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986063+0000", 2024-09-21T13:58:48.977 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00023781299999999999 2024-09-21T13:58:48.977 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.977 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.977 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:58:48.977 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986172+0000", 2024-09-21T13:58:48.977 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010923499999999999 2024-09-21T13:58:48.977 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.977 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.978 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:58:48.978 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986236+0000", 2024-09-21T13:58:48.978 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3606000000000001e-05 2024-09-21T13:58:48.978 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.978 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.978 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:58:48.978 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986257+0000", 2024-09-21T13:58:48.978 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1379999999999999e-05 2024-09-21T13:58:48.978 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:58:48.978 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:58:48.978 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:58:48.978 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.978 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.978 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:201496 7.8 7:10f836b0:::benchmark_data_smithi049_76798_object65076:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-21T13:58:48.979 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:42.012648+0000", 2024-09-21T13:58:48.979 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 366.92199536700002, 2024-09-21T13:58:48.979 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97380752599999998, 2024-09-21T13:58:48.979 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:58:48.979 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:58:48.979 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:58:48.979 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:58:48.979 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:58:48.979 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:58:48.979 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 201496 2024-09-21T13:58:48.979 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.979 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:58:48.979 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.979 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:58:48.979 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.012648+0000", 2024-09-21T13:58:48.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:48.980 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:48.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:48.981 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:58:48.981 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.981 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.981 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:58:48.981 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.012648+0000", 2024-09-21T13:58:48.981 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:58:48.981 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.982 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.982 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:58:48.982 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.012650+0000", 2024-09-21T13:58:48.982 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.513e-06 2024-09-21T13:58:48.982 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.982 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.982 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:58:48.982 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.012656+0000", 2024-09-21T13:58:48.982 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7300000000000002e-06 2024-09-21T13:58:48.982 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.982 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.982 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:58:48.982 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.012664+0000", 2024-09-21T13:58:48.983 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6090000000000001e-06 2024-09-21T13:58:48.983 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.983 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.983 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:58:48.983 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.012670+0000", 2024-09-21T13:58:48.983 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5829999999999997e-06 2024-09-21T13:58:48.983 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.983 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.983 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:58:48.983 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.012740+0000", 2024-09-21T13:58:48.983 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9834999999999998e-05 2024-09-21T13:58:48.983 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.983 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.983 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:58:48.984 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.012749+0000", 2024-09-21T13:58:48.984 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9889999999999998e-06 2024-09-21T13:58:48.984 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.984 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.984 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:58:48.984 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.983365+0000", 2024-09-21T13:58:48.984 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97061665600000002 2024-09-21T13:58:48.984 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.984 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.984 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:58:48.984 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.983473+0000", 2024-09-21T13:58:48.984 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000107507 2024-09-21T13:58:48.984 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.984 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.985 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-09-21T13:58:48.985 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.983595+0000", 2024-09-21T13:58:48.985 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000121614 2024-09-21T13:58:48.985 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.985 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.985 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:58:48.985 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986265+0000", 2024-09-21T13:58:48.985 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002670113 2024-09-21T13:58:48.985 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.985 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.985 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:58:48.985 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986301+0000", 2024-09-21T13:58:48.985 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6130999999999997e-05 2024-09-21T13:58:48.986 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.986 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.986 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:58:48.986 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986370+0000", 2024-09-21T13:58:48.986 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8994999999999999e-05 2024-09-21T13:58:48.986 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.986 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.986 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:58:48.986 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986417+0000", 2024-09-21T13:58:48.986 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7691000000000001e-05 2024-09-21T13:58:48.986 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.986 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.986 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:58:48.986 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986456+0000", 2024-09-21T13:58:48.986 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8559999999999997e-05 2024-09-21T13:58:48.987 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:58:48.987 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:58:48.987 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:58:48.987 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.987 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.987 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:201499 7.8 7:1329afa8:::benchmark_data_smithi049_76798_object65079:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-21T13:58:48.987 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:42.026955+0000", 2024-09-21T13:58:48.987 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 366.90768879199999, 2024-09-21T13:58:48.987 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97876118000000001, 2024-09-21T13:58:48.987 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:58:48.987 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:58:48.987 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:58:48.987 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:58:48.987 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:58:48.988 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:58:48.988 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 201499 2024-09-21T13:58:48.988 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.988 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:58:48.988 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.988 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:58:48.988 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.026955+0000", 2024-09-21T13:58:48.988 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:58:48.988 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.988 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.988 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:58:48.988 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.026955+0000", 2024-09-21T13:58:48.988 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:58:48.988 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.988 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.989 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:58:48.989 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.026957+0000", 2024-09-21T13:58:48.989 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7999999999999999e-06 2024-09-21T13:58:48.989 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.989 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.989 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:58:48.989 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.026962+0000", 2024-09-21T13:58:48.989 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9400000000000001e-06 2024-09-21T13:58:48.989 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.989 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.989 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:58:48.989 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.026977+0000", 2024-09-21T13:58:48.989 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5087e-05 2024-09-21T13:58:48.989 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.989 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.990 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:58:48.990 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.026983+0000", 2024-09-21T13:58:48.990 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0090000000000004e-06 2024-09-21T13:58:48.990 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.990 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.990 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:58:48.990 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027044+0000", 2024-09-21T13:58:48.990 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1543000000000004e-05 2024-09-21T13:58:48.990 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.990 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.990 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:58:48.990 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027051+0000", 2024-09-21T13:58:48.990 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4590000000000003e-06 2024-09-21T13:58:48.991 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.991 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.991 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:58:48.991 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986512+0000", 2024-09-21T13:58:48.991 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.95946141399999996 2024-09-21T13:58:48.991 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.991 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.991 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:58:48.991 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986631+0000", 2024-09-21T13:58:48.991 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011896300000000001 2024-09-21T13:58:48.991 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.991 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.991 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-09-21T13:58:48.991 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986738+0000", 2024-09-21T13:58:48.991 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000106766 2024-09-21T13:58:48.992 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.992 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.992 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:58:48.992 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.994021+0000", 2024-09-21T13:58:48.992 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0072827329999999996 2024-09-21T13:58:48.992 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.992 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.992 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:58:48.992 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.994059+0000", 2024-09-21T13:58:48.992 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8053999999999999e-05 2024-09-21T13:58:48.992 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.992 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.992 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:58:48.992 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.005582+0000", 2024-09-21T13:58:48.992 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.011523342000000001 2024-09-21T13:58:48.993 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.993 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.993 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:58:48.993 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.005671+0000", 2024-09-21T13:58:48.993 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.8708999999999994e-05 2024-09-21T13:58:48.993 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.993 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.993 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:58:48.993 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.005716+0000", 2024-09-21T13:58:48.993 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5361000000000002e-05 2024-09-21T13:58:48.993 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:58:48.993 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:58:48.993 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:58:48.993 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.993 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.994 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:201500 7.8 7:19fe3d31:::benchmark_data_smithi049_76798_object65080:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-21T13:58:48.994 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:42.027000+0000", 2024-09-21T13:58:48.994 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 366.90764335199998, 2024-09-21T13:58:48.994 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97887608500000001, 2024-09-21T13:58:48.994 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:58:48.994 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:58:48.994 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:58:48.994 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:58:48.994 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:58:48.994 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:58:48.994 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 201500 2024-09-21T13:58:48.994 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.994 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:58:48.995 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.995 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:58:48.995 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027000+0000", 2024-09-21T13:58:48.995 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:58:48.995 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.995 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.995 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:58:48.995 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027000+0000", 2024-09-21T13:58:48.995 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:58:48.995 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.995 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.995 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:58:48.995 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027001+0000", 2024-09-21T13:58:48.995 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3200000000000001e-07 2024-09-21T13:58:48.995 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.995 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.996 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:58:48.996 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027003+0000", 2024-09-21T13:58:48.996 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8980000000000001e-06 2024-09-21T13:58:48.996 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.996 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.996 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:58:48.996 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027007+0000", 2024-09-21T13:58:48.996 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4290000000000003e-06 2024-09-21T13:58:48.996 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.996 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.996 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:58:48.996 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027010+0000", 2024-09-21T13:58:48.996 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3419999999999999e-06 2024-09-21T13:58:48.996 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.996 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.996 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:58:48.997 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027064+0000", 2024-09-21T13:58:48.997 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3474000000000003e-05 2024-09-21T13:58:48.997 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.997 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.997 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:58:48.997 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027067+0000", 2024-09-21T13:58:48.997 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.932e-06 2024-09-21T13:58:48.997 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.997 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.997 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:58:48.997 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.989128+0000", 2024-09-21T13:58:48.997 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.96206106899999999 2024-09-21T13:58:48.997 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.998 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.998 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:58:48.998 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.990066+0000", 2024-09-21T13:58:48.998 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00093830900000000002 2024-09-21T13:58:48.998 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.998 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.998 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-09-21T13:58:48.998 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.990172+0000", 2024-09-21T13:58:48.998 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000105271 2024-09-21T13:58:48.998 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.998 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.998 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:58:48.998 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.996356+0000", 2024-09-21T13:58:48.998 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0061840100000000002 2024-09-21T13:58:48.999 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.999 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.999 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:58:48.999 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.996388+0000", 2024-09-21T13:58:48.999 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2539000000000001e-05 2024-09-21T13:58:48.999 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.999 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.999 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:58:48.999 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.005774+0000", 2024-09-21T13:58:48.999 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0093859679999999997 2024-09-21T13:58:48.999 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:48.999 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:48.999 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:58:48.999 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.005838+0000", 2024-09-21T13:58:48.999 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3724000000000002e-05 2024-09-21T13:58:49.000 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.000 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.000 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:58:49.000 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.005877+0000", 2024-09-21T13:58:49.000 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8788000000000002e-05 2024-09-21T13:58:49.000 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:58:49.000 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:58:49.000 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:58:49.000 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.000 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.000 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:201501 7.8 7:15f77bbd:::benchmark_data_smithi049_76798_object65081:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-21T13:58:49.000 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:42.027294+0000", 2024-09-21T13:58:49.000 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 366.90734938700001, 2024-09-21T13:58:49.000 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97870270599999998, 2024-09-21T13:58:49.000 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:58:49.001 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:58:49.001 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:58:49.001 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:58:49.001 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:58:49.001 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:58:49.001 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 201501 2024-09-21T13:58:49.001 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.001 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:58:49.001 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.001 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:58:49.001 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027294+0000", 2024-09-21T13:58:49.001 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:58:49.001 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.001 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.001 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:58:49.002 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027294+0000", 2024-09-21T13:58:49.002 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:58:49.002 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.002 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.002 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:58:49.002 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027295+0000", 2024-09-21T13:58:49.002 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7899999999999997e-07 2024-09-21T13:58:49.002 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.002 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.002 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:58:49.002 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027299+0000", 2024-09-21T13:58:49.002 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6449999999999998e-06 2024-09-21T13:58:49.002 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.002 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.002 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:58:49.003 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027310+0000", 2024-09-21T13:58:49.003 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0699999999999999e-05 2024-09-21T13:58:49.003 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.003 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.003 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:58:49.003 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027315+0000", 2024-09-21T13:58:49.003 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9729999999999997e-06 2024-09-21T13:58:49.003 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.003 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.003 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:58:49.003 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027363+0000", 2024-09-21T13:58:49.003 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8096000000000002e-05 2024-09-21T13:58:49.003 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.004 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.004 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:58:49.004 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027367+0000", 2024-09-21T13:58:49.004 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2930000000000002e-06 2024-09-21T13:58:49.004 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.004 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.004 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:58:49.004 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.992098+0000", 2024-09-21T13:58:49.004 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.96473078800000001 2024-09-21T13:58:49.004 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.004 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.004 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:58:49.004 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.992160+0000", 2024-09-21T13:58:49.005 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2133999999999998e-05 2024-09-21T13:58:49.005 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.005 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.005 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-09-21T13:58:49.005 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.992220+0000", 2024-09-21T13:58:49.005 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9911999999999998e-05 2024-09-21T13:58:49.005 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.005 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.005 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:58:49.005 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.003058+0000", 2024-09-21T13:58:49.005 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.010838441000000001 2024-09-21T13:58:49.005 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.005 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.005 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:58:49.006 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.003121+0000", 2024-09-21T13:58:49.006 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3118000000000001e-05 2024-09-21T13:58:49.006 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.006 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.006 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:58:49.006 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.005887+0000", 2024-09-21T13:58:49.006 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0027659500000000001 2024-09-21T13:58:49.006 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.006 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.006 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:58:49.006 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.005958+0000", 2024-09-21T13:58:49.006 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0842999999999999e-05 2024-09-21T13:58:49.006 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.006 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.007 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:58:49.007 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.005997+0000", 2024-09-21T13:58:49.007 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9033999999999999e-05 2024-09-21T13:58:49.007 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:58:49.007 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:58:49.007 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:58:49.007 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.007 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.007 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:201503 7.8 7:1bb4326b:::benchmark_data_smithi049_76798_object65083:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-21T13:58:49.007 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:42.027327+0000", 2024-09-21T13:58:49.007 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 366.90731731099999, 2024-09-21T13:58:49.007 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97877131900000003, 2024-09-21T13:58:49.007 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:58:49.008 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:58:49.008 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:58:49.008 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:58:49.008 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:58:49.008 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:58:49.008 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 201503 2024-09-21T13:58:49.008 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.009 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:58:49.009 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.009 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:58:49.009 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027327+0000", 2024-09-21T13:58:49.009 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:58:49.009 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.009 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.009 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:58:49.009 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027327+0000", 2024-09-21T13:58:49.009 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:58:49.009 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.009 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.009 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:58:49.009 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027327+0000", 2024-09-21T13:58:49.010 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4799999999999999e-07 2024-09-21T13:58:49.010 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.010 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.010 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:58:49.010 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027329+0000", 2024-09-21T13:58:49.010 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7400000000000001e-06 2024-09-21T13:58:49.010 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.010 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.010 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:58:49.010 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027333+0000", 2024-09-21T13:58:49.010 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0790000000000002e-06 2024-09-21T13:58:49.010 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.011 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.011 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:58:49.011 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027336+0000", 2024-09-21T13:58:49.011 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1190000000000001e-06 2024-09-21T13:58:49.011 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.011 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.011 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:58:49.011 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027379+0000", 2024-09-21T13:58:49.011 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3000000000000002e-05 2024-09-21T13:58:49.011 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.011 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.011 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:58:49.011 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027382+0000", 2024-09-21T13:58:49.011 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9399999999999998e-06 2024-09-21T13:58:49.012 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.012 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.012 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:58:49.012 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.994085+0000", 2024-09-21T13:58:49.012 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.96670302799999996 2024-09-21T13:58:49.012 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.012 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.012 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:58:49.012 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.994181+0000", 2024-09-21T13:58:49.012 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6340000000000003e-05 2024-09-21T13:58:49.012 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.012 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.012 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-09-21T13:58:49.012 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.994285+0000", 2024-09-21T13:58:49.012 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000103947 2024-09-21T13:58:49.013 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.013 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.013 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:58:49.013 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.003091+0000", 2024-09-21T13:58:49.013 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0088057599999999993 2024-09-21T13:58:49.013 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.013 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.013 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:58:49.013 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.003150+0000", 2024-09-21T13:58:49.013 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9670000000000003e-05 2024-09-21T13:58:49.013 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.013 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.013 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:58:49.014 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.006006+0000", 2024-09-21T13:58:49.014 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002855734 2024-09-21T13:58:49.014 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.014 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.014 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:58:49.014 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.006056+0000", 2024-09-21T13:58:49.014 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9619e-05 2024-09-21T13:58:49.014 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.014 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.014 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:58:49.014 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.006098+0000", 2024-09-21T13:58:49.014 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1995e-05 2024-09-21T13:58:49.014 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:58:49.014 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:58:49.015 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:58:49.015 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.015 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.015 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:201514 7.8 7:1c8e529c:::benchmark_data_smithi049_76798_object65094:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-21T13:58:49.015 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:42.044200+0000", 2024-09-21T13:58:49.015 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 366.890444204, 2024-09-21T13:58:49.015 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.96199924699999995, 2024-09-21T13:58:49.015 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:58:49.015 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:58:49.015 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:58:49.015 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:58:49.015 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:58:49.015 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:58:49.015 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 201514 2024-09-21T13:58:49.016 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.016 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:58:49.016 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.016 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:58:49.016 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.044200+0000", 2024-09-21T13:58:49.016 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:58:49.016 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.016 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.016 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:58:49.016 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.044200+0000", 2024-09-21T13:58:49.016 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:58:49.016 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.016 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.016 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:58:49.016 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.044201+0000", 2024-09-21T13:58:49.017 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7179999999999999e-06 2024-09-21T13:58:49.017 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.017 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.017 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:58:49.017 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.044208+0000", 2024-09-21T13:58:49.017 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1800000000000001e-06 2024-09-21T13:58:49.017 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.017 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.017 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:58:49.017 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.044226+0000", 2024-09-21T13:58:49.017 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8216999999999999e-05 2024-09-21T13:58:49.017 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.017 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.017 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:58:49.017 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.044235+0000", 2024-09-21T13:58:49.018 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1020000000000003e-06 2024-09-21T13:58:49.018 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.018 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.018 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:58:49.018 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.044482+0000", 2024-09-21T13:58:49.018 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00024732699999999999 2024-09-21T13:58:49.018 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.018 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.018 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:58:49.018 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.044499+0000", 2024-09-21T13:58:49.018 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7129000000000002e-05 2024-09-21T13:58:49.018 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.018 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.018 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:58:49.019 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.996416+0000", 2024-09-21T13:58:49.019 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.95191657200000002 2024-09-21T13:58:49.019 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.019 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.019 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:58:49.019 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.996528+0000", 2024-09-21T13:58:49.019 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000112118 2024-09-21T13:58:49.019 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.019 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.019 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-09-21T13:58:49.019 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.996652+0000", 2024-09-21T13:58:49.019 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012430500000000001 2024-09-21T13:58:49.019 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.019 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.020 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:58:49.020 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.003180+0000", 2024-09-21T13:58:49.020 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.006528056 2024-09-21T13:58:49.020 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.020 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.020 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:58:49.020 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.005760+0000", 2024-09-21T13:58:49.020 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0025797670000000002 2024-09-21T13:58:49.020 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.020 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.020 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:58:49.020 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.006108+0000", 2024-09-21T13:58:49.020 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00034834500000000001 2024-09-21T13:58:49.020 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.020 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.021 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:58:49.021 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.006155+0000", 2024-09-21T13:58:49.021 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6513e-05 2024-09-21T13:58:49.021 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.021 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.021 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:58:49.021 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.006199+0000", 2024-09-21T13:58:49.021 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3897999999999999e-05 2024-09-21T13:58:49.021 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:58:49.021 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:58:49.021 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:58:49.021 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.021 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.021 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:201539 7.8 7:1afa34ae:::benchmark_data_smithi049_76798_object65119:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-21T13:58:49.021 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:42.081266+0000", 2024-09-21T13:58:49.022 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 366.85337809599997, 2024-09-21T13:58:49.022 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.92917915600000001, 2024-09-21T13:58:49.022 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:58:49.022 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:58:49.022 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:58:49.022 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:58:49.022 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:58:49.022 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:58:49.022 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 201539 2024-09-21T13:58:49.022 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.022 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:58:49.022 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.022 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:58:49.022 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.081266+0000", 2024-09-21T13:58:49.022 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:58:49.023 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.023 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.023 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:58:49.023 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.081266+0000", 2024-09-21T13:58:49.023 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:58:49.023 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.023 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.023 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:58:49.023 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.081267+0000", 2024-09-21T13:58:49.023 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6559999999999999e-06 2024-09-21T13:58:49.023 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.023 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.023 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:58:49.023 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.081273+0000", 2024-09-21T13:58:49.024 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0839999999999997e-06 2024-09-21T13:58:49.024 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.024 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.024 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:58:49.024 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.081291+0000", 2024-09-21T13:58:49.024 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7827999999999999e-05 2024-09-21T13:58:49.024 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.024 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.024 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:58:49.024 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.081300+0000", 2024-09-21T13:58:49.024 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.7190000000000007e-06 2024-09-21T13:58:49.024 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.024 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.025 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:58:49.025 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.081360+0000", 2024-09-21T13:58:49.025 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9914000000000003e-05 2024-09-21T13:58:49.025 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.025 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.025 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:58:49.025 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.081369+0000", 2024-09-21T13:58:49.025 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1630000000000002e-06 2024-09-21T13:58:49.025 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.025 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.025 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:58:49.025 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.998700+0000", 2024-09-21T13:58:49.025 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.91733049300000002 2024-09-21T13:58:49.025 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.025 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.025 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:58:49.026 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.998798+0000", 2024-09-21T13:58:49.026 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8494999999999998e-05 2024-09-21T13:58:49.026 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.026 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.026 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-09-21T13:58:49.026 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.998903+0000", 2024-09-21T13:58:49.026 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000104782 2024-09-21T13:58:49.026 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.026 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.026 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:58:49.026 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.006208+0000", 2024-09-21T13:58:49.026 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0073052270000000001 2024-09-21T13:58:49.026 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.026 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.027 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:58:49.027 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.006259+0000", 2024-09-21T13:58:49.027 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1205999999999997e-05 2024-09-21T13:58:49.027 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.027 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.027 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:58:49.027 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.010350+0000", 2024-09-21T13:58:49.027 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0040908560000000004 2024-09-21T13:58:49.027 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.027 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.027 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:58:49.027 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.010400+0000", 2024-09-21T13:58:49.027 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9901000000000002e-05 2024-09-21T13:58:49.027 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.027 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.028 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:58:49.028 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.010445+0000", 2024-09-21T13:58:49.028 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4832e-05 2024-09-21T13:58:49.028 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:58:49.028 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:58:49.028 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:58:49.028 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.028 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.028 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:201541 7.8 7:135037ad:::benchmark_data_smithi049_76798_object65121:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-21T13:58:49.028 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:42.083222+0000", 2024-09-21T13:58:49.028 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 366.851421845, 2024-09-21T13:58:49.028 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.92733551700000005, 2024-09-21T13:58:49.028 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:58:49.028 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:58:49.028 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:58:49.029 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:58:49.029 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:58:49.029 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:58:49.029 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 201541 2024-09-21T13:58:49.029 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.029 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:58:49.029 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.029 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:58:49.029 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.083222+0000", 2024-09-21T13:58:49.029 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:58:49.029 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.029 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.029 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:58:49.029 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.083222+0000", 2024-09-21T13:58:49.029 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:58:49.030 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.030 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.030 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:58:49.030 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.083223+0000", 2024-09-21T13:58:49.030 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0020000000000001e-06 2024-09-21T13:58:49.030 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.030 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.030 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:58:49.030 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.083227+0000", 2024-09-21T13:58:49.030 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.636e-06 2024-09-21T13:58:49.030 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.030 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.030 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:58:49.030 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.083236+0000", 2024-09-21T13:58:49.030 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0180000000000007e-06 2024-09-21T13:58:49.031 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.031 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.031 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:58:49.031 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.083242+0000", 2024-09-21T13:58:49.031 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.891e-06 2024-09-21T13:58:49.031 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.031 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.031 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:58:49.031 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.083385+0000", 2024-09-21T13:58:49.031 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014374099999999999 2024-09-21T13:58:49.031 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.031 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.031 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:58:49.031 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.083394+0000", 2024-09-21T13:58:49.031 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0839999999999998e-06 2024-09-21T13:58:49.032 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.032 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.032 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:58:49.032 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.000839+0000", 2024-09-21T13:58:49.032 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.917444861 2024-09-21T13:58:49.032 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.032 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.032 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:58:49.032 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.000934+0000", 2024-09-21T13:58:49.032 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4943000000000002e-05 2024-09-21T13:58:49.032 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.032 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.032 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-09-21T13:58:49.032 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.001036+0000", 2024-09-21T13:58:49.033 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000101817 2024-09-21T13:58:49.033 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.033 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.033 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:58:49.033 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.006215+0000", 2024-09-21T13:58:49.033 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0051790020000000003 2024-09-21T13:58:49.033 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.033 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.033 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:58:49.033 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.006324+0000", 2024-09-21T13:58:49.033 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010858599999999999 2024-09-21T13:58:49.033 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.033 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.033 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:58:49.033 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.010481+0000", 2024-09-21T13:58:49.034 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0041575300000000004 2024-09-21T13:58:49.034 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.034 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.034 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:58:49.034 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.010528+0000", 2024-09-21T13:58:49.034 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6418000000000002e-05 2024-09-21T13:58:49.034 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.034 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.034 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:58:49.034 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.010558+0000", 2024-09-21T13:58:49.034 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9988000000000002e-05 2024-09-21T13:58:49.034 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:58:49.034 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:58:49.035 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:58:49.035 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.035 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.035 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:229964 7.e 7:7be4d3ef:::benchmark_data_smithi049_76798_object93544:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-21T13:58:49.035 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:53:16.693433+0000", 2024-09-21T13:58:49.035 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 332.24121128000002, 2024-09-21T13:58:49.035 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.99823727600000001, 2024-09-21T13:58:49.036 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:58:49.036 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:58:49.036 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:58:49.036 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:58:49.036 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:58:49.036 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:58:49.036 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 229964 2024-09-21T13:58:49.036 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.036 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:58:49.036 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.036 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:58:49.036 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.693433+0000", 2024-09-21T13:58:49.036 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:58:49.036 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.036 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.037 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:58:49.037 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.693433+0000", 2024-09-21T13:58:49.037 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:58:49.037 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.037 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.037 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:58:49.037 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.693434+0000", 2024-09-21T13:58:49.037 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3799999999999999e-06 2024-09-21T13:58:49.037 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.037 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.037 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:58:49.037 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.693438+0000", 2024-09-21T13:58:49.037 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.515e-06 2024-09-21T13:58:49.037 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.037 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.038 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:58:49.038 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.693455+0000", 2024-09-21T13:58:49.038 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6614000000000001e-05 2024-09-21T13:58:49.038 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.038 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.038 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:58:49.038 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.693463+0000", 2024-09-21T13:58:49.038 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9850000000000006e-06 2024-09-21T13:58:49.038 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.038 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.038 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:58:49.038 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.693542+0000", 2024-09-21T13:58:49.038 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8942000000000004e-05 2024-09-21T13:58:49.038 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.038 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.039 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:58:49.039 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.693550+0000", 2024-09-21T13:58:49.039 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7379999999999994e-06 2024-09-21T13:58:49.039 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.039 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.039 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:58:49.039 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.686321+0000", 2024-09-21T13:58:49.039 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.99277167399999999 2024-09-21T13:58:49.039 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.039 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.039 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:58:49.039 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.686413+0000", 2024-09-21T13:58:49.039 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1822000000000003e-05 2024-09-21T13:58:49.039 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.039 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.040 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,5", 2024-09-21T13:58:49.040 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.686547+0000", 2024-09-21T13:58:49.040 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000133516 2024-09-21T13:58:49.040 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.040 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.040 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:58:49.040 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.691512+0000", 2024-09-21T13:58:49.040 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0049651410000000002 2024-09-21T13:58:49.040 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.040 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.040 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:58:49.040 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.691558+0000", 2024-09-21T13:58:49.040 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6047999999999997e-05 2024-09-21T13:58:49.040 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.041 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.041 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:58:49.041 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.691610+0000", 2024-09-21T13:58:49.041 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1644000000000002e-05 2024-09-21T13:58:49.041 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.041 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.041 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:58:49.041 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.691642+0000", 2024-09-21T13:58:49.041 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2542000000000001e-05 2024-09-21T13:58:49.041 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.041 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.041 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:58:49.041 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.691670+0000", 2024-09-21T13:58:49.041 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.7715000000000001e-05 2024-09-21T13:58:49.041 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:58:49.041 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:58:49.042 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:58:49.042 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.042 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.042 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:230040 7.e 7:74df0b23:::benchmark_data_smithi049_76798_object93620:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-21T13:58:49.042 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:53:16.758488+0000", 2024-09-21T13:58:49.042 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 332.17615622300002, 2024-09-21T13:58:49.042 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.93513229499999995, 2024-09-21T13:58:49.042 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:58:49.042 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:58:49.042 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:58:49.042 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:58:49.042 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:58:49.042 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:58:49.043 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 230040 2024-09-21T13:58:49.043 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.043 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:58:49.043 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.043 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:58:49.043 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.758488+0000", 2024-09-21T13:58:49.043 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:58:49.043 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.043 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.043 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:58:49.043 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.758488+0000", 2024-09-21T13:58:49.043 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:58:49.043 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.043 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.043 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:58:49.044 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.758502+0000", 2024-09-21T13:58:49.044 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4207e-05 2024-09-21T13:58:49.044 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.044 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.044 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:58:49.044 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.758507+0000", 2024-09-21T13:58:49.044 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9429999999999999e-06 2024-09-21T13:58:49.044 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.044 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.044 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:58:49.044 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.758518+0000", 2024-09-21T13:58:49.044 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1447999999999999e-05 2024-09-21T13:58:49.044 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.045 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.045 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:58:49.045 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.758525+0000", 2024-09-21T13:58:49.045 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2389999999999996e-06 2024-09-21T13:58:49.045 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.045 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.045 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:58:49.045 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.758589+0000", 2024-09-21T13:58:49.045 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3262000000000005e-05 2024-09-21T13:58:49.045 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.045 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.045 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:58:49.045 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.758597+0000", 2024-09-21T13:58:49.045 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4100000000000008e-06 2024-09-21T13:58:49.046 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.046 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.046 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:58:49.046 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.689523+0000", 2024-09-21T13:58:49.046 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.93092584099999998 2024-09-21T13:58:49.046 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.046 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.046 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:58:49.046 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.689620+0000", 2024-09-21T13:58:49.046 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7081000000000001e-05 2024-09-21T13:58:49.046 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.046 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.046 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,5", 2024-09-21T13:58:49.046 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.689705+0000", 2024-09-21T13:58:49.046 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5074000000000006e-05 2024-09-21T13:58:49.046 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.047 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.047 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:58:49.047 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.691602+0000", 2024-09-21T13:58:49.047 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001896951 2024-09-21T13:58:49.047 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.047 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.047 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:58:49.047 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.693564+0000", 2024-09-21T13:58:49.047 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0019616630000000002 2024-09-21T13:58:49.047 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.047 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.047 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:58:49.047 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.693572+0000", 2024-09-21T13:58:49.047 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9160000000000001e-06 2024-09-21T13:58:49.048 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.048 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.048 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:58:49.048 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.693597+0000", 2024-09-21T13:58:49.048 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5812999999999998e-05 2024-09-21T13:58:49.048 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.048 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.048 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:58:49.048 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.693620+0000", 2024-09-21T13:58:49.048 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2447e-05 2024-09-21T13:58:49.048 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:58:49.048 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:58:49.048 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:58:49.048 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.048 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.048 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:230048 7.e 7:7e0d246c:::benchmark_data_smithi049_76798_object93628:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-21T13:58:49.049 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:53:16.763844+0000", 2024-09-21T13:58:49.049 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 332.17079934399999, 2024-09-21T13:58:49.049 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.93491194899999996, 2024-09-21T13:58:49.049 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:58:49.049 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:58:49.049 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:58:49.049 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:58:49.049 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:58:49.049 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:58:49.049 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 230048 2024-09-21T13:58:49.049 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.049 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:58:49.049 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.049 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:58:49.049 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.763844+0000", 2024-09-21T13:58:49.049 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:58:49.050 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.050 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.050 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:58:49.050 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.763844+0000", 2024-09-21T13:58:49.050 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:58:49.050 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.050 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.050 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:58:49.050 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.763846+0000", 2024-09-21T13:58:49.050 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6330000000000001e-06 2024-09-21T13:58:49.050 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.050 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.050 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:58:49.050 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.763852+0000", 2024-09-21T13:58:49.050 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3839999999999996e-06 2024-09-21T13:58:49.050 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.051 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.051 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:58:49.051 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.763915+0000", 2024-09-21T13:58:49.051 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3306999999999995e-05 2024-09-21T13:58:49.051 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.051 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.051 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:58:49.051 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.764047+0000", 2024-09-21T13:58:49.051 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013207599999999999 2024-09-21T13:58:49.051 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.051 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.051 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:58:49.051 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.764104+0000", 2024-09-21T13:58:49.051 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6758000000000001e-05 2024-09-21T13:58:49.051 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.052 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.052 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:58:49.052 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.764113+0000", 2024-09-21T13:58:49.052 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3700000000000001e-06 2024-09-21T13:58:49.052 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.052 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.052 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:58:49.052 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.691702+0000", 2024-09-21T13:58:49.052 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.92758885899999999 2024-09-21T13:58:49.052 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.052 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.052 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:58:49.052 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.691822+0000", 2024-09-21T13:58:49.052 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011997699999999999 2024-09-21T13:58:49.052 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.053 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.053 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,5", 2024-09-21T13:58:49.053 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.691939+0000", 2024-09-21T13:58:49.053 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011752 2024-09-21T13:58:49.053 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.053 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.053 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:58:49.053 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.698556+0000", 2024-09-21T13:58:49.053 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0066170739999999997 2024-09-21T13:58:49.053 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.053 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.053 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:58:49.053 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.698612+0000", 2024-09-21T13:58:49.053 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5362000000000003e-05 2024-09-21T13:58:49.053 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.053 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.054 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:58:49.054 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.698659+0000", 2024-09-21T13:58:49.054 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6931999999999997e-05 2024-09-21T13:58:49.054 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.054 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.054 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:58:49.054 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.698714+0000", 2024-09-21T13:58:49.054 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5402000000000003e-05 2024-09-21T13:58:49.054 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.054 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.054 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:58:49.054 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.698756+0000", 2024-09-21T13:58:49.054 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2295000000000001e-05 2024-09-21T13:58:49.054 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:58:49.055 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:58:49.055 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:58:49.055 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.055 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.055 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:230051 7.e 7:71fa5c2a:::benchmark_data_smithi049_76798_object93631:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-21T13:58:49.055 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:53:16.764458+0000", 2024-09-21T13:58:49.055 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 332.17018589899999, 2024-09-21T13:58:49.055 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.94179994899999997, 2024-09-21T13:58:49.055 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:58:49.055 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:58:49.055 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:58:49.055 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:58:49.055 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:58:49.055 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:58:49.055 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 230051 2024-09-21T13:58:49.055 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.056 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:58:49.056 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.056 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:58:49.056 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.764458+0000", 2024-09-21T13:58:49.056 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:58:49.056 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.056 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.056 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:58:49.056 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.764458+0000", 2024-09-21T13:58:49.056 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:58:49.056 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.056 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.056 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:58:49.056 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.764459+0000", 2024-09-21T13:58:49.056 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9400000000000004e-07 2024-09-21T13:58:49.056 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.057 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.057 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:58:49.057 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.764463+0000", 2024-09-21T13:58:49.057 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1960000000000001e-06 2024-09-21T13:58:49.057 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.057 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.057 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:58:49.057 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.764477+0000", 2024-09-21T13:58:49.057 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4426e-05 2024-09-21T13:58:49.057 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.057 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.057 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:58:49.057 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.764485+0000", 2024-09-21T13:58:49.057 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.148e-06 2024-09-21T13:58:49.057 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.058 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.058 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:58:49.058 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.764545+0000", 2024-09-21T13:58:49.058 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0278999999999997e-05 2024-09-21T13:58:49.058 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.058 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.058 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:58:49.058 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.764553+0000", 2024-09-21T13:58:49.058 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9170000000000006e-06 2024-09-21T13:58:49.058 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.058 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.058 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:58:49.058 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.693660+0000", 2024-09-21T13:58:49.058 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.92910747900000001 2024-09-21T13:58:49.058 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.059 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.059 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:58:49.059 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.693785+0000", 2024-09-21T13:58:49.059 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012436000000000001 2024-09-21T13:58:49.059 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.059 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.059 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,5", 2024-09-21T13:58:49.059 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.693907+0000", 2024-09-21T13:58:49.059 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012208600000000001 2024-09-21T13:58:49.059 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.059 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.059 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:58:49.059 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.698816+0000", 2024-09-21T13:58:49.059 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0049091279999999996 2024-09-21T13:58:49.060 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.060 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.060 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:58:49.060 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.698834+0000", 2024-09-21T13:58:49.060 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8236999999999998e-05 2024-09-21T13:58:49.060 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.060 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.060 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:58:49.060 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.706148+0000", 2024-09-21T13:58:49.060 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0073139709999999998 2024-09-21T13:58:49.060 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.060 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.060 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:58:49.060 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.706202+0000", 2024-09-21T13:58:49.060 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3779000000000001e-05 2024-09-21T13:58:49.060 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.061 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.061 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:58:49.061 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.706258+0000", 2024-09-21T13:58:49.061 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6048999999999998e-05 2024-09-21T13:58:49.061 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:58:49.061 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:58:49.061 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:58:49.061 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.061 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.061 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:230060 7.e 7:7289055b:::benchmark_data_smithi049_76798_object93640:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-21T13:58:49.061 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:53:16.772333+0000", 2024-09-21T13:58:49.061 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 332.16231077899999, 2024-09-21T13:58:49.061 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.93409186700000002, 2024-09-21T13:58:49.061 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:58:49.061 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:58:49.061 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:58:49.062 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:58:49.062 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T13:58:49.062 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:58:49.062 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 230060 2024-09-21T13:58:49.062 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.062 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:58:49.062 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.062 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:58:49.062 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.772333+0000", 2024-09-21T13:58:49.062 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:58:49.062 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.062 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.062 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:58:49.062 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.772333+0000", 2024-09-21T13:58:49.062 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:58:49.063 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.063 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.063 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:58:49.063 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.772334+0000", 2024-09-21T13:58:49.063 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1540000000000001e-06 2024-09-21T13:58:49.063 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.063 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.063 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:58:49.063 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.772339+0000", 2024-09-21T13:58:49.063 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8099999999999997e-06 2024-09-21T13:58:49.063 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.063 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.063 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:58:49.063 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.772351+0000", 2024-09-21T13:58:49.064 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2435999999999999e-05 2024-09-21T13:58:49.064 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.064 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.064 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:58:49.064 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.772359+0000", 2024-09-21T13:58:49.064 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7230000000000004e-06 2024-09-21T13:58:49.064 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.064 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.064 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:58:49.064 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.772435+0000", 2024-09-21T13:58:49.064 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5465999999999999e-05 2024-09-21T13:58:49.064 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.064 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.064 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:58:49.065 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.772443+0000", 2024-09-21T13:58:49.065 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9160000000000001e-06 2024-09-21T13:58:49.065 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.065 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.065 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:58:49.065 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.695975+0000", 2024-09-21T13:58:49.065 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.92353224599999995 2024-09-21T13:58:49.065 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.065 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.065 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:58:49.065 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.696104+0000", 2024-09-21T13:58:49.065 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012895499999999999 2024-09-21T13:58:49.065 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.065 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.065 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,5", 2024-09-21T13:58:49.066 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.696217+0000", 2024-09-21T13:58:49.066 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011362399999999999 2024-09-21T13:58:49.066 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.066 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.066 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:58:49.066 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.701076+0000", 2024-09-21T13:58:49.066 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0048582729999999998 2024-09-21T13:58:49.066 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.066 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.066 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:58:49.066 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.706310+0000", 2024-09-21T13:58:49.066 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0052347419999999997 2024-09-21T13:58:49.066 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.067 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.067 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:58:49.067 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.706347+0000", 2024-09-21T13:58:49.067 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6492999999999998e-05 2024-09-21T13:58:49.067 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.067 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.067 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:58:49.067 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.706392+0000", 2024-09-21T13:58:49.067 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5148999999999998e-05 2024-09-21T13:58:49.067 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:58:49.067 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:58:49.067 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:58:49.067 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.706425+0000", 2024-09-21T13:58:49.067 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2879999999999997e-05 2024-09-21T13:58:49.067 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:58:49.067 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:58:49.068 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:58:49.068 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:58:49.068 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:58:49.068 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T13:58:49.082 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:58:49.114 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:49.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:49.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:58:49.211 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:48 smithi049 ceph-mon[40489]: 6.10 scrub ok 2024-09-21T13:58:49.211 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:48 smithi049 ceph-mon[40489]: 4.1c scrub starts 2024-09-21T13:58:49.211 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:48 smithi049 ceph-mon[40489]: 4.1c scrub ok 2024-09-21T13:58:49.211 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:48 smithi049 ceph-mon[40489]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T13:58:49.211 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:48 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T13:58:49.211 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:48 smithi049 ceph-mon[40489]: osdmap e285: 8 total, 7 up, 8 in 2024-09-21T13:58:49.216 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:49.257 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:49.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:49.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:49.359 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:58:49.383 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:49.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:49.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:58:49.485 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:58:49.520 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:49.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:49.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:58:49.622 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:49.665 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:49.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:49.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:49.670 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 45 16 69564 69548 96.581 87.75 0.0270353 0.0103484 2024-09-21T13:58:49.767 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:58:49.794 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:49.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:49.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:58:49.842 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:58:49 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:58:49.896 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:49.938 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:49.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:49.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:49.961 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell -n osd.7 --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.3.14.7 2024-09-21T13:58:50.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:49 smithi049 ceph-mon[30080]: pgmap v494: 148 pgs: 1 active+recovering+degraded+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 76 MiB/s wr, 1.22k op/s; 35164/188037 objects degraded (18.701%); 326/188037 objects misplaced (0.173%); 4.9 MiB/s, 78 objects/s recovering 2024-09-21T13:58:50.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:49 smithi049 ceph-mon[30080]: 3.5 scrub starts 2024-09-21T13:58:50.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:49 smithi049 ceph-mon[30080]: 3.5 scrub ok 2024-09-21T13:58:50.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:49 smithi049 ceph-mon[40489]: pgmap v494: 148 pgs: 1 active+recovering+degraded+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 76 MiB/s wr, 1.22k op/s; 35164/188037 objects degraded (18.701%); 326/188037 objects misplaced (0.173%); 4.9 MiB/s, 78 objects/s recovering 2024-09-21T13:58:50.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:49 smithi049 ceph-mon[40489]: 3.5 scrub starts 2024-09-21T13:58:50.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:49 smithi049 ceph-mon[40489]: 3.5 scrub ok 2024-09-21T13:58:50.039 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:58:50.067 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:50.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:50.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:58:50.169 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:58:50.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:49 smithi187 ceph-mon[33032]: pgmap v494: 148 pgs: 1 active+recovering+degraded+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 76 MiB/s wr, 1.22k op/s; 35164/188037 objects degraded (18.701%); 326/188037 objects misplaced (0.173%); 4.9 MiB/s, 78 objects/s recovering 2024-09-21T13:58:50.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:49 smithi187 ceph-mon[33032]: 3.5 scrub starts 2024-09-21T13:58:50.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:49 smithi187 ceph-mon[33032]: 3.5 scrub ok 2024-09-21T13:58:50.200 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:50.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:50.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:58:50.255 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/osd.7/config 2024-09-21T13:58:50.303 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:50.340 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:50.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:50.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:50.442 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:50.481 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:50.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:50.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:50.583 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:50.623 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:50.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:50.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:50.670 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 46 16 71057 71041 96.5098 93.3125 0.0147324 0.0103562 2024-09-21T13:58:50.726 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:50.765 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:50.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:50.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:50.855 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:58:50 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:58:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:58:50.868 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:50.905 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:50.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:50.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:51.008 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:58:51.034 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:51.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:51.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:58:51.135 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:58:51.163 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:51.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:51.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:58:51.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:50 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:58:51.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:50 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 35164/188037 objects degraded (18.701%), 29 pgs degraded (PG_DEGRADED) 2024-09-21T13:58:51.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:50 smithi187 ceph-mon[33032]: osdmap e286: 8 total, 7 up, 8 in 2024-09-21T13:58:51.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:50 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:58:51.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:50 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 35164/188037 objects degraded (18.701%), 29 pgs degraded (PG_DEGRADED) 2024-09-21T13:58:51.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:50 smithi049 ceph-mon[30080]: osdmap e286: 8 total, 7 up, 8 in 2024-09-21T13:58:51.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:50 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:58:51.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:50 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 35164/188037 objects degraded (18.701%), 29 pgs degraded (PG_DEGRADED) 2024-09-21T13:58:51.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:50 smithi049 ceph-mon[40489]: osdmap e286: 8 total, 7 up, 8 in 2024-09-21T13:58:51.265 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:58:51.302 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:51.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:51.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:58:51.403 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:58:51.436 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:51.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:51.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:58:51.538 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:51.563 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:51.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:51.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:51.665 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:51.670 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 47 16 72289 72273 96.0946 77 0.00751516 0.0104014 2024-09-21T13:58:51.693 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:51.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:51.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:51.795 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:51.839 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:51.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:51.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:51.941 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:51.967 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:51.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:51.968 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:52.068 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:52.105 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:52.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:52.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:52.126 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:51 smithi187 ceph-mon[33032]: pgmap v495: 148 pgs: 1 active+recovering+degraded+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.4 KiB/s rd, 105 MiB/s wr, 1.68k op/s; 37744/201921 objects degraded (18.692%); 326/201921 objects misplaced (0.161%); 4.1 MiB/s, 66 objects/s recovering 2024-09-21T13:58:52.126 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:51 smithi187 ceph-mon[33032]: 4.b scrub starts 2024-09-21T13:58:52.126 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:51 smithi187 ceph-mon[33032]: 4.b scrub ok 2024-09-21T13:58:52.184 INFO:teuthology.orchestra.run.smithi187.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-21T13:58:52.207 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:52.246 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:52.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:52.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:52.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:51 smithi049 ceph-mon[40489]: pgmap v495: 148 pgs: 1 active+recovering+degraded+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.4 KiB/s rd, 105 MiB/s wr, 1.68k op/s; 37744/201921 objects degraded (18.692%); 326/201921 objects misplaced (0.161%); 4.1 MiB/s, 66 objects/s recovering 2024-09-21T13:58:52.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:51 smithi049 ceph-mon[40489]: 4.b scrub starts 2024-09-21T13:58:52.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:51 smithi049 ceph-mon[40489]: 4.b scrub ok 2024-09-21T13:58:52.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:51 smithi049 ceph-mon[30080]: pgmap v495: 148 pgs: 1 active+recovering+degraded+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.4 KiB/s rd, 105 MiB/s wr, 1.68k op/s; 37744/201921 objects degraded (18.692%); 326/201921 objects misplaced (0.161%); 4.1 MiB/s, 66 objects/s recovering 2024-09-21T13:58:52.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:51 smithi049 ceph-mon[30080]: 4.b scrub starts 2024-09-21T13:58:52.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:51 smithi049 ceph-mon[30080]: 4.b scrub ok 2024-09-21T13:58:52.348 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:58:52.375 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:52.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:52.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:58:52.477 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:52.506 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:52.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:52.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:52.607 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:58:52.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:52.640 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:52.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:58:52.670 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 48 16 73600 73584 95.7995 81.9375 0.00620695 0.0104333 2024-09-21T13:58:52.742 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:52.769 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:52.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:52.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:52.871 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:52.909 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:52.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:52.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:53.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:52 smithi049 ceph-mon[40489]: pgmap v497: 148 pgs: 1 active+clean+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 4.2 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 743 B/s rd, 82 MiB/s wr, 1.32k op/s; 34677/208311 objects degraded (16.647%); 4356/208311 objects misplaced (2.091%); 7.2 MiB/s, 115 objects/s recovering 2024-09-21T13:58:53.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:52 smithi049 ceph-mon[40489]: 5.c scrub starts 2024-09-21T13:58:53.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:52 smithi049 ceph-mon[40489]: 5.c scrub ok 2024-09-21T13:58:53.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:52 smithi049 ceph-mon[30080]: pgmap v497: 148 pgs: 1 active+clean+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 4.2 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 743 B/s rd, 82 MiB/s wr, 1.32k op/s; 34677/208311 objects degraded (16.647%); 4356/208311 objects misplaced (2.091%); 7.2 MiB/s, 115 objects/s recovering 2024-09-21T13:58:53.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:52 smithi049 ceph-mon[30080]: 5.c scrub starts 2024-09-21T13:58:53.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:52 smithi049 ceph-mon[30080]: 5.c scrub ok 2024-09-21T13:58:53.011 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:58:53.033 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:53.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:53.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:58:53.135 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:53.173 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:53.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:53.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:53.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:52 smithi187 ceph-mon[33032]: pgmap v497: 148 pgs: 1 active+clean+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 4.2 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 743 B/s rd, 82 MiB/s wr, 1.32k op/s; 34677/208311 objects degraded (16.647%); 4356/208311 objects misplaced (2.091%); 7.2 MiB/s, 115 objects/s recovering 2024-09-21T13:58:53.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:52 smithi187 ceph-mon[33032]: 5.c scrub starts 2024-09-21T13:58:53.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:52 smithi187 ceph-mon[33032]: 5.c scrub ok 2024-09-21T13:58:53.274 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:53.311 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:53.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:53.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:53.413 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:53.453 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:53.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:53.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:53.463 INFO:teuthology.orchestra.run.smithi187.stdout:get_pg_num_history pg_num_history pg_num_history(e272 pg_nums {1={20=1},2={59=3},3={60=32},4={62=32},5={64=32},6={66=32},8={183=16},9={260=16}} deleted_pools 256,8) 2024-09-21T13:58:53.463 INFO:teuthology.orchestra.run.smithi187.stdout:Importing pgid 3.14 2024-09-21T13:58:53.463 INFO:teuthology.orchestra.run.smithi187.stdout:write_pg epoch 270 info 3.14( empty local-lis/les=269/270 n=0 ec=60/60 lis/c=269/269 les/c/f=270/270/0 sis=269) 2024-09-21T13:58:53.463 INFO:teuthology.orchestra.run.smithi187.stdout:Import successful 2024-09-21T13:58:53.555 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:53.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:53.599 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:53.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:53.671 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 49 16 74911 74895 95.5162 81.9375 0.0100186 0.0104639 2024-09-21T13:58:53.700 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:53.725 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:53.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:53.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:53.810 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T13:58:53.826 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:53.910 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:53.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:53.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:54.011 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:58:54.037 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:54.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:54.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:58:54.038 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:58:54.139 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:58:54.166 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:54.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:54.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:58:54.182 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:53 smithi049 ceph-mon[30080]: 6.d scrub starts 2024-09-21T13:58:54.182 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:53 smithi049 ceph-mon[30080]: 6.d scrub ok 2024-09-21T13:58:54.183 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:53 smithi049 ceph-mon[40489]: 6.d scrub starts 2024-09-21T13:58:54.183 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:53 smithi049 ceph-mon[40489]: 6.d scrub ok 2024-09-21T13:58:54.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:53 smithi187 ceph-mon[33032]: 6.d scrub starts 2024-09-21T13:58:54.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:53 smithi187 ceph-mon[33032]: 6.d scrub ok 2024-09-21T13:58:54.269 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:54.312 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:54.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:54.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:54.414 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:54.459 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:54.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:54.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:54.561 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:58:54.588 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:54.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:54.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:58:54.671 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 50 16 76212 76196 95.232 81.3125 0.0279466 0.0104954 2024-09-21T13:58:54.690 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm -f /var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/exp.3.14.7 2024-09-21T13:58:54.690 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:54.729 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:54.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:54.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:54.791 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:58:54.792 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:58:54.831 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:58:54.832 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:58:54.833 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:58:54.843 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:58:54.856 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:54.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:54.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:58:54.867 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:58:54.867 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:58:54.901 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:58:54.901 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:58:54.938 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:58:54.938 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:58:54.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:54 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:58:54.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:54 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:58:54.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:54 smithi187 ceph-mon[33032]: 5.18 scrub starts 2024-09-21T13:58:54.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:54 smithi187 ceph-mon[33032]: 5.18 scrub ok 2024-09-21T13:58:54.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:54 smithi187 ceph-mon[33032]: osdmap e287: 8 total, 7 up, 8 in 2024-09-21T13:58:54.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:54 smithi187 ceph-mon[33032]: pgmap v499: 148 pgs: 1 active+clean+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 841 B/s rd, 105 MiB/s wr, 1.67k op/s; 36754/218553 objects degraded (16.817%); 4356/218553 objects misplaced (1.993%); 3.3 MiB/s, 53 objects/s recovering 2024-09-21T13:58:54.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:54 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2833724290' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:58:54.958 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:54.975 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:58:54.975 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:58:55.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:55.006 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:55.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:55.011 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:58:55.011 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:58:55.015 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T13:58:55.107 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:55.134 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:55.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:55.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:55.178 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:54 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:58:55.178 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:54 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:58:55.178 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:54 smithi049 ceph-mon[40489]: 5.18 scrub starts 2024-09-21T13:58:55.178 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:54 smithi049 ceph-mon[40489]: 5.18 scrub ok 2024-09-21T13:58:55.178 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:54 smithi049 ceph-mon[40489]: osdmap e287: 8 total, 7 up, 8 in 2024-09-21T13:58:55.178 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:54 smithi049 ceph-mon[40489]: pgmap v499: 148 pgs: 1 active+clean+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 841 B/s rd, 105 MiB/s wr, 1.67k op/s; 36754/218553 objects degraded (16.817%); 4356/218553 objects misplaced (1.993%); 3.3 MiB/s, 53 objects/s recovering 2024-09-21T13:58:55.179 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:54 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2833724290' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:58:55.179 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:54 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:58:55.179 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:54 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:58:55.179 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:54 smithi049 ceph-mon[30080]: 5.18 scrub starts 2024-09-21T13:58:55.179 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:54 smithi049 ceph-mon[30080]: 5.18 scrub ok 2024-09-21T13:58:55.179 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:54 smithi049 ceph-mon[30080]: osdmap e287: 8 total, 7 up, 8 in 2024-09-21T13:58:55.179 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:54 smithi049 ceph-mon[30080]: pgmap v499: 148 pgs: 1 active+clean+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 841 B/s rd, 105 MiB/s wr, 1.67k op/s; 36754/218553 objects degraded (16.817%); 4356/218553 objects misplaced (1.993%); 3.3 MiB/s, 53 objects/s recovering 2024-09-21T13:58:55.179 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:54 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2833724290' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:58:55.236 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:58:55.262 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:55.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:55.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:58:55.364 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:55.389 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:55.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:55.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:55.491 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:55.530 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:55.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:55.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:55.632 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:58:55.671 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:55.671 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 51 16 77546 77530 94.9994 83.375 0.0108096 0.0105212 2024-09-21T13:58:55.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:55.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:58:55.773 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:55.812 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:55.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:55.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:55.914 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:55.944 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:55.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:55.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:55.990 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T13:58:56.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:55 smithi049 ceph-mon[40489]: 6.17 scrub starts 2024-09-21T13:58:56.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:55 smithi049 ceph-mon[40489]: 6.17 scrub ok 2024-09-21T13:58:56.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:55 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 36754/218553 objects degraded (16.817%), 28 pgs degraded (PG_DEGRADED) 2024-09-21T13:58:56.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:55 smithi049 ceph-mon[40489]: osdmap e288: 8 total, 7 up, 8 in 2024-09-21T13:58:56.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:55 smithi049 ceph-mon[30080]: 6.17 scrub starts 2024-09-21T13:58:56.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:55 smithi049 ceph-mon[30080]: 6.17 scrub ok 2024-09-21T13:58:56.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:55 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 36754/218553 objects degraded (16.817%), 28 pgs degraded (PG_DEGRADED) 2024-09-21T13:58:56.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:55 smithi049 ceph-mon[30080]: osdmap e288: 8 total, 7 up, 8 in 2024-09-21T13:58:56.046 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:56.099 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:56.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:56.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:56.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:55 smithi187 ceph-mon[33032]: 6.17 scrub starts 2024-09-21T13:58:56.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:55 smithi187 ceph-mon[33032]: 6.17 scrub ok 2024-09-21T13:58:56.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:55 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 36754/218553 objects degraded (16.817%), 28 pgs degraded (PG_DEGRADED) 2024-09-21T13:58:56.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:55 smithi187 ceph-mon[33032]: osdmap e288: 8 total, 7 up, 8 in 2024-09-21T13:58:56.201 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:56.234 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T13:58:56.235 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T13:58:56.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:56.256 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:56.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:56.357 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:56.404 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:56.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:56.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:56.506 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:56.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:56.556 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:56.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:56.573 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T13:58:56.573 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:58:56.657 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:56.671 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 52 16 78857 78841 94.7476 81.9375 0.0150144 0.0105497 2024-09-21T13:58:56.699 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:56.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:56.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:56.801 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:56.842 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:56.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:56.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:56.944 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:56.986 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:56.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:56.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:57.088 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:57.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:57.124 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:57.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:57.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:56 smithi187 ceph-mon[33032]: 5.b scrub starts 2024-09-21T13:58:57.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:56 smithi187 ceph-mon[33032]: 5.b scrub ok 2024-09-21T13:58:57.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:56 smithi187 ceph-mon[33032]: pgmap v501: 148 pgs: 1 active+clean+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 92 MiB/s wr, 1.47k op/s; 37955/228309 objects degraded (16.624%); 4775/228309 objects misplaced (2.091%); 3.4 MiB/s, 54 objects/s recovering 2024-09-21T13:58:57.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:56 smithi187 ceph-mon[33032]: 4.4 scrub starts 2024-09-21T13:58:57.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:56 smithi187 ceph-mon[33032]: 4.4 scrub ok 2024-09-21T13:58:57.225 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:58:57.251 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:57.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:57.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:58:57.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:56 smithi049 ceph-mon[40489]: 5.b scrub starts 2024-09-21T13:58:57.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:56 smithi049 ceph-mon[40489]: 5.b scrub ok 2024-09-21T13:58:57.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:56 smithi049 ceph-mon[40489]: pgmap v501: 148 pgs: 1 active+clean+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 92 MiB/s wr, 1.47k op/s; 37955/228309 objects degraded (16.624%); 4775/228309 objects misplaced (2.091%); 3.4 MiB/s, 54 objects/s recovering 2024-09-21T13:58:57.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:56 smithi049 ceph-mon[40489]: 4.4 scrub starts 2024-09-21T13:58:57.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:56 smithi049 ceph-mon[40489]: 4.4 scrub ok 2024-09-21T13:58:57.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:56 smithi049 ceph-mon[30080]: 5.b scrub starts 2024-09-21T13:58:57.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:56 smithi049 ceph-mon[30080]: 5.b scrub ok 2024-09-21T13:58:57.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:56 smithi049 ceph-mon[30080]: pgmap v501: 148 pgs: 1 active+clean+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 92 MiB/s wr, 1.47k op/s; 37955/228309 objects degraded (16.624%); 4775/228309 objects misplaced (2.091%); 3.4 MiB/s, 54 objects/s recovering 2024-09-21T13:58:57.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:56 smithi049 ceph-mon[30080]: 4.4 scrub starts 2024-09-21T13:58:57.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:56 smithi049 ceph-mon[30080]: 4.4 scrub ok 2024-09-21T13:58:57.353 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:57.392 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:57.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:57.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:57.494 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:58:57.521 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:57.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:57.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:58:57.556 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T13:58:57.563 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:58:57.622 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:57.646 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:57.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:57.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:57.672 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 53 16 80235 80219 94.5826 86.125 0.0132521 0.0105669 2024-09-21T13:58:57.748 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:58:57.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:57.801 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:57.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:58:57.902 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:57.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:57.928 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:57.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:58.029 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:58:58.075 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:58.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:58.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:58:58.090 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:57 smithi049 ceph-mon[30080]: 3.2 scrub starts 2024-09-21T13:58:58.090 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:57 smithi049 ceph-mon[30080]: 3.2 scrub ok 2024-09-21T13:58:58.090 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:57 smithi049 ceph-mon[40489]: 3.2 scrub starts 2024-09-21T13:58:58.090 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:57 smithi049 ceph-mon[40489]: 3.2 scrub ok 2024-09-21T13:58:58.177 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:58.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:57 smithi187 ceph-mon[33032]: 3.2 scrub starts 2024-09-21T13:58:58.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:57 smithi187 ceph-mon[33032]: 3.2 scrub ok 2024-09-21T13:58:58.203 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:58.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:58.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:58.236 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T13:58:58.305 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:58:58.330 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:58.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:58.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:58:58.416 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:58:58.432 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:58:58.507 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:58.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:58.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:58:58.609 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:58:58.673 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 54 16 81391 81375 94.1682 72.25 0.0177885 0.010614 2024-09-21T13:58:58.675 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:58.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:58.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:58:58.776 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:58.803 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:58.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:58.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:58.905 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:58:58.945 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-09-21T13:58:58.955 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:58.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:58.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:58:59.057 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:59.084 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:59.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:59.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:59.103 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T13:58:59.104 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:58:59.185 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:59.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:58 smithi187 ceph-mon[33032]: 4.18 scrub starts 2024-09-21T13:58:59.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:58 smithi187 ceph-mon[33032]: 4.18 scrub ok 2024-09-21T13:58:59.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:58 smithi187 ceph-mon[33032]: pgmap v502: 148 pgs: 1 active+clean+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 649 B/s rd, 70 MiB/s wr, 1.12k op/s; 37955/228309 objects degraded (16.624%); 4775/228309 objects misplaced (2.091%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T13:58:59.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:58 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3969367411' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:58:59.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:58 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:58:59.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:58 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:58:59.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:58 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:58:59.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:58 smithi187 ceph-mon[33032]: 6.0 scrub starts 2024-09-21T13:58:59.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:58 smithi187 ceph-mon[33032]: 6.0 scrub ok 2024-09-21T13:58:59.214 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:59.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:59.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:59.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:58 smithi049 ceph-mon[40489]: 4.18 scrub starts 2024-09-21T13:58:59.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:58 smithi049 ceph-mon[40489]: 4.18 scrub ok 2024-09-21T13:58:59.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:58 smithi049 ceph-mon[40489]: pgmap v502: 148 pgs: 1 active+clean+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 649 B/s rd, 70 MiB/s wr, 1.12k op/s; 37955/228309 objects degraded (16.624%); 4775/228309 objects misplaced (2.091%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T13:58:59.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:58 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3969367411' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:58:59.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:58 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:58:59.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:58 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:58:59.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:58 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:58:59.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:58 smithi049 ceph-mon[40489]: 6.0 scrub starts 2024-09-21T13:58:59.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:58 smithi049 ceph-mon[40489]: 6.0 scrub ok 2024-09-21T13:58:59.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:58 smithi049 ceph-mon[30080]: 4.18 scrub starts 2024-09-21T13:58:59.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:58 smithi049 ceph-mon[30080]: 4.18 scrub ok 2024-09-21T13:58:59.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:58 smithi049 ceph-mon[30080]: pgmap v502: 148 pgs: 1 active+clean+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 649 B/s rd, 70 MiB/s wr, 1.12k op/s; 37955/228309 objects degraded (16.624%); 4775/228309 objects misplaced (2.091%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T13:58:59.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:58 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3969367411' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:58:59.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:58 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:58:59.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:58 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:58:59.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:58 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:58:59.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:58 smithi049 ceph-mon[30080]: 6.0 scrub starts 2024-09-21T13:58:59.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:58 smithi049 ceph-mon[30080]: 6.0 scrub ok 2024-09-21T13:58:59.316 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:58:59.354 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:59.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:59.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:58:59.456 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:58:59.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:59.498 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:58:59.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:58:59.599 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:59.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:59.630 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:59.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:59.673 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 55 16 82731 82715 93.9787 83.75 0.00642773 0.0106357 2024-09-21T13:58:59.694 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:58:59 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:58:59.723 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [7] live_osds: [0, 1, 4, 5, 6, 2, 3] 2024-09-21T13:58:59.724 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T13:58:59.724 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-21T13:58:59.724 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd dump -f json-pretty 2024-09-21T13:58:59.730 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:58:59.756 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:59.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:59.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:58:59.858 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:58:59.885 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:58:59.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:58:59.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:58:59.986 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:00.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:00.040 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:00.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:00.115 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T13:59:00.141 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:00.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:59 smithi187 ceph-mon[33032]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:59:00.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:59 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T13:59:00.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:59 smithi187 ceph-mon[33032]: osdmap e289: 8 total, 7 up, 8 in 2024-09-21T13:59:00.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:58:59 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:59:00.218 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:00.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:00.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:00.248 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:59 smithi049 ceph-mon[30080]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:59:00.248 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:59 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T13:59:00.248 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:59 smithi049 ceph-mon[30080]: osdmap e289: 8 total, 7 up, 8 in 2024-09-21T13:59:00.248 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:58:59 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:59:00.248 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:59 smithi049 ceph-mon[40489]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:59:00.248 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:59 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T13:59:00.249 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:59 smithi049 ceph-mon[40489]: osdmap e289: 8 total, 7 up, 8 in 2024-09-21T13:59:00.249 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:58:59 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:59:00.320 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:00.347 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:00.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:00.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:00.449 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:59:00.493 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:00.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:00.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:59:00.559 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:59:00.595 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:00.657 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:59:00.661 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:00.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:00.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:00.674 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 56 16 83920 83904 93.6259 74.3125 0.028514 0.0106745 2024-09-21T13:59:00.763 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:00.807 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:00.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:00.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:00.909 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:59:00.915 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:59:00 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:59:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:59:00.934 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:00.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:00.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:59:01.036 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:01.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:01.081 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:01.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:01.182 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:01.226 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:01.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:01.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:01.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:00 smithi049 ceph-mon[40489]: pgmap v504: 148 pgs: 1 active+clean+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.7 KiB/s rd, 80 MiB/s wr, 1.28k op/s; 40392/241665 objects degraded (16.714%); 4775/241665 objects misplaced (1.976%) 2024-09-21T13:59:01.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:00 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 40392/241665 objects degraded (16.714%), 28 pgs degraded (PG_DEGRADED) 2024-09-21T13:59:01.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:00 smithi049 ceph-mon[40489]: osdmap e290: 8 total, 7 up, 8 in 2024-09-21T13:59:01.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:00 smithi049 ceph-mon[30080]: pgmap v504: 148 pgs: 1 active+clean+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.7 KiB/s rd, 80 MiB/s wr, 1.28k op/s; 40392/241665 objects degraded (16.714%); 4775/241665 objects misplaced (1.976%) 2024-09-21T13:59:01.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:00 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 40392/241665 objects degraded (16.714%), 28 pgs degraded (PG_DEGRADED) 2024-09-21T13:59:01.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:00 smithi049 ceph-mon[30080]: osdmap e290: 8 total, 7 up, 8 in 2024-09-21T13:59:01.329 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:01.365 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:01.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:01.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:01.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:00 smithi187 ceph-mon[33032]: pgmap v504: 148 pgs: 1 active+clean+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.7 KiB/s rd, 80 MiB/s wr, 1.28k op/s; 40392/241665 objects degraded (16.714%); 4775/241665 objects misplaced (1.976%) 2024-09-21T13:59:01.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:00 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 40392/241665 objects degraded (16.714%), 28 pgs degraded (PG_DEGRADED) 2024-09-21T13:59:01.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:00 smithi187 ceph-mon[33032]: osdmap e290: 8 total, 7 up, 8 in 2024-09-21T13:59:01.467 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:59:01.492 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:01.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:01.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:59:01.594 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:01.645 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:01.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:01.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:01.674 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 57 16 85208 85192 93.3956 80.5 0.01726 0.0107012 2024-09-21T13:59:01.747 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:01.766 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:59:01.766 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:59:01.766 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:59:01.766 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:59:01.766 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:59:01.766 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:59:01.778 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:59:01.790 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:59:01.791 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:59:01.792 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:01.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:01.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:01.828 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:59:01.828 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:59:01.871 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:59:01.871 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:59:01.896 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:59:01.917 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:59:01.918 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:59:01.923 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:01.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:01.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:59:01.925 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T13:59:02.024 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:02.060 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:02.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:02.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:02.068 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:01 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3205731878' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:59:02.068 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:01 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3205731878' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:59:02.069 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:59:02.069 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T13:59:02.069 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 290, 2024-09-21T13:59:02.069 INFO:teuthology.orchestra.run.smithi049.stdout: "fsid": "40a3d2e8-781f-11ef-baf6-efdc52797490", 2024-09-21T13:59:02.069 INFO:teuthology.orchestra.run.smithi049.stdout: "created": "2024-09-21T13:42:57.838275+0000", 2024-09-21T13:59:02.069 INFO:teuthology.orchestra.run.smithi049.stdout: "modified": "2024-09-21T13:59:00.016334+0000", 2024-09-21T13:59:02.069 INFO:teuthology.orchestra.run.smithi049.stdout: "last_up_change": "2024-09-21T13:58:23.782318+0000", 2024-09-21T13:59:02.069 INFO:teuthology.orchestra.run.smithi049.stdout: "last_in_change": "2024-09-21T13:58:16.730861+0000", 2024-09-21T13:59:02.069 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-21T13:59:02.069 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_num": 5801984, 2024-09-21T13:59:02.069 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_set": [ 2024-09-21T13:59:02.069 INFO:teuthology.orchestra.run.smithi049.stdout: "noscrub", 2024-09-21T13:59:02.070 INFO:teuthology.orchestra.run.smithi049.stdout: "pglog_hardlimit", 2024-09-21T13:59:02.070 INFO:teuthology.orchestra.run.smithi049.stdout: "purged_snapdirs", 2024-09-21T13:59:02.070 INFO:teuthology.orchestra.run.smithi049.stdout: "recovery_deletes", 2024-09-21T13:59:02.070 INFO:teuthology.orchestra.run.smithi049.stdout: "sortbitwise" 2024-09-21T13:59:02.070 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T13:59:02.070 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_version": 20, 2024-09-21T13:59:02.070 INFO:teuthology.orchestra.run.smithi049.stdout: "full_ratio": 0.94999998807907104, 2024-09-21T13:59:02.070 INFO:teuthology.orchestra.run.smithi049.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-21T13:59:02.070 INFO:teuthology.orchestra.run.smithi049.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-21T13:59:02.070 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_snapshot": "", 2024-09-21T13:59:02.070 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_max": 9, 2024-09-21T13:59:02.070 INFO:teuthology.orchestra.run.smithi049.stdout: "max_osd": 8, 2024-09-21T13:59:02.070 INFO:teuthology.orchestra.run.smithi049.stdout: "require_min_compat_client": "luminous", 2024-09-21T13:59:02.071 INFO:teuthology.orchestra.run.smithi049.stdout: "min_compat_client": "luminous", 2024-09-21T13:59:02.071 INFO:teuthology.orchestra.run.smithi049.stdout: "require_osd_release": "squid", 2024-09-21T13:59:02.071 INFO:teuthology.orchestra.run.smithi049.stdout: "allow_crimson": false, 2024-09-21T13:59:02.071 INFO:teuthology.orchestra.run.smithi049.stdout: "pools": [ 2024-09-21T13:59:02.071 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.071 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 1, 2024-09-21T13:59:02.071 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": ".mgr", 2024-09-21T13:59:02.071 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:46:21.096244+0000", 2024-09-21T13:59:02.071 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T13:59:02.071 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T13:59:02.071 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T13:59:02.071 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T13:59:02.071 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T13:59:02.071 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T13:59:02.072 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T13:59:02.072 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T13:59:02.072 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T13:59:02.072 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T13:59:02.072 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T13:59:02.072 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T13:59:02.072 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T13:59:02.072 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 1, 2024-09-21T13:59:02.072 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 1, 2024-09-21T13:59:02.072 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 1, 2024-09-21T13:59:02.072 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 1, 2024-09-21T13:59:02.072 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 1, 2024-09-21T13:59:02.072 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T13:59:02.073 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T13:59:02.073 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T13:59:02.073 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T13:59:02.073 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T13:59:02.073 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T13:59:02.073 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T13:59:02.073 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.073 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "199", 2024-09-21T13:59:02.073 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T13:59:02.073 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T13:59:02.073 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T13:59:02.073 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T13:59:02.073 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T13:59:02.073 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T13:59:02.074 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T13:59:02.074 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T13:59:02.074 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T13:59:02.074 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T13:59:02.074 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T13:59:02.074 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T13:59:02.074 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T13:59:02.074 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T13:59:02.074 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T13:59:02.074 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T13:59:02.074 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T13:59:02.074 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T13:59:02.075 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T13:59:02.075 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T13:59:02.075 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T13:59:02.075 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T13:59:02.075 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T13:59:02.075 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T13:59:02.075 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T13:59:02.075 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T13:59:02.075 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.075 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T13:59:02.075 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T13:59:02.075 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T13:59:02.075 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T13:59:02.075 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T13:59:02.076 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T13:59:02.076 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T13:59:02.076 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T13:59:02.076 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T13:59:02.076 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T13:59:02.076 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T13:59:02.076 INFO:teuthology.orchestra.run.smithi049.stdout: "options": { 2024-09-21T13:59:02.076 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_max": 32, 2024-09-21T13:59:02.076 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_min": 1 2024-09-21T13:59:02.076 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.076 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T13:59:02.076 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr": {} 2024-09-21T13:59:02.076 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.077 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T13:59:02.077 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T13:59:02.077 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 8, 2024-09-21T13:59:02.077 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 8, 2024-09-21T13:59:02.077 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 0.25, 2024-09-21T13:59:02.077 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2, 2024-09-21T13:59:02.077 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2, 2024-09-21T13:59:02.077 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.86855697631835938, 2024-09-21T13:59:02.077 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.87000000476837158, 2024-09-21T13:59:02.077 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.87000000476837158 2024-09-21T13:59:02.077 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:02.077 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.077 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.077 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 2, 2024-09-21T13:59:02.078 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "datapool", 2024-09-21T13:59:02.078 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:48:55.540712+0000", 2024-09-21T13:59:02.078 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 8193, 2024-09-21T13:59:02.078 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T13:59:02.078 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T13:59:02.078 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T13:59:02.078 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T13:59:02.078 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T13:59:02.078 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T13:59:02.078 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T13:59:02.078 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T13:59:02.078 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T13:59:02.078 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T13:59:02.078 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T13:59:02.079 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T13:59:02.079 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 3, 2024-09-21T13:59:02.079 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 3, 2024-09-21T13:59:02.079 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 3, 2024-09-21T13:59:02.079 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 3, 2024-09-21T13:59:02.079 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 3, 2024-09-21T13:59:02.079 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T13:59:02.079 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T13:59:02.079 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T13:59:02.079 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T13:59:02.079 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T13:59:02.079 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T13:59:02.079 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T13:59:02.079 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.079 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "63", 2024-09-21T13:59:02.080 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T13:59:02.080 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T13:59:02.080 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T13:59:02.080 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T13:59:02.080 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T13:59:02.080 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 2, 2024-09-21T13:59:02.080 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 63, 2024-09-21T13:59:02.080 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T13:59:02.080 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T13:59:02.080 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T13:59:02.080 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T13:59:02.080 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T13:59:02.080 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T13:59:02.080 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T13:59:02.081 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T13:59:02.081 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T13:59:02.081 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T13:59:02.081 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T13:59:02.081 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T13:59:02.081 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T13:59:02.081 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T13:59:02.081 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T13:59:02.081 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T13:59:02.081 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T13:59:02.081 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T13:59:02.081 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T13:59:02.081 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.081 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T13:59:02.082 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T13:59:02.082 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T13:59:02.082 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T13:59:02.082 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T13:59:02.082 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T13:59:02.082 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T13:59:02.082 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T13:59:02.082 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T13:59:02.082 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T13:59:02.082 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T13:59:02.082 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T13:59:02.082 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T13:59:02.082 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd": {} 2024-09-21T13:59:02.082 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.083 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T13:59:02.083 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T13:59:02.083 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2.6700000762939453, 2024-09-21T13:59:02.083 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2.6700000762939453, 2024-09-21T13:59:02.083 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 0.75, 2024-09-21T13:59:02.083 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2, 2024-09-21T13:59:02.083 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2, 2024-09-21T13:59:02.083 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.86855697631835938, 2024-09-21T13:59:02.083 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.87000000476837158, 2024-09-21T13:59:02.083 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.87000000476837158 2024-09-21T13:59:02.083 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:02.083 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.083 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.083 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 3, 2024-09-21T13:59:02.084 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": ".rgw.root", 2024-09-21T13:59:02.084 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:48:56.417456+0000", 2024-09-21T13:59:02.084 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T13:59:02.084 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T13:59:02.084 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T13:59:02.084 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T13:59:02.084 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T13:59:02.084 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T13:59:02.084 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T13:59:02.084 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T13:59:02.084 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T13:59:02.084 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T13:59:02.084 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T13:59:02.084 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T13:59:02.084 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T13:59:02.085 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 32, 2024-09-21T13:59:02.085 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 32, 2024-09-21T13:59:02.085 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 32, 2024-09-21T13:59:02.085 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 32, 2024-09-21T13:59:02.085 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 32, 2024-09-21T13:59:02.085 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T13:59:02.085 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T13:59:02.085 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T13:59:02.085 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T13:59:02.086 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T13:59:02.086 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T13:59:02.086 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T13:59:02.086 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.086 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "62", 2024-09-21T13:59:02.086 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T13:59:02.086 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T13:59:02.086 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T13:59:02.086 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T13:59:02.086 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T13:59:02.086 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T13:59:02.086 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T13:59:02.087 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T13:59:02.087 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T13:59:02.087 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T13:59:02.087 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T13:59:02.087 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T13:59:02.087 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T13:59:02.087 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T13:59:02.087 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T13:59:02.087 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T13:59:02.087 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T13:59:02.087 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T13:59:02.087 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T13:59:02.087 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T13:59:02.087 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T13:59:02.088 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T13:59:02.088 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T13:59:02.088 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T13:59:02.088 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T13:59:02.088 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T13:59:02.088 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.088 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T13:59:02.088 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T13:59:02.088 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T13:59:02.088 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T13:59:02.088 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T13:59:02.088 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T13:59:02.088 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T13:59:02.088 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T13:59:02.089 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T13:59:02.089 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T13:59:02.089 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T13:59:02.089 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T13:59:02.089 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T13:59:02.089 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw": {} 2024-09-21T13:59:02.089 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.089 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T13:59:02.089 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T13:59:02.089 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2.2400000095367432, 2024-09-21T13:59:02.089 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2.2400000095367432, 2024-09-21T13:59:02.089 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 0.87999999523162842, 2024-09-21T13:59:02.089 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 1.9700000286102295, 2024-09-21T13:59:02.089 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 1.9700000286102295, 2024-09-21T13:59:02.089 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.86855697631835938, 2024-09-21T13:59:02.090 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.87000000476837158, 2024-09-21T13:59:02.090 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.87000000476837158 2024-09-21T13:59:02.090 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:02.090 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.090 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.090 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 4, 2024-09-21T13:59:02.090 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "default.rgw.log", 2024-09-21T13:59:02.090 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:48:58.128455+0000", 2024-09-21T13:59:02.090 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T13:59:02.090 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T13:59:02.090 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T13:59:02.090 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T13:59:02.090 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T13:59:02.090 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T13:59:02.091 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T13:59:02.091 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T13:59:02.091 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T13:59:02.091 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T13:59:02.091 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T13:59:02.091 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T13:59:02.091 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T13:59:02.091 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 32, 2024-09-21T13:59:02.091 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 32, 2024-09-21T13:59:02.091 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 32, 2024-09-21T13:59:02.091 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 32, 2024-09-21T13:59:02.091 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 32, 2024-09-21T13:59:02.091 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T13:59:02.091 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T13:59:02.092 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T13:59:02.092 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T13:59:02.092 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T13:59:02.092 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T13:59:02.092 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T13:59:02.092 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.092 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "64", 2024-09-21T13:59:02.092 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T13:59:02.092 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T13:59:02.092 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T13:59:02.092 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T13:59:02.092 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T13:59:02.092 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T13:59:02.092 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T13:59:02.093 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T13:59:02.093 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T13:59:02.093 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T13:59:02.093 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T13:59:02.093 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T13:59:02.093 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T13:59:02.093 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T13:59:02.093 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T13:59:02.093 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T13:59:02.093 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T13:59:02.093 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T13:59:02.093 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T13:59:02.093 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T13:59:02.093 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T13:59:02.093 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T13:59:02.094 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T13:59:02.094 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T13:59:02.094 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T13:59:02.094 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T13:59:02.094 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.094 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T13:59:02.094 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T13:59:02.094 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T13:59:02.094 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T13:59:02.094 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T13:59:02.094 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T13:59:02.094 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T13:59:02.094 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T13:59:02.095 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T13:59:02.095 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T13:59:02.095 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T13:59:02.095 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T13:59:02.095 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T13:59:02.095 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw": {} 2024-09-21T13:59:02.095 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.095 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T13:59:02.095 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T13:59:02.095 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2.2400000095367432, 2024-09-21T13:59:02.095 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2.2400000095367432, 2024-09-21T13:59:02.095 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 0.87999999523162842, 2024-09-21T13:59:02.095 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 1.9700000286102295, 2024-09-21T13:59:02.095 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 1.9700000286102295, 2024-09-21T13:59:02.095 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.86855697631835938, 2024-09-21T13:59:02.096 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.87000000476837158, 2024-09-21T13:59:02.096 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.87000000476837158 2024-09-21T13:59:02.096 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:02.096 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.096 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.096 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 5, 2024-09-21T13:59:02.096 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "default.rgw.control", 2024-09-21T13:59:02.096 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:49:00.069361+0000", 2024-09-21T13:59:02.096 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T13:59:02.096 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T13:59:02.096 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T13:59:02.096 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T13:59:02.096 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T13:59:02.096 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T13:59:02.097 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T13:59:02.097 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T13:59:02.097 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T13:59:02.097 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T13:59:02.097 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T13:59:02.097 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T13:59:02.097 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T13:59:02.097 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 32, 2024-09-21T13:59:02.097 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 32, 2024-09-21T13:59:02.097 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 32, 2024-09-21T13:59:02.097 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 32, 2024-09-21T13:59:02.097 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 32, 2024-09-21T13:59:02.097 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T13:59:02.097 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T13:59:02.097 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T13:59:02.098 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T13:59:02.098 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T13:59:02.098 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T13:59:02.098 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T13:59:02.098 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.098 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "66", 2024-09-21T13:59:02.098 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T13:59:02.098 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T13:59:02.098 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T13:59:02.098 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T13:59:02.099 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T13:59:02.099 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T13:59:02.099 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T13:59:02.099 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T13:59:02.099 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T13:59:02.099 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T13:59:02.099 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T13:59:02.099 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T13:59:02.099 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T13:59:02.099 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T13:59:02.099 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T13:59:02.100 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T13:59:02.100 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T13:59:02.100 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T13:59:02.100 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T13:59:02.100 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T13:59:02.100 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T13:59:02.100 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T13:59:02.100 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T13:59:02.100 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T13:59:02.100 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T13:59:02.100 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T13:59:02.100 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.100 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T13:59:02.100 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T13:59:02.101 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T13:59:02.101 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T13:59:02.101 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T13:59:02.101 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T13:59:02.101 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T13:59:02.101 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T13:59:02.101 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T13:59:02.101 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T13:59:02.101 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T13:59:02.101 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T13:59:02.101 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T13:59:02.101 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw": {} 2024-09-21T13:59:02.101 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.101 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T13:59:02.102 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T13:59:02.102 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 1.7400000095367432, 2024-09-21T13:59:02.102 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 1.7400000095367432, 2024-09-21T13:59:02.102 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 0.87999999523162842, 2024-09-21T13:59:02.102 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 1.5299999713897705, 2024-09-21T13:59:02.102 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 1.5299999713897705, 2024-09-21T13:59:02.102 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.86855697631835938, 2024-09-21T13:59:02.102 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.87000000476837158, 2024-09-21T13:59:02.103 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.87000000476837158 2024-09-21T13:59:02.103 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:02.103 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.103 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.103 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 6, 2024-09-21T13:59:02.103 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "default.rgw.meta", 2024-09-21T13:59:02.103 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:49:02.127399+0000", 2024-09-21T13:59:02.103 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T13:59:02.103 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T13:59:02.103 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T13:59:02.103 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T13:59:02.104 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T13:59:02.104 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T13:59:02.104 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T13:59:02.104 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T13:59:02.104 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T13:59:02.104 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T13:59:02.104 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T13:59:02.104 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T13:59:02.104 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T13:59:02.104 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 32, 2024-09-21T13:59:02.104 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 32, 2024-09-21T13:59:02.104 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 32, 2024-09-21T13:59:02.104 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 32, 2024-09-21T13:59:02.104 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 32, 2024-09-21T13:59:02.104 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T13:59:02.105 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T13:59:02.105 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T13:59:02.105 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T13:59:02.105 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T13:59:02.105 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T13:59:02.105 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T13:59:02.105 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.105 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "68", 2024-09-21T13:59:02.105 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T13:59:02.105 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T13:59:02.105 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T13:59:02.105 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T13:59:02.105 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T13:59:02.105 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T13:59:02.106 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T13:59:02.106 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T13:59:02.106 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T13:59:02.106 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T13:59:02.106 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T13:59:02.106 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T13:59:02.106 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T13:59:02.106 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T13:59:02.106 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T13:59:02.106 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T13:59:02.106 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T13:59:02.106 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T13:59:02.106 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T13:59:02.107 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T13:59:02.107 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T13:59:02.107 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T13:59:02.107 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T13:59:02.107 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T13:59:02.107 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T13:59:02.107 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T13:59:02.107 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.107 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T13:59:02.107 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T13:59:02.107 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T13:59:02.107 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T13:59:02.107 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T13:59:02.107 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T13:59:02.108 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T13:59:02.108 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T13:59:02.108 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T13:59:02.108 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T13:59:02.108 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T13:59:02.108 INFO:teuthology.orchestra.run.smithi049.stdout: "options": { 2024-09-21T13:59:02.108 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_bias": 4 2024-09-21T13:59:02.108 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.108 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T13:59:02.108 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw": {} 2024-09-21T13:59:02.108 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.108 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T13:59:02.108 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T13:59:02.108 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2.2400000095367432, 2024-09-21T13:59:02.108 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2.2400000095367432, 2024-09-21T13:59:02.109 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 0.87999999523162842, 2024-09-21T13:59:02.109 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 1.9700000286102295, 2024-09-21T13:59:02.109 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 1.9700000286102295, 2024-09-21T13:59:02.109 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.86855697631835938, 2024-09-21T13:59:02.109 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.87000000476837158, 2024-09-21T13:59:02.109 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.87000000476837158 2024-09-21T13:59:02.109 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:02.109 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.109 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.109 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 9, 2024-09-21T13:59:02.109 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "unique_pool_2", 2024-09-21T13:59:02.109 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:57:51.528024+0000", 2024-09-21T13:59:02.109 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T13:59:02.109 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T13:59:02.110 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T13:59:02.110 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T13:59:02.110 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T13:59:02.110 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T13:59:02.110 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T13:59:02.110 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T13:59:02.110 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T13:59:02.110 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T13:59:02.110 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T13:59:02.110 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T13:59:02.110 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T13:59:02.110 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 16, 2024-09-21T13:59:02.110 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 16, 2024-09-21T13:59:02.111 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 16, 2024-09-21T13:59:02.111 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 16, 2024-09-21T13:59:02.111 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 16, 2024-09-21T13:59:02.111 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T13:59:02.111 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T13:59:02.111 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T13:59:02.111 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T13:59:02.111 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T13:59:02.111 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T13:59:02.111 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T13:59:02.111 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.111 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "263", 2024-09-21T13:59:02.111 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T13:59:02.111 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T13:59:02.111 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T13:59:02.112 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T13:59:02.112 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T13:59:02.112 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T13:59:02.112 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T13:59:02.112 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T13:59:02.112 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T13:59:02.112 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T13:59:02.112 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T13:59:02.112 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T13:59:02.112 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T13:59:02.112 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T13:59:02.112 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T13:59:02.112 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T13:59:02.113 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T13:59:02.113 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T13:59:02.113 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T13:59:02.113 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T13:59:02.113 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T13:59:02.113 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T13:59:02.113 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T13:59:02.113 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T13:59:02.113 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T13:59:02.113 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T13:59:02.113 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.114 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T13:59:02.114 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T13:59:02.114 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T13:59:02.114 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T13:59:02.114 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T13:59:02.114 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T13:59:02.114 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T13:59:02.114 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T13:59:02.114 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T13:59:02.114 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T13:59:02.114 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T13:59:02.114 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T13:59:02.114 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T13:59:02.115 INFO:teuthology.orchestra.run.smithi049.stdout: "rados": {} 2024-09-21T13:59:02.115 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.115 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T13:59:02.115 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T13:59:02.115 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2.9900000095367432, 2024-09-21T13:59:02.115 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2.9900000095367432, 2024-09-21T13:59:02.115 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 0.87999999523162842, 2024-09-21T13:59:02.115 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2.630000114440918, 2024-09-21T13:59:02.115 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2.630000114440918, 2024-09-21T13:59:02.115 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.86855697631835938, 2024-09-21T13:59:02.115 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.87000000476837158, 2024-09-21T13:59:02.115 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.87000000476837158 2024-09-21T13:59:02.115 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:02.116 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:02.116 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T13:59:02.116 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-09-21T13:59:02.116 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.116 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 0, 2024-09-21T13:59:02.116 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "2595dca9-6219-4e0c-8728-751eb102e73a", 2024-09-21T13:59:02.116 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T13:59:02.116 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T13:59:02.116 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T13:59:02.116 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 0.1215972900390625, 2024-09-21T13:59:02.116 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T13:59:02.116 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T13:59:02.116 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 8, 2024-09-21T13:59:02.116 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 277, 2024-09-21T13:59:02.117 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T13:59:02.117 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T13:59:02.117 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T13:59:02.117 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:59:02.117 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.117 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:59:02.117 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6802", 2024-09-21T13:59:02.117 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T13:59:02.117 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.117 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.117 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:59:02.117 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6803", 2024-09-21T13:59:02.117 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T13:59:02.117 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:02.118 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:02.118 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.118 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T13:59:02.118 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:59:02.118 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.118 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:59:02.118 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6804", 2024-09-21T13:59:02.118 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T13:59:02.118 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.118 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.118 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:59:02.118 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6805", 2024-09-21T13:59:02.118 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T13:59:02.118 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:02.119 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:02.119 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.119 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T13:59:02.119 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:59:02.119 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.119 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:59:02.120 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6808", 2024-09-21T13:59:02.120 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T13:59:02.120 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.120 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.120 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:59:02.120 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6809", 2024-09-21T13:59:02.120 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T13:59:02.120 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:02.120 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:02.120 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.120 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T13:59:02.120 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:59:02.120 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.121 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:59:02.121 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6806", 2024-09-21T13:59:02.121 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T13:59:02.121 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.121 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.121 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:59:02.121 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6807", 2024-09-21T13:59:02.121 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T13:59:02.121 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:02.121 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:02.121 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.121 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6803/2095561412", 2024-09-21T13:59:02.121 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6805/2095561412", 2024-09-21T13:59:02.121 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6809/2095561412", 2024-09-21T13:59:02.122 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6807/2095561412", 2024-09-21T13:59:02.122 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T13:59:02.122 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T13:59:02.122 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T13:59:02.122 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:02.122 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.122 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.122 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 1, 2024-09-21T13:59:02.122 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "0ba327f9-5524-4a40-abcc-729ded0b1917", 2024-09-21T13:59:02.122 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T13:59:02.122 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T13:59:02.122 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T13:59:02.122 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T13:59:02.123 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T13:59:02.123 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T13:59:02.123 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 13, 2024-09-21T13:59:02.123 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 279, 2024-09-21T13:59:02.123 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T13:59:02.123 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T13:59:02.123 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T13:59:02.123 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:59:02.123 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.123 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:59:02.123 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6810", 2024-09-21T13:59:02.123 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T13:59:02.124 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.124 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.124 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:59:02.124 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6811", 2024-09-21T13:59:02.124 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T13:59:02.124 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:02.124 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:02.124 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.124 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T13:59:02.124 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:59:02.124 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.124 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:59:02.124 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6812", 2024-09-21T13:59:02.124 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T13:59:02.125 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.125 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.125 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:59:02.125 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6813", 2024-09-21T13:59:02.125 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T13:59:02.125 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:02.125 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:02.125 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.125 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T13:59:02.125 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:59:02.125 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.125 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:59:02.125 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6816", 2024-09-21T13:59:02.125 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T13:59:02.126 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.126 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.126 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:59:02.126 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6817", 2024-09-21T13:59:02.126 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T13:59:02.126 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:02.126 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:02.126 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.126 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T13:59:02.126 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:59:02.126 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.126 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:59:02.126 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6814", 2024-09-21T13:59:02.126 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T13:59:02.127 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.127 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.127 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:59:02.127 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6815", 2024-09-21T13:59:02.127 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T13:59:02.127 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:02.127 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:02.127 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.127 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6811/1382840527", 2024-09-21T13:59:02.127 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6813/1382840527", 2024-09-21T13:59:02.127 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6817/1382840527", 2024-09-21T13:59:02.127 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6815/1382840527", 2024-09-21T13:59:02.127 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T13:59:02.127 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T13:59:02.127 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T13:59:02.128 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:02.128 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.128 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.128 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 2, 2024-09-21T13:59:02.128 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "27a45750-204d-48fd-bb8f-ebb01273343d", 2024-09-21T13:59:02.128 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T13:59:02.128 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T13:59:02.128 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T13:59:02.128 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 0.8268585205078125, 2024-09-21T13:59:02.128 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 18, 2024-09-21T13:59:02.128 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 100, 2024-09-21T13:59:02.128 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 130, 2024-09-21T13:59:02.128 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 273, 2024-09-21T13:59:02.128 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 101, 2024-09-21T13:59:02.128 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T13:59:02.129 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T13:59:02.129 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:59:02.129 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.129 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:59:02.129 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6818", 2024-09-21T13:59:02.129 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T13:59:02.129 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.129 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.129 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:59:02.129 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6819", 2024-09-21T13:59:02.129 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T13:59:02.130 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:02.130 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:02.130 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.130 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T13:59:02.130 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:59:02.130 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.130 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:59:02.130 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6820", 2024-09-21T13:59:02.130 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T13:59:02.130 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.130 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.130 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:59:02.130 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6821", 2024-09-21T13:59:02.131 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T13:59:02.131 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:02.131 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:02.131 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.131 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T13:59:02.131 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:59:02.131 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.131 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:59:02.131 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6824", 2024-09-21T13:59:02.131 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T13:59:02.131 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.131 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.131 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:59:02.131 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6825", 2024-09-21T13:59:02.131 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T13:59:02.132 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:02.132 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:02.132 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.132 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T13:59:02.132 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:59:02.132 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.132 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:59:02.132 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6822", 2024-09-21T13:59:02.132 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T13:59:02.132 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.132 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.132 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:59:02.132 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6823", 2024-09-21T13:59:02.133 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T13:59:02.133 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:02.133 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:02.133 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.133 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6819/1945314741", 2024-09-21T13:59:02.133 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6821/1945314741", 2024-09-21T13:59:02.133 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6825/1945314741", 2024-09-21T13:59:02.133 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6823/1945314741", 2024-09-21T13:59:02.133 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T13:59:02.133 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T13:59:02.133 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T13:59:02.133 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:02.133 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.133 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.133 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 3, 2024-09-21T13:59:02.134 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "528bc390-3dc9-4f3b-ab75-d7412c9625db", 2024-09-21T13:59:02.134 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T13:59:02.134 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T13:59:02.134 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T13:59:02.134 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T13:59:02.134 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 27, 2024-09-21T13:59:02.134 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 214, 2024-09-21T13:59:02.134 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 240, 2024-09-21T13:59:02.134 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 274, 2024-09-21T13:59:02.134 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 215, 2024-09-21T13:59:02.134 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T13:59:02.134 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T13:59:02.135 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:59:02.135 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.135 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:59:02.135 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6826", 2024-09-21T13:59:02.135 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T13:59:02.135 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.135 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.135 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:59:02.135 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6827", 2024-09-21T13:59:02.135 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T13:59:02.135 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:02.135 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:02.135 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.136 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T13:59:02.136 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:59:02.136 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.136 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:59:02.136 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6828", 2024-09-21T13:59:02.136 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T13:59:02.136 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.137 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.137 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:59:02.137 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6829", 2024-09-21T13:59:02.137 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T13:59:02.137 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:02.137 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:02.137 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.137 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T13:59:02.137 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:59:02.138 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.138 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:59:02.138 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6832", 2024-09-21T13:59:02.138 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T13:59:02.138 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.138 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.138 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:59:02.138 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6833", 2024-09-21T13:59:02.138 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T13:59:02.138 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:02.138 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:02.138 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.138 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T13:59:02.138 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:59:02.139 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.139 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:59:02.139 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6830", 2024-09-21T13:59:02.139 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T13:59:02.139 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.139 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.139 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:59:02.139 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6831", 2024-09-21T13:59:02.139 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T13:59:02.139 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:02.139 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:02.139 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.139 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6827/886735721", 2024-09-21T13:59:02.139 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6829/886735721", 2024-09-21T13:59:02.140 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6833/886735721", 2024-09-21T13:59:02.140 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6831/886735721", 2024-09-21T13:59:02.140 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T13:59:02.140 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T13:59:02.140 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T13:59:02.140 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:02.140 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.140 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.140 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 4, 2024-09-21T13:59:02.140 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "6fc64698-2d37-4db0-bf3d-c7e612c45163", 2024-09-21T13:59:02.140 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T13:59:02.140 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T13:59:02.140 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T13:59:02.140 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T13:59:02.141 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T13:59:02.141 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T13:59:02.141 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 33, 2024-09-21T13:59:02.141 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 273, 2024-09-21T13:59:02.141 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T13:59:02.141 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T13:59:02.141 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T13:59:02.141 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:59:02.141 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.141 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:59:02.141 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6800", 2024-09-21T13:59:02.141 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T13:59:02.141 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.141 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.142 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:59:02.142 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6801", 2024-09-21T13:59:02.142 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T13:59:02.142 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:02.142 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:02.142 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.142 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T13:59:02.142 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:59:02.142 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.142 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:59:02.142 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6802", 2024-09-21T13:59:02.142 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T13:59:02.142 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.142 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.142 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:59:02.143 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6803", 2024-09-21T13:59:02.143 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T13:59:02.143 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:02.143 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:02.143 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.143 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T13:59:02.143 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:59:02.143 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.143 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:59:02.143 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6806", 2024-09-21T13:59:02.143 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T13:59:02.143 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.143 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.143 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:59:02.144 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6807", 2024-09-21T13:59:02.144 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T13:59:02.144 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:02.144 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:02.144 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.144 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T13:59:02.144 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:59:02.144 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.144 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:59:02.144 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6804", 2024-09-21T13:59:02.144 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T13:59:02.144 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.145 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.145 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:59:02.145 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6805", 2024-09-21T13:59:02.145 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T13:59:02.145 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:02.145 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:02.145 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.145 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.187:6801/1702447776", 2024-09-21T13:59:02.145 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.187:6803/1702447776", 2024-09-21T13:59:02.145 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.187:6807/1702447776", 2024-09-21T13:59:02.145 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.187:6805/1702447776", 2024-09-21T13:59:02.145 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T13:59:02.145 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T13:59:02.145 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T13:59:02.146 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:02.146 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.146 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.146 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 5, 2024-09-21T13:59:02.146 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "8d7f62a1-7e92-4e06-9cad-86b96be514d8", 2024-09-21T13:59:02.146 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T13:59:02.146 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T13:59:02.146 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T13:59:02.146 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T13:59:02.146 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T13:59:02.146 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T13:59:02.146 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 39, 2024-09-21T13:59:02.146 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 273, 2024-09-21T13:59:02.147 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T13:59:02.147 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T13:59:02.147 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T13:59:02.147 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:59:02.147 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.147 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:59:02.147 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6808", 2024-09-21T13:59:02.147 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3659050776 2024-09-21T13:59:02.147 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.147 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.147 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:59:02.147 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6809", 2024-09-21T13:59:02.147 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3659050776 2024-09-21T13:59:02.147 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:02.147 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:02.148 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.148 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T13:59:02.148 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:59:02.148 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.148 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:59:02.148 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6810", 2024-09-21T13:59:02.148 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3659050776 2024-09-21T13:59:02.148 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.148 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.148 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:59:02.148 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6811", 2024-09-21T13:59:02.148 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3659050776 2024-09-21T13:59:02.149 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:02.149 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:02.149 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.149 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T13:59:02.149 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:59:02.149 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.149 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:59:02.149 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6814", 2024-09-21T13:59:02.149 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3659050776 2024-09-21T13:59:02.149 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.149 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.149 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:59:02.150 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6815", 2024-09-21T13:59:02.150 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3659050776 2024-09-21T13:59:02.150 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:02.150 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:02.150 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.150 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T13:59:02.150 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:59:02.150 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.150 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:59:02.150 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6812", 2024-09-21T13:59:02.151 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3659050776 2024-09-21T13:59:02.151 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.151 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.151 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:59:02.151 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6813", 2024-09-21T13:59:02.151 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3659050776 2024-09-21T13:59:02.151 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:02.151 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:02.151 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.151 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.187:6809/3659050776", 2024-09-21T13:59:02.151 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.187:6811/3659050776", 2024-09-21T13:59:02.151 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.187:6815/3659050776", 2024-09-21T13:59:02.151 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.187:6813/3659050776", 2024-09-21T13:59:02.152 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T13:59:02.152 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T13:59:02.152 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T13:59:02.152 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:02.152 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.152 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.152 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 6, 2024-09-21T13:59:02.152 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "3a7afd13-26ce-444c-9cc0-81ce2eab6ce5", 2024-09-21T13:59:02.152 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T13:59:02.152 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T13:59:02.152 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T13:59:02.152 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T13:59:02.152 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T13:59:02.152 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T13:59:02.153 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 47, 2024-09-21T13:59:02.153 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 278, 2024-09-21T13:59:02.153 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T13:59:02.153 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T13:59:02.153 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T13:59:02.153 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:59:02.153 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.153 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:59:02.153 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6816", 2024-09-21T13:59:02.153 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T13:59:02.153 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.153 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.153 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:59:02.153 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6817", 2024-09-21T13:59:02.154 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T13:59:02.154 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:02.154 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:02.154 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.154 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T13:59:02.154 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:59:02.154 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.154 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:59:02.154 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6818", 2024-09-21T13:59:02.154 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T13:59:02.154 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.154 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.155 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:59:02.155 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6819", 2024-09-21T13:59:02.155 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T13:59:02.155 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:02.155 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:02.155 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.155 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T13:59:02.155 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:59:02.155 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.155 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:59:02.155 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6822", 2024-09-21T13:59:02.155 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T13:59:02.155 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.155 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.156 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:59:02.156 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6823", 2024-09-21T13:59:02.156 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T13:59:02.156 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:02.156 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:02.156 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.156 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T13:59:02.156 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:59:02.156 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.156 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:59:02.156 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6820", 2024-09-21T13:59:02.156 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T13:59:02.156 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.156 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.157 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:59:02.157 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6821", 2024-09-21T13:59:02.157 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T13:59:02.157 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:02.157 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:02.157 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.157 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.187:6817/2728426525", 2024-09-21T13:59:02.157 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.187:6819/2728426525", 2024-09-21T13:59:02.157 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.187:6823/2728426525", 2024-09-21T13:59:02.158 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.187:6821/2728426525", 2024-09-21T13:59:02.158 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T13:59:02.158 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T13:59:02.158 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T13:59:02.158 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:02.158 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.158 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.158 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 7, 2024-09-21T13:59:02.158 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "1e08d891-8747-402e-b4c2-aa3802ce8537", 2024-09-21T13:59:02.158 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 0, 2024-09-21T13:59:02.158 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T13:59:02.159 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T13:59:02.159 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T13:59:02.159 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T13:59:02.159 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T13:59:02.159 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 54, 2024-09-21T13:59:02.159 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 269, 2024-09-21T13:59:02.159 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 273, 2024-09-21T13:59:02.159 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T13:59:02.159 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T13:59:02.159 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:59:02.159 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.159 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:59:02.159 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6824", 2024-09-21T13:59:02.159 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3680850037 2024-09-21T13:59:02.160 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.160 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.160 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:59:02.160 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6825", 2024-09-21T13:59:02.160 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3680850037 2024-09-21T13:59:02.160 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:02.160 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:02.160 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.160 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T13:59:02.160 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:59:02.160 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.160 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:59:02.160 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6826", 2024-09-21T13:59:02.160 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3680850037 2024-09-21T13:59:02.161 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.161 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.161 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:59:02.161 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6827", 2024-09-21T13:59:02.161 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3680850037 2024-09-21T13:59:02.161 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:02.161 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:02.161 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.161 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T13:59:02.161 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:59:02.161 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.161 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:59:02.161 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6830", 2024-09-21T13:59:02.162 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3680850037 2024-09-21T13:59:02.162 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.162 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.162 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:59:02.162 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6831", 2024-09-21T13:59:02.162 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3680850037 2024-09-21T13:59:02.162 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:02.162 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:02.162 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.162 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T13:59:02.162 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T13:59:02.162 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.162 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T13:59:02.162 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6828", 2024-09-21T13:59:02.163 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3680850037 2024-09-21T13:59:02.163 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.163 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.163 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T13:59:02.163 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6829", 2024-09-21T13:59:02.163 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3680850037 2024-09-21T13:59:02.163 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:02.163 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:02.163 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.163 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.187:6825/3680850037", 2024-09-21T13:59:02.163 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.187:6827/3680850037", 2024-09-21T13:59:02.163 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.187:6831/3680850037", 2024-09-21T13:59:02.163 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.187:6829/3680850037", 2024-09-21T13:59:02.163 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T13:59:02.164 INFO:teuthology.orchestra.run.smithi049.stdout: "exists" 2024-09-21T13:59:02.164 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:02.164 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:02.164 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T13:59:02.164 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_xinfo": [ 2024-09-21T13:59:02.164 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.164 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 0, 2024-09-21T13:59:02.164 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T13:59:02.164 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T13:59:02.164 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T13:59:02.164 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T13:59:02.164 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T13:59:02.164 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T13:58:06.873692+0000", 2024-09-21T13:59:02.165 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T13:59:02.165 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.165 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.165 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 1, 2024-09-21T13:59:02.165 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T13:59:02.165 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T13:59:02.165 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T13:59:02.165 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T13:59:02.165 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T13:59:02.165 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T13:57:48.533160+0000", 2024-09-21T13:59:02.165 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T13:59:02.165 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.165 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.165 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 2, 2024-09-21T13:59:02.166 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T13:59:02.166 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T13:59:02.166 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T13:59:02.166 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T13:59:02.166 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T13:59:02.166 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T13:58:53.385022+0000", 2024-09-21T13:59:02.166 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 100 2024-09-21T13:59:02.166 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.166 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.166 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 3, 2024-09-21T13:59:02.166 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T13:59:02.166 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T13:59:02.166 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T13:59:02.166 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T13:59:02.167 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T13:59:02.167 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T13:58:30.838705+0000", 2024-09-21T13:59:02.167 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 214 2024-09-21T13:59:02.167 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.167 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.167 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 4, 2024-09-21T13:59:02.167 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T13:59:02.167 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T13:59:02.167 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T13:59:02.167 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T13:59:02.167 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T13:59:02.167 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T13:58:02.489794+0000", 2024-09-21T13:59:02.167 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T13:59:02.167 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.168 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.168 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 5, 2024-09-21T13:59:02.168 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T13:59:02.168 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T13:59:02.168 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T13:59:02.168 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T13:59:02.168 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T13:59:02.168 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T13:58:42.091015+0000", 2024-09-21T13:59:02.168 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T13:59:02.168 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.168 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.168 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 6, 2024-09-21T13:59:02.169 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T13:59:02.169 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T13:59:02.169 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T13:59:02.169 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T13:59:02.169 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T13:59:02.169 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T13:58:28.652486+0000", 2024-09-21T13:59:02.169 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T13:59:02.169 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.169 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.169 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 7, 2024-09-21T13:59:02.169 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T13:59:02.169 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T13:59:02.169 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T13:59:02.169 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T13:59:02.170 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T13:59:02.170 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T13:56:59.703741+0000", 2024-09-21T13:59:02.170 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 272 2024-09-21T13:59:02.170 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:02.170 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T13:59:02.170 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap": [ 2024-09-21T13:59:02.170 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.170 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "3.14", 2024-09-21T13:59:02.170 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-09-21T13:59:02.170 INFO:teuthology.orchestra.run.smithi049.stdout: 7, 2024-09-21T13:59:02.170 INFO:teuthology.orchestra.run.smithi049.stdout: 6, 2024-09-21T13:59:02.170 INFO:teuthology.orchestra.run.smithi049.stdout: 0 2024-09-21T13:59:02.170 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:02.171 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:02.171 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T13:59:02.171 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap_items": [ 2024-09-21T13:59:02.171 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.171 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "4.6", 2024-09-21T13:59:02.171 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-09-21T13:59:02.171 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.171 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 1, 2024-09-21T13:59:02.171 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 5 2024-09-21T13:59:02.171 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:02.171 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:02.171 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.171 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.171 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "4.1b", 2024-09-21T13:59:02.171 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-09-21T13:59:02.172 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.172 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 1, 2024-09-21T13:59:02.172 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 2 2024-09-21T13:59:02.172 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:02.172 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:02.172 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:02.172 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T13:59:02.172 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap_primaries": [], 2024-09-21T13:59:02.172 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_temp": [ 2024-09-21T13:59:02.172 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:02.172 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "9.1", 2024-09-21T13:59:02.172 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-09-21T13:59:02.172 INFO:teuthology.orchestra.run.smithi049.stdout: 3, 2024-09-21T13:59:02.172 INFO:teuthology.orchestra.run.smithi049.stdout: 6, 2024-09-21T13:59:02.172 INFO:teuthology.orchestra.run.smithi049.stdout: 4 2024-09-21T13:59:02.173 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:02.173 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:02.173 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T13:59:02.173 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_temp": [], 2024-09-21T13:59:02.173 INFO:teuthology.orchestra.run.smithi049.stdout: "blocklist": { 2024-09-21T13:59:02.173 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/2731793917": "2024-09-22T13:49:27.311313+0000", 2024-09-21T13:59:02.173 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/799831425": "2024-09-22T13:49:27.311313+0000", 2024-09-21T13:59:02.173 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/2518197280": "2024-09-22T13:43:28.110499+0000", 2024-09-21T13:59:02.173 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3556829486": "2024-09-22T13:43:28.110499+0000", 2024-09-21T13:59:02.173 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/1249926115": "2024-09-22T13:49:27.311313+0000", 2024-09-21T13:59:02.173 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/231530138": "2024-09-22T13:43:28.110499+0000", 2024-09-21T13:59:02.173 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/4220781134": "2024-09-22T13:49:27.311313+0000", 2024-09-21T13:59:02.173 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/202325455": "2024-09-22T13:43:28.110499+0000", 2024-09-21T13:59:02.173 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/4129636473": "2024-09-22T13:43:56.633723+0000", 2024-09-21T13:59:02.174 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/2518197280": "2024-09-22T13:43:28.110499+0000", 2024-09-21T13:59:02.174 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3514741273": "2024-09-22T13:43:56.633723+0000", 2024-09-21T13:59:02.174 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/4129636473": "2024-09-22T13:43:56.633723+0000", 2024-09-21T13:59:02.174 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3125743776": "2024-09-22T13:43:56.633723+0000", 2024-09-21T13:59:02.174 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3600977363": "2024-09-22T13:43:56.633723+0000", 2024-09-21T13:59:02.174 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/3453571598": "2024-09-22T13:49:27.311313+0000", 2024-09-21T13:59:02.174 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/3453571598": "2024-09-22T13:49:27.311313+0000", 2024-09-21T13:59:02.174 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3553734432": "2024-09-22T13:49:27.311313+0000" 2024-09-21T13:59:02.174 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.174 INFO:teuthology.orchestra.run.smithi049.stdout: "range_blocklist": {}, 2024-09-21T13:59:02.174 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profiles": { 2024-09-21T13:59:02.174 INFO:teuthology.orchestra.run.smithi049.stdout: "default": { 2024-09-21T13:59:02.174 INFO:teuthology.orchestra.run.smithi049.stdout: "crush-failure-domain": "osd", 2024-09-21T13:59:02.174 INFO:teuthology.orchestra.run.smithi049.stdout: "k": "2", 2024-09-21T13:59:02.175 INFO:teuthology.orchestra.run.smithi049.stdout: "m": "1", 2024-09-21T13:59:02.175 INFO:teuthology.orchestra.run.smithi049.stdout: "plugin": "jerasure", 2024-09-21T13:59:02.175 INFO:teuthology.orchestra.run.smithi049.stdout: "technique": "reed_sol_van" 2024-09-21T13:59:02.175 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:02.175 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:02.175 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps_queue": [], 2024-09-21T13:59:02.175 INFO:teuthology.orchestra.run.smithi049.stdout: "new_removed_snaps": [], 2024-09-21T13:59:02.175 INFO:teuthology.orchestra.run.smithi049.stdout: "new_purged_snaps": [], 2024-09-21T13:59:02.175 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_node_flags": {}, 2024-09-21T13:59:02.175 INFO:teuthology.orchestra.run.smithi049.stdout: "device_class_flags": {}, 2024-09-21T13:59:02.175 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode": { 2024-09-21T13:59:02.176 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode_enabled": false, 2024-09-21T13:59:02.176 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_bucket_count": 0, 2024-09-21T13:59:02.176 INFO:teuthology.orchestra.run.smithi049.stdout: "degraded_stretch_mode": 0, 2024-09-21T13:59:02.176 INFO:teuthology.orchestra.run.smithi049.stdout: "recovering_stretch_mode": 0, 2024-09-21T13:59:02.176 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode_bucket": 0 2024-09-21T13:59:02.176 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:02.176 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T13:59:02.176 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:02.217 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:02.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:02.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:02.319 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:02.354 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:02.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:02.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:02.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:01 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3205731878' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:59:02.456 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:02.497 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:02.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:02.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:02.598 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:59:02.633 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T13:59:02.640 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:02.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:02.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:59:02.674 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 58 16 86423 86407 93.0945 75.9375 0.0184223 0.0107372 2024-09-21T13:59:02.744 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:59:02.770 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:02.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:02.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:59:02.872 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:02.900 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:02.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:02.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:03.002 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:03.046 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:03.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:03.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:03.148 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:59:03.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:03.188 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:03.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:59:03.211 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:02 smithi049 ceph-mon[40489]: pgmap v506: 148 pgs: 1 active+clean+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 5.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 744 B/s rd, 77 MiB/s wr, 1.23k op/s; 41193/247749 objects degraded (16.627%); 5201/247749 objects misplaced (2.099%) 2024-09-21T13:59:03.212 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:02 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2427036755' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T13:59:03.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:02 smithi049 ceph-mon[30080]: pgmap v506: 148 pgs: 1 active+clean+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 5.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 744 B/s rd, 77 MiB/s wr, 1.23k op/s; 41193/247749 objects degraded (16.627%); 5201/247749 objects misplaced (2.099%) 2024-09-21T13:59:03.212 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:02 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2427036755' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T13:59:03.289 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:59:03.327 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:03.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:03.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:59:03.429 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:03.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:02 smithi187 ceph-mon[33032]: pgmap v506: 148 pgs: 1 active+clean+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 5.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 744 B/s rd, 77 MiB/s wr, 1.23k op/s; 41193/247749 objects degraded (16.627%); 5201/247749 objects misplaced (2.099%) 2024-09-21T13:59:03.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:02 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2427036755' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T13:59:03.467 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:03.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:03.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:03.569 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:59:03.598 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:03.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:03.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:59:03.674 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 59 16 87661 87645 92.828 77.375 0.0101705 0.0107674 2024-09-21T13:59:03.700 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:59:03.726 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:03.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:03.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:59:03.828 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:59:03.856 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:03.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:03.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:59:03.957 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:59:03.984 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:03.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:03.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:59:04.086 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:59:04.116 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:04.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:04.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:59:04.171 INFO:tasks.thrashosds.thrasher:j is {'epoch': 290, 'fsid': '40a3d2e8-781f-11ef-baf6-efdc52797490', 'created': '2024-09-21T13:42:57.838275+0000', 'modified': '2024-09-21T13:59:00.016334+0000', 'last_up_change': '2024-09-21T13:58:23.782318+0000', 'last_in_change': '2024-09-21T13:58:16.730861+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 20, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 9, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T13:46:21.096244+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '199', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8685569763183594, 'average_primary_affinity': 0.8700000047683716, 'average_primary_affinity_weighted': 0.8700000047683716}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-21T13:48:55.540712+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 63, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6700000762939453, 'score_stable': 2.6700000762939453, 'optimal_score': 0.75, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8685569763183594, 'average_primary_affinity': 0.8700000047683716, 'average_primary_affinity_weighted': 0.8700000047683716}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-21T13:48:56.417456+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 0.8685569763183594, 'average_primary_affinity': 0.8700000047683716, 'average_primary_affinity_weighted': 0.8700000047683716}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-21T13:48:58.128455+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 0.8685569763183594, 'average_primary_affinity': 0.8700000047683716, 'average_primary_affinity_weighted': 0.8700000047683716}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-21T13:49:00.069361+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 0.8685569763183594, 'average_primary_affinity': 0.8700000047683716, 'average_primary_affinity_weighted': 0.8700000047683716}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-21T13:49:02.127399+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 0.8685569763183594, 'average_primary_affinity': 0.8700000047683716, 'average_primary_affinity_weighted': 0.8700000047683716}}, {'pool': 9, 'pool_name': 'unique_pool_2', 'create_time': '2024-09-21T13:57:51.528024+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '263', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.990000009536743, 'score_stable': 2.990000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.630000114440918, 'raw_score_stable': 2.630000114440918, 'primary_affinity_weighted': 0.8685569763183594, 'average_primary_affinity': 0.8700000047683716, 'average_primary_affinity_weighted': 0.8700000047683716}}], 'osds': [{'osd': 0, 'uuid': '2595dca9-6219-4e0c-8728-751eb102e73a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.1215972900390625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 277, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6802', 'nonce': 2095561412}, {'type': 'v1', 'addr': '172.21.15.49:6803', 'nonce': 2095561412}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6804', 'nonce': 2095561412}, {'type': 'v1', 'addr': '172.21.15.49:6805', 'nonce': 2095561412}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6808', 'nonce': 2095561412}, {'type': 'v1', 'addr': '172.21.15.49:6809', 'nonce': 2095561412}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6806', 'nonce': 2095561412}, {'type': 'v1', 'addr': '172.21.15.49:6807', 'nonce': 2095561412}]}, 'public_addr': '172.21.15.49:6803/2095561412', 'cluster_addr': '172.21.15.49:6805/2095561412', 'heartbeat_back_addr': '172.21.15.49:6809/2095561412', 'heartbeat_front_addr': '172.21.15.49:6807/2095561412', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '0ba327f9-5524-4a40-abcc-729ded0b1917', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 279, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6810', 'nonce': 1382840527}, {'type': 'v1', 'addr': '172.21.15.49:6811', 'nonce': 1382840527}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6812', 'nonce': 1382840527}, {'type': 'v1', 'addr': '172.21.15.49:6813', 'nonce': 1382840527}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6816', 'nonce': 1382840527}, {'type': 'v1', 'addr': '172.21.15.49:6817', 'nonce': 1382840527}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6814', 'nonce': 1382840527}, {'type': 'v1', 'addr': '172.21.15.49:6815', 'nonce': 1382840527}]}, 'public_addr': '172.21.15.49:6811/1382840527', 'cluster_addr': '172.21.15.49:6813/1382840527', 'heartbeat_back_addr': '172.21.15.49:6817/1382840527', 'heartbeat_front_addr': '172.21.15.49:6815/1382840527', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '27a45750-204d-48fd-bb8f-ebb01273343d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.8268585205078125, 'last_clean_begin': 18, 'last_clean_end': 100, 'up_from': 130, 'up_thru': 273, 'down_at': 101, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6818', 'nonce': 1945314741}, {'type': 'v1', 'addr': '172.21.15.49:6819', 'nonce': 1945314741}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6820', 'nonce': 1945314741}, {'type': 'v1', 'addr': '172.21.15.49:6821', 'nonce': 1945314741}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6824', 'nonce': 1945314741}, {'type': 'v1', 'addr': '172.21.15.49:6825', 'nonce': 1945314741}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6822', 'nonce': 1945314741}, {'type': 'v1', 'addr': '172.21.15.49:6823', 'nonce': 1945314741}]}, 'public_addr': '172.21.15.49:6819/1945314741', 'cluster_addr': '172.21.15.49:6821/1945314741', 'heartbeat_back_addr': '172.21.15.49:6825/1945314741', 'heartbeat_front_addr': '172.21.15.49:6823/1945314741', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '528bc390-3dc9-4f3b-ab75-d7412c9625db', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 27, 'last_clean_end': 214, 'up_from': 240, 'up_thru': 274, 'down_at': 215, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6826', 'nonce': 886735721}, {'type': 'v1', 'addr': '172.21.15.49:6827', 'nonce': 886735721}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6828', 'nonce': 886735721}, {'type': 'v1', 'addr': '172.21.15.49:6829', 'nonce': 886735721}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6832', 'nonce': 886735721}, {'type': 'v1', 'addr': '172.21.15.49:6833', 'nonce': 886735721}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6830', 'nonce': 886735721}, {'type': 'v1', 'addr': '172.21.15.49:6831', 'nonce': 886735721}]}, 'public_addr': '172.21.15.49:6827/886735721', 'cluster_addr': '172.21.15.49:6829/886735721', 'heartbeat_back_addr': '172.21.15.49:6833/886735721', 'heartbeat_front_addr': '172.21.15.49:6831/886735721', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '6fc64698-2d37-4db0-bf3d-c7e612c45163', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 273, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6800', 'nonce': 1702447776}, {'type': 'v1', 'addr': '172.21.15.187:6801', 'nonce': 1702447776}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6802', 'nonce': 1702447776}, {'type': 'v1', 'addr': '172.21.15.187:6803', 'nonce': 1702447776}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6806', 'nonce': 1702447776}, {'type': 'v1', 'addr': '172.21.15.187:6807', 'nonce': 1702447776}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6804', 'nonce': 1702447776}, {'type': 'v1', 'addr': '172.21.15.187:6805', 'nonce': 1702447776}]}, 'public_addr': '172.21.15.187:6801/1702447776', 'cluster_addr': '172.21.15.187:6803/1702447776', 'heartbeat_back_addr': '172.21.15.187:6807/1702447776', 'heartbeat_front_addr': '172.21.15.187:6805/1702447776', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '8d7f62a1-7e92-4e06-9cad-86b96be514d8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 273, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6808', 'nonce': 3659050776}, {'type': 'v1', 'addr': '172.21.15.187:6809', 'nonce': 3659050776}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6810', 'nonce': 3659050776}, {'type': 'v1', 'addr': '172.21.15.187:6811', 'nonce': 3659050776}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6814', 'nonce': 3659050776}, {'type': 'v1', 'addr': '172.21.15.187:6815', 'nonce': 3659050776}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6812', 'nonce': 3659050776}, {'type': 'v1', 'addr': '172.21.15.187:6813', 'nonce': 3659050776}]}, 'public_addr': '172.21.15.187:6809/3659050776', 'cluster_addr': '172.21.15.187:6811/3659050776', 'heartbeat_back_addr': '172.21.15.187:6815/3659050776', 'heartbeat_front_addr': '172.21.15.187:6813/3659050776', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '3a7afd13-26ce-444c-9cc0-81ce2eab6ce5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 278, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6816', 'nonce': 2728426525}, {'type': 'v1', 'addr': '172.21.15.187:6817', 'nonce': 2728426525}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6818', 'nonce': 2728426525}, {'type': 'v1', 'addr': '172.21.15.187:6819', 'nonce': 2728426525}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6822', 'nonce': 2728426525}, {'type': 'v1', 'addr': '172.21.15.187:6823', 'nonce': 2728426525}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6820', 'nonce': 2728426525}, {'type': 'v1', 'addr': '172.21.15.187:6821', 'nonce': 2728426525}]}, 'public_addr': '172.21.15.187:6817/2728426525', 'cluster_addr': '172.21.15.187:6819/2728426525', 'heartbeat_back_addr': '172.21.15.187:6823/2728426525', 'heartbeat_front_addr': '172.21.15.187:6821/2728426525', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1e08d891-8747-402e-b4c2-aa3802ce8537', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 269, 'down_at': 273, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 3680850037}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 3680850037}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 3680850037}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 3680850037}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6830', 'nonce': 3680850037}, {'type': 'v1', 'addr': '172.21.15.187:6831', 'nonce': 3680850037}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6828', 'nonce': 3680850037}, {'type': 'v1', 'addr': '172.21.15.187:6829', 'nonce': 3680850037}]}, 'public_addr': '172.21.15.187:6825/3680850037', 'cluster_addr': '172.21.15.187:6827/3680850037', 'heartbeat_back_addr': '172.21.15.187:6831/3680850037', 'heartbeat_front_addr': '172.21.15.187:6829/3680850037', 'state': ['exists']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T13:58:06.873692+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T13:57:48.533160+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T13:58:53.385022+0000', 'dead_epoch': 100}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T13:58:30.838705+0000', 'dead_epoch': 214}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T13:58:02.489794+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T13:58:42.091015+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T13:58:28.652486+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T13:56:59.703741+0000', 'dead_epoch': 272}], 'pg_upmap': [{'pgid': '3.14', 'osds': [7, 6, 0]}], 'pg_upmap_items': [{'pgid': '4.6', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.1b', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '9.1', 'osds': [3, 6, 4]}], 'primary_temp': [], 'blocklist': {'172.21.15.49:0/2731793917': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:0/799831425': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:6801/2518197280': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:0/3556829486': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:0/1249926115': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:0/231530138': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:0/4220781134': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:0/202325455': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:6800/4129636473': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:6800/2518197280': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:0/3514741273': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:6801/4129636473': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:0/3125743776': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:0/3600977363': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:6801/3453571598': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:6800/3453571598': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:0/3553734432': '2024-09-22T13:49:27.311313+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-21T13:59:04.171 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 4.1b 2024-09-21T13:59:04.171 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd rm-pg-upmap-items 4.1b 2024-09-21T13:59:04.218 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:04.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:04.267 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:04.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:04.368 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:04.412 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:04.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:04.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:04.514 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:04.553 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:04.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:04.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:04.655 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:04.669 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T13:59:04.669 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T13:59:04.674 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:2024-09-21T13:59:04.675442+0000 min lat: 0.00125866 max lat: 2.32137 avg lat: 0.0107898 2024-09-21T13:59:04.674 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T13:59:04.674 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 60 16 88966 88950 92.6401 81.5625 0.0157707 0.0107898 2024-09-21T13:59:04.681 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:04.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:04.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:04.783 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:59:04.786 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T13:59:04.786 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-09-21T13:59:04.786 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-09-21T13:59:04.786 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-09-21T13:59:04.786 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.786 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232022 8.f 8.d7b7a40f (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T13:59:04.786 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.700289+0000", 2024-09-21T13:59:04.786 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 111.435740364, 2024-09-21T13:59:04.786 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97395471300000003, 2024-09-21T13:59:04.787 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:59:04.787 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:59:04.787 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T13:59:04.787 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:59:04.787 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:59:04.787 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:59:04.787 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232022 2024-09-21T13:59:04.787 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.787 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:59:04.787 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.787 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:59:04.787 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700289+0000", 2024-09-21T13:59:04.787 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:04.787 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.788 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.788 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:59:04.788 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700289+0000", 2024-09-21T13:59:04.788 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:04.788 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.788 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.788 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:59:04.788 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700291+0000", 2024-09-21T13:59:04.788 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5850000000000001e-06 2024-09-21T13:59:04.788 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.788 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.788 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:59:04.788 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700296+0000", 2024-09-21T13:59:04.788 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9749999999999998e-06 2024-09-21T13:59:04.789 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.789 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.789 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:59:04.789 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700306+0000", 2024-09-21T13:59:04.789 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9310000000000003e-06 2024-09-21T13:59:04.789 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.789 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.789 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:59:04.789 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700317+0000", 2024-09-21T13:59:04.789 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1199000000000001e-05 2024-09-21T13:59:04.789 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.789 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.789 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:59:04.790 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.711626+0000", 2024-09-21T13:59:04.790 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.011309009 2024-09-21T13:59:04.790 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.790 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.790 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:59:04.790 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.711637+0000", 2024-09-21T13:59:04.790 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1182000000000001e-05 2024-09-21T13:59:04.790 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.790 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.790 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:59:04.790 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674230+0000", 2024-09-21T13:59:04.790 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.962593535 2024-09-21T13:59:04.791 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.791 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.791 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:59:04.791 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674244+0000", 2024-09-21T13:59:04.791 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3297e-05 2024-09-21T13:59:04.791 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:04.791 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:04.791 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:04.791 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.791 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.791 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232025 8.f 8.4e5a668f (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T13:59:04.791 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.700329+0000", 2024-09-21T13:59:04.791 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 111.43570056599999, 2024-09-21T13:59:04.791 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97395089300000004, 2024-09-21T13:59:04.792 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:59:04.792 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:59:04.792 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T13:59:04.792 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:59:04.792 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:59:04.792 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:59:04.792 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232025 2024-09-21T13:59:04.792 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.792 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:59:04.792 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.792 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:59:04.792 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700329+0000", 2024-09-21T13:59:04.792 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:04.793 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.793 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.793 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:59:04.793 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700329+0000", 2024-09-21T13:59:04.793 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:04.793 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.793 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.793 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:59:04.793 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700329+0000", 2024-09-21T13:59:04.793 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7100000000000002e-07 2024-09-21T13:59:04.793 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.793 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.793 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:59:04.793 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700332+0000", 2024-09-21T13:59:04.794 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6589999999999999e-06 2024-09-21T13:59:04.794 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.794 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.794 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:59:04.794 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700339+0000", 2024-09-21T13:59:04.794 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8940000000000004e-06 2024-09-21T13:59:04.794 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.794 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.794 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:59:04.794 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700345+0000", 2024-09-21T13:59:04.794 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6219999999999997e-06 2024-09-21T13:59:04.794 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.794 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.794 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:59:04.794 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.711662+0000", 2024-09-21T13:59:04.795 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.011317522 2024-09-21T13:59:04.795 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.795 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.795 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:59:04.795 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.711669+0000", 2024-09-21T13:59:04.795 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2570000000000001e-06 2024-09-21T13:59:04.795 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.795 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.795 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:59:04.795 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674270+0000", 2024-09-21T13:59:04.795 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.96260044300000003 2024-09-21T13:59:04.795 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.795 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.795 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:59:04.796 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674280+0000", 2024-09-21T13:59:04.796 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0025e-05 2024-09-21T13:59:04.796 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:04.796 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:04.796 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:04.796 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.796 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.796 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232033 8.f 8.d949c02f (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T13:59:04.796 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.700352+0000", 2024-09-21T13:59:04.796 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 111.435677166, 2024-09-21T13:59:04.796 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97396006099999999, 2024-09-21T13:59:04.796 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:59:04.796 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:59:04.796 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T13:59:04.796 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:59:04.797 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:59:04.797 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:59:04.797 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232033 2024-09-21T13:59:04.797 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.797 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:59:04.797 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.797 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:59:04.797 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700352+0000", 2024-09-21T13:59:04.797 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:04.797 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.797 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.797 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:59:04.797 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700352+0000", 2024-09-21T13:59:04.798 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:04.798 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.798 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.798 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:59:04.798 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700353+0000", 2024-09-21T13:59:04.798 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.89e-07 2024-09-21T13:59:04.798 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.798 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.798 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:59:04.798 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700355+0000", 2024-09-21T13:59:04.798 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1840000000000002e-06 2024-09-21T13:59:04.798 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.798 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.798 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:59:04.799 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700364+0000", 2024-09-21T13:59:04.799 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4919999999999999e-06 2024-09-21T13:59:04.799 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.799 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.799 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:59:04.799 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700372+0000", 2024-09-21T13:59:04.799 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9659999999999996e-06 2024-09-21T13:59:04.799 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.799 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.799 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:59:04.799 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.711693+0000", 2024-09-21T13:59:04.799 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.011321059 2024-09-21T13:59:04.799 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.800 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.800 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:59:04.800 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.711701+0000", 2024-09-21T13:59:04.800 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3640000000000001e-06 2024-09-21T13:59:04.800 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.800 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.800 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:59:04.800 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674303+0000", 2024-09-21T13:59:04.800 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.96260253100000004 2024-09-21T13:59:04.800 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.800 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.800 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:59:04.800 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674312+0000", 2024-09-21T13:59:04.800 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0759999999999992e-06 2024-09-21T13:59:04.801 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:04.801 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:04.801 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:04.801 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.801 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.801 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232051 8.0 8.85c8f300 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T13:59:04.801 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.700380+0000", 2024-09-21T13:59:04.801 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 111.435649463, 2024-09-21T13:59:04.801 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97451152799999996, 2024-09-21T13:59:04.801 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:59:04.801 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:59:04.801 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T13:59:04.801 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:59:04.801 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:59:04.802 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:59:04.802 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232051 2024-09-21T13:59:04.802 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.802 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:59:04.802 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.802 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:59:04.802 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700380+0000", 2024-09-21T13:59:04.802 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:04.802 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.802 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.802 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:59:04.802 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700380+0000", 2024-09-21T13:59:04.802 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:04.802 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.803 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.803 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:59:04.803 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700380+0000", 2024-09-21T13:59:04.803 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1699999999999999e-07 2024-09-21T13:59:04.803 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.803 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.803 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:59:04.803 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700383+0000", 2024-09-21T13:59:04.803 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1600000000000001e-06 2024-09-21T13:59:04.803 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.803 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.803 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:59:04.803 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700389+0000", 2024-09-21T13:59:04.803 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6080000000000001e-06 2024-09-21T13:59:04.803 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.804 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.804 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:59:04.804 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700394+0000", 2024-09-21T13:59:04.804 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9810000000000003e-06 2024-09-21T13:59:04.804 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.804 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.804 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:59:04.804 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.715050+0000", 2024-09-21T13:59:04.804 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.014656269 2024-09-21T13:59:04.804 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.804 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.804 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:59:04.804 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.715056+0000", 2024-09-21T13:59:04.804 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0129999999999999e-06 2024-09-21T13:59:04.804 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.805 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.805 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:59:04.805 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674876+0000", 2024-09-21T13:59:04.805 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.95981962700000001 2024-09-21T13:59:04.805 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.805 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.805 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:59:04.805 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674892+0000", 2024-09-21T13:59:04.805 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5452999999999999e-05 2024-09-21T13:59:04.805 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:04.805 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:04.805 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:04.805 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.805 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.805 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232059 8.f 8.b757f3bf (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T13:59:04.805 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.889250+0000", 2024-09-21T13:59:04.806 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 111.24677964999999, 2024-09-21T13:59:04.806 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.78509475900000003, 2024-09-21T13:59:04.806 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:59:04.806 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:59:04.806 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T13:59:04.806 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:59:04.806 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:59:04.806 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:59:04.806 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232059 2024-09-21T13:59:04.806 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.806 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:59:04.806 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.806 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:59:04.806 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889250+0000", 2024-09-21T13:59:04.807 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:04.807 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.807 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.807 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:59:04.807 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889250+0000", 2024-09-21T13:59:04.807 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:04.807 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.807 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.807 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:59:04.807 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889252+0000", 2024-09-21T13:59:04.807 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9130000000000001e-06 2024-09-21T13:59:04.807 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.807 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.807 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:59:04.807 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889258+0000", 2024-09-21T13:59:04.808 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7410000000000003e-06 2024-09-21T13:59:04.808 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.808 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.808 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:59:04.808 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889360+0000", 2024-09-21T13:59:04.808 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000101382 2024-09-21T13:59:04.808 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.808 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.808 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:59:04.808 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889391+0000", 2024-09-21T13:59:04.808 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1032999999999999e-05 2024-09-21T13:59:04.808 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.808 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.808 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:59:04.808 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889500+0000", 2024-09-21T13:59:04.808 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000109194 2024-09-21T13:59:04.809 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.809 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.809 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:59:04.809 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889512+0000", 2024-09-21T13:59:04.809 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2368000000000001e-05 2024-09-21T13:59:04.809 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.809 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.809 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:59:04.809 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674335+0000", 2024-09-21T13:59:04.809 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.78482304000000003 2024-09-21T13:59:04.809 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.809 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.809 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:59:04.810 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674345+0000", 2024-09-21T13:59:04.810 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0880000000000001e-06 2024-09-21T13:59:04.810 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:04.810 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:04.810 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:04.810 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.810 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.810 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232060 8.b 8.6c14963b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T13:59:04.810 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.889442+0000", 2024-09-21T13:59:04.810 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 111.24658746199999, 2024-09-21T13:59:04.810 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.78518956100000004, 2024-09-21T13:59:04.810 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:59:04.810 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:59:04.810 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T13:59:04.811 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:59:04.811 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:59:04.811 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:59:04.811 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232060 2024-09-21T13:59:04.811 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.811 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:59:04.811 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.811 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:59:04.811 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889442+0000", 2024-09-21T13:59:04.811 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:04.811 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.811 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.811 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:59:04.811 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889442+0000", 2024-09-21T13:59:04.812 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:04.812 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.812 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.812 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:59:04.812 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889443+0000", 2024-09-21T13:59:04.812 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9100000000000003e-07 2024-09-21T13:59:04.812 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.812 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.812 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:59:04.812 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889447+0000", 2024-09-21T13:59:04.812 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4939999999999997e-06 2024-09-21T13:59:04.812 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.812 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.813 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:59:04.813 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889457+0000", 2024-09-21T13:59:04.813 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0145e-05 2024-09-21T13:59:04.813 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.813 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.813 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:59:04.813 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889465+0000", 2024-09-21T13:59:04.813 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2749999999999998e-06 2024-09-21T13:59:04.813 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.813 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.813 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:59:04.813 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889550+0000", 2024-09-21T13:59:04.813 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5000999999999995e-05 2024-09-21T13:59:04.813 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.814 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.814 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:59:04.814 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889561+0000", 2024-09-21T13:59:04.814 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.129e-05 2024-09-21T13:59:04.814 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.814 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.814 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:59:04.814 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674619+0000", 2024-09-21T13:59:04.814 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.785057703 2024-09-21T13:59:04.814 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.814 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.814 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:59:04.814 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674632+0000", 2024-09-21T13:59:04.815 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2962000000000001e-05 2024-09-21T13:59:04.815 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:04.815 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:04.815 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:04.815 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.815 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.815 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232068 8.f 8.d5b3687f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T13:59:04.815 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.894747+0000", 2024-09-21T13:59:04.815 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 111.241282774, 2024-09-21T13:59:04.815 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.77963143999999995, 2024-09-21T13:59:04.815 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:59:04.815 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:59:04.815 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T13:59:04.815 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:59:04.815 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:59:04.816 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:59:04.816 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232068 2024-09-21T13:59:04.816 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.816 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:59:04.816 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.816 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:59:04.816 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.894747+0000", 2024-09-21T13:59:04.816 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:04.816 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.816 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.816 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:59:04.816 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.894747+0000", 2024-09-21T13:59:04.816 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:04.816 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.816 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.817 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:59:04.817 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.894749+0000", 2024-09-21T13:59:04.817 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4849999999999999e-06 2024-09-21T13:59:04.817 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.817 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.817 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:59:04.817 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.894761+0000", 2024-09-21T13:59:04.817 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1809e-05 2024-09-21T13:59:04.817 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.817 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.817 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:59:04.817 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.894780+0000", 2024-09-21T13:59:04.817 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8593999999999998e-05 2024-09-21T13:59:04.817 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.817 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.818 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:59:04.818 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.894789+0000", 2024-09-21T13:59:04.818 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2839999999999995e-06 2024-09-21T13:59:04.818 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.818 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.818 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:59:04.818 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.894865+0000", 2024-09-21T13:59:04.818 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6150999999999996e-05 2024-09-21T13:59:04.818 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.818 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.818 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:59:04.818 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.894874+0000", 2024-09-21T13:59:04.818 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2450000000000004e-06 2024-09-21T13:59:04.819 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.819 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.819 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:59:04.819 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674368+0000", 2024-09-21T13:59:04.819 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.77949425999999999 2024-09-21T13:59:04.819 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.819 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.819 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:59:04.819 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674378+0000", 2024-09-21T13:59:04.819 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6120000000000005e-06 2024-09-21T13:59:04.819 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:04.820 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:04.820 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:04.820 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.820 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.820 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232074 8.0 8.57810 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T13:59:04.820 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.901181+0000", 2024-09-21T13:59:04.820 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 111.234848261, 2024-09-21T13:59:04.820 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.77382758200000001, 2024-09-21T13:59:04.820 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:59:04.820 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:59:04.820 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T13:59:04.820 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:59:04.820 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:59:04.820 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:59:04.821 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232074 2024-09-21T13:59:04.821 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.821 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:59:04.821 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.821 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:59:04.821 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.901181+0000", 2024-09-21T13:59:04.821 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:04.821 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.821 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.821 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:59:04.821 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.901181+0000", 2024-09-21T13:59:04.821 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:04.821 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.821 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.822 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:59:04.822 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.901184+0000", 2024-09-21T13:59:04.822 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.8660000000000002e-06 2024-09-21T13:59:04.822 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.822 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.822 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:59:04.822 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.901193+0000", 2024-09-21T13:59:04.822 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1549999999999996e-06 2024-09-21T13:59:04.822 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.822 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.822 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:59:04.823 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.901215+0000", 2024-09-21T13:59:04.823 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2272e-05 2024-09-21T13:59:04.823 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.823 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.823 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:59:04.823 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.901227+0000", 2024-09-21T13:59:04.823 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1116e-05 2024-09-21T13:59:04.823 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.823 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.823 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:59:04.823 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.901351+0000", 2024-09-21T13:59:04.823 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000124172 2024-09-21T13:59:04.824 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.824 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.824 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:59:04.824 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.901368+0000", 2024-09-21T13:59:04.824 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7008999999999999e-05 2024-09-21T13:59:04.824 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.824 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.824 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:59:04.824 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674970+0000", 2024-09-21T13:59:04.824 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.77360225199999999 2024-09-21T13:59:04.824 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.824 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.824 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:59:04.824 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.675009+0000", 2024-09-21T13:59:04.825 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8739999999999998e-05 2024-09-21T13:59:04.825 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:04.825 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:04.825 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:04.825 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.825 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.825 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232078 8.b 8.7f49789b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T13:59:04.825 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.904662+0000", 2024-09-21T13:59:04.825 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 111.23136785600001, 2024-09-21T13:59:04.825 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.77001060399999999, 2024-09-21T13:59:04.825 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:59:04.825 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:59:04.826 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T13:59:04.826 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:59:04.826 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:59:04.826 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:59:04.826 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232078 2024-09-21T13:59:04.826 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.826 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:59:04.826 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.826 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:59:04.826 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904662+0000", 2024-09-21T13:59:04.826 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:04.826 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.826 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.827 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:59:04.827 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904662+0000", 2024-09-21T13:59:04.827 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:04.827 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.827 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.827 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:59:04.827 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904663+0000", 2024-09-21T13:59:04.827 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.548e-06 2024-09-21T13:59:04.827 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.827 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.827 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:59:04.827 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904669+0000", 2024-09-21T13:59:04.827 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9619999999999998e-06 2024-09-21T13:59:04.828 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.828 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.828 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:59:04.828 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904684+0000", 2024-09-21T13:59:04.828 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4718e-05 2024-09-21T13:59:04.828 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.828 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.828 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:59:04.828 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904691+0000", 2024-09-21T13:59:04.828 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.861e-06 2024-09-21T13:59:04.828 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.828 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.829 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:59:04.829 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904750+0000", 2024-09-21T13:59:04.829 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9533000000000001e-05 2024-09-21T13:59:04.829 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.829 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.829 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:59:04.829 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904757+0000", 2024-09-21T13:59:04.829 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0369999999999997e-06 2024-09-21T13:59:04.829 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.829 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.829 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:59:04.829 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674661+0000", 2024-09-21T13:59:04.830 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.769903754 2024-09-21T13:59:04.830 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.830 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.830 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:59:04.830 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674672+0000", 2024-09-21T13:59:04.830 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1191e-05 2024-09-21T13:59:04.830 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:04.830 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:04.830 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:04.830 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.830 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.830 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232079 8.b 8.5c3e90b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T13:59:04.831 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.904710+0000", 2024-09-21T13:59:04.831 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 111.231319501, 2024-09-21T13:59:04.831 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.76999876, 2024-09-21T13:59:04.831 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:59:04.831 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:59:04.831 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T13:59:04.831 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:59:04.831 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:59:04.831 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:59:04.832 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232079 2024-09-21T13:59:04.832 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.832 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:59:04.832 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.832 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:59:04.832 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904710+0000", 2024-09-21T13:59:04.832 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:04.832 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.832 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.832 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:59:04.833 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904710+0000", 2024-09-21T13:59:04.833 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:04.833 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.833 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.833 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:59:04.833 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904711+0000", 2024-09-21T13:59:04.833 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8199999999999999e-07 2024-09-21T13:59:04.833 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.833 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.833 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:59:04.833 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904714+0000", 2024-09-21T13:59:04.833 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9079999999999999e-06 2024-09-21T13:59:04.833 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.834 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.834 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:59:04.834 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904726+0000", 2024-09-21T13:59:04.834 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2514000000000001e-05 2024-09-21T13:59:04.834 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.834 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.834 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:59:04.834 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904733+0000", 2024-09-21T13:59:04.834 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2690000000000002e-06 2024-09-21T13:59:04.834 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.834 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.834 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:59:04.834 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904784+0000", 2024-09-21T13:59:04.835 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1007000000000001e-05 2024-09-21T13:59:04.835 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.835 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.835 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T13:59:04.835 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904794+0000", 2024-09-21T13:59:04.835 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7419999999999992e-06 2024-09-21T13:59:04.835 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.835 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.835 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:59:04.835 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674699+0000", 2024-09-21T13:59:04.835 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.76990482900000001 2024-09-21T13:59:04.835 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.835 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.835 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:59:04.836 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674709+0000", 2024-09-21T13:59:04.836 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8090000000000004e-06 2024-09-21T13:59:04.836 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:04.836 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:04.836 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:04.836 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.836 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.836 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25421.0:71386 9.3 9:c0cb19f5:::benchmark_data_smithi049_138504_object71385:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-09-21T13:59:04.836 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:58:50.900796+0000", 2024-09-21T13:59:04.836 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 12.235233163, 2024-09-21T13:59:04.836 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.117207481, 2024-09-21T13:59:04.836 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:59:04.836 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:59:04.837 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:59:04.837 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:59:04.837 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25421", 2024-09-21T13:59:04.837 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T13:59:04.837 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 71386 2024-09-21T13:59:04.837 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.837 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:59:04.837 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.837 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:59:04.837 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:50.900796+0000", 2024-09-21T13:59:04.837 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:04.838 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.838 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.838 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:59:04.838 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:50.900796+0000", 2024-09-21T13:59:04.838 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:04.838 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.838 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.838 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:59:04.838 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:50.900798+0000", 2024-09-21T13:59:04.838 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.827e-06 2024-09-21T13:59:04.838 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.838 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.838 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:59:04.838 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:50.900825+0000", 2024-09-21T13:59:04.839 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6641000000000001e-05 2024-09-21T13:59:04.839 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.839 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.839 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:59:04.839 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:50.900838+0000", 2024-09-21T13:59:04.839 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2761e-05 2024-09-21T13:59:04.839 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.839 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.839 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:59:04.839 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:50.900845+0000", 2024-09-21T13:59:04.839 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5950000000000001e-06 2024-09-21T13:59:04.839 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.839 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.840 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:59:04.840 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:50.901185+0000", 2024-09-21T13:59:04.840 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00033987399999999999 2024-09-21T13:59:04.840 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.840 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.840 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:59:04.840 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:50.901316+0000", 2024-09-21T13:59:04.840 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013079199999999999 2024-09-21T13:59:04.840 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.840 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.840 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1", 2024-09-21T13:59:04.840 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:50.901454+0000", 2024-09-21T13:59:04.841 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013862999999999999 2024-09-21T13:59:04.841 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.841 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.841 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:59:04.841 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:50.908832+0000", 2024-09-21T13:59:04.841 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.007377565 2024-09-21T13:59:04.841 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.841 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.841 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:04.841 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:50.911234+0000", 2024-09-21T13:59:04.841 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0024021030000000001 2024-09-21T13:59:04.841 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.841 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.841 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:59:04.841 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:51.017916+0000", 2024-09-21T13:59:04.842 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.10668194 2024-09-21T13:59:04.842 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.842 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.842 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:59:04.842 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:51.018004+0000", 2024-09-21T13:59:04.842 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.7752999999999998e-05 2024-09-21T13:59:04.842 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:04.842 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:04.842 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:04.842 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.842 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.842 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25421.0:71404 9.8 9:1687319d:::benchmark_data_smithi049_138504_object71403:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-09-21T13:59:04.842 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:58:50.910652+0000", 2024-09-21T13:59:04.842 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 12.225377311000001, 2024-09-21T13:59:04.843 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.11538240900000001, 2024-09-21T13:59:04.843 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:59:04.843 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:59:04.843 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:59:04.843 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:59:04.843 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25421", 2024-09-21T13:59:04.843 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T13:59:04.843 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 71404 2024-09-21T13:59:04.843 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.843 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:59:04.843 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.843 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:59:04.843 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:50.910652+0000", 2024-09-21T13:59:04.843 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:04.844 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.844 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.844 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:59:04.844 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:50.910652+0000", 2024-09-21T13:59:04.844 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:04.844 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.844 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.844 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:59:04.844 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:50.910654+0000", 2024-09-21T13:59:04.844 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4139999999999999e-06 2024-09-21T13:59:04.844 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.845 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.845 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:59:04.845 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:50.910677+0000", 2024-09-21T13:59:04.845 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3710000000000002e-05 2024-09-21T13:59:04.845 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.845 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.845 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:59:04.845 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:50.910688+0000", 2024-09-21T13:59:04.845 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1219999999999999e-05 2024-09-21T13:59:04.845 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.845 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.845 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:59:04.845 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:50.910696+0000", 2024-09-21T13:59:04.845 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0790000000000003e-06 2024-09-21T13:59:04.846 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.846 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.846 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:59:04.846 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:50.910784+0000", 2024-09-21T13:59:04.846 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.8183999999999995e-05 2024-09-21T13:59:04.846 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.846 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.846 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:59:04.846 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:50.911109+0000", 2024-09-21T13:59:04.846 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00032485599999999999 2024-09-21T13:59:04.846 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.846 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.846 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,5", 2024-09-21T13:59:04.846 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:50.911313+0000", 2024-09-21T13:59:04.847 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00020416999999999999 2024-09-21T13:59:04.847 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.847 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.847 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:04.847 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:51.024895+0000", 2024-09-21T13:59:04.847 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.11358264999999999 2024-09-21T13:59:04.847 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.847 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.847 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:04.847 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:51.024960+0000", 2024-09-21T13:59:04.847 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5074999999999996e-05 2024-09-21T13:59:04.847 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.847 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.847 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:59:04.847 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:51.025687+0000", 2024-09-21T13:59:04.848 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00072619299999999998 2024-09-21T13:59:04.848 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.848 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.848 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:59:04.848 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:51.025984+0000", 2024-09-21T13:59:04.848 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00029699300000000001 2024-09-21T13:59:04.848 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.848 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.848 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:59:04.848 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:51.026035+0000", 2024-09-21T13:59:04.848 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0865000000000002e-05 2024-09-21T13:59:04.848 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:04.848 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:04.848 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:04.848 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.849 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.849 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25421.0:71403 9.f e286/260 9:f191666e:::benchmark_data_smithi049_138504_object71402:head v 286'4479, mlcod=286'4477)", 2024-09-21T13:59:04.849 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:58:50.910961+0000", 2024-09-21T13:59:04.849 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 12.225068049000001, 2024-09-21T13:59:04.849 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.11468961699999999, 2024-09-21T13:59:04.849 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:59:04.849 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:59:04.849 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:59:04.849 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:59:04.849 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.849 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:59:04.849 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:50.910961+0000", 2024-09-21T13:59:04.849 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:04.849 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.850 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.850 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:59:04.850 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:50.910961+0000", 2024-09-21T13:59:04.850 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:04.850 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.850 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.850 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:59:04.850 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:50.910962+0000", 2024-09-21T13:59:04.850 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5700000000000001e-07 2024-09-21T13:59:04.850 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.850 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.851 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:59:04.851 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:50.911009+0000", 2024-09-21T13:59:04.851 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6378000000000002e-05 2024-09-21T13:59:04.851 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.851 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.851 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:59:04.851 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:50.911020+0000", 2024-09-21T13:59:04.851 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1018e-05 2024-09-21T13:59:04.851 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.851 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.851 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:59:04.851 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:50.911026+0000", 2024-09-21T13:59:04.851 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2650000000000002e-06 2024-09-21T13:59:04.851 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.852 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.852 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:59:04.852 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:51.017635+0000", 2024-09-21T13:59:04.852 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.10660958299999999 2024-09-21T13:59:04.852 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.852 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.852 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:59:04.852 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:51.017655+0000", 2024-09-21T13:59:04.852 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9284e-05 2024-09-21T13:59:04.852 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.852 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.852 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:59:04.852 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:51.025586+0000", 2024-09-21T13:59:04.852 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0079313639999999998 2024-09-21T13:59:04.852 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.853 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.853 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:59:04.853 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:51.025651+0000", 2024-09-21T13:59:04.853 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4868000000000003e-05 2024-09-21T13:59:04.853 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:04.853 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:04.853 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:04.853 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.853 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.853 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25421.0:71405 9.8 9:1e7efdc6:::benchmark_data_smithi049_138504_object71404:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-09-21T13:59:04.853 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:58:50.911215+0000", 2024-09-21T13:59:04.853 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 12.224814393999999, 2024-09-21T13:59:04.853 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.117113283, 2024-09-21T13:59:04.853 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:59:04.854 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:59:04.854 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:59:04.854 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:59:04.854 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25421", 2024-09-21T13:59:04.854 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T13:59:04.854 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 71405 2024-09-21T13:59:04.854 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.854 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:59:04.854 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.854 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:59:04.855 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:50.911215+0000", 2024-09-21T13:59:04.855 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:04.855 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.855 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.855 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:59:04.855 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:50.911215+0000", 2024-09-21T13:59:04.855 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:04.855 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.855 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.855 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:59:04.855 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:50.911216+0000", 2024-09-21T13:59:04.855 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4300000000000002e-07 2024-09-21T13:59:04.855 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.855 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.855 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:59:04.856 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:50.911238+0000", 2024-09-21T13:59:04.856 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2271000000000001e-05 2024-09-21T13:59:04.856 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.856 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.856 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:59:04.856 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:50.911248+0000", 2024-09-21T13:59:04.856 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9830000000000008e-06 2024-09-21T13:59:04.856 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.856 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.856 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:59:04.856 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:50.911255+0000", 2024-09-21T13:59:04.856 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7719999999999997e-06 2024-09-21T13:59:04.856 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.857 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.857 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:59:04.857 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:51.019850+0000", 2024-09-21T13:59:04.857 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.10859517 2024-09-21T13:59:04.857 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.857 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.857 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:59:04.857 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:51.019972+0000", 2024-09-21T13:59:04.857 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012188000000000001 2024-09-21T13:59:04.857 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.858 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.858 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,5", 2024-09-21T13:59:04.858 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:51.020121+0000", 2024-09-21T13:59:04.858 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000149123 2024-09-21T13:59:04.858 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.858 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.858 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:04.858 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:51.025005+0000", 2024-09-21T13:59:04.858 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0048840660000000003 2024-09-21T13:59:04.858 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.858 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.859 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:59:04.859 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:51.026044+0000", 2024-09-21T13:59:04.859 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0010387160000000001 2024-09-21T13:59:04.859 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.859 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.859 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:04.859 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:51.028225+0000", 2024-09-21T13:59:04.859 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0021810060000000001 2024-09-21T13:59:04.859 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.859 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.859 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:59:04.859 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:51.028285+0000", 2024-09-21T13:59:04.859 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0136999999999997e-05 2024-09-21T13:59:04.859 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.860 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.860 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:59:04.860 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:51.028328+0000", 2024-09-21T13:59:04.860 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3316000000000003e-05 2024-09-21T13:59:04.860 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:04.860 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:04.860 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:04.860 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.860 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.860 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25421.0:71409 9.2 9:4b5be0cc:::benchmark_data_smithi049_138504_object71408:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-09-21T13:59:04.860 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:58:50.912324+0000", 2024-09-21T13:59:04.860 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 12.223705066999999, 2024-09-21T13:59:04.860 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.122262261, 2024-09-21T13:59:04.860 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:59:04.860 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:59:04.861 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:59:04.861 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:59:04.861 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25421", 2024-09-21T13:59:04.861 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T13:59:04.861 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 71409 2024-09-21T13:59:04.861 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.861 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:59:04.861 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.861 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:59:04.861 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:50.912324+0000", 2024-09-21T13:59:04.861 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:04.861 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.861 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.861 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:59:04.862 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:50.912324+0000", 2024-09-21T13:59:04.862 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:04.862 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.862 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.862 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:59:04.862 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:50.912326+0000", 2024-09-21T13:59:04.862 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5230000000000001e-06 2024-09-21T13:59:04.862 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.862 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.862 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:59:04.862 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:50.912357+0000", 2024-09-21T13:59:04.863 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1513000000000001e-05 2024-09-21T13:59:04.863 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.863 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.863 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:59:04.863 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:51.017291+0000", 2024-09-21T13:59:04.863 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.104933812 2024-09-21T13:59:04.863 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.863 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.863 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:59:04.863 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:51.017310+0000", 2024-09-21T13:59:04.863 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8621e-05 2024-09-21T13:59:04.863 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.863 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.863 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:59:04.864 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:51.019128+0000", 2024-09-21T13:59:04.864 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0018186210000000001 2024-09-21T13:59:04.864 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.864 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.864 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:59:04.864 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:51.019297+0000", 2024-09-21T13:59:04.864 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000168398 2024-09-21T13:59:04.864 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.864 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.864 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,2", 2024-09-21T13:59:04.864 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:51.019470+0000", 2024-09-21T13:59:04.864 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000173225 2024-09-21T13:59:04.864 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.864 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.865 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:04.865 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:51.027914+0000", 2024-09-21T13:59:04.865 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0084441580000000002 2024-09-21T13:59:04.865 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.865 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.865 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:04.865 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:51.034485+0000", 2024-09-21T13:59:04.865 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0065703410000000004 2024-09-21T13:59:04.865 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.865 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.865 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:59:04.865 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:51.034511+0000", 2024-09-21T13:59:04.865 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6795999999999999e-05 2024-09-21T13:59:04.865 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.866 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.866 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:59:04.866 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:51.034552+0000", 2024-09-21T13:59:04.866 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0692999999999998e-05 2024-09-21T13:59:04.866 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.866 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.866 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:59:04.866 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:51.034587+0000", 2024-09-21T13:59:04.866 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4560000000000001e-05 2024-09-21T13:59:04.866 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:04.866 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:04.866 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:04.866 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.867 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.867 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25421.0:73966 9.2 9:428f1206:::benchmark_data_smithi049_138504_object73965:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-09-21T13:59:04.867 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:58:52.923742+0000", 2024-09-21T13:59:04.867 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 10.212287665, 2024-09-21T13:59:04.867 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.117897382, 2024-09-21T13:59:04.867 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:59:04.867 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:59:04.867 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:59:04.867 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:59:04.867 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25421", 2024-09-21T13:59:04.867 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T13:59:04.867 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 73966 2024-09-21T13:59:04.867 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.867 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:59:04.867 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.868 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:59:04.868 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:52.923742+0000", 2024-09-21T13:59:04.868 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:04.868 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.868 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.868 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:59:04.868 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:52.923742+0000", 2024-09-21T13:59:04.868 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:04.868 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.868 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.868 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:59:04.868 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:52.923743+0000", 2024-09-21T13:59:04.868 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.358e-06 2024-09-21T13:59:04.869 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.869 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.869 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:59:04.869 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:52.923764+0000", 2024-09-21T13:59:04.869 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0707e-05 2024-09-21T13:59:04.869 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.869 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.869 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:59:04.869 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:52.923776+0000", 2024-09-21T13:59:04.869 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2632999999999999e-05 2024-09-21T13:59:04.869 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.870 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.870 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:59:04.870 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:52.923784+0000", 2024-09-21T13:59:04.870 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3730000000000004e-06 2024-09-21T13:59:04.870 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.870 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.870 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:59:04.870 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:52.923883+0000", 2024-09-21T13:59:04.870 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8795999999999994e-05 2024-09-21T13:59:04.870 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.870 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.870 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:59:04.870 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:52.923988+0000", 2024-09-21T13:59:04.870 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000105214 2024-09-21T13:59:04.871 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.871 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.871 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,2", 2024-09-21T13:59:04.871 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:52.924109+0000", 2024-09-21T13:59:04.871 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012086 2024-09-21T13:59:04.871 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.871 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.871 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:04.871 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:53.037798+0000", 2024-09-21T13:59:04.871 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.113689627 2024-09-21T13:59:04.871 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.871 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.871 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:59:04.871 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:53.038223+0000", 2024-09-21T13:59:04.871 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000424984 2024-09-21T13:59:04.871 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.872 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.872 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:04.872 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:53.041549+0000", 2024-09-21T13:59:04.872 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0033254869999999998 2024-09-21T13:59:04.872 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.872 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.874 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:59:04.874 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:53.041602+0000", 2024-09-21T13:59:04.874 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2882e-05 2024-09-21T13:59:04.874 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.874 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.875 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:59:04.875 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:53.041639+0000", 2024-09-21T13:59:04.875 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7460999999999999e-05 2024-09-21T13:59:04.875 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:04.875 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:04.875 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:04.875 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.875 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.875 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25421.0:73968 9.8 9:1c27e1b8:::benchmark_data_smithi049_138504_object73967:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-09-21T13:59:04.875 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:58:52.924527+0000", 2024-09-21T13:59:04.875 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 10.211502698, 2024-09-21T13:59:04.875 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.11625671999999999, 2024-09-21T13:59:04.876 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:59:04.876 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:59:04.876 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:59:04.876 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:59:04.876 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25421", 2024-09-21T13:59:04.876 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T13:59:04.876 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 73968 2024-09-21T13:59:04.876 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.876 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:59:04.876 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.876 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:59:04.876 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:52.924527+0000", 2024-09-21T13:59:04.876 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:04.876 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.877 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.877 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:59:04.877 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:52.924527+0000", 2024-09-21T13:59:04.877 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:04.877 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.877 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.877 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:59:04.877 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:52.924528+0000", 2024-09-21T13:59:04.877 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6099999999999999e-07 2024-09-21T13:59:04.877 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.877 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.877 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:59:04.877 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:52.924549+0000", 2024-09-21T13:59:04.877 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.126e-05 2024-09-21T13:59:04.878 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.878 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.878 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:59:04.878 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:52.924559+0000", 2024-09-21T13:59:04.878 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.023e-05 2024-09-21T13:59:04.878 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.878 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.878 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:59:04.878 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:52.924566+0000", 2024-09-21T13:59:04.878 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0770000000000002e-06 2024-09-21T13:59:04.878 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.878 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.878 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:59:04.879 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:52.924852+0000", 2024-09-21T13:59:04.879 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00028598300000000002 2024-09-21T13:59:04.879 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.879 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.879 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:59:04.879 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:52.924968+0000", 2024-09-21T13:59:04.879 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000116038 2024-09-21T13:59:04.879 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.879 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.879 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,5", 2024-09-21T13:59:04.879 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:52.925101+0000", 2024-09-21T13:59:04.879 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013222 2024-09-21T13:59:04.879 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.879 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.880 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:59:04.880 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:52.929959+0000", 2024-09-21T13:59:04.880 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0048585210000000002 2024-09-21T13:59:04.880 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.880 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.880 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:04.880 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:52.930353+0000", 2024-09-21T13:59:04.880 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00039374000000000001 2024-09-21T13:59:04.880 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.880 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.880 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:04.880 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:53.040696+0000", 2024-09-21T13:59:04.880 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.110343284 2024-09-21T13:59:04.880 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.881 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.881 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:59:04.881 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:53.040741+0000", 2024-09-21T13:59:04.881 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5426000000000002e-05 2024-09-21T13:59:04.881 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.881 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.881 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:59:04.881 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:53.040783+0000", 2024-09-21T13:59:04.881 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1980000000000001e-05 2024-09-21T13:59:04.881 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:04.881 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:04.881 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:04.881 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.881 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.882 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25421.0:73971 9.2 9:48769bc6:::benchmark_data_smithi049_138504_object73970:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-09-21T13:59:04.882 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:58:52.925821+0000", 2024-09-21T13:59:04.882 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 10.210208498, 2024-09-21T13:59:04.882 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.116008493, 2024-09-21T13:59:04.882 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:59:04.882 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:59:04.882 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:59:04.882 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:59:04.882 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25421", 2024-09-21T13:59:04.882 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T13:59:04.882 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 73971 2024-09-21T13:59:04.882 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.882 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:59:04.882 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.883 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:59:04.883 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:52.925821+0000", 2024-09-21T13:59:04.883 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:04.883 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.883 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.883 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:59:04.883 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:52.925821+0000", 2024-09-21T13:59:04.883 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:04.883 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.883 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.883 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:59:04.883 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:52.925822+0000", 2024-09-21T13:59:04.883 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1000000000000001e-06 2024-09-21T13:59:04.883 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.884 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.884 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:59:04.884 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:52.925842+0000", 2024-09-21T13:59:04.884 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0401999999999999e-05 2024-09-21T13:59:04.884 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.884 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.884 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:59:04.884 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:52.925855+0000", 2024-09-21T13:59:04.884 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2116000000000001e-05 2024-09-21T13:59:04.884 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.884 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.884 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:59:04.884 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:52.925862+0000", 2024-09-21T13:59:04.884 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1219999999999997e-06 2024-09-21T13:59:04.885 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.885 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.885 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:59:04.885 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:52.927212+0000", 2024-09-21T13:59:04.885 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0013500770000000001 2024-09-21T13:59:04.885 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.885 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.885 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:59:04.885 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:52.927330+0000", 2024-09-21T13:59:04.885 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011797499999999999 2024-09-21T13:59:04.885 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.885 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.885 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,2", 2024-09-21T13:59:04.885 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:52.927461+0000", 2024-09-21T13:59:04.886 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013125300000000001 2024-09-21T13:59:04.886 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.886 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.886 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:04.886 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:53.037837+0000", 2024-09-21T13:59:04.886 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.110376475 2024-09-21T13:59:04.886 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.886 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.886 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:59:04.886 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:53.038231+0000", 2024-09-21T13:59:04.886 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00039372000000000002 2024-09-21T13:59:04.886 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.886 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.886 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:04.886 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:53.041745+0000", 2024-09-21T13:59:04.887 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0035142469999999999 2024-09-21T13:59:04.887 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.887 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.887 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:59:04.887 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:53.041795+0000", 2024-09-21T13:59:04.887 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9997000000000003e-05 2024-09-21T13:59:04.887 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.887 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.887 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:59:04.887 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:53.041829+0000", 2024-09-21T13:59:04.887 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4008999999999999e-05 2024-09-21T13:59:04.887 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:04.887 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:04.887 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:04.888 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.888 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.888 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25421.0:73973 9.8 9:19fd4ef3:::benchmark_data_smithi049_138504_object73972:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-09-21T13:59:04.888 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:58:52.926858+0000", 2024-09-21T13:59:04.888 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 10.209171308, 2024-09-21T13:59:04.888 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.11404669200000001, 2024-09-21T13:59:04.888 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:59:04.888 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:59:04.888 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:59:04.888 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:59:04.888 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25421", 2024-09-21T13:59:04.888 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T13:59:04.888 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 73973 2024-09-21T13:59:04.889 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.889 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:59:04.889 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.889 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:59:04.889 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:52.926858+0000", 2024-09-21T13:59:04.889 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:04.889 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.889 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.889 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:59:04.889 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:52.926858+0000", 2024-09-21T13:59:04.889 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:04.889 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.889 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.890 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:59:04.890 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:52.926859+0000", 2024-09-21T13:59:04.890 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2600000000000001e-07 2024-09-21T13:59:04.890 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.890 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.890 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:59:04.890 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:52.926880+0000", 2024-09-21T13:59:04.890 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0599e-05 2024-09-21T13:59:04.890 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.890 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.890 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:59:04.890 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:52.926892+0000", 2024-09-21T13:59:04.890 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2065999999999999e-05 2024-09-21T13:59:04.891 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.891 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.891 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:59:04.891 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:52.926899+0000", 2024-09-21T13:59:04.891 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9079999999999998e-06 2024-09-21T13:59:04.891 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.891 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.891 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:59:04.891 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:52.927423+0000", 2024-09-21T13:59:04.891 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00052477899999999996 2024-09-21T13:59:04.891 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.891 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.891 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:59:04.891 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:52.927538+0000", 2024-09-21T13:59:04.892 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011503800000000001 2024-09-21T13:59:04.892 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.892 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.892 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,5", 2024-09-21T13:59:04.892 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:52.927668+0000", 2024-09-21T13:59:04.892 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000129935 2024-09-21T13:59:04.892 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.892 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.892 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:59:04.892 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:53.037299+0000", 2024-09-21T13:59:04.892 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.10963038899999999 2024-09-21T13:59:04.892 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.892 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.892 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:04.893 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:53.037897+0000", 2024-09-21T13:59:04.893 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00059838999999999997 2024-09-21T13:59:04.893 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.893 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.893 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:04.893 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:53.040823+0000", 2024-09-21T13:59:04.893 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0029257189999999998 2024-09-21T13:59:04.893 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.893 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.893 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:59:04.893 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:53.040866+0000", 2024-09-21T13:59:04.893 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2998999999999999e-05 2024-09-21T13:59:04.893 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.893 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.893 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:59:04.894 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:53.040905+0000", 2024-09-21T13:59:04.894 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8943999999999998e-05 2024-09-21T13:59:04.894 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:04.894 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:04.894 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:04.894 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.894 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.894 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25421.0:73980 9.1 9:863c6c65:::benchmark_data_smithi049_138504_object73979:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-09-21T13:59:04.894 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:58:52.930132+0000", 2024-09-21T13:59:04.894 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 10.205897215, 2024-09-21T13:59:04.894 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.116554138, 2024-09-21T13:59:04.894 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:59:04.894 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:59:04.894 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:59:04.895 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:59:04.895 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25421", 2024-09-21T13:59:04.895 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T13:59:04.895 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 73980 2024-09-21T13:59:04.895 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.895 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:59:04.895 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.895 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:59:04.895 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:52.930132+0000", 2024-09-21T13:59:04.895 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:04.895 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.895 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.895 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:59:04.895 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:52.930132+0000", 2024-09-21T13:59:04.896 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:04.896 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.896 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.896 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:59:04.896 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:52.930133+0000", 2024-09-21T13:59:04.896 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4499999999999996e-07 2024-09-21T13:59:04.896 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.896 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.896 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:59:04.896 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:52.930161+0000", 2024-09-21T13:59:04.896 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.773e-05 2024-09-21T13:59:04.896 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.896 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.897 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:59:04.897 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:52.930172+0000", 2024-09-21T13:59:04.897 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0783e-05 2024-09-21T13:59:04.897 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.897 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.897 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:59:04.897 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:52.930178+0000", 2024-09-21T13:59:04.897 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6869999999999997e-06 2024-09-21T13:59:04.897 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.897 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.897 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:59:04.897 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:52.930232+0000", 2024-09-21T13:59:04.897 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3789999999999998e-05 2024-09-21T13:59:04.898 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.898 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.898 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:59:04.898 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:52.930342+0000", 2024-09-21T13:59:04.898 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000109442 2024-09-21T13:59:04.898 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.898 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.898 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,6", 2024-09-21T13:59:04.898 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:52.930472+0000", 2024-09-21T13:59:04.898 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000130739 2024-09-21T13:59:04.898 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.898 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.898 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:04.898 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:53.043182+0000", 2024-09-21T13:59:04.899 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.112710195 2024-09-21T13:59:04.899 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.899 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.899 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:04.899 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:53.043222+0000", 2024-09-21T13:59:04.899 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9830000000000003e-05 2024-09-21T13:59:04.899 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.899 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.899 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:59:04.899 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:53.046448+0000", 2024-09-21T13:59:04.899 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00322587 2024-09-21T13:59:04.899 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.899 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.899 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:59:04.899 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:53.046579+0000", 2024-09-21T13:59:04.900 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013044900000000001 2024-09-21T13:59:04.900 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:04.900 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:04.900 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:59:04.900 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:58:53.046686+0000", 2024-09-21T13:59:04.900 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000107778 2024-09-21T13:59:04.900 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:04.900 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:04.900 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:04.900 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:04.900 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:04.900 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T13:59:04.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:04.902 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:04.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:59:05.003 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:05.027 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:05.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:05.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:05.129 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:05.173 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:05.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:05.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:05.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:04 smithi049 ceph-mon[40489]: pgmap v507: 148 pgs: 1 active+clean+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 75 MiB/s wr, 1.20k op/s; 43203/257178 objects degraded (16.799%); 5201/257178 objects misplaced (2.022%) 2024-09-21T13:59:05.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:04 smithi049 ceph-mon[30080]: pgmap v507: 148 pgs: 1 active+clean+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 75 MiB/s wr, 1.20k op/s; 43203/257178 objects degraded (16.799%); 5201/257178 objects misplaced (2.022%) 2024-09-21T13:59:05.276 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:05.315 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:05.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:05.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:05.417 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:59:05.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:04 smithi187 ceph-mon[33032]: pgmap v507: 148 pgs: 1 active+clean+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 75 MiB/s wr, 1.20k op/s; 43203/257178 objects degraded (16.799%); 5201/257178 objects misplaced (2.022%) 2024-09-21T13:59:05.460 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:05.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:05.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:59:05.562 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:59:05.586 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:05.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:05.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:59:05.619 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T13:59:05.679 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 61 16 90258 90242 92.441 80.75 0.00768992 0.0108119 2024-09-21T13:59:05.688 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:05.703 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:59:05.713 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:59:05.715 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:05.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:05.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:05.817 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:59:05.867 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:05.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:05.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:59:05.969 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:59:05.994 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:05.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:05.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:59:06.096 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:59:06.137 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:06.138 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:05 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 43203/257178 objects degraded (16.799%), 28 pgs degraded (PG_DEGRADED) 2024-09-21T13:59:06.138 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:05 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 43203/257178 objects degraded (16.799%), 28 pgs degraded (PG_DEGRADED) 2024-09-21T13:59:06.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:06.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:59:06.240 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:06.279 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:06.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:06.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:06.381 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:06.429 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:06.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:06.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:06.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:05 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 43203/257178 objects degraded (16.799%), 28 pgs degraded (PG_DEGRADED) 2024-09-21T13:59:06.531 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:06.556 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:06.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:06.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:06.658 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:59:06.670 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T13:59:06.680 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 62 16 91478 91462 92.1782 76.25 0.0269838 0.0108437 2024-09-21T13:59:06.685 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:06.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:06.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:59:06.787 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:06.854 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:06.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:06.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:06.956 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:07.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:07.012 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:07.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:07.097 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:59:07.114 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:07.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:07.180 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:07.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:07.281 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:07.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:07.325 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:07.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:07.426 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:07.438 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:07 smithi049 ceph-mon[40489]: pgmap v508: 148 pgs: 1 active+clean+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 5.4 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 7.2 KiB/s rd, 100 MiB/s wr, 1.61k op/s; 44395/266739 objects degraded (16.644%); 5588/266739 objects misplaced (2.095%) 2024-09-21T13:59:07.438 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:07 smithi049 ceph-mon[30080]: pgmap v508: 148 pgs: 1 active+clean+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 5.4 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 7.2 KiB/s rd, 100 MiB/s wr, 1.61k op/s; 44395/266739 objects degraded (16.644%); 5588/266739 objects misplaced (2.095%) 2024-09-21T13:59:07.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:07 smithi187 ceph-mon[33032]: pgmap v508: 148 pgs: 1 active+clean+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 5.4 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 7.2 KiB/s rd, 100 MiB/s wr, 1.61k op/s; 44395/266739 objects degraded (16.644%); 5588/266739 objects misplaced (2.095%) 2024-09-21T13:59:07.450 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:07.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:07.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:07.552 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:07.593 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:07.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:07.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:07.622 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:59:07.679 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 63 16 92552 92536 91.7801 67.125 0.0104639 0.0108911 2024-09-21T13:59:07.695 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:59:07.726 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:07.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:07.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:59:07.827 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:59:07.857 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:07.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:07.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:59:07.959 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:07.965 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T13:59:07.965 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-09-21T13:59:07.965 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:07.965 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25421.0:92937 9.9 e290/273 9:93bf080b:::benchmark_data_smithi049_138504_object92936:head v 290'5842, mlcod=290'5840)", 2024-09-21T13:59:07.965 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:59:07.957979+0000", 2024-09-21T13:59:07.965 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.0048090709999999998, 2024-09-21T13:59:07.965 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0048384329999999996, 2024-09-21T13:59:07.965 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:59:07.965 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:59:07.965 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:59:07.965 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:59:07.965 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:07.965 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:59:07.966 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:07.957979+0000", 2024-09-21T13:59:07.966 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:07.966 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:07.966 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:07.966 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:59:07.966 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:07.957979+0000", 2024-09-21T13:59:07.966 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:07.966 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:07.966 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:07.966 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:59:07.966 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:07.957980+0000", 2024-09-21T13:59:07.966 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5000000000000002e-07 2024-09-21T13:59:07.966 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:07.967 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:07.967 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:59:07.967 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:07.958037+0000", 2024-09-21T13:59:07.967 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7398000000000002e-05 2024-09-21T13:59:07.967 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:07.967 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:07.967 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:59:07.967 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:07.958066+0000", 2024-09-21T13:59:07.967 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9159999999999999e-05 2024-09-21T13:59:07.967 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:07.967 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:07.967 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:59:07.967 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:07.958072+0000", 2024-09-21T13:59:07.968 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4309999999999999e-06 2024-09-21T13:59:07.968 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:07.968 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:07.968 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:59:07.968 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:07.958267+0000", 2024-09-21T13:59:07.968 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000194876 2024-09-21T13:59:07.968 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:07.968 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:07.968 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:59:07.968 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:07.958343+0000", 2024-09-21T13:59:07.968 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5937000000000002e-05 2024-09-21T13:59:07.968 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:07.968 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:07.968 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:59:07.969 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:07.961842+0000", 2024-09-21T13:59:07.969 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0034990989999999999 2024-09-21T13:59:07.969 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:07.969 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:07.969 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:07.969 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:07.969 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:07.969 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25421.0:92945 9.5 e290/273 9:a18921b0:::benchmark_data_smithi049_138504_object92944:head v 290'5996, mlcod=290'5993)", 2024-09-21T13:59:07.969 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:59:07.961974+0000", 2024-09-21T13:59:07.969 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.0008139, 2024-09-21T13:59:07.969 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00090594700000000005, 2024-09-21T13:59:07.969 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:59:07.969 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:59:07.969 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T13:59:07.970 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:59:07.970 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:07.970 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:59:07.970 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:07.961974+0000", 2024-09-21T13:59:07.970 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:07.970 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:07.970 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:07.970 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:59:07.970 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:07.961974+0000", 2024-09-21T13:59:07.970 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:07.970 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:07.970 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:07.970 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:59:07.970 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:07.961975+0000", 2024-09-21T13:59:07.971 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.079e-06 2024-09-21T13:59:07.971 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:07.971 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:07.971 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:59:07.971 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:07.962061+0000", 2024-09-21T13:59:07.971 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6248000000000005e-05 2024-09-21T13:59:07.971 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:07.971 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:07.971 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:59:07.971 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:07.962071+0000", 2024-09-21T13:59:07.971 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1929999999999999e-06 2024-09-21T13:59:07.971 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:07.971 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:07.971 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:59:07.971 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:07.962077+0000", 2024-09-21T13:59:07.972 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6159999999999998e-06 2024-09-21T13:59:07.972 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:07.972 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:07.972 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:59:07.972 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:07.962402+0000", 2024-09-21T13:59:07.972 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00032496000000000001 2024-09-21T13:59:07.972 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:07.972 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:07.972 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:59:07.972 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:07.962473+0000", 2024-09-21T13:59:07.972 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0438999999999994e-05 2024-09-21T13:59:07.972 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:07.972 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:07.973 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:07.973 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:07.973 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T13:59:07.973 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 2 2024-09-21T13:59:07.973 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T13:59:08.015 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:08.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:08.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:08.033 INFO:teuthology.orchestra.run.smithi049.stderr:clear 4.1b pg_upmap_items mapping 2024-09-21T13:59:08.038 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-09-21T13:59:08.117 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:59:08.154 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:08.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:08.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:59:08.256 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:08.298 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:08.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:08.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:08.399 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:08.411 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:08 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/620740737' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:59:08.412 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:08 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:59:08.412 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:08 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3382036451' entity='client.admin' cmd={"prefix": "osd rm-pg-upmap-items", "pgid": "4.1b"} : dispatch 2024-09-21T13:59:08.412 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:08 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd rm-pg-upmap-items", "pgid": "4.1b"} : dispatch 2024-09-21T13:59:08.412 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:08 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/620740737' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:59:08.412 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:08 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:59:08.412 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:08 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3382036451' entity='client.admin' cmd={"prefix": "osd rm-pg-upmap-items", "pgid": "4.1b"} : dispatch 2024-09-21T13:59:08.412 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:08 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd rm-pg-upmap-items", "pgid": "4.1b"} : dispatch 2024-09-21T13:59:08.443 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:08.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:08 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/620740737' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:59:08.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:08 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:59:08.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:08 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3382036451' entity='client.admin' cmd={"prefix": "osd rm-pg-upmap-items", "pgid": "4.1b"} : dispatch 2024-09-21T13:59:08.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:08 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd rm-pg-upmap-items", "pgid": "4.1b"} : dispatch 2024-09-21T13:59:08.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:08.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:08.546 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:08.583 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:08.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:08.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:08.679 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 64 16 93865 93849 91.6281 82.0625 0.0123963 0.0109083 2024-09-21T13:59:08.685 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:59:08.714 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:08.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:08.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:59:08.816 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:59:08.842 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:08.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:08.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:59:08.943 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:59:08.972 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:08.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:08.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:59:09.073 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:09.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:09.114 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:09.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:09.216 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:09.227 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:59:09.227 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:59:09.259 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:09.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:09.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:09.263 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:59:09.263 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:59:09.263 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:09 smithi049 ceph-mon[40489]: pgmap v509: 148 pgs: 1 active+clean+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 5.4 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 8.5 KiB/s rd, 66 MiB/s wr, 1.08k op/s; 44395/266739 objects degraded (16.644%); 5588/266739 objects misplaced (2.095%) 2024-09-21T13:59:09.264 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:09 smithi049 ceph-mon[40489]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:59:09.264 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:09 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T13:59:09.264 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:09 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.1b"}]': finished 2024-09-21T13:59:09.264 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:09 smithi049 ceph-mon[40489]: osdmap e291: 8 total, 7 up, 8 in 2024-09-21T13:59:09.264 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:09 smithi049 ceph-mon[40489]: osdmap e292: 8 total, 7 up, 8 in 2024-09-21T13:59:09.264 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:09 smithi049 ceph-mon[30080]: pgmap v509: 148 pgs: 1 active+clean+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 5.4 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 8.5 KiB/s rd, 66 MiB/s wr, 1.08k op/s; 44395/266739 objects degraded (16.644%); 5588/266739 objects misplaced (2.095%) 2024-09-21T13:59:09.264 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:09 smithi049 ceph-mon[30080]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:59:09.264 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:09 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T13:59:09.264 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:09 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.1b"}]': finished 2024-09-21T13:59:09.264 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:09 smithi049 ceph-mon[30080]: osdmap e291: 8 total, 7 up, 8 in 2024-09-21T13:59:09.264 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:09 smithi049 ceph-mon[30080]: osdmap e292: 8 total, 7 up, 8 in 2024-09-21T13:59:09.298 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:59:09.298 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:59:09.330 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:59:09.330 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:59:09.361 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:59:09.362 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:59:09.362 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:59:09.400 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:59:09.401 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:59:09.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:09.413 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:09.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:59:09.436 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:59:09.436 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:59:09.440 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T13:59:09.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:09 smithi187 ceph-mon[33032]: pgmap v509: 148 pgs: 1 active+clean+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 5.4 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 8.5 KiB/s rd, 66 MiB/s wr, 1.08k op/s; 44395/266739 objects degraded (16.644%); 5588/266739 objects misplaced (2.095%) 2024-09-21T13:59:09.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:09 smithi187 ceph-mon[33032]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:59:09.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:09 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T13:59:09.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:09 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.1b"}]': finished 2024-09-21T13:59:09.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:09 smithi187 ceph-mon[33032]: osdmap e291: 8 total, 7 up, 8 in 2024-09-21T13:59:09.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:09 smithi187 ceph-mon[33032]: osdmap e292: 8 total, 7 up, 8 in 2024-09-21T13:59:09.515 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:59:09.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:09.540 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:09.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:59:09.641 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:09.679 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 65 16 95195 95179 91.4972 83.125 0.0082328 0.0109242 2024-09-21T13:59:09.681 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:09.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:09.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:09.783 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:09.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:09.827 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:09.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:09.928 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:09.944 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:59:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:59:09.950 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:09.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:09.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:10.052 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:59:10.080 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:10.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:10.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:59:10.182 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:10.230 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:10.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:10.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:10.332 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:10.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:10.380 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:10.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:10.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:10 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2936815766' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:59:10.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:10 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:59:10.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:10 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 44395/266739 objects degraded (16.644%), 28 pgs degraded (PG_DEGRADED) 2024-09-21T13:59:10.481 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:10.504 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:10.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:10.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:10.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:10 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2936815766' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:59:10.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:10 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:59:10.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:10 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 44395/266739 objects degraded (16.644%), 28 pgs degraded (PG_DEGRADED) 2024-09-21T13:59:10.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:10 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2936815766' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:59:10.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:10 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:59:10.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:10 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 44395/266739 objects degraded (16.644%), 28 pgs degraded (PG_DEGRADED) 2024-09-21T13:59:10.606 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:10.636 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:10.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:10.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:10.679 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 66 16 96236 96220 91.0967 65.0625 0.0154262 0.0109727 2024-09-21T13:59:10.737 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:10.777 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:10.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:10.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:10.879 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:10.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:10.915 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:10.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:11.004 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:59:10 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:59:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:59:11.016 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:11.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:11.040 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:11.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:11.141 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:11.176 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:11.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:11.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:11.278 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:59:11.306 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:11.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:11.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:59:11.408 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:11.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:11 smithi187 ceph-mon[33032]: pgmap v512: 148 pgs: 1 peering, 1 active+clean+remapped, 33 active+undersized, 28 active+undersized+degraded, 85 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 47 KiB/s rd, 82 MiB/s wr, 1.40k op/s; 46652/279453 objects degraded (16.694%); 5588/279453 objects misplaced (2.000%) 2024-09-21T13:59:11.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:11 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:59:11.451 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:11.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:11.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:11.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:11 smithi049 ceph-mon[40489]: pgmap v512: 148 pgs: 1 peering, 1 active+clean+remapped, 33 active+undersized, 28 active+undersized+degraded, 85 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 47 KiB/s rd, 82 MiB/s wr, 1.40k op/s; 46652/279453 objects degraded (16.694%); 5588/279453 objects misplaced (2.000%) 2024-09-21T13:59:11.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:11 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:59:11.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:11 smithi049 ceph-mon[30080]: pgmap v512: 148 pgs: 1 peering, 1 active+clean+remapped, 33 active+undersized, 28 active+undersized+degraded, 85 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 47 KiB/s rd, 82 MiB/s wr, 1.40k op/s; 46652/279453 objects degraded (16.694%); 5588/279453 objects misplaced (2.000%) 2024-09-21T13:59:11.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:11 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T13:59:11.553 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:11.575 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:11.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:11.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:11.677 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:11.680 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 67 16 97457 97441 90.8759 76.3125 0.00781909 0.0109993 2024-09-21T13:59:11.718 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:11.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:11.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:11.820 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:11.856 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:11.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:11.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:11.958 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:59:11.986 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:11.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:11.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:59:12.088 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:12.128 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:12.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:12.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:12.230 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:59:12.253 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:12.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:12.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:59:12.355 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:12.395 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:12.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:12.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:12.497 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:12.524 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:12.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:12.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:12.625 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:59:12.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:12.654 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:12.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:59:12.679 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 68 16 98681 98665 90.6645 76.5 0.0114021 0.0110249 2024-09-21T13:59:12.755 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:12.791 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:12.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:12.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:12.893 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:12.921 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:12.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:12.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:13.023 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:13.059 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:13.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:13.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:13.162 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:13.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:13.211 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:13.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:13.313 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:13.358 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:13.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:13.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:13.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:13 smithi187 ceph-mon[33032]: pgmap v513: 148 pgs: 1 active+clean+remapped, 1 peering, 33 active+undersized, 28 active+undersized+degraded, 85 active+clean; 5.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 55 KiB/s rd, 73 MiB/s wr, 1.25k op/s; 47439/285141 objects degraded (16.637%); 5973/285141 objects misplaced (2.095%) 2024-09-21T13:59:13.460 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:13.498 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:13.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:13.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:13.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:13 smithi049 ceph-mon[40489]: pgmap v513: 148 pgs: 1 active+clean+remapped, 1 peering, 33 active+undersized, 28 active+undersized+degraded, 85 active+clean; 5.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 55 KiB/s rd, 73 MiB/s wr, 1.25k op/s; 47439/285141 objects degraded (16.637%); 5973/285141 objects misplaced (2.095%) 2024-09-21T13:59:13.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:13 smithi049 ceph-mon[30080]: pgmap v513: 148 pgs: 1 active+clean+remapped, 1 peering, 33 active+undersized, 28 active+undersized+degraded, 85 active+clean; 5.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 55 KiB/s rd, 73 MiB/s wr, 1.25k op/s; 47439/285141 objects degraded (16.637%); 5973/285141 objects misplaced (2.095%) 2024-09-21T13:59:13.600 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:13.628 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:13.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:13.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:13.679 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 69 16 99928 99912 90.4799 77.9375 0.0184742 0.0110464 2024-09-21T13:59:13.729 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:13.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:13.756 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:13.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:13.857 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:13.896 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:13.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:13.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:13.998 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:14.023 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:14.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:14.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:14.125 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:59:14.159 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:14.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:14.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:59:14.243 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T13:59:14.244 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T13:59:14.260 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:14.299 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:14.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:14.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:14.361 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:14 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:59:14.362 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:14 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:59:14.362 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:14 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:59:14.362 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:14 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:59:14.362 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:14 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:59:14.362 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:14 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:59:14.362 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:14 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:59:14.362 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:14 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:59:14.362 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:14 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:59:14.363 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:14 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:59:14.401 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:14.427 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:14.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:14.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:14.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:14 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T13:59:14.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:14 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T13:59:14.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:14 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:59:14.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:14 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:59:14.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:14 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:59:14.529 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:59:14.554 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:14.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:14.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:59:14.656 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:14.679 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 70 16 101250 101234 90.3676 82.625 0.00948538 0.0110614 2024-09-21T13:59:14.694 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:14.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:14.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:14.757 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T13:59:14.795 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:14.860 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:14.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:14.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:14.962 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:15.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:15.001 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:15.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:15.102 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:15.152 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:15.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:15.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:15.217 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:59:15.254 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:15.301 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:15.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:15.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:15.404 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:59:15.447 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:15.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:15.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:59:15.549 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:15.601 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:15.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:15.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:15.679 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 71 16 102508 102492 90.2022 78.625 0.0182086 0.011081 2024-09-21T13:59:15.703 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:15.722 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T13:59:15.730 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:15.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:15.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:15.831 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:59:15.888 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T13:59:15.888 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:59:15.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:15.889 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:15.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:59:15.990 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:59:16.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:15 smithi049 ceph-mon[40489]: pgmap v514: 148 pgs: 1 active+clean+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 48 KiB/s rd, 72 MiB/s wr, 1.23k op/s; 49360/294255 objects degraded (16.775%); 5973/294255 objects misplaced (2.030%); 13 B/s, 0 objects/s recovering 2024-09-21T13:59:16.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:15 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 49360/294255 objects degraded (16.775%), 28 pgs degraded (PG_DEGRADED) 2024-09-21T13:59:16.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:15 smithi049 ceph-mon[30080]: pgmap v514: 148 pgs: 1 active+clean+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 48 KiB/s rd, 72 MiB/s wr, 1.23k op/s; 49360/294255 objects degraded (16.775%); 5973/294255 objects misplaced (2.030%); 13 B/s, 0 objects/s recovering 2024-09-21T13:59:16.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:15 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 49360/294255 objects degraded (16.775%), 28 pgs degraded (PG_DEGRADED) 2024-09-21T13:59:16.015 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:16.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:16.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:59:16.117 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:59:16.147 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:16.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:16.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:59:16.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:15 smithi187 ceph-mon[33032]: pgmap v514: 148 pgs: 1 active+clean+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 48 KiB/s rd, 72 MiB/s wr, 1.23k op/s; 49360/294255 objects degraded (16.775%); 5973/294255 objects misplaced (2.030%); 13 B/s, 0 objects/s recovering 2024-09-21T13:59:16.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:15 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 49360/294255 objects degraded (16.775%), 28 pgs degraded (PG_DEGRADED) 2024-09-21T13:59:16.245 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T13:59:16.248 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:59:16.310 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:16.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:16.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:59:16.412 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:16.436 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:16.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:16.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:16.538 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:16.564 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:16.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:16.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:16.665 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:59:16.679 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 72 16 103706 103690 89.9892 74.875 0.00529761 0.0111077 2024-09-21T13:59:16.708 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:16.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:16.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:59:16.810 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:59:16.840 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:16.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:16.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:59:16.942 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:16.964 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:16.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:16.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:17.065 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:17.095 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:17.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:17.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:17.197 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:17.232 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:17.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:17.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:17.334 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:59:17.364 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:17.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:17.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:59:17.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:17 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:59:17.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:17 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:59:17.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:17 smithi187 ceph-mon[33032]: pgmap v515: 148 pgs: 1 active+clean+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 47 KiB/s rd, 96 MiB/s wr, 1.61k op/s; 50533/303459 objects degraded (16.652%); 6353/303459 objects misplaced (2.094%); 13 B/s, 0 objects/s recovering 2024-09-21T13:59:17.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:17 smithi187 ceph-mon[33032]: osdmap e293: 8 total, 7 up, 8 in 2024-09-21T13:59:17.466 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:17.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:17 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:59:17.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:17 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:59:17.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:17 smithi049 ceph-mon[40489]: pgmap v515: 148 pgs: 1 active+clean+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 47 KiB/s rd, 96 MiB/s wr, 1.61k op/s; 50533/303459 objects degraded (16.652%); 6353/303459 objects misplaced (2.094%); 13 B/s, 0 objects/s recovering 2024-09-21T13:59:17.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:17 smithi049 ceph-mon[40489]: osdmap e293: 8 total, 7 up, 8 in 2024-09-21T13:59:17.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:17 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:59:17.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:17 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:59:17.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:17 smithi049 ceph-mon[30080]: pgmap v515: 148 pgs: 1 active+clean+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 47 KiB/s rd, 96 MiB/s wr, 1.61k op/s; 50533/303459 objects degraded (16.652%); 6353/303459 objects misplaced (2.094%); 13 B/s, 0 objects/s recovering 2024-09-21T13:59:17.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:17 smithi049 ceph-mon[30080]: osdmap e293: 8 total, 7 up, 8 in 2024-09-21T13:59:17.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:17.506 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:17.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:17.607 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:17.645 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:17.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:17.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:17.680 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 73 16 104819 104803 89.7093 69.5625 0.0122364 0.0111425 2024-09-21T13:59:17.747 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:17.784 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:17.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:17.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:17.886 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:17.925 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:17.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:17.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:18.027 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:59:18.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:18.060 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:18.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:59:18.092 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [7] live_osds: [0, 1, 4, 5, 6, 2, 3] 2024-09-21T13:59:18.092 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T13:59:18.092 INFO:tasks.thrashosds.thrasher:Reviving osd 7 2024-09-21T13:59:18.092 INFO:tasks.cephadm.osd.7:Restarting osd.7 (starting--it wasn't running)... 2024-09-21T13:59:18.093 DEBUG:teuthology.orchestra.run.smithi187:osd.7> sudo journalctl -f -n 0 -u ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@osd.7.service 2024-09-21T13:59:18.103 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl start ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@osd.7 2024-09-21T13:59:18.161 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:59:18.200 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:18.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:18.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:59:18.302 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:59:18.339 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:18.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:18.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:59:18.441 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:18.445 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 13:59:18 smithi187 systemd[1]: Starting Ceph osd.7 for 40a3d2e8-781f-11ef-baf6-efdc52797490... 2024-09-21T13:59:18.475 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:18.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:18.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:18.577 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:18.614 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:18.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:18.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:18.680 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 74 16 106028 106012 89.5181 75.5625 0.00834124 0.011166 2024-09-21T13:59:18.717 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:18.749 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 13:59:18 smithi187 podman[112358]: 2024-09-21 13:59:18.371424532 +0000 UTC m=+0.016105739 image pull b415816db8b41f544cdfdc71d193ff8b1ee008bf7b0ec4f64da8eb8f71b62e85 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5 2024-09-21T13:59:18.749 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 13:59:18 smithi187 podman[112358]: 2024-09-21 13:59:18.48696334 +0000 UTC m=+0.131644547 container create d53bfead2b25ed8bc2a71215817c7039e2af975d881d76ae5d03304435a880ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7-activate, ceph=True, GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-21T13:59:18.749 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 13:59:18 smithi187 podman[112358]: 2024-09-21 13:59:18.748695111 +0000 UTC m=+0.393376326 container init d53bfead2b25ed8bc2a71215817c7039e2af975d881d76ae5d03304435a880ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7-activate, GIT_BRANCH=HEAD, RELEASE=wip-guits-main-2024-09-21-0842-4417849, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-09-21T13:59:18.755 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:18.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:18.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:18.857 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:18.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:18.902 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:18.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:19.003 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:19.043 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:19.043 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 13:59:18 smithi187 podman[112358]: 2024-09-21 13:59:18.757301935 +0000 UTC m=+0.401983151 container start d53bfead2b25ed8bc2a71215817c7039e2af975d881d76ae5d03304435a880ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-guits-main-2024-09-21-0842-4417849) 2024-09-21T13:59:19.044 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 13:59:18 smithi187 podman[112358]: 2024-09-21 13:59:18.873711254 +0000 UTC m=+0.518392474 container attach d53bfead2b25ed8bc2a71215817c7039e2af975d881d76ae5d03304435a880ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7-activate, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, RELEASE=wip-guits-main-2024-09-21-0842-4417849, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=) 2024-09-21T13:59:19.044 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 13:59:18 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7-activate[112389]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T13:59:19.044 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 13:59:18 smithi187 bash[112358]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T13:59:19.044 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 13:59:18 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7-activate[112389]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T13:59:19.044 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 13:59:18 smithi187 bash[112358]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T13:59:19.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:19.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:19.145 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:19.186 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:19.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:19.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:19.288 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:19.327 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:19.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:19.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:19.428 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:59:19.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:19 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:59:19.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:19 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:59:19.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:19 smithi187 ceph-mon[33032]: pgmap v517: 148 pgs: 1 active+clean+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 6.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 19 KiB/s rd, 65 MiB/s wr, 1.06k op/s; 50533/303459 objects degraded (16.652%); 6353/303459 objects misplaced (2.094%); 12 B/s, 0 objects/s recovering 2024-09-21T13:59:19.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:19 smithi187 ceph-mon[33032]: osdmap e294: 8 total, 7 up, 8 in 2024-09-21T13:59:19.451 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:19.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:19.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:59:19.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:19 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:59:19.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:19 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:59:19.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:19 smithi049 ceph-mon[40489]: pgmap v517: 148 pgs: 1 active+clean+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 6.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 19 KiB/s rd, 65 MiB/s wr, 1.06k op/s; 50533/303459 objects degraded (16.652%); 6353/303459 objects misplaced (2.094%); 12 B/s, 0 objects/s recovering 2024-09-21T13:59:19.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:19 smithi049 ceph-mon[40489]: osdmap e294: 8 total, 7 up, 8 in 2024-09-21T13:59:19.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:19 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:59:19.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:19 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:59:19.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:19 smithi049 ceph-mon[30080]: pgmap v517: 148 pgs: 1 active+clean+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 6.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 19 KiB/s rd, 65 MiB/s wr, 1.06k op/s; 50533/303459 objects degraded (16.652%); 6353/303459 objects misplaced (2.094%); 12 B/s, 0 objects/s recovering 2024-09-21T13:59:19.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:19 smithi049 ceph-mon[30080]: osdmap e294: 8 total, 7 up, 8 in 2024-09-21T13:59:19.554 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:19.582 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:19.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:19.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:19.680 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 75 16 107237 107221 89.3316 75.5625 0.00746624 0.0111898 2024-09-21T13:59:19.684 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:59:19.708 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:19.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:19.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:59:19.810 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:59:19.852 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:19.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:19.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:59:19.944 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:59:19 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:59:19.955 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:19.996 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:19.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:19.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:20.097 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:20.135 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:20.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:20.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:20.236 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:20.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:20.273 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:20.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:20.374 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:20.383 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:20 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:59:20.383 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:20 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 50533/303459 objects degraded (16.652%), 28 pgs degraded (PG_DEGRADED) 2024-09-21T13:59:20.384 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:20 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:59:20.384 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:20 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 50533/303459 objects degraded (16.652%), 28 pgs degraded (PG_DEGRADED) 2024-09-21T13:59:20.399 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:20.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:20.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:20.445 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T13:59:20.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:20 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:59:20.446 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:20 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 50533/303459 objects degraded (16.652%), 28 pgs degraded (PG_DEGRADED) 2024-09-21T13:59:20.501 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:20.519 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:59:20.562 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:59:20.574 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:20.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:20.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:20.671 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:59:20 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:59:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:59:20.676 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:20.680 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 76 16 108393 108377 89.1064 72.25 0.00613997 0.0112183 2024-09-21T13:59:20.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:20.740 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:20.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:20.841 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:59:20.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:20.868 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:20.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:59:20.969 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:20.997 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:20.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:20.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:21.099 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:59:21.125 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:21.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:21.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:59:21.226 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:59:21.254 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:21.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:21 smithi049 ceph-mon[40489]: pgmap v519: 148 pgs: 1 active+clean+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s; 52820/315981 objects degraded (16.716%); 6353/315981 objects misplaced (2.011%); 13 B/s, 0 objects/s recovering 2024-09-21T13:59:21.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:21 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:59:21.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:21 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:59:21.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:21 smithi049 ceph-mon[30080]: pgmap v519: 148 pgs: 1 active+clean+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s; 52820/315981 objects degraded (16.716%); 6353/315981 objects misplaced (2.011%); 13 B/s, 0 objects/s recovering 2024-09-21T13:59:21.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:21 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:59:21.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:21 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:59:21.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:21.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:59:21.357 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:21.397 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:21.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:21.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:21.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:21 smithi187 ceph-mon[33032]: pgmap v519: 148 pgs: 1 active+clean+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s; 52820/315981 objects degraded (16.716%); 6353/315981 objects misplaced (2.011%); 13 B/s, 0 objects/s recovering 2024-09-21T13:59:21.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:21 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:59:21.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:21 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:59:21.499 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:21.529 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:21.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:21.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:21.631 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:21.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:21.663 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:21.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:21.680 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 77 16 109545 109529 88.8842 72 0.00432844 0.0112402 2024-09-21T13:59:21.764 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:59:21.786 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:21.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:21.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:59:21.888 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:59:21.911 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:21.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:21.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:59:22.012 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:22.051 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:22.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:22.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:22.085 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-09-21T13:59:22.153 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:22.157 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:59:22.157 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:59:22.157 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:59:22.157 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:59:22.157 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:59:22.157 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:59:22.157 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:59:22.157 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:59:22.157 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:59:22.157 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:59:22.157 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:59:22.158 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:59:22.158 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:59:22.158 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:59:22.158 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T13:59:22.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:22 smithi187 ceph-mon[33032]: osdmap e295: 8 total, 7 up, 8 in 2024-09-21T13:59:22.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:22 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2506643' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:59:22.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:22 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2667304919' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:59:22.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:22 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:59:22.195 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 13:59:21 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7-activate[112389]: --> Failed to activate via raw: did not find any matching OSD to activate 2024-09-21T13:59:22.195 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 13:59:21 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7-activate[112389]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T13:59:22.195 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 13:59:21 smithi187 bash[112358]: --> Failed to activate via raw: did not find any matching OSD to activate 2024-09-21T13:59:22.195 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 13:59:21 smithi187 bash[112358]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T13:59:22.195 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 13:59:21 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7-activate[112389]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T13:59:22.195 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 13:59:21 smithi187 bash[112358]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T13:59:22.195 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 13:59:21 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7-activate[112389]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-21T13:59:22.195 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 13:59:21 smithi187 bash[112358]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-21T13:59:22.195 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 13:59:21 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7-activate[112389]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_1 --path /var/lib/ceph/osd/ceph-7 --no-mon-config 2024-09-21T13:59:22.195 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 13:59:21 smithi187 bash[112358]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_1 --path /var/lib/ceph/osd/ceph-7 --no-mon-config 2024-09-21T13:59:22.203 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:22.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:22.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:22.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:22 smithi049 ceph-mon[40489]: osdmap e295: 8 total, 7 up, 8 in 2024-09-21T13:59:22.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:22 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2506643' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:59:22.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:22 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2667304919' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:59:22.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:22 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:59:22.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:22 smithi049 ceph-mon[30080]: osdmap e295: 8 total, 7 up, 8 in 2024-09-21T13:59:22.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:22 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2506643' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:59:22.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:22 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2667304919' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:59:22.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:22 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:59:22.306 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:59:22.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:22.345 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:22.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:59:22.446 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:22.477 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:22.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:22.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:22.579 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:59:22.604 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:22.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:22.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:59:22.681 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 78 16 110379 110363 88.4129 52.125 0.0224853 0.0113021 2024-09-21T13:59:22.695 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 13:59:22 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7-activate[112389]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-7/block 2024-09-21T13:59:22.695 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 13:59:22 smithi187 bash[112358]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-7/block 2024-09-21T13:59:22.695 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 13:59:22 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7-activate[112389]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-7/block 2024-09-21T13:59:22.695 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 13:59:22 smithi187 bash[112358]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-7/block 2024-09-21T13:59:22.695 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 13:59:22 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7-activate[112389]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-21T13:59:22.695 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 13:59:22 smithi187 bash[112358]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-21T13:59:22.695 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 13:59:22 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7-activate[112389]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-21T13:59:22.695 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 13:59:22 smithi187 bash[112358]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-21T13:59:22.695 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 13:59:22 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7-activate[112389]: --> ceph-volume lvm activate successful for osd ID: 7 2024-09-21T13:59:22.695 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 13:59:22 smithi187 bash[112358]: --> ceph-volume lvm activate successful for osd ID: 7 2024-09-21T13:59:22.696 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 13:59:22 smithi187 podman[112358]: 2024-09-21 13:59:22.309340026 +0000 UTC m=+3.954021239 container died d53bfead2b25ed8bc2a71215817c7039e2af975d881d76ae5d03304435a880ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7-activate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-21-0842-4417849, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T13:59:22.706 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:22.736 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:22.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:22.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:22.838 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:22.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:22.883 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:22.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:22.984 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:23.012 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:23.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:23.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:23.065 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T13:59:23.065 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T13:59:23.114 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:59:23.139 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:23.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:23.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:59:23.241 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:23.270 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:23.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:23.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:23.372 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:23.420 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:23.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:23.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:23.436 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:23 smithi049 ceph-mon[40489]: pgmap v521: 148 pgs: 1 active+clean+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 62 MiB/s wr, 1k op/s; 53549/321468 objects degraded (16.658%); 6744/321468 objects misplaced (2.098%) 2024-09-21T13:59:23.436 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:23 smithi049 ceph-mon[40489]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:59:23.437 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:23 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T13:59:23.437 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:23 smithi049 ceph-mon[40489]: osdmap e296: 8 total, 7 up, 8 in 2024-09-21T13:59:23.437 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:23 smithi049 ceph-mon[40489]: 2.2 scrub starts 2024-09-21T13:59:23.437 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:23 smithi049 ceph-mon[40489]: 2.2 scrub ok 2024-09-21T13:59:23.437 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:23 smithi049 ceph-mon[40489]: 9.0 scrub starts 2024-09-21T13:59:23.437 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:23 smithi049 ceph-mon[30080]: pgmap v521: 148 pgs: 1 active+clean+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 62 MiB/s wr, 1k op/s; 53549/321468 objects degraded (16.658%); 6744/321468 objects misplaced (2.098%) 2024-09-21T13:59:23.437 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:23 smithi049 ceph-mon[30080]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:59:23.437 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:23 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T13:59:23.437 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:23 smithi049 ceph-mon[30080]: osdmap e296: 8 total, 7 up, 8 in 2024-09-21T13:59:23.437 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:23 smithi049 ceph-mon[30080]: 2.2 scrub starts 2024-09-21T13:59:23.437 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:23 smithi049 ceph-mon[30080]: 2.2 scrub ok 2024-09-21T13:59:23.437 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:23 smithi049 ceph-mon[30080]: 9.0 scrub starts 2024-09-21T13:59:23.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:23 smithi187 ceph-mon[33032]: pgmap v521: 148 pgs: 1 active+clean+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 62 MiB/s wr, 1k op/s; 53549/321468 objects degraded (16.658%); 6744/321468 objects misplaced (2.098%) 2024-09-21T13:59:23.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:23 smithi187 ceph-mon[33032]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:59:23.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:23 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T13:59:23.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:23 smithi187 ceph-mon[33032]: osdmap e296: 8 total, 7 up, 8 in 2024-09-21T13:59:23.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:23 smithi187 ceph-mon[33032]: 2.2 scrub starts 2024-09-21T13:59:23.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:23 smithi187 ceph-mon[33032]: 2.2 scrub ok 2024-09-21T13:59:23.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:23 smithi187 ceph-mon[33032]: 9.0 scrub starts 2024-09-21T13:59:23.522 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:59:23.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:23.559 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:23.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:59:23.660 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:23.682 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 79 16 111624 111608 88.2765 77.8125 0.0181247 0.0113234 2024-09-21T13:59:23.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:23.700 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:23.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:23.778 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:59:23.801 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:23.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:23.867 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:23.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:23.968 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:59:23.997 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:23.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:23.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:59:24.100 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:24.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:24.144 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:24.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:24.189 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:24 smithi187 ceph-mon[33032]: 5.2 scrub starts 2024-09-21T13:59:24.189 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:24 smithi187 ceph-mon[33032]: 5.2 scrub ok 2024-09-21T13:59:24.190 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:24 smithi187 ceph-mon[33032]: 4.1d scrub starts 2024-09-21T13:59:24.190 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:24 smithi187 ceph-mon[33032]: 4.1d scrub ok 2024-09-21T13:59:24.190 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:24 smithi187 ceph-mon[33032]: 6.2 scrub starts 2024-09-21T13:59:24.190 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:24 smithi187 ceph-mon[33032]: 6.2 scrub ok 2024-09-21T13:59:24.190 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 13:59:23 smithi187 podman[112779]: 2024-09-21 13:59:23.936008841 +0000 UTC m=+1.619241130 container remove d53bfead2b25ed8bc2a71215817c7039e2af975d881d76ae5d03304435a880ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7-activate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-guits-main-2024-09-21-0842-4417849, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS) 2024-09-21T13:59:24.245 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:59:24.273 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:24.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:24.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:59:24.375 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:59:24.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:24.406 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:24.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:59:24.429 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:24 smithi049 ceph-mon[40489]: 5.2 scrub starts 2024-09-21T13:59:24.429 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:24 smithi049 ceph-mon[40489]: 5.2 scrub ok 2024-09-21T13:59:24.429 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:24 smithi049 ceph-mon[40489]: 4.1d scrub starts 2024-09-21T13:59:24.429 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:24 smithi049 ceph-mon[40489]: 4.1d scrub ok 2024-09-21T13:59:24.430 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:24 smithi049 ceph-mon[40489]: 6.2 scrub starts 2024-09-21T13:59:24.430 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:24 smithi049 ceph-mon[40489]: 6.2 scrub ok 2024-09-21T13:59:24.430 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:24 smithi049 ceph-mon[30080]: 5.2 scrub starts 2024-09-21T13:59:24.430 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:24 smithi049 ceph-mon[30080]: 5.2 scrub ok 2024-09-21T13:59:24.430 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:24 smithi049 ceph-mon[30080]: 4.1d scrub starts 2024-09-21T13:59:24.430 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:24 smithi049 ceph-mon[30080]: 4.1d scrub ok 2024-09-21T13:59:24.430 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:24 smithi049 ceph-mon[30080]: 6.2 scrub starts 2024-09-21T13:59:24.430 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:24 smithi049 ceph-mon[30080]: 6.2 scrub ok 2024-09-21T13:59:24.444 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 13:59:24 smithi187 podman[112940]: 2024-09-21 13:59:24.094772345 +0000 UTC m=+0.024504295 image pull b415816db8b41f544cdfdc71d193ff8b1ee008bf7b0ec4f64da8eb8f71b62e85 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5 2024-09-21T13:59:24.445 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 13:59:24 smithi187 podman[112940]: 2024-09-21 13:59:24.248590565 +0000 UTC m=+0.178322492 container create c1d12429d1b8305b6384fa8fe1242c1cf4de491ffc089af38043c0077038ac86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-21-0842-4417849, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T13:59:24.491 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T13:59:24.491 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-09-21T13:59:24.491 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-09-21T13:59:24.491 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-09-21T13:59:24.491 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.491 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.14817.0:49348 7.f e98/94 7:fcf6cd05:::benchmark_data_smithi049_76798_object49347:head v 98'3012, mlcod=98'3010)", 2024-09-21T13:59:24.491 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:38.253256+0000", 2024-09-21T13:59:24.491 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 526.23566896600005, 2024-09-21T13:59:24.491 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.155802151, 2024-09-21T13:59:24.491 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:59:24.492 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:59:24.492 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:59:24.492 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:59:24.492 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.492 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:59:24.492 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.253256+0000", 2024-09-21T13:59:24.492 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:24.492 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.492 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.492 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:59:24.492 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.253256+0000", 2024-09-21T13:59:24.492 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:24.492 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.492 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.492 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:59:24.493 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.253257+0000", 2024-09-21T13:59:24.493 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0599999999999999e-07 2024-09-21T13:59:24.493 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.493 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.493 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:59:24.493 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.253283+0000", 2024-09-21T13:59:24.493 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.584e-05 2024-09-21T13:59:24.493 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.493 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.493 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:59:24.493 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.253293+0000", 2024-09-21T13:59:24.493 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0341e-05 2024-09-21T13:59:24.493 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.493 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.493 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:59:24.494 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.253300+0000", 2024-09-21T13:59:24.494 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9249999999999998e-06 2024-09-21T13:59:24.494 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.494 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.494 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:59:24.494 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.253596+0000", 2024-09-21T13:59:24.494 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00029586800000000002 2024-09-21T13:59:24.494 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.494 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.494 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:59:24.494 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.253614+0000", 2024-09-21T13:59:24.494 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7682000000000001e-05 2024-09-21T13:59:24.494 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.495 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.495 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:59:24.495 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.409015+0000", 2024-09-21T13:59:24.495 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.155401716 2024-09-21T13:59:24.495 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.495 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.495 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:59:24.495 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.409058+0000", 2024-09-21T13:59:24.495 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2873000000000001e-05 2024-09-21T13:59:24.495 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:24.495 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:24.495 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:24.495 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.495 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.496 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.14817.0:49359 7.e e98/83 7:7ae75bfc:::benchmark_data_smithi049_76798_object49358:head v 98'3124, mlcod=98'3122)", 2024-09-21T13:59:24.496 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:38.254961+0000", 2024-09-21T13:59:24.496 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 526.23396374900005, 2024-09-21T13:59:24.496 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.152662036, 2024-09-21T13:59:24.496 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:59:24.496 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:59:24.496 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:59:24.496 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:59:24.496 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.496 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:59:24.496 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.254961+0000", 2024-09-21T13:59:24.496 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:24.496 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.496 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.497 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:59:24.497 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.254961+0000", 2024-09-21T13:59:24.497 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:24.497 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.497 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.497 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:59:24.497 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.254962+0000", 2024-09-21T13:59:24.497 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1969999999999999e-06 2024-09-21T13:59:24.497 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.497 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.497 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:59:24.497 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.254987+0000", 2024-09-21T13:59:24.497 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4510000000000001e-05 2024-09-21T13:59:24.497 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.498 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.498 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:59:24.498 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.255000+0000", 2024-09-21T13:59:24.498 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2543e-05 2024-09-21T13:59:24.498 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.498 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.498 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:59:24.498 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.255007+0000", 2024-09-21T13:59:24.498 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1840000000000002e-06 2024-09-21T13:59:24.498 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.498 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.498 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:59:24.498 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257538+0000", 2024-09-21T13:59:24.498 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0025315950000000002 2024-09-21T13:59:24.499 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.499 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.499 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:59:24.499 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257559+0000", 2024-09-21T13:59:24.499 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0812999999999999e-05 2024-09-21T13:59:24.499 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.499 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.499 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:59:24.499 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.259702+0000", 2024-09-21T13:59:24.499 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0021427920000000001 2024-09-21T13:59:24.499 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.499 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.499 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:59:24.499 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.407623+0000", 2024-09-21T13:59:24.500 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.14792140200000001 2024-09-21T13:59:24.500 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:24.500 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:24.500 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:24.500 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.500 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.500 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.14817.0:49358 7.d e98/83 7:bafe9a80:::benchmark_data_smithi049_76798_object49357:head v 98'3118, mlcod=98'3116)", 2024-09-21T13:59:24.500 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:38.257159+0000", 2024-09-21T13:59:24.500 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 526.23176597500003, 2024-09-21T13:59:24.500 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.14991167599999999, 2024-09-21T13:59:24.500 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:59:24.500 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:59:24.500 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:59:24.500 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:59:24.500 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.501 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:59:24.501 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257159+0000", 2024-09-21T13:59:24.501 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:24.501 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.501 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.501 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:59:24.501 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257159+0000", 2024-09-21T13:59:24.501 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:24.501 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.501 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.501 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:59:24.501 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257160+0000", 2024-09-21T13:59:24.501 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.198e-06 2024-09-21T13:59:24.501 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.502 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.502 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:59:24.502 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257209+0000", 2024-09-21T13:59:24.502 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9098000000000003e-05 2024-09-21T13:59:24.502 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.502 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.502 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:59:24.502 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257226+0000", 2024-09-21T13:59:24.502 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6657e-05 2024-09-21T13:59:24.502 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.502 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.502 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:59:24.502 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257236+0000", 2024-09-21T13:59:24.502 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6900000000000004e-06 2024-09-21T13:59:24.503 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.503 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.503 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:59:24.503 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257650+0000", 2024-09-21T13:59:24.503 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000414693 2024-09-21T13:59:24.503 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.503 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.503 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:59:24.503 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257669+0000", 2024-09-21T13:59:24.503 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8938999999999999e-05 2024-09-21T13:59:24.503 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.503 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.503 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:59:24.503 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.407019+0000", 2024-09-21T13:59:24.504 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.149349182 2024-09-21T13:59:24.504 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.504 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.504 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:59:24.504 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.407071+0000", 2024-09-21T13:59:24.504 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2219000000000003e-05 2024-09-21T13:59:24.504 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:24.504 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:24.504 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:24.504 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.504 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.504 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.14817.0:49364 7.d e98/83 7:ba37e2e5:::benchmark_data_smithi049_76798_object49363:head v 98'3119, mlcod=98'3116)", 2024-09-21T13:59:24.504 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:38.257264+0000", 2024-09-21T13:59:24.504 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 526.23166140900003, 2024-09-21T13:59:24.504 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.15315973299999999, 2024-09-21T13:59:24.505 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:59:24.505 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:59:24.505 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:59:24.505 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:59:24.505 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.505 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:59:24.505 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257264+0000", 2024-09-21T13:59:24.505 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:24.506 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.506 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.506 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:59:24.506 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257264+0000", 2024-09-21T13:59:24.506 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:24.506 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.506 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.506 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:59:24.506 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257264+0000", 2024-09-21T13:59:24.506 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9000000000000002e-07 2024-09-21T13:59:24.506 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.506 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.506 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:59:24.507 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257298+0000", 2024-09-21T13:59:24.507 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3547999999999998e-05 2024-09-21T13:59:24.507 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.507 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.507 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:59:24.507 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257306+0000", 2024-09-21T13:59:24.507 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0809999999999994e-06 2024-09-21T13:59:24.507 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.507 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.507 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:59:24.507 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257311+0000", 2024-09-21T13:59:24.507 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7950000000000004e-06 2024-09-21T13:59:24.507 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.507 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.508 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:59:24.508 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.407103+0000", 2024-09-21T13:59:24.508 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.14979118299999999 2024-09-21T13:59:24.508 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.508 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.508 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:59:24.508 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.407123+0000", 2024-09-21T13:59:24.508 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9993e-05 2024-09-21T13:59:24.508 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.508 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.508 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:59:24.508 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.410375+0000", 2024-09-21T13:59:24.508 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0032521910000000002 2024-09-21T13:59:24.508 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.509 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.509 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:59:24.509 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.410423+0000", 2024-09-21T13:59:24.509 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8551999999999999e-05 2024-09-21T13:59:24.509 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:24.509 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:24.509 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:24.509 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.509 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.509 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.14817.0:49355 7.f e98/94 7:f1158586:::benchmark_data_smithi049_76798_object49354:head v 98'3013, mlcod=98'3010)", 2024-09-21T13:59:24.509 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:38.257372+0000", 2024-09-21T13:59:24.509 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 526.23155339000004, 2024-09-21T13:59:24.509 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.15605590799999999, 2024-09-21T13:59:24.509 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:59:24.509 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:59:24.510 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:59:24.510 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:59:24.510 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.510 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:59:24.510 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257372+0000", 2024-09-21T13:59:24.510 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:24.510 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.510 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.510 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:59:24.510 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257372+0000", 2024-09-21T13:59:24.510 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:24.510 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.510 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.513 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:59:24.513 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257372+0000", 2024-09-21T13:59:24.513 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7599999999999997e-07 2024-09-21T13:59:24.513 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.513 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.514 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:59:24.514 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257400+0000", 2024-09-21T13:59:24.514 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.7801e-05 2024-09-21T13:59:24.514 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.514 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.514 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:59:24.514 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257410+0000", 2024-09-21T13:59:24.514 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.978e-06 2024-09-21T13:59:24.514 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.514 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.514 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:59:24.514 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.257415+0000", 2024-09-21T13:59:24.514 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3009999999999997e-06 2024-09-21T13:59:24.515 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.515 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.515 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:59:24.515 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.407104+0000", 2024-09-21T13:59:24.515 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.14968894099999999 2024-09-21T13:59:24.515 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.515 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.515 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:59:24.515 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.407125+0000", 2024-09-21T13:59:24.515 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1132999999999999e-05 2024-09-21T13:59:24.515 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.515 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.516 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:59:24.516 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.413375+0000", 2024-09-21T13:59:24.516 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0062498959999999996 2024-09-21T13:59:24.516 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.516 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.516 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:59:24.516 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.413428+0000", 2024-09-21T13:59:24.516 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2182000000000003e-05 2024-09-21T13:59:24.516 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:24.516 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:24.516 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:24.516 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.516 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.516 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.14817.0:49357 7.f e98/94 7:fffc650e:::benchmark_data_smithi049_76798_object49356:head v 98'3014, mlcod=98'3010)", 2024-09-21T13:59:24.517 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:38.259358+0000", 2024-09-21T13:59:24.517 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 526.22956676399997, 2024-09-21T13:59:24.517 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.160711944, 2024-09-21T13:59:24.517 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:59:24.517 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:59:24.517 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:59:24.517 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:59:24.517 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.517 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:59:24.517 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.259358+0000", 2024-09-21T13:59:24.517 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:24.517 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.517 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.517 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:59:24.518 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.259358+0000", 2024-09-21T13:59:24.518 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:24.518 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.518 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.518 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:59:24.518 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.259359+0000", 2024-09-21T13:59:24.518 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4099999999999998e-07 2024-09-21T13:59:24.518 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.518 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.518 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:59:24.518 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.259382+0000", 2024-09-21T13:59:24.518 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3357e-05 2024-09-21T13:59:24.518 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.519 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.519 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:59:24.519 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.259394+0000", 2024-09-21T13:59:24.519 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.183e-05 2024-09-21T13:59:24.519 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.519 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.519 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:59:24.519 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.259401+0000", 2024-09-21T13:59:24.519 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1930000000000004e-06 2024-09-21T13:59:24.519 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.519 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.519 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:59:24.519 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.409117+0000", 2024-09-21T13:59:24.519 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.149715182 2024-09-21T13:59:24.520 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.520 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.520 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:59:24.520 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.409151+0000", 2024-09-21T13:59:24.520 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4832999999999997e-05 2024-09-21T13:59:24.520 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.520 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.520 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:59:24.520 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.419333+0000", 2024-09-21T13:59:24.520 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.010181913000000001 2024-09-21T13:59:24.520 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.520 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.520 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:59:24.520 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.420070+0000", 2024-09-21T13:59:24.521 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000736895 2024-09-21T13:59:24.521 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:24.521 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:24.521 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:24.521 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.521 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.521 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.14817.0:49366 7.9 e98/94 7:956216ae:::benchmark_data_smithi049_76798_object49365:head v 98'3074, mlcod=98'3072)", 2024-09-21T13:59:24.521 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:38.259484+0000", 2024-09-21T13:59:24.521 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 526.22944082000004, 2024-09-21T13:59:24.521 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.15026700000000001, 2024-09-21T13:59:24.521 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:59:24.521 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:59:24.521 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:59:24.521 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:59:24.522 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.522 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:59:24.522 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.259484+0000", 2024-09-21T13:59:24.522 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:24.522 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.522 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.522 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:59:24.522 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.259484+0000", 2024-09-21T13:59:24.522 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:24.522 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.522 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.522 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:59:24.522 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.259485+0000", 2024-09-21T13:59:24.523 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0299999999999998e-07 2024-09-21T13:59:24.523 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.523 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.523 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:59:24.523 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.259543+0000", 2024-09-21T13:59:24.523 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7998000000000003e-05 2024-09-21T13:59:24.523 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.523 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.523 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:59:24.523 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.259554+0000", 2024-09-21T13:59:24.523 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0842000000000001e-05 2024-09-21T13:59:24.523 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.523 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.523 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:59:24.524 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.259561+0000", 2024-09-21T13:59:24.524 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6680000000000004e-06 2024-09-21T13:59:24.524 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.524 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.524 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:59:24.524 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.259639+0000", 2024-09-21T13:59:24.524 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7986999999999997e-05 2024-09-21T13:59:24.524 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.524 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.524 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:59:24.524 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.407005+0000", 2024-09-21T13:59:24.524 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.14736633399999999 2024-09-21T13:59:24.524 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.524 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.524 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:59:24.525 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.409267+0000", 2024-09-21T13:59:24.525 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0022621690000000001 2024-09-21T13:59:24.525 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.525 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.525 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:59:24.525 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.409751+0000", 2024-09-21T13:59:24.525 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00048419899999999999 2024-09-21T13:59:24.525 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:24.525 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:24.525 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:24.525 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.525 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.525 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.14817.0:49368 7.d e98/83 7:bfd0cd80:::benchmark_data_smithi049_76798_object49367:head v 98'3120, mlcod=98'3116)", 2024-09-21T13:59:24.525 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:38.260130+0000", 2024-09-21T13:59:24.526 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 526.22879538500001, 2024-09-21T13:59:24.526 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.151202637, 2024-09-21T13:59:24.526 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:59:24.526 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:59:24.526 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:59:24.526 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:59:24.526 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.526 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:59:24.526 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.260130+0000", 2024-09-21T13:59:24.526 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:24.526 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.526 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.526 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:59:24.526 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.260130+0000", 2024-09-21T13:59:24.527 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:24.527 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.527 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.527 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:59:24.527 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.260130+0000", 2024-09-21T13:59:24.527 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1100000000000005e-07 2024-09-21T13:59:24.527 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.527 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.527 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:59:24.527 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.260158+0000", 2024-09-21T13:59:24.527 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.7708e-05 2024-09-21T13:59:24.527 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.527 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.527 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:59:24.528 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.407103+0000", 2024-09-21T13:59:24.528 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.146944514 2024-09-21T13:59:24.528 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.528 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.528 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:59:24.528 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.407112+0000", 2024-09-21T13:59:24.528 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2569999999999996e-06 2024-09-21T13:59:24.528 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.528 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.528 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:59:24.528 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.408861+0000", 2024-09-21T13:59:24.528 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0017486590000000001 2024-09-21T13:59:24.528 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.528 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.529 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:59:24.529 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.408881+0000", 2024-09-21T13:59:24.529 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0038999999999999e-05 2024-09-21T13:59:24.529 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.529 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.529 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:59:24.529 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.411202+0000", 2024-09-21T13:59:24.529 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00232153 2024-09-21T13:59:24.529 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.529 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.529 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:59:24.529 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.411332+0000", 2024-09-21T13:59:24.529 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013011899999999999 2024-09-21T13:59:24.529 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:24.530 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:24.530 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:24.530 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.530 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.530 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.14817.0:49369 7.e e98/83 7:7b8e094d:::benchmark_data_smithi049_76798_object49368:head v 98'3125, mlcod=98'3122)", 2024-09-21T13:59:24.530 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:50:38.260639+0000", 2024-09-21T13:59:24.530 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 526.22828638999999, 2024-09-21T13:59:24.530 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.14984267100000001, 2024-09-21T13:59:24.530 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:59:24.530 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:59:24.530 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:59:24.530 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:59:24.530 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.530 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:59:24.531 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.260639+0000", 2024-09-21T13:59:24.531 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:24.531 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.531 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.531 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:59:24.531 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.260639+0000", 2024-09-21T13:59:24.531 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:24.531 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.531 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.531 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:59:24.531 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.260640+0000", 2024-09-21T13:59:24.531 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0400000000000005e-07 2024-09-21T13:59:24.531 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.532 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.532 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:59:24.532 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.260663+0000", 2024-09-21T13:59:24.532 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3859e-05 2024-09-21T13:59:24.532 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.532 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.532 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:59:24.532 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.406927+0000", 2024-09-21T13:59:24.532 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.146263533 2024-09-21T13:59:24.532 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.532 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.532 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:59:24.532 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.406950+0000", 2024-09-21T13:59:24.533 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3014e-05 2024-09-21T13:59:24.533 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.533 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.533 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:59:24.533 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.407706+0000", 2024-09-21T13:59:24.533 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00075584600000000001 2024-09-21T13:59:24.533 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.533 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.533 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:59:24.533 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.407723+0000", 2024-09-21T13:59:24.533 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7085999999999999e-05 2024-09-21T13:59:24.533 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.533 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.533 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:59:24.534 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.410449+0000", 2024-09-21T13:59:24.534 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0027258680000000002 2024-09-21T13:59:24.534 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.534 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.534 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:59:24.534 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:50:38.410481+0000", 2024-09-21T13:59:24.534 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2561000000000002e-05 2024-09-21T13:59:24.534 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:24.534 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:24.534 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:24.534 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.534 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.534 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:13660 8.e 8:7e25e821:::benchmark_data_smithi049_109311_object13659:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-09-21T13:59:24.535 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:54:22.839402+0000", 2024-09-21T13:59:24.535 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 301.64952327999998, 2024-09-21T13:59:24.535 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.106316493, 2024-09-21T13:59:24.535 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:59:24.535 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:59:24.535 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:59:24.535 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:59:24.535 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:59:24.535 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:59:24.535 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 13660 2024-09-21T13:59:24.535 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.535 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:59:24.535 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.535 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:59:24.535 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.839402+0000", 2024-09-21T13:59:24.535 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:24.536 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.536 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.536 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:59:24.536 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:59:24.537 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.839402+0000", 2024-09-21T13:59:24.537 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:24.537 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.537 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.537 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:59:24.537 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.839403+0000", 2024-09-21T13:59:24.537 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.573e-06 2024-09-21T13:59:24.537 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.537 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.537 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:59:24.537 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.839426+0000", 2024-09-21T13:59:24.538 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3059000000000001e-05 2024-09-21T13:59:24.538 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.538 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.538 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:59:24.538 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.839438+0000", 2024-09-21T13:59:24.538 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1851000000000001e-05 2024-09-21T13:59:24.538 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.538 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.538 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:59:24.538 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.839446+0000", 2024-09-21T13:59:24.538 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8609999999999998e-06 2024-09-21T13:59:24.538 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.538 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.539 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:59:24.539 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.839994+0000", 2024-09-21T13:59:24.539 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00054770399999999999 2024-09-21T13:59:24.539 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.539 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.539 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:59:24.539 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.840234+0000", 2024-09-21T13:59:24.539 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00023994700000000001 2024-09-21T13:59:24.539 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.539 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.539 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,4", 2024-09-21T13:59:24.539 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.840537+0000", 2024-09-21T13:59:24.539 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000303004 2024-09-21T13:59:24.539 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.539 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.539 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:24.540 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.842148+0000", 2024-09-21T13:59:24.540 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0016112570000000001 2024-09-21T13:59:24.540 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.540 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.540 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:24.540 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.842528+0000", 2024-09-21T13:59:24.540 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00038031000000000002 2024-09-21T13:59:24.540 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.540 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.540 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:59:24.540 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.842792+0000", 2024-09-21T13:59:24.540 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00026319199999999998 2024-09-21T13:59:24.540 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.540 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.540 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:59:24.541 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.945668+0000", 2024-09-21T13:59:24.541 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.10287652 2024-09-21T13:59:24.541 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.541 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.541 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:59:24.541 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:22.945718+0000", 2024-09-21T13:59:24.541 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0214999999999999e-05 2024-09-21T13:59:24.541 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:24.541 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:24.541 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:24.541 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.541 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.541 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:14213 8.c 8:38837bbf:::benchmark_data_smithi049_109311_object14212:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-09-21T13:59:24.541 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:54:23.116628+0000", 2024-09-21T13:59:24.541 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 301.37229724899998, 2024-09-21T13:59:24.542 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.120616289, 2024-09-21T13:59:24.542 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:59:24.542 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:59:24.542 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:59:24.542 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:59:24.542 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:59:24.542 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:59:24.542 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 14213 2024-09-21T13:59:24.542 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.542 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:59:24.542 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.542 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:59:24.542 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.116628+0000", 2024-09-21T13:59:24.542 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:24.542 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.543 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.543 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:59:24.543 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.116628+0000", 2024-09-21T13:59:24.543 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:24.543 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.543 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.543 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:59:24.543 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.116629+0000", 2024-09-21T13:59:24.543 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8299999999999995e-07 2024-09-21T13:59:24.543 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.543 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.543 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:59:24.543 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.116651+0000", 2024-09-21T13:59:24.543 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2005e-05 2024-09-21T13:59:24.543 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.544 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.544 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:59:24.544 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.116661+0000", 2024-09-21T13:59:24.544 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0533999999999999e-05 2024-09-21T13:59:24.544 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.544 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.544 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:59:24.544 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.116668+0000", 2024-09-21T13:59:24.544 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9349999999999997e-06 2024-09-21T13:59:24.544 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.544 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.544 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:59:24.544 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.116996+0000", 2024-09-21T13:59:24.544 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00032820400000000002 2024-09-21T13:59:24.545 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.545 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.545 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:59:24.545 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.117103+0000", 2024-09-21T13:59:24.545 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010673100000000001 2024-09-21T13:59:24.545 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.545 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.545 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,7", 2024-09-21T13:59:24.545 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.117264+0000", 2024-09-21T13:59:24.545 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016054200000000001 2024-09-21T13:59:24.545 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.545 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.545 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:59:24.545 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.118050+0000", 2024-09-21T13:59:24.546 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00078586699999999999 2024-09-21T13:59:24.546 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.546 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.546 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:24.546 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.118727+0000", 2024-09-21T13:59:24.546 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00067759600000000004 2024-09-21T13:59:24.546 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.546 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.546 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:24.546 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.119082+0000", 2024-09-21T13:59:24.546 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00035450800000000001 2024-09-21T13:59:24.546 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.546 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.546 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:59:24.546 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.119119+0000", 2024-09-21T13:59:24.546 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7568999999999999e-05 2024-09-21T13:59:24.547 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.547 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.547 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:59:24.547 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.237244+0000", 2024-09-21T13:59:24.547 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.11812481499999999 2024-09-21T13:59:24.547 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:24.547 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:24.547 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:24.547 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.547 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.547 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.15207.0:14214 8.2 e189/183 8:46d3a502:::benchmark_data_smithi049_109311_object14213:head v 189'885, mlcod=189'883)", 2024-09-21T13:59:24.547 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:54:23.118548+0000", 2024-09-21T13:59:24.547 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 301.37037719400001, 2024-09-21T13:59:24.547 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.118761609, 2024-09-21T13:59:24.547 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:59:24.547 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:59:24.548 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:59:24.548 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:59:24.548 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.548 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:59:24.548 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.118548+0000", 2024-09-21T13:59:24.548 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:24.548 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.548 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.548 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:59:24.548 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.118548+0000", 2024-09-21T13:59:24.548 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:24.548 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.548 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.548 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:59:24.548 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.118548+0000", 2024-09-21T13:59:24.548 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3e-07 2024-09-21T13:59:24.549 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.549 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.549 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:59:24.549 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.118593+0000", 2024-09-21T13:59:24.549 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4131999999999997e-05 2024-09-21T13:59:24.549 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.549 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.549 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:59:24.549 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.118604+0000", 2024-09-21T13:59:24.549 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.134e-05 2024-09-21T13:59:24.549 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.549 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.549 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:59:24.549 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.118611+0000", 2024-09-21T13:59:24.549 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4619999999999999e-06 2024-09-21T13:59:24.549 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.549 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.550 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:59:24.550 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.118669+0000", 2024-09-21T13:59:24.550 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7828999999999998e-05 2024-09-21T13:59:24.550 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.550 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.550 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:59:24.550 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.118690+0000", 2024-09-21T13:59:24.550 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0596999999999999e-05 2024-09-21T13:59:24.550 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.550 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.550 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:59:24.550 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.237258+0000", 2024-09-21T13:59:24.550 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.11856844599999999 2024-09-21T13:59:24.550 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.551 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.551 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:59:24.551 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.237309+0000", 2024-09-21T13:59:24.551 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1172999999999999e-05 2024-09-21T13:59:24.551 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:24.551 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:24.551 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:24.551 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.551 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.551 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.15207.0:14213 8.c e189/183 ondisk, result = 0)", 2024-09-21T13:59:24.551 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:54:23.118845+0000", 2024-09-21T13:59:24.551 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 301.370079756, 2024-09-21T13:59:24.551 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.11840756199999999, 2024-09-21T13:59:24.552 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:59:24.552 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:59:24.552 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T13:59:24.552 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:59:24.552 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.552 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:59:24.552 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.118845+0000", 2024-09-21T13:59:24.552 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:24.552 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.552 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.552 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:59:24.552 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.118845+0000", 2024-09-21T13:59:24.552 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:24.552 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.552 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.552 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:59:24.552 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.118846+0000", 2024-09-21T13:59:24.553 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3e-07 2024-09-21T13:59:24.553 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.553 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.553 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:59:24.553 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.118850+0000", 2024-09-21T13:59:24.553 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0219999999999998e-06 2024-09-21T13:59:24.553 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.553 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.553 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:59:24.553 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.118860+0000", 2024-09-21T13:59:24.553 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0162e-05 2024-09-21T13:59:24.553 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.553 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.553 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:59:24.553 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.118866+0000", 2024-09-21T13:59:24.554 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8980000000000001e-06 2024-09-21T13:59:24.554 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.554 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.554 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:59:24.554 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.119070+0000", 2024-09-21T13:59:24.554 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000204286 2024-09-21T13:59:24.554 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.554 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.554 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:59:24.554 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.119077+0000", 2024-09-21T13:59:24.554 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9909999999999996e-06 2024-09-21T13:59:24.554 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.554 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.554 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:59:24.554 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.237253+0000", 2024-09-21T13:59:24.554 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.118175473 2024-09-21T13:59:24.555 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:24.555 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:24.555 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:24.555 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.555 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.555 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.15207.0:14217 8.2 e189/183 8:42c0ac0e:::benchmark_data_smithi049_109311_object14216:head v 189'886, mlcod=189'883)", 2024-09-21T13:59:24.555 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:54:23.119299+0000", 2024-09-21T13:59:24.555 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 301.36962591299999, 2024-09-21T13:59:24.555 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.119049031, 2024-09-21T13:59:24.555 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:59:24.555 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:59:24.555 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:59:24.555 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:59:24.555 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.555 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:59:24.555 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.119299+0000", 2024-09-21T13:59:24.556 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:24.556 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.556 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.556 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:59:24.556 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.119299+0000", 2024-09-21T13:59:24.556 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:24.556 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.556 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.556 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:59:24.556 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.119304+0000", 2024-09-21T13:59:24.556 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9790000000000001e-06 2024-09-21T13:59:24.556 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.556 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.556 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:59:24.556 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.119344+0000", 2024-09-21T13:59:24.556 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9943e-05 2024-09-21T13:59:24.556 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.557 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.557 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:59:24.557 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.237161+0000", 2024-09-21T13:59:24.557 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.11781683 2024-09-21T13:59:24.557 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.557 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.557 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:59:24.557 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.237177+0000", 2024-09-21T13:59:24.557 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5709000000000002e-05 2024-09-21T13:59:24.557 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.557 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.557 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:59:24.557 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.237338+0000", 2024-09-21T13:59:24.557 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016102600000000001 2024-09-21T13:59:24.557 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.557 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.558 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:59:24.558 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.237359+0000", 2024-09-21T13:59:24.558 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1659999999999999e-05 2024-09-21T13:59:24.558 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.558 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.558 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:59:24.558 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.238317+0000", 2024-09-21T13:59:24.558 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00095745599999999998 2024-09-21T13:59:24.558 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.558 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.558 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:59:24.558 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.238348+0000", 2024-09-21T13:59:24.558 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1427999999999998e-05 2024-09-21T13:59:24.558 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:24.558 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:24.559 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:24.559 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.559 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.559 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:14223 8.e 8:753b28d9:::benchmark_data_smithi049_109311_object14222:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-09-21T13:59:24.559 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:54:23.121191+0000", 2024-09-21T13:59:24.559 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 301.36773401200003, 2024-09-21T13:59:24.559 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.119117285, 2024-09-21T13:59:24.559 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:59:24.559 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:59:24.559 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:59:24.559 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:59:24.559 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:59:24.559 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:59:24.559 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 14223 2024-09-21T13:59:24.559 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.559 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:59:24.560 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.560 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:59:24.560 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.121191+0000", 2024-09-21T13:59:24.560 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:24.560 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.560 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.560 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:59:24.560 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.121191+0000", 2024-09-21T13:59:24.560 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:24.560 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.560 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.560 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:59:24.560 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.121193+0000", 2024-09-21T13:59:24.560 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.486e-06 2024-09-21T13:59:24.560 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.561 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.561 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:59:24.561 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.121215+0000", 2024-09-21T13:59:24.561 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2371e-05 2024-09-21T13:59:24.561 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.561 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.561 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:59:24.561 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.237516+0000", 2024-09-21T13:59:24.561 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.116301107 2024-09-21T13:59:24.561 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.561 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.561 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:59:24.561 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.237527+0000", 2024-09-21T13:59:24.561 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1479e-05 2024-09-21T13:59:24.561 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.562 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.562 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:59:24.562 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.237593+0000", 2024-09-21T13:59:24.562 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5359999999999998e-05 2024-09-21T13:59:24.562 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.562 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.562 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:59:24.562 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.237741+0000", 2024-09-21T13:59:24.563 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014798799999999999 2024-09-21T13:59:24.563 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.563 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.563 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,4", 2024-09-21T13:59:24.563 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.237862+0000", 2024-09-21T13:59:24.563 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000121057 2024-09-21T13:59:24.563 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.563 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.564 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:59:24.564 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.238729+0000", 2024-09-21T13:59:24.564 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00086727799999999997 2024-09-21T13:59:24.564 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.564 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.564 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:24.564 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.239905+0000", 2024-09-21T13:59:24.564 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0011758280000000001 2024-09-21T13:59:24.564 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.564 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.564 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:24.564 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.240185+0000", 2024-09-21T13:59:24.564 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00028034499999999999 2024-09-21T13:59:24.565 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.565 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.565 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:59:24.565 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.240266+0000", 2024-09-21T13:59:24.565 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0969000000000004e-05 2024-09-21T13:59:24.565 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.565 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.565 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:59:24.565 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:54:23.240308+0000", 2024-09-21T13:59:24.565 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2017000000000001e-05 2024-09-21T13:59:24.565 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:24.565 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:24.565 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:24.565 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.566 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.566 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:212087 8.c 8:37868233:::benchmark_data_smithi049_109311_object85568:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-09-21T13:59:24.566 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:56:55.643930+0000", 2024-09-21T13:59:24.566 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 148.844994727, 2024-09-21T13:59:24.566 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.107147015, 2024-09-21T13:59:24.566 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:59:24.566 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:59:24.566 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:59:24.566 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:59:24.566 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:59:24.566 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:59:24.566 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 212087 2024-09-21T13:59:24.566 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.566 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:59:24.567 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.567 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:59:24.567 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:56:55.643930+0000", 2024-09-21T13:59:24.567 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:24.567 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.567 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.567 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:59:24.567 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:56:55.643930+0000", 2024-09-21T13:59:24.567 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:24.567 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.567 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.567 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:59:24.567 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:56:55.643932+0000", 2024-09-21T13:59:24.567 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.418e-06 2024-09-21T13:59:24.568 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.568 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.568 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:59:24.568 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:56:55.643938+0000", 2024-09-21T13:59:24.568 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9349999999999999e-06 2024-09-21T13:59:24.568 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.568 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.568 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:59:24.568 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:56:55.643953+0000", 2024-09-21T13:59:24.568 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5277999999999998e-05 2024-09-21T13:59:24.568 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.568 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.568 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:59:24.569 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:56:55.643961+0000", 2024-09-21T13:59:24.569 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6129999999999998e-06 2024-09-21T13:59:24.569 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.569 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.569 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:59:24.569 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:56:55.644034+0000", 2024-09-21T13:59:24.569 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3248000000000001e-05 2024-09-21T13:59:24.569 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.569 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.569 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:59:24.569 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:56:55.644172+0000", 2024-09-21T13:59:24.569 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013812500000000001 2024-09-21T13:59:24.569 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.570 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.570 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 7", 2024-09-21T13:59:24.570 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:56:55.644305+0000", 2024-09-21T13:59:24.570 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013281400000000001 2024-09-21T13:59:24.570 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.570 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.570 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:59:24.570 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:56:55.648266+0000", 2024-09-21T13:59:24.570 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.003961366 2024-09-21T13:59:24.570 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.570 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.570 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:24.570 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:56:55.750852+0000", 2024-09-21T13:59:24.570 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.102585553 2024-09-21T13:59:24.571 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.571 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.571 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:59:24.571 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:56:55.750985+0000", 2024-09-21T13:59:24.571 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013286999999999999 2024-09-21T13:59:24.571 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.571 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.571 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:59:24.571 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:56:55.751077+0000", 2024-09-21T13:59:24.571 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2794999999999995e-05 2024-09-21T13:59:24.571 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:24.571 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:24.571 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:24.571 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.572 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.572 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.15207.0:232030 8.2 e239/183 8:49ac6d95:::benchmark_data_smithi049_109311_object105511:head v 239'14230, mlcod=239'14227)", 2024-09-21T13:59:24.572 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.654181+0000", 2024-09-21T13:59:24.572 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 132.83474440200001, 2024-09-21T13:59:24.572 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.227698072, 2024-09-21T13:59:24.572 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:59:24.572 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:59:24.572 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:59:24.572 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:59:24.572 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.572 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:59:24.572 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.654181+0000", 2024-09-21T13:59:24.572 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:24.572 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.573 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.573 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:59:24.573 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.654181+0000", 2024-09-21T13:59:24.573 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:24.573 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.573 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.573 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:59:24.573 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.654181+0000", 2024-09-21T13:59:24.573 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.44e-07 2024-09-21T13:59:24.573 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.573 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.573 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:59:24.573 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.654188+0000", 2024-09-21T13:59:24.573 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1080000000000004e-06 2024-09-21T13:59:24.574 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.574 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.574 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:59:24.574 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.654202+0000", 2024-09-21T13:59:24.574 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3283e-05 2024-09-21T13:59:24.574 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.574 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.574 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:59:24.574 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.654208+0000", 2024-09-21T13:59:24.574 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1179999999999997e-06 2024-09-21T13:59:24.574 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.574 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.574 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:59:24.574 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.654644+0000", 2024-09-21T13:59:24.575 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000436191 2024-09-21T13:59:24.575 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.575 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.575 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:59:24.575 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.654661+0000", 2024-09-21T13:59:24.575 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6623e-05 2024-09-21T13:59:24.575 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.575 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.575 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:59:24.575 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.881820+0000", 2024-09-21T13:59:24.575 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22715939499999999 2024-09-21T13:59:24.575 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.575 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.575 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:59:24.575 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.881879+0000", 2024-09-21T13:59:24.576 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8709999999999999e-05 2024-09-21T13:59:24.576 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:24.576 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:24.576 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:24.576 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.576 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.576 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232032 8.e 8:755f5137:::benchmark_data_smithi049_109311_object105513:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T13:59:24.576 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.656185+0000", 2024-09-21T13:59:24.576 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 132.83274034799999, 2024-09-21T13:59:24.576 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22610913499999999, 2024-09-21T13:59:24.576 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:59:24.576 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:59:24.576 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:59:24.576 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T13:59:24.576 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T13:59:24.576 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:59:24.577 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232032 2024-09-21T13:59:24.577 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.577 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:59:24.577 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.577 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:59:24.577 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.656185+0000", 2024-09-21T13:59:24.577 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:24.577 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.577 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.577 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:59:24.577 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.656185+0000", 2024-09-21T13:59:24.577 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:24.577 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.577 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.577 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:59:24.578 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.656187+0000", 2024-09-21T13:59:24.578 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.215e-06 2024-09-21T13:59:24.578 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.578 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.578 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:59:24.578 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.656193+0000", 2024-09-21T13:59:24.578 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8900000000000004e-06 2024-09-21T13:59:24.578 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.578 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.578 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:59:24.578 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.656207+0000", 2024-09-21T13:59:24.578 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3886999999999999e-05 2024-09-21T13:59:24.578 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.578 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.578 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:59:24.579 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.656214+0000", 2024-09-21T13:59:24.579 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6210000000000004e-06 2024-09-21T13:59:24.579 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.579 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.579 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:59:24.579 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.658622+0000", 2024-09-21T13:59:24.579 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0024080999999999998 2024-09-21T13:59:24.579 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.579 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.579 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:59:24.579 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.658725+0000", 2024-09-21T13:59:24.579 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000102497 2024-09-21T13:59:24.579 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.579 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.579 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,4", 2024-09-21T13:59:24.579 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.658827+0000", 2024-09-21T13:59:24.579 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000101846 2024-09-21T13:59:24.580 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.580 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.580 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:24.580 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.666078+0000", 2024-09-21T13:59:24.580 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0072517420000000003 2024-09-21T13:59:24.580 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.580 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.580 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T13:59:24.580 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.666142+0000", 2024-09-21T13:59:24.580 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3538000000000001e-05 2024-09-21T13:59:24.580 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.580 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.580 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:24.580 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.882193+0000", 2024-09-21T13:59:24.580 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21605080600000001 2024-09-21T13:59:24.580 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.581 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.581 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:59:24.581 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.882250+0000", 2024-09-21T13:59:24.581 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6773e-05 2024-09-21T13:59:24.581 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.581 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.581 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:59:24.581 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.882294+0000", 2024-09-21T13:59:24.581 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.422e-05 2024-09-21T13:59:24.581 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:24.581 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:24.581 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:24.581 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.581 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.582 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.15207.0:232036 8.2 e239/183 8:4507ec93:::benchmark_data_smithi049_109311_object105517:head v 239'14231, mlcod=239'14227)", 2024-09-21T13:59:24.582 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.656873+0000", 2024-09-21T13:59:24.582 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 132.832051826, 2024-09-21T13:59:24.582 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22504153599999999, 2024-09-21T13:59:24.582 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:59:24.582 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:59:24.582 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:59:24.582 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:59:24.582 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.582 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:59:24.582 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.656873+0000", 2024-09-21T13:59:24.582 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:24.582 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.582 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.583 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:59:24.583 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.656873+0000", 2024-09-21T13:59:24.583 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:24.583 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.583 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.583 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:59:24.583 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.656874+0000", 2024-09-21T13:59:24.583 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7e-07 2024-09-21T13:59:24.583 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.583 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.583 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:59:24.583 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.656877+0000", 2024-09-21T13:59:24.583 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4489999999999999e-06 2024-09-21T13:59:24.583 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.583 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.583 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:59:24.584 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.656887+0000", 2024-09-21T13:59:24.584 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9550000000000004e-06 2024-09-21T13:59:24.584 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.584 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.584 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:59:24.584 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.656892+0000", 2024-09-21T13:59:24.584 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6630000000000001e-06 2024-09-21T13:59:24.584 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.584 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.584 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:59:24.584 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.670143+0000", 2024-09-21T13:59:24.584 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.013251447 2024-09-21T13:59:24.584 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.584 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.584 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:59:24.584 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.670163+0000", 2024-09-21T13:59:24.585 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0185000000000001e-05 2024-09-21T13:59:24.585 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.585 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.585 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T13:59:24.585 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.881884+0000", 2024-09-21T13:59:24.585 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21172070400000001 2024-09-21T13:59:24.585 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.585 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.585 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:59:24.585 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.881915+0000", 2024-09-21T13:59:24.585 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.0762999999999997e-05 2024-09-21T13:59:24.585 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:24.585 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:24.585 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:24.585 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.586 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.586 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.15207.0:232032 8.e e240/183 ondisk, result = 0)", 2024-09-21T13:59:24.586 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.685964+0000", 2024-09-21T13:59:24.586 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 132.80296088099999, 2024-09-21T13:59:24.586 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.19633603699999999, 2024-09-21T13:59:24.586 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T13:59:24.586 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T13:59:24.586 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T13:59:24.586 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T13:59:24.586 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.586 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T13:59:24.586 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.685964+0000", 2024-09-21T13:59:24.586 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:24.586 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.586 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.586 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T13:59:24.587 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.685964+0000", 2024-09-21T13:59:24.587 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T13:59:24.587 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.587 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.587 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T13:59:24.587 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.685965+0000", 2024-09-21T13:59:24.587 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2890000000000001e-06 2024-09-21T13:59:24.587 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.587 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.587 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T13:59:24.587 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.685971+0000", 2024-09-21T13:59:24.587 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4280000000000004e-06 2024-09-21T13:59:24.587 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.587 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.587 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T13:59:24.587 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.881522+0000", 2024-09-21T13:59:24.588 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.19555102499999999 2024-09-21T13:59:24.588 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.588 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.588 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T13:59:24.588 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.881535+0000", 2024-09-21T13:59:24.588 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3157e-05 2024-09-21T13:59:24.588 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.588 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.588 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T13:59:24.588 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.882178+0000", 2024-09-21T13:59:24.588 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00064299300000000002 2024-09-21T13:59:24.588 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.588 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.588 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T13:59:24.588 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.882187+0000", 2024-09-21T13:59:24.589 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2790000000000004e-06 2024-09-21T13:59:24.589 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T13:59:24.589 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T13:59:24.589 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T13:59:24.589 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.882300+0000", 2024-09-21T13:59:24.589 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000112866 2024-09-21T13:59:24.589 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:24.589 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:24.589 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:24.589 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T13:59:24.589 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T13:59:24.589 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T13:59:24.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:24.591 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:24.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:59:24.633 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T13:59:24.682 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:2024-09-21T13:59:24.683818+0000 min lat: 0.00125866 max lat: 2.32137 avg lat: 0.0113455 2024-09-21T13:59:24.683 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T13:59:24.683 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 80 16 112816 112800 88.1043 74.5 0.0138168 0.0113455 2024-09-21T13:59:24.692 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:24.750 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:24.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:24.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:24.852 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:59:24.885 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:24.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:24.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:59:24.945 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 13:59:24 smithi187 podman[112940]: 2024-09-21 13:59:24.518792093 +0000 UTC m=+0.448524043 container init c1d12429d1b8305b6384fa8fe1242c1cf4de491ffc089af38043c0077038ac86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7, ceph=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-21-0842-4417849, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T13:59:24.945 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 13:59:24 smithi187 podman[112940]: 2024-09-21 13:59:24.525465925 +0000 UTC m=+0.455197852 container start c1d12429d1b8305b6384fa8fe1242c1cf4de491ffc089af38043c0077038ac86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=) 2024-09-21T13:59:24.945 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 13:59:24 smithi187 ceph-osd[113010]: -- 172.21.15.187:0/2503671974 <== mon.1 v2:172.21.15.187:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556f8a145a40 con 0x556f8af14400 2024-09-21T13:59:24.945 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 13:59:24 smithi187 bash[112940]: c1d12429d1b8305b6384fa8fe1242c1cf4de491ffc089af38043c0077038ac86 2024-09-21T13:59:24.945 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 13:59:24 smithi187 systemd[1]: Started Ceph osd.7 for 40a3d2e8-781f-11ef-baf6-efdc52797490. 2024-09-21T13:59:24.988 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:25.033 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:25.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:25.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:25.059 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:59:25.066 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T13:59:25.135 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:59:25.184 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:25.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:25.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:59:25.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:25 smithi049 ceph-mon[40489]: pgmap v523: 148 pgs: 1 active+clean+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 6.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 91 MiB/s wr, 1.46k op/s; 55281/329775 objects degraded (16.763%); 6744/329775 objects misplaced (2.045%) 2024-09-21T13:59:25.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:25 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 55281/329775 objects degraded (16.763%), 28 pgs degraded (PG_DEGRADED) 2024-09-21T13:59:25.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:25 smithi049 ceph-mon[30080]: pgmap v523: 148 pgs: 1 active+clean+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 6.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 91 MiB/s wr, 1.46k op/s; 55281/329775 objects degraded (16.763%); 6744/329775 objects misplaced (2.045%) 2024-09-21T13:59:25.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:25 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 55281/329775 objects degraded (16.763%), 28 pgs degraded (PG_DEGRADED) 2024-09-21T13:59:25.285 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:25.326 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:25.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:25.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:25.366 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:25 smithi187 ceph-mon[33032]: pgmap v523: 148 pgs: 1 active+clean+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 6.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 91 MiB/s wr, 1.46k op/s; 55281/329775 objects degraded (16.763%); 6744/329775 objects misplaced (2.045%) 2024-09-21T13:59:25.366 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:25 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 55281/329775 objects degraded (16.763%), 28 pgs degraded (PG_DEGRADED) 2024-09-21T13:59:25.430 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:25.458 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:25.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:25.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:25.560 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:25.599 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:25.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:25.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:25.637 INFO:teuthology.orchestra.run.smithi187.stderr:no valid command found; 10 closest matches: 2024-09-21T13:59:25.638 INFO:teuthology.orchestra.run.smithi187.stderr:0 2024-09-21T13:59:25.638 INFO:teuthology.orchestra.run.smithi187.stderr:1 2024-09-21T13:59:25.638 INFO:teuthology.orchestra.run.smithi187.stderr:2 2024-09-21T13:59:25.638 INFO:teuthology.orchestra.run.smithi187.stderr:abort 2024-09-21T13:59:25.638 INFO:teuthology.orchestra.run.smithi187.stderr:assert 2024-09-21T13:59:25.638 INFO:teuthology.orchestra.run.smithi187.stderr:config diff 2024-09-21T13:59:25.638 INFO:teuthology.orchestra.run.smithi187.stderr:config diff get 2024-09-21T13:59:25.638 INFO:teuthology.orchestra.run.smithi187.stderr:config get 2024-09-21T13:59:25.638 INFO:teuthology.orchestra.run.smithi187.stderr:config help [] 2024-09-21T13:59:25.638 INFO:teuthology.orchestra.run.smithi187.stderr:config set ... 2024-09-21T13:59:25.638 INFO:teuthology.orchestra.run.smithi187.stderr:admin_socket: invalid command 2024-09-21T13:59:25.683 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 81 16 113957 113941 87.8968 71.3125 0.00282651 0.0113635 2024-09-21T13:59:25.700 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:59:25.723 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:25.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:25.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:59:25.824 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T13:59:25.825 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:59:25.846 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:25.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:25.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:59:25.948 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:26.007 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:26.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:26.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:26.109 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:59:26.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:26.139 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:26.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:59:26.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:26 smithi187 ceph-mon[33032]: 9.0 scrub ok 2024-09-21T13:59:26.195 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 13:59:26 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T13:59:26.163+0000 7f548ba3a640 -1 Falling back to public interface 2024-09-21T13:59:26.240 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:59:26.286 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:26.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:26.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:59:26.323 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T13:59:26.323 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-7, ['dump_ops_in_flight'] 2024-09-21T13:59:26.388 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:26.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:26.426 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:26.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:26.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:26 smithi049 ceph-mon[30080]: 9.0 scrub ok 2024-09-21T13:59:26.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:26 smithi049 ceph-mon[40489]: 9.0 scrub ok 2024-09-21T13:59:26.527 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:26.551 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:26.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:26.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:26.654 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:59:26.683 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 82 16 114858 114842 87.5117 56.3125 0.00622995 0.0114223 2024-09-21T13:59:26.691 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:26.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:26.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:59:26.793 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:59:26.799 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T13:59:26.819 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:26.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:26.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:59:26.910 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:59:26.920 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:26.960 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:59:26.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:26.975 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:26.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:27.076 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:59:27.104 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:27.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:27.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:59:27.205 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:59:27.261 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:27.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:27.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:59:27.363 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:27.406 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:27.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:27.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:27.413 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:27 smithi049 ceph-mon[40489]: pgmap v524: 148 pgs: 1 active+clean+scrubbing, 1 active+clean+remapped, 33 active+undersized, 28 active+undersized+degraded, 85 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.3 KiB/s rd, 78 MiB/s wr, 1.25k op/s; 55623/332847 objects degraded (16.711%); 6744/332847 objects misplaced (2.026%) 2024-09-21T13:59:27.413 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:27 smithi049 ceph-mon[40489]: 4.10 scrub starts 2024-09-21T13:59:27.413 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:27 smithi049 ceph-mon[40489]: 4.10 scrub ok 2024-09-21T13:59:27.413 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:27 smithi049 ceph-mon[40489]: 6.4 scrub starts 2024-09-21T13:59:27.413 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:27 smithi049 ceph-mon[40489]: 6.4 scrub ok 2024-09-21T13:59:27.413 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:27 smithi049 ceph-mon[30080]: pgmap v524: 148 pgs: 1 active+clean+scrubbing, 1 active+clean+remapped, 33 active+undersized, 28 active+undersized+degraded, 85 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.3 KiB/s rd, 78 MiB/s wr, 1.25k op/s; 55623/332847 objects degraded (16.711%); 6744/332847 objects misplaced (2.026%) 2024-09-21T13:59:27.413 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:27 smithi049 ceph-mon[30080]: 4.10 scrub starts 2024-09-21T13:59:27.413 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:27 smithi049 ceph-mon[30080]: 4.10 scrub ok 2024-09-21T13:59:27.414 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:27 smithi049 ceph-mon[30080]: 6.4 scrub starts 2024-09-21T13:59:27.414 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:27 smithi049 ceph-mon[30080]: 6.4 scrub ok 2024-09-21T13:59:27.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:27 smithi187 ceph-mon[33032]: pgmap v524: 148 pgs: 1 active+clean+scrubbing, 1 active+clean+remapped, 33 active+undersized, 28 active+undersized+degraded, 85 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.3 KiB/s rd, 78 MiB/s wr, 1.25k op/s; 55623/332847 objects degraded (16.711%); 6744/332847 objects misplaced (2.026%) 2024-09-21T13:59:27.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:27 smithi187 ceph-mon[33032]: 4.10 scrub starts 2024-09-21T13:59:27.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:27 smithi187 ceph-mon[33032]: 4.10 scrub ok 2024-09-21T13:59:27.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:27 smithi187 ceph-mon[33032]: 6.4 scrub starts 2024-09-21T13:59:27.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:27 smithi187 ceph-mon[33032]: 6.4 scrub ok 2024-09-21T13:59:27.507 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:59:27.536 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:27.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:27.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:59:27.638 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:27.678 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:27.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:27.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:27.683 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 83 16 116070 116054 87.3699 75.75 0.00956486 0.0114414 2024-09-21T13:59:27.780 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:27.827 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:27.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:27.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:27.929 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:27.962 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:27.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:27.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:28.064 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:28.095 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:28.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:28.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:28.197 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:28.224 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:28.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:28.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:28.277 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:59:28.277 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:59:28.277 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:59:28.277 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:59:28.277 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:59:28.277 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:59:28.278 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:59:28.278 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:59:28.278 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:59:28.278 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:59:28.278 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:59:28.278 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:59:28.278 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:59:28.278 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:59:28.278 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T13:59:28.279 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:28 smithi049 ceph-mon[30080]: 3.3 scrub starts 2024-09-21T13:59:28.279 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:28 smithi049 ceph-mon[30080]: 3.3 scrub ok 2024-09-21T13:59:28.279 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:28 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1378062637' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:59:28.280 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:28 smithi049 ceph-mon[40489]: 3.3 scrub starts 2024-09-21T13:59:28.280 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:28 smithi049 ceph-mon[40489]: 3.3 scrub ok 2024-09-21T13:59:28.280 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:28 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1378062637' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:59:28.326 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:28.359 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:28.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:28.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:28.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:28 smithi187 ceph-mon[33032]: 3.3 scrub starts 2024-09-21T13:59:28.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:28 smithi187 ceph-mon[33032]: 3.3 scrub ok 2024-09-21T13:59:28.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:28 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1378062637' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:59:28.460 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:28.482 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:28.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:28.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:28.584 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:59:28.622 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:28.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:28.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:59:28.683 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 84 16 117220 117204 87.1853 71.875 0.0034804 0.0114612 2024-09-21T13:59:28.724 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:59:28.752 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:28.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:28.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:59:28.854 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:59:28.882 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:28.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:28.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:59:28.984 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:59:29.013 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:29.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:29.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:59:29.115 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:59:29.137 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:29.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:29.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:59:29.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:29 smithi187 ceph-mon[33032]: pgmap v525: 148 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing, 33 active+undersized, 28 active+undersized+degraded, 85 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 917 op/s; 56303/337986 objects degraded (16.658%); 7089/337986 objects misplaced (2.097%) 2024-09-21T13:59:29.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:29 smithi187 ceph-mon[33032]: 6.13 scrub starts 2024-09-21T13:59:29.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:29 smithi187 ceph-mon[33032]: 6.13 scrub ok 2024-09-21T13:59:29.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:29 smithi187 ceph-mon[33032]: 4.9 scrub starts 2024-09-21T13:59:29.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:29 smithi187 ceph-mon[33032]: 4.9 scrub ok 2024-09-21T13:59:29.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:29 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:59:29.239 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:29.276 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:29.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:29.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:29.378 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:59:29.387 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:29 smithi049 ceph-mon[30080]: pgmap v525: 148 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing, 33 active+undersized, 28 active+undersized+degraded, 85 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 917 op/s; 56303/337986 objects degraded (16.658%); 7089/337986 objects misplaced (2.097%) 2024-09-21T13:59:29.387 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:29 smithi049 ceph-mon[30080]: 6.13 scrub starts 2024-09-21T13:59:29.387 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:29 smithi049 ceph-mon[30080]: 6.13 scrub ok 2024-09-21T13:59:29.388 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:29 smithi049 ceph-mon[30080]: 4.9 scrub starts 2024-09-21T13:59:29.388 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:29 smithi049 ceph-mon[30080]: 4.9 scrub ok 2024-09-21T13:59:29.388 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:29 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:59:29.388 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:29 smithi049 ceph-mon[40489]: pgmap v525: 148 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing, 33 active+undersized, 28 active+undersized+degraded, 85 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 917 op/s; 56303/337986 objects degraded (16.658%); 7089/337986 objects misplaced (2.097%) 2024-09-21T13:59:29.388 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:29 smithi049 ceph-mon[40489]: 6.13 scrub starts 2024-09-21T13:59:29.388 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:29 smithi049 ceph-mon[40489]: 6.13 scrub ok 2024-09-21T13:59:29.388 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:29 smithi049 ceph-mon[40489]: 4.9 scrub starts 2024-09-21T13:59:29.388 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:29 smithi049 ceph-mon[40489]: 4.9 scrub ok 2024-09-21T13:59:29.388 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:29 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:59:29.405 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:29.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:29.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:59:29.506 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:59:29.538 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:29.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:29.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:59:29.570 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:59:29.640 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:59:29.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:29.669 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:29.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:59:29.683 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 85 16 118454 118438 87.0668 77.125 0.0178356 0.0114804 2024-09-21T13:59:29.770 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:59:29.797 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:29.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:29.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:59:29.899 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:59:29.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:29.933 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:29.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:59:29.944 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:59:29 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:59:30.034 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:59:30.061 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:30.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:30.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:59:30.144 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-09-21T13:59:30.162 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:30.186 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T13:59:30.187 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T13:59:30.197 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:30.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:30.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:30.299 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:59:30.330 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:30.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:30.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:59:30.432 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:59:30.444 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 13:59:30 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T13:59:30.000+0000 7f548ba3a640 -1 osd.7 272 log_to_monitors true 2024-09-21T13:59:30.444 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 13:59:30 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T13:59:30.190+0000 7f54837e4640 -1 osd.7 272 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T13:59:30.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:30 smithi187 ceph-mon[33032]: 4.6 scrub starts 2024-09-21T13:59:30.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:30 smithi187 ceph-mon[33032]: 4.6 scrub ok 2024-09-21T13:59:30.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:30 smithi187 ceph-mon[33032]: 3.0 scrub starts 2024-09-21T13:59:30.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:30 smithi187 ceph-mon[33032]: 3.0 scrub ok 2024-09-21T13:59:30.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:30 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:59:30.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:30 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3630352436' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:59:30.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:30 smithi187 ceph-mon[33032]: from='osd.7 [v2:172.21.15.187:6824/13894964,v1:172.21.15.187:6825/13894964]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-21T13:59:30.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:30 smithi187 ceph-mon[33032]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-21T13:59:30.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:30 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 56303/337986 objects degraded (16.658%), 28 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-09-21T13:59:30.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:30.463 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:30.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:59:30.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:30 smithi049 ceph-mon[30080]: 4.6 scrub starts 2024-09-21T13:59:30.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:30 smithi049 ceph-mon[30080]: 4.6 scrub ok 2024-09-21T13:59:30.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:30 smithi049 ceph-mon[30080]: 3.0 scrub starts 2024-09-21T13:59:30.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:30 smithi049 ceph-mon[30080]: 3.0 scrub ok 2024-09-21T13:59:30.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:30 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:59:30.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:30 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3630352436' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:59:30.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:30 smithi049 ceph-mon[30080]: from='osd.7 [v2:172.21.15.187:6824/13894964,v1:172.21.15.187:6825/13894964]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-21T13:59:30.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:30 smithi049 ceph-mon[30080]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-21T13:59:30.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:30 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 56303/337986 objects degraded (16.658%), 28 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-09-21T13:59:30.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:30 smithi049 ceph-mon[40489]: 4.6 scrub starts 2024-09-21T13:59:30.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:30 smithi049 ceph-mon[40489]: 4.6 scrub ok 2024-09-21T13:59:30.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:30 smithi049 ceph-mon[40489]: 3.0 scrub starts 2024-09-21T13:59:30.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:30 smithi049 ceph-mon[40489]: 3.0 scrub ok 2024-09-21T13:59:30.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:30 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:59:30.507 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:30 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3630352436' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:59:30.507 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:30 smithi049 ceph-mon[40489]: from='osd.7 [v2:172.21.15.187:6824/13894964,v1:172.21.15.187:6825/13894964]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-21T13:59:30.507 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:30 smithi049 ceph-mon[40489]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-21T13:59:30.507 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:30 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 56303/337986 objects degraded (16.658%), 28 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-09-21T13:59:30.563 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:30.604 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:30.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:30.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:30.683 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 86 16 119759 119743 87.0027 81.5625 0.0100442 0.0114893 2024-09-21T13:59:30.706 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:59:30.736 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:30.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:30.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:59:30.838 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:59:30.868 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:30.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:30.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:59:30.901 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:59:30 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:59:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:59:30.970 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:59:30.999 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:31.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:31.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:59:31.101 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:31.144 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:31.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:31.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:31.171 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:31 smithi049 ceph-mon[40489]: pgmap v526: 148 pgs: 1 active+clean+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 67 MiB/s wr, 1.07k op/s; 58444/349734 objects degraded (16.711%); 7089/349734 objects misplaced (2.027%) 2024-09-21T13:59:31.171 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:31 smithi049 ceph-mon[30080]: pgmap v526: 148 pgs: 1 active+clean+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 67 MiB/s wr, 1.07k op/s; 58444/349734 objects degraded (16.711%); 7089/349734 objects misplaced (2.027%) 2024-09-21T13:59:31.246 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:59:31.267 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:31.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:31.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:59:31.325 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T13:59:31.369 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:31.393 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:31.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:31.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:31.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:31 smithi187 ceph-mon[33032]: pgmap v526: 148 pgs: 1 active+clean+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 67 MiB/s wr, 1.07k op/s; 58444/349734 objects degraded (16.711%); 7089/349734 objects misplaced (2.027%) 2024-09-21T13:59:31.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:31 smithi187 ceph-mon[33032]: 9.b scrub starts 2024-09-21T13:59:31.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:31 smithi187 ceph-mon[33032]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T13:59:31.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:31 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3630352436' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T13:59:31.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:31 smithi187 ceph-mon[33032]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-21T13:59:31.446 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:31 smithi187 ceph-mon[33032]: from='osd.7 [v2:172.21.15.187:6824/13894964,v1:172.21.15.187:6825/13894964]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-21T13:59:31.446 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:31 smithi187 ceph-mon[33032]: osdmap e297: 8 total, 7 up, 8 in 2024-09-21T13:59:31.446 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:31 smithi187 ceph-mon[33032]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-21T13:59:31.495 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:31.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:31 smithi049 ceph-mon[40489]: 9.b scrub starts 2024-09-21T13:59:31.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:31 smithi049 ceph-mon[40489]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T13:59:31.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:31 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3630352436' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T13:59:31.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:31 smithi049 ceph-mon[40489]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-21T13:59:31.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:31 smithi049 ceph-mon[40489]: from='osd.7 [v2:172.21.15.187:6824/13894964,v1:172.21.15.187:6825/13894964]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-21T13:59:31.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:31 smithi049 ceph-mon[40489]: osdmap e297: 8 total, 7 up, 8 in 2024-09-21T13:59:31.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:31 smithi049 ceph-mon[40489]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-21T13:59:31.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:31 smithi049 ceph-mon[30080]: 9.b scrub starts 2024-09-21T13:59:31.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:31 smithi049 ceph-mon[30080]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T13:59:31.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:31 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3630352436' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T13:59:31.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:31 smithi049 ceph-mon[30080]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-21T13:59:31.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:31 smithi049 ceph-mon[30080]: from='osd.7 [v2:172.21.15.187:6824/13894964,v1:172.21.15.187:6825/13894964]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-21T13:59:31.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:31 smithi049 ceph-mon[30080]: osdmap e297: 8 total, 7 up, 8 in 2024-09-21T13:59:31.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:31 smithi049 ceph-mon[30080]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-21T13:59:31.518 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:31.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:31.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:31.582 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:59:31.620 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:31.641 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:31.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:31.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:31.686 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 87 16 120386 120370 86.4531 39.1875 0.00329976 0.0114991 2024-09-21T13:59:31.743 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:59:31.765 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:31.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:31.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:59:31.867 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:59:31.888 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:31.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:31.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:59:31.990 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:59:32.011 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:32.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:32.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:59:32.077 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T13:59:32.077 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-21T13:59:32.077 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.077 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:120352 9.3 9.63b6fcb3 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-21T13:59:32.077 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:59:31.197504+0000", 2024-09-21T13:59:32.077 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.87865563800000002, 2024-09-21T13:59:32.077 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.87867357400000001, 2024-09-21T13:59:32.077 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:59:32.078 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:59:32.078 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for peered", 2024-09-21T13:59:32.078 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:59:32.078 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T13:59:32.078 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T13:59:32.078 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 120352 2024-09-21T13:59:32.078 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.078 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:59:32.078 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.078 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:59:32.078 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:31.197504+0000", 2024-09-21T13:59:32.079 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:32.079 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.079 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.079 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:59:32.079 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:31.197504+0000", 2024-09-21T13:59:32.079 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:32.079 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.079 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.079 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:59:32.079 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:31.197506+0000", 2024-09-21T13:59:32.079 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0829999999999998e-06 2024-09-21T13:59:32.079 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.079 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.079 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:59:32.080 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:31.197570+0000", 2024-09-21T13:59:32.080 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3708999999999996e-05 2024-09-21T13:59:32.080 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.080 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.080 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:59:32.080 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:31.197582+0000", 2024-09-21T13:59:32.080 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2206e-05 2024-09-21T13:59:32.080 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.080 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.080 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:59:32.080 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:31.197593+0000", 2024-09-21T13:59:32.080 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.024e-05 2024-09-21T13:59:32.080 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.080 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.081 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:59:32.081 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:31.208489+0000", 2024-09-21T13:59:32.081 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.010896029 2024-09-21T13:59:32.081 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.081 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.081 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:59:32.081 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:31.208497+0000", 2024-09-21T13:59:32.081 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5580000000000001e-06 2024-09-21T13:59:32.081 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:32.081 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:59:32.081 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:32.081 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.081 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.081 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:120363 9.3 9.cf7c6713 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e298)", 2024-09-21T13:59:32.081 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:59:31.197981+0000", 2024-09-21T13:59:32.081 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.87817933500000001, 2024-09-21T13:59:32.082 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.87824003500000003, 2024-09-21T13:59:32.082 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:59:32.082 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:59:32.082 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for peered", 2024-09-21T13:59:32.082 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:59:32.082 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T13:59:32.082 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T13:59:32.082 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 120363 2024-09-21T13:59:32.082 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.082 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:59:32.082 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.082 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:59:32.082 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:31.197981+0000", 2024-09-21T13:59:32.082 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:32.083 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.083 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.083 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:59:32.083 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:31.197981+0000", 2024-09-21T13:59:32.083 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:32.083 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.083 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.083 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:59:32.083 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:31.197982+0000", 2024-09-21T13:59:32.083 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.198e-06 2024-09-21T13:59:32.083 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.083 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.083 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:59:32.084 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:31.198052+0000", 2024-09-21T13:59:32.084 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.0531000000000006e-05 2024-09-21T13:59:32.084 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.084 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.084 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:59:32.084 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:31.198072+0000", 2024-09-21T13:59:32.084 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9758999999999998e-05 2024-09-21T13:59:32.084 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.084 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.084 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:59:32.084 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:31.198086+0000", 2024-09-21T13:59:32.084 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3723999999999999e-05 2024-09-21T13:59:32.084 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.084 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.084 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:59:32.085 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:31.208524+0000", 2024-09-21T13:59:32.085 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.010437979 2024-09-21T13:59:32.085 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.085 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.085 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:59:32.085 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:31.208530+0000", 2024-09-21T13:59:32.085 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0920000000000003e-06 2024-09-21T13:59:32.085 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:32.085 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:59:32.085 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:32.085 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.085 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.085 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:120377 9.9 9.d2151099 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-21T13:59:32.085 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:59:31.206582+0000", 2024-09-21T13:59:32.085 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.869577779, 2024-09-21T13:59:32.086 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.869668836, 2024-09-21T13:59:32.086 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:59:32.086 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:59:32.086 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for peered", 2024-09-21T13:59:32.086 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:59:32.086 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T13:59:32.086 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T13:59:32.086 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 120377 2024-09-21T13:59:32.086 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.086 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:59:32.086 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.086 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:59:32.086 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:31.206582+0000", 2024-09-21T13:59:32.086 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:32.087 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.087 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.087 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:59:32.087 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:31.206582+0000", 2024-09-21T13:59:32.087 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:32.087 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.087 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.087 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:59:32.087 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:31.206584+0000", 2024-09-21T13:59:32.087 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.578e-06 2024-09-21T13:59:32.087 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.087 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.087 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:59:32.087 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:31.206639+0000", 2024-09-21T13:59:32.087 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4945000000000003e-05 2024-09-21T13:59:32.088 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.088 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.088 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:59:32.088 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:31.206650+0000", 2024-09-21T13:59:32.088 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1263999999999999e-05 2024-09-21T13:59:32.088 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.088 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.088 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:59:32.088 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:31.206659+0000", 2024-09-21T13:59:32.088 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.9849999999999995e-06 2024-09-21T13:59:32.088 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.088 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.088 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:59:32.088 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:31.214398+0000", 2024-09-21T13:59:32.089 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0077386579999999998 2024-09-21T13:59:32.089 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.089 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.089 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:59:32.089 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:31.214406+0000", 2024-09-21T13:59:32.089 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5949999999999999e-06 2024-09-21T13:59:32.089 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:32.089 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:59:32.089 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:32.089 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.089 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.089 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:120378 9.e 9.b7ba179e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-21T13:59:32.089 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:59:31.210931+0000", 2024-09-21T13:59:32.089 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.86522903600000001, 2024-09-21T13:59:32.089 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.86534785599999997, 2024-09-21T13:59:32.090 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:59:32.090 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:59:32.090 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for peered", 2024-09-21T13:59:32.090 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:59:32.090 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T13:59:32.090 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T13:59:32.090 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 120378 2024-09-21T13:59:32.090 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.090 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:59:32.090 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.090 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:59:32.090 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:31.210931+0000", 2024-09-21T13:59:32.091 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:32.091 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.091 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.091 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:59:32.091 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:31.210931+0000", 2024-09-21T13:59:32.091 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:32.091 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.091 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.091 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:59:32.091 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:31.210933+0000", 2024-09-21T13:59:32.091 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5999999999999999e-06 2024-09-21T13:59:32.091 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.091 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.091 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:59:32.092 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:31.210987+0000", 2024-09-21T13:59:32.092 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4318999999999999e-05 2024-09-21T13:59:32.092 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.092 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.092 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:59:32.092 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:31.210998+0000", 2024-09-21T13:59:32.092 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0883000000000001e-05 2024-09-21T13:59:32.092 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.092 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.092 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:59:32.092 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:31.211008+0000", 2024-09-21T13:59:32.092 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0256e-05 2024-09-21T13:59:32.092 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.093 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.093 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:59:32.093 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:31.212296+0000", 2024-09-21T13:59:32.093 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001287503 2024-09-21T13:59:32.093 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.093 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.093 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:59:32.093 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:31.212440+0000", 2024-09-21T13:59:32.093 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014470600000000001 2024-09-21T13:59:32.093 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:32.093 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:59:32.093 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:32.093 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.093 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.094 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:120379 9.e 9.ab23a8fe (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-21T13:59:32.094 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:59:31.211020+0000", 2024-09-21T13:59:32.094 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.865140507, 2024-09-21T13:59:32.094 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.86528751199999998, 2024-09-21T13:59:32.094 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:59:32.094 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:59:32.094 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for peered", 2024-09-21T13:59:32.094 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:59:32.094 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T13:59:32.094 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T13:59:32.094 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 120379 2024-09-21T13:59:32.094 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.094 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:59:32.095 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.095 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:59:32.095 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:31.211020+0000", 2024-09-21T13:59:32.095 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:32.095 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.095 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.095 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:59:32.095 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:31.211020+0000", 2024-09-21T13:59:32.095 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:32.095 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.095 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.095 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:59:32.095 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:31.211020+0000", 2024-09-21T13:59:32.095 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5700000000000002e-07 2024-09-21T13:59:32.096 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.096 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.096 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:59:32.096 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:31.211071+0000", 2024-09-21T13:59:32.097 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0640999999999998e-05 2024-09-21T13:59:32.097 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.097 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.097 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:59:32.097 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:31.211079+0000", 2024-09-21T13:59:32.098 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0169999999999996e-06 2024-09-21T13:59:32.098 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.098 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.098 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:59:32.098 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:31.211085+0000", 2024-09-21T13:59:32.098 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9560000000000002e-06 2024-09-21T13:59:32.098 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.098 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.098 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:59:32.098 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:31.212573+0000", 2024-09-21T13:59:32.098 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001488627 2024-09-21T13:59:32.098 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.098 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.098 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:59:32.099 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:31.212585+0000", 2024-09-21T13:59:32.099 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1799e-05 2024-09-21T13:59:32.099 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:32.099 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:59:32.099 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:32.099 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.099 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.099 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:120386 9.3 9.64797f73 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-09-21T13:59:32.099 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:59:31.399147+0000", 2024-09-21T13:59:32.099 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.67701323999999996, 2024-09-21T13:59:32.099 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.67718781400000005, 2024-09-21T13:59:32.099 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:59:32.099 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:59:32.099 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for peered", 2024-09-21T13:59:32.099 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:59:32.100 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T13:59:32.100 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T13:59:32.100 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 120386 2024-09-21T13:59:32.100 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.100 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:59:32.100 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.100 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:59:32.100 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:31.399147+0000", 2024-09-21T13:59:32.100 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:32.100 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.100 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.100 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:59:32.100 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:31.399147+0000", 2024-09-21T13:59:32.100 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:32.100 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.101 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.101 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:59:32.101 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:31.399149+0000", 2024-09-21T13:59:32.101 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.77e-06 2024-09-21T13:59:32.101 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.101 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.101 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:59:32.101 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:31.399217+0000", 2024-09-21T13:59:32.101 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7999999999999999e-05 2024-09-21T13:59:32.101 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.101 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.101 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:59:32.101 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:31.399233+0000", 2024-09-21T13:59:32.101 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6141e-05 2024-09-21T13:59:32.101 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.102 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.102 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:59:32.102 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:31.399240+0000", 2024-09-21T13:59:32.102 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6450000000000005e-06 2024-09-21T13:59:32.102 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.102 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.102 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:59:32.102 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:31.399309+0000", 2024-09-21T13:59:32.102 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8410000000000004e-05 2024-09-21T13:59:32.102 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.102 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.102 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:59:32.102 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:31.399322+0000", 2024-09-21T13:59:32.102 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3404000000000001e-05 2024-09-21T13:59:32.103 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:32.103 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:59:32.103 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:32.103 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.103 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.103 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24442.0:622 2.0 2.9f137cac (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e298)", 2024-09-21T13:59:32.103 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:59:32.059200+0000", 2024-09-21T13:59:32.103 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.016960419000000001, 2024-09-21T13:59:32.103 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.017162402, 2024-09-21T13:59:32.103 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:59:32.103 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:59:32.103 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for peered", 2024-09-21T13:59:32.103 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:59:32.104 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24442", 2024-09-21T13:59:32.104 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3747604758", 2024-09-21T13:59:32.104 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 622 2024-09-21T13:59:32.104 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.104 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:59:32.104 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.104 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:59:32.104 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:32.059200+0000", 2024-09-21T13:59:32.104 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:32.104 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.104 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.104 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:59:32.104 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:32.059200+0000", 2024-09-21T13:59:32.104 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:32.104 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.105 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.105 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:59:32.105 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:32.059201+0000", 2024-09-21T13:59:32.105 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8199999999999999e-06 2024-09-21T13:59:32.105 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.105 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.105 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:59:32.105 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:32.059208+0000", 2024-09-21T13:59:32.105 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5370000000000003e-06 2024-09-21T13:59:32.105 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.105 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.105 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:59:32.105 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:32.059218+0000", 2024-09-21T13:59:32.105 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0087e-05 2024-09-21T13:59:32.106 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.106 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.106 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:59:32.106 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:32.059227+0000", 2024-09-21T13:59:32.106 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8529999999999998e-06 2024-09-21T13:59:32.106 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.106 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.106 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:59:32.106 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:32.059300+0000", 2024-09-21T13:59:32.106 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3448000000000006e-05 2024-09-21T13:59:32.106 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:32.106 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:32.106 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:59:32.106 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:32.059312+0000", 2024-09-21T13:59:32.107 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1829e-05 2024-09-21T13:59:32.107 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:32.107 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:59:32.107 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:32.107 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:32.107 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2024-09-21T13:59:32.107 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 7 2024-09-21T13:59:32.107 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T13:59:32.107 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T13:59:32.108 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:59:32.113 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:32.138 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:32.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:32.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:32.188 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T13:59:32.240 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:59:32.263 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:32.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:32.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:59:32.364 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:59:32.389 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:32.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:32.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:59:32.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:32 smithi187 ceph-mon[33032]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T13:59:32.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:32 smithi187 ceph-mon[33032]: osd.7 [v2:172.21.15.187:6824/13894964,v1:172.21.15.187:6825/13894964] boot 2024-09-21T13:59:32.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:32 smithi187 ceph-mon[33032]: osdmap e298: 8 total, 8 up, 8 in 2024-09-21T13:59:32.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:32 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T13:59:32.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:32 smithi187 ceph-mon[33032]: 6.e scrub starts 2024-09-21T13:59:32.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:32 smithi187 ceph-mon[33032]: 6.e scrub ok 2024-09-21T13:59:32.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:32 smithi187 ceph-mon[33032]: 9.b scrub ok 2024-09-21T13:59:32.475 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:32 smithi049 ceph-mon[40489]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T13:59:32.475 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:32 smithi049 ceph-mon[40489]: osd.7 [v2:172.21.15.187:6824/13894964,v1:172.21.15.187:6825/13894964] boot 2024-09-21T13:59:32.475 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:32 smithi049 ceph-mon[40489]: osdmap e298: 8 total, 8 up, 8 in 2024-09-21T13:59:32.476 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:32 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T13:59:32.476 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:32 smithi049 ceph-mon[40489]: 6.e scrub starts 2024-09-21T13:59:32.476 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:32 smithi049 ceph-mon[40489]: 6.e scrub ok 2024-09-21T13:59:32.476 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:32 smithi049 ceph-mon[40489]: 9.b scrub ok 2024-09-21T13:59:32.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:32 smithi049 ceph-mon[30080]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T13:59:32.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:32 smithi049 ceph-mon[30080]: osd.7 [v2:172.21.15.187:6824/13894964,v1:172.21.15.187:6825/13894964] boot 2024-09-21T13:59:32.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:32 smithi049 ceph-mon[30080]: osdmap e298: 8 total, 8 up, 8 in 2024-09-21T13:59:32.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:32 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T13:59:32.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:32 smithi049 ceph-mon[30080]: 6.e scrub starts 2024-09-21T13:59:32.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:32 smithi049 ceph-mon[30080]: 6.e scrub ok 2024-09-21T13:59:32.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:32 smithi049 ceph-mon[30080]: 9.b scrub ok 2024-09-21T13:59:32.491 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:59:32.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:32.513 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:32.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:59:32.614 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:32.635 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:32.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:32.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:32.684 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 88 16 120392 120376 85.4751 0.375 0.0033046 0.0115168 2024-09-21T13:59:32.737 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:32.762 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:32.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:32.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:32.852 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T13:59:32.864 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:59:32.885 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:32.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:32.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:59:32.987 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:59:33.008 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:33.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:33.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:59:33.063 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:59:33.110 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:59:33.134 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:33.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:33.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:59:33.236 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:33.258 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:33.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:33.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:33.360 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:59:33.385 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:33.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:33.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:59:33.487 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:33.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:33 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:59:33.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:33 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:59:33.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:33 smithi049 ceph-mon[40489]: pgmap v529: 148 pgs: 1 active+clean+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 7.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 67 MiB/s wr, 1.07k op/s; 59206/355401 objects degraded (16.659%); 7458/355401 objects misplaced (2.098%) 2024-09-21T13:59:33.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:33 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:59:33.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:33 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:59:33.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:33 smithi049 ceph-mon[40489]: 5.15 scrub starts 2024-09-21T13:59:33.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:33 smithi049 ceph-mon[40489]: 5.15 scrub ok 2024-09-21T13:59:33.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:33 smithi049 ceph-mon[40489]: osdmap e299: 8 total, 8 up, 8 in 2024-09-21T13:59:33.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:33 smithi049 ceph-mon[40489]: 9.1 starting backfill to osd.7 from (0'0,272'2312] MAX to 297'7480 2024-09-21T13:59:33.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:33 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:59:33.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:33 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:59:33.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:33 smithi049 ceph-mon[30080]: pgmap v529: 148 pgs: 1 active+clean+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 7.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 67 MiB/s wr, 1.07k op/s; 59206/355401 objects degraded (16.659%); 7458/355401 objects misplaced (2.098%) 2024-09-21T13:59:33.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:33 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:59:33.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:33 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:59:33.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:33 smithi049 ceph-mon[30080]: 5.15 scrub starts 2024-09-21T13:59:33.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:33 smithi049 ceph-mon[30080]: 5.15 scrub ok 2024-09-21T13:59:33.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:33 smithi049 ceph-mon[30080]: osdmap e299: 8 total, 8 up, 8 in 2024-09-21T13:59:33.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:33 smithi049 ceph-mon[30080]: 9.1 starting backfill to osd.7 from (0'0,272'2312] MAX to 297'7480 2024-09-21T13:59:33.527 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:33.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:33.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:33.630 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:33.678 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:33.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:33.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:33.684 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 89 16 120640 120624 84.689 15.5 0.0269615 0.011803 2024-09-21T13:59:33.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:33 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:59:33.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:33 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:59:33.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:33 smithi187 ceph-mon[33032]: pgmap v529: 148 pgs: 1 active+clean+remapped, 33 active+undersized, 28 active+undersized+degraded, 86 active+clean; 7.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 67 MiB/s wr, 1.07k op/s; 59206/355401 objects degraded (16.659%); 7458/355401 objects misplaced (2.098%) 2024-09-21T13:59:33.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:33 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:59:33.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:33 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:59:33.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:33 smithi187 ceph-mon[33032]: 5.15 scrub starts 2024-09-21T13:59:33.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:33 smithi187 ceph-mon[33032]: 5.15 scrub ok 2024-09-21T13:59:33.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:33 smithi187 ceph-mon[33032]: osdmap e299: 8 total, 8 up, 8 in 2024-09-21T13:59:33.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:33 smithi187 ceph-mon[33032]: 9.1 starting backfill to osd.7 from (0'0,272'2312] MAX to 297'7480 2024-09-21T13:59:33.738 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:59:33.738 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:59:33.777 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:59:33.778 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:59:33.780 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:33.813 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:59:33.813 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:59:33.827 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:59:33.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:33.834 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:33.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:33.850 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:59:33.850 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:59:33.878 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:59:33.878 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:59:33.913 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:59:33.913 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:59:33.937 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:33.944 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:59:33.944 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:59:33.977 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:33.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:33.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:33.980 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:59:33.980 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:59:34.079 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:34.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:34.103 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:34.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:34.204 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:59:34.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:34.247 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:34.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:59:34.348 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:59:34.385 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:34.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:34.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:59:34.487 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:34.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:34 smithi049 ceph-mon[40489]: 4.8 scrub starts 2024-09-21T13:59:34.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:34 smithi049 ceph-mon[40489]: 4.8 scrub ok 2024-09-21T13:59:34.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:34 smithi049 ceph-mon[40489]: 9.d starting backfill to osd.7 from (0'0,272'2312] MAX to 297'7529 2024-09-21T13:59:34.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:34 smithi049 ceph-mon[40489]: 9.3 starting backfill to osd.7 from (0'0,272'2328] MAX to 297'7527 2024-09-21T13:59:34.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:34 smithi049 ceph-mon[40489]: osdmap e300: 8 total, 8 up, 8 in 2024-09-21T13:59:34.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:34 smithi049 ceph-mon[40489]: 9.e starting backfill to osd.7 from (0'0,272'2295] MAX to 297'7398 2024-09-21T13:59:34.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:34 smithi049 ceph-mon[40489]: 9.c starting backfill to osd.7 from (0'0,272'2299] MAX to 297'7465 2024-09-21T13:59:34.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:34 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/882134927' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:59:34.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:34 smithi049 ceph-mon[30080]: 4.8 scrub starts 2024-09-21T13:59:34.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:34 smithi049 ceph-mon[30080]: 4.8 scrub ok 2024-09-21T13:59:34.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:34 smithi049 ceph-mon[30080]: 9.d starting backfill to osd.7 from (0'0,272'2312] MAX to 297'7529 2024-09-21T13:59:34.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:34 smithi049 ceph-mon[30080]: 9.3 starting backfill to osd.7 from (0'0,272'2328] MAX to 297'7527 2024-09-21T13:59:34.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:34 smithi049 ceph-mon[30080]: osdmap e300: 8 total, 8 up, 8 in 2024-09-21T13:59:34.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:34 smithi049 ceph-mon[30080]: 9.e starting backfill to osd.7 from (0'0,272'2295] MAX to 297'7398 2024-09-21T13:59:34.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:34 smithi049 ceph-mon[30080]: 9.c starting backfill to osd.7 from (0'0,272'2299] MAX to 297'7465 2024-09-21T13:59:34.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:34 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/882134927' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:59:34.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:34.530 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:34.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:34.632 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:59:34.672 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:34.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:34.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:59:34.684 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 90 6 121405 121399 84.2862 48.4375 0.0408087 0.0118579 2024-09-21T13:59:34.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:34 smithi187 ceph-mon[33032]: 4.8 scrub starts 2024-09-21T13:59:34.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:34 smithi187 ceph-mon[33032]: 4.8 scrub ok 2024-09-21T13:59:34.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:34 smithi187 ceph-mon[33032]: 9.d starting backfill to osd.7 from (0'0,272'2312] MAX to 297'7529 2024-09-21T13:59:34.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:34 smithi187 ceph-mon[33032]: 9.3 starting backfill to osd.7 from (0'0,272'2328] MAX to 297'7527 2024-09-21T13:59:34.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:34 smithi187 ceph-mon[33032]: osdmap e300: 8 total, 8 up, 8 in 2024-09-21T13:59:34.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:34 smithi187 ceph-mon[33032]: 9.e starting backfill to osd.7 from (0'0,272'2295] MAX to 297'7398 2024-09-21T13:59:34.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:34 smithi187 ceph-mon[33032]: 9.c starting backfill to osd.7 from (0'0,272'2299] MAX to 297'7465 2024-09-21T13:59:34.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:34 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/882134927' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:59:34.774 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:34.797 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:34.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:34.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:34.899 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:59:34.921 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:34.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:34.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:59:35.023 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:35.047 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:35.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:35.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:35.068 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T13:59:35.149 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:35.172 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:35.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:35.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:35.273 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:59:35.295 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:35.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:35.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:59:35.397 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:35.420 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:35.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:35.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:35.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:35 smithi049 ceph-mon[40489]: pgmap v532: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 25 active+undersized, 18 active+undersized+degraded, 101 active+clean; 7.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 50 MiB/s wr, 796 op/s; 52538/359292 objects degraded (14.623%); 7458/359292 objects misplaced (2.076%); 128 KiB/s, 8 objects/s recovering 2024-09-21T13:59:35.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:35 smithi049 ceph-mon[40489]: 5.1 scrub starts 2024-09-21T13:59:35.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:35 smithi049 ceph-mon[40489]: 5.1 scrub ok 2024-09-21T13:59:35.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:35 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 52538/359292 objects degraded (14.623%), 21 pgs degraded, 43 pgs undersized (PG_DEGRADED) 2024-09-21T13:59:35.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:35 smithi049 ceph-mon[30080]: pgmap v532: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 25 active+undersized, 18 active+undersized+degraded, 101 active+clean; 7.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 50 MiB/s wr, 796 op/s; 52538/359292 objects degraded (14.623%); 7458/359292 objects misplaced (2.076%); 128 KiB/s, 8 objects/s recovering 2024-09-21T13:59:35.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:35 smithi049 ceph-mon[30080]: 5.1 scrub starts 2024-09-21T13:59:35.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:35 smithi049 ceph-mon[30080]: 5.1 scrub ok 2024-09-21T13:59:35.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:35 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 52538/359292 objects degraded (14.623%), 21 pgs degraded, 43 pgs undersized (PG_DEGRADED) 2024-09-21T13:59:35.521 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:59:35.544 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:35.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:35.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:59:35.645 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:35.668 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:35.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:35.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:35.684 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Total time run: 90.0336 2024-09-21T13:59:35.684 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Total writes made: 121405 2024-09-21T13:59:35.684 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Write size: 65536 2024-09-21T13:59:35.684 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Object size: 65536 2024-09-21T13:59:35.684 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Bandwidth (MB/sec): 84.2776 2024-09-21T13:59:35.684 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Stddev Bandwidth: 37.5283 2024-09-21T13:59:35.684 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Max bandwidth (MB/sec): 202.062 2024-09-21T13:59:35.684 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Min bandwidth (MB/sec): 0 2024-09-21T13:59:35.685 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Average IOPS: 1348 2024-09-21T13:59:35.685 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Stddev IOPS: 600.461 2024-09-21T13:59:35.685 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Max IOPS: 3233 2024-09-21T13:59:35.685 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Min IOPS: 0 2024-09-21T13:59:35.685 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Average Latency(s): 0.0118594 2024-09-21T13:59:35.685 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Stddev Latency(s): 0.0427544 2024-09-21T13:59:35.685 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Max latency(s): 2.32137 2024-09-21T13:59:35.685 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Min latency(s): 0.00125866 2024-09-21T13:59:35.688 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Cleaning up (deleting benchmark objects) 2024-09-21T13:59:35.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:35 smithi187 ceph-mon[33032]: pgmap v532: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 25 active+undersized, 18 active+undersized+degraded, 101 active+clean; 7.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 50 MiB/s wr, 796 op/s; 52538/359292 objects degraded (14.623%); 7458/359292 objects misplaced (2.076%); 128 KiB/s, 8 objects/s recovering 2024-09-21T13:59:35.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:35 smithi187 ceph-mon[33032]: 5.1 scrub starts 2024-09-21T13:59:35.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:35 smithi187 ceph-mon[33032]: 5.1 scrub ok 2024-09-21T13:59:35.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:35 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 52538/359292 objects degraded (14.623%), 21 pgs degraded, 43 pgs undersized (PG_DEGRADED) 2024-09-21T13:59:35.770 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:59:35.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:35.807 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:35.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:59:35.908 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:59:35.948 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:35.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:35.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:59:36.050 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:59:36.084 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:36.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:36.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:59:36.187 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:36.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:36.234 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:36.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:36.335 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:59:36.364 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:36 smithi049 ceph-mon[40489]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T13:59:36.364 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:36 smithi049 ceph-mon[30080]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T13:59:36.374 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:36.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:36.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:59:36.419 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T13:59:36.419 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-09-21T13:59:36.419 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-09-21T13:59:36.419 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-09-21T13:59:36.419 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T13:59:36.476 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:59:36.512 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:36.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:36.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:59:36.613 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:36.646 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:36.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:36.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:36.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:36 smithi187 ceph-mon[33032]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T13:59:36.748 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:36.787 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:36.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:36.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:36.889 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:36.917 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:36.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:36.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:37.018 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:59:37.053 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:37.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:37.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:59:37.155 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:59:37.185 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T13:59:37.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:37.199 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:37.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:59:37.266 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:59:37.300 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:59:37.331 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:37.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:37.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:59:37.400 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:37 smithi049 ceph-mon[40489]: pgmap v533: 148 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 7 active+undersized+degraded, 127 active+clean; 7.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 47 MiB/s wr, 745 op/s; 50928/362646 objects degraded (14.043%); 7458/362646 objects misplaced (2.057%); 200 KiB/s, 10 objects/s recovering 2024-09-21T13:59:37.400 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:37 smithi049 ceph-mon[40489]: 5.1f scrub starts 2024-09-21T13:59:37.400 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:37 smithi049 ceph-mon[40489]: 5.1f scrub ok 2024-09-21T13:59:37.400 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:37 smithi049 ceph-mon[30080]: pgmap v533: 148 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 7 active+undersized+degraded, 127 active+clean; 7.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 47 MiB/s wr, 745 op/s; 50928/362646 objects degraded (14.043%); 7458/362646 objects misplaced (2.057%); 200 KiB/s, 10 objects/s recovering 2024-09-21T13:59:37.400 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:37 smithi049 ceph-mon[30080]: 5.1f scrub starts 2024-09-21T13:59:37.400 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:37 smithi049 ceph-mon[30080]: 5.1f scrub ok 2024-09-21T13:59:37.433 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:37.463 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:37.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:37.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:37.565 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:37.600 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:37.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:37.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:37.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:37 smithi187 ceph-mon[33032]: pgmap v533: 148 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized, 7 active+undersized+degraded, 127 active+clean; 7.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 47 MiB/s wr, 745 op/s; 50928/362646 objects degraded (14.043%); 7458/362646 objects misplaced (2.057%); 200 KiB/s, 10 objects/s recovering 2024-09-21T13:59:37.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:37 smithi187 ceph-mon[33032]: 5.1f scrub starts 2024-09-21T13:59:37.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:37 smithi187 ceph-mon[33032]: 5.1f scrub ok 2024-09-21T13:59:37.702 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:59:37.734 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:37.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:37.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:59:37.836 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:37.871 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:37.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:37.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:37.943 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 4, 5, 6, 2, 3, 7] 2024-09-21T13:59:37.943 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T13:59:37.944 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T13:59:37.973 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:38.024 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:38.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:38.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:38.127 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:38.161 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T13:59:38.164 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:38.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:38.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:38.266 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:59:38.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:38.305 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:38.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:59:38.406 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:59:38.448 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:38.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:38.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:59:38.549 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:38.572 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:38.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:38.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:38.575 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:38 smithi049 ceph-mon[40489]: 6.14 scrub starts 2024-09-21T13:59:38.575 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:38 smithi049 ceph-mon[40489]: 6.14 scrub ok 2024-09-21T13:59:38.575 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:38 smithi049 ceph-mon[30080]: 6.14 scrub starts 2024-09-21T13:59:38.575 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:38 smithi049 ceph-mon[30080]: 6.14 scrub ok 2024-09-21T13:59:38.674 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:59:38.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:38 smithi187 ceph-mon[33032]: 6.14 scrub starts 2024-09-21T13:59:38.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:38 smithi187 ceph-mon[33032]: 6.14 scrub ok 2024-09-21T13:59:38.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:38.721 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:38.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:59:38.822 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:38.864 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:38.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:38.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:38.966 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:39.000 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:39.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:39.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:39.102 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:39.141 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:39.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:39.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:39.186 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:59:39.205 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:59:39.243 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:39.279 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-09-21T13:59:39.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:39.285 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:39.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:39.293 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:59:39.327 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:39 smithi049 ceph-mon[40489]: pgmap v534: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 139 active+clean; 7.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 26 MiB/s wr, 417 op/s; 50933/363561 objects degraded (14.009%); 6751/363561 objects misplaced (1.857%); 5.4 MiB/s, 95 objects/s recovering 2024-09-21T13:59:39.327 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:39 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/309147' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:59:39.327 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:39 smithi049 ceph-mon[40489]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:59:39.327 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:39 smithi049 ceph-mon[30080]: pgmap v534: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 139 active+clean; 7.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 26 MiB/s wr, 417 op/s; 50933/363561 objects degraded (14.009%); 6751/363561 objects misplaced (1.857%); 5.4 MiB/s, 95 objects/s recovering 2024-09-21T13:59:39.327 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:39 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/309147' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:59:39.327 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:39 smithi049 ceph-mon[30080]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:59:39.386 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:59:39.427 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:39.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:39.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:59:39.530 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:39.563 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:39.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:39.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:39.565 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:39 smithi187 ceph-mon[33032]: pgmap v534: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 139 active+clean; 7.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 26 MiB/s wr, 417 op/s; 50933/363561 objects degraded (14.009%); 6751/363561 objects misplaced (1.857%); 5.4 MiB/s, 95 objects/s recovering 2024-09-21T13:59:39.565 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:39 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/309147' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T13:59:39.565 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:39 smithi187 ceph-mon[33032]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:59:39.665 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:39.695 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:39.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:39.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:39.797 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:59:39.831 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:39.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:39.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:59:39.933 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:59:39.944 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:59:39 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:59:39.968 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:39.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:39.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:59:40.070 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:40.102 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:40.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:40.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:40.204 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:59:40.236 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:40.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:40.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:59:40.338 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:40.368 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:40.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:40.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:40.471 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:40.503 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:40.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:40.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:40.551 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:40 smithi049 ceph-mon[30080]: 5.d scrub starts 2024-09-21T13:59:40.551 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:40 smithi049 ceph-mon[30080]: 5.d scrub ok 2024-09-21T13:59:40.551 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:40 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/309147' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T13:59:40.552 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:40 smithi049 ceph-mon[30080]: osdmap e301: 8 total, 8 up, 8 in 2024-09-21T13:59:40.552 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:40 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:59:40.552 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:40 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 50933/363561 objects degraded (14.009%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T13:59:40.552 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:40 smithi049 ceph-mon[30080]: osdmap e302: 8 total, 8 up, 8 in 2024-09-21T13:59:40.552 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:40 smithi049 ceph-mon[40489]: 5.d scrub starts 2024-09-21T13:59:40.552 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:40 smithi049 ceph-mon[40489]: 5.d scrub ok 2024-09-21T13:59:40.552 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:40 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/309147' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T13:59:40.552 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:40 smithi049 ceph-mon[40489]: osdmap e301: 8 total, 8 up, 8 in 2024-09-21T13:59:40.552 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:40 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:59:40.552 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:40 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 50933/363561 objects degraded (14.009%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T13:59:40.552 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:40 smithi049 ceph-mon[40489]: osdmap e302: 8 total, 8 up, 8 in 2024-09-21T13:59:40.605 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:59:40.638 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:40.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:40.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:59:40.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:40 smithi187 ceph-mon[33032]: 5.d scrub starts 2024-09-21T13:59:40.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:40 smithi187 ceph-mon[33032]: 5.d scrub ok 2024-09-21T13:59:40.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:40 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/309147' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T13:59:40.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:40 smithi187 ceph-mon[33032]: osdmap e301: 8 total, 8 up, 8 in 2024-09-21T13:59:40.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:40 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:59:40.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:40 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 50933/363561 objects degraded (14.009%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T13:59:40.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:40 smithi187 ceph-mon[33032]: osdmap e302: 8 total, 8 up, 8 in 2024-09-21T13:59:40.741 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:40.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:40.778 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:40.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:40.802 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:59:40 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:59:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:59:40.856 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:59:40.856 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:59:40.878 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:59:40.885 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:59:40.885 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:59:40.909 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:59:40.910 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:59:40.919 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:40.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:40.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:59:40.944 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:59:40.944 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:59:40.981 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:59:40.981 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:59:41.020 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:59:41.020 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:59:41.021 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:59:41.049 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:59:41.050 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:59:41.061 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:41.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:41.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:59:41.084 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:59:41.084 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:59:41.112 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T13:59:41.113 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:59:41.163 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:59:41.204 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:41.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:41.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:59:41.306 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:41.342 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:41.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:41.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:41.444 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:41.465 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:41 smithi049 ceph-mon[40489]: pgmap v536: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 139 active+clean; 7.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 25 MiB/s wr, 695 op/s; 49711/358080 objects degraded (13.883%); 6751/358080 objects misplaced (1.885%); 9.7 MiB/s, 162 objects/s recovering 2024-09-21T13:59:41.465 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:41 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1378484020' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:59:41.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:41 smithi049 ceph-mon[30080]: pgmap v536: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 139 active+clean; 7.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 25 MiB/s wr, 695 op/s; 49711/358080 objects degraded (13.883%); 6751/358080 objects misplaced (1.885%); 9.7 MiB/s, 162 objects/s recovering 2024-09-21T13:59:41.466 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:41 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1378484020' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:59:41.480 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:41.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:41.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:41.582 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:41.682 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:41.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:41.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:41.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:41 smithi187 ceph-mon[33032]: pgmap v536: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 139 active+clean; 7.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 25 MiB/s wr, 695 op/s; 49711/358080 objects degraded (13.883%); 6751/358080 objects misplaced (1.885%); 9.7 MiB/s, 162 objects/s recovering 2024-09-21T13:59:41.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:41 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1378484020' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:59:41.784 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:41.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:41.820 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:41.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:41.921 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:41.957 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:41.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:41.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:42.059 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:42.098 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:42.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:42.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:42.200 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:42.229 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:42.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:42.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:42.331 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:42.370 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:42.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:42.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:42.472 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:59:42.511 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:42.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:42.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:59:42.600 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T13:59:42.613 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:42.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:42.649 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:42.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:42.750 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:59:42.787 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:42.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:42.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:59:42.889 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:42.921 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:42.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:42.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:43.022 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:59:43.055 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:43.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:43.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:59:43.157 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:43.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:43.193 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:43.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:43.294 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:43.335 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:43.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:43.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:43.437 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:43.464 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:43.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:43.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:43.566 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:43.597 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:43.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:43.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:43.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:43 smithi187 ceph-mon[33032]: pgmap v538: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 139 active+clean; 7.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 14 MiB/s wr, 648 op/s; 49438/354756 objects degraded (13.936%); 6521/354756 objects misplaced (1.838%); 9.3 MiB/s, 152 objects/s recovering 2024-09-21T13:59:43.700 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:59:43.722 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:43.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:43.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:59:43.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:43 smithi049 ceph-mon[40489]: pgmap v538: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 139 active+clean; 7.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 14 MiB/s wr, 648 op/s; 49438/354756 objects degraded (13.936%); 6521/354756 objects misplaced (1.838%); 9.3 MiB/s, 152 objects/s recovering 2024-09-21T13:59:43.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:43 smithi049 ceph-mon[30080]: pgmap v538: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 139 active+clean; 7.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 14 MiB/s wr, 648 op/s; 49438/354756 objects degraded (13.936%); 6521/354756 objects misplaced (1.838%); 9.3 MiB/s, 152 objects/s recovering 2024-09-21T13:59:43.824 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:59:43.864 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:43.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:43.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:59:43.965 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:59:44.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:44.001 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:44.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:59:44.102 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:59:44.133 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:44.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:44.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:59:44.235 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:44.273 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:44.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:44.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:44.375 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:59:44.410 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:44.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:44.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:59:44.512 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:59:44.547 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:44.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:44.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:59:44.648 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:59:44.680 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:44.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:44.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:59:44.782 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:59:44.820 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:44.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:44.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:59:44.922 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:44.956 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:44.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:44.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:45.058 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:59:45.092 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:45.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:45.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:59:45.194 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:45.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:44 smithi187 ceph-mon[33032]: from='client.25565 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T13:59:45.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:44 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:59:45.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:44 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:59:45.229 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:45.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:45.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:45.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:44 smithi049 ceph-mon[40489]: from='client.25565 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T13:59:45.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:44 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:59:45.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:44 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:59:45.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:44 smithi049 ceph-mon[30080]: from='client.25565 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T13:59:45.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:44 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T13:59:45.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:44 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:59:45.331 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:45.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:45.368 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:45.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:45.412 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T13:59:45.469 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:59:45.507 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:45.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:45.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:59:45.609 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:45.640 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:45.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:45.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:45.742 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:45.770 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:45.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:45.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:45.865 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:59:45.871 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:45.903 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:59:45.905 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-09-21T13:59:45.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:45.920 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:45.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:46.021 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:46.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:46.064 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:46.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:46.166 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:46.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:45 smithi187 ceph-mon[33032]: pgmap v539: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 139 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 767 B/s rd, 5.6 MiB/s wr, 719 op/s; 49155/349728 objects degraded (14.055%); 6521/349728 objects misplaced (1.865%); 11 MiB/s, 185 objects/s recovering 2024-09-21T13:59:46.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:45 smithi187 ceph-mon[33032]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T13:59:46.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:45 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 49155/349728 objects degraded (14.055%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T13:59:46.198 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:46.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:46.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:46.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:45 smithi049 ceph-mon[40489]: pgmap v539: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 139 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 767 B/s rd, 5.6 MiB/s wr, 719 op/s; 49155/349728 objects degraded (14.055%); 6521/349728 objects misplaced (1.865%); 11 MiB/s, 185 objects/s recovering 2024-09-21T13:59:46.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:45 smithi049 ceph-mon[40489]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T13:59:46.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:45 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 49155/349728 objects degraded (14.055%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T13:59:46.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:45 smithi049 ceph-mon[30080]: pgmap v539: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 139 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 767 B/s rd, 5.6 MiB/s wr, 719 op/s; 49155/349728 objects degraded (14.055%); 6521/349728 objects misplaced (1.865%); 11 MiB/s, 185 objects/s recovering 2024-09-21T13:59:46.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:45 smithi049 ceph-mon[30080]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T13:59:46.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:45 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 49155/349728 objects degraded (14.055%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T13:59:46.300 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:59:46.327 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:46.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:46.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:59:46.429 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:46.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:46.461 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:46.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:46.562 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:59:46.602 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:46.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:46.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:59:46.649 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":538,"stamp":"2024-09-21T13:59:41.922592+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7737252403,"num_objects":118252,"num_object_clones":0,"num_object_copies":354756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":49438,"num_objects_misplaced":6521,"num_objects_unfound":0,"num_objects_dirty":118252,"num_whiteouts":0,"num_read":1248,"num_read_kb":1107,"num_write":125594,"num_write_kb":7771594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7521,"num_bytes_recovered":481495488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81655,"ondisk_log_size":81655,"up":444,"acting":436,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":446,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":23212884,"kb_used_data":20406152,"kb_used_omap":20,"kb_used_meta":2806507,"kb_avail":726748332,"statfs":{"total":767960285184,"available":744190291968,"internally_reserved":0,"allocated":20895899648,"data_stored":20891538964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21298,"internal_metadata":2873863374},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,4,8,2,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-99155944,"num_objects":-1512,"num_object_clones":0,"num_object_copies":-4536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3100,"num_objects_misplaced":-937,"num_objects_unfound":0,"num_objects_dirty":-1512,"num_whiteouts":0,"num_read":-71,"num_read_kb":-69,"num_write":5194,"num_write_kb":117761,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1223,"num_bytes_recovered":77987296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.005360"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":453,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.193273+0000","last_change":"2024-09-21T13:54:55.873382+0000","last_active":"2024-09-21T13:59:33.193273+0000","last_peered":"2024-09-21T13:59:33.193273+0000","last_clean":"2024-09-21T13:59:33.193273+0000","last_became_active":"2024-09-21T13:52:04.317322+0000","last_became_peered":"2024-09-21T13:52:04.317322+0000","last_unstale":"2024-09-21T13:59:33.193273+0000","last_undegraded":"2024-09-21T13:59:33.193273+0000","last_fullsized":"2024-09-21T13:59:33.193273+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:55.873301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:55.873301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:02.211126+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"290'15","reported_seq":582,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.206894+0000","last_change":"2024-09-21T13:58:57.766087+0000","last_active":"2024-09-21T13:59:33.206894+0000","last_peered":"2024-09-21T13:59:33.206894+0000","last_clean":"2024-09-21T13:59:33.206894+0000","last_became_active":"2024-09-21T13:58:17.771220+0000","last_became_peered":"2024-09-21T13:58:17.771220+0000","last_unstale":"2024-09-21T13:59:33.206894+0000","last_undegraded":"2024-09-21T13:59:33.206894+0000","last_fullsized":"2024-09-21T13:59:33.206894+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:58:57.766011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:57.766011+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:22.570440+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":514,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-21T13:59:39.286609+0000","last_change":"2024-09-21T13:59:32.201875+0000","last_active":"2024-09-21T13:59:39.286609+0000","last_peered":"2024-09-21T13:59:39.286609+0000","last_clean":"2024-09-21T13:59:39.286609+0000","last_became_active":"2024-09-21T13:59:32.201538+0000","last_became_peered":"2024-09-21T13:59:32.201538+0000","last_unstale":"2024-09-21T13:59:39.286609+0000","last_undegraded":"2024-09-21T13:59:39.286609+0000","last_fullsized":"2024-09-21T13:59:39.286609+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:40.452724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:40.452724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":496,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.207053+0000","last_change":"2024-09-21T13:57:00.320623+0000","last_active":"2024-09-21T13:59:33.207053+0000","last_peered":"2024-09-21T13:59:33.207053+0000","last_clean":"2024-09-21T13:59:33.207053+0000","last_became_active":"2024-09-21T13:49:04.072460+0000","last_became_peered":"2024-09-21T13:49:04.072460+0000","last_unstale":"2024-09-21T13:59:33.207053+0000","last_undegraded":"2024-09-21T13:59:33.207053+0000","last_fullsized":"2024-09-21T13:59:33.207053+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:00.320510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:00.320510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":481,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-21T13:59:40.042141+0000","last_change":"2024-09-21T13:59:32.208701+0000","last_active":"2024-09-21T13:59:40.042141+0000","last_peered":"2024-09-21T13:59:40.042141+0000","last_clean":"2024-09-21T13:59:40.042141+0000","last_became_active":"2024-09-21T13:59:32.203340+0000","last_became_peered":"2024-09-21T13:59:32.203340+0000","last_unstale":"2024-09-21T13:59:40.042141+0000","last_undegraded":"2024-09-21T13:59:40.042141+0000","last_fullsized":"2024-09-21T13:59:40.042141+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:34.696780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:34.696780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":493,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-21T13:59:40.040620+0000","last_change":"2024-09-21T13:58:17.778322+0000","last_active":"2024-09-21T13:59:40.040620+0000","last_peered":"2024-09-21T13:59:40.040620+0000","last_clean":"2024-09-21T13:59:40.040620+0000","last_became_active":"2024-09-21T13:58:17.777893+0000","last_became_peered":"2024-09-21T13:58:17.777893+0000","last_unstale":"2024-09-21T13:59:40.040620+0000","last_undegraded":"2024-09-21T13:59:40.040620+0000","last_fullsized":"2024-09-21T13:59:40.040620+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:07.001135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:55:07.001135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:18.028308+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"290'27","reported_seq":467,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-21T13:59:40.040743+0000","last_change":"2024-09-21T13:58:11.718421+0000","last_active":"2024-09-21T13:59:40.040743+0000","last_peered":"2024-09-21T13:59:40.040743+0000","last_clean":"2024-09-21T13:59:40.040743+0000","last_became_active":"2024-09-21T13:57:12.703469+0000","last_became_peered":"2024-09-21T13:57:12.703469+0000","last_unstale":"2024-09-21T13:59:40.040743+0000","last_undegraded":"2024-09-21T13:59:40.040743+0000","last_fullsized":"2024-09-21T13:59:40.040743+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:58:11.718334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:11.718334+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":522,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.206845+0000","last_change":"2024-09-21T13:58:53.761124+0000","last_active":"2024-09-21T13:59:33.206845+0000","last_peered":"2024-09-21T13:59:33.206845+0000","last_clean":"2024-09-21T13:59:33.206845+0000","last_became_active":"2024-09-21T13:58:17.773464+0000","last_became_peered":"2024-09-21T13:58:17.773464+0000","last_unstale":"2024-09-21T13:59:33.206845+0000","last_undegraded":"2024-09-21T13:59:33.206845+0000","last_fullsized":"2024-09-21T13:59:33.206845+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:53.761061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:53.761061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:05.719816+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":539,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.192653+0000","last_change":"2024-09-21T13:58:17.782330+0000","last_active":"2024-09-21T13:59:33.192653+0000","last_peered":"2024-09-21T13:59:33.192653+0000","last_clean":"2024-09-21T13:59:33.192653+0000","last_became_active":"2024-09-21T13:58:17.781904+0000","last_became_peered":"2024-09-21T13:58:17.781904+0000","last_unstale":"2024-09-21T13:59:33.192653+0000","last_undegraded":"2024-09-21T13:59:33.192653+0000","last_fullsized":"2024-09-21T13:59:33.192653+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:31.127926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:31.127926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"290'15","reported_seq":548,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.207701+0000","last_change":"2024-09-21T13:57:38.516376+0000","last_active":"2024-09-21T13:59:33.207701+0000","last_peered":"2024-09-21T13:59:33.207701+0000","last_clean":"2024-09-21T13:59:33.207701+0000","last_became_active":"2024-09-21T13:57:12.681408+0000","last_became_peered":"2024-09-21T13:57:12.681408+0000","last_unstale":"2024-09-21T13:59:33.207701+0000","last_undegraded":"2024-09-21T13:59:33.207701+0000","last_fullsized":"2024-09-21T13:59:33.207701+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:57:38.516309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:57:38.516309+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":504,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.191908+0000","last_change":"2024-09-21T13:59:32.213187+0000","last_active":"2024-09-21T13:59:33.191908+0000","last_peered":"2024-09-21T13:59:33.191908+0000","last_clean":"2024-09-21T13:59:33.191908+0000","last_became_active":"2024-09-21T13:59:32.212831+0000","last_became_peered":"2024-09-21T13:59:32.212831+0000","last_unstale":"2024-09-21T13:59:33.191908+0000","last_undegraded":"2024-09-21T13:59:33.191908+0000","last_fullsized":"2024-09-21T13:59:33.191908+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:21.284048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:56:21.284048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":462,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-21T13:59:39.288173+0000","last_change":"2024-09-21T13:59:32.202128+0000","last_active":"2024-09-21T13:59:39.288173+0000","last_peered":"2024-09-21T13:59:39.288173+0000","last_clean":"2024-09-21T13:59:39.288173+0000","last_became_active":"2024-09-21T13:59:32.201158+0000","last_became_peered":"2024-09-21T13:59:32.201158+0000","last_unstale":"2024-09-21T13:59:39.288173+0000","last_undegraded":"2024-09-21T13:59:39.288173+0000","last_fullsized":"2024-09-21T13:59:39.288173+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:25.792985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:54:25.792985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":509,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.191912+0000","last_change":"2024-09-21T13:56:54.328338+0000","last_active":"2024-09-21T13:59:33.191912+0000","last_peered":"2024-09-21T13:59:33.191912+0000","last_clean":"2024-09-21T13:59:33.191912+0000","last_became_active":"2024-09-21T13:48:58.050906+0000","last_became_peered":"2024-09-21T13:48:58.050906+0000","last_unstale":"2024-09-21T13:59:33.191912+0000","last_undegraded":"2024-09-21T13:59:33.191912+0000","last_fullsized":"2024-09-21T13:59:33.191912+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:54.328275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:54.328275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:07.169834+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"290'7","reported_seq":565,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.207290+0000","last_change":"2024-09-21T13:59:09.074586+0000","last_active":"2024-09-21T13:59:33.207290+0000","last_peered":"2024-09-21T13:59:33.207290+0000","last_clean":"2024-09-21T13:59:33.207290+0000","last_became_active":"2024-09-21T13:59:09.063053+0000","last_became_peered":"2024-09-21T13:59:09.063053+0000","last_unstale":"2024-09-21T13:59:33.207290+0000","last_undegraded":"2024-09-21T13:59:33.207290+0000","last_fullsized":"2024-09-21T13:59:33.207290+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-21T13:57:29.549403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.549403+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":456,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-21T13:59:40.045896+0000","last_change":"2024-09-21T13:59:32.204565+0000","last_active":"2024-09-21T13:59:40.045896+0000","last_peered":"2024-09-21T13:59:40.045896+0000","last_clean":"2024-09-21T13:59:40.045896+0000","last_became_active":"2024-09-21T13:59:32.204174+0000","last_became_peered":"2024-09-21T13:59:32.204174+0000","last_unstale":"2024-09-21T13:59:40.045896+0000","last_undegraded":"2024-09-21T13:59:40.045896+0000","last_fullsized":"2024-09-21T13:59:40.045896+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:06.722293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:06.722293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":524,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.191706+0000","last_change":"2024-09-21T13:58:44.376278+0000","last_active":"2024-09-21T13:59:33.191706+0000","last_peered":"2024-09-21T13:59:33.191706+0000","last_clean":"2024-09-21T13:59:33.191706+0000","last_became_active":"2024-09-21T13:57:12.684385+0000","last_became_peered":"2024-09-21T13:57:12.684385+0000","last_unstale":"2024-09-21T13:59:33.191706+0000","last_undegraded":"2024-09-21T13:59:33.191706+0000","last_fullsized":"2024-09-21T13:59:33.191706+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:44.376215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:44.376215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:59:51.576347+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":522,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.207978+0000","last_change":"2024-09-21T13:58:17.772250+0000","last_active":"2024-09-21T13:59:33.207978+0000","last_peered":"2024-09-21T13:59:33.207978+0000","last_clean":"2024-09-21T13:59:33.207978+0000","last_became_active":"2024-09-21T13:58:17.770940+0000","last_became_peered":"2024-09-21T13:58:17.770940+0000","last_unstale":"2024-09-21T13:59:33.207978+0000","last_undegraded":"2024-09-21T13:59:33.207978+0000","last_fullsized":"2024-09-21T13:59:33.207978+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:21.448653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:21.448653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":459,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.207549+0000","last_change":"2024-09-21T13:59:32.205037+0000","last_active":"2024-09-21T13:59:33.207549+0000","last_peered":"2024-09-21T13:59:33.207549+0000","last_clean":"2024-09-21T13:59:33.207549+0000","last_became_active":"2024-09-21T13:59:32.204749+0000","last_became_peered":"2024-09-21T13:59:32.204749+0000","last_unstale":"2024-09-21T13:59:33.207549+0000","last_undegraded":"2024-09-21T13:59:33.207549+0000","last_fullsized":"2024-09-21T13:59:33.207549+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:17.361699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:17.361699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"290'27","reported_seq":508,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.190614+0000","last_change":"2024-09-21T13:58:47.755659+0000","last_active":"2024-09-21T13:59:33.190614+0000","last_peered":"2024-09-21T13:59:33.190614+0000","last_clean":"2024-09-21T13:59:33.190614+0000","last_became_active":"2024-09-21T13:57:12.680783+0000","last_became_peered":"2024-09-21T13:57:12.680783+0000","last_unstale":"2024-09-21T13:59:33.190614+0000","last_undegraded":"2024-09-21T13:59:33.190614+0000","last_fullsized":"2024-09-21T13:59:33.190614+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:58:47.755561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:47.755561+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:01.959306+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":511,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-21T13:59:39.288154+0000","last_change":"2024-09-21T13:55:31.988001+0000","last_active":"2024-09-21T13:59:39.288154+0000","last_peered":"2024-09-21T13:59:39.288154+0000","last_clean":"2024-09-21T13:59:39.288154+0000","last_became_active":"2024-09-21T13:49:02.066796+0000","last_became_peered":"2024-09-21T13:49:02.066796+0000","last_unstale":"2024-09-21T13:59:39.288154+0000","last_undegraded":"2024-09-21T13:59:39.288154+0000","last_fullsized":"2024-09-21T13:59:39.288154+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:31.987854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:55:31.987854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:08.223013+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":462,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-21T13:59:40.043211+0000","last_change":"2024-09-21T13:58:17.779422+0000","last_active":"2024-09-21T13:59:40.043211+0000","last_peered":"2024-09-21T13:59:40.043211+0000","last_clean":"2024-09-21T13:59:40.043211+0000","last_became_active":"2024-09-21T13:58:17.779117+0000","last_became_peered":"2024-09-21T13:58:17.779117+0000","last_unstale":"2024-09-21T13:59:40.043211+0000","last_undegraded":"2024-09-21T13:59:40.043211+0000","last_fullsized":"2024-09-21T13:59:40.043211+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:57.962266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:57.962266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:59:59.730442+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":519,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.207789+0000","last_change":"2024-09-21T13:56:58.309928+0000","last_active":"2024-09-21T13:59:33.207789+0000","last_peered":"2024-09-21T13:59:33.207789+0000","last_clean":"2024-09-21T13:59:33.207789+0000","last_became_active":"2024-09-21T13:52:04.316016+0000","last_became_peered":"2024-09-21T13:52:04.316016+0000","last_unstale":"2024-09-21T13:59:33.207789+0000","last_undegraded":"2024-09-21T13:59:33.207789+0000","last_fullsized":"2024-09-21T13:59:33.207789+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:58.309863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:58.309863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"290'18","reported_seq":479,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-21T13:59:40.040720+0000","last_change":"2024-09-21T13:59:23.041396+0000","last_active":"2024-09-21T13:59:40.040720+0000","last_peered":"2024-09-21T13:59:40.040720+0000","last_clean":"2024-09-21T13:59:40.040720+0000","last_became_active":"2024-09-21T13:57:12.684454+0000","last_became_peered":"2024-09-21T13:57:12.684454+0000","last_unstale":"2024-09-21T13:59:40.040720+0000","last_undegraded":"2024-09-21T13:59:40.040720+0000","last_fullsized":"2024-09-21T13:59:40.040720+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'18","last_scrub_stamp":"2024-09-21T13:59:23.041311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:23.041311+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:25.910995+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":514,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.207884+0000","last_change":"2024-09-21T13:58:16.393763+0000","last_active":"2024-09-21T13:59:33.207884+0000","last_peered":"2024-09-21T13:59:33.207884+0000","last_clean":"2024-09-21T13:59:33.207884+0000","last_became_active":"2024-09-21T13:57:12.681897+0000","last_became_peered":"2024-09-21T13:57:12.681897+0000","last_unstale":"2024-09-21T13:59:33.207884+0000","last_undegraded":"2024-09-21T13:59:33.207884+0000","last_fullsized":"2024-09-21T13:59:33.207884+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:16.393696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:16.393696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:59:34.177409+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":429,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-21T13:59:40.047099+0000","last_change":"2024-09-21T13:59:32.216521+0000","last_active":"2024-09-21T13:59:40.047099+0000","last_peered":"2024-09-21T13:59:40.047099+0000","last_clean":"2024-09-21T13:59:40.047099+0000","last_became_active":"2024-09-21T13:59:32.215222+0000","last_became_peered":"2024-09-21T13:59:32.215222+0000","last_unstale":"2024-09-21T13:59:40.047099+0000","last_undegraded":"2024-09-21T13:59:40.047099+0000","last_fullsized":"2024-09-21T13:59:40.047099+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T13:57:26.536684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:26.536684+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":531,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-21T13:59:39.286761+0000","last_change":"2024-09-21T13:57:27.305798+0000","last_active":"2024-09-21T13:59:39.286761+0000","last_peered":"2024-09-21T13:59:39.286761+0000","last_clean":"2024-09-21T13:59:39.286761+0000","last_became_active":"2024-09-21T13:57:12.685742+0000","last_became_peered":"2024-09-21T13:57:12.685742+0000","last_unstale":"2024-09-21T13:59:39.286761+0000","last_undegraded":"2024-09-21T13:59:39.286761+0000","last_fullsized":"2024-09-21T13:59:39.286761+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:57:27.305724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:27.305724+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:21.365203+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"290'20","reported_seq":420,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-21T13:59:40.046105+0000","last_change":"2024-09-21T13:59:32.242771+0000","last_active":"2024-09-21T13:59:40.046105+0000","last_peered":"2024-09-21T13:59:40.046105+0000","last_clean":"2024-09-21T13:59:40.046105+0000","last_became_active":"2024-09-21T13:59:32.213865+0000","last_became_peered":"2024-09-21T13:59:32.213865+0000","last_unstale":"2024-09-21T13:59:40.046105+0000","last_undegraded":"2024-09-21T13:59:40.046105+0000","last_fullsized":"2024-09-21T13:59:40.046105+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:57:27.578738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:57:27.578738+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"300'12","reported_seq":114,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:36.012862+0000","last_change":"2024-09-21T13:59:36.012775+0000","last_active":"2024-09-21T13:59:36.012862+0000","last_peered":"2024-09-21T13:59:36.012862+0000","last_clean":"2024-09-21T13:59:36.012862+0000","last_became_active":"2024-09-21T13:59:32.205051+0000","last_became_peered":"2024-09-21T13:59:32.205051+0000","last_unstale":"2024-09-21T13:59:36.012862+0000","last_undegraded":"2024-09-21T13:59:36.012862+0000","last_fullsized":"2024-09-21T13:59:36.012862+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'12","last_scrub_stamp":"2024-09-21T13:59:36.012693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:36.012693+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:01:05.127536+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2","version":"302'7911","reported_seq":8006,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-21T13:59:40.895071+0000","last_change":"2024-09-21T13:57:53.208687+0000","last_active":"2024-09-21T13:59:40.895071+0000","last_peered":"2024-09-21T13:59:40.895071+0000","last_clean":"2024-09-21T13:59:40.895071+0000","last_became_active":"2024-09-21T13:57:53.208352+0000","last_became_peered":"2024-09-21T13:57:53.208352+0000","last_unstale":"2024-09-21T13:59:40.895071+0000","last_undegraded":"2024-09-21T13:59:40.895071+0000","last_fullsized":"2024-09-21T13:59:40.895071+0000","mapping_epoch":260,"log_start":"277'3200","ondisk_log_start":"277'3200","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4711,"log_dups_size":0,"ondisk_log_size":4711,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":478740480,"num_objects":7305,"num_object_clones":0,"num_object_copies":21915,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7911,"num_write_kb":486912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"290'27","reported_seq":577,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-21T13:59:39.287181+0000","last_change":"2024-09-21T13:58:45.984691+0000","last_active":"2024-09-21T13:59:39.287181+0000","last_peered":"2024-09-21T13:59:39.287181+0000","last_clean":"2024-09-21T13:59:39.287181+0000","last_became_active":"2024-09-21T13:57:12.695609+0000","last_became_peered":"2024-09-21T13:57:12.695609+0000","last_unstale":"2024-09-21T13:59:39.287181+0000","last_undegraded":"2024-09-21T13:59:39.287181+0000","last_fullsized":"2024-09-21T13:59:39.287181+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:58:45.984575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:45.984575+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:07.027041+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":465,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-21T13:59:40.040175+0000","last_change":"2024-09-21T13:59:32.205723+0000","last_active":"2024-09-21T13:59:40.040175+0000","last_peered":"2024-09-21T13:59:40.040175+0000","last_clean":"2024-09-21T13:59:40.040175+0000","last_became_active":"2024-09-21T13:59:32.205350+0000","last_became_peered":"2024-09-21T13:59:32.205350+0000","last_unstale":"2024-09-21T13:59:40.040175+0000","last_undegraded":"2024-09-21T13:59:40.040175+0000","last_fullsized":"2024-09-21T13:59:40.040175+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:32.668851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:32.668851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"68'8","reported_seq":640,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.245955+0000","last_change":"2024-09-21T13:56:59.335345+0000","last_active":"2024-09-21T13:59:33.245955+0000","last_peered":"2024-09-21T13:59:33.245955+0000","last_clean":"2024-09-21T13:59:33.245955+0000","last_became_active":"2024-09-21T13:49:02.061679+0000","last_became_peered":"2024-09-21T13:49:02.061679+0000","last_unstale":"2024-09-21T13:59:33.245955+0000","last_undegraded":"2024-09-21T13:59:33.245955+0000","last_fullsized":"2024-09-21T13:59:33.245955+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:56:59.335171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:56:59.335171+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":497,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.191923+0000","last_change":"2024-09-21T13:58:52.244527+0000","last_active":"2024-09-21T13:59:33.191923+0000","last_peered":"2024-09-21T13:59:33.191923+0000","last_clean":"2024-09-21T13:59:33.191923+0000","last_became_active":"2024-09-21T13:49:04.081905+0000","last_became_peered":"2024-09-21T13:49:04.081905+0000","last_unstale":"2024-09-21T13:59:33.191923+0000","last_undegraded":"2024-09-21T13:59:33.191923+0000","last_fullsized":"2024-09-21T13:59:33.191923+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:52.244483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:52.244483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:19.563849+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"301'7814","reported_seq":7923,"reported_epoch":301,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2024-09-21T13:59:39.494344+0000","last_change":"2024-09-21T13:59:33.321465+0000","last_active":"2024-09-21T13:59:39.494344+0000","last_peered":"2024-09-21T13:59:39.494344+0000","last_clean":"2024-09-21T13:58:23.757609+0000","last_became_active":"2024-09-21T13:59:33.315668+0000","last_became_peered":"2024-09-21T13:59:33.315668+0000","last_unstale":"2024-09-21T13:59:39.494344+0000","last_undegraded":"2024-09-21T13:59:33.208390+0000","last_fullsized":"2024-09-21T13:59:33.189502+0000","mapping_epoch":299,"log_start":"279'3165","ondisk_log_start":"279'3165","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4649,"log_dups_size":0,"ondisk_log_size":4649,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484638744,"num_objects":7396,"num_object_clones":0,"num_object_copies":22188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7396,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7396,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7814,"num_write_kb":486657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":7396}],"blocked_by":[],"up_primary":7,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"290'30","reported_seq":489,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-21T13:59:40.040656+0000","last_change":"2024-09-21T13:59:32.246090+0000","last_active":"2024-09-21T13:59:40.040656+0000","last_peered":"2024-09-21T13:59:40.040656+0000","last_clean":"2024-09-21T13:59:40.040656+0000","last_became_active":"2024-09-21T13:59:32.205887+0000","last_became_peered":"2024-09-21T13:59:32.205887+0000","last_unstale":"2024-09-21T13:59:40.040656+0000","last_undegraded":"2024-09-21T13:59:40.040656+0000","last_fullsized":"2024-09-21T13:59:40.040656+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'18","last_scrub_stamp":"2024-09-21T13:54:19.001030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:19.001030+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":417,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-21T13:59:40.042813+0000","last_change":"2024-09-21T13:59:32.212559+0000","last_active":"2024-09-21T13:59:40.042813+0000","last_peered":"2024-09-21T13:59:40.042813+0000","last_clean":"2024-09-21T13:59:40.042813+0000","last_became_active":"2024-09-21T13:59:32.211004+0000","last_became_peered":"2024-09-21T13:59:32.211004+0000","last_unstale":"2024-09-21T13:59:40.042813+0000","last_undegraded":"2024-09-21T13:59:40.042813+0000","last_fullsized":"2024-09-21T13:59:40.042813+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:25.736461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:25.736461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":531,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:34.293330+0000","last_change":"2024-09-21T13:59:34.293330+0000","last_active":"2024-09-21T13:59:34.293330+0000","last_peered":"2024-09-21T13:59:34.293330+0000","last_clean":"2024-09-21T13:59:34.293330+0000","last_became_active":"2024-09-21T13:59:32.220304+0000","last_became_peered":"2024-09-21T13:59:32.220304+0000","last_unstale":"2024-09-21T13:59:34.293330+0000","last_undegraded":"2024-09-21T13:59:34.293330+0000","last_fullsized":"2024-09-21T13:59:34.293330+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:34.292892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:34.292892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:46.211883+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":511,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.207698+0000","last_change":"2024-09-21T13:59:23.334230+0000","last_active":"2024-09-21T13:59:33.207698+0000","last_peered":"2024-09-21T13:59:33.207698+0000","last_clean":"2024-09-21T13:59:33.207698+0000","last_became_active":"2024-09-21T13:57:12.680372+0000","last_became_peered":"2024-09-21T13:57:12.680372+0000","last_unstale":"2024-09-21T13:59:33.207698+0000","last_undegraded":"2024-09-21T13:59:33.207698+0000","last_fullsized":"2024-09-21T13:59:33.207698+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:23.334162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:23.334162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:46.281593+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"302'7810","reported_seq":7966,"reported_epoch":302,"state":"active+undersized+degraded+remapped+backfill_wait","last_fresh":"2024-09-21T13:59:40.913806+0000","last_change":"2024-09-21T13:59:33.468803+0000","last_active":"2024-09-21T13:59:40.913806+0000","last_peered":"2024-09-21T13:59:40.913806+0000","last_clean":"2024-09-21T13:58:23.796606+0000","last_became_active":"2024-09-21T13:59:33.303769+0000","last_became_peered":"2024-09-21T13:59:33.303769+0000","last_unstale":"2024-09-21T13:59:40.913806+0000","last_undegraded":"2024-09-21T13:59:33.205632+0000","last_fullsized":"2024-09-21T13:59:33.192057+0000","mapping_epoch":299,"log_start":"277'3100","ondisk_log_start":"277'3100","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4710,"log_dups_size":0,"ondisk_log_size":4710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":474480640,"num_objects":7240,"num_object_clones":0,"num_object_copies":21720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7810,"num_write_kb":481600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1703936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":7240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"290'28","reported_seq":593,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.207189+0000","last_change":"2024-09-21T13:58:17.810036+0000","last_active":"2024-09-21T13:59:33.207189+0000","last_peered":"2024-09-21T13:59:33.207189+0000","last_clean":"2024-09-21T13:59:33.207189+0000","last_became_active":"2024-09-21T13:58:17.779342+0000","last_became_peered":"2024-09-21T13:58:17.779342+0000","last_unstale":"2024-09-21T13:59:33.207189+0000","last_undegraded":"2024-09-21T13:59:33.207189+0000","last_fullsized":"2024-09-21T13:59:33.207189+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'14","last_scrub_stamp":"2024-09-21T13:56:32.438459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:56:32.438459+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":518,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.194637+0000","last_change":"2024-09-21T13:57:38.046718+0000","last_active":"2024-09-21T13:59:33.194637+0000","last_peered":"2024-09-21T13:59:33.194637+0000","last_clean":"2024-09-21T13:59:33.194637+0000","last_became_active":"2024-09-21T13:48:58.053533+0000","last_became_peered":"2024-09-21T13:48:58.053533+0000","last_unstale":"2024-09-21T13:59:33.194637+0000","last_undegraded":"2024-09-21T13:59:33.194637+0000","last_fullsized":"2024-09-21T13:59:33.194637+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:57:38.046627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:38.046627+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:10.473009+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"241'10","reported_seq":527,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-21T13:59:40.041366+0000","last_change":"2024-09-21T13:58:06.789629+0000","last_active":"2024-09-21T13:59:40.041366+0000","last_peered":"2024-09-21T13:59:40.041366+0000","last_clean":"2024-09-21T13:59:40.041366+0000","last_became_active":"2024-09-21T13:57:12.686651+0000","last_became_peered":"2024-09-21T13:57:12.686651+0000","last_unstale":"2024-09-21T13:59:40.041366+0000","last_undegraded":"2024-09-21T13:59:40.041366+0000","last_fullsized":"2024-09-21T13:59:40.041366+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'10","last_scrub_stamp":"2024-09-21T13:58:06.789560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:06.789560+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":481,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-21T13:59:40.048067+0000","last_change":"2024-09-21T13:59:32.210257+0000","last_active":"2024-09-21T13:59:40.048067+0000","last_peered":"2024-09-21T13:59:40.048067+0000","last_clean":"2024-09-21T13:59:40.048067+0000","last_became_active":"2024-09-21T13:59:32.209615+0000","last_became_peered":"2024-09-21T13:59:32.209615+0000","last_unstale":"2024-09-21T13:59:40.048067+0000","last_undegraded":"2024-09-21T13:59:40.048067+0000","last_fullsized":"2024-09-21T13:59:40.048067+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:37.548064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:37.548064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"300'7834","reported_seq":7925,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:38.588410+0000","last_change":"2024-09-21T13:57:53.213439+0000","last_active":"2024-09-21T13:59:38.588410+0000","last_peered":"2024-09-21T13:59:38.588410+0000","last_clean":"2024-09-21T13:59:38.588410+0000","last_became_active":"2024-09-21T13:57:53.213139+0000","last_became_peered":"2024-09-21T13:57:53.213139+0000","last_unstale":"2024-09-21T13:59:38.588410+0000","last_undegraded":"2024-09-21T13:59:38.588410+0000","last_fullsized":"2024-09-21T13:59:38.588410+0000","mapping_epoch":260,"log_start":"275'2500","ondisk_log_start":"275'2500","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":5334,"log_dups_size":0,"ondisk_log_size":5334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":491782144,"num_objects":7504,"num_object_clones":0,"num_object_copies":22512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7834,"num_write_kb":490816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"290'20","reported_seq":546,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-21T13:59:39.288483+0000","last_change":"2024-09-21T13:54:19.593696+0000","last_active":"2024-09-21T13:59:39.288483+0000","last_peered":"2024-09-21T13:59:39.288483+0000","last_clean":"2024-09-21T13:59:39.288483+0000","last_became_active":"2024-09-21T13:49:00.054564+0000","last_became_peered":"2024-09-21T13:49:00.054564+0000","last_unstale":"2024-09-21T13:59:39.288483+0000","last_undegraded":"2024-09-21T13:59:39.288483+0000","last_fullsized":"2024-09-21T13:59:39.288483+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:54:19.593611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:19.593611+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:04.212544+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"61'1","reported_seq":562,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.193196+0000","last_change":"2024-09-21T13:58:48.407813+0000","last_active":"2024-09-21T13:59:33.193196+0000","last_peered":"2024-09-21T13:59:33.193196+0000","last_clean":"2024-09-21T13:59:33.193196+0000","last_became_active":"2024-09-21T13:57:12.684939+0000","last_became_peered":"2024-09-21T13:57:12.684939+0000","last_unstale":"2024-09-21T13:59:33.193196+0000","last_undegraded":"2024-09-21T13:59:33.193196+0000","last_fullsized":"2024-09-21T13:59:33.193196+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:58:48.407734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:48.407734+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:59:54.869007+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"270'13","reported_seq":93,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.251338+0000","last_change":"2024-09-21T13:58:17.788471+0000","last_active":"2024-09-21T13:59:33.251338+0000","last_peered":"2024-09-21T13:59:33.251338+0000","last_clean":"2024-09-21T13:59:33.251338+0000","last_became_active":"2024-09-21T13:58:17.777857+0000","last_became_peered":"2024-09-21T13:58:17.777857+0000","last_unstale":"2024-09-21T13:59:33.251338+0000","last_undegraded":"2024-09-21T13:59:33.251338+0000","last_fullsized":"2024-09-21T13:59:33.251338+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'12","last_scrub_stamp":"2024-09-21T13:57:30.133392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:30.133392+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:18.984922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":335,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-21T13:59:40.041270+0000","last_change":"2024-09-21T13:58:58.730439+0000","last_active":"2024-09-21T13:59:40.041270+0000","last_peered":"2024-09-21T13:59:40.041270+0000","last_clean":"2024-09-21T13:59:40.041270+0000","last_became_active":"2024-09-21T13:57:12.704897+0000","last_became_peered":"2024-09-21T13:57:12.704897+0000","last_unstale":"2024-09-21T13:59:40.041270+0000","last_undegraded":"2024-09-21T13:59:40.041270+0000","last_fullsized":"2024-09-21T13:59:40.041270+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:58.730368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:58.730368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:59:59.823526+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.e","version":"301'7661","reported_seq":7762,"reported_epoch":301,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2024-09-21T13:59:39.511757+0000","last_change":"2024-09-21T13:59:33.350619+0000","last_active":"2024-09-21T13:59:39.511757+0000","last_peered":"2024-09-21T13:59:39.511757+0000","last_clean":"2024-09-21T13:58:23.771204+0000","last_became_active":"2024-09-21T13:59:33.338259+0000","last_became_peered":"2024-09-21T13:59:33.338259+0000","last_unstale":"2024-09-21T13:59:39.511757+0000","last_undegraded":"2024-09-21T13:59:33.212884+0000","last_fullsized":"2024-09-21T13:59:33.191296+0000","mapping_epoch":299,"log_start":"277'2983","ondisk_log_start":"277'2983","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4678,"log_dups_size":0,"ondisk_log_size":4678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":476119040,"num_objects":7265,"num_object_clones":0,"num_object_copies":21795,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7661,"num_write_kb":477632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":7265}],"blocked_by":[],"up_primary":7,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"290'29","reported_seq":537,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.191987+0000","last_change":"2024-09-21T13:59:32.235547+0000","last_active":"2024-09-21T13:59:33.191987+0000","last_peered":"2024-09-21T13:59:33.191987+0000","last_clean":"2024-09-21T13:59:33.191987+0000","last_became_active":"2024-09-21T13:59:32.209177+0000","last_became_peered":"2024-09-21T13:59:32.209177+0000","last_unstale":"2024-09-21T13:59:33.191987+0000","last_undegraded":"2024-09-21T13:59:33.191987+0000","last_fullsized":"2024-09-21T13:59:33.191987+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-21T13:56:55.319759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:56:55.319759+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":523,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-21T13:59:39.288735+0000","last_change":"2024-09-21T13:58:16.713075+0000","last_active":"2024-09-21T13:59:39.288735+0000","last_peered":"2024-09-21T13:59:39.288735+0000","last_clean":"2024-09-21T13:59:39.288735+0000","last_became_active":"2024-09-21T13:52:04.321656+0000","last_became_peered":"2024-09-21T13:52:04.321656+0000","last_unstale":"2024-09-21T13:59:39.288735+0000","last_undegraded":"2024-09-21T13:59:39.288735+0000","last_fullsized":"2024-09-21T13:59:39.288735+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:16.712990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:16.712990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:59:42.679447+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":80,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.248649+0000","last_change":"2024-09-21T13:59:23.032196+0000","last_active":"2024-09-21T13:59:33.248649+0000","last_peered":"2024-09-21T13:59:33.248649+0000","last_clean":"2024-09-21T13:59:33.248649+0000","last_became_active":"2024-09-21T13:58:17.778679+0000","last_became_peered":"2024-09-21T13:58:17.778679+0000","last_unstale":"2024-09-21T13:59:33.248649+0000","last_undegraded":"2024-09-21T13:59:33.248649+0000","last_fullsized":"2024-09-21T13:59:33.248649+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:23.032143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:23.032143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:26.703801+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":525,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-21T13:59:39.286468+0000","last_change":"2024-09-21T13:58:43.952014+0000","last_active":"2024-09-21T13:59:39.286468+0000","last_peered":"2024-09-21T13:59:39.286468+0000","last_clean":"2024-09-21T13:59:39.286468+0000","last_became_active":"2024-09-21T13:58:17.757628+0000","last_became_peered":"2024-09-21T13:58:17.757628+0000","last_unstale":"2024-09-21T13:59:39.286468+0000","last_undegraded":"2024-09-21T13:59:39.286468+0000","last_fullsized":"2024-09-21T13:59:39.286468+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:43.951941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:43.951941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:01.473279+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"300'7821","reported_seq":8384,"reported_epoch":300,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-21T13:59:39.096258+0000","last_change":"2024-09-21T13:59:33.436280+0000","last_active":"2024-09-21T13:59:39.096258+0000","last_peered":"2024-09-21T13:59:39.096258+0000","last_clean":"2024-09-21T13:58:23.765960+0000","last_became_active":"2024-09-21T13:59:33.385921+0000","last_became_peered":"2024-09-21T13:59:33.385921+0000","last_unstale":"2024-09-21T13:59:39.096258+0000","last_undegraded":"2024-09-21T13:59:33.219145+0000","last_fullsized":"2024-09-21T13:59:33.193300+0000","mapping_epoch":299,"log_start":"272'2300","ondisk_log_start":"272'2300","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":5521,"log_dups_size":0,"ondisk_log_size":5521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":488308736,"num_objects":7451,"num_object_clones":0,"num_object_copies":22353,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7821,"num_write_kb":488704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":234,"num_bytes_recovered":15335424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[{"shards":"0,5","objects":2426}],"blocked_by":[],"up_primary":7,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"290'44","reported_seq":729,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-21T13:59:39.287462+0000","last_change":"2024-09-21T13:58:56.076153+0000","last_active":"2024-09-21T13:59:39.287462+0000","last_peered":"2024-09-21T13:59:39.287462+0000","last_clean":"2024-09-21T13:59:39.287462+0000","last_became_active":"2024-09-21T13:57:12.684690+0000","last_became_peered":"2024-09-21T13:57:12.684690+0000","last_unstale":"2024-09-21T13:59:39.287462+0000","last_undegraded":"2024-09-21T13:59:39.287462+0000","last_fullsized":"2024-09-21T13:59:39.287462+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'32","last_scrub_stamp":"2024-09-21T13:58:56.076074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:56.076074+0000","objects_scrubbed":10,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:04.712559+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":523,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.207624+0000","last_change":"2024-09-21T13:59:27.369582+0000","last_active":"2024-09-21T13:59:33.207624+0000","last_peered":"2024-09-21T13:59:33.207624+0000","last_clean":"2024-09-21T13:59:33.207624+0000","last_became_active":"2024-09-21T13:58:17.770937+0000","last_became_peered":"2024-09-21T13:58:17.770937+0000","last_unstale":"2024-09-21T13:59:33.207624+0000","last_undegraded":"2024-09-21T13:59:33.207624+0000","last_fullsized":"2024-09-21T13:59:33.207624+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:27.369511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:59:27.369511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:40.229805+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":558,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.237723+0000","last_change":"2024-09-21T13:59:22.186466+0000","last_active":"2024-09-21T13:59:33.237723+0000","last_peered":"2024-09-21T13:59:33.237723+0000","last_clean":"2024-09-21T13:59:33.237723+0000","last_became_active":"2024-09-21T13:58:17.776563+0000","last_became_peered":"2024-09-21T13:58:17.776563+0000","last_unstale":"2024-09-21T13:59:33.237723+0000","last_undegraded":"2024-09-21T13:59:33.237723+0000","last_fullsized":"2024-09-21T13:59:33.237723+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-21T13:59:22.186393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:59:22.186393+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:28.219690+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0015946669999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":452,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-21T13:59:39.287915+0000","last_change":"2024-09-21T13:55:07.021128+0000","last_active":"2024-09-21T13:59:39.287915+0000","last_peered":"2024-09-21T13:59:39.287915+0000","last_clean":"2024-09-21T13:59:39.287915+0000","last_became_active":"2024-09-21T13:50:29.688249+0000","last_became_peered":"2024-09-21T13:50:29.688249+0000","last_unstale":"2024-09-21T13:59:39.287915+0000","last_undegraded":"2024-09-21T13:59:39.287915+0000","last_fullsized":"2024-09-21T13:59:39.287915+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:07.021059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:55:07.021059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:06.200040+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":444,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-21T13:59:40.041304+0000","last_change":"2024-09-21T13:59:32.209357+0000","last_active":"2024-09-21T13:59:40.041304+0000","last_peered":"2024-09-21T13:59:40.041304+0000","last_clean":"2024-09-21T13:59:40.041304+0000","last_became_active":"2024-09-21T13:59:32.208980+0000","last_became_peered":"2024-09-21T13:59:32.208980+0000","last_unstale":"2024-09-21T13:59:40.041304+0000","last_undegraded":"2024-09-21T13:59:40.041304+0000","last_fullsized":"2024-09-21T13:59:40.041304+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:16.797227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:16.797227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"300'7601","reported_seq":7698,"reported_epoch":300,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T13:59:38.577263+0000","last_change":"2024-09-21T13:59:33.463191+0000","last_active":"2024-09-21T13:59:38.577263+0000","last_peered":"2024-09-21T13:59:38.577263+0000","last_clean":"2024-09-21T13:58:23.784671+0000","last_became_active":"2024-09-21T13:59:33.447969+0000","last_became_peered":"2024-09-21T13:59:33.447969+0000","last_unstale":"2024-09-21T13:59:38.577263+0000","last_undegraded":"2024-09-21T13:59:33.234331+0000","last_fullsized":"2024-09-21T13:59:33.191526+0000","mapping_epoch":299,"log_start":"274'2300","ondisk_log_start":"274'2300","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":5301,"log_dups_size":0,"ondisk_log_size":5301,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":475856896,"num_objects":7261,"num_object_clones":0,"num_object_copies":21783,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5080,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7601,"num_write_kb":475584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":2181}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"290'17","reported_seq":548,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-21T13:59:39.287286+0000","last_change":"2024-09-21T13:58:11.619937+0000","last_active":"2024-09-21T13:59:39.287286+0000","last_peered":"2024-09-21T13:59:39.287286+0000","last_clean":"2024-09-21T13:59:39.287286+0000","last_became_active":"2024-09-21T13:49:00.053323+0000","last_became_peered":"2024-09-21T13:49:00.053323+0000","last_unstale":"2024-09-21T13:59:39.287286+0000","last_undegraded":"2024-09-21T13:59:39.287286+0000","last_fullsized":"2024-09-21T13:59:39.287286+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'13","last_scrub_stamp":"2024-09-21T13:58:11.619848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:11.619848+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":561,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-21T13:59:39.287160+0000","last_change":"2024-09-21T13:59:29.454265+0000","last_active":"2024-09-21T13:59:39.287160+0000","last_peered":"2024-09-21T13:59:39.287160+0000","last_clean":"2024-09-21T13:59:39.287160+0000","last_became_active":"2024-09-21T13:58:17.759671+0000","last_became_peered":"2024-09-21T13:58:17.759671+0000","last_unstale":"2024-09-21T13:59:39.287160+0000","last_undegraded":"2024-09-21T13:59:39.287160+0000","last_fullsized":"2024-09-21T13:59:39.287160+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:29.454085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:59:29.454085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:43.157879+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":448,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.188890+0000","last_change":"2024-09-21T13:58:41.644876+0000","last_active":"2024-09-21T13:59:33.188890+0000","last_peered":"2024-09-21T13:59:33.188890+0000","last_clean":"2024-09-21T13:59:33.188890+0000","last_became_active":"2024-09-21T13:57:12.682147+0000","last_became_peered":"2024-09-21T13:57:12.682147+0000","last_unstale":"2024-09-21T13:59:33.188890+0000","last_undegraded":"2024-09-21T13:59:33.188890+0000","last_fullsized":"2024-09-21T13:59:33.188890+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:58:41.644789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:58:41.644789+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:59:42.011578+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":400,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.190928+0000","last_change":"2024-09-21T13:59:32.208163+0000","last_active":"2024-09-21T13:59:33.190928+0000","last_peered":"2024-09-21T13:59:33.190928+0000","last_clean":"2024-09-21T13:59:33.190928+0000","last_became_active":"2024-09-21T13:59:32.207740+0000","last_became_peered":"2024-09-21T13:59:32.207740+0000","last_unstale":"2024-09-21T13:59:33.190928+0000","last_undegraded":"2024-09-21T13:59:33.190928+0000","last_fullsized":"2024-09-21T13:59:33.190928+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:10.320333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:10.320333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:59:37.979045+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":489,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-21T13:59:40.048809+0000","last_change":"2024-09-21T13:59:32.222791+0000","last_active":"2024-09-21T13:59:40.048809+0000","last_peered":"2024-09-21T13:59:40.048809+0000","last_clean":"2024-09-21T13:59:40.048809+0000","last_became_active":"2024-09-21T13:59:32.221502+0000","last_became_peered":"2024-09-21T13:59:32.221502+0000","last_unstale":"2024-09-21T13:59:40.048809+0000","last_undegraded":"2024-09-21T13:59:40.048809+0000","last_fullsized":"2024-09-21T13:59:40.048809+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:14.673582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:14.673582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"300'7744","reported_seq":11716,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:38.504558+0000","last_change":"2024-09-21T13:59:31.471106+0000","last_active":"2024-09-21T13:59:38.504558+0000","last_peered":"2024-09-21T13:59:38.504558+0000","last_clean":"2024-09-21T13:59:38.504558+0000","last_became_active":"2024-09-21T13:58:35.721126+0000","last_became_peered":"2024-09-21T13:58:35.721126+0000","last_unstale":"2024-09-21T13:59:38.504558+0000","last_undegraded":"2024-09-21T13:59:38.504558+0000","last_fullsized":"2024-09-21T13:59:38.504558+0000","mapping_epoch":278,"log_start":"266'1200","ondisk_log_start":"266'1200","created":260,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"298'7530","last_scrub_stamp":"2024-09-21T13:59:31.471024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:59:31.471024+0000","objects_scrubbed":7502,"log_size":6544,"log_dups_size":0,"ondisk_log_size":6544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:33.949758+0000","scrub_duration":1396,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":487587840,"num_objects":7440,"num_object_clones":0,"num_object_copies":22320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7744,"num_write_kb":485888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1892,"num_bytes_recovered":123994112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"290'24","reported_seq":597,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.194457+0000","last_change":"2024-09-21T13:59:28.458847+0000","last_active":"2024-09-21T13:59:33.194457+0000","last_peered":"2024-09-21T13:59:33.194457+0000","last_clean":"2024-09-21T13:59:33.194457+0000","last_became_active":"2024-09-21T13:52:04.318875+0000","last_became_peered":"2024-09-21T13:52:04.318875+0000","last_unstale":"2024-09-21T13:59:33.194457+0000","last_undegraded":"2024-09-21T13:59:33.194457+0000","last_fullsized":"2024-09-21T13:59:33.194457+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'24","last_scrub_stamp":"2024-09-21T13:59:28.458768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:28.458768+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:51.851382+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":452,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.207245+0000","last_change":"2024-09-21T13:57:20.447876+0000","last_active":"2024-09-21T13:59:33.207245+0000","last_peered":"2024-09-21T13:59:33.207245+0000","last_clean":"2024-09-21T13:59:33.207245+0000","last_became_active":"2024-09-21T13:57:12.681030+0000","last_became_peered":"2024-09-21T13:57:12.681030+0000","last_unstale":"2024-09-21T13:59:33.207245+0000","last_undegraded":"2024-09-21T13:59:33.207245+0000","last_fullsized":"2024-09-21T13:59:33.207245+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:20.447801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:20.447801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":1031,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-21T13:59:40.044855+0000","last_change":"2024-09-21T13:59:32.218796+0000","last_active":"2024-09-21T13:59:40.044855+0000","last_peered":"2024-09-21T13:59:40.044855+0000","last_clean":"2024-09-21T13:59:40.044855+0000","last_became_active":"2024-09-21T13:59:32.203361+0000","last_became_peered":"2024-09-21T13:59:32.203361+0000","last_unstale":"2024-09-21T13:59:40.044855+0000","last_undegraded":"2024-09-21T13:59:40.044855+0000","last_fullsized":"2024-09-21T13:59:40.044855+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-21T13:56:29.617143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:56:29.617143+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047048000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":501,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.192711+0000","last_change":"2024-09-21T13:56:33.316275+0000","last_active":"2024-09-21T13:59:33.192711+0000","last_peered":"2024-09-21T13:59:33.192711+0000","last_clean":"2024-09-21T13:59:33.192711+0000","last_became_active":"2024-09-21T13:49:02.066641+0000","last_became_peered":"2024-09-21T13:49:02.066641+0000","last_unstale":"2024-09-21T13:59:33.192711+0000","last_undegraded":"2024-09-21T13:59:33.192711+0000","last_fullsized":"2024-09-21T13:59:33.192711+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:33.316230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:56:33.316230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:08.146604+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":535,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-21T13:59:39.287048+0000","last_change":"2024-09-21T13:59:26.557174+0000","last_active":"2024-09-21T13:59:39.287048+0000","last_peered":"2024-09-21T13:59:39.287048+0000","last_clean":"2024-09-21T13:59:39.287048+0000","last_became_active":"2024-09-21T13:57:12.684254+0000","last_became_peered":"2024-09-21T13:57:12.684254+0000","last_unstale":"2024-09-21T13:59:39.287048+0000","last_undegraded":"2024-09-21T13:59:39.287048+0000","last_fullsized":"2024-09-21T13:59:39.287048+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:26.557002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:26.557002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:28.551613+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"302'7997","reported_seq":8092,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-21T13:59:40.925131+0000","last_change":"2024-09-21T13:57:53.863404+0000","last_active":"2024-09-21T13:59:40.925131+0000","last_peered":"2024-09-21T13:59:40.925131+0000","last_clean":"2024-09-21T13:59:40.925131+0000","last_became_active":"2024-09-21T13:57:53.863113+0000","last_became_peered":"2024-09-21T13:57:53.863113+0000","last_unstale":"2024-09-21T13:59:40.925131+0000","last_undegraded":"2024-09-21T13:59:40.925131+0000","last_fullsized":"2024-09-21T13:59:40.925131+0000","mapping_epoch":260,"log_start":"280'3300","ondisk_log_start":"280'3300","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4697,"log_dups_size":0,"ondisk_log_size":4697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485687296,"num_objects":7411,"num_object_clones":0,"num_object_copies":22233,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7997,"num_write_kb":493056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"300'32","reported_seq":152,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.315060+0000","last_change":"2024-09-21T13:59:32.243718+0000","last_active":"2024-09-21T13:59:33.315060+0000","last_peered":"2024-09-21T13:59:33.315060+0000","last_clean":"2024-09-21T13:59:33.315060+0000","last_became_active":"2024-09-21T13:59:32.204624+0000","last_became_peered":"2024-09-21T13:59:32.204624+0000","last_unstale":"2024-09-21T13:59:33.315060+0000","last_undegraded":"2024-09-21T13:59:33.315060+0000","last_fullsized":"2024-09-21T13:59:33.315060+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'16","last_scrub_stamp":"2024-09-21T13:52:19.452163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:52:19.452163+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":491,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-21T13:59:40.042117+0000","last_change":"2024-09-21T13:58:57.773043+0000","last_active":"2024-09-21T13:59:40.042117+0000","last_peered":"2024-09-21T13:59:40.042117+0000","last_clean":"2024-09-21T13:59:40.042117+0000","last_became_active":"2024-09-21T13:58:17.778951+0000","last_became_peered":"2024-09-21T13:58:17.778951+0000","last_unstale":"2024-09-21T13:59:40.042117+0000","last_undegraded":"2024-09-21T13:59:40.042117+0000","last_fullsized":"2024-09-21T13:59:40.042117+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:57.772975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:57.772975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:09.786990+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"75'79","reported_seq":553,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-21T13:59:40.045027+0000","last_change":"2024-09-21T13:59:32.202714+0000","last_active":"2024-09-21T13:59:40.045027+0000","last_peered":"2024-09-21T13:59:40.045027+0000","last_clean":"2024-09-21T13:59:40.045027+0000","last_became_active":"2024-09-21T13:59:32.201469+0000","last_became_peered":"2024-09-21T13:59:32.201469+0000","last_unstale":"2024-09-21T13:59:40.045027+0000","last_undegraded":"2024-09-21T13:59:40.045027+0000","last_fullsized":"2024-09-21T13:59:40.045027+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":466,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-21T13:59:40.047190+0000","last_change":"2024-09-21T13:59:32.213870+0000","last_active":"2024-09-21T13:59:40.047190+0000","last_peered":"2024-09-21T13:59:40.047190+0000","last_clean":"2024-09-21T13:59:40.047190+0000","last_became_active":"2024-09-21T13:59:32.212228+0000","last_became_peered":"2024-09-21T13:59:32.212228+0000","last_unstale":"2024-09-21T13:59:40.047190+0000","last_undegraded":"2024-09-21T13:59:40.047190+0000","last_fullsized":"2024-09-21T13:59:40.047190+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:15.700409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:15.700409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":508,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.192366+0000","last_change":"2024-09-21T13:57:21.153067+0000","last_active":"2024-09-21T13:59:33.192366+0000","last_peered":"2024-09-21T13:59:33.192366+0000","last_clean":"2024-09-21T13:59:33.192366+0000","last_became_active":"2024-09-21T13:57:12.684495+0000","last_became_peered":"2024-09-21T13:57:12.684495+0000","last_unstale":"2024-09-21T13:59:33.192366+0000","last_undegraded":"2024-09-21T13:59:33.192366+0000","last_fullsized":"2024-09-21T13:59:33.192366+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:21.153018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:21.153018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:18.188142+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"301'7805","reported_seq":7904,"reported_epoch":301,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2024-09-21T13:59:39.511839+0000","last_change":"2024-09-21T13:59:33.339074+0000","last_active":"2024-09-21T13:59:39.511839+0000","last_peered":"2024-09-21T13:59:39.511839+0000","last_clean":"2024-09-21T13:58:23.728526+0000","last_became_active":"2024-09-21T13:59:33.326654+0000","last_became_peered":"2024-09-21T13:59:33.326654+0000","last_unstale":"2024-09-21T13:59:39.511839+0000","last_undegraded":"2024-09-21T13:59:33.211255+0000","last_fullsized":"2024-09-21T13:59:33.189504+0000","mapping_epoch":299,"log_start":"277'3112","ondisk_log_start":"277'3112","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4693,"log_dups_size":0,"ondisk_log_size":4693,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482541568,"num_objects":7363,"num_object_clones":0,"num_object_copies":22089,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7805,"num_write_kb":485376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":7363}],"blocked_by":[],"up_primary":7,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"290'19","reported_seq":576,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.208089+0000","last_change":"2024-09-21T13:58:17.811300+0000","last_active":"2024-09-21T13:59:33.208089+0000","last_peered":"2024-09-21T13:59:33.208089+0000","last_clean":"2024-09-21T13:59:33.208089+0000","last_became_active":"2024-09-21T13:58:17.776310+0000","last_became_peered":"2024-09-21T13:58:17.776310+0000","last_unstale":"2024-09-21T13:59:33.208089+0000","last_undegraded":"2024-09-21T13:59:33.208089+0000","last_fullsized":"2024-09-21T13:59:33.208089+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:54:16.451588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:16.451588+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:13.578582+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":530,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.207371+0000","last_change":"2024-09-21T13:59:32.223608+0000","last_active":"2024-09-21T13:59:33.207371+0000","last_peered":"2024-09-21T13:59:33.207371+0000","last_clean":"2024-09-21T13:59:33.207371+0000","last_became_active":"2024-09-21T13:59:32.223247+0000","last_became_peered":"2024-09-21T13:59:32.223247+0000","last_unstale":"2024-09-21T13:59:33.207371+0000","last_undegraded":"2024-09-21T13:59:33.207371+0000","last_fullsized":"2024-09-21T13:59:33.207371+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:53.377100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:53.377100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":523,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.192257+0000","last_change":"2024-09-21T13:58:17.782835+0000","last_active":"2024-09-21T13:59:33.192257+0000","last_peered":"2024-09-21T13:59:33.192257+0000","last_clean":"2024-09-21T13:59:33.192257+0000","last_became_active":"2024-09-21T13:58:17.782403+0000","last_became_peered":"2024-09-21T13:58:17.782403+0000","last_unstale":"2024-09-21T13:59:33.192257+0000","last_undegraded":"2024-09-21T13:59:33.192257+0000","last_fullsized":"2024-09-21T13:59:33.192257+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:34.299722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:55:34.299722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:59:37.325475+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":469,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-21T13:59:40.042327+0000","last_change":"2024-09-21T13:58:17.778317+0000","last_active":"2024-09-21T13:59:40.042327+0000","last_peered":"2024-09-21T13:59:40.042327+0000","last_clean":"2024-09-21T13:59:40.042327+0000","last_became_active":"2024-09-21T13:58:17.777954+0000","last_became_peered":"2024-09-21T13:58:17.777954+0000","last_unstale":"2024-09-21T13:59:40.042327+0000","last_undegraded":"2024-09-21T13:59:40.042327+0000","last_fullsized":"2024-09-21T13:59:40.042327+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:58.952218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:54:58.952218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:13.848511+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.0","version":"301'7739","reported_seq":11771,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-21T13:59:39.516315+0000","last_change":"2024-09-21T13:59:24.817101+0000","last_active":"2024-09-21T13:59:39.516315+0000","last_peered":"2024-09-21T13:59:39.516315+0000","last_clean":"2024-09-21T13:59:39.516315+0000","last_became_active":"2024-09-21T13:58:36.723110+0000","last_became_peered":"2024-09-21T13:58:36.723110+0000","last_unstale":"2024-09-21T13:59:39.516315+0000","last_undegraded":"2024-09-21T13:59:39.516315+0000","last_fullsized":"2024-09-21T13:59:39.516315+0000","mapping_epoch":279,"log_start":"277'3047","ondisk_log_start":"277'3047","created":260,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'7035","last_scrub_stamp":"2024-09-21T13:59:24.816903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:59:24.816903+0000","objects_scrubbed":6955,"log_size":4692,"log_dups_size":0,"ondisk_log_size":4692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:30.971537+0000","scrub_duration":2446,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":480575488,"num_objects":7333,"num_object_clones":0,"num_object_copies":21999,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7739,"num_write_kb":482304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1924,"num_bytes_recovered":126091264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"290'31","reported_seq":564,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.208153+0000","last_change":"2024-09-21T13:56:55.354217+0000","last_active":"2024-09-21T13:59:33.208153+0000","last_peered":"2024-09-21T13:59:33.208153+0000","last_clean":"2024-09-21T13:59:33.208153+0000","last_became_active":"2024-09-21T13:52:04.313218+0000","last_became_peered":"2024-09-21T13:52:04.313218+0000","last_unstale":"2024-09-21T13:59:33.208153+0000","last_undegraded":"2024-09-21T13:59:33.208153+0000","last_fullsized":"2024-09-21T13:59:33.208153+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'17","last_scrub_stamp":"2024-09-21T13:56:55.354126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:56:55.354126+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":72,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.248401+0000","last_change":"2024-09-21T13:58:17.762896+0000","last_active":"2024-09-21T13:59:33.248401+0000","last_peered":"2024-09-21T13:59:33.248401+0000","last_clean":"2024-09-21T13:59:33.248401+0000","last_became_active":"2024-09-21T13:58:17.762520+0000","last_became_peered":"2024-09-21T13:58:17.762520+0000","last_unstale":"2024-09-21T13:59:33.248401+0000","last_undegraded":"2024-09-21T13:59:33.248401+0000","last_fullsized":"2024-09-21T13:59:33.248401+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:04.216734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:55:04.216734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:04.858083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":503,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-21T13:59:39.287154+0000","last_change":"2024-09-21T13:58:52.030800+0000","last_active":"2024-09-21T13:59:39.287154+0000","last_peered":"2024-09-21T13:59:39.287154+0000","last_clean":"2024-09-21T13:59:39.287154+0000","last_became_active":"2024-09-21T13:49:02.067472+0000","last_became_peered":"2024-09-21T13:49:02.067472+0000","last_unstale":"2024-09-21T13:59:39.287154+0000","last_undegraded":"2024-09-21T13:59:39.287154+0000","last_fullsized":"2024-09-21T13:59:39.287154+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:52.030718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:52.030718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:59:55.779410+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":384,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.188709+0000","last_change":"2024-09-21T13:57:12.683321+0000","last_active":"2024-09-21T13:59:33.188709+0000","last_peered":"2024-09-21T13:59:33.188709+0000","last_clean":"2024-09-21T13:59:33.188709+0000","last_became_active":"2024-09-21T13:57:12.682979+0000","last_became_peered":"2024-09-21T13:57:12.682979+0000","last_unstale":"2024-09-21T13:59:33.188709+0000","last_undegraded":"2024-09-21T13:59:33.188709+0000","last_fullsized":"2024-09-21T13:59:33.188709+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:18.027944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:18.027944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:18.700579+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"302'7825","reported_seq":13652,"reported_epoch":302,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-21T13:59:40.912403+0000","last_change":"2024-09-21T13:59:33.434313+0000","last_active":"2024-09-21T13:59:40.912403+0000","last_peered":"2024-09-21T13:59:40.912403+0000","last_clean":"2024-09-21T13:59:31.188333+0000","last_became_active":"2024-09-21T13:59:32.273094+0000","last_became_peered":"2024-09-21T13:59:32.273094+0000","last_unstale":"2024-09-21T13:59:40.912403+0000","last_undegraded":"2024-09-21T13:59:40.912403+0000","last_fullsized":"2024-09-21T13:59:40.912403+0000","mapping_epoch":298,"log_start":"277'3100","ondisk_log_start":"277'3100","created":260,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4725,"log_dups_size":0,"ondisk_log_size":4725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":477298688,"num_objects":7283,"num_object_clones":0,"num_object_copies":21849,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6521,"num_objects_unfound":0,"num_objects_dirty":7283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7825,"num_write_kb":483456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2852,"num_bytes_recovered":186908672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,4],"avail_no_missing":["3","4","6"],"object_location_counts":[{"shards":"3,4,6","objects":7283}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"290'20","reported_seq":591,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.207985+0000","last_change":"2024-09-21T13:58:17.789407+0000","last_active":"2024-09-21T13:59:33.207985+0000","last_peered":"2024-09-21T13:59:33.207985+0000","last_clean":"2024-09-21T13:59:33.207985+0000","last_became_active":"2024-09-21T13:58:17.765427+0000","last_became_peered":"2024-09-21T13:58:17.765427+0000","last_unstale":"2024-09-21T13:59:33.207985+0000","last_undegraded":"2024-09-21T13:59:33.207985+0000","last_fullsized":"2024-09-21T13:59:33.207985+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:58:13.464196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:13.464196+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":437,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-21T13:59:40.041939+0000","last_change":"2024-09-21T13:57:36.707211+0000","last_active":"2024-09-21T13:59:40.041939+0000","last_peered":"2024-09-21T13:59:40.041939+0000","last_clean":"2024-09-21T13:59:40.041939+0000","last_became_active":"2024-09-21T13:57:12.704203+0000","last_became_peered":"2024-09-21T13:57:12.704203+0000","last_unstale":"2024-09-21T13:59:40.041939+0000","last_undegraded":"2024-09-21T13:59:40.041939+0000","last_fullsized":"2024-09-21T13:59:40.041939+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:36.707148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:36.707148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"300'12","reported_seq":503,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.189267+0000","last_change":"2024-09-21T13:59:32.225072+0000","last_active":"2024-09-21T13:59:33.189267+0000","last_peered":"2024-09-21T13:59:33.189267+0000","last_clean":"2024-09-21T13:59:33.189267+0000","last_became_active":"2024-09-21T13:59:32.224809+0000","last_became_peered":"2024-09-21T13:59:32.224809+0000","last_unstale":"2024-09-21T13:59:33.189267+0000","last_undegraded":"2024-09-21T13:59:33.189267+0000","last_fullsized":"2024-09-21T13:59:33.189267+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'10","last_scrub_stamp":"2024-09-21T13:57:40.261523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:40.261523+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"68'1","reported_seq":519,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.207916+0000","last_change":"2024-09-21T13:59:31.392186+0000","last_active":"2024-09-21T13:59:33.207916+0000","last_peered":"2024-09-21T13:59:33.207916+0000","last_clean":"2024-09-21T13:59:33.207916+0000","last_became_active":"2024-09-21T13:52:04.313210+0000","last_became_peered":"2024-09-21T13:52:04.313210+0000","last_unstale":"2024-09-21T13:59:33.207916+0000","last_undegraded":"2024-09-21T13:59:33.207916+0000","last_fullsized":"2024-09-21T13:59:33.207916+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T13:59:31.392105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:31.392105+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:01:00.512112+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"300'7603","reported_seq":7694,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:38.611773+0000","last_change":"2024-09-21T13:57:53.213656+0000","last_active":"2024-09-21T13:59:38.611773+0000","last_peered":"2024-09-21T13:59:38.611773+0000","last_clean":"2024-09-21T13:59:38.611773+0000","last_became_active":"2024-09-21T13:57:53.213314+0000","last_became_peered":"2024-09-21T13:57:53.213314+0000","last_unstale":"2024-09-21T13:59:38.611773+0000","last_undegraded":"2024-09-21T13:59:38.611773+0000","last_fullsized":"2024-09-21T13:59:38.611773+0000","mapping_epoch":260,"log_start":"272'2300","ondisk_log_start":"272'2300","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":5303,"log_dups_size":0,"ondisk_log_size":5303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":477560832,"num_objects":7287,"num_object_clones":0,"num_object_copies":21861,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7603,"num_write_kb":476480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"290'15","reported_seq":546,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.195516+0000","last_change":"2024-09-21T13:58:50.498597+0000","last_active":"2024-09-21T13:59:33.195516+0000","last_peered":"2024-09-21T13:59:33.195516+0000","last_clean":"2024-09-21T13:59:33.195516+0000","last_became_active":"2024-09-21T13:49:00.053016+0000","last_became_peered":"2024-09-21T13:49:00.053016+0000","last_unstale":"2024-09-21T13:59:33.195516+0000","last_undegraded":"2024-09-21T13:59:33.195516+0000","last_fullsized":"2024-09-21T13:59:33.195516+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:58:50.498369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:50.498369+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:09.820421+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":534,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.192005+0000","last_change":"2024-09-21T13:58:17.783765+0000","last_active":"2024-09-21T13:59:33.192005+0000","last_peered":"2024-09-21T13:59:33.192005+0000","last_clean":"2024-09-21T13:59:33.192005+0000","last_became_active":"2024-09-21T13:58:17.782747+0000","last_became_peered":"2024-09-21T13:58:17.782747+0000","last_unstale":"2024-09-21T13:59:33.192005+0000","last_undegraded":"2024-09-21T13:59:33.192005+0000","last_fullsized":"2024-09-21T13:59:33.192005+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:53.894506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:53.894506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:59:34.368653+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":442,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.207462+0000","last_change":"2024-09-21T13:57:12.683161+0000","last_active":"2024-09-21T13:59:33.207462+0000","last_peered":"2024-09-21T13:59:33.207462+0000","last_clean":"2024-09-21T13:59:33.207462+0000","last_became_active":"2024-09-21T13:57:12.680978+0000","last_became_peered":"2024-09-21T13:57:12.680978+0000","last_unstale":"2024-09-21T13:59:33.207462+0000","last_undegraded":"2024-09-21T13:59:33.207462+0000","last_fullsized":"2024-09-21T13:59:33.207462+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:21.402946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:21.402946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:08.693286+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":412,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-21T13:59:40.044771+0000","last_change":"2024-09-21T13:59:32.224001+0000","last_active":"2024-09-21T13:59:40.044771+0000","last_peered":"2024-09-21T13:59:40.044771+0000","last_clean":"2024-09-21T13:59:40.044771+0000","last_became_active":"2024-09-21T13:59:32.221628+0000","last_became_peered":"2024-09-21T13:59:32.221628+0000","last_unstale":"2024-09-21T13:59:40.044771+0000","last_undegraded":"2024-09-21T13:59:40.044771+0000","last_fullsized":"2024-09-21T13:59:40.044771+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:13.698091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:13.698091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"300'7967","reported_seq":8486,"reported_epoch":300,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-21T13:59:38.687071+0000","last_change":"2024-09-21T13:59:33.486474+0000","last_active":"2024-09-21T13:59:38.687071+0000","last_peered":"2024-09-21T13:59:38.687071+0000","last_clean":"2024-09-21T13:58:23.791074+0000","last_became_active":"2024-09-21T13:59:33.441400+0000","last_became_peered":"2024-09-21T13:59:33.441400+0000","last_unstale":"2024-09-21T13:59:38.687071+0000","last_undegraded":"2024-09-21T13:59:33.239445+0000","last_fullsized":"2024-09-21T13:59:33.192362+0000","mapping_epoch":299,"log_start":"275'2700","ondisk_log_start":"275'2700","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":5267,"log_dups_size":0,"ondisk_log_size":5267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":500891648,"num_objects":7643,"num_object_clones":0,"num_object_copies":22929,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7967,"num_write_kb":499520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":13762560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[{"shards":"3,5","objects":2599}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"290'35","reported_seq":146,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.248559+0000","last_change":"2024-09-21T13:59:32.234258+0000","last_active":"2024-09-21T13:59:33.248559+0000","last_peered":"2024-09-21T13:59:33.248559+0000","last_clean":"2024-09-21T13:59:33.248559+0000","last_became_active":"2024-09-21T13:59:32.204184+0000","last_became_peered":"2024-09-21T13:59:32.204184+0000","last_unstale":"2024-09-21T13:59:33.248559+0000","last_undegraded":"2024-09-21T13:59:33.248559+0000","last_fullsized":"2024-09-21T13:59:33.248559+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-21T13:58:12.650656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:12.650656+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:59:39.247813+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":493,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-21T13:59:40.049212+0000","last_change":"2024-09-21T13:59:32.222536+0000","last_active":"2024-09-21T13:59:40.049212+0000","last_peered":"2024-09-21T13:59:40.049212+0000","last_clean":"2024-09-21T13:59:40.049212+0000","last_became_active":"2024-09-21T13:59:32.221051+0000","last_became_peered":"2024-09-21T13:59:32.221051+0000","last_unstale":"2024-09-21T13:59:40.049212+0000","last_undegraded":"2024-09-21T13:59:40.049212+0000","last_fullsized":"2024-09-21T13:59:40.049212+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:21.587493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:21.587493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":467,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.189387+0000","last_change":"2024-09-21T13:58:55.588614+0000","last_active":"2024-09-21T13:59:33.189387+0000","last_peered":"2024-09-21T13:59:33.189387+0000","last_clean":"2024-09-21T13:59:33.189387+0000","last_became_active":"2024-09-21T13:52:04.321146+0000","last_became_peered":"2024-09-21T13:52:04.321146+0000","last_unstale":"2024-09-21T13:59:33.189387+0000","last_undegraded":"2024-09-21T13:59:33.189387+0000","last_fullsized":"2024-09-21T13:59:33.189387+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:55.588554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:55.588554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:23.995144+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":431,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-21T13:59:40.044945+0000","last_change":"2024-09-21T13:59:32.223123+0000","last_active":"2024-09-21T13:59:40.044945+0000","last_peered":"2024-09-21T13:59:40.044945+0000","last_clean":"2024-09-21T13:59:40.044945+0000","last_became_active":"2024-09-21T13:59:32.219332+0000","last_became_peered":"2024-09-21T13:59:32.219332+0000","last_unstale":"2024-09-21T13:59:40.044945+0000","last_undegraded":"2024-09-21T13:59:40.044945+0000","last_fullsized":"2024-09-21T13:59:40.044945+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:19.537177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:19.537177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"301'7647","reported_seq":7740,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-21T13:59:39.505330+0000","last_change":"2024-09-21T13:57:53.515705+0000","last_active":"2024-09-21T13:59:39.505330+0000","last_peered":"2024-09-21T13:59:39.505330+0000","last_clean":"2024-09-21T13:59:39.505330+0000","last_became_active":"2024-09-21T13:57:53.515313+0000","last_became_peered":"2024-09-21T13:57:53.515313+0000","last_unstale":"2024-09-21T13:59:39.505330+0000","last_undegraded":"2024-09-21T13:59:39.505330+0000","last_fullsized":"2024-09-21T13:59:39.505330+0000","mapping_epoch":260,"log_start":"277'2970","ondisk_log_start":"277'2970","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4677,"log_dups_size":0,"ondisk_log_size":4677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":473235456,"num_objects":7221,"num_object_clones":0,"num_object_copies":21663,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7647,"num_write_kb":475776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"290'18","reported_seq":563,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.207307+0000","last_change":"2024-09-21T13:59:28.340494+0000","last_active":"2024-09-21T13:59:33.207307+0000","last_peered":"2024-09-21T13:59:33.207307+0000","last_clean":"2024-09-21T13:59:33.207307+0000","last_became_active":"2024-09-21T13:57:12.680627+0000","last_became_peered":"2024-09-21T13:57:12.680627+0000","last_unstale":"2024-09-21T13:59:33.207307+0000","last_undegraded":"2024-09-21T13:59:33.207307+0000","last_fullsized":"2024-09-21T13:59:33.207307+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'18","last_scrub_stamp":"2024-09-21T13:59:28.340415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:28.340415+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:44.818651+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":459,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-21T13:59:40.045629+0000","last_change":"2024-09-21T13:59:32.201126+0000","last_active":"2024-09-21T13:59:40.045629+0000","last_peered":"2024-09-21T13:59:40.045629+0000","last_clean":"2024-09-21T13:59:40.045629+0000","last_became_active":"2024-09-21T13:59:32.200674+0000","last_became_peered":"2024-09-21T13:59:32.200674+0000","last_unstale":"2024-09-21T13:59:40.045629+0000","last_undegraded":"2024-09-21T13:59:40.045629+0000","last_fullsized":"2024-09-21T13:59:40.045629+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:21.582325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:21.582325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":432,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.189438+0000","last_change":"2024-09-21T13:58:11.377940+0000","last_active":"2024-09-21T13:59:33.189438+0000","last_peered":"2024-09-21T13:59:33.189438+0000","last_clean":"2024-09-21T13:59:33.189438+0000","last_became_active":"2024-09-21T13:52:04.322147+0000","last_became_peered":"2024-09-21T13:52:04.322147+0000","last_unstale":"2024-09-21T13:59:33.189438+0000","last_undegraded":"2024-09-21T13:59:33.189438+0000","last_fullsized":"2024-09-21T13:59:33.189438+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:11.377803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:11.377803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:19.656170+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":445,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-21T13:59:40.042394+0000","last_change":"2024-09-21T13:59:32.207006+0000","last_active":"2024-09-21T13:59:40.042394+0000","last_peered":"2024-09-21T13:59:40.042394+0000","last_clean":"2024-09-21T13:59:40.042394+0000","last_became_active":"2024-09-21T13:59:32.203767+0000","last_became_peered":"2024-09-21T13:59:32.203767+0000","last_unstale":"2024-09-21T13:59:40.042394+0000","last_undegraded":"2024-09-21T13:59:40.042394+0000","last_fullsized":"2024-09-21T13:59:40.042394+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:09.040062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:55:09.040062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"300'7980","reported_seq":8476,"reported_epoch":300,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-21T13:59:38.674742+0000","last_change":"2024-09-21T13:59:33.415030+0000","last_active":"2024-09-21T13:59:38.674742+0000","last_peered":"2024-09-21T13:59:38.674742+0000","last_clean":"2024-09-21T13:58:23.785212+0000","last_became_active":"2024-09-21T13:59:33.394359+0000","last_became_peered":"2024-09-21T13:59:33.394359+0000","last_unstale":"2024-09-21T13:59:38.674742+0000","last_undegraded":"2024-09-21T13:59:33.231231+0000","last_fullsized":"2024-09-21T13:59:33.191984+0000","mapping_epoch":299,"log_start":"275'2700","ondisk_log_start":"275'2700","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":5280,"log_dups_size":0,"ondisk_log_size":5280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":501350400,"num_objects":7650,"num_object_clones":0,"num_object_copies":22950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7980,"num_write_kb":500160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":198,"num_bytes_recovered":12976128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[{"shards":"0,5","objects":2590}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"290'27","reported_seq":636,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.191592+0000","last_change":"2024-09-21T13:59:33.058912+0000","last_active":"2024-09-21T13:59:33.191592+0000","last_peered":"2024-09-21T13:59:33.191592+0000","last_clean":"2024-09-21T13:59:33.191592+0000","last_became_active":"2024-09-21T13:59:32.205453+0000","last_became_peered":"2024-09-21T13:59:32.205453+0000","last_unstale":"2024-09-21T13:59:33.191592+0000","last_undegraded":"2024-09-21T13:59:33.191592+0000","last_fullsized":"2024-09-21T13:59:33.191592+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'27","last_scrub_stamp":"2024-09-21T13:59:33.058859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:33.058859+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:35.203746+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":431,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-21T13:59:40.045657+0000","last_change":"2024-09-21T13:59:32.203889+0000","last_active":"2024-09-21T13:59:40.045657+0000","last_peered":"2024-09-21T13:59:40.045657+0000","last_clean":"2024-09-21T13:59:40.045657+0000","last_became_active":"2024-09-21T13:59:32.203172+0000","last_became_peered":"2024-09-21T13:59:32.203172+0000","last_unstale":"2024-09-21T13:59:40.045657+0000","last_undegraded":"2024-09-21T13:59:40.045657+0000","last_fullsized":"2024-09-21T13:59:40.045657+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T13:55:03.879706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:55:03.879706+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"300'14","reported_seq":607,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-21T13:59:40.044625+0000","last_change":"2024-09-21T13:59:32.220898+0000","last_active":"2024-09-21T13:59:40.044625+0000","last_peered":"2024-09-21T13:59:40.044625+0000","last_clean":"2024-09-21T13:59:40.044625+0000","last_became_active":"2024-09-21T13:59:32.206956+0000","last_became_peered":"2024-09-21T13:59:32.206956+0000","last_unstale":"2024-09-21T13:59:40.044625+0000","last_undegraded":"2024-09-21T13:59:40.044625+0000","last_fullsized":"2024-09-21T13:59:40.044625+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'11","last_scrub_stamp":"2024-09-21T13:56:31.616003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:56:31.616003+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":535,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.237440+0000","last_change":"2024-09-21T13:58:39.258717+0000","last_active":"2024-09-21T13:59:33.237440+0000","last_peered":"2024-09-21T13:59:33.237440+0000","last_clean":"2024-09-21T13:59:33.237440+0000","last_became_active":"2024-09-21T13:58:17.781952+0000","last_became_peered":"2024-09-21T13:58:17.781952+0000","last_unstale":"2024-09-21T13:59:33.237440+0000","last_undegraded":"2024-09-21T13:59:33.237440+0000","last_fullsized":"2024-09-21T13:59:33.237440+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:39.258659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:39.258659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:59:54.287146+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":72,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.248435+0000","last_change":"2024-09-21T13:58:17.777374+0000","last_active":"2024-09-21T13:59:33.248435+0000","last_peered":"2024-09-21T13:59:33.248435+0000","last_clean":"2024-09-21T13:59:33.248435+0000","last_became_active":"2024-09-21T13:58:17.777025+0000","last_became_peered":"2024-09-21T13:58:17.777025+0000","last_unstale":"2024-09-21T13:59:33.248435+0000","last_undegraded":"2024-09-21T13:59:33.248435+0000","last_fullsized":"2024-09-21T13:59:33.248435+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:13.252516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:13.252516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"290'12","reported_seq":480,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-21T13:59:40.043253+0000","last_change":"2024-09-21T13:59:32.247422+0000","last_active":"2024-09-21T13:59:40.043253+0000","last_peered":"2024-09-21T13:59:40.043253+0000","last_clean":"2024-09-21T13:59:40.043253+0000","last_became_active":"2024-09-21T13:59:32.211306+0000","last_became_peered":"2024-09-21T13:59:32.211306+0000","last_unstale":"2024-09-21T13:59:40.043253+0000","last_undegraded":"2024-09-21T13:59:40.043253+0000","last_fullsized":"2024-09-21T13:59:40.043253+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'6","last_scrub_stamp":"2024-09-21T13:55:12.066873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:55:12.066873+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":528,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.192258+0000","last_change":"2024-09-21T13:57:37.106286+0000","last_active":"2024-09-21T13:59:33.192258+0000","last_peered":"2024-09-21T13:59:33.192258+0000","last_clean":"2024-09-21T13:59:33.192258+0000","last_became_active":"2024-09-21T13:57:12.702621+0000","last_became_peered":"2024-09-21T13:57:12.702621+0000","last_unstale":"2024-09-21T13:59:33.192258+0000","last_undegraded":"2024-09-21T13:59:33.192258+0000","last_fullsized":"2024-09-21T13:59:33.192258+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:37.106238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:37.106238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":477,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-21T13:59:40.049042+0000","last_change":"2024-09-21T13:59:32.222277+0000","last_active":"2024-09-21T13:59:40.049042+0000","last_peered":"2024-09-21T13:59:40.049042+0000","last_clean":"2024-09-21T13:59:40.049042+0000","last_became_active":"2024-09-21T13:59:32.219525+0000","last_became_peered":"2024-09-21T13:59:32.219525+0000","last_unstale":"2024-09-21T13:59:40.049042+0000","last_undegraded":"2024-09-21T13:59:40.049042+0000","last_fullsized":"2024-09-21T13:59:40.049042+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:02.838181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:55:02.838181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"290'15","reported_seq":443,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-21T13:59:40.041104+0000","last_change":"2024-09-21T13:59:32.238689+0000","last_active":"2024-09-21T13:59:40.041104+0000","last_peered":"2024-09-21T13:59:40.041104+0000","last_clean":"2024-09-21T13:59:40.041104+0000","last_became_active":"2024-09-21T13:59:32.209166+0000","last_became_peered":"2024-09-21T13:59:32.209166+0000","last_unstale":"2024-09-21T13:59:40.041104+0000","last_undegraded":"2024-09-21T13:59:40.041104+0000","last_fullsized":"2024-09-21T13:59:40.041104+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:55:04.990842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:55:04.990842+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"61'1","reported_seq":495,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-21T13:59:40.044994+0000","last_change":"2024-09-21T13:59:32.222826+0000","last_active":"2024-09-21T13:59:40.044994+0000","last_peered":"2024-09-21T13:59:40.044994+0000","last_clean":"2024-09-21T13:59:40.044994+0000","last_became_active":"2024-09-21T13:59:32.219515+0000","last_became_peered":"2024-09-21T13:59:32.219515+0000","last_unstale":"2024-09-21T13:59:40.044994+0000","last_undegraded":"2024-09-21T13:59:40.044994+0000","last_fullsized":"2024-09-21T13:59:40.044994+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:58:07.711028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:07.711028+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":428,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-21T13:59:40.042615+0000","last_change":"2024-09-21T13:59:32.211754+0000","last_active":"2024-09-21T13:59:40.042615+0000","last_peered":"2024-09-21T13:59:40.042615+0000","last_clean":"2024-09-21T13:59:40.042615+0000","last_became_active":"2024-09-21T13:59:32.210586+0000","last_became_peered":"2024-09-21T13:59:32.210586+0000","last_unstale":"2024-09-21T13:59:40.042615+0000","last_undegraded":"2024-09-21T13:59:40.042615+0000","last_fullsized":"2024-09-21T13:59:40.042615+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:28.014760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:28.014760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":466,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:36.677306+0000","last_change":"2024-09-21T13:59:36.677306+0000","last_active":"2024-09-21T13:59:36.677306+0000","last_peered":"2024-09-21T13:59:36.677306+0000","last_clean":"2024-09-21T13:59:36.677306+0000","last_became_active":"2024-09-21T13:59:32.205803+0000","last_became_peered":"2024-09-21T13:59:32.205803+0000","last_unstale":"2024-09-21T13:59:36.677306+0000","last_undegraded":"2024-09-21T13:59:36.677306+0000","last_fullsized":"2024-09-21T13:59:36.677306+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:36.677259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:36.677259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:49.697302+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"290'15","reported_seq":561,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.192540+0000","last_change":"2024-09-21T13:59:32.247060+0000","last_active":"2024-09-21T13:59:33.192540+0000","last_peered":"2024-09-21T13:59:33.192540+0000","last_clean":"2024-09-21T13:59:33.192540+0000","last_became_active":"2024-09-21T13:59:32.226975+0000","last_became_peered":"2024-09-21T13:59:32.226975+0000","last_unstale":"2024-09-21T13:59:33.192540+0000","last_undegraded":"2024-09-21T13:59:33.192540+0000","last_fullsized":"2024-09-21T13:59:33.192540+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:57:24.124604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:57:24.124604+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":364,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-21T13:59:40.046080+0000","last_change":"2024-09-21T13:59:32.212325+0000","last_active":"2024-09-21T13:59:40.046080+0000","last_peered":"2024-09-21T13:59:40.046080+0000","last_clean":"2024-09-21T13:59:40.046080+0000","last_became_active":"2024-09-21T13:59:32.208749+0000","last_became_peered":"2024-09-21T13:59:32.208749+0000","last_unstale":"2024-09-21T13:59:40.046080+0000","last_undegraded":"2024-09-21T13:59:40.046080+0000","last_fullsized":"2024-09-21T13:59:40.046080+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:20.574745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:20.574745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"300'14","reported_seq":540,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-21T13:59:40.042399+0000","last_change":"2024-09-21T13:59:32.208253+0000","last_active":"2024-09-21T13:59:40.042399+0000","last_peered":"2024-09-21T13:59:40.042399+0000","last_clean":"2024-09-21T13:59:40.042399+0000","last_became_active":"2024-09-21T13:59:32.206926+0000","last_became_peered":"2024-09-21T13:59:32.206926+0000","last_unstale":"2024-09-21T13:59:40.042399+0000","last_undegraded":"2024-09-21T13:59:40.042399+0000","last_fullsized":"2024-09-21T13:59:40.042399+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'10","last_scrub_stamp":"2024-09-21T13:54:20.946782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:20.946782+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":515,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.208070+0000","last_change":"2024-09-21T13:58:54.805670+0000","last_active":"2024-09-21T13:59:33.208070+0000","last_peered":"2024-09-21T13:59:33.208070+0000","last_clean":"2024-09-21T13:59:33.208070+0000","last_became_active":"2024-09-21T13:52:04.315865+0000","last_became_peered":"2024-09-21T13:52:04.315865+0000","last_unstale":"2024-09-21T13:59:33.208070+0000","last_undegraded":"2024-09-21T13:59:33.208070+0000","last_fullsized":"2024-09-21T13:59:33.208070+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:54.805619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:54.805619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:05.176950+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"290'20","reported_seq":479,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-21T13:59:40.042302+0000","last_change":"2024-09-21T13:59:32.236530+0000","last_active":"2024-09-21T13:59:40.042302+0000","last_peered":"2024-09-21T13:59:40.042302+0000","last_clean":"2024-09-21T13:59:40.042302+0000","last_became_active":"2024-09-21T13:59:32.208332+0000","last_became_peered":"2024-09-21T13:59:32.208332+0000","last_unstale":"2024-09-21T13:59:40.042302+0000","last_undegraded":"2024-09-21T13:59:40.042302+0000","last_fullsized":"2024-09-21T13:59:40.042302+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:57:37.671391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:57:37.671391+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":471,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-21T13:59:40.047957+0000","last_change":"2024-09-21T13:59:32.212737+0000","last_active":"2024-09-21T13:59:40.047957+0000","last_peered":"2024-09-21T13:59:40.047957+0000","last_clean":"2024-09-21T13:59:40.047957+0000","last_became_active":"2024-09-21T13:59:32.209804+0000","last_became_peered":"2024-09-21T13:59:32.209804+0000","last_unstale":"2024-09-21T13:59:40.047957+0000","last_undegraded":"2024-09-21T13:59:40.047957+0000","last_fullsized":"2024-09-21T13:59:40.047957+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:25.571256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:25.571256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"68'8","reported_seq":646,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.245920+0000","last_change":"2024-09-21T13:59:32.104294+0000","last_active":"2024-09-21T13:59:33.245920+0000","last_peered":"2024-09-21T13:59:33.245920+0000","last_clean":"2024-09-21T13:59:33.245920+0000","last_became_active":"2024-09-21T13:49:02.067487+0000","last_became_peered":"2024-09-21T13:49:02.067487+0000","last_unstale":"2024-09-21T13:59:33.245920+0000","last_undegraded":"2024-09-21T13:59:33.245920+0000","last_fullsized":"2024-09-21T13:59:33.245920+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:59:32.104224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:32.104224+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:36.249102+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":368,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-21T13:59:40.045461+0000","last_change":"2024-09-21T13:59:32.213015+0000","last_active":"2024-09-21T13:59:40.045461+0000","last_peered":"2024-09-21T13:59:40.045461+0000","last_clean":"2024-09-21T13:59:40.045461+0000","last_became_active":"2024-09-21T13:59:32.211472+0000","last_became_peered":"2024-09-21T13:59:32.211472+0000","last_unstale":"2024-09-21T13:59:40.045461+0000","last_undegraded":"2024-09-21T13:59:40.045461+0000","last_fullsized":"2024-09-21T13:59:40.045461+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:22.647191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:22.647191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"290'19","reported_seq":581,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.207376+0000","last_change":"2024-09-21T13:58:17.790158+0000","last_active":"2024-09-21T13:59:33.207376+0000","last_peered":"2024-09-21T13:59:33.207376+0000","last_clean":"2024-09-21T13:59:33.207376+0000","last_became_active":"2024-09-21T13:58:17.771497+0000","last_became_peered":"2024-09-21T13:58:17.771497+0000","last_unstale":"2024-09-21T13:59:33.207376+0000","last_undegraded":"2024-09-21T13:59:33.207376+0000","last_fullsized":"2024-09-21T13:59:33.207376+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:54:56.539374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:56.539374+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:16.539347+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":105,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-21T13:59:40.047261+0000","last_change":"2024-09-21T13:59:32.202097+0000","last_active":"2024-09-21T13:59:40.047261+0000","last_peered":"2024-09-21T13:59:40.047261+0000","last_clean":"2024-09-21T13:59:40.047261+0000","last_became_active":"2024-09-21T13:59:32.201568+0000","last_became_peered":"2024-09-21T13:59:32.201568+0000","last_unstale":"2024-09-21T13:59:40.047261+0000","last_undegraded":"2024-09-21T13:59:40.047261+0000","last_fullsized":"2024-09-21T13:59:40.047261+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:01.228213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:01.228213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":530,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-21T13:59:39.288348+0000","last_change":"2024-09-21T13:57:35.409292+0000","last_active":"2024-09-21T13:59:39.288348+0000","last_peered":"2024-09-21T13:59:39.288348+0000","last_clean":"2024-09-21T13:59:39.288348+0000","last_became_active":"2024-09-21T13:57:12.683646+0000","last_became_peered":"2024-09-21T13:57:12.683646+0000","last_unstale":"2024-09-21T13:59:39.288348+0000","last_undegraded":"2024-09-21T13:59:39.288348+0000","last_fullsized":"2024-09-21T13:59:39.288348+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:35.409237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:35.409237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:09.229079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":425,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-21T13:59:40.040190+0000","last_change":"2024-09-21T13:57:39.643527+0000","last_active":"2024-09-21T13:59:40.040190+0000","last_peered":"2024-09-21T13:59:40.040190+0000","last_clean":"2024-09-21T13:59:40.040190+0000","last_became_active":"2024-09-21T13:57:12.703037+0000","last_became_peered":"2024-09-21T13:57:12.703037+0000","last_unstale":"2024-09-21T13:59:40.040190+0000","last_undegraded":"2024-09-21T13:59:40.040190+0000","last_fullsized":"2024-09-21T13:59:40.040190+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:39.643458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:39.643458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"290'15","reported_seq":585,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-21T13:59:39.288578+0000","last_change":"2024-09-21T13:58:39.977717+0000","last_active":"2024-09-21T13:59:39.288578+0000","last_peered":"2024-09-21T13:59:39.288578+0000","last_clean":"2024-09-21T13:59:39.288578+0000","last_became_active":"2024-09-21T13:58:17.764120+0000","last_became_peered":"2024-09-21T13:58:17.764120+0000","last_unstale":"2024-09-21T13:59:39.288578+0000","last_undegraded":"2024-09-21T13:59:39.288578+0000","last_fullsized":"2024-09-21T13:59:39.288578+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:58:39.977466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:39.977466+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:03.738519+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":475,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-21T13:59:40.048893+0000","last_change":"2024-09-21T13:59:32.223082+0000","last_active":"2024-09-21T13:59:40.048893+0000","last_peered":"2024-09-21T13:59:40.048893+0000","last_clean":"2024-09-21T13:59:40.048893+0000","last_became_active":"2024-09-21T13:59:32.221854+0000","last_became_peered":"2024-09-21T13:59:32.221854+0000","last_unstale":"2024-09-21T13:59:40.048893+0000","last_undegraded":"2024-09-21T13:59:40.048893+0000","last_fullsized":"2024-09-21T13:59:40.048893+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:17.691193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:17.691193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":411,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-21T13:59:40.041778+0000","last_change":"2024-09-21T13:57:12.703746+0000","last_active":"2024-09-21T13:59:40.041778+0000","last_peered":"2024-09-21T13:59:40.041778+0000","last_clean":"2024-09-21T13:59:40.041778+0000","last_became_active":"2024-09-21T13:57:12.703441+0000","last_became_peered":"2024-09-21T13:57:12.703441+0000","last_unstale":"2024-09-21T13:59:40.041778+0000","last_undegraded":"2024-09-21T13:59:40.041778+0000","last_fullsized":"2024-09-21T13:59:40.041778+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:19.999125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:19.999125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:05.702853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":441,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.191535+0000","last_change":"2024-09-21T13:58:47.334564+0000","last_active":"2024-09-21T13:59:33.191535+0000","last_peered":"2024-09-21T13:59:33.191535+0000","last_clean":"2024-09-21T13:59:33.191535+0000","last_became_active":"2024-09-21T13:50:29.689045+0000","last_became_peered":"2024-09-21T13:50:29.689045+0000","last_unstale":"2024-09-21T13:59:33.191535+0000","last_undegraded":"2024-09-21T13:59:33.191535+0000","last_fullsized":"2024-09-21T13:59:33.191535+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:47.334504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:47.334504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:59:48.666934+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"290'19","reported_seq":555,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-21T13:59:40.040325+0000","last_change":"2024-09-21T13:59:32.277742+0000","last_active":"2024-09-21T13:59:40.040325+0000","last_peered":"2024-09-21T13:59:40.040325+0000","last_clean":"2024-09-21T13:59:40.040325+0000","last_became_active":"2024-09-21T13:59:32.231567+0000","last_became_peered":"2024-09-21T13:59:32.231567+0000","last_unstale":"2024-09-21T13:59:40.040325+0000","last_undegraded":"2024-09-21T13:59:40.040325+0000","last_fullsized":"2024-09-21T13:59:40.040325+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:54:22.978267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:22.978267+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":512,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.192282+0000","last_change":"2024-09-21T13:59:32.206128+0000","last_active":"2024-09-21T13:59:33.192282+0000","last_peered":"2024-09-21T13:59:33.192282+0000","last_clean":"2024-09-21T13:59:33.192282+0000","last_became_active":"2024-09-21T13:59:32.205826+0000","last_became_peered":"2024-09-21T13:59:32.205826+0000","last_unstale":"2024-09-21T13:59:33.192282+0000","last_undegraded":"2024-09-21T13:59:33.192282+0000","last_fullsized":"2024-09-21T13:59:33.192282+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:17.305144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:17.305144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":481,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-21T13:59:40.045112+0000","last_change":"2024-09-21T13:59:32.219508+0000","last_active":"2024-09-21T13:59:40.045112+0000","last_peered":"2024-09-21T13:59:40.045112+0000","last_clean":"2024-09-21T13:59:40.045112+0000","last_became_active":"2024-09-21T13:59:32.214004+0000","last_became_peered":"2024-09-21T13:59:32.214004+0000","last_unstale":"2024-09-21T13:59:40.045112+0000","last_undegraded":"2024-09-21T13:59:40.045112+0000","last_fullsized":"2024-09-21T13:59:40.045112+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:36.525417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:36.525417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":477,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-21T13:59:40.042018+0000","last_change":"2024-09-21T13:59:28.124373+0000","last_active":"2024-09-21T13:59:40.042018+0000","last_peered":"2024-09-21T13:59:40.042018+0000","last_clean":"2024-09-21T13:59:40.042018+0000","last_became_active":"2024-09-21T13:58:17.778115+0000","last_became_peered":"2024-09-21T13:58:17.778115+0000","last_unstale":"2024-09-21T13:59:40.042018+0000","last_undegraded":"2024-09-21T13:59:40.042018+0000","last_fullsized":"2024-09-21T13:59:40.042018+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:28.124296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:28.124296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:40.216842+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"290'8","reported_seq":501,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-21T13:59:40.041463+0000","last_change":"2024-09-21T13:59:26.034457+0000","last_active":"2024-09-21T13:59:40.041463+0000","last_peered":"2024-09-21T13:59:40.041463+0000","last_clean":"2024-09-21T13:59:40.041463+0000","last_became_active":"2024-09-21T13:58:17.775079+0000","last_became_peered":"2024-09-21T13:58:17.775079+0000","last_unstale":"2024-09-21T13:59:40.041463+0000","last_undegraded":"2024-09-21T13:59:40.041463+0000","last_fullsized":"2024-09-21T13:59:40.041463+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'8","last_scrub_stamp":"2024-09-21T13:59:26.034372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:26.034372+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:38.948765+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":460,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.192562+0000","last_change":"2024-09-21T13:58:16.457518+0000","last_active":"2024-09-21T13:59:33.192562+0000","last_peered":"2024-09-21T13:59:33.192562+0000","last_clean":"2024-09-21T13:59:33.192562+0000","last_became_active":"2024-09-21T13:57:12.684372+0000","last_became_peered":"2024-09-21T13:57:12.684372+0000","last_unstale":"2024-09-21T13:59:33.192562+0000","last_undegraded":"2024-09-21T13:59:33.192562+0000","last_fullsized":"2024-09-21T13:59:33.192562+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:16.457445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:16.457445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:59:38.691908+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":83,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.250730+0000","last_change":"2024-09-21T13:59:32.204646+0000","last_active":"2024-09-21T13:59:33.250730+0000","last_peered":"2024-09-21T13:59:33.250730+0000","last_clean":"2024-09-21T13:59:33.250730+0000","last_became_active":"2024-09-21T13:59:32.204380+0000","last_became_peered":"2024-09-21T13:59:32.204380+0000","last_unstale":"2024-09-21T13:59:33.250730+0000","last_undegraded":"2024-09-21T13:59:33.250730+0000","last_fullsized":"2024-09-21T13:59:33.250730+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:51.254456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:53:51.254456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":414,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-21T13:59:40.045979+0000","last_change":"2024-09-21T13:59:32.212615+0000","last_active":"2024-09-21T13:59:40.045979+0000","last_peered":"2024-09-21T13:59:40.045979+0000","last_clean":"2024-09-21T13:59:40.045979+0000","last_became_active":"2024-09-21T13:59:32.210134+0000","last_became_peered":"2024-09-21T13:59:32.210134+0000","last_unstale":"2024-09-21T13:59:40.045979+0000","last_undegraded":"2024-09-21T13:59:40.045979+0000","last_fullsized":"2024-09-21T13:59:40.045979+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:08.852789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:55:08.852789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":515,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-09-21T13:59:39.287821+0000","last_change":"2024-09-21T13:58:17.706373+0000","last_active":"2024-09-21T13:59:39.287821+0000","last_peered":"2024-09-21T13:59:39.287821+0000","last_clean":"2024-09-21T13:59:39.287821+0000","last_became_active":"2024-09-21T13:49:04.075562+0000","last_became_peered":"2024-09-21T13:49:04.075562+0000","last_unstale":"2024-09-21T13:59:39.287821+0000","last_undegraded":"2024-09-21T13:59:39.287821+0000","last_fullsized":"2024-09-21T13:59:39.287821+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:17.706325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:17.706325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":457,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-21T13:59:40.041779+0000","last_change":"2024-09-21T13:58:13.776632+0000","last_active":"2024-09-21T13:59:40.041779+0000","last_peered":"2024-09-21T13:59:40.041779+0000","last_clean":"2024-09-21T13:59:40.041779+0000","last_became_active":"2024-09-21T13:57:12.704042+0000","last_became_peered":"2024-09-21T13:57:12.704042+0000","last_unstale":"2024-09-21T13:59:40.041779+0000","last_undegraded":"2024-09-21T13:59:40.041779+0000","last_fullsized":"2024-09-21T13:59:40.041779+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:13.776572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:13.776572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"290'19","reported_seq":96,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-09-21T13:59:33.248908+0000","last_change":"2024-09-21T13:58:17.807643+0000","last_active":"2024-09-21T13:59:33.248908+0000","last_peered":"2024-09-21T13:59:33.248908+0000","last_clean":"2024-09-21T13:59:33.248908+0000","last_became_active":"2024-09-21T13:58:17.777573+0000","last_became_peered":"2024-09-21T13:58:17.777573+0000","last_unstale":"2024-09-21T13:59:33.248908+0000","last_undegraded":"2024-09-21T13:59:33.248908+0000","last_fullsized":"2024-09-21T13:59:33.248908+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:55:12.198562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:55:12.198562+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:07.882694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":383,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-09-21T13:59:40.045380+0000","last_change":"2024-09-21T13:59:32.203045+0000","last_active":"2024-09-21T13:59:40.045380+0000","last_peered":"2024-09-21T13:59:40.045380+0000","last_clean":"2024-09-21T13:59:40.045380+0000","last_became_active":"2024-09-21T13:59:32.202225+0000","last_became_peered":"2024-09-21T13:59:32.202225+0000","last_unstale":"2024-09-21T13:59:40.045380+0000","last_undegraded":"2024-09-21T13:59:40.045380+0000","last_fullsized":"2024-09-21T13:59:40.045380+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:40.650832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:40.650832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":7736655896,"num_objects":118053,"num_object_clones":0,"num_object_copies":354159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":49438,"num_objects_misplaced":6521,"num_objects_unfound":0,"num_objects_dirty":118053,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":124759,"num_write_kb":7769921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7338,"num_bytes_recovered":480903168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20785078272,"data_stored":20785070128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80782,"ondisk_log_size":80782,"up":48,"acting":40,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1114,"num_read_kb":871,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":176,"num_bytes_recovered":1914,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":687,"ondisk_log_size":687,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":298,"seq":1279900254211,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1459684,"kb_used_data":1390416,"kb_used_omap":4,"kb_used_meta":69243,"kb_avail":92285468,"statfs":{"total":95995035648,"available":94500319232,"internally_reserved":0,"allocated":1423785984,"data_stored":1423239552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":70905512},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463050,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1739968,"kb_used_data":1429744,"kb_used_omap":1,"kb_used_meta":310206,"kb_avail":92005184,"statfs":{"total":95995035648,"available":94213308416,"internally_reserved":0,"allocated":1464057856,"data_stored":1463534762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":317651398},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724689,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4765828,"kb_used_data":4323700,"kb_used_omap":1,"kb_used_meta":442110,"kb_avail":88979324,"statfs":{"total":95995035648,"available":91114827776,"internally_reserved":0,"allocated":4427468800,"data_stored":4426942006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":452721093},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,4,1,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920919,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2224924,"kb_used_data":1882700,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":91520228,"statfs":{"total":95995035648,"available":93716713472,"internally_reserved":0,"allocated":1927884800,"data_stored":1927333108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":350353861},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":240,"seq":1030792151071,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3961600,"kb_used_data":3771048,"kb_used_omap":4,"kb_used_meta":190523,"kb_avail":89783552,"statfs":{"total":95995035648,"available":91938357248,"internally_reserved":0,"allocated":3861553152,"data_stored":3860990308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4443,"internal_metadata":195096229},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":130,"seq":558345748572,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1927868,"kb_used_data":1426412,"kb_used_omap":4,"kb_used_meta":501435,"kb_avail":91817284,"statfs":{"total":95995035648,"available":94020898816,"internally_reserved":0,"allocated":1460645888,"data_stored":1460127064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4441,"internal_metadata":513470119},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575017,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4743424,"kb_used_data":4247152,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":89001728,"statfs":{"total":95995035648,"available":91137769472,"internally_reserved":0,"allocated":4349083648,"data_stored":4348495860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":508164549},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738543,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2389588,"kb_used_data":1934980,"kb_used_omap":1,"kb_used_meta":454590,"kb_avail":91355564,"statfs":{"total":95995035648,"available":93548097536,"internally_reserved":0,"allocated":1981419520,"data_stored":1980876304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":465500613},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1968570368,"data_stored":1968570368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4316336128,"data_stored":4316332056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1458765824,"data_stored":1458765824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3836280832,"data_stored":3836280832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1923022848,"data_stored":1923022848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4406054912,"data_stored":4406050840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1456799744,"data_stored":1456799744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1419247616,"data_stored":1419247616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T13:59:46.701 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph pg force-backfill 9.c 9.e 9.9 9.3 9.5 2024-09-21T13:59:46.702 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:59:46.704 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:46.739 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:46.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:46.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:46.841 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:46.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:46.885 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:46.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:46.986 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:47.037 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:47.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:47.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:47.138 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:47.179 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:47.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:47.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:47.281 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:47.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:47.321 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:47.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:47.375 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:59:47.422 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:59:47.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:47.454 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:47.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:59:47.555 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:47.582 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:47.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:47.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:47.684 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:47.722 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:47.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:47.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:47.818 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-09-21T13:59:47.821 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:47 smithi049 ceph-mon[40489]: pgmap v540: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 139 active+clean; 7.0 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.9 MiB/s wr, 878 op/s; 48160/344742 objects degraded (13.970%); 6521/344742 objects misplaced (1.892%); 8.2 MiB/s, 131 objects/s recovering 2024-09-21T13:59:47.821 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:47 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3277680941' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:59:47.821 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:47 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:59:47.821 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:47 smithi049 ceph-mon[30080]: pgmap v540: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 139 active+clean; 7.0 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.9 MiB/s wr, 878 op/s; 48160/344742 objects degraded (13.970%); 6521/344742 objects misplaced (1.892%); 8.2 MiB/s, 131 objects/s recovering 2024-09-21T13:59:47.821 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:47 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3277680941' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:59:47.822 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:47 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:59:47.824 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:59:47.832 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T13:59:47.846 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:47.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:47.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:59:47.947 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:48.011 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:48.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:48.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:48.113 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:48.130 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:47 smithi187 ceph-mon[33032]: pgmap v540: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 139 active+clean; 7.0 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.9 MiB/s wr, 878 op/s; 48160/344742 objects degraded (13.970%); 6521/344742 objects misplaced (1.892%); 8.2 MiB/s, 131 objects/s recovering 2024-09-21T13:59:48.130 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:47 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3277680941' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:59:48.130 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:47 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T13:59:48.134 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:48.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:48.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:48.236 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:59:48.287 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:48.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:48.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:59:48.318 INFO:teuthology.orchestra.run.smithi049.stderr:instructing pg(s) [9.9] on osd.0 to force-backfill; instructing pg(s) [9.3,9.e] on osd.1 to force-backfill; instructing pg(s) [9.c] on osd.3 to force-backfill; instructing pg(s) [9.5] on osd.5 to force-backfill; 2024-09-21T13:59:48.319 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-09-21T13:59:48.388 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:59:48.427 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:48.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:48.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:59:48.529 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:48.555 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:48.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:48.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:48.657 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:59:48.677 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:48.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:48.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:59:48.779 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:48.808 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:48.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:48.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:48.910 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:48.943 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:48.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:48.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:49.045 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:59:49.074 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:49.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:49.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:59:49.176 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:49.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:48 smithi187 ceph-mon[33032]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:59:49.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:48 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T13:59:49.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:48 smithi187 ceph-mon[33032]: osdmap e303: 8 total, 8 up, 8 in 2024-09-21T13:59:49.206 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:49.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:49.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:49.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:48 smithi049 ceph-mon[40489]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:59:49.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:48 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T13:59:49.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:48 smithi049 ceph-mon[40489]: osdmap e303: 8 total, 8 up, 8 in 2024-09-21T13:59:49.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:48 smithi049 ceph-mon[30080]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:59:49.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:48 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T13:59:49.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:48 smithi049 ceph-mon[30080]: osdmap e303: 8 total, 8 up, 8 in 2024-09-21T13:59:49.308 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:49.332 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:49.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:49.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:49.434 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:49.466 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:49.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:49.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:49.568 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:49.612 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:49.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:49.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:49.644 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:59:49 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:59:49.714 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:49.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:49.740 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:49.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:49.841 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:49.867 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:49.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:49.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:49.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:49 smithi187 ceph-mon[33032]: pgmap v542: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 139 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 691 op/s; 47925/341502 objects degraded (14.034%); 6280/341502 objects misplaced (1.839%); 3.4 MiB/s, 54 objects/s recovering 2024-09-21T13:59:49.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:49 smithi187 ceph-mon[33032]: from='client.25453 -' entity='client.admin' cmd=[{"prefix": "pg force-backfill", "pgid": ["9.c", "9.e", "9.9", "9.3", "9.5"], "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:59:49.969 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:50.001 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:50.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:50.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:50.104 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:59:50.127 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:50.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:50.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:59:50.231 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:50.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:49 smithi049 ceph-mon[40489]: pgmap v542: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 139 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 691 op/s; 47925/341502 objects degraded (14.034%); 6280/341502 objects misplaced (1.839%); 3.4 MiB/s, 54 objects/s recovering 2024-09-21T13:59:50.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:49 smithi049 ceph-mon[40489]: from='client.25453 -' entity='client.admin' cmd=[{"prefix": "pg force-backfill", "pgid": ["9.c", "9.e", "9.9", "9.3", "9.5"], "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:59:50.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:49 smithi049 ceph-mon[30080]: pgmap v542: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 139 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 691 op/s; 47925/341502 objects degraded (14.034%); 6280/341502 objects misplaced (1.839%); 3.4 MiB/s, 54 objects/s recovering 2024-09-21T13:59:50.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:49 smithi049 ceph-mon[30080]: from='client.25453 -' entity='client.admin' cmd=[{"prefix": "pg force-backfill", "pgid": ["9.c", "9.e", "9.9", "9.3", "9.5"], "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T13:59:50.258 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:50.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:50.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:50.360 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:50.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:50.409 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:50.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:50.510 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:59:50.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:50.551 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:50.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:59:50.562 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T13:59:50.562 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T13:59:50.653 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:50.692 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:50.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:50.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:50.794 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:50.829 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:50.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:50.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:50.931 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:59:50.963 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:50.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:50.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:59:51.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:50 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:59:51.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:50 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 47925/341502 objects degraded (14.034%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T13:59:51.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:50 smithi049 ceph-mon[40489]: osdmap e304: 8 total, 8 up, 8 in 2024-09-21T13:59:51.005 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 13:59:50 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:13:59:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T13:59:51.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:50 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:59:51.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:50 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 47925/341502 objects degraded (14.034%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T13:59:51.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:50 smithi049 ceph-mon[30080]: osdmap e304: 8 total, 8 up, 8 in 2024-09-21T13:59:51.065 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:59:51.107 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:51.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:51.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:59:51.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:50 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T13:59:51.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:50 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 47925/341502 objects degraded (14.034%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T13:59:51.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:50 smithi187 ceph-mon[33032]: osdmap e304: 8 total, 8 up, 8 in 2024-09-21T13:59:51.209 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:51.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:51.246 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:51.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:51.348 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:51.386 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:51.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:51.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:51.488 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:59:51.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:51.532 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:51.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:59:51.633 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:59:51.672 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:51.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:51.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:59:51.774 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:51.805 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:51.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:51.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:51.863 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:59:51.907 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:51.940 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:51.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:51.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:52.042 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:52.075 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:52.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:52.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:52.176 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:51 smithi049 ceph-mon[40489]: pgmap v543: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped+forced_backfill, 2 active+undersized+degraded+remapped+forced_backfill+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 139 active+clean; 6.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 876 op/s; 46790/332097 objects degraded (14.089%); 6280/332097 objects misplaced (1.891%); 5.3 MiB/s, 84 objects/s recovering 2024-09-21T13:59:52.177 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:51 smithi049 ceph-mon[30080]: pgmap v543: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped+forced_backfill, 2 active+undersized+degraded+remapped+forced_backfill+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 139 active+clean; 6.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 876 op/s; 46790/332097 objects degraded (14.089%); 6280/332097 objects misplaced (1.891%); 5.3 MiB/s, 84 objects/s recovering 2024-09-21T13:59:52.177 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:52.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:51 smithi187 ceph-mon[33032]: pgmap v543: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped+forced_backfill, 2 active+undersized+degraded+remapped+forced_backfill+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 139 active+clean; 6.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 876 op/s; 46790/332097 objects degraded (14.089%); 6280/332097 objects misplaced (1.891%); 5.3 MiB/s, 84 objects/s recovering 2024-09-21T13:59:52.219 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:52.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:52.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:52.321 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:52.344 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:52.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:52.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:52.446 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:52.459 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:59:52.459 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:59:52.468 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:52.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:52.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:52.477 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:59:52.478 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:59:52.500 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:59:52.500 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:59:52.530 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:59:52.530 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:59:52.557 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:59:52.558 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:59:52.570 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:52.599 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:59:52.599 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:59:52.604 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:52.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:52.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:52.634 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:59:52.634 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:59:52.665 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:59:52.666 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:59:52.705 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:52.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:52.746 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:52.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:52.847 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:59:52.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:52.881 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:52.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:59:52.981 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:59:53.025 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:53.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:53.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:59:53.127 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:59:53.149 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T13:59:53.171 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:53.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:53.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:59:53.183 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:52 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:59:53.183 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:52 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:59:53.183 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:52 smithi049 ceph-mon[30080]: osdmap e305: 8 total, 8 up, 8 in 2024-09-21T13:59:53.183 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:52 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3422388997' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:59:53.183 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:52 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:59:53.184 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:52 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:59:53.184 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:52 smithi049 ceph-mon[40489]: osdmap e305: 8 total, 8 up, 8 in 2024-09-21T13:59:53.184 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:52 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3422388997' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:59:53.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:52 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:59:53.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:52 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:59:53.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:52 smithi187 ceph-mon[33032]: osdmap e305: 8 total, 8 up, 8 in 2024-09-21T13:59:53.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:52 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3422388997' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T13:59:53.273 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:59:53.312 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:53.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:53.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:59:53.414 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:59:53.438 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T13:59:53.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:53.463 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:53.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:59:53.564 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:59:53.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:53.611 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:53.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:59:53.712 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:59:53.757 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:53.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:53.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:59:53.859 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:53.895 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:53.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:53.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:53.967 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:59:53.969 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:59:53.997 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:59:54.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:54.041 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:54.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:59:54.101 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:53 smithi049 ceph-mon[30080]: pgmap v546: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait+forced_backfill, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped+forced_backfill, 2 active+undersized+degraded+remapped+forced_backfill+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 139 active+clean; 6.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 862 op/s; 46556/329220 objects degraded (14.141%); 6063/329220 objects misplaced (1.842%); 4.1 MiB/s, 65 objects/s recovering 2024-09-21T13:59:54.101 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:53 smithi049 ceph-mon[40489]: pgmap v546: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait+forced_backfill, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped+forced_backfill, 2 active+undersized+degraded+remapped+forced_backfill+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 139 active+clean; 6.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 862 op/s; 46556/329220 objects degraded (14.141%); 6063/329220 objects misplaced (1.842%); 4.1 MiB/s, 65 objects/s recovering 2024-09-21T13:59:54.106 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:53 smithi187 ceph-mon[33032]: pgmap v546: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait+forced_backfill, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped+forced_backfill, 2 active+undersized+degraded+remapped+forced_backfill+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 139 active+clean; 6.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 862 op/s; 46556/329220 objects degraded (14.141%); 6063/329220 objects misplaced (1.842%); 4.1 MiB/s, 65 objects/s recovering 2024-09-21T13:59:54.143 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:54.174 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:54.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:54.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:54.275 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:54.319 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:54.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:54.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:54.421 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:54.459 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:54.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:54.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:54.561 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:54.592 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:54.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:54.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:54.638 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T13:59:54.638 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-21T13:59:54.638 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-21T13:59:54.638 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-21T13:59:54.638 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T13:59:54.694 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:59:54.733 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:54.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:54.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:59:54.834 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:54.868 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:54.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:54.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:54.970 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:59:55.006 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:55.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:55.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:59:55.046 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-09-21T13:59:55.107 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:55.144 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:55.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:55.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:55.246 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:55.271 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:55.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:55.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:55.373 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:55.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:55.406 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:55.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:55.507 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:59:55.538 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:55.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:55.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:59:55.600 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T13:59:55.640 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:59:55.672 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:55.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:55.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:59:55.774 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:55.805 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:55.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:55.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:55.866 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T13:59:55.907 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:59:55.942 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:55.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:55.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:59:56.006 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:59:56.044 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:59:56.092 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:56.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:56.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:59:56.138 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:55 smithi187 ceph-mon[33032]: pgmap v547: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait+forced_backfill, 2 active+recovering+undersized+degraded+remapped+forced_backfill, 2 active+undersized+degraded+remapped+forced_backfill+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfill_toofull, 139 active+clean; 6.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 836 B/s rd, 0 B/s wr, 918 op/s; 46322/324660 objects degraded (14.268%); 6063/324660 objects misplaced (1.867%); 6.4 MiB/s, 102 objects/s recovering 2024-09-21T13:59:56.138 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:55 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1163139893' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:59:56.138 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:55 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:59:56.138 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:55 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 46322/324660 objects degraded (14.268%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T13:59:56.138 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:55 smithi187 ceph-mon[33032]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:59:56.138 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:55 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T13:59:56.138 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:55 smithi187 ceph-mon[33032]: osdmap e306: 8 total, 8 up, 8 in 2024-09-21T13:59:56.194 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:56.232 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:56.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:56.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:56.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:55 smithi049 ceph-mon[40489]: pgmap v547: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait+forced_backfill, 2 active+recovering+undersized+degraded+remapped+forced_backfill, 2 active+undersized+degraded+remapped+forced_backfill+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfill_toofull, 139 active+clean; 6.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 836 B/s rd, 0 B/s wr, 918 op/s; 46322/324660 objects degraded (14.268%); 6063/324660 objects misplaced (1.867%); 6.4 MiB/s, 102 objects/s recovering 2024-09-21T13:59:56.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:55 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1163139893' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:59:56.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:55 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:59:56.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:55 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 46322/324660 objects degraded (14.268%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T13:59:56.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:55 smithi049 ceph-mon[40489]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:59:56.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:55 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T13:59:56.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:55 smithi049 ceph-mon[40489]: osdmap e306: 8 total, 8 up, 8 in 2024-09-21T13:59:56.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:55 smithi049 ceph-mon[30080]: pgmap v547: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait+forced_backfill, 2 active+recovering+undersized+degraded+remapped+forced_backfill, 2 active+undersized+degraded+remapped+forced_backfill+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfill_toofull, 139 active+clean; 6.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 836 B/s rd, 0 B/s wr, 918 op/s; 46322/324660 objects degraded (14.268%); 6063/324660 objects misplaced (1.867%); 6.4 MiB/s, 102 objects/s recovering 2024-09-21T13:59:56.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:55 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1163139893' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:59:56.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:55 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T13:59:56.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:55 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 46322/324660 objects degraded (14.268%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T13:59:56.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:55 smithi049 ceph-mon[30080]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T13:59:56.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:55 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T13:59:56.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:55 smithi049 ceph-mon[30080]: osdmap e306: 8 total, 8 up, 8 in 2024-09-21T13:59:56.334 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:56.359 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:56.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:56.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:56.461 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:59:56.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:56.513 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:56.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:59:56.614 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:59:56.649 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:56.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:56.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:59:56.751 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:59:56.798 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:56.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:56.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:59:56.810 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T13:59:56.810 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-21T13:59:56.810 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-21T13:59:56.810 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-21T13:59:56.810 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.810 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:35241 7.4 7:2f984d9b:::benchmark_data_smithi049_76798_object35240:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T13:59:56.810 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:28.687118+0000", 2024-09-21T13:59:56.811 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 568.120693437, 2024-09-21T13:59:56.811 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.011460528, 2024-09-21T13:59:56.811 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:59:56.811 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:59:56.811 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:59:56.811 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:59:56.811 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:59:56.811 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:59:56.811 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 35241 2024-09-21T13:59:56.811 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.811 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:59:56.811 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.811 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:59:56.811 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.687118+0000", 2024-09-21T13:59:56.812 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:56.812 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.812 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.812 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:59:56.812 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.687118+0000", 2024-09-21T13:59:56.812 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:56.812 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.812 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.812 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:59:56.812 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.687120+0000", 2024-09-21T13:59:56.812 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.573e-06 2024-09-21T13:59:56.812 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.812 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.813 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:59:56.813 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.687176+0000", 2024-09-21T13:59:56.813 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6502000000000002e-05 2024-09-21T13:59:56.813 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.813 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.813 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:59:56.813 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.687189+0000", 2024-09-21T13:59:56.813 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2747e-05 2024-09-21T13:59:56.813 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.813 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.813 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:59:56.813 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.687196+0000", 2024-09-21T13:59:56.813 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6050000000000003e-06 2024-09-21T13:59:56.813 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.814 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.814 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:59:56.814 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.687248+0000", 2024-09-21T13:59:56.814 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2234000000000002e-05 2024-09-21T13:59:56.814 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.814 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.814 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:59:56.814 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.687256+0000", 2024-09-21T13:59:56.814 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3299999999999999e-06 2024-09-21T13:59:56.814 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.814 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.814 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:59:56.814 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.691286+0000", 2024-09-21T13:59:56.815 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.004029828 2024-09-21T13:59:56.815 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.815 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.815 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:59:56.815 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.692244+0000", 2024-09-21T13:59:56.815 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00095738099999999999 2024-09-21T13:59:56.815 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.815 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.815 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-21T13:59:56.815 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.692408+0000", 2024-09-21T13:59:56.815 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000164041 2024-09-21T13:59:56.815 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.815 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.816 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:56.816 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.698261+0000", 2024-09-21T13:59:56.816 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.005852983 2024-09-21T13:59:56.816 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.816 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.816 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:59:56.816 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.698274+0000", 2024-09-21T13:59:56.816 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3485e-05 2024-09-21T13:59:56.816 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.816 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.816 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:56.816 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.698481+0000", 2024-09-21T13:59:56.816 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00020698099999999999 2024-09-21T13:59:56.817 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.817 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.817 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:59:56.817 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.698537+0000", 2024-09-21T13:59:56.817 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6082999999999998e-05 2024-09-21T13:59:56.817 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.817 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.817 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:59:56.817 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.698579+0000", 2024-09-21T13:59:56.817 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1755000000000003e-05 2024-09-21T13:59:56.817 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:56.817 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:59:56.817 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:56.817 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.818 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.818 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:35262 7.4 7:2f95173f:::benchmark_data_smithi049_76798_object35261:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T13:59:56.818 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:28.705575+0000", 2024-09-21T13:59:56.818 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 568.10223675500004, 2024-09-21T13:59:56.818 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.99486020900000005, 2024-09-21T13:59:56.818 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:59:56.818 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:59:56.818 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:59:56.818 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:59:56.818 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:59:56.819 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:59:56.819 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 35262 2024-09-21T13:59:56.819 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.819 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:59:56.819 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.819 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:59:56.819 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.705575+0000", 2024-09-21T13:59:56.819 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:56.819 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.819 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.819 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:59:56.819 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.705575+0000", 2024-09-21T13:59:56.819 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:56.819 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.820 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.820 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:59:56.820 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.705577+0000", 2024-09-21T13:59:56.820 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.485e-06 2024-09-21T13:59:56.820 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.820 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.820 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:59:56.820 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.705634+0000", 2024-09-21T13:59:56.820 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7954000000000001e-05 2024-09-21T13:59:56.820 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.820 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.820 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:59:56.820 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.705649+0000", 2024-09-21T13:59:56.821 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4209999999999999e-05 2024-09-21T13:59:56.821 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.821 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.821 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:59:56.821 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.705657+0000", 2024-09-21T13:59:56.821 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7910000000000005e-06 2024-09-21T13:59:56.821 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.821 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.821 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:59:56.821 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.705703+0000", 2024-09-21T13:59:56.821 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6712999999999998e-05 2024-09-21T13:59:56.821 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.821 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.821 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:59:56.822 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.705711+0000", 2024-09-21T13:59:56.822 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2050000000000002e-06 2024-09-21T13:59:56.822 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.822 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.822 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:59:56.822 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.694482+0000", 2024-09-21T13:59:56.822 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.98877102800000005 2024-09-21T13:59:56.822 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.822 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.822 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:59:56.822 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.694624+0000", 2024-09-21T13:59:56.822 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014155400000000001 2024-09-21T13:59:56.822 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.823 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.823 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-21T13:59:56.823 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.694770+0000", 2024-09-21T13:59:56.823 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014635399999999999 2024-09-21T13:59:56.823 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.823 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.823 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:59:56.823 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.698351+0000", 2024-09-21T13:59:56.823 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0035804869999999998 2024-09-21T13:59:56.823 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.823 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.823 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:56.823 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.698411+0000", 2024-09-21T13:59:56.823 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0132e-05 2024-09-21T13:59:56.824 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.824 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.824 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:56.824 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.700312+0000", 2024-09-21T13:59:56.824 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001901109 2024-09-21T13:59:56.824 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.824 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.824 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:59:56.824 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.700374+0000", 2024-09-21T13:59:56.825 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2137999999999994e-05 2024-09-21T13:59:56.825 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.825 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.825 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:59:56.825 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.700435+0000", 2024-09-21T13:59:56.825 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1049000000000005e-05 2024-09-21T13:59:56.825 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:56.825 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:59:56.825 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:56.825 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.826 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.826 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:35282 7.4 7:26bb5871:::benchmark_data_smithi049_76798_object35281:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T13:59:56.826 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:28.721297+0000", 2024-09-21T13:59:56.826 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 568.08651510000004, 2024-09-21T13:59:56.826 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.98226591299999999, 2024-09-21T13:59:56.826 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:59:56.826 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:59:56.826 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:59:56.826 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:59:56.826 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:59:56.826 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:59:56.826 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 35282 2024-09-21T13:59:56.827 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.827 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:59:56.827 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.827 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:59:56.827 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.721297+0000", 2024-09-21T13:59:56.827 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:56.827 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.827 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.827 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:59:56.827 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.721297+0000", 2024-09-21T13:59:56.827 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:56.827 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.827 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.828 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:59:56.828 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.721298+0000", 2024-09-21T13:59:56.828 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0920000000000001e-06 2024-09-21T13:59:56.828 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.828 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.828 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:59:56.828 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.721380+0000", 2024-09-21T13:59:56.828 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2422000000000005e-05 2024-09-21T13:59:56.828 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.828 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.828 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:59:56.828 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.721389+0000", 2024-09-21T13:59:56.828 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1789999999999997e-06 2024-09-21T13:59:56.828 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.829 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.829 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:59:56.829 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.721394+0000", 2024-09-21T13:59:56.829 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9080000000000003e-06 2024-09-21T13:59:56.829 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.829 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.829 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:59:56.829 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.721436+0000", 2024-09-21T13:59:56.829 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1887000000000001e-05 2024-09-21T13:59:56.829 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.829 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.829 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T13:59:56.829 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:28.721443+0000", 2024-09-21T13:59:56.829 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.725e-06 2024-09-21T13:59:56.830 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.830 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.830 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:59:56.830 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.696771+0000", 2024-09-21T13:59:56.830 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.97532851799999998 2024-09-21T13:59:56.830 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.830 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.830 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:59:56.830 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.696923+0000", 2024-09-21T13:59:56.830 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015162800000000001 2024-09-21T13:59:56.830 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.830 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.830 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-21T13:59:56.830 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.697094+0000", 2024-09-21T13:59:56.831 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000170884 2024-09-21T13:59:56.831 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.831 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.831 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:56.831 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.700495+0000", 2024-09-21T13:59:56.831 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0034005519999999998 2024-09-21T13:59:56.831 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.831 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.831 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:59:56.831 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.701924+0000", 2024-09-21T13:59:56.831 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001429303 2024-09-21T13:59:56.831 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.831 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.831 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:56.832 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.703461+0000", 2024-09-21T13:59:56.832 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001537395 2024-09-21T13:59:56.832 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.832 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.832 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:59:56.832 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.703519+0000", 2024-09-21T13:59:56.832 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7435999999999997e-05 2024-09-21T13:59:56.832 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.832 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.832 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:59:56.832 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:29.703563+0000", 2024-09-21T13:59:56.832 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3983999999999998e-05 2024-09-21T13:59:56.832 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:56.832 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:59:56.833 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:56.833 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.833 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.833 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:36866 7.1 7:81a03870:::benchmark_data_smithi049_76798_object36865:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-21T13:59:56.833 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:30.635343+0000", 2024-09-21T13:59:56.833 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 566.17246910100005, 2024-09-21T13:59:56.833 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13232190499999999, 2024-09-21T13:59:56.833 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:59:56.833 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:59:56.833 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:59:56.833 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:59:56.833 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:59:56.833 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:59:56.833 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 36866 2024-09-21T13:59:56.834 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.834 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:59:56.834 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.834 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:59:56.834 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.635343+0000", 2024-09-21T13:59:56.834 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:56.834 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.834 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.834 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:59:56.834 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.635343+0000", 2024-09-21T13:59:56.834 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:56.834 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.834 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.835 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:59:56.835 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.635344+0000", 2024-09-21T13:59:56.835 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6920000000000001e-06 2024-09-21T13:59:56.835 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.835 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.835 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:59:56.835 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.635406+0000", 2024-09-21T13:59:56.835 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1353999999999996e-05 2024-09-21T13:59:56.835 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.835 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.835 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:59:56.835 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.635418+0000", 2024-09-21T13:59:56.835 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2082000000000001e-05 2024-09-21T13:59:56.835 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.836 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.836 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:59:56.836 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.635425+0000", 2024-09-21T13:59:56.836 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1509999999999998e-06 2024-09-21T13:59:56.836 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.836 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.836 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:59:56.836 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.635486+0000", 2024-09-21T13:59:56.836 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0523999999999999e-05 2024-09-21T13:59:56.836 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.836 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.836 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:59:56.836 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.635599+0000", 2024-09-21T13:59:56.836 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000113731 2024-09-21T13:59:56.836 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.837 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.837 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,6", 2024-09-21T13:59:56.837 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.635724+0000", 2024-09-21T13:59:56.837 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000124292 2024-09-21T13:59:56.837 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.837 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.837 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:59:56.837 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.636808+0000", 2024-09-21T13:59:56.837 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001084631 2024-09-21T13:59:56.837 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.837 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.837 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:56.837 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.638114+0000", 2024-09-21T13:59:56.837 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0013061679999999999 2024-09-21T13:59:56.837 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.837 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.838 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:56.838 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.767545+0000", 2024-09-21T13:59:56.838 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.12943043400000001 2024-09-21T13:59:56.838 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.838 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.838 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:59:56.838 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.767607+0000", 2024-09-21T13:59:56.838 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2065999999999998e-05 2024-09-21T13:59:56.838 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.838 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.838 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:59:56.839 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.767665+0000", 2024-09-21T13:59:56.839 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7779999999999999e-05 2024-09-21T13:59:56.839 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:56.839 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:59:56.839 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:56.839 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.839 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.839 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:36886 7.1 7:8f0780fe:::benchmark_data_smithi049_76798_object36885:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-21T13:59:56.839 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:30.650584+0000", 2024-09-21T13:59:56.839 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 566.15722752199997, 2024-09-21T13:59:56.839 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.11732397999999999, 2024-09-21T13:59:56.839 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:59:56.839 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:59:56.839 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:59:56.840 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:59:56.840 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:59:56.840 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:59:56.840 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 36886 2024-09-21T13:59:56.840 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.840 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:59:56.840 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.840 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:59:56.840 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.650584+0000", 2024-09-21T13:59:56.840 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:56.840 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.840 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.840 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:59:56.841 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.650584+0000", 2024-09-21T13:59:56.841 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:56.841 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.841 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.841 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:59:56.841 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.650587+0000", 2024-09-21T13:59:56.841 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5600000000000001e-06 2024-09-21T13:59:56.841 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.841 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.841 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:59:56.841 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.650631+0000", 2024-09-21T13:59:56.841 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4499999999999997e-05 2024-09-21T13:59:56.841 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.842 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.842 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:59:56.842 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.650643+0000", 2024-09-21T13:59:56.842 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1404e-05 2024-09-21T13:59:56.842 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.842 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.842 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:59:56.842 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.650648+0000", 2024-09-21T13:59:56.842 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1800000000000004e-06 2024-09-21T13:59:56.842 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.842 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.842 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:59:56.842 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.650689+0000", 2024-09-21T13:59:56.842 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0784000000000001e-05 2024-09-21T13:59:56.843 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.843 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.843 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:59:56.843 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.650777+0000", 2024-09-21T13:59:56.843 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8154999999999999e-05 2024-09-21T13:59:56.843 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.843 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.843 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,6", 2024-09-21T13:59:56.843 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.650871+0000", 2024-09-21T13:59:56.843 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3738000000000003e-05 2024-09-21T13:59:56.843 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.843 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.844 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:59:56.844 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.651931+0000", 2024-09-21T13:59:56.844 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0010604270000000001 2024-09-21T13:59:56.844 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.844 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.844 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:56.844 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.652228+0000", 2024-09-21T13:59:56.844 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000296507 2024-09-21T13:59:56.844 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.844 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.844 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:56.844 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.767813+0000", 2024-09-21T13:59:56.845 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.115584956 2024-09-21T13:59:56.845 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.845 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.845 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:59:56.845 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.767869+0000", 2024-09-21T13:59:56.845 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6851999999999997e-05 2024-09-21T13:59:56.845 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.845 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.845 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:59:56.845 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.767908+0000", 2024-09-21T13:59:56.845 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8917e-05 2024-09-21T13:59:56.845 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:56.845 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:59:56.846 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:56.846 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.846 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.846 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:202117 7.1 7:8d183654:::benchmark_data_smithi049_76798_object65697:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-21T13:59:56.846 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:44.231638+0000", 2024-09-21T13:59:56.846 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 432.57617403900002, 2024-09-21T13:59:56.846 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.111972288, 2024-09-21T13:59:56.846 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:59:56.846 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:59:56.846 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:59:56.846 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:59:56.846 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:59:56.846 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:59:56.847 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 202117 2024-09-21T13:59:56.847 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.847 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:59:56.847 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.847 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:59:56.847 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.231638+0000", 2024-09-21T13:59:56.847 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:56.847 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.847 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.847 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:59:56.847 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.231638+0000", 2024-09-21T13:59:56.847 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:56.847 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.848 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.848 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:59:56.848 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.231640+0000", 2024-09-21T13:59:56.848 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3329999999999999e-06 2024-09-21T13:59:56.848 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.848 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.848 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:59:56.848 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.231649+0000", 2024-09-21T13:59:56.848 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0399999999999998e-06 2024-09-21T13:59:56.848 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.848 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.848 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:59:56.848 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.231674+0000", 2024-09-21T13:59:56.848 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.461e-05 2024-09-21T13:59:56.848 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.849 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.849 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:59:56.849 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.231688+0000", 2024-09-21T13:59:56.849 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4063000000000001e-05 2024-09-21T13:59:56.849 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.849 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.849 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:59:56.849 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.231803+0000", 2024-09-21T13:59:56.849 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000115295 2024-09-21T13:59:56.849 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.849 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.849 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:59:56.849 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.231950+0000", 2024-09-21T13:59:56.849 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014642 2024-09-21T13:59:56.849 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.850 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.850 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,6", 2024-09-21T13:59:56.850 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.232017+0000", 2024-09-21T13:59:56.850 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7754000000000002e-05 2024-09-21T13:59:56.850 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.850 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.850 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:59:56.850 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.233984+0000", 2024-09-21T13:59:56.850 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0019670320000000001 2024-09-21T13:59:56.850 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.850 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.850 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:56.850 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.236712+0000", 2024-09-21T13:59:56.850 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0027279040000000002 2024-09-21T13:59:56.851 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.851 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.851 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:56.851 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.343530+0000", 2024-09-21T13:59:56.851 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.106818107 2024-09-21T13:59:56.851 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.851 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.851 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:59:56.851 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.343568+0000", 2024-09-21T13:59:56.851 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7317000000000002e-05 2024-09-21T13:59:56.851 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.851 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.851 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:59:56.852 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.343610+0000", 2024-09-21T13:59:56.852 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2413000000000002e-05 2024-09-21T13:59:56.852 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:56.852 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:59:56.852 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:56.852 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.852 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.852 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:249864 7.1 7:8a44b6f3:::benchmark_data_smithi049_76798_object113444:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e168)", 2024-09-21T13:59:56.852 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:53:36.300462+0000", 2024-09-21T13:59:56.852 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 380.507350137, 2024-09-21T13:59:56.852 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.111981706, 2024-09-21T13:59:56.852 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:59:56.852 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:59:56.852 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:59:56.853 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:59:56.853 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T13:59:56.853 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T13:59:56.853 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 249864 2024-09-21T13:59:56.853 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.853 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:59:56.853 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.853 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:59:56.853 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:53:36.300462+0000", 2024-09-21T13:59:56.853 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:56.853 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.853 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.853 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:59:56.854 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:53:36.300462+0000", 2024-09-21T13:59:56.854 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:56.854 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.854 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.854 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:59:56.854 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:53:36.300463+0000", 2024-09-21T13:59:56.854 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7100000000000011e-07 2024-09-21T13:59:56.854 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.854 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.854 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:59:56.854 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:53:36.300467+0000", 2024-09-21T13:59:56.854 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3830000000000002e-06 2024-09-21T13:59:56.854 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.855 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.855 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:59:56.855 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:53:36.300478+0000", 2024-09-21T13:59:56.855 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.078e-05 2024-09-21T13:59:56.855 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.855 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.855 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:59:56.855 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:53:36.300486+0000", 2024-09-21T13:59:56.855 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6960000000000005e-06 2024-09-21T13:59:56.855 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.855 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.855 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:59:56.855 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:53:36.300533+0000", 2024-09-21T13:59:56.855 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7756999999999997e-05 2024-09-21T13:59:56.856 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.856 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.856 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:59:56.856 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:53:36.300635+0000", 2024-09-21T13:59:56.856 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000101618 2024-09-21T13:59:56.856 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.856 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.856 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,6", 2024-09-21T13:59:56.856 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:53:36.300736+0000", 2024-09-21T13:59:56.856 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000101516 2024-09-21T13:59:56.856 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.856 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.856 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:59:56.856 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:53:36.303427+0000", 2024-09-21T13:59:56.856 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.002690909 2024-09-21T13:59:56.857 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.857 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.857 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:56.857 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:53:36.303790+0000", 2024-09-21T13:59:56.857 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00036281099999999999 2024-09-21T13:59:56.857 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.857 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.857 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:56.857 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:53:36.412339+0000", 2024-09-21T13:59:56.857 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.108549004 2024-09-21T13:59:56.857 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.857 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.857 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:59:56.857 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:53:36.412400+0000", 2024-09-21T13:59:56.858 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0581000000000002e-05 2024-09-21T13:59:56.858 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.858 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.858 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:59:56.858 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:53:36.412443+0000", 2024-09-21T13:59:56.858 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3680000000000002e-05 2024-09-21T13:59:56.858 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:56.858 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:59:56.858 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:56.858 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.858 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.858 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.15207.0:14214 8.2 8:46d3a502:::benchmark_data_smithi049_109311_object14213:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-09-21T13:59:56.858 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:54:23.117083+0000", 2024-09-21T13:59:56.858 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 333.69072863700001, 2024-09-21T13:59:56.859 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.12033872, 2024-09-21T13:59:56.859 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:59:56.859 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:59:56.859 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:59:56.859 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:59:56.859 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.15207", 2024-09-21T13:59:56.859 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:59:56.859 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 14214 2024-09-21T13:59:56.859 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.859 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:59:56.859 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.859 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:59:56.859 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.117083+0000", 2024-09-21T13:59:56.859 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:56.860 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.860 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.860 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:59:56.860 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.117083+0000", 2024-09-21T13:59:56.860 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:56.860 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.860 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.860 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:59:56.860 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.117094+0000", 2024-09-21T13:59:56.860 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0915e-05 2024-09-21T13:59:56.860 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.860 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.860 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:59:56.860 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.117127+0000", 2024-09-21T13:59:56.861 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3222999999999997e-05 2024-09-21T13:59:56.861 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.861 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.861 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:59:56.861 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.117141+0000", 2024-09-21T13:59:56.861 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3604999999999999e-05 2024-09-21T13:59:56.861 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.861 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.861 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:59:56.861 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.117148+0000", 2024-09-21T13:59:56.861 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5739999999999998e-06 2024-09-21T13:59:56.861 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.861 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.861 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:59:56.862 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.117215+0000", 2024-09-21T13:59:56.862 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6672000000000001e-05 2024-09-21T13:59:56.862 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.862 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.862 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:59:56.862 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.117316+0000", 2024-09-21T13:59:56.862 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000100831 2024-09-21T13:59:56.862 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.862 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.862 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,5", 2024-09-21T13:59:56.862 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.117464+0000", 2024-09-21T13:59:56.862 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000148035 2024-09-21T13:59:56.862 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.862 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.863 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:59:56.863 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.118494+0000", 2024-09-21T13:59:56.863 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001029587 2024-09-21T13:59:56.863 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.863 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.863 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:56.863 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.118598+0000", 2024-09-21T13:59:56.863 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000104079 2024-09-21T13:59:56.863 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.863 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.863 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:56.863 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.237344+0000", 2024-09-21T13:59:56.863 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.118746166 2024-09-21T13:59:56.863 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.864 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.864 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:59:56.864 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.237385+0000", 2024-09-21T13:59:56.864 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1495000000000002e-05 2024-09-21T13:59:56.864 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.864 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.864 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:59:56.864 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.237422+0000", 2024-09-21T13:59:56.864 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6538000000000002e-05 2024-09-21T13:59:56.864 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:56.864 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:59:56.864 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:56.864 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.864 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.865 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.15207.0:14217 8.2 8:42c0ac0e:::benchmark_data_smithi049_109311_object14216:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-09-21T13:59:56.865 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:54:23.118628+0000", 2024-09-21T13:59:56.865 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 333.68918425099997, 2024-09-21T13:59:56.865 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.120069462, 2024-09-21T13:59:56.865 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:59:56.865 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:59:56.865 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:59:56.865 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:59:56.865 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.15207", 2024-09-21T13:59:56.865 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:59:56.865 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 14217 2024-09-21T13:59:56.865 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.865 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:59:56.866 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.866 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:59:56.866 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.118628+0000", 2024-09-21T13:59:56.866 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:56.866 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.866 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.866 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:59:56.866 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.118628+0000", 2024-09-21T13:59:56.866 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:56.866 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.866 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.866 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:59:56.866 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.118629+0000", 2024-09-21T13:59:56.867 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0499999999999999e-06 2024-09-21T13:59:56.867 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.867 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.868 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:59:56.868 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.118673+0000", 2024-09-21T13:59:56.868 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4274000000000003e-05 2024-09-21T13:59:56.868 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.868 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.868 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:59:56.868 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.118686+0000", 2024-09-21T13:59:56.868 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3189e-05 2024-09-21T13:59:56.868 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.868 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.868 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:59:56.868 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.118695+0000", 2024-09-21T13:59:56.869 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0690000000000008e-06 2024-09-21T13:59:56.869 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.869 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.869 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:59:56.869 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.118761+0000", 2024-09-21T13:59:56.869 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6203000000000003e-05 2024-09-21T13:59:56.869 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.869 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.869 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:59:56.869 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.118897+0000", 2024-09-21T13:59:56.869 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013574400000000001 2024-09-21T13:59:56.869 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.869 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.869 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,5", 2024-09-21T13:59:56.870 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.119033+0000", 2024-09-21T13:59:56.870 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013564499999999999 2024-09-21T13:59:56.870 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.870 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.870 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:59:56.870 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.119556+0000", 2024-09-21T13:59:56.870 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00052297800000000001 2024-09-21T13:59:56.870 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.870 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.870 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:56.870 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.120042+0000", 2024-09-21T13:59:56.870 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00048602399999999999 2024-09-21T13:59:56.870 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.871 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.871 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:56.871 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.238612+0000", 2024-09-21T13:59:56.871 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.118570721 2024-09-21T13:59:56.871 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.871 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.871 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:59:56.871 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.238657+0000", 2024-09-21T13:59:56.871 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4935000000000003e-05 2024-09-21T13:59:56.871 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.871 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.871 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:59:56.871 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.238697+0000", 2024-09-21T13:59:56.872 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9629999999999998e-05 2024-09-21T13:59:56.872 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:56.872 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:59:56.872 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:56.872 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.872 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.872 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.15207.0:211127 8.b 8:dd21c456:::benchmark_data_smithi049_109311_object84608:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-09-21T13:59:56.872 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:56:54.629514+0000", 2024-09-21T13:59:56.872 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 182.17829819400001, 2024-09-21T13:59:56.872 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13217907500000001, 2024-09-21T13:59:56.872 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:59:56.872 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:59:56.872 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:59:56.872 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:59:56.873 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.15207", 2024-09-21T13:59:56.873 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:59:56.873 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 211127 2024-09-21T13:59:56.873 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.873 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:59:56.873 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.873 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:59:56.873 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.629514+0000", 2024-09-21T13:59:56.873 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:56.873 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.873 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.873 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:59:56.873 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.629514+0000", 2024-09-21T13:59:56.873 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:56.874 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.874 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.874 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:59:56.874 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.629514+0000", 2024-09-21T13:59:56.874 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4999999999999998e-07 2024-09-21T13:59:56.874 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.874 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.874 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:59:56.874 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.629516+0000", 2024-09-21T13:59:56.874 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7010000000000001e-06 2024-09-21T13:59:56.874 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.874 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.874 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:59:56.874 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.629519+0000", 2024-09-21T13:59:56.875 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.766e-06 2024-09-21T13:59:56.875 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.875 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.875 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:59:56.875 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.629522+0000", 2024-09-21T13:59:56.875 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9519999999999999e-06 2024-09-21T13:59:56.875 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.875 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.875 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:59:56.875 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.707551+0000", 2024-09-21T13:59:56.875 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.078028231000000003 2024-09-21T13:59:56.875 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.875 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.875 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:59:56.875 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.707650+0000", 2024-09-21T13:59:56.876 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.9407999999999994e-05 2024-09-21T13:59:56.876 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.876 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.876 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1", 2024-09-21T13:59:56.876 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.707755+0000", 2024-09-21T13:59:56.876 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000105178 2024-09-21T13:59:56.876 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.876 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.876 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:59:56.876 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.716265+0000", 2024-09-21T13:59:56.876 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0085100430000000001 2024-09-21T13:59:56.876 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.877 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.877 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:56.877 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.761562+0000", 2024-09-21T13:59:56.877 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.045296967 2024-09-21T13:59:56.877 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.877 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.877 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:59:56.878 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.761602+0000", 2024-09-21T13:59:56.878 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9913999999999998e-05 2024-09-21T13:59:56.878 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.878 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.878 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:59:56.878 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.761693+0000", 2024-09-21T13:59:56.878 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0564999999999998e-05 2024-09-21T13:59:56.878 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:56.878 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:59:56.878 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:56.878 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.878 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.878 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.15207.0:211128 8.b 8:dde34708:::benchmark_data_smithi049_109311_object84609:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-09-21T13:59:56.878 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:56:54.629541+0000", 2024-09-21T13:59:56.879 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 182.17827082299999, 2024-09-21T13:59:56.879 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.14686216199999999, 2024-09-21T13:59:56.879 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:59:56.879 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:59:56.879 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:59:56.879 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:59:56.879 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.15207", 2024-09-21T13:59:56.879 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:59:56.879 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 211128 2024-09-21T13:59:56.879 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.879 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:59:56.879 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.880 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:59:56.880 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.629541+0000", 2024-09-21T13:59:56.880 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:56.880 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.880 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.880 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:59:56.880 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.629541+0000", 2024-09-21T13:59:56.880 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:56.880 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.880 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.880 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:59:56.880 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.629541+0000", 2024-09-21T13:59:56.880 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0699999999999998e-07 2024-09-21T13:59:56.880 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.880 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.881 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:59:56.881 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.629543+0000", 2024-09-21T13:59:56.881 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.513e-06 2024-09-21T13:59:56.881 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.881 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.881 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:59:56.881 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.629546+0000", 2024-09-21T13:59:56.881 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.642e-06 2024-09-21T13:59:56.881 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.881 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.881 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:59:56.881 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.629549+0000", 2024-09-21T13:59:56.881 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8480000000000001e-06 2024-09-21T13:59:56.881 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.882 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.882 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:59:56.882 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.710199+0000", 2024-09-21T13:59:56.882 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.080649352999999993 2024-09-21T13:59:56.882 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.882 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.882 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:59:56.882 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.710312+0000", 2024-09-21T13:59:56.882 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011377799999999999 2024-09-21T13:59:56.882 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.882 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.882 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1", 2024-09-21T13:59:56.882 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.710674+0000", 2024-09-21T13:59:56.883 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00036162500000000002 2024-09-21T13:59:56.883 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.883 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.883 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:59:56.883 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.716270+0000", 2024-09-21T13:59:56.883 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0055961520000000001 2024-09-21T13:59:56.883 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.883 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.883 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:56.883 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.776233+0000", 2024-09-21T13:59:56.883 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.059962834999999999 2024-09-21T13:59:56.883 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.883 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.884 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:59:56.884 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.776304+0000", 2024-09-21T13:59:56.884 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1371999999999994e-05 2024-09-21T13:59:56.884 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.884 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.884 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:59:56.884 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.776403+0000", 2024-09-21T13:59:56.884 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.8737e-05 2024-09-21T13:59:56.884 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:56.884 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:59:56.884 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:56.884 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.884 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.884 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.15207.0:212072 8.5 8:af76a4b5:::benchmark_data_smithi049_109311_object85553:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-09-21T13:59:56.885 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:56:55.635001+0000", 2024-09-21T13:59:56.885 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 181.172810795, 2024-09-21T13:59:56.885 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.10843544099999999, 2024-09-21T13:59:56.885 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:59:56.885 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:59:56.885 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:59:56.885 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:59:56.885 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.15207", 2024-09-21T13:59:56.885 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:59:56.885 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 212072 2024-09-21T13:59:56.885 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.885 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:59:56.885 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.886 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:59:56.886 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:55.635001+0000", 2024-09-21T13:59:56.886 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:56.886 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.886 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.886 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:59:56.886 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:55.635001+0000", 2024-09-21T13:59:56.886 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:56.886 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.886 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.886 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:59:56.886 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:55.635003+0000", 2024-09-21T13:59:56.886 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.533e-06 2024-09-21T13:59:56.886 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.887 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.887 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:59:56.887 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:55.635008+0000", 2024-09-21T13:59:56.887 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0839999999999999e-06 2024-09-21T13:59:56.887 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.887 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.887 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:59:56.887 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:55.635021+0000", 2024-09-21T13:59:56.887 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3047000000000001e-05 2024-09-21T13:59:56.887 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.887 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.887 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:59:56.887 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:55.635029+0000", 2024-09-21T13:59:56.887 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0499999999999992e-06 2024-09-21T13:59:56.888 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.888 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.888 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:59:56.888 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:55.635089+0000", 2024-09-21T13:59:56.888 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0118999999999998e-05 2024-09-21T13:59:56.888 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.888 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.888 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:59:56.888 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:55.635218+0000", 2024-09-21T13:59:56.888 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012881000000000001 2024-09-21T13:59:56.888 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.888 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.888 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-21T13:59:56.888 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:55.635330+0000", 2024-09-21T13:59:56.889 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000112575 2024-09-21T13:59:56.889 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.889 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.889 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:59:56.889 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:55.638008+0000", 2024-09-21T13:59:56.889 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0026773069999999999 2024-09-21T13:59:56.889 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.889 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.889 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:56.889 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:55.639072+0000", 2024-09-21T13:59:56.889 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00106401 2024-09-21T13:59:56.889 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.890 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.890 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:56.890 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:55.743329+0000", 2024-09-21T13:59:56.890 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.10425733 2024-09-21T13:59:56.890 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.890 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.890 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:59:56.890 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:55.743390+0000", 2024-09-21T13:59:56.890 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1204000000000006e-05 2024-09-21T13:59:56.890 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.890 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.890 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:59:56.890 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:55.743436+0000", 2024-09-21T13:59:56.891 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6372000000000003e-05 2024-09-21T13:59:56.891 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:56.891 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:59:56.891 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:56.891 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.891 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.891 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.15207.0:232030 8.2 8:49ac6d95:::benchmark_data_smithi049_109311_object105511:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T13:59:56.891 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:57:11.653701+0000", 2024-09-21T13:59:56.891 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 165.154110688, 2024-09-21T13:59:56.891 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.22905246600000001, 2024-09-21T13:59:56.891 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:59:56.891 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:59:56.892 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:59:56.892 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:59:56.892 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.15207", 2024-09-21T13:59:56.892 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:59:56.892 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 232030 2024-09-21T13:59:56.892 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.892 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:59:56.892 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.892 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:59:56.892 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.653701+0000", 2024-09-21T13:59:56.892 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:56.892 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.892 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.892 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:59:56.893 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.653701+0000", 2024-09-21T13:59:56.893 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:56.893 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.893 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.893 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:59:56.893 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.653702+0000", 2024-09-21T13:59:56.893 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3600000000000002e-07 2024-09-21T13:59:56.893 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.893 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.893 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:59:56.893 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.653706+0000", 2024-09-21T13:59:56.893 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6490000000000001e-06 2024-09-21T13:59:56.893 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.894 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.894 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:59:56.894 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.653716+0000", 2024-09-21T13:59:56.894 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0509999999999999e-05 2024-09-21T13:59:56.894 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.894 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.894 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:59:56.894 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.653721+0000", 2024-09-21T13:59:56.894 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1279999999999999e-06 2024-09-21T13:59:56.894 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.894 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.894 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:59:56.894 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.653764+0000", 2024-09-21T13:59:56.894 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2531999999999999e-05 2024-09-21T13:59:56.895 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.895 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.895 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:59:56.895 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.653848+0000", 2024-09-21T13:59:56.895 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4091000000000005e-05 2024-09-21T13:59:56.895 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.895 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.895 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,5", 2024-09-21T13:59:56.895 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.653914+0000", 2024-09-21T13:59:56.895 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5977999999999997e-05 2024-09-21T13:59:56.895 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.895 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.895 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:59:56.895 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.656207+0000", 2024-09-21T13:59:56.896 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0022932690000000001 2024-09-21T13:59:56.896 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.896 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.896 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:56.896 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.659356+0000", 2024-09-21T13:59:56.896 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0031493160000000001 2024-09-21T13:59:56.896 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.896 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.896 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:56.896 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.882645+0000", 2024-09-21T13:59:56.897 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.22328900700000001 2024-09-21T13:59:56.897 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.897 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.897 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:59:56.897 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.882706+0000", 2024-09-21T13:59:56.897 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0279999999999999e-05 2024-09-21T13:59:56.897 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.897 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.897 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:59:56.897 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.882754+0000", 2024-09-21T13:59:56.897 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7870000000000001e-05 2024-09-21T13:59:56.897 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:56.897 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:59:56.897 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:56.898 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.898 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.898 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.15207.0:232036 8.2 8:4507ec93:::benchmark_data_smithi049_109311_object105517:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T13:59:56.898 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:57:11.656238+0000", 2024-09-21T13:59:56.898 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 165.151573759, 2024-09-21T13:59:56.898 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.22665437799999999, 2024-09-21T13:59:56.898 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:59:56.898 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:59:56.898 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:59:56.898 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:59:56.898 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.15207", 2024-09-21T13:59:56.898 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:59:56.899 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 232036 2024-09-21T13:59:56.899 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.899 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:59:56.899 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.899 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:59:56.899 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.656238+0000", 2024-09-21T13:59:56.899 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:56.899 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.899 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.899 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:59:56.899 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.656238+0000", 2024-09-21T13:59:56.899 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:56.900 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.900 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.900 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:59:56.900 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.656239+0000", 2024-09-21T13:59:56.900 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9100000000000003e-07 2024-09-21T13:59:56.900 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.900 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.900 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:59:56.900 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.656243+0000", 2024-09-21T13:59:56.900 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1230000000000002e-06 2024-09-21T13:59:56.900 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.900 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.900 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:59:56.900 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.656252+0000", 2024-09-21T13:59:56.901 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7490000000000004e-06 2024-09-21T13:59:56.901 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.901 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.901 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:59:56.901 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.656259+0000", 2024-09-21T13:59:56.901 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7559999999999999e-06 2024-09-21T13:59:56.901 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.901 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.901 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:59:56.901 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.656310+0000", 2024-09-21T13:59:56.901 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0203e-05 2024-09-21T13:59:56.901 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.901 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.901 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:59:56.901 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.656440+0000", 2024-09-21T13:59:56.902 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000130376 2024-09-21T13:59:56.902 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.902 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.902 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,5", 2024-09-21T13:59:56.902 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.656550+0000", 2024-09-21T13:59:56.902 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000109812 2024-09-21T13:59:56.902 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.902 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.902 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:59:56.902 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.658548+0000", 2024-09-21T13:59:56.902 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0019982649999999999 2024-09-21T13:59:56.902 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.902 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.902 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:56.903 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.659437+0000", 2024-09-21T13:59:56.903 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000888743 2024-09-21T13:59:56.903 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.903 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.903 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:56.903 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.882792+0000", 2024-09-21T13:59:56.903 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.22335466200000001 2024-09-21T13:59:56.903 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.903 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.903 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:59:56.903 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.882858+0000", 2024-09-21T13:59:56.903 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6512000000000003e-05 2024-09-21T13:59:56.903 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.903 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.904 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:59:56.904 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.882892+0000", 2024-09-21T13:59:56.904 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4385999999999999e-05 2024-09-21T13:59:56.904 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:56.904 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:59:56.904 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:56.904 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.904 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.904 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.15207.0:232041 8.2 8:42fefff6:::benchmark_data_smithi049_109311_object105522:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T13:59:56.904 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:57:11.674073+0000", 2024-09-21T13:59:56.904 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 165.133738991, 2024-09-21T13:59:56.905 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21533840400000001, 2024-09-21T13:59:56.905 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:59:56.905 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:59:56.905 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:59:56.905 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:59:56.905 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.15207", 2024-09-21T13:59:56.905 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:59:56.905 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 232041 2024-09-21T13:59:56.905 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.905 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:59:56.905 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.905 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:59:56.906 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.674073+0000", 2024-09-21T13:59:56.906 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:56.906 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.906 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.906 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:59:56.906 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.674073+0000", 2024-09-21T13:59:56.906 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:56.906 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.906 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.906 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:59:56.906 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.674074+0000", 2024-09-21T13:59:56.906 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5889999999999999e-06 2024-09-21T13:59:56.906 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.906 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.906 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:59:56.907 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.674080+0000", 2024-09-21T13:59:56.907 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1290000000000004e-06 2024-09-21T13:59:56.907 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.907 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.907 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:59:56.907 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.674102+0000", 2024-09-21T13:59:56.907 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2541e-05 2024-09-21T13:59:56.907 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.907 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.907 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:59:56.907 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.674112+0000", 2024-09-21T13:59:56.907 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.5310000000000007e-06 2024-09-21T13:59:56.907 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.907 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.908 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:59:56.908 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.674234+0000", 2024-09-21T13:59:56.908 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012247000000000001 2024-09-21T13:59:56.908 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.908 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.908 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:59:56.908 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.674365+0000", 2024-09-21T13:59:56.908 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013083300000000001 2024-09-21T13:59:56.908 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.908 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.908 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,5", 2024-09-21T13:59:56.908 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.674489+0000", 2024-09-21T13:59:56.908 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000124079 2024-09-21T13:59:56.908 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.908 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.909 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:59:56.909 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.680300+0000", 2024-09-21T13:59:56.909 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0058106709999999999 2024-09-21T13:59:56.909 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.909 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.909 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:56.909 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.681669+0000", 2024-09-21T13:59:56.909 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0013690460000000001 2024-09-21T13:59:56.909 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.909 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.909 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:56.909 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.889300+0000", 2024-09-21T13:59:56.909 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20763156999999999 2024-09-21T13:59:56.909 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.909 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.910 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:59:56.910 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.889364+0000", 2024-09-21T13:59:56.910 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3979000000000005e-05 2024-09-21T13:59:56.910 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.910 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.910 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:59:56.910 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.889411+0000", 2024-09-21T13:59:56.910 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6965999999999997e-05 2024-09-21T13:59:56.910 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:56.910 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:59:56.910 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:56.910 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.910 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.911 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 4, 5, 6, 2, 3, 7] 2024-09-21T13:59:56.911 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T13:59:56.911 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T13:59:56.913 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:59:56.914 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.15207.0:232049 8.5 8:aa476d15:::benchmark_data_smithi049_109311_object105530:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T13:59:56.914 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:57:11.686170+0000", 2024-09-21T13:59:56.914 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 165.12164203, 2024-09-21T13:59:56.914 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20563606700000001, 2024-09-21T13:59:56.914 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:59:56.914 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:59:56.914 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:59:56.914 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:59:56.915 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.15207", 2024-09-21T13:59:56.915 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:59:56.915 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 232049 2024-09-21T13:59:56.915 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.915 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:59:56.915 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.915 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:59:56.915 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.686170+0000", 2024-09-21T13:59:56.915 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:56.915 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.915 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.915 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:59:56.915 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.686170+0000", 2024-09-21T13:59:56.915 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:56.916 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.916 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.916 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:59:56.916 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.686172+0000", 2024-09-21T13:59:56.916 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7710000000000001e-06 2024-09-21T13:59:56.916 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.916 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.916 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:59:56.916 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.686175+0000", 2024-09-21T13:59:56.916 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6239999999999999e-06 2024-09-21T13:59:56.916 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.916 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.916 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:59:56.916 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.686185+0000", 2024-09-21T13:59:56.917 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6909999999999992e-06 2024-09-21T13:59:56.917 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.917 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.917 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:59:56.917 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.686194+0000", 2024-09-21T13:59:56.917 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0769999999999997e-06 2024-09-21T13:59:56.917 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.917 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.917 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:59:56.917 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.686258+0000", 2024-09-21T13:59:56.917 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4239000000000006e-05 2024-09-21T13:59:56.917 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.917 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.918 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:59:56.918 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.686374+0000", 2024-09-21T13:59:56.918 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000116148 2024-09-21T13:59:56.918 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.918 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.918 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-21T13:59:56.918 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.686491+0000", 2024-09-21T13:59:56.918 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011667000000000001 2024-09-21T13:59:56.918 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.918 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.919 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:59:56.919 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.690120+0000", 2024-09-21T13:59:56.919 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003628797 2024-09-21T13:59:56.919 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.919 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.919 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:56.919 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.690188+0000", 2024-09-21T13:59:56.919 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8201000000000007e-05 2024-09-21T13:59:56.919 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.919 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.919 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:56.919 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.891730+0000", 2024-09-21T13:59:56.920 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20154166900000001 2024-09-21T13:59:56.920 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.920 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.920 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:59:56.920 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.891769+0000", 2024-09-21T13:59:56.920 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9409999999999997e-05 2024-09-21T13:59:56.920 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.920 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.920 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:59:56.920 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.891806+0000", 2024-09-21T13:59:56.920 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6770000000000002e-05 2024-09-21T13:59:56.920 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:56.920 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:59:56.921 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:56.921 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.921 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.921 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.15207.0:232052 8.5 8:aec88863:::benchmark_data_smithi049_109311_object105533:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T13:59:56.921 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:57:11.686395+0000", 2024-09-21T13:59:56.921 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 165.121416788, 2024-09-21T13:59:56.921 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20789333199999999, 2024-09-21T13:59:56.921 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:59:56.921 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:59:56.921 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:59:56.921 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:59:56.921 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.15207", 2024-09-21T13:59:56.921 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:59:56.921 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 232052 2024-09-21T13:59:56.921 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.922 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:59:56.922 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.922 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:59:56.922 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.686395+0000", 2024-09-21T13:59:56.922 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:56.922 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.922 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.922 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:59:56.922 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.686395+0000", 2024-09-21T13:59:56.922 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:56.922 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.922 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.922 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:59:56.922 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.686396+0000", 2024-09-21T13:59:56.923 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4600000000000004e-07 2024-09-21T13:59:56.923 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.923 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.923 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:59:56.923 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.686399+0000", 2024-09-21T13:59:56.923 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8329999999999998e-06 2024-09-21T13:59:56.923 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.923 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.923 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:59:56.923 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.686404+0000", 2024-09-21T13:59:56.923 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8220000000000003e-06 2024-09-21T13:59:56.923 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.923 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.923 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:59:56.924 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.686408+0000", 2024-09-21T13:59:56.924 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0040000000000001e-06 2024-09-21T13:59:56.924 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.924 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.924 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:59:56.924 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.688420+0000", 2024-09-21T13:59:56.924 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.002011896 2024-09-21T13:59:56.924 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.924 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.924 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:59:56.924 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.688526+0000", 2024-09-21T13:59:56.924 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000106031 2024-09-21T13:59:56.924 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.924 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.924 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-21T13:59:56.925 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.688626+0000", 2024-09-21T13:59:56.925 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000100185 2024-09-21T13:59:56.925 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.925 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.925 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:59:56.925 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.690125+0000", 2024-09-21T13:59:56.925 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0014986820000000001 2024-09-21T13:59:56.925 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.925 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.925 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:56.925 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.690262+0000", 2024-09-21T13:59:56.925 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000136444 2024-09-21T13:59:56.925 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.925 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.926 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:56.926 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.894191+0000", 2024-09-21T13:59:56.926 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.203929104 2024-09-21T13:59:56.926 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.926 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.926 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:59:56.926 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.894251+0000", 2024-09-21T13:59:56.926 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9868999999999999e-05 2024-09-21T13:59:56.926 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.926 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.926 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:59:56.926 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.894288+0000", 2024-09-21T13:59:56.926 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7815999999999998e-05 2024-09-21T13:59:56.926 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:56.926 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:59:56.927 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:56.927 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.927 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.927 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.15207.0:232055 8.2 8:459633e6:::benchmark_data_smithi049_109311_object105536:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T13:59:56.927 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:57:11.693418+0000", 2024-09-21T13:59:56.927 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 165.114394135, 2024-09-21T13:59:56.927 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.19827633, 2024-09-21T13:59:56.927 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:59:56.927 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:59:56.927 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:59:56.927 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:59:56.927 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.15207", 2024-09-21T13:59:56.927 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T13:59:56.927 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 232055 2024-09-21T13:59:56.928 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.928 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:59:56.928 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.928 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:59:56.928 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.693418+0000", 2024-09-21T13:59:56.928 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:56.928 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.928 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.928 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:59:56.928 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.693418+0000", 2024-09-21T13:59:56.928 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:56.928 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.929 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.929 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:59:56.929 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.693419+0000", 2024-09-21T13:59:56.929 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2100000000000001e-06 2024-09-21T13:59:56.929 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.929 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.929 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:59:56.929 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.693423+0000", 2024-09-21T13:59:56.929 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4669999999999998e-06 2024-09-21T13:59:56.929 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.929 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.929 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:59:56.929 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.693433+0000", 2024-09-21T13:59:56.929 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.8120000000000002e-06 2024-09-21T13:59:56.929 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.930 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.930 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:59:56.930 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.693438+0000", 2024-09-21T13:59:56.930 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2660000000000001e-06 2024-09-21T13:59:56.930 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.930 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.930 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:59:56.930 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.693476+0000", 2024-09-21T13:59:56.930 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7276e-05 2024-09-21T13:59:56.930 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.930 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.930 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:59:56.930 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.693549+0000", 2024-09-21T13:59:56.931 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3133999999999995e-05 2024-09-21T13:59:56.931 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.931 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.931 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,5", 2024-09-21T13:59:56.931 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.693612+0000", 2024-09-21T13:59:56.931 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3392000000000006e-05 2024-09-21T13:59:56.931 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.931 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.931 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:59:56.931 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.695497+0000", 2024-09-21T13:59:56.931 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0018845310000000001 2024-09-21T13:59:56.931 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.931 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.931 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:56.931 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.695811+0000", 2024-09-21T13:59:56.932 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00031459700000000001 2024-09-21T13:59:56.932 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.932 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.932 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:56.932 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.891590+0000", 2024-09-21T13:59:56.932 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.19577844999999999 2024-09-21T13:59:56.932 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.932 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.932 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:59:56.932 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.891645+0000", 2024-09-21T13:59:56.932 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4889999999999998e-05 2024-09-21T13:59:56.933 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.933 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.933 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:59:56.933 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.891694+0000", 2024-09-21T13:59:56.933 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9305000000000003e-05 2024-09-21T13:59:56.933 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:56.933 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:59:56.933 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:56.933 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.933 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.933 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:73960 9.e 9:7e61c187:::benchmark_data_smithi049_138504_object73959:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-09-21T13:59:56.933 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:58:52.916953+0000", 2024-09-21T13:59:56.933 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 63.890859280999997, 2024-09-21T13:59:56.934 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.12133540700000001, 2024-09-21T13:59:56.934 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:59:56.934 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:59:56.934 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:59:56.934 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:59:56.934 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T13:59:56.934 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T13:59:56.934 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 73960 2024-09-21T13:59:56.934 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.934 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:59:56.934 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.934 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:59:56.935 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:52.916953+0000", 2024-09-21T13:59:56.935 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:56.935 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.935 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.935 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:59:56.935 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:52.916953+0000", 2024-09-21T13:59:56.935 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:56.935 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.935 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.935 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:59:56.935 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:52.916964+0000", 2024-09-21T13:59:56.935 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1386000000000001e-05 2024-09-21T13:59:56.935 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.936 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.936 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:59:56.936 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:52.917074+0000", 2024-09-21T13:59:56.936 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010969299999999999 2024-09-21T13:59:56.936 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.936 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.936 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:59:56.936 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:52.917094+0000", 2024-09-21T13:59:56.936 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0251e-05 2024-09-21T13:59:56.936 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.936 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.936 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:59:56.936 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:52.917102+0000", 2024-09-21T13:59:56.936 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3000000000000002e-06 2024-09-21T13:59:56.936 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.937 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.937 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:59:56.937 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:52.917160+0000", 2024-09-21T13:59:56.937 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7405999999999999e-05 2024-09-21T13:59:56.937 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.937 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.937 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:59:56.937 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:52.917308+0000", 2024-09-21T13:59:56.937 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000148085 2024-09-21T13:59:56.937 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.937 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.937 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1", 2024-09-21T13:59:56.937 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:52.917467+0000", 2024-09-21T13:59:56.937 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015944200000000001 2024-09-21T13:59:56.937 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.938 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.938 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:59:56.938 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:52.919847+0000", 2024-09-21T13:59:56.938 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0023794250000000001 2024-09-21T13:59:56.938 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.938 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.938 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:56.938 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:53.038132+0000", 2024-09-21T13:59:56.938 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.11828538099999999 2024-09-21T13:59:56.938 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.938 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.938 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:59:56.938 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:53.038197+0000", 2024-09-21T13:59:56.938 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4744000000000002e-05 2024-09-21T13:59:56.939 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.939 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.939 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:59:56.939 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:53.038288+0000", 2024-09-21T13:59:56.939 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1293999999999997e-05 2024-09-21T13:59:56.939 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:56.939 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:59:56.939 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:56.939 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.939 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.939 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:73970 9.e 9:74881f58:::benchmark_data_smithi049_138504_object73969:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-09-21T13:59:56.939 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:58:52.924967+0000", 2024-09-21T13:59:56.939 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 63.882845146999998, 2024-09-21T13:59:56.939 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.113448892, 2024-09-21T13:59:56.939 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:59:56.940 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:59:56.940 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T13:59:56.940 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:59:56.940 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T13:59:56.940 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T13:59:56.940 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 73970 2024-09-21T13:59:56.940 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.940 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:59:56.940 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.940 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:59:56.940 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:52.924967+0000", 2024-09-21T13:59:56.940 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:56.940 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.940 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.941 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:59:56.941 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:52.924967+0000", 2024-09-21T13:59:56.941 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:56.941 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.941 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.941 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:59:56.941 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:52.924969+0000", 2024-09-21T13:59:56.941 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.03e-06 2024-09-21T13:59:56.941 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.941 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.941 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:59:56.941 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:52.925016+0000", 2024-09-21T13:59:56.941 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7216000000000003e-05 2024-09-21T13:59:56.941 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.942 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.942 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:59:56.942 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:52.925033+0000", 2024-09-21T13:59:56.942 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7079e-05 2024-09-21T13:59:56.942 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.942 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.942 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:59:56.942 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:52.925041+0000", 2024-09-21T13:59:56.942 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.106e-06 2024-09-21T13:59:56.942 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.942 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.942 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:59:56.942 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:52.925108+0000", 2024-09-21T13:59:56.943 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7175999999999995e-05 2024-09-21T13:59:56.943 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.943 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.943 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:59:56.943 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:52.925245+0000", 2024-09-21T13:59:56.943 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013656700000000001 2024-09-21T13:59:56.943 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.943 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.943 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1", 2024-09-21T13:59:56.943 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:52.925380+0000", 2024-09-21T13:59:56.943 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013513199999999999 2024-09-21T13:59:56.943 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.943 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.943 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:59:56.943 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:52.927803+0000", 2024-09-21T13:59:56.944 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0024229949999999998 2024-09-21T13:59:56.944 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.944 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.944 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:56.944 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:53.038326+0000", 2024-09-21T13:59:56.944 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.11052324700000001 2024-09-21T13:59:56.944 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.944 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.944 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T13:59:56.944 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:53.038366+0000", 2024-09-21T13:59:56.944 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9715999999999997e-05 2024-09-21T13:59:56.944 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:56.944 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:56.945 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T13:59:56.945 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:53.038416+0000", 2024-09-21T13:59:56.945 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9628e-05 2024-09-21T13:59:56.945 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:56.945 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:59:56.945 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:56.945 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:56.945 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:59:56.945 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T13:59:56.980 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:56.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:56.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:59:57.082 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:57.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:57.131 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:57.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:57.232 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:57.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:57 smithi049 ceph-mon[30080]: 5.e scrub starts 2024-09-21T13:59:57.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:57 smithi049 ceph-mon[30080]: 5.e scrub ok 2024-09-21T13:59:57.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:57 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T13:59:57.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:57 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T13:59:57.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:57 smithi049 ceph-mon[30080]: pgmap v549: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait+forced_backfill, 2 active+recovering+undersized+degraded+remapped+forced_backfill, 2 active+undersized+degraded+remapped+forced_backfill+backfill_toofull, 1 active+remapped+backfill_toofull, 139 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 672 op/s; 45358/320016 objects degraded (14.174%); 6063/320016 objects misplaced (1.895%); 4.4 MiB/s, 70 objects/s recovering 2024-09-21T13:59:57.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:57 smithi049 ceph-mon[30080]: osdmap e307: 8 total, 8 up, 8 in 2024-09-21T13:59:57.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:57 smithi049 ceph-mon[40489]: 5.e scrub starts 2024-09-21T13:59:57.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:57 smithi049 ceph-mon[40489]: 5.e scrub ok 2024-09-21T13:59:57.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:57 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T13:59:57.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:57 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T13:59:57.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:57 smithi049 ceph-mon[40489]: pgmap v549: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait+forced_backfill, 2 active+recovering+undersized+degraded+remapped+forced_backfill, 2 active+undersized+degraded+remapped+forced_backfill+backfill_toofull, 1 active+remapped+backfill_toofull, 139 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 672 op/s; 45358/320016 objects degraded (14.174%); 6063/320016 objects misplaced (1.895%); 4.4 MiB/s, 70 objects/s recovering 2024-09-21T13:59:57.256 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:57 smithi049 ceph-mon[40489]: osdmap e307: 8 total, 8 up, 8 in 2024-09-21T13:59:57.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:57.272 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:57.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:57.336 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:57 smithi187 ceph-mon[33032]: 5.e scrub starts 2024-09-21T13:59:57.337 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:57 smithi187 ceph-mon[33032]: 5.e scrub ok 2024-09-21T13:59:57.337 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:57 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T13:59:57.337 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:57 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T13:59:57.337 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:57 smithi187 ceph-mon[33032]: pgmap v549: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait+forced_backfill, 2 active+recovering+undersized+degraded+remapped+forced_backfill, 2 active+undersized+degraded+remapped+forced_backfill+backfill_toofull, 1 active+remapped+backfill_toofull, 139 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 672 op/s; 45358/320016 objects degraded (14.174%); 6063/320016 objects misplaced (1.895%); 4.4 MiB/s, 70 objects/s recovering 2024-09-21T13:59:57.337 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:57 smithi187 ceph-mon[33032]: osdmap e307: 8 total, 8 up, 8 in 2024-09-21T13:59:57.373 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:57.411 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:57.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:57.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:57.513 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:57.549 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:57.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:57.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:57.651 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:57.696 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:57.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:57.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:57.798 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:57.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:57.846 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:57.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:57.878 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T13:59:57.947 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:57.980 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:57.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:57.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:58.083 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T13:59:58.133 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:58.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:58.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T13:59:58.195 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:59:58.205 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T13:59:58.237 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:59:58.280 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:58.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:58.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:59:58.367 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T13:59:58.383 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T13:59:58.415 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:58.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:58.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T13:59:58.516 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:58.557 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:58.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:58.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:58.659 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:59:58.704 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:58.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:58.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:59:58.806 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:58.844 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:58.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:58.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:58.946 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T13:59:58.981 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:58.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:58.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T13:59:59.019 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T13:59:59.019 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-21T13:59:59.019 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.019 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:141221 9.7 9.def8d17 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-21T13:59:59.019 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:59:59.010642+0000", 2024-09-21T13:59:59.020 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0069475769999999999, 2024-09-21T13:59:59.020 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0069721419999999997, 2024-09-21T13:59:59.020 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:59:59.020 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:59:59.020 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "queued for pg", 2024-09-21T13:59:59.020 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:59:59.020 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T13:59:59.020 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T13:59:59.020 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 141221 2024-09-21T13:59:59.020 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.020 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:59:59.020 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.020 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:59:59.021 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.010642+0000", 2024-09-21T13:59:59.021 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:59.021 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.021 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.021 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:59:59.021 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.010642+0000", 2024-09-21T13:59:59.021 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:59.021 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.021 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.021 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:59:59.021 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.010643+0000", 2024-09-21T13:59:59.021 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8789999999999999e-06 2024-09-21T13:59:59.021 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.021 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.022 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:59:59.022 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.010664+0000", 2024-09-21T13:59:59.022 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0469e-05 2024-09-21T13:59:59.022 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.022 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.022 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:59:59.022 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.010675+0000", 2024-09-21T13:59:59.022 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1139e-05 2024-09-21T13:59:59.022 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.022 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.022 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:59:59.023 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.010682+0000", 2024-09-21T13:59:59.023 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9870000000000002e-06 2024-09-21T13:59:59.023 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:59.023 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:59:59.023 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:59.023 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.023 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.023 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:141224 9.a 9:5a14c328:::benchmark_data_smithi049_138504_object19816:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-21T13:59:59.023 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:59:59.011278+0000", 2024-09-21T13:59:59.023 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0063113010000000001, 2024-09-21T13:59:59.023 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0063871659999999997, 2024-09-21T13:59:59.023 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:59:59.023 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:59:59.023 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-21T13:59:59.024 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:59:59.024 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T13:59:59.024 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T13:59:59.024 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 141224 2024-09-21T13:59:59.024 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.024 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:59:59.024 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.024 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:59:59.024 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.011278+0000", 2024-09-21T13:59:59.024 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:59.024 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.024 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.024 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:59:59.025 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.011278+0000", 2024-09-21T13:59:59.025 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:59.025 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.025 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.025 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:59:59.025 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.011279+0000", 2024-09-21T13:59:59.025 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4600000000000003e-07 2024-09-21T13:59:59.025 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.025 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.025 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:59:59.025 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.011284+0000", 2024-09-21T13:59:59.025 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9899999999999997e-06 2024-09-21T13:59:59.025 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.025 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.025 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:59:59.026 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.011294+0000", 2024-09-21T13:59:59.026 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0186999999999999e-05 2024-09-21T13:59:59.026 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.026 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.026 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:59:59.026 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.011300+0000", 2024-09-21T13:59:59.026 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4999999999999996e-06 2024-09-21T13:59:59.026 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.026 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.026 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:59:59.026 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.017562+0000", 2024-09-21T13:59:59.026 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0062621049999999996 2024-09-21T13:59:59.026 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:59.027 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:59:59.027 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:59.027 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.027 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.027 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:141225 9.6 9:671d7a66:::benchmark_data_smithi049_138504_object19817:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-21T13:59:59.027 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:59:59.011389+0000", 2024-09-21T13:59:59.027 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0062004360000000001, 2024-09-21T13:59:59.027 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0063244140000000004, 2024-09-21T13:59:59.027 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:59:59.027 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:59:59.027 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-21T13:59:59.027 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:59:59.027 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T13:59:59.028 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T13:59:59.028 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 141225 2024-09-21T13:59:59.028 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.028 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:59:59.028 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.028 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:59:59.028 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.011389+0000", 2024-09-21T13:59:59.028 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:59.028 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.028 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.028 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:59:59.028 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.011389+0000", 2024-09-21T13:59:59.028 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:59.028 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.028 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.029 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:59:59.029 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.011389+0000", 2024-09-21T13:59:59.029 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8999999999999996e-07 2024-09-21T13:59:59.029 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.029 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.029 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:59:59.029 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.011392+0000", 2024-09-21T13:59:59.029 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9840000000000001e-06 2024-09-21T13:59:59.029 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.029 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.029 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:59:59.029 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.011399+0000", 2024-09-21T13:59:59.029 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.0140000000000001e-06 2024-09-21T13:59:59.029 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.029 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.029 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:59:59.030 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.011405+0000", 2024-09-21T13:59:59.030 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5419999999999996e-06 2024-09-21T13:59:59.030 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.030 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.030 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:59:59.030 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.011452+0000", 2024-09-21T13:59:59.030 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7364999999999998e-05 2024-09-21T13:59:59.030 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.030 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.030 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:59:59.030 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.011573+0000", 2024-09-21T13:59:59.030 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000121107 2024-09-21T13:59:59.030 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.030 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.031 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,2", 2024-09-21T13:59:59.031 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.011685+0000", 2024-09-21T13:59:59.031 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011123700000000001 2024-09-21T13:59:59.031 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.031 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.031 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:59:59.031 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.016284+0000", 2024-09-21T13:59:59.031 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0045991850000000004 2024-09-21T13:59:59.031 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:59.031 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:59:59.031 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:59.031 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.031 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.031 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25421.0:141223 9.8 e307/260 9:110614db:::benchmark_data_smithi049_138504_object19815:head v 307'8968, mlcod=307'8966)", 2024-09-21T13:59:59.031 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:59:59.014471+0000", 2024-09-21T13:59:59.032 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0031182710000000002, 2024-09-21T13:59:59.032 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0032970949999999999, 2024-09-21T13:59:59.032 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:59:59.032 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:59:59.032 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T13:59:59.032 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:59:59.032 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.032 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:59:59.032 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.014471+0000", 2024-09-21T13:59:59.032 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:59.032 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.032 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.032 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:59:59.033 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.014471+0000", 2024-09-21T13:59:59.033 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:59.033 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.033 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.033 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:59:59.033 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.014472+0000", 2024-09-21T13:59:59.033 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3710000000000001e-06 2024-09-21T13:59:59.033 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.033 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.033 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:59:59.034 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.014482+0000", 2024-09-21T13:59:59.034 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.9510000000000001e-06 2024-09-21T13:59:59.034 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.034 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.034 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:59:59.034 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.014499+0000", 2024-09-21T13:59:59.034 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6711e-05 2024-09-21T13:59:59.035 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.035 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.035 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:59:59.035 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.014507+0000", 2024-09-21T13:59:59.035 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3650000000000004e-06 2024-09-21T13:59:59.035 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.035 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.035 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:59:59.035 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.014605+0000", 2024-09-21T13:59:59.035 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7819999999999995e-05 2024-09-21T13:59:59.035 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.035 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.035 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:59:59.035 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.014628+0000", 2024-09-21T13:59:59.036 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2782999999999999e-05 2024-09-21T13:59:59.036 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:59.036 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:59:59.036 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:59.036 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.036 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.036 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25421.0:141226 9.8 e307/260 9:1aa10e43:::benchmark_data_smithi049_138504_object19818:head v 307'8969, mlcod=307'8966)", 2024-09-21T13:59:59.036 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:59:59.014534+0000", 2024-09-21T13:59:59.036 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0030546179999999998, 2024-09-21T13:59:59.036 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0032770260000000002, 2024-09-21T13:59:59.036 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:59:59.036 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:59:59.036 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T13:59:59.036 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:59:59.036 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.037 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:59:59.037 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.014534+0000", 2024-09-21T13:59:59.037 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:59.037 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.037 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.037 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:59:59.037 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.014534+0000", 2024-09-21T13:59:59.037 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:59.037 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.037 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.037 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:59:59.037 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.014535+0000", 2024-09-21T13:59:59.037 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3599999999999999e-07 2024-09-21T13:59:59.037 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.038 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.038 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:59:59.038 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.014538+0000", 2024-09-21T13:59:59.038 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4640000000000002e-06 2024-09-21T13:59:59.038 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.038 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.038 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:59:59.038 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.014544+0000", 2024-09-21T13:59:59.038 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6239999999999999e-06 2024-09-21T13:59:59.038 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.038 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.038 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:59:59.038 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.014548+0000", 2024-09-21T13:59:59.038 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8190000000000002e-06 2024-09-21T13:59:59.039 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.039 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.039 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:59:59.039 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.017156+0000", 2024-09-21T13:59:59.039 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0026078070000000002 2024-09-21T13:59:59.039 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.039 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.039 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:59:59.039 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.017173+0000", 2024-09-21T13:59:59.039 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6991999999999999e-05 2024-09-21T13:59:59.039 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:59.039 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:59:59.039 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:59.040 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.040 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.040 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.25421.0:141198 9.7 e307/299)", 2024-09-21T13:59:59.040 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:59:59.015551+0000", 2024-09-21T13:59:59.040 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0020377030000000001, 2024-09-21T13:59:59.040 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0023073780000000001, 2024-09-21T13:59:59.040 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:59:59.040 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:59:59.040 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "queued for pg", 2024-09-21T13:59:59.040 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:59:59.040 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.040 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:59:59.040 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.015551+0000", 2024-09-21T13:59:59.041 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:59.041 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.041 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.041 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:59:59.041 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.015551+0000", 2024-09-21T13:59:59.041 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:59.041 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.041 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.041 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:59:59.041 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.015552+0000", 2024-09-21T13:59:59.041 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.068e-06 2024-09-21T13:59:59.041 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.041 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.041 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:59:59.041 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.015559+0000", 2024-09-21T13:59:59.042 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3609999999999997e-06 2024-09-21T13:59:59.042 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.042 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.042 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:59:59.042 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.015571+0000", 2024-09-21T13:59:59.042 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2038000000000001e-05 2024-09-21T13:59:59.042 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.042 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.042 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:59:59.042 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.015579+0000", 2024-09-21T13:59:59.042 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0339999999999997e-06 2024-09-21T13:59:59.042 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:59.042 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:59:59.042 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:59.043 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.043 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.043 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "MOSDPGPushReply(9.5 307/299 [PushReplyOp(9:af7585a9:::benchmark_data_smithi049_138504_object48252:head)])", 2024-09-21T13:59:59.043 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:59:59.016421+0000", 2024-09-21T13:59:59.043 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.001167888, 2024-09-21T13:59:59.043 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00147043, 2024-09-21T13:59:59.043 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:59:59.043 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:59:59.043 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-21T13:59:59.043 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:59:59.043 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.043 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:59:59.044 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.016421+0000", 2024-09-21T13:59:59.044 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:59.044 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.044 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.044 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:59:59.044 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.016421+0000", 2024-09-21T13:59:59.044 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:59.044 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.044 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.044 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:59:59.044 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.016422+0000", 2024-09-21T13:59:59.044 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4199999999999995e-07 2024-09-21T13:59:59.044 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.044 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.044 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:59:59.045 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.016426+0000", 2024-09-21T13:59:59.045 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4009999999999999e-06 2024-09-21T13:59:59.045 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.045 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.045 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:59:59.045 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.016436+0000", 2024-09-21T13:59:59.045 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3670000000000003e-06 2024-09-21T13:59:59.045 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.045 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.045 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:59:59.045 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.016441+0000", 2024-09-21T13:59:59.045 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5949999999999998e-06 2024-09-21T13:59:59.045 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.045 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.046 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:59:59.046 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.016485+0000", 2024-09-21T13:59:59.046 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4134999999999997e-05 2024-09-21T13:59:59.046 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:59.046 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:59:59.046 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:59.046 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.046 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.046 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25421.0:141227 9.8 e307/260)", 2024-09-21T13:59:59.046 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:59:59.017528+0000", 2024-09-21T13:59:59.046 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 6.0986999999999997e-05, 2024-09-21T13:59:59.046 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00039716000000000003, 2024-09-21T13:59:59.047 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:59:59.047 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:59:59.047 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "queued for pg", 2024-09-21T13:59:59.047 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:59:59.047 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.047 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:59:59.047 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.017528+0000", 2024-09-21T13:59:59.047 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:59.047 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.047 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.047 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:59:59.047 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.017528+0000", 2024-09-21T13:59:59.047 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:59.047 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.048 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.048 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:59:59.048 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.017529+0000", 2024-09-21T13:59:59.048 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6299999999999993e-07 2024-09-21T13:59:59.048 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.048 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.048 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:59:59.048 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.017537+0000", 2024-09-21T13:59:59.048 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.593e-06 2024-09-21T13:59:59.048 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.048 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.048 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:59:59.048 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.017546+0000", 2024-09-21T13:59:59.048 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0790000000000007e-06 2024-09-21T13:59:59.049 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.049 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.049 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:59:59.049 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.017553+0000", 2024-09-21T13:59:59.049 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4549999999999998e-06 2024-09-21T13:59:59.049 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:59.049 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:59:59.049 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:59.049 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.049 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.049 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:141198 9.7 9:e363195b:::benchmark_data_smithi049_138504_object19790:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-21T13:59:59.049 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:59:58.987472+0000", 2024-09-21T13:59:59.049 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.03011664, 2024-09-21T13:59:59.050 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.030512574000000001, 2024-09-21T13:59:59.050 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:59:59.050 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:59:59.050 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-21T13:59:59.050 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:59:59.050 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T13:59:59.050 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T13:59:59.050 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 141198 2024-09-21T13:59:59.050 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.050 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:59:59.050 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.050 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:59:59.050 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:58.987472+0000", 2024-09-21T13:59:59.050 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:59.051 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.051 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.051 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:59:59.051 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:58.987472+0000", 2024-09-21T13:59:59.051 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:59.051 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.051 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.051 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:59:59.051 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:58.987474+0000", 2024-09-21T13:59:59.051 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3790000000000001e-06 2024-09-21T13:59:59.051 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.051 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.051 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:59:59.051 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:58.987479+0000", 2024-09-21T13:59:59.052 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0250000000000002e-06 2024-09-21T13:59:59.052 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.052 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.052 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:59:59.052 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:58.987488+0000", 2024-09-21T13:59:59.052 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.2890000000000003e-06 2024-09-21T13:59:59.052 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.052 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.052 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:59:59.052 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:58.987495+0000", 2024-09-21T13:59:59.052 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5440000000000004e-06 2024-09-21T13:59:59.052 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.052 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.052 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:59:59.053 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:58.993434+0000", 2024-09-21T13:59:59.053 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0059390509999999999 2024-09-21T13:59:59.053 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.053 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.053 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:59:59.053 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:58.993575+0000", 2024-09-21T13:59:59.053 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000140839 2024-09-21T13:59:59.053 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.053 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.053 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,7", 2024-09-21T13:59:59.053 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:58.993721+0000", 2024-09-21T13:59:59.053 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014677599999999999 2024-09-21T13:59:59.053 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.053 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.054 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T13:59:59.054 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.008634+0000", 2024-09-21T13:59:59.054 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.014912997000000001 2024-09-21T13:59:59.054 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.054 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.054 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T13:59:59.054 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.008690+0000", 2024-09-21T13:59:59.054 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5736999999999998e-05 2024-09-21T13:59:59.054 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:59.054 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:59:59.054 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:59.054 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.054 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.055 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:141220 9.7 9:eb62bcb2:::benchmark_data_smithi049_138504_object19812:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-21T13:59:59.055 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:59:59.007895+0000", 2024-09-21T13:59:59.055 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0096942339999999995, 2024-09-21T13:59:59.055 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.010150773, 2024-09-21T13:59:59.055 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T13:59:59.055 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T13:59:59.055 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-21T13:59:59.055 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T13:59:59.055 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T13:59:59.055 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T13:59:59.055 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 141220 2024-09-21T13:59:59.055 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.055 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T13:59:59.055 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.056 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T13:59:59.056 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.007895+0000", 2024-09-21T13:59:59.056 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:59.056 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.056 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.056 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T13:59:59.056 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.007895+0000", 2024-09-21T13:59:59.056 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T13:59:59.056 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.056 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.056 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T13:59:59.056 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.007896+0000", 2024-09-21T13:59:59.056 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4589999999999999e-06 2024-09-21T13:59:59.056 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.057 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.057 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T13:59:59.057 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.007902+0000", 2024-09-21T13:59:59.057 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.631e-06 2024-09-21T13:59:59.057 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.057 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.057 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T13:59:59.057 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.007916+0000", 2024-09-21T13:59:59.057 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4413000000000001e-05 2024-09-21T13:59:59.057 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.057 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.057 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T13:59:59.058 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.007924+0000", 2024-09-21T13:59:59.058 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6590000000000007e-06 2024-09-21T13:59:59.058 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.058 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.058 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T13:59:59.058 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.015208+0000", 2024-09-21T13:59:59.058 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0072842489999999996 2024-09-21T13:59:59.058 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.058 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.058 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T13:59:59.058 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.015348+0000", 2024-09-21T13:59:59.058 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013953899999999999 2024-09-21T13:59:59.058 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T13:59:59.059 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T13:59:59.059 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,7", 2024-09-21T13:59:59.059 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:59.015499+0000", 2024-09-21T13:59:59.059 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015068300000000001 2024-09-21T13:59:59.059 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:59.059 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T13:59:59.059 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:59.059 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T13:59:59.059 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2024-09-21T13:59:59.059 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 10 2024-09-21T13:59:59.059 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T13:59:59.083 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T13:59:59.117 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:59.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:59.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T13:59:59.158 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T13:59:59.160 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-09-21T13:59:59.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:59 smithi187 ceph-mon[33032]: pgmap v551: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait+forced_backfill, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped+forced_backfill, 2 active+undersized+degraded+remapped+forced_backfill+backfill_toofull, 1 active+remapped+backfill_toofull, 139 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 838 B/s rd, 0 B/s wr, 683 op/s; 45075/316725 objects degraded (14.232%); 5813/316725 objects misplaced (1.835%); 4.4 MiB/s, 69 objects/s recovering 2024-09-21T13:59:59.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 13:59:59 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:59:59.219 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:59.252 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:59.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:59.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:59.348 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:59 smithi049 ceph-mon[40489]: pgmap v551: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait+forced_backfill, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped+forced_backfill, 2 active+undersized+degraded+remapped+forced_backfill+backfill_toofull, 1 active+remapped+backfill_toofull, 139 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 838 B/s rd, 0 B/s wr, 683 op/s; 45075/316725 objects degraded (14.232%); 5813/316725 objects misplaced (1.835%); 4.4 MiB/s, 69 objects/s recovering 2024-09-21T13:59:59.348 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 13:59:59 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:59:59.348 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:59 smithi049 ceph-mon[30080]: pgmap v551: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait+forced_backfill, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped+forced_backfill, 2 active+undersized+degraded+remapped+forced_backfill+backfill_toofull, 1 active+remapped+backfill_toofull, 139 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 838 B/s rd, 0 B/s wr, 683 op/s; 45075/316725 objects degraded (14.232%); 5813/316725 objects misplaced (1.835%); 4.4 MiB/s, 69 objects/s recovering 2024-09-21T13:59:59.348 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 13:59:59 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T13:59:59.354 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T13:59:59.389 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:59.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:59.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T13:59:59.490 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:59:59.521 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T13:59:59.521 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T13:59:59.521 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T13:59:59.521 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T13:59:59.522 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T13:59:59.522 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T13:59:59.522 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T13:59:59.522 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T13:59:59.522 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T13:59:59.522 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T13:59:59.522 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T13:59:59.522 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T13:59:59.523 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T13:59:59.523 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T13:59:59.530 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:59.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:59.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:59:59.552 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T13:59:59.552 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T13:59:59.632 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T13:59:59.668 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:59.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:59.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T13:59:59.770 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T13:59:59.800 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T13:59:59.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:59.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T13:59:59.894 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 13:59:59 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T13:59:59.901 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T13:59:59.956 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T13:59:59.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T13:59:59.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T13:59:59.959 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T14:00:00.050 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T14:00:00.057 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:00:00.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:00.112 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:00.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:00:00.151 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":551,"stamp":"2024-09-21T13:59:57.930538+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6906452531,"num_objects":105575,"num_object_clones":0,"num_object_copies":316725,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":45075,"num_objects_misplaced":5813,"num_objects_unfound":0,"num_objects_dirty":105575,"num_whiteouts":0,"num_read":1264,"num_read_kb":1123,"num_write":138271,"num_write_kb":7771594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8779,"num_bytes_recovered":563939776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82232,"ondisk_log_size":82232,"up":444,"acting":436,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":445,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":21138360,"kb_used_data":18184488,"kb_used_omap":20,"kb_used_meta":2953643,"kb_avail":728822856,"statfs":{"total":767960285184,"available":746314604544,"internally_reserved":0,"allocated":18620915712,"data_stored":18616397758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21298,"internal_metadata":3024530638},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,5,8,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-272957440,"num_objects":-4165,"num_object_clones":0,"num_object_copies":-12495,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1481,"num_objects_misplaced":-250,"num_objects_unfound":0,"num_objects_dirty":-4165,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4165,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":425,"num_bytes_recovered":27852800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.103722"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":463,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.838782+0000","last_change":"2024-09-21T13:54:55.873382+0000","last_active":"2024-09-21T13:59:51.838782+0000","last_peered":"2024-09-21T13:59:51.838782+0000","last_clean":"2024-09-21T13:59:51.838782+0000","last_became_active":"2024-09-21T13:52:04.317322+0000","last_became_peered":"2024-09-21T13:52:04.317322+0000","last_unstale":"2024-09-21T13:59:51.838782+0000","last_undegraded":"2024-09-21T13:59:51.838782+0000","last_fullsized":"2024-09-21T13:59:51.838782+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:55.873301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:55.873301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:02.211126+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"290'15","reported_seq":592,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.843400+0000","last_change":"2024-09-21T13:58:57.766087+0000","last_active":"2024-09-21T13:59:51.843400+0000","last_peered":"2024-09-21T13:59:51.843400+0000","last_clean":"2024-09-21T13:59:51.843400+0000","last_became_active":"2024-09-21T13:58:17.771220+0000","last_became_peered":"2024-09-21T13:58:17.771220+0000","last_unstale":"2024-09-21T13:59:51.843400+0000","last_undegraded":"2024-09-21T13:59:51.843400+0000","last_fullsized":"2024-09-21T13:59:51.843400+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:58:57.766011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:57.766011+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:22.570440+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":522,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.831278+0000","last_change":"2024-09-21T13:59:32.201875+0000","last_active":"2024-09-21T13:59:51.831278+0000","last_peered":"2024-09-21T13:59:51.831278+0000","last_clean":"2024-09-21T13:59:51.831278+0000","last_became_active":"2024-09-21T13:59:32.201538+0000","last_became_peered":"2024-09-21T13:59:32.201538+0000","last_unstale":"2024-09-21T13:59:51.831278+0000","last_undegraded":"2024-09-21T13:59:51.831278+0000","last_fullsized":"2024-09-21T13:59:51.831278+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:40.452724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:40.452724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":506,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.843734+0000","last_change":"2024-09-21T13:57:00.320623+0000","last_active":"2024-09-21T13:59:51.843734+0000","last_peered":"2024-09-21T13:59:51.843734+0000","last_clean":"2024-09-21T13:59:51.843734+0000","last_became_active":"2024-09-21T13:49:04.072460+0000","last_became_peered":"2024-09-21T13:49:04.072460+0000","last_unstale":"2024-09-21T13:59:51.843734+0000","last_undegraded":"2024-09-21T13:59:51.843734+0000","last_fullsized":"2024-09-21T13:59:51.843734+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:00.320510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:00.320510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:49.358812+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":489,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-09-21T13:59:55.056580+0000","last_change":"2024-09-21T13:59:32.208701+0000","last_active":"2024-09-21T13:59:55.056580+0000","last_peered":"2024-09-21T13:59:55.056580+0000","last_clean":"2024-09-21T13:59:55.056580+0000","last_became_active":"2024-09-21T13:59:32.203340+0000","last_became_peered":"2024-09-21T13:59:32.203340+0000","last_unstale":"2024-09-21T13:59:55.056580+0000","last_undegraded":"2024-09-21T13:59:55.056580+0000","last_fullsized":"2024-09-21T13:59:55.056580+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:34.696780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:34.696780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":501,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-09-21T13:59:55.056765+0000","last_change":"2024-09-21T13:58:17.778322+0000","last_active":"2024-09-21T13:59:55.056765+0000","last_peered":"2024-09-21T13:59:55.056765+0000","last_clean":"2024-09-21T13:59:55.056765+0000","last_became_active":"2024-09-21T13:58:17.777893+0000","last_became_peered":"2024-09-21T13:58:17.777893+0000","last_unstale":"2024-09-21T13:59:55.056765+0000","last_undegraded":"2024-09-21T13:59:55.056765+0000","last_fullsized":"2024-09-21T13:59:55.056765+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:07.001135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:55:07.001135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:18.028308+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"290'27","reported_seq":475,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-09-21T13:59:55.056082+0000","last_change":"2024-09-21T13:58:11.718421+0000","last_active":"2024-09-21T13:59:55.056082+0000","last_peered":"2024-09-21T13:59:55.056082+0000","last_clean":"2024-09-21T13:59:55.056082+0000","last_became_active":"2024-09-21T13:57:12.703469+0000","last_became_peered":"2024-09-21T13:57:12.703469+0000","last_unstale":"2024-09-21T13:59:55.056082+0000","last_undegraded":"2024-09-21T13:59:55.056082+0000","last_fullsized":"2024-09-21T13:59:55.056082+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:58:11.718334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:11.718334+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":532,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.843316+0000","last_change":"2024-09-21T13:58:53.761124+0000","last_active":"2024-09-21T13:59:51.843316+0000","last_peered":"2024-09-21T13:59:51.843316+0000","last_clean":"2024-09-21T13:59:51.843316+0000","last_became_active":"2024-09-21T13:58:17.773464+0000","last_became_peered":"2024-09-21T13:58:17.773464+0000","last_unstale":"2024-09-21T13:59:51.843316+0000","last_undegraded":"2024-09-21T13:59:51.843316+0000","last_fullsized":"2024-09-21T13:59:51.843316+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:53.761061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:53.761061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:05.719816+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":549,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.838224+0000","last_change":"2024-09-21T13:58:17.782330+0000","last_active":"2024-09-21T13:59:51.838224+0000","last_peered":"2024-09-21T13:59:51.838224+0000","last_clean":"2024-09-21T13:59:51.838224+0000","last_became_active":"2024-09-21T13:58:17.781904+0000","last_became_peered":"2024-09-21T13:58:17.781904+0000","last_unstale":"2024-09-21T13:59:51.838224+0000","last_undegraded":"2024-09-21T13:59:51.838224+0000","last_fullsized":"2024-09-21T13:59:51.838224+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:31.127926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:31.127926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"290'15","reported_seq":558,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.844097+0000","last_change":"2024-09-21T13:57:38.516376+0000","last_active":"2024-09-21T13:59:51.844097+0000","last_peered":"2024-09-21T13:59:51.844097+0000","last_clean":"2024-09-21T13:59:51.844097+0000","last_became_active":"2024-09-21T13:57:12.681408+0000","last_became_peered":"2024-09-21T13:57:12.681408+0000","last_unstale":"2024-09-21T13:59:51.844097+0000","last_undegraded":"2024-09-21T13:59:51.844097+0000","last_fullsized":"2024-09-21T13:59:51.844097+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:57:38.516309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:57:38.516309+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":514,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.835818+0000","last_change":"2024-09-21T13:59:32.213187+0000","last_active":"2024-09-21T13:59:51.835818+0000","last_peered":"2024-09-21T13:59:51.835818+0000","last_clean":"2024-09-21T13:59:51.835818+0000","last_became_active":"2024-09-21T13:59:32.212831+0000","last_became_peered":"2024-09-21T13:59:32.212831+0000","last_unstale":"2024-09-21T13:59:51.835818+0000","last_undegraded":"2024-09-21T13:59:51.835818+0000","last_fullsized":"2024-09-21T13:59:51.835818+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:21.284048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:56:21.284048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":470,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.833711+0000","last_change":"2024-09-21T13:59:32.202128+0000","last_active":"2024-09-21T13:59:51.833711+0000","last_peered":"2024-09-21T13:59:51.833711+0000","last_clean":"2024-09-21T13:59:51.833711+0000","last_became_active":"2024-09-21T13:59:32.201158+0000","last_became_peered":"2024-09-21T13:59:32.201158+0000","last_unstale":"2024-09-21T13:59:51.833711+0000","last_undegraded":"2024-09-21T13:59:51.833711+0000","last_fullsized":"2024-09-21T13:59:51.833711+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:25.792985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:54:25.792985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":519,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.836029+0000","last_change":"2024-09-21T13:56:54.328338+0000","last_active":"2024-09-21T13:59:51.836029+0000","last_peered":"2024-09-21T13:59:51.836029+0000","last_clean":"2024-09-21T13:59:51.836029+0000","last_became_active":"2024-09-21T13:48:58.050906+0000","last_became_peered":"2024-09-21T13:48:58.050906+0000","last_unstale":"2024-09-21T13:59:51.836029+0000","last_undegraded":"2024-09-21T13:59:51.836029+0000","last_fullsized":"2024-09-21T13:59:51.836029+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:54.328275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:54.328275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:07.169834+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"290'7","reported_seq":575,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.843874+0000","last_change":"2024-09-21T13:59:09.074586+0000","last_active":"2024-09-21T13:59:51.843874+0000","last_peered":"2024-09-21T13:59:51.843874+0000","last_clean":"2024-09-21T13:59:51.843874+0000","last_became_active":"2024-09-21T13:59:09.063053+0000","last_became_peered":"2024-09-21T13:59:09.063053+0000","last_unstale":"2024-09-21T13:59:51.843874+0000","last_undegraded":"2024-09-21T13:59:51.843874+0000","last_fullsized":"2024-09-21T13:59:51.843874+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-21T13:57:29.549403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.549403+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":464,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-09-21T13:59:55.059586+0000","last_change":"2024-09-21T13:59:32.204565+0000","last_active":"2024-09-21T13:59:55.059586+0000","last_peered":"2024-09-21T13:59:55.059586+0000","last_clean":"2024-09-21T13:59:55.059586+0000","last_became_active":"2024-09-21T13:59:32.204174+0000","last_became_peered":"2024-09-21T13:59:32.204174+0000","last_unstale":"2024-09-21T13:59:55.059586+0000","last_undegraded":"2024-09-21T13:59:55.059586+0000","last_fullsized":"2024-09-21T13:59:55.059586+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:06.722293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:06.722293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":534,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.836949+0000","last_change":"2024-09-21T13:58:44.376278+0000","last_active":"2024-09-21T13:59:51.836949+0000","last_peered":"2024-09-21T13:59:51.836949+0000","last_clean":"2024-09-21T13:59:51.836949+0000","last_became_active":"2024-09-21T13:57:12.684385+0000","last_became_peered":"2024-09-21T13:57:12.684385+0000","last_unstale":"2024-09-21T13:59:51.836949+0000","last_undegraded":"2024-09-21T13:59:51.836949+0000","last_fullsized":"2024-09-21T13:59:51.836949+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:44.376215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:44.376215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":532,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.844698+0000","last_change":"2024-09-21T13:58:17.772250+0000","last_active":"2024-09-21T13:59:51.844698+0000","last_peered":"2024-09-21T13:59:51.844698+0000","last_clean":"2024-09-21T13:59:51.844698+0000","last_became_active":"2024-09-21T13:58:17.770940+0000","last_became_peered":"2024-09-21T13:58:17.770940+0000","last_unstale":"2024-09-21T13:59:51.844698+0000","last_undegraded":"2024-09-21T13:59:51.844698+0000","last_fullsized":"2024-09-21T13:59:51.844698+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:21.448653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:21.448653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":469,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.844088+0000","last_change":"2024-09-21T13:59:32.205037+0000","last_active":"2024-09-21T13:59:51.844088+0000","last_peered":"2024-09-21T13:59:51.844088+0000","last_clean":"2024-09-21T13:59:51.844088+0000","last_became_active":"2024-09-21T13:59:32.204749+0000","last_became_peered":"2024-09-21T13:59:32.204749+0000","last_unstale":"2024-09-21T13:59:51.844088+0000","last_undegraded":"2024-09-21T13:59:51.844088+0000","last_fullsized":"2024-09-21T13:59:51.844088+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:17.361699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:17.361699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"290'27","reported_seq":522,"reported_epoch":307,"state":"active+clean","last_fresh":"2024-09-21T13:59:56.045126+0000","last_change":"2024-09-21T13:58:47.755659+0000","last_active":"2024-09-21T13:59:56.045126+0000","last_peered":"2024-09-21T13:59:56.045126+0000","last_clean":"2024-09-21T13:59:56.045126+0000","last_became_active":"2024-09-21T13:57:12.680783+0000","last_became_peered":"2024-09-21T13:57:12.680783+0000","last_unstale":"2024-09-21T13:59:56.045126+0000","last_undegraded":"2024-09-21T13:59:56.045126+0000","last_fullsized":"2024-09-21T13:59:56.045126+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:58:47.755561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:47.755561+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:01.959306+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":519,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.836859+0000","last_change":"2024-09-21T13:55:31.988001+0000","last_active":"2024-09-21T13:59:51.836859+0000","last_peered":"2024-09-21T13:59:51.836859+0000","last_clean":"2024-09-21T13:59:51.836859+0000","last_became_active":"2024-09-21T13:49:02.066796+0000","last_became_peered":"2024-09-21T13:49:02.066796+0000","last_unstale":"2024-09-21T13:59:51.836859+0000","last_undegraded":"2024-09-21T13:59:51.836859+0000","last_fullsized":"2024-09-21T13:59:51.836859+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:31.987854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:55:31.987854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:08.223013+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":470,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-09-21T13:59:55.056561+0000","last_change":"2024-09-21T13:58:17.779422+0000","last_active":"2024-09-21T13:59:55.056561+0000","last_peered":"2024-09-21T13:59:55.056561+0000","last_clean":"2024-09-21T13:59:55.056561+0000","last_became_active":"2024-09-21T13:58:17.779117+0000","last_became_peered":"2024-09-21T13:58:17.779117+0000","last_unstale":"2024-09-21T13:59:55.056561+0000","last_undegraded":"2024-09-21T13:59:55.056561+0000","last_fullsized":"2024-09-21T13:59:55.056561+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:57.962266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:57.962266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:59:59.730442+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":529,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.844239+0000","last_change":"2024-09-21T13:56:58.309928+0000","last_active":"2024-09-21T13:59:51.844239+0000","last_peered":"2024-09-21T13:59:51.844239+0000","last_clean":"2024-09-21T13:59:51.844239+0000","last_became_active":"2024-09-21T13:52:04.316016+0000","last_became_peered":"2024-09-21T13:52:04.316016+0000","last_unstale":"2024-09-21T13:59:51.844239+0000","last_undegraded":"2024-09-21T13:59:51.844239+0000","last_fullsized":"2024-09-21T13:59:51.844239+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:58.309863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:58.309863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:47.291059+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"290'18","reported_seq":487,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-09-21T13:59:55.056543+0000","last_change":"2024-09-21T13:59:23.041396+0000","last_active":"2024-09-21T13:59:55.056543+0000","last_peered":"2024-09-21T13:59:55.056543+0000","last_clean":"2024-09-21T13:59:55.056543+0000","last_became_active":"2024-09-21T13:57:12.684454+0000","last_became_peered":"2024-09-21T13:57:12.684454+0000","last_unstale":"2024-09-21T13:59:55.056543+0000","last_undegraded":"2024-09-21T13:59:55.056543+0000","last_fullsized":"2024-09-21T13:59:55.056543+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'18","last_scrub_stamp":"2024-09-21T13:59:23.041311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:23.041311+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:25.910995+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":524,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.848560+0000","last_change":"2024-09-21T13:58:16.393763+0000","last_active":"2024-09-21T13:59:51.848560+0000","last_peered":"2024-09-21T13:59:51.848560+0000","last_clean":"2024-09-21T13:59:51.848560+0000","last_became_active":"2024-09-21T13:57:12.681897+0000","last_became_peered":"2024-09-21T13:57:12.681897+0000","last_unstale":"2024-09-21T13:59:51.848560+0000","last_undegraded":"2024-09-21T13:59:51.848560+0000","last_fullsized":"2024-09-21T13:59:51.848560+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:16.393696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:16.393696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":437,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-09-21T13:59:55.056074+0000","last_change":"2024-09-21T13:59:32.216521+0000","last_active":"2024-09-21T13:59:55.056074+0000","last_peered":"2024-09-21T13:59:55.056074+0000","last_clean":"2024-09-21T13:59:55.056074+0000","last_became_active":"2024-09-21T13:59:32.215222+0000","last_became_peered":"2024-09-21T13:59:32.215222+0000","last_unstale":"2024-09-21T13:59:55.056074+0000","last_undegraded":"2024-09-21T13:59:55.056074+0000","last_fullsized":"2024-09-21T13:59:55.056074+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T13:57:26.536684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:26.536684+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":539,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.831473+0000","last_change":"2024-09-21T13:57:27.305798+0000","last_active":"2024-09-21T13:59:51.831473+0000","last_peered":"2024-09-21T13:59:51.831473+0000","last_clean":"2024-09-21T13:59:51.831473+0000","last_became_active":"2024-09-21T13:57:12.685742+0000","last_became_peered":"2024-09-21T13:57:12.685742+0000","last_unstale":"2024-09-21T13:59:51.831473+0000","last_undegraded":"2024-09-21T13:59:51.831473+0000","last_fullsized":"2024-09-21T13:59:51.831473+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:57:27.305724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:27.305724+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:21.365203+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"290'20","reported_seq":428,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-09-21T13:59:55.057223+0000","last_change":"2024-09-21T13:59:32.242771+0000","last_active":"2024-09-21T13:59:55.057223+0000","last_peered":"2024-09-21T13:59:55.057223+0000","last_clean":"2024-09-21T13:59:55.057223+0000","last_became_active":"2024-09-21T13:59:32.213865+0000","last_became_peered":"2024-09-21T13:59:32.213865+0000","last_unstale":"2024-09-21T13:59:55.057223+0000","last_undegraded":"2024-09-21T13:59:55.057223+0000","last_fullsized":"2024-09-21T13:59:55.057223+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:57:27.578738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:57:27.578738+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"300'12","reported_seq":127,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.828271+0000","last_change":"2024-09-21T13:59:36.012775+0000","last_active":"2024-09-21T13:59:51.828271+0000","last_peered":"2024-09-21T13:59:51.828271+0000","last_clean":"2024-09-21T13:59:51.828271+0000","last_became_active":"2024-09-21T13:59:32.205051+0000","last_became_peered":"2024-09-21T13:59:32.205051+0000","last_unstale":"2024-09-21T13:59:51.828271+0000","last_undegraded":"2024-09-21T13:59:51.828271+0000","last_fullsized":"2024-09-21T13:59:51.828271+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'12","last_scrub_stamp":"2024-09-21T13:59:36.012693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:36.012693+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:01:05.127536+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2","version":"306'8694","reported_seq":8797,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-09-21T13:59:55.898788+0000","last_change":"2024-09-21T13:57:53.208687+0000","last_active":"2024-09-21T13:59:55.898788+0000","last_peered":"2024-09-21T13:59:55.898788+0000","last_clean":"2024-09-21T13:59:55.898788+0000","last_became_active":"2024-09-21T13:57:53.208352+0000","last_became_peered":"2024-09-21T13:57:53.208352+0000","last_unstale":"2024-09-21T13:59:55.898788+0000","last_undegraded":"2024-09-21T13:59:55.898788+0000","last_fullsized":"2024-09-21T13:59:55.898788+0000","mapping_epoch":260,"log_start":"284'4000","ondisk_log_start":"284'4000","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4694,"log_dups_size":0,"ondisk_log_size":4694,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":427425792,"num_objects":6522,"num_object_clones":0,"num_object_copies":19566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8694,"num_write_kb":486912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"290'27","reported_seq":585,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.834194+0000","last_change":"2024-09-21T13:58:45.984691+0000","last_active":"2024-09-21T13:59:51.834194+0000","last_peered":"2024-09-21T13:59:51.834194+0000","last_clean":"2024-09-21T13:59:51.834194+0000","last_became_active":"2024-09-21T13:57:12.695609+0000","last_became_peered":"2024-09-21T13:57:12.695609+0000","last_unstale":"2024-09-21T13:59:51.834194+0000","last_undegraded":"2024-09-21T13:59:51.834194+0000","last_fullsized":"2024-09-21T13:59:51.834194+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:58:45.984575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:45.984575+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:07.027041+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":473,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-09-21T13:59:55.049875+0000","last_change":"2024-09-21T13:59:32.205723+0000","last_active":"2024-09-21T13:59:55.049875+0000","last_peered":"2024-09-21T13:59:55.049875+0000","last_clean":"2024-09-21T13:59:55.049875+0000","last_became_active":"2024-09-21T13:59:32.205350+0000","last_became_peered":"2024-09-21T13:59:32.205350+0000","last_unstale":"2024-09-21T13:59:55.049875+0000","last_undegraded":"2024-09-21T13:59:55.049875+0000","last_fullsized":"2024-09-21T13:59:55.049875+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:32.668851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:32.668851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"68'8","reported_seq":653,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.844818+0000","last_change":"2024-09-21T13:56:59.335345+0000","last_active":"2024-09-21T13:59:51.844818+0000","last_peered":"2024-09-21T13:59:51.844818+0000","last_clean":"2024-09-21T13:59:51.844818+0000","last_became_active":"2024-09-21T13:49:02.061679+0000","last_became_peered":"2024-09-21T13:49:02.061679+0000","last_unstale":"2024-09-21T13:59:51.844818+0000","last_undegraded":"2024-09-21T13:59:51.844818+0000","last_fullsized":"2024-09-21T13:59:51.844818+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:56:59.335171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:56:59.335171+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":507,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.837765+0000","last_change":"2024-09-21T13:58:52.244527+0000","last_active":"2024-09-21T13:59:51.837765+0000","last_peered":"2024-09-21T13:59:51.837765+0000","last_clean":"2024-09-21T13:59:51.837765+0000","last_became_active":"2024-09-21T13:49:04.081905+0000","last_became_peered":"2024-09-21T13:49:04.081905+0000","last_unstale":"2024-09-21T13:59:51.837765+0000","last_undegraded":"2024-09-21T13:59:51.837765+0000","last_fullsized":"2024-09-21T13:59:51.837765+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:52.244483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:52.244483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:19.563849+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"305'8585","reported_seq":8701,"reported_epoch":305,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2024-09-21T13:59:54.507250+0000","last_change":"2024-09-21T13:59:33.321465+0000","last_active":"2024-09-21T13:59:54.507250+0000","last_peered":"2024-09-21T13:59:54.507250+0000","last_clean":"2024-09-21T13:58:23.757609+0000","last_became_active":"2024-09-21T13:59:33.315668+0000","last_became_peered":"2024-09-21T13:59:33.315668+0000","last_unstale":"2024-09-21T13:59:54.507250+0000","last_undegraded":"2024-09-21T13:59:33.208390+0000","last_fullsized":"2024-09-21T13:59:33.189502+0000","mapping_epoch":299,"log_start":"284'3965","ondisk_log_start":"284'3965","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4620,"log_dups_size":0,"ondisk_log_size":4620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":434110488,"num_objects":6625,"num_object_clones":0,"num_object_copies":19875,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6625,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6625,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8585,"num_write_kb":486657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":6625}],"blocked_by":[],"up_primary":7,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"290'30","reported_seq":497,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-09-21T13:59:55.052723+0000","last_change":"2024-09-21T13:59:32.246090+0000","last_active":"2024-09-21T13:59:55.052723+0000","last_peered":"2024-09-21T13:59:55.052723+0000","last_clean":"2024-09-21T13:59:55.052723+0000","last_became_active":"2024-09-21T13:59:32.205887+0000","last_became_peered":"2024-09-21T13:59:32.205887+0000","last_unstale":"2024-09-21T13:59:55.052723+0000","last_undegraded":"2024-09-21T13:59:55.052723+0000","last_fullsized":"2024-09-21T13:59:55.052723+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'18","last_scrub_stamp":"2024-09-21T13:54:19.001030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:19.001030+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":425,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-09-21T13:59:55.056894+0000","last_change":"2024-09-21T13:59:32.212559+0000","last_active":"2024-09-21T13:59:55.056894+0000","last_peered":"2024-09-21T13:59:55.056894+0000","last_clean":"2024-09-21T13:59:55.056894+0000","last_became_active":"2024-09-21T13:59:32.211004+0000","last_became_peered":"2024-09-21T13:59:32.211004+0000","last_unstale":"2024-09-21T13:59:55.056894+0000","last_undegraded":"2024-09-21T13:59:55.056894+0000","last_fullsized":"2024-09-21T13:59:55.056894+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:25.736461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:25.736461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":541,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.843639+0000","last_change":"2024-09-21T13:59:34.293330+0000","last_active":"2024-09-21T13:59:51.843639+0000","last_peered":"2024-09-21T13:59:51.843639+0000","last_clean":"2024-09-21T13:59:51.843639+0000","last_became_active":"2024-09-21T13:59:32.220304+0000","last_became_peered":"2024-09-21T13:59:32.220304+0000","last_unstale":"2024-09-21T13:59:51.843639+0000","last_undegraded":"2024-09-21T13:59:51.843639+0000","last_fullsized":"2024-09-21T13:59:51.843639+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:34.292892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:34.292892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:46.211883+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":521,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.844012+0000","last_change":"2024-09-21T13:59:23.334230+0000","last_active":"2024-09-21T13:59:51.844012+0000","last_peered":"2024-09-21T13:59:51.844012+0000","last_clean":"2024-09-21T13:59:51.844012+0000","last_became_active":"2024-09-21T13:57:12.680372+0000","last_became_peered":"2024-09-21T13:57:12.680372+0000","last_unstale":"2024-09-21T13:59:51.844012+0000","last_undegraded":"2024-09-21T13:59:51.844012+0000","last_fullsized":"2024-09-21T13:59:51.844012+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:23.334162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:23.334162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:46.281593+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"306'8566","reported_seq":8731,"reported_epoch":306,"state":"active+undersized+degraded+remapped+backfill_wait+forced_backfill","last_fresh":"2024-09-21T13:59:55.839299+0000","last_change":"2024-09-21T13:59:48.318724+0000","last_active":"2024-09-21T13:59:55.839299+0000","last_peered":"2024-09-21T13:59:55.839299+0000","last_clean":"2024-09-21T13:58:23.796606+0000","last_became_active":"2024-09-21T13:59:33.303769+0000","last_became_peered":"2024-09-21T13:59:33.303769+0000","last_unstale":"2024-09-21T13:59:55.839299+0000","last_undegraded":"2024-09-21T13:59:33.205632+0000","last_fullsized":"2024-09-21T13:59:33.192057+0000","mapping_epoch":299,"log_start":"283'3800","ondisk_log_start":"283'3800","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4766,"log_dups_size":0,"ondisk_log_size":4766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":424935424,"num_objects":6484,"num_object_clones":0,"num_object_copies":19452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8566,"num_write_kb":481600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1703936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":6484}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"290'28","reported_seq":603,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.843726+0000","last_change":"2024-09-21T13:58:17.810036+0000","last_active":"2024-09-21T13:59:51.843726+0000","last_peered":"2024-09-21T13:59:51.843726+0000","last_clean":"2024-09-21T13:59:51.843726+0000","last_became_active":"2024-09-21T13:58:17.779342+0000","last_became_peered":"2024-09-21T13:58:17.779342+0000","last_unstale":"2024-09-21T13:59:51.843726+0000","last_undegraded":"2024-09-21T13:59:51.843726+0000","last_fullsized":"2024-09-21T13:59:51.843726+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'14","last_scrub_stamp":"2024-09-21T13:56:32.438459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:56:32.438459+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:45.254297+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":529,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.834888+0000","last_change":"2024-09-21T13:57:38.046718+0000","last_active":"2024-09-21T13:59:51.834888+0000","last_peered":"2024-09-21T13:59:51.834888+0000","last_clean":"2024-09-21T13:59:51.834888+0000","last_became_active":"2024-09-21T13:48:58.053533+0000","last_became_peered":"2024-09-21T13:48:58.053533+0000","last_unstale":"2024-09-21T13:59:51.834888+0000","last_undegraded":"2024-09-21T13:59:51.834888+0000","last_fullsized":"2024-09-21T13:59:51.834888+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:57:38.046627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:38.046627+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:10.473009+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"241'10","reported_seq":538,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-09-21T13:59:55.052906+0000","last_change":"2024-09-21T13:58:06.789629+0000","last_active":"2024-09-21T13:59:55.052906+0000","last_peered":"2024-09-21T13:59:55.052906+0000","last_clean":"2024-09-21T13:59:55.052906+0000","last_became_active":"2024-09-21T13:57:12.686651+0000","last_became_peered":"2024-09-21T13:57:12.686651+0000","last_unstale":"2024-09-21T13:59:55.052906+0000","last_undegraded":"2024-09-21T13:59:55.052906+0000","last_fullsized":"2024-09-21T13:59:55.052906+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'10","last_scrub_stamp":"2024-09-21T13:58:06.789560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:06.789560+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":489,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-09-21T13:59:55.056108+0000","last_change":"2024-09-21T13:59:32.210257+0000","last_active":"2024-09-21T13:59:55.056108+0000","last_peered":"2024-09-21T13:59:55.056108+0000","last_clean":"2024-09-21T13:59:55.056108+0000","last_became_active":"2024-09-21T13:59:32.209615+0000","last_became_peered":"2024-09-21T13:59:32.209615+0000","last_unstale":"2024-09-21T13:59:55.056108+0000","last_undegraded":"2024-09-21T13:59:55.056108+0000","last_fullsized":"2024-09-21T13:59:55.056108+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:37.548064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:37.548064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"305'8663","reported_seq":8763,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:53.731158+0000","last_change":"2024-09-21T13:57:53.213439+0000","last_active":"2024-09-21T13:59:53.731158+0000","last_peered":"2024-09-21T13:59:53.731158+0000","last_clean":"2024-09-21T13:59:53.731158+0000","last_became_active":"2024-09-21T13:57:53.213139+0000","last_became_peered":"2024-09-21T13:57:53.213139+0000","last_unstale":"2024-09-21T13:59:53.731158+0000","last_undegraded":"2024-09-21T13:59:53.731158+0000","last_fullsized":"2024-09-21T13:59:53.731158+0000","mapping_epoch":260,"log_start":"281'3300","ondisk_log_start":"281'3300","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":5363,"log_dups_size":0,"ondisk_log_size":5363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":437452800,"num_objects":6675,"num_object_clones":0,"num_object_copies":20025,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8663,"num_write_kb":490816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"290'20","reported_seq":554,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.835887+0000","last_change":"2024-09-21T13:54:19.593696+0000","last_active":"2024-09-21T13:59:51.835887+0000","last_peered":"2024-09-21T13:59:51.835887+0000","last_clean":"2024-09-21T13:59:51.835887+0000","last_became_active":"2024-09-21T13:49:00.054564+0000","last_became_peered":"2024-09-21T13:49:00.054564+0000","last_unstale":"2024-09-21T13:59:51.835887+0000","last_undegraded":"2024-09-21T13:59:51.835887+0000","last_fullsized":"2024-09-21T13:59:51.835887+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:54:19.593611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:19.593611+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:04.212544+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"61'1","reported_seq":572,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.838305+0000","last_change":"2024-09-21T13:58:48.407813+0000","last_active":"2024-09-21T13:59:51.838305+0000","last_peered":"2024-09-21T13:59:51.838305+0000","last_clean":"2024-09-21T13:59:51.838305+0000","last_became_active":"2024-09-21T13:57:12.684939+0000","last_became_peered":"2024-09-21T13:57:12.684939+0000","last_unstale":"2024-09-21T13:59:51.838305+0000","last_undegraded":"2024-09-21T13:59:51.838305+0000","last_fullsized":"2024-09-21T13:59:51.838305+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:58:48.407734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:48.407734+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:59:54.869007+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"270'13","reported_seq":106,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.828390+0000","last_change":"2024-09-21T13:58:17.788471+0000","last_active":"2024-09-21T13:59:51.828390+0000","last_peered":"2024-09-21T13:59:51.828390+0000","last_clean":"2024-09-21T13:59:51.828390+0000","last_became_active":"2024-09-21T13:58:17.777857+0000","last_became_peered":"2024-09-21T13:58:17.777857+0000","last_unstale":"2024-09-21T13:59:51.828390+0000","last_undegraded":"2024-09-21T13:59:51.828390+0000","last_fullsized":"2024-09-21T13:59:51.828390+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'12","last_scrub_stamp":"2024-09-21T13:57:30.133392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:30.133392+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:18.984922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":343,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-09-21T13:59:55.052822+0000","last_change":"2024-09-21T13:58:58.730439+0000","last_active":"2024-09-21T13:59:55.052822+0000","last_peered":"2024-09-21T13:59:55.052822+0000","last_clean":"2024-09-21T13:59:55.052822+0000","last_became_active":"2024-09-21T13:57:12.704897+0000","last_became_peered":"2024-09-21T13:57:12.704897+0000","last_unstale":"2024-09-21T13:59:55.052822+0000","last_undegraded":"2024-09-21T13:59:55.052822+0000","last_fullsized":"2024-09-21T13:59:55.052822+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:58.730368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:58.730368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:59:59.823526+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.e","version":"305'8459","reported_seq":8569,"reported_epoch":305,"state":"active+undersized+degraded+remapped+forced_backfill+backfill_toofull","last_fresh":"2024-09-21T13:59:54.502837+0000","last_change":"2024-09-21T13:59:48.318744+0000","last_active":"2024-09-21T13:59:54.502837+0000","last_peered":"2024-09-21T13:59:54.502837+0000","last_clean":"2024-09-21T13:58:23.771204+0000","last_became_active":"2024-09-21T13:59:33.338259+0000","last_became_peered":"2024-09-21T13:59:33.338259+0000","last_unstale":"2024-09-21T13:59:54.502837+0000","last_undegraded":"2024-09-21T13:59:33.212884+0000","last_fullsized":"2024-09-21T13:59:33.191296+0000","mapping_epoch":299,"log_start":"283'3783","ondisk_log_start":"283'3783","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4676,"log_dups_size":0,"ondisk_log_size":4676,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":423821312,"num_objects":6467,"num_object_clones":0,"num_object_copies":19401,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6467,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8459,"num_write_kb":477632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":6467}],"blocked_by":[],"up_primary":7,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"290'29","reported_seq":547,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.835888+0000","last_change":"2024-09-21T13:59:32.235547+0000","last_active":"2024-09-21T13:59:51.835888+0000","last_peered":"2024-09-21T13:59:51.835888+0000","last_clean":"2024-09-21T13:59:51.835888+0000","last_became_active":"2024-09-21T13:59:32.209177+0000","last_became_peered":"2024-09-21T13:59:32.209177+0000","last_unstale":"2024-09-21T13:59:51.835888+0000","last_undegraded":"2024-09-21T13:59:51.835888+0000","last_fullsized":"2024-09-21T13:59:51.835888+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-21T13:56:55.319759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:56:55.319759+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":531,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.836362+0000","last_change":"2024-09-21T13:58:16.713075+0000","last_active":"2024-09-21T13:59:51.836362+0000","last_peered":"2024-09-21T13:59:51.836362+0000","last_clean":"2024-09-21T13:59:51.836362+0000","last_became_active":"2024-09-21T13:52:04.321656+0000","last_became_peered":"2024-09-21T13:52:04.321656+0000","last_unstale":"2024-09-21T13:59:51.836362+0000","last_undegraded":"2024-09-21T13:59:51.836362+0000","last_fullsized":"2024-09-21T13:59:51.836362+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:16.712990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:16.712990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":90,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.830648+0000","last_change":"2024-09-21T13:59:23.032196+0000","last_active":"2024-09-21T13:59:51.830648+0000","last_peered":"2024-09-21T13:59:51.830648+0000","last_clean":"2024-09-21T13:59:51.830648+0000","last_became_active":"2024-09-21T13:58:17.778679+0000","last_became_peered":"2024-09-21T13:58:17.778679+0000","last_unstale":"2024-09-21T13:59:51.830648+0000","last_undegraded":"2024-09-21T13:59:51.830648+0000","last_fullsized":"2024-09-21T13:59:51.830648+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:23.032143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:23.032143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:26.703801+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":533,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.830863+0000","last_change":"2024-09-21T13:58:43.952014+0000","last_active":"2024-09-21T13:59:51.830863+0000","last_peered":"2024-09-21T13:59:51.830863+0000","last_clean":"2024-09-21T13:59:51.830863+0000","last_became_active":"2024-09-21T13:58:17.757628+0000","last_became_peered":"2024-09-21T13:58:17.757628+0000","last_unstale":"2024-09-21T13:59:51.830863+0000","last_undegraded":"2024-09-21T13:59:51.830863+0000","last_fullsized":"2024-09-21T13:59:51.830863+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:43.951941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:43.951941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:01.473279+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"305'8599","reported_seq":10181,"reported_epoch":305,"state":"active+recovering+undersized+degraded+remapped+forced_backfill","last_fresh":"2024-09-21T13:59:54.119092+0000","last_change":"2024-09-21T13:59:48.318713+0000","last_active":"2024-09-21T13:59:54.119092+0000","last_peered":"2024-09-21T13:59:54.119092+0000","last_clean":"2024-09-21T13:58:23.765960+0000","last_became_active":"2024-09-21T13:59:33.385921+0000","last_became_peered":"2024-09-21T13:59:33.385921+0000","last_unstale":"2024-09-21T13:59:54.119092+0000","last_undegraded":"2024-09-21T13:59:33.219145+0000","last_fullsized":"2024-09-21T13:59:33.193300+0000","mapping_epoch":299,"log_start":"276'2700","ondisk_log_start":"276'2700","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":5899,"log_dups_size":0,"ondisk_log_size":5899,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":437321728,"num_objects":6673,"num_object_clones":0,"num_object_copies":20019,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4521,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8599,"num_write_kb":488704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":738,"num_bytes_recovered":48365568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[{"shards":"0,5","objects":2152}],"blocked_by":[],"up_primary":7,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"290'44","reported_seq":740,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.836671+0000","last_change":"2024-09-21T13:58:56.076153+0000","last_active":"2024-09-21T13:59:51.836671+0000","last_peered":"2024-09-21T13:59:51.836671+0000","last_clean":"2024-09-21T13:59:51.836671+0000","last_became_active":"2024-09-21T13:57:12.684690+0000","last_became_peered":"2024-09-21T13:57:12.684690+0000","last_unstale":"2024-09-21T13:59:51.836671+0000","last_undegraded":"2024-09-21T13:59:51.836671+0000","last_fullsized":"2024-09-21T13:59:51.836671+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'32","last_scrub_stamp":"2024-09-21T13:58:56.076074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:56.076074+0000","objects_scrubbed":10,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:04.712559+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":533,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.844165+0000","last_change":"2024-09-21T13:59:27.369582+0000","last_active":"2024-09-21T13:59:51.844165+0000","last_peered":"2024-09-21T13:59:51.844165+0000","last_clean":"2024-09-21T13:59:51.844165+0000","last_became_active":"2024-09-21T13:58:17.770937+0000","last_became_peered":"2024-09-21T13:58:17.770937+0000","last_unstale":"2024-09-21T13:59:51.844165+0000","last_undegraded":"2024-09-21T13:59:51.844165+0000","last_fullsized":"2024-09-21T13:59:51.844165+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:27.369511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:59:27.369511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:40.229805+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":568,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.837442+0000","last_change":"2024-09-21T13:59:22.186466+0000","last_active":"2024-09-21T13:59:51.837442+0000","last_peered":"2024-09-21T13:59:51.837442+0000","last_clean":"2024-09-21T13:59:51.837442+0000","last_became_active":"2024-09-21T13:58:17.776563+0000","last_became_peered":"2024-09-21T13:58:17.776563+0000","last_unstale":"2024-09-21T13:59:51.837442+0000","last_undegraded":"2024-09-21T13:59:51.837442+0000","last_fullsized":"2024-09-21T13:59:51.837442+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-21T13:59:22.186393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:59:22.186393+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:28.219690+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0015946669999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":460,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.836632+0000","last_change":"2024-09-21T13:55:07.021128+0000","last_active":"2024-09-21T13:59:51.836632+0000","last_peered":"2024-09-21T13:59:51.836632+0000","last_clean":"2024-09-21T13:59:51.836632+0000","last_became_active":"2024-09-21T13:50:29.688249+0000","last_became_peered":"2024-09-21T13:50:29.688249+0000","last_unstale":"2024-09-21T13:59:51.836632+0000","last_undegraded":"2024-09-21T13:59:51.836632+0000","last_fullsized":"2024-09-21T13:59:51.836632+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:07.021059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:55:07.021059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:06.200040+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":452,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-09-21T13:59:55.056986+0000","last_change":"2024-09-21T13:59:32.209357+0000","last_active":"2024-09-21T13:59:55.056986+0000","last_peered":"2024-09-21T13:59:55.056986+0000","last_clean":"2024-09-21T13:59:55.056986+0000","last_became_active":"2024-09-21T13:59:32.208980+0000","last_became_peered":"2024-09-21T13:59:32.208980+0000","last_unstale":"2024-09-21T13:59:55.056986+0000","last_undegraded":"2024-09-21T13:59:55.056986+0000","last_fullsized":"2024-09-21T13:59:55.056986+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:16.797227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:16.797227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"305'8361","reported_seq":8468,"reported_epoch":305,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T13:59:53.719491+0000","last_change":"2024-09-21T13:59:33.463191+0000","last_active":"2024-09-21T13:59:53.719491+0000","last_peered":"2024-09-21T13:59:53.719491+0000","last_clean":"2024-09-21T13:58:23.784671+0000","last_became_active":"2024-09-21T13:59:33.447969+0000","last_became_peered":"2024-09-21T13:59:33.447969+0000","last_unstale":"2024-09-21T13:59:53.719491+0000","last_undegraded":"2024-09-21T13:59:33.234331+0000","last_fullsized":"2024-09-21T13:59:33.191526+0000","mapping_epoch":299,"log_start":"277'3000","ondisk_log_start":"277'3000","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":5361,"log_dups_size":0,"ondisk_log_size":5361,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":426049536,"num_objects":6501,"num_object_clones":0,"num_object_copies":19503,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5080,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8361,"num_write_kb":475584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":1421}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"290'17","reported_seq":557,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.835746+0000","last_change":"2024-09-21T13:58:11.619937+0000","last_active":"2024-09-21T13:59:51.835746+0000","last_peered":"2024-09-21T13:59:51.835746+0000","last_clean":"2024-09-21T13:59:51.835746+0000","last_became_active":"2024-09-21T13:49:00.053323+0000","last_became_peered":"2024-09-21T13:49:00.053323+0000","last_unstale":"2024-09-21T13:59:51.835746+0000","last_undegraded":"2024-09-21T13:59:51.835746+0000","last_fullsized":"2024-09-21T13:59:51.835746+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'13","last_scrub_stamp":"2024-09-21T13:58:11.619848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:11.619848+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":569,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.830963+0000","last_change":"2024-09-21T13:59:29.454265+0000","last_active":"2024-09-21T13:59:51.830963+0000","last_peered":"2024-09-21T13:59:51.830963+0000","last_clean":"2024-09-21T13:59:51.830963+0000","last_became_active":"2024-09-21T13:58:17.759671+0000","last_became_peered":"2024-09-21T13:58:17.759671+0000","last_unstale":"2024-09-21T13:59:51.830963+0000","last_undegraded":"2024-09-21T13:59:51.830963+0000","last_fullsized":"2024-09-21T13:59:51.830963+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:29.454085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:59:29.454085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:43.157879+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":462,"reported_epoch":307,"state":"active+clean","last_fresh":"2024-09-21T13:59:56.044522+0000","last_change":"2024-09-21T13:58:41.644876+0000","last_active":"2024-09-21T13:59:56.044522+0000","last_peered":"2024-09-21T13:59:56.044522+0000","last_clean":"2024-09-21T13:59:56.044522+0000","last_became_active":"2024-09-21T13:57:12.682147+0000","last_became_peered":"2024-09-21T13:57:12.682147+0000","last_unstale":"2024-09-21T13:59:56.044522+0000","last_undegraded":"2024-09-21T13:59:56.044522+0000","last_fullsized":"2024-09-21T13:59:56.044522+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:58:41.644789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:58:41.644789+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:45.660503+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":414,"reported_epoch":307,"state":"active+clean","last_fresh":"2024-09-21T13:59:56.047281+0000","last_change":"2024-09-21T13:59:32.208163+0000","last_active":"2024-09-21T13:59:56.047281+0000","last_peered":"2024-09-21T13:59:56.047281+0000","last_clean":"2024-09-21T13:59:56.047281+0000","last_became_active":"2024-09-21T13:59:32.207740+0000","last_became_peered":"2024-09-21T13:59:32.207740+0000","last_unstale":"2024-09-21T13:59:56.047281+0000","last_undegraded":"2024-09-21T13:59:56.047281+0000","last_fullsized":"2024-09-21T13:59:56.047281+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:10.320333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:10.320333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:44.635553+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":497,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-09-21T13:59:55.056159+0000","last_change":"2024-09-21T13:59:32.222791+0000","last_active":"2024-09-21T13:59:55.056159+0000","last_peered":"2024-09-21T13:59:55.056159+0000","last_clean":"2024-09-21T13:59:55.056159+0000","last_became_active":"2024-09-21T13:59:32.221502+0000","last_became_peered":"2024-09-21T13:59:32.221502+0000","last_unstale":"2024-09-21T13:59:55.056159+0000","last_undegraded":"2024-09-21T13:59:55.056159+0000","last_fullsized":"2024-09-21T13:59:55.056159+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:14.673582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:14.673582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"305'8487","reported_seq":12469,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:53.488135+0000","last_change":"2024-09-21T13:59:31.471106+0000","last_active":"2024-09-21T13:59:53.488135+0000","last_peered":"2024-09-21T13:59:53.488135+0000","last_clean":"2024-09-21T13:59:53.488135+0000","last_became_active":"2024-09-21T13:58:35.721126+0000","last_became_peered":"2024-09-21T13:58:35.721126+0000","last_unstale":"2024-09-21T13:59:53.488135+0000","last_undegraded":"2024-09-21T13:59:53.488135+0000","last_fullsized":"2024-09-21T13:59:53.488135+0000","mapping_epoch":278,"log_start":"271'1900","ondisk_log_start":"271'1900","created":260,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"298'7530","last_scrub_stamp":"2024-09-21T13:59:31.471024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:59:31.471024+0000","objects_scrubbed":7502,"log_size":6587,"log_dups_size":0,"ondisk_log_size":6587,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:33.949758+0000","scrub_duration":1396,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":438894592,"num_objects":6697,"num_object_clones":0,"num_object_copies":20091,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8487,"num_write_kb":485888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1892,"num_bytes_recovered":123994112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"290'24","reported_seq":608,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.834733+0000","last_change":"2024-09-21T13:59:28.458847+0000","last_active":"2024-09-21T13:59:51.834733+0000","last_peered":"2024-09-21T13:59:51.834733+0000","last_clean":"2024-09-21T13:59:51.834733+0000","last_became_active":"2024-09-21T13:52:04.318875+0000","last_became_peered":"2024-09-21T13:52:04.318875+0000","last_unstale":"2024-09-21T13:59:51.834733+0000","last_undegraded":"2024-09-21T13:59:51.834733+0000","last_fullsized":"2024-09-21T13:59:51.834733+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'24","last_scrub_stamp":"2024-09-21T13:59:28.458768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:28.458768+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:51.851382+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":462,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.843874+0000","last_change":"2024-09-21T13:57:20.447876+0000","last_active":"2024-09-21T13:59:51.843874+0000","last_peered":"2024-09-21T13:59:51.843874+0000","last_clean":"2024-09-21T13:59:51.843874+0000","last_became_active":"2024-09-21T13:57:12.681030+0000","last_became_peered":"2024-09-21T13:57:12.681030+0000","last_unstale":"2024-09-21T13:59:51.843874+0000","last_undegraded":"2024-09-21T13:59:51.843874+0000","last_fullsized":"2024-09-21T13:59:51.843874+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:20.447801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:20.447801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":1055,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-09-21T13:59:55.053500+0000","last_change":"2024-09-21T13:59:32.218796+0000","last_active":"2024-09-21T13:59:55.053500+0000","last_peered":"2024-09-21T13:59:55.053500+0000","last_clean":"2024-09-21T13:59:55.053500+0000","last_became_active":"2024-09-21T13:59:32.203361+0000","last_became_peered":"2024-09-21T13:59:32.203361+0000","last_unstale":"2024-09-21T13:59:55.053500+0000","last_undegraded":"2024-09-21T13:59:55.053500+0000","last_fullsized":"2024-09-21T13:59:55.053500+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-21T13:56:29.617143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:56:29.617143+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047048000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":512,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.838739+0000","last_change":"2024-09-21T13:56:33.316275+0000","last_active":"2024-09-21T13:59:51.838739+0000","last_peered":"2024-09-21T13:59:51.838739+0000","last_clean":"2024-09-21T13:59:51.838739+0000","last_became_active":"2024-09-21T13:49:02.066641+0000","last_became_peered":"2024-09-21T13:49:02.066641+0000","last_unstale":"2024-09-21T13:59:51.838739+0000","last_undegraded":"2024-09-21T13:59:51.838739+0000","last_fullsized":"2024-09-21T13:59:51.838739+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:33.316230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:56:33.316230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:08.146604+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":543,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.831828+0000","last_change":"2024-09-21T13:59:26.557174+0000","last_active":"2024-09-21T13:59:51.831828+0000","last_peered":"2024-09-21T13:59:51.831828+0000","last_clean":"2024-09-21T13:59:51.831828+0000","last_became_active":"2024-09-21T13:57:12.684254+0000","last_became_peered":"2024-09-21T13:57:12.684254+0000","last_unstale":"2024-09-21T13:59:51.831828+0000","last_undegraded":"2024-09-21T13:59:51.831828+0000","last_fullsized":"2024-09-21T13:59:51.831828+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:26.557002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:26.557002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:28.551613+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"306'8809","reported_seq":8912,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-09-21T13:59:55.884134+0000","last_change":"2024-09-21T13:57:53.863404+0000","last_active":"2024-09-21T13:59:55.884134+0000","last_peered":"2024-09-21T13:59:55.884134+0000","last_clean":"2024-09-21T13:59:55.884134+0000","last_became_active":"2024-09-21T13:57:53.863113+0000","last_became_peered":"2024-09-21T13:57:53.863113+0000","last_unstale":"2024-09-21T13:59:55.884134+0000","last_undegraded":"2024-09-21T13:59:55.884134+0000","last_fullsized":"2024-09-21T13:59:55.884134+0000","mapping_epoch":260,"log_start":"284'4100","ondisk_log_start":"284'4100","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4709,"log_dups_size":0,"ondisk_log_size":4709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432472064,"num_objects":6599,"num_object_clones":0,"num_object_copies":19797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8809,"num_write_kb":493056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"300'32","reported_seq":165,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.828483+0000","last_change":"2024-09-21T13:59:32.243718+0000","last_active":"2024-09-21T13:59:51.828483+0000","last_peered":"2024-09-21T13:59:51.828483+0000","last_clean":"2024-09-21T13:59:51.828483+0000","last_became_active":"2024-09-21T13:59:32.204624+0000","last_became_peered":"2024-09-21T13:59:32.204624+0000","last_unstale":"2024-09-21T13:59:51.828483+0000","last_undegraded":"2024-09-21T13:59:51.828483+0000","last_fullsized":"2024-09-21T13:59:51.828483+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'16","last_scrub_stamp":"2024-09-21T13:52:19.452163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:52:19.452163+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:40.020421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":499,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-09-21T13:59:55.055198+0000","last_change":"2024-09-21T13:58:57.773043+0000","last_active":"2024-09-21T13:59:55.055198+0000","last_peered":"2024-09-21T13:59:55.055198+0000","last_clean":"2024-09-21T13:59:55.055198+0000","last_became_active":"2024-09-21T13:58:17.778951+0000","last_became_peered":"2024-09-21T13:58:17.778951+0000","last_unstale":"2024-09-21T13:59:55.055198+0000","last_undegraded":"2024-09-21T13:59:55.055198+0000","last_fullsized":"2024-09-21T13:59:55.055198+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:57.772975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:57.772975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:09.786990+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"75'79","reported_seq":561,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-09-21T13:59:55.053608+0000","last_change":"2024-09-21T13:59:32.202714+0000","last_active":"2024-09-21T13:59:55.053608+0000","last_peered":"2024-09-21T13:59:55.053608+0000","last_clean":"2024-09-21T13:59:55.053608+0000","last_became_active":"2024-09-21T13:59:32.201469+0000","last_became_peered":"2024-09-21T13:59:32.201469+0000","last_unstale":"2024-09-21T13:59:55.053608+0000","last_undegraded":"2024-09-21T13:59:55.053608+0000","last_fullsized":"2024-09-21T13:59:55.053608+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":474,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-09-21T13:59:55.056182+0000","last_change":"2024-09-21T13:59:32.213870+0000","last_active":"2024-09-21T13:59:55.056182+0000","last_peered":"2024-09-21T13:59:55.056182+0000","last_clean":"2024-09-21T13:59:55.056182+0000","last_became_active":"2024-09-21T13:59:32.212228+0000","last_became_peered":"2024-09-21T13:59:32.212228+0000","last_unstale":"2024-09-21T13:59:55.056182+0000","last_undegraded":"2024-09-21T13:59:55.056182+0000","last_fullsized":"2024-09-21T13:59:55.056182+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:15.700409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:15.700409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":519,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.838463+0000","last_change":"2024-09-21T13:57:21.153067+0000","last_active":"2024-09-21T13:59:51.838463+0000","last_peered":"2024-09-21T13:59:51.838463+0000","last_clean":"2024-09-21T13:59:51.838463+0000","last_became_active":"2024-09-21T13:57:12.684495+0000","last_became_peered":"2024-09-21T13:57:12.684495+0000","last_unstale":"2024-09-21T13:59:51.838463+0000","last_undegraded":"2024-09-21T13:59:51.838463+0000","last_fullsized":"2024-09-21T13:59:51.838463+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:21.153018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:21.153018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:18.188142+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"305'8589","reported_seq":8697,"reported_epoch":305,"state":"active+undersized+degraded+remapped+forced_backfill+backfill_toofull","last_fresh":"2024-09-21T13:59:54.504016+0000","last_change":"2024-09-21T13:59:48.318743+0000","last_active":"2024-09-21T13:59:54.504016+0000","last_peered":"2024-09-21T13:59:54.504016+0000","last_clean":"2024-09-21T13:58:23.728526+0000","last_became_active":"2024-09-21T13:59:33.326654+0000","last_became_peered":"2024-09-21T13:59:33.326654+0000","last_unstale":"2024-09-21T13:59:54.504016+0000","last_undegraded":"2024-09-21T13:59:33.211255+0000","last_fullsized":"2024-09-21T13:59:33.189504+0000","mapping_epoch":299,"log_start":"283'3912","ondisk_log_start":"283'3912","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4677,"log_dups_size":0,"ondisk_log_size":4677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":431161344,"num_objects":6579,"num_object_clones":0,"num_object_copies":19737,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6579,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8589,"num_write_kb":485376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":6579}],"blocked_by":[],"up_primary":7,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"290'19","reported_seq":586,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.844873+0000","last_change":"2024-09-21T13:58:17.811300+0000","last_active":"2024-09-21T13:59:51.844873+0000","last_peered":"2024-09-21T13:59:51.844873+0000","last_clean":"2024-09-21T13:59:51.844873+0000","last_became_active":"2024-09-21T13:58:17.776310+0000","last_became_peered":"2024-09-21T13:58:17.776310+0000","last_unstale":"2024-09-21T13:59:51.844873+0000","last_undegraded":"2024-09-21T13:59:51.844873+0000","last_fullsized":"2024-09-21T13:59:51.844873+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:54:16.451588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:16.451588+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:13.578582+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":541,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.844143+0000","last_change":"2024-09-21T13:59:32.223608+0000","last_active":"2024-09-21T13:59:51.844143+0000","last_peered":"2024-09-21T13:59:51.844143+0000","last_clean":"2024-09-21T13:59:51.844143+0000","last_became_active":"2024-09-21T13:59:32.223247+0000","last_became_peered":"2024-09-21T13:59:32.223247+0000","last_unstale":"2024-09-21T13:59:51.844143+0000","last_undegraded":"2024-09-21T13:59:51.844143+0000","last_fullsized":"2024-09-21T13:59:51.844143+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:53.377100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:53.377100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:42.213058+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":533,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.837074+0000","last_change":"2024-09-21T13:58:17.782835+0000","last_active":"2024-09-21T13:59:51.837074+0000","last_peered":"2024-09-21T13:59:51.837074+0000","last_clean":"2024-09-21T13:59:51.837074+0000","last_became_active":"2024-09-21T13:58:17.782403+0000","last_became_peered":"2024-09-21T13:58:17.782403+0000","last_unstale":"2024-09-21T13:59:51.837074+0000","last_undegraded":"2024-09-21T13:59:51.837074+0000","last_fullsized":"2024-09-21T13:59:51.837074+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:34.299722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:55:34.299722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":477,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-09-21T13:59:55.057524+0000","last_change":"2024-09-21T13:58:17.778317+0000","last_active":"2024-09-21T13:59:55.057524+0000","last_peered":"2024-09-21T13:59:55.057524+0000","last_clean":"2024-09-21T13:59:55.057524+0000","last_became_active":"2024-09-21T13:58:17.777954+0000","last_became_peered":"2024-09-21T13:58:17.777954+0000","last_unstale":"2024-09-21T13:59:55.057524+0000","last_undegraded":"2024-09-21T13:59:55.057524+0000","last_fullsized":"2024-09-21T13:59:55.057524+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:58.952218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:54:58.952218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:13.848511+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.0","version":"305'8525","reported_seq":12565,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:54.499043+0000","last_change":"2024-09-21T13:59:24.817101+0000","last_active":"2024-09-21T13:59:54.499043+0000","last_peered":"2024-09-21T13:59:54.499043+0000","last_clean":"2024-09-21T13:59:54.499043+0000","last_became_active":"2024-09-21T13:58:36.723110+0000","last_became_peered":"2024-09-21T13:58:36.723110+0000","last_unstale":"2024-09-21T13:59:54.499043+0000","last_undegraded":"2024-09-21T13:59:54.499043+0000","last_fullsized":"2024-09-21T13:59:54.499043+0000","mapping_epoch":279,"log_start":"283'3847","ondisk_log_start":"283'3847","created":260,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'7035","last_scrub_stamp":"2024-09-21T13:59:24.816903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:59:24.816903+0000","objects_scrubbed":6955,"log_size":4678,"log_dups_size":0,"ondisk_log_size":4678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:30.971537+0000","scrub_duration":2446,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":429064192,"num_objects":6547,"num_object_clones":0,"num_object_copies":19641,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8525,"num_write_kb":482304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1924,"num_bytes_recovered":126091264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"290'31","reported_seq":574,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.844460+0000","last_change":"2024-09-21T13:56:55.354217+0000","last_active":"2024-09-21T13:59:51.844460+0000","last_peered":"2024-09-21T13:59:51.844460+0000","last_clean":"2024-09-21T13:59:51.844460+0000","last_became_active":"2024-09-21T13:52:04.313218+0000","last_became_peered":"2024-09-21T13:52:04.313218+0000","last_unstale":"2024-09-21T13:59:51.844460+0000","last_undegraded":"2024-09-21T13:59:51.844460+0000","last_fullsized":"2024-09-21T13:59:51.844460+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'17","last_scrub_stamp":"2024-09-21T13:56:55.354126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:56:55.354126+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":82,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.830440+0000","last_change":"2024-09-21T13:58:17.762896+0000","last_active":"2024-09-21T13:59:51.830440+0000","last_peered":"2024-09-21T13:59:51.830440+0000","last_clean":"2024-09-21T13:59:51.830440+0000","last_became_active":"2024-09-21T13:58:17.762520+0000","last_became_peered":"2024-09-21T13:58:17.762520+0000","last_unstale":"2024-09-21T13:59:51.830440+0000","last_undegraded":"2024-09-21T13:59:51.830440+0000","last_fullsized":"2024-09-21T13:59:51.830440+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:04.216734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:55:04.216734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:04.858083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":511,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.831926+0000","last_change":"2024-09-21T13:58:52.030800+0000","last_active":"2024-09-21T13:59:51.831926+0000","last_peered":"2024-09-21T13:59:51.831926+0000","last_clean":"2024-09-21T13:59:51.831926+0000","last_became_active":"2024-09-21T13:49:02.067472+0000","last_became_peered":"2024-09-21T13:49:02.067472+0000","last_unstale":"2024-09-21T13:59:51.831926+0000","last_undegraded":"2024-09-21T13:59:51.831926+0000","last_fullsized":"2024-09-21T13:59:51.831926+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:52.030718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:52.030718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:59:55.779410+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":398,"reported_epoch":307,"state":"active+clean","last_fresh":"2024-09-21T13:59:56.045655+0000","last_change":"2024-09-21T13:57:12.683321+0000","last_active":"2024-09-21T13:59:56.045655+0000","last_peered":"2024-09-21T13:59:56.045655+0000","last_clean":"2024-09-21T13:59:56.045655+0000","last_became_active":"2024-09-21T13:57:12.682979+0000","last_became_peered":"2024-09-21T13:57:12.682979+0000","last_unstale":"2024-09-21T13:59:56.045655+0000","last_undegraded":"2024-09-21T13:59:56.045655+0000","last_fullsized":"2024-09-21T13:59:56.045655+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:18.027944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:18.027944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:18.700579+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"306'8610","reported_seq":14443,"reported_epoch":306,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-21T13:59:55.903691+0000","last_change":"2024-09-21T13:59:33.434313+0000","last_active":"2024-09-21T13:59:55.903691+0000","last_peered":"2024-09-21T13:59:55.903691+0000","last_clean":"2024-09-21T13:59:31.188333+0000","last_became_active":"2024-09-21T13:59:32.273094+0000","last_became_peered":"2024-09-21T13:59:32.273094+0000","last_unstale":"2024-09-21T13:59:55.903691+0000","last_undegraded":"2024-09-21T13:59:55.903691+0000","last_fullsized":"2024-09-21T13:59:55.903691+0000","mapping_epoch":298,"log_start":"284'3900","ondisk_log_start":"284'3900","created":260,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4710,"log_dups_size":0,"ondisk_log_size":4710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":425852928,"num_objects":6498,"num_object_clones":0,"num_object_copies":19494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5813,"num_objects_unfound":0,"num_objects_dirty":6498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8610,"num_write_kb":483456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2852,"num_bytes_recovered":186908672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,4],"avail_no_missing":["3","4","6"],"object_location_counts":[{"shards":"3,4,6","objects":6498}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"290'20","reported_seq":601,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.848577+0000","last_change":"2024-09-21T13:58:17.789407+0000","last_active":"2024-09-21T13:59:51.848577+0000","last_peered":"2024-09-21T13:59:51.848577+0000","last_clean":"2024-09-21T13:59:51.848577+0000","last_became_active":"2024-09-21T13:58:17.765427+0000","last_became_peered":"2024-09-21T13:58:17.765427+0000","last_unstale":"2024-09-21T13:59:51.848577+0000","last_undegraded":"2024-09-21T13:59:51.848577+0000","last_fullsized":"2024-09-21T13:59:51.848577+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:58:13.464196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:13.464196+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":445,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-09-21T13:59:55.056390+0000","last_change":"2024-09-21T13:57:36.707211+0000","last_active":"2024-09-21T13:59:55.056390+0000","last_peered":"2024-09-21T13:59:55.056390+0000","last_clean":"2024-09-21T13:59:55.056390+0000","last_became_active":"2024-09-21T13:57:12.704203+0000","last_became_peered":"2024-09-21T13:57:12.704203+0000","last_unstale":"2024-09-21T13:59:55.056390+0000","last_undegraded":"2024-09-21T13:59:55.056390+0000","last_fullsized":"2024-09-21T13:59:55.056390+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:36.707148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:36.707148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"300'12","reported_seq":529,"reported_epoch":307,"state":"active+clean","last_fresh":"2024-09-21T13:59:56.045408+0000","last_change":"2024-09-21T13:59:38.738198+0000","last_active":"2024-09-21T13:59:56.045408+0000","last_peered":"2024-09-21T13:59:56.045408+0000","last_clean":"2024-09-21T13:59:56.045408+0000","last_became_active":"2024-09-21T13:59:32.224809+0000","last_became_peered":"2024-09-21T13:59:32.224809+0000","last_unstale":"2024-09-21T13:59:56.045408+0000","last_undegraded":"2024-09-21T13:59:56.045408+0000","last_fullsized":"2024-09-21T13:59:56.045408+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'12","last_scrub_stamp":"2024-09-21T13:59:38.738120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:38.738120+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:01:05.042603+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"68'1","reported_seq":529,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.844599+0000","last_change":"2024-09-21T13:59:31.392186+0000","last_active":"2024-09-21T13:59:51.844599+0000","last_peered":"2024-09-21T13:59:51.844599+0000","last_clean":"2024-09-21T13:59:51.844599+0000","last_became_active":"2024-09-21T13:52:04.313210+0000","last_became_peered":"2024-09-21T13:52:04.313210+0000","last_unstale":"2024-09-21T13:59:51.844599+0000","last_undegraded":"2024-09-21T13:59:51.844599+0000","last_fullsized":"2024-09-21T13:59:51.844599+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T13:59:31.392105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:31.392105+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:01:00.512112+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"305'8397","reported_seq":8498,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:53.701787+0000","last_change":"2024-09-21T13:57:53.213656+0000","last_active":"2024-09-21T13:59:53.701787+0000","last_peered":"2024-09-21T13:59:53.701787+0000","last_clean":"2024-09-21T13:59:53.701787+0000","last_became_active":"2024-09-21T13:57:53.213314+0000","last_became_peered":"2024-09-21T13:57:53.213314+0000","last_unstale":"2024-09-21T13:59:53.701787+0000","last_undegraded":"2024-09-21T13:59:53.701787+0000","last_fullsized":"2024-09-21T13:59:53.701787+0000","mapping_epoch":260,"log_start":"277'3100","ondisk_log_start":"277'3100","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":5297,"log_dups_size":0,"ondisk_log_size":5297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":425525248,"num_objects":6493,"num_object_clones":0,"num_object_copies":19479,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8397,"num_write_kb":476480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"290'15","reported_seq":556,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.833782+0000","last_change":"2024-09-21T13:58:50.498597+0000","last_active":"2024-09-21T13:59:51.833782+0000","last_peered":"2024-09-21T13:59:51.833782+0000","last_clean":"2024-09-21T13:59:51.833782+0000","last_became_active":"2024-09-21T13:49:00.053016+0000","last_became_peered":"2024-09-21T13:49:00.053016+0000","last_unstale":"2024-09-21T13:59:51.833782+0000","last_undegraded":"2024-09-21T13:59:51.833782+0000","last_fullsized":"2024-09-21T13:59:51.833782+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:58:50.498369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:50.498369+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:09.820421+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":544,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.836095+0000","last_change":"2024-09-21T13:58:17.783765+0000","last_active":"2024-09-21T13:59:51.836095+0000","last_peered":"2024-09-21T13:59:51.836095+0000","last_clean":"2024-09-21T13:59:51.836095+0000","last_became_active":"2024-09-21T13:58:17.782747+0000","last_became_peered":"2024-09-21T13:58:17.782747+0000","last_unstale":"2024-09-21T13:59:51.836095+0000","last_undegraded":"2024-09-21T13:59:51.836095+0000","last_fullsized":"2024-09-21T13:59:51.836095+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:53.894506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:53.894506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":452,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.843826+0000","last_change":"2024-09-21T13:57:12.683161+0000","last_active":"2024-09-21T13:59:51.843826+0000","last_peered":"2024-09-21T13:59:51.843826+0000","last_clean":"2024-09-21T13:59:51.843826+0000","last_became_active":"2024-09-21T13:57:12.680978+0000","last_became_peered":"2024-09-21T13:57:12.680978+0000","last_unstale":"2024-09-21T13:59:51.843826+0000","last_undegraded":"2024-09-21T13:59:51.843826+0000","last_fullsized":"2024-09-21T13:59:51.843826+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:21.402946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:21.402946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:08.693286+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":420,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-09-21T13:59:55.053534+0000","last_change":"2024-09-21T13:59:32.224001+0000","last_active":"2024-09-21T13:59:55.053534+0000","last_peered":"2024-09-21T13:59:55.053534+0000","last_clean":"2024-09-21T13:59:55.053534+0000","last_became_active":"2024-09-21T13:59:32.221628+0000","last_became_peered":"2024-09-21T13:59:32.221628+0000","last_unstale":"2024-09-21T13:59:55.053534+0000","last_undegraded":"2024-09-21T13:59:55.053534+0000","last_fullsized":"2024-09-21T13:59:55.053534+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:13.698091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:13.698091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"305'8814","reported_seq":10146,"reported_epoch":305,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-21T13:59:53.737579+0000","last_change":"2024-09-21T13:59:33.486474+0000","last_active":"2024-09-21T13:59:53.737579+0000","last_peered":"2024-09-21T13:59:53.737579+0000","last_clean":"2024-09-21T13:58:23.791074+0000","last_became_active":"2024-09-21T13:59:33.441400+0000","last_became_peered":"2024-09-21T13:59:33.441400+0000","last_unstale":"2024-09-21T13:59:53.737579+0000","last_undegraded":"2024-09-21T13:59:33.239445+0000","last_fullsized":"2024-09-21T13:59:33.192362+0000","mapping_epoch":299,"log_start":"281'3500","ondisk_log_start":"281'3500","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":5314,"log_dups_size":0,"ondisk_log_size":5314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":445382656,"num_objects":6796,"num_object_clones":0,"num_object_copies":20388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4643,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8814,"num_write_kb":499520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":611,"num_bytes_recovered":40042496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[{"shards":"3,5","objects":2153}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"290'35","reported_seq":156,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.830569+0000","last_change":"2024-09-21T13:59:32.234258+0000","last_active":"2024-09-21T13:59:51.830569+0000","last_peered":"2024-09-21T13:59:51.830569+0000","last_clean":"2024-09-21T13:59:51.830569+0000","last_became_active":"2024-09-21T13:59:32.204184+0000","last_became_peered":"2024-09-21T13:59:32.204184+0000","last_unstale":"2024-09-21T13:59:51.830569+0000","last_undegraded":"2024-09-21T13:59:51.830569+0000","last_fullsized":"2024-09-21T13:59:51.830569+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-21T13:58:12.650656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:12.650656+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:43.038682+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":501,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-09-21T13:59:55.056541+0000","last_change":"2024-09-21T13:59:32.222536+0000","last_active":"2024-09-21T13:59:55.056541+0000","last_peered":"2024-09-21T13:59:55.056541+0000","last_clean":"2024-09-21T13:59:55.056541+0000","last_became_active":"2024-09-21T13:59:32.221051+0000","last_became_peered":"2024-09-21T13:59:32.221051+0000","last_unstale":"2024-09-21T13:59:55.056541+0000","last_undegraded":"2024-09-21T13:59:55.056541+0000","last_fullsized":"2024-09-21T13:59:55.056541+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:21.587493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:21.587493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":481,"reported_epoch":307,"state":"active+clean","last_fresh":"2024-09-21T13:59:56.044543+0000","last_change":"2024-09-21T13:58:55.588614+0000","last_active":"2024-09-21T13:59:56.044543+0000","last_peered":"2024-09-21T13:59:56.044543+0000","last_clean":"2024-09-21T13:59:56.044543+0000","last_became_active":"2024-09-21T13:52:04.321146+0000","last_became_peered":"2024-09-21T13:52:04.321146+0000","last_unstale":"2024-09-21T13:59:56.044543+0000","last_undegraded":"2024-09-21T13:59:56.044543+0000","last_fullsized":"2024-09-21T13:59:56.044543+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:55.588554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:55.588554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:23.995144+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":439,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-09-21T13:59:55.053540+0000","last_change":"2024-09-21T13:59:32.223123+0000","last_active":"2024-09-21T13:59:55.053540+0000","last_peered":"2024-09-21T13:59:55.053540+0000","last_clean":"2024-09-21T13:59:55.053540+0000","last_became_active":"2024-09-21T13:59:32.219332+0000","last_became_peered":"2024-09-21T13:59:32.219332+0000","last_unstale":"2024-09-21T13:59:55.053540+0000","last_undegraded":"2024-09-21T13:59:55.053540+0000","last_fullsized":"2024-09-21T13:59:55.053540+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:19.537177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:19.537177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"305'8453","reported_seq":8554,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:54.464289+0000","last_change":"2024-09-21T13:57:53.515705+0000","last_active":"2024-09-21T13:59:54.464289+0000","last_peered":"2024-09-21T13:59:54.464289+0000","last_clean":"2024-09-21T13:59:54.464289+0000","last_became_active":"2024-09-21T13:57:53.515313+0000","last_became_peered":"2024-09-21T13:57:53.515313+0000","last_unstale":"2024-09-21T13:59:54.464289+0000","last_undegraded":"2024-09-21T13:59:54.464289+0000","last_fullsized":"2024-09-21T13:59:54.464289+0000","mapping_epoch":260,"log_start":"283'3770","ondisk_log_start":"283'3770","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4683,"log_dups_size":0,"ondisk_log_size":4683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420413440,"num_objects":6415,"num_object_clones":0,"num_object_copies":19245,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8453,"num_write_kb":475776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"290'18","reported_seq":574,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.844003+0000","last_change":"2024-09-21T13:59:28.340494+0000","last_active":"2024-09-21T13:59:51.844003+0000","last_peered":"2024-09-21T13:59:51.844003+0000","last_clean":"2024-09-21T13:59:51.844003+0000","last_became_active":"2024-09-21T13:57:12.680627+0000","last_became_peered":"2024-09-21T13:57:12.680627+0000","last_unstale":"2024-09-21T13:59:51.844003+0000","last_undegraded":"2024-09-21T13:59:51.844003+0000","last_fullsized":"2024-09-21T13:59:51.844003+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'18","last_scrub_stamp":"2024-09-21T13:59:28.340415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:28.340415+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:44.818651+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":467,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-09-21T13:59:55.057142+0000","last_change":"2024-09-21T13:59:32.201126+0000","last_active":"2024-09-21T13:59:55.057142+0000","last_peered":"2024-09-21T13:59:55.057142+0000","last_clean":"2024-09-21T13:59:55.057142+0000","last_became_active":"2024-09-21T13:59:32.200674+0000","last_became_peered":"2024-09-21T13:59:32.200674+0000","last_unstale":"2024-09-21T13:59:55.057142+0000","last_undegraded":"2024-09-21T13:59:55.057142+0000","last_fullsized":"2024-09-21T13:59:55.057142+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:21.582325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:21.582325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":446,"reported_epoch":307,"state":"active+clean","last_fresh":"2024-09-21T13:59:56.044568+0000","last_change":"2024-09-21T13:58:11.377940+0000","last_active":"2024-09-21T13:59:56.044568+0000","last_peered":"2024-09-21T13:59:56.044568+0000","last_clean":"2024-09-21T13:59:56.044568+0000","last_became_active":"2024-09-21T13:52:04.322147+0000","last_became_peered":"2024-09-21T13:52:04.322147+0000","last_unstale":"2024-09-21T13:59:56.044568+0000","last_undegraded":"2024-09-21T13:59:56.044568+0000","last_fullsized":"2024-09-21T13:59:56.044568+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:11.377803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:11.377803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:19.656170+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":453,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-09-21T13:59:55.056744+0000","last_change":"2024-09-21T13:59:32.207006+0000","last_active":"2024-09-21T13:59:55.056744+0000","last_peered":"2024-09-21T13:59:55.056744+0000","last_clean":"2024-09-21T13:59:55.056744+0000","last_became_active":"2024-09-21T13:59:32.203767+0000","last_became_peered":"2024-09-21T13:59:32.203767+0000","last_unstale":"2024-09-21T13:59:55.056744+0000","last_undegraded":"2024-09-21T13:59:55.056744+0000","last_fullsized":"2024-09-21T13:59:55.056744+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:09.040062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:55:09.040062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"305'8825","reported_seq":10038,"reported_epoch":305,"state":"active+recovering+undersized+degraded+remapped+forced_backfill","last_fresh":"2024-09-21T13:59:53.731545+0000","last_change":"2024-09-21T13:59:48.331911+0000","last_active":"2024-09-21T13:59:53.731545+0000","last_peered":"2024-09-21T13:59:53.731545+0000","last_clean":"2024-09-21T13:58:23.785212+0000","last_became_active":"2024-09-21T13:59:33.394359+0000","last_became_peered":"2024-09-21T13:59:33.394359+0000","last_unstale":"2024-09-21T13:59:53.731545+0000","last_undegraded":"2024-09-21T13:59:33.231231+0000","last_fullsized":"2024-09-21T13:59:33.191984+0000","mapping_epoch":299,"log_start":"281'3500","ondisk_log_start":"281'3500","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":5325,"log_dups_size":0,"ondisk_log_size":5325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":445972480,"num_objects":6805,"num_object_clones":0,"num_object_copies":20415,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8825,"num_write_kb":500160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":551,"num_bytes_recovered":36110336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[{"shards":"0,5","objects":2098}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"290'27","reported_seq":647,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.835767+0000","last_change":"2024-09-21T13:59:33.058912+0000","last_active":"2024-09-21T13:59:51.835767+0000","last_peered":"2024-09-21T13:59:51.835767+0000","last_clean":"2024-09-21T13:59:51.835767+0000","last_became_active":"2024-09-21T13:59:32.205453+0000","last_became_peered":"2024-09-21T13:59:32.205453+0000","last_unstale":"2024-09-21T13:59:51.835767+0000","last_undegraded":"2024-09-21T13:59:51.835767+0000","last_fullsized":"2024-09-21T13:59:51.835767+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'27","last_scrub_stamp":"2024-09-21T13:59:33.058859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:33.058859+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:35.203746+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":439,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-09-21T13:59:55.057616+0000","last_change":"2024-09-21T13:59:32.203889+0000","last_active":"2024-09-21T13:59:55.057616+0000","last_peered":"2024-09-21T13:59:55.057616+0000","last_clean":"2024-09-21T13:59:55.057616+0000","last_became_active":"2024-09-21T13:59:32.203172+0000","last_became_peered":"2024-09-21T13:59:32.203172+0000","last_unstale":"2024-09-21T13:59:55.057616+0000","last_undegraded":"2024-09-21T13:59:55.057616+0000","last_fullsized":"2024-09-21T13:59:55.057616+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T13:55:03.879706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:55:03.879706+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"300'14","reported_seq":618,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-09-21T13:59:55.053398+0000","last_change":"2024-09-21T13:59:32.220898+0000","last_active":"2024-09-21T13:59:55.053398+0000","last_peered":"2024-09-21T13:59:55.053398+0000","last_clean":"2024-09-21T13:59:55.053398+0000","last_became_active":"2024-09-21T13:59:32.206956+0000","last_became_peered":"2024-09-21T13:59:32.206956+0000","last_unstale":"2024-09-21T13:59:55.053398+0000","last_undegraded":"2024-09-21T13:59:55.053398+0000","last_fullsized":"2024-09-21T13:59:55.053398+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'11","last_scrub_stamp":"2024-09-21T13:56:31.616003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:56:31.616003+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":545,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.837159+0000","last_change":"2024-09-21T13:58:39.258717+0000","last_active":"2024-09-21T13:59:51.837159+0000","last_peered":"2024-09-21T13:59:51.837159+0000","last_clean":"2024-09-21T13:59:51.837159+0000","last_became_active":"2024-09-21T13:58:17.781952+0000","last_became_peered":"2024-09-21T13:58:17.781952+0000","last_unstale":"2024-09-21T13:59:51.837159+0000","last_undegraded":"2024-09-21T13:59:51.837159+0000","last_fullsized":"2024-09-21T13:59:51.837159+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:39.258659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:39.258659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T13:59:54.287146+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":82,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.828593+0000","last_change":"2024-09-21T13:58:17.777374+0000","last_active":"2024-09-21T13:59:51.828593+0000","last_peered":"2024-09-21T13:59:51.828593+0000","last_clean":"2024-09-21T13:59:51.828593+0000","last_became_active":"2024-09-21T13:58:17.777025+0000","last_became_peered":"2024-09-21T13:58:17.777025+0000","last_unstale":"2024-09-21T13:59:51.828593+0000","last_undegraded":"2024-09-21T13:59:51.828593+0000","last_fullsized":"2024-09-21T13:59:51.828593+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:13.252516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:13.252516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:42.040885+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"290'12","reported_seq":488,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-09-21T13:59:55.057030+0000","last_change":"2024-09-21T13:59:32.247422+0000","last_active":"2024-09-21T13:59:55.057030+0000","last_peered":"2024-09-21T13:59:55.057030+0000","last_clean":"2024-09-21T13:59:55.057030+0000","last_became_active":"2024-09-21T13:59:32.211306+0000","last_became_peered":"2024-09-21T13:59:32.211306+0000","last_unstale":"2024-09-21T13:59:55.057030+0000","last_undegraded":"2024-09-21T13:59:55.057030+0000","last_fullsized":"2024-09-21T13:59:55.057030+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'6","last_scrub_stamp":"2024-09-21T13:55:12.066873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:55:12.066873+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":538,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.836514+0000","last_change":"2024-09-21T13:57:37.106286+0000","last_active":"2024-09-21T13:59:51.836514+0000","last_peered":"2024-09-21T13:59:51.836514+0000","last_clean":"2024-09-21T13:59:51.836514+0000","last_became_active":"2024-09-21T13:57:12.702621+0000","last_became_peered":"2024-09-21T13:57:12.702621+0000","last_unstale":"2024-09-21T13:59:51.836514+0000","last_undegraded":"2024-09-21T13:59:51.836514+0000","last_fullsized":"2024-09-21T13:59:51.836514+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:37.106238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:37.106238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":485,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-09-21T13:59:55.056394+0000","last_change":"2024-09-21T13:59:32.222277+0000","last_active":"2024-09-21T13:59:55.056394+0000","last_peered":"2024-09-21T13:59:55.056394+0000","last_clean":"2024-09-21T13:59:55.056394+0000","last_became_active":"2024-09-21T13:59:32.219525+0000","last_became_peered":"2024-09-21T13:59:32.219525+0000","last_unstale":"2024-09-21T13:59:55.056394+0000","last_undegraded":"2024-09-21T13:59:55.056394+0000","last_fullsized":"2024-09-21T13:59:55.056394+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:02.838181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:55:02.838181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"290'15","reported_seq":451,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-09-21T13:59:55.057032+0000","last_change":"2024-09-21T13:59:32.238689+0000","last_active":"2024-09-21T13:59:55.057032+0000","last_peered":"2024-09-21T13:59:55.057032+0000","last_clean":"2024-09-21T13:59:55.057032+0000","last_became_active":"2024-09-21T13:59:32.209166+0000","last_became_peered":"2024-09-21T13:59:32.209166+0000","last_unstale":"2024-09-21T13:59:55.057032+0000","last_undegraded":"2024-09-21T13:59:55.057032+0000","last_fullsized":"2024-09-21T13:59:55.057032+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:55:04.990842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:55:04.990842+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"61'1","reported_seq":503,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-09-21T13:59:55.053786+0000","last_change":"2024-09-21T13:59:32.222826+0000","last_active":"2024-09-21T13:59:55.053786+0000","last_peered":"2024-09-21T13:59:55.053786+0000","last_clean":"2024-09-21T13:59:55.053786+0000","last_became_active":"2024-09-21T13:59:32.219515+0000","last_became_peered":"2024-09-21T13:59:32.219515+0000","last_unstale":"2024-09-21T13:59:55.053786+0000","last_undegraded":"2024-09-21T13:59:55.053786+0000","last_fullsized":"2024-09-21T13:59:55.053786+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:58:07.711028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:07.711028+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":436,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-09-21T13:59:55.056650+0000","last_change":"2024-09-21T13:59:32.211754+0000","last_active":"2024-09-21T13:59:55.056650+0000","last_peered":"2024-09-21T13:59:55.056650+0000","last_clean":"2024-09-21T13:59:55.056650+0000","last_became_active":"2024-09-21T13:59:32.210586+0000","last_became_peered":"2024-09-21T13:59:32.210586+0000","last_unstale":"2024-09-21T13:59:55.056650+0000","last_undegraded":"2024-09-21T13:59:55.056650+0000","last_fullsized":"2024-09-21T13:59:55.056650+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:28.014760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:28.014760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":480,"reported_epoch":307,"state":"active+clean","last_fresh":"2024-09-21T13:59:56.044817+0000","last_change":"2024-09-21T13:59:36.677306+0000","last_active":"2024-09-21T13:59:56.044817+0000","last_peered":"2024-09-21T13:59:56.044817+0000","last_clean":"2024-09-21T13:59:56.044817+0000","last_became_active":"2024-09-21T13:59:32.205803+0000","last_became_peered":"2024-09-21T13:59:32.205803+0000","last_unstale":"2024-09-21T13:59:56.044817+0000","last_undegraded":"2024-09-21T13:59:56.044817+0000","last_fullsized":"2024-09-21T13:59:56.044817+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:36.677259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:36.677259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:49.697302+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"290'15","reported_seq":571,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.838144+0000","last_change":"2024-09-21T13:59:32.247060+0000","last_active":"2024-09-21T13:59:51.838144+0000","last_peered":"2024-09-21T13:59:51.838144+0000","last_clean":"2024-09-21T13:59:51.838144+0000","last_became_active":"2024-09-21T13:59:32.226975+0000","last_became_peered":"2024-09-21T13:59:32.226975+0000","last_unstale":"2024-09-21T13:59:51.838144+0000","last_undegraded":"2024-09-21T13:59:51.838144+0000","last_fullsized":"2024-09-21T13:59:51.838144+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:57:24.124604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:57:24.124604+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":372,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-09-21T13:59:55.060276+0000","last_change":"2024-09-21T13:59:32.212325+0000","last_active":"2024-09-21T13:59:55.060276+0000","last_peered":"2024-09-21T13:59:55.060276+0000","last_clean":"2024-09-21T13:59:55.060276+0000","last_became_active":"2024-09-21T13:59:32.208749+0000","last_became_peered":"2024-09-21T13:59:32.208749+0000","last_unstale":"2024-09-21T13:59:55.060276+0000","last_undegraded":"2024-09-21T13:59:55.060276+0000","last_fullsized":"2024-09-21T13:59:55.060276+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:20.574745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:20.574745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"300'14","reported_seq":551,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-09-21T13:59:55.057605+0000","last_change":"2024-09-21T13:59:32.208253+0000","last_active":"2024-09-21T13:59:55.057605+0000","last_peered":"2024-09-21T13:59:55.057605+0000","last_clean":"2024-09-21T13:59:55.057605+0000","last_became_active":"2024-09-21T13:59:32.206926+0000","last_became_peered":"2024-09-21T13:59:32.206926+0000","last_unstale":"2024-09-21T13:59:55.057605+0000","last_undegraded":"2024-09-21T13:59:55.057605+0000","last_fullsized":"2024-09-21T13:59:55.057605+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'10","last_scrub_stamp":"2024-09-21T13:54:20.946782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:20.946782+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":525,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.844748+0000","last_change":"2024-09-21T13:58:54.805670+0000","last_active":"2024-09-21T13:59:51.844748+0000","last_peered":"2024-09-21T13:59:51.844748+0000","last_clean":"2024-09-21T13:59:51.844748+0000","last_became_active":"2024-09-21T13:52:04.315865+0000","last_became_peered":"2024-09-21T13:52:04.315865+0000","last_unstale":"2024-09-21T13:59:51.844748+0000","last_undegraded":"2024-09-21T13:59:51.844748+0000","last_fullsized":"2024-09-21T13:59:51.844748+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:54.805619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:54.805619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:05.176950+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"290'20","reported_seq":487,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-09-21T13:59:55.057421+0000","last_change":"2024-09-21T13:59:32.236530+0000","last_active":"2024-09-21T13:59:55.057421+0000","last_peered":"2024-09-21T13:59:55.057421+0000","last_clean":"2024-09-21T13:59:55.057421+0000","last_became_active":"2024-09-21T13:59:32.208332+0000","last_became_peered":"2024-09-21T13:59:32.208332+0000","last_unstale":"2024-09-21T13:59:55.057421+0000","last_undegraded":"2024-09-21T13:59:55.057421+0000","last_fullsized":"2024-09-21T13:59:55.057421+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:57:37.671391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:57:37.671391+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":479,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-09-21T13:59:55.056024+0000","last_change":"2024-09-21T13:59:32.212737+0000","last_active":"2024-09-21T13:59:55.056024+0000","last_peered":"2024-09-21T13:59:55.056024+0000","last_clean":"2024-09-21T13:59:55.056024+0000","last_became_active":"2024-09-21T13:59:32.209804+0000","last_became_peered":"2024-09-21T13:59:32.209804+0000","last_unstale":"2024-09-21T13:59:55.056024+0000","last_undegraded":"2024-09-21T13:59:55.056024+0000","last_fullsized":"2024-09-21T13:59:55.056024+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:25.571256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:25.571256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"68'8","reported_seq":659,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.837951+0000","last_change":"2024-09-21T13:59:32.104294+0000","last_active":"2024-09-21T13:59:51.837951+0000","last_peered":"2024-09-21T13:59:51.837951+0000","last_clean":"2024-09-21T13:59:51.837951+0000","last_became_active":"2024-09-21T13:49:02.067487+0000","last_became_peered":"2024-09-21T13:49:02.067487+0000","last_unstale":"2024-09-21T13:59:51.837951+0000","last_undegraded":"2024-09-21T13:59:51.837951+0000","last_fullsized":"2024-09-21T13:59:51.837951+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:59:32.104224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:32.104224+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:36.249102+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":376,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-09-21T13:59:55.056954+0000","last_change":"2024-09-21T13:59:32.213015+0000","last_active":"2024-09-21T13:59:55.056954+0000","last_peered":"2024-09-21T13:59:55.056954+0000","last_clean":"2024-09-21T13:59:55.056954+0000","last_became_active":"2024-09-21T13:59:32.211472+0000","last_became_peered":"2024-09-21T13:59:32.211472+0000","last_unstale":"2024-09-21T13:59:55.056954+0000","last_undegraded":"2024-09-21T13:59:55.056954+0000","last_fullsized":"2024-09-21T13:59:55.056954+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:22.647191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:22.647191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"290'19","reported_seq":591,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.843912+0000","last_change":"2024-09-21T13:58:17.790158+0000","last_active":"2024-09-21T13:59:51.843912+0000","last_peered":"2024-09-21T13:59:51.843912+0000","last_clean":"2024-09-21T13:59:51.843912+0000","last_became_active":"2024-09-21T13:58:17.771497+0000","last_became_peered":"2024-09-21T13:58:17.771497+0000","last_unstale":"2024-09-21T13:59:51.843912+0000","last_undegraded":"2024-09-21T13:59:51.843912+0000","last_fullsized":"2024-09-21T13:59:51.843912+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:54:56.539374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:56.539374+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:16.539347+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":113,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-09-21T13:59:55.056270+0000","last_change":"2024-09-21T13:59:32.202097+0000","last_active":"2024-09-21T13:59:55.056270+0000","last_peered":"2024-09-21T13:59:55.056270+0000","last_clean":"2024-09-21T13:59:55.056270+0000","last_became_active":"2024-09-21T13:59:32.201568+0000","last_became_peered":"2024-09-21T13:59:32.201568+0000","last_unstale":"2024-09-21T13:59:55.056270+0000","last_undegraded":"2024-09-21T13:59:55.056270+0000","last_fullsized":"2024-09-21T13:59:55.056270+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:01.228213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:01.228213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":538,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.833485+0000","last_change":"2024-09-21T13:57:35.409292+0000","last_active":"2024-09-21T13:59:51.833485+0000","last_peered":"2024-09-21T13:59:51.833485+0000","last_clean":"2024-09-21T13:59:51.833485+0000","last_became_active":"2024-09-21T13:57:12.683646+0000","last_became_peered":"2024-09-21T13:57:12.683646+0000","last_unstale":"2024-09-21T13:59:51.833485+0000","last_undegraded":"2024-09-21T13:59:51.833485+0000","last_fullsized":"2024-09-21T13:59:51.833485+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:35.409237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:35.409237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:09.229079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":433,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-09-21T13:59:55.055388+0000","last_change":"2024-09-21T13:57:39.643527+0000","last_active":"2024-09-21T13:59:55.055388+0000","last_peered":"2024-09-21T13:59:55.055388+0000","last_clean":"2024-09-21T13:59:55.055388+0000","last_became_active":"2024-09-21T13:57:12.703037+0000","last_became_peered":"2024-09-21T13:57:12.703037+0000","last_unstale":"2024-09-21T13:59:55.055388+0000","last_undegraded":"2024-09-21T13:59:55.055388+0000","last_fullsized":"2024-09-21T13:59:55.055388+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:39.643458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:39.643458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"290'15","reported_seq":593,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.835980+0000","last_change":"2024-09-21T13:58:39.977717+0000","last_active":"2024-09-21T13:59:51.835980+0000","last_peered":"2024-09-21T13:59:51.835980+0000","last_clean":"2024-09-21T13:59:51.835980+0000","last_became_active":"2024-09-21T13:58:17.764120+0000","last_became_peered":"2024-09-21T13:58:17.764120+0000","last_unstale":"2024-09-21T13:59:51.835980+0000","last_undegraded":"2024-09-21T13:59:51.835980+0000","last_fullsized":"2024-09-21T13:59:51.835980+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:58:39.977466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:39.977466+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:03.738519+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":483,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-09-21T13:59:55.056258+0000","last_change":"2024-09-21T13:59:32.223082+0000","last_active":"2024-09-21T13:59:55.056258+0000","last_peered":"2024-09-21T13:59:55.056258+0000","last_clean":"2024-09-21T13:59:55.056258+0000","last_became_active":"2024-09-21T13:59:32.221854+0000","last_became_peered":"2024-09-21T13:59:32.221854+0000","last_unstale":"2024-09-21T13:59:55.056258+0000","last_undegraded":"2024-09-21T13:59:55.056258+0000","last_fullsized":"2024-09-21T13:59:55.056258+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:17.691193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:17.691193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":419,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-09-21T13:59:55.056301+0000","last_change":"2024-09-21T13:57:12.703746+0000","last_active":"2024-09-21T13:59:55.056301+0000","last_peered":"2024-09-21T13:59:55.056301+0000","last_clean":"2024-09-21T13:59:55.056301+0000","last_became_active":"2024-09-21T13:57:12.703441+0000","last_became_peered":"2024-09-21T13:57:12.703441+0000","last_unstale":"2024-09-21T13:59:55.056301+0000","last_undegraded":"2024-09-21T13:59:55.056301+0000","last_fullsized":"2024-09-21T13:59:55.056301+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:19.999125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:19.999125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:05.702853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":452,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.835672+0000","last_change":"2024-09-21T13:58:47.334564+0000","last_active":"2024-09-21T13:59:51.835672+0000","last_peered":"2024-09-21T13:59:51.835672+0000","last_clean":"2024-09-21T13:59:51.835672+0000","last_became_active":"2024-09-21T13:50:29.689045+0000","last_became_peered":"2024-09-21T13:50:29.689045+0000","last_unstale":"2024-09-21T13:59:51.835672+0000","last_undegraded":"2024-09-21T13:59:51.835672+0000","last_fullsized":"2024-09-21T13:59:51.835672+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:47.334504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:47.334504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"290'19","reported_seq":563,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-09-21T13:59:55.055812+0000","last_change":"2024-09-21T13:59:32.277742+0000","last_active":"2024-09-21T13:59:55.055812+0000","last_peered":"2024-09-21T13:59:55.055812+0000","last_clean":"2024-09-21T13:59:55.055812+0000","last_became_active":"2024-09-21T13:59:32.231567+0000","last_became_peered":"2024-09-21T13:59:32.231567+0000","last_unstale":"2024-09-21T13:59:55.055812+0000","last_undegraded":"2024-09-21T13:59:55.055812+0000","last_fullsized":"2024-09-21T13:59:55.055812+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:54:22.978267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:22.978267+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":522,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.836594+0000","last_change":"2024-09-21T13:59:32.206128+0000","last_active":"2024-09-21T13:59:51.836594+0000","last_peered":"2024-09-21T13:59:51.836594+0000","last_clean":"2024-09-21T13:59:51.836594+0000","last_became_active":"2024-09-21T13:59:32.205826+0000","last_became_peered":"2024-09-21T13:59:32.205826+0000","last_unstale":"2024-09-21T13:59:51.836594+0000","last_undegraded":"2024-09-21T13:59:51.836594+0000","last_fullsized":"2024-09-21T13:59:51.836594+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:17.305144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:17.305144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":489,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-09-21T13:59:55.053630+0000","last_change":"2024-09-21T13:59:32.219508+0000","last_active":"2024-09-21T13:59:55.053630+0000","last_peered":"2024-09-21T13:59:55.053630+0000","last_clean":"2024-09-21T13:59:55.053630+0000","last_became_active":"2024-09-21T13:59:32.214004+0000","last_became_peered":"2024-09-21T13:59:32.214004+0000","last_unstale":"2024-09-21T13:59:55.053630+0000","last_undegraded":"2024-09-21T13:59:55.053630+0000","last_fullsized":"2024-09-21T13:59:55.053630+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:36.525417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:36.525417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":485,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-09-21T13:59:55.056479+0000","last_change":"2024-09-21T13:59:28.124373+0000","last_active":"2024-09-21T13:59:55.056479+0000","last_peered":"2024-09-21T13:59:55.056479+0000","last_clean":"2024-09-21T13:59:55.056479+0000","last_became_active":"2024-09-21T13:58:17.778115+0000","last_became_peered":"2024-09-21T13:58:17.778115+0000","last_unstale":"2024-09-21T13:59:55.056479+0000","last_undegraded":"2024-09-21T13:59:55.056479+0000","last_fullsized":"2024-09-21T13:59:55.056479+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:28.124296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:28.124296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:40.216842+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"290'8","reported_seq":509,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-09-21T13:59:55.053050+0000","last_change":"2024-09-21T13:59:26.034457+0000","last_active":"2024-09-21T13:59:55.053050+0000","last_peered":"2024-09-21T13:59:55.053050+0000","last_clean":"2024-09-21T13:59:55.053050+0000","last_became_active":"2024-09-21T13:58:17.775079+0000","last_became_peered":"2024-09-21T13:58:17.775079+0000","last_unstale":"2024-09-21T13:59:55.053050+0000","last_undegraded":"2024-09-21T13:59:55.053050+0000","last_fullsized":"2024-09-21T13:59:55.053050+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'8","last_scrub_stamp":"2024-09-21T13:59:26.034372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:26.034372+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:38.948765+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":470,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.838626+0000","last_change":"2024-09-21T13:58:16.457518+0000","last_active":"2024-09-21T13:59:51.838626+0000","last_peered":"2024-09-21T13:59:51.838626+0000","last_clean":"2024-09-21T13:59:51.838626+0000","last_became_active":"2024-09-21T13:57:12.684372+0000","last_became_peered":"2024-09-21T13:57:12.684372+0000","last_unstale":"2024-09-21T13:59:51.838626+0000","last_undegraded":"2024-09-21T13:59:51.838626+0000","last_fullsized":"2024-09-21T13:59:51.838626+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:16.457445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:16.457445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":93,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.828041+0000","last_change":"2024-09-21T13:59:32.204646+0000","last_active":"2024-09-21T13:59:51.828041+0000","last_peered":"2024-09-21T13:59:51.828041+0000","last_clean":"2024-09-21T13:59:51.828041+0000","last_became_active":"2024-09-21T13:59:32.204380+0000","last_became_peered":"2024-09-21T13:59:32.204380+0000","last_unstale":"2024-09-21T13:59:51.828041+0000","last_undegraded":"2024-09-21T13:59:51.828041+0000","last_fullsized":"2024-09-21T13:59:51.828041+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:51.254456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:53:51.254456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:41.002185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":422,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-09-21T13:59:55.060123+0000","last_change":"2024-09-21T13:59:32.212615+0000","last_active":"2024-09-21T13:59:55.060123+0000","last_peered":"2024-09-21T13:59:55.060123+0000","last_clean":"2024-09-21T13:59:55.060123+0000","last_became_active":"2024-09-21T13:59:32.210134+0000","last_became_peered":"2024-09-21T13:59:32.210134+0000","last_unstale":"2024-09-21T13:59:55.060123+0000","last_undegraded":"2024-09-21T13:59:55.060123+0000","last_fullsized":"2024-09-21T13:59:55.060123+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:08.852789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:55:08.852789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":523,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.837188+0000","last_change":"2024-09-21T13:58:17.706373+0000","last_active":"2024-09-21T13:59:51.837188+0000","last_peered":"2024-09-21T13:59:51.837188+0000","last_clean":"2024-09-21T13:59:51.837188+0000","last_became_active":"2024-09-21T13:49:04.075562+0000","last_became_peered":"2024-09-21T13:49:04.075562+0000","last_unstale":"2024-09-21T13:59:51.837188+0000","last_undegraded":"2024-09-21T13:59:51.837188+0000","last_fullsized":"2024-09-21T13:59:51.837188+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:17.706325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:17.706325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":465,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-09-21T13:59:55.053243+0000","last_change":"2024-09-21T13:58:13.776632+0000","last_active":"2024-09-21T13:59:55.053243+0000","last_peered":"2024-09-21T13:59:55.053243+0000","last_clean":"2024-09-21T13:59:55.053243+0000","last_became_active":"2024-09-21T13:57:12.704042+0000","last_became_peered":"2024-09-21T13:57:12.704042+0000","last_unstale":"2024-09-21T13:59:55.053243+0000","last_undegraded":"2024-09-21T13:59:55.053243+0000","last_fullsized":"2024-09-21T13:59:55.053243+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:13.776572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:13.776572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"290'19","reported_seq":106,"reported_epoch":305,"state":"active+clean","last_fresh":"2024-09-21T13:59:51.828129+0000","last_change":"2024-09-21T13:58:17.807643+0000","last_active":"2024-09-21T13:59:51.828129+0000","last_peered":"2024-09-21T13:59:51.828129+0000","last_clean":"2024-09-21T13:59:51.828129+0000","last_became_active":"2024-09-21T13:58:17.777573+0000","last_became_peered":"2024-09-21T13:58:17.777573+0000","last_unstale":"2024-09-21T13:59:51.828129+0000","last_undegraded":"2024-09-21T13:59:51.828129+0000","last_fullsized":"2024-09-21T13:59:51.828129+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:55:12.198562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:55:12.198562+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:07.882694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":391,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-09-21T13:59:55.056867+0000","last_change":"2024-09-21T13:59:32.203045+0000","last_active":"2024-09-21T13:59:55.056867+0000","last_peered":"2024-09-21T13:59:55.056867+0000","last_clean":"2024-09-21T13:59:55.056867+0000","last_became_active":"2024-09-21T13:59:32.202225+0000","last_became_peered":"2024-09-21T13:59:32.202225+0000","last_unstale":"2024-09-21T13:59:55.056867+0000","last_undegraded":"2024-09-21T13:59:55.056867+0000","last_fullsized":"2024-09-21T13:59:55.056867+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:40.650832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:40.650832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":6905856024,"num_objects":105376,"num_object_clones":0,"num_object_copies":316128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":45075,"num_objects_misplaced":5813,"num_objects_unfound":0,"num_objects_dirty":105376,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137436,"num_write_kb":7769921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8596,"num_bytes_recovered":563347456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18526117888,"data_stored":18526109744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81359,"ondisk_log_size":81359,"up":48,"acting":40,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1114,"num_read_kb":871,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":176,"num_bytes_recovered":1914,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":687,"ondisk_log_size":687,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":298,"seq":1279900254214,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1359440,"kb_used_data":1259648,"kb_used_omap":4,"kb_used_meta":99771,"kb_avail":92385712,"statfs":{"total":95995035648,"available":94602969088,"internally_reserved":0,"allocated":1289879552,"data_stored":1289317398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":102166184},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463053,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1603004,"kb_used_data":1283052,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":92142148,"statfs":{"total":95995035648,"available":94353559552,"internally_reserved":0,"allocated":1313845248,"data_stored":1313302489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":327612870},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724692,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4319808,"kb_used_data":3869360,"kb_used_omap":1,"kb_used_meta":450430,"kb_avail":89425344,"statfs":{"total":95995035648,"available":91571552256,"internally_reserved":0,"allocated":3962224640,"data_stored":3961678181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":461240773},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920922,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2038552,"kb_used_data":1682504,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":91706600,"statfs":{"total":95995035648,"available":93907558400,"internally_reserved":0,"allocated":1722884096,"data_stored":1722312739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":240,"seq":1030792151074,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3564140,"kb_used_data":3363864,"kb_used_omap":4,"kb_used_meta":200251,"kb_avail":90181012,"statfs":{"total":95995035648,"available":92345356288,"internally_reserved":0,"allocated":3444596736,"data_stored":3444018170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4443,"internal_metadata":205057701},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":130,"seq":558345748576,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1766424,"kb_used_data":1234432,"kb_used_omap":4,"kb_used_meta":531963,"kb_avail":91978728,"statfs":{"total":95995035648,"available":94186217472,"internally_reserved":0,"allocated":1264058368,"data_stored":1263512043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4441,"internal_metadata":544730791},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575020,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4293180,"kb_used_data":3766316,"kb_used_omap":1,"kb_used_meta":526782,"kb_avail":89451972,"statfs":{"total":95995035648,"available":91598819328,"internally_reserved":0,"allocated":3856707584,"data_stored":3856100131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":539425221},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738546,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2193812,"kb_used_data":1725312,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":91551340,"statfs":{"total":95995035648,"available":93748572160,"internally_reserved":0,"allocated":1766719488,"data_stored":1766156607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":479721925},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1756626944,"data_stored":1756626944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3852800000,"data_stored":3852795928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1249968128,"data_stored":1249968128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3422289920,"data_stored":3422289920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1719861248,"data_stored":1719861248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3932688384,"data_stored":3932684312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1305804800,"data_stored":1305804800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1286078464,"data_stored":1286078464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T14:00:00.158 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph pg cancel-force-backfill 9.c 9.e 9.9 9.5 2024-09-21T14:00:00.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:00 smithi187 ceph-mon[33032]: from='client.15687 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:00:00.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:00 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2523538973' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:00:00.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:00 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:00:00.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:00 smithi187 ceph-mon[33032]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set; Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull; Degraded data redundancy: 45075/316725 objects degraded (14.232%), 8 pgs degraded 2024-09-21T14:00:00.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:00 smithi187 ceph-mon[33032]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-09-21T14:00:00.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:00 smithi187 ceph-mon[33032]: [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull 2024-09-21T14:00:00.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:00 smithi187 ceph-mon[33032]: pg 9.1 is active+remapped+backfill_toofull, acting [3,6,4] 2024-09-21T14:00:00.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:00 smithi187 ceph-mon[33032]: pg 9.3 is active+undersized+degraded+remapped+forced_backfill+backfill_toofull, acting [1,3] 2024-09-21T14:00:00.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:00 smithi187 ceph-mon[33032]: pg 9.d is active+undersized+degraded+remapped+backfill_toofull, acting [1,5] 2024-09-21T14:00:00.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:00 smithi187 ceph-mon[33032]: pg 9.e is active+undersized+degraded+remapped+forced_backfill+backfill_toofull, acting [1,4] 2024-09-21T14:00:00.196 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:00 smithi187 ceph-mon[33032]: [WRN] PG_DEGRADED: Degraded data redundancy: 45075/316725 objects degraded (14.232%), 8 pgs degraded 2024-09-21T14:00:00.196 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:00 smithi187 ceph-mon[33032]: pg 9.3 is active+undersized+degraded+remapped+forced_backfill+backfill_toofull, acting [1,3] 2024-09-21T14:00:00.196 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:00 smithi187 ceph-mon[33032]: pg 9.5 is active+recovering+undersized+degraded+remapped+forced_backfill, acting [5,0] 2024-09-21T14:00:00.196 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:00 smithi187 ceph-mon[33032]: pg 9.7 is active+recovering+undersized+degraded+remapped, acting [5,3] 2024-09-21T14:00:00.196 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:00 smithi187 ceph-mon[33032]: pg 9.9 is active+recovering+undersized+degraded+remapped+forced_backfill, acting [0,5] 2024-09-21T14:00:00.196 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:00 smithi187 ceph-mon[33032]: pg 9.a is active+recovery_wait+undersized+degraded+remapped, acting [5,1] 2024-09-21T14:00:00.196 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:00 smithi187 ceph-mon[33032]: pg 9.c is active+undersized+degraded+remapped+backfill_wait+forced_backfill, acting [3,4] 2024-09-21T14:00:00.196 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:00 smithi187 ceph-mon[33032]: pg 9.d is active+undersized+degraded+remapped+backfill_toofull, acting [1,5] 2024-09-21T14:00:00.196 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:00 smithi187 ceph-mon[33032]: pg 9.e is active+undersized+degraded+remapped+forced_backfill+backfill_toofull, acting [1,4] 2024-09-21T14:00:00.196 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:00 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 45075/316725 objects degraded (14.232%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T14:00:00.196 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:00 smithi187 ceph-mon[33032]: osdmap e308: 8 total, 8 up, 8 in 2024-09-21T14:00:00.214 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:00.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:00.262 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:00.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:00.363 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:00:00.422 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:00.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:00.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:00:00.449 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:00:00.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:00 smithi049 ceph-mon[30080]: from='client.15687 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:00:00.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:00 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2523538973' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:00:00.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:00 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:00:00.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:00 smithi049 ceph-mon[30080]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set; Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull; Degraded data redundancy: 45075/316725 objects degraded (14.232%), 8 pgs degraded 2024-09-21T14:00:00.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:00 smithi049 ceph-mon[30080]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-09-21T14:00:00.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:00 smithi049 ceph-mon[30080]: [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull 2024-09-21T14:00:00.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:00 smithi049 ceph-mon[30080]: pg 9.1 is active+remapped+backfill_toofull, acting [3,6,4] 2024-09-21T14:00:00.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:00 smithi049 ceph-mon[30080]: pg 9.3 is active+undersized+degraded+remapped+forced_backfill+backfill_toofull, acting [1,3] 2024-09-21T14:00:00.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:00 smithi049 ceph-mon[30080]: pg 9.d is active+undersized+degraded+remapped+backfill_toofull, acting [1,5] 2024-09-21T14:00:00.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:00 smithi049 ceph-mon[30080]: pg 9.e is active+undersized+degraded+remapped+forced_backfill+backfill_toofull, acting [1,4] 2024-09-21T14:00:00.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:00 smithi049 ceph-mon[30080]: [WRN] PG_DEGRADED: Degraded data redundancy: 45075/316725 objects degraded (14.232%), 8 pgs degraded 2024-09-21T14:00:00.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:00 smithi049 ceph-mon[30080]: pg 9.3 is active+undersized+degraded+remapped+forced_backfill+backfill_toofull, acting [1,3] 2024-09-21T14:00:00.508 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:00 smithi049 ceph-mon[30080]: pg 9.5 is active+recovering+undersized+degraded+remapped+forced_backfill, acting [5,0] 2024-09-21T14:00:00.508 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:00 smithi049 ceph-mon[30080]: pg 9.7 is active+recovering+undersized+degraded+remapped, acting [5,3] 2024-09-21T14:00:00.508 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:00 smithi049 ceph-mon[30080]: pg 9.9 is active+recovering+undersized+degraded+remapped+forced_backfill, acting [0,5] 2024-09-21T14:00:00.508 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:00 smithi049 ceph-mon[30080]: pg 9.a is active+recovery_wait+undersized+degraded+remapped, acting [5,1] 2024-09-21T14:00:00.508 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:00 smithi049 ceph-mon[30080]: pg 9.c is active+undersized+degraded+remapped+backfill_wait+forced_backfill, acting [3,4] 2024-09-21T14:00:00.508 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:00 smithi049 ceph-mon[30080]: pg 9.d is active+undersized+degraded+remapped+backfill_toofull, acting [1,5] 2024-09-21T14:00:00.508 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:00 smithi049 ceph-mon[30080]: pg 9.e is active+undersized+degraded+remapped+forced_backfill+backfill_toofull, acting [1,4] 2024-09-21T14:00:00.508 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:00 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 45075/316725 objects degraded (14.232%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T14:00:00.508 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:00 smithi049 ceph-mon[30080]: osdmap e308: 8 total, 8 up, 8 in 2024-09-21T14:00:00.508 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:00 smithi049 ceph-mon[40489]: from='client.15687 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:00:00.508 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:00 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2523538973' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:00:00.508 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:00 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:00:00.509 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:00 smithi049 ceph-mon[40489]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set; Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull; Degraded data redundancy: 45075/316725 objects degraded (14.232%), 8 pgs degraded 2024-09-21T14:00:00.509 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:00 smithi049 ceph-mon[40489]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-09-21T14:00:00.509 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:00 smithi049 ceph-mon[40489]: [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull 2024-09-21T14:00:00.509 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:00 smithi049 ceph-mon[40489]: pg 9.1 is active+remapped+backfill_toofull, acting [3,6,4] 2024-09-21T14:00:00.509 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:00 smithi049 ceph-mon[40489]: pg 9.3 is active+undersized+degraded+remapped+forced_backfill+backfill_toofull, acting [1,3] 2024-09-21T14:00:00.509 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:00 smithi049 ceph-mon[40489]: pg 9.d is active+undersized+degraded+remapped+backfill_toofull, acting [1,5] 2024-09-21T14:00:00.509 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:00 smithi049 ceph-mon[40489]: pg 9.e is active+undersized+degraded+remapped+forced_backfill+backfill_toofull, acting [1,4] 2024-09-21T14:00:00.509 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:00 smithi049 ceph-mon[40489]: [WRN] PG_DEGRADED: Degraded data redundancy: 45075/316725 objects degraded (14.232%), 8 pgs degraded 2024-09-21T14:00:00.509 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:00 smithi049 ceph-mon[40489]: pg 9.3 is active+undersized+degraded+remapped+forced_backfill+backfill_toofull, acting [1,3] 2024-09-21T14:00:00.509 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:00 smithi049 ceph-mon[40489]: pg 9.5 is active+recovering+undersized+degraded+remapped+forced_backfill, acting [5,0] 2024-09-21T14:00:00.509 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:00 smithi049 ceph-mon[40489]: pg 9.7 is active+recovering+undersized+degraded+remapped, acting [5,3] 2024-09-21T14:00:00.509 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:00 smithi049 ceph-mon[40489]: pg 9.9 is active+recovering+undersized+degraded+remapped+forced_backfill, acting [0,5] 2024-09-21T14:00:00.509 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:00 smithi049 ceph-mon[40489]: pg 9.a is active+recovery_wait+undersized+degraded+remapped, acting [5,1] 2024-09-21T14:00:00.509 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:00 smithi049 ceph-mon[40489]: pg 9.c is active+undersized+degraded+remapped+backfill_wait+forced_backfill, acting [3,4] 2024-09-21T14:00:00.510 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:00 smithi049 ceph-mon[40489]: pg 9.d is active+undersized+degraded+remapped+backfill_toofull, acting [1,5] 2024-09-21T14:00:00.510 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:00 smithi049 ceph-mon[40489]: pg 9.e is active+undersized+degraded+remapped+forced_backfill+backfill_toofull, acting [1,4] 2024-09-21T14:00:00.510 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:00 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 45075/316725 objects degraded (14.232%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T14:00:00.510 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:00 smithi049 ceph-mon[40489]: osdmap e308: 8 total, 8 up, 8 in 2024-09-21T14:00:00.524 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:00.558 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:00.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:00.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:00.660 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:00.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:00.702 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:00.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:00.803 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:00.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:00.836 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:00.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:00.841 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:00:00 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:00:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:00:00.937 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:00:00.957 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:00:00.977 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:00.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:00.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:00:01.012 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:00:01.078 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:01.088 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T14:00:01.088 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:00:01.114 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:01.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:01.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:01.142 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:01 smithi049 ceph-mon[40489]: 3.1 scrub starts 2024-09-21T14:00:01.142 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:01 smithi049 ceph-mon[40489]: 3.1 scrub ok 2024-09-21T14:00:01.142 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:01 smithi049 ceph-mon[40489]: pgmap v552: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait+forced_backfill, 2 active+recovering+undersized+degraded+remapped+forced_backfill, 2 active+undersized+degraded+remapped+forced_backfill+backfill_toofull, 1 active+remapped+backfill_toofull, 139 active+clean; 6.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 916 op/s; 43839/307230 objects degraded (14.269%); 5813/307230 objects misplaced (1.892%); 6.7 MiB/s, 107 objects/s recovering 2024-09-21T14:00:01.142 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:01 smithi049 ceph-mon[30080]: 3.1 scrub starts 2024-09-21T14:00:01.142 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:01 smithi049 ceph-mon[30080]: 3.1 scrub ok 2024-09-21T14:00:01.142 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:01 smithi049 ceph-mon[30080]: pgmap v552: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait+forced_backfill, 2 active+recovering+undersized+degraded+remapped+forced_backfill, 2 active+undersized+degraded+remapped+forced_backfill+backfill_toofull, 1 active+remapped+backfill_toofull, 139 active+clean; 6.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 916 op/s; 43839/307230 objects degraded (14.269%); 5813/307230 objects misplaced (1.892%); 6.7 MiB/s, 107 objects/s recovering 2024-09-21T14:00:01.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:01 smithi187 ceph-mon[33032]: 3.1 scrub starts 2024-09-21T14:00:01.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:01 smithi187 ceph-mon[33032]: 3.1 scrub ok 2024-09-21T14:00:01.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:01 smithi187 ceph-mon[33032]: pgmap v552: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait+forced_backfill, 2 active+recovering+undersized+degraded+remapped+forced_backfill, 2 active+undersized+degraded+remapped+forced_backfill+backfill_toofull, 1 active+remapped+backfill_toofull, 139 active+clean; 6.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 916 op/s; 43839/307230 objects degraded (14.269%); 5813/307230 objects misplaced (1.892%); 6.7 MiB/s, 107 objects/s recovering 2024-09-21T14:00:01.216 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:01.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:01.250 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:01.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:01.351 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:01.392 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:01.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:01.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:01.494 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:01.535 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:01.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:01.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:01.637 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:00:01.675 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:01.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:01.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:00:01.777 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:01.815 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:01.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:01.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:01.880 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T14:00:01.917 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:00:01.972 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:01.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:01.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:00:02.073 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:02.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:02.125 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:02.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:02.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:02 smithi187 ceph-mon[33032]: 6.1e scrub starts 2024-09-21T14:00:02.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:02 smithi187 ceph-mon[33032]: 6.1e scrub ok 2024-09-21T14:00:02.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:02 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:00:02.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:02 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:00:02.226 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:00:02.258 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:02.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:02.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:00:02.338 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:00:02.360 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:02.394 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:02.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:02.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:02.496 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:02.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:02 smithi049 ceph-mon[40489]: 6.1e scrub starts 2024-09-21T14:00:02.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:02 smithi049 ceph-mon[40489]: 6.1e scrub ok 2024-09-21T14:00:02.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:02 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:00:02.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:02 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:00:02.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:02 smithi049 ceph-mon[30080]: 6.1e scrub starts 2024-09-21T14:00:02.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:02 smithi049 ceph-mon[30080]: 6.1e scrub ok 2024-09-21T14:00:02.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:02 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:00:02.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:02 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:00:02.543 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:02.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:02.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:02.644 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:02.688 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:02.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:02.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:02.790 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:00:02.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:02.829 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:02.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:00:02.894 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T14:00:02.929 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:00:02.992 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:02.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:02.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:00:03.094 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:03.118 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:03.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:03.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:03.220 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:03.269 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T14:00:03.269 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:00:03.269 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:03 smithi187 ceph-mon[33032]: 4.1b scrub starts 2024-09-21T14:00:03.270 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:03 smithi187 ceph-mon[33032]: 4.1b scrub ok 2024-09-21T14:00:03.270 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:03 smithi187 ceph-mon[33032]: pgmap v554: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait+forced_backfill, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped+forced_backfill, 2 active+undersized+degraded+remapped+forced_backfill+backfill_toofull, 1 active+remapped+backfill_toofull, 139 active+clean; 6.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 778 op/s; 43549/303954 objects degraded (14.327%); 5592/303954 objects misplaced (1.840%); 4.0 MiB/s, 63 objects/s recovering 2024-09-21T14:00:03.270 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:03 smithi187 ceph-mon[33032]: osdmap e309: 8 total, 8 up, 8 in 2024-09-21T14:00:03.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:03.277 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:03.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:03.378 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:03.402 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:03 smithi049 ceph-mon[30080]: 4.1b scrub starts 2024-09-21T14:00:03.402 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:03 smithi049 ceph-mon[30080]: 4.1b scrub ok 2024-09-21T14:00:03.402 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:03 smithi049 ceph-mon[30080]: pgmap v554: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait+forced_backfill, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped+forced_backfill, 2 active+undersized+degraded+remapped+forced_backfill+backfill_toofull, 1 active+remapped+backfill_toofull, 139 active+clean; 6.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 778 op/s; 43549/303954 objects degraded (14.327%); 5592/303954 objects misplaced (1.840%); 4.0 MiB/s, 63 objects/s recovering 2024-09-21T14:00:03.402 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:03 smithi049 ceph-mon[30080]: osdmap e309: 8 total, 8 up, 8 in 2024-09-21T14:00:03.403 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:03 smithi049 ceph-mon[40489]: 4.1b scrub starts 2024-09-21T14:00:03.403 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:03 smithi049 ceph-mon[40489]: 4.1b scrub ok 2024-09-21T14:00:03.403 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:03 smithi049 ceph-mon[40489]: pgmap v554: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait+forced_backfill, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped+forced_backfill, 2 active+undersized+degraded+remapped+forced_backfill+backfill_toofull, 1 active+remapped+backfill_toofull, 139 active+clean; 6.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 778 op/s; 43549/303954 objects degraded (14.327%); 5592/303954 objects misplaced (1.840%); 4.0 MiB/s, 63 objects/s recovering 2024-09-21T14:00:03.403 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:03 smithi049 ceph-mon[40489]: osdmap e309: 8 total, 8 up, 8 in 2024-09-21T14:00:03.408 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:03.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:03.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:03.510 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:00:03.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:03.576 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:03.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:00:03.619 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:00:03.677 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:00:03.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:03.710 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:03.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:00:03.811 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:00:03.843 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:03.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:03.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:00:03.943 INFO:teuthology.orchestra.run.smithi049.stderr:instructing pg(s) [9.9] on osd.0 to cancel-force-backfill; instructing pg(s) [9.e] on osd.1 to cancel-force-backfill; instructing pg(s) [9.c] on osd.3 to cancel-force-backfill; instructing pg(s) [9.5] on osd.5 to cancel-force-backfill; 2024-09-21T14:00:03.943 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-09-21T14:00:03.945 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:03.976 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:03.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:03.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:04.078 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:04.085 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-09-21T14:00:04.117 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:04.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:04.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:04.206 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T14:00:04.218 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:00:04.252 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:04.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:04.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:00:04.354 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:04.387 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:04.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:04.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:04.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:04 smithi187 ceph-mon[33032]: 4.1a scrub starts 2024-09-21T14:00:04.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:04 smithi187 ceph-mon[33032]: 4.1a scrub ok 2024-09-21T14:00:04.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:04 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/240383295' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:00:04.489 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:04.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:04 smithi049 ceph-mon[40489]: 4.1a scrub starts 2024-09-21T14:00:04.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:04 smithi049 ceph-mon[40489]: 4.1a scrub ok 2024-09-21T14:00:04.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:04 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/240383295' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:00:04.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:04 smithi049 ceph-mon[30080]: 4.1a scrub starts 2024-09-21T14:00:04.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:04 smithi049 ceph-mon[30080]: 4.1a scrub ok 2024-09-21T14:00:04.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:04 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/240383295' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:00:04.531 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:04.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:04.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:04.633 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:04.634 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:00:04.668 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:04.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:04.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:04.762 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:00:04.762 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:00:04.770 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:04.805 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:04.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:04.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:04.811 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:00:04.811 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:00:04.845 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:00:04.845 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:00:04.879 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:00:04.879 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:00:04.907 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:04.916 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:00:04.916 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:00:04.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:04.944 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:04.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:04.957 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:00:04.957 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:00:04.989 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:00:04.989 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:00:05.025 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:00:05.025 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:00:05.044 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:00:05.076 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:05.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:05.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:00:05.177 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:05.216 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:05 smithi049 ceph-mon[40489]: pgmap v556: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped+forced_backfill, 1 active+undersized+degraded+remapped+backfill_wait+forced_backfill, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped+forced_backfill, 2 active+undersized+degraded+remapped+forced_backfill+backfill_toofull, 1 active+remapped+backfill_toofull, 139 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 648 B/s rd, 0 B/s wr, 760 op/s; 43339/298734 objects degraded (14.508%); 5592/298734 objects misplaced (1.872%); 5.1 MiB/s, 81 objects/s recovering 2024-09-21T14:00:05.216 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:05 smithi049 ceph-mon[40489]: from='client.25486 -' entity='client.admin' cmd=[{"prefix": "pg cancel-force-backfill", "pgid": ["9.c", "9.e", "9.9", "9.5"], "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T14:00:05.216 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:05 smithi049 ceph-mon[40489]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T14:00:05.216 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:05 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/240383295' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T14:00:05.216 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:05 smithi049 ceph-mon[40489]: osdmap e310: 8 total, 8 up, 8 in 2024-09-21T14:00:05.216 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:05 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2245321601' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:00:05.216 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:05 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 43339/298734 objects degraded (14.508%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T14:00:05.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:05 smithi049 ceph-mon[30080]: pgmap v556: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped+forced_backfill, 1 active+undersized+degraded+remapped+backfill_wait+forced_backfill, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped+forced_backfill, 2 active+undersized+degraded+remapped+forced_backfill+backfill_toofull, 1 active+remapped+backfill_toofull, 139 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 648 B/s rd, 0 B/s wr, 760 op/s; 43339/298734 objects degraded (14.508%); 5592/298734 objects misplaced (1.872%); 5.1 MiB/s, 81 objects/s recovering 2024-09-21T14:00:05.216 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:05 smithi049 ceph-mon[30080]: from='client.25486 -' entity='client.admin' cmd=[{"prefix": "pg cancel-force-backfill", "pgid": ["9.c", "9.e", "9.9", "9.5"], "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T14:00:05.217 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:05 smithi049 ceph-mon[30080]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T14:00:05.217 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:05 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/240383295' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T14:00:05.217 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:05 smithi049 ceph-mon[30080]: osdmap e310: 8 total, 8 up, 8 in 2024-09-21T14:00:05.217 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:05 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2245321601' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:00:05.217 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:05 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 43339/298734 objects degraded (14.508%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T14:00:05.222 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:05.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:05.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:05.308 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T14:00:05.308 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-21T14:00:05.308 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:05.308 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:146920 9.b 9:df8cdf12:::benchmark_data_smithi049_138504_object25512:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e310)", 2024-09-21T14:00:05.308 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:00:05.298942+0000", 2024-09-21T14:00:05.308 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0084768580000000003, 2024-09-21T14:00:05.308 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0085153260000000001, 2024-09-21T14:00:05.308 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:00:05.308 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:00:05.308 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-21T14:00:05.308 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:00:05.309 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:00:05.309 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:00:05.309 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 146920 2024-09-21T14:00:05.309 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:05.309 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:00:05.309 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:05.309 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:00:05.309 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:05.298942+0000", 2024-09-21T14:00:05.309 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:00:05.309 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:05.309 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:05.309 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:00:05.309 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:05.298942+0000", 2024-09-21T14:00:05.309 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:00:05.309 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:05.310 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:05.310 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:00:05.310 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:05.298945+0000", 2024-09-21T14:00:05.310 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.875e-06 2024-09-21T14:00:05.310 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:05.310 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:05.310 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:00:05.310 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:05.298954+0000", 2024-09-21T14:00:05.310 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.5929999999999995e-06 2024-09-21T14:00:05.310 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:05.310 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:05.310 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:00:05.310 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:05.298979+0000", 2024-09-21T14:00:05.311 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4635999999999999e-05 2024-09-21T14:00:05.311 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:05.311 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:05.311 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:00:05.311 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:05.298988+0000", 2024-09-21T14:00:05.311 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.2809999999999997e-06 2024-09-21T14:00:05.311 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:05.311 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:05.311 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:00:05.311 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:05.299167+0000", 2024-09-21T14:00:05.311 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000179171 2024-09-21T14:00:05.311 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:05.311 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:05.311 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:00:05.311 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:05.299466+0000", 2024-09-21T14:00:05.312 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000298366 2024-09-21T14:00:05.312 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:05.312 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:05.312 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,5", 2024-09-21T14:00:05.312 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:05.299985+0000", 2024-09-21T14:00:05.312 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000519624 2024-09-21T14:00:05.312 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:05.312 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:05.312 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:00:05.312 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:05.304272+0000", 2024-09-21T14:00:05.312 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0042868589999999996 2024-09-21T14:00:05.312 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:05.312 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:05.312 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:05.313 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:05.305721+0000", 2024-09-21T14:00:05.313 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001449107 2024-09-21T14:00:05.313 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:00:05.313 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:00:05.313 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:00:05.313 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:05.313 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:05.313 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25421.0:146925 9.0 e310/279 9:0c00184a:::benchmark_data_smithi049_138504_object25517:head v 310'9152, mlcod=310'9150)", 2024-09-21T14:00:05.313 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:00:05.305224+0000", 2024-09-21T14:00:05.313 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0021942300000000001, 2024-09-21T14:00:05.313 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0023104330000000002, 2024-09-21T14:00:05.313 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:00:05.313 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:00:05.313 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T14:00:05.313 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:00:05.314 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:05.314 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:00:05.314 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:05.305224+0000", 2024-09-21T14:00:05.314 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:00:05.314 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:05.314 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:05.314 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:00:05.314 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:05.305224+0000", 2024-09-21T14:00:05.314 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:00:05.314 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:05.314 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:05.314 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:00:05.314 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:05.305226+0000", 2024-09-21T14:00:05.314 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0920000000000001e-06 2024-09-21T14:00:05.315 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:05.315 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:05.315 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:00:05.315 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:05.305231+0000", 2024-09-21T14:00:05.315 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6960000000000002e-06 2024-09-21T14:00:05.315 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:05.315 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:05.315 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:00:05.315 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:05.305244+0000", 2024-09-21T14:00:05.315 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2802999999999999e-05 2024-09-21T14:00:05.315 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:05.315 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:05.315 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:00:05.315 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:05.305251+0000", 2024-09-21T14:00:05.316 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7209999999999997e-06 2024-09-21T14:00:05.316 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:05.316 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:05.316 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:00:05.316 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:05.305299+0000", 2024-09-21T14:00:05.316 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8090999999999997e-05 2024-09-21T14:00:05.316 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:05.316 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:05.316 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:00:05.316 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:05.305316+0000", 2024-09-21T14:00:05.316 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7445999999999999e-05 2024-09-21T14:00:05.316 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:00:05.316 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:00:05.316 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:00:05.316 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:00:05.317 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2024-09-21T14:00:05.317 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 2 2024-09-21T14:00:05.317 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T14:00:05.324 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:00:05.359 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:05.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:05.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:00:05.362 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:05 smithi187 ceph-mon[33032]: pgmap v556: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped+forced_backfill, 1 active+undersized+degraded+remapped+backfill_wait+forced_backfill, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped+forced_backfill, 2 active+undersized+degraded+remapped+forced_backfill+backfill_toofull, 1 active+remapped+backfill_toofull, 139 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 648 B/s rd, 0 B/s wr, 760 op/s; 43339/298734 objects degraded (14.508%); 5592/298734 objects misplaced (1.872%); 5.1 MiB/s, 81 objects/s recovering 2024-09-21T14:00:05.362 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:05 smithi187 ceph-mon[33032]: from='client.25486 -' entity='client.admin' cmd=[{"prefix": "pg cancel-force-backfill", "pgid": ["9.c", "9.e", "9.9", "9.5"], "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T14:00:05.362 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:05 smithi187 ceph-mon[33032]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T14:00:05.362 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:05 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/240383295' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T14:00:05.362 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:05 smithi187 ceph-mon[33032]: osdmap e310: 8 total, 8 up, 8 in 2024-09-21T14:00:05.363 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:05 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2245321601' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:00:05.363 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:05 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 43339/298734 objects degraded (14.508%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T14:00:05.461 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:00:05.501 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:05.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:05.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:00:05.602 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:05.641 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:05.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:05.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:05.743 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:05.782 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:05.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:05.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:05.884 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:00:05.918 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:05.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:05.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:00:06.019 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:00:06.054 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:06.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:06.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:00:06.155 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:00:06.183 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:06.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:06.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:00:06.202 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T14:00:06.285 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:00:06.316 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:06.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:06.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:00:06.418 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:06.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:06 smithi187 ceph-mon[33032]: 3.a scrub starts 2024-09-21T14:00:06.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:06 smithi187 ceph-mon[33032]: 3.a scrub ok 2024-09-21T14:00:06.458 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:06.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:06.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:06.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:06 smithi049 ceph-mon[40489]: 3.a scrub starts 2024-09-21T14:00:06.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:06 smithi049 ceph-mon[40489]: 3.a scrub ok 2024-09-21T14:00:06.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:06 smithi049 ceph-mon[30080]: 3.a scrub starts 2024-09-21T14:00:06.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:06 smithi049 ceph-mon[30080]: 3.a scrub ok 2024-09-21T14:00:06.560 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:00:06.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:06.593 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:06.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:00:06.626 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:00:06.694 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:06.732 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:06.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:06.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:06.834 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:06.871 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:06.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:06.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:06.973 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:07.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:07.012 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:07.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:07.113 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:07.148 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:07.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:07.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:07.250 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:07.270 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T14:00:07.270 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-21T14:00:07.270 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-21T14:00:07.270 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-21T14:00:07.270 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T14:00:07.291 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:07.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:07.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:07.393 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:00:07.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:07.434 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:07.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:00:07.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:07 smithi187 ceph-mon[33032]: pgmap v558: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+recovery_wait+undersized+degraded+remapped+forced_backfill, 1 active+undersized+degraded+remapped+backfill_wait+forced_backfill, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 139 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 746 op/s; 42104/293823 objects degraded (14.330%); 5592/293823 objects misplaced (1.903%); 6.3 MiB/s, 100 objects/s recovering 2024-09-21T14:00:07.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:07 smithi187 ceph-mon[33032]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T14:00:07.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:07 smithi049 ceph-mon[30080]: pgmap v558: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+recovery_wait+undersized+degraded+remapped+forced_backfill, 1 active+undersized+degraded+remapped+backfill_wait+forced_backfill, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 139 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 746 op/s; 42104/293823 objects degraded (14.330%); 5592/293823 objects misplaced (1.903%); 6.3 MiB/s, 100 objects/s recovering 2024-09-21T14:00:07.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:07 smithi049 ceph-mon[30080]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T14:00:07.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:07 smithi049 ceph-mon[40489]: pgmap v558: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+recovery_wait+undersized+degraded+remapped+forced_backfill, 1 active+undersized+degraded+remapped+backfill_wait+forced_backfill, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 139 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 746 op/s; 42104/293823 objects degraded (14.330%); 5592/293823 objects misplaced (1.903%); 6.3 MiB/s, 100 objects/s recovering 2024-09-21T14:00:07.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:07 smithi049 ceph-mon[40489]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T14:00:07.535 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:07.571 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:07.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:07.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:07.673 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:07.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:07.707 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:07.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:07.808 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:00:07.841 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:07.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:07.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:00:07.943 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:07.976 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:07.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:07.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:08.078 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:08.105 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T14:00:08.106 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T14:00:08.109 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:08.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:08.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:08.211 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:00:08.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:08.257 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:08.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:00:08.358 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:08.406 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:08.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:08.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:08.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:08 smithi187 ceph-mon[33032]: 4.1c scrub starts 2024-09-21T14:00:08.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:08 smithi187 ceph-mon[33032]: 4.1c scrub ok 2024-09-21T14:00:08.507 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:08.508 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:08 smithi049 ceph-mon[30080]: 4.1c scrub starts 2024-09-21T14:00:08.508 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:08 smithi049 ceph-mon[30080]: 4.1c scrub ok 2024-09-21T14:00:08.509 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:08 smithi049 ceph-mon[40489]: 4.1c scrub starts 2024-09-21T14:00:08.509 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:08 smithi049 ceph-mon[40489]: 4.1c scrub ok 2024-09-21T14:00:08.535 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:00:08.547 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:00:08.555 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:08.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:08.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:08.651 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T14:00:08.657 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:00:08.748 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:08.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:08.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:00:08.850 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:08.883 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:08.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:08.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:08.985 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:00:09.032 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:09.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:09.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:00:09.134 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:09.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:09.168 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:09.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:09.268 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:09.303 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:09.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:09.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:09.388 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:09 smithi187 ceph-mon[33032]: pgmap v559: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+undersized+degraded+remapped+forced_backfill, 2 active+recovery_wait+undersized+degraded+remapped, 139 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 717 op/s; 41814/290280 objects degraded (14.405%); 5323/290280 objects misplaced (1.834%); 4.8 MiB/s, 76 objects/s recovering 2024-09-21T14:00:09.404 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:09.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:09.455 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:09.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:09.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:09 smithi049 ceph-mon[30080]: pgmap v559: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+undersized+degraded+remapped+forced_backfill, 2 active+recovery_wait+undersized+degraded+remapped, 139 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 717 op/s; 41814/290280 objects degraded (14.405%); 5323/290280 objects misplaced (1.834%); 4.8 MiB/s, 76 objects/s recovering 2024-09-21T14:00:09.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:09 smithi049 ceph-mon[40489]: pgmap v559: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+recovery_wait+undersized+degraded+remapped+forced_backfill, 2 active+recovery_wait+undersized+degraded+remapped, 139 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 717 op/s; 41814/290280 objects degraded (14.405%); 5323/290280 objects misplaced (1.834%); 4.8 MiB/s, 76 objects/s recovering 2024-09-21T14:00:09.557 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:00:09.562 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T14:00:09.562 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-21T14:00:09.562 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-21T14:00:09.563 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-21T14:00:09.563 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.563 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:36843 7.d 7:b9f54a05:::benchmark_data_smithi049_76798_object36842:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-21T14:00:09.563 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:30.623325+0000", 2024-09-21T14:00:09.563 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 578.93643408000003, 2024-09-21T14:00:09.563 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.14164070300000001, 2024-09-21T14:00:09.563 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:00:09.563 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:00:09.563 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:00:09.563 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:00:09.563 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T14:00:09.563 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T14:00:09.563 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 36843 2024-09-21T14:00:09.563 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.564 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:00:09.564 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.564 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:00:09.564 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.623325+0000", 2024-09-21T14:00:09.564 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:00:09.564 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.564 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.564 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:00:09.564 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.623325+0000", 2024-09-21T14:00:09.564 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:00:09.564 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.564 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.564 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:00:09.564 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.623328+0000", 2024-09-21T14:00:09.565 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8120000000000001e-06 2024-09-21T14:00:09.565 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.565 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.565 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:00:09.565 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.623434+0000", 2024-09-21T14:00:09.565 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010542799999999999 2024-09-21T14:00:09.565 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.565 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.565 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:00:09.565 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.623443+0000", 2024-09-21T14:00:09.565 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1200000000000008e-06 2024-09-21T14:00:09.565 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.565 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.565 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:00:09.566 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.623448+0000", 2024-09-21T14:00:09.566 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2079999999999999e-06 2024-09-21T14:00:09.566 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.566 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.566 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:00:09.566 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.623483+0000", 2024-09-21T14:00:09.566 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5048000000000001e-05 2024-09-21T14:00:09.566 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.566 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.566 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:00:09.566 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.623592+0000", 2024-09-21T14:00:09.566 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000108585 2024-09-21T14:00:09.566 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.567 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.567 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-21T14:00:09.567 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.623673+0000", 2024-09-21T14:00:09.567 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1079000000000001e-05 2024-09-21T14:00:09.567 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.567 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.567 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:00:09.567 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.624761+0000", 2024-09-21T14:00:09.567 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001088449 2024-09-21T14:00:09.567 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.567 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.567 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:09.567 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.625971+0000", 2024-09-21T14:00:09.567 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0012098949999999999 2024-09-21T14:00:09.568 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.568 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.568 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:09.568 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.764850+0000", 2024-09-21T14:00:09.568 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13887893000000001 2024-09-21T14:00:09.568 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.568 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.568 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:00:09.568 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.764907+0000", 2024-09-21T14:00:09.568 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6901999999999998e-05 2024-09-21T14:00:09.568 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.568 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.568 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:00:09.568 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:30.764965+0000", 2024-09-21T14:00:09.568 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8247e-05 2024-09-21T14:00:09.569 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:00:09.569 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:00:09.569 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:00:09.569 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.569 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.569 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:49358 7.d 7:bafe9a80:::benchmark_data_smithi049_76798_object49357:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T14:00:09.569 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:38.254356+0000", 2024-09-21T14:00:09.569 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 571.30540262800002, 2024-09-21T14:00:09.569 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.153550043, 2024-09-21T14:00:09.569 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:00:09.569 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:00:09.569 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:00:09.569 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:00:09.569 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T14:00:09.570 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T14:00:09.570 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 49358 2024-09-21T14:00:09.570 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.570 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:00:09.570 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.570 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:00:09.570 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.254356+0000", 2024-09-21T14:00:09.570 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:00:09.570 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.570 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.570 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:00:09.570 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.254356+0000", 2024-09-21T14:00:09.570 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:00:09.570 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.571 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.571 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:00:09.571 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.254357+0000", 2024-09-21T14:00:09.571 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4600000000000004e-07 2024-09-21T14:00:09.571 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.571 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.571 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:00:09.571 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.254488+0000", 2024-09-21T14:00:09.571 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013163199999999999 2024-09-21T14:00:09.571 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.571 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.571 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:00:09.571 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.254503+0000", 2024-09-21T14:00:09.572 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.452e-05 2024-09-21T14:00:09.572 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.572 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.572 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:00:09.572 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.254510+0000", 2024-09-21T14:00:09.572 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.554e-06 2024-09-21T14:00:09.572 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.572 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.572 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:00:09.572 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.254578+0000", 2024-09-21T14:00:09.572 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7968999999999999e-05 2024-09-21T14:00:09.572 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.572 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.572 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:00:09.572 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.254736+0000", 2024-09-21T14:00:09.573 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000157219 2024-09-21T14:00:09.573 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.573 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.573 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-21T14:00:09.573 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.254886+0000", 2024-09-21T14:00:09.573 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000150667 2024-09-21T14:00:09.573 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.573 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.573 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:00:09.573 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.257811+0000", 2024-09-21T14:00:09.573 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00292436 2024-09-21T14:00:09.573 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.573 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.573 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:09.574 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.261523+0000", 2024-09-21T14:00:09.574 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0037119639999999999 2024-09-21T14:00:09.574 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.574 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.574 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:09.574 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.407826+0000", 2024-09-21T14:00:09.574 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.146303813 2024-09-21T14:00:09.574 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.574 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.574 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:00:09.574 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.407877+0000", 2024-09-21T14:00:09.574 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0603999999999999e-05 2024-09-21T14:00:09.574 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.575 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.575 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:00:09.575 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.407906+0000", 2024-09-21T14:00:09.575 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8994999999999999e-05 2024-09-21T14:00:09.575 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:00:09.575 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:00:09.575 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:00:09.575 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.575 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.575 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:49364 7.d 7:ba37e2e5:::benchmark_data_smithi049_76798_object49363:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T14:00:09.575 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:38.254570+0000", 2024-09-21T14:00:09.575 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 571.30518885699996, 2024-09-21T14:00:09.575 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.156026475, 2024-09-21T14:00:09.575 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:00:09.575 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:00:09.576 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:00:09.576 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:00:09.576 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T14:00:09.576 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T14:00:09.576 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 49364 2024-09-21T14:00:09.576 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.576 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:00:09.576 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.576 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:00:09.577 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.254570+0000", 2024-09-21T14:00:09.577 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:00:09.577 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.577 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.577 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:00:09.577 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.254570+0000", 2024-09-21T14:00:09.577 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:00:09.577 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.577 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.577 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:00:09.577 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.254571+0000", 2024-09-21T14:00:09.577 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.8700000000000004e-07 2024-09-21T14:00:09.577 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.577 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.578 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:00:09.578 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.254637+0000", 2024-09-21T14:00:09.578 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6704000000000004e-05 2024-09-21T14:00:09.578 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.578 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.578 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:00:09.578 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.254652+0000", 2024-09-21T14:00:09.578 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.438e-05 2024-09-21T14:00:09.578 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.578 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.578 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:00:09.578 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.254659+0000", 2024-09-21T14:00:09.578 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9589999999999998e-06 2024-09-21T14:00:09.579 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.579 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.579 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:00:09.579 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.256288+0000", 2024-09-21T14:00:09.579 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001629669 2024-09-21T14:00:09.579 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.579 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.579 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:00:09.579 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.256409+0000", 2024-09-21T14:00:09.579 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012085400000000001 2024-09-21T14:00:09.579 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.579 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.579 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-21T14:00:09.579 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.256520+0000", 2024-09-21T14:00:09.579 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000111141 2024-09-21T14:00:09.580 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.580 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.580 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:00:09.580 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.257926+0000", 2024-09-21T14:00:09.580 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001405959 2024-09-21T14:00:09.580 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.580 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.580 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:09.580 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.263043+0000", 2024-09-21T14:00:09.580 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0051160989999999998 2024-09-21T14:00:09.580 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.580 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.580 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:09.580 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.410522+0000", 2024-09-21T14:00:09.581 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.14747992600000001 2024-09-21T14:00:09.581 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.581 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.581 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:00:09.581 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.410571+0000", 2024-09-21T14:00:09.581 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8538000000000002e-05 2024-09-21T14:00:09.581 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.581 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.581 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:00:09.581 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.410596+0000", 2024-09-21T14:00:09.581 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5259e-05 2024-09-21T14:00:09.581 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:00:09.581 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:00:09.581 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:00:09.581 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.582 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.582 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:49368 7.d 7:bfd0cd80:::benchmark_data_smithi049_76798_object49367:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T14:00:09.582 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:38.259454+0000", 2024-09-21T14:00:09.582 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 571.30030464699996, 2024-09-21T14:00:09.582 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.15201478299999999, 2024-09-21T14:00:09.582 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:00:09.582 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:00:09.582 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:00:09.582 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:00:09.582 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T14:00:09.582 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T14:00:09.582 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 49368 2024-09-21T14:00:09.582 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.582 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:00:09.583 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.583 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:00:09.583 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.259454+0000", 2024-09-21T14:00:09.583 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:00:09.583 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.583 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.583 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:00:09.583 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.259454+0000", 2024-09-21T14:00:09.583 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:00:09.583 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.583 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.583 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:00:09.583 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.259455+0000", 2024-09-21T14:00:09.583 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3260000000000001e-06 2024-09-21T14:00:09.583 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.584 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.584 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:00:09.584 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.259505+0000", 2024-09-21T14:00:09.584 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9941999999999997e-05 2024-09-21T14:00:09.584 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.584 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.584 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:00:09.584 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.259516+0000", 2024-09-21T14:00:09.584 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0943e-05 2024-09-21T14:00:09.584 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.584 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.584 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:00:09.584 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.259523+0000", 2024-09-21T14:00:09.584 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2200000000000003e-06 2024-09-21T14:00:09.584 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.585 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.585 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:00:09.585 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.259582+0000", 2024-09-21T14:00:09.585 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8789000000000002e-05 2024-09-21T14:00:09.585 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.585 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.585 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:00:09.585 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.259708+0000", 2024-09-21T14:00:09.585 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012536700000000001 2024-09-21T14:00:09.585 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.585 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.585 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-21T14:00:09.585 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.259861+0000", 2024-09-21T14:00:09.585 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015389900000000001 2024-09-21T14:00:09.585 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.586 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.586 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:00:09.586 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.261617+0000", 2024-09-21T14:00:09.586 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0017559260000000001 2024-09-21T14:00:09.586 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.586 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.586 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:09.586 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.263374+0000", 2024-09-21T14:00:09.586 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0017561440000000001 2024-09-21T14:00:09.586 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.586 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.586 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:09.586 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.411384+0000", 2024-09-21T14:00:09.586 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.148010004 2024-09-21T14:00:09.587 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.587 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.587 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:00:09.587 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.411437+0000", 2024-09-21T14:00:09.587 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3600999999999997e-05 2024-09-21T14:00:09.587 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.587 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.587 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:00:09.587 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.411469+0000", 2024-09-21T14:00:09.587 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1622000000000003e-05 2024-09-21T14:00:09.587 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:00:09.587 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:00:09.587 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:00:09.587 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.587 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.588 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:49375 7.7 7:ecf0f90a:::benchmark_data_smithi049_76798_object49374:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T14:00:09.588 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:38.263906+0000", 2024-09-21T14:00:09.588 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 571.29585261199998, 2024-09-21T14:00:09.588 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.161950177, 2024-09-21T14:00:09.588 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:00:09.588 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:00:09.588 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:00:09.588 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:00:09.588 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T14:00:09.588 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T14:00:09.588 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 49375 2024-09-21T14:00:09.588 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.588 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:00:09.588 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.588 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:00:09.589 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.263906+0000", 2024-09-21T14:00:09.589 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:00:09.589 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.589 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.589 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:00:09.589 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.263906+0000", 2024-09-21T14:00:09.589 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:00:09.589 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.589 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.589 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:00:09.589 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.263907+0000", 2024-09-21T14:00:09.589 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.61e-07 2024-09-21T14:00:09.589 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.589 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.590 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:00:09.590 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.263963+0000", 2024-09-21T14:00:09.590 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5772000000000001e-05 2024-09-21T14:00:09.590 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.590 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.590 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:00:09.590 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.263969+0000", 2024-09-21T14:00:09.590 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4200000000000004e-06 2024-09-21T14:00:09.590 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.590 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.590 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:00:09.590 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.263973+0000", 2024-09-21T14:00:09.590 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1869999999999999e-06 2024-09-21T14:00:09.590 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.591 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.591 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:00:09.591 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.264028+0000", 2024-09-21T14:00:09.591 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4657999999999997e-05 2024-09-21T14:00:09.591 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.591 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.591 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:00:09.591 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.264143+0000", 2024-09-21T14:00:09.591 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000115223 2024-09-21T14:00:09.591 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.591 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.591 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3", 2024-09-21T14:00:09.591 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.264277+0000", 2024-09-21T14:00:09.591 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000133993 2024-09-21T14:00:09.591 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.592 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.592 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:00:09.592 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.265777+0000", 2024-09-21T14:00:09.592 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0014999740000000001 2024-09-21T14:00:09.592 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.592 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.592 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:09.592 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.268782+0000", 2024-09-21T14:00:09.592 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0030045929999999998 2024-09-21T14:00:09.592 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.592 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.592 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:09.592 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.425737+0000", 2024-09-21T14:00:09.592 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.156955024 2024-09-21T14:00:09.593 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.593 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.593 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:00:09.593 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.425795+0000", 2024-09-21T14:00:09.593 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8443000000000003e-05 2024-09-21T14:00:09.593 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.593 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.593 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:00:09.593 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.425856+0000", 2024-09-21T14:00:09.593 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1129000000000004e-05 2024-09-21T14:00:09.593 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:00:09.593 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:00:09.593 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:00:09.593 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.593 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.594 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:49376 7.7 7:e5b9203c:::benchmark_data_smithi049_76798_object49375:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T14:00:09.594 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:38.265943+0000", 2024-09-21T14:00:09.594 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 571.29381564899995, 2024-09-21T14:00:09.594 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.16002640000000001, 2024-09-21T14:00:09.594 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:00:09.594 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:00:09.594 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:00:09.594 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:00:09.594 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T14:00:09.594 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T14:00:09.594 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 49376 2024-09-21T14:00:09.594 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.594 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:00:09.594 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.595 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:00:09.595 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.265943+0000", 2024-09-21T14:00:09.595 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:00:09.595 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.595 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.595 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:00:09.595 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.265943+0000", 2024-09-21T14:00:09.595 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:00:09.595 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.595 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.595 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:00:09.595 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.265944+0000", 2024-09-21T14:00:09.596 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.237e-06 2024-09-21T14:00:09.596 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.596 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.596 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:00:09.596 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.266008+0000", 2024-09-21T14:00:09.596 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3382999999999999e-05 2024-09-21T14:00:09.596 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.596 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.596 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:00:09.596 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.266018+0000", 2024-09-21T14:00:09.596 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0225e-05 2024-09-21T14:00:09.596 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.596 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.596 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:00:09.597 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.266023+0000", 2024-09-21T14:00:09.597 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0810000000000001e-06 2024-09-21T14:00:09.597 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.597 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.597 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:00:09.597 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.266066+0000", 2024-09-21T14:00:09.597 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2939999999999999e-05 2024-09-21T14:00:09.597 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.597 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.597 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:00:09.597 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.266173+0000", 2024-09-21T14:00:09.597 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000107089 2024-09-21T14:00:09.597 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.597 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.597 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3", 2024-09-21T14:00:09.598 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.266305+0000", 2024-09-21T14:00:09.598 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013210699999999999 2024-09-21T14:00:09.598 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.598 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.598 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:00:09.598 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.268746+0000", 2024-09-21T14:00:09.598 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00244072 2024-09-21T14:00:09.598 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.598 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.598 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:09.598 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.269249+0000", 2024-09-21T14:00:09.598 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00050357600000000005 2024-09-21T14:00:09.598 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.598 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.599 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:09.599 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.425891+0000", 2024-09-21T14:00:09.599 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.15664171700000001 2024-09-21T14:00:09.599 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.599 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.599 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:00:09.599 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.425934+0000", 2024-09-21T14:00:09.599 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3136999999999997e-05 2024-09-21T14:00:09.599 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.599 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.599 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:00:09.599 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.425969+0000", 2024-09-21T14:00:09.599 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5188000000000003e-05 2024-09-21T14:00:09.599 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:00:09.600 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:00:09.600 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:00:09.600 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.600 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.600 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:49377 7.7 7:e5eecd6c:::benchmark_data_smithi049_76798_object49376:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-21T14:00:09.600 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:50:38.266042+0000", 2024-09-21T14:00:09.600 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 571.29371629800005, 2024-09-21T14:00:09.600 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.160051258, 2024-09-21T14:00:09.600 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:00:09.600 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:00:09.600 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:00:09.600 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:00:09.600 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T14:00:09.600 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T14:00:09.601 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 49377 2024-09-21T14:00:09.601 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.601 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:00:09.601 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.601 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:00:09.601 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.266042+0000", 2024-09-21T14:00:09.601 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:00:09.601 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.601 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.601 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:00:09.601 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.266042+0000", 2024-09-21T14:00:09.601 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:00:09.601 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.601 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.602 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:00:09.602 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.266043+0000", 2024-09-21T14:00:09.602 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2599999999999998e-07 2024-09-21T14:00:09.602 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.602 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.602 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:00:09.602 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.266116+0000", 2024-09-21T14:00:09.602 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3120999999999993e-05 2024-09-21T14:00:09.602 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.602 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.602 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:00:09.602 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.266126+0000", 2024-09-21T14:00:09.603 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6809999999999993e-06 2024-09-21T14:00:09.603 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.603 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:09.604 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:00:09.604 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.266133+0000", 2024-09-21T14:00:09.604 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1590000000000004e-06 2024-09-21T14:00:09.604 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.604 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.604 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:00:09.604 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.267417+0000", 2024-09-21T14:00:09.604 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00128472 2024-09-21T14:00:09.604 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.605 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.605 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:00:09.605 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.267503+0000", 2024-09-21T14:00:09.605 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5261999999999998e-05 2024-09-21T14:00:09.605 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.605 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.605 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3", 2024-09-21T14:00:09.605 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.267626+0000", 2024-09-21T14:00:09.605 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000123435 2024-09-21T14:00:09.605 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.605 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.605 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:00:09.605 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.268883+0000", 2024-09-21T14:00:09.605 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00125657 2024-09-21T14:00:09.606 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.606 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.606 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:09.606 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.269389+0000", 2024-09-21T14:00:09.606 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00050581200000000002 2024-09-21T14:00:09.606 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.606 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.606 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:09.606 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.426010+0000", 2024-09-21T14:00:09.606 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.156621173 2024-09-21T14:00:09.606 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.606 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.606 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:00:09.606 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.426059+0000", 2024-09-21T14:00:09.607 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8891000000000003e-05 2024-09-21T14:00:09.607 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.607 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.607 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:00:09.607 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:50:38.426094+0000", 2024-09-21T14:00:09.607 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5008000000000001e-05 2024-09-21T14:00:09.607 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:00:09.607 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:00:09.607 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:00:09.607 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.607 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.607 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:202104 7.7 7:e253bf76:::benchmark_data_smithi049_76798_object65684:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-21T14:00:09.607 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:44.217360+0000", 2024-09-21T14:00:09.607 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 445.34239847800001, 2024-09-21T14:00:09.608 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.15782971300000001, 2024-09-21T14:00:09.608 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:00:09.608 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:00:09.608 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:00:09.608 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:00:09.608 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T14:00:09.608 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T14:00:09.608 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 202104 2024-09-21T14:00:09.608 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.608 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:00:09.608 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.608 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:00:09.608 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.217360+0000", 2024-09-21T14:00:09.608 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:00:09.609 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.609 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.609 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:00:09.609 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.217360+0000", 2024-09-21T14:00:09.609 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:00:09.609 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.609 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.609 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:00:09.609 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.217361+0000", 2024-09-21T14:00:09.609 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.279e-06 2024-09-21T14:00:09.609 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.609 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.609 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:00:09.609 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.217367+0000", 2024-09-21T14:00:09.609 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2120000000000002e-06 2024-09-21T14:00:09.610 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.610 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.610 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:00:09.610 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.217379+0000", 2024-09-21T14:00:09.610 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2612000000000001e-05 2024-09-21T14:00:09.610 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.610 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.610 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:00:09.610 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.217387+0000", 2024-09-21T14:00:09.610 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8909999999999995e-06 2024-09-21T14:00:09.610 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.610 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.610 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:00:09.610 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.217429+0000", 2024-09-21T14:00:09.611 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1785000000000001e-05 2024-09-21T14:00:09.611 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.611 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.611 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:00:09.611 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.217506+0000", 2024-09-21T14:00:09.611 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7194000000000007e-05 2024-09-21T14:00:09.611 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.611 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.611 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3", 2024-09-21T14:00:09.611 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.340772+0000", 2024-09-21T14:00:09.611 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.12326546100000001 2024-09-21T14:00:09.611 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.611 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.612 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:00:09.612 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.344667+0000", 2024-09-21T14:00:09.612 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003894907 2024-09-21T14:00:09.612 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.612 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.612 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:09.612 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.369243+0000", 2024-09-21T14:00:09.612 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.024576745000000001 2024-09-21T14:00:09.612 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.612 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.612 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:09.612 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.375131+0000", 2024-09-21T14:00:09.612 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0058877240000000004 2024-09-21T14:00:09.612 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.613 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.613 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:00:09.613 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.375162+0000", 2024-09-21T14:00:09.613 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1198999999999997e-05 2024-09-21T14:00:09.613 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.613 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.613 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:00:09.613 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.375190+0000", 2024-09-21T14:00:09.613 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7704000000000001e-05 2024-09-21T14:00:09.613 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:00:09.613 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:00:09.613 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:00:09.613 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.613 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.613 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:29977 9.b 9.ed124ccb (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e269)", 2024-09-21T14:00:09.614 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:58:16.771091+0000", 2024-09-21T14:00:09.614 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 112.788667657, 2024-09-21T14:00:09.614 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.97926761900000003, 2024-09-21T14:00:09.614 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:00:09.614 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:00:09.614 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-21T14:00:09.614 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:00:09.614 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:00:09.614 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:00:09.614 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 29977 2024-09-21T14:00:09.614 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.614 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:00:09.614 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.614 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:00:09.615 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.771091+0000", 2024-09-21T14:00:09.615 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:00:09.615 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.615 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.615 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:00:09.615 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.771091+0000", 2024-09-21T14:00:09.615 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:00:09.615 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.615 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.615 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:00:09.615 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.771093+0000", 2024-09-21T14:00:09.615 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.606e-06 2024-09-21T14:00:09.615 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.615 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.616 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:00:09.616 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.771152+0000", 2024-09-21T14:00:09.616 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9827000000000002e-05 2024-09-21T14:00:09.616 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.616 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.616 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:00:09.616 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.771162+0000", 2024-09-21T14:00:09.616 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0010000000000001e-05 2024-09-21T14:00:09.616 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.616 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.616 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:00:09.616 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.771171+0000", 2024-09-21T14:00:09.616 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2970000000000004e-06 2024-09-21T14:00:09.616 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.616 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.617 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:00:09.617 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.771209+0000", 2024-09-21T14:00:09.617 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8182000000000001e-05 2024-09-21T14:00:09.617 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.617 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.617 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:00:09.617 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.771215+0000", 2024-09-21T14:00:09.617 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9599999999999997e-06 2024-09-21T14:00:09.617 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.617 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.617 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:00:09.617 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:17.750348+0000", 2024-09-21T14:00:09.617 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.97913300400000003 2024-09-21T14:00:09.617 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.617 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.618 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:00:09.618 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:17.750359+0000", 2024-09-21T14:00:09.618 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0733e-05 2024-09-21T14:00:09.618 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:00:09.618 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:00:09.618 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:00:09.618 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.618 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.618 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:29992 9.b 9.6daaec5b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-09-21T14:00:09.618 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:58:16.780680+0000", 2024-09-21T14:00:09.618 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 112.779078946, 2024-09-21T14:00:09.618 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.96971144600000003, 2024-09-21T14:00:09.618 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:00:09.618 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:00:09.619 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-21T14:00:09.619 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:00:09.619 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:00:09.619 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:00:09.619 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 29992 2024-09-21T14:00:09.619 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.619 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:00:09.619 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.619 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:00:09.619 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.780680+0000", 2024-09-21T14:00:09.619 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:00:09.619 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.619 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.619 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:00:09.619 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.780680+0000", 2024-09-21T14:00:09.620 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:00:09.620 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.620 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.620 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:00:09.620 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.780681+0000", 2024-09-21T14:00:09.620 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6509999999999999e-06 2024-09-21T14:00:09.620 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.620 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.620 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:00:09.620 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.780750+0000", 2024-09-21T14:00:09.620 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8570000000000002e-05 2024-09-21T14:00:09.620 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.620 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.620 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:00:09.620 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.780762+0000", 2024-09-21T14:00:09.621 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2109999999999999e-05 2024-09-21T14:00:09.621 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.621 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.621 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:00:09.621 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.780770+0000", 2024-09-21T14:00:09.621 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3690000000000007e-06 2024-09-21T14:00:09.621 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.621 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.621 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:00:09.621 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.780828+0000", 2024-09-21T14:00:09.621 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.749e-05 2024-09-21T14:00:09.621 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.621 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.622 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:00:09.622 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.780836+0000", 2024-09-21T14:00:09.622 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1219999999999995e-06 2024-09-21T14:00:09.622 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.622 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.622 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:00:09.622 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:17.750382+0000", 2024-09-21T14:00:09.622 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.96954600400000002 2024-09-21T14:00:09.622 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.622 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.622 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:00:09.622 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:17.750391+0000", 2024-09-21T14:00:09.622 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1300000000000007e-06 2024-09-21T14:00:09.623 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:00:09.623 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:00:09.623 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:00:09.623 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.623 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.623 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:29996 9.b 9.d0040c7b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-09-21T14:00:09.623 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:58:16.785408+0000", 2024-09-21T14:00:09.623 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 112.774350617, 2024-09-21T14:00:09.623 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.96553488099999996, 2024-09-21T14:00:09.623 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:00:09.623 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:00:09.623 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-21T14:00:09.623 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:00:09.624 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:00:09.624 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:00:09.624 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 29996 2024-09-21T14:00:09.624 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.624 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:00:09.624 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.624 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:00:09.624 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.785408+0000", 2024-09-21T14:00:09.624 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:00:09.624 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.624 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.624 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:00:09.624 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.785408+0000", 2024-09-21T14:00:09.624 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:00:09.625 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.625 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.625 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:00:09.625 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.785410+0000", 2024-09-21T14:00:09.625 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5030000000000001e-06 2024-09-21T14:00:09.625 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.625 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.625 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:00:09.625 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.785466+0000", 2024-09-21T14:00:09.625 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6938999999999998e-05 2024-09-21T14:00:09.625 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.625 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.625 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:00:09.625 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.785477+0000", 2024-09-21T14:00:09.626 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0543000000000001e-05 2024-09-21T14:00:09.626 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.626 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.626 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:00:09.626 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.785483+0000", 2024-09-21T14:00:09.626 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5999999999999997e-06 2024-09-21T14:00:09.626 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.626 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.626 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:00:09.626 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.785528+0000", 2024-09-21T14:00:09.626 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5286e-05 2024-09-21T14:00:09.626 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.626 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.627 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:00:09.627 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.785537+0000", 2024-09-21T14:00:09.627 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3600000000000002e-06 2024-09-21T14:00:09.627 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.627 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.627 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:00:09.627 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:17.750935+0000", 2024-09-21T14:00:09.627 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.96539792099999999 2024-09-21T14:00:09.627 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.627 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.627 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:00:09.627 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:17.750943+0000", 2024-09-21T14:00:09.627 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.729e-06 2024-09-21T14:00:09.628 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:00:09.628 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:00:09.628 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:00:09.628 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.628 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.628 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:30003 9.b 9.b7b5b7bb (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-09-21T14:00:09.628 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:58:16.789071+0000", 2024-09-21T14:00:09.628 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 112.770687897, 2024-09-21T14:00:09.628 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.96190145699999996, 2024-09-21T14:00:09.628 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:00:09.628 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:00:09.628 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-21T14:00:09.628 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:00:09.628 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:00:09.628 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:00:09.629 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 30003 2024-09-21T14:00:09.629 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.629 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:00:09.629 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.629 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:00:09.629 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.789071+0000", 2024-09-21T14:00:09.629 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:00:09.629 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.629 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.629 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:00:09.629 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.789071+0000", 2024-09-21T14:00:09.629 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:00:09.629 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.629 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.630 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:00:09.630 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.789074+0000", 2024-09-21T14:00:09.630 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8430000000000001e-06 2024-09-21T14:00:09.630 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.630 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.630 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:00:09.630 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.789182+0000", 2024-09-21T14:00:09.630 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000108019 2024-09-21T14:00:09.630 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.630 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.630 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:00:09.630 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.789210+0000", 2024-09-21T14:00:09.630 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8503000000000001e-05 2024-09-21T14:00:09.630 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.631 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.631 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:00:09.631 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.789226+0000", 2024-09-21T14:00:09.631 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5403000000000001e-05 2024-09-21T14:00:09.631 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.631 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.631 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:00:09.631 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.789287+0000", 2024-09-21T14:00:09.631 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1326000000000002e-05 2024-09-21T14:00:09.631 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.631 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.631 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:00:09.631 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.789293+0000", 2024-09-21T14:00:09.631 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3509999999999998e-06 2024-09-21T14:00:09.632 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.632 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.632 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:00:09.632 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:17.750965+0000", 2024-09-21T14:00:09.632 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.96167175199999999 2024-09-21T14:00:09.632 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.632 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.632 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:00:09.632 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:17.750972+0000", 2024-09-21T14:00:09.632 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2599999999999999e-06 2024-09-21T14:00:09.632 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:00:09.632 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:00:09.632 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:00:09.632 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.632 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.633 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:30009 9.b 9.13ae813b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-09-21T14:00:09.633 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:58:16.793372+0000", 2024-09-21T14:00:09.633 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 112.766386383, 2024-09-21T14:00:09.633 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.95778215200000005, 2024-09-21T14:00:09.633 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:00:09.633 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:00:09.633 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-21T14:00:09.633 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:00:09.633 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:00:09.633 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:00:09.633 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 30009 2024-09-21T14:00:09.633 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.633 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:00:09.633 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.633 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:00:09.634 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.793372+0000", 2024-09-21T14:00:09.634 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:00:09.634 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.634 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.634 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:00:09.634 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.793372+0000", 2024-09-21T14:00:09.634 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:00:09.634 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.634 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.634 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:00:09.634 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.793374+0000", 2024-09-21T14:00:09.634 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0190000000000001e-06 2024-09-21T14:00:09.634 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.635 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.635 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:00:09.635 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.793447+0000", 2024-09-21T14:00:09.635 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2768000000000005e-05 2024-09-21T14:00:09.635 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.635 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.635 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:00:09.635 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.793464+0000", 2024-09-21T14:00:09.635 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6612e-05 2024-09-21T14:00:09.635 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.635 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.635 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:00:09.635 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.793472+0000", 2024-09-21T14:00:09.635 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.276e-06 2024-09-21T14:00:09.636 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.636 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.636 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:00:09.636 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.793527+0000", 2024-09-21T14:00:09.636 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4752e-05 2024-09-21T14:00:09.636 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.636 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.636 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:00:09.636 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.793535+0000", 2024-09-21T14:00:09.636 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3890000000000005e-06 2024-09-21T14:00:09.636 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.636 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.636 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:00:09.636 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:17.751138+0000", 2024-09-21T14:00:09.636 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.95760284799999995 2024-09-21T14:00:09.637 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.637 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.637 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:00:09.637 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:17.751154+0000", 2024-09-21T14:00:09.637 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6487999999999999e-05 2024-09-21T14:00:09.637 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:00:09.637 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:00:09.637 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:00:09.637 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.637 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.637 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:30021 9.b 9.9be4c56b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-09-21T14:00:09.637 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:58:16.803723+0000", 2024-09-21T14:00:09.637 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 112.75603530799999, 2024-09-21T14:00:09.637 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.94748764900000004, 2024-09-21T14:00:09.637 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:00:09.638 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:00:09.638 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-21T14:00:09.638 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:00:09.638 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:00:09.638 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:00:09.638 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 30021 2024-09-21T14:00:09.638 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.638 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:00:09.638 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.638 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:00:09.638 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.803723+0000", 2024-09-21T14:00:09.638 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:00:09.638 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.638 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.639 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:00:09.639 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.803723+0000", 2024-09-21T14:00:09.639 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:00:09.639 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.639 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.639 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:00:09.639 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.803725+0000", 2024-09-21T14:00:09.639 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.973e-06 2024-09-21T14:00:09.639 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.639 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.639 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:00:09.639 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.803794+0000", 2024-09-21T14:00:09.639 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9092999999999997e-05 2024-09-21T14:00:09.639 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.639 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.640 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:00:09.640 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.803814+0000", 2024-09-21T14:00:09.640 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9618000000000001e-05 2024-09-21T14:00:09.640 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.640 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.640 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:00:09.640 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.803825+0000", 2024-09-21T14:00:09.640 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0788000000000001e-05 2024-09-21T14:00:09.640 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.640 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.640 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:00:09.640 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.803887+0000", 2024-09-21T14:00:09.640 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2618000000000003e-05 2024-09-21T14:00:09.640 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.640 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.641 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:00:09.641 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.803893+0000", 2024-09-21T14:00:09.641 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.874e-06 2024-09-21T14:00:09.641 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.641 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.641 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:00:09.641 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:17.751204+0000", 2024-09-21T14:00:09.641 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.94731062300000002 2024-09-21T14:00:09.641 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.641 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.641 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:00:09.641 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:17.751211+0000", 2024-09-21T14:00:09.641 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.0620000000000003e-06 2024-09-21T14:00:09.641 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:00:09.641 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:00:09.642 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:00:09.642 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.642 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.642 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:113894 9.b 9:ddec8d1f:::benchmark_data_smithi049_138504_object113893:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-21T14:00:09.642 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:59:25.585009+0000", 2024-09-21T14:00:09.642 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 43.974749246999998, 2024-09-21T14:00:09.642 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.35427235800000001, 2024-09-21T14:00:09.642 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:00:09.642 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:00:09.642 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:00:09.642 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:00:09.642 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:00:09.642 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:00:09.642 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 113894 2024-09-21T14:00:09.642 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.642 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:00:09.643 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.643 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:00:09.643 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.585009+0000", 2024-09-21T14:00:09.643 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:00:09.643 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.643 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.643 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:00:09.643 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.585009+0000", 2024-09-21T14:00:09.643 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:00:09.643 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.643 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.643 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:00:09.643 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.585011+0000", 2024-09-21T14:00:09.643 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1659999999999999e-06 2024-09-21T14:00:09.643 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.644 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.644 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:00:09.644 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.585112+0000", 2024-09-21T14:00:09.644 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000101773 2024-09-21T14:00:09.644 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.644 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.644 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:00:09.644 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.585130+0000", 2024-09-21T14:00:09.644 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7668000000000001e-05 2024-09-21T14:00:09.644 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.644 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.644 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:00:09.644 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.585138+0000", 2024-09-21T14:00:09.644 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0020000000000006e-06 2024-09-21T14:00:09.645 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.645 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.645 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:00:09.645 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.590959+0000", 2024-09-21T14:00:09.645 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0058207110000000001 2024-09-21T14:00:09.645 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:09.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:00:09.646 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.646 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.646 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:00:09.646 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.591076+0000", 2024-09-21T14:00:09.646 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000116903 2024-09-21T14:00:09.646 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.647 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.647 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,5", 2024-09-21T14:00:09.647 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.591201+0000", 2024-09-21T14:00:09.647 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012490099999999999 2024-09-21T14:00:09.647 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.647 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.647 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:00:09.647 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.593938+0000", 2024-09-21T14:00:09.647 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.002737644 2024-09-21T14:00:09.647 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.647 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.647 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:09.647 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.595211+0000", 2024-09-21T14:00:09.647 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00127315 2024-09-21T14:00:09.648 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.648 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.648 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:09.648 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.939188+0000", 2024-09-21T14:00:09.648 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.34397667799999998 2024-09-21T14:00:09.648 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.648 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.648 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:00:09.648 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.939242+0000", 2024-09-21T14:00:09.648 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3783999999999999e-05 2024-09-21T14:00:09.648 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.648 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.648 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:00:09.649 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.939282+0000", 2024-09-21T14:00:09.649 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9978000000000002e-05 2024-09-21T14:00:09.649 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:00:09.649 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:00:09.649 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:00:09.649 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.649 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.649 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:113922 9.b 9:db1a22c4:::benchmark_data_smithi049_138504_object113921:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-21T14:00:09.649 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:59:25.604822+0000", 2024-09-21T14:00:09.649 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 43.954937155000003, 2024-09-21T14:00:09.649 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.34532540099999998, 2024-09-21T14:00:09.649 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:00:09.649 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:00:09.649 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:00:09.650 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:00:09.650 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:00:09.650 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:00:09.650 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 113922 2024-09-21T14:00:09.650 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.650 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:00:09.650 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.650 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:00:09.650 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.604822+0000", 2024-09-21T14:00:09.650 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:00:09.650 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.650 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.650 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:00:09.650 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.604822+0000", 2024-09-21T14:00:09.651 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:00:09.651 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.651 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.651 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:00:09.651 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.604823+0000", 2024-09-21T14:00:09.651 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.871e-06 2024-09-21T14:00:09.651 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.651 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.651 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:00:09.651 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.604877+0000", 2024-09-21T14:00:09.651 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4112e-05 2024-09-21T14:00:09.651 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.651 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.651 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:00:09.651 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.604896+0000", 2024-09-21T14:00:09.652 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8569000000000001e-05 2024-09-21T14:00:09.652 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.652 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.652 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:00:09.652 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.604906+0000", 2024-09-21T14:00:09.652 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.8889999999999997e-06 2024-09-21T14:00:09.652 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.652 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.652 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:00:09.652 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.604971+0000", 2024-09-21T14:00:09.652 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5406000000000004e-05 2024-09-21T14:00:09.652 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.652 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.652 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:00:09.652 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.605106+0000", 2024-09-21T14:00:09.653 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013487199999999999 2024-09-21T14:00:09.653 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.653 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.653 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,5", 2024-09-21T14:00:09.653 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.605248+0000", 2024-09-21T14:00:09.653 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014186299999999999 2024-09-21T14:00:09.653 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.653 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.653 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:00:09.653 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.608440+0000", 2024-09-21T14:00:09.653 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0031917349999999998 2024-09-21T14:00:09.653 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.653 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.653 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:09.653 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.608968+0000", 2024-09-21T14:00:09.654 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00052824000000000002 2024-09-21T14:00:09.654 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.654 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.654 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:09.654 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.950032+0000", 2024-09-21T14:00:09.654 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.34106423200000002 2024-09-21T14:00:09.654 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.654 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.654 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:00:09.654 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.950093+0000", 2024-09-21T14:00:09.654 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1123000000000004e-05 2024-09-21T14:00:09.654 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.654 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.654 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:00:09.655 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.950147+0000", 2024-09-21T14:00:09.655 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3489000000000002e-05 2024-09-21T14:00:09.655 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:00:09.655 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:00:09.655 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:00:09.655 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.655 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.655 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:113929 9.b 9:d1fc3735:::benchmark_data_smithi049_138504_object113928:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-21T14:00:09.655 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:59:25.610482+0000", 2024-09-21T14:00:09.655 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 43.949276920999999, 2024-09-21T14:00:09.655 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.33987869100000001, 2024-09-21T14:00:09.655 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:00:09.655 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:00:09.655 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:00:09.655 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:00:09.655 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:00:09.656 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:00:09.656 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 113929 2024-09-21T14:00:09.656 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.656 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:00:09.656 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.656 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:00:09.656 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.610482+0000", 2024-09-21T14:00:09.656 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:00:09.656 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.656 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.656 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:00:09.656 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.610482+0000", 2024-09-21T14:00:09.656 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:00:09.656 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.657 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.657 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:00:09.657 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.610484+0000", 2024-09-21T14:00:09.657 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.069e-06 2024-09-21T14:00:09.657 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.657 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.657 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:00:09.657 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.610532+0000", 2024-09-21T14:00:09.657 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8508999999999999e-05 2024-09-21T14:00:09.657 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.657 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.657 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:00:09.657 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.610547+0000", 2024-09-21T14:00:09.657 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4443e-05 2024-09-21T14:00:09.657 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.658 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.658 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:00:09.658 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.610554+0000", 2024-09-21T14:00:09.658 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1420000000000004e-06 2024-09-21T14:00:09.658 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.658 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.658 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:00:09.658 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.610607+0000", 2024-09-21T14:00:09.658 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3111e-05 2024-09-21T14:00:09.658 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.658 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.658 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:00:09.658 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.610747+0000", 2024-09-21T14:00:09.658 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000139732 2024-09-21T14:00:09.658 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.659 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.659 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,5", 2024-09-21T14:00:09.659 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.610904+0000", 2024-09-21T14:00:09.659 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000157498 2024-09-21T14:00:09.659 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.659 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.659 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:00:09.659 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.614124+0000", 2024-09-21T14:00:09.659 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0032196059999999999 2024-09-21T14:00:09.659 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.659 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.659 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:09.659 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.615339+0000", 2024-09-21T14:00:09.659 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001215247 2024-09-21T14:00:09.660 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.660 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.660 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:09.660 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.950251+0000", 2024-09-21T14:00:09.660 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.33491227000000001 2024-09-21T14:00:09.660 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.660 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.660 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:00:09.660 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.950325+0000", 2024-09-21T14:00:09.660 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3919999999999997e-05 2024-09-21T14:00:09.660 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.660 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.660 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:00:09.660 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.950360+0000", 2024-09-21T14:00:09.661 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5144000000000001e-05 2024-09-21T14:00:09.661 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:00:09.661 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:00:09.661 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:00:09.661 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.661 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.661 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:113935 9.b 9:d6c67fb5:::benchmark_data_smithi049_138504_object113934:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-21T14:00:09.661 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:59:25.616010+0000", 2024-09-21T14:00:09.661 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 43.943748233999997, 2024-09-21T14:00:09.661 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.33448062299999998, 2024-09-21T14:00:09.661 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:00:09.661 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:00:09.661 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:00:09.661 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:00:09.661 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:00:09.662 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:00:09.662 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 113935 2024-09-21T14:00:09.662 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.662 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:00:09.662 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.662 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:00:09.662 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.616010+0000", 2024-09-21T14:00:09.662 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:00:09.662 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.662 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.662 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:00:09.662 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.616010+0000", 2024-09-21T14:00:09.662 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:00:09.662 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.663 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.663 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:00:09.663 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.616012+0000", 2024-09-21T14:00:09.663 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7090000000000001e-06 2024-09-21T14:00:09.663 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.663 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.663 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:00:09.663 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.616082+0000", 2024-09-21T14:00:09.663 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.0131999999999999e-05 2024-09-21T14:00:09.663 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.663 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.663 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:00:09.663 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.616097+0000", 2024-09-21T14:00:09.663 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4867e-05 2024-09-21T14:00:09.664 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.664 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.664 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:00:09.664 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.616104+0000", 2024-09-21T14:00:09.664 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1740000000000003e-06 2024-09-21T14:00:09.664 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.664 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.664 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:00:09.664 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.616165+0000", 2024-09-21T14:00:09.664 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0210000000000001e-05 2024-09-21T14:00:09.664 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.664 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.664 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:00:09.664 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.616298+0000", 2024-09-21T14:00:09.664 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000133137 2024-09-21T14:00:09.665 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.665 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.665 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,5", 2024-09-21T14:00:09.665 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.616443+0000", 2024-09-21T14:00:09.665 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000145408 2024-09-21T14:00:09.665 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.665 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.665 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:09.665 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.623090+0000", 2024-09-21T14:00:09.665 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0066470319999999998 2024-09-21T14:00:09.665 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.665 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.665 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:00:09.665 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.623155+0000", 2024-09-21T14:00:09.665 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4774e-05 2024-09-21T14:00:09.666 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.666 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.666 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:09.666 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.950407+0000", 2024-09-21T14:00:09.666 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.32725225899999999 2024-09-21T14:00:09.666 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.666 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.666 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:00:09.666 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.950457+0000", 2024-09-21T14:00:09.666 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9574999999999999e-05 2024-09-21T14:00:09.666 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.666 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.666 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:00:09.666 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.950491+0000", 2024-09-21T14:00:09.667 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4345999999999999e-05 2024-09-21T14:00:09.667 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:00:09.667 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:00:09.667 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:00:09.667 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.667 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.667 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:113936 9.b 9:dbd56865:::benchmark_data_smithi049_138504_object113935:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-21T14:00:09.667 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:59:25.616122+0000", 2024-09-21T14:00:09.667 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 43.943636830000003, 2024-09-21T14:00:09.667 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.33836170500000001, 2024-09-21T14:00:09.667 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:00:09.667 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:00:09.667 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:00:09.667 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:00:09.667 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:00:09.668 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:00:09.668 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 113936 2024-09-21T14:00:09.668 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.668 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:00:09.668 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.668 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:00:09.668 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.616122+0000", 2024-09-21T14:00:09.668 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:00:09.668 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.668 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.668 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:00:09.668 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.616122+0000", 2024-09-21T14:00:09.668 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:00:09.668 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.668 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.669 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:00:09.669 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.616122+0000", 2024-09-21T14:00:09.669 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1999999999999999e-07 2024-09-21T14:00:09.669 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.669 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.669 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:00:09.669 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.616151+0000", 2024-09-21T14:00:09.669 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8578e-05 2024-09-21T14:00:09.669 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.669 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.669 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:00:09.669 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.616158+0000", 2024-09-21T14:00:09.669 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.0439999999999998e-06 2024-09-21T14:00:09.669 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.669 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.670 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:00:09.670 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.616163+0000", 2024-09-21T14:00:09.670 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0849999999999996e-06 2024-09-21T14:00:09.670 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.670 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.670 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:00:09.670 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.619427+0000", 2024-09-21T14:00:09.670 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0032643390000000002 2024-09-21T14:00:09.670 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.670 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.670 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:00:09.670 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.619565+0000", 2024-09-21T14:00:09.670 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013736299999999999 2024-09-21T14:00:09.671 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.671 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.671 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,5", 2024-09-21T14:00:09.671 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.619724+0000", 2024-09-21T14:00:09.671 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000159536 2024-09-21T14:00:09.671 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.671 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.671 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:09.671 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.623143+0000", 2024-09-21T14:00:09.671 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0034181369999999999 2024-09-21T14:00:09.671 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.671 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.671 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:00:09.671 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.623159+0000", 2024-09-21T14:00:09.671 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6756999999999999e-05 2024-09-21T14:00:09.671 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.672 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.672 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:09.672 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.954404+0000", 2024-09-21T14:00:09.672 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.33124505199999998 2024-09-21T14:00:09.672 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.672 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.672 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:00:09.672 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.954442+0000", 2024-09-21T14:00:09.672 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7583999999999998e-05 2024-09-21T14:00:09.672 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.672 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.672 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:00:09.672 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.954484+0000", 2024-09-21T14:00:09.672 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1610000000000003e-05 2024-09-21T14:00:09.673 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:00:09.673 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:00:09.673 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:00:09.673 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.673 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.673 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:113957 9.b 9:de61005e:::benchmark_data_smithi049_138504_object113956:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-21T14:00:09.673 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:59:25.651274+0000", 2024-09-21T14:00:09.673 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 43.90848475, 2024-09-21T14:00:09.673 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.30796400600000001, 2024-09-21T14:00:09.673 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:00:09.673 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:00:09.673 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:00:09.673 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:00:09.673 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:00:09.674 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:00:09.674 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 113957 2024-09-21T14:00:09.674 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.674 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:00:09.674 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.674 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:00:09.674 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.651274+0000", 2024-09-21T14:00:09.674 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:00:09.674 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.674 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.674 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:00:09.674 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.651274+0000", 2024-09-21T14:00:09.674 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:00:09.674 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.675 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.675 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:00:09.675 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.651277+0000", 2024-09-21T14:00:09.675 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.061e-06 2024-09-21T14:00:09.675 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.675 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.675 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:00:09.675 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.651311+0000", 2024-09-21T14:00:09.675 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4131000000000002e-05 2024-09-21T14:00:09.675 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.675 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.675 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:00:09.675 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.651333+0000", 2024-09-21T14:00:09.675 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2393999999999999e-05 2024-09-21T14:00:09.675 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.675 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.676 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:00:09.676 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.651343+0000", 2024-09-21T14:00:09.676 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7729999999999994e-06 2024-09-21T14:00:09.676 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.676 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.676 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:00:09.676 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.651399+0000", 2024-09-21T14:00:09.676 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5911000000000001e-05 2024-09-21T14:00:09.676 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.676 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.676 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:00:09.676 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.651506+0000", 2024-09-21T14:00:09.676 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000107148 2024-09-21T14:00:09.676 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.677 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.677 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,5", 2024-09-21T14:00:09.677 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.651609+0000", 2024-09-21T14:00:09.677 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000103063 2024-09-21T14:00:09.677 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.677 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.677 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:00:09.677 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.654350+0000", 2024-09-21T14:00:09.677 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0027411079999999999 2024-09-21T14:00:09.677 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.677 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.677 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:09.677 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.655715+0000", 2024-09-21T14:00:09.677 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0013649790000000001 2024-09-21T14:00:09.678 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.678 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.678 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:09.678 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.959170+0000", 2024-09-21T14:00:09.678 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.30345403700000001 2024-09-21T14:00:09.678 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.678 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.678 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:00:09.678 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.959208+0000", 2024-09-21T14:00:09.678 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8757000000000002e-05 2024-09-21T14:00:09.678 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:00:09.678 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:00:09.678 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:00:09.678 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.959238+0000", 2024-09-21T14:00:09.678 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9643999999999999e-05 2024-09-21T14:00:09.679 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:00:09.679 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:00:09.679 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:00:09.679 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:00:09.679 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:00:09.679 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T14:00:09.679 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:00:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:00:09.746 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:09.792 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:09.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:09.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:09.894 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:00:09.934 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:09.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:09.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:00:10.036 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:10.073 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:10.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:10.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:10.175 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:00:10.213 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:10.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:10.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:00:10.265 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 4, 5, 6, 2, 3, 7] 2024-09-21T14:00:10.265 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T14:00:10.265 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-21T14:00:10.265 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T14:00:10.315 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:10.361 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:10.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:10.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:10.410 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:10 smithi049 ceph-mon[40489]: 4.1f scrub starts 2024-09-21T14:00:10.410 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:10 smithi049 ceph-mon[40489]: 4.1f scrub ok 2024-09-21T14:00:10.410 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:10 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:00:10.410 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:10 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 41814/290280 objects degraded (14.405%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T14:00:10.410 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:10 smithi049 ceph-mon[30080]: 4.1f scrub starts 2024-09-21T14:00:10.410 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:10 smithi049 ceph-mon[30080]: 4.1f scrub ok 2024-09-21T14:00:10.411 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:10 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:00:10.411 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:10 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 41814/290280 objects degraded (14.405%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T14:00:10.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:10 smithi187 ceph-mon[33032]: 4.1f scrub starts 2024-09-21T14:00:10.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:10 smithi187 ceph-mon[33032]: 4.1f scrub ok 2024-09-21T14:00:10.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:10 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:00:10.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:10 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 41814/290280 objects degraded (14.405%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T14:00:10.467 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:10.534 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:10.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:10.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:10.623 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T14:00:10.636 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:00:10.664 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:00:10 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:00:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:00:10.667 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:10.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:10.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:00:10.768 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:10.790 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:10.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:10.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:10.891 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:00:10.925 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:10.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:10.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:00:11.027 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:00:11.057 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:11.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:11.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:00:11.159 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:00:11.193 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:11.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:11.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:00:11.294 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:11.318 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:00:11.351 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:11.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:11.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:11.444 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 14:00:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=cleanup t=2024-09-21T14:00:10.995081603Z level=info msg="Completed cleanup jobs" duration=23.168513ms 2024-09-21T14:00:11.445 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 14:00:11 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=plugins.update.checker t=2024-09-21T14:00:11.116158098Z level=info msg="Update check succeeded" duration=71.334872ms 2024-09-21T14:00:11.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:11 smithi187 ceph-mon[33032]: pgmap v560: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 139 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.03k op/s; 39425/279234 objects degraded (14.119%); 5323/279234 objects misplaced (1.906%); 16 MiB/s, 248 objects/s recovering 2024-09-21T14:00:11.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:11 smithi187 ceph-mon[33032]: 5.a scrub starts 2024-09-21T14:00:11.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:11 smithi187 ceph-mon[33032]: 5.a scrub ok 2024-09-21T14:00:11.452 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:00:11.461 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:11 smithi049 ceph-mon[30080]: pgmap v560: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 139 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.03k op/s; 39425/279234 objects degraded (14.119%); 5323/279234 objects misplaced (1.906%); 16 MiB/s, 248 objects/s recovering 2024-09-21T14:00:11.461 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:11 smithi049 ceph-mon[30080]: 5.a scrub starts 2024-09-21T14:00:11.462 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:11 smithi049 ceph-mon[30080]: 5.a scrub ok 2024-09-21T14:00:11.462 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:11 smithi049 ceph-mon[40489]: pgmap v560: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait, 3 active+recovery_wait+undersized+degraded+remapped, 139 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.03k op/s; 39425/279234 objects degraded (14.119%); 5323/279234 objects misplaced (1.906%); 16 MiB/s, 248 objects/s recovering 2024-09-21T14:00:11.462 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:11 smithi049 ceph-mon[40489]: 5.a scrub starts 2024-09-21T14:00:11.462 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:11 smithi049 ceph-mon[40489]: 5.a scrub ok 2024-09-21T14:00:11.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:11.485 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:11.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:00:11.585 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:11.612 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:11.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:11.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:11.713 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:11.749 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:11.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:11.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:11.851 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:11.885 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:11.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:11.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:11.987 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:12.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:12.026 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:12.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:12.127 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:12.145 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-09-21T14:00:12.178 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:12.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:12.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:12.280 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:00:12.378 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:12.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:12.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:00:12.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:12 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/913153466' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:00:12.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:12 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:00:12.480 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:00:12.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:12 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/913153466' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:00:12.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:12 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:00:12.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:12 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/913153466' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:00:12.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:12 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:00:12.517 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:12.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:12.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:00:12.618 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:00:12.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:12.664 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:12.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:00:12.742 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:00:12.742 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:00:12.765 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:00:12.789 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:00:12.789 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:00:12.797 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:12.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:12.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:00:12.799 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:00:12.824 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:00:12.824 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:00:12.868 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:00:12.868 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:00:12.899 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:00:12.904 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:00:12.904 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:00:12.933 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:12.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:12.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:00:12.939 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:00:12.939 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:00:12.975 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:00:12.975 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:00:13.009 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:00:13.009 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:00:13.035 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:13.073 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:13.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:13.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:13.175 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:00:13.212 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:13.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:13.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:00:13.314 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:00:13.348 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:13.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:13.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:00:13.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:13 smithi187 ceph-mon[33032]: pgmap v561: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 3 active+recovery_wait+undersized+degraded+remapped, 139 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 956 op/s; 39138/275673 objects degraded (14.197%); 5053/275673 objects misplaced (1.833%); 13 MiB/s, 201 objects/s recovering 2024-09-21T14:00:13.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:13 smithi187 ceph-mon[33032]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:00:13.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:13 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T14:00:13.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:13 smithi187 ceph-mon[33032]: osdmap e311: 8 total, 8 up, 8 in 2024-09-21T14:00:13.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:13 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1979791519' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:00:13.450 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:00:13.482 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:13.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:13.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:00:13.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:13 smithi049 ceph-mon[30080]: pgmap v561: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 3 active+recovery_wait+undersized+degraded+remapped, 139 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 956 op/s; 39138/275673 objects degraded (14.197%); 5053/275673 objects misplaced (1.833%); 13 MiB/s, 201 objects/s recovering 2024-09-21T14:00:13.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:13 smithi049 ceph-mon[30080]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:00:13.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:13 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T14:00:13.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:13 smithi049 ceph-mon[30080]: osdmap e311: 8 total, 8 up, 8 in 2024-09-21T14:00:13.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:13 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1979791519' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:00:13.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:13 smithi049 ceph-mon[40489]: pgmap v561: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 3 active+recovery_wait+undersized+degraded+remapped, 139 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 956 op/s; 39138/275673 objects degraded (14.197%); 5053/275673 objects misplaced (1.833%); 13 MiB/s, 201 objects/s recovering 2024-09-21T14:00:13.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:13 smithi049 ceph-mon[40489]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:00:13.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:13 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T14:00:13.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:13 smithi049 ceph-mon[40489]: osdmap e311: 8 total, 8 up, 8 in 2024-09-21T14:00:13.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:13 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1979791519' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:00:13.584 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:13.613 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:13.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:13.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:13.714 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:00:13.753 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:13.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:13.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:00:13.854 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:13.892 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:13.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:13.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:13.994 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:14.023 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:14.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:14.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:14.125 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:00:14.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:14.173 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:14.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:00:14.274 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:00:14.304 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:14.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:14.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:00:14.406 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:00:14.448 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:14 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:00:14.448 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:14 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T14:00:14.454 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:14.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:14.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:00:14.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:14 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:00:14.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:14 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T14:00:14.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:14 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:00:14.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:14 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T14:00:14.556 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:00:14.601 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:14.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:14.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:00:14.703 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:00:14.736 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:14.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:14.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:00:14.838 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:14.880 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:14.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:14.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:14.982 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:15.002 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:15.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:15.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:15.104 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:15.142 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:15.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:15.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:15.243 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:15.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:15 smithi049 ceph-mon[40489]: pgmap v563: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 3 active+recovery_wait+undersized+degraded+remapped, 139 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 965 op/s; 39138/270216 objects degraded (14.484%); 5053/270216 objects misplaced (1.870%); 11 MiB/s, 180 objects/s recovering 2024-09-21T14:00:15.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:15 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T14:00:15.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:15 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T14:00:15.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:15 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T14:00:15.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:15 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T14:00:15.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:15 smithi049 ceph-mon[40489]: Adjusting osd_memory_target on smithi187 to 3248M 2024-09-21T14:00:15.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:15 smithi049 ceph-mon[30080]: pgmap v563: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 3 active+recovery_wait+undersized+degraded+remapped, 139 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 965 op/s; 39138/270216 objects degraded (14.484%); 5053/270216 objects misplaced (1.870%); 11 MiB/s, 180 objects/s recovering 2024-09-21T14:00:15.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:15 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T14:00:15.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:15 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T14:00:15.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:15 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T14:00:15.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:15 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T14:00:15.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:15 smithi049 ceph-mon[30080]: Adjusting osd_memory_target on smithi187 to 3248M 2024-09-21T14:00:15.265 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:15.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:15.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:15.367 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:00:15.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:15.406 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:15.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:00:15.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:15 smithi187 ceph-mon[33032]: pgmap v563: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 3 active+recovery_wait+undersized+degraded+remapped, 139 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 965 op/s; 39138/270216 objects degraded (14.484%); 5053/270216 objects misplaced (1.870%); 11 MiB/s, 180 objects/s recovering 2024-09-21T14:00:15.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:15 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T14:00:15.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:15 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T14:00:15.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:15 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T14:00:15.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:15 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T14:00:15.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:15 smithi187 ceph-mon[33032]: Adjusting osd_memory_target on smithi187 to 3248M 2024-09-21T14:00:15.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:15 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:00:15.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:15 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 39138/270216 objects degraded (14.484%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T14:00:15.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:15 smithi187 ceph-mon[33032]: osdmap e312: 8 total, 8 up, 8 in 2024-09-21T14:00:15.507 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:15.546 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:15.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:15.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:15.648 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:00:15.679 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:15.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:15.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:00:15.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:15 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:00:15.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:15 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 39138/270216 objects degraded (14.484%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T14:00:15.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:15 smithi049 ceph-mon[40489]: osdmap e312: 8 total, 8 up, 8 in 2024-09-21T14:00:15.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:15 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:00:15.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:15 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 39138/270216 objects degraded (14.484%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T14:00:15.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:15 smithi049 ceph-mon[30080]: osdmap e312: 8 total, 8 up, 8 in 2024-09-21T14:00:15.780 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:15.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:15.816 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:15.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:15.917 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:15.938 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:15.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:15.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:16.040 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:16.075 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:16.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:16.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:16.177 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:16.220 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:16.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:16.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:16.323 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:16.347 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:16.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:16.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:16.449 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:00:16.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:16.482 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:16.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:00:16.584 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:16.623 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:16.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:16.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:16.725 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:16.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:16.752 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:16.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:16.853 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:16.890 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:16.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:16.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:16.992 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:00:17.031 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:17.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:17.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:00:17.133 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:00:17.162 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:17.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:17.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:00:17.264 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:17.300 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:17.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:17.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:17.402 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:17.435 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:17.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:17.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:17.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:17 smithi187 ceph-mon[33032]: pgmap v565: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovery_wait+undersized+degraded+remapped, 139 active+clean; 5.4 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s; 36885/265218 objects degraded (13.907%); 5053/265218 objects misplaced (1.905%); 22 MiB/s, 357 objects/s recovering 2024-09-21T14:00:17.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:17 smithi049 ceph-mon[40489]: pgmap v565: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovery_wait+undersized+degraded+remapped, 139 active+clean; 5.4 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s; 36885/265218 objects degraded (13.907%); 5053/265218 objects misplaced (1.905%); 22 MiB/s, 357 objects/s recovering 2024-09-21T14:00:17.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:17 smithi049 ceph-mon[30080]: pgmap v565: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovery_wait+undersized+degraded+remapped, 139 active+clean; 5.4 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s; 36885/265218 objects degraded (13.907%); 5053/265218 objects misplaced (1.905%); 22 MiB/s, 357 objects/s recovering 2024-09-21T14:00:17.537 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:00:17.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:17.572 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:17.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:00:17.673 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:17.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:17.718 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:17.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:17.819 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:00:17.853 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:17.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:17.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:00:17.906 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T14:00:17.956 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:17.981 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:17.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:17.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:18.082 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:00:18.119 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:18.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:18.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:00:18.221 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:18.246 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:00:18.261 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:18.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:18.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:18.363 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:00:18.386 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:18.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:18.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:00:18.488 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:18.527 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:18.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:18.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:18.629 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:18.663 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:18.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:18.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:18.765 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:18.801 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:18.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:18.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:18.903 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:00:18.940 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:18.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:18.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:00:19.043 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:19.075 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:19.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:19.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:19.177 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:19.218 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:19.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:19.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:19.320 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:19.353 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:19.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:19.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:19.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:19 smithi187 ceph-mon[33032]: pgmap v566: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 3 active+recovery_wait+undersized+degraded+remapped, 139 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 721 op/s; 36635/261942 objects degraded (13.986%); 4825/261942 objects misplaced (1.842%); 12 MiB/s, 184 objects/s recovering 2024-09-21T14:00:19.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:19 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T14:00:19.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:19 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T14:00:19.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:19 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T14:00:19.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:19 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T14:00:19.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:19 smithi187 ceph-mon[33032]: Adjusting osd_memory_target on smithi049 to 3248M 2024-09-21T14:00:19.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:19 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:00:19.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:19 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T14:00:19.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:19 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T14:00:19.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:19 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:00:19.455 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:19.473 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:19 smithi049 ceph-mon[40489]: pgmap v566: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 3 active+recovery_wait+undersized+degraded+remapped, 139 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 721 op/s; 36635/261942 objects degraded (13.986%); 4825/261942 objects misplaced (1.842%); 12 MiB/s, 184 objects/s recovering 2024-09-21T14:00:19.473 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:19 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T14:00:19.473 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:19 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T14:00:19.473 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:19 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T14:00:19.473 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:19 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T14:00:19.474 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:19 smithi049 ceph-mon[40489]: Adjusting osd_memory_target on smithi049 to 3248M 2024-09-21T14:00:19.474 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:19 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:00:19.474 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:19 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T14:00:19.474 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:19 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T14:00:19.474 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:19 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:00:19.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:19 smithi049 ceph-mon[30080]: pgmap v566: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 3 active+recovery_wait+undersized+degraded+remapped, 139 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 721 op/s; 36635/261942 objects degraded (13.986%); 4825/261942 objects misplaced (1.842%); 12 MiB/s, 184 objects/s recovering 2024-09-21T14:00:19.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:19 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T14:00:19.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:19 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T14:00:19.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:19 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T14:00:19.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:19 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T14:00:19.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:19 smithi049 ceph-mon[30080]: Adjusting osd_memory_target on smithi049 to 3248M 2024-09-21T14:00:19.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:19 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:00:19.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:19 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T14:00:19.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:19 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T14:00:19.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:19 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:00:19.494 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:19.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:19.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:19.596 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:19.630 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:19.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:19.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:19.659 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T14:00:19.659 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:00:19.732 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:19.767 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:19.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:19.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:19.869 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:00:19.909 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:19.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:19.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:00:19.944 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:00:19 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:00:20.010 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:20.046 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:20.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:20.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:20.148 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:00:20.179 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:20.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:20.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:00:20.280 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:00:20.318 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:20.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:20.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:00:20.420 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:00:20.455 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:20.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:20.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:00:20.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:20 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:00:20.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:20 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 36635/261942 objects degraded (13.986%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T14:00:20.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:20 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:00:20.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:20 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 36635/261942 objects degraded (13.986%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T14:00:20.557 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:20.591 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:20.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:20.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:20.693 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:00:20.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:20 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:00:20.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:20 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 36635/261942 objects degraded (13.986%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T14:00:20.734 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:20.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:20.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:00:20.836 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:20.869 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:00:20 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:00:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:00:20.872 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:20.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:20.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:20.974 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:21.010 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:21.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:21.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:21.112 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:21.149 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:21.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:21.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:21.251 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:21.290 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:21.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:21.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:21.374 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T14:00:21.380 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-09-21T14:00:21.392 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:00:21.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:21.424 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:21.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:00:21.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:21 smithi049 ceph-mon[40489]: pgmap v567: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovery_wait+undersized+degraded+remapped, 139 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 940 op/s; 34675/253098 objects degraded (13.700%); 4825/253098 objects misplaced (1.906%); 22 MiB/s, 356 objects/s recovering 2024-09-21T14:00:21.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:21 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:00:21.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:21 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:00:21.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:21 smithi049 ceph-mon[30080]: pgmap v567: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovery_wait+undersized+degraded+remapped, 139 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 940 op/s; 34675/253098 objects degraded (13.700%); 4825/253098 objects misplaced (1.906%); 22 MiB/s, 356 objects/s recovering 2024-09-21T14:00:21.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:21 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:00:21.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:21 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:00:21.525 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:00:21.561 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:21.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:21.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:00:21.600 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T14:00:21.663 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:00:21.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:21 smithi187 ceph-mon[33032]: pgmap v567: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovery_wait+undersized+degraded+remapped, 139 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 940 op/s; 34675/253098 objects degraded (13.700%); 4825/253098 objects misplaced (1.906%); 22 MiB/s, 356 objects/s recovering 2024-09-21T14:00:21.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:21 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:00:21.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:21 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:00:21.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:21.697 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:21.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:00:21.798 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:21.802 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:00:21.859 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:21.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:21.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:21.961 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:22.000 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:22.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:22.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:22.102 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:22.134 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:22.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:22.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:22.236 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:00:22.270 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T14:00:22.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:22.275 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:22.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:00:22.376 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:00:22.425 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":568,"stamp":"2024-09-21T14:00:21.236054+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5443623475,"num_objects":83254,"num_object_clones":0,"num_object_copies":249762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34407,"num_objects_misplaced":4581,"num_objects_unfound":0,"num_objects_dirty":83254,"num_whiteouts":0,"num_read":1289,"num_read_kb":1148,"num_write":160592,"num_write_kb":7771594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14055,"num_bytes_recovered":909707712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82253,"ondisk_log_size":82253,"up":444,"acting":436,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":445,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":17541988,"kb_used_data":14519972,"kb_used_omap":20,"kb_used_meta":3021675,"kb_avail":732419228,"statfs":{"total":767960285184,"available":749997289472,"internally_reserved":0,"allocated":14868451328,"data_stored":14863725145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21298,"internal_metadata":3094195406},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-566034432,"num_objects":-8637,"num_object_clones":0,"num_object_copies":-25911,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4731,"num_objects_misplaced":-472,"num_objects_unfound":0,"num_objects_dirty":-8637,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":8637,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2849,"num_bytes_recovered":186712064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.091805"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":477,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.062393+0000","last_change":"2024-09-21T13:54:55.873382+0000","last_active":"2024-09-21T14:00:15.062393+0000","last_peered":"2024-09-21T14:00:15.062393+0000","last_clean":"2024-09-21T14:00:15.062393+0000","last_became_active":"2024-09-21T13:52:04.317322+0000","last_became_peered":"2024-09-21T13:52:04.317322+0000","last_unstale":"2024-09-21T14:00:15.062393+0000","last_undegraded":"2024-09-21T14:00:15.062393+0000","last_fullsized":"2024-09-21T14:00:15.062393+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:55.873301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:55.873301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"290'15","reported_seq":606,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.088961+0000","last_change":"2024-09-21T13:58:57.766087+0000","last_active":"2024-09-21T14:00:15.088961+0000","last_peered":"2024-09-21T14:00:15.088961+0000","last_clean":"2024-09-21T14:00:15.088961+0000","last_became_active":"2024-09-21T13:58:17.771220+0000","last_became_peered":"2024-09-21T13:58:17.771220+0000","last_unstale":"2024-09-21T14:00:15.088961+0000","last_undegraded":"2024-09-21T14:00:15.088961+0000","last_fullsized":"2024-09-21T14:00:15.088961+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:58:57.766011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:57.766011+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:22.570440+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":536,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.055940+0000","last_change":"2024-09-21T13:59:32.201875+0000","last_active":"2024-09-21T14:00:15.055940+0000","last_peered":"2024-09-21T14:00:15.055940+0000","last_clean":"2024-09-21T14:00:15.055940+0000","last_became_active":"2024-09-21T13:59:32.201538+0000","last_became_peered":"2024-09-21T13:59:32.201538+0000","last_unstale":"2024-09-21T14:00:15.055940+0000","last_undegraded":"2024-09-21T14:00:15.055940+0000","last_fullsized":"2024-09-21T14:00:15.055940+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:40.452724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:40.452724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":520,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.090155+0000","last_change":"2024-09-21T13:57:00.320623+0000","last_active":"2024-09-21T14:00:15.090155+0000","last_peered":"2024-09-21T14:00:15.090155+0000","last_clean":"2024-09-21T14:00:15.090155+0000","last_became_active":"2024-09-21T13:49:04.072460+0000","last_became_peered":"2024-09-21T13:49:04.072460+0000","last_unstale":"2024-09-21T14:00:15.090155+0000","last_undegraded":"2024-09-21T14:00:15.090155+0000","last_fullsized":"2024-09-21T14:00:15.090155+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:00.320510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:00.320510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:49.358812+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":501,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.061149+0000","last_change":"2024-09-21T13:59:32.208701+0000","last_active":"2024-09-21T14:00:15.061149+0000","last_peered":"2024-09-21T14:00:15.061149+0000","last_clean":"2024-09-21T14:00:15.061149+0000","last_became_active":"2024-09-21T13:59:32.203340+0000","last_became_peered":"2024-09-21T13:59:32.203340+0000","last_unstale":"2024-09-21T14:00:15.061149+0000","last_undegraded":"2024-09-21T14:00:15.061149+0000","last_fullsized":"2024-09-21T14:00:15.061149+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:34.696780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:34.696780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":513,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.061381+0000","last_change":"2024-09-21T13:58:17.778322+0000","last_active":"2024-09-21T14:00:15.061381+0000","last_peered":"2024-09-21T14:00:15.061381+0000","last_clean":"2024-09-21T14:00:15.061381+0000","last_became_active":"2024-09-21T13:58:17.777893+0000","last_became_peered":"2024-09-21T13:58:17.777893+0000","last_unstale":"2024-09-21T14:00:15.061381+0000","last_undegraded":"2024-09-21T14:00:15.061381+0000","last_fullsized":"2024-09-21T14:00:15.061381+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:07.001135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:55:07.001135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:18.028308+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"290'27","reported_seq":487,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.061377+0000","last_change":"2024-09-21T13:58:11.718421+0000","last_active":"2024-09-21T14:00:15.061377+0000","last_peered":"2024-09-21T14:00:15.061377+0000","last_clean":"2024-09-21T14:00:15.061377+0000","last_became_active":"2024-09-21T13:57:12.703469+0000","last_became_peered":"2024-09-21T13:57:12.703469+0000","last_unstale":"2024-09-21T14:00:15.061377+0000","last_undegraded":"2024-09-21T14:00:15.061377+0000","last_fullsized":"2024-09-21T14:00:15.061377+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:58:11.718334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:11.718334+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":546,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.088951+0000","last_change":"2024-09-21T13:58:53.761124+0000","last_active":"2024-09-21T14:00:15.088951+0000","last_peered":"2024-09-21T14:00:15.088951+0000","last_clean":"2024-09-21T14:00:15.088951+0000","last_became_active":"2024-09-21T13:58:17.773464+0000","last_became_peered":"2024-09-21T13:58:17.773464+0000","last_unstale":"2024-09-21T14:00:15.088951+0000","last_undegraded":"2024-09-21T14:00:15.088951+0000","last_fullsized":"2024-09-21T14:00:15.088951+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:53.761061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:53.761061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":563,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.052974+0000","last_change":"2024-09-21T13:58:17.782330+0000","last_active":"2024-09-21T14:00:15.052974+0000","last_peered":"2024-09-21T14:00:15.052974+0000","last_clean":"2024-09-21T14:00:15.052974+0000","last_became_active":"2024-09-21T13:58:17.781904+0000","last_became_peered":"2024-09-21T13:58:17.781904+0000","last_unstale":"2024-09-21T14:00:15.052974+0000","last_undegraded":"2024-09-21T14:00:15.052974+0000","last_fullsized":"2024-09-21T14:00:15.052974+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:31.127926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:31.127926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"290'15","reported_seq":580,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.089409+0000","last_change":"2024-09-21T14:00:02.475761+0000","last_active":"2024-09-21T14:00:15.089409+0000","last_peered":"2024-09-21T14:00:15.089409+0000","last_clean":"2024-09-21T14:00:15.089409+0000","last_became_active":"2024-09-21T13:57:12.681408+0000","last_became_peered":"2024-09-21T13:57:12.681408+0000","last_unstale":"2024-09-21T14:00:15.089409+0000","last_undegraded":"2024-09-21T14:00:15.089409+0000","last_fullsized":"2024-09-21T14:00:15.089409+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'15","last_scrub_stamp":"2024-09-21T14:00:02.475661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:00:02.475661+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:01:20.058343+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":528,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.057169+0000","last_change":"2024-09-21T13:59:32.213187+0000","last_active":"2024-09-21T14:00:15.057169+0000","last_peered":"2024-09-21T14:00:15.057169+0000","last_clean":"2024-09-21T14:00:15.057169+0000","last_became_active":"2024-09-21T13:59:32.212831+0000","last_became_peered":"2024-09-21T13:59:32.212831+0000","last_unstale":"2024-09-21T14:00:15.057169+0000","last_undegraded":"2024-09-21T14:00:15.057169+0000","last_fullsized":"2024-09-21T14:00:15.057169+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:21.284048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:56:21.284048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":485,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.057011+0000","last_change":"2024-09-21T13:59:32.202128+0000","last_active":"2024-09-21T14:00:15.057011+0000","last_peered":"2024-09-21T14:00:15.057011+0000","last_clean":"2024-09-21T14:00:15.057011+0000","last_became_active":"2024-09-21T13:59:32.201158+0000","last_became_peered":"2024-09-21T13:59:32.201158+0000","last_unstale":"2024-09-21T14:00:15.057011+0000","last_undegraded":"2024-09-21T14:00:15.057011+0000","last_fullsized":"2024-09-21T14:00:15.057011+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:25.792985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:54:25.792985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":533,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.052545+0000","last_change":"2024-09-21T13:56:54.328338+0000","last_active":"2024-09-21T14:00:15.052545+0000","last_peered":"2024-09-21T14:00:15.052545+0000","last_clean":"2024-09-21T14:00:15.052545+0000","last_became_active":"2024-09-21T13:48:58.050906+0000","last_became_peered":"2024-09-21T13:48:58.050906+0000","last_unstale":"2024-09-21T14:00:15.052545+0000","last_undegraded":"2024-09-21T14:00:15.052545+0000","last_fullsized":"2024-09-21T14:00:15.052545+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:54.328275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:54.328275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"290'7","reported_seq":597,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.089214+0000","last_change":"2024-09-21T14:00:01.507203+0000","last_active":"2024-09-21T14:00:15.089214+0000","last_peered":"2024-09-21T14:00:15.089214+0000","last_clean":"2024-09-21T14:00:15.089214+0000","last_became_active":"2024-09-21T13:59:09.063053+0000","last_became_peered":"2024-09-21T13:59:09.063053+0000","last_unstale":"2024-09-21T14:00:15.089214+0000","last_undegraded":"2024-09-21T14:00:15.089214+0000","last_fullsized":"2024-09-21T14:00:15.089214+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'7","last_scrub_stamp":"2024-09-21T14:00:01.507123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:00:01.507123+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:01:25.228964+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":476,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.059522+0000","last_change":"2024-09-21T13:59:32.204565+0000","last_active":"2024-09-21T14:00:15.059522+0000","last_peered":"2024-09-21T14:00:15.059522+0000","last_clean":"2024-09-21T14:00:15.059522+0000","last_became_active":"2024-09-21T13:59:32.204174+0000","last_became_peered":"2024-09-21T13:59:32.204174+0000","last_unstale":"2024-09-21T14:00:15.059522+0000","last_undegraded":"2024-09-21T14:00:15.059522+0000","last_fullsized":"2024-09-21T14:00:15.059522+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:06.722293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:06.722293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":548,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.052208+0000","last_change":"2024-09-21T13:58:44.376278+0000","last_active":"2024-09-21T14:00:15.052208+0000","last_peered":"2024-09-21T14:00:15.052208+0000","last_clean":"2024-09-21T14:00:15.052208+0000","last_became_active":"2024-09-21T13:57:12.684385+0000","last_became_peered":"2024-09-21T13:57:12.684385+0000","last_unstale":"2024-09-21T14:00:15.052208+0000","last_undegraded":"2024-09-21T14:00:15.052208+0000","last_fullsized":"2024-09-21T14:00:15.052208+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:44.376215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:44.376215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":554,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.091064+0000","last_change":"2024-09-21T14:00:00.497259+0000","last_active":"2024-09-21T14:00:15.091064+0000","last_peered":"2024-09-21T14:00:15.091064+0000","last_clean":"2024-09-21T14:00:15.091064+0000","last_became_active":"2024-09-21T13:58:17.770940+0000","last_became_peered":"2024-09-21T13:58:17.770940+0000","last_unstale":"2024-09-21T14:00:15.091064+0000","last_undegraded":"2024-09-21T14:00:15.091064+0000","last_fullsized":"2024-09-21T14:00:15.091064+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:00:00.497191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:00:00.497191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:01:15.404987+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":483,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.089492+0000","last_change":"2024-09-21T13:59:32.205037+0000","last_active":"2024-09-21T14:00:15.089492+0000","last_peered":"2024-09-21T14:00:15.089492+0000","last_clean":"2024-09-21T14:00:15.089492+0000","last_became_active":"2024-09-21T13:59:32.204749+0000","last_became_peered":"2024-09-21T13:59:32.204749+0000","last_unstale":"2024-09-21T14:00:15.089492+0000","last_undegraded":"2024-09-21T14:00:15.089492+0000","last_fullsized":"2024-09-21T14:00:15.089492+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:17.361699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:17.361699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"290'27","reported_seq":540,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.067620+0000","last_change":"2024-09-21T14:00:06.841023+0000","last_active":"2024-09-21T14:00:15.067620+0000","last_peered":"2024-09-21T14:00:15.067620+0000","last_clean":"2024-09-21T14:00:15.067620+0000","last_became_active":"2024-09-21T13:57:12.680783+0000","last_became_peered":"2024-09-21T13:57:12.680783+0000","last_unstale":"2024-09-21T14:00:15.067620+0000","last_undegraded":"2024-09-21T14:00:15.067620+0000","last_fullsized":"2024-09-21T14:00:15.067620+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'27","last_scrub_stamp":"2024-09-21T14:00:06.840937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:00:06.840937+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:01:16.134419+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":533,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.057376+0000","last_change":"2024-09-21T13:55:31.988001+0000","last_active":"2024-09-21T14:00:15.057376+0000","last_peered":"2024-09-21T14:00:15.057376+0000","last_clean":"2024-09-21T14:00:15.057376+0000","last_became_active":"2024-09-21T13:49:02.066796+0000","last_became_peered":"2024-09-21T13:49:02.066796+0000","last_unstale":"2024-09-21T14:00:15.057376+0000","last_undegraded":"2024-09-21T14:00:15.057376+0000","last_fullsized":"2024-09-21T14:00:15.057376+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:31.987854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:55:31.987854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":482,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.061568+0000","last_change":"2024-09-21T13:58:17.779422+0000","last_active":"2024-09-21T14:00:15.061568+0000","last_peered":"2024-09-21T14:00:15.061568+0000","last_clean":"2024-09-21T14:00:15.061568+0000","last_became_active":"2024-09-21T13:58:17.779117+0000","last_became_peered":"2024-09-21T13:58:17.779117+0000","last_unstale":"2024-09-21T14:00:15.061568+0000","last_undegraded":"2024-09-21T14:00:15.061568+0000","last_fullsized":"2024-09-21T14:00:15.061568+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:57.962266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:57.962266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":543,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.089529+0000","last_change":"2024-09-21T13:56:58.309928+0000","last_active":"2024-09-21T14:00:15.089529+0000","last_peered":"2024-09-21T14:00:15.089529+0000","last_clean":"2024-09-21T14:00:15.089529+0000","last_became_active":"2024-09-21T13:52:04.316016+0000","last_became_peered":"2024-09-21T13:52:04.316016+0000","last_unstale":"2024-09-21T14:00:15.089529+0000","last_undegraded":"2024-09-21T14:00:15.089529+0000","last_fullsized":"2024-09-21T14:00:15.089529+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:58.309863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:58.309863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:47.291059+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"290'18","reported_seq":499,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.061091+0000","last_change":"2024-09-21T13:59:23.041396+0000","last_active":"2024-09-21T14:00:15.061091+0000","last_peered":"2024-09-21T14:00:15.061091+0000","last_clean":"2024-09-21T14:00:15.061091+0000","last_became_active":"2024-09-21T13:57:12.684454+0000","last_became_peered":"2024-09-21T13:57:12.684454+0000","last_unstale":"2024-09-21T14:00:15.061091+0000","last_undegraded":"2024-09-21T14:00:15.061091+0000","last_fullsized":"2024-09-21T14:00:15.061091+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'18","last_scrub_stamp":"2024-09-21T13:59:23.041311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:23.041311+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:25.910995+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":538,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.090760+0000","last_change":"2024-09-21T13:58:16.393763+0000","last_active":"2024-09-21T14:00:15.090760+0000","last_peered":"2024-09-21T14:00:15.090760+0000","last_clean":"2024-09-21T14:00:15.090760+0000","last_became_active":"2024-09-21T13:57:12.681897+0000","last_became_peered":"2024-09-21T13:57:12.681897+0000","last_unstale":"2024-09-21T14:00:15.090760+0000","last_undegraded":"2024-09-21T14:00:15.090760+0000","last_fullsized":"2024-09-21T14:00:15.090760+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:16.393696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:16.393696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":449,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.059686+0000","last_change":"2024-09-21T13:59:32.216521+0000","last_active":"2024-09-21T14:00:15.059686+0000","last_peered":"2024-09-21T14:00:15.059686+0000","last_clean":"2024-09-21T14:00:15.059686+0000","last_became_active":"2024-09-21T13:59:32.215222+0000","last_became_peered":"2024-09-21T13:59:32.215222+0000","last_unstale":"2024-09-21T14:00:15.059686+0000","last_undegraded":"2024-09-21T14:00:15.059686+0000","last_fullsized":"2024-09-21T14:00:15.059686+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T13:57:26.536684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:26.536684+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":553,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.056074+0000","last_change":"2024-09-21T13:57:27.305798+0000","last_active":"2024-09-21T14:00:15.056074+0000","last_peered":"2024-09-21T14:00:15.056074+0000","last_clean":"2024-09-21T14:00:15.056074+0000","last_became_active":"2024-09-21T13:57:12.685742+0000","last_became_peered":"2024-09-21T13:57:12.685742+0000","last_unstale":"2024-09-21T14:00:15.056074+0000","last_undegraded":"2024-09-21T14:00:15.056074+0000","last_fullsized":"2024-09-21T14:00:15.056074+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:57:27.305724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:27.305724+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:21.365203+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"290'20","reported_seq":440,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.061120+0000","last_change":"2024-09-21T13:59:32.242771+0000","last_active":"2024-09-21T14:00:15.061120+0000","last_peered":"2024-09-21T14:00:15.061120+0000","last_clean":"2024-09-21T14:00:15.061120+0000","last_became_active":"2024-09-21T13:59:32.213865+0000","last_became_peered":"2024-09-21T13:59:32.213865+0000","last_unstale":"2024-09-21T14:00:15.061120+0000","last_undegraded":"2024-09-21T14:00:15.061120+0000","last_fullsized":"2024-09-21T14:00:15.061120+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:57:27.578738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:57:27.578738+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"300'12","reported_seq":146,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.060388+0000","last_change":"2024-09-21T13:59:36.012775+0000","last_active":"2024-09-21T14:00:15.060388+0000","last_peered":"2024-09-21T14:00:15.060388+0000","last_clean":"2024-09-21T14:00:15.060388+0000","last_became_active":"2024-09-21T13:59:32.205051+0000","last_became_peered":"2024-09-21T13:59:32.205051+0000","last_unstale":"2024-09-21T14:00:15.060388+0000","last_undegraded":"2024-09-21T14:00:15.060388+0000","last_fullsized":"2024-09-21T14:00:15.060388+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'12","last_scrub_stamp":"2024-09-21T13:59:36.012693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:36.012693+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:01:05.127536+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2","version":"312'10155","reported_seq":10271,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:20.960436+0000","last_change":"2024-09-21T13:57:53.208687+0000","last_active":"2024-09-21T14:00:20.960436+0000","last_peered":"2024-09-21T14:00:20.960436+0000","last_clean":"2024-09-21T14:00:20.960436+0000","last_became_active":"2024-09-21T13:57:53.208352+0000","last_became_peered":"2024-09-21T13:57:53.208352+0000","last_unstale":"2024-09-21T14:00:20.960436+0000","last_undegraded":"2024-09-21T14:00:20.960436+0000","last_fullsized":"2024-09-21T14:00:20.960436+0000","mapping_epoch":260,"log_start":"290'5400","ondisk_log_start":"290'5400","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4755,"log_dups_size":0,"ondisk_log_size":4755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331677696,"num_objects":5061,"num_object_clones":0,"num_object_copies":15183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10155,"num_write_kb":486912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"290'27","reported_seq":599,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.057725+0000","last_change":"2024-09-21T13:58:45.984691+0000","last_active":"2024-09-21T14:00:15.057725+0000","last_peered":"2024-09-21T14:00:15.057725+0000","last_clean":"2024-09-21T14:00:15.057725+0000","last_became_active":"2024-09-21T13:57:12.695609+0000","last_became_peered":"2024-09-21T13:57:12.695609+0000","last_unstale":"2024-09-21T14:00:15.057725+0000","last_undegraded":"2024-09-21T14:00:15.057725+0000","last_fullsized":"2024-09-21T14:00:15.057725+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:58:45.984575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:45.984575+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":485,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.061886+0000","last_change":"2024-09-21T13:59:32.205723+0000","last_active":"2024-09-21T14:00:15.061886+0000","last_peered":"2024-09-21T14:00:15.061886+0000","last_clean":"2024-09-21T14:00:15.061886+0000","last_became_active":"2024-09-21T13:59:32.205350+0000","last_became_peered":"2024-09-21T13:59:32.205350+0000","last_unstale":"2024-09-21T14:00:15.061886+0000","last_undegraded":"2024-09-21T14:00:15.061886+0000","last_fullsized":"2024-09-21T14:00:15.061886+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:32.668851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:32.668851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"68'8","reported_seq":681,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:18.262795+0000","last_change":"2024-09-21T13:59:55.499466+0000","last_active":"2024-09-21T14:00:18.262795+0000","last_peered":"2024-09-21T14:00:18.262795+0000","last_clean":"2024-09-21T14:00:18.262795+0000","last_became_active":"2024-09-21T13:49:02.061679+0000","last_became_peered":"2024-09-21T13:49:02.061679+0000","last_unstale":"2024-09-21T14:00:18.262795+0000","last_undegraded":"2024-09-21T14:00:18.262795+0000","last_fullsized":"2024-09-21T14:00:18.262795+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:59:55.499140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:55.499140+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:01:24.309628+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":521,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.052536+0000","last_change":"2024-09-21T13:58:52.244527+0000","last_active":"2024-09-21T14:00:15.052536+0000","last_peered":"2024-09-21T14:00:15.052536+0000","last_clean":"2024-09-21T14:00:15.052536+0000","last_became_active":"2024-09-21T13:49:04.081905+0000","last_became_peered":"2024-09-21T13:49:04.081905+0000","last_unstale":"2024-09-21T14:00:15.052536+0000","last_undegraded":"2024-09-21T14:00:15.052536+0000","last_fullsized":"2024-09-21T14:00:15.052536+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:52.244483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:52.244483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:19.563849+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"312'9940","reported_seq":10071,"reported_epoch":312,"state":"active+undersized+degraded+remapped+backfill_wait+backfill_toofull","last_fresh":"2024-09-21T14:00:19.531899+0000","last_change":"2024-09-21T14:00:03.321673+0000","last_active":"2024-09-21T14:00:19.531899+0000","last_peered":"2024-09-21T14:00:19.531899+0000","last_clean":"2024-09-21T13:58:23.757609+0000","last_became_active":"2024-09-21T13:59:33.315668+0000","last_became_peered":"2024-09-21T13:59:33.315668+0000","last_unstale":"2024-09-21T14:00:19.531899+0000","last_undegraded":"2024-09-21T13:59:33.208390+0000","last_fullsized":"2024-09-21T13:59:33.189502+0000","mapping_epoch":299,"log_start":"290'5265","ondisk_log_start":"290'5265","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4675,"log_dups_size":0,"ondisk_log_size":4675,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":345309208,"num_objects":5270,"num_object_clones":0,"num_object_copies":15810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5270,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9940,"num_write_kb":486657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":5270}],"blocked_by":[],"up_primary":7,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"290'30","reported_seq":509,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.060388+0000","last_change":"2024-09-21T13:59:32.246090+0000","last_active":"2024-09-21T14:00:15.060388+0000","last_peered":"2024-09-21T14:00:15.060388+0000","last_clean":"2024-09-21T14:00:15.060388+0000","last_became_active":"2024-09-21T13:59:32.205887+0000","last_became_peered":"2024-09-21T13:59:32.205887+0000","last_unstale":"2024-09-21T14:00:15.060388+0000","last_undegraded":"2024-09-21T14:00:15.060388+0000","last_fullsized":"2024-09-21T14:00:15.060388+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'18","last_scrub_stamp":"2024-09-21T13:54:19.001030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:19.001030+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":437,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.061937+0000","last_change":"2024-09-21T13:59:32.212559+0000","last_active":"2024-09-21T14:00:15.061937+0000","last_peered":"2024-09-21T14:00:15.061937+0000","last_clean":"2024-09-21T14:00:15.061937+0000","last_became_active":"2024-09-21T13:59:32.211004+0000","last_became_peered":"2024-09-21T13:59:32.211004+0000","last_unstale":"2024-09-21T14:00:15.061937+0000","last_undegraded":"2024-09-21T14:00:15.061937+0000","last_fullsized":"2024-09-21T14:00:15.061937+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:25.736461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:25.736461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":555,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.091332+0000","last_change":"2024-09-21T13:59:34.293330+0000","last_active":"2024-09-21T14:00:15.091332+0000","last_peered":"2024-09-21T14:00:15.091332+0000","last_clean":"2024-09-21T14:00:15.091332+0000","last_became_active":"2024-09-21T13:59:32.220304+0000","last_became_peered":"2024-09-21T13:59:32.220304+0000","last_unstale":"2024-09-21T14:00:15.091332+0000","last_undegraded":"2024-09-21T14:00:15.091332+0000","last_fullsized":"2024-09-21T14:00:15.091332+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:34.292892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:34.292892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:46.211883+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":535,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.089329+0000","last_change":"2024-09-21T13:59:23.334230+0000","last_active":"2024-09-21T14:00:15.089329+0000","last_peered":"2024-09-21T14:00:15.089329+0000","last_clean":"2024-09-21T14:00:15.089329+0000","last_became_active":"2024-09-21T13:57:12.680372+0000","last_became_peered":"2024-09-21T13:57:12.680372+0000","last_unstale":"2024-09-21T14:00:15.089329+0000","last_undegraded":"2024-09-21T14:00:15.089329+0000","last_fullsized":"2024-09-21T14:00:15.089329+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:23.334162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:23.334162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:46.281593+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"312'9956","reported_seq":10135,"reported_epoch":312,"state":"active+undersized+degraded+remapped+backfill_wait","last_fresh":"2024-09-21T14:00:20.971594+0000","last_change":"2024-09-21T14:00:03.946792+0000","last_active":"2024-09-21T14:00:20.971594+0000","last_peered":"2024-09-21T14:00:20.971594+0000","last_clean":"2024-09-21T13:58:23.796606+0000","last_became_active":"2024-09-21T13:59:33.303769+0000","last_became_peered":"2024-09-21T13:59:33.303769+0000","last_unstale":"2024-09-21T14:00:20.971594+0000","last_undegraded":"2024-09-21T13:59:33.205632+0000","last_fullsized":"2024-09-21T13:59:33.192057+0000","mapping_epoch":299,"log_start":"290'5200","ondisk_log_start":"290'5200","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4756,"log_dups_size":0,"ondisk_log_size":4756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333840384,"num_objects":5094,"num_object_clones":0,"num_object_copies":15282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5068,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9956,"num_write_kb":481600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1703936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":5094}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"290'28","reported_seq":617,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.091413+0000","last_change":"2024-09-21T13:58:17.810036+0000","last_active":"2024-09-21T14:00:15.091413+0000","last_peered":"2024-09-21T14:00:15.091413+0000","last_clean":"2024-09-21T14:00:15.091413+0000","last_became_active":"2024-09-21T13:58:17.779342+0000","last_became_peered":"2024-09-21T13:58:17.779342+0000","last_unstale":"2024-09-21T14:00:15.091413+0000","last_undegraded":"2024-09-21T14:00:15.091413+0000","last_fullsized":"2024-09-21T14:00:15.091413+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'14","last_scrub_stamp":"2024-09-21T13:56:32.438459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:56:32.438459+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:45.254297+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":543,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.055697+0000","last_change":"2024-09-21T13:57:38.046718+0000","last_active":"2024-09-21T14:00:15.055697+0000","last_peered":"2024-09-21T14:00:15.055697+0000","last_clean":"2024-09-21T14:00:15.055697+0000","last_became_active":"2024-09-21T13:48:58.053533+0000","last_became_peered":"2024-09-21T13:48:58.053533+0000","last_unstale":"2024-09-21T14:00:15.055697+0000","last_undegraded":"2024-09-21T14:00:15.055697+0000","last_fullsized":"2024-09-21T14:00:15.055697+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:57:38.046627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:38.046627+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"241'10","reported_seq":555,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.061996+0000","last_change":"2024-09-21T13:58:06.789629+0000","last_active":"2024-09-21T14:00:15.061996+0000","last_peered":"2024-09-21T14:00:15.061996+0000","last_clean":"2024-09-21T14:00:15.061996+0000","last_became_active":"2024-09-21T13:57:12.686651+0000","last_became_peered":"2024-09-21T13:57:12.686651+0000","last_unstale":"2024-09-21T14:00:15.061996+0000","last_undegraded":"2024-09-21T14:00:15.061996+0000","last_fullsized":"2024-09-21T14:00:15.061996+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'10","last_scrub_stamp":"2024-09-21T13:58:06.789560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:06.789560+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":501,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.059366+0000","last_change":"2024-09-21T13:59:32.210257+0000","last_active":"2024-09-21T14:00:15.059366+0000","last_peered":"2024-09-21T14:00:15.059366+0000","last_clean":"2024-09-21T14:00:15.059366+0000","last_became_active":"2024-09-21T13:59:32.209615+0000","last_became_peered":"2024-09-21T13:59:32.209615+0000","last_unstale":"2024-09-21T14:00:15.059366+0000","last_undegraded":"2024-09-21T14:00:15.059366+0000","last_fullsized":"2024-09-21T14:00:15.059366+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:37.548064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:37.548064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"312'10052","reported_seq":10168,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:18.841559+0000","last_change":"2024-09-21T13:57:53.213439+0000","last_active":"2024-09-21T14:00:18.841559+0000","last_peered":"2024-09-21T14:00:18.841559+0000","last_clean":"2024-09-21T14:00:18.841559+0000","last_became_active":"2024-09-21T13:57:53.213139+0000","last_became_peered":"2024-09-21T13:57:53.213139+0000","last_unstale":"2024-09-21T14:00:18.841559+0000","last_undegraded":"2024-09-21T14:00:18.841559+0000","last_fullsized":"2024-09-21T14:00:18.841559+0000","mapping_epoch":260,"log_start":"286'4700","ondisk_log_start":"286'4700","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":5352,"log_dups_size":0,"ondisk_log_size":5352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":346423296,"num_objects":5286,"num_object_clones":0,"num_object_copies":15858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10052,"num_write_kb":490816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"290'20","reported_seq":568,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.056394+0000","last_change":"2024-09-21T13:54:19.593696+0000","last_active":"2024-09-21T14:00:15.056394+0000","last_peered":"2024-09-21T14:00:15.056394+0000","last_clean":"2024-09-21T14:00:15.056394+0000","last_became_active":"2024-09-21T13:49:00.054564+0000","last_became_peered":"2024-09-21T13:49:00.054564+0000","last_unstale":"2024-09-21T14:00:15.056394+0000","last_undegraded":"2024-09-21T14:00:15.056394+0000","last_fullsized":"2024-09-21T14:00:15.056394+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:54:19.593611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:19.593611+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"61'1","reported_seq":586,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.053048+0000","last_change":"2024-09-21T13:58:48.407813+0000","last_active":"2024-09-21T14:00:15.053048+0000","last_peered":"2024-09-21T14:00:15.053048+0000","last_clean":"2024-09-21T14:00:15.053048+0000","last_became_active":"2024-09-21T13:57:12.684939+0000","last_became_peered":"2024-09-21T13:57:12.684939+0000","last_unstale":"2024-09-21T14:00:15.053048+0000","last_undegraded":"2024-09-21T14:00:15.053048+0000","last_fullsized":"2024-09-21T14:00:15.053048+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:58:48.407734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:48.407734+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"270'13","reported_seq":125,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.060449+0000","last_change":"2024-09-21T13:58:17.788471+0000","last_active":"2024-09-21T14:00:15.060449+0000","last_peered":"2024-09-21T14:00:15.060449+0000","last_clean":"2024-09-21T14:00:15.060449+0000","last_became_active":"2024-09-21T13:58:17.777857+0000","last_became_peered":"2024-09-21T13:58:17.777857+0000","last_unstale":"2024-09-21T14:00:15.060449+0000","last_undegraded":"2024-09-21T14:00:15.060449+0000","last_fullsized":"2024-09-21T14:00:15.060449+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'12","last_scrub_stamp":"2024-09-21T13:57:30.133392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:30.133392+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:18.984922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":355,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.061860+0000","last_change":"2024-09-21T13:58:58.730439+0000","last_active":"2024-09-21T14:00:15.061860+0000","last_peered":"2024-09-21T14:00:15.061860+0000","last_clean":"2024-09-21T14:00:15.061860+0000","last_became_active":"2024-09-21T13:57:12.704897+0000","last_became_peered":"2024-09-21T13:57:12.704897+0000","last_unstale":"2024-09-21T14:00:15.061860+0000","last_undegraded":"2024-09-21T14:00:15.061860+0000","last_fullsized":"2024-09-21T14:00:15.061860+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:58.730368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:58.730368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.e","version":"312'9780","reported_seq":13788,"reported_epoch":312,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-21T14:00:19.536342+0000","last_change":"2024-09-21T14:00:03.950127+0000","last_active":"2024-09-21T14:00:19.536342+0000","last_peered":"2024-09-21T14:00:19.536342+0000","last_clean":"2024-09-21T13:58:23.771204+0000","last_became_active":"2024-09-21T13:59:33.338259+0000","last_became_peered":"2024-09-21T13:59:33.338259+0000","last_unstale":"2024-09-21T14:00:19.536342+0000","last_undegraded":"2024-09-21T13:59:33.212884+0000","last_fullsized":"2024-09-21T13:59:33.191296+0000","mapping_epoch":299,"log_start":"290'5083","ondisk_log_start":"290'5083","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4697,"log_dups_size":0,"ondisk_log_size":4697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":337248256,"num_objects":5146,"num_object_clones":0,"num_object_copies":15438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3217,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9780,"num_write_kb":477632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1940,"num_bytes_recovered":127139840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":5146}],"blocked_by":[],"up_primary":7,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"290'29","reported_seq":561,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.057253+0000","last_change":"2024-09-21T13:59:32.235547+0000","last_active":"2024-09-21T14:00:15.057253+0000","last_peered":"2024-09-21T14:00:15.057253+0000","last_clean":"2024-09-21T14:00:15.057253+0000","last_became_active":"2024-09-21T13:59:32.209177+0000","last_became_peered":"2024-09-21T13:59:32.209177+0000","last_unstale":"2024-09-21T14:00:15.057253+0000","last_undegraded":"2024-09-21T14:00:15.057253+0000","last_fullsized":"2024-09-21T14:00:15.057253+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-21T13:56:55.319759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:56:55.319759+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":545,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.056840+0000","last_change":"2024-09-21T13:58:16.713075+0000","last_active":"2024-09-21T14:00:15.056840+0000","last_peered":"2024-09-21T14:00:15.056840+0000","last_clean":"2024-09-21T14:00:15.056840+0000","last_became_active":"2024-09-21T13:52:04.321656+0000","last_became_peered":"2024-09-21T13:52:04.321656+0000","last_unstale":"2024-09-21T14:00:15.056840+0000","last_undegraded":"2024-09-21T14:00:15.056840+0000","last_fullsized":"2024-09-21T14:00:15.056840+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:16.712990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:16.712990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":104,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.060801+0000","last_change":"2024-09-21T13:59:23.032196+0000","last_active":"2024-09-21T14:00:15.060801+0000","last_peered":"2024-09-21T14:00:15.060801+0000","last_clean":"2024-09-21T14:00:15.060801+0000","last_became_active":"2024-09-21T13:58:17.778679+0000","last_became_peered":"2024-09-21T13:58:17.778679+0000","last_unstale":"2024-09-21T14:00:15.060801+0000","last_undegraded":"2024-09-21T14:00:15.060801+0000","last_fullsized":"2024-09-21T14:00:15.060801+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:23.032143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:23.032143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:26.703801+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":547,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.055857+0000","last_change":"2024-09-21T13:58:43.952014+0000","last_active":"2024-09-21T14:00:15.055857+0000","last_peered":"2024-09-21T14:00:15.055857+0000","last_clean":"2024-09-21T14:00:15.055857+0000","last_became_active":"2024-09-21T13:58:17.757628+0000","last_became_peered":"2024-09-21T13:58:17.757628+0000","last_unstale":"2024-09-21T14:00:15.055857+0000","last_undegraded":"2024-09-21T14:00:15.055857+0000","last_fullsized":"2024-09-21T14:00:15.055857+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:43.951941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:43.951941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"312'9959","reported_seq":13230,"reported_epoch":312,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-21T14:00:19.152401+0000","last_change":"2024-09-21T14:00:03.947405+0000","last_active":"2024-09-21T14:00:19.152401+0000","last_peered":"2024-09-21T14:00:19.152401+0000","last_clean":"2024-09-21T13:58:23.765960+0000","last_became_active":"2024-09-21T13:59:33.385921+0000","last_became_peered":"2024-09-21T13:59:33.385921+0000","last_unstale":"2024-09-21T14:00:19.152401+0000","last_undegraded":"2024-09-21T13:59:33.219145+0000","last_fullsized":"2024-09-21T13:59:33.193300+0000","mapping_epoch":299,"log_start":"284'4000","ondisk_log_start":"284'4000","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":5959,"log_dups_size":0,"ondisk_log_size":5959,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":348192768,"num_objects":5313,"num_object_clones":0,"num_object_copies":15939,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3684,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9959,"num_write_kb":488704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1575,"num_bytes_recovered":103219200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[{"shards":"0,5","objects":1629}],"blocked_by":[],"up_primary":7,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"290'44","reported_seq":759,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.056742+0000","last_change":"2024-09-21T13:58:56.076153+0000","last_active":"2024-09-21T14:00:15.056742+0000","last_peered":"2024-09-21T14:00:15.056742+0000","last_clean":"2024-09-21T14:00:15.056742+0000","last_became_active":"2024-09-21T13:57:12.684690+0000","last_became_peered":"2024-09-21T13:57:12.684690+0000","last_unstale":"2024-09-21T14:00:15.056742+0000","last_undegraded":"2024-09-21T14:00:15.056742+0000","last_fullsized":"2024-09-21T14:00:15.056742+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'32","last_scrub_stamp":"2024-09-21T13:58:56.076074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:56.076074+0000","objects_scrubbed":10,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":547,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.089571+0000","last_change":"2024-09-21T13:59:27.369582+0000","last_active":"2024-09-21T14:00:15.089571+0000","last_peered":"2024-09-21T14:00:15.089571+0000","last_clean":"2024-09-21T14:00:15.089571+0000","last_became_active":"2024-09-21T13:58:17.770937+0000","last_became_peered":"2024-09-21T13:58:17.770937+0000","last_unstale":"2024-09-21T14:00:15.089571+0000","last_undegraded":"2024-09-21T14:00:15.089571+0000","last_fullsized":"2024-09-21T14:00:15.089571+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:27.369511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:59:27.369511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:40.229805+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":582,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.052740+0000","last_change":"2024-09-21T13:59:22.186466+0000","last_active":"2024-09-21T14:00:15.052740+0000","last_peered":"2024-09-21T14:00:15.052740+0000","last_clean":"2024-09-21T14:00:15.052740+0000","last_became_active":"2024-09-21T13:58:17.776563+0000","last_became_peered":"2024-09-21T13:58:17.776563+0000","last_unstale":"2024-09-21T14:00:15.052740+0000","last_undegraded":"2024-09-21T14:00:15.052740+0000","last_fullsized":"2024-09-21T14:00:15.052740+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-21T13:59:22.186393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:59:22.186393+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:28.219690+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0015946669999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":474,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.057304+0000","last_change":"2024-09-21T13:55:07.021128+0000","last_active":"2024-09-21T14:00:15.057304+0000","last_peered":"2024-09-21T14:00:15.057304+0000","last_clean":"2024-09-21T14:00:15.057304+0000","last_became_active":"2024-09-21T13:50:29.688249+0000","last_became_peered":"2024-09-21T13:50:29.688249+0000","last_unstale":"2024-09-21T14:00:15.057304+0000","last_undegraded":"2024-09-21T14:00:15.057304+0000","last_fullsized":"2024-09-21T14:00:15.057304+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:07.021059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:55:07.021059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":464,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.061753+0000","last_change":"2024-09-21T13:59:32.209357+0000","last_active":"2024-09-21T14:00:15.061753+0000","last_peered":"2024-09-21T14:00:15.061753+0000","last_clean":"2024-09-21T14:00:15.061753+0000","last_became_active":"2024-09-21T13:59:32.208980+0000","last_became_peered":"2024-09-21T13:59:32.208980+0000","last_unstale":"2024-09-21T14:00:15.061753+0000","last_undegraded":"2024-09-21T14:00:15.061753+0000","last_fullsized":"2024-09-21T14:00:15.061753+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:16.797227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:16.797227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"312'9700","reported_seq":9822,"reported_epoch":312,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T14:00:18.841922+0000","last_change":"2024-09-21T13:59:33.463191+0000","last_active":"2024-09-21T14:00:18.841922+0000","last_peered":"2024-09-21T14:00:18.841922+0000","last_clean":"2024-09-21T13:58:23.784671+0000","last_became_active":"2024-09-21T13:59:33.447969+0000","last_became_peered":"2024-09-21T13:59:33.447969+0000","last_unstale":"2024-09-21T14:00:18.841922+0000","last_undegraded":"2024-09-21T13:59:33.234331+0000","last_fullsized":"2024-09-21T13:59:33.191526+0000","mapping_epoch":299,"log_start":"286'4400","ondisk_log_start":"286'4400","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":5300,"log_dups_size":0,"ondisk_log_size":5300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338296832,"num_objects":5162,"num_object_clones":0,"num_object_copies":15486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5080,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9700,"num_write_kb":475584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":82}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"290'17","reported_seq":571,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.057756+0000","last_change":"2024-09-21T13:58:11.619937+0000","last_active":"2024-09-21T14:00:15.057756+0000","last_peered":"2024-09-21T14:00:15.057756+0000","last_clean":"2024-09-21T14:00:15.057756+0000","last_became_active":"2024-09-21T13:49:00.053323+0000","last_became_peered":"2024-09-21T13:49:00.053323+0000","last_unstale":"2024-09-21T14:00:15.057756+0000","last_undegraded":"2024-09-21T14:00:15.057756+0000","last_fullsized":"2024-09-21T14:00:15.057756+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'13","last_scrub_stamp":"2024-09-21T13:58:11.619848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:11.619848+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":583,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.056242+0000","last_change":"2024-09-21T13:59:29.454265+0000","last_active":"2024-09-21T14:00:15.056242+0000","last_peered":"2024-09-21T14:00:15.056242+0000","last_clean":"2024-09-21T14:00:15.056242+0000","last_became_active":"2024-09-21T13:58:17.759671+0000","last_became_peered":"2024-09-21T13:58:17.759671+0000","last_unstale":"2024-09-21T14:00:15.056242+0000","last_undegraded":"2024-09-21T14:00:15.056242+0000","last_fullsized":"2024-09-21T14:00:15.056242+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:29.454085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:59:29.454085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:43.157879+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":472,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.066793+0000","last_change":"2024-09-21T13:58:41.644876+0000","last_active":"2024-09-21T14:00:15.066793+0000","last_peered":"2024-09-21T14:00:15.066793+0000","last_clean":"2024-09-21T14:00:15.066793+0000","last_became_active":"2024-09-21T13:57:12.682147+0000","last_became_peered":"2024-09-21T13:57:12.682147+0000","last_unstale":"2024-09-21T14:00:15.066793+0000","last_undegraded":"2024-09-21T14:00:15.066793+0000","last_fullsized":"2024-09-21T14:00:15.066793+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:58:41.644789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:58:41.644789+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:45.660503+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":424,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.068526+0000","last_change":"2024-09-21T13:59:32.208163+0000","last_active":"2024-09-21T14:00:15.068526+0000","last_peered":"2024-09-21T14:00:15.068526+0000","last_clean":"2024-09-21T14:00:15.068526+0000","last_became_active":"2024-09-21T13:59:32.207740+0000","last_became_peered":"2024-09-21T13:59:32.207740+0000","last_unstale":"2024-09-21T14:00:15.068526+0000","last_undegraded":"2024-09-21T14:00:15.068526+0000","last_fullsized":"2024-09-21T14:00:15.068526+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:10.320333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:10.320333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:44.635553+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":509,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.059692+0000","last_change":"2024-09-21T13:59:32.222791+0000","last_active":"2024-09-21T14:00:15.059692+0000","last_peered":"2024-09-21T14:00:15.059692+0000","last_clean":"2024-09-21T14:00:15.059692+0000","last_became_active":"2024-09-21T13:59:32.221502+0000","last_became_peered":"2024-09-21T13:59:32.221502+0000","last_unstale":"2024-09-21T14:00:15.059692+0000","last_undegraded":"2024-09-21T14:00:15.059692+0000","last_fullsized":"2024-09-21T14:00:15.059692+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:14.673582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:14.673582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"312'9946","reported_seq":13942,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:18.466624+0000","last_change":"2024-09-21T13:59:31.471106+0000","last_active":"2024-09-21T14:00:18.466624+0000","last_peered":"2024-09-21T14:00:18.466624+0000","last_clean":"2024-09-21T14:00:18.466624+0000","last_became_active":"2024-09-21T13:58:35.721126+0000","last_became_peered":"2024-09-21T13:58:35.721126+0000","last_unstale":"2024-09-21T14:00:18.466624+0000","last_undegraded":"2024-09-21T14:00:18.466624+0000","last_fullsized":"2024-09-21T14:00:18.466624+0000","mapping_epoch":278,"log_start":"281'3400","ondisk_log_start":"281'3400","created":260,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"298'7530","last_scrub_stamp":"2024-09-21T13:59:31.471024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:59:31.471024+0000","objects_scrubbed":7502,"log_size":6546,"log_dups_size":0,"ondisk_log_size":6546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:33.949758+0000","scrub_duration":1396,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343277568,"num_objects":5238,"num_object_clones":0,"num_object_copies":15714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9946,"num_write_kb":485888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1892,"num_bytes_recovered":123994112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"290'24","reported_seq":622,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.054767+0000","last_change":"2024-09-21T13:59:28.458847+0000","last_active":"2024-09-21T14:00:15.054767+0000","last_peered":"2024-09-21T14:00:15.054767+0000","last_clean":"2024-09-21T14:00:15.054767+0000","last_became_active":"2024-09-21T13:52:04.318875+0000","last_became_peered":"2024-09-21T13:52:04.318875+0000","last_unstale":"2024-09-21T14:00:15.054767+0000","last_undegraded":"2024-09-21T14:00:15.054767+0000","last_fullsized":"2024-09-21T14:00:15.054767+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'24","last_scrub_stamp":"2024-09-21T13:59:28.458768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:28.458768+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:51.851382+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":484,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.091569+0000","last_change":"2024-09-21T13:59:59.450613+0000","last_active":"2024-09-21T14:00:15.091569+0000","last_peered":"2024-09-21T14:00:15.091569+0000","last_clean":"2024-09-21T14:00:15.091569+0000","last_became_active":"2024-09-21T13:57:12.681030+0000","last_became_peered":"2024-09-21T13:57:12.681030+0000","last_unstale":"2024-09-21T14:00:15.091569+0000","last_undegraded":"2024-09-21T14:00:15.091569+0000","last_fullsized":"2024-09-21T14:00:15.091569+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:59.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:59:59.450546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:01:05.581170+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":1092,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:19.300229+0000","last_change":"2024-09-21T13:59:32.218796+0000","last_active":"2024-09-21T14:00:19.300229+0000","last_peered":"2024-09-21T14:00:19.300229+0000","last_clean":"2024-09-21T14:00:19.300229+0000","last_became_active":"2024-09-21T13:59:32.203361+0000","last_became_peered":"2024-09-21T13:59:32.203361+0000","last_unstale":"2024-09-21T14:00:19.300229+0000","last_undegraded":"2024-09-21T14:00:19.300229+0000","last_fullsized":"2024-09-21T14:00:19.300229+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-21T13:56:29.617143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:56:29.617143+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047048000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":55,"num_read_kb":50,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":526,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.062315+0000","last_change":"2024-09-21T13:56:33.316275+0000","last_active":"2024-09-21T14:00:15.062315+0000","last_peered":"2024-09-21T14:00:15.062315+0000","last_clean":"2024-09-21T14:00:15.062315+0000","last_became_active":"2024-09-21T13:49:02.066641+0000","last_became_peered":"2024-09-21T13:49:02.066641+0000","last_unstale":"2024-09-21T14:00:15.062315+0000","last_undegraded":"2024-09-21T14:00:15.062315+0000","last_fullsized":"2024-09-21T14:00:15.062315+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:33.316230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:56:33.316230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":557,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.056242+0000","last_change":"2024-09-21T13:59:26.557174+0000","last_active":"2024-09-21T14:00:15.056242+0000","last_peered":"2024-09-21T14:00:15.056242+0000","last_clean":"2024-09-21T14:00:15.056242+0000","last_became_active":"2024-09-21T13:57:12.684254+0000","last_became_peered":"2024-09-21T13:57:12.684254+0000","last_unstale":"2024-09-21T14:00:15.056242+0000","last_undegraded":"2024-09-21T14:00:15.056242+0000","last_fullsized":"2024-09-21T14:00:15.056242+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:26.557002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:26.557002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:28.551613+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"312'10238","reported_seq":10354,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:20.971254+0000","last_change":"2024-09-21T13:57:53.863404+0000","last_active":"2024-09-21T14:00:20.971254+0000","last_peered":"2024-09-21T14:00:20.971254+0000","last_clean":"2024-09-21T14:00:20.971254+0000","last_became_active":"2024-09-21T13:57:53.863113+0000","last_became_peered":"2024-09-21T13:57:53.863113+0000","last_unstale":"2024-09-21T14:00:20.971254+0000","last_undegraded":"2024-09-21T14:00:20.971254+0000","last_fullsized":"2024-09-21T14:00:20.971254+0000","mapping_epoch":260,"log_start":"290'5500","ondisk_log_start":"290'5500","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4738,"log_dups_size":0,"ondisk_log_size":4738,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338821120,"num_objects":5170,"num_object_clones":0,"num_object_copies":15510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10238,"num_write_kb":493056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"300'32","reported_seq":184,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.059905+0000","last_change":"2024-09-21T13:59:32.243718+0000","last_active":"2024-09-21T14:00:15.059905+0000","last_peered":"2024-09-21T14:00:15.059905+0000","last_clean":"2024-09-21T14:00:15.059905+0000","last_became_active":"2024-09-21T13:59:32.204624+0000","last_became_peered":"2024-09-21T13:59:32.204624+0000","last_unstale":"2024-09-21T14:00:15.059905+0000","last_undegraded":"2024-09-21T14:00:15.059905+0000","last_fullsized":"2024-09-21T14:00:15.059905+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'16","last_scrub_stamp":"2024-09-21T13:52:19.452163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:52:19.452163+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:40.020421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":511,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.062396+0000","last_change":"2024-09-21T13:58:57.773043+0000","last_active":"2024-09-21T14:00:15.062396+0000","last_peered":"2024-09-21T14:00:15.062396+0000","last_clean":"2024-09-21T14:00:15.062396+0000","last_became_active":"2024-09-21T13:58:17.778951+0000","last_became_peered":"2024-09-21T13:58:17.778951+0000","last_unstale":"2024-09-21T14:00:15.062396+0000","last_undegraded":"2024-09-21T14:00:15.062396+0000","last_fullsized":"2024-09-21T14:00:15.062396+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:57.772975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:57.772975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"75'79","reported_seq":573,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.059373+0000","last_change":"2024-09-21T13:59:32.202714+0000","last_active":"2024-09-21T14:00:15.059373+0000","last_peered":"2024-09-21T14:00:15.059373+0000","last_clean":"2024-09-21T14:00:15.059373+0000","last_became_active":"2024-09-21T13:59:32.201469+0000","last_became_peered":"2024-09-21T13:59:32.201469+0000","last_unstale":"2024-09-21T14:00:15.059373+0000","last_undegraded":"2024-09-21T14:00:15.059373+0000","last_fullsized":"2024-09-21T14:00:15.059373+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":486,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.059769+0000","last_change":"2024-09-21T13:59:32.213870+0000","last_active":"2024-09-21T14:00:15.059769+0000","last_peered":"2024-09-21T14:00:15.059769+0000","last_clean":"2024-09-21T14:00:15.059769+0000","last_became_active":"2024-09-21T13:59:32.212228+0000","last_became_peered":"2024-09-21T13:59:32.212228+0000","last_unstale":"2024-09-21T14:00:15.059769+0000","last_undegraded":"2024-09-21T14:00:15.059769+0000","last_fullsized":"2024-09-21T14:00:15.059769+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:15.700409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:15.700409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":533,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.062025+0000","last_change":"2024-09-21T13:57:21.153067+0000","last_active":"2024-09-21T14:00:15.062025+0000","last_peered":"2024-09-21T14:00:15.062025+0000","last_clean":"2024-09-21T14:00:15.062025+0000","last_became_active":"2024-09-21T13:57:12.684495+0000","last_became_peered":"2024-09-21T13:57:12.684495+0000","last_unstale":"2024-09-21T14:00:15.062025+0000","last_undegraded":"2024-09-21T14:00:15.062025+0000","last_fullsized":"2024-09-21T14:00:15.062025+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:21.153018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:21.153018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:18.188142+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"312'9943","reported_seq":14127,"reported_epoch":312,"state":"active+undersized+degraded+remapped+backfilling+forced_backfill","last_fresh":"2024-09-21T14:00:19.543202+0000","last_change":"2024-09-21T14:00:03.340290+0000","last_active":"2024-09-21T14:00:19.543202+0000","last_peered":"2024-09-21T14:00:19.543202+0000","last_clean":"2024-09-21T13:58:23.728526+0000","last_became_active":"2024-09-21T13:59:33.326654+0000","last_became_peered":"2024-09-21T13:59:33.326654+0000","last_unstale":"2024-09-21T14:00:19.543202+0000","last_undegraded":"2024-09-21T13:59:33.211255+0000","last_fullsized":"2024-09-21T13:59:33.189504+0000","mapping_epoch":299,"log_start":"290'5312","ondisk_log_start":"290'5312","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4631,"log_dups_size":0,"ondisk_log_size":4631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":342425600,"num_objects":5225,"num_object_clones":0,"num_object_copies":15675,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3207,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9943,"num_write_kb":485376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2030,"num_bytes_recovered":133038080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":5225}],"blocked_by":[],"up_primary":7,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"290'19","reported_seq":601,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.091790+0000","last_change":"2024-09-21T13:58:17.811300+0000","last_active":"2024-09-21T14:00:15.091790+0000","last_peered":"2024-09-21T14:00:15.091790+0000","last_clean":"2024-09-21T14:00:15.091790+0000","last_became_active":"2024-09-21T13:58:17.776310+0000","last_became_peered":"2024-09-21T13:58:17.776310+0000","last_unstale":"2024-09-21T14:00:15.091790+0000","last_undegraded":"2024-09-21T14:00:15.091790+0000","last_fullsized":"2024-09-21T14:00:15.091790+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:54:16.451588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:16.451588+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:01:16.580358+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":555,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.091785+0000","last_change":"2024-09-21T13:59:32.223608+0000","last_active":"2024-09-21T14:00:15.091785+0000","last_peered":"2024-09-21T14:00:15.091785+0000","last_clean":"2024-09-21T14:00:15.091785+0000","last_became_active":"2024-09-21T13:59:32.223247+0000","last_became_peered":"2024-09-21T13:59:32.223247+0000","last_unstale":"2024-09-21T14:00:15.091785+0000","last_undegraded":"2024-09-21T14:00:15.091785+0000","last_fullsized":"2024-09-21T14:00:15.091785+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:53.377100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:53.377100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:42.213058+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":547,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.055026+0000","last_change":"2024-09-21T13:58:17.782835+0000","last_active":"2024-09-21T14:00:15.055026+0000","last_peered":"2024-09-21T14:00:15.055026+0000","last_clean":"2024-09-21T14:00:15.055026+0000","last_became_active":"2024-09-21T13:58:17.782403+0000","last_became_peered":"2024-09-21T13:58:17.782403+0000","last_unstale":"2024-09-21T14:00:15.055026+0000","last_undegraded":"2024-09-21T14:00:15.055026+0000","last_fullsized":"2024-09-21T14:00:15.055026+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:34.299722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:55:34.299722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":489,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.063223+0000","last_change":"2024-09-21T13:58:17.778317+0000","last_active":"2024-09-21T14:00:15.063223+0000","last_peered":"2024-09-21T14:00:15.063223+0000","last_clean":"2024-09-21T14:00:15.063223+0000","last_became_active":"2024-09-21T13:58:17.777954+0000","last_became_peered":"2024-09-21T13:58:17.777954+0000","last_unstale":"2024-09-21T14:00:15.063223+0000","last_undegraded":"2024-09-21T14:00:15.063223+0000","last_fullsized":"2024-09-21T14:00:15.063223+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:58.952218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:54:58.952218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.0","version":"312'9987","reported_seq":14041,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:19.533897+0000","last_change":"2024-09-21T13:59:24.817101+0000","last_active":"2024-09-21T14:00:19.533897+0000","last_peered":"2024-09-21T14:00:19.533897+0000","last_clean":"2024-09-21T14:00:19.533897+0000","last_became_active":"2024-09-21T13:58:36.723110+0000","last_became_peered":"2024-09-21T13:58:36.723110+0000","last_unstale":"2024-09-21T14:00:19.533897+0000","last_undegraded":"2024-09-21T14:00:19.533897+0000","last_fullsized":"2024-09-21T14:00:19.533897+0000","mapping_epoch":279,"log_start":"290'5347","ondisk_log_start":"290'5347","created":260,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'7035","last_scrub_stamp":"2024-09-21T13:59:24.816903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:59:24.816903+0000","objects_scrubbed":6955,"log_size":4640,"log_dups_size":0,"ondisk_log_size":4640,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:30.971537+0000","scrub_duration":2446,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333250560,"num_objects":5085,"num_object_clones":0,"num_object_copies":15255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9987,"num_write_kb":482304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1924,"num_bytes_recovered":126091264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"290'31","reported_seq":589,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.090021+0000","last_change":"2024-09-21T13:56:55.354217+0000","last_active":"2024-09-21T14:00:15.090021+0000","last_peered":"2024-09-21T14:00:15.090021+0000","last_clean":"2024-09-21T14:00:15.090021+0000","last_became_active":"2024-09-21T13:52:04.313218+0000","last_became_peered":"2024-09-21T13:52:04.313218+0000","last_unstale":"2024-09-21T14:00:15.090021+0000","last_undegraded":"2024-09-21T14:00:15.090021+0000","last_fullsized":"2024-09-21T14:00:15.090021+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'17","last_scrub_stamp":"2024-09-21T13:56:55.354126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:56:55.354126+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:54.483045+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":104,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.060578+0000","last_change":"2024-09-21T14:00:05.196400+0000","last_active":"2024-09-21T14:00:15.060578+0000","last_peered":"2024-09-21T14:00:15.060578+0000","last_clean":"2024-09-21T14:00:15.060578+0000","last_became_active":"2024-09-21T13:58:17.762520+0000","last_became_peered":"2024-09-21T13:58:17.762520+0000","last_unstale":"2024-09-21T14:00:15.060578+0000","last_undegraded":"2024-09-21T14:00:15.060578+0000","last_fullsized":"2024-09-21T14:00:15.060578+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:00:05.196318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:00:05.196318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:01:29.827986+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":526,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.056390+0000","last_change":"2024-09-21T13:58:52.030800+0000","last_active":"2024-09-21T14:00:15.056390+0000","last_peered":"2024-09-21T14:00:15.056390+0000","last_clean":"2024-09-21T14:00:15.056390+0000","last_became_active":"2024-09-21T13:49:02.067472+0000","last_became_peered":"2024-09-21T13:49:02.067472+0000","last_unstale":"2024-09-21T14:00:15.056390+0000","last_undegraded":"2024-09-21T14:00:15.056390+0000","last_fullsized":"2024-09-21T14:00:15.056390+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:52.030718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:52.030718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":408,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.068226+0000","last_change":"2024-09-21T13:57:12.683321+0000","last_active":"2024-09-21T14:00:15.068226+0000","last_peered":"2024-09-21T14:00:15.068226+0000","last_clean":"2024-09-21T14:00:15.068226+0000","last_became_active":"2024-09-21T13:57:12.682979+0000","last_became_peered":"2024-09-21T13:57:12.682979+0000","last_unstale":"2024-09-21T14:00:15.068226+0000","last_undegraded":"2024-09-21T14:00:15.068226+0000","last_fullsized":"2024-09-21T14:00:15.068226+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:18.027944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:18.027944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:18.700579+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"312'9994","reported_seq":15844,"reported_epoch":312,"state":"active+remapped+backfill_wait+backfill_toofull","last_fresh":"2024-09-21T14:00:20.975159+0000","last_change":"2024-09-21T14:00:03.435124+0000","last_active":"2024-09-21T14:00:20.975159+0000","last_peered":"2024-09-21T14:00:20.975159+0000","last_clean":"2024-09-21T13:59:31.188333+0000","last_became_active":"2024-09-21T13:59:32.273094+0000","last_became_peered":"2024-09-21T13:59:32.273094+0000","last_unstale":"2024-09-21T14:00:20.975159+0000","last_undegraded":"2024-09-21T14:00:20.975159+0000","last_fullsized":"2024-09-21T14:00:20.975159+0000","mapping_epoch":298,"log_start":"290'5300","ondisk_log_start":"290'5300","created":260,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4694,"log_dups_size":0,"ondisk_log_size":4694,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335151104,"num_objects":5114,"num_object_clones":0,"num_object_copies":15342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4581,"num_objects_unfound":0,"num_objects_dirty":5114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9994,"num_write_kb":483456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2852,"num_bytes_recovered":186908672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,4],"avail_no_missing":["3","4","6"],"object_location_counts":[{"shards":"3,4,6","objects":5114}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"290'20","reported_seq":616,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.090929+0000","last_change":"2024-09-21T13:58:17.789407+0000","last_active":"2024-09-21T14:00:15.090929+0000","last_peered":"2024-09-21T14:00:15.090929+0000","last_clean":"2024-09-21T14:00:15.090929+0000","last_became_active":"2024-09-21T13:58:17.765427+0000","last_became_peered":"2024-09-21T13:58:17.765427+0000","last_unstale":"2024-09-21T14:00:15.090929+0000","last_undegraded":"2024-09-21T14:00:15.090929+0000","last_fullsized":"2024-09-21T14:00:15.090929+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:58:13.464196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:13.464196+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:01:12.556495+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":457,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.060952+0000","last_change":"2024-09-21T13:57:36.707211+0000","last_active":"2024-09-21T14:00:15.060952+0000","last_peered":"2024-09-21T14:00:15.060952+0000","last_clean":"2024-09-21T14:00:15.060952+0000","last_became_active":"2024-09-21T13:57:12.704203+0000","last_became_peered":"2024-09-21T13:57:12.704203+0000","last_unstale":"2024-09-21T14:00:15.060952+0000","last_undegraded":"2024-09-21T14:00:15.060952+0000","last_fullsized":"2024-09-21T14:00:15.060952+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:36.707148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:36.707148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"300'12","reported_seq":543,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.067924+0000","last_change":"2024-09-21T13:59:38.738198+0000","last_active":"2024-09-21T14:00:15.067924+0000","last_peered":"2024-09-21T14:00:15.067924+0000","last_clean":"2024-09-21T14:00:15.067924+0000","last_became_active":"2024-09-21T13:59:32.224809+0000","last_became_peered":"2024-09-21T13:59:32.224809+0000","last_unstale":"2024-09-21T14:00:15.067924+0000","last_undegraded":"2024-09-21T14:00:15.067924+0000","last_fullsized":"2024-09-21T14:00:15.067924+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'12","last_scrub_stamp":"2024-09-21T13:59:38.738120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:38.738120+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:01:05.042603+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"68'1","reported_seq":543,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.091645+0000","last_change":"2024-09-21T13:59:31.392186+0000","last_active":"2024-09-21T14:00:15.091645+0000","last_peered":"2024-09-21T14:00:15.091645+0000","last_clean":"2024-09-21T14:00:15.091645+0000","last_became_active":"2024-09-21T13:52:04.313210+0000","last_became_peered":"2024-09-21T13:52:04.313210+0000","last_unstale":"2024-09-21T14:00:15.091645+0000","last_undegraded":"2024-09-21T14:00:15.091645+0000","last_fullsized":"2024-09-21T14:00:15.091645+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T13:59:31.392105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:31.392105+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:01:00.512112+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"312'9763","reported_seq":9879,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:18.885703+0000","last_change":"2024-09-21T13:57:53.213656+0000","last_active":"2024-09-21T14:00:18.885703+0000","last_peered":"2024-09-21T14:00:18.885703+0000","last_clean":"2024-09-21T14:00:18.885703+0000","last_became_active":"2024-09-21T13:57:53.213314+0000","last_became_peered":"2024-09-21T13:57:53.213314+0000","last_unstale":"2024-09-21T14:00:18.885703+0000","last_undegraded":"2024-09-21T14:00:18.885703+0000","last_fullsized":"2024-09-21T14:00:18.885703+0000","mapping_epoch":260,"log_start":"286'4400","ondisk_log_start":"286'4400","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":5363,"log_dups_size":0,"ondisk_log_size":5363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336003072,"num_objects":5127,"num_object_clones":0,"num_object_copies":15381,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9763,"num_write_kb":476480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"290'15","reported_seq":571,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.056173+0000","last_change":"2024-09-21T13:58:50.498597+0000","last_active":"2024-09-21T14:00:15.056173+0000","last_peered":"2024-09-21T14:00:15.056173+0000","last_clean":"2024-09-21T14:00:15.056173+0000","last_became_active":"2024-09-21T13:49:00.053016+0000","last_became_peered":"2024-09-21T13:49:00.053016+0000","last_unstale":"2024-09-21T14:00:15.056173+0000","last_undegraded":"2024-09-21T14:00:15.056173+0000","last_fullsized":"2024-09-21T14:00:15.056173+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:58:50.498369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:50.498369+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":559,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.052719+0000","last_change":"2024-09-21T13:58:17.783765+0000","last_active":"2024-09-21T14:00:15.052719+0000","last_peered":"2024-09-21T14:00:15.052719+0000","last_clean":"2024-09-21T14:00:15.052719+0000","last_became_active":"2024-09-21T13:58:17.782747+0000","last_became_peered":"2024-09-21T13:58:17.782747+0000","last_unstale":"2024-09-21T14:00:15.052719+0000","last_undegraded":"2024-09-21T14:00:15.052719+0000","last_fullsized":"2024-09-21T14:00:15.052719+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:53.894506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:53.894506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":474,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.089150+0000","last_change":"2024-09-21T14:00:10.562648+0000","last_active":"2024-09-21T14:00:15.089150+0000","last_peered":"2024-09-21T14:00:15.089150+0000","last_clean":"2024-09-21T14:00:15.089150+0000","last_became_active":"2024-09-21T13:57:12.680978+0000","last_became_peered":"2024-09-21T13:57:12.680978+0000","last_unstale":"2024-09-21T14:00:15.089150+0000","last_undegraded":"2024-09-21T14:00:15.089150+0000","last_fullsized":"2024-09-21T14:00:15.089150+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:00:10.562575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:00:10.562575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:01:19.199643+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":432,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.062884+0000","last_change":"2024-09-21T13:59:32.224001+0000","last_active":"2024-09-21T14:00:15.062884+0000","last_peered":"2024-09-21T14:00:15.062884+0000","last_clean":"2024-09-21T14:00:15.062884+0000","last_became_active":"2024-09-21T13:59:32.221628+0000","last_became_peered":"2024-09-21T13:59:32.221628+0000","last_unstale":"2024-09-21T14:00:15.062884+0000","last_undegraded":"2024-09-21T14:00:15.062884+0000","last_fullsized":"2024-09-21T14:00:15.062884+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:13.698091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:13.698091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"312'10274","reported_seq":12133,"reported_epoch":312,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T14:00:18.877097+0000","last_change":"2024-09-21T14:00:03.360270+0000","last_active":"2024-09-21T14:00:18.877097+0000","last_peered":"2024-09-21T14:00:18.877097+0000","last_clean":"2024-09-21T13:58:23.791074+0000","last_became_active":"2024-09-21T13:59:33.441400+0000","last_became_peered":"2024-09-21T13:59:33.441400+0000","last_unstale":"2024-09-21T14:00:18.877097+0000","last_undegraded":"2024-09-21T13:59:33.239445+0000","last_fullsized":"2024-09-21T13:59:33.192362+0000","mapping_epoch":299,"log_start":"288'5000","ondisk_log_start":"288'5000","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":5274,"log_dups_size":0,"ondisk_log_size":5274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":349700096,"num_objects":5336,"num_object_clones":0,"num_object_copies":16008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10274,"num_write_kb":499520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":866,"num_bytes_recovered":56754176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[{"shards":"3,5","objects":948}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"290'35","reported_seq":170,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.060717+0000","last_change":"2024-09-21T13:59:32.234258+0000","last_active":"2024-09-21T14:00:15.060717+0000","last_peered":"2024-09-21T14:00:15.060717+0000","last_clean":"2024-09-21T14:00:15.060717+0000","last_became_active":"2024-09-21T13:59:32.204184+0000","last_became_peered":"2024-09-21T13:59:32.204184+0000","last_unstale":"2024-09-21T14:00:15.060717+0000","last_undegraded":"2024-09-21T14:00:15.060717+0000","last_fullsized":"2024-09-21T14:00:15.060717+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-21T13:58:12.650656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:12.650656+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:43.038682+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":513,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.060133+0000","last_change":"2024-09-21T13:59:32.222536+0000","last_active":"2024-09-21T14:00:15.060133+0000","last_peered":"2024-09-21T14:00:15.060133+0000","last_clean":"2024-09-21T14:00:15.060133+0000","last_became_active":"2024-09-21T13:59:32.221051+0000","last_became_peered":"2024-09-21T13:59:32.221051+0000","last_unstale":"2024-09-21T14:00:15.060133+0000","last_undegraded":"2024-09-21T14:00:15.060133+0000","last_fullsized":"2024-09-21T14:00:15.060133+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:21.587493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:21.587493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":491,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.067088+0000","last_change":"2024-09-21T13:58:55.588614+0000","last_active":"2024-09-21T14:00:15.067088+0000","last_peered":"2024-09-21T14:00:15.067088+0000","last_clean":"2024-09-21T14:00:15.067088+0000","last_became_active":"2024-09-21T13:52:04.321146+0000","last_became_peered":"2024-09-21T13:52:04.321146+0000","last_unstale":"2024-09-21T14:00:15.067088+0000","last_undegraded":"2024-09-21T14:00:15.067088+0000","last_fullsized":"2024-09-21T14:00:15.067088+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:55.588554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:55.588554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:23.995144+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":451,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.059296+0000","last_change":"2024-09-21T13:59:32.223123+0000","last_active":"2024-09-21T14:00:15.059296+0000","last_peered":"2024-09-21T14:00:15.059296+0000","last_clean":"2024-09-21T14:00:15.059296+0000","last_became_active":"2024-09-21T13:59:32.219332+0000","last_became_peered":"2024-09-21T13:59:32.219332+0000","last_unstale":"2024-09-21T14:00:15.059296+0000","last_undegraded":"2024-09-21T14:00:15.059296+0000","last_fullsized":"2024-09-21T14:00:15.059296+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:19.537177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:19.537177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"312'9769","reported_seq":9885,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:19.519576+0000","last_change":"2024-09-21T13:57:53.515705+0000","last_active":"2024-09-21T14:00:19.519576+0000","last_peered":"2024-09-21T14:00:19.519576+0000","last_clean":"2024-09-21T14:00:19.519576+0000","last_became_active":"2024-09-21T13:57:53.515313+0000","last_became_peered":"2024-09-21T13:57:53.515313+0000","last_unstale":"2024-09-21T14:00:19.519576+0000","last_undegraded":"2024-09-21T14:00:19.519576+0000","last_fullsized":"2024-09-21T14:00:19.519576+0000","mapping_epoch":260,"log_start":"289'5070","ondisk_log_start":"289'5070","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4699,"log_dups_size":0,"ondisk_log_size":4699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334168064,"num_objects":5099,"num_object_clones":0,"num_object_copies":15297,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9769,"num_write_kb":475776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"290'18","reported_seq":588,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.091658+0000","last_change":"2024-09-21T13:59:28.340494+0000","last_active":"2024-09-21T14:00:15.091658+0000","last_peered":"2024-09-21T14:00:15.091658+0000","last_clean":"2024-09-21T14:00:15.091658+0000","last_became_active":"2024-09-21T13:57:12.680627+0000","last_became_peered":"2024-09-21T13:57:12.680627+0000","last_unstale":"2024-09-21T14:00:15.091658+0000","last_undegraded":"2024-09-21T14:00:15.091658+0000","last_fullsized":"2024-09-21T14:00:15.091658+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'18","last_scrub_stamp":"2024-09-21T13:59:28.340415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:28.340415+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:44.818651+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":479,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.061046+0000","last_change":"2024-09-21T13:59:32.201126+0000","last_active":"2024-09-21T14:00:15.061046+0000","last_peered":"2024-09-21T14:00:15.061046+0000","last_clean":"2024-09-21T14:00:15.061046+0000","last_became_active":"2024-09-21T13:59:32.200674+0000","last_became_peered":"2024-09-21T13:59:32.200674+0000","last_unstale":"2024-09-21T14:00:15.061046+0000","last_undegraded":"2024-09-21T14:00:15.061046+0000","last_fullsized":"2024-09-21T14:00:15.061046+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:21.582325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:21.582325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":456,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.069275+0000","last_change":"2024-09-21T13:58:11.377940+0000","last_active":"2024-09-21T14:00:15.069275+0000","last_peered":"2024-09-21T14:00:15.069275+0000","last_clean":"2024-09-21T14:00:15.069275+0000","last_became_active":"2024-09-21T13:52:04.322147+0000","last_became_peered":"2024-09-21T13:52:04.322147+0000","last_unstale":"2024-09-21T14:00:15.069275+0000","last_undegraded":"2024-09-21T14:00:15.069275+0000","last_fullsized":"2024-09-21T14:00:15.069275+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:11.377803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:11.377803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:19.656170+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":465,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.061312+0000","last_change":"2024-09-21T13:59:32.207006+0000","last_active":"2024-09-21T14:00:15.061312+0000","last_peered":"2024-09-21T14:00:15.061312+0000","last_clean":"2024-09-21T14:00:15.061312+0000","last_became_active":"2024-09-21T13:59:32.203767+0000","last_became_peered":"2024-09-21T13:59:32.203767+0000","last_unstale":"2024-09-21T14:00:15.061312+0000","last_undegraded":"2024-09-21T14:00:15.061312+0000","last_fullsized":"2024-09-21T14:00:15.061312+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:09.040062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:55:09.040062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"312'10301","reported_seq":11960,"reported_epoch":312,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T14:00:18.867742+0000","last_change":"2024-09-21T14:00:03.944965+0000","last_active":"2024-09-21T14:00:18.867742+0000","last_peered":"2024-09-21T14:00:18.867742+0000","last_clean":"2024-09-21T13:58:23.785212+0000","last_became_active":"2024-09-21T13:59:33.394359+0000","last_became_peered":"2024-09-21T13:59:33.394359+0000","last_unstale":"2024-09-21T14:00:18.867742+0000","last_undegraded":"2024-09-21T13:59:33.231231+0000","last_fullsized":"2024-09-21T13:59:33.191984+0000","mapping_epoch":299,"log_start":"288'5000","ondisk_log_start":"288'5000","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":5301,"log_dups_size":0,"ondisk_log_size":5301,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":349241344,"num_objects":5329,"num_object_clones":0,"num_object_copies":15987,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4493,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10301,"num_write_kb":500160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":765,"num_bytes_recovered":50135040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[{"shards":"0,5","objects":836}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"290'27","reported_seq":661,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.052286+0000","last_change":"2024-09-21T13:59:33.058912+0000","last_active":"2024-09-21T14:00:15.052286+0000","last_peered":"2024-09-21T14:00:15.052286+0000","last_clean":"2024-09-21T14:00:15.052286+0000","last_became_active":"2024-09-21T13:59:32.205453+0000","last_became_peered":"2024-09-21T13:59:32.205453+0000","last_unstale":"2024-09-21T14:00:15.052286+0000","last_undegraded":"2024-09-21T14:00:15.052286+0000","last_fullsized":"2024-09-21T14:00:15.052286+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'27","last_scrub_stamp":"2024-09-21T13:59:33.058859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:33.058859+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:35.203746+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":451,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.072185+0000","last_change":"2024-09-21T13:59:32.203889+0000","last_active":"2024-09-21T14:00:15.072185+0000","last_peered":"2024-09-21T14:00:15.072185+0000","last_clean":"2024-09-21T14:00:15.072185+0000","last_became_active":"2024-09-21T13:59:32.203172+0000","last_became_peered":"2024-09-21T13:59:32.203172+0000","last_unstale":"2024-09-21T14:00:15.072185+0000","last_undegraded":"2024-09-21T14:00:15.072185+0000","last_fullsized":"2024-09-21T14:00:15.072185+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T13:55:03.879706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:55:03.879706+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"300'14","reported_seq":635,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.063042+0000","last_change":"2024-09-21T13:59:32.220898+0000","last_active":"2024-09-21T14:00:15.063042+0000","last_peered":"2024-09-21T14:00:15.063042+0000","last_clean":"2024-09-21T14:00:15.063042+0000","last_became_active":"2024-09-21T13:59:32.206956+0000","last_became_peered":"2024-09-21T13:59:32.206956+0000","last_unstale":"2024-09-21T14:00:15.063042+0000","last_undegraded":"2024-09-21T14:00:15.063042+0000","last_fullsized":"2024-09-21T14:00:15.063042+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'11","last_scrub_stamp":"2024-09-21T13:56:31.616003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:56:31.616003+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":560,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.052459+0000","last_change":"2024-09-21T13:58:39.258717+0000","last_active":"2024-09-21T14:00:15.052459+0000","last_peered":"2024-09-21T14:00:15.052459+0000","last_clean":"2024-09-21T14:00:15.052459+0000","last_became_active":"2024-09-21T13:58:17.781952+0000","last_became_peered":"2024-09-21T13:58:17.781952+0000","last_unstale":"2024-09-21T14:00:15.052459+0000","last_undegraded":"2024-09-21T14:00:15.052459+0000","last_fullsized":"2024-09-21T14:00:15.052459+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:39.258659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:39.258659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":96,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.059630+0000","last_change":"2024-09-21T13:58:17.777374+0000","last_active":"2024-09-21T14:00:15.059630+0000","last_peered":"2024-09-21T14:00:15.059630+0000","last_clean":"2024-09-21T14:00:15.059630+0000","last_became_active":"2024-09-21T13:58:17.777025+0000","last_became_peered":"2024-09-21T13:58:17.777025+0000","last_unstale":"2024-09-21T14:00:15.059630+0000","last_undegraded":"2024-09-21T14:00:15.059630+0000","last_fullsized":"2024-09-21T14:00:15.059630+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:13.252516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:13.252516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:42.040885+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"290'12","reported_seq":500,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.062183+0000","last_change":"2024-09-21T13:59:32.247422+0000","last_active":"2024-09-21T14:00:15.062183+0000","last_peered":"2024-09-21T14:00:15.062183+0000","last_clean":"2024-09-21T14:00:15.062183+0000","last_became_active":"2024-09-21T13:59:32.211306+0000","last_became_peered":"2024-09-21T13:59:32.211306+0000","last_unstale":"2024-09-21T14:00:15.062183+0000","last_undegraded":"2024-09-21T14:00:15.062183+0000","last_fullsized":"2024-09-21T14:00:15.062183+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'6","last_scrub_stamp":"2024-09-21T13:55:12.066873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:55:12.066873+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":552,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.053152+0000","last_change":"2024-09-21T13:57:37.106286+0000","last_active":"2024-09-21T14:00:15.053152+0000","last_peered":"2024-09-21T14:00:15.053152+0000","last_clean":"2024-09-21T14:00:15.053152+0000","last_became_active":"2024-09-21T13:57:12.702621+0000","last_became_peered":"2024-09-21T13:57:12.702621+0000","last_unstale":"2024-09-21T14:00:15.053152+0000","last_undegraded":"2024-09-21T14:00:15.053152+0000","last_fullsized":"2024-09-21T14:00:15.053152+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:37.106238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:37.106238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":497,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.059921+0000","last_change":"2024-09-21T13:59:32.222277+0000","last_active":"2024-09-21T14:00:15.059921+0000","last_peered":"2024-09-21T14:00:15.059921+0000","last_clean":"2024-09-21T14:00:15.059921+0000","last_became_active":"2024-09-21T13:59:32.219525+0000","last_became_peered":"2024-09-21T13:59:32.219525+0000","last_unstale":"2024-09-21T14:00:15.059921+0000","last_undegraded":"2024-09-21T14:00:15.059921+0000","last_fullsized":"2024-09-21T14:00:15.059921+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:02.838181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:55:02.838181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"290'15","reported_seq":463,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.061547+0000","last_change":"2024-09-21T13:59:32.238689+0000","last_active":"2024-09-21T14:00:15.061547+0000","last_peered":"2024-09-21T14:00:15.061547+0000","last_clean":"2024-09-21T14:00:15.061547+0000","last_became_active":"2024-09-21T13:59:32.209166+0000","last_became_peered":"2024-09-21T13:59:32.209166+0000","last_unstale":"2024-09-21T14:00:15.061547+0000","last_undegraded":"2024-09-21T14:00:15.061547+0000","last_fullsized":"2024-09-21T14:00:15.061547+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:55:04.990842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:55:04.990842+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"61'1","reported_seq":515,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.063131+0000","last_change":"2024-09-21T13:59:32.222826+0000","last_active":"2024-09-21T14:00:15.063131+0000","last_peered":"2024-09-21T14:00:15.063131+0000","last_clean":"2024-09-21T14:00:15.063131+0000","last_became_active":"2024-09-21T13:59:32.219515+0000","last_became_peered":"2024-09-21T13:59:32.219515+0000","last_unstale":"2024-09-21T14:00:15.063131+0000","last_undegraded":"2024-09-21T14:00:15.063131+0000","last_fullsized":"2024-09-21T14:00:15.063131+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:58:07.711028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:07.711028+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":448,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.061712+0000","last_change":"2024-09-21T13:59:32.211754+0000","last_active":"2024-09-21T14:00:15.061712+0000","last_peered":"2024-09-21T14:00:15.061712+0000","last_clean":"2024-09-21T14:00:15.061712+0000","last_became_active":"2024-09-21T13:59:32.210586+0000","last_became_peered":"2024-09-21T13:59:32.210586+0000","last_unstale":"2024-09-21T14:00:15.061712+0000","last_undegraded":"2024-09-21T14:00:15.061712+0000","last_fullsized":"2024-09-21T14:00:15.061712+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:28.014760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:28.014760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":490,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.067308+0000","last_change":"2024-09-21T13:59:36.677306+0000","last_active":"2024-09-21T14:00:15.067308+0000","last_peered":"2024-09-21T14:00:15.067308+0000","last_clean":"2024-09-21T14:00:15.067308+0000","last_became_active":"2024-09-21T13:59:32.205803+0000","last_became_peered":"2024-09-21T13:59:32.205803+0000","last_unstale":"2024-09-21T14:00:15.067308+0000","last_undegraded":"2024-09-21T14:00:15.067308+0000","last_fullsized":"2024-09-21T14:00:15.067308+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:36.677259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:36.677259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:49.697302+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"290'15","reported_seq":585,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.052896+0000","last_change":"2024-09-21T13:59:32.247060+0000","last_active":"2024-09-21T14:00:15.052896+0000","last_peered":"2024-09-21T14:00:15.052896+0000","last_clean":"2024-09-21T14:00:15.052896+0000","last_became_active":"2024-09-21T13:59:32.226975+0000","last_became_peered":"2024-09-21T13:59:32.226975+0000","last_unstale":"2024-09-21T14:00:15.052896+0000","last_undegraded":"2024-09-21T14:00:15.052896+0000","last_fullsized":"2024-09-21T14:00:15.052896+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:57:24.124604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:57:24.124604+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":384,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.060312+0000","last_change":"2024-09-21T13:59:32.212325+0000","last_active":"2024-09-21T14:00:15.060312+0000","last_peered":"2024-09-21T14:00:15.060312+0000","last_clean":"2024-09-21T14:00:15.060312+0000","last_became_active":"2024-09-21T13:59:32.208749+0000","last_became_peered":"2024-09-21T13:59:32.208749+0000","last_unstale":"2024-09-21T14:00:15.060312+0000","last_undegraded":"2024-09-21T14:00:15.060312+0000","last_fullsized":"2024-09-21T14:00:15.060312+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:20.574745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:20.574745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"300'14","reported_seq":568,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.063321+0000","last_change":"2024-09-21T13:59:32.208253+0000","last_active":"2024-09-21T14:00:15.063321+0000","last_peered":"2024-09-21T14:00:15.063321+0000","last_clean":"2024-09-21T14:00:15.063321+0000","last_became_active":"2024-09-21T13:59:32.206926+0000","last_became_peered":"2024-09-21T13:59:32.206926+0000","last_unstale":"2024-09-21T14:00:15.063321+0000","last_undegraded":"2024-09-21T14:00:15.063321+0000","last_fullsized":"2024-09-21T14:00:15.063321+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'10","last_scrub_stamp":"2024-09-21T13:54:20.946782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:20.946782+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":539,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.089672+0000","last_change":"2024-09-21T13:58:54.805670+0000","last_active":"2024-09-21T14:00:15.089672+0000","last_peered":"2024-09-21T14:00:15.089672+0000","last_clean":"2024-09-21T14:00:15.089672+0000","last_became_active":"2024-09-21T13:52:04.315865+0000","last_became_peered":"2024-09-21T13:52:04.315865+0000","last_unstale":"2024-09-21T14:00:15.089672+0000","last_undegraded":"2024-09-21T14:00:15.089672+0000","last_fullsized":"2024-09-21T14:00:15.089672+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:54.805619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:54.805619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"290'20","reported_seq":499,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.063049+0000","last_change":"2024-09-21T13:59:32.236530+0000","last_active":"2024-09-21T14:00:15.063049+0000","last_peered":"2024-09-21T14:00:15.063049+0000","last_clean":"2024-09-21T14:00:15.063049+0000","last_became_active":"2024-09-21T13:59:32.208332+0000","last_became_peered":"2024-09-21T13:59:32.208332+0000","last_unstale":"2024-09-21T14:00:15.063049+0000","last_undegraded":"2024-09-21T14:00:15.063049+0000","last_fullsized":"2024-09-21T14:00:15.063049+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:57:37.671391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:57:37.671391+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":491,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.059233+0000","last_change":"2024-09-21T13:59:32.212737+0000","last_active":"2024-09-21T14:00:15.059233+0000","last_peered":"2024-09-21T14:00:15.059233+0000","last_clean":"2024-09-21T14:00:15.059233+0000","last_became_active":"2024-09-21T13:59:32.209804+0000","last_became_peered":"2024-09-21T13:59:32.209804+0000","last_unstale":"2024-09-21T14:00:15.059233+0000","last_undegraded":"2024-09-21T14:00:15.059233+0000","last_fullsized":"2024-09-21T14:00:15.059233+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:25.571256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:25.571256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"68'8","reported_seq":678,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.052702+0000","last_change":"2024-09-21T13:59:32.104294+0000","last_active":"2024-09-21T14:00:15.052702+0000","last_peered":"2024-09-21T14:00:15.052702+0000","last_clean":"2024-09-21T14:00:15.052702+0000","last_became_active":"2024-09-21T13:49:02.067487+0000","last_became_peered":"2024-09-21T13:49:02.067487+0000","last_unstale":"2024-09-21T14:00:15.052702+0000","last_undegraded":"2024-09-21T14:00:15.052702+0000","last_fullsized":"2024-09-21T14:00:15.052702+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:59:32.104224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:32.104224+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:36.249102+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":388,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.060878+0000","last_change":"2024-09-21T13:59:32.213015+0000","last_active":"2024-09-21T14:00:15.060878+0000","last_peered":"2024-09-21T14:00:15.060878+0000","last_clean":"2024-09-21T14:00:15.060878+0000","last_became_active":"2024-09-21T13:59:32.211472+0000","last_became_peered":"2024-09-21T13:59:32.211472+0000","last_unstale":"2024-09-21T14:00:15.060878+0000","last_undegraded":"2024-09-21T14:00:15.060878+0000","last_fullsized":"2024-09-21T14:00:15.060878+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:22.647191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:22.647191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"290'19","reported_seq":605,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.089308+0000","last_change":"2024-09-21T13:58:17.790158+0000","last_active":"2024-09-21T14:00:15.089308+0000","last_peered":"2024-09-21T14:00:15.089308+0000","last_clean":"2024-09-21T14:00:15.089308+0000","last_became_active":"2024-09-21T13:58:17.771497+0000","last_became_peered":"2024-09-21T13:58:17.771497+0000","last_unstale":"2024-09-21T14:00:15.089308+0000","last_undegraded":"2024-09-21T14:00:15.089308+0000","last_fullsized":"2024-09-21T14:00:15.089308+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:54:56.539374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:56.539374+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:16.539347+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":125,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.059852+0000","last_change":"2024-09-21T13:59:32.202097+0000","last_active":"2024-09-21T14:00:15.059852+0000","last_peered":"2024-09-21T14:00:15.059852+0000","last_clean":"2024-09-21T14:00:15.059852+0000","last_became_active":"2024-09-21T13:59:32.201568+0000","last_became_peered":"2024-09-21T13:59:32.201568+0000","last_unstale":"2024-09-21T14:00:15.059852+0000","last_undegraded":"2024-09-21T14:00:15.059852+0000","last_fullsized":"2024-09-21T14:00:15.059852+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:01.228213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:01.228213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":552,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.056250+0000","last_change":"2024-09-21T13:57:35.409292+0000","last_active":"2024-09-21T14:00:15.056250+0000","last_peered":"2024-09-21T14:00:15.056250+0000","last_clean":"2024-09-21T14:00:15.056250+0000","last_became_active":"2024-09-21T13:57:12.683646+0000","last_became_peered":"2024-09-21T13:57:12.683646+0000","last_unstale":"2024-09-21T14:00:15.056250+0000","last_undegraded":"2024-09-21T14:00:15.056250+0000","last_fullsized":"2024-09-21T14:00:15.056250+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:35.409237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:35.409237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":445,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.060728+0000","last_change":"2024-09-21T13:57:39.643527+0000","last_active":"2024-09-21T14:00:15.060728+0000","last_peered":"2024-09-21T14:00:15.060728+0000","last_clean":"2024-09-21T14:00:15.060728+0000","last_became_active":"2024-09-21T13:57:12.703037+0000","last_became_peered":"2024-09-21T13:57:12.703037+0000","last_unstale":"2024-09-21T14:00:15.060728+0000","last_undegraded":"2024-09-21T14:00:15.060728+0000","last_fullsized":"2024-09-21T14:00:15.060728+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:39.643458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:39.643458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"290'15","reported_seq":607,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.056485+0000","last_change":"2024-09-21T13:58:39.977717+0000","last_active":"2024-09-21T14:00:15.056485+0000","last_peered":"2024-09-21T14:00:15.056485+0000","last_clean":"2024-09-21T14:00:15.056485+0000","last_became_active":"2024-09-21T13:58:17.764120+0000","last_became_peered":"2024-09-21T13:58:17.764120+0000","last_unstale":"2024-09-21T14:00:15.056485+0000","last_undegraded":"2024-09-21T14:00:15.056485+0000","last_fullsized":"2024-09-21T14:00:15.056485+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:58:39.977466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:39.977466+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":495,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.059776+0000","last_change":"2024-09-21T13:59:32.223082+0000","last_active":"2024-09-21T14:00:15.059776+0000","last_peered":"2024-09-21T14:00:15.059776+0000","last_clean":"2024-09-21T14:00:15.059776+0000","last_became_active":"2024-09-21T13:59:32.221854+0000","last_became_peered":"2024-09-21T13:59:32.221854+0000","last_unstale":"2024-09-21T14:00:15.059776+0000","last_undegraded":"2024-09-21T14:00:15.059776+0000","last_fullsized":"2024-09-21T14:00:15.059776+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:17.691193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:17.691193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":431,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.060862+0000","last_change":"2024-09-21T13:57:12.703746+0000","last_active":"2024-09-21T14:00:15.060862+0000","last_peered":"2024-09-21T14:00:15.060862+0000","last_clean":"2024-09-21T14:00:15.060862+0000","last_became_active":"2024-09-21T13:57:12.703441+0000","last_became_peered":"2024-09-21T13:57:12.703441+0000","last_unstale":"2024-09-21T14:00:15.060862+0000","last_undegraded":"2024-09-21T14:00:15.060862+0000","last_fullsized":"2024-09-21T14:00:15.060862+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:19.999125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:19.999125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":466,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.052195+0000","last_change":"2024-09-21T13:58:47.334564+0000","last_active":"2024-09-21T14:00:15.052195+0000","last_peered":"2024-09-21T14:00:15.052195+0000","last_clean":"2024-09-21T14:00:15.052195+0000","last_became_active":"2024-09-21T13:50:29.689045+0000","last_became_peered":"2024-09-21T13:50:29.689045+0000","last_unstale":"2024-09-21T14:00:15.052195+0000","last_undegraded":"2024-09-21T14:00:15.052195+0000","last_fullsized":"2024-09-21T14:00:15.052195+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:47.334504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:47.334504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"290'19","reported_seq":575,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.061167+0000","last_change":"2024-09-21T13:59:32.277742+0000","last_active":"2024-09-21T14:00:15.061167+0000","last_peered":"2024-09-21T14:00:15.061167+0000","last_clean":"2024-09-21T14:00:15.061167+0000","last_became_active":"2024-09-21T13:59:32.231567+0000","last_became_peered":"2024-09-21T13:59:32.231567+0000","last_unstale":"2024-09-21T14:00:15.061167+0000","last_undegraded":"2024-09-21T14:00:15.061167+0000","last_fullsized":"2024-09-21T14:00:15.061167+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:54:22.978267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:22.978267+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":536,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.053235+0000","last_change":"2024-09-21T13:59:32.206128+0000","last_active":"2024-09-21T14:00:15.053235+0000","last_peered":"2024-09-21T14:00:15.053235+0000","last_clean":"2024-09-21T14:00:15.053235+0000","last_became_active":"2024-09-21T13:59:32.205826+0000","last_became_peered":"2024-09-21T13:59:32.205826+0000","last_unstale":"2024-09-21T14:00:15.053235+0000","last_undegraded":"2024-09-21T14:00:15.053235+0000","last_fullsized":"2024-09-21T14:00:15.053235+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:17.305144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:17.305144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":501,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.059453+0000","last_change":"2024-09-21T13:59:32.219508+0000","last_active":"2024-09-21T14:00:15.059453+0000","last_peered":"2024-09-21T14:00:15.059453+0000","last_clean":"2024-09-21T14:00:15.059453+0000","last_became_active":"2024-09-21T13:59:32.214004+0000","last_became_peered":"2024-09-21T13:59:32.214004+0000","last_unstale":"2024-09-21T14:00:15.059453+0000","last_undegraded":"2024-09-21T14:00:15.059453+0000","last_fullsized":"2024-09-21T14:00:15.059453+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:36.525417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:36.525417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":497,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.061042+0000","last_change":"2024-09-21T13:59:28.124373+0000","last_active":"2024-09-21T14:00:15.061042+0000","last_peered":"2024-09-21T14:00:15.061042+0000","last_clean":"2024-09-21T14:00:15.061042+0000","last_became_active":"2024-09-21T13:58:17.778115+0000","last_became_peered":"2024-09-21T13:58:17.778115+0000","last_unstale":"2024-09-21T14:00:15.061042+0000","last_undegraded":"2024-09-21T14:00:15.061042+0000","last_fullsized":"2024-09-21T14:00:15.061042+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:28.124296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:28.124296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:40.216842+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"290'8","reported_seq":521,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.061972+0000","last_change":"2024-09-21T13:59:26.034457+0000","last_active":"2024-09-21T14:00:15.061972+0000","last_peered":"2024-09-21T14:00:15.061972+0000","last_clean":"2024-09-21T14:00:15.061972+0000","last_became_active":"2024-09-21T13:58:17.775079+0000","last_became_peered":"2024-09-21T13:58:17.775079+0000","last_unstale":"2024-09-21T14:00:15.061972+0000","last_undegraded":"2024-09-21T14:00:15.061972+0000","last_fullsized":"2024-09-21T14:00:15.061972+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'8","last_scrub_stamp":"2024-09-21T13:59:26.034372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:26.034372+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:38.948765+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":485,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.062230+0000","last_change":"2024-09-21T13:58:16.457518+0000","last_active":"2024-09-21T14:00:15.062230+0000","last_peered":"2024-09-21T14:00:15.062230+0000","last_clean":"2024-09-21T14:00:15.062230+0000","last_became_active":"2024-09-21T13:57:12.684372+0000","last_became_peered":"2024-09-21T13:57:12.684372+0000","last_unstale":"2024-09-21T14:00:15.062230+0000","last_undegraded":"2024-09-21T14:00:15.062230+0000","last_fullsized":"2024-09-21T14:00:15.062230+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:16.457445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:16.457445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":107,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.060026+0000","last_change":"2024-09-21T13:59:32.204646+0000","last_active":"2024-09-21T14:00:15.060026+0000","last_peered":"2024-09-21T14:00:15.060026+0000","last_clean":"2024-09-21T14:00:15.060026+0000","last_became_active":"2024-09-21T13:59:32.204380+0000","last_became_peered":"2024-09-21T13:59:32.204380+0000","last_unstale":"2024-09-21T14:00:15.060026+0000","last_undegraded":"2024-09-21T14:00:15.060026+0000","last_fullsized":"2024-09-21T14:00:15.060026+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:51.254456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:53:51.254456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:00:41.002185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":434,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.059539+0000","last_change":"2024-09-21T13:59:32.212615+0000","last_active":"2024-09-21T14:00:15.059539+0000","last_peered":"2024-09-21T14:00:15.059539+0000","last_clean":"2024-09-21T14:00:15.059539+0000","last_became_active":"2024-09-21T13:59:32.210134+0000","last_became_peered":"2024-09-21T13:59:32.210134+0000","last_unstale":"2024-09-21T14:00:15.059539+0000","last_undegraded":"2024-09-21T14:00:15.059539+0000","last_fullsized":"2024-09-21T14:00:15.059539+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:08.852789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:55:08.852789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":537,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.057113+0000","last_change":"2024-09-21T13:58:17.706373+0000","last_active":"2024-09-21T14:00:15.057113+0000","last_peered":"2024-09-21T14:00:15.057113+0000","last_clean":"2024-09-21T14:00:15.057113+0000","last_became_active":"2024-09-21T13:49:04.075562+0000","last_became_peered":"2024-09-21T13:49:04.075562+0000","last_unstale":"2024-09-21T14:00:15.057113+0000","last_undegraded":"2024-09-21T14:00:15.057113+0000","last_fullsized":"2024-09-21T14:00:15.057113+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:17.706325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:17.706325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":477,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.062064+0000","last_change":"2024-09-21T13:58:13.776632+0000","last_active":"2024-09-21T14:00:15.062064+0000","last_peered":"2024-09-21T14:00:15.062064+0000","last_clean":"2024-09-21T14:00:15.062064+0000","last_became_active":"2024-09-21T13:57:12.704042+0000","last_became_peered":"2024-09-21T13:57:12.704042+0000","last_unstale":"2024-09-21T14:00:15.062064+0000","last_undegraded":"2024-09-21T14:00:15.062064+0000","last_fullsized":"2024-09-21T14:00:15.062064+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:13.776572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:13.776572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"290'19","reported_seq":128,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.060247+0000","last_change":"2024-09-21T14:00:09.169916+0000","last_active":"2024-09-21T14:00:15.060247+0000","last_peered":"2024-09-21T14:00:15.060247+0000","last_clean":"2024-09-21T14:00:15.060247+0000","last_became_active":"2024-09-21T13:58:17.777573+0000","last_became_peered":"2024-09-21T13:58:17.777573+0000","last_unstale":"2024-09-21T14:00:15.060247+0000","last_undegraded":"2024-09-21T14:00:15.060247+0000","last_fullsized":"2024-09-21T14:00:15.060247+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'19","last_scrub_stamp":"2024-09-21T14:00:09.168921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:00:09.168921+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:01:22.900349+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":403,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T14:00:15.060792+0000","last_change":"2024-09-21T13:59:32.203045+0000","last_active":"2024-09-21T14:00:15.060792+0000","last_peered":"2024-09-21T14:00:15.060792+0000","last_clean":"2024-09-21T14:00:15.060792+0000","last_became_active":"2024-09-21T13:59:32.202225+0000","last_became_peered":"2024-09-21T13:59:32.202225+0000","last_unstale":"2024-09-21T14:00:15.060792+0000","last_undegraded":"2024-09-21T14:00:15.060792+0000","last_fullsized":"2024-09-21T14:00:15.060792+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:40.650832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:40.650832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":5443026968,"num_objects":83055,"num_object_clones":0,"num_object_copies":249165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34407,"num_objects_misplaced":4581,"num_objects_unfound":0,"num_objects_dirty":83055,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159757,"num_write_kb":7769921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13872,"num_bytes_recovered":909115392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14771691520,"data_stored":14771683376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81380,"ondisk_log_size":81380,"up":48,"acting":40,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1114,"num_read_kb":871,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":176,"num_bytes_recovered":1914,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":687,"ondisk_log_size":687,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":55,"num_read_kb":50,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":219,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":298,"seq":1279900254219,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1225956,"kb_used_data":1116436,"kb_used_omap":4,"kb_used_meta":109499,"kb_avail":92519196,"statfs":{"total":95995035648,"available":94739656704,"internally_reserved":0,"allocated":1143230464,"data_stored":1142640809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":112127656},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463058,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1326928,"kb_used_data":1006976,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":92418224,"statfs":{"total":95995035648,"available":94636261376,"internally_reserved":0,"allocated":1031143424,"data_stored":1030573164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":327612870},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724697,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3523796,"kb_used_data":3059460,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":90221356,"statfs":{"total":95995035648,"available":92386668544,"internally_reserved":0,"allocated":3132887040,"data_stored":3132313080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":475462085},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920927,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1706476,"kb_used_data":1346268,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":92038676,"statfs":{"total":95995035648,"available":94247604224,"internally_reserved":0,"allocated":1378578432,"data_stored":1377979574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":368835013},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":240,"seq":1030792151079,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2873264,"kb_used_data":2646624,"kb_used_omap":4,"kb_used_meta":226619,"kb_avail":90871888,"statfs":{"total":95995035648,"available":93052813312,"internally_reserved":0,"allocated":2710142976,"data_stored":2709540852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4443,"internal_metadata":232058533},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":130,"seq":558345748580,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1536012,"kb_used_data":1008188,"kb_used_omap":4,"kb_used_meta":527803,"kb_avail":92209140,"statfs":{"total":95995035648,"available":94422159360,"internally_reserved":0,"allocated":1032384512,"data_stored":1031818522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4441,"internal_metadata":540470951},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575025,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3514384,"kb_used_data":2973504,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":90230768,"statfs":{"total":95995035648,"available":92396306432,"internally_reserved":0,"allocated":3044868096,"data_stored":3044233142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":553646533},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738551,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1835172,"kb_used_data":1362516,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":91909980,"statfs":{"total":95995035648,"available":94115819520,"internally_reserved":0,"allocated":1395216384,"data_stored":1394626002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1383923712,"data_stored":1383923712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3042775040,"data_stored":3042770968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1024000000,"data_stored":1024000000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2684354560,"data_stored":2684354560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1366491136,"data_stored":1366491136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3104575488,"data_stored":3104571416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1025835008,"data_stored":1025835008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1139736576,"data_stored":1139736576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T14:00:22.431 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-21T14:00:22.431 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd pool set .mgr pg_num 11 2024-09-21T14:00:22.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:22.448 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:22.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:00:22.483 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:22 smithi049 ceph-mon[40489]: osdmap e313: 8 total, 8 up, 8 in 2024-09-21T14:00:22.483 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:22 smithi049 ceph-mon[40489]: from='client.25622 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:00:22.483 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:22 smithi049 ceph-mon[30080]: osdmap e313: 8 total, 8 up, 8 in 2024-09-21T14:00:22.483 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:22 smithi049 ceph-mon[30080]: from='client.25622 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:00:22.549 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:22.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:22.620 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:22.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:22.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:22 smithi187 ceph-mon[33032]: osdmap e313: 8 total, 8 up, 8 in 2024-09-21T14:00:22.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:22 smithi187 ceph-mon[33032]: from='client.25622 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:00:22.721 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:22.749 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:22.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:22.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:22.851 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:00:22.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:22.885 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:22.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:00:22.986 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:00:23.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:23.030 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:23.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:00:23.109 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:00:23.117 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:00:23.131 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:23.210 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:23.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:23.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:23.253 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:23 smithi049 ceph-mon[30080]: pgmap v569: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 3 active+recovery_wait+undersized+degraded+remapped, 139 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 853 op/s; 34407/249762 objects degraded (13.776%); 4581/249762 objects misplaced (1.834%); 22 MiB/s, 356 objects/s recovering 2024-09-21T14:00:23.253 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:23 smithi049 ceph-mon[40489]: pgmap v569: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 3 active+recovery_wait+undersized+degraded+remapped, 139 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 853 op/s; 34407/249762 objects degraded (13.776%); 4581/249762 objects misplaced (1.834%); 22 MiB/s, 356 objects/s recovering 2024-09-21T14:00:23.312 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:00:23.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:23.355 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:23.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:00:23.456 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:23.495 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:23.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:23.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:23.596 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:23.632 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:23.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:23.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:23.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:23 smithi187 ceph-mon[33032]: pgmap v569: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 3 active+recovery_wait+undersized+degraded+remapped, 139 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 853 op/s; 34407/249762 objects degraded (13.776%); 4581/249762 objects misplaced (1.834%); 22 MiB/s, 356 objects/s recovering 2024-09-21T14:00:23.733 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:23.774 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:23.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:23.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:23.876 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:23.911 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:23.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:23.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:24.013 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:24.047 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:24.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:24.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:24.149 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:24.187 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:24.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:24.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:24.259 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-09-21T14:00:24.289 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:00:24.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:24.330 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:24.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:00:24.431 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:24.470 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:24.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:24.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:24.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:24 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/97573738' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:00:24.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:24 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/97573738' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:00:24.572 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:24.607 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:24.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:24.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:24.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:24 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/97573738' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:00:24.709 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:00:24.739 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:24.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:24.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:00:24.840 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:00:24.875 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:24.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:24.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:00:24.977 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:00:25.007 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:25.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:25.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:00:25.109 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:25.142 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:25.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:25.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:25.243 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:25.270 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:25.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:25.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:25.371 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:25.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:25.404 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:25.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:25.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:25 smithi049 ceph-mon[40489]: pgmap v570: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 3 active+recovery_wait+undersized+degraded+remapped, 139 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 777 op/s; 34407/244497 objects degraded (14.073%); 4581/244497 objects misplaced (1.874%); 9.7 MiB/s, 154 objects/s recovering 2024-09-21T14:00:25.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:25 smithi049 ceph-mon[40489]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:00:25.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:25 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/97573738' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T14:00:25.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:25 smithi049 ceph-mon[40489]: osdmap e314: 8 total, 8 up, 8 in 2024-09-21T14:00:25.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:25 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 34407/244497 objects degraded (14.073%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T14:00:25.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:25 smithi049 ceph-mon[30080]: pgmap v570: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 3 active+recovery_wait+undersized+degraded+remapped, 139 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 777 op/s; 34407/244497 objects degraded (14.073%); 4581/244497 objects misplaced (1.874%); 9.7 MiB/s, 154 objects/s recovering 2024-09-21T14:00:25.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:25 smithi049 ceph-mon[30080]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:00:25.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:25 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/97573738' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T14:00:25.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:25 smithi049 ceph-mon[30080]: osdmap e314: 8 total, 8 up, 8 in 2024-09-21T14:00:25.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:25 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 34407/244497 objects degraded (14.073%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T14:00:25.506 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:25.537 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:25.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:25.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:25.639 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:25.672 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:25.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:25.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:25.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:25 smithi187 ceph-mon[33032]: pgmap v570: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 3 active+recovery_wait+undersized+degraded+remapped, 139 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 777 op/s; 34407/244497 objects degraded (14.073%); 4581/244497 objects misplaced (1.874%); 9.7 MiB/s, 154 objects/s recovering 2024-09-21T14:00:25.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:25 smithi187 ceph-mon[33032]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:00:25.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:25 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/97573738' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T14:00:25.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:25 smithi187 ceph-mon[33032]: osdmap e314: 8 total, 8 up, 8 in 2024-09-21T14:00:25.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:25 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 34407/244497 objects degraded (14.073%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T14:00:25.773 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:25.808 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:25.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:25.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:25.909 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:00:25.948 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:25.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:25.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:00:26.050 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:26.074 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:26.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:26.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:26.176 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:00:26.216 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:26.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:26.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:00:26.318 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:26.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:26.354 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:26.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:26.455 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:26.486 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:26.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:26.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:26.588 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:26.622 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:26.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:26.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:26.724 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:26.762 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:26.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:26.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:26.863 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:00:26.896 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:26.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:26.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:00:26.998 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:27.035 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:27.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:27.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:27.137 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:27.172 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:27.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:27.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:27.273 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:27.311 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:27.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:27.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:27.413 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:00:27.449 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:27.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:27.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:00:27.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:27 smithi049 ceph-mon[30080]: pgmap v572: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovery_wait+undersized+degraded+remapped, 139 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 950 op/s; 32412/239157 objects degraded (13.553%); 4581/239157 objects misplaced (1.915%); 21 MiB/s, 342 objects/s recovering 2024-09-21T14:00:27.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:27 smithi049 ceph-mon[40489]: pgmap v572: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovery_wait+undersized+degraded+remapped, 139 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 950 op/s; 32412/239157 objects degraded (13.553%); 4581/239157 objects misplaced (1.915%); 21 MiB/s, 342 objects/s recovering 2024-09-21T14:00:27.551 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:00:27.594 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:27.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:27.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:00:27.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:27 smithi187 ceph-mon[33032]: pgmap v572: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovery_wait+undersized+degraded+remapped, 139 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 950 op/s; 32412/239157 objects degraded (13.553%); 4581/239157 objects misplaced (1.915%); 21 MiB/s, 342 objects/s recovering 2024-09-21T14:00:27.696 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:00:27.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:27.739 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:27.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:00:27.840 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:00:27.881 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:27.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:27.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:00:27.983 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:28.018 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:28.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:28.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:28.119 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:00:28.166 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:28.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:28.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:00:28.267 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:00:28.268 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:00:28.268 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:28.309 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:28.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:28.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:28.321 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:00:28.321 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:00:28.351 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:00:28.352 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:00:28.385 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:00:28.386 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:00:28.412 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:28.414 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:00:28.414 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:00:28.445 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:28.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:28.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:28.453 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:00:28.454 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:00:28.454 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:28 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/387628398' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:00:28.454 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:28 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/387628398' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:00:28.488 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:00:28.488 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:00:28.523 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:00:28.523 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:00:28.547 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:28.588 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:28.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:28.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:28.689 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:28.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:28 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/387628398' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:00:28.723 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:28.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:28.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:28.825 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:00:28.854 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:28.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:28.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:00:28.956 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:28.988 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:28.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:28.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:29.091 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:29.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:29.136 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:29.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:29.237 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:00:29.274 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:29.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:29.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:00:29.376 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:29.414 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:29.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:29.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:29.431 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T14:00:29.431 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:00:29.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:29 smithi187 ceph-mon[33032]: pgmap v573: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 3 active+recovery_wait+undersized+degraded+remapped, 139 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 723 op/s; 32137/235758 objects degraded (13.631%); 4325/235758 objects misplaced (1.835%); 11 MiB/s, 171 objects/s recovering 2024-09-21T14:00:29.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:29 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:00:29.516 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:29.547 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:29.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:29.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:29.649 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:29.678 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T14:00:29.688 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:29.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:29.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:29.713 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:29 smithi049 ceph-mon[40489]: pgmap v573: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 3 active+recovery_wait+undersized+degraded+remapped, 139 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 723 op/s; 32137/235758 objects degraded (13.631%); 4325/235758 objects misplaced (1.835%); 11 MiB/s, 171 objects/s recovering 2024-09-21T14:00:29.713 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:29 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:00:29.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:29 smithi049 ceph-mon[30080]: pgmap v573: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 3 active+recovery_wait+undersized+degraded+remapped, 139 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 723 op/s; 32137/235758 objects degraded (13.631%); 4325/235758 objects misplaced (1.835%); 11 MiB/s, 171 objects/s recovering 2024-09-21T14:00:29.713 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:29 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:00:29.790 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:29.820 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:29.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:29.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:29.922 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:00:29.944 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:00:29 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:00:29.969 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:29.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:29.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:00:30.070 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:00:30.114 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:30.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:30.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:00:30.216 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:00:30.253 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:30.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:30.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:00:30.355 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:30.397 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:30.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:30.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:30.498 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:00:30.535 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:30.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:30.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:00:30.580 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:30 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:00:30.580 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:30 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 32137/235758 objects degraded (13.631%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T14:00:30.580 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:30 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:00:30.580 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:30 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 32137/235758 objects degraded (13.631%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T14:00:30.637 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:30.675 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:30.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:30.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:30.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:30 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:00:30.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:30 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 32137/235758 objects degraded (13.631%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T14:00:30.719 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:00:30.777 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:00:30.824 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:30.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:30.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:00:30.840 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:00:30 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:00:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:00:30.925 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:00:30.963 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:30.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:30.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:00:31.065 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:31.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:31.099 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:31.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:31.200 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:00:31.234 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:31.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:31.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:00:31.335 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:31.357 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:31.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:31.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:31.459 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:00:31.496 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:31.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:31.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:00:31.598 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:00:31.631 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:31.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:31.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:00:31.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:31 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:00:31.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:31 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:00:31.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:31 smithi187 ceph-mon[33032]: pgmap v574: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovery_wait+undersized+degraded+remapped, 139 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 588 B/s rd, 0 B/s wr, 935 op/s; 29971/225333 objects degraded (13.301%); 4325/225333 objects misplaced (1.919%); 21 MiB/s, 338 objects/s recovering 2024-09-21T14:00:31.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:31 smithi187 ceph-mon[33032]: osdmap e315: 8 total, 8 up, 8 in 2024-09-21T14:00:31.733 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:00:31.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:31 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:00:31.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:31 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:00:31.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:31 smithi049 ceph-mon[40489]: pgmap v574: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovery_wait+undersized+degraded+remapped, 139 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 588 B/s rd, 0 B/s wr, 935 op/s; 29971/225333 objects degraded (13.301%); 4325/225333 objects misplaced (1.919%); 21 MiB/s, 338 objects/s recovering 2024-09-21T14:00:31.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:31 smithi049 ceph-mon[40489]: osdmap e315: 8 total, 8 up, 8 in 2024-09-21T14:00:31.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:31 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:00:31.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:31 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:00:31.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:31 smithi049 ceph-mon[30080]: pgmap v574: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovery_wait+undersized+degraded+remapped, 139 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 588 B/s rd, 0 B/s wr, 935 op/s; 29971/225333 objects degraded (13.301%); 4325/225333 objects misplaced (1.919%); 21 MiB/s, 338 objects/s recovering 2024-09-21T14:00:31.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:31 smithi049 ceph-mon[30080]: osdmap e315: 8 total, 8 up, 8 in 2024-09-21T14:00:31.758 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:31.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:31.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:00:31.859 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:31.890 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:31.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:31.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:31.992 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:32.027 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:32.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:32.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:32.129 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:00:32.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:32.169 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:32.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:00:32.255 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T14:00:32.270 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:32.338 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:32.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:32.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:32.440 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:32.488 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:32.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:32.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:32.590 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:32.628 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:32.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:32.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:32.730 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:00:32.765 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:32.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:32.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:00:32.867 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:00:32.906 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:32.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:32.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:00:33.008 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:00:33.040 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:33.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:33.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:00:33.142 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:33.177 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:33.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:33.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:33.279 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:33.304 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:33.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:33.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:33.406 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:00:33.429 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:33.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:33.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:00:33.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:33 smithi049 ceph-mon[40489]: pgmap v576: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 3 active+recovery_wait+undersized+degraded+remapped, 139 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 950 op/s; 29671/221715 objects degraded (13.382%); 4064/221715 objects misplaced (1.833%); 23 MiB/s, 368 objects/s recovering 2024-09-21T14:00:33.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:33 smithi049 ceph-mon[30080]: pgmap v576: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 3 active+recovery_wait+undersized+degraded+remapped, 139 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 950 op/s; 29671/221715 objects degraded (13.382%); 4064/221715 objects misplaced (1.833%); 23 MiB/s, 368 objects/s recovering 2024-09-21T14:00:33.530 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:33.561 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:33.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:33.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:33.663 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:00:33.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:33 smithi187 ceph-mon[33032]: pgmap v576: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 3 active+recovery_wait+undersized+degraded+remapped, 139 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 950 op/s; 29671/221715 objects degraded (13.382%); 4064/221715 objects misplaced (1.833%); 23 MiB/s, 368 objects/s recovering 2024-09-21T14:00:33.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:33.700 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:33.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:00:33.801 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:33.836 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:33.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:33.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:33.938 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:00:33.960 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:33.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:33.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:00:34.062 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:34.097 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:34.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:34.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:34.199 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:00:34.237 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:34.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:34.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:00:34.339 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:00:34.371 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:34.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:34.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:00:34.473 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:00:34.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:34.508 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:34.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:00:34.609 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:00:34.638 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:34.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:34.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:00:34.739 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:00:34.783 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:34.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:34.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:00:34.885 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:00:34.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:34.920 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:34.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:00:35.021 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:35.059 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:35.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:35.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:35.161 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:35.198 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:35.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:35.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:35.300 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:00:35.334 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:35.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:35.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:00:35.436 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:00:35.477 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:35.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:35.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:00:35.579 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:00:35.615 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:35.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:35.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:00:35.633 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:35 smithi049 ceph-mon[30080]: pgmap v577: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 3 active+recovery_wait+undersized+degraded+remapped, 139 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 924 op/s; 29671/216771 objects degraded (13.688%); 4064/216771 objects misplaced (1.875%); 18 MiB/s, 287 objects/s recovering 2024-09-21T14:00:35.633 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:35 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 29671/216771 objects degraded (13.688%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-09-21T14:00:35.633 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:35 smithi049 ceph-mon[30080]: osdmap e316: 8 total, 8 up, 8 in 2024-09-21T14:00:35.635 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:35 smithi049 ceph-mon[40489]: pgmap v577: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 3 active+recovery_wait+undersized+degraded+remapped, 139 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 924 op/s; 29671/216771 objects degraded (13.688%); 4064/216771 objects misplaced (1.875%); 18 MiB/s, 287 objects/s recovering 2024-09-21T14:00:35.635 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:35 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 29671/216771 objects degraded (13.688%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-09-21T14:00:35.635 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:35 smithi049 ceph-mon[40489]: osdmap e316: 8 total, 8 up, 8 in 2024-09-21T14:00:35.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:35 smithi187 ceph-mon[33032]: pgmap v577: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 3 active+recovery_wait+undersized+degraded+remapped, 139 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 924 op/s; 29671/216771 objects degraded (13.688%); 4064/216771 objects misplaced (1.875%); 18 MiB/s, 287 objects/s recovering 2024-09-21T14:00:35.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:35 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 29671/216771 objects degraded (13.688%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-09-21T14:00:35.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:35 smithi187 ceph-mon[33032]: osdmap e316: 8 total, 8 up, 8 in 2024-09-21T14:00:35.717 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:00:35.760 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:35.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:35.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:00:35.864 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:35.895 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:35.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:35.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:35.996 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:00:36.041 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:36.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:36.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:00:36.084 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:00:36.143 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:00:36.159 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:00:36.205 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:36.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:36.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:00:36.307 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:36.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:36.342 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:36.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:36.442 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:00:36.489 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:36.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:36.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:00:36.590 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:36.626 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:36.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:36.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:36.727 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:00:36.766 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:36.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:36.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:00:36.868 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:00:36.910 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:36.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:36.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:00:37.013 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:37.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:37.041 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:37.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:37.087 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T14:00:37.088 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-09-21T14:00:37.088 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-09-21T14:00:37.088 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-09-21T14:00:37.088 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.088 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:163570 7.8 7.330e1638 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e130)", 2024-09-21T14:00:37.088 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:03.335126+0000", 2024-09-21T14:00:37.088 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 513.75002032600003, 2024-09-21T14:00:37.088 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97989348499999995, 2024-09-21T14:00:37.088 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:00:37.088 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:00:37.088 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T14:00:37.089 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:00:37.089 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T14:00:37.089 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T14:00:37.089 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 163570 2024-09-21T14:00:37.089 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.089 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:00:37.089 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.089 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:00:37.089 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.335126+0000", 2024-09-21T14:00:37.089 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:37.089 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.089 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.089 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:00:37.089 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.335126+0000", 2024-09-21T14:00:37.090 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:37.090 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.090 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.090 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:00:37.090 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.335127+0000", 2024-09-21T14:00:37.090 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5209999999999999e-06 2024-09-21T14:00:37.090 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.090 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.090 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:00:37.090 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.335132+0000", 2024-09-21T14:00:37.090 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4730000000000002e-06 2024-09-21T14:00:37.090 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.090 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.090 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:00:37.091 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.335142+0000", 2024-09-21T14:00:37.091 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8770000000000005e-06 2024-09-21T14:00:37.091 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.091 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.091 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:00:37.091 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.335151+0000", 2024-09-21T14:00:37.091 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7650000000000005e-06 2024-09-21T14:00:37.091 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.091 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.091 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:37.091 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.336965+0000", 2024-09-21T14:00:37.091 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001813907 2024-09-21T14:00:37.091 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.091 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.092 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:00:37.092 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.336978+0000", 2024-09-21T14:00:37.092 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2685e-05 2024-09-21T14:00:37.092 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.092 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.092 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:37.092 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:04.315001+0000", 2024-09-21T14:00:37.092 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97802330000000004 2024-09-21T14:00:37.092 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.092 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.092 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:00:37.092 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:04.315019+0000", 2024-09-21T14:00:37.092 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7957000000000001e-05 2024-09-21T14:00:37.092 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:37.093 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:00:37.093 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:37.093 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.093 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.093 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:163585 7.8 7.9fcafe08 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-21T14:00:37.093 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:03.339560+0000", 2024-09-21T14:00:37.093 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 513.745586151, 2024-09-21T14:00:37.093 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97550176099999997, 2024-09-21T14:00:37.093 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:00:37.093 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:00:37.093 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T14:00:37.093 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:00:37.093 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T14:00:37.093 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T14:00:37.094 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 163585 2024-09-21T14:00:37.094 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.094 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:00:37.094 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.094 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:00:37.094 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.339560+0000", 2024-09-21T14:00:37.094 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:37.094 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.094 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.094 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:00:37.094 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.339560+0000", 2024-09-21T14:00:37.094 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:37.094 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.094 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.095 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:00:37.095 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.339562+0000", 2024-09-21T14:00:37.095 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6449999999999999e-06 2024-09-21T14:00:37.095 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.095 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.095 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:00:37.095 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.339568+0000", 2024-09-21T14:00:37.095 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2169999999999999e-06 2024-09-21T14:00:37.095 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.095 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.095 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:00:37.095 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.339580+0000", 2024-09-21T14:00:37.095 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1846e-05 2024-09-21T14:00:37.095 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.095 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.096 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:00:37.096 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.339589+0000", 2024-09-21T14:00:37.096 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.798e-06 2024-09-21T14:00:37.096 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.096 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.096 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:37.096 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.342967+0000", 2024-09-21T14:00:37.096 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0033776990000000001 2024-09-21T14:00:37.096 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.096 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.096 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:00:37.096 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.342980+0000", 2024-09-21T14:00:37.096 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2758000000000001e-05 2024-09-21T14:00:37.096 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.096 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.097 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:37.097 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:04.315052+0000", 2024-09-21T14:00:37.097 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97207221799999999 2024-09-21T14:00:37.097 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.097 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.097 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:00:37.097 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:04.315062+0000", 2024-09-21T14:00:37.097 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.5799999999999998e-06 2024-09-21T14:00:37.097 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:37.097 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:00:37.097 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:37.097 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.097 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.098 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:163595 7.8 7.9f5fc9f8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-21T14:00:37.098 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:03.360541+0000", 2024-09-21T14:00:37.098 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 513.72460498099997, 2024-09-21T14:00:37.098 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.95455601800000001, 2024-09-21T14:00:37.098 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:00:37.098 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:00:37.098 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T14:00:37.098 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:00:37.098 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T14:00:37.098 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T14:00:37.099 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 163595 2024-09-21T14:00:37.099 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.099 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:00:37.099 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.099 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:00:37.099 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.360541+0000", 2024-09-21T14:00:37.099 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:37.099 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.099 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.099 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:00:37.099 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.360541+0000", 2024-09-21T14:00:37.099 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:37.099 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.099 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.100 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:00:37.100 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.360544+0000", 2024-09-21T14:00:37.100 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.632e-06 2024-09-21T14:00:37.100 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.100 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.100 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:00:37.100 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.360554+0000", 2024-09-21T14:00:37.100 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0210000000000001e-05 2024-09-21T14:00:37.100 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.100 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.100 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:00:37.100 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.360579+0000", 2024-09-21T14:00:37.100 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4878000000000001e-05 2024-09-21T14:00:37.100 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.101 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.101 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:00:37.101 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.360589+0000", 2024-09-21T14:00:37.101 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7960000000000007e-06 2024-09-21T14:00:37.101 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.101 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.101 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:37.101 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.360662+0000", 2024-09-21T14:00:37.101 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3514000000000001e-05 2024-09-21T14:00:37.101 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.101 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.101 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:00:37.101 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.360673+0000", 2024-09-21T14:00:37.101 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1080000000000001e-05 2024-09-21T14:00:37.102 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.102 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.102 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:37.102 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:04.315088+0000", 2024-09-21T14:00:37.102 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.954414859 2024-09-21T14:00:37.102 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.102 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.103 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:00:37.103 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:04.315097+0000", 2024-09-21T14:00:37.103 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0489999999999993e-06 2024-09-21T14:00:37.103 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:37.103 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:00:37.103 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:37.103 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.104 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.104 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:163602 7.8 7.3bec8d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-21T14:00:37.104 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:03.369662+0000", 2024-09-21T14:00:37.104 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 513.71548422499995, 2024-09-21T14:00:37.104 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.945473868, 2024-09-21T14:00:37.104 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:00:37.104 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:00:37.104 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T14:00:37.104 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:00:37.104 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T14:00:37.104 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T14:00:37.104 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 163602 2024-09-21T14:00:37.105 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.105 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:00:37.105 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.105 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:00:37.105 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.369662+0000", 2024-09-21T14:00:37.105 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:37.105 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.105 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.105 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:00:37.105 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.369662+0000", 2024-09-21T14:00:37.105 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:37.106 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.106 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.106 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:00:37.106 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.369664+0000", 2024-09-21T14:00:37.106 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.136e-06 2024-09-21T14:00:37.106 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.106 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.106 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:00:37.106 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.369671+0000", 2024-09-21T14:00:37.106 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9110000000000004e-06 2024-09-21T14:00:37.106 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.106 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.106 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:00:37.107 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.369689+0000", 2024-09-21T14:00:37.107 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7757999999999998e-05 2024-09-21T14:00:37.107 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.107 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.107 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:00:37.107 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.369700+0000", 2024-09-21T14:00:37.107 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.146e-05 2024-09-21T14:00:37.107 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.107 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.107 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:37.107 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.369779+0000", 2024-09-21T14:00:37.107 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9260999999999999e-05 2024-09-21T14:00:37.107 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.107 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.108 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:00:37.108 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:03.369793+0000", 2024-09-21T14:00:37.108 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3200000000000001e-05 2024-09-21T14:00:37.108 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.108 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.108 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:37.108 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:04.315126+0000", 2024-09-21T14:00:37.108 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.94533302500000005 2024-09-21T14:00:37.108 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.108 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.109 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:00:37.109 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:04.315136+0000", 2024-09-21T14:00:37.109 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0117e-05 2024-09-21T14:00:37.109 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:37.109 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:00:37.109 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:37.109 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.109 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.109 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:201459 7.8 7:1473b567:::benchmark_data_smithi049_76798_object65039:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-21T14:00:37.109 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:41.960649+0000", 2024-09-21T14:00:37.109 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 475.124497056, 2024-09-21T14:00:37.109 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.025332715, 2024-09-21T14:00:37.109 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:00:37.110 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:00:37.110 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:00:37.110 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:00:37.110 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T14:00:37.110 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T14:00:37.110 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 201459 2024-09-21T14:00:37.110 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.110 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:00:37.110 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.110 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:00:37.110 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:41.960649+0000", 2024-09-21T14:00:37.110 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:37.110 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.110 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.111 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:00:37.111 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:41.960649+0000", 2024-09-21T14:00:37.111 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:37.111 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.111 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.111 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:00:37.111 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:41.960651+0000", 2024-09-21T14:00:37.111 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7990000000000001e-06 2024-09-21T14:00:37.111 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.111 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.111 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:00:37.111 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:41.960657+0000", 2024-09-21T14:00:37.111 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2260000000000002e-06 2024-09-21T14:00:37.112 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.112 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.112 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:00:37.112 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:41.960677+0000", 2024-09-21T14:00:37.112 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9783999999999999e-05 2024-09-21T14:00:37.112 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.112 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.112 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:00:37.112 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:41.960686+0000", 2024-09-21T14:00:37.112 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0710000000000001e-06 2024-09-21T14:00:37.112 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.112 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.112 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:37.112 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:41.964807+0000", 2024-09-21T14:00:37.113 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0041212560000000002 2024-09-21T14:00:37.113 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.113 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.113 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:00:37.113 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:41.964817+0000", 2024-09-21T14:00:37.113 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0030000000000001e-05 2024-09-21T14:00:37.113 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.113 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.113 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:37.113 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.976018+0000", 2024-09-21T14:00:37.113 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.011200412 2024-09-21T14:00:37.113 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.113 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.113 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:00:37.114 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.976136+0000", 2024-09-21T14:00:37.114 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000118106 2024-09-21T14:00:37.114 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.114 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.114 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-09-21T14:00:37.114 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.976247+0000", 2024-09-21T14:00:37.114 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000111418 2024-09-21T14:00:37.114 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.114 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.114 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:37.114 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.980913+0000", 2024-09-21T14:00:37.114 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0046656850000000001 2024-09-21T14:00:37.114 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.115 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.115 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:37.115 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.980959+0000", 2024-09-21T14:00:37.115 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6402000000000001e-05 2024-09-21T14:00:37.115 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.115 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.115 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:00:37.115 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.985848+0000", 2024-09-21T14:00:37.115 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0048883210000000002 2024-09-21T14:00:37.115 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.115 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.115 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:00:37.115 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.985906+0000", 2024-09-21T14:00:37.115 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8242000000000002e-05 2024-09-21T14:00:37.115 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.116 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.116 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:00:37.116 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.985982+0000", 2024-09-21T14:00:37.116 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5963000000000004e-05 2024-09-21T14:00:37.116 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:37.116 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:00:37.116 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:37.116 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.116 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.116 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:201494 7.8 7:1c4e9ccf:::benchmark_data_smithi049_76798_object65074:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-21T14:00:37.116 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:42.006815+0000", 2024-09-21T14:00:37.116 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 475.07833065, 2024-09-21T14:00:37.116 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97934932200000002, 2024-09-21T14:00:37.116 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:00:37.116 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:00:37.117 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:00:37.117 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:00:37.117 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T14:00:37.117 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T14:00:37.117 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 201494 2024-09-21T14:00:37.117 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.117 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:00:37.117 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.117 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:00:37.117 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006815+0000", 2024-09-21T14:00:37.117 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:37.117 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.117 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.117 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:00:37.117 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006815+0000", 2024-09-21T14:00:37.118 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:37.118 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.118 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.118 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:00:37.118 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006817+0000", 2024-09-21T14:00:37.118 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.618e-06 2024-09-21T14:00:37.118 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.118 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.118 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:00:37.118 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006823+0000", 2024-09-21T14:00:37.118 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0859999999999998e-06 2024-09-21T14:00:37.118 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.118 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.119 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:00:37.119 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006840+0000", 2024-09-21T14:00:37.119 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7149000000000001e-05 2024-09-21T14:00:37.119 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.119 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.119 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:00:37.119 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006848+0000", 2024-09-21T14:00:37.119 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9340000000000006e-06 2024-09-21T14:00:37.119 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.119 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.119 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:37.119 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006949+0000", 2024-09-21T14:00:37.119 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010043399999999999 2024-09-21T14:00:37.119 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.120 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.120 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:00:37.120 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006957+0000", 2024-09-21T14:00:37.120 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6580000000000008e-06 2024-09-21T14:00:37.120 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.120 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.120 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:37.120 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.979032+0000", 2024-09-21T14:00:37.120 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97207428699999998 2024-09-21T14:00:37.120 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.120 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.120 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:00:37.120 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.979198+0000", 2024-09-21T14:00:37.120 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016622800000000001 2024-09-21T14:00:37.120 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.121 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.121 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-09-21T14:00:37.121 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.979306+0000", 2024-09-21T14:00:37.121 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000108356 2024-09-21T14:00:37.121 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.121 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.121 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:37.121 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.980997+0000", 2024-09-21T14:00:37.121 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0016909640000000001 2024-09-21T14:00:37.121 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.121 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.121 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:37.121 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.981060+0000", 2024-09-21T14:00:37.121 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2516999999999998e-05 2024-09-21T14:00:37.122 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.122 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.122 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:00:37.122 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986075+0000", 2024-09-21T14:00:37.122 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.005014971 2024-09-21T14:00:37.122 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.122 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.122 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:00:37.122 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986121+0000", 2024-09-21T14:00:37.122 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6313999999999997e-05 2024-09-21T14:00:37.122 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.122 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.122 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:00:37.123 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986165+0000", 2024-09-21T14:00:37.123 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3806e-05 2024-09-21T14:00:37.123 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:37.123 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:00:37.123 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:37.123 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.123 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.123 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:201495 7.8 7:18dd0180:::benchmark_data_smithi049_76798_object65075:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-21T14:00:37.123 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:42.006872+0000", 2024-09-21T14:00:37.123 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 475.07827357100001, 2024-09-21T14:00:37.123 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97938486800000002, 2024-09-21T14:00:37.123 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:00:37.123 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:00:37.123 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:00:37.123 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:00:37.124 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T14:00:37.124 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T14:00:37.124 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 201495 2024-09-21T14:00:37.124 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.124 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:00:37.124 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.124 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:00:37.124 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006872+0000", 2024-09-21T14:00:37.124 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:37.124 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.124 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.124 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:00:37.124 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006872+0000", 2024-09-21T14:00:37.124 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:37.125 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.125 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.125 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:00:37.125 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006873+0000", 2024-09-21T14:00:37.125 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4899999999999995e-07 2024-09-21T14:00:37.125 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.125 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.125 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:00:37.125 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006878+0000", 2024-09-21T14:00:37.125 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0429999999999998e-06 2024-09-21T14:00:37.125 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.125 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.125 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:00:37.125 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006886+0000", 2024-09-21T14:00:37.125 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3399999999999998e-06 2024-09-21T14:00:37.126 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.126 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.126 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:00:37.126 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006894+0000", 2024-09-21T14:00:37.126 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1359999999999999e-06 2024-09-21T14:00:37.126 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.126 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.126 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:37.126 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006982+0000", 2024-09-21T14:00:37.126 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.8022000000000005e-05 2024-09-21T14:00:37.126 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.126 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.126 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:00:37.126 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.006988+0000", 2024-09-21T14:00:37.127 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3470000000000004e-06 2024-09-21T14:00:37.127 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.127 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.127 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:37.127 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.981113+0000", 2024-09-21T14:00:37.127 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97412548499999996 2024-09-21T14:00:37.127 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.127 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.127 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:00:37.127 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.981230+0000", 2024-09-21T14:00:37.127 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000116666 2024-09-21T14:00:37.127 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.127 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.127 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-09-21T14:00:37.128 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.981349+0000", 2024-09-21T14:00:37.128 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000118572 2024-09-21T14:00:37.128 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.128 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.128 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:37.128 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.985825+0000", 2024-09-21T14:00:37.128 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0044766739999999999 2024-09-21T14:00:37.128 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.128 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.128 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:37.128 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986063+0000", 2024-09-21T14:00:37.128 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00023781299999999999 2024-09-21T14:00:37.128 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.128 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.129 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:00:37.129 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986172+0000", 2024-09-21T14:00:37.129 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010923499999999999 2024-09-21T14:00:37.129 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.129 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.129 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:00:37.129 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986236+0000", 2024-09-21T14:00:37.129 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3606000000000001e-05 2024-09-21T14:00:37.129 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.129 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.129 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:00:37.129 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986257+0000", 2024-09-21T14:00:37.129 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1379999999999999e-05 2024-09-21T14:00:37.129 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:37.129 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:00:37.130 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:37.130 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.130 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.130 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:201496 7.8 7:10f836b0:::benchmark_data_smithi049_76798_object65076:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-21T14:00:37.130 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:42.012648+0000", 2024-09-21T14:00:37.130 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 475.07249760100001, 2024-09-21T14:00:37.130 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97380752599999998, 2024-09-21T14:00:37.130 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:00:37.130 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:00:37.130 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:00:37.130 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:00:37.130 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T14:00:37.130 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T14:00:37.130 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 201496 2024-09-21T14:00:37.131 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.131 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:00:37.131 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.131 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:00:37.131 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.012648+0000", 2024-09-21T14:00:37.131 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:37.131 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.132 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.132 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:00:37.132 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.012648+0000", 2024-09-21T14:00:37.132 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:37.132 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.132 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.132 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:00:37.132 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.012650+0000", 2024-09-21T14:00:37.132 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.513e-06 2024-09-21T14:00:37.132 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.132 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.132 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:00:37.132 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.012656+0000", 2024-09-21T14:00:37.132 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7300000000000002e-06 2024-09-21T14:00:37.133 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.133 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.133 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:00:37.133 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.012664+0000", 2024-09-21T14:00:37.133 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6090000000000001e-06 2024-09-21T14:00:37.133 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.133 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.133 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:00:37.133 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.012670+0000", 2024-09-21T14:00:37.133 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5829999999999997e-06 2024-09-21T14:00:37.133 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.133 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.133 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:37.133 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.012740+0000", 2024-09-21T14:00:37.133 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9834999999999998e-05 2024-09-21T14:00:37.134 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.134 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.134 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:00:37.134 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.012749+0000", 2024-09-21T14:00:37.134 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9889999999999998e-06 2024-09-21T14:00:37.134 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.134 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.134 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:37.134 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.983365+0000", 2024-09-21T14:00:37.134 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97061665600000002 2024-09-21T14:00:37.134 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.134 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.134 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:00:37.134 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.983473+0000", 2024-09-21T14:00:37.135 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000107507 2024-09-21T14:00:37.135 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.135 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.135 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-09-21T14:00:37.135 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.983595+0000", 2024-09-21T14:00:37.135 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000121614 2024-09-21T14:00:37.135 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.135 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.135 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:00:37.135 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986265+0000", 2024-09-21T14:00:37.135 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002670113 2024-09-21T14:00:37.135 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.135 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.135 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:37.136 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986301+0000", 2024-09-21T14:00:37.136 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6130999999999997e-05 2024-09-21T14:00:37.136 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.136 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.136 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:37.136 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986370+0000", 2024-09-21T14:00:37.136 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8994999999999999e-05 2024-09-21T14:00:37.136 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.136 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.136 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:00:37.136 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986417+0000", 2024-09-21T14:00:37.136 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7691000000000001e-05 2024-09-21T14:00:37.136 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.137 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.137 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:00:37.137 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986456+0000", 2024-09-21T14:00:37.137 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8559999999999997e-05 2024-09-21T14:00:37.137 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:37.137 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:00:37.137 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:37.137 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.137 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.137 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:201499 7.8 7:1329afa8:::benchmark_data_smithi049_76798_object65079:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-21T14:00:37.137 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:42.026955+0000", 2024-09-21T14:00:37.137 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 475.05819102599997, 2024-09-21T14:00:37.137 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97876118000000001, 2024-09-21T14:00:37.137 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:00:37.138 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:00:37.138 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:00:37.138 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:00:37.138 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T14:00:37.138 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T14:00:37.138 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 201499 2024-09-21T14:00:37.138 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.138 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:00:37.138 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.138 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:00:37.138 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.026955+0000", 2024-09-21T14:00:37.138 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:37.138 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.138 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.139 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:00:37.139 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.026955+0000", 2024-09-21T14:00:37.139 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:37.139 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.139 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.139 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:00:37.139 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.026957+0000", 2024-09-21T14:00:37.139 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7999999999999999e-06 2024-09-21T14:00:37.139 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.139 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.139 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:00:37.139 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.026962+0000", 2024-09-21T14:00:37.139 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9400000000000001e-06 2024-09-21T14:00:37.139 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.140 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.140 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:00:37.140 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.026977+0000", 2024-09-21T14:00:37.140 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5087e-05 2024-09-21T14:00:37.140 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.140 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.140 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:00:37.140 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.026983+0000", 2024-09-21T14:00:37.140 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0090000000000004e-06 2024-09-21T14:00:37.140 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.140 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.140 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:37.140 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027044+0000", 2024-09-21T14:00:37.140 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1543000000000004e-05 2024-09-21T14:00:37.140 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.141 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.141 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:00:37.141 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027051+0000", 2024-09-21T14:00:37.141 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4590000000000003e-06 2024-09-21T14:00:37.141 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.141 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.141 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:37.141 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986512+0000", 2024-09-21T14:00:37.141 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.95946141399999996 2024-09-21T14:00:37.141 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.141 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.141 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:00:37.141 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986631+0000", 2024-09-21T14:00:37.141 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011896300000000001 2024-09-21T14:00:37.142 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.142 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.142 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-09-21T14:00:37.142 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.986738+0000", 2024-09-21T14:00:37.142 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000106766 2024-09-21T14:00:37.142 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.142 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.142 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:37.142 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.994021+0000", 2024-09-21T14:00:37.142 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0072827329999999996 2024-09-21T14:00:37.142 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.142 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.142 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:37.142 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.994059+0000", 2024-09-21T14:00:37.143 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8053999999999999e-05 2024-09-21T14:00:37.143 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.143 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.143 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:00:37.143 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.005582+0000", 2024-09-21T14:00:37.143 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.011523342000000001 2024-09-21T14:00:37.143 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.143 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.143 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:00:37.143 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.005671+0000", 2024-09-21T14:00:37.143 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.8708999999999994e-05 2024-09-21T14:00:37.143 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.143 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.143 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:00:37.143 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.005716+0000", 2024-09-21T14:00:37.143 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5361000000000002e-05 2024-09-21T14:00:37.144 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:37.144 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:00:37.144 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:37.144 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.144 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.144 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:201500 7.8 7:19fe3d31:::benchmark_data_smithi049_76798_object65080:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-21T14:00:37.144 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:42.027000+0000", 2024-09-21T14:00:37.144 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 475.05814558600002, 2024-09-21T14:00:37.144 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97887608500000001, 2024-09-21T14:00:37.144 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:00:37.144 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:00:37.144 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:00:37.144 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:00:37.144 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T14:00:37.144 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T14:00:37.145 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 201500 2024-09-21T14:00:37.145 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.145 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:00:37.145 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.145 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:00:37.145 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027000+0000", 2024-09-21T14:00:37.145 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:37.145 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.145 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.145 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:00:37.145 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027000+0000", 2024-09-21T14:00:37.145 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:37.145 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.145 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.146 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:00:37.146 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027001+0000", 2024-09-21T14:00:37.146 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3200000000000001e-07 2024-09-21T14:00:37.146 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.146 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.146 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:00:37.146 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027003+0000", 2024-09-21T14:00:37.146 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8980000000000001e-06 2024-09-21T14:00:37.146 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.146 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.146 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:00:37.146 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027007+0000", 2024-09-21T14:00:37.146 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4290000000000003e-06 2024-09-21T14:00:37.146 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.147 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.147 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:00:37.147 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027010+0000", 2024-09-21T14:00:37.147 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3419999999999999e-06 2024-09-21T14:00:37.147 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.147 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.147 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:37.147 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027064+0000", 2024-09-21T14:00:37.147 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3474000000000003e-05 2024-09-21T14:00:37.147 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.147 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.147 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:00:37.147 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027067+0000", 2024-09-21T14:00:37.148 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.932e-06 2024-09-21T14:00:37.148 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.148 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.148 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:37.148 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.989128+0000", 2024-09-21T14:00:37.148 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.96206106899999999 2024-09-21T14:00:37.148 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.148 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.148 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:00:37.148 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.990066+0000", 2024-09-21T14:00:37.148 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00093830900000000002 2024-09-21T14:00:37.148 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.148 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.148 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-09-21T14:00:37.148 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.990172+0000", 2024-09-21T14:00:37.149 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000105271 2024-09-21T14:00:37.149 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.149 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.149 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:37.149 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.996356+0000", 2024-09-21T14:00:37.149 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0061840100000000002 2024-09-21T14:00:37.149 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.149 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.149 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:37.149 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.996388+0000", 2024-09-21T14:00:37.149 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2539000000000001e-05 2024-09-21T14:00:37.149 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.149 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.149 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:00:37.150 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.005774+0000", 2024-09-21T14:00:37.150 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0093859679999999997 2024-09-21T14:00:37.150 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.150 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.150 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:00:37.150 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.005838+0000", 2024-09-21T14:00:37.150 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3724000000000002e-05 2024-09-21T14:00:37.150 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.150 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.150 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:00:37.150 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.005877+0000", 2024-09-21T14:00:37.150 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8788000000000002e-05 2024-09-21T14:00:37.150 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:37.150 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:00:37.151 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:37.151 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.151 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.151 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:201501 7.8 7:15f77bbd:::benchmark_data_smithi049_76798_object65081:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-21T14:00:37.151 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:42.027294+0000", 2024-09-21T14:00:37.151 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 475.057851621, 2024-09-21T14:00:37.151 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97870270599999998, 2024-09-21T14:00:37.151 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:00:37.151 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:00:37.151 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:00:37.151 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:00:37.151 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T14:00:37.151 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T14:00:37.151 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 201501 2024-09-21T14:00:37.152 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.152 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:00:37.152 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.152 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:00:37.152 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027294+0000", 2024-09-21T14:00:37.152 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:37.152 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.152 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.152 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:00:37.152 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027294+0000", 2024-09-21T14:00:37.152 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:37.152 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.152 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.152 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:00:37.152 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027295+0000", 2024-09-21T14:00:37.153 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7899999999999997e-07 2024-09-21T14:00:37.153 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.153 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.153 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:00:37.153 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027299+0000", 2024-09-21T14:00:37.153 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6449999999999998e-06 2024-09-21T14:00:37.153 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.153 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.153 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:00:37.153 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027310+0000", 2024-09-21T14:00:37.153 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0699999999999999e-05 2024-09-21T14:00:37.153 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.153 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.153 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:00:37.153 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027315+0000", 2024-09-21T14:00:37.154 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9729999999999997e-06 2024-09-21T14:00:37.154 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.154 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.154 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:37.154 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027363+0000", 2024-09-21T14:00:37.154 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8096000000000002e-05 2024-09-21T14:00:37.154 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.154 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.154 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:00:37.154 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027367+0000", 2024-09-21T14:00:37.154 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2930000000000002e-06 2024-09-21T14:00:37.154 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.154 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.154 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:37.155 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.992098+0000", 2024-09-21T14:00:37.155 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.96473078800000001 2024-09-21T14:00:37.155 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.155 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.155 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:00:37.155 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.992160+0000", 2024-09-21T14:00:37.155 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2133999999999998e-05 2024-09-21T14:00:37.155 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.155 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.155 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-09-21T14:00:37.155 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.992220+0000", 2024-09-21T14:00:37.155 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9911999999999998e-05 2024-09-21T14:00:37.155 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.155 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.155 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:37.156 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.003058+0000", 2024-09-21T14:00:37.156 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.010838441000000001 2024-09-21T14:00:37.156 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.156 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.156 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:37.156 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.003121+0000", 2024-09-21T14:00:37.156 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3118000000000001e-05 2024-09-21T14:00:37.156 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.156 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.156 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:00:37.156 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.005887+0000", 2024-09-21T14:00:37.156 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0027659500000000001 2024-09-21T14:00:37.156 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.156 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.156 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:00:37.157 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.005958+0000", 2024-09-21T14:00:37.157 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0842999999999999e-05 2024-09-21T14:00:37.157 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.157 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.157 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:00:37.157 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.005997+0000", 2024-09-21T14:00:37.157 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9033999999999999e-05 2024-09-21T14:00:37.157 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:37.157 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:00:37.157 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:37.157 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.157 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.157 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:201503 7.8 7:1bb4326b:::benchmark_data_smithi049_76798_object65083:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-21T14:00:37.157 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:42.027327+0000", 2024-09-21T14:00:37.157 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 475.05781954499997, 2024-09-21T14:00:37.158 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97877131900000003, 2024-09-21T14:00:37.158 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:00:37.158 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:00:37.158 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:00:37.158 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:37.159 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:00:37.159 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T14:00:37.159 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T14:00:37.159 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 201503 2024-09-21T14:00:37.159 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.160 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:00:37.160 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.160 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:00:37.160 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027327+0000", 2024-09-21T14:00:37.160 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:37.160 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.160 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.160 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:00:37.160 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027327+0000", 2024-09-21T14:00:37.160 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:37.160 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.160 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.160 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:00:37.160 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027327+0000", 2024-09-21T14:00:37.161 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4799999999999999e-07 2024-09-21T14:00:37.161 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.161 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.161 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:00:37.161 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027329+0000", 2024-09-21T14:00:37.161 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7400000000000001e-06 2024-09-21T14:00:37.161 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.161 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.161 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:00:37.161 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027333+0000", 2024-09-21T14:00:37.161 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0790000000000002e-06 2024-09-21T14:00:37.161 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.161 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.161 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:00:37.162 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027336+0000", 2024-09-21T14:00:37.162 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1190000000000001e-06 2024-09-21T14:00:37.162 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.162 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.162 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:37.162 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027379+0000", 2024-09-21T14:00:37.162 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3000000000000002e-05 2024-09-21T14:00:37.162 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.162 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.162 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:00:37.162 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.027382+0000", 2024-09-21T14:00:37.162 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9399999999999998e-06 2024-09-21T14:00:37.162 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.162 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.163 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:37.163 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.994085+0000", 2024-09-21T14:00:37.163 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.96670302799999996 2024-09-21T14:00:37.163 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.163 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.163 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:00:37.163 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.994181+0000", 2024-09-21T14:00:37.163 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6340000000000003e-05 2024-09-21T14:00:37.163 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.163 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.163 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-09-21T14:00:37.163 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.994285+0000", 2024-09-21T14:00:37.163 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000103947 2024-09-21T14:00:37.163 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.164 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.164 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:37.164 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.003091+0000", 2024-09-21T14:00:37.164 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0088057599999999993 2024-09-21T14:00:37.164 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.164 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.164 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:37.164 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.003150+0000", 2024-09-21T14:00:37.164 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9670000000000003e-05 2024-09-21T14:00:37.164 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.164 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.164 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:00:37.164 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.006006+0000", 2024-09-21T14:00:37.164 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002855734 2024-09-21T14:00:37.165 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.165 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.165 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:00:37.165 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.006056+0000", 2024-09-21T14:00:37.165 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9619e-05 2024-09-21T14:00:37.165 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.165 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.165 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:00:37.165 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.006098+0000", 2024-09-21T14:00:37.165 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1995e-05 2024-09-21T14:00:37.165 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:37.165 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:00:37.165 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:37.165 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.166 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.166 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:201514 7.8 7:1c8e529c:::benchmark_data_smithi049_76798_object65094:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-21T14:00:37.166 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:42.044200+0000", 2024-09-21T14:00:37.166 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 475.04094643799999, 2024-09-21T14:00:37.166 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.96199924699999995, 2024-09-21T14:00:37.166 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:00:37.166 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:00:37.166 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:00:37.166 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:00:37.166 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T14:00:37.166 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T14:00:37.166 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 201514 2024-09-21T14:00:37.166 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.166 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:00:37.167 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.167 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:00:37.167 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.044200+0000", 2024-09-21T14:00:37.167 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:37.167 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.167 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.167 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:00:37.167 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.044200+0000", 2024-09-21T14:00:37.167 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:37.167 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.167 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.167 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:00:37.167 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.044201+0000", 2024-09-21T14:00:37.167 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7179999999999999e-06 2024-09-21T14:00:37.168 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.168 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.168 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:00:37.168 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.044208+0000", 2024-09-21T14:00:37.168 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1800000000000001e-06 2024-09-21T14:00:37.168 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.168 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.168 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:00:37.168 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.044226+0000", 2024-09-21T14:00:37.168 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8216999999999999e-05 2024-09-21T14:00:37.168 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.168 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.168 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:00:37.168 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.044235+0000", 2024-09-21T14:00:37.168 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1020000000000003e-06 2024-09-21T14:00:37.169 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.169 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.169 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:37.169 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.044482+0000", 2024-09-21T14:00:37.169 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00024732699999999999 2024-09-21T14:00:37.169 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.169 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.169 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:00:37.169 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.044499+0000", 2024-09-21T14:00:37.169 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7129000000000002e-05 2024-09-21T14:00:37.169 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.169 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.169 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:37.169 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.996416+0000", 2024-09-21T14:00:37.169 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.95191657200000002 2024-09-21T14:00:37.170 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.170 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.170 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:00:37.170 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.996528+0000", 2024-09-21T14:00:37.170 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000112118 2024-09-21T14:00:37.170 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.170 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.170 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-09-21T14:00:37.170 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.996652+0000", 2024-09-21T14:00:37.170 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012430500000000001 2024-09-21T14:00:37.170 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.170 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.170 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:37.170 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.003180+0000", 2024-09-21T14:00:37.170 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.006528056 2024-09-21T14:00:37.171 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.171 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.171 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:37.171 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.005760+0000", 2024-09-21T14:00:37.171 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0025797670000000002 2024-09-21T14:00:37.171 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.171 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.171 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:00:37.171 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.006108+0000", 2024-09-21T14:00:37.171 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00034834500000000001 2024-09-21T14:00:37.171 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.171 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.171 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:00:37.171 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.006155+0000", 2024-09-21T14:00:37.172 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6513e-05 2024-09-21T14:00:37.172 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.172 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.172 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:00:37.172 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.006199+0000", 2024-09-21T14:00:37.172 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3897999999999999e-05 2024-09-21T14:00:37.172 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:37.172 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:00:37.172 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:37.172 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.172 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.172 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:201539 7.8 7:1afa34ae:::benchmark_data_smithi049_76798_object65119:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-21T14:00:37.172 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:42.081266+0000", 2024-09-21T14:00:37.172 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 475.00388033000002, 2024-09-21T14:00:37.172 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.92917915600000001, 2024-09-21T14:00:37.173 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:00:37.173 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:00:37.173 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:00:37.173 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:00:37.173 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T14:00:37.173 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T14:00:37.173 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 201539 2024-09-21T14:00:37.173 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.173 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:00:37.173 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.173 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:00:37.173 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.081266+0000", 2024-09-21T14:00:37.173 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:37.173 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.174 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.174 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:00:37.174 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.081266+0000", 2024-09-21T14:00:37.174 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:37.174 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.174 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.174 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:00:37.174 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.081267+0000", 2024-09-21T14:00:37.174 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6559999999999999e-06 2024-09-21T14:00:37.174 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.174 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.174 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:00:37.174 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.081273+0000", 2024-09-21T14:00:37.174 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0839999999999997e-06 2024-09-21T14:00:37.175 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.175 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.175 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:00:37.175 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.081291+0000", 2024-09-21T14:00:37.175 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7827999999999999e-05 2024-09-21T14:00:37.175 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.175 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.175 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:00:37.175 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.081300+0000", 2024-09-21T14:00:37.175 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.7190000000000007e-06 2024-09-21T14:00:37.175 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.175 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.175 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:37.175 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.081360+0000", 2024-09-21T14:00:37.176 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9914000000000003e-05 2024-09-21T14:00:37.176 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.176 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.176 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:00:37.176 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.081369+0000", 2024-09-21T14:00:37.176 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1630000000000002e-06 2024-09-21T14:00:37.176 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.176 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.176 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:37.176 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.998700+0000", 2024-09-21T14:00:37.176 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.91733049300000002 2024-09-21T14:00:37.176 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.176 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.176 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:00:37.176 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.998798+0000", 2024-09-21T14:00:37.177 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8494999999999998e-05 2024-09-21T14:00:37.177 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.177 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.177 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-09-21T14:00:37.177 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.998903+0000", 2024-09-21T14:00:37.177 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000104782 2024-09-21T14:00:37.177 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.177 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.177 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:00:37.177 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.006208+0000", 2024-09-21T14:00:37.177 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0073052270000000001 2024-09-21T14:00:37.177 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.177 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.177 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:37.177 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.006259+0000", 2024-09-21T14:00:37.177 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1205999999999997e-05 2024-09-21T14:00:37.178 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.178 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.178 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:37.178 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.010350+0000", 2024-09-21T14:00:37.178 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0040908560000000004 2024-09-21T14:00:37.178 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.178 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.178 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:00:37.178 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.010400+0000", 2024-09-21T14:00:37.178 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9901000000000002e-05 2024-09-21T14:00:37.178 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.178 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.178 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:00:37.178 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.010445+0000", 2024-09-21T14:00:37.178 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4832e-05 2024-09-21T14:00:37.179 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:37.179 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:00:37.179 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:37.179 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.179 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.179 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:201541 7.8 7:135037ad:::benchmark_data_smithi049_76798_object65121:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-21T14:00:37.179 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:52:42.083222+0000", 2024-09-21T14:00:37.179 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 475.00192407899999, 2024-09-21T14:00:37.179 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.92733551700000005, 2024-09-21T14:00:37.179 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:00:37.179 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:00:37.179 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:00:37.179 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:00:37.179 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T14:00:37.180 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T14:00:37.180 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 201541 2024-09-21T14:00:37.180 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.180 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:00:37.180 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.180 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:00:37.180 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.083222+0000", 2024-09-21T14:00:37.180 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:37.180 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.180 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.180 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:00:37.180 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.083222+0000", 2024-09-21T14:00:37.180 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:37.180 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.181 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.181 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:00:37.181 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.083223+0000", 2024-09-21T14:00:37.181 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0020000000000001e-06 2024-09-21T14:00:37.181 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.181 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.181 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:00:37.181 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.083227+0000", 2024-09-21T14:00:37.181 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.636e-06 2024-09-21T14:00:37.181 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.181 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.181 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:00:37.181 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.083236+0000", 2024-09-21T14:00:37.181 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0180000000000007e-06 2024-09-21T14:00:37.181 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.182 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.182 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:00:37.182 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.083242+0000", 2024-09-21T14:00:37.182 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.891e-06 2024-09-21T14:00:37.182 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.182 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.182 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:37.182 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.083385+0000", 2024-09-21T14:00:37.182 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014374099999999999 2024-09-21T14:00:37.182 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.182 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.182 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:00:37.182 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:42.083394+0000", 2024-09-21T14:00:37.182 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0839999999999998e-06 2024-09-21T14:00:37.182 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.183 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.183 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:37.183 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.000839+0000", 2024-09-21T14:00:37.183 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.917444861 2024-09-21T14:00:37.183 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.183 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.183 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:00:37.183 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.000934+0000", 2024-09-21T14:00:37.183 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4943000000000002e-05 2024-09-21T14:00:37.183 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.183 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.183 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,7", 2024-09-21T14:00:37.183 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.001036+0000", 2024-09-21T14:00:37.183 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000101817 2024-09-21T14:00:37.183 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.184 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.184 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:00:37.184 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.006215+0000", 2024-09-21T14:00:37.184 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0051790020000000003 2024-09-21T14:00:37.184 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.184 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.184 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:37.184 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.006324+0000", 2024-09-21T14:00:37.184 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010858599999999999 2024-09-21T14:00:37.184 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.184 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.184 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:37.184 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.010481+0000", 2024-09-21T14:00:37.185 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0041575300000000004 2024-09-21T14:00:37.185 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.185 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.185 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:00:37.185 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.010528+0000", 2024-09-21T14:00:37.185 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6418000000000002e-05 2024-09-21T14:00:37.185 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.185 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.185 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:00:37.185 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:52:43.010558+0000", 2024-09-21T14:00:37.185 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9988000000000002e-05 2024-09-21T14:00:37.185 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:37.185 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:00:37.186 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:37.186 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.186 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.187 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:229964 7.e 7:7be4d3ef:::benchmark_data_smithi049_76798_object93544:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-21T14:00:37.187 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:53:16.693433+0000", 2024-09-21T14:00:37.187 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 440.391713514, 2024-09-21T14:00:37.187 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.99823727600000001, 2024-09-21T14:00:37.187 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:00:37.187 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:00:37.187 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:00:37.187 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:00:37.188 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T14:00:37.188 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T14:00:37.188 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 229964 2024-09-21T14:00:37.188 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.188 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:00:37.188 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.188 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:00:37.188 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.693433+0000", 2024-09-21T14:00:37.188 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:37.188 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.188 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.188 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:00:37.188 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.693433+0000", 2024-09-21T14:00:37.188 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:37.188 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.189 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.189 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:00:37.189 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.693434+0000", 2024-09-21T14:00:37.189 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3799999999999999e-06 2024-09-21T14:00:37.189 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.189 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.189 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:00:37.189 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.693438+0000", 2024-09-21T14:00:37.189 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.515e-06 2024-09-21T14:00:37.189 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.189 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.189 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:00:37.189 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.693455+0000", 2024-09-21T14:00:37.189 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6614000000000001e-05 2024-09-21T14:00:37.190 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.190 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.190 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:00:37.190 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.693463+0000", 2024-09-21T14:00:37.190 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9850000000000006e-06 2024-09-21T14:00:37.190 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.190 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.190 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:37.190 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.693542+0000", 2024-09-21T14:00:37.190 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8942000000000004e-05 2024-09-21T14:00:37.190 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.190 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.190 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:00:37.190 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.693550+0000", 2024-09-21T14:00:37.190 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7379999999999994e-06 2024-09-21T14:00:37.191 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.191 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.191 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:37.191 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.686321+0000", 2024-09-21T14:00:37.191 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.99277167399999999 2024-09-21T14:00:37.191 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.191 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.191 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:00:37.191 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.686413+0000", 2024-09-21T14:00:37.191 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1822000000000003e-05 2024-09-21T14:00:37.191 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.191 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.191 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,5", 2024-09-21T14:00:37.191 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.686547+0000", 2024-09-21T14:00:37.191 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000133516 2024-09-21T14:00:37.192 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.192 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.192 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:37.192 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.691512+0000", 2024-09-21T14:00:37.192 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0049651410000000002 2024-09-21T14:00:37.192 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.192 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.192 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:37.192 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.691558+0000", 2024-09-21T14:00:37.192 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6047999999999997e-05 2024-09-21T14:00:37.192 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.192 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.192 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:00:37.192 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.691610+0000", 2024-09-21T14:00:37.193 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1644000000000002e-05 2024-09-21T14:00:37.193 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.193 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.193 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:00:37.193 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.691642+0000", 2024-09-21T14:00:37.193 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2542000000000001e-05 2024-09-21T14:00:37.193 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.193 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.193 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:00:37.193 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.691670+0000", 2024-09-21T14:00:37.193 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.7715000000000001e-05 2024-09-21T14:00:37.193 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:37.193 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:00:37.193 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:37.193 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.194 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.194 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:230040 7.e 7:74df0b23:::benchmark_data_smithi049_76798_object93620:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-21T14:00:37.194 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:53:16.758488+0000", 2024-09-21T14:00:37.194 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 440.32665845700001, 2024-09-21T14:00:37.194 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.93513229499999995, 2024-09-21T14:00:37.194 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:00:37.194 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:00:37.194 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:00:37.194 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:00:37.194 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T14:00:37.194 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T14:00:37.194 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 230040 2024-09-21T14:00:37.194 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.194 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:00:37.195 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.195 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:00:37.195 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.758488+0000", 2024-09-21T14:00:37.195 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:37.195 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.195 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.195 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:00:37.195 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.758488+0000", 2024-09-21T14:00:37.195 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:37.195 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.195 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.195 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:00:37.195 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.758502+0000", 2024-09-21T14:00:37.195 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4207e-05 2024-09-21T14:00:37.196 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.196 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.196 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:00:37.196 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.758507+0000", 2024-09-21T14:00:37.196 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9429999999999999e-06 2024-09-21T14:00:37.196 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.196 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.196 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:00:37.196 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.758518+0000", 2024-09-21T14:00:37.196 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1447999999999999e-05 2024-09-21T14:00:37.196 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.196 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.196 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:00:37.196 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.758525+0000", 2024-09-21T14:00:37.196 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2389999999999996e-06 2024-09-21T14:00:37.197 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.197 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.197 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:37.197 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.758589+0000", 2024-09-21T14:00:37.197 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3262000000000005e-05 2024-09-21T14:00:37.197 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.197 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.197 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:00:37.197 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.758597+0000", 2024-09-21T14:00:37.197 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4100000000000008e-06 2024-09-21T14:00:37.197 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.197 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.197 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:37.197 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.689523+0000", 2024-09-21T14:00:37.198 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.93092584099999998 2024-09-21T14:00:37.198 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.198 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.198 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:00:37.198 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.689620+0000", 2024-09-21T14:00:37.198 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7081000000000001e-05 2024-09-21T14:00:37.198 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.198 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.198 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,5", 2024-09-21T14:00:37.198 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.689705+0000", 2024-09-21T14:00:37.198 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5074000000000006e-05 2024-09-21T14:00:37.198 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.198 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.198 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:37.199 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.691602+0000", 2024-09-21T14:00:37.199 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001896951 2024-09-21T14:00:37.199 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.199 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.199 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:37.199 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.693564+0000", 2024-09-21T14:00:37.199 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0019616630000000002 2024-09-21T14:00:37.199 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.199 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.199 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:00:37.199 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.693572+0000", 2024-09-21T14:00:37.199 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9160000000000001e-06 2024-09-21T14:00:37.199 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.199 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.199 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:00:37.199 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.693597+0000", 2024-09-21T14:00:37.200 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5812999999999998e-05 2024-09-21T14:00:37.200 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.200 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.200 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:00:37.200 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.693620+0000", 2024-09-21T14:00:37.200 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2447e-05 2024-09-21T14:00:37.200 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:37.200 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:00:37.200 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:37.200 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.200 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.200 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:230048 7.e 7:7e0d246c:::benchmark_data_smithi049_76798_object93628:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-21T14:00:37.200 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:53:16.763844+0000", 2024-09-21T14:00:37.200 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 440.32130157799998, 2024-09-21T14:00:37.201 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.93491194899999996, 2024-09-21T14:00:37.201 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:00:37.201 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:00:37.201 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:00:37.201 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:00:37.201 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T14:00:37.201 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T14:00:37.201 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 230048 2024-09-21T14:00:37.201 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.201 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:00:37.201 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.201 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:00:37.201 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.763844+0000", 2024-09-21T14:00:37.201 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:37.201 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.202 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.202 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:00:37.202 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.763844+0000", 2024-09-21T14:00:37.202 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:37.202 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.202 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.202 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:00:37.202 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.763846+0000", 2024-09-21T14:00:37.202 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6330000000000001e-06 2024-09-21T14:00:37.202 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.202 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.202 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:00:37.202 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.763852+0000", 2024-09-21T14:00:37.202 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3839999999999996e-06 2024-09-21T14:00:37.202 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.203 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.203 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:00:37.203 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.763915+0000", 2024-09-21T14:00:37.203 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3306999999999995e-05 2024-09-21T14:00:37.203 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.203 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.203 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:00:37.203 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.764047+0000", 2024-09-21T14:00:37.203 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013207599999999999 2024-09-21T14:00:37.203 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.203 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.203 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:37.203 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.764104+0000", 2024-09-21T14:00:37.203 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6758000000000001e-05 2024-09-21T14:00:37.203 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.204 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.204 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:00:37.204 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.764113+0000", 2024-09-21T14:00:37.204 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3700000000000001e-06 2024-09-21T14:00:37.204 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.204 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.204 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:37.204 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.691702+0000", 2024-09-21T14:00:37.204 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.92758885899999999 2024-09-21T14:00:37.204 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.204 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.204 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:00:37.204 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.691822+0000", 2024-09-21T14:00:37.204 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011997699999999999 2024-09-21T14:00:37.204 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.205 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.205 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,5", 2024-09-21T14:00:37.205 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.691939+0000", 2024-09-21T14:00:37.205 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011752 2024-09-21T14:00:37.205 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.205 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.205 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:00:37.205 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.698556+0000", 2024-09-21T14:00:37.205 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0066170739999999997 2024-09-21T14:00:37.205 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.205 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.205 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:37.205 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.698612+0000", 2024-09-21T14:00:37.205 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5362000000000003e-05 2024-09-21T14:00:37.206 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.206 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.206 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:37.206 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.698659+0000", 2024-09-21T14:00:37.206 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6931999999999997e-05 2024-09-21T14:00:37.206 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.206 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.206 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:00:37.206 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.698714+0000", 2024-09-21T14:00:37.206 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5402000000000003e-05 2024-09-21T14:00:37.206 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.206 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.206 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:00:37.207 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.698756+0000", 2024-09-21T14:00:37.207 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2295000000000001e-05 2024-09-21T14:00:37.207 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:37.207 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:00:37.207 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:37.207 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.207 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.207 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:230051 7.e 7:71fa5c2a:::benchmark_data_smithi049_76798_object93631:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-21T14:00:37.207 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:53:16.764458+0000", 2024-09-21T14:00:37.207 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 440.32068813299998, 2024-09-21T14:00:37.207 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.94179994899999997, 2024-09-21T14:00:37.207 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:00:37.207 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:00:37.207 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:00:37.208 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:00:37.208 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T14:00:37.208 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T14:00:37.208 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 230051 2024-09-21T14:00:37.208 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.208 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:00:37.208 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.208 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:00:37.208 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.764458+0000", 2024-09-21T14:00:37.208 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:37.208 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.208 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.208 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:00:37.208 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.764458+0000", 2024-09-21T14:00:37.208 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:37.209 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.209 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.209 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:00:37.209 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.764459+0000", 2024-09-21T14:00:37.209 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9400000000000004e-07 2024-09-21T14:00:37.209 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.209 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.209 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:00:37.209 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.764463+0000", 2024-09-21T14:00:37.209 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1960000000000001e-06 2024-09-21T14:00:37.209 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.209 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.209 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:00:37.209 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.764477+0000", 2024-09-21T14:00:37.210 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4426e-05 2024-09-21T14:00:37.210 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.210 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.210 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:00:37.210 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.764485+0000", 2024-09-21T14:00:37.210 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.148e-06 2024-09-21T14:00:37.210 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.210 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.210 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:37.210 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.764545+0000", 2024-09-21T14:00:37.210 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0278999999999997e-05 2024-09-21T14:00:37.210 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.210 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.210 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:00:37.210 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.764553+0000", 2024-09-21T14:00:37.211 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9170000000000006e-06 2024-09-21T14:00:37.211 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.211 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.211 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:37.211 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.693660+0000", 2024-09-21T14:00:37.211 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.92910747900000001 2024-09-21T14:00:37.211 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.211 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.211 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:00:37.211 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.693785+0000", 2024-09-21T14:00:37.211 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012436000000000001 2024-09-21T14:00:37.211 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.211 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.211 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,5", 2024-09-21T14:00:37.211 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.693907+0000", 2024-09-21T14:00:37.212 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012208600000000001 2024-09-21T14:00:37.212 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.212 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.212 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:37.212 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.698816+0000", 2024-09-21T14:00:37.212 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0049091279999999996 2024-09-21T14:00:37.212 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.212 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.212 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:00:37.212 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.698834+0000", 2024-09-21T14:00:37.212 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8236999999999998e-05 2024-09-21T14:00:37.212 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.212 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.212 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:37.213 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.706148+0000", 2024-09-21T14:00:37.213 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0073139709999999998 2024-09-21T14:00:37.213 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.213 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.213 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:00:37.213 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.706202+0000", 2024-09-21T14:00:37.213 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3779000000000001e-05 2024-09-21T14:00:37.213 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.214 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.214 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:00:37.214 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.706258+0000", 2024-09-21T14:00:37.214 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6048999999999998e-05 2024-09-21T14:00:37.214 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:37.214 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:00:37.214 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:37.214 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.214 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.214 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.14817.0:230060 7.e 7:7289055b:::benchmark_data_smithi049_76798_object93640:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-21T14:00:37.215 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:53:16.772333+0000", 2024-09-21T14:00:37.215 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 440.31281301299998, 2024-09-21T14:00:37.215 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.93409186700000002, 2024-09-21T14:00:37.215 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:00:37.215 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:00:37.215 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:00:37.215 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:00:37.215 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.14817", 2024-09-21T14:00:37.215 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T14:00:37.215 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 230060 2024-09-21T14:00:37.215 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.215 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:00:37.215 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.215 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:00:37.216 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.772333+0000", 2024-09-21T14:00:37.216 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:37.216 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.216 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.216 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:00:37.216 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.772333+0000", 2024-09-21T14:00:37.216 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:37.216 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.216 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.216 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:00:37.216 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.772334+0000", 2024-09-21T14:00:37.216 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1540000000000001e-06 2024-09-21T14:00:37.216 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.216 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.217 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:00:37.217 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.772339+0000", 2024-09-21T14:00:37.217 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8099999999999997e-06 2024-09-21T14:00:37.217 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.217 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.217 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:00:37.217 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.772351+0000", 2024-09-21T14:00:37.217 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2435999999999999e-05 2024-09-21T14:00:37.217 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.217 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.217 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:00:37.217 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.772359+0000", 2024-09-21T14:00:37.217 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7230000000000004e-06 2024-09-21T14:00:37.217 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.217 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.218 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:37.218 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.772435+0000", 2024-09-21T14:00:37.218 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5465999999999999e-05 2024-09-21T14:00:37.218 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.218 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.218 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:00:37.218 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:16.772443+0000", 2024-09-21T14:00:37.218 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9160000000000001e-06 2024-09-21T14:00:37.218 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.218 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.218 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:37.218 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.695975+0000", 2024-09-21T14:00:37.218 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.92353224599999995 2024-09-21T14:00:37.218 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.219 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.219 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:00:37.219 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.696104+0000", 2024-09-21T14:00:37.219 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012895499999999999 2024-09-21T14:00:37.219 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.219 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.219 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,5", 2024-09-21T14:00:37.219 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.696217+0000", 2024-09-21T14:00:37.219 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011362399999999999 2024-09-21T14:00:37.219 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.219 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.219 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:00:37.219 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.701076+0000", 2024-09-21T14:00:37.219 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0048582729999999998 2024-09-21T14:00:37.220 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.220 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.220 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:37.220 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.706310+0000", 2024-09-21T14:00:37.220 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0052347419999999997 2024-09-21T14:00:37.220 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.220 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.220 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:37.220 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.706347+0000", 2024-09-21T14:00:37.220 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6492999999999998e-05 2024-09-21T14:00:37.220 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.220 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.220 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:00:37.220 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.706392+0000", 2024-09-21T14:00:37.220 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5148999999999998e-05 2024-09-21T14:00:37.221 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:37.221 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:37.221 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:00:37.221 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:53:17.706425+0000", 2024-09-21T14:00:37.221 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2879999999999997e-05 2024-09-21T14:00:37.221 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:37.221 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:00:37.221 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:37.221 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:37.221 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:00:37.221 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T14:00:37.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:37.222 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:37.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:37.301 INFO:teuthology.orchestra.run.smithi049.stderr:set pool 1 pg_num to 11 2024-09-21T14:00:37.322 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:37.338 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:37 smithi049 ceph-mon[30080]: pgmap v579: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovery_wait+undersized+degraded+remapped, 139 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 990 op/s; 27694/212010 objects degraded (13.063%); 4064/212010 objects misplaced (1.917%); 25 MiB/s, 394 objects/s recovering 2024-09-21T14:00:37.338 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:37 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1941535596' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-21T14:00:37.338 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:37 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-21T14:00:37.338 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:37 smithi049 ceph-mon[40489]: pgmap v579: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovery_wait+undersized+degraded+remapped, 139 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 990 op/s; 27694/212010 objects degraded (13.063%); 4064/212010 objects misplaced (1.917%); 25 MiB/s, 394 objects/s recovering 2024-09-21T14:00:37.338 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:37 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1941535596' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-21T14:00:37.338 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:37 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-21T14:00:37.355 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:37.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:37.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:37.457 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:37.489 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:37.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:37.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:37.517 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T14:00:37.591 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:37.642 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:37.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:37.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:37.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:37 smithi187 ceph-mon[33032]: pgmap v579: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovery_wait+undersized+degraded+remapped, 139 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 990 op/s; 27694/212010 objects degraded (13.063%); 4064/212010 objects misplaced (1.917%); 25 MiB/s, 394 objects/s recovering 2024-09-21T14:00:37.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:37 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1941535596' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-21T14:00:37.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:37 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-21T14:00:37.744 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:37.778 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:37.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:37.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:37.879 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:00:37.917 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:37.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:37.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:00:38.018 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:38.058 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:38.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:38.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:38.160 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:00:38.202 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:38.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:38.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:00:38.304 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:38.317 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-09-21T14:00:38.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:38.360 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:38.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:38.461 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:00:38.500 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:38.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:38.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:00:38.602 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:00:38.631 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:38.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:38.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:00:38.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:38 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-21T14:00:38.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:38 smithi187 ceph-mon[33032]: osdmap e317: 8 total, 8 up, 8 in 2024-09-21T14:00:38.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:38 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1631131105' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:00:38.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:38 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-21T14:00:38.733 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:38.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:38 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-21T14:00:38.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:38 smithi049 ceph-mon[40489]: osdmap e317: 8 total, 8 up, 8 in 2024-09-21T14:00:38.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:38 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1631131105' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:00:38.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:38 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-21T14:00:38.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:38 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-21T14:00:38.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:38 smithi049 ceph-mon[30080]: osdmap e317: 8 total, 8 up, 8 in 2024-09-21T14:00:38.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:38 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1631131105' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:00:38.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:38 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-21T14:00:38.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:38.768 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:38.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:38.869 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:00:38.904 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:38.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:38.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:00:39.006 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:00:39.041 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:39.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:39.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:00:39.143 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:00:39.175 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:39.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:39.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:00:39.276 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:39.315 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:39.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:39.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:39.418 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:39.445 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:39.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:39.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:39.548 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:00:39.591 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:39.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:39.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:00:39.638 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:39 smithi187 ceph-mon[33032]: pgmap v581: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 3 active+recovery_wait+undersized+degraded+remapped, 139 active+clean; 4.2 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 718 op/s; 27435/208836 objects degraded (13.137%); 3835/208836 objects misplaced (1.836%); 13 MiB/s, 205 objects/s recovering 2024-09-21T14:00:39.638 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:39 smithi187 ceph-mon[33032]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:00:39.638 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:39 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1631131105' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T14:00:39.638 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:39 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-21T14:00:39.638 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:39 smithi187 ceph-mon[33032]: osdmap e318: 8 total, 8 up, 8 in 2024-09-21T14:00:39.693 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:00:39.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:39.733 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:39.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:00:39.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:39 smithi049 ceph-mon[40489]: pgmap v581: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 3 active+recovery_wait+undersized+degraded+remapped, 139 active+clean; 4.2 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 718 op/s; 27435/208836 objects degraded (13.137%); 3835/208836 objects misplaced (1.836%); 13 MiB/s, 205 objects/s recovering 2024-09-21T14:00:39.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:39 smithi049 ceph-mon[40489]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:00:39.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:39 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1631131105' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T14:00:39.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:39 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-21T14:00:39.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:39 smithi049 ceph-mon[40489]: osdmap e318: 8 total, 8 up, 8 in 2024-09-21T14:00:39.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:39 smithi049 ceph-mon[30080]: pgmap v581: 148 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 3 active+recovery_wait+undersized+degraded+remapped, 139 active+clean; 4.2 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 718 op/s; 27435/208836 objects degraded (13.137%); 3835/208836 objects misplaced (1.836%); 13 MiB/s, 205 objects/s recovering 2024-09-21T14:00:39.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:39 smithi049 ceph-mon[30080]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:00:39.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:39 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1631131105' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T14:00:39.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:39 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-21T14:00:39.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:39 smithi049 ceph-mon[30080]: osdmap e318: 8 total, 8 up, 8 in 2024-09-21T14:00:39.835 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:39.872 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:39.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:39.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:39.944 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:00:39 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:00:39.974 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:40.008 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:40.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:40.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:40.109 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:40.150 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:40.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:40.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:40.251 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:00:40.273 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:40.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:40.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:00:40.375 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:40.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:40.412 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:40.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:40.513 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:00:40.552 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:40.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:40.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:00:40.654 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:00:40.687 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:40.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:40.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:00:40.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:40 smithi187 ceph-mon[33032]: osdmap e319: 8 total, 8 up, 8 in 2024-09-21T14:00:40.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:40 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:00:40.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:40 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-21T14:00:40.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:40 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 27435/208836 objects degraded (13.137%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-09-21T14:00:40.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:40 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-21T14:00:40.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:40 smithi187 ceph-mon[33032]: osdmap e320: 8 total, 8 up, 8 in 2024-09-21T14:00:40.754 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:00:40 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:00:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:00:40.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:40 smithi049 ceph-mon[30080]: osdmap e319: 8 total, 8 up, 8 in 2024-09-21T14:00:40.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:40 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:00:40.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:40 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-21T14:00:40.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:40 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 27435/208836 objects degraded (13.137%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-09-21T14:00:40.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:40 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-21T14:00:40.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:40 smithi049 ceph-mon[30080]: osdmap e320: 8 total, 8 up, 8 in 2024-09-21T14:00:40.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:40 smithi049 ceph-mon[40489]: osdmap e319: 8 total, 8 up, 8 in 2024-09-21T14:00:40.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:40 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:00:40.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:40 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-21T14:00:40.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:40 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 27435/208836 objects degraded (13.137%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-09-21T14:00:40.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:40 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-21T14:00:40.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:40 smithi049 ceph-mon[40489]: osdmap e320: 8 total, 8 up, 8 in 2024-09-21T14:00:40.789 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:40.824 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:40.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:40.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:40.926 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:40.961 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:40.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:40.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:41.062 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:00:41.093 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:41.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:41.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:00:41.195 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:00:41.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:41.222 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:41.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:00:41.323 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:00:41.349 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:41.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:41.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:00:41.450 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:41.473 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:41.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:41.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:41.575 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:00:41.597 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:41.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:41.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:00:41.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:41 smithi187 ceph-mon[33032]: pgmap v584: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+recovering+undersized+degraded+remapped, 10 unknown, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovery_wait+undersized+degraded+remapped, 139 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 945 op/s; 25446/198165 objects degraded (12.841%); 3835/198165 objects misplaced (1.935%); 20 MiB/s, 327 objects/s recovering 2024-09-21T14:00:41.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:41 smithi187 ceph-mon[33032]: 3.3 scrub starts 2024-09-21T14:00:41.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:41 smithi187 ceph-mon[33032]: 3.3 scrub ok 2024-09-21T14:00:41.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:41 smithi187 ceph-mon[33032]: osdmap e321: 8 total, 8 up, 8 in 2024-09-21T14:00:41.699 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:41.721 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:41.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:41.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:41.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:41 smithi049 ceph-mon[30080]: pgmap v584: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+recovering+undersized+degraded+remapped, 10 unknown, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovery_wait+undersized+degraded+remapped, 139 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 945 op/s; 25446/198165 objects degraded (12.841%); 3835/198165 objects misplaced (1.935%); 20 MiB/s, 327 objects/s recovering 2024-09-21T14:00:41.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:41 smithi049 ceph-mon[30080]: 3.3 scrub starts 2024-09-21T14:00:41.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:41 smithi049 ceph-mon[30080]: 3.3 scrub ok 2024-09-21T14:00:41.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:41 smithi049 ceph-mon[30080]: osdmap e321: 8 total, 8 up, 8 in 2024-09-21T14:00:41.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:41 smithi049 ceph-mon[40489]: pgmap v584: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+recovering+undersized+degraded+remapped, 10 unknown, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_wait+backfill_toofull, 3 active+recovery_wait+undersized+degraded+remapped, 139 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 945 op/s; 25446/198165 objects degraded (12.841%); 3835/198165 objects misplaced (1.935%); 20 MiB/s, 327 objects/s recovering 2024-09-21T14:00:41.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:41 smithi049 ceph-mon[40489]: 3.3 scrub starts 2024-09-21T14:00:41.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:41 smithi049 ceph-mon[40489]: 3.3 scrub ok 2024-09-21T14:00:41.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:41 smithi049 ceph-mon[40489]: osdmap e321: 8 total, 8 up, 8 in 2024-09-21T14:00:41.823 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:00:41.845 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:41.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:41.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:00:41.947 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:41.971 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:41.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:41.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:42.073 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:42.097 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:42.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:42.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:42.199 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:00:42.228 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:42.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:42.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:00:42.330 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:42.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:42.367 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:42.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:42.467 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:42.504 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:42.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:42.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:42.605 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:42.639 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:42.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:42.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:42.740 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:42.767 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:42.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:42.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:42.869 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:42.897 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:42.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:42.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:42.999 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:43.081 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:43.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:43.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:43.183 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:00:43.214 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:43.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:43.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:00:43.316 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:00:43.344 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:43.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:43.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:00:43.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:43 smithi187 ceph-mon[33032]: pgmap v587: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 7 unknown, 1 active+remapped+backfill_wait+backfill_toofull, 3 peering, 1 active+undersized+degraded+remapped+backfilling, 3 active+recovery_wait+undersized+degraded+remapped, 139 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s; 25153/194781 objects degraded (12.913%); 3596/194781 objects misplaced (1.846%); 25 MiB/s, 400 objects/s recovering 2024-09-21T14:00:43.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:43 smithi187 ceph-mon[33032]: osdmap e322: 8 total, 8 up, 8 in 2024-09-21T14:00:43.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:43 smithi187 ceph-mon[33032]: 1.6 scrub starts 2024-09-21T14:00:43.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:43 smithi187 ceph-mon[33032]: 1.6 scrub ok 2024-09-21T14:00:43.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:43 smithi187 ceph-mon[33032]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-21T14:00:43.445 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:43.480 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:43.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:43.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:43.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:43 smithi049 ceph-mon[40489]: pgmap v587: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 7 unknown, 1 active+remapped+backfill_wait+backfill_toofull, 3 peering, 1 active+undersized+degraded+remapped+backfilling, 3 active+recovery_wait+undersized+degraded+remapped, 139 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s; 25153/194781 objects degraded (12.913%); 3596/194781 objects misplaced (1.846%); 25 MiB/s, 400 objects/s recovering 2024-09-21T14:00:43.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:43 smithi049 ceph-mon[40489]: osdmap e322: 8 total, 8 up, 8 in 2024-09-21T14:00:43.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:43 smithi049 ceph-mon[40489]: 1.6 scrub starts 2024-09-21T14:00:43.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:43 smithi049 ceph-mon[40489]: 1.6 scrub ok 2024-09-21T14:00:43.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:43 smithi049 ceph-mon[40489]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-21T14:00:43.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:43 smithi049 ceph-mon[30080]: pgmap v587: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 7 unknown, 1 active+remapped+backfill_wait+backfill_toofull, 3 peering, 1 active+undersized+degraded+remapped+backfilling, 3 active+recovery_wait+undersized+degraded+remapped, 139 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s; 25153/194781 objects degraded (12.913%); 3596/194781 objects misplaced (1.846%); 25 MiB/s, 400 objects/s recovering 2024-09-21T14:00:43.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:43 smithi049 ceph-mon[30080]: osdmap e322: 8 total, 8 up, 8 in 2024-09-21T14:00:43.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:43 smithi049 ceph-mon[30080]: 1.6 scrub starts 2024-09-21T14:00:43.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:43 smithi049 ceph-mon[30080]: 1.6 scrub ok 2024-09-21T14:00:43.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:43 smithi049 ceph-mon[30080]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-21T14:00:43.582 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:43.619 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:43.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:43.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:43.721 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:00:43.759 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:43.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:43.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:00:43.861 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:43.899 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:43.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:43.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:44.001 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:00:44.035 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:44.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:44.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:00:44.137 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:44.181 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:44.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:44.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:44.283 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:44.321 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:44.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:44.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:44.422 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:00:44.456 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:44.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:44.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:00:44.558 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:44.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:44.600 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:44.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:44.679 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T14:00:44.685 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:00:44.701 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:00:44.736 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:44.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:44.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:00:44.838 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:44.884 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:44.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:44.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:44.986 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:45.020 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:45.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:45.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:45.122 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:45.158 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:45.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:45.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:45.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:44 smithi187 ceph-mon[33032]: 1.7 scrub starts 2024-09-21T14:00:45.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:44 smithi187 ceph-mon[33032]: 1.7 scrub ok 2024-09-21T14:00:45.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:44 smithi187 ceph-mon[33032]: 1.5 scrub starts 2024-09-21T14:00:45.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:44 smithi187 ceph-mon[33032]: 1.5 scrub ok 2024-09-21T14:00:45.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:44 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:00:45.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:44 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:00:45.222 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:44 smithi049 ceph-mon[40489]: 1.7 scrub starts 2024-09-21T14:00:45.222 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:44 smithi049 ceph-mon[40489]: 1.7 scrub ok 2024-09-21T14:00:45.222 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:44 smithi049 ceph-mon[40489]: 1.5 scrub starts 2024-09-21T14:00:45.222 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:44 smithi049 ceph-mon[40489]: 1.5 scrub ok 2024-09-21T14:00:45.222 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:44 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:00:45.222 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:44 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:00:45.222 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:44 smithi049 ceph-mon[30080]: 1.7 scrub starts 2024-09-21T14:00:45.222 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:44 smithi049 ceph-mon[30080]: 1.7 scrub ok 2024-09-21T14:00:45.222 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:44 smithi049 ceph-mon[30080]: 1.5 scrub starts 2024-09-21T14:00:45.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:44 smithi049 ceph-mon[30080]: 1.5 scrub ok 2024-09-21T14:00:45.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:44 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:00:45.223 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:44 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:00:45.260 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:45.300 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:45.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:45.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:45.372 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:00:45.379 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:00:45.379 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:00:45.401 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:45.414 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:00:45.414 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:00:45.438 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:45.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:45.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:45.443 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:00:45.443 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:00:45.475 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:00:45.475 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:00:45.502 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:00:45.502 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:00:45.540 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:45.541 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:00:45.542 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:00:45.578 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:00:45.578 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:00:45.579 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:45.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:45.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:45.614 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:00:45.614 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:00:45.659 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T14:00:45.681 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:45.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:45.749 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:45.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:45.850 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:45.885 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:45.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:45.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:45.987 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:46.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:45 smithi049 ceph-mon[40489]: pgmap v589: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 4 unknown, 1 active+remapped+backfill_wait+backfill_toofull, 3 peering, 1 active+undersized+degraded+remapped+backfilling, 3 active+recovery_wait+undersized+degraded+remapped, 142 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 588 op/s; 25153/194100 objects degraded (12.959%); 3596/194100 objects misplaced (1.853%); 19 MiB/s, 303 objects/s recovering 2024-09-21T14:00:46.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:45 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 25153/194100 objects degraded (12.959%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-09-21T14:00:46.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:45 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2798912959' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:00:46.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:45 smithi049 ceph-mon[30080]: pgmap v589: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 4 unknown, 1 active+remapped+backfill_wait+backfill_toofull, 3 peering, 1 active+undersized+degraded+remapped+backfilling, 3 active+recovery_wait+undersized+degraded+remapped, 142 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 588 op/s; 25153/194100 objects degraded (12.959%); 3596/194100 objects misplaced (1.853%); 19 MiB/s, 303 objects/s recovering 2024-09-21T14:00:46.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:45 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 25153/194100 objects degraded (12.959%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-09-21T14:00:46.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:45 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2798912959' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:00:46.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:46.020 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:46.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:46.120 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:46.155 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:46.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:46.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:46.192 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T14:00:46.193 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:00:46.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:45 smithi187 ceph-mon[33032]: pgmap v589: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling+forced_backfill, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 4 unknown, 1 active+remapped+backfill_wait+backfill_toofull, 3 peering, 1 active+undersized+degraded+remapped+backfilling, 3 active+recovery_wait+undersized+degraded+remapped, 142 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 588 op/s; 25153/194100 objects degraded (12.959%); 3596/194100 objects misplaced (1.853%); 19 MiB/s, 303 objects/s recovering 2024-09-21T14:00:46.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:45 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 25153/194100 objects degraded (12.959%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-09-21T14:00:46.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:45 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2798912959' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:00:46.257 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:00:46.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:46.296 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:46.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:00:46.397 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:00:46.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:46.432 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:46.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:00:46.533 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:00:46.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:46.573 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:46.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:00:46.674 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:00:46.709 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:46.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:46.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:00:46.811 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:46.855 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:46.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:46.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:46.956 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:46.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:46.994 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:46.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:47.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:46 smithi049 ceph-mon[30080]: 3.9 scrub starts 2024-09-21T14:00:47.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:46 smithi049 ceph-mon[30080]: 3.9 scrub ok 2024-09-21T14:00:47.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:46 smithi049 ceph-mon[30080]: 5.6 scrub starts 2024-09-21T14:00:47.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:46 smithi049 ceph-mon[30080]: 5.6 scrub ok 2024-09-21T14:00:47.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:46 smithi049 ceph-mon[40489]: 3.9 scrub starts 2024-09-21T14:00:47.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:46 smithi049 ceph-mon[40489]: 3.9 scrub ok 2024-09-21T14:00:47.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:46 smithi049 ceph-mon[40489]: 5.6 scrub starts 2024-09-21T14:00:47.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:46 smithi049 ceph-mon[40489]: 5.6 scrub ok 2024-09-21T14:00:47.095 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:47.122 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:47.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:47.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:47.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:46 smithi187 ceph-mon[33032]: 3.9 scrub starts 2024-09-21T14:00:47.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:46 smithi187 ceph-mon[33032]: 3.9 scrub ok 2024-09-21T14:00:47.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:46 smithi187 ceph-mon[33032]: 5.6 scrub starts 2024-09-21T14:00:47.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:46 smithi187 ceph-mon[33032]: 5.6 scrub ok 2024-09-21T14:00:47.224 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:00:47.252 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:47.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:47.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:00:47.354 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:00:47.381 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:47.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:47.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:00:47.439 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T14:00:47.462 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:00:47.483 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:47.541 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:47.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:47.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:47.643 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:47.676 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:47.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:47.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:47.777 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:47.816 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:47.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:47.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:47.918 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:47.950 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:47.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:47.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:47.968 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:47 smithi049 ceph-mon[40489]: pgmap v590: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfill_wait+backfill_toofull, 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 150 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 667 op/s; 24050/186057 objects degraded (12.926%); 3596/186057 objects misplaced (1.933%); 9.0 MiB/s, 144 objects/s recovering 2024-09-21T14:00:47.968 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:47 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:00:47.968 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:47 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:00:47.969 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:47 smithi049 ceph-mon[30080]: pgmap v590: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfill_wait+backfill_toofull, 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 150 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 667 op/s; 24050/186057 objects degraded (12.926%); 3596/186057 objects misplaced (1.933%); 9.0 MiB/s, 144 objects/s recovering 2024-09-21T14:00:47.969 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:47 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:00:47.969 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:47 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:00:48.051 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:00:48.097 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:48.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:48.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:00:48.187 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:00:48.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:47 smithi187 ceph-mon[33032]: pgmap v590: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfill_wait+backfill_toofull, 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 150 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 667 op/s; 24050/186057 objects degraded (12.926%); 3596/186057 objects misplaced (1.933%); 9.0 MiB/s, 144 objects/s recovering 2024-09-21T14:00:48.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:47 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:00:48.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:47 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:00:48.199 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:00:48.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:48.243 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:48.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:00:48.344 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:48.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:48.380 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:48.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:48.481 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:48.516 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:48.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:48.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:48.618 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:00:48.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:48.649 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:48.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:00:48.714 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T14:00:48.750 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:48.801 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T14:00:48.801 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:00:48.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:48.803 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:48.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:48.904 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:48.923 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-09-21T14:00:48.943 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:48.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:48.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:49.045 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:49.077 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:49.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:49.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:49.149 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 4, 5, 6, 2, 3, 7] 2024-09-21T14:00:49.149 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T14:00:49.150 INFO:tasks.thrashosds.thrasher:Killing osd 5, live_osds are [0, 1, 4, 5, 6, 2, 3, 7] 2024-09-21T14:00:49.150 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-21T14:00:49.150 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@osd.5 2024-09-21T14:00:49.169 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:48 smithi187 ceph-mon[33032]: 4.5 scrub starts 2024-09-21T14:00:49.169 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:48 smithi187 ceph-mon[33032]: 4.5 scrub ok 2024-09-21T14:00:49.169 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:48 smithi187 ceph-mon[33032]: osdmap e323: 8 total, 8 up, 8 in 2024-09-21T14:00:49.169 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:48 smithi187 ceph-mon[33032]: pgmap v592: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+degraded, 3 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 150 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 12 KiB/s rd, 86 KiB/s wr, 580 op/s; 23785/183045 objects degraded (12.994%); 3390/183045 objects misplaced (1.852%); 7.9 MiB/s, 126 objects/s recovering 2024-09-21T14:00:49.169 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:48 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3572792813' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:00:49.169 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:48 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:00:49.179 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:49.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:49.218 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:49.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:49.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:48 smithi049 ceph-mon[40489]: 4.5 scrub starts 2024-09-21T14:00:49.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:48 smithi049 ceph-mon[40489]: 4.5 scrub ok 2024-09-21T14:00:49.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:48 smithi049 ceph-mon[40489]: osdmap e323: 8 total, 8 up, 8 in 2024-09-21T14:00:49.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:48 smithi049 ceph-mon[40489]: pgmap v592: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+degraded, 3 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 150 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 12 KiB/s rd, 86 KiB/s wr, 580 op/s; 23785/183045 objects degraded (12.994%); 3390/183045 objects misplaced (1.852%); 7.9 MiB/s, 126 objects/s recovering 2024-09-21T14:00:49.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:48 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3572792813' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:00:49.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:48 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:00:49.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:48 smithi049 ceph-mon[30080]: 4.5 scrub starts 2024-09-21T14:00:49.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:48 smithi049 ceph-mon[30080]: 4.5 scrub ok 2024-09-21T14:00:49.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:48 smithi049 ceph-mon[30080]: osdmap e323: 8 total, 8 up, 8 in 2024-09-21T14:00:49.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:48 smithi049 ceph-mon[30080]: pgmap v592: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+degraded, 3 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 150 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 12 KiB/s rd, 86 KiB/s wr, 580 op/s; 23785/183045 objects degraded (12.994%); 3390/183045 objects misplaced (1.852%); 7.9 MiB/s, 126 objects/s recovering 2024-09-21T14:00:49.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:48 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3572792813' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:00:49.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:48 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:00:49.319 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:49.352 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:49.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:49.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:49.444 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:00:49 smithi187 systemd[1]: Stopping Ceph osd.5 for 40a3d2e8-781f-11ef-baf6-efdc52797490... 2024-09-21T14:00:49.454 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:00:49.477 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:49.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:49.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:00:49.579 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:00:49.627 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:49.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:49.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:00:49.729 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:49.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:49.764 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:49.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:49.865 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:49.898 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:49.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:49.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:49.906 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:00:49 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:00:49.999 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:00:50.045 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:50.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:50.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:00:50.146 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:50.167 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:50.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:50.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:50.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:49 smithi187 ceph-mon[33032]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-21T14:00:50.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:49 smithi187 ceph-mon[33032]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T14:00:50.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:49 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T14:00:50.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:49 smithi187 ceph-mon[33032]: osdmap e324: 8 total, 8 up, 8 in 2024-09-21T14:00:50.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:49 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:00:50.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:49 smithi049 ceph-mon[40489]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-21T14:00:50.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:49 smithi049 ceph-mon[40489]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T14:00:50.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:49 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T14:00:50.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:49 smithi049 ceph-mon[40489]: osdmap e324: 8 total, 8 up, 8 in 2024-09-21T14:00:50.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:49 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:00:50.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:49 smithi049 ceph-mon[30080]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-21T14:00:50.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:49 smithi049 ceph-mon[30080]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T14:00:50.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:49 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T14:00:50.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:49 smithi049 ceph-mon[30080]: osdmap e324: 8 total, 8 up, 8 in 2024-09-21T14:00:50.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:49 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:00:50.269 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:50.317 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:50.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:50.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:50.418 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:50.457 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:50.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:50.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:50.559 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:50.584 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:50.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:50.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:50.686 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:00:50.718 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:50.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:50.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:00:50.754 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:00:50 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:00:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:00:50.820 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:00:50.855 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:50.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:50.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:00:50.957 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:50.981 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:50.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:50.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:50.991 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:00:51.023 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T14:00:51.082 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:00:51.131 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:51.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:51.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:00:51.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:50 smithi187 ceph-mon[33032]: pgmap v594: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+degraded, 3 active+recovering+undersized+degraded+remapped, 150 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 11 KiB/s rd, 75 KiB/s wr, 860 op/s; 23031/174684 objects degraded (13.184%); 3390/174684 objects misplaced (1.941%); 11 MiB/s, 172 objects/s recovering 2024-09-21T14:00:51.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:50 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 23785/183045 objects degraded (12.994%), 7 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-21T14:00:51.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:50 smithi187 ceph-mon[33032]: osd.5 marked itself down and dead 2024-09-21T14:00:51.195 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:00:50 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5[43330]: 2024-09-21T14:00:50.763+0000 7f7fe53b1640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T14:00:51.195 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:00:50 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5[43330]: 2024-09-21T14:00:50.763+0000 7f7fe53b1640 -1 osd.5 324 *** Got signal Terminated *** 2024-09-21T14:00:51.195 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:00:50 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5[43330]: 2024-09-21T14:00:50.763+0000 7f7fe53b1640 -1 osd.5 324 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T14:00:51.232 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:51.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:50 smithi049 ceph-mon[40489]: pgmap v594: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+degraded, 3 active+recovering+undersized+degraded+remapped, 150 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 11 KiB/s rd, 75 KiB/s wr, 860 op/s; 23031/174684 objects degraded (13.184%); 3390/174684 objects misplaced (1.941%); 11 MiB/s, 172 objects/s recovering 2024-09-21T14:00:51.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:50 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 23785/183045 objects degraded (12.994%), 7 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-21T14:00:51.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:50 smithi049 ceph-mon[40489]: osd.5 marked itself down and dead 2024-09-21T14:00:51.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:50 smithi049 ceph-mon[30080]: pgmap v594: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+degraded, 3 active+recovering+undersized+degraded+remapped, 150 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 11 KiB/s rd, 75 KiB/s wr, 860 op/s; 23031/174684 objects degraded (13.184%); 3390/174684 objects misplaced (1.941%); 11 MiB/s, 172 objects/s recovering 2024-09-21T14:00:51.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:50 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 23785/183045 objects degraded (12.994%), 7 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-21T14:00:51.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:50 smithi049 ceph-mon[30080]: osd.5 marked itself down and dead 2024-09-21T14:00:51.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:51.256 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:51.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:51.357 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:51.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:51.382 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:51.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:51.483 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:51.509 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:51.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:51.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:51.608 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:00:51.608 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:00:51.611 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:00:51.630 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:00:51.630 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:00:51.634 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:51.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:51.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:00:51.649 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:00:51.649 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:00:51.670 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:00:51.670 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:00:51.691 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:00:51.692 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:00:51.692 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-21T14:00:51.717 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:00:51.718 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:00:51.718 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:00:51.736 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:51.741 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:00:51.741 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:00:51.766 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:51.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:51.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:51.867 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:00:51.893 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:51.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:51.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:00:51.910 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:00:51 smithi187 podman[120146]: 2024-09-21 14:00:51.604764688 +0000 UTC m=+1.868679665 container died 77ec40e12497099fc7bf455699c25fd729c32c96f096ad62fb9e68d04f2543e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-guits-main-2024-09-21-0842-4417849, ceph=True, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T14:00:51.994 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:52.017 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:52.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:52.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:52.119 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:00:52.145 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:52.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:52.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:00:52.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:51 smithi187 ceph-mon[33032]: 4.1 scrub starts 2024-09-21T14:00:52.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:51 smithi187 ceph-mon[33032]: 4.1 scrub ok 2024-09-21T14:00:52.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:51 smithi187 ceph-mon[33032]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T14:00:52.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:51 smithi187 ceph-mon[33032]: osdmap e325: 8 total, 7 up, 8 in 2024-09-21T14:00:52.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:51 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1662149514' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:00:52.195 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:00:52 smithi187 podman[120146]: 2024-09-21 14:00:52.145847212 +0000 UTC m=+2.409762182 container cleanup 77ec40e12497099fc7bf455699c25fd729c32c96f096ad62fb9e68d04f2543e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T14:00:52.195 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:00:52 smithi187 bash[120146]: ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5 2024-09-21T14:00:52.247 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:00:52.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:51 smithi049 ceph-mon[40489]: 4.1 scrub starts 2024-09-21T14:00:52.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:51 smithi049 ceph-mon[40489]: 4.1 scrub ok 2024-09-21T14:00:52.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:51 smithi049 ceph-mon[40489]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T14:00:52.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:51 smithi049 ceph-mon[40489]: osdmap e325: 8 total, 7 up, 8 in 2024-09-21T14:00:52.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:51 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1662149514' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:00:52.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:51 smithi049 ceph-mon[30080]: 4.1 scrub starts 2024-09-21T14:00:52.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:51 smithi049 ceph-mon[30080]: 4.1 scrub ok 2024-09-21T14:00:52.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:51 smithi049 ceph-mon[30080]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T14:00:52.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:51 smithi049 ceph-mon[30080]: osdmap e325: 8 total, 7 up, 8 in 2024-09-21T14:00:52.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:51 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1662149514' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:00:52.272 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:52.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:52.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:00:52.374 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:52.399 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:52.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:52.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:52.501 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:52.518 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T14:00:52.525 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:52.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:52.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:52.535 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T14:00:52.535 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T14:00:52.627 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:52.655 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:52.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:52.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:52.694 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:00:52 smithi187 podman[120269]: 2024-09-21 14:00:52.389761501 +0000 UTC m=+0.778746516 container remove 77ec40e12497099fc7bf455699c25fd729c32c96f096ad62fb9e68d04f2543e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-21-0842-4417849, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux ) 2024-09-21T14:00:52.757 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:52.785 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:52.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:52.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:52.887 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:52.909 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:52.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:52.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:53.011 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:00:53.040 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:53.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:53.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:00:53.064 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:52 smithi049 ceph-mon[40489]: 3.1a scrub starts 2024-09-21T14:00:53.064 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:52 smithi049 ceph-mon[40489]: 3.1a scrub ok 2024-09-21T14:00:53.064 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:52 smithi049 ceph-mon[40489]: 9.d continuing backfill to osd.7 from (296'7120,324'11721] MIN to 324'11721 2024-09-21T14:00:53.064 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:52 smithi049 ceph-mon[40489]: osdmap e326: 8 total, 7 up, 8 in 2024-09-21T14:00:53.064 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:52 smithi049 ceph-mon[40489]: pgmap v597: 158 pgs: 1 active+remapped+backfilling, 1 stale+active+recovering+undersized+degraded+remapped, 1 stale+active+recovery_wait+undersized+degraded+remapped, 1 remapped, 21 stale+active+clean, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 11 peering, 1 active+recovering+undersized+degraded+remapped, 119 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s; 18626/169515 objects degraded (10.988%); 3103/169515 objects misplaced (1.831%); 9.9 MiB/s, 159 objects/s recovering 2024-09-21T14:00:53.065 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:52 smithi049 ceph-mon[30080]: 3.1a scrub starts 2024-09-21T14:00:53.065 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:52 smithi049 ceph-mon[30080]: 3.1a scrub ok 2024-09-21T14:00:53.065 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:52 smithi049 ceph-mon[30080]: 9.d continuing backfill to osd.7 from (296'7120,324'11721] MIN to 324'11721 2024-09-21T14:00:53.065 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:52 smithi049 ceph-mon[30080]: osdmap e326: 8 total, 7 up, 8 in 2024-09-21T14:00:53.065 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:52 smithi049 ceph-mon[30080]: pgmap v597: 158 pgs: 1 active+remapped+backfilling, 1 stale+active+recovering+undersized+degraded+remapped, 1 stale+active+recovery_wait+undersized+degraded+remapped, 1 remapped, 21 stale+active+clean, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 11 peering, 1 active+recovering+undersized+degraded+remapped, 119 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s; 18626/169515 objects degraded (10.988%); 3103/169515 objects misplaced (1.831%); 9.9 MiB/s, 159 objects/s recovering 2024-09-21T14:00:53.093 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T14:00:53.093 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-09-21T14:00:53.093 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-09-21T14:00:53.093 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-09-21T14:00:53.093 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.093 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232022 8.f 8.d7b7a40f (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T14:00:53.093 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.700289+0000", 2024-09-21T14:00:53.093 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 221.38985372100001, 2024-09-21T14:00:53.094 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97395471300000003, 2024-09-21T14:00:53.094 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:00:53.094 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:00:53.094 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T14:00:53.094 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:00:53.094 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T14:00:53.094 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:00:53.094 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232022 2024-09-21T14:00:53.094 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.094 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:00:53.094 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.095 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:00:53.095 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700289+0000", 2024-09-21T14:00:53.095 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:53.095 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.095 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.095 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:00:53.095 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700289+0000", 2024-09-21T14:00:53.095 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:53.095 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.095 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.095 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:00:53.095 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700291+0000", 2024-09-21T14:00:53.096 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5850000000000001e-06 2024-09-21T14:00:53.096 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.096 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.096 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:00:53.096 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700296+0000", 2024-09-21T14:00:53.096 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9749999999999998e-06 2024-09-21T14:00:53.096 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.096 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.096 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:00:53.096 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700306+0000", 2024-09-21T14:00:53.096 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9310000000000003e-06 2024-09-21T14:00:53.097 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.097 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.097 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:00:53.097 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700317+0000", 2024-09-21T14:00:53.097 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1199000000000001e-05 2024-09-21T14:00:53.097 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.097 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.097 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:53.097 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.711626+0000", 2024-09-21T14:00:53.097 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.011309009 2024-09-21T14:00:53.097 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.097 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.097 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:00:53.098 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.711637+0000", 2024-09-21T14:00:53.098 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1182000000000001e-05 2024-09-21T14:00:53.098 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.098 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.098 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:53.098 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674230+0000", 2024-09-21T14:00:53.098 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.962593535 2024-09-21T14:00:53.098 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.098 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.098 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:00:53.098 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674244+0000", 2024-09-21T14:00:53.099 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3297e-05 2024-09-21T14:00:53.099 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:53.099 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:00:53.099 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:53.099 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.099 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.099 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232025 8.f 8.4e5a668f (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T14:00:53.099 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.700329+0000", 2024-09-21T14:00:53.099 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 221.38981392299999, 2024-09-21T14:00:53.099 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97395089300000004, 2024-09-21T14:00:53.099 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:00:53.099 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:00:53.099 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T14:00:53.100 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:00:53.100 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T14:00:53.100 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:00:53.100 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232025 2024-09-21T14:00:53.100 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.100 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:00:53.100 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.100 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:00:53.100 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700329+0000", 2024-09-21T14:00:53.100 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:53.100 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.100 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.100 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:00:53.101 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700329+0000", 2024-09-21T14:00:53.101 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:53.101 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.101 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.101 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:00:53.101 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700329+0000", 2024-09-21T14:00:53.101 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7100000000000002e-07 2024-09-21T14:00:53.101 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.101 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.101 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:00:53.101 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700332+0000", 2024-09-21T14:00:53.101 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6589999999999999e-06 2024-09-21T14:00:53.101 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.102 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.102 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:00:53.102 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700339+0000", 2024-09-21T14:00:53.102 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8940000000000004e-06 2024-09-21T14:00:53.102 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.102 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.102 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:00:53.102 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700345+0000", 2024-09-21T14:00:53.103 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6219999999999997e-06 2024-09-21T14:00:53.103 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.103 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.103 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:53.103 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.711662+0000", 2024-09-21T14:00:53.103 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.011317522 2024-09-21T14:00:53.103 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.103 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.103 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:00:53.103 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.711669+0000", 2024-09-21T14:00:53.103 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2570000000000001e-06 2024-09-21T14:00:53.103 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.104 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.104 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:53.104 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674270+0000", 2024-09-21T14:00:53.104 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.96260044300000003 2024-09-21T14:00:53.104 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.104 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.104 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:00:53.104 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674280+0000", 2024-09-21T14:00:53.105 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0025e-05 2024-09-21T14:00:53.105 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:53.105 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:00:53.105 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:53.105 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.105 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.105 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232033 8.f 8.d949c02f (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T14:00:53.105 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.700352+0000", 2024-09-21T14:00:53.105 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 221.38979052299999, 2024-09-21T14:00:53.105 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97396006099999999, 2024-09-21T14:00:53.105 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:00:53.105 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:00:53.105 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T14:00:53.105 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:00:53.106 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T14:00:53.106 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:00:53.106 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232033 2024-09-21T14:00:53.106 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.106 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:00:53.106 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.106 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:00:53.106 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700352+0000", 2024-09-21T14:00:53.106 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:53.106 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.107 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.107 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:00:53.107 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700352+0000", 2024-09-21T14:00:53.107 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:53.107 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.107 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.107 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:00:53.107 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700353+0000", 2024-09-21T14:00:53.107 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.89e-07 2024-09-21T14:00:53.107 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.107 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.108 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:00:53.108 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700355+0000", 2024-09-21T14:00:53.108 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1840000000000002e-06 2024-09-21T14:00:53.108 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.108 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.108 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:00:53.108 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700364+0000", 2024-09-21T14:00:53.108 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4919999999999999e-06 2024-09-21T14:00:53.108 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.108 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.108 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:00:53.108 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700372+0000", 2024-09-21T14:00:53.108 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9659999999999996e-06 2024-09-21T14:00:53.109 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.109 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.109 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:53.109 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.711693+0000", 2024-09-21T14:00:53.109 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.011321059 2024-09-21T14:00:53.109 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.109 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.109 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:00:53.109 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.711701+0000", 2024-09-21T14:00:53.109 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3640000000000001e-06 2024-09-21T14:00:53.109 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.109 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.109 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:53.109 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674303+0000", 2024-09-21T14:00:53.109 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.96260253100000004 2024-09-21T14:00:53.110 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.110 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.110 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:00:53.110 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674312+0000", 2024-09-21T14:00:53.110 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0759999999999992e-06 2024-09-21T14:00:53.110 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:53.110 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:00:53.110 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:53.110 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.110 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.110 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232051 8.0 8.85c8f300 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T14:00:53.110 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.700380+0000", 2024-09-21T14:00:53.111 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 221.38976281999999, 2024-09-21T14:00:53.111 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97451152799999996, 2024-09-21T14:00:53.111 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:00:53.111 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:00:53.111 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T14:00:53.111 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:00:53.111 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T14:00:53.111 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:00:53.111 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232051 2024-09-21T14:00:53.111 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.111 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:00:53.111 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.111 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:00:53.111 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700380+0000", 2024-09-21T14:00:53.112 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:53.112 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.112 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.112 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:00:53.112 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700380+0000", 2024-09-21T14:00:53.112 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:53.112 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.112 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.112 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:00:53.112 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700380+0000", 2024-09-21T14:00:53.112 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1699999999999999e-07 2024-09-21T14:00:53.112 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.113 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.113 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:00:53.113 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700383+0000", 2024-09-21T14:00:53.113 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1600000000000001e-06 2024-09-21T14:00:53.113 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.113 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.113 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:00:53.113 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700389+0000", 2024-09-21T14:00:53.114 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6080000000000001e-06 2024-09-21T14:00:53.114 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.114 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.114 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:00:53.114 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700394+0000", 2024-09-21T14:00:53.114 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9810000000000003e-06 2024-09-21T14:00:53.114 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.114 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.114 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:53.114 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.715050+0000", 2024-09-21T14:00:53.114 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.014656269 2024-09-21T14:00:53.114 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.115 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.115 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:00:53.115 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.715056+0000", 2024-09-21T14:00:53.115 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0129999999999999e-06 2024-09-21T14:00:53.115 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.115 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.115 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:53.115 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674876+0000", 2024-09-21T14:00:53.115 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.95981962700000001 2024-09-21T14:00:53.115 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.115 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.115 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:00:53.115 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674892+0000", 2024-09-21T14:00:53.115 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5452999999999999e-05 2024-09-21T14:00:53.116 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:53.116 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:00:53.116 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:53.116 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.116 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.116 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232059 8.f 8.b757f3bf (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T14:00:53.116 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.889250+0000", 2024-09-21T14:00:53.116 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 221.20089300699999, 2024-09-21T14:00:53.116 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.78509475900000003, 2024-09-21T14:00:53.116 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:00:53.116 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:00:53.116 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T14:00:53.116 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:00:53.117 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T14:00:53.117 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:00:53.117 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232059 2024-09-21T14:00:53.117 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.117 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:00:53.117 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.117 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:00:53.117 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889250+0000", 2024-09-21T14:00:53.117 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:53.118 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.118 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.118 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:00:53.118 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889250+0000", 2024-09-21T14:00:53.118 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:53.118 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.118 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.118 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:00:53.118 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889252+0000", 2024-09-21T14:00:53.118 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9130000000000001e-06 2024-09-21T14:00:53.118 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.118 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.119 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:00:53.119 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889258+0000", 2024-09-21T14:00:53.119 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7410000000000003e-06 2024-09-21T14:00:53.119 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.119 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.119 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:00:53.119 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889360+0000", 2024-09-21T14:00:53.119 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000101382 2024-09-21T14:00:53.119 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.119 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.119 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:00:53.119 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889391+0000", 2024-09-21T14:00:53.119 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1032999999999999e-05 2024-09-21T14:00:53.119 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.120 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.120 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:53.120 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889500+0000", 2024-09-21T14:00:53.120 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000109194 2024-09-21T14:00:53.120 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.120 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.120 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:00:53.120 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889512+0000", 2024-09-21T14:00:53.120 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2368000000000001e-05 2024-09-21T14:00:53.120 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.120 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.120 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:53.120 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674335+0000", 2024-09-21T14:00:53.120 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.78482304000000003 2024-09-21T14:00:53.120 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.121 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.121 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:00:53.121 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674345+0000", 2024-09-21T14:00:53.121 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0880000000000001e-06 2024-09-21T14:00:53.121 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:53.121 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:00:53.121 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:53.121 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.122 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.122 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232060 8.b 8.6c14963b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T14:00:53.122 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.889442+0000", 2024-09-21T14:00:53.122 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 221.20070081899999, 2024-09-21T14:00:53.122 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.78518956100000004, 2024-09-21T14:00:53.122 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:00:53.122 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:00:53.122 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T14:00:53.122 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:00:53.122 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T14:00:53.123 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:00:53.123 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232060 2024-09-21T14:00:53.123 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.123 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:00:53.123 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.123 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:00:53.123 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889442+0000", 2024-09-21T14:00:53.123 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:53.123 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.123 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.123 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:00:53.124 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889442+0000", 2024-09-21T14:00:53.124 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:53.124 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.124 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.124 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:00:53.124 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889443+0000", 2024-09-21T14:00:53.124 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9100000000000003e-07 2024-09-21T14:00:53.124 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.124 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.124 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:00:53.125 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889447+0000", 2024-09-21T14:00:53.125 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4939999999999997e-06 2024-09-21T14:00:53.125 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.125 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.125 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:00:53.125 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889457+0000", 2024-09-21T14:00:53.125 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0145e-05 2024-09-21T14:00:53.125 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.125 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.125 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:00:53.125 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889465+0000", 2024-09-21T14:00:53.125 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2749999999999998e-06 2024-09-21T14:00:53.126 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.126 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.126 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:53.126 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889550+0000", 2024-09-21T14:00:53.126 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5000999999999995e-05 2024-09-21T14:00:53.126 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.126 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.126 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:00:53.126 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889561+0000", 2024-09-21T14:00:53.126 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.129e-05 2024-09-21T14:00:53.126 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.126 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.126 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:53.127 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:00:52 smithi187 podman[120409]: 2024-09-21 14:00:52.735970043 +0000 UTC m=+0.019065745 image pull b415816db8b41f544cdfdc71d193ff8b1ee008bf7b0ec4f64da8eb8f71b62e85 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5 2024-09-21T14:00:53.127 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:00:52 smithi187 podman[120409]: 2024-09-21 14:00:52.849143882 +0000 UTC m=+0.132239570 container create c757988e69ba2426ecae268eefd3ba6efd9b39791aa57f4c0568038bfc57f327 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5-deactivate, GIT_CLEAN=True, ceph=True, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-main-2024-09-21-0842-4417849, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-09-21T14:00:53.128 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:52 smithi187 ceph-mon[33032]: 3.1a scrub starts 2024-09-21T14:00:53.128 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:52 smithi187 ceph-mon[33032]: 3.1a scrub ok 2024-09-21T14:00:53.128 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:52 smithi187 ceph-mon[33032]: 9.d continuing backfill to osd.7 from (296'7120,324'11721] MIN to 324'11721 2024-09-21T14:00:53.128 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:52 smithi187 ceph-mon[33032]: osdmap e326: 8 total, 7 up, 8 in 2024-09-21T14:00:53.128 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:52 smithi187 ceph-mon[33032]: pgmap v597: 158 pgs: 1 active+remapped+backfilling, 1 stale+active+recovering+undersized+degraded+remapped, 1 stale+active+recovery_wait+undersized+degraded+remapped, 1 remapped, 21 stale+active+clean, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 11 peering, 1 active+recovering+undersized+degraded+remapped, 119 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s; 18626/169515 objects degraded (10.988%); 3103/169515 objects misplaced (1.831%); 9.9 MiB/s, 159 objects/s recovering 2024-09-21T14:00:53.128 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674619+0000", 2024-09-21T14:00:53.128 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.785057703 2024-09-21T14:00:53.128 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.128 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.128 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:00:53.128 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674632+0000", 2024-09-21T14:00:53.129 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2962000000000001e-05 2024-09-21T14:00:53.129 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:53.129 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:00:53.129 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:53.129 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.129 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.129 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232068 8.f 8.d5b3687f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T14:00:53.129 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.894747+0000", 2024-09-21T14:00:53.129 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 221.195396131, 2024-09-21T14:00:53.129 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.77963143999999995, 2024-09-21T14:00:53.129 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:00:53.129 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:00:53.129 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T14:00:53.130 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:00:53.130 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T14:00:53.130 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:00:53.130 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232068 2024-09-21T14:00:53.130 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.130 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:00:53.130 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.130 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:00:53.130 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.894747+0000", 2024-09-21T14:00:53.130 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:53.130 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.130 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.130 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:00:53.130 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.894747+0000", 2024-09-21T14:00:53.130 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:53.131 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.131 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.131 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:00:53.131 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.894749+0000", 2024-09-21T14:00:53.131 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4849999999999999e-06 2024-09-21T14:00:53.131 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.131 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.131 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:00:53.131 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.894761+0000", 2024-09-21T14:00:53.131 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1809e-05 2024-09-21T14:00:53.131 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.131 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.131 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:00:53.131 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.894780+0000", 2024-09-21T14:00:53.132 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8593999999999998e-05 2024-09-21T14:00:53.132 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.132 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.132 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:00:53.132 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.894789+0000", 2024-09-21T14:00:53.132 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2839999999999995e-06 2024-09-21T14:00:53.132 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.132 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.132 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:53.132 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.894865+0000", 2024-09-21T14:00:53.132 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6150999999999996e-05 2024-09-21T14:00:53.132 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.132 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.132 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:00:53.133 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.894874+0000", 2024-09-21T14:00:53.133 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2450000000000004e-06 2024-09-21T14:00:53.133 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.133 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.133 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:53.133 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674368+0000", 2024-09-21T14:00:53.133 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.77949425999999999 2024-09-21T14:00:53.133 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.133 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.133 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:00:53.133 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674378+0000", 2024-09-21T14:00:53.133 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6120000000000005e-06 2024-09-21T14:00:53.134 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:53.134 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:00:53.134 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:53.134 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.134 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.134 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232074 8.0 8.57810 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T14:00:53.134 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.901181+0000", 2024-09-21T14:00:53.134 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 221.18896161800001, 2024-09-21T14:00:53.134 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.77382758200000001, 2024-09-21T14:00:53.134 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:00:53.134 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:00:53.134 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T14:00:53.134 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:00:53.134 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T14:00:53.135 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:00:53.135 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232074 2024-09-21T14:00:53.135 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.135 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:00:53.135 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.135 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:00:53.135 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.901181+0000", 2024-09-21T14:00:53.135 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:53.135 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.135 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.135 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:00:53.135 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.901181+0000", 2024-09-21T14:00:53.135 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:53.135 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.135 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.136 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:00:53.136 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.901184+0000", 2024-09-21T14:00:53.136 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.8660000000000002e-06 2024-09-21T14:00:53.136 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.136 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.136 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:00:53.136 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.901193+0000", 2024-09-21T14:00:53.136 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1549999999999996e-06 2024-09-21T14:00:53.136 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.136 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.136 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:00:53.136 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.901215+0000", 2024-09-21T14:00:53.136 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2272e-05 2024-09-21T14:00:53.136 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.136 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.136 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:00:53.137 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.901227+0000", 2024-09-21T14:00:53.137 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1116e-05 2024-09-21T14:00:53.137 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.137 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.137 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:53.137 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.901351+0000", 2024-09-21T14:00:53.137 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000124172 2024-09-21T14:00:53.137 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.137 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.137 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:00:53.137 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.901368+0000", 2024-09-21T14:00:53.137 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7008999999999999e-05 2024-09-21T14:00:53.137 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.137 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.137 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:53.137 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674970+0000", 2024-09-21T14:00:53.138 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.77360225199999999 2024-09-21T14:00:53.138 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.138 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.138 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:00:53.138 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.675009+0000", 2024-09-21T14:00:53.138 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8739999999999998e-05 2024-09-21T14:00:53.138 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:53.138 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:00:53.138 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:53.138 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.138 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.138 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232078 8.b 8.7f49789b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T14:00:53.138 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.904662+0000", 2024-09-21T14:00:53.139 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 221.185481213, 2024-09-21T14:00:53.139 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.77001060399999999, 2024-09-21T14:00:53.139 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:00:53.139 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:00:53.139 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T14:00:53.139 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:00:53.139 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T14:00:53.139 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:00:53.139 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232078 2024-09-21T14:00:53.139 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.139 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:00:53.139 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.140 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:00:53.140 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904662+0000", 2024-09-21T14:00:53.140 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:53.140 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.140 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.140 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:00:53.140 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904662+0000", 2024-09-21T14:00:53.140 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:53.140 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.140 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.140 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:00:53.140 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904663+0000", 2024-09-21T14:00:53.140 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.548e-06 2024-09-21T14:00:53.140 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.141 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.141 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:00:53.141 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904669+0000", 2024-09-21T14:00:53.141 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9619999999999998e-06 2024-09-21T14:00:53.141 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.141 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.141 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:00:53.141 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904684+0000", 2024-09-21T14:00:53.141 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4718e-05 2024-09-21T14:00:53.141 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.141 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.141 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:00:53.141 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904691+0000", 2024-09-21T14:00:53.141 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.861e-06 2024-09-21T14:00:53.142 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.142 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.142 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:53.142 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904750+0000", 2024-09-21T14:00:53.142 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9533000000000001e-05 2024-09-21T14:00:53.142 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.142 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.142 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:00:53.142 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904757+0000", 2024-09-21T14:00:53.142 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0369999999999997e-06 2024-09-21T14:00:53.142 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.142 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.142 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:53.143 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674661+0000", 2024-09-21T14:00:53.143 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.769903754 2024-09-21T14:00:53.143 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.143 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.143 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:00:53.143 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674672+0000", 2024-09-21T14:00:53.143 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1191e-05 2024-09-21T14:00:53.143 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:53.143 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:00:53.143 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:53.143 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.144 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.144 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232079 8.b 8.5c3e90b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T14:00:53.144 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.904710+0000", 2024-09-21T14:00:53.144 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 221.18543285800001, 2024-09-21T14:00:53.144 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.76999876, 2024-09-21T14:00:53.144 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:00:53.144 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:00:53.144 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T14:00:53.144 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:00:53.144 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T14:00:53.144 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:00:53.144 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232079 2024-09-21T14:00:53.144 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.145 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:00:53.145 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.145 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:00:53.145 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904710+0000", 2024-09-21T14:00:53.145 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:53.145 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.145 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.145 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:00:53.145 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904710+0000", 2024-09-21T14:00:53.145 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:53.145 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.145 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.145 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:00:53.145 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904711+0000", 2024-09-21T14:00:53.146 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8199999999999999e-07 2024-09-21T14:00:53.146 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.146 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.146 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:00:53.146 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904714+0000", 2024-09-21T14:00:53.146 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9079999999999999e-06 2024-09-21T14:00:53.146 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.146 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.146 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:00:53.146 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904726+0000", 2024-09-21T14:00:53.146 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2514000000000001e-05 2024-09-21T14:00:53.146 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.146 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.146 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:00:53.147 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904733+0000", 2024-09-21T14:00:53.147 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2690000000000002e-06 2024-09-21T14:00:53.147 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.147 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.147 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:53.147 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904784+0000", 2024-09-21T14:00:53.147 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1007000000000001e-05 2024-09-21T14:00:53.147 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.147 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.147 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:00:53.147 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904794+0000", 2024-09-21T14:00:53.147 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7419999999999992e-06 2024-09-21T14:00:53.147 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.147 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.148 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:53.148 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674699+0000", 2024-09-21T14:00:53.148 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.76990482900000001 2024-09-21T14:00:53.148 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.148 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.148 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:00:53.148 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674709+0000", 2024-09-21T14:00:53.148 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8090000000000004e-06 2024-09-21T14:00:53.148 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:53.148 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:00:53.148 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:53.148 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.148 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.149 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25421.0:113916 9.8 9:1f612498:::benchmark_data_smithi049_138504_object113915:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-21T14:00:53.149 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:59:25.602805+0000", 2024-09-21T14:00:53.149 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 87.487337425999996, 2024-09-21T14:00:53.149 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.34059980099999998, 2024-09-21T14:00:53.149 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:00:53.149 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:00:53.149 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:00:53.149 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:00:53.149 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25421", 2024-09-21T14:00:53.149 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:00:53.149 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 113916 2024-09-21T14:00:53.149 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.149 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:00:53.149 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.150 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:00:53.150 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.602805+0000", 2024-09-21T14:00:53.150 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:53.150 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.150 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.150 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:00:53.150 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.602805+0000", 2024-09-21T14:00:53.150 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:53.150 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.150 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.150 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:00:53.150 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.602807+0000", 2024-09-21T14:00:53.150 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7099999999999999e-06 2024-09-21T14:00:53.150 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.151 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.151 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:00:53.151 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.602830+0000", 2024-09-21T14:00:53.151 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2758000000000001e-05 2024-09-21T14:00:53.151 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.151 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.151 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:00:53.151 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.602847+0000", 2024-09-21T14:00:53.151 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6721000000000001e-05 2024-09-21T14:00:53.151 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.151 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.151 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:00:53.151 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.602854+0000", 2024-09-21T14:00:53.152 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9100000000000005e-06 2024-09-21T14:00:53.152 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.152 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.152 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:53.152 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.602925+0000", 2024-09-21T14:00:53.152 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0088999999999999e-05 2024-09-21T14:00:53.152 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.152 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.152 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:00:53.152 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.603043+0000", 2024-09-21T14:00:53.152 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011804200000000001 2024-09-21T14:00:53.152 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.152 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.152 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,5", 2024-09-21T14:00:53.152 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.603177+0000", 2024-09-21T14:00:53.153 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000134821 2024-09-21T14:00:53.153 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.153 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.153 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:00:53.153 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.606802+0000", 2024-09-21T14:00:53.153 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0036243880000000001 2024-09-21T14:00:53.153 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.153 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.153 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:53.153 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.608461+0000", 2024-09-21T14:00:53.153 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0016595710000000001 2024-09-21T14:00:53.154 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.154 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.154 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:53.154 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.943300+0000", 2024-09-21T14:00:53.154 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.33483894400000003 2024-09-21T14:00:53.154 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.154 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.154 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:00:53.154 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.943360+0000", 2024-09-21T14:00:53.154 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9401999999999998e-05 2024-09-21T14:00:53.154 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.154 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.154 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:00:53.155 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.943405+0000", 2024-09-21T14:00:53.155 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5445000000000003e-05 2024-09-21T14:00:53.155 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:53.155 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:00:53.155 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:53.155 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.155 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.155 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25421.0:113943 9.8 9:1e81a0a3:::benchmark_data_smithi049_138504_object113942:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-21T14:00:53.155 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:59:25.623062+0000", 2024-09-21T14:00:53.155 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 87.467080663000004, 2024-09-21T14:00:53.155 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.32046552900000003, 2024-09-21T14:00:53.155 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:00:53.156 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:00:53.156 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:00:53.156 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:00:53.156 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25421", 2024-09-21T14:00:53.156 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:00:53.156 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 113943 2024-09-21T14:00:53.156 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.156 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:00:53.156 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.156 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:00:53.156 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.623062+0000", 2024-09-21T14:00:53.156 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:53.156 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.157 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.157 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:00:53.157 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:00:53.158 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.623062+0000", 2024-09-21T14:00:53.158 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:53.158 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.158 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.158 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:00:53.158 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.623063+0000", 2024-09-21T14:00:53.158 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0210000000000001e-06 2024-09-21T14:00:53.158 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.159 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.159 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:00:53.159 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.623089+0000", 2024-09-21T14:00:53.159 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.567e-05 2024-09-21T14:00:53.159 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.159 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.159 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:00:53.159 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.623103+0000", 2024-09-21T14:00:53.159 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4248e-05 2024-09-21T14:00:53.159 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.159 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.159 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:00:53.159 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.623109+0000", 2024-09-21T14:00:53.159 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5949999999999998e-06 2024-09-21T14:00:53.159 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.160 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.160 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:53.160 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.623150+0000", 2024-09-21T14:00:53.160 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0868999999999998e-05 2024-09-21T14:00:53.160 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.160 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.160 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:00:53.160 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.623240+0000", 2024-09-21T14:00:53.160 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0562000000000005e-05 2024-09-21T14:00:53.160 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.160 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.160 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,5", 2024-09-21T14:00:53.160 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.623353+0000", 2024-09-21T14:00:53.160 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000113249 2024-09-21T14:00:53.161 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.161 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.161 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:00:53.161 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.625438+0000", 2024-09-21T14:00:53.161 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0020847750000000001 2024-09-21T14:00:53.161 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.161 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.161 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:53.161 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.626378+0000", 2024-09-21T14:00:53.161 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00093971199999999999 2024-09-21T14:00:53.161 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.161 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.161 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:53.161 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.943444+0000", 2024-09-21T14:00:53.161 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.317066615 2024-09-21T14:00:53.162 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.162 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.162 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:00:53.162 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.943487+0000", 2024-09-21T14:00:53.162 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2933999999999999e-05 2024-09-21T14:00:53.162 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.162 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.162 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:00:53.162 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.943528+0000", 2024-09-21T14:00:53.162 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0278999999999998e-05 2024-09-21T14:00:53.162 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:53.162 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:00:53.162 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:53.162 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.163 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.163 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25421.0:113948 9.8 9:1fea698d:::benchmark_data_smithi049_138504_object113947:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-21T14:00:53.163 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:59:25.630511+0000", 2024-09-21T14:00:53.163 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 87.459631376999994, 2024-09-21T14:00:53.163 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.31617158400000001, 2024-09-21T14:00:53.163 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:00:53.163 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:00:53.163 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:00:53.163 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:00:53.163 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25421", 2024-09-21T14:00:53.163 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:00:53.163 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 113948 2024-09-21T14:00:53.164 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.164 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:00:53.164 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.164 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:00:53.164 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.630511+0000", 2024-09-21T14:00:53.164 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:53.164 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.164 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.164 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:00:53.164 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.630511+0000", 2024-09-21T14:00:53.164 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:53.164 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.164 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.165 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:00:53.165 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.630512+0000", 2024-09-21T14:00:53.165 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.16e-07 2024-09-21T14:00:53.165 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.165 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.165 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:00:53.165 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.630531+0000", 2024-09-21T14:00:53.165 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8845999999999999e-05 2024-09-21T14:00:53.165 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.165 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.165 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:00:53.166 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.630542+0000", 2024-09-21T14:00:53.166 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0651e-05 2024-09-21T14:00:53.166 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.166 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.166 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:00:53.166 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.630547+0000", 2024-09-21T14:00:53.166 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8770000000000001e-06 2024-09-21T14:00:53.166 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.166 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.166 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:53.166 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.630585+0000", 2024-09-21T14:00:53.166 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8488999999999997e-05 2024-09-21T14:00:53.167 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.167 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.167 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:00:53.167 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.630660+0000", 2024-09-21T14:00:53.167 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5290000000000006e-05 2024-09-21T14:00:53.167 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.167 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.167 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,5", 2024-09-21T14:00:53.167 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.630760+0000", 2024-09-21T14:00:53.167 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9834000000000007e-05 2024-09-21T14:00:53.167 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.167 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.167 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:00:53.167 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.632607+0000", 2024-09-21T14:00:53.167 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001847174 2024-09-21T14:00:53.168 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.168 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.168 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:53.168 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.633151+0000", 2024-09-21T14:00:53.168 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00054377600000000005 2024-09-21T14:00:53.168 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.168 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.168 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:53.168 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.946575+0000", 2024-09-21T14:00:53.168 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.31342341099999999 2024-09-21T14:00:53.168 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.168 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.168 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:00:53.168 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.946632+0000", 2024-09-21T14:00:53.169 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7055999999999997e-05 2024-09-21T14:00:53.169 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.169 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.169 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:00:53.169 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.946683+0000", 2024-09-21T14:00:53.169 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1264000000000002e-05 2024-09-21T14:00:53.169 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:53.169 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:00:53.169 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:53.169 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.169 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.169 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25421.0:172314 9.4 e315/260 9:2b749f7e:::benchmark_data_smithi049_138504_object50906:head v 315'10563, mlcod=315'10559)", 2024-09-21T14:00:53.170 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:00:33.250136+0000", 2024-09-21T14:00:53.170 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 19.840006745, 2024-09-21T14:00:53.170 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.270468655, 2024-09-21T14:00:53.170 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:00:53.170 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:00:53.170 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:00:53.170 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:00:53.170 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.170 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:00:53.170 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250136+0000", 2024-09-21T14:00:53.170 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:53.170 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.170 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.170 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:00:53.171 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250136+0000", 2024-09-21T14:00:53.171 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:53.171 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.171 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.171 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:00:53.171 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250137+0000", 2024-09-21T14:00:53.171 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1999999999999999e-07 2024-09-21T14:00:53.171 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.171 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.171 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:00:53.171 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250142+0000", 2024-09-21T14:00:53.171 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5609999999999998e-06 2024-09-21T14:00:53.171 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.171 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.172 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:00:53.172 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250150+0000", 2024-09-21T14:00:53.172 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1310000000000006e-06 2024-09-21T14:00:53.172 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.172 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.172 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:00:53.172 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250156+0000", 2024-09-21T14:00:53.172 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7740000000000001e-06 2024-09-21T14:00:53.172 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.172 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.172 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:53.172 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.253391+0000", 2024-09-21T14:00:53.172 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0032349969999999999 2024-09-21T14:00:53.172 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.173 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.173 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:00:53.173 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.253410+0000", 2024-09-21T14:00:53.173 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9097e-05 2024-09-21T14:00:53.173 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.173 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.173 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:00:53.173 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.520555+0000", 2024-09-21T14:00:53.173 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26714502600000001 2024-09-21T14:00:53.173 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.173 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.173 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:00:53.174 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.520605+0000", 2024-09-21T14:00:53.174 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9449e-05 2024-09-21T14:00:53.174 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:53.174 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:00:53.174 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:53.174 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.174 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.174 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25421.0:172319 9.4 e315/260 9:21e132db:::benchmark_data_smithi049_138504_object50911:head v 315'10564, mlcod=315'10559)", 2024-09-21T14:00:53.174 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:00:33.250191+0000", 2024-09-21T14:00:53.174 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 19.839951761000002, 2024-09-21T14:00:53.174 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.270657121, 2024-09-21T14:00:53.174 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:00:53.175 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:00:53.175 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:00:53.175 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:00:53.175 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.175 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:00:53.175 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250191+0000", 2024-09-21T14:00:53.175 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:53.175 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.175 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.175 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:00:53.175 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250191+0000", 2024-09-21T14:00:53.175 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:53.176 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.176 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.176 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:00:53.176 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250191+0000", 2024-09-21T14:00:53.176 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8799999999999998e-07 2024-09-21T14:00:53.176 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.176 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.176 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:00:53.176 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250195+0000", 2024-09-21T14:00:53.176 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.647e-06 2024-09-21T14:00:53.176 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.176 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.176 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:00:53.176 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250201+0000", 2024-09-21T14:00:53.177 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4200000000000004e-06 2024-09-21T14:00:53.177 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.177 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.177 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:00:53.177 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250206+0000", 2024-09-21T14:00:53.177 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.994e-06 2024-09-21T14:00:53.177 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.177 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.177 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:53.177 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.256110+0000", 2024-09-21T14:00:53.177 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0059039840000000001 2024-09-21T14:00:53.178 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.178 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.178 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:00:53.178 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.256130+0000", 2024-09-21T14:00:53.178 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.98e-05 2024-09-21T14:00:53.178 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.178 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.178 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:00:53.178 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.520816+0000", 2024-09-21T14:00:53.178 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.264685528 2024-09-21T14:00:53.178 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.178 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.178 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:00:53.179 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.520848+0000", 2024-09-21T14:00:53.179 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2360000000000002e-05 2024-09-21T14:00:53.179 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:53.179 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:00:53.179 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:53.179 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.179 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.179 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25421.0:172326 9.4 e315/260 9:22a8fc8a:::benchmark_data_smithi049_138504_object50918:head v 315'10565, mlcod=315'10560)", 2024-09-21T14:00:53.179 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:00:33.254759+0000", 2024-09-21T14:00:53.179 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 19.835383471, 2024-09-21T14:00:53.179 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26611502300000001, 2024-09-21T14:00:53.179 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:00:53.179 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:00:53.180 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:00:53.180 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:00:53.180 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.180 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:00:53.180 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254759+0000", 2024-09-21T14:00:53.180 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:53.180 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.180 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.180 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:00:53.180 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254759+0000", 2024-09-21T14:00:53.180 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:53.180 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.180 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.180 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:00:53.181 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254760+0000", 2024-09-21T14:00:53.181 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6400000000000001e-07 2024-09-21T14:00:53.181 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.181 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.181 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:00:53.181 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254854+0000", 2024-09-21T14:00:53.181 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4320999999999994e-05 2024-09-21T14:00:53.181 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.181 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.181 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:00:53.181 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254865+0000", 2024-09-21T14:00:53.181 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.101e-05 2024-09-21T14:00:53.182 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.182 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.182 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:00:53.182 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254872+0000", 2024-09-21T14:00:53.182 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2539999999999998e-06 2024-09-21T14:00:53.182 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.183 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.183 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:53.183 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.258679+0000", 2024-09-21T14:00:53.183 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.003807342 2024-09-21T14:00:53.183 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.183 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.183 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:00:53.183 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.258699+0000", 2024-09-21T14:00:53.184 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.039e-05 2024-09-21T14:00:53.184 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.184 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.184 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:00:53.184 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.520851+0000", 2024-09-21T14:00:53.184 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26215118900000001 2024-09-21T14:00:53.184 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.184 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.184 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:00:53.184 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.520874+0000", 2024-09-21T14:00:53.185 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3753000000000001e-05 2024-09-21T14:00:53.185 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:53.185 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:00:53.185 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:53.185 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.185 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.185 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25421.0:172331 9.f e315/260 9:f6449e5b:::benchmark_data_smithi049_138504_object50923:head v 315'10871, mlcod=315'10867)", 2024-09-21T14:00:53.185 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:00:33.254942+0000", 2024-09-21T14:00:53.185 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 19.835201093999999, 2024-09-21T14:00:53.185 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.274133181, 2024-09-21T14:00:53.185 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:00:53.185 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:00:53.185 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:00:53.185 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:00:53.186 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.186 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:00:53.186 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254942+0000", 2024-09-21T14:00:53.186 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:53.186 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.186 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.187 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:00:53.187 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254942+0000", 2024-09-21T14:00:53.187 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:53.187 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.187 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.187 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:00:53.187 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254942+0000", 2024-09-21T14:00:53.187 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5700000000000002e-07 2024-09-21T14:00:53.187 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.187 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.188 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:00:53.188 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254949+0000", 2024-09-21T14:00:53.188 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.024e-06 2024-09-21T14:00:53.188 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.188 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.188 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:00:53.188 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254959+0000", 2024-09-21T14:00:53.188 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6390000000000004e-06 2024-09-21T14:00:53.188 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.189 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.189 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:00:53.189 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254965+0000", 2024-09-21T14:00:53.189 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6420000000000004e-06 2024-09-21T14:00:53.189 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.189 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.189 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:53.189 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.259407+0000", 2024-09-21T14:00:53.189 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0044420140000000002 2024-09-21T14:00:53.190 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.190 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.190 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:00:53.190 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.259425+0000", 2024-09-21T14:00:53.190 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7937000000000001e-05 2024-09-21T14:00:53.190 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.190 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.190 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:00:53.190 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.529033+0000", 2024-09-21T14:00:53.190 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26960846599999999 2024-09-21T14:00:53.190 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.190 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.190 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:00:53.191 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.529075+0000", 2024-09-21T14:00:53.191 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1801999999999997e-05 2024-09-21T14:00:53.191 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:53.191 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:00:53.191 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:53.191 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.191 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.191 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25421.0:172333 9.2 9:461458cc:::benchmark_data_smithi049_138504_object50925:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-09-21T14:00:53.191 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:00:33.258960+0000", 2024-09-21T14:00:53.191 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 19.831182958999999, 2024-09-21T14:00:53.191 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.261926308, 2024-09-21T14:00:53.191 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:00:53.191 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:00:53.191 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:00:53.192 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:00:53.192 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25421", 2024-09-21T14:00:53.192 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:00:53.192 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 172333 2024-09-21T14:00:53.192 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.192 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:00:53.192 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.192 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:00:53.192 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.258960+0000", 2024-09-21T14:00:53.192 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:53.192 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.192 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.192 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:00:53.192 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.258960+0000", 2024-09-21T14:00:53.193 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:53.193 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.193 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.193 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:00:53.193 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.258961+0000", 2024-09-21T14:00:53.193 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5629999999999999e-06 2024-09-21T14:00:53.193 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.193 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.193 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:00:53.193 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.258967+0000", 2024-09-21T14:00:53.193 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5079999999999996e-06 2024-09-21T14:00:53.193 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.193 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.193 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:00:53.194 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.258979+0000", 2024-09-21T14:00:53.194 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1897e-05 2024-09-21T14:00:53.194 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.194 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.194 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:00:53.194 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.258986+0000", 2024-09-21T14:00:53.194 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8739999999999997e-06 2024-09-21T14:00:53.194 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.194 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.194 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:53.194 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.259036+0000", 2024-09-21T14:00:53.194 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0695000000000002e-05 2024-09-21T14:00:53.194 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.194 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.195 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:00:53.195 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.259144+0000", 2024-09-21T14:00:53.195 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000107462 2024-09-21T14:00:53.195 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.195 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.195 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,2", 2024-09-21T14:00:53.195 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.259252+0000", 2024-09-21T14:00:53.195 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000107667 2024-09-21T14:00:53.195 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.195 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.195 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:00:53.195 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.517843+0000", 2024-09-21T14:00:53.195 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.25859111400000001 2024-09-21T14:00:53.195 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.196 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.196 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:53.196 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.517994+0000", 2024-09-21T14:00:53.196 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00015150299999999999 2024-09-21T14:00:53.196 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.196 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.196 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:00:53.196 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.520805+0000", 2024-09-21T14:00:53.196 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0028113489999999999 2024-09-21T14:00:53.196 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.196 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.196 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:00:53.196 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.520850+0000", 2024-09-21T14:00:53.196 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4333999999999999e-05 2024-09-21T14:00:53.197 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.197 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.197 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:00:53.197 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.520886+0000", 2024-09-21T14:00:53.197 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6341999999999999e-05 2024-09-21T14:00:53.197 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:53.197 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:00:53.197 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:53.197 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.197 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.197 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25421.0:172330 9.4 e315/260 9:2882ed62:::benchmark_data_smithi049_138504_object50922:head v 315'10566, mlcod=315'10560)", 2024-09-21T14:00:53.197 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:00:33.259976+0000", 2024-09-21T14:00:53.197 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 19.830166464000001, 2024-09-21T14:00:53.197 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26092217499999998, 2024-09-21T14:00:53.197 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:00:53.198 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:00:53.198 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:00:53.198 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:00:53.198 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.198 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:00:53.198 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.259976+0000", 2024-09-21T14:00:53.198 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:53.198 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.198 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.198 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:00:53.198 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.259976+0000", 2024-09-21T14:00:53.198 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:53.198 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.198 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.199 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:00:53.199 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.259977+0000", 2024-09-21T14:00:53.199 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1e-07 2024-09-21T14:00:53.199 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.199 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.199 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:00:53.199 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.259980+0000", 2024-09-21T14:00:53.199 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.58e-06 2024-09-21T14:00:53.199 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.199 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.199 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:00:53.199 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.259986+0000", 2024-09-21T14:00:53.199 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9109999999999998e-06 2024-09-21T14:00:53.199 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.200 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.200 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:00:53.200 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.259991+0000", 2024-09-21T14:00:53.200 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4510000000000002e-06 2024-09-21T14:00:53.200 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.200 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.200 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:53.200 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.261224+0000", 2024-09-21T14:00:53.200 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0012338169999999999 2024-09-21T14:00:53.200 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.200 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.200 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:00:53.200 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.261241+0000", 2024-09-21T14:00:53.200 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6762e-05 2024-09-21T14:00:53.200 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.201 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.201 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:00:53.201 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.520876+0000", 2024-09-21T14:00:53.201 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.25963512500000002 2024-09-21T14:00:53.201 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.201 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.201 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:00:53.201 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.520899+0000", 2024-09-21T14:00:53.201 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2218999999999999e-05 2024-09-21T14:00:53.201 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:53.201 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:00:53.201 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:53.201 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.201 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.201 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25421.0:172334 9.4 e315/260 9:24fa2bd3:::benchmark_data_smithi049_138504_object50926:head v 315'10567, mlcod=315'10561)", 2024-09-21T14:00:53.201 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:00:33.262387+0000", 2024-09-21T14:00:53.202 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 19.827756252, 2024-09-21T14:00:53.202 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26075252599999998, 2024-09-21T14:00:53.202 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:00:53.202 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:00:53.202 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:00:53.202 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:00:53.202 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.202 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:00:53.202 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.262387+0000", 2024-09-21T14:00:53.202 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:53.202 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.202 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.202 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:00:53.202 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.262387+0000", 2024-09-21T14:00:53.203 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:00:53.203 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.203 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.203 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:00:53.203 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.262388+0000", 2024-09-21T14:00:53.203 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6099999999999999e-07 2024-09-21T14:00:53.203 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.203 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.203 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:00:53.203 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.262393+0000", 2024-09-21T14:00:53.203 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8289999999999996e-06 2024-09-21T14:00:53.203 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.203 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.203 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:00:53.203 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.262488+0000", 2024-09-21T14:00:53.204 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4914000000000006e-05 2024-09-21T14:00:53.204 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.204 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.204 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:00:53.204 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.262504+0000", 2024-09-21T14:00:53.204 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5498999999999998e-05 2024-09-21T14:00:53.204 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.204 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.204 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:00:53.204 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.263573+0000", 2024-09-21T14:00:53.204 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0010692119999999999 2024-09-21T14:00:53.204 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.204 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.204 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:00:53.205 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.263590+0000", 2024-09-21T14:00:53.205 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7331000000000001e-05 2024-09-21T14:00:53.205 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.205 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.205 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:00:53.205 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.523109+0000", 2024-09-21T14:00:53.205 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.25951882100000001 2024-09-21T14:00:53.205 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:00:53.205 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:00:53.205 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:00:53.205 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.523139+0000", 2024-09-21T14:00:53.205 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9958999999999999e-05 2024-09-21T14:00:53.205 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:53.205 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:00:53.206 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:53.206 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:00:53.206 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:00:53.206 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T14:00:53.206 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:53.231 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:53.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:53.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:53.333 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:53.359 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:53.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:53.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:53.445 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:00:53 smithi187 podman[120409]: 2024-09-21 14:00:53.118986764 +0000 UTC m=+0.402082459 container init c757988e69ba2426ecae268eefd3ba6efd9b39791aa57f4c0568038bfc57f327 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-guits-main-2024-09-21-0842-4417849, ceph=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T14:00:53.445 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:00:53 smithi187 podman[120409]: 2024-09-21 14:00:53.125905207 +0000 UTC m=+0.409000904 container start c757988e69ba2426ecae268eefd3ba6efd9b39791aa57f4c0568038bfc57f327 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-21-0842-4417849, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-09-21T14:00:53.445 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:00:53 smithi187 podman[120409]: 2024-09-21 14:00:53.218887914 +0000 UTC m=+0.501983611 container attach c757988e69ba2426ecae268eefd3ba6efd9b39791aa57f4c0568038bfc57f327 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-21-0842-4417849, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, ceph=True, io.buildah.version=1.37.2) 2024-09-21T14:00:53.461 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:53.486 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:53.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:53.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:53.588 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:00:53.612 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:53.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:53.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:00:53.714 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:53.738 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:53.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:53.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:53.831 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:00:53 smithi187 podman[120409]: 2024-09-21 14:00:53.458915637 +0000 UTC m=+0.742011332 container died c757988e69ba2426ecae268eefd3ba6efd9b39791aa57f4c0568038bfc57f327 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-21-0842-4417849, ceph=True) 2024-09-21T14:00:53.840 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:53.863 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:53.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:53.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:53.965 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:00:53.991 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:53.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:53.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:00:54.093 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:54.117 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:54.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:54.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:54.140 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:53 smithi187 ceph-mon[33032]: Health check failed: Reduced data availability: 4 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-09-21T14:00:54.140 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:53 smithi187 ceph-mon[33032]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T14:00:54.140 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:53 smithi187 ceph-mon[33032]: osdmap e327: 8 total, 7 up, 8 in 2024-09-21T14:00:54.157 DEBUG:teuthology.orchestra.run.smithi187:> sudo pkill -f 'journalctl -f -n 0 -u ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@osd.5.service' 2024-09-21T14:00:54.206 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T14:00:54.206 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-21T14:00:54.206 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.5 2024-09-21T14:00:54.207 INFO:tasks.thrashosds.thrasher:remote for osd 5 is ubuntu@smithi187.front.sepia.ceph.com 2024-09-21T14:00:54.207 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell -n osd.5 --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --op list-pgs 2024-09-21T14:00:54.218 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:54.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:53 smithi049 ceph-mon[30080]: Health check failed: Reduced data availability: 4 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-09-21T14:00:54.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:53 smithi049 ceph-mon[30080]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T14:00:54.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:53 smithi049 ceph-mon[30080]: osdmap e327: 8 total, 7 up, 8 in 2024-09-21T14:00:54.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:53 smithi049 ceph-mon[40489]: Health check failed: Reduced data availability: 4 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-09-21T14:00:54.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:53 smithi049 ceph-mon[40489]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T14:00:54.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:53 smithi049 ceph-mon[40489]: osdmap e327: 8 total, 7 up, 8 in 2024-09-21T14:00:54.277 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:54.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:54.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:54.379 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:54.412 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:54.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:54.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:54.495 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/osd.5/config 2024-09-21T14:00:54.514 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:00:54.536 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T14:00:54.539 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:54.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:54.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:00:54.642 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:00:54.671 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:54.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:54.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:00:54.773 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:54.802 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:54.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:54.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:54.904 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:00:54.931 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:54.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:54.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:00:55.033 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:00:55.058 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:55.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:55.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:00:55.160 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:00:55.185 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:55.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:55.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:00:55.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:54 smithi187 ceph-mon[33032]: pgmap v599: 158 pgs: 1 active+remapped+backfilling, 1 stale+active+recovering+undersized+degraded+remapped, 1 stale+active+recovery_wait+undersized+degraded+remapped, 1 remapped, 13 stale+active+clean, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 11 peering, 14 active+undersized, 1 active+recovering+undersized+degraded+remapped, 7 active+undersized+degraded, 106 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 876 op/s; 25542/168978 objects degraded (15.116%); 3103/168978 objects misplaced (1.836%); 7.9 MiB/s, 127 objects/s recovering 2024-09-21T14:00:55.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:54 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:00:55.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:54 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:00:55.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:54 smithi049 ceph-mon[40489]: pgmap v599: 158 pgs: 1 active+remapped+backfilling, 1 stale+active+recovering+undersized+degraded+remapped, 1 stale+active+recovery_wait+undersized+degraded+remapped, 1 remapped, 13 stale+active+clean, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 11 peering, 14 active+undersized, 1 active+recovering+undersized+degraded+remapped, 7 active+undersized+degraded, 106 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 876 op/s; 25542/168978 objects degraded (15.116%); 3103/168978 objects misplaced (1.836%); 7.9 MiB/s, 127 objects/s recovering 2024-09-21T14:00:55.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:54 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:00:55.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:54 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:00:55.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:54 smithi049 ceph-mon[30080]: pgmap v599: 158 pgs: 1 active+remapped+backfilling, 1 stale+active+recovering+undersized+degraded+remapped, 1 stale+active+recovery_wait+undersized+degraded+remapped, 1 remapped, 13 stale+active+clean, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 11 peering, 14 active+undersized, 1 active+recovering+undersized+degraded+remapped, 7 active+undersized+degraded, 106 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 876 op/s; 25542/168978 objects degraded (15.116%); 3103/168978 objects misplaced (1.836%); 7.9 MiB/s, 127 objects/s recovering 2024-09-21T14:00:55.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:54 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:00:55.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:54 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:00:55.287 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:00:55.314 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:55.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:55.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:00:55.415 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:00:55.440 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:55.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:55.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:00:55.542 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:55.566 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:55.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:55.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:55.668 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:55.694 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:55.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:55.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:55.796 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:00:55.815 INFO:teuthology.orchestra.run.smithi187.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-21T14:00:55.824 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:55.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:55.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:00:55.925 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:55.952 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:55.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:55.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:56.014 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:00:56.054 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:56.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:56.079 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:56.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:56.089 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:55 smithi187 ceph-mon[33032]: osdmap e328: 8 total, 7 up, 8 in 2024-09-21T14:00:56.089 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:55 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 25542/168978 objects degraded (15.116%), 12 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T14:00:56.180 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:56.208 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:56.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:56.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:56.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:55 smithi049 ceph-mon[40489]: osdmap e328: 8 total, 7 up, 8 in 2024-09-21T14:00:56.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:55 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 25542/168978 objects degraded (15.116%), 12 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T14:00:56.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:55 smithi049 ceph-mon[30080]: osdmap e328: 8 total, 7 up, 8 in 2024-09-21T14:00:56.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:55 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 25542/168978 objects degraded (15.116%), 12 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T14:00:56.309 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:56.335 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:56.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:56.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:56.437 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:56.461 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:56.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:56.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:56.563 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:00:56.588 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:56.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:56.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:00:56.689 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:56.712 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:56.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:56.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:56.813 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:56.843 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:56.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:56.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:56.944 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:00:56.970 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:56.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:56.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:00:57.072 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:00:57.094 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:57.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:57.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:00:57.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:56 smithi187 ceph-mon[33032]: pgmap v601: 158 pgs: 1 undersized+degraded+remapped+backfill_toofull+peered, 3 active+recovering+undersized+degraded, 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfilling, 11 peering, 27 active+undersized, 19 active+undersized+degraded, 95 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 394 op/s; 47701/167034 objects degraded (28.558%); 3103/167034 objects misplaced (1.858%); 6.4 MiB/s, 99 objects/s recovering 2024-09-21T14:00:57.196 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:00:57.221 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:57.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:57.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:00:57.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:56 smithi049 ceph-mon[30080]: pgmap v601: 158 pgs: 1 undersized+degraded+remapped+backfill_toofull+peered, 3 active+recovering+undersized+degraded, 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfilling, 11 peering, 27 active+undersized, 19 active+undersized+degraded, 95 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 394 op/s; 47701/167034 objects degraded (28.558%); 3103/167034 objects misplaced (1.858%); 6.4 MiB/s, 99 objects/s recovering 2024-09-21T14:00:57.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:56 smithi049 ceph-mon[40489]: pgmap v601: 158 pgs: 1 undersized+degraded+remapped+backfill_toofull+peered, 3 active+recovering+undersized+degraded, 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfilling, 11 peering, 27 active+undersized, 19 active+undersized+degraded, 95 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 394 op/s; 47701/167034 objects degraded (28.558%); 3103/167034 objects misplaced (1.858%); 6.4 MiB/s, 99 objects/s recovering 2024-09-21T14:00:57.323 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:57.349 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:57.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:57.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:57.451 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:57.477 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:57.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:57.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:57.579 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:00:57.608 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:57.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:57.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:00:57.710 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:00:57.737 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:57.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:57.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:00:57.838 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:57.866 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:57.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:57.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:57.968 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:00:57.992 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:57.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:57.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:00:58.093 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:00:58.121 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:58.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:58.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:00:58.224 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:00:58.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:58.249 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:58.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:00:58.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:57 smithi049 ceph-mon[40489]: 5.11 scrub starts 2024-09-21T14:00:58.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:58 smithi049 ceph-mon[40489]: 5.11 scrub ok 2024-09-21T14:00:58.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:57 smithi049 ceph-mon[30080]: 5.11 scrub starts 2024-09-21T14:00:58.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:58 smithi049 ceph-mon[30080]: 5.11 scrub ok 2024-09-21T14:00:58.306 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:57 smithi187 ceph-mon[33032]: 5.11 scrub starts 2024-09-21T14:00:58.307 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:57 smithi187 ceph-mon[33032]: 5.11 scrub ok 2024-09-21T14:00:58.351 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:00:58.376 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:58.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:58.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:00:58.478 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:00:58.505 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:58.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:58.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:00:58.607 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:58.630 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:58.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:58.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:58.732 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:58.755 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:58.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:58.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:58.857 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:00:58.880 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:58.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:58.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:00:58.982 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:00:59.006 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:59.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:59.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:00:59.108 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:00:59.130 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:59.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:59.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:00:59.232 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:00:59.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:58 smithi049 ceph-mon[40489]: pgmap v602: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+clean+remapped, 1 active+remapped+backfilling, 1 undersized+degraded+remapped+backfill_toofull+peered, 3 active+recovering+undersized+degraded, 35 active+undersized, 21 active+undersized+degraded, 95 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 848 B/s rd, 0 B/s wr, 140 op/s; 48700/166983 objects degraded (29.165%); 1805/166983 objects misplaced (1.081%); 43 MiB/s, 678 objects/s recovering 2024-09-21T14:00:59.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:58 smithi049 ceph-mon[40489]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 5 pgs peering) 2024-09-21T14:00:59.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:58 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:00:59.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:00:58 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:00:59.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:58 smithi049 ceph-mon[30080]: pgmap v602: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+clean+remapped, 1 active+remapped+backfilling, 1 undersized+degraded+remapped+backfill_toofull+peered, 3 active+recovering+undersized+degraded, 35 active+undersized, 21 active+undersized+degraded, 95 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 848 B/s rd, 0 B/s wr, 140 op/s; 48700/166983 objects degraded (29.165%); 1805/166983 objects misplaced (1.081%); 43 MiB/s, 678 objects/s recovering 2024-09-21T14:00:59.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:58 smithi049 ceph-mon[30080]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 5 pgs peering) 2024-09-21T14:00:59.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:58 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:00:59.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:00:58 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:00:59.255 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:59.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:59.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:00:59.257 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:58 smithi187 ceph-mon[33032]: pgmap v602: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+clean+remapped, 1 active+remapped+backfilling, 1 undersized+degraded+remapped+backfill_toofull+peered, 3 active+recovering+undersized+degraded, 35 active+undersized, 21 active+undersized+degraded, 95 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 848 B/s rd, 0 B/s wr, 140 op/s; 48700/166983 objects degraded (29.165%); 1805/166983 objects misplaced (1.081%); 43 MiB/s, 678 objects/s recovering 2024-09-21T14:00:59.257 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:58 smithi187 ceph-mon[33032]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 5 pgs peering) 2024-09-21T14:00:59.257 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:58 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:00:59.257 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:00:58 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:00:59.357 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:00:59.381 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:59.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:59.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:00:59.483 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:00:59.488 INFO:teuthology.orchestra.run.smithi187.stdout:9.f 2024-09-21T14:00:59.488 INFO:teuthology.orchestra.run.smithi187.stdout:9.d 2024-09-21T14:00:59.488 INFO:teuthology.orchestra.run.smithi187.stdout:5.15 2024-09-21T14:00:59.488 INFO:teuthology.orchestra.run.smithi187.stdout:3.d 2024-09-21T14:00:59.488 INFO:teuthology.orchestra.run.smithi187.stdout:5.7 2024-09-21T14:00:59.488 INFO:teuthology.orchestra.run.smithi187.stdout:4.7 2024-09-21T14:00:59.488 INFO:teuthology.orchestra.run.smithi187.stdout:4.6 2024-09-21T14:00:59.488 INFO:teuthology.orchestra.run.smithi187.stdout:5.4 2024-09-21T14:00:59.488 INFO:teuthology.orchestra.run.smithi187.stdout:5.6 2024-09-21T14:00:59.488 INFO:teuthology.orchestra.run.smithi187.stdout:4.3 2024-09-21T14:00:59.488 INFO:teuthology.orchestra.run.smithi187.stdout:5.b 2024-09-21T14:00:59.488 INFO:teuthology.orchestra.run.smithi187.stdout:1.a 2024-09-21T14:00:59.489 INFO:teuthology.orchestra.run.smithi187.stdout:3.1d 2024-09-21T14:00:59.489 INFO:teuthology.orchestra.run.smithi187.stdout:2.2 2024-09-21T14:00:59.489 INFO:teuthology.orchestra.run.smithi187.stdout:5.19 2024-09-21T14:00:59.489 INFO:teuthology.orchestra.run.smithi187.stdout:4.8 2024-09-21T14:00:59.489 INFO:teuthology.orchestra.run.smithi187.stdout:6.1e 2024-09-21T14:00:59.489 INFO:teuthology.orchestra.run.smithi187.stdout:6.4 2024-09-21T14:00:59.489 INFO:teuthology.orchestra.run.smithi187.stdout:9.9 2024-09-21T14:00:59.489 INFO:teuthology.orchestra.run.smithi187.stdout:3.c 2024-09-21T14:00:59.489 INFO:teuthology.orchestra.run.smithi187.stdout:3.2 2024-09-21T14:00:59.489 INFO:teuthology.orchestra.run.smithi187.stdout:4.1f 2024-09-21T14:00:59.489 INFO:teuthology.orchestra.run.smithi187.stdout:3.10 2024-09-21T14:00:59.489 INFO:teuthology.orchestra.run.smithi187.stdout:4.15 2024-09-21T14:00:59.489 INFO:teuthology.orchestra.run.smithi187.stdout:5.1b 2024-09-21T14:00:59.489 INFO:teuthology.orchestra.run.smithi187.stdout:5.f 2024-09-21T14:00:59.489 INFO:teuthology.orchestra.run.smithi187.stdout:3.16 2024-09-21T14:00:59.490 INFO:teuthology.orchestra.run.smithi187.stdout:1.4 2024-09-21T14:00:59.490 INFO:teuthology.orchestra.run.smithi187.stdout:4.1 2024-09-21T14:00:59.490 INFO:teuthology.orchestra.run.smithi187.stdout:6.a 2024-09-21T14:00:59.490 INFO:teuthology.orchestra.run.smithi187.stdout:6.1d 2024-09-21T14:00:59.490 INFO:teuthology.orchestra.run.smithi187.stdout:3.1c 2024-09-21T14:00:59.490 INFO:teuthology.orchestra.run.smithi187.stdout:1.1 2024-09-21T14:00:59.490 INFO:teuthology.orchestra.run.smithi187.stdout:6.19 2024-09-21T14:00:59.490 INFO:teuthology.orchestra.run.smithi187.stdout:3.1f 2024-09-21T14:00:59.490 INFO:teuthology.orchestra.run.smithi187.stdout:5.3 2024-09-21T14:00:59.490 INFO:teuthology.orchestra.run.smithi187.stdout:5.2 2024-09-21T14:00:59.490 INFO:teuthology.orchestra.run.smithi187.stdout:1.8 2024-09-21T14:00:59.490 INFO:teuthology.orchestra.run.smithi187.stdout:5.d 2024-09-21T14:00:59.490 INFO:teuthology.orchestra.run.smithi187.stdout:5.e 2024-09-21T14:00:59.490 INFO:teuthology.orchestra.run.smithi187.stdout:3.5 2024-09-21T14:00:59.490 INFO:teuthology.orchestra.run.smithi187.stdout:4.2 2024-09-21T14:00:59.490 INFO:teuthology.orchestra.run.smithi187.stdout:6.10 2024-09-21T14:00:59.491 INFO:teuthology.orchestra.run.smithi187.stdout:6.11 2024-09-21T14:00:59.491 INFO:teuthology.orchestra.run.smithi187.stdout:6.17 2024-09-21T14:00:59.491 INFO:teuthology.orchestra.run.smithi187.stdout:6.1a 2024-09-21T14:00:59.491 INFO:teuthology.orchestra.run.smithi187.stdout:6.c 2024-09-21T14:00:59.491 INFO:teuthology.orchestra.run.smithi187.stdout:6.1f 2024-09-21T14:00:59.491 INFO:teuthology.orchestra.run.smithi187.stdout:5.12 2024-09-21T14:00:59.491 INFO:teuthology.orchestra.run.smithi187.stdout:6.1c 2024-09-21T14:00:59.491 INFO:teuthology.orchestra.run.smithi187.stdout:6.7 2024-09-21T14:00:59.491 INFO:teuthology.orchestra.run.smithi187.stdout:6.d 2024-09-21T14:00:59.491 INFO:teuthology.orchestra.run.smithi187.stdout:5.16 2024-09-21T14:00:59.491 INFO:teuthology.orchestra.run.smithi187.stdout:3.4 2024-09-21T14:00:59.491 INFO:teuthology.orchestra.run.smithi187.stdout:9.5 2024-09-21T14:00:59.491 INFO:teuthology.orchestra.run.smithi187.stdout:9.6 2024-09-21T14:00:59.491 INFO:teuthology.orchestra.run.smithi187.stdout:9.7 2024-09-21T14:00:59.491 INFO:teuthology.orchestra.run.smithi187.stdout:9.8 2024-09-21T14:00:59.491 INFO:teuthology.orchestra.run.smithi187.stdout:3.17 2024-09-21T14:00:59.491 INFO:teuthology.orchestra.run.smithi187.stdout:9.a 2024-09-21T14:00:59.492 INFO:teuthology.orchestra.run.smithi187.stdout:9.b 2024-09-21T14:00:59.505 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:59.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:59.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:00:59.607 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:00:59.629 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:00:59.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:59.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:00:59.731 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:00:59.754 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:59.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:59.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:00:59.856 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:00:59.880 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:00:59.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:00:59.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:00:59.944 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:00:59 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:00:59.982 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:01:00.004 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:00.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:00.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:01:00.105 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:00.128 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:00.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:00.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:00.230 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:00.255 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:00.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:00.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:00.357 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:01:00.380 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:00.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:00.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:01:00.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:00 smithi187 ceph-mon[33032]: osdmap e329: 8 total, 7 up, 8 in 2024-09-21T14:01:00.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:00 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:01:00.481 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:01:00.499 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell -n osd.5 --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --op export --pgid 1.a --file /var/log/ceph/exp.1.a.5 2024-09-21T14:01:00.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:00 smithi049 ceph-mon[40489]: osdmap e329: 8 total, 7 up, 8 in 2024-09-21T14:01:00.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:00 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:01:00.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:00 smithi049 ceph-mon[30080]: osdmap e329: 8 total, 7 up, 8 in 2024-09-21T14:01:00.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:00 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:01:00.505 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:00.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:00.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:01:00.606 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:00.635 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:00.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:00.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:00.737 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:00.763 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:00.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:00.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:00.772 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/osd.5/config 2024-09-21T14:01:00.865 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:01:00.889 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:00.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:00.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:01:00.991 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:01.004 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:01:00 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:01:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:01:01.017 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:01.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:01.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:01.118 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:01.142 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:01.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:01.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:01.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:01 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:01:01.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:01 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:01:01.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:01 smithi187 ceph-mon[33032]: pgmap v604: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+clean+remapped, 1 undersized+degraded+remapped+backfill_toofull+peered, 1 active+remapped+backfilling, 3 active+recovering+undersized+degraded, 35 active+undersized, 21 active+undersized+degraded, 95 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 727 B/s rd, 0 B/s wr, 116 op/s; 48194/166791 objects degraded (28.895%); 1805/166791 objects misplaced (1.082%); 41 MiB/s, 647 objects/s recovering 2024-09-21T14:01:01.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:01 smithi187 ceph-mon[33032]: osdmap e330: 8 total, 7 up, 8 in 2024-09-21T14:01:01.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:01 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 48194/166791 objects degraded (28.895%), 26 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T14:01:01.244 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:01.273 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:01.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:01.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:01.375 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:01.396 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:01.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:01.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:01.492 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:01 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:01:01.492 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:01 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:01:01.492 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:01 smithi049 ceph-mon[30080]: pgmap v604: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+clean+remapped, 1 undersized+degraded+remapped+backfill_toofull+peered, 1 active+remapped+backfilling, 3 active+recovering+undersized+degraded, 35 active+undersized, 21 active+undersized+degraded, 95 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 727 B/s rd, 0 B/s wr, 116 op/s; 48194/166791 objects degraded (28.895%); 1805/166791 objects misplaced (1.082%); 41 MiB/s, 647 objects/s recovering 2024-09-21T14:01:01.493 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:01 smithi049 ceph-mon[30080]: osdmap e330: 8 total, 7 up, 8 in 2024-09-21T14:01:01.493 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:01 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 48194/166791 objects degraded (28.895%), 26 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T14:01:01.493 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:01 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:01:01.493 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:01 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:01:01.493 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:01 smithi049 ceph-mon[40489]: pgmap v604: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+clean+remapped, 1 undersized+degraded+remapped+backfill_toofull+peered, 1 active+remapped+backfilling, 3 active+recovering+undersized+degraded, 35 active+undersized, 21 active+undersized+degraded, 95 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 727 B/s rd, 0 B/s wr, 116 op/s; 48194/166791 objects degraded (28.895%); 1805/166791 objects misplaced (1.082%); 41 MiB/s, 647 objects/s recovering 2024-09-21T14:01:01.493 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:01 smithi049 ceph-mon[40489]: osdmap e330: 8 total, 7 up, 8 in 2024-09-21T14:01:01.493 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:01 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 48194/166791 objects degraded (28.895%), 26 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T14:01:01.499 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:01:01.523 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:01.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:01.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:01:01.624 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:01.651 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:01.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:01.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:01.752 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:01.778 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:01.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:01.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:01.880 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:01:01.902 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:01.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:01.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:01:02.004 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:01:02.027 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:02.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:02.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:01:02.130 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:02.140 INFO:teuthology.orchestra.run.smithi187.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-21T14:01:02.160 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:02.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:02.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:02.262 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:01:02.286 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:02.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:02.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:01:02.387 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:01:02.412 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:02.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:02.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:01:02.513 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:01:02.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:02.539 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:02.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:01:02.640 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:02.666 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:02.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:02.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:02.767 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:01:02.791 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:02.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:02.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:01:02.893 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:01:02.916 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:02.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:02.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:01:03.017 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:01:03.045 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:03.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:03.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:01:03.098 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:03 smithi187 ceph-mon[33032]: pgmap v606: 158 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 1 undersized+degraded+remapped+backfill_toofull+peered, 4 active+recovering+undersized+degraded, 35 active+undersized, 20 active+undersized+degraded, 96 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 68 op/s; 46230/166392 objects degraded (27.784%); 1102/166392 objects misplaced (0.662%); 57 MiB/s, 903 objects/s recovering 2024-09-21T14:01:03.098 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:03 smithi187 ceph-mon[33032]: 4.a scrub starts 2024-09-21T14:01:03.098 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:03 smithi187 ceph-mon[33032]: 4.a scrub ok 2024-09-21T14:01:03.147 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:03.173 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:03.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:03.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:03.275 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:03.301 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:03.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:03.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:03.402 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:03.430 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:03.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:03.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:03.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:03 smithi049 ceph-mon[40489]: pgmap v606: 158 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 1 undersized+degraded+remapped+backfill_toofull+peered, 4 active+recovering+undersized+degraded, 35 active+undersized, 20 active+undersized+degraded, 96 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 68 op/s; 46230/166392 objects degraded (27.784%); 1102/166392 objects misplaced (0.662%); 57 MiB/s, 903 objects/s recovering 2024-09-21T14:01:03.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:03 smithi049 ceph-mon[40489]: 4.a scrub starts 2024-09-21T14:01:03.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:03 smithi049 ceph-mon[40489]: 4.a scrub ok 2024-09-21T14:01:03.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:03 smithi049 ceph-mon[30080]: pgmap v606: 158 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 1 undersized+degraded+remapped+backfill_toofull+peered, 4 active+recovering+undersized+degraded, 35 active+undersized, 20 active+undersized+degraded, 96 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 68 op/s; 46230/166392 objects degraded (27.784%); 1102/166392 objects misplaced (0.662%); 57 MiB/s, 903 objects/s recovering 2024-09-21T14:01:03.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:03 smithi049 ceph-mon[30080]: 4.a scrub starts 2024-09-21T14:01:03.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:03 smithi049 ceph-mon[30080]: 4.a scrub ok 2024-09-21T14:01:03.531 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:03.558 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:03.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:03.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:03.659 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:03.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:03.684 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:03.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:03.785 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:01:03.808 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:03.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:03.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:01:03.909 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:01:03.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:03.933 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:03.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:01:04.034 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:04.060 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:04.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:04.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:04.150 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T14:01:04.162 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:01:04.217 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:04.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:04.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:01:04.319 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:01:04.342 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:04.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:04.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:01:04.443 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:01:04.469 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:04.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:04.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:01:04.571 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:01:04.594 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:04.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:04.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:01:04.696 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:04.720 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:04.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:04.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:04.822 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:04.850 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:04.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:04.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:04.952 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:01:04.976 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:04.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:04.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:01:05.078 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:05.102 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:05.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:05.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:05.204 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:05.226 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:05.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:05.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:05.328 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:05.355 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:05.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:05.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:05.381 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:05 smithi049 ceph-mon[40489]: 4.d scrub starts 2024-09-21T14:01:05.381 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:05 smithi049 ceph-mon[40489]: 4.d scrub ok 2024-09-21T14:01:05.382 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:05 smithi049 ceph-mon[40489]: pgmap v607: 158 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 1 undersized+degraded+remapped+backfill_toofull+peered, 4 active+recovering+undersized+degraded, 35 active+undersized, 20 active+undersized+degraded, 96 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 28 op/s; 46221/166365 objects degraded (27.783%); 1102/166365 objects misplaced (0.662%); 48 MiB/s, 773 objects/s recovering 2024-09-21T14:01:05.382 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:05 smithi049 ceph-mon[30080]: 4.d scrub starts 2024-09-21T14:01:05.382 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:05 smithi049 ceph-mon[30080]: 4.d scrub ok 2024-09-21T14:01:05.382 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:05 smithi049 ceph-mon[30080]: pgmap v607: 158 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 1 undersized+degraded+remapped+backfill_toofull+peered, 4 active+recovering+undersized+degraded, 35 active+undersized, 20 active+undersized+degraded, 96 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 28 op/s; 46221/166365 objects degraded (27.783%); 1102/166365 objects misplaced (0.662%); 48 MiB/s, 773 objects/s recovering 2024-09-21T14:01:05.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:05 smithi187 ceph-mon[33032]: 4.d scrub starts 2024-09-21T14:01:05.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:05 smithi187 ceph-mon[33032]: 4.d scrub ok 2024-09-21T14:01:05.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:05 smithi187 ceph-mon[33032]: pgmap v607: 158 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 1 undersized+degraded+remapped+backfill_toofull+peered, 4 active+recovering+undersized+degraded, 35 active+undersized, 20 active+undersized+degraded, 96 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 28 op/s; 46221/166365 objects degraded (27.783%); 1102/166365 objects misplaced (0.662%); 48 MiB/s, 773 objects/s recovering 2024-09-21T14:01:05.457 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:01:05.481 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:05.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:05.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:01:05.583 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:05.610 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:05.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:05.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:05.712 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:05.736 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:05.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:05.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:05.837 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:05.861 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:05.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:05.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:05.963 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:05.989 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:05.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:05.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:06.091 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:01:06.116 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:06.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:06.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:01:06.188 INFO:teuthology.orchestra.run.smithi187.stderr:Exporting 1.a info 1.a( v 75'79 (0'0,75'79] local-lis/les=320/321 n=0 ec=318/20 lis/c=320/320 les/c/f=321/321/0 sis=320) 2024-09-21T14:01:06.188 INFO:teuthology.orchestra.run.smithi187.stderr:Export successful 2024-09-21T14:01:06.218 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:06.240 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:06.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:06.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:06.332 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:06 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 46221/166365 objects degraded (27.783%), 25 pgs degraded (PG_DEGRADED) 2024-09-21T14:01:06.332 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:06 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/68439718' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:01:06.333 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:06 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 46221/166365 objects degraded (27.783%), 25 pgs degraded (PG_DEGRADED) 2024-09-21T14:01:06.333 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:06 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/68439718' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:01:06.333 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-09-21T14:01:06.343 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:06.369 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:06.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:06.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:06.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:06 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 46221/166365 objects degraded (27.783%), 25 pgs degraded (PG_DEGRADED) 2024-09-21T14:01:06.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:06 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/68439718' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:01:06.471 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:06.493 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:06.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:06.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:06.595 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:06.621 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:06.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:06.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:06.723 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:06.748 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:06.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:06.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:06.849 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:06.874 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:06.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:06.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:06.976 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:07.003 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:07.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:07.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:07.105 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:01:07.127 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:07.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:07.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:01:07.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:07 smithi187 ceph-mon[33032]: pgmap v608: 158 pgs: 1 active+clean+remapped, 1 undersized+degraded+remapped+backfill_toofull+peered, 1 active+remapped+backfilling, 4 active+recovering+undersized+degraded, 35 active+undersized, 20 active+undersized+degraded, 96 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 26 op/s; 45654/166365 objects degraded (27.442%); 1102/166365 objects misplaced (0.662%); 24 MiB/s, 376 objects/s recovering 2024-09-21T14:01:07.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:07 smithi187 ceph-mon[33032]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:01:07.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:07 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/68439718' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T14:01:07.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:07 smithi187 ceph-mon[33032]: osdmap e331: 8 total, 7 up, 8 in 2024-09-21T14:01:07.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:07 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:01:07.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:07 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:01:07.229 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:01:07.251 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:07.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:07.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:01:07.262 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell -n osd.5 --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --force --op remove --pgid 1.a 2024-09-21T14:01:07.353 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:07.380 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:07.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:07.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:07.482 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:01:07.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:07 smithi049 ceph-mon[30080]: pgmap v608: 158 pgs: 1 active+clean+remapped, 1 undersized+degraded+remapped+backfill_toofull+peered, 1 active+remapped+backfilling, 4 active+recovering+undersized+degraded, 35 active+undersized, 20 active+undersized+degraded, 96 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 26 op/s; 45654/166365 objects degraded (27.442%); 1102/166365 objects misplaced (0.662%); 24 MiB/s, 376 objects/s recovering 2024-09-21T14:01:07.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:07 smithi049 ceph-mon[30080]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:01:07.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:07 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/68439718' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T14:01:07.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:07 smithi049 ceph-mon[30080]: osdmap e331: 8 total, 7 up, 8 in 2024-09-21T14:01:07.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:07 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:01:07.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:07 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:01:07.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:07 smithi049 ceph-mon[40489]: pgmap v608: 158 pgs: 1 active+clean+remapped, 1 undersized+degraded+remapped+backfill_toofull+peered, 1 active+remapped+backfilling, 4 active+recovering+undersized+degraded, 35 active+undersized, 20 active+undersized+degraded, 96 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 26 op/s; 45654/166365 objects degraded (27.442%); 1102/166365 objects misplaced (0.662%); 24 MiB/s, 376 objects/s recovering 2024-09-21T14:01:07.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:07 smithi049 ceph-mon[40489]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:01:07.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:07 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/68439718' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T14:01:07.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:07 smithi049 ceph-mon[40489]: osdmap e331: 8 total, 7 up, 8 in 2024-09-21T14:01:07.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:07 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:01:07.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:07 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:01:07.506 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:07.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:07.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:01:07.525 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/osd.5/config 2024-09-21T14:01:07.607 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:07.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:07.635 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:07.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:07.736 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:01:07.759 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:07.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:07.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:01:07.861 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:07.886 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:07.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:07.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:07.988 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:01:08.011 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:08.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:08.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:01:08.112 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:08.139 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:08.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:08.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:08.241 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:01:08.263 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:08.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:08.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:01:08.367 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:08.383 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:08 smithi187 ceph-mon[33032]: osdmap e332: 8 total, 7 up, 8 in 2024-09-21T14:01:08.393 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:08.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:08.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:08.495 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:08.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:08 smithi049 ceph-mon[40489]: osdmap e332: 8 total, 7 up, 8 in 2024-09-21T14:01:08.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:08 smithi049 ceph-mon[30080]: osdmap e332: 8 total, 7 up, 8 in 2024-09-21T14:01:08.519 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:08.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:08.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:08.621 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:01:08.645 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:08.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:08.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:01:08.747 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:08.770 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:08.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:08.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:08.872 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:01:08.895 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:08.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:08.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:01:08.997 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:09.019 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:09.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:09.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:09.121 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:01:09.144 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:09.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:09.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:01:09.187 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:09 smithi187 ceph-mon[33032]: pgmap v611: 158 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 1 undersized+degraded+remapped+backfill_toofull+peered, 4 active+recovering+undersized+degraded, 35 active+undersized, 20 active+undersized+degraded, 96 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 19 op/s; 45183/166365 objects degraded (27.159%); 131/166365 objects misplaced (0.079%); 31 MiB/s, 503 objects/s recovering 2024-09-21T14:01:09.240 INFO:teuthology.orchestra.run.smithi187.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-21T14:01:09.246 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:09.273 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:09.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:09.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:09.375 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:01:09.399 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:09.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:09.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:01:09.501 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:01:09.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:09 smithi049 ceph-mon[40489]: pgmap v611: 158 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 1 undersized+degraded+remapped+backfill_toofull+peered, 4 active+recovering+undersized+degraded, 35 active+undersized, 20 active+undersized+degraded, 96 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 19 op/s; 45183/166365 objects degraded (27.159%); 131/166365 objects misplaced (0.079%); 31 MiB/s, 503 objects/s recovering 2024-09-21T14:01:09.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:09 smithi049 ceph-mon[30080]: pgmap v611: 158 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 1 undersized+degraded+remapped+backfill_toofull+peered, 4 active+recovering+undersized+degraded, 35 active+undersized, 20 active+undersized+degraded, 96 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 19 op/s; 45183/166365 objects degraded (27.159%); 131/166365 objects misplaced (0.079%); 31 MiB/s, 503 objects/s recovering 2024-09-21T14:01:09.523 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:09.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:09.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:01:09.625 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:01:09.648 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:09.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:09.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:01:09.749 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:09.775 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:09.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:09.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:09.799 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:01:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:01:09.877 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:09.900 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:09.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:09.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:10.002 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:10.024 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:10.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:10.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:10.125 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:10.151 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:10.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:10.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:10.253 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:01:10.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:10.279 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:10.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:01:10.380 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:01:10.403 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:10.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:10.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:01:10.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:10 smithi187 ceph-mon[33032]: osdmap e333: 8 total, 7 up, 8 in 2024-09-21T14:01:10.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:10 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:01:10.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:10 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 45183/166365 objects degraded (27.159%), 25 pgs degraded (PG_DEGRADED) 2024-09-21T14:01:10.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:10 smithi187 ceph-mon[33032]: osdmap e334: 8 total, 7 up, 8 in 2024-09-21T14:01:10.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:10 smithi049 ceph-mon[40489]: osdmap e333: 8 total, 7 up, 8 in 2024-09-21T14:01:10.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:10 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:01:10.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:10 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 45183/166365 objects degraded (27.159%), 25 pgs degraded (PG_DEGRADED) 2024-09-21T14:01:10.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:10 smithi049 ceph-mon[40489]: osdmap e334: 8 total, 7 up, 8 in 2024-09-21T14:01:10.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:10 smithi049 ceph-mon[30080]: osdmap e333: 8 total, 7 up, 8 in 2024-09-21T14:01:10.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:10 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:01:10.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:10 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 45183/166365 objects degraded (27.159%), 25 pgs degraded (PG_DEGRADED) 2024-09-21T14:01:10.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:10 smithi049 ceph-mon[30080]: osdmap e334: 8 total, 7 up, 8 in 2024-09-21T14:01:10.505 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:10.527 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:10.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:10.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:10.628 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:10.654 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:10.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:10.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:10.756 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:01:10.778 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:10.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:10.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:01:10.880 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:10.902 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:10.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:10.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:11.003 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:01:11.004 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:01:10 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:01:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:01:11.025 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:11.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:11.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:01:11.127 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:11.149 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:11.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:11.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:11.251 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:11.275 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:11.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:11.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:11.376 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:11.399 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:11.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:11.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:11.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:11 smithi187 ceph-mon[33032]: pgmap v613: 158 pgs: 1 active+clean+remapped, 1 undersized+degraded+remapped+backfill_toofull+peered, 1 active+remapped+backfilling, 4 active+recovering+undersized+degraded, 35 active+undersized, 20 active+undersized+degraded, 96 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 44332/166365 objects degraded (26.647%); 131/166365 objects misplaced (0.079%); 30 MiB/s, 476 objects/s recovering 2024-09-21T14:01:11.501 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:11.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:11 smithi049 ceph-mon[40489]: pgmap v613: 158 pgs: 1 active+clean+remapped, 1 undersized+degraded+remapped+backfill_toofull+peered, 1 active+remapped+backfilling, 4 active+recovering+undersized+degraded, 35 active+undersized, 20 active+undersized+degraded, 96 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 44332/166365 objects degraded (26.647%); 131/166365 objects misplaced (0.079%); 30 MiB/s, 476 objects/s recovering 2024-09-21T14:01:11.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:11 smithi049 ceph-mon[30080]: pgmap v613: 158 pgs: 1 active+clean+remapped, 1 undersized+degraded+remapped+backfill_toofull+peered, 1 active+remapped+backfilling, 4 active+recovering+undersized+degraded, 35 active+undersized, 20 active+undersized+degraded, 96 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 44332/166365 objects degraded (26.647%); 131/166365 objects misplaced (0.079%); 30 MiB/s, 476 objects/s recovering 2024-09-21T14:01:11.523 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:11.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:11.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:11.625 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:11.650 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:11.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:11.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:11.751 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:11.776 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:11.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:11.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:11.877 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:11.904 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:11.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:11.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:12.006 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:12.030 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:12.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:12.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:12.132 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:12.157 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:12.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:12.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:12.259 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:12.284 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:12.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:12.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:12.386 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:12.408 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:12.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:12.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:12.510 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:01:12.533 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:12.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:12.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:01:12.634 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:01:12.656 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:12.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:12.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:01:12.758 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:01:12.781 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:12.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:12.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:01:12.883 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:01:12.905 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:12.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:12.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:01:13.007 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:13.032 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:13.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:13.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:13.134 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:13.159 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:13.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:13.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:13.260 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:13.283 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:13.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:13.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:13.364 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:13 smithi049 ceph-mon[40489]: pgmap v615: 158 pgs: 1 active+clean+remapped, 1 undersized+degraded+remapped+backfill_toofull+peered, 4 active+recovering+undersized+degraded, 35 active+undersized, 20 active+undersized+degraded, 97 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 868 B/s rd, 0 op/s; 43819/166365 objects degraded (26.339%); 16 MiB/s, 253 objects/s recovering 2024-09-21T14:01:13.364 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:13 smithi049 ceph-mon[30080]: pgmap v615: 158 pgs: 1 active+clean+remapped, 1 undersized+degraded+remapped+backfill_toofull+peered, 4 active+recovering+undersized+degraded, 35 active+undersized, 20 active+undersized+degraded, 97 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 868 B/s rd, 0 op/s; 43819/166365 objects degraded (26.339%); 16 MiB/s, 253 objects/s recovering 2024-09-21T14:01:13.385 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:13.409 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:13.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:13.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:13.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:13 smithi187 ceph-mon[33032]: pgmap v615: 158 pgs: 1 active+clean+remapped, 1 undersized+degraded+remapped+backfill_toofull+peered, 4 active+recovering+undersized+degraded, 35 active+undersized, 20 active+undersized+degraded, 97 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 868 B/s rd, 0 op/s; 43819/166365 objects degraded (26.339%); 16 MiB/s, 253 objects/s recovering 2024-09-21T14:01:13.511 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:13.536 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:13.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:13.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:13.638 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:13.665 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:13.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:13.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:13.766 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:13.799 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:13.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:13.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:13.882 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:01:13.883 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:01:13.902 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:13.906 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:01:13.907 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:01:13.914 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:01:13.931 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:13.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:13.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:13.934 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:01:13.934 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:01:13.952 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:01:13.953 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:01:13.971 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:01:13.971 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:01:13.971 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-21T14:01:13.991 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:01:13.991 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:01:14.012 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:01:14.012 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:01:14.032 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:14.060 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:14.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:14.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:14.162 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:14.187 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:14.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:14.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:14.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:14 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3093420999' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:01:14.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:14 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:01:14.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:14 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:01:14.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:14 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3093420999' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:01:14.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:14 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:01:14.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:14 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:01:14.289 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:14.314 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:14.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:14.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:14.416 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:01:14.441 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:14.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:14.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:01:14.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:14 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3093420999' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:01:14.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:14 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:01:14.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:14 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:01:14.543 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:01:14.565 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:14.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:14.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:01:14.667 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:14.690 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T14:01:14.691 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:01:14.696 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:14.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:14.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:14.798 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:14.824 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:14.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:14.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:14.925 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:01:14.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:14.949 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:14.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:01:15.050 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:15.075 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:15.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:15.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:15.177 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:01:15.200 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:15.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:15.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:01:15.302 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:15.325 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:15.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:15.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:15.366 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T14:01:15.369 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:15 smithi049 ceph-mon[30080]: pgmap v616: 158 pgs: 1 active+clean+remapped, 1 undersized+degraded+remapped+backfill_toofull+peered, 4 active+recovering+undersized+degraded, 35 active+undersized, 20 active+undersized+degraded, 97 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 742 B/s rd, 0 op/s; 43819/166365 objects degraded (26.339%); 14 MiB/s, 216 objects/s recovering 2024-09-21T14:01:15.369 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:15 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 43819/166365 objects degraded (26.339%), 25 pgs degraded (PG_DEGRADED) 2024-09-21T14:01:15.370 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:15 smithi049 ceph-mon[40489]: pgmap v616: 158 pgs: 1 active+clean+remapped, 1 undersized+degraded+remapped+backfill_toofull+peered, 4 active+recovering+undersized+degraded, 35 active+undersized, 20 active+undersized+degraded, 97 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 742 B/s rd, 0 op/s; 43819/166365 objects degraded (26.339%); 14 MiB/s, 216 objects/s recovering 2024-09-21T14:01:15.370 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:15 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 43819/166365 objects degraded (26.339%), 25 pgs degraded (PG_DEGRADED) 2024-09-21T14:01:15.405 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T14:01:15.427 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:15.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:15 smithi187 ceph-mon[33032]: pgmap v616: 158 pgs: 1 active+clean+remapped, 1 undersized+degraded+remapped+backfill_toofull+peered, 4 active+recovering+undersized+degraded, 35 active+undersized, 20 active+undersized+degraded, 97 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 742 B/s rd, 0 op/s; 43819/166365 objects degraded (26.339%); 14 MiB/s, 216 objects/s recovering 2024-09-21T14:01:15.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:15 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 43819/166365 objects degraded (26.339%), 25 pgs degraded (PG_DEGRADED) 2024-09-21T14:01:15.460 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-21T14:01:15.455+0000 7efdba353580 -1 bluestore(/var/lib/ceph/osd/ceph-5) _main_bdev_label_try_reserve bdev label location 0x280000000 occupied by BlueStore object or BlueFS file, disabling 2024-09-21T14:01:15.468 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:15.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:15.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:15.481 INFO:teuthology.orchestra.run.smithi187.stdout: marking collection for removal 2024-09-21T14:01:15.482 INFO:teuthology.orchestra.run.smithi187.stdout:setting '_remove' omap key 2024-09-21T14:01:15.482 INFO:teuthology.orchestra.run.smithi187.stdout:finish_remove_pgs 1.a_head removing 1.a 2024-09-21T14:01:15.487 INFO:teuthology.orchestra.run.smithi187.stdout:Remove successful 2024-09-21T14:01:15.570 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:01:15.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:15.595 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:15.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:01:15.696 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:15.722 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:15.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:15.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:15.823 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:15.848 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:15.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:15.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:15.853 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T14:01:15.854 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T14:01:15.865 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:01:15.885 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:01:15.949 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:15.973 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:15.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:15.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:16.074 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:01:16.097 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:16.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:16.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:01:16.199 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:01:16.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:16.224 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:16.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:01:16.325 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:16.352 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:16.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:16.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:16.453 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:16.478 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:16.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:16.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:16.580 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:16.605 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:16.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:16.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:16.632 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T14:01:16.632 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:01:16.707 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:16.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:16.733 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:16.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:16.772 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell -n osd.5 --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.1.a.5 2024-09-21T14:01:16.834 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:01:16.858 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:16.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:16.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:01:16.960 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:16.990 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:16.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:16.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:17.067 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/osd.5/config 2024-09-21T14:01:17.092 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:17.122 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:17.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:17.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:17.224 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:01:17.247 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:17.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:17.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:01:17.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:17 smithi049 ceph-mon[40489]: pgmap v617: 158 pgs: 1 active+clean+remapped, 1 undersized+degraded+remapped+backfill_toofull+peered, 4 active+recovering+undersized+degraded, 35 active+undersized, 20 active+undersized+degraded, 97 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 42961/166365 objects degraded (25.823%); 18 MiB/s, 293 objects/s recovering 2024-09-21T14:01:17.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:17 smithi049 ceph-mon[30080]: pgmap v617: 158 pgs: 1 active+clean+remapped, 1 undersized+degraded+remapped+backfill_toofull+peered, 4 active+recovering+undersized+degraded, 35 active+undersized, 20 active+undersized+degraded, 97 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 42961/166365 objects degraded (25.823%); 18 MiB/s, 293 objects/s recovering 2024-09-21T14:01:17.348 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:17.370 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:17 smithi187 ceph-mon[33032]: pgmap v617: 158 pgs: 1 active+clean+remapped, 1 undersized+degraded+remapped+backfill_toofull+peered, 4 active+recovering+undersized+degraded, 35 active+undersized, 20 active+undersized+degraded, 97 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 42961/166365 objects degraded (25.823%); 18 MiB/s, 293 objects/s recovering 2024-09-21T14:01:17.377 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:17.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:17.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:17.481 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:17.504 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:17.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:17.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:17.575 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T14:01:17.606 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:01:17.641 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:17.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:17.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:01:17.742 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:17.768 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:17.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:17.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:17.842 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:01:17.854 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T14:01:17.870 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:17.921 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:17.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:17.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:18.023 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:01:18.046 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:18.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:18.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:01:18.111 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-09-21T14:01:18.149 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:18.172 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:18.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:18.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:18.176 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:18 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1387845019' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:01:18.176 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:18 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:01:18.176 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:18 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1387845019' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:01:18.176 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:18 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:01:18.274 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:18.285 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:18 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1387845019' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:01:18.285 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:18 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:01:18.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:18.302 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:18.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:18.389 INFO:teuthology.orchestra.run.smithi187.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-21T14:01:18.403 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:18.427 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:18.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:18.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:18.528 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:18.558 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:18.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:18.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:18.660 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:01:18.683 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:18.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:18.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:01:18.785 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:01:18.809 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:18.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:18.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:01:18.911 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:01:18.934 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:18.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:18.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:01:19.036 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:19.065 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:19.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:19.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:19.167 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:01:19.190 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:19.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:19.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:01:19.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:19 smithi187 ceph-mon[33032]: pgmap v618: 158 pgs: 1 active+clean+remapped, 1 undersized+degraded+remapped+backfill_toofull+peered, 4 active+recovering+undersized+degraded, 35 active+undersized, 20 active+undersized+degraded, 97 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 42389/166365 objects degraded (25.480%); 21 MiB/s, 329 objects/s recovering 2024-09-21T14:01:19.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:19 smithi187 ceph-mon[33032]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:01:19.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:19 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T14:01:19.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:19 smithi187 ceph-mon[33032]: osdmap e335: 8 total, 7 up, 8 in 2024-09-21T14:01:19.291 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:19.318 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:19.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:19.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:19.419 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:19.442 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:19.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:19.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:19.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:19 smithi049 ceph-mon[40489]: pgmap v618: 158 pgs: 1 active+clean+remapped, 1 undersized+degraded+remapped+backfill_toofull+peered, 4 active+recovering+undersized+degraded, 35 active+undersized, 20 active+undersized+degraded, 97 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 42389/166365 objects degraded (25.480%); 21 MiB/s, 329 objects/s recovering 2024-09-21T14:01:19.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:19 smithi049 ceph-mon[40489]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:01:19.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:19 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T14:01:19.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:19 smithi049 ceph-mon[40489]: osdmap e335: 8 total, 7 up, 8 in 2024-09-21T14:01:19.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:19 smithi049 ceph-mon[30080]: pgmap v618: 158 pgs: 1 active+clean+remapped, 1 undersized+degraded+remapped+backfill_toofull+peered, 4 active+recovering+undersized+degraded, 35 active+undersized, 20 active+undersized+degraded, 97 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 42389/166365 objects degraded (25.480%); 21 MiB/s, 329 objects/s recovering 2024-09-21T14:01:19.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:19 smithi049 ceph-mon[30080]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:01:19.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:19 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T14:01:19.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:19 smithi049 ceph-mon[30080]: osdmap e335: 8 total, 7 up, 8 in 2024-09-21T14:01:19.544 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:19.575 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:19.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:19.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:19.677 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:19.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:19.709 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:19.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:19.731 INFO:teuthology.orchestra.run.smithi187.stderr:2024-09-21T14:01:19.729+0000 7f29b64ef580 -1 bluestore(/var/lib/ceph/osd/ceph-5) _main_bdev_label_try_reserve bdev label location 0x280000000 occupied by BlueStore object or BlueFS file, disabling 2024-09-21T14:01:19.736 INFO:teuthology.orchestra.run.smithi187.stdout:get_pg_num_history pg_num_history pg_num_history(e324 pg_nums {1={20=1,318=11},2={59=3},3={60=32},4={62=32},5={64=32},6={66=32},8={183=16},9={260=16}} deleted_pools 256,8) 2024-09-21T14:01:19.737 INFO:teuthology.orchestra.run.smithi187.stdout:Importing pgid 1.a 2024-09-21T14:01:19.738 INFO:teuthology.orchestra.run.smithi187.stdout:write_pg epoch 321 info 1.a( v 75'79 (0'0,75'79] local-lis/les=320/321 n=0 ec=318/20 lis/c=320/320 les/c/f=321/321/0 sis=320) 2024-09-21T14:01:19.741 INFO:teuthology.orchestra.run.smithi187.stdout:Import successful 2024-09-21T14:01:19.809 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:19.833 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:19.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:19.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:19.935 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:19.944 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:01:19 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:01:19.959 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:19.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:19.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:20.061 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:20.086 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:20.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:20.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:20.187 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:20.212 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:20.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:20.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:20.258 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:20 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T14:01:20.259 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:20 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:01:20.259 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:20 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 42389/166365 objects degraded (25.480%), 25 pgs degraded (PG_DEGRADED) 2024-09-21T14:01:20.259 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:20 smithi187 ceph-mon[33032]: osdmap e336: 8 total, 7 up, 8 in 2024-09-21T14:01:20.314 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:20.338 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:20.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:20.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:20.362 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:20 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T14:01:20.362 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:20 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:01:20.362 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:20 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 42389/166365 objects degraded (25.480%), 25 pgs degraded (PG_DEGRADED) 2024-09-21T14:01:20.362 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:20 smithi049 ceph-mon[40489]: osdmap e336: 8 total, 7 up, 8 in 2024-09-21T14:01:20.362 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:20 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T14:01:20.362 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:20 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:01:20.362 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:20 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 42389/166365 objects degraded (25.480%), 25 pgs degraded (PG_DEGRADED) 2024-09-21T14:01:20.362 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:20 smithi049 ceph-mon[30080]: osdmap e336: 8 total, 7 up, 8 in 2024-09-21T14:01:20.440 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:01:20.462 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:20.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:20.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:01:20.564 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:20.590 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:20.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:20.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:20.692 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:20.719 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:20.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:20.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:20.754 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:01:20 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:01:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:01:20.820 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:01:20.844 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:20.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:20.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:01:20.946 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:01:20.968 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:20.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:20.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:01:20.970 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm -f /var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/exp.1.a.5 2024-09-21T14:01:21.070 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:01:21.092 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:21.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:21.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:01:21.121 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T14:01:21.121 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:01:21.193 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:21.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:21 smithi187 ceph-mon[33032]: pgmap v620: 158 pgs: 1 active+clean+remapped, 1 undersized+degraded+remapped+backfill_toofull+peered, 4 active+recovering+undersized+degraded, 35 active+undersized, 20 active+undersized+degraded, 97 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 41938/166365 objects degraded (25.208%); 16 MiB/s, 334 objects/s recovering 2024-09-21T14:01:21.218 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:21.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:21.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:21.320 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:21.346 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:21.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:21.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:21.448 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:21.472 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:21.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:21.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:21.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:21 smithi049 ceph-mon[40489]: pgmap v620: 158 pgs: 1 active+clean+remapped, 1 undersized+degraded+remapped+backfill_toofull+peered, 4 active+recovering+undersized+degraded, 35 active+undersized, 20 active+undersized+degraded, 97 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 41938/166365 objects degraded (25.208%); 16 MiB/s, 334 objects/s recovering 2024-09-21T14:01:21.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:21 smithi049 ceph-mon[30080]: pgmap v620: 158 pgs: 1 active+clean+remapped, 1 undersized+degraded+remapped+backfill_toofull+peered, 4 active+recovering+undersized+degraded, 35 active+undersized, 20 active+undersized+degraded, 97 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 41938/166365 objects degraded (25.208%); 16 MiB/s, 334 objects/s recovering 2024-09-21T14:01:21.574 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:21.596 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:21.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:21.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:21.698 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:21.722 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:21.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:21.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:21.824 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:21.849 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:21.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:21.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:21.951 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:01:21.973 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:21.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:21.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:01:22.075 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:22.097 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:22.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:22.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:22.199 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:22.221 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:22.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:22.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:22.323 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:22.346 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:22.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:22.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:22.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:22 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T14:01:22.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:22 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T14:01:22.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:22 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:01:22.448 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:01:22.470 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:22.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:22.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:01:22.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:22 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T14:01:22.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:22 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T14:01:22.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:22 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:01:22.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:22 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T14:01:22.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:22 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T14:01:22.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:22 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:01:22.571 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:22.594 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:22.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:22.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:22.601 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T14:01:22.690 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:01:22.696 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:01:22.721 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:22.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:22.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:01:22.823 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:01:22.847 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:22.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:22.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:01:22.948 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:01:22.972 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:22.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:22.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:01:23.074 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:01:23.096 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:23.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:23.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:01:23.198 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:23.220 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:23.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:23.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:23.303 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:01:23.303 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:01:23.303 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:01:23.303 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:01:23.312 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:01:23.312 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:01:23.322 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:01:23.334 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:01:23.334 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:01:23.346 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:01:23.347 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:23.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:23.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:01:23.353 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:01:23.353 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:01:23.353 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-21T14:01:23.372 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:01:23.372 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:01:23.389 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:01:23.390 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:01:23.448 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:23.472 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:23.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:23.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:23.574 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:23.596 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:23.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:23.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:23.698 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:23.721 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:23.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:23.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:23.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:23 smithi049 ceph-mon[40489]: pgmap v622: 158 pgs: 1 active+clean+remapped, 1 undersized+degraded+remapped+backfill_toofull+peered, 4 active+recovering+undersized+degraded, 35 active+undersized, 20 active+undersized+degraded, 97 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 41271/166365 objects degraded (24.808%); 20 MiB/s, 416 objects/s recovering 2024-09-21T14:01:23.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:23 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:01:23.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:23 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:01:23.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:23 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/97318538' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:01:23.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:23 smithi049 ceph-mon[30080]: pgmap v622: 158 pgs: 1 active+clean+remapped, 1 undersized+degraded+remapped+backfill_toofull+peered, 4 active+recovering+undersized+degraded, 35 active+undersized, 20 active+undersized+degraded, 97 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 41271/166365 objects degraded (24.808%); 20 MiB/s, 416 objects/s recovering 2024-09-21T14:01:23.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:23 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:01:23.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:23 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:01:23.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:23 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/97318538' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:01:23.823 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:23.846 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:23.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:23.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:23.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:23 smithi187 ceph-mon[33032]: pgmap v622: 158 pgs: 1 active+clean+remapped, 1 undersized+degraded+remapped+backfill_toofull+peered, 4 active+recovering+undersized+degraded, 35 active+undersized, 20 active+undersized+degraded, 97 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 41271/166365 objects degraded (24.808%); 20 MiB/s, 416 objects/s recovering 2024-09-21T14:01:23.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:23 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:01:23.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:23 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:01:23.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:23 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/97318538' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:01:23.948 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:23.972 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:23.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:23.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:24.037 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T14:01:24.073 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:24.084 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T14:01:24.085 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-09-21T14:01:24.085 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 0 2024-09-21T14:01:24.085 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T14:01:24.097 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:24.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:24.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:24.199 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:24.225 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:24.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:24.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:24.327 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:24.353 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:24.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:24.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:24.455 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:24.479 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:24.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:24.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:24.580 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:24.604 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:24.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:24.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:24.706 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:01:24.728 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:24.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:24.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:01:24.829 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:24.852 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:24.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:24.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:24.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:24 smithi187 ceph-mon[33032]: osdmap e337: 8 total, 7 up, 8 in 2024-09-21T14:01:24.954 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:24.979 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:24.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:24.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:25.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:24 smithi049 ceph-mon[40489]: osdmap e337: 8 total, 7 up, 8 in 2024-09-21T14:01:25.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:24 smithi049 ceph-mon[30080]: osdmap e337: 8 total, 7 up, 8 in 2024-09-21T14:01:25.035 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T14:01:25.035 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T14:01:25.081 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:25.104 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:25.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:25.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:25.206 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:25.228 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:25.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:25.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:25.330 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:25.356 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:25.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:25.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:25.458 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:01:25.480 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:25.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:25.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:01:25.582 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:01:25.604 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:25.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:25.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:01:25.689 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T14:01:25.692 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:01:25.706 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:01:25.732 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:25.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:25.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:01:25.833 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:25.860 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:25.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:25.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:25.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:25 smithi187 ceph-mon[33032]: pgmap v624: 158 pgs: 1 active+clean+remapped, 1 undersized+degraded+remapped+backfill_toofull+peered, 4 active+recovering+undersized+degraded, 35 active+undersized, 20 active+undersized+degraded, 97 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 41271/166365 objects degraded (24.808%); 12 MiB/s, 316 objects/s recovering 2024-09-21T14:01:25.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:25 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 41271/166365 objects degraded (24.808%), 25 pgs degraded (PG_DEGRADED) 2024-09-21T14:01:25.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:25 smithi187 ceph-mon[33032]: osdmap e338: 8 total, 7 up, 8 in 2024-09-21T14:01:25.959 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:25 smithi049 ceph-mon[40489]: pgmap v624: 158 pgs: 1 active+clean+remapped, 1 undersized+degraded+remapped+backfill_toofull+peered, 4 active+recovering+undersized+degraded, 35 active+undersized, 20 active+undersized+degraded, 97 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 41271/166365 objects degraded (24.808%); 12 MiB/s, 316 objects/s recovering 2024-09-21T14:01:25.959 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:25 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 41271/166365 objects degraded (24.808%), 25 pgs degraded (PG_DEGRADED) 2024-09-21T14:01:25.959 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:25 smithi049 ceph-mon[40489]: osdmap e338: 8 total, 7 up, 8 in 2024-09-21T14:01:25.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:25 smithi049 ceph-mon[30080]: pgmap v624: 158 pgs: 1 active+clean+remapped, 1 undersized+degraded+remapped+backfill_toofull+peered, 4 active+recovering+undersized+degraded, 35 active+undersized, 20 active+undersized+degraded, 97 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 41271/166365 objects degraded (24.808%); 12 MiB/s, 316 objects/s recovering 2024-09-21T14:01:25.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:25 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 41271/166365 objects degraded (24.808%), 25 pgs degraded (PG_DEGRADED) 2024-09-21T14:01:25.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:25 smithi049 ceph-mon[30080]: osdmap e338: 8 total, 7 up, 8 in 2024-09-21T14:01:25.962 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:25.983 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:25.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:25.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:26.003 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [5] live_osds: [0, 1, 4, 6, 2, 3, 7] 2024-09-21T14:01:26.004 INFO:tasks.thrashosds.thrasher:Reviving osd 5 2024-09-21T14:01:26.004 INFO:tasks.cephadm.osd.5:Restarting osd.5 (starting--it wasn't running)... 2024-09-21T14:01:26.004 DEBUG:teuthology.orchestra.run.smithi187:osd.5> sudo journalctl -f -n 0 -u ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@osd.5.service 2024-09-21T14:01:26.026 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl start ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@osd.5 2024-09-21T14:01:26.085 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:26.141 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:26.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:26.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:26.243 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:01:26.256 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:01:26.267 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:26.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:26.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:01:26.354 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:01:26 smithi187 systemd[1]: Starting Ceph osd.5 for 40a3d2e8-781f-11ef-baf6-efdc52797490... 2024-09-21T14:01:26.368 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:01:26.391 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:26.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:26.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:01:26.493 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:01:26.516 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:26.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:26.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:01:26.618 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:26.629 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:01:26 smithi187 podman[124607]: 2024-09-21 14:01:26.258935997 +0000 UTC m=+0.021857540 image pull b415816db8b41f544cdfdc71d193ff8b1ee008bf7b0ec4f64da8eb8f71b62e85 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5 2024-09-21T14:01:26.629 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:01:26 smithi187 podman[124607]: 2024-09-21 14:01:26.392489033 +0000 UTC m=+0.155410552 container create 611e3be4d321a0ac3cb72059969f83776ca208d8a493a6094a4c897646470d51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5-activate, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, ceph=True, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T14:01:26.645 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:26.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:26.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:26.655 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-09-21T14:01:26.747 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:01:26.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:26.770 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:26.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:01:26.871 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:01:26.894 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:26.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:26.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:01:26.898 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:26 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2503449905' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:01:26.898 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:26 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:01:26.898 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:26 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2503449905' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:01:26.898 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:26 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:01:26.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:26 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2503449905' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:01:26.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:26 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:01:26.945 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:01:26 smithi187 podman[124607]: 2024-09-21 14:01:26.657297703 +0000 UTC m=+0.420219232 container init 611e3be4d321a0ac3cb72059969f83776ca208d8a493a6094a4c897646470d51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5-activate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-guits-main-2024-09-21-0842-4417849, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-09-21T14:01:26.945 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:01:26 smithi187 podman[124607]: 2024-09-21 14:01:26.662695794 +0000 UTC m=+0.425617316 container start 611e3be4d321a0ac3cb72059969f83776ca208d8a493a6094a4c897646470d51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5-activate, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-21-0842-4417849, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux ) 2024-09-21T14:01:26.945 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:01:26 smithi187 podman[124607]: 2024-09-21 14:01:26.762425156 +0000 UTC m=+0.525346674 container attach 611e3be4d321a0ac3cb72059969f83776ca208d8a493a6094a4c897646470d51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5-activate, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-21-0842-4417849, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-21T14:01:26.945 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:01:26 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5-activate[124622]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T14:01:26.945 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:01:26 smithi187 bash[124607]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T14:01:26.945 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:01:26 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5-activate[124622]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T14:01:26.945 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:01:26 smithi187 bash[124607]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T14:01:26.997 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:27.019 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:27.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:27.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:27.037 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T14:01:27.121 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:01:27.143 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:27.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:27.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:01:27.245 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:27.268 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:27.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:27.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:27.327 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T14:01:27.327 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-09-21T14:01:27.327 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-09-21T14:01:27.328 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-09-21T14:01:27.328 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T14:01:27.370 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:27.393 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:27.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:27.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:27.495 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:01:27.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:27.519 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:27.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:01:27.620 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:01:27.644 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:27.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:27.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:01:27.745 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:27.768 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:27.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:27.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:27.870 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:27.892 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:27.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:27.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:27.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:27 smithi187 ceph-mon[33032]: pgmap v626: 158 pgs: 1 undersized+degraded+remapped+backfilling+peered, 1 active+clean+remapped, 3 active+recovering+undersized+degraded, 35 active+undersized, 21 active+undersized+degraded, 97 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 41177/166365 objects degraded (24.751%); 7.9 MiB/s, 228 objects/s recovering 2024-09-21T14:01:27.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:27 smithi187 ceph-mon[33032]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-21T14:01:27.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:27 smithi187 ceph-mon[33032]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:01:27.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:27 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T14:01:27.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:27 smithi187 ceph-mon[33032]: osdmap e339: 8 total, 7 up, 8 in 2024-09-21T14:01:27.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:27 smithi187 ceph-mon[33032]: 6.14 scrub starts 2024-09-21T14:01:27.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:27 smithi187 ceph-mon[33032]: 6.14 scrub ok 2024-09-21T14:01:27.994 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:28.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:27 smithi049 ceph-mon[40489]: pgmap v626: 158 pgs: 1 undersized+degraded+remapped+backfilling+peered, 1 active+clean+remapped, 3 active+recovering+undersized+degraded, 35 active+undersized, 21 active+undersized+degraded, 97 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 41177/166365 objects degraded (24.751%); 7.9 MiB/s, 228 objects/s recovering 2024-09-21T14:01:28.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:27 smithi049 ceph-mon[40489]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-21T14:01:28.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:27 smithi049 ceph-mon[40489]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:01:28.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:27 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T14:01:28.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:27 smithi049 ceph-mon[40489]: osdmap e339: 8 total, 7 up, 8 in 2024-09-21T14:01:28.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:27 smithi049 ceph-mon[40489]: 6.14 scrub starts 2024-09-21T14:01:28.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:27 smithi049 ceph-mon[40489]: 6.14 scrub ok 2024-09-21T14:01:28.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:27 smithi049 ceph-mon[30080]: pgmap v626: 158 pgs: 1 undersized+degraded+remapped+backfilling+peered, 1 active+clean+remapped, 3 active+recovering+undersized+degraded, 35 active+undersized, 21 active+undersized+degraded, 97 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 41177/166365 objects degraded (24.751%); 7.9 MiB/s, 228 objects/s recovering 2024-09-21T14:01:28.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:27 smithi049 ceph-mon[30080]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-21T14:01:28.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:27 smithi049 ceph-mon[30080]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:01:28.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:27 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T14:01:28.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:27 smithi049 ceph-mon[30080]: osdmap e339: 8 total, 7 up, 8 in 2024-09-21T14:01:28.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:27 smithi049 ceph-mon[30080]: 6.14 scrub starts 2024-09-21T14:01:28.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:27 smithi049 ceph-mon[30080]: 6.14 scrub ok 2024-09-21T14:01:28.016 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:28.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:28.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:28.118 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:28.140 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:28.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:28.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:28.164 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T14:01:28.242 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:28.263 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:28.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:28.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:28.365 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:28.388 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:28.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:28.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:28.490 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:28.512 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:28.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:28.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:28.614 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:28.638 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:28.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:28.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:28.740 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:01:28.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:28.763 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:28.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:01:28.864 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:01:28.871 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:01:28.887 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:28.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:28.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:01:28.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:28 smithi187 ceph-mon[33032]: 5.a scrub starts 2024-09-21T14:01:28.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:28 smithi187 ceph-mon[33032]: 5.a scrub ok 2024-09-21T14:01:28.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:28 smithi187 ceph-mon[33032]: 4.1c scrub starts 2024-09-21T14:01:28.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:28 smithi187 ceph-mon[33032]: 4.1c scrub ok 2024-09-21T14:01:28.989 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:01:28.994 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:28 smithi049 ceph-mon[40489]: 5.a scrub starts 2024-09-21T14:01:28.994 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:28 smithi049 ceph-mon[40489]: 5.a scrub ok 2024-09-21T14:01:28.994 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:28 smithi049 ceph-mon[40489]: 4.1c scrub starts 2024-09-21T14:01:28.994 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:28 smithi049 ceph-mon[40489]: 4.1c scrub ok 2024-09-21T14:01:28.994 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:28 smithi049 ceph-mon[30080]: 5.a scrub starts 2024-09-21T14:01:28.994 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:28 smithi049 ceph-mon[30080]: 5.a scrub ok 2024-09-21T14:01:28.994 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:28 smithi049 ceph-mon[30080]: 4.1c scrub starts 2024-09-21T14:01:28.995 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:28 smithi049 ceph-mon[30080]: 4.1c scrub ok 2024-09-21T14:01:29.011 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:29.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:29.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:01:29.113 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:01:29.135 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:29.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:29.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:01:29.238 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:29.268 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:29.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:29.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:29.370 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:29.394 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:29.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:29.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:29.408 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:01:29.408 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:01:29.426 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:01:29.426 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:01:29.435 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:01:29.443 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:01:29.444 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:01:29.462 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:01:29.462 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:01:29.479 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:01:29.480 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:01:29.480 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-21T14:01:29.495 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:29.497 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:01:29.497 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:01:29.514 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:01:29.514 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:01:29.518 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:29.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:29.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:29.620 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:01:29.643 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:29.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:29.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:01:29.745 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:29.766 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:29.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:29.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:29.868 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:01:29.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:29.891 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:29.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:01:29.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:29 smithi187 ceph-mon[33032]: pgmap v628: 158 pgs: 1 active+clean+remapped, 1 undersized+degraded+remapped+backfilling+peered, 2 active+recovering+undersized+degraded, 35 active+undersized, 22 active+undersized+degraded, 97 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 40062/166365 objects degraded (24.081%); 13 MiB/s, 303 objects/s recovering 2024-09-21T14:01:29.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:29 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:01:29.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:29 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:01:29.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:29 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2709325683' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:01:29.945 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:01:29 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:01:29.945 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:01:29 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5-activate[124622]: --> Failed to activate via raw: did not find any matching OSD to activate 2024-09-21T14:01:29.945 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:01:29 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5-activate[124622]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T14:01:29.945 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:01:29 smithi187 bash[124607]: --> Failed to activate via raw: did not find any matching OSD to activate 2024-09-21T14:01:29.946 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:01:29 smithi187 bash[124607]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T14:01:29.946 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:01:29 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5-activate[124622]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T14:01:29.946 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:01:29 smithi187 bash[124607]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T14:01:29.946 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:01:29 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5-activate[124622]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-21T14:01:29.946 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:01:29 smithi187 bash[124607]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-21T14:01:29.946 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:01:29 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5-activate[124622]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-5 --no-mon-config 2024-09-21T14:01:29.946 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:01:29 smithi187 bash[124607]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-5 --no-mon-config 2024-09-21T14:01:29.993 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:30.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:29 smithi049 ceph-mon[40489]: pgmap v628: 158 pgs: 1 active+clean+remapped, 1 undersized+degraded+remapped+backfilling+peered, 2 active+recovering+undersized+degraded, 35 active+undersized, 22 active+undersized+degraded, 97 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 40062/166365 objects degraded (24.081%); 13 MiB/s, 303 objects/s recovering 2024-09-21T14:01:30.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:29 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:01:30.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:29 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:01:30.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:29 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2709325683' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:01:30.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:29 smithi049 ceph-mon[30080]: pgmap v628: 158 pgs: 1 active+clean+remapped, 1 undersized+degraded+remapped+backfilling+peered, 2 active+recovering+undersized+degraded, 35 active+undersized, 22 active+undersized+degraded, 97 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 40062/166365 objects degraded (24.081%); 13 MiB/s, 303 objects/s recovering 2024-09-21T14:01:30.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:29 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:01:30.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:29 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:01:30.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:29 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2709325683' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:01:30.019 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:30.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:30.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:30.121 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:01:30.143 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:30.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:30.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:01:30.244 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:01:30.266 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:30.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:30.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:01:30.368 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:01:30.389 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:30.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:30.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:01:30.445 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:01:30 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5-activate[124622]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5/block 2024-09-21T14:01:30.445 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:01:30 smithi187 bash[124607]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5/block 2024-09-21T14:01:30.445 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:01:30 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5-activate[124622]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-5/block 2024-09-21T14:01:30.445 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:01:30 smithi187 bash[124607]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-5/block 2024-09-21T14:01:30.445 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:01:30 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5-activate[124622]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-21T14:01:30.445 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:01:30 smithi187 bash[124607]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-21T14:01:30.445 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:01:30 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5-activate[124622]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-21T14:01:30.445 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:01:30 smithi187 bash[124607]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-21T14:01:30.445 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:01:30 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5-activate[124622]: --> ceph-volume lvm activate successful for osd ID: 5 2024-09-21T14:01:30.445 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:01:30 smithi187 bash[124607]: --> ceph-volume lvm activate successful for osd ID: 5 2024-09-21T14:01:30.445 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:01:30 smithi187 podman[124607]: 2024-09-21 14:01:30.198723407 +0000 UTC m=+3.961644931 container died 611e3be4d321a0ac3cb72059969f83776ca208d8a493a6094a4c897646470d51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5-activate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-21-0842-4417849, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2) 2024-09-21T14:01:30.491 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:01:30.513 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:30.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:30.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:01:30.614 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:30.617 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T14:01:30.618 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T14:01:30.637 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:30.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:30.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:30.739 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:01:30.761 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:30.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:30.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:01:30.826 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:30 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:01:30.826 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:30 smithi187 ceph-mon[33032]: 3.a scrub starts 2024-09-21T14:01:30.826 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:30 smithi187 ceph-mon[33032]: 3.a scrub ok 2024-09-21T14:01:30.826 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:30 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 40062/166365 objects degraded (24.081%), 25 pgs degraded (PG_DEGRADED) 2024-09-21T14:01:30.826 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:30 smithi187 ceph-mon[33032]: osdmap e340: 8 total, 7 up, 8 in 2024-09-21T14:01:30.826 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T14:01:30.849 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:01:30 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:01:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:01:30.849 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:30 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:01:30.849 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:30 smithi049 ceph-mon[40489]: 3.a scrub starts 2024-09-21T14:01:30.849 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:30 smithi049 ceph-mon[40489]: 3.a scrub ok 2024-09-21T14:01:30.849 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:30 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 40062/166365 objects degraded (24.081%), 25 pgs degraded (PG_DEGRADED) 2024-09-21T14:01:30.849 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:30 smithi049 ceph-mon[40489]: osdmap e340: 8 total, 7 up, 8 in 2024-09-21T14:01:30.850 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:30 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:01:30.850 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:30 smithi049 ceph-mon[30080]: 3.a scrub starts 2024-09-21T14:01:30.850 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:30 smithi049 ceph-mon[30080]: 3.a scrub ok 2024-09-21T14:01:30.850 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:30 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 40062/166365 objects degraded (24.081%), 25 pgs degraded (PG_DEGRADED) 2024-09-21T14:01:30.850 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:30 smithi049 ceph-mon[30080]: osdmap e340: 8 total, 7 up, 8 in 2024-09-21T14:01:30.863 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:01:30.886 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:30.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:30.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:01:30.915 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T14:01:30.915 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:01:30.988 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:31.010 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:31.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:31.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:31.077 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:01:30 smithi187 podman[125097]: 2024-09-21 14:01:30.825719294 +0000 UTC m=+0.615448515 container remove 611e3be4d321a0ac3cb72059969f83776ca208d8a493a6094a4c897646470d51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5-activate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-21-0842-4417849, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, ceph=True) 2024-09-21T14:01:31.077 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:01:31 smithi187 podman[125159]: 2024-09-21 14:01:30.962193797 +0000 UTC m=+0.023231646 image pull b415816db8b41f544cdfdc71d193ff8b1ee008bf7b0ec4f64da8eb8f71b62e85 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5 2024-09-21T14:01:31.077 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:01:31 smithi187 podman[125159]: 2024-09-21 14:01:31.077144638 +0000 UTC m=+0.138182471 container create 1fc4fc5f79021f071b26a40fca744526db585ab2a34443ff1cbbccb5272ff339 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5, RELEASE=wip-guits-main-2024-09-21-0842-4417849, io.buildah.version=1.37.2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T14:01:31.112 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:01:31.134 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:31.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:31.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:01:31.236 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:01:31.259 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:31.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:31.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:01:31.344 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:01:31 smithi187 podman[125159]: 2024-09-21 14:01:31.313315086 +0000 UTC m=+0.374352925 container init 1fc4fc5f79021f071b26a40fca744526db585ab2a34443ff1cbbccb5272ff339 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-guits-main-2024-09-21-0842-4417849, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=) 2024-09-21T14:01:31.344 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:01:31 smithi187 podman[125159]: 2024-09-21 14:01:31.320238521 +0000 UTC m=+0.381276348 container start 1fc4fc5f79021f071b26a40fca744526db585ab2a34443ff1cbbccb5272ff339 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-guits-main-2024-09-21-0842-4417849, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T14:01:31.360 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:31.385 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:31.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:31.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:31.444 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T14:01:31.486 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:31.508 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:31.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:31.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:31.610 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:01:31.632 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:31.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:31.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:01:31.642 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:01:31 smithi187 ceph-osd[125198]: -- 172.21.15.187:0/1989416333 <== mon.1 v2:172.21.15.187:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55dce3f0da40 con 0x55dce4cdc400 2024-09-21T14:01:31.642 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:01:31 smithi187 bash[125159]: 1fc4fc5f79021f071b26a40fca744526db585ab2a34443ff1cbbccb5272ff339 2024-09-21T14:01:31.642 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:01:31 smithi187 systemd[1]: Started Ceph osd.5 for 40a3d2e8-781f-11ef-baf6-efdc52797490. 2024-09-21T14:01:31.734 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:31.752 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:01:31.758 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:31.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:31.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:31.766 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T14:01:31.859 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:01:31.860 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:31.883 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:31.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:31.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:31.898 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:31 smithi187 ceph-mon[33032]: pgmap v629: 158 pgs: 1 undersized+degraded+remapped+backfilling+peered, 1 active+clean+remapped, 2 active+recovering+undersized+degraded, 35 active+undersized, 22 active+undersized+degraded, 97 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 807 B/s rd, 0 op/s; 39624/166365 objects degraded (23.818%); 16 MiB/s, 356 objects/s recovering 2024-09-21T14:01:31.980 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:31 smithi049 ceph-mon[40489]: pgmap v629: 158 pgs: 1 undersized+degraded+remapped+backfilling+peered, 1 active+clean+remapped, 2 active+recovering+undersized+degraded, 35 active+undersized, 22 active+undersized+degraded, 97 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 807 B/s rd, 0 op/s; 39624/166365 objects degraded (23.818%); 16 MiB/s, 356 objects/s recovering 2024-09-21T14:01:31.981 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:31 smithi049 ceph-mon[30080]: pgmap v629: 158 pgs: 1 undersized+degraded+remapped+backfilling+peered, 1 active+clean+remapped, 2 active+recovering+undersized+degraded, 35 active+undersized, 22 active+undersized+degraded, 97 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 807 B/s rd, 0 op/s; 39624/166365 objects degraded (23.818%); 16 MiB/s, 356 objects/s recovering 2024-09-21T14:01:31.985 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:32.012 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:32.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:32.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:32.114 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:32.137 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:32.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:32.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:32.239 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:01:32.261 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:32.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:32.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:01:32.299 INFO:teuthology.orchestra.run.smithi187.stderr:no valid command found; 10 closest matches: 2024-09-21T14:01:32.300 INFO:teuthology.orchestra.run.smithi187.stderr:0 2024-09-21T14:01:32.300 INFO:teuthology.orchestra.run.smithi187.stderr:1 2024-09-21T14:01:32.300 INFO:teuthology.orchestra.run.smithi187.stderr:2 2024-09-21T14:01:32.300 INFO:teuthology.orchestra.run.smithi187.stderr:abort 2024-09-21T14:01:32.300 INFO:teuthology.orchestra.run.smithi187.stderr:assert 2024-09-21T14:01:32.300 INFO:teuthology.orchestra.run.smithi187.stderr:config diff 2024-09-21T14:01:32.300 INFO:teuthology.orchestra.run.smithi187.stderr:config diff get 2024-09-21T14:01:32.300 INFO:teuthology.orchestra.run.smithi187.stderr:config get 2024-09-21T14:01:32.300 INFO:teuthology.orchestra.run.smithi187.stderr:config help [] 2024-09-21T14:01:32.300 INFO:teuthology.orchestra.run.smithi187.stderr:config set ... 2024-09-21T14:01:32.300 INFO:teuthology.orchestra.run.smithi187.stderr:admin_socket: invalid command 2024-09-21T14:01:32.363 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:32.388 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:32.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:32.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:32.490 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:32.513 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:32.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:32.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:32.614 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:32.619 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T14:01:32.637 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:32.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:32.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:32.738 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:32.760 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:32.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:32.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:32.862 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:01:32.885 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:01:32.887 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:32.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:32.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:01:32.957 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:01:32.989 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:01:32.990 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T14:01:32.990 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-5, ['dump_ops_in_flight'] 2024-09-21T14:01:33.012 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:33.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:33.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:01:33.114 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:33.135 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:33.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:33.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:33.237 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:01:33.259 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:33.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:33.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:01:33.361 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:33.384 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:33.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:33.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:33.422 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:01:33.423 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:01:33.435 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:01:33.436 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:01:33.452 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:01:33.452 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:01:33.468 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:01:33.468 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:01:33.485 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:01:33.485 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:01:33.485 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-21T14:01:33.486 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:01:33.486 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:01:33 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5[125175]: 2024-09-21T14:01:33.211+0000 7f29ed132640 -1 Falling back to public interface 2024-09-21T14:01:33.504 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:01:33.504 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:01:33.508 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T14:01:33.508 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-21T14:01:33.508 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 0 2024-09-21T14:01:33.508 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T14:01:33.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:33.509 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:33.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:01:33.522 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:01:33.522 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:01:33.610 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:01:33.632 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:33.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:33.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:01:33.657 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-09-21T14:01:33.734 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:33.756 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:33.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:33.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:33.857 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:01:33.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:33.881 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:33.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:01:33.890 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:33 smithi187 ceph-mon[33032]: pgmap v631: 158 pgs: 1 active+clean+remapped, 1 undersized+degraded+remapped+backfilling+peered, 2 active+recovering+undersized+degraded, 35 active+undersized, 22 active+undersized+degraded, 97 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 39265/166365 objects degraded (23.602%); 17 MiB/s, 277 objects/s recovering 2024-09-21T14:01:33.890 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:33 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:01:33.890 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:33 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:01:33.890 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:33 smithi187 ceph-mon[33032]: osdmap e341: 8 total, 7 up, 8 in 2024-09-21T14:01:33.890 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:33 smithi187 ceph-mon[33032]: 5.18 scrub starts 2024-09-21T14:01:33.890 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:33 smithi187 ceph-mon[33032]: 5.18 scrub ok 2024-09-21T14:01:33.890 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:33 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3003940986' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:01:33.890 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:33 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:01:33.890 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:33 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/963991653' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:01:33.982 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:34.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:33 smithi049 ceph-mon[40489]: pgmap v631: 158 pgs: 1 active+clean+remapped, 1 undersized+degraded+remapped+backfilling+peered, 2 active+recovering+undersized+degraded, 35 active+undersized, 22 active+undersized+degraded, 97 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 39265/166365 objects degraded (23.602%); 17 MiB/s, 277 objects/s recovering 2024-09-21T14:01:34.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:33 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:01:34.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:33 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:01:34.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:33 smithi049 ceph-mon[40489]: osdmap e341: 8 total, 7 up, 8 in 2024-09-21T14:01:34.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:33 smithi049 ceph-mon[40489]: 5.18 scrub starts 2024-09-21T14:01:34.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:33 smithi049 ceph-mon[40489]: 5.18 scrub ok 2024-09-21T14:01:34.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:33 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3003940986' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:01:34.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:33 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:01:34.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:33 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/963991653' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:01:34.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:33 smithi049 ceph-mon[30080]: pgmap v631: 158 pgs: 1 active+clean+remapped, 1 undersized+degraded+remapped+backfilling+peered, 2 active+recovering+undersized+degraded, 35 active+undersized, 22 active+undersized+degraded, 97 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 39265/166365 objects degraded (23.602%); 17 MiB/s, 277 objects/s recovering 2024-09-21T14:01:34.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:33 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:01:34.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:33 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:01:34.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:33 smithi049 ceph-mon[30080]: osdmap e341: 8 total, 7 up, 8 in 2024-09-21T14:01:34.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:33 smithi049 ceph-mon[30080]: 5.18 scrub starts 2024-09-21T14:01:34.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:33 smithi049 ceph-mon[30080]: 5.18 scrub ok 2024-09-21T14:01:34.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:33 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3003940986' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:01:34.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:33 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:01:34.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:33 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/963991653' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:01:34.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:34.006 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:34.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:34.107 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:34.130 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:34.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:34.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:34.231 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:01:34.254 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:34.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:34.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:01:34.293 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T14:01:34.356 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:01:34.378 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:34.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:34.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:01:34.480 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:01:34.502 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:34.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:34.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:01:34.583 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:01:34.604 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:01:34.627 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:34.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:34.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:01:34.650 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:34 smithi187 ceph-mon[33032]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T14:01:34.650 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:34 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T14:01:34.650 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:34 smithi187 ceph-mon[33032]: osdmap e342: 8 total, 7 up, 8 in 2024-09-21T14:01:34.728 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:34.752 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:34.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:34.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:34.854 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:34.878 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:34.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:34.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:34.979 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:35.001 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:35.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:35.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:35.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:34 smithi049 ceph-mon[40489]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T14:01:35.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:34 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T14:01:35.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:34 smithi049 ceph-mon[40489]: osdmap e342: 8 total, 7 up, 8 in 2024-09-21T14:01:35.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:34 smithi049 ceph-mon[30080]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T14:01:35.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:34 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T14:01:35.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:34 smithi049 ceph-mon[30080]: osdmap e342: 8 total, 7 up, 8 in 2024-09-21T14:01:35.103 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:35.127 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:35.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:35.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:35.130 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T14:01:35.130 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-21T14:01:35.130 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-21T14:01:35.130 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-21T14:01:35.130 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T14:01:35.178 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T14:01:35.178 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T14:01:35.229 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:35.251 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:35.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:35.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:35.353 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:01:35.375 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:35.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:35.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:01:35.477 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:01:35.499 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:35.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:35.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:01:35.603 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:01:35.625 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:35.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:35.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:01:35.727 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:35.748 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:35.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:35.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:35.799 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:35 smithi187 ceph-mon[33032]: pgmap v634: 158 pgs: 1 active+clean+remapped, 1 undersized+degraded+remapped+backfilling+peered, 2 active+recovering+undersized+degraded, 35 active+undersized, 22 active+undersized+degraded, 97 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 39265/166365 objects degraded (23.602%); 8.3 MiB/s, 132 objects/s recovering 2024-09-21T14:01:35.799 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:35 smithi187 ceph-mon[33032]: 4.18 scrub starts 2024-09-21T14:01:35.799 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:35 smithi187 ceph-mon[33032]: 4.18 scrub ok 2024-09-21T14:01:35.799 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:35 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 39265/166365 objects degraded (23.602%), 25 pgs degraded (PG_DEGRADED) 2024-09-21T14:01:35.834 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T14:01:35.850 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:35.901 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:35.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:35.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:36.003 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:36.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:35 smithi049 ceph-mon[40489]: pgmap v634: 158 pgs: 1 active+clean+remapped, 1 undersized+degraded+remapped+backfilling+peered, 2 active+recovering+undersized+degraded, 35 active+undersized, 22 active+undersized+degraded, 97 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 39265/166365 objects degraded (23.602%); 8.3 MiB/s, 132 objects/s recovering 2024-09-21T14:01:36.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:35 smithi049 ceph-mon[40489]: 4.18 scrub starts 2024-09-21T14:01:36.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:35 smithi049 ceph-mon[40489]: 4.18 scrub ok 2024-09-21T14:01:36.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:35 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 39265/166365 objects degraded (23.602%), 25 pgs degraded (PG_DEGRADED) 2024-09-21T14:01:36.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:35 smithi049 ceph-mon[30080]: pgmap v634: 158 pgs: 1 active+clean+remapped, 1 undersized+degraded+remapped+backfilling+peered, 2 active+recovering+undersized+degraded, 35 active+undersized, 22 active+undersized+degraded, 97 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 39265/166365 objects degraded (23.602%); 8.3 MiB/s, 132 objects/s recovering 2024-09-21T14:01:36.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:35 smithi049 ceph-mon[30080]: 4.18 scrub starts 2024-09-21T14:01:36.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:35 smithi049 ceph-mon[30080]: 4.18 scrub ok 2024-09-21T14:01:36.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:35 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 39265/166365 objects degraded (23.602%), 25 pgs degraded (PG_DEGRADED) 2024-09-21T14:01:36.027 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:36.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:36.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:36.121 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:01:36.129 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:36.154 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:36.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:36.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:36.256 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:01:36.278 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:36.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:36.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:01:36.380 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:36.402 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:36.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:36.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:36.503 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:01:36.525 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:36.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:36.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:01:36.627 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:36.652 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T14:01:36.653 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:36.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:36.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:36.726 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T14:01:36.726 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-21T14:01:36.726 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-21T14:01:36.726 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-21T14:01:36.726 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.726 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:202117 7.1 7:8d183654:::benchmark_data_smithi049_76798_object65697:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-21T14:01:36.726 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:52:44.231638+0000", 2024-09-21T14:01:36.726 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 532.49025989100005, 2024-09-21T14:01:36.727 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.111972288, 2024-09-21T14:01:36.727 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:01:36.727 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:01:36.727 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:01:36.727 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:01:36.727 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T14:01:36.727 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T14:01:36.727 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 202117 2024-09-21T14:01:36.727 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.727 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:01:36.727 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.727 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:01:36.727 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.231638+0000", 2024-09-21T14:01:36.728 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:01:36.728 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.728 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.728 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:01:36.728 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.231638+0000", 2024-09-21T14:01:36.728 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:01:36.728 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.728 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.728 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:01:36.728 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.231640+0000", 2024-09-21T14:01:36.728 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3329999999999999e-06 2024-09-21T14:01:36.728 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.728 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.728 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:01:36.729 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.231649+0000", 2024-09-21T14:01:36.729 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0399999999999998e-06 2024-09-21T14:01:36.729 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.729 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.729 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:01:36.729 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.231674+0000", 2024-09-21T14:01:36.729 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.461e-05 2024-09-21T14:01:36.729 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.729 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.729 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:01:36.729 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.231688+0000", 2024-09-21T14:01:36.729 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4063000000000001e-05 2024-09-21T14:01:36.729 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.730 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.730 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:01:36.730 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.231803+0000", 2024-09-21T14:01:36.730 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000115295 2024-09-21T14:01:36.730 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.730 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.730 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:01:36.730 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.231950+0000", 2024-09-21T14:01:36.730 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014642 2024-09-21T14:01:36.730 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.730 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.730 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,6", 2024-09-21T14:01:36.730 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.232017+0000", 2024-09-21T14:01:36.731 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7754000000000002e-05 2024-09-21T14:01:36.731 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.731 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.731 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:01:36.731 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.233984+0000", 2024-09-21T14:01:36.731 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0019670320000000001 2024-09-21T14:01:36.731 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.731 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.731 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:01:36.731 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.236712+0000", 2024-09-21T14:01:36.731 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0027279040000000002 2024-09-21T14:01:36.731 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.731 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.732 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:01:36.732 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.343530+0000", 2024-09-21T14:01:36.732 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.106818107 2024-09-21T14:01:36.732 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.732 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.732 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:01:36.732 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.343568+0000", 2024-09-21T14:01:36.732 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7317000000000002e-05 2024-09-21T14:01:36.732 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.732 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.732 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:01:36.732 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:52:44.343610+0000", 2024-09-21T14:01:36.732 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2413000000000002e-05 2024-09-21T14:01:36.732 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:01:36.733 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:01:36.733 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:01:36.733 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.733 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.733 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.14817.0:249864 7.1 7:8a44b6f3:::benchmark_data_smithi049_76798_object113444:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e168)", 2024-09-21T14:01:36.733 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:53:36.300462+0000", 2024-09-21T14:01:36.733 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 480.42143598899997, 2024-09-21T14:01:36.733 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.111981706, 2024-09-21T14:01:36.733 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:01:36.733 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:01:36.733 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:01:36.733 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:01:36.733 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.14817", 2024-09-21T14:01:36.734 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/1926912850", 2024-09-21T14:01:36.734 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 249864 2024-09-21T14:01:36.734 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.734 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:01:36.734 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.734 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:01:36.734 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:53:36.300462+0000", 2024-09-21T14:01:36.734 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:01:36.734 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.734 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.734 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:01:36.734 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:53:36.300462+0000", 2024-09-21T14:01:36.734 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:01:36.735 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.735 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.735 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:01:36.735 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:53:36.300463+0000", 2024-09-21T14:01:36.735 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7100000000000011e-07 2024-09-21T14:01:36.735 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.735 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.735 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:01:36.735 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:53:36.300467+0000", 2024-09-21T14:01:36.735 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3830000000000002e-06 2024-09-21T14:01:36.735 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.735 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.735 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:01:36.736 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:53:36.300478+0000", 2024-09-21T14:01:36.736 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.078e-05 2024-09-21T14:01:36.736 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.736 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.736 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:01:36.736 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:53:36.300486+0000", 2024-09-21T14:01:36.736 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6960000000000005e-06 2024-09-21T14:01:36.736 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.736 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.736 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:01:36.736 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:53:36.300533+0000", 2024-09-21T14:01:36.736 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7756999999999997e-05 2024-09-21T14:01:36.736 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.736 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.737 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:01:36.737 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:53:36.300635+0000", 2024-09-21T14:01:36.737 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000101618 2024-09-21T14:01:36.737 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.737 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.737 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,6", 2024-09-21T14:01:36.737 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:53:36.300736+0000", 2024-09-21T14:01:36.737 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000101516 2024-09-21T14:01:36.737 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.738 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.738 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:01:36.738 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:53:36.303427+0000", 2024-09-21T14:01:36.738 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.002690909 2024-09-21T14:01:36.738 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.738 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.738 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:01:36.738 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:53:36.303790+0000", 2024-09-21T14:01:36.738 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00036281099999999999 2024-09-21T14:01:36.738 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.738 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.738 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:01:36.738 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:53:36.412339+0000", 2024-09-21T14:01:36.739 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.108549004 2024-09-21T14:01:36.739 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.739 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.739 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:01:36.739 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:53:36.412400+0000", 2024-09-21T14:01:36.739 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0581000000000002e-05 2024-09-21T14:01:36.739 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.739 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.739 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:01:36.739 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:53:36.412443+0000", 2024-09-21T14:01:36.739 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3680000000000002e-05 2024-09-21T14:01:36.739 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:01:36.739 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:01:36.739 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:01:36.740 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.740 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.740 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.15207.0:14214 8.2 8:46d3a502:::benchmark_data_smithi049_109311_object14213:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-09-21T14:01:36.740 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:54:23.117083+0000", 2024-09-21T14:01:36.740 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 433.60481448899998, 2024-09-21T14:01:36.740 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.12033872, 2024-09-21T14:01:36.740 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:01:36.740 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:01:36.740 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:01:36.740 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:01:36.740 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.15207", 2024-09-21T14:01:36.740 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:01:36.740 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 14214 2024-09-21T14:01:36.740 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.740 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:01:36.741 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.741 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:01:36.741 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.117083+0000", 2024-09-21T14:01:36.741 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:01:36.741 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.741 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.741 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:01:36.741 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.117083+0000", 2024-09-21T14:01:36.741 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:01:36.741 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.741 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.741 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:01:36.741 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.117094+0000", 2024-09-21T14:01:36.741 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0915e-05 2024-09-21T14:01:36.742 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.742 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.742 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:01:36.742 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.117127+0000", 2024-09-21T14:01:36.742 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3222999999999997e-05 2024-09-21T14:01:36.742 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.742 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.742 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:01:36.742 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.117141+0000", 2024-09-21T14:01:36.742 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3604999999999999e-05 2024-09-21T14:01:36.742 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.742 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.742 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:01:36.743 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.117148+0000", 2024-09-21T14:01:36.743 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5739999999999998e-06 2024-09-21T14:01:36.743 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.743 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.743 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:01:36.743 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.117215+0000", 2024-09-21T14:01:36.743 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6672000000000001e-05 2024-09-21T14:01:36.743 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.743 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.743 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:01:36.743 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.117316+0000", 2024-09-21T14:01:36.743 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000100831 2024-09-21T14:01:36.743 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.744 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.744 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,5", 2024-09-21T14:01:36.744 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.117464+0000", 2024-09-21T14:01:36.744 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000148035 2024-09-21T14:01:36.744 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.744 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.744 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:01:36.744 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.118494+0000", 2024-09-21T14:01:36.744 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001029587 2024-09-21T14:01:36.744 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.744 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.744 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:01:36.744 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.118598+0000", 2024-09-21T14:01:36.745 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000104079 2024-09-21T14:01:36.745 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.745 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.745 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:01:36.745 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.237344+0000", 2024-09-21T14:01:36.745 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.118746166 2024-09-21T14:01:36.745 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.745 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.745 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:01:36.745 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.237385+0000", 2024-09-21T14:01:36.745 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1495000000000002e-05 2024-09-21T14:01:36.745 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.745 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.745 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:01:36.746 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.237422+0000", 2024-09-21T14:01:36.746 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6538000000000002e-05 2024-09-21T14:01:36.746 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:01:36.746 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:01:36.746 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:01:36.746 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.746 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.746 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.15207.0:14217 8.2 8:42c0ac0e:::benchmark_data_smithi049_109311_object14216:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-09-21T14:01:36.746 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:54:23.118628+0000", 2024-09-21T14:01:36.746 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 433.603270103, 2024-09-21T14:01:36.746 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.120069462, 2024-09-21T14:01:36.746 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:01:36.746 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:01:36.746 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:01:36.747 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:01:36.747 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.15207", 2024-09-21T14:01:36.747 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:01:36.747 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 14217 2024-09-21T14:01:36.747 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.747 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:01:36.747 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.747 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:01:36.747 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.118628+0000", 2024-09-21T14:01:36.747 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:01:36.747 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.747 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.747 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:01:36.747 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.118628+0000", 2024-09-21T14:01:36.747 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:01:36.748 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.748 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.748 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:01:36.748 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.118629+0000", 2024-09-21T14:01:36.748 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0499999999999999e-06 2024-09-21T14:01:36.748 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.748 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.748 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:01:36.748 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.118673+0000", 2024-09-21T14:01:36.748 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4274000000000003e-05 2024-09-21T14:01:36.748 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.748 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.748 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:01:36.748 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.118686+0000", 2024-09-21T14:01:36.749 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3189e-05 2024-09-21T14:01:36.749 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.749 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.749 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:01:36.749 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.118695+0000", 2024-09-21T14:01:36.749 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0690000000000008e-06 2024-09-21T14:01:36.749 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.749 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.749 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:01:36.749 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.118761+0000", 2024-09-21T14:01:36.749 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6203000000000003e-05 2024-09-21T14:01:36.749 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.749 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.749 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:01:36.749 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.118897+0000", 2024-09-21T14:01:36.750 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013574400000000001 2024-09-21T14:01:36.750 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.750 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.750 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,5", 2024-09-21T14:01:36.750 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.119033+0000", 2024-09-21T14:01:36.750 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013564499999999999 2024-09-21T14:01:36.750 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.750 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.750 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:01:36.750 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.119556+0000", 2024-09-21T14:01:36.750 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00052297800000000001 2024-09-21T14:01:36.750 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.750 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.750 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:01:36.751 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.120042+0000", 2024-09-21T14:01:36.751 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00048602399999999999 2024-09-21T14:01:36.751 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.751 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.751 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:01:36.751 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.238612+0000", 2024-09-21T14:01:36.751 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.118570721 2024-09-21T14:01:36.751 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.751 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.751 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:01:36.751 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.238657+0000", 2024-09-21T14:01:36.751 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4935000000000003e-05 2024-09-21T14:01:36.751 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.751 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.752 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:01:36.752 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.238697+0000", 2024-09-21T14:01:36.752 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9629999999999998e-05 2024-09-21T14:01:36.752 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:01:36.752 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:01:36.752 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:01:36.752 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.752 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.752 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.15207.0:211127 8.b 8:dd21c456:::benchmark_data_smithi049_109311_object84608:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-09-21T14:01:36.752 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:56:54.629514+0000", 2024-09-21T14:01:36.752 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 282.09238404600001, 2024-09-21T14:01:36.752 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13217907500000001, 2024-09-21T14:01:36.752 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:01:36.752 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:01:36.753 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:01:36.753 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:01:36.753 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.15207", 2024-09-21T14:01:36.753 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:01:36.753 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 211127 2024-09-21T14:01:36.753 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.753 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:01:36.753 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.753 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:01:36.753 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.629514+0000", 2024-09-21T14:01:36.753 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:01:36.753 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.753 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.753 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:01:36.754 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.629514+0000", 2024-09-21T14:01:36.754 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:01:36.754 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.754 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.754 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:01:36.754 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.629514+0000", 2024-09-21T14:01:36.754 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4999999999999998e-07 2024-09-21T14:01:36.754 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.754 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:36.756 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.756 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:01:36.756 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.629516+0000", 2024-09-21T14:01:36.756 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7010000000000001e-06 2024-09-21T14:01:36.756 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.756 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.756 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:01:36.756 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.629519+0000", 2024-09-21T14:01:36.756 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.766e-06 2024-09-21T14:01:36.756 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.756 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.756 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:01:36.757 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.629522+0000", 2024-09-21T14:01:36.757 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9519999999999999e-06 2024-09-21T14:01:36.757 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.757 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.757 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:01:36.757 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.707551+0000", 2024-09-21T14:01:36.757 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.078028231000000003 2024-09-21T14:01:36.757 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.757 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.757 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:01:36.757 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.707650+0000", 2024-09-21T14:01:36.757 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.9407999999999994e-05 2024-09-21T14:01:36.758 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.758 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.758 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1", 2024-09-21T14:01:36.758 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.707755+0000", 2024-09-21T14:01:36.758 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000105178 2024-09-21T14:01:36.758 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.758 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.758 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:01:36.758 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.716265+0000", 2024-09-21T14:01:36.758 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0085100430000000001 2024-09-21T14:01:36.758 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.758 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.758 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:01:36.759 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.761562+0000", 2024-09-21T14:01:36.759 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.045296967 2024-09-21T14:01:36.759 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.759 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.759 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:01:36.759 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.761602+0000", 2024-09-21T14:01:36.759 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9913999999999998e-05 2024-09-21T14:01:36.759 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.759 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.759 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:01:36.759 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.761693+0000", 2024-09-21T14:01:36.759 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0564999999999998e-05 2024-09-21T14:01:36.759 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:01:36.759 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:01:36.760 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:01:36.760 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.760 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.760 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.15207.0:211128 8.b 8:dde34708:::benchmark_data_smithi049_109311_object84609:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-09-21T14:01:36.760 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:56:54.629541+0000", 2024-09-21T14:01:36.760 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 282.09235667500002, 2024-09-21T14:01:36.760 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.14686216199999999, 2024-09-21T14:01:36.760 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:01:36.760 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:01:36.760 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:01:36.760 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:01:36.760 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.15207", 2024-09-21T14:01:36.760 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:01:36.761 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 211128 2024-09-21T14:01:36.761 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.761 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:01:36.761 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.761 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:01:36.761 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.629541+0000", 2024-09-21T14:01:36.761 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:01:36.761 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.761 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.761 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:01:36.761 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.629541+0000", 2024-09-21T14:01:36.761 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:01:36.761 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.762 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.762 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:01:36.762 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.629541+0000", 2024-09-21T14:01:36.762 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0699999999999998e-07 2024-09-21T14:01:36.762 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.762 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.762 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:01:36.762 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.629543+0000", 2024-09-21T14:01:36.762 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.513e-06 2024-09-21T14:01:36.762 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.762 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.762 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:01:36.762 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.629546+0000", 2024-09-21T14:01:36.763 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.642e-06 2024-09-21T14:01:36.763 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.763 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.763 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:01:36.763 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.629549+0000", 2024-09-21T14:01:36.763 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8480000000000001e-06 2024-09-21T14:01:36.763 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.763 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.763 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:01:36.763 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.710199+0000", 2024-09-21T14:01:36.763 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.080649352999999993 2024-09-21T14:01:36.763 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.763 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.763 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:01:36.764 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.710312+0000", 2024-09-21T14:01:36.764 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011377799999999999 2024-09-21T14:01:36.764 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.764 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.764 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1", 2024-09-21T14:01:36.764 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.710674+0000", 2024-09-21T14:01:36.764 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00036162500000000002 2024-09-21T14:01:36.764 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.764 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.764 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:01:36.764 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.716270+0000", 2024-09-21T14:01:36.764 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0055961520000000001 2024-09-21T14:01:36.764 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.765 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.765 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:01:36.765 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.776233+0000", 2024-09-21T14:01:36.765 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.059962834999999999 2024-09-21T14:01:36.765 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.765 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.765 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:01:36.765 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.776304+0000", 2024-09-21T14:01:36.765 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1371999999999994e-05 2024-09-21T14:01:36.765 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.765 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.765 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:01:36.765 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.776403+0000", 2024-09-21T14:01:36.766 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.8737e-05 2024-09-21T14:01:36.766 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:01:36.766 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:01:36.766 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:01:36.766 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.766 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.766 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.15207.0:212072 8.5 8:af76a4b5:::benchmark_data_smithi049_109311_object85553:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-09-21T14:01:36.766 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:56:55.635001+0000", 2024-09-21T14:01:36.766 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 281.086896647, 2024-09-21T14:01:36.766 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.10843544099999999, 2024-09-21T14:01:36.766 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:01:36.766 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:01:36.766 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:01:36.767 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:01:36.767 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.15207", 2024-09-21T14:01:36.767 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:01:36.767 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 212072 2024-09-21T14:01:36.767 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.767 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:01:36.767 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.767 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:01:36.767 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:55.635001+0000", 2024-09-21T14:01:36.767 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:01:36.767 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.767 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.767 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:01:36.767 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:55.635001+0000", 2024-09-21T14:01:36.767 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:01:36.768 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.768 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.768 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:01:36.768 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:55.635003+0000", 2024-09-21T14:01:36.768 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.533e-06 2024-09-21T14:01:36.768 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.768 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.768 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:01:36.768 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:55.635008+0000", 2024-09-21T14:01:36.768 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0839999999999999e-06 2024-09-21T14:01:36.768 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.768 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.768 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:01:36.768 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:55.635021+0000", 2024-09-21T14:01:36.769 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3047000000000001e-05 2024-09-21T14:01:36.769 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.769 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.769 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:01:36.769 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:55.635029+0000", 2024-09-21T14:01:36.769 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0499999999999992e-06 2024-09-21T14:01:36.769 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.769 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.769 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:01:36.769 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:55.635089+0000", 2024-09-21T14:01:36.769 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0118999999999998e-05 2024-09-21T14:01:36.769 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.769 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.769 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:01:36.769 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:55.635218+0000", 2024-09-21T14:01:36.770 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012881000000000001 2024-09-21T14:01:36.770 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.770 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.770 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:01:36.770 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:55.635330+0000", 2024-09-21T14:01:36.770 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000112575 2024-09-21T14:01:36.770 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.770 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.770 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:01:36.770 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:55.638008+0000", 2024-09-21T14:01:36.770 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0026773069999999999 2024-09-21T14:01:36.770 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.770 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.771 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:01:36.771 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:55.639072+0000", 2024-09-21T14:01:36.771 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00106401 2024-09-21T14:01:36.771 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.771 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.771 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:01:36.771 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:55.743329+0000", 2024-09-21T14:01:36.771 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.10425733 2024-09-21T14:01:36.771 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.771 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.771 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:01:36.771 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:55.743390+0000", 2024-09-21T14:01:36.771 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1204000000000006e-05 2024-09-21T14:01:36.771 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.772 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.772 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:01:36.772 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:55.743436+0000", 2024-09-21T14:01:36.772 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6372000000000003e-05 2024-09-21T14:01:36.772 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:01:36.772 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:01:36.772 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:01:36.772 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.772 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.772 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.15207.0:232030 8.2 8:49ac6d95:::benchmark_data_smithi049_109311_object105511:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T14:01:36.772 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:57:11.653701+0000", 2024-09-21T14:01:36.772 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 265.06819653999997, 2024-09-21T14:01:36.772 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.22905246600000001, 2024-09-21T14:01:36.773 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:01:36.773 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:01:36.773 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:01:36.773 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:01:36.773 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.15207", 2024-09-21T14:01:36.773 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:01:36.773 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 232030 2024-09-21T14:01:36.773 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.773 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:01:36.773 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.773 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:01:36.773 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.653701+0000", 2024-09-21T14:01:36.773 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:01:36.774 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.774 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.774 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:01:36.774 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.653701+0000", 2024-09-21T14:01:36.774 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:01:36.774 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.774 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.774 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:01:36.774 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.653702+0000", 2024-09-21T14:01:36.774 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3600000000000002e-07 2024-09-21T14:01:36.774 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.774 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.775 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:01:36.775 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.653706+0000", 2024-09-21T14:01:36.775 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6490000000000001e-06 2024-09-21T14:01:36.775 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.775 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.775 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:01:36.775 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.653716+0000", 2024-09-21T14:01:36.775 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0509999999999999e-05 2024-09-21T14:01:36.775 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.775 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.775 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:01:36.775 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.653721+0000", 2024-09-21T14:01:36.775 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1279999999999999e-06 2024-09-21T14:01:36.775 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.775 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.776 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:01:36.776 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.653764+0000", 2024-09-21T14:01:36.776 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2531999999999999e-05 2024-09-21T14:01:36.776 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.776 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.776 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:01:36.776 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.653848+0000", 2024-09-21T14:01:36.776 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4091000000000005e-05 2024-09-21T14:01:36.776 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.776 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.776 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,5", 2024-09-21T14:01:36.776 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.653914+0000", 2024-09-21T14:01:36.776 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5977999999999997e-05 2024-09-21T14:01:36.777 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.777 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.777 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:01:36.777 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.656207+0000", 2024-09-21T14:01:36.777 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0022932690000000001 2024-09-21T14:01:36.777 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.777 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.777 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:01:36.777 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.659356+0000", 2024-09-21T14:01:36.777 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0031493160000000001 2024-09-21T14:01:36.777 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.777 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.777 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:01:36.777 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.882645+0000", 2024-09-21T14:01:36.778 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.22328900700000001 2024-09-21T14:01:36.778 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.778 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.778 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:01:36.778 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.882706+0000", 2024-09-21T14:01:36.778 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0279999999999999e-05 2024-09-21T14:01:36.778 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.778 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.778 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:01:36.778 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.882754+0000", 2024-09-21T14:01:36.778 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7870000000000001e-05 2024-09-21T14:01:36.778 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:01:36.778 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:01:36.778 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:01:36.779 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.779 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.779 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.15207.0:232036 8.2 8:4507ec93:::benchmark_data_smithi049_109311_object105517:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T14:01:36.779 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:57:11.656238+0000", 2024-09-21T14:01:36.779 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 265.065659611, 2024-09-21T14:01:36.779 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.22665437799999999, 2024-09-21T14:01:36.779 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:01:36.779 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:01:36.779 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:01:36.779 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:01:36.779 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.15207", 2024-09-21T14:01:36.779 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:01:36.779 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 232036 2024-09-21T14:01:36.779 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.780 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:01:36.780 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.780 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:01:36.780 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.656238+0000", 2024-09-21T14:01:36.780 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:01:36.780 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.780 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.780 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:01:36.780 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.656238+0000", 2024-09-21T14:01:36.780 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:01:36.780 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.780 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.780 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:01:36.780 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.656239+0000", 2024-09-21T14:01:36.780 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9100000000000003e-07 2024-09-21T14:01:36.781 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.781 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.781 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:01:36.781 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.656243+0000", 2024-09-21T14:01:36.781 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1230000000000002e-06 2024-09-21T14:01:36.781 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.781 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.781 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:01:36.781 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.656252+0000", 2024-09-21T14:01:36.781 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7490000000000004e-06 2024-09-21T14:01:36.781 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.781 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.781 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:01:36.781 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.656259+0000", 2024-09-21T14:01:36.782 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7559999999999999e-06 2024-09-21T14:01:36.782 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.782 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.782 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:01:36.782 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.656310+0000", 2024-09-21T14:01:36.782 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0203e-05 2024-09-21T14:01:36.782 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.782 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.782 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:01:36.782 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.656440+0000", 2024-09-21T14:01:36.782 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000130376 2024-09-21T14:01:36.782 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.782 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.783 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,5", 2024-09-21T14:01:36.783 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.656550+0000", 2024-09-21T14:01:36.783 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000109812 2024-09-21T14:01:36.783 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.783 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.783 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:01:36.783 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.658548+0000", 2024-09-21T14:01:36.783 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0019982649999999999 2024-09-21T14:01:36.783 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.783 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.783 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:01:36.783 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.659437+0000", 2024-09-21T14:01:36.783 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000888743 2024-09-21T14:01:36.783 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.784 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.784 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:01:36.784 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.882792+0000", 2024-09-21T14:01:36.784 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.22335466200000001 2024-09-21T14:01:36.784 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.784 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.784 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:01:36.784 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.882858+0000", 2024-09-21T14:01:36.784 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6512000000000003e-05 2024-09-21T14:01:36.784 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.784 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.784 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:01:36.784 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.882892+0000", 2024-09-21T14:01:36.785 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4385999999999999e-05 2024-09-21T14:01:36.785 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:01:36.785 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:01:36.785 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:01:36.785 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.785 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.785 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.15207.0:232041 8.2 8:42fefff6:::benchmark_data_smithi049_109311_object105522:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T14:01:36.785 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:57:11.674073+0000", 2024-09-21T14:01:36.785 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 265.047824843, 2024-09-21T14:01:36.785 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21533840400000001, 2024-09-21T14:01:36.785 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:01:36.785 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:01:36.785 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:01:36.786 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:01:36.786 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.15207", 2024-09-21T14:01:36.786 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:01:36.786 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 232041 2024-09-21T14:01:36.786 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.786 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:01:36.786 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.786 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:01:36.786 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.674073+0000", 2024-09-21T14:01:36.786 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:01:36.786 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.786 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.786 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:01:36.786 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.674073+0000", 2024-09-21T14:01:36.787 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:01:36.787 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.787 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.787 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:01:36.787 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.674074+0000", 2024-09-21T14:01:36.787 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5889999999999999e-06 2024-09-21T14:01:36.787 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.787 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.787 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:01:36.787 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.674080+0000", 2024-09-21T14:01:36.787 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1290000000000004e-06 2024-09-21T14:01:36.787 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.787 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.787 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:01:36.788 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.674102+0000", 2024-09-21T14:01:36.788 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2541e-05 2024-09-21T14:01:36.788 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.788 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.788 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:01:36.788 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.674112+0000", 2024-09-21T14:01:36.788 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.5310000000000007e-06 2024-09-21T14:01:36.788 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.788 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.788 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:01:36.788 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.674234+0000", 2024-09-21T14:01:36.788 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012247000000000001 2024-09-21T14:01:36.788 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.788 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.789 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:01:36.789 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.674365+0000", 2024-09-21T14:01:36.789 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013083300000000001 2024-09-21T14:01:36.789 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.789 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.789 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,5", 2024-09-21T14:01:36.789 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.674489+0000", 2024-09-21T14:01:36.789 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000124079 2024-09-21T14:01:36.789 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.789 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.789 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:01:36.789 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.680300+0000", 2024-09-21T14:01:36.789 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0058106709999999999 2024-09-21T14:01:36.789 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.790 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.790 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:01:36.790 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.681669+0000", 2024-09-21T14:01:36.790 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0013690460000000001 2024-09-21T14:01:36.790 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.790 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.790 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:01:36.790 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.889300+0000", 2024-09-21T14:01:36.790 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20763156999999999 2024-09-21T14:01:36.790 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.790 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.790 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:01:36.790 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.889364+0000", 2024-09-21T14:01:36.790 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3979000000000005e-05 2024-09-21T14:01:36.791 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.791 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.791 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:01:36.791 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.889411+0000", 2024-09-21T14:01:36.791 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6965999999999997e-05 2024-09-21T14:01:36.791 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:01:36.791 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:01:36.791 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:01:36.791 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.791 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.791 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.15207.0:232049 8.5 8:aa476d15:::benchmark_data_smithi049_109311_object105530:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T14:01:36.791 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:57:11.686170+0000", 2024-09-21T14:01:36.791 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 265.035727882, 2024-09-21T14:01:36.791 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20563606700000001, 2024-09-21T14:01:36.791 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:01:36.792 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:01:36.792 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:01:36.792 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:01:36.792 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.15207", 2024-09-21T14:01:36.792 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:01:36.792 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 232049 2024-09-21T14:01:36.792 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.792 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:01:36.792 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.792 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:01:36.792 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.686170+0000", 2024-09-21T14:01:36.792 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:01:36.792 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.792 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.793 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:01:36.793 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.686170+0000", 2024-09-21T14:01:36.793 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:01:36.793 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.793 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.793 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:01:36.793 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.686172+0000", 2024-09-21T14:01:36.793 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7710000000000001e-06 2024-09-21T14:01:36.793 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.793 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.793 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:01:36.793 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.686175+0000", 2024-09-21T14:01:36.793 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6239999999999999e-06 2024-09-21T14:01:36.793 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.793 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.794 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:01:36.794 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.686185+0000", 2024-09-21T14:01:36.794 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6909999999999992e-06 2024-09-21T14:01:36.794 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.794 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.794 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:01:36.794 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.686194+0000", 2024-09-21T14:01:36.794 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0769999999999997e-06 2024-09-21T14:01:36.794 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.794 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.794 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:01:36.794 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.686258+0000", 2024-09-21T14:01:36.794 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4239000000000006e-05 2024-09-21T14:01:36.795 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.795 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.795 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:01:36.795 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.686374+0000", 2024-09-21T14:01:36.795 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000116148 2024-09-21T14:01:36.795 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.795 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.795 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:01:36.795 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.686491+0000", 2024-09-21T14:01:36.795 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011667000000000001 2024-09-21T14:01:36.795 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.795 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.795 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:01:36.795 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.690120+0000", 2024-09-21T14:01:36.795 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003628797 2024-09-21T14:01:36.796 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.796 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.796 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:01:36.796 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.690188+0000", 2024-09-21T14:01:36.796 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8201000000000007e-05 2024-09-21T14:01:36.796 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.796 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.796 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:01:36.796 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.891730+0000", 2024-09-21T14:01:36.796 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20154166900000001 2024-09-21T14:01:36.796 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.796 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.796 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:01:36.796 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.891769+0000", 2024-09-21T14:01:36.796 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9409999999999997e-05 2024-09-21T14:01:36.796 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.797 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.797 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:01:36.797 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.891806+0000", 2024-09-21T14:01:36.797 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6770000000000002e-05 2024-09-21T14:01:36.797 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:01:36.797 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:01:36.797 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:01:36.797 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.797 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.797 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.15207.0:232052 8.5 8:aec88863:::benchmark_data_smithi049_109311_object105533:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T14:01:36.797 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:57:11.686395+0000", 2024-09-21T14:01:36.797 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 265.03550264, 2024-09-21T14:01:36.797 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20789333199999999, 2024-09-21T14:01:36.797 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:01:36.797 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:01:36.798 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:01:36.798 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:01:36.798 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.15207", 2024-09-21T14:01:36.798 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:01:36.798 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 232052 2024-09-21T14:01:36.798 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.798 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:01:36.798 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.798 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:01:36.798 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.686395+0000", 2024-09-21T14:01:36.798 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:01:36.798 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.798 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.799 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:01:36.799 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.686395+0000", 2024-09-21T14:01:36.799 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:01:36.799 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.799 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.799 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:01:36.799 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.686396+0000", 2024-09-21T14:01:36.799 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4600000000000004e-07 2024-09-21T14:01:36.799 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.799 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.799 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:01:36.799 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.686399+0000", 2024-09-21T14:01:36.799 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8329999999999998e-06 2024-09-21T14:01:36.799 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.800 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.800 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:01:36.800 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.686404+0000", 2024-09-21T14:01:36.800 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8220000000000003e-06 2024-09-21T14:01:36.800 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.800 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.800 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:01:36.800 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.686408+0000", 2024-09-21T14:01:36.800 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0040000000000001e-06 2024-09-21T14:01:36.800 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.800 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.800 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:01:36.801 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.688420+0000", 2024-09-21T14:01:36.801 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.002011896 2024-09-21T14:01:36.801 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.801 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.801 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:01:36.801 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.688526+0000", 2024-09-21T14:01:36.801 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000106031 2024-09-21T14:01:36.801 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.801 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.801 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:01:36.801 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.688626+0000", 2024-09-21T14:01:36.801 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000100185 2024-09-21T14:01:36.801 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.802 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.802 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:01:36.802 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.690125+0000", 2024-09-21T14:01:36.802 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0014986820000000001 2024-09-21T14:01:36.802 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.802 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.802 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:01:36.802 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.690262+0000", 2024-09-21T14:01:36.802 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000136444 2024-09-21T14:01:36.802 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.802 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.802 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:01:36.802 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.894191+0000", 2024-09-21T14:01:36.802 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.203929104 2024-09-21T14:01:36.803 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.803 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.803 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:01:36.803 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.894251+0000", 2024-09-21T14:01:36.803 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9868999999999999e-05 2024-09-21T14:01:36.803 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.803 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.803 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:01:36.803 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.894288+0000", 2024-09-21T14:01:36.803 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7815999999999998e-05 2024-09-21T14:01:36.803 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:01:36.803 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:01:36.803 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:01:36.803 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.804 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.804 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.15207.0:232055 8.2 8:459633e6:::benchmark_data_smithi049_109311_object105536:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T14:01:36.804 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:57:11.693418+0000", 2024-09-21T14:01:36.804 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 265.02847998700003, 2024-09-21T14:01:36.804 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.19827633, 2024-09-21T14:01:36.804 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:01:36.804 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:01:36.804 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:01:36.804 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:01:36.804 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.15207", 2024-09-21T14:01:36.804 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:01:36.804 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 232055 2024-09-21T14:01:36.804 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.804 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:01:36.805 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.805 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:01:36.805 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.693418+0000", 2024-09-21T14:01:36.805 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:01:36.805 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.805 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.805 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:01:36.805 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.693418+0000", 2024-09-21T14:01:36.805 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:01:36.805 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.805 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.805 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:01:36.805 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.693419+0000", 2024-09-21T14:01:36.805 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2100000000000001e-06 2024-09-21T14:01:36.805 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.805 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.806 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:01:36.806 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.693423+0000", 2024-09-21T14:01:36.806 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4669999999999998e-06 2024-09-21T14:01:36.806 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.806 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.806 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:01:36.806 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.693433+0000", 2024-09-21T14:01:36.806 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.8120000000000002e-06 2024-09-21T14:01:36.806 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.806 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.806 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:01:36.806 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.693438+0000", 2024-09-21T14:01:36.806 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2660000000000001e-06 2024-09-21T14:01:36.807 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.807 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.807 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:01:36.807 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.693476+0000", 2024-09-21T14:01:36.807 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7276e-05 2024-09-21T14:01:36.807 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.807 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.807 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:01:36.807 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.693549+0000", 2024-09-21T14:01:36.807 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3133999999999995e-05 2024-09-21T14:01:36.807 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.807 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.807 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,5", 2024-09-21T14:01:36.807 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.693612+0000", 2024-09-21T14:01:36.807 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3392000000000006e-05 2024-09-21T14:01:36.808 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.808 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.808 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:01:36.808 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.695497+0000", 2024-09-21T14:01:36.808 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0018845310000000001 2024-09-21T14:01:36.808 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.808 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.808 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:01:36.808 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.695811+0000", 2024-09-21T14:01:36.808 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00031459700000000001 2024-09-21T14:01:36.808 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.808 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.808 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:01:36.808 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.891590+0000", 2024-09-21T14:01:36.809 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.19577844999999999 2024-09-21T14:01:36.809 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.809 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.809 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:01:36.809 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.891645+0000", 2024-09-21T14:01:36.809 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4889999999999998e-05 2024-09-21T14:01:36.809 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.809 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.809 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:01:36.809 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.891694+0000", 2024-09-21T14:01:36.809 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9305000000000003e-05 2024-09-21T14:01:36.809 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:01:36.809 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:01:36.809 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:01:36.809 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.810 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.810 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:73960 9.e 9:7e61c187:::benchmark_data_smithi049_138504_object73959:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-09-21T14:01:36.810 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:58:52.916953+0000", 2024-09-21T14:01:36.810 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 163.80494513299999, 2024-09-21T14:01:36.810 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.12133540700000001, 2024-09-21T14:01:36.810 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:01:36.810 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:01:36.810 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:01:36.810 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:01:36.810 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:01:36.810 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:01:36.810 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 73960 2024-09-21T14:01:36.810 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.810 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:01:36.811 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.811 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:01:36.811 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:52.916953+0000", 2024-09-21T14:01:36.811 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:01:36.811 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.812 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.812 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:01:36.812 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:52.916953+0000", 2024-09-21T14:01:36.812 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:01:36.812 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.812 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.812 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:01:36.812 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:52.916964+0000", 2024-09-21T14:01:36.812 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1386000000000001e-05 2024-09-21T14:01:36.812 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.812 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.812 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:01:36.812 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:52.917074+0000", 2024-09-21T14:01:36.812 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010969299999999999 2024-09-21T14:01:36.813 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.813 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.813 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:01:36.813 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:52.917094+0000", 2024-09-21T14:01:36.813 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0251e-05 2024-09-21T14:01:36.813 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.813 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.813 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:01:36.813 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:52.917102+0000", 2024-09-21T14:01:36.813 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3000000000000002e-06 2024-09-21T14:01:36.813 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.813 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.813 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:01:36.813 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:52.917160+0000", 2024-09-21T14:01:36.814 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7405999999999999e-05 2024-09-21T14:01:36.814 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.814 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.814 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:01:36.814 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:52.917308+0000", 2024-09-21T14:01:36.814 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000148085 2024-09-21T14:01:36.814 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.814 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.814 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1", 2024-09-21T14:01:36.814 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:52.917467+0000", 2024-09-21T14:01:36.814 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015944200000000001 2024-09-21T14:01:36.814 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.814 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.815 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:01:36.815 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:52.919847+0000", 2024-09-21T14:01:36.815 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0023794250000000001 2024-09-21T14:01:36.815 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.815 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.815 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:01:36.815 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:53.038132+0000", 2024-09-21T14:01:36.815 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.11828538099999999 2024-09-21T14:01:36.815 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.815 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.815 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:01:36.815 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:53.038197+0000", 2024-09-21T14:01:36.815 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4744000000000002e-05 2024-09-21T14:01:36.816 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.816 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.816 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:01:36.816 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:53.038288+0000", 2024-09-21T14:01:36.816 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1293999999999997e-05 2024-09-21T14:01:36.816 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:01:36.816 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:01:36.816 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:01:36.816 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.816 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.816 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:73970 9.e 9:74881f58:::benchmark_data_smithi049_138504_object73969:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-09-21T14:01:36.816 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:58:52.924967+0000", 2024-09-21T14:01:36.817 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 163.79693099900001, 2024-09-21T14:01:36.817 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.113448892, 2024-09-21T14:01:36.817 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:01:36.817 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:01:36.817 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:01:36.817 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:01:36.817 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:01:36.817 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:01:36.817 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 73970 2024-09-21T14:01:36.817 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.817 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:01:36.817 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.817 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:01:36.818 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:52.924967+0000", 2024-09-21T14:01:36.818 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:01:36.818 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.818 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.818 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:01:36.818 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:52.924967+0000", 2024-09-21T14:01:36.818 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:01:36.818 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.818 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.818 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:01:36.818 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:52.924969+0000", 2024-09-21T14:01:36.818 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.03e-06 2024-09-21T14:01:36.818 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.819 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.819 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:01:36.819 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:52.925016+0000", 2024-09-21T14:01:36.819 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7216000000000003e-05 2024-09-21T14:01:36.819 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.819 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.819 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:01:36.819 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:52.925033+0000", 2024-09-21T14:01:36.819 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7079e-05 2024-09-21T14:01:36.819 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.819 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.819 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:01:36.819 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:52.925041+0000", 2024-09-21T14:01:36.820 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.106e-06 2024-09-21T14:01:36.820 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.820 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.820 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:01:36.820 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:52.925108+0000", 2024-09-21T14:01:36.820 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7175999999999995e-05 2024-09-21T14:01:36.820 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.820 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.820 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:01:36.820 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:52.925245+0000", 2024-09-21T14:01:36.820 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013656700000000001 2024-09-21T14:01:36.820 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.820 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.821 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1", 2024-09-21T14:01:36.821 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:52.925380+0000", 2024-09-21T14:01:36.821 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013513199999999999 2024-09-21T14:01:36.821 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.821 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.821 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:01:36.821 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:52.927803+0000", 2024-09-21T14:01:36.821 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0024229949999999998 2024-09-21T14:01:36.821 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.821 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.821 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:01:36.821 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:53.038326+0000", 2024-09-21T14:01:36.821 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.11052324700000001 2024-09-21T14:01:36.821 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.822 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.822 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:01:36.822 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:53.038366+0000", 2024-09-21T14:01:36.822 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9715999999999997e-05 2024-09-21T14:01:36.822 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.822 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.822 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:01:36.822 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:53.038416+0000", 2024-09-21T14:01:36.822 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9628e-05 2024-09-21T14:01:36.822 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:01:36.822 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:01:36.822 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:01:36.822 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.823 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.823 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25421.0:179024 9.4 e320/260 9:208b33bc:::benchmark_data_smithi049_138504_object57616:head v 320'10951, mlcod=320'10948)", 2024-09-21T14:01:36.823 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:00:40.673314+0000", 2024-09-21T14:01:36.823 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 56.048583770999997, 2024-09-21T14:01:36.823 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.017150953, 2024-09-21T14:01:36.823 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:01:36.823 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:01:36.823 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:01:36.823 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:01:36.823 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.823 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:01:36.823 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:40.673314+0000", 2024-09-21T14:01:36.823 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:01:36.823 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.824 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.824 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:01:36.824 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:40.673314+0000", 2024-09-21T14:01:36.824 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:01:36.824 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.824 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.824 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:01:36.824 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:40.673314+0000", 2024-09-21T14:01:36.824 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3200000000000005e-07 2024-09-21T14:01:36.824 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.824 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.824 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:01:36.824 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:40.673319+0000", 2024-09-21T14:01:36.824 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1269999999999996e-06 2024-09-21T14:01:36.825 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.825 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.825 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:01:36.825 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:40.673326+0000", 2024-09-21T14:01:36.825 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8900000000000007e-06 2024-09-21T14:01:36.825 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.825 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.825 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:01:36.825 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:40.673331+0000", 2024-09-21T14:01:36.825 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.51e-06 2024-09-21T14:01:36.825 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.825 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.825 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:01:36.825 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:40.680489+0000", 2024-09-21T14:01:36.825 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0071584609999999996 2024-09-21T14:01:36.826 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.826 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:01:36 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5[125175]: 2024-09-21T14:01:36.715+0000 7f29ed132640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _main_bdev_label_try_reserve bdev label location 0x280000000 occupied by BlueStore object or BlueFS file, disabling 2024-09-21T14:01:36.826 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.826 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:01:36.826 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:40.680533+0000", 2024-09-21T14:01:36.827 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3909000000000003e-05 2024-09-21T14:01:36.827 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.827 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.827 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:01:36.827 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:40.690432+0000", 2024-09-21T14:01:36.827 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0098982410000000003 2024-09-21T14:01:36.827 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.827 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.827 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:01:36.827 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:40.690465+0000", 2024-09-21T14:01:36.827 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3283e-05 2024-09-21T14:01:36.827 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:01:36.827 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:01:36.827 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:01:36.828 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.828 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.828 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25421.0:179026 9.4 e320/260 9:256dedb2:::benchmark_data_smithi049_138504_object57618:head v 320'10952, mlcod=320'10948)", 2024-09-21T14:01:36.828 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:00:40.676282+0000", 2024-09-21T14:01:36.828 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 56.045615976999997, 2024-09-21T14:01:36.828 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.016241876999999998, 2024-09-21T14:01:36.828 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:01:36.828 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:01:36.828 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:01:36.828 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:01:36.828 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.828 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:01:36.828 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:40.676282+0000", 2024-09-21T14:01:36.828 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:01:36.829 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.829 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.829 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:01:36.829 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:40.676282+0000", 2024-09-21T14:01:36.829 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:01:36.829 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.829 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.829 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:01:36.829 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:40.676282+0000", 2024-09-21T14:01:36.829 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7100000000000001e-07 2024-09-21T14:01:36.829 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.829 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.829 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:01:36.829 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:40.676289+0000", 2024-09-21T14:01:36.830 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0650000000000004e-06 2024-09-21T14:01:36.830 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.830 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.830 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:01:36.830 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:40.676298+0000", 2024-09-21T14:01:36.830 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.9219999999999992e-06 2024-09-21T14:01:36.830 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.830 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.830 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:01:36.830 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:40.676305+0000", 2024-09-21T14:01:36.830 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9129999999999997e-06 2024-09-21T14:01:36.830 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.830 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.830 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:01:36.831 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:40.687935+0000", 2024-09-21T14:01:36.831 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.011629168 2024-09-21T14:01:36.831 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.831 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.831 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:01:36.831 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:40.687956+0000", 2024-09-21T14:01:36.831 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0988999999999998e-05 2024-09-21T14:01:36.831 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.831 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.831 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:01:36.831 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:40.692390+0000", 2024-09-21T14:01:36.831 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.004434632 2024-09-21T14:01:36.831 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.831 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.832 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:01:36.832 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:40.692524+0000", 2024-09-21T14:01:36.832 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013341699999999999 2024-09-21T14:01:36.832 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:01:36.832 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:01:36.832 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:01:36.832 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.832 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.832 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25421.0:179468 9.e e322/321 9:71f026da:::benchmark_data_smithi049_138504_object58060:head v 322'10951, mlcod=322'10948)", 2024-09-21T14:01:36.832 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:00:42.169017+0000", 2024-09-21T14:01:36.832 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 54.552880406, 2024-09-21T14:01:36.832 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.014278903000000001, 2024-09-21T14:01:36.832 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:01:36.832 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:01:36.833 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:01:36.833 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:01:36.833 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.833 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:01:36.833 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:42.169017+0000", 2024-09-21T14:01:36.833 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:01:36.833 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.833 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.833 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:01:36.833 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:42.169017+0000", 2024-09-21T14:01:36.833 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:01:36.833 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.833 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.833 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:01:36.834 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:42.169019+0000", 2024-09-21T14:01:36.834 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5460000000000001e-06 2024-09-21T14:01:36.834 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.834 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.834 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:01:36.834 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:42.169027+0000", 2024-09-21T14:01:36.834 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4959999999999996e-06 2024-09-21T14:01:36.834 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.834 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.834 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:01:36.834 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:42.169042+0000", 2024-09-21T14:01:36.834 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4745e-05 2024-09-21T14:01:36.834 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.834 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.835 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:01:36.835 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:42.169050+0000", 2024-09-21T14:01:36.835 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4149999999999999e-06 2024-09-21T14:01:36.835 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.835 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.835 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:01:36.835 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:42.169097+0000", 2024-09-21T14:01:36.835 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6653000000000002e-05 2024-09-21T14:01:36.835 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.835 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.835 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:01:36.835 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:42.169118+0000", 2024-09-21T14:01:36.835 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.056e-05 2024-09-21T14:01:36.835 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.836 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.836 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:01:36.836 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:42.183242+0000", 2024-09-21T14:01:36.836 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.014124121999999999 2024-09-21T14:01:36.836 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.836 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.836 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:01:36.836 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:42.183296+0000", 2024-09-21T14:01:36.836 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4366000000000001e-05 2024-09-21T14:01:36.836 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:01:36.836 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:01:36.836 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:01:36.836 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.836 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.836 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25421.0:182211 9.c e322/299 9:3f4e9b88:::benchmark_data_smithi049_138504_object60803:head v 322'11290, mlcod=322'11288)", 2024-09-21T14:01:36.837 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:00:44.972654+0000", 2024-09-21T14:01:36.837 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 51.749243493000002, 2024-09-21T14:01:36.837 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.019121724999999999, 2024-09-21T14:01:36.837 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:01:36.837 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:01:36.837 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:01:36.837 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:01:36.837 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.837 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:01:36.837 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:44.972654+0000", 2024-09-21T14:01:36.837 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:01:36.837 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.837 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.837 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:01:36.838 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:44.972654+0000", 2024-09-21T14:01:36.838 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:01:36.838 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.838 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.838 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:01:36.838 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:44.972656+0000", 2024-09-21T14:01:36.838 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6980000000000001e-06 2024-09-21T14:01:36.838 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.838 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.838 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:01:36.838 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:44.972665+0000", 2024-09-21T14:01:36.838 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.5610000000000004e-06 2024-09-21T14:01:36.838 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.838 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.839 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:01:36.839 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:44.972685+0000", 2024-09-21T14:01:36.839 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9797999999999999e-05 2024-09-21T14:01:36.839 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.839 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.839 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:01:36.839 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:44.972694+0000", 2024-09-21T14:01:36.839 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.9320000000000002e-06 2024-09-21T14:01:36.839 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.839 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.839 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:01:36.839 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:44.972752+0000", 2024-09-21T14:01:36.839 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8162999999999999e-05 2024-09-21T14:01:36.839 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.839 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.840 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:01:36.840 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:44.972775+0000", 2024-09-21T14:01:36.840 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2578999999999998e-05 2024-09-21T14:01:36.840 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.840 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.840 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:01:36.840 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:44.991246+0000", 2024-09-21T14:01:36.840 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.018470751000000001 2024-09-21T14:01:36.840 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.840 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.840 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:01:36.840 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:44.991776+0000", 2024-09-21T14:01:36.840 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00053024300000000001 2024-09-21T14:01:36.840 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:01:36.840 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:01:36.840 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:01:36.841 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.841 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.841 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25421.0:182457 9.c e322/299 9:38e70577:::benchmark_data_smithi049_138504_object61049:head v 322'11307, mlcod=322'11305)", 2024-09-21T14:01:36.841 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:00:45.237016+0000", 2024-09-21T14:01:36.841 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 51.48488176, 2024-09-21T14:01:36.841 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.014243393999999999, 2024-09-21T14:01:36.841 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:01:36.841 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:01:36.841 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:01:36.841 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:01:36.841 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.841 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:01:36.841 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:45.237016+0000", 2024-09-21T14:01:36.842 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:01:36.842 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.842 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.842 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:01:36.842 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:45.237016+0000", 2024-09-21T14:01:36.842 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:01:36.842 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.842 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.842 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:01:36.842 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:45.237018+0000", 2024-09-21T14:01:36.842 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.787e-06 2024-09-21T14:01:36.842 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.842 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.843 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:01:36.843 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:45.237027+0000", 2024-09-21T14:01:36.843 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1819999999999995e-06 2024-09-21T14:01:36.843 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.843 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.843 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:01:36.843 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:45.237047+0000", 2024-09-21T14:01:36.843 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0097000000000001e-05 2024-09-21T14:01:36.843 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.843 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.843 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:01:36.843 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:45.237056+0000", 2024-09-21T14:01:36.843 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5900000000000008e-06 2024-09-21T14:01:36.843 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.844 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.844 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:01:36.844 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:45.237125+0000", 2024-09-21T14:01:36.844 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9023000000000006e-05 2024-09-21T14:01:36.844 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.844 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.844 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:01:36.844 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:45.237151+0000", 2024-09-21T14:01:36.844 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.633e-05 2024-09-21T14:01:36.844 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.844 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.844 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:01:36.844 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:45.251203+0000", 2024-09-21T14:01:36.844 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.014052386 2024-09-21T14:01:36.844 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:01:36.845 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:01:36.845 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:01:36.845 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:00:45.251259+0000", 2024-09-21T14:01:36.845 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5998999999999997e-05 2024-09-21T14:01:36.845 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:01:36.845 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:01:36.845 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:01:36.845 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:01:36.845 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:01:36.845 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T14:01:36.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:36.847 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:36.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:36.914 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:01:36.947 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:36.972 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:36.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:36.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:37.074 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:37.102 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:37.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:37.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:37.180 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T14:01:37.204 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:01:37.245 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:37.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:37.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:01:37.346 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:37.372 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:37.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:37.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:37.473 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:37.499 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:37.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:37.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:37.531 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:01:37.600 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:37.623 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:37.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:37.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:37.665 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-09-21T14:01:37.698 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T14:01:37.725 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:01:37.749 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:37.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:37.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:01:37.850 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:37.873 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:37.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:37.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:37.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:37 smithi187 ceph-mon[33032]: pgmap v635: 158 pgs: 1 undersized+degraded+remapped+backfilling+peered, 1 active+clean+remapped, 2 active+recovering+undersized+degraded, 35 active+undersized, 22 active+undersized+degraded, 97 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 39074/166365 objects degraded (23.487%); 5.7 MiB/s, 91 objects/s recovering 2024-09-21T14:01:37.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:37 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/786977412' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:01:37.945 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:01:37 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5[125175]: 2024-09-21T14:01:37.653+0000 7f29ed132640 -1 osd.5 324 log_to_monitors true 2024-09-21T14:01:37.974 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:37.991 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T14:01:37.993 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:01:38.000 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:38.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:38.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:38.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:37 smithi049 ceph-mon[30080]: pgmap v635: 158 pgs: 1 undersized+degraded+remapped+backfilling+peered, 1 active+clean+remapped, 2 active+recovering+undersized+degraded, 35 active+undersized, 22 active+undersized+degraded, 97 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 39074/166365 objects degraded (23.487%); 5.7 MiB/s, 91 objects/s recovering 2024-09-21T14:01:38.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:37 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/786977412' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:01:38.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:37 smithi049 ceph-mon[40489]: pgmap v635: 158 pgs: 1 undersized+degraded+remapped+backfilling+peered, 1 active+clean+remapped, 2 active+recovering+undersized+degraded, 35 active+undersized, 22 active+undersized+degraded, 97 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 39074/166365 objects degraded (23.487%); 5.7 MiB/s, 91 objects/s recovering 2024-09-21T14:01:38.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:37 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/786977412' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:01:38.102 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:38.124 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:38.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:38.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:38.226 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:01:38.248 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:38.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:38.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:01:38.350 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:01:38.372 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:38.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:38.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:01:38.474 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:38.498 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:38.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:38.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:38.541 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T14:01:38.542 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-21T14:01:38.542 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 0 2024-09-21T14:01:38.542 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T14:01:38.600 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:38.624 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:38.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:38.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:38.725 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:38.751 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:38.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:38.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:38.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:38 smithi049 ceph-mon[30080]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:01:38.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:38 smithi049 ceph-mon[30080]: from='osd.5 [v2:172.21.15.187:6808/1595353965,v1:172.21.15.187:6809/1595353965]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T14:01:38.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:38 smithi049 ceph-mon[30080]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T14:01:38.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:38 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/786977412' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T14:01:38.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:38 smithi049 ceph-mon[30080]: osdmap e343: 8 total, 7 up, 8 in 2024-09-21T14:01:38.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:38 smithi049 ceph-mon[40489]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:01:38.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:38 smithi049 ceph-mon[40489]: from='osd.5 [v2:172.21.15.187:6808/1595353965,v1:172.21.15.187:6809/1595353965]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T14:01:38.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:38 smithi049 ceph-mon[40489]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T14:01:38.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:38 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/786977412' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T14:01:38.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:38 smithi049 ceph-mon[40489]: osdmap e343: 8 total, 7 up, 8 in 2024-09-21T14:01:38.797 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:01:38.853 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:38.877 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:38.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:38.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:38.899 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:01:38.899 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:01:38.904 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:38 smithi187 ceph-mon[33032]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:01:38.904 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:38 smithi187 ceph-mon[33032]: from='osd.5 [v2:172.21.15.187:6808/1595353965,v1:172.21.15.187:6809/1595353965]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T14:01:38.904 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:38 smithi187 ceph-mon[33032]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T14:01:38.905 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:38 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/786977412' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T14:01:38.905 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:38 smithi187 ceph-mon[33032]: osdmap e343: 8 total, 7 up, 8 in 2024-09-21T14:01:38.913 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:01:38.913 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:01:38.929 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:01:38.929 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:01:38.945 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:01:38.945 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:01:38.962 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:01:38.963 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:01:38.963 INFO:teuthology.orchestra.run.smithi049.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-21T14:01:38.980 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:38.982 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:01:38.982 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:01:38.997 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:01:38.997 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:01:39.004 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:39.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:39.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:39.106 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:01:39.128 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:39.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:39.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:01:39.230 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:39.258 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:39.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:39.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:39.260 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T14:01:39.360 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:01:39.382 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:39.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:39.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:01:39.414 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:01:39 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5[125175]: 2024-09-21T14:01:39.163+0000 7f29e0ed4640 -1 osd.5 324 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T14:01:39.483 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:39.509 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:39.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:39.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:39.611 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:39.640 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:39.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:39.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:39.742 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:39.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:39 smithi049 ceph-mon[30080]: pgmap v637: 158 pgs: 1 active+clean+remapped, 1 undersized+degraded+remapped+backfilling+peered, 2 active+recovering+undersized+degraded, 35 active+undersized, 22 active+undersized+degraded, 97 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 38710/166365 objects degraded (23.268%); 5.8 MiB/s, 92 objects/s recovering 2024-09-21T14:01:39.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:39 smithi049 ceph-mon[30080]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-21T14:01:39.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:39 smithi049 ceph-mon[30080]: osdmap e344: 8 total, 7 up, 8 in 2024-09-21T14:01:39.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:39 smithi049 ceph-mon[30080]: from='osd.5 [v2:172.21.15.187:6808/1595353965,v1:172.21.15.187:6809/1595353965]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-21T14:01:39.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:39 smithi049 ceph-mon[30080]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-21T14:01:39.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:39 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2086262057' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:01:39.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:39 smithi049 ceph-mon[30080]: from='osd.5 ' entity='osd.5' 2024-09-21T14:01:39.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:39 smithi049 ceph-mon[40489]: pgmap v637: 158 pgs: 1 active+clean+remapped, 1 undersized+degraded+remapped+backfilling+peered, 2 active+recovering+undersized+degraded, 35 active+undersized, 22 active+undersized+degraded, 97 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 38710/166365 objects degraded (23.268%); 5.8 MiB/s, 92 objects/s recovering 2024-09-21T14:01:39.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:39 smithi049 ceph-mon[40489]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-21T14:01:39.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:39 smithi049 ceph-mon[40489]: osdmap e344: 8 total, 7 up, 8 in 2024-09-21T14:01:39.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:39 smithi049 ceph-mon[40489]: from='osd.5 [v2:172.21.15.187:6808/1595353965,v1:172.21.15.187:6809/1595353965]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-21T14:01:39.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:39 smithi049 ceph-mon[40489]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-21T14:01:39.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:39 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2086262057' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:01:39.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:39 smithi049 ceph-mon[40489]: from='osd.5 ' entity='osd.5' 2024-09-21T14:01:39.764 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:39.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:39.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:39.866 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:39.889 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:39.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:39.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:39.944 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:01:39 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:01:39.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:39 smithi187 ceph-mon[33032]: pgmap v637: 158 pgs: 1 active+clean+remapped, 1 undersized+degraded+remapped+backfilling+peered, 2 active+recovering+undersized+degraded, 35 active+undersized, 22 active+undersized+degraded, 97 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 38710/166365 objects degraded (23.268%); 5.8 MiB/s, 92 objects/s recovering 2024-09-21T14:01:39.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:39 smithi187 ceph-mon[33032]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-21T14:01:39.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:39 smithi187 ceph-mon[33032]: osdmap e344: 8 total, 7 up, 8 in 2024-09-21T14:01:39.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:39 smithi187 ceph-mon[33032]: from='osd.5 [v2:172.21.15.187:6808/1595353965,v1:172.21.15.187:6809/1595353965]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-21T14:01:39.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:39 smithi187 ceph-mon[33032]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-21T14:01:39.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:39 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2086262057' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:01:39.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:39 smithi187 ceph-mon[33032]: from='osd.5 ' entity='osd.5' 2024-09-21T14:01:39.959 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T14:01:39.959 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T14:01:39.991 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:40.014 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:40.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:40.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:40.115 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:40.126 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T14:01:40.126 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-21T14:01:40.126 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 0 2024-09-21T14:01:40.126 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T14:01:40.138 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:40.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:40.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:40.240 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:40.261 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:40.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:40.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:40.357 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:01:40.363 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:40.385 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:40.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:40.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:40.487 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:01:40.509 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:40.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:40.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:01:40.516 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T14:01:40.610 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:01:40.633 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:40.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:40.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:01:40.733 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:40 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:01:40.734 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:40 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:01:40.734 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:40 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:01:40.734 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:40 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 38710/166365 objects degraded (23.268%), 25 pgs degraded (PG_DEGRADED) 2024-09-21T14:01:40.734 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:40 smithi187 ceph-mon[33032]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T14:01:40.734 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:40 smithi187 ceph-mon[33032]: osd.5 [v2:172.21.15.187:6808/1595353965,v1:172.21.15.187:6809/1595353965] boot 2024-09-21T14:01:40.734 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:40 smithi187 ceph-mon[33032]: osdmap e345: 8 total, 8 up, 8 in 2024-09-21T14:01:40.734 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:40 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T14:01:40.734 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:40.755 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:40.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:40.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:40.810 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:01:40.856 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:01:40.879 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:40.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:40.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:01:40.932 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:01:40 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:01:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:01:40.932 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:40 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:01:40.932 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:40 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:01:40.932 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:40 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:01:40.932 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:40 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 38710/166365 objects degraded (23.268%), 25 pgs degraded (PG_DEGRADED) 2024-09-21T14:01:40.932 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:40 smithi049 ceph-mon[40489]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T14:01:40.933 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:40 smithi049 ceph-mon[40489]: osd.5 [v2:172.21.15.187:6808/1595353965,v1:172.21.15.187:6809/1595353965] boot 2024-09-21T14:01:40.933 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:40 smithi049 ceph-mon[40489]: osdmap e345: 8 total, 8 up, 8 in 2024-09-21T14:01:40.933 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:40 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T14:01:40.933 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:40 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:01:40.933 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:40 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:01:40.933 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:40 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:01:40.933 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:40 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 38710/166365 objects degraded (23.268%), 25 pgs degraded (PG_DEGRADED) 2024-09-21T14:01:40.933 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:40 smithi049 ceph-mon[30080]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T14:01:40.933 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:40 smithi049 ceph-mon[30080]: osd.5 [v2:172.21.15.187:6808/1595353965,v1:172.21.15.187:6809/1595353965] boot 2024-09-21T14:01:40.933 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:40 smithi049 ceph-mon[30080]: osdmap e345: 8 total, 8 up, 8 in 2024-09-21T14:01:40.933 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:40 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T14:01:40.981 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:41.003 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:41.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:41.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:41.053 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd reweight 0 1 2024-09-21T14:01:41.104 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:01:41.127 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:41.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:41.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:01:41.228 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:41.259 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:41.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:41.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:41.295 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T14:01:41.295 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:01:41.361 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:01:41.381 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:41.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:41.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:01:41.431 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:01:41.483 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:41.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:41.506 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:41.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:41.607 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:41.628 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:41.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:41.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:41.730 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:41.752 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:41.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:41.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:41.854 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:41.875 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:41.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:41.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:41.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:41 smithi187 ceph-mon[33032]: pgmap v639: 158 pgs: 1 undersized+degraded+remapped+backfilling+peered, 1 active+clean+remapped, 2 active+recovering+undersized+degraded, 35 active+undersized, 22 active+undersized+degraded, 97 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 811 B/s rd, 0 op/s; 38527/166365 objects degraded (23.158%); 7.3 MiB/s, 116 objects/s recovering 2024-09-21T14:01:41.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:41 smithi187 ceph-mon[33032]: osdmap e346: 8 total, 8 up, 8 in 2024-09-21T14:01:41.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:41 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/435215281' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:01:41.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:41 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:01:41.961 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T14:01:41.977 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:01:41.983 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:41 smithi049 ceph-mon[30080]: pgmap v639: 158 pgs: 1 undersized+degraded+remapped+backfilling+peered, 1 active+clean+remapped, 2 active+recovering+undersized+degraded, 35 active+undersized, 22 active+undersized+degraded, 97 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 811 B/s rd, 0 op/s; 38527/166365 objects degraded (23.158%); 7.3 MiB/s, 116 objects/s recovering 2024-09-21T14:01:41.983 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:41 smithi049 ceph-mon[30080]: osdmap e346: 8 total, 8 up, 8 in 2024-09-21T14:01:41.983 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:41 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/435215281' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:01:41.983 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:41 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:01:41.983 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:41 smithi049 ceph-mon[40489]: pgmap v639: 158 pgs: 1 undersized+degraded+remapped+backfilling+peered, 1 active+clean+remapped, 2 active+recovering+undersized+degraded, 35 active+undersized, 22 active+undersized+degraded, 97 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 811 B/s rd, 0 op/s; 38527/166365 objects degraded (23.158%); 7.3 MiB/s, 116 objects/s recovering 2024-09-21T14:01:41.983 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:41 smithi049 ceph-mon[40489]: osdmap e346: 8 total, 8 up, 8 in 2024-09-21T14:01:41.983 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:41 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/435215281' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:01:41.983 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:41 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:01:41.999 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:41.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:42.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:01:42.027 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T14:01:42.100 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:42.123 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:42.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:42.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:42.168 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-09-21T14:01:42.225 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:42.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:42.257 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:42.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:42.345 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:01:42.358 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:01:42.382 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:42.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:42.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:01:42.483 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:42.504 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:42.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:42.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:42.606 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:01:42.629 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:42.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:42.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:01:42.731 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:42.753 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:42.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:42.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:42.855 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:42.877 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:42.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:42.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:42.880 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:42 smithi187 ceph-mon[33032]: 9.d continuing backfill to osd.7 from (296'7120,324'11721] 9:b4276f3e:::benchmark_data_smithi049_138504_object33880:head to 324'11721 2024-09-21T14:01:42.880 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:42 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T14:01:42.880 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:42 smithi187 ceph-mon[33032]: osdmap e347: 8 total, 8 up, 8 in 2024-09-21T14:01:42.880 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:42 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2042913129' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T14:01:42.902 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:42 smithi049 ceph-mon[40489]: 9.d continuing backfill to osd.7 from (296'7120,324'11721] 9:b4276f3e:::benchmark_data_smithi049_138504_object33880:head to 324'11721 2024-09-21T14:01:42.902 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:42 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T14:01:42.902 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:42 smithi049 ceph-mon[40489]: osdmap e347: 8 total, 8 up, 8 in 2024-09-21T14:01:42.902 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:42 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2042913129' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T14:01:42.902 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:42 smithi049 ceph-mon[30080]: 9.d continuing backfill to osd.7 from (296'7120,324'11721] 9:b4276f3e:::benchmark_data_smithi049_138504_object33880:head to 324'11721 2024-09-21T14:01:42.902 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:42 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T14:01:42.902 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:42 smithi049 ceph-mon[30080]: osdmap e347: 8 total, 8 up, 8 in 2024-09-21T14:01:42.902 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:42 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2042913129' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T14:01:42.979 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:42.991 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:01:42.997 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T14:01:42.997 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:01:43.002 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:43.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:43.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:43.103 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:43.126 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:43.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:43.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:43.164 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.0 to 1 (10000) 2024-09-21T14:01:43.228 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:43.251 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:43.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:43.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:43.353 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:01:43.376 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:43.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:43.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:01:43.478 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:43.499 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:43.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:43.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:43.601 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:43.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:43.624 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:43.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:43.631 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:01:43.631 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:01:43.644 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:01:43.644 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:01:43.660 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:01:43.660 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:01:43.676 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:01:43.677 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:01:43.694 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:01:43.694 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:01:43.711 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:01:43.711 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:01:43.725 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:43.727 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:01:43.727 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:01:43.746 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:01:43.747 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:01:43.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:43.747 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:43.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:43.843 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:43 smithi049 ceph-mon[40489]: pgmap v642: 158 pgs: 1 unknown, 1 remapped+peering, 8 peering, 1 undersized+degraded+remapped+backfilling+peered, 2 active+recovering+undersized+degraded, 29 active+undersized, 19 active+undersized+degraded, 97 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 31611/166365 objects degraded (19.001%); 2.7 MiB/s, 42 objects/s recovering 2024-09-21T14:01:43.843 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:43 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2042913129' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T14:01:43.843 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:43 smithi049 ceph-mon[40489]: osdmap e348: 8 total, 8 up, 8 in 2024-09-21T14:01:43.843 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:43 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/25679598' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:01:43.844 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:43 smithi049 ceph-mon[30080]: pgmap v642: 158 pgs: 1 unknown, 1 remapped+peering, 8 peering, 1 undersized+degraded+remapped+backfilling+peered, 2 active+recovering+undersized+degraded, 29 active+undersized, 19 active+undersized+degraded, 97 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 31611/166365 objects degraded (19.001%); 2.7 MiB/s, 42 objects/s recovering 2024-09-21T14:01:43.844 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:43 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2042913129' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T14:01:43.844 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:43 smithi049 ceph-mon[30080]: osdmap e348: 8 total, 8 up, 8 in 2024-09-21T14:01:43.844 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:43 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/25679598' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:01:43.848 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:43.871 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:43.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:43.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:43.872 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T14:01:43.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:43 smithi187 ceph-mon[33032]: pgmap v642: 158 pgs: 1 unknown, 1 remapped+peering, 8 peering, 1 undersized+degraded+remapped+backfilling+peered, 2 active+recovering+undersized+degraded, 29 active+undersized, 19 active+undersized+degraded, 97 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 31611/166365 objects degraded (19.001%); 2.7 MiB/s, 42 objects/s recovering 2024-09-21T14:01:43.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:43 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2042913129' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T14:01:43.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:43 smithi187 ceph-mon[33032]: osdmap e348: 8 total, 8 up, 8 in 2024-09-21T14:01:43.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:43 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/25679598' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:01:43.972 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:43.994 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:43.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:43.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:44.095 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:44.116 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:44.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:44.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:44.217 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:44.241 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd reweight 1 1 2024-09-21T14:01:44.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:44.242 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:44.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:44.343 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:01:44.364 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:44.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:44.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:01:44.466 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:01:44.496 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:44.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:44.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:01:44.598 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:01:44.621 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:44.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:44.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:01:44.722 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:01:44.745 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:44.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:44.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:01:44.847 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:44.869 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:44.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:44.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:44.930 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T14:01:44.947 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:44 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:01:44.947 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:44 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:01:44.947 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:44 smithi049 ceph-mon[30080]: pgmap v645: 158 pgs: 1 unknown, 1 remapped+peering, 8 peering, 1 undersized+degraded+remapped+backfilling+peered, 2 active+recovering+undersized+degraded, 22 active+undersized, 13 active+undersized+degraded, 110 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 3 op/s; 28114/166341 objects degraded (16.901%); 0 B/s, 4 objects/s recovering 2024-09-21T14:01:44.947 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:44 smithi049 ceph-mon[30080]: osdmap e349: 8 total, 8 up, 8 in 2024-09-21T14:01:44.971 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:44.992 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:44.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:44.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:45.070 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:01:45.094 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:01:45.114 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:45.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:45.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:01:45.118 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:01:45.204 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:44 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:01:45.204 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:44 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:01:45.204 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:44 smithi049 ceph-mon[40489]: pgmap v645: 158 pgs: 1 unknown, 1 remapped+peering, 8 peering, 1 undersized+degraded+remapped+backfilling+peered, 2 active+recovering+undersized+degraded, 22 active+undersized, 13 active+undersized+degraded, 110 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 3 op/s; 28114/166341 objects degraded (16.901%); 0 B/s, 4 objects/s recovering 2024-09-21T14:01:45.204 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:44 smithi049 ceph-mon[40489]: osdmap e349: 8 total, 8 up, 8 in 2024-09-21T14:01:45.216 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:01:45.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:45.239 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:45.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:01:45.340 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:01:45.365 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:45.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:45.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:01:45.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:44 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:01:45.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:44 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:01:45.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:44 smithi187 ceph-mon[33032]: pgmap v645: 158 pgs: 1 unknown, 1 remapped+peering, 8 peering, 1 undersized+degraded+remapped+backfilling+peered, 2 active+recovering+undersized+degraded, 22 active+undersized, 13 active+undersized+degraded, 110 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 3 op/s; 28114/166341 objects degraded (16.901%); 0 B/s, 4 objects/s recovering 2024-09-21T14:01:45.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:44 smithi187 ceph-mon[33032]: osdmap e349: 8 total, 8 up, 8 in 2024-09-21T14:01:45.467 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:45.488 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:45.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:45.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:45.589 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:45.614 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:45.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:45.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:45.715 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:01:45.742 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:45.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:45.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:01:45.843 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:01:45.869 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:45.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:45.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:01:45.971 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:01:46.001 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:46.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:46.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:01:46.103 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:46.106 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:45 smithi049 ceph-mon[30080]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:01:46.106 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:45 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 28114/166341 objects degraded (16.901%), 16 pgs degraded (PG_DEGRADED) 2024-09-21T14:01:46.106 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:45 smithi049 ceph-mon[30080]: osdmap e350: 8 total, 8 up, 8 in 2024-09-21T14:01:46.106 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:45 smithi049 ceph-mon[40489]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:01:46.106 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:45 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 28114/166341 objects degraded (16.901%), 16 pgs degraded (PG_DEGRADED) 2024-09-21T14:01:46.106 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:45 smithi049 ceph-mon[40489]: osdmap e350: 8 total, 8 up, 8 in 2024-09-21T14:01:46.127 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:46.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:46.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:46.165 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T14:01:46.165 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-09-21T14:01:46.166 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 0 2024-09-21T14:01:46.166 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T14:01:46.229 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:46.253 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:46.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:46.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:46.355 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:46.382 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:46.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:46.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:46.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:45 smithi187 ceph-mon[33032]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:01:46.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:45 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 28114/166341 objects degraded (16.901%), 16 pgs degraded (PG_DEGRADED) 2024-09-21T14:01:46.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:45 smithi187 ceph-mon[33032]: osdmap e350: 8 total, 8 up, 8 in 2024-09-21T14:01:46.484 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:01:46.511 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:46.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:46.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:01:46.613 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:01:46.638 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:46.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:46.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:01:46.740 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:01:46.770 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:46.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:46.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:01:46.872 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:01:46.899 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:46.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:46.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:01:46.928 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:01:46.978 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T14:01:46.995 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T14:01:47.001 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:47.049 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:47.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:47.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:47.084 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.1 to 1 (10000) 2024-09-21T14:01:47.151 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:01:47.179 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:47.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:47.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:01:47.239 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:47 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:01:47.239 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:47 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:01:47.239 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:47 smithi049 ceph-mon[40489]: pgmap v648: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 1 unknown, 1 remapped+peering, 8 peering, 1 active+undersized+degraded+remapped+backfill_toofull, 145 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 16 op/s; 8036/166164 objects degraded (4.836%) 2024-09-21T14:01:47.239 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:47 smithi049 ceph-mon[40489]: osdmap e351: 8 total, 8 up, 8 in 2024-09-21T14:01:47.239 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:47 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3376190571' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T14:01:47.239 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:47 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T14:01:47.240 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:47 smithi049 ceph-mon[40489]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T14:01:47.240 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:47 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:01:47.240 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:47 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:01:47.240 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:47 smithi049 ceph-mon[30080]: pgmap v648: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 1 unknown, 1 remapped+peering, 8 peering, 1 active+undersized+degraded+remapped+backfill_toofull, 145 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 16 op/s; 8036/166164 objects degraded (4.836%) 2024-09-21T14:01:47.240 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:47 smithi049 ceph-mon[30080]: osdmap e351: 8 total, 8 up, 8 in 2024-09-21T14:01:47.240 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:47 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3376190571' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T14:01:47.240 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:47 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T14:01:47.240 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:47 smithi049 ceph-mon[30080]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T14:01:47.281 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:47.314 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:47.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:47.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:47.345 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:47 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:01:47.345 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:47 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:01:47.345 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:47 smithi187 ceph-mon[33032]: pgmap v648: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 1 unknown, 1 remapped+peering, 8 peering, 1 active+undersized+degraded+remapped+backfill_toofull, 145 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 16 op/s; 8036/166164 objects degraded (4.836%) 2024-09-21T14:01:47.346 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:47 smithi187 ceph-mon[33032]: osdmap e351: 8 total, 8 up, 8 in 2024-09-21T14:01:47.346 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:47 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3376190571' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T14:01:47.346 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:47 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T14:01:47.346 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:47 smithi187 ceph-mon[33032]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T14:01:47.416 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:47.444 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:47.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:47.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:47.546 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:47.577 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:47.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:47.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:47.678 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:01:47.704 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:47.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:47.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:01:47.805 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:47.842 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:47.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:47.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:47.944 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:01:47.974 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:47.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:47.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:01:48.075 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:01:48.099 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd reweight 2 1 2024-09-21T14:01:48.112 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:48.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:48.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:01:48.213 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:48.242 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:48.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:48.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:48.344 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:01:48.366 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:48 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T14:01:48.366 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:48 smithi049 ceph-mon[40489]: osdmap e352: 8 total, 8 up, 8 in 2024-09-21T14:01:48.367 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:48 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T14:01:48.367 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:48 smithi049 ceph-mon[30080]: osdmap e352: 8 total, 8 up, 8 in 2024-09-21T14:01:48.385 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:48.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:48.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:01:48.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:48 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T14:01:48.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:48 smithi187 ceph-mon[33032]: osdmap e352: 8 total, 8 up, 8 in 2024-09-21T14:01:48.470 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:01:48.481 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:01:48.487 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:01:48.533 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:01:48.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:48.534 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:48.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:01:48.635 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:48.666 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:48.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:48.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:48.768 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:01:48.802 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:48.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:48.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:01:48.904 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:48.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:48.940 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:48.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:49.041 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:49.074 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:49.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:49.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:49.113 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-09-21T14:01:49.176 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:01:49.206 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:49.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:49.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:01:49.247 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:49 smithi049 ceph-mon[30080]: pgmap v651: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 155 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 54 op/s; 8036/165693 objects degraded (4.850%) 2024-09-21T14:01:49.247 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:49 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3813992466' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:01:49.247 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:49 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:01:49.247 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:49 smithi049 ceph-mon[40489]: pgmap v651: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 155 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 54 op/s; 8036/165693 objects degraded (4.850%) 2024-09-21T14:01:49.247 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:49 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3813992466' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:01:49.248 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:49 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:01:49.308 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:01:49.341 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:49.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:49.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:01:49.383 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:01:49.384 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:01:49.405 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:01:49.405 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:01:49.425 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:01:49.425 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:01:49.443 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:49.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:49 smithi187 ceph-mon[33032]: pgmap v651: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 155 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 54 op/s; 8036/165693 objects degraded (4.850%) 2024-09-21T14:01:49.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:49 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3813992466' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:01:49.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:49 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:01:49.445 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:01:49.446 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:01:49.474 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:49.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:49.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:49.477 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:01:49.477 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:01:49.504 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:01:49.504 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:01:49.531 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:01:49.531 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:01:49.561 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:01:49.561 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:01:49.576 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:01:49.610 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:49.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:49.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:01:49.712 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:49.743 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:49.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:49.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:49.845 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:49.873 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:49.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:49.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:49.956 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T14:01:49.956 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-09-21T14:01:49.956 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-09-21T14:01:49.956 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-09-21T14:01:49.956 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T14:01:49.975 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:01:50.003 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:50.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:50.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:01:50.105 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:50.111 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:01:49 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:01:50.132 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:50.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:50.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:50.234 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:01:50.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:50 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T14:01:50.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:50 smithi049 ceph-mon[30080]: osdmap e353: 8 total, 8 up, 8 in 2024-09-21T14:01:50.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:50 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2970125729' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:01:50.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:50 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:01:50.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:50 smithi049 ceph-mon[30080]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:01:50.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:50 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 8036/165693 objects degraded (4.850%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T14:01:50.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:50 smithi049 ceph-mon[30080]: osdmap e354: 8 total, 8 up, 8 in 2024-09-21T14:01:50.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:50 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T14:01:50.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:50 smithi049 ceph-mon[40489]: osdmap e353: 8 total, 8 up, 8 in 2024-09-21T14:01:50.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:50 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2970125729' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:01:50.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:50 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:01:50.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:50 smithi049 ceph-mon[40489]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:01:50.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:50 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 8036/165693 objects degraded (4.850%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T14:01:50.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:50 smithi049 ceph-mon[40489]: osdmap e354: 8 total, 8 up, 8 in 2024-09-21T14:01:50.263 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:50.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:50.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:01:50.365 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:50.388 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:50.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:50.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:50.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:50 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T14:01:50.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:50 smithi187 ceph-mon[33032]: osdmap e353: 8 total, 8 up, 8 in 2024-09-21T14:01:50.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:50 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2970125729' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:01:50.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:50 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:01:50.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:50 smithi187 ceph-mon[33032]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:01:50.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:50 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 8036/165693 objects degraded (4.850%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T14:01:50.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:50 smithi187 ceph-mon[33032]: osdmap e354: 8 total, 8 up, 8 in 2024-09-21T14:01:50.489 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:01:50.513 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:50.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:50.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:01:50.615 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:01:50.643 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:50.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:50.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:01:50.745 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:50.754 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:01:50 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:01:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:01:50.771 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:50.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:50.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:50.816 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T14:01:50.872 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:01:50.908 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:50.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:50.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:01:51.010 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:01:51.044 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:51.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:51.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:01:51.146 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:01:51.176 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:51.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:51.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:01:51.277 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:51.308 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:51.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:51.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:51.410 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:51.442 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:51.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:51.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:51.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:51 smithi187 ceph-mon[33032]: pgmap v653: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 155 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 418 B/s rd, 0 B/s wr, 169 op/s; 7904/163758 objects degraded (4.827%) 2024-09-21T14:01:51.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:51 smithi187 ceph-mon[33032]: 3.10 scrub starts 2024-09-21T14:01:51.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:51 smithi187 ceph-mon[33032]: 3.10 scrub ok 2024-09-21T14:01:51.503 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:51 smithi049 ceph-mon[40489]: pgmap v653: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 155 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 418 B/s rd, 0 B/s wr, 169 op/s; 7904/163758 objects degraded (4.827%) 2024-09-21T14:01:51.503 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:51 smithi049 ceph-mon[40489]: 3.10 scrub starts 2024-09-21T14:01:51.503 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:51 smithi049 ceph-mon[40489]: 3.10 scrub ok 2024-09-21T14:01:51.503 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:51 smithi049 ceph-mon[30080]: pgmap v653: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 155 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 418 B/s rd, 0 B/s wr, 169 op/s; 7904/163758 objects degraded (4.827%) 2024-09-21T14:01:51.503 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:51 smithi049 ceph-mon[30080]: 3.10 scrub starts 2024-09-21T14:01:51.503 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:51 smithi049 ceph-mon[30080]: 3.10 scrub ok 2024-09-21T14:01:51.544 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:01:51.582 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:51.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:51.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:01:51.684 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:51.713 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:51.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:51.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:51.815 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:51.840 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:51.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:51.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:51.942 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:01:51.967 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:51.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:51.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:01:52.069 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:52.094 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:52.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:52.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:52.125 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.2 to 1 (10000) 2024-09-21T14:01:52.141 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:52 smithi049 ceph-mon[40489]: 5.6 scrub starts 2024-09-21T14:01:52.141 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:52 smithi049 ceph-mon[40489]: 5.6 scrub ok 2024-09-21T14:01:52.141 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:52 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/645805380' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T14:01:52.141 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:52 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T14:01:52.141 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:52 smithi049 ceph-mon[30080]: 5.6 scrub starts 2024-09-21T14:01:52.141 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:52 smithi049 ceph-mon[30080]: 5.6 scrub ok 2024-09-21T14:01:52.141 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:52 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/645805380' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T14:01:52.142 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:52 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T14:01:52.196 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:52.223 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:52.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:52.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:52.324 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:01:52.351 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:52.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:52.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:01:52.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:52 smithi187 ceph-mon[33032]: 5.6 scrub starts 2024-09-21T14:01:52.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:52 smithi187 ceph-mon[33032]: 5.6 scrub ok 2024-09-21T14:01:52.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:52 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/645805380' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T14:01:52.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:52 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T14:01:52.453 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:01:52.477 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:52.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:52.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:01:52.579 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:52.612 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:52.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:52.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:52.714 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:52.739 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:52.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:52.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:52.841 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:52.871 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:52.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:52.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:52.973 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:01:53.001 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:53.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:53.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:01:53.102 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:01:53.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:53.132 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:53.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:01:53.188 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd reweight 3 1 2024-09-21T14:01:53.198 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:01:53.233 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:53.262 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:53.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:53.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:53.363 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:53.387 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:53.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:53.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:53.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:53 smithi187 ceph-mon[33032]: pgmap v655: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 155 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 574 op/s; 6577/155772 objects degraded (4.222%); 2.8 MiB/s, 48 objects/s recovering 2024-09-21T14:01:53.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:53 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T14:01:53.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:53 smithi187 ceph-mon[33032]: osdmap e355: 8 total, 8 up, 8 in 2024-09-21T14:01:53.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:53 smithi187 ceph-mon[33032]: 5.2 scrub starts 2024-09-21T14:01:53.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:53 smithi187 ceph-mon[33032]: 5.2 scrub ok 2024-09-21T14:01:53.488 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:53.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:53 smithi049 ceph-mon[30080]: pgmap v655: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 155 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 574 op/s; 6577/155772 objects degraded (4.222%); 2.8 MiB/s, 48 objects/s recovering 2024-09-21T14:01:53.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:53 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T14:01:53.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:53 smithi049 ceph-mon[30080]: osdmap e355: 8 total, 8 up, 8 in 2024-09-21T14:01:53.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:53 smithi049 ceph-mon[30080]: 5.2 scrub starts 2024-09-21T14:01:53.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:53 smithi049 ceph-mon[30080]: 5.2 scrub ok 2024-09-21T14:01:53.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:53 smithi049 ceph-mon[40489]: pgmap v655: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 155 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 574 op/s; 6577/155772 objects degraded (4.222%); 2.8 MiB/s, 48 objects/s recovering 2024-09-21T14:01:53.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:53 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T14:01:53.507 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:53 smithi049 ceph-mon[40489]: osdmap e355: 8 total, 8 up, 8 in 2024-09-21T14:01:53.507 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:53 smithi049 ceph-mon[40489]: 5.2 scrub starts 2024-09-21T14:01:53.507 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:53 smithi049 ceph-mon[40489]: 5.2 scrub ok 2024-09-21T14:01:53.515 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:53.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:53.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:53.549 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T14:01:53.619 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:53.651 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:53.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:53.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:53.753 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:01:53.783 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:53.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:53.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:01:53.885 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:01:53.917 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:53.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:53.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:01:53.976 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T14:01:53.976 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:01:54.019 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:54.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:54.050 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:54.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:54.151 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:54.182 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:54.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:54.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:54.230 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T14:01:54.284 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:01:54.318 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:54.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:54.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:01:54.420 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:54.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:54 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:01:54.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:54 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:01:54.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:54 smithi187 ceph-mon[33032]: osdmap e356: 8 total, 8 up, 8 in 2024-09-21T14:01:54.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:54 smithi187 ceph-mon[33032]: 9.b scrub starts 2024-09-21T14:01:54.448 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:54.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:54.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:54.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:54 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:01:54.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:54 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:01:54.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:54 smithi049 ceph-mon[40489]: osdmap e356: 8 total, 8 up, 8 in 2024-09-21T14:01:54.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:54 smithi049 ceph-mon[40489]: 9.b scrub starts 2024-09-21T14:01:54.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:54 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:01:54.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:54 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:01:54.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:54 smithi049 ceph-mon[30080]: osdmap e356: 8 total, 8 up, 8 in 2024-09-21T14:01:54.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:54 smithi049 ceph-mon[30080]: 9.b scrub starts 2024-09-21T14:01:54.549 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:54.579 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:54.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:54.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:54.681 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:01:54.708 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:54.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:54.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:01:54.810 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:54.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:54.840 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:54.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:54.941 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:01:54.967 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:54.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:54.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:01:55.069 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:01:55.095 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:55.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:55.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:01:55.196 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:55.226 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:55.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:55.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:55.238 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T14:01:55.274 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:01:55.328 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:55.354 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:01:55.358 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:55.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:55.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:55.361 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:01:55.431 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:55 smithi049 ceph-mon[30080]: 6.1e scrub starts 2024-09-21T14:01:55.431 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:55 smithi049 ceph-mon[30080]: 6.1e scrub ok 2024-09-21T14:01:55.431 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:55 smithi049 ceph-mon[30080]: pgmap v658: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 155 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 707 op/s; 6577/155022 objects degraded (4.243%); 3.4 MiB/s, 57 objects/s recovering 2024-09-21T14:01:55.431 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:55 smithi049 ceph-mon[30080]: 5.1d scrub starts 2024-09-21T14:01:55.431 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:55 smithi049 ceph-mon[30080]: 5.1d scrub ok 2024-09-21T14:01:55.431 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:55 smithi049 ceph-mon[30080]: 9.b scrub ok 2024-09-21T14:01:55.431 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:55 smithi049 ceph-mon[30080]: 4.1f scrub starts 2024-09-21T14:01:55.432 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:55 smithi049 ceph-mon[30080]: 4.1f scrub ok 2024-09-21T14:01:55.432 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:55 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 6577/155022 objects degraded (4.243%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T14:01:55.432 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:55 smithi049 ceph-mon[40489]: 6.1e scrub starts 2024-09-21T14:01:55.432 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:55 smithi049 ceph-mon[40489]: 6.1e scrub ok 2024-09-21T14:01:55.432 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:55 smithi049 ceph-mon[40489]: pgmap v658: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 155 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 707 op/s; 6577/155022 objects degraded (4.243%); 3.4 MiB/s, 57 objects/s recovering 2024-09-21T14:01:55.432 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:55 smithi049 ceph-mon[40489]: 5.1d scrub starts 2024-09-21T14:01:55.432 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:55 smithi049 ceph-mon[40489]: 5.1d scrub ok 2024-09-21T14:01:55.432 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:55 smithi049 ceph-mon[40489]: 9.b scrub ok 2024-09-21T14:01:55.432 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:55 smithi049 ceph-mon[40489]: 4.1f scrub starts 2024-09-21T14:01:55.432 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:55 smithi049 ceph-mon[40489]: 4.1f scrub ok 2024-09-21T14:01:55.433 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:55 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 6577/155022 objects degraded (4.243%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T14:01:55.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:55 smithi187 ceph-mon[33032]: 6.1e scrub starts 2024-09-21T14:01:55.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:55 smithi187 ceph-mon[33032]: 6.1e scrub ok 2024-09-21T14:01:55.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:55 smithi187 ceph-mon[33032]: pgmap v658: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 155 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 707 op/s; 6577/155022 objects degraded (4.243%); 3.4 MiB/s, 57 objects/s recovering 2024-09-21T14:01:55.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:55 smithi187 ceph-mon[33032]: 5.1d scrub starts 2024-09-21T14:01:55.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:55 smithi187 ceph-mon[33032]: 5.1d scrub ok 2024-09-21T14:01:55.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:55 smithi187 ceph-mon[33032]: 9.b scrub ok 2024-09-21T14:01:55.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:55 smithi187 ceph-mon[33032]: 4.1f scrub starts 2024-09-21T14:01:55.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:55 smithi187 ceph-mon[33032]: 4.1f scrub ok 2024-09-21T14:01:55.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:55 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 6577/155022 objects degraded (4.243%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T14:01:55.460 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:01:55.487 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:55.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:55.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:01:55.589 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:55.614 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:55.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:55.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:55.716 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:01:55.744 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:55.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:55.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:01:55.845 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:55.871 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:55.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:55.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:55.973 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:56.006 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:56.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:56.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:56.108 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:01:56.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:56.137 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:56.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:01:56.238 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:56.267 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:56.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:56.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:56.369 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:01:56.396 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:56.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:56.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:01:56.498 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:56.528 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:56.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:56.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:56.631 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:01:56.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:56.661 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:56.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:01:56.762 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:56.790 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:56.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:56.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:56.891 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:56.922 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:56.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:56.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:57.024 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:57.050 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:57.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:57.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:57.141 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.3 to 1 (10000) 2024-09-21T14:01:57.152 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:01:57.179 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:57.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:57.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:01:57.201 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:01:57.202 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:01:57.202 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:01:57.202 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:01:57.202 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:01:57.202 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:01:57.202 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:01:57.202 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:01:57.203 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:01:57.203 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:01:57.203 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:01:57.203 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:01:57.203 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:01:57.203 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:01:57.203 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:01:57.203 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:01:57.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:57 smithi049 ceph-mon[40489]: 5.e scrub starts 2024-09-21T14:01:57.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:57 smithi049 ceph-mon[40489]: 5.e scrub ok 2024-09-21T14:01:57.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:57 smithi049 ceph-mon[40489]: pgmap v659: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 155 active+clean; 3.0 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 776 op/s; 5884/149784 objects degraded (3.928%); 8.2 MiB/s, 134 objects/s recovering 2024-09-21T14:01:57.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:57 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/369403874' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T14:01:57.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:57 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2741344027' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:01:57.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:57 smithi049 ceph-mon[30080]: 5.e scrub starts 2024-09-21T14:01:57.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:57 smithi049 ceph-mon[30080]: 5.e scrub ok 2024-09-21T14:01:57.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:57 smithi049 ceph-mon[30080]: pgmap v659: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 155 active+clean; 3.0 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 776 op/s; 5884/149784 objects degraded (3.928%); 8.2 MiB/s, 134 objects/s recovering 2024-09-21T14:01:57.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:57 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/369403874' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T14:01:57.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:57 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2741344027' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:01:57.280 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:01:57.308 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:57.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:57.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:01:57.410 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:01:57.436 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:57.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:57.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:01:57.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:57 smithi187 ceph-mon[33032]: 5.e scrub starts 2024-09-21T14:01:57.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:57 smithi187 ceph-mon[33032]: 5.e scrub ok 2024-09-21T14:01:57.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:57 smithi187 ceph-mon[33032]: pgmap v659: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 155 active+clean; 3.0 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 776 op/s; 5884/149784 objects degraded (3.928%); 8.2 MiB/s, 134 objects/s recovering 2024-09-21T14:01:57.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:57 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/369403874' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T14:01:57.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:57 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2741344027' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:01:57.538 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:57.563 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:57.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:57.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:57.665 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:01:57.693 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:57.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:57.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:01:57.795 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:01:57.819 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:57.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:57.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:01:57.921 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:01:57.943 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:57.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:57.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:01:58.045 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:01:58.073 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:58.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:58.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:01:58.175 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:01:58.201 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:58.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:58.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:01:58.303 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:01:58.328 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:58.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:58.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:01:58.430 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:58.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:58 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/369403874' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T14:01:58.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:58 smithi187 ceph-mon[33032]: osdmap e357: 8 total, 8 up, 8 in 2024-09-21T14:01:58.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:58 smithi187 ceph-mon[33032]: 4.1b scrub starts 2024-09-21T14:01:58.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:58 smithi187 ceph-mon[33032]: 4.1b scrub ok 2024-09-21T14:01:58.459 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:58.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:58.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:58.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:58 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/369403874' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T14:01:58.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:58 smithi049 ceph-mon[40489]: osdmap e357: 8 total, 8 up, 8 in 2024-09-21T14:01:58.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:58 smithi049 ceph-mon[40489]: 4.1b scrub starts 2024-09-21T14:01:58.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:58 smithi049 ceph-mon[40489]: 4.1b scrub ok 2024-09-21T14:01:58.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:58 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/369403874' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T14:01:58.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:58 smithi049 ceph-mon[30080]: osdmap e357: 8 total, 8 up, 8 in 2024-09-21T14:01:58.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:58 smithi049 ceph-mon[30080]: 4.1b scrub starts 2024-09-21T14:01:58.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:58 smithi049 ceph-mon[30080]: 4.1b scrub ok 2024-09-21T14:01:58.560 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:01:58.589 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:58.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:58.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:01:58.690 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:01:58.707 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd reweight 4 1 2024-09-21T14:01:58.720 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:58.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:58.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:01:58.822 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:01:58.850 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:58.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:58.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:01:58.952 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:58.978 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:58.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:58.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:59.080 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:01:59.108 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:59.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:59.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:01:59.163 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-09-21T14:01:59.210 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:01:59.235 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:59.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:59.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:01:59.336 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:01:59.361 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:01:59.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:59.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:01:59.440 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:59 smithi049 ceph-mon[40489]: pgmap v661: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 155 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 579 op/s; 5884/145344 objects degraded (4.048%); 5.5 MiB/s, 87 objects/s recovering 2024-09-21T14:01:59.440 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:59 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3446086168' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:01:59.440 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:59 smithi049 ceph-mon[40489]: 3.3 scrub starts 2024-09-21T14:01:59.440 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:59 smithi049 ceph-mon[40489]: 3.3 scrub ok 2024-09-21T14:01:59.440 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:59 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:01:59.441 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:01:59 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:01:59.441 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:59 smithi049 ceph-mon[30080]: pgmap v661: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 155 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 579 op/s; 5884/145344 objects degraded (4.048%); 5.5 MiB/s, 87 objects/s recovering 2024-09-21T14:01:59.441 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:59 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3446086168' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:01:59.441 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:59 smithi049 ceph-mon[30080]: 3.3 scrub starts 2024-09-21T14:01:59.441 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:59 smithi049 ceph-mon[30080]: 3.3 scrub ok 2024-09-21T14:01:59.441 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:59 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:01:59.441 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:01:59 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:01:59.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:59 smithi187 ceph-mon[33032]: pgmap v661: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 155 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 579 op/s; 5884/145344 objects degraded (4.048%); 5.5 MiB/s, 87 objects/s recovering 2024-09-21T14:01:59.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:59 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3446086168' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:01:59.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:59 smithi187 ceph-mon[33032]: 3.3 scrub starts 2024-09-21T14:01:59.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:59 smithi187 ceph-mon[33032]: 3.3 scrub ok 2024-09-21T14:01:59.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:59 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:01:59.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:01:59 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:01:59.463 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:59.490 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:59.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:59.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:59.592 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:01:59.617 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:59.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:59.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:01:59.719 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:01:59.745 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:59.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:59.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:01:59.847 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:01:59.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:01:59.875 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:01:59.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:01:59.976 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:00.003 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:00.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:00.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:00.075 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:02:00.079 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:02:00.106 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:00.136 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:00.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:00.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:00.159 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:01:59 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:02:00.237 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:02:00.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:00 smithi049 ceph-mon[40489]: 3.19 scrub starts 2024-09-21T14:02:00.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:00 smithi049 ceph-mon[40489]: 3.19 scrub ok 2024-09-21T14:02:00.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:00 smithi049 ceph-mon[40489]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T14:02:00.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:00 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3446086168' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T14:02:00.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:00 smithi049 ceph-mon[40489]: osdmap e358: 8 total, 8 up, 8 in 2024-09-21T14:02:00.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:00 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:02:00.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:00 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 5884/145344 objects degraded (4.048%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T14:02:00.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:00 smithi049 ceph-mon[30080]: 3.19 scrub starts 2024-09-21T14:02:00.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:00 smithi049 ceph-mon[30080]: 3.19 scrub ok 2024-09-21T14:02:00.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:00 smithi049 ceph-mon[30080]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T14:02:00.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:00 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3446086168' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T14:02:00.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:00 smithi049 ceph-mon[30080]: osdmap e358: 8 total, 8 up, 8 in 2024-09-21T14:02:00.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:00 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:02:00.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:00 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 5884/145344 objects degraded (4.048%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T14:02:00.268 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:00.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:00.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:02:00.369 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:00.401 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:00.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:00.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:00.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:00 smithi187 ceph-mon[33032]: 3.19 scrub starts 2024-09-21T14:02:00.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:00 smithi187 ceph-mon[33032]: 3.19 scrub ok 2024-09-21T14:02:00.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:00 smithi187 ceph-mon[33032]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T14:02:00.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:00 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3446086168' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T14:02:00.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:00 smithi187 ceph-mon[33032]: osdmap e358: 8 total, 8 up, 8 in 2024-09-21T14:02:00.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:00 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:02:00.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:00 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 5884/145344 objects degraded (4.048%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T14:02:00.503 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:00.531 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:00.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:00.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:00.633 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:02:00.661 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:00.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:00.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:02:00.762 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:02:00.788 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:00.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:00.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:02:00.816 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:02:00 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:02:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:02:00.890 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:02:00.917 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:00.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:00.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:02:01.019 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:02:01.044 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:01.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:01.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:02:01.146 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:01.165 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.4 to 1 (10000) 2024-09-21T14:02:01.172 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:01.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:01.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:01.234 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T14:02:01.234 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-09-21T14:02:01.234 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 0 2024-09-21T14:02:01.234 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T14:02:01.275 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:02:01.302 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:01.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:01.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:02:01.404 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:02:01.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:01.438 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:01.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:02:01.443 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T14:02:01.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:01 smithi187 ceph-mon[33032]: pgmap v663: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 155 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 748 B/s rd, 0 B/s wr, 623 op/s; 5735/142986 objects degraded (4.011%); 4.8 MiB/s, 76 objects/s recovering 2024-09-21T14:02:01.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:01 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3625524806' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T14:02:01.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:01 smithi049 ceph-mon[30080]: pgmap v663: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 155 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 748 B/s rd, 0 B/s wr, 623 op/s; 5735/142986 objects degraded (4.011%); 4.8 MiB/s, 76 objects/s recovering 2024-09-21T14:02:01.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:01 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3625524806' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T14:02:01.476 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:01 smithi049 ceph-mon[40489]: pgmap v663: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 155 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 748 B/s rd, 0 B/s wr, 623 op/s; 5735/142986 objects degraded (4.011%); 4.8 MiB/s, 76 objects/s recovering 2024-09-21T14:02:01.476 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:01 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3625524806' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T14:02:01.540 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:02:01.572 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:01.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:01.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:02:01.674 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:02:01.706 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:01.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:01.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:02:01.808 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:02:01.840 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:01.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:01.841 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:02:01.942 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:01.966 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:01.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:01.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:02.042 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T14:02:02.043 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd reweight 5 1 2024-09-21T14:02:02.068 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:02:02.116 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:02.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:02.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:02:02.218 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:02:02.245 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:02.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:02.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:02:02.347 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:02:02.372 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:02.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:02.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:02:02.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:02 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3625524806' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T14:02:02.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:02 smithi187 ceph-mon[33032]: osdmap e359: 8 total, 8 up, 8 in 2024-09-21T14:02:02.474 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:02:02.501 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:02.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:02.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:02:02.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:02 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3625524806' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T14:02:02.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:02 smithi049 ceph-mon[40489]: osdmap e359: 8 total, 8 up, 8 in 2024-09-21T14:02:02.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:02 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3625524806' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T14:02:02.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:02 smithi049 ceph-mon[30080]: osdmap e359: 8 total, 8 up, 8 in 2024-09-21T14:02:02.603 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:02:02.631 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:02.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:02.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:02:02.734 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:02.763 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:02.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:02.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:02.865 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:02:02.892 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:02.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:02.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:02:02.995 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:03.018 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:03.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:03.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:03.120 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:03.144 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:03.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:03.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:03.246 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:02:03.276 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:03.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:03.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:02:03.378 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:02:03.414 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:03.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:03.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:02:03.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:03 smithi187 ceph-mon[33032]: pgmap v665: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 155 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 766 op/s; 5070/136005 objects degraded (3.728%); 6.9 MiB/s, 110 objects/s recovering 2024-09-21T14:02:03.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:03 smithi187 ceph-mon[33032]: 4.1 scrub starts 2024-09-21T14:02:03.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:03 smithi187 ceph-mon[33032]: 4.1 scrub ok 2024-09-21T14:02:03.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:03 smithi049 ceph-mon[40489]: pgmap v665: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 155 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 766 op/s; 5070/136005 objects degraded (3.728%); 6.9 MiB/s, 110 objects/s recovering 2024-09-21T14:02:03.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:03 smithi049 ceph-mon[40489]: 4.1 scrub starts 2024-09-21T14:02:03.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:03 smithi049 ceph-mon[40489]: 4.1 scrub ok 2024-09-21T14:02:03.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:03 smithi049 ceph-mon[30080]: pgmap v665: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 155 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 766 op/s; 5070/136005 objects degraded (3.728%); 6.9 MiB/s, 110 objects/s recovering 2024-09-21T14:02:03.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:03 smithi049 ceph-mon[30080]: 4.1 scrub starts 2024-09-21T14:02:03.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:03 smithi049 ceph-mon[30080]: 4.1 scrub ok 2024-09-21T14:02:03.515 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:02:03.544 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:03.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:03.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:02:03.645 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:02:03.669 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:03.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:03.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:02:03.771 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:02:03.799 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:03.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:03.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:02:03.901 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:03.925 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:03.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:03.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:04.027 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:02:04.059 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:04.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:04.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:02:04.161 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:04.185 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:04.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:04.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:04.287 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:02:04.313 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:04.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:04.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:02:04.414 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:02:04.442 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:04.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:04.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:02:04.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:04 smithi187 ceph-mon[33032]: 3.9 scrub starts 2024-09-21T14:02:04.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:04 smithi187 ceph-mon[33032]: 3.9 scrub ok 2024-09-21T14:02:04.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:04 smithi187 ceph-mon[33032]: 1.6 scrub starts 2024-09-21T14:02:04.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:04 smithi187 ceph-mon[33032]: 1.6 scrub ok 2024-09-21T14:02:04.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:04 smithi049 ceph-mon[30080]: 3.9 scrub starts 2024-09-21T14:02:04.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:04 smithi049 ceph-mon[30080]: 3.9 scrub ok 2024-09-21T14:02:04.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:04 smithi049 ceph-mon[30080]: 1.6 scrub starts 2024-09-21T14:02:04.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:04 smithi049 ceph-mon[30080]: 1.6 scrub ok 2024-09-21T14:02:04.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:04 smithi049 ceph-mon[40489]: 3.9 scrub starts 2024-09-21T14:02:04.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:04 smithi049 ceph-mon[40489]: 3.9 scrub ok 2024-09-21T14:02:04.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:04 smithi049 ceph-mon[40489]: 1.6 scrub starts 2024-09-21T14:02:04.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:04 smithi049 ceph-mon[40489]: 1.6 scrub ok 2024-09-21T14:02:04.544 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:02:04.570 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:04.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:04.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:02:04.668 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T14:02:04.672 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:02:04.702 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:04.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:04.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:02:04.728 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:02:04.732 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:02:04.733 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:02:04.804 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:02:04.833 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:04.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:04.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:02:04.935 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:04.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:04.961 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:04.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:05.062 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:05.097 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:05.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:05.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:05.199 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:02:05.226 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:05.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:05.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:02:05.328 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:05.357 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:05.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:05.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:05.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:05 smithi187 ceph-mon[33032]: pgmap v666: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 155 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 749 B/s rd, 0 B/s wr, 482 op/s; 5070/135474 objects degraded (3.742%); 6.1 MiB/s, 97 objects/s recovering 2024-09-21T14:02:05.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:05 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 5070/135474 objects degraded (3.742%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T14:02:05.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:05 smithi187 ceph-mon[33032]: osdmap e360: 8 total, 8 up, 8 in 2024-09-21T14:02:05.459 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:02:05.465 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:05 smithi049 ceph-mon[40489]: pgmap v666: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 155 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 749 B/s rd, 0 B/s wr, 482 op/s; 5070/135474 objects degraded (3.742%); 6.1 MiB/s, 97 objects/s recovering 2024-09-21T14:02:05.465 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:05 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 5070/135474 objects degraded (3.742%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T14:02:05.465 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:05 smithi049 ceph-mon[40489]: osdmap e360: 8 total, 8 up, 8 in 2024-09-21T14:02:05.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:05 smithi049 ceph-mon[30080]: pgmap v666: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+recovering+undersized+degraded+remapped, 155 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 749 B/s rd, 0 B/s wr, 482 op/s; 5070/135474 objects degraded (3.742%); 6.1 MiB/s, 97 objects/s recovering 2024-09-21T14:02:05.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:05 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 5070/135474 objects degraded (3.742%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T14:02:05.465 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:05 smithi049 ceph-mon[30080]: osdmap e360: 8 total, 8 up, 8 in 2024-09-21T14:02:05.495 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:05.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:05.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:02:05.597 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:02:05.622 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:05.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:05.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:02:05.724 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:02:05.759 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:05.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:05.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:02:05.813 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:02:05.813 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:02:05.813 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:02:05.814 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:02:05.817 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:02:05.818 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:02:05.843 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:02:05.843 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:02:05.861 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:05.874 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:02:05.874 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:02:05.887 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:05.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:05.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:05.900 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:02:05.900 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:02:05.919 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:02:05.919 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:02:05.941 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:02:05.941 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:02:05.989 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:02:06.015 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:06.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:06.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:02:06.117 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:02:06.140 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:06.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:06.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:02:06.242 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:02:06.276 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:06.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:06.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:02:06.378 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:02:06.381 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.5 to 1 (10000) 2024-09-21T14:02:06.404 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:06.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:06.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:02:06.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:06 smithi049 ceph-mon[40489]: 1.7 scrub starts 2024-09-21T14:02:06.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:06 smithi049 ceph-mon[40489]: 1.7 scrub ok 2024-09-21T14:02:06.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:06 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1567819700' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:02:06.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:06 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/4207477426' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T14:02:06.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:06 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T14:02:06.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:06 smithi049 ceph-mon[30080]: 1.7 scrub starts 2024-09-21T14:02:06.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:06 smithi049 ceph-mon[30080]: 1.7 scrub ok 2024-09-21T14:02:06.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:06 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1567819700' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:02:06.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:06 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/4207477426' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T14:02:06.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:06 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T14:02:06.505 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:02:06.534 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:06.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:06.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:02:06.636 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:02:06.660 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:06.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:06.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:02:06.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:06 smithi187 ceph-mon[33032]: 1.7 scrub starts 2024-09-21T14:02:06.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:06 smithi187 ceph-mon[33032]: 1.7 scrub ok 2024-09-21T14:02:06.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:06 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1567819700' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:02:06.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:06 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/4207477426' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T14:02:06.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:06 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T14:02:06.762 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:02:06.791 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:06.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:06.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:02:06.893 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:06.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:06.921 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:06.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:07.022 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:07.047 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:07.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:07.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:07.149 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:02:07.176 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:07.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:07.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:02:07.277 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:07.302 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:07.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:07.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:07.404 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:07.429 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:07.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:07.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:07.459 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:07 smithi049 ceph-mon[40489]: pgmap v668: 158 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 155 active+clean; 2.6 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 751 B/s rd, 0 B/s wr, 688 op/s; 4140/130623 objects degraded (3.169%); 13 MiB/s, 216 objects/s recovering 2024-09-21T14:02:07.459 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:07 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T14:02:07.459 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:07 smithi049 ceph-mon[40489]: osdmap e361: 8 total, 8 up, 8 in 2024-09-21T14:02:07.459 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:07 smithi049 ceph-mon[40489]: 1.5 scrub starts 2024-09-21T14:02:07.460 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:07 smithi049 ceph-mon[40489]: 1.5 scrub ok 2024-09-21T14:02:07.460 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:07 smithi049 ceph-mon[30080]: pgmap v668: 158 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 155 active+clean; 2.6 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 751 B/s rd, 0 B/s wr, 688 op/s; 4140/130623 objects degraded (3.169%); 13 MiB/s, 216 objects/s recovering 2024-09-21T14:02:07.460 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:07 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T14:02:07.460 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:07 smithi049 ceph-mon[30080]: osdmap e361: 8 total, 8 up, 8 in 2024-09-21T14:02:07.460 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:07 smithi049 ceph-mon[30080]: 1.5 scrub starts 2024-09-21T14:02:07.460 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:07 smithi049 ceph-mon[30080]: 1.5 scrub ok 2024-09-21T14:02:07.532 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:02:07.558 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:07.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:07.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:02:07.660 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:02:07.687 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:07.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:07.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:02:07.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:07 smithi187 ceph-mon[33032]: pgmap v668: 158 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 155 active+clean; 2.6 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 751 B/s rd, 0 B/s wr, 688 op/s; 4140/130623 objects degraded (3.169%); 13 MiB/s, 216 objects/s recovering 2024-09-21T14:02:07.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:07 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T14:02:07.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:07 smithi187 ceph-mon[33032]: osdmap e361: 8 total, 8 up, 8 in 2024-09-21T14:02:07.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:07 smithi187 ceph-mon[33032]: 1.5 scrub starts 2024-09-21T14:02:07.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:07 smithi187 ceph-mon[33032]: 1.5 scrub ok 2024-09-21T14:02:07.789 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:02:07.823 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:07.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:07.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:02:07.925 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:07.958 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:07.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:07.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:08.060 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:02:08.089 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:08.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:08.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:02:08.191 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:02:08.209 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-09-21T14:02:08.212 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:08.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:08.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:02:08.314 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:02:08.337 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:08.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:08.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:02:08.358 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd reweight 7 1 2024-09-21T14:02:08.439 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:08.463 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:08.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:08.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:08.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:08 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3967470601' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:02:08.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:08 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:02:08.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:08 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3967470601' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:02:08.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:08 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:02:08.550 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:02:08.565 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:02:08.596 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:08.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:08.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:02:08.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:08 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3967470601' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:02:08.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:08 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:02:08.698 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:02:08.719 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:08.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:08.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:02:08.821 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:08.844 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:08.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:08.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:08.945 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:08.967 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:08.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:08.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:09.069 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:09.093 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:09.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:09.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:09.108 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:02:09.196 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:09.221 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:09.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:09.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:09.323 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:09.357 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:09.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:09.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:09.359 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:09 smithi049 ceph-mon[40489]: pgmap v670: 158 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 155 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 752 B/s rd, 0 B/s wr, 460 op/s; 4140/126618 objects degraded (3.270%); 7.2 MiB/s, 118 objects/s recovering 2024-09-21T14:02:09.359 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:09 smithi049 ceph-mon[40489]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:02:09.359 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:09 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T14:02:09.359 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:09 smithi049 ceph-mon[40489]: osdmap e362: 8 total, 8 up, 8 in 2024-09-21T14:02:09.359 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:09 smithi049 ceph-mon[30080]: pgmap v670: 158 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 155 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 752 B/s rd, 0 B/s wr, 460 op/s; 4140/126618 objects degraded (3.270%); 7.2 MiB/s, 118 objects/s recovering 2024-09-21T14:02:09.360 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:09 smithi049 ceph-mon[30080]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:02:09.360 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:09 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T14:02:09.360 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:09 smithi049 ceph-mon[30080]: osdmap e362: 8 total, 8 up, 8 in 2024-09-21T14:02:09.459 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:02:09.497 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:09.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:09.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:02:09.598 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:09.638 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:09.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:09.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:09.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:09 smithi187 ceph-mon[33032]: pgmap v670: 158 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 155 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 752 B/s rd, 0 B/s wr, 460 op/s; 4140/126618 objects degraded (3.270%); 7.2 MiB/s, 118 objects/s recovering 2024-09-21T14:02:09.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:09 smithi187 ceph-mon[33032]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:02:09.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:09 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T14:02:09.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:09 smithi187 ceph-mon[33032]: osdmap e362: 8 total, 8 up, 8 in 2024-09-21T14:02:09.740 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:02:09.775 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:09.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:09.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:02:09.876 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:09.913 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:09.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:09.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:09.933 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T14:02:09.967 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T14:02:09.967 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:02:10.016 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:02:10.049 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:10.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:10.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:02:10.150 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:10.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:10.193 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:10.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:10.194 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:02:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:02:10.295 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:02:10.324 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:10.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:10.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:02:10.427 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:02:10.455 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:10.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:10.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:02:10.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:10 smithi049 ceph-mon[30080]: osdmap e363: 8 total, 8 up, 8 in 2024-09-21T14:02:10.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:10 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:02:10.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:10 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 4140/126618 objects degraded (3.270%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T14:02:10.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:10 smithi049 ceph-mon[30080]: osdmap e364: 8 total, 8 up, 8 in 2024-09-21T14:02:10.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:10 smithi049 ceph-mon[40489]: osdmap e363: 8 total, 8 up, 8 in 2024-09-21T14:02:10.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:10 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:02:10.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:10 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 4140/126618 objects degraded (3.270%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T14:02:10.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:10 smithi049 ceph-mon[40489]: osdmap e364: 8 total, 8 up, 8 in 2024-09-21T14:02:10.557 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:02:10.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:10.603 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:10.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:02:10.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:10 smithi187 ceph-mon[33032]: osdmap e363: 8 total, 8 up, 8 in 2024-09-21T14:02:10.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:10 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:02:10.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:10 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 4140/126618 objects degraded (3.270%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T14:02:10.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:10 smithi187 ceph-mon[33032]: osdmap e364: 8 total, 8 up, 8 in 2024-09-21T14:02:10.704 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:02:10.739 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:10.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:10.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:02:10.841 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:10.887 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:10.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:10.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:10.989 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:11.004 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:02:10 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:02:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:02:11.019 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:11.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:11.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:11.098 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T14:02:11.119 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T14:02:11.120 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:11.146 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:02:11.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:11.216 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:11.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:11.234 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.7 to 1 (10000) 2024-09-21T14:02:11.317 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:11.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:11.372 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:11.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:11.398 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:11 smithi049 ceph-mon[40489]: pgmap v673: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 155 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 630 op/s; 3994/124443 objects degraded (3.210%); 9.9 MiB/s, 164 objects/s recovering 2024-09-21T14:02:11.399 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:11 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3439481938' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T14:02:11.399 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:11 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T14:02:11.399 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:11 smithi049 ceph-mon[30080]: pgmap v673: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 155 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 630 op/s; 3994/124443 objects degraded (3.210%); 9.9 MiB/s, 164 objects/s recovering 2024-09-21T14:02:11.399 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:11 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3439481938' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T14:02:11.399 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:11 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T14:02:11.473 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:11.520 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:11.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:11.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:11.622 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:11.658 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:11.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:11.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:11.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:11 smithi187 ceph-mon[33032]: pgmap v673: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 155 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 630 op/s; 3994/124443 objects degraded (3.210%); 9.9 MiB/s, 164 objects/s recovering 2024-09-21T14:02:11.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:11 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3439481938' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T14:02:11.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:11 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T14:02:11.760 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:02:11.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:11.799 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:11.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:02:11.900 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:02:11.932 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:11.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:11.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:02:12.034 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:02:12.072 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:12.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:12.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:02:12.174 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:12.201 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:12.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:12.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:12.303 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:12.337 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:12.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:12.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:12.439 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:12.476 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:12.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:12.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:12.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:12 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T14:02:12.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:12 smithi049 ceph-mon[40489]: osdmap e365: 8 total, 8 up, 8 in 2024-09-21T14:02:12.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:12 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T14:02:12.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:12 smithi049 ceph-mon[30080]: osdmap e365: 8 total, 8 up, 8 in 2024-09-21T14:02:12.578 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:02:12.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:12.614 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:12.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:02:12.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:12 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T14:02:12.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:12 smithi187 ceph-mon[33032]: osdmap e365: 8 total, 8 up, 8 in 2024-09-21T14:02:12.715 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:02:12.747 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:12.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:12.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:02:12.849 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:12.885 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:12.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:12.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:12.987 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:13.023 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:13.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:13.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:13.125 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:13.169 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:13.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:13.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:13.271 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:02:13.320 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:13.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:13.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:02:13.323 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:02:13.323 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:02:13.363 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:02:13.363 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:02:13.396 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:02:13.396 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:02:13.396 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:13 smithi049 ceph-mon[40489]: pgmap v676: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 156 active+clean; 2.3 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 933 op/s; 3860/115422 objects degraded (3.344%); 2.1 MiB/s, 183 objects/s recovering 2024-09-21T14:02:13.397 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:13 smithi049 ceph-mon[30080]: pgmap v676: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 156 active+clean; 2.3 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 933 op/s; 3860/115422 objects degraded (3.344%); 2.1 MiB/s, 183 objects/s recovering 2024-09-21T14:02:13.421 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:02:13.433 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:02:13.433 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:02:13.457 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:13.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:13.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:02:13.468 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:02:13.468 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:02:13.500 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:02:13.500 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:02:13.535 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:02:13.536 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:02:13.560 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:02:13.571 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:02:13.571 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:02:13.591 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:13.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:13.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:02:13.692 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:13.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:13 smithi187 ceph-mon[33032]: pgmap v676: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 156 active+clean; 2.3 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 933 op/s; 3860/115422 objects degraded (3.344%); 2.1 MiB/s, 183 objects/s recovering 2024-09-21T14:02:13.726 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:13.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:13.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:13.828 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:13.867 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd reweight 6 1 2024-09-21T14:02:13.874 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:13.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:13.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:13.976 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:14.026 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:14.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:14.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:14.127 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:14.171 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:14.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:14.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:14.272 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:14.311 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:14.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:14.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:14.412 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:02:14.450 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:14.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:14.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:02:14.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:14 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1155858272' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:02:14.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:14 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:02:14.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:14 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1155858272' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:02:14.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:14 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:02:14.552 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:02:14.590 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:14.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:14.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:02:14.693 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:14.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:14 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1155858272' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:02:14.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:14 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:02:14.744 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:14.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:14.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:14.825 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:02:14.833 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:02:14.838 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:02:14.845 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:14.907 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:14.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:14.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:15.012 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:15.056 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:15.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:15.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:15.158 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:02:15.188 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:15.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:15.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:02:15.290 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:02:15.329 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:15.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:15.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:02:15.431 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:15.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:15.469 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:15.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:15.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:15 smithi049 ceph-mon[40489]: pgmap v677: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 156 active+clean; 2.3 GiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 886 B/s rd, 0 B/s wr, 701 op/s; 3860/114486 objects degraded (3.372%); 1.5 MiB/s, 126 objects/s recovering 2024-09-21T14:02:15.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:15 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 3860/114486 objects degraded (3.372%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T14:02:15.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:15 smithi049 ceph-mon[40489]: osdmap e366: 8 total, 8 up, 8 in 2024-09-21T14:02:15.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:15 smithi049 ceph-mon[30080]: pgmap v677: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 156 active+clean; 2.3 GiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 886 B/s rd, 0 B/s wr, 701 op/s; 3860/114486 objects degraded (3.372%); 1.5 MiB/s, 126 objects/s recovering 2024-09-21T14:02:15.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:15 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 3860/114486 objects degraded (3.372%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T14:02:15.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:15 smithi049 ceph-mon[30080]: osdmap e366: 8 total, 8 up, 8 in 2024-09-21T14:02:15.570 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:02:15.611 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:15.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:15.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:02:15.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:15 smithi187 ceph-mon[33032]: pgmap v677: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 156 active+clean; 2.3 GiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 886 B/s rd, 0 B/s wr, 701 op/s; 3860/114486 objects degraded (3.372%); 1.5 MiB/s, 126 objects/s recovering 2024-09-21T14:02:15.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:15 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 3860/114486 objects degraded (3.372%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T14:02:15.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:15 smithi187 ceph-mon[33032]: osdmap e366: 8 total, 8 up, 8 in 2024-09-21T14:02:15.713 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:15.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:15.752 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:15.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:15.853 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:02:15.899 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:15.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:15.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:02:16.001 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:02:16.037 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:16.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:16.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:02:16.138 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:02:16.170 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:16.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:16.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:02:16.271 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:16.315 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:16.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:16.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:16.372 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T14:02:16.373 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-09-21T14:02:16.373 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-09-21T14:02:16.373 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-09-21T14:02:16.373 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.373 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232022 8.f 8.d7b7a40f (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T14:02:16.373 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.700289+0000", 2024-09-21T14:02:16.373 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 304.67035679899999, 2024-09-21T14:02:16.373 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97395471300000003, 2024-09-21T14:02:16.373 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:02:16.373 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:02:16.373 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T14:02:16.373 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:02:16.374 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T14:02:16.374 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:02:16.374 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232022 2024-09-21T14:02:16.374 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.374 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:02:16.374 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.374 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:02:16.374 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700289+0000", 2024-09-21T14:02:16.374 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:02:16.374 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.374 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.375 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:02:16.375 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700289+0000", 2024-09-21T14:02:16.375 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:02:16.375 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.375 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.375 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:02:16.375 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700291+0000", 2024-09-21T14:02:16.375 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5850000000000001e-06 2024-09-21T14:02:16.376 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.376 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.376 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:02:16.376 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700296+0000", 2024-09-21T14:02:16.376 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9749999999999998e-06 2024-09-21T14:02:16.376 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.376 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.376 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:02:16.376 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700306+0000", 2024-09-21T14:02:16.376 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9310000000000003e-06 2024-09-21T14:02:16.376 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.376 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.376 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:02:16.376 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700317+0000", 2024-09-21T14:02:16.377 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1199000000000001e-05 2024-09-21T14:02:16.377 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.377 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.377 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:02:16.377 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.711626+0000", 2024-09-21T14:02:16.377 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.011309009 2024-09-21T14:02:16.377 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.377 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.377 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:02:16.377 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.711637+0000", 2024-09-21T14:02:16.377 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1182000000000001e-05 2024-09-21T14:02:16.377 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.378 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.378 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:02:16.378 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674230+0000", 2024-09-21T14:02:16.378 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.962593535 2024-09-21T14:02:16.378 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.378 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.378 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:02:16.378 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674244+0000", 2024-09-21T14:02:16.378 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3297e-05 2024-09-21T14:02:16.378 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:02:16.378 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:02:16.379 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:02:16.379 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.379 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.380 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232025 8.f 8.4e5a668f (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T14:02:16.380 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.700329+0000", 2024-09-21T14:02:16.380 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 304.670317001, 2024-09-21T14:02:16.380 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97395089300000004, 2024-09-21T14:02:16.380 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:02:16.380 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:02:16.380 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T14:02:16.380 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:02:16.380 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T14:02:16.380 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:02:16.380 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232025 2024-09-21T14:02:16.380 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.380 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:02:16.380 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.381 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:02:16.381 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700329+0000", 2024-09-21T14:02:16.381 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:02:16.381 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.381 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.381 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:02:16.381 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700329+0000", 2024-09-21T14:02:16.381 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:02:16.381 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.381 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.382 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:02:16.382 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700329+0000", 2024-09-21T14:02:16.382 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7100000000000002e-07 2024-09-21T14:02:16.382 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.382 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.382 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:02:16.382 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700332+0000", 2024-09-21T14:02:16.382 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6589999999999999e-06 2024-09-21T14:02:16.382 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.382 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.382 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:02:16.383 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700339+0000", 2024-09-21T14:02:16.383 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8940000000000004e-06 2024-09-21T14:02:16.383 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.383 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.383 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:02:16.383 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700345+0000", 2024-09-21T14:02:16.383 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6219999999999997e-06 2024-09-21T14:02:16.383 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.383 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.383 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:02:16.383 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.711662+0000", 2024-09-21T14:02:16.383 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.011317522 2024-09-21T14:02:16.383 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.383 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.383 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:02:16.384 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.711669+0000", 2024-09-21T14:02:16.384 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2570000000000001e-06 2024-09-21T14:02:16.384 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.384 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.384 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:02:16.384 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674270+0000", 2024-09-21T14:02:16.384 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.96260044300000003 2024-09-21T14:02:16.384 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.384 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.384 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:02:16.384 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674280+0000", 2024-09-21T14:02:16.384 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0025e-05 2024-09-21T14:02:16.384 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:02:16.384 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:02:16.385 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:02:16.385 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.385 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.385 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232033 8.f 8.d949c02f (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T14:02:16.385 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.700352+0000", 2024-09-21T14:02:16.385 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 304.67029360100003, 2024-09-21T14:02:16.385 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97396006099999999, 2024-09-21T14:02:16.385 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:02:16.385 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:02:16.385 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T14:02:16.385 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:02:16.385 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T14:02:16.385 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:02:16.385 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232033 2024-09-21T14:02:16.386 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.386 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:02:16.386 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.386 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:02:16.386 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700352+0000", 2024-09-21T14:02:16.386 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:02:16.386 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.386 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.386 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:02:16.386 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700352+0000", 2024-09-21T14:02:16.386 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:02:16.386 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.386 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.386 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:02:16.387 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700353+0000", 2024-09-21T14:02:16.387 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.89e-07 2024-09-21T14:02:16.387 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.387 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.387 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:02:16.387 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700355+0000", 2024-09-21T14:02:16.387 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1840000000000002e-06 2024-09-21T14:02:16.387 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.387 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.387 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:02:16.387 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700364+0000", 2024-09-21T14:02:16.387 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4919999999999999e-06 2024-09-21T14:02:16.387 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.388 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.388 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:02:16.388 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700372+0000", 2024-09-21T14:02:16.388 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9659999999999996e-06 2024-09-21T14:02:16.388 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.388 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.388 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:02:16.388 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.711693+0000", 2024-09-21T14:02:16.388 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.011321059 2024-09-21T14:02:16.388 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.388 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.388 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:02:16.388 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.711701+0000", 2024-09-21T14:02:16.389 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3640000000000001e-06 2024-09-21T14:02:16.389 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.389 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.389 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:02:16.389 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674303+0000", 2024-09-21T14:02:16.389 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.96260253100000004 2024-09-21T14:02:16.389 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.389 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.390 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:02:16.390 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674312+0000", 2024-09-21T14:02:16.390 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0759999999999992e-06 2024-09-21T14:02:16.390 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:02:16.390 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:02:16.390 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:02:16.390 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.391 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.391 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232051 8.0 8.85c8f300 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T14:02:16.391 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.700380+0000", 2024-09-21T14:02:16.391 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 304.67026589800003, 2024-09-21T14:02:16.391 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97451152799999996, 2024-09-21T14:02:16.391 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:02:16.391 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:02:16.391 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T14:02:16.391 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:02:16.391 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T14:02:16.391 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:02:16.391 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232051 2024-09-21T14:02:16.391 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.392 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:02:16.392 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.392 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:02:16.392 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700380+0000", 2024-09-21T14:02:16.392 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:02:16.392 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.392 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.392 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:02:16.392 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700380+0000", 2024-09-21T14:02:16.392 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:02:16.392 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.392 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.392 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:02:16.392 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700380+0000", 2024-09-21T14:02:16.392 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1699999999999999e-07 2024-09-21T14:02:16.393 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.393 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.393 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:02:16.393 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700383+0000", 2024-09-21T14:02:16.393 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1600000000000001e-06 2024-09-21T14:02:16.393 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.393 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.393 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:02:16.393 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700389+0000", 2024-09-21T14:02:16.393 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6080000000000001e-06 2024-09-21T14:02:16.393 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.393 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.393 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:02:16.393 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700394+0000", 2024-09-21T14:02:16.394 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9810000000000003e-06 2024-09-21T14:02:16.394 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.394 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.394 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:02:16.394 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.715050+0000", 2024-09-21T14:02:16.394 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.014656269 2024-09-21T14:02:16.394 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.394 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.394 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:02:16.394 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.715056+0000", 2024-09-21T14:02:16.394 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0129999999999999e-06 2024-09-21T14:02:16.394 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.395 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.395 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:02:16.395 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674876+0000", 2024-09-21T14:02:16.395 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.95981962700000001 2024-09-21T14:02:16.395 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.395 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.395 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:02:16.395 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674892+0000", 2024-09-21T14:02:16.395 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5452999999999999e-05 2024-09-21T14:02:16.395 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:02:16.395 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:02:16.395 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:02:16.395 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.395 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.396 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232059 8.f 8.b757f3bf (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T14:02:16.396 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.889250+0000", 2024-09-21T14:02:16.396 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 304.48139608499997, 2024-09-21T14:02:16.396 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.78509475900000003, 2024-09-21T14:02:16.396 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:02:16.396 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:02:16.396 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T14:02:16.396 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:02:16.396 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T14:02:16.396 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:02:16.396 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232059 2024-09-21T14:02:16.396 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.396 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:02:16.396 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.397 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:02:16.397 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889250+0000", 2024-09-21T14:02:16.397 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:02:16.397 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.397 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.397 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:02:16.397 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889250+0000", 2024-09-21T14:02:16.397 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:02:16.397 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.397 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.397 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:02:16.397 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889252+0000", 2024-09-21T14:02:16.397 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9130000000000001e-06 2024-09-21T14:02:16.397 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.398 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.398 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:02:16.398 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889258+0000", 2024-09-21T14:02:16.398 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7410000000000003e-06 2024-09-21T14:02:16.398 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.398 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.398 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:02:16.398 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889360+0000", 2024-09-21T14:02:16.398 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000101382 2024-09-21T14:02:16.398 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.398 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.398 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:02:16.398 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889391+0000", 2024-09-21T14:02:16.399 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1032999999999999e-05 2024-09-21T14:02:16.399 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.399 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.399 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:02:16.399 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889500+0000", 2024-09-21T14:02:16.399 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000109194 2024-09-21T14:02:16.399 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.399 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.399 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:02:16.399 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889512+0000", 2024-09-21T14:02:16.399 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2368000000000001e-05 2024-09-21T14:02:16.399 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.399 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.399 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:02:16.400 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674335+0000", 2024-09-21T14:02:16.400 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.78482304000000003 2024-09-21T14:02:16.400 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.400 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.400 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:02:16.400 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674345+0000", 2024-09-21T14:02:16.400 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0880000000000001e-06 2024-09-21T14:02:16.400 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:02:16.400 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:02:16.400 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:02:16.400 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.400 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.400 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232060 8.b 8.6c14963b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T14:02:16.400 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.889442+0000", 2024-09-21T14:02:16.401 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 304.481203897, 2024-09-21T14:02:16.401 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.78518956100000004, 2024-09-21T14:02:16.401 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:02:16.401 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:02:16.401 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T14:02:16.401 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:02:16.401 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T14:02:16.401 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:02:16.401 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232060 2024-09-21T14:02:16.401 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.401 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:02:16.401 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.401 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:02:16.401 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889442+0000", 2024-09-21T14:02:16.402 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:02:16.402 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.402 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.402 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:02:16.402 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889442+0000", 2024-09-21T14:02:16.402 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:02:16.402 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.402 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.402 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:02:16.402 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889443+0000", 2024-09-21T14:02:16.402 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9100000000000003e-07 2024-09-21T14:02:16.402 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.402 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.402 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:02:16.403 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889447+0000", 2024-09-21T14:02:16.403 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4939999999999997e-06 2024-09-21T14:02:16.403 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.403 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.403 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:02:16.403 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889457+0000", 2024-09-21T14:02:16.403 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0145e-05 2024-09-21T14:02:16.403 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.403 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.403 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:02:16.403 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889465+0000", 2024-09-21T14:02:16.403 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2749999999999998e-06 2024-09-21T14:02:16.403 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.404 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.404 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:02:16.404 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889550+0000", 2024-09-21T14:02:16.404 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5000999999999995e-05 2024-09-21T14:02:16.404 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.404 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.404 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:02:16.404 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889561+0000", 2024-09-21T14:02:16.404 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.129e-05 2024-09-21T14:02:16.404 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.404 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.404 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:02:16.404 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674619+0000", 2024-09-21T14:02:16.404 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.785057703 2024-09-21T14:02:16.405 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.405 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.405 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:02:16.405 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674632+0000", 2024-09-21T14:02:16.405 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2962000000000001e-05 2024-09-21T14:02:16.405 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:02:16.405 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:02:16.405 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:02:16.405 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.405 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.405 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232068 8.f 8.d5b3687f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T14:02:16.405 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.894747+0000", 2024-09-21T14:02:16.405 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 304.47589920899998, 2024-09-21T14:02:16.405 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.77963143999999995, 2024-09-21T14:02:16.405 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:02:16.406 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:02:16.406 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T14:02:16.406 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:02:16.406 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T14:02:16.406 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:02:16.406 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232068 2024-09-21T14:02:16.406 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.406 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:02:16.406 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.406 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:02:16.406 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.894747+0000", 2024-09-21T14:02:16.406 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:02:16.406 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.406 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.406 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:02:16.407 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.894747+0000", 2024-09-21T14:02:16.407 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:02:16.407 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.407 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.407 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:02:16.407 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.894749+0000", 2024-09-21T14:02:16.407 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4849999999999999e-06 2024-09-21T14:02:16.407 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.407 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.407 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:02:16.407 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.894761+0000", 2024-09-21T14:02:16.407 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1809e-05 2024-09-21T14:02:16.407 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.407 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.408 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:02:16.408 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.894780+0000", 2024-09-21T14:02:16.408 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8593999999999998e-05 2024-09-21T14:02:16.408 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.408 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.408 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:02:16.408 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.894789+0000", 2024-09-21T14:02:16.408 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2839999999999995e-06 2024-09-21T14:02:16.408 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.408 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.408 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:02:16.408 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.894865+0000", 2024-09-21T14:02:16.409 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6150999999999996e-05 2024-09-21T14:02:16.409 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.409 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.409 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:02:16.409 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.894874+0000", 2024-09-21T14:02:16.409 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2450000000000004e-06 2024-09-21T14:02:16.409 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.409 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.409 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:02:16.409 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674368+0000", 2024-09-21T14:02:16.409 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.77949425999999999 2024-09-21T14:02:16.409 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.409 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.409 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:02:16.410 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674378+0000", 2024-09-21T14:02:16.410 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6120000000000005e-06 2024-09-21T14:02:16.410 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:02:16.410 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:02:16.410 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:02:16.410 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.410 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.410 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232074 8.0 8.57810 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T14:02:16.410 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.901181+0000", 2024-09-21T14:02:16.410 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 304.46946469599999, 2024-09-21T14:02:16.410 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.77382758200000001, 2024-09-21T14:02:16.410 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:02:16.410 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:02:16.411 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T14:02:16.411 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:02:16.411 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T14:02:16.411 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:02:16.411 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232074 2024-09-21T14:02:16.411 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.411 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:02:16.411 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.411 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:02:16.411 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.901181+0000", 2024-09-21T14:02:16.411 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:02:16.411 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.411 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.411 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:02:16.411 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.901181+0000", 2024-09-21T14:02:16.412 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:02:16.412 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.412 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.412 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:02:16.412 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.901184+0000", 2024-09-21T14:02:16.412 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.8660000000000002e-06 2024-09-21T14:02:16.412 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.412 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.412 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:02:16.412 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.901193+0000", 2024-09-21T14:02:16.412 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1549999999999996e-06 2024-09-21T14:02:16.412 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.412 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.412 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:02:16.412 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.901215+0000", 2024-09-21T14:02:16.413 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2272e-05 2024-09-21T14:02:16.413 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.413 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.413 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:02:16.413 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.901227+0000", 2024-09-21T14:02:16.413 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1116e-05 2024-09-21T14:02:16.413 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.413 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.413 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:02:16.413 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.901351+0000", 2024-09-21T14:02:16.413 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000124172 2024-09-21T14:02:16.413 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.413 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.413 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:02:16.414 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.901368+0000", 2024-09-21T14:02:16.414 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7008999999999999e-05 2024-09-21T14:02:16.414 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.414 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.414 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:02:16.414 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674970+0000", 2024-09-21T14:02:16.414 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.77360225199999999 2024-09-21T14:02:16.414 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.414 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.414 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:02:16.414 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.675009+0000", 2024-09-21T14:02:16.414 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8739999999999998e-05 2024-09-21T14:02:16.414 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:02:16.414 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:02:16.415 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:02:16.415 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.415 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.415 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232078 8.b 8.7f49789b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T14:02:16.415 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.904662+0000", 2024-09-21T14:02:16.415 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 304.46598429099998, 2024-09-21T14:02:16.415 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.77001060399999999, 2024-09-21T14:02:16.415 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:02:16.415 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:02:16.415 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T14:02:16.415 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:02:16.415 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T14:02:16.415 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:02:16.415 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232078 2024-09-21T14:02:16.415 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.416 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:02:16.416 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.416 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:02:16.416 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904662+0000", 2024-09-21T14:02:16.416 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:02:16.416 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.416 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.416 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:02:16.416 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904662+0000", 2024-09-21T14:02:16.416 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:02:16.416 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.416 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.416 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:02:16.417 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904663+0000", 2024-09-21T14:02:16.417 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.548e-06 2024-09-21T14:02:16.417 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.417 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.417 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:02:16.417 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904669+0000", 2024-09-21T14:02:16.417 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9619999999999998e-06 2024-09-21T14:02:16.417 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.417 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.417 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:02:16.417 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904684+0000", 2024-09-21T14:02:16.417 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4718e-05 2024-09-21T14:02:16.417 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.417 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.418 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:02:16.418 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904691+0000", 2024-09-21T14:02:16.418 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.861e-06 2024-09-21T14:02:16.418 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.418 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.418 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:02:16.418 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904750+0000", 2024-09-21T14:02:16.418 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9533000000000001e-05 2024-09-21T14:02:16.418 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.418 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.418 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:02:16.418 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904757+0000", 2024-09-21T14:02:16.418 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0369999999999997e-06 2024-09-21T14:02:16.419 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.419 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.419 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:02:16.419 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674661+0000", 2024-09-21T14:02:16.419 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.769903754 2024-09-21T14:02:16.419 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.419 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.419 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:02:16.419 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674672+0000", 2024-09-21T14:02:16.419 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1191e-05 2024-09-21T14:02:16.419 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:02:16.419 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:02:16.419 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:02:16.419 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.420 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.420 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232079 8.b 8.5c3e90b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T14:02:16.420 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.904710+0000", 2024-09-21T14:02:16.420 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 304.46593593599999, 2024-09-21T14:02:16.420 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.76999876, 2024-09-21T14:02:16.420 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:02:16.420 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:02:16.420 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T14:02:16.420 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:02:16.420 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T14:02:16.420 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:02:16.420 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232079 2024-09-21T14:02:16.420 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.421 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:02:16.421 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.421 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:02:16.421 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904710+0000", 2024-09-21T14:02:16.421 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:02:16.421 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.421 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.421 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:02:16.421 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904710+0000", 2024-09-21T14:02:16.421 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:02:16.421 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.421 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.421 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:02:16.421 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904711+0000", 2024-09-21T14:02:16.421 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8199999999999999e-07 2024-09-21T14:02:16.422 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.422 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.422 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:02:16.422 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904714+0000", 2024-09-21T14:02:16.422 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9079999999999999e-06 2024-09-21T14:02:16.422 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.422 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.422 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:02:16.422 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904726+0000", 2024-09-21T14:02:16.422 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2514000000000001e-05 2024-09-21T14:02:16.422 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.422 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.423 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:02:16.423 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904733+0000", 2024-09-21T14:02:16.423 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2690000000000002e-06 2024-09-21T14:02:16.423 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.423 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.423 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:02:16.423 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904784+0000", 2024-09-21T14:02:16.423 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1007000000000001e-05 2024-09-21T14:02:16.423 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.423 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.423 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:02:16.423 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904794+0000", 2024-09-21T14:02:16.423 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7419999999999992e-06 2024-09-21T14:02:16.424 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.424 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.424 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:02:16.424 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674699+0000", 2024-09-21T14:02:16.424 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.76990482900000001 2024-09-21T14:02:16.424 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.424 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.424 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:02:16.424 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674709+0000", 2024-09-21T14:02:16.424 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8090000000000004e-06 2024-09-21T14:02:16.424 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:02:16.424 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:02:16.424 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:02:16.424 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.425 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.425 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25421.0:113916 9.8 9:1f612498:::benchmark_data_smithi049_138504_object113915:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-21T14:02:16.425 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:59:25.602805+0000", 2024-09-21T14:02:16.425 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 170.76784050399999, 2024-09-21T14:02:16.425 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.34059980099999998, 2024-09-21T14:02:16.425 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:02:16.425 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:02:16.425 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:02:16.425 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:02:16.425 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25421", 2024-09-21T14:02:16.425 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:02:16.425 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 113916 2024-09-21T14:02:16.425 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.425 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:02:16.426 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.426 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:02:16.426 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.602805+0000", 2024-09-21T14:02:16.426 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:02:16.426 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.426 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.426 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:02:16.426 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.602805+0000", 2024-09-21T14:02:16.426 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:02:16.426 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.426 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.426 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:02:16.426 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.602807+0000", 2024-09-21T14:02:16.427 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7099999999999999e-06 2024-09-21T14:02:16.427 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.427 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.427 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:02:16.427 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.602830+0000", 2024-09-21T14:02:16.427 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2758000000000001e-05 2024-09-21T14:02:16.427 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.427 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.427 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:02:16.427 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.602847+0000", 2024-09-21T14:02:16.427 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6721000000000001e-05 2024-09-21T14:02:16.427 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.427 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.428 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:02:16.428 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.602854+0000", 2024-09-21T14:02:16.428 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9100000000000005e-06 2024-09-21T14:02:16.428 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.428 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.428 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:02:16.428 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.602925+0000", 2024-09-21T14:02:16.428 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0088999999999999e-05 2024-09-21T14:02:16.428 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.428 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.428 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:02:16.428 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.603043+0000", 2024-09-21T14:02:16.428 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011804200000000001 2024-09-21T14:02:16.428 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.429 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.429 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,5", 2024-09-21T14:02:16.429 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.603177+0000", 2024-09-21T14:02:16.429 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000134821 2024-09-21T14:02:16.429 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.429 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.429 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:02:16.429 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.606802+0000", 2024-09-21T14:02:16.429 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0036243880000000001 2024-09-21T14:02:16.429 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.429 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.429 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:02:16.429 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.608461+0000", 2024-09-21T14:02:16.429 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0016595710000000001 2024-09-21T14:02:16.430 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.430 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.430 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:02:16.430 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.943300+0000", 2024-09-21T14:02:16.430 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.33483894400000003 2024-09-21T14:02:16.430 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.430 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.430 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:02:16.430 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.943360+0000", 2024-09-21T14:02:16.430 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9401999999999998e-05 2024-09-21T14:02:16.430 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.430 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.430 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:02:16.430 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.943405+0000", 2024-09-21T14:02:16.431 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5445000000000003e-05 2024-09-21T14:02:16.431 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:02:16.431 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:02:16.431 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:02:16.431 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.431 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.431 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25421.0:113943 9.8 9:1e81a0a3:::benchmark_data_smithi049_138504_object113942:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-21T14:02:16.431 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:59:25.623062+0000", 2024-09-21T14:02:16.431 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 170.747583741, 2024-09-21T14:02:16.431 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.32046552900000003, 2024-09-21T14:02:16.431 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:02:16.431 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:02:16.431 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:02:16.431 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:02:16.432 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25421", 2024-09-21T14:02:16.432 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:02:16.432 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 113943 2024-09-21T14:02:16.432 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.432 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:02:16.432 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.432 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:02:16.432 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.623062+0000", 2024-09-21T14:02:16.432 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:02:16.432 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.432 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.432 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:02:16.433 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:16.434 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.623062+0000", 2024-09-21T14:02:16.434 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:02:16.434 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.434 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.434 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:02:16.434 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.623063+0000", 2024-09-21T14:02:16.434 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0210000000000001e-06 2024-09-21T14:02:16.434 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.434 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.434 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:02:16.434 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.623089+0000", 2024-09-21T14:02:16.435 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.567e-05 2024-09-21T14:02:16.435 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.435 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.435 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:02:16.435 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.623103+0000", 2024-09-21T14:02:16.435 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4248e-05 2024-09-21T14:02:16.435 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.435 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.435 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:02:16.435 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.623109+0000", 2024-09-21T14:02:16.435 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5949999999999998e-06 2024-09-21T14:02:16.435 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.435 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.435 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:02:16.435 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.623150+0000", 2024-09-21T14:02:16.436 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0868999999999998e-05 2024-09-21T14:02:16.436 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.436 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.436 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:02:16.436 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.623240+0000", 2024-09-21T14:02:16.436 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0562000000000005e-05 2024-09-21T14:02:16.436 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.436 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.436 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,5", 2024-09-21T14:02:16.436 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.623353+0000", 2024-09-21T14:02:16.436 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000113249 2024-09-21T14:02:16.436 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.436 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.436 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:02:16.437 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.625438+0000", 2024-09-21T14:02:16.437 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0020847750000000001 2024-09-21T14:02:16.437 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.437 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.437 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:02:16.437 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.626378+0000", 2024-09-21T14:02:16.437 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00093971199999999999 2024-09-21T14:02:16.437 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.437 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.437 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:02:16.437 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.943444+0000", 2024-09-21T14:02:16.437 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.317066615 2024-09-21T14:02:16.437 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.437 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.437 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:02:16.438 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.943487+0000", 2024-09-21T14:02:16.438 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2933999999999999e-05 2024-09-21T14:02:16.438 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.438 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.438 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:02:16.438 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.943528+0000", 2024-09-21T14:02:16.438 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0278999999999998e-05 2024-09-21T14:02:16.438 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:02:16.438 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:02:16.438 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:02:16.438 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.438 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.438 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25421.0:113948 9.8 9:1fea698d:::benchmark_data_smithi049_138504_object113947:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-21T14:02:16.439 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:59:25.630511+0000", 2024-09-21T14:02:16.439 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 170.740134455, 2024-09-21T14:02:16.439 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.31617158400000001, 2024-09-21T14:02:16.439 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:02:16.439 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:02:16.439 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:02:16.439 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:02:16.439 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25421", 2024-09-21T14:02:16.439 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:02:16.439 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 113948 2024-09-21T14:02:16.439 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.439 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:02:16.439 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.439 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:02:16.440 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.630511+0000", 2024-09-21T14:02:16.440 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:02:16.440 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.440 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.440 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:02:16.440 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.630511+0000", 2024-09-21T14:02:16.440 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:02:16.440 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.440 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.440 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:02:16.440 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.630512+0000", 2024-09-21T14:02:16.440 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.16e-07 2024-09-21T14:02:16.440 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.440 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.441 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:02:16.441 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.630531+0000", 2024-09-21T14:02:16.441 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8845999999999999e-05 2024-09-21T14:02:16.441 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.441 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.441 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:02:16.441 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.630542+0000", 2024-09-21T14:02:16.441 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0651e-05 2024-09-21T14:02:16.441 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.441 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.441 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:02:16.441 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.630547+0000", 2024-09-21T14:02:16.441 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8770000000000001e-06 2024-09-21T14:02:16.441 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.442 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.442 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:02:16.442 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.630585+0000", 2024-09-21T14:02:16.442 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8488999999999997e-05 2024-09-21T14:02:16.442 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.442 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.442 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:02:16.442 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.630660+0000", 2024-09-21T14:02:16.442 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5290000000000006e-05 2024-09-21T14:02:16.442 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.442 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.442 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,5", 2024-09-21T14:02:16.443 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.630760+0000", 2024-09-21T14:02:16.443 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9834000000000007e-05 2024-09-21T14:02:16.443 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.443 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.443 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:02:16.443 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.632607+0000", 2024-09-21T14:02:16.443 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001847174 2024-09-21T14:02:16.443 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.443 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.443 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:02:16.443 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.633151+0000", 2024-09-21T14:02:16.443 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00054377600000000005 2024-09-21T14:02:16.443 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.443 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.444 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:02:16.444 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.946575+0000", 2024-09-21T14:02:16.444 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.31342341099999999 2024-09-21T14:02:16.444 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.444 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.444 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:02:16.444 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.946632+0000", 2024-09-21T14:02:16.444 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7055999999999997e-05 2024-09-21T14:02:16.444 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.444 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.444 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:02:16.444 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.946683+0000", 2024-09-21T14:02:16.444 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1264000000000002e-05 2024-09-21T14:02:16.444 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:02:16.445 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:02:16.445 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:02:16.445 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.445 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.445 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25421.0:172314 9.4 e315/260 9:2b749f7e:::benchmark_data_smithi049_138504_object50906:head v 315'10563, mlcod=315'10559)", 2024-09-21T14:02:16.445 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:00:33.250136+0000", 2024-09-21T14:02:16.445 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 103.12050982300001, 2024-09-21T14:02:16.445 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.270468655, 2024-09-21T14:02:16.445 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:02:16.445 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:02:16.445 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:02:16.445 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:02:16.445 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.445 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:02:16.446 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250136+0000", 2024-09-21T14:02:16.446 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:02:16.446 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.446 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.446 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:02:16.446 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250136+0000", 2024-09-21T14:02:16.446 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:02:16.446 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.446 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.446 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:02:16.446 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250137+0000", 2024-09-21T14:02:16.446 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1999999999999999e-07 2024-09-21T14:02:16.446 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.446 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.446 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:02:16.447 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250142+0000", 2024-09-21T14:02:16.447 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5609999999999998e-06 2024-09-21T14:02:16.447 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.447 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.447 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:02:16.447 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250150+0000", 2024-09-21T14:02:16.447 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1310000000000006e-06 2024-09-21T14:02:16.447 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.447 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.447 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:02:16.447 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250156+0000", 2024-09-21T14:02:16.447 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7740000000000001e-06 2024-09-21T14:02:16.448 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.448 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.448 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:02:16.448 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.253391+0000", 2024-09-21T14:02:16.448 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0032349969999999999 2024-09-21T14:02:16.448 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.448 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.448 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:02:16.448 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.253410+0000", 2024-09-21T14:02:16.448 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9097e-05 2024-09-21T14:02:16.448 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.448 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.448 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:02:16.449 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.520555+0000", 2024-09-21T14:02:16.449 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26714502600000001 2024-09-21T14:02:16.449 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.449 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.449 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:02:16.449 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.520605+0000", 2024-09-21T14:02:16.449 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9449e-05 2024-09-21T14:02:16.449 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:02:16.449 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:02:16.449 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:02:16.449 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.449 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.449 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25421.0:172319 9.4 e315/260 9:21e132db:::benchmark_data_smithi049_138504_object50911:head v 315'10564, mlcod=315'10559)", 2024-09-21T14:02:16.449 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:00:33.250191+0000", 2024-09-21T14:02:16.450 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 103.120454839, 2024-09-21T14:02:16.450 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.270657121, 2024-09-21T14:02:16.450 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:02:16.450 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:02:16.450 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:02:16.450 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:02:16.450 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.450 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:02:16.450 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250191+0000", 2024-09-21T14:02:16.450 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:02:16.450 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.450 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.450 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:02:16.450 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250191+0000", 2024-09-21T14:02:16.451 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:02:16.451 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.451 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.451 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:02:16.451 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250191+0000", 2024-09-21T14:02:16.451 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8799999999999998e-07 2024-09-21T14:02:16.451 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.451 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.451 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:02:16.451 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250195+0000", 2024-09-21T14:02:16.451 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.647e-06 2024-09-21T14:02:16.451 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.451 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.451 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:02:16.452 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250201+0000", 2024-09-21T14:02:16.452 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4200000000000004e-06 2024-09-21T14:02:16.452 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.452 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.452 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:02:16.452 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250206+0000", 2024-09-21T14:02:16.452 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.994e-06 2024-09-21T14:02:16.452 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.452 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.452 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:02:16.452 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.256110+0000", 2024-09-21T14:02:16.452 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0059039840000000001 2024-09-21T14:02:16.452 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.452 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.453 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:02:16.453 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.256130+0000", 2024-09-21T14:02:16.453 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.98e-05 2024-09-21T14:02:16.453 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.453 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.453 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:02:16.453 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.520816+0000", 2024-09-21T14:02:16.453 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.264685528 2024-09-21T14:02:16.453 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.453 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.453 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:02:16.453 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.520848+0000", 2024-09-21T14:02:16.453 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2360000000000002e-05 2024-09-21T14:02:16.453 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:02:16.454 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:02:16.454 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:02:16.454 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.454 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.454 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25421.0:172326 9.4 e315/260 9:22a8fc8a:::benchmark_data_smithi049_138504_object50918:head v 315'10565, mlcod=315'10560)", 2024-09-21T14:02:16.454 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:00:33.254759+0000", 2024-09-21T14:02:16.454 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 103.115886549, 2024-09-21T14:02:16.454 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26611502300000001, 2024-09-21T14:02:16.454 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:02:16.454 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:02:16.454 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:02:16.454 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:02:16.454 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.454 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:02:16.455 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254759+0000", 2024-09-21T14:02:16.455 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:02:16.455 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.455 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.455 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:02:16.455 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254759+0000", 2024-09-21T14:02:16.455 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:02:16.455 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.455 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.455 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:02:16.455 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254760+0000", 2024-09-21T14:02:16.455 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6400000000000001e-07 2024-09-21T14:02:16.455 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.455 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.456 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:02:16.456 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254854+0000", 2024-09-21T14:02:16.456 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4320999999999994e-05 2024-09-21T14:02:16.456 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.456 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.456 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:02:16.456 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254865+0000", 2024-09-21T14:02:16.456 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.101e-05 2024-09-21T14:02:16.456 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.456 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.456 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:02:16.456 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254872+0000", 2024-09-21T14:02:16.456 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2539999999999998e-06 2024-09-21T14:02:16.457 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.457 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.457 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:02:16.457 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.258679+0000", 2024-09-21T14:02:16.457 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.003807342 2024-09-21T14:02:16.457 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.457 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.457 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:02:16.457 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.258699+0000", 2024-09-21T14:02:16.457 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.039e-05 2024-09-21T14:02:16.457 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.457 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.457 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:02:16.457 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.520851+0000", 2024-09-21T14:02:16.458 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26215118900000001 2024-09-21T14:02:16.458 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.458 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.458 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:02:16.458 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.520874+0000", 2024-09-21T14:02:16.458 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3753000000000001e-05 2024-09-21T14:02:16.458 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:02:16.458 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:02:16.458 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:02:16.458 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.458 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.458 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25421.0:172331 9.f e315/260 9:f6449e5b:::benchmark_data_smithi049_138504_object50923:head v 315'10871, mlcod=315'10867)", 2024-09-21T14:02:16.458 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:00:33.254942+0000", 2024-09-21T14:02:16.459 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 103.11570417199999, 2024-09-21T14:02:16.459 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.274133181, 2024-09-21T14:02:16.459 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:02:16.459 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:02:16.459 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:02:16.459 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:02:16.459 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.459 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:02:16.459 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254942+0000", 2024-09-21T14:02:16.459 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:02:16.459 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.460 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.460 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:02:16.460 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254942+0000", 2024-09-21T14:02:16.460 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:02:16.460 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.460 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.460 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:02:16.460 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254942+0000", 2024-09-21T14:02:16.460 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5700000000000002e-07 2024-09-21T14:02:16.460 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.460 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.460 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:02:16.460 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254949+0000", 2024-09-21T14:02:16.460 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.024e-06 2024-09-21T14:02:16.460 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.461 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.461 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:02:16.461 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254959+0000", 2024-09-21T14:02:16.461 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6390000000000004e-06 2024-09-21T14:02:16.461 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.461 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.461 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:02:16.461 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254965+0000", 2024-09-21T14:02:16.461 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6420000000000004e-06 2024-09-21T14:02:16.461 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.461 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.461 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:02:16.461 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.259407+0000", 2024-09-21T14:02:16.461 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0044420140000000002 2024-09-21T14:02:16.462 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.462 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.462 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:02:16.462 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.259425+0000", 2024-09-21T14:02:16.462 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7937000000000001e-05 2024-09-21T14:02:16.462 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.462 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.462 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:02:16.462 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.529033+0000", 2024-09-21T14:02:16.462 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26960846599999999 2024-09-21T14:02:16.462 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.462 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.462 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:02:16.463 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.529075+0000", 2024-09-21T14:02:16.463 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1801999999999997e-05 2024-09-21T14:02:16.463 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:02:16.463 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:02:16.463 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:02:16.463 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.463 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.463 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25421.0:172333 9.2 9:461458cc:::benchmark_data_smithi049_138504_object50925:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-09-21T14:02:16.463 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:00:33.258960+0000", 2024-09-21T14:02:16.463 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 103.111686037, 2024-09-21T14:02:16.463 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.261926308, 2024-09-21T14:02:16.463 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:02:16.463 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:02:16.463 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:02:16.464 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:02:16.464 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25421", 2024-09-21T14:02:16.464 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:02:16.464 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 172333 2024-09-21T14:02:16.464 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.464 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:02:16.464 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.464 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:02:16.464 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.258960+0000", 2024-09-21T14:02:16.464 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:02:16.464 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.464 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.464 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:02:16.465 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.258960+0000", 2024-09-21T14:02:16.465 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:02:16.465 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.465 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.465 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:02:16.465 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.258961+0000", 2024-09-21T14:02:16.465 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5629999999999999e-06 2024-09-21T14:02:16.465 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.465 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.465 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:02:16.465 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.258967+0000", 2024-09-21T14:02:16.465 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5079999999999996e-06 2024-09-21T14:02:16.465 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.465 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.465 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:02:16.466 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.258979+0000", 2024-09-21T14:02:16.466 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1897e-05 2024-09-21T14:02:16.466 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.466 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.466 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:02:16.466 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.258986+0000", 2024-09-21T14:02:16.466 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8739999999999997e-06 2024-09-21T14:02:16.466 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.466 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.466 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:02:16.466 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.259036+0000", 2024-09-21T14:02:16.466 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0695000000000002e-05 2024-09-21T14:02:16.466 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.467 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.467 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:02:16.467 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.259144+0000", 2024-09-21T14:02:16.467 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000107462 2024-09-21T14:02:16.467 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.467 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.467 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,2", 2024-09-21T14:02:16.467 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.259252+0000", 2024-09-21T14:02:16.467 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000107667 2024-09-21T14:02:16.467 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.467 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.467 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:02:16.467 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.517843+0000", 2024-09-21T14:02:16.467 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.25859111400000001 2024-09-21T14:02:16.468 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.468 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.468 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:02:16.468 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.517994+0000", 2024-09-21T14:02:16.468 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00015150299999999999 2024-09-21T14:02:16.468 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.468 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.468 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:02:16.468 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.520805+0000", 2024-09-21T14:02:16.468 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0028113489999999999 2024-09-21T14:02:16.468 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.468 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.468 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:02:16.468 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.520850+0000", 2024-09-21T14:02:16.469 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4333999999999999e-05 2024-09-21T14:02:16.469 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.469 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.469 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:02:16.469 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.520886+0000", 2024-09-21T14:02:16.469 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6341999999999999e-05 2024-09-21T14:02:16.469 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:02:16.469 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:02:16.469 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:02:16.469 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.469 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.469 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25421.0:172330 9.4 e315/260 9:2882ed62:::benchmark_data_smithi049_138504_object50922:head v 315'10566, mlcod=315'10560)", 2024-09-21T14:02:16.469 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:00:33.259976+0000", 2024-09-21T14:02:16.469 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 103.110669542, 2024-09-21T14:02:16.469 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26092217499999998, 2024-09-21T14:02:16.470 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:02:16.470 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:02:16.470 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:02:16.470 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:02:16.470 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.470 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:02:16.470 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.259976+0000", 2024-09-21T14:02:16.470 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:02:16.470 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.470 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.470 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:02:16.470 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.259976+0000", 2024-09-21T14:02:16.470 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:02:16.470 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.471 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.471 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:02:16.471 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.259977+0000", 2024-09-21T14:02:16.471 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1e-07 2024-09-21T14:02:16.471 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.471 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.471 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:02:16.471 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.259980+0000", 2024-09-21T14:02:16.471 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.58e-06 2024-09-21T14:02:16.471 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.471 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.471 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:02:16.471 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.259986+0000", 2024-09-21T14:02:16.471 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9109999999999998e-06 2024-09-21T14:02:16.471 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.472 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.472 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:02:16.472 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.259991+0000", 2024-09-21T14:02:16.472 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4510000000000002e-06 2024-09-21T14:02:16.472 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.472 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.472 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:02:16.472 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.261224+0000", 2024-09-21T14:02:16.472 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0012338169999999999 2024-09-21T14:02:16.472 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.472 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.472 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:02:16.472 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.261241+0000", 2024-09-21T14:02:16.472 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6762e-05 2024-09-21T14:02:16.473 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.473 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.473 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:02:16.473 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.520876+0000", 2024-09-21T14:02:16.473 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.25963512500000002 2024-09-21T14:02:16.473 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.473 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.473 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:02:16.473 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.520899+0000", 2024-09-21T14:02:16.473 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2218999999999999e-05 2024-09-21T14:02:16.473 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:02:16.473 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:02:16.473 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:02:16.473 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.473 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.474 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25421.0:172334 9.4 e315/260 9:24fa2bd3:::benchmark_data_smithi049_138504_object50926:head v 315'10567, mlcod=315'10561)", 2024-09-21T14:02:16.474 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:00:33.262387+0000", 2024-09-21T14:02:16.474 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 103.10825933, 2024-09-21T14:02:16.474 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26075252599999998, 2024-09-21T14:02:16.474 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:02:16.474 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:02:16.474 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:02:16.474 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:02:16.474 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.474 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:02:16.474 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.262387+0000", 2024-09-21T14:02:16.474 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:02:16.475 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.475 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.475 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:02:16.475 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.262387+0000", 2024-09-21T14:02:16.475 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:02:16.475 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.475 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.475 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:02:16.475 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.262388+0000", 2024-09-21T14:02:16.475 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6099999999999999e-07 2024-09-21T14:02:16.475 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.475 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.475 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:02:16.475 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.262393+0000", 2024-09-21T14:02:16.476 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8289999999999996e-06 2024-09-21T14:02:16.476 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.476 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.476 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:02:16.476 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.262488+0000", 2024-09-21T14:02:16.476 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4914000000000006e-05 2024-09-21T14:02:16.476 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.476 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.476 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:02:16.476 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.262504+0000", 2024-09-21T14:02:16.476 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5498999999999998e-05 2024-09-21T14:02:16.476 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.476 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.476 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:02:16.477 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.263573+0000", 2024-09-21T14:02:16.477 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0010692119999999999 2024-09-21T14:02:16.477 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.477 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.477 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:02:16.477 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.263590+0000", 2024-09-21T14:02:16.477 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7331000000000001e-05 2024-09-21T14:02:16.477 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.477 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.477 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:02:16.477 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.523109+0000", 2024-09-21T14:02:16.477 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.25951882100000001 2024-09-21T14:02:16.477 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:16.477 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:16.478 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:02:16.478 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.523139+0000", 2024-09-21T14:02:16.478 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9958999999999999e-05 2024-09-21T14:02:16.478 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:02:16.478 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:02:16.478 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:02:16.478 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:02:16.478 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:02:16.478 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T14:02:16.479 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:16 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:02:16.479 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:16 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:02:16.479 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:16 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:02:16.479 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:16 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:02:16.479 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:16 smithi049 ceph-mon[40489]: osdmap e367: 8 total, 8 up, 8 in 2024-09-21T14:02:16.479 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:16 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:02:16.480 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:16 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:02:16.480 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:16 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:02:16.480 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:16 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:02:16.480 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:16 smithi049 ceph-mon[30080]: osdmap e367: 8 total, 8 up, 8 in 2024-09-21T14:02:16.519 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:16.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:16.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:16.621 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:16.664 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:16.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:16.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:16.684 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T14:02:16.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:16 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:02:16.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:16 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:02:16.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:16 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:02:16.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:16 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:02:16.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:16 smithi187 ceph-mon[33032]: osdmap e367: 8 total, 8 up, 8 in 2024-09-21T14:02:16.766 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:02:16.799 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:16.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:16.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:02:16.900 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:16.946 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:16.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:16.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:17.047 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:17.069 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:17.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:17.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:17.171 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:02:17.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:17.208 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:17.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:02:17.288 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.6 to 1 (10000) 2024-09-21T14:02:17.289 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-09-21T14:02:17.309 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:02:17.353 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:17.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:17.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:02:17.455 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:17.488 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:17.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:17.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:17.590 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:17.595 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:17 smithi187 ceph-mon[33032]: pgmap v679: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 156 active+clean; 2.1 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.07k op/s; 3060/105171 objects degraded (2.910%); 7.0 MiB/s, 211 objects/s recovering 2024-09-21T14:02:17.595 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:17 smithi187 ceph-mon[33032]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-21T14:02:17.595 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:17 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3685174748' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:02:17.595 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:17 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:02:17.595 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:17 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3942864027' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T14:02:17.595 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:17 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T14:02:17.613 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:17.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:17.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:17.715 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:17.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:17 smithi049 ceph-mon[30080]: pgmap v679: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 156 active+clean; 2.1 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.07k op/s; 3060/105171 objects degraded (2.910%); 7.0 MiB/s, 211 objects/s recovering 2024-09-21T14:02:17.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:17 smithi049 ceph-mon[30080]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-21T14:02:17.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:17 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3685174748' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:02:17.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:17 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:02:17.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:17 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3942864027' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T14:02:17.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:17 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T14:02:17.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:17 smithi049 ceph-mon[40489]: pgmap v679: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 156 active+clean; 2.1 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.07k op/s; 3060/105171 objects degraded (2.910%); 7.0 MiB/s, 211 objects/s recovering 2024-09-21T14:02:17.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:17 smithi049 ceph-mon[40489]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-21T14:02:17.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:17 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3685174748' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:02:17.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:17 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:02:17.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:17 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3942864027' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T14:02:17.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:17 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T14:02:17.757 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:17.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:17.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:17.859 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:17.890 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:17.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:17.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:17.992 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:02:18.028 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:18.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:18.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:02:18.130 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:02:18.153 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:18.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:18.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:02:18.255 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:02:18.300 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:18.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:18.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:02:18.402 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:02:18.435 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:18.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:18.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:02:18.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:18 smithi187 ceph-mon[33032]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:02:18.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:18 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T14:02:18.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:18 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T14:02:18.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:18 smithi187 ceph-mon[33032]: osdmap e368: 8 total, 8 up, 8 in 2024-09-21T14:02:18.537 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:02:18.570 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:18.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:18.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:02:18.672 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:02:18.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:18.702 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:18.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:02:18.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:18 smithi049 ceph-mon[40489]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:02:18.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:18 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T14:02:18.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:18 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T14:02:18.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:18 smithi049 ceph-mon[40489]: osdmap e368: 8 total, 8 up, 8 in 2024-09-21T14:02:18.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:18 smithi049 ceph-mon[30080]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:02:18.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:18 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T14:02:18.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:18 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T14:02:18.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:18 smithi049 ceph-mon[30080]: osdmap e368: 8 total, 8 up, 8 in 2024-09-21T14:02:18.804 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:02:18.841 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:18.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:18.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:02:18.943 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:18.976 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:18.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:18.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:19.077 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:02:19.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:19.125 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:19.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:02:19.226 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:19.248 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:19.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:19.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:19.349 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:02:19.387 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:19.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:19.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:02:19.489 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:19.522 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:19.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:19.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:19.612 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:19 smithi049 ceph-mon[40489]: pgmap v682: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 156 active+clean; 2.0 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.02k op/s; 3060/97026 objects degraded (3.154%); 5.6 MiB/s, 89 objects/s recovering 2024-09-21T14:02:19.613 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:19 smithi049 ceph-mon[30080]: pgmap v682: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 156 active+clean; 2.0 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.02k op/s; 3060/97026 objects degraded (3.154%); 5.6 MiB/s, 89 objects/s recovering 2024-09-21T14:02:19.614 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T14:02:19.623 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:02:19.653 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:19 smithi187 ceph-mon[33032]: pgmap v682: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 156 active+clean; 2.0 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.02k op/s; 3060/97026 objects degraded (3.154%); 5.6 MiB/s, 89 objects/s recovering 2024-09-21T14:02:19.692 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:19.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:19.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:02:19.794 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:02:19.843 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:19.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:19.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:02:19.945 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:19.946 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:02:19 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:02:19.976 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:19.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:19.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:20.078 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:02:20.085 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:02:20.118 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:20.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:20.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:02:20.220 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:20.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:20.254 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:20.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:20.355 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:02:20.389 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:20.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:20.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:02:20.442 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:20 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:02:20.442 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:20 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 3060/97026 objects degraded (3.154%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T14:02:20.491 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:20.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:20 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:02:20.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:20 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 3060/97026 objects degraded (3.154%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T14:02:20.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:20 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:02:20.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:20 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 3060/97026 objects degraded (3.154%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T14:02:20.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:20.525 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:20.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:20.627 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:02:20.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:20.670 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:20.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:02:20.761 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T14:02:20.761 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-21T14:02:20.761 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.761 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25421.0:216308 9.5 e368/346 9:a78bfe52:::benchmark_data_smithi049_138504_object94900:head v 368'13937, mlcod=0'0)", 2024-09-21T14:02:20.762 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:02:20.749235+0000", 2024-09-21T14:02:20.762 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0094252120000000005, 2024-09-21T14:02:20.762 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0094530159999999998, 2024-09-21T14:02:20.762 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:02:20.762 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:02:20.762 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T14:02:20.762 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:02:20.762 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.762 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:02:20.762 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.749235+0000", 2024-09-21T14:02:20.762 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:02:20.762 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.762 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.762 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:02:20.763 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.749235+0000", 2024-09-21T14:02:20.763 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:02:20.763 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.763 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.763 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:02:20.763 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.749235+0000", 2024-09-21T14:02:20.763 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4999999999999998e-07 2024-09-21T14:02:20.763 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.763 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.763 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:02:20.763 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.749240+0000", 2024-09-21T14:02:20.763 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3970000000000004e-06 2024-09-21T14:02:20.763 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.763 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.764 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:02:20.764 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.749246+0000", 2024-09-21T14:02:20.764 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6579999999999996e-06 2024-09-21T14:02:20.764 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.764 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.764 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:02:20.764 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.749251+0000", 2024-09-21T14:02:20.764 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6519999999999997e-06 2024-09-21T14:02:20.764 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.764 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.764 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:02:20.764 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.754993+0000", 2024-09-21T14:02:20.764 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0057423079999999998 2024-09-21T14:02:20.764 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.765 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.765 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:02:20.765 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.755014+0000", 2024-09-21T14:02:20.765 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0727999999999999e-05 2024-09-21T14:02:20.765 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:02:20.765 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:02:20.765 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:02:20.765 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.765 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.765 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:216309 9.9 9:97c8631c:::benchmark_data_smithi049_138504_object94901:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e368)", 2024-09-21T14:02:20.765 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:02:20.749331+0000", 2024-09-21T14:02:20.765 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0093293379999999995, 2024-09-21T14:02:20.765 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0094115450000000003, 2024-09-21T14:02:20.765 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:02:20.765 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:02:20.766 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-21T14:02:20.766 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:02:20.766 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:02:20.766 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:02:20.766 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 216309 2024-09-21T14:02:20.766 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.766 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:02:20.766 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.766 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:02:20.766 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.749331+0000", 2024-09-21T14:02:20.766 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:02:20.766 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.766 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.766 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:02:20.767 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.749331+0000", 2024-09-21T14:02:20.767 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:02:20.767 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.767 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.767 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:02:20.767 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.749332+0000", 2024-09-21T14:02:20.767 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4160000000000001e-06 2024-09-21T14:02:20.767 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.767 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.767 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:02:20.767 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.749336+0000", 2024-09-21T14:02:20.767 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1880000000000004e-06 2024-09-21T14:02:20.767 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.767 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.768 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:02:20.768 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.749347+0000", 2024-09-21T14:02:20.768 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0349999999999999e-05 2024-09-21T14:02:20.768 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.768 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.768 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:02:20.768 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.749410+0000", 2024-09-21T14:02:20.768 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3310000000000002e-05 2024-09-21T14:02:20.768 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.768 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.768 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:02:20.768 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.750707+0000", 2024-09-21T14:02:20.768 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001296682 2024-09-21T14:02:20.768 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.768 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.769 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:02:20.769 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.750832+0000", 2024-09-21T14:02:20.769 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012557000000000001 2024-09-21T14:02:20.769 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.769 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.769 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,5", 2024-09-21T14:02:20.769 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.750965+0000", 2024-09-21T14:02:20.769 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000132356 2024-09-21T14:02:20.769 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:02:20.769 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:02:20.769 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:02:20.769 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.769 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.769 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:216310 9.9 9:92623d3b:::benchmark_data_smithi049_138504_object94902:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e368)", 2024-09-21T14:02:20.769 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:02:20.749421+0000", 2024-09-21T14:02:20.770 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0092386740000000005, 2024-09-21T14:02:20.770 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0093817840000000007, 2024-09-21T14:02:20.770 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:02:20.770 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:02:20.770 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-21T14:02:20.770 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:02:20.770 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:02:20.770 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:02:20.770 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 216310 2024-09-21T14:02:20.770 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.770 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:02:20.770 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.770 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:02:20.770 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.749421+0000", 2024-09-21T14:02:20.771 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:02:20.771 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.771 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.771 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:02:20.771 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.749421+0000", 2024-09-21T14:02:20.771 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:02:20.771 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.771 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.771 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:02:20.771 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.749422+0000", 2024-09-21T14:02:20.771 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7599999999999997e-07 2024-09-21T14:02:20.771 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.771 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.771 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:02:20.772 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.749425+0000", 2024-09-21T14:02:20.772 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2009999999999999e-06 2024-09-21T14:02:20.772 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.772 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.772 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:02:20.772 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.749432+0000", 2024-09-21T14:02:20.772 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6440000000000003e-06 2024-09-21T14:02:20.772 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.772 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.772 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:02:20.772 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.749437+0000", 2024-09-21T14:02:20.772 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1959999999999999e-06 2024-09-21T14:02:20.772 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.772 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.772 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:02:20.773 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.753602+0000", 2024-09-21T14:02:20.773 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0041655720000000002 2024-09-21T14:02:20.773 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.773 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.773 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:02:20.773 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.753785+0000", 2024-09-21T14:02:20.773 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000182621 2024-09-21T14:02:20.773 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.773 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.773 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,5", 2024-09-21T14:02:20.773 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.753915+0000", 2024-09-21T14:02:20.773 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012972600000000001 2024-09-21T14:02:20.773 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:02:20.773 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:02:20.773 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:02:20.774 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.774 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.774 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:216311 9.e 9:72039af0:::benchmark_data_smithi049_138504_object94903:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e368)", 2024-09-21T14:02:20.774 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:02:20.749443+0000", 2024-09-21T14:02:20.774 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0092170059999999998, 2024-09-21T14:02:20.774 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0094226150000000005, 2024-09-21T14:02:20.774 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:02:20.774 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:02:20.774 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-21T14:02:20.774 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:02:20.774 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:02:20.774 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:02:20.774 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 216311 2024-09-21T14:02:20.775 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:20.775 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.776 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:02:20.776 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.776 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:02:20.776 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.749443+0000", 2024-09-21T14:02:20.776 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:02:20.776 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.776 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.776 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:02:20.776 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.749443+0000", 2024-09-21T14:02:20.776 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:02:20.776 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.776 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.776 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:02:20.776 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.749443+0000", 2024-09-21T14:02:20.777 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9299999999999999e-07 2024-09-21T14:02:20.777 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.777 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.777 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:02:20.777 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.749446+0000", 2024-09-21T14:02:20.777 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1550000000000001e-06 2024-09-21T14:02:20.777 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.777 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.777 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:02:20.777 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.749451+0000", 2024-09-21T14:02:20.777 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6330000000000001e-06 2024-09-21T14:02:20.777 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.777 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.777 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:02:20.778 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.749456+0000", 2024-09-21T14:02:20.778 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7169999999999999e-06 2024-09-21T14:02:20.778 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.778 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.778 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:02:20.778 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.752898+0000", 2024-09-21T14:02:20.778 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0034420259999999999 2024-09-21T14:02:20.778 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.778 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.778 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:02:20.778 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.753027+0000", 2024-09-21T14:02:20.778 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000128947 2024-09-21T14:02:20.778 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.778 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.779 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,4", 2024-09-21T14:02:20.779 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.753134+0000", 2024-09-21T14:02:20.779 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000107408 2024-09-21T14:02:20.779 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.779 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.779 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:02:20.779 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.757475+0000", 2024-09-21T14:02:20.779 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0043403349999999999 2024-09-21T14:02:20.779 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:02:20.779 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:02:20.779 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:02:20.779 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.779 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.779 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:216314 9.e 9:74358e09:::benchmark_data_smithi049_138504_object94906:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e368)", 2024-09-21T14:02:20.779 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:02:20.749463+0000", 2024-09-21T14:02:20.780 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0091968459999999998, 2024-09-21T14:02:20.780 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0094634320000000008, 2024-09-21T14:02:20.780 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:02:20.780 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:02:20.780 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-21T14:02:20.780 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:02:20.780 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:02:20.780 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:02:20.780 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 216314 2024-09-21T14:02:20.780 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.780 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:02:20.780 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.780 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:02:20.780 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.749463+0000", 2024-09-21T14:02:20.781 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:02:20.781 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.781 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.781 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:02:20.781 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.749463+0000", 2024-09-21T14:02:20.781 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:02:20.781 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.781 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.781 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:02:20.781 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.749464+0000", 2024-09-21T14:02:20.781 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5999999999999999e-07 2024-09-21T14:02:20.781 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.781 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.781 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:02:20.782 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.749466+0000", 2024-09-21T14:02:20.782 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1289999999999999e-06 2024-09-21T14:02:20.782 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.782 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.782 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:02:20.782 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.749472+0000", 2024-09-21T14:02:20.782 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8109999999999999e-06 2024-09-21T14:02:20.782 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.782 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.782 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:02:20.782 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.749476+0000", 2024-09-21T14:02:20.782 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8489999999999997e-06 2024-09-21T14:02:20.782 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.782 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.783 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:02:20.783 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.757507+0000", 2024-09-21T14:02:20.783 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.008030934 2024-09-21T14:02:20.783 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.783 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.783 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:02:20.783 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.757615+0000", 2024-09-21T14:02:20.783 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000107785 2024-09-21T14:02:20.783 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.783 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.783 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,4", 2024-09-21T14:02:20.783 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.757724+0000", 2024-09-21T14:02:20.783 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010874099999999999 2024-09-21T14:02:20.783 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:02:20.783 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:02:20.784 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:02:20.784 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.784 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.784 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25421.0:216315 9.d e368/345 9:b0f6a87d:::benchmark_data_smithi049_138504_object94907:head v 368'13528, mlcod=368'13526)", 2024-09-21T14:02:20.784 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:02:20.750797+0000", 2024-09-21T14:02:20.784 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0078632140000000003, 2024-09-21T14:02:20.784 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0081849690000000003, 2024-09-21T14:02:20.784 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:02:20.784 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:02:20.784 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T14:02:20.784 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:02:20.784 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.784 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:02:20.784 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.750797+0000", 2024-09-21T14:02:20.784 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:02:20.785 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.785 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.785 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:02:20.785 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.750797+0000", 2024-09-21T14:02:20.785 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:02:20.785 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.785 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.785 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:02:20.785 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.750798+0000", 2024-09-21T14:02:20.785 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4300000000000002e-07 2024-09-21T14:02:20.785 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.785 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.785 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:02:20.785 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.750806+0000", 2024-09-21T14:02:20.786 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.994e-06 2024-09-21T14:02:20.786 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.786 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.786 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:02:20.786 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.750815+0000", 2024-09-21T14:02:20.786 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3249999999999997e-06 2024-09-21T14:02:20.786 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.786 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.786 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:02:20.786 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.750822+0000", 2024-09-21T14:02:20.786 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.985e-06 2024-09-21T14:02:20.786 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.786 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.786 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:02:20.786 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.754911+0000", 2024-09-21T14:02:20.787 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0040890859999999996 2024-09-21T14:02:20.787 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.787 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.787 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:02:20.787 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.754931+0000", 2024-09-21T14:02:20.787 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9884000000000001e-05 2024-09-21T14:02:20.787 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:02:20.787 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:02:20.787 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:02:20.787 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.787 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.787 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25421.0:216318 9.1 e368/333 9:882bea7f:::benchmark_data_smithi049_138504_object94910:head v 368'13408, mlcod=368'13406)", 2024-09-21T14:02:20.787 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:02:20.750851+0000", 2024-09-21T14:02:20.787 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0078092029999999998, 2024-09-21T14:02:20.788 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0081764360000000005, 2024-09-21T14:02:20.788 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:02:20.788 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:02:20.788 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T14:02:20.788 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:02:20.788 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.788 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:02:20.788 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.750851+0000", 2024-09-21T14:02:20.788 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:02:20.788 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.788 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.788 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:02:20.788 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.750851+0000", 2024-09-21T14:02:20.788 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:02:20.788 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.789 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.789 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:02:20.789 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.750851+0000", 2024-09-21T14:02:20.789 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9000000000000002e-07 2024-09-21T14:02:20.789 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.789 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.789 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:02:20.789 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.750857+0000", 2024-09-21T14:02:20.789 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6400000000000002e-06 2024-09-21T14:02:20.789 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.789 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.789 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:02:20.789 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.750864+0000", 2024-09-21T14:02:20.789 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.657e-06 2024-09-21T14:02:20.790 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.790 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.790 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:02:20.790 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.750869+0000", 2024-09-21T14:02:20.790 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1320000000000002e-06 2024-09-21T14:02:20.790 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.790 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.790 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:02:20.790 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.753424+0000", 2024-09-21T14:02:20.790 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0025555550000000002 2024-09-21T14:02:20.790 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.790 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.790 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:02:20.790 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.753444+0000", 2024-09-21T14:02:20.790 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9519000000000001e-05 2024-09-21T14:02:20.791 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:02:20.791 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:02:20.791 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:02:20.791 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.791 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.791 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:216320 9.e 9.7091335e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e368)", 2024-09-21T14:02:20.792 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:02:20.752555+0000", 2024-09-21T14:02:20.792 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0061046970000000001, 2024-09-21T14:02:20.792 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0065172659999999999, 2024-09-21T14:02:20.792 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:02:20.792 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:02:20.792 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "queued for pg", 2024-09-21T14:02:20.792 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:02:20.792 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:02:20.793 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:02:20.793 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 216320 2024-09-21T14:02:20.793 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.793 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:02:20.793 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.793 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:02:20.793 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.752555+0000", 2024-09-21T14:02:20.793 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:02:20.793 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.793 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.794 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:02:20.794 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.752555+0000", 2024-09-21T14:02:20.794 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:02:20.794 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.794 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.794 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:02:20.794 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.752557+0000", 2024-09-21T14:02:20.794 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4470000000000001e-06 2024-09-21T14:02:20.794 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.794 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.794 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:02:20.794 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.752562+0000", 2024-09-21T14:02:20.795 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4099999999999999e-06 2024-09-21T14:02:20.795 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.795 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.795 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:02:20.795 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.752574+0000", 2024-09-21T14:02:20.795 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.183e-05 2024-09-21T14:02:20.795 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.795 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.795 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:02:20.795 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.752581+0000", 2024-09-21T14:02:20.796 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.25e-06 2024-09-21T14:02:20.796 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:02:20.796 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:02:20.796 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:02:20.796 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.796 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.796 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25421.0:216323 9.c e368/329 9:3514f6e3:::benchmark_data_smithi049_138504_object94915:head v 368'13393, mlcod=368'13391)", 2024-09-21T14:02:20.796 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:02:20.755034+0000", 2024-09-21T14:02:20.796 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0036258729999999999, 2024-09-21T14:02:20.796 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0040813550000000001, 2024-09-21T14:02:20.796 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:02:20.797 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:02:20.797 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T14:02:20.797 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:02:20.797 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.797 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:02:20.797 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.755034+0000", 2024-09-21T14:02:20.797 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:02:20.797 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.797 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.797 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:02:20.798 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.755034+0000", 2024-09-21T14:02:20.798 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:02:20.798 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.798 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.798 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:02:20.798 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.755035+0000", 2024-09-21T14:02:20.798 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1310000000000001e-06 2024-09-21T14:02:20.798 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.798 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.798 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:02:20.798 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.755044+0000", 2024-09-21T14:02:20.798 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3489999999999992e-06 2024-09-21T14:02:20.799 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.799 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.799 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:02:20.799 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.755055+0000", 2024-09-21T14:02:20.799 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0835999999999999e-05 2024-09-21T14:02:20.799 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.799 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.799 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:02:20.799 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.755061+0000", 2024-09-21T14:02:20.799 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7460000000000003e-06 2024-09-21T14:02:20.800 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.800 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.800 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:02:20.800 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.755162+0000", 2024-09-21T14:02:20.800 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010075 2024-09-21T14:02:20.800 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.800 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.800 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:02:20.800 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.755182+0000", 2024-09-21T14:02:20.800 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9542e-05 2024-09-21T14:02:20.801 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:02:20.801 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:02:20.801 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:02:20.801 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.801 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.801 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25421.0:216322 9.1 e368/333 9:8369b222:::benchmark_data_smithi049_138504_object94914:head v 368'13409, mlcod=368'13406)", 2024-09-21T14:02:20.801 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:02:20.755085+0000", 2024-09-21T14:02:20.801 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0035750690000000002, 2024-09-21T14:02:20.801 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0040743120000000001, 2024-09-21T14:02:20.802 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:02:20.802 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:02:20.802 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T14:02:20.802 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:02:20.802 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.802 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:02:20.802 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.755085+0000", 2024-09-21T14:02:20.802 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:02:20.802 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.802 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.802 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:02:20.802 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.755085+0000", 2024-09-21T14:02:20.803 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:02:20.803 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.803 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.803 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:02:20.803 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.755085+0000", 2024-09-21T14:02:20.803 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8099999999999998e-07 2024-09-21T14:02:20.803 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.804 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.804 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:02:20.804 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.755091+0000", 2024-09-21T14:02:20.804 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9150000000000001e-06 2024-09-21T14:02:20.804 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.804 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.804 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:02:20.804 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.755098+0000", 2024-09-21T14:02:20.804 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3080000000000004e-06 2024-09-21T14:02:20.804 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.804 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.804 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:02:20.805 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.755102+0000", 2024-09-21T14:02:20.805 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8330000000000001e-06 2024-09-21T14:02:20.805 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.805 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.805 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:02:20.805 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.758263+0000", 2024-09-21T14:02:20.805 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0031603769999999998 2024-09-21T14:02:20.805 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.806 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.806 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:02:20.806 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.758279+0000", 2024-09-21T14:02:20.806 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6560999999999999e-05 2024-09-21T14:02:20.806 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:02:20.806 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:02:20.806 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:02:20.806 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.806 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.806 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.25421.0:216309 9.9 e368/348)", 2024-09-21T14:02:20.806 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:02:20.755643+0000", 2024-09-21T14:02:20.807 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0030168389999999999, 2024-09-21T14:02:20.807 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003557593, 2024-09-21T14:02:20.807 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:02:20.807 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:02:20.807 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "queued for pg", 2024-09-21T14:02:20.807 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:02:20.807 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.807 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:02:20.808 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.755643+0000", 2024-09-21T14:02:20.808 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:02:20.808 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.808 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.808 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:02:20.808 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.755643+0000", 2024-09-21T14:02:20.808 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:02:20.808 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.808 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.808 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:02:20.808 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.755644+0000", 2024-09-21T14:02:20.809 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2699999999999999e-07 2024-09-21T14:02:20.809 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.809 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.809 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:02:20.809 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.755647+0000", 2024-09-21T14:02:20.809 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4149999999999999e-06 2024-09-21T14:02:20.809 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.809 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.809 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:02:20.809 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.756420+0000", 2024-09-21T14:02:20.809 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00077243600000000004 2024-09-21T14:02:20.809 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.810 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.810 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:02:20.810 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.756429+0000", 2024-09-21T14:02:20.810 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.4900000000000006e-06 2024-09-21T14:02:20.810 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:02:20.810 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:02:20.810 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:02:20.810 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.810 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.810 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25421.0:216302 9.5 e368/346 9:afd81a3f:::benchmark_data_smithi049_138504_object94894:head v 368'13936, mlcod=0'0)", 2024-09-21T14:02:20.810 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:02:20.744359+0000", 2024-09-21T14:02:20.810 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.014300856000000001, 2024-09-21T14:02:20.811 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.014879559000000001, 2024-09-21T14:02:20.811 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:02:20.811 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:02:20.811 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T14:02:20.811 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:02:20.811 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.811 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:02:20.811 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.744359+0000", 2024-09-21T14:02:20.811 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:02:20.811 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.811 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.811 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:02:20.811 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.744359+0000", 2024-09-21T14:02:20.811 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:02:20.812 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.812 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.812 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:02:20.812 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.744360+0000", 2024-09-21T14:02:20.812 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3399999999999998e-07 2024-09-21T14:02:20.812 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.812 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.812 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:02:20.812 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.744366+0000", 2024-09-21T14:02:20.812 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.331e-06 2024-09-21T14:02:20.812 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.812 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.813 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:02:20.813 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.744377+0000", 2024-09-21T14:02:20.813 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0453e-05 2024-09-21T14:02:20.813 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.813 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.813 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:02:20.813 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.744383+0000", 2024-09-21T14:02:20.813 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.55e-06 2024-09-21T14:02:20.813 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.813 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.814 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:02:20.814 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.744490+0000", 2024-09-21T14:02:20.814 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010697899999999999 2024-09-21T14:02:20.814 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:20.814 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:20.814 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:02:20.814 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:20.744512+0000", 2024-09-21T14:02:20.814 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1957999999999999e-05 2024-09-21T14:02:20.814 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:02:20.814 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:02:20.815 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:02:20.815 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:02:20.815 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2024-09-21T14:02:20.815 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 12 2024-09-21T14:02:20.815 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T14:02:20.853 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:20.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:20.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:20.956 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:02:20.991 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:20.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:20.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:02:21.004 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:02:20 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:02:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:02:21.085 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-21T14:02:21.086 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T14:02:21.093 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:02:21.130 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:21.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:21.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:02:21.208 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:02:21.233 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:21.280 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:21.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:21.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:21.381 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:21.426 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:21.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:21.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:21.528 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:02:21.561 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:21.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:21.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:02:21.663 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:02:21.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:21 smithi187 ceph-mon[33032]: pgmap v683: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 156 active+clean; 1.9 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.20k op/s; 2472/92868 objects degraded (2.662%); 10 MiB/s, 160 objects/s recovering 2024-09-21T14:02:21.700 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:21 smithi049 ceph-mon[30080]: pgmap v683: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 156 active+clean; 1.9 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.20k op/s; 2472/92868 objects degraded (2.662%); 10 MiB/s, 160 objects/s recovering 2024-09-21T14:02:21.700 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:21 smithi049 ceph-mon[40489]: pgmap v683: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 156 active+clean; 1.9 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.20k op/s; 2472/92868 objects degraded (2.662%); 10 MiB/s, 160 objects/s recovering 2024-09-21T14:02:21.702 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:21.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:21.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:02:21.804 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:21.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:21.848 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:21.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:21.892 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:02:21.897 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:02:21.898 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:02:21.930 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:02:21.930 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:02:21.948 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:02:21.948 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:02:21.949 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:21.984 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:02:21.984 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:02:21.992 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:21.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:21.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:22.016 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:02:22.016 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:02:22.049 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:02:22.050 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:02:22.083 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:02:22.083 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:02:22.094 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:22.129 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:02:22.129 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:02:22.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:22.142 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:22.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:22.243 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:02:22.282 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:22.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:22.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:02:22.384 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:02:22.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:22.410 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:22.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:02:22.511 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:02:22.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:22.547 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:22.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:02:22.592 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T14:02:22.638 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:22 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/515421330' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:02:22.638 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:22 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T14:02:22.639 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:22 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/515421330' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:02:22.639 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:22 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T14:02:22.648 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:22.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:22.692 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:22.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:22.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:22 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/515421330' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:02:22.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:22 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T14:02:22.792 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:02:22.815 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:22.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:22.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:02:22.916 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:02:22.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:22.943 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:22.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:02:23.044 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:02:23.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:23.081 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:23.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:02:23.182 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:02:23.219 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:23.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:23.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:02:23.268 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:02:23.321 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:02:23.378 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:23.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:23.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:02:23.480 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:02:23.488 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:23 smithi049 ceph-mon[30080]: pgmap v684: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 156 active+clean; 1.6 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.47k op/s; 2207/80673 objects degraded (2.736%); 8.6 MiB/s, 138 objects/s recovering 2024-09-21T14:02:23.488 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:23 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:02:23.488 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:23 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:02:23.490 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:23 smithi049 ceph-mon[40489]: pgmap v684: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 156 active+clean; 1.6 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.47k op/s; 2207/80673 objects degraded (2.736%); 8.6 MiB/s, 138 objects/s recovering 2024-09-21T14:02:23.490 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:23 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:02:23.490 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:23 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:02:23.517 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:23.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:23.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:02:23.554 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T14:02:23.619 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:02:23.645 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T14:02:23.648 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-09-21T14:02:23.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:23.657 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:23.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:02:23.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:23 smithi187 ceph-mon[33032]: pgmap v684: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 156 active+clean; 1.6 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.47k op/s; 2207/80673 objects degraded (2.736%); 8.6 MiB/s, 138 objects/s recovering 2024-09-21T14:02:23.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:23 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:02:23.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:23 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:02:23.758 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:02:23.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:23.804 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:23.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:02:23.905 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:23.940 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:23.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:23.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:23.990 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:02:24.042 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:24.075 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:24.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:24.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:24.177 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:02:24.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:24.219 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:24.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:02:24.320 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:02:24.356 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:24.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:24.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:02:24.440 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:24 smithi187 ceph-mon[33032]: osdmap e369: 8 total, 8 up, 8 in 2024-09-21T14:02:24.440 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:24 smithi187 ceph-mon[33032]: from='client.25838 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:02:24.441 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:24 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T14:02:24.441 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:24 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T14:02:24.441 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:24 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:02:24.458 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:02:24.499 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:24.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:24.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:02:24.591 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T14:02:24.591 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:02:24.600 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:02:24.642 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:24.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:24.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:02:24.706 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:24 smithi049 ceph-mon[40489]: osdmap e369: 8 total, 8 up, 8 in 2024-09-21T14:02:24.706 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:24 smithi049 ceph-mon[40489]: from='client.25838 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:02:24.706 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:24 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T14:02:24.706 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:24 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T14:02:24.706 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:24 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:02:24.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:24 smithi049 ceph-mon[30080]: osdmap e369: 8 total, 8 up, 8 in 2024-09-21T14:02:24.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:24 smithi049 ceph-mon[30080]: from='client.25838 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:02:24.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:24 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T14:02:24.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:24 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T14:02:24.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:24 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:02:24.743 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:24.757 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":685,"stamp":"2024-09-21T14:02:23.370861+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1749656897,"num_objects":26891,"num_object_clones":0,"num_object_copies":80673,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2207,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26891,"num_whiteouts":0,"num_read":1369,"num_read_kb":1198,"num_write":216925,"num_write_kb":7771150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40666,"num_bytes_recovered":2515047616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77562,"ondisk_log_size":77562,"up":474,"acting":472,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":475,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":8217132,"kb_used_data":5312040,"kb_used_omap":23,"kb_used_meta":2904936,"kb_avail":741744084,"statfs":{"total":767960285184,"available":759545942016,"internally_reserved":0,"allocated":5439528960,"data_stored":5433017441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24148,"internal_metadata":2974654892},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,3,2,3,6,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-535166976,"num_objects":-8166,"num_object_clones":0,"num_object_copies":-24498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-853,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8166,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":8166,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":689,"num_bytes_recovered":45088792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.391769"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":552,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.293724+0000","last_change":"2024-09-21T14:01:41.189741+0000","last_active":"2024-09-21T14:02:17.293724+0000","last_peered":"2024-09-21T14:02:17.293724+0000","last_clean":"2024-09-21T14:02:17.293724+0000","last_became_active":"2024-09-21T14:01:41.189398+0000","last_became_peered":"2024-09-21T14:01:41.189398+0000","last_unstale":"2024-09-21T14:02:17.293724+0000","last_undegraded":"2024-09-21T14:02:17.293724+0000","last_fullsized":"2024-09-21T14:02:17.293724+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:55.873301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:55.873301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"290'15","reported_seq":727,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.291866+0000","last_change":"2024-09-21T14:01:34.785322+0000","last_active":"2024-09-21T14:02:17.291866+0000","last_peered":"2024-09-21T14:02:17.291866+0000","last_clean":"2024-09-21T14:02:17.291866+0000","last_became_active":"2024-09-21T13:58:17.771220+0000","last_became_peered":"2024-09-21T13:58:17.771220+0000","last_unstale":"2024-09-21T14:02:17.291866+0000","last_undegraded":"2024-09-21T14:02:17.291866+0000","last_fullsized":"2024-09-21T14:02:17.291866+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'15","last_scrub_stamp":"2024-09-21T14:01:34.785267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:34.785267+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:36.742233+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":660,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.292207+0000","last_change":"2024-09-21T14:01:41.202503+0000","last_active":"2024-09-21T14:02:17.292207+0000","last_peered":"2024-09-21T14:02:17.292207+0000","last_clean":"2024-09-21T14:02:17.292207+0000","last_became_active":"2024-09-21T14:01:41.202168+0000","last_became_peered":"2024-09-21T14:01:41.202168+0000","last_unstale":"2024-09-21T14:02:17.292207+0000","last_undegraded":"2024-09-21T14:02:17.292207+0000","last_fullsized":"2024-09-21T14:02:17.292207+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:40.452724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:40.452724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":644,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.291409+0000","last_change":"2024-09-21T14:01:41.203762+0000","last_active":"2024-09-21T14:02:17.291409+0000","last_peered":"2024-09-21T14:02:17.291409+0000","last_clean":"2024-09-21T14:02:17.291409+0000","last_became_active":"2024-09-21T14:01:41.203275+0000","last_became_peered":"2024-09-21T14:01:41.203275+0000","last_unstale":"2024-09-21T14:02:17.291409+0000","last_undegraded":"2024-09-21T14:02:17.291409+0000","last_fullsized":"2024-09-21T14:02:17.291409+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:00.320510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:00.320510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:46.565572+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":614,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.299821+0000","last_change":"2024-09-21T13:59:32.208701+0000","last_active":"2024-09-21T14:02:17.299821+0000","last_peered":"2024-09-21T14:02:17.299821+0000","last_clean":"2024-09-21T14:02:17.299821+0000","last_became_active":"2024-09-21T13:59:32.203340+0000","last_became_peered":"2024-09-21T13:59:32.203340+0000","last_unstale":"2024-09-21T14:02:17.299821+0000","last_undegraded":"2024-09-21T14:02:17.299821+0000","last_fullsized":"2024-09-21T14:02:17.299821+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:34.696780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:34.696780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":626,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.301584+0000","last_change":"2024-09-21T13:58:17.778322+0000","last_active":"2024-09-21T14:02:17.301584+0000","last_peered":"2024-09-21T14:02:17.301584+0000","last_clean":"2024-09-21T14:02:17.301584+0000","last_became_active":"2024-09-21T13:58:17.777893+0000","last_became_peered":"2024-09-21T13:58:17.777893+0000","last_unstale":"2024-09-21T14:02:17.301584+0000","last_undegraded":"2024-09-21T14:02:17.301584+0000","last_fullsized":"2024-09-21T14:02:17.301584+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:07.001135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:55:07.001135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"290'27","reported_seq":600,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.298277+0000","last_change":"2024-09-21T13:58:11.718421+0000","last_active":"2024-09-21T14:02:17.298277+0000","last_peered":"2024-09-21T14:02:17.298277+0000","last_clean":"2024-09-21T14:02:17.298277+0000","last_became_active":"2024-09-21T13:57:12.703469+0000","last_became_peered":"2024-09-21T13:57:12.703469+0000","last_unstale":"2024-09-21T14:02:17.298277+0000","last_undegraded":"2024-09-21T14:02:17.298277+0000","last_fullsized":"2024-09-21T14:02:17.298277+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:58:11.718334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:11.718334+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":667,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.290007+0000","last_change":"2024-09-21T14:01:32.717551+0000","last_active":"2024-09-21T14:02:17.290007+0000","last_peered":"2024-09-21T14:02:17.290007+0000","last_clean":"2024-09-21T14:02:17.290007+0000","last_became_active":"2024-09-21T13:58:17.773464+0000","last_became_peered":"2024-09-21T13:58:17.773464+0000","last_unstale":"2024-09-21T14:02:17.290007+0000","last_undegraded":"2024-09-21T14:02:17.290007+0000","last_fullsized":"2024-09-21T14:02:17.290007+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:32.717480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:01:32.717480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:02.205485+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":619,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.295198+0000","last_change":"2024-09-21T14:01:41.194807+0000","last_active":"2024-09-21T14:02:17.295198+0000","last_peered":"2024-09-21T14:02:17.295198+0000","last_clean":"2024-09-21T14:02:17.295198+0000","last_became_active":"2024-09-21T14:01:41.194517+0000","last_became_peered":"2024-09-21T14:01:41.194517+0000","last_unstale":"2024-09-21T14:02:17.295198+0000","last_undegraded":"2024-09-21T14:02:17.295198+0000","last_fullsized":"2024-09-21T14:02:17.295198+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:31.127926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:31.127926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"290'15","reported_seq":693,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.291762+0000","last_change":"2024-09-21T14:00:02.475761+0000","last_active":"2024-09-21T14:02:17.291762+0000","last_peered":"2024-09-21T14:02:17.291762+0000","last_clean":"2024-09-21T14:02:17.291762+0000","last_became_active":"2024-09-21T13:57:12.681408+0000","last_became_peered":"2024-09-21T13:57:12.681408+0000","last_unstale":"2024-09-21T14:02:17.291762+0000","last_undegraded":"2024-09-21T14:02:17.291762+0000","last_fullsized":"2024-09-21T14:02:17.291762+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'15","last_scrub_stamp":"2024-09-21T14:00:02.475661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:00:02.475661+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:37.694022+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":560,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.293492+0000","last_change":"2024-09-21T14:01:41.188585+0000","last_active":"2024-09-21T14:02:17.293492+0000","last_peered":"2024-09-21T14:02:17.293492+0000","last_clean":"2024-09-21T14:02:17.293492+0000","last_became_active":"2024-09-21T14:01:41.188248+0000","last_became_peered":"2024-09-21T14:01:41.188248+0000","last_unstale":"2024-09-21T14:02:17.293492+0000","last_undegraded":"2024-09-21T14:02:17.293492+0000","last_fullsized":"2024-09-21T14:02:17.293492+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:21.284048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:56:21.284048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":597,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.294043+0000","last_change":"2024-09-21T13:59:32.202128+0000","last_active":"2024-09-21T14:02:17.294043+0000","last_peered":"2024-09-21T14:02:17.294043+0000","last_clean":"2024-09-21T14:02:17.294043+0000","last_became_active":"2024-09-21T13:59:32.201158+0000","last_became_peered":"2024-09-21T13:59:32.201158+0000","last_unstale":"2024-09-21T14:02:17.294043+0000","last_undegraded":"2024-09-21T14:02:17.294043+0000","last_fullsized":"2024-09-21T14:02:17.294043+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:25.792985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:54:25.792985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:46.991952+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":596,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.291293+0000","last_change":"2024-09-21T14:01:41.189797+0000","last_active":"2024-09-21T14:02:17.291293+0000","last_peered":"2024-09-21T14:02:17.291293+0000","last_clean":"2024-09-21T14:02:17.291293+0000","last_became_active":"2024-09-21T14:01:41.189373+0000","last_became_peered":"2024-09-21T14:01:41.189373+0000","last_unstale":"2024-09-21T14:02:17.291293+0000","last_undegraded":"2024-09-21T14:02:17.291293+0000","last_fullsized":"2024-09-21T14:02:17.291293+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:54.328275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:54.328275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"290'7","reported_seq":718,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.292093+0000","last_change":"2024-09-21T14:01:57.563250+0000","last_active":"2024-09-21T14:02:17.292093+0000","last_peered":"2024-09-21T14:02:17.292093+0000","last_clean":"2024-09-21T14:02:17.292093+0000","last_became_active":"2024-09-21T13:59:09.063053+0000","last_became_peered":"2024-09-21T13:59:09.063053+0000","last_unstale":"2024-09-21T14:02:17.292093+0000","last_undegraded":"2024-09-21T14:02:17.292093+0000","last_fullsized":"2024-09-21T14:02:17.292093+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'7","last_scrub_stamp":"2024-09-21T14:01:57.563184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:57.563184+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:19.737603+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":589,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.287420+0000","last_change":"2024-09-21T13:59:32.204565+0000","last_active":"2024-09-21T14:02:17.287420+0000","last_peered":"2024-09-21T14:02:17.287420+0000","last_clean":"2024-09-21T14:02:17.287420+0000","last_became_active":"2024-09-21T13:59:32.204174+0000","last_became_peered":"2024-09-21T13:59:32.204174+0000","last_unstale":"2024-09-21T14:02:17.287420+0000","last_undegraded":"2024-09-21T14:02:17.287420+0000","last_fullsized":"2024-09-21T14:02:17.287420+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:06.722293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:06.722293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":630,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.293128+0000","last_change":"2024-09-21T14:01:41.199085+0000","last_active":"2024-09-21T14:02:17.293128+0000","last_peered":"2024-09-21T14:02:17.293128+0000","last_clean":"2024-09-21T14:02:17.293128+0000","last_became_active":"2024-09-21T14:01:41.198672+0000","last_became_peered":"2024-09-21T14:01:41.198672+0000","last_unstale":"2024-09-21T14:02:17.293128+0000","last_undegraded":"2024-09-21T14:02:17.293128+0000","last_fullsized":"2024-09-21T14:02:17.293128+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:44.376215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:44.376215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":686,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.293180+0000","last_change":"2024-09-21T14:01:53.460257+0000","last_active":"2024-09-21T14:02:17.293180+0000","last_peered":"2024-09-21T14:02:17.293180+0000","last_clean":"2024-09-21T14:02:17.293180+0000","last_became_active":"2024-09-21T14:01:41.200360+0000","last_became_peered":"2024-09-21T14:01:41.200360+0000","last_unstale":"2024-09-21T14:02:17.293180+0000","last_undegraded":"2024-09-21T14:02:17.293180+0000","last_fullsized":"2024-09-21T14:02:17.293180+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:53.460180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:01:53.460180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:15.802657+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":596,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.292331+0000","last_change":"2024-09-21T13:59:32.205037+0000","last_active":"2024-09-21T14:02:17.292331+0000","last_peered":"2024-09-21T14:02:17.292331+0000","last_clean":"2024-09-21T14:02:17.292331+0000","last_became_active":"2024-09-21T13:59:32.204749+0000","last_became_peered":"2024-09-21T13:59:32.204749+0000","last_unstale":"2024-09-21T14:02:17.292331+0000","last_undegraded":"2024-09-21T14:02:17.292331+0000","last_fullsized":"2024-09-21T14:02:17.292331+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:17.361699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:17.361699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:23.593131+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"290'27","reported_seq":661,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.306570+0000","last_change":"2024-09-21T14:01:27.893648+0000","last_active":"2024-09-21T14:02:17.306570+0000","last_peered":"2024-09-21T14:02:17.306570+0000","last_clean":"2024-09-21T14:02:17.306570+0000","last_became_active":"2024-09-21T13:57:12.680783+0000","last_became_peered":"2024-09-21T13:57:12.680783+0000","last_unstale":"2024-09-21T14:02:17.306570+0000","last_undegraded":"2024-09-21T14:02:17.306570+0000","last_fullsized":"2024-09-21T14:02:17.306570+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'27","last_scrub_stamp":"2024-09-21T14:01:27.893587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:27.893587+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:53.939497+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":654,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.294502+0000","last_change":"2024-09-21T14:01:54.024745+0000","last_active":"2024-09-21T14:02:17.294502+0000","last_peered":"2024-09-21T14:02:17.294502+0000","last_clean":"2024-09-21T14:02:17.294502+0000","last_became_active":"2024-09-21T13:49:02.066796+0000","last_became_peered":"2024-09-21T13:49:02.066796+0000","last_unstale":"2024-09-21T14:02:17.294502+0000","last_undegraded":"2024-09-21T14:02:17.294502+0000","last_fullsized":"2024-09-21T14:02:17.294502+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:54.024702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:01:54.024702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:06.646444+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":606,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.298234+0000","last_change":"2024-09-21T14:01:41.191486+0000","last_active":"2024-09-21T14:02:17.298234+0000","last_peered":"2024-09-21T14:02:17.298234+0000","last_clean":"2024-09-21T14:02:17.298234+0000","last_became_active":"2024-09-21T14:01:41.191116+0000","last_became_peered":"2024-09-21T14:01:41.191116+0000","last_unstale":"2024-09-21T14:02:17.298234+0000","last_undegraded":"2024-09-21T14:02:17.298234+0000","last_fullsized":"2024-09-21T14:02:17.298234+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:57.962266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:57.962266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:08.315476+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":664,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.291895+0000","last_change":"2024-09-21T14:00:51.495433+0000","last_active":"2024-09-21T14:02:17.291895+0000","last_peered":"2024-09-21T14:02:17.291895+0000","last_clean":"2024-09-21T14:02:17.291895+0000","last_became_active":"2024-09-21T13:52:04.316016+0000","last_became_peered":"2024-09-21T13:52:04.316016+0000","last_unstale":"2024-09-21T14:02:17.291895+0000","last_undegraded":"2024-09-21T14:02:17.291895+0000","last_fullsized":"2024-09-21T14:02:17.291895+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:00:51.495368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:00:51.495368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:10.415465+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"290'18","reported_seq":612,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.301229+0000","last_change":"2024-09-21T13:59:23.041396+0000","last_active":"2024-09-21T14:02:17.301229+0000","last_peered":"2024-09-21T14:02:17.301229+0000","last_clean":"2024-09-21T14:02:17.301229+0000","last_became_active":"2024-09-21T13:57:12.684454+0000","last_became_peered":"2024-09-21T13:57:12.684454+0000","last_unstale":"2024-09-21T14:02:17.301229+0000","last_undegraded":"2024-09-21T14:02:17.301229+0000","last_fullsized":"2024-09-21T14:02:17.301229+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'18","last_scrub_stamp":"2024-09-21T13:59:23.041311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:23.041311+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":651,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.292368+0000","last_change":"2024-09-21T13:58:16.393763+0000","last_active":"2024-09-21T14:02:17.292368+0000","last_peered":"2024-09-21T14:02:17.292368+0000","last_clean":"2024-09-21T14:02:17.292368+0000","last_became_active":"2024-09-21T13:57:12.681897+0000","last_became_peered":"2024-09-21T13:57:12.681897+0000","last_unstale":"2024-09-21T14:02:17.292368+0000","last_undegraded":"2024-09-21T14:02:17.292368+0000","last_fullsized":"2024-09-21T14:02:17.292368+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:16.393696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:16.393696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:24.632574+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":573,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.287260+0000","last_change":"2024-09-21T14:01:41.182300+0000","last_active":"2024-09-21T14:02:17.287260+0000","last_peered":"2024-09-21T14:02:17.287260+0000","last_clean":"2024-09-21T14:02:17.287260+0000","last_became_active":"2024-09-21T14:01:41.181946+0000","last_became_peered":"2024-09-21T14:01:41.181946+0000","last_unstale":"2024-09-21T14:02:17.287260+0000","last_undegraded":"2024-09-21T14:02:17.287260+0000","last_fullsized":"2024-09-21T14:02:17.287260+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T13:57:26.536684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:26.536684+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":674,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.292342+0000","last_change":"2024-09-21T14:01:58.994867+0000","last_active":"2024-09-21T14:02:17.292342+0000","last_peered":"2024-09-21T14:02:17.292342+0000","last_clean":"2024-09-21T14:02:17.292342+0000","last_became_active":"2024-09-21T13:57:12.685742+0000","last_became_peered":"2024-09-21T13:57:12.685742+0000","last_unstale":"2024-09-21T14:02:17.292342+0000","last_undegraded":"2024-09-21T14:02:17.292342+0000","last_fullsized":"2024-09-21T14:02:17.292342+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T14:01:58.994811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:01:58.994811+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:17.687542+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"290'20","reported_seq":553,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.289515+0000","last_change":"2024-09-21T13:59:32.242771+0000","last_active":"2024-09-21T14:02:17.289515+0000","last_peered":"2024-09-21T14:02:17.289515+0000","last_clean":"2024-09-21T14:02:17.289515+0000","last_became_active":"2024-09-21T13:59:32.213865+0000","last_became_peered":"2024-09-21T13:59:32.213865+0000","last_unstale":"2024-09-21T14:02:17.289515+0000","last_undegraded":"2024-09-21T14:02:17.289515+0000","last_fullsized":"2024-09-21T14:02:17.289515+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:57:27.578738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:57:27.578738+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"300'12","reported_seq":283,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.295926+0000","last_change":"2024-09-21T13:59:36.012775+0000","last_active":"2024-09-21T14:02:17.295926+0000","last_peered":"2024-09-21T14:02:17.295926+0000","last_clean":"2024-09-21T14:02:17.295926+0000","last_became_active":"2024-09-21T13:59:32.205051+0000","last_became_peered":"2024-09-21T13:59:32.205051+0000","last_unstale":"2024-09-21T14:02:17.295926+0000","last_undegraded":"2024-09-21T14:02:17.295926+0000","last_fullsized":"2024-09-21T14:02:17.295926+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'12","last_scrub_stamp":"2024-09-21T13:59:36.012693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:36.012693+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"75'79","reported_seq":69,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.292987+0000","last_change":"2024-09-21T14:01:41.248772+0000","last_active":"2024-09-21T14:02:17.292987+0000","last_peered":"2024-09-21T14:02:17.292987+0000","last_clean":"2024-09-21T14:02:17.292987+0000","last_became_active":"2024-09-21T14:01:41.246888+0000","last_became_peered":"2024-09-21T14:01:41.246888+0000","last_unstale":"2024-09-21T14:02:17.292987+0000","last_undegraded":"2024-09-21T14:02:17.292987+0000","last_fullsized":"2024-09-21T14:02:17.292987+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53669,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"368'13593","reported_seq":13822,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:21.136694+0000","last_change":"2024-09-21T13:57:53.208687+0000","last_active":"2024-09-21T14:02:21.136694+0000","last_peered":"2024-09-21T14:02:21.136694+0000","last_clean":"2024-09-21T14:02:21.136694+0000","last_became_active":"2024-09-21T13:57:53.208352+0000","last_became_peered":"2024-09-21T13:57:53.208352+0000","last_unstale":"2024-09-21T14:02:21.136694+0000","last_undegraded":"2024-09-21T14:02:21.136694+0000","last_fullsized":"2024-09-21T14:02:21.136694+0000","mapping_epoch":260,"log_start":"310'9215","ondisk_log_start":"310'9215","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4378,"log_dups_size":0,"ondisk_log_size":4378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106364928,"num_objects":1623,"num_object_clones":0,"num_object_copies":4869,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13593,"num_write_kb":486912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"290'27","reported_seq":712,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.294141+0000","last_change":"2024-09-21T13:58:45.984691+0000","last_active":"2024-09-21T14:02:17.294141+0000","last_peered":"2024-09-21T14:02:17.294141+0000","last_clean":"2024-09-21T14:02:17.294141+0000","last_became_active":"2024-09-21T13:57:12.695609+0000","last_became_peered":"2024-09-21T13:57:12.695609+0000","last_unstale":"2024-09-21T14:02:17.294141+0000","last_undegraded":"2024-09-21T14:02:17.294141+0000","last_fullsized":"2024-09-21T14:02:17.294141+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:58:45.984575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:45.984575+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":598,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.300250+0000","last_change":"2024-09-21T13:59:32.205723+0000","last_active":"2024-09-21T14:02:17.300250+0000","last_peered":"2024-09-21T14:02:17.300250+0000","last_clean":"2024-09-21T14:02:17.300250+0000","last_became_active":"2024-09-21T13:59:32.205350+0000","last_became_peered":"2024-09-21T13:59:32.205350+0000","last_unstale":"2024-09-21T14:02:17.300250+0000","last_undegraded":"2024-09-21T14:02:17.300250+0000","last_fullsized":"2024-09-21T14:02:17.300250+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:32.668851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:32.668851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"346'10","reported_seq":839,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.293221+0000","last_change":"2024-09-21T14:01:55.521513+0000","last_active":"2024-09-21T14:02:17.293221+0000","last_peered":"2024-09-21T14:02:17.293221+0000","last_clean":"2024-09-21T14:02:17.293221+0000","last_became_active":"2024-09-21T14:01:41.200218+0000","last_became_peered":"2024-09-21T14:01:41.200218+0000","last_unstale":"2024-09-21T14:02:17.293221+0000","last_undegraded":"2024-09-21T14:02:17.293221+0000","last_fullsized":"2024-09-21T14:02:17.293221+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'10","last_scrub_stamp":"2024-09-21T14:01:55.521438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:01:55.521438+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:55.829821+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":525,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.294504+0000","last_change":"2024-09-21T14:01:41.207310+0000","last_active":"2024-09-21T14:02:17.294504+0000","last_peered":"2024-09-21T14:02:17.294504+0000","last_clean":"2024-09-21T14:02:17.294504+0000","last_became_active":"2024-09-21T14:01:41.206720+0000","last_became_peered":"2024-09-21T14:01:41.206720+0000","last_unstale":"2024-09-21T14:02:17.294504+0000","last_undegraded":"2024-09-21T14:02:17.294504+0000","last_fullsized":"2024-09-21T14:02:17.294504+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:52.244483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:52.244483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"75'79","reported_seq":122,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.293027+0000","last_change":"2024-09-21T14:02:06.395083+0000","last_active":"2024-09-21T14:02:17.293027+0000","last_peered":"2024-09-21T14:02:17.293027+0000","last_clean":"2024-09-21T14:02:17.293027+0000","last_became_active":"2024-09-21T14:00:41.137756+0000","last_became_peered":"2024-09-21T14:00:41.137756+0000","last_unstale":"2024-09-21T14:02:17.293027+0000","last_undegraded":"2024-09-21T14:02:17.293027+0000","last_fullsized":"2024-09-21T14:02:17.293027+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:02:06.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:02:06.395011+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:29.136654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"368'13426","reported_seq":16942,"reported_epoch":368,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-21T14:02:19.664064+0000","last_change":"2024-09-21T14:02:11.235474+0000","last_active":"2024-09-21T14:02:19.664064+0000","last_peered":"2024-09-21T14:02:19.664064+0000","last_clean":"2024-09-21T13:58:23.757609+0000","last_became_active":"2024-09-21T14:01:41.187681+0000","last_became_peered":"2024-09-21T14:01:41.187681+0000","last_unstale":"2024-09-21T14:02:19.664064+0000","last_undegraded":"2024-09-21T14:01:41.168266+0000","last_fullsized":"2024-09-21T14:01:41.168096+0000","mapping_epoch":345,"log_start":"307'8820","ondisk_log_start":"307'8820","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4606,"log_dups_size":0,"ondisk_log_size":4606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116850712,"num_objects":1784,"num_object_clones":0,"num_object_copies":5352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":675,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1784,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13426,"num_write_kb":486657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1628,"num_bytes_recovered":106627096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":1784}],"blocked_by":[],"up_primary":7,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"290'30","reported_seq":622,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.300353+0000","last_change":"2024-09-21T13:59:32.246090+0000","last_active":"2024-09-21T14:02:17.300353+0000","last_peered":"2024-09-21T14:02:17.300353+0000","last_clean":"2024-09-21T14:02:17.300353+0000","last_became_active":"2024-09-21T13:59:32.205887+0000","last_became_peered":"2024-09-21T13:59:32.205887+0000","last_unstale":"2024-09-21T14:02:17.300353+0000","last_undegraded":"2024-09-21T14:02:17.300353+0000","last_fullsized":"2024-09-21T14:02:17.300353+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'18","last_scrub_stamp":"2024-09-21T13:54:19.001030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:19.001030+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:40.231306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":550,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.300932+0000","last_change":"2024-09-21T13:59:32.212559+0000","last_active":"2024-09-21T14:02:17.300932+0000","last_peered":"2024-09-21T14:02:17.300932+0000","last_clean":"2024-09-21T14:02:17.300932+0000","last_became_active":"2024-09-21T13:59:32.211004+0000","last_became_peered":"2024-09-21T13:59:32.211004+0000","last_unstale":"2024-09-21T14:02:17.300932+0000","last_undegraded":"2024-09-21T14:02:17.300932+0000","last_fullsized":"2024-09-21T14:02:17.300932+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:25.736461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:25.736461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":668,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.290761+0000","last_change":"2024-09-21T13:59:34.293330+0000","last_active":"2024-09-21T14:02:17.290761+0000","last_peered":"2024-09-21T14:02:17.290761+0000","last_clean":"2024-09-21T14:02:17.290761+0000","last_became_active":"2024-09-21T13:59:32.220304+0000","last_became_peered":"2024-09-21T13:59:32.220304+0000","last_unstale":"2024-09-21T14:02:17.290761+0000","last_undegraded":"2024-09-21T14:02:17.290761+0000","last_fullsized":"2024-09-21T14:02:17.290761+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:34.292892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:34.292892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:15.468420+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":648,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.291677+0000","last_change":"2024-09-21T13:59:23.334230+0000","last_active":"2024-09-21T14:02:17.291677+0000","last_peered":"2024-09-21T14:02:17.291677+0000","last_clean":"2024-09-21T14:02:17.291677+0000","last_became_active":"2024-09-21T13:57:12.680372+0000","last_became_peered":"2024-09-21T13:57:12.680372+0000","last_unstale":"2024-09-21T14:02:17.291677+0000","last_undegraded":"2024-09-21T14:02:17.291677+0000","last_fullsized":"2024-09-21T14:02:17.291677+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:23.334162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:23.334162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"75'79","reported_seq":125,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.298429+0000","last_change":"2024-09-21T14:01:42.177646+0000","last_active":"2024-09-21T14:02:17.298429+0000","last_peered":"2024-09-21T14:02:17.298429+0000","last_clean":"2024-09-21T14:02:17.298429+0000","last_became_active":"2024-09-21T14:01:42.173042+0000","last_became_peered":"2024-09-21T14:01:42.173042+0000","last_unstale":"2024-09-21T14:02:17.298429+0000","last_undegraded":"2024-09-21T14:02:17.298429+0000","last_fullsized":"2024-09-21T14:02:17.298429+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:02:42.254915+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":709030,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"368'13422","reported_seq":20583,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:21.153130+0000","last_change":"2024-09-21T14:01:00.058151+0000","last_active":"2024-09-21T14:02:21.153130+0000","last_peered":"2024-09-21T14:02:21.153130+0000","last_clean":"2024-09-21T14:02:21.153130+0000","last_became_active":"2024-09-21T14:01:00.026605+0000","last_became_peered":"2024-09-21T14:01:00.026605+0000","last_unstale":"2024-09-21T14:02:21.153130+0000","last_undegraded":"2024-09-21T14:02:21.153130+0000","last_fullsized":"2024-09-21T14:02:21.153130+0000","mapping_epoch":329,"log_start":"310'9042","ondisk_log_start":"310'9042","created":260,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4380,"log_dups_size":0,"ondisk_log_size":4380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106692608,"num_objects":1628,"num_object_clones":0,"num_object_copies":4884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13422,"num_write_kb":481600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3456,"num_bytes_recovered":226492416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"290'28","reported_seq":757,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.290848+0000","last_change":"2024-09-21T14:02:02.471226+0000","last_active":"2024-09-21T14:02:17.290848+0000","last_peered":"2024-09-21T14:02:17.290848+0000","last_clean":"2024-09-21T14:02:17.290848+0000","last_became_active":"2024-09-21T14:01:41.200859+0000","last_became_peered":"2024-09-21T14:01:41.200859+0000","last_unstale":"2024-09-21T14:02:17.290848+0000","last_undegraded":"2024-09-21T14:02:17.290848+0000","last_fullsized":"2024-09-21T14:02:17.290848+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'28","last_scrub_stamp":"2024-09-21T14:02:02.471133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:02.471133+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:14.278651+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":656,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.290639+0000","last_change":"2024-09-21T13:57:38.046718+0000","last_active":"2024-09-21T14:02:17.290639+0000","last_peered":"2024-09-21T14:02:17.290639+0000","last_clean":"2024-09-21T14:02:17.290639+0000","last_became_active":"2024-09-21T13:48:58.053533+0000","last_became_peered":"2024-09-21T13:48:58.053533+0000","last_unstale":"2024-09-21T14:02:17.290639+0000","last_undegraded":"2024-09-21T14:02:17.290639+0000","last_fullsized":"2024-09-21T14:02:17.290639+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:57:38.046627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:38.046627+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:41.742834+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"241'10","reported_seq":692,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.300609+0000","last_change":"2024-09-21T13:58:06.789629+0000","last_active":"2024-09-21T14:02:17.300609+0000","last_peered":"2024-09-21T14:02:17.300609+0000","last_clean":"2024-09-21T14:02:17.300609+0000","last_became_active":"2024-09-21T13:57:12.686651+0000","last_became_peered":"2024-09-21T13:57:12.686651+0000","last_unstale":"2024-09-21T14:02:17.300609+0000","last_undegraded":"2024-09-21T14:02:17.300609+0000","last_fullsized":"2024-09-21T14:02:17.300609+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'10","last_scrub_stamp":"2024-09-21T13:58:06.789560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:06.789560+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":614,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.288477+0000","last_change":"2024-09-21T13:59:32.210257+0000","last_active":"2024-09-21T14:02:17.288477+0000","last_peered":"2024-09-21T14:02:17.288477+0000","last_clean":"2024-09-21T14:02:17.288477+0000","last_became_active":"2024-09-21T13:59:32.209615+0000","last_became_peered":"2024-09-21T13:59:32.209615+0000","last_unstale":"2024-09-21T14:02:17.288477+0000","last_undegraded":"2024-09-21T14:02:17.288477+0000","last_fullsized":"2024-09-21T14:02:17.288477+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:37.548064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:37.548064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"75'79","reported_seq":695,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.295715+0000","last_change":"2024-09-21T14:02:05.693250+0000","last_active":"2024-09-21T14:02:17.295715+0000","last_peered":"2024-09-21T14:02:17.295715+0000","last_clean":"2024-09-21T14:02:17.295715+0000","last_became_active":"2024-09-21T14:00:41.083425+0000","last_became_peered":"2024-09-21T14:00:41.083425+0000","last_unstale":"2024-09-21T14:02:17.295715+0000","last_undegraded":"2024-09-21T14:02:17.295715+0000","last_fullsized":"2024-09-21T14:02:17.295715+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:02:05.693193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:02:05.693193+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:28.620866+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.f","version":"368'13654","reported_seq":13870,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:20.796473+0000","last_change":"2024-09-21T14:01:41.336305+0000","last_active":"2024-09-21T14:02:20.796473+0000","last_peered":"2024-09-21T14:02:20.796473+0000","last_clean":"2024-09-21T14:02:20.796473+0000","last_became_active":"2024-09-21T14:01:41.202052+0000","last_became_peered":"2024-09-21T14:01:41.202052+0000","last_unstale":"2024-09-21T14:02:20.796473+0000","last_undegraded":"2024-09-21T14:02:20.796473+0000","last_fullsized":"2024-09-21T14:02:20.796473+0000","mapping_epoch":345,"log_start":"305'8702","ondisk_log_start":"305'8702","created":260,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4952,"log_dups_size":0,"ondisk_log_size":4952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110362624,"num_objects":1684,"num_object_clones":0,"num_object_copies":5052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13654,"num_write_kb":490816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"290'20","reported_seq":692,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.295653+0000","last_change":"2024-09-21T14:01:41.187743+0000","last_active":"2024-09-21T14:02:17.295653+0000","last_peered":"2024-09-21T14:02:17.295653+0000","last_clean":"2024-09-21T14:02:17.295653+0000","last_became_active":"2024-09-21T14:01:41.187060+0000","last_became_peered":"2024-09-21T14:01:41.187060+0000","last_unstale":"2024-09-21T14:02:17.295653+0000","last_undegraded":"2024-09-21T14:02:17.295653+0000","last_fullsized":"2024-09-21T14:02:17.295653+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:54:19.593611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:19.593611+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:45.999996+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"61'1","reported_seq":587,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.294738+0000","last_change":"2024-09-21T14:01:41.195580+0000","last_active":"2024-09-21T14:02:17.294738+0000","last_peered":"2024-09-21T14:02:17.294738+0000","last_clean":"2024-09-21T14:02:17.294738+0000","last_became_active":"2024-09-21T14:01:41.195268+0000","last_became_peered":"2024-09-21T14:01:41.195268+0000","last_unstale":"2024-09-21T14:02:17.294738+0000","last_undegraded":"2024-09-21T14:02:17.294738+0000","last_fullsized":"2024-09-21T14:02:17.294738+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:58:48.407734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:48.407734+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"346'15","reported_seq":276,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.295185+0000","last_change":"2024-09-21T14:01:41.209662+0000","last_active":"2024-09-21T14:02:17.295185+0000","last_peered":"2024-09-21T14:02:17.295185+0000","last_clean":"2024-09-21T14:02:17.295185+0000","last_became_active":"2024-09-21T14:01:41.209296+0000","last_became_peered":"2024-09-21T14:01:41.209296+0000","last_unstale":"2024-09-21T14:02:17.295185+0000","last_undegraded":"2024-09-21T14:02:17.295185+0000","last_fullsized":"2024-09-21T14:02:17.295185+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'12","last_scrub_stamp":"2024-09-21T13:57:30.133392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:30.133392+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:41.760065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":468,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.300448+0000","last_change":"2024-09-21T13:58:58.730439+0000","last_active":"2024-09-21T14:02:17.300448+0000","last_peered":"2024-09-21T14:02:17.300448+0000","last_clean":"2024-09-21T14:02:17.300448+0000","last_became_active":"2024-09-21T13:57:12.704897+0000","last_became_peered":"2024-09-21T13:57:12.704897+0000","last_unstale":"2024-09-21T14:02:17.300448+0000","last_undegraded":"2024-09-21T14:02:17.300448+0000","last_fullsized":"2024-09-21T14:02:17.300448+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:58.730368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:58.730368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"75'79","reported_seq":695,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.295577+0000","last_change":"2024-09-21T14:02:03.737005+0000","last_active":"2024-09-21T14:02:17.295577+0000","last_peered":"2024-09-21T14:02:17.295577+0000","last_clean":"2024-09-21T14:02:17.295577+0000","last_became_active":"2024-09-21T14:00:41.073047+0000","last_became_peered":"2024-09-21T14:00:41.073047+0000","last_unstale":"2024-09-21T14:02:17.295577+0000","last_undegraded":"2024-09-21T14:02:17.295577+0000","last_fullsized":"2024-09-21T14:02:17.295577+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:02:03.736928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:02:03.736928+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:21.563427+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"368'13295","reported_seq":23078,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:21.097087+0000","last_change":"2024-09-21T14:00:42.188751+0000","last_active":"2024-09-21T14:02:21.097087+0000","last_peered":"2024-09-21T14:02:21.097087+0000","last_clean":"2024-09-21T14:02:21.097087+0000","last_became_active":"2024-09-21T14:00:42.129476+0000","last_became_peered":"2024-09-21T14:00:42.129476+0000","last_unstale":"2024-09-21T14:02:21.097087+0000","last_undegraded":"2024-09-21T14:02:21.097087+0000","last_fullsized":"2024-09-21T14:02:21.097087+0000","mapping_epoch":321,"log_start":"307'8669","ondisk_log_start":"307'8669","created":260,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4626,"log_dups_size":0,"ondisk_log_size":4626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106889216,"num_objects":1631,"num_object_clones":0,"num_object_copies":4893,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13295,"num_write_kb":477632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4765,"num_bytes_recovered":312213504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"290'29","reported_seq":593,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.293701+0000","last_change":"2024-09-21T14:01:41.188261+0000","last_active":"2024-09-21T14:02:17.293701+0000","last_peered":"2024-09-21T14:02:17.293701+0000","last_clean":"2024-09-21T14:02:17.293701+0000","last_became_active":"2024-09-21T14:01:41.186960+0000","last_became_peered":"2024-09-21T14:01:41.186960+0000","last_unstale":"2024-09-21T14:02:17.293701+0000","last_undegraded":"2024-09-21T14:02:17.293701+0000","last_fullsized":"2024-09-21T14:02:17.293701+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-21T13:56:55.319759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:56:55.319759+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":669,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.292653+0000","last_change":"2024-09-21T14:01:41.185706+0000","last_active":"2024-09-21T14:02:17.292653+0000","last_peered":"2024-09-21T14:02:17.292653+0000","last_clean":"2024-09-21T14:02:17.292653+0000","last_became_active":"2024-09-21T14:01:41.185400+0000","last_became_peered":"2024-09-21T14:01:41.185400+0000","last_unstale":"2024-09-21T14:02:17.292653+0000","last_undegraded":"2024-09-21T14:02:17.292653+0000","last_fullsized":"2024-09-21T14:02:17.292653+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:16.712990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:16.712990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":236,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.295357+0000","last_change":"2024-09-21T14:01:52.671029+0000","last_active":"2024-09-21T14:02:17.295357+0000","last_peered":"2024-09-21T14:02:17.295357+0000","last_clean":"2024-09-21T14:02:17.295357+0000","last_became_active":"2024-09-21T14:01:41.193973+0000","last_became_peered":"2024-09-21T14:01:41.193973+0000","last_unstale":"2024-09-21T14:02:17.295357+0000","last_undegraded":"2024-09-21T14:02:17.295357+0000","last_fullsized":"2024-09-21T14:02:17.295357+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:52.670974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:01:52.670974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:58.750020+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":660,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.292117+0000","last_change":"2024-09-21T13:58:43.952014+0000","last_active":"2024-09-21T14:02:17.292117+0000","last_peered":"2024-09-21T14:02:17.292117+0000","last_clean":"2024-09-21T14:02:17.292117+0000","last_became_active":"2024-09-21T13:58:17.757628+0000","last_became_peered":"2024-09-21T13:58:17.757628+0000","last_unstale":"2024-09-21T14:02:17.292117+0000","last_undegraded":"2024-09-21T14:02:17.292117+0000","last_fullsized":"2024-09-21T14:02:17.292117+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:43.951941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:43.951941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"75'79","reported_seq":69,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.293255+0000","last_change":"2024-09-21T14:01:41.199377+0000","last_active":"2024-09-21T14:02:17.293255+0000","last_peered":"2024-09-21T14:02:17.293255+0000","last_clean":"2024-09-21T14:02:17.293255+0000","last_became_active":"2024-09-21T14:01:41.198750+0000","last_became_peered":"2024-09-21T14:01:41.198750+0000","last_unstale":"2024-09-21T14:02:17.293255+0000","last_undegraded":"2024-09-21T14:02:17.293255+0000","last_fullsized":"2024-09-21T14:02:17.293255+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:02:43.568139+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"368'13639","reported_seq":24452,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:21.160533+0000","last_change":"2024-09-21T14:01:44.191023+0000","last_active":"2024-09-21T14:02:21.160533+0000","last_peered":"2024-09-21T14:02:21.160533+0000","last_clean":"2024-09-21T14:02:21.160533+0000","last_became_active":"2024-09-21T14:01:44.174262+0000","last_became_peered":"2024-09-21T14:01:44.174262+0000","last_unstale":"2024-09-21T14:02:21.160533+0000","last_undegraded":"2024-09-21T14:02:21.160533+0000","last_fullsized":"2024-09-21T14:02:21.160533+0000","mapping_epoch":348,"log_start":"309'9031","ondisk_log_start":"309'9031","created":260,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4608,"log_dups_size":0,"ondisk_log_size":4608,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107020288,"num_objects":1633,"num_object_clones":0,"num_object_copies":4899,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13639,"num_write_kb":488704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5260,"num_bytes_recovered":344588288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"290'44","reported_seq":896,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.292548+0000","last_change":"2024-09-21T13:58:56.076153+0000","last_active":"2024-09-21T14:02:17.292548+0000","last_peered":"2024-09-21T14:02:17.292548+0000","last_clean":"2024-09-21T14:02:17.292548+0000","last_became_active":"2024-09-21T13:57:12.684690+0000","last_became_peered":"2024-09-21T13:57:12.684690+0000","last_unstale":"2024-09-21T14:02:17.292548+0000","last_undegraded":"2024-09-21T14:02:17.292548+0000","last_fullsized":"2024-09-21T14:02:17.292548+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'32","last_scrub_stamp":"2024-09-21T13:58:56.076074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:56.076074+0000","objects_scrubbed":10,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":676,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.292455+0000","last_change":"2024-09-21T14:01:58.524159+0000","last_active":"2024-09-21T14:02:17.292455+0000","last_peered":"2024-09-21T14:02:17.292455+0000","last_clean":"2024-09-21T14:02:17.292455+0000","last_became_active":"2024-09-21T13:58:17.770937+0000","last_became_peered":"2024-09-21T13:58:17.770937+0000","last_unstale":"2024-09-21T14:02:17.292455+0000","last_undegraded":"2024-09-21T14:02:17.292455+0000","last_fullsized":"2024-09-21T14:02:17.292455+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:58.524114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:01:58.524114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:07.944133+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":606,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.292910+0000","last_change":"2024-09-21T14:01:41.249061+0000","last_active":"2024-09-21T14:02:17.292910+0000","last_peered":"2024-09-21T14:02:17.292910+0000","last_clean":"2024-09-21T14:02:17.292910+0000","last_became_active":"2024-09-21T14:01:41.247642+0000","last_became_peered":"2024-09-21T14:01:41.247642+0000","last_unstale":"2024-09-21T14:02:17.292910+0000","last_undegraded":"2024-09-21T14:02:17.292910+0000","last_fullsized":"2024-09-21T14:02:17.292910+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-21T13:59:22.186393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:59:22.186393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":587,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.293979+0000","last_change":"2024-09-21T13:55:07.021128+0000","last_active":"2024-09-21T14:02:17.293979+0000","last_peered":"2024-09-21T14:02:17.293979+0000","last_clean":"2024-09-21T14:02:17.293979+0000","last_became_active":"2024-09-21T13:50:29.688249+0000","last_became_peered":"2024-09-21T13:50:29.688249+0000","last_unstale":"2024-09-21T14:02:17.293979+0000","last_undegraded":"2024-09-21T14:02:17.293979+0000","last_fullsized":"2024-09-21T14:02:17.293979+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:07.021059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:55:07.021059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:47.974697+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":577,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.301939+0000","last_change":"2024-09-21T13:59:32.209357+0000","last_active":"2024-09-21T14:02:17.301939+0000","last_peered":"2024-09-21T14:02:17.301939+0000","last_clean":"2024-09-21T14:02:17.301939+0000","last_became_active":"2024-09-21T13:59:32.208980+0000","last_became_peered":"2024-09-21T13:59:32.208980+0000","last_unstale":"2024-09-21T14:02:17.301939+0000","last_undegraded":"2024-09-21T14:02:17.301939+0000","last_fullsized":"2024-09-21T14:02:17.301939+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:16.797227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:16.797227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"75'79","reported_seq":701,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.287216+0000","last_change":"2024-09-21T14:00:41.077286+0000","last_active":"2024-09-21T14:02:17.287216+0000","last_peered":"2024-09-21T14:02:17.287216+0000","last_clean":"2024-09-21T14:02:17.287216+0000","last_became_active":"2024-09-21T14:00:41.076948+0000","last_became_peered":"2024-09-21T14:00:41.076948+0000","last_unstale":"2024-09-21T14:02:17.287216+0000","last_undegraded":"2024-09-21T14:02:17.287216+0000","last_fullsized":"2024-09-21T14:02:17.287216+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:02:44.823429+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.a","version":"368'13257","reported_seq":14181,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:20.793487+0000","last_change":"2024-09-21T14:01:41.339174+0000","last_active":"2024-09-21T14:02:20.793487+0000","last_peered":"2024-09-21T14:02:20.793487+0000","last_clean":"2024-09-21T14:02:20.793487+0000","last_became_active":"2024-09-21T14:01:41.190267+0000","last_became_peered":"2024-09-21T14:01:41.190267+0000","last_unstale":"2024-09-21T14:02:20.793487+0000","last_undegraded":"2024-09-21T14:02:20.793487+0000","last_fullsized":"2024-09-21T14:02:20.793487+0000","mapping_epoch":345,"log_start":"305'8274","ondisk_log_start":"305'8274","created":260,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4983,"log_dups_size":0,"ondisk_log_size":4983,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105185280,"num_objects":1605,"num_object_clones":0,"num_object_copies":4815,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13257,"num_write_kb":475584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5098,"num_bytes_recovered":241565696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"290'17","reported_seq":695,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.293853+0000","last_change":"2024-09-21T14:01:41.200809+0000","last_active":"2024-09-21T14:02:17.293853+0000","last_peered":"2024-09-21T14:02:17.293853+0000","last_clean":"2024-09-21T14:02:17.293853+0000","last_became_active":"2024-09-21T14:01:41.200365+0000","last_became_peered":"2024-09-21T14:01:41.200365+0000","last_unstale":"2024-09-21T14:02:17.293853+0000","last_undegraded":"2024-09-21T14:02:17.293853+0000","last_fullsized":"2024-09-21T14:02:17.293853+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'13","last_scrub_stamp":"2024-09-21T13:58:11.619848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:11.619848+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":696,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.293330+0000","last_change":"2024-09-21T13:59:29.454265+0000","last_active":"2024-09-21T14:02:17.293330+0000","last_peered":"2024-09-21T14:02:17.293330+0000","last_clean":"2024-09-21T14:02:17.293330+0000","last_became_active":"2024-09-21T13:58:17.759671+0000","last_became_peered":"2024-09-21T13:58:17.759671+0000","last_unstale":"2024-09-21T14:02:17.293330+0000","last_undegraded":"2024-09-21T14:02:17.293330+0000","last_fullsized":"2024-09-21T14:02:17.293330+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:29.454085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:59:29.454085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":585,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.302637+0000","last_change":"2024-09-21T13:58:41.644876+0000","last_active":"2024-09-21T14:02:17.302637+0000","last_peered":"2024-09-21T14:02:17.302637+0000","last_clean":"2024-09-21T14:02:17.302637+0000","last_became_active":"2024-09-21T13:57:12.682147+0000","last_became_peered":"2024-09-21T13:57:12.682147+0000","last_unstale":"2024-09-21T14:02:17.302637+0000","last_undegraded":"2024-09-21T14:02:17.302637+0000","last_fullsized":"2024-09-21T14:02:17.302637+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:58:41.644789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:58:41.644789+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:13.796854+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":564,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.307100+0000","last_change":"2024-09-21T14:01:50.880840+0000","last_active":"2024-09-21T14:02:17.307100+0000","last_peered":"2024-09-21T14:02:17.307100+0000","last_clean":"2024-09-21T14:02:17.307100+0000","last_became_active":"2024-09-21T14:01:41.182000+0000","last_became_peered":"2024-09-21T14:01:41.182000+0000","last_unstale":"2024-09-21T14:02:17.307100+0000","last_undegraded":"2024-09-21T14:02:17.307100+0000","last_fullsized":"2024-09-21T14:02:17.307100+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:50.880471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:01:50.880471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:14.118692+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":622,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.291231+0000","last_change":"2024-09-21T13:59:32.222791+0000","last_active":"2024-09-21T14:02:17.291231+0000","last_peered":"2024-09-21T14:02:17.291231+0000","last_clean":"2024-09-21T14:02:17.291231+0000","last_became_active":"2024-09-21T13:59:32.221502+0000","last_became_peered":"2024-09-21T13:59:32.221502+0000","last_unstale":"2024-09-21T14:02:17.291231+0000","last_undegraded":"2024-09-21T14:02:17.291231+0000","last_fullsized":"2024-09-21T14:02:17.291231+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:14.673582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:14.673582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"75'79","reported_seq":106,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.292732+0000","last_change":"2024-09-21T14:00:41.071631+0000","last_active":"2024-09-21T14:02:17.292732+0000","last_peered":"2024-09-21T14:02:17.292732+0000","last_clean":"2024-09-21T14:02:17.292732+0000","last_became_active":"2024-09-21T14:00:41.071042+0000","last_became_peered":"2024-09-21T14:00:41.071042+0000","last_unstale":"2024-09-21T14:02:17.292732+0000","last_undegraded":"2024-09-21T14:02:17.292732+0000","last_fullsized":"2024-09-21T14:02:17.292732+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:02:42.091028+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.b","version":"368'13359","reported_seq":17556,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:18.517572+0000","last_change":"2024-09-21T14:01:54.238547+0000","last_active":"2024-09-21T14:02:18.517572+0000","last_peered":"2024-09-21T14:02:18.517572+0000","last_clean":"2024-09-21T14:02:18.517572+0000","last_became_active":"2024-09-21T14:01:41.210137+0000","last_became_peered":"2024-09-21T14:01:41.210137+0000","last_unstale":"2024-09-21T14:02:18.517572+0000","last_undegraded":"2024-09-21T14:02:18.517572+0000","last_fullsized":"2024-09-21T14:02:18.517572+0000","mapping_epoch":345,"log_start":"296'7432","ondisk_log_start":"296'7432","created":260,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"356'12122","last_scrub_stamp":"2024-09-21T14:01:54.238344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T14:01:54.238344+0000","objects_scrubbed":3072,"log_size":5927,"log_dups_size":0,"ondisk_log_size":5927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:15.605090+0000","scrub_duration":597,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119603200,"num_objects":1825,"num_object_clones":0,"num_object_copies":5475,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13359,"num_write_kb":485888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1910,"num_bytes_recovered":123994112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"290'24","reported_seq":746,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.290454+0000","last_change":"2024-09-21T14:01:41.185105+0000","last_active":"2024-09-21T14:02:17.290454+0000","last_peered":"2024-09-21T14:02:17.290454+0000","last_clean":"2024-09-21T14:02:17.290454+0000","last_became_active":"2024-09-21T14:01:41.184728+0000","last_became_peered":"2024-09-21T14:01:41.184728+0000","last_unstale":"2024-09-21T14:02:17.290454+0000","last_undegraded":"2024-09-21T14:02:17.290454+0000","last_fullsized":"2024-09-21T14:02:17.290454+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'24","last_scrub_stamp":"2024-09-21T13:59:28.458768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:28.458768+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":597,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.290987+0000","last_change":"2024-09-21T13:59:59.450613+0000","last_active":"2024-09-21T14:02:17.290987+0000","last_peered":"2024-09-21T14:02:17.290987+0000","last_clean":"2024-09-21T14:02:17.290987+0000","last_became_active":"2024-09-21T13:57:12.681030+0000","last_became_peered":"2024-09-21T13:57:12.681030+0000","last_unstale":"2024-09-21T14:02:17.290987+0000","last_undegraded":"2024-09-21T14:02:17.290987+0000","last_fullsized":"2024-09-21T14:02:17.290987+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:59.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:59:59.450546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:26.702708+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":1325,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:19.453184+0000","last_change":"2024-09-21T13:59:32.218796+0000","last_active":"2024-09-21T14:02:19.453184+0000","last_peered":"2024-09-21T14:02:19.453184+0000","last_clean":"2024-09-21T14:02:19.453184+0000","last_became_active":"2024-09-21T13:59:32.203361+0000","last_became_peered":"2024-09-21T13:59:32.203361+0000","last_unstale":"2024-09-21T14:02:19.453184+0000","last_undegraded":"2024-09-21T14:02:19.453184+0000","last_fullsized":"2024-09-21T14:02:19.453184+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-21T13:56:29.617143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:56:29.617143+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047048000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":175,"num_read_kb":170,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":577,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.293746+0000","last_change":"2024-09-21T14:01:41.212595+0000","last_active":"2024-09-21T14:02:17.293746+0000","last_peered":"2024-09-21T14:02:17.293746+0000","last_clean":"2024-09-21T14:02:17.293746+0000","last_became_active":"2024-09-21T14:01:41.200531+0000","last_became_peered":"2024-09-21T14:01:41.200531+0000","last_unstale":"2024-09-21T14:02:17.293746+0000","last_undegraded":"2024-09-21T14:02:17.293746+0000","last_fullsized":"2024-09-21T14:02:17.293746+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:33.316230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:56:33.316230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":681,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.292048+0000","last_change":"2024-09-21T14:01:41.184939+0000","last_active":"2024-09-21T14:02:17.292048+0000","last_peered":"2024-09-21T14:02:17.292048+0000","last_clean":"2024-09-21T14:02:17.292048+0000","last_became_active":"2024-09-21T14:01:41.184664+0000","last_became_peered":"2024-09-21T14:01:41.184664+0000","last_unstale":"2024-09-21T14:02:17.292048+0000","last_undegraded":"2024-09-21T14:02:17.292048+0000","last_fullsized":"2024-09-21T14:02:17.292048+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:26.557002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:26.557002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"368'13792","reported_seq":14060,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:21.142455+0000","last_change":"2024-09-21T14:01:41.317185+0000","last_active":"2024-09-21T14:02:21.142455+0000","last_peered":"2024-09-21T14:02:21.142455+0000","last_clean":"2024-09-21T14:02:21.142455+0000","last_became_active":"2024-09-21T14:01:41.197526+0000","last_became_peered":"2024-09-21T14:01:41.197526+0000","last_unstale":"2024-09-21T14:02:21.142455+0000","last_undegraded":"2024-09-21T14:02:21.142455+0000","last_fullsized":"2024-09-21T14:02:21.142455+0000","mapping_epoch":345,"log_start":"310'9387","ondisk_log_start":"310'9387","created":260,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4405,"log_dups_size":0,"ondisk_log_size":4405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105906176,"num_objects":1616,"num_object_clones":0,"num_object_copies":4848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13792,"num_write_kb":493056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"300'32","reported_seq":329,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.296227+0000","last_change":"2024-09-21T14:00:47.130089+0000","last_active":"2024-09-21T14:02:17.296227+0000","last_peered":"2024-09-21T14:02:17.296227+0000","last_clean":"2024-09-21T14:02:17.296227+0000","last_became_active":"2024-09-21T13:59:32.204624+0000","last_became_peered":"2024-09-21T13:59:32.204624+0000","last_unstale":"2024-09-21T14:02:17.296227+0000","last_undegraded":"2024-09-21T14:02:17.296227+0000","last_fullsized":"2024-09-21T14:02:17.296227+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'32","last_scrub_stamp":"2024-09-21T14:00:47.129789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:00:47.129789+0000","objects_scrubbed":8,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:15.616969+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":635,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.298680+0000","last_change":"2024-09-21T14:01:41.190537+0000","last_active":"2024-09-21T14:02:17.298680+0000","last_peered":"2024-09-21T14:02:17.298680+0000","last_clean":"2024-09-21T14:02:17.298680+0000","last_became_active":"2024-09-21T14:01:41.190210+0000","last_became_peered":"2024-09-21T14:01:41.190210+0000","last_unstale":"2024-09-21T14:02:17.298680+0000","last_undegraded":"2024-09-21T14:02:17.298680+0000","last_fullsized":"2024-09-21T14:02:17.298680+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:57.772975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:57.772975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"75'79","reported_seq":696,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.287708+0000","last_change":"2024-09-21T14:00:39.339360+0000","last_active":"2024-09-21T14:02:17.287708+0000","last_peered":"2024-09-21T14:02:17.287708+0000","last_clean":"2024-09-21T14:02:17.287708+0000","last_became_active":"2024-09-21T14:00:39.339077+0000","last_became_peered":"2024-09-21T14:00:39.339077+0000","last_unstale":"2024-09-21T14:02:17.287708+0000","last_undegraded":"2024-09-21T14:02:17.287708+0000","last_fullsized":"2024-09-21T14:02:17.287708+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:02:42.782456+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":19,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53669,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":610,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.287371+0000","last_change":"2024-09-21T14:01:41.182186+0000","last_active":"2024-09-21T14:02:17.287371+0000","last_peered":"2024-09-21T14:02:17.287371+0000","last_clean":"2024-09-21T14:02:17.287371+0000","last_became_active":"2024-09-21T14:01:41.181899+0000","last_became_peered":"2024-09-21T14:01:41.181899+0000","last_unstale":"2024-09-21T14:02:17.287371+0000","last_undegraded":"2024-09-21T14:02:17.287371+0000","last_fullsized":"2024-09-21T14:02:17.287371+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:15.700409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:15.700409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":535,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.291641+0000","last_change":"2024-09-21T14:01:41.196509+0000","last_active":"2024-09-21T14:02:17.291641+0000","last_peered":"2024-09-21T14:02:17.291641+0000","last_clean":"2024-09-21T14:02:17.291641+0000","last_became_active":"2024-09-21T14:01:41.196243+0000","last_became_peered":"2024-09-21T14:01:41.196243+0000","last_unstale":"2024-09-21T14:02:17.291641+0000","last_undegraded":"2024-09-21T14:02:17.291641+0000","last_fullsized":"2024-09-21T14:02:17.291641+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:21.153018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:21.153018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"368'13539","reported_seq":23439,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:21.145144+0000","last_change":"2024-09-21T14:00:42.107898+0000","last_active":"2024-09-21T14:02:21.145144+0000","last_peered":"2024-09-21T14:02:21.145144+0000","last_clean":"2024-09-21T14:02:21.145144+0000","last_became_active":"2024-09-21T14:00:42.085387+0000","last_became_peered":"2024-09-21T14:00:42.085387+0000","last_unstale":"2024-09-21T14:02:21.145144+0000","last_undegraded":"2024-09-21T14:02:21.145144+0000","last_fullsized":"2024-09-21T14:02:21.145144+0000","mapping_epoch":321,"log_start":"308'8912","ondisk_log_start":"308'8912","created":260,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4627,"log_dups_size":0,"ondisk_log_size":4627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106758144,"num_objects":1629,"num_object_clones":0,"num_object_copies":4887,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13539,"num_write_kb":485376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4827,"num_bytes_recovered":316342272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"290'19","reported_seq":713,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.293262+0000","last_change":"2024-09-21T13:58:17.811300+0000","last_active":"2024-09-21T14:02:17.293262+0000","last_peered":"2024-09-21T14:02:17.293262+0000","last_clean":"2024-09-21T14:02:17.293262+0000","last_became_active":"2024-09-21T13:58:17.776310+0000","last_became_peered":"2024-09-21T13:58:17.776310+0000","last_unstale":"2024-09-21T14:02:17.293262+0000","last_undegraded":"2024-09-21T14:02:17.293262+0000","last_fullsized":"2024-09-21T14:02:17.293262+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:54:16.451588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:16.451588+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:20.664330+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":684,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.291206+0000","last_change":"2024-09-21T14:02:03.437581+0000","last_active":"2024-09-21T14:02:17.291206+0000","last_peered":"2024-09-21T14:02:17.291206+0000","last_clean":"2024-09-21T14:02:17.291206+0000","last_became_active":"2024-09-21T13:59:32.223247+0000","last_became_peered":"2024-09-21T13:59:32.223247+0000","last_unstale":"2024-09-21T14:02:17.291206+0000","last_undegraded":"2024-09-21T14:02:17.291206+0000","last_fullsized":"2024-09-21T14:02:17.291206+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:03.437510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:02:03.437510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:14.620200+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":603,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.291472+0000","last_change":"2024-09-21T14:01:41.194538+0000","last_active":"2024-09-21T14:02:17.291472+0000","last_peered":"2024-09-21T14:02:17.291472+0000","last_clean":"2024-09-21T14:02:17.291472+0000","last_became_active":"2024-09-21T14:01:41.193941+0000","last_became_peered":"2024-09-21T14:01:41.193941+0000","last_unstale":"2024-09-21T14:02:17.291472+0000","last_undegraded":"2024-09-21T14:02:17.291472+0000","last_fullsized":"2024-09-21T14:02:17.291472+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:34.299722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:55:34.299722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":613,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.303678+0000","last_change":"2024-09-21T14:01:41.190501+0000","last_active":"2024-09-21T14:02:17.303678+0000","last_peered":"2024-09-21T14:02:17.303678+0000","last_clean":"2024-09-21T14:02:17.303678+0000","last_became_active":"2024-09-21T14:01:41.190192+0000","last_became_peered":"2024-09-21T14:01:41.190192+0000","last_unstale":"2024-09-21T14:02:17.303678+0000","last_undegraded":"2024-09-21T14:02:17.303678+0000","last_fullsized":"2024-09-21T14:02:17.303678+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:58.952218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:54:58.952218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"75'79","reported_seq":76,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.291018+0000","last_change":"2024-09-21T14:01:41.189146+0000","last_active":"2024-09-21T14:02:17.291018+0000","last_peered":"2024-09-21T14:02:17.291018+0000","last_clean":"2024-09-21T14:02:17.291018+0000","last_became_active":"2024-09-21T14:01:41.188685+0000","last_became_peered":"2024-09-21T14:01:41.188685+0000","last_unstale":"2024-09-21T14:02:17.291018+0000","last_undegraded":"2024-09-21T14:02:17.291018+0000","last_fullsized":"2024-09-21T14:02:17.291018+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:02:42.589731+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.0","version":"368'13348","reported_seq":17515,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:19.679855+0000","last_change":"2024-09-21T13:59:24.817101+0000","last_active":"2024-09-21T14:02:19.679855+0000","last_peered":"2024-09-21T14:02:19.679855+0000","last_clean":"2024-09-21T14:02:19.679855+0000","last_became_active":"2024-09-21T13:58:36.723110+0000","last_became_peered":"2024-09-21T13:58:36.723110+0000","last_unstale":"2024-09-21T14:02:19.679855+0000","last_undegraded":"2024-09-21T14:02:19.679855+0000","last_fullsized":"2024-09-21T14:02:19.679855+0000","mapping_epoch":279,"log_start":"307'8781","ondisk_log_start":"307'8781","created":260,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'7035","last_scrub_stamp":"2024-09-21T13:59:24.816903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:59:24.816903+0000","objects_scrubbed":6955,"log_size":4567,"log_dups_size":0,"ondisk_log_size":4567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2446,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112984064,"num_objects":1724,"num_object_clones":0,"num_object_copies":5172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13348,"num_write_kb":482304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1924,"num_bytes_recovered":126091264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"290'31","reported_seq":709,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.292887+0000","last_change":"2024-09-21T14:01:03.541547+0000","last_active":"2024-09-21T14:02:17.292887+0000","last_peered":"2024-09-21T14:02:17.292887+0000","last_clean":"2024-09-21T14:02:17.292887+0000","last_became_active":"2024-09-21T13:52:04.313218+0000","last_became_peered":"2024-09-21T13:52:04.313218+0000","last_unstale":"2024-09-21T14:02:17.292887+0000","last_undegraded":"2024-09-21T14:02:17.292887+0000","last_fullsized":"2024-09-21T14:02:17.292887+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'31","last_scrub_stamp":"2024-09-21T14:01:03.541432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:03.541432+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:21.057603+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":225,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.295129+0000","last_change":"2024-09-21T14:01:29.924408+0000","last_active":"2024-09-21T14:02:17.295129+0000","last_peered":"2024-09-21T14:02:17.295129+0000","last_clean":"2024-09-21T14:02:17.295129+0000","last_became_active":"2024-09-21T13:58:17.762520+0000","last_became_peered":"2024-09-21T13:58:17.762520+0000","last_unstale":"2024-09-21T14:02:17.295129+0000","last_undegraded":"2024-09-21T14:02:17.295129+0000","last_fullsized":"2024-09-21T14:02:17.295129+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:29.924331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:01:29.924331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:43.977176+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":638,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.292157+0000","last_change":"2024-09-21T13:58:52.030800+0000","last_active":"2024-09-21T14:02:17.292157+0000","last_peered":"2024-09-21T14:02:17.292157+0000","last_clean":"2024-09-21T14:02:17.292157+0000","last_became_active":"2024-09-21T13:49:02.067472+0000","last_became_peered":"2024-09-21T13:49:02.067472+0000","last_unstale":"2024-09-21T14:02:17.292157+0000","last_undegraded":"2024-09-21T14:02:17.292157+0000","last_fullsized":"2024-09-21T14:02:17.292157+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:52.030718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:52.030718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":521,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.307544+0000","last_change":"2024-09-21T13:57:12.683321+0000","last_active":"2024-09-21T14:02:17.307544+0000","last_peered":"2024-09-21T14:02:17.307544+0000","last_clean":"2024-09-21T14:02:17.307544+0000","last_became_active":"2024-09-21T13:57:12.682979+0000","last_became_peered":"2024-09-21T13:57:12.682979+0000","last_unstale":"2024-09-21T14:02:17.307544+0000","last_undegraded":"2024-09-21T14:02:17.307544+0000","last_fullsized":"2024-09-21T14:02:17.307544+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:18.027944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:18.027944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:20.920553+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"75'79","reported_seq":701,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.289696+0000","last_change":"2024-09-21T14:00:41.140108+0000","last_active":"2024-09-21T14:02:17.289696+0000","last_peered":"2024-09-21T14:02:17.289696+0000","last_clean":"2024-09-21T14:02:17.289696+0000","last_became_active":"2024-09-21T14:00:41.139315+0000","last_became_peered":"2024-09-21T14:00:41.139315+0000","last_unstale":"2024-09-21T14:02:17.289696+0000","last_undegraded":"2024-09-21T14:02:17.289696+0000","last_fullsized":"2024-09-21T14:02:17.289696+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1","version":"368'13436","reported_seq":25705,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:21.112867+0000","last_change":"2024-09-21T14:01:10.113546+0000","last_active":"2024-09-21T14:02:21.112867+0000","last_peered":"2024-09-21T14:02:21.112867+0000","last_clean":"2024-09-21T14:02:21.112867+0000","last_became_active":"2024-09-21T14:01:10.092192+0000","last_became_peered":"2024-09-21T14:01:10.092192+0000","last_unstale":"2024-09-21T14:02:21.112867+0000","last_undegraded":"2024-09-21T14:02:21.112867+0000","last_fullsized":"2024-09-21T14:02:21.112867+0000","mapping_epoch":333,"log_start":"309'9045","ondisk_log_start":"309'9045","created":260,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4391,"log_dups_size":0,"ondisk_log_size":4391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109576192,"num_objects":1672,"num_object_clones":0,"num_object_copies":5016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13436,"num_write_kb":483456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5999,"num_bytes_recovered":393150464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"290'20","reported_seq":728,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.293424+0000","last_change":"2024-09-21T13:58:17.789407+0000","last_active":"2024-09-21T14:02:17.293424+0000","last_peered":"2024-09-21T14:02:17.293424+0000","last_clean":"2024-09-21T14:02:17.293424+0000","last_became_active":"2024-09-21T13:58:17.765427+0000","last_became_peered":"2024-09-21T13:58:17.765427+0000","last_unstale":"2024-09-21T14:02:17.293424+0000","last_undegraded":"2024-09-21T14:02:17.293424+0000","last_fullsized":"2024-09-21T14:02:17.293424+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:58:13.464196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:13.464196+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:16.467667+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":570,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.299666+0000","last_change":"2024-09-21T13:57:36.707211+0000","last_active":"2024-09-21T14:02:17.299666+0000","last_peered":"2024-09-21T14:02:17.299666+0000","last_clean":"2024-09-21T14:02:17.299666+0000","last_became_active":"2024-09-21T13:57:12.704203+0000","last_became_peered":"2024-09-21T13:57:12.704203+0000","last_unstale":"2024-09-21T14:02:17.299666+0000","last_undegraded":"2024-09-21T14:02:17.299666+0000","last_fullsized":"2024-09-21T14:02:17.299666+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:36.707148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:36.707148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"346'14","reported_seq":695,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.306363+0000","last_change":"2024-09-21T14:01:41.186960+0000","last_active":"2024-09-21T14:02:17.306363+0000","last_peered":"2024-09-21T14:02:17.306363+0000","last_clean":"2024-09-21T14:02:17.306363+0000","last_became_active":"2024-09-21T14:01:41.186564+0000","last_became_peered":"2024-09-21T14:01:41.186564+0000","last_unstale":"2024-09-21T14:02:17.306363+0000","last_undegraded":"2024-09-21T14:02:17.306363+0000","last_fullsized":"2024-09-21T14:02:17.306363+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'12","last_scrub_stamp":"2024-09-21T13:59:38.738120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:38.738120+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:44.832105+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"68'1","reported_seq":656,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.293145+0000","last_change":"2024-09-21T13:59:31.392186+0000","last_active":"2024-09-21T14:02:17.293145+0000","last_peered":"2024-09-21T14:02:17.293145+0000","last_clean":"2024-09-21T14:02:17.293145+0000","last_became_active":"2024-09-21T13:52:04.313210+0000","last_became_peered":"2024-09-21T13:52:04.313210+0000","last_unstale":"2024-09-21T14:02:17.293145+0000","last_undegraded":"2024-09-21T14:02:17.293145+0000","last_fullsized":"2024-09-21T14:02:17.293145+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T13:59:31.392105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:31.392105+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"368'13342","reported_seq":13557,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:20.785221+0000","last_change":"2024-09-21T14:01:41.341139+0000","last_active":"2024-09-21T14:02:20.785221+0000","last_peered":"2024-09-21T14:02:20.785221+0000","last_clean":"2024-09-21T14:02:20.785221+0000","last_became_active":"2024-09-21T14:01:41.199821+0000","last_became_peered":"2024-09-21T14:01:41.199821+0000","last_unstale":"2024-09-21T14:02:20.785221+0000","last_undegraded":"2024-09-21T14:02:20.785221+0000","last_fullsized":"2024-09-21T14:02:20.785221+0000","mapping_epoch":345,"log_start":"305'8399","ondisk_log_start":"305'8399","created":260,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4943,"log_dups_size":0,"ondisk_log_size":4943,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101449728,"num_objects":1548,"num_object_clones":0,"num_object_copies":4644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13342,"num_write_kb":476480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"290'15","reported_seq":683,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.292828+0000","last_change":"2024-09-21T13:58:50.498597+0000","last_active":"2024-09-21T14:02:17.292828+0000","last_peered":"2024-09-21T14:02:17.292828+0000","last_clean":"2024-09-21T14:02:17.292828+0000","last_became_active":"2024-09-21T13:49:00.053016+0000","last_became_peered":"2024-09-21T13:49:00.053016+0000","last_unstale":"2024-09-21T14:02:17.292828+0000","last_undegraded":"2024-09-21T14:02:17.292828+0000","last_fullsized":"2024-09-21T14:02:17.292828+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:58:50.498369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:50.498369+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":615,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.291430+0000","last_change":"2024-09-21T14:01:41.192717+0000","last_active":"2024-09-21T14:02:17.291430+0000","last_peered":"2024-09-21T14:02:17.291430+0000","last_clean":"2024-09-21T14:02:17.291430+0000","last_became_active":"2024-09-21T14:01:41.192443+0000","last_became_peered":"2024-09-21T14:01:41.192443+0000","last_unstale":"2024-09-21T14:02:17.291430+0000","last_undegraded":"2024-09-21T14:02:17.291430+0000","last_fullsized":"2024-09-21T14:02:17.291430+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:53.894506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:53.894506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":595,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.291496+0000","last_change":"2024-09-21T14:01:27.739999+0000","last_active":"2024-09-21T14:02:17.291496+0000","last_peered":"2024-09-21T14:02:17.291496+0000","last_clean":"2024-09-21T14:02:17.291496+0000","last_became_active":"2024-09-21T13:57:12.680978+0000","last_became_peered":"2024-09-21T13:57:12.680978+0000","last_unstale":"2024-09-21T14:02:17.291496+0000","last_undegraded":"2024-09-21T14:02:17.291496+0000","last_fullsized":"2024-09-21T14:02:17.291496+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:27.739921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:01:27.739921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:50.205235+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":545,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.287881+0000","last_change":"2024-09-21T13:59:32.224001+0000","last_active":"2024-09-21T14:02:17.287881+0000","last_peered":"2024-09-21T14:02:17.287881+0000","last_clean":"2024-09-21T14:02:17.287881+0000","last_became_active":"2024-09-21T13:59:32.221628+0000","last_became_peered":"2024-09-21T13:59:32.221628+0000","last_unstale":"2024-09-21T14:02:17.287881+0000","last_undegraded":"2024-09-21T14:02:17.287881+0000","last_fullsized":"2024-09-21T14:02:17.287881+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:13.698091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:13.698091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"368'13900","reported_seq":22194,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:20.799495+0000","last_change":"2024-09-21T14:02:09.252872+0000","last_active":"2024-09-21T14:02:20.799495+0000","last_peered":"2024-09-21T14:02:20.799495+0000","last_clean":"2024-09-21T14:02:20.799495+0000","last_became_active":"2024-09-21T14:02:09.218449+0000","last_became_peered":"2024-09-21T14:02:09.218449+0000","last_unstale":"2024-09-21T14:02:20.799495+0000","last_undegraded":"2024-09-21T14:02:20.799495+0000","last_fullsized":"2024-09-21T14:02:20.799495+0000","mapping_epoch":362,"log_start":"307'8955","ondisk_log_start":"307'8955","created":260,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4945,"log_dups_size":0,"ondisk_log_size":4945,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112066560,"num_objects":1710,"num_object_clones":0,"num_object_copies":5130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13900,"num_write_kb":499520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4082,"num_bytes_recovered":226033664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"290'35","reported_seq":291,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.295266+0000","last_change":"2024-09-21T14:01:02.006619+0000","last_active":"2024-09-21T14:02:17.295266+0000","last_peered":"2024-09-21T14:02:17.295266+0000","last_clean":"2024-09-21T14:02:17.295266+0000","last_became_active":"2024-09-21T13:59:32.204184+0000","last_became_peered":"2024-09-21T13:59:32.204184+0000","last_unstale":"2024-09-21T14:02:17.295266+0000","last_undegraded":"2024-09-21T14:02:17.295266+0000","last_fullsized":"2024-09-21T14:02:17.295266+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'35","last_scrub_stamp":"2024-09-21T14:01:02.006504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:02.006504+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":637,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.292424+0000","last_change":"2024-09-21T14:01:41.184937+0000","last_active":"2024-09-21T14:02:17.292424+0000","last_peered":"2024-09-21T14:02:17.292424+0000","last_clean":"2024-09-21T14:02:17.292424+0000","last_became_active":"2024-09-21T14:01:41.184567+0000","last_became_peered":"2024-09-21T14:01:41.184567+0000","last_unstale":"2024-09-21T14:02:17.292424+0000","last_undegraded":"2024-09-21T14:02:17.292424+0000","last_fullsized":"2024-09-21T14:02:17.292424+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:21.587493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:21.587493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":615,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.302901+0000","last_change":"2024-09-21T14:01:41.182250+0000","last_active":"2024-09-21T14:02:17.302901+0000","last_peered":"2024-09-21T14:02:17.302901+0000","last_clean":"2024-09-21T14:02:17.302901+0000","last_became_active":"2024-09-21T14:01:41.181911+0000","last_became_peered":"2024-09-21T14:01:41.181911+0000","last_unstale":"2024-09-21T14:02:17.302901+0000","last_undegraded":"2024-09-21T14:02:17.302901+0000","last_fullsized":"2024-09-21T14:02:17.302901+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:55.588554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:55.588554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:41.898392+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":564,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.287615+0000","last_change":"2024-09-21T13:59:32.223123+0000","last_active":"2024-09-21T14:02:17.287615+0000","last_peered":"2024-09-21T14:02:17.287615+0000","last_clean":"2024-09-21T14:02:17.287615+0000","last_became_active":"2024-09-21T13:59:32.219332+0000","last_became_peered":"2024-09-21T13:59:32.219332+0000","last_unstale":"2024-09-21T14:02:17.287615+0000","last_undegraded":"2024-09-21T14:02:17.287615+0000","last_fullsized":"2024-09-21T14:02:17.287615+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:19.537177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:19.537177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"368'13179","reported_seq":13408,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:19.681935+0000","last_change":"2024-09-21T13:57:53.515705+0000","last_active":"2024-09-21T14:02:19.681935+0000","last_peered":"2024-09-21T14:02:19.681935+0000","last_clean":"2024-09-21T14:02:19.681935+0000","last_became_active":"2024-09-21T13:57:53.515313+0000","last_became_peered":"2024-09-21T13:57:53.515313+0000","last_unstale":"2024-09-21T14:02:19.681935+0000","last_undegraded":"2024-09-21T14:02:19.681935+0000","last_fullsized":"2024-09-21T14:02:19.681935+0000","mapping_epoch":260,"log_start":"307'8579","ondisk_log_start":"307'8579","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4600,"log_dups_size":0,"ondisk_log_size":4600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110690304,"num_objects":1689,"num_object_clones":0,"num_object_copies":5067,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13179,"num_write_kb":475776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"290'18","reported_seq":701,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.291067+0000","last_change":"2024-09-21T13:59:28.340494+0000","last_active":"2024-09-21T14:02:17.291067+0000","last_peered":"2024-09-21T14:02:17.291067+0000","last_clean":"2024-09-21T14:02:17.291067+0000","last_became_active":"2024-09-21T13:57:12.680627+0000","last_became_peered":"2024-09-21T13:57:12.680627+0000","last_unstale":"2024-09-21T14:02:17.291067+0000","last_undegraded":"2024-09-21T14:02:17.291067+0000","last_fullsized":"2024-09-21T14:02:17.291067+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'18","last_scrub_stamp":"2024-09-21T13:59:28.340415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:28.340415+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:11.383693+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":592,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.289689+0000","last_change":"2024-09-21T13:59:32.201126+0000","last_active":"2024-09-21T14:02:17.289689+0000","last_peered":"2024-09-21T14:02:17.289689+0000","last_clean":"2024-09-21T14:02:17.289689+0000","last_became_active":"2024-09-21T13:59:32.200674+0000","last_became_peered":"2024-09-21T13:59:32.200674+0000","last_unstale":"2024-09-21T14:02:17.289689+0000","last_undegraded":"2024-09-21T14:02:17.289689+0000","last_fullsized":"2024-09-21T14:02:17.289689+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:21.582325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:21.582325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":569,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.302708+0000","last_change":"2024-09-21T13:58:11.377940+0000","last_active":"2024-09-21T14:02:17.302708+0000","last_peered":"2024-09-21T14:02:17.302708+0000","last_clean":"2024-09-21T14:02:17.302708+0000","last_became_active":"2024-09-21T13:52:04.322147+0000","last_became_peered":"2024-09-21T13:52:04.322147+0000","last_unstale":"2024-09-21T14:02:17.302708+0000","last_undegraded":"2024-09-21T14:02:17.302708+0000","last_fullsized":"2024-09-21T14:02:17.302708+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:11.377803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:11.377803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:25.932593+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":578,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.299959+0000","last_change":"2024-09-21T13:59:32.207006+0000","last_active":"2024-09-21T14:02:17.299959+0000","last_peered":"2024-09-21T14:02:17.299959+0000","last_clean":"2024-09-21T14:02:17.299959+0000","last_became_active":"2024-09-21T13:59:32.203767+0000","last_became_peered":"2024-09-21T13:59:32.203767+0000","last_unstale":"2024-09-21T14:02:17.299959+0000","last_undegraded":"2024-09-21T14:02:17.299959+0000","last_fullsized":"2024-09-21T14:02:17.299959+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:09.040062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:55:09.040062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"368'13939","reported_seq":17040,"reported_epoch":368,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-21T14:02:20.786653+0000","last_change":"2024-09-21T14:01:45.658247+0000","last_active":"2024-09-21T14:02:20.786653+0000","last_peered":"2024-09-21T14:02:20.786653+0000","last_clean":"2024-09-21T13:58:23.785212+0000","last_became_active":"2024-09-21T14:01:44.478305+0000","last_became_peered":"2024-09-21T14:01:44.478305+0000","last_unstale":"2024-09-21T14:02:20.786653+0000","last_undegraded":"2024-09-21T14:01:44.440463+0000","last_fullsized":"2024-09-21T14:01:42.170394+0000","mapping_epoch":346,"log_start":"307'8986","ondisk_log_start":"307'8986","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4953,"log_dups_size":0,"ondisk_log_size":4953,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110821376,"num_objects":1691,"num_object_clones":0,"num_object_copies":5073,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1532,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13939,"num_write_kb":500160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1500,"num_bytes_recovered":96862208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,7],"avail_no_missing":["5","0"],"object_location_counts":[{"shards":"0,5","objects":1532},{"shards":"5,7","objects":159}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"290'27","reported_seq":691,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.291165+0000","last_change":"2024-09-21T14:01:41.193143+0000","last_active":"2024-09-21T14:02:17.291165+0000","last_peered":"2024-09-21T14:02:17.291165+0000","last_clean":"2024-09-21T14:02:17.291165+0000","last_became_active":"2024-09-21T14:01:41.192481+0000","last_became_peered":"2024-09-21T14:01:41.192481+0000","last_unstale":"2024-09-21T14:02:17.291165+0000","last_undegraded":"2024-09-21T14:02:17.291165+0000","last_fullsized":"2024-09-21T14:02:17.291165+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'27","last_scrub_stamp":"2024-09-21T13:59:33.058859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:33.058859+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":564,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.288138+0000","last_change":"2024-09-21T13:59:32.203889+0000","last_active":"2024-09-21T14:02:17.288138+0000","last_peered":"2024-09-21T14:02:17.288138+0000","last_clean":"2024-09-21T14:02:17.288138+0000","last_became_active":"2024-09-21T13:59:32.203172+0000","last_became_peered":"2024-09-21T13:59:32.203172+0000","last_unstale":"2024-09-21T14:02:17.288138+0000","last_undegraded":"2024-09-21T14:02:17.288138+0000","last_fullsized":"2024-09-21T14:02:17.288138+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T13:55:03.879706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:55:03.879706+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"300'14","reported_seq":772,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.287506+0000","last_change":"2024-09-21T13:59:32.220898+0000","last_active":"2024-09-21T14:02:17.287506+0000","last_peered":"2024-09-21T14:02:17.287506+0000","last_clean":"2024-09-21T14:02:17.287506+0000","last_became_active":"2024-09-21T13:59:32.206956+0000","last_became_peered":"2024-09-21T13:59:32.206956+0000","last_unstale":"2024-09-21T14:02:17.287506+0000","last_undegraded":"2024-09-21T14:02:17.287506+0000","last_fullsized":"2024-09-21T14:02:17.287506+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'11","last_scrub_stamp":"2024-09-21T13:56:31.616003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:56:31.616003+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":639,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.292775+0000","last_change":"2024-09-21T14:01:41.248408+0000","last_active":"2024-09-21T14:02:17.292775+0000","last_peered":"2024-09-21T14:02:17.292775+0000","last_clean":"2024-09-21T14:02:17.292775+0000","last_became_active":"2024-09-21T14:01:41.246106+0000","last_became_peered":"2024-09-21T14:01:41.246106+0000","last_unstale":"2024-09-21T14:02:17.292775+0000","last_undegraded":"2024-09-21T14:02:17.292775+0000","last_fullsized":"2024-09-21T14:02:17.292775+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:39.258659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:39.258659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":228,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.295614+0000","last_change":"2024-09-21T14:01:50.733055+0000","last_active":"2024-09-21T14:02:17.295614+0000","last_peered":"2024-09-21T14:02:17.295614+0000","last_clean":"2024-09-21T14:02:17.295614+0000","last_became_active":"2024-09-21T14:01:41.197501+0000","last_became_peered":"2024-09-21T14:01:41.197501+0000","last_unstale":"2024-09-21T14:02:17.295614+0000","last_undegraded":"2024-09-21T14:02:17.295614+0000","last_fullsized":"2024-09-21T14:02:17.295614+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:50.733007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:01:50.733007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:08.367766+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"290'12","reported_seq":613,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.301299+0000","last_change":"2024-09-21T13:59:32.247422+0000","last_active":"2024-09-21T14:02:17.301299+0000","last_peered":"2024-09-21T14:02:17.301299+0000","last_clean":"2024-09-21T14:02:17.301299+0000","last_became_active":"2024-09-21T13:59:32.211306+0000","last_became_peered":"2024-09-21T13:59:32.211306+0000","last_unstale":"2024-09-21T14:02:17.301299+0000","last_undegraded":"2024-09-21T14:02:17.301299+0000","last_fullsized":"2024-09-21T14:02:17.301299+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'6","last_scrub_stamp":"2024-09-21T13:55:12.066873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:55:12.066873+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":571,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.291281+0000","last_change":"2024-09-21T14:01:41.199742+0000","last_active":"2024-09-21T14:02:17.291281+0000","last_peered":"2024-09-21T14:02:17.291281+0000","last_clean":"2024-09-21T14:02:17.291281+0000","last_became_active":"2024-09-21T14:01:41.199321+0000","last_became_peered":"2024-09-21T14:01:41.199321+0000","last_unstale":"2024-09-21T14:02:17.291281+0000","last_undegraded":"2024-09-21T14:02:17.291281+0000","last_fullsized":"2024-09-21T14:02:17.291281+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:37.106238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:37.106238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":610,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.291444+0000","last_change":"2024-09-21T13:59:32.222277+0000","last_active":"2024-09-21T14:02:17.291444+0000","last_peered":"2024-09-21T14:02:17.291444+0000","last_clean":"2024-09-21T14:02:17.291444+0000","last_became_active":"2024-09-21T13:59:32.219525+0000","last_became_peered":"2024-09-21T13:59:32.219525+0000","last_unstale":"2024-09-21T14:02:17.291444+0000","last_undegraded":"2024-09-21T14:02:17.291444+0000","last_fullsized":"2024-09-21T14:02:17.291444+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:02.838181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:55:02.838181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"290'15","reported_seq":576,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.301689+0000","last_change":"2024-09-21T13:59:32.238689+0000","last_active":"2024-09-21T14:02:17.301689+0000","last_peered":"2024-09-21T14:02:17.301689+0000","last_clean":"2024-09-21T14:02:17.301689+0000","last_became_active":"2024-09-21T13:59:32.209166+0000","last_became_peered":"2024-09-21T13:59:32.209166+0000","last_unstale":"2024-09-21T14:02:17.301689+0000","last_undegraded":"2024-09-21T14:02:17.301689+0000","last_fullsized":"2024-09-21T14:02:17.301689+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:55:04.990842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:55:04.990842+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"61'1","reported_seq":628,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.289616+0000","last_change":"2024-09-21T13:59:32.222826+0000","last_active":"2024-09-21T14:02:17.289616+0000","last_peered":"2024-09-21T14:02:17.289616+0000","last_clean":"2024-09-21T14:02:17.289616+0000","last_became_active":"2024-09-21T13:59:32.219515+0000","last_became_peered":"2024-09-21T13:59:32.219515+0000","last_unstale":"2024-09-21T14:02:17.289616+0000","last_undegraded":"2024-09-21T14:02:17.289616+0000","last_fullsized":"2024-09-21T14:02:17.289616+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:58:07.711028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:07.711028+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":561,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.298333+0000","last_change":"2024-09-21T13:59:32.211754+0000","last_active":"2024-09-21T14:02:17.298333+0000","last_peered":"2024-09-21T14:02:17.298333+0000","last_clean":"2024-09-21T14:02:17.298333+0000","last_became_active":"2024-09-21T13:59:32.210586+0000","last_became_peered":"2024-09-21T13:59:32.210586+0000","last_unstale":"2024-09-21T14:02:17.298333+0000","last_undegraded":"2024-09-21T14:02:17.298333+0000","last_fullsized":"2024-09-21T14:02:17.298333+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:28.014760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:28.014760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:17.280712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":611,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.302445+0000","last_change":"2024-09-21T14:01:26.918280+0000","last_active":"2024-09-21T14:02:17.302445+0000","last_peered":"2024-09-21T14:02:17.302445+0000","last_clean":"2024-09-21T14:02:17.302445+0000","last_became_active":"2024-09-21T13:59:32.205803+0000","last_became_peered":"2024-09-21T13:59:32.205803+0000","last_unstale":"2024-09-21T14:02:17.302445+0000","last_undegraded":"2024-09-21T14:02:17.302445+0000","last_fullsized":"2024-09-21T14:02:17.302445+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:26.918225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:01:26.918225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:27.514271+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"290'15","reported_seq":616,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.295116+0000","last_change":"2024-09-21T14:01:41.189676+0000","last_active":"2024-09-21T14:02:17.295116+0000","last_peered":"2024-09-21T14:02:17.295116+0000","last_clean":"2024-09-21T14:02:17.295116+0000","last_became_active":"2024-09-21T14:01:41.187904+0000","last_became_peered":"2024-09-21T14:01:41.187904+0000","last_unstale":"2024-09-21T14:02:17.295116+0000","last_undegraded":"2024-09-21T14:02:17.295116+0000","last_fullsized":"2024-09-21T14:02:17.295116+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:57:24.124604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:57:24.124604+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":497,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.287744+0000","last_change":"2024-09-21T13:59:32.212325+0000","last_active":"2024-09-21T14:02:17.287744+0000","last_peered":"2024-09-21T14:02:17.287744+0000","last_clean":"2024-09-21T14:02:17.287744+0000","last_became_active":"2024-09-21T13:59:32.208749+0000","last_became_peered":"2024-09-21T13:59:32.208749+0000","last_unstale":"2024-09-21T14:02:17.287744+0000","last_undegraded":"2024-09-21T14:02:17.287744+0000","last_fullsized":"2024-09-21T14:02:17.287744+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:20.574745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:20.574745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"300'14","reported_seq":705,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.303147+0000","last_change":"2024-09-21T13:59:32.208253+0000","last_active":"2024-09-21T14:02:17.303147+0000","last_peered":"2024-09-21T14:02:17.303147+0000","last_clean":"2024-09-21T14:02:17.303147+0000","last_became_active":"2024-09-21T13:59:32.206926+0000","last_became_peered":"2024-09-21T13:59:32.206926+0000","last_unstale":"2024-09-21T14:02:17.303147+0000","last_undegraded":"2024-09-21T14:02:17.303147+0000","last_fullsized":"2024-09-21T14:02:17.303147+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'10","last_scrub_stamp":"2024-09-21T13:54:20.946782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:20.946782+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:10.314285+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":663,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.292647+0000","last_change":"2024-09-21T14:01:41.200564+0000","last_active":"2024-09-21T14:02:17.292647+0000","last_peered":"2024-09-21T14:02:17.292647+0000","last_clean":"2024-09-21T14:02:17.292647+0000","last_became_active":"2024-09-21T14:01:41.200113+0000","last_became_peered":"2024-09-21T14:01:41.200113+0000","last_unstale":"2024-09-21T14:02:17.292647+0000","last_undegraded":"2024-09-21T14:02:17.292647+0000","last_fullsized":"2024-09-21T14:02:17.292647+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:54.805619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:54.805619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:48.557673+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"290'20","reported_seq":612,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.302399+0000","last_change":"2024-09-21T13:59:32.236530+0000","last_active":"2024-09-21T14:02:17.302399+0000","last_peered":"2024-09-21T14:02:17.302399+0000","last_clean":"2024-09-21T14:02:17.302399+0000","last_became_active":"2024-09-21T13:59:32.208332+0000","last_became_peered":"2024-09-21T13:59:32.208332+0000","last_unstale":"2024-09-21T14:02:17.302399+0000","last_undegraded":"2024-09-21T14:02:17.302399+0000","last_fullsized":"2024-09-21T14:02:17.302399+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:57:37.671391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:57:37.671391+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":604,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.288395+0000","last_change":"2024-09-21T13:59:32.212737+0000","last_active":"2024-09-21T14:02:17.288395+0000","last_peered":"2024-09-21T14:02:17.288395+0000","last_clean":"2024-09-21T14:02:17.288395+0000","last_became_active":"2024-09-21T13:59:32.209804+0000","last_became_peered":"2024-09-21T13:59:32.209804+0000","last_unstale":"2024-09-21T14:02:17.288395+0000","last_undegraded":"2024-09-21T14:02:17.288395+0000","last_fullsized":"2024-09-21T14:02:17.288395+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:25.571256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:25.571256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"346'10","reported_seq":705,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.295046+0000","last_change":"2024-09-21T14:01:41.207030+0000","last_active":"2024-09-21T14:02:17.295046+0000","last_peered":"2024-09-21T14:02:17.295046+0000","last_clean":"2024-09-21T14:02:17.295046+0000","last_became_active":"2024-09-21T14:01:41.206303+0000","last_became_peered":"2024-09-21T14:01:41.206303+0000","last_unstale":"2024-09-21T14:02:17.295046+0000","last_undegraded":"2024-09-21T14:02:17.295046+0000","last_fullsized":"2024-09-21T14:02:17.295046+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:59:32.104224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:32.104224+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":501,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.289289+0000","last_change":"2024-09-21T13:59:32.213015+0000","last_active":"2024-09-21T14:02:17.289289+0000","last_peered":"2024-09-21T14:02:17.289289+0000","last_clean":"2024-09-21T14:02:17.289289+0000","last_became_active":"2024-09-21T13:59:32.211472+0000","last_became_peered":"2024-09-21T13:59:32.211472+0000","last_unstale":"2024-09-21T14:02:17.289289+0000","last_undegraded":"2024-09-21T14:02:17.289289+0000","last_fullsized":"2024-09-21T14:02:17.289289+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:22.647191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:22.647191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"290'19","reported_seq":718,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.292175+0000","last_change":"2024-09-21T13:58:17.790158+0000","last_active":"2024-09-21T14:02:17.292175+0000","last_peered":"2024-09-21T14:02:17.292175+0000","last_clean":"2024-09-21T14:02:17.292175+0000","last_became_active":"2024-09-21T13:58:17.771497+0000","last_became_peered":"2024-09-21T13:58:17.771497+0000","last_unstale":"2024-09-21T14:02:17.292175+0000","last_undegraded":"2024-09-21T14:02:17.292175+0000","last_fullsized":"2024-09-21T14:02:17.292175+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:54:56.539374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:56.539374+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:22.599916+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":238,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.287524+0000","last_change":"2024-09-21T13:59:32.202097+0000","last_active":"2024-09-21T14:02:17.287524+0000","last_peered":"2024-09-21T14:02:17.287524+0000","last_clean":"2024-09-21T14:02:17.287524+0000","last_became_active":"2024-09-21T13:59:32.201568+0000","last_became_peered":"2024-09-21T13:59:32.201568+0000","last_unstale":"2024-09-21T14:02:17.287524+0000","last_undegraded":"2024-09-21T14:02:17.287524+0000","last_fullsized":"2024-09-21T14:02:17.287524+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:01.228213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:01.228213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":676,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.295503+0000","last_change":"2024-09-21T14:01:41.185588+0000","last_active":"2024-09-21T14:02:17.295503+0000","last_peered":"2024-09-21T14:02:17.295503+0000","last_clean":"2024-09-21T14:02:17.295503+0000","last_became_active":"2024-09-21T14:01:41.185293+0000","last_became_peered":"2024-09-21T14:01:41.185293+0000","last_unstale":"2024-09-21T14:02:17.295503+0000","last_undegraded":"2024-09-21T14:02:17.295503+0000","last_fullsized":"2024-09-21T14:02:17.295503+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:35.409237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:35.409237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:10.948556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":569,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.296818+0000","last_change":"2024-09-21T14:01:41.200439+0000","last_active":"2024-09-21T14:02:17.296818+0000","last_peered":"2024-09-21T14:02:17.296818+0000","last_clean":"2024-09-21T14:02:17.296818+0000","last_became_active":"2024-09-21T14:01:41.199671+0000","last_became_peered":"2024-09-21T14:01:41.199671+0000","last_unstale":"2024-09-21T14:02:17.296818+0000","last_undegraded":"2024-09-21T14:02:17.296818+0000","last_fullsized":"2024-09-21T14:02:17.296818+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:39.643458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:39.643458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"290'15","reported_seq":720,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.295654+0000","last_change":"2024-09-21T13:58:39.977717+0000","last_active":"2024-09-21T14:02:17.295654+0000","last_peered":"2024-09-21T14:02:17.295654+0000","last_clean":"2024-09-21T14:02:17.295654+0000","last_became_active":"2024-09-21T13:58:17.764120+0000","last_became_peered":"2024-09-21T13:58:17.764120+0000","last_unstale":"2024-09-21T14:02:17.295654+0000","last_undegraded":"2024-09-21T14:02:17.295654+0000","last_fullsized":"2024-09-21T14:02:17.295654+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:58:39.977466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:39.977466+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":608,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.291317+0000","last_change":"2024-09-21T13:59:32.223082+0000","last_active":"2024-09-21T14:02:17.291317+0000","last_peered":"2024-09-21T14:02:17.291317+0000","last_clean":"2024-09-21T14:02:17.291317+0000","last_became_active":"2024-09-21T13:59:32.221854+0000","last_became_peered":"2024-09-21T13:59:32.221854+0000","last_unstale":"2024-09-21T14:02:17.291317+0000","last_undegraded":"2024-09-21T14:02:17.291317+0000","last_fullsized":"2024-09-21T14:02:17.291317+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:17.691193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:17.691193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":544,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.299587+0000","last_change":"2024-09-21T13:57:12.703746+0000","last_active":"2024-09-21T14:02:17.299587+0000","last_peered":"2024-09-21T14:02:17.299587+0000","last_clean":"2024-09-21T14:02:17.299587+0000","last_became_active":"2024-09-21T13:57:12.703441+0000","last_became_peered":"2024-09-21T13:57:12.703441+0000","last_unstale":"2024-09-21T14:02:17.299587+0000","last_undegraded":"2024-09-21T14:02:17.299587+0000","last_fullsized":"2024-09-21T14:02:17.299587+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:19.999125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:19.999125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:13.302246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":466,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.290915+0000","last_change":"2024-09-21T14:01:41.188075+0000","last_active":"2024-09-21T14:02:17.290915+0000","last_peered":"2024-09-21T14:02:17.290915+0000","last_clean":"2024-09-21T14:02:17.290915+0000","last_became_active":"2024-09-21T14:01:41.186934+0000","last_became_peered":"2024-09-21T14:01:41.186934+0000","last_unstale":"2024-09-21T14:02:17.290915+0000","last_undegraded":"2024-09-21T14:02:17.290915+0000","last_fullsized":"2024-09-21T14:02:17.290915+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:47.334504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:47.334504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"290'19","reported_seq":688,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.298080+0000","last_change":"2024-09-21T13:59:32.277742+0000","last_active":"2024-09-21T14:02:17.298080+0000","last_peered":"2024-09-21T14:02:17.298080+0000","last_clean":"2024-09-21T14:02:17.298080+0000","last_became_active":"2024-09-21T13:59:32.231567+0000","last_became_peered":"2024-09-21T13:59:32.231567+0000","last_unstale":"2024-09-21T14:02:17.298080+0000","last_undegraded":"2024-09-21T14:02:17.298080+0000","last_fullsized":"2024-09-21T14:02:17.298080+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:54:22.978267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:22.978267+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:12.294653+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":568,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.291366+0000","last_change":"2024-09-21T14:01:41.191090+0000","last_active":"2024-09-21T14:02:17.291366+0000","last_peered":"2024-09-21T14:02:17.291366+0000","last_clean":"2024-09-21T14:02:17.291366+0000","last_became_active":"2024-09-21T14:01:41.190760+0000","last_became_peered":"2024-09-21T14:01:41.190760+0000","last_unstale":"2024-09-21T14:02:17.291366+0000","last_undegraded":"2024-09-21T14:02:17.291366+0000","last_fullsized":"2024-09-21T14:02:17.291366+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:17.305144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:17.305144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":614,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.287805+0000","last_change":"2024-09-21T13:59:32.219508+0000","last_active":"2024-09-21T14:02:17.287805+0000","last_peered":"2024-09-21T14:02:17.287805+0000","last_clean":"2024-09-21T14:02:17.287805+0000","last_became_active":"2024-09-21T13:59:32.214004+0000","last_became_peered":"2024-09-21T13:59:32.214004+0000","last_unstale":"2024-09-21T14:02:17.287805+0000","last_undegraded":"2024-09-21T14:02:17.287805+0000","last_fullsized":"2024-09-21T14:02:17.287805+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:36.525417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:36.525417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":610,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.299740+0000","last_change":"2024-09-21T13:59:28.124373+0000","last_active":"2024-09-21T14:02:17.299740+0000","last_peered":"2024-09-21T14:02:17.299740+0000","last_clean":"2024-09-21T14:02:17.299740+0000","last_became_active":"2024-09-21T13:58:17.778115+0000","last_became_peered":"2024-09-21T13:58:17.778115+0000","last_unstale":"2024-09-21T14:02:17.299740+0000","last_undegraded":"2024-09-21T14:02:17.299740+0000","last_fullsized":"2024-09-21T14:02:17.299740+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:28.124296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:28.124296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"290'8","reported_seq":634,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.300865+0000","last_change":"2024-09-21T13:59:26.034457+0000","last_active":"2024-09-21T14:02:17.300865+0000","last_peered":"2024-09-21T14:02:17.300865+0000","last_clean":"2024-09-21T14:02:17.300865+0000","last_became_active":"2024-09-21T13:58:17.775079+0000","last_became_peered":"2024-09-21T13:58:17.775079+0000","last_unstale":"2024-09-21T14:02:17.300865+0000","last_undegraded":"2024-09-21T14:02:17.300865+0000","last_fullsized":"2024-09-21T14:02:17.300865+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'8","last_scrub_stamp":"2024-09-21T13:59:26.034372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:26.034372+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":536,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.291276+0000","last_change":"2024-09-21T14:01:41.196335+0000","last_active":"2024-09-21T14:02:17.291276+0000","last_peered":"2024-09-21T14:02:17.291276+0000","last_clean":"2024-09-21T14:02:17.291276+0000","last_became_active":"2024-09-21T14:01:41.195989+0000","last_became_peered":"2024-09-21T14:01:41.195989+0000","last_unstale":"2024-09-21T14:02:17.291276+0000","last_undegraded":"2024-09-21T14:02:17.291276+0000","last_fullsized":"2024-09-21T14:02:17.291276+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:16.457445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:16.457445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":228,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.295259+0000","last_change":"2024-09-21T14:00:57.128764+0000","last_active":"2024-09-21T14:02:17.295259+0000","last_peered":"2024-09-21T14:02:17.295259+0000","last_clean":"2024-09-21T14:02:17.295259+0000","last_became_active":"2024-09-21T13:59:32.204380+0000","last_became_peered":"2024-09-21T13:59:32.204380+0000","last_unstale":"2024-09-21T14:02:17.295259+0000","last_undegraded":"2024-09-21T14:02:17.295259+0000","last_fullsized":"2024-09-21T14:02:17.295259+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:00:57.128713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:00:57.128713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:23.643628+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":547,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.287662+0000","last_change":"2024-09-21T13:59:32.212615+0000","last_active":"2024-09-21T14:02:17.287662+0000","last_peered":"2024-09-21T14:02:17.287662+0000","last_clean":"2024-09-21T14:02:17.287662+0000","last_became_active":"2024-09-21T13:59:32.210134+0000","last_became_peered":"2024-09-21T13:59:32.210134+0000","last_unstale":"2024-09-21T14:02:17.287662+0000","last_undegraded":"2024-09-21T14:02:17.287662+0000","last_fullsized":"2024-09-21T14:02:17.287662+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:08.852789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:55:08.852789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":661,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.293355+0000","last_change":"2024-09-21T14:01:41.187792+0000","last_active":"2024-09-21T14:02:17.293355+0000","last_peered":"2024-09-21T14:02:17.293355+0000","last_clean":"2024-09-21T14:02:17.293355+0000","last_became_active":"2024-09-21T14:01:41.187438+0000","last_became_peered":"2024-09-21T14:01:41.187438+0000","last_unstale":"2024-09-21T14:02:17.293355+0000","last_undegraded":"2024-09-21T14:02:17.293355+0000","last_fullsized":"2024-09-21T14:02:17.293355+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:17.706325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:17.706325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":590,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.301061+0000","last_change":"2024-09-21T13:58:13.776632+0000","last_active":"2024-09-21T14:02:17.301061+0000","last_peered":"2024-09-21T14:02:17.301061+0000","last_clean":"2024-09-21T14:02:17.301061+0000","last_became_active":"2024-09-21T13:57:12.704042+0000","last_became_peered":"2024-09-21T13:57:12.704042+0000","last_unstale":"2024-09-21T14:02:17.301061+0000","last_undegraded":"2024-09-21T14:02:17.301061+0000","last_fullsized":"2024-09-21T14:02:17.301061+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:13.776572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:13.776572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"290'19","reported_seq":260,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.295863+0000","last_change":"2024-09-21T14:01:54.680822+0000","last_active":"2024-09-21T14:02:17.295863+0000","last_peered":"2024-09-21T14:02:17.295863+0000","last_clean":"2024-09-21T14:02:17.295863+0000","last_became_active":"2024-09-21T14:01:41.194606+0000","last_became_peered":"2024-09-21T14:01:41.194606+0000","last_unstale":"2024-09-21T14:02:17.295863+0000","last_undegraded":"2024-09-21T14:02:17.295863+0000","last_fullsized":"2024-09-21T14:02:17.295863+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'19","last_scrub_stamp":"2024-09-21T14:01:54.680735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:54.680735+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:21.312249+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":516,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.289216+0000","last_change":"2024-09-21T13:59:32.203045+0000","last_active":"2024-09-21T14:02:17.289216+0000","last_peered":"2024-09-21T14:02:17.289216+0000","last_clean":"2024-09-21T14:02:17.289216+0000","last_became_active":"2024-09-21T13:59:32.202225+0000","last_became_peered":"2024-09-21T13:59:32.202225+0000","last_unstale":"2024-09-21T14:02:17.289216+0000","last_undegraded":"2024-09-21T14:02:17.289216+0000","last_fullsized":"2024-09-21T14:02:17.289216+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:40.650832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:40.650832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":1749221400,"num_objects":26692,"num_object_clones":0,"num_object_copies":80076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2207,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26692,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":216120,"num_write_kb":7769921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40497,"num_bytes_recovered":2513960984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5299974144,"data_stored":5299961928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75891,"ondisk_log_size":75891,"up":48,"acting":46,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1111,"num_read_kb":868,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":1914,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":687,"ondisk_log_size":687,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":175,"num_read_kb":170,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":429358,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":159,"num_write":83,"num_write_kb":1184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1084718,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":298,"seq":1279900254243,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1161988,"kb_used_data":881716,"kb_used_omap":4,"kb_used_meta":280251,"kb_avail":92583164,"statfs":{"total":95995035648,"available":94805159936,"internally_reserved":0,"allocated":902877184,"data_stored":902064351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":286977704},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463082,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":692528,"kb_used_data":364252,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":93052624,"statfs":{"total":95995035648,"available":95285886976,"internally_reserved":0,"allocated":372994048,"data_stored":372204682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":336132550},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":345,"seq":1481763717129,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1054964,"kb_used_data":969768,"kb_used_omap":4,"kb_used_meta":85179,"kb_avail":92690188,"statfs":{"total":95995035648,"available":94914752512,"internally_reserved":0,"allocated":993042432,"data_stored":992242262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":87223971},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,0,2,4,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920951,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":817744,"kb_used_data":439484,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":92927408,"statfs":{"total":95995035648,"available":95157665792,"internally_reserved":0,"allocated":450031616,"data_stored":449210100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":387316165},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":240,"seq":1030792151103,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1110420,"kb_used_data":874052,"kb_used_omap":4,"kb_used_meta":236347,"kb_avail":92634732,"statfs":{"total":95995035648,"available":94857965568,"internally_reserved":0,"allocated":895029248,"data_stored":894200914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4443,"internal_metadata":242020005},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":130,"seq":558345748605,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":840368,"kb_used_data":298656,"kb_used_omap":4,"kb_used_meta":541691,"kb_avail":92904784,"statfs":{"total":95995035648,"available":95134498816,"internally_reserved":0,"allocated":305823744,"data_stored":305031544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4441,"internal_metadata":554692263},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575049,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1585072,"kb_used_data":1012448,"kb_used_omap":1,"kb_used_meta":572606,"kb_avail":92160080,"statfs":{"total":95995035648,"available":94371921920,"internally_reserved":0,"allocated":1036746752,"data_stored":1035889140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":586348997},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738575,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":954048,"kb_used_data":471664,"kb_used_omap":1,"kb_used_meta":482366,"kb_avail":92791104,"statfs":{"total":95995035648,"available":95018090496,"internally_reserved":0,"allocated":482983936,"data_stored":482174448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":493943237},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":466616320,"data_stored":466616320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1015549952,"data_stored":1015545880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":291438592,"data_stored":291438592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":857866240,"data_stored":857866240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":446955520,"data_stored":446955520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":974721024,"data_stored":974716952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":359137280,"data_stored":359137280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":887689216,"data_stored":887685144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T14:02:24.765 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T14:02:24.784 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:24.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:24.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:24.886 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:02:24.923 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T14:02:24.926 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:24.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:24.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:02:25.031 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:02:25.063 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:25.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:25.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:02:25.092 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-09-21T14:02:25.165 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:02:25.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:25.196 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:25.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:02:25.298 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:25.342 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:25.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:25.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:25.375 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T14:02:25.443 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:02:25.487 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:25.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:25.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:02:25.589 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:25.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:25.629 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:25.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:25.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:25 smithi187 ceph-mon[33032]: pgmap v686: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 156 active+clean; 1.6 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.08k op/s; 2207/79605 objects degraded (2.772%); 5.4 MiB/s, 87 objects/s recovering 2024-09-21T14:02:25.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:25 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3614471595' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:02:25.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:25 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:02:25.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:25 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 2207/79605 objects degraded (2.772%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T14:02:25.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:25 smithi187 ceph-mon[33032]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:02:25.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:25 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T14:02:25.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:25 smithi187 ceph-mon[33032]: osdmap e370: 8 total, 8 up, 8 in 2024-09-21T14:02:25.730 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:02:25.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:25 smithi049 ceph-mon[30080]: pgmap v686: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 156 active+clean; 1.6 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.08k op/s; 2207/79605 objects degraded (2.772%); 5.4 MiB/s, 87 objects/s recovering 2024-09-21T14:02:25.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:25 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3614471595' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:02:25.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:25 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:02:25.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:25 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 2207/79605 objects degraded (2.772%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T14:02:25.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:25 smithi049 ceph-mon[30080]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:02:25.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:25 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T14:02:25.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:25 smithi049 ceph-mon[30080]: osdmap e370: 8 total, 8 up, 8 in 2024-09-21T14:02:25.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:25 smithi049 ceph-mon[40489]: pgmap v686: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 156 active+clean; 1.6 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.08k op/s; 2207/79605 objects degraded (2.772%); 5.4 MiB/s, 87 objects/s recovering 2024-09-21T14:02:25.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:25 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3614471595' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:02:25.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:25 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:02:25.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:25 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 2207/79605 objects degraded (2.772%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T14:02:25.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:25 smithi049 ceph-mon[40489]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:02:25.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:25 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T14:02:25.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:25 smithi049 ceph-mon[40489]: osdmap e370: 8 total, 8 up, 8 in 2024-09-21T14:02:25.784 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:25.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:25.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:02:25.808 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:02:25.886 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:02:25.923 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:25.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:25.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:02:25.987 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:02:26.024 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:26.055 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:02:26.077 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:26.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:26.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:26.179 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:02:26.212 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:26.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:26.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:02:26.314 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:02:26.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:26.359 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:26.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:02:26.460 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:26.483 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:26.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:26.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:26.585 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:02:26.616 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:26.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:26.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:02:26.718 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:02:26.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:26.752 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:26.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:02:26.853 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:26.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:26.879 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:26.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:26.970 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:02:26.970 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:02:26.980 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:02:27.002 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:02:27.002 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:02:27.012 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:27.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:27.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:02:27.036 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:02:27.036 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:02:27.071 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:02:27.072 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:02:27.104 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:02:27.105 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:02:27.114 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:02:27.135 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:02:27.135 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:02:27.143 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:27.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:27.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:02:27.238 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:02:27.238 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:02:27.245 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:02:27.274 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:27.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:27.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:02:27.278 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:02:27.278 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:02:27.376 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:02:27.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:27.410 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:27.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:02:27.494 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T14:02:27.494 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:02:27.512 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:02:27.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:27.539 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:27.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:02:27.568 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T14:02:27.572 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-09-21T14:02:27.640 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:27.664 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:27.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:27.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:27.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:27 smithi187 ceph-mon[33032]: pgmap v688: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 156 active+clean; 1.4 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.08k op/s; 1443/71142 objects degraded (2.028%); 11 MiB/s, 169 objects/s recovering 2024-09-21T14:02:27.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:27 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1551251423' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:02:27.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:27 smithi049 ceph-mon[30080]: pgmap v688: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 156 active+clean; 1.4 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.08k op/s; 1443/71142 objects degraded (2.028%); 11 MiB/s, 169 objects/s recovering 2024-09-21T14:02:27.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:27 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1551251423' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:02:27.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:27 smithi049 ceph-mon[40489]: pgmap v688: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 156 active+clean; 1.4 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.08k op/s; 1443/71142 objects degraded (2.028%); 11 MiB/s, 169 objects/s recovering 2024-09-21T14:02:27.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:27 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1551251423' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:02:27.767 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:02:27.788 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:27.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:27.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:02:27.891 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:02:27.910 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T14:02:27.912 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:27.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:27.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:02:28.015 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:28.037 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:28.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:28.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:28.139 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:02:28.160 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:28.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:28.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:02:28.262 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:02:28.284 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:28.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:28.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:02:28.386 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:02:28.413 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:28.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:28.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:02:28.515 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:28.558 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:28.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:28.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:28.659 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:02:28.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:28 smithi187 ceph-mon[33032]: osdmap e371: 8 total, 8 up, 8 in 2024-09-21T14:02:28.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:28 smithi187 ceph-mon[33032]: from='client.15942 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:02:28.696 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:28.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:28.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:02:28.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:28 smithi049 ceph-mon[30080]: osdmap e371: 8 total, 8 up, 8 in 2024-09-21T14:02:28.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:28 smithi049 ceph-mon[30080]: from='client.15942 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:02:28.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:28 smithi049 ceph-mon[40489]: osdmap e371: 8 total, 8 up, 8 in 2024-09-21T14:02:28.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:28 smithi049 ceph-mon[40489]: from='client.15942 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:02:28.797 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:02:28.833 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:28.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:28.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:02:28.899 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":689,"stamp":"2024-09-21T14:02:27.391831+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1382065473,"num_objects":21282,"num_object_clones":0,"num_object_copies":63846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1443,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21282,"num_whiteouts":0,"num_read":1374,"num_read_kb":1203,"num_write":222534,"num_write_kb":7771150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41335,"num_bytes_recovered":2558891200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77571,"ondisk_log_size":77571,"up":474,"acting":472,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":475,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":7216684,"kb_used_data":4297556,"kb_used_omap":23,"kb_used_meta":2918824,"kb_avail":742744532,"statfs":{"total":767960285184,"available":760570400768,"internally_reserved":0,"allocated":4400697344,"data_stored":4394148775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24148,"internal_metadata":2988876204},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-633995264,"num_objects":-9674,"num_object_clones":0,"num_object_copies":-29022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1029,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9674,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":9674,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":934,"num_bytes_recovered":61210624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.411612"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":556,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.100236+0000","last_change":"2024-09-21T14:01:41.189741+0000","last_active":"2024-09-21T14:02:25.100236+0000","last_peered":"2024-09-21T14:02:25.100236+0000","last_clean":"2024-09-21T14:02:25.100236+0000","last_became_active":"2024-09-21T14:01:41.189398+0000","last_became_peered":"2024-09-21T14:01:41.189398+0000","last_unstale":"2024-09-21T14:02:25.100236+0000","last_undegraded":"2024-09-21T14:02:25.100236+0000","last_fullsized":"2024-09-21T14:02:25.100236+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:55.873301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:55.873301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"290'15","reported_seq":729,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:23.381826+0000","last_change":"2024-09-21T14:01:34.785322+0000","last_active":"2024-09-21T14:02:23.381826+0000","last_peered":"2024-09-21T14:02:23.381826+0000","last_clean":"2024-09-21T14:02:23.381826+0000","last_became_active":"2024-09-21T13:58:17.771220+0000","last_became_peered":"2024-09-21T13:58:17.771220+0000","last_unstale":"2024-09-21T14:02:23.381826+0000","last_undegraded":"2024-09-21T14:02:23.381826+0000","last_fullsized":"2024-09-21T14:02:23.381826+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'15","last_scrub_stamp":"2024-09-21T14:01:34.785267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:34.785267+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:36.742233+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":662,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:23.376545+0000","last_change":"2024-09-21T14:01:41.202503+0000","last_active":"2024-09-21T14:02:23.376545+0000","last_peered":"2024-09-21T14:02:23.376545+0000","last_clean":"2024-09-21T14:02:23.376545+0000","last_became_active":"2024-09-21T14:01:41.202168+0000","last_became_peered":"2024-09-21T14:01:41.202168+0000","last_unstale":"2024-09-21T14:02:23.376545+0000","last_undegraded":"2024-09-21T14:02:23.376545+0000","last_fullsized":"2024-09-21T14:02:23.376545+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:40.452724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:40.452724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":646,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:23.382527+0000","last_change":"2024-09-21T14:01:41.203762+0000","last_active":"2024-09-21T14:02:23.382527+0000","last_peered":"2024-09-21T14:02:23.382527+0000","last_clean":"2024-09-21T14:02:23.382527+0000","last_became_active":"2024-09-21T14:01:41.203275+0000","last_became_peered":"2024-09-21T14:01:41.203275+0000","last_unstale":"2024-09-21T14:02:23.382527+0000","last_undegraded":"2024-09-21T14:02:23.382527+0000","last_fullsized":"2024-09-21T14:02:23.382527+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:00.320510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:00.320510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:46.565572+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":618,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.091233+0000","last_change":"2024-09-21T13:59:32.208701+0000","last_active":"2024-09-21T14:02:25.091233+0000","last_peered":"2024-09-21T14:02:25.091233+0000","last_clean":"2024-09-21T14:02:25.091233+0000","last_became_active":"2024-09-21T13:59:32.203340+0000","last_became_peered":"2024-09-21T13:59:32.203340+0000","last_unstale":"2024-09-21T14:02:25.091233+0000","last_undegraded":"2024-09-21T14:02:25.091233+0000","last_fullsized":"2024-09-21T14:02:25.091233+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:34.696780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:34.696780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":630,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.094637+0000","last_change":"2024-09-21T13:58:17.778322+0000","last_active":"2024-09-21T14:02:25.094637+0000","last_peered":"2024-09-21T14:02:25.094637+0000","last_clean":"2024-09-21T14:02:25.094637+0000","last_became_active":"2024-09-21T13:58:17.777893+0000","last_became_peered":"2024-09-21T13:58:17.777893+0000","last_unstale":"2024-09-21T14:02:25.094637+0000","last_undegraded":"2024-09-21T14:02:25.094637+0000","last_fullsized":"2024-09-21T14:02:25.094637+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:07.001135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:55:07.001135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:22.269018+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"290'27","reported_seq":604,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.100600+0000","last_change":"2024-09-21T13:58:11.718421+0000","last_active":"2024-09-21T14:02:25.100600+0000","last_peered":"2024-09-21T14:02:25.100600+0000","last_clean":"2024-09-21T14:02:25.100600+0000","last_became_active":"2024-09-21T13:57:12.703469+0000","last_became_peered":"2024-09-21T13:57:12.703469+0000","last_unstale":"2024-09-21T14:02:25.100600+0000","last_undegraded":"2024-09-21T14:02:25.100600+0000","last_fullsized":"2024-09-21T14:02:25.100600+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:58:11.718334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:11.718334+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":669,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:23.381736+0000","last_change":"2024-09-21T14:01:32.717551+0000","last_active":"2024-09-21T14:02:23.381736+0000","last_peered":"2024-09-21T14:02:23.381736+0000","last_clean":"2024-09-21T14:02:23.381736+0000","last_became_active":"2024-09-21T13:58:17.773464+0000","last_became_peered":"2024-09-21T13:58:17.773464+0000","last_unstale":"2024-09-21T14:02:23.381736+0000","last_undegraded":"2024-09-21T14:02:23.381736+0000","last_fullsized":"2024-09-21T14:02:23.381736+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:32.717480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:01:32.717480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:02.205485+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":623,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.109299+0000","last_change":"2024-09-21T14:01:41.194807+0000","last_active":"2024-09-21T14:02:25.109299+0000","last_peered":"2024-09-21T14:02:25.109299+0000","last_clean":"2024-09-21T14:02:25.109299+0000","last_became_active":"2024-09-21T14:01:41.194517+0000","last_became_peered":"2024-09-21T14:01:41.194517+0000","last_unstale":"2024-09-21T14:02:25.109299+0000","last_undegraded":"2024-09-21T14:02:25.109299+0000","last_fullsized":"2024-09-21T14:02:25.109299+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:31.127926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:31.127926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"290'15","reported_seq":695,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:23.382727+0000","last_change":"2024-09-21T14:00:02.475761+0000","last_active":"2024-09-21T14:02:23.382727+0000","last_peered":"2024-09-21T14:02:23.382727+0000","last_clean":"2024-09-21T14:02:23.382727+0000","last_became_active":"2024-09-21T13:57:12.681408+0000","last_became_peered":"2024-09-21T13:57:12.681408+0000","last_unstale":"2024-09-21T14:02:23.382727+0000","last_undegraded":"2024-09-21T14:02:23.382727+0000","last_fullsized":"2024-09-21T14:02:23.382727+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'15","last_scrub_stamp":"2024-09-21T14:00:02.475661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:00:02.475661+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:37.694022+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":564,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.099494+0000","last_change":"2024-09-21T14:01:41.188585+0000","last_active":"2024-09-21T14:02:25.099494+0000","last_peered":"2024-09-21T14:02:25.099494+0000","last_clean":"2024-09-21T14:02:25.099494+0000","last_became_active":"2024-09-21T14:01:41.188248+0000","last_became_peered":"2024-09-21T14:01:41.188248+0000","last_unstale":"2024-09-21T14:02:25.099494+0000","last_undegraded":"2024-09-21T14:02:25.099494+0000","last_fullsized":"2024-09-21T14:02:25.099494+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:21.284048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:56:21.284048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":599,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:23.384752+0000","last_change":"2024-09-21T13:59:32.202128+0000","last_active":"2024-09-21T14:02:23.384752+0000","last_peered":"2024-09-21T14:02:23.384752+0000","last_clean":"2024-09-21T14:02:23.384752+0000","last_became_active":"2024-09-21T13:59:32.201158+0000","last_became_peered":"2024-09-21T13:59:32.201158+0000","last_unstale":"2024-09-21T14:02:23.384752+0000","last_undegraded":"2024-09-21T14:02:23.384752+0000","last_fullsized":"2024-09-21T14:02:23.384752+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:25.792985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:54:25.792985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:46.991952+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":600,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.099536+0000","last_change":"2024-09-21T14:01:41.189797+0000","last_active":"2024-09-21T14:02:25.099536+0000","last_peered":"2024-09-21T14:02:25.099536+0000","last_clean":"2024-09-21T14:02:25.099536+0000","last_became_active":"2024-09-21T14:01:41.189373+0000","last_became_peered":"2024-09-21T14:01:41.189373+0000","last_unstale":"2024-09-21T14:02:25.099536+0000","last_undegraded":"2024-09-21T14:02:25.099536+0000","last_fullsized":"2024-09-21T14:02:25.099536+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:54.328275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:54.328275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"290'7","reported_seq":720,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:23.382942+0000","last_change":"2024-09-21T14:01:57.563250+0000","last_active":"2024-09-21T14:02:23.382942+0000","last_peered":"2024-09-21T14:02:23.382942+0000","last_clean":"2024-09-21T14:02:23.382942+0000","last_became_active":"2024-09-21T13:59:09.063053+0000","last_became_peered":"2024-09-21T13:59:09.063053+0000","last_unstale":"2024-09-21T14:02:23.382942+0000","last_undegraded":"2024-09-21T14:02:23.382942+0000","last_fullsized":"2024-09-21T14:02:23.382942+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'7","last_scrub_stamp":"2024-09-21T14:01:57.563184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:57.563184+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:19.737603+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":593,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.106302+0000","last_change":"2024-09-21T13:59:32.204565+0000","last_active":"2024-09-21T14:02:25.106302+0000","last_peered":"2024-09-21T14:02:25.106302+0000","last_clean":"2024-09-21T14:02:25.106302+0000","last_became_active":"2024-09-21T13:59:32.204174+0000","last_became_peered":"2024-09-21T13:59:32.204174+0000","last_unstale":"2024-09-21T14:02:25.106302+0000","last_undegraded":"2024-09-21T14:02:25.106302+0000","last_fullsized":"2024-09-21T14:02:25.106302+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:06.722293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:06.722293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":634,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.099850+0000","last_change":"2024-09-21T14:01:41.199085+0000","last_active":"2024-09-21T14:02:25.099850+0000","last_peered":"2024-09-21T14:02:25.099850+0000","last_clean":"2024-09-21T14:02:25.099850+0000","last_became_active":"2024-09-21T14:01:41.198672+0000","last_became_peered":"2024-09-21T14:01:41.198672+0000","last_unstale":"2024-09-21T14:02:25.099850+0000","last_undegraded":"2024-09-21T14:02:25.099850+0000","last_fullsized":"2024-09-21T14:02:25.099850+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:44.376215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:44.376215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":688,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:23.384184+0000","last_change":"2024-09-21T14:01:53.460257+0000","last_active":"2024-09-21T14:02:23.384184+0000","last_peered":"2024-09-21T14:02:23.384184+0000","last_clean":"2024-09-21T14:02:23.384184+0000","last_became_active":"2024-09-21T14:01:41.200360+0000","last_became_peered":"2024-09-21T14:01:41.200360+0000","last_unstale":"2024-09-21T14:02:23.384184+0000","last_undegraded":"2024-09-21T14:02:23.384184+0000","last_fullsized":"2024-09-21T14:02:23.384184+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:53.460180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:01:53.460180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:15.802657+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":598,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:23.383072+0000","last_change":"2024-09-21T13:59:32.205037+0000","last_active":"2024-09-21T14:02:23.383072+0000","last_peered":"2024-09-21T14:02:23.383072+0000","last_clean":"2024-09-21T14:02:23.383072+0000","last_became_active":"2024-09-21T13:59:32.204749+0000","last_became_peered":"2024-09-21T13:59:32.204749+0000","last_unstale":"2024-09-21T14:02:23.383072+0000","last_undegraded":"2024-09-21T14:02:23.383072+0000","last_fullsized":"2024-09-21T14:02:23.383072+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:17.361699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:17.361699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:23.593131+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"290'27","reported_seq":661,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.306570+0000","last_change":"2024-09-21T14:01:27.893648+0000","last_active":"2024-09-21T14:02:17.306570+0000","last_peered":"2024-09-21T14:02:17.306570+0000","last_clean":"2024-09-21T14:02:17.306570+0000","last_became_active":"2024-09-21T13:57:12.680783+0000","last_became_peered":"2024-09-21T13:57:12.680783+0000","last_unstale":"2024-09-21T14:02:17.306570+0000","last_undegraded":"2024-09-21T14:02:17.306570+0000","last_fullsized":"2024-09-21T14:02:17.306570+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'27","last_scrub_stamp":"2024-09-21T14:01:27.893587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:27.893587+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:53.939497+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":656,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:23.382587+0000","last_change":"2024-09-21T14:01:54.024745+0000","last_active":"2024-09-21T14:02:23.382587+0000","last_peered":"2024-09-21T14:02:23.382587+0000","last_clean":"2024-09-21T14:02:23.382587+0000","last_became_active":"2024-09-21T13:49:02.066796+0000","last_became_peered":"2024-09-21T13:49:02.066796+0000","last_unstale":"2024-09-21T14:02:23.382587+0000","last_undegraded":"2024-09-21T14:02:23.382587+0000","last_fullsized":"2024-09-21T14:02:23.382587+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:54.024702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:01:54.024702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:06.646444+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":610,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.094293+0000","last_change":"2024-09-21T14:01:41.191486+0000","last_active":"2024-09-21T14:02:25.094293+0000","last_peered":"2024-09-21T14:02:25.094293+0000","last_clean":"2024-09-21T14:02:25.094293+0000","last_became_active":"2024-09-21T14:01:41.191116+0000","last_became_peered":"2024-09-21T14:01:41.191116+0000","last_unstale":"2024-09-21T14:02:25.094293+0000","last_undegraded":"2024-09-21T14:02:25.094293+0000","last_fullsized":"2024-09-21T14:02:25.094293+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:57.962266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:57.962266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:08.315476+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":666,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:23.382847+0000","last_change":"2024-09-21T14:00:51.495433+0000","last_active":"2024-09-21T14:02:23.382847+0000","last_peered":"2024-09-21T14:02:23.382847+0000","last_clean":"2024-09-21T14:02:23.382847+0000","last_became_active":"2024-09-21T13:52:04.316016+0000","last_became_peered":"2024-09-21T13:52:04.316016+0000","last_unstale":"2024-09-21T14:02:23.382847+0000","last_undegraded":"2024-09-21T14:02:23.382847+0000","last_fullsized":"2024-09-21T14:02:23.382847+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:00:51.495368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:00:51.495368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:10.415465+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"290'18","reported_seq":616,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.091197+0000","last_change":"2024-09-21T13:59:23.041396+0000","last_active":"2024-09-21T14:02:25.091197+0000","last_peered":"2024-09-21T14:02:25.091197+0000","last_clean":"2024-09-21T14:02:25.091197+0000","last_became_active":"2024-09-21T13:57:12.684454+0000","last_became_peered":"2024-09-21T13:57:12.684454+0000","last_unstale":"2024-09-21T14:02:25.091197+0000","last_undegraded":"2024-09-21T14:02:25.091197+0000","last_fullsized":"2024-09-21T14:02:25.091197+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'18","last_scrub_stamp":"2024-09-21T13:59:23.041311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:23.041311+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":653,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:23.383203+0000","last_change":"2024-09-21T13:58:16.393763+0000","last_active":"2024-09-21T14:02:23.383203+0000","last_peered":"2024-09-21T14:02:23.383203+0000","last_clean":"2024-09-21T14:02:23.383203+0000","last_became_active":"2024-09-21T13:57:12.681897+0000","last_became_peered":"2024-09-21T13:57:12.681897+0000","last_unstale":"2024-09-21T14:02:23.383203+0000","last_undegraded":"2024-09-21T14:02:23.383203+0000","last_fullsized":"2024-09-21T14:02:23.383203+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:16.393696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:16.393696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:24.632574+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":577,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.107083+0000","last_change":"2024-09-21T14:01:41.182300+0000","last_active":"2024-09-21T14:02:25.107083+0000","last_peered":"2024-09-21T14:02:25.107083+0000","last_clean":"2024-09-21T14:02:25.107083+0000","last_became_active":"2024-09-21T14:01:41.181946+0000","last_became_peered":"2024-09-21T14:01:41.181946+0000","last_unstale":"2024-09-21T14:02:25.107083+0000","last_undegraded":"2024-09-21T14:02:25.107083+0000","last_fullsized":"2024-09-21T14:02:25.107083+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T13:57:26.536684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:26.536684+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":676,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:23.376724+0000","last_change":"2024-09-21T14:01:58.994867+0000","last_active":"2024-09-21T14:02:23.376724+0000","last_peered":"2024-09-21T14:02:23.376724+0000","last_clean":"2024-09-21T14:02:23.376724+0000","last_became_active":"2024-09-21T13:57:12.685742+0000","last_became_peered":"2024-09-21T13:57:12.685742+0000","last_unstale":"2024-09-21T14:02:23.376724+0000","last_undegraded":"2024-09-21T14:02:23.376724+0000","last_fullsized":"2024-09-21T14:02:23.376724+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T14:01:58.994811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:01:58.994811+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:17.687542+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"290'20","reported_seq":557,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.108292+0000","last_change":"2024-09-21T13:59:32.242771+0000","last_active":"2024-09-21T14:02:25.108292+0000","last_peered":"2024-09-21T14:02:25.108292+0000","last_clean":"2024-09-21T14:02:25.108292+0000","last_became_active":"2024-09-21T13:59:32.213865+0000","last_became_peered":"2024-09-21T13:59:32.213865+0000","last_unstale":"2024-09-21T14:02:25.108292+0000","last_undegraded":"2024-09-21T14:02:25.108292+0000","last_fullsized":"2024-09-21T14:02:25.108292+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:57:27.578738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:57:27.578738+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"300'12","reported_seq":286,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:23.389881+0000","last_change":"2024-09-21T13:59:36.012775+0000","last_active":"2024-09-21T14:02:23.389881+0000","last_peered":"2024-09-21T14:02:23.389881+0000","last_clean":"2024-09-21T14:02:23.389881+0000","last_became_active":"2024-09-21T13:59:32.205051+0000","last_became_peered":"2024-09-21T13:59:32.205051+0000","last_unstale":"2024-09-21T14:02:23.389881+0000","last_undegraded":"2024-09-21T14:02:23.389881+0000","last_fullsized":"2024-09-21T14:02:23.389881+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'12","last_scrub_stamp":"2024-09-21T13:59:36.012693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:36.012693+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:20.587646+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"75'79","reported_seq":73,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.099567+0000","last_change":"2024-09-21T14:01:41.248772+0000","last_active":"2024-09-21T14:02:25.099567+0000","last_peered":"2024-09-21T14:02:25.099567+0000","last_clean":"2024-09-21T14:02:25.099567+0000","last_became_active":"2024-09-21T14:01:41.246888+0000","last_became_peered":"2024-09-21T14:01:41.246888+0000","last_unstale":"2024-09-21T14:02:25.099567+0000","last_undegraded":"2024-09-21T14:02:25.099567+0000","last_fullsized":"2024-09-21T14:02:25.099567+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53669,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"370'13930","reported_seq":14163,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:26.127474+0000","last_change":"2024-09-21T13:57:53.208687+0000","last_active":"2024-09-21T14:02:26.127474+0000","last_peered":"2024-09-21T14:02:26.127474+0000","last_clean":"2024-09-21T14:02:26.127474+0000","last_became_active":"2024-09-21T13:57:53.208352+0000","last_became_peered":"2024-09-21T13:57:53.208352+0000","last_unstale":"2024-09-21T14:02:26.127474+0000","last_undegraded":"2024-09-21T14:02:26.127474+0000","last_fullsized":"2024-09-21T14:02:26.127474+0000","mapping_epoch":260,"log_start":"310'9515","ondisk_log_start":"310'9515","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4415,"log_dups_size":0,"ondisk_log_size":4415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84279296,"num_objects":1286,"num_object_clones":0,"num_object_copies":3858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13930,"num_write_kb":486912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"290'27","reported_seq":714,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:23.377363+0000","last_change":"2024-09-21T13:58:45.984691+0000","last_active":"2024-09-21T14:02:23.377363+0000","last_peered":"2024-09-21T14:02:23.377363+0000","last_clean":"2024-09-21T14:02:23.377363+0000","last_became_active":"2024-09-21T13:57:12.695609+0000","last_became_peered":"2024-09-21T13:57:12.695609+0000","last_unstale":"2024-09-21T14:02:23.377363+0000","last_undegraded":"2024-09-21T14:02:23.377363+0000","last_fullsized":"2024-09-21T14:02:23.377363+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:58:45.984575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:45.984575+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":602,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.090706+0000","last_change":"2024-09-21T13:59:32.205723+0000","last_active":"2024-09-21T14:02:25.090706+0000","last_peered":"2024-09-21T14:02:25.090706+0000","last_clean":"2024-09-21T14:02:25.090706+0000","last_became_active":"2024-09-21T13:59:32.205350+0000","last_became_peered":"2024-09-21T13:59:32.205350+0000","last_unstale":"2024-09-21T14:02:25.090706+0000","last_undegraded":"2024-09-21T14:02:25.090706+0000","last_fullsized":"2024-09-21T14:02:25.090706+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:32.668851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:32.668851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"346'10","reported_seq":842,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:23.384283+0000","last_change":"2024-09-21T14:01:55.521513+0000","last_active":"2024-09-21T14:02:23.384283+0000","last_peered":"2024-09-21T14:02:23.384283+0000","last_clean":"2024-09-21T14:02:23.384283+0000","last_became_active":"2024-09-21T14:01:41.200218+0000","last_became_peered":"2024-09-21T14:01:41.200218+0000","last_unstale":"2024-09-21T14:02:23.384283+0000","last_undegraded":"2024-09-21T14:02:23.384283+0000","last_fullsized":"2024-09-21T14:02:23.384283+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'10","last_scrub_stamp":"2024-09-21T14:01:55.521438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:01:55.521438+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:55.829821+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":529,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.108568+0000","last_change":"2024-09-21T14:01:41.207310+0000","last_active":"2024-09-21T14:02:25.108568+0000","last_peered":"2024-09-21T14:02:25.108568+0000","last_clean":"2024-09-21T14:02:25.108568+0000","last_became_active":"2024-09-21T14:01:41.206720+0000","last_became_peered":"2024-09-21T14:01:41.206720+0000","last_unstale":"2024-09-21T14:02:25.108568+0000","last_undegraded":"2024-09-21T14:02:25.108568+0000","last_fullsized":"2024-09-21T14:02:25.108568+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:52.244483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:52.244483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"75'79","reported_seq":124,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:23.383820+0000","last_change":"2024-09-21T14:02:06.395083+0000","last_active":"2024-09-21T14:02:23.383820+0000","last_peered":"2024-09-21T14:02:23.383820+0000","last_clean":"2024-09-21T14:02:23.383820+0000","last_became_active":"2024-09-21T14:00:41.137756+0000","last_became_peered":"2024-09-21T14:00:41.137756+0000","last_unstale":"2024-09-21T14:02:23.383820+0000","last_undegraded":"2024-09-21T14:02:23.383820+0000","last_fullsized":"2024-09-21T14:02:23.383820+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:02:06.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:02:06.395011+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:29.136654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"369'13800","reported_seq":18068,"reported_epoch":369,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-21T14:02:24.686320+0000","last_change":"2024-09-21T14:02:11.235474+0000","last_active":"2024-09-21T14:02:24.686320+0000","last_peered":"2024-09-21T14:02:24.686320+0000","last_clean":"2024-09-21T13:58:23.757609+0000","last_became_active":"2024-09-21T14:01:41.187681+0000","last_became_peered":"2024-09-21T14:01:41.187681+0000","last_unstale":"2024-09-21T14:02:24.686320+0000","last_undegraded":"2024-09-21T14:01:41.168266+0000","last_fullsized":"2024-09-21T14:01:41.168096+0000","mapping_epoch":345,"log_start":"310'9220","ondisk_log_start":"310'9220","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4580,"log_dups_size":0,"ondisk_log_size":4580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92340248,"num_objects":1410,"num_object_clones":0,"num_object_copies":4230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1410,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13800,"num_write_kb":486657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2003,"num_bytes_recovered":131203096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":1410}],"blocked_by":[],"up_primary":7,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"290'30","reported_seq":626,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.090731+0000","last_change":"2024-09-21T13:59:32.246090+0000","last_active":"2024-09-21T14:02:25.090731+0000","last_peered":"2024-09-21T14:02:25.090731+0000","last_clean":"2024-09-21T14:02:25.090731+0000","last_became_active":"2024-09-21T13:59:32.205887+0000","last_became_peered":"2024-09-21T13:59:32.205887+0000","last_unstale":"2024-09-21T14:02:25.090731+0000","last_undegraded":"2024-09-21T14:02:25.090731+0000","last_fullsized":"2024-09-21T14:02:25.090731+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'18","last_scrub_stamp":"2024-09-21T13:54:19.001030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:19.001030+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:40.231306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":554,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.094613+0000","last_change":"2024-09-21T13:59:32.212559+0000","last_active":"2024-09-21T14:02:25.094613+0000","last_peered":"2024-09-21T14:02:25.094613+0000","last_clean":"2024-09-21T14:02:25.094613+0000","last_became_active":"2024-09-21T13:59:32.211004+0000","last_became_peered":"2024-09-21T13:59:32.211004+0000","last_unstale":"2024-09-21T14:02:25.094613+0000","last_undegraded":"2024-09-21T14:02:25.094613+0000","last_fullsized":"2024-09-21T14:02:25.094613+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:25.736461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:25.736461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":670,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:23.382210+0000","last_change":"2024-09-21T13:59:34.293330+0000","last_active":"2024-09-21T14:02:23.382210+0000","last_peered":"2024-09-21T14:02:23.382210+0000","last_clean":"2024-09-21T14:02:23.382210+0000","last_became_active":"2024-09-21T13:59:32.220304+0000","last_became_peered":"2024-09-21T13:59:32.220304+0000","last_unstale":"2024-09-21T14:02:23.382210+0000","last_undegraded":"2024-09-21T14:02:23.382210+0000","last_fullsized":"2024-09-21T14:02:23.382210+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:34.292892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:34.292892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:15.468420+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":650,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:23.382698+0000","last_change":"2024-09-21T13:59:23.334230+0000","last_active":"2024-09-21T14:02:23.382698+0000","last_peered":"2024-09-21T14:02:23.382698+0000","last_clean":"2024-09-21T14:02:23.382698+0000","last_became_active":"2024-09-21T13:57:12.680372+0000","last_became_peered":"2024-09-21T13:57:12.680372+0000","last_unstale":"2024-09-21T14:02:23.382698+0000","last_undegraded":"2024-09-21T14:02:23.382698+0000","last_fullsized":"2024-09-21T14:02:23.382698+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:23.334162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:23.334162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:17.515611+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"75'79","reported_seq":129,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.091877+0000","last_change":"2024-09-21T14:01:42.177646+0000","last_active":"2024-09-21T14:02:25.091877+0000","last_peered":"2024-09-21T14:02:25.091877+0000","last_clean":"2024-09-21T14:02:25.091877+0000","last_became_active":"2024-09-21T14:01:42.173042+0000","last_became_peered":"2024-09-21T14:01:42.173042+0000","last_unstale":"2024-09-21T14:02:25.091877+0000","last_undegraded":"2024-09-21T14:02:25.091877+0000","last_fullsized":"2024-09-21T14:02:25.091877+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:02:42.254915+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":709030,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"370'13777","reported_seq":20942,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:26.150989+0000","last_change":"2024-09-21T14:01:00.058151+0000","last_active":"2024-09-21T14:02:26.150989+0000","last_peered":"2024-09-21T14:02:26.150989+0000","last_clean":"2024-09-21T14:02:26.150989+0000","last_became_active":"2024-09-21T14:01:00.026605+0000","last_became_peered":"2024-09-21T14:01:00.026605+0000","last_unstale":"2024-09-21T14:02:26.150989+0000","last_undegraded":"2024-09-21T14:02:26.150989+0000","last_fullsized":"2024-09-21T14:02:26.150989+0000","mapping_epoch":329,"log_start":"310'9342","ondisk_log_start":"310'9342","created":260,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4435,"log_dups_size":0,"ondisk_log_size":4435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83427328,"num_objects":1273,"num_object_clones":0,"num_object_copies":3819,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13777,"num_write_kb":481600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3456,"num_bytes_recovered":226492416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"290'28","reported_seq":759,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:23.382262+0000","last_change":"2024-09-21T14:02:02.471226+0000","last_active":"2024-09-21T14:02:23.382262+0000","last_peered":"2024-09-21T14:02:23.382262+0000","last_clean":"2024-09-21T14:02:23.382262+0000","last_became_active":"2024-09-21T14:01:41.200859+0000","last_became_peered":"2024-09-21T14:01:41.200859+0000","last_unstale":"2024-09-21T14:02:23.382262+0000","last_undegraded":"2024-09-21T14:02:23.382262+0000","last_fullsized":"2024-09-21T14:02:23.382262+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'28","last_scrub_stamp":"2024-09-21T14:02:02.471133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:02.471133+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:14.278651+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":658,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:23.390894+0000","last_change":"2024-09-21T13:57:38.046718+0000","last_active":"2024-09-21T14:02:23.390894+0000","last_peered":"2024-09-21T14:02:23.390894+0000","last_clean":"2024-09-21T14:02:23.390894+0000","last_became_active":"2024-09-21T13:48:58.053533+0000","last_became_peered":"2024-09-21T13:48:58.053533+0000","last_unstale":"2024-09-21T14:02:23.390894+0000","last_undegraded":"2024-09-21T14:02:23.390894+0000","last_fullsized":"2024-09-21T14:02:23.390894+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:57:38.046627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:38.046627+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:41.742834+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"241'10","reported_seq":697,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.090840+0000","last_change":"2024-09-21T13:58:06.789629+0000","last_active":"2024-09-21T14:02:25.090840+0000","last_peered":"2024-09-21T14:02:25.090840+0000","last_clean":"2024-09-21T14:02:25.090840+0000","last_became_active":"2024-09-21T13:57:12.686651+0000","last_became_peered":"2024-09-21T13:57:12.686651+0000","last_unstale":"2024-09-21T14:02:25.090840+0000","last_undegraded":"2024-09-21T14:02:25.090840+0000","last_fullsized":"2024-09-21T14:02:25.090840+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'10","last_scrub_stamp":"2024-09-21T13:58:06.789560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:06.789560+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":618,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.107048+0000","last_change":"2024-09-21T13:59:32.210257+0000","last_active":"2024-09-21T14:02:25.107048+0000","last_peered":"2024-09-21T14:02:25.107048+0000","last_clean":"2024-09-21T14:02:25.107048+0000","last_became_active":"2024-09-21T13:59:32.209615+0000","last_became_peered":"2024-09-21T13:59:32.209615+0000","last_unstale":"2024-09-21T14:02:25.107048+0000","last_undegraded":"2024-09-21T14:02:25.107048+0000","last_fullsized":"2024-09-21T14:02:25.107048+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:37.548064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:37.548064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"75'79","reported_seq":697,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:23.389578+0000","last_change":"2024-09-21T14:02:05.693250+0000","last_active":"2024-09-21T14:02:23.389578+0000","last_peered":"2024-09-21T14:02:23.389578+0000","last_clean":"2024-09-21T14:02:23.389578+0000","last_became_active":"2024-09-21T14:00:41.083425+0000","last_became_peered":"2024-09-21T14:00:41.083425+0000","last_unstale":"2024-09-21T14:02:23.389578+0000","last_undegraded":"2024-09-21T14:02:23.389578+0000","last_fullsized":"2024-09-21T14:02:23.389578+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:02:05.693193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:02:05.693193+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:28.620866+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.f","version":"370'14008","reported_seq":14228,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.814970+0000","last_change":"2024-09-21T14:01:41.336305+0000","last_active":"2024-09-21T14:02:25.814970+0000","last_peered":"2024-09-21T14:02:25.814970+0000","last_clean":"2024-09-21T14:02:25.814970+0000","last_became_active":"2024-09-21T14:01:41.202052+0000","last_became_peered":"2024-09-21T14:01:41.202052+0000","last_unstale":"2024-09-21T14:02:25.814970+0000","last_undegraded":"2024-09-21T14:02:25.814970+0000","last_fullsized":"2024-09-21T14:02:25.814970+0000","mapping_epoch":345,"log_start":"308'9002","ondisk_log_start":"308'9002","created":260,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":5006,"log_dups_size":0,"ondisk_log_size":5006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87162880,"num_objects":1330,"num_object_clones":0,"num_object_copies":3990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14008,"num_write_kb":490816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"290'20","reported_seq":694,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:23.377228+0000","last_change":"2024-09-21T14:01:41.187743+0000","last_active":"2024-09-21T14:02:23.377228+0000","last_peered":"2024-09-21T14:02:23.377228+0000","last_clean":"2024-09-21T14:02:23.377228+0000","last_became_active":"2024-09-21T14:01:41.187060+0000","last_became_peered":"2024-09-21T14:01:41.187060+0000","last_unstale":"2024-09-21T14:02:23.377228+0000","last_undegraded":"2024-09-21T14:02:23.377228+0000","last_fullsized":"2024-09-21T14:02:23.377228+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:54:19.593611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:19.593611+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:45.999996+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"61'1","reported_seq":591,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.108789+0000","last_change":"2024-09-21T14:01:41.195580+0000","last_active":"2024-09-21T14:02:25.108789+0000","last_peered":"2024-09-21T14:02:25.108789+0000","last_clean":"2024-09-21T14:02:25.108789+0000","last_became_active":"2024-09-21T14:01:41.195268+0000","last_became_peered":"2024-09-21T14:01:41.195268+0000","last_unstale":"2024-09-21T14:02:25.108789+0000","last_undegraded":"2024-09-21T14:02:25.108789+0000","last_fullsized":"2024-09-21T14:02:25.108789+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:58:48.407734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:48.407734+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"346'15","reported_seq":279,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:23.389174+0000","last_change":"2024-09-21T14:01:41.209662+0000","last_active":"2024-09-21T14:02:23.389174+0000","last_peered":"2024-09-21T14:02:23.389174+0000","last_clean":"2024-09-21T14:02:23.389174+0000","last_became_active":"2024-09-21T14:01:41.209296+0000","last_became_peered":"2024-09-21T14:01:41.209296+0000","last_unstale":"2024-09-21T14:02:23.389174+0000","last_undegraded":"2024-09-21T14:02:23.389174+0000","last_fullsized":"2024-09-21T14:02:23.389174+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'12","last_scrub_stamp":"2024-09-21T13:57:30.133392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:30.133392+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:41.760065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":472,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.091737+0000","last_change":"2024-09-21T13:58:58.730439+0000","last_active":"2024-09-21T14:02:25.091737+0000","last_peered":"2024-09-21T14:02:25.091737+0000","last_clean":"2024-09-21T14:02:25.091737+0000","last_became_active":"2024-09-21T13:57:12.704897+0000","last_became_peered":"2024-09-21T13:57:12.704897+0000","last_unstale":"2024-09-21T14:02:25.091737+0000","last_undegraded":"2024-09-21T14:02:25.091737+0000","last_fullsized":"2024-09-21T14:02:25.091737+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:58.730368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:58.730368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"75'79","reported_seq":697,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:23.389673+0000","last_change":"2024-09-21T14:02:03.737005+0000","last_active":"2024-09-21T14:02:23.389673+0000","last_peered":"2024-09-21T14:02:23.389673+0000","last_clean":"2024-09-21T14:02:23.389673+0000","last_became_active":"2024-09-21T14:00:41.073047+0000","last_became_peered":"2024-09-21T14:00:41.073047+0000","last_unstale":"2024-09-21T14:02:23.389673+0000","last_undegraded":"2024-09-21T14:02:23.389673+0000","last_fullsized":"2024-09-21T14:02:23.389673+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:02:03.736928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:02:03.736928+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:21.563427+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"370'13647","reported_seq":23434,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:26.164772+0000","last_change":"2024-09-21T14:00:42.188751+0000","last_active":"2024-09-21T14:02:26.164772+0000","last_peered":"2024-09-21T14:02:26.164772+0000","last_clean":"2024-09-21T14:02:26.164772+0000","last_became_active":"2024-09-21T14:00:42.129476+0000","last_became_peered":"2024-09-21T14:00:42.129476+0000","last_unstale":"2024-09-21T14:02:26.164772+0000","last_undegraded":"2024-09-21T14:02:26.164772+0000","last_fullsized":"2024-09-21T14:02:26.164772+0000","mapping_epoch":321,"log_start":"310'9069","ondisk_log_start":"310'9069","created":260,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4578,"log_dups_size":0,"ondisk_log_size":4578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83820544,"num_objects":1279,"num_object_clones":0,"num_object_copies":3837,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13647,"num_write_kb":477632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4765,"num_bytes_recovered":312213504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"290'29","reported_seq":597,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.099541+0000","last_change":"2024-09-21T14:01:41.188261+0000","last_active":"2024-09-21T14:02:25.099541+0000","last_peered":"2024-09-21T14:02:25.099541+0000","last_clean":"2024-09-21T14:02:25.099541+0000","last_became_active":"2024-09-21T14:01:41.186960+0000","last_became_peered":"2024-09-21T14:01:41.186960+0000","last_unstale":"2024-09-21T14:02:25.099541+0000","last_undegraded":"2024-09-21T14:02:25.099541+0000","last_fullsized":"2024-09-21T14:02:25.099541+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-21T13:56:55.319759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:56:55.319759+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":671,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:23.378028+0000","last_change":"2024-09-21T14:01:41.185706+0000","last_active":"2024-09-21T14:02:23.378028+0000","last_peered":"2024-09-21T14:02:23.378028+0000","last_clean":"2024-09-21T14:02:23.378028+0000","last_became_active":"2024-09-21T14:01:41.185400+0000","last_became_peered":"2024-09-21T14:01:41.185400+0000","last_unstale":"2024-09-21T14:02:23.378028+0000","last_undegraded":"2024-09-21T14:02:23.378028+0000","last_fullsized":"2024-09-21T14:02:23.378028+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:16.712990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:16.712990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":238,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:23.388502+0000","last_change":"2024-09-21T14:01:52.671029+0000","last_active":"2024-09-21T14:02:23.388502+0000","last_peered":"2024-09-21T14:02:23.388502+0000","last_clean":"2024-09-21T14:02:23.388502+0000","last_became_active":"2024-09-21T14:01:41.193973+0000","last_became_peered":"2024-09-21T14:01:41.193973+0000","last_unstale":"2024-09-21T14:02:23.388502+0000","last_undegraded":"2024-09-21T14:02:23.388502+0000","last_fullsized":"2024-09-21T14:02:23.388502+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:52.670974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:01:52.670974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:58.750020+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":662,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:23.376479+0000","last_change":"2024-09-21T13:58:43.952014+0000","last_active":"2024-09-21T14:02:23.376479+0000","last_peered":"2024-09-21T14:02:23.376479+0000","last_clean":"2024-09-21T14:02:23.376479+0000","last_became_active":"2024-09-21T13:58:17.757628+0000","last_became_peered":"2024-09-21T13:58:17.757628+0000","last_unstale":"2024-09-21T14:02:23.376479+0000","last_undegraded":"2024-09-21T14:02:23.376479+0000","last_fullsized":"2024-09-21T14:02:23.376479+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:43.951941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:43.951941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"75'79","reported_seq":73,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.099352+0000","last_change":"2024-09-21T14:01:41.199377+0000","last_active":"2024-09-21T14:02:25.099352+0000","last_peered":"2024-09-21T14:02:25.099352+0000","last_clean":"2024-09-21T14:02:25.099352+0000","last_became_active":"2024-09-21T14:01:41.198750+0000","last_became_peered":"2024-09-21T14:01:41.198750+0000","last_unstale":"2024-09-21T14:02:25.099352+0000","last_undegraded":"2024-09-21T14:02:25.099352+0000","last_fullsized":"2024-09-21T14:02:25.099352+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:02:43.568139+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"370'13963","reported_seq":24781,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:26.157415+0000","last_change":"2024-09-21T14:01:44.191023+0000","last_active":"2024-09-21T14:02:26.157415+0000","last_peered":"2024-09-21T14:02:26.157415+0000","last_clean":"2024-09-21T14:02:26.157415+0000","last_became_active":"2024-09-21T14:01:44.174262+0000","last_became_peered":"2024-09-21T14:01:44.174262+0000","last_unstale":"2024-09-21T14:02:26.157415+0000","last_undegraded":"2024-09-21T14:02:26.157415+0000","last_fullsized":"2024-09-21T14:02:26.157415+0000","mapping_epoch":348,"log_start":"310'9331","ondisk_log_start":"310'9331","created":260,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4632,"log_dups_size":0,"ondisk_log_size":4632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85786624,"num_objects":1309,"num_object_clones":0,"num_object_copies":3927,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13963,"num_write_kb":488704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5260,"num_bytes_recovered":344588288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"290'44","reported_seq":899,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:23.377928+0000","last_change":"2024-09-21T13:58:56.076153+0000","last_active":"2024-09-21T14:02:23.377928+0000","last_peered":"2024-09-21T14:02:23.377928+0000","last_clean":"2024-09-21T14:02:23.377928+0000","last_became_active":"2024-09-21T13:57:12.684690+0000","last_became_peered":"2024-09-21T13:57:12.684690+0000","last_unstale":"2024-09-21T14:02:23.377928+0000","last_undegraded":"2024-09-21T14:02:23.377928+0000","last_fullsized":"2024-09-21T14:02:23.377928+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'32","last_scrub_stamp":"2024-09-21T13:58:56.076074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:56.076074+0000","objects_scrubbed":10,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":678,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:23.383093+0000","last_change":"2024-09-21T14:01:58.524159+0000","last_active":"2024-09-21T14:02:23.383093+0000","last_peered":"2024-09-21T14:02:23.383093+0000","last_clean":"2024-09-21T14:02:23.383093+0000","last_became_active":"2024-09-21T13:58:17.770937+0000","last_became_peered":"2024-09-21T13:58:17.770937+0000","last_unstale":"2024-09-21T14:02:23.383093+0000","last_undegraded":"2024-09-21T14:02:23.383093+0000","last_fullsized":"2024-09-21T14:02:23.383093+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:58.524114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:01:58.524114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:07.944133+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":610,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.099515+0000","last_change":"2024-09-21T14:01:41.249061+0000","last_active":"2024-09-21T14:02:25.099515+0000","last_peered":"2024-09-21T14:02:25.099515+0000","last_clean":"2024-09-21T14:02:25.099515+0000","last_became_active":"2024-09-21T14:01:41.247642+0000","last_became_peered":"2024-09-21T14:01:41.247642+0000","last_unstale":"2024-09-21T14:02:25.099515+0000","last_undegraded":"2024-09-21T14:02:25.099515+0000","last_fullsized":"2024-09-21T14:02:25.099515+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-21T13:59:22.186393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:59:22.186393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":589,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:23.382384+0000","last_change":"2024-09-21T13:55:07.021128+0000","last_active":"2024-09-21T14:02:23.382384+0000","last_peered":"2024-09-21T14:02:23.382384+0000","last_clean":"2024-09-21T14:02:23.382384+0000","last_became_active":"2024-09-21T13:50:29.688249+0000","last_became_peered":"2024-09-21T13:50:29.688249+0000","last_unstale":"2024-09-21T14:02:23.382384+0000","last_undegraded":"2024-09-21T14:02:23.382384+0000","last_fullsized":"2024-09-21T14:02:23.382384+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:07.021059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:55:07.021059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:47.974697+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":581,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.095019+0000","last_change":"2024-09-21T13:59:32.209357+0000","last_active":"2024-09-21T14:02:25.095019+0000","last_peered":"2024-09-21T14:02:25.095019+0000","last_clean":"2024-09-21T14:02:25.095019+0000","last_became_active":"2024-09-21T13:59:32.208980+0000","last_became_peered":"2024-09-21T13:59:32.208980+0000","last_unstale":"2024-09-21T14:02:25.095019+0000","last_undegraded":"2024-09-21T14:02:25.095019+0000","last_fullsized":"2024-09-21T14:02:25.095019+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:16.797227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:16.797227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"75'79","reported_seq":705,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.106145+0000","last_change":"2024-09-21T14:00:41.077286+0000","last_active":"2024-09-21T14:02:25.106145+0000","last_peered":"2024-09-21T14:02:25.106145+0000","last_clean":"2024-09-21T14:02:25.106145+0000","last_became_active":"2024-09-21T14:00:41.076948+0000","last_became_peered":"2024-09-21T14:00:41.076948+0000","last_unstale":"2024-09-21T14:02:25.106145+0000","last_undegraded":"2024-09-21T14:02:25.106145+0000","last_fullsized":"2024-09-21T14:02:25.106145+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:02:44.823429+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.a","version":"370'13594","reported_seq":14522,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.781666+0000","last_change":"2024-09-21T14:01:41.339174+0000","last_active":"2024-09-21T14:02:25.781666+0000","last_peered":"2024-09-21T14:02:25.781666+0000","last_clean":"2024-09-21T14:02:25.781666+0000","last_became_active":"2024-09-21T14:01:41.190267+0000","last_became_peered":"2024-09-21T14:01:41.190267+0000","last_unstale":"2024-09-21T14:02:25.781666+0000","last_undegraded":"2024-09-21T14:02:25.781666+0000","last_fullsized":"2024-09-21T14:02:25.781666+0000","mapping_epoch":345,"log_start":"307'8674","ondisk_log_start":"307'8674","created":260,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4920,"log_dups_size":0,"ondisk_log_size":4920,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83099648,"num_objects":1268,"num_object_clones":0,"num_object_copies":3804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13594,"num_write_kb":475584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5098,"num_bytes_recovered":241565696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"290'17","reported_seq":697,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:23.377421+0000","last_change":"2024-09-21T14:01:41.200809+0000","last_active":"2024-09-21T14:02:23.377421+0000","last_peered":"2024-09-21T14:02:23.377421+0000","last_clean":"2024-09-21T14:02:23.377421+0000","last_became_active":"2024-09-21T14:01:41.200365+0000","last_became_peered":"2024-09-21T14:01:41.200365+0000","last_unstale":"2024-09-21T14:02:23.377421+0000","last_undegraded":"2024-09-21T14:02:23.377421+0000","last_fullsized":"2024-09-21T14:02:23.377421+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'13","last_scrub_stamp":"2024-09-21T13:58:11.619848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:11.619848+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":698,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:23.383806+0000","last_change":"2024-09-21T13:59:29.454265+0000","last_active":"2024-09-21T14:02:23.383806+0000","last_peered":"2024-09-21T14:02:23.383806+0000","last_clean":"2024-09-21T14:02:23.383806+0000","last_became_active":"2024-09-21T13:58:17.759671+0000","last_became_peered":"2024-09-21T13:58:17.759671+0000","last_unstale":"2024-09-21T14:02:23.383806+0000","last_undegraded":"2024-09-21T14:02:23.383806+0000","last_fullsized":"2024-09-21T14:02:23.383806+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:29.454085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:59:29.454085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":585,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.302637+0000","last_change":"2024-09-21T13:58:41.644876+0000","last_active":"2024-09-21T14:02:17.302637+0000","last_peered":"2024-09-21T14:02:17.302637+0000","last_clean":"2024-09-21T14:02:17.302637+0000","last_became_active":"2024-09-21T13:57:12.682147+0000","last_became_peered":"2024-09-21T13:57:12.682147+0000","last_unstale":"2024-09-21T14:02:17.302637+0000","last_undegraded":"2024-09-21T14:02:17.302637+0000","last_fullsized":"2024-09-21T14:02:17.302637+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:58:41.644789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:58:41.644789+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:13.796854+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":564,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.307100+0000","last_change":"2024-09-21T14:01:50.880840+0000","last_active":"2024-09-21T14:02:17.307100+0000","last_peered":"2024-09-21T14:02:17.307100+0000","last_clean":"2024-09-21T14:02:17.307100+0000","last_became_active":"2024-09-21T14:01:41.182000+0000","last_became_peered":"2024-09-21T14:01:41.182000+0000","last_unstale":"2024-09-21T14:02:17.307100+0000","last_undegraded":"2024-09-21T14:02:17.307100+0000","last_fullsized":"2024-09-21T14:02:17.307100+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:50.880471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:01:50.880471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:14.118692+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":626,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.108066+0000","last_change":"2024-09-21T13:59:32.222791+0000","last_active":"2024-09-21T14:02:25.108066+0000","last_peered":"2024-09-21T14:02:25.108066+0000","last_clean":"2024-09-21T14:02:25.108066+0000","last_became_active":"2024-09-21T13:59:32.221502+0000","last_became_peered":"2024-09-21T13:59:32.221502+0000","last_unstale":"2024-09-21T14:02:25.108066+0000","last_undegraded":"2024-09-21T14:02:25.108066+0000","last_fullsized":"2024-09-21T14:02:25.108066+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:14.673582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:14.673582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"75'79","reported_seq":108,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:23.380828+0000","last_change":"2024-09-21T14:00:41.071631+0000","last_active":"2024-09-21T14:02:23.380828+0000","last_peered":"2024-09-21T14:02:23.380828+0000","last_clean":"2024-09-21T14:02:23.380828+0000","last_became_active":"2024-09-21T14:00:41.071042+0000","last_became_peered":"2024-09-21T14:00:41.071042+0000","last_unstale":"2024-09-21T14:02:23.380828+0000","last_undegraded":"2024-09-21T14:02:23.380828+0000","last_fullsized":"2024-09-21T14:02:23.380828+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:02:42.091028+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.b","version":"369'13715","reported_seq":17914,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:23.538842+0000","last_change":"2024-09-21T14:01:54.238547+0000","last_active":"2024-09-21T14:02:23.538842+0000","last_peered":"2024-09-21T14:02:23.538842+0000","last_clean":"2024-09-21T14:02:23.538842+0000","last_became_active":"2024-09-21T14:01:41.210137+0000","last_became_peered":"2024-09-21T14:01:41.210137+0000","last_unstale":"2024-09-21T14:02:23.538842+0000","last_undegraded":"2024-09-21T14:02:23.538842+0000","last_fullsized":"2024-09-21T14:02:23.538842+0000","mapping_epoch":345,"log_start":"302'7832","ondisk_log_start":"302'7832","created":260,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"356'12122","last_scrub_stamp":"2024-09-21T14:01:54.238344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T14:01:54.238344+0000","objects_scrubbed":3072,"log_size":5883,"log_dups_size":0,"ondisk_log_size":5883,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:15.605090+0000","scrub_duration":597,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96272384,"num_objects":1469,"num_object_clones":0,"num_object_copies":4407,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13715,"num_write_kb":485888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1910,"num_bytes_recovered":123994112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"290'24","reported_seq":748,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:23.390755+0000","last_change":"2024-09-21T14:01:41.185105+0000","last_active":"2024-09-21T14:02:23.390755+0000","last_peered":"2024-09-21T14:02:23.390755+0000","last_clean":"2024-09-21T14:02:23.390755+0000","last_became_active":"2024-09-21T14:01:41.184728+0000","last_became_peered":"2024-09-21T14:01:41.184728+0000","last_unstale":"2024-09-21T14:02:23.390755+0000","last_undegraded":"2024-09-21T14:02:23.390755+0000","last_fullsized":"2024-09-21T14:02:23.390755+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'24","last_scrub_stamp":"2024-09-21T13:59:28.458768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:28.458768+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":599,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:23.382359+0000","last_change":"2024-09-21T13:59:59.450613+0000","last_active":"2024-09-21T14:02:23.382359+0000","last_peered":"2024-09-21T14:02:23.382359+0000","last_clean":"2024-09-21T14:02:23.382359+0000","last_became_active":"2024-09-21T13:57:12.681030+0000","last_became_peered":"2024-09-21T13:57:12.681030+0000","last_unstale":"2024-09-21T14:02:23.382359+0000","last_undegraded":"2024-09-21T14:02:23.382359+0000","last_fullsized":"2024-09-21T14:02:23.382359+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:59.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:59:59.450546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:26.702708+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":1334,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.101985+0000","last_change":"2024-09-21T13:59:32.218796+0000","last_active":"2024-09-21T14:02:25.101985+0000","last_peered":"2024-09-21T14:02:25.101985+0000","last_clean":"2024-09-21T14:02:25.101985+0000","last_became_active":"2024-09-21T13:59:32.203361+0000","last_became_peered":"2024-09-21T13:59:32.203361+0000","last_unstale":"2024-09-21T14:02:25.101985+0000","last_undegraded":"2024-09-21T14:02:25.101985+0000","last_fullsized":"2024-09-21T14:02:25.101985+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-21T13:56:29.617143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:56:29.617143+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047048000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":180,"num_read_kb":175,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":581,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.100272+0000","last_change":"2024-09-21T14:01:41.212595+0000","last_active":"2024-09-21T14:02:25.100272+0000","last_peered":"2024-09-21T14:02:25.100272+0000","last_clean":"2024-09-21T14:02:25.100272+0000","last_became_active":"2024-09-21T14:01:41.200531+0000","last_became_peered":"2024-09-21T14:01:41.200531+0000","last_unstale":"2024-09-21T14:02:25.100272+0000","last_undegraded":"2024-09-21T14:02:25.100272+0000","last_fullsized":"2024-09-21T14:02:25.100272+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:33.316230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:56:33.316230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":683,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:23.377735+0000","last_change":"2024-09-21T14:01:41.184939+0000","last_active":"2024-09-21T14:02:23.377735+0000","last_peered":"2024-09-21T14:02:23.377735+0000","last_clean":"2024-09-21T14:02:23.377735+0000","last_became_active":"2024-09-21T14:01:41.184664+0000","last_became_peered":"2024-09-21T14:01:41.184664+0000","last_unstale":"2024-09-21T14:02:23.377735+0000","last_undegraded":"2024-09-21T14:02:23.377735+0000","last_fullsized":"2024-09-21T14:02:23.377735+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:26.557002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:26.557002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"370'14136","reported_seq":14408,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:26.137294+0000","last_change":"2024-09-21T14:01:41.317185+0000","last_active":"2024-09-21T14:02:26.137294+0000","last_peered":"2024-09-21T14:02:26.137294+0000","last_clean":"2024-09-21T14:02:26.137294+0000","last_became_active":"2024-09-21T14:01:41.197526+0000","last_became_peered":"2024-09-21T14:01:41.197526+0000","last_unstale":"2024-09-21T14:02:26.137294+0000","last_undegraded":"2024-09-21T14:02:26.137294+0000","last_fullsized":"2024-09-21T14:02:26.137294+0000","mapping_epoch":345,"log_start":"311'9787","ondisk_log_start":"311'9787","created":260,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4349,"log_dups_size":0,"ondisk_log_size":4349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83361792,"num_objects":1272,"num_object_clones":0,"num_object_copies":3816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14136,"num_write_kb":493056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"300'32","reported_seq":332,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:23.390225+0000","last_change":"2024-09-21T14:00:47.130089+0000","last_active":"2024-09-21T14:02:23.390225+0000","last_peered":"2024-09-21T14:02:23.390225+0000","last_clean":"2024-09-21T14:02:23.390225+0000","last_became_active":"2024-09-21T13:59:32.204624+0000","last_became_peered":"2024-09-21T13:59:32.204624+0000","last_unstale":"2024-09-21T14:02:23.390225+0000","last_undegraded":"2024-09-21T14:02:23.390225+0000","last_fullsized":"2024-09-21T14:02:23.390225+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'32","last_scrub_stamp":"2024-09-21T14:00:47.129789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:00:47.129789+0000","objects_scrubbed":8,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:15.616969+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":639,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.091518+0000","last_change":"2024-09-21T14:01:41.190537+0000","last_active":"2024-09-21T14:02:25.091518+0000","last_peered":"2024-09-21T14:02:25.091518+0000","last_clean":"2024-09-21T14:02:25.091518+0000","last_became_active":"2024-09-21T14:01:41.190210+0000","last_became_peered":"2024-09-21T14:01:41.190210+0000","last_unstale":"2024-09-21T14:02:25.091518+0000","last_undegraded":"2024-09-21T14:02:25.091518+0000","last_fullsized":"2024-09-21T14:02:25.091518+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:57.772975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:57.772975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"75'79","reported_seq":700,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.104267+0000","last_change":"2024-09-21T14:00:39.339360+0000","last_active":"2024-09-21T14:02:25.104267+0000","last_peered":"2024-09-21T14:02:25.104267+0000","last_clean":"2024-09-21T14:02:25.104267+0000","last_became_active":"2024-09-21T14:00:39.339077+0000","last_became_peered":"2024-09-21T14:00:39.339077+0000","last_unstale":"2024-09-21T14:02:25.104267+0000","last_undegraded":"2024-09-21T14:02:25.104267+0000","last_fullsized":"2024-09-21T14:02:25.104267+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:02:42.782456+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":19,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53669,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":614,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.107183+0000","last_change":"2024-09-21T14:01:41.182186+0000","last_active":"2024-09-21T14:02:25.107183+0000","last_peered":"2024-09-21T14:02:25.107183+0000","last_clean":"2024-09-21T14:02:25.107183+0000","last_became_active":"2024-09-21T14:01:41.181899+0000","last_became_peered":"2024-09-21T14:01:41.181899+0000","last_unstale":"2024-09-21T14:02:25.107183+0000","last_undegraded":"2024-09-21T14:02:25.107183+0000","last_fullsized":"2024-09-21T14:02:25.107183+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:15.700409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:15.700409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":539,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.100143+0000","last_change":"2024-09-21T14:01:41.196509+0000","last_active":"2024-09-21T14:02:25.100143+0000","last_peered":"2024-09-21T14:02:25.100143+0000","last_clean":"2024-09-21T14:02:25.100143+0000","last_became_active":"2024-09-21T14:01:41.196243+0000","last_became_peered":"2024-09-21T14:01:41.196243+0000","last_unstale":"2024-09-21T14:02:25.100143+0000","last_undegraded":"2024-09-21T14:02:25.100143+0000","last_fullsized":"2024-09-21T14:02:25.100143+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:21.153018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:21.153018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"370'13897","reported_seq":23801,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:26.152764+0000","last_change":"2024-09-21T14:00:42.107898+0000","last_active":"2024-09-21T14:02:26.152764+0000","last_peered":"2024-09-21T14:02:26.152764+0000","last_clean":"2024-09-21T14:02:26.152764+0000","last_became_active":"2024-09-21T14:00:42.085387+0000","last_became_peered":"2024-09-21T14:00:42.085387+0000","last_unstale":"2024-09-21T14:02:26.152764+0000","last_undegraded":"2024-09-21T14:02:26.152764+0000","last_fullsized":"2024-09-21T14:02:26.152764+0000","mapping_epoch":321,"log_start":"310'9312","ondisk_log_start":"310'9312","created":260,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4585,"log_dups_size":0,"ondisk_log_size":4585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83296256,"num_objects":1271,"num_object_clones":0,"num_object_copies":3813,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13897,"num_write_kb":485376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4827,"num_bytes_recovered":316342272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"290'19","reported_seq":715,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:23.384376+0000","last_change":"2024-09-21T13:58:17.811300+0000","last_active":"2024-09-21T14:02:23.384376+0000","last_peered":"2024-09-21T14:02:23.384376+0000","last_clean":"2024-09-21T14:02:23.384376+0000","last_became_active":"2024-09-21T13:58:17.776310+0000","last_became_peered":"2024-09-21T13:58:17.776310+0000","last_unstale":"2024-09-21T14:02:23.384376+0000","last_undegraded":"2024-09-21T14:02:23.384376+0000","last_fullsized":"2024-09-21T14:02:23.384376+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:54:16.451588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:16.451588+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":686,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:23.382567+0000","last_change":"2024-09-21T14:02:03.437581+0000","last_active":"2024-09-21T14:02:23.382567+0000","last_peered":"2024-09-21T14:02:23.382567+0000","last_clean":"2024-09-21T14:02:23.382567+0000","last_became_active":"2024-09-21T13:59:32.223247+0000","last_became_peered":"2024-09-21T13:59:32.223247+0000","last_unstale":"2024-09-21T14:02:23.382567+0000","last_undegraded":"2024-09-21T14:02:23.382567+0000","last_fullsized":"2024-09-21T14:02:23.382567+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:03.437510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:02:03.437510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:14.620200+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":607,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.100047+0000","last_change":"2024-09-21T14:01:41.194538+0000","last_active":"2024-09-21T14:02:25.100047+0000","last_peered":"2024-09-21T14:02:25.100047+0000","last_clean":"2024-09-21T14:02:25.100047+0000","last_became_active":"2024-09-21T14:01:41.193941+0000","last_became_peered":"2024-09-21T14:01:41.193941+0000","last_unstale":"2024-09-21T14:02:25.100047+0000","last_undegraded":"2024-09-21T14:02:25.100047+0000","last_fullsized":"2024-09-21T14:02:25.100047+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:34.299722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:55:34.299722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":617,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.092475+0000","last_change":"2024-09-21T14:01:41.190501+0000","last_active":"2024-09-21T14:02:25.092475+0000","last_peered":"2024-09-21T14:02:25.092475+0000","last_clean":"2024-09-21T14:02:25.092475+0000","last_became_active":"2024-09-21T14:01:41.190192+0000","last_became_peered":"2024-09-21T14:01:41.190192+0000","last_unstale":"2024-09-21T14:02:25.092475+0000","last_undegraded":"2024-09-21T14:02:25.092475+0000","last_fullsized":"2024-09-21T14:02:25.092475+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:58.952218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:54:58.952218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:18.255572+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"75'79","reported_seq":80,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.099233+0000","last_change":"2024-09-21T14:01:41.189146+0000","last_active":"2024-09-21T14:02:25.099233+0000","last_peered":"2024-09-21T14:02:25.099233+0000","last_clean":"2024-09-21T14:02:25.099233+0000","last_became_active":"2024-09-21T14:01:41.188685+0000","last_became_peered":"2024-09-21T14:01:41.188685+0000","last_unstale":"2024-09-21T14:02:25.099233+0000","last_undegraded":"2024-09-21T14:02:25.099233+0000","last_fullsized":"2024-09-21T14:02:25.099233+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:02:42.589731+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.0","version":"369'13701","reported_seq":17870,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:24.658020+0000","last_change":"2024-09-21T13:59:24.817101+0000","last_active":"2024-09-21T14:02:24.658020+0000","last_peered":"2024-09-21T14:02:24.658020+0000","last_clean":"2024-09-21T14:02:24.658020+0000","last_became_active":"2024-09-21T13:58:36.723110+0000","last_became_peered":"2024-09-21T13:58:36.723110+0000","last_unstale":"2024-09-21T14:02:24.658020+0000","last_undegraded":"2024-09-21T14:02:24.658020+0000","last_fullsized":"2024-09-21T14:02:24.658020+0000","mapping_epoch":279,"log_start":"310'9081","ondisk_log_start":"310'9081","created":260,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'7035","last_scrub_stamp":"2024-09-21T13:59:24.816903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:59:24.816903+0000","objects_scrubbed":6955,"log_size":4620,"log_dups_size":0,"ondisk_log_size":4620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2446,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89849856,"num_objects":1371,"num_object_clones":0,"num_object_copies":4113,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13701,"num_write_kb":482304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1924,"num_bytes_recovered":126091264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"290'31","reported_seq":711,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:23.383651+0000","last_change":"2024-09-21T14:01:03.541547+0000","last_active":"2024-09-21T14:02:23.383651+0000","last_peered":"2024-09-21T14:02:23.383651+0000","last_clean":"2024-09-21T14:02:23.383651+0000","last_became_active":"2024-09-21T13:52:04.313218+0000","last_became_peered":"2024-09-21T13:52:04.313218+0000","last_unstale":"2024-09-21T14:02:23.383651+0000","last_undegraded":"2024-09-21T14:02:23.383651+0000","last_fullsized":"2024-09-21T14:02:23.383651+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'31","last_scrub_stamp":"2024-09-21T14:01:03.541432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:03.541432+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":227,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:23.388294+0000","last_change":"2024-09-21T14:01:29.924408+0000","last_active":"2024-09-21T14:02:23.388294+0000","last_peered":"2024-09-21T14:02:23.388294+0000","last_clean":"2024-09-21T14:02:23.388294+0000","last_became_active":"2024-09-21T13:58:17.762520+0000","last_became_peered":"2024-09-21T13:58:17.762520+0000","last_unstale":"2024-09-21T14:02:23.388294+0000","last_undegraded":"2024-09-21T14:02:23.388294+0000","last_fullsized":"2024-09-21T14:02:23.388294+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:29.924331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:01:29.924331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:43.977176+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":640,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:23.377743+0000","last_change":"2024-09-21T13:58:52.030800+0000","last_active":"2024-09-21T14:02:23.377743+0000","last_peered":"2024-09-21T14:02:23.377743+0000","last_clean":"2024-09-21T14:02:23.377743+0000","last_became_active":"2024-09-21T13:49:02.067472+0000","last_became_peered":"2024-09-21T13:49:02.067472+0000","last_unstale":"2024-09-21T14:02:23.377743+0000","last_undegraded":"2024-09-21T14:02:23.377743+0000","last_fullsized":"2024-09-21T14:02:23.377743+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:52.030718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:52.030718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":521,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.307544+0000","last_change":"2024-09-21T13:57:12.683321+0000","last_active":"2024-09-21T14:02:17.307544+0000","last_peered":"2024-09-21T14:02:17.307544+0000","last_clean":"2024-09-21T14:02:17.307544+0000","last_became_active":"2024-09-21T13:57:12.682979+0000","last_became_peered":"2024-09-21T13:57:12.682979+0000","last_unstale":"2024-09-21T14:02:17.307544+0000","last_undegraded":"2024-09-21T14:02:17.307544+0000","last_fullsized":"2024-09-21T14:02:17.307544+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:18.027944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:18.027944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:20.920553+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"75'79","reported_seq":705,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.106311+0000","last_change":"2024-09-21T14:00:41.140108+0000","last_active":"2024-09-21T14:02:25.106311+0000","last_peered":"2024-09-21T14:02:25.106311+0000","last_clean":"2024-09-21T14:02:25.106311+0000","last_became_active":"2024-09-21T14:00:41.139315+0000","last_became_peered":"2024-09-21T14:00:41.139315+0000","last_unstale":"2024-09-21T14:02:25.106311+0000","last_undegraded":"2024-09-21T14:02:25.106311+0000","last_fullsized":"2024-09-21T14:02:25.106311+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1","version":"370'13798","reported_seq":26071,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:26.151007+0000","last_change":"2024-09-21T14:01:10.113546+0000","last_active":"2024-09-21T14:02:26.151007+0000","last_peered":"2024-09-21T14:02:26.151007+0000","last_clean":"2024-09-21T14:02:26.151007+0000","last_became_active":"2024-09-21T14:01:10.092192+0000","last_became_peered":"2024-09-21T14:01:10.092192+0000","last_unstale":"2024-09-21T14:02:26.151007+0000","last_undegraded":"2024-09-21T14:02:26.151007+0000","last_fullsized":"2024-09-21T14:02:26.151007+0000","mapping_epoch":333,"log_start":"310'9445","ondisk_log_start":"310'9445","created":260,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4353,"log_dups_size":0,"ondisk_log_size":4353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85852160,"num_objects":1310,"num_object_clones":0,"num_object_copies":3930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13798,"num_write_kb":483456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5999,"num_bytes_recovered":393150464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"290'20","reported_seq":730,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:23.383288+0000","last_change":"2024-09-21T13:58:17.789407+0000","last_active":"2024-09-21T14:02:23.383288+0000","last_peered":"2024-09-21T14:02:23.383288+0000","last_clean":"2024-09-21T14:02:23.383288+0000","last_became_active":"2024-09-21T13:58:17.765427+0000","last_became_peered":"2024-09-21T13:58:17.765427+0000","last_unstale":"2024-09-21T14:02:23.383288+0000","last_undegraded":"2024-09-21T14:02:23.383288+0000","last_fullsized":"2024-09-21T14:02:23.383288+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:58:13.464196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:13.464196+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:16.467667+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":574,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.091040+0000","last_change":"2024-09-21T13:57:36.707211+0000","last_active":"2024-09-21T14:02:25.091040+0000","last_peered":"2024-09-21T14:02:25.091040+0000","last_clean":"2024-09-21T14:02:25.091040+0000","last_became_active":"2024-09-21T13:57:12.704203+0000","last_became_peered":"2024-09-21T13:57:12.704203+0000","last_unstale":"2024-09-21T14:02:25.091040+0000","last_undegraded":"2024-09-21T14:02:25.091040+0000","last_fullsized":"2024-09-21T14:02:25.091040+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:36.707148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:36.707148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"346'14","reported_seq":695,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.306363+0000","last_change":"2024-09-21T14:01:41.186960+0000","last_active":"2024-09-21T14:02:17.306363+0000","last_peered":"2024-09-21T14:02:17.306363+0000","last_clean":"2024-09-21T14:02:17.306363+0000","last_became_active":"2024-09-21T14:01:41.186564+0000","last_became_peered":"2024-09-21T14:01:41.186564+0000","last_unstale":"2024-09-21T14:02:17.306363+0000","last_undegraded":"2024-09-21T14:02:17.306363+0000","last_fullsized":"2024-09-21T14:02:17.306363+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'12","last_scrub_stamp":"2024-09-21T13:59:38.738120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:38.738120+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:44.832105+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"68'1","reported_seq":658,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:23.383354+0000","last_change":"2024-09-21T13:59:31.392186+0000","last_active":"2024-09-21T14:02:23.383354+0000","last_peered":"2024-09-21T14:02:23.383354+0000","last_clean":"2024-09-21T14:02:23.383354+0000","last_became_active":"2024-09-21T13:52:04.313210+0000","last_became_peered":"2024-09-21T13:52:04.313210+0000","last_unstale":"2024-09-21T14:02:23.383354+0000","last_undegraded":"2024-09-21T14:02:23.383354+0000","last_fullsized":"2024-09-21T14:02:23.383354+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T13:59:31.392105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:31.392105+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"370'13670","reported_seq":13889,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.833367+0000","last_change":"2024-09-21T14:01:41.341139+0000","last_active":"2024-09-21T14:02:25.833367+0000","last_peered":"2024-09-21T14:02:25.833367+0000","last_clean":"2024-09-21T14:02:25.833367+0000","last_became_active":"2024-09-21T14:01:41.199821+0000","last_became_peered":"2024-09-21T14:01:41.199821+0000","last_unstale":"2024-09-21T14:02:25.833367+0000","last_undegraded":"2024-09-21T14:02:25.833367+0000","last_fullsized":"2024-09-21T14:02:25.833367+0000","mapping_epoch":345,"log_start":"307'8699","ondisk_log_start":"307'8699","created":260,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4971,"log_dups_size":0,"ondisk_log_size":4971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79953920,"num_objects":1220,"num_object_clones":0,"num_object_copies":3660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13670,"num_write_kb":476480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"290'15","reported_seq":685,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:23.391769+0000","last_change":"2024-09-21T13:58:50.498597+0000","last_active":"2024-09-21T14:02:23.391769+0000","last_peered":"2024-09-21T14:02:23.391769+0000","last_clean":"2024-09-21T14:02:23.391769+0000","last_became_active":"2024-09-21T13:49:00.053016+0000","last_became_peered":"2024-09-21T13:49:00.053016+0000","last_unstale":"2024-09-21T14:02:23.391769+0000","last_undegraded":"2024-09-21T14:02:23.391769+0000","last_fullsized":"2024-09-21T14:02:23.391769+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:58:50.498369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:50.498369+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":619,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.099651+0000","last_change":"2024-09-21T14:01:41.192717+0000","last_active":"2024-09-21T14:02:25.099651+0000","last_peered":"2024-09-21T14:02:25.099651+0000","last_clean":"2024-09-21T14:02:25.099651+0000","last_became_active":"2024-09-21T14:01:41.192443+0000","last_became_peered":"2024-09-21T14:01:41.192443+0000","last_unstale":"2024-09-21T14:02:25.099651+0000","last_undegraded":"2024-09-21T14:02:25.099651+0000","last_fullsized":"2024-09-21T14:02:25.099651+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:53.894506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:53.894506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":597,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:23.382605+0000","last_change":"2024-09-21T14:01:27.739999+0000","last_active":"2024-09-21T14:02:23.382605+0000","last_peered":"2024-09-21T14:02:23.382605+0000","last_clean":"2024-09-21T14:02:23.382605+0000","last_became_active":"2024-09-21T13:57:12.680978+0000","last_became_peered":"2024-09-21T13:57:12.680978+0000","last_unstale":"2024-09-21T14:02:23.382605+0000","last_undegraded":"2024-09-21T14:02:23.382605+0000","last_fullsized":"2024-09-21T14:02:23.382605+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:27.739921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:01:27.739921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:50.205235+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":549,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.102006+0000","last_change":"2024-09-21T13:59:32.224001+0000","last_active":"2024-09-21T14:02:25.102006+0000","last_peered":"2024-09-21T14:02:25.102006+0000","last_clean":"2024-09-21T14:02:25.102006+0000","last_became_active":"2024-09-21T13:59:32.221628+0000","last_became_peered":"2024-09-21T13:59:32.221628+0000","last_unstale":"2024-09-21T14:02:25.102006+0000","last_undegraded":"2024-09-21T14:02:25.102006+0000","last_fullsized":"2024-09-21T14:02:25.102006+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:13.698091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:13.698091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"370'14253","reported_seq":22551,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.829887+0000","last_change":"2024-09-21T14:02:09.252872+0000","last_active":"2024-09-21T14:02:25.829887+0000","last_peered":"2024-09-21T14:02:25.829887+0000","last_clean":"2024-09-21T14:02:25.829887+0000","last_became_active":"2024-09-21T14:02:09.218449+0000","last_became_peered":"2024-09-21T14:02:09.218449+0000","last_unstale":"2024-09-21T14:02:25.829887+0000","last_undegraded":"2024-09-21T14:02:25.829887+0000","last_fullsized":"2024-09-21T14:02:25.829887+0000","mapping_epoch":362,"log_start":"308'9255","ondisk_log_start":"308'9255","created":260,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4998,"log_dups_size":0,"ondisk_log_size":4998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88932352,"num_objects":1357,"num_object_clones":0,"num_object_copies":4071,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14253,"num_write_kb":499520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4082,"num_bytes_recovered":226033664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"290'35","reported_seq":293,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:23.388419+0000","last_change":"2024-09-21T14:01:02.006619+0000","last_active":"2024-09-21T14:02:23.388419+0000","last_peered":"2024-09-21T14:02:23.388419+0000","last_clean":"2024-09-21T14:02:23.388419+0000","last_became_active":"2024-09-21T13:59:32.204184+0000","last_became_peered":"2024-09-21T13:59:32.204184+0000","last_unstale":"2024-09-21T14:02:23.388419+0000","last_undegraded":"2024-09-21T14:02:23.388419+0000","last_fullsized":"2024-09-21T14:02:23.388419+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'35","last_scrub_stamp":"2024-09-21T14:01:02.006504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:02.006504+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:22.557193+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":641,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.108535+0000","last_change":"2024-09-21T14:01:41.184937+0000","last_active":"2024-09-21T14:02:25.108535+0000","last_peered":"2024-09-21T14:02:25.108535+0000","last_clean":"2024-09-21T14:02:25.108535+0000","last_became_active":"2024-09-21T14:01:41.184567+0000","last_became_peered":"2024-09-21T14:01:41.184567+0000","last_unstale":"2024-09-21T14:02:25.108535+0000","last_undegraded":"2024-09-21T14:02:25.108535+0000","last_fullsized":"2024-09-21T14:02:25.108535+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:21.587493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:21.587493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":615,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.302901+0000","last_change":"2024-09-21T14:01:41.182250+0000","last_active":"2024-09-21T14:02:17.302901+0000","last_peered":"2024-09-21T14:02:17.302901+0000","last_clean":"2024-09-21T14:02:17.302901+0000","last_became_active":"2024-09-21T14:01:41.181911+0000","last_became_peered":"2024-09-21T14:01:41.181911+0000","last_unstale":"2024-09-21T14:02:17.302901+0000","last_undegraded":"2024-09-21T14:02:17.302901+0000","last_fullsized":"2024-09-21T14:02:17.302901+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:55.588554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:55.588554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:41.898392+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":568,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.102026+0000","last_change":"2024-09-21T13:59:32.223123+0000","last_active":"2024-09-21T14:02:25.102026+0000","last_peered":"2024-09-21T14:02:25.102026+0000","last_clean":"2024-09-21T14:02:25.102026+0000","last_became_active":"2024-09-21T13:59:32.219332+0000","last_became_peered":"2024-09-21T13:59:32.219332+0000","last_unstale":"2024-09-21T14:02:25.102026+0000","last_undegraded":"2024-09-21T14:02:25.102026+0000","last_fullsized":"2024-09-21T14:02:25.102026+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:19.537177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:19.537177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"369'13506","reported_seq":13737,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:24.680500+0000","last_change":"2024-09-21T13:57:53.515705+0000","last_active":"2024-09-21T14:02:24.680500+0000","last_peered":"2024-09-21T14:02:24.680500+0000","last_clean":"2024-09-21T14:02:24.680500+0000","last_became_active":"2024-09-21T13:57:53.515313+0000","last_became_peered":"2024-09-21T13:57:53.515313+0000","last_unstale":"2024-09-21T14:02:24.680500+0000","last_undegraded":"2024-09-21T14:02:24.680500+0000","last_fullsized":"2024-09-21T14:02:24.680500+0000","mapping_epoch":260,"log_start":"309'8879","ondisk_log_start":"309'8879","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4627,"log_dups_size":0,"ondisk_log_size":4627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89260032,"num_objects":1362,"num_object_clones":0,"num_object_copies":4086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13506,"num_write_kb":475776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"290'18","reported_seq":703,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:23.382436+0000","last_change":"2024-09-21T13:59:28.340494+0000","last_active":"2024-09-21T14:02:23.382436+0000","last_peered":"2024-09-21T14:02:23.382436+0000","last_clean":"2024-09-21T14:02:23.382436+0000","last_became_active":"2024-09-21T13:57:12.680627+0000","last_became_peered":"2024-09-21T13:57:12.680627+0000","last_unstale":"2024-09-21T14:02:23.382436+0000","last_undegraded":"2024-09-21T14:02:23.382436+0000","last_fullsized":"2024-09-21T14:02:23.382436+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'18","last_scrub_stamp":"2024-09-21T13:59:28.340415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:28.340415+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:11.383693+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":596,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.108204+0000","last_change":"2024-09-21T13:59:32.201126+0000","last_active":"2024-09-21T14:02:25.108204+0000","last_peered":"2024-09-21T14:02:25.108204+0000","last_clean":"2024-09-21T14:02:25.108204+0000","last_became_active":"2024-09-21T13:59:32.200674+0000","last_became_peered":"2024-09-21T13:59:32.200674+0000","last_unstale":"2024-09-21T14:02:25.108204+0000","last_undegraded":"2024-09-21T14:02:25.108204+0000","last_fullsized":"2024-09-21T14:02:25.108204+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:21.582325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:21.582325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":569,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.302708+0000","last_change":"2024-09-21T13:58:11.377940+0000","last_active":"2024-09-21T14:02:17.302708+0000","last_peered":"2024-09-21T14:02:17.302708+0000","last_clean":"2024-09-21T14:02:17.302708+0000","last_became_active":"2024-09-21T13:52:04.322147+0000","last_became_peered":"2024-09-21T13:52:04.322147+0000","last_unstale":"2024-09-21T14:02:17.302708+0000","last_undegraded":"2024-09-21T14:02:17.302708+0000","last_fullsized":"2024-09-21T14:02:17.302708+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:11.377803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:11.377803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:25.932593+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":582,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.091393+0000","last_change":"2024-09-21T13:59:32.207006+0000","last_active":"2024-09-21T14:02:25.091393+0000","last_peered":"2024-09-21T14:02:25.091393+0000","last_clean":"2024-09-21T14:02:25.091393+0000","last_became_active":"2024-09-21T13:59:32.203767+0000","last_became_peered":"2024-09-21T13:59:32.203767+0000","last_unstale":"2024-09-21T14:02:25.091393+0000","last_undegraded":"2024-09-21T14:02:25.091393+0000","last_fullsized":"2024-09-21T14:02:25.091393+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:09.040062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:55:09.040062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"370'14334","reported_seq":18025,"reported_epoch":370,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-21T14:02:25.834753+0000","last_change":"2024-09-21T14:01:45.658247+0000","last_active":"2024-09-21T14:02:25.834753+0000","last_peered":"2024-09-21T14:02:25.834753+0000","last_clean":"2024-09-21T13:58:23.785212+0000","last_became_active":"2024-09-21T14:01:44.478305+0000","last_became_peered":"2024-09-21T14:01:44.478305+0000","last_unstale":"2024-09-21T14:02:25.834753+0000","last_undegraded":"2024-09-21T14:01:44.440463+0000","last_fullsized":"2024-09-21T14:01:42.170394+0000","mapping_epoch":346,"log_start":"309'9386","ondisk_log_start":"309'9386","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4948,"log_dups_size":0,"ondisk_log_size":4948,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84934656,"num_objects":1296,"num_object_clones":0,"num_object_copies":3888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1238,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14334,"num_write_kb":500160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1794,"num_bytes_recovered":116129792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,7],"avail_no_missing":["5","0"],"object_location_counts":[{"shards":"0,5","objects":1238},{"shards":"5,7","objects":58}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"290'27","reported_seq":695,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.099310+0000","last_change":"2024-09-21T14:01:41.193143+0000","last_active":"2024-09-21T14:02:25.099310+0000","last_peered":"2024-09-21T14:02:25.099310+0000","last_clean":"2024-09-21T14:02:25.099310+0000","last_became_active":"2024-09-21T14:01:41.192481+0000","last_became_peered":"2024-09-21T14:01:41.192481+0000","last_unstale":"2024-09-21T14:02:25.099310+0000","last_undegraded":"2024-09-21T14:02:25.099310+0000","last_fullsized":"2024-09-21T14:02:25.099310+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'27","last_scrub_stamp":"2024-09-21T13:59:33.058859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:33.058859+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":568,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.112007+0000","last_change":"2024-09-21T13:59:32.203889+0000","last_active":"2024-09-21T14:02:25.112007+0000","last_peered":"2024-09-21T14:02:25.112007+0000","last_clean":"2024-09-21T14:02:25.112007+0000","last_became_active":"2024-09-21T13:59:32.203172+0000","last_became_peered":"2024-09-21T13:59:32.203172+0000","last_unstale":"2024-09-21T14:02:25.112007+0000","last_undegraded":"2024-09-21T14:02:25.112007+0000","last_fullsized":"2024-09-21T14:02:25.112007+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T13:55:03.879706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:55:03.879706+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"300'14","reported_seq":777,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.101853+0000","last_change":"2024-09-21T13:59:32.220898+0000","last_active":"2024-09-21T14:02:25.101853+0000","last_peered":"2024-09-21T14:02:25.101853+0000","last_clean":"2024-09-21T14:02:25.101853+0000","last_became_active":"2024-09-21T13:59:32.206956+0000","last_became_peered":"2024-09-21T13:59:32.206956+0000","last_unstale":"2024-09-21T14:02:25.101853+0000","last_undegraded":"2024-09-21T14:02:25.101853+0000","last_fullsized":"2024-09-21T14:02:25.101853+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'11","last_scrub_stamp":"2024-09-21T13:56:31.616003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:56:31.616003+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":643,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.099459+0000","last_change":"2024-09-21T14:01:41.248408+0000","last_active":"2024-09-21T14:02:25.099459+0000","last_peered":"2024-09-21T14:02:25.099459+0000","last_clean":"2024-09-21T14:02:25.099459+0000","last_became_active":"2024-09-21T14:01:41.246106+0000","last_became_peered":"2024-09-21T14:01:41.246106+0000","last_unstale":"2024-09-21T14:02:25.099459+0000","last_undegraded":"2024-09-21T14:02:25.099459+0000","last_fullsized":"2024-09-21T14:02:25.099459+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:39.258659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:39.258659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":230,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:23.388076+0000","last_change":"2024-09-21T14:01:50.733055+0000","last_active":"2024-09-21T14:02:23.388076+0000","last_peered":"2024-09-21T14:02:23.388076+0000","last_clean":"2024-09-21T14:02:23.388076+0000","last_became_active":"2024-09-21T14:01:41.197501+0000","last_became_peered":"2024-09-21T14:01:41.197501+0000","last_unstale":"2024-09-21T14:02:23.388076+0000","last_undegraded":"2024-09-21T14:02:23.388076+0000","last_fullsized":"2024-09-21T14:02:23.388076+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:50.733007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:01:50.733007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:08.367766+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"290'12","reported_seq":617,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.094925+0000","last_change":"2024-09-21T13:59:32.247422+0000","last_active":"2024-09-21T14:02:25.094925+0000","last_peered":"2024-09-21T14:02:25.094925+0000","last_clean":"2024-09-21T14:02:25.094925+0000","last_became_active":"2024-09-21T13:59:32.211306+0000","last_became_peered":"2024-09-21T13:59:32.211306+0000","last_unstale":"2024-09-21T14:02:25.094925+0000","last_undegraded":"2024-09-21T14:02:25.094925+0000","last_fullsized":"2024-09-21T14:02:25.094925+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'6","last_scrub_stamp":"2024-09-21T13:55:12.066873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:55:12.066873+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":575,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.099869+0000","last_change":"2024-09-21T14:01:41.199742+0000","last_active":"2024-09-21T14:02:25.099869+0000","last_peered":"2024-09-21T14:02:25.099869+0000","last_clean":"2024-09-21T14:02:25.099869+0000","last_became_active":"2024-09-21T14:01:41.199321+0000","last_became_peered":"2024-09-21T14:01:41.199321+0000","last_unstale":"2024-09-21T14:02:25.099869+0000","last_undegraded":"2024-09-21T14:02:25.099869+0000","last_fullsized":"2024-09-21T14:02:25.099869+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:37.106238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:37.106238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":614,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.108286+0000","last_change":"2024-09-21T13:59:32.222277+0000","last_active":"2024-09-21T14:02:25.108286+0000","last_peered":"2024-09-21T14:02:25.108286+0000","last_clean":"2024-09-21T14:02:25.108286+0000","last_became_active":"2024-09-21T13:59:32.219525+0000","last_became_peered":"2024-09-21T13:59:32.219525+0000","last_unstale":"2024-09-21T14:02:25.108286+0000","last_undegraded":"2024-09-21T14:02:25.108286+0000","last_fullsized":"2024-09-21T14:02:25.108286+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:02.838181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:55:02.838181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"290'15","reported_seq":580,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.094849+0000","last_change":"2024-09-21T13:59:32.238689+0000","last_active":"2024-09-21T14:02:25.094849+0000","last_peered":"2024-09-21T14:02:25.094849+0000","last_clean":"2024-09-21T14:02:25.094849+0000","last_became_active":"2024-09-21T13:59:32.209166+0000","last_became_peered":"2024-09-21T13:59:32.209166+0000","last_unstale":"2024-09-21T14:02:25.094849+0000","last_undegraded":"2024-09-21T14:02:25.094849+0000","last_fullsized":"2024-09-21T14:02:25.094849+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:55:04.990842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:55:04.990842+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:21.278052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"61'1","reported_seq":632,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.106023+0000","last_change":"2024-09-21T13:59:32.222826+0000","last_active":"2024-09-21T14:02:25.106023+0000","last_peered":"2024-09-21T14:02:25.106023+0000","last_clean":"2024-09-21T14:02:25.106023+0000","last_became_active":"2024-09-21T13:59:32.219515+0000","last_became_peered":"2024-09-21T13:59:32.219515+0000","last_unstale":"2024-09-21T14:02:25.106023+0000","last_undegraded":"2024-09-21T14:02:25.106023+0000","last_fullsized":"2024-09-21T14:02:25.106023+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:58:07.711028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:07.711028+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":565,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.094408+0000","last_change":"2024-09-21T13:59:32.211754+0000","last_active":"2024-09-21T14:02:25.094408+0000","last_peered":"2024-09-21T14:02:25.094408+0000","last_clean":"2024-09-21T14:02:25.094408+0000","last_became_active":"2024-09-21T13:59:32.210586+0000","last_became_peered":"2024-09-21T13:59:32.210586+0000","last_unstale":"2024-09-21T14:02:25.094408+0000","last_undegraded":"2024-09-21T14:02:25.094408+0000","last_fullsized":"2024-09-21T14:02:25.094408+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:28.014760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:28.014760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:17.280712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":611,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T14:02:17.302445+0000","last_change":"2024-09-21T14:01:26.918280+0000","last_active":"2024-09-21T14:02:17.302445+0000","last_peered":"2024-09-21T14:02:17.302445+0000","last_clean":"2024-09-21T14:02:17.302445+0000","last_became_active":"2024-09-21T13:59:32.205803+0000","last_became_peered":"2024-09-21T13:59:32.205803+0000","last_unstale":"2024-09-21T14:02:17.302445+0000","last_undegraded":"2024-09-21T14:02:17.302445+0000","last_fullsized":"2024-09-21T14:02:17.302445+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:26.918225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:01:26.918225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:27.514271+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"290'15","reported_seq":620,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.109202+0000","last_change":"2024-09-21T14:01:41.189676+0000","last_active":"2024-09-21T14:02:25.109202+0000","last_peered":"2024-09-21T14:02:25.109202+0000","last_clean":"2024-09-21T14:02:25.109202+0000","last_became_active":"2024-09-21T14:01:41.187904+0000","last_became_peered":"2024-09-21T14:01:41.187904+0000","last_unstale":"2024-09-21T14:02:25.109202+0000","last_undegraded":"2024-09-21T14:02:25.109202+0000","last_fullsized":"2024-09-21T14:02:25.109202+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:57:24.124604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:57:24.124604+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":501,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.106499+0000","last_change":"2024-09-21T13:59:32.212325+0000","last_active":"2024-09-21T14:02:25.106499+0000","last_peered":"2024-09-21T14:02:25.106499+0000","last_clean":"2024-09-21T14:02:25.106499+0000","last_became_active":"2024-09-21T13:59:32.208749+0000","last_became_peered":"2024-09-21T13:59:32.208749+0000","last_unstale":"2024-09-21T14:02:25.106499+0000","last_undegraded":"2024-09-21T14:02:25.106499+0000","last_fullsized":"2024-09-21T14:02:25.106499+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:20.574745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:20.574745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"300'14","reported_seq":710,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.092566+0000","last_change":"2024-09-21T13:59:32.208253+0000","last_active":"2024-09-21T14:02:25.092566+0000","last_peered":"2024-09-21T14:02:25.092566+0000","last_clean":"2024-09-21T14:02:25.092566+0000","last_became_active":"2024-09-21T13:59:32.206926+0000","last_became_peered":"2024-09-21T13:59:32.206926+0000","last_unstale":"2024-09-21T14:02:25.092566+0000","last_undegraded":"2024-09-21T14:02:25.092566+0000","last_fullsized":"2024-09-21T14:02:25.092566+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'10","last_scrub_stamp":"2024-09-21T13:54:20.946782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:20.946782+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:10.314285+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":665,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:23.384129+0000","last_change":"2024-09-21T14:01:41.200564+0000","last_active":"2024-09-21T14:02:23.384129+0000","last_peered":"2024-09-21T14:02:23.384129+0000","last_clean":"2024-09-21T14:02:23.384129+0000","last_became_active":"2024-09-21T14:01:41.200113+0000","last_became_peered":"2024-09-21T14:01:41.200113+0000","last_unstale":"2024-09-21T14:02:23.384129+0000","last_undegraded":"2024-09-21T14:02:23.384129+0000","last_fullsized":"2024-09-21T14:02:23.384129+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:54.805619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:54.805619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:48.557673+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"290'20","reported_seq":616,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.092391+0000","last_change":"2024-09-21T13:59:32.236530+0000","last_active":"2024-09-21T14:02:25.092391+0000","last_peered":"2024-09-21T14:02:25.092391+0000","last_clean":"2024-09-21T14:02:25.092391+0000","last_became_active":"2024-09-21T13:59:32.208332+0000","last_became_peered":"2024-09-21T13:59:32.208332+0000","last_unstale":"2024-09-21T14:02:25.092391+0000","last_undegraded":"2024-09-21T14:02:25.092391+0000","last_fullsized":"2024-09-21T14:02:25.092391+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:57:37.671391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:57:37.671391+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":608,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.106902+0000","last_change":"2024-09-21T13:59:32.212737+0000","last_active":"2024-09-21T14:02:25.106902+0000","last_peered":"2024-09-21T14:02:25.106902+0000","last_clean":"2024-09-21T14:02:25.106902+0000","last_became_active":"2024-09-21T13:59:32.209804+0000","last_became_peered":"2024-09-21T13:59:32.209804+0000","last_unstale":"2024-09-21T14:02:25.106902+0000","last_undegraded":"2024-09-21T14:02:25.106902+0000","last_fullsized":"2024-09-21T14:02:25.106902+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:25.571256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:25.571256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"346'10","reported_seq":710,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.109094+0000","last_change":"2024-09-21T14:01:41.207030+0000","last_active":"2024-09-21T14:02:25.109094+0000","last_peered":"2024-09-21T14:02:25.109094+0000","last_clean":"2024-09-21T14:02:25.109094+0000","last_became_active":"2024-09-21T14:01:41.206303+0000","last_became_peered":"2024-09-21T14:01:41.206303+0000","last_unstale":"2024-09-21T14:02:25.109094+0000","last_undegraded":"2024-09-21T14:02:25.109094+0000","last_fullsized":"2024-09-21T14:02:25.109094+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:59:32.104224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:32.104224+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":505,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.108010+0000","last_change":"2024-09-21T13:59:32.213015+0000","last_active":"2024-09-21T14:02:25.108010+0000","last_peered":"2024-09-21T14:02:25.108010+0000","last_clean":"2024-09-21T14:02:25.108010+0000","last_became_active":"2024-09-21T13:59:32.211472+0000","last_became_peered":"2024-09-21T13:59:32.211472+0000","last_unstale":"2024-09-21T14:02:25.108010+0000","last_undegraded":"2024-09-21T14:02:25.108010+0000","last_fullsized":"2024-09-21T14:02:25.108010+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:22.647191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:22.647191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"290'19","reported_seq":720,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:23.382964+0000","last_change":"2024-09-21T13:58:17.790158+0000","last_active":"2024-09-21T14:02:23.382964+0000","last_peered":"2024-09-21T14:02:23.382964+0000","last_clean":"2024-09-21T14:02:23.382964+0000","last_became_active":"2024-09-21T13:58:17.771497+0000","last_became_peered":"2024-09-21T13:58:17.771497+0000","last_unstale":"2024-09-21T14:02:23.382964+0000","last_undegraded":"2024-09-21T14:02:23.382964+0000","last_fullsized":"2024-09-21T14:02:23.382964+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:54:56.539374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:56.539374+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":242,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.107261+0000","last_change":"2024-09-21T13:59:32.202097+0000","last_active":"2024-09-21T14:02:25.107261+0000","last_peered":"2024-09-21T14:02:25.107261+0000","last_clean":"2024-09-21T14:02:25.107261+0000","last_became_active":"2024-09-21T13:59:32.201568+0000","last_became_peered":"2024-09-21T13:59:32.201568+0000","last_unstale":"2024-09-21T14:02:25.107261+0000","last_undegraded":"2024-09-21T14:02:25.107261+0000","last_fullsized":"2024-09-21T14:02:25.107261+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:01.228213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:01.228213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":678,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:23.377054+0000","last_change":"2024-09-21T14:01:41.185588+0000","last_active":"2024-09-21T14:02:23.377054+0000","last_peered":"2024-09-21T14:02:23.377054+0000","last_clean":"2024-09-21T14:02:23.377054+0000","last_became_active":"2024-09-21T14:01:41.185293+0000","last_became_peered":"2024-09-21T14:01:41.185293+0000","last_unstale":"2024-09-21T14:02:23.377054+0000","last_undegraded":"2024-09-21T14:02:23.377054+0000","last_fullsized":"2024-09-21T14:02:23.377054+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:35.409237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:35.409237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:10.948556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":573,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.096356+0000","last_change":"2024-09-21T14:01:41.200439+0000","last_active":"2024-09-21T14:02:25.096356+0000","last_peered":"2024-09-21T14:02:25.096356+0000","last_clean":"2024-09-21T14:02:25.096356+0000","last_became_active":"2024-09-21T14:01:41.199671+0000","last_became_peered":"2024-09-21T14:01:41.199671+0000","last_unstale":"2024-09-21T14:02:25.096356+0000","last_undegraded":"2024-09-21T14:02:25.096356+0000","last_fullsized":"2024-09-21T14:02:25.096356+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:39.643458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:39.643458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"290'15","reported_seq":722,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:23.377340+0000","last_change":"2024-09-21T13:58:39.977717+0000","last_active":"2024-09-21T14:02:23.377340+0000","last_peered":"2024-09-21T14:02:23.377340+0000","last_clean":"2024-09-21T14:02:23.377340+0000","last_became_active":"2024-09-21T13:58:17.764120+0000","last_became_peered":"2024-09-21T13:58:17.764120+0000","last_unstale":"2024-09-21T14:02:23.377340+0000","last_undegraded":"2024-09-21T14:02:23.377340+0000","last_fullsized":"2024-09-21T14:02:23.377340+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:58:39.977466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:39.977466+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":612,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.108156+0000","last_change":"2024-09-21T13:59:32.223082+0000","last_active":"2024-09-21T14:02:25.108156+0000","last_peered":"2024-09-21T14:02:25.108156+0000","last_clean":"2024-09-21T14:02:25.108156+0000","last_became_active":"2024-09-21T13:59:32.221854+0000","last_became_peered":"2024-09-21T13:59:32.221854+0000","last_unstale":"2024-09-21T14:02:25.108156+0000","last_undegraded":"2024-09-21T14:02:25.108156+0000","last_fullsized":"2024-09-21T14:02:25.108156+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:17.691193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:17.691193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":548,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.090943+0000","last_change":"2024-09-21T13:57:12.703746+0000","last_active":"2024-09-21T14:02:25.090943+0000","last_peered":"2024-09-21T14:02:25.090943+0000","last_clean":"2024-09-21T14:02:25.090943+0000","last_became_active":"2024-09-21T13:57:12.703441+0000","last_became_peered":"2024-09-21T13:57:12.703441+0000","last_unstale":"2024-09-21T14:02:25.090943+0000","last_undegraded":"2024-09-21T14:02:25.090943+0000","last_fullsized":"2024-09-21T14:02:25.090943+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:19.999125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:19.999125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:13.302246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":470,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.099152+0000","last_change":"2024-09-21T14:01:41.188075+0000","last_active":"2024-09-21T14:02:25.099152+0000","last_peered":"2024-09-21T14:02:25.099152+0000","last_clean":"2024-09-21T14:02:25.099152+0000","last_became_active":"2024-09-21T14:01:41.186934+0000","last_became_peered":"2024-09-21T14:01:41.186934+0000","last_unstale":"2024-09-21T14:02:25.099152+0000","last_undegraded":"2024-09-21T14:02:25.099152+0000","last_fullsized":"2024-09-21T14:02:25.099152+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:47.334504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:47.334504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"290'19","reported_seq":692,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.100371+0000","last_change":"2024-09-21T13:59:32.277742+0000","last_active":"2024-09-21T14:02:25.100371+0000","last_peered":"2024-09-21T14:02:25.100371+0000","last_clean":"2024-09-21T14:02:25.100371+0000","last_became_active":"2024-09-21T13:59:32.231567+0000","last_became_peered":"2024-09-21T13:59:32.231567+0000","last_unstale":"2024-09-21T14:02:25.100371+0000","last_undegraded":"2024-09-21T14:02:25.100371+0000","last_fullsized":"2024-09-21T14:02:25.100371+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:54:22.978267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:22.978267+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:12.294653+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":572,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.099956+0000","last_change":"2024-09-21T14:01:41.191090+0000","last_active":"2024-09-21T14:02:25.099956+0000","last_peered":"2024-09-21T14:02:25.099956+0000","last_clean":"2024-09-21T14:02:25.099956+0000","last_became_active":"2024-09-21T14:01:41.190760+0000","last_became_peered":"2024-09-21T14:01:41.190760+0000","last_unstale":"2024-09-21T14:02:25.099956+0000","last_undegraded":"2024-09-21T14:02:25.099956+0000","last_fullsized":"2024-09-21T14:02:25.099956+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:17.305144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:17.305144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":618,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.104371+0000","last_change":"2024-09-21T13:59:32.219508+0000","last_active":"2024-09-21T14:02:25.104371+0000","last_peered":"2024-09-21T14:02:25.104371+0000","last_clean":"2024-09-21T14:02:25.104371+0000","last_became_active":"2024-09-21T13:59:32.214004+0000","last_became_peered":"2024-09-21T13:59:32.214004+0000","last_unstale":"2024-09-21T14:02:25.104371+0000","last_undegraded":"2024-09-21T14:02:25.104371+0000","last_fullsized":"2024-09-21T14:02:25.104371+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:36.525417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:36.525417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":614,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.091135+0000","last_change":"2024-09-21T13:59:28.124373+0000","last_active":"2024-09-21T14:02:25.091135+0000","last_peered":"2024-09-21T14:02:25.091135+0000","last_clean":"2024-09-21T14:02:25.091135+0000","last_became_active":"2024-09-21T13:58:17.778115+0000","last_became_peered":"2024-09-21T13:58:17.778115+0000","last_unstale":"2024-09-21T14:02:25.091135+0000","last_undegraded":"2024-09-21T14:02:25.091135+0000","last_fullsized":"2024-09-21T14:02:25.091135+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:28.124296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:28.124296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"290'8","reported_seq":638,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.090938+0000","last_change":"2024-09-21T13:59:26.034457+0000","last_active":"2024-09-21T14:02:25.090938+0000","last_peered":"2024-09-21T14:02:25.090938+0000","last_clean":"2024-09-21T14:02:25.090938+0000","last_became_active":"2024-09-21T13:58:17.775079+0000","last_became_peered":"2024-09-21T13:58:17.775079+0000","last_unstale":"2024-09-21T14:02:25.090938+0000","last_undegraded":"2024-09-21T14:02:25.090938+0000","last_fullsized":"2024-09-21T14:02:25.090938+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'8","last_scrub_stamp":"2024-09-21T13:59:26.034372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:26.034372+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":540,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.099916+0000","last_change":"2024-09-21T14:01:41.196335+0000","last_active":"2024-09-21T14:02:25.099916+0000","last_peered":"2024-09-21T14:02:25.099916+0000","last_clean":"2024-09-21T14:02:25.099916+0000","last_became_active":"2024-09-21T14:01:41.195989+0000","last_became_peered":"2024-09-21T14:01:41.195989+0000","last_unstale":"2024-09-21T14:02:25.099916+0000","last_undegraded":"2024-09-21T14:02:25.099916+0000","last_fullsized":"2024-09-21T14:02:25.099916+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:16.457445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:16.457445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":230,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:23.388015+0000","last_change":"2024-09-21T14:00:57.128764+0000","last_active":"2024-09-21T14:02:23.388015+0000","last_peered":"2024-09-21T14:02:23.388015+0000","last_clean":"2024-09-21T14:02:23.388015+0000","last_became_active":"2024-09-21T13:59:32.204380+0000","last_became_peered":"2024-09-21T13:59:32.204380+0000","last_unstale":"2024-09-21T14:02:23.388015+0000","last_undegraded":"2024-09-21T14:02:23.388015+0000","last_fullsized":"2024-09-21T14:02:23.388015+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:00:57.128713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:00:57.128713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:23.643628+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":551,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.106387+0000","last_change":"2024-09-21T13:59:32.212615+0000","last_active":"2024-09-21T14:02:25.106387+0000","last_peered":"2024-09-21T14:02:25.106387+0000","last_clean":"2024-09-21T14:02:25.106387+0000","last_became_active":"2024-09-21T13:59:32.210134+0000","last_became_peered":"2024-09-21T13:59:32.210134+0000","last_unstale":"2024-09-21T14:02:25.106387+0000","last_undegraded":"2024-09-21T14:02:25.106387+0000","last_fullsized":"2024-09-21T14:02:25.106387+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:08.852789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:55:08.852789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":663,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:23.382300+0000","last_change":"2024-09-21T14:01:41.187792+0000","last_active":"2024-09-21T14:02:23.382300+0000","last_peered":"2024-09-21T14:02:23.382300+0000","last_clean":"2024-09-21T14:02:23.382300+0000","last_became_active":"2024-09-21T14:01:41.187438+0000","last_became_peered":"2024-09-21T14:01:41.187438+0000","last_unstale":"2024-09-21T14:02:23.382300+0000","last_undegraded":"2024-09-21T14:02:23.382300+0000","last_fullsized":"2024-09-21T14:02:23.382300+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:17.706325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:17.706325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":594,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.091148+0000","last_change":"2024-09-21T13:58:13.776632+0000","last_active":"2024-09-21T14:02:25.091148+0000","last_peered":"2024-09-21T14:02:25.091148+0000","last_clean":"2024-09-21T14:02:25.091148+0000","last_became_active":"2024-09-21T13:57:12.704042+0000","last_became_peered":"2024-09-21T13:57:12.704042+0000","last_unstale":"2024-09-21T14:02:25.091148+0000","last_undegraded":"2024-09-21T14:02:25.091148+0000","last_fullsized":"2024-09-21T14:02:25.091148+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:13.776572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:13.776572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"290'19","reported_seq":262,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T14:02:23.389777+0000","last_change":"2024-09-21T14:01:54.680822+0000","last_active":"2024-09-21T14:02:23.389777+0000","last_peered":"2024-09-21T14:02:23.389777+0000","last_clean":"2024-09-21T14:02:23.389777+0000","last_became_active":"2024-09-21T14:01:41.194606+0000","last_became_peered":"2024-09-21T14:01:41.194606+0000","last_unstale":"2024-09-21T14:02:23.389777+0000","last_undegraded":"2024-09-21T14:02:23.389777+0000","last_fullsized":"2024-09-21T14:02:23.389777+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'19","last_scrub_stamp":"2024-09-21T14:01:54.680735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:54.680735+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:21.312249+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":520,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T14:02:25.108523+0000","last_change":"2024-09-21T13:59:32.203045+0000","last_active":"2024-09-21T14:02:25.108523+0000","last_peered":"2024-09-21T14:02:25.108523+0000","last_clean":"2024-09-21T14:02:25.108523+0000","last_became_active":"2024-09-21T13:59:32.202225+0000","last_became_peered":"2024-09-21T13:59:32.202225+0000","last_unstale":"2024-09-21T14:02:25.108523+0000","last_undegraded":"2024-09-21T14:02:25.108523+0000","last_fullsized":"2024-09-21T14:02:25.108523+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:40.650832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:40.650832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":1381629976,"num_objects":21083,"num_object_clones":0,"num_object_copies":63249,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1443,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21083,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":221729,"num_write_kb":7769921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41166,"num_bytes_recovered":2557804568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4259852288,"data_stored":4259840072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75900,"ondisk_log_size":75900,"up":48,"acting":46,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1111,"num_read_kb":868,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":1914,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":687,"ondisk_log_size":687,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":180,"num_read_kb":175,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":429358,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":159,"num_write":83,"num_write_kb":1184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1084718,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":298,"seq":1279900254244,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":959136,"kb_used_data":678860,"kb_used_omap":4,"kb_used_meta":280251,"kb_avail":92786016,"statfs":{"total":95995035648,"available":95012880384,"internally_reserved":0,"allocated":695152640,"data_stored":694331557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":286977704},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463083,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":612780,"kb_used_data":296988,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":93132372,"statfs":{"total":95995035648,"available":95367548928,"internally_reserved":0,"allocated":304115712,"data_stored":303326346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":323353030},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":345,"seq":1481763717130,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":865168,"kb_used_data":770240,"kb_used_omap":4,"kb_used_meta":94907,"kb_avail":92879984,"statfs":{"total":95995035648,"available":95109103616,"internally_reserved":0,"allocated":788725760,"data_stored":787917340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":97185443},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920952,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":747164,"kb_used_data":364744,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":92997988,"statfs":{"total":95995035648,"available":95229939712,"internally_reserved":0,"allocated":373497856,"data_stored":372672240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":391576005},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":240,"seq":1030792151104,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":938612,"kb_used_data":693916,"kb_used_omap":4,"kb_used_meta":244667,"kb_avail":92806540,"statfs":{"total":95995035648,"available":95033896960,"internally_reserved":0,"allocated":710569984,"data_stored":709733400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4443,"internal_metadata":250539685},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":130,"seq":558345748605,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":840368,"kb_used_data":298656,"kb_used_omap":4,"kb_used_meta":541691,"kb_avail":92904784,"statfs":{"total":95995035648,"available":95134498816,"internally_reserved":0,"allocated":305823744,"data_stored":305031544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4441,"internal_metadata":554692263},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575050,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1382400,"kb_used_data":813804,"kb_used_omap":1,"kb_used_meta":568446,"kb_avail":92362752,"statfs":{"total":95995035648,"available":94579458048,"internally_reserved":0,"allocated":833335296,"data_stored":832473584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":582089157},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738576,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":871056,"kb_used_data":380348,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":92874096,"statfs":{"total":95995035648,"available":95103074304,"internally_reserved":0,"allocated":389476352,"data_stored":388662764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":502462917},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":372047872,"data_stored":372047872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":811208704,"data_stored":811204632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":291438592,"data_stored":291438592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":675217408,"data_stored":675217408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":370540544,"data_stored":370540544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":766906368,"data_stored":766902296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":289472512,"data_stored":289472512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":683020288,"data_stored":683016216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T14:02:28.906 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph status --format=json 2024-09-21T14:02:28.935 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:02:28.962 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:02:29.029 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:29.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:29.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:02:29.130 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:29.165 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:29.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:29.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:29.266 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:02:29.287 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:29.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:29.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:02:29.389 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:29.436 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:29.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:29.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:29.538 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:29.590 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:29.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:29.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:29.610 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:02:29.692 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:29.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:29 smithi187 ceph-mon[33032]: 5.8 scrub starts 2024-09-21T14:02:29.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:29 smithi187 ceph-mon[33032]: 5.8 scrub ok 2024-09-21T14:02:29.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:29 smithi187 ceph-mon[33032]: pgmap v690: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 156 active+clean; 1.3 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 935 op/s; 1443/63846 objects degraded (2.260%); 7.0 MiB/s, 111 objects/s recovering 2024-09-21T14:02:29.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:29 smithi187 ceph-mon[33032]: osdmap e372: 8 total, 8 up, 8 in 2024-09-21T14:02:29.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:29 smithi187 ceph-mon[33032]: 4.e scrub starts 2024-09-21T14:02:29.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:29 smithi187 ceph-mon[33032]: 4.e scrub ok 2024-09-21T14:02:29.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:29 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:02:29.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:29 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:02:29.739 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:29.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:29.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:29.751 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:29 smithi049 ceph-mon[40489]: 5.8 scrub starts 2024-09-21T14:02:29.751 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:29 smithi049 ceph-mon[40489]: 5.8 scrub ok 2024-09-21T14:02:29.751 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:29 smithi049 ceph-mon[40489]: pgmap v690: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 156 active+clean; 1.3 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 935 op/s; 1443/63846 objects degraded (2.260%); 7.0 MiB/s, 111 objects/s recovering 2024-09-21T14:02:29.751 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:29 smithi049 ceph-mon[40489]: osdmap e372: 8 total, 8 up, 8 in 2024-09-21T14:02:29.751 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:29 smithi049 ceph-mon[40489]: 4.e scrub starts 2024-09-21T14:02:29.751 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:29 smithi049 ceph-mon[40489]: 4.e scrub ok 2024-09-21T14:02:29.751 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:29 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:02:29.752 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:29 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:02:29.752 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:29 smithi049 ceph-mon[30080]: 5.8 scrub starts 2024-09-21T14:02:29.752 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:29 smithi049 ceph-mon[30080]: 5.8 scrub ok 2024-09-21T14:02:29.752 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:29 smithi049 ceph-mon[30080]: pgmap v690: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 156 active+clean; 1.3 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 935 op/s; 1443/63846 objects degraded (2.260%); 7.0 MiB/s, 111 objects/s recovering 2024-09-21T14:02:29.752 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:29 smithi049 ceph-mon[30080]: osdmap e372: 8 total, 8 up, 8 in 2024-09-21T14:02:29.752 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:29 smithi049 ceph-mon[30080]: 4.e scrub starts 2024-09-21T14:02:29.752 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:29 smithi049 ceph-mon[30080]: 4.e scrub ok 2024-09-21T14:02:29.752 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:29 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:02:29.752 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:29 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:02:29.841 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:02:29.848 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T14:02:29.877 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:29.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:29.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:02:29.979 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:02:30.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:30.021 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:30.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:02:30.122 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:02:30.161 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:30.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:30.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:02:30.194 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:02:29 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:02:30.263 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:30.298 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:30.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:30.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:30.311 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:02:30.341 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T14:02:30.400 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:30.407 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-09-21T14:02:30.455 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:30.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:30.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:30.557 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:30.604 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:30 smithi187 ceph-mon[33032]: 5.19 scrub starts 2024-09-21T14:02:30.604 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:30 smithi187 ceph-mon[33032]: 5.19 scrub ok 2024-09-21T14:02:30.605 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:30 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:02:30.605 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:30 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3073064166' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:02:30.605 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:30 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 1443/63846 objects degraded (2.260%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T14:02:30.610 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:30.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:30.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:30.712 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:30.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:30 smithi049 ceph-mon[30080]: 5.19 scrub starts 2024-09-21T14:02:30.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:30 smithi049 ceph-mon[30080]: 5.19 scrub ok 2024-09-21T14:02:30.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:30 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:02:30.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:30 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3073064166' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:02:30.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:30 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 1443/63846 objects degraded (2.260%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T14:02:30.755 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:02:30 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:02:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:02:30.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:30 smithi049 ceph-mon[40489]: 5.19 scrub starts 2024-09-21T14:02:30.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:30 smithi049 ceph-mon[40489]: 5.19 scrub ok 2024-09-21T14:02:30.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:30 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:02:30.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:30 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3073064166' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:02:30.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:30 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 1443/63846 objects degraded (2.260%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T14:02:30.772 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T14:02:30.772 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"40a3d2e8-781f-11ef-baf6-efdc52797490","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1443/61152 objects degraded (2.360%), 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1066,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":373,"num_osds":8,"num_up_osds":8,"osd_up_since":1726927300,"num_in_osds":8,"osd_in_since":1726927096,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":156},{"state_name":"active+undersized+degraded+remapped+backfilling","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":158,"num_pools":7,"num_objects":20384,"data_bytes":1323214145,"bytes_used":6951497728,"bytes_avail":761008787456,"bytes_total":767960285184,"degraded_objects":1443,"degraded_total":61152,"degraded_ratio":0.023596938775510203,"recovering_objects_per_sec":111,"recovering_bytes_per_sec":7302928,"recovering_keys_per_sec":0,"num_objects_recovered":669,"num_bytes_recovered":43843584,"num_keys_recovered":0,"read_bytes_sec":852,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":1024},"fsmap":{"epoch":1,"btime":"2024-09-21T13:42:57:838094+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.49:8443/","prometheus":"http://172.21.15.49:9283/"}},"servicemap":{"epoch":17,"modified":"2024-09-21T14:00:31.944278+0000","services":{"rgw":{"daemons":{"summary":"","24365":{"start_epoch":12,"start_stamp":"2024-09-21T13:49:04.899719+0000","gid":24365,"addr":"172.21.15.49:0/3404764347","metadata":{"arch":"x86_64","ceph_release":"squid","ceph_version":"ceph version 19.3.0-5104-g44178494 (441784946648ff53dd310293f9dd752afeeea72a) squid (dev)","ceph_version_short":"19.3.0-5104-g44178494","container_hostname":"smithi049","container_image":"quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5","cpu":"Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz","distro":"centos","distro_description":"CentOS Stream 9","distro_version":"9","frontend_config#0":"beast port=80","frontend_type#0":"beast","hostname":"smithi049","id":"foo.a","kernel_description":"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024","kernel_version":"5.14.0-508.el9.x86_64","mem_swap_kb":"0","mem_total_kb":"32487164","num_handles":"1","os":"Linux","pid":"7","realm_id":"","realm_name":"","zone_id":"d447add1-8b31-4043-a929-4cb437e5f078","zone_name":"default","zonegroup_id":"06d4f733-b51c-4f26-9844-acd95f223f5b","zonegroup_name":"default"},"task_status":{}}}}}},"progress_events":{}} 2024-09-21T14:02:30.777 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:30.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:30.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:30.879 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:02:30.904 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T14:02:30.904 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-21T14:02:30.905 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.905 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25421.0:226629 9.8 e373/345 9:1a93ec7a:::benchmark_data_smithi049_138504_object105221:head v 373'14410, mlcod=373'14407)", 2024-09-21T14:02:30.905 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:02:30.898204+0000", 2024-09-21T14:02:30.905 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0021651700000000001, 2024-09-21T14:02:30.905 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0021937430000000002, 2024-09-21T14:02:30.905 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:02:30.905 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:02:30.905 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T14:02:30.905 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:02:30.905 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.906 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:02:30.906 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.898204+0000", 2024-09-21T14:02:30.906 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:02:30.906 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.906 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.906 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:02:30.906 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.898204+0000", 2024-09-21T14:02:30.906 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:02:30.906 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.906 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.906 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:02:30.907 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.898205+0000", 2024-09-21T14:02:30.907 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3349999999999999e-06 2024-09-21T14:02:30.907 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.907 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.907 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:02:30.907 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.898213+0000", 2024-09-21T14:02:30.907 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2090000000000005e-06 2024-09-21T14:02:30.907 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.907 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.907 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:02:30.907 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.898226+0000", 2024-09-21T14:02:30.907 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2116999999999999e-05 2024-09-21T14:02:30.908 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.908 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.908 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:02:30.908 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.898233+0000", 2024-09-21T14:02:30.908 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.644e-06 2024-09-21T14:02:30.908 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.908 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.908 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:02:30.908 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.899102+0000", 2024-09-21T14:02:30.908 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00086887900000000003 2024-09-21T14:02:30.909 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.909 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.909 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:02:30.909 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.899120+0000", 2024-09-21T14:02:30.909 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8437999999999999e-05 2024-09-21T14:02:30.909 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:02:30.909 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:02:30.909 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:02:30.909 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.909 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.909 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25421.0:226631 9.b e373/345 9:d86083ab:::benchmark_data_smithi049_138504_object105223:head v 373'14176, mlcod=373'14173)", 2024-09-21T14:02:30.910 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:02:30.898266+0000", 2024-09-21T14:02:30.910 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0021033779999999999, 2024-09-21T14:02:30.910 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0021849679999999998, 2024-09-21T14:02:30.910 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:02:30.910 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:02:30.910 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T14:02:30.910 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:02:30.910 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.910 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:02:30.910 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.898266+0000", 2024-09-21T14:02:30.910 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:02:30.910 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.910 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.911 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:02:30.911 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.898266+0000", 2024-09-21T14:02:30.911 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:02:30.911 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.911 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.911 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:02:30.911 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.898266+0000", 2024-09-21T14:02:30.911 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0100000000000005e-07 2024-09-21T14:02:30.911 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.911 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.911 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:02:30.911 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.898270+0000", 2024-09-21T14:02:30.911 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3270000000000002e-06 2024-09-21T14:02:30.911 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.912 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.912 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:02:30.912 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.898278+0000", 2024-09-21T14:02:30.912 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3799999999999996e-06 2024-09-21T14:02:30.912 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.912 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.912 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:02:30.912 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.898288+0000", 2024-09-21T14:02:30.912 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0519e-05 2024-09-21T14:02:30.912 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.912 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.912 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:02:30.912 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.898481+0000", 2024-09-21T14:02:30.913 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000192702 2024-09-21T14:02:30.913 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.913 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.913 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:02:30.913 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.898501+0000", 2024-09-21T14:02:30.913 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0194e-05 2024-09-21T14:02:30.913 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:02:30.913 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:02:30.913 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:02:30.914 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.914 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.914 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:226635 9.a 9:5f225e2e:::benchmark_data_smithi049_138504_object105227:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-09-21T14:02:30.914 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:02:30.898375+0000", 2024-09-21T14:02:30.914 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0019940280000000001, 2024-09-21T14:02:30.914 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0021245420000000001, 2024-09-21T14:02:30.914 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:02:30.914 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:02:30.914 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-21T14:02:30.914 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:02:30.915 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:02:30.915 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:02:30.915 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 226635 2024-09-21T14:02:30.915 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.915 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:02:30.915 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.915 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:02:30.915 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.898375+0000", 2024-09-21T14:02:30.915 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:02:30.915 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.915 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.915 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:02:30.915 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.898375+0000", 2024-09-21T14:02:30.915 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:02:30.915 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.916 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.916 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:02:30.916 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.898377+0000", 2024-09-21T14:02:30.916 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.655e-06 2024-09-21T14:02:30.916 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.916 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.916 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:02:30.916 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.898382+0000", 2024-09-21T14:02:30.916 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7819999999999999e-06 2024-09-21T14:02:30.916 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.916 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.916 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:02:30.916 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.898397+0000", 2024-09-21T14:02:30.916 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4504e-05 2024-09-21T14:02:30.916 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.917 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.917 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:02:30.917 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.898405+0000", 2024-09-21T14:02:30.917 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0150000000000003e-06 2024-09-21T14:02:30.917 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.917 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.917 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:02:30.917 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.898705+0000", 2024-09-21T14:02:30.917 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000300236 2024-09-21T14:02:30.917 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.917 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.917 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:02:30.917 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.899106+0000", 2024-09-21T14:02:30.917 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00040077400000000001 2024-09-21T14:02:30.917 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.918 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.918 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,7", 2024-09-21T14:02:30.918 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.899221+0000", 2024-09-21T14:02:30.918 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011534799999999999 2024-09-21T14:02:30.918 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:02:30.918 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:02:30.918 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:02:30.918 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.918 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.918 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:226636 9.5 9:a92a2244:::benchmark_data_smithi049_138504_object105228:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-09-21T14:02:30.918 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:02:30.898428+0000", 2024-09-21T14:02:30.918 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.001941311, 2024-09-21T14:02:30.918 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0021400590000000001, 2024-09-21T14:02:30.918 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:02:30.919 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:02:30.919 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-21T14:02:30.919 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:02:30.919 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:02:30.919 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:02:30.919 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 226636 2024-09-21T14:02:30.919 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.919 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:02:30.919 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.919 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:02:30.919 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.898428+0000", 2024-09-21T14:02:30.919 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:02:30.919 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.919 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.920 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:02:30.920 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.898428+0000", 2024-09-21T14:02:30.920 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:02:30.920 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.920 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.920 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:02:30.920 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.898428+0000", 2024-09-21T14:02:30.920 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.82e-07 2024-09-21T14:02:30.920 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.920 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.920 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:02:30.920 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.898431+0000", 2024-09-21T14:02:30.920 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.6759999999999999e-06 2024-09-21T14:02:30.920 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.921 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.921 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:02:30.921 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.898438+0000", 2024-09-21T14:02:30.921 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.464e-06 2024-09-21T14:02:30.921 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.921 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.921 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:02:30.921 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.898444+0000", 2024-09-21T14:02:30.921 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8000000000000004e-06 2024-09-21T14:02:30.921 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.921 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.921 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:02:30.921 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.898608+0000", 2024-09-21T14:02:30.921 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000163373 2024-09-21T14:02:30.922 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.922 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.922 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:02:30.922 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.898733+0000", 2024-09-21T14:02:30.922 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012517299999999999 2024-09-21T14:02:30.922 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.922 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.922 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:02:30.922 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.898872+0000", 2024-09-21T14:02:30.922 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000139474 2024-09-21T14:02:30.922 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:02:30.922 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:02:30.922 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:02:30.922 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.923 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.923 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.25421.0:226606 9.5 e373/346)", 2024-09-21T14:02:30.923 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:02:30.898872+0000", 2024-09-21T14:02:30.923 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0014970859999999999, 2024-09-21T14:02:30.923 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0017498769999999999, 2024-09-21T14:02:30.923 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:02:30.923 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:02:30.923 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "queued for pg", 2024-09-21T14:02:30.923 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:02:30.923 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.923 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:02:30.923 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.898872+0000", 2024-09-21T14:02:30.923 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:02:30.923 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.924 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.924 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:02:30.924 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.898872+0000", 2024-09-21T14:02:30.924 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:02:30.924 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.924 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.924 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:02:30.924 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.898873+0000", 2024-09-21T14:02:30.924 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6199999999999998e-07 2024-09-21T14:02:30.924 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.924 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.924 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:02:30.924 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.898876+0000", 2024-09-21T14:02:30.924 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6660000000000001e-06 2024-09-21T14:02:30.925 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.925 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.925 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:02:30.925 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.898885+0000", 2024-09-21T14:02:30.925 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1789999999999997e-06 2024-09-21T14:02:30.925 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.925 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.925 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:02:30.925 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.898892+0000", 2024-09-21T14:02:30.925 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1739999999999997e-06 2024-09-21T14:02:30.925 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:02:30.925 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:02:30.925 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:02:30.925 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.925 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.926 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:226606 9.5 9:ab55d2ad:::benchmark_data_smithi049_138504_object105198:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-09-21T14:02:30.926 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:02:30.873203+0000", 2024-09-21T14:02:30.926 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.027165989000000001, 2024-09-21T14:02:30.926 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.027457220000000001, 2024-09-21T14:02:30.926 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:02:30.926 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:02:30.926 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-21T14:02:30.926 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:02:30.926 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:02:30.926 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:02:30.926 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 226606 2024-09-21T14:02:30.926 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.926 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:02:30.927 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.927 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:02:30.927 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.873203+0000", 2024-09-21T14:02:30.927 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:02:30.927 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.927 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.927 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:02:30.927 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.873203+0000", 2024-09-21T14:02:30.927 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:02:30.927 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.927 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.928 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:02:30.928 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.873203+0000", 2024-09-21T14:02:30.928 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2399999999999999e-07 2024-09-21T14:02:30.928 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.928 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.928 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:02:30.928 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.873206+0000", 2024-09-21T14:02:30.928 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4990000000000001e-06 2024-09-21T14:02:30.928 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.928 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.928 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:02:30.928 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.873212+0000", 2024-09-21T14:02:30.928 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1369999999999999e-06 2024-09-21T14:02:30.928 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.929 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.929 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:02:30.929 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.873217+0000", 2024-09-21T14:02:30.929 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7129999999999996e-06 2024-09-21T14:02:30.929 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.929 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.929 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:02:30.929 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.880380+0000", 2024-09-21T14:02:30.929 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0071630319999999997 2024-09-21T14:02:30.929 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.929 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.929 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:02:30.929 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.880507+0000", 2024-09-21T14:02:30.929 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012729899999999999 2024-09-21T14:02:30.930 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.930 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.930 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:02:30.930 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.880587+0000", 2024-09-21T14:02:30.930 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0259999999999994e-05 2024-09-21T14:02:30.930 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.930 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.930 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:02:30.930 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.883083+0000", 2024-09-21T14:02:30.930 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0024955590000000001 2024-09-21T14:02:30.930 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.930 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.930 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:02:30.930 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.885346+0000", 2024-09-21T14:02:30.931 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.002262881 2024-09-21T14:02:30.931 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:02:30.931 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:02:30.931 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:02:30.931 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.931 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.931 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "MOSDPGPushReply(9.5 373/346 [PushReplyOp(9:a913e9ff:::benchmark_data_smithi049_138504_object107778:head)])", 2024-09-21T14:02:30.931 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:02:30.898903+0000", 2024-09-21T14:02:30.931 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0014657439999999999, 2024-09-21T14:02:30.931 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00181869, 2024-09-21T14:02:30.931 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:02:30.931 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:02:30.931 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "queued for pg", 2024-09-21T14:02:30.931 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:02:30.931 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.932 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:02:30.932 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.898903+0000", 2024-09-21T14:02:30.932 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:02:30.932 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.932 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.932 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:02:30.932 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.898903+0000", 2024-09-21T14:02:30.932 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:02:30.932 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.932 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.932 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:02:30.932 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.898904+0000", 2024-09-21T14:02:30.932 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4999999999999999e-07 2024-09-21T14:02:30.932 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.933 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.933 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:02:30.933 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.898908+0000", 2024-09-21T14:02:30.933 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0090000000000001e-06 2024-09-21T14:02:30.933 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.933 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.933 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:02:30.933 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.898914+0000", 2024-09-21T14:02:30.933 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9349999999999997e-06 2024-09-21T14:02:30.933 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.933 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.933 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:02:30.933 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.898921+0000", 2024-09-21T14:02:30.933 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1419999999999998e-06 2024-09-21T14:02:30.933 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:02:30.934 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:02:30.934 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:02:30.934 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.934 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.934 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25421.0:226634 9.d e373/371)", 2024-09-21T14:02:30.934 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:02:30.898946+0000", 2024-09-21T14:02:30.934 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.001423505, 2024-09-21T14:02:30.934 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0018097199999999999, 2024-09-21T14:02:30.934 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:02:30.934 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:02:30.934 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "queued for pg", 2024-09-21T14:02:30.934 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:02:30.934 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.934 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:02:30.935 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.898946+0000", 2024-09-21T14:02:30.935 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:02:30.935 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.935 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.935 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:02:30.935 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.898946+0000", 2024-09-21T14:02:30.935 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:02:30.935 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.935 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.935 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:02:30.935 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.898946+0000", 2024-09-21T14:02:30.936 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7599999999999998e-07 2024-09-21T14:02:30.936 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.936 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.936 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:02:30.936 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.898950+0000", 2024-09-21T14:02:30.936 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.16e-06 2024-09-21T14:02:30.936 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.936 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.936 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:02:30.936 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.898958+0000", 2024-09-21T14:02:30.936 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4649999999999997e-06 2024-09-21T14:02:30.937 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.937 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.937 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:02:30.937 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.898963+0000", 2024-09-21T14:02:30.937 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1669999999999998e-06 2024-09-21T14:02:30.937 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:02:30.937 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:02:30.937 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:02:30.937 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.937 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.937 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:226640 9.f 9.a5ee135f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-09-21T14:02:30.937 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:02:30.899010+0000", 2024-09-21T14:02:30.937 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0013588020000000001, 2024-09-21T14:02:30.937 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001783319, 2024-09-21T14:02:30.938 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:02:30.938 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:02:30.938 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "queued for pg", 2024-09-21T14:02:30.938 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:02:30.938 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:02:30.938 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:02:30.938 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 226640 2024-09-21T14:02:30.938 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.938 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:02:30.938 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.938 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:02:30.939 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.899010+0000", 2024-09-21T14:02:30.939 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:02:30.939 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.939 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.939 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:02:30.939 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.899010+0000", 2024-09-21T14:02:30.939 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:02:30.939 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.939 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.939 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:02:30.939 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.899011+0000", 2024-09-21T14:02:30.939 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6499999999999999e-07 2024-09-21T14:02:30.939 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.940 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.940 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:02:30.940 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.899015+0000", 2024-09-21T14:02:30.940 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7359999999999999e-06 2024-09-21T14:02:30.940 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.940 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.940 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:02:30.940 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.899035+0000", 2024-09-21T14:02:30.940 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0831000000000001e-05 2024-09-21T14:02:30.940 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.940 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.940 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:02:30.941 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.899042+0000", 2024-09-21T14:02:30.941 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6290000000000004e-06 2024-09-21T14:02:30.941 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:02:30.941 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:02:30.941 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:02:30.941 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.941 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.941 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25421.0:226624 9.b e373/345 9:d3e58074:::benchmark_data_smithi049_138504_object105216:head v 373'14175, mlcod=373'14173)", 2024-09-21T14:02:30.941 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:02:30.888350+0000", 2024-09-21T14:02:30.941 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.012019416999999999, 2024-09-21T14:02:30.941 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.012494402999999999, 2024-09-21T14:02:30.941 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:02:30.941 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:02:30.941 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T14:02:30.942 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:02:30.942 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.942 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:02:30.942 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.888350+0000", 2024-09-21T14:02:30.942 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:02:30.942 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.942 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.942 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:02:30.942 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.888350+0000", 2024-09-21T14:02:30.942 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:02:30.943 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.943 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.943 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:02:30.943 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.888351+0000", 2024-09-21T14:02:30.943 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.569999999999999e-07 2024-09-21T14:02:30.943 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.943 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.943 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:02:30.943 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.888357+0000", 2024-09-21T14:02:30.943 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8619999999999997e-06 2024-09-21T14:02:30.943 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.943 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.944 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:02:30.944 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.888374+0000", 2024-09-21T14:02:30.944 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6719e-05 2024-09-21T14:02:30.944 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.944 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.944 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:02:30.944 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.888382+0000", 2024-09-21T14:02:30.944 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4000000000000003e-06 2024-09-21T14:02:30.944 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.944 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.944 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:02:30.945 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.891573+0000", 2024-09-21T14:02:30.945 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0031917310000000002 2024-09-21T14:02:30.945 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:02:30.945 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:02:30.945 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:02:30.945 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:02:30.891593+0000", 2024-09-21T14:02:30.945 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9848999999999999e-05 2024-09-21T14:02:30.945 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:02:30.945 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:02:30.945 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:02:30.945 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:02:30.946 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2024-09-21T14:02:30.946 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 10 2024-09-21T14:02:30.946 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T14:02:30.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:30.947 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:30.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:02:31.049 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:02:31.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:31.071 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:31.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:02:31.174 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:02:31.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:31.213 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:31.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:02:31.315 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:02:31.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:31.354 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:31.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:02:31.455 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:02:31.478 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:31.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:31.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:02:31.581 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:31.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:31.619 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:31.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:31.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:31 smithi187 ceph-mon[33032]: 6.14 scrub starts 2024-09-21T14:02:31.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:31 smithi187 ceph-mon[33032]: 6.14 scrub ok 2024-09-21T14:02:31.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:31 smithi187 ceph-mon[33032]: pgmap v692: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 156 active+clean; 1.2 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.02k op/s; 1443/61152 objects degraded (2.360%); 7.0 MiB/s, 111 objects/s recovering 2024-09-21T14:02:31.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:31 smithi187 ceph-mon[33032]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T14:02:31.696 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:31 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3073064166' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T14:02:31.696 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:31 smithi187 ceph-mon[33032]: osdmap e373: 8 total, 8 up, 8 in 2024-09-21T14:02:31.696 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:31 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/611967193' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T14:02:31.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:31 smithi049 ceph-mon[30080]: 6.14 scrub starts 2024-09-21T14:02:31.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:31 smithi049 ceph-mon[30080]: 6.14 scrub ok 2024-09-21T14:02:31.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:31 smithi049 ceph-mon[30080]: pgmap v692: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 156 active+clean; 1.2 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.02k op/s; 1443/61152 objects degraded (2.360%); 7.0 MiB/s, 111 objects/s recovering 2024-09-21T14:02:31.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:31 smithi049 ceph-mon[30080]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T14:02:31.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:31 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3073064166' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T14:02:31.714 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:31 smithi049 ceph-mon[30080]: osdmap e373: 8 total, 8 up, 8 in 2024-09-21T14:02:31.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:31 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/611967193' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T14:02:31.715 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:31 smithi049 ceph-mon[40489]: 6.14 scrub starts 2024-09-21T14:02:31.715 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:31 smithi049 ceph-mon[40489]: 6.14 scrub ok 2024-09-21T14:02:31.715 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:31 smithi049 ceph-mon[40489]: pgmap v692: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 156 active+clean; 1.2 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.02k op/s; 1443/61152 objects degraded (2.360%); 7.0 MiB/s, 111 objects/s recovering 2024-09-21T14:02:31.715 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:31 smithi049 ceph-mon[40489]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T14:02:31.715 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:31 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3073064166' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T14:02:31.715 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:31 smithi049 ceph-mon[40489]: osdmap e373: 8 total, 8 up, 8 in 2024-09-21T14:02:31.715 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:31 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/611967193' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T14:02:31.721 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:31.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:31.757 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:31.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:31.858 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:31.896 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:31.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:31.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:31.999 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:02:32.037 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:32.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:32.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:02:32.138 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:32.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:32.179 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:32.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:32.280 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:02:32.319 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:32.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:32.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:02:32.387 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T14:02:32.421 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:32.455 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:32.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:32.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:32.556 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:02:32.601 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:32.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:32.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:02:32.703 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:32.711 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:02:32.752 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-21T14:02:32.753 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T14:02:32.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:32.762 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:32.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:32.820 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:02:32.863 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:32.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:32.902 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:32.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:33.003 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:33.065 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:33.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:33.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:33.167 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:33.199 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:33.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:33.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:33.301 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:33.354 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:33.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:33.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:33.382 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:02:33.382 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:02:33.416 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:02:33.416 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:02:33.426 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:02:33.453 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:02:33.454 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:02:33.456 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:33.468 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T14:02:33.468 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-21T14:02:33.469 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-21T14:02:33.469 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-21T14:02:33.469 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T14:02:33.488 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:02:33.488 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:02:33.495 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:33.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:33.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:33.521 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:02:33.521 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:02:33.565 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:02:33.565 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:02:33.565 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:33 smithi049 ceph-mon[40489]: pgmap v694: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 1023 MiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.19k op/s; 795/49701 objects degraded (1.600%); 6.6 MiB/s, 105 objects/s recovering 2024-09-21T14:02:33.565 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:33 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/864711741' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:02:33.566 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:33 smithi049 ceph-mon[30080]: pgmap v694: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 1023 MiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.19k op/s; 795/49701 objects degraded (1.600%); 6.6 MiB/s, 105 objects/s recovering 2024-09-21T14:02:33.566 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:33 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/864711741' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:02:33.597 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:33.599 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:02:33.599 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:02:33.632 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:33.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:33.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:33.634 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:02:33.634 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:02:33.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:33 smithi187 ceph-mon[33032]: pgmap v694: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 1023 MiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.19k op/s; 795/49701 objects degraded (1.600%); 6.6 MiB/s, 105 objects/s recovering 2024-09-21T14:02:33.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:33 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/864711741' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:02:33.735 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:33.771 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:33.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:33.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:33.873 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:33.899 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T14:02:33.913 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:33.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:33.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:34.015 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:34.057 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:34.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:34.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:34.159 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:02:34.205 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:34.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:34.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:02:34.229 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T14:02:34.307 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:34.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:34.353 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:34.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:34.454 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:02:34.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:34.492 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:34.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:02:34.585 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T14:02:34.592 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-09-21T14:02:34.592 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:34.633 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:34.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:34.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:34.687 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:02:34.736 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:02:34.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:34.776 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:34.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:02:34.877 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:34.911 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:34.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:34.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:35.013 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:02:35.052 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:35.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:35.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:02:35.154 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:35.189 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:35.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:35.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:35.280 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T14:02:35.281 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:02:35.291 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:35.330 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:35.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:35.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:35.431 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:02:35.465 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:35.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:35.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:02:35.567 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:35.582 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":695,"stamp":"2024-09-21T14:02:33.987722+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1049208129,"num_objects":16203,"num_object_clones":0,"num_object_copies":48609,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":795,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16203,"num_whiteouts":0,"num_read":1379,"num_read_kb":1208,"num_write":227613,"num_write_kb":7771150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41970,"num_bytes_recovered":2600441024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77650,"ondisk_log_size":77650,"up":474,"acting":473,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":475,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6154544,"kb_used_data":3211684,"kb_used_omap":23,"kb_used_meta":2942440,"kb_avail":743806672,"statfs":{"total":767960285184,"available":761658032128,"internally_reserved":0,"allocated":3288764416,"data_stored":3282099434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24148,"internal_metadata":3013058988},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,6,5,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-332857344,"num_objects":-5079,"num_object_clones":0,"num_object_copies":-15237,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-648,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5079,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":5079,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":635,"num_bytes_recovered":41549824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.595890"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":562,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.414208+0000","last_change":"2024-09-21T14:01:41.189741+0000","last_active":"2024-09-21T14:02:30.414208+0000","last_peered":"2024-09-21T14:02:30.414208+0000","last_clean":"2024-09-21T14:02:30.414208+0000","last_became_active":"2024-09-21T14:01:41.189398+0000","last_became_peered":"2024-09-21T14:01:41.189398+0000","last_unstale":"2024-09-21T14:02:30.414208+0000","last_undegraded":"2024-09-21T14:02:30.414208+0000","last_fullsized":"2024-09-21T14:02:30.414208+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:55.873301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:55.873301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"290'15","reported_seq":737,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.424438+0000","last_change":"2024-09-21T14:01:34.785322+0000","last_active":"2024-09-21T14:02:30.424438+0000","last_peered":"2024-09-21T14:02:30.424438+0000","last_clean":"2024-09-21T14:02:30.424438+0000","last_became_active":"2024-09-21T13:58:17.771220+0000","last_became_peered":"2024-09-21T13:58:17.771220+0000","last_unstale":"2024-09-21T14:02:30.424438+0000","last_undegraded":"2024-09-21T14:02:30.424438+0000","last_fullsized":"2024-09-21T14:02:30.424438+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'15","last_scrub_stamp":"2024-09-21T14:01:34.785267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:34.785267+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:36.742233+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":676,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T14:02:29.205437+0000","last_change":"2024-09-21T14:02:29.205437+0000","last_active":"2024-09-21T14:02:29.205437+0000","last_peered":"2024-09-21T14:02:29.205437+0000","last_clean":"2024-09-21T14:02:29.205437+0000","last_became_active":"2024-09-21T14:01:41.202168+0000","last_became_peered":"2024-09-21T14:01:41.202168+0000","last_unstale":"2024-09-21T14:02:29.205437+0000","last_undegraded":"2024-09-21T14:02:29.205437+0000","last_fullsized":"2024-09-21T14:02:29.205437+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:29.205343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:02:29.205343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:29.369063+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":654,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.424788+0000","last_change":"2024-09-21T14:01:41.203762+0000","last_active":"2024-09-21T14:02:30.424788+0000","last_peered":"2024-09-21T14:02:30.424788+0000","last_clean":"2024-09-21T14:02:30.424788+0000","last_became_active":"2024-09-21T14:01:41.203275+0000","last_became_peered":"2024-09-21T14:01:41.203275+0000","last_unstale":"2024-09-21T14:02:30.424788+0000","last_undegraded":"2024-09-21T14:02:30.424788+0000","last_fullsized":"2024-09-21T14:02:30.424788+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:00.320510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:00.320510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:46.565572+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":624,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.423530+0000","last_change":"2024-09-21T13:59:32.208701+0000","last_active":"2024-09-21T14:02:30.423530+0000","last_peered":"2024-09-21T14:02:30.423530+0000","last_clean":"2024-09-21T14:02:30.423530+0000","last_became_active":"2024-09-21T13:59:32.203340+0000","last_became_peered":"2024-09-21T13:59:32.203340+0000","last_unstale":"2024-09-21T14:02:30.423530+0000","last_undegraded":"2024-09-21T14:02:30.423530+0000","last_fullsized":"2024-09-21T14:02:30.423530+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:34.696780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:34.696780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":636,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.422262+0000","last_change":"2024-09-21T13:58:17.778322+0000","last_active":"2024-09-21T14:02:30.422262+0000","last_peered":"2024-09-21T14:02:30.422262+0000","last_clean":"2024-09-21T14:02:30.422262+0000","last_became_active":"2024-09-21T13:58:17.777893+0000","last_became_peered":"2024-09-21T13:58:17.777893+0000","last_unstale":"2024-09-21T14:02:30.422262+0000","last_undegraded":"2024-09-21T14:02:30.422262+0000","last_fullsized":"2024-09-21T14:02:30.422262+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:07.001135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:55:07.001135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:22.269018+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"290'27","reported_seq":610,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.430308+0000","last_change":"2024-09-21T13:58:11.718421+0000","last_active":"2024-09-21T14:02:30.430308+0000","last_peered":"2024-09-21T14:02:30.430308+0000","last_clean":"2024-09-21T14:02:30.430308+0000","last_became_active":"2024-09-21T13:57:12.703469+0000","last_became_peered":"2024-09-21T13:57:12.703469+0000","last_unstale":"2024-09-21T14:02:30.430308+0000","last_undegraded":"2024-09-21T14:02:30.430308+0000","last_fullsized":"2024-09-21T14:02:30.430308+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:58:11.718334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:11.718334+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":677,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.424349+0000","last_change":"2024-09-21T14:01:32.717551+0000","last_active":"2024-09-21T14:02:30.424349+0000","last_peered":"2024-09-21T14:02:30.424349+0000","last_clean":"2024-09-21T14:02:30.424349+0000","last_became_active":"2024-09-21T13:58:17.773464+0000","last_became_peered":"2024-09-21T13:58:17.773464+0000","last_unstale":"2024-09-21T14:02:30.424349+0000","last_undegraded":"2024-09-21T14:02:30.424349+0000","last_fullsized":"2024-09-21T14:02:30.424349+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:32.717480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:01:32.717480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:02.205485+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":629,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.413593+0000","last_change":"2024-09-21T14:01:41.194807+0000","last_active":"2024-09-21T14:02:30.413593+0000","last_peered":"2024-09-21T14:02:30.413593+0000","last_clean":"2024-09-21T14:02:30.413593+0000","last_became_active":"2024-09-21T14:01:41.194517+0000","last_became_peered":"2024-09-21T14:01:41.194517+0000","last_unstale":"2024-09-21T14:02:30.413593+0000","last_undegraded":"2024-09-21T14:02:30.413593+0000","last_fullsized":"2024-09-21T14:02:30.413593+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:31.127926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:31.127926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"290'15","reported_seq":703,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.424981+0000","last_change":"2024-09-21T14:00:02.475761+0000","last_active":"2024-09-21T14:02:30.424981+0000","last_peered":"2024-09-21T14:02:30.424981+0000","last_clean":"2024-09-21T14:02:30.424981+0000","last_became_active":"2024-09-21T13:57:12.681408+0000","last_became_peered":"2024-09-21T13:57:12.681408+0000","last_unstale":"2024-09-21T14:02:30.424981+0000","last_undegraded":"2024-09-21T14:02:30.424981+0000","last_fullsized":"2024-09-21T14:02:30.424981+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'15","last_scrub_stamp":"2024-09-21T14:00:02.475661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:00:02.475661+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:37.694022+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":570,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.409596+0000","last_change":"2024-09-21T14:01:41.188585+0000","last_active":"2024-09-21T14:02:30.409596+0000","last_peered":"2024-09-21T14:02:30.409596+0000","last_clean":"2024-09-21T14:02:30.409596+0000","last_became_active":"2024-09-21T14:01:41.188248+0000","last_became_peered":"2024-09-21T14:01:41.188248+0000","last_unstale":"2024-09-21T14:02:30.409596+0000","last_undegraded":"2024-09-21T14:02:30.409596+0000","last_fullsized":"2024-09-21T14:02:30.409596+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:21.284048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:56:21.284048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":605,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T14:02:28.393695+0000","last_change":"2024-09-21T13:59:32.202128+0000","last_active":"2024-09-21T14:02:28.393695+0000","last_peered":"2024-09-21T14:02:28.393695+0000","last_clean":"2024-09-21T14:02:28.393695+0000","last_became_active":"2024-09-21T13:59:32.201158+0000","last_became_peered":"2024-09-21T13:59:32.201158+0000","last_unstale":"2024-09-21T14:02:28.393695+0000","last_undegraded":"2024-09-21T14:02:28.393695+0000","last_fullsized":"2024-09-21T14:02:28.393695+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:25.792985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:54:25.792985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:46.991952+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":606,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.412936+0000","last_change":"2024-09-21T14:01:41.189797+0000","last_active":"2024-09-21T14:02:30.412936+0000","last_peered":"2024-09-21T14:02:30.412936+0000","last_clean":"2024-09-21T14:02:30.412936+0000","last_became_active":"2024-09-21T14:01:41.189373+0000","last_became_peered":"2024-09-21T14:01:41.189373+0000","last_unstale":"2024-09-21T14:02:30.412936+0000","last_undegraded":"2024-09-21T14:02:30.412936+0000","last_fullsized":"2024-09-21T14:02:30.412936+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:54.328275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:54.328275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"290'7","reported_seq":728,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.425181+0000","last_change":"2024-09-21T14:01:57.563250+0000","last_active":"2024-09-21T14:02:30.425181+0000","last_peered":"2024-09-21T14:02:30.425181+0000","last_clean":"2024-09-21T14:02:30.425181+0000","last_became_active":"2024-09-21T13:59:09.063053+0000","last_became_peered":"2024-09-21T13:59:09.063053+0000","last_unstale":"2024-09-21T14:02:30.425181+0000","last_undegraded":"2024-09-21T14:02:30.425181+0000","last_fullsized":"2024-09-21T14:02:30.425181+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'7","last_scrub_stamp":"2024-09-21T14:01:57.563184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:57.563184+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:19.737603+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":599,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.414470+0000","last_change":"2024-09-21T13:59:32.204565+0000","last_active":"2024-09-21T14:02:30.414470+0000","last_peered":"2024-09-21T14:02:30.414470+0000","last_clean":"2024-09-21T14:02:30.414470+0000","last_became_active":"2024-09-21T13:59:32.204174+0000","last_became_peered":"2024-09-21T13:59:32.204174+0000","last_unstale":"2024-09-21T14:02:30.414470+0000","last_undegraded":"2024-09-21T14:02:30.414470+0000","last_fullsized":"2024-09-21T14:02:30.414470+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:06.722293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:06.722293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":640,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.409456+0000","last_change":"2024-09-21T14:01:41.199085+0000","last_active":"2024-09-21T14:02:30.409456+0000","last_peered":"2024-09-21T14:02:30.409456+0000","last_clean":"2024-09-21T14:02:30.409456+0000","last_became_active":"2024-09-21T14:01:41.198672+0000","last_became_peered":"2024-09-21T14:01:41.198672+0000","last_unstale":"2024-09-21T14:02:30.409456+0000","last_undegraded":"2024-09-21T14:02:30.409456+0000","last_fullsized":"2024-09-21T14:02:30.409456+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:44.376215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:44.376215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":696,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.425818+0000","last_change":"2024-09-21T14:01:53.460257+0000","last_active":"2024-09-21T14:02:30.425818+0000","last_peered":"2024-09-21T14:02:30.425818+0000","last_clean":"2024-09-21T14:02:30.425818+0000","last_became_active":"2024-09-21T14:01:41.200360+0000","last_became_peered":"2024-09-21T14:01:41.200360+0000","last_unstale":"2024-09-21T14:02:30.425818+0000","last_undegraded":"2024-09-21T14:02:30.425818+0000","last_fullsized":"2024-09-21T14:02:30.425818+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:53.460180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:01:53.460180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:15.802657+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":606,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.425504+0000","last_change":"2024-09-21T13:59:32.205037+0000","last_active":"2024-09-21T14:02:30.425504+0000","last_peered":"2024-09-21T14:02:30.425504+0000","last_clean":"2024-09-21T14:02:30.425504+0000","last_became_active":"2024-09-21T13:59:32.204749+0000","last_became_peered":"2024-09-21T13:59:32.204749+0000","last_unstale":"2024-09-21T14:02:30.425504+0000","last_undegraded":"2024-09-21T14:02:30.425504+0000","last_fullsized":"2024-09-21T14:02:30.425504+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:17.361699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:17.361699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"290'27","reported_seq":671,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.412891+0000","last_change":"2024-09-21T14:01:27.893648+0000","last_active":"2024-09-21T14:02:30.412891+0000","last_peered":"2024-09-21T14:02:30.412891+0000","last_clean":"2024-09-21T14:02:30.412891+0000","last_became_active":"2024-09-21T13:57:12.680783+0000","last_became_peered":"2024-09-21T13:57:12.680783+0000","last_unstale":"2024-09-21T14:02:30.412891+0000","last_undegraded":"2024-09-21T14:02:30.412891+0000","last_fullsized":"2024-09-21T14:02:30.412891+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'27","last_scrub_stamp":"2024-09-21T14:01:27.893587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:27.893587+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:53.939497+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":662,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T14:02:28.395150+0000","last_change":"2024-09-21T14:01:54.024745+0000","last_active":"2024-09-21T14:02:28.395150+0000","last_peered":"2024-09-21T14:02:28.395150+0000","last_clean":"2024-09-21T14:02:28.395150+0000","last_became_active":"2024-09-21T13:49:02.066796+0000","last_became_peered":"2024-09-21T13:49:02.066796+0000","last_unstale":"2024-09-21T14:02:28.395150+0000","last_undegraded":"2024-09-21T14:02:28.395150+0000","last_fullsized":"2024-09-21T14:02:28.395150+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:54.024702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:01:54.024702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:06.646444+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":616,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.430661+0000","last_change":"2024-09-21T14:01:41.191486+0000","last_active":"2024-09-21T14:02:30.430661+0000","last_peered":"2024-09-21T14:02:30.430661+0000","last_clean":"2024-09-21T14:02:30.430661+0000","last_became_active":"2024-09-21T14:01:41.191116+0000","last_became_peered":"2024-09-21T14:01:41.191116+0000","last_unstale":"2024-09-21T14:02:30.430661+0000","last_undegraded":"2024-09-21T14:02:30.430661+0000","last_fullsized":"2024-09-21T14:02:30.430661+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:57.962266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:57.962266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:08.315476+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":674,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.425083+0000","last_change":"2024-09-21T14:00:51.495433+0000","last_active":"2024-09-21T14:02:30.425083+0000","last_peered":"2024-09-21T14:02:30.425083+0000","last_clean":"2024-09-21T14:02:30.425083+0000","last_became_active":"2024-09-21T13:52:04.316016+0000","last_became_peered":"2024-09-21T13:52:04.316016+0000","last_unstale":"2024-09-21T14:02:30.425083+0000","last_undegraded":"2024-09-21T14:02:30.425083+0000","last_fullsized":"2024-09-21T14:02:30.425083+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:00:51.495368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:00:51.495368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:10.415465+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"290'18","reported_seq":622,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.434820+0000","last_change":"2024-09-21T13:59:23.041396+0000","last_active":"2024-09-21T14:02:30.434820+0000","last_peered":"2024-09-21T14:02:30.434820+0000","last_clean":"2024-09-21T14:02:30.434820+0000","last_became_active":"2024-09-21T13:57:12.684454+0000","last_became_peered":"2024-09-21T13:57:12.684454+0000","last_unstale":"2024-09-21T14:02:30.434820+0000","last_undegraded":"2024-09-21T14:02:30.434820+0000","last_fullsized":"2024-09-21T14:02:30.434820+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'18","last_scrub_stamp":"2024-09-21T13:59:23.041311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:23.041311+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":661,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.425423+0000","last_change":"2024-09-21T13:58:16.393763+0000","last_active":"2024-09-21T14:02:30.425423+0000","last_peered":"2024-09-21T14:02:30.425423+0000","last_clean":"2024-09-21T14:02:30.425423+0000","last_became_active":"2024-09-21T13:57:12.681897+0000","last_became_peered":"2024-09-21T13:57:12.681897+0000","last_unstale":"2024-09-21T14:02:30.425423+0000","last_undegraded":"2024-09-21T14:02:30.425423+0000","last_fullsized":"2024-09-21T14:02:30.425423+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:16.393696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:16.393696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":583,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.415846+0000","last_change":"2024-09-21T14:01:41.182300+0000","last_active":"2024-09-21T14:02:30.415846+0000","last_peered":"2024-09-21T14:02:30.415846+0000","last_clean":"2024-09-21T14:02:30.415846+0000","last_became_active":"2024-09-21T14:01:41.181946+0000","last_became_peered":"2024-09-21T14:01:41.181946+0000","last_unstale":"2024-09-21T14:02:30.415846+0000","last_undegraded":"2024-09-21T14:02:30.415846+0000","last_fullsized":"2024-09-21T14:02:30.415846+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T13:57:26.536684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:26.536684+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":682,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T14:02:28.393577+0000","last_change":"2024-09-21T14:01:58.994867+0000","last_active":"2024-09-21T14:02:28.393577+0000","last_peered":"2024-09-21T14:02:28.393577+0000","last_clean":"2024-09-21T14:02:28.393577+0000","last_became_active":"2024-09-21T13:57:12.685742+0000","last_became_peered":"2024-09-21T13:57:12.685742+0000","last_unstale":"2024-09-21T14:02:28.393577+0000","last_undegraded":"2024-09-21T14:02:28.393577+0000","last_fullsized":"2024-09-21T14:02:28.393577+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T14:01:58.994811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:01:58.994811+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:17.687542+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"290'20","reported_seq":563,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.415623+0000","last_change":"2024-09-21T13:59:32.242771+0000","last_active":"2024-09-21T14:02:30.415623+0000","last_peered":"2024-09-21T14:02:30.415623+0000","last_clean":"2024-09-21T14:02:30.415623+0000","last_became_active":"2024-09-21T13:59:32.213865+0000","last_became_peered":"2024-09-21T13:59:32.213865+0000","last_unstale":"2024-09-21T14:02:30.415623+0000","last_undegraded":"2024-09-21T14:02:30.415623+0000","last_fullsized":"2024-09-21T14:02:30.415623+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:57:27.578738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:57:27.578738+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"300'12","reported_seq":297,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.418183+0000","last_change":"2024-09-21T13:59:36.012775+0000","last_active":"2024-09-21T14:02:30.418183+0000","last_peered":"2024-09-21T14:02:30.418183+0000","last_clean":"2024-09-21T14:02:30.418183+0000","last_became_active":"2024-09-21T13:59:32.205051+0000","last_became_peered":"2024-09-21T13:59:32.205051+0000","last_unstale":"2024-09-21T14:02:30.418183+0000","last_undegraded":"2024-09-21T14:02:30.418183+0000","last_fullsized":"2024-09-21T14:02:30.418183+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'12","last_scrub_stamp":"2024-09-21T13:59:36.012693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:36.012693+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:20.587646+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"75'79","reported_seq":79,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.413440+0000","last_change":"2024-09-21T14:01:41.248772+0000","last_active":"2024-09-21T14:02:30.413440+0000","last_peered":"2024-09-21T14:02:30.413440+0000","last_clean":"2024-09-21T14:02:30.413440+0000","last_became_active":"2024-09-21T14:01:41.246888+0000","last_became_peered":"2024-09-21T14:01:41.246888+0000","last_unstale":"2024-09-21T14:02:30.413440+0000","last_undegraded":"2024-09-21T14:02:30.413440+0000","last_fullsized":"2024-09-21T14:02:30.413440+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53669,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"373'14196","reported_seq":14435,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:31.164893+0000","last_change":"2024-09-21T13:57:53.208687+0000","last_active":"2024-09-21T14:02:31.164893+0000","last_peered":"2024-09-21T14:02:31.164893+0000","last_clean":"2024-09-21T14:02:31.164893+0000","last_became_active":"2024-09-21T13:57:53.208352+0000","last_became_peered":"2024-09-21T13:57:53.208352+0000","last_unstale":"2024-09-21T14:02:31.164893+0000","last_undegraded":"2024-09-21T14:02:31.164893+0000","last_fullsized":"2024-09-21T14:02:31.164893+0000","mapping_epoch":260,"log_start":"312'9815","ondisk_log_start":"312'9815","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4381,"log_dups_size":0,"ondisk_log_size":4381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66846720,"num_objects":1020,"num_object_clones":0,"num_object_copies":3060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14196,"num_write_kb":486912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"290'27","reported_seq":720,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T14:02:28.395333+0000","last_change":"2024-09-21T13:58:45.984691+0000","last_active":"2024-09-21T14:02:28.395333+0000","last_peered":"2024-09-21T14:02:28.395333+0000","last_clean":"2024-09-21T14:02:28.395333+0000","last_became_active":"2024-09-21T13:57:12.695609+0000","last_became_peered":"2024-09-21T13:57:12.695609+0000","last_unstale":"2024-09-21T14:02:28.395333+0000","last_undegraded":"2024-09-21T14:02:28.395333+0000","last_fullsized":"2024-09-21T14:02:28.395333+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:58:45.984575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:45.984575+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":608,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.422576+0000","last_change":"2024-09-21T13:59:32.205723+0000","last_active":"2024-09-21T14:02:30.422576+0000","last_peered":"2024-09-21T14:02:30.422576+0000","last_clean":"2024-09-21T14:02:30.422576+0000","last_became_active":"2024-09-21T13:59:32.205350+0000","last_became_peered":"2024-09-21T13:59:32.205350+0000","last_unstale":"2024-09-21T14:02:30.422576+0000","last_undegraded":"2024-09-21T14:02:30.422576+0000","last_fullsized":"2024-09-21T14:02:30.422576+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:32.668851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:32.668851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"346'10","reported_seq":853,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:33.288310+0000","last_change":"2024-09-21T14:01:55.521513+0000","last_active":"2024-09-21T14:02:33.288310+0000","last_peered":"2024-09-21T14:02:33.288310+0000","last_clean":"2024-09-21T14:02:33.288310+0000","last_became_active":"2024-09-21T14:01:41.200218+0000","last_became_peered":"2024-09-21T14:01:41.200218+0000","last_unstale":"2024-09-21T14:02:33.288310+0000","last_undegraded":"2024-09-21T14:02:33.288310+0000","last_fullsized":"2024-09-21T14:02:33.288310+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'10","last_scrub_stamp":"2024-09-21T14:01:55.521438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:01:55.521438+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:55.829821+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":535,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.413162+0000","last_change":"2024-09-21T14:01:41.207310+0000","last_active":"2024-09-21T14:02:30.413162+0000","last_peered":"2024-09-21T14:02:30.413162+0000","last_clean":"2024-09-21T14:02:30.413162+0000","last_became_active":"2024-09-21T14:01:41.206720+0000","last_became_peered":"2024-09-21T14:01:41.206720+0000","last_unstale":"2024-09-21T14:02:30.413162+0000","last_undegraded":"2024-09-21T14:02:30.413162+0000","last_fullsized":"2024-09-21T14:02:30.413162+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:52.244483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:52.244483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"75'79","reported_seq":132,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.425701+0000","last_change":"2024-09-21T14:02:06.395083+0000","last_active":"2024-09-21T14:02:30.425701+0000","last_peered":"2024-09-21T14:02:30.425701+0000","last_clean":"2024-09-21T14:02:30.425701+0000","last_became_active":"2024-09-21T14:00:41.137756+0000","last_became_peered":"2024-09-21T14:00:41.137756+0000","last_unstale":"2024-09-21T14:02:30.425701+0000","last_undegraded":"2024-09-21T14:02:30.425701+0000","last_fullsized":"2024-09-21T14:02:30.425701+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:02:06.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:02:06.395011+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:29.136654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"373'14174","reported_seq":18846,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:31.180772+0000","last_change":"2024-09-21T14:02:28.496980+0000","last_active":"2024-09-21T14:02:31.180772+0000","last_peered":"2024-09-21T14:02:31.180772+0000","last_clean":"2024-09-21T14:02:31.180772+0000","last_became_active":"2024-09-21T14:02:28.412817+0000","last_became_peered":"2024-09-21T14:02:28.412817+0000","last_unstale":"2024-09-21T14:02:31.180772+0000","last_undegraded":"2024-09-21T14:02:31.180772+0000","last_fullsized":"2024-09-21T14:02:31.180772+0000","mapping_epoch":371,"log_start":"311'9620","ondisk_log_start":"311'9620","created":260,"last_epoch_clean":372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4554,"log_dups_size":0,"ondisk_log_size":4554,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67829784,"num_objects":1036,"num_object_clones":0,"num_object_copies":3108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1036,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14174,"num_write_kb":486657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2195,"num_bytes_recovered":143720472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"290'30","reported_seq":632,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.422676+0000","last_change":"2024-09-21T13:59:32.246090+0000","last_active":"2024-09-21T14:02:30.422676+0000","last_peered":"2024-09-21T14:02:30.422676+0000","last_clean":"2024-09-21T14:02:30.422676+0000","last_became_active":"2024-09-21T13:59:32.205887+0000","last_became_peered":"2024-09-21T13:59:32.205887+0000","last_unstale":"2024-09-21T14:02:30.422676+0000","last_undegraded":"2024-09-21T14:02:30.422676+0000","last_fullsized":"2024-09-21T14:02:30.422676+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'18","last_scrub_stamp":"2024-09-21T13:54:19.001030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:19.001030+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:40.231306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":560,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.430964+0000","last_change":"2024-09-21T13:59:32.212559+0000","last_active":"2024-09-21T14:02:30.430964+0000","last_peered":"2024-09-21T14:02:30.430964+0000","last_clean":"2024-09-21T14:02:30.430964+0000","last_became_active":"2024-09-21T13:59:32.211004+0000","last_became_peered":"2024-09-21T13:59:32.211004+0000","last_unstale":"2024-09-21T14:02:30.430964+0000","last_undegraded":"2024-09-21T14:02:30.430964+0000","last_fullsized":"2024-09-21T14:02:30.430964+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:25.736461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:25.736461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":678,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.424448+0000","last_change":"2024-09-21T13:59:34.293330+0000","last_active":"2024-09-21T14:02:30.424448+0000","last_peered":"2024-09-21T14:02:30.424448+0000","last_clean":"2024-09-21T14:02:30.424448+0000","last_became_active":"2024-09-21T13:59:32.220304+0000","last_became_peered":"2024-09-21T13:59:32.220304+0000","last_unstale":"2024-09-21T14:02:30.424448+0000","last_undegraded":"2024-09-21T14:02:30.424448+0000","last_fullsized":"2024-09-21T14:02:30.424448+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:34.292892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:34.292892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:15.468420+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":658,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.424945+0000","last_change":"2024-09-21T13:59:23.334230+0000","last_active":"2024-09-21T14:02:30.424945+0000","last_peered":"2024-09-21T14:02:30.424945+0000","last_clean":"2024-09-21T14:02:30.424945+0000","last_became_active":"2024-09-21T13:57:12.680372+0000","last_became_peered":"2024-09-21T13:57:12.680372+0000","last_unstale":"2024-09-21T14:02:30.424945+0000","last_undegraded":"2024-09-21T14:02:30.424945+0000","last_fullsized":"2024-09-21T14:02:30.424945+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:23.334162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:23.334162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:17.515611+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"75'79","reported_seq":135,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.421857+0000","last_change":"2024-09-21T14:01:42.177646+0000","last_active":"2024-09-21T14:02:30.421857+0000","last_peered":"2024-09-21T14:02:30.421857+0000","last_clean":"2024-09-21T14:02:30.421857+0000","last_became_active":"2024-09-21T14:01:42.173042+0000","last_became_peered":"2024-09-21T14:01:42.173042+0000","last_unstale":"2024-09-21T14:02:30.421857+0000","last_undegraded":"2024-09-21T14:02:30.421857+0000","last_fullsized":"2024-09-21T14:02:30.421857+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:02:42.254915+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":709030,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"373'14066","reported_seq":21239,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:31.135545+0000","last_change":"2024-09-21T14:01:00.058151+0000","last_active":"2024-09-21T14:02:31.135545+0000","last_peered":"2024-09-21T14:02:31.135545+0000","last_clean":"2024-09-21T14:02:31.135545+0000","last_became_active":"2024-09-21T14:01:00.026605+0000","last_became_peered":"2024-09-21T14:01:00.026605+0000","last_unstale":"2024-09-21T14:02:31.135545+0000","last_undegraded":"2024-09-21T14:02:31.135545+0000","last_fullsized":"2024-09-21T14:02:31.135545+0000","mapping_epoch":329,"log_start":"311'9642","ondisk_log_start":"311'9642","created":260,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4424,"log_dups_size":0,"ondisk_log_size":4424,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64487424,"num_objects":984,"num_object_clones":0,"num_object_copies":2952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14066,"num_write_kb":481600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3456,"num_bytes_recovered":226492416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"290'28","reported_seq":767,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.424464+0000","last_change":"2024-09-21T14:02:02.471226+0000","last_active":"2024-09-21T14:02:30.424464+0000","last_peered":"2024-09-21T14:02:30.424464+0000","last_clean":"2024-09-21T14:02:30.424464+0000","last_became_active":"2024-09-21T14:01:41.200859+0000","last_became_peered":"2024-09-21T14:01:41.200859+0000","last_unstale":"2024-09-21T14:02:30.424464+0000","last_undegraded":"2024-09-21T14:02:30.424464+0000","last_fullsized":"2024-09-21T14:02:30.424464+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'28","last_scrub_stamp":"2024-09-21T14:02:02.471133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:02.471133+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:14.278651+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":664,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T14:02:28.394190+0000","last_change":"2024-09-21T13:57:38.046718+0000","last_active":"2024-09-21T14:02:28.394190+0000","last_peered":"2024-09-21T14:02:28.394190+0000","last_clean":"2024-09-21T14:02:28.394190+0000","last_became_active":"2024-09-21T13:48:58.053533+0000","last_became_peered":"2024-09-21T13:48:58.053533+0000","last_unstale":"2024-09-21T14:02:28.394190+0000","last_undegraded":"2024-09-21T14:02:28.394190+0000","last_fullsized":"2024-09-21T14:02:28.394190+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:57:38.046627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:38.046627+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:41.742834+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"241'10","reported_seq":704,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.422785+0000","last_change":"2024-09-21T13:58:06.789629+0000","last_active":"2024-09-21T14:02:30.422785+0000","last_peered":"2024-09-21T14:02:30.422785+0000","last_clean":"2024-09-21T14:02:30.422785+0000","last_became_active":"2024-09-21T13:57:12.686651+0000","last_became_peered":"2024-09-21T13:57:12.686651+0000","last_unstale":"2024-09-21T14:02:30.422785+0000","last_undegraded":"2024-09-21T14:02:30.422785+0000","last_fullsized":"2024-09-21T14:02:30.422785+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'10","last_scrub_stamp":"2024-09-21T13:58:06.789560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:06.789560+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":624,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.416259+0000","last_change":"2024-09-21T13:59:32.210257+0000","last_active":"2024-09-21T14:02:30.416259+0000","last_peered":"2024-09-21T14:02:30.416259+0000","last_clean":"2024-09-21T14:02:30.416259+0000","last_became_active":"2024-09-21T13:59:32.209615+0000","last_became_peered":"2024-09-21T13:59:32.209615+0000","last_unstale":"2024-09-21T14:02:30.416259+0000","last_undegraded":"2024-09-21T14:02:30.416259+0000","last_fullsized":"2024-09-21T14:02:30.416259+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:37.548064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:37.548064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"75'79","reported_seq":705,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.417793+0000","last_change":"2024-09-21T14:02:05.693250+0000","last_active":"2024-09-21T14:02:30.417793+0000","last_peered":"2024-09-21T14:02:30.417793+0000","last_clean":"2024-09-21T14:02:30.417793+0000","last_became_active":"2024-09-21T14:00:41.083425+0000","last_became_peered":"2024-09-21T14:00:41.083425+0000","last_unstale":"2024-09-21T14:02:30.417793+0000","last_undegraded":"2024-09-21T14:02:30.417793+0000","last_fullsized":"2024-09-21T14:02:30.417793+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:02:05.693193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:02:05.693193+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:28.620866+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.f","version":"373'14306","reported_seq":14532,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.854326+0000","last_change":"2024-09-21T14:01:41.336305+0000","last_active":"2024-09-21T14:02:30.854326+0000","last_peered":"2024-09-21T14:02:30.854326+0000","last_clean":"2024-09-21T14:02:30.854326+0000","last_became_active":"2024-09-21T14:01:41.202052+0000","last_became_peered":"2024-09-21T14:01:41.202052+0000","last_unstale":"2024-09-21T14:02:30.854326+0000","last_undegraded":"2024-09-21T14:02:30.854326+0000","last_fullsized":"2024-09-21T14:02:30.854326+0000","mapping_epoch":345,"log_start":"310'9302","ondisk_log_start":"310'9302","created":260,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":5004,"log_dups_size":0,"ondisk_log_size":5004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67633152,"num_objects":1032,"num_object_clones":0,"num_object_copies":3096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14306,"num_write_kb":490816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"290'20","reported_seq":700,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T14:02:28.393502+0000","last_change":"2024-09-21T14:01:41.187743+0000","last_active":"2024-09-21T14:02:28.393502+0000","last_peered":"2024-09-21T14:02:28.393502+0000","last_clean":"2024-09-21T14:02:28.393502+0000","last_became_active":"2024-09-21T14:01:41.187060+0000","last_became_peered":"2024-09-21T14:01:41.187060+0000","last_unstale":"2024-09-21T14:02:28.393502+0000","last_undegraded":"2024-09-21T14:02:28.393502+0000","last_fullsized":"2024-09-21T14:02:28.393502+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:54:19.593611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:19.593611+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:45.999996+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"61'1","reported_seq":597,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.413254+0000","last_change":"2024-09-21T14:01:41.195580+0000","last_active":"2024-09-21T14:02:30.413254+0000","last_peered":"2024-09-21T14:02:30.413254+0000","last_clean":"2024-09-21T14:02:30.413254+0000","last_became_active":"2024-09-21T14:01:41.195268+0000","last_became_peered":"2024-09-21T14:01:41.195268+0000","last_unstale":"2024-09-21T14:02:30.413254+0000","last_undegraded":"2024-09-21T14:02:30.413254+0000","last_fullsized":"2024-09-21T14:02:30.413254+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:58:48.407734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:48.407734+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"346'15","reported_seq":290,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.417647+0000","last_change":"2024-09-21T14:01:41.209662+0000","last_active":"2024-09-21T14:02:30.417647+0000","last_peered":"2024-09-21T14:02:30.417647+0000","last_clean":"2024-09-21T14:02:30.417647+0000","last_became_active":"2024-09-21T14:01:41.209296+0000","last_became_peered":"2024-09-21T14:01:41.209296+0000","last_unstale":"2024-09-21T14:02:30.417647+0000","last_undegraded":"2024-09-21T14:02:30.417647+0000","last_fullsized":"2024-09-21T14:02:30.417647+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'12","last_scrub_stamp":"2024-09-21T13:57:30.133392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:30.133392+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:41.760065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":478,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.422757+0000","last_change":"2024-09-21T13:58:58.730439+0000","last_active":"2024-09-21T14:02:30.422757+0000","last_peered":"2024-09-21T14:02:30.422757+0000","last_clean":"2024-09-21T14:02:30.422757+0000","last_became_active":"2024-09-21T13:57:12.704897+0000","last_became_peered":"2024-09-21T13:57:12.704897+0000","last_unstale":"2024-09-21T14:02:30.422757+0000","last_undegraded":"2024-09-21T14:02:30.422757+0000","last_fullsized":"2024-09-21T14:02:30.422757+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:58.730368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:58.730368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"75'79","reported_seq":705,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.417802+0000","last_change":"2024-09-21T14:02:03.737005+0000","last_active":"2024-09-21T14:02:30.417802+0000","last_peered":"2024-09-21T14:02:30.417802+0000","last_clean":"2024-09-21T14:02:30.417802+0000","last_became_active":"2024-09-21T14:00:41.073047+0000","last_became_peered":"2024-09-21T14:00:41.073047+0000","last_unstale":"2024-09-21T14:02:30.417802+0000","last_undegraded":"2024-09-21T14:02:30.417802+0000","last_fullsized":"2024-09-21T14:02:30.417802+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:02:03.736928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:02:03.736928+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:21.563427+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"373'13944","reported_seq":23737,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:31.165763+0000","last_change":"2024-09-21T14:00:42.188751+0000","last_active":"2024-09-21T14:02:31.165763+0000","last_peered":"2024-09-21T14:02:31.165763+0000","last_clean":"2024-09-21T14:02:31.165763+0000","last_became_active":"2024-09-21T14:00:42.129476+0000","last_became_peered":"2024-09-21T14:00:42.129476+0000","last_unstale":"2024-09-21T14:02:31.165763+0000","last_undegraded":"2024-09-21T14:02:31.165763+0000","last_fullsized":"2024-09-21T14:02:31.165763+0000","mapping_epoch":321,"log_start":"310'9369","ondisk_log_start":"310'9369","created":260,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4575,"log_dups_size":0,"ondisk_log_size":4575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64356352,"num_objects":982,"num_object_clones":0,"num_object_copies":2946,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13944,"num_write_kb":477632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4765,"num_bytes_recovered":312213504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"290'29","reported_seq":603,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.409729+0000","last_change":"2024-09-21T14:01:41.188261+0000","last_active":"2024-09-21T14:02:30.409729+0000","last_peered":"2024-09-21T14:02:30.409729+0000","last_clean":"2024-09-21T14:02:30.409729+0000","last_became_active":"2024-09-21T14:01:41.186960+0000","last_became_peered":"2024-09-21T14:01:41.186960+0000","last_unstale":"2024-09-21T14:02:30.409729+0000","last_undegraded":"2024-09-21T14:02:30.409729+0000","last_fullsized":"2024-09-21T14:02:30.409729+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-21T13:56:55.319759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:56:55.319759+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":677,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T14:02:28.395390+0000","last_change":"2024-09-21T14:01:41.185706+0000","last_active":"2024-09-21T14:02:28.395390+0000","last_peered":"2024-09-21T14:02:28.395390+0000","last_clean":"2024-09-21T14:02:28.395390+0000","last_became_active":"2024-09-21T14:01:41.185400+0000","last_became_peered":"2024-09-21T14:01:41.185400+0000","last_unstale":"2024-09-21T14:02:28.395390+0000","last_undegraded":"2024-09-21T14:02:28.395390+0000","last_fullsized":"2024-09-21T14:02:28.395390+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:16.712990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:16.712990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":247,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.417290+0000","last_change":"2024-09-21T14:01:52.671029+0000","last_active":"2024-09-21T14:02:30.417290+0000","last_peered":"2024-09-21T14:02:30.417290+0000","last_clean":"2024-09-21T14:02:30.417290+0000","last_became_active":"2024-09-21T14:01:41.193973+0000","last_became_peered":"2024-09-21T14:01:41.193973+0000","last_unstale":"2024-09-21T14:02:30.417290+0000","last_undegraded":"2024-09-21T14:02:30.417290+0000","last_fullsized":"2024-09-21T14:02:30.417290+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:52.670974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:01:52.670974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:58.750020+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":668,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T14:02:28.393317+0000","last_change":"2024-09-21T13:58:43.952014+0000","last_active":"2024-09-21T14:02:28.393317+0000","last_peered":"2024-09-21T14:02:28.393317+0000","last_clean":"2024-09-21T14:02:28.393317+0000","last_became_active":"2024-09-21T13:58:17.757628+0000","last_became_peered":"2024-09-21T13:58:17.757628+0000","last_unstale":"2024-09-21T14:02:28.393317+0000","last_undegraded":"2024-09-21T14:02:28.393317+0000","last_fullsized":"2024-09-21T14:02:28.393317+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:43.951941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:43.951941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"75'79","reported_seq":79,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.409610+0000","last_change":"2024-09-21T14:01:41.199377+0000","last_active":"2024-09-21T14:02:30.409610+0000","last_peered":"2024-09-21T14:02:30.409610+0000","last_clean":"2024-09-21T14:02:30.409610+0000","last_became_active":"2024-09-21T14:01:41.198750+0000","last_became_peered":"2024-09-21T14:01:41.198750+0000","last_unstale":"2024-09-21T14:02:30.409610+0000","last_undegraded":"2024-09-21T14:02:30.409610+0000","last_fullsized":"2024-09-21T14:02:30.409610+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:02:43.568139+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"373'14252","reported_seq":25076,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:31.154660+0000","last_change":"2024-09-21T14:01:44.191023+0000","last_active":"2024-09-21T14:02:31.154660+0000","last_peered":"2024-09-21T14:02:31.154660+0000","last_clean":"2024-09-21T14:02:31.154660+0000","last_became_active":"2024-09-21T14:01:44.174262+0000","last_became_peered":"2024-09-21T14:01:44.174262+0000","last_unstale":"2024-09-21T14:02:31.154660+0000","last_undegraded":"2024-09-21T14:02:31.154660+0000","last_fullsized":"2024-09-21T14:02:31.154660+0000","mapping_epoch":348,"log_start":"311'9631","ondisk_log_start":"311'9631","created":260,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4621,"log_dups_size":0,"ondisk_log_size":4621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66846720,"num_objects":1020,"num_object_clones":0,"num_object_copies":3060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14252,"num_write_kb":488704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5260,"num_bytes_recovered":344588288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"290'44","reported_seq":906,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T14:02:28.395241+0000","last_change":"2024-09-21T13:58:56.076153+0000","last_active":"2024-09-21T14:02:28.395241+0000","last_peered":"2024-09-21T14:02:28.395241+0000","last_clean":"2024-09-21T14:02:28.395241+0000","last_became_active":"2024-09-21T13:57:12.684690+0000","last_became_peered":"2024-09-21T13:57:12.684690+0000","last_unstale":"2024-09-21T14:02:28.395241+0000","last_undegraded":"2024-09-21T14:02:28.395241+0000","last_fullsized":"2024-09-21T14:02:28.395241+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'32","last_scrub_stamp":"2024-09-21T13:58:56.076074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:56.076074+0000","objects_scrubbed":10,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":686,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.425601+0000","last_change":"2024-09-21T14:01:58.524159+0000","last_active":"2024-09-21T14:02:30.425601+0000","last_peered":"2024-09-21T14:02:30.425601+0000","last_clean":"2024-09-21T14:02:30.425601+0000","last_became_active":"2024-09-21T13:58:17.770937+0000","last_became_peered":"2024-09-21T13:58:17.770937+0000","last_unstale":"2024-09-21T14:02:30.425601+0000","last_undegraded":"2024-09-21T14:02:30.425601+0000","last_fullsized":"2024-09-21T14:02:30.425601+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:58.524114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:01:58.524114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:07.944133+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":616,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.413346+0000","last_change":"2024-09-21T14:01:41.249061+0000","last_active":"2024-09-21T14:02:30.413346+0000","last_peered":"2024-09-21T14:02:30.413346+0000","last_clean":"2024-09-21T14:02:30.413346+0000","last_became_active":"2024-09-21T14:01:41.247642+0000","last_became_peered":"2024-09-21T14:01:41.247642+0000","last_unstale":"2024-09-21T14:02:30.413346+0000","last_undegraded":"2024-09-21T14:02:30.413346+0000","last_fullsized":"2024-09-21T14:02:30.413346+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-21T13:59:22.186393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:59:22.186393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":595,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T14:02:28.394940+0000","last_change":"2024-09-21T13:55:07.021128+0000","last_active":"2024-09-21T14:02:28.394940+0000","last_peered":"2024-09-21T14:02:28.394940+0000","last_clean":"2024-09-21T14:02:28.394940+0000","last_became_active":"2024-09-21T13:50:29.688249+0000","last_became_peered":"2024-09-21T13:50:29.688249+0000","last_unstale":"2024-09-21T14:02:28.394940+0000","last_undegraded":"2024-09-21T14:02:28.394940+0000","last_fullsized":"2024-09-21T14:02:28.394940+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:07.021059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:55:07.021059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:47.974697+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":587,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.422639+0000","last_change":"2024-09-21T13:59:32.209357+0000","last_active":"2024-09-21T14:02:30.422639+0000","last_peered":"2024-09-21T14:02:30.422639+0000","last_clean":"2024-09-21T14:02:30.422639+0000","last_became_active":"2024-09-21T13:59:32.208980+0000","last_became_peered":"2024-09-21T13:59:32.208980+0000","last_unstale":"2024-09-21T14:02:30.422639+0000","last_undegraded":"2024-09-21T14:02:30.422639+0000","last_fullsized":"2024-09-21T14:02:30.422639+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:16.797227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:16.797227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"75'79","reported_seq":711,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.414305+0000","last_change":"2024-09-21T14:00:41.077286+0000","last_active":"2024-09-21T14:02:30.414305+0000","last_peered":"2024-09-21T14:02:30.414305+0000","last_clean":"2024-09-21T14:02:30.414305+0000","last_became_active":"2024-09-21T14:00:41.076948+0000","last_became_peered":"2024-09-21T14:00:41.076948+0000","last_unstale":"2024-09-21T14:02:30.414305+0000","last_undegraded":"2024-09-21T14:02:30.414305+0000","last_fullsized":"2024-09-21T14:02:30.414305+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:02:44.823429+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.a","version":"373'13862","reported_seq":14796,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.839596+0000","last_change":"2024-09-21T14:01:41.339174+0000","last_active":"2024-09-21T14:02:30.839596+0000","last_peered":"2024-09-21T14:02:30.839596+0000","last_clean":"2024-09-21T14:02:30.839596+0000","last_became_active":"2024-09-21T14:01:41.190267+0000","last_became_peered":"2024-09-21T14:01:41.190267+0000","last_unstale":"2024-09-21T14:02:30.839596+0000","last_undegraded":"2024-09-21T14:02:30.839596+0000","last_fullsized":"2024-09-21T14:02:30.839596+0000","mapping_epoch":345,"log_start":"309'8874","ondisk_log_start":"309'8874","created":260,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4988,"log_dups_size":0,"ondisk_log_size":4988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536000,"num_objects":1000,"num_object_clones":0,"num_object_copies":3000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13862,"num_write_kb":475584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5098,"num_bytes_recovered":241565696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"290'17","reported_seq":703,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T14:02:28.395483+0000","last_change":"2024-09-21T14:01:41.200809+0000","last_active":"2024-09-21T14:02:28.395483+0000","last_peered":"2024-09-21T14:02:28.395483+0000","last_clean":"2024-09-21T14:02:28.395483+0000","last_became_active":"2024-09-21T14:01:41.200365+0000","last_became_peered":"2024-09-21T14:01:41.200365+0000","last_unstale":"2024-09-21T14:02:28.395483+0000","last_undegraded":"2024-09-21T14:02:28.395483+0000","last_fullsized":"2024-09-21T14:02:28.395483+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'13","last_scrub_stamp":"2024-09-21T13:58:11.619848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:11.619848+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":704,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T14:02:28.392821+0000","last_change":"2024-09-21T13:59:29.454265+0000","last_active":"2024-09-21T14:02:28.392821+0000","last_peered":"2024-09-21T14:02:28.392821+0000","last_clean":"2024-09-21T14:02:28.392821+0000","last_became_active":"2024-09-21T13:58:17.759671+0000","last_became_peered":"2024-09-21T13:58:17.759671+0000","last_unstale":"2024-09-21T14:02:28.392821+0000","last_undegraded":"2024-09-21T14:02:28.392821+0000","last_fullsized":"2024-09-21T14:02:28.392821+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:29.454085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:59:29.454085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":596,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.429338+0000","last_change":"2024-09-21T13:58:41.644876+0000","last_active":"2024-09-21T14:02:30.429338+0000","last_peered":"2024-09-21T14:02:30.429338+0000","last_clean":"2024-09-21T14:02:30.429338+0000","last_became_active":"2024-09-21T13:57:12.682147+0000","last_became_peered":"2024-09-21T13:57:12.682147+0000","last_unstale":"2024-09-21T14:02:30.429338+0000","last_undegraded":"2024-09-21T14:02:30.429338+0000","last_fullsized":"2024-09-21T14:02:30.429338+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:58:41.644789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:58:41.644789+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:13.796854+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":574,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.414263+0000","last_change":"2024-09-21T14:01:50.880840+0000","last_active":"2024-09-21T14:02:30.414263+0000","last_peered":"2024-09-21T14:02:30.414263+0000","last_clean":"2024-09-21T14:02:30.414263+0000","last_became_active":"2024-09-21T14:01:41.182000+0000","last_became_peered":"2024-09-21T14:01:41.182000+0000","last_unstale":"2024-09-21T14:02:30.414263+0000","last_undegraded":"2024-09-21T14:02:30.414263+0000","last_fullsized":"2024-09-21T14:02:30.414263+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:50.880471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:01:50.880471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:14.118692+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":632,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.416483+0000","last_change":"2024-09-21T13:59:32.222791+0000","last_active":"2024-09-21T14:02:30.416483+0000","last_peered":"2024-09-21T14:02:30.416483+0000","last_clean":"2024-09-21T14:02:30.416483+0000","last_became_active":"2024-09-21T13:59:32.221502+0000","last_became_peered":"2024-09-21T13:59:32.221502+0000","last_unstale":"2024-09-21T14:02:30.416483+0000","last_undegraded":"2024-09-21T14:02:30.416483+0000","last_fullsized":"2024-09-21T14:02:30.416483+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:14.673582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:14.673582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"75'79","reported_seq":114,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T14:02:28.394735+0000","last_change":"2024-09-21T14:00:41.071631+0000","last_active":"2024-09-21T14:02:28.394735+0000","last_peered":"2024-09-21T14:02:28.394735+0000","last_clean":"2024-09-21T14:02:28.394735+0000","last_became_active":"2024-09-21T14:00:41.071042+0000","last_became_peered":"2024-09-21T14:00:41.071042+0000","last_unstale":"2024-09-21T14:02:28.394735+0000","last_undegraded":"2024-09-21T14:02:28.394735+0000","last_fullsized":"2024-09-21T14:02:28.394735+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:02:42.091028+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.b","version":"373'14379","reported_seq":18586,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:33.546381+0000","last_change":"2024-09-21T14:01:54.238547+0000","last_active":"2024-09-21T14:02:33.546381+0000","last_peered":"2024-09-21T14:02:33.546381+0000","last_clean":"2024-09-21T14:02:33.546381+0000","last_became_active":"2024-09-21T14:01:41.210137+0000","last_became_peered":"2024-09-21T14:01:41.210137+0000","last_unstale":"2024-09-21T14:02:33.546381+0000","last_undegraded":"2024-09-21T14:02:33.546381+0000","last_fullsized":"2024-09-21T14:02:33.546381+0000","mapping_epoch":345,"log_start":"305'8432","ondisk_log_start":"305'8432","created":260,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"356'12122","last_scrub_stamp":"2024-09-21T14:01:54.238344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T14:01:54.238344+0000","objects_scrubbed":3072,"log_size":5947,"log_dups_size":0,"ondisk_log_size":5947,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:15.605090+0000","scrub_duration":597,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52756480,"num_objects":805,"num_object_clones":0,"num_object_copies":2415,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14379,"num_write_kb":485888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1910,"num_bytes_recovered":123994112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"290'24","reported_seq":754,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T14:02:28.394029+0000","last_change":"2024-09-21T14:01:41.185105+0000","last_active":"2024-09-21T14:02:28.394029+0000","last_peered":"2024-09-21T14:02:28.394029+0000","last_clean":"2024-09-21T14:02:28.394029+0000","last_became_active":"2024-09-21T14:01:41.184728+0000","last_became_peered":"2024-09-21T14:01:41.184728+0000","last_unstale":"2024-09-21T14:02:28.394029+0000","last_undegraded":"2024-09-21T14:02:28.394029+0000","last_fullsized":"2024-09-21T14:02:28.394029+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'24","last_scrub_stamp":"2024-09-21T13:59:28.458768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:28.458768+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":607,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.424598+0000","last_change":"2024-09-21T13:59:59.450613+0000","last_active":"2024-09-21T14:02:30.424598+0000","last_peered":"2024-09-21T14:02:30.424598+0000","last_clean":"2024-09-21T14:02:30.424598+0000","last_became_active":"2024-09-21T13:57:12.681030+0000","last_became_peered":"2024-09-21T13:57:12.681030+0000","last_unstale":"2024-09-21T14:02:30.424598+0000","last_undegraded":"2024-09-21T14:02:30.424598+0000","last_fullsized":"2024-09-21T14:02:30.424598+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:59.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:59:59.450546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":1345,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.413974+0000","last_change":"2024-09-21T13:59:32.218796+0000","last_active":"2024-09-21T14:02:30.413974+0000","last_peered":"2024-09-21T14:02:30.413974+0000","last_clean":"2024-09-21T14:02:30.413974+0000","last_became_active":"2024-09-21T13:59:32.203361+0000","last_became_peered":"2024-09-21T13:59:32.203361+0000","last_unstale":"2024-09-21T14:02:30.413974+0000","last_undegraded":"2024-09-21T14:02:30.413974+0000","last_fullsized":"2024-09-21T14:02:30.413974+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-21T13:56:29.617143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:56:29.617143+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047048000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":185,"num_read_kb":180,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":587,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.414288+0000","last_change":"2024-09-21T14:01:41.212595+0000","last_active":"2024-09-21T14:02:30.414288+0000","last_peered":"2024-09-21T14:02:30.414288+0000","last_clean":"2024-09-21T14:02:30.414288+0000","last_became_active":"2024-09-21T14:01:41.200531+0000","last_became_peered":"2024-09-21T14:01:41.200531+0000","last_unstale":"2024-09-21T14:02:30.414288+0000","last_undegraded":"2024-09-21T14:02:30.414288+0000","last_fullsized":"2024-09-21T14:02:30.414288+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:33.316230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:56:33.316230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":689,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T14:02:28.394417+0000","last_change":"2024-09-21T14:01:41.184939+0000","last_active":"2024-09-21T14:02:28.394417+0000","last_peered":"2024-09-21T14:02:28.394417+0000","last_clean":"2024-09-21T14:02:28.394417+0000","last_became_active":"2024-09-21T14:01:41.184664+0000","last_became_peered":"2024-09-21T14:01:41.184664+0000","last_unstale":"2024-09-21T14:02:28.394417+0000","last_undegraded":"2024-09-21T14:02:28.394417+0000","last_fullsized":"2024-09-21T14:02:28.394417+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:26.557002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:26.557002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"373'14433","reported_seq":14710,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:31.171510+0000","last_change":"2024-09-21T14:01:41.317185+0000","last_active":"2024-09-21T14:02:31.171510+0000","last_peered":"2024-09-21T14:02:31.171510+0000","last_clean":"2024-09-21T14:02:31.171510+0000","last_became_active":"2024-09-21T14:01:41.197526+0000","last_became_peered":"2024-09-21T14:01:41.197526+0000","last_unstale":"2024-09-21T14:02:31.171510+0000","last_undegraded":"2024-09-21T14:02:31.171510+0000","last_fullsized":"2024-09-21T14:02:31.171510+0000","mapping_epoch":345,"log_start":"312'9987","ondisk_log_start":"312'9987","created":260,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4446,"log_dups_size":0,"ondisk_log_size":4446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63897600,"num_objects":975,"num_object_clones":0,"num_object_copies":2925,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14433,"num_write_kb":493056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"300'32","reported_seq":343,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.418140+0000","last_change":"2024-09-21T14:00:47.130089+0000","last_active":"2024-09-21T14:02:30.418140+0000","last_peered":"2024-09-21T14:02:30.418140+0000","last_clean":"2024-09-21T14:02:30.418140+0000","last_became_active":"2024-09-21T13:59:32.204624+0000","last_became_peered":"2024-09-21T13:59:32.204624+0000","last_unstale":"2024-09-21T14:02:30.418140+0000","last_undegraded":"2024-09-21T14:02:30.418140+0000","last_fullsized":"2024-09-21T14:02:30.418140+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'32","last_scrub_stamp":"2024-09-21T14:00:47.129789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:00:47.129789+0000","objects_scrubbed":8,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:15.616969+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":645,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.423239+0000","last_change":"2024-09-21T14:01:41.190537+0000","last_active":"2024-09-21T14:02:30.423239+0000","last_peered":"2024-09-21T14:02:30.423239+0000","last_clean":"2024-09-21T14:02:30.423239+0000","last_became_active":"2024-09-21T14:01:41.190210+0000","last_became_peered":"2024-09-21T14:01:41.190210+0000","last_unstale":"2024-09-21T14:02:30.423239+0000","last_undegraded":"2024-09-21T14:02:30.423239+0000","last_fullsized":"2024-09-21T14:02:30.423239+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:57.772975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:57.772975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"75'79","reported_seq":706,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.414053+0000","last_change":"2024-09-21T14:00:39.339360+0000","last_active":"2024-09-21T14:02:30.414053+0000","last_peered":"2024-09-21T14:02:30.414053+0000","last_clean":"2024-09-21T14:02:30.414053+0000","last_became_active":"2024-09-21T14:00:39.339077+0000","last_became_peered":"2024-09-21T14:00:39.339077+0000","last_unstale":"2024-09-21T14:02:30.414053+0000","last_undegraded":"2024-09-21T14:02:30.414053+0000","last_fullsized":"2024-09-21T14:02:30.414053+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:02:42.782456+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":19,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53669,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":620,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.415882+0000","last_change":"2024-09-21T14:01:41.182186+0000","last_active":"2024-09-21T14:02:30.415882+0000","last_peered":"2024-09-21T14:02:30.415882+0000","last_clean":"2024-09-21T14:02:30.415882+0000","last_became_active":"2024-09-21T14:01:41.181899+0000","last_became_peered":"2024-09-21T14:01:41.181899+0000","last_unstale":"2024-09-21T14:02:30.415882+0000","last_undegraded":"2024-09-21T14:02:30.415882+0000","last_fullsized":"2024-09-21T14:02:30.415882+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:15.700409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:15.700409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":545,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.414029+0000","last_change":"2024-09-21T14:01:41.196509+0000","last_active":"2024-09-21T14:02:30.414029+0000","last_peered":"2024-09-21T14:02:30.414029+0000","last_clean":"2024-09-21T14:02:30.414029+0000","last_became_active":"2024-09-21T14:01:41.196243+0000","last_became_peered":"2024-09-21T14:01:41.196243+0000","last_unstale":"2024-09-21T14:02:30.414029+0000","last_undegraded":"2024-09-21T14:02:30.414029+0000","last_fullsized":"2024-09-21T14:02:30.414029+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:21.153018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:21.153018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"373'14177","reported_seq":24087,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:31.175686+0000","last_change":"2024-09-21T14:00:42.107898+0000","last_active":"2024-09-21T14:02:31.175686+0000","last_peered":"2024-09-21T14:02:31.175686+0000","last_clean":"2024-09-21T14:02:31.175686+0000","last_became_active":"2024-09-21T14:00:42.085387+0000","last_became_peered":"2024-09-21T14:00:42.085387+0000","last_unstale":"2024-09-21T14:02:31.175686+0000","last_undegraded":"2024-09-21T14:02:31.175686+0000","last_fullsized":"2024-09-21T14:02:31.175686+0000","mapping_epoch":321,"log_start":"311'9612","ondisk_log_start":"311'9612","created":260,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4565,"log_dups_size":0,"ondisk_log_size":4565,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64946176,"num_objects":991,"num_object_clones":0,"num_object_copies":2973,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14177,"num_write_kb":485376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4827,"num_bytes_recovered":316342272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"290'19","reported_seq":732,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.426034+0000","last_change":"2024-09-21T14:02:28.553549+0000","last_active":"2024-09-21T14:02:30.426034+0000","last_peered":"2024-09-21T14:02:30.426034+0000","last_clean":"2024-09-21T14:02:30.426034+0000","last_became_active":"2024-09-21T13:58:17.776310+0000","last_became_peered":"2024-09-21T13:58:17.776310+0000","last_unstale":"2024-09-21T14:02:30.426034+0000","last_undegraded":"2024-09-21T14:02:30.426034+0000","last_fullsized":"2024-09-21T14:02:30.426034+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'19","last_scrub_stamp":"2024-09-21T14:02:28.553065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:28.553065+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:33.636087+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":694,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.424705+0000","last_change":"2024-09-21T14:02:03.437581+0000","last_active":"2024-09-21T14:02:30.424705+0000","last_peered":"2024-09-21T14:02:30.424705+0000","last_clean":"2024-09-21T14:02:30.424705+0000","last_became_active":"2024-09-21T13:59:32.223247+0000","last_became_peered":"2024-09-21T13:59:32.223247+0000","last_unstale":"2024-09-21T14:02:30.424705+0000","last_undegraded":"2024-09-21T14:02:30.424705+0000","last_fullsized":"2024-09-21T14:02:30.424705+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:03.437510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:02:03.437510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:14.620200+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":613,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.413876+0000","last_change":"2024-09-21T14:01:41.194538+0000","last_active":"2024-09-21T14:02:30.413876+0000","last_peered":"2024-09-21T14:02:30.413876+0000","last_clean":"2024-09-21T14:02:30.413876+0000","last_became_active":"2024-09-21T14:01:41.193941+0000","last_became_peered":"2024-09-21T14:01:41.193941+0000","last_unstale":"2024-09-21T14:02:30.413876+0000","last_undegraded":"2024-09-21T14:02:30.413876+0000","last_fullsized":"2024-09-21T14:02:30.413876+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:34.299722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:55:34.299722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":623,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.422705+0000","last_change":"2024-09-21T14:01:41.190501+0000","last_active":"2024-09-21T14:02:30.422705+0000","last_peered":"2024-09-21T14:02:30.422705+0000","last_clean":"2024-09-21T14:02:30.422705+0000","last_became_active":"2024-09-21T14:01:41.190192+0000","last_became_peered":"2024-09-21T14:01:41.190192+0000","last_unstale":"2024-09-21T14:02:30.422705+0000","last_undegraded":"2024-09-21T14:02:30.422705+0000","last_fullsized":"2024-09-21T14:02:30.422705+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:58.952218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:54:58.952218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:18.255572+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"75'79","reported_seq":86,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.409470+0000","last_change":"2024-09-21T14:01:41.189146+0000","last_active":"2024-09-21T14:02:30.409470+0000","last_peered":"2024-09-21T14:02:30.409470+0000","last_clean":"2024-09-21T14:02:30.409470+0000","last_became_active":"2024-09-21T14:01:41.188685+0000","last_became_peered":"2024-09-21T14:01:41.188685+0000","last_unstale":"2024-09-21T14:02:30.409470+0000","last_undegraded":"2024-09-21T14:02:30.409470+0000","last_fullsized":"2024-09-21T14:02:30.409470+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:02:42.589731+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.0","version":"372'14019","reported_seq":18194,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T14:02:29.689281+0000","last_change":"2024-09-21T13:59:24.817101+0000","last_active":"2024-09-21T14:02:29.689281+0000","last_peered":"2024-09-21T14:02:29.689281+0000","last_clean":"2024-09-21T14:02:29.689281+0000","last_became_active":"2024-09-21T13:58:36.723110+0000","last_became_peered":"2024-09-21T13:58:36.723110+0000","last_unstale":"2024-09-21T14:02:29.689281+0000","last_undegraded":"2024-09-21T14:02:29.689281+0000","last_fullsized":"2024-09-21T14:02:29.689281+0000","mapping_epoch":279,"log_start":"310'9381","ondisk_log_start":"310'9381","created":260,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'7035","last_scrub_stamp":"2024-09-21T13:59:24.816903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:59:24.816903+0000","objects_scrubbed":6955,"log_size":4638,"log_dups_size":0,"ondisk_log_size":4638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2446,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69009408,"num_objects":1053,"num_object_clones":0,"num_object_copies":3159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14019,"num_write_kb":482304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1924,"num_bytes_recovered":126091264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"290'31","reported_seq":720,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.425616+0000","last_change":"2024-09-21T14:01:03.541547+0000","last_active":"2024-09-21T14:02:30.425616+0000","last_peered":"2024-09-21T14:02:30.425616+0000","last_clean":"2024-09-21T14:02:30.425616+0000","last_became_active":"2024-09-21T13:52:04.313218+0000","last_became_peered":"2024-09-21T13:52:04.313218+0000","last_unstale":"2024-09-21T14:02:30.425616+0000","last_undegraded":"2024-09-21T14:02:30.425616+0000","last_fullsized":"2024-09-21T14:02:30.425616+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'31","last_scrub_stamp":"2024-09-21T14:01:03.541432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:03.541432+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":235,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.417044+0000","last_change":"2024-09-21T14:01:29.924408+0000","last_active":"2024-09-21T14:02:30.417044+0000","last_peered":"2024-09-21T14:02:30.417044+0000","last_clean":"2024-09-21T14:02:30.417044+0000","last_became_active":"2024-09-21T13:58:17.762520+0000","last_became_peered":"2024-09-21T13:58:17.762520+0000","last_unstale":"2024-09-21T14:02:30.417044+0000","last_undegraded":"2024-09-21T14:02:30.417044+0000","last_fullsized":"2024-09-21T14:02:30.417044+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:29.924331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:01:29.924331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:43.977176+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":646,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T14:02:28.394524+0000","last_change":"2024-09-21T13:58:52.030800+0000","last_active":"2024-09-21T14:02:28.394524+0000","last_peered":"2024-09-21T14:02:28.394524+0000","last_clean":"2024-09-21T14:02:28.394524+0000","last_became_active":"2024-09-21T13:49:02.067472+0000","last_became_peered":"2024-09-21T13:49:02.067472+0000","last_unstale":"2024-09-21T14:02:28.394524+0000","last_undegraded":"2024-09-21T14:02:28.394524+0000","last_fullsized":"2024-09-21T14:02:28.394524+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:52.030718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:52.030718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":531,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.413546+0000","last_change":"2024-09-21T13:57:12.683321+0000","last_active":"2024-09-21T14:02:30.413546+0000","last_peered":"2024-09-21T14:02:30.413546+0000","last_clean":"2024-09-21T14:02:30.413546+0000","last_became_active":"2024-09-21T13:57:12.682979+0000","last_became_peered":"2024-09-21T13:57:12.682979+0000","last_unstale":"2024-09-21T14:02:30.413546+0000","last_undegraded":"2024-09-21T14:02:30.413546+0000","last_fullsized":"2024-09-21T14:02:30.413546+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:18.027944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:18.027944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:21.753981+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"75'79","reported_seq":711,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.414638+0000","last_change":"2024-09-21T14:00:41.140108+0000","last_active":"2024-09-21T14:02:30.414638+0000","last_peered":"2024-09-21T14:02:30.414638+0000","last_clean":"2024-09-21T14:02:30.414638+0000","last_became_active":"2024-09-21T14:00:41.139315+0000","last_became_peered":"2024-09-21T14:00:41.139315+0000","last_unstale":"2024-09-21T14:02:30.414638+0000","last_undegraded":"2024-09-21T14:02:30.414638+0000","last_fullsized":"2024-09-21T14:02:30.414638+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1","version":"373'14082","reported_seq":26361,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:31.162522+0000","last_change":"2024-09-21T14:01:10.113546+0000","last_active":"2024-09-21T14:02:31.162522+0000","last_peered":"2024-09-21T14:02:31.162522+0000","last_clean":"2024-09-21T14:02:31.162522+0000","last_became_active":"2024-09-21T14:01:10.092192+0000","last_became_peered":"2024-09-21T14:01:10.092192+0000","last_unstale":"2024-09-21T14:02:31.162522+0000","last_undegraded":"2024-09-21T14:02:31.162522+0000","last_fullsized":"2024-09-21T14:02:31.162522+0000","mapping_epoch":333,"log_start":"311'9645","ondisk_log_start":"311'9645","created":260,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4437,"log_dups_size":0,"ondisk_log_size":4437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67239936,"num_objects":1026,"num_object_clones":0,"num_object_copies":3078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14082,"num_write_kb":483456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5999,"num_bytes_recovered":393150464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"290'20","reported_seq":739,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.425655+0000","last_change":"2024-09-21T13:58:17.789407+0000","last_active":"2024-09-21T14:02:30.425655+0000","last_peered":"2024-09-21T14:02:30.425655+0000","last_clean":"2024-09-21T14:02:30.425655+0000","last_became_active":"2024-09-21T13:58:17.765427+0000","last_became_peered":"2024-09-21T13:58:17.765427+0000","last_unstale":"2024-09-21T14:02:30.425655+0000","last_undegraded":"2024-09-21T14:02:30.425655+0000","last_fullsized":"2024-09-21T14:02:30.425655+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:58:13.464196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:13.464196+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:16.467667+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":580,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.423345+0000","last_change":"2024-09-21T13:57:36.707211+0000","last_active":"2024-09-21T14:02:30.423345+0000","last_peered":"2024-09-21T14:02:30.423345+0000","last_clean":"2024-09-21T14:02:30.423345+0000","last_became_active":"2024-09-21T13:57:12.704203+0000","last_became_peered":"2024-09-21T13:57:12.704203+0000","last_unstale":"2024-09-21T14:02:30.423345+0000","last_undegraded":"2024-09-21T14:02:30.423345+0000","last_fullsized":"2024-09-21T14:02:30.423345+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:36.707148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:36.707148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"346'14","reported_seq":707,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.413254+0000","last_change":"2024-09-21T14:01:41.186960+0000","last_active":"2024-09-21T14:02:30.413254+0000","last_peered":"2024-09-21T14:02:30.413254+0000","last_clean":"2024-09-21T14:02:30.413254+0000","last_became_active":"2024-09-21T14:01:41.186564+0000","last_became_peered":"2024-09-21T14:01:41.186564+0000","last_unstale":"2024-09-21T14:02:30.413254+0000","last_undegraded":"2024-09-21T14:02:30.413254+0000","last_fullsized":"2024-09-21T14:02:30.413254+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'12","last_scrub_stamp":"2024-09-21T13:59:38.738120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:38.738120+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:44.832105+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"68'1","reported_seq":666,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.425725+0000","last_change":"2024-09-21T13:59:31.392186+0000","last_active":"2024-09-21T14:02:30.425725+0000","last_peered":"2024-09-21T14:02:30.425725+0000","last_clean":"2024-09-21T14:02:30.425725+0000","last_became_active":"2024-09-21T13:52:04.313210+0000","last_became_peered":"2024-09-21T13:52:04.313210+0000","last_unstale":"2024-09-21T14:02:30.425725+0000","last_undegraded":"2024-09-21T14:02:30.425725+0000","last_fullsized":"2024-09-21T14:02:30.425725+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T13:59:31.392105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:31.392105+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"373'13931","reported_seq":14156,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.842616+0000","last_change":"2024-09-21T14:01:41.341139+0000","last_active":"2024-09-21T14:02:30.842616+0000","last_peered":"2024-09-21T14:02:30.842616+0000","last_clean":"2024-09-21T14:02:30.842616+0000","last_became_active":"2024-09-21T14:01:41.199821+0000","last_became_peered":"2024-09-21T14:01:41.199821+0000","last_unstale":"2024-09-21T14:02:30.842616+0000","last_undegraded":"2024-09-21T14:02:30.842616+0000","last_fullsized":"2024-09-21T14:02:30.842616+0000","mapping_epoch":345,"log_start":"310'8999","ondisk_log_start":"310'8999","created":260,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4932,"log_dups_size":0,"ondisk_log_size":4932,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62849024,"num_objects":959,"num_object_clones":0,"num_object_copies":2877,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13931,"num_write_kb":476480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"290'15","reported_seq":692,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T14:02:28.394211+0000","last_change":"2024-09-21T13:58:50.498597+0000","last_active":"2024-09-21T14:02:28.394211+0000","last_peered":"2024-09-21T14:02:28.394211+0000","last_clean":"2024-09-21T14:02:28.394211+0000","last_became_active":"2024-09-21T13:49:00.053016+0000","last_became_peered":"2024-09-21T13:49:00.053016+0000","last_unstale":"2024-09-21T14:02:28.394211+0000","last_undegraded":"2024-09-21T14:02:28.394211+0000","last_fullsized":"2024-09-21T14:02:28.394211+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:58:50.498369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:50.498369+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":625,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.412999+0000","last_change":"2024-09-21T14:01:41.192717+0000","last_active":"2024-09-21T14:02:30.412999+0000","last_peered":"2024-09-21T14:02:30.412999+0000","last_clean":"2024-09-21T14:02:30.412999+0000","last_became_active":"2024-09-21T14:01:41.192443+0000","last_became_peered":"2024-09-21T14:01:41.192443+0000","last_unstale":"2024-09-21T14:02:30.412999+0000","last_undegraded":"2024-09-21T14:02:30.412999+0000","last_fullsized":"2024-09-21T14:02:30.412999+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:53.894506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:53.894506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":605,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.424862+0000","last_change":"2024-09-21T14:01:27.739999+0000","last_active":"2024-09-21T14:02:30.424862+0000","last_peered":"2024-09-21T14:02:30.424862+0000","last_clean":"2024-09-21T14:02:30.424862+0000","last_became_active":"2024-09-21T13:57:12.680978+0000","last_became_peered":"2024-09-21T13:57:12.680978+0000","last_unstale":"2024-09-21T14:02:30.424862+0000","last_undegraded":"2024-09-21T14:02:30.424862+0000","last_fullsized":"2024-09-21T14:02:30.424862+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:27.739921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:01:27.739921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:50.205235+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":555,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.414432+0000","last_change":"2024-09-21T13:59:32.224001+0000","last_active":"2024-09-21T14:02:30.414432+0000","last_peered":"2024-09-21T14:02:30.414432+0000","last_clean":"2024-09-21T14:02:30.414432+0000","last_became_active":"2024-09-21T13:59:32.221628+0000","last_became_peered":"2024-09-21T13:59:32.221628+0000","last_unstale":"2024-09-21T14:02:30.414432+0000","last_undegraded":"2024-09-21T14:02:30.414432+0000","last_fullsized":"2024-09-21T14:02:30.414432+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:13.698091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:13.698091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"373'14578","reported_seq":22880,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.774513+0000","last_change":"2024-09-21T14:02:09.252872+0000","last_active":"2024-09-21T14:02:30.774513+0000","last_peered":"2024-09-21T14:02:30.774513+0000","last_clean":"2024-09-21T14:02:30.774513+0000","last_became_active":"2024-09-21T14:02:09.218449+0000","last_became_peered":"2024-09-21T14:02:09.218449+0000","last_unstale":"2024-09-21T14:02:30.774513+0000","last_undegraded":"2024-09-21T14:02:30.774513+0000","last_fullsized":"2024-09-21T14:02:30.774513+0000","mapping_epoch":362,"log_start":"310'9655","ondisk_log_start":"310'9655","created":260,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4923,"log_dups_size":0,"ondisk_log_size":4923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67633152,"num_objects":1032,"num_object_clones":0,"num_object_copies":3096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14578,"num_write_kb":499520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4082,"num_bytes_recovered":226033664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"290'35","reported_seq":301,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.417189+0000","last_change":"2024-09-21T14:01:02.006619+0000","last_active":"2024-09-21T14:02:30.417189+0000","last_peered":"2024-09-21T14:02:30.417189+0000","last_clean":"2024-09-21T14:02:30.417189+0000","last_became_active":"2024-09-21T13:59:32.204184+0000","last_became_peered":"2024-09-21T13:59:32.204184+0000","last_unstale":"2024-09-21T14:02:30.417189+0000","last_undegraded":"2024-09-21T14:02:30.417189+0000","last_fullsized":"2024-09-21T14:02:30.417189+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'35","last_scrub_stamp":"2024-09-21T14:01:02.006504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:02.006504+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:22.557193+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":647,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.417123+0000","last_change":"2024-09-21T14:01:41.184937+0000","last_active":"2024-09-21T14:02:30.417123+0000","last_peered":"2024-09-21T14:02:30.417123+0000","last_clean":"2024-09-21T14:02:30.417123+0000","last_became_active":"2024-09-21T14:01:41.184567+0000","last_became_peered":"2024-09-21T14:01:41.184567+0000","last_unstale":"2024-09-21T14:02:30.417123+0000","last_undegraded":"2024-09-21T14:02:30.417123+0000","last_fullsized":"2024-09-21T14:02:30.417123+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:21.587493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:21.587493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":625,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.412396+0000","last_change":"2024-09-21T14:01:41.182250+0000","last_active":"2024-09-21T14:02:30.412396+0000","last_peered":"2024-09-21T14:02:30.412396+0000","last_clean":"2024-09-21T14:02:30.412396+0000","last_became_active":"2024-09-21T14:01:41.181911+0000","last_became_peered":"2024-09-21T14:01:41.181911+0000","last_unstale":"2024-09-21T14:02:30.412396+0000","last_undegraded":"2024-09-21T14:02:30.412396+0000","last_fullsized":"2024-09-21T14:02:30.412396+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:55.588554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:55.588554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:41.898392+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":574,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.414047+0000","last_change":"2024-09-21T13:59:32.223123+0000","last_active":"2024-09-21T14:02:30.414047+0000","last_peered":"2024-09-21T14:02:30.414047+0000","last_clean":"2024-09-21T14:02:30.414047+0000","last_became_active":"2024-09-21T13:59:32.219332+0000","last_became_peered":"2024-09-21T13:59:32.219332+0000","last_unstale":"2024-09-21T14:02:30.414047+0000","last_undegraded":"2024-09-21T14:02:30.414047+0000","last_fullsized":"2024-09-21T14:02:30.414047+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:19.537177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:19.537177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"372'13786","reported_seq":14023,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T14:02:29.683020+0000","last_change":"2024-09-21T13:57:53.515705+0000","last_active":"2024-09-21T14:02:29.683020+0000","last_peered":"2024-09-21T14:02:29.683020+0000","last_clean":"2024-09-21T14:02:29.683020+0000","last_became_active":"2024-09-21T13:57:53.515313+0000","last_became_peered":"2024-09-21T13:57:53.515313+0000","last_unstale":"2024-09-21T14:02:29.683020+0000","last_undegraded":"2024-09-21T14:02:29.683020+0000","last_fullsized":"2024-09-21T14:02:29.683020+0000","mapping_epoch":260,"log_start":"310'9179","ondisk_log_start":"310'9179","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4607,"log_dups_size":0,"ondisk_log_size":4607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70909952,"num_objects":1082,"num_object_clones":0,"num_object_copies":3246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13786,"num_write_kb":475776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"290'18","reported_seq":711,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.424606+0000","last_change":"2024-09-21T13:59:28.340494+0000","last_active":"2024-09-21T14:02:30.424606+0000","last_peered":"2024-09-21T14:02:30.424606+0000","last_clean":"2024-09-21T14:02:30.424606+0000","last_became_active":"2024-09-21T13:57:12.680627+0000","last_became_peered":"2024-09-21T13:57:12.680627+0000","last_unstale":"2024-09-21T14:02:30.424606+0000","last_undegraded":"2024-09-21T14:02:30.424606+0000","last_fullsized":"2024-09-21T14:02:30.424606+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'18","last_scrub_stamp":"2024-09-21T13:59:28.340415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:28.340415+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:11.383693+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":602,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.415527+0000","last_change":"2024-09-21T13:59:32.201126+0000","last_active":"2024-09-21T14:02:30.415527+0000","last_peered":"2024-09-21T14:02:30.415527+0000","last_clean":"2024-09-21T14:02:30.415527+0000","last_became_active":"2024-09-21T13:59:32.200674+0000","last_became_peered":"2024-09-21T13:59:32.200674+0000","last_unstale":"2024-09-21T14:02:30.415527+0000","last_undegraded":"2024-09-21T14:02:30.415527+0000","last_fullsized":"2024-09-21T14:02:30.415527+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:21.582325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:21.582325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":587,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.413839+0000","last_change":"2024-09-21T14:02:27.791041+0000","last_active":"2024-09-21T14:02:30.413839+0000","last_peered":"2024-09-21T14:02:30.413839+0000","last_clean":"2024-09-21T14:02:30.413839+0000","last_became_active":"2024-09-21T13:52:04.322147+0000","last_became_peered":"2024-09-21T13:52:04.322147+0000","last_unstale":"2024-09-21T14:02:30.413839+0000","last_undegraded":"2024-09-21T14:02:30.413839+0000","last_fullsized":"2024-09-21T14:02:30.413839+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:27.790994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:02:27.790994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:51.879201+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":588,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.423681+0000","last_change":"2024-09-21T13:59:32.207006+0000","last_active":"2024-09-21T14:02:30.423681+0000","last_peered":"2024-09-21T14:02:30.423681+0000","last_clean":"2024-09-21T14:02:30.423681+0000","last_became_active":"2024-09-21T13:59:32.203767+0000","last_became_peered":"2024-09-21T13:59:32.203767+0000","last_unstale":"2024-09-21T14:02:30.423681+0000","last_undegraded":"2024-09-21T14:02:30.423681+0000","last_fullsized":"2024-09-21T14:02:30.423681+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:09.040062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:55:09.040062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"373'14623","reported_seq":19208,"reported_epoch":373,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-21T14:02:30.853938+0000","last_change":"2024-09-21T14:01:45.658247+0000","last_active":"2024-09-21T14:02:30.853938+0000","last_peered":"2024-09-21T14:02:30.853938+0000","last_clean":"2024-09-21T13:58:23.785212+0000","last_became_active":"2024-09-21T14:01:44.478305+0000","last_became_peered":"2024-09-21T14:01:44.478305+0000","last_unstale":"2024-09-21T14:02:30.853938+0000","last_undegraded":"2024-09-21T14:01:44.440463+0000","last_fullsized":"2024-09-21T14:01:42.170394+0000","mapping_epoch":346,"log_start":"310'9686","ondisk_log_start":"310'9686","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4937,"log_dups_size":0,"ondisk_log_size":4937,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65994752,"num_objects":1007,"num_object_clones":0,"num_object_copies":3021,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":795,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14623,"num_write_kb":500160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2237,"num_bytes_recovered":145162240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,7],"avail_no_missing":["5","0"],"object_location_counts":[{"shards":"0,5","objects":795},{"shards":"5,7","objects":212}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"290'27","reported_seq":701,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.409924+0000","last_change":"2024-09-21T14:01:41.193143+0000","last_active":"2024-09-21T14:02:30.409924+0000","last_peered":"2024-09-21T14:02:30.409924+0000","last_clean":"2024-09-21T14:02:30.409924+0000","last_became_active":"2024-09-21T14:01:41.192481+0000","last_became_peered":"2024-09-21T14:01:41.192481+0000","last_unstale":"2024-09-21T14:02:30.409924+0000","last_undegraded":"2024-09-21T14:02:30.409924+0000","last_fullsized":"2024-09-21T14:02:30.409924+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'27","last_scrub_stamp":"2024-09-21T13:59:33.058859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:33.058859+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":574,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.415473+0000","last_change":"2024-09-21T13:59:32.203889+0000","last_active":"2024-09-21T14:02:30.415473+0000","last_peered":"2024-09-21T14:02:30.415473+0000","last_clean":"2024-09-21T14:02:30.415473+0000","last_became_active":"2024-09-21T13:59:32.203172+0000","last_became_peered":"2024-09-21T13:59:32.203172+0000","last_unstale":"2024-09-21T14:02:30.415473+0000","last_undegraded":"2024-09-21T14:02:30.415473+0000","last_fullsized":"2024-09-21T14:02:30.415473+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T13:55:03.879706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:55:03.879706+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"300'14","reported_seq":784,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.414339+0000","last_change":"2024-09-21T13:59:32.220898+0000","last_active":"2024-09-21T14:02:30.414339+0000","last_peered":"2024-09-21T14:02:30.414339+0000","last_clean":"2024-09-21T14:02:30.414339+0000","last_became_active":"2024-09-21T13:59:32.206956+0000","last_became_peered":"2024-09-21T13:59:32.206956+0000","last_unstale":"2024-09-21T14:02:30.414339+0000","last_undegraded":"2024-09-21T14:02:30.414339+0000","last_fullsized":"2024-09-21T14:02:30.414339+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'11","last_scrub_stamp":"2024-09-21T13:56:31.616003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:56:31.616003+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":649,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.413278+0000","last_change":"2024-09-21T14:01:41.248408+0000","last_active":"2024-09-21T14:02:30.413278+0000","last_peered":"2024-09-21T14:02:30.413278+0000","last_clean":"2024-09-21T14:02:30.413278+0000","last_became_active":"2024-09-21T14:01:41.246106+0000","last_became_peered":"2024-09-21T14:01:41.246106+0000","last_unstale":"2024-09-21T14:02:30.413278+0000","last_undegraded":"2024-09-21T14:02:30.413278+0000","last_fullsized":"2024-09-21T14:02:30.413278+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:39.258659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:39.258659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":238,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.417177+0000","last_change":"2024-09-21T14:01:50.733055+0000","last_active":"2024-09-21T14:02:30.417177+0000","last_peered":"2024-09-21T14:02:30.417177+0000","last_clean":"2024-09-21T14:02:30.417177+0000","last_became_active":"2024-09-21T14:01:41.197501+0000","last_became_peered":"2024-09-21T14:01:41.197501+0000","last_unstale":"2024-09-21T14:02:30.417177+0000","last_undegraded":"2024-09-21T14:02:30.417177+0000","last_fullsized":"2024-09-21T14:02:30.417177+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:50.733007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:01:50.733007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:08.367766+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"290'12","reported_seq":623,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.432560+0000","last_change":"2024-09-21T13:59:32.247422+0000","last_active":"2024-09-21T14:02:30.432560+0000","last_peered":"2024-09-21T14:02:30.432560+0000","last_clean":"2024-09-21T14:02:30.432560+0000","last_became_active":"2024-09-21T13:59:32.211306+0000","last_became_peered":"2024-09-21T13:59:32.211306+0000","last_unstale":"2024-09-21T14:02:30.432560+0000","last_undegraded":"2024-09-21T14:02:30.432560+0000","last_fullsized":"2024-09-21T14:02:30.432560+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'6","last_scrub_stamp":"2024-09-21T13:55:12.066873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:55:12.066873+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":581,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.409627+0000","last_change":"2024-09-21T14:01:41.199742+0000","last_active":"2024-09-21T14:02:30.409627+0000","last_peered":"2024-09-21T14:02:30.409627+0000","last_clean":"2024-09-21T14:02:30.409627+0000","last_became_active":"2024-09-21T14:01:41.199321+0000","last_became_peered":"2024-09-21T14:01:41.199321+0000","last_unstale":"2024-09-21T14:02:30.409627+0000","last_undegraded":"2024-09-21T14:02:30.409627+0000","last_fullsized":"2024-09-21T14:02:30.409627+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:37.106238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:37.106238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":620,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.416783+0000","last_change":"2024-09-21T13:59:32.222277+0000","last_active":"2024-09-21T14:02:30.416783+0000","last_peered":"2024-09-21T14:02:30.416783+0000","last_clean":"2024-09-21T14:02:30.416783+0000","last_became_active":"2024-09-21T13:59:32.219525+0000","last_became_peered":"2024-09-21T13:59:32.219525+0000","last_unstale":"2024-09-21T14:02:30.416783+0000","last_undegraded":"2024-09-21T14:02:30.416783+0000","last_fullsized":"2024-09-21T14:02:30.416783+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:02.838181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:55:02.838181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"290'15","reported_seq":586,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.422425+0000","last_change":"2024-09-21T13:59:32.238689+0000","last_active":"2024-09-21T14:02:30.422425+0000","last_peered":"2024-09-21T14:02:30.422425+0000","last_clean":"2024-09-21T14:02:30.422425+0000","last_became_active":"2024-09-21T13:59:32.209166+0000","last_became_peered":"2024-09-21T13:59:32.209166+0000","last_unstale":"2024-09-21T14:02:30.422425+0000","last_undegraded":"2024-09-21T14:02:30.422425+0000","last_fullsized":"2024-09-21T14:02:30.422425+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:55:04.990842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:55:04.990842+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:21.278052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"61'1","reported_seq":638,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.414562+0000","last_change":"2024-09-21T13:59:32.222826+0000","last_active":"2024-09-21T14:02:30.414562+0000","last_peered":"2024-09-21T14:02:30.414562+0000","last_clean":"2024-09-21T14:02:30.414562+0000","last_became_active":"2024-09-21T13:59:32.219515+0000","last_became_peered":"2024-09-21T13:59:32.219515+0000","last_unstale":"2024-09-21T14:02:30.414562+0000","last_undegraded":"2024-09-21T14:02:30.414562+0000","last_fullsized":"2024-09-21T14:02:30.414562+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:58:07.711028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:07.711028+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":571,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.430767+0000","last_change":"2024-09-21T13:59:32.211754+0000","last_active":"2024-09-21T14:02:30.430767+0000","last_peered":"2024-09-21T14:02:30.430767+0000","last_clean":"2024-09-21T14:02:30.430767+0000","last_became_active":"2024-09-21T13:59:32.210586+0000","last_became_peered":"2024-09-21T13:59:32.210586+0000","last_unstale":"2024-09-21T14:02:30.430767+0000","last_undegraded":"2024-09-21T14:02:30.430767+0000","last_fullsized":"2024-09-21T14:02:30.430767+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:28.014760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:28.014760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:17.280712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":629,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.412585+0000","last_change":"2024-09-21T14:02:29.731635+0000","last_active":"2024-09-21T14:02:30.412585+0000","last_peered":"2024-09-21T14:02:30.412585+0000","last_clean":"2024-09-21T14:02:30.412585+0000","last_became_active":"2024-09-21T13:59:32.205803+0000","last_became_peered":"2024-09-21T13:59:32.205803+0000","last_unstale":"2024-09-21T14:02:30.412585+0000","last_undegraded":"2024-09-21T14:02:30.412585+0000","last_fullsized":"2024-09-21T14:02:30.412585+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:29.731565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:02:29.731565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:53.473477+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"290'15","reported_seq":626,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.413571+0000","last_change":"2024-09-21T14:01:41.189676+0000","last_active":"2024-09-21T14:02:30.413571+0000","last_peered":"2024-09-21T14:02:30.413571+0000","last_clean":"2024-09-21T14:02:30.413571+0000","last_became_active":"2024-09-21T14:01:41.187904+0000","last_became_peered":"2024-09-21T14:01:41.187904+0000","last_unstale":"2024-09-21T14:02:30.413571+0000","last_undegraded":"2024-09-21T14:02:30.413571+0000","last_fullsized":"2024-09-21T14:02:30.413571+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:57:24.124604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:57:24.124604+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":507,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.414660+0000","last_change":"2024-09-21T13:59:32.212325+0000","last_active":"2024-09-21T14:02:30.414660+0000","last_peered":"2024-09-21T14:02:30.414660+0000","last_clean":"2024-09-21T14:02:30.414660+0000","last_became_active":"2024-09-21T13:59:32.208749+0000","last_became_peered":"2024-09-21T13:59:32.208749+0000","last_unstale":"2024-09-21T14:02:30.414660+0000","last_undegraded":"2024-09-21T14:02:30.414660+0000","last_fullsized":"2024-09-21T14:02:30.414660+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:20.574745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:20.574745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"300'14","reported_seq":717,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.422806+0000","last_change":"2024-09-21T13:59:32.208253+0000","last_active":"2024-09-21T14:02:30.422806+0000","last_peered":"2024-09-21T14:02:30.422806+0000","last_clean":"2024-09-21T14:02:30.422806+0000","last_became_active":"2024-09-21T13:59:32.206926+0000","last_became_peered":"2024-09-21T13:59:32.206926+0000","last_unstale":"2024-09-21T14:02:30.422806+0000","last_undegraded":"2024-09-21T14:02:30.422806+0000","last_fullsized":"2024-09-21T14:02:30.422806+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'10","last_scrub_stamp":"2024-09-21T13:54:20.946782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:20.946782+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:10.314285+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":673,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.425853+0000","last_change":"2024-09-21T14:01:41.200564+0000","last_active":"2024-09-21T14:02:30.425853+0000","last_peered":"2024-09-21T14:02:30.425853+0000","last_clean":"2024-09-21T14:02:30.425853+0000","last_became_active":"2024-09-21T14:01:41.200113+0000","last_became_peered":"2024-09-21T14:01:41.200113+0000","last_unstale":"2024-09-21T14:02:30.425853+0000","last_undegraded":"2024-09-21T14:02:30.425853+0000","last_fullsized":"2024-09-21T14:02:30.425853+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:54.805619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:54.805619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:48.557673+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"290'20","reported_seq":622,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.422606+0000","last_change":"2024-09-21T13:59:32.236530+0000","last_active":"2024-09-21T14:02:30.422606+0000","last_peered":"2024-09-21T14:02:30.422606+0000","last_clean":"2024-09-21T14:02:30.422606+0000","last_became_active":"2024-09-21T13:59:32.208332+0000","last_became_peered":"2024-09-21T13:59:32.208332+0000","last_unstale":"2024-09-21T14:02:30.422606+0000","last_undegraded":"2024-09-21T14:02:30.422606+0000","last_fullsized":"2024-09-21T14:02:30.422606+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:57:37.671391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:57:37.671391+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":614,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.416144+0000","last_change":"2024-09-21T13:59:32.212737+0000","last_active":"2024-09-21T14:02:30.416144+0000","last_peered":"2024-09-21T14:02:30.416144+0000","last_clean":"2024-09-21T14:02:30.416144+0000","last_became_active":"2024-09-21T13:59:32.209804+0000","last_became_peered":"2024-09-21T13:59:32.209804+0000","last_unstale":"2024-09-21T14:02:30.416144+0000","last_undegraded":"2024-09-21T14:02:30.416144+0000","last_fullsized":"2024-09-21T14:02:30.416144+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:25.571256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:25.571256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"346'10","reported_seq":717,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.413486+0000","last_change":"2024-09-21T14:01:41.207030+0000","last_active":"2024-09-21T14:02:30.413486+0000","last_peered":"2024-09-21T14:02:30.413486+0000","last_clean":"2024-09-21T14:02:30.413486+0000","last_became_active":"2024-09-21T14:01:41.206303+0000","last_became_peered":"2024-09-21T14:01:41.206303+0000","last_unstale":"2024-09-21T14:02:30.413486+0000","last_undegraded":"2024-09-21T14:02:30.413486+0000","last_fullsized":"2024-09-21T14:02:30.413486+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:59:32.104224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:32.104224+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":511,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.415349+0000","last_change":"2024-09-21T13:59:32.213015+0000","last_active":"2024-09-21T14:02:30.415349+0000","last_peered":"2024-09-21T14:02:30.415349+0000","last_clean":"2024-09-21T14:02:30.415349+0000","last_became_active":"2024-09-21T13:59:32.211472+0000","last_became_peered":"2024-09-21T13:59:32.211472+0000","last_unstale":"2024-09-21T14:02:30.415349+0000","last_undegraded":"2024-09-21T14:02:30.415349+0000","last_fullsized":"2024-09-21T14:02:30.415349+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:22.647191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:22.647191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"290'19","reported_seq":728,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.425282+0000","last_change":"2024-09-21T13:58:17.790158+0000","last_active":"2024-09-21T14:02:30.425282+0000","last_peered":"2024-09-21T14:02:30.425282+0000","last_clean":"2024-09-21T14:02:30.425282+0000","last_became_active":"2024-09-21T13:58:17.771497+0000","last_became_peered":"2024-09-21T13:58:17.771497+0000","last_unstale":"2024-09-21T14:02:30.425282+0000","last_undegraded":"2024-09-21T14:02:30.425282+0000","last_fullsized":"2024-09-21T14:02:30.425282+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:54:56.539374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:56.539374+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":248,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.415958+0000","last_change":"2024-09-21T13:59:32.202097+0000","last_active":"2024-09-21T14:02:30.415958+0000","last_peered":"2024-09-21T14:02:30.415958+0000","last_clean":"2024-09-21T14:02:30.415958+0000","last_became_active":"2024-09-21T13:59:32.201568+0000","last_became_peered":"2024-09-21T13:59:32.201568+0000","last_unstale":"2024-09-21T14:02:30.415958+0000","last_undegraded":"2024-09-21T14:02:30.415958+0000","last_fullsized":"2024-09-21T14:02:30.415958+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:01.228213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:01.228213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":684,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T14:02:28.393344+0000","last_change":"2024-09-21T14:01:41.185588+0000","last_active":"2024-09-21T14:02:28.393344+0000","last_peered":"2024-09-21T14:02:28.393344+0000","last_clean":"2024-09-21T14:02:28.393344+0000","last_became_active":"2024-09-21T14:01:41.185293+0000","last_became_peered":"2024-09-21T14:01:41.185293+0000","last_unstale":"2024-09-21T14:02:28.393344+0000","last_undegraded":"2024-09-21T14:02:28.393344+0000","last_fullsized":"2024-09-21T14:02:28.393344+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:35.409237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:35.409237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:10.948556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":579,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.429343+0000","last_change":"2024-09-21T14:01:41.200439+0000","last_active":"2024-09-21T14:02:30.429343+0000","last_peered":"2024-09-21T14:02:30.429343+0000","last_clean":"2024-09-21T14:02:30.429343+0000","last_became_active":"2024-09-21T14:01:41.199671+0000","last_became_peered":"2024-09-21T14:01:41.199671+0000","last_unstale":"2024-09-21T14:02:30.429343+0000","last_undegraded":"2024-09-21T14:02:30.429343+0000","last_fullsized":"2024-09-21T14:02:30.429343+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:39.643458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:39.643458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"290'15","reported_seq":728,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T14:02:28.393596+0000","last_change":"2024-09-21T13:58:39.977717+0000","last_active":"2024-09-21T14:02:28.393596+0000","last_peered":"2024-09-21T14:02:28.393596+0000","last_clean":"2024-09-21T14:02:28.393596+0000","last_became_active":"2024-09-21T13:58:17.764120+0000","last_became_peered":"2024-09-21T13:58:17.764120+0000","last_unstale":"2024-09-21T14:02:28.393596+0000","last_undegraded":"2024-09-21T14:02:28.393596+0000","last_fullsized":"2024-09-21T14:02:28.393596+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:58:39.977466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:39.977466+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":618,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.417302+0000","last_change":"2024-09-21T13:59:32.223082+0000","last_active":"2024-09-21T14:02:30.417302+0000","last_peered":"2024-09-21T14:02:30.417302+0000","last_clean":"2024-09-21T14:02:30.417302+0000","last_became_active":"2024-09-21T13:59:32.221854+0000","last_became_peered":"2024-09-21T13:59:32.221854+0000","last_unstale":"2024-09-21T14:02:30.417302+0000","last_undegraded":"2024-09-21T14:02:30.417302+0000","last_fullsized":"2024-09-21T14:02:30.417302+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:17.691193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:17.691193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":554,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.423258+0000","last_change":"2024-09-21T13:57:12.703746+0000","last_active":"2024-09-21T14:02:30.423258+0000","last_peered":"2024-09-21T14:02:30.423258+0000","last_clean":"2024-09-21T14:02:30.423258+0000","last_became_active":"2024-09-21T13:57:12.703441+0000","last_became_peered":"2024-09-21T13:57:12.703441+0000","last_unstale":"2024-09-21T14:02:30.423258+0000","last_undegraded":"2024-09-21T14:02:30.423258+0000","last_fullsized":"2024-09-21T14:02:30.423258+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:19.999125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:19.999125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:13.302246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":476,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.409411+0000","last_change":"2024-09-21T14:01:41.188075+0000","last_active":"2024-09-21T14:02:30.409411+0000","last_peered":"2024-09-21T14:02:30.409411+0000","last_clean":"2024-09-21T14:02:30.409411+0000","last_became_active":"2024-09-21T14:01:41.186934+0000","last_became_peered":"2024-09-21T14:01:41.186934+0000","last_unstale":"2024-09-21T14:02:30.409411+0000","last_undegraded":"2024-09-21T14:02:30.409411+0000","last_fullsized":"2024-09-21T14:02:30.409411+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:47.334504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:47.334504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"290'19","reported_seq":698,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.430117+0000","last_change":"2024-09-21T13:59:32.277742+0000","last_active":"2024-09-21T14:02:30.430117+0000","last_peered":"2024-09-21T14:02:30.430117+0000","last_clean":"2024-09-21T14:02:30.430117+0000","last_became_active":"2024-09-21T13:59:32.231567+0000","last_became_peered":"2024-09-21T13:59:32.231567+0000","last_unstale":"2024-09-21T14:02:30.430117+0000","last_undegraded":"2024-09-21T14:02:30.430117+0000","last_fullsized":"2024-09-21T14:02:30.430117+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:54:22.978267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:22.978267+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:12.294653+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":578,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.409728+0000","last_change":"2024-09-21T14:01:41.191090+0000","last_active":"2024-09-21T14:02:30.409728+0000","last_peered":"2024-09-21T14:02:30.409728+0000","last_clean":"2024-09-21T14:02:30.409728+0000","last_became_active":"2024-09-21T14:01:41.190760+0000","last_became_peered":"2024-09-21T14:01:41.190760+0000","last_unstale":"2024-09-21T14:02:30.409728+0000","last_undegraded":"2024-09-21T14:02:30.409728+0000","last_fullsized":"2024-09-21T14:02:30.409728+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:17.305144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:17.305144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":624,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.414133+0000","last_change":"2024-09-21T13:59:32.219508+0000","last_active":"2024-09-21T14:02:30.414133+0000","last_peered":"2024-09-21T14:02:30.414133+0000","last_clean":"2024-09-21T14:02:30.414133+0000","last_became_active":"2024-09-21T13:59:32.214004+0000","last_became_peered":"2024-09-21T13:59:32.214004+0000","last_unstale":"2024-09-21T14:02:30.414133+0000","last_undegraded":"2024-09-21T14:02:30.414133+0000","last_fullsized":"2024-09-21T14:02:30.414133+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:36.525417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:36.525417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":620,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.423431+0000","last_change":"2024-09-21T13:59:28.124373+0000","last_active":"2024-09-21T14:02:30.423431+0000","last_peered":"2024-09-21T14:02:30.423431+0000","last_clean":"2024-09-21T14:02:30.423431+0000","last_became_active":"2024-09-21T13:58:17.778115+0000","last_became_peered":"2024-09-21T13:58:17.778115+0000","last_unstale":"2024-09-21T14:02:30.423431+0000","last_undegraded":"2024-09-21T14:02:30.423431+0000","last_fullsized":"2024-09-21T14:02:30.423431+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:28.124296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:28.124296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"290'8","reported_seq":644,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.422853+0000","last_change":"2024-09-21T13:59:26.034457+0000","last_active":"2024-09-21T14:02:30.422853+0000","last_peered":"2024-09-21T14:02:30.422853+0000","last_clean":"2024-09-21T14:02:30.422853+0000","last_became_active":"2024-09-21T13:58:17.775079+0000","last_became_peered":"2024-09-21T13:58:17.775079+0000","last_unstale":"2024-09-21T14:02:30.422853+0000","last_undegraded":"2024-09-21T14:02:30.422853+0000","last_fullsized":"2024-09-21T14:02:30.422853+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'8","last_scrub_stamp":"2024-09-21T13:59:26.034372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:26.034372+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":546,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.413711+0000","last_change":"2024-09-21T14:01:41.196335+0000","last_active":"2024-09-21T14:02:30.413711+0000","last_peered":"2024-09-21T14:02:30.413711+0000","last_clean":"2024-09-21T14:02:30.413711+0000","last_became_active":"2024-09-21T14:01:41.195989+0000","last_became_peered":"2024-09-21T14:01:41.195989+0000","last_unstale":"2024-09-21T14:02:30.413711+0000","last_undegraded":"2024-09-21T14:02:30.413711+0000","last_fullsized":"2024-09-21T14:02:30.413711+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:16.457445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:16.457445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":238,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.417337+0000","last_change":"2024-09-21T14:00:57.128764+0000","last_active":"2024-09-21T14:02:30.417337+0000","last_peered":"2024-09-21T14:02:30.417337+0000","last_clean":"2024-09-21T14:02:30.417337+0000","last_became_active":"2024-09-21T13:59:32.204380+0000","last_became_peered":"2024-09-21T13:59:32.204380+0000","last_unstale":"2024-09-21T14:02:30.417337+0000","last_undegraded":"2024-09-21T14:02:30.417337+0000","last_fullsized":"2024-09-21T14:02:30.417337+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:00:57.128713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:00:57.128713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:24.563060+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":557,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.414564+0000","last_change":"2024-09-21T13:59:32.212615+0000","last_active":"2024-09-21T14:02:30.414564+0000","last_peered":"2024-09-21T14:02:30.414564+0000","last_clean":"2024-09-21T14:02:30.414564+0000","last_became_active":"2024-09-21T13:59:32.210134+0000","last_became_peered":"2024-09-21T13:59:32.210134+0000","last_unstale":"2024-09-21T14:02:30.414564+0000","last_undegraded":"2024-09-21T14:02:30.414564+0000","last_fullsized":"2024-09-21T14:02:30.414564+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:08.852789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:55:08.852789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":669,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T14:02:28.394876+0000","last_change":"2024-09-21T14:01:41.187792+0000","last_active":"2024-09-21T14:02:28.394876+0000","last_peered":"2024-09-21T14:02:28.394876+0000","last_clean":"2024-09-21T14:02:28.394876+0000","last_became_active":"2024-09-21T14:01:41.187438+0000","last_became_peered":"2024-09-21T14:01:41.187438+0000","last_unstale":"2024-09-21T14:02:28.394876+0000","last_undegraded":"2024-09-21T14:02:28.394876+0000","last_fullsized":"2024-09-21T14:02:28.394876+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:17.706325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:17.706325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":600,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.423031+0000","last_change":"2024-09-21T13:58:13.776632+0000","last_active":"2024-09-21T14:02:30.423031+0000","last_peered":"2024-09-21T14:02:30.423031+0000","last_clean":"2024-09-21T14:02:30.423031+0000","last_became_active":"2024-09-21T13:57:12.704042+0000","last_became_peered":"2024-09-21T13:57:12.704042+0000","last_unstale":"2024-09-21T14:02:30.423031+0000","last_undegraded":"2024-09-21T14:02:30.423031+0000","last_fullsized":"2024-09-21T14:02:30.423031+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:13.776572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:13.776572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"290'19","reported_seq":271,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.418050+0000","last_change":"2024-09-21T14:01:54.680822+0000","last_active":"2024-09-21T14:02:30.418050+0000","last_peered":"2024-09-21T14:02:30.418050+0000","last_clean":"2024-09-21T14:02:30.418050+0000","last_became_active":"2024-09-21T14:01:41.194606+0000","last_became_peered":"2024-09-21T14:01:41.194606+0000","last_unstale":"2024-09-21T14:02:30.418050+0000","last_undegraded":"2024-09-21T14:02:30.418050+0000","last_fullsized":"2024-09-21T14:02:30.418050+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'19","last_scrub_stamp":"2024-09-21T14:01:54.680735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:54.680735+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:21.312249+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":526,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T14:02:30.415277+0000","last_change":"2024-09-21T13:59:32.203045+0000","last_active":"2024-09-21T14:02:30.415277+0000","last_peered":"2024-09-21T14:02:30.415277+0000","last_clean":"2024-09-21T14:02:30.415277+0000","last_became_active":"2024-09-21T13:59:32.202225+0000","last_became_peered":"2024-09-21T13:59:32.202225+0000","last_unstale":"2024-09-21T14:02:30.415277+0000","last_undegraded":"2024-09-21T14:02:30.415277+0000","last_fullsized":"2024-09-21T14:02:30.415277+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:40.650832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:40.650832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":1048772632,"num_objects":16004,"num_object_clones":0,"num_object_copies":48012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":795,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16004,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":226808,"num_write_kb":7769921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41801,"num_bytes_recovered":2599354392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3145478144,"data_stored":3145465928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75979,"ondisk_log_size":75979,"up":48,"acting":47,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1111,"num_read_kb":868,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":1914,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":687,"ondisk_log_size":687,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":185,"num_read_kb":180,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":429358,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":159,"num_write":83,"num_write_kb":1184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1084718,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":298,"seq":1279900254245,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":816960,"kb_used_data":540784,"kb_used_omap":4,"kb_used_meta":276091,"kb_avail":92928192,"statfs":{"total":95995035648,"available":95158468608,"internally_reserved":0,"allocated":553762816,"data_stored":552929233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":282717864},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,2,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463085,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":494504,"kb_used_data":170264,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":93250648,"statfs":{"total":95995035648,"available":95488663552,"internally_reserved":0,"allocated":174350336,"data_stored":173540220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":331872710},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":345,"seq":1481763717131,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":711736,"kb_used_data":602852,"kb_used_omap":4,"kb_used_meta":108795,"kb_avail":93033416,"statfs":{"total":95995035648,"available":95266217984,"internally_reserved":0,"allocated":617320448,"data_stored":616499528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":111406755},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920954,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":609928,"kb_used_data":235832,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":93135224,"statfs":{"total":95995035648,"available":95370469376,"internally_reserved":0,"allocated":241491968,"data_stored":240649702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":383056325},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":240,"seq":1030792151105,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":797572,"kb_used_data":543156,"kb_used_omap":4,"kb_used_meta":254395,"kb_avail":92947580,"statfs":{"total":95995035648,"available":95178321920,"internally_reserved":0,"allocated":556191744,"data_stored":555346839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4443,"internal_metadata":260501157},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":130,"seq":558345748607,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":716332,"kb_used_data":166300,"kb_used_omap":4,"kb_used_meta":550011,"kb_avail":93028820,"statfs":{"total":95995035648,"available":95261511680,"internally_reserved":0,"allocated":170291200,"data_stored":169478250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4441,"internal_metadata":563211943},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575051,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1224100,"kb_used_data":647312,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":92521052,"statfs":{"total":95995035648,"available":94741557248,"internally_reserved":0,"allocated":662847488,"data_stored":661973305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":590608837},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738577,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":783412,"kb_used_data":305184,"kb_used_omap":1,"kb_used_meta":478206,"kb_avail":92961740,"statfs":{"total":95995035648,"available":95192821760,"internally_reserved":0,"allocated":312508416,"data_stored":311682357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":489683397},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":294060032,"data_stored":294060032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":642387968,"data_stored":642383896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":167116800,"data_stored":167116800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":524156928,"data_stored":524156928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":221904896,"data_stored":221904896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":599003136,"data_stored":598999064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":157220864,"data_stored":157220864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":539627520,"data_stored":539623448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T14:02:35.618 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:35.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:35.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:35.650 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:02:35.701 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:35 smithi187 ceph-mon[33032]: pgmap v695: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 1001 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 776 B/s rd, 0 B/s wr, 770 op/s; 795/48609 objects degraded (1.635%); 6.0 MiB/s, 96 objects/s recovering 2024-09-21T14:02:35.701 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:35 smithi187 ceph-mon[33032]: from='client.25877 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:02:35.701 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:35 smithi187 ceph-mon[33032]: 4.13 scrub starts 2024-09-21T14:02:35.701 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:35 smithi187 ceph-mon[33032]: 4.13 scrub ok 2024-09-21T14:02:35.701 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:35 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 795/48609 objects degraded (1.635%), 1 pg degraded (PG_DEGRADED) 2024-09-21T14:02:35.702 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:35 smithi187 ceph-mon[33032]: osdmap e374: 8 total, 8 up, 8 in 2024-09-21T14:02:35.720 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:02:35.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:35 smithi049 ceph-mon[30080]: pgmap v695: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 1001 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 776 B/s rd, 0 B/s wr, 770 op/s; 795/48609 objects degraded (1.635%); 6.0 MiB/s, 96 objects/s recovering 2024-09-21T14:02:35.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:35 smithi049 ceph-mon[30080]: from='client.25877 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:02:35.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:35 smithi049 ceph-mon[30080]: 4.13 scrub starts 2024-09-21T14:02:35.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:35 smithi049 ceph-mon[30080]: 4.13 scrub ok 2024-09-21T14:02:35.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:35 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 795/48609 objects degraded (1.635%), 1 pg degraded (PG_DEGRADED) 2024-09-21T14:02:35.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:35 smithi049 ceph-mon[30080]: osdmap e374: 8 total, 8 up, 8 in 2024-09-21T14:02:35.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:35 smithi049 ceph-mon[40489]: pgmap v695: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 1001 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 776 B/s rd, 0 B/s wr, 770 op/s; 795/48609 objects degraded (1.635%); 6.0 MiB/s, 96 objects/s recovering 2024-09-21T14:02:35.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:35 smithi049 ceph-mon[40489]: from='client.25877 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:02:35.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:35 smithi049 ceph-mon[40489]: 4.13 scrub starts 2024-09-21T14:02:35.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:35 smithi049 ceph-mon[40489]: 4.13 scrub ok 2024-09-21T14:02:35.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:35 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 795/48609 objects degraded (1.635%), 1 pg degraded (PG_DEGRADED) 2024-09-21T14:02:35.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:35 smithi049 ceph-mon[40489]: osdmap e374: 8 total, 8 up, 8 in 2024-09-21T14:02:35.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:35.757 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:35.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:02:35.858 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:02:35.891 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:35.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:35.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:02:35.995 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:02:36.041 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:36.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:36.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:02:36.056 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T14:02:36.143 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:36.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:36.198 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:36.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:36.299 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:02:36.338 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:36.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:36.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:02:36.440 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:02:36.481 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:36.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:36.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:02:36.583 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:36.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:36.638 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:36.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:36.739 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:02:36.775 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:36.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:36.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:02:36.877 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:02:36.901 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:36.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:36.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:02:37.002 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:02:37.038 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:37.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:37.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:02:37.052 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T14:02:37.141 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:02:37.162 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:37.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:37.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:02:37.264 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:02:37.285 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:37.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:37.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:02:37.387 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:37.423 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:37.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:37.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:37.525 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:37.562 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:37.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:37.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:37.663 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:02:37.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:37.697 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:37.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:02:37.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:37 smithi049 ceph-mon[40489]: pgmap v697: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 833 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 673 B/s rd, 0 B/s wr, 1.02k op/s; 521/40569 objects degraded (1.284%); 7.5 MiB/s, 119 objects/s recovering 2024-09-21T14:02:37.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:37 smithi049 ceph-mon[40489]: 6.b scrub starts 2024-09-21T14:02:37.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:37 smithi049 ceph-mon[40489]: 6.b scrub ok 2024-09-21T14:02:37.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:37 smithi049 ceph-mon[40489]: 3.1b scrub starts 2024-09-21T14:02:37.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:37 smithi049 ceph-mon[40489]: 3.1b scrub ok 2024-09-21T14:02:37.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:37 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/197273641' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:02:37.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:37 smithi049 ceph-mon[30080]: pgmap v697: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 833 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 673 B/s rd, 0 B/s wr, 1.02k op/s; 521/40569 objects degraded (1.284%); 7.5 MiB/s, 119 objects/s recovering 2024-09-21T14:02:37.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:37 smithi049 ceph-mon[30080]: 6.b scrub starts 2024-09-21T14:02:37.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:37 smithi049 ceph-mon[30080]: 6.b scrub ok 2024-09-21T14:02:37.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:37 smithi049 ceph-mon[30080]: 3.1b scrub starts 2024-09-21T14:02:37.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:37 smithi049 ceph-mon[30080]: 3.1b scrub ok 2024-09-21T14:02:37.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:37 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/197273641' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:02:37.798 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:02:37.831 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:37.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:37.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:02:37.904 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-09-21T14:02:37.933 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:02:37.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:37 smithi187 ceph-mon[33032]: pgmap v697: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 833 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 673 B/s rd, 0 B/s wr, 1.02k op/s; 521/40569 objects degraded (1.284%); 7.5 MiB/s, 119 objects/s recovering 2024-09-21T14:02:37.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:37 smithi187 ceph-mon[33032]: 6.b scrub starts 2024-09-21T14:02:37.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:37 smithi187 ceph-mon[33032]: 6.b scrub ok 2024-09-21T14:02:37.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:37 smithi187 ceph-mon[33032]: 3.1b scrub starts 2024-09-21T14:02:37.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:37 smithi187 ceph-mon[33032]: 3.1b scrub ok 2024-09-21T14:02:37.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:37 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/197273641' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:02:37.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:37.971 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:37.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:02:38.072 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:38.104 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:38.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:38.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:38.206 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:38.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:38.232 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:38.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:38.333 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:02:38.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:38.373 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:38.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:02:38.474 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:38.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:38.508 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:38.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:38.589 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T14:02:38.609 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:02:38.630 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:38.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:38.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:02:38.732 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:02:38.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:38 smithi049 ceph-mon[30080]: 9.4 scrub starts 2024-09-21T14:02:38.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:38 smithi049 ceph-mon[30080]: 9.4 scrub ok 2024-09-21T14:02:38.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:38 smithi049 ceph-mon[30080]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:02:38.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:38 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/197273641' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T14:02:38.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:38 smithi049 ceph-mon[30080]: osdmap e375: 8 total, 8 up, 8 in 2024-09-21T14:02:38.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:38 smithi049 ceph-mon[40489]: 9.4 scrub starts 2024-09-21T14:02:38.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:38 smithi049 ceph-mon[40489]: 9.4 scrub ok 2024-09-21T14:02:38.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:38 smithi049 ceph-mon[40489]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:02:38.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:38 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/197273641' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T14:02:38.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:38 smithi049 ceph-mon[40489]: osdmap e375: 8 total, 8 up, 8 in 2024-09-21T14:02:38.758 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:38.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:38.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:02:38.860 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:02:38.892 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:38.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:38.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:02:38.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:38 smithi187 ceph-mon[33032]: 9.4 scrub starts 2024-09-21T14:02:38.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:38 smithi187 ceph-mon[33032]: 9.4 scrub ok 2024-09-21T14:02:38.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:38 smithi187 ceph-mon[33032]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:02:38.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:38 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/197273641' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T14:02:38.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:38 smithi187 ceph-mon[33032]: osdmap e375: 8 total, 8 up, 8 in 2024-09-21T14:02:38.994 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:39.035 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:39.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:39.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:39.136 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:39.173 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:39.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:39.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:39.275 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:02:39.306 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:39.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:39.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:02:39.408 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:02:39.434 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:39.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:39.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:02:39.536 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:02:39.567 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:39.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:39.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:02:39.669 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:39.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:39.703 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:39.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:39.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:39 smithi049 ceph-mon[40489]: pgmap v699: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 630 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.33k op/s; 521/30819 objects degraded (1.691%); 7.5 MiB/s, 119 objects/s recovering 2024-09-21T14:02:39.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:39 smithi049 ceph-mon[30080]: pgmap v699: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 630 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.33k op/s; 521/30819 objects degraded (1.691%); 7.5 MiB/s, 119 objects/s recovering 2024-09-21T14:02:39.767 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:39 smithi187 ceph-mon[33032]: pgmap v699: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 630 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.33k op/s; 521/30819 objects degraded (1.691%); 7.5 MiB/s, 119 objects/s recovering 2024-09-21T14:02:39.804 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:39.841 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:39.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:39.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:39.943 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:02:39.974 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:39.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:39.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:02:40.076 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:02:40.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:40.116 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:40.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:02:40.194 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:02:39 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:02:40.217 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:02:40.253 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:40.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:40.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:02:40.355 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:40.377 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:40.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:40.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:40.478 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:40.515 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:40.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:40.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:40.617 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:40.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:40.655 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:40.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:40.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:40 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:02:40.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:40 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 521/30819 objects degraded (1.691%), 1 pg degraded (PG_DEGRADED) 2024-09-21T14:02:40.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:40 smithi049 ceph-mon[40489]: osdmap e376: 8 total, 8 up, 8 in 2024-09-21T14:02:40.755 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:02:40 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:02:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:02:40.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:40 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:02:40.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:40 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 521/30819 objects degraded (1.691%), 1 pg degraded (PG_DEGRADED) 2024-09-21T14:02:40.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:40 smithi049 ceph-mon[30080]: osdmap e376: 8 total, 8 up, 8 in 2024-09-21T14:02:40.755 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:02:40.790 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:40.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:40.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:02:40.892 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:02:40.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:40.931 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:40.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:02:40.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:40 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:02:40.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:40 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 521/30819 objects degraded (1.691%), 1 pg degraded (PG_DEGRADED) 2024-09-21T14:02:40.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:40 smithi187 ceph-mon[33032]: osdmap e376: 8 total, 8 up, 8 in 2024-09-21T14:02:41.033 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:02:41.054 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:41.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:41.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:02:41.156 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:02:41.188 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:41.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:41.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:02:41.290 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:02:41.325 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:41.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:41.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:02:41.427 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:02:41.463 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:41.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:41.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:02:41.565 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:02:41.598 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:41.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:41.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:02:41.700 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:02:41.733 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:41.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:41.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:02:41.735 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:02:41.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:41 smithi049 ceph-mon[30080]: pgmap v700: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 563 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 920 op/s; 521/27624 objects degraded (1.886%); 2.1 MiB/s, 34 objects/s recovering 2024-09-21T14:02:41.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:41 smithi049 ceph-mon[40489]: pgmap v700: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 563 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 920 op/s; 521/27624 objects degraded (1.886%); 2.1 MiB/s, 34 objects/s recovering 2024-09-21T14:02:41.781 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:02:41.787 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:02:41.835 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:02:41.868 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:41.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:41.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:02:41.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:41 smithi187 ceph-mon[33032]: pgmap v700: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 563 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 920 op/s; 521/27624 objects degraded (1.886%); 2.1 MiB/s, 34 objects/s recovering 2024-09-21T14:02:41.970 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:42.010 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:42.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:42.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:42.112 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:42.159 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:42.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:42.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:42.260 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:42.297 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:42.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:42.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:42.399 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:02:42.428 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:42.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:42.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:02:42.529 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:42.563 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:42.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:42.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:42.665 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:42.701 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:42.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:42.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:42.803 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:02:42.844 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:42.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:42.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:02:42.945 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:42.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:42.990 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:42.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:43.091 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:02:43.136 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:43.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:43.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:02:43.238 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:43.251 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:02:43.251 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:02:43.282 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:43.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:43.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:43.296 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:02:43.297 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:02:43.340 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:02:43.340 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:02:43.384 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:43.387 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:02:43.387 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:02:43.434 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:43.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:43.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:43.440 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:02:43.441 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:02:43.481 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T14:02:43.481 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-09-21T14:02:43.481 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:43.481 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25421.0:241808 9.9 e376/348 9:9b964702:::benchmark_data_smithi049_138504_object120400:head v 376'15218, mlcod=376'15216)", 2024-09-21T14:02:43.481 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:02:43.478525+0000", 2024-09-21T14:02:43.481 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.0018570640000000001, 2024-09-21T14:02:43.481 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001882917, 2024-09-21T14:02:43.481 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:02:43.481 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:02:43.481 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T14:02:43.482 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:02:43.482 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:43.482 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:02:43.482 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:43.478525+0000", 2024-09-21T14:02:43.482 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:02:43.482 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:43.482 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:43.482 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:02:43.482 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:43.478525+0000", 2024-09-21T14:02:43.482 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:02:43.482 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:43.482 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:43.482 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:02:43.482 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:43.478526+0000", 2024-09-21T14:02:43.483 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6800000000000009e-07 2024-09-21T14:02:43.483 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:43.483 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:43.483 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:02:43.483 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:43.478533+0000", 2024-09-21T14:02:43.483 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6100000000000002e-06 2024-09-21T14:02:43.483 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:43.483 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:43.483 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:02:43.483 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:43.478550+0000", 2024-09-21T14:02:43.483 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6742999999999999e-05 2024-09-21T14:02:43.483 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:43.483 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:43.483 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:02:43.484 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:43.478558+0000", 2024-09-21T14:02:43.484 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8490000000000006e-06 2024-09-21T14:02:43.484 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:43.484 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:43.484 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:02:43.484 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:43.478630+0000", 2024-09-21T14:02:43.484 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2898999999999994e-05 2024-09-21T14:02:43.484 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:02:43.484 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:02:43.484 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:02:43.484 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:43.478653+0000", 2024-09-21T14:02:43.484 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2704000000000001e-05 2024-09-21T14:02:43.484 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:02:43.484 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:02:43.485 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:02:43.485 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:02:43.485 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T14:02:43.485 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 1 2024-09-21T14:02:43.485 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T14:02:43.487 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:02:43.487 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:02:43.536 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:43.538 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:02:43.538 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:02:43.576 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:02:43.576 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:02:43.577 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:43 smithi049 ceph-mon[40489]: pgmap v702: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 284 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.55k op/s; 43/14235 objects degraded (0.302%); 6.8 MiB/s, 108 objects/s recovering 2024-09-21T14:02:43.577 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:43 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/85615883' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:02:43.577 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:43 smithi049 ceph-mon[30080]: pgmap v702: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 284 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.55k op/s; 43/14235 objects degraded (0.302%); 6.8 MiB/s, 108 objects/s recovering 2024-09-21T14:02:43.577 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:43 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/85615883' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:02:43.579 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:43.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:43.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:43.640 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T14:02:43.645 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-09-21T14:02:43.680 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:43.717 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T14:02:43.720 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:43.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:43.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:43.822 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:02:43.856 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:43.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:43.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:02:43.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:43 smithi187 ceph-mon[33032]: pgmap v702: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 284 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.55k op/s; 43/14235 objects degraded (0.302%); 6.8 MiB/s, 108 objects/s recovering 2024-09-21T14:02:43.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:43 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/85615883' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:02:43.958 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:02:43.989 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:43.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:43.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:02:44.091 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:02:44.127 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:44.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:44.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:02:44.229 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:44.255 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Removed 121405 objects 2024-09-21T14:02:44.255 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Clean up completed and total clean up time :188.567 2024-09-21T14:02:44.264 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:44.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:44.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:44.366 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:02:44.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:44.390 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:44.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:02:44.490 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:02:44.513 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:44.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:44.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:02:44.615 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:02:44.637 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:44.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:44.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:02:44.740 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:44.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:44 smithi049 ceph-mon[40489]: from='client.25883 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:02:44.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:44 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:02:44.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:44 smithi049 ceph-mon[30080]: from='client.25883 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:02:44.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:44 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:02:44.763 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:44.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:44.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:44.801 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":702,"stamp":"2024-09-21T14:02:41.990354+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":298296641,"num_objects":4745,"num_object_clones":0,"num_object_copies":14235,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":43,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4745,"num_whiteouts":0,"num_read":1389,"num_read_kb":1218,"num_write":239071,"num_write_kb":7771150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42722,"num_bytes_recovered":2649724096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77608,"ondisk_log_size":77608,"up":474,"acting":473,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":475,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":4167624,"kb_used_data":1188908,"kb_used_omap":23,"kb_used_meta":2978536,"kb_avail":745793592,"statfs":{"total":767960285184,"available":763692638208,"internally_reserved":0,"allocated":1217441792,"data_stored":1210684872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24148,"internal_metadata":3050021292},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-702545920,"num_objects":-10720,"num_object_clones":0,"num_object_copies":-32160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-10720,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10720,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":752,"num_bytes_recovered":49283072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.899908"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":568,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.102931+0000","last_change":"2024-09-21T14:01:41.189741+0000","last_active":"2024-09-21T14:02:40.102931+0000","last_peered":"2024-09-21T14:02:40.102931+0000","last_clean":"2024-09-21T14:02:40.102931+0000","last_became_active":"2024-09-21T14:01:41.189398+0000","last_became_peered":"2024-09-21T14:01:41.189398+0000","last_unstale":"2024-09-21T14:02:40.102931+0000","last_undegraded":"2024-09-21T14:02:40.102931+0000","last_fullsized":"2024-09-21T14:02:40.102931+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:55.873301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:55.873301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"290'15","reported_seq":741,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.468528+0000","last_change":"2024-09-21T14:01:34.785322+0000","last_active":"2024-09-21T14:02:37.468528+0000","last_peered":"2024-09-21T14:02:37.468528+0000","last_clean":"2024-09-21T14:02:37.468528+0000","last_became_active":"2024-09-21T13:58:17.771220+0000","last_became_peered":"2024-09-21T13:58:17.771220+0000","last_unstale":"2024-09-21T14:02:37.468528+0000","last_undegraded":"2024-09-21T14:02:37.468528+0000","last_fullsized":"2024-09-21T14:02:37.468528+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'15","last_scrub_stamp":"2024-09-21T14:01:34.785267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:34.785267+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":682,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.466647+0000","last_change":"2024-09-21T14:02:29.205437+0000","last_active":"2024-09-21T14:02:37.466647+0000","last_peered":"2024-09-21T14:02:37.466647+0000","last_clean":"2024-09-21T14:02:37.466647+0000","last_became_active":"2024-09-21T14:01:41.202168+0000","last_became_peered":"2024-09-21T14:01:41.202168+0000","last_unstale":"2024-09-21T14:02:37.466647+0000","last_undegraded":"2024-09-21T14:02:37.466647+0000","last_fullsized":"2024-09-21T14:02:37.466647+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:29.205343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:02:29.205343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:29.369063+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":658,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.469341+0000","last_change":"2024-09-21T14:01:41.203762+0000","last_active":"2024-09-21T14:02:37.469341+0000","last_peered":"2024-09-21T14:02:37.469341+0000","last_clean":"2024-09-21T14:02:37.469341+0000","last_became_active":"2024-09-21T14:01:41.203275+0000","last_became_peered":"2024-09-21T14:01:41.203275+0000","last_unstale":"2024-09-21T14:02:37.469341+0000","last_undegraded":"2024-09-21T14:02:37.469341+0000","last_fullsized":"2024-09-21T14:02:37.469341+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:00.320510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:00.320510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:46.565572+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":630,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.100974+0000","last_change":"2024-09-21T13:59:32.208701+0000","last_active":"2024-09-21T14:02:40.100974+0000","last_peered":"2024-09-21T14:02:40.100974+0000","last_clean":"2024-09-21T14:02:40.100974+0000","last_became_active":"2024-09-21T13:59:32.203340+0000","last_became_peered":"2024-09-21T13:59:32.203340+0000","last_unstale":"2024-09-21T14:02:40.100974+0000","last_undegraded":"2024-09-21T14:02:40.100974+0000","last_fullsized":"2024-09-21T14:02:40.100974+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:34.696780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:34.696780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":642,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.100711+0000","last_change":"2024-09-21T13:58:17.778322+0000","last_active":"2024-09-21T14:02:40.100711+0000","last_peered":"2024-09-21T14:02:40.100711+0000","last_clean":"2024-09-21T14:02:40.100711+0000","last_became_active":"2024-09-21T13:58:17.777893+0000","last_became_peered":"2024-09-21T13:58:17.777893+0000","last_unstale":"2024-09-21T14:02:40.100711+0000","last_undegraded":"2024-09-21T14:02:40.100711+0000","last_fullsized":"2024-09-21T14:02:40.100711+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:07.001135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:55:07.001135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:22.269018+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"290'27","reported_seq":616,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.099176+0000","last_change":"2024-09-21T13:58:11.718421+0000","last_active":"2024-09-21T14:02:40.099176+0000","last_peered":"2024-09-21T14:02:40.099176+0000","last_clean":"2024-09-21T14:02:40.099176+0000","last_became_active":"2024-09-21T13:57:12.703469+0000","last_became_peered":"2024-09-21T13:57:12.703469+0000","last_unstale":"2024-09-21T14:02:40.099176+0000","last_undegraded":"2024-09-21T14:02:40.099176+0000","last_fullsized":"2024-09-21T14:02:40.099176+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:58:11.718334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:11.718334+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":681,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.468443+0000","last_change":"2024-09-21T14:01:32.717551+0000","last_active":"2024-09-21T14:02:37.468443+0000","last_peered":"2024-09-21T14:02:37.468443+0000","last_clean":"2024-09-21T14:02:37.468443+0000","last_became_active":"2024-09-21T13:58:17.773464+0000","last_became_peered":"2024-09-21T13:58:17.773464+0000","last_unstale":"2024-09-21T14:02:37.468443+0000","last_undegraded":"2024-09-21T14:02:37.468443+0000","last_fullsized":"2024-09-21T14:02:37.468443+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:32.717480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:01:32.717480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:02.205485+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":635,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.107085+0000","last_change":"2024-09-21T14:01:41.194807+0000","last_active":"2024-09-21T14:02:40.107085+0000","last_peered":"2024-09-21T14:02:40.107085+0000","last_clean":"2024-09-21T14:02:40.107085+0000","last_became_active":"2024-09-21T14:01:41.194517+0000","last_became_peered":"2024-09-21T14:01:41.194517+0000","last_unstale":"2024-09-21T14:02:40.107085+0000","last_undegraded":"2024-09-21T14:02:40.107085+0000","last_fullsized":"2024-09-21T14:02:40.107085+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:31.127926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:31.127926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"290'15","reported_seq":707,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.469489+0000","last_change":"2024-09-21T14:00:02.475761+0000","last_active":"2024-09-21T14:02:37.469489+0000","last_peered":"2024-09-21T14:02:37.469489+0000","last_clean":"2024-09-21T14:02:37.469489+0000","last_became_active":"2024-09-21T13:57:12.681408+0000","last_became_peered":"2024-09-21T13:57:12.681408+0000","last_unstale":"2024-09-21T14:02:37.469489+0000","last_undegraded":"2024-09-21T14:02:37.469489+0000","last_fullsized":"2024-09-21T14:02:37.469489+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'15","last_scrub_stamp":"2024-09-21T14:00:02.475661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:00:02.475661+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:37.694022+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":576,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.103489+0000","last_change":"2024-09-21T14:01:41.188585+0000","last_active":"2024-09-21T14:02:40.103489+0000","last_peered":"2024-09-21T14:02:40.103489+0000","last_clean":"2024-09-21T14:02:40.103489+0000","last_became_active":"2024-09-21T14:01:41.188248+0000","last_became_peered":"2024-09-21T14:01:41.188248+0000","last_unstale":"2024-09-21T14:02:40.103489+0000","last_undegraded":"2024-09-21T14:02:40.103489+0000","last_fullsized":"2024-09-21T14:02:40.103489+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:21.284048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:56:21.284048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":612,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.471591+0000","last_change":"2024-09-21T13:59:32.202128+0000","last_active":"2024-09-21T14:02:37.471591+0000","last_peered":"2024-09-21T14:02:37.471591+0000","last_clean":"2024-09-21T14:02:37.471591+0000","last_became_active":"2024-09-21T13:59:32.201158+0000","last_became_peered":"2024-09-21T13:59:32.201158+0000","last_unstale":"2024-09-21T14:02:37.471591+0000","last_undegraded":"2024-09-21T14:02:37.471591+0000","last_fullsized":"2024-09-21T14:02:37.471591+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:25.792985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:54:25.792985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:46.991952+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":612,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.103754+0000","last_change":"2024-09-21T14:01:41.189797+0000","last_active":"2024-09-21T14:02:40.103754+0000","last_peered":"2024-09-21T14:02:40.103754+0000","last_clean":"2024-09-21T14:02:40.103754+0000","last_became_active":"2024-09-21T14:01:41.189373+0000","last_became_peered":"2024-09-21T14:01:41.189373+0000","last_unstale":"2024-09-21T14:02:40.103754+0000","last_undegraded":"2024-09-21T14:02:40.103754+0000","last_fullsized":"2024-09-21T14:02:40.103754+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:54.328275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:54.328275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"290'7","reported_seq":732,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.469480+0000","last_change":"2024-09-21T14:01:57.563250+0000","last_active":"2024-09-21T14:02:37.469480+0000","last_peered":"2024-09-21T14:02:37.469480+0000","last_clean":"2024-09-21T14:02:37.469480+0000","last_became_active":"2024-09-21T13:59:09.063053+0000","last_became_peered":"2024-09-21T13:59:09.063053+0000","last_unstale":"2024-09-21T14:02:37.469480+0000","last_undegraded":"2024-09-21T14:02:37.469480+0000","last_fullsized":"2024-09-21T14:02:37.469480+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'7","last_scrub_stamp":"2024-09-21T14:01:57.563184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:57.563184+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:19.737603+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":605,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.102972+0000","last_change":"2024-09-21T13:59:32.204565+0000","last_active":"2024-09-21T14:02:40.102972+0000","last_peered":"2024-09-21T14:02:40.102972+0000","last_clean":"2024-09-21T14:02:40.102972+0000","last_became_active":"2024-09-21T13:59:32.204174+0000","last_became_peered":"2024-09-21T13:59:32.204174+0000","last_unstale":"2024-09-21T14:02:40.102972+0000","last_undegraded":"2024-09-21T14:02:40.102972+0000","last_fullsized":"2024-09-21T14:02:40.102972+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:06.722293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:06.722293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":646,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.103574+0000","last_change":"2024-09-21T14:01:41.199085+0000","last_active":"2024-09-21T14:02:40.103574+0000","last_peered":"2024-09-21T14:02:40.103574+0000","last_clean":"2024-09-21T14:02:40.103574+0000","last_became_active":"2024-09-21T14:01:41.198672+0000","last_became_peered":"2024-09-21T14:01:41.198672+0000","last_unstale":"2024-09-21T14:02:40.103574+0000","last_undegraded":"2024-09-21T14:02:40.103574+0000","last_fullsized":"2024-09-21T14:02:40.103574+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:44.376215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:44.376215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":700,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.469202+0000","last_change":"2024-09-21T14:01:53.460257+0000","last_active":"2024-09-21T14:02:37.469202+0000","last_peered":"2024-09-21T14:02:37.469202+0000","last_clean":"2024-09-21T14:02:37.469202+0000","last_became_active":"2024-09-21T14:01:41.200360+0000","last_became_peered":"2024-09-21T14:01:41.200360+0000","last_unstale":"2024-09-21T14:02:37.469202+0000","last_undegraded":"2024-09-21T14:02:37.469202+0000","last_fullsized":"2024-09-21T14:02:37.469202+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:53.460180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:01:53.460180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:15.802657+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":618,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.469691+0000","last_change":"2024-09-21T14:02:36.679396+0000","last_active":"2024-09-21T14:02:37.469691+0000","last_peered":"2024-09-21T14:02:37.469691+0000","last_clean":"2024-09-21T14:02:37.469691+0000","last_became_active":"2024-09-21T13:59:32.204749+0000","last_became_peered":"2024-09-21T13:59:32.204749+0000","last_unstale":"2024-09-21T14:02:37.469691+0000","last_undegraded":"2024-09-21T14:02:37.469691+0000","last_fullsized":"2024-09-21T14:02:37.469691+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:36.679321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:02:36.679321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:43.067225+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"290'27","reported_seq":675,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.455223+0000","last_change":"2024-09-21T14:01:27.893648+0000","last_active":"2024-09-21T14:02:37.455223+0000","last_peered":"2024-09-21T14:02:37.455223+0000","last_clean":"2024-09-21T14:02:37.455223+0000","last_became_active":"2024-09-21T13:57:12.680783+0000","last_became_peered":"2024-09-21T13:57:12.680783+0000","last_unstale":"2024-09-21T14:02:37.455223+0000","last_undegraded":"2024-09-21T14:02:37.455223+0000","last_fullsized":"2024-09-21T14:02:37.455223+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'27","last_scrub_stamp":"2024-09-21T14:01:27.893587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:27.893587+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:53.939497+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":668,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.467107+0000","last_change":"2024-09-21T14:01:54.024745+0000","last_active":"2024-09-21T14:02:37.467107+0000","last_peered":"2024-09-21T14:02:37.467107+0000","last_clean":"2024-09-21T14:02:37.467107+0000","last_became_active":"2024-09-21T13:49:02.066796+0000","last_became_peered":"2024-09-21T13:49:02.066796+0000","last_unstale":"2024-09-21T14:02:37.467107+0000","last_undegraded":"2024-09-21T14:02:37.467107+0000","last_fullsized":"2024-09-21T14:02:37.467107+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:54.024702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:01:54.024702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:06.646444+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":622,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.102869+0000","last_change":"2024-09-21T14:01:41.191486+0000","last_active":"2024-09-21T14:02:40.102869+0000","last_peered":"2024-09-21T14:02:40.102869+0000","last_clean":"2024-09-21T14:02:40.102869+0000","last_became_active":"2024-09-21T14:01:41.191116+0000","last_became_peered":"2024-09-21T14:01:41.191116+0000","last_unstale":"2024-09-21T14:02:40.102869+0000","last_undegraded":"2024-09-21T14:02:40.102869+0000","last_fullsized":"2024-09-21T14:02:40.102869+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:57.962266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:57.962266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:08.315476+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":678,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.469586+0000","last_change":"2024-09-21T14:00:51.495433+0000","last_active":"2024-09-21T14:02:37.469586+0000","last_peered":"2024-09-21T14:02:37.469586+0000","last_clean":"2024-09-21T14:02:37.469586+0000","last_became_active":"2024-09-21T13:52:04.316016+0000","last_became_peered":"2024-09-21T13:52:04.316016+0000","last_unstale":"2024-09-21T14:02:37.469586+0000","last_undegraded":"2024-09-21T14:02:37.469586+0000","last_fullsized":"2024-09-21T14:02:37.469586+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:00:51.495368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:00:51.495368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:10.415465+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"290'18","reported_seq":628,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.100711+0000","last_change":"2024-09-21T13:59:23.041396+0000","last_active":"2024-09-21T14:02:40.100711+0000","last_peered":"2024-09-21T14:02:40.100711+0000","last_clean":"2024-09-21T14:02:40.100711+0000","last_became_active":"2024-09-21T13:57:12.684454+0000","last_became_peered":"2024-09-21T13:57:12.684454+0000","last_unstale":"2024-09-21T14:02:40.100711+0000","last_undegraded":"2024-09-21T14:02:40.100711+0000","last_fullsized":"2024-09-21T14:02:40.100711+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'18","last_scrub_stamp":"2024-09-21T13:59:23.041311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:23.041311+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":665,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.470098+0000","last_change":"2024-09-21T13:58:16.393763+0000","last_active":"2024-09-21T14:02:37.470098+0000","last_peered":"2024-09-21T14:02:37.470098+0000","last_clean":"2024-09-21T14:02:37.470098+0000","last_became_active":"2024-09-21T13:57:12.681897+0000","last_became_peered":"2024-09-21T13:57:12.681897+0000","last_unstale":"2024-09-21T14:02:37.470098+0000","last_undegraded":"2024-09-21T14:02:37.470098+0000","last_fullsized":"2024-09-21T14:02:37.470098+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:16.393696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:16.393696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":589,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.104482+0000","last_change":"2024-09-21T14:01:41.182300+0000","last_active":"2024-09-21T14:02:40.104482+0000","last_peered":"2024-09-21T14:02:40.104482+0000","last_clean":"2024-09-21T14:02:40.104482+0000","last_became_active":"2024-09-21T14:01:41.181946+0000","last_became_peered":"2024-09-21T14:01:41.181946+0000","last_unstale":"2024-09-21T14:02:40.104482+0000","last_undegraded":"2024-09-21T14:02:40.104482+0000","last_fullsized":"2024-09-21T14:02:40.104482+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T13:57:26.536684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:26.536684+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":688,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.466746+0000","last_change":"2024-09-21T14:01:58.994867+0000","last_active":"2024-09-21T14:02:37.466746+0000","last_peered":"2024-09-21T14:02:37.466746+0000","last_clean":"2024-09-21T14:02:37.466746+0000","last_became_active":"2024-09-21T13:57:12.685742+0000","last_became_peered":"2024-09-21T13:57:12.685742+0000","last_unstale":"2024-09-21T14:02:37.466746+0000","last_undegraded":"2024-09-21T14:02:37.466746+0000","last_fullsized":"2024-09-21T14:02:37.466746+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T14:01:58.994811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:01:58.994811+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:17.687542+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"290'20","reported_seq":569,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.105342+0000","last_change":"2024-09-21T13:59:32.242771+0000","last_active":"2024-09-21T14:02:40.105342+0000","last_peered":"2024-09-21T14:02:40.105342+0000","last_clean":"2024-09-21T14:02:40.105342+0000","last_became_active":"2024-09-21T13:59:32.213865+0000","last_became_peered":"2024-09-21T13:59:32.213865+0000","last_unstale":"2024-09-21T14:02:40.105342+0000","last_undegraded":"2024-09-21T14:02:40.105342+0000","last_fullsized":"2024-09-21T14:02:40.105342+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:57:27.578738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:57:27.578738+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"300'12","reported_seq":302,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.468801+0000","last_change":"2024-09-21T13:59:36.012775+0000","last_active":"2024-09-21T14:02:37.468801+0000","last_peered":"2024-09-21T14:02:37.468801+0000","last_clean":"2024-09-21T14:02:37.468801+0000","last_became_active":"2024-09-21T13:59:32.205051+0000","last_became_peered":"2024-09-21T13:59:32.205051+0000","last_unstale":"2024-09-21T14:02:37.468801+0000","last_undegraded":"2024-09-21T14:02:37.468801+0000","last_fullsized":"2024-09-21T14:02:37.468801+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'12","last_scrub_stamp":"2024-09-21T13:59:36.012693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:36.012693+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:20.587646+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"75'79","reported_seq":85,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.102403+0000","last_change":"2024-09-21T14:01:41.248772+0000","last_active":"2024-09-21T14:02:40.102403+0000","last_peered":"2024-09-21T14:02:40.102403+0000","last_clean":"2024-09-21T14:02:40.102403+0000","last_became_active":"2024-09-21T14:01:41.246888+0000","last_became_peered":"2024-09-21T14:01:41.246888+0000","last_unstale":"2024-09-21T14:02:40.102403+0000","last_undegraded":"2024-09-21T14:02:40.102403+0000","last_fullsized":"2024-09-21T14:02:40.102403+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53669,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"376'14953","reported_seq":15198,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:41.138687+0000","last_change":"2024-09-21T13:57:53.208687+0000","last_active":"2024-09-21T14:02:41.138687+0000","last_peered":"2024-09-21T14:02:41.138687+0000","last_clean":"2024-09-21T14:02:41.138687+0000","last_became_active":"2024-09-21T13:57:53.208352+0000","last_became_peered":"2024-09-21T13:57:53.208352+0000","last_unstale":"2024-09-21T14:02:41.138687+0000","last_undegraded":"2024-09-21T14:02:41.138687+0000","last_fullsized":"2024-09-21T14:02:41.138687+0000","mapping_epoch":260,"log_start":"314'10515","ondisk_log_start":"314'10515","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4438,"log_dups_size":0,"ondisk_log_size":4438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17235968,"num_objects":263,"num_object_clones":0,"num_object_copies":789,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14953,"num_write_kb":486912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"290'27","reported_seq":727,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.469678+0000","last_change":"2024-09-21T13:58:45.984691+0000","last_active":"2024-09-21T14:02:37.469678+0000","last_peered":"2024-09-21T14:02:37.469678+0000","last_clean":"2024-09-21T14:02:37.469678+0000","last_became_active":"2024-09-21T13:57:12.695609+0000","last_became_peered":"2024-09-21T13:57:12.695609+0000","last_unstale":"2024-09-21T14:02:37.469678+0000","last_undegraded":"2024-09-21T14:02:37.469678+0000","last_fullsized":"2024-09-21T14:02:37.469678+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:58:45.984575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:45.984575+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":614,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.097829+0000","last_change":"2024-09-21T13:59:32.205723+0000","last_active":"2024-09-21T14:02:40.097829+0000","last_peered":"2024-09-21T14:02:40.097829+0000","last_clean":"2024-09-21T14:02:40.097829+0000","last_became_active":"2024-09-21T13:59:32.205350+0000","last_became_peered":"2024-09-21T13:59:32.205350+0000","last_unstale":"2024-09-21T14:02:40.097829+0000","last_undegraded":"2024-09-21T14:02:40.097829+0000","last_fullsized":"2024-09-21T14:02:40.097829+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:32.668851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:32.668851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"346'10","reported_seq":858,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.469325+0000","last_change":"2024-09-21T14:01:55.521513+0000","last_active":"2024-09-21T14:02:37.469325+0000","last_peered":"2024-09-21T14:02:37.469325+0000","last_clean":"2024-09-21T14:02:37.469325+0000","last_became_active":"2024-09-21T14:01:41.200218+0000","last_became_peered":"2024-09-21T14:01:41.200218+0000","last_unstale":"2024-09-21T14:02:37.469325+0000","last_undegraded":"2024-09-21T14:02:37.469325+0000","last_fullsized":"2024-09-21T14:02:37.469325+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'10","last_scrub_stamp":"2024-09-21T14:01:55.521438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:01:55.521438+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:55.829821+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":541,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.106597+0000","last_change":"2024-09-21T14:01:41.207310+0000","last_active":"2024-09-21T14:02:40.106597+0000","last_peered":"2024-09-21T14:02:40.106597+0000","last_clean":"2024-09-21T14:02:40.106597+0000","last_became_active":"2024-09-21T14:01:41.206720+0000","last_became_peered":"2024-09-21T14:01:41.206720+0000","last_unstale":"2024-09-21T14:02:40.106597+0000","last_undegraded":"2024-09-21T14:02:40.106597+0000","last_fullsized":"2024-09-21T14:02:40.106597+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:52.244483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:52.244483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"75'79","reported_seq":136,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.469970+0000","last_change":"2024-09-21T14:02:06.395083+0000","last_active":"2024-09-21T14:02:37.469970+0000","last_peered":"2024-09-21T14:02:37.469970+0000","last_clean":"2024-09-21T14:02:37.469970+0000","last_became_active":"2024-09-21T14:00:41.137756+0000","last_became_peered":"2024-09-21T14:00:41.137756+0000","last_unstale":"2024-09-21T14:02:37.469970+0000","last_undegraded":"2024-09-21T14:02:37.469970+0000","last_fullsized":"2024-09-21T14:02:37.469970+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:02:06.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:02:06.395011+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:29.136654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"376'14937","reported_seq":19616,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:41.181854+0000","last_change":"2024-09-21T14:02:28.496980+0000","last_active":"2024-09-21T14:02:41.181854+0000","last_peered":"2024-09-21T14:02:41.181854+0000","last_clean":"2024-09-21T14:02:41.181854+0000","last_became_active":"2024-09-21T14:02:28.412817+0000","last_became_peered":"2024-09-21T14:02:28.412817+0000","last_unstale":"2024-09-21T14:02:41.181854+0000","last_undegraded":"2024-09-21T14:02:41.181854+0000","last_fullsized":"2024-09-21T14:02:41.181854+0000","mapping_epoch":371,"log_start":"314'10320","ondisk_log_start":"314'10320","created":260,"last_epoch_clean":372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4617,"log_dups_size":0,"ondisk_log_size":4617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17825816,"num_objects":273,"num_object_clones":0,"num_object_copies":819,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":273,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14937,"num_write_kb":486657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2195,"num_bytes_recovered":143720472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"290'30","reported_seq":638,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.097920+0000","last_change":"2024-09-21T13:59:32.246090+0000","last_active":"2024-09-21T14:02:40.097920+0000","last_peered":"2024-09-21T14:02:40.097920+0000","last_clean":"2024-09-21T14:02:40.097920+0000","last_became_active":"2024-09-21T13:59:32.205887+0000","last_became_peered":"2024-09-21T13:59:32.205887+0000","last_unstale":"2024-09-21T14:02:40.097920+0000","last_undegraded":"2024-09-21T14:02:40.097920+0000","last_fullsized":"2024-09-21T14:02:40.097920+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'18","last_scrub_stamp":"2024-09-21T13:54:19.001030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:19.001030+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:40.231306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":566,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.103131+0000","last_change":"2024-09-21T13:59:32.212559+0000","last_active":"2024-09-21T14:02:40.103131+0000","last_peered":"2024-09-21T14:02:40.103131+0000","last_clean":"2024-09-21T14:02:40.103131+0000","last_became_active":"2024-09-21T13:59:32.211004+0000","last_became_peered":"2024-09-21T13:59:32.211004+0000","last_unstale":"2024-09-21T14:02:40.103131+0000","last_undegraded":"2024-09-21T14:02:40.103131+0000","last_fullsized":"2024-09-21T14:02:40.103131+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:25.736461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:25.736461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":682,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.468857+0000","last_change":"2024-09-21T13:59:34.293330+0000","last_active":"2024-09-21T14:02:37.468857+0000","last_peered":"2024-09-21T14:02:37.468857+0000","last_clean":"2024-09-21T14:02:37.468857+0000","last_became_active":"2024-09-21T13:59:32.220304+0000","last_became_peered":"2024-09-21T13:59:32.220304+0000","last_unstale":"2024-09-21T14:02:37.468857+0000","last_undegraded":"2024-09-21T14:02:37.468857+0000","last_fullsized":"2024-09-21T14:02:37.468857+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:34.292892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:34.292892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:15.468420+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":662,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.469454+0000","last_change":"2024-09-21T13:59:23.334230+0000","last_active":"2024-09-21T14:02:37.469454+0000","last_peered":"2024-09-21T14:02:37.469454+0000","last_clean":"2024-09-21T14:02:37.469454+0000","last_became_active":"2024-09-21T13:57:12.680372+0000","last_became_peered":"2024-09-21T13:57:12.680372+0000","last_unstale":"2024-09-21T14:02:37.469454+0000","last_undegraded":"2024-09-21T14:02:37.469454+0000","last_fullsized":"2024-09-21T14:02:37.469454+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:23.334162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:23.334162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:17.515611+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"75'79","reported_seq":141,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.101309+0000","last_change":"2024-09-21T14:01:42.177646+0000","last_active":"2024-09-21T14:02:40.101309+0000","last_peered":"2024-09-21T14:02:40.101309+0000","last_clean":"2024-09-21T14:02:40.101309+0000","last_became_active":"2024-09-21T14:01:42.173042+0000","last_became_peered":"2024-09-21T14:01:42.173042+0000","last_unstale":"2024-09-21T14:02:40.101309+0000","last_undegraded":"2024-09-21T14:02:40.101309+0000","last_fullsized":"2024-09-21T14:02:40.101309+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:02:42.254915+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":709030,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"376'14800","reported_seq":21979,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:41.196187+0000","last_change":"2024-09-21T14:01:00.058151+0000","last_active":"2024-09-21T14:02:41.196187+0000","last_peered":"2024-09-21T14:02:41.196187+0000","last_clean":"2024-09-21T14:02:41.196187+0000","last_became_active":"2024-09-21T14:01:00.026605+0000","last_became_peered":"2024-09-21T14:01:00.026605+0000","last_unstale":"2024-09-21T14:02:41.196187+0000","last_undegraded":"2024-09-21T14:02:41.196187+0000","last_fullsized":"2024-09-21T14:02:41.196187+0000","mapping_epoch":329,"log_start":"314'10442","ondisk_log_start":"314'10442","created":260,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4358,"log_dups_size":0,"ondisk_log_size":4358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16384000,"num_objects":250,"num_object_clones":0,"num_object_copies":750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14800,"num_write_kb":481600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3456,"num_bytes_recovered":226492416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"290'28","reported_seq":771,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.468934+0000","last_change":"2024-09-21T14:02:02.471226+0000","last_active":"2024-09-21T14:02:37.468934+0000","last_peered":"2024-09-21T14:02:37.468934+0000","last_clean":"2024-09-21T14:02:37.468934+0000","last_became_active":"2024-09-21T14:01:41.200859+0000","last_became_peered":"2024-09-21T14:01:41.200859+0000","last_unstale":"2024-09-21T14:02:37.468934+0000","last_undegraded":"2024-09-21T14:02:37.468934+0000","last_fullsized":"2024-09-21T14:02:37.468934+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'28","last_scrub_stamp":"2024-09-21T14:02:02.471133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:02.471133+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:14.278651+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":670,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.476580+0000","last_change":"2024-09-21T13:57:38.046718+0000","last_active":"2024-09-21T14:02:37.476580+0000","last_peered":"2024-09-21T14:02:37.476580+0000","last_clean":"2024-09-21T14:02:37.476580+0000","last_became_active":"2024-09-21T13:48:58.053533+0000","last_became_peered":"2024-09-21T13:48:58.053533+0000","last_unstale":"2024-09-21T14:02:37.476580+0000","last_undegraded":"2024-09-21T14:02:37.476580+0000","last_fullsized":"2024-09-21T14:02:37.476580+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:57:38.046627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:38.046627+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:41.742834+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"241'10","reported_seq":712,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.098096+0000","last_change":"2024-09-21T13:58:06.789629+0000","last_active":"2024-09-21T14:02:40.098096+0000","last_peered":"2024-09-21T14:02:40.098096+0000","last_clean":"2024-09-21T14:02:40.098096+0000","last_became_active":"2024-09-21T13:57:12.686651+0000","last_became_peered":"2024-09-21T13:57:12.686651+0000","last_unstale":"2024-09-21T14:02:40.098096+0000","last_undegraded":"2024-09-21T14:02:40.098096+0000","last_fullsized":"2024-09-21T14:02:40.098096+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'10","last_scrub_stamp":"2024-09-21T13:58:06.789560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:06.789560+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":630,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.105045+0000","last_change":"2024-09-21T13:59:32.210257+0000","last_active":"2024-09-21T14:02:40.105045+0000","last_peered":"2024-09-21T14:02:40.105045+0000","last_clean":"2024-09-21T14:02:40.105045+0000","last_became_active":"2024-09-21T13:59:32.209615+0000","last_became_peered":"2024-09-21T13:59:32.209615+0000","last_unstale":"2024-09-21T14:02:40.105045+0000","last_undegraded":"2024-09-21T14:02:40.105045+0000","last_fullsized":"2024-09-21T14:02:40.105045+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:37.548064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:37.548064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"75'79","reported_seq":709,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.467967+0000","last_change":"2024-09-21T14:02:05.693250+0000","last_active":"2024-09-21T14:02:37.467967+0000","last_peered":"2024-09-21T14:02:37.467967+0000","last_clean":"2024-09-21T14:02:37.467967+0000","last_became_active":"2024-09-21T14:00:41.083425+0000","last_became_peered":"2024-09-21T14:00:41.083425+0000","last_unstale":"2024-09-21T14:02:37.467967+0000","last_undegraded":"2024-09-21T14:02:37.467967+0000","last_fullsized":"2024-09-21T14:02:37.467967+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:02:05.693193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:02:05.693193+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:28.620866+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.f","version":"376'15064","reported_seq":15296,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.859472+0000","last_change":"2024-09-21T14:01:41.336305+0000","last_active":"2024-09-21T14:02:40.859472+0000","last_peered":"2024-09-21T14:02:40.859472+0000","last_clean":"2024-09-21T14:02:40.859472+0000","last_became_active":"2024-09-21T14:01:41.202052+0000","last_became_peered":"2024-09-21T14:01:41.202052+0000","last_unstale":"2024-09-21T14:02:40.859472+0000","last_undegraded":"2024-09-21T14:02:40.859472+0000","last_fullsized":"2024-09-21T14:02:40.859472+0000","mapping_epoch":345,"log_start":"312'10102","ondisk_log_start":"312'10102","created":260,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4962,"log_dups_size":0,"ondisk_log_size":4962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17956864,"num_objects":274,"num_object_clones":0,"num_object_copies":822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15064,"num_write_kb":490816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"290'20","reported_seq":706,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.477334+0000","last_change":"2024-09-21T14:01:41.187743+0000","last_active":"2024-09-21T14:02:37.477334+0000","last_peered":"2024-09-21T14:02:37.477334+0000","last_clean":"2024-09-21T14:02:37.477334+0000","last_became_active":"2024-09-21T14:01:41.187060+0000","last_became_peered":"2024-09-21T14:01:41.187060+0000","last_unstale":"2024-09-21T14:02:37.477334+0000","last_undegraded":"2024-09-21T14:02:37.477334+0000","last_fullsized":"2024-09-21T14:02:37.477334+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:54:19.593611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:19.593611+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:45.999996+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"61'1","reported_seq":603,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.106692+0000","last_change":"2024-09-21T14:01:41.195580+0000","last_active":"2024-09-21T14:02:40.106692+0000","last_peered":"2024-09-21T14:02:40.106692+0000","last_clean":"2024-09-21T14:02:40.106692+0000","last_became_active":"2024-09-21T14:01:41.195268+0000","last_became_peered":"2024-09-21T14:01:41.195268+0000","last_unstale":"2024-09-21T14:02:40.106692+0000","last_undegraded":"2024-09-21T14:02:40.106692+0000","last_fullsized":"2024-09-21T14:02:40.106692+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:58:48.407734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:48.407734+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"346'15","reported_seq":295,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.467729+0000","last_change":"2024-09-21T14:01:41.209662+0000","last_active":"2024-09-21T14:02:37.467729+0000","last_peered":"2024-09-21T14:02:37.467729+0000","last_clean":"2024-09-21T14:02:37.467729+0000","last_became_active":"2024-09-21T14:01:41.209296+0000","last_became_peered":"2024-09-21T14:01:41.209296+0000","last_unstale":"2024-09-21T14:02:37.467729+0000","last_undegraded":"2024-09-21T14:02:37.467729+0000","last_fullsized":"2024-09-21T14:02:37.467729+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'12","last_scrub_stamp":"2024-09-21T13:57:30.133392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:30.133392+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:41.760065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":484,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.098009+0000","last_change":"2024-09-21T13:58:58.730439+0000","last_active":"2024-09-21T14:02:40.098009+0000","last_peered":"2024-09-21T14:02:40.098009+0000","last_clean":"2024-09-21T14:02:40.098009+0000","last_became_active":"2024-09-21T13:57:12.704897+0000","last_became_peered":"2024-09-21T13:57:12.704897+0000","last_unstale":"2024-09-21T14:02:40.098009+0000","last_undegraded":"2024-09-21T14:02:40.098009+0000","last_fullsized":"2024-09-21T14:02:40.098009+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:58.730368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:58.730368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"75'79","reported_seq":709,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.468337+0000","last_change":"2024-09-21T14:02:03.737005+0000","last_active":"2024-09-21T14:02:37.468337+0000","last_peered":"2024-09-21T14:02:37.468337+0000","last_clean":"2024-09-21T14:02:37.468337+0000","last_became_active":"2024-09-21T14:00:41.073047+0000","last_became_peered":"2024-09-21T14:00:41.073047+0000","last_unstale":"2024-09-21T14:02:37.468337+0000","last_undegraded":"2024-09-21T14:02:37.468337+0000","last_fullsized":"2024-09-21T14:02:37.468337+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:02:03.736928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:02:03.736928+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:21.563427+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"376'14674","reported_seq":24471,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:41.200793+0000","last_change":"2024-09-21T14:00:42.188751+0000","last_active":"2024-09-21T14:02:41.200793+0000","last_peered":"2024-09-21T14:02:41.200793+0000","last_clean":"2024-09-21T14:02:41.200793+0000","last_became_active":"2024-09-21T14:00:42.129476+0000","last_became_peered":"2024-09-21T14:00:42.129476+0000","last_unstale":"2024-09-21T14:02:41.200793+0000","last_undegraded":"2024-09-21T14:02:41.200793+0000","last_fullsized":"2024-09-21T14:02:41.200793+0000","mapping_epoch":321,"log_start":"314'10069","ondisk_log_start":"314'10069","created":260,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4605,"log_dups_size":0,"ondisk_log_size":4605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16515072,"num_objects":252,"num_object_clones":0,"num_object_copies":756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14674,"num_write_kb":477632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4765,"num_bytes_recovered":312213504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"290'29","reported_seq":609,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.103720+0000","last_change":"2024-09-21T14:01:41.188261+0000","last_active":"2024-09-21T14:02:40.103720+0000","last_peered":"2024-09-21T14:02:40.103720+0000","last_clean":"2024-09-21T14:02:40.103720+0000","last_became_active":"2024-09-21T14:01:41.186960+0000","last_became_peered":"2024-09-21T14:01:41.186960+0000","last_unstale":"2024-09-21T14:02:40.103720+0000","last_undegraded":"2024-09-21T14:02:40.103720+0000","last_fullsized":"2024-09-21T14:02:40.103720+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-21T13:56:55.319759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:56:55.319759+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":683,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.468291+0000","last_change":"2024-09-21T14:01:41.185706+0000","last_active":"2024-09-21T14:02:37.468291+0000","last_peered":"2024-09-21T14:02:37.468291+0000","last_clean":"2024-09-21T14:02:37.468291+0000","last_became_active":"2024-09-21T14:01:41.185400+0000","last_became_peered":"2024-09-21T14:01:41.185400+0000","last_unstale":"2024-09-21T14:02:37.468291+0000","last_undegraded":"2024-09-21T14:02:37.468291+0000","last_fullsized":"2024-09-21T14:02:37.468291+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:16.712990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:16.712990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":251,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.467750+0000","last_change":"2024-09-21T14:01:52.671029+0000","last_active":"2024-09-21T14:02:37.467750+0000","last_peered":"2024-09-21T14:02:37.467750+0000","last_clean":"2024-09-21T14:02:37.467750+0000","last_became_active":"2024-09-21T14:01:41.193973+0000","last_became_peered":"2024-09-21T14:01:41.193973+0000","last_unstale":"2024-09-21T14:02:37.467750+0000","last_undegraded":"2024-09-21T14:02:37.467750+0000","last_fullsized":"2024-09-21T14:02:37.467750+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:52.670974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:01:52.670974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:58.750020+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":674,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.466444+0000","last_change":"2024-09-21T13:58:43.952014+0000","last_active":"2024-09-21T14:02:37.466444+0000","last_peered":"2024-09-21T14:02:37.466444+0000","last_clean":"2024-09-21T14:02:37.466444+0000","last_became_active":"2024-09-21T13:58:17.757628+0000","last_became_peered":"2024-09-21T13:58:17.757628+0000","last_unstale":"2024-09-21T14:02:37.466444+0000","last_undegraded":"2024-09-21T14:02:37.466444+0000","last_fullsized":"2024-09-21T14:02:37.466444+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:43.951941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:43.951941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"75'79","reported_seq":85,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.103722+0000","last_change":"2024-09-21T14:01:41.199377+0000","last_active":"2024-09-21T14:02:40.103722+0000","last_peered":"2024-09-21T14:02:40.103722+0000","last_clean":"2024-09-21T14:02:40.103722+0000","last_became_active":"2024-09-21T14:01:41.198750+0000","last_became_peered":"2024-09-21T14:01:41.198750+0000","last_unstale":"2024-09-21T14:02:40.103722+0000","last_undegraded":"2024-09-21T14:02:40.103722+0000","last_fullsized":"2024-09-21T14:02:40.103722+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:02:43.568139+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"376'15035","reported_seq":25866,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:41.181557+0000","last_change":"2024-09-21T14:01:44.191023+0000","last_active":"2024-09-21T14:02:41.181557+0000","last_peered":"2024-09-21T14:02:41.181557+0000","last_clean":"2024-09-21T14:02:41.181557+0000","last_became_active":"2024-09-21T14:01:44.174262+0000","last_became_peered":"2024-09-21T14:01:44.174262+0000","last_unstale":"2024-09-21T14:02:41.181557+0000","last_undegraded":"2024-09-21T14:02:41.181557+0000","last_fullsized":"2024-09-21T14:02:41.181557+0000","mapping_epoch":348,"log_start":"314'10431","ondisk_log_start":"314'10431","created":260,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4604,"log_dups_size":0,"ondisk_log_size":4604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15532032,"num_objects":237,"num_object_clones":0,"num_object_copies":711,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15035,"num_write_kb":488704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5260,"num_bytes_recovered":344588288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"290'44","reported_seq":914,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.468191+0000","last_change":"2024-09-21T13:58:56.076153+0000","last_active":"2024-09-21T14:02:37.468191+0000","last_peered":"2024-09-21T14:02:37.468191+0000","last_clean":"2024-09-21T14:02:37.468191+0000","last_became_active":"2024-09-21T13:57:12.684690+0000","last_became_peered":"2024-09-21T13:57:12.684690+0000","last_unstale":"2024-09-21T14:02:37.468191+0000","last_undegraded":"2024-09-21T14:02:37.468191+0000","last_fullsized":"2024-09-21T14:02:37.468191+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'32","last_scrub_stamp":"2024-09-21T13:58:56.076074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:56.076074+0000","objects_scrubbed":10,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":690,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.469699+0000","last_change":"2024-09-21T14:01:58.524159+0000","last_active":"2024-09-21T14:02:37.469699+0000","last_peered":"2024-09-21T14:02:37.469699+0000","last_clean":"2024-09-21T14:02:37.469699+0000","last_became_active":"2024-09-21T13:58:17.770937+0000","last_became_peered":"2024-09-21T13:58:17.770937+0000","last_unstale":"2024-09-21T14:02:37.469699+0000","last_undegraded":"2024-09-21T14:02:37.469699+0000","last_fullsized":"2024-09-21T14:02:37.469699+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:58.524114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:01:58.524114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:07.944133+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":622,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.102312+0000","last_change":"2024-09-21T14:01:41.249061+0000","last_active":"2024-09-21T14:02:40.102312+0000","last_peered":"2024-09-21T14:02:40.102312+0000","last_clean":"2024-09-21T14:02:40.102312+0000","last_became_active":"2024-09-21T14:01:41.247642+0000","last_became_peered":"2024-09-21T14:01:41.247642+0000","last_unstale":"2024-09-21T14:02:40.102312+0000","last_undegraded":"2024-09-21T14:02:40.102312+0000","last_fullsized":"2024-09-21T14:02:40.102312+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-21T13:59:22.186393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:59:22.186393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":601,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.466929+0000","last_change":"2024-09-21T13:55:07.021128+0000","last_active":"2024-09-21T14:02:37.466929+0000","last_peered":"2024-09-21T14:02:37.466929+0000","last_clean":"2024-09-21T14:02:37.466929+0000","last_became_active":"2024-09-21T13:50:29.688249+0000","last_became_peered":"2024-09-21T13:50:29.688249+0000","last_unstale":"2024-09-21T14:02:37.466929+0000","last_undegraded":"2024-09-21T14:02:37.466929+0000","last_fullsized":"2024-09-21T14:02:37.466929+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:07.021059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:55:07.021059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:47.974697+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":593,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.101068+0000","last_change":"2024-09-21T13:59:32.209357+0000","last_active":"2024-09-21T14:02:40.101068+0000","last_peered":"2024-09-21T14:02:40.101068+0000","last_clean":"2024-09-21T14:02:40.101068+0000","last_became_active":"2024-09-21T13:59:32.208980+0000","last_became_peered":"2024-09-21T13:59:32.208980+0000","last_unstale":"2024-09-21T14:02:40.101068+0000","last_undegraded":"2024-09-21T14:02:40.101068+0000","last_fullsized":"2024-09-21T14:02:40.101068+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:16.797227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:16.797227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"75'79","reported_seq":717,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.102797+0000","last_change":"2024-09-21T14:00:41.077286+0000","last_active":"2024-09-21T14:02:40.102797+0000","last_peered":"2024-09-21T14:02:40.102797+0000","last_clean":"2024-09-21T14:02:40.102797+0000","last_became_active":"2024-09-21T14:00:41.076948+0000","last_became_peered":"2024-09-21T14:00:41.076948+0000","last_unstale":"2024-09-21T14:02:40.102797+0000","last_undegraded":"2024-09-21T14:02:40.102797+0000","last_fullsized":"2024-09-21T14:02:40.102797+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:02:44.823429+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.a","version":"376'14572","reported_seq":15511,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.871601+0000","last_change":"2024-09-21T14:01:41.339174+0000","last_active":"2024-09-21T14:02:40.871601+0000","last_peered":"2024-09-21T14:02:40.871601+0000","last_clean":"2024-09-21T14:02:40.871601+0000","last_became_active":"2024-09-21T14:01:41.190267+0000","last_became_peered":"2024-09-21T14:01:41.190267+0000","last_unstale":"2024-09-21T14:02:40.871601+0000","last_undegraded":"2024-09-21T14:02:40.871601+0000","last_fullsized":"2024-09-21T14:02:40.871601+0000","mapping_epoch":345,"log_start":"312'9574","ondisk_log_start":"312'9574","created":260,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4998,"log_dups_size":0,"ondisk_log_size":4998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19005440,"num_objects":290,"num_object_clones":0,"num_object_copies":870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14572,"num_write_kb":475584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5098,"num_bytes_recovered":241565696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"290'17","reported_seq":709,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.469724+0000","last_change":"2024-09-21T14:01:41.200809+0000","last_active":"2024-09-21T14:02:37.469724+0000","last_peered":"2024-09-21T14:02:37.469724+0000","last_clean":"2024-09-21T14:02:37.469724+0000","last_became_active":"2024-09-21T14:01:41.200365+0000","last_became_peered":"2024-09-21T14:01:41.200365+0000","last_unstale":"2024-09-21T14:02:37.469724+0000","last_undegraded":"2024-09-21T14:02:37.469724+0000","last_fullsized":"2024-09-21T14:02:37.469724+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'13","last_scrub_stamp":"2024-09-21T13:58:11.619848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:11.619848+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":710,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.470883+0000","last_change":"2024-09-21T13:59:29.454265+0000","last_active":"2024-09-21T14:02:37.470883+0000","last_peered":"2024-09-21T14:02:37.470883+0000","last_clean":"2024-09-21T14:02:37.470883+0000","last_became_active":"2024-09-21T13:58:17.759671+0000","last_became_peered":"2024-09-21T13:58:17.759671+0000","last_unstale":"2024-09-21T14:02:37.470883+0000","last_undegraded":"2024-09-21T14:02:37.470883+0000","last_fullsized":"2024-09-21T14:02:37.470883+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:29.454085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:59:29.454085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":600,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.454199+0000","last_change":"2024-09-21T13:58:41.644876+0000","last_active":"2024-09-21T14:02:37.454199+0000","last_peered":"2024-09-21T14:02:37.454199+0000","last_clean":"2024-09-21T14:02:37.454199+0000","last_became_active":"2024-09-21T13:57:12.682147+0000","last_became_peered":"2024-09-21T13:57:12.682147+0000","last_unstale":"2024-09-21T14:02:37.454199+0000","last_undegraded":"2024-09-21T14:02:37.454199+0000","last_fullsized":"2024-09-21T14:02:37.454199+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:58:41.644789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:58:41.644789+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:13.796854+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":578,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.454719+0000","last_change":"2024-09-21T14:01:50.880840+0000","last_active":"2024-09-21T14:02:37.454719+0000","last_peered":"2024-09-21T14:02:37.454719+0000","last_clean":"2024-09-21T14:02:37.454719+0000","last_became_active":"2024-09-21T14:01:41.182000+0000","last_became_peered":"2024-09-21T14:01:41.182000+0000","last_unstale":"2024-09-21T14:02:37.454719+0000","last_undegraded":"2024-09-21T14:02:37.454719+0000","last_fullsized":"2024-09-21T14:02:37.454719+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:50.880471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:01:50.880471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:14.118692+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":638,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.105287+0000","last_change":"2024-09-21T13:59:32.222791+0000","last_active":"2024-09-21T14:02:40.105287+0000","last_peered":"2024-09-21T14:02:40.105287+0000","last_clean":"2024-09-21T14:02:40.105287+0000","last_became_active":"2024-09-21T13:59:32.221502+0000","last_became_peered":"2024-09-21T13:59:32.221502+0000","last_unstale":"2024-09-21T14:02:40.105287+0000","last_undegraded":"2024-09-21T14:02:40.105287+0000","last_fullsized":"2024-09-21T14:02:40.105287+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:14.673582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:14.673582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"75'79","reported_seq":120,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.466913+0000","last_change":"2024-09-21T14:00:41.071631+0000","last_active":"2024-09-21T14:02:37.466913+0000","last_peered":"2024-09-21T14:02:37.466913+0000","last_clean":"2024-09-21T14:02:37.466913+0000","last_became_active":"2024-09-21T14:00:41.071042+0000","last_became_peered":"2024-09-21T14:00:41.071042+0000","last_unstale":"2024-09-21T14:02:37.466913+0000","last_undegraded":"2024-09-21T14:02:37.466913+0000","last_fullsized":"2024-09-21T14:02:37.466913+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:02:42.091028+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.b","version":"375'14754","reported_seq":18965,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:38.537796+0000","last_change":"2024-09-21T14:01:54.238547+0000","last_active":"2024-09-21T14:02:38.537796+0000","last_peered":"2024-09-21T14:02:38.537796+0000","last_clean":"2024-09-21T14:02:38.537796+0000","last_became_active":"2024-09-21T14:01:41.210137+0000","last_became_peered":"2024-09-21T14:01:41.210137+0000","last_unstale":"2024-09-21T14:02:38.537796+0000","last_undegraded":"2024-09-21T14:02:38.537796+0000","last_fullsized":"2024-09-21T14:02:38.537796+0000","mapping_epoch":345,"log_start":"307'8832","ondisk_log_start":"307'8832","created":260,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"356'12122","last_scrub_stamp":"2024-09-21T14:01:54.238344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T14:01:54.238344+0000","objects_scrubbed":3072,"log_size":5922,"log_dups_size":0,"ondisk_log_size":5922,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:15.605090+0000","scrub_duration":597,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28180480,"num_objects":430,"num_object_clones":0,"num_object_copies":1290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14754,"num_write_kb":485888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1910,"num_bytes_recovered":123994112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"290'24","reported_seq":760,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.476412+0000","last_change":"2024-09-21T14:01:41.185105+0000","last_active":"2024-09-21T14:02:37.476412+0000","last_peered":"2024-09-21T14:02:37.476412+0000","last_clean":"2024-09-21T14:02:37.476412+0000","last_became_active":"2024-09-21T14:01:41.184728+0000","last_became_peered":"2024-09-21T14:01:41.184728+0000","last_unstale":"2024-09-21T14:02:37.476412+0000","last_undegraded":"2024-09-21T14:02:37.476412+0000","last_fullsized":"2024-09-21T14:02:37.476412+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'24","last_scrub_stamp":"2024-09-21T13:59:28.458768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:28.458768+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":611,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.469056+0000","last_change":"2024-09-21T13:59:59.450613+0000","last_active":"2024-09-21T14:02:37.469056+0000","last_peered":"2024-09-21T14:02:37.469056+0000","last_clean":"2024-09-21T14:02:37.469056+0000","last_became_active":"2024-09-21T13:57:12.681030+0000","last_became_peered":"2024-09-21T13:57:12.681030+0000","last_unstale":"2024-09-21T14:02:37.469056+0000","last_undegraded":"2024-09-21T14:02:37.469056+0000","last_fullsized":"2024-09-21T14:02:37.469056+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:59.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:59:59.450546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":1361,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.102976+0000","last_change":"2024-09-21T13:59:32.218796+0000","last_active":"2024-09-21T14:02:40.102976+0000","last_peered":"2024-09-21T14:02:40.102976+0000","last_clean":"2024-09-21T14:02:40.102976+0000","last_became_active":"2024-09-21T13:59:32.203361+0000","last_became_peered":"2024-09-21T13:59:32.203361+0000","last_unstale":"2024-09-21T14:02:40.102976+0000","last_undegraded":"2024-09-21T14:02:40.102976+0000","last_fullsized":"2024-09-21T14:02:40.102976+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-21T13:56:29.617143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:56:29.617143+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047048000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":195,"num_read_kb":190,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":593,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.103010+0000","last_change":"2024-09-21T14:01:41.212595+0000","last_active":"2024-09-21T14:02:40.103010+0000","last_peered":"2024-09-21T14:02:40.103010+0000","last_clean":"2024-09-21T14:02:40.103010+0000","last_became_active":"2024-09-21T14:01:41.200531+0000","last_became_peered":"2024-09-21T14:01:41.200531+0000","last_unstale":"2024-09-21T14:02:40.103010+0000","last_undegraded":"2024-09-21T14:02:40.103010+0000","last_fullsized":"2024-09-21T14:02:40.103010+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:33.316230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:56:33.316230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":695,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.466323+0000","last_change":"2024-09-21T14:01:41.184939+0000","last_active":"2024-09-21T14:02:37.466323+0000","last_peered":"2024-09-21T14:02:37.466323+0000","last_clean":"2024-09-21T14:02:37.466323+0000","last_became_active":"2024-09-21T14:01:41.184664+0000","last_became_peered":"2024-09-21T14:01:41.184664+0000","last_unstale":"2024-09-21T14:02:37.466323+0000","last_undegraded":"2024-09-21T14:02:37.466323+0000","last_fullsized":"2024-09-21T14:02:37.466323+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:26.557002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:26.557002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"376'15156","reported_seq":15440,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:41.165478+0000","last_change":"2024-09-21T14:01:41.317185+0000","last_active":"2024-09-21T14:02:41.165478+0000","last_peered":"2024-09-21T14:02:41.165478+0000","last_clean":"2024-09-21T14:02:41.165478+0000","last_became_active":"2024-09-21T14:01:41.197526+0000","last_became_peered":"2024-09-21T14:01:41.197526+0000","last_unstale":"2024-09-21T14:02:41.165478+0000","last_undegraded":"2024-09-21T14:02:41.165478+0000","last_fullsized":"2024-09-21T14:02:41.165478+0000","mapping_epoch":345,"log_start":"314'10787","ondisk_log_start":"314'10787","created":260,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4369,"log_dups_size":0,"ondisk_log_size":4369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16515072,"num_objects":252,"num_object_clones":0,"num_object_copies":756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15156,"num_write_kb":493056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"300'32","reported_seq":348,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.468275+0000","last_change":"2024-09-21T14:00:47.130089+0000","last_active":"2024-09-21T14:02:37.468275+0000","last_peered":"2024-09-21T14:02:37.468275+0000","last_clean":"2024-09-21T14:02:37.468275+0000","last_became_active":"2024-09-21T13:59:32.204624+0000","last_became_peered":"2024-09-21T13:59:32.204624+0000","last_unstale":"2024-09-21T14:02:37.468275+0000","last_undegraded":"2024-09-21T14:02:37.468275+0000","last_fullsized":"2024-09-21T14:02:37.468275+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'32","last_scrub_stamp":"2024-09-21T14:00:47.129789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:00:47.129789+0000","objects_scrubbed":8,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:15.616969+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":651,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.101097+0000","last_change":"2024-09-21T14:01:41.190537+0000","last_active":"2024-09-21T14:02:40.101097+0000","last_peered":"2024-09-21T14:02:40.101097+0000","last_clean":"2024-09-21T14:02:40.101097+0000","last_became_active":"2024-09-21T14:01:41.190210+0000","last_became_peered":"2024-09-21T14:01:41.190210+0000","last_unstale":"2024-09-21T14:02:40.101097+0000","last_undegraded":"2024-09-21T14:02:40.101097+0000","last_fullsized":"2024-09-21T14:02:40.101097+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:57.772975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:57.772975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"75'79","reported_seq":712,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.103127+0000","last_change":"2024-09-21T14:00:39.339360+0000","last_active":"2024-09-21T14:02:40.103127+0000","last_peered":"2024-09-21T14:02:40.103127+0000","last_clean":"2024-09-21T14:02:40.103127+0000","last_became_active":"2024-09-21T14:00:39.339077+0000","last_became_peered":"2024-09-21T14:00:39.339077+0000","last_unstale":"2024-09-21T14:02:40.103127+0000","last_undegraded":"2024-09-21T14:02:40.103127+0000","last_fullsized":"2024-09-21T14:02:40.103127+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:02:42.782456+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":19,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53669,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":626,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.104588+0000","last_change":"2024-09-21T14:01:41.182186+0000","last_active":"2024-09-21T14:02:40.104588+0000","last_peered":"2024-09-21T14:02:40.104588+0000","last_clean":"2024-09-21T14:02:40.104588+0000","last_became_active":"2024-09-21T14:01:41.181899+0000","last_became_peered":"2024-09-21T14:01:41.181899+0000","last_unstale":"2024-09-21T14:02:40.104588+0000","last_undegraded":"2024-09-21T14:02:40.104588+0000","last_fullsized":"2024-09-21T14:02:40.104588+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:15.700409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:15.700409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":551,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.102733+0000","last_change":"2024-09-21T14:01:41.196509+0000","last_active":"2024-09-21T14:02:40.102733+0000","last_peered":"2024-09-21T14:02:40.102733+0000","last_clean":"2024-09-21T14:02:40.102733+0000","last_became_active":"2024-09-21T14:01:41.196243+0000","last_became_peered":"2024-09-21T14:01:41.196243+0000","last_unstale":"2024-09-21T14:02:40.102733+0000","last_undegraded":"2024-09-21T14:02:40.102733+0000","last_fullsized":"2024-09-21T14:02:40.102733+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:21.153018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:21.153018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"376'14917","reported_seq":24833,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:41.198267+0000","last_change":"2024-09-21T14:00:42.107898+0000","last_active":"2024-09-21T14:02:41.198267+0000","last_peered":"2024-09-21T14:02:41.198267+0000","last_clean":"2024-09-21T14:02:41.198267+0000","last_became_active":"2024-09-21T14:00:42.085387+0000","last_became_peered":"2024-09-21T14:00:42.085387+0000","last_unstale":"2024-09-21T14:02:41.198267+0000","last_undegraded":"2024-09-21T14:02:41.198267+0000","last_fullsized":"2024-09-21T14:02:41.198267+0000","mapping_epoch":321,"log_start":"314'10312","ondisk_log_start":"314'10312","created":260,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4605,"log_dups_size":0,"ondisk_log_size":4605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16449536,"num_objects":251,"num_object_clones":0,"num_object_copies":753,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14917,"num_write_kb":485376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4827,"num_bytes_recovered":316342272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"290'19","reported_seq":736,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.469386+0000","last_change":"2024-09-21T14:02:28.553549+0000","last_active":"2024-09-21T14:02:37.469386+0000","last_peered":"2024-09-21T14:02:37.469386+0000","last_clean":"2024-09-21T14:02:37.469386+0000","last_became_active":"2024-09-21T13:58:17.776310+0000","last_became_peered":"2024-09-21T13:58:17.776310+0000","last_unstale":"2024-09-21T14:02:37.469386+0000","last_undegraded":"2024-09-21T14:02:37.469386+0000","last_fullsized":"2024-09-21T14:02:37.469386+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'19","last_scrub_stamp":"2024-09-21T14:02:28.553065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:28.553065+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:33.636087+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":698,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.469175+0000","last_change":"2024-09-21T14:02:03.437581+0000","last_active":"2024-09-21T14:02:37.469175+0000","last_peered":"2024-09-21T14:02:37.469175+0000","last_clean":"2024-09-21T14:02:37.469175+0000","last_became_active":"2024-09-21T13:59:32.223247+0000","last_became_peered":"2024-09-21T13:59:32.223247+0000","last_unstale":"2024-09-21T14:02:37.469175+0000","last_undegraded":"2024-09-21T14:02:37.469175+0000","last_fullsized":"2024-09-21T14:02:37.469175+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:03.437510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:02:03.437510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:14.620200+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":619,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.102561+0000","last_change":"2024-09-21T14:01:41.194538+0000","last_active":"2024-09-21T14:02:40.102561+0000","last_peered":"2024-09-21T14:02:40.102561+0000","last_clean":"2024-09-21T14:02:40.102561+0000","last_became_active":"2024-09-21T14:01:41.193941+0000","last_became_peered":"2024-09-21T14:01:41.193941+0000","last_unstale":"2024-09-21T14:02:40.102561+0000","last_undegraded":"2024-09-21T14:02:40.102561+0000","last_fullsized":"2024-09-21T14:02:40.102561+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:34.299722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:55:34.299722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":629,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.102108+0000","last_change":"2024-09-21T14:01:41.190501+0000","last_active":"2024-09-21T14:02:40.102108+0000","last_peered":"2024-09-21T14:02:40.102108+0000","last_clean":"2024-09-21T14:02:40.102108+0000","last_became_active":"2024-09-21T14:01:41.190192+0000","last_became_peered":"2024-09-21T14:01:41.190192+0000","last_unstale":"2024-09-21T14:02:40.102108+0000","last_undegraded":"2024-09-21T14:02:40.102108+0000","last_fullsized":"2024-09-21T14:02:40.102108+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:58.952218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:54:58.952218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:18.255572+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"75'79","reported_seq":92,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.102144+0000","last_change":"2024-09-21T14:01:41.189146+0000","last_active":"2024-09-21T14:02:40.102144+0000","last_peered":"2024-09-21T14:02:40.102144+0000","last_clean":"2024-09-21T14:02:40.102144+0000","last_became_active":"2024-09-21T14:01:41.188685+0000","last_became_peered":"2024-09-21T14:01:41.188685+0000","last_unstale":"2024-09-21T14:02:40.102144+0000","last_undegraded":"2024-09-21T14:02:40.102144+0000","last_fullsized":"2024-09-21T14:02:40.102144+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:02:42.589731+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.0","version":"375'14730","reported_seq":18911,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:39.693282+0000","last_change":"2024-09-21T13:59:24.817101+0000","last_active":"2024-09-21T14:02:39.693282+0000","last_peered":"2024-09-21T14:02:39.693282+0000","last_clean":"2024-09-21T14:02:39.693282+0000","last_became_active":"2024-09-21T13:58:36.723110+0000","last_became_peered":"2024-09-21T13:58:36.723110+0000","last_unstale":"2024-09-21T14:02:39.693282+0000","last_undegraded":"2024-09-21T14:02:39.693282+0000","last_fullsized":"2024-09-21T14:02:39.693282+0000","mapping_epoch":279,"log_start":"313'10181","ondisk_log_start":"313'10181","created":260,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'7035","last_scrub_stamp":"2024-09-21T13:59:24.816903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:59:24.816903+0000","objects_scrubbed":6955,"log_size":4549,"log_dups_size":0,"ondisk_log_size":4549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2446,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22413312,"num_objects":342,"num_object_clones":0,"num_object_copies":1026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14730,"num_write_kb":482304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1924,"num_bytes_recovered":126091264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"290'31","reported_seq":724,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.469832+0000","last_change":"2024-09-21T14:01:03.541547+0000","last_active":"2024-09-21T14:02:37.469832+0000","last_peered":"2024-09-21T14:02:37.469832+0000","last_clean":"2024-09-21T14:02:37.469832+0000","last_became_active":"2024-09-21T13:52:04.313218+0000","last_became_peered":"2024-09-21T13:52:04.313218+0000","last_unstale":"2024-09-21T14:02:37.469832+0000","last_undegraded":"2024-09-21T14:02:37.469832+0000","last_fullsized":"2024-09-21T14:02:37.469832+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'31","last_scrub_stamp":"2024-09-21T14:01:03.541432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:03.541432+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":240,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.467577+0000","last_change":"2024-09-21T14:01:29.924408+0000","last_active":"2024-09-21T14:02:37.467577+0000","last_peered":"2024-09-21T14:02:37.467577+0000","last_clean":"2024-09-21T14:02:37.467577+0000","last_became_active":"2024-09-21T13:58:17.762520+0000","last_became_peered":"2024-09-21T13:58:17.762520+0000","last_unstale":"2024-09-21T14:02:37.467577+0000","last_undegraded":"2024-09-21T14:02:37.467577+0000","last_fullsized":"2024-09-21T14:02:37.467577+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:29.924331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:01:29.924331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:43.977176+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":653,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.466426+0000","last_change":"2024-09-21T13:58:52.030800+0000","last_active":"2024-09-21T14:02:37.466426+0000","last_peered":"2024-09-21T14:02:37.466426+0000","last_clean":"2024-09-21T14:02:37.466426+0000","last_became_active":"2024-09-21T13:49:02.067472+0000","last_became_peered":"2024-09-21T13:49:02.067472+0000","last_unstale":"2024-09-21T14:02:37.466426+0000","last_undegraded":"2024-09-21T14:02:37.466426+0000","last_fullsized":"2024-09-21T14:02:37.466426+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:52.030718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:52.030718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":535,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.455033+0000","last_change":"2024-09-21T13:57:12.683321+0000","last_active":"2024-09-21T14:02:37.455033+0000","last_peered":"2024-09-21T14:02:37.455033+0000","last_clean":"2024-09-21T14:02:37.455033+0000","last_became_active":"2024-09-21T13:57:12.682979+0000","last_became_peered":"2024-09-21T13:57:12.682979+0000","last_unstale":"2024-09-21T14:02:37.455033+0000","last_undegraded":"2024-09-21T14:02:37.455033+0000","last_fullsized":"2024-09-21T14:02:37.455033+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:18.027944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:18.027944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:21.753981+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"75'79","reported_seq":717,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.107035+0000","last_change":"2024-09-21T14:00:41.140108+0000","last_active":"2024-09-21T14:02:40.107035+0000","last_peered":"2024-09-21T14:02:40.107035+0000","last_clean":"2024-09-21T14:02:40.107035+0000","last_became_active":"2024-09-21T14:00:41.139315+0000","last_became_peered":"2024-09-21T14:00:41.139315+0000","last_unstale":"2024-09-21T14:02:40.107035+0000","last_undegraded":"2024-09-21T14:02:40.107035+0000","last_fullsized":"2024-09-21T14:02:40.107035+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1","version":"376'14843","reported_seq":27128,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:41.158847+0000","last_change":"2024-09-21T14:01:10.113546+0000","last_active":"2024-09-21T14:02:41.158847+0000","last_peered":"2024-09-21T14:02:41.158847+0000","last_clean":"2024-09-21T14:02:41.158847+0000","last_became_active":"2024-09-21T14:01:10.092192+0000","last_became_peered":"2024-09-21T14:01:10.092192+0000","last_unstale":"2024-09-21T14:02:41.158847+0000","last_undegraded":"2024-09-21T14:02:41.158847+0000","last_fullsized":"2024-09-21T14:02:41.158847+0000","mapping_epoch":333,"log_start":"314'10445","ondisk_log_start":"314'10445","created":260,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4398,"log_dups_size":0,"ondisk_log_size":4398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17367040,"num_objects":265,"num_object_clones":0,"num_object_copies":795,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14843,"num_write_kb":483456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5999,"num_bytes_recovered":393150464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"290'20","reported_seq":743,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.470188+0000","last_change":"2024-09-21T13:58:17.789407+0000","last_active":"2024-09-21T14:02:37.470188+0000","last_peered":"2024-09-21T14:02:37.470188+0000","last_clean":"2024-09-21T14:02:37.470188+0000","last_became_active":"2024-09-21T13:58:17.765427+0000","last_became_peered":"2024-09-21T13:58:17.765427+0000","last_unstale":"2024-09-21T14:02:37.470188+0000","last_undegraded":"2024-09-21T14:02:37.470188+0000","last_fullsized":"2024-09-21T14:02:37.470188+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:58:13.464196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:13.464196+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:16.467667+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":586,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.100548+0000","last_change":"2024-09-21T13:57:36.707211+0000","last_active":"2024-09-21T14:02:40.100548+0000","last_peered":"2024-09-21T14:02:40.100548+0000","last_clean":"2024-09-21T14:02:40.100548+0000","last_became_active":"2024-09-21T13:57:12.704203+0000","last_became_peered":"2024-09-21T13:57:12.704203+0000","last_unstale":"2024-09-21T14:02:40.100548+0000","last_undegraded":"2024-09-21T14:02:40.100548+0000","last_fullsized":"2024-09-21T14:02:40.100548+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:36.707148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:36.707148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"346'14","reported_seq":712,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.454788+0000","last_change":"2024-09-21T14:01:41.186960+0000","last_active":"2024-09-21T14:02:37.454788+0000","last_peered":"2024-09-21T14:02:37.454788+0000","last_clean":"2024-09-21T14:02:37.454788+0000","last_became_active":"2024-09-21T14:01:41.186564+0000","last_became_peered":"2024-09-21T14:01:41.186564+0000","last_unstale":"2024-09-21T14:02:37.454788+0000","last_undegraded":"2024-09-21T14:02:37.454788+0000","last_fullsized":"2024-09-21T14:02:37.454788+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'12","last_scrub_stamp":"2024-09-21T13:59:38.738120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:38.738120+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:44.832105+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"68'1","reported_seq":671,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.469159+0000","last_change":"2024-09-21T13:59:31.392186+0000","last_active":"2024-09-21T14:02:37.469159+0000","last_peered":"2024-09-21T14:02:37.469159+0000","last_clean":"2024-09-21T14:02:37.469159+0000","last_became_active":"2024-09-21T13:52:04.313210+0000","last_became_peered":"2024-09-21T13:52:04.313210+0000","last_unstale":"2024-09-21T14:02:37.469159+0000","last_undegraded":"2024-09-21T14:02:37.469159+0000","last_fullsized":"2024-09-21T14:02:37.469159+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T13:59:31.392105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:31.392105+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"376'14648","reported_seq":14878,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.875416+0000","last_change":"2024-09-21T14:01:41.341139+0000","last_active":"2024-09-21T14:02:40.875416+0000","last_peered":"2024-09-21T14:02:40.875416+0000","last_clean":"2024-09-21T14:02:40.875416+0000","last_became_active":"2024-09-21T14:01:41.199821+0000","last_became_peered":"2024-09-21T14:01:41.199821+0000","last_unstale":"2024-09-21T14:02:40.875416+0000","last_undegraded":"2024-09-21T14:02:40.875416+0000","last_fullsized":"2024-09-21T14:02:40.875416+0000","mapping_epoch":345,"log_start":"312'9699","ondisk_log_start":"312'9699","created":260,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4949,"log_dups_size":0,"ondisk_log_size":4949,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15859712,"num_objects":242,"num_object_clones":0,"num_object_copies":726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14648,"num_write_kb":476480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"290'15","reported_seq":698,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.475410+0000","last_change":"2024-09-21T13:58:50.498597+0000","last_active":"2024-09-21T14:02:37.475410+0000","last_peered":"2024-09-21T14:02:37.475410+0000","last_clean":"2024-09-21T14:02:37.475410+0000","last_became_active":"2024-09-21T13:49:00.053016+0000","last_became_peered":"2024-09-21T13:49:00.053016+0000","last_unstale":"2024-09-21T14:02:37.475410+0000","last_undegraded":"2024-09-21T14:02:37.475410+0000","last_fullsized":"2024-09-21T14:02:37.475410+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:58:50.498369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:50.498369+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":631,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.103893+0000","last_change":"2024-09-21T14:01:41.192717+0000","last_active":"2024-09-21T14:02:40.103893+0000","last_peered":"2024-09-21T14:02:40.103893+0000","last_clean":"2024-09-21T14:02:40.103893+0000","last_became_active":"2024-09-21T14:01:41.192443+0000","last_became_peered":"2024-09-21T14:01:41.192443+0000","last_unstale":"2024-09-21T14:02:40.103893+0000","last_undegraded":"2024-09-21T14:02:40.103893+0000","last_fullsized":"2024-09-21T14:02:40.103893+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:53.894506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:53.894506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":609,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.469318+0000","last_change":"2024-09-21T14:01:27.739999+0000","last_active":"2024-09-21T14:02:37.469318+0000","last_peered":"2024-09-21T14:02:37.469318+0000","last_clean":"2024-09-21T14:02:37.469318+0000","last_became_active":"2024-09-21T13:57:12.680978+0000","last_became_peered":"2024-09-21T13:57:12.680978+0000","last_unstale":"2024-09-21T14:02:37.469318+0000","last_undegraded":"2024-09-21T14:02:37.469318+0000","last_fullsized":"2024-09-21T14:02:37.469318+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:27.739921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:01:27.739921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:50.205235+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":561,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.104190+0000","last_change":"2024-09-21T13:59:32.224001+0000","last_active":"2024-09-21T14:02:40.104190+0000","last_peered":"2024-09-21T14:02:40.104190+0000","last_clean":"2024-09-21T14:02:40.104190+0000","last_became_active":"2024-09-21T13:59:32.221628+0000","last_became_peered":"2024-09-21T13:59:32.221628+0000","last_unstale":"2024-09-21T14:02:40.104190+0000","last_undegraded":"2024-09-21T14:02:40.104190+0000","last_fullsized":"2024-09-21T14:02:40.104190+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:13.698091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:13.698091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"376'15318","reported_seq":23628,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.878229+0000","last_change":"2024-09-21T14:02:09.252872+0000","last_active":"2024-09-21T14:02:40.878229+0000","last_peered":"2024-09-21T14:02:40.878229+0000","last_clean":"2024-09-21T14:02:40.878229+0000","last_became_active":"2024-09-21T14:02:09.218449+0000","last_became_peered":"2024-09-21T14:02:09.218449+0000","last_unstale":"2024-09-21T14:02:40.878229+0000","last_undegraded":"2024-09-21T14:02:40.878229+0000","last_fullsized":"2024-09-21T14:02:40.878229+0000","mapping_epoch":362,"log_start":"312'10355","ondisk_log_start":"312'10355","created":260,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4963,"log_dups_size":0,"ondisk_log_size":4963,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19136512,"num_objects":292,"num_object_clones":0,"num_object_copies":876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15318,"num_write_kb":499520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4082,"num_bytes_recovered":226033664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"290'35","reported_seq":306,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.467707+0000","last_change":"2024-09-21T14:01:02.006619+0000","last_active":"2024-09-21T14:02:37.467707+0000","last_peered":"2024-09-21T14:02:37.467707+0000","last_clean":"2024-09-21T14:02:37.467707+0000","last_became_active":"2024-09-21T13:59:32.204184+0000","last_became_peered":"2024-09-21T13:59:32.204184+0000","last_unstale":"2024-09-21T14:02:37.467707+0000","last_undegraded":"2024-09-21T14:02:37.467707+0000","last_fullsized":"2024-09-21T14:02:37.467707+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'35","last_scrub_stamp":"2024-09-21T14:01:02.006504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:02.006504+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:22.557193+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":653,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.105699+0000","last_change":"2024-09-21T14:01:41.184937+0000","last_active":"2024-09-21T14:02:40.105699+0000","last_peered":"2024-09-21T14:02:40.105699+0000","last_clean":"2024-09-21T14:02:40.105699+0000","last_became_active":"2024-09-21T14:01:41.184567+0000","last_became_peered":"2024-09-21T14:01:41.184567+0000","last_unstale":"2024-09-21T14:02:40.105699+0000","last_undegraded":"2024-09-21T14:02:40.105699+0000","last_fullsized":"2024-09-21T14:02:40.105699+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:21.587493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:21.587493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":629,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.455133+0000","last_change":"2024-09-21T14:01:41.182250+0000","last_active":"2024-09-21T14:02:37.455133+0000","last_peered":"2024-09-21T14:02:37.455133+0000","last_clean":"2024-09-21T14:02:37.455133+0000","last_became_active":"2024-09-21T14:01:41.181911+0000","last_became_peered":"2024-09-21T14:01:41.181911+0000","last_unstale":"2024-09-21T14:02:37.455133+0000","last_undegraded":"2024-09-21T14:02:37.455133+0000","last_fullsized":"2024-09-21T14:02:37.455133+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:55.588554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:55.588554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:41.898392+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":580,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.103028+0000","last_change":"2024-09-21T13:59:32.223123+0000","last_active":"2024-09-21T14:02:40.103028+0000","last_peered":"2024-09-21T14:02:40.103028+0000","last_clean":"2024-09-21T14:02:40.103028+0000","last_became_active":"2024-09-21T13:59:32.219332+0000","last_became_peered":"2024-09-21T13:59:32.219332+0000","last_unstale":"2024-09-21T14:02:40.103028+0000","last_undegraded":"2024-09-21T14:02:40.103028+0000","last_fullsized":"2024-09-21T14:02:40.103028+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:19.537177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:19.537177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"375'14503","reported_seq":14757,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:39.701705+0000","last_change":"2024-09-21T14:02:37.242601+0000","last_active":"2024-09-21T14:02:39.701705+0000","last_peered":"2024-09-21T14:02:39.701705+0000","last_clean":"2024-09-21T14:02:39.701705+0000","last_became_active":"2024-09-21T13:57:53.515313+0000","last_became_peered":"2024-09-21T13:57:53.515313+0000","last_unstale":"2024-09-21T14:02:39.701705+0000","last_undegraded":"2024-09-21T14:02:39.701705+0000","last_fullsized":"2024-09-21T14:02:39.701705+0000","mapping_epoch":260,"log_start":"312'9879","ondisk_log_start":"312'9879","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"374'14313","last_scrub_stamp":"2024-09-21T14:02:37.242486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T14:02:37.242486+0000","objects_scrubbed":555,"log_size":4624,"log_dups_size":0,"ondisk_log_size":4624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:03.083004+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23920640,"num_objects":365,"num_object_clones":0,"num_object_copies":1095,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14503,"num_write_kb":475776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"290'18","reported_seq":715,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.469123+0000","last_change":"2024-09-21T13:59:28.340494+0000","last_active":"2024-09-21T14:02:37.469123+0000","last_peered":"2024-09-21T14:02:37.469123+0000","last_clean":"2024-09-21T14:02:37.469123+0000","last_became_active":"2024-09-21T13:57:12.680627+0000","last_became_peered":"2024-09-21T13:57:12.680627+0000","last_unstale":"2024-09-21T14:02:37.469123+0000","last_undegraded":"2024-09-21T14:02:37.469123+0000","last_fullsized":"2024-09-21T14:02:37.469123+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'18","last_scrub_stamp":"2024-09-21T13:59:28.340415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:28.340415+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:11.383693+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":608,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.105202+0000","last_change":"2024-09-21T13:59:32.201126+0000","last_active":"2024-09-21T14:02:40.105202+0000","last_peered":"2024-09-21T14:02:40.105202+0000","last_clean":"2024-09-21T14:02:40.105202+0000","last_became_active":"2024-09-21T13:59:32.200674+0000","last_became_peered":"2024-09-21T13:59:32.200674+0000","last_unstale":"2024-09-21T14:02:40.105202+0000","last_undegraded":"2024-09-21T14:02:40.105202+0000","last_fullsized":"2024-09-21T14:02:40.105202+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:21.582325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:21.582325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":591,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.454132+0000","last_change":"2024-09-21T14:02:27.791041+0000","last_active":"2024-09-21T14:02:37.454132+0000","last_peered":"2024-09-21T14:02:37.454132+0000","last_clean":"2024-09-21T14:02:37.454132+0000","last_became_active":"2024-09-21T13:52:04.322147+0000","last_became_peered":"2024-09-21T13:52:04.322147+0000","last_unstale":"2024-09-21T14:02:37.454132+0000","last_undegraded":"2024-09-21T14:02:37.454132+0000","last_fullsized":"2024-09-21T14:02:37.454132+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:27.790994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:02:27.790994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:51.879201+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":602,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.101080+0000","last_change":"2024-09-21T14:02:36.371955+0000","last_active":"2024-09-21T14:02:40.101080+0000","last_peered":"2024-09-21T14:02:40.101080+0000","last_clean":"2024-09-21T14:02:40.101080+0000","last_became_active":"2024-09-21T13:59:32.203767+0000","last_became_peered":"2024-09-21T13:59:32.203767+0000","last_unstale":"2024-09-21T14:02:40.101080+0000","last_undegraded":"2024-09-21T14:02:40.101080+0000","last_fullsized":"2024-09-21T14:02:40.101080+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:36.371747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:02:36.371747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:54.168810+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"376'15362","reported_seq":21457,"reported_epoch":376,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-21T14:02:40.869278+0000","last_change":"2024-09-21T14:01:45.658247+0000","last_active":"2024-09-21T14:02:40.869278+0000","last_peered":"2024-09-21T14:02:40.869278+0000","last_clean":"2024-09-21T13:58:23.785212+0000","last_became_active":"2024-09-21T14:01:44.478305+0000","last_became_peered":"2024-09-21T14:01:44.478305+0000","last_unstale":"2024-09-21T14:02:40.869278+0000","last_undegraded":"2024-09-21T14:01:44.440463+0000","last_fullsized":"2024-09-21T14:01:42.170394+0000","mapping_epoch":346,"log_start":"312'10386","ondisk_log_start":"312'10386","created":260,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:57:52.193000+0000","last_clean_scrub_stamp":"2024-09-21T13:57:52.193000+0000","objects_scrubbed":0,"log_size":4976,"log_dups_size":0,"ondisk_log_size":4976,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17563648,"num_objects":268,"num_object_clones":0,"num_object_copies":804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":43,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15362,"num_write_kb":500160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2989,"num_bytes_recovered":194445312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,7],"avail_no_missing":["5","0"],"object_location_counts":[{"shards":"0,5","objects":43},{"shards":"5,7","objects":225}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"290'27","reported_seq":707,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.102294+0000","last_change":"2024-09-21T14:01:41.193143+0000","last_active":"2024-09-21T14:02:40.102294+0000","last_peered":"2024-09-21T14:02:40.102294+0000","last_clean":"2024-09-21T14:02:40.102294+0000","last_became_active":"2024-09-21T14:01:41.192481+0000","last_became_peered":"2024-09-21T14:01:41.192481+0000","last_unstale":"2024-09-21T14:02:40.102294+0000","last_undegraded":"2024-09-21T14:02:40.102294+0000","last_fullsized":"2024-09-21T14:02:40.102294+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'27","last_scrub_stamp":"2024-09-21T13:59:33.058859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:33.058859+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":580,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.105117+0000","last_change":"2024-09-21T13:59:32.203889+0000","last_active":"2024-09-21T14:02:40.105117+0000","last_peered":"2024-09-21T14:02:40.105117+0000","last_clean":"2024-09-21T14:02:40.105117+0000","last_became_active":"2024-09-21T13:59:32.203172+0000","last_became_peered":"2024-09-21T13:59:32.203172+0000","last_unstale":"2024-09-21T14:02:40.105117+0000","last_undegraded":"2024-09-21T14:02:40.105117+0000","last_fullsized":"2024-09-21T14:02:40.105117+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T13:55:03.879706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:55:03.879706+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"300'14","reported_seq":792,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.104058+0000","last_change":"2024-09-21T13:59:32.220898+0000","last_active":"2024-09-21T14:02:40.104058+0000","last_peered":"2024-09-21T14:02:40.104058+0000","last_clean":"2024-09-21T14:02:40.104058+0000","last_became_active":"2024-09-21T13:59:32.206956+0000","last_became_peered":"2024-09-21T13:59:32.206956+0000","last_unstale":"2024-09-21T14:02:40.104058+0000","last_undegraded":"2024-09-21T14:02:40.104058+0000","last_fullsized":"2024-09-21T14:02:40.104058+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'11","last_scrub_stamp":"2024-09-21T13:56:31.616003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:56:31.616003+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":655,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.102175+0000","last_change":"2024-09-21T14:01:41.248408+0000","last_active":"2024-09-21T14:02:40.102175+0000","last_peered":"2024-09-21T14:02:40.102175+0000","last_clean":"2024-09-21T14:02:40.102175+0000","last_became_active":"2024-09-21T14:01:41.246106+0000","last_became_peered":"2024-09-21T14:01:41.246106+0000","last_unstale":"2024-09-21T14:02:40.102175+0000","last_undegraded":"2024-09-21T14:02:40.102175+0000","last_fullsized":"2024-09-21T14:02:40.102175+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:39.258659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:39.258659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":242,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.468996+0000","last_change":"2024-09-21T14:01:50.733055+0000","last_active":"2024-09-21T14:02:37.468996+0000","last_peered":"2024-09-21T14:02:37.468996+0000","last_clean":"2024-09-21T14:02:37.468996+0000","last_became_active":"2024-09-21T14:01:41.197501+0000","last_became_peered":"2024-09-21T14:01:41.197501+0000","last_unstale":"2024-09-21T14:02:37.468996+0000","last_undegraded":"2024-09-21T14:02:37.468996+0000","last_fullsized":"2024-09-21T14:02:37.468996+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:50.733007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:01:50.733007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:08.367766+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"290'12","reported_seq":629,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.104168+0000","last_change":"2024-09-21T13:59:32.247422+0000","last_active":"2024-09-21T14:02:40.104168+0000","last_peered":"2024-09-21T14:02:40.104168+0000","last_clean":"2024-09-21T14:02:40.104168+0000","last_became_active":"2024-09-21T13:59:32.211306+0000","last_became_peered":"2024-09-21T13:59:32.211306+0000","last_unstale":"2024-09-21T14:02:40.104168+0000","last_undegraded":"2024-09-21T14:02:40.104168+0000","last_fullsized":"2024-09-21T14:02:40.104168+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'6","last_scrub_stamp":"2024-09-21T13:55:12.066873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:55:12.066873+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:38.360712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":587,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.102112+0000","last_change":"2024-09-21T14:01:41.199742+0000","last_active":"2024-09-21T14:02:40.102112+0000","last_peered":"2024-09-21T14:02:40.102112+0000","last_clean":"2024-09-21T14:02:40.102112+0000","last_became_active":"2024-09-21T14:01:41.199321+0000","last_became_peered":"2024-09-21T14:01:41.199321+0000","last_unstale":"2024-09-21T14:02:40.102112+0000","last_undegraded":"2024-09-21T14:02:40.102112+0000","last_fullsized":"2024-09-21T14:02:40.102112+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:37.106238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:37.106238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":626,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.105524+0000","last_change":"2024-09-21T13:59:32.222277+0000","last_active":"2024-09-21T14:02:40.105524+0000","last_peered":"2024-09-21T14:02:40.105524+0000","last_clean":"2024-09-21T14:02:40.105524+0000","last_became_active":"2024-09-21T13:59:32.219525+0000","last_became_peered":"2024-09-21T13:59:32.219525+0000","last_unstale":"2024-09-21T14:02:40.105524+0000","last_undegraded":"2024-09-21T14:02:40.105524+0000","last_fullsized":"2024-09-21T14:02:40.105524+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:02.838181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:55:02.838181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"290'15","reported_seq":592,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.100859+0000","last_change":"2024-09-21T13:59:32.238689+0000","last_active":"2024-09-21T14:02:40.100859+0000","last_peered":"2024-09-21T14:02:40.100859+0000","last_clean":"2024-09-21T14:02:40.100859+0000","last_became_active":"2024-09-21T13:59:32.209166+0000","last_became_peered":"2024-09-21T13:59:32.209166+0000","last_unstale":"2024-09-21T14:02:40.100859+0000","last_undegraded":"2024-09-21T14:02:40.100859+0000","last_fullsized":"2024-09-21T14:02:40.100859+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:55:04.990842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:55:04.990842+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:21.278052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"61'1","reported_seq":644,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.107312+0000","last_change":"2024-09-21T13:59:32.222826+0000","last_active":"2024-09-21T14:02:40.107312+0000","last_peered":"2024-09-21T14:02:40.107312+0000","last_clean":"2024-09-21T14:02:40.107312+0000","last_became_active":"2024-09-21T13:59:32.219515+0000","last_became_peered":"2024-09-21T13:59:32.219515+0000","last_unstale":"2024-09-21T14:02:40.107312+0000","last_undegraded":"2024-09-21T14:02:40.107312+0000","last_fullsized":"2024-09-21T14:02:40.107312+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:58:07.711028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:07.711028+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":577,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.102957+0000","last_change":"2024-09-21T13:59:32.211754+0000","last_active":"2024-09-21T14:02:40.102957+0000","last_peered":"2024-09-21T14:02:40.102957+0000","last_clean":"2024-09-21T14:02:40.102957+0000","last_became_active":"2024-09-21T13:59:32.210586+0000","last_became_peered":"2024-09-21T13:59:32.210586+0000","last_unstale":"2024-09-21T14:02:40.102957+0000","last_undegraded":"2024-09-21T14:02:40.102957+0000","last_fullsized":"2024-09-21T14:02:40.102957+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:28.014760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:28.014760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:17.280712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":633,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.454172+0000","last_change":"2024-09-21T14:02:29.731635+0000","last_active":"2024-09-21T14:02:37.454172+0000","last_peered":"2024-09-21T14:02:37.454172+0000","last_clean":"2024-09-21T14:02:37.454172+0000","last_became_active":"2024-09-21T13:59:32.205803+0000","last_became_peered":"2024-09-21T13:59:32.205803+0000","last_unstale":"2024-09-21T14:02:37.454172+0000","last_undegraded":"2024-09-21T14:02:37.454172+0000","last_fullsized":"2024-09-21T14:02:37.454172+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:29.731565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:02:29.731565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:53.473477+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"290'15","reported_seq":632,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.107010+0000","last_change":"2024-09-21T14:01:41.189676+0000","last_active":"2024-09-21T14:02:40.107010+0000","last_peered":"2024-09-21T14:02:40.107010+0000","last_clean":"2024-09-21T14:02:40.107010+0000","last_became_active":"2024-09-21T14:01:41.187904+0000","last_became_peered":"2024-09-21T14:01:41.187904+0000","last_unstale":"2024-09-21T14:02:40.107010+0000","last_undegraded":"2024-09-21T14:02:40.107010+0000","last_fullsized":"2024-09-21T14:02:40.107010+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:57:24.124604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:57:24.124604+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":513,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.103121+0000","last_change":"2024-09-21T13:59:32.212325+0000","last_active":"2024-09-21T14:02:40.103121+0000","last_peered":"2024-09-21T14:02:40.103121+0000","last_clean":"2024-09-21T14:02:40.103121+0000","last_became_active":"2024-09-21T13:59:32.208749+0000","last_became_peered":"2024-09-21T13:59:32.208749+0000","last_unstale":"2024-09-21T14:02:40.103121+0000","last_undegraded":"2024-09-21T14:02:40.103121+0000","last_fullsized":"2024-09-21T14:02:40.103121+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:20.574745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:20.574745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"300'14","reported_seq":725,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.102192+0000","last_change":"2024-09-21T13:59:32.208253+0000","last_active":"2024-09-21T14:02:40.102192+0000","last_peered":"2024-09-21T14:02:40.102192+0000","last_clean":"2024-09-21T14:02:40.102192+0000","last_became_active":"2024-09-21T13:59:32.206926+0000","last_became_peered":"2024-09-21T13:59:32.206926+0000","last_unstale":"2024-09-21T14:02:40.102192+0000","last_undegraded":"2024-09-21T14:02:40.102192+0000","last_fullsized":"2024-09-21T14:02:40.102192+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'10","last_scrub_stamp":"2024-09-21T13:54:20.946782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:20.946782+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:10.314285+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":677,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.469987+0000","last_change":"2024-09-21T14:01:41.200564+0000","last_active":"2024-09-21T14:02:37.469987+0000","last_peered":"2024-09-21T14:02:37.469987+0000","last_clean":"2024-09-21T14:02:37.469987+0000","last_became_active":"2024-09-21T14:01:41.200113+0000","last_became_peered":"2024-09-21T14:01:41.200113+0000","last_unstale":"2024-09-21T14:02:37.469987+0000","last_undegraded":"2024-09-21T14:02:37.469987+0000","last_fullsized":"2024-09-21T14:02:37.469987+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:54.805619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:54.805619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:48.557673+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"290'20","reported_seq":628,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.102017+0000","last_change":"2024-09-21T13:59:32.236530+0000","last_active":"2024-09-21T14:02:40.102017+0000","last_peered":"2024-09-21T14:02:40.102017+0000","last_clean":"2024-09-21T14:02:40.102017+0000","last_became_active":"2024-09-21T13:59:32.208332+0000","last_became_peered":"2024-09-21T13:59:32.208332+0000","last_unstale":"2024-09-21T14:02:40.102017+0000","last_undegraded":"2024-09-21T14:02:40.102017+0000","last_fullsized":"2024-09-21T14:02:40.102017+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:57:37.671391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:57:37.671391+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":620,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.104793+0000","last_change":"2024-09-21T13:59:32.212737+0000","last_active":"2024-09-21T14:02:40.104793+0000","last_peered":"2024-09-21T14:02:40.104793+0000","last_clean":"2024-09-21T14:02:40.104793+0000","last_became_active":"2024-09-21T13:59:32.209804+0000","last_became_peered":"2024-09-21T13:59:32.209804+0000","last_unstale":"2024-09-21T14:02:40.104793+0000","last_undegraded":"2024-09-21T14:02:40.104793+0000","last_fullsized":"2024-09-21T14:02:40.104793+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:25.571256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:25.571256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"346'10","reported_seq":725,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.106920+0000","last_change":"2024-09-21T14:01:41.207030+0000","last_active":"2024-09-21T14:02:40.106920+0000","last_peered":"2024-09-21T14:02:40.106920+0000","last_clean":"2024-09-21T14:02:40.106920+0000","last_became_active":"2024-09-21T14:01:41.206303+0000","last_became_peered":"2024-09-21T14:01:41.206303+0000","last_unstale":"2024-09-21T14:02:40.106920+0000","last_undegraded":"2024-09-21T14:02:40.106920+0000","last_fullsized":"2024-09-21T14:02:40.106920+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:59:32.104224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:32.104224+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":517,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.104868+0000","last_change":"2024-09-21T13:59:32.213015+0000","last_active":"2024-09-21T14:02:40.104868+0000","last_peered":"2024-09-21T14:02:40.104868+0000","last_clean":"2024-09-21T14:02:40.104868+0000","last_became_active":"2024-09-21T13:59:32.211472+0000","last_became_peered":"2024-09-21T13:59:32.211472+0000","last_unstale":"2024-09-21T14:02:40.104868+0000","last_undegraded":"2024-09-21T14:02:40.104868+0000","last_fullsized":"2024-09-21T14:02:40.104868+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:22.647191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:22.647191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"290'19","reported_seq":740,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.469597+0000","last_change":"2024-09-21T14:02:34.670058+0000","last_active":"2024-09-21T14:02:37.469597+0000","last_peered":"2024-09-21T14:02:37.469597+0000","last_clean":"2024-09-21T14:02:37.469597+0000","last_became_active":"2024-09-21T13:58:17.771497+0000","last_became_peered":"2024-09-21T13:58:17.771497+0000","last_unstale":"2024-09-21T14:02:37.469597+0000","last_undegraded":"2024-09-21T14:02:37.469597+0000","last_fullsized":"2024-09-21T14:02:37.469597+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'19","last_scrub_stamp":"2024-09-21T14:02:34.669962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:34.669962+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:44.716931+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":254,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.104948+0000","last_change":"2024-09-21T13:59:32.202097+0000","last_active":"2024-09-21T14:02:40.104948+0000","last_peered":"2024-09-21T14:02:40.104948+0000","last_clean":"2024-09-21T14:02:40.104948+0000","last_became_active":"2024-09-21T13:59:32.201568+0000","last_became_peered":"2024-09-21T13:59:32.201568+0000","last_unstale":"2024-09-21T14:02:40.104948+0000","last_undegraded":"2024-09-21T14:02:40.104948+0000","last_fullsized":"2024-09-21T14:02:40.104948+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:01.228213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:01.228213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":690,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.477272+0000","last_change":"2024-09-21T14:01:41.185588+0000","last_active":"2024-09-21T14:02:37.477272+0000","last_peered":"2024-09-21T14:02:37.477272+0000","last_clean":"2024-09-21T14:02:37.477272+0000","last_became_active":"2024-09-21T14:01:41.185293+0000","last_became_peered":"2024-09-21T14:01:41.185293+0000","last_unstale":"2024-09-21T14:02:37.477272+0000","last_undegraded":"2024-09-21T14:02:37.477272+0000","last_fullsized":"2024-09-21T14:02:37.477272+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:35.409237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:35.409237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:10.948556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":585,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.098743+0000","last_change":"2024-09-21T14:01:41.200439+0000","last_active":"2024-09-21T14:02:40.098743+0000","last_peered":"2024-09-21T14:02:40.098743+0000","last_clean":"2024-09-21T14:02:40.098743+0000","last_became_active":"2024-09-21T14:01:41.199671+0000","last_became_peered":"2024-09-21T14:01:41.199671+0000","last_unstale":"2024-09-21T14:02:40.098743+0000","last_undegraded":"2024-09-21T14:02:40.098743+0000","last_fullsized":"2024-09-21T14:02:40.098743+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:39.643458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:39.643458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"290'15","reported_seq":734,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.477382+0000","last_change":"2024-09-21T13:58:39.977717+0000","last_active":"2024-09-21T14:02:37.477382+0000","last_peered":"2024-09-21T14:02:37.477382+0000","last_clean":"2024-09-21T14:02:37.477382+0000","last_became_active":"2024-09-21T13:58:17.764120+0000","last_became_peered":"2024-09-21T13:58:17.764120+0000","last_unstale":"2024-09-21T14:02:37.477382+0000","last_undegraded":"2024-09-21T14:02:37.477382+0000","last_fullsized":"2024-09-21T14:02:37.477382+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:58:39.977466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:39.977466+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":624,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.105382+0000","last_change":"2024-09-21T13:59:32.223082+0000","last_active":"2024-09-21T14:02:40.105382+0000","last_peered":"2024-09-21T14:02:40.105382+0000","last_clean":"2024-09-21T14:02:40.105382+0000","last_became_active":"2024-09-21T13:59:32.221854+0000","last_became_peered":"2024-09-21T13:59:32.221854+0000","last_unstale":"2024-09-21T14:02:40.105382+0000","last_undegraded":"2024-09-21T14:02:40.105382+0000","last_fullsized":"2024-09-21T14:02:40.105382+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:17.691193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:17.691193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":560,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.100452+0000","last_change":"2024-09-21T13:57:12.703746+0000","last_active":"2024-09-21T14:02:40.100452+0000","last_peered":"2024-09-21T14:02:40.100452+0000","last_clean":"2024-09-21T14:02:40.100452+0000","last_became_active":"2024-09-21T13:57:12.703441+0000","last_became_peered":"2024-09-21T13:57:12.703441+0000","last_unstale":"2024-09-21T14:02:40.100452+0000","last_undegraded":"2024-09-21T14:02:40.100452+0000","last_fullsized":"2024-09-21T14:02:40.100452+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:19.999125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:19.999125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:13.302246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":482,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.102046+0000","last_change":"2024-09-21T14:01:41.188075+0000","last_active":"2024-09-21T14:02:40.102046+0000","last_peered":"2024-09-21T14:02:40.102046+0000","last_clean":"2024-09-21T14:02:40.102046+0000","last_became_active":"2024-09-21T14:01:41.186934+0000","last_became_peered":"2024-09-21T14:01:41.186934+0000","last_unstale":"2024-09-21T14:02:40.102046+0000","last_undegraded":"2024-09-21T14:02:40.102046+0000","last_fullsized":"2024-09-21T14:02:40.102046+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:47.334504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:47.334504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"290'19","reported_seq":704,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.098987+0000","last_change":"2024-09-21T13:59:32.277742+0000","last_active":"2024-09-21T14:02:40.098987+0000","last_peered":"2024-09-21T14:02:40.098987+0000","last_clean":"2024-09-21T14:02:40.098987+0000","last_became_active":"2024-09-21T13:59:32.231567+0000","last_became_peered":"2024-09-21T13:59:32.231567+0000","last_unstale":"2024-09-21T14:02:40.098987+0000","last_undegraded":"2024-09-21T14:02:40.098987+0000","last_fullsized":"2024-09-21T14:02:40.098987+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:54:22.978267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:22.978267+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:12.294653+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":584,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.102210+0000","last_change":"2024-09-21T14:01:41.191090+0000","last_active":"2024-09-21T14:02:40.102210+0000","last_peered":"2024-09-21T14:02:40.102210+0000","last_clean":"2024-09-21T14:02:40.102210+0000","last_became_active":"2024-09-21T14:01:41.190760+0000","last_became_peered":"2024-09-21T14:01:41.190760+0000","last_unstale":"2024-09-21T14:02:40.102210+0000","last_undegraded":"2024-09-21T14:02:40.102210+0000","last_fullsized":"2024-09-21T14:02:40.102210+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:17.305144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:17.305144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":630,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.103238+0000","last_change":"2024-09-21T13:59:32.219508+0000","last_active":"2024-09-21T14:02:40.103238+0000","last_peered":"2024-09-21T14:02:40.103238+0000","last_clean":"2024-09-21T14:02:40.103238+0000","last_became_active":"2024-09-21T13:59:32.214004+0000","last_became_peered":"2024-09-21T13:59:32.214004+0000","last_unstale":"2024-09-21T14:02:40.103238+0000","last_undegraded":"2024-09-21T14:02:40.103238+0000","last_fullsized":"2024-09-21T14:02:40.103238+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:36.525417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:36.525417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":626,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.100937+0000","last_change":"2024-09-21T13:59:28.124373+0000","last_active":"2024-09-21T14:02:40.100937+0000","last_peered":"2024-09-21T14:02:40.100937+0000","last_clean":"2024-09-21T14:02:40.100937+0000","last_became_active":"2024-09-21T13:58:17.778115+0000","last_became_peered":"2024-09-21T13:58:17.778115+0000","last_unstale":"2024-09-21T14:02:40.100937+0000","last_undegraded":"2024-09-21T14:02:40.100937+0000","last_fullsized":"2024-09-21T14:02:40.100937+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:28.124296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:28.124296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"290'8","reported_seq":650,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.098185+0000","last_change":"2024-09-21T13:59:26.034457+0000","last_active":"2024-09-21T14:02:40.098185+0000","last_peered":"2024-09-21T14:02:40.098185+0000","last_clean":"2024-09-21T14:02:40.098185+0000","last_became_active":"2024-09-21T13:58:17.775079+0000","last_became_peered":"2024-09-21T13:58:17.775079+0000","last_unstale":"2024-09-21T14:02:40.098185+0000","last_undegraded":"2024-09-21T14:02:40.098185+0000","last_fullsized":"2024-09-21T14:02:40.098185+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'8","last_scrub_stamp":"2024-09-21T13:59:26.034372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:26.034372+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":552,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.102295+0000","last_change":"2024-09-21T14:01:41.196335+0000","last_active":"2024-09-21T14:02:40.102295+0000","last_peered":"2024-09-21T14:02:40.102295+0000","last_clean":"2024-09-21T14:02:40.102295+0000","last_became_active":"2024-09-21T14:01:41.195989+0000","last_became_peered":"2024-09-21T14:01:41.195989+0000","last_unstale":"2024-09-21T14:02:40.102295+0000","last_undegraded":"2024-09-21T14:02:40.102295+0000","last_fullsized":"2024-09-21T14:02:40.102295+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:16.457445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:16.457445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":242,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.467581+0000","last_change":"2024-09-21T14:00:57.128764+0000","last_active":"2024-09-21T14:02:37.467581+0000","last_peered":"2024-09-21T14:02:37.467581+0000","last_clean":"2024-09-21T14:02:37.467581+0000","last_became_active":"2024-09-21T13:59:32.204380+0000","last_became_peered":"2024-09-21T13:59:32.204380+0000","last_unstale":"2024-09-21T14:02:37.467581+0000","last_undegraded":"2024-09-21T14:02:37.467581+0000","last_fullsized":"2024-09-21T14:02:37.467581+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:00:57.128713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:00:57.128713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:24.563060+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":563,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.103044+0000","last_change":"2024-09-21T13:59:32.212615+0000","last_active":"2024-09-21T14:02:40.103044+0000","last_peered":"2024-09-21T14:02:40.103044+0000","last_clean":"2024-09-21T14:02:40.103044+0000","last_became_active":"2024-09-21T13:59:32.210134+0000","last_became_peered":"2024-09-21T13:59:32.210134+0000","last_unstale":"2024-09-21T14:02:40.103044+0000","last_undegraded":"2024-09-21T14:02:40.103044+0000","last_fullsized":"2024-09-21T14:02:40.103044+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:08.852789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:55:08.852789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":675,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.466824+0000","last_change":"2024-09-21T14:01:41.187792+0000","last_active":"2024-09-21T14:02:37.466824+0000","last_peered":"2024-09-21T14:02:37.466824+0000","last_clean":"2024-09-21T14:02:37.466824+0000","last_became_active":"2024-09-21T14:01:41.187438+0000","last_became_peered":"2024-09-21T14:01:41.187438+0000","last_unstale":"2024-09-21T14:02:37.466824+0000","last_undegraded":"2024-09-21T14:02:37.466824+0000","last_fullsized":"2024-09-21T14:02:37.466824+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:17.706325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:17.706325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":606,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.098541+0000","last_change":"2024-09-21T13:58:13.776632+0000","last_active":"2024-09-21T14:02:40.098541+0000","last_peered":"2024-09-21T14:02:40.098541+0000","last_clean":"2024-09-21T14:02:40.098541+0000","last_became_active":"2024-09-21T13:57:12.704042+0000","last_became_peered":"2024-09-21T13:57:12.704042+0000","last_unstale":"2024-09-21T14:02:40.098541+0000","last_undegraded":"2024-09-21T14:02:40.098541+0000","last_fullsized":"2024-09-21T14:02:40.098541+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:13.776572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:13.776572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"290'19","reported_seq":275,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T14:02:37.468680+0000","last_change":"2024-09-21T14:01:54.680822+0000","last_active":"2024-09-21T14:02:37.468680+0000","last_peered":"2024-09-21T14:02:37.468680+0000","last_clean":"2024-09-21T14:02:37.468680+0000","last_became_active":"2024-09-21T14:01:41.194606+0000","last_became_peered":"2024-09-21T14:01:41.194606+0000","last_unstale":"2024-09-21T14:02:37.468680+0000","last_undegraded":"2024-09-21T14:02:37.468680+0000","last_fullsized":"2024-09-21T14:02:37.468680+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'19","last_scrub_stamp":"2024-09-21T14:01:54.680735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:54.680735+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:21.312249+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":532,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T14:02:40.104776+0000","last_change":"2024-09-21T13:59:32.203045+0000","last_active":"2024-09-21T14:02:40.104776+0000","last_peered":"2024-09-21T14:02:40.104776+0000","last_clean":"2024-09-21T14:02:40.104776+0000","last_became_active":"2024-09-21T13:59:32.202225+0000","last_became_peered":"2024-09-21T13:59:32.202225+0000","last_unstale":"2024-09-21T14:02:40.104776+0000","last_undegraded":"2024-09-21T14:02:40.104776+0000","last_fullsized":"2024-09-21T14:02:40.104776+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:40.650832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:40.650832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":297861144,"num_objects":4546,"num_object_clones":0,"num_object_copies":13638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":43,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4546,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":238266,"num_write_kb":7769921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42553,"num_bytes_recovered":2648637464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1077489664,"data_stored":1077477448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75937,"ondisk_log_size":75937,"up":48,"acting":47,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1111,"num_read_kb":868,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":1914,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":687,"ondisk_log_size":687,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":195,"num_read_kb":190,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":429358,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":159,"num_write":83,"num_write_kb":1184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1084718,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":298,"seq":1279900254247,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":467880,"kb_used_data":163988,"kb_used_omap":4,"kb_used_meta":303867,"kb_avail":93277272,"statfs":{"total":95995035648,"available":95515926528,"internally_reserved":0,"allocated":167923712,"data_stored":167077592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":311160488},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463086,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":426312,"kb_used_data":98032,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":93318840,"statfs":{"total":95995035648,"available":95558492160,"internally_reserved":0,"allocated":100384768,"data_stored":99566294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":336132550},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":345,"seq":1481763717133,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":288148,"kb_used_data":175176,"kb_used_omap":4,"kb_used_meta":112955,"kb_avail":93457004,"statfs":{"total":95995035648,"available":95699972096,"internally_reserved":0,"allocated":179380224,"data_stored":178546767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":115666595},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920955,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":533860,"kb_used_data":155600,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":93211292,"statfs":{"total":95995035648,"available":95448363008,"internally_reserved":0,"allocated":159334400,"data_stored":158483776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":387316165},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":240,"seq":1030792151107,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":419896,"kb_used_data":161316,"kb_used_omap":4,"kb_used_meta":258555,"kb_avail":93325256,"statfs":{"total":95995035648,"available":95565062144,"internally_reserved":0,"allocated":165187584,"data_stored":164325963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4443,"internal_metadata":264760997},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":130,"seq":558345748608,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":633736,"kb_used_data":96180,"kb_used_omap":4,"kb_used_meta":537531,"kb_avail":93111416,"statfs":{"total":95995035648,"available":95346089984,"internally_reserved":0,"allocated":98488320,"data_stored":97667012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4441,"internal_metadata":550432423},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575053,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":793672,"kb_used_data":225204,"kb_used_omap":1,"kb_used_meta":568446,"kb_avail":92951480,"statfs":{"total":95995035648,"available":95182315520,"internally_reserved":0,"allocated":230608896,"data_stored":229722176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":582089157},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738579,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":604120,"kb_used_data":113412,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":93141032,"statfs":{"total":95995035648,"available":95376416768,"internally_reserved":0,"allocated":116133888,"data_stored":115295292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":502462917},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":98172928,"data_stored":98172928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":213127168,"data_stored":213123096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":93978624,"data_stored":93978624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":134938624,"data_stored":134938624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":140771328,"data_stored":140771328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":159387648,"data_stored":159383576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":84017152,"data_stored":84017152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":153096192,"data_stored":153092120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T14:02:44.807 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph status --format=json 2024-09-21T14:02:44.865 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:02:44.908 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:44.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:44.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:02:44.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:44 smithi187 ceph-mon[33032]: from='client.25883 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:02:44.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:44 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:02:45.010 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:02:45.033 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:45.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:45.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:02:45.134 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:45.159 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:45.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:45.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:45.261 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:02:45.283 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:45.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:45.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:02:45.316 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-09-21T14:02:45.316 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-09-21T14:02:45.384 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:45.406 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:45.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:45.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:45.508 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:45.531 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:45.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:45.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:45.632 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:45.654 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:45.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:45.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:45.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:45 smithi049 ceph-mon[30080]: pgmap v703: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 261 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s; 43/13095 objects degraded (0.328%); 3.7 MiB/s, 59 objects/s recovering 2024-09-21T14:02:45.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:45 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 43/13095 objects degraded (0.328%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T14:02:45.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:45 smithi049 ceph-mon[40489]: pgmap v703: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 261 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s; 43/13095 objects degraded (0.328%); 3.7 MiB/s, 59 objects/s recovering 2024-09-21T14:02:45.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:45 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 43/13095 objects degraded (0.328%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T14:02:45.756 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:02:45.778 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:45.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:45.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:02:45.880 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:02:45.906 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:45.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:45.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:02:45.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:45 smithi187 ceph-mon[33032]: pgmap v703: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 261 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s; 43/13095 objects degraded (0.328%); 3.7 MiB/s, 59 objects/s recovering 2024-09-21T14:02:45.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:45 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 43/13095 objects degraded (0.328%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T14:02:46.001 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T14:02:46.004 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:02:46.007 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:02:46.009 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:02:46.033 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:46.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:46.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:02:46.135 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:46.158 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:46.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:46.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:46.260 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:46.284 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:46.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:46.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:46.386 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:46.409 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:46.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:46.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:46.510 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:02:46.537 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:46.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:46.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:02:46.639 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:02:46.665 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:46.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:46.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:02:46.994 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T14:02:46.995 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"40a3d2e8-781f-11ef-baf6-efdc52797490","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1 pg undersized","count":1},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1082,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":377,"num_osds":8,"num_up_osds":8,"osd_up_since":1726927300,"num_in_osds":8,"osd_in_since":1726927096,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":157},{"state_name":"active+undersized+remapped","count":1}],"num_pgs":158,"num_pools":7,"num_objects":2292,"data_bytes":137536833,"bytes_used":3523055616,"bytes_avail":764437229568,"bytes_total":767960285184,"recovering_objects_per_sec":182,"recovering_bytes_per_sec":3999508,"recovering_keys_per_sec":0,"num_objects_recovered":1558,"num_bytes_recovered":34144256,"num_keys_recovered":0,"read_bytes_sec":599,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":934},"fsmap":{"epoch":1,"btime":"2024-09-21T13:42:57:838094+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.49:8443/","prometheus":"http://172.21.15.49:9283/"}},"servicemap":{"epoch":18,"modified":"2024-09-21T14:02:39.989968+0000","services":{"rgw":{"daemons":{"summary":"","24365":{"start_epoch":12,"start_stamp":"2024-09-21T13:49:04.899719+0000","gid":24365,"addr":"172.21.15.49:0/3404764347","metadata":{"arch":"x86_64","ceph_release":"squid","ceph_version":"ceph version 19.3.0-5104-g44178494 (441784946648ff53dd310293f9dd752afeeea72a) squid (dev)","ceph_version_short":"19.3.0-5104-g44178494","container_hostname":"smithi049","container_image":"quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5","cpu":"Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz","distro":"centos","distro_description":"CentOS Stream 9","distro_version":"9","frontend_config#0":"beast port=80","frontend_type#0":"beast","hostname":"smithi049","id":"foo.a","kernel_description":"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024","kernel_version":"5.14.0-508.el9.x86_64","mem_swap_kb":"0","mem_total_kb":"32487164","num_handles":"1","os":"Linux","pid":"7","realm_id":"","realm_name":"","zone_id":"d447add1-8b31-4043-a929-4cb437e5f078","zone_name":"default","zonegroup_id":"06d4f733-b51c-4f26-9844-acd95f223f5b","zonegroup_name":"default"},"task_status":{}}}}}},"progress_events":{}} 2024-09-21T14:02:47.063 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:47.086 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:47.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:47.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:47.174 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T14:02:47.188 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:47.241 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:47.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:47.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:47.261 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:02:47.343 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:02:47.369 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:47.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:47.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:02:47.470 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:47.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:47.493 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:47.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:47.518 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-09-21T14:02:47.595 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:02:47.619 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:47.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:47.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:02:47.721 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:02:47.744 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:47.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:47.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:02:47.764 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:47 smithi187 ceph-mon[33032]: pgmap v704: 158 pgs: 1 active+undersized+remapped, 157 active+clean; 131 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 599 B/s rd, 0 B/s wr, 935 op/s; 3.8 MiB/s, 182 objects/s recovering 2024-09-21T14:02:47.764 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:47 smithi187 ceph-mon[33032]: osdmap e377: 8 total, 8 up, 8 in 2024-09-21T14:02:47.764 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:47 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2277594342' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T14:02:47.764 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:47 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1418204661' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:02:47.764 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:47 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:02:47.847 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:02:47.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:47.871 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:47.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:02:47.930 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:47 smithi049 ceph-mon[30080]: pgmap v704: 158 pgs: 1 active+undersized+remapped, 157 active+clean; 131 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 599 B/s rd, 0 B/s wr, 935 op/s; 3.8 MiB/s, 182 objects/s recovering 2024-09-21T14:02:47.930 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:47 smithi049 ceph-mon[30080]: osdmap e377: 8 total, 8 up, 8 in 2024-09-21T14:02:47.930 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:47 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2277594342' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T14:02:47.930 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:47 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1418204661' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:02:47.931 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:47 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:02:47.931 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:47 smithi049 ceph-mon[40489]: pgmap v704: 158 pgs: 1 active+undersized+remapped, 157 active+clean; 131 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 599 B/s rd, 0 B/s wr, 935 op/s; 3.8 MiB/s, 182 objects/s recovering 2024-09-21T14:02:47.931 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:47 smithi049 ceph-mon[40489]: osdmap e377: 8 total, 8 up, 8 in 2024-09-21T14:02:47.931 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:47 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2277594342' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T14:02:47.931 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:47 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1418204661' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:02:47.931 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:47 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:02:47.972 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:47.997 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:47.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:47.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:48.099 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:48.122 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:48.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:48.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:48.224 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:02:48.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:48.248 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:48.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:02:48.350 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:02:48.374 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:48.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:48.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:02:48.475 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:02:48.496 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:48.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:48.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:02:48.598 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:02:48.619 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:48.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:48.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:02:48.680 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:48 smithi049 ceph-mon[30080]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:02:48.680 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:48 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T14:02:48.680 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:48 smithi049 ceph-mon[30080]: osdmap e378: 8 total, 8 up, 8 in 2024-09-21T14:02:48.681 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:48 smithi049 ceph-mon[40489]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:02:48.681 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:48 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T14:02:48.681 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:48 smithi049 ceph-mon[40489]: osdmap e378: 8 total, 8 up, 8 in 2024-09-21T14:02:48.712 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-21T14:02:48.712 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T14:02:48.717 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:02:48.721 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:02:48.741 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:48.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:48.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:02:48.843 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:02:48.863 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:48.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:48.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:02:48.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:48 smithi187 ceph-mon[33032]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:02:48.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:48 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T14:02:48.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:48 smithi187 ceph-mon[33032]: osdmap e378: 8 total, 8 up, 8 in 2024-09-21T14:02:48.965 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:48.991 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:48.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:48.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:49.092 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:02:49.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:49.116 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:49.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:02:49.217 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:02:49.238 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:49.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:49.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:02:49.339 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:49.363 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:49.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:49.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:49.464 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:49.489 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:49.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:49.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:49.518 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'unique_pool_2' removed 2024-09-21T14:02:49.591 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:49.614 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:49.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:49.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:49.716 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:02:49.730 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:49 smithi049 ceph-mon[40489]: pgmap v707: 158 pgs: 1 active+undersized+remapped, 157 active+clean; 3.5 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.14k op/s; 4.1 MiB/s, 197 objects/s recovering 2024-09-21T14:02:49.730 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:49 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:02:49.730 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:49 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:02:49.730 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:49 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1928343196' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T14:02:49.730 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:49 smithi049 ceph-mon[30080]: pgmap v707: 158 pgs: 1 active+undersized+remapped, 157 active+clean; 3.5 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.14k op/s; 4.1 MiB/s, 197 objects/s recovering 2024-09-21T14:02:49.730 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:49 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:02:49.730 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:49 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:02:49.730 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:49 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1928343196' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T14:02:49.737 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:49.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:49.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:02:49.743 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T14:02:49.743 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-09-21T14:02:49.743 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-09-21T14:02:49.743 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-09-21T14:02:49.743 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T14:02:49.774 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:49 smithi187 ceph-mon[33032]: pgmap v707: 158 pgs: 1 active+undersized+remapped, 157 active+clean; 3.5 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.14k op/s; 4.1 MiB/s, 197 objects/s recovering 2024-09-21T14:02:49.775 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:49 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:02:49.775 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:49 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:02:49.775 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:49 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1928343196' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T14:02:49.839 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:02:49.861 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:49.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:49.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:02:49.963 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:02:49.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:49.996 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:49.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:02:50.097 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:02:50.118 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:50.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:50.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:02:50.194 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:02:49 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:02:50.220 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:02:50.241 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:50.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:50.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:02:50.342 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:02:50.365 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:50.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:50.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:02:50.391 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T14:02:50.465 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-21T14:02:50.465 INFO:tasks.radosbench:Beginning radosbench... 2024-09-21T14:02:50.465 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-09-21T14:02:50.466 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd pool create unique_pool_3 16 2024-09-21T14:02:50.466 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:50.489 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:50.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:50.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:50.591 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:50.614 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:50.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:50.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:50.707 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:02:50.707 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:02:50.716 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:50.748 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:50.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:50.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:50.754 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:02:50 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:02:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:02:50.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:50 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1928343196' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T14:02:50.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:50 smithi049 ceph-mon[40489]: osdmap e379: 8 total, 8 up, 8 in 2024-09-21T14:02:50.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:50 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:02:50.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:50 smithi049 ceph-mon[40489]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-09-21T14:02:50.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:50 smithi049 ceph-mon[40489]: osdmap e380: 8 total, 8 up, 8 in 2024-09-21T14:02:50.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:50 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1928343196' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T14:02:50.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:50 smithi049 ceph-mon[30080]: osdmap e379: 8 total, 8 up, 8 in 2024-09-21T14:02:50.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:50 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:02:50.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:50 smithi049 ceph-mon[30080]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-09-21T14:02:50.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:50 smithi049 ceph-mon[30080]: osdmap e380: 8 total, 8 up, 8 in 2024-09-21T14:02:50.850 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:02:50.873 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:50.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:50.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:02:50.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:50 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1928343196' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T14:02:50.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:50 smithi187 ceph-mon[33032]: osdmap e379: 8 total, 8 up, 8 in 2024-09-21T14:02:50.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:50 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:02:50.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:50 smithi187 ceph-mon[33032]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-09-21T14:02:50.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:50 smithi187 ceph-mon[33032]: osdmap e380: 8 total, 8 up, 8 in 2024-09-21T14:02:50.975 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:02:50.997 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:50.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:50.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:02:51.099 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:02:51.120 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:51.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:51.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:02:51.222 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:51.244 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:51.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:51.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:51.346 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:02:51.369 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:51.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:51.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:02:51.471 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:02:51.493 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:51.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:51.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:02:51.594 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:51.617 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:51.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:51.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:51.719 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:02:51.741 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:51.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:51.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:02:51.837 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:51 smithi049 ceph-mon[30080]: pgmap v709: 142 pgs: 142 active+clean; 425 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-21T14:02:51.837 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:51 smithi049 ceph-mon[30080]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-21T14:02:51.837 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:51 smithi049 ceph-mon[40489]: pgmap v709: 142 pgs: 142 active+clean; 425 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-21T14:02:51.837 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:51 smithi049 ceph-mon[40489]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-21T14:02:51.842 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:51.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:51.867 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:51.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:51.869 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T14:02:51.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:51 smithi187 ceph-mon[33032]: pgmap v709: 142 pgs: 142 active+clean; 425 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-21T14:02:51.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:51 smithi187 ceph-mon[33032]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-21T14:02:51.968 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:02:51.990 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:51.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:51.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:02:52.092 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:02:52.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:52.116 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:52.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:02:52.217 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:02:52.239 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:52.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:52.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:02:52.341 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:52.359 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T14:02:52.365 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-09-21T14:02:52.372 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:52.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:52.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:52.471 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:02:52.471 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:02:52.474 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:52.490 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:02:52.491 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:02:52.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:52.498 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:52.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:52.506 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:02:52.506 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:02:52.522 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:02:52.522 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:02:52.539 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:02:52.540 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:02:52.556 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:02:52.557 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:02:52.573 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:02:52.573 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:02:52.590 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:02:52.590 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:02:52.600 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:52.623 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:52.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:52.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:52.725 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:02:52.748 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:52.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:52.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:02:52.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:52 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3535783029' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:02:52.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:52 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3535783029' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:02:52.850 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:02:52.871 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:52.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:52.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:02:52.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:52 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3535783029' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:02:52.973 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:02:52.998 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:52.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:52.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:02:53.100 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:02:53.124 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:53.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:53.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:02:53.226 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:53.233 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":711,"stamp":"2024-09-21T14:02:51.993352+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":435497,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":1398,"num_read_kb":1227,"num_write":805,"num_write_kb":1229,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":169,"num_bytes_recovered":1086632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1671,"ondisk_log_size":1671,"up":426,"acting":426,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":466,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":3036876,"kb_used_data":58136,"kb_used_omap":23,"kb_used_meta":2978536,"kb_avail":746924340,"statfs":{"total":767960285184,"available":764850524160,"internally_reserved":0,"allocated":59531264,"data_stored":52668949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24148,"internal_metadata":3050021292},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":49,"apply_latency_ms":49,"commit_latency_ns":49000000,"apply_latency_ns":49000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-3276800,"num_objects":-50,"num_object_clones":0,"num_object_copies":-150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-50,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":-242762,"num_write_kb":-7769921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-43633,"num_bytes_recovered":-2651455512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.483850"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":576,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.272197+0000","last_change":"2024-09-21T14:01:41.189741+0000","last_active":"2024-09-21T14:02:50.272197+0000","last_peered":"2024-09-21T14:02:50.272197+0000","last_clean":"2024-09-21T14:02:50.272197+0000","last_became_active":"2024-09-21T14:01:41.189398+0000","last_became_peered":"2024-09-21T14:01:41.189398+0000","last_unstale":"2024-09-21T14:02:50.272197+0000","last_undegraded":"2024-09-21T14:02:50.272197+0000","last_fullsized":"2024-09-21T14:02:50.272197+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:55.873301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:55.873301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"290'15","reported_seq":747,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T14:02:47.516568+0000","last_change":"2024-09-21T14:01:34.785322+0000","last_active":"2024-09-21T14:02:47.516568+0000","last_peered":"2024-09-21T14:02:47.516568+0000","last_clean":"2024-09-21T14:02:47.516568+0000","last_became_active":"2024-09-21T13:58:17.771220+0000","last_became_peered":"2024-09-21T13:58:17.771220+0000","last_unstale":"2024-09-21T14:02:47.516568+0000","last_undegraded":"2024-09-21T14:02:47.516568+0000","last_fullsized":"2024-09-21T14:02:47.516568+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'15","last_scrub_stamp":"2024-09-21T14:01:34.785267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:34.785267+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":690,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T14:02:49.520436+0000","last_change":"2024-09-21T14:02:29.205437+0000","last_active":"2024-09-21T14:02:49.520436+0000","last_peered":"2024-09-21T14:02:49.520436+0000","last_clean":"2024-09-21T14:02:49.520436+0000","last_became_active":"2024-09-21T14:01:41.202168+0000","last_became_peered":"2024-09-21T14:01:41.202168+0000","last_unstale":"2024-09-21T14:02:49.520436+0000","last_undegraded":"2024-09-21T14:02:49.520436+0000","last_fullsized":"2024-09-21T14:02:49.520436+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:29.205343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:02:29.205343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:29.369063+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":664,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T14:02:47.516652+0000","last_change":"2024-09-21T14:01:41.203762+0000","last_active":"2024-09-21T14:02:47.516652+0000","last_peered":"2024-09-21T14:02:47.516652+0000","last_clean":"2024-09-21T14:02:47.516652+0000","last_became_active":"2024-09-21T14:01:41.203275+0000","last_became_peered":"2024-09-21T14:01:41.203275+0000","last_unstale":"2024-09-21T14:02:47.516652+0000","last_undegraded":"2024-09-21T14:02:47.516652+0000","last_fullsized":"2024-09-21T14:02:47.516652+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:00.320510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:00.320510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:46.689692+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":638,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.095339+0000","last_change":"2024-09-21T13:59:32.208701+0000","last_active":"2024-09-21T14:02:50.095339+0000","last_peered":"2024-09-21T14:02:50.095339+0000","last_clean":"2024-09-21T14:02:50.095339+0000","last_became_active":"2024-09-21T13:59:32.203340+0000","last_became_peered":"2024-09-21T13:59:32.203340+0000","last_unstale":"2024-09-21T14:02:50.095339+0000","last_undegraded":"2024-09-21T14:02:50.095339+0000","last_fullsized":"2024-09-21T14:02:50.095339+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:34.696780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:34.696780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:44.274358+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":650,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.095528+0000","last_change":"2024-09-21T13:58:17.778322+0000","last_active":"2024-09-21T14:02:50.095528+0000","last_peered":"2024-09-21T14:02:50.095528+0000","last_clean":"2024-09-21T14:02:50.095528+0000","last_became_active":"2024-09-21T13:58:17.777893+0000","last_became_peered":"2024-09-21T13:58:17.777893+0000","last_unstale":"2024-09-21T14:02:50.095528+0000","last_undegraded":"2024-09-21T14:02:50.095528+0000","last_fullsized":"2024-09-21T14:02:50.095528+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:07.001135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:55:07.001135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:22.269018+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"290'27","reported_seq":624,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.095534+0000","last_change":"2024-09-21T13:58:11.718421+0000","last_active":"2024-09-21T14:02:50.095534+0000","last_peered":"2024-09-21T14:02:50.095534+0000","last_clean":"2024-09-21T14:02:50.095534+0000","last_became_active":"2024-09-21T13:57:12.703469+0000","last_became_peered":"2024-09-21T13:57:12.703469+0000","last_unstale":"2024-09-21T14:02:50.095534+0000","last_undegraded":"2024-09-21T14:02:50.095534+0000","last_fullsized":"2024-09-21T14:02:50.095534+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:58:11.718334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:11.718334+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":687,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T14:02:47.516449+0000","last_change":"2024-09-21T14:01:32.717551+0000","last_active":"2024-09-21T14:02:47.516449+0000","last_peered":"2024-09-21T14:02:47.516449+0000","last_clean":"2024-09-21T14:02:47.516449+0000","last_became_active":"2024-09-21T13:58:17.773464+0000","last_became_peered":"2024-09-21T13:58:17.773464+0000","last_unstale":"2024-09-21T14:02:47.516449+0000","last_undegraded":"2024-09-21T14:02:47.516449+0000","last_fullsized":"2024-09-21T14:02:47.516449+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:32.717480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:01:32.717480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:02.205485+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":643,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.272212+0000","last_change":"2024-09-21T14:01:41.194807+0000","last_active":"2024-09-21T14:02:50.272212+0000","last_peered":"2024-09-21T14:02:50.272212+0000","last_clean":"2024-09-21T14:02:50.272212+0000","last_became_active":"2024-09-21T14:01:41.194517+0000","last_became_peered":"2024-09-21T14:01:41.194517+0000","last_unstale":"2024-09-21T14:02:50.272212+0000","last_undegraded":"2024-09-21T14:02:50.272212+0000","last_fullsized":"2024-09-21T14:02:50.272212+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:31.127926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:31.127926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"290'15","reported_seq":713,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T14:02:47.517146+0000","last_change":"2024-09-21T14:00:02.475761+0000","last_active":"2024-09-21T14:02:47.517146+0000","last_peered":"2024-09-21T14:02:47.517146+0000","last_clean":"2024-09-21T14:02:47.517146+0000","last_became_active":"2024-09-21T13:57:12.681408+0000","last_became_peered":"2024-09-21T13:57:12.681408+0000","last_unstale":"2024-09-21T14:02:47.517146+0000","last_undegraded":"2024-09-21T14:02:47.517146+0000","last_fullsized":"2024-09-21T14:02:47.517146+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'15","last_scrub_stamp":"2024-09-21T14:00:02.475661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:00:02.475661+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:43.713097+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":584,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.271701+0000","last_change":"2024-09-21T14:01:41.188585+0000","last_active":"2024-09-21T14:02:50.271701+0000","last_peered":"2024-09-21T14:02:50.271701+0000","last_clean":"2024-09-21T14:02:50.271701+0000","last_became_active":"2024-09-21T14:01:41.188248+0000","last_became_peered":"2024-09-21T14:01:41.188248+0000","last_unstale":"2024-09-21T14:02:50.271701+0000","last_undegraded":"2024-09-21T14:02:50.271701+0000","last_fullsized":"2024-09-21T14:02:50.271701+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:21.284048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:56:21.284048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":620,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T14:02:49.521968+0000","last_change":"2024-09-21T13:59:32.202128+0000","last_active":"2024-09-21T14:02:49.521968+0000","last_peered":"2024-09-21T14:02:49.521968+0000","last_clean":"2024-09-21T14:02:49.521968+0000","last_became_active":"2024-09-21T13:59:32.201158+0000","last_became_peered":"2024-09-21T13:59:32.201158+0000","last_unstale":"2024-09-21T14:02:49.521968+0000","last_undegraded":"2024-09-21T14:02:49.521968+0000","last_fullsized":"2024-09-21T14:02:49.521968+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:25.792985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:54:25.792985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:49.267653+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":620,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.271731+0000","last_change":"2024-09-21T14:01:41.189797+0000","last_active":"2024-09-21T14:02:50.271731+0000","last_peered":"2024-09-21T14:02:50.271731+0000","last_clean":"2024-09-21T14:02:50.271731+0000","last_became_active":"2024-09-21T14:01:41.189373+0000","last_became_peered":"2024-09-21T14:01:41.189373+0000","last_unstale":"2024-09-21T14:02:50.271731+0000","last_undegraded":"2024-09-21T14:02:50.271731+0000","last_fullsized":"2024-09-21T14:02:50.271731+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:54.328275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:54.328275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"290'7","reported_seq":738,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T14:02:47.516880+0000","last_change":"2024-09-21T14:01:57.563250+0000","last_active":"2024-09-21T14:02:47.516880+0000","last_peered":"2024-09-21T14:02:47.516880+0000","last_clean":"2024-09-21T14:02:47.516880+0000","last_became_active":"2024-09-21T13:59:09.063053+0000","last_became_peered":"2024-09-21T13:59:09.063053+0000","last_unstale":"2024-09-21T14:02:47.516880+0000","last_undegraded":"2024-09-21T14:02:47.516880+0000","last_fullsized":"2024-09-21T14:02:47.516880+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'7","last_scrub_stamp":"2024-09-21T14:01:57.563184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:57.563184+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:19.737603+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":611,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T14:02:49.947115+0000","last_change":"2024-09-21T13:59:32.204565+0000","last_active":"2024-09-21T14:02:49.947115+0000","last_peered":"2024-09-21T14:02:49.947115+0000","last_clean":"2024-09-21T14:02:49.947115+0000","last_became_active":"2024-09-21T13:59:32.204174+0000","last_became_peered":"2024-09-21T13:59:32.204174+0000","last_unstale":"2024-09-21T14:02:49.947115+0000","last_undegraded":"2024-09-21T14:02:49.947115+0000","last_fullsized":"2024-09-21T14:02:49.947115+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:06.722293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:06.722293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":654,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.271461+0000","last_change":"2024-09-21T14:01:41.199085+0000","last_active":"2024-09-21T14:02:50.271461+0000","last_peered":"2024-09-21T14:02:50.271461+0000","last_clean":"2024-09-21T14:02:50.271461+0000","last_became_active":"2024-09-21T14:01:41.198672+0000","last_became_peered":"2024-09-21T14:01:41.198672+0000","last_unstale":"2024-09-21T14:02:50.271461+0000","last_undegraded":"2024-09-21T14:02:50.271461+0000","last_fullsized":"2024-09-21T14:02:50.271461+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:44.376215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:44.376215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":706,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T14:02:47.517633+0000","last_change":"2024-09-21T14:01:53.460257+0000","last_active":"2024-09-21T14:02:47.517633+0000","last_peered":"2024-09-21T14:02:47.517633+0000","last_clean":"2024-09-21T14:02:47.517633+0000","last_became_active":"2024-09-21T14:01:41.200360+0000","last_became_peered":"2024-09-21T14:01:41.200360+0000","last_unstale":"2024-09-21T14:02:47.517633+0000","last_undegraded":"2024-09-21T14:02:47.517633+0000","last_fullsized":"2024-09-21T14:02:47.517633+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:53.460180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:01:53.460180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:15.802657+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":624,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T14:02:47.517114+0000","last_change":"2024-09-21T14:02:36.679396+0000","last_active":"2024-09-21T14:02:47.517114+0000","last_peered":"2024-09-21T14:02:47.517114+0000","last_clean":"2024-09-21T14:02:47.517114+0000","last_became_active":"2024-09-21T13:59:32.204749+0000","last_became_peered":"2024-09-21T13:59:32.204749+0000","last_unstale":"2024-09-21T14:02:47.517114+0000","last_undegraded":"2024-09-21T14:02:47.517114+0000","last_fullsized":"2024-09-21T14:02:47.517114+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:36.679321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:02:36.679321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:43.067225+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"290'27","reported_seq":681,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T14:02:47.518566+0000","last_change":"2024-09-21T14:01:27.893648+0000","last_active":"2024-09-21T14:02:47.518566+0000","last_peered":"2024-09-21T14:02:47.518566+0000","last_clean":"2024-09-21T14:02:47.518566+0000","last_became_active":"2024-09-21T13:57:12.680783+0000","last_became_peered":"2024-09-21T13:57:12.680783+0000","last_unstale":"2024-09-21T14:02:47.518566+0000","last_undegraded":"2024-09-21T14:02:47.518566+0000","last_fullsized":"2024-09-21T14:02:47.518566+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'27","last_scrub_stamp":"2024-09-21T14:01:27.893587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:27.893587+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:53.939497+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":676,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T14:02:49.523410+0000","last_change":"2024-09-21T14:01:54.024745+0000","last_active":"2024-09-21T14:02:49.523410+0000","last_peered":"2024-09-21T14:02:49.523410+0000","last_clean":"2024-09-21T14:02:49.523410+0000","last_became_active":"2024-09-21T13:49:02.066796+0000","last_became_peered":"2024-09-21T13:49:02.066796+0000","last_unstale":"2024-09-21T14:02:49.523410+0000","last_undegraded":"2024-09-21T14:02:49.523410+0000","last_fullsized":"2024-09-21T14:02:49.523410+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:54.024702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:01:54.024702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:06.646444+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":630,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.096450+0000","last_change":"2024-09-21T14:01:41.191486+0000","last_active":"2024-09-21T14:02:50.096450+0000","last_peered":"2024-09-21T14:02:50.096450+0000","last_clean":"2024-09-21T14:02:50.096450+0000","last_became_active":"2024-09-21T14:01:41.191116+0000","last_became_peered":"2024-09-21T14:01:41.191116+0000","last_unstale":"2024-09-21T14:02:50.096450+0000","last_undegraded":"2024-09-21T14:02:50.096450+0000","last_fullsized":"2024-09-21T14:02:50.096450+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:57.962266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:57.962266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:08.315476+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":684,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T14:02:47.517481+0000","last_change":"2024-09-21T14:00:51.495433+0000","last_active":"2024-09-21T14:02:47.517481+0000","last_peered":"2024-09-21T14:02:47.517481+0000","last_clean":"2024-09-21T14:02:47.517481+0000","last_became_active":"2024-09-21T13:52:04.316016+0000","last_became_peered":"2024-09-21T13:52:04.316016+0000","last_unstale":"2024-09-21T14:02:47.517481+0000","last_undegraded":"2024-09-21T14:02:47.517481+0000","last_fullsized":"2024-09-21T14:02:47.517481+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:00:51.495368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:00:51.495368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:10.415465+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"290'18","reported_seq":636,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.096040+0000","last_change":"2024-09-21T13:59:23.041396+0000","last_active":"2024-09-21T14:02:50.096040+0000","last_peered":"2024-09-21T14:02:50.096040+0000","last_clean":"2024-09-21T14:02:50.096040+0000","last_became_active":"2024-09-21T13:57:12.684454+0000","last_became_peered":"2024-09-21T13:57:12.684454+0000","last_unstale":"2024-09-21T14:02:50.096040+0000","last_undegraded":"2024-09-21T14:02:50.096040+0000","last_fullsized":"2024-09-21T14:02:50.096040+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'18","last_scrub_stamp":"2024-09-21T13:59:23.041311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:23.041311+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":671,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T14:02:47.516927+0000","last_change":"2024-09-21T13:58:16.393763+0000","last_active":"2024-09-21T14:02:47.516927+0000","last_peered":"2024-09-21T14:02:47.516927+0000","last_clean":"2024-09-21T14:02:47.516927+0000","last_became_active":"2024-09-21T13:57:12.681897+0000","last_became_peered":"2024-09-21T13:57:12.681897+0000","last_unstale":"2024-09-21T14:02:47.516927+0000","last_undegraded":"2024-09-21T14:02:47.516927+0000","last_fullsized":"2024-09-21T14:02:47.516927+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:16.393696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:16.393696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:37.708367+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":595,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T14:02:49.946655+0000","last_change":"2024-09-21T14:01:41.182300+0000","last_active":"2024-09-21T14:02:49.946655+0000","last_peered":"2024-09-21T14:02:49.946655+0000","last_clean":"2024-09-21T14:02:49.946655+0000","last_became_active":"2024-09-21T14:01:41.181946+0000","last_became_peered":"2024-09-21T14:01:41.181946+0000","last_unstale":"2024-09-21T14:02:49.946655+0000","last_undegraded":"2024-09-21T14:02:49.946655+0000","last_fullsized":"2024-09-21T14:02:49.946655+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T13:57:26.536684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:26.536684+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":696,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T14:02:49.520593+0000","last_change":"2024-09-21T14:01:58.994867+0000","last_active":"2024-09-21T14:02:49.520593+0000","last_peered":"2024-09-21T14:02:49.520593+0000","last_clean":"2024-09-21T14:02:49.520593+0000","last_became_active":"2024-09-21T13:57:12.685742+0000","last_became_peered":"2024-09-21T13:57:12.685742+0000","last_unstale":"2024-09-21T14:02:49.520593+0000","last_undegraded":"2024-09-21T14:02:49.520593+0000","last_fullsized":"2024-09-21T14:02:49.520593+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T14:01:58.994811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:01:58.994811+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:17.687542+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"290'20","reported_seq":575,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T14:02:49.951024+0000","last_change":"2024-09-21T13:59:32.242771+0000","last_active":"2024-09-21T14:02:49.951024+0000","last_peered":"2024-09-21T14:02:49.951024+0000","last_clean":"2024-09-21T14:02:49.951024+0000","last_became_active":"2024-09-21T13:59:32.213865+0000","last_became_peered":"2024-09-21T13:59:32.213865+0000","last_unstale":"2024-09-21T14:02:49.951024+0000","last_undegraded":"2024-09-21T14:02:49.951024+0000","last_fullsized":"2024-09-21T14:02:49.951024+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:57:27.578738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:57:27.578738+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"300'12","reported_seq":310,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T14:02:47.746177+0000","last_change":"2024-09-21T13:59:36.012775+0000","last_active":"2024-09-21T14:02:47.746177+0000","last_peered":"2024-09-21T14:02:47.746177+0000","last_clean":"2024-09-21T14:02:47.746177+0000","last_became_active":"2024-09-21T13:59:32.205051+0000","last_became_peered":"2024-09-21T13:59:32.205051+0000","last_unstale":"2024-09-21T14:02:47.746177+0000","last_undegraded":"2024-09-21T14:02:47.746177+0000","last_fullsized":"2024-09-21T14:02:47.746177+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'12","last_scrub_stamp":"2024-09-21T13:59:36.012693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:36.012693+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:20.587646+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"75'79","reported_seq":93,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.271861+0000","last_change":"2024-09-21T14:01:41.248772+0000","last_active":"2024-09-21T14:02:50.271861+0000","last_peered":"2024-09-21T14:02:50.271861+0000","last_clean":"2024-09-21T14:02:50.271861+0000","last_became_active":"2024-09-21T14:01:41.246888+0000","last_became_peered":"2024-09-21T14:01:41.246888+0000","last_unstale":"2024-09-21T14:02:50.271861+0000","last_undegraded":"2024-09-21T14:02:50.271861+0000","last_fullsized":"2024-09-21T14:02:50.271861+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53669,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"290'27","reported_seq":735,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T14:02:49.524025+0000","last_change":"2024-09-21T13:58:45.984691+0000","last_active":"2024-09-21T14:02:49.524025+0000","last_peered":"2024-09-21T14:02:49.524025+0000","last_clean":"2024-09-21T14:02:49.524025+0000","last_became_active":"2024-09-21T13:57:12.695609+0000","last_became_peered":"2024-09-21T13:57:12.695609+0000","last_unstale":"2024-09-21T14:02:49.524025+0000","last_undegraded":"2024-09-21T14:02:49.524025+0000","last_fullsized":"2024-09-21T14:02:49.524025+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:58:45.984575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:45.984575+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":622,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.095285+0000","last_change":"2024-09-21T13:59:32.205723+0000","last_active":"2024-09-21T14:02:50.095285+0000","last_peered":"2024-09-21T14:02:50.095285+0000","last_clean":"2024-09-21T14:02:50.095285+0000","last_became_active":"2024-09-21T13:59:32.205350+0000","last_became_peered":"2024-09-21T13:59:32.205350+0000","last_unstale":"2024-09-21T14:02:50.095285+0000","last_undegraded":"2024-09-21T14:02:50.095285+0000","last_fullsized":"2024-09-21T14:02:50.095285+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:32.668851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:32.668851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:47.252073+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"346'10","reported_seq":866,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T14:02:47.517781+0000","last_change":"2024-09-21T14:01:55.521513+0000","last_active":"2024-09-21T14:02:47.517781+0000","last_peered":"2024-09-21T14:02:47.517781+0000","last_clean":"2024-09-21T14:02:47.517781+0000","last_became_active":"2024-09-21T14:01:41.200218+0000","last_became_peered":"2024-09-21T14:01:41.200218+0000","last_unstale":"2024-09-21T14:02:47.517781+0000","last_undegraded":"2024-09-21T14:02:47.517781+0000","last_fullsized":"2024-09-21T14:02:47.517781+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'10","last_scrub_stamp":"2024-09-21T14:01:55.521438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:01:55.521438+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:55.829821+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":549,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.271936+0000","last_change":"2024-09-21T14:01:41.207310+0000","last_active":"2024-09-21T14:02:50.271936+0000","last_peered":"2024-09-21T14:02:50.271936+0000","last_clean":"2024-09-21T14:02:50.271936+0000","last_became_active":"2024-09-21T14:01:41.206720+0000","last_became_peered":"2024-09-21T14:01:41.206720+0000","last_unstale":"2024-09-21T14:02:50.271936+0000","last_undegraded":"2024-09-21T14:02:50.271936+0000","last_fullsized":"2024-09-21T14:02:50.271936+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:52.244483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:52.244483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"75'79","reported_seq":143,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T14:02:47.517653+0000","last_change":"2024-09-21T14:02:06.395083+0000","last_active":"2024-09-21T14:02:47.517653+0000","last_peered":"2024-09-21T14:02:47.517653+0000","last_clean":"2024-09-21T14:02:47.517653+0000","last_became_active":"2024-09-21T14:00:41.137756+0000","last_became_peered":"2024-09-21T14:00:41.137756+0000","last_unstale":"2024-09-21T14:02:47.517653+0000","last_undegraded":"2024-09-21T14:02:47.517653+0000","last_fullsized":"2024-09-21T14:02:47.517653+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:02:06.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:02:06.395011+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:29.136654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"290'30","reported_seq":646,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.095382+0000","last_change":"2024-09-21T13:59:32.246090+0000","last_active":"2024-09-21T14:02:50.095382+0000","last_peered":"2024-09-21T14:02:50.095382+0000","last_clean":"2024-09-21T14:02:50.095382+0000","last_became_active":"2024-09-21T13:59:32.205887+0000","last_became_peered":"2024-09-21T13:59:32.205887+0000","last_unstale":"2024-09-21T14:02:50.095382+0000","last_undegraded":"2024-09-21T14:02:50.095382+0000","last_fullsized":"2024-09-21T14:02:50.095382+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'18","last_scrub_stamp":"2024-09-21T13:54:19.001030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:19.001030+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:43.319659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":574,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.096557+0000","last_change":"2024-09-21T13:59:32.212559+0000","last_active":"2024-09-21T14:02:50.096557+0000","last_peered":"2024-09-21T14:02:50.096557+0000","last_clean":"2024-09-21T14:02:50.096557+0000","last_became_active":"2024-09-21T13:59:32.211004+0000","last_became_peered":"2024-09-21T13:59:32.211004+0000","last_unstale":"2024-09-21T14:02:50.096557+0000","last_undegraded":"2024-09-21T14:02:50.096557+0000","last_fullsized":"2024-09-21T14:02:50.096557+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:25.736461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:25.736461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:48.292957+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":688,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T14:02:47.517209+0000","last_change":"2024-09-21T13:59:34.293330+0000","last_active":"2024-09-21T14:02:47.517209+0000","last_peered":"2024-09-21T14:02:47.517209+0000","last_clean":"2024-09-21T14:02:47.517209+0000","last_became_active":"2024-09-21T13:59:32.220304+0000","last_became_peered":"2024-09-21T13:59:32.220304+0000","last_unstale":"2024-09-21T14:02:47.517209+0000","last_undegraded":"2024-09-21T14:02:47.517209+0000","last_fullsized":"2024-09-21T14:02:47.517209+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:34.292892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:34.292892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:15.468420+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":668,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T14:02:47.517040+0000","last_change":"2024-09-21T13:59:23.334230+0000","last_active":"2024-09-21T14:02:47.517040+0000","last_peered":"2024-09-21T14:02:47.517040+0000","last_clean":"2024-09-21T14:02:47.517040+0000","last_became_active":"2024-09-21T13:57:12.680372+0000","last_became_peered":"2024-09-21T13:57:12.680372+0000","last_unstale":"2024-09-21T14:02:47.517040+0000","last_undegraded":"2024-09-21T14:02:47.517040+0000","last_fullsized":"2024-09-21T14:02:47.517040+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:23.334162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:23.334162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:17.515611+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"75'79","reported_seq":150,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.095716+0000","last_change":"2024-09-21T14:01:42.177646+0000","last_active":"2024-09-21T14:02:50.095716+0000","last_peered":"2024-09-21T14:02:50.095716+0000","last_clean":"2024-09-21T14:02:50.095716+0000","last_became_active":"2024-09-21T14:01:42.173042+0000","last_became_peered":"2024-09-21T14:01:42.173042+0000","last_unstale":"2024-09-21T14:02:50.095716+0000","last_undegraded":"2024-09-21T14:02:50.095716+0000","last_fullsized":"2024-09-21T14:02:50.095716+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:03:42.356436+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":709030,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"290'28","reported_seq":777,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T14:02:47.516641+0000","last_change":"2024-09-21T14:02:02.471226+0000","last_active":"2024-09-21T14:02:47.516641+0000","last_peered":"2024-09-21T14:02:47.516641+0000","last_clean":"2024-09-21T14:02:47.516641+0000","last_became_active":"2024-09-21T14:01:41.200859+0000","last_became_peered":"2024-09-21T14:01:41.200859+0000","last_unstale":"2024-09-21T14:02:47.516641+0000","last_undegraded":"2024-09-21T14:02:47.516641+0000","last_fullsized":"2024-09-21T14:02:47.516641+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'28","last_scrub_stamp":"2024-09-21T14:02:02.471133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:02.471133+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:14.278651+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":676,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T14:02:47.522589+0000","last_change":"2024-09-21T13:57:38.046718+0000","last_active":"2024-09-21T14:02:47.522589+0000","last_peered":"2024-09-21T14:02:47.522589+0000","last_clean":"2024-09-21T14:02:47.522589+0000","last_became_active":"2024-09-21T13:48:58.053533+0000","last_became_peered":"2024-09-21T13:48:58.053533+0000","last_unstale":"2024-09-21T14:02:47.522589+0000","last_undegraded":"2024-09-21T14:02:47.522589+0000","last_fullsized":"2024-09-21T14:02:47.522589+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:57:38.046627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:38.046627+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"241'10","reported_seq":722,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.096146+0000","last_change":"2024-09-21T13:58:06.789629+0000","last_active":"2024-09-21T14:02:50.096146+0000","last_peered":"2024-09-21T14:02:50.096146+0000","last_clean":"2024-09-21T14:02:50.096146+0000","last_became_active":"2024-09-21T13:57:12.686651+0000","last_became_peered":"2024-09-21T13:57:12.686651+0000","last_unstale":"2024-09-21T14:02:50.096146+0000","last_undegraded":"2024-09-21T14:02:50.096146+0000","last_fullsized":"2024-09-21T14:02:50.096146+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'10","last_scrub_stamp":"2024-09-21T13:58:06.789560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:06.789560+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":636,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T14:02:49.946965+0000","last_change":"2024-09-21T13:59:32.210257+0000","last_active":"2024-09-21T14:02:49.946965+0000","last_peered":"2024-09-21T14:02:49.946965+0000","last_clean":"2024-09-21T14:02:49.946965+0000","last_became_active":"2024-09-21T13:59:32.209615+0000","last_became_peered":"2024-09-21T13:59:32.209615+0000","last_unstale":"2024-09-21T14:02:49.946965+0000","last_undegraded":"2024-09-21T14:02:49.946965+0000","last_fullsized":"2024-09-21T14:02:49.946965+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:37.548064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:37.548064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"75'79","reported_seq":715,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T14:02:47.746065+0000","last_change":"2024-09-21T14:02:05.693250+0000","last_active":"2024-09-21T14:02:47.746065+0000","last_peered":"2024-09-21T14:02:47.746065+0000","last_clean":"2024-09-21T14:02:47.746065+0000","last_became_active":"2024-09-21T14:00:41.083425+0000","last_became_peered":"2024-09-21T14:00:41.083425+0000","last_unstale":"2024-09-21T14:02:47.746065+0000","last_undegraded":"2024-09-21T14:02:47.746065+0000","last_fullsized":"2024-09-21T14:02:47.746065+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:02:05.693193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:02:05.693193+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:28.620866+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"290'20","reported_seq":714,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T14:02:49.521239+0000","last_change":"2024-09-21T14:01:41.187743+0000","last_active":"2024-09-21T14:02:49.521239+0000","last_peered":"2024-09-21T14:02:49.521239+0000","last_clean":"2024-09-21T14:02:49.521239+0000","last_became_active":"2024-09-21T14:01:41.187060+0000","last_became_peered":"2024-09-21T14:01:41.187060+0000","last_unstale":"2024-09-21T14:02:49.521239+0000","last_undegraded":"2024-09-21T14:02:49.521239+0000","last_fullsized":"2024-09-21T14:02:49.521239+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:54:19.593611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:19.593611+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:47.290628+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"61'1","reported_seq":611,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.271734+0000","last_change":"2024-09-21T14:01:41.195580+0000","last_active":"2024-09-21T14:02:50.271734+0000","last_peered":"2024-09-21T14:02:50.271734+0000","last_clean":"2024-09-21T14:02:50.271734+0000","last_became_active":"2024-09-21T14:01:41.195268+0000","last_became_peered":"2024-09-21T14:01:41.195268+0000","last_unstale":"2024-09-21T14:02:50.271734+0000","last_undegraded":"2024-09-21T14:02:50.271734+0000","last_fullsized":"2024-09-21T14:02:50.271734+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:58:48.407734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:48.407734+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"346'15","reported_seq":303,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T14:02:47.745808+0000","last_change":"2024-09-21T14:01:41.209662+0000","last_active":"2024-09-21T14:02:47.745808+0000","last_peered":"2024-09-21T14:02:47.745808+0000","last_clean":"2024-09-21T14:02:47.745808+0000","last_became_active":"2024-09-21T14:01:41.209296+0000","last_became_peered":"2024-09-21T14:01:41.209296+0000","last_unstale":"2024-09-21T14:02:47.745808+0000","last_undegraded":"2024-09-21T14:02:47.745808+0000","last_fullsized":"2024-09-21T14:02:47.745808+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'12","last_scrub_stamp":"2024-09-21T13:57:30.133392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:30.133392+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:43.651875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":492,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.096047+0000","last_change":"2024-09-21T13:58:58.730439+0000","last_active":"2024-09-21T14:02:50.096047+0000","last_peered":"2024-09-21T14:02:50.096047+0000","last_clean":"2024-09-21T14:02:50.096047+0000","last_became_active":"2024-09-21T13:57:12.704897+0000","last_became_peered":"2024-09-21T13:57:12.704897+0000","last_unstale":"2024-09-21T14:02:50.096047+0000","last_undegraded":"2024-09-21T14:02:50.096047+0000","last_fullsized":"2024-09-21T14:02:50.096047+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:58.730368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:58.730368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"75'79","reported_seq":715,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T14:02:47.745948+0000","last_change":"2024-09-21T14:02:03.737005+0000","last_active":"2024-09-21T14:02:47.745948+0000","last_peered":"2024-09-21T14:02:47.745948+0000","last_clean":"2024-09-21T14:02:47.745948+0000","last_became_active":"2024-09-21T14:00:41.073047+0000","last_became_peered":"2024-09-21T14:00:41.073047+0000","last_unstale":"2024-09-21T14:02:47.745948+0000","last_undegraded":"2024-09-21T14:02:47.745948+0000","last_fullsized":"2024-09-21T14:02:47.745948+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:02:03.736928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:02:03.736928+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:21.563427+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"290'29","reported_seq":617,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.271845+0000","last_change":"2024-09-21T14:01:41.188261+0000","last_active":"2024-09-21T14:02:50.271845+0000","last_peered":"2024-09-21T14:02:50.271845+0000","last_clean":"2024-09-21T14:02:50.271845+0000","last_became_active":"2024-09-21T14:01:41.186960+0000","last_became_peered":"2024-09-21T14:01:41.186960+0000","last_unstale":"2024-09-21T14:02:50.271845+0000","last_undegraded":"2024-09-21T14:02:50.271845+0000","last_fullsized":"2024-09-21T14:02:50.271845+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-21T13:56:55.319759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:56:55.319759+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":691,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T14:02:49.522601+0000","last_change":"2024-09-21T14:01:41.185706+0000","last_active":"2024-09-21T14:02:49.522601+0000","last_peered":"2024-09-21T14:02:49.522601+0000","last_clean":"2024-09-21T14:02:49.522601+0000","last_became_active":"2024-09-21T14:01:41.185400+0000","last_became_peered":"2024-09-21T14:01:41.185400+0000","last_unstale":"2024-09-21T14:02:49.522601+0000","last_undegraded":"2024-09-21T14:02:49.522601+0000","last_fullsized":"2024-09-21T14:02:49.522601+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:16.712990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:16.712990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:38.309423+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":257,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T14:02:47.745770+0000","last_change":"2024-09-21T14:01:52.671029+0000","last_active":"2024-09-21T14:02:47.745770+0000","last_peered":"2024-09-21T14:02:47.745770+0000","last_clean":"2024-09-21T14:02:47.745770+0000","last_became_active":"2024-09-21T14:01:41.193973+0000","last_became_peered":"2024-09-21T14:01:41.193973+0000","last_unstale":"2024-09-21T14:02:47.745770+0000","last_undegraded":"2024-09-21T14:02:47.745770+0000","last_fullsized":"2024-09-21T14:02:47.745770+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:52.670974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:01:52.670974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:58.750020+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":682,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T14:02:49.520341+0000","last_change":"2024-09-21T13:58:43.952014+0000","last_active":"2024-09-21T14:02:49.520341+0000","last_peered":"2024-09-21T14:02:49.520341+0000","last_clean":"2024-09-21T14:02:49.520341+0000","last_became_active":"2024-09-21T13:58:17.757628+0000","last_became_peered":"2024-09-21T13:58:17.757628+0000","last_unstale":"2024-09-21T14:02:49.520341+0000","last_undegraded":"2024-09-21T14:02:49.520341+0000","last_fullsized":"2024-09-21T14:02:49.520341+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:43.951941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:43.951941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"75'79","reported_seq":93,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.271570+0000","last_change":"2024-09-21T14:01:41.199377+0000","last_active":"2024-09-21T14:02:50.271570+0000","last_peered":"2024-09-21T14:02:50.271570+0000","last_clean":"2024-09-21T14:02:50.271570+0000","last_became_active":"2024-09-21T14:01:41.198750+0000","last_became_peered":"2024-09-21T14:01:41.198750+0000","last_unstale":"2024-09-21T14:02:50.271570+0000","last_undegraded":"2024-09-21T14:02:50.271570+0000","last_fullsized":"2024-09-21T14:02:50.271570+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:03:47.451019+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"290'44","reported_seq":924,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T14:02:49.522410+0000","last_change":"2024-09-21T13:58:56.076153+0000","last_active":"2024-09-21T14:02:49.522410+0000","last_peered":"2024-09-21T14:02:49.522410+0000","last_clean":"2024-09-21T14:02:49.522410+0000","last_became_active":"2024-09-21T13:57:12.684690+0000","last_became_peered":"2024-09-21T13:57:12.684690+0000","last_unstale":"2024-09-21T14:02:49.522410+0000","last_undegraded":"2024-09-21T14:02:49.522410+0000","last_fullsized":"2024-09-21T14:02:49.522410+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'32","last_scrub_stamp":"2024-09-21T13:58:56.076074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:56.076074+0000","objects_scrubbed":10,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":696,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T14:02:47.517332+0000","last_change":"2024-09-21T14:01:58.524159+0000","last_active":"2024-09-21T14:02:47.517332+0000","last_peered":"2024-09-21T14:02:47.517332+0000","last_clean":"2024-09-21T14:02:47.517332+0000","last_became_active":"2024-09-21T13:58:17.770937+0000","last_became_peered":"2024-09-21T13:58:17.770937+0000","last_unstale":"2024-09-21T14:02:47.517332+0000","last_undegraded":"2024-09-21T14:02:47.517332+0000","last_fullsized":"2024-09-21T14:02:47.517332+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:58.524114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:01:58.524114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:07.944133+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":630,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.271909+0000","last_change":"2024-09-21T14:01:41.249061+0000","last_active":"2024-09-21T14:02:50.271909+0000","last_peered":"2024-09-21T14:02:50.271909+0000","last_clean":"2024-09-21T14:02:50.271909+0000","last_became_active":"2024-09-21T14:01:41.247642+0000","last_became_peered":"2024-09-21T14:01:41.247642+0000","last_unstale":"2024-09-21T14:02:50.271909+0000","last_undegraded":"2024-09-21T14:02:50.271909+0000","last_fullsized":"2024-09-21T14:02:50.271909+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-21T13:59:22.186393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:59:22.186393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":609,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T14:02:49.523673+0000","last_change":"2024-09-21T13:55:07.021128+0000","last_active":"2024-09-21T14:02:49.523673+0000","last_peered":"2024-09-21T14:02:49.523673+0000","last_clean":"2024-09-21T14:02:49.523673+0000","last_became_active":"2024-09-21T13:50:29.688249+0000","last_became_peered":"2024-09-21T13:50:29.688249+0000","last_unstale":"2024-09-21T14:02:49.523673+0000","last_undegraded":"2024-09-21T14:02:49.523673+0000","last_fullsized":"2024-09-21T14:02:49.523673+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:07.021059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:55:07.021059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":601,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.095886+0000","last_change":"2024-09-21T13:59:32.209357+0000","last_active":"2024-09-21T14:02:50.095886+0000","last_peered":"2024-09-21T14:02:50.095886+0000","last_clean":"2024-09-21T14:02:50.095886+0000","last_became_active":"2024-09-21T13:59:32.208980+0000","last_became_peered":"2024-09-21T13:59:32.208980+0000","last_unstale":"2024-09-21T14:02:50.095886+0000","last_undegraded":"2024-09-21T14:02:50.095886+0000","last_fullsized":"2024-09-21T14:02:50.095886+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:16.797227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:16.797227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"75'79","reported_seq":724,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T14:02:49.946639+0000","last_change":"2024-09-21T14:00:41.077286+0000","last_active":"2024-09-21T14:02:49.946639+0000","last_peered":"2024-09-21T14:02:49.946639+0000","last_clean":"2024-09-21T14:02:49.946639+0000","last_became_active":"2024-09-21T14:00:41.076948+0000","last_became_peered":"2024-09-21T14:00:41.076948+0000","last_unstale":"2024-09-21T14:02:49.946639+0000","last_undegraded":"2024-09-21T14:02:49.946639+0000","last_fullsized":"2024-09-21T14:02:49.946639+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"290'17","reported_seq":717,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T14:02:49.524151+0000","last_change":"2024-09-21T14:01:41.200809+0000","last_active":"2024-09-21T14:02:49.524151+0000","last_peered":"2024-09-21T14:02:49.524151+0000","last_clean":"2024-09-21T14:02:49.524151+0000","last_became_active":"2024-09-21T14:01:41.200365+0000","last_became_peered":"2024-09-21T14:01:41.200365+0000","last_unstale":"2024-09-21T14:02:49.524151+0000","last_undegraded":"2024-09-21T14:02:49.524151+0000","last_fullsized":"2024-09-21T14:02:49.524151+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'13","last_scrub_stamp":"2024-09-21T13:58:11.619848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:11.619848+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:39.361066+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":718,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T14:02:49.521286+0000","last_change":"2024-09-21T13:59:29.454265+0000","last_active":"2024-09-21T14:02:49.521286+0000","last_peered":"2024-09-21T14:02:49.521286+0000","last_clean":"2024-09-21T14:02:49.521286+0000","last_became_active":"2024-09-21T13:58:17.759671+0000","last_became_peered":"2024-09-21T13:58:17.759671+0000","last_unstale":"2024-09-21T14:02:49.521286+0000","last_undegraded":"2024-09-21T14:02:49.521286+0000","last_fullsized":"2024-09-21T14:02:49.521286+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:29.454085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:59:29.454085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":606,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T14:02:47.516308+0000","last_change":"2024-09-21T13:58:41.644876+0000","last_active":"2024-09-21T14:02:47.516308+0000","last_peered":"2024-09-21T14:02:47.516308+0000","last_clean":"2024-09-21T14:02:47.516308+0000","last_became_active":"2024-09-21T13:57:12.682147+0000","last_became_peered":"2024-09-21T13:57:12.682147+0000","last_unstale":"2024-09-21T14:02:47.516308+0000","last_undegraded":"2024-09-21T14:02:47.516308+0000","last_fullsized":"2024-09-21T14:02:47.516308+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:58:41.644789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:58:41.644789+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:13.796854+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":584,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T14:02:47.517265+0000","last_change":"2024-09-21T14:01:50.880840+0000","last_active":"2024-09-21T14:02:47.517265+0000","last_peered":"2024-09-21T14:02:47.517265+0000","last_clean":"2024-09-21T14:02:47.517265+0000","last_became_active":"2024-09-21T14:01:41.182000+0000","last_became_peered":"2024-09-21T14:01:41.182000+0000","last_unstale":"2024-09-21T14:02:47.517265+0000","last_undegraded":"2024-09-21T14:02:47.517265+0000","last_fullsized":"2024-09-21T14:02:47.517265+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:50.880471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:01:50.880471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:14.118692+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":644,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T14:02:49.948625+0000","last_change":"2024-09-21T13:59:32.222791+0000","last_active":"2024-09-21T14:02:49.948625+0000","last_peered":"2024-09-21T14:02:49.948625+0000","last_clean":"2024-09-21T14:02:49.948625+0000","last_became_active":"2024-09-21T13:59:32.221502+0000","last_became_peered":"2024-09-21T13:59:32.221502+0000","last_unstale":"2024-09-21T14:02:49.948625+0000","last_undegraded":"2024-09-21T14:02:49.948625+0000","last_fullsized":"2024-09-21T14:02:49.948625+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:14.673582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:14.673582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"75'79","reported_seq":129,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T14:02:49.522387+0000","last_change":"2024-09-21T14:00:41.071631+0000","last_active":"2024-09-21T14:02:49.522387+0000","last_peered":"2024-09-21T14:02:49.522387+0000","last_clean":"2024-09-21T14:02:49.522387+0000","last_became_active":"2024-09-21T14:00:41.071042+0000","last_became_peered":"2024-09-21T14:00:41.071042+0000","last_unstale":"2024-09-21T14:02:49.522387+0000","last_undegraded":"2024-09-21T14:02:49.522387+0000","last_fullsized":"2024-09-21T14:02:49.522387+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:03:42.335979+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"290'24","reported_seq":766,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T14:02:47.522475+0000","last_change":"2024-09-21T14:01:41.185105+0000","last_active":"2024-09-21T14:02:47.522475+0000","last_peered":"2024-09-21T14:02:47.522475+0000","last_clean":"2024-09-21T14:02:47.522475+0000","last_became_active":"2024-09-21T14:01:41.184728+0000","last_became_peered":"2024-09-21T14:01:41.184728+0000","last_unstale":"2024-09-21T14:02:47.522475+0000","last_undegraded":"2024-09-21T14:02:47.522475+0000","last_fullsized":"2024-09-21T14:02:47.522475+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'24","last_scrub_stamp":"2024-09-21T13:59:28.458768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:28.458768+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":617,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T14:02:47.517169+0000","last_change":"2024-09-21T13:59:59.450613+0000","last_active":"2024-09-21T14:02:47.517169+0000","last_peered":"2024-09-21T14:02:47.517169+0000","last_clean":"2024-09-21T14:02:47.517169+0000","last_became_active":"2024-09-21T13:57:12.681030+0000","last_became_peered":"2024-09-21T13:57:12.681030+0000","last_unstale":"2024-09-21T14:02:47.517169+0000","last_undegraded":"2024-09-21T14:02:47.517169+0000","last_fullsized":"2024-09-21T14:02:47.517169+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:59.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:59:59.450546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:40.683320+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":1377,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T14:02:49.946492+0000","last_change":"2024-09-21T13:59:32.218796+0000","last_active":"2024-09-21T14:02:49.946492+0000","last_peered":"2024-09-21T14:02:49.946492+0000","last_clean":"2024-09-21T14:02:49.946492+0000","last_became_active":"2024-09-21T13:59:32.203361+0000","last_became_peered":"2024-09-21T13:59:32.203361+0000","last_unstale":"2024-09-21T14:02:49.946492+0000","last_undegraded":"2024-09-21T14:02:49.946492+0000","last_fullsized":"2024-09-21T14:02:49.946492+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-21T13:56:29.617143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:56:29.617143+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047048000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":205,"num_read_kb":200,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":601,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.272208+0000","last_change":"2024-09-21T14:01:41.212595+0000","last_active":"2024-09-21T14:02:50.272208+0000","last_peered":"2024-09-21T14:02:50.272208+0000","last_clean":"2024-09-21T14:02:50.272208+0000","last_became_active":"2024-09-21T14:01:41.200531+0000","last_became_peered":"2024-09-21T14:01:41.200531+0000","last_unstale":"2024-09-21T14:02:50.272208+0000","last_undegraded":"2024-09-21T14:02:50.272208+0000","last_fullsized":"2024-09-21T14:02:50.272208+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:33.316230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:56:33.316230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":703,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T14:02:49.520713+0000","last_change":"2024-09-21T14:01:41.184939+0000","last_active":"2024-09-21T14:02:49.520713+0000","last_peered":"2024-09-21T14:02:49.520713+0000","last_clean":"2024-09-21T14:02:49.520713+0000","last_became_active":"2024-09-21T14:01:41.184664+0000","last_became_peered":"2024-09-21T14:01:41.184664+0000","last_unstale":"2024-09-21T14:02:49.520713+0000","last_undegraded":"2024-09-21T14:02:49.520713+0000","last_fullsized":"2024-09-21T14:02:49.520713+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:26.557002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:26.557002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"300'32","reported_seq":356,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T14:02:47.745986+0000","last_change":"2024-09-21T14:00:47.130089+0000","last_active":"2024-09-21T14:02:47.745986+0000","last_peered":"2024-09-21T14:02:47.745986+0000","last_clean":"2024-09-21T14:02:47.745986+0000","last_became_active":"2024-09-21T13:59:32.204624+0000","last_became_peered":"2024-09-21T13:59:32.204624+0000","last_unstale":"2024-09-21T14:02:47.745986+0000","last_undegraded":"2024-09-21T14:02:47.745986+0000","last_fullsized":"2024-09-21T14:02:47.745986+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'32","last_scrub_stamp":"2024-09-21T14:00:47.129789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:00:47.129789+0000","objects_scrubbed":8,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:15.616969+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":659,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.095813+0000","last_change":"2024-09-21T14:01:41.190537+0000","last_active":"2024-09-21T14:02:50.095813+0000","last_peered":"2024-09-21T14:02:50.095813+0000","last_clean":"2024-09-21T14:02:50.095813+0000","last_became_active":"2024-09-21T14:01:41.190210+0000","last_became_peered":"2024-09-21T14:01:41.190210+0000","last_unstale":"2024-09-21T14:02:50.095813+0000","last_undegraded":"2024-09-21T14:02:50.095813+0000","last_fullsized":"2024-09-21T14:02:50.095813+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:57.772975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:57.772975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"75'79","reported_seq":718,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T14:02:49.947121+0000","last_change":"2024-09-21T14:00:39.339360+0000","last_active":"2024-09-21T14:02:49.947121+0000","last_peered":"2024-09-21T14:02:49.947121+0000","last_clean":"2024-09-21T14:02:49.947121+0000","last_became_active":"2024-09-21T14:00:39.339077+0000","last_became_peered":"2024-09-21T14:00:39.339077+0000","last_unstale":"2024-09-21T14:02:49.947121+0000","last_undegraded":"2024-09-21T14:02:49.947121+0000","last_fullsized":"2024-09-21T14:02:49.947121+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:03:49.763645+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":19,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53669,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":632,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T14:02:49.946742+0000","last_change":"2024-09-21T14:01:41.182186+0000","last_active":"2024-09-21T14:02:49.946742+0000","last_peered":"2024-09-21T14:02:49.946742+0000","last_clean":"2024-09-21T14:02:49.946742+0000","last_became_active":"2024-09-21T14:01:41.181899+0000","last_became_peered":"2024-09-21T14:01:41.181899+0000","last_unstale":"2024-09-21T14:02:49.946742+0000","last_undegraded":"2024-09-21T14:02:49.946742+0000","last_fullsized":"2024-09-21T14:02:49.946742+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:15.700409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:15.700409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":559,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.272090+0000","last_change":"2024-09-21T14:01:41.196509+0000","last_active":"2024-09-21T14:02:50.272090+0000","last_peered":"2024-09-21T14:02:50.272090+0000","last_clean":"2024-09-21T14:02:50.272090+0000","last_became_active":"2024-09-21T14:01:41.196243+0000","last_became_peered":"2024-09-21T14:01:41.196243+0000","last_unstale":"2024-09-21T14:02:50.272090+0000","last_undegraded":"2024-09-21T14:02:50.272090+0000","last_fullsized":"2024-09-21T14:02:50.272090+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:21.153018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:21.153018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"290'19","reported_seq":742,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T14:02:47.517751+0000","last_change":"2024-09-21T14:02:28.553549+0000","last_active":"2024-09-21T14:02:47.517751+0000","last_peered":"2024-09-21T14:02:47.517751+0000","last_clean":"2024-09-21T14:02:47.517751+0000","last_became_active":"2024-09-21T13:58:17.776310+0000","last_became_peered":"2024-09-21T13:58:17.776310+0000","last_unstale":"2024-09-21T14:02:47.517751+0000","last_undegraded":"2024-09-21T14:02:47.517751+0000","last_fullsized":"2024-09-21T14:02:47.517751+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'19","last_scrub_stamp":"2024-09-21T14:02:28.553065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:28.553065+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:33.636087+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":704,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T14:02:47.517401+0000","last_change":"2024-09-21T14:02:03.437581+0000","last_active":"2024-09-21T14:02:47.517401+0000","last_peered":"2024-09-21T14:02:47.517401+0000","last_clean":"2024-09-21T14:02:47.517401+0000","last_became_active":"2024-09-21T13:59:32.223247+0000","last_became_peered":"2024-09-21T13:59:32.223247+0000","last_unstale":"2024-09-21T14:02:47.517401+0000","last_undegraded":"2024-09-21T14:02:47.517401+0000","last_fullsized":"2024-09-21T14:02:47.517401+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:03.437510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:02:03.437510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:14.620200+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":627,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.272089+0000","last_change":"2024-09-21T14:01:41.194538+0000","last_active":"2024-09-21T14:02:50.272089+0000","last_peered":"2024-09-21T14:02:50.272089+0000","last_clean":"2024-09-21T14:02:50.272089+0000","last_became_active":"2024-09-21T14:01:41.193941+0000","last_became_peered":"2024-09-21T14:01:41.193941+0000","last_unstale":"2024-09-21T14:02:50.272089+0000","last_undegraded":"2024-09-21T14:02:50.272089+0000","last_fullsized":"2024-09-21T14:02:50.272089+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:34.299722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:55:34.299722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":637,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.096287+0000","last_change":"2024-09-21T14:01:41.190501+0000","last_active":"2024-09-21T14:02:50.096287+0000","last_peered":"2024-09-21T14:02:50.096287+0000","last_clean":"2024-09-21T14:02:50.096287+0000","last_became_active":"2024-09-21T14:01:41.190192+0000","last_became_peered":"2024-09-21T14:01:41.190192+0000","last_unstale":"2024-09-21T14:02:50.096287+0000","last_undegraded":"2024-09-21T14:02:50.096287+0000","last_fullsized":"2024-09-21T14:02:50.096287+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:58.952218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:54:58.952218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:18.255572+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"75'79","reported_seq":100,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.271425+0000","last_change":"2024-09-21T14:01:41.189146+0000","last_active":"2024-09-21T14:02:50.271425+0000","last_peered":"2024-09-21T14:02:50.271425+0000","last_clean":"2024-09-21T14:02:50.271425+0000","last_became_active":"2024-09-21T14:01:41.188685+0000","last_became_peered":"2024-09-21T14:01:41.188685+0000","last_unstale":"2024-09-21T14:02:50.271425+0000","last_undegraded":"2024-09-21T14:02:50.271425+0000","last_fullsized":"2024-09-21T14:02:50.271425+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:03:46.437011+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"290'31","reported_seq":730,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T14:02:47.517538+0000","last_change":"2024-09-21T14:01:03.541547+0000","last_active":"2024-09-21T14:02:47.517538+0000","last_peered":"2024-09-21T14:02:47.517538+0000","last_clean":"2024-09-21T14:02:47.517538+0000","last_became_active":"2024-09-21T13:52:04.313218+0000","last_became_peered":"2024-09-21T13:52:04.313218+0000","last_unstale":"2024-09-21T14:02:47.517538+0000","last_undegraded":"2024-09-21T14:02:47.517538+0000","last_fullsized":"2024-09-21T14:02:47.517538+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'31","last_scrub_stamp":"2024-09-21T14:01:03.541432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:03.541432+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":246,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T14:02:47.745543+0000","last_change":"2024-09-21T14:01:29.924408+0000","last_active":"2024-09-21T14:02:47.745543+0000","last_peered":"2024-09-21T14:02:47.745543+0000","last_clean":"2024-09-21T14:02:47.745543+0000","last_became_active":"2024-09-21T13:58:17.762520+0000","last_became_peered":"2024-09-21T13:58:17.762520+0000","last_unstale":"2024-09-21T14:02:47.745543+0000","last_undegraded":"2024-09-21T14:02:47.745543+0000","last_fullsized":"2024-09-21T14:02:47.745543+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:29.924331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:01:29.924331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:47.626829+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":661,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T14:02:49.520805+0000","last_change":"2024-09-21T13:58:52.030800+0000","last_active":"2024-09-21T14:02:49.520805+0000","last_peered":"2024-09-21T14:02:49.520805+0000","last_clean":"2024-09-21T14:02:49.520805+0000","last_became_active":"2024-09-21T13:49:02.067472+0000","last_became_peered":"2024-09-21T13:49:02.067472+0000","last_unstale":"2024-09-21T14:02:49.520805+0000","last_undegraded":"2024-09-21T14:02:49.520805+0000","last_fullsized":"2024-09-21T14:02:49.520805+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:52.030718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:52.030718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:43.346814+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":541,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T14:02:47.518012+0000","last_change":"2024-09-21T13:57:12.683321+0000","last_active":"2024-09-21T14:02:47.518012+0000","last_peered":"2024-09-21T14:02:47.518012+0000","last_clean":"2024-09-21T14:02:47.518012+0000","last_became_active":"2024-09-21T13:57:12.682979+0000","last_became_peered":"2024-09-21T13:57:12.682979+0000","last_unstale":"2024-09-21T14:02:47.518012+0000","last_undegraded":"2024-09-21T14:02:47.518012+0000","last_fullsized":"2024-09-21T14:02:47.518012+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:18.027944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:18.027944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:21.753981+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"75'79","reported_seq":724,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T14:02:49.947629+0000","last_change":"2024-09-21T14:00:41.140108+0000","last_active":"2024-09-21T14:02:49.947629+0000","last_peered":"2024-09-21T14:02:49.947629+0000","last_clean":"2024-09-21T14:02:49.947629+0000","last_became_active":"2024-09-21T14:00:41.139315+0000","last_became_peered":"2024-09-21T14:00:41.139315+0000","last_unstale":"2024-09-21T14:02:49.947629+0000","last_undegraded":"2024-09-21T14:02:49.947629+0000","last_fullsized":"2024-09-21T14:02:49.947629+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"290'20","reported_seq":749,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T14:02:47.516949+0000","last_change":"2024-09-21T13:58:17.789407+0000","last_active":"2024-09-21T14:02:47.516949+0000","last_peered":"2024-09-21T14:02:47.516949+0000","last_clean":"2024-09-21T14:02:47.516949+0000","last_became_active":"2024-09-21T13:58:17.765427+0000","last_became_peered":"2024-09-21T13:58:17.765427+0000","last_unstale":"2024-09-21T14:02:47.516949+0000","last_undegraded":"2024-09-21T14:02:47.516949+0000","last_fullsized":"2024-09-21T14:02:47.516949+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:58:13.464196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:13.464196+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:16.467667+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":594,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.095139+0000","last_change":"2024-09-21T13:57:36.707211+0000","last_active":"2024-09-21T14:02:50.095139+0000","last_peered":"2024-09-21T14:02:50.095139+0000","last_clean":"2024-09-21T14:02:50.095139+0000","last_became_active":"2024-09-21T13:57:12.704203+0000","last_became_peered":"2024-09-21T13:57:12.704203+0000","last_unstale":"2024-09-21T14:02:50.095139+0000","last_undegraded":"2024-09-21T14:02:50.095139+0000","last_fullsized":"2024-09-21T14:02:50.095139+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:36.707148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:36.707148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"346'14","reported_seq":720,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T14:02:47.518242+0000","last_change":"2024-09-21T14:01:41.186960+0000","last_active":"2024-09-21T14:02:47.518242+0000","last_peered":"2024-09-21T14:02:47.518242+0000","last_clean":"2024-09-21T14:02:47.518242+0000","last_became_active":"2024-09-21T14:01:41.186564+0000","last_became_peered":"2024-09-21T14:01:41.186564+0000","last_unstale":"2024-09-21T14:02:47.518242+0000","last_undegraded":"2024-09-21T14:02:47.518242+0000","last_fullsized":"2024-09-21T14:02:47.518242+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'12","last_scrub_stamp":"2024-09-21T13:59:38.738120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:38.738120+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"68'1","reported_seq":677,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T14:02:47.517532+0000","last_change":"2024-09-21T13:59:31.392186+0000","last_active":"2024-09-21T14:02:47.517532+0000","last_peered":"2024-09-21T14:02:47.517532+0000","last_clean":"2024-09-21T14:02:47.517532+0000","last_became_active":"2024-09-21T13:52:04.313210+0000","last_became_peered":"2024-09-21T13:52:04.313210+0000","last_unstale":"2024-09-21T14:02:47.517532+0000","last_undegraded":"2024-09-21T14:02:47.517532+0000","last_fullsized":"2024-09-21T14:02:47.517532+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T13:59:31.392105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:31.392105+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:39.716003+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"290'15","reported_seq":704,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T14:02:47.522024+0000","last_change":"2024-09-21T13:58:50.498597+0000","last_active":"2024-09-21T14:02:47.522024+0000","last_peered":"2024-09-21T14:02:47.522024+0000","last_clean":"2024-09-21T14:02:47.522024+0000","last_became_active":"2024-09-21T13:49:00.053016+0000","last_became_peered":"2024-09-21T13:49:00.053016+0000","last_unstale":"2024-09-21T14:02:47.522024+0000","last_undegraded":"2024-09-21T14:02:47.522024+0000","last_fullsized":"2024-09-21T14:02:47.522024+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:58:50.498369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:50.498369+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":639,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.271854+0000","last_change":"2024-09-21T14:01:41.192717+0000","last_active":"2024-09-21T14:02:50.271854+0000","last_peered":"2024-09-21T14:02:50.271854+0000","last_clean":"2024-09-21T14:02:50.271854+0000","last_became_active":"2024-09-21T14:01:41.192443+0000","last_became_peered":"2024-09-21T14:01:41.192443+0000","last_unstale":"2024-09-21T14:02:50.271854+0000","last_undegraded":"2024-09-21T14:02:50.271854+0000","last_fullsized":"2024-09-21T14:02:50.271854+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:53.894506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:53.894506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":615,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T14:02:47.516760+0000","last_change":"2024-09-21T14:01:27.739999+0000","last_active":"2024-09-21T14:02:47.516760+0000","last_peered":"2024-09-21T14:02:47.516760+0000","last_clean":"2024-09-21T14:02:47.516760+0000","last_became_active":"2024-09-21T13:57:12.680978+0000","last_became_peered":"2024-09-21T13:57:12.680978+0000","last_unstale":"2024-09-21T14:02:47.516760+0000","last_undegraded":"2024-09-21T14:02:47.516760+0000","last_fullsized":"2024-09-21T14:02:47.516760+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:27.739921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:01:27.739921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:50.205235+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":567,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T14:02:49.946485+0000","last_change":"2024-09-21T13:59:32.224001+0000","last_active":"2024-09-21T14:02:49.946485+0000","last_peered":"2024-09-21T14:02:49.946485+0000","last_clean":"2024-09-21T14:02:49.946485+0000","last_became_active":"2024-09-21T13:59:32.221628+0000","last_became_peered":"2024-09-21T13:59:32.221628+0000","last_unstale":"2024-09-21T14:02:49.946485+0000","last_undegraded":"2024-09-21T14:02:49.946485+0000","last_fullsized":"2024-09-21T14:02:49.946485+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:13.698091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:13.698091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"290'35","reported_seq":312,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T14:02:47.745620+0000","last_change":"2024-09-21T14:01:02.006619+0000","last_active":"2024-09-21T14:02:47.745620+0000","last_peered":"2024-09-21T14:02:47.745620+0000","last_clean":"2024-09-21T14:02:47.745620+0000","last_became_active":"2024-09-21T13:59:32.204184+0000","last_became_peered":"2024-09-21T13:59:32.204184+0000","last_unstale":"2024-09-21T14:02:47.745620+0000","last_undegraded":"2024-09-21T14:02:47.745620+0000","last_fullsized":"2024-09-21T14:02:47.745620+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'35","last_scrub_stamp":"2024-09-21T14:01:02.006504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:02.006504+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:22.557193+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":659,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T14:02:49.950220+0000","last_change":"2024-09-21T14:01:41.184937+0000","last_active":"2024-09-21T14:02:49.950220+0000","last_peered":"2024-09-21T14:02:49.950220+0000","last_clean":"2024-09-21T14:02:49.950220+0000","last_became_active":"2024-09-21T14:01:41.184567+0000","last_became_peered":"2024-09-21T14:01:41.184567+0000","last_unstale":"2024-09-21T14:02:49.950220+0000","last_undegraded":"2024-09-21T14:02:49.950220+0000","last_fullsized":"2024-09-21T14:02:49.950220+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:21.587493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:21.587493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":635,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T14:02:47.517760+0000","last_change":"2024-09-21T14:01:41.182250+0000","last_active":"2024-09-21T14:02:47.517760+0000","last_peered":"2024-09-21T14:02:47.517760+0000","last_clean":"2024-09-21T14:02:47.517760+0000","last_became_active":"2024-09-21T14:01:41.181911+0000","last_became_peered":"2024-09-21T14:01:41.181911+0000","last_unstale":"2024-09-21T14:02:47.517760+0000","last_undegraded":"2024-09-21T14:02:47.517760+0000","last_fullsized":"2024-09-21T14:02:47.517760+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:55.588554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:55.588554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:45.648976+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":586,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T14:02:49.946543+0000","last_change":"2024-09-21T13:59:32.223123+0000","last_active":"2024-09-21T14:02:49.946543+0000","last_peered":"2024-09-21T14:02:49.946543+0000","last_clean":"2024-09-21T14:02:49.946543+0000","last_became_active":"2024-09-21T13:59:32.219332+0000","last_became_peered":"2024-09-21T13:59:32.219332+0000","last_unstale":"2024-09-21T14:02:49.946543+0000","last_undegraded":"2024-09-21T14:02:49.946543+0000","last_fullsized":"2024-09-21T14:02:49.946543+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:19.537177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:19.537177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"290'18","reported_seq":721,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T14:02:47.517287+0000","last_change":"2024-09-21T13:59:28.340494+0000","last_active":"2024-09-21T14:02:47.517287+0000","last_peered":"2024-09-21T14:02:47.517287+0000","last_clean":"2024-09-21T14:02:47.517287+0000","last_became_active":"2024-09-21T13:57:12.680627+0000","last_became_peered":"2024-09-21T13:57:12.680627+0000","last_unstale":"2024-09-21T14:02:47.517287+0000","last_undegraded":"2024-09-21T14:02:47.517287+0000","last_fullsized":"2024-09-21T14:02:47.517287+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'18","last_scrub_stamp":"2024-09-21T13:59:28.340415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:28.340415+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:11.383693+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":614,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T14:02:49.950937+0000","last_change":"2024-09-21T13:59:32.201126+0000","last_active":"2024-09-21T14:02:49.950937+0000","last_peered":"2024-09-21T14:02:49.950937+0000","last_clean":"2024-09-21T14:02:49.950937+0000","last_became_active":"2024-09-21T13:59:32.200674+0000","last_became_peered":"2024-09-21T13:59:32.200674+0000","last_unstale":"2024-09-21T14:02:49.950937+0000","last_undegraded":"2024-09-21T14:02:49.950937+0000","last_fullsized":"2024-09-21T14:02:49.950937+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:21.582325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:21.582325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":597,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T14:02:47.516131+0000","last_change":"2024-09-21T14:02:27.791041+0000","last_active":"2024-09-21T14:02:47.516131+0000","last_peered":"2024-09-21T14:02:47.516131+0000","last_clean":"2024-09-21T14:02:47.516131+0000","last_became_active":"2024-09-21T13:52:04.322147+0000","last_became_peered":"2024-09-21T13:52:04.322147+0000","last_unstale":"2024-09-21T14:02:47.516131+0000","last_undegraded":"2024-09-21T14:02:47.516131+0000","last_fullsized":"2024-09-21T14:02:47.516131+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:27.790994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:02:27.790994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:51.879201+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":610,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.096511+0000","last_change":"2024-09-21T14:02:36.371955+0000","last_active":"2024-09-21T14:02:50.096511+0000","last_peered":"2024-09-21T14:02:50.096511+0000","last_clean":"2024-09-21T14:02:50.096511+0000","last_became_active":"2024-09-21T13:59:32.203767+0000","last_became_peered":"2024-09-21T13:59:32.203767+0000","last_unstale":"2024-09-21T14:02:50.096511+0000","last_undegraded":"2024-09-21T14:02:50.096511+0000","last_fullsized":"2024-09-21T14:02:50.096511+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:36.371747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:02:36.371747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:54.168810+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"290'27","reported_seq":715,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.271547+0000","last_change":"2024-09-21T14:01:41.193143+0000","last_active":"2024-09-21T14:02:50.271547+0000","last_peered":"2024-09-21T14:02:50.271547+0000","last_clean":"2024-09-21T14:02:50.271547+0000","last_became_active":"2024-09-21T14:01:41.192481+0000","last_became_peered":"2024-09-21T14:01:41.192481+0000","last_unstale":"2024-09-21T14:02:50.271547+0000","last_undegraded":"2024-09-21T14:02:50.271547+0000","last_fullsized":"2024-09-21T14:02:50.271547+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'27","last_scrub_stamp":"2024-09-21T13:59:33.058859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:33.058859+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":586,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T14:02:49.951289+0000","last_change":"2024-09-21T13:59:32.203889+0000","last_active":"2024-09-21T14:02:49.951289+0000","last_peered":"2024-09-21T14:02:49.951289+0000","last_clean":"2024-09-21T14:02:49.951289+0000","last_became_active":"2024-09-21T13:59:32.203172+0000","last_became_peered":"2024-09-21T13:59:32.203172+0000","last_unstale":"2024-09-21T14:02:49.951289+0000","last_undegraded":"2024-09-21T14:02:49.951289+0000","last_fullsized":"2024-09-21T14:02:49.951289+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T13:55:03.879706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:55:03.879706+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"300'14","reported_seq":800,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T14:02:49.946329+0000","last_change":"2024-09-21T13:59:32.220898+0000","last_active":"2024-09-21T14:02:49.946329+0000","last_peered":"2024-09-21T14:02:49.946329+0000","last_clean":"2024-09-21T14:02:49.946329+0000","last_became_active":"2024-09-21T13:59:32.206956+0000","last_became_peered":"2024-09-21T13:59:32.206956+0000","last_unstale":"2024-09-21T14:02:49.946329+0000","last_undegraded":"2024-09-21T14:02:49.946329+0000","last_fullsized":"2024-09-21T14:02:49.946329+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'11","last_scrub_stamp":"2024-09-21T13:56:31.616003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:56:31.616003+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":663,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.271686+0000","last_change":"2024-09-21T14:01:41.248408+0000","last_active":"2024-09-21T14:02:50.271686+0000","last_peered":"2024-09-21T14:02:50.271686+0000","last_clean":"2024-09-21T14:02:50.271686+0000","last_became_active":"2024-09-21T14:01:41.246106+0000","last_became_peered":"2024-09-21T14:01:41.246106+0000","last_unstale":"2024-09-21T14:02:50.271686+0000","last_undegraded":"2024-09-21T14:02:50.271686+0000","last_fullsized":"2024-09-21T14:02:50.271686+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:39.258659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:39.258659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":248,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T14:02:47.745627+0000","last_change":"2024-09-21T14:01:50.733055+0000","last_active":"2024-09-21T14:02:47.745627+0000","last_peered":"2024-09-21T14:02:47.745627+0000","last_clean":"2024-09-21T14:02:47.745627+0000","last_became_active":"2024-09-21T14:01:41.197501+0000","last_became_peered":"2024-09-21T14:01:41.197501+0000","last_unstale":"2024-09-21T14:02:47.745627+0000","last_undegraded":"2024-09-21T14:02:47.745627+0000","last_fullsized":"2024-09-21T14:02:47.745627+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:50.733007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:01:50.733007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:08.367766+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"290'12","reported_seq":637,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.096677+0000","last_change":"2024-09-21T13:59:32.247422+0000","last_active":"2024-09-21T14:02:50.096677+0000","last_peered":"2024-09-21T14:02:50.096677+0000","last_clean":"2024-09-21T14:02:50.096677+0000","last_became_active":"2024-09-21T13:59:32.211306+0000","last_became_peered":"2024-09-21T13:59:32.211306+0000","last_unstale":"2024-09-21T14:02:50.096677+0000","last_undegraded":"2024-09-21T14:02:50.096677+0000","last_fullsized":"2024-09-21T14:02:50.096677+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'6","last_scrub_stamp":"2024-09-21T13:55:12.066873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:55:12.066873+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:38.360712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":595,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.271871+0000","last_change":"2024-09-21T14:01:41.199742+0000","last_active":"2024-09-21T14:02:50.271871+0000","last_peered":"2024-09-21T14:02:50.271871+0000","last_clean":"2024-09-21T14:02:50.271871+0000","last_became_active":"2024-09-21T14:01:41.199321+0000","last_became_peered":"2024-09-21T14:01:41.199321+0000","last_unstale":"2024-09-21T14:02:50.271871+0000","last_undegraded":"2024-09-21T14:02:50.271871+0000","last_fullsized":"2024-09-21T14:02:50.271871+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:37.106238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:37.106238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":632,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T14:02:49.948901+0000","last_change":"2024-09-21T13:59:32.222277+0000","last_active":"2024-09-21T14:02:49.948901+0000","last_peered":"2024-09-21T14:02:49.948901+0000","last_clean":"2024-09-21T14:02:49.948901+0000","last_became_active":"2024-09-21T13:59:32.219525+0000","last_became_peered":"2024-09-21T13:59:32.219525+0000","last_unstale":"2024-09-21T14:02:49.948901+0000","last_undegraded":"2024-09-21T14:02:49.948901+0000","last_fullsized":"2024-09-21T14:02:49.948901+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:02.838181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:55:02.838181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"290'15","reported_seq":600,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.095686+0000","last_change":"2024-09-21T13:59:32.238689+0000","last_active":"2024-09-21T14:02:50.095686+0000","last_peered":"2024-09-21T14:02:50.095686+0000","last_clean":"2024-09-21T14:02:50.095686+0000","last_became_active":"2024-09-21T13:59:32.209166+0000","last_became_peered":"2024-09-21T13:59:32.209166+0000","last_unstale":"2024-09-21T14:02:50.095686+0000","last_undegraded":"2024-09-21T14:02:50.095686+0000","last_fullsized":"2024-09-21T14:02:50.095686+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:55:04.990842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:55:04.990842+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:21.278052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"61'1","reported_seq":650,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T14:02:49.947474+0000","last_change":"2024-09-21T13:59:32.222826+0000","last_active":"2024-09-21T14:02:49.947474+0000","last_peered":"2024-09-21T14:02:49.947474+0000","last_clean":"2024-09-21T14:02:49.947474+0000","last_became_active":"2024-09-21T13:59:32.219515+0000","last_became_peered":"2024-09-21T13:59:32.219515+0000","last_unstale":"2024-09-21T14:02:49.947474+0000","last_undegraded":"2024-09-21T14:02:49.947474+0000","last_fullsized":"2024-09-21T14:02:49.947474+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:58:07.711028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:07.711028+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":585,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.096399+0000","last_change":"2024-09-21T13:59:32.211754+0000","last_active":"2024-09-21T14:02:50.096399+0000","last_peered":"2024-09-21T14:02:50.096399+0000","last_clean":"2024-09-21T14:02:50.096399+0000","last_became_active":"2024-09-21T13:59:32.210586+0000","last_became_peered":"2024-09-21T13:59:32.210586+0000","last_unstale":"2024-09-21T14:02:50.096399+0000","last_undegraded":"2024-09-21T14:02:50.096399+0000","last_fullsized":"2024-09-21T14:02:50.096399+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:28.014760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:28.014760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:17.280712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":639,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T14:02:47.518341+0000","last_change":"2024-09-21T14:02:29.731635+0000","last_active":"2024-09-21T14:02:47.518341+0000","last_peered":"2024-09-21T14:02:47.518341+0000","last_clean":"2024-09-21T14:02:47.518341+0000","last_became_active":"2024-09-21T13:59:32.205803+0000","last_became_peered":"2024-09-21T13:59:32.205803+0000","last_unstale":"2024-09-21T14:02:47.518341+0000","last_undegraded":"2024-09-21T14:02:47.518341+0000","last_fullsized":"2024-09-21T14:02:47.518341+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:29.731565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:02:29.731565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:53.473477+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"290'15","reported_seq":640,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.272125+0000","last_change":"2024-09-21T14:01:41.189676+0000","last_active":"2024-09-21T14:02:50.272125+0000","last_peered":"2024-09-21T14:02:50.272125+0000","last_clean":"2024-09-21T14:02:50.272125+0000","last_became_active":"2024-09-21T14:01:41.187904+0000","last_became_peered":"2024-09-21T14:01:41.187904+0000","last_unstale":"2024-09-21T14:02:50.272125+0000","last_undegraded":"2024-09-21T14:02:50.272125+0000","last_fullsized":"2024-09-21T14:02:50.272125+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:57:24.124604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:57:24.124604+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":519,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T14:02:49.947259+0000","last_change":"2024-09-21T13:59:32.212325+0000","last_active":"2024-09-21T14:02:49.947259+0000","last_peered":"2024-09-21T14:02:49.947259+0000","last_clean":"2024-09-21T14:02:49.947259+0000","last_became_active":"2024-09-21T13:59:32.208749+0000","last_became_peered":"2024-09-21T13:59:32.208749+0000","last_unstale":"2024-09-21T14:02:49.947259+0000","last_undegraded":"2024-09-21T14:02:49.947259+0000","last_fullsized":"2024-09-21T14:02:49.947259+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:20.574745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:20.574745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"300'14","reported_seq":735,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.096312+0000","last_change":"2024-09-21T13:59:32.208253+0000","last_active":"2024-09-21T14:02:50.096312+0000","last_peered":"2024-09-21T14:02:50.096312+0000","last_clean":"2024-09-21T14:02:50.096312+0000","last_became_active":"2024-09-21T13:59:32.206926+0000","last_became_peered":"2024-09-21T13:59:32.206926+0000","last_unstale":"2024-09-21T14:02:50.096312+0000","last_undegraded":"2024-09-21T14:02:50.096312+0000","last_fullsized":"2024-09-21T14:02:50.096312+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'10","last_scrub_stamp":"2024-09-21T13:54:20.946782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:20.946782+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:10.314285+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":683,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T14:02:47.517945+0000","last_change":"2024-09-21T14:01:41.200564+0000","last_active":"2024-09-21T14:02:47.517945+0000","last_peered":"2024-09-21T14:02:47.517945+0000","last_clean":"2024-09-21T14:02:47.517945+0000","last_became_active":"2024-09-21T14:01:41.200113+0000","last_became_peered":"2024-09-21T14:01:41.200113+0000","last_unstale":"2024-09-21T14:02:47.517945+0000","last_undegraded":"2024-09-21T14:02:47.517945+0000","last_fullsized":"2024-09-21T14:02:47.517945+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:54.805619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:54.805619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:48.557673+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"290'20","reported_seq":636,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.096213+0000","last_change":"2024-09-21T13:59:32.236530+0000","last_active":"2024-09-21T14:02:50.096213+0000","last_peered":"2024-09-21T14:02:50.096213+0000","last_clean":"2024-09-21T14:02:50.096213+0000","last_became_active":"2024-09-21T13:59:32.208332+0000","last_became_peered":"2024-09-21T13:59:32.208332+0000","last_unstale":"2024-09-21T14:02:50.096213+0000","last_undegraded":"2024-09-21T14:02:50.096213+0000","last_fullsized":"2024-09-21T14:02:50.096213+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:57:37.671391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:57:37.671391+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":626,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T14:02:49.946833+0000","last_change":"2024-09-21T13:59:32.212737+0000","last_active":"2024-09-21T14:02:49.946833+0000","last_peered":"2024-09-21T14:02:49.946833+0000","last_clean":"2024-09-21T14:02:49.946833+0000","last_became_active":"2024-09-21T13:59:32.209804+0000","last_became_peered":"2024-09-21T13:59:32.209804+0000","last_unstale":"2024-09-21T14:02:49.946833+0000","last_undegraded":"2024-09-21T14:02:49.946833+0000","last_fullsized":"2024-09-21T14:02:49.946833+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:25.571256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:25.571256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"346'10","reported_seq":735,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.272121+0000","last_change":"2024-09-21T14:01:41.207030+0000","last_active":"2024-09-21T14:02:50.272121+0000","last_peered":"2024-09-21T14:02:50.272121+0000","last_clean":"2024-09-21T14:02:50.272121+0000","last_became_active":"2024-09-21T14:01:41.206303+0000","last_became_peered":"2024-09-21T14:01:41.206303+0000","last_unstale":"2024-09-21T14:02:50.272121+0000","last_undegraded":"2024-09-21T14:02:50.272121+0000","last_fullsized":"2024-09-21T14:02:50.272121+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:59:32.104224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:32.104224+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":523,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T14:02:49.950733+0000","last_change":"2024-09-21T13:59:32.213015+0000","last_active":"2024-09-21T14:02:49.950733+0000","last_peered":"2024-09-21T14:02:49.950733+0000","last_clean":"2024-09-21T14:02:49.950733+0000","last_became_active":"2024-09-21T13:59:32.211472+0000","last_became_peered":"2024-09-21T13:59:32.211472+0000","last_unstale":"2024-09-21T14:02:49.950733+0000","last_undegraded":"2024-09-21T14:02:49.950733+0000","last_fullsized":"2024-09-21T14:02:49.950733+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:22.647191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:22.647191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"290'19","reported_seq":746,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T14:02:47.516956+0000","last_change":"2024-09-21T14:02:34.670058+0000","last_active":"2024-09-21T14:02:47.516956+0000","last_peered":"2024-09-21T14:02:47.516956+0000","last_clean":"2024-09-21T14:02:47.516956+0000","last_became_active":"2024-09-21T13:58:17.771497+0000","last_became_peered":"2024-09-21T13:58:17.771497+0000","last_unstale":"2024-09-21T14:02:47.516956+0000","last_undegraded":"2024-09-21T14:02:47.516956+0000","last_fullsized":"2024-09-21T14:02:47.516956+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'19","last_scrub_stamp":"2024-09-21T14:02:34.669962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:34.669962+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:44.716931+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":260,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T14:02:49.946822+0000","last_change":"2024-09-21T13:59:32.202097+0000","last_active":"2024-09-21T14:02:49.946822+0000","last_peered":"2024-09-21T14:02:49.946822+0000","last_clean":"2024-09-21T14:02:49.946822+0000","last_became_active":"2024-09-21T13:59:32.201568+0000","last_became_peered":"2024-09-21T13:59:32.201568+0000","last_unstale":"2024-09-21T14:02:49.946822+0000","last_undegraded":"2024-09-21T14:02:49.946822+0000","last_fullsized":"2024-09-21T14:02:49.946822+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:01.228213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:01.228213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":698,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T14:02:49.521106+0000","last_change":"2024-09-21T14:01:41.185588+0000","last_active":"2024-09-21T14:02:49.521106+0000","last_peered":"2024-09-21T14:02:49.521106+0000","last_clean":"2024-09-21T14:02:49.521106+0000","last_became_active":"2024-09-21T14:01:41.185293+0000","last_became_peered":"2024-09-21T14:01:41.185293+0000","last_unstale":"2024-09-21T14:02:49.521106+0000","last_undegraded":"2024-09-21T14:02:49.521106+0000","last_fullsized":"2024-09-21T14:02:49.521106+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:35.409237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:35.409237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:10.948556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":593,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.095142+0000","last_change":"2024-09-21T14:01:41.200439+0000","last_active":"2024-09-21T14:02:50.095142+0000","last_peered":"2024-09-21T14:02:50.095142+0000","last_clean":"2024-09-21T14:02:50.095142+0000","last_became_active":"2024-09-21T14:01:41.199671+0000","last_became_peered":"2024-09-21T14:01:41.199671+0000","last_unstale":"2024-09-21T14:02:50.095142+0000","last_undegraded":"2024-09-21T14:02:50.095142+0000","last_fullsized":"2024-09-21T14:02:50.095142+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:39.643458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:39.643458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"290'15","reported_seq":742,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T14:02:49.521372+0000","last_change":"2024-09-21T13:58:39.977717+0000","last_active":"2024-09-21T14:02:49.521372+0000","last_peered":"2024-09-21T14:02:49.521372+0000","last_clean":"2024-09-21T14:02:49.521372+0000","last_became_active":"2024-09-21T13:58:17.764120+0000","last_became_peered":"2024-09-21T13:58:17.764120+0000","last_unstale":"2024-09-21T14:02:49.521372+0000","last_undegraded":"2024-09-21T14:02:49.521372+0000","last_fullsized":"2024-09-21T14:02:49.521372+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:58:39.977466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:39.977466+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":630,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T14:02:49.948706+0000","last_change":"2024-09-21T13:59:32.223082+0000","last_active":"2024-09-21T14:02:49.948706+0000","last_peered":"2024-09-21T14:02:49.948706+0000","last_clean":"2024-09-21T14:02:49.948706+0000","last_became_active":"2024-09-21T13:59:32.221854+0000","last_became_peered":"2024-09-21T13:59:32.221854+0000","last_unstale":"2024-09-21T14:02:49.948706+0000","last_undegraded":"2024-09-21T14:02:49.948706+0000","last_fullsized":"2024-09-21T14:02:49.948706+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:17.691193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:17.691193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":568,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.095031+0000","last_change":"2024-09-21T13:57:12.703746+0000","last_active":"2024-09-21T14:02:50.095031+0000","last_peered":"2024-09-21T14:02:50.095031+0000","last_clean":"2024-09-21T14:02:50.095031+0000","last_became_active":"2024-09-21T13:57:12.703441+0000","last_became_peered":"2024-09-21T13:57:12.703441+0000","last_unstale":"2024-09-21T14:02:50.095031+0000","last_undegraded":"2024-09-21T14:02:50.095031+0000","last_fullsized":"2024-09-21T14:02:50.095031+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:19.999125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:19.999125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:13.302246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":490,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.271420+0000","last_change":"2024-09-21T14:01:41.188075+0000","last_active":"2024-09-21T14:02:50.271420+0000","last_peered":"2024-09-21T14:02:50.271420+0000","last_clean":"2024-09-21T14:02:50.271420+0000","last_became_active":"2024-09-21T14:01:41.186934+0000","last_became_peered":"2024-09-21T14:01:41.186934+0000","last_unstale":"2024-09-21T14:02:50.271420+0000","last_undegraded":"2024-09-21T14:02:50.271420+0000","last_fullsized":"2024-09-21T14:02:50.271420+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:47.334504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:47.334504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"290'19","reported_seq":712,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.095380+0000","last_change":"2024-09-21T13:59:32.277742+0000","last_active":"2024-09-21T14:02:50.095380+0000","last_peered":"2024-09-21T14:02:50.095380+0000","last_clean":"2024-09-21T14:02:50.095380+0000","last_became_active":"2024-09-21T13:59:32.231567+0000","last_became_peered":"2024-09-21T13:59:32.231567+0000","last_unstale":"2024-09-21T14:02:50.095380+0000","last_undegraded":"2024-09-21T14:02:50.095380+0000","last_fullsized":"2024-09-21T14:02:50.095380+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:54:22.978267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:22.978267+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:12.294653+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":592,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.271847+0000","last_change":"2024-09-21T14:01:41.191090+0000","last_active":"2024-09-21T14:02:50.271847+0000","last_peered":"2024-09-21T14:02:50.271847+0000","last_clean":"2024-09-21T14:02:50.271847+0000","last_became_active":"2024-09-21T14:01:41.190760+0000","last_became_peered":"2024-09-21T14:01:41.190760+0000","last_unstale":"2024-09-21T14:02:50.271847+0000","last_undegraded":"2024-09-21T14:02:50.271847+0000","last_fullsized":"2024-09-21T14:02:50.271847+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:17.305144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:17.305144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":636,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T14:02:49.947053+0000","last_change":"2024-09-21T13:59:32.219508+0000","last_active":"2024-09-21T14:02:49.947053+0000","last_peered":"2024-09-21T14:02:49.947053+0000","last_clean":"2024-09-21T14:02:49.947053+0000","last_became_active":"2024-09-21T13:59:32.214004+0000","last_became_peered":"2024-09-21T13:59:32.214004+0000","last_unstale":"2024-09-21T14:02:49.947053+0000","last_undegraded":"2024-09-21T14:02:49.947053+0000","last_fullsized":"2024-09-21T14:02:49.947053+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:36.525417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:36.525417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":634,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.095235+0000","last_change":"2024-09-21T13:59:28.124373+0000","last_active":"2024-09-21T14:02:50.095235+0000","last_peered":"2024-09-21T14:02:50.095235+0000","last_clean":"2024-09-21T14:02:50.095235+0000","last_became_active":"2024-09-21T13:58:17.778115+0000","last_became_peered":"2024-09-21T13:58:17.778115+0000","last_unstale":"2024-09-21T14:02:50.095235+0000","last_undegraded":"2024-09-21T14:02:50.095235+0000","last_fullsized":"2024-09-21T14:02:50.095235+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:28.124296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:28.124296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"290'8","reported_seq":658,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.096253+0000","last_change":"2024-09-21T13:59:26.034457+0000","last_active":"2024-09-21T14:02:50.096253+0000","last_peered":"2024-09-21T14:02:50.096253+0000","last_clean":"2024-09-21T14:02:50.096253+0000","last_became_active":"2024-09-21T13:58:17.775079+0000","last_became_peered":"2024-09-21T13:58:17.775079+0000","last_unstale":"2024-09-21T14:02:50.096253+0000","last_undegraded":"2024-09-21T14:02:50.096253+0000","last_fullsized":"2024-09-21T14:02:50.096253+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'8","last_scrub_stamp":"2024-09-21T13:59:26.034372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:26.034372+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":560,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.271924+0000","last_change":"2024-09-21T14:01:41.196335+0000","last_active":"2024-09-21T14:02:50.271924+0000","last_peered":"2024-09-21T14:02:50.271924+0000","last_clean":"2024-09-21T14:02:50.271924+0000","last_became_active":"2024-09-21T14:01:41.195989+0000","last_became_peered":"2024-09-21T14:01:41.195989+0000","last_unstale":"2024-09-21T14:02:50.271924+0000","last_undegraded":"2024-09-21T14:02:50.271924+0000","last_fullsized":"2024-09-21T14:02:50.271924+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:16.457445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:16.457445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":248,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T14:02:47.745559+0000","last_change":"2024-09-21T14:00:57.128764+0000","last_active":"2024-09-21T14:02:47.745559+0000","last_peered":"2024-09-21T14:02:47.745559+0000","last_clean":"2024-09-21T14:02:47.745559+0000","last_became_active":"2024-09-21T13:59:32.204380+0000","last_became_peered":"2024-09-21T13:59:32.204380+0000","last_unstale":"2024-09-21T14:02:47.745559+0000","last_undegraded":"2024-09-21T14:02:47.745559+0000","last_fullsized":"2024-09-21T14:02:47.745559+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:00:57.128713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:00:57.128713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:24.563060+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":569,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T14:02:49.947189+0000","last_change":"2024-09-21T13:59:32.212615+0000","last_active":"2024-09-21T14:02:49.947189+0000","last_peered":"2024-09-21T14:02:49.947189+0000","last_clean":"2024-09-21T14:02:49.947189+0000","last_became_active":"2024-09-21T13:59:32.210134+0000","last_became_peered":"2024-09-21T13:59:32.210134+0000","last_unstale":"2024-09-21T14:02:49.947189+0000","last_undegraded":"2024-09-21T14:02:49.947189+0000","last_fullsized":"2024-09-21T14:02:49.947189+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:08.852789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:55:08.852789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":683,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T14:02:49.523009+0000","last_change":"2024-09-21T14:01:41.187792+0000","last_active":"2024-09-21T14:02:49.523009+0000","last_peered":"2024-09-21T14:02:49.523009+0000","last_clean":"2024-09-21T14:02:49.523009+0000","last_became_active":"2024-09-21T14:01:41.187438+0000","last_became_peered":"2024-09-21T14:01:41.187438+0000","last_unstale":"2024-09-21T14:02:49.523009+0000","last_undegraded":"2024-09-21T14:02:49.523009+0000","last_fullsized":"2024-09-21T14:02:49.523009+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:17.706325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:17.706325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:40.383161+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":614,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.096358+0000","last_change":"2024-09-21T13:58:13.776632+0000","last_active":"2024-09-21T14:02:50.096358+0000","last_peered":"2024-09-21T14:02:50.096358+0000","last_clean":"2024-09-21T14:02:50.096358+0000","last_became_active":"2024-09-21T13:57:12.704042+0000","last_became_peered":"2024-09-21T13:57:12.704042+0000","last_unstale":"2024-09-21T14:02:50.096358+0000","last_undegraded":"2024-09-21T14:02:50.096358+0000","last_fullsized":"2024-09-21T14:02:50.096358+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:13.776572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:13.776572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"290'19","reported_seq":281,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-21T14:02:47.746094+0000","last_change":"2024-09-21T14:01:54.680822+0000","last_active":"2024-09-21T14:02:47.746094+0000","last_peered":"2024-09-21T14:02:47.746094+0000","last_clean":"2024-09-21T14:02:47.746094+0000","last_became_active":"2024-09-21T14:01:41.194606+0000","last_became_peered":"2024-09-21T14:01:41.194606+0000","last_unstale":"2024-09-21T14:02:47.746094+0000","last_undegraded":"2024-09-21T14:02:47.746094+0000","last_fullsized":"2024-09-21T14:02:47.746094+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'19","last_scrub_stamp":"2024-09-21T14:01:54.680735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:54.680735+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:21.312249+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":538,"reported_epoch":379,"state":"active+clean","last_fresh":"2024-09-21T14:02:49.950289+0000","last_change":"2024-09-21T13:59:32.203045+0000","last_active":"2024-09-21T14:02:49.950289+0000","last_peered":"2024-09-21T14:02:49.950289+0000","last_clean":"2024-09-21T14:02:49.950289+0000","last_became_active":"2024-09-21T13:59:32.202225+0000","last_became_peered":"2024-09-21T13:59:32.202225+0000","last_unstale":"2024-09-21T14:02:49.950289+0000","last_undegraded":"2024-09-21T14:02:49.950289+0000","last_fullsized":"2024-09-21T14:02:49.950289+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:40.650832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:40.650832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1111,"num_read_kb":868,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":1914,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":687,"ondisk_log_size":687,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":205,"num_read_kb":200,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":429358,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":159,"num_write":83,"num_write_kb":1184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1084718,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":298,"seq":1279900254249,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":310796,"kb_used_data":2744,"kb_used_omap":4,"kb_used_meta":308027,"kb_avail":93434356,"statfs":{"total":95995035648,"available":95676780544,"internally_reserved":0,"allocated":2809856,"data_stored":1951183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":315420328},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463088,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":322332,"kb_used_data":2376,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":93422820,"statfs":{"total":95995035648,"available":95664967680,"internally_reserved":0,"allocated":2433024,"data_stored":1606192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":327612870},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":345,"seq":1481763717135,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":111820,"kb_used_data":3000,"kb_used_omap":4,"kb_used_meta":108795,"kb_avail":93633332,"statfs":{"total":95995035648,"available":95880531968,"internally_reserved":0,"allocated":3072000,"data_stored":2221417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":111406755},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920957,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":421896,"kb_used_data":39476,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":93323256,"statfs":{"total":95995035648,"available":95563014144,"internally_reserved":0,"allocated":40423424,"data_stored":39560242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":391576005},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":240,"seq":1030792151109,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":257460,"kb_used_data":3032,"kb_used_omap":4,"kb_used_meta":254395,"kb_avail":93487692,"statfs":{"total":95995035648,"available":95731396608,"internally_reserved":0,"allocated":3104768,"data_stored":2221949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4443,"internal_metadata":260501157},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":130,"seq":558345748610,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":548528,"kb_used_data":2648,"kb_used_omap":4,"kb_used_meta":545851,"kb_avail":93196624,"statfs":{"total":95995035648,"available":95433342976,"internally_reserved":0,"allocated":2711552,"data_stored":1877686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4441,"internal_metadata":558952103},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575055,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":583404,"kb_used_data":2452,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":93161748,"statfs":{"total":95995035648,"available":95397629952,"internally_reserved":0,"allocated":2510848,"data_stored":1615642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":594868677},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738581,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":480640,"kb_used_data":2408,"kb_used_omap":1,"kb_used_meta":478206,"kb_avail":93264512,"statfs":{"total":95995035648,"available":95502860288,"internally_reserved":0,"allocated":2465792,"data_stored":1614638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":489683397},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T14:02:53.252 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:53.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:53.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:53.355 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:02:53.376 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:53.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:53.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:02:53.441 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:02:53.478 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:53.501 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:53.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:53.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:53.603 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:53.625 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:53.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:53.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:53.727 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:02:53.748 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:53.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:53.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:02:53.850 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:53.866 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:53 smithi049 ceph-mon[40489]: pgmap v711: 142 pgs: 142 active+clean; 425 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 746 B/s rd, 0 op/s 2024-09-21T14:02:53.866 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:53 smithi049 ceph-mon[40489]: from='client.25913 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:02:53.867 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:53 smithi049 ceph-mon[30080]: pgmap v711: 142 pgs: 142 active+clean; 425 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 746 B/s rd, 0 op/s 2024-09-21T14:02:53.867 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:53 smithi049 ceph-mon[30080]: from='client.25913 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:02:53.873 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:53.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:53.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:53.907 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:02:53.910 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:02:53.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:53 smithi187 ceph-mon[33032]: pgmap v711: 142 pgs: 142 active+clean; 425 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 746 B/s rd, 0 op/s 2024-09-21T14:02:53.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:53 smithi187 ceph-mon[33032]: from='client.25913 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:02:53.975 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:02:53.997 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:53.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:53.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:02:54.098 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:02:54.120 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:54.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:54.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:02:54.221 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:02:54.243 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:54.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:54.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:02:54.344 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:54.367 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:54.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:54.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:54.469 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:02:54.490 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:54.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:54.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:02:54.592 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:02:54.614 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:54.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:54.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:02:54.715 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:02:54.736 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:54.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:54.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:02:54.838 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:54.860 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:54.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:54.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:54.962 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:02:54.984 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:54.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:54.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:02:55.086 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:55.108 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:55.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:55.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:55.210 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:02:55.232 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:55.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:55.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:02:55.323 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T14:02:55.323 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:02:55.334 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:55.356 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:55.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:55.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:55.458 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:02:55.482 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:55.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:55.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:02:55.544 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-09-21T14:02:55.552 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'unique_pool_3' created 2024-09-21T14:02:55.583 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:55.606 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:55.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:55.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:55.707 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:02:55.729 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:55.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:55.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:02:55.831 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:02:55.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:55.857 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:55.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:02:55.874 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:55 smithi049 ceph-mon[40489]: pgmap v712: 142 pgs: 142 active+clean; 425 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 632 B/s rd, 0 op/s 2024-09-21T14:02:55.875 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:55 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2381121225' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:02:55.875 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:55 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3025664765' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-21T14:02:55.875 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:55 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-21T14:02:55.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:55 smithi049 ceph-mon[30080]: pgmap v712: 142 pgs: 142 active+clean; 425 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 632 B/s rd, 0 op/s 2024-09-21T14:02:55.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:55 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2381121225' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:02:55.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:55 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3025664765' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-21T14:02:55.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:55 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-21T14:02:55.896 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T14:02:55.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:55 smithi187 ceph-mon[33032]: pgmap v712: 142 pgs: 142 active+clean; 425 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 632 B/s rd, 0 op/s 2024-09-21T14:02:55.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:55 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2381121225' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:02:55.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:55 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3025664765' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-21T14:02:55.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:55 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-21T14:02:55.959 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:55.981 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:55.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:55.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:56.084 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:56.110 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:56.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:56.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:56.213 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:56.236 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:56.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:56.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:56.239 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T14:02:56.339 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:02:56.366 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:56.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:56.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:02:56.468 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:02:56.494 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:56.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:56.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:02:56.596 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:56.618 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:56.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:56.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:56.719 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:02:56.739 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T14:02:56.746 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:56.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:56.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:02:56.795 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:56 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:02:56.795 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:56 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:02:56.796 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:56 smithi049 ceph-mon[40489]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:02:56.796 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:56 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2381121225' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T14:02:56.796 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:56 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-21T14:02:56.796 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:56 smithi049 ceph-mon[40489]: osdmap e381: 8 total, 8 up, 8 in 2024-09-21T14:02:56.796 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:56 smithi049 ceph-mon[40489]: 4.1c scrub starts 2024-09-21T14:02:56.796 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:56 smithi049 ceph-mon[40489]: 4.1c scrub ok 2024-09-21T14:02:56.796 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:56 smithi049 ceph-mon[40489]: 1.9 scrub starts 2024-09-21T14:02:56.796 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:56 smithi049 ceph-mon[40489]: 1.9 scrub ok 2024-09-21T14:02:56.796 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:56 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:02:56.796 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:56 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:02:56.797 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:56 smithi049 ceph-mon[30080]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:02:56.797 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:56 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2381121225' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T14:02:56.797 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:56 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-21T14:02:56.797 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:56 smithi049 ceph-mon[30080]: osdmap e381: 8 total, 8 up, 8 in 2024-09-21T14:02:56.797 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:56 smithi049 ceph-mon[30080]: 4.1c scrub starts 2024-09-21T14:02:56.797 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:56 smithi049 ceph-mon[30080]: 4.1c scrub ok 2024-09-21T14:02:56.797 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:56 smithi049 ceph-mon[30080]: 1.9 scrub starts 2024-09-21T14:02:56.797 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:56 smithi049 ceph-mon[30080]: 1.9 scrub ok 2024-09-21T14:02:56.847 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:56.870 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:56.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:56.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:56.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:56 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:02:56.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:56 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:02:56.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:56 smithi187 ceph-mon[33032]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:02:56.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:56 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2381121225' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T14:02:56.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:56 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-21T14:02:56.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:56 smithi187 ceph-mon[33032]: osdmap e381: 8 total, 8 up, 8 in 2024-09-21T14:02:56.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:56 smithi187 ceph-mon[33032]: 4.1c scrub starts 2024-09-21T14:02:56.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:56 smithi187 ceph-mon[33032]: 4.1c scrub ok 2024-09-21T14:02:56.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:56 smithi187 ceph-mon[33032]: 1.9 scrub starts 2024-09-21T14:02:56.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:56 smithi187 ceph-mon[33032]: 1.9 scrub ok 2024-09-21T14:02:56.972 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:02:56.998 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:56.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:56.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:02:57.100 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:02:57.125 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:57.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:57.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:02:57.227 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:57.249 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:57.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:57.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:57.351 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:02:57.376 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:57.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:57.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:02:57.478 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:57.501 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:57.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:57.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:57.574 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:02:57.594 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-09-21T14:02:57.604 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:57.617 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:02:57.642 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:57.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:57.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:57.744 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:02:57.767 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:57.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:57.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:02:57.869 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:57.892 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:57 smithi049 ceph-mon[40489]: pgmap v714: 158 pgs: 2 creating+peering, 14 unknown, 142 active+clean; 425 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 790 B/s rd, 0 op/s 2024-09-21T14:02:57.892 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:57 smithi049 ceph-mon[40489]: 4.12 scrub starts 2024-09-21T14:02:57.892 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:57 smithi049 ceph-mon[40489]: 4.12 scrub ok 2024-09-21T14:02:57.892 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:57 smithi049 ceph-mon[40489]: 3.1f scrub starts 2024-09-21T14:02:57.892 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:57 smithi049 ceph-mon[40489]: 3.1f scrub ok 2024-09-21T14:02:57.893 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:57 smithi049 ceph-mon[40489]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T14:02:57.893 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:57 smithi049 ceph-mon[40489]: osdmap e382: 8 total, 8 up, 8 in 2024-09-21T14:02:57.893 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:57 smithi049 ceph-mon[40489]: 5.a scrub starts 2024-09-21T14:02:57.893 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:57 smithi049 ceph-mon[40489]: 5.a scrub ok 2024-09-21T14:02:57.893 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:57 smithi049 ceph-mon[30080]: pgmap v714: 158 pgs: 2 creating+peering, 14 unknown, 142 active+clean; 425 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 790 B/s rd, 0 op/s 2024-09-21T14:02:57.893 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:57 smithi049 ceph-mon[30080]: 4.12 scrub starts 2024-09-21T14:02:57.893 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:57 smithi049 ceph-mon[30080]: 4.12 scrub ok 2024-09-21T14:02:57.893 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:57 smithi049 ceph-mon[30080]: 3.1f scrub starts 2024-09-21T14:02:57.893 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:57 smithi049 ceph-mon[30080]: 3.1f scrub ok 2024-09-21T14:02:57.893 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:57 smithi049 ceph-mon[30080]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T14:02:57.894 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:57 smithi049 ceph-mon[30080]: osdmap e382: 8 total, 8 up, 8 in 2024-09-21T14:02:57.894 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:57 smithi049 ceph-mon[30080]: 5.a scrub starts 2024-09-21T14:02:57.894 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:57 smithi049 ceph-mon[30080]: 5.a scrub ok 2024-09-21T14:02:57.897 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:57.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:57.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:57.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:57 smithi187 ceph-mon[33032]: pgmap v714: 158 pgs: 2 creating+peering, 14 unknown, 142 active+clean; 425 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 790 B/s rd, 0 op/s 2024-09-21T14:02:57.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:57 smithi187 ceph-mon[33032]: 4.12 scrub starts 2024-09-21T14:02:57.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:57 smithi187 ceph-mon[33032]: 4.12 scrub ok 2024-09-21T14:02:57.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:57 smithi187 ceph-mon[33032]: 3.1f scrub starts 2024-09-21T14:02:57.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:57 smithi187 ceph-mon[33032]: 3.1f scrub ok 2024-09-21T14:02:57.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:57 smithi187 ceph-mon[33032]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T14:02:57.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:57 smithi187 ceph-mon[33032]: osdmap e382: 8 total, 8 up, 8 in 2024-09-21T14:02:57.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:57 smithi187 ceph-mon[33032]: 5.a scrub starts 2024-09-21T14:02:57.947 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:57 smithi187 ceph-mon[33032]: 5.a scrub ok 2024-09-21T14:02:57.999 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:58.022 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:58.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:58.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:58.124 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:02:58.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:58.151 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:58.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:02:58.230 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T14:02:58.251 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:02:58.276 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:58.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:58.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:02:58.377 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:02:58.387 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:02:58.387 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:02:58.401 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:58.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:58.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:02:58.420 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:02:58.420 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:02:58.437 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:02:58.437 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:02:58.454 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:02:58.454 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:02:58.471 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:02:58.471 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:02:58.490 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:02:58.490 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:02:58.502 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:02:58.511 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:02:58.512 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:02:58.526 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:58.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:58.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:02:58.529 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:02:58.529 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:02:58.628 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:58.651 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:58.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:58.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:58.753 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:02:58.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:58 smithi049 ceph-mon[40489]: 6.11 scrub starts 2024-09-21T14:02:58.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:58 smithi049 ceph-mon[40489]: 6.11 scrub ok 2024-09-21T14:02:58.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:58 smithi049 ceph-mon[40489]: 4.4 scrub starts 2024-09-21T14:02:58.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:58 smithi049 ceph-mon[40489]: 4.4 scrub ok 2024-09-21T14:02:58.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:58 smithi049 ceph-mon[40489]: osdmap e383: 8 total, 8 up, 8 in 2024-09-21T14:02:58.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:58 smithi049 ceph-mon[40489]: 4.6 scrub starts 2024-09-21T14:02:58.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:58 smithi049 ceph-mon[40489]: 4.6 scrub ok 2024-09-21T14:02:58.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:58 smithi049 ceph-mon[40489]: 3.14 scrub starts 2024-09-21T14:02:58.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:58 smithi049 ceph-mon[40489]: 3.14 scrub ok 2024-09-21T14:02:58.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:58 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3938191916' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:02:58.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:58 smithi049 ceph-mon[30080]: 6.11 scrub starts 2024-09-21T14:02:58.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:58 smithi049 ceph-mon[30080]: 6.11 scrub ok 2024-09-21T14:02:58.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:58 smithi049 ceph-mon[30080]: 4.4 scrub starts 2024-09-21T14:02:58.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:58 smithi049 ceph-mon[30080]: 4.4 scrub ok 2024-09-21T14:02:58.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:58 smithi049 ceph-mon[30080]: osdmap e383: 8 total, 8 up, 8 in 2024-09-21T14:02:58.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:58 smithi049 ceph-mon[30080]: 4.6 scrub starts 2024-09-21T14:02:58.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:58 smithi049 ceph-mon[30080]: 4.6 scrub ok 2024-09-21T14:02:58.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:58 smithi049 ceph-mon[30080]: 3.14 scrub starts 2024-09-21T14:02:58.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:58 smithi049 ceph-mon[30080]: 3.14 scrub ok 2024-09-21T14:02:58.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:58 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3938191916' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:02:58.775 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:58.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:58.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:02:58.785 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T14:02:58.787 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-09-21T14:02:58.877 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:58.900 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:58.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:58.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:58.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:58 smithi187 ceph-mon[33032]: 6.11 scrub starts 2024-09-21T14:02:58.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:58 smithi187 ceph-mon[33032]: 6.11 scrub ok 2024-09-21T14:02:58.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:58 smithi187 ceph-mon[33032]: 4.4 scrub starts 2024-09-21T14:02:58.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:58 smithi187 ceph-mon[33032]: 4.4 scrub ok 2024-09-21T14:02:58.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:58 smithi187 ceph-mon[33032]: osdmap e383: 8 total, 8 up, 8 in 2024-09-21T14:02:58.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:58 smithi187 ceph-mon[33032]: 4.6 scrub starts 2024-09-21T14:02:58.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:58 smithi187 ceph-mon[33032]: 4.6 scrub ok 2024-09-21T14:02:58.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:58 smithi187 ceph-mon[33032]: 3.14 scrub starts 2024-09-21T14:02:58.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:58 smithi187 ceph-mon[33032]: 3.14 scrub ok 2024-09-21T14:02:58.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:58 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3938191916' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:02:59.002 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:02:59.029 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:59.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:59.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:02:59.131 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:02:59.154 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:59.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:59.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:02:59.256 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:59.278 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:59.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:59.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:59.380 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:02:59.402 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:59.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:59.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:02:59.504 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:59.527 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:59.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:59.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:59.616 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":717,"stamp":"2024-09-21T14:02:57.995002+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":381827,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":1393,"num_read_kb":1212,"num_write":795,"num_write_kb":1081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":169,"num_bytes_recovered":1032962,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1671,"ondisk_log_size":1671,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":434,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":3045604,"kb_used_data":48812,"kb_used_omap":23,"kb_used_meta":2996584,"kb_avail":746915612,"statfs":{"total":767960285184,"available":764841586688,"internally_reserved":0,"allocated":49983488,"data_stored":43032640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24148,"internal_metadata":3068502444},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-5,"num_read_kb":-15,"num_write":-10,"num_write_kb":-148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":-53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001650"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":578,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.550979+0000","last_change":"2024-09-21T14:01:41.189741+0000","last_active":"2024-09-21T14:02:55.550979+0000","last_peered":"2024-09-21T14:02:55.550979+0000","last_clean":"2024-09-21T14:02:55.550979+0000","last_became_active":"2024-09-21T14:01:41.189398+0000","last_became_peered":"2024-09-21T14:01:41.189398+0000","last_unstale":"2024-09-21T14:02:55.550979+0000","last_undegraded":"2024-09-21T14:02:55.550979+0000","last_fullsized":"2024-09-21T14:02:55.550979+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:55.873301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:55.873301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"290'15","reported_seq":751,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.708857+0000","last_change":"2024-09-21T14:01:34.785322+0000","last_active":"2024-09-21T14:02:50.708857+0000","last_peered":"2024-09-21T14:02:50.708857+0000","last_clean":"2024-09-21T14:02:50.708857+0000","last_became_active":"2024-09-21T13:58:17.771220+0000","last_became_peered":"2024-09-21T13:58:17.771220+0000","last_unstale":"2024-09-21T14:02:50.708857+0000","last_undegraded":"2024-09-21T14:02:50.708857+0000","last_fullsized":"2024-09-21T14:02:50.708857+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'15","last_scrub_stamp":"2024-09-21T14:01:34.785267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:34.785267+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":692,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.093693+0000","last_change":"2024-09-21T14:02:29.205437+0000","last_active":"2024-09-21T14:02:50.093693+0000","last_peered":"2024-09-21T14:02:50.093693+0000","last_clean":"2024-09-21T14:02:50.093693+0000","last_became_active":"2024-09-21T14:01:41.202168+0000","last_became_peered":"2024-09-21T14:01:41.202168+0000","last_unstale":"2024-09-21T14:02:50.093693+0000","last_undegraded":"2024-09-21T14:02:50.093693+0000","last_fullsized":"2024-09-21T14:02:50.093693+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:29.205343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:02:29.205343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:29.369063+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":668,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.709061+0000","last_change":"2024-09-21T14:01:41.203762+0000","last_active":"2024-09-21T14:02:50.709061+0000","last_peered":"2024-09-21T14:02:50.709061+0000","last_clean":"2024-09-21T14:02:50.709061+0000","last_became_active":"2024-09-21T14:01:41.203275+0000","last_became_peered":"2024-09-21T14:01:41.203275+0000","last_unstale":"2024-09-21T14:02:50.709061+0000","last_undegraded":"2024-09-21T14:02:50.709061+0000","last_fullsized":"2024-09-21T14:02:50.709061+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:00.320510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:00.320510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:46.689692+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":640,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.551045+0000","last_change":"2024-09-21T13:59:32.208701+0000","last_active":"2024-09-21T14:02:55.551045+0000","last_peered":"2024-09-21T14:02:55.551045+0000","last_clean":"2024-09-21T14:02:55.551045+0000","last_became_active":"2024-09-21T13:59:32.203340+0000","last_became_peered":"2024-09-21T13:59:32.203340+0000","last_unstale":"2024-09-21T14:02:55.551045+0000","last_undegraded":"2024-09-21T14:02:55.551045+0000","last_fullsized":"2024-09-21T14:02:55.551045+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:34.696780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:34.696780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:44.274358+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":652,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.548028+0000","last_change":"2024-09-21T13:58:17.778322+0000","last_active":"2024-09-21T14:02:55.548028+0000","last_peered":"2024-09-21T14:02:55.548028+0000","last_clean":"2024-09-21T14:02:55.548028+0000","last_became_active":"2024-09-21T13:58:17.777893+0000","last_became_peered":"2024-09-21T13:58:17.777893+0000","last_unstale":"2024-09-21T14:02:55.548028+0000","last_undegraded":"2024-09-21T14:02:55.548028+0000","last_fullsized":"2024-09-21T14:02:55.548028+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:07.001135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:55:07.001135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:22.269018+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"290'27","reported_seq":626,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.550738+0000","last_change":"2024-09-21T13:58:11.718421+0000","last_active":"2024-09-21T14:02:55.550738+0000","last_peered":"2024-09-21T14:02:55.550738+0000","last_clean":"2024-09-21T14:02:55.550738+0000","last_became_active":"2024-09-21T13:57:12.703469+0000","last_became_peered":"2024-09-21T13:57:12.703469+0000","last_unstale":"2024-09-21T14:02:55.550738+0000","last_undegraded":"2024-09-21T14:02:55.550738+0000","last_fullsized":"2024-09-21T14:02:55.550738+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:58:11.718334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:11.718334+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":691,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.708852+0000","last_change":"2024-09-21T14:01:32.717551+0000","last_active":"2024-09-21T14:02:50.708852+0000","last_peered":"2024-09-21T14:02:50.708852+0000","last_clean":"2024-09-21T14:02:50.708852+0000","last_became_active":"2024-09-21T13:58:17.773464+0000","last_became_peered":"2024-09-21T13:58:17.773464+0000","last_unstale":"2024-09-21T14:02:50.708852+0000","last_undegraded":"2024-09-21T14:02:50.708852+0000","last_fullsized":"2024-09-21T14:02:50.708852+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:32.717480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:01:32.717480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:02.205485+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":645,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.550123+0000","last_change":"2024-09-21T14:01:41.194807+0000","last_active":"2024-09-21T14:02:55.550123+0000","last_peered":"2024-09-21T14:02:55.550123+0000","last_clean":"2024-09-21T14:02:55.550123+0000","last_became_active":"2024-09-21T14:01:41.194517+0000","last_became_peered":"2024-09-21T14:01:41.194517+0000","last_unstale":"2024-09-21T14:02:55.550123+0000","last_undegraded":"2024-09-21T14:02:55.550123+0000","last_fullsized":"2024-09-21T14:02:55.550123+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:31.127926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:31.127926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"290'15","reported_seq":717,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.709283+0000","last_change":"2024-09-21T14:00:02.475761+0000","last_active":"2024-09-21T14:02:50.709283+0000","last_peered":"2024-09-21T14:02:50.709283+0000","last_clean":"2024-09-21T14:02:50.709283+0000","last_became_active":"2024-09-21T13:57:12.681408+0000","last_became_peered":"2024-09-21T13:57:12.681408+0000","last_unstale":"2024-09-21T14:02:50.709283+0000","last_undegraded":"2024-09-21T14:02:50.709283+0000","last_fullsized":"2024-09-21T14:02:50.709283+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'15","last_scrub_stamp":"2024-09-21T14:00:02.475661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:00:02.475661+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:43.713097+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":586,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.549683+0000","last_change":"2024-09-21T14:01:41.188585+0000","last_active":"2024-09-21T14:02:55.549683+0000","last_peered":"2024-09-21T14:02:55.549683+0000","last_clean":"2024-09-21T14:02:55.549683+0000","last_became_active":"2024-09-21T14:01:41.188248+0000","last_became_peered":"2024-09-21T14:01:41.188248+0000","last_unstale":"2024-09-21T14:02:55.549683+0000","last_undegraded":"2024-09-21T14:02:55.549683+0000","last_fullsized":"2024-09-21T14:02:55.549683+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:21.284048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:56:21.284048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":622,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.094288+0000","last_change":"2024-09-21T13:59:32.202128+0000","last_active":"2024-09-21T14:02:50.094288+0000","last_peered":"2024-09-21T14:02:50.094288+0000","last_clean":"2024-09-21T14:02:50.094288+0000","last_became_active":"2024-09-21T13:59:32.201158+0000","last_became_peered":"2024-09-21T13:59:32.201158+0000","last_unstale":"2024-09-21T14:02:50.094288+0000","last_undegraded":"2024-09-21T14:02:50.094288+0000","last_fullsized":"2024-09-21T14:02:50.094288+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:25.792985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:54:25.792985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:49.267653+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":622,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.549746+0000","last_change":"2024-09-21T14:01:41.189797+0000","last_active":"2024-09-21T14:02:55.549746+0000","last_peered":"2024-09-21T14:02:55.549746+0000","last_clean":"2024-09-21T14:02:55.549746+0000","last_became_active":"2024-09-21T14:01:41.189373+0000","last_became_peered":"2024-09-21T14:01:41.189373+0000","last_unstale":"2024-09-21T14:02:55.549746+0000","last_undegraded":"2024-09-21T14:02:55.549746+0000","last_fullsized":"2024-09-21T14:02:55.549746+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:54.328275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:54.328275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"290'7","reported_seq":742,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.709200+0000","last_change":"2024-09-21T14:01:57.563250+0000","last_active":"2024-09-21T14:02:50.709200+0000","last_peered":"2024-09-21T14:02:50.709200+0000","last_clean":"2024-09-21T14:02:50.709200+0000","last_became_active":"2024-09-21T13:59:09.063053+0000","last_became_peered":"2024-09-21T13:59:09.063053+0000","last_unstale":"2024-09-21T14:02:50.709200+0000","last_undegraded":"2024-09-21T14:02:50.709200+0000","last_fullsized":"2024-09-21T14:02:50.709200+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'7","last_scrub_stamp":"2024-09-21T14:01:57.563184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:57.563184+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:19.737603+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":615,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.551151+0000","last_change":"2024-09-21T13:59:32.204565+0000","last_active":"2024-09-21T14:02:55.551151+0000","last_peered":"2024-09-21T14:02:55.551151+0000","last_clean":"2024-09-21T14:02:55.551151+0000","last_became_active":"2024-09-21T13:59:32.204174+0000","last_became_peered":"2024-09-21T13:59:32.204174+0000","last_unstale":"2024-09-21T14:02:55.551151+0000","last_undegraded":"2024-09-21T14:02:55.551151+0000","last_fullsized":"2024-09-21T14:02:55.551151+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:06.722293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:06.722293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":656,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.549465+0000","last_change":"2024-09-21T14:01:41.199085+0000","last_active":"2024-09-21T14:02:55.549465+0000","last_peered":"2024-09-21T14:02:55.549465+0000","last_clean":"2024-09-21T14:02:55.549465+0000","last_became_active":"2024-09-21T14:01:41.198672+0000","last_became_peered":"2024-09-21T14:01:41.198672+0000","last_unstale":"2024-09-21T14:02:55.549465+0000","last_undegraded":"2024-09-21T14:02:55.549465+0000","last_fullsized":"2024-09-21T14:02:55.549465+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:44.376215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:44.376215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":710,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.709579+0000","last_change":"2024-09-21T14:01:53.460257+0000","last_active":"2024-09-21T14:02:50.709579+0000","last_peered":"2024-09-21T14:02:50.709579+0000","last_clean":"2024-09-21T14:02:50.709579+0000","last_became_active":"2024-09-21T14:01:41.200360+0000","last_became_peered":"2024-09-21T14:01:41.200360+0000","last_unstale":"2024-09-21T14:02:50.709579+0000","last_undegraded":"2024-09-21T14:02:50.709579+0000","last_fullsized":"2024-09-21T14:02:50.709579+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:53.460180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:01:53.460180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:15.802657+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":628,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.709426+0000","last_change":"2024-09-21T14:02:36.679396+0000","last_active":"2024-09-21T14:02:50.709426+0000","last_peered":"2024-09-21T14:02:50.709426+0000","last_clean":"2024-09-21T14:02:50.709426+0000","last_became_active":"2024-09-21T13:59:32.204749+0000","last_became_peered":"2024-09-21T13:59:32.204749+0000","last_unstale":"2024-09-21T14:02:50.709426+0000","last_undegraded":"2024-09-21T14:02:50.709426+0000","last_fullsized":"2024-09-21T14:02:50.709426+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:36.679321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:02:36.679321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:43.067225+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"290'27","reported_seq":699,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T14:02:57.553061+0000","last_change":"2024-09-21T14:02:55.620755+0000","last_active":"2024-09-21T14:02:57.553061+0000","last_peered":"2024-09-21T14:02:57.553061+0000","last_clean":"2024-09-21T14:02:57.553061+0000","last_became_active":"2024-09-21T13:57:12.680783+0000","last_became_peered":"2024-09-21T13:57:12.680783+0000","last_unstale":"2024-09-21T14:02:57.553061+0000","last_undegraded":"2024-09-21T14:02:57.553061+0000","last_fullsized":"2024-09-21T14:02:57.553061+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'27","last_scrub_stamp":"2024-09-21T14:02:55.620700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.620700+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:01.085594+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":678,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.095158+0000","last_change":"2024-09-21T14:01:54.024745+0000","last_active":"2024-09-21T14:02:50.095158+0000","last_peered":"2024-09-21T14:02:50.095158+0000","last_clean":"2024-09-21T14:02:50.095158+0000","last_became_active":"2024-09-21T13:49:02.066796+0000","last_became_peered":"2024-09-21T13:49:02.066796+0000","last_unstale":"2024-09-21T14:02:50.095158+0000","last_undegraded":"2024-09-21T14:02:50.095158+0000","last_fullsized":"2024-09-21T14:02:50.095158+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:54.024702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:01:54.024702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:06.646444+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":632,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.551286+0000","last_change":"2024-09-21T14:01:41.191486+0000","last_active":"2024-09-21T14:02:55.551286+0000","last_peered":"2024-09-21T14:02:55.551286+0000","last_clean":"2024-09-21T14:02:55.551286+0000","last_became_active":"2024-09-21T14:01:41.191116+0000","last_became_peered":"2024-09-21T14:01:41.191116+0000","last_unstale":"2024-09-21T14:02:55.551286+0000","last_undegraded":"2024-09-21T14:02:55.551286+0000","last_fullsized":"2024-09-21T14:02:55.551286+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:57.962266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:57.962266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:08.315476+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":688,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.709410+0000","last_change":"2024-09-21T14:00:51.495433+0000","last_active":"2024-09-21T14:02:50.709410+0000","last_peered":"2024-09-21T14:02:50.709410+0000","last_clean":"2024-09-21T14:02:50.709410+0000","last_became_active":"2024-09-21T13:52:04.316016+0000","last_became_peered":"2024-09-21T13:52:04.316016+0000","last_unstale":"2024-09-21T14:02:50.709410+0000","last_undegraded":"2024-09-21T14:02:50.709410+0000","last_fullsized":"2024-09-21T14:02:50.709410+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:00:51.495368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:00:51.495368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:10.415465+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"290'18","reported_seq":638,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.548177+0000","last_change":"2024-09-21T13:59:23.041396+0000","last_active":"2024-09-21T14:02:55.548177+0000","last_peered":"2024-09-21T14:02:55.548177+0000","last_clean":"2024-09-21T14:02:55.548177+0000","last_became_active":"2024-09-21T13:57:12.684454+0000","last_became_peered":"2024-09-21T13:57:12.684454+0000","last_unstale":"2024-09-21T14:02:55.548177+0000","last_undegraded":"2024-09-21T14:02:55.548177+0000","last_fullsized":"2024-09-21T14:02:55.548177+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'18","last_scrub_stamp":"2024-09-21T13:59:23.041311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:23.041311+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":675,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.709257+0000","last_change":"2024-09-21T13:58:16.393763+0000","last_active":"2024-09-21T14:02:50.709257+0000","last_peered":"2024-09-21T14:02:50.709257+0000","last_clean":"2024-09-21T14:02:50.709257+0000","last_became_active":"2024-09-21T13:57:12.681897+0000","last_became_peered":"2024-09-21T13:57:12.681897+0000","last_unstale":"2024-09-21T14:02:50.709257+0000","last_undegraded":"2024-09-21T14:02:50.709257+0000","last_fullsized":"2024-09-21T14:02:50.709257+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:16.393696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:16.393696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:37.708367+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":599,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.552199+0000","last_change":"2024-09-21T14:01:41.182300+0000","last_active":"2024-09-21T14:02:55.552199+0000","last_peered":"2024-09-21T14:02:55.552199+0000","last_clean":"2024-09-21T14:02:55.552199+0000","last_became_active":"2024-09-21T14:01:41.181946+0000","last_became_peered":"2024-09-21T14:01:41.181946+0000","last_unstale":"2024-09-21T14:02:55.552199+0000","last_undegraded":"2024-09-21T14:02:55.552199+0000","last_fullsized":"2024-09-21T14:02:55.552199+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T13:57:26.536684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:26.536684+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":698,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.093841+0000","last_change":"2024-09-21T14:01:58.994867+0000","last_active":"2024-09-21T14:02:50.093841+0000","last_peered":"2024-09-21T14:02:50.093841+0000","last_clean":"2024-09-21T14:02:50.093841+0000","last_became_active":"2024-09-21T13:57:12.685742+0000","last_became_peered":"2024-09-21T13:57:12.685742+0000","last_unstale":"2024-09-21T14:02:50.093841+0000","last_undegraded":"2024-09-21T14:02:50.093841+0000","last_fullsized":"2024-09-21T14:02:50.093841+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T14:01:58.994811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:01:58.994811+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:17.687542+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"290'20","reported_seq":579,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.554527+0000","last_change":"2024-09-21T13:59:32.242771+0000","last_active":"2024-09-21T14:02:55.554527+0000","last_peered":"2024-09-21T14:02:55.554527+0000","last_clean":"2024-09-21T14:02:55.554527+0000","last_became_active":"2024-09-21T13:59:32.213865+0000","last_became_peered":"2024-09-21T13:59:32.213865+0000","last_unstale":"2024-09-21T14:02:55.554527+0000","last_undegraded":"2024-09-21T14:02:55.554527+0000","last_fullsized":"2024-09-21T14:02:55.554527+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:57:27.578738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:57:27.578738+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"300'12","reported_seq":315,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:51.647351+0000","last_change":"2024-09-21T13:59:36.012775+0000","last_active":"2024-09-21T14:02:51.647351+0000","last_peered":"2024-09-21T14:02:51.647351+0000","last_clean":"2024-09-21T14:02:51.647351+0000","last_became_active":"2024-09-21T13:59:32.205051+0000","last_became_peered":"2024-09-21T13:59:32.205051+0000","last_unstale":"2024-09-21T14:02:51.647351+0000","last_undegraded":"2024-09-21T14:02:51.647351+0000","last_fullsized":"2024-09-21T14:02:51.647351+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'12","last_scrub_stamp":"2024-09-21T13:59:36.012693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:36.012693+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:20.587646+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-21T14:02:55.537905+0000","last_change":"2024-09-21T14:02:55.537905+0000","last_active":"2024-09-21T14:02:55.537905+0000","last_peered":"2024-09-21T14:02:55.537905+0000","last_clean":"2024-09-21T14:02:55.537905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T14:02:55.537905+0000","last_undegraded":"2024-09-21T14:02:55.537905+0000","last_fullsized":"2024-09-21T14:02:55.537905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.a","version":"75'79","reported_seq":95,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.549975+0000","last_change":"2024-09-21T14:01:41.248772+0000","last_active":"2024-09-21T14:02:55.549975+0000","last_peered":"2024-09-21T14:02:55.549975+0000","last_clean":"2024-09-21T14:02:55.549975+0000","last_became_active":"2024-09-21T14:01:41.246888+0000","last_became_peered":"2024-09-21T14:01:41.246888+0000","last_unstale":"2024-09-21T14:02:55.549975+0000","last_undegraded":"2024-09-21T14:02:55.549975+0000","last_fullsized":"2024-09-21T14:02:55.549975+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:03:52.423493+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53669,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"290'27","reported_seq":737,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.094715+0000","last_change":"2024-09-21T13:58:45.984691+0000","last_active":"2024-09-21T14:02:50.094715+0000","last_peered":"2024-09-21T14:02:50.094715+0000","last_clean":"2024-09-21T14:02:50.094715+0000","last_became_active":"2024-09-21T13:57:12.695609+0000","last_became_peered":"2024-09-21T13:57:12.695609+0000","last_unstale":"2024-09-21T14:02:50.094715+0000","last_undegraded":"2024-09-21T14:02:50.094715+0000","last_fullsized":"2024-09-21T14:02:50.094715+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:58:45.984575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:45.984575+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":624,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.546611+0000","last_change":"2024-09-21T13:59:32.205723+0000","last_active":"2024-09-21T14:02:55.546611+0000","last_peered":"2024-09-21T14:02:55.546611+0000","last_clean":"2024-09-21T14:02:55.546611+0000","last_became_active":"2024-09-21T13:59:32.205350+0000","last_became_peered":"2024-09-21T13:59:32.205350+0000","last_unstale":"2024-09-21T14:02:55.546611+0000","last_undegraded":"2024-09-21T14:02:55.546611+0000","last_fullsized":"2024-09-21T14:02:55.546611+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:32.668851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:32.668851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:47.252073+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"346'10","reported_seq":871,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.709728+0000","last_change":"2024-09-21T14:01:55.521513+0000","last_active":"2024-09-21T14:02:50.709728+0000","last_peered":"2024-09-21T14:02:50.709728+0000","last_clean":"2024-09-21T14:02:50.709728+0000","last_became_active":"2024-09-21T14:01:41.200218+0000","last_became_peered":"2024-09-21T14:01:41.200218+0000","last_unstale":"2024-09-21T14:02:50.709728+0000","last_undegraded":"2024-09-21T14:02:50.709728+0000","last_fullsized":"2024-09-21T14:02:50.709728+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'10","last_scrub_stamp":"2024-09-21T14:01:55.521438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:01:55.521438+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:55.829821+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":551,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.549524+0000","last_change":"2024-09-21T14:01:41.207310+0000","last_active":"2024-09-21T14:02:55.549524+0000","last_peered":"2024-09-21T14:02:55.549524+0000","last_clean":"2024-09-21T14:02:55.549524+0000","last_became_active":"2024-09-21T14:01:41.206720+0000","last_became_peered":"2024-09-21T14:01:41.206720+0000","last_unstale":"2024-09-21T14:02:55.549524+0000","last_undegraded":"2024-09-21T14:02:55.549524+0000","last_fullsized":"2024-09-21T14:02:55.549524+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:52.244483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:52.244483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-21T14:02:55.537905+0000","last_change":"2024-09-21T14:02:55.537905+0000","last_active":"2024-09-21T14:02:55.537905+0000","last_peered":"2024-09-21T14:02:55.537905+0000","last_clean":"2024-09-21T14:02:55.537905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T14:02:55.537905+0000","last_undegraded":"2024-09-21T14:02:55.537905+0000","last_fullsized":"2024-09-21T14:02:55.537905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.5","version":"75'79","reported_seq":147,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.709637+0000","last_change":"2024-09-21T14:02:06.395083+0000","last_active":"2024-09-21T14:02:50.709637+0000","last_peered":"2024-09-21T14:02:50.709637+0000","last_clean":"2024-09-21T14:02:50.709637+0000","last_became_active":"2024-09-21T14:00:41.137756+0000","last_became_peered":"2024-09-21T14:00:41.137756+0000","last_unstale":"2024-09-21T14:02:50.709637+0000","last_undegraded":"2024-09-21T14:02:50.709637+0000","last_fullsized":"2024-09-21T14:02:50.709637+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:02:06.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:02:06.395011+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:29.136654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"290'30","reported_seq":648,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.547124+0000","last_change":"2024-09-21T13:59:32.246090+0000","last_active":"2024-09-21T14:02:55.547124+0000","last_peered":"2024-09-21T14:02:55.547124+0000","last_clean":"2024-09-21T14:02:55.547124+0000","last_became_active":"2024-09-21T13:59:32.205887+0000","last_became_peered":"2024-09-21T13:59:32.205887+0000","last_unstale":"2024-09-21T14:02:55.547124+0000","last_undegraded":"2024-09-21T14:02:55.547124+0000","last_fullsized":"2024-09-21T14:02:55.547124+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'18","last_scrub_stamp":"2024-09-21T13:54:19.001030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:19.001030+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:43.319659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":576,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.551590+0000","last_change":"2024-09-21T13:59:32.212559+0000","last_active":"2024-09-21T14:02:55.551590+0000","last_peered":"2024-09-21T14:02:55.551590+0000","last_clean":"2024-09-21T14:02:55.551590+0000","last_became_active":"2024-09-21T13:59:32.211004+0000","last_became_peered":"2024-09-21T13:59:32.211004+0000","last_unstale":"2024-09-21T14:02:55.551590+0000","last_undegraded":"2024-09-21T14:02:55.551590+0000","last_fullsized":"2024-09-21T14:02:55.551590+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:25.736461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:25.736461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:48.292957+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":692,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.708845+0000","last_change":"2024-09-21T13:59:34.293330+0000","last_active":"2024-09-21T14:02:50.708845+0000","last_peered":"2024-09-21T14:02:50.708845+0000","last_clean":"2024-09-21T14:02:50.708845+0000","last_became_active":"2024-09-21T13:59:32.220304+0000","last_became_peered":"2024-09-21T13:59:32.220304+0000","last_unstale":"2024-09-21T14:02:50.708845+0000","last_undegraded":"2024-09-21T14:02:50.708845+0000","last_fullsized":"2024-09-21T14:02:50.708845+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:34.292892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:34.292892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:15.468420+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":672,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.709279+0000","last_change":"2024-09-21T13:59:23.334230+0000","last_active":"2024-09-21T14:02:50.709279+0000","last_peered":"2024-09-21T14:02:50.709279+0000","last_clean":"2024-09-21T14:02:50.709279+0000","last_became_active":"2024-09-21T13:57:12.680372+0000","last_became_peered":"2024-09-21T13:57:12.680372+0000","last_unstale":"2024-09-21T14:02:50.709279+0000","last_undegraded":"2024-09-21T14:02:50.709279+0000","last_fullsized":"2024-09-21T14:02:50.709279+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:23.334162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:23.334162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:17.515611+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-21T14:02:55.537905+0000","last_change":"2024-09-21T14:02:55.537905+0000","last_active":"2024-09-21T14:02:55.537905+0000","last_peered":"2024-09-21T14:02:55.537905+0000","last_clean":"2024-09-21T14:02:55.537905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T14:02:55.537905+0000","last_undegraded":"2024-09-21T14:02:55.537905+0000","last_fullsized":"2024-09-21T14:02:55.537905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.4","version":"75'79","reported_seq":152,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.549336+0000","last_change":"2024-09-21T14:01:42.177646+0000","last_active":"2024-09-21T14:02:55.549336+0000","last_peered":"2024-09-21T14:02:55.549336+0000","last_clean":"2024-09-21T14:02:55.549336+0000","last_became_active":"2024-09-21T14:01:42.173042+0000","last_became_peered":"2024-09-21T14:01:42.173042+0000","last_unstale":"2024-09-21T14:02:55.549336+0000","last_undegraded":"2024-09-21T14:02:55.549336+0000","last_fullsized":"2024-09-21T14:02:55.549336+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:03:42.356436+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":709030,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"290'28","reported_seq":781,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.708940+0000","last_change":"2024-09-21T14:02:02.471226+0000","last_active":"2024-09-21T14:02:50.708940+0000","last_peered":"2024-09-21T14:02:50.708940+0000","last_clean":"2024-09-21T14:02:50.708940+0000","last_became_active":"2024-09-21T14:01:41.200859+0000","last_became_peered":"2024-09-21T14:01:41.200859+0000","last_unstale":"2024-09-21T14:02:50.708940+0000","last_undegraded":"2024-09-21T14:02:50.708940+0000","last_fullsized":"2024-09-21T14:02:50.708940+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'28","last_scrub_stamp":"2024-09-21T14:02:02.471133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:02.471133+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:14.278651+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":680,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.096545+0000","last_change":"2024-09-21T13:57:38.046718+0000","last_active":"2024-09-21T14:02:50.096545+0000","last_peered":"2024-09-21T14:02:50.096545+0000","last_clean":"2024-09-21T14:02:50.096545+0000","last_became_active":"2024-09-21T13:48:58.053533+0000","last_became_peered":"2024-09-21T13:48:58.053533+0000","last_unstale":"2024-09-21T14:02:50.096545+0000","last_undegraded":"2024-09-21T14:02:50.096545+0000","last_fullsized":"2024-09-21T14:02:50.096545+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:57:38.046627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:38.046627+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:49.580099+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"241'10","reported_seq":725,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.547365+0000","last_change":"2024-09-21T13:58:06.789629+0000","last_active":"2024-09-21T14:02:55.547365+0000","last_peered":"2024-09-21T14:02:55.547365+0000","last_clean":"2024-09-21T14:02:55.547365+0000","last_became_active":"2024-09-21T13:57:12.686651+0000","last_became_peered":"2024-09-21T13:57:12.686651+0000","last_unstale":"2024-09-21T14:02:55.547365+0000","last_undegraded":"2024-09-21T14:02:55.547365+0000","last_fullsized":"2024-09-21T14:02:55.547365+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'10","last_scrub_stamp":"2024-09-21T13:58:06.789560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:06.789560+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":640,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.551425+0000","last_change":"2024-09-21T13:59:32.210257+0000","last_active":"2024-09-21T14:02:55.551425+0000","last_peered":"2024-09-21T14:02:55.551425+0000","last_clean":"2024-09-21T14:02:55.551425+0000","last_became_active":"2024-09-21T13:59:32.209615+0000","last_became_peered":"2024-09-21T13:59:32.209615+0000","last_unstale":"2024-09-21T14:02:55.551425+0000","last_undegraded":"2024-09-21T14:02:55.551425+0000","last_fullsized":"2024-09-21T14:02:55.551425+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:37.548064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:37.548064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":1,"reported_epoch":381,"state":"creating+peering","last_fresh":"2024-09-21T14:02:55.550740+0000","last_change":"2024-09-21T14:02:55.550740+0000","last_active":"2024-09-21T14:02:55.537905+0000","last_peered":"2024-09-21T14:02:55.537905+0000","last_clean":"2024-09-21T14:02:55.537905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T14:02:55.550740+0000","last_undegraded":"2024-09-21T14:02:55.550740+0000","last_fullsized":"2024-09-21T14:02:55.550740+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"75'79","reported_seq":720,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:51.647081+0000","last_change":"2024-09-21T14:02:05.693250+0000","last_active":"2024-09-21T14:02:51.647081+0000","last_peered":"2024-09-21T14:02:51.647081+0000","last_clean":"2024-09-21T14:02:51.647081+0000","last_became_active":"2024-09-21T14:00:41.083425+0000","last_became_peered":"2024-09-21T14:00:41.083425+0000","last_unstale":"2024-09-21T14:02:51.647081+0000","last_undegraded":"2024-09-21T14:02:51.647081+0000","last_fullsized":"2024-09-21T14:02:51.647081+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:02:05.693193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:02:05.693193+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:28.620866+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"290'20","reported_seq":716,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.094742+0000","last_change":"2024-09-21T14:01:41.187743+0000","last_active":"2024-09-21T14:02:50.094742+0000","last_peered":"2024-09-21T14:02:50.094742+0000","last_clean":"2024-09-21T14:02:50.094742+0000","last_became_active":"2024-09-21T14:01:41.187060+0000","last_became_peered":"2024-09-21T14:01:41.187060+0000","last_unstale":"2024-09-21T14:02:50.094742+0000","last_undegraded":"2024-09-21T14:02:50.094742+0000","last_fullsized":"2024-09-21T14:02:50.094742+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:54:19.593611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:19.593611+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:47.290628+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"61'1","reported_seq":613,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.549625+0000","last_change":"2024-09-21T14:01:41.195580+0000","last_active":"2024-09-21T14:02:55.549625+0000","last_peered":"2024-09-21T14:02:55.549625+0000","last_clean":"2024-09-21T14:02:55.549625+0000","last_became_active":"2024-09-21T14:01:41.195268+0000","last_became_peered":"2024-09-21T14:01:41.195268+0000","last_unstale":"2024-09-21T14:02:55.549625+0000","last_undegraded":"2024-09-21T14:02:55.549625+0000","last_fullsized":"2024-09-21T14:02:55.549625+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:58:48.407734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:48.407734+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"346'15","reported_seq":308,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:51.646720+0000","last_change":"2024-09-21T14:01:41.209662+0000","last_active":"2024-09-21T14:02:51.646720+0000","last_peered":"2024-09-21T14:02:51.646720+0000","last_clean":"2024-09-21T14:02:51.646720+0000","last_became_active":"2024-09-21T14:01:41.209296+0000","last_became_peered":"2024-09-21T14:01:41.209296+0000","last_unstale":"2024-09-21T14:02:51.646720+0000","last_undegraded":"2024-09-21T14:02:51.646720+0000","last_fullsized":"2024-09-21T14:02:51.646720+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'12","last_scrub_stamp":"2024-09-21T13:57:30.133392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:30.133392+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:43.651875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":494,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.547260+0000","last_change":"2024-09-21T13:58:58.730439+0000","last_active":"2024-09-21T14:02:55.547260+0000","last_peered":"2024-09-21T14:02:55.547260+0000","last_clean":"2024-09-21T14:02:55.547260+0000","last_became_active":"2024-09-21T13:57:12.704897+0000","last_became_peered":"2024-09-21T13:57:12.704897+0000","last_unstale":"2024-09-21T14:02:55.547260+0000","last_undegraded":"2024-09-21T14:02:55.547260+0000","last_fullsized":"2024-09-21T14:02:55.547260+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:58.730368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:58.730368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-21T14:02:55.537905+0000","last_change":"2024-09-21T14:02:55.537905+0000","last_active":"2024-09-21T14:02:55.537905+0000","last_peered":"2024-09-21T14:02:55.537905+0000","last_clean":"2024-09-21T14:02:55.537905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T14:02:55.537905+0000","last_undegraded":"2024-09-21T14:02:55.537905+0000","last_fullsized":"2024-09-21T14:02:55.537905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.6","version":"75'79","reported_seq":720,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:51.646906+0000","last_change":"2024-09-21T14:02:03.737005+0000","last_active":"2024-09-21T14:02:51.646906+0000","last_peered":"2024-09-21T14:02:51.646906+0000","last_clean":"2024-09-21T14:02:51.646906+0000","last_became_active":"2024-09-21T14:00:41.073047+0000","last_became_peered":"2024-09-21T14:00:41.073047+0000","last_unstale":"2024-09-21T14:02:51.646906+0000","last_undegraded":"2024-09-21T14:02:51.646906+0000","last_fullsized":"2024-09-21T14:02:51.646906+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:02:03.736928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:02:03.736928+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:21.563427+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"290'29","reported_seq":619,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.549964+0000","last_change":"2024-09-21T14:01:41.188261+0000","last_active":"2024-09-21T14:02:55.549964+0000","last_peered":"2024-09-21T14:02:55.549964+0000","last_clean":"2024-09-21T14:02:55.549964+0000","last_became_active":"2024-09-21T14:01:41.186960+0000","last_became_peered":"2024-09-21T14:01:41.186960+0000","last_unstale":"2024-09-21T14:02:55.549964+0000","last_undegraded":"2024-09-21T14:02:55.549964+0000","last_fullsized":"2024-09-21T14:02:55.549964+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-21T13:56:55.319759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:56:55.319759+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":693,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.094555+0000","last_change":"2024-09-21T14:01:41.185706+0000","last_active":"2024-09-21T14:02:50.094555+0000","last_peered":"2024-09-21T14:02:50.094555+0000","last_clean":"2024-09-21T14:02:50.094555+0000","last_became_active":"2024-09-21T14:01:41.185400+0000","last_became_peered":"2024-09-21T14:01:41.185400+0000","last_unstale":"2024-09-21T14:02:50.094555+0000","last_undegraded":"2024-09-21T14:02:50.094555+0000","last_fullsized":"2024-09-21T14:02:50.094555+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:16.712990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:16.712990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:38.309423+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":261,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:51.646754+0000","last_change":"2024-09-21T14:01:52.671029+0000","last_active":"2024-09-21T14:02:51.646754+0000","last_peered":"2024-09-21T14:02:51.646754+0000","last_clean":"2024-09-21T14:02:51.646754+0000","last_became_active":"2024-09-21T14:01:41.193973+0000","last_became_peered":"2024-09-21T14:01:41.193973+0000","last_unstale":"2024-09-21T14:02:51.646754+0000","last_undegraded":"2024-09-21T14:02:51.646754+0000","last_fullsized":"2024-09-21T14:02:51.646754+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:52.670974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:01:52.670974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:02:58.750020+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":684,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.093595+0000","last_change":"2024-09-21T13:58:43.952014+0000","last_active":"2024-09-21T14:02:50.093595+0000","last_peered":"2024-09-21T14:02:50.093595+0000","last_clean":"2024-09-21T14:02:50.093595+0000","last_became_active":"2024-09-21T13:58:17.757628+0000","last_became_peered":"2024-09-21T13:58:17.757628+0000","last_unstale":"2024-09-21T14:02:50.093595+0000","last_undegraded":"2024-09-21T14:02:50.093595+0000","last_fullsized":"2024-09-21T14:02:50.093595+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:43.951941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:43.951941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-21T14:02:55.537905+0000","last_change":"2024-09-21T14:02:55.537905+0000","last_active":"2024-09-21T14:02:55.537905+0000","last_peered":"2024-09-21T14:02:55.537905+0000","last_clean":"2024-09-21T14:02:55.537905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T14:02:55.537905+0000","last_undegraded":"2024-09-21T14:02:55.537905+0000","last_fullsized":"2024-09-21T14:02:55.537905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.1","version":"75'79","reported_seq":95,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.549679+0000","last_change":"2024-09-21T14:01:41.199377+0000","last_active":"2024-09-21T14:02:55.549679+0000","last_peered":"2024-09-21T14:02:55.549679+0000","last_clean":"2024-09-21T14:02:55.549679+0000","last_became_active":"2024-09-21T14:01:41.198750+0000","last_became_peered":"2024-09-21T14:01:41.198750+0000","last_unstale":"2024-09-21T14:02:55.549679+0000","last_undegraded":"2024-09-21T14:02:55.549679+0000","last_fullsized":"2024-09-21T14:02:55.549679+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:03:47.451019+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"290'44","reported_seq":927,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.094439+0000","last_change":"2024-09-21T13:58:56.076153+0000","last_active":"2024-09-21T14:02:50.094439+0000","last_peered":"2024-09-21T14:02:50.094439+0000","last_clean":"2024-09-21T14:02:50.094439+0000","last_became_active":"2024-09-21T13:57:12.684690+0000","last_became_peered":"2024-09-21T13:57:12.684690+0000","last_unstale":"2024-09-21T14:02:50.094439+0000","last_undegraded":"2024-09-21T14:02:50.094439+0000","last_fullsized":"2024-09-21T14:02:50.094439+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'32","last_scrub_stamp":"2024-09-21T13:58:56.076074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:56.076074+0000","objects_scrubbed":10,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":700,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.709454+0000","last_change":"2024-09-21T14:01:58.524159+0000","last_active":"2024-09-21T14:02:50.709454+0000","last_peered":"2024-09-21T14:02:50.709454+0000","last_clean":"2024-09-21T14:02:50.709454+0000","last_became_active":"2024-09-21T13:58:17.770937+0000","last_became_peered":"2024-09-21T13:58:17.770937+0000","last_unstale":"2024-09-21T14:02:50.709454+0000","last_undegraded":"2024-09-21T14:02:50.709454+0000","last_fullsized":"2024-09-21T14:02:50.709454+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:58.524114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:01:58.524114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:07.944133+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":632,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.549885+0000","last_change":"2024-09-21T14:01:41.249061+0000","last_active":"2024-09-21T14:02:55.549885+0000","last_peered":"2024-09-21T14:02:55.549885+0000","last_clean":"2024-09-21T14:02:55.549885+0000","last_became_active":"2024-09-21T14:01:41.247642+0000","last_became_peered":"2024-09-21T14:01:41.247642+0000","last_unstale":"2024-09-21T14:02:55.549885+0000","last_undegraded":"2024-09-21T14:02:55.549885+0000","last_fullsized":"2024-09-21T14:02:55.549885+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-21T13:59:22.186393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:59:22.186393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":611,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.094977+0000","last_change":"2024-09-21T13:55:07.021128+0000","last_active":"2024-09-21T14:02:50.094977+0000","last_peered":"2024-09-21T14:02:50.094977+0000","last_clean":"2024-09-21T14:02:50.094977+0000","last_became_active":"2024-09-21T13:50:29.688249+0000","last_became_peered":"2024-09-21T13:50:29.688249+0000","last_unstale":"2024-09-21T14:02:50.094977+0000","last_undegraded":"2024-09-21T14:02:50.094977+0000","last_fullsized":"2024-09-21T14:02:50.094977+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:07.021059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:55:07.021059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":603,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.548792+0000","last_change":"2024-09-21T13:59:32.209357+0000","last_active":"2024-09-21T14:02:55.548792+0000","last_peered":"2024-09-21T14:02:55.548792+0000","last_clean":"2024-09-21T14:02:55.548792+0000","last_became_active":"2024-09-21T13:59:32.208980+0000","last_became_peered":"2024-09-21T13:59:32.208980+0000","last_unstale":"2024-09-21T14:02:55.548792+0000","last_undegraded":"2024-09-21T14:02:55.548792+0000","last_fullsized":"2024-09-21T14:02:55.548792+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:16.797227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:16.797227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":1,"reported_epoch":381,"state":"creating+peering","last_fresh":"2024-09-21T14:02:55.549007+0000","last_change":"2024-09-21T14:02:55.549007+0000","last_active":"2024-09-21T14:02:55.537905+0000","last_peered":"2024-09-21T14:02:55.537905+0000","last_clean":"2024-09-21T14:02:55.537905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T14:02:55.549007+0000","last_undegraded":"2024-09-21T14:02:55.549007+0000","last_fullsized":"2024-09-21T14:02:55.549007+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"75'79","reported_seq":728,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.551040+0000","last_change":"2024-09-21T14:00:41.077286+0000","last_active":"2024-09-21T14:02:55.551040+0000","last_peered":"2024-09-21T14:02:55.551040+0000","last_clean":"2024-09-21T14:02:55.551040+0000","last_became_active":"2024-09-21T14:00:41.076948+0000","last_became_peered":"2024-09-21T14:00:41.076948+0000","last_unstale":"2024-09-21T14:02:55.551040+0000","last_undegraded":"2024-09-21T14:02:55.551040+0000","last_fullsized":"2024-09-21T14:02:55.551040+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:03:51.730082+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"290'17","reported_seq":719,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.094743+0000","last_change":"2024-09-21T14:01:41.200809+0000","last_active":"2024-09-21T14:02:50.094743+0000","last_peered":"2024-09-21T14:02:50.094743+0000","last_clean":"2024-09-21T14:02:50.094743+0000","last_became_active":"2024-09-21T14:01:41.200365+0000","last_became_peered":"2024-09-21T14:01:41.200365+0000","last_unstale":"2024-09-21T14:02:50.094743+0000","last_undegraded":"2024-09-21T14:02:50.094743+0000","last_fullsized":"2024-09-21T14:02:50.094743+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'13","last_scrub_stamp":"2024-09-21T13:58:11.619848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:11.619848+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:39.361066+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":720,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.093433+0000","last_change":"2024-09-21T13:59:29.454265+0000","last_active":"2024-09-21T14:02:50.093433+0000","last_peered":"2024-09-21T14:02:50.093433+0000","last_clean":"2024-09-21T14:02:50.093433+0000","last_became_active":"2024-09-21T13:58:17.759671+0000","last_became_peered":"2024-09-21T13:58:17.759671+0000","last_unstale":"2024-09-21T14:02:50.093433+0000","last_undegraded":"2024-09-21T14:02:50.093433+0000","last_fullsized":"2024-09-21T14:02:50.093433+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:29.454085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:59:29.454085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":616,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T14:02:57.551700+0000","last_change":"2024-09-21T13:58:41.644876+0000","last_active":"2024-09-21T14:02:57.551700+0000","last_peered":"2024-09-21T14:02:57.551700+0000","last_clean":"2024-09-21T14:02:57.551700+0000","last_became_active":"2024-09-21T13:57:12.682147+0000","last_became_peered":"2024-09-21T13:57:12.682147+0000","last_unstale":"2024-09-21T14:02:57.551700+0000","last_undegraded":"2024-09-21T14:02:57.551700+0000","last_fullsized":"2024-09-21T14:02:57.551700+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:58:41.644789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:58:41.644789+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:13.796854+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":595,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T14:02:57.551946+0000","last_change":"2024-09-21T14:01:50.880840+0000","last_active":"2024-09-21T14:02:57.551946+0000","last_peered":"2024-09-21T14:02:57.551946+0000","last_clean":"2024-09-21T14:02:57.551946+0000","last_became_active":"2024-09-21T14:01:41.182000+0000","last_became_peered":"2024-09-21T14:01:41.182000+0000","last_unstale":"2024-09-21T14:02:57.551946+0000","last_undegraded":"2024-09-21T14:02:57.551946+0000","last_fullsized":"2024-09-21T14:02:57.551946+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:50.880471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:01:50.880471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:14.118692+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":648,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.554117+0000","last_change":"2024-09-21T13:59:32.222791+0000","last_active":"2024-09-21T14:02:55.554117+0000","last_peered":"2024-09-21T14:02:55.554117+0000","last_clean":"2024-09-21T14:02:55.554117+0000","last_became_active":"2024-09-21T13:59:32.221502+0000","last_became_peered":"2024-09-21T13:59:32.221502+0000","last_unstale":"2024-09-21T14:02:55.554117+0000","last_undegraded":"2024-09-21T14:02:55.554117+0000","last_fullsized":"2024-09-21T14:02:55.554117+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:14.673582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:14.673582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":14,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T14:02:57.550583+0000","last_change":"2024-09-21T14:02:56.559108+0000","last_active":"2024-09-21T14:02:57.550583+0000","last_peered":"2024-09-21T14:02:57.550583+0000","last_clean":"2024-09-21T14:02:57.550583+0000","last_became_active":"2024-09-21T14:02:56.558826+0000","last_became_peered":"2024-09-21T14:02:56.558826+0000","last_unstale":"2024-09-21T14:02:57.550583+0000","last_undegraded":"2024-09-21T14:02:57.550583+0000","last_fullsized":"2024-09-21T14:02:57.550583+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:15.358106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"75'79","reported_seq":131,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.093891+0000","last_change":"2024-09-21T14:00:41.071631+0000","last_active":"2024-09-21T14:02:50.093891+0000","last_peered":"2024-09-21T14:02:50.093891+0000","last_clean":"2024-09-21T14:02:50.093891+0000","last_became_active":"2024-09-21T14:00:41.071042+0000","last_became_peered":"2024-09-21T14:00:41.071042+0000","last_unstale":"2024-09-21T14:02:50.093891+0000","last_undegraded":"2024-09-21T14:02:50.093891+0000","last_fullsized":"2024-09-21T14:02:50.093891+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:03:42.335979+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"290'24","reported_seq":770,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.096349+0000","last_change":"2024-09-21T14:01:41.185105+0000","last_active":"2024-09-21T14:02:50.096349+0000","last_peered":"2024-09-21T14:02:50.096349+0000","last_clean":"2024-09-21T14:02:50.096349+0000","last_became_active":"2024-09-21T14:01:41.184728+0000","last_became_peered":"2024-09-21T14:01:41.184728+0000","last_unstale":"2024-09-21T14:02:50.096349+0000","last_undegraded":"2024-09-21T14:02:50.096349+0000","last_fullsized":"2024-09-21T14:02:50.096349+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'24","last_scrub_stamp":"2024-09-21T13:59:28.458768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:28.458768+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":621,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.709024+0000","last_change":"2024-09-21T13:59:59.450613+0000","last_active":"2024-09-21T14:02:50.709024+0000","last_peered":"2024-09-21T14:02:50.709024+0000","last_clean":"2024-09-21T14:02:50.709024+0000","last_became_active":"2024-09-21T13:57:12.681030+0000","last_became_peered":"2024-09-21T13:57:12.681030+0000","last_unstale":"2024-09-21T14:02:50.709024+0000","last_undegraded":"2024-09-21T14:02:50.709024+0000","last_fullsized":"2024-09-21T14:02:50.709024+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:59.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:59:59.450546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:40.683320+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":1386,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.551358+0000","last_change":"2024-09-21T13:59:32.218796+0000","last_active":"2024-09-21T14:02:55.551358+0000","last_peered":"2024-09-21T14:02:55.551358+0000","last_clean":"2024-09-21T14:02:55.551358+0000","last_became_active":"2024-09-21T13:59:32.203361+0000","last_became_peered":"2024-09-21T13:59:32.203361+0000","last_unstale":"2024-09-21T14:02:55.551358+0000","last_undegraded":"2024-09-21T14:02:55.551358+0000","last_fullsized":"2024-09-21T14:02:55.551358+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-21T13:56:29.617143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:56:29.617143+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047048000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":210,"num_read_kb":205,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":603,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.551064+0000","last_change":"2024-09-21T14:01:41.212595+0000","last_active":"2024-09-21T14:02:55.551064+0000","last_peered":"2024-09-21T14:02:55.551064+0000","last_clean":"2024-09-21T14:02:55.551064+0000","last_became_active":"2024-09-21T14:01:41.200531+0000","last_became_peered":"2024-09-21T14:01:41.200531+0000","last_unstale":"2024-09-21T14:02:55.551064+0000","last_undegraded":"2024-09-21T14:02:55.551064+0000","last_fullsized":"2024-09-21T14:02:55.551064+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:33.316230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:56:33.316230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":705,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.094056+0000","last_change":"2024-09-21T14:01:41.184939+0000","last_active":"2024-09-21T14:02:50.094056+0000","last_peered":"2024-09-21T14:02:50.094056+0000","last_clean":"2024-09-21T14:02:50.094056+0000","last_became_active":"2024-09-21T14:01:41.184664+0000","last_became_peered":"2024-09-21T14:01:41.184664+0000","last_unstale":"2024-09-21T14:02:50.094056+0000","last_undegraded":"2024-09-21T14:02:50.094056+0000","last_fullsized":"2024-09-21T14:02:50.094056+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:26.557002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:26.557002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-21T14:02:55.537905+0000","last_change":"2024-09-21T14:02:55.537905+0000","last_active":"2024-09-21T14:02:55.537905+0000","last_peered":"2024-09-21T14:02:55.537905+0000","last_clean":"2024-09-21T14:02:55.537905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T14:02:55.537905+0000","last_undegraded":"2024-09-21T14:02:55.537905+0000","last_fullsized":"2024-09-21T14:02:55.537905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.5","version":"300'32","reported_seq":361,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:51.646968+0000","last_change":"2024-09-21T14:00:47.130089+0000","last_active":"2024-09-21T14:02:51.646968+0000","last_peered":"2024-09-21T14:02:51.646968+0000","last_clean":"2024-09-21T14:02:51.646968+0000","last_became_active":"2024-09-21T13:59:32.204624+0000","last_became_peered":"2024-09-21T13:59:32.204624+0000","last_unstale":"2024-09-21T14:02:51.646968+0000","last_undegraded":"2024-09-21T14:02:51.646968+0000","last_fullsized":"2024-09-21T14:02:51.646968+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'32","last_scrub_stamp":"2024-09-21T14:00:47.129789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:00:47.129789+0000","objects_scrubbed":8,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:15.616969+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":661,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.547618+0000","last_change":"2024-09-21T14:01:41.190537+0000","last_active":"2024-09-21T14:02:55.547618+0000","last_peered":"2024-09-21T14:02:55.547618+0000","last_clean":"2024-09-21T14:02:55.547618+0000","last_became_active":"2024-09-21T14:01:41.190210+0000","last_became_peered":"2024-09-21T14:01:41.190210+0000","last_unstale":"2024-09-21T14:02:55.547618+0000","last_undegraded":"2024-09-21T14:02:55.547618+0000","last_fullsized":"2024-09-21T14:02:55.547618+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:57.772975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:57.772975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"75'79","reported_seq":722,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.551567+0000","last_change":"2024-09-21T14:00:39.339360+0000","last_active":"2024-09-21T14:02:55.551567+0000","last_peered":"2024-09-21T14:02:55.551567+0000","last_clean":"2024-09-21T14:02:55.551567+0000","last_became_active":"2024-09-21T14:00:39.339077+0000","last_became_peered":"2024-09-21T14:00:39.339077+0000","last_unstale":"2024-09-21T14:02:55.551567+0000","last_undegraded":"2024-09-21T14:02:55.551567+0000","last_fullsized":"2024-09-21T14:02:55.551567+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:03:49.763645+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":19,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53669,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":636,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.552416+0000","last_change":"2024-09-21T14:01:41.182186+0000","last_active":"2024-09-21T14:02:55.552416+0000","last_peered":"2024-09-21T14:02:55.552416+0000","last_clean":"2024-09-21T14:02:55.552416+0000","last_became_active":"2024-09-21T14:01:41.181899+0000","last_became_peered":"2024-09-21T14:01:41.181899+0000","last_unstale":"2024-09-21T14:02:55.552416+0000","last_undegraded":"2024-09-21T14:02:55.552416+0000","last_fullsized":"2024-09-21T14:02:55.552416+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:15.700409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:15.700409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":561,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.550766+0000","last_change":"2024-09-21T14:01:41.196509+0000","last_active":"2024-09-21T14:02:55.550766+0000","last_peered":"2024-09-21T14:02:55.550766+0000","last_clean":"2024-09-21T14:02:55.550766+0000","last_became_active":"2024-09-21T14:01:41.196243+0000","last_became_peered":"2024-09-21T14:01:41.196243+0000","last_unstale":"2024-09-21T14:02:55.550766+0000","last_undegraded":"2024-09-21T14:02:55.550766+0000","last_fullsized":"2024-09-21T14:02:55.550766+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:21.153018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:21.153018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-21T14:02:55.537905+0000","last_change":"2024-09-21T14:02:55.537905+0000","last_active":"2024-09-21T14:02:55.537905+0000","last_peered":"2024-09-21T14:02:55.537905+0000","last_clean":"2024-09-21T14:02:55.537905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T14:02:55.537905+0000","last_undegraded":"2024-09-21T14:02:55.537905+0000","last_fullsized":"2024-09-21T14:02:55.537905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.e","version":"290'19","reported_seq":746,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.709714+0000","last_change":"2024-09-21T14:02:28.553549+0000","last_active":"2024-09-21T14:02:50.709714+0000","last_peered":"2024-09-21T14:02:50.709714+0000","last_clean":"2024-09-21T14:02:50.709714+0000","last_became_active":"2024-09-21T13:58:17.776310+0000","last_became_peered":"2024-09-21T13:58:17.776310+0000","last_unstale":"2024-09-21T14:02:50.709714+0000","last_undegraded":"2024-09-21T14:02:50.709714+0000","last_fullsized":"2024-09-21T14:02:50.709714+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'19","last_scrub_stamp":"2024-09-21T14:02:28.553065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:28.553065+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:33.636087+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":708,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.709126+0000","last_change":"2024-09-21T14:02:03.437581+0000","last_active":"2024-09-21T14:02:50.709126+0000","last_peered":"2024-09-21T14:02:50.709126+0000","last_clean":"2024-09-21T14:02:50.709126+0000","last_became_active":"2024-09-21T13:59:32.223247+0000","last_became_peered":"2024-09-21T13:59:32.223247+0000","last_unstale":"2024-09-21T14:02:50.709126+0000","last_undegraded":"2024-09-21T14:02:50.709126+0000","last_fullsized":"2024-09-21T14:02:50.709126+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:03.437510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:02:03.437510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:14.620200+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":629,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.550683+0000","last_change":"2024-09-21T14:01:41.194538+0000","last_active":"2024-09-21T14:02:55.550683+0000","last_peered":"2024-09-21T14:02:55.550683+0000","last_clean":"2024-09-21T14:02:55.550683+0000","last_became_active":"2024-09-21T14:01:41.193941+0000","last_became_peered":"2024-09-21T14:01:41.193941+0000","last_unstale":"2024-09-21T14:02:55.550683+0000","last_undegraded":"2024-09-21T14:02:55.550683+0000","last_fullsized":"2024-09-21T14:02:55.550683+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:34.299722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:55:34.299722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":639,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.549937+0000","last_change":"2024-09-21T14:01:41.190501+0000","last_active":"2024-09-21T14:02:55.549937+0000","last_peered":"2024-09-21T14:02:55.549937+0000","last_clean":"2024-09-21T14:02:55.549937+0000","last_became_active":"2024-09-21T14:01:41.190192+0000","last_became_peered":"2024-09-21T14:01:41.190192+0000","last_unstale":"2024-09-21T14:02:55.549937+0000","last_undegraded":"2024-09-21T14:02:55.549937+0000","last_fullsized":"2024-09-21T14:02:55.549937+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:58.952218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:54:58.952218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:18.255572+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":14,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T14:02:57.551668+0000","last_change":"2024-09-21T14:02:56.560359+0000","last_active":"2024-09-21T14:02:57.551668+0000","last_peered":"2024-09-21T14:02:57.551668+0000","last_clean":"2024-09-21T14:02:57.551668+0000","last_became_active":"2024-09-21T14:02:56.560070+0000","last_became_peered":"2024-09-21T14:02:56.560070+0000","last_unstale":"2024-09-21T14:02:57.551668+0000","last_undegraded":"2024-09-21T14:02:57.551668+0000","last_fullsized":"2024-09-21T14:02:57.551668+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:22.233109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"75'79","reported_seq":102,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.549546+0000","last_change":"2024-09-21T14:01:41.189146+0000","last_active":"2024-09-21T14:02:55.549546+0000","last_peered":"2024-09-21T14:02:55.549546+0000","last_clean":"2024-09-21T14:02:55.549546+0000","last_became_active":"2024-09-21T14:01:41.188685+0000","last_became_peered":"2024-09-21T14:01:41.188685+0000","last_unstale":"2024-09-21T14:02:55.549546+0000","last_undegraded":"2024-09-21T14:02:55.549546+0000","last_fullsized":"2024-09-21T14:02:55.549546+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:03:46.437011+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"290'31","reported_seq":734,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.709538+0000","last_change":"2024-09-21T14:01:03.541547+0000","last_active":"2024-09-21T14:02:50.709538+0000","last_peered":"2024-09-21T14:02:50.709538+0000","last_clean":"2024-09-21T14:02:50.709538+0000","last_became_active":"2024-09-21T13:52:04.313218+0000","last_became_peered":"2024-09-21T13:52:04.313218+0000","last_unstale":"2024-09-21T14:02:50.709538+0000","last_undegraded":"2024-09-21T14:02:50.709538+0000","last_fullsized":"2024-09-21T14:02:50.709538+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'31","last_scrub_stamp":"2024-09-21T14:01:03.541432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:03.541432+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:50.707905+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":250,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:51.646571+0000","last_change":"2024-09-21T14:01:29.924408+0000","last_active":"2024-09-21T14:02:51.646571+0000","last_peered":"2024-09-21T14:02:51.646571+0000","last_clean":"2024-09-21T14:02:51.646571+0000","last_became_active":"2024-09-21T13:58:17.762520+0000","last_became_peered":"2024-09-21T13:58:17.762520+0000","last_unstale":"2024-09-21T14:02:51.646571+0000","last_undegraded":"2024-09-21T14:02:51.646571+0000","last_fullsized":"2024-09-21T14:02:51.646571+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:29.924331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:01:29.924331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:47.626829+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":663,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.094142+0000","last_change":"2024-09-21T13:58:52.030800+0000","last_active":"2024-09-21T14:02:50.094142+0000","last_peered":"2024-09-21T14:02:50.094142+0000","last_clean":"2024-09-21T14:02:50.094142+0000","last_became_active":"2024-09-21T13:49:02.067472+0000","last_became_peered":"2024-09-21T13:49:02.067472+0000","last_unstale":"2024-09-21T14:02:50.094142+0000","last_undegraded":"2024-09-21T14:02:50.094142+0000","last_fullsized":"2024-09-21T14:02:50.094142+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:52.030718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:52.030718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:43.346814+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":551,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T14:02:57.551946+0000","last_change":"2024-09-21T13:57:12.683321+0000","last_active":"2024-09-21T14:02:57.551946+0000","last_peered":"2024-09-21T14:02:57.551946+0000","last_clean":"2024-09-21T14:02:57.551946+0000","last_became_active":"2024-09-21T13:57:12.682979+0000","last_became_peered":"2024-09-21T13:57:12.682979+0000","last_unstale":"2024-09-21T14:02:57.551946+0000","last_undegraded":"2024-09-21T14:02:57.551946+0000","last_fullsized":"2024-09-21T14:02:57.551946+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:18.027944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:18.027944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:21.753981+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-21T14:02:55.537905+0000","last_change":"2024-09-21T14:02:55.537905+0000","last_active":"2024-09-21T14:02:55.537905+0000","last_peered":"2024-09-21T14:02:55.537905+0000","last_clean":"2024-09-21T14:02:55.537905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T14:02:55.537905+0000","last_undegraded":"2024-09-21T14:02:55.537905+0000","last_fullsized":"2024-09-21T14:02:55.537905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.9","version":"75'79","reported_seq":736,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.806160+0000","last_change":"2024-09-21T14:02:55.806160+0000","last_active":"2024-09-21T14:02:55.806160+0000","last_peered":"2024-09-21T14:02:55.806160+0000","last_clean":"2024-09-21T14:02:55.806160+0000","last_became_active":"2024-09-21T14:00:41.139315+0000","last_became_peered":"2024-09-21T14:00:41.139315+0000","last_unstale":"2024-09-21T14:02:55.806160+0000","last_undegraded":"2024-09-21T14:02:55.806160+0000","last_fullsized":"2024-09-21T14:02:55.806160+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:02:55.806077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.806077+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:59.364707+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"290'20","reported_seq":753,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.709464+0000","last_change":"2024-09-21T13:58:17.789407+0000","last_active":"2024-09-21T14:02:50.709464+0000","last_peered":"2024-09-21T14:02:50.709464+0000","last_clean":"2024-09-21T14:02:50.709464+0000","last_became_active":"2024-09-21T13:58:17.765427+0000","last_became_peered":"2024-09-21T13:58:17.765427+0000","last_unstale":"2024-09-21T14:02:50.709464+0000","last_undegraded":"2024-09-21T14:02:50.709464+0000","last_fullsized":"2024-09-21T14:02:50.709464+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:58:13.464196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:13.464196+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:16.467667+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":596,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.550993+0000","last_change":"2024-09-21T13:57:36.707211+0000","last_active":"2024-09-21T14:02:55.550993+0000","last_peered":"2024-09-21T14:02:55.550993+0000","last_clean":"2024-09-21T14:02:55.550993+0000","last_became_active":"2024-09-21T13:57:12.704203+0000","last_became_peered":"2024-09-21T13:57:12.704203+0000","last_unstale":"2024-09-21T14:02:55.550993+0000","last_undegraded":"2024-09-21T14:02:55.550993+0000","last_fullsized":"2024-09-21T14:02:55.550993+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:36.707148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:36.707148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"346'14","reported_seq":732,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T14:02:57.552816+0000","last_change":"2024-09-21T14:01:41.186960+0000","last_active":"2024-09-21T14:02:57.552816+0000","last_peered":"2024-09-21T14:02:57.552816+0000","last_clean":"2024-09-21T14:02:57.552816+0000","last_became_active":"2024-09-21T14:01:41.186564+0000","last_became_peered":"2024-09-21T14:01:41.186564+0000","last_unstale":"2024-09-21T14:02:57.552816+0000","last_undegraded":"2024-09-21T14:02:57.552816+0000","last_fullsized":"2024-09-21T14:02:57.552816+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'12","last_scrub_stamp":"2024-09-21T13:59:38.738120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:38.738120+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:47.648985+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"68'1","reported_seq":681,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.709575+0000","last_change":"2024-09-21T13:59:31.392186+0000","last_active":"2024-09-21T14:02:50.709575+0000","last_peered":"2024-09-21T14:02:50.709575+0000","last_clean":"2024-09-21T14:02:50.709575+0000","last_became_active":"2024-09-21T13:52:04.313210+0000","last_became_peered":"2024-09-21T13:52:04.313210+0000","last_unstale":"2024-09-21T14:02:50.709575+0000","last_undegraded":"2024-09-21T14:02:50.709575+0000","last_fullsized":"2024-09-21T14:02:50.709575+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T13:59:31.392105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:31.392105+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:39.716003+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":14,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T14:02:57.551648+0000","last_change":"2024-09-21T14:02:56.557127+0000","last_active":"2024-09-21T14:02:57.551648+0000","last_peered":"2024-09-21T14:02:57.551648+0000","last_clean":"2024-09-21T14:02:57.551648+0000","last_became_active":"2024-09-21T14:02:56.556786+0000","last_became_peered":"2024-09-21T14:02:56.556786+0000","last_unstale":"2024-09-21T14:02:57.551648+0000","last_undegraded":"2024-09-21T14:02:57.551648+0000","last_fullsized":"2024-09-21T14:02:57.551648+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:08.748194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"290'15","reported_seq":708,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.094306+0000","last_change":"2024-09-21T13:58:50.498597+0000","last_active":"2024-09-21T14:02:50.094306+0000","last_peered":"2024-09-21T14:02:50.094306+0000","last_clean":"2024-09-21T14:02:50.094306+0000","last_became_active":"2024-09-21T13:49:00.053016+0000","last_became_peered":"2024-09-21T13:49:00.053016+0000","last_unstale":"2024-09-21T14:02:50.094306+0000","last_undegraded":"2024-09-21T14:02:50.094306+0000","last_fullsized":"2024-09-21T14:02:50.094306+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:58:50.498369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:50.498369+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":641,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.549868+0000","last_change":"2024-09-21T14:01:41.192717+0000","last_active":"2024-09-21T14:02:55.549868+0000","last_peered":"2024-09-21T14:02:55.549868+0000","last_clean":"2024-09-21T14:02:55.549868+0000","last_became_active":"2024-09-21T14:01:41.192443+0000","last_became_peered":"2024-09-21T14:01:41.192443+0000","last_unstale":"2024-09-21T14:02:55.549868+0000","last_undegraded":"2024-09-21T14:02:55.549868+0000","last_fullsized":"2024-09-21T14:02:55.549868+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:53.894506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:53.894506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:53.450976+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":619,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.709067+0000","last_change":"2024-09-21T14:01:27.739999+0000","last_active":"2024-09-21T14:02:50.709067+0000","last_peered":"2024-09-21T14:02:50.709067+0000","last_clean":"2024-09-21T14:02:50.709067+0000","last_became_active":"2024-09-21T13:57:12.680978+0000","last_became_peered":"2024-09-21T13:57:12.680978+0000","last_unstale":"2024-09-21T14:02:50.709067+0000","last_undegraded":"2024-09-21T14:02:50.709067+0000","last_fullsized":"2024-09-21T14:02:50.709067+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:27.739921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:01:27.739921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":571,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.550990+0000","last_change":"2024-09-21T13:59:32.224001+0000","last_active":"2024-09-21T14:02:55.550990+0000","last_peered":"2024-09-21T14:02:55.550990+0000","last_clean":"2024-09-21T14:02:55.550990+0000","last_became_active":"2024-09-21T13:59:32.221628+0000","last_became_peered":"2024-09-21T13:59:32.221628+0000","last_unstale":"2024-09-21T14:02:55.550990+0000","last_undegraded":"2024-09-21T14:02:55.550990+0000","last_fullsized":"2024-09-21T14:02:55.550990+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:13.698091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:13.698091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-21T14:02:55.537905+0000","last_change":"2024-09-21T14:02:55.537905+0000","last_active":"2024-09-21T14:02:55.537905+0000","last_peered":"2024-09-21T14:02:55.537905+0000","last_clean":"2024-09-21T14:02:55.537905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T14:02:55.537905+0000","last_undegraded":"2024-09-21T14:02:55.537905+0000","last_fullsized":"2024-09-21T14:02:55.537905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.a","version":"290'35","reported_seq":316,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:51.646738+0000","last_change":"2024-09-21T14:01:02.006619+0000","last_active":"2024-09-21T14:02:51.646738+0000","last_peered":"2024-09-21T14:02:51.646738+0000","last_clean":"2024-09-21T14:02:51.646738+0000","last_became_active":"2024-09-21T13:59:32.204184+0000","last_became_peered":"2024-09-21T13:59:32.204184+0000","last_unstale":"2024-09-21T14:02:51.646738+0000","last_undegraded":"2024-09-21T14:02:51.646738+0000","last_fullsized":"2024-09-21T14:02:51.646738+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'35","last_scrub_stamp":"2024-09-21T14:01:02.006504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:02.006504+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:22.557193+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":663,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.554556+0000","last_change":"2024-09-21T14:01:41.184937+0000","last_active":"2024-09-21T14:02:55.554556+0000","last_peered":"2024-09-21T14:02:55.554556+0000","last_clean":"2024-09-21T14:02:55.554556+0000","last_became_active":"2024-09-21T14:01:41.184567+0000","last_became_peered":"2024-09-21T14:01:41.184567+0000","last_unstale":"2024-09-21T14:02:55.554556+0000","last_undegraded":"2024-09-21T14:02:55.554556+0000","last_fullsized":"2024-09-21T14:02:55.554556+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:21.587493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:21.587493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":645,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T14:02:57.551726+0000","last_change":"2024-09-21T14:01:41.182250+0000","last_active":"2024-09-21T14:02:57.551726+0000","last_peered":"2024-09-21T14:02:57.551726+0000","last_clean":"2024-09-21T14:02:57.551726+0000","last_became_active":"2024-09-21T14:01:41.181911+0000","last_became_peered":"2024-09-21T14:01:41.181911+0000","last_unstale":"2024-09-21T14:02:57.551726+0000","last_undegraded":"2024-09-21T14:02:57.551726+0000","last_fullsized":"2024-09-21T14:02:57.551726+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:55.588554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:55.588554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:45.648976+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":590,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.551469+0000","last_change":"2024-09-21T13:59:32.223123+0000","last_active":"2024-09-21T14:02:55.551469+0000","last_peered":"2024-09-21T14:02:55.551469+0000","last_clean":"2024-09-21T14:02:55.551469+0000","last_became_active":"2024-09-21T13:59:32.219332+0000","last_became_peered":"2024-09-21T13:59:32.219332+0000","last_unstale":"2024-09-21T14:02:55.551469+0000","last_undegraded":"2024-09-21T14:02:55.551469+0000","last_fullsized":"2024-09-21T14:02:55.551469+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:19.537177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:19.537177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-21T14:02:55.537905+0000","last_change":"2024-09-21T14:02:55.537905+0000","last_active":"2024-09-21T14:02:55.537905+0000","last_peered":"2024-09-21T14:02:55.537905+0000","last_clean":"2024-09-21T14:02:55.537905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T14:02:55.537905+0000","last_undegraded":"2024-09-21T14:02:55.537905+0000","last_fullsized":"2024-09-21T14:02:55.537905+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.9","version":"290'18","reported_seq":725,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.709029+0000","last_change":"2024-09-21T13:59:28.340494+0000","last_active":"2024-09-21T14:02:50.709029+0000","last_peered":"2024-09-21T14:02:50.709029+0000","last_clean":"2024-09-21T14:02:50.709029+0000","last_became_active":"2024-09-21T13:57:12.680627+0000","last_became_peered":"2024-09-21T13:57:12.680627+0000","last_unstale":"2024-09-21T14:02:50.709029+0000","last_undegraded":"2024-09-21T14:02:50.709029+0000","last_fullsized":"2024-09-21T14:02:50.709029+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'18","last_scrub_stamp":"2024-09-21T13:59:28.340415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:28.340415+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:11.383693+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":618,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.554416+0000","last_change":"2024-09-21T13:59:32.201126+0000","last_active":"2024-09-21T14:02:55.554416+0000","last_peered":"2024-09-21T14:02:55.554416+0000","last_clean":"2024-09-21T14:02:55.554416+0000","last_became_active":"2024-09-21T13:59:32.200674+0000","last_became_peered":"2024-09-21T13:59:32.200674+0000","last_unstale":"2024-09-21T14:02:55.554416+0000","last_undegraded":"2024-09-21T14:02:55.554416+0000","last_fullsized":"2024-09-21T14:02:55.554416+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:21.582325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:21.582325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":608,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T14:02:57.550653+0000","last_change":"2024-09-21T14:02:27.791041+0000","last_active":"2024-09-21T14:02:57.550653+0000","last_peered":"2024-09-21T14:02:57.550653+0000","last_clean":"2024-09-21T14:02:57.550653+0000","last_became_active":"2024-09-21T13:52:04.322147+0000","last_became_peered":"2024-09-21T13:52:04.322147+0000","last_unstale":"2024-09-21T14:02:57.550653+0000","last_undegraded":"2024-09-21T14:02:57.550653+0000","last_fullsized":"2024-09-21T14:02:57.550653+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:27.790994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:02:27.790994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:51.879201+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":612,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.551158+0000","last_change":"2024-09-21T14:02:36.371955+0000","last_active":"2024-09-21T14:02:55.551158+0000","last_peered":"2024-09-21T14:02:55.551158+0000","last_clean":"2024-09-21T14:02:55.551158+0000","last_became_active":"2024-09-21T13:59:32.203767+0000","last_became_peered":"2024-09-21T13:59:32.203767+0000","last_unstale":"2024-09-21T14:02:55.551158+0000","last_undegraded":"2024-09-21T14:02:55.551158+0000","last_fullsized":"2024-09-21T14:02:55.551158+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:36.371747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:02:36.371747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:54.168810+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":1,"reported_epoch":381,"state":"creating+peering","last_fresh":"2024-09-21T14:02:55.549078+0000","last_change":"2024-09-21T14:02:55.549078+0000","last_active":"2024-09-21T14:02:55.537905+0000","last_peered":"2024-09-21T14:02:55.537905+0000","last_clean":"2024-09-21T14:02:55.537905+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T14:02:55.549078+0000","last_undegraded":"2024-09-21T14:02:55.549078+0000","last_fullsized":"2024-09-21T14:02:55.549078+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"290'27","reported_seq":717,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.549723+0000","last_change":"2024-09-21T14:01:41.193143+0000","last_active":"2024-09-21T14:02:55.549723+0000","last_peered":"2024-09-21T14:02:55.549723+0000","last_clean":"2024-09-21T14:02:55.549723+0000","last_became_active":"2024-09-21T14:01:41.192481+0000","last_became_peered":"2024-09-21T14:01:41.192481+0000","last_unstale":"2024-09-21T14:02:55.549723+0000","last_undegraded":"2024-09-21T14:02:55.549723+0000","last_fullsized":"2024-09-21T14:02:55.549723+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'27","last_scrub_stamp":"2024-09-21T13:59:33.058859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:33.058859+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":590,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.554856+0000","last_change":"2024-09-21T13:59:32.203889+0000","last_active":"2024-09-21T14:02:55.554856+0000","last_peered":"2024-09-21T14:02:55.554856+0000","last_clean":"2024-09-21T14:02:55.554856+0000","last_became_active":"2024-09-21T13:59:32.203172+0000","last_became_peered":"2024-09-21T13:59:32.203172+0000","last_unstale":"2024-09-21T14:02:55.554856+0000","last_undegraded":"2024-09-21T14:02:55.554856+0000","last_fullsized":"2024-09-21T14:02:55.554856+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T13:55:03.879706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:55:03.879706+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"300'14","reported_seq":805,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.550823+0000","last_change":"2024-09-21T13:59:32.220898+0000","last_active":"2024-09-21T14:02:55.550823+0000","last_peered":"2024-09-21T14:02:55.550823+0000","last_clean":"2024-09-21T14:02:55.550823+0000","last_became_active":"2024-09-21T13:59:32.206956+0000","last_became_peered":"2024-09-21T13:59:32.206956+0000","last_unstale":"2024-09-21T14:02:55.550823+0000","last_undegraded":"2024-09-21T14:02:55.550823+0000","last_fullsized":"2024-09-21T14:02:55.550823+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'11","last_scrub_stamp":"2024-09-21T13:56:31.616003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:56:31.616003+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":665,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.549648+0000","last_change":"2024-09-21T14:01:41.248408+0000","last_active":"2024-09-21T14:02:55.549648+0000","last_peered":"2024-09-21T14:02:55.549648+0000","last_clean":"2024-09-21T14:02:55.549648+0000","last_became_active":"2024-09-21T14:01:41.246106+0000","last_became_peered":"2024-09-21T14:01:41.246106+0000","last_unstale":"2024-09-21T14:02:55.549648+0000","last_undegraded":"2024-09-21T14:02:55.549648+0000","last_fullsized":"2024-09-21T14:02:55.549648+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:39.258659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:39.258659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":252,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:51.646480+0000","last_change":"2024-09-21T14:01:50.733055+0000","last_active":"2024-09-21T14:02:51.646480+0000","last_peered":"2024-09-21T14:02:51.646480+0000","last_clean":"2024-09-21T14:02:51.646480+0000","last_became_active":"2024-09-21T14:01:41.197501+0000","last_became_peered":"2024-09-21T14:01:41.197501+0000","last_unstale":"2024-09-21T14:02:51.646480+0000","last_undegraded":"2024-09-21T14:02:51.646480+0000","last_fullsized":"2024-09-21T14:02:51.646480+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:50.733007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:01:50.733007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:08.367766+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"290'12","reported_seq":639,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.551905+0000","last_change":"2024-09-21T13:59:32.247422+0000","last_active":"2024-09-21T14:02:55.551905+0000","last_peered":"2024-09-21T14:02:55.551905+0000","last_clean":"2024-09-21T14:02:55.551905+0000","last_became_active":"2024-09-21T13:59:32.211306+0000","last_became_peered":"2024-09-21T13:59:32.211306+0000","last_unstale":"2024-09-21T14:02:55.551905+0000","last_undegraded":"2024-09-21T14:02:55.551905+0000","last_fullsized":"2024-09-21T14:02:55.551905+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'6","last_scrub_stamp":"2024-09-21T13:55:12.066873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:55:12.066873+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:38.360712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":597,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.550154+0000","last_change":"2024-09-21T14:01:41.199742+0000","last_active":"2024-09-21T14:02:55.550154+0000","last_peered":"2024-09-21T14:02:55.550154+0000","last_clean":"2024-09-21T14:02:55.550154+0000","last_became_active":"2024-09-21T14:01:41.199321+0000","last_became_peered":"2024-09-21T14:01:41.199321+0000","last_unstale":"2024-09-21T14:02:55.550154+0000","last_undegraded":"2024-09-21T14:02:55.550154+0000","last_fullsized":"2024-09-21T14:02:55.550154+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:37.106238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:37.106238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":636,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.554398+0000","last_change":"2024-09-21T13:59:32.222277+0000","last_active":"2024-09-21T14:02:55.554398+0000","last_peered":"2024-09-21T14:02:55.554398+0000","last_clean":"2024-09-21T14:02:55.554398+0000","last_became_active":"2024-09-21T13:59:32.219525+0000","last_became_peered":"2024-09-21T13:59:32.219525+0000","last_unstale":"2024-09-21T14:02:55.554398+0000","last_undegraded":"2024-09-21T14:02:55.554398+0000","last_fullsized":"2024-09-21T14:02:55.554398+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:02.838181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:55:02.838181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"290'15","reported_seq":602,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.548448+0000","last_change":"2024-09-21T13:59:32.238689+0000","last_active":"2024-09-21T14:02:55.548448+0000","last_peered":"2024-09-21T14:02:55.548448+0000","last_clean":"2024-09-21T14:02:55.548448+0000","last_became_active":"2024-09-21T13:59:32.209166+0000","last_became_peered":"2024-09-21T13:59:32.209166+0000","last_unstale":"2024-09-21T14:02:55.548448+0000","last_undegraded":"2024-09-21T14:02:55.548448+0000","last_fullsized":"2024-09-21T14:02:55.548448+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:55:04.990842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:55:04.990842+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:21.278052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"61'1","reported_seq":654,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.551275+0000","last_change":"2024-09-21T13:59:32.222826+0000","last_active":"2024-09-21T14:02:55.551275+0000","last_peered":"2024-09-21T14:02:55.551275+0000","last_clean":"2024-09-21T14:02:55.551275+0000","last_became_active":"2024-09-21T13:59:32.219515+0000","last_became_peered":"2024-09-21T13:59:32.219515+0000","last_unstale":"2024-09-21T14:02:55.551275+0000","last_undegraded":"2024-09-21T14:02:55.551275+0000","last_fullsized":"2024-09-21T14:02:55.551275+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:58:07.711028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:07.711028+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":587,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.551416+0000","last_change":"2024-09-21T13:59:32.211754+0000","last_active":"2024-09-21T14:02:55.551416+0000","last_peered":"2024-09-21T14:02:55.551416+0000","last_clean":"2024-09-21T14:02:55.551416+0000","last_became_active":"2024-09-21T13:59:32.210586+0000","last_became_peered":"2024-09-21T13:59:32.210586+0000","last_unstale":"2024-09-21T14:02:55.551416+0000","last_undegraded":"2024-09-21T14:02:55.551416+0000","last_fullsized":"2024-09-21T14:02:55.551416+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:28.014760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:28.014760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:17.280712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":649,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-09-21T14:02:57.551726+0000","last_change":"2024-09-21T14:02:29.731635+0000","last_active":"2024-09-21T14:02:57.551726+0000","last_peered":"2024-09-21T14:02:57.551726+0000","last_clean":"2024-09-21T14:02:57.551726+0000","last_became_active":"2024-09-21T13:59:32.205803+0000","last_became_peered":"2024-09-21T13:59:32.205803+0000","last_unstale":"2024-09-21T14:02:57.551726+0000","last_undegraded":"2024-09-21T14:02:57.551726+0000","last_fullsized":"2024-09-21T14:02:57.551726+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:29.731565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:02:29.731565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:53.473477+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"290'15","reported_seq":642,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.550059+0000","last_change":"2024-09-21T14:01:41.189676+0000","last_active":"2024-09-21T14:02:55.550059+0000","last_peered":"2024-09-21T14:02:55.550059+0000","last_clean":"2024-09-21T14:02:55.550059+0000","last_became_active":"2024-09-21T14:01:41.187904+0000","last_became_peered":"2024-09-21T14:01:41.187904+0000","last_unstale":"2024-09-21T14:02:55.550059+0000","last_undegraded":"2024-09-21T14:02:55.550059+0000","last_fullsized":"2024-09-21T14:02:55.550059+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:57:24.124604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:57:24.124604+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":523,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.551568+0000","last_change":"2024-09-21T13:59:32.212325+0000","last_active":"2024-09-21T14:02:55.551568+0000","last_peered":"2024-09-21T14:02:55.551568+0000","last_clean":"2024-09-21T14:02:55.551568+0000","last_became_active":"2024-09-21T13:59:32.208749+0000","last_became_peered":"2024-09-21T13:59:32.208749+0000","last_unstale":"2024-09-21T14:02:55.551568+0000","last_undegraded":"2024-09-21T14:02:55.551568+0000","last_fullsized":"2024-09-21T14:02:55.551568+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:20.574745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:20.574745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"300'14","reported_seq":738,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.550037+0000","last_change":"2024-09-21T13:59:32.208253+0000","last_active":"2024-09-21T14:02:55.550037+0000","last_peered":"2024-09-21T14:02:55.550037+0000","last_clean":"2024-09-21T14:02:55.550037+0000","last_became_active":"2024-09-21T13:59:32.206926+0000","last_became_peered":"2024-09-21T13:59:32.206926+0000","last_unstale":"2024-09-21T14:02:55.550037+0000","last_undegraded":"2024-09-21T14:02:55.550037+0000","last_fullsized":"2024-09-21T14:02:55.550037+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'10","last_scrub_stamp":"2024-09-21T13:54:20.946782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:20.946782+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:10.314285+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":687,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.709665+0000","last_change":"2024-09-21T14:01:41.200564+0000","last_active":"2024-09-21T14:02:50.709665+0000","last_peered":"2024-09-21T14:02:50.709665+0000","last_clean":"2024-09-21T14:02:50.709665+0000","last_became_active":"2024-09-21T14:01:41.200113+0000","last_became_peered":"2024-09-21T14:01:41.200113+0000","last_unstale":"2024-09-21T14:02:50.709665+0000","last_undegraded":"2024-09-21T14:02:50.709665+0000","last_fullsized":"2024-09-21T14:02:50.709665+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:54.805619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:54.805619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:48.669884+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"290'20","reported_seq":638,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.549796+0000","last_change":"2024-09-21T13:59:32.236530+0000","last_active":"2024-09-21T14:02:55.549796+0000","last_peered":"2024-09-21T14:02:55.549796+0000","last_clean":"2024-09-21T14:02:55.549796+0000","last_became_active":"2024-09-21T13:59:32.208332+0000","last_became_peered":"2024-09-21T13:59:32.208332+0000","last_unstale":"2024-09-21T14:02:55.549796+0000","last_undegraded":"2024-09-21T14:02:55.549796+0000","last_fullsized":"2024-09-21T14:02:55.549796+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:57:37.671391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:57:37.671391+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:55.297241+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":630,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.551259+0000","last_change":"2024-09-21T13:59:32.212737+0000","last_active":"2024-09-21T14:02:55.551259+0000","last_peered":"2024-09-21T14:02:55.551259+0000","last_clean":"2024-09-21T14:02:55.551259+0000","last_became_active":"2024-09-21T13:59:32.209804+0000","last_became_peered":"2024-09-21T13:59:32.209804+0000","last_unstale":"2024-09-21T14:02:55.551259+0000","last_undegraded":"2024-09-21T14:02:55.551259+0000","last_fullsized":"2024-09-21T14:02:55.551259+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:25.571256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:25.571256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"346'10","reported_seq":738,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.549933+0000","last_change":"2024-09-21T14:01:41.207030+0000","last_active":"2024-09-21T14:02:55.549933+0000","last_peered":"2024-09-21T14:02:55.549933+0000","last_clean":"2024-09-21T14:02:55.549933+0000","last_became_active":"2024-09-21T14:01:41.206303+0000","last_became_peered":"2024-09-21T14:01:41.206303+0000","last_unstale":"2024-09-21T14:02:55.549933+0000","last_undegraded":"2024-09-21T14:02:55.549933+0000","last_fullsized":"2024-09-21T14:02:55.549933+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:59:32.104224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:32.104224+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":527,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.554188+0000","last_change":"2024-09-21T13:59:32.213015+0000","last_active":"2024-09-21T14:02:55.554188+0000","last_peered":"2024-09-21T14:02:55.554188+0000","last_clean":"2024-09-21T14:02:55.554188+0000","last_became_active":"2024-09-21T13:59:32.211472+0000","last_became_peered":"2024-09-21T13:59:32.211472+0000","last_unstale":"2024-09-21T14:02:55.554188+0000","last_undegraded":"2024-09-21T14:02:55.554188+0000","last_fullsized":"2024-09-21T14:02:55.554188+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:22.647191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:22.647191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"290'19","reported_seq":750,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.709228+0000","last_change":"2024-09-21T14:02:34.670058+0000","last_active":"2024-09-21T14:02:50.709228+0000","last_peered":"2024-09-21T14:02:50.709228+0000","last_clean":"2024-09-21T14:02:50.709228+0000","last_became_active":"2024-09-21T13:58:17.771497+0000","last_became_peered":"2024-09-21T13:58:17.771497+0000","last_unstale":"2024-09-21T14:02:50.709228+0000","last_undegraded":"2024-09-21T14:02:50.709228+0000","last_fullsized":"2024-09-21T14:02:50.709228+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'19","last_scrub_stamp":"2024-09-21T14:02:34.669962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:34.669962+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:44.716931+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":264,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.552499+0000","last_change":"2024-09-21T13:59:32.202097+0000","last_active":"2024-09-21T14:02:55.552499+0000","last_peered":"2024-09-21T14:02:55.552499+0000","last_clean":"2024-09-21T14:02:55.552499+0000","last_became_active":"2024-09-21T13:59:32.201568+0000","last_became_peered":"2024-09-21T13:59:32.201568+0000","last_unstale":"2024-09-21T14:02:55.552499+0000","last_undegraded":"2024-09-21T14:02:55.552499+0000","last_fullsized":"2024-09-21T14:02:55.552499+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:01.228213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:01.228213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":700,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.094612+0000","last_change":"2024-09-21T14:01:41.185588+0000","last_active":"2024-09-21T14:02:50.094612+0000","last_peered":"2024-09-21T14:02:50.094612+0000","last_clean":"2024-09-21T14:02:50.094612+0000","last_became_active":"2024-09-21T14:01:41.185293+0000","last_became_peered":"2024-09-21T14:01:41.185293+0000","last_unstale":"2024-09-21T14:02:50.094612+0000","last_undegraded":"2024-09-21T14:02:50.094612+0000","last_fullsized":"2024-09-21T14:02:50.094612+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:35.409237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:35.409237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:10.948556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":595,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.549901+0000","last_change":"2024-09-21T14:01:41.200439+0000","last_active":"2024-09-21T14:02:55.549901+0000","last_peered":"2024-09-21T14:02:55.549901+0000","last_clean":"2024-09-21T14:02:55.549901+0000","last_became_active":"2024-09-21T14:01:41.199671+0000","last_became_peered":"2024-09-21T14:01:41.199671+0000","last_unstale":"2024-09-21T14:02:55.549901+0000","last_undegraded":"2024-09-21T14:02:55.549901+0000","last_fullsized":"2024-09-21T14:02:55.549901+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:39.643458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:39.643458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"290'15","reported_seq":744,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.094823+0000","last_change":"2024-09-21T13:58:39.977717+0000","last_active":"2024-09-21T14:02:50.094823+0000","last_peered":"2024-09-21T14:02:50.094823+0000","last_clean":"2024-09-21T14:02:50.094823+0000","last_became_active":"2024-09-21T13:58:17.764120+0000","last_became_peered":"2024-09-21T13:58:17.764120+0000","last_unstale":"2024-09-21T14:02:50.094823+0000","last_undegraded":"2024-09-21T14:02:50.094823+0000","last_fullsized":"2024-09-21T14:02:50.094823+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:58:39.977466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:39.977466+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":634,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.554231+0000","last_change":"2024-09-21T13:59:32.223082+0000","last_active":"2024-09-21T14:02:55.554231+0000","last_peered":"2024-09-21T14:02:55.554231+0000","last_clean":"2024-09-21T14:02:55.554231+0000","last_became_active":"2024-09-21T13:59:32.221854+0000","last_became_peered":"2024-09-21T13:59:32.221854+0000","last_unstale":"2024-09-21T14:02:55.554231+0000","last_undegraded":"2024-09-21T14:02:55.554231+0000","last_fullsized":"2024-09-21T14:02:55.554231+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:17.691193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:17.691193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":570,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.547800+0000","last_change":"2024-09-21T13:57:12.703746+0000","last_active":"2024-09-21T14:02:55.547800+0000","last_peered":"2024-09-21T14:02:55.547800+0000","last_clean":"2024-09-21T14:02:55.547800+0000","last_became_active":"2024-09-21T13:57:12.703441+0000","last_became_peered":"2024-09-21T13:57:12.703441+0000","last_unstale":"2024-09-21T14:02:55.547800+0000","last_undegraded":"2024-09-21T14:02:55.547800+0000","last_fullsized":"2024-09-21T14:02:55.547800+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:19.999125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:19.999125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:13.302246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":492,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.549339+0000","last_change":"2024-09-21T14:01:41.188075+0000","last_active":"2024-09-21T14:02:55.549339+0000","last_peered":"2024-09-21T14:02:55.549339+0000","last_clean":"2024-09-21T14:02:55.549339+0000","last_became_active":"2024-09-21T14:01:41.186934+0000","last_became_peered":"2024-09-21T14:01:41.186934+0000","last_unstale":"2024-09-21T14:02:55.549339+0000","last_undegraded":"2024-09-21T14:02:55.549339+0000","last_fullsized":"2024-09-21T14:02:55.549339+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:47.334504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:47.334504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"290'19","reported_seq":714,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.550112+0000","last_change":"2024-09-21T13:59:32.277742+0000","last_active":"2024-09-21T14:02:55.550112+0000","last_peered":"2024-09-21T14:02:55.550112+0000","last_clean":"2024-09-21T14:02:55.550112+0000","last_became_active":"2024-09-21T13:59:32.231567+0000","last_became_peered":"2024-09-21T13:59:32.231567+0000","last_unstale":"2024-09-21T14:02:55.550112+0000","last_undegraded":"2024-09-21T14:02:55.550112+0000","last_fullsized":"2024-09-21T14:02:55.550112+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:54:22.978267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:22.978267+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:12.294653+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":594,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.550257+0000","last_change":"2024-09-21T14:01:41.191090+0000","last_active":"2024-09-21T14:02:55.550257+0000","last_peered":"2024-09-21T14:02:55.550257+0000","last_clean":"2024-09-21T14:02:55.550257+0000","last_became_active":"2024-09-21T14:01:41.190760+0000","last_became_peered":"2024-09-21T14:01:41.190760+0000","last_unstale":"2024-09-21T14:02:55.550257+0000","last_undegraded":"2024-09-21T14:02:55.550257+0000","last_fullsized":"2024-09-21T14:02:55.550257+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:17.305144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:17.305144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":640,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.551596+0000","last_change":"2024-09-21T13:59:32.219508+0000","last_active":"2024-09-21T14:02:55.551596+0000","last_peered":"2024-09-21T14:02:55.551596+0000","last_clean":"2024-09-21T14:02:55.551596+0000","last_became_active":"2024-09-21T13:59:32.214004+0000","last_became_peered":"2024-09-21T13:59:32.214004+0000","last_unstale":"2024-09-21T14:02:55.551596+0000","last_undegraded":"2024-09-21T14:02:55.551596+0000","last_fullsized":"2024-09-21T14:02:55.551596+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:36.525417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:36.525417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":636,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.550947+0000","last_change":"2024-09-21T13:59:28.124373+0000","last_active":"2024-09-21T14:02:55.550947+0000","last_peered":"2024-09-21T14:02:55.550947+0000","last_clean":"2024-09-21T14:02:55.550947+0000","last_became_active":"2024-09-21T13:58:17.778115+0000","last_became_peered":"2024-09-21T13:58:17.778115+0000","last_unstale":"2024-09-21T14:02:55.550947+0000","last_undegraded":"2024-09-21T14:02:55.550947+0000","last_fullsized":"2024-09-21T14:02:55.550947+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:28.124296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:28.124296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"290'8","reported_seq":660,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.547472+0000","last_change":"2024-09-21T13:59:26.034457+0000","last_active":"2024-09-21T14:02:55.547472+0000","last_peered":"2024-09-21T14:02:55.547472+0000","last_clean":"2024-09-21T14:02:55.547472+0000","last_became_active":"2024-09-21T13:58:17.775079+0000","last_became_peered":"2024-09-21T13:58:17.775079+0000","last_unstale":"2024-09-21T14:02:55.547472+0000","last_undegraded":"2024-09-21T14:02:55.547472+0000","last_fullsized":"2024-09-21T14:02:55.547472+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'8","last_scrub_stamp":"2024-09-21T13:59:26.034372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:26.034372+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":562,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.550489+0000","last_change":"2024-09-21T14:01:41.196335+0000","last_active":"2024-09-21T14:02:55.550489+0000","last_peered":"2024-09-21T14:02:55.550489+0000","last_clean":"2024-09-21T14:02:55.550489+0000","last_became_active":"2024-09-21T14:01:41.195989+0000","last_became_peered":"2024-09-21T14:01:41.195989+0000","last_unstale":"2024-09-21T14:02:55.550489+0000","last_undegraded":"2024-09-21T14:02:55.550489+0000","last_fullsized":"2024-09-21T14:02:55.550489+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:16.457445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:16.457445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":252,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:51.646526+0000","last_change":"2024-09-21T14:00:57.128764+0000","last_active":"2024-09-21T14:02:51.646526+0000","last_peered":"2024-09-21T14:02:51.646526+0000","last_clean":"2024-09-21T14:02:51.646526+0000","last_became_active":"2024-09-21T13:59:32.204380+0000","last_became_peered":"2024-09-21T13:59:32.204380+0000","last_unstale":"2024-09-21T14:02:51.646526+0000","last_undegraded":"2024-09-21T14:02:51.646526+0000","last_fullsized":"2024-09-21T14:02:51.646526+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:00:57.128713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:00:57.128713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:24.563060+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":573,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.551278+0000","last_change":"2024-09-21T13:59:32.212615+0000","last_active":"2024-09-21T14:02:55.551278+0000","last_peered":"2024-09-21T14:02:55.551278+0000","last_clean":"2024-09-21T14:02:55.551278+0000","last_became_active":"2024-09-21T13:59:32.210134+0000","last_became_peered":"2024-09-21T13:59:32.210134+0000","last_unstale":"2024-09-21T14:02:55.551278+0000","last_undegraded":"2024-09-21T14:02:55.551278+0000","last_fullsized":"2024-09-21T14:02:55.551278+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:08.852789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:55:08.852789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":685,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:50.094885+0000","last_change":"2024-09-21T14:01:41.187792+0000","last_active":"2024-09-21T14:02:50.094885+0000","last_peered":"2024-09-21T14:02:50.094885+0000","last_clean":"2024-09-21T14:02:50.094885+0000","last_became_active":"2024-09-21T14:01:41.187438+0000","last_became_peered":"2024-09-21T14:01:41.187438+0000","last_unstale":"2024-09-21T14:02:50.094885+0000","last_undegraded":"2024-09-21T14:02:50.094885+0000","last_fullsized":"2024-09-21T14:02:50.094885+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:17.706325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:17.706325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:40.383161+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":616,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.547589+0000","last_change":"2024-09-21T13:58:13.776632+0000","last_active":"2024-09-21T14:02:55.547589+0000","last_peered":"2024-09-21T14:02:55.547589+0000","last_clean":"2024-09-21T14:02:55.547589+0000","last_became_active":"2024-09-21T13:57:12.704042+0000","last_became_peered":"2024-09-21T13:57:12.704042+0000","last_unstale":"2024-09-21T14:02:55.547589+0000","last_undegraded":"2024-09-21T14:02:55.547589+0000","last_fullsized":"2024-09-21T14:02:55.547589+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:13.776572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:13.776572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"290'19","reported_seq":285,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-09-21T14:02:51.647225+0000","last_change":"2024-09-21T14:01:54.680822+0000","last_active":"2024-09-21T14:02:51.647225+0000","last_peered":"2024-09-21T14:02:51.647225+0000","last_clean":"2024-09-21T14:02:51.647225+0000","last_became_active":"2024-09-21T14:01:41.194606+0000","last_became_peered":"2024-09-21T14:01:41.194606+0000","last_unstale":"2024-09-21T14:02:51.647225+0000","last_undegraded":"2024-09-21T14:02:51.647225+0000","last_fullsized":"2024-09-21T14:02:51.647225+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'19","last_scrub_stamp":"2024-09-21T14:01:54.680735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:54.680735+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:21.312249+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":542,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-09-21T14:02:55.554067+0000","last_change":"2024-09-21T13:59:32.203045+0000","last_active":"2024-09-21T14:02:55.554067+0000","last_peered":"2024-09-21T14:02:55.554067+0000","last_clean":"2024-09-21T14:02:55.554067+0000","last_became_active":"2024-09-21T13:59:32.202225+0000","last_became_peered":"2024-09-21T13:59:32.202225+0000","last_unstale":"2024-09-21T14:02:55.554067+0000","last_undegraded":"2024-09-21T14:02:55.554067+0000","last_fullsized":"2024-09-21T14:02:55.554067+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:40.650832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:40.650832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":3},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1111,"num_read_kb":868,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":1914,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":687,"ondisk_log_size":687,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":210,"num_read_kb":205,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":375688,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":139,"num_write":73,"num_write_kb":1036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1031048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":298,"seq":1279900254250,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":310820,"kb_used_data":2768,"kb_used_omap":4,"kb_used_meta":308027,"kb_avail":93434332,"statfs":{"total":95995035648,"available":95676755968,"internally_reserved":0,"allocated":2834432,"data_stored":1967387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":315420328},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463089,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":340420,"kb_used_data":2416,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":93404732,"statfs":{"total":95995035648,"available":95646445568,"internally_reserved":0,"allocated":2473984,"data_stored":1630133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":346094022},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":5,"up_from":345,"seq":1481763717136,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":111820,"kb_used_data":3000,"kb_used_omap":4,"kb_used_meta":108795,"kb_avail":93633332,"statfs":{"total":95995035648,"available":95880531968,"internally_reserved":0,"allocated":3072000,"data_stored":2221417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":111406755},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920958,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":412388,"kb_used_data":29968,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":93332764,"statfs":{"total":95995035648,"available":95572750336,"internally_reserved":0,"allocated":30687232,"data_stored":29811231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":391576005},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":240,"seq":1030792151110,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":257460,"kb_used_data":3032,"kb_used_omap":4,"kb_used_meta":254395,"kb_avail":93487692,"statfs":{"total":95995035648,"available":95731396608,"internally_reserved":0,"allocated":3104768,"data_stored":2221949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4443,"internal_metadata":260501157},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":130,"seq":558345748612,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":548596,"kb_used_data":2712,"kb_used_omap":4,"kb_used_meta":545851,"kb_avail":93196556,"statfs":{"total":95995035648,"available":95433273344,"internally_reserved":0,"allocated":2777088,"data_stored":1918537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4441,"internal_metadata":558952103},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575056,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":583432,"kb_used_data":2480,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":93161720,"statfs":{"total":95995035648,"available":95397601280,"internally_reserved":0,"allocated":2539520,"data_stored":1631495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":594868677},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738582,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":480668,"kb_used_data":2436,"kb_used_omap":1,"kb_used_meta":478206,"kb_avail":93264484,"statfs":{"total":95995035648,"available":95502831616,"internally_reserved":0,"allocated":2494464,"data_stored":1630491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":489683397},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T14:02:59.622 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph status --format=json 2024-09-21T14:02:59.623 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:02:59.629 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:02:59.652 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:59.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:59.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:02:59.753 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:02:59.776 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:02:59.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:59.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:02:59.828 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:59 smithi049 ceph-mon[40489]: pgmap v717: 158 pgs: 3 creating+peering, 10 unknown, 145 active+clean; 373 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-09-21T14:02:59.828 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:59 smithi049 ceph-mon[40489]: 3.b scrub starts 2024-09-21T14:02:59.828 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:59 smithi049 ceph-mon[40489]: 3.b scrub ok 2024-09-21T14:02:59.829 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:59 smithi049 ceph-mon[40489]: 4.f scrub starts 2024-09-21T14:02:59.829 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:59 smithi049 ceph-mon[40489]: 4.f scrub ok 2024-09-21T14:02:59.829 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:59 smithi049 ceph-mon[40489]: 5.f scrub starts 2024-09-21T14:02:59.829 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:59 smithi049 ceph-mon[40489]: 5.f scrub ok 2024-09-21T14:02:59.829 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:59 smithi049 ceph-mon[40489]: from='client.25943 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:02:59.829 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:59 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:02:59.829 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:02:59 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:02:59.829 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:59 smithi049 ceph-mon[30080]: pgmap v717: 158 pgs: 3 creating+peering, 10 unknown, 145 active+clean; 373 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-09-21T14:02:59.829 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:59 smithi049 ceph-mon[30080]: 3.b scrub starts 2024-09-21T14:02:59.829 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:59 smithi049 ceph-mon[30080]: 3.b scrub ok 2024-09-21T14:02:59.829 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:59 smithi049 ceph-mon[30080]: 4.f scrub starts 2024-09-21T14:02:59.830 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:59 smithi049 ceph-mon[30080]: 4.f scrub ok 2024-09-21T14:02:59.830 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:59 smithi049 ceph-mon[30080]: 5.f scrub starts 2024-09-21T14:02:59.830 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:59 smithi049 ceph-mon[30080]: 5.f scrub ok 2024-09-21T14:02:59.830 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:59 smithi049 ceph-mon[30080]: from='client.25943 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:02:59.830 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:59 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:02:59.830 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:02:59 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:02:59.877 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:02:59.899 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:02:59.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:02:59.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:02:59.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:59 smithi187 ceph-mon[33032]: pgmap v717: 158 pgs: 3 creating+peering, 10 unknown, 145 active+clean; 373 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-09-21T14:02:59.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:59 smithi187 ceph-mon[33032]: 3.b scrub starts 2024-09-21T14:02:59.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:59 smithi187 ceph-mon[33032]: 3.b scrub ok 2024-09-21T14:02:59.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:59 smithi187 ceph-mon[33032]: 4.f scrub starts 2024-09-21T14:02:59.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:59 smithi187 ceph-mon[33032]: 4.f scrub ok 2024-09-21T14:02:59.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:59 smithi187 ceph-mon[33032]: 5.f scrub starts 2024-09-21T14:02:59.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:59 smithi187 ceph-mon[33032]: 5.f scrub ok 2024-09-21T14:02:59.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:59 smithi187 ceph-mon[33032]: from='client.25943 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:02:59.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:59 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:02:59.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:02:59 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:02:59.945 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:02:59 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:03:00.001 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:00.024 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:00.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:00.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:00.086 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:03:00.093 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:03:00.126 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:03:00.147 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:00.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:00.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:03:00.249 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:03:00.271 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:00.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:00.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:03:00.373 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:00.395 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:00.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:00.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:00.498 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:03:00.520 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:00.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:00.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:03:00.622 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:03:00.645 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:00.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:00.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:03:00.742 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:03:00 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:03:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:03:00.742 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:00 smithi049 ceph-mon[40489]: 5.2 scrub starts 2024-09-21T14:03:00.742 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:00 smithi049 ceph-mon[40489]: 5.2 scrub ok 2024-09-21T14:03:00.742 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:00 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:03:00.742 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:00 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:03:00.743 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:00 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:03:00.743 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:00 smithi049 ceph-mon[40489]: 5.e scrub starts 2024-09-21T14:03:00.743 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:00 smithi049 ceph-mon[40489]: 5.e scrub ok 2024-09-21T14:03:00.743 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:00 smithi049 ceph-mon[40489]: osdmap e384: 8 total, 8 up, 8 in 2024-09-21T14:03:00.743 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:00 smithi049 ceph-mon[30080]: 5.2 scrub starts 2024-09-21T14:03:00.743 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:00 smithi049 ceph-mon[30080]: 5.2 scrub ok 2024-09-21T14:03:00.743 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:00 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:03:00.743 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:00 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:03:00.743 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:00 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:03:00.743 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:00 smithi049 ceph-mon[30080]: 5.e scrub starts 2024-09-21T14:03:00.743 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:00 smithi049 ceph-mon[30080]: 5.e scrub ok 2024-09-21T14:03:00.744 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:00 smithi049 ceph-mon[30080]: osdmap e384: 8 total, 8 up, 8 in 2024-09-21T14:03:00.747 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:03:00.770 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:00.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:00.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:03:00.871 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:03:00.893 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:00.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:00.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:03:00.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:00 smithi187 ceph-mon[33032]: 5.2 scrub starts 2024-09-21T14:03:00.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:00 smithi187 ceph-mon[33032]: 5.2 scrub ok 2024-09-21T14:03:00.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:00 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:03:00.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:00 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:03:00.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:00 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:03:00.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:00 smithi187 ceph-mon[33032]: 5.e scrub starts 2024-09-21T14:03:00.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:00 smithi187 ceph-mon[33032]: 5.e scrub ok 2024-09-21T14:03:00.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:00 smithi187 ceph-mon[33032]: osdmap e384: 8 total, 8 up, 8 in 2024-09-21T14:03:00.995 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:01.017 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:01.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:01.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:01.118 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:03:01.141 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:01.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:01.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:03:01.228 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T14:03:01.228 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-09-21T14:03:01.228 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 0 2024-09-21T14:03:01.228 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T14:03:01.244 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:03:01.266 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:01.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:01.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:03:01.368 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:03:01.390 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:01.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:01.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:03:01.491 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:03:01.513 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:01.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:01.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:03:01.565 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-09-21T14:03:01.616 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:01.638 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:01.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:01.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:01.740 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:03:01.761 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:01.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:01.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:03:01.863 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:03:01.885 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:01.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:01.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:03:01.934 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:01 smithi049 ceph-mon[40489]: pgmap v718: 158 pgs: 3 creating+peering, 155 active+clean; 373 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-09-21T14:03:01.934 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:01 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1354053692' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:03:01.934 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:01 smithi049 ceph-mon[30080]: pgmap v718: 158 pgs: 3 creating+peering, 155 active+clean; 373 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-09-21T14:03:01.934 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:01 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1354053692' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:03:01.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:01 smithi187 ceph-mon[33032]: pgmap v718: 158 pgs: 3 creating+peering, 155 active+clean; 373 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-09-21T14:03:01.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:01 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1354053692' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:03:01.961 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T14:03:01.987 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:03:02.026 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:02.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:02.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:03:02.112 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T14:03:02.128 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:03:02.177 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:02.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:02.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:03:02.279 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:03:02.301 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:02.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:02.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:03:02.403 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:03:02.426 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:02.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:02.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:03:02.527 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:03:02.550 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:02.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:02.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:03:02.652 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:03:02.675 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:02.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:02.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:03:02.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:02 smithi049 ceph-mon[40489]: 3.0 scrub starts 2024-09-21T14:03:02.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:02 smithi049 ceph-mon[40489]: 3.0 scrub ok 2024-09-21T14:03:02.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:02 smithi049 ceph-mon[40489]: 3.16 scrub starts 2024-09-21T14:03:02.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:02 smithi049 ceph-mon[40489]: 3.16 scrub ok 2024-09-21T14:03:02.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:02 smithi049 ceph-mon[40489]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T14:03:02.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:02 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1354053692' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T14:03:02.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:02 smithi049 ceph-mon[40489]: osdmap e385: 8 total, 8 up, 8 in 2024-09-21T14:03:02.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:02 smithi049 ceph-mon[40489]: 6.15 scrub starts 2024-09-21T14:03:02.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:02 smithi049 ceph-mon[40489]: 6.15 scrub ok 2024-09-21T14:03:02.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:02 smithi049 ceph-mon[30080]: 3.0 scrub starts 2024-09-21T14:03:02.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:02 smithi049 ceph-mon[30080]: 3.0 scrub ok 2024-09-21T14:03:02.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:02 smithi049 ceph-mon[30080]: 3.16 scrub starts 2024-09-21T14:03:02.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:02 smithi049 ceph-mon[30080]: 3.16 scrub ok 2024-09-21T14:03:02.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:02 smithi049 ceph-mon[30080]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T14:03:02.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:02 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1354053692' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T14:03:02.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:02 smithi049 ceph-mon[30080]: osdmap e385: 8 total, 8 up, 8 in 2024-09-21T14:03:02.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:02 smithi049 ceph-mon[30080]: 6.15 scrub starts 2024-09-21T14:03:02.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:02 smithi049 ceph-mon[30080]: 6.15 scrub ok 2024-09-21T14:03:02.777 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:03:02.800 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:02.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:02.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:03:02.901 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:03:02.924 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:02.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:02.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:03:02.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:02 smithi187 ceph-mon[33032]: 3.0 scrub starts 2024-09-21T14:03:02.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:02 smithi187 ceph-mon[33032]: 3.0 scrub ok 2024-09-21T14:03:02.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:02 smithi187 ceph-mon[33032]: 3.16 scrub starts 2024-09-21T14:03:02.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:02 smithi187 ceph-mon[33032]: 3.16 scrub ok 2024-09-21T14:03:02.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:02 smithi187 ceph-mon[33032]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T14:03:02.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:02 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1354053692' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T14:03:02.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:02 smithi187 ceph-mon[33032]: osdmap e385: 8 total, 8 up, 8 in 2024-09-21T14:03:02.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:02 smithi187 ceph-mon[33032]: 6.15 scrub starts 2024-09-21T14:03:02.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:02 smithi187 ceph-mon[33032]: 6.15 scrub ok 2024-09-21T14:03:03.026 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:03.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:03.055 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:03.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:03.156 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:03:03.176 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:03:03.182 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:03.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:03.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:03:03.217 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:03:03.284 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:03:03.310 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:03.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:03.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:03:03.412 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:03:03.438 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:03.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:03.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:03:03.539 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:03:03.562 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:03.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:03.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:03:03.576 INFO:teuthology.orchestra.run.smithi049.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-09-21T14:03:03.665 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:03:03.688 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:03.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:03.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:03:03.789 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:03:03.815 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:03.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:03.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:03:03.917 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:03:03.940 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:03.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:03.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:03:03.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:03 smithi187 ceph-mon[33032]: pgmap v721: 158 pgs: 158 active+clean; 373 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 939 B/s rd, 0 op/s 2024-09-21T14:03:03.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:03 smithi187 ceph-mon[33032]: 6.6 scrub starts 2024-09-21T14:03:03.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:03 smithi187 ceph-mon[33032]: 6.6 scrub ok 2024-09-21T14:03:03.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:03 smithi187 ceph-mon[33032]: 5.18 scrub starts 2024-09-21T14:03:03.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:03 smithi187 ceph-mon[33032]: 5.18 scrub ok 2024-09-21T14:03:03.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:03 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1192500662' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T14:03:03.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:03 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T14:03:04.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:03 smithi049 ceph-mon[40489]: pgmap v721: 158 pgs: 158 active+clean; 373 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 939 B/s rd, 0 op/s 2024-09-21T14:03:04.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:03 smithi049 ceph-mon[40489]: 6.6 scrub starts 2024-09-21T14:03:04.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:03 smithi049 ceph-mon[40489]: 6.6 scrub ok 2024-09-21T14:03:04.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:03 smithi049 ceph-mon[40489]: 5.18 scrub starts 2024-09-21T14:03:04.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:03 smithi049 ceph-mon[40489]: 5.18 scrub ok 2024-09-21T14:03:04.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:03 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1192500662' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T14:03:04.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:03 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T14:03:04.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:03 smithi049 ceph-mon[30080]: pgmap v721: 158 pgs: 158 active+clean; 373 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 939 B/s rd, 0 op/s 2024-09-21T14:03:04.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:03 smithi049 ceph-mon[30080]: 6.6 scrub starts 2024-09-21T14:03:04.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:03 smithi049 ceph-mon[30080]: 6.6 scrub ok 2024-09-21T14:03:04.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:03 smithi049 ceph-mon[30080]: 5.18 scrub starts 2024-09-21T14:03:04.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:03 smithi049 ceph-mon[30080]: 5.18 scrub ok 2024-09-21T14:03:04.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:03 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1192500662' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T14:03:04.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:03 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T14:03:04.042 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:03:04.068 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:04.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:04.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:03:04.169 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:03:04.194 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:04.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:04.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:03:04.296 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:03:04.322 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:04.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:04.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:03:04.412 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T14:03:04.412 INFO:teuthology.orchestra.run.smithi049.stdout:{"fsid":"40a3d2e8-781f-11ef-baf6-efdc52797490","health":{"status":"HEALTH_WARN","checks":{"POOL_APP_NOT_ENABLED":{"severity":"HEALTH_WARN","summary":{"message":"1 pool(s) do not have an application enabled","count":1},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1100,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":386,"num_osds":8,"num_up_osds":8,"osd_up_since":1726927300,"num_in_osds":8,"osd_in_since":1726927096,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":158}],"num_pgs":158,"num_pools":7,"num_objects":199,"data_bytes":381827,"bytes_used":3109117952,"bytes_avail":764851167232,"bytes_total":767960285184,"read_bytes_sec":939,"read_op_per_sec":0,"write_op_per_sec":0},"fsmap":{"epoch":1,"btime":"2024-09-21T13:42:57:838094+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.49:8443/","prometheus":"http://172.21.15.49:9283/"}},"servicemap":{"epoch":18,"modified":"2024-09-21T14:02:39.989968+0000","services":{"rgw":{"daemons":{"summary":"","24365":{"start_epoch":12,"start_stamp":"2024-09-21T13:49:04.899719+0000","gid":24365,"addr":"172.21.15.49:0/3404764347","metadata":{"arch":"x86_64","ceph_release":"squid","ceph_version":"ceph version 19.3.0-5104-g44178494 (441784946648ff53dd310293f9dd752afeeea72a) squid (dev)","ceph_version_short":"19.3.0-5104-g44178494","container_hostname":"smithi049","container_image":"quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5","cpu":"Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz","distro":"centos","distro_description":"CentOS Stream 9","distro_version":"9","frontend_config#0":"beast port=80","frontend_type#0":"beast","hostname":"smithi049","id":"foo.a","kernel_description":"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024","kernel_version":"5.14.0-508.el9.x86_64","mem_swap_kb":"0","mem_total_kb":"32487164","num_handles":"1","os":"Linux","pid":"7","realm_id":"","realm_name":"","zone_id":"d447add1-8b31-4043-a929-4cb437e5f078","zone_name":"default","zonegroup_id":"06d4f733-b51c-4f26-9844-acd95f223f5b","zonegroup_name":"default"},"task_status":{}}}}}},"progress_events":{}} 2024-09-21T14:03:04.424 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:03:04.449 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:04.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:04.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:03:04.551 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:04.578 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:04.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:04.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:04.679 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:03:04.702 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:04.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:04.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:03:04.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:04 smithi049 ceph-mon[40489]: 5.0 scrub starts 2024-09-21T14:03:04.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:04 smithi049 ceph-mon[40489]: 5.0 scrub ok 2024-09-21T14:03:04.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:04 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T14:03:04.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:04 smithi049 ceph-mon[40489]: osdmap e386: 8 total, 8 up, 8 in 2024-09-21T14:03:04.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:04 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2848439534' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T14:03:04.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:04 smithi049 ceph-mon[30080]: 5.0 scrub starts 2024-09-21T14:03:04.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:04 smithi049 ceph-mon[30080]: 5.0 scrub ok 2024-09-21T14:03:04.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:04 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T14:03:04.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:04 smithi049 ceph-mon[30080]: osdmap e386: 8 total, 8 up, 8 in 2024-09-21T14:03:04.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:04 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2848439534' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T14:03:04.804 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:03:04.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:04.827 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:04.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:03:04.928 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:03:04.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:04 smithi187 ceph-mon[33032]: 5.0 scrub starts 2024-09-21T14:03:04.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:04 smithi187 ceph-mon[33032]: 5.0 scrub ok 2024-09-21T14:03:04.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:04 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T14:03:04.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:04 smithi187 ceph-mon[33032]: osdmap e386: 8 total, 8 up, 8 in 2024-09-21T14:03:04.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:04 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2848439534' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T14:03:04.955 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:04.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:04.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:03:04.977 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T14:03:05.057 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:03:05.079 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:05.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:05.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:03:05.080 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:03:05.080 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:03:05.094 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:03:05.094 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:03:05.125 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:03:05.125 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:03:05.142 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:03:05.143 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:03:05.177 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:03:05.177 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:03:05.180 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:05.204 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:05.204 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:03:05.205 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:03:05.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:05.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:05.222 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:03:05.223 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:03:05.240 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:03:05.240 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:03:05.306 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:03:05.329 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:05.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:05.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:03:05.431 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:03:05.453 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:05.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:05.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:03:05.549 INFO:tasks.cephadm.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-09-21T14:03:05.549 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T14:03:05.555 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:03:05.612 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:05.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:05.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:03:05.648 DEBUG:teuthology.orchestra.run.smithi049:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-09-21T14:03:05.657 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-21T14:03:05.657 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-21T14:03:05.714 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:03:05.736 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:05.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:05.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:03:05.838 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:03:05.850 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:03:05.862 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:05.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:05.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:03:05.864 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:hints = 1 2024-09-21T14:03:05.865 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-21T14:03:05.865 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Object prefix: benchmark_data_smithi049_176811 2024-09-21T14:03:05.865 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T14:03:05.865 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 0 0 0 0 0 0 - 0 2024-09-21T14:03:05.911 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:03:05.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:05 smithi187 ceph-mon[33032]: pgmap v723: 158 pgs: 158 active+clean; 373 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T14:03:05.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:05 smithi187 ceph-mon[33032]: 6.4 scrub starts 2024-09-21T14:03:05.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:05 smithi187 ceph-mon[33032]: 6.4 scrub ok 2024-09-21T14:03:05.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:05 smithi187 ceph-mon[33032]: 5.1b scrub starts 2024-09-21T14:03:05.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:05 smithi187 ceph-mon[33032]: 5.1b scrub ok 2024-09-21T14:03:05.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:05 smithi187 ceph-mon[33032]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T14:03:05.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:05 smithi187 ceph-mon[33032]: Cluster is now healthy 2024-09-21T14:03:05.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:05 smithi187 ceph-mon[33032]: 3.f scrub starts 2024-09-21T14:03:05.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:05 smithi187 ceph-mon[33032]: 3.f scrub ok 2024-09-21T14:03:05.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:05 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1827242971' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:03:05.963 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:03:05.983 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:05 smithi049 ceph-mon[30080]: pgmap v723: 158 pgs: 158 active+clean; 373 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T14:03:05.984 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:05 smithi049 ceph-mon[30080]: 6.4 scrub starts 2024-09-21T14:03:05.984 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:05 smithi049 ceph-mon[30080]: 6.4 scrub ok 2024-09-21T14:03:05.984 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:05 smithi049 ceph-mon[30080]: 5.1b scrub starts 2024-09-21T14:03:05.984 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:05 smithi049 ceph-mon[30080]: 5.1b scrub ok 2024-09-21T14:03:05.984 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:05 smithi049 ceph-mon[30080]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T14:03:05.984 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:05 smithi049 ceph-mon[30080]: Cluster is now healthy 2024-09-21T14:03:05.984 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:05 smithi049 ceph-mon[30080]: 3.f scrub starts 2024-09-21T14:03:05.984 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:05 smithi049 ceph-mon[30080]: 3.f scrub ok 2024-09-21T14:03:05.984 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:05 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1827242971' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:03:05.984 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:05 smithi049 ceph-mon[40489]: pgmap v723: 158 pgs: 158 active+clean; 373 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T14:03:05.984 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:05 smithi049 ceph-mon[40489]: 6.4 scrub starts 2024-09-21T14:03:05.984 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:05 smithi049 ceph-mon[40489]: 6.4 scrub ok 2024-09-21T14:03:05.985 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:05 smithi049 ceph-mon[40489]: 5.1b scrub starts 2024-09-21T14:03:05.985 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:05 smithi049 ceph-mon[40489]: 5.1b scrub ok 2024-09-21T14:03:05.985 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:05 smithi049 ceph-mon[40489]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T14:03:05.985 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:05 smithi049 ceph-mon[40489]: Cluster is now healthy 2024-09-21T14:03:05.985 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:05 smithi049 ceph-mon[40489]: 3.f scrub starts 2024-09-21T14:03:05.985 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:05 smithi049 ceph-mon[40489]: 3.f scrub ok 2024-09-21T14:03:05.985 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:05 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1827242971' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:03:06.002 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:06.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:06.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:03:06.105 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:03:06.141 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:06.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:06.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:03:06.243 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:03:06.279 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:06.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:06.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:03:06.381 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:03:06.414 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:06.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:06.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:03:06.516 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:03:06.555 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:06.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:06.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:03:06.657 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:06.691 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:06.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:06.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:06.793 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:03:06.832 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:06.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:06.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:03:06.865 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 1 15 3799 3784 236.487 236.5 0.00276145 0.00421465 2024-09-21T14:03:06.934 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:06.955 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:06.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:06.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:07.057 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:03:07.095 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:07.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:07.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:03:07.197 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:03:07.230 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:07.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:07.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:03:07.332 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:03:07.367 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:07.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:07.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:03:07.469 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:03:07.504 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:07.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:07.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:03:07.606 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:03:07.642 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:07.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:07.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:03:07.744 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:03:07.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:07 smithi049 ceph-mon[40489]: pgmap v724: 158 pgs: 158 active+clean; 501 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 21 KiB/s wr, 1 op/s 2024-09-21T14:03:07.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:07 smithi049 ceph-mon[40489]: 5.7 scrub starts 2024-09-21T14:03:07.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:07 smithi049 ceph-mon[40489]: 5.7 scrub ok 2024-09-21T14:03:07.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:07 smithi049 ceph-mon[30080]: pgmap v724: 158 pgs: 158 active+clean; 501 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 21 KiB/s wr, 1 op/s 2024-09-21T14:03:07.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:07 smithi049 ceph-mon[30080]: 5.7 scrub starts 2024-09-21T14:03:07.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:07 smithi049 ceph-mon[30080]: 5.7 scrub ok 2024-09-21T14:03:07.777 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:07.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:07.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:03:07.866 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 2 16 6983 6967 217.698 198.938 0.00442798 0.00458247 2024-09-21T14:03:07.878 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:03:07.911 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:07.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:07.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:03:07.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:07 smithi187 ceph-mon[33032]: pgmap v724: 158 pgs: 158 active+clean; 501 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 21 KiB/s wr, 1 op/s 2024-09-21T14:03:07.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:07 smithi187 ceph-mon[33032]: 5.7 scrub starts 2024-09-21T14:03:07.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:07 smithi187 ceph-mon[33032]: 5.7 scrub ok 2024-09-21T14:03:08.013 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:03:08.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:08.056 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:08.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:03:08.110 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T14:03:08.110 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:03:08.157 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:03:08.191 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:08.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:08.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:03:08.293 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:03:08.325 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:08.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:08.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:03:08.426 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:08.461 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:08.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:08.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:08.563 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:03:08.602 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:08.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:08.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:03:08.611 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-09-21T14:03:08.704 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:03:08.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:08.740 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:08.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:03:08.841 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:03:08.866 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 3 16 9942 9926 206.766 184.938 0.00395104 0.00482591 2024-09-21T14:03:08.881 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:08.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:08.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:03:08.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:08 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:03:08.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:08 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:03:08.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:08 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:03:08.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:08 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:03:08.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:08 smithi187 ceph-mon[33032]: osdmap e387: 8 total, 8 up, 8 in 2024-09-21T14:03:08.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:08 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1443807801' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:03:08.984 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:03:09.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:08 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:03:09.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:08 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:03:09.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:08 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:03:09.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:08 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:03:09.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:08 smithi049 ceph-mon[40489]: osdmap e387: 8 total, 8 up, 8 in 2024-09-21T14:03:09.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:08 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1443807801' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:03:09.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:08 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:03:09.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:08 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:03:09.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:08 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:03:09.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:08 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:03:09.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:08 smithi049 ceph-mon[30080]: osdmap e387: 8 total, 8 up, 8 in 2024-09-21T14:03:09.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:08 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1443807801' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:03:09.024 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:09.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:09.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:03:09.126 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:03:09.165 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:09.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:09.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:03:09.228 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T14:03:09.268 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:03:09.302 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:09.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:09.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:03:09.404 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:09.436 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:09.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:09.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:09.538 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:03:09.585 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:09.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:09.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:03:09.687 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:03:09.722 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:09.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:09.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:03:09.824 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:09.858 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:09.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:09.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:09.866 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 4 16 12885 12869 201.055 183.938 0.00354773 0.00496697 2024-09-21T14:03:09.944 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:03:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:03:09.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:09 smithi187 ceph-mon[33032]: pgmap v726: 158 pgs: 158 active+clean; 84 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 796 B/s rd, 13 MiB/s wr, 209 op/s 2024-09-21T14:03:09.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:09 smithi187 ceph-mon[33032]: 5.1d scrub starts 2024-09-21T14:03:09.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:09 smithi187 ceph-mon[33032]: 5.1d scrub ok 2024-09-21T14:03:09.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:09 smithi187 ceph-mon[33032]: 4.3 scrub starts 2024-09-21T14:03:09.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:09 smithi187 ceph-mon[33032]: 4.3 scrub ok 2024-09-21T14:03:09.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:09 smithi187 ceph-mon[33032]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:03:09.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:09 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1443807801' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T14:03:09.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:09 smithi187 ceph-mon[33032]: osdmap e388: 8 total, 8 up, 8 in 2024-09-21T14:03:09.961 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:03:10.000 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:10.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:10.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:03:10.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:09 smithi049 ceph-mon[40489]: pgmap v726: 158 pgs: 158 active+clean; 84 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 796 B/s rd, 13 MiB/s wr, 209 op/s 2024-09-21T14:03:10.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:09 smithi049 ceph-mon[40489]: 5.1d scrub starts 2024-09-21T14:03:10.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:09 smithi049 ceph-mon[40489]: 5.1d scrub ok 2024-09-21T14:03:10.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:09 smithi049 ceph-mon[40489]: 4.3 scrub starts 2024-09-21T14:03:10.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:09 smithi049 ceph-mon[40489]: 4.3 scrub ok 2024-09-21T14:03:10.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:09 smithi049 ceph-mon[40489]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:03:10.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:09 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1443807801' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T14:03:10.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:09 smithi049 ceph-mon[40489]: osdmap e388: 8 total, 8 up, 8 in 2024-09-21T14:03:10.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:09 smithi049 ceph-mon[30080]: pgmap v726: 158 pgs: 158 active+clean; 84 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 796 B/s rd, 13 MiB/s wr, 209 op/s 2024-09-21T14:03:10.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:09 smithi049 ceph-mon[30080]: 5.1d scrub starts 2024-09-21T14:03:10.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:09 smithi049 ceph-mon[30080]: 5.1d scrub ok 2024-09-21T14:03:10.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:09 smithi049 ceph-mon[30080]: 4.3 scrub starts 2024-09-21T14:03:10.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:09 smithi049 ceph-mon[30080]: 4.3 scrub ok 2024-09-21T14:03:10.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:09 smithi049 ceph-mon[30080]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:03:10.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:09 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1443807801' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T14:03:10.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:09 smithi049 ceph-mon[30080]: osdmap e388: 8 total, 8 up, 8 in 2024-09-21T14:03:10.103 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:03:10.135 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:10.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:10.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:03:10.237 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:10.266 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T14:03:10.268 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:10.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:10.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:10.370 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:03:10.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:10.413 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:10.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:03:10.514 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:03:10.545 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:10.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:10.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:03:10.647 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:03:10.699 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:10.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:10.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:03:10.801 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:03:10.836 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:10.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:10.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:03:10.866 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 5 16 15750 15734 196.653 179.062 0.00702424 0.00507877 2024-09-21T14:03:10.938 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:03:10.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:10 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:03:10.978 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:10.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:10.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:03:11.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:10 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:03:11.004 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:03:10 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:03:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:03:11.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:10 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:03:11.079 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:03:11.120 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:11.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:11.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:03:11.221 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:03:11.255 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:11.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:11.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:03:11.357 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:03:11.396 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:11.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:11.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:03:11.498 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:03:11.531 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:11.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:11.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:03:11.596 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:03:11.597 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:03:11.633 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:11.662 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:11.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:11.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:11.764 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:03:11.803 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:11.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:11.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:03:11.866 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 6 16 18534 18518 192.875 174 0.00545574 0.00517766 2024-09-21T14:03:11.905 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:03:11.912 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:11 smithi049 ceph-mon[40489]: pgmap v728: 158 pgs: 158 active+clean; 449 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 796 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-09-21T14:03:11.913 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:11 smithi049 ceph-mon[30080]: pgmap v728: 158 pgs: 158 active+clean; 449 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 796 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-09-21T14:03:11.936 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:11.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:11.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:03:11.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:11 smithi187 ceph-mon[33032]: pgmap v728: 158 pgs: 158 active+clean; 449 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 796 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-09-21T14:03:12.039 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:03:12.080 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:12.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:12.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:03:12.182 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:03:12.222 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:12.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:12.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:03:12.324 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:03:12.359 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:12.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:12.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:03:12.461 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:03:12.505 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:12.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:12.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:03:12.607 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:03:12.619 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:03:12.620 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:03:12.631 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:03:12.632 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:03:12.637 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:12.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:12.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:03:12.676 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:03:12.676 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:03:12.713 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:03:12.713 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:03:12.739 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:03:12.745 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:03:12.750 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:03:12.750 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:03:12.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:12 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/796577982' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:03:12.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:12 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/796577982' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:03:12.787 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:12.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:12.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:03:12.792 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:03:12.792 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:03:12.828 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:03:12.828 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:03:12.856 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T14:03:12.864 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-09-21T14:03:12.866 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:03:12.866 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:03:12.866 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 7 16 20952 20936 186.908 151.125 0.0115236 0.00534308 2024-09-21T14:03:12.890 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:03:12.924 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:12.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:12.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:03:12.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:12 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/796577982' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:03:13.026 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:03:13.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:13.070 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:13.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:03:13.171 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:03:13.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:13.215 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:13.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:03:13.316 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:03:13.355 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:13.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:13.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:03:13.457 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:03:13.492 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:13.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:13.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:03:13.594 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:03:13.627 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:13.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:13.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:03:13.663 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T14:03:13.729 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:03:13.751 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:13.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:13.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:03:13.853 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:03:13.867 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 8 15 23324 23309 182.072 148.312 0.00719888 0.00548608 2024-09-21T14:03:13.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:13.894 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:13.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:03:13.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:13 smithi187 ceph-mon[33032]: pgmap v729: 158 pgs: 158 active+clean; 630 MiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-09-21T14:03:13.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:13 smithi187 ceph-mon[33032]: from='client.16080 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:03:13.995 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:03:14.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:13 smithi049 ceph-mon[40489]: pgmap v729: 158 pgs: 158 active+clean; 630 MiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-09-21T14:03:14.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:13 smithi049 ceph-mon[40489]: from='client.16080 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:03:14.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:13 smithi049 ceph-mon[30080]: pgmap v729: 158 pgs: 158 active+clean; 630 MiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-09-21T14:03:14.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:13 smithi049 ceph-mon[30080]: from='client.16080 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:03:14.031 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:14.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:14.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:03:14.133 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:03:14.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:14.178 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:14.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:03:14.279 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:03:14.318 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:14.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:14.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:03:14.421 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:14.455 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:14.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:14.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:14.557 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:03:14.592 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:14.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:14.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:03:14.694 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:03:14.721 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:14.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:14.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:03:14.823 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:03:14.846 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:14.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:14.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:03:14.867 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 9 16 25514 25498 177.042 136.812 0.0018277 0.00556782 2024-09-21T14:03:14.948 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:03:14.983 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:14.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:14.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:03:15.086 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:15.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:15.111 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:15.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:15.212 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:03:15.251 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:15.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:15.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:03:15.353 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:03:15.392 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:15.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:15.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:03:15.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:15 smithi187 ceph-mon[33032]: pgmap v730: 158 pgs: 158 active+clean; 1.2 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 160 MiB/s wr, 2.56k op/s 2024-09-21T14:03:15.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:15 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:03:15.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:15 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:03:15.494 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:03:15.501 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:15 smithi049 ceph-mon[40489]: pgmap v730: 158 pgs: 158 active+clean; 1.2 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 160 MiB/s wr, 2.56k op/s 2024-09-21T14:03:15.502 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:15 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:03:15.502 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:15 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:03:15.502 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:15 smithi049 ceph-mon[30080]: pgmap v730: 158 pgs: 158 active+clean; 1.2 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 160 MiB/s wr, 2.56k op/s 2024-09-21T14:03:15.502 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:15 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:03:15.502 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:15 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:03:15.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:15.533 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:15.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:03:15.634 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:03:15.670 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:15.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:15.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:03:15.772 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:03:15.805 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:15.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:15.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:03:15.867 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 10 16 27526 27510 171.912 125.75 0.00573841 0.00581142 2024-09-21T14:03:15.906 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:15.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:15.943 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:15.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:16.044 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:03:16.076 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:16.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:16.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:03:16.178 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:16.212 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:16.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:16.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:16.314 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:16.346 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:16.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:16.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:16.448 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:16.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:16.484 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:16.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:16.585 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:03:16.621 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:16.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:16.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:03:16.723 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:03:16.748 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:16.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:16.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:03:16.849 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:03:16.867 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 11 16 29793 29777 169.163 141.688 0.00679275 0.00590577 2024-09-21T14:03:16.883 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:16.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:16.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:03:16.985 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:03:17.021 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:17.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:17.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:03:17.123 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:17.158 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:17.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:17.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:17.260 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:03:17.303 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:17.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:17.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:03:17.361 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:17 smithi049 ceph-mon[40489]: pgmap v731: 158 pgs: 158 active+clean; 1.4 GiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 609 B/s rd, 169 MiB/s wr, 2.70k op/s 2024-09-21T14:03:17.361 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:17 smithi049 ceph-mon[30080]: pgmap v731: 158 pgs: 158 active+clean; 1.4 GiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 609 B/s rd, 169 MiB/s wr, 2.70k op/s 2024-09-21T14:03:17.405 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:03:17.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:17 smithi187 ceph-mon[33032]: pgmap v731: 158 pgs: 158 active+clean; 1.4 GiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 609 B/s rd, 169 MiB/s wr, 2.70k op/s 2024-09-21T14:03:17.453 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T14:03:17.453 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-09-21T14:03:17.453 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-09-21T14:03:17.454 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-09-21T14:03:17.454 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.454 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24365.0:1780 4.4 4:22d26bf9:::data_loggenerations_metadata:head [watch reconnect cookie 94025139211520 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-09-21T14:03:17.454 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:55:40.878748+0000", 2024-09-21T14:03:17.454 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 456.57254687099999, 2024-09-21T14:03:17.454 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.39793440499999999, 2024-09-21T14:03:17.454 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:03:17.454 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:03:17.454 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T14:03:17.454 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:03:17.454 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24365", 2024-09-21T14:03:17.455 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3404764347", 2024-09-21T14:03:17.455 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 1780 2024-09-21T14:03:17.455 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.455 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:03:17.455 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.455 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:03:17.455 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:40.878748+0000", 2024-09-21T14:03:17.455 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:03:17.455 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.455 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.455 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:03:17.455 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:40.878748+0000", 2024-09-21T14:03:17.455 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:03:17.455 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.456 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.456 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:03:17.456 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:40.878750+0000", 2024-09-21T14:03:17.456 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.962e-06 2024-09-21T14:03:17.456 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.456 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.456 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:03:17.456 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:40.878756+0000", 2024-09-21T14:03:17.456 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3080000000000004e-06 2024-09-21T14:03:17.456 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.456 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.456 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:03:17.456 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:40.878768+0000", 2024-09-21T14:03:17.456 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.203e-05 2024-09-21T14:03:17.457 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.457 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.457 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:03:17.457 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:40.878776+0000", 2024-09-21T14:03:17.457 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4849999999999992e-06 2024-09-21T14:03:17.457 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.457 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.457 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:03:17.457 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:40.878849+0000", 2024-09-21T14:03:17.457 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2670999999999996e-05 2024-09-21T14:03:17.457 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.457 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.457 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:03:17.457 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:40.878859+0000", 2024-09-21T14:03:17.458 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6099999999999995e-06 2024-09-21T14:03:17.458 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.458 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.458 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:03:17.458 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:41.271074+0000", 2024-09-21T14:03:17.458 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.39221556499999999 2024-09-21T14:03:17.458 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.458 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.458 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:03:17.458 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:41.271224+0000", 2024-09-21T14:03:17.458 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00015018400000000001 2024-09-21T14:03:17.458 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.458 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.459 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:03:17.459 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:41.276682+0000", 2024-09-21T14:03:17.459 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00545759 2024-09-21T14:03:17.459 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:17.459 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:17.459 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:17.459 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.459 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.459 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24365.0:1781 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 94025139203456 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-09-21T14:03:17.459 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:55:40.878794+0000", 2024-09-21T14:03:17.459 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 456.57250041499998, 2024-09-21T14:03:17.459 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.39819017699999998, 2024-09-21T14:03:17.459 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:03:17.460 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:03:17.460 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T14:03:17.460 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:03:17.460 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24365", 2024-09-21T14:03:17.460 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3404764347", 2024-09-21T14:03:17.460 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 1781 2024-09-21T14:03:17.460 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.460 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:03:17.460 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.460 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:03:17.460 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:40.878794+0000", 2024-09-21T14:03:17.460 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:03:17.460 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.461 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.461 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:03:17.461 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:40.878794+0000", 2024-09-21T14:03:17.461 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:03:17.461 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.461 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.461 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:03:17.461 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:40.878795+0000", 2024-09-21T14:03:17.461 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9299999999999998e-07 2024-09-21T14:03:17.461 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.461 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.461 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:03:17.461 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:40.878798+0000", 2024-09-21T14:03:17.461 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.0120000000000001e-06 2024-09-21T14:03:17.461 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.462 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.462 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:03:17.462 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:40.878805+0000", 2024-09-21T14:03:17.462 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6830000000000008e-06 2024-09-21T14:03:17.462 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.462 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.462 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:03:17.462 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:40.878811+0000", 2024-09-21T14:03:17.462 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9009999999999999e-06 2024-09-21T14:03:17.462 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.462 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.462 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:03:17.462 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:40.878873+0000", 2024-09-21T14:03:17.462 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1412000000000001e-05 2024-09-21T14:03:17.463 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.463 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.463 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:03:17.463 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:40.878882+0000", 2024-09-21T14:03:17.463 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2289999999999992e-06 2024-09-21T14:03:17.463 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.463 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.463 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:03:17.463 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:41.274347+0000", 2024-09-21T14:03:17.463 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.39546476200000003 2024-09-21T14:03:17.463 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.463 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.463 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:03:17.463 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:41.274501+0000", 2024-09-21T14:03:17.464 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00015485499999999999 2024-09-21T14:03:17.464 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.464 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.464 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:03:17.464 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:41.276984+0000", 2024-09-21T14:03:17.464 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0024828300000000001 2024-09-21T14:03:17.464 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:17.464 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:17.464 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:17.464 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.464 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.464 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232045 8.7 8:e258b742:::benchmark_data_smithi049_109311_object105526:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T14:03:17.464 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.681733+0000", 2024-09-21T14:03:17.464 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 365.76956191199997, 2024-09-21T14:03:17.465 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0897055980000001, 2024-09-21T14:03:17.465 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:03:17.465 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:03:17.465 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:03:17.465 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:03:17.465 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T14:03:17.465 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:03:17.465 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232045 2024-09-21T14:03:17.465 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.465 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:03:17.465 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.465 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:03:17.465 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.681733+0000", 2024-09-21T14:03:17.465 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:03:17.466 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.466 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.466 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:03:17.466 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.681733+0000", 2024-09-21T14:03:17.466 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:03:17.466 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.466 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.466 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:03:17.466 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.681734+0000", 2024-09-21T14:03:17.466 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6020000000000001e-06 2024-09-21T14:03:17.466 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.466 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.466 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:03:17.466 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.681740+0000", 2024-09-21T14:03:17.467 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.592e-06 2024-09-21T14:03:17.467 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.467 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.467 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:03:17.467 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.681751+0000", 2024-09-21T14:03:17.467 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1640999999999999e-05 2024-09-21T14:03:17.467 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.467 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.467 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:03:17.467 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.681759+0000", 2024-09-21T14:03:17.467 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5399999999999998e-06 2024-09-21T14:03:17.467 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.467 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.468 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:03:17.468 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.686829+0000", 2024-09-21T14:03:17.468 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0050702300000000002 2024-09-21T14:03:17.468 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.468 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.468 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:03:17.468 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.686838+0000", 2024-09-21T14:03:17.469 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.884e-06 2024-09-21T14:03:17.469 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.469 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.469 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:03:17.469 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.767863+0000", 2024-09-21T14:03:17.469 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.081025358 2024-09-21T14:03:17.469 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.469 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.469 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:03:17.469 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.768022+0000", 2024-09-21T14:03:17.469 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00015892599999999999 2024-09-21T14:03:17.469 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.469 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.470 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,3,6", 2024-09-21T14:03:17.470 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.768190+0000", 2024-09-21T14:03:17.470 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016811899999999999 2024-09-21T14:03:17.470 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.470 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.470 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:03:17.470 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.770833+0000", 2024-09-21T14:03:17.470 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0026427410000000001 2024-09-21T14:03:17.470 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.470 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.470 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:17.470 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.771163+0000", 2024-09-21T14:03:17.470 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000329618 2024-09-21T14:03:17.471 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.471 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.471 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:17.471 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.771221+0000", 2024-09-21T14:03:17.471 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8322000000000002e-05 2024-09-21T14:03:17.471 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.471 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.471 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:17.471 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.771281+0000", 2024-09-21T14:03:17.471 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0053999999999998e-05 2024-09-21T14:03:17.471 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.471 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.471 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:03:17.471 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.771351+0000", 2024-09-21T14:03:17.472 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9828999999999998e-05 2024-09-21T14:03:17.472 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.472 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.472 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:03:17.472 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.771438+0000", 2024-09-21T14:03:17.472 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.7142e-05 2024-09-21T14:03:17.472 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:17.472 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:17.472 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:17.472 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.472 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.472 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232047 8.7 8:e0ec053c:::benchmark_data_smithi049_109311_object105528:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T14:03:17.472 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.686294+0000", 2024-09-21T14:03:17.473 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 365.765000493, 2024-09-21T14:03:17.473 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0967370940000001, 2024-09-21T14:03:17.473 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:03:17.473 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:03:17.473 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:03:17.473 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:03:17.473 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T14:03:17.473 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:03:17.473 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232047 2024-09-21T14:03:17.473 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.473 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:03:17.473 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.473 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:03:17.474 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.686294+0000", 2024-09-21T14:03:17.474 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:03:17.474 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.474 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.474 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:03:17.474 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.686294+0000", 2024-09-21T14:03:17.474 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:03:17.474 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.474 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.474 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:03:17.474 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.686295+0000", 2024-09-21T14:03:17.474 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2890000000000001e-06 2024-09-21T14:03:17.474 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.475 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.475 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:03:17.475 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.686300+0000", 2024-09-21T14:03:17.475 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1900000000000003e-06 2024-09-21T14:03:17.475 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.475 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.475 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:03:17.475 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.686332+0000", 2024-09-21T14:03:17.475 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1986000000000002e-05 2024-09-21T14:03:17.475 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.475 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.475 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:03:17.475 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.686341+0000", 2024-09-21T14:03:17.475 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1869999999999997e-06 2024-09-21T14:03:17.476 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.476 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.476 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:03:17.476 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.686860+0000", 2024-09-21T14:03:17.476 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00051973299999999998 2024-09-21T14:03:17.476 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.476 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.476 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:03:17.476 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.686870+0000", 2024-09-21T14:03:17.476 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2250000000000006e-06 2024-09-21T14:03:17.476 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.476 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.476 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:03:17.476 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.771686+0000", 2024-09-21T14:03:17.477 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.084816461 2024-09-21T14:03:17.477 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.477 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.477 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:03:17.477 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.771804+0000", 2024-09-21T14:03:17.477 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000117829 2024-09-21T14:03:17.477 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.477 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.477 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,3,6", 2024-09-21T14:03:17.477 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.771895+0000", 2024-09-21T14:03:17.477 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0903999999999996e-05 2024-09-21T14:03:17.477 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.477 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.477 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:17.478 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.777261+0000", 2024-09-21T14:03:17.478 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.005366081 2024-09-21T14:03:17.478 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.478 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.478 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:17.478 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.777313+0000", 2024-09-21T14:03:17.478 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1749000000000002e-05 2024-09-21T14:03:17.478 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.478 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.478 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:17.478 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.777359+0000", 2024-09-21T14:03:17.478 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6221999999999999e-05 2024-09-21T14:03:17.478 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.478 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.479 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:03:17.479 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.782926+0000", 2024-09-21T14:03:17.479 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0055667060000000003 2024-09-21T14:03:17.479 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.479 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.479 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:03:17.479 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.782961+0000", 2024-09-21T14:03:17.479 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5394e-05 2024-09-21T14:03:17.479 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.479 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.479 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:03:17.479 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.783031+0000", 2024-09-21T14:03:17.479 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0137999999999998e-05 2024-09-21T14:03:17.479 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:17.480 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:17.480 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:17.480 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.480 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.480 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232056 8.7 8:e847aa11:::benchmark_data_smithi049_109311_object105537:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T14:03:17.480 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.884592+0000", 2024-09-21T14:03:17.480 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 365.56670259399999, 2024-09-21T14:03:17.480 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.89863333199999995, 2024-09-21T14:03:17.480 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:03:17.480 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:03:17.480 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:03:17.480 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:03:17.480 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T14:03:17.480 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:03:17.481 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232056 2024-09-21T14:03:17.481 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.481 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:03:17.481 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.481 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:03:17.481 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.884592+0000", 2024-09-21T14:03:17.481 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:03:17.481 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.481 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.481 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:03:17.481 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.884592+0000", 2024-09-21T14:03:17.481 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:03:17.482 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.482 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.482 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:03:17.482 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.884594+0000", 2024-09-21T14:03:17.482 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1619999999999998e-06 2024-09-21T14:03:17.482 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.482 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.482 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:03:17.482 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.884602+0000", 2024-09-21T14:03:17.482 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4649999999999994e-06 2024-09-21T14:03:17.482 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.482 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.482 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:03:17.483 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.884625+0000", 2024-09-21T14:03:17.483 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2904e-05 2024-09-21T14:03:17.483 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.483 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.483 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:03:17.483 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.884635+0000", 2024-09-21T14:03:17.483 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8190000000000003e-06 2024-09-21T14:03:17.483 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.483 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.483 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:03:17.483 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.884711+0000", 2024-09-21T14:03:17.483 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5407000000000005e-05 2024-09-21T14:03:17.483 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.483 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.484 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:03:17.484 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.884721+0000", 2024-09-21T14:03:17.484 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0631999999999999e-05 2024-09-21T14:03:17.484 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.484 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.484 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:03:17.484 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.774310+0000", 2024-09-21T14:03:17.484 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.88958867699999999 2024-09-21T14:03:17.484 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.484 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.484 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:03:17.484 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.774481+0000", 2024-09-21T14:03:17.484 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00017151699999999999 2024-09-21T14:03:17.484 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.485 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.485 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,3,6", 2024-09-21T14:03:17.485 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.774664+0000", 2024-09-21T14:03:17.485 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00018272000000000001 2024-09-21T14:03:17.485 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.485 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.485 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:17.485 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.782843+0000", 2024-09-21T14:03:17.485 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0081790089999999992 2024-09-21T14:03:17.485 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.485 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.485 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:17.485 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.782904+0000", 2024-09-21T14:03:17.485 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0591999999999999e-05 2024-09-21T14:03:17.486 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.486 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.486 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:03:17.486 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.783040+0000", 2024-09-21T14:03:17.486 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013623300000000001 2024-09-21T14:03:17.486 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.486 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.486 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:17.486 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.783085+0000", 2024-09-21T14:03:17.486 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4948999999999999e-05 2024-09-21T14:03:17.486 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.486 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.486 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:03:17.487 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.783122+0000", 2024-09-21T14:03:17.487 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7268000000000003e-05 2024-09-21T14:03:17.487 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.487 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.487 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:03:17.487 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.783225+0000", 2024-09-21T14:03:17.487 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000102978 2024-09-21T14:03:17.487 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:17.487 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:17.487 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:17.487 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.487 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.487 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25421.0:138802 9.d 9:b628721e:::benchmark_data_smithi049_138504_object17394:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-21T14:03:17.488 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:59:56.212970+0000", 2024-09-21T14:03:17.488 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 201.238324815, 2024-09-21T14:03:17.488 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.153997041, 2024-09-21T14:03:17.488 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:03:17.488 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:03:17.488 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:03:17.488 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:03:17.488 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25421", 2024-09-21T14:03:17.488 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:03:17.488 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 138802 2024-09-21T14:03:17.488 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.488 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:03:17.488 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.488 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:03:17.489 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:56.212970+0000", 2024-09-21T14:03:17.489 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:03:17.489 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.489 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.489 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:03:17.489 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:56.212970+0000", 2024-09-21T14:03:17.489 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:03:17.489 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.489 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.489 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:03:17.489 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:56.212971+0000", 2024-09-21T14:03:17.489 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2190000000000001e-06 2024-09-21T14:03:17.489 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.489 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.490 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:03:17.490 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:56.212975+0000", 2024-09-21T14:03:17.490 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0799999999999999e-06 2024-09-21T14:03:17.490 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.490 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.490 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:03:17.490 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:56.212985+0000", 2024-09-21T14:03:17.490 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0149000000000001e-05 2024-09-21T14:03:17.490 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.490 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.490 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:03:17.490 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:56.212992+0000", 2024-09-21T14:03:17.490 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.046e-06 2024-09-21T14:03:17.491 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.491 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.491 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:03:17.491 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:56.213149+0000", 2024-09-21T14:03:17.491 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00015712500000000001 2024-09-21T14:03:17.491 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.491 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.491 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:03:17.491 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:56.213286+0000", 2024-09-21T14:03:17.491 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000136501 2024-09-21T14:03:17.491 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.491 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.491 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,7", 2024-09-21T14:03:17.491 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:56.213416+0000", 2024-09-21T14:03:17.492 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000130097 2024-09-21T14:03:17.492 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.492 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.492 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:03:17.492 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:56.355414+0000", 2024-09-21T14:03:17.492 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.141998605 2024-09-21T14:03:17.492 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.492 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.492 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:17.492 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:56.356856+0000", 2024-09-21T14:03:17.492 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0014415610000000001 2024-09-21T14:03:17.492 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.492 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.493 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:17.493 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:56.366798+0000", 2024-09-21T14:03:17.493 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0099419250000000008 2024-09-21T14:03:17.493 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.493 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.493 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:03:17.493 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:56.366854+0000", 2024-09-21T14:03:17.493 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5869999999999999e-05 2024-09-21T14:03:17.493 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.493 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.493 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:03:17.493 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:56.366967+0000", 2024-09-21T14:03:17.493 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011286299999999999 2024-09-21T14:03:17.493 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:17.494 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:17.494 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:17.494 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.494 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.494 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25421.0:138808 9.0 9:05e48645:::benchmark_data_smithi049_138504_object17400:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-21T14:03:17.494 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:59:56.214820+0000", 2024-09-21T14:03:17.494 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 201.23647425300001, 2024-09-21T14:03:17.494 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.15134097899999999, 2024-09-21T14:03:17.494 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:03:17.494 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:03:17.494 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:03:17.494 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:03:17.494 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25421", 2024-09-21T14:03:17.494 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:03:17.495 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 138808 2024-09-21T14:03:17.495 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.495 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:03:17.495 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.495 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:03:17.495 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:56.214820+0000", 2024-09-21T14:03:17.495 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:03:17.495 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.495 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.495 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:03:17.495 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:56.214820+0000", 2024-09-21T14:03:17.495 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:03:17.495 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.495 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.496 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:03:17.496 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:56.214821+0000", 2024-09-21T14:03:17.496 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2e-07 2024-09-21T14:03:17.496 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.496 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.496 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:03:17.496 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:56.214823+0000", 2024-09-21T14:03:17.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:17.497 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:17.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:03:17.497 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5670000000000002e-06 2024-09-21T14:03:17.498 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.498 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.498 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:03:17.498 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:56.214830+0000", 2024-09-21T14:03:17.498 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7839999999999998e-06 2024-09-21T14:03:17.498 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.498 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.498 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:03:17.498 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:56.214835+0000", 2024-09-21T14:03:17.498 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3580000000000002e-06 2024-09-21T14:03:17.498 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.498 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.498 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:03:17.499 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:56.356223+0000", 2024-09-21T14:03:17.499 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.14138791000000001 2024-09-21T14:03:17.499 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.499 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.499 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:03:17.499 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:56.356528+0000", 2024-09-21T14:03:17.499 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00030416999999999998 2024-09-21T14:03:17.499 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.499 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.499 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,6", 2024-09-21T14:03:17.499 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:56.356657+0000", 2024-09-21T14:03:17.499 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012965000000000001 2024-09-21T14:03:17.499 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.500 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.500 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:03:17.500 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:56.365359+0000", 2024-09-21T14:03:17.500 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0087019780000000008 2024-09-21T14:03:17.500 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.500 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.500 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:17.500 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:56.365621+0000", 2024-09-21T14:03:17.500 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00026207999999999998 2024-09-21T14:03:17.500 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.500 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.501 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:17.501 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:56.366066+0000", 2024-09-21T14:03:17.501 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00044490000000000003 2024-09-21T14:03:17.501 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.501 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.501 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:03:17.501 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:56.366126+0000", 2024-09-21T14:03:17.501 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9590999999999999e-05 2024-09-21T14:03:17.501 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.501 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.501 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:03:17.501 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:56.366161+0000", 2024-09-21T14:03:17.501 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5471e-05 2024-09-21T14:03:17.501 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:17.502 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:17.502 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:17.502 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.502 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.502 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25421.0:172314 9.4 9:2b749f7e:::benchmark_data_smithi049_138504_object50906:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-09-21T14:03:17.502 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:00:33.178595+0000", 2024-09-21T14:03:17.502 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 164.272699313, 2024-09-21T14:03:17.502 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.342539818, 2024-09-21T14:03:17.502 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:03:17.502 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:03:17.502 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:03:17.502 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:03:17.502 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25421", 2024-09-21T14:03:17.503 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:03:17.503 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 172314 2024-09-21T14:03:17.503 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.503 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:03:17.503 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.503 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:03:17.503 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.178595+0000", 2024-09-21T14:03:17.503 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:03:17.503 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.503 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.503 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:03:17.503 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.178595+0000", 2024-09-21T14:03:17.503 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:03:17.504 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.504 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.504 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:03:17.504 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.178596+0000", 2024-09-21T14:03:17.504 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5599999999999995e-07 2024-09-21T14:03:17.504 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.504 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.504 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:03:17.504 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.178599+0000", 2024-09-21T14:03:17.504 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3919999999999999e-06 2024-09-21T14:03:17.504 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.504 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.504 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:03:17.504 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.178607+0000", 2024-09-21T14:03:17.505 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3780000000000001e-06 2024-09-21T14:03:17.505 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.505 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.505 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:03:17.505 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.178613+0000", 2024-09-21T14:03:17.505 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8629999999999996e-06 2024-09-21T14:03:17.505 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.505 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.505 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:03:17.505 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.178862+0000", 2024-09-21T14:03:17.505 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00024909299999999998 2024-09-21T14:03:17.505 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.505 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.505 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:03:17.506 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.178965+0000", 2024-09-21T14:03:17.506 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000102204 2024-09-21T14:03:17.506 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.506 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.506 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,4", 2024-09-21T14:03:17.506 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.179255+0000", 2024-09-21T14:03:17.506 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00029060400000000003 2024-09-21T14:03:17.506 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.506 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.506 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:03:17.506 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.181536+0000", 2024-09-21T14:03:17.506 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0022811630000000001 2024-09-21T14:03:17.506 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.506 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.507 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:17.507 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.187746+0000", 2024-09-21T14:03:17.507 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0062093790000000001 2024-09-21T14:03:17.507 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.507 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.507 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:17.507 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.521042+0000", 2024-09-21T14:03:17.507 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.33329632300000001 2024-09-21T14:03:17.507 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.507 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.507 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:03:17.507 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.521095+0000", 2024-09-21T14:03:17.507 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3365999999999997e-05 2024-09-21T14:03:17.507 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.508 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.508 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:03:17.508 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.521135+0000", 2024-09-21T14:03:17.508 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9496999999999998e-05 2024-09-21T14:03:17.508 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:17.508 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:17.508 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:17.508 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.508 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.508 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25421.0:172319 9.4 9:21e132db:::benchmark_data_smithi049_138504_object50911:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-09-21T14:03:17.508 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:00:33.184640+0000", 2024-09-21T14:03:17.508 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 164.26665398099999, 2024-09-21T14:03:17.508 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.338985429, 2024-09-21T14:03:17.508 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:03:17.509 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:03:17.509 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:03:17.509 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:03:17.509 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25421", 2024-09-21T14:03:17.509 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:03:17.509 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 172319 2024-09-21T14:03:17.509 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.509 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:03:17.509 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.509 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:03:17.509 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.184640+0000", 2024-09-21T14:03:17.509 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:03:17.509 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.509 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.510 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:03:17.510 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.184640+0000", 2024-09-21T14:03:17.510 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:03:17.510 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.510 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.510 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:03:17.510 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.184642+0000", 2024-09-21T14:03:17.510 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.488e-06 2024-09-21T14:03:17.510 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.510 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.510 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:03:17.510 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.184647+0000", 2024-09-21T14:03:17.510 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4e-06 2024-09-21T14:03:17.510 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.510 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.511 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:03:17.511 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.184679+0000", 2024-09-21T14:03:17.511 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1482000000000001e-05 2024-09-21T14:03:17.511 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.511 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.511 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:03:17.511 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.184688+0000", 2024-09-21T14:03:17.511 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0380000000000005e-06 2024-09-21T14:03:17.511 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.511 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.511 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:03:17.511 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.184742+0000", 2024-09-21T14:03:17.511 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3888999999999999e-05 2024-09-21T14:03:17.512 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.512 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.512 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:03:17.512 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.184861+0000", 2024-09-21T14:03:17.512 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011920499999999999 2024-09-21T14:03:17.512 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.512 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.512 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,4", 2024-09-21T14:03:17.512 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.185030+0000", 2024-09-21T14:03:17.512 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016910499999999999 2024-09-21T14:03:17.512 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.512 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.512 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:03:17.512 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.187701+0000", 2024-09-21T14:03:17.513 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0026706659999999999 2024-09-21T14:03:17.513 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.513 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.513 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:17.513 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.188554+0000", 2024-09-21T14:03:17.513 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00085298600000000002 2024-09-21T14:03:17.513 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.513 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.513 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:17.513 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.523534+0000", 2024-09-21T14:03:17.513 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.33498041099999998 2024-09-21T14:03:17.513 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.513 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.514 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:03:17.514 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.523587+0000", 2024-09-21T14:03:17.514 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2604e-05 2024-09-21T14:03:17.514 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.514 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.514 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:03:17.514 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.523626+0000", 2024-09-21T14:03:17.514 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9155e-05 2024-09-21T14:03:17.514 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:17.514 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:17.514 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:17.514 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.514 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.515 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25421.0:172326 9.4 9:22a8fc8a:::benchmark_data_smithi049_138504_object50918:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-09-21T14:03:17.515 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:00:33.254009+0000", 2024-09-21T14:03:17.515 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 164.19728567799999, 2024-09-21T14:03:17.515 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.269760204, 2024-09-21T14:03:17.515 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:03:17.515 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:03:17.515 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:03:17.515 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:03:17.515 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25421", 2024-09-21T14:03:17.515 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:03:17.515 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 172326 2024-09-21T14:03:17.515 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.515 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:03:17.515 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.515 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:03:17.516 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254009+0000", 2024-09-21T14:03:17.516 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:03:17.516 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.516 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.516 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:03:17.516 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254009+0000", 2024-09-21T14:03:17.516 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:03:17.516 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.516 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.516 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:03:17.516 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254010+0000", 2024-09-21T14:03:17.516 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.42e-06 2024-09-21T14:03:17.516 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.516 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.517 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:03:17.517 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254014+0000", 2024-09-21T14:03:17.517 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9550000000000002e-06 2024-09-21T14:03:17.517 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.517 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.517 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:03:17.517 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254023+0000", 2024-09-21T14:03:17.517 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3330000000000003e-06 2024-09-21T14:03:17.517 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.517 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.517 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:03:17.517 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254030+0000", 2024-09-21T14:03:17.517 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5719999999999999e-06 2024-09-21T14:03:17.517 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.517 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.518 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:03:17.518 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254298+0000", 2024-09-21T14:03:17.518 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00026834 2024-09-21T14:03:17.518 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.518 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.518 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:03:17.518 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254412+0000", 2024-09-21T14:03:17.518 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011395000000000001 2024-09-21T14:03:17.518 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.518 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.518 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,4", 2024-09-21T14:03:17.518 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254540+0000", 2024-09-21T14:03:17.518 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012807199999999999 2024-09-21T14:03:17.519 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.519 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.519 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:17.519 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.259890+0000", 2024-09-21T14:03:17.519 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0053499230000000004 2024-09-21T14:03:17.519 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.519 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.519 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:03:17.519 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.259900+0000", 2024-09-21T14:03:17.519 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8919999999999995e-06 2024-09-21T14:03:17.519 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.519 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.519 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:17.519 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.523677+0000", 2024-09-21T14:03:17.520 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26377665300000003 2024-09-21T14:03:17.520 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.520 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.520 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:03:17.520 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.523732+0000", 2024-09-21T14:03:17.520 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5605e-05 2024-09-21T14:03:17.520 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.520 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.520 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:03:17.520 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.523769+0000", 2024-09-21T14:03:17.520 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6489000000000002e-05 2024-09-21T14:03:17.520 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:17.520 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:17.521 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:17.521 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.521 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.521 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25421.0:172330 9.4 9:2882ed62:::benchmark_data_smithi049_138504_object50922:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-09-21T14:03:17.521 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:00:33.254698+0000", 2024-09-21T14:03:17.521 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 164.19659615500001, 2024-09-21T14:03:17.521 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26919505599999999, 2024-09-21T14:03:17.521 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:03:17.521 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:03:17.521 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:03:17.521 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:03:17.521 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25421", 2024-09-21T14:03:17.521 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:03:17.521 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 172330 2024-09-21T14:03:17.522 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.522 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:03:17.522 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.522 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:03:17.522 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254698+0000", 2024-09-21T14:03:17.522 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:03:17.522 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.522 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.522 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:03:17.522 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254698+0000", 2024-09-21T14:03:17.522 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:03:17.522 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.522 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.523 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:03:17.523 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254699+0000", 2024-09-21T14:03:17.523 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3799999999999998e-07 2024-09-21T14:03:17.523 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.523 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.523 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:03:17.523 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254701+0000", 2024-09-21T14:03:17.523 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5409999999999999e-06 2024-09-21T14:03:17.523 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.523 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.523 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:03:17.523 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254708+0000", 2024-09-21T14:03:17.523 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6089999999999997e-06 2024-09-21T14:03:17.523 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.524 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.524 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:03:17.524 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254713+0000", 2024-09-21T14:03:17.524 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2449999999999998e-06 2024-09-21T14:03:17.524 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.524 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.524 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:03:17.524 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.257115+0000", 2024-09-21T14:03:17.524 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0024021030000000001 2024-09-21T14:03:17.524 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.524 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.524 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:03:17.524 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.257220+0000", 2024-09-21T14:03:17.524 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010508 2024-09-21T14:03:17.525 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.525 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.525 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,4", 2024-09-21T14:03:17.525 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.257330+0000", 2024-09-21T14:03:17.525 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000110165 2024-09-21T14:03:17.525 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.525 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.525 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:03:17.525 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.260045+0000", 2024-09-21T14:03:17.525 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0027141890000000001 2024-09-21T14:03:17.525 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.525 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.526 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:17.526 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.274124+0000", 2024-09-21T14:03:17.526 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.014079473 2024-09-21T14:03:17.526 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.526 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.526 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:17.526 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.523815+0000", 2024-09-21T14:03:17.526 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.249690836 2024-09-21T14:03:17.527 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.527 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.527 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:03:17.527 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.523860+0000", 2024-09-21T14:03:17.527 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4935999999999998e-05 2024-09-21T14:03:17.527 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.527 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.527 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:03:17.527 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.523893+0000", 2024-09-21T14:03:17.527 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3441000000000001e-05 2024-09-21T14:03:17.527 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:17.527 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:17.527 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:17.527 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.528 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.528 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25421.0:172334 9.4 9:24fa2bd3:::benchmark_data_smithi049_138504_object50926:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-09-21T14:03:17.528 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:00:33.260571+0000", 2024-09-21T14:03:17.528 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 164.190722999, 2024-09-21T14:03:17.528 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26343834999999999, 2024-09-21T14:03:17.528 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:03:17.528 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:03:17.528 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:03:17.528 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:03:17.528 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25421", 2024-09-21T14:03:17.528 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:03:17.528 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 172334 2024-09-21T14:03:17.528 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.529 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:03:17.529 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.529 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:03:17.529 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.260571+0000", 2024-09-21T14:03:17.529 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:03:17.529 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.529 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.529 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:03:17.529 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.260571+0000", 2024-09-21T14:03:17.529 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:03:17.529 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.529 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.529 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:03:17.530 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.260573+0000", 2024-09-21T14:03:17.530 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2839999999999999e-06 2024-09-21T14:03:17.530 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.530 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.530 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:03:17.530 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.260577+0000", 2024-09-21T14:03:17.530 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5940000000000004e-06 2024-09-21T14:03:17.530 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.530 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.530 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:03:17.530 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.260588+0000", 2024-09-21T14:03:17.530 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1163e-05 2024-09-21T14:03:17.530 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.531 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.531 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:03:17.531 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.260596+0000", 2024-09-21T14:03:17.531 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0520000000000004e-06 2024-09-21T14:03:17.531 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.531 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.531 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:03:17.531 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.260646+0000", 2024-09-21T14:03:17.531 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0593999999999997e-05 2024-09-21T14:03:17.531 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.531 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.531 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:03:17.531 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.260743+0000", 2024-09-21T14:03:17.532 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6998999999999997e-05 2024-09-21T14:03:17.532 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.532 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.532 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,4", 2024-09-21T14:03:17.532 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.260842+0000", 2024-09-21T14:03:17.532 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9185999999999995e-05 2024-09-21T14:03:17.532 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.532 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.532 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:03:17.532 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.274135+0000", 2024-09-21T14:03:17.532 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.013293142000000001 2024-09-21T14:03:17.532 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.532 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.532 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:17.533 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.274165+0000", 2024-09-21T14:03:17.533 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9277000000000001e-05 2024-09-21T14:03:17.533 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.533 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.533 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:17.533 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.523933+0000", 2024-09-21T14:03:17.533 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.249768038 2024-09-21T14:03:17.533 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.533 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.533 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:03:17.533 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.523976+0000", 2024-09-21T14:03:17.533 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3120000000000001e-05 2024-09-21T14:03:17.533 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.533 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.534 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:03:17.534 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.524010+0000", 2024-09-21T14:03:17.534 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3900999999999999e-05 2024-09-21T14:03:17.534 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:17.534 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:17.534 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:17.534 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.534 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.534 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25421.0:172337 9.3 9:c4d16778:::benchmark_data_smithi049_138504_object50929:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-09-21T14:03:17.534 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:00:33.262471+0000", 2024-09-21T14:03:17.534 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 164.188823923, 2024-09-21T14:03:17.534 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26159086300000001, 2024-09-21T14:03:17.534 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:03:17.534 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:03:17.535 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:03:17.535 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:03:17.535 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25421", 2024-09-21T14:03:17.535 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:03:17.535 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 172337 2024-09-21T14:03:17.535 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.535 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:03:17.535 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.535 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:03:17.535 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.262471+0000", 2024-09-21T14:03:17.535 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:03:17.535 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.535 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.535 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:03:17.536 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.262471+0000", 2024-09-21T14:03:17.536 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:03:17.536 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.536 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.536 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:03:17.536 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.262471+0000", 2024-09-21T14:03:17.536 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2500000000000004e-07 2024-09-21T14:03:17.536 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.536 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.536 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:03:17.536 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.262835+0000", 2024-09-21T14:03:17.536 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00036389300000000001 2024-09-21T14:03:17.536 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.536 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.536 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:03:17.537 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.262844+0000", 2024-09-21T14:03:17.537 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0299999999999999e-06 2024-09-21T14:03:17.537 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.537 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.537 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:03:17.537 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.262851+0000", 2024-09-21T14:03:17.537 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3600000000000001e-06 2024-09-21T14:03:17.537 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.537 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.537 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:03:17.537 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.263740+0000", 2024-09-21T14:03:17.537 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00088968700000000001 2024-09-21T14:03:17.537 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.537 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.538 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:03:17.538 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.273507+0000", 2024-09-21T14:03:17.538 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0097663909999999993 2024-09-21T14:03:17.538 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.538 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.538 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,7", 2024-09-21T14:03:17.538 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.273631+0000", 2024-09-21T14:03:17.538 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012441500000000001 2024-09-21T14:03:17.538 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.538 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.538 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:03:17.538 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.279244+0000", 2024-09-21T14:03:17.538 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0056131870000000004 2024-09-21T14:03:17.538 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.539 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.539 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:17.539 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.283719+0000", 2024-09-21T14:03:17.539 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0044745640000000003 2024-09-21T14:03:17.539 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.539 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.539 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:17.539 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.523913+0000", 2024-09-21T14:03:17.539 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.24019366 2024-09-21T14:03:17.539 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.539 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.539 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:03:17.539 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.523964+0000", 2024-09-21T14:03:17.539 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1039999999999999e-05 2024-09-21T14:03:17.540 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.540 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.540 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:03:17.540 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.524061+0000", 2024-09-21T14:03:17.540 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7711e-05 2024-09-21T14:03:17.540 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:17.540 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:17.540 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:17.540 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.540 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.540 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25421.0:172338 9.3 9:cb49996e:::benchmark_data_smithi049_138504_object50930:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-09-21T14:03:17.540 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:00:33.262872+0000", 2024-09-21T14:03:17.540 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 164.18842241799999, 2024-09-21T14:03:17.540 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26143280099999999, 2024-09-21T14:03:17.541 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:03:17.541 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:03:17.541 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:03:17.541 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:03:17.541 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25421", 2024-09-21T14:03:17.541 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:03:17.541 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 172338 2024-09-21T14:03:17.541 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.541 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:03:17.541 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.541 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:03:17.541 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.262872+0000", 2024-09-21T14:03:17.541 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:03:17.542 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.542 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.542 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:03:17.542 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.262872+0000", 2024-09-21T14:03:17.542 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:03:17.542 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.542 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.542 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:03:17.542 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.262873+0000", 2024-09-21T14:03:17.542 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5300000000000004e-07 2024-09-21T14:03:17.542 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.542 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.542 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:03:17.542 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.262875+0000", 2024-09-21T14:03:17.543 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.706e-06 2024-09-21T14:03:17.543 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.543 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.543 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:03:17.543 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.262882+0000", 2024-09-21T14:03:17.543 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4989999999999999e-06 2024-09-21T14:03:17.543 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.543 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.543 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:03:17.543 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.262887+0000", 2024-09-21T14:03:17.543 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9509999999999997e-06 2024-09-21T14:03:17.543 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.543 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.543 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:03:17.544 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.276120+0000", 2024-09-21T14:03:17.544 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.013233300999999999 2024-09-21T14:03:17.544 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.544 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.544 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:03:17.544 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.276230+0000", 2024-09-21T14:03:17.544 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000110229 2024-09-21T14:03:17.544 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.544 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.544 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,7", 2024-09-21T14:03:17.544 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.276349+0000", 2024-09-21T14:03:17.544 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011883900000000001 2024-09-21T14:03:17.544 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.544 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.545 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:03:17.545 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.283223+0000", 2024-09-21T14:03:17.545 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.006873891 2024-09-21T14:03:17.545 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.545 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.545 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:17.545 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.283857+0000", 2024-09-21T14:03:17.545 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00063396899999999996 2024-09-21T14:03:17.545 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.545 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.545 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:17.545 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.524123+0000", 2024-09-21T14:03:17.545 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.24026641700000001 2024-09-21T14:03:17.546 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.546 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.546 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:03:17.546 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.524197+0000", 2024-09-21T14:03:17.546 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3915e-05 2024-09-21T14:03:17.546 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.546 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.546 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:03:17.546 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.524305+0000", 2024-09-21T14:03:17.546 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000107431 2024-09-21T14:03:17.546 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:17.546 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:17.546 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:17.546 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.547 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.547 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25421.0:187674 9.d 9:b1fcb922:::benchmark_data_smithi049_138504_object66266:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e350)", 2024-09-21T14:03:17.547 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:01:45.364290+0000", 2024-09-21T14:03:17.547 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 92.087004577000002, 2024-09-21T14:03:17.547 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.23297105300000001, 2024-09-21T14:03:17.547 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:03:17.547 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:03:17.547 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:03:17.547 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:03:17.547 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25421", 2024-09-21T14:03:17.547 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:03:17.547 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 187674 2024-09-21T14:03:17.547 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.547 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:03:17.548 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.548 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:03:17.548 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:01:45.364290+0000", 2024-09-21T14:03:17.548 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:03:17.548 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.548 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.548 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:03:17.548 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:01:45.364290+0000", 2024-09-21T14:03:17.548 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:03:17.548 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.548 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.548 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:03:17.548 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:01:45.364292+0000", 2024-09-21T14:03:17.549 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5069999999999999e-06 2024-09-21T14:03:17.549 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.549 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.549 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:03:17.549 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:01:45.364302+0000", 2024-09-21T14:03:17.549 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.764e-06 2024-09-21T14:03:17.549 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.549 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.549 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:03:17.549 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:01:45.364325+0000", 2024-09-21T14:03:17.549 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2481e-05 2024-09-21T14:03:17.549 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.549 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.549 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:03:17.550 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:01:45.364335+0000", 2024-09-21T14:03:17.550 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0591000000000001e-05 2024-09-21T14:03:17.550 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.550 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.550 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:03:17.550 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:01:45.364417+0000", 2024-09-21T14:03:17.550 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2250999999999996e-05 2024-09-21T14:03:17.550 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.550 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.550 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:03:17.550 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:01:45.364547+0000", 2024-09-21T14:03:17.550 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012936100000000001 2024-09-21T14:03:17.550 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.551 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.551 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,7", 2024-09-21T14:03:17.551 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:01:45.364720+0000", 2024-09-21T14:03:17.551 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000173139 2024-09-21T14:03:17.551 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.551 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.551 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:03:17.551 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:01:45.367514+0000", 2024-09-21T14:03:17.551 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0027938400000000001 2024-09-21T14:03:17.551 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.551 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.551 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:17.551 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:01:45.367749+0000", 2024-09-21T14:03:17.551 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000234792 2024-09-21T14:03:17.552 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.552 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.552 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:17.552 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:01:45.597049+0000", 2024-09-21T14:03:17.552 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22929991799999999 2024-09-21T14:03:17.552 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.552 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.552 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:03:17.552 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:01:45.597148+0000", 2024-09-21T14:03:17.552 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9321000000000007e-05 2024-09-21T14:03:17.552 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.552 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.552 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:03:17.553 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:01:45.597261+0000", 2024-09-21T14:03:17.553 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000113088 2024-09-21T14:03:17.553 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:17.553 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:17.553 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:17.553 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.553 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.553 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25421.0:187697 9.d 9:bc2d8559:::benchmark_data_smithi049_138504_object66289:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e350)", 2024-09-21T14:03:17.553 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:01:45.452843+0000", 2024-09-21T14:03:17.553 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 91.998451697999997, 2024-09-21T14:03:17.553 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.15371763299999999, 2024-09-21T14:03:17.553 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:03:17.553 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:03:17.553 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:03:17.553 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:03:17.554 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25421", 2024-09-21T14:03:17.554 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:03:17.554 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 187697 2024-09-21T14:03:17.554 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.555 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:03:17.555 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.555 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:03:17.555 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:01:45.452843+0000", 2024-09-21T14:03:17.555 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:03:17.555 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.555 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.555 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:03:17.555 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:01:45.452843+0000", 2024-09-21T14:03:17.555 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:03:17.555 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.555 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.555 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:03:17.555 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:01:45.452844+0000", 2024-09-21T14:03:17.556 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0750000000000001e-06 2024-09-21T14:03:17.556 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.556 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.556 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:03:17.556 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:01:45.452848+0000", 2024-09-21T14:03:17.556 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1989999999999999e-06 2024-09-21T14:03:17.556 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.556 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.556 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:03:17.556 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:01:45.452860+0000", 2024-09-21T14:03:17.556 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2435999999999999e-05 2024-09-21T14:03:17.556 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.556 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.557 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:03:17.557 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:01:45.452868+0000", 2024-09-21T14:03:17.557 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4399999999999999e-06 2024-09-21T14:03:17.557 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.557 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.557 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:03:17.557 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:01:45.452919+0000", 2024-09-21T14:03:17.557 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0658000000000002e-05 2024-09-21T14:03:17.557 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.557 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.557 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:03:17.557 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:01:45.452995+0000", 2024-09-21T14:03:17.557 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6448999999999999e-05 2024-09-21T14:03:17.557 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.558 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.558 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,7", 2024-09-21T14:03:17.558 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:01:45.453075+0000", 2024-09-21T14:03:17.558 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9943000000000004e-05 2024-09-21T14:03:17.558 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.558 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.558 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:03:17.558 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:01:45.455073+0000", 2024-09-21T14:03:17.558 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0019981669999999999 2024-09-21T14:03:17.558 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.558 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.558 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:17.558 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:01:45.455911+0000", 2024-09-21T14:03:17.559 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00083750299999999997 2024-09-21T14:03:17.559 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.559 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.559 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:17.559 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:01:45.606398+0000", 2024-09-21T14:03:17.559 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.15048757900000001 2024-09-21T14:03:17.559 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.559 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.559 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:03:17.559 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:01:45.606445+0000", 2024-09-21T14:03:17.559 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6950999999999998e-05 2024-09-21T14:03:17.559 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.559 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.559 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:03:17.560 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:01:45.606560+0000", 2024-09-21T14:03:17.560 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000115233 2024-09-21T14:03:17.560 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:17.560 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:17.560 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:17.560 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.560 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.560 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25421.0:212251 9.3 e367/321 9:c5428cfe:::benchmark_data_smithi049_138504_object90843:head v 367'13243, mlcod=367'13240)", 2024-09-21T14:03:17.560 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:02:16.942631+0000", 2024-09-21T14:03:17.560 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 60.508663593999998, 2024-09-21T14:03:17.560 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.12989590400000001, 2024-09-21T14:03:17.560 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:03:17.560 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:03:17.561 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:03:17.561 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:03:17.561 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.561 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:03:17.561 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:16.942631+0000", 2024-09-21T14:03:17.561 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:03:17.561 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.561 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.561 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:03:17.561 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:16.942631+0000", 2024-09-21T14:03:17.561 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:03:17.561 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.561 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.561 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:03:17.562 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:16.942632+0000", 2024-09-21T14:03:17.562 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2699999999999999e-06 2024-09-21T14:03:17.562 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.562 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.562 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:03:17.562 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:16.942639+0000", 2024-09-21T14:03:17.562 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1659999999999997e-06 2024-09-21T14:03:17.562 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.562 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.562 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:03:17.562 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:16.942652+0000", 2024-09-21T14:03:17.562 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2593e-05 2024-09-21T14:03:17.562 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.562 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.563 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:03:17.563 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:16.942659+0000", 2024-09-21T14:03:17.563 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4649999999999997e-06 2024-09-21T14:03:17.563 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.563 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.563 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:03:17.563 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:16.947475+0000", 2024-09-21T14:03:17.563 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0048152029999999997 2024-09-21T14:03:17.563 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.563 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.563 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:03:17.563 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:16.947506+0000", 2024-09-21T14:03:17.563 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1154e-05 2024-09-21T14:03:17.563 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.563 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.564 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:03:17.564 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:17.072427+0000", 2024-09-21T14:03:17.564 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.124921768 2024-09-21T14:03:17.564 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.564 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.564 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:03:17.564 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:17.072527+0000", 2024-09-21T14:03:17.564 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9284999999999996e-05 2024-09-21T14:03:17.564 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:17.564 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:17.564 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:17.564 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.564 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.564 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25421.0:212266 9.d 9:b4489ba5:::benchmark_data_smithi049_138504_object90858:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e367)", 2024-09-21T14:03:17.565 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:02:16.949472+0000", 2024-09-21T14:03:17.565 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 60.501822210999997, 2024-09-21T14:03:17.565 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.12987005200000001, 2024-09-21T14:03:17.565 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:03:17.565 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:03:17.565 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:03:17.565 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:03:17.565 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25421", 2024-09-21T14:03:17.565 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:03:17.565 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 212266 2024-09-21T14:03:17.565 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.565 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:03:17.565 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.566 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:03:17.566 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:16.949472+0000", 2024-09-21T14:03:17.566 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:03:17.566 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.566 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.566 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:03:17.566 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:16.949472+0000", 2024-09-21T14:03:17.566 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:03:17.566 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.566 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.566 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:03:17.566 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:16.949475+0000", 2024-09-21T14:03:17.566 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.65e-06 2024-09-21T14:03:17.567 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.567 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.567 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:03:17.567 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:16.949481+0000", 2024-09-21T14:03:17.567 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0449999999999997e-06 2024-09-21T14:03:17.567 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.567 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.567 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:03:17.567 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:16.949501+0000", 2024-09-21T14:03:17.567 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0007e-05 2024-09-21T14:03:17.567 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.567 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.567 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:03:17.567 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:16.949509+0000", 2024-09-21T14:03:17.567 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5569999999999996e-06 2024-09-21T14:03:17.568 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.568 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.568 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:03:17.568 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:16.959031+0000", 2024-09-21T14:03:17.568 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0095210369999999996 2024-09-21T14:03:17.568 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.568 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.568 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:03:17.568 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:16.959204+0000", 2024-09-21T14:03:17.568 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00017299399999999999 2024-09-21T14:03:17.568 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.568 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.568 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,7", 2024-09-21T14:03:17.568 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:16.959345+0000", 2024-09-21T14:03:17.569 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014109699999999999 2024-09-21T14:03:17.569 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.569 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.569 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:17.569 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:17.079064+0000", 2024-09-21T14:03:17.569 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.119719719 2024-09-21T14:03:17.569 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.569 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.569 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:17.569 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:17.079109+0000", 2024-09-21T14:03:17.569 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4872e-05 2024-09-21T14:03:17.569 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.569 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.569 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:03:17.570 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:17.079122+0000", 2024-09-21T14:03:17.570 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2357999999999999e-05 2024-09-21T14:03:17.570 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.570 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.570 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:03:17.570 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:17.079173+0000", 2024-09-21T14:03:17.570 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0930999999999997e-05 2024-09-21T14:03:17.570 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.570 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.570 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:03:17.570 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:17.079342+0000", 2024-09-21T14:03:17.570 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016978499999999999 2024-09-21T14:03:17.570 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:17.571 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:17.571 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:17.571 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.571 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.571 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25421.0:213862 9.d 9:b68c085b:::benchmark_data_smithi049_138504_object92454:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e368)", 2024-09-21T14:03:17.571 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:02:18.554190+0000", 2024-09-21T14:03:17.571 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 58.897104890999998, 2024-09-21T14:03:17.571 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.133874673, 2024-09-21T14:03:17.571 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:03:17.571 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:03:17.571 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:03:17.571 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:03:17.571 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25421", 2024-09-21T14:03:17.571 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:03:17.571 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 213862 2024-09-21T14:03:17.572 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.572 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:03:17.572 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.572 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:03:17.572 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:18.554190+0000", 2024-09-21T14:03:17.572 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:03:17.572 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.572 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.572 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:03:17.572 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:18.554190+0000", 2024-09-21T14:03:17.572 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:03:17.572 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.572 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.572 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:03:17.573 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:18.554191+0000", 2024-09-21T14:03:17.573 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3820000000000001e-06 2024-09-21T14:03:17.573 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.573 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.573 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:03:17.573 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:18.554195+0000", 2024-09-21T14:03:17.573 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.267e-06 2024-09-21T14:03:17.573 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.573 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.573 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:03:17.573 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:18.554211+0000", 2024-09-21T14:03:17.573 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5358000000000001e-05 2024-09-21T14:03:17.573 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.573 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.573 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:03:17.574 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:18.554218+0000", 2024-09-21T14:03:17.574 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7840000000000004e-06 2024-09-21T14:03:17.574 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.574 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.574 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:03:17.574 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:18.554287+0000", 2024-09-21T14:03:17.574 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8806000000000005e-05 2024-09-21T14:03:17.574 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.574 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.574 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:03:17.574 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:18.554423+0000", 2024-09-21T14:03:17.574 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013583599999999999 2024-09-21T14:03:17.574 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.575 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.575 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,7", 2024-09-21T14:03:17.575 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:18.554583+0000", 2024-09-21T14:03:17.575 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000160123 2024-09-21T14:03:17.575 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.575 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.575 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:03:17.575 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:18.556781+0000", 2024-09-21T14:03:17.575 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0021979080000000002 2024-09-21T14:03:17.575 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.575 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.575 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:17.575 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:18.567388+0000", 2024-09-21T14:03:17.575 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.010607042000000001 2024-09-21T14:03:17.576 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.576 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.576 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:17.576 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:18.687870+0000", 2024-09-21T14:03:17.576 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.12048213300000001 2024-09-21T14:03:17.576 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.576 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.576 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:03:17.576 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:18.687941+0000", 2024-09-21T14:03:17.576 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0908000000000006e-05 2024-09-21T14:03:17.576 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.576 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.576 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:03:17.576 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:18.688064+0000", 2024-09-21T14:03:17.576 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012312600000000001 2024-09-21T14:03:17.577 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:17.577 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:17.577 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:17.577 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.577 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.577 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25421.0:213864 9.d 9:b994402e:::benchmark_data_smithi049_138504_object92456:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e368)", 2024-09-21T14:03:17.577 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:02:18.559275+0000", 2024-09-21T14:03:17.577 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 58.892019222999998, 2024-09-21T14:03:17.577 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.13393707599999999, 2024-09-21T14:03:17.577 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:03:17.577 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:03:17.577 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:03:17.577 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:03:17.577 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25421", 2024-09-21T14:03:17.577 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:03:17.578 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 213864 2024-09-21T14:03:17.578 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.578 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:03:17.578 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.578 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:03:17.578 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:18.559275+0000", 2024-09-21T14:03:17.578 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:03:17.578 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.578 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.578 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:03:17.578 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:18.559275+0000", 2024-09-21T14:03:17.578 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:03:17.578 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.579 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.579 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:03:17.579 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:18.559277+0000", 2024-09-21T14:03:17.579 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.28e-06 2024-09-21T14:03:17.579 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.579 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.579 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:03:17.579 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:18.559281+0000", 2024-09-21T14:03:17.579 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.425e-06 2024-09-21T14:03:17.579 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.579 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.579 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:03:17.579 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:18.559291+0000", 2024-09-21T14:03:17.579 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8679999999999994e-06 2024-09-21T14:03:17.579 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.580 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.580 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:03:17.580 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:18.559296+0000", 2024-09-21T14:03:17.580 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0370000000000002e-06 2024-09-21T14:03:17.580 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.580 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.580 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:03:17.580 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:18.559338+0000", 2024-09-21T14:03:17.580 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1989000000000001e-05 2024-09-21T14:03:17.580 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.580 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.580 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:03:17.580 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:18.559428+0000", 2024-09-21T14:03:17.580 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0432000000000004e-05 2024-09-21T14:03:17.581 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.581 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.581 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,7", 2024-09-21T14:03:17.581 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:18.559523+0000", 2024-09-21T14:03:17.581 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.5043000000000005e-05 2024-09-21T14:03:17.581 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.581 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.581 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:17.581 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:18.567422+0000", 2024-09-21T14:03:17.581 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0078987120000000004 2024-09-21T14:03:17.581 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.581 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.581 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:03:17.581 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:18.567723+0000", 2024-09-21T14:03:17.581 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00030147399999999998 2024-09-21T14:03:17.582 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.582 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.582 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:17.582 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:18.693018+0000", 2024-09-21T14:03:17.582 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.125294766 2024-09-21T14:03:17.582 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.582 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.582 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:03:17.582 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:18.693115+0000", 2024-09-21T14:03:17.582 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7076000000000004e-05 2024-09-21T14:03:17.582 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:17.582 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:17.582 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:03:17.582 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:02:18.693212+0000", 2024-09-21T14:03:17.583 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6973999999999997e-05 2024-09-21T14:03:17.583 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:17.583 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:17.583 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:17.583 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:17.583 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:17.583 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T14:03:17.598 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:03:17.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:17.637 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:17.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:03:17.738 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:03:17.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:17.770 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:17.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:03:17.867 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 12 16 32004 31988 166.581 138.188 0.00836108 0.00599769 2024-09-21T14:03:17.871 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:03:17.905 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:18.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:18.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:03:18.133 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":729,"stamp":"2024-09-21T14:03:12.000008+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":661050243,"num_objects":10280,"num_object_clones":0,"num_object_copies":30840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10280,"num_whiteouts":0,"num_read":1408,"num_read_kb":1227,"num_write":10876,"num_write_kb":646265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":169,"num_bytes_recovered":1032962,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11752,"ondisk_log_size":11752,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":475,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":4968336,"kb_used_data":1952128,"kb_used_omap":23,"kb_used_meta":3016040,"kb_avail":744992880,"statfs":{"total":767960285184,"available":762872709120,"internally_reserved":0,"allocated":1998979072,"data_stored":1991805293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24148,"internal_metadata":3088425388},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":660668416,"num_objects":10081,"num_object_clones":0,"num_object_copies":30243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10081,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10081,"num_write_kb":645184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003397"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":600,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.624828+0000","last_change":"2024-09-21T14:02:56.458231+0000","last_active":"2024-09-21T14:03:08.624828+0000","last_peered":"2024-09-21T14:03:08.624828+0000","last_clean":"2024-09-21T14:03:08.624828+0000","last_became_active":"2024-09-21T14:01:41.189398+0000","last_became_peered":"2024-09-21T14:01:41.189398+0000","last_unstale":"2024-09-21T14:03:08.624828+0000","last_undegraded":"2024-09-21T14:03:08.624828+0000","last_fullsized":"2024-09-21T14:03:08.624828+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:56.458125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:02:56.458125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:20.867402+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"290'15","reported_seq":765,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T14:03:07.610986+0000","last_change":"2024-09-21T14:01:34.785322+0000","last_active":"2024-09-21T14:03:07.610986+0000","last_peered":"2024-09-21T14:03:07.610986+0000","last_clean":"2024-09-21T14:03:07.610986+0000","last_became_active":"2024-09-21T13:58:17.771220+0000","last_became_peered":"2024-09-21T13:58:17.771220+0000","last_unstale":"2024-09-21T14:03:07.610986+0000","last_undegraded":"2024-09-21T14:03:07.610986+0000","last_fullsized":"2024-09-21T14:03:07.610986+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'15","last_scrub_stamp":"2024-09-21T14:01:34.785267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:34.785267+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:53.703961+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":708,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.614729+0000","last_change":"2024-09-21T14:02:29.205437+0000","last_active":"2024-09-21T14:03:08.614729+0000","last_peered":"2024-09-21T14:03:08.614729+0000","last_clean":"2024-09-21T14:03:08.614729+0000","last_became_active":"2024-09-21T14:01:41.202168+0000","last_became_peered":"2024-09-21T14:01:41.202168+0000","last_unstale":"2024-09-21T14:03:08.614729+0000","last_undegraded":"2024-09-21T14:03:08.614729+0000","last_fullsized":"2024-09-21T14:03:08.614729+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:29.205343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:02:29.205343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:29.369063+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":682,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T14:03:07.611258+0000","last_change":"2024-09-21T14:01:41.203762+0000","last_active":"2024-09-21T14:03:07.611258+0000","last_peered":"2024-09-21T14:03:07.611258+0000","last_clean":"2024-09-21T14:03:07.611258+0000","last_became_active":"2024-09-21T14:01:41.203275+0000","last_became_peered":"2024-09-21T14:01:41.203275+0000","last_unstale":"2024-09-21T14:03:07.611258+0000","last_undegraded":"2024-09-21T14:03:07.611258+0000","last_fullsized":"2024-09-21T14:03:07.611258+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:00.320510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:00.320510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:46.689692+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":654,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.619256+0000","last_change":"2024-09-21T13:59:32.208701+0000","last_active":"2024-09-21T14:03:08.619256+0000","last_peered":"2024-09-21T14:03:08.619256+0000","last_clean":"2024-09-21T14:03:08.619256+0000","last_became_active":"2024-09-21T13:59:32.203340+0000","last_became_peered":"2024-09-21T13:59:32.203340+0000","last_unstale":"2024-09-21T14:03:08.619256+0000","last_undegraded":"2024-09-21T14:03:08.619256+0000","last_fullsized":"2024-09-21T14:03:08.619256+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:34.696780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:34.696780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:44.274358+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":667,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.619744+0000","last_change":"2024-09-21T13:58:17.778322+0000","last_active":"2024-09-21T14:03:08.619744+0000","last_peered":"2024-09-21T14:03:08.619744+0000","last_clean":"2024-09-21T14:03:08.619744+0000","last_became_active":"2024-09-21T13:58:17.777893+0000","last_became_peered":"2024-09-21T13:58:17.777893+0000","last_unstale":"2024-09-21T14:03:08.619744+0000","last_undegraded":"2024-09-21T14:03:08.619744+0000","last_fullsized":"2024-09-21T14:03:08.619744+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:07.001135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:55:07.001135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:22.269018+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"290'27","reported_seq":640,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.617023+0000","last_change":"2024-09-21T13:58:11.718421+0000","last_active":"2024-09-21T14:03:08.617023+0000","last_peered":"2024-09-21T14:03:08.617023+0000","last_clean":"2024-09-21T14:03:08.617023+0000","last_became_active":"2024-09-21T13:57:12.703469+0000","last_became_peered":"2024-09-21T13:57:12.703469+0000","last_unstale":"2024-09-21T14:03:08.617023+0000","last_undegraded":"2024-09-21T14:03:08.617023+0000","last_fullsized":"2024-09-21T14:03:08.617023+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:58:11.718334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:11.718334+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":713,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T14:03:07.610955+0000","last_change":"2024-09-21T14:03:02.862866+0000","last_active":"2024-09-21T14:03:07.610955+0000","last_peered":"2024-09-21T14:03:07.610955+0000","last_clean":"2024-09-21T14:03:07.610955+0000","last_became_active":"2024-09-21T13:58:17.773464+0000","last_became_peered":"2024-09-21T13:58:17.773464+0000","last_unstale":"2024-09-21T14:03:07.610955+0000","last_undegraded":"2024-09-21T14:03:07.610955+0000","last_fullsized":"2024-09-21T14:03:07.610955+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:02.862787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:02.862787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:17.663483+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":659,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.624227+0000","last_change":"2024-09-21T14:01:41.194807+0000","last_active":"2024-09-21T14:03:08.624227+0000","last_peered":"2024-09-21T14:03:08.624227+0000","last_clean":"2024-09-21T14:03:08.624227+0000","last_became_active":"2024-09-21T14:01:41.194517+0000","last_became_peered":"2024-09-21T14:01:41.194517+0000","last_unstale":"2024-09-21T14:03:08.624227+0000","last_undegraded":"2024-09-21T14:03:08.624227+0000","last_fullsized":"2024-09-21T14:03:08.624227+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:31.127926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:31.127926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"290'15","reported_seq":731,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T14:03:07.611539+0000","last_change":"2024-09-21T14:00:02.475761+0000","last_active":"2024-09-21T14:03:07.611539+0000","last_peered":"2024-09-21T14:03:07.611539+0000","last_clean":"2024-09-21T14:03:07.611539+0000","last_became_active":"2024-09-21T13:57:12.681408+0000","last_became_peered":"2024-09-21T13:57:12.681408+0000","last_unstale":"2024-09-21T14:03:07.611539+0000","last_undegraded":"2024-09-21T14:03:07.611539+0000","last_fullsized":"2024-09-21T14:03:07.611539+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'15","last_scrub_stamp":"2024-09-21T14:00:02.475661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:00:02.475661+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:43.713097+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":608,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.623330+0000","last_change":"2024-09-21T14:03:04.516642+0000","last_active":"2024-09-21T14:03:08.623330+0000","last_peered":"2024-09-21T14:03:08.623330+0000","last_clean":"2024-09-21T14:03:08.623330+0000","last_became_active":"2024-09-21T14:01:41.188248+0000","last_became_peered":"2024-09-21T14:01:41.188248+0000","last_unstale":"2024-09-21T14:03:08.623330+0000","last_undegraded":"2024-09-21T14:03:08.623330+0000","last_fullsized":"2024-09-21T14:03:08.623330+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:04.516583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:04.516583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:26.082827+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":638,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.617379+0000","last_change":"2024-09-21T13:59:32.202128+0000","last_active":"2024-09-21T14:03:08.617379+0000","last_peered":"2024-09-21T14:03:08.617379+0000","last_clean":"2024-09-21T14:03:08.617379+0000","last_became_active":"2024-09-21T13:59:32.201158+0000","last_became_peered":"2024-09-21T13:59:32.201158+0000","last_unstale":"2024-09-21T14:03:08.617379+0000","last_undegraded":"2024-09-21T14:03:08.617379+0000","last_fullsized":"2024-09-21T14:03:08.617379+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:25.792985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:54:25.792985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:49.267653+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":636,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.623403+0000","last_change":"2024-09-21T14:01:41.189797+0000","last_active":"2024-09-21T14:03:08.623403+0000","last_peered":"2024-09-21T14:03:08.623403+0000","last_clean":"2024-09-21T14:03:08.623403+0000","last_became_active":"2024-09-21T14:01:41.189373+0000","last_became_peered":"2024-09-21T14:01:41.189373+0000","last_unstale":"2024-09-21T14:03:08.623403+0000","last_undegraded":"2024-09-21T14:03:08.623403+0000","last_fullsized":"2024-09-21T14:03:08.623403+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:54.328275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:54.328275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"290'7","reported_seq":756,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T14:03:07.611281+0000","last_change":"2024-09-21T14:01:57.563250+0000","last_active":"2024-09-21T14:03:07.611281+0000","last_peered":"2024-09-21T14:03:07.611281+0000","last_clean":"2024-09-21T14:03:07.611281+0000","last_became_active":"2024-09-21T13:59:09.063053+0000","last_became_peered":"2024-09-21T13:59:09.063053+0000","last_unstale":"2024-09-21T14:03:07.611281+0000","last_undegraded":"2024-09-21T14:03:07.611281+0000","last_fullsized":"2024-09-21T14:03:07.611281+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'7","last_scrub_stamp":"2024-09-21T14:01:57.563184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:57.563184+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:19.737603+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":629,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.625933+0000","last_change":"2024-09-21T13:59:32.204565+0000","last_active":"2024-09-21T14:03:08.625933+0000","last_peered":"2024-09-21T14:03:08.625933+0000","last_clean":"2024-09-21T14:03:08.625933+0000","last_became_active":"2024-09-21T13:59:32.204174+0000","last_became_peered":"2024-09-21T13:59:32.204174+0000","last_unstale":"2024-09-21T14:03:08.625933+0000","last_undegraded":"2024-09-21T14:03:08.625933+0000","last_fullsized":"2024-09-21T14:03:08.625933+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:06.722293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:06.722293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":670,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.622851+0000","last_change":"2024-09-21T14:01:41.199085+0000","last_active":"2024-09-21T14:03:08.622851+0000","last_peered":"2024-09-21T14:03:08.622851+0000","last_clean":"2024-09-21T14:03:08.622851+0000","last_became_active":"2024-09-21T14:01:41.198672+0000","last_became_peered":"2024-09-21T14:01:41.198672+0000","last_unstale":"2024-09-21T14:03:08.622851+0000","last_undegraded":"2024-09-21T14:03:08.622851+0000","last_fullsized":"2024-09-21T14:03:08.622851+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:44.376215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:44.376215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":725,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T14:03:07.613109+0000","last_change":"2024-09-21T14:01:53.460257+0000","last_active":"2024-09-21T14:03:07.613109+0000","last_peered":"2024-09-21T14:03:07.613109+0000","last_clean":"2024-09-21T14:03:07.613109+0000","last_became_active":"2024-09-21T14:01:41.200360+0000","last_became_peered":"2024-09-21T14:01:41.200360+0000","last_unstale":"2024-09-21T14:03:07.613109+0000","last_undegraded":"2024-09-21T14:03:07.613109+0000","last_fullsized":"2024-09-21T14:03:07.613109+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:53.460180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:01:53.460180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:15.802657+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":643,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T14:03:07.611541+0000","last_change":"2024-09-21T14:02:36.679396+0000","last_active":"2024-09-21T14:03:07.611541+0000","last_peered":"2024-09-21T14:03:07.611541+0000","last_clean":"2024-09-21T14:03:07.611541+0000","last_became_active":"2024-09-21T13:59:32.204749+0000","last_became_peered":"2024-09-21T13:59:32.204749+0000","last_unstale":"2024-09-21T14:03:07.611541+0000","last_undegraded":"2024-09-21T14:03:07.611541+0000","last_fullsized":"2024-09-21T14:03:07.611541+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:36.679321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:02:36.679321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:43.067225+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"290'27","reported_seq":707,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T14:03:07.609548+0000","last_change":"2024-09-21T14:02:55.620755+0000","last_active":"2024-09-21T14:03:07.609548+0000","last_peered":"2024-09-21T14:03:07.609548+0000","last_clean":"2024-09-21T14:03:07.609548+0000","last_became_active":"2024-09-21T13:57:12.680783+0000","last_became_peered":"2024-09-21T13:57:12.680783+0000","last_unstale":"2024-09-21T14:03:07.609548+0000","last_undegraded":"2024-09-21T14:03:07.609548+0000","last_fullsized":"2024-09-21T14:03:07.609548+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'27","last_scrub_stamp":"2024-09-21T14:02:55.620700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.620700+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:01.085594+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":702,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.616652+0000","last_change":"2024-09-21T14:03:08.299319+0000","last_active":"2024-09-21T14:03:08.616652+0000","last_peered":"2024-09-21T14:03:08.616652+0000","last_clean":"2024-09-21T14:03:08.616652+0000","last_became_active":"2024-09-21T13:49:02.066796+0000","last_became_peered":"2024-09-21T13:49:02.066796+0000","last_unstale":"2024-09-21T14:03:08.616652+0000","last_undegraded":"2024-09-21T14:03:08.616652+0000","last_fullsized":"2024-09-21T14:03:08.616652+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:08.299252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:08.299252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:29.982454+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":646,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.620854+0000","last_change":"2024-09-21T14:01:41.191486+0000","last_active":"2024-09-21T14:03:08.620854+0000","last_peered":"2024-09-21T14:03:08.620854+0000","last_clean":"2024-09-21T14:03:08.620854+0000","last_became_active":"2024-09-21T14:01:41.191116+0000","last_became_peered":"2024-09-21T14:01:41.191116+0000","last_unstale":"2024-09-21T14:03:08.620854+0000","last_undegraded":"2024-09-21T14:03:08.620854+0000","last_fullsized":"2024-09-21T14:03:08.620854+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:57.962266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:57.962266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":702,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T14:03:07.611661+0000","last_change":"2024-09-21T14:00:51.495433+0000","last_active":"2024-09-21T14:03:07.611661+0000","last_peered":"2024-09-21T14:03:07.611661+0000","last_clean":"2024-09-21T14:03:07.611661+0000","last_became_active":"2024-09-21T13:52:04.316016+0000","last_became_peered":"2024-09-21T13:52:04.316016+0000","last_unstale":"2024-09-21T14:03:07.611661+0000","last_undegraded":"2024-09-21T14:03:07.611661+0000","last_fullsized":"2024-09-21T14:03:07.611661+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:00:51.495368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:00:51.495368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:10.415465+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"290'18","reported_seq":652,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.620717+0000","last_change":"2024-09-21T13:59:23.041396+0000","last_active":"2024-09-21T14:03:08.620717+0000","last_peered":"2024-09-21T14:03:08.620717+0000","last_clean":"2024-09-21T14:03:08.620717+0000","last_became_active":"2024-09-21T13:57:12.684454+0000","last_became_peered":"2024-09-21T13:57:12.684454+0000","last_unstale":"2024-09-21T14:03:08.620717+0000","last_undegraded":"2024-09-21T14:03:08.620717+0000","last_fullsized":"2024-09-21T14:03:08.620717+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'18","last_scrub_stamp":"2024-09-21T13:59:23.041311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:23.041311+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":689,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T14:03:07.611791+0000","last_change":"2024-09-21T13:58:16.393763+0000","last_active":"2024-09-21T14:03:07.611791+0000","last_peered":"2024-09-21T14:03:07.611791+0000","last_clean":"2024-09-21T14:03:07.611791+0000","last_became_active":"2024-09-21T13:57:12.681897+0000","last_became_peered":"2024-09-21T13:57:12.681897+0000","last_unstale":"2024-09-21T14:03:07.611791+0000","last_undegraded":"2024-09-21T14:03:07.611791+0000","last_fullsized":"2024-09-21T14:03:07.611791+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:16.393696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:16.393696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:37.708367+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":613,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.626525+0000","last_change":"2024-09-21T14:01:41.182300+0000","last_active":"2024-09-21T14:03:08.626525+0000","last_peered":"2024-09-21T14:03:08.626525+0000","last_clean":"2024-09-21T14:03:08.626525+0000","last_became_active":"2024-09-21T14:01:41.181946+0000","last_became_peered":"2024-09-21T14:01:41.181946+0000","last_unstale":"2024-09-21T14:03:08.626525+0000","last_undegraded":"2024-09-21T14:03:08.626525+0000","last_fullsized":"2024-09-21T14:03:08.626525+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T13:57:26.536684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:26.536684+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":714,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.614861+0000","last_change":"2024-09-21T14:01:58.994867+0000","last_active":"2024-09-21T14:03:08.614861+0000","last_peered":"2024-09-21T14:03:08.614861+0000","last_clean":"2024-09-21T14:03:08.614861+0000","last_became_active":"2024-09-21T13:57:12.685742+0000","last_became_peered":"2024-09-21T13:57:12.685742+0000","last_unstale":"2024-09-21T14:03:08.614861+0000","last_undegraded":"2024-09-21T14:03:08.614861+0000","last_fullsized":"2024-09-21T14:03:08.614861+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T14:01:58.994811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:01:58.994811+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:17.687542+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"290'20","reported_seq":593,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.627153+0000","last_change":"2024-09-21T13:59:32.242771+0000","last_active":"2024-09-21T14:03:08.627153+0000","last_peered":"2024-09-21T14:03:08.627153+0000","last_clean":"2024-09-21T14:03:08.627153+0000","last_became_active":"2024-09-21T13:59:32.213865+0000","last_became_peered":"2024-09-21T13:59:32.213865+0000","last_unstale":"2024-09-21T14:03:08.627153+0000","last_undegraded":"2024-09-21T14:03:08.627153+0000","last_fullsized":"2024-09-21T14:03:08.627153+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:57:27.578738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:57:27.578738+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"300'12","reported_seq":332,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T14:03:07.608645+0000","last_change":"2024-09-21T13:59:36.012775+0000","last_active":"2024-09-21T14:03:07.608645+0000","last_peered":"2024-09-21T14:03:07.608645+0000","last_clean":"2024-09-21T14:03:07.608645+0000","last_became_active":"2024-09-21T13:59:32.205051+0000","last_became_peered":"2024-09-21T13:59:32.205051+0000","last_unstale":"2024-09-21T14:03:07.608645+0000","last_undegraded":"2024-09-21T14:03:07.608645+0000","last_fullsized":"2024-09-21T14:03:07.608645+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'12","last_scrub_stamp":"2024-09-21T13:59:36.012693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:36.012693+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:20.587646+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"387'570","reported_seq":593,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.600794+0000","last_change":"2024-09-21T14:02:56.560786+0000","last_active":"2024-09-21T14:03:08.600794+0000","last_peered":"2024-09-21T14:03:08.600794+0000","last_clean":"2024-09-21T14:03:08.600794+0000","last_became_active":"2024-09-21T14:02:56.559536+0000","last_became_peered":"2024-09-21T14:02:56.559536+0000","last_unstale":"2024-09-21T14:03:08.600794+0000","last_undegraded":"2024-09-21T14:03:08.600794+0000","last_fullsized":"2024-09-21T14:03:08.600794+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":570,"log_dups_size":0,"ondisk_log_size":570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:03.627467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37355520,"num_objects":570,"num_object_clones":0,"num_object_copies":1710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":570,"num_write_kb":36480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"75'79","reported_seq":109,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.623290+0000","last_change":"2024-09-21T14:01:41.248772+0000","last_active":"2024-09-21T14:03:08.623290+0000","last_peered":"2024-09-21T14:03:08.623290+0000","last_clean":"2024-09-21T14:03:08.623290+0000","last_became_active":"2024-09-21T14:01:41.246888+0000","last_became_peered":"2024-09-21T14:01:41.246888+0000","last_unstale":"2024-09-21T14:03:08.623290+0000","last_undegraded":"2024-09-21T14:03:08.623290+0000","last_fullsized":"2024-09-21T14:03:08.623290+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:03:52.423493+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53669,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"290'27","reported_seq":761,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.617486+0000","last_change":"2024-09-21T14:02:58.293228+0000","last_active":"2024-09-21T14:03:08.617486+0000","last_peered":"2024-09-21T14:03:08.617486+0000","last_clean":"2024-09-21T14:03:08.617486+0000","last_became_active":"2024-09-21T13:57:12.695609+0000","last_became_peered":"2024-09-21T13:57:12.695609+0000","last_unstale":"2024-09-21T14:03:08.617486+0000","last_undegraded":"2024-09-21T14:03:08.617486+0000","last_fullsized":"2024-09-21T14:03:08.617486+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'27","last_scrub_stamp":"2024-09-21T14:02:58.293160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:58.293160+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:01.203771+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":638,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.616008+0000","last_change":"2024-09-21T13:59:32.205723+0000","last_active":"2024-09-21T14:03:08.616008+0000","last_peered":"2024-09-21T14:03:08.616008+0000","last_clean":"2024-09-21T14:03:08.616008+0000","last_became_active":"2024-09-21T13:59:32.205350+0000","last_became_peered":"2024-09-21T13:59:32.205350+0000","last_unstale":"2024-09-21T14:03:08.616008+0000","last_undegraded":"2024-09-21T14:03:08.616008+0000","last_fullsized":"2024-09-21T14:03:08.616008+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:32.668851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:32.668851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:47.252073+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"346'10","reported_seq":896,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T14:03:07.613204+0000","last_change":"2024-09-21T14:02:59.831183+0000","last_active":"2024-09-21T14:03:07.613204+0000","last_peered":"2024-09-21T14:03:07.613204+0000","last_clean":"2024-09-21T14:03:07.613204+0000","last_became_active":"2024-09-21T14:01:41.200218+0000","last_became_peered":"2024-09-21T14:01:41.200218+0000","last_unstale":"2024-09-21T14:03:07.613204+0000","last_undegraded":"2024-09-21T14:03:07.613204+0000","last_fullsized":"2024-09-21T14:03:07.613204+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'10","last_scrub_stamp":"2024-09-21T14:02:59.831128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:02:59.831128+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:08.243919+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":565,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.623457+0000","last_change":"2024-09-21T14:01:41.207310+0000","last_active":"2024-09-21T14:03:08.623457+0000","last_peered":"2024-09-21T14:03:08.623457+0000","last_clean":"2024-09-21T14:03:08.623457+0000","last_became_active":"2024-09-21T14:01:41.206720+0000","last_became_peered":"2024-09-21T14:01:41.206720+0000","last_unstale":"2024-09-21T14:03:08.623457+0000","last_undegraded":"2024-09-21T14:03:08.623457+0000","last_fullsized":"2024-09-21T14:03:08.623457+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:52.244483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:52.244483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"387'530","reported_seq":553,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.594808+0000","last_change":"2024-09-21T14:02:56.562998+0000","last_active":"2024-09-21T14:03:08.594808+0000","last_peered":"2024-09-21T14:03:08.594808+0000","last_clean":"2024-09-21T14:03:08.594808+0000","last_became_active":"2024-09-21T14:02:56.562524+0000","last_became_peered":"2024-09-21T14:02:56.562524+0000","last_unstale":"2024-09-21T14:03:08.594808+0000","last_undegraded":"2024-09-21T14:03:08.594808+0000","last_fullsized":"2024-09-21T14:03:08.594808+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":530,"log_dups_size":0,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:01.881695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34734080,"num_objects":530,"num_object_clones":0,"num_object_copies":1590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":530,"num_write_kb":33920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"75'79","reported_seq":161,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T14:03:07.612706+0000","last_change":"2024-09-21T14:02:06.395083+0000","last_active":"2024-09-21T14:03:07.612706+0000","last_peered":"2024-09-21T14:03:07.612706+0000","last_clean":"2024-09-21T14:03:07.612706+0000","last_became_active":"2024-09-21T14:00:41.137756+0000","last_became_peered":"2024-09-21T14:00:41.137756+0000","last_unstale":"2024-09-21T14:03:07.612706+0000","last_undegraded":"2024-09-21T14:03:07.612706+0000","last_fullsized":"2024-09-21T14:03:07.612706+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:02:06.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:02:06.395011+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:29.136654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"290'30","reported_seq":663,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.616148+0000","last_change":"2024-09-21T13:59:32.246090+0000","last_active":"2024-09-21T14:03:08.616148+0000","last_peered":"2024-09-21T14:03:08.616148+0000","last_clean":"2024-09-21T14:03:08.616148+0000","last_became_active":"2024-09-21T13:59:32.205887+0000","last_became_peered":"2024-09-21T13:59:32.205887+0000","last_unstale":"2024-09-21T14:03:08.616148+0000","last_undegraded":"2024-09-21T14:03:08.616148+0000","last_fullsized":"2024-09-21T14:03:08.616148+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'18","last_scrub_stamp":"2024-09-21T13:54:19.001030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:19.001030+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:43.319659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":590,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.621088+0000","last_change":"2024-09-21T13:59:32.212559+0000","last_active":"2024-09-21T14:03:08.621088+0000","last_peered":"2024-09-21T14:03:08.621088+0000","last_clean":"2024-09-21T14:03:08.621088+0000","last_became_active":"2024-09-21T13:59:32.211004+0000","last_became_peered":"2024-09-21T13:59:32.211004+0000","last_unstale":"2024-09-21T14:03:08.621088+0000","last_undegraded":"2024-09-21T14:03:08.621088+0000","last_fullsized":"2024-09-21T14:03:08.621088+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:25.736461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:25.736461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:48.292957+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":706,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T14:03:07.611129+0000","last_change":"2024-09-21T13:59:34.293330+0000","last_active":"2024-09-21T14:03:07.611129+0000","last_peered":"2024-09-21T14:03:07.611129+0000","last_clean":"2024-09-21T14:03:07.611129+0000","last_became_active":"2024-09-21T13:59:32.220304+0000","last_became_peered":"2024-09-21T13:59:32.220304+0000","last_unstale":"2024-09-21T14:03:07.611129+0000","last_undegraded":"2024-09-21T14:03:07.611129+0000","last_fullsized":"2024-09-21T14:03:07.611129+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:34.292892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:34.292892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:15.468420+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":686,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T14:03:07.611445+0000","last_change":"2024-09-21T13:59:23.334230+0000","last_active":"2024-09-21T14:03:07.611445+0000","last_peered":"2024-09-21T14:03:07.611445+0000","last_clean":"2024-09-21T14:03:07.611445+0000","last_became_active":"2024-09-21T13:57:12.680372+0000","last_became_peered":"2024-09-21T13:57:12.680372+0000","last_unstale":"2024-09-21T14:03:07.611445+0000","last_undegraded":"2024-09-21T14:03:07.611445+0000","last_fullsized":"2024-09-21T14:03:07.611445+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:23.334162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:23.334162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:17.515611+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"387'582","reported_seq":605,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.508998+0000","last_change":"2024-09-21T14:02:56.557348+0000","last_active":"2024-09-21T14:03:08.508998+0000","last_peered":"2024-09-21T14:03:08.508998+0000","last_clean":"2024-09-21T14:03:08.508998+0000","last_became_active":"2024-09-21T14:02:56.557055+0000","last_became_peered":"2024-09-21T14:02:56.557055+0000","last_unstale":"2024-09-21T14:03:08.508998+0000","last_undegraded":"2024-09-21T14:03:08.508998+0000","last_fullsized":"2024-09-21T14:03:08.508998+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":582,"log_dups_size":0,"ondisk_log_size":582,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:20.950710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38141952,"num_objects":582,"num_object_clones":0,"num_object_copies":1746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":582,"num_write_kb":37248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"75'79","reported_seq":166,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.619279+0000","last_change":"2024-09-21T14:01:42.177646+0000","last_active":"2024-09-21T14:03:08.619279+0000","last_peered":"2024-09-21T14:03:08.619279+0000","last_clean":"2024-09-21T14:03:08.619279+0000","last_became_active":"2024-09-21T14:01:42.173042+0000","last_became_peered":"2024-09-21T14:01:42.173042+0000","last_unstale":"2024-09-21T14:03:08.619279+0000","last_undegraded":"2024-09-21T14:03:08.619279+0000","last_fullsized":"2024-09-21T14:03:08.619279+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:03:42.356436+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":709030,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"290'28","reported_seq":795,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T14:03:07.611135+0000","last_change":"2024-09-21T14:02:02.471226+0000","last_active":"2024-09-21T14:03:07.611135+0000","last_peered":"2024-09-21T14:03:07.611135+0000","last_clean":"2024-09-21T14:03:07.611135+0000","last_became_active":"2024-09-21T14:01:41.200859+0000","last_became_peered":"2024-09-21T14:01:41.200859+0000","last_unstale":"2024-09-21T14:03:07.611135+0000","last_undegraded":"2024-09-21T14:03:07.611135+0000","last_fullsized":"2024-09-21T14:03:07.611135+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'28","last_scrub_stamp":"2024-09-21T14:02:02.471133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:02.471133+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:14.278651+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":696,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.627953+0000","last_change":"2024-09-21T13:57:38.046718+0000","last_active":"2024-09-21T14:03:08.627953+0000","last_peered":"2024-09-21T14:03:08.627953+0000","last_clean":"2024-09-21T14:03:08.627953+0000","last_became_active":"2024-09-21T13:48:58.053533+0000","last_became_peered":"2024-09-21T13:48:58.053533+0000","last_unstale":"2024-09-21T14:03:08.627953+0000","last_undegraded":"2024-09-21T14:03:08.627953+0000","last_fullsized":"2024-09-21T14:03:08.627953+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:57:38.046627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:38.046627+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:49.580099+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"241'10","reported_seq":751,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.616330+0000","last_change":"2024-09-21T14:03:03.228551+0000","last_active":"2024-09-21T14:03:08.616330+0000","last_peered":"2024-09-21T14:03:08.616330+0000","last_clean":"2024-09-21T14:03:08.616330+0000","last_became_active":"2024-09-21T13:57:12.686651+0000","last_became_peered":"2024-09-21T13:57:12.686651+0000","last_unstale":"2024-09-21T14:03:08.616330+0000","last_undegraded":"2024-09-21T14:03:08.616330+0000","last_fullsized":"2024-09-21T14:03:08.616330+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'10","last_scrub_stamp":"2024-09-21T14:03:03.228443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:03.228443+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:24.135971+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":654,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.626227+0000","last_change":"2024-09-21T13:59:32.210257+0000","last_active":"2024-09-21T14:03:08.626227+0000","last_peered":"2024-09-21T14:03:08.626227+0000","last_clean":"2024-09-21T14:03:08.626227+0000","last_became_active":"2024-09-21T13:59:32.209615+0000","last_became_peered":"2024-09-21T13:59:32.209615+0000","last_unstale":"2024-09-21T14:03:08.626227+0000","last_undegraded":"2024-09-21T14:03:08.626227+0000","last_fullsized":"2024-09-21T14:03:08.626227+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:37.548064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:37.548064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.c","version":"388'1053","reported_seq":1078,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:11.192683+0000","last_change":"2024-09-21T14:02:56.563139+0000","last_active":"2024-09-21T14:03:11.192683+0000","last_peered":"2024-09-21T14:03:11.192683+0000","last_clean":"2024-09-21T14:03:11.192683+0000","last_became_active":"2024-09-21T14:02:56.562806+0000","last_became_peered":"2024-09-21T14:02:56.562806+0000","last_unstale":"2024-09-21T14:03:11.192683+0000","last_undegraded":"2024-09-21T14:03:11.192683+0000","last_fullsized":"2024-09-21T14:03:11.192683+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":1053,"log_dups_size":0,"ondisk_log_size":1053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:18.206635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69009408,"num_objects":1053,"num_object_clones":0,"num_object_copies":3159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1053,"num_write_kb":67392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"75'79","reported_seq":734,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T14:03:07.608392+0000","last_change":"2024-09-21T14:02:05.693250+0000","last_active":"2024-09-21T14:03:07.608392+0000","last_peered":"2024-09-21T14:03:07.608392+0000","last_clean":"2024-09-21T14:03:07.608392+0000","last_became_active":"2024-09-21T14:00:41.083425+0000","last_became_peered":"2024-09-21T14:00:41.083425+0000","last_unstale":"2024-09-21T14:03:07.608392+0000","last_undegraded":"2024-09-21T14:03:07.608392+0000","last_fullsized":"2024-09-21T14:03:07.608392+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:02:05.693193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:02:05.693193+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:28.620866+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"290'20","reported_seq":732,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.614934+0000","last_change":"2024-09-21T14:01:41.187743+0000","last_active":"2024-09-21T14:03:08.614934+0000","last_peered":"2024-09-21T14:03:08.614934+0000","last_clean":"2024-09-21T14:03:08.614934+0000","last_became_active":"2024-09-21T14:01:41.187060+0000","last_became_peered":"2024-09-21T14:01:41.187060+0000","last_unstale":"2024-09-21T14:03:08.614934+0000","last_undegraded":"2024-09-21T14:03:08.614934+0000","last_fullsized":"2024-09-21T14:03:08.614934+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:54:19.593611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:19.593611+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:47.290628+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"61'1","reported_seq":627,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.623554+0000","last_change":"2024-09-21T14:01:41.195580+0000","last_active":"2024-09-21T14:03:08.623554+0000","last_peered":"2024-09-21T14:03:08.623554+0000","last_clean":"2024-09-21T14:03:08.623554+0000","last_became_active":"2024-09-21T14:01:41.195268+0000","last_became_peered":"2024-09-21T14:01:41.195268+0000","last_unstale":"2024-09-21T14:03:08.623554+0000","last_undegraded":"2024-09-21T14:03:08.623554+0000","last_fullsized":"2024-09-21T14:03:08.623554+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:58:48.407734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:48.407734+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"346'15","reported_seq":325,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T14:03:07.608092+0000","last_change":"2024-09-21T14:01:41.209662+0000","last_active":"2024-09-21T14:03:07.608092+0000","last_peered":"2024-09-21T14:03:07.608092+0000","last_clean":"2024-09-21T14:03:07.608092+0000","last_became_active":"2024-09-21T14:01:41.209296+0000","last_became_peered":"2024-09-21T14:01:41.209296+0000","last_unstale":"2024-09-21T14:03:07.608092+0000","last_undegraded":"2024-09-21T14:03:07.608092+0000","last_fullsized":"2024-09-21T14:03:07.608092+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'12","last_scrub_stamp":"2024-09-21T13:57:30.133392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:30.133392+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:43.651875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":509,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.616246+0000","last_change":"2024-09-21T13:58:58.730439+0000","last_active":"2024-09-21T14:03:08.616246+0000","last_peered":"2024-09-21T14:03:08.616246+0000","last_clean":"2024-09-21T14:03:08.616246+0000","last_became_active":"2024-09-21T13:57:12.704897+0000","last_became_peered":"2024-09-21T13:57:12.704897+0000","last_unstale":"2024-09-21T14:03:08.616246+0000","last_undegraded":"2024-09-21T14:03:08.616246+0000","last_fullsized":"2024-09-21T14:03:08.616246+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:58.730368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:58.730368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.d","version":"387'587","reported_seq":610,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.548616+0000","last_change":"2024-09-21T14:02:56.560958+0000","last_active":"2024-09-21T14:03:08.548616+0000","last_peered":"2024-09-21T14:03:08.548616+0000","last_clean":"2024-09-21T14:03:08.548616+0000","last_became_active":"2024-09-21T14:02:56.560664+0000","last_became_peered":"2024-09-21T14:02:56.560664+0000","last_unstale":"2024-09-21T14:03:08.548616+0000","last_undegraded":"2024-09-21T14:03:08.548616+0000","last_fullsized":"2024-09-21T14:03:08.548616+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":587,"log_dups_size":0,"ondisk_log_size":587,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:06.081463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38469632,"num_objects":587,"num_object_clones":0,"num_object_copies":1761,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":587,"num_write_kb":37568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"75'79","reported_seq":734,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T14:03:07.608292+0000","last_change":"2024-09-21T14:02:03.737005+0000","last_active":"2024-09-21T14:03:07.608292+0000","last_peered":"2024-09-21T14:03:07.608292+0000","last_clean":"2024-09-21T14:03:07.608292+0000","last_became_active":"2024-09-21T14:00:41.073047+0000","last_became_peered":"2024-09-21T14:00:41.073047+0000","last_unstale":"2024-09-21T14:03:07.608292+0000","last_undegraded":"2024-09-21T14:03:07.608292+0000","last_fullsized":"2024-09-21T14:03:07.608292+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:02:03.736928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:02:03.736928+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:21.563427+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"290'29","reported_seq":641,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.623447+0000","last_change":"2024-09-21T14:03:08.399687+0000","last_active":"2024-09-21T14:03:08.623447+0000","last_peered":"2024-09-21T14:03:08.623447+0000","last_clean":"2024-09-21T14:03:08.623447+0000","last_became_active":"2024-09-21T14:01:41.186960+0000","last_became_peered":"2024-09-21T14:01:41.186960+0000","last_unstale":"2024-09-21T14:03:08.623447+0000","last_undegraded":"2024-09-21T14:03:08.623447+0000","last_fullsized":"2024-09-21T14:03:08.623447+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'29","last_scrub_stamp":"2024-09-21T14:03:08.399606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:03:08.399606+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:19.025883+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":709,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.616225+0000","last_change":"2024-09-21T14:01:41.185706+0000","last_active":"2024-09-21T14:03:08.616225+0000","last_peered":"2024-09-21T14:03:08.616225+0000","last_clean":"2024-09-21T14:03:08.616225+0000","last_became_active":"2024-09-21T14:01:41.185400+0000","last_became_peered":"2024-09-21T14:01:41.185400+0000","last_unstale":"2024-09-21T14:03:08.616225+0000","last_undegraded":"2024-09-21T14:03:08.616225+0000","last_fullsized":"2024-09-21T14:03:08.616225+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:16.712990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:16.712990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:38.309423+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":283,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T14:03:07.608145+0000","last_change":"2024-09-21T14:02:59.469906+0000","last_active":"2024-09-21T14:03:07.608145+0000","last_peered":"2024-09-21T14:03:07.608145+0000","last_clean":"2024-09-21T14:03:07.608145+0000","last_became_active":"2024-09-21T14:01:41.193973+0000","last_became_peered":"2024-09-21T14:01:41.193973+0000","last_unstale":"2024-09-21T14:03:07.608145+0000","last_undegraded":"2024-09-21T14:03:07.608145+0000","last_fullsized":"2024-09-21T14:03:07.608145+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:59.469835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:02:59.469835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:00.302431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":700,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.614634+0000","last_change":"2024-09-21T13:58:43.952014+0000","last_active":"2024-09-21T14:03:08.614634+0000","last_peered":"2024-09-21T14:03:08.614634+0000","last_clean":"2024-09-21T14:03:08.614634+0000","last_became_active":"2024-09-21T13:58:17.757628+0000","last_became_peered":"2024-09-21T13:58:17.757628+0000","last_unstale":"2024-09-21T14:03:08.614634+0000","last_undegraded":"2024-09-21T14:03:08.614634+0000","last_fullsized":"2024-09-21T14:03:08.614634+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:43.951941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:43.951941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:52.357919+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"387'585","reported_seq":608,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.551475+0000","last_change":"2024-09-21T14:02:56.559655+0000","last_active":"2024-09-21T14:03:08.551475+0000","last_peered":"2024-09-21T14:03:08.551475+0000","last_clean":"2024-09-21T14:03:08.551475+0000","last_became_active":"2024-09-21T14:02:56.559346+0000","last_became_peered":"2024-09-21T14:02:56.559346+0000","last_unstale":"2024-09-21T14:03:08.551475+0000","last_undegraded":"2024-09-21T14:03:08.551475+0000","last_fullsized":"2024-09-21T14:03:08.551475+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":585,"log_dups_size":0,"ondisk_log_size":585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:10.203798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38338560,"num_objects":585,"num_object_clones":0,"num_object_copies":1755,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":585,"num_write_kb":37440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"75'79","reported_seq":109,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.623015+0000","last_change":"2024-09-21T14:01:41.199377+0000","last_active":"2024-09-21T14:03:08.623015+0000","last_peered":"2024-09-21T14:03:08.623015+0000","last_clean":"2024-09-21T14:03:08.623015+0000","last_became_active":"2024-09-21T14:01:41.198750+0000","last_became_peered":"2024-09-21T14:01:41.198750+0000","last_unstale":"2024-09-21T14:03:08.623015+0000","last_undegraded":"2024-09-21T14:03:08.623015+0000","last_fullsized":"2024-09-21T14:03:08.623015+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:03:47.451019+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"290'44","reported_seq":954,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.616087+0000","last_change":"2024-09-21T14:02:57.316829+0000","last_active":"2024-09-21T14:03:08.616087+0000","last_peered":"2024-09-21T14:03:08.616087+0000","last_clean":"2024-09-21T14:03:08.616087+0000","last_became_active":"2024-09-21T13:57:12.684690+0000","last_became_peered":"2024-09-21T13:57:12.684690+0000","last_unstale":"2024-09-21T14:03:08.616087+0000","last_undegraded":"2024-09-21T14:03:08.616087+0000","last_fullsized":"2024-09-21T14:03:08.616087+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'44","last_scrub_stamp":"2024-09-21T14:02:57.316763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:57.316763+0000","objects_scrubbed":10,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:57.769620+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":714,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T14:03:07.611589+0000","last_change":"2024-09-21T14:01:58.524159+0000","last_active":"2024-09-21T14:03:07.611589+0000","last_peered":"2024-09-21T14:03:07.611589+0000","last_clean":"2024-09-21T14:03:07.611589+0000","last_became_active":"2024-09-21T13:58:17.770937+0000","last_became_peered":"2024-09-21T13:58:17.770937+0000","last_unstale":"2024-09-21T14:03:07.611589+0000","last_undegraded":"2024-09-21T14:03:07.611589+0000","last_fullsized":"2024-09-21T14:03:07.611589+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:58.524114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:01:58.524114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:07.944133+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":646,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.623269+0000","last_change":"2024-09-21T14:01:41.249061+0000","last_active":"2024-09-21T14:03:08.623269+0000","last_peered":"2024-09-21T14:03:08.623269+0000","last_clean":"2024-09-21T14:03:08.623269+0000","last_became_active":"2024-09-21T14:01:41.247642+0000","last_became_peered":"2024-09-21T14:01:41.247642+0000","last_unstale":"2024-09-21T14:03:08.623269+0000","last_undegraded":"2024-09-21T14:03:08.623269+0000","last_fullsized":"2024-09-21T14:03:08.623269+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-21T13:59:22.186393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:59:22.186393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":628,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.616412+0000","last_change":"2024-09-21T13:55:07.021128+0000","last_active":"2024-09-21T14:03:08.616412+0000","last_peered":"2024-09-21T14:03:08.616412+0000","last_clean":"2024-09-21T14:03:08.616412+0000","last_became_active":"2024-09-21T13:50:29.688249+0000","last_became_peered":"2024-09-21T13:50:29.688249+0000","last_unstale":"2024-09-21T14:03:08.616412+0000","last_undegraded":"2024-09-21T14:03:08.616412+0000","last_fullsized":"2024-09-21T14:03:08.616412+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:07.021059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:55:07.021059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:50.302160+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":625,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.619631+0000","last_change":"2024-09-21T14:03:02.238409+0000","last_active":"2024-09-21T14:03:08.619631+0000","last_peered":"2024-09-21T14:03:08.619631+0000","last_clean":"2024-09-21T14:03:08.619631+0000","last_became_active":"2024-09-21T13:59:32.208980+0000","last_became_peered":"2024-09-21T13:59:32.208980+0000","last_unstale":"2024-09-21T14:03:08.619631+0000","last_undegraded":"2024-09-21T14:03:08.619631+0000","last_fullsized":"2024-09-21T14:03:08.619631+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:02.238357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:02.238357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:09.206892+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"388'1000","reported_seq":1025,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:10.875859+0000","last_change":"2024-09-21T14:02:56.559275+0000","last_active":"2024-09-21T14:03:10.875859+0000","last_peered":"2024-09-21T14:03:10.875859+0000","last_clean":"2024-09-21T14:03:10.875859+0000","last_became_active":"2024-09-21T14:02:56.558773+0000","last_became_peered":"2024-09-21T14:02:56.558773+0000","last_unstale":"2024-09-21T14:03:10.875859+0000","last_undegraded":"2024-09-21T14:03:10.875859+0000","last_fullsized":"2024-09-21T14:03:10.875859+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":1000,"log_dups_size":0,"ondisk_log_size":1000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:02.709376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536000,"num_objects":1000,"num_object_clones":0,"num_object_copies":3000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1000,"num_write_kb":64000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"75'79","reported_seq":742,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.625821+0000","last_change":"2024-09-21T14:00:41.077286+0000","last_active":"2024-09-21T14:03:08.625821+0000","last_peered":"2024-09-21T14:03:08.625821+0000","last_clean":"2024-09-21T14:03:08.625821+0000","last_became_active":"2024-09-21T14:00:41.076948+0000","last_became_peered":"2024-09-21T14:00:41.076948+0000","last_unstale":"2024-09-21T14:03:08.625821+0000","last_undegraded":"2024-09-21T14:03:08.625821+0000","last_fullsized":"2024-09-21T14:03:08.625821+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:03:51.730082+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"290'17","reported_seq":735,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.617594+0000","last_change":"2024-09-21T14:01:41.200809+0000","last_active":"2024-09-21T14:03:08.617594+0000","last_peered":"2024-09-21T14:03:08.617594+0000","last_clean":"2024-09-21T14:03:08.617594+0000","last_became_active":"2024-09-21T14:01:41.200365+0000","last_became_peered":"2024-09-21T14:01:41.200365+0000","last_unstale":"2024-09-21T14:03:08.617594+0000","last_undegraded":"2024-09-21T14:03:08.617594+0000","last_fullsized":"2024-09-21T14:03:08.617594+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'13","last_scrub_stamp":"2024-09-21T13:58:11.619848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:11.619848+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:39.361066+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":745,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.615019+0000","last_change":"2024-09-21T14:03:01.310647+0000","last_active":"2024-09-21T14:03:08.615019+0000","last_peered":"2024-09-21T14:03:08.615019+0000","last_clean":"2024-09-21T14:03:08.615019+0000","last_became_active":"2024-09-21T13:58:17.759671+0000","last_became_peered":"2024-09-21T13:58:17.759671+0000","last_unstale":"2024-09-21T14:03:08.615019+0000","last_undegraded":"2024-09-21T14:03:08.615019+0000","last_fullsized":"2024-09-21T14:03:08.615019+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:01.310603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:03:01.310603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:10.066440+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":624,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T14:03:07.608670+0000","last_change":"2024-09-21T13:58:41.644876+0000","last_active":"2024-09-21T14:03:07.608670+0000","last_peered":"2024-09-21T14:03:07.608670+0000","last_clean":"2024-09-21T14:03:07.608670+0000","last_became_active":"2024-09-21T13:57:12.682147+0000","last_became_peered":"2024-09-21T13:57:12.682147+0000","last_unstale":"2024-09-21T14:03:07.608670+0000","last_undegraded":"2024-09-21T14:03:07.608670+0000","last_fullsized":"2024-09-21T14:03:07.608670+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:58:41.644789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:58:41.644789+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:13.796854+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":603,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T14:03:07.609178+0000","last_change":"2024-09-21T14:01:50.880840+0000","last_active":"2024-09-21T14:03:07.609178+0000","last_peered":"2024-09-21T14:03:07.609178+0000","last_clean":"2024-09-21T14:03:07.609178+0000","last_became_active":"2024-09-21T14:01:41.182000+0000","last_became_peered":"2024-09-21T14:01:41.182000+0000","last_unstale":"2024-09-21T14:03:07.609178+0000","last_undegraded":"2024-09-21T14:03:07.609178+0000","last_fullsized":"2024-09-21T14:03:07.609178+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:50.880471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:01:50.880471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:14.118692+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":662,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.626572+0000","last_change":"2024-09-21T13:59:32.222791+0000","last_active":"2024-09-21T14:03:08.626572+0000","last_peered":"2024-09-21T14:03:08.626572+0000","last_clean":"2024-09-21T14:03:08.626572+0000","last_became_active":"2024-09-21T13:59:32.221502+0000","last_became_peered":"2024-09-21T13:59:32.221502+0000","last_unstale":"2024-09-21T14:03:08.626572+0000","last_undegraded":"2024-09-21T14:03:08.626572+0000","last_fullsized":"2024-09-21T14:03:08.626572+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:14.673582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:14.673582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"387'410","reported_seq":433,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T14:03:07.735974+0000","last_change":"2024-09-21T14:02:56.559108+0000","last_active":"2024-09-21T14:03:07.735974+0000","last_peered":"2024-09-21T14:03:07.735974+0000","last_clean":"2024-09-21T14:03:07.735974+0000","last_became_active":"2024-09-21T14:02:56.558826+0000","last_became_peered":"2024-09-21T14:02:56.558826+0000","last_unstale":"2024-09-21T14:03:07.735974+0000","last_undegraded":"2024-09-21T14:03:07.735974+0000","last_fullsized":"2024-09-21T14:03:07.735974+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":410,"log_dups_size":0,"ondisk_log_size":410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:03.816041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26869760,"num_objects":410,"num_object_clones":0,"num_object_copies":1230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":410,"num_write_kb":26240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"75'79","reported_seq":147,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.616090+0000","last_change":"2024-09-21T14:00:41.071631+0000","last_active":"2024-09-21T14:03:08.616090+0000","last_peered":"2024-09-21T14:03:08.616090+0000","last_clean":"2024-09-21T14:03:08.616090+0000","last_became_active":"2024-09-21T14:00:41.071042+0000","last_became_peered":"2024-09-21T14:00:41.071042+0000","last_unstale":"2024-09-21T14:03:08.616090+0000","last_undegraded":"2024-09-21T14:03:08.616090+0000","last_fullsized":"2024-09-21T14:03:08.616090+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:03:42.335979+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"290'24","reported_seq":794,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.627743+0000","last_change":"2024-09-21T14:02:57.560019+0000","last_active":"2024-09-21T14:03:08.627743+0000","last_peered":"2024-09-21T14:03:08.627743+0000","last_clean":"2024-09-21T14:03:08.627743+0000","last_became_active":"2024-09-21T14:01:41.184728+0000","last_became_peered":"2024-09-21T14:01:41.184728+0000","last_unstale":"2024-09-21T14:03:08.627743+0000","last_undegraded":"2024-09-21T14:03:08.627743+0000","last_fullsized":"2024-09-21T14:03:08.627743+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'24","last_scrub_stamp":"2024-09-21T14:02:57.559936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:57.559936+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:26.628231+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":636,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T14:03:07.611481+0000","last_change":"2024-09-21T13:59:59.450613+0000","last_active":"2024-09-21T14:03:07.611481+0000","last_peered":"2024-09-21T14:03:07.611481+0000","last_clean":"2024-09-21T14:03:07.611481+0000","last_became_active":"2024-09-21T13:57:12.681030+0000","last_became_peered":"2024-09-21T13:57:12.681030+0000","last_unstale":"2024-09-21T14:03:07.611481+0000","last_undegraded":"2024-09-21T14:03:07.611481+0000","last_fullsized":"2024-09-21T14:03:07.611481+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:59.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:59:59.450546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:40.683320+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":1415,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:09.546294+0000","last_change":"2024-09-21T13:59:32.218796+0000","last_active":"2024-09-21T14:03:09.546294+0000","last_peered":"2024-09-21T14:03:09.546294+0000","last_clean":"2024-09-21T14:03:09.546294+0000","last_became_active":"2024-09-21T13:59:32.203361+0000","last_became_peered":"2024-09-21T13:59:32.203361+0000","last_unstale":"2024-09-21T14:03:09.546294+0000","last_undegraded":"2024-09-21T14:03:09.546294+0000","last_fullsized":"2024-09-21T14:03:09.546294+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-21T13:56:29.617143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:56:29.617143+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047048000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":225,"num_read_kb":220,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":625,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.624913+0000","last_change":"2024-09-21T14:03:06.470855+0000","last_active":"2024-09-21T14:03:08.624913+0000","last_peered":"2024-09-21T14:03:08.624913+0000","last_clean":"2024-09-21T14:03:08.624913+0000","last_became_active":"2024-09-21T14:01:41.200531+0000","last_became_peered":"2024-09-21T14:01:41.200531+0000","last_unstale":"2024-09-21T14:03:08.624913+0000","last_undegraded":"2024-09-21T14:03:08.624913+0000","last_fullsized":"2024-09-21T14:03:08.624913+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:06.470705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:06.470705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:32.367248+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":729,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.615199+0000","last_change":"2024-09-21T14:03:04.357026+0000","last_active":"2024-09-21T14:03:08.615199+0000","last_peered":"2024-09-21T14:03:08.615199+0000","last_clean":"2024-09-21T14:03:08.615199+0000","last_became_active":"2024-09-21T14:01:41.184664+0000","last_became_peered":"2024-09-21T14:01:41.184664+0000","last_unstale":"2024-09-21T14:03:08.615199+0000","last_undegraded":"2024-09-21T14:03:08.615199+0000","last_fullsized":"2024-09-21T14:03:08.615199+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:04.356979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:04.356979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:19.365633+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"387'560","reported_seq":583,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.545539+0000","last_change":"2024-09-21T14:02:56.557541+0000","last_active":"2024-09-21T14:03:08.545539+0000","last_peered":"2024-09-21T14:03:08.545539+0000","last_clean":"2024-09-21T14:03:08.545539+0000","last_became_active":"2024-09-21T14:02:56.557263+0000","last_became_peered":"2024-09-21T14:02:56.557263+0000","last_unstale":"2024-09-21T14:03:08.545539+0000","last_undegraded":"2024-09-21T14:03:08.545539+0000","last_fullsized":"2024-09-21T14:03:08.545539+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":560,"log_dups_size":0,"ondisk_log_size":560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:05.629424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36700160,"num_objects":560,"num_object_clones":0,"num_object_copies":1680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":560,"num_write_kb":35840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"300'32","reported_seq":378,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T14:03:07.608423+0000","last_change":"2024-09-21T14:00:47.130089+0000","last_active":"2024-09-21T14:03:07.608423+0000","last_peered":"2024-09-21T14:03:07.608423+0000","last_clean":"2024-09-21T14:03:07.608423+0000","last_became_active":"2024-09-21T13:59:32.204624+0000","last_became_peered":"2024-09-21T13:59:32.204624+0000","last_unstale":"2024-09-21T14:03:07.608423+0000","last_undegraded":"2024-09-21T14:03:07.608423+0000","last_fullsized":"2024-09-21T14:03:07.608423+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'32","last_scrub_stamp":"2024-09-21T14:00:47.129789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:00:47.129789+0000","objects_scrubbed":8,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:15.616969+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":675,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.617361+0000","last_change":"2024-09-21T14:01:41.190537+0000","last_active":"2024-09-21T14:03:08.617361+0000","last_peered":"2024-09-21T14:03:08.617361+0000","last_clean":"2024-09-21T14:03:08.617361+0000","last_became_active":"2024-09-21T14:01:41.190210+0000","last_became_peered":"2024-09-21T14:01:41.190210+0000","last_unstale":"2024-09-21T14:03:08.617361+0000","last_undegraded":"2024-09-21T14:03:08.617361+0000","last_fullsized":"2024-09-21T14:03:08.617361+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:57.772975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:57.772975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"75'79","reported_seq":736,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.625937+0000","last_change":"2024-09-21T14:00:39.339360+0000","last_active":"2024-09-21T14:03:08.625937+0000","last_peered":"2024-09-21T14:03:08.625937+0000","last_clean":"2024-09-21T14:03:08.625937+0000","last_became_active":"2024-09-21T14:00:39.339077+0000","last_became_peered":"2024-09-21T14:00:39.339077+0000","last_unstale":"2024-09-21T14:03:08.625937+0000","last_undegraded":"2024-09-21T14:03:08.625937+0000","last_fullsized":"2024-09-21T14:03:08.625937+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:03:49.763645+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":19,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53669,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":650,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.626626+0000","last_change":"2024-09-21T14:01:41.182186+0000","last_active":"2024-09-21T14:03:08.626626+0000","last_peered":"2024-09-21T14:03:08.626626+0000","last_clean":"2024-09-21T14:03:08.626626+0000","last_became_active":"2024-09-21T14:01:41.181899+0000","last_became_peered":"2024-09-21T14:01:41.181899+0000","last_unstale":"2024-09-21T14:03:08.626626+0000","last_undegraded":"2024-09-21T14:03:08.626626+0000","last_fullsized":"2024-09-21T14:03:08.626626+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:15.700409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:15.700409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":575,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.624670+0000","last_change":"2024-09-21T14:01:41.196509+0000","last_active":"2024-09-21T14:03:08.624670+0000","last_peered":"2024-09-21T14:03:08.624670+0000","last_clean":"2024-09-21T14:03:08.624670+0000","last_became_active":"2024-09-21T14:01:41.196243+0000","last_became_peered":"2024-09-21T14:01:41.196243+0000","last_unstale":"2024-09-21T14:03:08.624670+0000","last_undegraded":"2024-09-21T14:03:08.624670+0000","last_fullsized":"2024-09-21T14:03:08.624670+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:21.153018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:21.153018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"387'569","reported_seq":592,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.551639+0000","last_change":"2024-09-21T14:02:56.561047+0000","last_active":"2024-09-21T14:03:08.551639+0000","last_peered":"2024-09-21T14:03:08.551639+0000","last_clean":"2024-09-21T14:03:08.551639+0000","last_became_active":"2024-09-21T14:02:56.558774+0000","last_became_peered":"2024-09-21T14:02:56.558774+0000","last_unstale":"2024-09-21T14:03:08.551639+0000","last_undegraded":"2024-09-21T14:03:08.551639+0000","last_fullsized":"2024-09-21T14:03:08.551639+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":569,"log_dups_size":0,"ondisk_log_size":569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:07.590313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37289984,"num_objects":569,"num_object_clones":0,"num_object_copies":1707,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":569,"num_write_kb":36416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"290'19","reported_seq":760,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T14:03:07.613257+0000","last_change":"2024-09-21T14:02:28.553549+0000","last_active":"2024-09-21T14:03:07.613257+0000","last_peered":"2024-09-21T14:03:07.613257+0000","last_clean":"2024-09-21T14:03:07.613257+0000","last_became_active":"2024-09-21T13:58:17.776310+0000","last_became_peered":"2024-09-21T13:58:17.776310+0000","last_unstale":"2024-09-21T14:03:07.613257+0000","last_undegraded":"2024-09-21T14:03:07.613257+0000","last_fullsized":"2024-09-21T14:03:07.613257+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'19","last_scrub_stamp":"2024-09-21T14:02:28.553065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:28.553065+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:33.636087+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":722,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T14:03:07.611415+0000","last_change":"2024-09-21T14:02:03.437581+0000","last_active":"2024-09-21T14:03:07.611415+0000","last_peered":"2024-09-21T14:03:07.611415+0000","last_clean":"2024-09-21T14:03:07.611415+0000","last_became_active":"2024-09-21T13:59:32.223247+0000","last_became_peered":"2024-09-21T13:59:32.223247+0000","last_unstale":"2024-09-21T14:03:07.611415+0000","last_undegraded":"2024-09-21T14:03:07.611415+0000","last_fullsized":"2024-09-21T14:03:07.611415+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:03.437510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:02:03.437510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:14.620200+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":651,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.624576+0000","last_change":"2024-09-21T14:02:58.461716+0000","last_active":"2024-09-21T14:03:08.624576+0000","last_peered":"2024-09-21T14:03:08.624576+0000","last_clean":"2024-09-21T14:03:08.624576+0000","last_became_active":"2024-09-21T14:01:41.193941+0000","last_became_peered":"2024-09-21T14:01:41.193941+0000","last_unstale":"2024-09-21T14:03:08.624576+0000","last_undegraded":"2024-09-21T14:03:08.624576+0000","last_fullsized":"2024-09-21T14:03:08.624576+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:58.461663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:02:58.461663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:20.802609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":653,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.620392+0000","last_change":"2024-09-21T14:01:41.190501+0000","last_active":"2024-09-21T14:03:08.620392+0000","last_peered":"2024-09-21T14:03:08.620392+0000","last_clean":"2024-09-21T14:03:08.620392+0000","last_became_active":"2024-09-21T14:01:41.190192+0000","last_became_peered":"2024-09-21T14:01:41.190192+0000","last_unstale":"2024-09-21T14:03:08.620392+0000","last_undegraded":"2024-09-21T14:03:08.620392+0000","last_fullsized":"2024-09-21T14:03:08.620392+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:58.952218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:54:58.952218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:18.255572+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"387'417","reported_seq":440,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T14:03:07.742020+0000","last_change":"2024-09-21T14:02:56.560359+0000","last_active":"2024-09-21T14:03:07.742020+0000","last_peered":"2024-09-21T14:03:07.742020+0000","last_clean":"2024-09-21T14:03:07.742020+0000","last_became_active":"2024-09-21T14:02:56.560070+0000","last_became_peered":"2024-09-21T14:02:56.560070+0000","last_unstale":"2024-09-21T14:03:07.742020+0000","last_undegraded":"2024-09-21T14:03:07.742020+0000","last_fullsized":"2024-09-21T14:03:07.742020+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":417,"log_dups_size":0,"ondisk_log_size":417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:16.453643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27328512,"num_objects":417,"num_object_clones":0,"num_object_copies":1251,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":417,"num_write_kb":26688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"75'79","reported_seq":116,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.622783+0000","last_change":"2024-09-21T14:01:41.189146+0000","last_active":"2024-09-21T14:03:08.622783+0000","last_peered":"2024-09-21T14:03:08.622783+0000","last_clean":"2024-09-21T14:03:08.622783+0000","last_became_active":"2024-09-21T14:01:41.188685+0000","last_became_peered":"2024-09-21T14:01:41.188685+0000","last_unstale":"2024-09-21T14:03:08.622783+0000","last_undegraded":"2024-09-21T14:03:08.622783+0000","last_fullsized":"2024-09-21T14:03:08.622783+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:03:46.437011+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"290'31","reported_seq":748,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T14:03:07.612635+0000","last_change":"2024-09-21T14:01:03.541547+0000","last_active":"2024-09-21T14:03:07.612635+0000","last_peered":"2024-09-21T14:03:07.612635+0000","last_clean":"2024-09-21T14:03:07.612635+0000","last_became_active":"2024-09-21T13:52:04.313218+0000","last_became_peered":"2024-09-21T13:52:04.313218+0000","last_unstale":"2024-09-21T14:03:07.612635+0000","last_undegraded":"2024-09-21T14:03:07.612635+0000","last_fullsized":"2024-09-21T14:03:07.612635+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'31","last_scrub_stamp":"2024-09-21T14:01:03.541432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:03.541432+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:50.707905+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":264,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T14:03:07.608021+0000","last_change":"2024-09-21T14:01:29.924408+0000","last_active":"2024-09-21T14:03:07.608021+0000","last_peered":"2024-09-21T14:03:07.608021+0000","last_clean":"2024-09-21T14:03:07.608021+0000","last_became_active":"2024-09-21T13:58:17.762520+0000","last_became_peered":"2024-09-21T13:58:17.762520+0000","last_unstale":"2024-09-21T14:03:07.608021+0000","last_undegraded":"2024-09-21T14:03:07.608021+0000","last_fullsized":"2024-09-21T14:03:07.608021+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:29.924331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:01:29.924331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:47.626829+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":679,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.615303+0000","last_change":"2024-09-21T13:58:52.030800+0000","last_active":"2024-09-21T14:03:08.615303+0000","last_peered":"2024-09-21T14:03:08.615303+0000","last_clean":"2024-09-21T14:03:08.615303+0000","last_became_active":"2024-09-21T13:49:02.067472+0000","last_became_peered":"2024-09-21T13:49:02.067472+0000","last_unstale":"2024-09-21T14:03:08.615303+0000","last_undegraded":"2024-09-21T14:03:08.615303+0000","last_fullsized":"2024-09-21T14:03:08.615303+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:52.030718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:52.030718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:43.346814+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":559,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T14:03:07.613534+0000","last_change":"2024-09-21T13:57:12.683321+0000","last_active":"2024-09-21T14:03:07.613534+0000","last_peered":"2024-09-21T14:03:07.613534+0000","last_clean":"2024-09-21T14:03:07.613534+0000","last_became_active":"2024-09-21T13:57:12.682979+0000","last_became_peered":"2024-09-21T13:57:12.682979+0000","last_unstale":"2024-09-21T14:03:07.613534+0000","last_undegraded":"2024-09-21T14:03:07.613534+0000","last_fullsized":"2024-09-21T14:03:07.613534+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:18.027944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:18.027944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:21.753981+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2","version":"388'758","reported_seq":783,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:09.701980+0000","last_change":"2024-09-21T14:02:56.559657+0000","last_active":"2024-09-21T14:03:09.701980+0000","last_peered":"2024-09-21T14:03:09.701980+0000","last_clean":"2024-09-21T14:03:09.701980+0000","last_became_active":"2024-09-21T14:02:56.559368+0000","last_became_peered":"2024-09-21T14:02:56.559368+0000","last_unstale":"2024-09-21T14:03:09.701980+0000","last_undegraded":"2024-09-21T14:03:09.701980+0000","last_fullsized":"2024-09-21T14:03:09.701980+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":758,"log_dups_size":0,"ondisk_log_size":758,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:23.196521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49676288,"num_objects":758,"num_object_clones":0,"num_object_copies":2274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":758,"num_write_kb":48512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"75'79","reported_seq":750,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.627061+0000","last_change":"2024-09-21T14:02:55.806160+0000","last_active":"2024-09-21T14:03:08.627061+0000","last_peered":"2024-09-21T14:03:08.627061+0000","last_clean":"2024-09-21T14:03:08.627061+0000","last_became_active":"2024-09-21T14:00:41.139315+0000","last_became_peered":"2024-09-21T14:00:41.139315+0000","last_unstale":"2024-09-21T14:03:08.627061+0000","last_undegraded":"2024-09-21T14:03:08.627061+0000","last_fullsized":"2024-09-21T14:03:08.627061+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:02:55.806077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.806077+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:59.364707+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"290'20","reported_seq":767,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T14:03:07.611885+0000","last_change":"2024-09-21T13:58:17.789407+0000","last_active":"2024-09-21T14:03:07.611885+0000","last_peered":"2024-09-21T14:03:07.611885+0000","last_clean":"2024-09-21T14:03:07.611885+0000","last_became_active":"2024-09-21T13:58:17.765427+0000","last_became_peered":"2024-09-21T13:58:17.765427+0000","last_unstale":"2024-09-21T14:03:07.611885+0000","last_undegraded":"2024-09-21T14:03:07.611885+0000","last_fullsized":"2024-09-21T14:03:07.611885+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:58:13.464196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:13.464196+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:16.467667+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":618,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.619046+0000","last_change":"2024-09-21T14:02:58.236695+0000","last_active":"2024-09-21T14:03:08.619046+0000","last_peered":"2024-09-21T14:03:08.619046+0000","last_clean":"2024-09-21T14:03:08.619046+0000","last_became_active":"2024-09-21T13:57:12.704203+0000","last_became_peered":"2024-09-21T13:57:12.704203+0000","last_unstale":"2024-09-21T14:03:08.619046+0000","last_undegraded":"2024-09-21T14:03:08.619046+0000","last_fullsized":"2024-09-21T14:03:08.619046+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:58.236645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:02:58.236645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:21.993648+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"346'14","reported_seq":742,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T14:03:07.610177+0000","last_change":"2024-09-21T14:01:41.186960+0000","last_active":"2024-09-21T14:03:07.610177+0000","last_peered":"2024-09-21T14:03:07.610177+0000","last_clean":"2024-09-21T14:03:07.610177+0000","last_became_active":"2024-09-21T14:01:41.186564+0000","last_became_peered":"2024-09-21T14:01:41.186564+0000","last_unstale":"2024-09-21T14:03:07.610177+0000","last_undegraded":"2024-09-21T14:03:07.610177+0000","last_fullsized":"2024-09-21T14:03:07.610177+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'12","last_scrub_stamp":"2024-09-21T13:59:38.738120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:38.738120+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:47.648985+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"68'1","reported_seq":695,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T14:03:07.612935+0000","last_change":"2024-09-21T13:59:31.392186+0000","last_active":"2024-09-21T14:03:07.612935+0000","last_peered":"2024-09-21T14:03:07.612935+0000","last_clean":"2024-09-21T14:03:07.612935+0000","last_became_active":"2024-09-21T13:52:04.313210+0000","last_became_peered":"2024-09-21T13:52:04.313210+0000","last_unstale":"2024-09-21T14:03:07.612935+0000","last_undegraded":"2024-09-21T14:03:07.612935+0000","last_fullsized":"2024-09-21T14:03:07.612935+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T13:59:31.392105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:31.392105+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:39.716003+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.5","version":"387'425","reported_seq":448,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T14:03:07.734432+0000","last_change":"2024-09-21T14:02:56.557127+0000","last_active":"2024-09-21T14:03:07.734432+0000","last_peered":"2024-09-21T14:03:07.734432+0000","last_clean":"2024-09-21T14:03:07.734432+0000","last_became_active":"2024-09-21T14:02:56.556786+0000","last_became_peered":"2024-09-21T14:02:56.556786+0000","last_unstale":"2024-09-21T14:03:07.734432+0000","last_undegraded":"2024-09-21T14:03:07.734432+0000","last_fullsized":"2024-09-21T14:03:07.734432+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":425,"log_dups_size":0,"ondisk_log_size":425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:16.085110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27852800,"num_objects":425,"num_object_clones":0,"num_object_copies":1275,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":425,"num_write_kb":27200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"290'15","reported_seq":724,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.627722+0000","last_change":"2024-09-21T13:58:50.498597+0000","last_active":"2024-09-21T14:03:08.627722+0000","last_peered":"2024-09-21T14:03:08.627722+0000","last_clean":"2024-09-21T14:03:08.627722+0000","last_became_active":"2024-09-21T13:49:00.053016+0000","last_became_peered":"2024-09-21T13:49:00.053016+0000","last_unstale":"2024-09-21T14:03:08.627722+0000","last_undegraded":"2024-09-21T14:03:08.627722+0000","last_fullsized":"2024-09-21T14:03:08.627722+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:58:50.498369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:50.498369+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:54.564652+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":655,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.623595+0000","last_change":"2024-09-21T14:01:41.192717+0000","last_active":"2024-09-21T14:03:08.623595+0000","last_peered":"2024-09-21T14:03:08.623595+0000","last_clean":"2024-09-21T14:03:08.623595+0000","last_became_active":"2024-09-21T14:01:41.192443+0000","last_became_peered":"2024-09-21T14:01:41.192443+0000","last_unstale":"2024-09-21T14:03:08.623595+0000","last_undegraded":"2024-09-21T14:03:08.623595+0000","last_fullsized":"2024-09-21T14:03:08.623595+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:53.894506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:53.894506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:53.450976+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":641,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T14:03:07.611358+0000","last_change":"2024-09-21T14:02:56.796917+0000","last_active":"2024-09-21T14:03:07.611358+0000","last_peered":"2024-09-21T14:03:07.611358+0000","last_clean":"2024-09-21T14:03:07.611358+0000","last_became_active":"2024-09-21T13:57:12.680978+0000","last_became_peered":"2024-09-21T13:57:12.680978+0000","last_unstale":"2024-09-21T14:03:07.611358+0000","last_undegraded":"2024-09-21T14:03:07.611358+0000","last_fullsized":"2024-09-21T14:03:07.611358+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:56.796861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:02:56.796861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:10.252837+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":585,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.626678+0000","last_change":"2024-09-21T13:59:32.224001+0000","last_active":"2024-09-21T14:03:08.626678+0000","last_peered":"2024-09-21T14:03:08.626678+0000","last_clean":"2024-09-21T14:03:08.626678+0000","last_became_active":"2024-09-21T13:59:32.221628+0000","last_became_peered":"2024-09-21T13:59:32.221628+0000","last_unstale":"2024-09-21T14:03:08.626678+0000","last_undegraded":"2024-09-21T14:03:08.626678+0000","last_fullsized":"2024-09-21T14:03:08.626678+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:13.698091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:13.698091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"387'570","reported_seq":593,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.591762+0000","last_change":"2024-09-21T14:02:56.562973+0000","last_active":"2024-09-21T14:03:08.591762+0000","last_peered":"2024-09-21T14:03:08.591762+0000","last_clean":"2024-09-21T14:03:08.591762+0000","last_became_active":"2024-09-21T14:02:56.562256+0000","last_became_peered":"2024-09-21T14:02:56.562256+0000","last_unstale":"2024-09-21T14:03:08.591762+0000","last_undegraded":"2024-09-21T14:03:08.591762+0000","last_fullsized":"2024-09-21T14:03:08.591762+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":570,"log_dups_size":0,"ondisk_log_size":570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:09.952633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37355520,"num_objects":570,"num_object_clones":0,"num_object_copies":1710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":570,"num_write_kb":36480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"290'35","reported_seq":330,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T14:03:07.608079+0000","last_change":"2024-09-21T14:01:02.006619+0000","last_active":"2024-09-21T14:03:07.608079+0000","last_peered":"2024-09-21T14:03:07.608079+0000","last_clean":"2024-09-21T14:03:07.608079+0000","last_became_active":"2024-09-21T13:59:32.204184+0000","last_became_peered":"2024-09-21T13:59:32.204184+0000","last_unstale":"2024-09-21T14:03:07.608079+0000","last_undegraded":"2024-09-21T14:03:07.608079+0000","last_fullsized":"2024-09-21T14:03:07.608079+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'35","last_scrub_stamp":"2024-09-21T14:01:02.006504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:02.006504+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:22.557193+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":677,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.626959+0000","last_change":"2024-09-21T14:01:41.184937+0000","last_active":"2024-09-21T14:03:08.626959+0000","last_peered":"2024-09-21T14:03:08.626959+0000","last_clean":"2024-09-21T14:03:08.626959+0000","last_became_active":"2024-09-21T14:01:41.184567+0000","last_became_peered":"2024-09-21T14:01:41.184567+0000","last_unstale":"2024-09-21T14:03:08.626959+0000","last_undegraded":"2024-09-21T14:03:08.626959+0000","last_fullsized":"2024-09-21T14:03:08.626959+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:21.587493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:21.587493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":653,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T14:03:07.610856+0000","last_change":"2024-09-21T14:01:41.182250+0000","last_active":"2024-09-21T14:03:07.610856+0000","last_peered":"2024-09-21T14:03:07.610856+0000","last_clean":"2024-09-21T14:03:07.610856+0000","last_became_active":"2024-09-21T14:01:41.181911+0000","last_became_peered":"2024-09-21T14:01:41.181911+0000","last_unstale":"2024-09-21T14:03:07.610856+0000","last_undegraded":"2024-09-21T14:03:07.610856+0000","last_fullsized":"2024-09-21T14:03:07.610856+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:55.588554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:55.588554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:45.648976+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":604,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.625801+0000","last_change":"2024-09-21T13:59:32.223123+0000","last_active":"2024-09-21T14:03:08.625801+0000","last_peered":"2024-09-21T14:03:08.625801+0000","last_clean":"2024-09-21T14:03:08.625801+0000","last_became_active":"2024-09-21T13:59:32.219332+0000","last_became_peered":"2024-09-21T13:59:32.219332+0000","last_unstale":"2024-09-21T14:03:08.625801+0000","last_undegraded":"2024-09-21T14:03:08.625801+0000","last_fullsized":"2024-09-21T14:03:08.625801+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:19.537177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:19.537177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"387'529","reported_seq":552,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.512690+0000","last_change":"2024-09-21T14:02:56.559626+0000","last_active":"2024-09-21T14:03:08.512690+0000","last_peered":"2024-09-21T14:03:08.512690+0000","last_clean":"2024-09-21T14:03:08.512690+0000","last_became_active":"2024-09-21T14:02:56.559331+0000","last_became_peered":"2024-09-21T14:02:56.559331+0000","last_unstale":"2024-09-21T14:03:08.512690+0000","last_undegraded":"2024-09-21T14:03:08.512690+0000","last_fullsized":"2024-09-21T14:03:08.512690+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":529,"log_dups_size":0,"ondisk_log_size":529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:20.185982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34668544,"num_objects":529,"num_object_clones":0,"num_object_copies":1587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":529,"num_write_kb":33856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"290'18","reported_seq":739,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T14:03:07.611276+0000","last_change":"2024-09-21T13:59:28.340494+0000","last_active":"2024-09-21T14:03:07.611276+0000","last_peered":"2024-09-21T14:03:07.611276+0000","last_clean":"2024-09-21T14:03:07.611276+0000","last_became_active":"2024-09-21T13:57:12.680627+0000","last_became_peered":"2024-09-21T13:57:12.680627+0000","last_unstale":"2024-09-21T14:03:07.611276+0000","last_undegraded":"2024-09-21T14:03:07.611276+0000","last_fullsized":"2024-09-21T14:03:07.611276+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'18","last_scrub_stamp":"2024-09-21T13:59:28.340415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:28.340415+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:11.383693+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":632,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.627063+0000","last_change":"2024-09-21T13:59:32.201126+0000","last_active":"2024-09-21T14:03:08.627063+0000","last_peered":"2024-09-21T14:03:08.627063+0000","last_clean":"2024-09-21T14:03:08.627063+0000","last_became_active":"2024-09-21T13:59:32.200674+0000","last_became_peered":"2024-09-21T13:59:32.200674+0000","last_unstale":"2024-09-21T14:03:08.627063+0000","last_undegraded":"2024-09-21T14:03:08.627063+0000","last_fullsized":"2024-09-21T14:03:08.627063+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:21.582325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:21.582325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":616,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T14:03:07.608987+0000","last_change":"2024-09-21T14:02:27.791041+0000","last_active":"2024-09-21T14:03:07.608987+0000","last_peered":"2024-09-21T14:03:07.608987+0000","last_clean":"2024-09-21T14:03:07.608987+0000","last_became_active":"2024-09-21T13:52:04.322147+0000","last_became_peered":"2024-09-21T13:52:04.322147+0000","last_unstale":"2024-09-21T14:03:07.608987+0000","last_undegraded":"2024-09-21T14:03:07.608987+0000","last_fullsized":"2024-09-21T14:03:07.608987+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:27.790994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:02:27.790994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:51.879201+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":626,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.619410+0000","last_change":"2024-09-21T14:02:36.371955+0000","last_active":"2024-09-21T14:03:08.619410+0000","last_peered":"2024-09-21T14:03:08.619410+0000","last_clean":"2024-09-21T14:03:08.619410+0000","last_became_active":"2024-09-21T13:59:32.203767+0000","last_became_peered":"2024-09-21T13:59:32.203767+0000","last_unstale":"2024-09-21T14:03:08.619410+0000","last_undegraded":"2024-09-21T14:03:08.619410+0000","last_fullsized":"2024-09-21T14:03:08.619410+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:36.371747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:02:36.371747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:54.168810+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"388'936","reported_seq":961,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:10.879776+0000","last_change":"2024-09-21T14:02:56.564166+0000","last_active":"2024-09-21T14:03:10.879776+0000","last_peered":"2024-09-21T14:03:10.879776+0000","last_clean":"2024-09-21T14:03:10.879776+0000","last_became_active":"2024-09-21T14:02:56.563788+0000","last_became_peered":"2024-09-21T14:02:56.563788+0000","last_unstale":"2024-09-21T14:03:10.879776+0000","last_undegraded":"2024-09-21T14:03:10.879776+0000","last_fullsized":"2024-09-21T14:03:10.879776+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":936,"log_dups_size":0,"ondisk_log_size":936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:20.466245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61341696,"num_objects":936,"num_object_clones":0,"num_object_copies":2808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":936,"num_write_kb":59904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"290'27","reported_seq":731,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.622979+0000","last_change":"2024-09-21T14:01:41.193143+0000","last_active":"2024-09-21T14:03:08.622979+0000","last_peered":"2024-09-21T14:03:08.622979+0000","last_clean":"2024-09-21T14:03:08.622979+0000","last_became_active":"2024-09-21T14:01:41.192481+0000","last_became_peered":"2024-09-21T14:01:41.192481+0000","last_unstale":"2024-09-21T14:03:08.622979+0000","last_undegraded":"2024-09-21T14:03:08.622979+0000","last_fullsized":"2024-09-21T14:03:08.622979+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'27","last_scrub_stamp":"2024-09-21T13:59:33.058859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:33.058859+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":612,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.627330+0000","last_change":"2024-09-21T14:03:04.953724+0000","last_active":"2024-09-21T14:03:08.627330+0000","last_peered":"2024-09-21T14:03:08.627330+0000","last_clean":"2024-09-21T14:03:08.627330+0000","last_became_active":"2024-09-21T13:59:32.203172+0000","last_became_peered":"2024-09-21T13:59:32.203172+0000","last_unstale":"2024-09-21T14:03:08.627330+0000","last_undegraded":"2024-09-21T14:03:08.627330+0000","last_fullsized":"2024-09-21T14:03:08.627330+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T14:03:04.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:03:04.953658+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:14.479415+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"300'14","reported_seq":822,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.626642+0000","last_change":"2024-09-21T13:59:32.220898+0000","last_active":"2024-09-21T14:03:08.626642+0000","last_peered":"2024-09-21T14:03:08.626642+0000","last_clean":"2024-09-21T14:03:08.626642+0000","last_became_active":"2024-09-21T13:59:32.206956+0000","last_became_peered":"2024-09-21T13:59:32.206956+0000","last_unstale":"2024-09-21T14:03:08.626642+0000","last_undegraded":"2024-09-21T14:03:08.626642+0000","last_fullsized":"2024-09-21T14:03:08.626642+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'11","last_scrub_stamp":"2024-09-21T13:56:31.616003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:56:31.616003+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":679,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.623167+0000","last_change":"2024-09-21T14:01:41.248408+0000","last_active":"2024-09-21T14:03:08.623167+0000","last_peered":"2024-09-21T14:03:08.623167+0000","last_clean":"2024-09-21T14:03:08.623167+0000","last_became_active":"2024-09-21T14:01:41.246106+0000","last_became_peered":"2024-09-21T14:01:41.246106+0000","last_unstale":"2024-09-21T14:03:08.623167+0000","last_undegraded":"2024-09-21T14:03:08.623167+0000","last_fullsized":"2024-09-21T14:03:08.623167+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:39.258659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:39.258659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":267,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T14:03:07.607834+0000","last_change":"2024-09-21T14:01:50.733055+0000","last_active":"2024-09-21T14:03:07.607834+0000","last_peered":"2024-09-21T14:03:07.607834+0000","last_clean":"2024-09-21T14:03:07.607834+0000","last_became_active":"2024-09-21T14:01:41.197501+0000","last_became_peered":"2024-09-21T14:01:41.197501+0000","last_unstale":"2024-09-21T14:03:07.607834+0000","last_undegraded":"2024-09-21T14:03:07.607834+0000","last_fullsized":"2024-09-21T14:03:07.607834+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:50.733007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:01:50.733007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:08.367766+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"290'12","reported_seq":653,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.621449+0000","last_change":"2024-09-21T13:59:32.247422+0000","last_active":"2024-09-21T14:03:08.621449+0000","last_peered":"2024-09-21T14:03:08.621449+0000","last_clean":"2024-09-21T14:03:08.621449+0000","last_became_active":"2024-09-21T13:59:32.211306+0000","last_became_peered":"2024-09-21T13:59:32.211306+0000","last_unstale":"2024-09-21T14:03:08.621449+0000","last_undegraded":"2024-09-21T14:03:08.621449+0000","last_fullsized":"2024-09-21T14:03:08.621449+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'6","last_scrub_stamp":"2024-09-21T13:55:12.066873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:55:12.066873+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:38.360712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":611,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.624026+0000","last_change":"2024-09-21T14:01:41.199742+0000","last_active":"2024-09-21T14:03:08.624026+0000","last_peered":"2024-09-21T14:03:08.624026+0000","last_clean":"2024-09-21T14:03:08.624026+0000","last_became_active":"2024-09-21T14:01:41.199321+0000","last_became_peered":"2024-09-21T14:01:41.199321+0000","last_unstale":"2024-09-21T14:03:08.624026+0000","last_undegraded":"2024-09-21T14:03:08.624026+0000","last_fullsized":"2024-09-21T14:03:08.624026+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:37.106238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:37.106238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":658,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.626721+0000","last_change":"2024-09-21T14:03:01.872986+0000","last_active":"2024-09-21T14:03:08.626721+0000","last_peered":"2024-09-21T14:03:08.626721+0000","last_clean":"2024-09-21T14:03:08.626721+0000","last_became_active":"2024-09-21T13:59:32.219525+0000","last_became_peered":"2024-09-21T13:59:32.219525+0000","last_unstale":"2024-09-21T14:03:08.626721+0000","last_undegraded":"2024-09-21T14:03:08.626721+0000","last_fullsized":"2024-09-21T14:03:08.626721+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:01.872928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:01.872928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:09.296117+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"290'15","reported_seq":616,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.619420+0000","last_change":"2024-09-21T13:59:32.238689+0000","last_active":"2024-09-21T14:03:08.619420+0000","last_peered":"2024-09-21T14:03:08.619420+0000","last_clean":"2024-09-21T14:03:08.619420+0000","last_became_active":"2024-09-21T13:59:32.209166+0000","last_became_peered":"2024-09-21T13:59:32.209166+0000","last_unstale":"2024-09-21T14:03:08.619420+0000","last_undegraded":"2024-09-21T14:03:08.619420+0000","last_fullsized":"2024-09-21T14:03:08.619420+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:55:04.990842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:55:04.990842+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:21.278052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"61'1","reported_seq":668,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.626957+0000","last_change":"2024-09-21T13:59:32.222826+0000","last_active":"2024-09-21T14:03:08.626957+0000","last_peered":"2024-09-21T14:03:08.626957+0000","last_clean":"2024-09-21T14:03:08.626957+0000","last_became_active":"2024-09-21T13:59:32.219515+0000","last_became_peered":"2024-09-21T13:59:32.219515+0000","last_unstale":"2024-09-21T14:03:08.626957+0000","last_undegraded":"2024-09-21T14:03:08.626957+0000","last_fullsized":"2024-09-21T14:03:08.626957+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:58:07.711028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:07.711028+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":601,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.621811+0000","last_change":"2024-09-21T13:59:32.211754+0000","last_active":"2024-09-21T14:03:08.621811+0000","last_peered":"2024-09-21T14:03:08.621811+0000","last_clean":"2024-09-21T14:03:08.621811+0000","last_became_active":"2024-09-21T13:59:32.210586+0000","last_became_peered":"2024-09-21T13:59:32.210586+0000","last_unstale":"2024-09-21T14:03:08.621811+0000","last_undegraded":"2024-09-21T14:03:08.621811+0000","last_fullsized":"2024-09-21T14:03:08.621811+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:28.014760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:28.014760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:17.280712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":657,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T14:03:07.609019+0000","last_change":"2024-09-21T14:02:29.731635+0000","last_active":"2024-09-21T14:03:07.609019+0000","last_peered":"2024-09-21T14:03:07.609019+0000","last_clean":"2024-09-21T14:03:07.609019+0000","last_became_active":"2024-09-21T13:59:32.205803+0000","last_became_peered":"2024-09-21T13:59:32.205803+0000","last_unstale":"2024-09-21T14:03:07.609019+0000","last_undegraded":"2024-09-21T14:03:07.609019+0000","last_fullsized":"2024-09-21T14:03:07.609019+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:29.731565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:02:29.731565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:53.473477+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"290'15","reported_seq":656,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.624302+0000","last_change":"2024-09-21T14:01:41.189676+0000","last_active":"2024-09-21T14:03:08.624302+0000","last_peered":"2024-09-21T14:03:08.624302+0000","last_clean":"2024-09-21T14:03:08.624302+0000","last_became_active":"2024-09-21T14:01:41.187904+0000","last_became_peered":"2024-09-21T14:01:41.187904+0000","last_unstale":"2024-09-21T14:03:08.624302+0000","last_undegraded":"2024-09-21T14:03:08.624302+0000","last_fullsized":"2024-09-21T14:03:08.624302+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:57:24.124604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:57:24.124604+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":537,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.626124+0000","last_change":"2024-09-21T13:59:32.212325+0000","last_active":"2024-09-21T14:03:08.626124+0000","last_peered":"2024-09-21T14:03:08.626124+0000","last_clean":"2024-09-21T14:03:08.626124+0000","last_became_active":"2024-09-21T13:59:32.208749+0000","last_became_peered":"2024-09-21T13:59:32.208749+0000","last_unstale":"2024-09-21T14:03:08.626124+0000","last_undegraded":"2024-09-21T14:03:08.626124+0000","last_fullsized":"2024-09-21T14:03:08.626124+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:20.574745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:20.574745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"300'14","reported_seq":755,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.620484+0000","last_change":"2024-09-21T13:59:32.208253+0000","last_active":"2024-09-21T14:03:08.620484+0000","last_peered":"2024-09-21T14:03:08.620484+0000","last_clean":"2024-09-21T14:03:08.620484+0000","last_became_active":"2024-09-21T13:59:32.206926+0000","last_became_peered":"2024-09-21T13:59:32.206926+0000","last_unstale":"2024-09-21T14:03:08.620484+0000","last_undegraded":"2024-09-21T14:03:08.620484+0000","last_fullsized":"2024-09-21T14:03:08.620484+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'10","last_scrub_stamp":"2024-09-21T13:54:20.946782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:20.946782+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:10.314285+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":701,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T14:03:07.613030+0000","last_change":"2024-09-21T14:01:41.200564+0000","last_active":"2024-09-21T14:03:07.613030+0000","last_peered":"2024-09-21T14:03:07.613030+0000","last_clean":"2024-09-21T14:03:07.613030+0000","last_became_active":"2024-09-21T14:01:41.200113+0000","last_became_peered":"2024-09-21T14:01:41.200113+0000","last_unstale":"2024-09-21T14:03:07.613030+0000","last_undegraded":"2024-09-21T14:03:07.613030+0000","last_fullsized":"2024-09-21T14:03:07.613030+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:54.805619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:54.805619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:48.669884+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"290'20","reported_seq":652,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.620281+0000","last_change":"2024-09-21T13:59:32.236530+0000","last_active":"2024-09-21T14:03:08.620281+0000","last_peered":"2024-09-21T14:03:08.620281+0000","last_clean":"2024-09-21T14:03:08.620281+0000","last_became_active":"2024-09-21T13:59:32.208332+0000","last_became_peered":"2024-09-21T13:59:32.208332+0000","last_unstale":"2024-09-21T14:03:08.620281+0000","last_undegraded":"2024-09-21T14:03:08.620281+0000","last_fullsized":"2024-09-21T14:03:08.620281+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:57:37.671391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:57:37.671391+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:55.297241+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":644,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.626093+0000","last_change":"2024-09-21T13:59:32.212737+0000","last_active":"2024-09-21T14:03:08.626093+0000","last_peered":"2024-09-21T14:03:08.626093+0000","last_clean":"2024-09-21T14:03:08.626093+0000","last_became_active":"2024-09-21T13:59:32.209804+0000","last_became_peered":"2024-09-21T13:59:32.209804+0000","last_unstale":"2024-09-21T14:03:08.626093+0000","last_undegraded":"2024-09-21T14:03:08.626093+0000","last_fullsized":"2024-09-21T14:03:08.626093+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:25.571256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:25.571256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"346'10","reported_seq":755,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.624027+0000","last_change":"2024-09-21T14:01:41.207030+0000","last_active":"2024-09-21T14:03:08.624027+0000","last_peered":"2024-09-21T14:03:08.624027+0000","last_clean":"2024-09-21T14:03:08.624027+0000","last_became_active":"2024-09-21T14:01:41.206303+0000","last_became_peered":"2024-09-21T14:01:41.206303+0000","last_unstale":"2024-09-21T14:03:08.624027+0000","last_undegraded":"2024-09-21T14:03:08.624027+0000","last_fullsized":"2024-09-21T14:03:08.624027+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:59:32.104224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:32.104224+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":541,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.626876+0000","last_change":"2024-09-21T13:59:32.213015+0000","last_active":"2024-09-21T14:03:08.626876+0000","last_peered":"2024-09-21T14:03:08.626876+0000","last_clean":"2024-09-21T14:03:08.626876+0000","last_became_active":"2024-09-21T13:59:32.211472+0000","last_became_peered":"2024-09-21T13:59:32.211472+0000","last_unstale":"2024-09-21T14:03:08.626876+0000","last_undegraded":"2024-09-21T14:03:08.626876+0000","last_fullsized":"2024-09-21T14:03:08.626876+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:22.647191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:22.647191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"290'19","reported_seq":765,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T14:03:07.611385+0000","last_change":"2024-09-21T14:02:34.670058+0000","last_active":"2024-09-21T14:03:07.611385+0000","last_peered":"2024-09-21T14:03:07.611385+0000","last_clean":"2024-09-21T14:03:07.611385+0000","last_became_active":"2024-09-21T13:58:17.771497+0000","last_became_peered":"2024-09-21T13:58:17.771497+0000","last_unstale":"2024-09-21T14:03:07.611385+0000","last_undegraded":"2024-09-21T14:03:07.611385+0000","last_fullsized":"2024-09-21T14:03:07.611385+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'19","last_scrub_stamp":"2024-09-21T14:02:34.669962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:34.669962+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:44.716931+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":286,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.626708+0000","last_change":"2024-09-21T14:02:57.807136+0000","last_active":"2024-09-21T14:03:08.626708+0000","last_peered":"2024-09-21T14:03:08.626708+0000","last_clean":"2024-09-21T14:03:08.626708+0000","last_became_active":"2024-09-21T13:59:32.201568+0000","last_became_peered":"2024-09-21T13:59:32.201568+0000","last_unstale":"2024-09-21T14:03:08.626708+0000","last_undegraded":"2024-09-21T14:03:08.626708+0000","last_fullsized":"2024-09-21T14:03:08.626708+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:57.807059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:02:57.807059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:09.290442+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":717,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.614771+0000","last_change":"2024-09-21T14:01:41.185588+0000","last_active":"2024-09-21T14:03:08.614771+0000","last_peered":"2024-09-21T14:03:08.614771+0000","last_clean":"2024-09-21T14:03:08.614771+0000","last_became_active":"2024-09-21T14:01:41.185293+0000","last_became_peered":"2024-09-21T14:01:41.185293+0000","last_unstale":"2024-09-21T14:03:08.614771+0000","last_undegraded":"2024-09-21T14:03:08.614771+0000","last_fullsized":"2024-09-21T14:03:08.614771+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:35.409237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:35.409237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:10.948556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":617,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.616660+0000","last_change":"2024-09-21T14:02:57.225650+0000","last_active":"2024-09-21T14:03:08.616660+0000","last_peered":"2024-09-21T14:03:08.616660+0000","last_clean":"2024-09-21T14:03:08.616660+0000","last_became_active":"2024-09-21T14:01:41.199671+0000","last_became_peered":"2024-09-21T14:01:41.199671+0000","last_unstale":"2024-09-21T14:03:08.616660+0000","last_undegraded":"2024-09-21T14:03:08.616660+0000","last_fullsized":"2024-09-21T14:03:08.616660+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:57.225602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:02:57.225602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:26.897407+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"290'15","reported_seq":769,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.615040+0000","last_change":"2024-09-21T14:02:56.315766+0000","last_active":"2024-09-21T14:03:08.615040+0000","last_peered":"2024-09-21T14:03:08.615040+0000","last_clean":"2024-09-21T14:03:08.615040+0000","last_became_active":"2024-09-21T13:58:17.764120+0000","last_became_peered":"2024-09-21T13:58:17.764120+0000","last_unstale":"2024-09-21T14:03:08.615040+0000","last_undegraded":"2024-09-21T14:03:08.615040+0000","last_fullsized":"2024-09-21T14:03:08.615040+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'15","last_scrub_stamp":"2024-09-21T14:02:56.315702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:56.315702+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:11.328147+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":648,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.626633+0000","last_change":"2024-09-21T13:59:32.223082+0000","last_active":"2024-09-21T14:03:08.626633+0000","last_peered":"2024-09-21T14:03:08.626633+0000","last_clean":"2024-09-21T14:03:08.626633+0000","last_became_active":"2024-09-21T13:59:32.221854+0000","last_became_peered":"2024-09-21T13:59:32.221854+0000","last_unstale":"2024-09-21T14:03:08.626633+0000","last_undegraded":"2024-09-21T14:03:08.626633+0000","last_fullsized":"2024-09-21T14:03:08.626633+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:17.691193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:17.691193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":584,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.616004+0000","last_change":"2024-09-21T13:57:12.703746+0000","last_active":"2024-09-21T14:03:08.616004+0000","last_peered":"2024-09-21T14:03:08.616004+0000","last_clean":"2024-09-21T14:03:08.616004+0000","last_became_active":"2024-09-21T13:57:12.703441+0000","last_became_peered":"2024-09-21T13:57:12.703441+0000","last_unstale":"2024-09-21T14:03:08.616004+0000","last_undegraded":"2024-09-21T14:03:08.616004+0000","last_fullsized":"2024-09-21T14:03:08.616004+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:19.999125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:19.999125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:13.302246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":506,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.622649+0000","last_change":"2024-09-21T14:01:41.188075+0000","last_active":"2024-09-21T14:03:08.622649+0000","last_peered":"2024-09-21T14:03:08.622649+0000","last_clean":"2024-09-21T14:03:08.622649+0000","last_became_active":"2024-09-21T14:01:41.186934+0000","last_became_peered":"2024-09-21T14:01:41.186934+0000","last_unstale":"2024-09-21T14:03:08.622649+0000","last_undegraded":"2024-09-21T14:03:08.622649+0000","last_fullsized":"2024-09-21T14:03:08.622649+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:47.334504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:47.334504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"290'19","reported_seq":728,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.616835+0000","last_change":"2024-09-21T13:59:32.277742+0000","last_active":"2024-09-21T14:03:08.616835+0000","last_peered":"2024-09-21T14:03:08.616835+0000","last_clean":"2024-09-21T14:03:08.616835+0000","last_became_active":"2024-09-21T13:59:32.231567+0000","last_became_peered":"2024-09-21T13:59:32.231567+0000","last_unstale":"2024-09-21T14:03:08.616835+0000","last_undegraded":"2024-09-21T14:03:08.616835+0000","last_fullsized":"2024-09-21T14:03:08.616835+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:54:22.978267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:22.978267+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:12.294653+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":616,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.624069+0000","last_change":"2024-09-21T14:03:01.448170+0000","last_active":"2024-09-21T14:03:08.624069+0000","last_peered":"2024-09-21T14:03:08.624069+0000","last_clean":"2024-09-21T14:03:08.624069+0000","last_became_active":"2024-09-21T14:01:41.190760+0000","last_became_peered":"2024-09-21T14:01:41.190760+0000","last_unstale":"2024-09-21T14:03:08.624069+0000","last_undegraded":"2024-09-21T14:03:08.624069+0000","last_fullsized":"2024-09-21T14:03:08.624069+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:01.448109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:03:01.448109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:28.748585+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":654,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.625942+0000","last_change":"2024-09-21T13:59:32.219508+0000","last_active":"2024-09-21T14:03:08.625942+0000","last_peered":"2024-09-21T14:03:08.625942+0000","last_clean":"2024-09-21T14:03:08.625942+0000","last_became_active":"2024-09-21T13:59:32.214004+0000","last_became_peered":"2024-09-21T13:59:32.214004+0000","last_unstale":"2024-09-21T14:03:08.625942+0000","last_undegraded":"2024-09-21T14:03:08.625942+0000","last_fullsized":"2024-09-21T14:03:08.625942+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:36.525417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:36.525417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":650,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.619150+0000","last_change":"2024-09-21T13:59:28.124373+0000","last_active":"2024-09-21T14:03:08.619150+0000","last_peered":"2024-09-21T14:03:08.619150+0000","last_clean":"2024-09-21T14:03:08.619150+0000","last_became_active":"2024-09-21T13:58:17.778115+0000","last_became_peered":"2024-09-21T13:58:17.778115+0000","last_unstale":"2024-09-21T14:03:08.619150+0000","last_undegraded":"2024-09-21T14:03:08.619150+0000","last_fullsized":"2024-09-21T14:03:08.619150+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:28.124296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:28.124296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"290'8","reported_seq":674,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.616378+0000","last_change":"2024-09-21T13:59:26.034457+0000","last_active":"2024-09-21T14:03:08.616378+0000","last_peered":"2024-09-21T14:03:08.616378+0000","last_clean":"2024-09-21T14:03:08.616378+0000","last_became_active":"2024-09-21T13:58:17.775079+0000","last_became_peered":"2024-09-21T13:58:17.775079+0000","last_unstale":"2024-09-21T14:03:08.616378+0000","last_undegraded":"2024-09-21T14:03:08.616378+0000","last_fullsized":"2024-09-21T14:03:08.616378+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'8","last_scrub_stamp":"2024-09-21T13:59:26.034372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:26.034372+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":576,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.623876+0000","last_change":"2024-09-21T14:01:41.196335+0000","last_active":"2024-09-21T14:03:08.623876+0000","last_peered":"2024-09-21T14:03:08.623876+0000","last_clean":"2024-09-21T14:03:08.623876+0000","last_became_active":"2024-09-21T14:01:41.195989+0000","last_became_peered":"2024-09-21T14:01:41.195989+0000","last_unstale":"2024-09-21T14:03:08.623876+0000","last_undegraded":"2024-09-21T14:03:08.623876+0000","last_fullsized":"2024-09-21T14:03:08.623876+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:16.457445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:16.457445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":267,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T14:03:07.607793+0000","last_change":"2024-09-21T14:00:57.128764+0000","last_active":"2024-09-21T14:03:07.607793+0000","last_peered":"2024-09-21T14:03:07.607793+0000","last_clean":"2024-09-21T14:03:07.607793+0000","last_became_active":"2024-09-21T13:59:32.204380+0000","last_became_peered":"2024-09-21T13:59:32.204380+0000","last_unstale":"2024-09-21T14:03:07.607793+0000","last_undegraded":"2024-09-21T14:03:07.607793+0000","last_fullsized":"2024-09-21T14:03:07.607793+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:00:57.128713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:00:57.128713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:24.563060+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":587,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.626037+0000","last_change":"2024-09-21T13:59:32.212615+0000","last_active":"2024-09-21T14:03:08.626037+0000","last_peered":"2024-09-21T14:03:08.626037+0000","last_clean":"2024-09-21T14:03:08.626037+0000","last_became_active":"2024-09-21T13:59:32.210134+0000","last_became_peered":"2024-09-21T13:59:32.210134+0000","last_unstale":"2024-09-21T14:03:08.626037+0000","last_undegraded":"2024-09-21T14:03:08.626037+0000","last_fullsized":"2024-09-21T14:03:08.626037+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:08.852789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:55:08.852789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":701,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.616321+0000","last_change":"2024-09-21T14:01:41.187792+0000","last_active":"2024-09-21T14:03:08.616321+0000","last_peered":"2024-09-21T14:03:08.616321+0000","last_clean":"2024-09-21T14:03:08.616321+0000","last_became_active":"2024-09-21T14:01:41.187438+0000","last_became_peered":"2024-09-21T14:01:41.187438+0000","last_unstale":"2024-09-21T14:03:08.616321+0000","last_undegraded":"2024-09-21T14:03:08.616321+0000","last_fullsized":"2024-09-21T14:03:08.616321+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:17.706325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:17.706325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:40.383161+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":630,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.616467+0000","last_change":"2024-09-21T13:58:13.776632+0000","last_active":"2024-09-21T14:03:08.616467+0000","last_peered":"2024-09-21T14:03:08.616467+0000","last_clean":"2024-09-21T14:03:08.616467+0000","last_became_active":"2024-09-21T13:57:12.704042+0000","last_became_peered":"2024-09-21T13:57:12.704042+0000","last_unstale":"2024-09-21T14:03:08.616467+0000","last_undegraded":"2024-09-21T14:03:08.616467+0000","last_fullsized":"2024-09-21T14:03:08.616467+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:13.776572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:13.776572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"290'19","reported_seq":299,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-21T14:03:07.608518+0000","last_change":"2024-09-21T14:01:54.680822+0000","last_active":"2024-09-21T14:03:07.608518+0000","last_peered":"2024-09-21T14:03:07.608518+0000","last_clean":"2024-09-21T14:03:07.608518+0000","last_became_active":"2024-09-21T14:01:41.194606+0000","last_became_peered":"2024-09-21T14:01:41.194606+0000","last_unstale":"2024-09-21T14:03:07.608518+0000","last_undegraded":"2024-09-21T14:03:07.608518+0000","last_fullsized":"2024-09-21T14:03:07.608518+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'19","last_scrub_stamp":"2024-09-21T14:01:54.680735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:54.680735+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:21.312249+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":556,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.626833+0000","last_change":"2024-09-21T13:59:32.203045+0000","last_active":"2024-09-21T14:03:08.626833+0000","last_peered":"2024-09-21T14:03:08.626833+0000","last_clean":"2024-09-21T14:03:08.626833+0000","last_became_active":"2024-09-21T13:59:32.202225+0000","last_became_peered":"2024-09-21T13:59:32.202225+0000","last_unstale":"2024-09-21T14:03:08.626833+0000","last_undegraded":"2024-09-21T14:03:08.626833+0000","last_fullsized":"2024-09-21T14:03:08.626833+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:40.650832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:40.650832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":660668416,"num_objects":10081,"num_object_clones":0,"num_object_copies":30243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10081,"num_write_kb":645184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2271215616,"data_stored":2271215616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10081,"ondisk_log_size":10081,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1111,"num_read_kb":868,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":1914,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":687,"ondisk_log_size":687,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":225,"num_read_kb":220,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":375688,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":139,"num_write":73,"num_write_kb":1036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1031048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":298,"seq":1279900254253,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":564404,"kb_used_data":260516,"kb_used_omap":4,"kb_used_meta":303867,"kb_avail":93180748,"statfs":{"total":95995035648,"available":95417085952,"internally_reserved":0,"allocated":266768384,"data_stored":265872696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":311160488},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463092,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":583188,"kb_used_data":249348,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":93161964,"statfs":{"total":95995035648,"available":95397851136,"internally_reserved":0,"allocated":255332352,"data_stored":254460267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":341834182},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":345,"seq":1481763717139,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":398436,"kb_used_data":289624,"kb_used_omap":4,"kb_used_meta":108795,"kb_avail":93346716,"statfs":{"total":95995035648,"available":95587037184,"internally_reserved":0,"allocated":296574976,"data_stored":295691951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":111406755},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920961,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":611192,"kb_used_data":232932,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":93133960,"statfs":{"total":95995035648,"available":95369175040,"internally_reserved":0,"allocated":238522368,"data_stored":237618133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":387316165},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":240,"seq":1030792151113,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":591628,"kb_used_data":337208,"kb_used_omap":4,"kb_used_meta":254395,"kb_avail":93153524,"statfs":{"total":95995035648,"available":95389208576,"internally_reserved":0,"allocated":345300992,"data_stored":344385731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4443,"internal_metadata":260501157},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":130,"seq":558345748614,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":633484,"kb_used_data":83452,"kb_used_omap":4,"kb_used_meta":550011,"kb_avail":93111668,"statfs":{"total":95995035648,"available":95346348032,"internally_reserved":0,"allocated":85454848,"data_stored":84584037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4441,"internal_metadata":563211943},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575059,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":708648,"kb_used_data":136016,"kb_used_omap":1,"kb_used_meta":572606,"kb_avail":93036504,"statfs":{"total":95995035648,"available":95269380096,"internally_reserved":0,"allocated":139280384,"data_stored":138339917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":586348997},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738585,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":877356,"kb_used_data":363032,"kb_used_omap":1,"kb_used_meta":514302,"kb_avail":92867796,"statfs":{"total":95995035648,"available":95096623104,"internally_reserved":0,"allocated":371744768,"data_stored":370852561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":526645701},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":459997184,"data_stored":459997184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151519232,"data_stored":151519232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":162463744,"data_stored":162463744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":344391680,"data_stored":344391680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":297533440,"data_stored":297533440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":319881216,"data_stored":319881216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":258211840,"data_stored":258211840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":277217280,"data_stored":277217280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T14:03:18.151 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:03:18.184 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:18.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:18.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:03:18.287 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:03:18.323 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:18.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:18.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:03:18.425 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:03:18.460 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:18.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:18.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:03:18.508 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:03:18.561 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:03:18.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:18.599 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:18.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:03:18.700 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:03:18.738 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:18.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:18.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:03:18.840 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:18.867 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 13 16 33915 33899 162.954 119.438 0.00277061 0.00613044 2024-09-21T14:03:18.875 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:18.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:18.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:18.978 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:03:19.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:19.002 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:19.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:03:19.006 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T14:03:19.104 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:03:19.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:19.142 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:19.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:03:19.180 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T14:03:19.243 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:03:19.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:19 smithi049 ceph-mon[30080]: pgmap v732: 158 pgs: 158 active+clean; 1.6 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 152 MiB/s wr, 2.42k op/s 2024-09-21T14:03:19.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:19 smithi049 ceph-mon[40489]: pgmap v732: 158 pgs: 158 active+clean; 1.6 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 152 MiB/s wr, 2.42k op/s 2024-09-21T14:03:19.288 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:19.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:19.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:03:19.390 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:03:19.423 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:19.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:19.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:03:19.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:19 smithi187 ceph-mon[33032]: pgmap v732: 158 pgs: 158 active+clean; 1.6 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 152 MiB/s wr, 2.42k op/s 2024-09-21T14:03:19.525 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:03:19.578 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:19.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:19.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:03:19.626 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:03:19.679 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:03:19.711 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:19.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:19.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:03:19.747 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:03:19.813 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:19.836 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:19.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:19.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:19.867 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 14 16 35581 35565 158.751 104.125 0.0018596 0.00627384 2024-09-21T14:03:19.938 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:03:19.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:19.983 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:19.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:03:20.075 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:03:19 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:03:20.083 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:03:20.085 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-09-21T14:03:20.143 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:20.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:20.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:03:20.246 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:03:20.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:20 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:03:20.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:20 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/630205676' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:03:20.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:20 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:03:20.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:20 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:03:20.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:20 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/630205676' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:03:20.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:20 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:03:20.280 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:20.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:20.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:03:20.382 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:03:20.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:20.420 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:20.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:03:20.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:20 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:03:20.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:20 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/630205676' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:03:20.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:20 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:03:20.521 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:03:20.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:20.556 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:20.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:03:20.657 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:03:20.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:20.698 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:20.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:03:20.799 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:03:20.834 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:20.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:20.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:03:20.867 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 15 15 37380 37365 155.667 112.5 0.00827623 0.00641786 2024-09-21T14:03:20.889 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:03:20 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:03:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:03:20.936 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:03:20.969 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:20.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:20.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:03:20.977 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:03:20.977 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:03:21.015 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:03:21.015 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:03:21.049 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:03:21.049 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:03:21.070 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:03:21.085 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:03:21.085 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:03:21.116 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:21.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:21.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:03:21.141 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T14:03:21.180 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:21 smithi049 ceph-mon[40489]: pgmap v733: 158 pgs: 158 active+clean; 2.0 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 898 B/s rd, 141 MiB/s wr, 2.26k op/s 2024-09-21T14:03:21.180 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:21 smithi049 ceph-mon[40489]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:03:21.180 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:21 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T14:03:21.180 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:21 smithi049 ceph-mon[40489]: osdmap e389: 8 total, 8 up, 8 in 2024-09-21T14:03:21.180 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:21 smithi049 ceph-mon[40489]: osdmap e390: 8 total, 8 up, 8 in 2024-09-21T14:03:21.180 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:21 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3430068745' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:03:21.181 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:21 smithi049 ceph-mon[30080]: pgmap v733: 158 pgs: 158 active+clean; 2.0 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 898 B/s rd, 141 MiB/s wr, 2.26k op/s 2024-09-21T14:03:21.181 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:21 smithi049 ceph-mon[30080]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:03:21.181 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:21 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T14:03:21.181 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:21 smithi049 ceph-mon[30080]: osdmap e389: 8 total, 8 up, 8 in 2024-09-21T14:03:21.181 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:21 smithi049 ceph-mon[30080]: osdmap e390: 8 total, 8 up, 8 in 2024-09-21T14:03:21.181 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:21 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3430068745' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:03:21.217 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:21.239 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:03:21.239 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:03:21.247 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:21.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:21.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:21.276 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:03:21.276 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:03:21.314 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:03:21.314 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:03:21.348 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:03:21.351 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:03:21.351 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:03:21.372 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:21.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:21.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:03:21.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:21 smithi187 ceph-mon[33032]: pgmap v733: 158 pgs: 158 active+clean; 2.0 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 898 B/s rd, 141 MiB/s wr, 2.26k op/s 2024-09-21T14:03:21.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:21 smithi187 ceph-mon[33032]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:03:21.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:21 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T14:03:21.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:21 smithi187 ceph-mon[33032]: osdmap e389: 8 total, 8 up, 8 in 2024-09-21T14:03:21.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:21 smithi187 ceph-mon[33032]: osdmap e390: 8 total, 8 up, 8 in 2024-09-21T14:03:21.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:21 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3430068745' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:03:21.473 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:03:21.506 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:21.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:21.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:03:21.608 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:03:21.644 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:21.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:21.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:03:21.746 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:03:21.785 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:21.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:21.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:03:21.867 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 16 16 38946 38930 152.051 97.8125 0.0046807 0.00657146 2024-09-21T14:03:21.887 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:03:21.922 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:21.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:21.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:03:22.024 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:03:22.045 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:22.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:22.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:03:22.147 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:03:22.181 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:22.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:22.182 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:03:22.240 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:03:22.283 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:03:22.313 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:22.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:22.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:03:22.415 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:03:22.445 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:22.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:22.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:03:22.547 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:03:22.587 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:22.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:22.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:03:22.689 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:03:22.724 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:22.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:22.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:03:22.826 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:03:22.860 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:22.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:22.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:03:22.867 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 17 16 40633 40617 149.309 105.438 0.00599581 0.00669237 2024-09-21T14:03:22.962 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:03:23.003 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:23.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:23.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:03:23.105 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:03:23.144 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:23.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:23.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:03:23.246 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:03:23.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:23 smithi049 ceph-mon[30080]: pgmap v736: 158 pgs: 158 active+clean; 2.1 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 105 MiB/s wr, 1.68k op/s 2024-09-21T14:03:23.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:23 smithi049 ceph-mon[40489]: pgmap v736: 158 pgs: 158 active+clean; 2.1 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 105 MiB/s wr, 1.68k op/s 2024-09-21T14:03:23.281 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:23.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:23.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:03:23.383 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:23.419 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:23.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:23.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:23.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:23 smithi187 ceph-mon[33032]: pgmap v736: 158 pgs: 158 active+clean; 2.1 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 105 MiB/s wr, 1.68k op/s 2024-09-21T14:03:23.521 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:23.553 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T14:03:23.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:23.559 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:23.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:23.660 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:03:23.704 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:23.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:23.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:03:23.806 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:03:23.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:23.858 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:23.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:03:23.869 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 18 16 42344 42328 146.954 106.938 0.0125495 0.00679995 2024-09-21T14:03:23.959 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:03:23.992 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:23.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:23.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:03:24.088 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:03:24.094 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:03:24.108 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T14:03:24.108 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:03:24.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:24.147 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:24.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:03:24.248 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:03:24.281 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:24.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:24.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:03:24.341 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T14:03:24.343 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-09-21T14:03:24.382 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:03:24.428 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:24.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:24.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:03:24.530 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:03:24.566 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:24.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:24.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:03:24.668 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:03:24.714 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:24.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:24.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:03:24.818 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:03:24.868 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 19 16 43895 43879 144.321 96.9375 0.00516086 0.00692216 2024-09-21T14:03:24.871 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:24.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:24.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:03:24.973 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:03:24.996 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:24.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:24.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:03:25.090 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:25 smithi049 ceph-mon[30080]: pgmap v737: 158 pgs: 158 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 138 MiB/s wr, 2.20k op/s 2024-09-21T14:03:25.090 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:25 smithi049 ceph-mon[30080]: from='client.16095 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:03:25.098 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:03:25.144 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:25.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:25.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:03:25.246 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:03:25.278 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:25.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:25.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:03:25.356 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T14:03:25.373 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:25 smithi187 ceph-mon[33032]: pgmap v737: 158 pgs: 158 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 138 MiB/s wr, 2.20k op/s 2024-09-21T14:03:25.373 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:25 smithi187 ceph-mon[33032]: from='client.16095 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:03:25.373 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:25 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T14:03:25.380 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:03:25.391 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:25 smithi049 ceph-mon[40489]: pgmap v737: 158 pgs: 158 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 138 MiB/s wr, 2.20k op/s 2024-09-21T14:03:25.391 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:25 smithi049 ceph-mon[40489]: from='client.16095 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:03:25.391 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:25 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T14:03:25.392 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:25 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T14:03:25.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:25.435 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:25.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:03:25.536 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:03:25.564 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:25.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:25.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:03:25.611 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":737,"stamp":"2024-09-21T14:03:24.006341+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2649936771,"num_objects":40628,"num_object_clones":0,"num_object_copies":121884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40628,"num_whiteouts":0,"num_read":1418,"num_read_kb":1237,"num_write":41224,"num_write_kb":2588537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":169,"num_bytes_recovered":1032962,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42100,"ondisk_log_size":42100,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":10525912,"kb_used_data":7290380,"kb_used_omap":23,"kb_used_meta":3235368,"kb_avail":739435304,"statfs":{"total":767960285184,"available":757181751296,"internally_reserved":0,"allocated":7465349120,"data_stored":7458103905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24148,"internal_metadata":3313017260},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,3,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1154744320,"num_objects":17620,"num_object_clones":0,"num_object_copies":52860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17620,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":17620,"num_write_kb":1127680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.004145"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":604,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.115463+0000","last_change":"2024-09-21T14:02:56.458231+0000","last_active":"2024-09-21T14:03:20.115463+0000","last_peered":"2024-09-21T14:03:20.115463+0000","last_clean":"2024-09-21T14:03:20.115463+0000","last_became_active":"2024-09-21T14:01:41.189398+0000","last_became_peered":"2024-09-21T14:01:41.189398+0000","last_unstale":"2024-09-21T14:03:20.115463+0000","last_undegraded":"2024-09-21T14:03:20.115463+0000","last_fullsized":"2024-09-21T14:03:20.115463+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:56.458125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:02:56.458125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:20.867402+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"290'15","reported_seq":771,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.116355+0000","last_change":"2024-09-21T14:01:34.785322+0000","last_active":"2024-09-21T14:03:20.116355+0000","last_peered":"2024-09-21T14:03:20.116355+0000","last_clean":"2024-09-21T14:03:20.116355+0000","last_became_active":"2024-09-21T13:58:17.771220+0000","last_became_peered":"2024-09-21T13:58:17.771220+0000","last_unstale":"2024-09-21T14:03:20.116355+0000","last_undegraded":"2024-09-21T14:03:20.116355+0000","last_fullsized":"2024-09-21T14:03:20.116355+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'15","last_scrub_stamp":"2024-09-21T14:01:34.785267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:34.785267+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:53.703961+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":708,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.614729+0000","last_change":"2024-09-21T14:02:29.205437+0000","last_active":"2024-09-21T14:03:08.614729+0000","last_peered":"2024-09-21T14:03:08.614729+0000","last_clean":"2024-09-21T14:03:08.614729+0000","last_became_active":"2024-09-21T14:01:41.202168+0000","last_became_peered":"2024-09-21T14:01:41.202168+0000","last_unstale":"2024-09-21T14:03:08.614729+0000","last_undegraded":"2024-09-21T14:03:08.614729+0000","last_fullsized":"2024-09-21T14:03:08.614729+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:29.205343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:02:29.205343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:29.369063+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":688,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.112386+0000","last_change":"2024-09-21T14:01:41.203762+0000","last_active":"2024-09-21T14:03:20.112386+0000","last_peered":"2024-09-21T14:03:20.112386+0000","last_clean":"2024-09-21T14:03:20.112386+0000","last_became_active":"2024-09-21T14:01:41.203275+0000","last_became_peered":"2024-09-21T14:01:41.203275+0000","last_unstale":"2024-09-21T14:03:20.112386+0000","last_undegraded":"2024-09-21T14:03:20.112386+0000","last_fullsized":"2024-09-21T14:03:20.112386+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:00.320510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:00.320510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:46.689692+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":658,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.113755+0000","last_change":"2024-09-21T13:59:32.208701+0000","last_active":"2024-09-21T14:03:20.113755+0000","last_peered":"2024-09-21T14:03:20.113755+0000","last_clean":"2024-09-21T14:03:20.113755+0000","last_became_active":"2024-09-21T13:59:32.203340+0000","last_became_peered":"2024-09-21T13:59:32.203340+0000","last_unstale":"2024-09-21T14:03:20.113755+0000","last_undegraded":"2024-09-21T14:03:20.113755+0000","last_fullsized":"2024-09-21T14:03:20.113755+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:34.696780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:34.696780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:44.274358+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":671,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.117702+0000","last_change":"2024-09-21T13:58:17.778322+0000","last_active":"2024-09-21T14:03:20.117702+0000","last_peered":"2024-09-21T14:03:20.117702+0000","last_clean":"2024-09-21T14:03:20.117702+0000","last_became_active":"2024-09-21T13:58:17.777893+0000","last_became_peered":"2024-09-21T13:58:17.777893+0000","last_unstale":"2024-09-21T14:03:20.117702+0000","last_undegraded":"2024-09-21T14:03:20.117702+0000","last_fullsized":"2024-09-21T14:03:20.117702+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:07.001135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:55:07.001135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:22.269018+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"290'27","reported_seq":644,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.111310+0000","last_change":"2024-09-21T13:58:11.718421+0000","last_active":"2024-09-21T14:03:20.111310+0000","last_peered":"2024-09-21T14:03:20.111310+0000","last_clean":"2024-09-21T14:03:20.111310+0000","last_became_active":"2024-09-21T13:57:12.703469+0000","last_became_peered":"2024-09-21T13:57:12.703469+0000","last_unstale":"2024-09-21T14:03:20.111310+0000","last_undegraded":"2024-09-21T14:03:20.111310+0000","last_fullsized":"2024-09-21T14:03:20.111310+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:58:11.718334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:11.718334+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":719,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.116268+0000","last_change":"2024-09-21T14:03:02.862866+0000","last_active":"2024-09-21T14:03:20.116268+0000","last_peered":"2024-09-21T14:03:20.116268+0000","last_clean":"2024-09-21T14:03:20.116268+0000","last_became_active":"2024-09-21T13:58:17.773464+0000","last_became_peered":"2024-09-21T13:58:17.773464+0000","last_unstale":"2024-09-21T14:03:20.116268+0000","last_undegraded":"2024-09-21T14:03:20.116268+0000","last_fullsized":"2024-09-21T14:03:20.116268+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:02.862787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:02.862787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:17.663483+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":663,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.114735+0000","last_change":"2024-09-21T14:01:41.194807+0000","last_active":"2024-09-21T14:03:20.114735+0000","last_peered":"2024-09-21T14:03:20.114735+0000","last_clean":"2024-09-21T14:03:20.114735+0000","last_became_active":"2024-09-21T14:01:41.194517+0000","last_became_peered":"2024-09-21T14:01:41.194517+0000","last_unstale":"2024-09-21T14:03:20.114735+0000","last_undegraded":"2024-09-21T14:03:20.114735+0000","last_fullsized":"2024-09-21T14:03:20.114735+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:31.127926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:31.127926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:10.380240+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"290'15","reported_seq":737,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.113297+0000","last_change":"2024-09-21T14:00:02.475761+0000","last_active":"2024-09-21T14:03:20.113297+0000","last_peered":"2024-09-21T14:03:20.113297+0000","last_clean":"2024-09-21T14:03:20.113297+0000","last_became_active":"2024-09-21T13:57:12.681408+0000","last_became_peered":"2024-09-21T13:57:12.681408+0000","last_unstale":"2024-09-21T14:03:20.113297+0000","last_undegraded":"2024-09-21T14:03:20.113297+0000","last_fullsized":"2024-09-21T14:03:20.113297+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'15","last_scrub_stamp":"2024-09-21T14:00:02.475661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:00:02.475661+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:43.713097+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":612,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.114518+0000","last_change":"2024-09-21T14:03:04.516642+0000","last_active":"2024-09-21T14:03:20.114518+0000","last_peered":"2024-09-21T14:03:20.114518+0000","last_clean":"2024-09-21T14:03:20.114518+0000","last_became_active":"2024-09-21T14:01:41.188248+0000","last_became_peered":"2024-09-21T14:01:41.188248+0000","last_unstale":"2024-09-21T14:03:20.114518+0000","last_undegraded":"2024-09-21T14:03:20.114518+0000","last_fullsized":"2024-09-21T14:03:20.114518+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:04.516583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:04.516583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:26.082827+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":638,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.617379+0000","last_change":"2024-09-21T13:59:32.202128+0000","last_active":"2024-09-21T14:03:08.617379+0000","last_peered":"2024-09-21T14:03:08.617379+0000","last_clean":"2024-09-21T14:03:08.617379+0000","last_became_active":"2024-09-21T13:59:32.201158+0000","last_became_peered":"2024-09-21T13:59:32.201158+0000","last_unstale":"2024-09-21T14:03:08.617379+0000","last_undegraded":"2024-09-21T14:03:08.617379+0000","last_fullsized":"2024-09-21T14:03:08.617379+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:25.792985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:54:25.792985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:49.267653+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":640,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.113934+0000","last_change":"2024-09-21T14:01:41.189797+0000","last_active":"2024-09-21T14:03:20.113934+0000","last_peered":"2024-09-21T14:03:20.113934+0000","last_clean":"2024-09-21T14:03:20.113934+0000","last_became_active":"2024-09-21T14:01:41.189373+0000","last_became_peered":"2024-09-21T14:01:41.189373+0000","last_unstale":"2024-09-21T14:03:20.113934+0000","last_undegraded":"2024-09-21T14:03:20.113934+0000","last_fullsized":"2024-09-21T14:03:20.113934+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:54.328275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:54.328275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:09.403771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"290'7","reported_seq":762,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.112565+0000","last_change":"2024-09-21T14:01:57.563250+0000","last_active":"2024-09-21T14:03:20.112565+0000","last_peered":"2024-09-21T14:03:20.112565+0000","last_clean":"2024-09-21T14:03:20.112565+0000","last_became_active":"2024-09-21T13:59:09.063053+0000","last_became_peered":"2024-09-21T13:59:09.063053+0000","last_unstale":"2024-09-21T14:03:20.112565+0000","last_undegraded":"2024-09-21T14:03:20.112565+0000","last_fullsized":"2024-09-21T14:03:20.112565+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'7","last_scrub_stamp":"2024-09-21T14:01:57.563184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:57.563184+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":633,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.115649+0000","last_change":"2024-09-21T13:59:32.204565+0000","last_active":"2024-09-21T14:03:20.115649+0000","last_peered":"2024-09-21T14:03:20.115649+0000","last_clean":"2024-09-21T14:03:20.115649+0000","last_became_active":"2024-09-21T13:59:32.204174+0000","last_became_peered":"2024-09-21T13:59:32.204174+0000","last_unstale":"2024-09-21T14:03:20.115649+0000","last_undegraded":"2024-09-21T14:03:20.115649+0000","last_fullsized":"2024-09-21T14:03:20.115649+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:06.722293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:06.722293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":674,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.115071+0000","last_change":"2024-09-21T14:01:41.199085+0000","last_active":"2024-09-21T14:03:20.115071+0000","last_peered":"2024-09-21T14:03:20.115071+0000","last_clean":"2024-09-21T14:03:20.115071+0000","last_became_active":"2024-09-21T14:01:41.198672+0000","last_became_peered":"2024-09-21T14:01:41.198672+0000","last_unstale":"2024-09-21T14:03:20.115071+0000","last_undegraded":"2024-09-21T14:03:20.115071+0000","last_fullsized":"2024-09-21T14:03:20.115071+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:44.376215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:44.376215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":731,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.113520+0000","last_change":"2024-09-21T14:01:53.460257+0000","last_active":"2024-09-21T14:03:20.113520+0000","last_peered":"2024-09-21T14:03:20.113520+0000","last_clean":"2024-09-21T14:03:20.113520+0000","last_became_active":"2024-09-21T14:01:41.200360+0000","last_became_peered":"2024-09-21T14:01:41.200360+0000","last_unstale":"2024-09-21T14:03:20.113520+0000","last_undegraded":"2024-09-21T14:03:20.113520+0000","last_fullsized":"2024-09-21T14:03:20.113520+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:53.460180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:01:53.460180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":649,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.112741+0000","last_change":"2024-09-21T14:02:36.679396+0000","last_active":"2024-09-21T14:03:20.112741+0000","last_peered":"2024-09-21T14:03:20.112741+0000","last_clean":"2024-09-21T14:03:20.112741+0000","last_became_active":"2024-09-21T13:59:32.204749+0000","last_became_peered":"2024-09-21T13:59:32.204749+0000","last_unstale":"2024-09-21T14:03:20.112741+0000","last_undegraded":"2024-09-21T14:03:20.112741+0000","last_fullsized":"2024-09-21T14:03:20.112741+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:36.679321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:02:36.679321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:43.067225+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"290'27","reported_seq":713,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.115264+0000","last_change":"2024-09-21T14:02:55.620755+0000","last_active":"2024-09-21T14:03:20.115264+0000","last_peered":"2024-09-21T14:03:20.115264+0000","last_clean":"2024-09-21T14:03:20.115264+0000","last_became_active":"2024-09-21T13:57:12.680783+0000","last_became_peered":"2024-09-21T13:57:12.680783+0000","last_unstale":"2024-09-21T14:03:20.115264+0000","last_undegraded":"2024-09-21T14:03:20.115264+0000","last_fullsized":"2024-09-21T14:03:20.115264+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'27","last_scrub_stamp":"2024-09-21T14:02:55.620700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.620700+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:01.085594+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":702,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.616652+0000","last_change":"2024-09-21T14:03:08.299319+0000","last_active":"2024-09-21T14:03:08.616652+0000","last_peered":"2024-09-21T14:03:08.616652+0000","last_clean":"2024-09-21T14:03:08.616652+0000","last_became_active":"2024-09-21T13:49:02.066796+0000","last_became_peered":"2024-09-21T13:49:02.066796+0000","last_unstale":"2024-09-21T14:03:08.616652+0000","last_undegraded":"2024-09-21T14:03:08.616652+0000","last_fullsized":"2024-09-21T14:03:08.616652+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:08.299252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:08.299252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:29.982454+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":650,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.120108+0000","last_change":"2024-09-21T14:01:41.191486+0000","last_active":"2024-09-21T14:03:20.120108+0000","last_peered":"2024-09-21T14:03:20.120108+0000","last_clean":"2024-09-21T14:03:20.120108+0000","last_became_active":"2024-09-21T14:01:41.191116+0000","last_became_peered":"2024-09-21T14:01:41.191116+0000","last_unstale":"2024-09-21T14:03:20.120108+0000","last_undegraded":"2024-09-21T14:03:20.120108+0000","last_fullsized":"2024-09-21T14:03:20.120108+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:57.962266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:57.962266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:10.160181+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":708,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.113433+0000","last_change":"2024-09-21T14:00:51.495433+0000","last_active":"2024-09-21T14:03:20.113433+0000","last_peered":"2024-09-21T14:03:20.113433+0000","last_clean":"2024-09-21T14:03:20.113433+0000","last_became_active":"2024-09-21T13:52:04.316016+0000","last_became_peered":"2024-09-21T13:52:04.316016+0000","last_unstale":"2024-09-21T14:03:20.113433+0000","last_undegraded":"2024-09-21T14:03:20.113433+0000","last_fullsized":"2024-09-21T14:03:20.113433+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:00:51.495368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:00:51.495368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:12.877210+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"290'18","reported_seq":656,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.119960+0000","last_change":"2024-09-21T13:59:23.041396+0000","last_active":"2024-09-21T14:03:20.119960+0000","last_peered":"2024-09-21T14:03:20.119960+0000","last_clean":"2024-09-21T14:03:20.119960+0000","last_became_active":"2024-09-21T13:57:12.684454+0000","last_became_peered":"2024-09-21T13:57:12.684454+0000","last_unstale":"2024-09-21T14:03:20.119960+0000","last_undegraded":"2024-09-21T14:03:20.119960+0000","last_fullsized":"2024-09-21T14:03:20.119960+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'18","last_scrub_stamp":"2024-09-21T13:59:23.041311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:23.041311+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":695,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.112435+0000","last_change":"2024-09-21T13:58:16.393763+0000","last_active":"2024-09-21T14:03:20.112435+0000","last_peered":"2024-09-21T14:03:20.112435+0000","last_clean":"2024-09-21T14:03:20.112435+0000","last_became_active":"2024-09-21T13:57:12.681897+0000","last_became_peered":"2024-09-21T13:57:12.681897+0000","last_unstale":"2024-09-21T14:03:20.112435+0000","last_undegraded":"2024-09-21T14:03:20.112435+0000","last_fullsized":"2024-09-21T14:03:20.112435+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:16.393696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:16.393696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:37.708367+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":617,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.115799+0000","last_change":"2024-09-21T14:01:41.182300+0000","last_active":"2024-09-21T14:03:20.115799+0000","last_peered":"2024-09-21T14:03:20.115799+0000","last_clean":"2024-09-21T14:03:20.115799+0000","last_became_active":"2024-09-21T14:01:41.181946+0000","last_became_peered":"2024-09-21T14:01:41.181946+0000","last_unstale":"2024-09-21T14:03:20.115799+0000","last_undegraded":"2024-09-21T14:03:20.115799+0000","last_fullsized":"2024-09-21T14:03:20.115799+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T13:57:26.536684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:26.536684+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":714,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.614861+0000","last_change":"2024-09-21T14:01:58.994867+0000","last_active":"2024-09-21T14:03:08.614861+0000","last_peered":"2024-09-21T14:03:08.614861+0000","last_clean":"2024-09-21T14:03:08.614861+0000","last_became_active":"2024-09-21T13:57:12.685742+0000","last_became_peered":"2024-09-21T13:57:12.685742+0000","last_unstale":"2024-09-21T14:03:08.614861+0000","last_undegraded":"2024-09-21T14:03:08.614861+0000","last_fullsized":"2024-09-21T14:03:08.614861+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T14:01:58.994811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:01:58.994811+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:17.687542+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"290'20","reported_seq":597,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.117028+0000","last_change":"2024-09-21T13:59:32.242771+0000","last_active":"2024-09-21T14:03:20.117028+0000","last_peered":"2024-09-21T14:03:20.117028+0000","last_clean":"2024-09-21T14:03:20.117028+0000","last_became_active":"2024-09-21T13:59:32.213865+0000","last_became_peered":"2024-09-21T13:59:32.213865+0000","last_unstale":"2024-09-21T14:03:20.117028+0000","last_undegraded":"2024-09-21T14:03:20.117028+0000","last_fullsized":"2024-09-21T14:03:20.117028+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:57:27.578738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:57:27.578738+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"300'12","reported_seq":341,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.111358+0000","last_change":"2024-09-21T13:59:36.012775+0000","last_active":"2024-09-21T14:03:20.111358+0000","last_peered":"2024-09-21T14:03:20.111358+0000","last_clean":"2024-09-21T14:03:20.111358+0000","last_became_active":"2024-09-21T13:59:32.205051+0000","last_became_peered":"2024-09-21T13:59:32.205051+0000","last_unstale":"2024-09-21T14:03:20.111358+0000","last_undegraded":"2024-09-21T14:03:20.111358+0000","last_fullsized":"2024-09-21T14:03:20.111358+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'12","last_scrub_stamp":"2024-09-21T13:59:36.012693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:36.012693+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:21.546366+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"390'2664","reported_seq":2693,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:23.584927+0000","last_change":"2024-09-21T14:02:56.560786+0000","last_active":"2024-09-21T14:03:23.584927+0000","last_peered":"2024-09-21T14:03:23.584927+0000","last_clean":"2024-09-21T14:03:23.584927+0000","last_became_active":"2024-09-21T14:02:56.559536+0000","last_became_peered":"2024-09-21T14:02:56.559536+0000","last_unstale":"2024-09-21T14:03:23.584927+0000","last_undegraded":"2024-09-21T14:03:23.584927+0000","last_fullsized":"2024-09-21T14:03:23.584927+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":2664,"log_dups_size":0,"ondisk_log_size":2664,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:03.627467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174587904,"num_objects":2664,"num_object_clones":0,"num_object_copies":7992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2664,"num_write_kb":170496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"75'79","reported_seq":113,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.113975+0000","last_change":"2024-09-21T14:01:41.248772+0000","last_active":"2024-09-21T14:03:20.113975+0000","last_peered":"2024-09-21T14:03:20.113975+0000","last_clean":"2024-09-21T14:03:20.113975+0000","last_became_active":"2024-09-21T14:01:41.246888+0000","last_became_peered":"2024-09-21T14:01:41.246888+0000","last_unstale":"2024-09-21T14:03:20.113975+0000","last_undegraded":"2024-09-21T14:03:20.113975+0000","last_fullsized":"2024-09-21T14:03:20.113975+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:03:52.423493+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53669,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"290'27","reported_seq":761,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.617486+0000","last_change":"2024-09-21T14:02:58.293228+0000","last_active":"2024-09-21T14:03:08.617486+0000","last_peered":"2024-09-21T14:03:08.617486+0000","last_clean":"2024-09-21T14:03:08.617486+0000","last_became_active":"2024-09-21T13:57:12.695609+0000","last_became_peered":"2024-09-21T13:57:12.695609+0000","last_unstale":"2024-09-21T14:03:08.617486+0000","last_undegraded":"2024-09-21T14:03:08.617486+0000","last_fullsized":"2024-09-21T14:03:08.617486+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'27","last_scrub_stamp":"2024-09-21T14:02:58.293160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:58.293160+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:01.203771+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":642,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.108465+0000","last_change":"2024-09-21T13:59:32.205723+0000","last_active":"2024-09-21T14:03:20.108465+0000","last_peered":"2024-09-21T14:03:20.108465+0000","last_clean":"2024-09-21T14:03:20.108465+0000","last_became_active":"2024-09-21T13:59:32.205350+0000","last_became_peered":"2024-09-21T13:59:32.205350+0000","last_unstale":"2024-09-21T14:03:20.108465+0000","last_undegraded":"2024-09-21T14:03:20.108465+0000","last_fullsized":"2024-09-21T14:03:20.108465+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:32.668851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:32.668851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:47.252073+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"346'10","reported_seq":905,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.113645+0000","last_change":"2024-09-21T14:02:59.831183+0000","last_active":"2024-09-21T14:03:20.113645+0000","last_peered":"2024-09-21T14:03:20.113645+0000","last_clean":"2024-09-21T14:03:20.113645+0000","last_became_active":"2024-09-21T14:01:41.200218+0000","last_became_peered":"2024-09-21T14:01:41.200218+0000","last_unstale":"2024-09-21T14:03:20.113645+0000","last_undegraded":"2024-09-21T14:03:20.113645+0000","last_fullsized":"2024-09-21T14:03:20.113645+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'10","last_scrub_stamp":"2024-09-21T14:02:59.831128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:02:59.831128+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:08.243919+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":569,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.114241+0000","last_change":"2024-09-21T14:01:41.207310+0000","last_active":"2024-09-21T14:03:20.114241+0000","last_peered":"2024-09-21T14:03:20.114241+0000","last_clean":"2024-09-21T14:03:20.114241+0000","last_became_active":"2024-09-21T14:01:41.206720+0000","last_became_peered":"2024-09-21T14:01:41.206720+0000","last_unstale":"2024-09-21T14:03:20.114241+0000","last_undegraded":"2024-09-21T14:03:20.114241+0000","last_fullsized":"2024-09-21T14:03:20.114241+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:52.244483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:52.244483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"390'2513","reported_seq":2542,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:23.607918+0000","last_change":"2024-09-21T14:02:56.562998+0000","last_active":"2024-09-21T14:03:23.607918+0000","last_peered":"2024-09-21T14:03:23.607918+0000","last_clean":"2024-09-21T14:03:23.607918+0000","last_became_active":"2024-09-21T14:02:56.562524+0000","last_became_peered":"2024-09-21T14:02:56.562524+0000","last_unstale":"2024-09-21T14:03:23.607918+0000","last_undegraded":"2024-09-21T14:03:23.607918+0000","last_fullsized":"2024-09-21T14:03:23.607918+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":2513,"log_dups_size":0,"ondisk_log_size":2513,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:01.881695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164691968,"num_objects":2513,"num_object_clones":0,"num_object_copies":7539,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2513,"num_write_kb":160832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"75'79","reported_seq":167,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.117031+0000","last_change":"2024-09-21T14:02:06.395083+0000","last_active":"2024-09-21T14:03:20.117031+0000","last_peered":"2024-09-21T14:03:20.117031+0000","last_clean":"2024-09-21T14:03:20.117031+0000","last_became_active":"2024-09-21T14:00:41.137756+0000","last_became_peered":"2024-09-21T14:00:41.137756+0000","last_unstale":"2024-09-21T14:03:20.117031+0000","last_undegraded":"2024-09-21T14:03:20.117031+0000","last_fullsized":"2024-09-21T14:03:20.117031+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:02:06.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:02:06.395011+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:29.136654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"290'30","reported_seq":667,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.108581+0000","last_change":"2024-09-21T13:59:32.246090+0000","last_active":"2024-09-21T14:03:20.108581+0000","last_peered":"2024-09-21T14:03:20.108581+0000","last_clean":"2024-09-21T14:03:20.108581+0000","last_became_active":"2024-09-21T13:59:32.205887+0000","last_became_peered":"2024-09-21T13:59:32.205887+0000","last_unstale":"2024-09-21T14:03:20.108581+0000","last_undegraded":"2024-09-21T14:03:20.108581+0000","last_fullsized":"2024-09-21T14:03:20.108581+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'18","last_scrub_stamp":"2024-09-21T13:54:19.001030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:19.001030+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:43.319659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":594,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.120388+0000","last_change":"2024-09-21T13:59:32.212559+0000","last_active":"2024-09-21T14:03:20.120388+0000","last_peered":"2024-09-21T14:03:20.120388+0000","last_clean":"2024-09-21T14:03:20.120388+0000","last_became_active":"2024-09-21T13:59:32.211004+0000","last_became_peered":"2024-09-21T13:59:32.211004+0000","last_unstale":"2024-09-21T14:03:20.120388+0000","last_undegraded":"2024-09-21T14:03:20.120388+0000","last_fullsized":"2024-09-21T14:03:20.120388+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:25.736461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:25.736461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:48.292957+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":712,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.117033+0000","last_change":"2024-09-21T13:59:34.293330+0000","last_active":"2024-09-21T14:03:20.117033+0000","last_peered":"2024-09-21T14:03:20.117033+0000","last_clean":"2024-09-21T14:03:20.117033+0000","last_became_active":"2024-09-21T13:59:32.220304+0000","last_became_peered":"2024-09-21T13:59:32.220304+0000","last_unstale":"2024-09-21T14:03:20.117033+0000","last_undegraded":"2024-09-21T14:03:20.117033+0000","last_fullsized":"2024-09-21T14:03:20.117033+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:34.292892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:34.292892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:19.896473+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":692,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.113199+0000","last_change":"2024-09-21T13:59:23.334230+0000","last_active":"2024-09-21T14:03:20.113199+0000","last_peered":"2024-09-21T14:03:20.113199+0000","last_clean":"2024-09-21T14:03:20.113199+0000","last_became_active":"2024-09-21T13:57:12.680372+0000","last_became_peered":"2024-09-21T13:57:12.680372+0000","last_unstale":"2024-09-21T14:03:20.113199+0000","last_undegraded":"2024-09-21T14:03:20.113199+0000","last_fullsized":"2024-09-21T14:03:20.113199+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:23.334162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:23.334162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"390'2698","reported_seq":2727,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:23.559552+0000","last_change":"2024-09-21T14:02:56.557348+0000","last_active":"2024-09-21T14:03:23.559552+0000","last_peered":"2024-09-21T14:03:23.559552+0000","last_clean":"2024-09-21T14:03:23.559552+0000","last_became_active":"2024-09-21T14:02:56.557055+0000","last_became_peered":"2024-09-21T14:02:56.557055+0000","last_unstale":"2024-09-21T14:03:23.559552+0000","last_undegraded":"2024-09-21T14:03:23.559552+0000","last_fullsized":"2024-09-21T14:03:23.559552+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":2698,"log_dups_size":0,"ondisk_log_size":2698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:20.950710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176816128,"num_objects":2698,"num_object_clones":0,"num_object_copies":8094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2698,"num_write_kb":172672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"75'79","reported_seq":170,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.109523+0000","last_change":"2024-09-21T14:01:42.177646+0000","last_active":"2024-09-21T14:03:20.109523+0000","last_peered":"2024-09-21T14:03:20.109523+0000","last_clean":"2024-09-21T14:03:20.109523+0000","last_became_active":"2024-09-21T14:01:42.173042+0000","last_became_peered":"2024-09-21T14:01:42.173042+0000","last_unstale":"2024-09-21T14:03:20.109523+0000","last_undegraded":"2024-09-21T14:03:20.109523+0000","last_fullsized":"2024-09-21T14:03:20.109523+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:03:42.356436+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":709030,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"290'28","reported_seq":801,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.117091+0000","last_change":"2024-09-21T14:02:02.471226+0000","last_active":"2024-09-21T14:03:20.117091+0000","last_peered":"2024-09-21T14:03:20.117091+0000","last_clean":"2024-09-21T14:03:20.117091+0000","last_became_active":"2024-09-21T14:01:41.200859+0000","last_became_peered":"2024-09-21T14:01:41.200859+0000","last_unstale":"2024-09-21T14:03:20.117091+0000","last_undegraded":"2024-09-21T14:03:20.117091+0000","last_fullsized":"2024-09-21T14:03:20.117091+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'28","last_scrub_stamp":"2024-09-21T14:02:02.471133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:02.471133+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":696,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.627953+0000","last_change":"2024-09-21T13:57:38.046718+0000","last_active":"2024-09-21T14:03:08.627953+0000","last_peered":"2024-09-21T14:03:08.627953+0000","last_clean":"2024-09-21T14:03:08.627953+0000","last_became_active":"2024-09-21T13:48:58.053533+0000","last_became_peered":"2024-09-21T13:48:58.053533+0000","last_unstale":"2024-09-21T14:03:08.627953+0000","last_undegraded":"2024-09-21T14:03:08.627953+0000","last_fullsized":"2024-09-21T14:03:08.627953+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:57:38.046627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:38.046627+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:49.580099+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"241'10","reported_seq":757,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.108775+0000","last_change":"2024-09-21T14:03:03.228551+0000","last_active":"2024-09-21T14:03:20.108775+0000","last_peered":"2024-09-21T14:03:20.108775+0000","last_clean":"2024-09-21T14:03:20.108775+0000","last_became_active":"2024-09-21T13:57:12.686651+0000","last_became_peered":"2024-09-21T13:57:12.686651+0000","last_unstale":"2024-09-21T14:03:20.108775+0000","last_undegraded":"2024-09-21T14:03:20.108775+0000","last_fullsized":"2024-09-21T14:03:20.108775+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'10","last_scrub_stamp":"2024-09-21T14:03:03.228443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:03.228443+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:24.135971+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":658,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.116173+0000","last_change":"2024-09-21T13:59:32.210257+0000","last_active":"2024-09-21T14:03:20.116173+0000","last_peered":"2024-09-21T14:03:20.116173+0000","last_clean":"2024-09-21T14:03:20.116173+0000","last_became_active":"2024-09-21T13:59:32.209615+0000","last_became_peered":"2024-09-21T13:59:32.209615+0000","last_unstale":"2024-09-21T14:03:20.116173+0000","last_undegraded":"2024-09-21T14:03:20.116173+0000","last_fullsized":"2024-09-21T14:03:20.116173+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:37.548064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:37.548064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.c","version":"390'2316","reported_seq":2343,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:21.114934+0000","last_change":"2024-09-21T14:02:56.563139+0000","last_active":"2024-09-21T14:03:21.114934+0000","last_peered":"2024-09-21T14:03:21.114934+0000","last_clean":"2024-09-21T14:03:21.114934+0000","last_became_active":"2024-09-21T14:02:56.562806+0000","last_became_peered":"2024-09-21T14:02:56.562806+0000","last_unstale":"2024-09-21T14:03:21.114934+0000","last_undegraded":"2024-09-21T14:03:21.114934+0000","last_fullsized":"2024-09-21T14:03:21.114934+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":2316,"log_dups_size":0,"ondisk_log_size":2316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:18.206635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151781376,"num_objects":2316,"num_object_clones":0,"num_object_copies":6948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2316,"num_write_kb":148224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"75'79","reported_seq":740,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.110997+0000","last_change":"2024-09-21T14:02:05.693250+0000","last_active":"2024-09-21T14:03:20.110997+0000","last_peered":"2024-09-21T14:03:20.110997+0000","last_clean":"2024-09-21T14:03:20.110997+0000","last_became_active":"2024-09-21T14:00:41.083425+0000","last_became_peered":"2024-09-21T14:00:41.083425+0000","last_unstale":"2024-09-21T14:03:20.110997+0000","last_undegraded":"2024-09-21T14:03:20.110997+0000","last_fullsized":"2024-09-21T14:03:20.110997+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:02:05.693193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:02:05.693193+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:28.620866+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"290'20","reported_seq":732,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.614934+0000","last_change":"2024-09-21T14:01:41.187743+0000","last_active":"2024-09-21T14:03:08.614934+0000","last_peered":"2024-09-21T14:03:08.614934+0000","last_clean":"2024-09-21T14:03:08.614934+0000","last_became_active":"2024-09-21T14:01:41.187060+0000","last_became_peered":"2024-09-21T14:01:41.187060+0000","last_unstale":"2024-09-21T14:03:08.614934+0000","last_undegraded":"2024-09-21T14:03:08.614934+0000","last_fullsized":"2024-09-21T14:03:08.614934+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:54:19.593611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:19.593611+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:47.290628+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"61'1","reported_seq":631,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.114317+0000","last_change":"2024-09-21T14:01:41.195580+0000","last_active":"2024-09-21T14:03:20.114317+0000","last_peered":"2024-09-21T14:03:20.114317+0000","last_clean":"2024-09-21T14:03:20.114317+0000","last_became_active":"2024-09-21T14:01:41.195268+0000","last_became_peered":"2024-09-21T14:01:41.195268+0000","last_unstale":"2024-09-21T14:03:20.114317+0000","last_undegraded":"2024-09-21T14:03:20.114317+0000","last_fullsized":"2024-09-21T14:03:20.114317+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:58:48.407734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:48.407734+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"346'15","reported_seq":334,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.111674+0000","last_change":"2024-09-21T14:01:41.209662+0000","last_active":"2024-09-21T14:03:20.111674+0000","last_peered":"2024-09-21T14:03:20.111674+0000","last_clean":"2024-09-21T14:03:20.111674+0000","last_became_active":"2024-09-21T14:01:41.209296+0000","last_became_peered":"2024-09-21T14:01:41.209296+0000","last_unstale":"2024-09-21T14:03:20.111674+0000","last_undegraded":"2024-09-21T14:03:20.111674+0000","last_fullsized":"2024-09-21T14:03:20.111674+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'12","last_scrub_stamp":"2024-09-21T13:57:30.133392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:30.133392+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:43.651875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":513,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.108675+0000","last_change":"2024-09-21T13:58:58.730439+0000","last_active":"2024-09-21T14:03:20.108675+0000","last_peered":"2024-09-21T14:03:20.108675+0000","last_clean":"2024-09-21T14:03:20.108675+0000","last_became_active":"2024-09-21T13:57:12.704897+0000","last_became_peered":"2024-09-21T13:57:12.704897+0000","last_unstale":"2024-09-21T14:03:20.108675+0000","last_undegraded":"2024-09-21T14:03:20.108675+0000","last_fullsized":"2024-09-21T14:03:20.108675+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:58.730368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:58.730368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.d","version":"390'2633","reported_seq":2662,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:23.556227+0000","last_change":"2024-09-21T14:02:56.560958+0000","last_active":"2024-09-21T14:03:23.556227+0000","last_peered":"2024-09-21T14:03:23.556227+0000","last_clean":"2024-09-21T14:03:23.556227+0000","last_became_active":"2024-09-21T14:02:56.560664+0000","last_became_peered":"2024-09-21T14:02:56.560664+0000","last_unstale":"2024-09-21T14:03:23.556227+0000","last_undegraded":"2024-09-21T14:03:23.556227+0000","last_fullsized":"2024-09-21T14:03:23.556227+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":2633,"log_dups_size":0,"ondisk_log_size":2633,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:06.081463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172556288,"num_objects":2633,"num_object_clones":0,"num_object_copies":7899,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2633,"num_write_kb":168512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"75'79","reported_seq":740,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.110408+0000","last_change":"2024-09-21T14:02:03.737005+0000","last_active":"2024-09-21T14:03:20.110408+0000","last_peered":"2024-09-21T14:03:20.110408+0000","last_clean":"2024-09-21T14:03:20.110408+0000","last_became_active":"2024-09-21T14:00:41.073047+0000","last_became_peered":"2024-09-21T14:00:41.073047+0000","last_unstale":"2024-09-21T14:03:20.110408+0000","last_undegraded":"2024-09-21T14:03:20.110408+0000","last_fullsized":"2024-09-21T14:03:20.110408+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:02:03.736928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:02:03.736928+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:21.563427+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"290'29","reported_seq":645,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.114637+0000","last_change":"2024-09-21T14:03:08.399687+0000","last_active":"2024-09-21T14:03:20.114637+0000","last_peered":"2024-09-21T14:03:20.114637+0000","last_clean":"2024-09-21T14:03:20.114637+0000","last_became_active":"2024-09-21T14:01:41.186960+0000","last_became_peered":"2024-09-21T14:01:41.186960+0000","last_unstale":"2024-09-21T14:03:20.114637+0000","last_undegraded":"2024-09-21T14:03:20.114637+0000","last_fullsized":"2024-09-21T14:03:20.114637+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'29","last_scrub_stamp":"2024-09-21T14:03:08.399606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:03:08.399606+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:19.025883+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":709,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.616225+0000","last_change":"2024-09-21T14:01:41.185706+0000","last_active":"2024-09-21T14:03:08.616225+0000","last_peered":"2024-09-21T14:03:08.616225+0000","last_clean":"2024-09-21T14:03:08.616225+0000","last_became_active":"2024-09-21T14:01:41.185400+0000","last_became_peered":"2024-09-21T14:01:41.185400+0000","last_unstale":"2024-09-21T14:03:08.616225+0000","last_undegraded":"2024-09-21T14:03:08.616225+0000","last_fullsized":"2024-09-21T14:03:08.616225+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:16.712990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:16.712990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:38.309423+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":289,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.110985+0000","last_change":"2024-09-21T14:02:59.469906+0000","last_active":"2024-09-21T14:03:20.110985+0000","last_peered":"2024-09-21T14:03:20.110985+0000","last_clean":"2024-09-21T14:03:20.110985+0000","last_became_active":"2024-09-21T14:01:41.193973+0000","last_became_peered":"2024-09-21T14:01:41.193973+0000","last_unstale":"2024-09-21T14:03:20.110985+0000","last_undegraded":"2024-09-21T14:03:20.110985+0000","last_fullsized":"2024-09-21T14:03:20.110985+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:59.469835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:02:59.469835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:00.302431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":700,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.614634+0000","last_change":"2024-09-21T13:58:43.952014+0000","last_active":"2024-09-21T14:03:08.614634+0000","last_peered":"2024-09-21T14:03:08.614634+0000","last_clean":"2024-09-21T14:03:08.614634+0000","last_became_active":"2024-09-21T13:58:17.757628+0000","last_became_peered":"2024-09-21T13:58:17.757628+0000","last_unstale":"2024-09-21T14:03:08.614634+0000","last_undegraded":"2024-09-21T14:03:08.614634+0000","last_fullsized":"2024-09-21T14:03:08.614634+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:43.951941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:43.951941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:52.357919+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"390'2633","reported_seq":2662,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:23.536905+0000","last_change":"2024-09-21T14:02:56.559655+0000","last_active":"2024-09-21T14:03:23.536905+0000","last_peered":"2024-09-21T14:03:23.536905+0000","last_clean":"2024-09-21T14:03:23.536905+0000","last_became_active":"2024-09-21T14:02:56.559346+0000","last_became_peered":"2024-09-21T14:02:56.559346+0000","last_unstale":"2024-09-21T14:03:23.536905+0000","last_undegraded":"2024-09-21T14:03:23.536905+0000","last_fullsized":"2024-09-21T14:03:23.536905+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":2633,"log_dups_size":0,"ondisk_log_size":2633,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:10.203798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172556288,"num_objects":2633,"num_object_clones":0,"num_object_copies":7899,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2633,"num_write_kb":168512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"75'79","reported_seq":113,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.114449+0000","last_change":"2024-09-21T14:01:41.199377+0000","last_active":"2024-09-21T14:03:20.114449+0000","last_peered":"2024-09-21T14:03:20.114449+0000","last_clean":"2024-09-21T14:03:20.114449+0000","last_became_active":"2024-09-21T14:01:41.198750+0000","last_became_peered":"2024-09-21T14:01:41.198750+0000","last_unstale":"2024-09-21T14:03:20.114449+0000","last_undegraded":"2024-09-21T14:03:20.114449+0000","last_fullsized":"2024-09-21T14:03:20.114449+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:03:47.451019+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"290'44","reported_seq":956,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:13.296180+0000","last_change":"2024-09-21T14:02:57.316829+0000","last_active":"2024-09-21T14:03:13.296180+0000","last_peered":"2024-09-21T14:03:13.296180+0000","last_clean":"2024-09-21T14:03:13.296180+0000","last_became_active":"2024-09-21T13:57:12.684690+0000","last_became_peered":"2024-09-21T13:57:12.684690+0000","last_unstale":"2024-09-21T14:03:13.296180+0000","last_undegraded":"2024-09-21T14:03:13.296180+0000","last_fullsized":"2024-09-21T14:03:13.296180+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'44","last_scrub_stamp":"2024-09-21T14:02:57.316763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:57.316763+0000","objects_scrubbed":10,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:57.769620+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":720,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.113146+0000","last_change":"2024-09-21T14:01:58.524159+0000","last_active":"2024-09-21T14:03:20.113146+0000","last_peered":"2024-09-21T14:03:20.113146+0000","last_clean":"2024-09-21T14:03:20.113146+0000","last_became_active":"2024-09-21T13:58:17.770937+0000","last_became_peered":"2024-09-21T13:58:17.770937+0000","last_unstale":"2024-09-21T14:03:20.113146+0000","last_undegraded":"2024-09-21T14:03:20.113146+0000","last_fullsized":"2024-09-21T14:03:20.113146+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:58.524114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:01:58.524114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:08.916792+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":650,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.113892+0000","last_change":"2024-09-21T14:01:41.249061+0000","last_active":"2024-09-21T14:03:20.113892+0000","last_peered":"2024-09-21T14:03:20.113892+0000","last_clean":"2024-09-21T14:03:20.113892+0000","last_became_active":"2024-09-21T14:01:41.247642+0000","last_became_peered":"2024-09-21T14:01:41.247642+0000","last_unstale":"2024-09-21T14:03:20.113892+0000","last_undegraded":"2024-09-21T14:03:20.113892+0000","last_fullsized":"2024-09-21T14:03:20.113892+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-21T13:59:22.186393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:59:22.186393+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":628,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.616412+0000","last_change":"2024-09-21T13:55:07.021128+0000","last_active":"2024-09-21T14:03:08.616412+0000","last_peered":"2024-09-21T14:03:08.616412+0000","last_clean":"2024-09-21T14:03:08.616412+0000","last_became_active":"2024-09-21T13:50:29.688249+0000","last_became_peered":"2024-09-21T13:50:29.688249+0000","last_unstale":"2024-09-21T14:03:08.616412+0000","last_undegraded":"2024-09-21T14:03:08.616412+0000","last_fullsized":"2024-09-21T14:03:08.616412+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:07.021059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:55:07.021059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:50.302160+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":629,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.118055+0000","last_change":"2024-09-21T14:03:02.238409+0000","last_active":"2024-09-21T14:03:20.118055+0000","last_peered":"2024-09-21T14:03:20.118055+0000","last_clean":"2024-09-21T14:03:20.118055+0000","last_became_active":"2024-09-21T13:59:32.208980+0000","last_became_peered":"2024-09-21T13:59:32.208980+0000","last_unstale":"2024-09-21T14:03:20.118055+0000","last_undegraded":"2024-09-21T14:03:20.118055+0000","last_fullsized":"2024-09-21T14:03:20.118055+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:02.238357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:02.238357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:09.206892+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"390'2323","reported_seq":2351,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.881924+0000","last_change":"2024-09-21T14:02:56.559275+0000","last_active":"2024-09-21T14:03:20.881924+0000","last_peered":"2024-09-21T14:03:20.881924+0000","last_clean":"2024-09-21T14:03:20.881924+0000","last_became_active":"2024-09-21T14:02:56.558773+0000","last_became_peered":"2024-09-21T14:02:56.558773+0000","last_unstale":"2024-09-21T14:03:20.881924+0000","last_undegraded":"2024-09-21T14:03:20.881924+0000","last_fullsized":"2024-09-21T14:03:20.881924+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":2323,"log_dups_size":0,"ondisk_log_size":2323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:02.709376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152240128,"num_objects":2323,"num_object_clones":0,"num_object_copies":6969,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2323,"num_write_kb":148672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"75'79","reported_seq":746,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.115645+0000","last_change":"2024-09-21T14:00:41.077286+0000","last_active":"2024-09-21T14:03:20.115645+0000","last_peered":"2024-09-21T14:03:20.115645+0000","last_clean":"2024-09-21T14:03:20.115645+0000","last_became_active":"2024-09-21T14:00:41.076948+0000","last_became_peered":"2024-09-21T14:00:41.076948+0000","last_unstale":"2024-09-21T14:03:20.115645+0000","last_undegraded":"2024-09-21T14:03:20.115645+0000","last_fullsized":"2024-09-21T14:03:20.115645+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:03:51.730082+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"290'17","reported_seq":735,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.617594+0000","last_change":"2024-09-21T14:01:41.200809+0000","last_active":"2024-09-21T14:03:08.617594+0000","last_peered":"2024-09-21T14:03:08.617594+0000","last_clean":"2024-09-21T14:03:08.617594+0000","last_became_active":"2024-09-21T14:01:41.200365+0000","last_became_peered":"2024-09-21T14:01:41.200365+0000","last_unstale":"2024-09-21T14:03:08.617594+0000","last_undegraded":"2024-09-21T14:03:08.617594+0000","last_fullsized":"2024-09-21T14:03:08.617594+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'13","last_scrub_stamp":"2024-09-21T13:58:11.619848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:11.619848+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:39.361066+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":745,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.615019+0000","last_change":"2024-09-21T14:03:01.310647+0000","last_active":"2024-09-21T14:03:08.615019+0000","last_peered":"2024-09-21T14:03:08.615019+0000","last_clean":"2024-09-21T14:03:08.615019+0000","last_became_active":"2024-09-21T13:58:17.759671+0000","last_became_peered":"2024-09-21T13:58:17.759671+0000","last_unstale":"2024-09-21T14:03:08.615019+0000","last_undegraded":"2024-09-21T14:03:08.615019+0000","last_fullsized":"2024-09-21T14:03:08.615019+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:01.310603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:03:01.310603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:10.066440+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":630,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.114865+0000","last_change":"2024-09-21T13:58:41.644876+0000","last_active":"2024-09-21T14:03:20.114865+0000","last_peered":"2024-09-21T14:03:20.114865+0000","last_clean":"2024-09-21T14:03:20.114865+0000","last_became_active":"2024-09-21T13:57:12.682147+0000","last_became_peered":"2024-09-21T13:57:12.682147+0000","last_unstale":"2024-09-21T14:03:20.114865+0000","last_undegraded":"2024-09-21T14:03:20.114865+0000","last_fullsized":"2024-09-21T14:03:20.114865+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:58:41.644789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:58:41.644789+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:15.691519+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":609,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.115200+0000","last_change":"2024-09-21T14:01:50.880840+0000","last_active":"2024-09-21T14:03:20.115200+0000","last_peered":"2024-09-21T14:03:20.115200+0000","last_clean":"2024-09-21T14:03:20.115200+0000","last_became_active":"2024-09-21T14:01:41.182000+0000","last_became_peered":"2024-09-21T14:01:41.182000+0000","last_unstale":"2024-09-21T14:03:20.115200+0000","last_undegraded":"2024-09-21T14:03:20.115200+0000","last_fullsized":"2024-09-21T14:03:20.115200+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:50.880471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:01:50.880471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:17.729591+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":666,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.116417+0000","last_change":"2024-09-21T13:59:32.222791+0000","last_active":"2024-09-21T14:03:20.116417+0000","last_peered":"2024-09-21T14:03:20.116417+0000","last_clean":"2024-09-21T14:03:20.116417+0000","last_became_active":"2024-09-21T13:59:32.221502+0000","last_became_peered":"2024-09-21T13:59:32.221502+0000","last_unstale":"2024-09-21T14:03:20.116417+0000","last_undegraded":"2024-09-21T14:03:20.116417+0000","last_fullsized":"2024-09-21T14:03:20.116417+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:14.673582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:14.673582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"390'2546","reported_seq":2573,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:22.761563+0000","last_change":"2024-09-21T14:02:56.559108+0000","last_active":"2024-09-21T14:03:22.761563+0000","last_peered":"2024-09-21T14:03:22.761563+0000","last_clean":"2024-09-21T14:03:22.761563+0000","last_became_active":"2024-09-21T14:02:56.558826+0000","last_became_peered":"2024-09-21T14:02:56.558826+0000","last_unstale":"2024-09-21T14:03:22.761563+0000","last_undegraded":"2024-09-21T14:03:22.761563+0000","last_fullsized":"2024-09-21T14:03:22.761563+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":2546,"log_dups_size":0,"ondisk_log_size":2546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:03.816041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166854656,"num_objects":2546,"num_object_clones":0,"num_object_copies":7638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2546,"num_write_kb":162944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"75'79","reported_seq":147,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.616090+0000","last_change":"2024-09-21T14:00:41.071631+0000","last_active":"2024-09-21T14:03:08.616090+0000","last_peered":"2024-09-21T14:03:08.616090+0000","last_clean":"2024-09-21T14:03:08.616090+0000","last_became_active":"2024-09-21T14:00:41.071042+0000","last_became_peered":"2024-09-21T14:00:41.071042+0000","last_unstale":"2024-09-21T14:03:08.616090+0000","last_undegraded":"2024-09-21T14:03:08.616090+0000","last_fullsized":"2024-09-21T14:03:08.616090+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:03:42.335979+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"290'24","reported_seq":794,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.627743+0000","last_change":"2024-09-21T14:02:57.560019+0000","last_active":"2024-09-21T14:03:08.627743+0000","last_peered":"2024-09-21T14:03:08.627743+0000","last_clean":"2024-09-21T14:03:08.627743+0000","last_became_active":"2024-09-21T14:01:41.184728+0000","last_became_peered":"2024-09-21T14:01:41.184728+0000","last_unstale":"2024-09-21T14:03:08.627743+0000","last_undegraded":"2024-09-21T14:03:08.627743+0000","last_fullsized":"2024-09-21T14:03:08.627743+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'24","last_scrub_stamp":"2024-09-21T14:02:57.559936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:57.559936+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:26.628231+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":642,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.117239+0000","last_change":"2024-09-21T13:59:59.450613+0000","last_active":"2024-09-21T14:03:20.117239+0000","last_peered":"2024-09-21T14:03:20.117239+0000","last_clean":"2024-09-21T14:03:20.117239+0000","last_became_active":"2024-09-21T13:57:12.681030+0000","last_became_peered":"2024-09-21T13:57:12.681030+0000","last_unstale":"2024-09-21T14:03:20.117239+0000","last_undegraded":"2024-09-21T14:03:20.117239+0000","last_fullsized":"2024-09-21T14:03:20.117239+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:59.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:59:59.450546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:40.683320+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":1429,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.116373+0000","last_change":"2024-09-21T13:59:32.218796+0000","last_active":"2024-09-21T14:03:20.116373+0000","last_peered":"2024-09-21T14:03:20.116373+0000","last_clean":"2024-09-21T14:03:20.116373+0000","last_became_active":"2024-09-21T13:59:32.203361+0000","last_became_peered":"2024-09-21T13:59:32.203361+0000","last_unstale":"2024-09-21T14:03:20.116373+0000","last_undegraded":"2024-09-21T14:03:20.116373+0000","last_fullsized":"2024-09-21T14:03:20.116373+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-21T13:56:29.617143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:56:29.617143+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047048000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":235,"num_read_kb":230,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":629,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.115556+0000","last_change":"2024-09-21T14:03:06.470855+0000","last_active":"2024-09-21T14:03:20.115556+0000","last_peered":"2024-09-21T14:03:20.115556+0000","last_clean":"2024-09-21T14:03:20.115556+0000","last_became_active":"2024-09-21T14:01:41.200531+0000","last_became_peered":"2024-09-21T14:01:41.200531+0000","last_unstale":"2024-09-21T14:03:20.115556+0000","last_undegraded":"2024-09-21T14:03:20.115556+0000","last_fullsized":"2024-09-21T14:03:20.115556+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:06.470705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:06.470705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:32.367248+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":729,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.615199+0000","last_change":"2024-09-21T14:03:04.357026+0000","last_active":"2024-09-21T14:03:08.615199+0000","last_peered":"2024-09-21T14:03:08.615199+0000","last_clean":"2024-09-21T14:03:08.615199+0000","last_became_active":"2024-09-21T14:01:41.184664+0000","last_became_peered":"2024-09-21T14:01:41.184664+0000","last_unstale":"2024-09-21T14:03:08.615199+0000","last_undegraded":"2024-09-21T14:03:08.615199+0000","last_fullsized":"2024-09-21T14:03:08.615199+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:04.356979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:04.356979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:19.365633+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"390'2541","reported_seq":2570,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:23.563510+0000","last_change":"2024-09-21T14:02:56.557541+0000","last_active":"2024-09-21T14:03:23.563510+0000","last_peered":"2024-09-21T14:03:23.563510+0000","last_clean":"2024-09-21T14:03:23.563510+0000","last_became_active":"2024-09-21T14:02:56.557263+0000","last_became_peered":"2024-09-21T14:02:56.557263+0000","last_unstale":"2024-09-21T14:03:23.563510+0000","last_undegraded":"2024-09-21T14:03:23.563510+0000","last_fullsized":"2024-09-21T14:03:23.563510+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":2541,"log_dups_size":0,"ondisk_log_size":2541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:05.629424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166526976,"num_objects":2541,"num_object_clones":0,"num_object_copies":7623,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2541,"num_write_kb":162624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"300'32","reported_seq":387,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.111074+0000","last_change":"2024-09-21T14:00:47.130089+0000","last_active":"2024-09-21T14:03:20.111074+0000","last_peered":"2024-09-21T14:03:20.111074+0000","last_clean":"2024-09-21T14:03:20.111074+0000","last_became_active":"2024-09-21T13:59:32.204624+0000","last_became_peered":"2024-09-21T13:59:32.204624+0000","last_unstale":"2024-09-21T14:03:20.111074+0000","last_undegraded":"2024-09-21T14:03:20.111074+0000","last_fullsized":"2024-09-21T14:03:20.111074+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'32","last_scrub_stamp":"2024-09-21T14:00:47.129789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:00:47.129789+0000","objects_scrubbed":8,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:16.636929+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":679,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.111506+0000","last_change":"2024-09-21T14:01:41.190537+0000","last_active":"2024-09-21T14:03:20.111506+0000","last_peered":"2024-09-21T14:03:20.111506+0000","last_clean":"2024-09-21T14:03:20.111506+0000","last_became_active":"2024-09-21T14:01:41.190210+0000","last_became_peered":"2024-09-21T14:01:41.190210+0000","last_unstale":"2024-09-21T14:03:20.111506+0000","last_undegraded":"2024-09-21T14:03:20.111506+0000","last_fullsized":"2024-09-21T14:03:20.111506+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:57.772975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:57.772975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"75'79","reported_seq":740,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.116525+0000","last_change":"2024-09-21T14:00:39.339360+0000","last_active":"2024-09-21T14:03:20.116525+0000","last_peered":"2024-09-21T14:03:20.116525+0000","last_clean":"2024-09-21T14:03:20.116525+0000","last_became_active":"2024-09-21T14:00:39.339077+0000","last_became_peered":"2024-09-21T14:00:39.339077+0000","last_unstale":"2024-09-21T14:03:20.116525+0000","last_undegraded":"2024-09-21T14:03:20.116525+0000","last_fullsized":"2024-09-21T14:03:20.116525+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:03:49.763645+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":19,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53669,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":654,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.116006+0000","last_change":"2024-09-21T14:01:41.182186+0000","last_active":"2024-09-21T14:03:20.116006+0000","last_peered":"2024-09-21T14:03:20.116006+0000","last_clean":"2024-09-21T14:03:20.116006+0000","last_became_active":"2024-09-21T14:01:41.181899+0000","last_became_peered":"2024-09-21T14:01:41.181899+0000","last_unstale":"2024-09-21T14:03:20.116006+0000","last_undegraded":"2024-09-21T14:03:20.116006+0000","last_fullsized":"2024-09-21T14:03:20.116006+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:15.700409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:15.700409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":579,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.115292+0000","last_change":"2024-09-21T14:01:41.196509+0000","last_active":"2024-09-21T14:03:20.115292+0000","last_peered":"2024-09-21T14:03:20.115292+0000","last_clean":"2024-09-21T14:03:20.115292+0000","last_became_active":"2024-09-21T14:01:41.196243+0000","last_became_peered":"2024-09-21T14:01:41.196243+0000","last_unstale":"2024-09-21T14:03:20.115292+0000","last_undegraded":"2024-09-21T14:03:20.115292+0000","last_fullsized":"2024-09-21T14:03:20.115292+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:21.153018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:21.153018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:17.291763+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"390'2670","reported_seq":2699,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:23.546296+0000","last_change":"2024-09-21T14:02:56.561047+0000","last_active":"2024-09-21T14:03:23.546296+0000","last_peered":"2024-09-21T14:03:23.546296+0000","last_clean":"2024-09-21T14:03:23.546296+0000","last_became_active":"2024-09-21T14:02:56.558774+0000","last_became_peered":"2024-09-21T14:02:56.558774+0000","last_unstale":"2024-09-21T14:03:23.546296+0000","last_undegraded":"2024-09-21T14:03:23.546296+0000","last_fullsized":"2024-09-21T14:03:23.546296+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":2670,"log_dups_size":0,"ondisk_log_size":2670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:07.590313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174981120,"num_objects":2670,"num_object_clones":0,"num_object_copies":8010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2670,"num_write_kb":170880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"290'19","reported_seq":766,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.113697+0000","last_change":"2024-09-21T14:02:28.553549+0000","last_active":"2024-09-21T14:03:20.113697+0000","last_peered":"2024-09-21T14:03:20.113697+0000","last_clean":"2024-09-21T14:03:20.113697+0000","last_became_active":"2024-09-21T13:58:17.776310+0000","last_became_peered":"2024-09-21T13:58:17.776310+0000","last_unstale":"2024-09-21T14:03:20.113697+0000","last_undegraded":"2024-09-21T14:03:20.113697+0000","last_fullsized":"2024-09-21T14:03:20.113697+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'19","last_scrub_stamp":"2024-09-21T14:02:28.553065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:28.553065+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:33.636087+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":728,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.117338+0000","last_change":"2024-09-21T14:02:03.437581+0000","last_active":"2024-09-21T14:03:20.117338+0000","last_peered":"2024-09-21T14:03:20.117338+0000","last_clean":"2024-09-21T14:03:20.117338+0000","last_became_active":"2024-09-21T13:59:32.223247+0000","last_became_peered":"2024-09-21T13:59:32.223247+0000","last_unstale":"2024-09-21T14:03:20.117338+0000","last_undegraded":"2024-09-21T14:03:20.117338+0000","last_fullsized":"2024-09-21T14:03:20.117338+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:03.437510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:02:03.437510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":655,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.115211+0000","last_change":"2024-09-21T14:02:58.461716+0000","last_active":"2024-09-21T14:03:20.115211+0000","last_peered":"2024-09-21T14:03:20.115211+0000","last_clean":"2024-09-21T14:03:20.115211+0000","last_became_active":"2024-09-21T14:01:41.193941+0000","last_became_peered":"2024-09-21T14:01:41.193941+0000","last_unstale":"2024-09-21T14:03:20.115211+0000","last_undegraded":"2024-09-21T14:03:20.115211+0000","last_fullsized":"2024-09-21T14:03:20.115211+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:58.461663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:02:58.461663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:20.802609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":657,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.110000+0000","last_change":"2024-09-21T14:01:41.190501+0000","last_active":"2024-09-21T14:03:20.110000+0000","last_peered":"2024-09-21T14:03:20.110000+0000","last_clean":"2024-09-21T14:03:20.110000+0000","last_became_active":"2024-09-21T14:01:41.190192+0000","last_became_peered":"2024-09-21T14:01:41.190192+0000","last_unstale":"2024-09-21T14:03:20.110000+0000","last_undegraded":"2024-09-21T14:03:20.110000+0000","last_fullsized":"2024-09-21T14:03:20.110000+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:58.952218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:54:58.952218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"390'2493","reported_seq":2522,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:22.735677+0000","last_change":"2024-09-21T14:02:56.560359+0000","last_active":"2024-09-21T14:03:22.735677+0000","last_peered":"2024-09-21T14:03:22.735677+0000","last_clean":"2024-09-21T14:03:22.735677+0000","last_became_active":"2024-09-21T14:02:56.560070+0000","last_became_peered":"2024-09-21T14:02:56.560070+0000","last_unstale":"2024-09-21T14:03:22.735677+0000","last_undegraded":"2024-09-21T14:03:22.735677+0000","last_fullsized":"2024-09-21T14:03:22.735677+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":2493,"log_dups_size":0,"ondisk_log_size":2493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:16.453643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163381248,"num_objects":2493,"num_object_clones":0,"num_object_copies":7479,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2493,"num_write_kb":159552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"75'79","reported_seq":120,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.113339+0000","last_change":"2024-09-21T14:01:41.189146+0000","last_active":"2024-09-21T14:03:20.113339+0000","last_peered":"2024-09-21T14:03:20.113339+0000","last_clean":"2024-09-21T14:03:20.113339+0000","last_became_active":"2024-09-21T14:01:41.188685+0000","last_became_peered":"2024-09-21T14:01:41.188685+0000","last_unstale":"2024-09-21T14:03:20.113339+0000","last_undegraded":"2024-09-21T14:03:20.113339+0000","last_fullsized":"2024-09-21T14:03:20.113339+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:03:46.437011+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"290'31","reported_seq":754,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.116977+0000","last_change":"2024-09-21T14:01:03.541547+0000","last_active":"2024-09-21T14:03:20.116977+0000","last_peered":"2024-09-21T14:03:20.116977+0000","last_clean":"2024-09-21T14:03:20.116977+0000","last_became_active":"2024-09-21T13:52:04.313218+0000","last_became_peered":"2024-09-21T13:52:04.313218+0000","last_unstale":"2024-09-21T14:03:20.116977+0000","last_undegraded":"2024-09-21T14:03:20.116977+0000","last_fullsized":"2024-09-21T14:03:20.116977+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'31","last_scrub_stamp":"2024-09-21T14:01:03.541432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:03.541432+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:50.707905+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":270,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.110748+0000","last_change":"2024-09-21T14:01:29.924408+0000","last_active":"2024-09-21T14:03:20.110748+0000","last_peered":"2024-09-21T14:03:20.110748+0000","last_clean":"2024-09-21T14:03:20.110748+0000","last_became_active":"2024-09-21T13:58:17.762520+0000","last_became_peered":"2024-09-21T13:58:17.762520+0000","last_unstale":"2024-09-21T14:03:20.110748+0000","last_undegraded":"2024-09-21T14:03:20.110748+0000","last_fullsized":"2024-09-21T14:03:20.110748+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:29.924331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:01:29.924331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:47.626829+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":679,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.615303+0000","last_change":"2024-09-21T13:58:52.030800+0000","last_active":"2024-09-21T14:03:08.615303+0000","last_peered":"2024-09-21T14:03:08.615303+0000","last_clean":"2024-09-21T14:03:08.615303+0000","last_became_active":"2024-09-21T13:49:02.067472+0000","last_became_peered":"2024-09-21T13:49:02.067472+0000","last_unstale":"2024-09-21T14:03:08.615303+0000","last_undegraded":"2024-09-21T14:03:08.615303+0000","last_fullsized":"2024-09-21T14:03:08.615303+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:52.030718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:52.030718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:43.346814+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":566,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.116037+0000","last_change":"2024-09-21T13:57:12.683321+0000","last_active":"2024-09-21T14:03:20.116037+0000","last_peered":"2024-09-21T14:03:20.116037+0000","last_clean":"2024-09-21T14:03:20.116037+0000","last_became_active":"2024-09-21T13:57:12.682979+0000","last_became_peered":"2024-09-21T13:57:12.682979+0000","last_unstale":"2024-09-21T14:03:20.116037+0000","last_undegraded":"2024-09-21T14:03:20.116037+0000","last_fullsized":"2024-09-21T14:03:20.116037+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:18.027944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:18.027944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:21.753981+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2","version":"388'2265","reported_seq":2290,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:19.705403+0000","last_change":"2024-09-21T14:02:56.559657+0000","last_active":"2024-09-21T14:03:19.705403+0000","last_peered":"2024-09-21T14:03:19.705403+0000","last_clean":"2024-09-21T14:03:19.705403+0000","last_became_active":"2024-09-21T14:02:56.559368+0000","last_became_peered":"2024-09-21T14:02:56.559368+0000","last_unstale":"2024-09-21T14:03:19.705403+0000","last_undegraded":"2024-09-21T14:03:19.705403+0000","last_fullsized":"2024-09-21T14:03:19.705403+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":2265,"log_dups_size":0,"ondisk_log_size":2265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:23.196521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148439040,"num_objects":2265,"num_object_clones":0,"num_object_copies":6795,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2265,"num_write_kb":144960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"75'79","reported_seq":754,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.116325+0000","last_change":"2024-09-21T14:02:55.806160+0000","last_active":"2024-09-21T14:03:20.116325+0000","last_peered":"2024-09-21T14:03:20.116325+0000","last_clean":"2024-09-21T14:03:20.116325+0000","last_became_active":"2024-09-21T14:00:41.139315+0000","last_became_peered":"2024-09-21T14:00:41.139315+0000","last_unstale":"2024-09-21T14:03:20.116325+0000","last_undegraded":"2024-09-21T14:03:20.116325+0000","last_fullsized":"2024-09-21T14:03:20.116325+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:02:55.806077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.806077+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:59.364707+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"290'20","reported_seq":773,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.112821+0000","last_change":"2024-09-21T13:58:17.789407+0000","last_active":"2024-09-21T14:03:20.112821+0000","last_peered":"2024-09-21T14:03:20.112821+0000","last_clean":"2024-09-21T14:03:20.112821+0000","last_became_active":"2024-09-21T13:58:17.765427+0000","last_became_peered":"2024-09-21T13:58:17.765427+0000","last_unstale":"2024-09-21T14:03:20.112821+0000","last_undegraded":"2024-09-21T14:03:20.112821+0000","last_fullsized":"2024-09-21T14:03:20.112821+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:58:13.464196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:13.464196+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:18.848758+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":622,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.113564+0000","last_change":"2024-09-21T14:02:58.236695+0000","last_active":"2024-09-21T14:03:20.113564+0000","last_peered":"2024-09-21T14:03:20.113564+0000","last_clean":"2024-09-21T14:03:20.113564+0000","last_became_active":"2024-09-21T13:57:12.704203+0000","last_became_peered":"2024-09-21T13:57:12.704203+0000","last_unstale":"2024-09-21T14:03:20.113564+0000","last_undegraded":"2024-09-21T14:03:20.113564+0000","last_fullsized":"2024-09-21T14:03:20.113564+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:58.236645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:02:58.236645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:21.993648+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"346'14","reported_seq":752,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.115604+0000","last_change":"2024-09-21T14:01:41.186960+0000","last_active":"2024-09-21T14:03:20.115604+0000","last_peered":"2024-09-21T14:03:20.115604+0000","last_clean":"2024-09-21T14:03:20.115604+0000","last_became_active":"2024-09-21T14:01:41.186564+0000","last_became_peered":"2024-09-21T14:01:41.186564+0000","last_unstale":"2024-09-21T14:03:20.115604+0000","last_undegraded":"2024-09-21T14:03:20.115604+0000","last_fullsized":"2024-09-21T14:03:20.115604+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'12","last_scrub_stamp":"2024-09-21T13:59:38.738120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:38.738120+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:47.648985+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"68'1","reported_seq":701,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.113428+0000","last_change":"2024-09-21T13:59:31.392186+0000","last_active":"2024-09-21T14:03:20.113428+0000","last_peered":"2024-09-21T14:03:20.113428+0000","last_clean":"2024-09-21T14:03:20.113428+0000","last_became_active":"2024-09-21T13:52:04.313210+0000","last_became_peered":"2024-09-21T13:52:04.313210+0000","last_unstale":"2024-09-21T14:03:20.113428+0000","last_undegraded":"2024-09-21T14:03:20.113428+0000","last_fullsized":"2024-09-21T14:03:20.113428+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T13:59:31.392105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:31.392105+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:39.716003+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.5","version":"390'2576","reported_seq":2605,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:22.752156+0000","last_change":"2024-09-21T14:02:56.557127+0000","last_active":"2024-09-21T14:03:22.752156+0000","last_peered":"2024-09-21T14:03:22.752156+0000","last_clean":"2024-09-21T14:03:22.752156+0000","last_became_active":"2024-09-21T14:02:56.556786+0000","last_became_peered":"2024-09-21T14:02:56.556786+0000","last_unstale":"2024-09-21T14:03:22.752156+0000","last_undegraded":"2024-09-21T14:03:22.752156+0000","last_fullsized":"2024-09-21T14:03:22.752156+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":2576,"log_dups_size":0,"ondisk_log_size":2576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:16.085110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168820736,"num_objects":2576,"num_object_clones":0,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2576,"num_write_kb":164864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"290'15","reported_seq":724,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.627722+0000","last_change":"2024-09-21T13:58:50.498597+0000","last_active":"2024-09-21T14:03:08.627722+0000","last_peered":"2024-09-21T14:03:08.627722+0000","last_clean":"2024-09-21T14:03:08.627722+0000","last_became_active":"2024-09-21T13:49:00.053016+0000","last_became_peered":"2024-09-21T13:49:00.053016+0000","last_unstale":"2024-09-21T14:03:08.627722+0000","last_undegraded":"2024-09-21T14:03:08.627722+0000","last_fullsized":"2024-09-21T14:03:08.627722+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:58:50.498369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:50.498369+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:54.564652+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":659,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.114064+0000","last_change":"2024-09-21T14:01:41.192717+0000","last_active":"2024-09-21T14:03:20.114064+0000","last_peered":"2024-09-21T14:03:20.114064+0000","last_clean":"2024-09-21T14:03:20.114064+0000","last_became_active":"2024-09-21T14:01:41.192443+0000","last_became_peered":"2024-09-21T14:01:41.192443+0000","last_unstale":"2024-09-21T14:03:20.114064+0000","last_undegraded":"2024-09-21T14:03:20.114064+0000","last_fullsized":"2024-09-21T14:03:20.114064+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:53.894506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:53.894506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:53.450976+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":647,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.112474+0000","last_change":"2024-09-21T14:02:56.796917+0000","last_active":"2024-09-21T14:03:20.112474+0000","last_peered":"2024-09-21T14:03:20.112474+0000","last_clean":"2024-09-21T14:03:20.112474+0000","last_became_active":"2024-09-21T13:57:12.680978+0000","last_became_peered":"2024-09-21T13:57:12.680978+0000","last_unstale":"2024-09-21T14:03:20.112474+0000","last_undegraded":"2024-09-21T14:03:20.112474+0000","last_fullsized":"2024-09-21T14:03:20.112474+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:56.796861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:02:56.796861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:10.252837+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":589,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.116014+0000","last_change":"2024-09-21T13:59:32.224001+0000","last_active":"2024-09-21T14:03:20.116014+0000","last_peered":"2024-09-21T14:03:20.116014+0000","last_clean":"2024-09-21T14:03:20.116014+0000","last_became_active":"2024-09-21T13:59:32.221628+0000","last_became_peered":"2024-09-21T13:59:32.221628+0000","last_unstale":"2024-09-21T14:03:20.116014+0000","last_undegraded":"2024-09-21T14:03:20.116014+0000","last_fullsized":"2024-09-21T14:03:20.116014+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:13.698091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:13.698091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"390'2650","reported_seq":2679,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:23.580462+0000","last_change":"2024-09-21T14:02:56.562973+0000","last_active":"2024-09-21T14:03:23.580462+0000","last_peered":"2024-09-21T14:03:23.580462+0000","last_clean":"2024-09-21T14:03:23.580462+0000","last_became_active":"2024-09-21T14:02:56.562256+0000","last_became_peered":"2024-09-21T14:02:56.562256+0000","last_unstale":"2024-09-21T14:03:23.580462+0000","last_undegraded":"2024-09-21T14:03:23.580462+0000","last_fullsized":"2024-09-21T14:03:23.580462+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":2650,"log_dups_size":0,"ondisk_log_size":2650,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:09.952633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173670400,"num_objects":2650,"num_object_clones":0,"num_object_copies":7950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2650,"num_write_kb":169600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"290'35","reported_seq":336,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.110893+0000","last_change":"2024-09-21T14:01:02.006619+0000","last_active":"2024-09-21T14:03:20.110893+0000","last_peered":"2024-09-21T14:03:20.110893+0000","last_clean":"2024-09-21T14:03:20.110893+0000","last_became_active":"2024-09-21T13:59:32.204184+0000","last_became_peered":"2024-09-21T13:59:32.204184+0000","last_unstale":"2024-09-21T14:03:20.110893+0000","last_undegraded":"2024-09-21T14:03:20.110893+0000","last_fullsized":"2024-09-21T14:03:20.110893+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'35","last_scrub_stamp":"2024-09-21T14:01:02.006504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:02.006504+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:22.557193+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":681,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.117263+0000","last_change":"2024-09-21T14:01:41.184937+0000","last_active":"2024-09-21T14:03:20.117263+0000","last_peered":"2024-09-21T14:03:20.117263+0000","last_clean":"2024-09-21T14:03:20.117263+0000","last_became_active":"2024-09-21T14:01:41.184567+0000","last_became_peered":"2024-09-21T14:01:41.184567+0000","last_unstale":"2024-09-21T14:03:20.117263+0000","last_undegraded":"2024-09-21T14:03:20.117263+0000","last_fullsized":"2024-09-21T14:03:20.117263+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:21.587493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:21.587493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":660,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.115427+0000","last_change":"2024-09-21T14:01:41.182250+0000","last_active":"2024-09-21T14:03:20.115427+0000","last_peered":"2024-09-21T14:03:20.115427+0000","last_clean":"2024-09-21T14:03:20.115427+0000","last_became_active":"2024-09-21T14:01:41.181911+0000","last_became_peered":"2024-09-21T14:01:41.181911+0000","last_unstale":"2024-09-21T14:03:20.115427+0000","last_undegraded":"2024-09-21T14:03:20.115427+0000","last_fullsized":"2024-09-21T14:03:20.115427+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:55.588554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:55.588554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:45.648976+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":608,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.116424+0000","last_change":"2024-09-21T13:59:32.223123+0000","last_active":"2024-09-21T14:03:20.116424+0000","last_peered":"2024-09-21T14:03:20.116424+0000","last_clean":"2024-09-21T14:03:20.116424+0000","last_became_active":"2024-09-21T13:59:32.219332+0000","last_became_peered":"2024-09-21T13:59:32.219332+0000","last_unstale":"2024-09-21T14:03:20.116424+0000","last_undegraded":"2024-09-21T14:03:20.116424+0000","last_fullsized":"2024-09-21T14:03:20.116424+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:19.537177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:19.537177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"390'2604","reported_seq":2633,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:23.567448+0000","last_change":"2024-09-21T14:02:56.559626+0000","last_active":"2024-09-21T14:03:23.567448+0000","last_peered":"2024-09-21T14:03:23.567448+0000","last_clean":"2024-09-21T14:03:23.567448+0000","last_became_active":"2024-09-21T14:02:56.559331+0000","last_became_peered":"2024-09-21T14:02:56.559331+0000","last_unstale":"2024-09-21T14:03:23.567448+0000","last_undegraded":"2024-09-21T14:03:23.567448+0000","last_fullsized":"2024-09-21T14:03:23.567448+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":2604,"log_dups_size":0,"ondisk_log_size":2604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:20.185982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170655744,"num_objects":2604,"num_object_clones":0,"num_object_copies":7812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2604,"num_write_kb":166656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"290'18","reported_seq":745,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.117251+0000","last_change":"2024-09-21T13:59:28.340494+0000","last_active":"2024-09-21T14:03:20.117251+0000","last_peered":"2024-09-21T14:03:20.117251+0000","last_clean":"2024-09-21T14:03:20.117251+0000","last_became_active":"2024-09-21T13:57:12.680627+0000","last_became_peered":"2024-09-21T13:57:12.680627+0000","last_unstale":"2024-09-21T14:03:20.117251+0000","last_undegraded":"2024-09-21T14:03:20.117251+0000","last_fullsized":"2024-09-21T14:03:20.117251+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'18","last_scrub_stamp":"2024-09-21T13:59:28.340415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:28.340415+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:11.926616+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":636,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.116948+0000","last_change":"2024-09-21T13:59:32.201126+0000","last_active":"2024-09-21T14:03:20.116948+0000","last_peered":"2024-09-21T14:03:20.116948+0000","last_clean":"2024-09-21T14:03:20.116948+0000","last_became_active":"2024-09-21T13:59:32.200674+0000","last_became_peered":"2024-09-21T13:59:32.200674+0000","last_unstale":"2024-09-21T14:03:20.116948+0000","last_undegraded":"2024-09-21T14:03:20.116948+0000","last_fullsized":"2024-09-21T14:03:20.116948+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:21.582325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:21.582325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:12.941161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":622,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.114804+0000","last_change":"2024-09-21T14:02:27.791041+0000","last_active":"2024-09-21T14:03:20.114804+0000","last_peered":"2024-09-21T14:03:20.114804+0000","last_clean":"2024-09-21T14:03:20.114804+0000","last_became_active":"2024-09-21T13:52:04.322147+0000","last_became_peered":"2024-09-21T13:52:04.322147+0000","last_unstale":"2024-09-21T14:03:20.114804+0000","last_undegraded":"2024-09-21T14:03:20.114804+0000","last_fullsized":"2024-09-21T14:03:20.114804+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:27.790994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:02:27.790994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:51.879201+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":630,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.113897+0000","last_change":"2024-09-21T14:02:36.371955+0000","last_active":"2024-09-21T14:03:20.113897+0000","last_peered":"2024-09-21T14:03:20.113897+0000","last_clean":"2024-09-21T14:03:20.113897+0000","last_became_active":"2024-09-21T13:59:32.203767+0000","last_became_peered":"2024-09-21T13:59:32.203767+0000","last_unstale":"2024-09-21T14:03:20.113897+0000","last_undegraded":"2024-09-21T14:03:20.113897+0000","last_fullsized":"2024-09-21T14:03:20.113897+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:36.371747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:02:36.371747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:54.168810+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"390'2304","reported_seq":2333,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.879872+0000","last_change":"2024-09-21T14:02:56.564166+0000","last_active":"2024-09-21T14:03:20.879872+0000","last_peered":"2024-09-21T14:03:20.879872+0000","last_clean":"2024-09-21T14:03:20.879872+0000","last_became_active":"2024-09-21T14:02:56.563788+0000","last_became_peered":"2024-09-21T14:02:56.563788+0000","last_unstale":"2024-09-21T14:03:20.879872+0000","last_undegraded":"2024-09-21T14:03:20.879872+0000","last_fullsized":"2024-09-21T14:03:20.879872+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":2304,"log_dups_size":0,"ondisk_log_size":2304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:20.466245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":2304,"num_object_clones":0,"num_object_copies":6912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2304,"num_write_kb":147456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"290'27","reported_seq":735,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.113466+0000","last_change":"2024-09-21T14:01:41.193143+0000","last_active":"2024-09-21T14:03:20.113466+0000","last_peered":"2024-09-21T14:03:20.113466+0000","last_clean":"2024-09-21T14:03:20.113466+0000","last_became_active":"2024-09-21T14:01:41.192481+0000","last_became_peered":"2024-09-21T14:01:41.192481+0000","last_unstale":"2024-09-21T14:03:20.113466+0000","last_undegraded":"2024-09-21T14:03:20.113466+0000","last_fullsized":"2024-09-21T14:03:20.113466+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'27","last_scrub_stamp":"2024-09-21T13:59:33.058859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:33.058859+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":616,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.116757+0000","last_change":"2024-09-21T14:03:04.953724+0000","last_active":"2024-09-21T14:03:20.116757+0000","last_peered":"2024-09-21T14:03:20.116757+0000","last_clean":"2024-09-21T14:03:20.116757+0000","last_became_active":"2024-09-21T13:59:32.203172+0000","last_became_peered":"2024-09-21T13:59:32.203172+0000","last_unstale":"2024-09-21T14:03:20.116757+0000","last_undegraded":"2024-09-21T14:03:20.116757+0000","last_fullsized":"2024-09-21T14:03:20.116757+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T14:03:04.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:03:04.953658+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:14.479415+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"300'14","reported_seq":828,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.115860+0000","last_change":"2024-09-21T13:59:32.220898+0000","last_active":"2024-09-21T14:03:20.115860+0000","last_peered":"2024-09-21T14:03:20.115860+0000","last_clean":"2024-09-21T14:03:20.115860+0000","last_became_active":"2024-09-21T13:59:32.206956+0000","last_became_peered":"2024-09-21T13:59:32.206956+0000","last_unstale":"2024-09-21T14:03:20.115860+0000","last_undegraded":"2024-09-21T14:03:20.115860+0000","last_fullsized":"2024-09-21T14:03:20.115860+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'11","last_scrub_stamp":"2024-09-21T13:56:31.616003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:56:31.616003+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:11.986942+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":683,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.113686+0000","last_change":"2024-09-21T14:01:41.248408+0000","last_active":"2024-09-21T14:03:20.113686+0000","last_peered":"2024-09-21T14:03:20.113686+0000","last_clean":"2024-09-21T14:03:20.113686+0000","last_became_active":"2024-09-21T14:01:41.246106+0000","last_became_peered":"2024-09-21T14:01:41.246106+0000","last_unstale":"2024-09-21T14:03:20.113686+0000","last_undegraded":"2024-09-21T14:03:20.113686+0000","last_fullsized":"2024-09-21T14:03:20.113686+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:39.258659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:39.258659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":273,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.110370+0000","last_change":"2024-09-21T14:01:50.733055+0000","last_active":"2024-09-21T14:03:20.110370+0000","last_peered":"2024-09-21T14:03:20.110370+0000","last_clean":"2024-09-21T14:03:20.110370+0000","last_became_active":"2024-09-21T14:01:41.197501+0000","last_became_peered":"2024-09-21T14:01:41.197501+0000","last_unstale":"2024-09-21T14:03:20.110370+0000","last_undegraded":"2024-09-21T14:03:20.110370+0000","last_fullsized":"2024-09-21T14:03:20.110370+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:50.733007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:01:50.733007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:17.685126+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"290'12","reported_seq":657,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.120602+0000","last_change":"2024-09-21T13:59:32.247422+0000","last_active":"2024-09-21T14:03:20.120602+0000","last_peered":"2024-09-21T14:03:20.120602+0000","last_clean":"2024-09-21T14:03:20.120602+0000","last_became_active":"2024-09-21T13:59:32.211306+0000","last_became_peered":"2024-09-21T13:59:32.211306+0000","last_unstale":"2024-09-21T14:03:20.120602+0000","last_undegraded":"2024-09-21T14:03:20.120602+0000","last_fullsized":"2024-09-21T14:03:20.120602+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'6","last_scrub_stamp":"2024-09-21T13:55:12.066873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:55:12.066873+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:38.360712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":615,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.113888+0000","last_change":"2024-09-21T14:01:41.199742+0000","last_active":"2024-09-21T14:03:20.113888+0000","last_peered":"2024-09-21T14:03:20.113888+0000","last_clean":"2024-09-21T14:03:20.113888+0000","last_became_active":"2024-09-21T14:01:41.199321+0000","last_became_peered":"2024-09-21T14:01:41.199321+0000","last_unstale":"2024-09-21T14:03:20.113888+0000","last_undegraded":"2024-09-21T14:03:20.113888+0000","last_fullsized":"2024-09-21T14:03:20.113888+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:37.106238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:37.106238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:16.289450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":662,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.117108+0000","last_change":"2024-09-21T14:03:01.872986+0000","last_active":"2024-09-21T14:03:20.117108+0000","last_peered":"2024-09-21T14:03:20.117108+0000","last_clean":"2024-09-21T14:03:20.117108+0000","last_became_active":"2024-09-21T13:59:32.219525+0000","last_became_peered":"2024-09-21T13:59:32.219525+0000","last_unstale":"2024-09-21T14:03:20.117108+0000","last_undegraded":"2024-09-21T14:03:20.117108+0000","last_fullsized":"2024-09-21T14:03:20.117108+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:01.872928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:01.872928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:09.296117+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"290'15","reported_seq":620,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.117839+0000","last_change":"2024-09-21T13:59:32.238689+0000","last_active":"2024-09-21T14:03:20.117839+0000","last_peered":"2024-09-21T14:03:20.117839+0000","last_clean":"2024-09-21T14:03:20.117839+0000","last_became_active":"2024-09-21T13:59:32.209166+0000","last_became_peered":"2024-09-21T13:59:32.209166+0000","last_unstale":"2024-09-21T14:03:20.117839+0000","last_undegraded":"2024-09-21T14:03:20.117839+0000","last_fullsized":"2024-09-21T14:03:20.117839+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:55:04.990842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:55:04.990842+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:21.278052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"61'1","reported_seq":672,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.116237+0000","last_change":"2024-09-21T13:59:32.222826+0000","last_active":"2024-09-21T14:03:20.116237+0000","last_peered":"2024-09-21T14:03:20.116237+0000","last_clean":"2024-09-21T14:03:20.116237+0000","last_became_active":"2024-09-21T13:59:32.219515+0000","last_became_peered":"2024-09-21T13:59:32.219515+0000","last_unstale":"2024-09-21T14:03:20.116237+0000","last_undegraded":"2024-09-21T14:03:20.116237+0000","last_fullsized":"2024-09-21T14:03:20.116237+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:58:07.711028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:07.711028+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":605,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.120272+0000","last_change":"2024-09-21T13:59:32.211754+0000","last_active":"2024-09-21T14:03:20.120272+0000","last_peered":"2024-09-21T14:03:20.120272+0000","last_clean":"2024-09-21T14:03:20.120272+0000","last_became_active":"2024-09-21T13:59:32.210586+0000","last_became_peered":"2024-09-21T13:59:32.210586+0000","last_unstale":"2024-09-21T14:03:20.120272+0000","last_undegraded":"2024-09-21T14:03:20.120272+0000","last_fullsized":"2024-09-21T14:03:20.120272+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:28.014760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:28.014760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":663,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.114989+0000","last_change":"2024-09-21T14:02:29.731635+0000","last_active":"2024-09-21T14:03:20.114989+0000","last_peered":"2024-09-21T14:03:20.114989+0000","last_clean":"2024-09-21T14:03:20.114989+0000","last_became_active":"2024-09-21T13:59:32.205803+0000","last_became_peered":"2024-09-21T13:59:32.205803+0000","last_unstale":"2024-09-21T14:03:20.114989+0000","last_undegraded":"2024-09-21T14:03:20.114989+0000","last_fullsized":"2024-09-21T14:03:20.114989+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:29.731565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:02:29.731565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:53.473477+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"290'15","reported_seq":660,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.114637+0000","last_change":"2024-09-21T14:01:41.189676+0000","last_active":"2024-09-21T14:03:20.114637+0000","last_peered":"2024-09-21T14:03:20.114637+0000","last_clean":"2024-09-21T14:03:20.114637+0000","last_became_active":"2024-09-21T14:01:41.187904+0000","last_became_peered":"2024-09-21T14:01:41.187904+0000","last_unstale":"2024-09-21T14:03:20.114637+0000","last_undegraded":"2024-09-21T14:03:20.114637+0000","last_fullsized":"2024-09-21T14:03:20.114637+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:57:24.124604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:57:24.124604+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:18.325159+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":541,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.115894+0000","last_change":"2024-09-21T13:59:32.212325+0000","last_active":"2024-09-21T14:03:20.115894+0000","last_peered":"2024-09-21T14:03:20.115894+0000","last_clean":"2024-09-21T14:03:20.115894+0000","last_became_active":"2024-09-21T13:59:32.208749+0000","last_became_peered":"2024-09-21T13:59:32.208749+0000","last_unstale":"2024-09-21T14:03:20.115894+0000","last_undegraded":"2024-09-21T14:03:20.115894+0000","last_fullsized":"2024-09-21T14:03:20.115894+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:20.574745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:20.574745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"300'14","reported_seq":761,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.110098+0000","last_change":"2024-09-21T13:59:32.208253+0000","last_active":"2024-09-21T14:03:20.110098+0000","last_peered":"2024-09-21T14:03:20.110098+0000","last_clean":"2024-09-21T14:03:20.110098+0000","last_became_active":"2024-09-21T13:59:32.206926+0000","last_became_peered":"2024-09-21T13:59:32.206926+0000","last_unstale":"2024-09-21T14:03:20.110098+0000","last_undegraded":"2024-09-21T14:03:20.110098+0000","last_fullsized":"2024-09-21T14:03:20.110098+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'10","last_scrub_stamp":"2024-09-21T13:54:20.946782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:20.946782+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:15.133039+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":707,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.112370+0000","last_change":"2024-09-21T14:01:41.200564+0000","last_active":"2024-09-21T14:03:20.112370+0000","last_peered":"2024-09-21T14:03:20.112370+0000","last_clean":"2024-09-21T14:03:20.112370+0000","last_became_active":"2024-09-21T14:01:41.200113+0000","last_became_peered":"2024-09-21T14:01:41.200113+0000","last_unstale":"2024-09-21T14:03:20.112370+0000","last_undegraded":"2024-09-21T14:03:20.112370+0000","last_fullsized":"2024-09-21T14:03:20.112370+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:54.805619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:54.805619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:48.669884+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"290'20","reported_seq":656,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.110005+0000","last_change":"2024-09-21T13:59:32.236530+0000","last_active":"2024-09-21T14:03:20.110005+0000","last_peered":"2024-09-21T14:03:20.110005+0000","last_clean":"2024-09-21T14:03:20.110005+0000","last_became_active":"2024-09-21T13:59:32.208332+0000","last_became_peered":"2024-09-21T13:59:32.208332+0000","last_unstale":"2024-09-21T14:03:20.110005+0000","last_undegraded":"2024-09-21T14:03:20.110005+0000","last_fullsized":"2024-09-21T14:03:20.110005+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:57:37.671391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:57:37.671391+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:55.297241+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":648,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.116048+0000","last_change":"2024-09-21T13:59:32.212737+0000","last_active":"2024-09-21T14:03:20.116048+0000","last_peered":"2024-09-21T14:03:20.116048+0000","last_clean":"2024-09-21T14:03:20.116048+0000","last_became_active":"2024-09-21T13:59:32.209804+0000","last_became_peered":"2024-09-21T13:59:32.209804+0000","last_unstale":"2024-09-21T14:03:20.116048+0000","last_undegraded":"2024-09-21T14:03:20.116048+0000","last_fullsized":"2024-09-21T14:03:20.116048+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:25.571256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:25.571256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:15.928161+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"346'10","reported_seq":761,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.114545+0000","last_change":"2024-09-21T14:01:41.207030+0000","last_active":"2024-09-21T14:03:20.114545+0000","last_peered":"2024-09-21T14:03:20.114545+0000","last_clean":"2024-09-21T14:03:20.114545+0000","last_became_active":"2024-09-21T14:01:41.206303+0000","last_became_peered":"2024-09-21T14:01:41.206303+0000","last_unstale":"2024-09-21T14:03:20.114545+0000","last_undegraded":"2024-09-21T14:03:20.114545+0000","last_fullsized":"2024-09-21T14:03:20.114545+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:59:32.104224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:32.104224+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":545,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.116765+0000","last_change":"2024-09-21T13:59:32.213015+0000","last_active":"2024-09-21T14:03:20.116765+0000","last_peered":"2024-09-21T14:03:20.116765+0000","last_clean":"2024-09-21T14:03:20.116765+0000","last_became_active":"2024-09-21T13:59:32.211472+0000","last_became_peered":"2024-09-21T13:59:32.211472+0000","last_unstale":"2024-09-21T14:03:20.116765+0000","last_undegraded":"2024-09-21T14:03:20.116765+0000","last_fullsized":"2024-09-21T14:03:20.116765+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:22.647191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:22.647191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"290'19","reported_seq":771,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.113044+0000","last_change":"2024-09-21T14:02:34.670058+0000","last_active":"2024-09-21T14:03:20.113044+0000","last_peered":"2024-09-21T14:03:20.113044+0000","last_clean":"2024-09-21T14:03:20.113044+0000","last_became_active":"2024-09-21T13:58:17.771497+0000","last_became_peered":"2024-09-21T13:58:17.771497+0000","last_unstale":"2024-09-21T14:03:20.113044+0000","last_undegraded":"2024-09-21T14:03:20.113044+0000","last_fullsized":"2024-09-21T14:03:20.113044+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'19","last_scrub_stamp":"2024-09-21T14:02:34.669962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:34.669962+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:44.716931+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":290,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.116335+0000","last_change":"2024-09-21T14:02:57.807136+0000","last_active":"2024-09-21T14:03:20.116335+0000","last_peered":"2024-09-21T14:03:20.116335+0000","last_clean":"2024-09-21T14:03:20.116335+0000","last_became_active":"2024-09-21T13:59:32.201568+0000","last_became_peered":"2024-09-21T13:59:32.201568+0000","last_unstale":"2024-09-21T14:03:20.116335+0000","last_undegraded":"2024-09-21T14:03:20.116335+0000","last_fullsized":"2024-09-21T14:03:20.116335+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:57.807059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:02:57.807059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:09.290442+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":717,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.614771+0000","last_change":"2024-09-21T14:01:41.185588+0000","last_active":"2024-09-21T14:03:08.614771+0000","last_peered":"2024-09-21T14:03:08.614771+0000","last_clean":"2024-09-21T14:03:08.614771+0000","last_became_active":"2024-09-21T14:01:41.185293+0000","last_became_peered":"2024-09-21T14:01:41.185293+0000","last_unstale":"2024-09-21T14:03:08.614771+0000","last_undegraded":"2024-09-21T14:03:08.614771+0000","last_fullsized":"2024-09-21T14:03:08.614771+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:35.409237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:35.409237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:10.948556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":621,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.110947+0000","last_change":"2024-09-21T14:02:57.225650+0000","last_active":"2024-09-21T14:03:20.110947+0000","last_peered":"2024-09-21T14:03:20.110947+0000","last_clean":"2024-09-21T14:03:20.110947+0000","last_became_active":"2024-09-21T14:01:41.199671+0000","last_became_peered":"2024-09-21T14:01:41.199671+0000","last_unstale":"2024-09-21T14:03:20.110947+0000","last_undegraded":"2024-09-21T14:03:20.110947+0000","last_fullsized":"2024-09-21T14:03:20.110947+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:57.225602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:02:57.225602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:26.897407+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"290'15","reported_seq":769,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.615040+0000","last_change":"2024-09-21T14:02:56.315766+0000","last_active":"2024-09-21T14:03:08.615040+0000","last_peered":"2024-09-21T14:03:08.615040+0000","last_clean":"2024-09-21T14:03:08.615040+0000","last_became_active":"2024-09-21T13:58:17.764120+0000","last_became_peered":"2024-09-21T13:58:17.764120+0000","last_unstale":"2024-09-21T14:03:08.615040+0000","last_undegraded":"2024-09-21T14:03:08.615040+0000","last_fullsized":"2024-09-21T14:03:08.615040+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'15","last_scrub_stamp":"2024-09-21T14:02:56.315702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:56.315702+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:11.328147+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":652,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.116501+0000","last_change":"2024-09-21T13:59:32.223082+0000","last_active":"2024-09-21T14:03:20.116501+0000","last_peered":"2024-09-21T14:03:20.116501+0000","last_clean":"2024-09-21T14:03:20.116501+0000","last_became_active":"2024-09-21T13:59:32.221854+0000","last_became_peered":"2024-09-21T13:59:32.221854+0000","last_unstale":"2024-09-21T14:03:20.116501+0000","last_undegraded":"2024-09-21T14:03:20.116501+0000","last_fullsized":"2024-09-21T14:03:20.116501+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:17.691193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:17.691193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":588,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.113056+0000","last_change":"2024-09-21T13:57:12.703746+0000","last_active":"2024-09-21T14:03:20.113056+0000","last_peered":"2024-09-21T14:03:20.113056+0000","last_clean":"2024-09-21T14:03:20.113056+0000","last_became_active":"2024-09-21T13:57:12.703441+0000","last_became_peered":"2024-09-21T13:57:12.703441+0000","last_unstale":"2024-09-21T14:03:20.113056+0000","last_undegraded":"2024-09-21T14:03:20.113056+0000","last_fullsized":"2024-09-21T14:03:20.113056+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:19.999125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:19.999125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":510,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.113234+0000","last_change":"2024-09-21T14:01:41.188075+0000","last_active":"2024-09-21T14:03:20.113234+0000","last_peered":"2024-09-21T14:03:20.113234+0000","last_clean":"2024-09-21T14:03:20.113234+0000","last_became_active":"2024-09-21T14:01:41.186934+0000","last_became_peered":"2024-09-21T14:01:41.186934+0000","last_unstale":"2024-09-21T14:03:20.113234+0000","last_undegraded":"2024-09-21T14:03:20.113234+0000","last_fullsized":"2024-09-21T14:03:20.113234+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:47.334504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:47.334504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"290'19","reported_seq":732,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.111171+0000","last_change":"2024-09-21T13:59:32.277742+0000","last_active":"2024-09-21T14:03:20.111171+0000","last_peered":"2024-09-21T14:03:20.111171+0000","last_clean":"2024-09-21T14:03:20.111171+0000","last_became_active":"2024-09-21T13:59:32.231567+0000","last_became_peered":"2024-09-21T13:59:32.231567+0000","last_unstale":"2024-09-21T14:03:20.111171+0000","last_undegraded":"2024-09-21T14:03:20.111171+0000","last_fullsized":"2024-09-21T14:03:20.111171+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:54:22.978267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:22.978267+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":620,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.113976+0000","last_change":"2024-09-21T14:03:01.448170+0000","last_active":"2024-09-21T14:03:20.113976+0000","last_peered":"2024-09-21T14:03:20.113976+0000","last_clean":"2024-09-21T14:03:20.113976+0000","last_became_active":"2024-09-21T14:01:41.190760+0000","last_became_peered":"2024-09-21T14:01:41.190760+0000","last_unstale":"2024-09-21T14:03:20.113976+0000","last_undegraded":"2024-09-21T14:03:20.113976+0000","last_fullsized":"2024-09-21T14:03:20.113976+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:01.448109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:03:01.448109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:28.748585+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":658,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.116626+0000","last_change":"2024-09-21T13:59:32.219508+0000","last_active":"2024-09-21T14:03:20.116626+0000","last_peered":"2024-09-21T14:03:20.116626+0000","last_clean":"2024-09-21T14:03:20.116626+0000","last_became_active":"2024-09-21T13:59:32.214004+0000","last_became_peered":"2024-09-21T13:59:32.214004+0000","last_unstale":"2024-09-21T14:03:20.116626+0000","last_undegraded":"2024-09-21T14:03:20.116626+0000","last_fullsized":"2024-09-21T14:03:20.116626+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:36.525417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:36.525417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":654,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.113666+0000","last_change":"2024-09-21T13:59:28.124373+0000","last_active":"2024-09-21T14:03:20.113666+0000","last_peered":"2024-09-21T14:03:20.113666+0000","last_clean":"2024-09-21T14:03:20.113666+0000","last_became_active":"2024-09-21T13:58:17.778115+0000","last_became_peered":"2024-09-21T13:58:17.778115+0000","last_unstale":"2024-09-21T14:03:20.113666+0000","last_undegraded":"2024-09-21T14:03:20.113666+0000","last_fullsized":"2024-09-21T14:03:20.113666+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:28.124296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:28.124296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"290'8","reported_seq":678,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.108879+0000","last_change":"2024-09-21T13:59:26.034457+0000","last_active":"2024-09-21T14:03:20.108879+0000","last_peered":"2024-09-21T14:03:20.108879+0000","last_clean":"2024-09-21T14:03:20.108879+0000","last_became_active":"2024-09-21T13:58:17.775079+0000","last_became_peered":"2024-09-21T13:58:17.775079+0000","last_unstale":"2024-09-21T14:03:20.108879+0000","last_undegraded":"2024-09-21T14:03:20.108879+0000","last_fullsized":"2024-09-21T14:03:20.108879+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'8","last_scrub_stamp":"2024-09-21T13:59:26.034372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:26.034372+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":580,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.115025+0000","last_change":"2024-09-21T14:01:41.196335+0000","last_active":"2024-09-21T14:03:20.115025+0000","last_peered":"2024-09-21T14:03:20.115025+0000","last_clean":"2024-09-21T14:03:20.115025+0000","last_became_active":"2024-09-21T14:01:41.195989+0000","last_became_peered":"2024-09-21T14:01:41.195989+0000","last_unstale":"2024-09-21T14:03:20.115025+0000","last_undegraded":"2024-09-21T14:03:20.115025+0000","last_fullsized":"2024-09-21T14:03:20.115025+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:16.457445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:16.457445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":273,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.111282+0000","last_change":"2024-09-21T14:00:57.128764+0000","last_active":"2024-09-21T14:03:20.111282+0000","last_peered":"2024-09-21T14:03:20.111282+0000","last_clean":"2024-09-21T14:03:20.111282+0000","last_became_active":"2024-09-21T13:59:32.204380+0000","last_became_peered":"2024-09-21T13:59:32.204380+0000","last_unstale":"2024-09-21T14:03:20.111282+0000","last_undegraded":"2024-09-21T14:03:20.111282+0000","last_fullsized":"2024-09-21T14:03:20.111282+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:00:57.128713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:00:57.128713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:24.563060+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":591,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.115794+0000","last_change":"2024-09-21T13:59:32.212615+0000","last_active":"2024-09-21T14:03:20.115794+0000","last_peered":"2024-09-21T14:03:20.115794+0000","last_clean":"2024-09-21T14:03:20.115794+0000","last_became_active":"2024-09-21T13:59:32.210134+0000","last_became_peered":"2024-09-21T13:59:32.210134+0000","last_unstale":"2024-09-21T14:03:20.115794+0000","last_undegraded":"2024-09-21T14:03:20.115794+0000","last_fullsized":"2024-09-21T14:03:20.115794+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:08.852789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:55:08.852789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:10.940052+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":701,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-21T14:03:08.616321+0000","last_change":"2024-09-21T14:01:41.187792+0000","last_active":"2024-09-21T14:03:08.616321+0000","last_peered":"2024-09-21T14:03:08.616321+0000","last_clean":"2024-09-21T14:03:08.616321+0000","last_became_active":"2024-09-21T14:01:41.187438+0000","last_became_peered":"2024-09-21T14:01:41.187438+0000","last_unstale":"2024-09-21T14:03:08.616321+0000","last_undegraded":"2024-09-21T14:03:08.616321+0000","last_fullsized":"2024-09-21T14:03:08.616321+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:17.706325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:17.706325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:40.383161+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":634,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.110877+0000","last_change":"2024-09-21T13:58:13.776632+0000","last_active":"2024-09-21T14:03:20.110877+0000","last_peered":"2024-09-21T14:03:20.110877+0000","last_clean":"2024-09-21T14:03:20.110877+0000","last_became_active":"2024-09-21T13:57:12.704042+0000","last_became_peered":"2024-09-21T13:57:12.704042+0000","last_unstale":"2024-09-21T14:03:20.110877+0000","last_undegraded":"2024-09-21T14:03:20.110877+0000","last_fullsized":"2024-09-21T14:03:20.110877+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:13.776572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:13.776572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"290'19","reported_seq":305,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.111167+0000","last_change":"2024-09-21T14:01:54.680822+0000","last_active":"2024-09-21T14:03:20.111167+0000","last_peered":"2024-09-21T14:03:20.111167+0000","last_clean":"2024-09-21T14:03:20.111167+0000","last_became_active":"2024-09-21T14:01:41.194606+0000","last_became_peered":"2024-09-21T14:01:41.194606+0000","last_unstale":"2024-09-21T14:03:20.111167+0000","last_undegraded":"2024-09-21T14:03:20.111167+0000","last_fullsized":"2024-09-21T14:03:20.111167+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'19","last_scrub_stamp":"2024-09-21T14:01:54.680735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:54.680735+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:21.312249+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":560,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-09-21T14:03:20.116688+0000","last_change":"2024-09-21T13:59:32.203045+0000","last_active":"2024-09-21T14:03:20.116688+0000","last_peered":"2024-09-21T14:03:20.116688+0000","last_clean":"2024-09-21T14:03:20.116688+0000","last_became_active":"2024-09-21T13:59:32.202225+0000","last_became_peered":"2024-09-21T13:59:32.202225+0000","last_unstale":"2024-09-21T14:03:20.116688+0000","last_undegraded":"2024-09-21T14:03:20.116688+0000","last_fullsized":"2024-09-21T14:03:20.116688+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:40.650832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:40.650832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":2649554944,"num_objects":40429,"num_object_clones":0,"num_object_copies":121287,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40429,"num_write_kb":2587456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7580614656,"data_stored":7580614656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40429,"ondisk_log_size":40429,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1111,"num_read_kb":868,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":1914,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":687,"ondisk_log_size":687,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":235,"num_read_kb":230,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":375688,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":139,"num_write":73,"num_write_kb":1036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1031048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":298,"seq":1279900254255,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":911892,"kb_used_data":603836,"kb_used_omap":4,"kb_used_meta":308027,"kb_avail":92833260,"statfs":{"total":95995035648,"available":95061258240,"internally_reserved":0,"allocated":618328064,"data_stored":617423960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":315420328},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463095,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1523896,"kb_used_data":1172008,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":92221256,"statfs":{"total":95995035648,"available":94434566144,"internally_reserved":0,"allocated":1200136192,"data_stored":1199251483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":360315334},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":345,"seq":1481763717141,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":859780,"kb_used_data":724592,"kb_used_omap":4,"kb_used_meta":135163,"kb_avail":92885372,"statfs":{"total":95995035648,"available":95114620928,"internally_reserved":0,"allocated":741982208,"data_stored":741090767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":138407587},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920964,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1691800,"kb_used_data":1303816,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":92053352,"statfs":{"total":95995035648,"available":94262632448,"internally_reserved":0,"allocated":1335107584,"data_stored":1334190725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":397277637},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":240,"seq":1030792151115,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1083360,"kb_used_data":766480,"kb_used_omap":4,"kb_used_meta":316859,"kb_avail":92661792,"statfs":{"total":95995035648,"available":94885675008,"internally_reserved":0,"allocated":784875520,"data_stored":783951843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4443,"internal_metadata":324464293},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":130,"seq":558345748617,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1535676,"kb_used_data":971756,"kb_used_omap":4,"kb_used_meta":563899,"kb_avail":92209476,"statfs":{"total":95995035648,"available":94422503424,"internally_reserved":0,"allocated":995078144,"data_stored":994190601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4441,"internal_metadata":577433255},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575061,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1027744,"kb_used_data":419024,"kb_used_omap":1,"kb_used_meta":608702,"kb_avail":92717408,"statfs":{"total":95995035648,"available":94942625792,"internally_reserved":0,"allocated":429080576,"data_stored":428140109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":623311301},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738587,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1891764,"kb_used_data":1328868,"kb_used_omap":1,"kb_used_meta":562878,"kb_avail":91853388,"statfs":{"total":95995035648,"available":94057869312,"internally_reserved":0,"allocated":1360760832,"data_stored":1359864417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":576387525},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1417412608,"data_stored":1417412608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":436404224,"data_stored":436404224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":994246656,"data_stored":994246656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":781647872,"data_stored":781647872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1371406336,"data_stored":1371406336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":761200640,"data_stored":761200640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1198915584,"data_stored":1198915584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":619380736,"data_stored":619380736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T14:03:25.618 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-21T14:03:25.618 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T14:03:25.618 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-09-21T14:03:25.618 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T14:03:25.618 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T14:03:25.618 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-09-21T14:03:25.665 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:03:25.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:25.721 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:25.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:03:25.821 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:03:25.852 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:25.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:25.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:03:25.868 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:2024-09-21T14:03:25.869086+0000 min lat: 0.00132701 max lat: 0.17048 avg lat: 0.00705654 2024-09-21T14:03:25.868 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T14:03:25.868 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 20 16 45336 45320 141.607 90.0625 0.00812083 0.00705654 2024-09-21T14:03:25.873 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T14:03:25.954 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:03:26.033 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:26.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:26.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:03:26.080 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:03:26.127 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-09-21T14:03:26.134 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:03:26.166 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:26.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:26.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:03:26.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:26 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2121466384' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:03:26.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:26 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:03:26.269 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:03:26.314 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:26.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:26.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:03:26.416 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:03:26.451 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:26.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:26.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:03:26.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:26 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2121466384' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:03:26.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:26 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:03:26.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:26 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2121466384' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:03:26.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:26 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:03:26.553 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:03:26.590 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:26.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:26.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:03:26.692 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:03:26.727 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:26.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:26.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:03:26.741 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T14:03:26.741 INFO:teuthology.orchestra.run.smithi187.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-21T14:03:26.741 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T14:03:26.829 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:03:26.861 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:26.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:26.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:03:26.868 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 21 16 46941 46925 139.641 100.312 0.00824056 0.00715534 2024-09-21T14:03:26.963 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:27.005 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:27.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:27.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:27.107 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:03:27.144 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:27.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:27.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:03:27.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:27 smithi187 ceph-mon[33032]: pgmap v738: 158 pgs: 158 active+clean; 2.6 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 128 MiB/s wr, 2.04k op/s 2024-09-21T14:03:27.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:27 smithi187 ceph-mon[33032]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:03:27.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:27 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T14:03:27.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:27 smithi187 ceph-mon[33032]: osdmap e391: 8 total, 8 up, 8 in 2024-09-21T14:03:27.246 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:03:27.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:27 smithi049 ceph-mon[40489]: pgmap v738: 158 pgs: 158 active+clean; 2.6 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 128 MiB/s wr, 2.04k op/s 2024-09-21T14:03:27.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:27 smithi049 ceph-mon[40489]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:03:27.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:27 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T14:03:27.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:27 smithi049 ceph-mon[40489]: osdmap e391: 8 total, 8 up, 8 in 2024-09-21T14:03:27.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:27 smithi049 ceph-mon[30080]: pgmap v738: 158 pgs: 158 active+clean; 2.6 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 128 MiB/s wr, 2.04k op/s 2024-09-21T14:03:27.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:27 smithi049 ceph-mon[30080]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:03:27.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:27 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T14:03:27.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:27 smithi049 ceph-mon[30080]: osdmap e391: 8 total, 8 up, 8 in 2024-09-21T14:03:27.282 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:27.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:27.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:03:27.384 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:03:27.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:27.447 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:27.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:03:27.473 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:03:27.548 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:03:27.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:27.596 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:27.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:03:27.697 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:03:27.732 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:27.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:27.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:03:27.834 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:27.865 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:27.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:27.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:27.869 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 22 16 48631 48615 138.086 105.625 0.00717723 0.00723547 2024-09-21T14:03:27.966 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:03:27.998 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:27.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:27.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:03:28.099 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:28.131 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:28.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:28.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:28.190 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:03:28.191 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:03:28.227 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:03:28.227 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:03:28.228 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:28 smithi049 ceph-mon[40489]: 4.1 scrub starts 2024-09-21T14:03:28.228 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:28 smithi049 ceph-mon[40489]: 4.1 scrub ok 2024-09-21T14:03:28.228 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:28 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:03:28.228 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:28 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:03:28.228 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:28 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]} : dispatch 2024-09-21T14:03:28.228 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:28 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]} : dispatch 2024-09-21T14:03:28.228 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:28 smithi049 ceph-mon[40489]: 6.16 scrub starts 2024-09-21T14:03:28.228 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:28 smithi049 ceph-mon[40489]: 6.16 scrub ok 2024-09-21T14:03:28.229 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:28 smithi049 ceph-mon[30080]: 4.1 scrub starts 2024-09-21T14:03:28.229 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:28 smithi049 ceph-mon[30080]: 4.1 scrub ok 2024-09-21T14:03:28.229 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:28 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:03:28.229 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:28 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:03:28.229 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:28 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]} : dispatch 2024-09-21T14:03:28.229 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:28 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]} : dispatch 2024-09-21T14:03:28.229 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:28 smithi049 ceph-mon[30080]: 6.16 scrub starts 2024-09-21T14:03:28.229 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:28 smithi049 ceph-mon[30080]: 6.16 scrub ok 2024-09-21T14:03:28.232 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:03:28.261 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:03:28.261 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:03:28.284 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:28.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:28.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:03:28.292 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:03:28.297 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:03:28.297 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:03:28.337 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:03:28.337 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:03:28.378 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:03:28.378 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:03:28.386 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:03:28.413 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:03:28.414 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:03:28.432 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:28 smithi187 ceph-mon[33032]: 4.1 scrub starts 2024-09-21T14:03:28.432 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:28 smithi187 ceph-mon[33032]: 4.1 scrub ok 2024-09-21T14:03:28.432 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:28 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:03:28.432 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:28 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:03:28.432 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:28 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]} : dispatch 2024-09-21T14:03:28.432 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:28 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]} : dispatch 2024-09-21T14:03:28.433 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:28 smithi187 ceph-mon[33032]: 6.16 scrub starts 2024-09-21T14:03:28.433 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:28 smithi187 ceph-mon[33032]: 6.16 scrub ok 2024-09-21T14:03:28.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:28.443 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:28.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:03:28.451 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:03:28.452 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:03:28.543 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:03:28.590 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:28.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:28.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:03:28.692 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:03:28.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:28.724 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:28.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:03:28.825 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:03:28.862 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:28.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:28.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:03:28.869 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 23 16 50003 49987 135.811 85.75 0.0112874 0.00735797 2024-09-21T14:03:28.907 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T14:03:28.908 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-21T14:03:28.908 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-21T14:03:28.908 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-21T14:03:28.908 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T14:03:28.964 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:03:29.006 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:29.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:29.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:03:29.089 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T14:03:29.108 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:03:29.142 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:29.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:29.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:03:29.244 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:03:29.272 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:29.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:29.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:03:29.374 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:03:29.405 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:29 smithi049 ceph-mon[40489]: pgmap v740: 158 pgs: 158 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.3 KiB/s rd, 93 MiB/s wr, 1.48k op/s 2024-09-21T14:03:29.406 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:29 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-09-21T14:03:29.406 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:29 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-09-21T14:03:29.406 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:29 smithi049 ceph-mon[40489]: osdmap e392: 8 total, 8 up, 8 in 2024-09-21T14:03:29.406 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:29 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/826469065' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:03:29.406 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:29 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:03:29.406 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:29 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:03:29.406 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:29 smithi049 ceph-mon[40489]: 6.1c scrub starts 2024-09-21T14:03:29.406 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:29 smithi049 ceph-mon[40489]: 6.1c scrub ok 2024-09-21T14:03:29.406 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:29 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:03:29.407 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:29 smithi049 ceph-mon[30080]: pgmap v740: 158 pgs: 158 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.3 KiB/s rd, 93 MiB/s wr, 1.48k op/s 2024-09-21T14:03:29.407 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:29 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-09-21T14:03:29.407 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:29 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-09-21T14:03:29.407 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:29 smithi049 ceph-mon[30080]: osdmap e392: 8 total, 8 up, 8 in 2024-09-21T14:03:29.407 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:29 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/826469065' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:03:29.407 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:29 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:03:29.407 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:29 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:03:29.407 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:29 smithi049 ceph-mon[30080]: 6.1c scrub starts 2024-09-21T14:03:29.407 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:29 smithi049 ceph-mon[30080]: 6.1c scrub ok 2024-09-21T14:03:29.407 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:29 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:03:29.425 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:29.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:29.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:03:29.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:29 smithi187 ceph-mon[33032]: pgmap v740: 158 pgs: 158 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.3 KiB/s rd, 93 MiB/s wr, 1.48k op/s 2024-09-21T14:03:29.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:29 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-09-21T14:03:29.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:29 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-09-21T14:03:29.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:29 smithi187 ceph-mon[33032]: osdmap e392: 8 total, 8 up, 8 in 2024-09-21T14:03:29.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:29 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/826469065' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:03:29.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:29 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:03:29.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:29 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:03:29.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:29 smithi187 ceph-mon[33032]: 6.1c scrub starts 2024-09-21T14:03:29.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:29 smithi187 ceph-mon[33032]: 6.1c scrub ok 2024-09-21T14:03:29.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:29 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:03:29.527 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:03:29.560 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:29.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:29.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:03:29.662 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:03:29.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:29.699 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:29.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:03:29.800 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:03:29.839 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:29.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:29.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:03:29.869 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 24 16 51602 51586 134.316 99.9375 0.00675929 0.00743979 2024-09-21T14:03:29.941 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:29.978 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:29.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:29.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:30.079 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:03:30.124 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:30.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:30.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:03:30.128 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:03:29 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:03:30.226 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:03:30.262 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:30.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:30.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:03:30.364 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:03:30.414 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:30.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:30.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:03:30.429 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:03:30.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:30 smithi187 ceph-mon[33032]: osdmap e393: 8 total, 8 up, 8 in 2024-09-21T14:03:30.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:30 smithi187 ceph-mon[33032]: 6.10 scrub starts 2024-09-21T14:03:30.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:30 smithi187 ceph-mon[33032]: 6.10 scrub ok 2024-09-21T14:03:30.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:30 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:03:30.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:30 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:03:30.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:30 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:03:30.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:30 smithi187 ceph-mon[33032]: osdmap e394: 8 total, 8 up, 8 in 2024-09-21T14:03:30.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:30 smithi049 ceph-mon[30080]: osdmap e393: 8 total, 8 up, 8 in 2024-09-21T14:03:30.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:30 smithi049 ceph-mon[30080]: 6.10 scrub starts 2024-09-21T14:03:30.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:30 smithi049 ceph-mon[30080]: 6.10 scrub ok 2024-09-21T14:03:30.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:30 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:03:30.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:30 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:03:30.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:30 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:03:30.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:30 smithi049 ceph-mon[30080]: osdmap e394: 8 total, 8 up, 8 in 2024-09-21T14:03:30.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:30 smithi049 ceph-mon[40489]: osdmap e393: 8 total, 8 up, 8 in 2024-09-21T14:03:30.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:30 smithi049 ceph-mon[40489]: 6.10 scrub starts 2024-09-21T14:03:30.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:30 smithi049 ceph-mon[40489]: 6.10 scrub ok 2024-09-21T14:03:30.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:30 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:03:30.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:30 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:03:30.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:30 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:03:30.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:30 smithi049 ceph-mon[40489]: osdmap e394: 8 total, 8 up, 8 in 2024-09-21T14:03:30.516 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:03:30.548 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:30.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:30.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:03:30.649 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:03:30.689 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:30.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:30.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:03:30.791 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:03:30.822 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:30.823 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:03:30 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:03:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:03:30.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:30.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:03:30.870 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 25 16 53074 53058 132.623 92 0.00437265 0.00753538 2024-09-21T14:03:30.925 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:03:30.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:30.962 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:30.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:03:31.063 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:03:31.103 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:31.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:31.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:03:31.205 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:31.242 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:31.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:31.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:31.297 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T14:03:31.343 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:03:31.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:31 smithi049 ceph-mon[30080]: pgmap v743: 158 pgs: 1 peering, 157 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 88 MiB/s wr, 1.40k op/s 2024-09-21T14:03:31.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:31 smithi049 ceph-mon[30080]: 2.2 scrub starts 2024-09-21T14:03:31.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:31 smithi049 ceph-mon[30080]: 2.2 scrub ok 2024-09-21T14:03:31.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:31 smithi049 ceph-mon[30080]: 5.11 scrub starts 2024-09-21T14:03:31.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:31 smithi049 ceph-mon[30080]: 5.11 scrub ok 2024-09-21T14:03:31.376 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:31 smithi049 ceph-mon[40489]: pgmap v743: 158 pgs: 1 peering, 157 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 88 MiB/s wr, 1.40k op/s 2024-09-21T14:03:31.376 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:31 smithi049 ceph-mon[40489]: 2.2 scrub starts 2024-09-21T14:03:31.376 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:31 smithi049 ceph-mon[40489]: 2.2 scrub ok 2024-09-21T14:03:31.376 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:31 smithi049 ceph-mon[40489]: 5.11 scrub starts 2024-09-21T14:03:31.376 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:31 smithi049 ceph-mon[40489]: 5.11 scrub ok 2024-09-21T14:03:31.389 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:31.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:31.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:03:31.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:31 smithi187 ceph-mon[33032]: pgmap v743: 158 pgs: 1 peering, 157 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 88 MiB/s wr, 1.40k op/s 2024-09-21T14:03:31.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:31 smithi187 ceph-mon[33032]: 2.2 scrub starts 2024-09-21T14:03:31.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:31 smithi187 ceph-mon[33032]: 2.2 scrub ok 2024-09-21T14:03:31.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:31 smithi187 ceph-mon[33032]: 5.11 scrub starts 2024-09-21T14:03:31.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:31 smithi187 ceph-mon[33032]: 5.11 scrub ok 2024-09-21T14:03:31.491 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:03:31.528 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:31.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:31.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:03:31.630 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:03:31.667 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:31.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:31.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:03:31.768 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:03:31.820 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:31.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:31.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:03:31.869 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 26 16 54646 54630 131.301 98.25 0.00580922 0.00761098 2024-09-21T14:03:31.902 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T14:03:31.921 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:03:31.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:31.957 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:31.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:03:32.057 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:32.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:32.099 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:32.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:32.169 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-09-21T14:03:32.200 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:03:32.222 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:03:32.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:32 smithi049 ceph-mon[40489]: 4.d scrub starts 2024-09-21T14:03:32.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:32 smithi049 ceph-mon[40489]: 4.d scrub ok 2024-09-21T14:03:32.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:32 smithi049 ceph-mon[40489]: 4.8 scrub starts 2024-09-21T14:03:32.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:32 smithi049 ceph-mon[40489]: 4.8 scrub ok 2024-09-21T14:03:32.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:32 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2781825246' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:03:32.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:32 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:03:32.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:32 smithi049 ceph-mon[40489]: 4.1f scrub starts 2024-09-21T14:03:32.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:32 smithi049 ceph-mon[40489]: 4.1f scrub ok 2024-09-21T14:03:32.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:32 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:03:32.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:32 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:03:32.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:32 smithi049 ceph-mon[40489]: 3.d scrub starts 2024-09-21T14:03:32.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:32 smithi049 ceph-mon[40489]: 3.d scrub ok 2024-09-21T14:03:32.257 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:32.257 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:32 smithi049 ceph-mon[30080]: 4.d scrub starts 2024-09-21T14:03:32.258 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:32 smithi049 ceph-mon[30080]: 4.d scrub ok 2024-09-21T14:03:32.258 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:32 smithi049 ceph-mon[30080]: 4.8 scrub starts 2024-09-21T14:03:32.258 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:32 smithi049 ceph-mon[30080]: 4.8 scrub ok 2024-09-21T14:03:32.258 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:32 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2781825246' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:03:32.258 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:32 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:03:32.258 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:32 smithi049 ceph-mon[30080]: 4.1f scrub starts 2024-09-21T14:03:32.258 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:32 smithi049 ceph-mon[30080]: 4.1f scrub ok 2024-09-21T14:03:32.258 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:32 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:03:32.258 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:32 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:03:32.258 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:32 smithi049 ceph-mon[30080]: 3.d scrub starts 2024-09-21T14:03:32.258 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:32 smithi049 ceph-mon[30080]: 3.d scrub ok 2024-09-21T14:03:32.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:32.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:03:32.360 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:03:32.410 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:32.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:32.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:03:32.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:32 smithi187 ceph-mon[33032]: 4.d scrub starts 2024-09-21T14:03:32.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:32 smithi187 ceph-mon[33032]: 4.d scrub ok 2024-09-21T14:03:32.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:32 smithi187 ceph-mon[33032]: 4.8 scrub starts 2024-09-21T14:03:32.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:32 smithi187 ceph-mon[33032]: 4.8 scrub ok 2024-09-21T14:03:32.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:32 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2781825246' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:03:32.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:32 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:03:32.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:32 smithi187 ceph-mon[33032]: 4.1f scrub starts 2024-09-21T14:03:32.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:32 smithi187 ceph-mon[33032]: 4.1f scrub ok 2024-09-21T14:03:32.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:32 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:03:32.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:32 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:03:32.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:32 smithi187 ceph-mon[33032]: 3.d scrub starts 2024-09-21T14:03:32.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:32 smithi187 ceph-mon[33032]: 3.d scrub ok 2024-09-21T14:03:32.512 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:03:32.534 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:32.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:32.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:03:32.635 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:03:32.677 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:32.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:32.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:03:32.779 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:03:32.813 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:32.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:32.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:03:32.840 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 4, 6, 2, 3, 7, 5] 2024-09-21T14:03:32.840 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T14:03:32.840 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T14:03:32.870 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 27 16 56067 56051 129.727 88.8125 0.00665395 0.00770365 2024-09-21T14:03:32.914 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:32.949 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:32.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:32.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:33.051 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:33.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:33.090 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:33.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:33.122 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T14:03:33.123 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-21T14:03:33.123 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-21T14:03:33.123 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-21T14:03:33.123 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.123 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.15207.0:14214 8.2 8:46d3a502:::benchmark_data_smithi049_109311_object14213:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-09-21T14:03:33.123 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:54:23.117083+0000", 2024-09-21T14:03:33.123 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 550.00259150099998, 2024-09-21T14:03:33.123 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.12033872, 2024-09-21T14:03:33.123 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:03:33.123 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:03:33.123 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:03:33.123 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:03:33.124 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.15207", 2024-09-21T14:03:33.124 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:03:33.124 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 14214 2024-09-21T14:03:33.124 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.124 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:03:33.124 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.124 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:03:33.124 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.117083+0000", 2024-09-21T14:03:33.124 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:03:33.124 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.124 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.124 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:03:33.125 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.117083+0000", 2024-09-21T14:03:33.125 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:03:33.125 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.125 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.125 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:03:33.125 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.117094+0000", 2024-09-21T14:03:33.125 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0915e-05 2024-09-21T14:03:33.125 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.125 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.125 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:03:33.125 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.117127+0000", 2024-09-21T14:03:33.125 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3222999999999997e-05 2024-09-21T14:03:33.125 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.125 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.126 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:03:33.126 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.117141+0000", 2024-09-21T14:03:33.126 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3604999999999999e-05 2024-09-21T14:03:33.126 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.126 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.126 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:03:33.126 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.117148+0000", 2024-09-21T14:03:33.126 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5739999999999998e-06 2024-09-21T14:03:33.126 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.126 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.126 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:03:33.126 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.117215+0000", 2024-09-21T14:03:33.126 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6672000000000001e-05 2024-09-21T14:03:33.126 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.127 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.127 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:03:33.127 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.117316+0000", 2024-09-21T14:03:33.127 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000100831 2024-09-21T14:03:33.127 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.127 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.127 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,5", 2024-09-21T14:03:33.127 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.117464+0000", 2024-09-21T14:03:33.127 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000148035 2024-09-21T14:03:33.127 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.127 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.127 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:03:33.127 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.118494+0000", 2024-09-21T14:03:33.127 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001029587 2024-09-21T14:03:33.127 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.128 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.128 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:33.128 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.118598+0000", 2024-09-21T14:03:33.128 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000104079 2024-09-21T14:03:33.128 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.128 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.128 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:33.128 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.237344+0000", 2024-09-21T14:03:33.128 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.118746166 2024-09-21T14:03:33.128 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.128 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.128 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:03:33.128 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.237385+0000", 2024-09-21T14:03:33.128 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1495000000000002e-05 2024-09-21T14:03:33.129 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.129 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.129 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:03:33.129 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.237422+0000", 2024-09-21T14:03:33.129 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6538000000000002e-05 2024-09-21T14:03:33.129 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:03:33.129 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:03:33.129 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:03:33.129 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.129 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.129 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.15207.0:14217 8.2 8:42c0ac0e:::benchmark_data_smithi049_109311_object14216:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-09-21T14:03:33.129 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:54:23.118628+0000", 2024-09-21T14:03:33.129 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 550.00104711500001, 2024-09-21T14:03:33.129 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.120069462, 2024-09-21T14:03:33.130 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:03:33.130 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:03:33.130 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:03:33.130 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:03:33.130 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.15207", 2024-09-21T14:03:33.130 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:03:33.130 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 14217 2024-09-21T14:03:33.130 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.130 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:03:33.130 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.130 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:03:33.130 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.118628+0000", 2024-09-21T14:03:33.130 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:03:33.130 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.131 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.131 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:03:33.131 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.118628+0000", 2024-09-21T14:03:33.131 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:03:33.131 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.131 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.131 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:03:33.131 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.118629+0000", 2024-09-21T14:03:33.131 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0499999999999999e-06 2024-09-21T14:03:33.131 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.131 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.131 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:03:33.131 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.118673+0000", 2024-09-21T14:03:33.132 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4274000000000003e-05 2024-09-21T14:03:33.132 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.132 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.132 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:03:33.132 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.118686+0000", 2024-09-21T14:03:33.132 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3189e-05 2024-09-21T14:03:33.132 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.132 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.132 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:03:33.132 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.118695+0000", 2024-09-21T14:03:33.132 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0690000000000008e-06 2024-09-21T14:03:33.132 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.132 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.132 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:03:33.132 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.118761+0000", 2024-09-21T14:03:33.133 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6203000000000003e-05 2024-09-21T14:03:33.133 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.133 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.133 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:03:33.133 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.118897+0000", 2024-09-21T14:03:33.133 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013574400000000001 2024-09-21T14:03:33.133 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.133 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.133 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,5", 2024-09-21T14:03:33.133 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.119033+0000", 2024-09-21T14:03:33.133 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013564499999999999 2024-09-21T14:03:33.133 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.133 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.133 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:03:33.133 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.119556+0000", 2024-09-21T14:03:33.133 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00052297800000000001 2024-09-21T14:03:33.134 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.134 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.134 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:33.134 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.120042+0000", 2024-09-21T14:03:33.134 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00048602399999999999 2024-09-21T14:03:33.134 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.134 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.134 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:33.134 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.238612+0000", 2024-09-21T14:03:33.134 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.118570721 2024-09-21T14:03:33.134 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.134 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.134 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:03:33.135 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.238657+0000", 2024-09-21T14:03:33.135 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4935000000000003e-05 2024-09-21T14:03:33.135 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.135 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.135 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:03:33.135 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:54:23.238697+0000", 2024-09-21T14:03:33.135 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9629999999999998e-05 2024-09-21T14:03:33.135 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:03:33.135 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:03:33.135 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:03:33.135 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.135 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.135 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.15207.0:211127 8.b 8:dd21c456:::benchmark_data_smithi049_109311_object84608:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-09-21T14:03:33.135 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:56:54.629514+0000", 2024-09-21T14:03:33.135 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 398.49016105800001, 2024-09-21T14:03:33.135 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13217907500000001, 2024-09-21T14:03:33.136 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:03:33.136 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:03:33.136 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:03:33.136 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:03:33.136 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.15207", 2024-09-21T14:03:33.136 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:03:33.136 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 211127 2024-09-21T14:03:33.136 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.136 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:03:33.136 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.136 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:03:33.136 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.629514+0000", 2024-09-21T14:03:33.136 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:03:33.136 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.137 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.137 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:03:33.137 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.629514+0000", 2024-09-21T14:03:33.137 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:03:33.137 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.137 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.137 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:03:33.137 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.629514+0000", 2024-09-21T14:03:33.137 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4999999999999998e-07 2024-09-21T14:03:33.137 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.137 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.137 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:03:33.137 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.629516+0000", 2024-09-21T14:03:33.137 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7010000000000001e-06 2024-09-21T14:03:33.137 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.138 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.138 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:03:33.138 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.629519+0000", 2024-09-21T14:03:33.138 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.766e-06 2024-09-21T14:03:33.138 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.138 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.138 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:03:33.138 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.629522+0000", 2024-09-21T14:03:33.138 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9519999999999999e-06 2024-09-21T14:03:33.138 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.138 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.138 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:03:33.138 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.707551+0000", 2024-09-21T14:03:33.139 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.078028231000000003 2024-09-21T14:03:33.139 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.139 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.139 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:03:33.139 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.707650+0000", 2024-09-21T14:03:33.139 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.9407999999999994e-05 2024-09-21T14:03:33.139 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.139 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.139 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1", 2024-09-21T14:03:33.139 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.707755+0000", 2024-09-21T14:03:33.139 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000105178 2024-09-21T14:03:33.139 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.139 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.139 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:03:33.139 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.716265+0000", 2024-09-21T14:03:33.140 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0085100430000000001 2024-09-21T14:03:33.140 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.140 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.140 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:33.140 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.761562+0000", 2024-09-21T14:03:33.140 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.045296967 2024-09-21T14:03:33.140 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.140 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.140 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:03:33.140 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.761602+0000", 2024-09-21T14:03:33.140 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9913999999999998e-05 2024-09-21T14:03:33.140 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.140 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.140 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:03:33.141 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.761693+0000", 2024-09-21T14:03:33.141 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0564999999999998e-05 2024-09-21T14:03:33.141 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:03:33.141 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:03:33.141 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:03:33.141 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.141 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.141 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.15207.0:211128 8.b 8:dde34708:::benchmark_data_smithi049_109311_object84609:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-09-21T14:03:33.141 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:56:54.629541+0000", 2024-09-21T14:03:33.141 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 398.49013368700003, 2024-09-21T14:03:33.141 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.14686216199999999, 2024-09-21T14:03:33.141 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:03:33.141 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:03:33.141 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:03:33.141 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:03:33.142 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.15207", 2024-09-21T14:03:33.142 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:03:33.142 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 211128 2024-09-21T14:03:33.142 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.142 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:03:33.142 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.142 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:03:33.142 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.629541+0000", 2024-09-21T14:03:33.142 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:03:33.142 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.142 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.142 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:03:33.142 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.629541+0000", 2024-09-21T14:03:33.142 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:03:33.142 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.143 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.143 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:03:33.143 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.629541+0000", 2024-09-21T14:03:33.143 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0699999999999998e-07 2024-09-21T14:03:33.143 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.143 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.143 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:03:33.143 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.629543+0000", 2024-09-21T14:03:33.143 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.513e-06 2024-09-21T14:03:33.143 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.143 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.143 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:03:33.143 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.629546+0000", 2024-09-21T14:03:33.143 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.642e-06 2024-09-21T14:03:33.144 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.144 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.144 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:03:33.144 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.629549+0000", 2024-09-21T14:03:33.144 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8480000000000001e-06 2024-09-21T14:03:33.144 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.144 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.144 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:03:33.144 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.710199+0000", 2024-09-21T14:03:33.144 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.080649352999999993 2024-09-21T14:03:33.144 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.144 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.144 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:03:33.144 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.710312+0000", 2024-09-21T14:03:33.144 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011377799999999999 2024-09-21T14:03:33.145 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.145 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.145 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1", 2024-09-21T14:03:33.145 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.710674+0000", 2024-09-21T14:03:33.145 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00036162500000000002 2024-09-21T14:03:33.145 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.145 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.145 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:03:33.145 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.716270+0000", 2024-09-21T14:03:33.145 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0055961520000000001 2024-09-21T14:03:33.145 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.145 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.145 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:33.145 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.776233+0000", 2024-09-21T14:03:33.145 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.059962834999999999 2024-09-21T14:03:33.146 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.146 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.146 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:03:33.146 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.776304+0000", 2024-09-21T14:03:33.146 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1371999999999994e-05 2024-09-21T14:03:33.146 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.146 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.146 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:03:33.146 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:56:54.776403+0000", 2024-09-21T14:03:33.146 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.8737e-05 2024-09-21T14:03:33.146 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:03:33.146 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:03:33.146 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:03:33.146 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.147 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.147 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.15207.0:232030 8.2 8:49ac6d95:::benchmark_data_smithi049_109311_object105511:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T14:03:33.147 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:57:11.653701+0000", 2024-09-21T14:03:33.147 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 381.46597355199998, 2024-09-21T14:03:33.147 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.22905246600000001, 2024-09-21T14:03:33.147 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:03:33.147 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:03:33.147 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:03:33.147 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:03:33.147 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.15207", 2024-09-21T14:03:33.147 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:03:33.147 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 232030 2024-09-21T14:03:33.147 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.147 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:03:33.147 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.147 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:03:33.148 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.653701+0000", 2024-09-21T14:03:33.148 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:03:33.148 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.148 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.148 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:03:33.148 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.653701+0000", 2024-09-21T14:03:33.148 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:03:33.148 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.148 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.148 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:03:33.148 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.653702+0000", 2024-09-21T14:03:33.148 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3600000000000002e-07 2024-09-21T14:03:33.148 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.148 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.148 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:03:33.148 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.653706+0000", 2024-09-21T14:03:33.149 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6490000000000001e-06 2024-09-21T14:03:33.149 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.149 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.149 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:03:33.149 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.653716+0000", 2024-09-21T14:03:33.149 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0509999999999999e-05 2024-09-21T14:03:33.149 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.149 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.149 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:03:33.150 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.653721+0000", 2024-09-21T14:03:33.150 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1279999999999999e-06 2024-09-21T14:03:33.150 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.150 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.150 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:03:33.150 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.653764+0000", 2024-09-21T14:03:33.150 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2531999999999999e-05 2024-09-21T14:03:33.150 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.150 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.150 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:03:33.150 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.653848+0000", 2024-09-21T14:03:33.150 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4091000000000005e-05 2024-09-21T14:03:33.150 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.151 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.151 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,5", 2024-09-21T14:03:33.151 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.653914+0000", 2024-09-21T14:03:33.151 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5977999999999997e-05 2024-09-21T14:03:33.151 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.151 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.151 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:03:33.151 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.656207+0000", 2024-09-21T14:03:33.151 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0022932690000000001 2024-09-21T14:03:33.151 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.151 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.151 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:33.151 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.659356+0000", 2024-09-21T14:03:33.151 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0031493160000000001 2024-09-21T14:03:33.151 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.152 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.152 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:33.152 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.882645+0000", 2024-09-21T14:03:33.152 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.22328900700000001 2024-09-21T14:03:33.152 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.152 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.152 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:03:33.152 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.882706+0000", 2024-09-21T14:03:33.152 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0279999999999999e-05 2024-09-21T14:03:33.152 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.152 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.152 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:03:33.152 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.882754+0000", 2024-09-21T14:03:33.152 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7870000000000001e-05 2024-09-21T14:03:33.152 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:03:33.152 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:03:33.153 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:03:33.153 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.153 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.153 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.15207.0:232036 8.2 8:4507ec93:::benchmark_data_smithi049_109311_object105517:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T14:03:33.153 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:57:11.656238+0000", 2024-09-21T14:03:33.153 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 381.46343662300001, 2024-09-21T14:03:33.153 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.22665437799999999, 2024-09-21T14:03:33.153 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:03:33.153 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:03:33.153 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:03:33.153 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:03:33.154 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.15207", 2024-09-21T14:03:33.154 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:03:33.154 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 232036 2024-09-21T14:03:33.154 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.154 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:03:33.154 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.154 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:03:33.154 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.656238+0000", 2024-09-21T14:03:33.154 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:03:33.154 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.154 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.154 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:03:33.154 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.656238+0000", 2024-09-21T14:03:33.154 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:03:33.154 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.155 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.155 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:03:33.155 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.656239+0000", 2024-09-21T14:03:33.155 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9100000000000003e-07 2024-09-21T14:03:33.155 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.155 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.155 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:03:33.155 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.656243+0000", 2024-09-21T14:03:33.155 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1230000000000002e-06 2024-09-21T14:03:33.155 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.155 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.155 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:03:33.155 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.656252+0000", 2024-09-21T14:03:33.155 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7490000000000004e-06 2024-09-21T14:03:33.155 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.156 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.156 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:03:33.156 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.656259+0000", 2024-09-21T14:03:33.156 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7559999999999999e-06 2024-09-21T14:03:33.156 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.156 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.156 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:03:33.156 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.656310+0000", 2024-09-21T14:03:33.156 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0203e-05 2024-09-21T14:03:33.156 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.156 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.156 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:03:33.156 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.656440+0000", 2024-09-21T14:03:33.156 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000130376 2024-09-21T14:03:33.156 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.157 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.157 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,5", 2024-09-21T14:03:33.157 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.656550+0000", 2024-09-21T14:03:33.157 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000109812 2024-09-21T14:03:33.157 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.157 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.157 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:03:33.157 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.658548+0000", 2024-09-21T14:03:33.157 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0019982649999999999 2024-09-21T14:03:33.157 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.157 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.157 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:33.157 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.659437+0000", 2024-09-21T14:03:33.157 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000888743 2024-09-21T14:03:33.157 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.158 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.158 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:33.158 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.882792+0000", 2024-09-21T14:03:33.158 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.22335466200000001 2024-09-21T14:03:33.158 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.158 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.158 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:03:33.158 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.882858+0000", 2024-09-21T14:03:33.158 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6512000000000003e-05 2024-09-21T14:03:33.158 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.158 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.158 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:03:33.158 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.882892+0000", 2024-09-21T14:03:33.158 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4385999999999999e-05 2024-09-21T14:03:33.159 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:03:33.159 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:03:33.159 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:03:33.159 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.159 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.159 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.15207.0:232041 8.2 8:42fefff6:::benchmark_data_smithi049_109311_object105522:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T14:03:33.159 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:57:11.674073+0000", 2024-09-21T14:03:33.159 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 381.44560185500001, 2024-09-21T14:03:33.159 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21533840400000001, 2024-09-21T14:03:33.159 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:03:33.159 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:03:33.159 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:03:33.159 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:03:33.159 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.15207", 2024-09-21T14:03:33.159 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:03:33.159 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 232041 2024-09-21T14:03:33.160 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.160 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:03:33.160 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.160 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:03:33.160 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.674073+0000", 2024-09-21T14:03:33.160 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:03:33.160 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.160 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.160 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:03:33.160 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.674073+0000", 2024-09-21T14:03:33.160 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:03:33.160 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.160 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.160 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:03:33.161 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.674074+0000", 2024-09-21T14:03:33.161 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5889999999999999e-06 2024-09-21T14:03:33.161 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.161 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.161 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:03:33.161 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.674080+0000", 2024-09-21T14:03:33.161 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1290000000000004e-06 2024-09-21T14:03:33.161 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.161 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.161 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:03:33.161 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.674102+0000", 2024-09-21T14:03:33.161 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2541e-05 2024-09-21T14:03:33.161 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.161 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.161 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:03:33.161 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.674112+0000", 2024-09-21T14:03:33.162 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.5310000000000007e-06 2024-09-21T14:03:33.162 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.162 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.162 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:03:33.162 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.674234+0000", 2024-09-21T14:03:33.162 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012247000000000001 2024-09-21T14:03:33.162 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.162 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.162 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:03:33.162 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.674365+0000", 2024-09-21T14:03:33.162 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013083300000000001 2024-09-21T14:03:33.162 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.163 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.163 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,5", 2024-09-21T14:03:33.163 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.674489+0000", 2024-09-21T14:03:33.163 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000124079 2024-09-21T14:03:33.163 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.163 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.163 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:03:33.163 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.680300+0000", 2024-09-21T14:03:33.163 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0058106709999999999 2024-09-21T14:03:33.163 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.163 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.163 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:33.163 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.681669+0000", 2024-09-21T14:03:33.164 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0013690460000000001 2024-09-21T14:03:33.164 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.164 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.164 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:33.164 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.889300+0000", 2024-09-21T14:03:33.164 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20763156999999999 2024-09-21T14:03:33.164 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.164 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.164 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:03:33.164 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.889364+0000", 2024-09-21T14:03:33.164 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3979000000000005e-05 2024-09-21T14:03:33.164 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.165 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.165 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:03:33.165 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.889411+0000", 2024-09-21T14:03:33.165 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6965999999999997e-05 2024-09-21T14:03:33.165 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:03:33.165 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:03:33.165 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:03:33.165 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.165 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.165 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.15207.0:232049 8.5 8:aa476d15:::benchmark_data_smithi049_109311_object105530:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T14:03:33.165 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:57:11.686170+0000", 2024-09-21T14:03:33.166 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 381.43350489400001, 2024-09-21T14:03:33.166 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20563606700000001, 2024-09-21T14:03:33.166 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:03:33.166 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:03:33.166 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:03:33.166 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:03:33.166 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.15207", 2024-09-21T14:03:33.166 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:03:33.166 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 232049 2024-09-21T14:03:33.166 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.166 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:03:33.167 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.167 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:03:33.167 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.686170+0000", 2024-09-21T14:03:33.167 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:03:33.167 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.167 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.167 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:03:33.167 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.686170+0000", 2024-09-21T14:03:33.167 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:03:33.167 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.167 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.167 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:03:33.167 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.686172+0000", 2024-09-21T14:03:33.167 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7710000000000001e-06 2024-09-21T14:03:33.168 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.168 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.168 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:03:33.168 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.686175+0000", 2024-09-21T14:03:33.168 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6239999999999999e-06 2024-09-21T14:03:33.168 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.168 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.168 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:03:33.168 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.686185+0000", 2024-09-21T14:03:33.168 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6909999999999992e-06 2024-09-21T14:03:33.168 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.168 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.168 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:03:33.169 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.686194+0000", 2024-09-21T14:03:33.169 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0769999999999997e-06 2024-09-21T14:03:33.169 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.169 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.169 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:03:33.169 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.686258+0000", 2024-09-21T14:03:33.169 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4239000000000006e-05 2024-09-21T14:03:33.169 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.169 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.169 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:03:33.169 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.686374+0000", 2024-09-21T14:03:33.170 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000116148 2024-09-21T14:03:33.170 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.170 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.170 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:03:33.170 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.686491+0000", 2024-09-21T14:03:33.170 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011667000000000001 2024-09-21T14:03:33.170 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.170 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.170 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:03:33.170 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.690120+0000", 2024-09-21T14:03:33.170 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003628797 2024-09-21T14:03:33.170 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.170 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.171 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:33.171 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.690188+0000", 2024-09-21T14:03:33.171 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8201000000000007e-05 2024-09-21T14:03:33.171 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.171 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.171 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:33.171 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.891730+0000", 2024-09-21T14:03:33.171 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20154166900000001 2024-09-21T14:03:33.171 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.171 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.171 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:03:33.171 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.891769+0000", 2024-09-21T14:03:33.171 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9409999999999997e-05 2024-09-21T14:03:33.171 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.172 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.172 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:03:33.172 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.891806+0000", 2024-09-21T14:03:33.172 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6770000000000002e-05 2024-09-21T14:03:33.172 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:03:33.172 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:03:33.172 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:03:33.172 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.172 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.172 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.15207.0:232052 8.5 8:aec88863:::benchmark_data_smithi049_109311_object105533:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T14:03:33.172 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:57:11.686395+0000", 2024-09-21T14:03:33.172 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 381.43327965200001, 2024-09-21T14:03:33.172 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20789333199999999, 2024-09-21T14:03:33.172 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:03:33.172 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:03:33.172 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:03:33.173 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:03:33.173 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.15207", 2024-09-21T14:03:33.173 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:03:33.173 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 232052 2024-09-21T14:03:33.173 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.173 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:03:33.173 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.173 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:03:33.173 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.686395+0000", 2024-09-21T14:03:33.173 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:03:33.173 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.173 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.173 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:03:33.173 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.686395+0000", 2024-09-21T14:03:33.173 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:03:33.174 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.174 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.174 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:03:33.174 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.686396+0000", 2024-09-21T14:03:33.174 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4600000000000004e-07 2024-09-21T14:03:33.174 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.174 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.174 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:03:33.174 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.686399+0000", 2024-09-21T14:03:33.174 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8329999999999998e-06 2024-09-21T14:03:33.174 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.174 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.174 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:03:33.174 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.686404+0000", 2024-09-21T14:03:33.175 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8220000000000003e-06 2024-09-21T14:03:33.175 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.175 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.175 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:03:33.175 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.686408+0000", 2024-09-21T14:03:33.175 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0040000000000001e-06 2024-09-21T14:03:33.175 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.175 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.175 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:03:33.175 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.688420+0000", 2024-09-21T14:03:33.175 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.002011896 2024-09-21T14:03:33.175 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.175 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.175 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:03:33.175 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.688526+0000", 2024-09-21T14:03:33.176 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000106031 2024-09-21T14:03:33.176 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.176 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.176 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:03:33.176 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.688626+0000", 2024-09-21T14:03:33.176 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000100185 2024-09-21T14:03:33.176 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.176 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.176 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:03:33.176 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.690125+0000", 2024-09-21T14:03:33.176 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0014986820000000001 2024-09-21T14:03:33.176 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.176 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.176 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:33.176 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.690262+0000", 2024-09-21T14:03:33.177 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000136444 2024-09-21T14:03:33.177 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.177 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.178 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:33.178 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.894191+0000", 2024-09-21T14:03:33.178 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.203929104 2024-09-21T14:03:33.178 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.178 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.178 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:03:33.178 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.894251+0000", 2024-09-21T14:03:33.178 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9868999999999999e-05 2024-09-21T14:03:33.178 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.178 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.178 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:03:33.178 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.894288+0000", 2024-09-21T14:03:33.178 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7815999999999998e-05 2024-09-21T14:03:33.179 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:03:33.179 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:03:33.179 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:03:33.179 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.179 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.179 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.15207.0:232055 8.2 8:459633e6:::benchmark_data_smithi049_109311_object105536:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T14:03:33.179 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:57:11.693418+0000", 2024-09-21T14:03:33.179 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 381.42625699899997, 2024-09-21T14:03:33.179 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.19827633, 2024-09-21T14:03:33.179 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:03:33.179 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:03:33.179 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:03:33.179 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:03:33.179 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.15207", 2024-09-21T14:03:33.180 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:03:33.180 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 232055 2024-09-21T14:03:33.180 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.180 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:03:33.180 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.180 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:03:33.180 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.693418+0000", 2024-09-21T14:03:33.180 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:03:33.181 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.181 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.181 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:03:33.181 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.693418+0000", 2024-09-21T14:03:33.181 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:03:33.181 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.181 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.182 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:03:33.182 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.693419+0000", 2024-09-21T14:03:33.182 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2100000000000001e-06 2024-09-21T14:03:33.182 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.182 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.182 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:03:33.182 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.693423+0000", 2024-09-21T14:03:33.182 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4669999999999998e-06 2024-09-21T14:03:33.182 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.182 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.182 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:03:33.182 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.693433+0000", 2024-09-21T14:03:33.182 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.8120000000000002e-06 2024-09-21T14:03:33.182 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.183 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.183 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:03:33.183 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.693438+0000", 2024-09-21T14:03:33.183 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2660000000000001e-06 2024-09-21T14:03:33.183 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.183 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.183 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:03:33.183 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.693476+0000", 2024-09-21T14:03:33.183 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7276e-05 2024-09-21T14:03:33.183 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.183 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.183 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:03:33.183 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.693549+0000", 2024-09-21T14:03:33.183 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3133999999999995e-05 2024-09-21T14:03:33.184 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.184 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.184 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,5", 2024-09-21T14:03:33.184 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.693612+0000", 2024-09-21T14:03:33.184 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3392000000000006e-05 2024-09-21T14:03:33.184 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.184 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.184 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:03:33.184 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.695497+0000", 2024-09-21T14:03:33.184 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0018845310000000001 2024-09-21T14:03:33.184 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.184 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.184 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:33.184 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.695811+0000", 2024-09-21T14:03:33.184 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00031459700000000001 2024-09-21T14:03:33.185 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.185 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.185 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:33.185 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.891590+0000", 2024-09-21T14:03:33.185 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.19577844999999999 2024-09-21T14:03:33.185 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.185 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.185 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:03:33.185 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.891645+0000", 2024-09-21T14:03:33.185 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4889999999999998e-05 2024-09-21T14:03:33.185 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.185 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.185 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:03:33.185 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.891694+0000", 2024-09-21T14:03:33.185 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9305000000000003e-05 2024-09-21T14:03:33.186 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:03:33.186 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:03:33.186 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:03:33.186 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.186 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.186 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:73960 9.e 9:7e61c187:::benchmark_data_smithi049_138504_object73959:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-09-21T14:03:33.186 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:58:52.916953+0000", 2024-09-21T14:03:33.186 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 280.202722145, 2024-09-21T14:03:33.186 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.12133540700000001, 2024-09-21T14:03:33.186 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:03:33.186 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:03:33.186 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:03:33.186 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:03:33.186 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:03:33.187 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:03:33.187 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 73960 2024-09-21T14:03:33.187 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.187 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:03:33.187 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.187 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:03:33.187 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:52.916953+0000", 2024-09-21T14:03:33.187 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:03:33.187 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.187 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.187 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:03:33.187 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:52.916953+0000", 2024-09-21T14:03:33.187 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:03:33.187 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.187 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.188 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:03:33.188 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:52.916964+0000", 2024-09-21T14:03:33.188 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1386000000000001e-05 2024-09-21T14:03:33.188 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.188 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.188 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:03:33.188 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:52.917074+0000", 2024-09-21T14:03:33.188 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010969299999999999 2024-09-21T14:03:33.189 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.189 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.189 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:03:33.189 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:52.917094+0000", 2024-09-21T14:03:33.189 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0251e-05 2024-09-21T14:03:33.189 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.190 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.190 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:03:33.190 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:52.917102+0000", 2024-09-21T14:03:33.190 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3000000000000002e-06 2024-09-21T14:03:33.190 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.190 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.190 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:03:33.190 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:52.917160+0000", 2024-09-21T14:03:33.190 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7405999999999999e-05 2024-09-21T14:03:33.190 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.190 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.190 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:03:33.191 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:52.917308+0000", 2024-09-21T14:03:33.191 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000148085 2024-09-21T14:03:33.191 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.191 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.191 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1", 2024-09-21T14:03:33.191 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:52.917467+0000", 2024-09-21T14:03:33.191 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015944200000000001 2024-09-21T14:03:33.191 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.191 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.191 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:03:33.191 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:52.919847+0000", 2024-09-21T14:03:33.191 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0023794250000000001 2024-09-21T14:03:33.191 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.191 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.191 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:33.192 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:53.038132+0000", 2024-09-21T14:03:33.192 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.11828538099999999 2024-09-21T14:03:33.192 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.192 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.192 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:03:33.192 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:53.038197+0000", 2024-09-21T14:03:33.192 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4744000000000002e-05 2024-09-21T14:03:33.192 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.192 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.192 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:03:33.192 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:53.038288+0000", 2024-09-21T14:03:33.192 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1293999999999997e-05 2024-09-21T14:03:33.192 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:03:33.192 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:03:33.192 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:03:33.193 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.193 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.193 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:25489 10.4 10:255de292:::benchmark_data_smithi049_176811_object25488:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e388)", 2024-09-21T14:03:33.193 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:03:14.743046+0000", 2024-09-21T14:03:33.193 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 18.376628849999999, 2024-09-21T14:03:33.193 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.166022157, 2024-09-21T14:03:33.193 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:03:33.193 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:03:33.193 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:03:33.193 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:03:33.193 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:03:33.193 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:03:33.193 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 25489 2024-09-21T14:03:33.193 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.194 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:03:33.194 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.194 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:03:33.194 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:14.743046+0000", 2024-09-21T14:03:33.194 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:03:33.194 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.194 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.194 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:03:33.194 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:14.743046+0000", 2024-09-21T14:03:33.194 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:03:33.194 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.194 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.194 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:03:33.194 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:14.743047+0000", 2024-09-21T14:03:33.195 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.409e-06 2024-09-21T14:03:33.195 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.195 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.195 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:03:33.195 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:14.743084+0000", 2024-09-21T14:03:33.195 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7015999999999999e-05 2024-09-21T14:03:33.195 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.195 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.195 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:03:33.195 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:14.743099+0000", 2024-09-21T14:03:33.195 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4929000000000001e-05 2024-09-21T14:03:33.195 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.195 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.195 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:03:33.195 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:14.743107+0000", 2024-09-21T14:03:33.196 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4760000000000001e-06 2024-09-21T14:03:33.196 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.196 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.196 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:03:33.196 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:14.743269+0000", 2024-09-21T14:03:33.196 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00016269 2024-09-21T14:03:33.196 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.196 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.196 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:03:33.196 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:14.743412+0000", 2024-09-21T14:03:33.196 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000142826 2024-09-21T14:03:33.196 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.196 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.196 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3", 2024-09-21T14:03:33.197 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:14.743537+0000", 2024-09-21T14:03:33.197 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012446000000000001 2024-09-21T14:03:33.197 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.197 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.197 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:03:33.197 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:14.744832+0000", 2024-09-21T14:03:33.197 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0012954920000000001 2024-09-21T14:03:33.197 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.197 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.197 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:33.197 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:14.746895+0000", 2024-09-21T14:03:33.197 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0020632710000000002 2024-09-21T14:03:33.197 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.197 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.197 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:33.198 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:14.908974+0000", 2024-09-21T14:03:33.198 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.16207866200000001 2024-09-21T14:03:33.198 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.198 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.198 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:03:33.198 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:14.909035+0000", 2024-09-21T14:03:33.198 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1138999999999999e-05 2024-09-21T14:03:33.198 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.198 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.198 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:03:33.198 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:14.909068+0000", 2024-09-21T14:03:33.198 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2787000000000003e-05 2024-09-21T14:03:33.198 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:03:33.199 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:03:33.199 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:03:33.199 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.199 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.199 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:25497 10.4 10:297fffc1:::benchmark_data_smithi049_176811_object25496:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e388)", 2024-09-21T14:03:33.199 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:03:14.745399+0000", 2024-09-21T14:03:33.199 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 18.374276130999998, 2024-09-21T14:03:33.199 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.16377804100000001, 2024-09-21T14:03:33.199 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:03:33.199 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:03:33.199 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:03:33.199 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:03:33.199 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:03:33.199 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:03:33.199 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 25497 2024-09-21T14:03:33.200 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.200 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:03:33.200 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.200 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:03:33.200 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:14.745399+0000", 2024-09-21T14:03:33.200 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:03:33.200 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.200 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.200 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:03:33.200 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:14.745399+0000", 2024-09-21T14:03:33.200 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:03:33.200 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.200 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.200 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:03:33.200 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:14.745400+0000", 2024-09-21T14:03:33.200 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7e-06 2024-09-21T14:03:33.201 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.201 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.201 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:03:33.201 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:14.745435+0000", 2024-09-21T14:03:33.201 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4690999999999997e-05 2024-09-21T14:03:33.201 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.201 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.201 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:03:33.201 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:14.745451+0000", 2024-09-21T14:03:33.201 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6184e-05 2024-09-21T14:03:33.201 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.201 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.201 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:03:33.201 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:14.745459+0000", 2024-09-21T14:03:33.201 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9240000000000007e-06 2024-09-21T14:03:33.201 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.202 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.202 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:03:33.202 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:14.745510+0000", 2024-09-21T14:03:33.202 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1396000000000001e-05 2024-09-21T14:03:33.202 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.202 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.202 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:03:33.202 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:14.745638+0000", 2024-09-21T14:03:33.202 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012744399999999999 2024-09-21T14:03:33.202 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.202 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.202 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3", 2024-09-21T14:03:33.202 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:14.745824+0000", 2024-09-21T14:03:33.202 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000186186 2024-09-21T14:03:33.203 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.203 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.203 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:03:33.203 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:14.747155+0000", 2024-09-21T14:03:33.203 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001330695 2024-09-21T14:03:33.203 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.203 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.203 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:33.203 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:14.748710+0000", 2024-09-21T14:03:33.203 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0015550710000000001 2024-09-21T14:03:33.203 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.203 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.203 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:33.203 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:14.909113+0000", 2024-09-21T14:03:33.204 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.160403184 2024-09-21T14:03:33.204 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.204 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.204 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:03:33.204 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:14.909149+0000", 2024-09-21T14:03:33.204 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6192000000000002e-05 2024-09-21T14:03:33.204 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.204 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.204 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:03:33.204 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:14.909177+0000", 2024-09-21T14:03:33.204 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7373999999999999e-05 2024-09-21T14:03:33.204 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:03:33.204 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:03:33.204 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:03:33.204 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.205 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.205 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:25500 10.4 10:2097fbf4:::benchmark_data_smithi049_176811_object25499:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e388)", 2024-09-21T14:03:33.205 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:03:14.745798+0000", 2024-09-21T14:03:33.205 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 18.373876783, 2024-09-21T14:03:33.205 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.163479705, 2024-09-21T14:03:33.205 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:03:33.205 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:03:33.205 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:03:33.205 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:03:33.205 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:03:33.205 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:03:33.205 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 25500 2024-09-21T14:03:33.205 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.205 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:03:33.205 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.206 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:03:33.206 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:14.745798+0000", 2024-09-21T14:03:33.206 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:03:33.206 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.206 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.206 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:03:33.206 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:14.745798+0000", 2024-09-21T14:03:33.206 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:03:33.206 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.206 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.206 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:03:33.206 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:14.745799+0000", 2024-09-21T14:03:33.206 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7700000000000004e-07 2024-09-21T14:03:33.206 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.207 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.207 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:03:33.207 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:14.745842+0000", 2024-09-21T14:03:33.207 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3884999999999997e-05 2024-09-21T14:03:33.207 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.207 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.207 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:03:33.207 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:14.745854+0000", 2024-09-21T14:03:33.207 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1093e-05 2024-09-21T14:03:33.207 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.207 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.207 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:03:33.207 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:14.745860+0000", 2024-09-21T14:03:33.208 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2859999999999996e-06 2024-09-21T14:03:33.208 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.208 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.208 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:03:33.208 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:14.747190+0000", 2024-09-21T14:03:33.208 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0013304600000000001 2024-09-21T14:03:33.208 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.208 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.208 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:03:33.208 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:14.747320+0000", 2024-09-21T14:03:33.208 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012929900000000001 2024-09-21T14:03:33.209 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.209 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.209 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3", 2024-09-21T14:03:33.209 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:14.747522+0000", 2024-09-21T14:03:33.209 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000202425 2024-09-21T14:03:33.209 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.209 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.209 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:03:33.209 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:14.748875+0000", 2024-09-21T14:03:33.209 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0013527050000000001 2024-09-21T14:03:33.209 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.209 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.209 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:33.209 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:14.748954+0000", 2024-09-21T14:03:33.210 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9065000000000003e-05 2024-09-21T14:03:33.210 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.210 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.210 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:33.210 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:14.909209+0000", 2024-09-21T14:03:33.210 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.160255587 2024-09-21T14:03:33.210 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.210 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.210 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:03:33.211 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:14.909245+0000", 2024-09-21T14:03:33.211 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5544999999999999e-05 2024-09-21T14:03:33.211 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.211 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.211 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:03:33.211 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:14.909278+0000", 2024-09-21T14:03:33.211 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2678000000000001e-05 2024-09-21T14:03:33.211 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:03:33.211 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:03:33.211 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:03:33.211 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.211 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.211 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:32192 10.e 10:7692c258:::benchmark_data_smithi049_176811_object32191:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e388)", 2024-09-21T14:03:33.212 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:03:17.954917+0000", 2024-09-21T14:03:33.212 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 15.164757218, 2024-09-21T14:03:33.212 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.142285729, 2024-09-21T14:03:33.212 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:03:33.212 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:03:33.212 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:03:33.212 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:03:33.212 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:03:33.212 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:03:33.212 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 32192 2024-09-21T14:03:33.212 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.212 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:03:33.212 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.212 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:03:33.212 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:17.954917+0000", 2024-09-21T14:03:33.213 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:03:33.213 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.213 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.213 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:03:33.213 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:17.954917+0000", 2024-09-21T14:03:33.213 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:03:33.213 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.213 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.213 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:03:33.213 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:17.954918+0000", 2024-09-21T14:03:33.213 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0699999999999996e-07 2024-09-21T14:03:33.213 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.213 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.213 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:03:33.213 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:17.954965+0000", 2024-09-21T14:03:33.213 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6718999999999998e-05 2024-09-21T14:03:33.214 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.214 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.214 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:03:33.214 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:17.954973+0000", 2024-09-21T14:03:33.214 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.994e-06 2024-09-21T14:03:33.214 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.214 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.214 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:03:33.214 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:17.954978+0000", 2024-09-21T14:03:33.214 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9520000000000002e-06 2024-09-21T14:03:33.214 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.214 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.214 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:03:33.214 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:17.955018+0000", 2024-09-21T14:03:33.214 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9864999999999999e-05 2024-09-21T14:03:33.215 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.215 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.215 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:03:33.215 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:17.955127+0000", 2024-09-21T14:03:33.215 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010930299999999999 2024-09-21T14:03:33.215 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.215 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.215 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,6", 2024-09-21T14:03:33.215 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:17.955226+0000", 2024-09-21T14:03:33.215 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.8728999999999996e-05 2024-09-21T14:03:33.215 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.215 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.215 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:03:33.215 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:17.956307+0000", 2024-09-21T14:03:33.215 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0010813649999999999 2024-09-21T14:03:33.215 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.216 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.216 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:33.216 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:18.094584+0000", 2024-09-21T14:03:33.216 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13827711100000001 2024-09-21T14:03:33.216 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.216 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.216 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:33.216 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:18.097110+0000", 2024-09-21T14:03:33.216 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0025258979999999999 2024-09-21T14:03:33.216 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.216 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.216 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:03:33.216 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:18.097165+0000", 2024-09-21T14:03:33.216 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5161000000000003e-05 2024-09-21T14:03:33.216 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.217 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.217 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:03:33.217 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:18.097203+0000", 2024-09-21T14:03:33.217 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7824999999999998e-05 2024-09-21T14:03:33.217 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:03:33.217 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:03:33.217 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:03:33.217 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.217 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.217 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.16068.0:32194 10.2 e388/381 10:4ce3fa9c:::benchmark_data_smithi049_176811_object32193:head v 388'2044, mlcod=388'2042)", 2024-09-21T14:03:33.217 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:03:17.956584+0000", 2024-09-21T14:03:33.217 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 15.163090949000001, 2024-09-21T14:03:33.217 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.139027913, 2024-09-21T14:03:33.217 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:03:33.217 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:03:33.218 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:03:33.218 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:03:33.218 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.218 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:03:33.218 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:17.956584+0000", 2024-09-21T14:03:33.218 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:03:33.218 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.218 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.218 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:03:33.218 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:17.956584+0000", 2024-09-21T14:03:33.218 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:03:33.218 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.218 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.218 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:03:33.219 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:17.956585+0000", 2024-09-21T14:03:33.219 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.192e-06 2024-09-21T14:03:33.219 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.219 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.219 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:03:33.219 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:17.956657+0000", 2024-09-21T14:03:33.219 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1956e-05 2024-09-21T14:03:33.219 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.219 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.220 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:03:33.220 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:17.956665+0000", 2024-09-21T14:03:33.220 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8320000000000006e-06 2024-09-21T14:03:33.220 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:03:33.221 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:33 smithi049 ceph-mon[40489]: 4.2 scrub starts 2024-09-21T14:03:33.221 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:33 smithi049 ceph-mon[40489]: 4.2 scrub ok 2024-09-21T14:03:33.221 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:33 smithi049 ceph-mon[40489]: pgmap v745: 158 pgs: 1 peering, 157 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.7 KiB/s rd, 87 MiB/s wr, 1.40k op/s 2024-09-21T14:03:33.221 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:33 smithi049 ceph-mon[40489]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T14:03:33.221 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:33 smithi049 ceph-mon[40489]: Cluster is now healthy 2024-09-21T14:03:33.222 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:33 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T14:03:33.222 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:33 smithi049 ceph-mon[40489]: osdmap e395: 8 total, 8 up, 8 in 2024-09-21T14:03:33.222 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:33 smithi049 ceph-mon[40489]: 5.13 scrub starts 2024-09-21T14:03:33.222 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:33 smithi049 ceph-mon[40489]: 5.13 scrub ok 2024-09-21T14:03:33.222 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.223 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.223 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:03:33.223 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:17.956669+0000", 2024-09-21T14:03:33.223 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4959999999999998e-06 2024-09-21T14:03:33.223 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.223 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.223 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:03:33.223 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:17.956861+0000", 2024-09-21T14:03:33.223 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00019152199999999999 2024-09-21T14:03:33.224 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.224 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.224 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:03:33.224 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:17.956873+0000", 2024-09-21T14:03:33.224 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2517999999999999e-05 2024-09-21T14:03:33.224 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.224 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.224 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:03:33.224 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:18.095538+0000", 2024-09-21T14:03:33.224 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.138664815 2024-09-21T14:03:33.224 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.224 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.224 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:03:33.224 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:18.095612+0000", 2024-09-21T14:03:33.224 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3582000000000001e-05 2024-09-21T14:03:33.225 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:03:33.225 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:03:33.225 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:03:33.225 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.225 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.225 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.16068.0:32192 10.e e388/381 ondisk, result = 0)", 2024-09-21T14:03:33.225 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:03:17.957324+0000", 2024-09-21T14:03:33.225 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 15.162350819, 2024-09-21T14:03:33.225 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13726892099999999, 2024-09-21T14:03:33.225 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:03:33.225 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:03:33.225 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T14:03:33.226 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:03:33.226 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.226 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:03:33.226 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:17.957324+0000", 2024-09-21T14:03:33.226 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:03:33.226 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.226 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.226 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:03:33.226 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:17.957324+0000", 2024-09-21T14:03:33.227 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:03:33.227 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.227 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.227 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:03:33.227 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:17.957325+0000", 2024-09-21T14:03:33.227 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3099999999999997e-07 2024-09-21T14:03:33.227 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.228 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.228 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:03:33.228 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:17.957329+0000", 2024-09-21T14:03:33.228 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3669999999999999e-06 2024-09-21T14:03:33.228 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.228 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.228 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:03:33.228 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:18.094361+0000", 2024-09-21T14:03:33.228 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.137032501 2024-09-21T14:03:33.228 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.228 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.229 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:03:33.229 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:18.094373+0000", 2024-09-21T14:03:33.229 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1627999999999999e-05 2024-09-21T14:03:33.229 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.229 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.229 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:03:33.229 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:18.094570+0000", 2024-09-21T14:03:33.229 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000196775 2024-09-21T14:03:33.229 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.229 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.229 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:03:33.229 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:18.094578+0000", 2024-09-21T14:03:33.230 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0930000000000003e-06 2024-09-21T14:03:33.230 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.230 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.230 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:03:33.230 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:18.094593+0000", 2024-09-21T14:03:33.230 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4826e-05 2024-09-21T14:03:33.230 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:03:33.230 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:03:33.230 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:03:33.230 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.231 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.231 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:32197 10.4 10:245f1f7d:::benchmark_data_smithi049_176811_object32196:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e388)", 2024-09-21T14:03:33.231 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:03:17.958250+0000", 2024-09-21T14:03:33.231 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 15.161424212, 2024-09-21T14:03:33.231 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.14462983300000001, 2024-09-21T14:03:33.231 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:03:33.231 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:03:33.231 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:03:33.231 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:03:33.232 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:03:33.232 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:03:33.232 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 32197 2024-09-21T14:03:33.232 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.232 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:03:33.232 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.232 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:03:33.232 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:17.958250+0000", 2024-09-21T14:03:33.232 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:03:33.233 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.233 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.233 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:03:33.233 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:17.958250+0000", 2024-09-21T14:03:33.233 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:03:33.233 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.233 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.233 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:03:33.234 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:17.958251+0000", 2024-09-21T14:03:33.234 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.4300000000000001e-07 2024-09-21T14:03:33.234 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.234 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.234 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:03:33.234 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:17.958310+0000", 2024-09-21T14:03:33.234 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8949000000000001e-05 2024-09-21T14:03:33.234 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.234 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.234 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:03:33.234 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:18.093999+0000", 2024-09-21T14:03:33.234 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13568814400000001 2024-09-21T14:03:33.234 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.234 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.235 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:03:33.235 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:18.094012+0000", 2024-09-21T14:03:33.235 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.382e-05 2024-09-21T14:03:33.235 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.235 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.235 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:03:33.235 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:18.094521+0000", 2024-09-21T14:03:33.235 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00050858000000000001 2024-09-21T14:03:33.235 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.235 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.235 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:03:33.235 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:18.094661+0000", 2024-09-21T14:03:33.235 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000140283 2024-09-21T14:03:33.235 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.235 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.236 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3", 2024-09-21T14:03:33.236 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:18.094800+0000", 2024-09-21T14:03:33.236 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000138739 2024-09-21T14:03:33.236 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.236 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.236 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:03:33.236 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:18.097286+0000", 2024-09-21T14:03:33.236 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0024862479999999999 2024-09-21T14:03:33.236 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.236 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.236 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:33.236 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:18.098728+0000", 2024-09-21T14:03:33.236 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0014418879999999999 2024-09-21T14:03:33.236 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.236 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.236 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:33.237 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:18.102750+0000", 2024-09-21T14:03:33.237 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0040221670000000001 2024-09-21T14:03:33.237 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.237 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.237 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:03:33.237 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:18.102828+0000", 2024-09-21T14:03:33.237 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7471000000000004e-05 2024-09-21T14:03:33.237 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.237 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.237 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:03:33.237 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:18.102880+0000", 2024-09-21T14:03:33.237 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2601e-05 2024-09-21T14:03:33.237 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:03:33.237 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:03:33.237 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:03:33.238 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.238 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.238 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.16068.0:37852 10.e e390/381 ondisk, result = 0)", 2024-09-21T14:03:33.238 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:03:21.118888+0000", 2024-09-21T14:03:33.238 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 12.000786808000001, 2024-09-21T14:03:33.238 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.117971009, 2024-09-21T14:03:33.238 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:03:33.238 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:03:33.238 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T14:03:33.239 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:03:33.239 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.239 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:03:33.239 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:21.118888+0000", 2024-09-21T14:03:33.239 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:03:33.239 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.239 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.239 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:03:33.239 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:21.118888+0000", 2024-09-21T14:03:33.239 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:03:33.239 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.240 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.240 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:03:33.240 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:21.118888+0000", 2024-09-21T14:03:33.240 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8500000000000001e-07 2024-09-21T14:03:33.240 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.240 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.240 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:03:33.240 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:21.118891+0000", 2024-09-21T14:03:33.240 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9459999999999998e-06 2024-09-21T14:03:33.240 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.240 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.240 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:03:33.240 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:21.236632+0000", 2024-09-21T14:03:33.241 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.117740419 2024-09-21T14:03:33.241 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.241 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.241 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:03:33.241 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:21.236640+0000", 2024-09-21T14:03:33.241 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.259e-06 2024-09-21T14:03:33.241 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.241 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.241 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:03:33.241 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:21.236826+0000", 2024-09-21T14:03:33.241 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00018583799999999999 2024-09-21T14:03:33.241 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.241 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.241 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:03:33.242 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:21.236836+0000", 2024-09-21T14:03:33.242 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6579999999999997e-06 2024-09-21T14:03:33.242 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.242 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.242 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:03:33.242 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:21.236859+0000", 2024-09-21T14:03:33.242 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3303999999999999e-05 2024-09-21T14:03:33.242 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:03:33.242 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:03:33.242 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:03:33.242 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.242 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.242 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:49739 10.e 10:71726312:::benchmark_data_smithi049_176811_object49738:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e392)", 2024-09-21T14:03:33.242 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:03:28.592043+0000", 2024-09-21T14:03:33.243 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 4.5276317800000001, 2024-09-21T14:03:33.243 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.12562675300000001, 2024-09-21T14:03:33.243 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:03:33.243 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:03:33.243 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:03:33.243 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:03:33.243 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:03:33.243 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:03:33.243 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 49739 2024-09-21T14:03:33.243 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.244 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:03:33.244 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.244 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:03:33.244 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:28.592043+0000", 2024-09-21T14:03:33.244 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:03:33.244 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.244 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.244 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:03:33.244 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:28.592043+0000", 2024-09-21T14:03:33.244 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:03:33.244 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.244 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.244 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:03:33.244 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:28.592045+0000", 2024-09-21T14:03:33.245 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6479999999999999e-06 2024-09-21T14:03:33.245 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.245 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.245 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:03:33.245 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:28.592088+0000", 2024-09-21T14:03:33.245 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3727999999999999e-05 2024-09-21T14:03:33.245 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.245 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.245 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:03:33.245 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:28.592104+0000", 2024-09-21T14:03:33.245 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5366000000000002e-05 2024-09-21T14:03:33.245 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.245 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.245 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:03:33.246 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:28.592112+0000", 2024-09-21T14:03:33.246 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3429999999999996e-06 2024-09-21T14:03:33.246 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.246 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.246 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:03:33.246 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:28.592174+0000", 2024-09-21T14:03:33.246 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1551999999999996e-05 2024-09-21T14:03:33.246 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.246 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.246 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:03:33.246 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:28.592300+0000", 2024-09-21T14:03:33.246 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012604700000000001 2024-09-21T14:03:33.246 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.246 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.247 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,6", 2024-09-21T14:03:33.247 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:28.592444+0000", 2024-09-21T14:03:33.247 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000144167 2024-09-21T14:03:33.247 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.247 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.247 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:03:33.247 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:28.594522+0000", 2024-09-21T14:03:33.247 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.002078444 2024-09-21T14:03:33.247 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.247 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.247 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:33.247 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:28.595052+0000", 2024-09-21T14:03:33.247 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00052941399999999997 2024-09-21T14:03:33.247 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.247 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.248 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:03:33.248 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:28.717567+0000", 2024-09-21T14:03:33.248 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.12251582499999999 2024-09-21T14:03:33.248 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.248 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.248 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:03:33.248 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:28.717625+0000", 2024-09-21T14:03:33.248 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7652000000000003e-05 2024-09-21T14:03:33.248 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:03:33.248 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:03:33.248 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:03:33.248 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:28.717670+0000", 2024-09-21T14:03:33.248 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4567000000000002e-05 2024-09-21T14:03:33.248 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:03:33.249 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:03:33.249 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:03:33.249 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:03:33.249 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:03:33.249 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T14:03:33.282 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:33.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:33.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:03:33.384 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:33.418 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:33.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:33.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:33.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:33 smithi187 ceph-mon[33032]: 4.2 scrub starts 2024-09-21T14:03:33.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:33 smithi187 ceph-mon[33032]: 4.2 scrub ok 2024-09-21T14:03:33.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:33 smithi187 ceph-mon[33032]: pgmap v745: 158 pgs: 1 peering, 157 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.7 KiB/s rd, 87 MiB/s wr, 1.40k op/s 2024-09-21T14:03:33.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:33 smithi187 ceph-mon[33032]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T14:03:33.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:33 smithi187 ceph-mon[33032]: Cluster is now healthy 2024-09-21T14:03:33.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:33 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T14:03:33.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:33 smithi187 ceph-mon[33032]: osdmap e395: 8 total, 8 up, 8 in 2024-09-21T14:03:33.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:33 smithi187 ceph-mon[33032]: 5.13 scrub starts 2024-09-21T14:03:33.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:33 smithi187 ceph-mon[33032]: 5.13 scrub ok 2024-09-21T14:03:33.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:33 smithi049 ceph-mon[30080]: 4.2 scrub starts 2024-09-21T14:03:33.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:33 smithi049 ceph-mon[30080]: 4.2 scrub ok 2024-09-21T14:03:33.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:33 smithi049 ceph-mon[30080]: pgmap v745: 158 pgs: 1 peering, 157 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.7 KiB/s rd, 87 MiB/s wr, 1.40k op/s 2024-09-21T14:03:33.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:33 smithi049 ceph-mon[30080]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T14:03:33.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:33 smithi049 ceph-mon[30080]: Cluster is now healthy 2024-09-21T14:03:33.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:33 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T14:03:33.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:33 smithi049 ceph-mon[30080]: osdmap e395: 8 total, 8 up, 8 in 2024-09-21T14:03:33.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:33 smithi049 ceph-mon[30080]: 5.13 scrub starts 2024-09-21T14:03:33.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:33 smithi049 ceph-mon[30080]: 5.13 scrub ok 2024-09-21T14:03:33.520 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:03:33.559 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:33.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:33.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:03:33.661 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:03:33.719 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:33.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:33.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:03:33.734 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:03:33.755 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:03:33.822 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:03:33.870 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 28 16 57522 57506 128.341 90.9375 0.0078406 0.00778653 2024-09-21T14:03:33.884 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:33.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:33.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:03:33.986 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:34.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:34.022 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:34.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:34.123 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:34.156 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:34.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:34.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:34.258 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:03:34.292 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:34.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:34.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:03:34.394 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:03:34.433 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:34.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:34.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:03:34.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:34 smithi187 ceph-mon[33032]: 3.9 scrub starts 2024-09-21T14:03:34.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:34 smithi187 ceph-mon[33032]: 3.9 scrub ok 2024-09-21T14:03:34.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:34 smithi187 ceph-mon[33032]: 5.19 scrub starts 2024-09-21T14:03:34.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:34 smithi187 ceph-mon[33032]: 5.19 scrub ok 2024-09-21T14:03:34.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:34 smithi187 ceph-mon[33032]: 5.17 scrub starts 2024-09-21T14:03:34.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:34 smithi187 ceph-mon[33032]: 5.17 scrub ok 2024-09-21T14:03:34.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:34 smithi187 ceph-mon[33032]: 6.8 scrub starts 2024-09-21T14:03:34.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:34 smithi187 ceph-mon[33032]: 6.8 scrub ok 2024-09-21T14:03:34.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:34 smithi049 ceph-mon[30080]: 3.9 scrub starts 2024-09-21T14:03:34.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:34 smithi049 ceph-mon[30080]: 3.9 scrub ok 2024-09-21T14:03:34.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:34 smithi049 ceph-mon[30080]: 5.19 scrub starts 2024-09-21T14:03:34.472 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:34 smithi049 ceph-mon[30080]: 5.19 scrub ok 2024-09-21T14:03:34.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:34 smithi049 ceph-mon[30080]: 5.17 scrub starts 2024-09-21T14:03:34.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:34 smithi049 ceph-mon[30080]: 5.17 scrub ok 2024-09-21T14:03:34.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:34 smithi049 ceph-mon[30080]: 6.8 scrub starts 2024-09-21T14:03:34.473 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:34 smithi049 ceph-mon[30080]: 6.8 scrub ok 2024-09-21T14:03:34.473 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:34 smithi049 ceph-mon[40489]: 3.9 scrub starts 2024-09-21T14:03:34.473 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:34 smithi049 ceph-mon[40489]: 3.9 scrub ok 2024-09-21T14:03:34.473 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:34 smithi049 ceph-mon[40489]: 5.19 scrub starts 2024-09-21T14:03:34.473 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:34 smithi049 ceph-mon[40489]: 5.19 scrub ok 2024-09-21T14:03:34.473 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:34 smithi049 ceph-mon[40489]: 5.17 scrub starts 2024-09-21T14:03:34.473 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:34 smithi049 ceph-mon[40489]: 5.17 scrub ok 2024-09-21T14:03:34.473 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:34 smithi049 ceph-mon[40489]: 6.8 scrub starts 2024-09-21T14:03:34.474 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:34 smithi049 ceph-mon[40489]: 6.8 scrub ok 2024-09-21T14:03:34.480 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T14:03:34.535 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:03:34.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:34.573 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:34.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:03:34.657 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:03:34.657 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:03:34.674 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:34.687 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:03:34.687 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:03:34.724 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:03:34.724 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:03:34.739 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:34.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:34.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:34.761 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:03:34.761 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:03:34.798 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:03:34.799 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:03:34.836 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:03:34.836 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:03:34.842 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:34.870 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 29 16 59006 58990 127.114 92.75 0.0112898 0.00786022 2024-09-21T14:03:34.872 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:03:34.872 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:03:34.892 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:34.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:34.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:34.909 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:03:34.909 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:03:34.993 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:03:35.025 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:35.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:35.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:03:35.127 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:35.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:35.170 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:35.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:35.237 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T14:03:35.240 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-09-21T14:03:35.272 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:03:35.296 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:03:35.321 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:35.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:35.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:03:35.424 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:03:35.438 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:35 smithi187 ceph-mon[33032]: pgmap v747: 158 pgs: 158 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 871 B/s rd, 127 MiB/s wr, 2.04k op/s; 18 B/s, 2 objects/s recovering 2024-09-21T14:03:35.438 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:35 smithi187 ceph-mon[33032]: 5.15 scrub starts 2024-09-21T14:03:35.438 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:35 smithi187 ceph-mon[33032]: 5.15 scrub ok 2024-09-21T14:03:35.438 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:35 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2369441395' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:03:35.438 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:35 smithi187 ceph-mon[33032]: 1.6 scrub starts 2024-09-21T14:03:35.438 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:35 smithi187 ceph-mon[33032]: 1.6 scrub ok 2024-09-21T14:03:35.439 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:35 smithi187 ceph-mon[33032]: osdmap e396: 8 total, 8 up, 8 in 2024-09-21T14:03:35.461 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:35.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:35.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:03:35.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:35 smithi049 ceph-mon[40489]: pgmap v747: 158 pgs: 158 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 871 B/s rd, 127 MiB/s wr, 2.04k op/s; 18 B/s, 2 objects/s recovering 2024-09-21T14:03:35.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:35 smithi049 ceph-mon[40489]: 5.15 scrub starts 2024-09-21T14:03:35.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:35 smithi049 ceph-mon[40489]: 5.15 scrub ok 2024-09-21T14:03:35.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:35 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2369441395' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:03:35.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:35 smithi049 ceph-mon[40489]: 1.6 scrub starts 2024-09-21T14:03:35.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:35 smithi049 ceph-mon[40489]: 1.6 scrub ok 2024-09-21T14:03:35.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:35 smithi049 ceph-mon[40489]: osdmap e396: 8 total, 8 up, 8 in 2024-09-21T14:03:35.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:35 smithi049 ceph-mon[30080]: pgmap v747: 158 pgs: 158 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 871 B/s rd, 127 MiB/s wr, 2.04k op/s; 18 B/s, 2 objects/s recovering 2024-09-21T14:03:35.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:35 smithi049 ceph-mon[30080]: 5.15 scrub starts 2024-09-21T14:03:35.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:35 smithi049 ceph-mon[30080]: 5.15 scrub ok 2024-09-21T14:03:35.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:35 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2369441395' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:03:35.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:35 smithi049 ceph-mon[30080]: 1.6 scrub starts 2024-09-21T14:03:35.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:35 smithi049 ceph-mon[30080]: 1.6 scrub ok 2024-09-21T14:03:35.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:35 smithi049 ceph-mon[30080]: osdmap e396: 8 total, 8 up, 8 in 2024-09-21T14:03:35.563 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:03:35.591 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T14:03:35.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:35.601 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:35.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:03:35.702 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:03:35.738 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:35.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:35.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:03:35.840 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:03:35.867 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:35.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:35.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:03:35.870 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 30 16 60424 60408 125.831 88.625 0.00287303 0.00793743 2024-09-21T14:03:35.969 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:03:36.009 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:36.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:36.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:03:36.111 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:03:36.147 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:36.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:36.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:03:36.249 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:03:36.275 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:36.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:36.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:03:36.377 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:03:36.414 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:36.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:36.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:03:36.483 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:36 smithi049 ceph-mon[40489]: from='client.16122 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:03:36.488 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:36 smithi049 ceph-mon[30080]: from='client.16122 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:03:36.516 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:03:36.548 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":748,"stamp":"2024-09-21T14:03:35.115128+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3694056323,"num_objects":56560,"num_object_clones":0,"num_object_copies":169680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56560,"num_whiteouts":0,"num_read":1428,"num_read_kb":1247,"num_write":57156,"num_write_kb":3608185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":1033072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58032,"ondisk_log_size":58032,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":473,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":14001608,"kb_used_data":10618924,"kb_used_omap":23,"kb_used_meta":3382504,"kb_avail":735959608,"statfs":{"total":767960285184,"available":753622638592,"internally_reserved":0,"allocated":10873778176,"data_stored":10866368057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24148,"internal_metadata":3463684524},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,4,7,9,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":525008896,"num_objects":8011,"num_object_clones":0,"num_object_copies":24033,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8011,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":8011,"num_write_kb":512704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.974666"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":612,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.134326+0000","last_change":"2024-09-21T14:02:56.458231+0000","last_active":"2024-09-21T14:03:30.134326+0000","last_peered":"2024-09-21T14:03:30.134326+0000","last_clean":"2024-09-21T14:03:30.134326+0000","last_became_active":"2024-09-21T14:01:41.189398+0000","last_became_peered":"2024-09-21T14:01:41.189398+0000","last_unstale":"2024-09-21T14:03:30.134326+0000","last_undegraded":"2024-09-21T14:03:30.134326+0000","last_fullsized":"2024-09-21T14:03:30.134326+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:56.458125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:02:56.458125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:20.867402+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"290'15","reported_seq":782,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.180519+0000","last_change":"2024-09-21T14:01:34.785322+0000","last_active":"2024-09-21T14:03:32.180519+0000","last_peered":"2024-09-21T14:03:32.180519+0000","last_clean":"2024-09-21T14:03:32.180519+0000","last_became_active":"2024-09-21T13:58:17.771220+0000","last_became_peered":"2024-09-21T13:58:17.771220+0000","last_unstale":"2024-09-21T14:03:32.180519+0000","last_undegraded":"2024-09-21T14:03:32.180519+0000","last_fullsized":"2024-09-21T14:03:32.180519+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'15","last_scrub_stamp":"2024-09-21T14:01:34.785267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:34.785267+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:53.703961+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":730,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:33.301558+0000","last_change":"2024-09-21T14:03:33.301558+0000","last_active":"2024-09-21T14:03:33.301558+0000","last_peered":"2024-09-21T14:03:33.301558+0000","last_clean":"2024-09-21T14:03:33.301558+0000","last_became_active":"2024-09-21T14:01:41.202168+0000","last_became_peered":"2024-09-21T14:01:41.202168+0000","last_unstale":"2024-09-21T14:03:33.301558+0000","last_undegraded":"2024-09-21T14:03:33.301558+0000","last_fullsized":"2024-09-21T14:03:33.301558+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:33.301487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:33.301487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:54.547538+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":698,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.181173+0000","last_change":"2024-09-21T14:01:41.203762+0000","last_active":"2024-09-21T14:03:32.181173+0000","last_peered":"2024-09-21T14:03:32.181173+0000","last_clean":"2024-09-21T14:03:32.181173+0000","last_became_active":"2024-09-21T14:01:41.203275+0000","last_became_peered":"2024-09-21T14:01:41.203275+0000","last_unstale":"2024-09-21T14:03:32.181173+0000","last_undegraded":"2024-09-21T14:03:32.181173+0000","last_fullsized":"2024-09-21T14:03:32.181173+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:00.320510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:00.320510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:46.689692+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":666,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.123827+0000","last_change":"2024-09-21T13:59:32.208701+0000","last_active":"2024-09-21T14:03:30.123827+0000","last_peered":"2024-09-21T14:03:30.123827+0000","last_clean":"2024-09-21T14:03:30.123827+0000","last_became_active":"2024-09-21T13:59:32.203340+0000","last_became_peered":"2024-09-21T13:59:32.203340+0000","last_unstale":"2024-09-21T14:03:30.123827+0000","last_undegraded":"2024-09-21T14:03:30.123827+0000","last_fullsized":"2024-09-21T14:03:30.123827+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:34.696780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:34.696780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:44.274358+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":679,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.124351+0000","last_change":"2024-09-21T13:58:17.778322+0000","last_active":"2024-09-21T14:03:30.124351+0000","last_peered":"2024-09-21T14:03:30.124351+0000","last_clean":"2024-09-21T14:03:30.124351+0000","last_became_active":"2024-09-21T13:58:17.777893+0000","last_became_peered":"2024-09-21T13:58:17.777893+0000","last_unstale":"2024-09-21T14:03:30.124351+0000","last_undegraded":"2024-09-21T14:03:30.124351+0000","last_fullsized":"2024-09-21T14:03:30.124351+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:07.001135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:55:07.001135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"290'27","reported_seq":652,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.123734+0000","last_change":"2024-09-21T13:58:11.718421+0000","last_active":"2024-09-21T14:03:30.123734+0000","last_peered":"2024-09-21T14:03:30.123734+0000","last_clean":"2024-09-21T14:03:30.123734+0000","last_became_active":"2024-09-21T13:57:12.703469+0000","last_became_peered":"2024-09-21T13:57:12.703469+0000","last_unstale":"2024-09-21T14:03:30.123734+0000","last_undegraded":"2024-09-21T14:03:30.123734+0000","last_fullsized":"2024-09-21T14:03:30.123734+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:58:11.718334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:11.718334+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":730,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.180436+0000","last_change":"2024-09-21T14:03:02.862866+0000","last_active":"2024-09-21T14:03:32.180436+0000","last_peered":"2024-09-21T14:03:32.180436+0000","last_clean":"2024-09-21T14:03:32.180436+0000","last_became_active":"2024-09-21T13:58:17.773464+0000","last_became_peered":"2024-09-21T13:58:17.773464+0000","last_unstale":"2024-09-21T14:03:32.180436+0000","last_undegraded":"2024-09-21T14:03:32.180436+0000","last_fullsized":"2024-09-21T14:03:32.180436+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:02.862787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:02.862787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:17.663483+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":671,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.133969+0000","last_change":"2024-09-21T14:01:41.194807+0000","last_active":"2024-09-21T14:03:30.133969+0000","last_peered":"2024-09-21T14:03:30.133969+0000","last_clean":"2024-09-21T14:03:30.133969+0000","last_became_active":"2024-09-21T14:01:41.194517+0000","last_became_peered":"2024-09-21T14:01:41.194517+0000","last_unstale":"2024-09-21T14:03:30.133969+0000","last_undegraded":"2024-09-21T14:03:30.133969+0000","last_fullsized":"2024-09-21T14:03:30.133969+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:31.127926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:31.127926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:10.380240+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"290'15","reported_seq":747,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.181542+0000","last_change":"2024-09-21T14:00:02.475761+0000","last_active":"2024-09-21T14:03:32.181542+0000","last_peered":"2024-09-21T14:03:32.181542+0000","last_clean":"2024-09-21T14:03:32.181542+0000","last_became_active":"2024-09-21T13:57:12.681408+0000","last_became_peered":"2024-09-21T13:57:12.681408+0000","last_unstale":"2024-09-21T14:03:32.181542+0000","last_undegraded":"2024-09-21T14:03:32.181542+0000","last_fullsized":"2024-09-21T14:03:32.181542+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'15","last_scrub_stamp":"2024-09-21T14:00:02.475661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:00:02.475661+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:43.713097+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":620,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.133200+0000","last_change":"2024-09-21T14:03:04.516642+0000","last_active":"2024-09-21T14:03:30.133200+0000","last_peered":"2024-09-21T14:03:30.133200+0000","last_clean":"2024-09-21T14:03:30.133200+0000","last_became_active":"2024-09-21T14:01:41.188248+0000","last_became_peered":"2024-09-21T14:01:41.188248+0000","last_unstale":"2024-09-21T14:03:30.133200+0000","last_undegraded":"2024-09-21T14:03:30.133200+0000","last_fullsized":"2024-09-21T14:03:30.133200+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:04.516583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:04.516583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:26.082827+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":652,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.165030+0000","last_change":"2024-09-21T13:59:32.202128+0000","last_active":"2024-09-21T14:03:32.165030+0000","last_peered":"2024-09-21T14:03:32.165030+0000","last_clean":"2024-09-21T14:03:32.165030+0000","last_became_active":"2024-09-21T13:59:32.201158+0000","last_became_peered":"2024-09-21T13:59:32.201158+0000","last_unstale":"2024-09-21T14:03:32.165030+0000","last_undegraded":"2024-09-21T14:03:32.165030+0000","last_fullsized":"2024-09-21T14:03:32.165030+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:25.792985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:54:25.792985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:49.267653+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":648,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.133179+0000","last_change":"2024-09-21T14:01:41.189797+0000","last_active":"2024-09-21T14:03:30.133179+0000","last_peered":"2024-09-21T14:03:30.133179+0000","last_clean":"2024-09-21T14:03:30.133179+0000","last_became_active":"2024-09-21T14:01:41.189373+0000","last_became_peered":"2024-09-21T14:01:41.189373+0000","last_unstale":"2024-09-21T14:03:30.133179+0000","last_undegraded":"2024-09-21T14:03:30.133179+0000","last_fullsized":"2024-09-21T14:03:30.133179+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:54.328275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:54.328275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:09.403771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"290'7","reported_seq":772,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.179608+0000","last_change":"2024-09-21T14:01:57.563250+0000","last_active":"2024-09-21T14:03:32.179608+0000","last_peered":"2024-09-21T14:03:32.179608+0000","last_clean":"2024-09-21T14:03:32.179608+0000","last_became_active":"2024-09-21T13:59:09.063053+0000","last_became_peered":"2024-09-21T13:59:09.063053+0000","last_unstale":"2024-09-21T14:03:32.179608+0000","last_undegraded":"2024-09-21T14:03:32.179608+0000","last_fullsized":"2024-09-21T14:03:32.179608+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'7","last_scrub_stamp":"2024-09-21T14:01:57.563184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:57.563184+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":641,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.130445+0000","last_change":"2024-09-21T13:59:32.204565+0000","last_active":"2024-09-21T14:03:30.130445+0000","last_peered":"2024-09-21T14:03:30.130445+0000","last_clean":"2024-09-21T14:03:30.130445+0000","last_became_active":"2024-09-21T13:59:32.204174+0000","last_became_peered":"2024-09-21T13:59:32.204174+0000","last_unstale":"2024-09-21T14:03:30.130445+0000","last_undegraded":"2024-09-21T14:03:30.130445+0000","last_fullsized":"2024-09-21T14:03:30.130445+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:06.722293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:06.722293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":682,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.132137+0000","last_change":"2024-09-21T14:01:41.199085+0000","last_active":"2024-09-21T14:03:30.132137+0000","last_peered":"2024-09-21T14:03:30.132137+0000","last_clean":"2024-09-21T14:03:30.132137+0000","last_became_active":"2024-09-21T14:01:41.198672+0000","last_became_peered":"2024-09-21T14:01:41.198672+0000","last_unstale":"2024-09-21T14:03:30.132137+0000","last_undegraded":"2024-09-21T14:03:30.132137+0000","last_fullsized":"2024-09-21T14:03:30.132137+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:44.376215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:44.376215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:23.240484+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":741,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.181705+0000","last_change":"2024-09-21T14:01:53.460257+0000","last_active":"2024-09-21T14:03:32.181705+0000","last_peered":"2024-09-21T14:03:32.181705+0000","last_clean":"2024-09-21T14:03:32.181705+0000","last_became_active":"2024-09-21T14:01:41.200360+0000","last_became_peered":"2024-09-21T14:01:41.200360+0000","last_unstale":"2024-09-21T14:03:32.181705+0000","last_undegraded":"2024-09-21T14:03:32.181705+0000","last_fullsized":"2024-09-21T14:03:32.181705+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:53.460180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:01:53.460180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":659,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.179922+0000","last_change":"2024-09-21T14:02:36.679396+0000","last_active":"2024-09-21T14:03:32.179922+0000","last_peered":"2024-09-21T14:03:32.179922+0000","last_clean":"2024-09-21T14:03:32.179922+0000","last_became_active":"2024-09-21T13:59:32.204749+0000","last_became_peered":"2024-09-21T13:59:32.204749+0000","last_unstale":"2024-09-21T14:03:32.179922+0000","last_undegraded":"2024-09-21T14:03:32.179922+0000","last_fullsized":"2024-09-21T14:03:32.179922+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:36.679321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:02:36.679321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:43.067225+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"290'27","reported_seq":723,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.167885+0000","last_change":"2024-09-21T14:02:55.620755+0000","last_active":"2024-09-21T14:03:32.167885+0000","last_peered":"2024-09-21T14:03:32.167885+0000","last_clean":"2024-09-21T14:03:32.167885+0000","last_became_active":"2024-09-21T13:57:12.680783+0000","last_became_peered":"2024-09-21T13:57:12.680783+0000","last_unstale":"2024-09-21T14:03:32.167885+0000","last_undegraded":"2024-09-21T14:03:32.167885+0000","last_fullsized":"2024-09-21T14:03:32.167885+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'27","last_scrub_stamp":"2024-09-21T14:02:55.620700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.620700+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:01.085594+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":716,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.167000+0000","last_change":"2024-09-21T14:03:08.299319+0000","last_active":"2024-09-21T14:03:32.167000+0000","last_peered":"2024-09-21T14:03:32.167000+0000","last_clean":"2024-09-21T14:03:32.167000+0000","last_became_active":"2024-09-21T13:49:02.066796+0000","last_became_peered":"2024-09-21T13:49:02.066796+0000","last_unstale":"2024-09-21T14:03:32.167000+0000","last_undegraded":"2024-09-21T14:03:32.167000+0000","last_fullsized":"2024-09-21T14:03:32.167000+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:08.299252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:08.299252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:29.982454+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":659,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.123247+0000","last_change":"2024-09-21T14:01:41.191486+0000","last_active":"2024-09-21T14:03:30.123247+0000","last_peered":"2024-09-21T14:03:30.123247+0000","last_clean":"2024-09-21T14:03:30.123247+0000","last_became_active":"2024-09-21T14:01:41.191116+0000","last_became_peered":"2024-09-21T14:01:41.191116+0000","last_unstale":"2024-09-21T14:03:30.123247+0000","last_undegraded":"2024-09-21T14:03:30.123247+0000","last_fullsized":"2024-09-21T14:03:30.123247+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:57.962266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:57.962266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:10.160181+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":718,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.181685+0000","last_change":"2024-09-21T14:00:51.495433+0000","last_active":"2024-09-21T14:03:32.181685+0000","last_peered":"2024-09-21T14:03:32.181685+0000","last_clean":"2024-09-21T14:03:32.181685+0000","last_became_active":"2024-09-21T13:52:04.316016+0000","last_became_peered":"2024-09-21T13:52:04.316016+0000","last_unstale":"2024-09-21T14:03:32.181685+0000","last_undegraded":"2024-09-21T14:03:32.181685+0000","last_fullsized":"2024-09-21T14:03:32.181685+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:00:51.495368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:00:51.495368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:12.877210+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"290'18","reported_seq":664,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.121451+0000","last_change":"2024-09-21T13:59:23.041396+0000","last_active":"2024-09-21T14:03:30.121451+0000","last_peered":"2024-09-21T14:03:30.121451+0000","last_clean":"2024-09-21T14:03:30.121451+0000","last_became_active":"2024-09-21T13:57:12.684454+0000","last_became_peered":"2024-09-21T13:57:12.684454+0000","last_unstale":"2024-09-21T14:03:30.121451+0000","last_undegraded":"2024-09-21T14:03:30.121451+0000","last_fullsized":"2024-09-21T14:03:30.121451+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'18","last_scrub_stamp":"2024-09-21T13:59:23.041311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:23.041311+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":705,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.181968+0000","last_change":"2024-09-21T13:58:16.393763+0000","last_active":"2024-09-21T14:03:32.181968+0000","last_peered":"2024-09-21T14:03:32.181968+0000","last_clean":"2024-09-21T14:03:32.181968+0000","last_became_active":"2024-09-21T13:57:12.681897+0000","last_became_peered":"2024-09-21T13:57:12.681897+0000","last_unstale":"2024-09-21T14:03:32.181968+0000","last_undegraded":"2024-09-21T14:03:32.181968+0000","last_fullsized":"2024-09-21T14:03:32.181968+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:16.393696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:16.393696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:37.708367+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":633,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.130432+0000","last_change":"2024-09-21T14:03:28.933216+0000","last_active":"2024-09-21T14:03:30.130432+0000","last_peered":"2024-09-21T14:03:30.130432+0000","last_clean":"2024-09-21T14:03:30.130432+0000","last_became_active":"2024-09-21T14:01:41.181946+0000","last_became_peered":"2024-09-21T14:01:41.181946+0000","last_unstale":"2024-09-21T14:03:30.130432+0000","last_undegraded":"2024-09-21T14:03:30.130432+0000","last_fullsized":"2024-09-21T14:03:30.130432+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T14:03:28.933143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:28.933143+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:46.365067+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":729,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.164600+0000","last_change":"2024-09-21T14:01:58.994867+0000","last_active":"2024-09-21T14:03:32.164600+0000","last_peered":"2024-09-21T14:03:32.164600+0000","last_clean":"2024-09-21T14:03:32.164600+0000","last_became_active":"2024-09-21T13:57:12.685742+0000","last_became_peered":"2024-09-21T13:57:12.685742+0000","last_unstale":"2024-09-21T14:03:32.164600+0000","last_undegraded":"2024-09-21T14:03:32.164600+0000","last_fullsized":"2024-09-21T14:03:32.164600+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T14:01:58.994811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:01:58.994811+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:25.355083+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"290'20","reported_seq":606,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.132656+0000","last_change":"2024-09-21T13:59:32.242771+0000","last_active":"2024-09-21T14:03:30.132656+0000","last_peered":"2024-09-21T14:03:30.132656+0000","last_clean":"2024-09-21T14:03:30.132656+0000","last_became_active":"2024-09-21T13:59:32.213865+0000","last_became_peered":"2024-09-21T13:59:32.213865+0000","last_unstale":"2024-09-21T14:03:30.132656+0000","last_undegraded":"2024-09-21T14:03:30.132656+0000","last_fullsized":"2024-09-21T14:03:30.132656+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:57:27.578738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:57:27.578738+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"300'12","reported_seq":353,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.177071+0000","last_change":"2024-09-21T13:59:36.012775+0000","last_active":"2024-09-21T14:03:32.177071+0000","last_peered":"2024-09-21T14:03:32.177071+0000","last_clean":"2024-09-21T14:03:32.177071+0000","last_became_active":"2024-09-21T13:59:32.205051+0000","last_became_peered":"2024-09-21T13:59:32.205051+0000","last_unstale":"2024-09-21T14:03:32.177071+0000","last_undegraded":"2024-09-21T14:03:32.177071+0000","last_fullsized":"2024-09-21T14:03:32.177071+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'12","last_scrub_stamp":"2024-09-21T13:59:36.012693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:36.012693+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:21.546366+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"395'3630","reported_seq":3669,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:33.609295+0000","last_change":"2024-09-21T14:02:56.560786+0000","last_active":"2024-09-21T14:03:33.609295+0000","last_peered":"2024-09-21T14:03:33.609295+0000","last_clean":"2024-09-21T14:03:33.609295+0000","last_became_active":"2024-09-21T14:02:56.559536+0000","last_became_peered":"2024-09-21T14:02:56.559536+0000","last_unstale":"2024-09-21T14:03:33.609295+0000","last_undegraded":"2024-09-21T14:03:33.609295+0000","last_fullsized":"2024-09-21T14:03:33.609295+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":3630,"log_dups_size":0,"ondisk_log_size":3630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:03.627467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":237895680,"num_objects":3630,"num_object_clones":0,"num_object_copies":10890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3630,"num_write_kb":232320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"75'79","reported_seq":121,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.133357+0000","last_change":"2024-09-21T14:01:41.248772+0000","last_active":"2024-09-21T14:03:30.133357+0000","last_peered":"2024-09-21T14:03:30.133357+0000","last_clean":"2024-09-21T14:03:30.133357+0000","last_became_active":"2024-09-21T14:01:41.246888+0000","last_became_peered":"2024-09-21T14:01:41.246888+0000","last_unstale":"2024-09-21T14:03:30.133357+0000","last_undegraded":"2024-09-21T14:03:30.133357+0000","last_fullsized":"2024-09-21T14:03:30.133357+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:03:52.423493+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53669,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"290'27","reported_seq":775,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.167343+0000","last_change":"2024-09-21T14:02:58.293228+0000","last_active":"2024-09-21T14:03:32.167343+0000","last_peered":"2024-09-21T14:03:32.167343+0000","last_clean":"2024-09-21T14:03:32.167343+0000","last_became_active":"2024-09-21T13:57:12.695609+0000","last_became_peered":"2024-09-21T13:57:12.695609+0000","last_unstale":"2024-09-21T14:03:32.167343+0000","last_undegraded":"2024-09-21T14:03:32.167343+0000","last_fullsized":"2024-09-21T14:03:32.167343+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'27","last_scrub_stamp":"2024-09-21T14:02:58.293160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:58.293160+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:01.203771+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":650,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.121422+0000","last_change":"2024-09-21T13:59:32.205723+0000","last_active":"2024-09-21T14:03:30.121422+0000","last_peered":"2024-09-21T14:03:30.121422+0000","last_clean":"2024-09-21T14:03:30.121422+0000","last_became_active":"2024-09-21T13:59:32.205350+0000","last_became_peered":"2024-09-21T13:59:32.205350+0000","last_unstale":"2024-09-21T14:03:30.121422+0000","last_undegraded":"2024-09-21T14:03:30.121422+0000","last_fullsized":"2024-09-21T14:03:30.121422+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:32.668851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:32.668851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:47.252073+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"346'10","reported_seq":917,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.181935+0000","last_change":"2024-09-21T14:02:59.831183+0000","last_active":"2024-09-21T14:03:32.181935+0000","last_peered":"2024-09-21T14:03:32.181935+0000","last_clean":"2024-09-21T14:03:32.181935+0000","last_became_active":"2024-09-21T14:01:41.200218+0000","last_became_peered":"2024-09-21T14:01:41.200218+0000","last_unstale":"2024-09-21T14:03:32.181935+0000","last_undegraded":"2024-09-21T14:03:32.181935+0000","last_fullsized":"2024-09-21T14:03:32.181935+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'10","last_scrub_stamp":"2024-09-21T14:02:59.831128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:02:59.831128+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:08.243919+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":577,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.133520+0000","last_change":"2024-09-21T14:01:41.207310+0000","last_active":"2024-09-21T14:03:30.133520+0000","last_peered":"2024-09-21T14:03:30.133520+0000","last_clean":"2024-09-21T14:03:30.133520+0000","last_became_active":"2024-09-21T14:01:41.206720+0000","last_became_peered":"2024-09-21T14:01:41.206720+0000","last_unstale":"2024-09-21T14:03:30.133520+0000","last_undegraded":"2024-09-21T14:03:30.133520+0000","last_fullsized":"2024-09-21T14:03:30.133520+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:52.244483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:52.244483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:21.273004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"395'3457","reported_seq":3496,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:33.581397+0000","last_change":"2024-09-21T14:02:56.562998+0000","last_active":"2024-09-21T14:03:33.581397+0000","last_peered":"2024-09-21T14:03:33.581397+0000","last_clean":"2024-09-21T14:03:33.581397+0000","last_became_active":"2024-09-21T14:02:56.562524+0000","last_became_peered":"2024-09-21T14:02:56.562524+0000","last_unstale":"2024-09-21T14:03:33.581397+0000","last_undegraded":"2024-09-21T14:03:33.581397+0000","last_fullsized":"2024-09-21T14:03:33.581397+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":3457,"log_dups_size":0,"ondisk_log_size":3457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:01.881695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226557952,"num_objects":3457,"num_object_clones":0,"num_object_copies":10371,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3457,"num_write_kb":221248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"75'79","reported_seq":177,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.181298+0000","last_change":"2024-09-21T14:02:06.395083+0000","last_active":"2024-09-21T14:03:32.181298+0000","last_peered":"2024-09-21T14:03:32.181298+0000","last_clean":"2024-09-21T14:03:32.181298+0000","last_became_active":"2024-09-21T14:00:41.137756+0000","last_became_peered":"2024-09-21T14:00:41.137756+0000","last_unstale":"2024-09-21T14:03:32.181298+0000","last_undegraded":"2024-09-21T14:03:32.181298+0000","last_fullsized":"2024-09-21T14:03:32.181298+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:02:06.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:02:06.395011+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"290'30","reported_seq":675,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.121528+0000","last_change":"2024-09-21T13:59:32.246090+0000","last_active":"2024-09-21T14:03:30.121528+0000","last_peered":"2024-09-21T14:03:30.121528+0000","last_clean":"2024-09-21T14:03:30.121528+0000","last_became_active":"2024-09-21T13:59:32.205887+0000","last_became_peered":"2024-09-21T13:59:32.205887+0000","last_unstale":"2024-09-21T14:03:30.121528+0000","last_undegraded":"2024-09-21T14:03:30.121528+0000","last_fullsized":"2024-09-21T14:03:30.121528+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'18","last_scrub_stamp":"2024-09-21T13:54:19.001030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:19.001030+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:43.319659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":602,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.123561+0000","last_change":"2024-09-21T13:59:32.212559+0000","last_active":"2024-09-21T14:03:30.123561+0000","last_peered":"2024-09-21T14:03:30.123561+0000","last_clean":"2024-09-21T14:03:30.123561+0000","last_became_active":"2024-09-21T13:59:32.211004+0000","last_became_peered":"2024-09-21T13:59:32.211004+0000","last_unstale":"2024-09-21T14:03:30.123561+0000","last_undegraded":"2024-09-21T14:03:30.123561+0000","last_fullsized":"2024-09-21T14:03:30.123561+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:25.736461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:25.736461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:48.292957+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":723,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.180023+0000","last_change":"2024-09-21T13:59:34.293330+0000","last_active":"2024-09-21T14:03:32.180023+0000","last_peered":"2024-09-21T14:03:32.180023+0000","last_clean":"2024-09-21T14:03:32.180023+0000","last_became_active":"2024-09-21T13:59:32.220304+0000","last_became_peered":"2024-09-21T13:59:32.220304+0000","last_unstale":"2024-09-21T14:03:32.180023+0000","last_undegraded":"2024-09-21T14:03:32.180023+0000","last_fullsized":"2024-09-21T14:03:32.180023+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:34.292892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:34.292892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:19.896473+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":702,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.181453+0000","last_change":"2024-09-21T13:59:23.334230+0000","last_active":"2024-09-21T14:03:32.181453+0000","last_peered":"2024-09-21T14:03:32.181453+0000","last_clean":"2024-09-21T14:03:32.181453+0000","last_became_active":"2024-09-21T13:57:12.680372+0000","last_became_peered":"2024-09-21T13:57:12.680372+0000","last_unstale":"2024-09-21T14:03:32.181453+0000","last_undegraded":"2024-09-21T14:03:32.181453+0000","last_fullsized":"2024-09-21T14:03:32.181453+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:23.334162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:23.334162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:23.951901+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"395'3690","reported_seq":3729,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:33.563640+0000","last_change":"2024-09-21T14:02:56.557348+0000","last_active":"2024-09-21T14:03:33.563640+0000","last_peered":"2024-09-21T14:03:33.563640+0000","last_clean":"2024-09-21T14:03:33.563640+0000","last_became_active":"2024-09-21T14:02:56.557055+0000","last_became_peered":"2024-09-21T14:02:56.557055+0000","last_unstale":"2024-09-21T14:03:33.563640+0000","last_undegraded":"2024-09-21T14:03:33.563640+0000","last_fullsized":"2024-09-21T14:03:33.563640+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":3690,"log_dups_size":0,"ondisk_log_size":3690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:20.950710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":241827840,"num_objects":3690,"num_object_clones":0,"num_object_copies":11070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3690,"num_write_kb":236160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"75'79","reported_seq":178,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.122301+0000","last_change":"2024-09-21T14:01:42.177646+0000","last_active":"2024-09-21T14:03:30.122301+0000","last_peered":"2024-09-21T14:03:30.122301+0000","last_clean":"2024-09-21T14:03:30.122301+0000","last_became_active":"2024-09-21T14:01:42.173042+0000","last_became_peered":"2024-09-21T14:01:42.173042+0000","last_unstale":"2024-09-21T14:03:30.122301+0000","last_undegraded":"2024-09-21T14:03:30.122301+0000","last_fullsized":"2024-09-21T14:03:30.122301+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:03:42.356436+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":709030,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"290'28","reported_seq":820,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.180117+0000","last_change":"2024-09-21T14:03:27.047111+0000","last_active":"2024-09-21T14:03:32.180117+0000","last_peered":"2024-09-21T14:03:32.180117+0000","last_clean":"2024-09-21T14:03:32.180117+0000","last_became_active":"2024-09-21T14:01:41.200859+0000","last_became_peered":"2024-09-21T14:01:41.200859+0000","last_unstale":"2024-09-21T14:03:32.180117+0000","last_undegraded":"2024-09-21T14:03:32.180117+0000","last_fullsized":"2024-09-21T14:03:32.180117+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'28","last_scrub_stamp":"2024-09-21T14:03:27.046906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:03:27.046906+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:29.157667+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":710,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.179437+0000","last_change":"2024-09-21T13:57:38.046718+0000","last_active":"2024-09-21T14:03:32.179437+0000","last_peered":"2024-09-21T14:03:32.179437+0000","last_clean":"2024-09-21T14:03:32.179437+0000","last_became_active":"2024-09-21T13:48:58.053533+0000","last_became_peered":"2024-09-21T13:48:58.053533+0000","last_unstale":"2024-09-21T14:03:32.179437+0000","last_undegraded":"2024-09-21T14:03:32.179437+0000","last_fullsized":"2024-09-21T14:03:32.179437+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:57:38.046627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:38.046627+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:49.580099+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"241'10","reported_seq":767,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.121775+0000","last_change":"2024-09-21T14:03:03.228551+0000","last_active":"2024-09-21T14:03:30.121775+0000","last_peered":"2024-09-21T14:03:30.121775+0000","last_clean":"2024-09-21T14:03:30.121775+0000","last_became_active":"2024-09-21T13:57:12.686651+0000","last_became_peered":"2024-09-21T13:57:12.686651+0000","last_unstale":"2024-09-21T14:03:30.121775+0000","last_undegraded":"2024-09-21T14:03:30.121775+0000","last_fullsized":"2024-09-21T14:03:30.121775+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'10","last_scrub_stamp":"2024-09-21T14:03:03.228443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:03.228443+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:24.135971+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":666,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.131185+0000","last_change":"2024-09-21T13:59:32.210257+0000","last_active":"2024-09-21T14:03:30.131185+0000","last_peered":"2024-09-21T14:03:30.131185+0000","last_clean":"2024-09-21T14:03:30.131185+0000","last_became_active":"2024-09-21T13:59:32.209615+0000","last_became_peered":"2024-09-21T13:59:32.209615+0000","last_unstale":"2024-09-21T14:03:30.131185+0000","last_undegraded":"2024-09-21T14:03:30.131185+0000","last_fullsized":"2024-09-21T14:03:30.131185+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:37.548064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:37.548064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.c","version":"394'3237","reported_seq":3274,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:31.174312+0000","last_change":"2024-09-21T14:02:56.563139+0000","last_active":"2024-09-21T14:03:31.174312+0000","last_peered":"2024-09-21T14:03:31.174312+0000","last_clean":"2024-09-21T14:03:31.174312+0000","last_became_active":"2024-09-21T14:02:56.562806+0000","last_became_peered":"2024-09-21T14:02:56.562806+0000","last_unstale":"2024-09-21T14:03:31.174312+0000","last_undegraded":"2024-09-21T14:03:31.174312+0000","last_fullsized":"2024-09-21T14:03:31.174312+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":3237,"log_dups_size":0,"ondisk_log_size":3237,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:18.206635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212140032,"num_objects":3237,"num_object_clones":0,"num_object_copies":9711,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3237,"num_write_kb":207168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"75'79","reported_seq":750,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.176655+0000","last_change":"2024-09-21T14:02:05.693250+0000","last_active":"2024-09-21T14:03:32.176655+0000","last_peered":"2024-09-21T14:03:32.176655+0000","last_clean":"2024-09-21T14:03:32.176655+0000","last_became_active":"2024-09-21T14:00:41.083425+0000","last_became_peered":"2024-09-21T14:00:41.083425+0000","last_unstale":"2024-09-21T14:03:32.176655+0000","last_undegraded":"2024-09-21T14:03:32.176655+0000","last_fullsized":"2024-09-21T14:03:32.176655+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:02:05.693193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:02:05.693193+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"290'20","reported_seq":36,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.167142+0000","last_change":"2024-09-21T14:03:31.791421+0000","last_active":"2024-09-21T14:03:32.167142+0000","last_peered":"2024-09-21T14:03:32.167142+0000","last_clean":"2024-09-21T14:03:32.167142+0000","last_became_active":"2024-09-21T14:03:29.158218+0000","last_became_peered":"2024-09-21T14:03:29.158218+0000","last_unstale":"2024-09-21T14:03:32.167142+0000","last_undegraded":"2024-09-21T14:03:32.167142+0000","last_fullsized":"2024-09-21T14:03:32.167142+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'20","last_scrub_stamp":"2024-09-21T14:03:31.791317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:03:31.791317+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:39.052613+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"61'1","reported_seq":639,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.133605+0000","last_change":"2024-09-21T14:01:41.195580+0000","last_active":"2024-09-21T14:03:30.133605+0000","last_peered":"2024-09-21T14:03:30.133605+0000","last_clean":"2024-09-21T14:03:30.133605+0000","last_became_active":"2024-09-21T14:01:41.195268+0000","last_became_peered":"2024-09-21T14:01:41.195268+0000","last_unstale":"2024-09-21T14:03:30.133605+0000","last_undegraded":"2024-09-21T14:03:30.133605+0000","last_fullsized":"2024-09-21T14:03:30.133605+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:58:48.407734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:48.407734+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:25.283190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"346'15","reported_seq":346,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.177671+0000","last_change":"2024-09-21T14:01:41.209662+0000","last_active":"2024-09-21T14:03:32.177671+0000","last_peered":"2024-09-21T14:03:32.177671+0000","last_clean":"2024-09-21T14:03:32.177671+0000","last_became_active":"2024-09-21T14:01:41.209296+0000","last_became_peered":"2024-09-21T14:01:41.209296+0000","last_unstale":"2024-09-21T14:03:32.177671+0000","last_undegraded":"2024-09-21T14:03:32.177671+0000","last_fullsized":"2024-09-21T14:03:32.177671+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'12","last_scrub_stamp":"2024-09-21T13:57:30.133392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:30.133392+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:43.651875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":521,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.121668+0000","last_change":"2024-09-21T13:58:58.730439+0000","last_active":"2024-09-21T14:03:30.121668+0000","last_peered":"2024-09-21T14:03:30.121668+0000","last_clean":"2024-09-21T14:03:30.121668+0000","last_became_active":"2024-09-21T13:57:12.704897+0000","last_became_peered":"2024-09-21T13:57:12.704897+0000","last_unstale":"2024-09-21T14:03:30.121668+0000","last_undegraded":"2024-09-21T14:03:30.121668+0000","last_fullsized":"2024-09-21T14:03:30.121668+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:58.730368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:58.730368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.d","version":"395'3595","reported_seq":3633,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:33.573967+0000","last_change":"2024-09-21T14:02:56.560958+0000","last_active":"2024-09-21T14:03:33.573967+0000","last_peered":"2024-09-21T14:03:33.573967+0000","last_clean":"2024-09-21T14:03:33.573967+0000","last_became_active":"2024-09-21T14:02:56.560664+0000","last_became_peered":"2024-09-21T14:02:56.560664+0000","last_unstale":"2024-09-21T14:03:33.573967+0000","last_undegraded":"2024-09-21T14:03:33.573967+0000","last_fullsized":"2024-09-21T14:03:33.573967+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":3595,"log_dups_size":0,"ondisk_log_size":3595,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:06.081463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235601920,"num_objects":3595,"num_object_clones":0,"num_object_copies":10785,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3595,"num_write_kb":230080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"75'79","reported_seq":750,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.176383+0000","last_change":"2024-09-21T14:02:03.737005+0000","last_active":"2024-09-21T14:03:32.176383+0000","last_peered":"2024-09-21T14:03:32.176383+0000","last_clean":"2024-09-21T14:03:32.176383+0000","last_became_active":"2024-09-21T14:00:41.073047+0000","last_became_peered":"2024-09-21T14:00:41.073047+0000","last_unstale":"2024-09-21T14:03:32.176383+0000","last_undegraded":"2024-09-21T14:03:32.176383+0000","last_fullsized":"2024-09-21T14:03:32.176383+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:02:03.736928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:02:03.736928+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"290'29","reported_seq":653,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.133340+0000","last_change":"2024-09-21T14:03:08.399687+0000","last_active":"2024-09-21T14:03:30.133340+0000","last_peered":"2024-09-21T14:03:30.133340+0000","last_clean":"2024-09-21T14:03:30.133340+0000","last_became_active":"2024-09-21T14:01:41.186960+0000","last_became_peered":"2024-09-21T14:01:41.186960+0000","last_unstale":"2024-09-21T14:03:30.133340+0000","last_undegraded":"2024-09-21T14:03:30.133340+0000","last_fullsized":"2024-09-21T14:03:30.133340+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'29","last_scrub_stamp":"2024-09-21T14:03:08.399606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:03:08.399606+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:19.025883+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":724,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.167660+0000","last_change":"2024-09-21T14:01:41.185706+0000","last_active":"2024-09-21T14:03:32.167660+0000","last_peered":"2024-09-21T14:03:32.167660+0000","last_clean":"2024-09-21T14:03:32.167660+0000","last_became_active":"2024-09-21T14:01:41.185400+0000","last_became_peered":"2024-09-21T14:01:41.185400+0000","last_unstale":"2024-09-21T14:03:32.167660+0000","last_undegraded":"2024-09-21T14:03:32.167660+0000","last_fullsized":"2024-09-21T14:03:32.167660+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:16.712990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:16.712990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:38.309423+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":299,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.177000+0000","last_change":"2024-09-21T14:02:59.469906+0000","last_active":"2024-09-21T14:03:32.177000+0000","last_peered":"2024-09-21T14:03:32.177000+0000","last_clean":"2024-09-21T14:03:32.177000+0000","last_became_active":"2024-09-21T14:01:41.193973+0000","last_became_peered":"2024-09-21T14:01:41.193973+0000","last_unstale":"2024-09-21T14:03:32.177000+0000","last_undegraded":"2024-09-21T14:03:32.177000+0000","last_fullsized":"2024-09-21T14:03:32.177000+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:59.469835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:02:59.469835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:00.302431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":714,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.164237+0000","last_change":"2024-09-21T13:58:43.952014+0000","last_active":"2024-09-21T14:03:32.164237+0000","last_peered":"2024-09-21T14:03:32.164237+0000","last_clean":"2024-09-21T14:03:32.164237+0000","last_became_active":"2024-09-21T13:58:17.757628+0000","last_became_peered":"2024-09-21T13:58:17.757628+0000","last_unstale":"2024-09-21T14:03:32.164237+0000","last_undegraded":"2024-09-21T14:03:32.164237+0000","last_fullsized":"2024-09-21T14:03:32.164237+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:43.951941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:43.951941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:52.357919+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"395'3544","reported_seq":3583,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:33.588264+0000","last_change":"2024-09-21T14:02:56.559655+0000","last_active":"2024-09-21T14:03:33.588264+0000","last_peered":"2024-09-21T14:03:33.588264+0000","last_clean":"2024-09-21T14:03:33.588264+0000","last_became_active":"2024-09-21T14:02:56.559346+0000","last_became_peered":"2024-09-21T14:02:56.559346+0000","last_unstale":"2024-09-21T14:03:33.588264+0000","last_undegraded":"2024-09-21T14:03:33.588264+0000","last_fullsized":"2024-09-21T14:03:33.588264+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":3544,"log_dups_size":0,"ondisk_log_size":3544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:10.203798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":232259584,"num_objects":3544,"num_object_clones":0,"num_object_copies":10632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3544,"num_write_kb":226816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"75'79","reported_seq":121,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.132285+0000","last_change":"2024-09-21T14:01:41.199377+0000","last_active":"2024-09-21T14:03:30.132285+0000","last_peered":"2024-09-21T14:03:30.132285+0000","last_clean":"2024-09-21T14:03:30.132285+0000","last_became_active":"2024-09-21T14:01:41.198750+0000","last_became_peered":"2024-09-21T14:01:41.198750+0000","last_unstale":"2024-09-21T14:03:30.132285+0000","last_undegraded":"2024-09-21T14:03:30.132285+0000","last_fullsized":"2024-09-21T14:03:30.132285+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:03:47.451019+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"290'44","reported_seq":973,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.165268+0000","last_change":"2024-09-21T14:02:57.316829+0000","last_active":"2024-09-21T14:03:32.165268+0000","last_peered":"2024-09-21T14:03:32.165268+0000","last_clean":"2024-09-21T14:03:32.165268+0000","last_became_active":"2024-09-21T13:57:12.684690+0000","last_became_peered":"2024-09-21T13:57:12.684690+0000","last_unstale":"2024-09-21T14:03:32.165268+0000","last_undegraded":"2024-09-21T14:03:32.165268+0000","last_fullsized":"2024-09-21T14:03:32.165268+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'44","last_scrub_stamp":"2024-09-21T14:02:57.316763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:57.316763+0000","objects_scrubbed":10,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:57.769620+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":730,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.180006+0000","last_change":"2024-09-21T14:01:58.524159+0000","last_active":"2024-09-21T14:03:32.180006+0000","last_peered":"2024-09-21T14:03:32.180006+0000","last_clean":"2024-09-21T14:03:32.180006+0000","last_became_active":"2024-09-21T13:58:17.770937+0000","last_became_peered":"2024-09-21T13:58:17.770937+0000","last_unstale":"2024-09-21T14:03:32.180006+0000","last_undegraded":"2024-09-21T14:03:32.180006+0000","last_fullsized":"2024-09-21T14:03:32.180006+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:58.524114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:01:58.524114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:08.916792+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":666,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.390161+0000","last_change":"2024-09-21T14:03:30.390161+0000","last_active":"2024-09-21T14:03:30.390161+0000","last_peered":"2024-09-21T14:03:30.390161+0000","last_clean":"2024-09-21T14:03:30.390161+0000","last_became_active":"2024-09-21T14:01:41.247642+0000","last_became_peered":"2024-09-21T14:01:41.247642+0000","last_unstale":"2024-09-21T14:03:30.390161+0000","last_undegraded":"2024-09-21T14:03:30.390161+0000","last_fullsized":"2024-09-21T14:03:30.390161+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-21T14:03:30.390086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T14:03:30.390086+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:32.519440+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":642,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.166778+0000","last_change":"2024-09-21T13:55:07.021128+0000","last_active":"2024-09-21T14:03:32.166778+0000","last_peered":"2024-09-21T14:03:32.166778+0000","last_clean":"2024-09-21T14:03:32.166778+0000","last_became_active":"2024-09-21T13:50:29.688249+0000","last_became_peered":"2024-09-21T13:50:29.688249+0000","last_unstale":"2024-09-21T14:03:32.166778+0000","last_undegraded":"2024-09-21T14:03:32.166778+0000","last_fullsized":"2024-09-21T14:03:32.166778+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:07.021059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:55:07.021059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:50.302160+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":637,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.124710+0000","last_change":"2024-09-21T14:03:02.238409+0000","last_active":"2024-09-21T14:03:30.124710+0000","last_peered":"2024-09-21T14:03:30.124710+0000","last_clean":"2024-09-21T14:03:30.124710+0000","last_became_active":"2024-09-21T13:59:32.208980+0000","last_became_peered":"2024-09-21T13:59:32.208980+0000","last_unstale":"2024-09-21T14:03:30.124710+0000","last_undegraded":"2024-09-21T14:03:30.124710+0000","last_fullsized":"2024-09-21T14:03:30.124710+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:02.238357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:02.238357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:09.206892+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"394'3260","reported_seq":3297,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.869302+0000","last_change":"2024-09-21T14:02:56.559275+0000","last_active":"2024-09-21T14:03:30.869302+0000","last_peered":"2024-09-21T14:03:30.869302+0000","last_clean":"2024-09-21T14:03:30.869302+0000","last_became_active":"2024-09-21T14:02:56.558773+0000","last_became_peered":"2024-09-21T14:02:56.558773+0000","last_unstale":"2024-09-21T14:03:30.869302+0000","last_undegraded":"2024-09-21T14:03:30.869302+0000","last_fullsized":"2024-09-21T14:03:30.869302+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":3260,"log_dups_size":0,"ondisk_log_size":3260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:02.709376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213647360,"num_objects":3260,"num_object_clones":0,"num_object_copies":9780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3260,"num_write_kb":208640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"75'79","reported_seq":754,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.130329+0000","last_change":"2024-09-21T14:00:41.077286+0000","last_active":"2024-09-21T14:03:30.130329+0000","last_peered":"2024-09-21T14:03:30.130329+0000","last_clean":"2024-09-21T14:03:30.130329+0000","last_became_active":"2024-09-21T14:00:41.076948+0000","last_became_peered":"2024-09-21T14:00:41.076948+0000","last_unstale":"2024-09-21T14:03:30.130329+0000","last_undegraded":"2024-09-21T14:03:30.130329+0000","last_fullsized":"2024-09-21T14:03:30.130329+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:03:51.730082+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"290'17","reported_seq":749,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.167428+0000","last_change":"2024-09-21T14:01:41.200809+0000","last_active":"2024-09-21T14:03:32.167428+0000","last_peered":"2024-09-21T14:03:32.167428+0000","last_clean":"2024-09-21T14:03:32.167428+0000","last_became_active":"2024-09-21T14:01:41.200365+0000","last_became_peered":"2024-09-21T14:01:41.200365+0000","last_unstale":"2024-09-21T14:03:32.167428+0000","last_undegraded":"2024-09-21T14:03:32.167428+0000","last_fullsized":"2024-09-21T14:03:32.167428+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'13","last_scrub_stamp":"2024-09-21T13:58:11.619848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:11.619848+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:39.361066+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":759,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.164394+0000","last_change":"2024-09-21T14:03:01.310647+0000","last_active":"2024-09-21T14:03:32.164394+0000","last_peered":"2024-09-21T14:03:32.164394+0000","last_clean":"2024-09-21T14:03:32.164394+0000","last_became_active":"2024-09-21T13:58:17.759671+0000","last_became_peered":"2024-09-21T13:58:17.759671+0000","last_unstale":"2024-09-21T14:03:32.164394+0000","last_undegraded":"2024-09-21T14:03:32.164394+0000","last_fullsized":"2024-09-21T14:03:32.164394+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:01.310603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:03:01.310603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:10.066440+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":640,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.169321+0000","last_change":"2024-09-21T13:58:41.644876+0000","last_active":"2024-09-21T14:03:32.169321+0000","last_peered":"2024-09-21T14:03:32.169321+0000","last_clean":"2024-09-21T14:03:32.169321+0000","last_became_active":"2024-09-21T13:57:12.682147+0000","last_became_peered":"2024-09-21T13:57:12.682147+0000","last_unstale":"2024-09-21T14:03:32.169321+0000","last_undegraded":"2024-09-21T14:03:32.169321+0000","last_fullsized":"2024-09-21T14:03:32.169321+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:58:41.644789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:58:41.644789+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:15.691519+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":619,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.167938+0000","last_change":"2024-09-21T14:01:50.880840+0000","last_active":"2024-09-21T14:03:32.167938+0000","last_peered":"2024-09-21T14:03:32.167938+0000","last_clean":"2024-09-21T14:03:32.167938+0000","last_became_active":"2024-09-21T14:01:41.182000+0000","last_became_peered":"2024-09-21T14:01:41.182000+0000","last_unstale":"2024-09-21T14:03:32.167938+0000","last_undegraded":"2024-09-21T14:03:32.167938+0000","last_fullsized":"2024-09-21T14:03:32.167938+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:50.880471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:01:50.880471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:17.729591+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":674,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.131851+0000","last_change":"2024-09-21T13:59:32.222791+0000","last_active":"2024-09-21T14:03:30.131851+0000","last_peered":"2024-09-21T14:03:30.131851+0000","last_clean":"2024-09-21T14:03:30.131851+0000","last_became_active":"2024-09-21T13:59:32.221502+0000","last_became_peered":"2024-09-21T13:59:32.221502+0000","last_unstale":"2024-09-21T14:03:30.131851+0000","last_undegraded":"2024-09-21T14:03:30.131851+0000","last_fullsized":"2024-09-21T14:03:30.131851+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:14.673582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:14.673582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"395'3532","reported_seq":3568,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.765623+0000","last_change":"2024-09-21T14:02:56.559108+0000","last_active":"2024-09-21T14:03:32.765623+0000","last_peered":"2024-09-21T14:03:32.765623+0000","last_clean":"2024-09-21T14:03:32.765623+0000","last_became_active":"2024-09-21T14:02:56.558826+0000","last_became_peered":"2024-09-21T14:02:56.558826+0000","last_unstale":"2024-09-21T14:03:32.765623+0000","last_undegraded":"2024-09-21T14:03:32.765623+0000","last_fullsized":"2024-09-21T14:03:32.765623+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":3532,"log_dups_size":0,"ondisk_log_size":3532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:03.816041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":231473152,"num_objects":3532,"num_object_clones":0,"num_object_copies":10596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3532,"num_write_kb":226048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"75'79","reported_seq":161,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.170148+0000","last_change":"2024-09-21T14:00:41.071631+0000","last_active":"2024-09-21T14:03:32.170148+0000","last_peered":"2024-09-21T14:03:32.170148+0000","last_clean":"2024-09-21T14:03:32.170148+0000","last_became_active":"2024-09-21T14:00:41.071042+0000","last_became_peered":"2024-09-21T14:00:41.071042+0000","last_unstale":"2024-09-21T14:03:32.170148+0000","last_undegraded":"2024-09-21T14:03:32.170148+0000","last_fullsized":"2024-09-21T14:03:32.170148+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:03:42.335979+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"290'24","reported_seq":808,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.179202+0000","last_change":"2024-09-21T14:02:57.560019+0000","last_active":"2024-09-21T14:03:32.179202+0000","last_peered":"2024-09-21T14:03:32.179202+0000","last_clean":"2024-09-21T14:03:32.179202+0000","last_became_active":"2024-09-21T14:01:41.184728+0000","last_became_peered":"2024-09-21T14:01:41.184728+0000","last_unstale":"2024-09-21T14:03:32.179202+0000","last_undegraded":"2024-09-21T14:03:32.179202+0000","last_fullsized":"2024-09-21T14:03:32.179202+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'24","last_scrub_stamp":"2024-09-21T14:02:57.559936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:57.559936+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:26.628231+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":652,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.180216+0000","last_change":"2024-09-21T13:59:59.450613+0000","last_active":"2024-09-21T14:03:32.180216+0000","last_peered":"2024-09-21T14:03:32.180216+0000","last_clean":"2024-09-21T14:03:32.180216+0000","last_became_active":"2024-09-21T13:57:12.681030+0000","last_became_peered":"2024-09-21T13:57:12.681030+0000","last_unstale":"2024-09-21T14:03:32.180216+0000","last_undegraded":"2024-09-21T14:03:32.180216+0000","last_fullsized":"2024-09-21T14:03:32.180216+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:59.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:59:59.450546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:40.683320+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":1448,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.573217+0000","last_change":"2024-09-21T13:59:32.218796+0000","last_active":"2024-09-21T14:03:30.573217+0000","last_peered":"2024-09-21T14:03:30.573217+0000","last_clean":"2024-09-21T14:03:30.573217+0000","last_became_active":"2024-09-21T13:59:32.203361+0000","last_became_peered":"2024-09-21T13:59:32.203361+0000","last_unstale":"2024-09-21T14:03:30.573217+0000","last_undegraded":"2024-09-21T14:03:30.573217+0000","last_fullsized":"2024-09-21T14:03:30.573217+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-21T13:56:29.617143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:56:29.617143+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:20.983081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047048000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":245,"num_read_kb":240,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":637,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.134397+0000","last_change":"2024-09-21T14:03:06.470855+0000","last_active":"2024-09-21T14:03:30.134397+0000","last_peered":"2024-09-21T14:03:30.134397+0000","last_clean":"2024-09-21T14:03:30.134397+0000","last_became_active":"2024-09-21T14:01:41.200531+0000","last_became_peered":"2024-09-21T14:01:41.200531+0000","last_unstale":"2024-09-21T14:03:30.134397+0000","last_undegraded":"2024-09-21T14:03:30.134397+0000","last_fullsized":"2024-09-21T14:03:30.134397+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:06.470705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:06.470705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:32.367248+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":743,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.164936+0000","last_change":"2024-09-21T14:03:04.357026+0000","last_active":"2024-09-21T14:03:32.164936+0000","last_peered":"2024-09-21T14:03:32.164936+0000","last_clean":"2024-09-21T14:03:32.164936+0000","last_became_active":"2024-09-21T14:01:41.184664+0000","last_became_peered":"2024-09-21T14:01:41.184664+0000","last_unstale":"2024-09-21T14:03:32.164936+0000","last_undegraded":"2024-09-21T14:03:32.164936+0000","last_fullsized":"2024-09-21T14:03:32.164936+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:04.356979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:04.356979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:19.365633+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"395'3496","reported_seq":3535,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:33.572230+0000","last_change":"2024-09-21T14:02:56.557541+0000","last_active":"2024-09-21T14:03:33.572230+0000","last_peered":"2024-09-21T14:03:33.572230+0000","last_clean":"2024-09-21T14:03:33.572230+0000","last_became_active":"2024-09-21T14:02:56.557263+0000","last_became_peered":"2024-09-21T14:02:56.557263+0000","last_unstale":"2024-09-21T14:03:33.572230+0000","last_undegraded":"2024-09-21T14:03:33.572230+0000","last_fullsized":"2024-09-21T14:03:33.572230+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":3496,"log_dups_size":0,"ondisk_log_size":3496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:05.629424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":229113856,"num_objects":3496,"num_object_clones":0,"num_object_copies":10488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3496,"num_write_kb":223744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"300'32","reported_seq":399,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.176308+0000","last_change":"2024-09-21T14:00:47.130089+0000","last_active":"2024-09-21T14:03:32.176308+0000","last_peered":"2024-09-21T14:03:32.176308+0000","last_clean":"2024-09-21T14:03:32.176308+0000","last_became_active":"2024-09-21T13:59:32.204624+0000","last_became_peered":"2024-09-21T13:59:32.204624+0000","last_unstale":"2024-09-21T14:03:32.176308+0000","last_undegraded":"2024-09-21T14:03:32.176308+0000","last_fullsized":"2024-09-21T14:03:32.176308+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'32","last_scrub_stamp":"2024-09-21T14:00:47.129789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:00:47.129789+0000","objects_scrubbed":8,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:16.636929+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":688,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.124061+0000","last_change":"2024-09-21T14:01:41.190537+0000","last_active":"2024-09-21T14:03:30.124061+0000","last_peered":"2024-09-21T14:03:30.124061+0000","last_clean":"2024-09-21T14:03:30.124061+0000","last_became_active":"2024-09-21T14:01:41.190210+0000","last_became_peered":"2024-09-21T14:01:41.190210+0000","last_unstale":"2024-09-21T14:03:30.124061+0000","last_undegraded":"2024-09-21T14:03:30.124061+0000","last_fullsized":"2024-09-21T14:03:30.124061+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:57.772975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:57.772975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"75'79","reported_seq":749,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.132635+0000","last_change":"2024-09-21T14:00:39.339360+0000","last_active":"2024-09-21T14:03:30.132635+0000","last_peered":"2024-09-21T14:03:30.132635+0000","last_clean":"2024-09-21T14:03:30.132635+0000","last_became_active":"2024-09-21T14:00:39.339077+0000","last_became_peered":"2024-09-21T14:00:39.339077+0000","last_unstale":"2024-09-21T14:03:30.132635+0000","last_undegraded":"2024-09-21T14:03:30.132635+0000","last_fullsized":"2024-09-21T14:03:30.132635+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:03:49.763645+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":19,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53669,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":662,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.131406+0000","last_change":"2024-09-21T14:01:41.182186+0000","last_active":"2024-09-21T14:03:30.131406+0000","last_peered":"2024-09-21T14:03:30.131406+0000","last_clean":"2024-09-21T14:03:30.131406+0000","last_became_active":"2024-09-21T14:01:41.181899+0000","last_became_peered":"2024-09-21T14:01:41.181899+0000","last_unstale":"2024-09-21T14:03:30.131406+0000","last_undegraded":"2024-09-21T14:03:30.131406+0000","last_fullsized":"2024-09-21T14:03:30.131406+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:15.700409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:15.700409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":587,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.134230+0000","last_change":"2024-09-21T14:01:41.196509+0000","last_active":"2024-09-21T14:03:30.134230+0000","last_peered":"2024-09-21T14:03:30.134230+0000","last_clean":"2024-09-21T14:03:30.134230+0000","last_became_active":"2024-09-21T14:01:41.196243+0000","last_became_peered":"2024-09-21T14:01:41.196243+0000","last_unstale":"2024-09-21T14:03:30.134230+0000","last_undegraded":"2024-09-21T14:03:30.134230+0000","last_fullsized":"2024-09-21T14:03:30.134230+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:21.153018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:21.153018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:17.291763+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"395'3662","reported_seq":3701,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:33.575146+0000","last_change":"2024-09-21T14:02:56.561047+0000","last_active":"2024-09-21T14:03:33.575146+0000","last_peered":"2024-09-21T14:03:33.575146+0000","last_clean":"2024-09-21T14:03:33.575146+0000","last_became_active":"2024-09-21T14:02:56.558774+0000","last_became_peered":"2024-09-21T14:02:56.558774+0000","last_unstale":"2024-09-21T14:03:33.575146+0000","last_undegraded":"2024-09-21T14:03:33.575146+0000","last_fullsized":"2024-09-21T14:03:33.575146+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":3662,"log_dups_size":0,"ondisk_log_size":3662,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:07.590313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239992832,"num_objects":3662,"num_object_clones":0,"num_object_copies":10986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3662,"num_write_kb":234368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"290'19","reported_seq":776,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.181984+0000","last_change":"2024-09-21T14:02:28.553549+0000","last_active":"2024-09-21T14:03:32.181984+0000","last_peered":"2024-09-21T14:03:32.181984+0000","last_clean":"2024-09-21T14:03:32.181984+0000","last_became_active":"2024-09-21T13:58:17.776310+0000","last_became_peered":"2024-09-21T13:58:17.776310+0000","last_unstale":"2024-09-21T14:03:32.181984+0000","last_undegraded":"2024-09-21T14:03:32.181984+0000","last_fullsized":"2024-09-21T14:03:32.181984+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'19","last_scrub_stamp":"2024-09-21T14:02:28.553065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:28.553065+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:33.636087+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":746,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.957705+0000","last_change":"2024-09-21T14:03:32.957705+0000","last_active":"2024-09-21T14:03:32.957705+0000","last_peered":"2024-09-21T14:03:32.957705+0000","last_clean":"2024-09-21T14:03:32.957705+0000","last_became_active":"2024-09-21T13:59:32.223247+0000","last_became_peered":"2024-09-21T13:59:32.223247+0000","last_unstale":"2024-09-21T14:03:32.957705+0000","last_undegraded":"2024-09-21T14:03:32.957705+0000","last_fullsized":"2024-09-21T14:03:32.957705+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:32.957649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:03:32.957649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:44.854711+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":663,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.134176+0000","last_change":"2024-09-21T14:02:58.461716+0000","last_active":"2024-09-21T14:03:30.134176+0000","last_peered":"2024-09-21T14:03:30.134176+0000","last_clean":"2024-09-21T14:03:30.134176+0000","last_became_active":"2024-09-21T14:01:41.193941+0000","last_became_peered":"2024-09-21T14:01:41.193941+0000","last_unstale":"2024-09-21T14:03:30.134176+0000","last_undegraded":"2024-09-21T14:03:30.134176+0000","last_fullsized":"2024-09-21T14:03:30.134176+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:58.461663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:02:58.461663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:20.802609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":665,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.122806+0000","last_change":"2024-09-21T14:01:41.190501+0000","last_active":"2024-09-21T14:03:30.122806+0000","last_peered":"2024-09-21T14:03:30.122806+0000","last_clean":"2024-09-21T14:03:30.122806+0000","last_became_active":"2024-09-21T14:01:41.190192+0000","last_became_peered":"2024-09-21T14:01:41.190192+0000","last_unstale":"2024-09-21T14:03:30.122806+0000","last_undegraded":"2024-09-21T14:03:30.122806+0000","last_fullsized":"2024-09-21T14:03:30.122806+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:58.952218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:54:58.952218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"395'3440","reported_seq":3479,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.742956+0000","last_change":"2024-09-21T14:02:56.560359+0000","last_active":"2024-09-21T14:03:32.742956+0000","last_peered":"2024-09-21T14:03:32.742956+0000","last_clean":"2024-09-21T14:03:32.742956+0000","last_became_active":"2024-09-21T14:02:56.560070+0000","last_became_peered":"2024-09-21T14:02:56.560070+0000","last_unstale":"2024-09-21T14:03:32.742956+0000","last_undegraded":"2024-09-21T14:03:32.742956+0000","last_fullsized":"2024-09-21T14:03:32.742956+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":3440,"log_dups_size":0,"ondisk_log_size":3440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:16.453643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":225443840,"num_objects":3440,"num_object_clones":0,"num_object_copies":10320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3440,"num_write_kb":220160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"75'79","reported_seq":128,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.131903+0000","last_change":"2024-09-21T14:01:41.189146+0000","last_active":"2024-09-21T14:03:30.131903+0000","last_peered":"2024-09-21T14:03:30.131903+0000","last_clean":"2024-09-21T14:03:30.131903+0000","last_became_active":"2024-09-21T14:01:41.188685+0000","last_became_peered":"2024-09-21T14:01:41.188685+0000","last_unstale":"2024-09-21T14:03:30.131903+0000","last_undegraded":"2024-09-21T14:03:30.131903+0000","last_fullsized":"2024-09-21T14:03:30.131903+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:03:46.437011+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"290'31","reported_seq":797,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.181145+0000","last_change":"2024-09-21T14:03:31.017284+0000","last_active":"2024-09-21T14:03:32.181145+0000","last_peered":"2024-09-21T14:03:32.181145+0000","last_clean":"2024-09-21T14:03:32.181145+0000","last_became_active":"2024-09-21T14:03:29.162483+0000","last_became_peered":"2024-09-21T14:03:29.162483+0000","last_unstale":"2024-09-21T14:03:32.181145+0000","last_undegraded":"2024-09-21T14:03:32.181145+0000","last_fullsized":"2024-09-21T14:03:32.181145+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'31","last_scrub_stamp":"2024-09-21T14:03:31.017119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:03:31.017119+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:54.975325+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":280,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.176795+0000","last_change":"2024-09-21T14:01:29.924408+0000","last_active":"2024-09-21T14:03:32.176795+0000","last_peered":"2024-09-21T14:03:32.176795+0000","last_clean":"2024-09-21T14:03:32.176795+0000","last_became_active":"2024-09-21T13:58:17.762520+0000","last_became_peered":"2024-09-21T13:58:17.762520+0000","last_unstale":"2024-09-21T14:03:32.176795+0000","last_undegraded":"2024-09-21T14:03:32.176795+0000","last_fullsized":"2024-09-21T14:03:32.176795+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:29.924331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:01:29.924331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:47.626829+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":693,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.165030+0000","last_change":"2024-09-21T13:58:52.030800+0000","last_active":"2024-09-21T14:03:32.165030+0000","last_peered":"2024-09-21T14:03:32.165030+0000","last_clean":"2024-09-21T14:03:32.165030+0000","last_became_active":"2024-09-21T13:49:02.067472+0000","last_became_peered":"2024-09-21T13:49:02.067472+0000","last_unstale":"2024-09-21T14:03:32.165030+0000","last_undegraded":"2024-09-21T14:03:32.165030+0000","last_fullsized":"2024-09-21T14:03:32.165030+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:52.030718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:52.030718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:43.346814+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":576,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.167521+0000","last_change":"2024-09-21T13:57:12.683321+0000","last_active":"2024-09-21T14:03:32.167521+0000","last_peered":"2024-09-21T14:03:32.167521+0000","last_clean":"2024-09-21T14:03:32.167521+0000","last_became_active":"2024-09-21T13:57:12.682979+0000","last_became_peered":"2024-09-21T13:57:12.682979+0000","last_unstale":"2024-09-21T14:03:32.167521+0000","last_undegraded":"2024-09-21T14:03:32.167521+0000","last_fullsized":"2024-09-21T14:03:32.167521+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:18.027944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:18.027944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:22.725610+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2","version":"395'3732","reported_seq":3771,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:34.703291+0000","last_change":"2024-09-21T14:02:56.559657+0000","last_active":"2024-09-21T14:03:34.703291+0000","last_peered":"2024-09-21T14:03:34.703291+0000","last_clean":"2024-09-21T14:03:34.703291+0000","last_became_active":"2024-09-21T14:02:56.559368+0000","last_became_peered":"2024-09-21T14:02:56.559368+0000","last_unstale":"2024-09-21T14:03:34.703291+0000","last_undegraded":"2024-09-21T14:03:34.703291+0000","last_fullsized":"2024-09-21T14:03:34.703291+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":3732,"log_dups_size":0,"ondisk_log_size":3732,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:23.196521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244580352,"num_objects":3732,"num_object_clones":0,"num_object_copies":11196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3732,"num_write_kb":238848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"75'79","reported_seq":762,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.131635+0000","last_change":"2024-09-21T14:02:55.806160+0000","last_active":"2024-09-21T14:03:30.131635+0000","last_peered":"2024-09-21T14:03:30.131635+0000","last_clean":"2024-09-21T14:03:30.131635+0000","last_became_active":"2024-09-21T14:00:41.139315+0000","last_became_peered":"2024-09-21T14:00:41.139315+0000","last_unstale":"2024-09-21T14:03:30.131635+0000","last_undegraded":"2024-09-21T14:03:30.131635+0000","last_fullsized":"2024-09-21T14:03:30.131635+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:02:55.806077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.806077+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:59.364707+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"290'20","reported_seq":783,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.182052+0000","last_change":"2024-09-21T13:58:17.789407+0000","last_active":"2024-09-21T14:03:32.182052+0000","last_peered":"2024-09-21T14:03:32.182052+0000","last_clean":"2024-09-21T14:03:32.182052+0000","last_became_active":"2024-09-21T13:58:17.765427+0000","last_became_peered":"2024-09-21T13:58:17.765427+0000","last_unstale":"2024-09-21T14:03:32.182052+0000","last_undegraded":"2024-09-21T14:03:32.182052+0000","last_fullsized":"2024-09-21T14:03:32.182052+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:58:13.464196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:13.464196+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:18.848758+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":630,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.123639+0000","last_change":"2024-09-21T14:02:58.236695+0000","last_active":"2024-09-21T14:03:30.123639+0000","last_peered":"2024-09-21T14:03:30.123639+0000","last_clean":"2024-09-21T14:03:30.123639+0000","last_became_active":"2024-09-21T13:57:12.704203+0000","last_became_peered":"2024-09-21T13:57:12.704203+0000","last_unstale":"2024-09-21T14:03:30.123639+0000","last_undegraded":"2024-09-21T14:03:30.123639+0000","last_fullsized":"2024-09-21T14:03:30.123639+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:58.236645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:02:58.236645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:21.993648+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"346'14","reported_seq":764,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.168057+0000","last_change":"2024-09-21T14:01:41.186960+0000","last_active":"2024-09-21T14:03:32.168057+0000","last_peered":"2024-09-21T14:03:32.168057+0000","last_clean":"2024-09-21T14:03:32.168057+0000","last_became_active":"2024-09-21T14:01:41.186564+0000","last_became_peered":"2024-09-21T14:01:41.186564+0000","last_unstale":"2024-09-21T14:03:32.168057+0000","last_undegraded":"2024-09-21T14:03:32.168057+0000","last_fullsized":"2024-09-21T14:03:32.168057+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'12","last_scrub_stamp":"2024-09-21T13:59:38.738120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:38.738120+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:47.648985+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"68'1","reported_seq":711,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.181590+0000","last_change":"2024-09-21T13:59:31.392186+0000","last_active":"2024-09-21T14:03:32.181590+0000","last_peered":"2024-09-21T14:03:32.181590+0000","last_clean":"2024-09-21T14:03:32.181590+0000","last_became_active":"2024-09-21T13:52:04.313210+0000","last_became_peered":"2024-09-21T13:52:04.313210+0000","last_unstale":"2024-09-21T14:03:32.181590+0000","last_undegraded":"2024-09-21T14:03:32.181590+0000","last_fullsized":"2024-09-21T14:03:32.181590+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T13:59:31.392105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:31.392105+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:39.716003+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.5","version":"395'3588","reported_seq":3627,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.759823+0000","last_change":"2024-09-21T14:02:56.557127+0000","last_active":"2024-09-21T14:03:32.759823+0000","last_peered":"2024-09-21T14:03:32.759823+0000","last_clean":"2024-09-21T14:03:32.759823+0000","last_became_active":"2024-09-21T14:02:56.556786+0000","last_became_peered":"2024-09-21T14:02:56.556786+0000","last_unstale":"2024-09-21T14:03:32.759823+0000","last_undegraded":"2024-09-21T14:03:32.759823+0000","last_fullsized":"2024-09-21T14:03:32.759823+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":3588,"log_dups_size":0,"ondisk_log_size":3588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:16.085110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235143168,"num_objects":3588,"num_object_clones":0,"num_object_copies":10764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3588,"num_write_kb":229632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"290'15","reported_seq":738,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.179736+0000","last_change":"2024-09-21T13:58:50.498597+0000","last_active":"2024-09-21T14:03:32.179736+0000","last_peered":"2024-09-21T14:03:32.179736+0000","last_clean":"2024-09-21T14:03:32.179736+0000","last_became_active":"2024-09-21T13:49:00.053016+0000","last_became_peered":"2024-09-21T13:49:00.053016+0000","last_unstale":"2024-09-21T14:03:32.179736+0000","last_undegraded":"2024-09-21T14:03:32.179736+0000","last_fullsized":"2024-09-21T14:03:32.179736+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:58:50.498369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:50.498369+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:54.564652+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":667,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.133080+0000","last_change":"2024-09-21T14:01:41.192717+0000","last_active":"2024-09-21T14:03:30.133080+0000","last_peered":"2024-09-21T14:03:30.133080+0000","last_clean":"2024-09-21T14:03:30.133080+0000","last_became_active":"2024-09-21T14:01:41.192443+0000","last_became_peered":"2024-09-21T14:01:41.192443+0000","last_unstale":"2024-09-21T14:03:30.133080+0000","last_undegraded":"2024-09-21T14:03:30.133080+0000","last_fullsized":"2024-09-21T14:03:30.133080+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:53.894506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:53.894506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:53.450976+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":657,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.181266+0000","last_change":"2024-09-21T14:02:56.796917+0000","last_active":"2024-09-21T14:03:32.181266+0000","last_peered":"2024-09-21T14:03:32.181266+0000","last_clean":"2024-09-21T14:03:32.181266+0000","last_became_active":"2024-09-21T13:57:12.680978+0000","last_became_peered":"2024-09-21T13:57:12.680978+0000","last_unstale":"2024-09-21T14:03:32.181266+0000","last_undegraded":"2024-09-21T14:03:32.181266+0000","last_fullsized":"2024-09-21T14:03:32.181266+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:56.796861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:02:56.796861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:10.252837+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":597,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.131310+0000","last_change":"2024-09-21T13:59:32.224001+0000","last_active":"2024-09-21T14:03:30.131310+0000","last_peered":"2024-09-21T14:03:30.131310+0000","last_clean":"2024-09-21T14:03:30.131310+0000","last_became_active":"2024-09-21T13:59:32.221628+0000","last_became_peered":"2024-09-21T13:59:32.221628+0000","last_unstale":"2024-09-21T14:03:30.131310+0000","last_undegraded":"2024-09-21T14:03:30.131310+0000","last_fullsized":"2024-09-21T14:03:30.131310+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:13.698091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:13.698091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"395'3606","reported_seq":3645,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:33.609898+0000","last_change":"2024-09-21T14:02:56.562973+0000","last_active":"2024-09-21T14:03:33.609898+0000","last_peered":"2024-09-21T14:03:33.609898+0000","last_clean":"2024-09-21T14:03:33.609898+0000","last_became_active":"2024-09-21T14:02:56.562256+0000","last_became_peered":"2024-09-21T14:02:56.562256+0000","last_unstale":"2024-09-21T14:03:33.609898+0000","last_undegraded":"2024-09-21T14:03:33.609898+0000","last_fullsized":"2024-09-21T14:03:33.609898+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":3606,"log_dups_size":0,"ondisk_log_size":3606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:09.952633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236322816,"num_objects":3606,"num_object_clones":0,"num_object_copies":10818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3606,"num_write_kb":230784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"290'35","reported_seq":346,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.176920+0000","last_change":"2024-09-21T14:01:02.006619+0000","last_active":"2024-09-21T14:03:32.176920+0000","last_peered":"2024-09-21T14:03:32.176920+0000","last_clean":"2024-09-21T14:03:32.176920+0000","last_became_active":"2024-09-21T13:59:32.204184+0000","last_became_peered":"2024-09-21T13:59:32.204184+0000","last_unstale":"2024-09-21T14:03:32.176920+0000","last_undegraded":"2024-09-21T14:03:32.176920+0000","last_fullsized":"2024-09-21T14:03:32.176920+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'35","last_scrub_stamp":"2024-09-21T14:01:02.006504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:02.006504+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:24.584005+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":689,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.132224+0000","last_change":"2024-09-21T14:01:41.184937+0000","last_active":"2024-09-21T14:03:30.132224+0000","last_peered":"2024-09-21T14:03:30.132224+0000","last_clean":"2024-09-21T14:03:30.132224+0000","last_became_active":"2024-09-21T14:01:41.184567+0000","last_became_peered":"2024-09-21T14:01:41.184567+0000","last_unstale":"2024-09-21T14:03:30.132224+0000","last_undegraded":"2024-09-21T14:03:30.132224+0000","last_fullsized":"2024-09-21T14:03:30.132224+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:21.587493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:21.587493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":670,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.168874+0000","last_change":"2024-09-21T14:01:41.182250+0000","last_active":"2024-09-21T14:03:32.168874+0000","last_peered":"2024-09-21T14:03:32.168874+0000","last_clean":"2024-09-21T14:03:32.168874+0000","last_became_active":"2024-09-21T14:01:41.181911+0000","last_became_peered":"2024-09-21T14:01:41.181911+0000","last_unstale":"2024-09-21T14:03:32.168874+0000","last_undegraded":"2024-09-21T14:03:32.168874+0000","last_fullsized":"2024-09-21T14:03:32.168874+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:55.588554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:55.588554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:45.648976+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":616,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.132510+0000","last_change":"2024-09-21T13:59:32.223123+0000","last_active":"2024-09-21T14:03:30.132510+0000","last_peered":"2024-09-21T14:03:30.132510+0000","last_clean":"2024-09-21T14:03:30.132510+0000","last_became_active":"2024-09-21T13:59:32.219332+0000","last_became_peered":"2024-09-21T13:59:32.219332+0000","last_unstale":"2024-09-21T14:03:30.132510+0000","last_undegraded":"2024-09-21T14:03:30.132510+0000","last_fullsized":"2024-09-21T14:03:30.132510+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:19.537177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:19.537177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"395'3616","reported_seq":3655,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:33.568404+0000","last_change":"2024-09-21T14:02:56.559626+0000","last_active":"2024-09-21T14:03:33.568404+0000","last_peered":"2024-09-21T14:03:33.568404+0000","last_clean":"2024-09-21T14:03:33.568404+0000","last_became_active":"2024-09-21T14:02:56.559331+0000","last_became_peered":"2024-09-21T14:02:56.559331+0000","last_unstale":"2024-09-21T14:03:33.568404+0000","last_undegraded":"2024-09-21T14:03:33.568404+0000","last_fullsized":"2024-09-21T14:03:33.568404+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":3616,"log_dups_size":0,"ondisk_log_size":3616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:20.185982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236978176,"num_objects":3616,"num_object_clones":0,"num_object_copies":10848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3616,"num_write_kb":231424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"290'18","reported_seq":755,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.180296+0000","last_change":"2024-09-21T13:59:28.340494+0000","last_active":"2024-09-21T14:03:32.180296+0000","last_peered":"2024-09-21T14:03:32.180296+0000","last_clean":"2024-09-21T14:03:32.180296+0000","last_became_active":"2024-09-21T13:57:12.680627+0000","last_became_peered":"2024-09-21T13:57:12.680627+0000","last_unstale":"2024-09-21T14:03:32.180296+0000","last_undegraded":"2024-09-21T14:03:32.180296+0000","last_fullsized":"2024-09-21T14:03:32.180296+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'18","last_scrub_stamp":"2024-09-21T13:59:28.340415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:28.340415+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:11.926616+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":644,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.132538+0000","last_change":"2024-09-21T13:59:32.201126+0000","last_active":"2024-09-21T14:03:30.132538+0000","last_peered":"2024-09-21T14:03:30.132538+0000","last_clean":"2024-09-21T14:03:30.132538+0000","last_became_active":"2024-09-21T13:59:32.200674+0000","last_became_peered":"2024-09-21T13:59:32.200674+0000","last_unstale":"2024-09-21T14:03:30.132538+0000","last_undegraded":"2024-09-21T14:03:30.132538+0000","last_fullsized":"2024-09-21T14:03:30.132538+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:21.582325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:21.582325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:12.941161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":632,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.167136+0000","last_change":"2024-09-21T14:02:27.791041+0000","last_active":"2024-09-21T14:03:32.167136+0000","last_peered":"2024-09-21T14:03:32.167136+0000","last_clean":"2024-09-21T14:03:32.167136+0000","last_became_active":"2024-09-21T13:52:04.322147+0000","last_became_peered":"2024-09-21T13:52:04.322147+0000","last_unstale":"2024-09-21T14:03:32.167136+0000","last_undegraded":"2024-09-21T14:03:32.167136+0000","last_fullsized":"2024-09-21T14:03:32.167136+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:27.790994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:02:27.790994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:51.879201+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":638,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.123976+0000","last_change":"2024-09-21T14:02:36.371955+0000","last_active":"2024-09-21T14:03:30.123976+0000","last_peered":"2024-09-21T14:03:30.123976+0000","last_clean":"2024-09-21T14:03:30.123976+0000","last_became_active":"2024-09-21T13:59:32.203767+0000","last_became_peered":"2024-09-21T13:59:32.203767+0000","last_unstale":"2024-09-21T14:03:30.123976+0000","last_undegraded":"2024-09-21T14:03:30.123976+0000","last_fullsized":"2024-09-21T14:03:30.123976+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:36.371747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:02:36.371747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:54.168810+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"394'3276","reported_seq":3313,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.864775+0000","last_change":"2024-09-21T14:02:56.564166+0000","last_active":"2024-09-21T14:03:30.864775+0000","last_peered":"2024-09-21T14:03:30.864775+0000","last_clean":"2024-09-21T14:03:30.864775+0000","last_became_active":"2024-09-21T14:02:56.563788+0000","last_became_peered":"2024-09-21T14:02:56.563788+0000","last_unstale":"2024-09-21T14:03:30.864775+0000","last_undegraded":"2024-09-21T14:03:30.864775+0000","last_fullsized":"2024-09-21T14:03:30.864775+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":3276,"log_dups_size":0,"ondisk_log_size":3276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:20.466245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":214695936,"num_objects":3276,"num_object_clones":0,"num_object_copies":9828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3276,"num_write_kb":209664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"290'27","reported_seq":743,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.132046+0000","last_change":"2024-09-21T14:01:41.193143+0000","last_active":"2024-09-21T14:03:30.132046+0000","last_peered":"2024-09-21T14:03:30.132046+0000","last_clean":"2024-09-21T14:03:30.132046+0000","last_became_active":"2024-09-21T14:01:41.192481+0000","last_became_peered":"2024-09-21T14:01:41.192481+0000","last_unstale":"2024-09-21T14:03:30.132046+0000","last_undegraded":"2024-09-21T14:03:30.132046+0000","last_fullsized":"2024-09-21T14:03:30.132046+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'27","last_scrub_stamp":"2024-09-21T13:59:33.058859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:33.058859+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":624,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.132517+0000","last_change":"2024-09-21T14:03:04.953724+0000","last_active":"2024-09-21T14:03:30.132517+0000","last_peered":"2024-09-21T14:03:30.132517+0000","last_clean":"2024-09-21T14:03:30.132517+0000","last_became_active":"2024-09-21T13:59:32.203172+0000","last_became_peered":"2024-09-21T13:59:32.203172+0000","last_unstale":"2024-09-21T14:03:30.132517+0000","last_undegraded":"2024-09-21T14:03:30.132517+0000","last_fullsized":"2024-09-21T14:03:30.132517+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T14:03:04.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:03:04.953658+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:14.479415+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"300'14","reported_seq":838,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.131308+0000","last_change":"2024-09-21T13:59:32.220898+0000","last_active":"2024-09-21T14:03:30.131308+0000","last_peered":"2024-09-21T14:03:30.131308+0000","last_clean":"2024-09-21T14:03:30.131308+0000","last_became_active":"2024-09-21T13:59:32.206956+0000","last_became_peered":"2024-09-21T13:59:32.206956+0000","last_unstale":"2024-09-21T14:03:30.131308+0000","last_undegraded":"2024-09-21T14:03:30.131308+0000","last_fullsized":"2024-09-21T14:03:30.131308+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'11","last_scrub_stamp":"2024-09-21T13:56:31.616003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:56:31.616003+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:11.986942+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":691,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.133085+0000","last_change":"2024-09-21T14:01:41.248408+0000","last_active":"2024-09-21T14:03:30.133085+0000","last_peered":"2024-09-21T14:03:30.133085+0000","last_clean":"2024-09-21T14:03:30.133085+0000","last_became_active":"2024-09-21T14:01:41.246106+0000","last_became_peered":"2024-09-21T14:01:41.246106+0000","last_unstale":"2024-09-21T14:03:30.133085+0000","last_undegraded":"2024-09-21T14:03:30.133085+0000","last_fullsized":"2024-09-21T14:03:30.133085+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:39.258659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:39.258659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:22.284546+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":283,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.176815+0000","last_change":"2024-09-21T14:01:50.733055+0000","last_active":"2024-09-21T14:03:32.176815+0000","last_peered":"2024-09-21T14:03:32.176815+0000","last_clean":"2024-09-21T14:03:32.176815+0000","last_became_active":"2024-09-21T14:01:41.197501+0000","last_became_peered":"2024-09-21T14:01:41.197501+0000","last_unstale":"2024-09-21T14:03:32.176815+0000","last_undegraded":"2024-09-21T14:03:32.176815+0000","last_fullsized":"2024-09-21T14:03:32.176815+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:50.733007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:01:50.733007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:17.685126+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"290'12","reported_seq":665,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.123813+0000","last_change":"2024-09-21T13:59:32.247422+0000","last_active":"2024-09-21T14:03:30.123813+0000","last_peered":"2024-09-21T14:03:30.123813+0000","last_clean":"2024-09-21T14:03:30.123813+0000","last_became_active":"2024-09-21T13:59:32.211306+0000","last_became_peered":"2024-09-21T13:59:32.211306+0000","last_unstale":"2024-09-21T14:03:30.123813+0000","last_undegraded":"2024-09-21T14:03:30.123813+0000","last_fullsized":"2024-09-21T14:03:30.123813+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'6","last_scrub_stamp":"2024-09-21T13:55:12.066873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:55:12.066873+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:38.360712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":623,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.134571+0000","last_change":"2024-09-21T14:01:41.199742+0000","last_active":"2024-09-21T14:03:30.134571+0000","last_peered":"2024-09-21T14:03:30.134571+0000","last_clean":"2024-09-21T14:03:30.134571+0000","last_became_active":"2024-09-21T14:01:41.199321+0000","last_became_peered":"2024-09-21T14:01:41.199321+0000","last_unstale":"2024-09-21T14:03:30.134571+0000","last_undegraded":"2024-09-21T14:03:30.134571+0000","last_fullsized":"2024-09-21T14:03:30.134571+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:37.106238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:37.106238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:16.289450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":670,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.132095+0000","last_change":"2024-09-21T14:03:01.872986+0000","last_active":"2024-09-21T14:03:30.132095+0000","last_peered":"2024-09-21T14:03:30.132095+0000","last_clean":"2024-09-21T14:03:30.132095+0000","last_became_active":"2024-09-21T13:59:32.219525+0000","last_became_peered":"2024-09-21T13:59:32.219525+0000","last_unstale":"2024-09-21T14:03:30.132095+0000","last_undegraded":"2024-09-21T14:03:30.132095+0000","last_fullsized":"2024-09-21T14:03:30.132095+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:01.872928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:01.872928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:09.296117+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"290'15","reported_seq":628,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.124503+0000","last_change":"2024-09-21T13:59:32.238689+0000","last_active":"2024-09-21T14:03:30.124503+0000","last_peered":"2024-09-21T14:03:30.124503+0000","last_clean":"2024-09-21T14:03:30.124503+0000","last_became_active":"2024-09-21T13:59:32.209166+0000","last_became_peered":"2024-09-21T13:59:32.209166+0000","last_unstale":"2024-09-21T14:03:30.124503+0000","last_undegraded":"2024-09-21T14:03:30.124503+0000","last_fullsized":"2024-09-21T14:03:30.124503+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:55:04.990842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:55:04.990842+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"61'1","reported_seq":680,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.131546+0000","last_change":"2024-09-21T13:59:32.222826+0000","last_active":"2024-09-21T14:03:30.131546+0000","last_peered":"2024-09-21T14:03:30.131546+0000","last_clean":"2024-09-21T14:03:30.131546+0000","last_became_active":"2024-09-21T13:59:32.219515+0000","last_became_peered":"2024-09-21T13:59:32.219515+0000","last_unstale":"2024-09-21T14:03:30.131546+0000","last_undegraded":"2024-09-21T14:03:30.131546+0000","last_fullsized":"2024-09-21T14:03:30.131546+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:58:07.711028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:07.711028+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":613,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.123404+0000","last_change":"2024-09-21T13:59:32.211754+0000","last_active":"2024-09-21T14:03:30.123404+0000","last_peered":"2024-09-21T14:03:30.123404+0000","last_clean":"2024-09-21T14:03:30.123404+0000","last_became_active":"2024-09-21T13:59:32.210586+0000","last_became_peered":"2024-09-21T13:59:32.210586+0000","last_unstale":"2024-09-21T14:03:30.123404+0000","last_undegraded":"2024-09-21T14:03:30.123404+0000","last_fullsized":"2024-09-21T14:03:30.123404+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:28.014760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:28.014760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":673,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.167568+0000","last_change":"2024-09-21T14:02:29.731635+0000","last_active":"2024-09-21T14:03:32.167568+0000","last_peered":"2024-09-21T14:03:32.167568+0000","last_clean":"2024-09-21T14:03:32.167568+0000","last_became_active":"2024-09-21T13:59:32.205803+0000","last_became_peered":"2024-09-21T13:59:32.205803+0000","last_unstale":"2024-09-21T14:03:32.167568+0000","last_undegraded":"2024-09-21T14:03:32.167568+0000","last_fullsized":"2024-09-21T14:03:32.167568+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:29.731565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:02:29.731565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:53.473477+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"290'15","reported_seq":668,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.133949+0000","last_change":"2024-09-21T14:01:41.189676+0000","last_active":"2024-09-21T14:03:30.133949+0000","last_peered":"2024-09-21T14:03:30.133949+0000","last_clean":"2024-09-21T14:03:30.133949+0000","last_became_active":"2024-09-21T14:01:41.187904+0000","last_became_peered":"2024-09-21T14:01:41.187904+0000","last_unstale":"2024-09-21T14:03:30.133949+0000","last_undegraded":"2024-09-21T14:03:30.133949+0000","last_fullsized":"2024-09-21T14:03:30.133949+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:57:24.124604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:57:24.124604+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:18.325159+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":549,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.130615+0000","last_change":"2024-09-21T13:59:32.212325+0000","last_active":"2024-09-21T14:03:30.130615+0000","last_peered":"2024-09-21T14:03:30.130615+0000","last_clean":"2024-09-21T14:03:30.130615+0000","last_became_active":"2024-09-21T13:59:32.208749+0000","last_became_peered":"2024-09-21T13:59:32.208749+0000","last_unstale":"2024-09-21T14:03:30.130615+0000","last_undegraded":"2024-09-21T14:03:30.130615+0000","last_fullsized":"2024-09-21T14:03:30.130615+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:20.574745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:20.574745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:22.976952+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"300'14","reported_seq":771,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.122896+0000","last_change":"2024-09-21T13:59:32.208253+0000","last_active":"2024-09-21T14:03:30.122896+0000","last_peered":"2024-09-21T14:03:30.122896+0000","last_clean":"2024-09-21T14:03:30.122896+0000","last_became_active":"2024-09-21T13:59:32.206926+0000","last_became_peered":"2024-09-21T13:59:32.206926+0000","last_unstale":"2024-09-21T14:03:30.122896+0000","last_undegraded":"2024-09-21T14:03:30.122896+0000","last_fullsized":"2024-09-21T14:03:30.122896+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'10","last_scrub_stamp":"2024-09-21T13:54:20.946782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:20.946782+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:15.133039+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":717,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.180458+0000","last_change":"2024-09-21T14:01:41.200564+0000","last_active":"2024-09-21T14:03:32.180458+0000","last_peered":"2024-09-21T14:03:32.180458+0000","last_clean":"2024-09-21T14:03:32.180458+0000","last_became_active":"2024-09-21T14:01:41.200113+0000","last_became_peered":"2024-09-21T14:01:41.200113+0000","last_unstale":"2024-09-21T14:03:32.180458+0000","last_undegraded":"2024-09-21T14:03:32.180458+0000","last_fullsized":"2024-09-21T14:03:32.180458+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:54.805619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:54.805619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:48.669884+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"290'20","reported_seq":664,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.122711+0000","last_change":"2024-09-21T13:59:32.236530+0000","last_active":"2024-09-21T14:03:30.122711+0000","last_peered":"2024-09-21T14:03:30.122711+0000","last_clean":"2024-09-21T14:03:30.122711+0000","last_became_active":"2024-09-21T13:59:32.208332+0000","last_became_peered":"2024-09-21T13:59:32.208332+0000","last_unstale":"2024-09-21T14:03:30.122711+0000","last_undegraded":"2024-09-21T14:03:30.122711+0000","last_fullsized":"2024-09-21T14:03:30.122711+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:57:37.671391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:57:37.671391+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:55.297241+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":656,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.131058+0000","last_change":"2024-09-21T13:59:32.212737+0000","last_active":"2024-09-21T14:03:30.131058+0000","last_peered":"2024-09-21T14:03:30.131058+0000","last_clean":"2024-09-21T14:03:30.131058+0000","last_became_active":"2024-09-21T13:59:32.209804+0000","last_became_peered":"2024-09-21T13:59:32.209804+0000","last_unstale":"2024-09-21T14:03:30.131058+0000","last_undegraded":"2024-09-21T14:03:30.131058+0000","last_fullsized":"2024-09-21T14:03:30.131058+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:25.571256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:25.571256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:15.928161+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"346'10","reported_seq":771,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.133850+0000","last_change":"2024-09-21T14:01:41.207030+0000","last_active":"2024-09-21T14:03:30.133850+0000","last_peered":"2024-09-21T14:03:30.133850+0000","last_clean":"2024-09-21T14:03:30.133850+0000","last_became_active":"2024-09-21T14:01:41.206303+0000","last_became_peered":"2024-09-21T14:01:41.206303+0000","last_unstale":"2024-09-21T14:03:30.133850+0000","last_undegraded":"2024-09-21T14:03:30.133850+0000","last_fullsized":"2024-09-21T14:03:30.133850+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-21T13:59:32.104224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:32.104224+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":561,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.132371+0000","last_change":"2024-09-21T14:03:27.961613+0000","last_active":"2024-09-21T14:03:30.132371+0000","last_peered":"2024-09-21T14:03:30.132371+0000","last_clean":"2024-09-21T14:03:30.132371+0000","last_became_active":"2024-09-21T13:59:32.211472+0000","last_became_peered":"2024-09-21T13:59:32.211472+0000","last_unstale":"2024-09-21T14:03:30.132371+0000","last_undegraded":"2024-09-21T14:03:30.132371+0000","last_fullsized":"2024-09-21T14:03:30.132371+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:27.961544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:27.961544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:33.496172+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"290'19","reported_seq":781,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.179722+0000","last_change":"2024-09-21T14:02:34.670058+0000","last_active":"2024-09-21T14:03:32.179722+0000","last_peered":"2024-09-21T14:03:32.179722+0000","last_clean":"2024-09-21T14:03:32.179722+0000","last_became_active":"2024-09-21T13:58:17.771497+0000","last_became_peered":"2024-09-21T13:58:17.771497+0000","last_unstale":"2024-09-21T14:03:32.179722+0000","last_undegraded":"2024-09-21T14:03:32.179722+0000","last_fullsized":"2024-09-21T14:03:32.179722+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'19","last_scrub_stamp":"2024-09-21T14:02:34.669962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:34.669962+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:44.716931+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":298,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.132320+0000","last_change":"2024-09-21T14:02:57.807136+0000","last_active":"2024-09-21T14:03:30.132320+0000","last_peered":"2024-09-21T14:03:30.132320+0000","last_clean":"2024-09-21T14:03:30.132320+0000","last_became_active":"2024-09-21T13:59:32.201568+0000","last_became_peered":"2024-09-21T13:59:32.201568+0000","last_unstale":"2024-09-21T14:03:30.132320+0000","last_undegraded":"2024-09-21T14:03:30.132320+0000","last_fullsized":"2024-09-21T14:03:30.132320+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:57.807059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:02:57.807059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:09.290442+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":731,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.164699+0000","last_change":"2024-09-21T14:01:41.185588+0000","last_active":"2024-09-21T14:03:32.164699+0000","last_peered":"2024-09-21T14:03:32.164699+0000","last_clean":"2024-09-21T14:03:32.164699+0000","last_became_active":"2024-09-21T14:01:41.185293+0000","last_became_peered":"2024-09-21T14:01:41.185293+0000","last_unstale":"2024-09-21T14:03:32.164699+0000","last_undegraded":"2024-09-21T14:03:32.164699+0000","last_fullsized":"2024-09-21T14:03:32.164699+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:35.409237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:35.409237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:18.234150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":629,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.123237+0000","last_change":"2024-09-21T14:02:57.225650+0000","last_active":"2024-09-21T14:03:30.123237+0000","last_peered":"2024-09-21T14:03:30.123237+0000","last_clean":"2024-09-21T14:03:30.123237+0000","last_became_active":"2024-09-21T14:01:41.199671+0000","last_became_peered":"2024-09-21T14:01:41.199671+0000","last_unstale":"2024-09-21T14:03:30.123237+0000","last_undegraded":"2024-09-21T14:03:30.123237+0000","last_fullsized":"2024-09-21T14:03:30.123237+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:57.225602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:02:57.225602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:26.897407+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"290'15","reported_seq":783,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.164871+0000","last_change":"2024-09-21T14:02:56.315766+0000","last_active":"2024-09-21T14:03:32.164871+0000","last_peered":"2024-09-21T14:03:32.164871+0000","last_clean":"2024-09-21T14:03:32.164871+0000","last_became_active":"2024-09-21T13:58:17.764120+0000","last_became_peered":"2024-09-21T13:58:17.764120+0000","last_unstale":"2024-09-21T14:03:32.164871+0000","last_undegraded":"2024-09-21T14:03:32.164871+0000","last_fullsized":"2024-09-21T14:03:32.164871+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'15","last_scrub_stamp":"2024-09-21T14:02:56.315702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:56.315702+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:11.328147+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":660,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.131943+0000","last_change":"2024-09-21T13:59:32.223082+0000","last_active":"2024-09-21T14:03:30.131943+0000","last_peered":"2024-09-21T14:03:30.131943+0000","last_clean":"2024-09-21T14:03:30.131943+0000","last_became_active":"2024-09-21T13:59:32.221854+0000","last_became_peered":"2024-09-21T13:59:32.221854+0000","last_unstale":"2024-09-21T14:03:30.131943+0000","last_undegraded":"2024-09-21T14:03:30.131943+0000","last_fullsized":"2024-09-21T14:03:30.131943+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:17.691193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:17.691193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":596,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.123551+0000","last_change":"2024-09-21T13:57:12.703746+0000","last_active":"2024-09-21T14:03:30.123551+0000","last_peered":"2024-09-21T14:03:30.123551+0000","last_clean":"2024-09-21T14:03:30.123551+0000","last_became_active":"2024-09-21T13:57:12.703441+0000","last_became_peered":"2024-09-21T13:57:12.703441+0000","last_unstale":"2024-09-21T14:03:30.123551+0000","last_undegraded":"2024-09-21T14:03:30.123551+0000","last_fullsized":"2024-09-21T14:03:30.123551+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:19.999125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:19.999125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":526,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.131792+0000","last_change":"2024-09-21T14:03:29.345499+0000","last_active":"2024-09-21T14:03:30.131792+0000","last_peered":"2024-09-21T14:03:30.131792+0000","last_clean":"2024-09-21T14:03:30.131792+0000","last_became_active":"2024-09-21T14:01:41.186934+0000","last_became_peered":"2024-09-21T14:01:41.186934+0000","last_unstale":"2024-09-21T14:03:30.131792+0000","last_undegraded":"2024-09-21T14:03:30.131792+0000","last_fullsized":"2024-09-21T14:03:30.131792+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:29.345427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:29.345427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:45.842768+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"290'19","reported_seq":740,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.123429+0000","last_change":"2024-09-21T13:59:32.277742+0000","last_active":"2024-09-21T14:03:30.123429+0000","last_peered":"2024-09-21T14:03:30.123429+0000","last_clean":"2024-09-21T14:03:30.123429+0000","last_became_active":"2024-09-21T13:59:32.231567+0000","last_became_peered":"2024-09-21T13:59:32.231567+0000","last_unstale":"2024-09-21T14:03:30.123429+0000","last_undegraded":"2024-09-21T14:03:30.123429+0000","last_fullsized":"2024-09-21T14:03:30.123429+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:54:22.978267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:22.978267+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:20.135882+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":628,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.134641+0000","last_change":"2024-09-21T14:03:01.448170+0000","last_active":"2024-09-21T14:03:30.134641+0000","last_peered":"2024-09-21T14:03:30.134641+0000","last_clean":"2024-09-21T14:03:30.134641+0000","last_became_active":"2024-09-21T14:01:41.190760+0000","last_became_peered":"2024-09-21T14:01:41.190760+0000","last_unstale":"2024-09-21T14:03:30.134641+0000","last_undegraded":"2024-09-21T14:03:30.134641+0000","last_fullsized":"2024-09-21T14:03:30.134641+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:01.448109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:03:01.448109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:28.748585+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":666,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.132678+0000","last_change":"2024-09-21T13:59:32.219508+0000","last_active":"2024-09-21T14:03:30.132678+0000","last_peered":"2024-09-21T14:03:30.132678+0000","last_clean":"2024-09-21T14:03:30.132678+0000","last_became_active":"2024-09-21T13:59:32.214004+0000","last_became_peered":"2024-09-21T13:59:32.214004+0000","last_unstale":"2024-09-21T14:03:30.132678+0000","last_undegraded":"2024-09-21T14:03:30.132678+0000","last_fullsized":"2024-09-21T14:03:30.132678+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:36.525417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:36.525417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":662,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.123734+0000","last_change":"2024-09-21T13:59:28.124373+0000","last_active":"2024-09-21T14:03:30.123734+0000","last_peered":"2024-09-21T14:03:30.123734+0000","last_clean":"2024-09-21T14:03:30.123734+0000","last_became_active":"2024-09-21T13:58:17.778115+0000","last_became_peered":"2024-09-21T13:58:17.778115+0000","last_unstale":"2024-09-21T14:03:30.123734+0000","last_undegraded":"2024-09-21T14:03:30.123734+0000","last_fullsized":"2024-09-21T14:03:30.123734+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:28.124296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:28.124296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"290'8","reported_seq":686,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.121868+0000","last_change":"2024-09-21T13:59:26.034457+0000","last_active":"2024-09-21T14:03:30.121868+0000","last_peered":"2024-09-21T14:03:30.121868+0000","last_clean":"2024-09-21T14:03:30.121868+0000","last_became_active":"2024-09-21T13:58:17.775079+0000","last_became_peered":"2024-09-21T13:58:17.775079+0000","last_unstale":"2024-09-21T14:03:30.121868+0000","last_undegraded":"2024-09-21T14:03:30.121868+0000","last_fullsized":"2024-09-21T14:03:30.121868+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'8","last_scrub_stamp":"2024-09-21T13:59:26.034372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:26.034372+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":588,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.134082+0000","last_change":"2024-09-21T14:01:41.196335+0000","last_active":"2024-09-21T14:03:30.134082+0000","last_peered":"2024-09-21T14:03:30.134082+0000","last_clean":"2024-09-21T14:03:30.134082+0000","last_became_active":"2024-09-21T14:01:41.195989+0000","last_became_peered":"2024-09-21T14:01:41.195989+0000","last_unstale":"2024-09-21T14:03:30.134082+0000","last_undegraded":"2024-09-21T14:03:30.134082+0000","last_fullsized":"2024-09-21T14:03:30.134082+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:16.457445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:16.457445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:20.288594+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":291,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.175954+0000","last_change":"2024-09-21T14:03:30.728296+0000","last_active":"2024-09-21T14:03:32.175954+0000","last_peered":"2024-09-21T14:03:32.175954+0000","last_clean":"2024-09-21T14:03:32.175954+0000","last_became_active":"2024-09-21T13:59:32.204380+0000","last_became_peered":"2024-09-21T13:59:32.204380+0000","last_unstale":"2024-09-21T14:03:32.175954+0000","last_undegraded":"2024-09-21T14:03:32.175954+0000","last_fullsized":"2024-09-21T14:03:32.175954+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:30.727972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:30.727972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:43.035198+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":599,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.130533+0000","last_change":"2024-09-21T13:59:32.212615+0000","last_active":"2024-09-21T14:03:30.130533+0000","last_peered":"2024-09-21T14:03:30.130533+0000","last_clean":"2024-09-21T14:03:30.130533+0000","last_became_active":"2024-09-21T13:59:32.210134+0000","last_became_peered":"2024-09-21T13:59:32.210134+0000","last_unstale":"2024-09-21T14:03:30.130533+0000","last_undegraded":"2024-09-21T14:03:30.130533+0000","last_fullsized":"2024-09-21T14:03:30.130533+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:08.852789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:55:08.852789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:10.940052+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":715,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.166679+0000","last_change":"2024-09-21T14:01:41.187792+0000","last_active":"2024-09-21T14:03:32.166679+0000","last_peered":"2024-09-21T14:03:32.166679+0000","last_clean":"2024-09-21T14:03:32.166679+0000","last_became_active":"2024-09-21T14:01:41.187438+0000","last_became_peered":"2024-09-21T14:01:41.187438+0000","last_unstale":"2024-09-21T14:03:32.166679+0000","last_undegraded":"2024-09-21T14:03:32.166679+0000","last_fullsized":"2024-09-21T14:03:32.166679+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:17.706325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:17.706325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:40.383161+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":642,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.121961+0000","last_change":"2024-09-21T13:58:13.776632+0000","last_active":"2024-09-21T14:03:30.121961+0000","last_peered":"2024-09-21T14:03:30.121961+0000","last_clean":"2024-09-21T14:03:30.121961+0000","last_became_active":"2024-09-21T13:57:12.704042+0000","last_became_peered":"2024-09-21T13:57:12.704042+0000","last_unstale":"2024-09-21T14:03:30.121961+0000","last_undegraded":"2024-09-21T14:03:30.121961+0000","last_fullsized":"2024-09-21T14:03:30.121961+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:13.776572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:13.776572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"290'19","reported_seq":323,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-09-21T14:03:32.176863+0000","last_change":"2024-09-21T14:03:31.762681+0000","last_active":"2024-09-21T14:03:32.176863+0000","last_peered":"2024-09-21T14:03:32.176863+0000","last_clean":"2024-09-21T14:03:32.176863+0000","last_became_active":"2024-09-21T14:01:41.194606+0000","last_became_peered":"2024-09-21T14:01:41.194606+0000","last_unstale":"2024-09-21T14:03:32.176863+0000","last_undegraded":"2024-09-21T14:03:32.176863+0000","last_fullsized":"2024-09-21T14:03:32.176863+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'19","last_scrub_stamp":"2024-09-21T14:03:31.762605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:03:31.762605+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:58.699819+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":569,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T14:03:30.132323+0000","last_change":"2024-09-21T13:59:32.203045+0000","last_active":"2024-09-21T14:03:30.132323+0000","last_peered":"2024-09-21T14:03:30.132323+0000","last_clean":"2024-09-21T14:03:30.132323+0000","last_became_active":"2024-09-21T13:59:32.202225+0000","last_became_peered":"2024-09-21T13:59:32.202225+0000","last_unstale":"2024-09-21T14:03:30.132323+0000","last_undegraded":"2024-09-21T14:03:30.132323+0000","last_fullsized":"2024-09-21T14:03:30.132323+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:40.650832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:40.650832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":3693674496,"num_objects":56361,"num_object_clones":0,"num_object_copies":169083,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56361,"num_write_kb":3607104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11056447488,"data_stored":11056447488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56361,"ondisk_log_size":56361,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1111,"num_read_kb":868,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":175,"num_bytes_recovered":2024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":687,"ondisk_log_size":687,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":245,"num_read_kb":240,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":375688,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":139,"num_write":73,"num_write_kb":1036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1031048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":298,"seq":1279900254257,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1171336,"kb_used_data":845228,"kb_used_omap":4,"kb_used_meta":326075,"kb_avail":92573816,"statfs":{"total":95995035648,"available":94795587584,"internally_reserved":0,"allocated":865513472,"data_stored":864592934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":333901480},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463097,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1923956,"kb_used_data":1567908,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":91821196,"statfs":{"total":95995035648,"available":94024904704,"internally_reserved":0,"allocated":1605537792,"data_stored":1604632499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":364575174},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":345,"seq":1481763717143,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1193464,"kb_used_data":1040228,"kb_used_omap":4,"kb_used_meta":153211,"kb_avail":92551688,"statfs":{"total":95995035648,"available":94772928512,"internally_reserved":0,"allocated":1065193472,"data_stored":1064281611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":156888739},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920966,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2212564,"kb_used_data":1798212,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":91532588,"statfs":{"total":95995035648,"available":93729370112,"internally_reserved":0,"allocated":1841369088,"data_stored":1840431645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":424278469},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,5],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":240,"seq":1030792151117,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1410580,"kb_used_data":1057600,"kb_used_omap":4,"kb_used_meta":352955,"kb_avail":92334572,"statfs":{"total":95995035648,"available":94550601728,"internally_reserved":0,"allocated":1082982400,"data_stored":1082042289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4443,"internal_metadata":361426597},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":130,"seq":558345748619,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1896556,"kb_used_data":1310428,"kb_used_omap":4,"kb_used_meta":586107,"kb_avail":91848596,"statfs":{"total":95995035648,"available":94052962304,"internally_reserved":0,"allocated":1341878272,"data_stored":1340974295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4441,"internal_metadata":600174247},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575064,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1318260,"kb_used_data":705376,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":92426892,"statfs":{"total":95995035648,"available":94645137408,"internally_reserved":0,"allocated":722305024,"data_stored":721339543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":627571141},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738590,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2874892,"kb_used_data":2293944,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":90870260,"statfs":{"total":95995035648,"available":93051146240,"internally_reserved":0,"allocated":2348998656,"data_stored":2348073241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":594868677},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2381971456,"data_stored":2381971456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":726073344,"data_stored":726073344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1373962240,"data_stored":1373962240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1110704128,"data_stored":1110704128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1872166912,"data_stored":1872166912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1076232192,"data_stored":1076232192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1642463232,"data_stored":1642463232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":872873984,"data_stored":872873984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T14:03:36.564 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:03:36.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:36.575 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:36.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:03:36.676 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:03:36.680 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:36 smithi187 ceph-mon[33032]: from='client.16122 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:03:36.715 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:36.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:36.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:03:36.817 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:36.852 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T14:03:36.852 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:03:36.858 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:36.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:36.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:36.870 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 31 16 61793 61777 124.53 85.5625 0.0124977 0.00802492 2024-09-21T14:03:36.961 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:03:36.998 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:36.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:36.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:03:37.100 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:03:37.144 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:37.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:37.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:03:37.246 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:03:37.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:37.287 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:37.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:03:37.388 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:03:37.433 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:37.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:37.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:03:37.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:37 smithi049 ceph-mon[30080]: pgmap v749: 158 pgs: 158 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 88 MiB/s wr, 1.40k op/s; 18 B/s, 2 objects/s recovering 2024-09-21T14:03:37.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:37 smithi049 ceph-mon[30080]: 6.1e scrub starts 2024-09-21T14:03:37.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:37 smithi049 ceph-mon[30080]: 6.1e scrub ok 2024-09-21T14:03:37.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:37 smithi049 ceph-mon[30080]: 6.c scrub starts 2024-09-21T14:03:37.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:37 smithi049 ceph-mon[30080]: 6.c scrub ok 2024-09-21T14:03:37.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:37 smithi049 ceph-mon[40489]: pgmap v749: 158 pgs: 158 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 88 MiB/s wr, 1.40k op/s; 18 B/s, 2 objects/s recovering 2024-09-21T14:03:37.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:37 smithi049 ceph-mon[40489]: 6.1e scrub starts 2024-09-21T14:03:37.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:37 smithi049 ceph-mon[40489]: 6.1e scrub ok 2024-09-21T14:03:37.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:37 smithi049 ceph-mon[40489]: 6.c scrub starts 2024-09-21T14:03:37.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:37 smithi049 ceph-mon[40489]: 6.c scrub ok 2024-09-21T14:03:37.535 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:03:37.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:37.573 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:37.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:03:37.674 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:03:37.685 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:37 smithi187 ceph-mon[33032]: pgmap v749: 158 pgs: 158 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 88 MiB/s wr, 1.40k op/s; 18 B/s, 2 objects/s recovering 2024-09-21T14:03:37.685 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:37 smithi187 ceph-mon[33032]: 6.1e scrub starts 2024-09-21T14:03:37.685 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:37 smithi187 ceph-mon[33032]: 6.1e scrub ok 2024-09-21T14:03:37.686 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:37 smithi187 ceph-mon[33032]: 6.c scrub starts 2024-09-21T14:03:37.686 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:37 smithi187 ceph-mon[33032]: 6.c scrub ok 2024-09-21T14:03:37.719 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:37.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:37.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:03:37.740 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T14:03:37.800 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T14:03:37.821 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:03:37.870 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 32 16 63264 63248 123.512 91.9375 0.0152279 0.00809154 2024-09-21T14:03:37.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:37.892 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:37.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:03:37.993 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:03:38.061 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:38.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:38.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:03:38.162 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:03:38.211 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:38.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:38.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:03:38.219 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-09-21T14:03:38.229 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:03:38.313 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:38.347 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:38.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:38.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:38.449 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:03:38.494 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:38.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:38.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:03:38.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:38 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3092008893' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:03:38.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:38 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:03:38.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:38 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3092008893' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:03:38.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:38 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:03:38.560 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:03:38.595 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:03:38.625 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:38.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:38.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:03:38.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:38 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3092008893' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:03:38.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:38 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:03:38.727 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:03:38.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:38.762 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:38.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:03:38.864 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:03:38.870 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 33 16 64584 64568 122.269 82.5 0.0111807 0.00817372 2024-09-21T14:03:38.917 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:38.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:38.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:03:39.019 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:03:39.054 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:39.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:39.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:03:39.156 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:39.203 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:39.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:39.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:39.235 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T14:03:39.235 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:03:39.305 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:39.326 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:39.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:39.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:39.428 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:03:39.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:39 smithi187 ceph-mon[33032]: pgmap v750: 158 pgs: 158 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.3 KiB/s rd, 81 MiB/s wr, 1.29k op/s; 13 B/s, 1 objects/s recovering 2024-09-21T14:03:39.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:39 smithi187 ceph-mon[33032]: 5.1c scrub starts 2024-09-21T14:03:39.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:39 smithi187 ceph-mon[33032]: 5.1c scrub ok 2024-09-21T14:03:39.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:39 smithi187 ceph-mon[33032]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:03:39.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:39 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T14:03:39.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:39 smithi187 ceph-mon[33032]: osdmap e397: 8 total, 8 up, 8 in 2024-09-21T14:03:39.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:39 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:03:39.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:39 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:03:39.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:39 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T14:03:39.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:39 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T14:03:39.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:39 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:03:39.449 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:39.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:39.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:03:39.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:39 smithi049 ceph-mon[40489]: pgmap v750: 158 pgs: 158 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.3 KiB/s rd, 81 MiB/s wr, 1.29k op/s; 13 B/s, 1 objects/s recovering 2024-09-21T14:03:39.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:39 smithi049 ceph-mon[40489]: 5.1c scrub starts 2024-09-21T14:03:39.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:39 smithi049 ceph-mon[40489]: 5.1c scrub ok 2024-09-21T14:03:39.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:39 smithi049 ceph-mon[40489]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:03:39.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:39 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T14:03:39.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:39 smithi049 ceph-mon[40489]: osdmap e397: 8 total, 8 up, 8 in 2024-09-21T14:03:39.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:39 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:03:39.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:39 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:03:39.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:39 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T14:03:39.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:39 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T14:03:39.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:39 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:03:39.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:39 smithi049 ceph-mon[30080]: pgmap v750: 158 pgs: 158 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.3 KiB/s rd, 81 MiB/s wr, 1.29k op/s; 13 B/s, 1 objects/s recovering 2024-09-21T14:03:39.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:39 smithi049 ceph-mon[30080]: 5.1c scrub starts 2024-09-21T14:03:39.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:39 smithi049 ceph-mon[30080]: 5.1c scrub ok 2024-09-21T14:03:39.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:39 smithi049 ceph-mon[30080]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:03:39.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:39 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T14:03:39.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:39 smithi049 ceph-mon[30080]: osdmap e397: 8 total, 8 up, 8 in 2024-09-21T14:03:39.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:39 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:03:39.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:39 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:03:39.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:39 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T14:03:39.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:39 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T14:03:39.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:39 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:03:39.550 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:03:39.580 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:39.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:39.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:03:39.682 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:39.714 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:39.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:39.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:39.815 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:39.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:39.850 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:39.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:39.871 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 34 16 65587 65571 120.516 62.6875 0.00333991 0.00828952 2024-09-21T14:03:39.944 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:03:39 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:03:39.951 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:03:39.987 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:39.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:39.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:03:40.089 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:03:40.122 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:40.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:40.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:03:40.153 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T14:03:40.224 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:03:40.245 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:40.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:40.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:03:40.347 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:03:40.394 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:40.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:40.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:03:40.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:40 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:03:40.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:40 smithi187 ceph-mon[33032]: osdmap e398: 8 total, 8 up, 8 in 2024-09-21T14:03:40.496 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:03:40.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:40 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:03:40.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:40 smithi049 ceph-mon[40489]: osdmap e398: 8 total, 8 up, 8 in 2024-09-21T14:03:40.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:40 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:03:40.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:40 smithi049 ceph-mon[30080]: osdmap e398: 8 total, 8 up, 8 in 2024-09-21T14:03:40.534 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:03:40.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:40.547 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:40.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:03:40.648 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:03:40.683 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:40.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:40.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:03:40.785 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:03:40.821 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:40.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:40.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:03:40.871 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 35 16 66824 66808 119.282 77.3125 0.0161284 0.00837835 2024-09-21T14:03:40.922 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:03:40.952 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:40.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:40.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:03:41.004 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:03:40 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:03:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:03:41.054 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:03:41.088 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:41.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:41.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:03:41.190 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:03:41.227 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:41.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:41.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:03:41.329 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:41.358 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T14:03:41.358 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:03:41.366 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:41.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:41.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:41.468 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:03:41.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:41 smithi049 ceph-mon[40489]: pgmap v752: 158 pgs: 158 active+clean; 3.9 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 652 B/s rd, 105 MiB/s wr, 1.68k op/s; 14 B/s, 1 objects/s recovering 2024-09-21T14:03:41.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:41 smithi049 ceph-mon[30080]: pgmap v752: 158 pgs: 158 active+clean; 3.9 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 652 B/s rd, 105 MiB/s wr, 1.68k op/s; 14 B/s, 1 objects/s recovering 2024-09-21T14:03:41.506 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:41.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:41.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:03:41.555 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 4, 6, 2, 3, 7, 5] 2024-09-21T14:03:41.555 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T14:03:41.555 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-21T14:03:41.555 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd dump -f json-pretty 2024-09-21T14:03:41.609 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:03:41.660 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:41.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:41.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:03:41.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:41 smithi187 ceph-mon[33032]: pgmap v752: 158 pgs: 158 active+clean; 3.9 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 652 B/s rd, 105 MiB/s wr, 1.68k op/s; 14 B/s, 1 objects/s recovering 2024-09-21T14:03:41.761 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:03:41.783 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:41.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:41.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:03:41.872 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 36 16 68021 68005 118.042 74.8125 0.00534681 0.00846662 2024-09-21T14:03:41.885 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:03:41.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:41.923 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:41.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:03:42.024 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:03:42.061 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:42.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:42.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:03:42.163 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:03:42.200 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:42.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:42.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:03:42.279 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T14:03:42.302 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:42.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:42.330 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:42.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:42.431 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:03:42.459 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:42.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:42.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:03:42.561 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:03:42.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:42.639 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:42.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:03:42.740 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:03:42.775 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:42.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:42.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:03:42.872 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 37 16 69408 69392 117.194 86.6875 0.00747948 0.00852749 2024-09-21T14:03:42.877 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:03:42.913 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:42.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:42.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:03:43.014 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:03:43.027 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:03:43.038 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:03:43.039 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:03:43.039 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:03:43.080 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:03:43.080 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:03:43.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:43.081 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:43.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:03:43.123 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:03:43.123 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:03:43.159 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:03:43.159 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:03:43.182 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:43.195 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:03:43.195 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:03:43.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:43.216 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:43.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:43.233 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:03:43.234 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:03:43.271 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:03:43.271 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:03:43.303 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:03:43.303 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:03:43.317 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:03:43.371 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:43.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:43.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:03:43.473 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:43.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:43 smithi049 ceph-mon[40489]: pgmap v754: 158 pgs: 158 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.4 KiB/s rd, 75 MiB/s wr, 1.20k op/s 2024-09-21T14:03:43.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:43 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2968942362' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:03:43.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:43 smithi049 ceph-mon[30080]: pgmap v754: 158 pgs: 158 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.4 KiB/s rd, 75 MiB/s wr, 1.20k op/s 2024-09-21T14:03:43.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:43 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2968942362' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:03:43.510 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:43.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:43.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:43.611 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:03:43.651 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:43.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:43.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:03:43.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:43 smithi187 ceph-mon[33032]: pgmap v754: 158 pgs: 158 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.4 KiB/s rd, 75 MiB/s wr, 1.20k op/s 2024-09-21T14:03:43.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:43 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2968942362' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:03:43.753 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:03:43.787 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:43.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:43.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:03:43.872 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 38 16 70768 70752 116.347 85 0.00968713 0.00859027 2024-09-21T14:03:43.890 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:03:43.927 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:43.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:43.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:03:44.029 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:03:44.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:44.064 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:44.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:03:44.166 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:03:44.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:44.190 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:44.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:03:44.291 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:03:44.312 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:44.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:44.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:03:44.323 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T14:03:44.414 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:03:44.444 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:44.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:44.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:03:44.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:44 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:03:44.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:44 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:03:44.546 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:03:44.584 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:44.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:44.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:03:44.686 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:03:44.688 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T14:03:44.688 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-09-21T14:03:44.688 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:44.688 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.16068.0:71157 10.f e398/381 10:f1554f68:::benchmark_data_smithi049_176811_object71156:head v 398'4541, mlcod=398'4539)", 2024-09-21T14:03:44.688 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:03:44.541317+0000", 2024-09-21T14:03:44.688 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.0033141199999999998, 2024-09-21T14:03:44.688 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0033405869999999999, 2024-09-21T14:03:44.688 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:03:44.688 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:03:44.688 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T14:03:44.688 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:03:44.688 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:44.689 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:03:44.689 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.541317+0000", 2024-09-21T14:03:44.689 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:03:44.689 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:44.689 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:44.689 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:03:44.689 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.541317+0000", 2024-09-21T14:03:44.689 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:03:44.689 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:44.689 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:44.689 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:03:44.689 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.541318+0000", 2024-09-21T14:03:44.689 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9299999999999998e-07 2024-09-21T14:03:44.689 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:44.690 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:44.690 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:03:44.690 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.541352+0000", 2024-09-21T14:03:44.690 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3778000000000001e-05 2024-09-21T14:03:44.690 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:44.690 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:44.690 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:03:44.690 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.541361+0000", 2024-09-21T14:03:44.690 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3139999999999993e-06 2024-09-21T14:03:44.690 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:44.690 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:44.690 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:03:44.690 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.541367+0000", 2024-09-21T14:03:44.690 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0270000000000001e-06 2024-09-21T14:03:44.691 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:44.691 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:44.691 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:03:44.691 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.541539+0000", 2024-09-21T14:03:44.691 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00017204699999999999 2024-09-21T14:03:44.691 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:44.691 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:44.691 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:03:44.691 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.541560+0000", 2024-09-21T14:03:44.691 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0676e-05 2024-09-21T14:03:44.691 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:44.691 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:44.691 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:44.691 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:44.691 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:44.692 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:71159 10.8 10:1b756742:::benchmark_data_smithi049_176811_object71158:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-09-21T14:03:44.692 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:03:44.541403+0000", 2024-09-21T14:03:44.692 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.003228416, 2024-09-21T14:03:44.692 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0033161940000000002, 2024-09-21T14:03:44.692 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:03:44.692 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:03:44.692 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "waiting for sub ops", 2024-09-21T14:03:44.692 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:03:44.692 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:03:44.692 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:03:44.692 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 71159 2024-09-21T14:03:44.692 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:44.692 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:03:44.692 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:44.692 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:03:44.693 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.541403+0000", 2024-09-21T14:03:44.693 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:03:44.693 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:44.693 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:44.693 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:03:44.693 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.541403+0000", 2024-09-21T14:03:44.693 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:03:44.693 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:44.693 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:44.693 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:03:44.693 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.541405+0000", 2024-09-21T14:03:44.693 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.463e-06 2024-09-21T14:03:44.693 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:44.693 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:44.693 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:03:44.694 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.541431+0000", 2024-09-21T14:03:44.694 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6485000000000001e-05 2024-09-21T14:03:44.694 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:44.694 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:44.694 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:03:44.694 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.541446+0000", 2024-09-21T14:03:44.694 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5150999999999999e-05 2024-09-21T14:03:44.694 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:44.694 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:44.694 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:03:44.694 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.541453+0000", 2024-09-21T14:03:44.694 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8279999999999997e-06 2024-09-21T14:03:44.694 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:44.695 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:44.695 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:03:44.695 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.541533+0000", 2024-09-21T14:03:44.695 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9882999999999994e-05 2024-09-21T14:03:44.695 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:44.695 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:44.695 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:03:44.695 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.541818+0000", 2024-09-21T14:03:44.695 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00028529299999999999 2024-09-21T14:03:44.695 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:44.695 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:44.695 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:03:44.695 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.541961+0000", 2024-09-21T14:03:44.695 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000143127 2024-09-21T14:03:44.696 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:44.696 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:44.696 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:44.696 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:44.696 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T14:03:44.696 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 2 2024-09-21T14:03:44.696 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T14:03:44.697 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:44 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:03:44.723 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:44.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:44.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:03:44.825 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:03:44.861 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:44.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:44.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:03:44.872 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 39 16 71619 71603 114.727 53.1875 0.00977333 0.00871147 2024-09-21T14:03:44.964 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:44.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:45.000 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:45.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:45.101 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:03:45.133 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:45.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:45.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:03:45.235 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:03:45.270 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:45.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:45.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:03:45.372 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:45.406 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:45.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:45.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:45.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:45 smithi049 ceph-mon[40489]: pgmap v755: 158 pgs: 158 active+clean; 4.2 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.50k op/s 2024-09-21T14:03:45.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:45 smithi049 ceph-mon[30080]: pgmap v755: 158 pgs: 158 active+clean; 4.2 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.50k op/s 2024-09-21T14:03:45.508 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:03:45.547 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:45.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:45.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:03:45.649 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:03:45.681 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:45.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:45.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:03:45.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:45 smithi187 ceph-mon[33032]: pgmap v755: 158 pgs: 158 active+clean; 4.2 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.50k op/s 2024-09-21T14:03:45.783 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:45.819 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:45.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:45.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:45.874 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:2024-09-21T14:03:45.875522+0000 min lat: 0.00132701 max lat: 0.311772 avg lat: 0.00876124 2024-09-21T14:03:45.874 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T14:03:45.874 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 40 16 73032 73016 114.062 88.3125 0.010021 0.00876124 2024-09-21T14:03:45.921 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:03:45.954 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:45.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:45.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:03:46.056 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:03:46.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:46.097 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:46.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:03:46.198 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:03:46.249 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:46.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:46.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:03:46.300 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T14:03:46.300 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T14:03:46.300 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 398, 2024-09-21T14:03:46.300 INFO:teuthology.orchestra.run.smithi049.stdout: "fsid": "40a3d2e8-781f-11ef-baf6-efdc52797490", 2024-09-21T14:03:46.300 INFO:teuthology.orchestra.run.smithi049.stdout: "created": "2024-09-21T13:42:57.838275+0000", 2024-09-21T14:03:46.301 INFO:teuthology.orchestra.run.smithi049.stdout: "modified": "2024-09-21T14:03:40.109444+0000", 2024-09-21T14:03:46.301 INFO:teuthology.orchestra.run.smithi049.stdout: "last_up_change": "2024-09-21T14:01:40.158454+0000", 2024-09-21T14:03:46.301 INFO:teuthology.orchestra.run.smithi049.stdout: "last_in_change": "2024-09-21T13:58:16.730861+0000", 2024-09-21T14:03:46.301 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-21T14:03:46.301 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_num": 5801984, 2024-09-21T14:03:46.301 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_set": [ 2024-09-21T14:03:46.301 INFO:teuthology.orchestra.run.smithi049.stdout: "noscrub", 2024-09-21T14:03:46.301 INFO:teuthology.orchestra.run.smithi049.stdout: "pglog_hardlimit", 2024-09-21T14:03:46.301 INFO:teuthology.orchestra.run.smithi049.stdout: "purged_snapdirs", 2024-09-21T14:03:46.301 INFO:teuthology.orchestra.run.smithi049.stdout: "recovery_deletes", 2024-09-21T14:03:46.301 INFO:teuthology.orchestra.run.smithi049.stdout: "sortbitwise" 2024-09-21T14:03:46.301 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T14:03:46.301 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_version": 22, 2024-09-21T14:03:46.301 INFO:teuthology.orchestra.run.smithi049.stdout: "full_ratio": 0.94999998807907104, 2024-09-21T14:03:46.301 INFO:teuthology.orchestra.run.smithi049.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-21T14:03:46.302 INFO:teuthology.orchestra.run.smithi049.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-21T14:03:46.302 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_snapshot": "", 2024-09-21T14:03:46.302 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_max": 10, 2024-09-21T14:03:46.302 INFO:teuthology.orchestra.run.smithi049.stdout: "max_osd": 8, 2024-09-21T14:03:46.302 INFO:teuthology.orchestra.run.smithi049.stdout: "require_min_compat_client": "luminous", 2024-09-21T14:03:46.302 INFO:teuthology.orchestra.run.smithi049.stdout: "min_compat_client": "luminous", 2024-09-21T14:03:46.302 INFO:teuthology.orchestra.run.smithi049.stdout: "require_osd_release": "squid", 2024-09-21T14:03:46.302 INFO:teuthology.orchestra.run.smithi049.stdout: "allow_crimson": false, 2024-09-21T14:03:46.302 INFO:teuthology.orchestra.run.smithi049.stdout: "pools": [ 2024-09-21T14:03:46.302 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.302 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 1, 2024-09-21T14:03:46.302 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": ".mgr", 2024-09-21T14:03:46.302 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:46:21.096244+0000", 2024-09-21T14:03:46.302 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T14:03:46.303 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T14:03:46.303 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T14:03:46.303 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T14:03:46.303 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T14:03:46.303 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T14:03:46.303 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T14:03:46.303 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T14:03:46.303 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T14:03:46.303 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T14:03:46.303 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T14:03:46.303 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T14:03:46.303 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T14:03:46.303 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 11, 2024-09-21T14:03:46.303 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 11, 2024-09-21T14:03:46.304 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 11, 2024-09-21T14:03:46.304 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 11, 2024-09-21T14:03:46.304 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 11, 2024-09-21T14:03:46.304 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T14:03:46.304 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T14:03:46.304 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T14:03:46.304 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T14:03:46.304 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T14:03:46.304 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T14:03:46.304 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T14:03:46.304 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.304 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "320", 2024-09-21T14:03:46.304 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T14:03:46.304 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T14:03:46.304 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "318", 2024-09-21T14:03:46.305 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T14:03:46.305 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T14:03:46.305 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T14:03:46.305 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T14:03:46.305 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T14:03:46.305 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T14:03:46.305 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T14:03:46.305 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T14:03:46.305 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T14:03:46.305 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T14:03:46.305 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T14:03:46.305 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T14:03:46.305 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T14:03:46.306 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T14:03:46.306 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T14:03:46.306 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T14:03:46.306 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T14:03:46.306 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T14:03:46.306 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T14:03:46.306 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T14:03:46.306 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T14:03:46.306 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T14:03:46.306 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T14:03:46.306 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.306 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T14:03:46.306 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T14:03:46.307 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T14:03:46.307 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T14:03:46.307 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T14:03:46.307 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T14:03:46.307 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T14:03:46.307 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T14:03:46.307 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T14:03:46.307 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T14:03:46.307 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T14:03:46.307 INFO:teuthology.orchestra.run.smithi049.stdout: "options": { 2024-09-21T14:03:46.307 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_max": 32, 2024-09-21T14:03:46.307 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_min": 1 2024-09-21T14:03:46.307 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.307 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T14:03:46.308 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr": {} 2024-09-21T14:03:46.308 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.308 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T14:03:46.308 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T14:03:46.308 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2.1800000667572021, 2024-09-21T14:03:46.308 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2.1800000667572021, 2024-09-21T14:03:46.308 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1, 2024-09-21T14:03:46.308 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2.1800000667572021, 2024-09-21T14:03:46.308 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2.1800000667572021, 2024-09-21T14:03:46.308 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.86855697631835938, 2024-09-21T14:03:46.308 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.87000000476837158, 2024-09-21T14:03:46.308 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.87000000476837158 2024-09-21T14:03:46.309 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:46.309 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.309 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.309 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 2, 2024-09-21T14:03:46.309 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "datapool", 2024-09-21T14:03:46.309 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:48:55.540712+0000", 2024-09-21T14:03:46.309 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 8193, 2024-09-21T14:03:46.309 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T14:03:46.309 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T14:03:46.309 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T14:03:46.309 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T14:03:46.309 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T14:03:46.309 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T14:03:46.309 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T14:03:46.309 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T14:03:46.310 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T14:03:46.310 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T14:03:46.310 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T14:03:46.310 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T14:03:46.310 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 3, 2024-09-21T14:03:46.310 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 3, 2024-09-21T14:03:46.310 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 3, 2024-09-21T14:03:46.310 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 3, 2024-09-21T14:03:46.310 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 3, 2024-09-21T14:03:46.310 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T14:03:46.310 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T14:03:46.310 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T14:03:46.310 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T14:03:46.311 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T14:03:46.311 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T14:03:46.311 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T14:03:46.311 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.311 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "63", 2024-09-21T14:03:46.311 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T14:03:46.311 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T14:03:46.311 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T14:03:46.311 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T14:03:46.311 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T14:03:46.311 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 2, 2024-09-21T14:03:46.311 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 63, 2024-09-21T14:03:46.311 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T14:03:46.312 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T14:03:46.312 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T14:03:46.312 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T14:03:46.312 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T14:03:46.312 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T14:03:46.312 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T14:03:46.312 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T14:03:46.312 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T14:03:46.312 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T14:03:46.312 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T14:03:46.312 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T14:03:46.312 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T14:03:46.312 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T14:03:46.312 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T14:03:46.312 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T14:03:46.313 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T14:03:46.313 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T14:03:46.313 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T14:03:46.313 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.313 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T14:03:46.313 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T14:03:46.313 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T14:03:46.313 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T14:03:46.313 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T14:03:46.313 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T14:03:46.313 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T14:03:46.313 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T14:03:46.313 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T14:03:46.313 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T14:03:46.314 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T14:03:46.314 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T14:03:46.314 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T14:03:46.314 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd": {} 2024-09-21T14:03:46.314 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.314 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T14:03:46.314 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T14:03:46.314 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2.6500000953674316, 2024-09-21T14:03:46.314 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2.6500000953674316, 2024-09-21T14:03:46.314 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 0.87999999523162842, 2024-09-21T14:03:46.314 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2.3299999237060547, 2024-09-21T14:03:46.314 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2.3299999237060547, 2024-09-21T14:03:46.314 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.86855697631835938, 2024-09-21T14:03:46.314 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.87000000476837158, 2024-09-21T14:03:46.314 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.87000000476837158 2024-09-21T14:03:46.315 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:46.315 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.315 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.315 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 3, 2024-09-21T14:03:46.315 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": ".rgw.root", 2024-09-21T14:03:46.315 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:48:56.417456+0000", 2024-09-21T14:03:46.315 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T14:03:46.315 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T14:03:46.315 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T14:03:46.315 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T14:03:46.315 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T14:03:46.315 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T14:03:46.316 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T14:03:46.316 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T14:03:46.316 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T14:03:46.316 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T14:03:46.316 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T14:03:46.316 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T14:03:46.316 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T14:03:46.316 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 32, 2024-09-21T14:03:46.316 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 32, 2024-09-21T14:03:46.316 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 32, 2024-09-21T14:03:46.316 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 32, 2024-09-21T14:03:46.316 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 32, 2024-09-21T14:03:46.317 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T14:03:46.317 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T14:03:46.317 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T14:03:46.317 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T14:03:46.317 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T14:03:46.317 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T14:03:46.317 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T14:03:46.317 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.317 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "62", 2024-09-21T14:03:46.317 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T14:03:46.317 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T14:03:46.318 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T14:03:46.318 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T14:03:46.318 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T14:03:46.318 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T14:03:46.318 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T14:03:46.318 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T14:03:46.318 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T14:03:46.318 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T14:03:46.318 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T14:03:46.318 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T14:03:46.318 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T14:03:46.319 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T14:03:46.319 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T14:03:46.319 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T14:03:46.319 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T14:03:46.319 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T14:03:46.319 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T14:03:46.319 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T14:03:46.319 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T14:03:46.319 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T14:03:46.319 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T14:03:46.319 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T14:03:46.319 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T14:03:46.319 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T14:03:46.319 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.320 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T14:03:46.320 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T14:03:46.320 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T14:03:46.320 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T14:03:46.320 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T14:03:46.320 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T14:03:46.320 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T14:03:46.320 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T14:03:46.320 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T14:03:46.320 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T14:03:46.320 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T14:03:46.321 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T14:03:46.321 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T14:03:46.321 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw": {} 2024-09-21T14:03:46.321 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.321 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T14:03:46.321 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T14:03:46.321 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2, 2024-09-21T14:03:46.321 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2, 2024-09-21T14:03:46.321 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1, 2024-09-21T14:03:46.321 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2, 2024-09-21T14:03:46.321 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2, 2024-09-21T14:03:46.321 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.86855697631835938, 2024-09-21T14:03:46.322 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.87000000476837158, 2024-09-21T14:03:46.322 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.87000000476837158 2024-09-21T14:03:46.322 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:46.322 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.322 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.322 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 4, 2024-09-21T14:03:46.322 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "default.rgw.log", 2024-09-21T14:03:46.322 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:48:58.128455+0000", 2024-09-21T14:03:46.322 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T14:03:46.322 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T14:03:46.322 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T14:03:46.322 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T14:03:46.323 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T14:03:46.323 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T14:03:46.323 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T14:03:46.323 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T14:03:46.323 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T14:03:46.323 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T14:03:46.323 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T14:03:46.323 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T14:03:46.323 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T14:03:46.323 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 32, 2024-09-21T14:03:46.323 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 32, 2024-09-21T14:03:46.324 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 32, 2024-09-21T14:03:46.324 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 32, 2024-09-21T14:03:46.324 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 32, 2024-09-21T14:03:46.324 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T14:03:46.324 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T14:03:46.324 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T14:03:46.324 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T14:03:46.324 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T14:03:46.324 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T14:03:46.324 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T14:03:46.324 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.324 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "64", 2024-09-21T14:03:46.324 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T14:03:46.325 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T14:03:46.325 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T14:03:46.325 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T14:03:46.325 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T14:03:46.325 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T14:03:46.325 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T14:03:46.325 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T14:03:46.325 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T14:03:46.325 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T14:03:46.325 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T14:03:46.325 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T14:03:46.325 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T14:03:46.326 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T14:03:46.326 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T14:03:46.326 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T14:03:46.326 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T14:03:46.326 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T14:03:46.326 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T14:03:46.326 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T14:03:46.326 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T14:03:46.326 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T14:03:46.326 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T14:03:46.327 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T14:03:46.327 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T14:03:46.327 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T14:03:46.327 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.327 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T14:03:46.327 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T14:03:46.327 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T14:03:46.327 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T14:03:46.327 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T14:03:46.328 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T14:03:46.328 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T14:03:46.328 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T14:03:46.328 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T14:03:46.328 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T14:03:46.328 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T14:03:46.328 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T14:03:46.328 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T14:03:46.328 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw": {} 2024-09-21T14:03:46.328 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.328 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T14:03:46.328 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T14:03:46.328 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2.25, 2024-09-21T14:03:46.328 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2.25, 2024-09-21T14:03:46.329 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1, 2024-09-21T14:03:46.329 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2.25, 2024-09-21T14:03:46.329 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2.25, 2024-09-21T14:03:46.329 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.86855697631835938, 2024-09-21T14:03:46.329 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.87000000476837158, 2024-09-21T14:03:46.329 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.87000000476837158 2024-09-21T14:03:46.329 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:46.329 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.329 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.329 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 5, 2024-09-21T14:03:46.329 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "default.rgw.control", 2024-09-21T14:03:46.329 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:49:00.069361+0000", 2024-09-21T14:03:46.329 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T14:03:46.329 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T14:03:46.329 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T14:03:46.330 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T14:03:46.330 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T14:03:46.330 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T14:03:46.330 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T14:03:46.330 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T14:03:46.330 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T14:03:46.330 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T14:03:46.330 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T14:03:46.330 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T14:03:46.330 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T14:03:46.330 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 32, 2024-09-21T14:03:46.330 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 32, 2024-09-21T14:03:46.330 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 32, 2024-09-21T14:03:46.330 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 32, 2024-09-21T14:03:46.331 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 32, 2024-09-21T14:03:46.331 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T14:03:46.331 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T14:03:46.331 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T14:03:46.331 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T14:03:46.331 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T14:03:46.331 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T14:03:46.331 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T14:03:46.331 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.331 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "66", 2024-09-21T14:03:46.331 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T14:03:46.331 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T14:03:46.331 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T14:03:46.331 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T14:03:46.331 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T14:03:46.332 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T14:03:46.332 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T14:03:46.332 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T14:03:46.332 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T14:03:46.332 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T14:03:46.332 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T14:03:46.332 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T14:03:46.332 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T14:03:46.332 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T14:03:46.332 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T14:03:46.332 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T14:03:46.332 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T14:03:46.332 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T14:03:46.332 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T14:03:46.333 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T14:03:46.333 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T14:03:46.333 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T14:03:46.333 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T14:03:46.333 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T14:03:46.333 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T14:03:46.333 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T14:03:46.333 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.333 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T14:03:46.333 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T14:03:46.333 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T14:03:46.333 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T14:03:46.333 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T14:03:46.333 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T14:03:46.333 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T14:03:46.334 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T14:03:46.334 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T14:03:46.334 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T14:03:46.334 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T14:03:46.334 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T14:03:46.334 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T14:03:46.334 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw": {} 2024-09-21T14:03:46.334 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.334 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T14:03:46.334 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T14:03:46.334 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 1.25, 2024-09-21T14:03:46.334 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 1.25, 2024-09-21T14:03:46.335 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1, 2024-09-21T14:03:46.336 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T14:03:46.336 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 1.25, 2024-09-21T14:03:46.336 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 1.25, 2024-09-21T14:03:46.337 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.86855697631835938, 2024-09-21T14:03:46.337 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.87000000476837158, 2024-09-21T14:03:46.337 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.87000000476837158 2024-09-21T14:03:46.337 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:46.337 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.337 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.337 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 6, 2024-09-21T14:03:46.337 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "default.rgw.meta", 2024-09-21T14:03:46.337 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:49:02.127399+0000", 2024-09-21T14:03:46.337 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T14:03:46.337 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T14:03:46.337 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T14:03:46.337 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T14:03:46.337 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T14:03:46.338 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T14:03:46.338 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T14:03:46.338 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T14:03:46.338 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T14:03:46.338 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T14:03:46.338 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T14:03:46.338 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T14:03:46.338 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T14:03:46.338 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 32, 2024-09-21T14:03:46.338 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 32, 2024-09-21T14:03:46.338 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 32, 2024-09-21T14:03:46.338 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 32, 2024-09-21T14:03:46.338 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 32, 2024-09-21T14:03:46.338 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T14:03:46.339 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T14:03:46.339 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T14:03:46.339 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T14:03:46.339 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T14:03:46.339 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T14:03:46.339 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T14:03:46.339 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.339 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "68", 2024-09-21T14:03:46.339 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T14:03:46.339 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T14:03:46.339 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T14:03:46.339 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T14:03:46.339 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T14:03:46.340 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T14:03:46.340 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T14:03:46.340 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T14:03:46.340 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T14:03:46.340 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T14:03:46.340 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T14:03:46.340 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T14:03:46.340 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T14:03:46.340 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T14:03:46.340 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T14:03:46.340 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T14:03:46.340 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T14:03:46.340 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T14:03:46.340 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T14:03:46.341 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T14:03:46.341 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T14:03:46.341 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T14:03:46.341 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T14:03:46.341 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T14:03:46.341 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T14:03:46.341 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T14:03:46.341 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.341 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T14:03:46.341 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T14:03:46.341 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T14:03:46.341 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T14:03:46.341 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T14:03:46.341 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T14:03:46.341 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T14:03:46.342 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T14:03:46.342 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T14:03:46.342 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T14:03:46.342 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T14:03:46.342 INFO:teuthology.orchestra.run.smithi049.stdout: "options": { 2024-09-21T14:03:46.342 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_bias": 4 2024-09-21T14:03:46.342 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.342 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T14:03:46.342 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw": {} 2024-09-21T14:03:46.342 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.342 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T14:03:46.342 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T14:03:46.342 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2, 2024-09-21T14:03:46.342 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2, 2024-09-21T14:03:46.343 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1, 2024-09-21T14:03:46.343 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2, 2024-09-21T14:03:46.343 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2, 2024-09-21T14:03:46.343 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.86855697631835938, 2024-09-21T14:03:46.343 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.87000000476837158, 2024-09-21T14:03:46.343 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.87000000476837158 2024-09-21T14:03:46.343 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:46.343 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.343 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.343 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 10, 2024-09-21T14:03:46.343 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "unique_pool_3", 2024-09-21T14:03:46.343 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T14:02:55.287153+0000", 2024-09-21T14:03:46.343 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T14:03:46.343 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T14:03:46.343 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T14:03:46.344 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T14:03:46.344 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T14:03:46.344 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T14:03:46.344 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T14:03:46.344 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T14:03:46.344 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T14:03:46.344 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T14:03:46.344 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T14:03:46.344 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T14:03:46.344 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T14:03:46.344 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 16, 2024-09-21T14:03:46.344 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 16, 2024-09-21T14:03:46.344 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 16, 2024-09-21T14:03:46.344 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 16, 2024-09-21T14:03:46.344 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 16, 2024-09-21T14:03:46.344 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T14:03:46.345 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T14:03:46.345 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T14:03:46.345 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T14:03:46.345 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T14:03:46.345 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T14:03:46.345 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T14:03:46.345 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.345 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "386", 2024-09-21T14:03:46.345 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T14:03:46.345 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T14:03:46.345 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T14:03:46.345 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T14:03:46.345 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T14:03:46.345 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T14:03:46.345 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T14:03:46.346 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T14:03:46.346 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T14:03:46.346 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T14:03:46.346 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T14:03:46.346 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T14:03:46.346 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T14:03:46.346 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T14:03:46.346 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T14:03:46.346 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T14:03:46.346 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T14:03:46.346 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T14:03:46.346 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T14:03:46.346 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T14:03:46.346 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T14:03:46.347 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T14:03:46.347 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T14:03:46.347 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T14:03:46.347 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T14:03:46.347 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T14:03:46.347 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.347 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T14:03:46.347 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T14:03:46.347 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T14:03:46.347 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T14:03:46.347 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T14:03:46.347 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T14:03:46.347 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T14:03:46.348 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T14:03:46.348 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T14:03:46.348 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T14:03:46.348 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T14:03:46.348 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T14:03:46.348 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T14:03:46.348 INFO:teuthology.orchestra.run.smithi049.stdout: "rados": {} 2024-09-21T14:03:46.348 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.348 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T14:03:46.348 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T14:03:46.348 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 3, 2024-09-21T14:03:46.349 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 3, 2024-09-21T14:03:46.349 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1, 2024-09-21T14:03:46.349 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 3, 2024-09-21T14:03:46.349 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 3, 2024-09-21T14:03:46.349 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.86855697631835938, 2024-09-21T14:03:46.349 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.87000000476837158, 2024-09-21T14:03:46.349 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.87000000476837158 2024-09-21T14:03:46.349 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:46.349 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:46.349 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T14:03:46.349 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-09-21T14:03:46.349 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.349 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 0, 2024-09-21T14:03:46.349 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "2595dca9-6219-4e0c-8728-751eb102e73a", 2024-09-21T14:03:46.349 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T14:03:46.349 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T14:03:46.350 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T14:03:46.350 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 0.1215972900390625, 2024-09-21T14:03:46.350 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T14:03:46.350 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T14:03:46.350 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 8, 2024-09-21T14:03:46.350 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 345, 2024-09-21T14:03:46.350 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T14:03:46.350 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T14:03:46.350 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T14:03:46.351 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:03:46.351 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.351 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:03:46.351 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6802", 2024-09-21T14:03:46.351 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T14:03:46.351 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.351 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.351 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:03:46.351 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6803", 2024-09-21T14:03:46.351 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T14:03:46.351 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:46.351 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:46.351 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.352 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T14:03:46.352 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:03:46.352 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.352 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:03:46.352 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6804", 2024-09-21T14:03:46.352 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T14:03:46.352 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.352 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.352 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:03:46.352 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6805", 2024-09-21T14:03:46.353 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T14:03:46.353 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:46.353 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:46.353 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.353 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T14:03:46.353 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:03:46.353 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.353 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:03:46.353 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6808", 2024-09-21T14:03:46.353 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T14:03:46.353 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.353 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.353 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:03:46.353 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6809", 2024-09-21T14:03:46.354 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T14:03:46.354 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:46.354 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:46.354 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.354 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T14:03:46.354 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:03:46.354 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.354 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:03:46.354 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6806", 2024-09-21T14:03:46.355 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T14:03:46.355 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.355 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.355 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:03:46.355 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6807", 2024-09-21T14:03:46.355 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T14:03:46.355 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:46.355 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:46.355 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.355 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6803/2095561412", 2024-09-21T14:03:46.355 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6805/2095561412", 2024-09-21T14:03:46.355 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6809/2095561412", 2024-09-21T14:03:46.356 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6807/2095561412", 2024-09-21T14:03:46.356 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T14:03:46.356 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T14:03:46.356 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T14:03:46.356 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:46.356 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.356 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.356 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 1, 2024-09-21T14:03:46.356 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "0ba327f9-5524-4a40-abcc-729ded0b1917", 2024-09-21T14:03:46.356 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T14:03:46.356 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T14:03:46.357 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T14:03:46.357 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T14:03:46.357 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T14:03:46.357 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T14:03:46.357 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 13, 2024-09-21T14:03:46.357 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 381, 2024-09-21T14:03:46.357 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T14:03:46.357 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T14:03:46.357 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T14:03:46.357 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:03:46.357 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.358 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:03:46.358 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6810", 2024-09-21T14:03:46.358 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T14:03:46.358 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.358 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.358 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:03:46.358 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6811", 2024-09-21T14:03:46.358 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T14:03:46.358 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:46.358 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:46.358 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.359 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T14:03:46.359 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:03:46.359 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.359 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:03:46.359 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6812", 2024-09-21T14:03:46.359 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T14:03:46.359 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.359 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.360 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:03:46.360 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6813", 2024-09-21T14:03:46.360 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T14:03:46.360 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:46.360 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:46.360 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.360 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T14:03:46.360 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:03:46.360 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.360 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:03:46.360 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6816", 2024-09-21T14:03:46.360 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T14:03:46.361 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.361 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.361 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:03:46.361 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6817", 2024-09-21T14:03:46.361 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T14:03:46.361 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:46.361 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:46.361 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.361 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T14:03:46.361 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:03:46.361 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.361 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:03:46.361 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6814", 2024-09-21T14:03:46.361 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T14:03:46.361 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.362 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.362 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:03:46.362 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6815", 2024-09-21T14:03:46.362 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T14:03:46.362 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:46.362 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:46.362 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.362 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6811/1382840527", 2024-09-21T14:03:46.362 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6813/1382840527", 2024-09-21T14:03:46.362 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6817/1382840527", 2024-09-21T14:03:46.362 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6815/1382840527", 2024-09-21T14:03:46.362 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T14:03:46.363 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T14:03:46.363 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T14:03:46.363 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:46.363 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.363 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.363 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 2, 2024-09-21T14:03:46.363 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "27a45750-204d-48fd-bb8f-ebb01273343d", 2024-09-21T14:03:46.363 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T14:03:46.363 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T14:03:46.363 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T14:03:46.363 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 0.8268585205078125, 2024-09-21T14:03:46.363 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 18, 2024-09-21T14:03:46.363 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 100, 2024-09-21T14:03:46.364 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 130, 2024-09-21T14:03:46.364 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 392, 2024-09-21T14:03:46.364 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 101, 2024-09-21T14:03:46.364 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T14:03:46.364 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T14:03:46.364 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:03:46.364 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.364 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:03:46.364 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6818", 2024-09-21T14:03:46.364 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T14:03:46.364 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.364 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.364 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:03:46.364 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6819", 2024-09-21T14:03:46.364 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T14:03:46.365 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:46.365 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:46.365 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.365 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T14:03:46.365 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:03:46.365 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.365 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:03:46.365 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6820", 2024-09-21T14:03:46.365 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T14:03:46.365 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.365 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.365 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:03:46.365 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6821", 2024-09-21T14:03:46.365 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T14:03:46.366 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:46.366 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:46.366 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.366 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T14:03:46.366 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:03:46.366 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.366 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:03:46.366 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6824", 2024-09-21T14:03:46.366 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T14:03:46.366 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.366 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.366 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:03:46.366 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6825", 2024-09-21T14:03:46.366 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T14:03:46.366 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:46.367 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:46.367 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.367 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T14:03:46.367 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:03:46.367 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.367 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:03:46.367 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6822", 2024-09-21T14:03:46.367 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T14:03:46.367 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.367 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.367 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:03:46.367 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6823", 2024-09-21T14:03:46.367 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T14:03:46.367 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:46.368 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:46.368 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.368 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6819/1945314741", 2024-09-21T14:03:46.368 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6821/1945314741", 2024-09-21T14:03:46.368 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6825/1945314741", 2024-09-21T14:03:46.368 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6823/1945314741", 2024-09-21T14:03:46.368 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T14:03:46.368 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T14:03:46.368 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T14:03:46.368 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:46.368 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.368 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.368 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 3, 2024-09-21T14:03:46.368 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "528bc390-3dc9-4f3b-ab75-d7412c9625db", 2024-09-21T14:03:46.368 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T14:03:46.369 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T14:03:46.369 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T14:03:46.369 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T14:03:46.369 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 27, 2024-09-21T14:03:46.369 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 214, 2024-09-21T14:03:46.369 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 240, 2024-09-21T14:03:46.369 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 346, 2024-09-21T14:03:46.369 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 215, 2024-09-21T14:03:46.369 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T14:03:46.369 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T14:03:46.369 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:03:46.369 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.369 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:03:46.369 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6826", 2024-09-21T14:03:46.370 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T14:03:46.370 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.370 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.370 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:03:46.370 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6827", 2024-09-21T14:03:46.370 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T14:03:46.370 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:46.370 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:46.370 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.370 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T14:03:46.370 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:03:46.370 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.370 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:03:46.370 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6828", 2024-09-21T14:03:46.370 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T14:03:46.371 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.371 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.371 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:03:46.371 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6829", 2024-09-21T14:03:46.371 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T14:03:46.371 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:46.371 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:46.371 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.371 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T14:03:46.371 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:03:46.371 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.371 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:03:46.371 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6832", 2024-09-21T14:03:46.371 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T14:03:46.371 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.372 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.372 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:03:46.372 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6833", 2024-09-21T14:03:46.372 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T14:03:46.372 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:46.372 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:46.372 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.372 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T14:03:46.372 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:03:46.372 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.372 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:03:46.372 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6830", 2024-09-21T14:03:46.372 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T14:03:46.372 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.373 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.373 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:03:46.373 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6831", 2024-09-21T14:03:46.373 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T14:03:46.373 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:46.373 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:46.373 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:03:46.374 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.374 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6827/886735721", 2024-09-21T14:03:46.374 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6829/886735721", 2024-09-21T14:03:46.374 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6833/886735721", 2024-09-21T14:03:46.374 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6831/886735721", 2024-09-21T14:03:46.374 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T14:03:46.374 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T14:03:46.374 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T14:03:46.374 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:46.375 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.375 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.375 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 4, 2024-09-21T14:03:46.375 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "6fc64698-2d37-4db0-bf3d-c7e612c45163", 2024-09-21T14:03:46.375 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T14:03:46.375 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T14:03:46.375 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T14:03:46.375 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T14:03:46.375 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T14:03:46.375 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T14:03:46.375 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 33, 2024-09-21T14:03:46.375 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 392, 2024-09-21T14:03:46.375 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T14:03:46.376 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T14:03:46.376 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T14:03:46.376 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:03:46.376 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.376 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:03:46.376 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6800", 2024-09-21T14:03:46.376 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T14:03:46.376 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.376 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.376 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:03:46.376 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6801", 2024-09-21T14:03:46.376 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T14:03:46.376 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:46.376 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:46.377 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.377 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T14:03:46.377 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:03:46.377 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.377 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:03:46.377 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6802", 2024-09-21T14:03:46.377 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T14:03:46.377 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.377 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.377 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:03:46.377 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6803", 2024-09-21T14:03:46.377 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T14:03:46.377 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:46.377 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:46.378 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.378 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T14:03:46.378 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:03:46.378 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.378 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:03:46.378 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6806", 2024-09-21T14:03:46.378 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T14:03:46.378 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.378 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.378 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:03:46.378 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6807", 2024-09-21T14:03:46.378 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T14:03:46.378 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:46.378 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:46.379 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.379 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T14:03:46.379 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:03:46.379 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.379 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:03:46.379 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6804", 2024-09-21T14:03:46.379 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T14:03:46.379 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.379 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.379 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:03:46.379 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6805", 2024-09-21T14:03:46.379 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T14:03:46.379 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:46.379 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:46.380 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.380 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.187:6801/1702447776", 2024-09-21T14:03:46.380 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.187:6803/1702447776", 2024-09-21T14:03:46.380 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.187:6807/1702447776", 2024-09-21T14:03:46.380 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.187:6805/1702447776", 2024-09-21T14:03:46.380 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T14:03:46.380 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T14:03:46.380 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T14:03:46.380 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:46.380 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.380 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.380 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 5, 2024-09-21T14:03:46.380 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "8d7f62a1-7e92-4e06-9cad-86b96be514d8", 2024-09-21T14:03:46.380 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T14:03:46.380 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T14:03:46.381 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T14:03:46.381 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T14:03:46.381 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 39, 2024-09-21T14:03:46.381 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 324, 2024-09-21T14:03:46.381 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 345, 2024-09-21T14:03:46.381 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 381, 2024-09-21T14:03:46.381 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 325, 2024-09-21T14:03:46.381 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T14:03:46.381 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T14:03:46.381 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:03:46.381 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.381 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:03:46.381 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6808", 2024-09-21T14:03:46.382 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1595353965 2024-09-21T14:03:46.382 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.382 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.382 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:03:46.382 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6809", 2024-09-21T14:03:46.382 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1595353965 2024-09-21T14:03:46.382 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:46.382 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:46.382 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.382 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T14:03:46.382 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:03:46.382 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.382 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:03:46.382 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6810", 2024-09-21T14:03:46.383 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1595353965 2024-09-21T14:03:46.383 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.383 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.383 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:03:46.383 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6811", 2024-09-21T14:03:46.383 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1595353965 2024-09-21T14:03:46.383 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:46.383 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:46.383 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.383 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T14:03:46.383 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:03:46.383 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.383 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:03:46.384 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6814", 2024-09-21T14:03:46.384 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1595353965 2024-09-21T14:03:46.384 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.384 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.384 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:03:46.384 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6815", 2024-09-21T14:03:46.384 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1595353965 2024-09-21T14:03:46.384 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:46.384 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:46.384 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.384 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T14:03:46.384 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:03:46.384 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.384 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:03:46.385 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6812", 2024-09-21T14:03:46.385 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1595353965 2024-09-21T14:03:46.385 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.385 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.385 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:03:46.385 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6813", 2024-09-21T14:03:46.385 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1595353965 2024-09-21T14:03:46.385 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:46.385 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:46.385 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.385 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.187:6809/1595353965", 2024-09-21T14:03:46.385 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.187:6811/1595353965", 2024-09-21T14:03:46.385 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.187:6815/1595353965", 2024-09-21T14:03:46.386 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.187:6813/1595353965", 2024-09-21T14:03:46.386 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T14:03:46.386 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T14:03:46.386 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T14:03:46.386 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:46.386 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.386 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.386 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 6, 2024-09-21T14:03:46.386 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "3a7afd13-26ce-444c-9cc0-81ce2eab6ce5", 2024-09-21T14:03:46.386 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T14:03:46.386 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T14:03:46.386 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T14:03:46.387 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T14:03:46.387 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T14:03:46.387 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T14:03:46.387 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 47, 2024-09-21T14:03:46.387 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 381, 2024-09-21T14:03:46.387 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T14:03:46.387 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T14:03:46.387 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T14:03:46.387 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:03:46.387 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.387 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:03:46.387 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6816", 2024-09-21T14:03:46.387 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T14:03:46.388 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.388 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.388 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:03:46.388 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6817", 2024-09-21T14:03:46.388 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T14:03:46.388 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:46.388 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:46.388 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.388 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T14:03:46.388 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:03:46.388 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.389 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:03:46.389 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6818", 2024-09-21T14:03:46.389 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T14:03:46.389 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.389 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.389 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:03:46.389 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6819", 2024-09-21T14:03:46.389 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T14:03:46.389 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:46.389 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:46.389 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.389 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T14:03:46.389 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:03:46.390 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.390 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:03:46.390 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6822", 2024-09-21T14:03:46.390 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T14:03:46.390 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.390 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.390 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:03:46.390 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6823", 2024-09-21T14:03:46.390 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T14:03:46.390 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:46.390 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:46.391 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.391 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T14:03:46.391 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:03:46.391 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.391 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:03:46.391 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6820", 2024-09-21T14:03:46.391 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T14:03:46.391 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.391 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.391 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:03:46.391 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6821", 2024-09-21T14:03:46.391 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T14:03:46.392 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:46.392 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:46.392 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.392 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.187:6817/2728426525", 2024-09-21T14:03:46.392 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.187:6819/2728426525", 2024-09-21T14:03:46.392 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.187:6823/2728426525", 2024-09-21T14:03:46.392 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.187:6821/2728426525", 2024-09-21T14:03:46.392 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T14:03:46.392 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T14:03:46.392 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T14:03:46.392 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:46.392 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.393 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.393 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 7, 2024-09-21T14:03:46.393 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "1e08d891-8747-402e-b4c2-aa3802ce8537", 2024-09-21T14:03:46.393 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T14:03:46.393 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T14:03:46.393 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T14:03:46.393 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T14:03:46.393 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 54, 2024-09-21T14:03:46.393 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 272, 2024-09-21T14:03:46.393 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 298, 2024-09-21T14:03:46.393 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 381, 2024-09-21T14:03:46.394 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 273, 2024-09-21T14:03:46.394 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T14:03:46.394 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T14:03:46.394 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:03:46.394 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.394 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:03:46.394 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6824", 2024-09-21T14:03:46.394 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 13894964 2024-09-21T14:03:46.394 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.394 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.395 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:03:46.395 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6825", 2024-09-21T14:03:46.395 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 13894964 2024-09-21T14:03:46.395 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:46.395 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:46.395 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.395 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T14:03:46.395 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:03:46.395 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.396 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:03:46.396 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6826", 2024-09-21T14:03:46.396 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 13894964 2024-09-21T14:03:46.396 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.396 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.396 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:03:46.396 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6827", 2024-09-21T14:03:46.396 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 13894964 2024-09-21T14:03:46.396 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:46.396 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:46.396 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.396 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T14:03:46.397 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:03:46.397 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.397 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:03:46.397 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6830", 2024-09-21T14:03:46.397 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 13894964 2024-09-21T14:03:46.397 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.397 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.397 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:03:46.397 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6831", 2024-09-21T14:03:46.397 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 13894964 2024-09-21T14:03:46.397 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:46.397 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:46.397 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.397 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T14:03:46.398 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:03:46.398 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.398 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:03:46.398 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6828", 2024-09-21T14:03:46.398 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 13894964 2024-09-21T14:03:46.398 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.398 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.398 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:03:46.398 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6829", 2024-09-21T14:03:46.398 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 13894964 2024-09-21T14:03:46.398 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:46.398 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:46.398 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.399 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.187:6825/13894964", 2024-09-21T14:03:46.399 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.187:6827/13894964", 2024-09-21T14:03:46.399 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.187:6831/13894964", 2024-09-21T14:03:46.399 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.187:6829/13894964", 2024-09-21T14:03:46.399 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T14:03:46.399 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T14:03:46.399 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T14:03:46.399 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:46.399 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:46.399 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T14:03:46.399 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_xinfo": [ 2024-09-21T14:03:46.399 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.399 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 0, 2024-09-21T14:03:46.399 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T14:03:46.400 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T14:03:46.400 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T14:03:46.400 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T14:03:46.400 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T14:03:46.400 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T14:02:59.701694+0000", 2024-09-21T14:03:46.400 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T14:03:46.400 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.400 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.400 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 1, 2024-09-21T14:03:46.400 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T14:03:46.400 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T14:03:46.400 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T14:03:46.400 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T14:03:46.400 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T14:03:46.401 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T14:02:55.079021+0000", 2024-09-21T14:03:46.401 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T14:03:46.401 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.401 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.401 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 2, 2024-09-21T14:03:46.401 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T14:03:46.401 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T14:03:46.401 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T14:03:46.401 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T14:03:46.401 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T14:03:46.401 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T14:02:48.643793+0000", 2024-09-21T14:03:46.401 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 100 2024-09-21T14:03:46.401 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.401 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.402 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 3, 2024-09-21T14:03:46.402 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T14:03:46.402 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T14:03:46.402 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T14:03:46.402 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T14:03:46.402 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T14:03:46.402 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T14:03:28.883427+0000", 2024-09-21T14:03:46.402 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 214 2024-09-21T14:03:46.402 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.402 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.402 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 4, 2024-09-21T14:03:46.402 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T14:03:46.402 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T14:03:46.402 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T14:03:46.403 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T14:03:46.403 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T14:03:46.403 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T14:03:07.515930+0000", 2024-09-21T14:03:46.403 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T14:03:46.403 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.403 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.403 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 5, 2024-09-21T14:03:46.403 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T14:03:46.403 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T14:03:46.403 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T14:03:46.403 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T14:03:46.403 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T14:03:46.403 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T14:03:06.752661+0000", 2024-09-21T14:03:46.403 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 324 2024-09-21T14:03:46.404 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.404 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.404 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 6, 2024-09-21T14:03:46.404 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T14:03:46.404 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T14:03:46.404 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T14:03:46.404 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T14:03:46.404 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T14:03:46.404 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T14:03:27.736192+0000", 2024-09-21T14:03:46.404 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T14:03:46.404 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.404 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.404 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 7, 2024-09-21T14:03:46.404 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T14:03:46.404 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T14:03:46.405 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T14:03:46.405 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T14:03:46.405 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T14:03:46.405 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T14:03:29.485204+0000", 2024-09-21T14:03:46.405 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 272 2024-09-21T14:03:46.405 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:46.405 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T14:03:46.405 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap": [ 2024-09-21T14:03:46.405 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.405 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "3.14", 2024-09-21T14:03:46.405 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-09-21T14:03:46.405 INFO:teuthology.orchestra.run.smithi049.stdout: 7, 2024-09-21T14:03:46.405 INFO:teuthology.orchestra.run.smithi049.stdout: 6, 2024-09-21T14:03:46.405 INFO:teuthology.orchestra.run.smithi049.stdout: 0 2024-09-21T14:03:46.406 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:46.406 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:46.406 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T14:03:46.406 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap_items": [ 2024-09-21T14:03:46.406 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.406 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "4.2", 2024-09-21T14:03:46.406 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-09-21T14:03:46.406 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.406 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 1, 2024-09-21T14:03:46.406 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 2 2024-09-21T14:03:46.406 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:46.406 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:46.406 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.406 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.407 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "4.6", 2024-09-21T14:03:46.407 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-09-21T14:03:46.407 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.407 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 1, 2024-09-21T14:03:46.407 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 5 2024-09-21T14:03:46.407 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:46.407 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:46.407 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.407 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.407 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "4.d", 2024-09-21T14:03:46.407 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-09-21T14:03:46.407 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:03:46.407 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 1, 2024-09-21T14:03:46.407 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 5 2024-09-21T14:03:46.407 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:46.408 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:03:46.408 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:46.408 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T14:03:46.408 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap_primaries": [], 2024-09-21T14:03:46.408 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_temp": [], 2024-09-21T14:03:46.408 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_temp": [], 2024-09-21T14:03:46.408 INFO:teuthology.orchestra.run.smithi049.stdout: "blocklist": { 2024-09-21T14:03:46.408 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/2731793917": "2024-09-22T13:49:27.311313+0000", 2024-09-21T14:03:46.408 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/799831425": "2024-09-22T13:49:27.311313+0000", 2024-09-21T14:03:46.408 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/2518197280": "2024-09-22T13:43:28.110499+0000", 2024-09-21T14:03:46.408 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3556829486": "2024-09-22T13:43:28.110499+0000", 2024-09-21T14:03:46.408 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/1249926115": "2024-09-22T13:49:27.311313+0000", 2024-09-21T14:03:46.408 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/231530138": "2024-09-22T13:43:28.110499+0000", 2024-09-21T14:03:46.408 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/4220781134": "2024-09-22T13:49:27.311313+0000", 2024-09-21T14:03:46.409 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/202325455": "2024-09-22T13:43:28.110499+0000", 2024-09-21T14:03:46.409 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/4129636473": "2024-09-22T13:43:56.633723+0000", 2024-09-21T14:03:46.409 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/2518197280": "2024-09-22T13:43:28.110499+0000", 2024-09-21T14:03:46.409 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3514741273": "2024-09-22T13:43:56.633723+0000", 2024-09-21T14:03:46.409 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/4129636473": "2024-09-22T13:43:56.633723+0000", 2024-09-21T14:03:46.409 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3125743776": "2024-09-22T13:43:56.633723+0000", 2024-09-21T14:03:46.409 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3600977363": "2024-09-22T13:43:56.633723+0000", 2024-09-21T14:03:46.409 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/3453571598": "2024-09-22T13:49:27.311313+0000", 2024-09-21T14:03:46.409 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/3453571598": "2024-09-22T13:49:27.311313+0000", 2024-09-21T14:03:46.409 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3553734432": "2024-09-22T13:49:27.311313+0000" 2024-09-21T14:03:46.409 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.409 INFO:teuthology.orchestra.run.smithi049.stdout: "range_blocklist": {}, 2024-09-21T14:03:46.409 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profiles": { 2024-09-21T14:03:46.409 INFO:teuthology.orchestra.run.smithi049.stdout: "default": { 2024-09-21T14:03:46.409 INFO:teuthology.orchestra.run.smithi049.stdout: "crush-failure-domain": "osd", 2024-09-21T14:03:46.410 INFO:teuthology.orchestra.run.smithi049.stdout: "k": "2", 2024-09-21T14:03:46.410 INFO:teuthology.orchestra.run.smithi049.stdout: "m": "1", 2024-09-21T14:03:46.410 INFO:teuthology.orchestra.run.smithi049.stdout: "plugin": "jerasure", 2024-09-21T14:03:46.410 INFO:teuthology.orchestra.run.smithi049.stdout: "technique": "reed_sol_van" 2024-09-21T14:03:46.410 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:46.410 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:03:46.410 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps_queue": [], 2024-09-21T14:03:46.410 INFO:teuthology.orchestra.run.smithi049.stdout: "new_removed_snaps": [], 2024-09-21T14:03:46.410 INFO:teuthology.orchestra.run.smithi049.stdout: "new_purged_snaps": [], 2024-09-21T14:03:46.410 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_node_flags": {}, 2024-09-21T14:03:46.410 INFO:teuthology.orchestra.run.smithi049.stdout: "device_class_flags": {}, 2024-09-21T14:03:46.410 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode": { 2024-09-21T14:03:46.410 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode_enabled": false, 2024-09-21T14:03:46.411 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_bucket_count": 0, 2024-09-21T14:03:46.411 INFO:teuthology.orchestra.run.smithi049.stdout: "degraded_stretch_mode": 0, 2024-09-21T14:03:46.411 INFO:teuthology.orchestra.run.smithi049.stdout: "recovering_stretch_mode": 0, 2024-09-21T14:03:46.411 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode_bucket": 0 2024-09-21T14:03:46.411 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:03:46.411 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T14:03:46.411 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:03:46.445 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:46.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:46.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:03:46.547 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:03:46.590 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:46.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:46.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:03:46.692 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:03:46.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:46.737 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:46.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:03:46.838 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:03:46.874 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 41 16 74292 74276 113.201 78.75 0.00850947 0.00882882 2024-09-21T14:03:46.876 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:46.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:46.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:03:46.978 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:03:47.017 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:47.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:47.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:03:47.119 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:03:47.159 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:47.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:47.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:03:47.261 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:03:47.304 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:47.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:47.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:03:47.406 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:03:47.436 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:47.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:47.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:03:47.479 INFO:tasks.thrashosds.thrasher:j is {'epoch': 398, 'fsid': '40a3d2e8-781f-11ef-baf6-efdc52797490', 'created': '2024-09-21T13:42:57.838275+0000', 'modified': '2024-09-21T14:03:40.109444+0000', 'last_up_change': '2024-09-21T14:01:40.158454+0000', 'last_in_change': '2024-09-21T13:58:16.730861+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 22, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 10, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T13:46:21.096244+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '320', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '318', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.180000066757202, 'score_stable': 2.180000066757202, 'optimal_score': 1, 'raw_score_acting': 2.180000066757202, 'raw_score_stable': 2.180000066757202, 'primary_affinity_weighted': 0.8685569763183594, 'average_primary_affinity': 0.8700000047683716, 'average_primary_affinity_weighted': 0.8700000047683716}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-21T13:48:55.540712+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 63, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 0.8685569763183594, 'average_primary_affinity': 0.8700000047683716, 'average_primary_affinity_weighted': 0.8700000047683716}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-21T13:48:56.417456+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8685569763183594, 'average_primary_affinity': 0.8700000047683716, 'average_primary_affinity_weighted': 0.8700000047683716}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-21T13:48:58.128455+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.8685569763183594, 'average_primary_affinity': 0.8700000047683716, 'average_primary_affinity_weighted': 0.8700000047683716}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-21T13:49:00.069361+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 0.8685569763183594, 'average_primary_affinity': 0.8700000047683716, 'average_primary_affinity_weighted': 0.8700000047683716}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-21T13:49:02.127399+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8685569763183594, 'average_primary_affinity': 0.8700000047683716, 'average_primary_affinity_weighted': 0.8700000047683716}}, {'pool': 10, 'pool_name': 'unique_pool_3', 'create_time': '2024-09-21T14:02:55.287153+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '386', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3, 'score_stable': 3, 'optimal_score': 1, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.8685569763183594, 'average_primary_affinity': 0.8700000047683716, 'average_primary_affinity_weighted': 0.8700000047683716}}], 'osds': [{'osd': 0, 'uuid': '2595dca9-6219-4e0c-8728-751eb102e73a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.1215972900390625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 345, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6802', 'nonce': 2095561412}, {'type': 'v1', 'addr': '172.21.15.49:6803', 'nonce': 2095561412}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6804', 'nonce': 2095561412}, {'type': 'v1', 'addr': '172.21.15.49:6805', 'nonce': 2095561412}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6808', 'nonce': 2095561412}, {'type': 'v1', 'addr': '172.21.15.49:6809', 'nonce': 2095561412}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6806', 'nonce': 2095561412}, {'type': 'v1', 'addr': '172.21.15.49:6807', 'nonce': 2095561412}]}, 'public_addr': '172.21.15.49:6803/2095561412', 'cluster_addr': '172.21.15.49:6805/2095561412', 'heartbeat_back_addr': '172.21.15.49:6809/2095561412', 'heartbeat_front_addr': '172.21.15.49:6807/2095561412', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '0ba327f9-5524-4a40-abcc-729ded0b1917', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 381, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6810', 'nonce': 1382840527}, {'type': 'v1', 'addr': '172.21.15.49:6811', 'nonce': 1382840527}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6812', 'nonce': 1382840527}, {'type': 'v1', 'addr': '172.21.15.49:6813', 'nonce': 1382840527}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6816', 'nonce': 1382840527}, {'type': 'v1', 'addr': '172.21.15.49:6817', 'nonce': 1382840527}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6814', 'nonce': 1382840527}, {'type': 'v1', 'addr': '172.21.15.49:6815', 'nonce': 1382840527}]}, 'public_addr': '172.21.15.49:6811/1382840527', 'cluster_addr': '172.21.15.49:6813/1382840527', 'heartbeat_back_addr': '172.21.15.49:6817/1382840527', 'heartbeat_front_addr': '172.21.15.49:6815/1382840527', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '27a45750-204d-48fd-bb8f-ebb01273343d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.8268585205078125, 'last_clean_begin': 18, 'last_clean_end': 100, 'up_from': 130, 'up_thru': 392, 'down_at': 101, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6818', 'nonce': 1945314741}, {'type': 'v1', 'addr': '172.21.15.49:6819', 'nonce': 1945314741}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6820', 'nonce': 1945314741}, {'type': 'v1', 'addr': '172.21.15.49:6821', 'nonce': 1945314741}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6824', 'nonce': 1945314741}, {'type': 'v1', 'addr': '172.21.15.49:6825', 'nonce': 1945314741}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6822', 'nonce': 1945314741}, {'type': 'v1', 'addr': '172.21.15.49:6823', 'nonce': 1945314741}]}, 'public_addr': '172.21.15.49:6819/1945314741', 'cluster_addr': '172.21.15.49:6821/1945314741', 'heartbeat_back_addr': '172.21.15.49:6825/1945314741', 'heartbeat_front_addr': '172.21.15.49:6823/1945314741', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '528bc390-3dc9-4f3b-ab75-d7412c9625db', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 27, 'last_clean_end': 214, 'up_from': 240, 'up_thru': 346, 'down_at': 215, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6826', 'nonce': 886735721}, {'type': 'v1', 'addr': '172.21.15.49:6827', 'nonce': 886735721}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6828', 'nonce': 886735721}, {'type': 'v1', 'addr': '172.21.15.49:6829', 'nonce': 886735721}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6832', 'nonce': 886735721}, {'type': 'v1', 'addr': '172.21.15.49:6833', 'nonce': 886735721}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6830', 'nonce': 886735721}, {'type': 'v1', 'addr': '172.21.15.49:6831', 'nonce': 886735721}]}, 'public_addr': '172.21.15.49:6827/886735721', 'cluster_addr': '172.21.15.49:6829/886735721', 'heartbeat_back_addr': '172.21.15.49:6833/886735721', 'heartbeat_front_addr': '172.21.15.49:6831/886735721', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '6fc64698-2d37-4db0-bf3d-c7e612c45163', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 392, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6800', 'nonce': 1702447776}, {'type': 'v1', 'addr': '172.21.15.187:6801', 'nonce': 1702447776}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6802', 'nonce': 1702447776}, {'type': 'v1', 'addr': '172.21.15.187:6803', 'nonce': 1702447776}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6806', 'nonce': 1702447776}, {'type': 'v1', 'addr': '172.21.15.187:6807', 'nonce': 1702447776}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6804', 'nonce': 1702447776}, {'type': 'v1', 'addr': '172.21.15.187:6805', 'nonce': 1702447776}]}, 'public_addr': '172.21.15.187:6801/1702447776', 'cluster_addr': '172.21.15.187:6803/1702447776', 'heartbeat_back_addr': '172.21.15.187:6807/1702447776', 'heartbeat_front_addr': '172.21.15.187:6805/1702447776', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '8d7f62a1-7e92-4e06-9cad-86b96be514d8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 39, 'last_clean_end': 324, 'up_from': 345, 'up_thru': 381, 'down_at': 325, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6808', 'nonce': 1595353965}, {'type': 'v1', 'addr': '172.21.15.187:6809', 'nonce': 1595353965}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6810', 'nonce': 1595353965}, {'type': 'v1', 'addr': '172.21.15.187:6811', 'nonce': 1595353965}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6814', 'nonce': 1595353965}, {'type': 'v1', 'addr': '172.21.15.187:6815', 'nonce': 1595353965}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6812', 'nonce': 1595353965}, {'type': 'v1', 'addr': '172.21.15.187:6813', 'nonce': 1595353965}]}, 'public_addr': '172.21.15.187:6809/1595353965', 'cluster_addr': '172.21.15.187:6811/1595353965', 'heartbeat_back_addr': '172.21.15.187:6815/1595353965', 'heartbeat_front_addr': '172.21.15.187:6813/1595353965', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '3a7afd13-26ce-444c-9cc0-81ce2eab6ce5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 381, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6816', 'nonce': 2728426525}, {'type': 'v1', 'addr': '172.21.15.187:6817', 'nonce': 2728426525}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6818', 'nonce': 2728426525}, {'type': 'v1', 'addr': '172.21.15.187:6819', 'nonce': 2728426525}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6822', 'nonce': 2728426525}, {'type': 'v1', 'addr': '172.21.15.187:6823', 'nonce': 2728426525}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6820', 'nonce': 2728426525}, {'type': 'v1', 'addr': '172.21.15.187:6821', 'nonce': 2728426525}]}, 'public_addr': '172.21.15.187:6817/2728426525', 'cluster_addr': '172.21.15.187:6819/2728426525', 'heartbeat_back_addr': '172.21.15.187:6823/2728426525', 'heartbeat_front_addr': '172.21.15.187:6821/2728426525', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1e08d891-8747-402e-b4c2-aa3802ce8537', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 54, 'last_clean_end': 272, 'up_from': 298, 'up_thru': 381, 'down_at': 273, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 13894964}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 13894964}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 13894964}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 13894964}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6830', 'nonce': 13894964}, {'type': 'v1', 'addr': '172.21.15.187:6831', 'nonce': 13894964}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6828', 'nonce': 13894964}, {'type': 'v1', 'addr': '172.21.15.187:6829', 'nonce': 13894964}]}, 'public_addr': '172.21.15.187:6825/13894964', 'cluster_addr': '172.21.15.187:6827/13894964', 'heartbeat_back_addr': '172.21.15.187:6831/13894964', 'heartbeat_front_addr': '172.21.15.187:6829/13894964', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T14:02:59.701694+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T14:02:55.079021+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T14:02:48.643793+0000', 'dead_epoch': 100}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T14:03:28.883427+0000', 'dead_epoch': 214}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T14:03:07.515930+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T14:03:06.752661+0000', 'dead_epoch': 324}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T14:03:27.736192+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T14:03:29.485204+0000', 'dead_epoch': 272}], 'pg_upmap': [{'pgid': '3.14', 'osds': [7, 6, 0]}], 'pg_upmap_items': [{'pgid': '4.2', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.6', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.d', 'mappings': [{'from': 1, 'to': 5}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.49:0/2731793917': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:0/799831425': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:6801/2518197280': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:0/3556829486': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:0/1249926115': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:0/231530138': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:0/4220781134': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:0/202325455': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:6800/4129636473': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:6800/2518197280': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:0/3514741273': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:6801/4129636473': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:0/3125743776': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:0/3600977363': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:6801/3453571598': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:6800/3453571598': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:0/3553734432': '2024-09-22T13:49:27.311313+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-21T14:03:47.479 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T14:03:47.517 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:47 smithi049 ceph-mon[30080]: pgmap v756: 158 pgs: 158 active+clean; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 89 MiB/s wr, 1.42k op/s 2024-09-21T14:03:47.517 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:47 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/945453054' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T14:03:47.517 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:47 smithi049 ceph-mon[40489]: pgmap v756: 158 pgs: 158 active+clean; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 89 MiB/s wr, 1.42k op/s 2024-09-21T14:03:47.517 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:47 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/945453054' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T14:03:47.538 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:03:47.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:47.595 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:47.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:03:47.687 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T14:03:47.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:47 smithi187 ceph-mon[33032]: pgmap v756: 158 pgs: 158 active+clean; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 89 MiB/s wr, 1.42k op/s 2024-09-21T14:03:47.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:47 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/945453054' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T14:03:47.696 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:03:47.732 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:47.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:47.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:03:47.834 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:03:47.864 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:47.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:47.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:03:47.874 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 42 16 75606 75590 112.461 82.125 0.0130786 0.00888668 2024-09-21T14:03:47.966 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:03:48.004 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:48.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:48.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:03:48.105 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:48.140 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:48.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:48.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:48.171 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:03:48.243 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:48.272 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:48.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:48.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:48.374 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:03:48.415 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:48.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:48.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:03:48.517 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:03:48.554 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:48.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:48.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:03:48.593 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:03:48.656 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:48.687 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:48.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:48.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:48.789 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:03:48.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:48.823 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:48.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:03:48.824 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:03:48.875 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 43 16 76799 76783 111.579 74.5625 0.00853493 0.00895731 2024-09-21T14:03:48.885 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T14:03:48.885 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-09-21T14:03:48.885 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-09-21T14:03:48.885 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-09-21T14:03:48.885 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T14:03:48.924 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:03:48.968 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:48.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:48.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:03:49.070 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:03:49.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:49.114 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:49.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:03:49.215 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:03:49.255 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:49.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:49.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:03:49.276 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-09-21T14:03:49.357 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:03:49.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:49.448 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:49.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:03:49.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:49 smithi049 ceph-mon[30080]: pgmap v757: 158 pgs: 158 active+clean; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-09-21T14:03:49.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:49 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2988516306' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:03:49.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:49 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:03:49.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:49 smithi049 ceph-mon[40489]: pgmap v757: 158 pgs: 158 active+clean; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-09-21T14:03:49.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:49 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2988516306' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:03:49.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:49 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:03:49.549 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:03:49.586 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:49.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:49.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:03:49.689 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:03:49.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:49 smithi187 ceph-mon[33032]: pgmap v757: 158 pgs: 158 active+clean; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-09-21T14:03:49.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:49 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2988516306' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:03:49.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:49 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:03:49.723 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:49.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:49.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:03:49.825 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:03:49.858 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:49.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:49.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:03:49.875 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 44 16 78133 78117 110.938 83.375 0.011978 0.0090092 2024-09-21T14:03:49.959 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:03:49.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:49.998 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:49.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:03:50.099 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:03:50.138 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:50.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:50.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:03:50.194 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:03:49 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:03:50.241 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:03:50.273 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:50.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:50.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:03:50.374 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:03:50.410 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:50.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:50.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:03:50.514 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:50.549 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:50.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:50.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:50.579 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:50 smithi049 ceph-mon[40489]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:03:50.580 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:50 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T14:03:50.580 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:50 smithi049 ceph-mon[40489]: osdmap e399: 8 total, 8 up, 8 in 2024-09-21T14:03:50.580 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:50 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:03:50.580 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:50 smithi049 ceph-mon[40489]: osdmap e400: 8 total, 8 up, 8 in 2024-09-21T14:03:50.580 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:50 smithi049 ceph-mon[30080]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:03:50.580 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:50 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T14:03:50.580 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:50 smithi049 ceph-mon[30080]: osdmap e399: 8 total, 8 up, 8 in 2024-09-21T14:03:50.580 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:50 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:03:50.580 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:50 smithi049 ceph-mon[30080]: osdmap e400: 8 total, 8 up, 8 in 2024-09-21T14:03:50.650 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:03:50.693 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:50.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:50.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:03:50.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:50 smithi187 ceph-mon[33032]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:03:50.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:50 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T14:03:50.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:50 smithi187 ceph-mon[33032]: osdmap e399: 8 total, 8 up, 8 in 2024-09-21T14:03:50.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:50 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:03:50.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:50 smithi187 ceph-mon[33032]: osdmap e400: 8 total, 8 up, 8 in 2024-09-21T14:03:50.714 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T14:03:50.795 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:03:50.820 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T14:03:50.829 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-09-21T14:03:50.850 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:03:50 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:03:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:03:50.859 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:50.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:50.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:03:50.875 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 45 16 79361 79345 110.178 76.75 0.011025 0.00907046 2024-09-21T14:03:50.961 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:03:50.993 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:50.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:50.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:03:51.095 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:03:51.127 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:51.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:51.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:03:51.229 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:03:51.267 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:51.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:51.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:03:51.369 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:03:51.410 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:51.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:51.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:03:51.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:51 smithi049 ceph-mon[30080]: pgmap v759: 158 pgs: 158 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.0 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-09-21T14:03:51.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:51 smithi049 ceph-mon[30080]: from='client.16149 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:03:51.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:51 smithi049 ceph-mon[40489]: pgmap v759: 158 pgs: 158 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.0 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-09-21T14:03:51.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:51 smithi049 ceph-mon[40489]: from='client.16149 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:03:51.512 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:03:51.550 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:51.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:51.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:03:51.652 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:51.687 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:51.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:51.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:51.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:51 smithi187 ceph-mon[33032]: pgmap v759: 158 pgs: 158 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.0 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-09-21T14:03:51.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:51 smithi187 ceph-mon[33032]: from='client.16149 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:03:51.789 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:51.824 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:51.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:51.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:51.875 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 46 16 80685 80669 109.582 82.75 0.00835654 0.00912067 2024-09-21T14:03:51.925 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:03:51.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:51.967 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:51.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:03:52.068 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:03:52.104 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:52.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:52.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:03:52.206 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:03:52.244 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:52.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:52.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:03:52.345 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":760,"stamp":"2024-09-21T14:03:50.123989+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4964799363,"num_objects":75950,"num_object_clones":0,"num_object_copies":227850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75950,"num_whiteouts":0,"num_read":1443,"num_read_kb":1262,"num_write":76546,"num_write_kb":4849145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":1033072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76422,"ondisk_log_size":76422,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":17832136,"kb_used_data":14424456,"kb_used_omap":23,"kb_used_meta":3407528,"kb_avail":732129080,"statfs":{"total":767960285184,"available":749700177920,"internally_reserved":0,"allocated":14770642944,"data_stored":14763112475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24148,"internal_metadata":3489309100},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,4,6,5,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":730529792,"num_objects":11147,"num_object_clones":0,"num_object_copies":33441,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11147,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":11147,"num_write_kb":713408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.109826"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":621,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.118743+0000","last_change":"2024-09-21T14:02:56.458231+0000","last_active":"2024-09-21T14:03:40.118743+0000","last_peered":"2024-09-21T14:03:40.118743+0000","last_clean":"2024-09-21T14:03:40.118743+0000","last_became_active":"2024-09-21T14:01:41.189398+0000","last_became_peered":"2024-09-21T14:01:41.189398+0000","last_unstale":"2024-09-21T14:03:40.118743+0000","last_undegraded":"2024-09-21T14:03:40.118743+0000","last_fullsized":"2024-09-21T14:03:40.118743+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:56.458125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:02:56.458125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:20.867402+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"290'15","reported_seq":788,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.114870+0000","last_change":"2024-09-21T14:01:34.785322+0000","last_active":"2024-09-21T14:03:40.114870+0000","last_peered":"2024-09-21T14:03:40.114870+0000","last_clean":"2024-09-21T14:03:40.114870+0000","last_became_active":"2024-09-21T13:58:17.771220+0000","last_became_peered":"2024-09-21T13:58:17.771220+0000","last_unstale":"2024-09-21T14:03:40.114870+0000","last_undegraded":"2024-09-21T14:03:40.114870+0000","last_fullsized":"2024-09-21T14:03:40.114870+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'15","last_scrub_stamp":"2024-09-21T14:01:34.785267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:34.785267+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:53.703961+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":739,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-21T14:03:49.274018+0000","last_change":"2024-09-21T14:03:33.301558+0000","last_active":"2024-09-21T14:03:49.274018+0000","last_peered":"2024-09-21T14:03:49.274018+0000","last_clean":"2024-09-21T14:03:49.274018+0000","last_became_active":"2024-09-21T14:01:41.202168+0000","last_became_peered":"2024-09-21T14:01:41.202168+0000","last_unstale":"2024-09-21T14:03:49.274018+0000","last_undegraded":"2024-09-21T14:03:49.274018+0000","last_fullsized":"2024-09-21T14:03:49.274018+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:33.301487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:33.301487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:54.547538+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":704,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.115529+0000","last_change":"2024-09-21T14:01:41.203762+0000","last_active":"2024-09-21T14:03:40.115529+0000","last_peered":"2024-09-21T14:03:40.115529+0000","last_clean":"2024-09-21T14:03:40.115529+0000","last_became_active":"2024-09-21T14:01:41.203275+0000","last_became_peered":"2024-09-21T14:01:41.203275+0000","last_unstale":"2024-09-21T14:03:40.115529+0000","last_undegraded":"2024-09-21T14:03:40.115529+0000","last_fullsized":"2024-09-21T14:03:40.115529+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:00.320510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:00.320510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:46.689692+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":674,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.260098+0000","last_change":"2024-09-21T13:59:32.208701+0000","last_active":"2024-09-21T14:03:40.260098+0000","last_peered":"2024-09-21T14:03:40.260098+0000","last_clean":"2024-09-21T14:03:40.260098+0000","last_became_active":"2024-09-21T13:59:32.203340+0000","last_became_peered":"2024-09-21T13:59:32.203340+0000","last_unstale":"2024-09-21T14:03:40.260098+0000","last_undegraded":"2024-09-21T14:03:40.260098+0000","last_fullsized":"2024-09-21T14:03:40.260098+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:34.696780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:34.696780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:44.274358+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":687,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.259231+0000","last_change":"2024-09-21T13:58:17.778322+0000","last_active":"2024-09-21T14:03:40.259231+0000","last_peered":"2024-09-21T14:03:40.259231+0000","last_clean":"2024-09-21T14:03:40.259231+0000","last_became_active":"2024-09-21T13:58:17.777893+0000","last_became_peered":"2024-09-21T13:58:17.777893+0000","last_unstale":"2024-09-21T14:03:40.259231+0000","last_undegraded":"2024-09-21T14:03:40.259231+0000","last_fullsized":"2024-09-21T14:03:40.259231+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:07.001135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:55:07.001135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"290'27","reported_seq":660,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.258426+0000","last_change":"2024-09-21T13:58:11.718421+0000","last_active":"2024-09-21T14:03:40.258426+0000","last_peered":"2024-09-21T14:03:40.258426+0000","last_clean":"2024-09-21T14:03:40.258426+0000","last_became_active":"2024-09-21T13:57:12.703469+0000","last_became_peered":"2024-09-21T13:57:12.703469+0000","last_unstale":"2024-09-21T14:03:40.258426+0000","last_undegraded":"2024-09-21T14:03:40.258426+0000","last_fullsized":"2024-09-21T14:03:40.258426+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'15","last_scrub_stamp":"2024-09-21T13:58:11.718334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:11.718334+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":736,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.114763+0000","last_change":"2024-09-21T14:03:02.862866+0000","last_active":"2024-09-21T14:03:40.114763+0000","last_peered":"2024-09-21T14:03:40.114763+0000","last_clean":"2024-09-21T14:03:40.114763+0000","last_became_active":"2024-09-21T13:58:17.773464+0000","last_became_peered":"2024-09-21T13:58:17.773464+0000","last_unstale":"2024-09-21T14:03:40.114763+0000","last_undegraded":"2024-09-21T14:03:40.114763+0000","last_fullsized":"2024-09-21T14:03:40.114763+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:02.862787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:02.862787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:17.663483+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":679,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.117601+0000","last_change":"2024-09-21T14:01:41.194807+0000","last_active":"2024-09-21T14:03:40.117601+0000","last_peered":"2024-09-21T14:03:40.117601+0000","last_clean":"2024-09-21T14:03:40.117601+0000","last_became_active":"2024-09-21T14:01:41.194517+0000","last_became_peered":"2024-09-21T14:01:41.194517+0000","last_unstale":"2024-09-21T14:03:40.117601+0000","last_undegraded":"2024-09-21T14:03:40.117601+0000","last_fullsized":"2024-09-21T14:03:40.117601+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:31.127926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:31.127926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:10.380240+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"290'15","reported_seq":753,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.115308+0000","last_change":"2024-09-21T14:00:02.475761+0000","last_active":"2024-09-21T14:03:40.115308+0000","last_peered":"2024-09-21T14:03:40.115308+0000","last_clean":"2024-09-21T14:03:40.115308+0000","last_became_active":"2024-09-21T13:57:12.681408+0000","last_became_peered":"2024-09-21T13:57:12.681408+0000","last_unstale":"2024-09-21T14:03:40.115308+0000","last_undegraded":"2024-09-21T14:03:40.115308+0000","last_fullsized":"2024-09-21T14:03:40.115308+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'15","last_scrub_stamp":"2024-09-21T14:00:02.475661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:00:02.475661+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:43.713097+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":628,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.116673+0000","last_change":"2024-09-21T14:03:04.516642+0000","last_active":"2024-09-21T14:03:40.116673+0000","last_peered":"2024-09-21T14:03:40.116673+0000","last_clean":"2024-09-21T14:03:40.116673+0000","last_became_active":"2024-09-21T14:01:41.188248+0000","last_became_peered":"2024-09-21T14:01:41.188248+0000","last_unstale":"2024-09-21T14:03:40.116673+0000","last_undegraded":"2024-09-21T14:03:40.116673+0000","last_fullsized":"2024-09-21T14:03:40.116673+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:04.516583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:04.516583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:26.082827+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":660,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-21T14:03:49.275701+0000","last_change":"2024-09-21T13:59:32.202128+0000","last_active":"2024-09-21T14:03:49.275701+0000","last_peered":"2024-09-21T14:03:49.275701+0000","last_clean":"2024-09-21T14:03:49.275701+0000","last_became_active":"2024-09-21T13:59:32.201158+0000","last_became_peered":"2024-09-21T13:59:32.201158+0000","last_unstale":"2024-09-21T14:03:49.275701+0000","last_undegraded":"2024-09-21T14:03:49.275701+0000","last_fullsized":"2024-09-21T14:03:49.275701+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:25.792985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:54:25.792985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":656,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.116738+0000","last_change":"2024-09-21T14:01:41.189797+0000","last_active":"2024-09-21T14:03:40.116738+0000","last_peered":"2024-09-21T14:03:40.116738+0000","last_clean":"2024-09-21T14:03:40.116738+0000","last_became_active":"2024-09-21T14:01:41.189373+0000","last_became_peered":"2024-09-21T14:01:41.189373+0000","last_unstale":"2024-09-21T14:03:40.116738+0000","last_undegraded":"2024-09-21T14:03:40.116738+0000","last_fullsized":"2024-09-21T14:03:40.116738+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:54.328275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:54.328275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:09.403771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"290'7","reported_seq":778,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.114707+0000","last_change":"2024-09-21T14:01:57.563250+0000","last_active":"2024-09-21T14:03:40.114707+0000","last_peered":"2024-09-21T14:03:40.114707+0000","last_clean":"2024-09-21T14:03:40.114707+0000","last_became_active":"2024-09-21T13:59:09.063053+0000","last_became_peered":"2024-09-21T13:59:09.063053+0000","last_unstale":"2024-09-21T14:03:40.114707+0000","last_undegraded":"2024-09-21T14:03:40.114707+0000","last_fullsized":"2024-09-21T14:03:40.114707+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'7","last_scrub_stamp":"2024-09-21T14:01:57.563184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:57.563184+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:39.020043+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":649,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.126688+0000","last_change":"2024-09-21T13:59:32.204565+0000","last_active":"2024-09-21T14:03:40.126688+0000","last_peered":"2024-09-21T14:03:40.126688+0000","last_clean":"2024-09-21T14:03:40.126688+0000","last_became_active":"2024-09-21T13:59:32.204174+0000","last_became_peered":"2024-09-21T13:59:32.204174+0000","last_unstale":"2024-09-21T14:03:40.126688+0000","last_undegraded":"2024-09-21T14:03:40.126688+0000","last_fullsized":"2024-09-21T14:03:40.126688+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:06.722293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:06.722293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":690,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.117614+0000","last_change":"2024-09-21T14:01:41.199085+0000","last_active":"2024-09-21T14:03:40.117614+0000","last_peered":"2024-09-21T14:03:40.117614+0000","last_clean":"2024-09-21T14:03:40.117614+0000","last_became_active":"2024-09-21T14:01:41.198672+0000","last_became_peered":"2024-09-21T14:01:41.198672+0000","last_unstale":"2024-09-21T14:03:40.117614+0000","last_undegraded":"2024-09-21T14:03:40.117614+0000","last_fullsized":"2024-09-21T14:03:40.117614+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:44.376215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:44.376215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:23.240484+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":755,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.115879+0000","last_change":"2024-09-21T14:03:36.018910+0000","last_active":"2024-09-21T14:03:40.115879+0000","last_peered":"2024-09-21T14:03:40.115879+0000","last_clean":"2024-09-21T14:03:40.115879+0000","last_became_active":"2024-09-21T14:01:41.200360+0000","last_became_peered":"2024-09-21T14:01:41.200360+0000","last_unstale":"2024-09-21T14:03:40.115879+0000","last_undegraded":"2024-09-21T14:03:40.115879+0000","last_fullsized":"2024-09-21T14:03:40.115879+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:36.018851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:36.018851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:05.142781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":665,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.115038+0000","last_change":"2024-09-21T14:02:36.679396+0000","last_active":"2024-09-21T14:03:40.115038+0000","last_peered":"2024-09-21T14:03:40.115038+0000","last_clean":"2024-09-21T14:03:40.115038+0000","last_became_active":"2024-09-21T13:59:32.204749+0000","last_became_peered":"2024-09-21T13:59:32.204749+0000","last_unstale":"2024-09-21T14:03:40.115038+0000","last_undegraded":"2024-09-21T14:03:40.115038+0000","last_fullsized":"2024-09-21T14:03:40.115038+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:36.679321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:02:36.679321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:43.067225+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"290'27","reported_seq":729,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.114205+0000","last_change":"2024-09-21T14:02:55.620755+0000","last_active":"2024-09-21T14:03:40.114205+0000","last_peered":"2024-09-21T14:03:40.114205+0000","last_clean":"2024-09-21T14:03:40.114205+0000","last_became_active":"2024-09-21T13:57:12.680783+0000","last_became_peered":"2024-09-21T13:57:12.680783+0000","last_unstale":"2024-09-21T14:03:40.114205+0000","last_undegraded":"2024-09-21T14:03:40.114205+0000","last_fullsized":"2024-09-21T14:03:40.114205+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'27","last_scrub_stamp":"2024-09-21T14:02:55.620700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.620700+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:01.085594+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":724,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-21T14:03:49.276193+0000","last_change":"2024-09-21T14:03:08.299319+0000","last_active":"2024-09-21T14:03:49.276193+0000","last_peered":"2024-09-21T14:03:49.276193+0000","last_clean":"2024-09-21T14:03:49.276193+0000","last_became_active":"2024-09-21T13:49:02.066796+0000","last_became_peered":"2024-09-21T13:49:02.066796+0000","last_unstale":"2024-09-21T14:03:49.276193+0000","last_undegraded":"2024-09-21T14:03:49.276193+0000","last_fullsized":"2024-09-21T14:03:49.276193+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:08.299252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:08.299252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:29.982454+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":667,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.259907+0000","last_change":"2024-09-21T14:01:41.191486+0000","last_active":"2024-09-21T14:03:40.259907+0000","last_peered":"2024-09-21T14:03:40.259907+0000","last_clean":"2024-09-21T14:03:40.259907+0000","last_became_active":"2024-09-21T14:01:41.191116+0000","last_became_peered":"2024-09-21T14:01:41.191116+0000","last_unstale":"2024-09-21T14:03:40.259907+0000","last_undegraded":"2024-09-21T14:03:40.259907+0000","last_fullsized":"2024-09-21T14:03:40.259907+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:57.962266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:57.962266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:10.160181+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":724,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.115455+0000","last_change":"2024-09-21T14:00:51.495433+0000","last_active":"2024-09-21T14:03:40.115455+0000","last_peered":"2024-09-21T14:03:40.115455+0000","last_clean":"2024-09-21T14:03:40.115455+0000","last_became_active":"2024-09-21T13:52:04.316016+0000","last_became_peered":"2024-09-21T13:52:04.316016+0000","last_unstale":"2024-09-21T14:03:40.115455+0000","last_undegraded":"2024-09-21T14:03:40.115455+0000","last_fullsized":"2024-09-21T14:03:40.115455+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:00:51.495368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:00:51.495368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:12.877210+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"290'18","reported_seq":672,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.262202+0000","last_change":"2024-09-21T13:59:23.041396+0000","last_active":"2024-09-21T14:03:40.262202+0000","last_peered":"2024-09-21T14:03:40.262202+0000","last_clean":"2024-09-21T14:03:40.262202+0000","last_became_active":"2024-09-21T13:57:12.684454+0000","last_became_peered":"2024-09-21T13:57:12.684454+0000","last_unstale":"2024-09-21T14:03:40.262202+0000","last_undegraded":"2024-09-21T14:03:40.262202+0000","last_fullsized":"2024-09-21T14:03:40.262202+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'18","last_scrub_stamp":"2024-09-21T13:59:23.041311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:23.041311+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":719,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.115537+0000","last_change":"2024-09-21T14:03:38.065257+0000","last_active":"2024-09-21T14:03:40.115537+0000","last_peered":"2024-09-21T14:03:40.115537+0000","last_clean":"2024-09-21T14:03:40.115537+0000","last_became_active":"2024-09-21T13:57:12.681897+0000","last_became_peered":"2024-09-21T13:57:12.681897+0000","last_unstale":"2024-09-21T14:03:40.115537+0000","last_undegraded":"2024-09-21T14:03:40.115537+0000","last_fullsized":"2024-09-21T14:03:40.115537+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:38.065196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:38.065196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:03.431512+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":641,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.124195+0000","last_change":"2024-09-21T14:03:28.933216+0000","last_active":"2024-09-21T14:03:40.124195+0000","last_peered":"2024-09-21T14:03:40.124195+0000","last_clean":"2024-09-21T14:03:40.124195+0000","last_became_active":"2024-09-21T14:01:41.181946+0000","last_became_peered":"2024-09-21T14:01:41.181946+0000","last_unstale":"2024-09-21T14:03:40.124195+0000","last_undegraded":"2024-09-21T14:03:40.124195+0000","last_fullsized":"2024-09-21T14:03:40.124195+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T14:03:28.933143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:28.933143+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:46.365067+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":737,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-21T14:03:49.274161+0000","last_change":"2024-09-21T14:01:58.994867+0000","last_active":"2024-09-21T14:03:49.274161+0000","last_peered":"2024-09-21T14:03:49.274161+0000","last_clean":"2024-09-21T14:03:49.274161+0000","last_became_active":"2024-09-21T13:57:12.685742+0000","last_became_peered":"2024-09-21T13:57:12.685742+0000","last_unstale":"2024-09-21T14:03:49.274161+0000","last_undegraded":"2024-09-21T14:03:49.274161+0000","last_fullsized":"2024-09-21T14:03:49.274161+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T14:01:58.994811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:01:58.994811+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:25.355083+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"290'20","reported_seq":614,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.126378+0000","last_change":"2024-09-21T13:59:32.242771+0000","last_active":"2024-09-21T14:03:40.126378+0000","last_peered":"2024-09-21T14:03:40.126378+0000","last_clean":"2024-09-21T14:03:40.126378+0000","last_became_active":"2024-09-21T13:59:32.213865+0000","last_became_peered":"2024-09-21T13:59:32.213865+0000","last_unstale":"2024-09-21T14:03:40.126378+0000","last_undegraded":"2024-09-21T14:03:40.126378+0000","last_fullsized":"2024-09-21T14:03:40.126378+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:57:27.578738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:57:27.578738+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"300'12","reported_seq":362,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:43.302756+0000","last_change":"2024-09-21T13:59:36.012775+0000","last_active":"2024-09-21T14:03:43.302756+0000","last_peered":"2024-09-21T14:03:43.302756+0000","last_clean":"2024-09-21T14:03:43.302756+0000","last_became_active":"2024-09-21T13:59:32.205051+0000","last_became_peered":"2024-09-21T13:59:32.205051+0000","last_unstale":"2024-09-21T14:03:43.302756+0000","last_undegraded":"2024-09-21T14:03:43.302756+0000","last_fullsized":"2024-09-21T14:03:43.302756+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'12","last_scrub_stamp":"2024-09-21T13:59:36.012693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:36.012693+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:21.546366+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"398'4931","reported_seq":4976,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:48.580430+0000","last_change":"2024-09-21T14:02:56.560786+0000","last_active":"2024-09-21T14:03:48.580430+0000","last_peered":"2024-09-21T14:03:48.580430+0000","last_clean":"2024-09-21T14:03:48.580430+0000","last_became_active":"2024-09-21T14:02:56.559536+0000","last_became_peered":"2024-09-21T14:02:56.559536+0000","last_unstale":"2024-09-21T14:03:48.580430+0000","last_undegraded":"2024-09-21T14:03:48.580430+0000","last_fullsized":"2024-09-21T14:03:48.580430+0000","mapping_epoch":381,"log_start":"387'500","ondisk_log_start":"387'500","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":4431,"log_dups_size":0,"ondisk_log_size":4431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:03.627467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":323158016,"num_objects":4931,"num_object_clones":0,"num_object_copies":14793,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4931,"num_write_kb":315584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"75'79","reported_seq":129,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.117115+0000","last_change":"2024-09-21T14:01:41.248772+0000","last_active":"2024-09-21T14:03:40.117115+0000","last_peered":"2024-09-21T14:03:40.117115+0000","last_clean":"2024-09-21T14:03:40.117115+0000","last_became_active":"2024-09-21T14:01:41.246888+0000","last_became_peered":"2024-09-21T14:01:41.246888+0000","last_unstale":"2024-09-21T14:03:40.117115+0000","last_undegraded":"2024-09-21T14:03:40.117115+0000","last_fullsized":"2024-09-21T14:03:40.117115+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:03:52.423493+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53669,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"290'27","reported_seq":783,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-21T14:03:49.275709+0000","last_change":"2024-09-21T14:02:58.293228+0000","last_active":"2024-09-21T14:03:49.275709+0000","last_peered":"2024-09-21T14:03:49.275709+0000","last_clean":"2024-09-21T14:03:49.275709+0000","last_became_active":"2024-09-21T13:57:12.695609+0000","last_became_peered":"2024-09-21T13:57:12.695609+0000","last_unstale":"2024-09-21T14:03:49.275709+0000","last_undegraded":"2024-09-21T14:03:49.275709+0000","last_fullsized":"2024-09-21T14:03:49.275709+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'27","last_scrub_stamp":"2024-09-21T14:02:58.293160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:58.293160+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:01.203771+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":659,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.258015+0000","last_change":"2024-09-21T13:59:32.205723+0000","last_active":"2024-09-21T14:03:40.258015+0000","last_peered":"2024-09-21T14:03:40.258015+0000","last_clean":"2024-09-21T14:03:40.258015+0000","last_became_active":"2024-09-21T13:59:32.205350+0000","last_became_peered":"2024-09-21T13:59:32.205350+0000","last_unstale":"2024-09-21T14:03:40.258015+0000","last_undegraded":"2024-09-21T14:03:40.258015+0000","last_fullsized":"2024-09-21T14:03:40.258015+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:32.668851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:32.668851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:47.252073+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"346'10","reported_seq":926,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:43.302551+0000","last_change":"2024-09-21T14:02:59.831183+0000","last_active":"2024-09-21T14:03:43.302551+0000","last_peered":"2024-09-21T14:03:43.302551+0000","last_clean":"2024-09-21T14:03:43.302551+0000","last_became_active":"2024-09-21T14:01:41.200218+0000","last_became_peered":"2024-09-21T14:01:41.200218+0000","last_unstale":"2024-09-21T14:03:43.302551+0000","last_undegraded":"2024-09-21T14:03:43.302551+0000","last_fullsized":"2024-09-21T14:03:43.302551+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'10","last_scrub_stamp":"2024-09-21T14:02:59.831128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:02:59.831128+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:08.243919+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":585,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.117119+0000","last_change":"2024-09-21T14:01:41.207310+0000","last_active":"2024-09-21T14:03:40.117119+0000","last_peered":"2024-09-21T14:03:40.117119+0000","last_clean":"2024-09-21T14:03:40.117119+0000","last_became_active":"2024-09-21T14:01:41.206720+0000","last_became_peered":"2024-09-21T14:01:41.206720+0000","last_unstale":"2024-09-21T14:03:40.117119+0000","last_undegraded":"2024-09-21T14:03:40.117119+0000","last_fullsized":"2024-09-21T14:03:40.117119+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:52.244483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:52.244483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:21.273004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"398'4665","reported_seq":4710,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:48.609783+0000","last_change":"2024-09-21T14:02:56.562998+0000","last_active":"2024-09-21T14:03:48.609783+0000","last_peered":"2024-09-21T14:03:48.609783+0000","last_clean":"2024-09-21T14:03:48.609783+0000","last_became_active":"2024-09-21T14:02:56.562524+0000","last_became_peered":"2024-09-21T14:02:56.562524+0000","last_unstale":"2024-09-21T14:03:48.609783+0000","last_undegraded":"2024-09-21T14:03:48.609783+0000","last_fullsized":"2024-09-21T14:03:48.609783+0000","mapping_epoch":381,"log_start":"386'200","ondisk_log_start":"386'200","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":4465,"log_dups_size":0,"ondisk_log_size":4465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:01.881695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":305725440,"num_objects":4665,"num_object_clones":0,"num_object_copies":13995,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4665,"num_write_kb":298560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"75'79","reported_seq":183,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.115293+0000","last_change":"2024-09-21T14:02:06.395083+0000","last_active":"2024-09-21T14:03:40.115293+0000","last_peered":"2024-09-21T14:03:40.115293+0000","last_clean":"2024-09-21T14:03:40.115293+0000","last_became_active":"2024-09-21T14:00:41.137756+0000","last_became_peered":"2024-09-21T14:00:41.137756+0000","last_unstale":"2024-09-21T14:03:40.115293+0000","last_undegraded":"2024-09-21T14:03:40.115293+0000","last_fullsized":"2024-09-21T14:03:40.115293+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:02:06.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:02:06.395011+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"290'30","reported_seq":683,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.257859+0000","last_change":"2024-09-21T13:59:32.246090+0000","last_active":"2024-09-21T14:03:40.257859+0000","last_peered":"2024-09-21T14:03:40.257859+0000","last_clean":"2024-09-21T14:03:40.257859+0000","last_became_active":"2024-09-21T13:59:32.205887+0000","last_became_peered":"2024-09-21T13:59:32.205887+0000","last_unstale":"2024-09-21T14:03:40.257859+0000","last_undegraded":"2024-09-21T14:03:40.257859+0000","last_fullsized":"2024-09-21T14:03:40.257859+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'18","last_scrub_stamp":"2024-09-21T13:54:19.001030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:19.001030+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:43.319659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":611,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.260170+0000","last_change":"2024-09-21T13:59:32.212559+0000","last_active":"2024-09-21T14:03:40.260170+0000","last_peered":"2024-09-21T14:03:40.260170+0000","last_clean":"2024-09-21T14:03:40.260170+0000","last_became_active":"2024-09-21T13:59:32.211004+0000","last_became_peered":"2024-09-21T13:59:32.211004+0000","last_unstale":"2024-09-21T14:03:40.260170+0000","last_undegraded":"2024-09-21T14:03:40.260170+0000","last_fullsized":"2024-09-21T14:03:40.260170+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:25.736461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:25.736461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:48.292957+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":729,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.114536+0000","last_change":"2024-09-21T13:59:34.293330+0000","last_active":"2024-09-21T14:03:40.114536+0000","last_peered":"2024-09-21T14:03:40.114536+0000","last_clean":"2024-09-21T14:03:40.114536+0000","last_became_active":"2024-09-21T13:59:32.220304+0000","last_became_peered":"2024-09-21T13:59:32.220304+0000","last_unstale":"2024-09-21T14:03:40.114536+0000","last_undegraded":"2024-09-21T14:03:40.114536+0000","last_fullsized":"2024-09-21T14:03:40.114536+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:34.292892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:34.292892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:19.896473+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":709,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.115260+0000","last_change":"2024-09-21T13:59:23.334230+0000","last_active":"2024-09-21T14:03:40.115260+0000","last_peered":"2024-09-21T14:03:40.115260+0000","last_clean":"2024-09-21T14:03:40.115260+0000","last_became_active":"2024-09-21T13:57:12.680372+0000","last_became_peered":"2024-09-21T13:57:12.680372+0000","last_unstale":"2024-09-21T14:03:40.115260+0000","last_undegraded":"2024-09-21T14:03:40.115260+0000","last_fullsized":"2024-09-21T14:03:40.115260+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:23.334162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:23.334162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:23.951901+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"398'4840","reported_seq":4885,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:48.600298+0000","last_change":"2024-09-21T14:02:56.557348+0000","last_active":"2024-09-21T14:03:48.600298+0000","last_peered":"2024-09-21T14:03:48.600298+0000","last_clean":"2024-09-21T14:03:48.600298+0000","last_became_active":"2024-09-21T14:02:56.557055+0000","last_became_peered":"2024-09-21T14:02:56.557055+0000","last_unstale":"2024-09-21T14:03:48.600298+0000","last_undegraded":"2024-09-21T14:03:48.600298+0000","last_fullsized":"2024-09-21T14:03:48.600298+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":4840,"log_dups_size":0,"ondisk_log_size":4840,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:20.950710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":317194240,"num_objects":4840,"num_object_clones":0,"num_object_copies":14520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4840,"num_write_kb":309760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"75'79","reported_seq":186,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.262097+0000","last_change":"2024-09-21T14:01:42.177646+0000","last_active":"2024-09-21T14:03:40.262097+0000","last_peered":"2024-09-21T14:03:40.262097+0000","last_clean":"2024-09-21T14:03:40.262097+0000","last_became_active":"2024-09-21T14:01:42.173042+0000","last_became_peered":"2024-09-21T14:01:42.173042+0000","last_unstale":"2024-09-21T14:03:40.262097+0000","last_undegraded":"2024-09-21T14:03:40.262097+0000","last_fullsized":"2024-09-21T14:03:40.262097+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:03:42.356436+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":709030,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"290'28","reported_seq":826,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.114865+0000","last_change":"2024-09-21T14:03:27.047111+0000","last_active":"2024-09-21T14:03:40.114865+0000","last_peered":"2024-09-21T14:03:40.114865+0000","last_clean":"2024-09-21T14:03:40.114865+0000","last_became_active":"2024-09-21T14:01:41.200859+0000","last_became_peered":"2024-09-21T14:01:41.200859+0000","last_unstale":"2024-09-21T14:03:40.114865+0000","last_undegraded":"2024-09-21T14:03:40.114865+0000","last_fullsized":"2024-09-21T14:03:40.114865+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'28","last_scrub_stamp":"2024-09-21T14:03:27.046906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:03:27.046906+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:29.157667+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":718,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-21T14:03:49.291611+0000","last_change":"2024-09-21T13:57:38.046718+0000","last_active":"2024-09-21T14:03:49.291611+0000","last_peered":"2024-09-21T14:03:49.291611+0000","last_clean":"2024-09-21T14:03:49.291611+0000","last_became_active":"2024-09-21T13:48:58.053533+0000","last_became_peered":"2024-09-21T13:48:58.053533+0000","last_unstale":"2024-09-21T14:03:49.291611+0000","last_undegraded":"2024-09-21T14:03:49.291611+0000","last_fullsized":"2024-09-21T14:03:49.291611+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:57:38.046627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:38.046627+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:49.580099+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"241'10","reported_seq":778,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.258342+0000","last_change":"2024-09-21T14:03:03.228551+0000","last_active":"2024-09-21T14:03:40.258342+0000","last_peered":"2024-09-21T14:03:40.258342+0000","last_clean":"2024-09-21T14:03:40.258342+0000","last_became_active":"2024-09-21T13:57:12.686651+0000","last_became_peered":"2024-09-21T13:57:12.686651+0000","last_unstale":"2024-09-21T14:03:40.258342+0000","last_undegraded":"2024-09-21T14:03:40.258342+0000","last_fullsized":"2024-09-21T14:03:40.258342+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'10","last_scrub_stamp":"2024-09-21T14:03:03.228443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:03.228443+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:24.135971+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":675,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.126460+0000","last_change":"2024-09-21T13:59:32.210257+0000","last_active":"2024-09-21T14:03:40.126460+0000","last_peered":"2024-09-21T14:03:40.126460+0000","last_clean":"2024-09-21T14:03:40.126460+0000","last_became_active":"2024-09-21T13:59:32.209615+0000","last_became_peered":"2024-09-21T13:59:32.209615+0000","last_unstale":"2024-09-21T14:03:40.126460+0000","last_undegraded":"2024-09-21T14:03:40.126460+0000","last_fullsized":"2024-09-21T14:03:40.126460+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:37.548064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:37.548064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.c","version":"398'4429","reported_seq":4474,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:46.227999+0000","last_change":"2024-09-21T14:02:56.563139+0000","last_active":"2024-09-21T14:03:46.227999+0000","last_peered":"2024-09-21T14:03:46.227999+0000","last_clean":"2024-09-21T14:03:46.227999+0000","last_became_active":"2024-09-21T14:02:56.562806+0000","last_became_peered":"2024-09-21T14:02:56.562806+0000","last_unstale":"2024-09-21T14:03:46.227999+0000","last_undegraded":"2024-09-21T14:03:46.227999+0000","last_fullsized":"2024-09-21T14:03:46.227999+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":4429,"log_dups_size":0,"ondisk_log_size":4429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:18.206635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":290258944,"num_objects":4429,"num_object_clones":0,"num_object_copies":13287,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4429,"num_write_kb":283456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"75'79","reported_seq":756,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.117491+0000","last_change":"2024-09-21T14:02:05.693250+0000","last_active":"2024-09-21T14:03:40.117491+0000","last_peered":"2024-09-21T14:03:40.117491+0000","last_clean":"2024-09-21T14:03:40.117491+0000","last_became_active":"2024-09-21T14:00:41.083425+0000","last_became_peered":"2024-09-21T14:00:41.083425+0000","last_unstale":"2024-09-21T14:03:40.117491+0000","last_undegraded":"2024-09-21T14:03:40.117491+0000","last_fullsized":"2024-09-21T14:03:40.117491+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:02:05.693193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:02:05.693193+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:38.858726+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"290'20","reported_seq":42,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.113668+0000","last_change":"2024-09-21T14:03:31.791421+0000","last_active":"2024-09-21T14:03:40.113668+0000","last_peered":"2024-09-21T14:03:40.113668+0000","last_clean":"2024-09-21T14:03:40.113668+0000","last_became_active":"2024-09-21T14:03:29.158218+0000","last_became_peered":"2024-09-21T14:03:29.158218+0000","last_unstale":"2024-09-21T14:03:40.113668+0000","last_undegraded":"2024-09-21T14:03:40.113668+0000","last_fullsized":"2024-09-21T14:03:40.113668+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'20","last_scrub_stamp":"2024-09-21T14:03:31.791317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:03:31.791317+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:39.052613+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"61'1","reported_seq":648,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.117237+0000","last_change":"2024-09-21T14:01:41.195580+0000","last_active":"2024-09-21T14:03:40.117237+0000","last_peered":"2024-09-21T14:03:40.117237+0000","last_clean":"2024-09-21T14:03:40.117237+0000","last_became_active":"2024-09-21T14:01:41.195268+0000","last_became_peered":"2024-09-21T14:01:41.195268+0000","last_unstale":"2024-09-21T14:03:40.117237+0000","last_undegraded":"2024-09-21T14:03:40.117237+0000","last_fullsized":"2024-09-21T14:03:40.117237+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:58:48.407734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:48.407734+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:25.283190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"346'15","reported_seq":355,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:43.301712+0000","last_change":"2024-09-21T14:01:41.209662+0000","last_active":"2024-09-21T14:03:43.301712+0000","last_peered":"2024-09-21T14:03:43.301712+0000","last_clean":"2024-09-21T14:03:43.301712+0000","last_became_active":"2024-09-21T14:01:41.209296+0000","last_became_peered":"2024-09-21T14:01:41.209296+0000","last_unstale":"2024-09-21T14:03:43.301712+0000","last_undegraded":"2024-09-21T14:03:43.301712+0000","last_fullsized":"2024-09-21T14:03:43.301712+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'12","last_scrub_stamp":"2024-09-21T13:57:30.133392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:30.133392+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:43.907864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":529,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.258191+0000","last_change":"2024-09-21T13:58:58.730439+0000","last_active":"2024-09-21T14:03:40.258191+0000","last_peered":"2024-09-21T14:03:40.258191+0000","last_clean":"2024-09-21T14:03:40.258191+0000","last_became_active":"2024-09-21T13:57:12.704897+0000","last_became_peered":"2024-09-21T13:57:12.704897+0000","last_unstale":"2024-09-21T14:03:40.258191+0000","last_undegraded":"2024-09-21T14:03:40.258191+0000","last_fullsized":"2024-09-21T14:03:40.258191+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:58.730368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:58.730368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.d","version":"398'4846","reported_seq":4891,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:48.573304+0000","last_change":"2024-09-21T14:02:56.560958+0000","last_active":"2024-09-21T14:03:48.573304+0000","last_peered":"2024-09-21T14:03:48.573304+0000","last_clean":"2024-09-21T14:03:48.573304+0000","last_became_active":"2024-09-21T14:02:56.560664+0000","last_became_peered":"2024-09-21T14:02:56.560664+0000","last_unstale":"2024-09-21T14:03:48.573304+0000","last_undegraded":"2024-09-21T14:03:48.573304+0000","last_fullsized":"2024-09-21T14:03:48.573304+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":4846,"log_dups_size":0,"ondisk_log_size":4846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:06.081463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":317587456,"num_objects":4846,"num_object_clones":0,"num_object_copies":14538,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4846,"num_write_kb":310144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"75'79","reported_seq":764,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.116922+0000","last_change":"2024-09-21T14:03:34.762945+0000","last_active":"2024-09-21T14:03:40.116922+0000","last_peered":"2024-09-21T14:03:40.116922+0000","last_clean":"2024-09-21T14:03:40.116922+0000","last_became_active":"2024-09-21T14:00:41.073047+0000","last_became_peered":"2024-09-21T14:00:41.073047+0000","last_unstale":"2024-09-21T14:03:40.116922+0000","last_undegraded":"2024-09-21T14:03:40.116922+0000","last_fullsized":"2024-09-21T14:03:40.116922+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:03:34.762689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:03:34.762689+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:41.051553+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"290'29","reported_seq":662,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.116859+0000","last_change":"2024-09-21T14:03:08.399687+0000","last_active":"2024-09-21T14:03:40.116859+0000","last_peered":"2024-09-21T14:03:40.116859+0000","last_clean":"2024-09-21T14:03:40.116859+0000","last_became_active":"2024-09-21T14:01:41.186960+0000","last_became_peered":"2024-09-21T14:01:41.186960+0000","last_unstale":"2024-09-21T14:03:40.116859+0000","last_undegraded":"2024-09-21T14:03:40.116859+0000","last_fullsized":"2024-09-21T14:03:40.116859+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'29","last_scrub_stamp":"2024-09-21T14:03:08.399606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:03:08.399606+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:19.025883+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":732,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-21T14:03:49.275028+0000","last_change":"2024-09-21T14:01:41.185706+0000","last_active":"2024-09-21T14:03:49.275028+0000","last_peered":"2024-09-21T14:03:49.275028+0000","last_clean":"2024-09-21T14:03:49.275028+0000","last_became_active":"2024-09-21T14:01:41.185400+0000","last_became_peered":"2024-09-21T14:01:41.185400+0000","last_unstale":"2024-09-21T14:03:49.275028+0000","last_undegraded":"2024-09-21T14:03:49.275028+0000","last_fullsized":"2024-09-21T14:03:49.275028+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:16.712990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:16.712990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:39.515666+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":305,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.117314+0000","last_change":"2024-09-21T14:02:59.469906+0000","last_active":"2024-09-21T14:03:40.117314+0000","last_peered":"2024-09-21T14:03:40.117314+0000","last_clean":"2024-09-21T14:03:40.117314+0000","last_became_active":"2024-09-21T14:01:41.193973+0000","last_became_peered":"2024-09-21T14:01:41.193973+0000","last_unstale":"2024-09-21T14:03:40.117314+0000","last_undegraded":"2024-09-21T14:03:40.117314+0000","last_fullsized":"2024-09-21T14:03:40.117314+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:59.469835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:02:59.469835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:00.302431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":723,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-21T14:03:49.273900+0000","last_change":"2024-09-21T13:58:43.952014+0000","last_active":"2024-09-21T14:03:49.273900+0000","last_peered":"2024-09-21T14:03:49.273900+0000","last_clean":"2024-09-21T14:03:49.273900+0000","last_became_active":"2024-09-21T13:58:17.757628+0000","last_became_peered":"2024-09-21T13:58:17.757628+0000","last_unstale":"2024-09-21T14:03:49.273900+0000","last_undegraded":"2024-09-21T14:03:49.273900+0000","last_fullsized":"2024-09-21T14:03:49.273900+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:43.951941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:43.951941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:52.357919+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"398'4706","reported_seq":4751,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:48.593998+0000","last_change":"2024-09-21T14:02:56.559655+0000","last_active":"2024-09-21T14:03:48.593998+0000","last_peered":"2024-09-21T14:03:48.593998+0000","last_clean":"2024-09-21T14:03:48.593998+0000","last_became_active":"2024-09-21T14:02:56.559346+0000","last_became_peered":"2024-09-21T14:02:56.559346+0000","last_unstale":"2024-09-21T14:03:48.593998+0000","last_undegraded":"2024-09-21T14:03:48.593998+0000","last_fullsized":"2024-09-21T14:03:48.593998+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":4706,"log_dups_size":0,"ondisk_log_size":4706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:10.203798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":308412416,"num_objects":4706,"num_object_clones":0,"num_object_copies":14118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4706,"num_write_kb":301184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"75'79","reported_seq":130,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.117791+0000","last_change":"2024-09-21T14:01:41.199377+0000","last_active":"2024-09-21T14:03:40.117791+0000","last_peered":"2024-09-21T14:03:40.117791+0000","last_clean":"2024-09-21T14:03:40.117791+0000","last_became_active":"2024-09-21T14:01:41.198750+0000","last_became_peered":"2024-09-21T14:01:41.198750+0000","last_unstale":"2024-09-21T14:03:40.117791+0000","last_undegraded":"2024-09-21T14:03:40.117791+0000","last_fullsized":"2024-09-21T14:03:40.117791+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:03:47.451019+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"290'44","reported_seq":985,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-21T14:03:49.274889+0000","last_change":"2024-09-21T14:02:57.316829+0000","last_active":"2024-09-21T14:03:49.274889+0000","last_peered":"2024-09-21T14:03:49.274889+0000","last_clean":"2024-09-21T14:03:49.274889+0000","last_became_active":"2024-09-21T13:57:12.684690+0000","last_became_peered":"2024-09-21T13:57:12.684690+0000","last_unstale":"2024-09-21T14:03:49.274889+0000","last_undegraded":"2024-09-21T14:03:49.274889+0000","last_fullsized":"2024-09-21T14:03:49.274889+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'44","last_scrub_stamp":"2024-09-21T14:02:57.316763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:57.316763+0000","objects_scrubbed":10,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:57.769620+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":737,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.114987+0000","last_change":"2024-09-21T14:01:58.524159+0000","last_active":"2024-09-21T14:03:40.114987+0000","last_peered":"2024-09-21T14:03:40.114987+0000","last_clean":"2024-09-21T14:03:40.114987+0000","last_became_active":"2024-09-21T13:58:17.770937+0000","last_became_peered":"2024-09-21T13:58:17.770937+0000","last_unstale":"2024-09-21T14:03:40.114987+0000","last_undegraded":"2024-09-21T14:03:40.114987+0000","last_fullsized":"2024-09-21T14:03:40.114987+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:58.524114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:01:58.524114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:08.916792+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":675,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.117067+0000","last_change":"2024-09-21T14:03:30.390161+0000","last_active":"2024-09-21T14:03:40.117067+0000","last_peered":"2024-09-21T14:03:40.117067+0000","last_clean":"2024-09-21T14:03:40.117067+0000","last_became_active":"2024-09-21T14:01:41.247642+0000","last_became_peered":"2024-09-21T14:01:41.247642+0000","last_unstale":"2024-09-21T14:03:40.117067+0000","last_undegraded":"2024-09-21T14:03:40.117067+0000","last_fullsized":"2024-09-21T14:03:40.117067+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-21T14:03:30.390086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T14:03:30.390086+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:32.519440+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":650,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-21T14:03:49.276004+0000","last_change":"2024-09-21T13:55:07.021128+0000","last_active":"2024-09-21T14:03:49.276004+0000","last_peered":"2024-09-21T14:03:49.276004+0000","last_clean":"2024-09-21T14:03:49.276004+0000","last_became_active":"2024-09-21T13:50:29.688249+0000","last_became_peered":"2024-09-21T13:50:29.688249+0000","last_unstale":"2024-09-21T14:03:49.276004+0000","last_undegraded":"2024-09-21T14:03:49.276004+0000","last_fullsized":"2024-09-21T14:03:49.276004+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:07.021059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:55:07.021059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:50.302160+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":646,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.257260+0000","last_change":"2024-09-21T14:03:02.238409+0000","last_active":"2024-09-21T14:03:40.257260+0000","last_peered":"2024-09-21T14:03:40.257260+0000","last_clean":"2024-09-21T14:03:40.257260+0000","last_became_active":"2024-09-21T13:59:32.208980+0000","last_became_peered":"2024-09-21T13:59:32.208980+0000","last_unstale":"2024-09-21T14:03:40.257260+0000","last_undegraded":"2024-09-21T14:03:40.257260+0000","last_fullsized":"2024-09-21T14:03:40.257260+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:02.238357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:02.238357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:09.206892+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"398'4474","reported_seq":4519,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:45.881008+0000","last_change":"2024-09-21T14:02:56.559275+0000","last_active":"2024-09-21T14:03:45.881008+0000","last_peered":"2024-09-21T14:03:45.881008+0000","last_clean":"2024-09-21T14:03:45.881008+0000","last_became_active":"2024-09-21T14:02:56.558773+0000","last_became_peered":"2024-09-21T14:02:56.558773+0000","last_unstale":"2024-09-21T14:03:45.881008+0000","last_undegraded":"2024-09-21T14:03:45.881008+0000","last_fullsized":"2024-09-21T14:03:45.881008+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":4474,"log_dups_size":0,"ondisk_log_size":4474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:02.709376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293208064,"num_objects":4474,"num_object_clones":0,"num_object_copies":13422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4474,"num_write_kb":286336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"75'79","reported_seq":762,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.126732+0000","last_change":"2024-09-21T14:00:41.077286+0000","last_active":"2024-09-21T14:03:40.126732+0000","last_peered":"2024-09-21T14:03:40.126732+0000","last_clean":"2024-09-21T14:03:40.126732+0000","last_became_active":"2024-09-21T14:00:41.076948+0000","last_became_peered":"2024-09-21T14:00:41.076948+0000","last_unstale":"2024-09-21T14:03:40.126732+0000","last_undegraded":"2024-09-21T14:03:40.126732+0000","last_fullsized":"2024-09-21T14:03:40.126732+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:03:51.730082+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"290'17","reported_seq":757,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-21T14:03:49.275801+0000","last_change":"2024-09-21T14:01:41.200809+0000","last_active":"2024-09-21T14:03:49.275801+0000","last_peered":"2024-09-21T14:03:49.275801+0000","last_clean":"2024-09-21T14:03:49.275801+0000","last_became_active":"2024-09-21T14:01:41.200365+0000","last_became_peered":"2024-09-21T14:01:41.200365+0000","last_unstale":"2024-09-21T14:03:49.275801+0000","last_undegraded":"2024-09-21T14:03:49.275801+0000","last_fullsized":"2024-09-21T14:03:49.275801+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'13","last_scrub_stamp":"2024-09-21T13:58:11.619848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:11.619848+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:43.404997+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":767,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-21T14:03:49.275155+0000","last_change":"2024-09-21T14:03:01.310647+0000","last_active":"2024-09-21T14:03:49.275155+0000","last_peered":"2024-09-21T14:03:49.275155+0000","last_clean":"2024-09-21T14:03:49.275155+0000","last_became_active":"2024-09-21T13:58:17.759671+0000","last_became_peered":"2024-09-21T13:58:17.759671+0000","last_unstale":"2024-09-21T14:03:49.275155+0000","last_undegraded":"2024-09-21T14:03:49.275155+0000","last_fullsized":"2024-09-21T14:03:49.275155+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:01.310603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:03:01.310603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:10.066440+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":646,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.113480+0000","last_change":"2024-09-21T13:58:41.644876+0000","last_active":"2024-09-21T14:03:40.113480+0000","last_peered":"2024-09-21T14:03:40.113480+0000","last_clean":"2024-09-21T14:03:40.113480+0000","last_became_active":"2024-09-21T13:57:12.682147+0000","last_became_peered":"2024-09-21T13:57:12.682147+0000","last_unstale":"2024-09-21T14:03:40.113480+0000","last_undegraded":"2024-09-21T14:03:40.113480+0000","last_fullsized":"2024-09-21T14:03:40.113480+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:58:41.644789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:58:41.644789+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:15.691519+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":625,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.116661+0000","last_change":"2024-09-21T14:01:50.880840+0000","last_active":"2024-09-21T14:03:40.116661+0000","last_peered":"2024-09-21T14:03:40.116661+0000","last_clean":"2024-09-21T14:03:40.116661+0000","last_became_active":"2024-09-21T14:01:41.182000+0000","last_became_peered":"2024-09-21T14:01:41.182000+0000","last_unstale":"2024-09-21T14:03:40.116661+0000","last_undegraded":"2024-09-21T14:03:40.116661+0000","last_fullsized":"2024-09-21T14:03:40.116661+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:50.880471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:01:50.880471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:17.729591+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":683,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.125097+0000","last_change":"2024-09-21T13:59:32.222791+0000","last_active":"2024-09-21T14:03:40.125097+0000","last_peered":"2024-09-21T14:03:40.125097+0000","last_clean":"2024-09-21T14:03:40.125097+0000","last_became_active":"2024-09-21T13:59:32.221502+0000","last_became_peered":"2024-09-21T13:59:32.221502+0000","last_unstale":"2024-09-21T14:03:40.125097+0000","last_undegraded":"2024-09-21T14:03:40.125097+0000","last_fullsized":"2024-09-21T14:03:40.125097+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:14.673582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:14.673582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"398'4727","reported_seq":4772,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:47.762697+0000","last_change":"2024-09-21T14:02:56.559108+0000","last_active":"2024-09-21T14:03:47.762697+0000","last_peered":"2024-09-21T14:03:47.762697+0000","last_clean":"2024-09-21T14:03:47.762697+0000","last_became_active":"2024-09-21T14:02:56.558826+0000","last_became_peered":"2024-09-21T14:02:56.558826+0000","last_unstale":"2024-09-21T14:03:47.762697+0000","last_undegraded":"2024-09-21T14:03:47.762697+0000","last_fullsized":"2024-09-21T14:03:47.762697+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":4727,"log_dups_size":0,"ondisk_log_size":4727,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:03.816041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":309788672,"num_objects":4727,"num_object_clones":0,"num_object_copies":14181,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4727,"num_write_kb":302528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"75'79","reported_seq":169,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-21T14:03:49.274650+0000","last_change":"2024-09-21T14:00:41.071631+0000","last_active":"2024-09-21T14:03:49.274650+0000","last_peered":"2024-09-21T14:03:49.274650+0000","last_clean":"2024-09-21T14:03:49.274650+0000","last_became_active":"2024-09-21T14:00:41.071042+0000","last_became_peered":"2024-09-21T14:00:41.071042+0000","last_unstale":"2024-09-21T14:03:49.274650+0000","last_undegraded":"2024-09-21T14:03:49.274650+0000","last_fullsized":"2024-09-21T14:03:49.274650+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:04:42.447828+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"290'24","reported_seq":816,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-21T14:03:49.291459+0000","last_change":"2024-09-21T14:02:57.560019+0000","last_active":"2024-09-21T14:03:49.291459+0000","last_peered":"2024-09-21T14:03:49.291459+0000","last_clean":"2024-09-21T14:03:49.291459+0000","last_became_active":"2024-09-21T14:01:41.184728+0000","last_became_peered":"2024-09-21T14:01:41.184728+0000","last_unstale":"2024-09-21T14:03:49.291459+0000","last_undegraded":"2024-09-21T14:03:49.291459+0000","last_fullsized":"2024-09-21T14:03:49.291459+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'24","last_scrub_stamp":"2024-09-21T14:02:57.559936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:57.559936+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:26.628231+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":658,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.114709+0000","last_change":"2024-09-21T13:59:59.450613+0000","last_active":"2024-09-21T14:03:40.114709+0000","last_peered":"2024-09-21T14:03:40.114709+0000","last_clean":"2024-09-21T14:03:40.114709+0000","last_became_active":"2024-09-21T13:57:12.681030+0000","last_became_peered":"2024-09-21T13:57:12.681030+0000","last_unstale":"2024-09-21T14:03:40.114709+0000","last_undegraded":"2024-09-21T14:03:40.114709+0000","last_fullsized":"2024-09-21T14:03:40.114709+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:59.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:59:59.450546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:40.683320+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":1471,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:44.593120+0000","last_change":"2024-09-21T13:59:32.218796+0000","last_active":"2024-09-21T14:03:44.593120+0000","last_peered":"2024-09-21T14:03:44.593120+0000","last_clean":"2024-09-21T14:03:44.593120+0000","last_became_active":"2024-09-21T13:59:32.203361+0000","last_became_peered":"2024-09-21T13:59:32.203361+0000","last_unstale":"2024-09-21T14:03:44.593120+0000","last_undegraded":"2024-09-21T14:03:44.593120+0000","last_fullsized":"2024-09-21T14:03:44.593120+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-21T13:56:29.617143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:56:29.617143+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:20.983081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047048000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":260,"num_read_kb":255,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":646,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.118819+0000","last_change":"2024-09-21T14:03:06.470855+0000","last_active":"2024-09-21T14:03:40.118819+0000","last_peered":"2024-09-21T14:03:40.118819+0000","last_clean":"2024-09-21T14:03:40.118819+0000","last_became_active":"2024-09-21T14:01:41.200531+0000","last_became_peered":"2024-09-21T14:01:41.200531+0000","last_unstale":"2024-09-21T14:03:40.118819+0000","last_undegraded":"2024-09-21T14:03:40.118819+0000","last_fullsized":"2024-09-21T14:03:40.118819+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:06.470705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:06.470705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:32.367248+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":752,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-21T14:03:49.274638+0000","last_change":"2024-09-21T14:03:04.357026+0000","last_active":"2024-09-21T14:03:49.274638+0000","last_peered":"2024-09-21T14:03:49.274638+0000","last_clean":"2024-09-21T14:03:49.274638+0000","last_became_active":"2024-09-21T14:01:41.184664+0000","last_became_peered":"2024-09-21T14:01:41.184664+0000","last_unstale":"2024-09-21T14:03:49.274638+0000","last_undegraded":"2024-09-21T14:03:49.274638+0000","last_fullsized":"2024-09-21T14:03:49.274638+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:04.356979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:04.356979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:19.365633+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"398'4734","reported_seq":4779,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:48.568269+0000","last_change":"2024-09-21T14:02:56.557541+0000","last_active":"2024-09-21T14:03:48.568269+0000","last_peered":"2024-09-21T14:03:48.568269+0000","last_clean":"2024-09-21T14:03:48.568269+0000","last_became_active":"2024-09-21T14:02:56.557263+0000","last_became_peered":"2024-09-21T14:02:56.557263+0000","last_unstale":"2024-09-21T14:03:48.568269+0000","last_undegraded":"2024-09-21T14:03:48.568269+0000","last_fullsized":"2024-09-21T14:03:48.568269+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":4734,"log_dups_size":0,"ondisk_log_size":4734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:05.629424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310247424,"num_objects":4734,"num_object_clones":0,"num_object_copies":14202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4734,"num_write_kb":302976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"300'32","reported_seq":408,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:43.302064+0000","last_change":"2024-09-21T14:00:47.130089+0000","last_active":"2024-09-21T14:03:43.302064+0000","last_peered":"2024-09-21T14:03:43.302064+0000","last_clean":"2024-09-21T14:03:43.302064+0000","last_became_active":"2024-09-21T13:59:32.204624+0000","last_became_peered":"2024-09-21T13:59:32.204624+0000","last_unstale":"2024-09-21T14:03:43.302064+0000","last_undegraded":"2024-09-21T14:03:43.302064+0000","last_fullsized":"2024-09-21T14:03:43.302064+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'32","last_scrub_stamp":"2024-09-21T14:00:47.129789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:00:47.129789+0000","objects_scrubbed":8,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:16.636929+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":696,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.258924+0000","last_change":"2024-09-21T14:01:41.190537+0000","last_active":"2024-09-21T14:03:40.258924+0000","last_peered":"2024-09-21T14:03:40.258924+0000","last_clean":"2024-09-21T14:03:40.258924+0000","last_became_active":"2024-09-21T14:01:41.190210+0000","last_became_peered":"2024-09-21T14:01:41.190210+0000","last_unstale":"2024-09-21T14:03:40.258924+0000","last_undegraded":"2024-09-21T14:03:40.258924+0000","last_fullsized":"2024-09-21T14:03:40.258924+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:57.772975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:57.772975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"75'79","reported_seq":757,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.125629+0000","last_change":"2024-09-21T14:00:39.339360+0000","last_active":"2024-09-21T14:03:40.125629+0000","last_peered":"2024-09-21T14:03:40.125629+0000","last_clean":"2024-09-21T14:03:40.125629+0000","last_became_active":"2024-09-21T14:00:39.339077+0000","last_became_peered":"2024-09-21T14:00:39.339077+0000","last_unstale":"2024-09-21T14:03:40.125629+0000","last_undegraded":"2024-09-21T14:03:40.125629+0000","last_fullsized":"2024-09-21T14:03:40.125629+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:03:49.763645+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":19,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53669,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":671,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.124316+0000","last_change":"2024-09-21T14:01:41.182186+0000","last_active":"2024-09-21T14:03:40.124316+0000","last_peered":"2024-09-21T14:03:40.124316+0000","last_clean":"2024-09-21T14:03:40.124316+0000","last_became_active":"2024-09-21T14:01:41.181899+0000","last_became_peered":"2024-09-21T14:01:41.181899+0000","last_unstale":"2024-09-21T14:03:40.124316+0000","last_undegraded":"2024-09-21T14:03:40.124316+0000","last_fullsized":"2024-09-21T14:03:40.124316+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:15.700409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:15.700409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":596,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.118515+0000","last_change":"2024-09-21T14:01:41.196509+0000","last_active":"2024-09-21T14:03:40.118515+0000","last_peered":"2024-09-21T14:03:40.118515+0000","last_clean":"2024-09-21T14:03:40.118515+0000","last_became_active":"2024-09-21T14:01:41.196243+0000","last_became_peered":"2024-09-21T14:01:41.196243+0000","last_unstale":"2024-09-21T14:03:40.118515+0000","last_undegraded":"2024-09-21T14:03:40.118515+0000","last_fullsized":"2024-09-21T14:03:40.118515+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:21.153018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:21.153018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:17.291763+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"398'4826","reported_seq":4871,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:48.525277+0000","last_change":"2024-09-21T14:02:56.561047+0000","last_active":"2024-09-21T14:03:48.525277+0000","last_peered":"2024-09-21T14:03:48.525277+0000","last_clean":"2024-09-21T14:03:48.525277+0000","last_became_active":"2024-09-21T14:02:56.558774+0000","last_became_peered":"2024-09-21T14:02:56.558774+0000","last_unstale":"2024-09-21T14:03:48.525277+0000","last_undegraded":"2024-09-21T14:03:48.525277+0000","last_fullsized":"2024-09-21T14:03:48.525277+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":4826,"log_dups_size":0,"ondisk_log_size":4826,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:07.590313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":316276736,"num_objects":4826,"num_object_clones":0,"num_object_copies":14478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4826,"num_write_kb":308864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"290'19","reported_seq":782,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.116046+0000","last_change":"2024-09-21T14:02:28.553549+0000","last_active":"2024-09-21T14:03:40.116046+0000","last_peered":"2024-09-21T14:03:40.116046+0000","last_clean":"2024-09-21T14:03:40.116046+0000","last_became_active":"2024-09-21T13:58:17.776310+0000","last_became_peered":"2024-09-21T13:58:17.776310+0000","last_unstale":"2024-09-21T14:03:40.116046+0000","last_undegraded":"2024-09-21T14:03:40.116046+0000","last_fullsized":"2024-09-21T14:03:40.116046+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'19","last_scrub_stamp":"2024-09-21T14:02:28.553065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:28.553065+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":752,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.115469+0000","last_change":"2024-09-21T14:03:32.957705+0000","last_active":"2024-09-21T14:03:40.115469+0000","last_peered":"2024-09-21T14:03:40.115469+0000","last_clean":"2024-09-21T14:03:40.115469+0000","last_became_active":"2024-09-21T13:59:32.223247+0000","last_became_peered":"2024-09-21T13:59:32.223247+0000","last_unstale":"2024-09-21T14:03:40.115469+0000","last_undegraded":"2024-09-21T14:03:40.115469+0000","last_fullsized":"2024-09-21T14:03:40.115469+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:32.957649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:03:32.957649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:44.854711+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":671,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.118393+0000","last_change":"2024-09-21T14:02:58.461716+0000","last_active":"2024-09-21T14:03:40.118393+0000","last_peered":"2024-09-21T14:03:40.118393+0000","last_clean":"2024-09-21T14:03:40.118393+0000","last_became_active":"2024-09-21T14:01:41.193941+0000","last_became_peered":"2024-09-21T14:01:41.193941+0000","last_unstale":"2024-09-21T14:03:40.118393+0000","last_undegraded":"2024-09-21T14:03:40.118393+0000","last_fullsized":"2024-09-21T14:03:40.118393+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:58.461663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:02:58.461663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:20.802609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":681,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.262480+0000","last_change":"2024-09-21T14:03:36.302360+0000","last_active":"2024-09-21T14:03:40.262480+0000","last_peered":"2024-09-21T14:03:40.262480+0000","last_clean":"2024-09-21T14:03:40.262480+0000","last_became_active":"2024-09-21T14:01:41.190192+0000","last_became_peered":"2024-09-21T14:01:41.190192+0000","last_unstale":"2024-09-21T14:03:40.262480+0000","last_undegraded":"2024-09-21T14:03:40.262480+0000","last_fullsized":"2024-09-21T14:03:40.262480+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:36.302281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:36.302281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:37.898600+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"398'4665","reported_seq":4710,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:47.742553+0000","last_change":"2024-09-21T14:02:56.560359+0000","last_active":"2024-09-21T14:03:47.742553+0000","last_peered":"2024-09-21T14:03:47.742553+0000","last_clean":"2024-09-21T14:03:47.742553+0000","last_became_active":"2024-09-21T14:02:56.560070+0000","last_became_peered":"2024-09-21T14:02:56.560070+0000","last_unstale":"2024-09-21T14:03:47.742553+0000","last_undegraded":"2024-09-21T14:03:47.742553+0000","last_fullsized":"2024-09-21T14:03:47.742553+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":4665,"log_dups_size":0,"ondisk_log_size":4665,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:16.453643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":305725440,"num_objects":4665,"num_object_clones":0,"num_object_copies":13995,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4665,"num_write_kb":298560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"75'79","reported_seq":136,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.116128+0000","last_change":"2024-09-21T14:01:41.189146+0000","last_active":"2024-09-21T14:03:40.116128+0000","last_peered":"2024-09-21T14:03:40.116128+0000","last_clean":"2024-09-21T14:03:40.116128+0000","last_became_active":"2024-09-21T14:01:41.188685+0000","last_became_peered":"2024-09-21T14:01:41.188685+0000","last_unstale":"2024-09-21T14:03:40.116128+0000","last_undegraded":"2024-09-21T14:03:40.116128+0000","last_fullsized":"2024-09-21T14:03:40.116128+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:03:46.437011+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"290'31","reported_seq":803,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.115000+0000","last_change":"2024-09-21T14:03:31.017284+0000","last_active":"2024-09-21T14:03:40.115000+0000","last_peered":"2024-09-21T14:03:40.115000+0000","last_clean":"2024-09-21T14:03:40.115000+0000","last_became_active":"2024-09-21T14:03:29.162483+0000","last_became_peered":"2024-09-21T14:03:29.162483+0000","last_unstale":"2024-09-21T14:03:40.115000+0000","last_undegraded":"2024-09-21T14:03:40.115000+0000","last_fullsized":"2024-09-21T14:03:40.115000+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'31","last_scrub_stamp":"2024-09-21T14:03:31.017119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:03:31.017119+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:54.975325+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":286,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.117085+0000","last_change":"2024-09-21T14:01:29.924408+0000","last_active":"2024-09-21T14:03:40.117085+0000","last_peered":"2024-09-21T14:03:40.117085+0000","last_clean":"2024-09-21T14:03:40.117085+0000","last_became_active":"2024-09-21T13:58:17.762520+0000","last_became_peered":"2024-09-21T13:58:17.762520+0000","last_unstale":"2024-09-21T14:03:40.117085+0000","last_undegraded":"2024-09-21T14:03:40.117085+0000","last_fullsized":"2024-09-21T14:03:40.117085+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:29.924331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:01:29.924331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:47.626829+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":701,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-21T14:03:49.274722+0000","last_change":"2024-09-21T13:58:52.030800+0000","last_active":"2024-09-21T14:03:49.274722+0000","last_peered":"2024-09-21T14:03:49.274722+0000","last_clean":"2024-09-21T14:03:49.274722+0000","last_became_active":"2024-09-21T13:49:02.067472+0000","last_became_peered":"2024-09-21T13:49:02.067472+0000","last_unstale":"2024-09-21T14:03:49.274722+0000","last_undegraded":"2024-09-21T14:03:49.274722+0000","last_fullsized":"2024-09-21T14:03:49.274722+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:52.030718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:52.030718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":582,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.114206+0000","last_change":"2024-09-21T13:57:12.683321+0000","last_active":"2024-09-21T14:03:40.114206+0000","last_peered":"2024-09-21T14:03:40.114206+0000","last_clean":"2024-09-21T14:03:40.114206+0000","last_became_active":"2024-09-21T13:57:12.682979+0000","last_became_peered":"2024-09-21T13:57:12.682979+0000","last_unstale":"2024-09-21T14:03:40.114206+0000","last_undegraded":"2024-09-21T14:03:40.114206+0000","last_fullsized":"2024-09-21T14:03:40.114206+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:18.027944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:18.027944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:22.725610+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2","version":"399'4914","reported_seq":4960,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-21T14:03:49.721860+0000","last_change":"2024-09-21T14:02:56.559657+0000","last_active":"2024-09-21T14:03:49.721860+0000","last_peered":"2024-09-21T14:03:49.721860+0000","last_clean":"2024-09-21T14:03:49.721860+0000","last_became_active":"2024-09-21T14:02:56.559368+0000","last_became_peered":"2024-09-21T14:02:56.559368+0000","last_unstale":"2024-09-21T14:03:49.721860+0000","last_undegraded":"2024-09-21T14:03:49.721860+0000","last_fullsized":"2024-09-21T14:03:49.721860+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":4914,"log_dups_size":0,"ondisk_log_size":4914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:23.196521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":322043904,"num_objects":4914,"num_object_clones":0,"num_object_copies":14742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4914,"num_write_kb":314496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"75'79","reported_seq":770,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.124771+0000","last_change":"2024-09-21T14:02:55.806160+0000","last_active":"2024-09-21T14:03:40.124771+0000","last_peered":"2024-09-21T14:03:40.124771+0000","last_clean":"2024-09-21T14:03:40.124771+0000","last_became_active":"2024-09-21T14:00:41.139315+0000","last_became_peered":"2024-09-21T14:00:41.139315+0000","last_unstale":"2024-09-21T14:03:40.124771+0000","last_undegraded":"2024-09-21T14:03:40.124771+0000","last_fullsized":"2024-09-21T14:03:40.124771+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:02:55.806077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.806077+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:59.364707+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"290'20","reported_seq":789,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.115610+0000","last_change":"2024-09-21T13:58:17.789407+0000","last_active":"2024-09-21T14:03:40.115610+0000","last_peered":"2024-09-21T14:03:40.115610+0000","last_clean":"2024-09-21T14:03:40.115610+0000","last_became_active":"2024-09-21T13:58:17.765427+0000","last_became_peered":"2024-09-21T13:58:17.765427+0000","last_unstale":"2024-09-21T14:03:40.115610+0000","last_undegraded":"2024-09-21T14:03:40.115610+0000","last_fullsized":"2024-09-21T14:03:40.115610+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:58:13.464196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:13.464196+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:18.848758+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":638,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.260423+0000","last_change":"2024-09-21T14:02:58.236695+0000","last_active":"2024-09-21T14:03:40.260423+0000","last_peered":"2024-09-21T14:03:40.260423+0000","last_clean":"2024-09-21T14:03:40.260423+0000","last_became_active":"2024-09-21T13:57:12.704203+0000","last_became_peered":"2024-09-21T13:57:12.704203+0000","last_unstale":"2024-09-21T14:03:40.260423+0000","last_undegraded":"2024-09-21T14:03:40.260423+0000","last_fullsized":"2024-09-21T14:03:40.260423+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:58.236645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:02:58.236645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:21.993648+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"346'14","reported_seq":773,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.113779+0000","last_change":"2024-09-21T14:01:41.186960+0000","last_active":"2024-09-21T14:03:40.113779+0000","last_peered":"2024-09-21T14:03:40.113779+0000","last_clean":"2024-09-21T14:03:40.113779+0000","last_became_active":"2024-09-21T14:01:41.186564+0000","last_became_peered":"2024-09-21T14:01:41.186564+0000","last_unstale":"2024-09-21T14:03:40.113779+0000","last_undegraded":"2024-09-21T14:03:40.113779+0000","last_fullsized":"2024-09-21T14:03:40.113779+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'12","last_scrub_stamp":"2024-09-21T13:59:38.738120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:38.738120+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:47.648985+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"68'1","reported_seq":717,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.115873+0000","last_change":"2024-09-21T13:59:31.392186+0000","last_active":"2024-09-21T14:03:40.115873+0000","last_peered":"2024-09-21T14:03:40.115873+0000","last_clean":"2024-09-21T14:03:40.115873+0000","last_became_active":"2024-09-21T13:52:04.313210+0000","last_became_peered":"2024-09-21T13:52:04.313210+0000","last_unstale":"2024-09-21T14:03:40.115873+0000","last_undegraded":"2024-09-21T14:03:40.115873+0000","last_fullsized":"2024-09-21T14:03:40.115873+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T13:59:31.392105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:31.392105+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.5","version":"398'4829","reported_seq":4874,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:47.784777+0000","last_change":"2024-09-21T14:02:56.557127+0000","last_active":"2024-09-21T14:03:47.784777+0000","last_peered":"2024-09-21T14:03:47.784777+0000","last_clean":"2024-09-21T14:03:47.784777+0000","last_became_active":"2024-09-21T14:02:56.556786+0000","last_became_peered":"2024-09-21T14:02:56.556786+0000","last_unstale":"2024-09-21T14:03:47.784777+0000","last_undegraded":"2024-09-21T14:03:47.784777+0000","last_fullsized":"2024-09-21T14:03:47.784777+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":4829,"log_dups_size":0,"ondisk_log_size":4829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:16.085110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":316473344,"num_objects":4829,"num_object_clones":0,"num_object_copies":14487,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4829,"num_write_kb":309056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"290'15","reported_seq":746,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-21T14:03:49.290059+0000","last_change":"2024-09-21T13:58:50.498597+0000","last_active":"2024-09-21T14:03:49.290059+0000","last_peered":"2024-09-21T14:03:49.290059+0000","last_clean":"2024-09-21T14:03:49.290059+0000","last_became_active":"2024-09-21T13:49:00.053016+0000","last_became_peered":"2024-09-21T13:49:00.053016+0000","last_unstale":"2024-09-21T14:03:49.290059+0000","last_undegraded":"2024-09-21T14:03:49.290059+0000","last_fullsized":"2024-09-21T14:03:49.290059+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:58:50.498369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:50.498369+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:54.564652+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":675,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.116881+0000","last_change":"2024-09-21T14:01:41.192717+0000","last_active":"2024-09-21T14:03:40.116881+0000","last_peered":"2024-09-21T14:03:40.116881+0000","last_clean":"2024-09-21T14:03:40.116881+0000","last_became_active":"2024-09-21T14:01:41.192443+0000","last_became_peered":"2024-09-21T14:01:41.192443+0000","last_unstale":"2024-09-21T14:03:40.116881+0000","last_undegraded":"2024-09-21T14:03:40.116881+0000","last_fullsized":"2024-09-21T14:03:40.116881+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:53.894506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:54:53.894506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:53.450976+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":664,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.115046+0000","last_change":"2024-09-21T14:02:56.796917+0000","last_active":"2024-09-21T14:03:40.115046+0000","last_peered":"2024-09-21T14:03:40.115046+0000","last_clean":"2024-09-21T14:03:40.115046+0000","last_became_active":"2024-09-21T13:57:12.680978+0000","last_became_peered":"2024-09-21T13:57:12.680978+0000","last_unstale":"2024-09-21T14:03:40.115046+0000","last_undegraded":"2024-09-21T14:03:40.115046+0000","last_fullsized":"2024-09-21T14:03:40.115046+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:56.796861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:02:56.796861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:10.252837+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":605,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.124608+0000","last_change":"2024-09-21T13:59:32.224001+0000","last_active":"2024-09-21T14:03:40.124608+0000","last_peered":"2024-09-21T14:03:40.124608+0000","last_clean":"2024-09-21T14:03:40.124608+0000","last_became_active":"2024-09-21T13:59:32.221628+0000","last_became_peered":"2024-09-21T13:59:32.221628+0000","last_unstale":"2024-09-21T14:03:40.124608+0000","last_undegraded":"2024-09-21T14:03:40.124608+0000","last_fullsized":"2024-09-21T14:03:40.124608+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:13.698091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:13.698091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"398'4748","reported_seq":4793,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:48.523529+0000","last_change":"2024-09-21T14:02:56.562973+0000","last_active":"2024-09-21T14:03:48.523529+0000","last_peered":"2024-09-21T14:03:48.523529+0000","last_clean":"2024-09-21T14:03:48.523529+0000","last_became_active":"2024-09-21T14:02:56.562256+0000","last_became_peered":"2024-09-21T14:02:56.562256+0000","last_unstale":"2024-09-21T14:03:48.523529+0000","last_undegraded":"2024-09-21T14:03:48.523529+0000","last_fullsized":"2024-09-21T14:03:48.523529+0000","mapping_epoch":381,"log_start":"386'300","ondisk_log_start":"386'300","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":4448,"log_dups_size":0,"ondisk_log_size":4448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:09.952633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":311164928,"num_objects":4748,"num_object_clones":0,"num_object_copies":14244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4748,"num_write_kb":303872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"290'35","reported_seq":352,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.117210+0000","last_change":"2024-09-21T14:01:02.006619+0000","last_active":"2024-09-21T14:03:40.117210+0000","last_peered":"2024-09-21T14:03:40.117210+0000","last_clean":"2024-09-21T14:03:40.117210+0000","last_became_active":"2024-09-21T13:59:32.204184+0000","last_became_peered":"2024-09-21T13:59:32.204184+0000","last_unstale":"2024-09-21T14:03:40.117210+0000","last_undegraded":"2024-09-21T14:03:40.117210+0000","last_fullsized":"2024-09-21T14:03:40.117210+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'35","last_scrub_stamp":"2024-09-21T14:01:02.006504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:02.006504+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:24.584005+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":705,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.125420+0000","last_change":"2024-09-21T14:03:31.882264+0000","last_active":"2024-09-21T14:03:40.125420+0000","last_peered":"2024-09-21T14:03:40.125420+0000","last_clean":"2024-09-21T14:03:40.125420+0000","last_became_active":"2024-09-21T14:01:41.184567+0000","last_became_peered":"2024-09-21T14:01:41.184567+0000","last_unstale":"2024-09-21T14:03:40.125420+0000","last_undegraded":"2024-09-21T14:03:40.125420+0000","last_fullsized":"2024-09-21T14:03:40.125420+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:31.882215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:03:31.882215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:38.486020+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":676,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.115195+0000","last_change":"2024-09-21T14:01:41.182250+0000","last_active":"2024-09-21T14:03:40.115195+0000","last_peered":"2024-09-21T14:03:40.115195+0000","last_clean":"2024-09-21T14:03:40.115195+0000","last_became_active":"2024-09-21T14:01:41.181911+0000","last_became_peered":"2024-09-21T14:01:41.181911+0000","last_unstale":"2024-09-21T14:03:40.115195+0000","last_undegraded":"2024-09-21T14:03:40.115195+0000","last_fullsized":"2024-09-21T14:03:40.115195+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:55.588554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:55.588554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:45.648976+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":632,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.125151+0000","last_change":"2024-09-21T14:03:33.893158+0000","last_active":"2024-09-21T14:03:40.125151+0000","last_peered":"2024-09-21T14:03:40.125151+0000","last_clean":"2024-09-21T14:03:40.125151+0000","last_became_active":"2024-09-21T13:59:32.219332+0000","last_became_peered":"2024-09-21T13:59:32.219332+0000","last_unstale":"2024-09-21T14:03:40.125151+0000","last_undegraded":"2024-09-21T14:03:40.125151+0000","last_fullsized":"2024-09-21T14:03:40.125151+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:33.893072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:33.893072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:40.141218+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"398'4831","reported_seq":4876,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:48.580252+0000","last_change":"2024-09-21T14:02:56.559626+0000","last_active":"2024-09-21T14:03:48.580252+0000","last_peered":"2024-09-21T14:03:48.580252+0000","last_clean":"2024-09-21T14:03:48.580252+0000","last_became_active":"2024-09-21T14:02:56.559331+0000","last_became_peered":"2024-09-21T14:02:56.559331+0000","last_unstale":"2024-09-21T14:03:48.580252+0000","last_undegraded":"2024-09-21T14:03:48.580252+0000","last_fullsized":"2024-09-21T14:03:48.580252+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":4831,"log_dups_size":0,"ondisk_log_size":4831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:20.185982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":316604416,"num_objects":4831,"num_object_clones":0,"num_object_copies":14493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4831,"num_write_kb":309184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"290'18","reported_seq":761,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.115553+0000","last_change":"2024-09-21T13:59:28.340494+0000","last_active":"2024-09-21T14:03:40.115553+0000","last_peered":"2024-09-21T14:03:40.115553+0000","last_clean":"2024-09-21T14:03:40.115553+0000","last_became_active":"2024-09-21T13:57:12.680627+0000","last_became_peered":"2024-09-21T13:57:12.680627+0000","last_unstale":"2024-09-21T14:03:40.115553+0000","last_undegraded":"2024-09-21T14:03:40.115553+0000","last_fullsized":"2024-09-21T14:03:40.115553+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'18","last_scrub_stamp":"2024-09-21T13:59:28.340415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:28.340415+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:11.926616+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":652,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.126296+0000","last_change":"2024-09-21T13:59:32.201126+0000","last_active":"2024-09-21T14:03:40.126296+0000","last_peered":"2024-09-21T14:03:40.126296+0000","last_clean":"2024-09-21T14:03:40.126296+0000","last_became_active":"2024-09-21T13:59:32.200674+0000","last_became_peered":"2024-09-21T13:59:32.200674+0000","last_unstale":"2024-09-21T14:03:40.126296+0000","last_undegraded":"2024-09-21T14:03:40.126296+0000","last_fullsized":"2024-09-21T14:03:40.126296+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:21.582325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:21.582325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:12.941161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":638,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.113384+0000","last_change":"2024-09-21T14:02:27.791041+0000","last_active":"2024-09-21T14:03:40.113384+0000","last_peered":"2024-09-21T14:03:40.113384+0000","last_clean":"2024-09-21T14:03:40.113384+0000","last_became_active":"2024-09-21T13:52:04.322147+0000","last_became_peered":"2024-09-21T13:52:04.322147+0000","last_unstale":"2024-09-21T14:03:40.113384+0000","last_undegraded":"2024-09-21T14:03:40.113384+0000","last_fullsized":"2024-09-21T14:03:40.113384+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:27.790994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:02:27.790994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:51.879201+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":646,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.259980+0000","last_change":"2024-09-21T14:02:36.371955+0000","last_active":"2024-09-21T14:03:40.259980+0000","last_peered":"2024-09-21T14:03:40.259980+0000","last_clean":"2024-09-21T14:03:40.259980+0000","last_became_active":"2024-09-21T13:59:32.203767+0000","last_became_peered":"2024-09-21T13:59:32.203767+0000","last_unstale":"2024-09-21T14:03:40.259980+0000","last_undegraded":"2024-09-21T14:03:40.259980+0000","last_fullsized":"2024-09-21T14:03:40.259980+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:36.371747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:02:36.371747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:54.168810+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"398'4586","reported_seq":4631,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:45.854735+0000","last_change":"2024-09-21T14:02:56.564166+0000","last_active":"2024-09-21T14:03:45.854735+0000","last_peered":"2024-09-21T14:03:45.854735+0000","last_clean":"2024-09-21T14:03:45.854735+0000","last_became_active":"2024-09-21T14:02:56.563788+0000","last_became_peered":"2024-09-21T14:02:56.563788+0000","last_unstale":"2024-09-21T14:03:45.854735+0000","last_undegraded":"2024-09-21T14:03:45.854735+0000","last_fullsized":"2024-09-21T14:03:45.854735+0000","mapping_epoch":381,"log_start":"0'0","ondisk_log_start":"0'0","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":4586,"log_dups_size":0,"ondisk_log_size":4586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:20.466245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":300548096,"num_objects":4586,"num_object_clones":0,"num_object_copies":13758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4586,"num_write_kb":293504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"290'27","reported_seq":759,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.116242+0000","last_change":"2024-09-21T14:03:31.360992+0000","last_active":"2024-09-21T14:03:40.116242+0000","last_peered":"2024-09-21T14:03:40.116242+0000","last_clean":"2024-09-21T14:03:40.116242+0000","last_became_active":"2024-09-21T14:01:41.192481+0000","last_became_peered":"2024-09-21T14:01:41.192481+0000","last_unstale":"2024-09-21T14:03:40.116242+0000","last_undegraded":"2024-09-21T14:03:40.116242+0000","last_fullsized":"2024-09-21T14:03:40.116242+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'27","last_scrub_stamp":"2024-09-21T14:03:31.360906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:03:31.360906+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:41.422161+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":632,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.126048+0000","last_change":"2024-09-21T14:03:04.953724+0000","last_active":"2024-09-21T14:03:40.126048+0000","last_peered":"2024-09-21T14:03:40.126048+0000","last_clean":"2024-09-21T14:03:40.126048+0000","last_became_active":"2024-09-21T13:59:32.203172+0000","last_became_peered":"2024-09-21T13:59:32.203172+0000","last_unstale":"2024-09-21T14:03:40.126048+0000","last_undegraded":"2024-09-21T14:03:40.126048+0000","last_fullsized":"2024-09-21T14:03:40.126048+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T14:03:04.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:03:04.953658+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:14.479415+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"300'14","reported_seq":849,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.124516+0000","last_change":"2024-09-21T13:59:32.220898+0000","last_active":"2024-09-21T14:03:40.124516+0000","last_peered":"2024-09-21T14:03:40.124516+0000","last_clean":"2024-09-21T14:03:40.124516+0000","last_became_active":"2024-09-21T13:59:32.206956+0000","last_became_peered":"2024-09-21T13:59:32.206956+0000","last_unstale":"2024-09-21T14:03:40.124516+0000","last_undegraded":"2024-09-21T14:03:40.124516+0000","last_fullsized":"2024-09-21T14:03:40.124516+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'11","last_scrub_stamp":"2024-09-21T13:56:31.616003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:56:31.616003+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:11.986942+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":699,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.116602+0000","last_change":"2024-09-21T14:01:41.248408+0000","last_active":"2024-09-21T14:03:40.116602+0000","last_peered":"2024-09-21T14:03:40.116602+0000","last_clean":"2024-09-21T14:03:40.116602+0000","last_became_active":"2024-09-21T14:01:41.246106+0000","last_became_peered":"2024-09-21T14:01:41.246106+0000","last_unstale":"2024-09-21T14:03:40.116602+0000","last_undegraded":"2024-09-21T14:03:40.116602+0000","last_fullsized":"2024-09-21T14:03:40.116602+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:39.258659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:39.258659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:22.284546+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":289,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.118109+0000","last_change":"2024-09-21T14:01:50.733055+0000","last_active":"2024-09-21T14:03:40.118109+0000","last_peered":"2024-09-21T14:03:40.118109+0000","last_clean":"2024-09-21T14:03:40.118109+0000","last_became_active":"2024-09-21T14:01:41.197501+0000","last_became_peered":"2024-09-21T14:01:41.197501+0000","last_unstale":"2024-09-21T14:03:40.118109+0000","last_undegraded":"2024-09-21T14:03:40.118109+0000","last_fullsized":"2024-09-21T14:03:40.118109+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:50.733007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:01:50.733007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:17.685126+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"290'12","reported_seq":673,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.260387+0000","last_change":"2024-09-21T13:59:32.247422+0000","last_active":"2024-09-21T14:03:40.260387+0000","last_peered":"2024-09-21T14:03:40.260387+0000","last_clean":"2024-09-21T14:03:40.260387+0000","last_became_active":"2024-09-21T13:59:32.211306+0000","last_became_peered":"2024-09-21T13:59:32.211306+0000","last_unstale":"2024-09-21T14:03:40.260387+0000","last_undegraded":"2024-09-21T14:03:40.260387+0000","last_fullsized":"2024-09-21T14:03:40.260387+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'6","last_scrub_stamp":"2024-09-21T13:55:12.066873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:55:12.066873+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":631,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.117855+0000","last_change":"2024-09-21T14:01:41.199742+0000","last_active":"2024-09-21T14:03:40.117855+0000","last_peered":"2024-09-21T14:03:40.117855+0000","last_clean":"2024-09-21T14:03:40.117855+0000","last_became_active":"2024-09-21T14:01:41.199321+0000","last_became_peered":"2024-09-21T14:01:41.199321+0000","last_unstale":"2024-09-21T14:03:40.117855+0000","last_undegraded":"2024-09-21T14:03:40.117855+0000","last_fullsized":"2024-09-21T14:03:40.117855+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:37.106238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:37.106238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:16.289450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":678,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.125294+0000","last_change":"2024-09-21T14:03:01.872986+0000","last_active":"2024-09-21T14:03:40.125294+0000","last_peered":"2024-09-21T14:03:40.125294+0000","last_clean":"2024-09-21T14:03:40.125294+0000","last_became_active":"2024-09-21T13:59:32.219525+0000","last_became_peered":"2024-09-21T13:59:32.219525+0000","last_unstale":"2024-09-21T14:03:40.125294+0000","last_undegraded":"2024-09-21T14:03:40.125294+0000","last_fullsized":"2024-09-21T14:03:40.125294+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:01.872928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:01.872928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:09.296117+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"290'15","reported_seq":636,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.257430+0000","last_change":"2024-09-21T13:59:32.238689+0000","last_active":"2024-09-21T14:03:40.257430+0000","last_peered":"2024-09-21T14:03:40.257430+0000","last_clean":"2024-09-21T14:03:40.257430+0000","last_became_active":"2024-09-21T13:59:32.209166+0000","last_became_peered":"2024-09-21T13:59:32.209166+0000","last_unstale":"2024-09-21T14:03:40.257430+0000","last_undegraded":"2024-09-21T14:03:40.257430+0000","last_fullsized":"2024-09-21T14:03:40.257430+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:55:04.990842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:55:04.990842+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"61'1","reported_seq":688,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.124766+0000","last_change":"2024-09-21T13:59:32.222826+0000","last_active":"2024-09-21T14:03:40.124766+0000","last_peered":"2024-09-21T14:03:40.124766+0000","last_clean":"2024-09-21T14:03:40.124766+0000","last_became_active":"2024-09-21T13:59:32.219515+0000","last_became_peered":"2024-09-21T13:59:32.219515+0000","last_unstale":"2024-09-21T14:03:40.124766+0000","last_undegraded":"2024-09-21T14:03:40.124766+0000","last_fullsized":"2024-09-21T14:03:40.124766+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:58:07.711028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:07.711028+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":629,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.257430+0000","last_change":"2024-09-21T14:03:33.309440+0000","last_active":"2024-09-21T14:03:40.257430+0000","last_peered":"2024-09-21T14:03:40.257430+0000","last_clean":"2024-09-21T14:03:40.257430+0000","last_became_active":"2024-09-21T13:59:32.210586+0000","last_became_peered":"2024-09-21T13:59:32.210586+0000","last_unstale":"2024-09-21T14:03:40.257430+0000","last_undegraded":"2024-09-21T14:03:40.257430+0000","last_fullsized":"2024-09-21T14:03:40.257430+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:33.309248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:33.309248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:57.005556+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":680,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.113959+0000","last_change":"2024-09-21T14:02:29.731635+0000","last_active":"2024-09-21T14:03:40.113959+0000","last_peered":"2024-09-21T14:03:40.113959+0000","last_clean":"2024-09-21T14:03:40.113959+0000","last_became_active":"2024-09-21T13:59:32.205803+0000","last_became_peered":"2024-09-21T13:59:32.205803+0000","last_unstale":"2024-09-21T14:03:40.113959+0000","last_undegraded":"2024-09-21T14:03:40.113959+0000","last_fullsized":"2024-09-21T14:03:40.113959+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:29.731565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:02:29.731565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:53.473477+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"290'15","reported_seq":676,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.117512+0000","last_change":"2024-09-21T14:01:41.189676+0000","last_active":"2024-09-21T14:03:40.117512+0000","last_peered":"2024-09-21T14:03:40.117512+0000","last_clean":"2024-09-21T14:03:40.117512+0000","last_became_active":"2024-09-21T14:01:41.187904+0000","last_became_peered":"2024-09-21T14:01:41.187904+0000","last_unstale":"2024-09-21T14:03:40.117512+0000","last_undegraded":"2024-09-21T14:03:40.117512+0000","last_fullsized":"2024-09-21T14:03:40.117512+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:57:24.124604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:57:24.124604+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:18.325159+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":557,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.126834+0000","last_change":"2024-09-21T13:59:32.212325+0000","last_active":"2024-09-21T14:03:40.126834+0000","last_peered":"2024-09-21T14:03:40.126834+0000","last_clean":"2024-09-21T14:03:40.126834+0000","last_became_active":"2024-09-21T13:59:32.208749+0000","last_became_peered":"2024-09-21T13:59:32.208749+0000","last_unstale":"2024-09-21T14:03:40.126834+0000","last_undegraded":"2024-09-21T14:03:40.126834+0000","last_fullsized":"2024-09-21T14:03:40.126834+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:20.574745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:20.574745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:22.976952+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"300'14","reported_seq":782,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.258123+0000","last_change":"2024-09-21T13:59:32.208253+0000","last_active":"2024-09-21T14:03:40.258123+0000","last_peered":"2024-09-21T14:03:40.258123+0000","last_clean":"2024-09-21T14:03:40.258123+0000","last_became_active":"2024-09-21T13:59:32.206926+0000","last_became_peered":"2024-09-21T13:59:32.206926+0000","last_unstale":"2024-09-21T14:03:40.258123+0000","last_undegraded":"2024-09-21T14:03:40.258123+0000","last_fullsized":"2024-09-21T14:03:40.258123+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'10","last_scrub_stamp":"2024-09-21T13:54:20.946782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:20.946782+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:15.133039+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":724,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.115227+0000","last_change":"2024-09-21T14:01:41.200564+0000","last_active":"2024-09-21T14:03:40.115227+0000","last_peered":"2024-09-21T14:03:40.115227+0000","last_clean":"2024-09-21T14:03:40.115227+0000","last_became_active":"2024-09-21T14:01:41.200113+0000","last_became_peered":"2024-09-21T14:01:41.200113+0000","last_unstale":"2024-09-21T14:03:40.115227+0000","last_undegraded":"2024-09-21T14:03:40.115227+0000","last_fullsized":"2024-09-21T14:03:40.115227+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:54.805619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:54.805619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:48.669884+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"290'20","reported_seq":672,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.258200+0000","last_change":"2024-09-21T13:59:32.236530+0000","last_active":"2024-09-21T14:03:40.258200+0000","last_peered":"2024-09-21T14:03:40.258200+0000","last_clean":"2024-09-21T14:03:40.258200+0000","last_became_active":"2024-09-21T13:59:32.208332+0000","last_became_peered":"2024-09-21T13:59:32.208332+0000","last_unstale":"2024-09-21T14:03:40.258200+0000","last_undegraded":"2024-09-21T14:03:40.258200+0000","last_fullsized":"2024-09-21T14:03:40.258200+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:57:37.671391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:57:37.671391+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:55.297241+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":664,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.124394+0000","last_change":"2024-09-21T13:59:32.212737+0000","last_active":"2024-09-21T14:03:40.124394+0000","last_peered":"2024-09-21T14:03:40.124394+0000","last_clean":"2024-09-21T14:03:40.124394+0000","last_became_active":"2024-09-21T13:59:32.209804+0000","last_became_peered":"2024-09-21T13:59:32.209804+0000","last_unstale":"2024-09-21T14:03:40.124394+0000","last_undegraded":"2024-09-21T14:03:40.124394+0000","last_fullsized":"2024-09-21T14:03:40.124394+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:25.571256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:25.571256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:15.928161+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"346'10","reported_seq":790,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.117465+0000","last_change":"2024-09-21T14:03:34.365517+0000","last_active":"2024-09-21T14:03:40.117465+0000","last_peered":"2024-09-21T14:03:40.117465+0000","last_clean":"2024-09-21T14:03:40.117465+0000","last_became_active":"2024-09-21T14:01:41.206303+0000","last_became_peered":"2024-09-21T14:01:41.206303+0000","last_unstale":"2024-09-21T14:03:40.117465+0000","last_undegraded":"2024-09-21T14:03:40.117465+0000","last_fullsized":"2024-09-21T14:03:40.117465+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'10","last_scrub_stamp":"2024-09-21T14:03:34.365428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:34.365428+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:00.856228+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":569,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.126125+0000","last_change":"2024-09-21T14:03:27.961613+0000","last_active":"2024-09-21T14:03:40.126125+0000","last_peered":"2024-09-21T14:03:40.126125+0000","last_clean":"2024-09-21T14:03:40.126125+0000","last_became_active":"2024-09-21T13:59:32.211472+0000","last_became_peered":"2024-09-21T13:59:32.211472+0000","last_unstale":"2024-09-21T14:03:40.126125+0000","last_undegraded":"2024-09-21T14:03:40.126125+0000","last_fullsized":"2024-09-21T14:03:40.126125+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:27.961544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:27.961544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:33.496172+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"290'19","reported_seq":787,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.114891+0000","last_change":"2024-09-21T14:02:34.670058+0000","last_active":"2024-09-21T14:03:40.114891+0000","last_peered":"2024-09-21T14:03:40.114891+0000","last_clean":"2024-09-21T14:03:40.114891+0000","last_became_active":"2024-09-21T13:58:17.771497+0000","last_became_peered":"2024-09-21T13:58:17.771497+0000","last_unstale":"2024-09-21T14:03:40.114891+0000","last_undegraded":"2024-09-21T14:03:40.114891+0000","last_fullsized":"2024-09-21T14:03:40.114891+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'19","last_scrub_stamp":"2024-09-21T14:02:34.669962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:34.669962+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:03:44.716931+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":306,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.124359+0000","last_change":"2024-09-21T14:02:57.807136+0000","last_active":"2024-09-21T14:03:40.124359+0000","last_peered":"2024-09-21T14:03:40.124359+0000","last_clean":"2024-09-21T14:03:40.124359+0000","last_became_active":"2024-09-21T13:59:32.201568+0000","last_became_peered":"2024-09-21T13:59:32.201568+0000","last_unstale":"2024-09-21T14:03:40.124359+0000","last_undegraded":"2024-09-21T14:03:40.124359+0000","last_fullsized":"2024-09-21T14:03:40.124359+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:57.807059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:02:57.807059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:09.290442+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":739,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-21T14:03:49.277032+0000","last_change":"2024-09-21T14:01:41.185588+0000","last_active":"2024-09-21T14:03:49.277032+0000","last_peered":"2024-09-21T14:03:49.277032+0000","last_clean":"2024-09-21T14:03:49.277032+0000","last_became_active":"2024-09-21T14:01:41.185293+0000","last_became_peered":"2024-09-21T14:01:41.185293+0000","last_unstale":"2024-09-21T14:03:49.277032+0000","last_undegraded":"2024-09-21T14:03:49.277032+0000","last_fullsized":"2024-09-21T14:03:49.277032+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:35.409237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:35.409237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:18.234150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":637,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.257044+0000","last_change":"2024-09-21T14:02:57.225650+0000","last_active":"2024-09-21T14:03:40.257044+0000","last_peered":"2024-09-21T14:03:40.257044+0000","last_clean":"2024-09-21T14:03:40.257044+0000","last_became_active":"2024-09-21T14:01:41.199671+0000","last_became_peered":"2024-09-21T14:01:41.199671+0000","last_unstale":"2024-09-21T14:03:40.257044+0000","last_undegraded":"2024-09-21T14:03:40.257044+0000","last_fullsized":"2024-09-21T14:03:40.257044+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:57.225602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:02:57.225602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:26.897407+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"290'15","reported_seq":791,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-21T14:03:49.277173+0000","last_change":"2024-09-21T14:02:56.315766+0000","last_active":"2024-09-21T14:03:49.277173+0000","last_peered":"2024-09-21T14:03:49.277173+0000","last_clean":"2024-09-21T14:03:49.277173+0000","last_became_active":"2024-09-21T13:58:17.764120+0000","last_became_peered":"2024-09-21T13:58:17.764120+0000","last_unstale":"2024-09-21T14:03:49.277173+0000","last_undegraded":"2024-09-21T14:03:49.277173+0000","last_fullsized":"2024-09-21T14:03:49.277173+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'15","last_scrub_stamp":"2024-09-21T14:02:56.315702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:56.315702+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:11.328147+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":668,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.125149+0000","last_change":"2024-09-21T13:59:32.223082+0000","last_active":"2024-09-21T14:03:40.125149+0000","last_peered":"2024-09-21T14:03:40.125149+0000","last_clean":"2024-09-21T14:03:40.125149+0000","last_became_active":"2024-09-21T13:59:32.221854+0000","last_became_peered":"2024-09-21T13:59:32.221854+0000","last_unstale":"2024-09-21T14:03:40.125149+0000","last_undegraded":"2024-09-21T14:03:40.125149+0000","last_fullsized":"2024-09-21T14:03:40.125149+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:17.691193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:17.691193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":612,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.260324+0000","last_change":"2024-09-21T14:03:32.262147+0000","last_active":"2024-09-21T14:03:40.260324+0000","last_peered":"2024-09-21T14:03:40.260324+0000","last_clean":"2024-09-21T14:03:40.260324+0000","last_became_active":"2024-09-21T13:57:12.703441+0000","last_became_peered":"2024-09-21T13:57:12.703441+0000","last_unstale":"2024-09-21T14:03:40.260324+0000","last_undegraded":"2024-09-21T14:03:40.260324+0000","last_fullsized":"2024-09-21T14:03:40.260324+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:32.262067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:32.262067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:00.749089+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":534,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.116114+0000","last_change":"2024-09-21T14:03:29.345499+0000","last_active":"2024-09-21T14:03:40.116114+0000","last_peered":"2024-09-21T14:03:40.116114+0000","last_clean":"2024-09-21T14:03:40.116114+0000","last_became_active":"2024-09-21T14:01:41.186934+0000","last_became_peered":"2024-09-21T14:01:41.186934+0000","last_unstale":"2024-09-21T14:03:40.116114+0000","last_undegraded":"2024-09-21T14:03:40.116114+0000","last_fullsized":"2024-09-21T14:03:40.116114+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:29.345427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:29.345427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:45.842768+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"290'19","reported_seq":748,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.258154+0000","last_change":"2024-09-21T13:59:32.277742+0000","last_active":"2024-09-21T14:03:40.258154+0000","last_peered":"2024-09-21T14:03:40.258154+0000","last_clean":"2024-09-21T14:03:40.258154+0000","last_became_active":"2024-09-21T13:59:32.231567+0000","last_became_peered":"2024-09-21T13:59:32.231567+0000","last_unstale":"2024-09-21T14:03:40.258154+0000","last_undegraded":"2024-09-21T14:03:40.258154+0000","last_fullsized":"2024-09-21T14:03:40.258154+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:54:22.978267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:22.978267+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:20.135882+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":636,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.117930+0000","last_change":"2024-09-21T14:03:01.448170+0000","last_active":"2024-09-21T14:03:40.117930+0000","last_peered":"2024-09-21T14:03:40.117930+0000","last_clean":"2024-09-21T14:03:40.117930+0000","last_became_active":"2024-09-21T14:01:41.190760+0000","last_became_peered":"2024-09-21T14:01:41.190760+0000","last_unstale":"2024-09-21T14:03:40.117930+0000","last_undegraded":"2024-09-21T14:03:40.117930+0000","last_fullsized":"2024-09-21T14:03:40.117930+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:01.448109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:03:01.448109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:28.748585+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":674,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.125718+0000","last_change":"2024-09-21T13:59:32.219508+0000","last_active":"2024-09-21T14:03:40.125718+0000","last_peered":"2024-09-21T14:03:40.125718+0000","last_clean":"2024-09-21T14:03:40.125718+0000","last_became_active":"2024-09-21T13:59:32.214004+0000","last_became_peered":"2024-09-21T13:59:32.214004+0000","last_unstale":"2024-09-21T14:03:40.125718+0000","last_undegraded":"2024-09-21T14:03:40.125718+0000","last_fullsized":"2024-09-21T14:03:40.125718+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:36.525417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:36.525417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:39.980347+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":670,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.260522+0000","last_change":"2024-09-21T13:59:28.124373+0000","last_active":"2024-09-21T14:03:40.260522+0000","last_peered":"2024-09-21T14:03:40.260522+0000","last_clean":"2024-09-21T14:03:40.260522+0000","last_became_active":"2024-09-21T13:58:17.778115+0000","last_became_peered":"2024-09-21T13:58:17.778115+0000","last_unstale":"2024-09-21T14:03:40.260522+0000","last_undegraded":"2024-09-21T14:03:40.260522+0000","last_fullsized":"2024-09-21T14:03:40.260522+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:28.124296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:28.124296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"290'8","reported_seq":694,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.258396+0000","last_change":"2024-09-21T13:59:26.034457+0000","last_active":"2024-09-21T14:03:40.258396+0000","last_peered":"2024-09-21T14:03:40.258396+0000","last_clean":"2024-09-21T14:03:40.258396+0000","last_became_active":"2024-09-21T13:58:17.775079+0000","last_became_peered":"2024-09-21T13:58:17.775079+0000","last_unstale":"2024-09-21T14:03:40.258396+0000","last_undegraded":"2024-09-21T14:03:40.258396+0000","last_fullsized":"2024-09-21T14:03:40.258396+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'8","last_scrub_stamp":"2024-09-21T13:59:26.034372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:26.034372+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":596,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.116641+0000","last_change":"2024-09-21T14:01:41.196335+0000","last_active":"2024-09-21T14:03:40.116641+0000","last_peered":"2024-09-21T14:03:40.116641+0000","last_clean":"2024-09-21T14:03:40.116641+0000","last_became_active":"2024-09-21T14:01:41.195989+0000","last_became_peered":"2024-09-21T14:01:41.195989+0000","last_unstale":"2024-09-21T14:03:40.116641+0000","last_undegraded":"2024-09-21T14:03:40.116641+0000","last_fullsized":"2024-09-21T14:03:40.116641+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:16.457445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:16.457445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:20.288594+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":297,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.116821+0000","last_change":"2024-09-21T14:03:30.728296+0000","last_active":"2024-09-21T14:03:40.116821+0000","last_peered":"2024-09-21T14:03:40.116821+0000","last_clean":"2024-09-21T14:03:40.116821+0000","last_became_active":"2024-09-21T13:59:32.204380+0000","last_became_peered":"2024-09-21T13:59:32.204380+0000","last_unstale":"2024-09-21T14:03:40.116821+0000","last_undegraded":"2024-09-21T14:03:40.116821+0000","last_fullsized":"2024-09-21T14:03:40.116821+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:30.727972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:30.727972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:43.035198+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":607,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.126775+0000","last_change":"2024-09-21T13:59:32.212615+0000","last_active":"2024-09-21T14:03:40.126775+0000","last_peered":"2024-09-21T14:03:40.126775+0000","last_clean":"2024-09-21T14:03:40.126775+0000","last_became_active":"2024-09-21T13:59:32.210134+0000","last_became_peered":"2024-09-21T13:59:32.210134+0000","last_unstale":"2024-09-21T14:03:40.126775+0000","last_undegraded":"2024-09-21T14:03:40.126775+0000","last_fullsized":"2024-09-21T14:03:40.126775+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:08.852789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:55:08.852789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:10.940052+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":723,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-09-21T14:03:49.275918+0000","last_change":"2024-09-21T14:01:41.187792+0000","last_active":"2024-09-21T14:03:49.275918+0000","last_peered":"2024-09-21T14:03:49.275918+0000","last_clean":"2024-09-21T14:03:49.275918+0000","last_became_active":"2024-09-21T14:01:41.187438+0000","last_became_peered":"2024-09-21T14:01:41.187438+0000","last_unstale":"2024-09-21T14:03:49.275918+0000","last_undegraded":"2024-09-21T14:03:49.275918+0000","last_fullsized":"2024-09-21T14:03:49.275918+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:17.706325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:17.706325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:45.351033+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":650,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.258499+0000","last_change":"2024-09-21T13:58:13.776632+0000","last_active":"2024-09-21T14:03:40.258499+0000","last_peered":"2024-09-21T14:03:40.258499+0000","last_clean":"2024-09-21T14:03:40.258499+0000","last_became_active":"2024-09-21T13:57:12.704042+0000","last_became_peered":"2024-09-21T13:57:12.704042+0000","last_unstale":"2024-09-21T14:03:40.258499+0000","last_undegraded":"2024-09-21T14:03:40.258499+0000","last_fullsized":"2024-09-21T14:03:40.258499+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:13.776572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:13.776572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"290'19","reported_seq":329,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.117679+0000","last_change":"2024-09-21T14:03:31.762681+0000","last_active":"2024-09-21T14:03:40.117679+0000","last_peered":"2024-09-21T14:03:40.117679+0000","last_clean":"2024-09-21T14:03:40.117679+0000","last_became_active":"2024-09-21T14:01:41.194606+0000","last_became_peered":"2024-09-21T14:01:41.194606+0000","last_unstale":"2024-09-21T14:03:40.117679+0000","last_undegraded":"2024-09-21T14:03:40.117679+0000","last_fullsized":"2024-09-21T14:03:40.117679+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'19","last_scrub_stamp":"2024-09-21T14:03:31.762605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:03:31.762605+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:58.699819+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":577,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-09-21T14:03:40.126043+0000","last_change":"2024-09-21T13:59:32.203045+0000","last_active":"2024-09-21T14:03:40.126043+0000","last_peered":"2024-09-21T14:03:40.126043+0000","last_clean":"2024-09-21T14:03:40.126043+0000","last_became_active":"2024-09-21T13:59:32.202225+0000","last_became_peered":"2024-09-21T13:59:32.202225+0000","last_unstale":"2024-09-21T14:03:40.126043+0000","last_undegraded":"2024-09-21T14:03:40.126043+0000","last_fullsized":"2024-09-21T14:03:40.126043+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:40.650832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:40.650832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":4964417536,"num_objects":75751,"num_object_clones":0,"num_object_copies":227253,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":75751,"num_write_kb":4848064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14893711360,"data_stored":14893711360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74751,"ondisk_log_size":74751,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1111,"num_read_kb":868,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":175,"num_bytes_recovered":2024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":687,"ondisk_log_size":687,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":260,"num_read_kb":255,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":375688,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":139,"num_write":73,"num_write_kb":1036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1031048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":298,"seq":1279900254260,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1517356,"kb_used_data":1169052,"kb_used_omap":4,"kb_used_meta":348283,"kb_avail":92227796,"statfs":{"total":95995035648,"available":94441263104,"internally_reserved":0,"allocated":1197109248,"data_stored":1196172110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":356642472},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463100,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2496344,"kb_used_data":2122248,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":91248808,"statfs":{"total":95995035648,"available":93438779392,"internally_reserved":0,"allocated":2173181952,"data_stored":2172264209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":383056326},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,1,5,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":345,"seq":1481763717146,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1592224,"kb_used_data":1447316,"kb_used_omap":4,"kb_used_meta":144891,"kb_avail":92152928,"statfs":{"total":95995035648,"available":94364598272,"internally_reserved":0,"allocated":1482051584,"data_stored":1481123123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":148369059},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920969,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2847992,"kb_used_data":2429480,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":90897160,"statfs":{"total":95995035648,"available":93078691840,"internally_reserved":0,"allocated":2487787520,"data_stored":2486837627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":428538309},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":240,"seq":1030792151120,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1804288,"kb_used_data":1459632,"kb_used_omap":4,"kb_used_meta":344635,"kb_avail":91940864,"statfs":{"total":95995035648,"available":94147444736,"internally_reserved":0,"allocated":1494663168,"data_stored":1493706457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4443,"internal_metadata":352906917},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":130,"seq":558345748622,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2357660,"kb_used_data":1779852,"kb_used_omap":4,"kb_used_meta":577787,"kb_avail":91387492,"statfs":{"total":95995035648,"available":93580791808,"internally_reserved":0,"allocated":1822568448,"data_stored":1821647871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4441,"internal_metadata":591654567},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575067,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1543780,"kb_used_data":935056,"kb_used_omap":1,"kb_used_meta":608702,"kb_avail":92201372,"statfs":{"total":95995035648,"available":94414204928,"internally_reserved":0,"allocated":957497344,"data_stored":956515263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":623311301},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738593,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3672492,"kb_used_data":3081820,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":90072660,"statfs":{"total":95995035648,"available":92234403840,"internally_reserved":0,"allocated":3155783680,"data_stored":3154845815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":604830149},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3177971712,"data_stored":3177971712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":961216512,"data_stored":961216512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1849491456,"data_stored":1849491456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1517158400,"data_stored":1517158400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2508849152,"data_stored":2508849152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1488715776,"data_stored":1488715776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2192179200,"data_stored":2192179200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1198129152,"data_stored":1198129152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T14:03:52.352 INFO:tasks.thrashosds.thrasher:Setting 6.7 to [3, 7, 6, 1, 0, 2] 2024-09-21T14:03:52.353 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '6.7', '3', '7', '6', '1', '0', '2'] 2024-09-21T14:03:52.353 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd pg-upmap-items 6.7 3 7 6 1 0 2 2024-09-21T14:03:52.353 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:03:52.398 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:52.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:52.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:03:52.500 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:03:52.557 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:52.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:52.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:03:52.642 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:03:52.642 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:03:52.659 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:03:52.673 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:03:52.675 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:03:52.675 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:03:52.719 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:03:52.720 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:03:52.723 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:52.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:52.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:03:52.758 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:03:52.758 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:03:52.793 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:03:52.793 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:03:52.826 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:52.831 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:03:52.831 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:03:52.865 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:03:52.866 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:03:52.867 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:52.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:52.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:52.876 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 47 16 81861 81845 108.812 73.5 0.00870139 0.00918459 2024-09-21T14:03:52.902 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:03:52.903 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:03:52.969 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:03:53.007 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:53.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:53.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:03:53.109 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:03:53.150 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:53.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:53.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:03:53.252 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:03:53.290 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:53.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:53.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:03:53.393 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:53.426 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:53.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:53.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:53.528 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:03:53.548 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T14:03:53.548 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:03:53.561 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:53 smithi049 ceph-mon[40489]: pgmap v761: 158 pgs: 158 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 60 MiB/s wr, 961 op/s 2024-09-21T14:03:53.561 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:53 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2391518287' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:03:53.561 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:53 smithi049 ceph-mon[30080]: pgmap v761: 158 pgs: 158 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 60 MiB/s wr, 961 op/s 2024-09-21T14:03:53.562 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:53 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2391518287' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:03:53.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:53.586 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:53.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:03:53.629 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T14:03:53.687 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:03:53.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:53 smithi187 ceph-mon[33032]: pgmap v761: 158 pgs: 158 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 60 MiB/s wr, 961 op/s 2024-09-21T14:03:53.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:53 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2391518287' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:03:53.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:53.736 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:53.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:03:53.837 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:03:53.876 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:53.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:53.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:03:53.879 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 48 16 83109 83093 108.165 78 0.0116876 0.00924012 2024-09-21T14:03:53.979 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:03:54.010 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:54.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:54.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:03:54.112 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:03:54.151 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:54.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:54.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:03:54.253 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:03:54.290 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:54.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:54.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:03:54.392 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:03:54.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:54.432 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:54.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:03:54.533 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:03:54.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:54.565 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:54.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:03:54.666 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:03:54.694 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:54.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:54.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:03:54.796 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:03:54.819 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:54.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:54.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:03:54.829 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:03:54.860 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T14:03:54.869 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:03:54.878 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 49 16 84142 84126 107.275 64.5625 0.00304456 0.00928401 2024-09-21T14:03:54.921 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:03:54.943 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:54.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:54.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:03:55.045 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:55.066 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:55.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:55.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:55.169 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:03:55.201 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:55.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:55.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:03:55.303 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:03:55.343 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:55.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:55.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:03:55.400 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:55 smithi049 ceph-mon[40489]: pgmap v762: 158 pgs: 158 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 88 MiB/s wr, 1.41k op/s 2024-09-21T14:03:55.400 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:55 smithi049 ceph-mon[30080]: pgmap v762: 158 pgs: 158 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 88 MiB/s wr, 1.41k op/s 2024-09-21T14:03:55.445 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:03:55.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:55.482 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:55.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:03:55.583 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:03:55.620 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:55.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:55.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:03:55.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:55 smithi187 ceph-mon[33032]: pgmap v762: 158 pgs: 158 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 88 MiB/s wr, 1.41k op/s 2024-09-21T14:03:55.723 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:03:55.751 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:55.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:55.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:03:55.853 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:03:55.878 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 50 16 85052 85036 106.267 56.875 0.0100055 0.00940446 2024-09-21T14:03:55.896 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:55.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:55.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:03:55.998 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:03:56.032 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:56.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:56.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:03:56.134 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:03:56.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:56.172 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:56.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:03:56.273 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:03:56.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:56.314 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:56.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:03:56.415 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:03:56.451 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:56.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:56.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:03:56.553 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:03:56.570 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:56 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/397887908' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "6.7", "id": [3, 7, 6, 1, 0, 2]} : dispatch 2024-09-21T14:03:56.570 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:56 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "6.7", "id": [3, 7, 6, 1, 0, 2]} : dispatch 2024-09-21T14:03:56.571 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:56 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/397887908' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "6.7", "id": [3, 7, 6, 1, 0, 2]} : dispatch 2024-09-21T14:03:56.571 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:56 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "6.7", "id": [3, 7, 6, 1, 0, 2]} : dispatch 2024-09-21T14:03:56.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:56.591 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:56.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:03:56.620 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T14:03:56.692 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:56.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:56 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/397887908' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "6.7", "id": [3, 7, 6, 1, 0, 2]} : dispatch 2024-09-21T14:03:56.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:56 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "6.7", "id": [3, 7, 6, 1, 0, 2]} : dispatch 2024-09-21T14:03:56.724 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:56.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:56.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:56.826 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:03:56.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:56.879 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:56.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:03:56.880 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 51 16 86178 86162 105.562 70.375 0.0230889 0.00946783 2024-09-21T14:03:56.981 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:03:57.007 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:57.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:57.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:03:57.109 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:03:57.151 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:57.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:57.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:03:57.253 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:03:57.297 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:57.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:57.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:03:57.398 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:03:57.436 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:57.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:57.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:03:57.538 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:03:57.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:57.577 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:57.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:03:57.677 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:03:57.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:57 smithi187 ceph-mon[33032]: pgmap v763: 158 pgs: 158 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 85 MiB/s wr, 1.36k op/s 2024-09-21T14:03:57.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:57 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.7", "id": [3, 7, 6, 1, 0, 2]}]': finished 2024-09-21T14:03:57.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:57 smithi187 ceph-mon[33032]: osdmap e401: 8 total, 8 up, 8 in 2024-09-21T14:03:57.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:57.719 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:57.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:03:57.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:57 smithi049 ceph-mon[40489]: pgmap v763: 158 pgs: 158 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 85 MiB/s wr, 1.36k op/s 2024-09-21T14:03:57.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:57 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.7", "id": [3, 7, 6, 1, 0, 2]}]': finished 2024-09-21T14:03:57.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:57 smithi049 ceph-mon[40489]: osdmap e401: 8 total, 8 up, 8 in 2024-09-21T14:03:57.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:57 smithi049 ceph-mon[30080]: pgmap v763: 158 pgs: 158 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 85 MiB/s wr, 1.36k op/s 2024-09-21T14:03:57.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:57 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.7", "id": [3, 7, 6, 1, 0, 2]}]': finished 2024-09-21T14:03:57.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:57 smithi049 ceph-mon[30080]: osdmap e401: 8 total, 8 up, 8 in 2024-09-21T14:03:57.820 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:03:57.841 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:57.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:57.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:03:57.879 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 52 16 87388 87372 104.987 75.625 0.0049122 0.00951855 2024-09-21T14:03:57.887 INFO:teuthology.orchestra.run.smithi049.stderr:set 6.7 pg_upmap_items mapping to [3->7,6->1,0->2] 2024-09-21T14:03:57.943 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:03:57.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:57.983 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:57.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:03:58.084 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:03:58.126 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:58.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:58.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:03:58.228 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:03:58.262 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:58.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:58.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:03:58.364 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:03:58.408 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:58.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:58.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:03:58.511 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:03:58.554 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:58.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:58.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:03:58.655 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:58 smithi049 ceph-mon[30080]: osdmap e402: 8 total, 8 up, 8 in 2024-09-21T14:03:58.655 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:58 smithi049 ceph-mon[40489]: osdmap e402: 8 total, 8 up, 8 in 2024-09-21T14:03:58.656 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:03:58.686 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:58.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:58.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:03:58.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:58 smithi187 ceph-mon[33032]: osdmap e402: 8 total, 8 up, 8 in 2024-09-21T14:03:58.788 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:03:58.818 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:58.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:58.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:03:58.879 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 53 16 88623 88607 104.462 77.1875 0.0149183 0.00956774 2024-09-21T14:03:58.920 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:03:58.960 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:58.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:58.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:03:59.062 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:03:59.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:59.102 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:59.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:03:59.107 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:03:59.203 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:03:59.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:59.249 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:59.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:03:59.350 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:03:59.359 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-09-21T14:03:59.383 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:59.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:59.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:03:59.485 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:03:59.521 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:59.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:59.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:03:59.623 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:03:59.655 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:03:59.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:59.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:03:59.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:59 smithi187 ceph-mon[33032]: pgmap v766: 158 pgs: 158 active+clean; 5.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.3 KiB/s rd, 67 MiB/s wr, 1.07k op/s 2024-09-21T14:03:59.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:59 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2382973817' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:03:59.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:59 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:03:59.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:03:59 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:03:59.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:59 smithi049 ceph-mon[40489]: pgmap v766: 158 pgs: 158 active+clean; 5.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.3 KiB/s rd, 67 MiB/s wr, 1.07k op/s 2024-09-21T14:03:59.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:59 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2382973817' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:03:59.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:59 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:03:59.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:03:59 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:03:59.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:59 smithi049 ceph-mon[30080]: pgmap v766: 158 pgs: 158 active+clean; 5.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.3 KiB/s rd, 67 MiB/s wr, 1.07k op/s 2024-09-21T14:03:59.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:59 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2382973817' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:03:59.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:59 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:03:59.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:03:59 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:03:59.757 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:03:59.802 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:59.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:59.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:03:59.879 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 54 15 89859 89844 103.959 77.3125 0.00553683 0.00961435 2024-09-21T14:03:59.904 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:03:59.938 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:03:59.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:03:59.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:04:00.040 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:04:00.076 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:00.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:00.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:04:00.178 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:04:00.194 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:03:59 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:04:00.215 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:00.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:00.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:04:00.317 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:04:00.350 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:00.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:00.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:04:00.451 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:00 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:04:00.452 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:00 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:04:00.452 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:00 smithi049 ceph-mon[40489]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:04:00.452 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:00 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T14:04:00.452 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:00 smithi049 ceph-mon[40489]: osdmap e403: 8 total, 8 up, 8 in 2024-09-21T14:04:00.452 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:00 smithi049 ceph-mon[40489]: 1.a scrub starts 2024-09-21T14:04:00.452 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:00 smithi049 ceph-mon[40489]: 1.a scrub ok 2024-09-21T14:04:00.452 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:00 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:04:00.452 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:00 smithi049 ceph-mon[40489]: osdmap e404: 8 total, 8 up, 8 in 2024-09-21T14:04:00.452 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:00 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:04:00.452 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:00 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:04:00.452 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:00 smithi049 ceph-mon[30080]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:04:00.453 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:00 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T14:04:00.453 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:00 smithi049 ceph-mon[30080]: osdmap e403: 8 total, 8 up, 8 in 2024-09-21T14:04:00.453 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:00 smithi049 ceph-mon[30080]: 1.a scrub starts 2024-09-21T14:04:00.453 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:00 smithi049 ceph-mon[30080]: 1.a scrub ok 2024-09-21T14:04:00.453 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:00 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:04:00.453 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:00 smithi049 ceph-mon[30080]: osdmap e404: 8 total, 8 up, 8 in 2024-09-21T14:04:00.453 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:04:00.484 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:00.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:00.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:04:00.493 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:04:00.522 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T14:04:00.522 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:04:00.587 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:00.633 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:00.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:00.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:00.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:00 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:04:00.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:00 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:04:00.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:00 smithi187 ceph-mon[33032]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:04:00.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:00 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T14:04:00.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:00 smithi187 ceph-mon[33032]: osdmap e403: 8 total, 8 up, 8 in 2024-09-21T14:04:00.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:00 smithi187 ceph-mon[33032]: 1.a scrub starts 2024-09-21T14:04:00.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:00 smithi187 ceph-mon[33032]: 1.a scrub ok 2024-09-21T14:04:00.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:00 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:04:00.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:00 smithi187 ceph-mon[33032]: osdmap e404: 8 total, 8 up, 8 in 2024-09-21T14:04:00.735 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:04:00.754 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:04:00 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:04:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:04:00.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:00.780 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:00.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:04:00.879 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 55 16 91120 91104 103.501 78.75 0.00722475 0.00965697 2024-09-21T14:04:00.881 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:04:00.921 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:00.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:00.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:04:01.023 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:04:01.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:01.066 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:01.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:04:01.167 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:04:01.211 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:01.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:01.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:04:01.313 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:04:01.347 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:01.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:01.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:04:01.448 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:04:01.469 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:01.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:01.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:04:01.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:01 smithi049 ceph-mon[40489]: pgmap v768: 158 pgs: 158 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 62 MiB/s wr, 991 op/s 2024-09-21T14:04:01.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:01 smithi049 ceph-mon[40489]: 6.1b scrub starts 2024-09-21T14:04:01.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:01 smithi049 ceph-mon[40489]: 6.1b scrub ok 2024-09-21T14:04:01.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:01 smithi049 ceph-mon[40489]: 3.c scrub starts 2024-09-21T14:04:01.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:01 smithi049 ceph-mon[40489]: 3.c scrub ok 2024-09-21T14:04:01.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:01 smithi049 ceph-mon[30080]: pgmap v768: 158 pgs: 158 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 62 MiB/s wr, 991 op/s 2024-09-21T14:04:01.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:01 smithi049 ceph-mon[30080]: 6.1b scrub starts 2024-09-21T14:04:01.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:01 smithi049 ceph-mon[30080]: 6.1b scrub ok 2024-09-21T14:04:01.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:01 smithi049 ceph-mon[30080]: 3.c scrub starts 2024-09-21T14:04:01.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:01 smithi049 ceph-mon[30080]: 3.c scrub ok 2024-09-21T14:04:01.571 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:04:01.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:01.602 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:01.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:04:01.607 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T14:04:01.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:01 smithi187 ceph-mon[33032]: pgmap v768: 158 pgs: 158 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 62 MiB/s wr, 991 op/s 2024-09-21T14:04:01.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:01 smithi187 ceph-mon[33032]: 6.1b scrub starts 2024-09-21T14:04:01.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:01 smithi187 ceph-mon[33032]: 6.1b scrub ok 2024-09-21T14:04:01.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:01 smithi187 ceph-mon[33032]: 3.c scrub starts 2024-09-21T14:04:01.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:01 smithi187 ceph-mon[33032]: 3.c scrub ok 2024-09-21T14:04:01.703 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:01.744 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:01.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:01.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:01.846 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:04:01.878 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:04:01.878 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:04:01.878 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:01.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:01.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:04:01.880 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 56 16 92205 92189 102.864 67.8125 0.0121567 0.00971627 2024-09-21T14:04:01.914 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:04:01.914 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:04:01.951 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:04:01.951 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:04:01.981 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:01.990 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:04:01.991 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:04:02.027 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:04:02.027 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:04:02.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:02.043 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:02.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:02.061 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:04:02.062 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:04:02.095 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:04:02.095 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:04:02.097 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:04:02.136 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:04:02.136 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:04:02.144 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:04:02.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:02.199 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:02.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:04:02.248 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T14:04:02.300 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:04:02.328 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:02.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:02.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:04:02.429 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:02.472 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:02.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:02.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:02.573 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:02.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:02.617 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:02.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:02.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:02 smithi187 ceph-mon[33032]: 3.8 scrub starts 2024-09-21T14:04:02.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:02 smithi187 ceph-mon[33032]: 3.8 scrub ok 2024-09-21T14:04:02.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:02 smithi187 ceph-mon[33032]: 6.7 scrub starts 2024-09-21T14:04:02.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:02 smithi187 ceph-mon[33032]: 6.7 scrub ok 2024-09-21T14:04:02.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:02 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2091957074' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:04:02.718 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:04:02.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:02 smithi049 ceph-mon[40489]: 3.8 scrub starts 2024-09-21T14:04:02.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:02 smithi049 ceph-mon[40489]: 3.8 scrub ok 2024-09-21T14:04:02.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:02 smithi049 ceph-mon[40489]: 6.7 scrub starts 2024-09-21T14:04:02.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:02 smithi049 ceph-mon[40489]: 6.7 scrub ok 2024-09-21T14:04:02.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:02 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2091957074' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:04:02.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:02 smithi049 ceph-mon[30080]: 3.8 scrub starts 2024-09-21T14:04:02.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:02 smithi049 ceph-mon[30080]: 3.8 scrub ok 2024-09-21T14:04:02.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:02 smithi049 ceph-mon[30080]: 6.7 scrub starts 2024-09-21T14:04:02.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:02 smithi049 ceph-mon[30080]: 6.7 scrub ok 2024-09-21T14:04:02.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:02 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2091957074' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:04:02.759 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:02.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:02.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:04:02.861 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:04:02.879 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 57 16 93361 93345 102.326 72.25 0.00846768 0.00976775 2024-09-21T14:04:02.898 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:02.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:02.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:04:02.999 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:04:03.064 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:03.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:03.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:04:03.166 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:04:03.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:03.209 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:03.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:04:03.310 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:04:03.363 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:03.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:03.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:04:03.465 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:04:03.513 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:03.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:03.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:04:03.591 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:03 smithi049 ceph-mon[30080]: 4.1c scrub starts 2024-09-21T14:04:03.591 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:03 smithi049 ceph-mon[30080]: 4.1c scrub ok 2024-09-21T14:04:03.591 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:03 smithi049 ceph-mon[30080]: pgmap v770: 158 pgs: 158 active+clean; 5.4 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-09-21T14:04:03.592 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:03 smithi049 ceph-mon[30080]: 3.7 scrub starts 2024-09-21T14:04:03.592 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:03 smithi049 ceph-mon[30080]: 3.7 scrub ok 2024-09-21T14:04:03.592 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:03 smithi049 ceph-mon[30080]: 5.2 scrub starts 2024-09-21T14:04:03.592 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:03 smithi049 ceph-mon[30080]: 5.2 scrub ok 2024-09-21T14:04:03.592 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:03 smithi049 ceph-mon[40489]: 4.1c scrub starts 2024-09-21T14:04:03.592 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:03 smithi049 ceph-mon[40489]: 4.1c scrub ok 2024-09-21T14:04:03.592 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:03 smithi049 ceph-mon[40489]: pgmap v770: 158 pgs: 158 active+clean; 5.4 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-09-21T14:04:03.592 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:03 smithi049 ceph-mon[40489]: 3.7 scrub starts 2024-09-21T14:04:03.592 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:03 smithi049 ceph-mon[40489]: 3.7 scrub ok 2024-09-21T14:04:03.592 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:03 smithi049 ceph-mon[40489]: 5.2 scrub starts 2024-09-21T14:04:03.592 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:03 smithi049 ceph-mon[40489]: 5.2 scrub ok 2024-09-21T14:04:03.615 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:04:03.649 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T14:04:03.649 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-09-21T14:04:03.649 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-09-21T14:04:03.649 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-09-21T14:04:03.650 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T14:04:03.665 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:03.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:03.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:04:03.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:03 smithi187 ceph-mon[33032]: 4.1c scrub starts 2024-09-21T14:04:03.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:03 smithi187 ceph-mon[33032]: 4.1c scrub ok 2024-09-21T14:04:03.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:03 smithi187 ceph-mon[33032]: pgmap v770: 158 pgs: 158 active+clean; 5.4 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-09-21T14:04:03.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:03 smithi187 ceph-mon[33032]: 3.7 scrub starts 2024-09-21T14:04:03.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:03 smithi187 ceph-mon[33032]: 3.7 scrub ok 2024-09-21T14:04:03.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:03 smithi187 ceph-mon[33032]: 5.2 scrub starts 2024-09-21T14:04:03.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:03 smithi187 ceph-mon[33032]: 5.2 scrub ok 2024-09-21T14:04:03.766 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:04:03.805 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:03.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:03.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:04:03.879 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 58 16 94573 94557 101.868 75.75 0.0150668 0.009812 2024-09-21T14:04:03.907 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:04:03.947 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:03.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:03.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:04:04.048 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:04:04.086 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:04.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:04.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:04:04.150 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 4, 6, 2, 3, 7, 5] 2024-09-21T14:04:04.150 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T14:04:04.150 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-09-21T14:04:04.151 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd test-reweight-by-utilization 1000 0.05 2024-09-21T14:04:04.188 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:04.243 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:04.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:04.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:04.344 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:04:04.380 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:04.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:04.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:04:04.481 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:04.519 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:04:04.546 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:04.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:04.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:04.578 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T14:04:04.647 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:04:04.662 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:04 smithi049 ceph-mon[40489]: 4.f scrub starts 2024-09-21T14:04:04.662 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:04 smithi049 ceph-mon[40489]: 4.f scrub ok 2024-09-21T14:04:04.662 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:04 smithi049 ceph-mon[40489]: 3.11 scrub starts 2024-09-21T14:04:04.663 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:04 smithi049 ceph-mon[40489]: 3.11 scrub ok 2024-09-21T14:04:04.663 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:04 smithi049 ceph-mon[30080]: 4.f scrub starts 2024-09-21T14:04:04.663 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:04 smithi049 ceph-mon[30080]: 4.f scrub ok 2024-09-21T14:04:04.663 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:04 smithi049 ceph-mon[30080]: 3.11 scrub starts 2024-09-21T14:04:04.663 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:04 smithi049 ceph-mon[30080]: 3.11 scrub ok 2024-09-21T14:04:04.686 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:04.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:04.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:04:04.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:04 smithi187 ceph-mon[33032]: 4.f scrub starts 2024-09-21T14:04:04.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:04 smithi187 ceph-mon[33032]: 4.f scrub ok 2024-09-21T14:04:04.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:04 smithi187 ceph-mon[33032]: 3.11 scrub starts 2024-09-21T14:04:04.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:04 smithi187 ceph-mon[33032]: 3.11 scrub ok 2024-09-21T14:04:04.787 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:04:04.821 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:04.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:04.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:04:04.879 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 59 16 95682 95666 101.316 69.3125 0.00763515 0.00986413 2024-09-21T14:04:04.922 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:04:04.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:04.983 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:04.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:04:05.045 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T14:04:05.083 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:04:05.120 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:05.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:05.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:04:05.222 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:04:05.229 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:04:05.254 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:05.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:05.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:04:05.356 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:04:05.389 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:05.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:05.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:04:05.491 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:04:05.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:05 smithi049 ceph-mon[30080]: pgmap v771: 158 pgs: 158 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 94 MiB/s wr, 1.51k op/s 2024-09-21T14:04:05.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:05 smithi049 ceph-mon[30080]: 4.14 scrub starts 2024-09-21T14:04:05.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:05 smithi049 ceph-mon[30080]: 4.14 scrub ok 2024-09-21T14:04:05.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:05 smithi049 ceph-mon[30080]: 10.9 scrub starts 2024-09-21T14:04:05.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:05 smithi049 ceph-mon[30080]: 4.13 scrub starts 2024-09-21T14:04:05.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:05 smithi049 ceph-mon[30080]: 4.13 scrub ok 2024-09-21T14:04:05.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:05 smithi049 ceph-mon[40489]: pgmap v771: 158 pgs: 158 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 94 MiB/s wr, 1.51k op/s 2024-09-21T14:04:05.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:05 smithi049 ceph-mon[40489]: 4.14 scrub starts 2024-09-21T14:04:05.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:05 smithi049 ceph-mon[40489]: 4.14 scrub ok 2024-09-21T14:04:05.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:05 smithi049 ceph-mon[40489]: 10.9 scrub starts 2024-09-21T14:04:05.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:05 smithi049 ceph-mon[40489]: 4.13 scrub starts 2024-09-21T14:04:05.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:05 smithi049 ceph-mon[40489]: 4.13 scrub ok 2024-09-21T14:04:05.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:05.543 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:05.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:04:05.644 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:04:05.690 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:05.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:05.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:04:05.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:05 smithi187 ceph-mon[33032]: pgmap v771: 158 pgs: 158 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 94 MiB/s wr, 1.51k op/s 2024-09-21T14:04:05.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:05 smithi187 ceph-mon[33032]: 4.14 scrub starts 2024-09-21T14:04:05.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:05 smithi187 ceph-mon[33032]: 4.14 scrub ok 2024-09-21T14:04:05.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:05 smithi187 ceph-mon[33032]: 10.9 scrub starts 2024-09-21T14:04:05.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:05 smithi187 ceph-mon[33032]: 4.13 scrub starts 2024-09-21T14:04:05.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:05 smithi187 ceph-mon[33032]: 4.13 scrub ok 2024-09-21T14:04:05.792 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:04:05.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:05.824 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:05.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:04:05.880 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:2024-09-21T14:04:05.881072+0000 min lat: 0.00132701 max lat: 0.461209 avg lat: 0.00993328 2024-09-21T14:04:05.880 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T14:04:05.880 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 60 16 96621 96605 100.606 58.6875 0.0142609 0.00993328 2024-09-21T14:04:05.924 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:04:05.954 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:05.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:05.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:04:05.966 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:04:05.979 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:04:06.057 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:04:06.090 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:06.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:06.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:04:06.192 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:06.248 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:06.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:06.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:06.313 INFO:teuthology.orchestra.run.smithi049.stdout:moved 0 / 474 (0%) 2024-09-21T14:04:06.313 INFO:teuthology.orchestra.run.smithi049.stdout:avg 59.25 2024-09-21T14:04:06.313 INFO:teuthology.orchestra.run.smithi049.stdout:stddev 5.76086 -> 5.76086 (expected baseline 7.20026) 2024-09-21T14:04:06.313 INFO:teuthology.orchestra.run.smithi049.stdout:min osd.2 with 48 -> 48 pgs (0.810127 -> 0.810127 * mean) 2024-09-21T14:04:06.313 INFO:teuthology.orchestra.run.smithi049.stdout:max osd.4 with 68 -> 68 pgs (1.14768 -> 1.14768 * mean) 2024-09-21T14:04:06.314 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T14:04:06.314 INFO:teuthology.orchestra.run.smithi049.stdout:oload 1000 2024-09-21T14:04:06.314 INFO:teuthology.orchestra.run.smithi049.stdout:max_change 0.05 2024-09-21T14:04:06.314 INFO:teuthology.orchestra.run.smithi049.stdout:max_change_osds 4 2024-09-21T14:04:06.314 INFO:teuthology.orchestra.run.smithi049.stdout:average_utilization 0.0287 2024-09-21T14:04:06.314 INFO:teuthology.orchestra.run.smithi049.stdout:overload_utilization 0.2866 2024-09-21T14:04:06.314 INFO:teuthology.orchestra.run.smithi049.stderr:no change 2024-09-21T14:04:06.350 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:04:06.393 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:06.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:06.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:04:06.433 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-09-21T14:04:06.496 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:04:06.538 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:06.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:06.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:04:06.639 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:04:06.677 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:06.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:06.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:04:06.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:06 smithi187 ceph-mon[33032]: 10.8 scrub starts 2024-09-21T14:04:06.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:06 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2913558231' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:04:06.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:06 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:04:06.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:06 smithi049 ceph-mon[40489]: 10.8 scrub starts 2024-09-21T14:04:06.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:06 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2913558231' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:04:06.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:06 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:04:06.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:06 smithi049 ceph-mon[30080]: 10.8 scrub starts 2024-09-21T14:04:06.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:06 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2913558231' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:04:06.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:06 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:04:06.779 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:04:06.816 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:06.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:06.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:04:06.880 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 61 16 97585 97569 99.9444 60.25 0.0360716 0.0099964 2024-09-21T14:04:06.917 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:04:06.939 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:06.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:06.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:04:07.041 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:04:07.081 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:07.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:07.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:04:07.182 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:04:07.219 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:07.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:07.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:04:07.320 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:04:07.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:07.356 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:07.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:04:07.456 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:04:07.500 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:07.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:07.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:04:07.601 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:04:07.641 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:07.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:07.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:04:07.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:07 smithi187 ceph-mon[33032]: pgmap v772: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-09-21T14:04:07.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:07 smithi187 ceph-mon[33032]: 10.b scrub starts 2024-09-21T14:04:07.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:07 smithi187 ceph-mon[33032]: 10.9 scrub ok 2024-09-21T14:04:07.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:07 smithi187 ceph-mon[33032]: from='client.26087 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T14:04:07.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:07 smithi187 ceph-mon[33032]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T14:04:07.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:07 smithi187 ceph-mon[33032]: Cluster is now healthy 2024-09-21T14:04:07.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:07 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T14:04:07.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:07 smithi187 ceph-mon[33032]: osdmap e405: 8 total, 8 up, 8 in 2024-09-21T14:04:07.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:07 smithi187 ceph-mon[33032]: 3.15 scrub starts 2024-09-21T14:04:07.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:07 smithi187 ceph-mon[33032]: 3.15 scrub ok 2024-09-21T14:04:07.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:07 smithi187 ceph-mon[33032]: 10.e scrub starts 2024-09-21T14:04:07.743 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:04:07.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:07 smithi049 ceph-mon[30080]: pgmap v772: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-09-21T14:04:07.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:07 smithi049 ceph-mon[30080]: 10.b scrub starts 2024-09-21T14:04:07.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:07 smithi049 ceph-mon[30080]: 10.9 scrub ok 2024-09-21T14:04:07.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:07 smithi049 ceph-mon[30080]: from='client.26087 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T14:04:07.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:07 smithi049 ceph-mon[30080]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T14:04:07.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:07 smithi049 ceph-mon[30080]: Cluster is now healthy 2024-09-21T14:04:07.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:07 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T14:04:07.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:07 smithi049 ceph-mon[30080]: osdmap e405: 8 total, 8 up, 8 in 2024-09-21T14:04:07.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:07 smithi049 ceph-mon[30080]: 3.15 scrub starts 2024-09-21T14:04:07.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:07 smithi049 ceph-mon[30080]: 3.15 scrub ok 2024-09-21T14:04:07.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:07 smithi049 ceph-mon[30080]: 10.e scrub starts 2024-09-21T14:04:07.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:07 smithi049 ceph-mon[40489]: pgmap v772: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-09-21T14:04:07.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:07 smithi049 ceph-mon[40489]: 10.b scrub starts 2024-09-21T14:04:07.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:07 smithi049 ceph-mon[40489]: 10.9 scrub ok 2024-09-21T14:04:07.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:07 smithi049 ceph-mon[40489]: from='client.26087 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T14:04:07.757 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:07 smithi049 ceph-mon[40489]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T14:04:07.757 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:07 smithi049 ceph-mon[40489]: Cluster is now healthy 2024-09-21T14:04:07.757 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:07 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T14:04:07.757 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:07 smithi049 ceph-mon[40489]: osdmap e405: 8 total, 8 up, 8 in 2024-09-21T14:04:07.757 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:07 smithi049 ceph-mon[40489]: 3.15 scrub starts 2024-09-21T14:04:07.757 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:07 smithi049 ceph-mon[40489]: 3.15 scrub ok 2024-09-21T14:04:07.757 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:07 smithi049 ceph-mon[40489]: 10.e scrub starts 2024-09-21T14:04:07.780 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:07.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:07.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:04:07.880 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 62 16 98664 98648 99.4201 67.4375 0.00804299 0.0100533 2024-09-21T14:04:07.882 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:04:07.934 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:07.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:07.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:04:08.036 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:04:08.048 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T14:04:08.049 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-09-21T14:04:08.049 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-09-21T14:04:08.049 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-09-21T14:04:08.049 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.049 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232022 8.f 8.d7b7a40f (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T14:04:08.049 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.700289+0000", 2024-09-21T14:04:08.049 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 416.34586974400003, 2024-09-21T14:04:08.049 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97395471300000003, 2024-09-21T14:04:08.049 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:08.049 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:08.049 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T14:04:08.050 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:04:08.050 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T14:04:08.050 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:04:08.050 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232022 2024-09-21T14:04:08.050 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.050 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:08.050 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.050 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:08.050 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700289+0000", 2024-09-21T14:04:08.050 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:08.050 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.050 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.050 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:08.051 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700289+0000", 2024-09-21T14:04:08.051 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:08.051 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.051 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.051 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:08.051 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700291+0000", 2024-09-21T14:04:08.051 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5850000000000001e-06 2024-09-21T14:04:08.051 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.051 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.051 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:08.051 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700296+0000", 2024-09-21T14:04:08.051 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9749999999999998e-06 2024-09-21T14:04:08.052 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.052 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.052 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:08.052 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700306+0000", 2024-09-21T14:04:08.052 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9310000000000003e-06 2024-09-21T14:04:08.052 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.052 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.052 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:08.052 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700317+0000", 2024-09-21T14:04:08.052 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1199000000000001e-05 2024-09-21T14:04:08.052 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.052 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.052 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:08.053 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.711626+0000", 2024-09-21T14:04:08.053 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.011309009 2024-09-21T14:04:08.053 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.053 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.053 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:04:08.053 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.711637+0000", 2024-09-21T14:04:08.053 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1182000000000001e-05 2024-09-21T14:04:08.053 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.053 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.053 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:08.053 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674230+0000", 2024-09-21T14:04:08.053 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.962593535 2024-09-21T14:04:08.053 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.054 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.054 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:08.054 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674244+0000", 2024-09-21T14:04:08.054 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3297e-05 2024-09-21T14:04:08.054 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:08.054 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:08.054 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:08.054 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.054 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.054 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232025 8.f 8.4e5a668f (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T14:04:08.054 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.700329+0000", 2024-09-21T14:04:08.054 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 416.34582994599998, 2024-09-21T14:04:08.054 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97395089300000004, 2024-09-21T14:04:08.054 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:08.055 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:08.055 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T14:04:08.055 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:04:08.055 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T14:04:08.055 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:04:08.055 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232025 2024-09-21T14:04:08.055 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.055 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:08.055 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.055 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:08.055 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700329+0000", 2024-09-21T14:04:08.055 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:08.055 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.056 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.056 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:08.056 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700329+0000", 2024-09-21T14:04:08.056 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:08.056 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.056 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.056 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:08.056 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700329+0000", 2024-09-21T14:04:08.056 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7100000000000002e-07 2024-09-21T14:04:08.056 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.056 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.056 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:08.056 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700332+0000", 2024-09-21T14:04:08.056 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6589999999999999e-06 2024-09-21T14:04:08.056 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.057 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.057 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:08.057 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700339+0000", 2024-09-21T14:04:08.057 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8940000000000004e-06 2024-09-21T14:04:08.057 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.057 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.057 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:08.057 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700345+0000", 2024-09-21T14:04:08.057 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6219999999999997e-06 2024-09-21T14:04:08.057 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.057 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.057 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:08.057 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.711662+0000", 2024-09-21T14:04:08.057 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.011317522 2024-09-21T14:04:08.058 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.058 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.058 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:04:08.058 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.711669+0000", 2024-09-21T14:04:08.058 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2570000000000001e-06 2024-09-21T14:04:08.058 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.058 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.058 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:08.058 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674270+0000", 2024-09-21T14:04:08.058 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.96260044300000003 2024-09-21T14:04:08.058 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.058 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.058 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:08.058 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674280+0000", 2024-09-21T14:04:08.059 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0025e-05 2024-09-21T14:04:08.059 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:08.059 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:08.059 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:08.059 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.059 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.059 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232033 8.f 8.d949c02f (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T14:04:08.059 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.700352+0000", 2024-09-21T14:04:08.059 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 416.34580654600001, 2024-09-21T14:04:08.059 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97396006099999999, 2024-09-21T14:04:08.059 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:08.059 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:08.060 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T14:04:08.060 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:04:08.060 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T14:04:08.060 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:04:08.060 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232033 2024-09-21T14:04:08.060 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.060 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:08.060 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.060 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:08.060 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700352+0000", 2024-09-21T14:04:08.060 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:08.060 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.060 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.060 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:08.060 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700352+0000", 2024-09-21T14:04:08.061 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:08.061 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.061 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.061 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:08.061 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700353+0000", 2024-09-21T14:04:08.061 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.89e-07 2024-09-21T14:04:08.061 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.061 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.061 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:08.061 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700355+0000", 2024-09-21T14:04:08.061 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1840000000000002e-06 2024-09-21T14:04:08.061 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.061 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.061 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:08.061 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700364+0000", 2024-09-21T14:04:08.062 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4919999999999999e-06 2024-09-21T14:04:08.062 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.062 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.062 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:08.062 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700372+0000", 2024-09-21T14:04:08.062 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9659999999999996e-06 2024-09-21T14:04:08.062 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.062 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.062 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:08.062 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.711693+0000", 2024-09-21T14:04:08.062 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.011321059 2024-09-21T14:04:08.062 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.062 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.062 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:04:08.063 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.711701+0000", 2024-09-21T14:04:08.063 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3640000000000001e-06 2024-09-21T14:04:08.063 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.063 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.063 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:08.063 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674303+0000", 2024-09-21T14:04:08.063 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.96260253100000004 2024-09-21T14:04:08.063 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.063 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.064 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:08.064 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674312+0000", 2024-09-21T14:04:08.064 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0759999999999992e-06 2024-09-21T14:04:08.064 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:08.064 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:08.064 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:08.064 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.064 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.064 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232051 8.0 8.85c8f300 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T14:04:08.064 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.700380+0000", 2024-09-21T14:04:08.064 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 416.34577884300001, 2024-09-21T14:04:08.064 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97451152799999996, 2024-09-21T14:04:08.064 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:08.064 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:08.064 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T14:04:08.065 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:04:08.065 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T14:04:08.065 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:04:08.065 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232051 2024-09-21T14:04:08.065 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.065 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:08.065 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.065 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:08.065 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700380+0000", 2024-09-21T14:04:08.065 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:08.065 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.065 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.065 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:08.066 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700380+0000", 2024-09-21T14:04:08.066 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:08.066 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.066 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.066 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:08.066 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700380+0000", 2024-09-21T14:04:08.066 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1699999999999999e-07 2024-09-21T14:04:08.066 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.066 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.066 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:08.066 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700383+0000", 2024-09-21T14:04:08.066 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1600000000000001e-06 2024-09-21T14:04:08.066 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.066 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.066 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:08.067 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700389+0000", 2024-09-21T14:04:08.067 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6080000000000001e-06 2024-09-21T14:04:08.067 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.067 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.067 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:08.067 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700394+0000", 2024-09-21T14:04:08.067 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9810000000000003e-06 2024-09-21T14:04:08.067 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.067 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.067 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:08.067 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.715050+0000", 2024-09-21T14:04:08.067 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.014656269 2024-09-21T14:04:08.067 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.067 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.068 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:04:08.068 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.715056+0000", 2024-09-21T14:04:08.068 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0129999999999999e-06 2024-09-21T14:04:08.068 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.068 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.068 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:08.068 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674876+0000", 2024-09-21T14:04:08.068 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.95981962700000001 2024-09-21T14:04:08.068 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.068 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.068 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:08.068 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674892+0000", 2024-09-21T14:04:08.068 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5452999999999999e-05 2024-09-21T14:04:08.068 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:08.068 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:08.068 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:08.069 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.069 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.069 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232059 8.f 8.b757f3bf (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T14:04:08.069 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.889250+0000", 2024-09-21T14:04:08.069 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 416.15690903000001, 2024-09-21T14:04:08.069 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.78509475900000003, 2024-09-21T14:04:08.069 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:08.069 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:08.069 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T14:04:08.069 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:04:08.069 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T14:04:08.069 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:04:08.069 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232059 2024-09-21T14:04:08.069 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.069 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:08.070 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.070 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:08.070 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889250+0000", 2024-09-21T14:04:08.070 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:08.070 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.070 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.070 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:08.070 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889250+0000", 2024-09-21T14:04:08.070 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:08.070 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.070 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.070 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:08.070 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889252+0000", 2024-09-21T14:04:08.070 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9130000000000001e-06 2024-09-21T14:04:08.071 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.071 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.071 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:08.071 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889258+0000", 2024-09-21T14:04:08.071 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7410000000000003e-06 2024-09-21T14:04:08.071 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.071 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.071 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:08.071 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889360+0000", 2024-09-21T14:04:08.071 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000101382 2024-09-21T14:04:08.071 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.071 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.071 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:08.071 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889391+0000", 2024-09-21T14:04:08.071 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1032999999999999e-05 2024-09-21T14:04:08.071 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.072 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.072 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:08.072 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889500+0000", 2024-09-21T14:04:08.072 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000109194 2024-09-21T14:04:08.072 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.072 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.072 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:04:08.072 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889512+0000", 2024-09-21T14:04:08.072 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2368000000000001e-05 2024-09-21T14:04:08.072 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.072 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.072 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:08.072 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674335+0000", 2024-09-21T14:04:08.072 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.78482304000000003 2024-09-21T14:04:08.072 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.073 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.073 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:08.073 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674345+0000", 2024-09-21T14:04:08.073 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0880000000000001e-06 2024-09-21T14:04:08.073 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:08.073 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:08.073 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:08.073 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.073 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.073 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232060 8.b 8.6c14963b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T14:04:08.073 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.889442+0000", 2024-09-21T14:04:08.073 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 416.15671684199998, 2024-09-21T14:04:08.073 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.78518956100000004, 2024-09-21T14:04:08.073 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:08.074 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:08.074 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T14:04:08.074 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:04:08.074 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T14:04:08.074 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:04:08.074 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232060 2024-09-21T14:04:08.074 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.074 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:08.074 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.074 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:08.074 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889442+0000", 2024-09-21T14:04:08.074 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:08.074 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.075 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.075 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:08.075 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889442+0000", 2024-09-21T14:04:08.075 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:08.075 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.075 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.075 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:08.075 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889443+0000", 2024-09-21T14:04:08.075 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9100000000000003e-07 2024-09-21T14:04:08.075 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.075 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.075 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:08.075 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889447+0000", 2024-09-21T14:04:08.075 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4939999999999997e-06 2024-09-21T14:04:08.076 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.076 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.076 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:08.076 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889457+0000", 2024-09-21T14:04:08.076 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0145e-05 2024-09-21T14:04:08.076 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.076 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.076 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:08.076 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889465+0000", 2024-09-21T14:04:08.076 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2749999999999998e-06 2024-09-21T14:04:08.076 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.076 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.076 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:08.076 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889550+0000", 2024-09-21T14:04:08.077 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5000999999999995e-05 2024-09-21T14:04:08.077 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.077 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.077 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:04:08.077 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889561+0000", 2024-09-21T14:04:08.077 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.129e-05 2024-09-21T14:04:08.077 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.077 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.077 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:08.077 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674619+0000", 2024-09-21T14:04:08.077 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.785057703 2024-09-21T14:04:08.077 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.077 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.077 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:08.078 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674632+0000", 2024-09-21T14:04:08.078 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2962000000000001e-05 2024-09-21T14:04:08.078 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:08.078 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:08.078 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:08.078 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.078 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.078 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232068 8.f 8.d5b3687f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T14:04:08.078 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.894747+0000", 2024-09-21T14:04:08.078 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 416.15141215400001, 2024-09-21T14:04:08.078 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.77963143999999995, 2024-09-21T14:04:08.078 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:08.078 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:08.078 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T14:04:08.078 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:04:08.079 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T14:04:08.079 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:04:08.079 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232068 2024-09-21T14:04:08.079 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.079 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:08.079 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.079 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:08.079 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.894747+0000", 2024-09-21T14:04:08.079 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:08.079 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.079 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.079 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:08.079 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.894747+0000", 2024-09-21T14:04:08.079 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:08.080 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.080 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.080 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:08.080 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.894749+0000", 2024-09-21T14:04:08.080 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4849999999999999e-06 2024-09-21T14:04:08.080 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.080 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.080 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:08.080 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.894761+0000", 2024-09-21T14:04:08.080 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1809e-05 2024-09-21T14:04:08.080 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.080 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.080 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:08.080 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.894780+0000", 2024-09-21T14:04:08.080 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8593999999999998e-05 2024-09-21T14:04:08.081 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.081 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.081 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:08.081 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.894789+0000", 2024-09-21T14:04:08.081 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2839999999999995e-06 2024-09-21T14:04:08.081 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.081 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.081 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:08.081 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.894865+0000", 2024-09-21T14:04:08.081 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6150999999999996e-05 2024-09-21T14:04:08.081 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.081 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.081 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:04:08.081 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.894874+0000", 2024-09-21T14:04:08.082 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2450000000000004e-06 2024-09-21T14:04:08.082 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.082 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.082 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:08.082 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674368+0000", 2024-09-21T14:04:08.082 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.77949425999999999 2024-09-21T14:04:08.082 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.082 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.082 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:08.082 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674378+0000", 2024-09-21T14:04:08.082 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6120000000000005e-06 2024-09-21T14:04:08.082 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:08.082 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:08.083 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:08.083 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.083 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.083 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232074 8.0 8.57810 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T14:04:08.083 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.901181+0000", 2024-09-21T14:04:08.083 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 416.14497764100003, 2024-09-21T14:04:08.083 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.77382758200000001, 2024-09-21T14:04:08.083 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:08.083 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:08.083 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T14:04:08.083 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:04:08.083 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T14:04:08.083 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:04:08.083 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232074 2024-09-21T14:04:08.084 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.084 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:08.084 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.084 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:08.084 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.901181+0000", 2024-09-21T14:04:08.084 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:08.084 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.084 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.084 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:08.084 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.901181+0000", 2024-09-21T14:04:08.084 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:08.084 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.084 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.084 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:08.085 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.901184+0000", 2024-09-21T14:04:08.085 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.8660000000000002e-06 2024-09-21T14:04:08.085 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.085 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.085 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:08.085 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.901193+0000", 2024-09-21T14:04:08.085 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1549999999999996e-06 2024-09-21T14:04:08.085 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.085 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.085 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:08.085 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.901215+0000", 2024-09-21T14:04:08.085 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2272e-05 2024-09-21T14:04:08.085 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.085 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.086 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:08.086 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.901227+0000", 2024-09-21T14:04:08.086 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1116e-05 2024-09-21T14:04:08.086 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.086 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.086 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:08.086 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.901351+0000", 2024-09-21T14:04:08.086 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000124172 2024-09-21T14:04:08.086 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.086 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.086 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:04:08.086 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.901368+0000", 2024-09-21T14:04:08.086 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7008999999999999e-05 2024-09-21T14:04:08.086 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.086 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.087 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:08.087 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674970+0000", 2024-09-21T14:04:08.087 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.77360225199999999 2024-09-21T14:04:08.087 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.087 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.087 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:08.087 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.675009+0000", 2024-09-21T14:04:08.087 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8739999999999998e-05 2024-09-21T14:04:08.087 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:08.087 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:08.087 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:08.087 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.087 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.087 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232078 8.b 8.7f49789b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T14:04:08.088 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.904662+0000", 2024-09-21T14:04:08.088 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 416.14149723600002, 2024-09-21T14:04:08.088 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.77001060399999999, 2024-09-21T14:04:08.088 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:08.088 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:08.088 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T14:04:08.088 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:04:08.088 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T14:04:08.088 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:04:08.088 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232078 2024-09-21T14:04:08.088 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.088 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:08.088 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.088 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:08.089 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904662+0000", 2024-09-21T14:04:08.089 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:08.089 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.089 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.089 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:08.089 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904662+0000", 2024-09-21T14:04:08.089 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:08.089 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.089 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.089 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:08.089 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904663+0000", 2024-09-21T14:04:08.089 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.548e-06 2024-09-21T14:04:08.089 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.089 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.089 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:08.090 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904669+0000", 2024-09-21T14:04:08.090 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9619999999999998e-06 2024-09-21T14:04:08.090 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.090 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.090 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:08.090 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904684+0000", 2024-09-21T14:04:08.090 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4718e-05 2024-09-21T14:04:08.090 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.090 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.090 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:08.090 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904691+0000", 2024-09-21T14:04:08.090 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.861e-06 2024-09-21T14:04:08.090 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.090 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.091 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:08.091 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904750+0000", 2024-09-21T14:04:08.091 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9533000000000001e-05 2024-09-21T14:04:08.091 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.091 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.091 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:04:08.091 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904757+0000", 2024-09-21T14:04:08.091 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0369999999999997e-06 2024-09-21T14:04:08.091 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.092 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.092 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:08.092 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674661+0000", 2024-09-21T14:04:08.092 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.769903754 2024-09-21T14:04:08.092 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.092 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.092 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:08.092 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674672+0000", 2024-09-21T14:04:08.092 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1191e-05 2024-09-21T14:04:08.092 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:08.092 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:08.092 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:08.092 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.092 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.093 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232079 8.b 8.5c3e90b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T14:04:08.093 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.904710+0000", 2024-09-21T14:04:08.093 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 416.14144888099997, 2024-09-21T14:04:08.093 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.76999876, 2024-09-21T14:04:08.093 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:08.093 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:08.093 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T14:04:08.093 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:04:08.093 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T14:04:08.093 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:04:08.093 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232079 2024-09-21T14:04:08.093 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.093 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:08.093 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.094 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:08.094 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904710+0000", 2024-09-21T14:04:08.094 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:08.094 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.094 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.094 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:08.094 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904710+0000", 2024-09-21T14:04:08.094 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:08.094 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.094 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.094 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:08.094 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904711+0000", 2024-09-21T14:04:08.094 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8199999999999999e-07 2024-09-21T14:04:08.095 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.095 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.095 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:08.095 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904714+0000", 2024-09-21T14:04:08.095 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9079999999999999e-06 2024-09-21T14:04:08.095 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.095 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.095 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:08.095 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904726+0000", 2024-09-21T14:04:08.095 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2514000000000001e-05 2024-09-21T14:04:08.095 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.095 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.095 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:08.095 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904733+0000", 2024-09-21T14:04:08.095 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2690000000000002e-06 2024-09-21T14:04:08.096 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.096 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.096 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:08.096 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904784+0000", 2024-09-21T14:04:08.096 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1007000000000001e-05 2024-09-21T14:04:08.096 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.096 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.096 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:04:08.096 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904794+0000", 2024-09-21T14:04:08.096 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7419999999999992e-06 2024-09-21T14:04:08.096 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.096 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.096 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:08.096 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674699+0000", 2024-09-21T14:04:08.097 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.76990482900000001 2024-09-21T14:04:08.097 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.097 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.097 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:08.097 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674709+0000", 2024-09-21T14:04:08.097 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8090000000000004e-06 2024-09-21T14:04:08.097 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:08.097 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:08.097 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:08.097 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.097 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.097 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25421.0:113916 9.8 9:1f612498:::benchmark_data_smithi049_138504_object113915:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-21T14:04:08.097 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:59:25.602805+0000", 2024-09-21T14:04:08.098 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 282.44335344900003, 2024-09-21T14:04:08.098 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.34059980099999998, 2024-09-21T14:04:08.098 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:08.098 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:08.098 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:08.098 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:04:08.098 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25421", 2024-09-21T14:04:08.098 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:04:08.098 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 113916 2024-09-21T14:04:08.098 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.098 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:08.098 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.098 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:08.098 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.602805+0000", 2024-09-21T14:04:08.098 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:08.099 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.099 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.099 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:08.099 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.602805+0000", 2024-09-21T14:04:08.099 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:08.099 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.099 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.099 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:08.099 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.602807+0000", 2024-09-21T14:04:08.099 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7099999999999999e-06 2024-09-21T14:04:08.100 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.100 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.100 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:08.100 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.602830+0000", 2024-09-21T14:04:08.100 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2758000000000001e-05 2024-09-21T14:04:08.100 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.100 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.100 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:08.100 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.602847+0000", 2024-09-21T14:04:08.100 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6721000000000001e-05 2024-09-21T14:04:08.101 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.101 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.101 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:08.101 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.602854+0000", 2024-09-21T14:04:08.101 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9100000000000005e-06 2024-09-21T14:04:08.101 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.101 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.101 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:08.101 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.602925+0000", 2024-09-21T14:04:08.101 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0088999999999999e-05 2024-09-21T14:04:08.101 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.101 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.102 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:04:08.102 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.603043+0000", 2024-09-21T14:04:08.102 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011804200000000001 2024-09-21T14:04:08.102 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.102 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.102 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,5", 2024-09-21T14:04:08.102 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.603177+0000", 2024-09-21T14:04:08.102 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000134821 2024-09-21T14:04:08.102 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.102 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.102 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:04:08.102 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.606802+0000", 2024-09-21T14:04:08.102 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0036243880000000001 2024-09-21T14:04:08.103 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.103 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.103 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:08.103 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.608461+0000", 2024-09-21T14:04:08.103 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0016595710000000001 2024-09-21T14:04:08.103 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.103 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.103 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:08.103 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.943300+0000", 2024-09-21T14:04:08.103 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.33483894400000003 2024-09-21T14:04:08.103 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.103 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.103 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:04:08.104 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.943360+0000", 2024-09-21T14:04:08.104 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9401999999999998e-05 2024-09-21T14:04:08.104 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.104 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.104 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:08.104 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.943405+0000", 2024-09-21T14:04:08.104 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5445000000000003e-05 2024-09-21T14:04:08.104 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:08.104 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:08.104 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:08.104 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.104 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.104 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25421.0:113943 9.8 9:1e81a0a3:::benchmark_data_smithi049_138504_object113942:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-21T14:04:08.104 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:59:25.623062+0000", 2024-09-21T14:04:08.104 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 282.42309668600001, 2024-09-21T14:04:08.105 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.32046552900000003, 2024-09-21T14:04:08.105 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:08.105 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:08.105 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:08.105 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:04:08.105 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25421", 2024-09-21T14:04:08.105 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:04:08.105 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 113943 2024-09-21T14:04:08.105 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.105 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:08.105 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.105 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:08.105 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.623062+0000", 2024-09-21T14:04:08.105 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:08.106 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.106 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.106 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:08.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:08.107 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:08.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:04:08.107 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.623062+0000", 2024-09-21T14:04:08.108 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:08.108 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.108 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.108 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:08.108 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.623063+0000", 2024-09-21T14:04:08.108 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0210000000000001e-06 2024-09-21T14:04:08.108 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.108 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.108 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:08.108 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.623089+0000", 2024-09-21T14:04:08.108 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.567e-05 2024-09-21T14:04:08.108 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.108 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.109 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:08.109 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.623103+0000", 2024-09-21T14:04:08.109 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4248e-05 2024-09-21T14:04:08.109 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.109 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.109 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:08.109 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.623109+0000", 2024-09-21T14:04:08.109 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5949999999999998e-06 2024-09-21T14:04:08.109 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.109 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.109 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:08.109 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.623150+0000", 2024-09-21T14:04:08.109 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0868999999999998e-05 2024-09-21T14:04:08.109 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.110 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.110 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:04:08.110 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.623240+0000", 2024-09-21T14:04:08.110 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0562000000000005e-05 2024-09-21T14:04:08.110 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.110 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.110 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,5", 2024-09-21T14:04:08.110 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.623353+0000", 2024-09-21T14:04:08.110 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000113249 2024-09-21T14:04:08.110 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.110 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.110 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:04:08.110 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.625438+0000", 2024-09-21T14:04:08.110 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0020847750000000001 2024-09-21T14:04:08.111 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.111 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.111 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:08.111 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.626378+0000", 2024-09-21T14:04:08.111 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00093971199999999999 2024-09-21T14:04:08.111 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.111 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.111 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:08.111 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.943444+0000", 2024-09-21T14:04:08.111 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.317066615 2024-09-21T14:04:08.111 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.111 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.111 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:04:08.111 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.943487+0000", 2024-09-21T14:04:08.112 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2933999999999999e-05 2024-09-21T14:04:08.112 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.112 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.112 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:08.112 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.943528+0000", 2024-09-21T14:04:08.112 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0278999999999998e-05 2024-09-21T14:04:08.112 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:08.112 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:08.112 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:08.112 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.112 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.112 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25421.0:113948 9.8 9:1fea698d:::benchmark_data_smithi049_138504_object113947:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-21T14:04:08.112 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:59:25.630511+0000", 2024-09-21T14:04:08.113 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 282.41564740000001, 2024-09-21T14:04:08.113 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.31617158400000001, 2024-09-21T14:04:08.113 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:08.113 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:08.113 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:08.113 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:04:08.113 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25421", 2024-09-21T14:04:08.113 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:04:08.113 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 113948 2024-09-21T14:04:08.113 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.113 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:08.113 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.113 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:08.114 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.630511+0000", 2024-09-21T14:04:08.114 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:08.114 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.114 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.114 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:08.114 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.630511+0000", 2024-09-21T14:04:08.114 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:08.114 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.114 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.114 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:08.114 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.630512+0000", 2024-09-21T14:04:08.114 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.16e-07 2024-09-21T14:04:08.114 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.114 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.114 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:08.115 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.630531+0000", 2024-09-21T14:04:08.115 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8845999999999999e-05 2024-09-21T14:04:08.115 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.115 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.115 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:08.115 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.630542+0000", 2024-09-21T14:04:08.115 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0651e-05 2024-09-21T14:04:08.115 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.115 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.115 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:08.115 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.630547+0000", 2024-09-21T14:04:08.115 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8770000000000001e-06 2024-09-21T14:04:08.115 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.116 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.116 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:08.116 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.630585+0000", 2024-09-21T14:04:08.116 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8488999999999997e-05 2024-09-21T14:04:08.116 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.116 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.116 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:04:08.116 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.630660+0000", 2024-09-21T14:04:08.116 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5290000000000006e-05 2024-09-21T14:04:08.116 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.116 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.116 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,5", 2024-09-21T14:04:08.116 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.630760+0000", 2024-09-21T14:04:08.116 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9834000000000007e-05 2024-09-21T14:04:08.117 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.117 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.117 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:04:08.117 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.632607+0000", 2024-09-21T14:04:08.117 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001847174 2024-09-21T14:04:08.117 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.117 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.117 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:08.117 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.633151+0000", 2024-09-21T14:04:08.117 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00054377600000000005 2024-09-21T14:04:08.117 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.117 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.117 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:08.117 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.946575+0000", 2024-09-21T14:04:08.118 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.31342341099999999 2024-09-21T14:04:08.118 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.118 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.118 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:04:08.118 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.946632+0000", 2024-09-21T14:04:08.118 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7055999999999997e-05 2024-09-21T14:04:08.118 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.118 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.118 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:08.118 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.946683+0000", 2024-09-21T14:04:08.118 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1264000000000002e-05 2024-09-21T14:04:08.118 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:08.118 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:08.119 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:08.119 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.119 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.119 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25421.0:172314 9.4 e315/260 9:2b749f7e:::benchmark_data_smithi049_138504_object50906:head v 315'10563, mlcod=315'10559)", 2024-09-21T14:04:08.119 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:00:33.250136+0000", 2024-09-21T14:04:08.119 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 214.796022768, 2024-09-21T14:04:08.119 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.270468655, 2024-09-21T14:04:08.119 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:08.119 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:08.119 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:08.119 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:08.119 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.119 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:08.119 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250136+0000", 2024-09-21T14:04:08.120 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:08.120 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.120 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.120 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:08.120 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250136+0000", 2024-09-21T14:04:08.120 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:08.120 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.120 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.120 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:08.120 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250137+0000", 2024-09-21T14:04:08.120 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1999999999999999e-07 2024-09-21T14:04:08.120 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.120 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.120 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:08.120 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250142+0000", 2024-09-21T14:04:08.121 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5609999999999998e-06 2024-09-21T14:04:08.121 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.121 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.121 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:08.121 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250150+0000", 2024-09-21T14:04:08.121 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1310000000000006e-06 2024-09-21T14:04:08.121 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.121 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.121 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:08.121 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250156+0000", 2024-09-21T14:04:08.121 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7740000000000001e-06 2024-09-21T14:04:08.121 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.121 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.121 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:08.122 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.253391+0000", 2024-09-21T14:04:08.122 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0032349969999999999 2024-09-21T14:04:08.122 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.122 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.122 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:04:08.122 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.253410+0000", 2024-09-21T14:04:08.122 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9097e-05 2024-09-21T14:04:08.122 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.122 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.122 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:04:08.122 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.520555+0000", 2024-09-21T14:04:08.122 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26714502600000001 2024-09-21T14:04:08.122 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.122 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.123 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:08.123 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.520605+0000", 2024-09-21T14:04:08.123 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9449e-05 2024-09-21T14:04:08.123 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:08.123 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:08.123 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:08.123 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.123 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.123 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25421.0:172319 9.4 e315/260 9:21e132db:::benchmark_data_smithi049_138504_object50911:head v 315'10564, mlcod=315'10559)", 2024-09-21T14:04:08.123 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:00:33.250191+0000", 2024-09-21T14:04:08.123 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 214.795967784, 2024-09-21T14:04:08.123 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.270657121, 2024-09-21T14:04:08.123 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:08.123 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:08.124 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:08.124 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:08.124 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.124 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:08.124 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250191+0000", 2024-09-21T14:04:08.124 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:08.124 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.124 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.124 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:08.124 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250191+0000", 2024-09-21T14:04:08.125 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:08.125 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.125 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.125 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:08.125 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250191+0000", 2024-09-21T14:04:08.125 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8799999999999998e-07 2024-09-21T14:04:08.125 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.125 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.125 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:08.125 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250195+0000", 2024-09-21T14:04:08.125 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.647e-06 2024-09-21T14:04:08.125 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.125 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.125 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:08.125 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250201+0000", 2024-09-21T14:04:08.125 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4200000000000004e-06 2024-09-21T14:04:08.126 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.126 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.126 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:08.126 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250206+0000", 2024-09-21T14:04:08.126 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.994e-06 2024-09-21T14:04:08.126 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.126 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.126 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:08.126 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.256110+0000", 2024-09-21T14:04:08.126 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0059039840000000001 2024-09-21T14:04:08.126 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.126 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.130 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:04:08.130 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.256130+0000", 2024-09-21T14:04:08.130 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.98e-05 2024-09-21T14:04:08.130 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.130 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.130 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:04:08.130 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.520816+0000", 2024-09-21T14:04:08.130 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.264685528 2024-09-21T14:04:08.130 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.130 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.130 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:08.131 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.520848+0000", 2024-09-21T14:04:08.131 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2360000000000002e-05 2024-09-21T14:04:08.131 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:08.131 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:08.131 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:08.131 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.131 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.131 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25421.0:172326 9.4 e315/260 9:22a8fc8a:::benchmark_data_smithi049_138504_object50918:head v 315'10565, mlcod=315'10560)", 2024-09-21T14:04:08.131 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:00:33.254759+0000", 2024-09-21T14:04:08.131 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 214.79139949399999, 2024-09-21T14:04:08.131 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26611502300000001, 2024-09-21T14:04:08.131 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:08.131 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:08.131 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:08.131 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:08.132 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.132 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:08.132 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254759+0000", 2024-09-21T14:04:08.132 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:08.132 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.132 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.132 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:08.132 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254759+0000", 2024-09-21T14:04:08.132 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:08.132 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.132 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.132 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:08.132 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254760+0000", 2024-09-21T14:04:08.132 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6400000000000001e-07 2024-09-21T14:04:08.132 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.132 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.133 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:08.133 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254854+0000", 2024-09-21T14:04:08.133 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4320999999999994e-05 2024-09-21T14:04:08.133 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.133 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.133 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:08.133 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254865+0000", 2024-09-21T14:04:08.133 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.101e-05 2024-09-21T14:04:08.133 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.133 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.133 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:08.133 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254872+0000", 2024-09-21T14:04:08.133 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2539999999999998e-06 2024-09-21T14:04:08.133 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.134 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.134 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:08.134 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.258679+0000", 2024-09-21T14:04:08.134 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.003807342 2024-09-21T14:04:08.134 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.134 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.134 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:04:08.134 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.258699+0000", 2024-09-21T14:04:08.134 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.039e-05 2024-09-21T14:04:08.134 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.134 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.134 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:04:08.134 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.520851+0000", 2024-09-21T14:04:08.135 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26215118900000001 2024-09-21T14:04:08.135 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.135 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.135 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:08.135 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.520874+0000", 2024-09-21T14:04:08.135 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3753000000000001e-05 2024-09-21T14:04:08.135 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:08.135 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:08.135 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:08.135 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.135 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.135 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25421.0:172331 9.f e315/260 9:f6449e5b:::benchmark_data_smithi049_138504_object50923:head v 315'10871, mlcod=315'10867)", 2024-09-21T14:04:08.135 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:00:33.254942+0000", 2024-09-21T14:04:08.135 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 214.791217117, 2024-09-21T14:04:08.135 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.274133181, 2024-09-21T14:04:08.136 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:08.136 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:08.136 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:08.136 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:08.136 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.136 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:08.136 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254942+0000", 2024-09-21T14:04:08.136 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:08.136 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.136 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.136 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:08.136 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254942+0000", 2024-09-21T14:04:08.136 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:08.136 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.136 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.137 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:08.137 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254942+0000", 2024-09-21T14:04:08.137 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5700000000000002e-07 2024-09-21T14:04:08.137 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.137 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.137 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:08.137 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254949+0000", 2024-09-21T14:04:08.137 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.024e-06 2024-09-21T14:04:08.137 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.137 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.137 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:08.137 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254959+0000", 2024-09-21T14:04:08.137 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6390000000000004e-06 2024-09-21T14:04:08.137 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.137 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.138 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:08.138 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254965+0000", 2024-09-21T14:04:08.138 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6420000000000004e-06 2024-09-21T14:04:08.138 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.138 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.138 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:08.138 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.259407+0000", 2024-09-21T14:04:08.138 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0044420140000000002 2024-09-21T14:04:08.138 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.138 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.138 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:04:08.138 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.259425+0000", 2024-09-21T14:04:08.139 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7937000000000001e-05 2024-09-21T14:04:08.139 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.139 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.139 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:04:08.139 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.529033+0000", 2024-09-21T14:04:08.139 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26960846599999999 2024-09-21T14:04:08.139 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.140 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.140 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:08.140 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.529075+0000", 2024-09-21T14:04:08.140 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1801999999999997e-05 2024-09-21T14:04:08.140 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:08.140 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:08.140 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:08.140 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.140 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.140 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25421.0:172333 9.2 9:461458cc:::benchmark_data_smithi049_138504_object50925:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-09-21T14:04:08.140 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:00:33.258960+0000", 2024-09-21T14:04:08.140 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 214.78719898200001, 2024-09-21T14:04:08.141 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.261926308, 2024-09-21T14:04:08.141 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:08.141 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:08.141 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:08.141 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:04:08.141 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25421", 2024-09-21T14:04:08.141 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:04:08.141 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 172333 2024-09-21T14:04:08.141 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.141 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:08.141 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.141 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:08.141 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.258960+0000", 2024-09-21T14:04:08.141 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:08.141 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.142 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.142 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:08.142 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.258960+0000", 2024-09-21T14:04:08.142 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:08.142 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.142 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.142 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:08.142 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.258961+0000", 2024-09-21T14:04:08.142 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5629999999999999e-06 2024-09-21T14:04:08.142 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.142 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.142 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:08.142 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.258967+0000", 2024-09-21T14:04:08.142 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5079999999999996e-06 2024-09-21T14:04:08.142 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.143 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.143 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:08.143 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.258979+0000", 2024-09-21T14:04:08.143 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1897e-05 2024-09-21T14:04:08.143 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.143 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.143 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:08.143 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.258986+0000", 2024-09-21T14:04:08.143 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8739999999999997e-06 2024-09-21T14:04:08.143 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.143 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.143 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:08.143 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.259036+0000", 2024-09-21T14:04:08.143 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0695000000000002e-05 2024-09-21T14:04:08.143 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.144 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.144 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:04:08.144 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.259144+0000", 2024-09-21T14:04:08.144 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000107462 2024-09-21T14:04:08.144 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.144 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.144 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,2", 2024-09-21T14:04:08.144 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.259252+0000", 2024-09-21T14:04:08.144 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000107667 2024-09-21T14:04:08.144 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.144 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.144 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:04:08.144 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.517843+0000", 2024-09-21T14:04:08.144 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.25859111400000001 2024-09-21T14:04:08.145 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.145 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.145 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:08.145 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.517994+0000", 2024-09-21T14:04:08.145 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00015150299999999999 2024-09-21T14:04:08.145 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.145 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.145 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:08.145 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.520805+0000", 2024-09-21T14:04:08.145 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0028113489999999999 2024-09-21T14:04:08.145 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.145 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.145 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:04:08.145 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.520850+0000", 2024-09-21T14:04:08.145 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4333999999999999e-05 2024-09-21T14:04:08.145 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.146 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.146 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:08.146 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.520886+0000", 2024-09-21T14:04:08.146 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6341999999999999e-05 2024-09-21T14:04:08.146 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:08.146 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:08.146 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:08.146 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.146 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.146 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25421.0:172330 9.4 e315/260 9:2882ed62:::benchmark_data_smithi049_138504_object50922:head v 315'10566, mlcod=315'10560)", 2024-09-21T14:04:08.146 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:00:33.259976+0000", 2024-09-21T14:04:08.146 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 214.78618248699999, 2024-09-21T14:04:08.146 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26092217499999998, 2024-09-21T14:04:08.146 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:08.146 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:08.147 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:08.147 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:08.147 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.147 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:08.147 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.259976+0000", 2024-09-21T14:04:08.147 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:08.147 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.147 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.147 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:08.147 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.259976+0000", 2024-09-21T14:04:08.147 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:08.147 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.147 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.148 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:08.148 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.259977+0000", 2024-09-21T14:04:08.148 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1e-07 2024-09-21T14:04:08.148 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.148 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.148 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:08.148 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.259980+0000", 2024-09-21T14:04:08.148 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.58e-06 2024-09-21T14:04:08.148 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.148 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.148 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:08.148 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.259986+0000", 2024-09-21T14:04:08.148 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9109999999999998e-06 2024-09-21T14:04:08.148 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.148 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.148 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:08.149 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.259991+0000", 2024-09-21T14:04:08.149 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4510000000000002e-06 2024-09-21T14:04:08.149 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.149 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.149 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:08.149 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.261224+0000", 2024-09-21T14:04:08.149 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0012338169999999999 2024-09-21T14:04:08.149 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.149 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.149 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:04:08.149 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.261241+0000", 2024-09-21T14:04:08.149 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6762e-05 2024-09-21T14:04:08.149 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.149 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.149 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:04:08.150 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.520876+0000", 2024-09-21T14:04:08.150 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.25963512500000002 2024-09-21T14:04:08.150 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.150 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.150 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:08.150 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.520899+0000", 2024-09-21T14:04:08.150 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2218999999999999e-05 2024-09-21T14:04:08.150 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:08.150 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:08.150 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:08.150 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.150 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.150 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25421.0:172334 9.4 e315/260 9:24fa2bd3:::benchmark_data_smithi049_138504_object50926:head v 315'10567, mlcod=315'10561)", 2024-09-21T14:04:08.151 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:00:33.262387+0000", 2024-09-21T14:04:08.151 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 214.78377227499999, 2024-09-21T14:04:08.151 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26075252599999998, 2024-09-21T14:04:08.151 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:08.151 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:08.151 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:08.151 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:08.151 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.151 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:08.151 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.262387+0000", 2024-09-21T14:04:08.151 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:08.151 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.151 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.151 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:08.151 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.262387+0000", 2024-09-21T14:04:08.152 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:08.152 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.152 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.152 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:08.152 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.262388+0000", 2024-09-21T14:04:08.152 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6099999999999999e-07 2024-09-21T14:04:08.152 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.152 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.152 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:08.152 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.262393+0000", 2024-09-21T14:04:08.152 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8289999999999996e-06 2024-09-21T14:04:08.152 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.152 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.152 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:08.152 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.262488+0000", 2024-09-21T14:04:08.152 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4914000000000006e-05 2024-09-21T14:04:08.152 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.153 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.153 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:08.153 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.262504+0000", 2024-09-21T14:04:08.153 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5498999999999998e-05 2024-09-21T14:04:08.153 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.153 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.153 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:08.153 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.263573+0000", 2024-09-21T14:04:08.153 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0010692119999999999 2024-09-21T14:04:08.153 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.153 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.153 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:04:08.153 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.263590+0000", 2024-09-21T14:04:08.153 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7331000000000001e-05 2024-09-21T14:04:08.153 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.154 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.154 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:04:08.154 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.523109+0000", 2024-09-21T14:04:08.154 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.25951882100000001 2024-09-21T14:04:08.154 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:08.154 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:08.154 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:08.154 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.523139+0000", 2024-09-21T14:04:08.154 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9958999999999999e-05 2024-09-21T14:04:08.154 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:08.154 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:08.154 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:08.154 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:08.154 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:08.155 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T14:04:08.208 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:04:08.251 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:08.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:08.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:04:08.353 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:04:08.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:08.393 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:08.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:04:08.494 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:04:08.537 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:08.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:08.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:04:08.640 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:04:08.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:08.676 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:08.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:04:08.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:08 smithi049 ceph-mon[30080]: 10.8 scrub ok 2024-09-21T14:04:08.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:08 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:04:08.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:08 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:04:08.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:08 smithi049 ceph-mon[30080]: osdmap e406: 8 total, 8 up, 8 in 2024-09-21T14:04:08.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:08 smithi049 ceph-mon[30080]: 10.b scrub ok 2024-09-21T14:04:08.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:08 smithi049 ceph-mon[30080]: 10.1 scrub starts 2024-09-21T14:04:08.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:08 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2710414510' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:04:08.757 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:08 smithi049 ceph-mon[40489]: 10.8 scrub ok 2024-09-21T14:04:08.757 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:08 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:04:08.757 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:08 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:04:08.757 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:08 smithi049 ceph-mon[40489]: osdmap e406: 8 total, 8 up, 8 in 2024-09-21T14:04:08.757 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:08 smithi049 ceph-mon[40489]: 10.b scrub ok 2024-09-21T14:04:08.757 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:08 smithi049 ceph-mon[40489]: 10.1 scrub starts 2024-09-21T14:04:08.757 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:08 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2710414510' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:04:08.778 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:08.815 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:08.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:08.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:08.880 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 63 16 99704 99688 98.8737 65 0.00603065 0.0101088 2024-09-21T14:04:08.917 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:04:08.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:08 smithi187 ceph-mon[33032]: 10.8 scrub ok 2024-09-21T14:04:08.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:08 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:04:08.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:08 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:04:08.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:08 smithi187 ceph-mon[33032]: osdmap e406: 8 total, 8 up, 8 in 2024-09-21T14:04:08.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:08 smithi187 ceph-mon[33032]: 10.b scrub ok 2024-09-21T14:04:08.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:08 smithi187 ceph-mon[33032]: 10.1 scrub starts 2024-09-21T14:04:08.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:08 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2710414510' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:04:08.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:08.955 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:08.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:04:09.056 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:04:09.096 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:09.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:09.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:04:09.198 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:04:09.235 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:09.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:09.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:04:09.337 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:04:09.374 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:04:09.374 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:04:09.374 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:04:09.374 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:04:09.374 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:04:09.374 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:04:09.374 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:04:09.374 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:04:09.375 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:04:09.375 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:04:09.375 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:04:09.375 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:04:09.375 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:04:09.375 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:04:09.375 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:04:09.375 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:04:09.380 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:09.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:09.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:04:09.482 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:04:09.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:09 smithi049 ceph-mon[30080]: pgmap v775: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-21T14:04:09.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:09 smithi049 ceph-mon[30080]: 4.19 scrub starts 2024-09-21T14:04:09.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:09 smithi049 ceph-mon[30080]: 4.19 scrub ok 2024-09-21T14:04:09.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:09 smithi049 ceph-mon[30080]: 10.e scrub ok 2024-09-21T14:04:09.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:09 smithi049 ceph-mon[30080]: 10.d scrub starts 2024-09-21T14:04:09.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:09 smithi049 ceph-mon[40489]: pgmap v775: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-21T14:04:09.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:09 smithi049 ceph-mon[40489]: 4.19 scrub starts 2024-09-21T14:04:09.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:09 smithi049 ceph-mon[40489]: 4.19 scrub ok 2024-09-21T14:04:09.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:09 smithi049 ceph-mon[40489]: 10.e scrub ok 2024-09-21T14:04:09.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:09 smithi049 ceph-mon[40489]: 10.d scrub starts 2024-09-21T14:04:09.528 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:09.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:09.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:04:09.630 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:04:09.670 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:09.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:09.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:04:09.772 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:04:09.809 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:09.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:09.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:04:09.859 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:09 smithi187 ceph-mon[33032]: pgmap v775: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-21T14:04:09.859 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:09 smithi187 ceph-mon[33032]: 4.19 scrub starts 2024-09-21T14:04:09.859 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:09 smithi187 ceph-mon[33032]: 4.19 scrub ok 2024-09-21T14:04:09.859 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:09 smithi187 ceph-mon[33032]: 10.e scrub ok 2024-09-21T14:04:09.859 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:09 smithi187 ceph-mon[33032]: 10.d scrub starts 2024-09-21T14:04:09.880 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 64 16 100817 100801 98.4157 69.5625 0.0318019 0.0101558 2024-09-21T14:04:09.910 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:04:09.955 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:09.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:09.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:04:10.057 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:04:10.093 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:10.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:10.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:04:10.195 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:04:10.196 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:04:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:04:10.228 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:10.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:10.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:04:10.330 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:04:10.365 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:10.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:10.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:04:10.467 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:04:10.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:10 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:04:10.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:10 smithi049 ceph-mon[40489]: 10.1 scrub ok 2024-09-21T14:04:10.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:10 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:04:10.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:10 smithi049 ceph-mon[30080]: 10.1 scrub ok 2024-09-21T14:04:10.510 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:10.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:10.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:04:10.612 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:04:10.650 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:10.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:10.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:04:10.718 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-21T14:04:10.719 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd reweight-by-utilization 1000 3.0 2024-09-21T14:04:10.752 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:04:10.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:10.811 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:10.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:04:10.849 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:04:10 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:04:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:04:10.880 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 65 16 101988 101972 98.0275 73.1875 0.00500435 0.0101963 2024-09-21T14:04:10.912 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:04:10.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:10 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:04:10.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:10 smithi187 ceph-mon[33032]: 10.1 scrub ok 2024-09-21T14:04:10.965 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:10.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:10.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:04:11.067 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:04:11.103 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:11.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:11.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:04:11.205 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:04:11.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:11.241 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:11.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:04:11.280 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T14:04:11.306 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T14:04:11.342 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:11.394 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:04:11.430 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:11.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:11.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:11.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:11 smithi049 ceph-mon[30080]: pgmap v776: 158 pgs: 3 active+clean+scrubbing, 155 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-09-21T14:04:11.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:11 smithi049 ceph-mon[30080]: 3.0 scrub starts 2024-09-21T14:04:11.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:11 smithi049 ceph-mon[30080]: 3.0 scrub ok 2024-09-21T14:04:11.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:11 smithi049 ceph-mon[30080]: 3.1c scrub starts 2024-09-21T14:04:11.508 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:11 smithi049 ceph-mon[30080]: 3.1c scrub ok 2024-09-21T14:04:11.508 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:11 smithi049 ceph-mon[30080]: 10.d scrub ok 2024-09-21T14:04:11.508 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:11 smithi049 ceph-mon[30080]: 3.3 scrub starts 2024-09-21T14:04:11.508 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:11 smithi049 ceph-mon[30080]: 3.3 scrub ok 2024-09-21T14:04:11.508 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:11 smithi049 ceph-mon[40489]: pgmap v776: 158 pgs: 3 active+clean+scrubbing, 155 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-09-21T14:04:11.508 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:11 smithi049 ceph-mon[40489]: 3.0 scrub starts 2024-09-21T14:04:11.508 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:11 smithi049 ceph-mon[40489]: 3.0 scrub ok 2024-09-21T14:04:11.508 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:11 smithi049 ceph-mon[40489]: 3.1c scrub starts 2024-09-21T14:04:11.508 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:11 smithi049 ceph-mon[40489]: 3.1c scrub ok 2024-09-21T14:04:11.508 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:11 smithi049 ceph-mon[40489]: 10.d scrub ok 2024-09-21T14:04:11.509 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:11 smithi049 ceph-mon[40489]: 3.3 scrub starts 2024-09-21T14:04:11.509 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:11 smithi049 ceph-mon[40489]: 3.3 scrub ok 2024-09-21T14:04:11.532 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:04:11.576 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:11.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:11.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:04:11.678 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:04:11.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:11 smithi187 ceph-mon[33032]: pgmap v776: 158 pgs: 3 active+clean+scrubbing, 155 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-09-21T14:04:11.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:11 smithi187 ceph-mon[33032]: 3.0 scrub starts 2024-09-21T14:04:11.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:11 smithi187 ceph-mon[33032]: 3.0 scrub ok 2024-09-21T14:04:11.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:11 smithi187 ceph-mon[33032]: 3.1c scrub starts 2024-09-21T14:04:11.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:11 smithi187 ceph-mon[33032]: 3.1c scrub ok 2024-09-21T14:04:11.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:11 smithi187 ceph-mon[33032]: 10.d scrub ok 2024-09-21T14:04:11.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:11 smithi187 ceph-mon[33032]: 3.3 scrub starts 2024-09-21T14:04:11.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:11 smithi187 ceph-mon[33032]: 3.3 scrub ok 2024-09-21T14:04:11.705 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:04:11.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:11.728 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:11.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:04:11.829 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:04:11.862 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:11.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:11.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:04:11.880 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 66 16 103099 103083 97.5943 69.4375 0.0100834 0.010242 2024-09-21T14:04:11.963 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:04:12.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:12.015 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:12.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:04:12.104 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:04:12.116 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:04:12.153 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:12.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:12.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:04:12.255 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:12.292 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:12.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:12.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:12.302 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T14:04:12.393 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:04:12.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:12.436 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:12.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:04:12.481 INFO:teuthology.orchestra.run.smithi049.stdout:moved 0 / 474 (0%) 2024-09-21T14:04:12.481 INFO:teuthology.orchestra.run.smithi049.stdout:avg 59.25 2024-09-21T14:04:12.481 INFO:teuthology.orchestra.run.smithi049.stdout:stddev 5.76086 -> 5.76086 (expected baseline 7.20026) 2024-09-21T14:04:12.481 INFO:teuthology.orchestra.run.smithi049.stdout:min osd.2 with 48 -> 48 pgs (0.810127 -> 0.810127 * mean) 2024-09-21T14:04:12.481 INFO:teuthology.orchestra.run.smithi049.stdout:max osd.4 with 68 -> 68 pgs (1.14768 -> 1.14768 * mean) 2024-09-21T14:04:12.481 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T14:04:12.481 INFO:teuthology.orchestra.run.smithi049.stdout:oload 1000 2024-09-21T14:04:12.482 INFO:teuthology.orchestra.run.smithi049.stdout:max_change 3 2024-09-21T14:04:12.482 INFO:teuthology.orchestra.run.smithi049.stdout:max_change_osds 4 2024-09-21T14:04:12.482 INFO:teuthology.orchestra.run.smithi049.stdout:average_utilization 0.0302 2024-09-21T14:04:12.482 INFO:teuthology.orchestra.run.smithi049.stdout:overload_utilization 0.3020 2024-09-21T14:04:12.482 INFO:teuthology.orchestra.run.smithi049.stderr:no change 2024-09-21T14:04:12.492 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T14:04:12.492 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:04:12.499 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:12 smithi049 ceph-mon[40489]: 3.1d scrub starts 2024-09-21T14:04:12.499 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:12 smithi049 ceph-mon[30080]: 3.1d scrub starts 2024-09-21T14:04:12.537 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:04:12.563 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:12.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:12.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:04:12.665 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:04:12.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:12 smithi187 ceph-mon[33032]: 3.1d scrub starts 2024-09-21T14:04:12.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:12 smithi187 ceph-mon[33032]: 3.1d scrub ok 2024-09-21T14:04:12.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:12 smithi187 ceph-mon[33032]: 10.0 scrub starts 2024-09-21T14:04:12.700 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:12.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:12.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:04:12.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:12 smithi049 ceph-mon[40489]: 3.1d scrub ok 2024-09-21T14:04:12.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:12 smithi049 ceph-mon[40489]: 10.0 scrub starts 2024-09-21T14:04:12.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:12 smithi049 ceph-mon[30080]: 3.1d scrub ok 2024-09-21T14:04:12.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:12 smithi049 ceph-mon[30080]: 10.0 scrub starts 2024-09-21T14:04:12.802 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:12.838 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:12.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:12.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:12.880 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 67 16 104139 104123 97.1078 65 0.011295 0.0102928 2024-09-21T14:04:12.940 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:04:12.973 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:12.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:12.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:04:13.075 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:04:13.116 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:13.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:13.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:04:13.218 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:04:13.250 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:13.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:13.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:04:13.295 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T14:04:13.352 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:13.391 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:13.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:13.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:13.493 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:04:13.535 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:13.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:13.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:04:13.637 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:04:13.675 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:13.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:13.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:04:13.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:13 smithi187 ceph-mon[33032]: pgmap v777: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 6.1 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 52 MiB/s wr, 830 op/s 2024-09-21T14:04:13.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:13 smithi187 ceph-mon[33032]: 4.12 scrub starts 2024-09-21T14:04:13.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:13 smithi187 ceph-mon[33032]: 4.12 scrub ok 2024-09-21T14:04:13.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:13 smithi187 ceph-mon[33032]: from='client.16188 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T14:04:13.725 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:04:13.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:13 smithi049 ceph-mon[40489]: pgmap v777: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 6.1 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 52 MiB/s wr, 830 op/s 2024-09-21T14:04:13.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:13 smithi049 ceph-mon[40489]: 4.12 scrub starts 2024-09-21T14:04:13.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:13 smithi049 ceph-mon[40489]: 4.12 scrub ok 2024-09-21T14:04:13.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:13 smithi049 ceph-mon[40489]: from='client.16188 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T14:04:13.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:13 smithi049 ceph-mon[30080]: pgmap v777: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 6.1 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 52 MiB/s wr, 830 op/s 2024-09-21T14:04:13.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:13 smithi049 ceph-mon[30080]: 4.12 scrub starts 2024-09-21T14:04:13.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:13 smithi049 ceph-mon[30080]: 4.12 scrub ok 2024-09-21T14:04:13.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:13 smithi049 ceph-mon[30080]: from='client.16188 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T14:04:13.776 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:04:13.825 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:13.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:13.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:04:13.880 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 68 16 105227 105211 96.6797 68 0.0035639 0.0103373 2024-09-21T14:04:13.928 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:04:13.969 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:13.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:13.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:04:14.071 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:04:14.108 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:14.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:14.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:04:14.210 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:04:14.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:14.250 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:14.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:04:14.351 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:04:14.385 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:14.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:14.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:04:14.409 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T14:04:14.410 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:04:14.487 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:04:14.524 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:14.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:14.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:04:14.626 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:04:14.660 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:14.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:14.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:04:14.762 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:04:14.798 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:14.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:14.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:04:14.880 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 69 16 106340 106324 96.2867 69.5625 0.018042 0.0103807 2024-09-21T14:04:14.900 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:04:14.939 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:14.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:14.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:04:15.041 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:04:15.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:15.087 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:15.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:04:15.098 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T14:04:15.189 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:04:15.224 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:15.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:15.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:04:15.326 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:04:15.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:15.377 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:15.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:04:15.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:15 smithi187 ceph-mon[33032]: pgmap v778: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 76 MiB/s wr, 1.21k op/s 2024-09-21T14:04:15.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:15 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:04:15.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:15 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:04:15.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:15 smithi187 ceph-mon[33032]: 10.0 scrub ok 2024-09-21T14:04:15.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:15 smithi187 ceph-mon[33032]: 10.a scrub starts 2024-09-21T14:04:15.478 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:04:15.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:15 smithi049 ceph-mon[40489]: pgmap v778: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 76 MiB/s wr, 1.21k op/s 2024-09-21T14:04:15.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:15 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:04:15.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:15 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:04:15.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:15 smithi049 ceph-mon[40489]: 10.0 scrub ok 2024-09-21T14:04:15.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:15 smithi049 ceph-mon[40489]: 10.a scrub starts 2024-09-21T14:04:15.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:15 smithi049 ceph-mon[30080]: pgmap v778: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 76 MiB/s wr, 1.21k op/s 2024-09-21T14:04:15.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:15 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:04:15.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:15 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:04:15.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:15 smithi049 ceph-mon[30080]: 10.0 scrub ok 2024-09-21T14:04:15.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:15 smithi049 ceph-mon[30080]: 10.a scrub starts 2024-09-21T14:04:15.514 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:15.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:15.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:04:15.519 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:04:15.616 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:04:15.649 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:15.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:15.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:04:15.751 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:15.786 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:15.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:15.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:15.880 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 70 16 107373 107357 95.8335 64.5625 0.0107662 0.0104299 2024-09-21T14:04:15.888 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:04:15.928 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:15.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:15.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:04:16.030 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:04:16.062 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:16.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:16.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:04:16.155 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-09-21T14:04:16.164 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:04:16.168 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T14:04:16.168 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:04:16.197 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:16.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:16.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:04:16.298 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:04:16.341 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:16.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:16.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:04:16.444 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:04:16.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:16 smithi187 ceph-mon[33032]: 6.12 scrub starts 2024-09-21T14:04:16.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:16 smithi187 ceph-mon[33032]: 6.12 scrub ok 2024-09-21T14:04:16.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:16 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2588317084' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:04:16.480 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:16.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:16.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:04:16.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:16 smithi049 ceph-mon[40489]: 6.12 scrub starts 2024-09-21T14:04:16.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:16 smithi049 ceph-mon[40489]: 6.12 scrub ok 2024-09-21T14:04:16.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:16 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2588317084' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:04:16.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:16 smithi049 ceph-mon[30080]: 6.12 scrub starts 2024-09-21T14:04:16.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:16 smithi049 ceph-mon[30080]: 6.12 scrub ok 2024-09-21T14:04:16.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:16 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2588317084' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:04:16.582 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:04:16.605 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:16.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:16.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:04:16.708 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:04:16.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:16.746 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:16.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:04:16.847 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:04:16.880 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 71 16 108406 108390 95.393 64.5625 0.00852226 0.0104785 2024-09-21T14:04:16.883 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:16.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:16.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:04:16.919 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T14:04:16.976 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:04:16.985 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:16.997 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-09-21T14:04:16.997 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd reweight-by-utilization 110 3.0 2024-09-21T14:04:17.023 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:17.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:17.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:17.125 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:04:17.168 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:17.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:17.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:04:17.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:17 smithi049 ceph-mon[40489]: pgmap v779: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 533 B/s rd, 68 MiB/s wr, 1.08k op/s 2024-09-21T14:04:17.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:17 smithi049 ceph-mon[40489]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:04:17.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:17 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2588317084' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T14:04:17.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:17 smithi049 ceph-mon[40489]: osdmap e407: 8 total, 8 up, 8 in 2024-09-21T14:04:17.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:17 smithi049 ceph-mon[30080]: pgmap v779: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 533 B/s rd, 68 MiB/s wr, 1.08k op/s 2024-09-21T14:04:17.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:17 smithi049 ceph-mon[30080]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:04:17.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:17 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2588317084' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T14:04:17.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:17 smithi049 ceph-mon[30080]: osdmap e407: 8 total, 8 up, 8 in 2024-09-21T14:04:17.269 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:04:17.317 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:17.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:17.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:04:17.325 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:04:17.420 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:04:17.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:17 smithi187 ceph-mon[33032]: pgmap v779: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 533 B/s rd, 68 MiB/s wr, 1.08k op/s 2024-09-21T14:04:17.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:17 smithi187 ceph-mon[33032]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:04:17.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:17 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2588317084' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T14:04:17.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:17 smithi187 ceph-mon[33032]: osdmap e407: 8 total, 8 up, 8 in 2024-09-21T14:04:17.464 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:17.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:17.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:04:17.566 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:04:17.605 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:17.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:17.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:04:17.707 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:04:17.742 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:17.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:17.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:04:17.844 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:04:17.880 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 72 16 109559 109543 95.0689 72.0625 0.0170151 0.0105146 2024-09-21T14:04:17.884 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:17.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:17.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:04:17.971 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T14:04:17.971 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:04:17.987 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:18.030 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:18.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:18.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:18.111 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:04:18.111 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:04:18.131 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:04:18.151 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:04:18.151 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:04:18.172 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:18.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:18.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:04:18.194 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:04:18.194 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:04:18.229 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:04:18.229 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:04:18.234 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:04:18.263 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:04:18.263 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:04:18.274 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:04:18.299 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:04:18.299 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:04:18.304 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:18.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:18.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:04:18.340 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:04:18.340 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:04:18.371 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:18 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1314192511' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:04:18.372 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:04:18.372 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:04:18.372 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:18 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1314192511' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:04:18.405 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:04:18.438 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:18.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:18.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:04:18.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:18 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1314192511' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:04:18.541 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:04:18.573 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:18.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:18.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:04:18.674 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:04:18.678 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T14:04:18.715 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:18.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:18.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:04:18.816 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:04:18.868 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:18.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:18.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:04:18.881 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 73 16 110702 110686 94.7452 71.4375 0.00813819 0.0105499 2024-09-21T14:04:18.970 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:04:19.010 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:19.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:19.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:04:19.112 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:19.144 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:19.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:19.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:19.152 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:04:19.246 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:04:19.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:19.299 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:19.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:04:19.401 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:19.442 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:19.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:19.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:19.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:19 smithi187 ceph-mon[33032]: pgmap v781: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-09-21T14:04:19.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:19 smithi049 ceph-mon[30080]: pgmap v781: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-09-21T14:04:19.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:19 smithi049 ceph-mon[40489]: pgmap v781: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-09-21T14:04:19.544 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:19.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:19.583 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:19.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:19.684 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:04:19.719 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:19.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:19.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:04:19.805 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T14:04:19.805 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-21T14:04:19.805 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-21T14:04:19.805 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-21T14:04:19.805 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T14:04:19.822 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:04:19.849 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T14:04:19.851 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:19.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:19.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:04:19.881 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 74 16 111852 111836 94.4361 71.875 0.0133028 0.0105845 2024-09-21T14:04:19.953 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:04:19.990 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:19.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:19.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:04:20.092 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:20.134 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:20.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:20.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:20.159 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:04:19 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:04:20.179 INFO:teuthology.orchestra.run.smithi049.stdout:moved 38 / 474 (8.01688%) 2024-09-21T14:04:20.179 INFO:teuthology.orchestra.run.smithi049.stdout:avg 59.25 2024-09-21T14:04:20.179 INFO:teuthology.orchestra.run.smithi049.stdout:stddev 5.76086 -> 8.71421 (expected baseline 7.20026) 2024-09-21T14:04:20.179 INFO:teuthology.orchestra.run.smithi049.stdout:min osd.2 with 48 -> 61 pgs (0.810127 -> 1.02954 * mean) 2024-09-21T14:04:20.179 INFO:teuthology.orchestra.run.smithi049.stdout:max osd.4 with 68 -> 54 pgs (1.14768 -> 0.911392 * mean) 2024-09-21T14:04:20.180 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T14:04:20.180 INFO:teuthology.orchestra.run.smithi049.stdout:oload 110 2024-09-21T14:04:20.180 INFO:teuthology.orchestra.run.smithi049.stdout:max_change 3 2024-09-21T14:04:20.180 INFO:teuthology.orchestra.run.smithi049.stdout:max_change_osds 4 2024-09-21T14:04:20.180 INFO:teuthology.orchestra.run.smithi049.stdout:average_utilization 0.0319 2024-09-21T14:04:20.180 INFO:teuthology.orchestra.run.smithi049.stdout:overload_utilization 0.0351 2024-09-21T14:04:20.180 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0 weight 1.0000 -> 0.6177 2024-09-21T14:04:20.180 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4 weight 1.0000 -> 0.7889 2024-09-21T14:04:20.180 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6 weight 1.0000 -> 0.9076 2024-09-21T14:04:20.239 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:04:20.263 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:20.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:20.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:04:20.365 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:04:20.385 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:20.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:20.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:04:20.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:20 smithi187 ceph-mon[33032]: from='client.16206 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T14:04:20.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:20 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'40481','4':'51701','6':'59481'}"} : dispatch 2024-09-21T14:04:20.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:20 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:04:20.488 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:04:20.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:20 smithi049 ceph-mon[40489]: from='client.16206 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T14:04:20.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:20 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'40481','4':'51701','6':'59481'}"} : dispatch 2024-09-21T14:04:20.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:20 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:04:20.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:20 smithi049 ceph-mon[30080]: from='client.16206 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T14:04:20.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:20 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'40481','4':'51701','6':'59481'}"} : dispatch 2024-09-21T14:04:20.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:20 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:04:20.509 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:20.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:20.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:04:20.515 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T14:04:20.611 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:04:20.715 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:20.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:20.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:04:20.792 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:04:20 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:04:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:04:20.802 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:04:20.817 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:04:20.819 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-21T14:04:20.819 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd reweight-by-utilization 1000 3.0 2024-09-21T14:04:20.842 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:20.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:20.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:04:20.881 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 75 16 112262 112246 93.5187 25.625 0.00406973 0.0105988 2024-09-21T14:04:20.900 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:04:20.943 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:04:20.968 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:20.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:20.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:04:21.070 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:04:21.093 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:21.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:21.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:04:21.195 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:04:21.232 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:21.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:21.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:04:21.309 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T14:04:21.309 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-21T14:04:21.309 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-21T14:04:21.309 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-21T14:04:21.309 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.309 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187592 9.5 9:a5cf25e0:::benchmark_data_smithi049_138504_object66184:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:04:21.309 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:41.268428+0000", 2024-09-21T14:04:21.310 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 160.03963189000001, 2024-09-21T14:04:21.310 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3375785200000001, 2024-09-21T14:04:21.310 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:04:21.310 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:04:21.310 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:21.310 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:04:21.310 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:04:21.310 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:04:21.310 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187592 2024-09-21T14:04:21.310 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.310 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:04:21.310 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.310 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:04:21.310 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268428+0000", 2024-09-21T14:04:21.311 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:04:21.311 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.311 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.311 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:04:21.311 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268428+0000", 2024-09-21T14:04:21.311 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:04:21.311 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.311 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.311 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:04:21.311 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268429+0000", 2024-09-21T14:04:21.311 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.466e-06 2024-09-21T14:04:21.311 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.311 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.311 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:04:21.312 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268435+0000", 2024-09-21T14:04:21.312 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5539999999999997e-06 2024-09-21T14:04:21.312 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.312 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.312 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:04:21.312 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268445+0000", 2024-09-21T14:04:21.312 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0366999999999999e-05 2024-09-21T14:04:21.312 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.312 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.312 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:04:21.312 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268452+0000", 2024-09-21T14:04:21.312 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8870000000000003e-06 2024-09-21T14:04:21.312 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.313 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.313 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:04:21.313 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.289931+0000", 2024-09-21T14:04:21.313 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.021479166000000001 2024-09-21T14:04:21.313 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.313 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.313 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:04:21.313 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.289937+0000", 2024-09-21T14:04:21.313 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4140000000000002e-06 2024-09-21T14:04:21.313 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.313 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.314 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:04:21.314 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.566606+0000", 2024-09-21T14:04:21.314 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2766695319999997 2024-09-21T14:04:21.314 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.314 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.314 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:04:21.314 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.569571+0000", 2024-09-21T14:04:21.314 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0029643489999999998 2024-09-21T14:04:21.314 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.314 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.314 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:04:21.314 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.573938+0000", 2024-09-21T14:04:21.314 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0043675099999999998 2024-09-21T14:04:21.314 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.315 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.315 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:04:21.315 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.589876+0000", 2024-09-21T14:04:21.315 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.015938249000000002 2024-09-21T14:04:21.315 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.315 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.315 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:21.315 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.597274+0000", 2024-09-21T14:04:21.315 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0073979509999999998 2024-09-21T14:04:21.315 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.315 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.315 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:21.315 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.598324+0000", 2024-09-21T14:04:21.315 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001050156 2024-09-21T14:04:21.316 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.316 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.316 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:04:21.316 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.599807+0000", 2024-09-21T14:04:21.316 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0014825439999999999 2024-09-21T14:04:21.316 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.316 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.316 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:04:21.316 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.606006+0000", 2024-09-21T14:04:21.316 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.006199375 2024-09-21T14:04:21.316 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:04:21.316 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:04:21.316 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:04:21.317 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.317 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.317 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187594 9.5 9:a4584f0e:::benchmark_data_smithi049_138504_object66186:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:04:21.317 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:41.268527+0000", 2024-09-21T14:04:21.317 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 160.03953237499999, 2024-09-21T14:04:21.317 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3614060480000001, 2024-09-21T14:04:21.317 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:04:21.317 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:04:21.317 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:21.317 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:04:21.317 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:04:21.317 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:04:21.317 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187594 2024-09-21T14:04:21.317 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.318 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:04:21.318 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.318 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:04:21.318 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268527+0000", 2024-09-21T14:04:21.318 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:04:21.318 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.318 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.318 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:04:21.318 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268527+0000", 2024-09-21T14:04:21.318 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:04:21.318 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.318 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.318 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:04:21.318 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268528+0000", 2024-09-21T14:04:21.319 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5299999999999999e-07 2024-09-21T14:04:21.319 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.319 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.319 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:04:21.319 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268531+0000", 2024-09-21T14:04:21.319 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.157e-06 2024-09-21T14:04:21.319 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.319 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.319 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:04:21.319 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268538+0000", 2024-09-21T14:04:21.319 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9290000000000001e-06 2024-09-21T14:04:21.319 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.319 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.319 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:04:21.320 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268543+0000", 2024-09-21T14:04:21.320 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0409999999999997e-06 2024-09-21T14:04:21.320 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.320 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.320 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:04:21.320 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.289953+0000", 2024-09-21T14:04:21.320 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.021409811000000001 2024-09-21T14:04:21.320 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.320 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.320 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:04:21.320 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.289957+0000", 2024-09-21T14:04:21.320 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9489999999999998e-06 2024-09-21T14:04:21.320 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.320 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.321 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:04:21.321 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.606351+0000", 2024-09-21T14:04:21.321 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3163947599999997 2024-09-21T14:04:21.321 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.321 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.321 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:04:21.321 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.610368+0000", 2024-09-21T14:04:21.321 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0040163289999999999 2024-09-21T14:04:21.321 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.321 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.321 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:04:21.321 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.615560+0000", 2024-09-21T14:04:21.321 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0051918939999999998 2024-09-21T14:04:21.321 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.321 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.322 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:21.322 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.622425+0000", 2024-09-21T14:04:21.322 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0068655959999999999 2024-09-21T14:04:21.322 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.322 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.322 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:21.322 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.623509+0000", 2024-09-21T14:04:21.322 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001083669 2024-09-21T14:04:21.322 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.322 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.322 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:04:21.322 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.624009+0000", 2024-09-21T14:04:21.322 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00050001600000000005 2024-09-21T14:04:21.322 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.323 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.323 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:04:21.323 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.625282+0000", 2024-09-21T14:04:21.323 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0012730599999999999 2024-09-21T14:04:21.323 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.323 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.324 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:04:21.324 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.629933+0000", 2024-09-21T14:04:21.324 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0046513839999999997 2024-09-21T14:04:21.324 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:04:21.324 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:04:21.324 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:04:21.324 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.324 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.324 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187607 9.7 9:ea996e28:::benchmark_data_smithi049_138504_object66199:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:04:21.324 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:41.286116+0000", 2024-09-21T14:04:21.324 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 160.021943725, 2024-09-21T14:04:21.325 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0827087349999998, 2024-09-21T14:04:21.325 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:04:21.325 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:04:21.325 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:21.325 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:04:21.325 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:04:21.325 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:04:21.325 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187607 2024-09-21T14:04:21.325 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.325 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:04:21.325 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.325 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:04:21.325 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.286116+0000", 2024-09-21T14:04:21.325 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:04:21.325 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.326 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.326 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:04:21.326 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.286116+0000", 2024-09-21T14:04:21.326 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:04:21.326 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.326 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.326 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:04:21.326 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.286117+0000", 2024-09-21T14:04:21.326 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.097e-06 2024-09-21T14:04:21.326 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.326 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.326 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:04:21.326 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.286120+0000", 2024-09-21T14:04:21.326 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.224e-06 2024-09-21T14:04:21.327 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.327 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.327 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:04:21.327 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.286129+0000", 2024-09-21T14:04:21.327 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3659999999999992e-06 2024-09-21T14:04:21.327 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.327 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.327 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:04:21.327 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.286134+0000", 2024-09-21T14:04:21.327 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6130000000000003e-06 2024-09-21T14:04:21.327 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.327 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.327 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:04:21.327 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.286627+0000", 2024-09-21T14:04:21.328 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00049266000000000002 2024-09-21T14:04:21.328 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.328 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.328 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:04:21.328 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.286663+0000", 2024-09-21T14:04:21.328 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6164999999999997e-05 2024-09-21T14:04:21.328 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.328 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.328 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:04:21.328 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.330333+0000", 2024-09-21T14:04:21.328 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0436700029999999 2024-09-21T14:04:21.328 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.328 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.328 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:04:21.329 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.333462+0000", 2024-09-21T14:04:21.329 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003128881 2024-09-21T14:04:21.329 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.329 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.329 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,7", 2024-09-21T14:04:21.329 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.337950+0000", 2024-09-21T14:04:21.329 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0044878089999999997 2024-09-21T14:04:21.329 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.329 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.329 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:04:21.329 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353871+0000", 2024-09-21T14:04:21.329 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.015921270000000001 2024-09-21T14:04:21.329 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.329 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.329 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:21.330 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.360705+0000", 2024-09-21T14:04:21.330 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0068337110000000001 2024-09-21T14:04:21.330 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.330 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.330 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:21.330 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.362017+0000", 2024-09-21T14:04:21.330 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0013119749999999999 2024-09-21T14:04:21.330 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.330 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.330 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:04:21.330 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.364166+0000", 2024-09-21T14:04:21.330 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0021495030000000001 2024-09-21T14:04:21.330 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.330 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.331 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:04:21.331 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.368825+0000", 2024-09-21T14:04:21.331 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0046584579999999999 2024-09-21T14:04:21.331 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:04:21.331 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:04:21.331 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:04:21.331 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.331 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.331 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187609 9.5 9:a0a81d79:::benchmark_data_smithi049_138504_object66201:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:04:21.331 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:41.287003+0000", 2024-09-21T14:04:21.331 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 160.021057114, 2024-09-21T14:04:21.331 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3857519199999997, 2024-09-21T14:04:21.331 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:04:21.332 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:04:21.332 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:21.332 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:04:21.332 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:04:21.332 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:04:21.332 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187609 2024-09-21T14:04:21.332 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.332 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:04:21.332 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.332 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:04:21.332 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.287003+0000", 2024-09-21T14:04:21.332 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:04:21.332 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.332 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.332 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:04:21.333 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.287003+0000", 2024-09-21T14:04:21.333 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:04:21.333 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.333 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.333 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:04:21.333 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.287003+0000", 2024-09-21T14:04:21.333 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8400000000000003e-07 2024-09-21T14:04:21.333 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.333 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.333 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:04:21.333 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.287017+0000", 2024-09-21T14:04:21.333 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3831e-05 2024-09-21T14:04:21.333 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.333 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.333 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:04:21.334 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.287026+0000", 2024-09-21T14:04:21.334 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8389999999999996e-06 2024-09-21T14:04:21.334 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.334 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.334 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:04:21.334 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.287032+0000", 2024-09-21T14:04:21.334 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3509999999999998e-06 2024-09-21T14:04:21.334 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.334 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.334 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:04:21.334 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.289969+0000", 2024-09-21T14:04:21.334 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0029364399999999998 2024-09-21T14:04:21.334 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.334 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.335 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:04:21.335 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.289973+0000", 2024-09-21T14:04:21.335 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6339999999999999e-06 2024-09-21T14:04:21.335 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.335 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.335 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:04:21.335 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.631003+0000", 2024-09-21T14:04:21.335 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3410304320000002 2024-09-21T14:04:21.335 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.335 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.335 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:04:21.335 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.634397+0000", 2024-09-21T14:04:21.335 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0033938369999999998 2024-09-21T14:04:21.335 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.335 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.336 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:04:21.336 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.639017+0000", 2024-09-21T14:04:21.336 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0046205889999999996 2024-09-21T14:04:21.336 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.336 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.336 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:04:21.336 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.662137+0000", 2024-09-21T14:04:21.336 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.023119475 2024-09-21T14:04:21.336 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.336 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.336 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:21.336 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.663533+0000", 2024-09-21T14:04:21.336 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0013956159999999999 2024-09-21T14:04:21.336 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.336 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.337 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:21.337 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.664730+0000", 2024-09-21T14:04:21.337 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0011970139999999999 2024-09-21T14:04:21.337 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.337 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.337 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:04:21.337 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.666523+0000", 2024-09-21T14:04:21.337 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001793046 2024-09-21T14:04:21.337 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.337 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.337 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:04:21.337 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.672754+0000", 2024-09-21T14:04:21.337 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0062319319999999999 2024-09-21T14:04:21.337 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:04:21.338 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:04:21.338 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:04:21.338 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.338 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.338 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187621 9.5 9:a3e366d7:::benchmark_data_smithi049_138504_object66213:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:04:21.338 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:41.300495+0000", 2024-09-21T14:04:21.338 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 160.00756446599999, 2024-09-21T14:04:21.338 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3801383449999998, 2024-09-21T14:04:21.338 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:04:21.338 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:04:21.338 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:21.338 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:04:21.338 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:04:21.338 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:04:21.339 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187621 2024-09-21T14:04:21.339 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.339 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:04:21.339 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.339 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:04:21.339 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300495+0000", 2024-09-21T14:04:21.339 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:04:21.339 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.339 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.339 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:04:21.339 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300495+0000", 2024-09-21T14:04:21.339 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:04:21.339 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.339 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.339 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:04:21.340 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300497+0000", 2024-09-21T14:04:21.340 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3710000000000001e-06 2024-09-21T14:04:21.340 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.340 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.340 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:04:21.340 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300502+0000", 2024-09-21T14:04:21.340 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3569999999999997e-06 2024-09-21T14:04:21.340 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.340 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.340 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:04:21.340 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300511+0000", 2024-09-21T14:04:21.340 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.9509999999999995e-06 2024-09-21T14:04:21.340 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.340 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.340 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:04:21.341 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300517+0000", 2024-09-21T14:04:21.341 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2489999999999998e-06 2024-09-21T14:04:21.341 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.341 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.341 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:04:21.341 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300606+0000", 2024-09-21T14:04:21.341 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8794000000000004e-05 2024-09-21T14:04:21.341 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.341 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.341 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:04:21.341 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300616+0000", 2024-09-21T14:04:21.341 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0338e-05 2024-09-21T14:04:21.341 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.341 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.342 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:04:21.342 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.643702+0000", 2024-09-21T14:04:21.342 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.343085759 2024-09-21T14:04:21.342 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.342 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.342 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:04:21.342 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.648030+0000", 2024-09-21T14:04:21.342 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0043275859999999996 2024-09-21T14:04:21.342 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.342 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.342 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:04:21.342 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.653318+0000", 2024-09-21T14:04:21.342 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.005287936 2024-09-21T14:04:21.342 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.343 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.343 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:04:21.343 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.662422+0000", 2024-09-21T14:04:21.343 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0091040340000000004 2024-09-21T14:04:21.343 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.343 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.343 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:21.343 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.673842+0000", 2024-09-21T14:04:21.343 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.011420382999999999 2024-09-21T14:04:21.343 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.343 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.343 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:21.343 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.675052+0000", 2024-09-21T14:04:21.343 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0012104290000000001 2024-09-21T14:04:21.344 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.344 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.344 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:04:21.344 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.676489+0000", 2024-09-21T14:04:21.344 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0014364309999999999 2024-09-21T14:04:21.344 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.344 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.344 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:04:21.344 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.680634+0000", 2024-09-21T14:04:21.344 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.004144727 2024-09-21T14:04:21.344 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:04:21.344 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:04:21.344 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:04:21.344 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.344 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.345 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187623 9.7 9:e79a683b:::benchmark_data_smithi049_138504_object66215:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:04:21.345 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:41.300818+0000", 2024-09-21T14:04:21.345 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 160.00724206199999, 2024-09-21T14:04:21.345 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1014428919999997, 2024-09-21T14:04:21.345 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:04:21.345 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:04:21.345 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:21.345 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:04:21.345 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:04:21.345 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:04:21.345 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187623 2024-09-21T14:04:21.345 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.345 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:04:21.345 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.345 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:04:21.346 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300818+0000", 2024-09-21T14:04:21.346 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:04:21.346 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.346 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.346 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:04:21.346 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300818+0000", 2024-09-21T14:04:21.346 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:04:21.346 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.346 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.346 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:04:21.346 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300818+0000", 2024-09-21T14:04:21.346 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.54e-07 2024-09-21T14:04:21.346 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.346 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.347 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:04:21.347 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300822+0000", 2024-09-21T14:04:21.347 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6770000000000001e-06 2024-09-21T14:04:21.347 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.347 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.347 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:04:21.347 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300832+0000", 2024-09-21T14:04:21.347 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0202e-05 2024-09-21T14:04:21.347 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.347 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.347 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:04:21.347 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300847+0000", 2024-09-21T14:04:21.347 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4992e-05 2024-09-21T14:04:21.347 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.348 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.348 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:04:21.348 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300931+0000", 2024-09-21T14:04:21.348 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3577000000000003e-05 2024-09-21T14:04:21.348 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.348 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.348 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:04:21.348 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300940+0000", 2024-09-21T14:04:21.348 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8009999999999993e-06 2024-09-21T14:04:21.348 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.348 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.348 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:04:21.348 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.370531+0000", 2024-09-21T14:04:21.348 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0695910870000001 2024-09-21T14:04:21.349 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.349 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.349 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:04:21.349 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.373842+0000", 2024-09-21T14:04:21.349 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0033110259999999999 2024-09-21T14:04:21.349 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.349 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.349 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,7", 2024-09-21T14:04:21.349 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.378337+0000", 2024-09-21T14:04:21.349 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0044951100000000001 2024-09-21T14:04:21.349 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.349 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.349 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:21.349 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.394514+0000", 2024-09-21T14:04:21.349 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.016176877999999999 2024-09-21T14:04:21.350 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.350 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.350 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:04:21.350 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.395030+0000", 2024-09-21T14:04:21.350 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00051615000000000005 2024-09-21T14:04:21.350 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.350 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.350 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:21.350 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.395803+0000", 2024-09-21T14:04:21.350 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00077358799999999997 2024-09-21T14:04:21.350 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.350 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.350 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:04:21.350 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.397305+0000", 2024-09-21T14:04:21.351 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001501811 2024-09-21T14:04:21.351 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.351 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.351 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:04:21.351 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.402261+0000", 2024-09-21T14:04:21.351 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0049552390000000002 2024-09-21T14:04:21.351 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:04:21.351 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:04:21.351 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:04:21.351 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.351 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.352 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187624 9.5 9:a181e4d0:::benchmark_data_smithi049_138504_object66216:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:04:21.352 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:41.300871+0000", 2024-09-21T14:04:21.352 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 160.00718905900001, 2024-09-21T14:04:21.352 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4164464030000001, 2024-09-21T14:04:21.352 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:04:21.352 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:04:21.352 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:21.352 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:04:21.352 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:04:21.352 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:04:21.352 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187624 2024-09-21T14:04:21.352 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.352 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:04:21.352 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.353 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:04:21.353 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300871+0000", 2024-09-21T14:04:21.353 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:04:21.353 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.353 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.353 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:04:21.353 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300871+0000", 2024-09-21T14:04:21.353 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:04:21.353 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.353 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.353 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:04:21.353 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300871+0000", 2024-09-21T14:04:21.353 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8599999999999998e-07 2024-09-21T14:04:21.353 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.354 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.354 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:04:21.354 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300875+0000", 2024-09-21T14:04:21.354 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7720000000000001e-06 2024-09-21T14:04:21.354 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.354 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.354 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:04:21.354 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300886+0000", 2024-09-21T14:04:21.354 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1058e-05 2024-09-21T14:04:21.354 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.354 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.354 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:04:21.354 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300892+0000", 2024-09-21T14:04:21.354 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.161e-06 2024-09-21T14:04:21.355 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.355 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.355 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:04:21.355 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.301287+0000", 2024-09-21T14:04:21.355 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00039522800000000002 2024-09-21T14:04:21.355 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.355 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.355 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:04:21.355 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.301297+0000", 2024-09-21T14:04:21.355 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6209999999999999e-06 2024-09-21T14:04:21.355 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.355 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.355 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:04:21.356 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.681021+0000", 2024-09-21T14:04:21.356 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3797241769999999 2024-09-21T14:04:21.356 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.356 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.356 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:04:21.356 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.685054+0000", 2024-09-21T14:04:21.356 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0040323909999999998 2024-09-21T14:04:21.356 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.356 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.356 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:04:21.356 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.689479+0000", 2024-09-21T14:04:21.356 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0044253740000000001 2024-09-21T14:04:21.356 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.356 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.357 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:21.357 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.710060+0000", 2024-09-21T14:04:21.357 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.020581051999999999 2024-09-21T14:04:21.357 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.357 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.357 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:04:21.357 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.710641+0000", 2024-09-21T14:04:21.357 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00058055800000000001 2024-09-21T14:04:21.357 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.357 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.357 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:21.357 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.711399+0000", 2024-09-21T14:04:21.357 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00075836299999999996 2024-09-21T14:04:21.357 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.358 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.358 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:04:21.358 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.712857+0000", 2024-09-21T14:04:21.358 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001458373 2024-09-21T14:04:21.358 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.358 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.358 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:04:21.358 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.717317+0000", 2024-09-21T14:04:21.358 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0044596890000000002 2024-09-21T14:04:21.358 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:04:21.358 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:04:21.358 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:04:21.358 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.358 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.358 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187626 9.7 9:e3df99cb:::benchmark_data_smithi049_138504_object66218:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:04:21.359 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:41.304965+0000", 2024-09-21T14:04:21.359 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 160.00309492299999, 2024-09-21T14:04:21.359 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1061633610000001, 2024-09-21T14:04:21.359 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:04:21.359 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:04:21.359 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:21.359 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:04:21.359 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:04:21.359 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:04:21.359 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187626 2024-09-21T14:04:21.359 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.359 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:04:21.359 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.359 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:04:21.360 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.304965+0000", 2024-09-21T14:04:21.360 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:04:21.360 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.360 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.360 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:04:21.360 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.304965+0000", 2024-09-21T14:04:21.360 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:04:21.360 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.360 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.360 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:04:21.360 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.304966+0000", 2024-09-21T14:04:21.360 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6619999999999999e-06 2024-09-21T14:04:21.360 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.361 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.361 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:04:21.361 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.304973+0000", 2024-09-21T14:04:21.361 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1759999999999998e-06 2024-09-21T14:04:21.361 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.361 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.361 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:04:21.361 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.304987+0000", 2024-09-21T14:04:21.361 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4372999999999999e-05 2024-09-21T14:04:21.361 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.361 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.361 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:04:21.361 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.304995+0000", 2024-09-21T14:04:21.361 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4109999999999996e-06 2024-09-21T14:04:21.362 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.362 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.362 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:04:21.362 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.305067+0000", 2024-09-21T14:04:21.362 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1833000000000002e-05 2024-09-21T14:04:21.362 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.362 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.362 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:04:21.362 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.305077+0000", 2024-09-21T14:04:21.362 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0161e-05 2024-09-21T14:04:21.362 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.362 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.362 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:04:21.362 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.382147+0000", 2024-09-21T14:04:21.363 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0770691709999998 2024-09-21T14:04:21.363 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.363 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.363 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:04:21.363 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.385209+0000", 2024-09-21T14:04:21.363 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0030624139999999998 2024-09-21T14:04:21.363 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.363 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.363 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,7", 2024-09-21T14:04:21.363 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.389597+0000", 2024-09-21T14:04:21.363 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.004388477 2024-09-21T14:04:21.363 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.363 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.363 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:21.364 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.403080+0000", 2024-09-21T14:04:21.364 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.013482984 2024-09-21T14:04:21.364 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.364 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.364 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:04:21.364 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.403583+0000", 2024-09-21T14:04:21.364 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00050300099999999999 2024-09-21T14:04:21.364 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.364 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:21.365 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:21 smithi049 ceph-mon[40489]: pgmap v782: 158 pgs: 158 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-09-21T14:04:21.365 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:21 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'40481','4':'51701','6':'59481'}"}]': finished 2024-09-21T14:04:21.365 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:21 smithi049 ceph-mon[40489]: osdmap e408: 8 total, 8 up, 8 in 2024-09-21T14:04:21.366 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:21 smithi049 ceph-mon[30080]: pgmap v782: 158 pgs: 158 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-09-21T14:04:21.366 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:21 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'40481','4':'51701','6':'59481'}"}]': finished 2024-09-21T14:04:21.366 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:21 smithi049 ceph-mon[30080]: osdmap e408: 8 total, 8 up, 8 in 2024-09-21T14:04:21.367 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.367 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:21.367 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.404464+0000", 2024-09-21T14:04:21.367 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00088034599999999995 2024-09-21T14:04:21.367 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.367 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.367 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:04:21.367 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.406114+0000", 2024-09-21T14:04:21.368 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0016504499999999999 2024-09-21T14:04:21.368 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.368 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.368 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:04:21.368 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.411128+0000", 2024-09-21T14:04:21.368 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0050139019999999998 2024-09-21T14:04:21.368 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:04:21.368 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:04:21.368 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:04:21.368 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.368 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.368 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187630 9.5 9:afb0e788:::benchmark_data_smithi049_138504_object66222:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:04:21.368 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:41.309934+0000", 2024-09-21T14:04:21.369 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 159.99812534200001, 2024-09-21T14:04:21.369 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4164949509999998, 2024-09-21T14:04:21.369 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:04:21.369 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:04:21.369 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:21.369 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:04:21.369 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:04:21.369 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:04:21.369 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187630 2024-09-21T14:04:21.369 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.369 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:04:21.369 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.369 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:04:21.369 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.309934+0000", 2024-09-21T14:04:21.369 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:04:21.369 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.370 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.370 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:04:21.370 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.309934+0000", 2024-09-21T14:04:21.370 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:04:21.370 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.370 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.370 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:04:21.370 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.309936+0000", 2024-09-21T14:04:21.370 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7460000000000001e-06 2024-09-21T14:04:21.370 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.370 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.370 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:04:21.370 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.309943+0000", 2024-09-21T14:04:21.371 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6579999999999996e-06 2024-09-21T14:04:21.371 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.371 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.371 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:04:21.371 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.309957+0000", 2024-09-21T14:04:21.371 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4496e-05 2024-09-21T14:04:21.371 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.371 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.371 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:04:21.371 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.309964+0000", 2024-09-21T14:04:21.371 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2219999999999996e-06 2024-09-21T14:04:21.371 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.371 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.371 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:04:21.372 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.310043+0000", 2024-09-21T14:04:21.372 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8801999999999995e-05 2024-09-21T14:04:21.372 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.372 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.372 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:04:21.372 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.310055+0000", 2024-09-21T14:04:21.372 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1637999999999999e-05 2024-09-21T14:04:21.372 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.372 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.372 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:04:21.372 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.693253+0000", 2024-09-21T14:04:21.372 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3831976189999997 2024-09-21T14:04:21.372 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.372 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.373 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:04:21.373 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.697465+0000", 2024-09-21T14:04:21.373 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0042119770000000004 2024-09-21T14:04:21.373 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.373 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.373 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:04:21.373 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.703657+0000", 2024-09-21T14:04:21.373 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0061928870000000002 2024-09-21T14:04:21.373 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.373 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.373 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:04:21.373 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.717616+0000", 2024-09-21T14:04:21.373 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.013958435999999999 2024-09-21T14:04:21.373 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.374 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.374 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:21.374 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.718555+0000", 2024-09-21T14:04:21.374 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00093888999999999995 2024-09-21T14:04:21.374 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.374 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.374 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:21.374 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.719633+0000", 2024-09-21T14:04:21.374 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0010787150000000001 2024-09-21T14:04:21.374 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.374 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.374 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:04:21.374 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.721631+0000", 2024-09-21T14:04:21.374 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0019973220000000002 2024-09-21T14:04:21.375 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.375 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.375 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:04:21.375 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.726429+0000", 2024-09-21T14:04:21.375 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0047985429999999997 2024-09-21T14:04:21.375 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:04:21.375 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:04:21.375 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:04:21.375 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.375 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.375 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187631 9.7 9:eb4378cd:::benchmark_data_smithi049_138504_object66223:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:04:21.375 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:41.310203+0000", 2024-09-21T14:04:21.375 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 159.997856834, 2024-09-21T14:04:21.375 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1434362220000001, 2024-09-21T14:04:21.375 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:04:21.376 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:04:21.376 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:21.376 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:04:21.376 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:04:21.376 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:04:21.376 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187631 2024-09-21T14:04:21.376 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.376 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:04:21.376 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.376 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:04:21.376 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.310203+0000", 2024-09-21T14:04:21.376 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:04:21.376 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.376 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.377 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:04:21.377 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.310203+0000", 2024-09-21T14:04:21.377 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:04:21.377 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.377 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.377 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:04:21.377 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.310204+0000", 2024-09-21T14:04:21.377 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3699999999999999e-07 2024-09-21T14:04:21.377 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.377 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.377 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:04:21.377 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.310209+0000", 2024-09-21T14:04:21.377 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7039999999999999e-06 2024-09-21T14:04:21.377 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.377 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.378 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:04:21.378 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.310219+0000", 2024-09-21T14:04:21.378 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7319999999999993e-06 2024-09-21T14:04:21.378 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.378 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.378 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:04:21.378 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.310226+0000", 2024-09-21T14:04:21.378 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4239999999999998e-06 2024-09-21T14:04:21.378 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.378 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.378 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:04:21.378 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.310279+0000", 2024-09-21T14:04:21.378 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3124999999999997e-05 2024-09-21T14:04:21.378 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.379 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.379 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:04:21.379 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.310288+0000", 2024-09-21T14:04:21.379 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3149999999999998e-06 2024-09-21T14:04:21.379 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.379 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.379 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:04:21.379 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.416040+0000", 2024-09-21T14:04:21.379 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1057518990000004 2024-09-21T14:04:21.379 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.379 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.379 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:04:21.379 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.419285+0000", 2024-09-21T14:04:21.379 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0032447399999999999 2024-09-21T14:04:21.380 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.380 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.380 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,7", 2024-09-21T14:04:21.380 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.423815+0000", 2024-09-21T14:04:21.380 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0045301509999999996 2024-09-21T14:04:21.380 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.380 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.380 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:21.380 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.439415+0000", 2024-09-21T14:04:21.380 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.015600334 2024-09-21T14:04:21.380 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.380 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.380 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:21.380 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.440566+0000", 2024-09-21T14:04:21.381 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0011505129999999999 2024-09-21T14:04:21.381 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.381 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.381 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:04:21.381 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.447676+0000", 2024-09-21T14:04:21.381 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0071103659999999999 2024-09-21T14:04:21.381 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.381 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.381 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:04:21.381 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.448927+0000", 2024-09-21T14:04:21.381 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0012508759999999999 2024-09-21T14:04:21.381 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.381 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.381 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:04:21.381 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.453639+0000", 2024-09-21T14:04:21.382 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0047121059999999998 2024-09-21T14:04:21.382 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:04:21.382 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:04:21.382 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:04:21.382 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.382 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.382 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187650 9.5 9:ab0f6d84:::benchmark_data_smithi049_138504_object66242:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:04:21.382 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:41.359224+0000", 2024-09-21T14:04:21.382 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 159.94883533300001, 2024-09-21T14:04:21.382 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4044763060000003, 2024-09-21T14:04:21.382 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:04:21.382 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:04:21.382 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:21.383 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:04:21.383 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:04:21.383 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:04:21.383 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187650 2024-09-21T14:04:21.383 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.383 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:04:21.383 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.383 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:04:21.383 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.359224+0000", 2024-09-21T14:04:21.383 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:04:21.383 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.383 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.383 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:04:21.383 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.359224+0000", 2024-09-21T14:04:21.384 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:04:21.384 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.384 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.384 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:04:21.384 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.359226+0000", 2024-09-21T14:04:21.384 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.423e-06 2024-09-21T14:04:21.384 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.384 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.384 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:04:21.384 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.359231+0000", 2024-09-21T14:04:21.384 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0980000000000001e-06 2024-09-21T14:04:21.384 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.384 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.384 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:04:21.385 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.359243+0000", 2024-09-21T14:04:21.385 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2316e-05 2024-09-21T14:04:21.385 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.385 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.385 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:04:21.385 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.359250+0000", 2024-09-21T14:04:21.385 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8820000000000003e-06 2024-09-21T14:04:21.385 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.385 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.385 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:04:21.386 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.359329+0000", 2024-09-21T14:04:21.386 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8541000000000006e-05 2024-09-21T14:04:21.386 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.386 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.386 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:04:21.386 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.359341+0000", 2024-09-21T14:04:21.386 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2238e-05 2024-09-21T14:04:21.386 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.386 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.386 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:04:21.386 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.726850+0000", 2024-09-21T14:04:21.386 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3675093619999998 2024-09-21T14:04:21.386 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.386 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.387 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:04:21.387 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.731153+0000", 2024-09-21T14:04:21.387 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0043027860000000003 2024-09-21T14:04:21.387 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.387 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.387 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:04:21.387 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.737448+0000", 2024-09-21T14:04:21.387 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0062951559999999997 2024-09-21T14:04:21.387 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.387 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.387 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:21.387 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.754776+0000", 2024-09-21T14:04:21.387 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.017328300000000001 2024-09-21T14:04:21.387 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.388 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.388 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:21.388 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.756004+0000", 2024-09-21T14:04:21.388 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001227337 2024-09-21T14:04:21.388 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.388 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.388 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:04:21.388 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.757990+0000", 2024-09-21T14:04:21.388 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0019857809999999998 2024-09-21T14:04:21.388 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.388 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.388 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:04:21.388 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.759321+0000", 2024-09-21T14:04:21.388 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001331203 2024-09-21T14:04:21.389 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.389 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.389 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:04:21.389 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.763701+0000", 2024-09-21T14:04:21.389 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0043798830000000002 2024-09-21T14:04:21.389 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:04:21.389 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:04:21.389 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:04:21.389 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.389 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.389 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187651 9.f 9:fed2c87a:::benchmark_data_smithi049_138504_object66243:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e347)", 2024-09-21T14:04:21.389 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:42.222057+0000", 2024-09-21T14:04:21.389 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 159.08600220400001, 2024-09-21T14:04:21.389 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1315866940000001, 2024-09-21T14:04:21.390 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:04:21.390 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:04:21.390 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:21.390 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:04:21.390 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:04:21.390 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:04:21.390 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187651 2024-09-21T14:04:21.390 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.390 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:04:21.390 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.390 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:04:21.390 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.222057+0000", 2024-09-21T14:04:21.390 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:04:21.390 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.391 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.391 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:04:21.391 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.222057+0000", 2024-09-21T14:04:21.391 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:04:21.391 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.391 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.391 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:04:21.391 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.222059+0000", 2024-09-21T14:04:21.391 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8029999999999999e-06 2024-09-21T14:04:21.391 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.391 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.391 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:04:21.391 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.222066+0000", 2024-09-21T14:04:21.391 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2670000000000003e-06 2024-09-21T14:04:21.392 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.392 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.392 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:04:21.392 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.222078+0000", 2024-09-21T14:04:21.392 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2899e-05 2024-09-21T14:04:21.392 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.392 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.392 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:04:21.392 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.222086+0000", 2024-09-21T14:04:21.392 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8080000000000005e-06 2024-09-21T14:04:21.392 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.392 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.392 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:04:21.392 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.233188+0000", 2024-09-21T14:04:21.393 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0111019990000001 2024-09-21T14:04:21.393 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.393 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.393 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:04:21.393 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.233317+0000", 2024-09-21T14:04:21.393 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012857999999999999 2024-09-21T14:04:21.393 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.393 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.393 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3", 2024-09-21T14:04:21.393 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.233386+0000", 2024-09-21T14:04:21.393 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8901999999999999e-05 2024-09-21T14:04:21.393 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.393 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.393 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:04:21.393 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.342687+0000", 2024-09-21T14:04:21.394 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.109301519 2024-09-21T14:04:21.394 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.394 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.394 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:21.394 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.342738+0000", 2024-09-21T14:04:21.394 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0756e-05 2024-09-21T14:04:21.394 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.394 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.394 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:21.394 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353571+0000", 2024-09-21T14:04:21.394 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.01083253 2024-09-21T14:04:21.394 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.394 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.394 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:04:21.395 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353614+0000", 2024-09-21T14:04:21.395 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3129e-05 2024-09-21T14:04:21.395 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.395 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.395 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:04:21.395 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353644+0000", 2024-09-21T14:04:21.395 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0502e-05 2024-09-21T14:04:21.395 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:04:21.395 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:04:21.395 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:04:21.395 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.395 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.395 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187654 9.5 9:a73ea609:::benchmark_data_smithi049_138504_object66246:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e347)", 2024-09-21T14:04:21.395 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:42.234021+0000", 2024-09-21T14:04:21.396 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 159.07403834900001, 2024-09-21T14:04:21.396 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5379708970000001, 2024-09-21T14:04:21.396 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:04:21.396 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:04:21.396 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:21.396 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:04:21.396 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:04:21.396 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:04:21.396 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187654 2024-09-21T14:04:21.396 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.396 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:04:21.396 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.396 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:04:21.396 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.234021+0000", 2024-09-21T14:04:21.396 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:04:21.396 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.397 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.397 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:04:21.397 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.234021+0000", 2024-09-21T14:04:21.397 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:04:21.397 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.397 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.397 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:04:21.397 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.234023+0000", 2024-09-21T14:04:21.397 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5460000000000001e-06 2024-09-21T14:04:21.397 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.397 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.397 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:04:21.397 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.234029+0000", 2024-09-21T14:04:21.397 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8069999999999996e-06 2024-09-21T14:04:21.397 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.398 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.398 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:04:21.398 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.234041+0000", 2024-09-21T14:04:21.398 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2189e-05 2024-09-21T14:04:21.398 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.398 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.398 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:04:21.398 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.234052+0000", 2024-09-21T14:04:21.398 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0713999999999999e-05 2024-09-21T14:04:21.398 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.398 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.398 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:04:21.398 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.743160+0000", 2024-09-21T14:04:21.398 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5091079629999999 2024-09-21T14:04:21.399 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.399 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.399 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:04:21.399 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.746100+0000", 2024-09-21T14:04:21.399 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0029400559999999999 2024-09-21T14:04:21.399 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.399 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.399 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:04:21.399 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.750393+0000", 2024-09-21T14:04:21.399 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0042938280000000004 2024-09-21T14:04:21.399 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.399 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.399 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:21.399 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.757289+0000", 2024-09-21T14:04:21.400 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0068951739999999996 2024-09-21T14:04:21.400 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.400 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.400 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:21.400 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.764759+0000", 2024-09-21T14:04:21.400 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0074705689999999998 2024-09-21T14:04:21.400 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.400 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.400 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:04:21.400 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.765446+0000", 2024-09-21T14:04:21.400 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00068706799999999999 2024-09-21T14:04:21.400 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.400 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.400 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:04:21.401 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.766945+0000", 2024-09-21T14:04:21.401 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0014990509999999999 2024-09-21T14:04:21.401 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.401 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.401 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:04:21.401 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.771992+0000", 2024-09-21T14:04:21.401 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0050469319999999996 2024-09-21T14:04:21.401 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:04:21.401 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:04:21.401 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:04:21.401 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.401 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.401 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187662 9.f 9:f2269e64:::benchmark_data_smithi049_138504_object66254:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e347)", 2024-09-21T14:04:21.401 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:42.244949+0000", 2024-09-21T14:04:21.402 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 159.06311056000001, 2024-09-21T14:04:21.402 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1087973170000001, 2024-09-21T14:04:21.402 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:04:21.402 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:04:21.402 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:21.402 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:04:21.402 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:04:21.402 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:04:21.402 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187662 2024-09-21T14:04:21.402 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.402 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:04:21.402 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.402 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:04:21.402 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.244949+0000", 2024-09-21T14:04:21.403 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:04:21.403 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.403 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.403 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:04:21.403 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.244949+0000", 2024-09-21T14:04:21.403 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:04:21.403 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.403 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.403 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:04:21.403 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.244951+0000", 2024-09-21T14:04:21.403 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.962e-06 2024-09-21T14:04:21.403 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.403 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.403 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:04:21.403 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.244957+0000", 2024-09-21T14:04:21.404 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8170000000000004e-06 2024-09-21T14:04:21.404 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.404 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.404 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:04:21.404 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.244972+0000", 2024-09-21T14:04:21.404 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5204999999999999e-05 2024-09-21T14:04:21.404 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.404 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.404 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:04:21.404 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.244981+0000", 2024-09-21T14:04:21.404 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5990000000000002e-06 2024-09-21T14:04:21.404 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.404 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.404 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:04:21.404 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.234563+0000", 2024-09-21T14:04:21.405 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.989582548 2024-09-21T14:04:21.405 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.405 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.405 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:04:21.405 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.234696+0000", 2024-09-21T14:04:21.405 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013280200000000001 2024-09-21T14:04:21.405 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.405 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.405 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3", 2024-09-21T14:04:21.405 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.234783+0000", 2024-09-21T14:04:21.405 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.6990999999999994e-05 2024-09-21T14:04:21.405 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.405 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.405 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:04:21.405 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.342749+0000", 2024-09-21T14:04:21.406 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1079658480000001 2024-09-21T14:04:21.406 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.406 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.406 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:21.406 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353673+0000", 2024-09-21T14:04:21.406 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.010924473000000001 2024-09-21T14:04:21.406 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.406 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.406 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:21.406 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353696+0000", 2024-09-21T14:04:21.406 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2792000000000001e-05 2024-09-21T14:04:21.406 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.406 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.406 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:04:21.407 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353726+0000", 2024-09-21T14:04:21.407 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9509000000000002e-05 2024-09-21T14:04:21.407 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.407 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.407 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:04:21.407 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353746+0000", 2024-09-21T14:04:21.407 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0771000000000002e-05 2024-09-21T14:04:21.407 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:04:21.407 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:04:21.407 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:04:21.407 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.407 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.407 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187665 9.f 9:fde6db3c:::benchmark_data_smithi049_138504_object66257:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e347)", 2024-09-21T14:04:21.407 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:42.254849+0000", 2024-09-21T14:04:21.408 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 159.05321069799999, 2024-09-21T14:04:21.408 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.186045939, 2024-09-21T14:04:21.408 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:04:21.408 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:04:21.408 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:21.408 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:04:21.408 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:04:21.408 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:04:21.408 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187665 2024-09-21T14:04:21.408 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.408 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:04:21.408 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.408 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:04:21.408 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.254849+0000", 2024-09-21T14:04:21.408 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:04:21.409 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.409 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.409 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:04:21.409 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.254849+0000", 2024-09-21T14:04:21.409 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:04:21.409 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.409 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.409 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:04:21.409 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.254851+0000", 2024-09-21T14:04:21.409 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9209999999999999e-06 2024-09-21T14:04:21.409 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.409 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.409 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:04:21.409 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.254857+0000", 2024-09-21T14:04:21.409 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3659999999999997e-06 2024-09-21T14:04:21.410 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.410 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.410 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:04:21.410 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.254870+0000", 2024-09-21T14:04:21.410 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2892e-05 2024-09-21T14:04:21.410 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.410 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.410 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:04:21.410 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.254882+0000", 2024-09-21T14:04:21.410 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1647999999999999e-05 2024-09-21T14:04:21.410 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.410 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.410 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:04:21.411 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.415623+0000", 2024-09-21T14:04:21.411 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.160741582 2024-09-21T14:04:21.411 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.411 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.411 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:04:21.411 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.415771+0000", 2024-09-21T14:04:21.411 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014724000000000001 2024-09-21T14:04:21.411 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.411 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.411 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3", 2024-09-21T14:04:21.411 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.415898+0000", 2024-09-21T14:04:21.411 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012708399999999999 2024-09-21T14:04:21.411 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.411 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.412 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:21.412 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.427284+0000", 2024-09-21T14:04:21.412 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.011386624 2024-09-21T14:04:21.412 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.412 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.412 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:21.412 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.427305+0000", 2024-09-21T14:04:21.412 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0956000000000001e-05 2024-09-21T14:04:21.412 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.412 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.412 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:04:21.412 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.440820+0000", 2024-09-21T14:04:21.412 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.013514511999999999 2024-09-21T14:04:21.412 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.413 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.413 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:04:21.413 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.440860+0000", 2024-09-21T14:04:21.413 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0448000000000003e-05 2024-09-21T14:04:21.413 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.413 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.413 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:04:21.413 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.440895+0000", 2024-09-21T14:04:21.413 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4666000000000003e-05 2024-09-21T14:04:21.413 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:04:21.413 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:04:21.413 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:04:21.414 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.414 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.414 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25421.0:187664 9.d e347/345 9:bfa1ff03:::benchmark_data_smithi049_138504_object66256:head v 347'11740, mlcod=346'11738)", 2024-09-21T14:04:21.414 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:42.255288+0000", 2024-09-21T14:04:21.414 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 159.05277181899999, 2024-09-21T14:04:21.414 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2055402320000002, 2024-09-21T14:04:21.414 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:04:21.414 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:04:21.414 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:21.414 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:04:21.414 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.414 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:04:21.414 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.255288+0000", 2024-09-21T14:04:21.415 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:04:21.415 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.415 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.415 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:04:21.415 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.255288+0000", 2024-09-21T14:04:21.415 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:04:21.415 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.415 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.415 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:04:21.415 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.255289+0000", 2024-09-21T14:04:21.415 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2200000000000003e-07 2024-09-21T14:04:21.415 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.415 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.415 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:04:21.416 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.255296+0000", 2024-09-21T14:04:21.416 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3440000000000003e-06 2024-09-21T14:04:21.416 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.416 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.416 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:04:21.416 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.255312+0000", 2024-09-21T14:04:21.416 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5968999999999999e-05 2024-09-21T14:04:21.416 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.416 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.416 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:04:21.416 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.255319+0000", 2024-09-21T14:04:21.416 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9199999999999998e-06 2024-09-21T14:04:21.416 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.416 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.417 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:04:21.417 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.459355+0000", 2024-09-21T14:04:21.417 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2040357249999998 2024-09-21T14:04:21.417 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.417 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.417 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:04:21.417 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.459378+0000", 2024-09-21T14:04:21.417 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3757e-05 2024-09-21T14:04:21.417 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.417 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.417 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:04:21.417 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.460796+0000", 2024-09-21T14:04:21.417 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001417721 2024-09-21T14:04:21.417 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.418 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.418 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:04:21.418 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.460828+0000", 2024-09-21T14:04:21.418 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1974000000000003e-05 2024-09-21T14:04:21.418 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:04:21.418 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:04:21.418 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:04:21.418 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.418 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.418 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24365.0:2978 5.15 5:a93a5511:::notify.2:head [watch ping cookie 94025139204608 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:04:21.418 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:43.276775+0000", 2024-09-21T14:04:21.418 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 158.03128429700001, 2024-09-21T14:04:21.418 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4885784160000002, 2024-09-21T14:04:21.418 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:04:21.419 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:04:21.419 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T14:04:21.419 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:04:21.419 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24365", 2024-09-21T14:04:21.419 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3404764347", 2024-09-21T14:04:21.419 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 2978 2024-09-21T14:04:21.419 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.419 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:04:21.419 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.419 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:04:21.419 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:43.276775+0000", 2024-09-21T14:04:21.419 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:04:21.419 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.419 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.419 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:04:21.420 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:43.276775+0000", 2024-09-21T14:04:21.420 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:04:21.420 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.420 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.420 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:04:21.420 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:43.276777+0000", 2024-09-21T14:04:21.420 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9630000000000001e-06 2024-09-21T14:04:21.420 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.420 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.420 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:04:21.420 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:43.276784+0000", 2024-09-21T14:04:21.420 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1219999999999997e-06 2024-09-21T14:04:21.420 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.420 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.420 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:04:21.421 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:43.276802+0000", 2024-09-21T14:04:21.421 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7096999999999999e-05 2024-09-21T14:04:21.421 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.421 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.421 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:04:21.421 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:43.276811+0000", 2024-09-21T14:04:21.421 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7780000000000002e-06 2024-09-21T14:04:21.421 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.421 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.421 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:04:21.421 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.765168+0000", 2024-09-21T14:04:21.421 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4883561849999998 2024-09-21T14:04:21.421 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.421 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.422 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:04:21.422 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.765252+0000", 2024-09-21T14:04:21.422 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4201000000000002e-05 2024-09-21T14:04:21.422 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.422 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.422 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:04:21.422 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.765354+0000", 2024-09-21T14:04:21.422 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010207 2024-09-21T14:04:21.422 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:04:21.422 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:04:21.422 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:04:21.422 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.422 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.422 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.25421.0:187651 9.f e349/345 ondisk, result = 0)", 2024-09-21T14:04:21.423 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:44.238795+0000", 2024-09-21T14:04:21.423 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 157.06926501000001, 2024-09-21T14:04:21.423 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.115079242, 2024-09-21T14:04:21.423 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:04:21.423 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:04:21.423 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T14:04:21.423 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:04:21.423 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.423 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:04:21.423 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238795+0000", 2024-09-21T14:04:21.423 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:04:21.423 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.423 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.423 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:04:21.423 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238795+0000", 2024-09-21T14:04:21.424 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:04:21.424 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.424 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.424 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:04:21.424 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238795+0000", 2024-09-21T14:04:21.424 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9800000000000003e-07 2024-09-21T14:04:21.424 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.424 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.424 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:04:21.424 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238799+0000", 2024-09-21T14:04:21.424 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9920000000000001e-06 2024-09-21T14:04:21.424 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.424 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.424 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:04:21.424 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238809+0000", 2024-09-21T14:04:21.425 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6120000000000005e-06 2024-09-21T14:04:21.425 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.425 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.425 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:04:21.425 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238816+0000", 2024-09-21T14:04:21.425 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6229999999999999e-06 2024-09-21T14:04:21.425 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.425 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.425 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:04:21.425 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.342725+0000", 2024-09-21T14:04:21.425 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1039093099999999 2024-09-21T14:04:21.425 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.425 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.425 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:04:21.425 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.342733+0000", 2024-09-21T14:04:21.425 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9189999999999999e-06 2024-09-21T14:04:21.426 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.426 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.426 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:04:21.426 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353874+0000", 2024-09-21T14:04:21.426 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.011141088 2024-09-21T14:04:21.426 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:04:21.426 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:04:21.426 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:04:21.426 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.426 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.426 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.25421.0:187651 9.f e349/345 ondisk, result = 0)", 2024-09-21T14:04:21.426 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:44.238813+0000", 2024-09-21T14:04:21.426 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 157.06924666800001, 2024-09-21T14:04:21.426 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1148360479999999, 2024-09-21T14:04:21.427 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:04:21.427 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:04:21.427 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T14:04:21.427 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:04:21.427 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.427 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:04:21.427 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238813+0000", 2024-09-21T14:04:21.427 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:04:21.427 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.427 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.427 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:04:21.427 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238813+0000", 2024-09-21T14:04:21.427 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:04:21.427 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.427 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.428 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:04:21.428 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238814+0000", 2024-09-21T14:04:21.428 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4700000000000001e-07 2024-09-21T14:04:21.428 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.428 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.428 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:04:21.428 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238817+0000", 2024-09-21T14:04:21.428 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3469999999999999e-06 2024-09-21T14:04:21.428 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.428 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.428 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:04:21.428 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238827+0000", 2024-09-21T14:04:21.428 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0011e-05 2024-09-21T14:04:21.428 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.428 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.428 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:04:21.429 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238832+0000", 2024-09-21T14:04:21.429 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1139999999999997e-06 2024-09-21T14:04:21.429 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.429 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.429 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:04:21.429 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353559+0000", 2024-09-21T14:04:21.429 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1147265909999999 2024-09-21T14:04:21.429 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.429 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.429 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:04:21.429 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353566+0000", 2024-09-21T14:04:21.429 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8669999999999994e-06 2024-09-21T14:04:21.429 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.429 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.430 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:04:21.430 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353649+0000", 2024-09-21T14:04:21.430 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2571000000000006e-05 2024-09-21T14:04:21.430 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:04:21.430 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:04:21.430 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:04:21.430 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.430 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.430 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187668 9.5 9:acffd95c:::benchmark_data_smithi049_138504_object66260:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e349)", 2024-09-21T14:04:21.430 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:44.468347+0000", 2024-09-21T14:04:21.430 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 156.839713171, 2024-09-21T14:04:21.430 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3419073799999999, 2024-09-21T14:04:21.430 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:04:21.430 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:04:21.431 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:21.431 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:04:21.431 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:04:21.431 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:04:21.431 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187668 2024-09-21T14:04:21.431 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.431 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:04:21.431 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.431 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:04:21.431 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.468347+0000", 2024-09-21T14:04:21.431 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:04:21.431 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.431 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.431 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:04:21.432 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.468347+0000", 2024-09-21T14:04:21.432 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:04:21.432 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.432 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.432 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:04:21.432 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.468347+0000", 2024-09-21T14:04:21.432 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2399999999999997e-07 2024-09-21T14:04:21.432 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.432 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.432 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:04:21.432 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.468351+0000", 2024-09-21T14:04:21.432 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6049999999999998e-06 2024-09-21T14:04:21.432 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.432 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.432 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:04:21.433 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.468358+0000", 2024-09-21T14:04:21.433 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4510000000000003e-06 2024-09-21T14:04:21.433 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.433 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.433 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:04:21.433 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.468366+0000", 2024-09-21T14:04:21.433 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6140000000000003e-06 2024-09-21T14:04:21.433 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.433 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.433 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:04:21.433 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.772366+0000", 2024-09-21T14:04:21.433 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3040005530000001 2024-09-21T14:04:21.433 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.433 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.433 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:04:21.434 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.775717+0000", 2024-09-21T14:04:21.434 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0033501080000000001 2024-09-21T14:04:21.434 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.434 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.434 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:04:21.434 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.781090+0000", 2024-09-21T14:04:21.434 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0053731990000000004 2024-09-21T14:04:21.434 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.434 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.434 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:21.434 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.798846+0000", 2024-09-21T14:04:21.434 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.017756042999999999 2024-09-21T14:04:21.434 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.434 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.435 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:21.435 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.800017+0000", 2024-09-21T14:04:21.435 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0011714430000000001 2024-09-21T14:04:21.435 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.435 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.435 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:04:21.435 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.800552+0000", 2024-09-21T14:04:21.435 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00053442500000000005 2024-09-21T14:04:21.435 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.435 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.435 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:04:21.435 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.801930+0000", 2024-09-21T14:04:21.435 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0013783109999999999 2024-09-21T14:04:21.435 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:04:21.436 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:04:21.436 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:04:21.436 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.810254+0000", 2024-09-21T14:04:21.436 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.008323904 2024-09-21T14:04:21.436 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:04:21.436 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:04:21.436 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:04:21.436 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:04:21.436 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:04:21.436 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T14:04:21.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:21.438 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:21.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:21.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:21 smithi187 ceph-mon[33032]: pgmap v782: 158 pgs: 158 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-09-21T14:04:21.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:21 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'40481','4':'51701','6':'59481'}"}]': finished 2024-09-21T14:04:21.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:21 smithi187 ceph-mon[33032]: osdmap e408: 8 total, 8 up, 8 in 2024-09-21T14:04:21.477 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:04:21.539 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:04:21.560 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:21.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:21.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:04:21.656 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T14:04:21.662 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:04:21.721 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:21.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:21.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:04:21.823 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:04:21.844 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:21.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:21.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:04:21.881 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 76 16 112262 112246 92.2883 0 - 0.0105988 2024-09-21T14:04:21.945 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:21.968 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:21.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:21.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:22.069 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:04:22.091 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:22.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:22.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:04:22.144 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T14:04:22.152 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:04:22.189 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-09-21T14:04:22.193 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:04:22.236 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:22.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:22.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:04:22.250 INFO:teuthology.orchestra.run.smithi049.stdout:moved 0 / 474 (0%) 2024-09-21T14:04:22.250 INFO:teuthology.orchestra.run.smithi049.stdout:avg 59.25 2024-09-21T14:04:22.250 INFO:teuthology.orchestra.run.smithi049.stdout:stddev 8.71421 -> 8.71421 (expected baseline 7.20026) 2024-09-21T14:04:22.250 INFO:teuthology.orchestra.run.smithi049.stdout:min osd.0 with 39 -> 39 pgs (0.658228 -> 0.658228 * mean) 2024-09-21T14:04:22.250 INFO:teuthology.orchestra.run.smithi049.stdout:max osd.3 with 68 -> 68 pgs (1.14768 -> 1.14768 * mean) 2024-09-21T14:04:22.250 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T14:04:22.250 INFO:teuthology.orchestra.run.smithi049.stdout:oload 1000 2024-09-21T14:04:22.251 INFO:teuthology.orchestra.run.smithi049.stdout:max_change 3 2024-09-21T14:04:22.251 INFO:teuthology.orchestra.run.smithi049.stdout:max_change_osds 4 2024-09-21T14:04:22.251 INFO:teuthology.orchestra.run.smithi049.stdout:average_utilization 0.0331 2024-09-21T14:04:22.251 INFO:teuthology.orchestra.run.smithi049.stdout:overload_utilization 0.3310 2024-09-21T14:04:22.251 INFO:teuthology.orchestra.run.smithi049.stderr:no change 2024-09-21T14:04:22.338 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:04:22.370 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:22.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:22.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:04:22.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:22 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:04:22.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:22 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:04:22.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:22 smithi187 ceph-mon[33032]: osdmap e409: 8 total, 8 up, 8 in 2024-09-21T14:04:22.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:22 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1607545679' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:04:22.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:22 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:04:22.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:22 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:04:22.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:22 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:04:22.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:22 smithi187 ceph-mon[33032]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:04:22.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:22 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T14:04:22.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:22 smithi187 ceph-mon[33032]: osdmap e410: 8 total, 8 up, 8 in 2024-09-21T14:04:22.473 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:04:22.495 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:22.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:22.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:04:22.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:22 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:04:22.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:22 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:04:22.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:22 smithi049 ceph-mon[30080]: osdmap e409: 8 total, 8 up, 8 in 2024-09-21T14:04:22.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:22 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1607545679' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:04:22.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:22 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:04:22.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:22 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:04:22.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:22 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:04:22.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:22 smithi049 ceph-mon[30080]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:04:22.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:22 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T14:04:22.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:22 smithi049 ceph-mon[30080]: osdmap e410: 8 total, 8 up, 8 in 2024-09-21T14:04:22.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:22 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:04:22.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:22 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:04:22.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:22 smithi049 ceph-mon[40489]: osdmap e409: 8 total, 8 up, 8 in 2024-09-21T14:04:22.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:22 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1607545679' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:04:22.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:22 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:04:22.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:22 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:04:22.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:22 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:04:22.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:22 smithi049 ceph-mon[40489]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:04:22.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:22 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T14:04:22.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:22 smithi049 ceph-mon[40489]: osdmap e410: 8 total, 8 up, 8 in 2024-09-21T14:04:22.597 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:04:22.628 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:22.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:22.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:04:22.730 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:04:22.768 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:22.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:22.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:04:22.870 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:04:22.881 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 77 16 112845 112829 91.5626 18.2188 0.0110034 0.0109168 2024-09-21T14:04:22.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:22.912 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:22.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:04:23.013 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:04:23.055 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:23.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:23.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:04:23.157 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:04:23.196 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:23.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:23.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:04:23.298 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:23.339 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:23.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:23.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:23.441 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:04:23.479 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:23.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:23.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:04:23.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:23 smithi049 ceph-mon[40489]: pgmap v785: 158 pgs: 4 peering, 154 active+clean; 6.8 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 944 op/s; 88 B/s, 2 objects/s recovering 2024-09-21T14:04:23.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:23 smithi049 ceph-mon[40489]: 10.a starting backfill to osd.2 from (0'0,0'0] MAX to 407'6932 2024-09-21T14:04:23.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:23 smithi049 ceph-mon[40489]: 10.0 starting backfill to osd.7 from (0'0,0'0] MAX to 407'7020 2024-09-21T14:04:23.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:23 smithi049 ceph-mon[40489]: 10.4 starting backfill to osd.1 from (0'0,0'0] MAX to 407'6963 2024-09-21T14:04:23.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:23 smithi049 ceph-mon[40489]: 10.d starting backfill to osd.3 from (0'0,0'0] MAX to 407'7052 2024-09-21T14:04:23.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:23 smithi049 ceph-mon[40489]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T14:04:23.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:23 smithi049 ceph-mon[40489]: from='client.16218 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T14:04:23.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:23 smithi049 ceph-mon[30080]: pgmap v785: 158 pgs: 4 peering, 154 active+clean; 6.8 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 944 op/s; 88 B/s, 2 objects/s recovering 2024-09-21T14:04:23.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:23 smithi049 ceph-mon[30080]: 10.a starting backfill to osd.2 from (0'0,0'0] MAX to 407'6932 2024-09-21T14:04:23.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:23 smithi049 ceph-mon[30080]: 10.0 starting backfill to osd.7 from (0'0,0'0] MAX to 407'7020 2024-09-21T14:04:23.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:23 smithi049 ceph-mon[30080]: 10.4 starting backfill to osd.1 from (0'0,0'0] MAX to 407'6963 2024-09-21T14:04:23.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:23 smithi049 ceph-mon[30080]: 10.d starting backfill to osd.3 from (0'0,0'0] MAX to 407'7052 2024-09-21T14:04:23.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:23 smithi049 ceph-mon[30080]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T14:04:23.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:23 smithi049 ceph-mon[30080]: from='client.16218 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T14:04:23.581 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:04:23.620 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:23.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:23.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:04:23.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:23 smithi187 ceph-mon[33032]: pgmap v785: 158 pgs: 4 peering, 154 active+clean; 6.8 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 944 op/s; 88 B/s, 2 objects/s recovering 2024-09-21T14:04:23.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:23 smithi187 ceph-mon[33032]: 10.a starting backfill to osd.2 from (0'0,0'0] MAX to 407'6932 2024-09-21T14:04:23.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:23 smithi187 ceph-mon[33032]: 10.0 starting backfill to osd.7 from (0'0,0'0] MAX to 407'7020 2024-09-21T14:04:23.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:23 smithi187 ceph-mon[33032]: 10.4 starting backfill to osd.1 from (0'0,0'0] MAX to 407'6963 2024-09-21T14:04:23.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:23 smithi187 ceph-mon[33032]: 10.d starting backfill to osd.3 from (0'0,0'0] MAX to 407'7052 2024-09-21T14:04:23.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:23 smithi187 ceph-mon[33032]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T14:04:23.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:23 smithi187 ceph-mon[33032]: from='client.16218 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T14:04:23.722 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:04:23.755 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:23.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:23.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:04:23.856 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:04:23.881 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 78 16 114091 114075 91.3871 77.875 0.00635181 0.0109378 2024-09-21T14:04:23.891 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:23.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:23.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:04:23.994 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:04:24.029 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:24.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:24.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:04:24.105 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-21T14:04:24.105 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd reweight-by-utilization 1000 1.0 2024-09-21T14:04:24.131 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:04:24.170 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:24.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:24.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:04:24.272 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:04:24.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:24.308 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:24.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:04:24.409 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:04:24.443 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:24.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:24.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:04:24.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:24 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:04:24.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:24 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:04:24.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:24 smithi049 ceph-mon[30080]: osdmap e411: 8 total, 8 up, 8 in 2024-09-21T14:04:24.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:24 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:04:24.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:24 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:04:24.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:24 smithi049 ceph-mon[40489]: osdmap e411: 8 total, 8 up, 8 in 2024-09-21T14:04:24.545 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:24.585 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:24.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:24.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:24.688 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:04:24.696 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:24 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:04:24.696 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:24 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:04:24.696 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:24 smithi187 ceph-mon[33032]: osdmap e411: 8 total, 8 up, 8 in 2024-09-21T14:04:24.723 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:24.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:24.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:04:24.825 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:24.868 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:24.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:24.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:24.881 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 79 16 115252 115236 91.1488 72.5625 0.00928897 0.0109663 2024-09-21T14:04:24.970 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:04:24.979 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:04:24.979 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:04:25.022 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:04:25.028 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:04:25.028 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:04:25.040 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:25.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:25.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:04:25.064 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:04:25.065 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:04:25.074 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:04:25.101 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:04:25.102 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:04:25.139 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:04:25.139 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:04:25.142 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:04:25.170 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:04:25.171 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:04:25.174 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:25.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:25.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:04:25.212 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:04:25.213 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:04:25.247 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:04:25.247 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:04:25.275 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:04:25.306 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:25.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:25.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:04:25.408 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:04:25.455 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:25.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:25.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:04:25.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:25 smithi049 ceph-mon[30080]: pgmap v788: 158 pgs: 1 active+remapped+backfilling, 4 peering, 3 active+remapped+backfill_toofull, 150 active+clean; 6.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 255 B/s rd, 53 MiB/s wr, 854 op/s; 28165/340323 objects misplaced (8.276%); 3.4 MiB/s, 59 objects/s recovering 2024-09-21T14:04:25.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:25 smithi049 ceph-mon[30080]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T14:04:25.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:25 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/240367087' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:04:25.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:25 smithi049 ceph-mon[40489]: pgmap v788: 158 pgs: 1 active+remapped+backfilling, 4 peering, 3 active+remapped+backfill_toofull, 150 active+clean; 6.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 255 B/s rd, 53 MiB/s wr, 854 op/s; 28165/340323 objects misplaced (8.276%); 3.4 MiB/s, 59 objects/s recovering 2024-09-21T14:04:25.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:25 smithi049 ceph-mon[40489]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T14:04:25.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:25 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/240367087' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:04:25.557 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:04:25.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:25.601 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:25.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:04:25.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:25 smithi187 ceph-mon[33032]: pgmap v788: 158 pgs: 1 active+remapped+backfilling, 4 peering, 3 active+remapped+backfill_toofull, 150 active+clean; 6.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 255 B/s rd, 53 MiB/s wr, 854 op/s; 28165/340323 objects misplaced (8.276%); 3.4 MiB/s, 59 objects/s recovering 2024-09-21T14:04:25.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:25 smithi187 ceph-mon[33032]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T14:04:25.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:25 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/240367087' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:04:25.702 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:04:25.741 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:25.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:25.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:04:25.843 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:04:25.883 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:2024-09-21T14:04:25.883681+0000 min lat: 0.00132701 max lat: 2.18205 avg lat: 0.0109972 2024-09-21T14:04:25.883 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T14:04:25.883 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 80 16 116383 116367 90.8923 70.6875 0.0116287 0.0109972 2024-09-21T14:04:25.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:25.886 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:25.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:04:25.987 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:26.026 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:26.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:26.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:26.129 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:04:26.163 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:26.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:26.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:04:26.264 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:04:26.304 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:26.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:26.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:04:26.391 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:26 smithi049 ceph-mon[40489]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-09-21T14:04:26.391 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:26 smithi049 ceph-mon[30080]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-09-21T14:04:26.406 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:04:26.452 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:26.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:26.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:04:26.492 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T14:04:26.492 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:04:26.554 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:04:26.598 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:26.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:26.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:04:26.685 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T14:04:26.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:26 smithi187 ceph-mon[33032]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-09-21T14:04:26.700 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:04:26.729 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:26.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:26.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:04:26.830 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:26.880 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:26.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:26.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:26.884 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 81 16 117510 117494 90.6393 70.4375 0.0129639 0.0110276 2024-09-21T14:04:26.947 INFO:teuthology.orchestra.run.smithi049.stdout:moved 0 / 474 (0%) 2024-09-21T14:04:26.947 INFO:teuthology.orchestra.run.smithi049.stdout:avg 59.25 2024-09-21T14:04:26.947 INFO:teuthology.orchestra.run.smithi049.stdout:stddev 8.71421 -> 8.71421 (expected baseline 7.20026) 2024-09-21T14:04:26.947 INFO:teuthology.orchestra.run.smithi049.stdout:min osd.0 with 39 -> 39 pgs (0.658228 -> 0.658228 * mean) 2024-09-21T14:04:26.947 INFO:teuthology.orchestra.run.smithi049.stdout:max osd.3 with 68 -> 68 pgs (1.14768 -> 1.14768 * mean) 2024-09-21T14:04:26.948 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T14:04:26.948 INFO:teuthology.orchestra.run.smithi049.stdout:oload 1000 2024-09-21T14:04:26.948 INFO:teuthology.orchestra.run.smithi049.stdout:max_change 1 2024-09-21T14:04:26.948 INFO:teuthology.orchestra.run.smithi049.stdout:max_change_osds 4 2024-09-21T14:04:26.948 INFO:teuthology.orchestra.run.smithi049.stdout:average_utilization 0.0339 2024-09-21T14:04:26.948 INFO:teuthology.orchestra.run.smithi049.stdout:overload_utilization 0.3388 2024-09-21T14:04:26.948 INFO:teuthology.orchestra.run.smithi049.stderr:no change 2024-09-21T14:04:26.982 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:04:27.010 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:27.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:27.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:04:27.113 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:04:27.151 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:27.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:27.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:04:27.252 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:04:27.281 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:27.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:27.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:04:27.382 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:27.426 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:27.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:27.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:27.528 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:04:27.567 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:27.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:27.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:04:27.669 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:04:27.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:27 smithi187 ceph-mon[33032]: pgmap v789: 158 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 154 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 174 B/s rd, 44 MiB/s wr, 709 op/s; 28165/342519 objects misplaced (8.223%); 2.3 MiB/s, 43 objects/s recovering 2024-09-21T14:04:27.703 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:27.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:27.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:04:27.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:27 smithi049 ceph-mon[40489]: pgmap v789: 158 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 154 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 174 B/s rd, 44 MiB/s wr, 709 op/s; 28165/342519 objects misplaced (8.223%); 2.3 MiB/s, 43 objects/s recovering 2024-09-21T14:04:27.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:27 smithi049 ceph-mon[30080]: pgmap v789: 158 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 154 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 174 B/s rd, 44 MiB/s wr, 709 op/s; 28165/342519 objects misplaced (8.223%); 2.3 MiB/s, 43 objects/s recovering 2024-09-21T14:04:27.805 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:04:27.839 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:27.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:27.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:04:27.883 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 82 15 118688 118673 90.4326 73.6875 0.00653693 0.0110539 2024-09-21T14:04:27.941 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:04:27.982 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:27.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:27.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:04:28.084 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:04:28.123 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:28.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:28.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:04:28.224 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:04:28.261 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:28.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:28.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:04:28.363 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:04:28.405 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:28.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:28.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:04:28.507 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:04:28.546 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:28.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:28.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:04:28.648 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:04:28.690 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:28.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:28.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:04:28.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:28 smithi187 ceph-mon[33032]: from='client.26129 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T14:04:28.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:28 smithi049 ceph-mon[40489]: from='client.26129 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T14:04:28.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:28 smithi049 ceph-mon[30080]: from='client.26129 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T14:04:28.792 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:04:28.822 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:28.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:28.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:04:28.883 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 83 16 119865 119849 90.2285 73.5 0.0119982 0.0110783 2024-09-21T14:04:28.923 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:04:28.965 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:28.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:28.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:04:29.041 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T14:04:29.067 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:29.129 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:29.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:29.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:29.230 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:04:29.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:29.271 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:29.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:04:29.372 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:04:29.401 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:29.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:29.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:04:29.503 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:04:29.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:29 smithi049 ceph-mon[40489]: pgmap v790: 158 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 154 active+clean; 7.0 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.0 KiB/s rd, 45 MiB/s wr, 726 op/s; 28165/346767 objects misplaced (8.122%); 2.0 MiB/s, 36 objects/s recovering 2024-09-21T14:04:29.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:29 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:04:29.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:29 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:04:29.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:29 smithi049 ceph-mon[30080]: pgmap v790: 158 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 154 active+clean; 7.0 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.0 KiB/s rd, 45 MiB/s wr, 726 op/s; 28165/346767 objects misplaced (8.122%); 2.0 MiB/s, 36 objects/s recovering 2024-09-21T14:04:29.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:29 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:04:29.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:29 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:04:29.540 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:29.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:29.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:04:29.642 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:29.680 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:29.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:29.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:29.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:29 smithi187 ceph-mon[33032]: pgmap v790: 158 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 154 active+clean; 7.0 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.0 KiB/s rd, 45 MiB/s wr, 726 op/s; 28165/346767 objects misplaced (8.122%); 2.0 MiB/s, 36 objects/s recovering 2024-09-21T14:04:29.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:29 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:04:29.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:29 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:04:29.783 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:04:29.820 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:29.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:29.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:04:29.883 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 84 16 120945 120929 89.9579 67.5 0.00513985 0.0111108 2024-09-21T14:04:29.922 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:04:29.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:29.964 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:29.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:04:30.065 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:04:30.107 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:30.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:30.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:04:30.194 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:04:29 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:04:30.209 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:04:30.246 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:30.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:30.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:04:30.348 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:04:30.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:30.389 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:30.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:04:30.490 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:04:30.522 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:30.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:30.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:04:30.580 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:30 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:04:30.581 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:30 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:04:30.623 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:04:30.630 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:04:30.693 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:30.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:30 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:04:30.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:30.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:04:30.699 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T14:04:30.796 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:04:30.851 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:04:30 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:04:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:04:30.855 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:30.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:30.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:04:30.884 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 85 16 122129 122113 89.7694 74 0.00972893 0.011135 2024-09-21T14:04:30.956 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:04:31.022 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:31.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:31.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:04:31.124 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:04:31.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:31.173 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:31.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:04:31.274 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:31.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:31.313 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:31.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:31.376 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:04:31.396 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:04:31.414 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:04:31.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:31.487 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:31.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:04:31.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:31 smithi049 ceph-mon[30080]: pgmap v791: 158 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 154 active+clean; 7.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 511 B/s rd, 64 MiB/s wr, 1.03k op/s; 28836/357363 objects misplaced (8.069%); 7.2 MiB/s, 117 objects/s recovering 2024-09-21T14:04:31.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:31 smithi049 ceph-mon[40489]: pgmap v791: 158 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 154 active+clean; 7.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 511 B/s rd, 64 MiB/s wr, 1.03k op/s; 28836/357363 objects misplaced (8.069%); 7.2 MiB/s, 117 objects/s recovering 2024-09-21T14:04:31.587 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:04:31.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:31.641 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:31.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:04:31.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:31 smithi187 ceph-mon[33032]: pgmap v791: 158 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 154 active+clean; 7.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 511 B/s rd, 64 MiB/s wr, 1.03k op/s; 28836/357363 objects misplaced (8.069%); 7.2 MiB/s, 117 objects/s recovering 2024-09-21T14:04:31.743 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:04:31.786 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:31.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:31.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:04:31.884 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 86 16 123148 123132 89.4661 63.6875 0.0152392 0.0111729 2024-09-21T14:04:31.888 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:04:31.929 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:31.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:31.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:04:32.031 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:04:32.066 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:32.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:32.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:04:32.167 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:32.210 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:32.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:32.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:32.312 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:04:32.343 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-09-21T14:04:32.345 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:32.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:32.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:04:32.350 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T14:04:32.350 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:04:32.447 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:04:32.483 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:32.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:32.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:04:32.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:32 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1146064744' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:04:32.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:32 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:04:32.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:32 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1146064744' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:04:32.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:32 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:04:32.585 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:04:32.618 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:32.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:32.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:04:32.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:32 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1146064744' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:04:32.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:32 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:04:32.720 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:04:32.762 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:32.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:32.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:04:32.864 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:32.885 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 87 16 124275 124259 89.2457 70.4375 0.0102053 0.0112003 2024-09-21T14:04:32.892 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:04:32.893 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:04:32.893 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:04:32.893 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:04:32.893 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:04:32.893 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:04:32.893 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:04:32.893 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:04:32.893 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:04:32.893 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:04:32.893 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:04:32.893 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:04:32.893 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:04:32.893 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:04:32.894 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:04:32.894 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:04:32.904 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:32.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:32.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:33.007 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:04:33.043 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:33.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:33.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:04:33.144 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:04:33.177 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:33.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:33.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:04:33.279 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:04:33.318 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:33.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:33.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:04:33.420 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:04:33.459 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:33.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:33.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:04:33.479 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 4, 6, 2, 3, 7, 5] 2024-09-21T14:04:33.479 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T14:04:33.479 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_3 2024-09-21T14:04:33.480 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T14:04:33.539 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:33 smithi049 ceph-mon[30080]: pgmap v792: 158 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 154 active+clean; 7.3 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 935 B/s rd, 59 MiB/s wr, 940 op/s; 28836/360480 objects misplaced (7.999%); 5.8 MiB/s, 95 objects/s recovering 2024-09-21T14:04:33.539 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:33 smithi049 ceph-mon[30080]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:04:33.539 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:33 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T14:04:33.539 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:33 smithi049 ceph-mon[30080]: osdmap e412: 8 total, 8 up, 8 in 2024-09-21T14:04:33.539 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:33 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/45173573' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:04:33.539 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:33 smithi049 ceph-mon[40489]: pgmap v792: 158 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 154 active+clean; 7.3 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 935 B/s rd, 59 MiB/s wr, 940 op/s; 28836/360480 objects misplaced (7.999%); 5.8 MiB/s, 95 objects/s recovering 2024-09-21T14:04:33.539 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:33 smithi049 ceph-mon[40489]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:04:33.539 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:33 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T14:04:33.540 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:33 smithi049 ceph-mon[40489]: osdmap e412: 8 total, 8 up, 8 in 2024-09-21T14:04:33.540 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:33 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/45173573' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:04:33.561 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:04:33.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:33.596 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:33.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:04:33.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:33 smithi187 ceph-mon[33032]: pgmap v792: 158 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 154 active+clean; 7.3 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 935 B/s rd, 59 MiB/s wr, 940 op/s; 28836/360480 objects misplaced (7.999%); 5.8 MiB/s, 95 objects/s recovering 2024-09-21T14:04:33.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:33 smithi187 ceph-mon[33032]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:04:33.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:33 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T14:04:33.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:33 smithi187 ceph-mon[33032]: osdmap e412: 8 total, 8 up, 8 in 2024-09-21T14:04:33.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:33 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/45173573' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:04:33.697 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:04:33.729 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:33.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:33.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:04:33.831 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:04:33.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:33.868 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:33.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:04:33.886 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 88 16 125390 125374 89.0234 69.6875 0.0114066 0.0112284 2024-09-21T14:04:33.968 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:04:34.009 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:34.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:34.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:04:34.111 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:34.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:34.153 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:34.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:34.254 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:04:34.295 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:34.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:34.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:04:34.397 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:04:34.432 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:34.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:34.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:04:34.534 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:04:34.566 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:34.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:34.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:04:34.667 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:34.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:34 smithi187 ceph-mon[33032]: 3.18 scrub starts 2024-09-21T14:04:34.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:34 smithi187 ceph-mon[33032]: 3.18 scrub ok 2024-09-21T14:04:34.710 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:34.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:34.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:34.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:34 smithi049 ceph-mon[30080]: 3.18 scrub starts 2024-09-21T14:04:34.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:34 smithi049 ceph-mon[30080]: 3.18 scrub ok 2024-09-21T14:04:34.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:34 smithi049 ceph-mon[40489]: 3.18 scrub starts 2024-09-21T14:04:34.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:34 smithi049 ceph-mon[40489]: 3.18 scrub ok 2024-09-21T14:04:34.812 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:04:34.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:34.858 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:34.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:04:34.886 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 89 16 126491 126475 88.7963 68.8125 0.0301812 0.0112567 2024-09-21T14:04:34.959 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:04:34.994 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:34.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:34.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:04:35.096 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:04:35.142 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:35.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:35.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:04:35.244 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:04:35.278 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:35.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:35.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:04:35.380 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:04:35.418 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:35.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:35.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:04:35.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:35 smithi049 ceph-mon[40489]: pgmap v794: 158 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 154 active+clean; 7.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1023 B/s rd, 68 MiB/s wr, 1.10k op/s; 29496/373215 objects misplaced (7.903%); 8.5 MiB/s, 137 objects/s recovering 2024-09-21T14:04:35.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:35 smithi049 ceph-mon[40489]: 3.17 scrub starts 2024-09-21T14:04:35.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:35 smithi049 ceph-mon[40489]: 3.17 scrub ok 2024-09-21T14:04:35.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:35 smithi049 ceph-mon[30080]: pgmap v794: 158 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 154 active+clean; 7.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1023 B/s rd, 68 MiB/s wr, 1.10k op/s; 29496/373215 objects misplaced (7.903%); 8.5 MiB/s, 137 objects/s recovering 2024-09-21T14:04:35.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:35 smithi049 ceph-mon[30080]: 3.17 scrub starts 2024-09-21T14:04:35.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:35 smithi049 ceph-mon[30080]: 3.17 scrub ok 2024-09-21T14:04:35.520 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:04:35.566 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:35.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:35.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:04:35.668 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:04:35.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:35 smithi187 ceph-mon[33032]: pgmap v794: 158 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 154 active+clean; 7.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1023 B/s rd, 68 MiB/s wr, 1.10k op/s; 29496/373215 objects misplaced (7.903%); 8.5 MiB/s, 137 objects/s recovering 2024-09-21T14:04:35.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:35 smithi187 ceph-mon[33032]: 3.17 scrub starts 2024-09-21T14:04:35.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:35 smithi187 ceph-mon[33032]: 3.17 scrub ok 2024-09-21T14:04:35.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:35.703 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:35.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:04:35.804 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:04:35.825 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:35.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:35.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:04:35.886 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Total time run: 90.0167 2024-09-21T14:04:35.886 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Total writes made: 127574 2024-09-21T14:04:35.886 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Write size: 65536 2024-09-21T14:04:35.886 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Object size: 65536 2024-09-21T14:04:35.886 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Bandwidth (MB/sec): 88.5766 2024-09-21T14:04:35.886 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Stddev Bandwidth: 37.245 2024-09-21T14:04:35.886 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Max bandwidth (MB/sec): 236.5 2024-09-21T14:04:35.886 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Min bandwidth (MB/sec): 0 2024-09-21T14:04:35.886 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Average IOPS: 1417 2024-09-21T14:04:35.886 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Stddev IOPS: 595.931 2024-09-21T14:04:35.886 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Max IOPS: 3784 2024-09-21T14:04:35.886 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Min IOPS: 0 2024-09-21T14:04:35.887 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Average Latency(s): 0.0112848 2024-09-21T14:04:35.887 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Stddev Latency(s): 0.0257625 2024-09-21T14:04:35.887 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Max latency(s): 2.18205 2024-09-21T14:04:35.887 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Min latency(s): 0.00132701 2024-09-21T14:04:35.891 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Cleaning up (deleting benchmark objects) 2024-09-21T14:04:35.927 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:04:35.967 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:35.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:35.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:04:36.069 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:36.096 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:04:36.134 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T14:04:36.143 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:36.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:36.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:36.251 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:36.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:36.294 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:36.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:36.395 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:36.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:36.450 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:36.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:36.552 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:36.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:36.597 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:36.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:36.666 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:36 smithi049 ceph-mon[40489]: 5.18 scrub starts 2024-09-21T14:04:36.666 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:36 smithi049 ceph-mon[40489]: 5.18 scrub ok 2024-09-21T14:04:36.666 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:36 smithi049 ceph-mon[30080]: 5.18 scrub starts 2024-09-21T14:04:36.666 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:36 smithi049 ceph-mon[30080]: 5.18 scrub ok 2024-09-21T14:04:36.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:36 smithi187 ceph-mon[33032]: 5.18 scrub starts 2024-09-21T14:04:36.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:36 smithi187 ceph-mon[33032]: 5.18 scrub ok 2024-09-21T14:04:36.697 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:04:36.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:36.736 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:36.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:04:36.837 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:04:36.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:36.868 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:36.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:04:36.896 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:04:36.909 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T14:04:36.968 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:37.050 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:37.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:37.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:37.151 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:04:37.174 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:37.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:37.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:04:37.234 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T14:04:37.243 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-09-21T14:04:37.276 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:04:37.307 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:37.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:37.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:04:37.409 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:04:37.448 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:37.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:37.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:04:37.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:37 smithi049 ceph-mon[30080]: pgmap v795: 158 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 154 active+clean; 7.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1023 B/s rd, 70 MiB/s wr, 1.12k op/s; 29496/376245 objects misplaced (7.840%); 8.5 MiB/s, 136 objects/s recovering 2024-09-21T14:04:37.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:37 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:04:37.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:37 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:04:37.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:37 smithi049 ceph-mon[30080]: 3.12 scrub starts 2024-09-21T14:04:37.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:37 smithi049 ceph-mon[30080]: 3.12 scrub ok 2024-09-21T14:04:37.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:37 smithi049 ceph-mon[40489]: pgmap v795: 158 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 154 active+clean; 7.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1023 B/s rd, 70 MiB/s wr, 1.12k op/s; 29496/376245 objects misplaced (7.840%); 8.5 MiB/s, 136 objects/s recovering 2024-09-21T14:04:37.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:37 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:04:37.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:37 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:04:37.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:37 smithi049 ceph-mon[40489]: 3.12 scrub starts 2024-09-21T14:04:37.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:37 smithi049 ceph-mon[40489]: 3.12 scrub ok 2024-09-21T14:04:37.550 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:04:37.588 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:37.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:37.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:04:37.689 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:37.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:37 smithi187 ceph-mon[33032]: pgmap v795: 158 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 154 active+clean; 7.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1023 B/s rd, 70 MiB/s wr, 1.12k op/s; 29496/376245 objects misplaced (7.840%); 8.5 MiB/s, 136 objects/s recovering 2024-09-21T14:04:37.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:37 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:04:37.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:37 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:04:37.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:37 smithi187 ceph-mon[33032]: 3.12 scrub starts 2024-09-21T14:04:37.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:37 smithi187 ceph-mon[33032]: 3.12 scrub ok 2024-09-21T14:04:37.704 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T14:04:37.725 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:37.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:37.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:37.827 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:04:37.856 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:37.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:37.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:04:37.958 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:04:37.990 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:37.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:37.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:04:38.095 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:04:38.135 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:38.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:38.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:04:38.231 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":795,"stamp":"2024-09-21T14:04:36.036985+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8206483934,"num_objects":125415,"num_object_clones":0,"num_object_copies":376245,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":29496,"num_objects_unfound":0,"num_objects_dirty":125415,"num_whiteouts":0,"num_read":1474,"num_read_kb":1283,"num_write":126003,"num_write_kb":8014821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1795,"num_bytes_recovered":104527237,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82069,"ondisk_log_size":82069,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":478,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":27770664,"kb_used_data":24056844,"kb_used_omap":23,"kb_used_meta":3712936,"kb_avail":722190552,"statfs":{"total":767960285184,"available":739523125248,"internally_reserved":0,"allocated":24634208256,"data_stored":24628982787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24148,"internal_metadata":3802046892},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,4,2,4,5,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":736755712,"num_objects":11242,"num_object_clones":0,"num_object_copies":33726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1331,"num_objects_unfound":0,"num_objects_dirty":11242,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":11244,"num_write_kb":719552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1361,"num_bytes_recovered":89194496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004687"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":656,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.349680+0000","last_change":"2024-09-21T14:04:21.204685+0000","last_active":"2024-09-21T14:04:32.349680+0000","last_peered":"2024-09-21T14:04:32.349680+0000","last_clean":"2024-09-21T14:04:32.349680+0000","last_became_active":"2024-09-21T14:04:21.204299+0000","last_became_peered":"2024-09-21T14:04:21.204299+0000","last_unstale":"2024-09-21T14:04:32.349680+0000","last_undegraded":"2024-09-21T14:04:32.349680+0000","last_fullsized":"2024-09-21T14:04:32.349680+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:56.458125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:02:56.458125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"290'15","reported_seq":816,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.352568+0000","last_change":"2024-09-21T14:01:34.785322+0000","last_active":"2024-09-21T14:04:32.352568+0000","last_peered":"2024-09-21T14:04:32.352568+0000","last_clean":"2024-09-21T14:04:32.352568+0000","last_became_active":"2024-09-21T13:58:17.771220+0000","last_became_peered":"2024-09-21T13:58:17.771220+0000","last_unstale":"2024-09-21T14:04:32.352568+0000","last_undegraded":"2024-09-21T14:04:32.352568+0000","last_fullsized":"2024-09-21T14:04:32.352568+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'15","last_scrub_stamp":"2024-09-21T14:01:34.785267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:34.785267+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:54.084432+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":765,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.348623+0000","last_change":"2024-09-21T14:03:33.301558+0000","last_active":"2024-09-21T14:04:32.348623+0000","last_peered":"2024-09-21T14:04:32.348623+0000","last_clean":"2024-09-21T14:04:32.348623+0000","last_became_active":"2024-09-21T14:01:41.202168+0000","last_became_peered":"2024-09-21T14:01:41.202168+0000","last_unstale":"2024-09-21T14:04:32.348623+0000","last_undegraded":"2024-09-21T14:04:32.348623+0000","last_fullsized":"2024-09-21T14:04:32.348623+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:33.301487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:33.301487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:54.547538+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":733,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.353353+0000","last_change":"2024-09-21T14:01:41.203762+0000","last_active":"2024-09-21T14:04:32.353353+0000","last_peered":"2024-09-21T14:04:32.353353+0000","last_clean":"2024-09-21T14:04:32.353353+0000","last_became_active":"2024-09-21T14:01:41.203275+0000","last_became_peered":"2024-09-21T14:01:41.203275+0000","last_unstale":"2024-09-21T14:04:32.353353+0000","last_undegraded":"2024-09-21T14:04:32.353353+0000","last_fullsized":"2024-09-21T14:04:32.353353+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:00.320510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:00.320510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:46.994503+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":708,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T14:04:23.215224+0000","last_change":"2024-09-21T14:04:00.241774+0000","last_active":"2024-09-21T14:04:23.215224+0000","last_peered":"2024-09-21T14:04:23.215224+0000","last_clean":"2024-09-21T14:04:23.215224+0000","last_became_active":"2024-09-21T13:59:32.203340+0000","last_became_peered":"2024-09-21T13:59:32.203340+0000","last_unstale":"2024-09-21T14:04:23.215224+0000","last_undegraded":"2024-09-21T14:04:23.215224+0000","last_fullsized":"2024-09-21T14:04:23.215224+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:00.241562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:04:00.241562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:19.693680+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":713,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T14:04:23.215295+0000","last_change":"2024-09-21T13:58:17.778322+0000","last_active":"2024-09-21T14:04:23.215295+0000","last_peered":"2024-09-21T14:04:23.215295+0000","last_clean":"2024-09-21T14:04:23.215295+0000","last_became_active":"2024-09-21T13:58:17.777893+0000","last_became_peered":"2024-09-21T13:58:17.777893+0000","last_unstale":"2024-09-21T14:04:23.215295+0000","last_undegraded":"2024-09-21T14:04:23.215295+0000","last_fullsized":"2024-09-21T14:04:23.215295+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:07.001135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:55:07.001135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:55.260422+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"290'27","reported_seq":695,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T14:04:23.215048+0000","last_change":"2024-09-21T14:04:08.320458+0000","last_active":"2024-09-21T14:04:23.215048+0000","last_peered":"2024-09-21T14:04:23.215048+0000","last_clean":"2024-09-21T14:04:23.215048+0000","last_became_active":"2024-09-21T13:57:12.703469+0000","last_became_peered":"2024-09-21T13:57:12.703469+0000","last_unstale":"2024-09-21T14:04:23.215048+0000","last_undegraded":"2024-09-21T14:04:23.215048+0000","last_fullsized":"2024-09-21T14:04:23.215048+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'27","last_scrub_stamp":"2024-09-21T14:04:08.320339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:04:08.320339+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:32.005889+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":462,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.354815+0000","last_change":"2024-09-21T14:04:21.208603+0000","last_active":"2024-09-21T14:04:32.354815+0000","last_peered":"2024-09-21T14:04:32.354815+0000","last_clean":"2024-09-21T14:04:32.354815+0000","last_became_active":"2024-09-21T14:04:21.208237+0000","last_became_peered":"2024-09-21T14:04:21.208237+0000","last_unstale":"2024-09-21T14:04:32.354815+0000","last_undegraded":"2024-09-21T14:04:32.354815+0000","last_fullsized":"2024-09-21T14:04:32.354815+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:02.862787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:02.862787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":716,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.349597+0000","last_change":"2024-09-21T14:04:11.718218+0000","last_active":"2024-09-21T14:04:32.349597+0000","last_peered":"2024-09-21T14:04:32.349597+0000","last_clean":"2024-09-21T14:04:32.349597+0000","last_became_active":"2024-09-21T14:01:41.194517+0000","last_became_peered":"2024-09-21T14:01:41.194517+0000","last_unstale":"2024-09-21T14:04:32.349597+0000","last_undegraded":"2024-09-21T14:04:32.349597+0000","last_fullsized":"2024-09-21T14:04:32.349597+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:11.718143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:11.718143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:19.307011+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"290'15","reported_seq":553,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T14:04:23.214434+0000","last_change":"2024-09-21T14:04:21.228412+0000","last_active":"2024-09-21T14:04:23.214434+0000","last_peered":"2024-09-21T14:04:23.214434+0000","last_clean":"2024-09-21T14:04:23.214434+0000","last_became_active":"2024-09-21T14:04:21.208244+0000","last_became_peered":"2024-09-21T14:04:21.208244+0000","last_unstale":"2024-09-21T14:04:23.214434+0000","last_undegraded":"2024-09-21T14:04:23.214434+0000","last_fullsized":"2024-09-21T14:04:23.214434+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'15","last_scrub_stamp":"2024-09-21T14:00:02.475661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:00:02.475661+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":657,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.349487+0000","last_change":"2024-09-21T14:03:04.516642+0000","last_active":"2024-09-21T14:04:32.349487+0000","last_peered":"2024-09-21T14:04:32.349487+0000","last_clean":"2024-09-21T14:04:32.349487+0000","last_became_active":"2024-09-21T14:01:41.188248+0000","last_became_peered":"2024-09-21T14:01:41.188248+0000","last_unstale":"2024-09-21T14:04:32.349487+0000","last_undegraded":"2024-09-21T14:04:32.349487+0000","last_fullsized":"2024-09-21T14:04:32.349487+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:04.516583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:04.516583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":693,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.350321+0000","last_change":"2024-09-21T14:04:21.205796+0000","last_active":"2024-09-21T14:04:32.350321+0000","last_peered":"2024-09-21T14:04:32.350321+0000","last_clean":"2024-09-21T14:04:32.350321+0000","last_became_active":"2024-09-21T14:04:21.205416+0000","last_became_peered":"2024-09-21T14:04:21.205416+0000","last_unstale":"2024-09-21T14:04:32.350321+0000","last_undegraded":"2024-09-21T14:04:32.350321+0000","last_fullsized":"2024-09-21T14:04:32.350321+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:25.792985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:54:25.792985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":693,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.348651+0000","last_change":"2024-09-21T14:04:10.722881+0000","last_active":"2024-09-21T14:04:32.348651+0000","last_peered":"2024-09-21T14:04:32.348651+0000","last_clean":"2024-09-21T14:04:32.348651+0000","last_became_active":"2024-09-21T14:01:41.189373+0000","last_became_peered":"2024-09-21T14:01:41.189373+0000","last_unstale":"2024-09-21T14:04:32.348651+0000","last_undegraded":"2024-09-21T14:04:32.348651+0000","last_fullsized":"2024-09-21T14:04:32.348651+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:10.722799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:10.722799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:25.238119+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"290'7","reported_seq":553,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T14:04:23.215315+0000","last_change":"2024-09-21T14:04:21.231643+0000","last_active":"2024-09-21T14:04:23.215315+0000","last_peered":"2024-09-21T14:04:23.215315+0000","last_clean":"2024-09-21T14:04:23.215315+0000","last_became_active":"2024-09-21T14:04:21.213102+0000","last_became_peered":"2024-09-21T14:04:21.213102+0000","last_unstale":"2024-09-21T14:04:23.215315+0000","last_undegraded":"2024-09-21T14:04:23.215315+0000","last_fullsized":"2024-09-21T14:04:23.215315+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'7","last_scrub_stamp":"2024-09-21T14:01:57.563184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:57.563184+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":676,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T14:04:23.211067+0000","last_change":"2024-09-21T13:59:32.204565+0000","last_active":"2024-09-21T14:04:23.211067+0000","last_peered":"2024-09-21T14:04:23.211067+0000","last_clean":"2024-09-21T14:04:23.211067+0000","last_became_active":"2024-09-21T13:59:32.204174+0000","last_became_peered":"2024-09-21T13:59:32.204174+0000","last_unstale":"2024-09-21T14:04:23.211067+0000","last_undegraded":"2024-09-21T14:04:23.211067+0000","last_fullsized":"2024-09-21T14:04:23.211067+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:06.722293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:06.722293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:47.974782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":719,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.352399+0000","last_change":"2024-09-21T14:01:41.199085+0000","last_active":"2024-09-21T14:04:32.352399+0000","last_peered":"2024-09-21T14:04:32.352399+0000","last_clean":"2024-09-21T14:04:32.352399+0000","last_became_active":"2024-09-21T14:01:41.198672+0000","last_became_peered":"2024-09-21T14:01:41.198672+0000","last_unstale":"2024-09-21T14:04:32.352399+0000","last_undegraded":"2024-09-21T14:04:32.352399+0000","last_fullsized":"2024-09-21T14:04:32.352399+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:44.376215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:44.376215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":783,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.353309+0000","last_change":"2024-09-21T14:03:36.018910+0000","last_active":"2024-09-21T14:04:32.353309+0000","last_peered":"2024-09-21T14:04:32.353309+0000","last_clean":"2024-09-21T14:04:32.353309+0000","last_became_active":"2024-09-21T14:01:41.200360+0000","last_became_peered":"2024-09-21T14:01:41.200360+0000","last_unstale":"2024-09-21T14:04:32.353309+0000","last_undegraded":"2024-09-21T14:04:32.353309+0000","last_fullsized":"2024-09-21T14:04:32.353309+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:36.018851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:36.018851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:05.142781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":700,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.353291+0000","last_change":"2024-09-21T14:04:21.202574+0000","last_active":"2024-09-21T14:04:32.353291+0000","last_peered":"2024-09-21T14:04:32.353291+0000","last_clean":"2024-09-21T14:04:32.353291+0000","last_became_active":"2024-09-21T14:04:21.202178+0000","last_became_peered":"2024-09-21T14:04:21.202178+0000","last_unstale":"2024-09-21T14:04:32.353291+0000","last_undegraded":"2024-09-21T14:04:32.353291+0000","last_fullsized":"2024-09-21T14:04:32.353291+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:36.679321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:02:36.679321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"290'27","reported_seq":766,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.354422+0000","last_change":"2024-09-21T14:04:01.621388+0000","last_active":"2024-09-21T14:04:32.354422+0000","last_peered":"2024-09-21T14:04:32.354422+0000","last_clean":"2024-09-21T14:04:32.354422+0000","last_became_active":"2024-09-21T13:57:12.680783+0000","last_became_peered":"2024-09-21T13:57:12.680783+0000","last_unstale":"2024-09-21T14:04:32.354422+0000","last_undegraded":"2024-09-21T14:04:32.354422+0000","last_fullsized":"2024-09-21T14:04:32.354422+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'27","last_scrub_stamp":"2024-09-21T14:04:01.621303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:04:01.621303+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:23.163153+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":758,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.348643+0000","last_change":"2024-09-21T14:04:21.204283+0000","last_active":"2024-09-21T14:04:32.348643+0000","last_peered":"2024-09-21T14:04:32.348643+0000","last_clean":"2024-09-21T14:04:32.348643+0000","last_became_active":"2024-09-21T14:04:21.203937+0000","last_became_peered":"2024-09-21T14:04:21.203937+0000","last_unstale":"2024-09-21T14:04:32.348643+0000","last_undegraded":"2024-09-21T14:04:32.348643+0000","last_fullsized":"2024-09-21T14:04:32.348643+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:08.299252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:08.299252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":693,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T14:04:23.215604+0000","last_change":"2024-09-21T14:01:41.191486+0000","last_active":"2024-09-21T14:04:23.215604+0000","last_peered":"2024-09-21T14:04:23.215604+0000","last_clean":"2024-09-21T14:04:23.215604+0000","last_became_active":"2024-09-21T14:01:41.191116+0000","last_became_peered":"2024-09-21T14:01:41.191116+0000","last_unstale":"2024-09-21T14:04:23.215604+0000","last_undegraded":"2024-09-21T14:04:23.215604+0000","last_fullsized":"2024-09-21T14:04:23.215604+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:57.962266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:57.962266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:18.210525+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":753,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.353635+0000","last_change":"2024-09-21T14:00:51.495433+0000","last_active":"2024-09-21T14:04:32.353635+0000","last_peered":"2024-09-21T14:04:32.353635+0000","last_clean":"2024-09-21T14:04:32.353635+0000","last_became_active":"2024-09-21T13:52:04.316016+0000","last_became_peered":"2024-09-21T13:52:04.316016+0000","last_unstale":"2024-09-21T14:04:32.353635+0000","last_undegraded":"2024-09-21T14:04:32.353635+0000","last_fullsized":"2024-09-21T14:04:32.353635+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:00:51.495368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:00:51.495368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"290'18","reported_seq":698,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T14:04:23.217233+0000","last_change":"2024-09-21T13:59:23.041396+0000","last_active":"2024-09-21T14:04:23.217233+0000","last_peered":"2024-09-21T14:04:23.217233+0000","last_clean":"2024-09-21T14:04:23.217233+0000","last_became_active":"2024-09-21T13:57:12.684454+0000","last_became_peered":"2024-09-21T13:57:12.684454+0000","last_unstale":"2024-09-21T14:04:23.217233+0000","last_undegraded":"2024-09-21T14:04:23.217233+0000","last_fullsized":"2024-09-21T14:04:23.217233+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'18","last_scrub_stamp":"2024-09-21T13:59:23.041311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:23.041311+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":748,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.353039+0000","last_change":"2024-09-21T14:03:38.065257+0000","last_active":"2024-09-21T14:04:32.353039+0000","last_peered":"2024-09-21T14:04:32.353039+0000","last_clean":"2024-09-21T14:04:32.353039+0000","last_became_active":"2024-09-21T13:57:12.681897+0000","last_became_peered":"2024-09-21T13:57:12.681897+0000","last_unstale":"2024-09-21T14:04:32.353039+0000","last_undegraded":"2024-09-21T14:04:32.353039+0000","last_fullsized":"2024-09-21T14:04:32.353039+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:38.065196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:38.065196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:03.431512+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":668,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T14:04:23.211842+0000","last_change":"2024-09-21T14:03:28.933216+0000","last_active":"2024-09-21T14:04:23.211842+0000","last_peered":"2024-09-21T14:04:23.211842+0000","last_clean":"2024-09-21T14:04:23.211842+0000","last_became_active":"2024-09-21T14:01:41.181946+0000","last_became_peered":"2024-09-21T14:01:41.181946+0000","last_unstale":"2024-09-21T14:04:23.211842+0000","last_undegraded":"2024-09-21T14:04:23.211842+0000","last_fullsized":"2024-09-21T14:04:23.211842+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T14:03:28.933143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:28.933143+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:46.365067+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":774,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.348796+0000","last_change":"2024-09-21T14:04:21.215322+0000","last_active":"2024-09-21T14:04:32.348796+0000","last_peered":"2024-09-21T14:04:32.348796+0000","last_clean":"2024-09-21T14:04:32.348796+0000","last_became_active":"2024-09-21T14:04:21.200889+0000","last_became_peered":"2024-09-21T14:04:21.200889+0000","last_unstale":"2024-09-21T14:04:32.348796+0000","last_undegraded":"2024-09-21T14:04:32.348796+0000","last_fullsized":"2024-09-21T14:04:32.348796+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T14:01:58.994811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:01:58.994811+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"290'20","reported_seq":640,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T14:04:23.212716+0000","last_change":"2024-09-21T13:59:32.242771+0000","last_active":"2024-09-21T14:04:23.212716+0000","last_peered":"2024-09-21T14:04:23.212716+0000","last_clean":"2024-09-21T14:04:23.212716+0000","last_became_active":"2024-09-21T13:59:32.213865+0000","last_became_peered":"2024-09-21T13:59:32.213865+0000","last_unstale":"2024-09-21T14:04:23.212716+0000","last_undegraded":"2024-09-21T14:04:23.212716+0000","last_fullsized":"2024-09-21T14:04:23.212716+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:57:27.578738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:57:27.578738+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:40.978862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"409'13","reported_seq":412,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:33.316930+0000","last_change":"2024-09-21T14:04:21.222657+0000","last_active":"2024-09-21T14:04:33.316930+0000","last_peered":"2024-09-21T14:04:33.316930+0000","last_clean":"2024-09-21T14:04:33.316930+0000","last_became_active":"2024-09-21T14:04:21.206136+0000","last_became_peered":"2024-09-21T14:04:21.206136+0000","last_unstale":"2024-09-21T14:04:33.316930+0000","last_undegraded":"2024-09-21T14:04:33.316930+0000","last_fullsized":"2024-09-21T14:04:33.316930+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'12","last_scrub_stamp":"2024-09-21T13:59:36.012693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:36.012693+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:27.834286+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"412'7966","reported_seq":8112,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:33.713561+0000","last_change":"2024-09-21T14:04:09.971242+0000","last_active":"2024-09-21T14:04:33.713561+0000","last_peered":"2024-09-21T14:04:33.713561+0000","last_clean":"2024-09-21T14:04:33.713561+0000","last_became_active":"2024-09-21T14:02:56.559536+0000","last_became_peered":"2024-09-21T14:02:56.559536+0000","last_unstale":"2024-09-21T14:04:33.713561+0000","last_undegraded":"2024-09-21T14:04:33.713561+0000","last_fullsized":"2024-09-21T14:04:33.713561+0000","mapping_epoch":381,"log_start":"390'2700","ondisk_log_start":"390'2700","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"406'6478","last_scrub_stamp":"2024-09-21T14:04:09.971039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:04:09.971039+0000","objects_scrubbed":6417,"log_size":5266,"log_dups_size":0,"ondisk_log_size":5266,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:27.890235+0000","scrub_duration":1835,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":522059776,"num_objects":7966,"num_object_clones":0,"num_object_copies":23898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7966,"num_write_kb":509824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"75'79","reported_seq":166,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.348278+0000","last_change":"2024-09-21T14:03:59.776621+0000","last_active":"2024-09-21T14:04:32.348278+0000","last_peered":"2024-09-21T14:04:32.348278+0000","last_clean":"2024-09-21T14:04:32.348278+0000","last_became_active":"2024-09-21T14:01:41.246888+0000","last_became_peered":"2024-09-21T14:01:41.246888+0000","last_unstale":"2024-09-21T14:04:32.348278+0000","last_undegraded":"2024-09-21T14:04:32.348278+0000","last_fullsized":"2024-09-21T14:04:32.348278+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:03:59.776552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:03:59.776552+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:21.450064+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"290'27","reported_seq":817,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.349008+0000","last_change":"2024-09-21T14:04:03.259647+0000","last_active":"2024-09-21T14:04:32.349008+0000","last_peered":"2024-09-21T14:04:32.349008+0000","last_clean":"2024-09-21T14:04:32.349008+0000","last_became_active":"2024-09-21T13:57:12.695609+0000","last_became_peered":"2024-09-21T13:57:12.695609+0000","last_unstale":"2024-09-21T14:04:32.349008+0000","last_undegraded":"2024-09-21T14:04:32.349008+0000","last_fullsized":"2024-09-21T14:04:32.349008+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'27","last_scrub_stamp":"2024-09-21T14:04:03.259554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:04:03.259554+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:20.677105+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":693,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T14:04:23.213765+0000","last_change":"2024-09-21T14:04:01.183193+0000","last_active":"2024-09-21T14:04:23.213765+0000","last_peered":"2024-09-21T14:04:23.213765+0000","last_clean":"2024-09-21T14:04:23.213765+0000","last_became_active":"2024-09-21T13:59:32.205350+0000","last_became_peered":"2024-09-21T13:59:32.205350+0000","last_unstale":"2024-09-21T14:04:23.213765+0000","last_undegraded":"2024-09-21T14:04:23.213765+0000","last_fullsized":"2024-09-21T14:04:23.213765+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:01.183123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:01.183123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:13.677443+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"409'11","reported_seq":795,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.350039+0000","last_change":"2024-09-21T14:04:21.211284+0000","last_active":"2024-09-21T14:04:32.350039+0000","last_peered":"2024-09-21T14:04:32.350039+0000","last_clean":"2024-09-21T14:04:32.350039+0000","last_became_active":"2024-09-21T14:04:21.206029+0000","last_became_peered":"2024-09-21T14:04:21.206029+0000","last_unstale":"2024-09-21T14:04:32.350039+0000","last_undegraded":"2024-09-21T14:04:32.350039+0000","last_fullsized":"2024-09-21T14:04:32.350039+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'10","last_scrub_stamp":"2024-09-21T14:02:59.831128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:02:59.831128+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":621,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.349091+0000","last_change":"2024-09-21T14:04:21.203815+0000","last_active":"2024-09-21T14:04:32.349091+0000","last_peered":"2024-09-21T14:04:32.349091+0000","last_clean":"2024-09-21T14:04:32.349091+0000","last_became_active":"2024-09-21T14:04:21.203505+0000","last_became_peered":"2024-09-21T14:04:21.203505+0000","last_unstale":"2024-09-21T14:04:32.349091+0000","last_undegraded":"2024-09-21T14:04:32.349091+0000","last_fullsized":"2024-09-21T14:04:32.349091+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:52.244483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:52.244483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"412'7659","reported_seq":7801,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:33.726709+0000","last_change":"2024-09-21T14:04:08.894792+0000","last_active":"2024-09-21T14:04:33.726709+0000","last_peered":"2024-09-21T14:04:33.726709+0000","last_clean":"2024-09-21T14:04:33.726709+0000","last_became_active":"2024-09-21T14:02:56.562524+0000","last_became_peered":"2024-09-21T14:02:56.562524+0000","last_unstale":"2024-09-21T14:04:33.726709+0000","last_undegraded":"2024-09-21T14:04:33.726709+0000","last_fullsized":"2024-09-21T14:04:33.726709+0000","mapping_epoch":381,"log_start":"390'2400","ondisk_log_start":"390'2400","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"406'6081","last_scrub_stamp":"2024-09-21T14:04:08.894653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:04:08.894653+0000","objects_scrubbed":6008,"log_size":5259,"log_dups_size":0,"ondisk_log_size":5259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:32.085642+0000","scrub_duration":1756,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":501940224,"num_objects":7659,"num_object_clones":0,"num_object_copies":22977,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7659,"num_write_kb":490176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"75'79","reported_seq":211,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.353738+0000","last_change":"2024-09-21T14:02:06.395083+0000","last_active":"2024-09-21T14:04:32.353738+0000","last_peered":"2024-09-21T14:04:32.353738+0000","last_clean":"2024-09-21T14:04:32.353738+0000","last_became_active":"2024-09-21T14:00:41.137756+0000","last_became_peered":"2024-09-21T14:00:41.137756+0000","last_unstale":"2024-09-21T14:04:32.353738+0000","last_undegraded":"2024-09-21T14:04:32.353738+0000","last_fullsized":"2024-09-21T14:04:32.353738+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:02:06.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:02:06.395011+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:55.076258+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"290'30","reported_seq":734,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T14:04:23.213861+0000","last_change":"2024-09-21T14:04:21.228589+0000","last_active":"2024-09-21T14:04:23.213861+0000","last_peered":"2024-09-21T14:04:23.213861+0000","last_clean":"2024-09-21T14:04:23.213861+0000","last_became_active":"2024-09-21T14:04:21.208982+0000","last_became_peered":"2024-09-21T14:04:21.208982+0000","last_unstale":"2024-09-21T14:04:23.213861+0000","last_undegraded":"2024-09-21T14:04:23.213861+0000","last_fullsized":"2024-09-21T14:04:23.213861+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'18","last_scrub_stamp":"2024-09-21T13:54:19.001030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:19.001030+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:23.192760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":645,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T14:04:23.215827+0000","last_change":"2024-09-21T14:04:02.214555+0000","last_active":"2024-09-21T14:04:23.215827+0000","last_peered":"2024-09-21T14:04:23.215827+0000","last_clean":"2024-09-21T14:04:23.215827+0000","last_became_active":"2024-09-21T13:59:32.211004+0000","last_became_peered":"2024-09-21T13:59:32.211004+0000","last_unstale":"2024-09-21T14:04:23.215827+0000","last_undegraded":"2024-09-21T14:04:23.215827+0000","last_fullsized":"2024-09-21T14:04:23.215827+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:02.214464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:02.214464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:04.300176+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":757,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.352823+0000","last_change":"2024-09-21T13:59:34.293330+0000","last_active":"2024-09-21T14:04:32.352823+0000","last_peered":"2024-09-21T14:04:32.352823+0000","last_clean":"2024-09-21T14:04:32.352823+0000","last_became_active":"2024-09-21T13:59:32.220304+0000","last_became_peered":"2024-09-21T13:59:32.220304+0000","last_unstale":"2024-09-21T14:04:32.352823+0000","last_undegraded":"2024-09-21T14:04:32.352823+0000","last_fullsized":"2024-09-21T14:04:32.352823+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:34.292892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:34.292892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":391,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T14:04:23.214528+0000","last_change":"2024-09-21T14:04:21.210131+0000","last_active":"2024-09-21T14:04:23.214528+0000","last_peered":"2024-09-21T14:04:23.214528+0000","last_clean":"2024-09-21T14:04:23.214528+0000","last_became_active":"2024-09-21T14:04:21.207782+0000","last_became_peered":"2024-09-21T14:04:21.207782+0000","last_unstale":"2024-09-21T14:04:23.214528+0000","last_undegraded":"2024-09-21T14:04:23.214528+0000","last_fullsized":"2024-09-21T14:04:23.214528+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:23.334162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:23.334162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"412'7920","reported_seq":7992,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:33.561091+0000","last_change":"2024-09-21T14:02:56.557348+0000","last_active":"2024-09-21T14:04:33.561091+0000","last_peered":"2024-09-21T14:04:33.561091+0000","last_clean":"2024-09-21T14:04:33.561091+0000","last_became_active":"2024-09-21T14:02:56.557055+0000","last_became_peered":"2024-09-21T14:02:56.557055+0000","last_unstale":"2024-09-21T14:04:33.561091+0000","last_undegraded":"2024-09-21T14:04:33.561091+0000","last_fullsized":"2024-09-21T14:04:33.561091+0000","mapping_epoch":381,"log_start":"390'2810","ondisk_log_start":"390'2810","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":5110,"log_dups_size":0,"ondisk_log_size":5110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":519045120,"num_objects":7920,"num_object_clones":0,"num_object_copies":23760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7920,"num_write_kb":506880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"75'79","reported_seq":212,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T14:04:23.214614+0000","last_change":"2024-09-21T14:01:42.177646+0000","last_active":"2024-09-21T14:04:23.214614+0000","last_peered":"2024-09-21T14:04:23.214614+0000","last_clean":"2024-09-21T14:04:23.214614+0000","last_became_active":"2024-09-21T14:01:42.173042+0000","last_became_peered":"2024-09-21T14:01:42.173042+0000","last_unstale":"2024-09-21T14:04:23.214614+0000","last_undegraded":"2024-09-21T14:04:23.214614+0000","last_fullsized":"2024-09-21T14:04:23.214614+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:04:45.239779+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":709030,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"290'28","reported_seq":854,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.352935+0000","last_change":"2024-09-21T14:03:27.047111+0000","last_active":"2024-09-21T14:04:32.352935+0000","last_peered":"2024-09-21T14:04:32.352935+0000","last_clean":"2024-09-21T14:04:32.352935+0000","last_became_active":"2024-09-21T14:01:41.200859+0000","last_became_peered":"2024-09-21T14:01:41.200859+0000","last_unstale":"2024-09-21T14:04:32.352935+0000","last_undegraded":"2024-09-21T14:04:32.352935+0000","last_fullsized":"2024-09-21T14:04:32.352935+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'28","last_scrub_stamp":"2024-09-21T14:03:27.046906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:03:27.046906+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":32,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.348265+0000","last_change":"2024-09-21T14:04:21.216890+0000","last_active":"2024-09-21T14:04:32.348265+0000","last_peered":"2024-09-21T14:04:32.348265+0000","last_clean":"2024-09-21T14:04:32.348265+0000","last_became_active":"2024-09-21T14:04:21.205354+0000","last_became_peered":"2024-09-21T14:04:21.205354+0000","last_unstale":"2024-09-21T14:04:32.348265+0000","last_undegraded":"2024-09-21T14:04:32.348265+0000","last_fullsized":"2024-09-21T14:04:32.348265+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:57:38.046627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:38.046627+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"241'10","reported_seq":813,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T14:04:23.313377+0000","last_change":"2024-09-21T14:03:03.228551+0000","last_active":"2024-09-21T14:04:23.313377+0000","last_peered":"2024-09-21T14:04:23.313377+0000","last_clean":"2024-09-21T14:04:23.313377+0000","last_became_active":"2024-09-21T13:57:12.686651+0000","last_became_peered":"2024-09-21T13:57:12.686651+0000","last_unstale":"2024-09-21T14:04:23.313377+0000","last_undegraded":"2024-09-21T14:04:23.313377+0000","last_fullsized":"2024-09-21T14:04:23.313377+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'10","last_scrub_stamp":"2024-09-21T14:03:03.228443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:03.228443+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":701,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T14:04:23.211697+0000","last_change":"2024-09-21T13:59:32.210257+0000","last_active":"2024-09-21T14:04:23.211697+0000","last_peered":"2024-09-21T14:04:23.211697+0000","last_clean":"2024-09-21T14:04:23.211697+0000","last_became_active":"2024-09-21T13:59:32.209615+0000","last_became_peered":"2024-09-21T13:59:32.209615+0000","last_unstale":"2024-09-21T14:04:23.211697+0000","last_undegraded":"2024-09-21T14:04:23.211697+0000","last_fullsized":"2024-09-21T14:04:23.211697+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:37.548064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:37.548064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:43.986906+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.c","version":"411'7503","reported_seq":7574,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T14:04:31.229094+0000","last_change":"2024-09-21T14:02:56.563139+0000","last_active":"2024-09-21T14:04:31.229094+0000","last_peered":"2024-09-21T14:04:31.229094+0000","last_clean":"2024-09-21T14:04:31.229094+0000","last_became_active":"2024-09-21T14:02:56.562806+0000","last_became_peered":"2024-09-21T14:02:56.562806+0000","last_unstale":"2024-09-21T14:04:31.229094+0000","last_undegraded":"2024-09-21T14:04:31.229094+0000","last_fullsized":"2024-09-21T14:04:31.229094+0000","mapping_epoch":381,"log_start":"392'3009","ondisk_log_start":"392'3009","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":4494,"log_dups_size":0,"ondisk_log_size":4494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":491716608,"num_objects":7503,"num_object_clones":0,"num_object_copies":22509,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7503,"num_write_kb":480192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"75'79","reported_seq":784,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.356091+0000","last_change":"2024-09-21T14:02:05.693250+0000","last_active":"2024-09-21T14:04:32.356091+0000","last_peered":"2024-09-21T14:04:32.356091+0000","last_clean":"2024-09-21T14:04:32.356091+0000","last_became_active":"2024-09-21T14:00:41.083425+0000","last_became_peered":"2024-09-21T14:00:41.083425+0000","last_unstale":"2024-09-21T14:04:32.356091+0000","last_undegraded":"2024-09-21T14:04:32.356091+0000","last_fullsized":"2024-09-21T14:04:32.356091+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:02:05.693193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:02:05.693193+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:38.858726+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"290'20","reported_seq":70,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.354842+0000","last_change":"2024-09-21T14:03:31.791421+0000","last_active":"2024-09-21T14:04:32.354842+0000","last_peered":"2024-09-21T14:04:32.354842+0000","last_clean":"2024-09-21T14:04:32.354842+0000","last_became_active":"2024-09-21T14:03:29.158218+0000","last_became_peered":"2024-09-21T14:03:29.158218+0000","last_unstale":"2024-09-21T14:04:32.354842+0000","last_undegraded":"2024-09-21T14:04:32.354842+0000","last_fullsized":"2024-09-21T14:04:32.354842+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'20","last_scrub_stamp":"2024-09-21T14:03:31.791317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:03:31.791317+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:39.052613+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"61'1","reported_seq":676,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.349177+0000","last_change":"2024-09-21T14:01:41.195580+0000","last_active":"2024-09-21T14:04:32.349177+0000","last_peered":"2024-09-21T14:04:32.349177+0000","last_clean":"2024-09-21T14:04:32.349177+0000","last_became_active":"2024-09-21T14:01:41.195268+0000","last_became_peered":"2024-09-21T14:01:41.195268+0000","last_unstale":"2024-09-21T14:04:32.349177+0000","last_undegraded":"2024-09-21T14:04:32.349177+0000","last_fullsized":"2024-09-21T14:04:32.349177+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:58:48.407734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:48.407734+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"346'15","reported_seq":393,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:33.316755+0000","last_change":"2024-09-21T14:01:41.209662+0000","last_active":"2024-09-21T14:04:33.316755+0000","last_peered":"2024-09-21T14:04:33.316755+0000","last_clean":"2024-09-21T14:04:33.316755+0000","last_became_active":"2024-09-21T14:01:41.209296+0000","last_became_peered":"2024-09-21T14:01:41.209296+0000","last_unstale":"2024-09-21T14:04:33.316755+0000","last_undegraded":"2024-09-21T14:04:33.316755+0000","last_fullsized":"2024-09-21T14:04:33.316755+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'12","last_scrub_stamp":"2024-09-21T13:57:30.133392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:30.133392+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:43.907864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":555,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T14:04:23.214000+0000","last_change":"2024-09-21T13:58:58.730439+0000","last_active":"2024-09-21T14:04:23.214000+0000","last_peered":"2024-09-21T14:04:23.214000+0000","last_clean":"2024-09-21T14:04:23.214000+0000","last_became_active":"2024-09-21T13:57:12.704897+0000","last_became_peered":"2024-09-21T13:57:12.704897+0000","last_unstale":"2024-09-21T14:04:23.214000+0000","last_undegraded":"2024-09-21T14:04:23.214000+0000","last_fullsized":"2024-09-21T14:04:23.214000+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:58.730368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:58.730368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.d","version":"412'7839","reported_seq":7991,"reported_epoch":412,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-21T14:04:33.594691+0000","last_change":"2024-09-21T14:04:22.307554+0000","last_active":"2024-09-21T14:04:33.594691+0000","last_peered":"2024-09-21T14:04:33.594691+0000","last_clean":"2024-09-21T14:04:20.166030+0000","last_became_active":"2024-09-21T14:04:22.281826+0000","last_became_peered":"2024-09-21T14:04:22.281826+0000","last_unstale":"2024-09-21T14:04:33.594691+0000","last_undegraded":"2024-09-21T14:04:33.594691+0000","last_fullsized":"2024-09-21T14:04:33.594691+0000","mapping_epoch":409,"log_start":"390'2668","ondisk_log_start":"390'2668","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"406'6403","last_scrub_stamp":"2024-09-21T14:04:11.019087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:04:11.019087+0000","objects_scrubbed":6346,"log_size":5171,"log_dups_size":0,"ondisk_log_size":5171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1942,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":513736704,"num_objects":7839,"num_object_clones":0,"num_object_copies":23517,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7839,"num_objects_unfound":0,"num_objects_dirty":7839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7839,"num_write_kb":501696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,0],"avail_no_missing":["6","0","4"],"object_location_counts":[{"shards":"0,4,6","objects":7839}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"75'79","reported_seq":792,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.355142+0000","last_change":"2024-09-21T14:03:34.762945+0000","last_active":"2024-09-21T14:04:32.355142+0000","last_peered":"2024-09-21T14:04:32.355142+0000","last_clean":"2024-09-21T14:04:32.355142+0000","last_became_active":"2024-09-21T14:00:41.073047+0000","last_became_peered":"2024-09-21T14:00:41.073047+0000","last_unstale":"2024-09-21T14:04:32.355142+0000","last_undegraded":"2024-09-21T14:04:32.355142+0000","last_fullsized":"2024-09-21T14:04:32.355142+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:03:34.762689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:03:34.762689+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:41.051553+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"290'29","reported_seq":690,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.349624+0000","last_change":"2024-09-21T14:03:08.399687+0000","last_active":"2024-09-21T14:04:32.349624+0000","last_peered":"2024-09-21T14:04:32.349624+0000","last_clean":"2024-09-21T14:04:32.349624+0000","last_became_active":"2024-09-21T14:01:41.186960+0000","last_became_peered":"2024-09-21T14:01:41.186960+0000","last_unstale":"2024-09-21T14:04:32.349624+0000","last_undegraded":"2024-09-21T14:04:32.349624+0000","last_fullsized":"2024-09-21T14:04:32.349624+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'29","last_scrub_stamp":"2024-09-21T14:03:08.399606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:03:08.399606+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":758,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.352767+0000","last_change":"2024-09-21T14:01:41.185706+0000","last_active":"2024-09-21T14:04:32.352767+0000","last_peered":"2024-09-21T14:04:32.352767+0000","last_clean":"2024-09-21T14:04:32.352767+0000","last_became_active":"2024-09-21T14:01:41.185400+0000","last_became_peered":"2024-09-21T14:01:41.185400+0000","last_unstale":"2024-09-21T14:04:32.352767+0000","last_undegraded":"2024-09-21T14:04:32.352767+0000","last_fullsized":"2024-09-21T14:04:32.352767+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:16.712990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:16.712990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:39.515666+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":342,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.355379+0000","last_change":"2024-09-21T14:04:03.070260+0000","last_active":"2024-09-21T14:04:32.355379+0000","last_peered":"2024-09-21T14:04:32.355379+0000","last_clean":"2024-09-21T14:04:32.355379+0000","last_became_active":"2024-09-21T14:01:41.193973+0000","last_became_peered":"2024-09-21T14:01:41.193973+0000","last_unstale":"2024-09-21T14:04:32.355379+0000","last_undegraded":"2024-09-21T14:04:32.355379+0000","last_fullsized":"2024-09-21T14:04:32.355379+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:03.069973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:04:03.069973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:26.954137+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":749,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.348518+0000","last_change":"2024-09-21T13:58:43.952014+0000","last_active":"2024-09-21T14:04:32.348518+0000","last_peered":"2024-09-21T14:04:32.348518+0000","last_clean":"2024-09-21T14:04:32.348518+0000","last_became_active":"2024-09-21T13:58:17.757628+0000","last_became_peered":"2024-09-21T13:58:17.757628+0000","last_unstale":"2024-09-21T14:04:32.348518+0000","last_undegraded":"2024-09-21T14:04:32.348518+0000","last_fullsized":"2024-09-21T14:04:32.348518+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:43.951941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:43.951941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:54.273193+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"412'7745","reported_seq":7860,"reported_epoch":412,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-21T14:04:33.525298+0000","last_change":"2024-09-21T14:04:22.387863+0000","last_active":"2024-09-21T14:04:33.525298+0000","last_peered":"2024-09-21T14:04:33.525298+0000","last_clean":"2024-09-21T14:04:20.165921+0000","last_became_active":"2024-09-21T14:04:22.361054+0000","last_became_peered":"2024-09-21T14:04:22.361054+0000","last_unstale":"2024-09-21T14:04:33.525298+0000","last_undegraded":"2024-09-21T14:04:33.525298+0000","last_fullsized":"2024-09-21T14:04:33.525298+0000","mapping_epoch":409,"log_start":"390'2648","ondisk_log_start":"390'2648","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":3149,"log_size":5097,"log_dups_size":0,"ondisk_log_size":5097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":507576320,"num_objects":7745,"num_object_clones":0,"num_object_copies":23235,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7745,"num_objects_unfound":0,"num_objects_dirty":7745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7745,"num_write_kb":495680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,0],"avail_no_missing":["6","0","5"],"object_location_counts":[{"shards":"0,5,6","objects":7745}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"75'79","reported_seq":158,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.352717+0000","last_change":"2024-09-21T14:01:41.199377+0000","last_active":"2024-09-21T14:04:32.352717+0000","last_peered":"2024-09-21T14:04:32.352717+0000","last_clean":"2024-09-21T14:04:32.352717+0000","last_became_active":"2024-09-21T14:01:41.198750+0000","last_became_peered":"2024-09-21T14:01:41.198750+0000","last_unstale":"2024-09-21T14:04:32.352717+0000","last_undegraded":"2024-09-21T14:04:32.352717+0000","last_fullsized":"2024-09-21T14:04:32.352717+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:04:54.631861+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"290'44","reported_seq":1020,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.352674+0000","last_change":"2024-09-21T14:02:57.316829+0000","last_active":"2024-09-21T14:04:32.352674+0000","last_peered":"2024-09-21T14:04:32.352674+0000","last_clean":"2024-09-21T14:04:32.352674+0000","last_became_active":"2024-09-21T13:57:12.684690+0000","last_became_peered":"2024-09-21T13:57:12.684690+0000","last_unstale":"2024-09-21T14:04:32.352674+0000","last_undegraded":"2024-09-21T14:04:32.352674+0000","last_fullsized":"2024-09-21T14:04:32.352674+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'44","last_scrub_stamp":"2024-09-21T14:02:57.316763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:57.316763+0000","objects_scrubbed":10,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:58.296085+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":773,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.353329+0000","last_change":"2024-09-21T14:04:11.158776+0000","last_active":"2024-09-21T14:04:32.353329+0000","last_peered":"2024-09-21T14:04:32.353329+0000","last_clean":"2024-09-21T14:04:32.353329+0000","last_became_active":"2024-09-21T13:58:17.770937+0000","last_became_peered":"2024-09-21T13:58:17.770937+0000","last_unstale":"2024-09-21T14:04:32.353329+0000","last_undegraded":"2024-09-21T14:04:32.353329+0000","last_fullsized":"2024-09-21T14:04:32.353329+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:11.158703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:11.158703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:26.630567+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":703,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.348221+0000","last_change":"2024-09-21T14:03:30.390161+0000","last_active":"2024-09-21T14:04:32.348221+0000","last_peered":"2024-09-21T14:04:32.348221+0000","last_clean":"2024-09-21T14:04:32.348221+0000","last_became_active":"2024-09-21T14:01:41.247642+0000","last_became_peered":"2024-09-21T14:01:41.247642+0000","last_unstale":"2024-09-21T14:04:32.348221+0000","last_undegraded":"2024-09-21T14:04:32.348221+0000","last_fullsized":"2024-09-21T14:04:32.348221+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-21T14:03:30.390086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T14:03:30.390086+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:32.519440+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":683,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.348228+0000","last_change":"2024-09-21T14:04:21.204705+0000","last_active":"2024-09-21T14:04:32.348228+0000","last_peered":"2024-09-21T14:04:32.348228+0000","last_clean":"2024-09-21T14:04:32.348228+0000","last_became_active":"2024-09-21T14:04:21.204315+0000","last_became_peered":"2024-09-21T14:04:21.204315+0000","last_unstale":"2024-09-21T14:04:32.348228+0000","last_undegraded":"2024-09-21T14:04:32.348228+0000","last_fullsized":"2024-09-21T14:04:32.348228+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:07.021059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:55:07.021059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":672,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T14:04:23.215642+0000","last_change":"2024-09-21T14:03:02.238409+0000","last_active":"2024-09-21T14:04:23.215642+0000","last_peered":"2024-09-21T14:04:23.215642+0000","last_clean":"2024-09-21T14:04:23.215642+0000","last_became_active":"2024-09-21T13:59:32.208980+0000","last_became_peered":"2024-09-21T13:59:32.208980+0000","last_unstale":"2024-09-21T14:04:23.215642+0000","last_undegraded":"2024-09-21T14:04:23.215642+0000","last_fullsized":"2024-09-21T14:04:23.215642+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:02.238357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:02.238357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"412'7859","reported_seq":8001,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:35.793613+0000","last_change":"2024-09-21T14:04:06.243298+0000","last_active":"2024-09-21T14:04:35.793613+0000","last_peered":"2024-09-21T14:04:35.793613+0000","last_clean":"2024-09-21T14:04:35.793613+0000","last_became_active":"2024-09-21T14:02:56.558773+0000","last_became_peered":"2024-09-21T14:02:56.558773+0000","last_unstale":"2024-09-21T14:04:35.793613+0000","last_undegraded":"2024-09-21T14:04:35.793613+0000","last_fullsized":"2024-09-21T14:04:35.793613+0000","mapping_epoch":381,"log_start":"391'2969","ondisk_log_start":"391'2969","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'5948","last_scrub_stamp":"2024-09-21T14:04:06.243212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:04:06.243212+0000","objects_scrubbed":5906,"log_size":4890,"log_dups_size":0,"ondisk_log_size":4890,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:18.713403+0000","scrub_duration":1458,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":515047424,"num_objects":7859,"num_object_clones":0,"num_object_copies":23577,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7859,"num_write_kb":502976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"75'79","reported_seq":795,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T14:04:23.210933+0000","last_change":"2024-09-21T14:04:21.206885+0000","last_active":"2024-09-21T14:04:23.210933+0000","last_peered":"2024-09-21T14:04:23.210933+0000","last_clean":"2024-09-21T14:04:23.210933+0000","last_became_active":"2024-09-21T14:04:21.206472+0000","last_became_peered":"2024-09-21T14:04:21.206472+0000","last_unstale":"2024-09-21T14:04:23.210933+0000","last_undegraded":"2024-09-21T14:04:23.210933+0000","last_fullsized":"2024-09-21T14:04:23.210933+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:05:23.132366+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"290'17","reported_seq":803,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.349107+0000","last_change":"2024-09-21T14:04:21.224066+0000","last_active":"2024-09-21T14:04:32.349107+0000","last_peered":"2024-09-21T14:04:32.349107+0000","last_clean":"2024-09-21T14:04:32.349107+0000","last_became_active":"2024-09-21T14:04:21.205738+0000","last_became_peered":"2024-09-21T14:04:21.205738+0000","last_unstale":"2024-09-21T14:04:32.349107+0000","last_undegraded":"2024-09-21T14:04:32.349107+0000","last_fullsized":"2024-09-21T14:04:32.349107+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'13","last_scrub_stamp":"2024-09-21T13:58:11.619848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:11.619848+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":801,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.349341+0000","last_change":"2024-09-21T14:04:10.193524+0000","last_active":"2024-09-21T14:04:32.349341+0000","last_peered":"2024-09-21T14:04:32.349341+0000","last_clean":"2024-09-21T14:04:32.349341+0000","last_became_active":"2024-09-21T13:58:17.759671+0000","last_became_peered":"2024-09-21T13:58:17.759671+0000","last_unstale":"2024-09-21T14:04:32.349341+0000","last_undegraded":"2024-09-21T14:04:32.349341+0000","last_fullsized":"2024-09-21T14:04:32.349341+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:10.193318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:10.193318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:25.269486+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":675,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.358050+0000","last_change":"2024-09-21T13:58:41.644876+0000","last_active":"2024-09-21T14:04:32.358050+0000","last_peered":"2024-09-21T14:04:32.358050+0000","last_clean":"2024-09-21T14:04:32.358050+0000","last_became_active":"2024-09-21T13:57:12.682147+0000","last_became_peered":"2024-09-21T13:57:12.682147+0000","last_unstale":"2024-09-21T14:04:32.358050+0000","last_undegraded":"2024-09-21T14:04:32.358050+0000","last_fullsized":"2024-09-21T14:04:32.358050+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:58:41.644789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:58:41.644789+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:17.567651+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":653,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.356691+0000","last_change":"2024-09-21T14:01:50.880840+0000","last_active":"2024-09-21T14:04:32.356691+0000","last_peered":"2024-09-21T14:04:32.356691+0000","last_clean":"2024-09-21T14:04:32.356691+0000","last_became_active":"2024-09-21T14:01:41.182000+0000","last_became_peered":"2024-09-21T14:01:41.182000+0000","last_unstale":"2024-09-21T14:04:32.356691+0000","last_undegraded":"2024-09-21T14:04:32.356691+0000","last_fullsized":"2024-09-21T14:04:32.356691+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:50.880471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:01:50.880471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:20.548012+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":709,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T14:04:23.211303+0000","last_change":"2024-09-21T13:59:32.222791+0000","last_active":"2024-09-21T14:04:23.211303+0000","last_peered":"2024-09-21T14:04:23.211303+0000","last_clean":"2024-09-21T14:04:23.211303+0000","last_became_active":"2024-09-21T13:59:32.221502+0000","last_became_peered":"2024-09-21T13:59:32.221502+0000","last_unstale":"2024-09-21T14:04:23.211303+0000","last_undegraded":"2024-09-21T14:04:23.211303+0000","last_fullsized":"2024-09-21T14:04:23.211303+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:14.673582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:14.673582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:48.983706+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"412'7853","reported_seq":7995,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.946110+0000","last_change":"2024-09-21T14:04:06.996232+0000","last_active":"2024-09-21T14:04:32.946110+0000","last_peered":"2024-09-21T14:04:32.946110+0000","last_clean":"2024-09-21T14:04:32.946110+0000","last_became_active":"2024-09-21T14:02:56.558826+0000","last_became_peered":"2024-09-21T14:02:56.558826+0000","last_unstale":"2024-09-21T14:04:32.946110+0000","last_undegraded":"2024-09-21T14:04:32.946110+0000","last_fullsized":"2024-09-21T14:04:32.946110+0000","mapping_epoch":381,"log_start":"391'2882","ondisk_log_start":"391'2882","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'6143","last_scrub_stamp":"2024-09-21T14:04:06.996122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:04:06.996122+0000","objects_scrubbed":6071,"log_size":4971,"log_dups_size":0,"ondisk_log_size":4971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:28.340008+0000","scrub_duration":2407,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":514654208,"num_objects":7853,"num_object_clones":0,"num_object_copies":23559,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7853,"num_write_kb":502592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"75'79","reported_seq":202,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.351741+0000","last_change":"2024-09-21T14:04:21.211219+0000","last_active":"2024-09-21T14:04:32.351741+0000","last_peered":"2024-09-21T14:04:32.351741+0000","last_clean":"2024-09-21T14:04:32.351741+0000","last_became_active":"2024-09-21T14:04:21.210586+0000","last_became_peered":"2024-09-21T14:04:21.210586+0000","last_unstale":"2024-09-21T14:04:32.351741+0000","last_undegraded":"2024-09-21T14:04:32.351741+0000","last_fullsized":"2024-09-21T14:04:32.351741+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"290'24","reported_seq":600,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.350170+0000","last_change":"2024-09-21T14:04:21.221906+0000","last_active":"2024-09-21T14:04:32.350170+0000","last_peered":"2024-09-21T14:04:32.350170+0000","last_clean":"2024-09-21T14:04:32.350170+0000","last_became_active":"2024-09-21T14:04:21.206881+0000","last_became_peered":"2024-09-21T14:04:21.206881+0000","last_unstale":"2024-09-21T14:04:32.350170+0000","last_undegraded":"2024-09-21T14:04:32.350170+0000","last_fullsized":"2024-09-21T14:04:32.350170+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'24","last_scrub_stamp":"2024-09-21T14:02:57.559936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:57.559936+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":333,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T14:04:23.214436+0000","last_change":"2024-09-21T14:04:21.208903+0000","last_active":"2024-09-21T14:04:23.214436+0000","last_peered":"2024-09-21T14:04:23.214436+0000","last_clean":"2024-09-21T14:04:23.214436+0000","last_became_active":"2024-09-21T14:04:21.207578+0000","last_became_peered":"2024-09-21T14:04:21.207578+0000","last_unstale":"2024-09-21T14:04:23.214436+0000","last_undegraded":"2024-09-21T14:04:23.214436+0000","last_fullsized":"2024-09-21T14:04:23.214436+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:59.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:59:59.450546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":1542,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T14:04:29.690959+0000","last_change":"2024-09-21T13:59:32.218796+0000","last_active":"2024-09-21T14:04:29.690959+0000","last_peered":"2024-09-21T14:04:29.690959+0000","last_clean":"2024-09-21T14:04:29.690959+0000","last_became_active":"2024-09-21T13:59:32.203361+0000","last_became_peered":"2024-09-21T13:59:32.203361+0000","last_unstale":"2024-09-21T14:04:29.690959+0000","last_undegraded":"2024-09-21T14:04:29.690959+0000","last_fullsized":"2024-09-21T14:04:29.690959+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-21T13:56:29.617143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:56:29.617143+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047048000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":305,"num_read_kb":300,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":674,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.349760+0000","last_change":"2024-09-21T14:03:06.470855+0000","last_active":"2024-09-21T14:04:32.349760+0000","last_peered":"2024-09-21T14:04:32.349760+0000","last_clean":"2024-09-21T14:04:32.349760+0000","last_became_active":"2024-09-21T14:01:41.200531+0000","last_became_peered":"2024-09-21T14:01:41.200531+0000","last_unstale":"2024-09-21T14:04:32.349760+0000","last_undegraded":"2024-09-21T14:04:32.349760+0000","last_fullsized":"2024-09-21T14:04:32.349760+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:06.470705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:06.470705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:32.367248+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":778,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.352331+0000","last_change":"2024-09-21T14:03:04.357026+0000","last_active":"2024-09-21T14:04:32.352331+0000","last_peered":"2024-09-21T14:04:32.352331+0000","last_clean":"2024-09-21T14:04:32.352331+0000","last_became_active":"2024-09-21T14:01:41.184664+0000","last_became_peered":"2024-09-21T14:01:41.184664+0000","last_unstale":"2024-09-21T14:04:32.352331+0000","last_undegraded":"2024-09-21T14:04:32.352331+0000","last_fullsized":"2024-09-21T14:04:32.352331+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:04.356979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:04.356979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"412'7816","reported_seq":7958,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:33.590102+0000","last_change":"2024-09-21T14:04:08.011633+0000","last_active":"2024-09-21T14:04:33.590102+0000","last_peered":"2024-09-21T14:04:33.590102+0000","last_clean":"2024-09-21T14:04:33.590102+0000","last_became_active":"2024-09-21T14:02:56.557263+0000","last_became_peered":"2024-09-21T14:02:56.557263+0000","last_unstale":"2024-09-21T14:04:33.590102+0000","last_undegraded":"2024-09-21T14:04:33.590102+0000","last_fullsized":"2024-09-21T14:04:33.590102+0000","mapping_epoch":381,"log_start":"390'2719","ondisk_log_start":"390'2719","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"406'6147","last_scrub_stamp":"2024-09-21T14:04:08.011556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:04:08.011556+0000","objects_scrubbed":6074,"log_size":5097,"log_dups_size":0,"ondisk_log_size":5097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:19.661805+0000","scrub_duration":1868,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":512229376,"num_objects":7816,"num_object_clones":0,"num_object_copies":23448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7816,"num_write_kb":500224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"300'32","reported_seq":446,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:33.317430+0000","last_change":"2024-09-21T14:00:47.130089+0000","last_active":"2024-09-21T14:04:33.317430+0000","last_peered":"2024-09-21T14:04:33.317430+0000","last_clean":"2024-09-21T14:04:33.317430+0000","last_became_active":"2024-09-21T13:59:32.204624+0000","last_became_peered":"2024-09-21T13:59:32.204624+0000","last_unstale":"2024-09-21T14:04:33.317430+0000","last_undegraded":"2024-09-21T14:04:33.317430+0000","last_fullsized":"2024-09-21T14:04:33.317430+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'32","last_scrub_stamp":"2024-09-21T14:00:47.129789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:00:47.129789+0000","objects_scrubbed":8,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:20.900768+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":722,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T14:04:23.214339+0000","last_change":"2024-09-21T14:01:41.190537+0000","last_active":"2024-09-21T14:04:23.214339+0000","last_peered":"2024-09-21T14:04:23.214339+0000","last_clean":"2024-09-21T14:04:23.214339+0000","last_became_active":"2024-09-21T14:01:41.190210+0000","last_became_peered":"2024-09-21T14:01:41.190210+0000","last_unstale":"2024-09-21T14:04:23.214339+0000","last_undegraded":"2024-09-21T14:04:23.214339+0000","last_fullsized":"2024-09-21T14:04:23.214339+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:57.772975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:57.772975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"75'79","reported_seq":790,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T14:04:23.212158+0000","last_change":"2024-09-21T14:04:21.205378+0000","last_active":"2024-09-21T14:04:23.212158+0000","last_peered":"2024-09-21T14:04:23.212158+0000","last_clean":"2024-09-21T14:04:23.212158+0000","last_became_active":"2024-09-21T14:04:21.204537+0000","last_became_peered":"2024-09-21T14:04:21.204537+0000","last_unstale":"2024-09-21T14:04:23.212158+0000","last_undegraded":"2024-09-21T14:04:23.212158+0000","last_fullsized":"2024-09-21T14:04:23.212158+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:05:22.117825+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":19,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53669,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":697,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T14:04:23.211915+0000","last_change":"2024-09-21T14:01:41.182186+0000","last_active":"2024-09-21T14:04:23.211915+0000","last_peered":"2024-09-21T14:04:23.211915+0000","last_clean":"2024-09-21T14:04:23.211915+0000","last_became_active":"2024-09-21T14:01:41.181899+0000","last_became_peered":"2024-09-21T14:01:41.181899+0000","last_unstale":"2024-09-21T14:04:23.211915+0000","last_undegraded":"2024-09-21T14:04:23.211915+0000","last_fullsized":"2024-09-21T14:04:23.211915+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:15.700409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:15.700409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":639,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.349433+0000","last_change":"2024-09-21T14:04:01.760941+0000","last_active":"2024-09-21T14:04:32.349433+0000","last_peered":"2024-09-21T14:04:32.349433+0000","last_clean":"2024-09-21T14:04:32.349433+0000","last_became_active":"2024-09-21T14:03:57.346453+0000","last_became_peered":"2024-09-21T14:03:57.346453+0000","last_unstale":"2024-09-21T14:04:32.349433+0000","last_undegraded":"2024-09-21T14:04:32.349433+0000","last_fullsized":"2024-09-21T14:04:32.349433+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:01.760755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:04:01.760755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:08.954955+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"412'7833","reported_seq":7988,"reported_epoch":412,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-21T14:04:33.608118+0000","last_change":"2024-09-21T14:04:22.361693+0000","last_active":"2024-09-21T14:04:33.608118+0000","last_peered":"2024-09-21T14:04:33.608118+0000","last_clean":"2024-09-21T14:04:20.179742+0000","last_became_active":"2024-09-21T14:04:22.332419+0000","last_became_peered":"2024-09-21T14:04:22.332419+0000","last_unstale":"2024-09-21T14:04:33.608118+0000","last_undegraded":"2024-09-21T14:04:33.608118+0000","last_fullsized":"2024-09-21T14:04:33.608118+0000","mapping_epoch":409,"log_start":"390'2736","ondisk_log_start":"390'2736","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"406'6598","last_scrub_stamp":"2024-09-21T14:04:14.205496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:04:14.205496+0000","objects_scrubbed":6523,"log_size":5097,"log_dups_size":0,"ondisk_log_size":5097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":513343488,"num_objects":7833,"num_object_clones":0,"num_object_copies":23499,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7833,"num_objects_unfound":0,"num_objects_dirty":7833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7833,"num_write_kb":501312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,4],"avail_no_missing":["6","1","4"],"object_location_counts":[{"shards":"1,4,6","objects":7833}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"290'19","reported_seq":810,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.353444+0000","last_change":"2024-09-21T14:02:28.553549+0000","last_active":"2024-09-21T14:04:32.353444+0000","last_peered":"2024-09-21T14:04:32.353444+0000","last_clean":"2024-09-21T14:04:32.353444+0000","last_became_active":"2024-09-21T13:58:17.776310+0000","last_became_peered":"2024-09-21T13:58:17.776310+0000","last_unstale":"2024-09-21T14:04:32.353444+0000","last_undegraded":"2024-09-21T14:04:32.353444+0000","last_fullsized":"2024-09-21T14:04:32.353444+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'19","last_scrub_stamp":"2024-09-21T14:02:28.553065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:28.553065+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:56.054978+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":781,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.353205+0000","last_change":"2024-09-21T14:03:32.957705+0000","last_active":"2024-09-21T14:04:32.353205+0000","last_peered":"2024-09-21T14:04:32.353205+0000","last_clean":"2024-09-21T14:04:32.353205+0000","last_became_active":"2024-09-21T13:59:32.223247+0000","last_became_peered":"2024-09-21T13:59:32.223247+0000","last_unstale":"2024-09-21T14:04:32.353205+0000","last_undegraded":"2024-09-21T14:04:32.353205+0000","last_fullsized":"2024-09-21T14:04:32.353205+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:32.957649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:03:32.957649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:44.854711+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":700,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.349353+0000","last_change":"2024-09-21T14:02:58.461716+0000","last_active":"2024-09-21T14:04:32.349353+0000","last_peered":"2024-09-21T14:04:32.349353+0000","last_clean":"2024-09-21T14:04:32.349353+0000","last_became_active":"2024-09-21T14:01:41.193941+0000","last_became_peered":"2024-09-21T14:01:41.193941+0000","last_unstale":"2024-09-21T14:04:32.349353+0000","last_undegraded":"2024-09-21T14:04:32.349353+0000","last_fullsized":"2024-09-21T14:04:32.349353+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:58.461663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:02:58.461663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":708,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T14:04:23.215193+0000","last_change":"2024-09-21T14:03:36.302360+0000","last_active":"2024-09-21T14:04:23.215193+0000","last_peered":"2024-09-21T14:04:23.215193+0000","last_clean":"2024-09-21T14:04:23.215193+0000","last_became_active":"2024-09-21T14:01:41.190192+0000","last_became_peered":"2024-09-21T14:01:41.190192+0000","last_unstale":"2024-09-21T14:04:23.215193+0000","last_undegraded":"2024-09-21T14:04:23.215193+0000","last_fullsized":"2024-09-21T14:04:23.215193+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:36.302281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:36.302281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:37.898600+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"412'7727","reported_seq":7800,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.971185+0000","last_change":"2024-09-21T14:02:56.560359+0000","last_active":"2024-09-21T14:04:32.971185+0000","last_peered":"2024-09-21T14:04:32.971185+0000","last_clean":"2024-09-21T14:04:32.971185+0000","last_became_active":"2024-09-21T14:02:56.560070+0000","last_became_peered":"2024-09-21T14:02:56.560070+0000","last_unstale":"2024-09-21T14:04:32.971185+0000","last_undegraded":"2024-09-21T14:04:32.971185+0000","last_fullsized":"2024-09-21T14:04:32.971185+0000","mapping_epoch":381,"log_start":"390'2787","ondisk_log_start":"390'2787","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":4940,"log_dups_size":0,"ondisk_log_size":4940,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:29.621915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":506396672,"num_objects":7727,"num_object_clones":0,"num_object_copies":23181,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7727,"num_write_kb":494528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"75'79","reported_seq":165,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.348246+0000","last_change":"2024-09-21T14:01:41.189146+0000","last_active":"2024-09-21T14:04:32.348246+0000","last_peered":"2024-09-21T14:04:32.348246+0000","last_clean":"2024-09-21T14:04:32.348246+0000","last_became_active":"2024-09-21T14:01:41.188685+0000","last_became_peered":"2024-09-21T14:01:41.188685+0000","last_unstale":"2024-09-21T14:04:32.348246+0000","last_undegraded":"2024-09-21T14:04:32.348246+0000","last_fullsized":"2024-09-21T14:04:32.348246+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:04:50.615548+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"290'31","reported_seq":579,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.354447+0000","last_change":"2024-09-21T14:04:21.226319+0000","last_active":"2024-09-21T14:04:32.354447+0000","last_peered":"2024-09-21T14:04:32.354447+0000","last_clean":"2024-09-21T14:04:32.354447+0000","last_became_active":"2024-09-21T14:04:21.204164+0000","last_became_peered":"2024-09-21T14:04:21.204164+0000","last_unstale":"2024-09-21T14:04:32.354447+0000","last_undegraded":"2024-09-21T14:04:32.354447+0000","last_fullsized":"2024-09-21T14:04:32.354447+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'31","last_scrub_stamp":"2024-09-21T14:03:31.017119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:03:31.017119+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:36.091113+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":314,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.355081+0000","last_change":"2024-09-21T14:01:29.924408+0000","last_active":"2024-09-21T14:04:32.355081+0000","last_peered":"2024-09-21T14:04:32.355081+0000","last_clean":"2024-09-21T14:04:32.355081+0000","last_became_active":"2024-09-21T13:58:17.762520+0000","last_became_peered":"2024-09-21T13:58:17.762520+0000","last_unstale":"2024-09-21T14:04:32.355081+0000","last_undegraded":"2024-09-21T14:04:32.355081+0000","last_fullsized":"2024-09-21T14:04:32.355081+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:29.924331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:01:29.924331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:50.002080+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":727,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.352432+0000","last_change":"2024-09-21T13:58:52.030800+0000","last_active":"2024-09-21T14:04:32.352432+0000","last_peered":"2024-09-21T14:04:32.352432+0000","last_clean":"2024-09-21T14:04:32.352432+0000","last_became_active":"2024-09-21T13:49:02.067472+0000","last_became_peered":"2024-09-21T13:49:02.067472+0000","last_unstale":"2024-09-21T14:04:32.352432+0000","last_undegraded":"2024-09-21T14:04:32.352432+0000","last_fullsized":"2024-09-21T14:04:32.352432+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:52.030718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:52.030718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:53.320004+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":610,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.366231+0000","last_change":"2024-09-21T13:57:12.683321+0000","last_active":"2024-09-21T14:04:32.366231+0000","last_peered":"2024-09-21T14:04:32.366231+0000","last_clean":"2024-09-21T14:04:32.366231+0000","last_became_active":"2024-09-21T13:57:12.682979+0000","last_became_peered":"2024-09-21T13:57:12.682979+0000","last_unstale":"2024-09-21T14:04:32.366231+0000","last_undegraded":"2024-09-21T14:04:32.366231+0000","last_fullsized":"2024-09-21T14:04:32.366231+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:18.027944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:18.027944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:24.573702+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2","version":"412'7892","reported_seq":7965,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:34.736853+0000","last_change":"2024-09-21T14:02:56.559657+0000","last_active":"2024-09-21T14:04:34.736853+0000","last_peered":"2024-09-21T14:04:34.736853+0000","last_clean":"2024-09-21T14:04:34.736853+0000","last_became_active":"2024-09-21T14:02:56.559368+0000","last_became_peered":"2024-09-21T14:02:56.559368+0000","last_unstale":"2024-09-21T14:04:34.736853+0000","last_undegraded":"2024-09-21T14:04:34.736853+0000","last_fullsized":"2024-09-21T14:04:34.736853+0000","mapping_epoch":381,"log_start":"392'3142","ondisk_log_start":"392'3142","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":4750,"log_dups_size":0,"ondisk_log_size":4750,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":517210112,"num_objects":7892,"num_object_clones":0,"num_object_copies":23676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7892,"num_write_kb":505088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"75'79","reported_seq":796,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T14:04:23.211430+0000","last_change":"2024-09-21T14:02:55.806160+0000","last_active":"2024-09-21T14:04:23.211430+0000","last_peered":"2024-09-21T14:04:23.211430+0000","last_clean":"2024-09-21T14:04:23.211430+0000","last_became_active":"2024-09-21T14:00:41.139315+0000","last_became_peered":"2024-09-21T14:00:41.139315+0000","last_unstale":"2024-09-21T14:04:23.211430+0000","last_undegraded":"2024-09-21T14:04:23.211430+0000","last_fullsized":"2024-09-21T14:04:23.211430+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:02:55.806077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.806077+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"290'20","reported_seq":817,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.353094+0000","last_change":"2024-09-21T13:58:17.789407+0000","last_active":"2024-09-21T14:04:32.353094+0000","last_peered":"2024-09-21T14:04:32.353094+0000","last_clean":"2024-09-21T14:04:32.353094+0000","last_became_active":"2024-09-21T13:58:17.765427+0000","last_became_peered":"2024-09-21T13:58:17.765427+0000","last_unstale":"2024-09-21T14:04:32.353094+0000","last_undegraded":"2024-09-21T14:04:32.353094+0000","last_fullsized":"2024-09-21T14:04:32.353094+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:58:13.464196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:13.464196+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":672,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T14:04:23.215045+0000","last_change":"2024-09-21T14:04:21.215419+0000","last_active":"2024-09-21T14:04:23.215045+0000","last_peered":"2024-09-21T14:04:23.215045+0000","last_clean":"2024-09-21T14:04:23.215045+0000","last_became_active":"2024-09-21T14:04:21.215102+0000","last_became_peered":"2024-09-21T14:04:21.215102+0000","last_unstale":"2024-09-21T14:04:23.215045+0000","last_undegraded":"2024-09-21T14:04:23.215045+0000","last_fullsized":"2024-09-21T14:04:23.215045+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:58.236645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:02:58.236645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"346'14","reported_seq":810,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.354550+0000","last_change":"2024-09-21T14:01:41.186960+0000","last_active":"2024-09-21T14:04:32.354550+0000","last_peered":"2024-09-21T14:04:32.354550+0000","last_clean":"2024-09-21T14:04:32.354550+0000","last_became_active":"2024-09-21T14:01:41.186564+0000","last_became_peered":"2024-09-21T14:01:41.186564+0000","last_unstale":"2024-09-21T14:04:32.354550+0000","last_undegraded":"2024-09-21T14:04:32.354550+0000","last_fullsized":"2024-09-21T14:04:32.354550+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'12","last_scrub_stamp":"2024-09-21T13:59:38.738120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:38.738120+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:50.767839+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"68'1","reported_seq":745,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.353207+0000","last_change":"2024-09-21T13:59:31.392186+0000","last_active":"2024-09-21T14:04:32.353207+0000","last_peered":"2024-09-21T14:04:32.353207+0000","last_clean":"2024-09-21T14:04:32.353207+0000","last_became_active":"2024-09-21T13:52:04.313210+0000","last_became_peered":"2024-09-21T13:52:04.313210+0000","last_unstale":"2024-09-21T14:04:32.353207+0000","last_undegraded":"2024-09-21T14:04:32.353207+0000","last_fullsized":"2024-09-21T14:04:32.353207+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T13:59:31.392105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:31.392105+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:40.945663+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.5","version":"412'7901","reported_seq":7974,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.976578+0000","last_change":"2024-09-21T14:02:56.557127+0000","last_active":"2024-09-21T14:04:32.976578+0000","last_peered":"2024-09-21T14:04:32.976578+0000","last_clean":"2024-09-21T14:04:32.976578+0000","last_became_active":"2024-09-21T14:02:56.556786+0000","last_became_peered":"2024-09-21T14:02:56.556786+0000","last_unstale":"2024-09-21T14:04:32.976578+0000","last_undegraded":"2024-09-21T14:04:32.976578+0000","last_fullsized":"2024-09-21T14:04:32.976578+0000","mapping_epoch":381,"log_start":"391'2925","ondisk_log_start":"391'2925","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":4976,"log_dups_size":0,"ondisk_log_size":4976,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:26.615982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":517799936,"num_objects":7901,"num_object_clones":0,"num_object_copies":23703,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7901,"num_write_kb":505664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"290'15","reported_seq":772,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.346392+0000","last_change":"2024-09-21T13:58:50.498597+0000","last_active":"2024-09-21T14:04:32.346392+0000","last_peered":"2024-09-21T14:04:32.346392+0000","last_clean":"2024-09-21T14:04:32.346392+0000","last_became_active":"2024-09-21T13:49:00.053016+0000","last_became_peered":"2024-09-21T13:49:00.053016+0000","last_unstale":"2024-09-21T14:04:32.346392+0000","last_undegraded":"2024-09-21T14:04:32.346392+0000","last_fullsized":"2024-09-21T14:04:32.346392+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:58:50.498369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:50.498369+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:58.890531+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":712,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.348781+0000","last_change":"2024-09-21T14:04:00.767591+0000","last_active":"2024-09-21T14:04:32.348781+0000","last_peered":"2024-09-21T14:04:32.348781+0000","last_clean":"2024-09-21T14:04:32.348781+0000","last_became_active":"2024-09-21T14:01:41.192443+0000","last_became_peered":"2024-09-21T14:01:41.192443+0000","last_unstale":"2024-09-21T14:04:32.348781+0000","last_undegraded":"2024-09-21T14:04:32.348781+0000","last_fullsized":"2024-09-21T14:04:32.348781+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:00.767519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:00.767519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:08.722933+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":692,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.353358+0000","last_change":"2024-09-21T14:02:56.796917+0000","last_active":"2024-09-21T14:04:32.353358+0000","last_peered":"2024-09-21T14:04:32.353358+0000","last_clean":"2024-09-21T14:04:32.353358+0000","last_became_active":"2024-09-21T13:57:12.680978+0000","last_became_peered":"2024-09-21T13:57:12.680978+0000","last_unstale":"2024-09-21T14:04:32.353358+0000","last_undegraded":"2024-09-21T14:04:32.353358+0000","last_fullsized":"2024-09-21T14:04:32.353358+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:56.796861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:02:56.796861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":639,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T14:04:23.211100+0000","last_change":"2024-09-21T14:04:21.210096+0000","last_active":"2024-09-21T14:04:23.211100+0000","last_peered":"2024-09-21T14:04:23.211100+0000","last_clean":"2024-09-21T14:04:23.211100+0000","last_became_active":"2024-09-21T14:04:21.207534+0000","last_became_peered":"2024-09-21T14:04:21.207534+0000","last_unstale":"2024-09-21T14:04:23.211100+0000","last_undegraded":"2024-09-21T14:04:23.211100+0000","last_fullsized":"2024-09-21T14:04:23.211100+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:13.698091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:13.698091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"412'7757","reported_seq":10996,"reported_epoch":412,"state":"active+remapped+backfilling","last_fresh":"2024-09-21T14:04:33.741510+0000","last_change":"2024-09-21T14:04:22.282881+0000","last_active":"2024-09-21T14:04:33.741510+0000","last_peered":"2024-09-21T14:04:33.741510+0000","last_clean":"2024-09-21T14:04:20.165463+0000","last_became_active":"2024-09-21T14:04:22.266519+0000","last_became_peered":"2024-09-21T14:04:22.266519+0000","last_unstale":"2024-09-21T14:04:33.741510+0000","last_undegraded":"2024-09-21T14:04:33.741510+0000","last_fullsized":"2024-09-21T14:04:33.741510+0000","mapping_epoch":409,"log_start":"390'2500","ondisk_log_start":"390'2500","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":5257,"log_dups_size":0,"ondisk_log_size":5257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":508362752,"num_objects":7757,"num_object_clones":0,"num_object_copies":23271,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6079,"num_objects_unfound":0,"num_objects_dirty":7757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7757,"num_write_kb":496448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1580,"num_bytes_recovered":103546880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,0],"avail_no_missing":["4","0","3"],"object_location_counts":[{"shards":"0,3,4","objects":7757}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"290'35","reported_seq":380,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.355249+0000","last_change":"2024-09-21T14:01:02.006619+0000","last_active":"2024-09-21T14:04:32.355249+0000","last_peered":"2024-09-21T14:04:32.355249+0000","last_clean":"2024-09-21T14:04:32.355249+0000","last_became_active":"2024-09-21T13:59:32.204184+0000","last_became_peered":"2024-09-21T13:59:32.204184+0000","last_unstale":"2024-09-21T14:04:32.355249+0000","last_undegraded":"2024-09-21T14:04:32.355249+0000","last_fullsized":"2024-09-21T14:04:32.355249+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'35","last_scrub_stamp":"2024-09-21T14:01:02.006504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:02.006504+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:29.829621+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":732,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T14:04:23.211649+0000","last_change":"2024-09-21T14:03:31.882264+0000","last_active":"2024-09-21T14:04:23.211649+0000","last_peered":"2024-09-21T14:04:23.211649+0000","last_clean":"2024-09-21T14:04:23.211649+0000","last_became_active":"2024-09-21T14:01:41.184567+0000","last_became_peered":"2024-09-21T14:01:41.184567+0000","last_unstale":"2024-09-21T14:04:23.211649+0000","last_undegraded":"2024-09-21T14:04:23.211649+0000","last_fullsized":"2024-09-21T14:04:23.211649+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:31.882215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:03:31.882215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:38.486020+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":704,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.357290+0000","last_change":"2024-09-21T14:01:41.182250+0000","last_active":"2024-09-21T14:04:32.357290+0000","last_peered":"2024-09-21T14:04:32.357290+0000","last_clean":"2024-09-21T14:04:32.357290+0000","last_became_active":"2024-09-21T14:01:41.181911+0000","last_became_peered":"2024-09-21T14:01:41.181911+0000","last_unstale":"2024-09-21T14:04:32.357290+0000","last_undegraded":"2024-09-21T14:04:32.357290+0000","last_fullsized":"2024-09-21T14:04:32.357290+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:55.588554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:55.588554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:45.881738+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":659,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T14:04:23.211393+0000","last_change":"2024-09-21T14:03:33.893158+0000","last_active":"2024-09-21T14:04:23.211393+0000","last_peered":"2024-09-21T14:04:23.211393+0000","last_clean":"2024-09-21T14:04:23.211393+0000","last_became_active":"2024-09-21T13:59:32.219332+0000","last_became_peered":"2024-09-21T13:59:32.219332+0000","last_unstale":"2024-09-21T14:04:23.211393+0000","last_undegraded":"2024-09-21T14:04:23.211393+0000","last_fullsized":"2024-09-21T14:04:23.211393+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:33.893072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:33.893072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:40.141218+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"412'7926","reported_seq":7999,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:33.584835+0000","last_change":"2024-09-21T14:02:56.559626+0000","last_active":"2024-09-21T14:04:33.584835+0000","last_peered":"2024-09-21T14:04:33.584835+0000","last_clean":"2024-09-21T14:04:33.584835+0000","last_became_active":"2024-09-21T14:02:56.559331+0000","last_became_peered":"2024-09-21T14:02:56.559331+0000","last_unstale":"2024-09-21T14:04:33.584835+0000","last_undegraded":"2024-09-21T14:04:33.584835+0000","last_fullsized":"2024-09-21T14:04:33.584835+0000","mapping_epoch":381,"log_start":"390'2790","ondisk_log_start":"390'2790","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":5136,"log_dups_size":0,"ondisk_log_size":5136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":519438336,"num_objects":7926,"num_object_clones":0,"num_object_copies":23778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7926,"num_write_kb":507264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"290'18","reported_seq":790,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.353110+0000","last_change":"2024-09-21T13:59:28.340494+0000","last_active":"2024-09-21T14:04:32.353110+0000","last_peered":"2024-09-21T14:04:32.353110+0000","last_clean":"2024-09-21T14:04:32.353110+0000","last_became_active":"2024-09-21T13:57:12.680627+0000","last_became_peered":"2024-09-21T13:57:12.680627+0000","last_unstale":"2024-09-21T14:04:32.353110+0000","last_undegraded":"2024-09-21T14:04:32.353110+0000","last_fullsized":"2024-09-21T14:04:32.353110+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'18","last_scrub_stamp":"2024-09-21T13:59:28.340415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:28.340415+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":686,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T14:04:23.212693+0000","last_change":"2024-09-21T14:04:21.206034+0000","last_active":"2024-09-21T14:04:23.212693+0000","last_peered":"2024-09-21T14:04:23.212693+0000","last_clean":"2024-09-21T14:04:23.212693+0000","last_became_active":"2024-09-21T14:04:21.205699+0000","last_became_peered":"2024-09-21T14:04:21.205699+0000","last_unstale":"2024-09-21T14:04:23.212693+0000","last_undegraded":"2024-09-21T14:04:23.212693+0000","last_fullsized":"2024-09-21T14:04:23.212693+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:21.582325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:21.582325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":666,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.354170+0000","last_change":"2024-09-21T14:02:27.791041+0000","last_active":"2024-09-21T14:04:32.354170+0000","last_peered":"2024-09-21T14:04:32.354170+0000","last_clean":"2024-09-21T14:04:32.354170+0000","last_became_active":"2024-09-21T13:52:04.322147+0000","last_became_peered":"2024-09-21T13:52:04.322147+0000","last_unstale":"2024-09-21T14:04:32.354170+0000","last_undegraded":"2024-09-21T14:04:32.354170+0000","last_fullsized":"2024-09-21T14:04:32.354170+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:27.790994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:02:27.790994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:53.761101+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":673,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T14:04:23.215409+0000","last_change":"2024-09-21T14:02:36.371955+0000","last_active":"2024-09-21T14:04:23.215409+0000","last_peered":"2024-09-21T14:04:23.215409+0000","last_clean":"2024-09-21T14:04:23.215409+0000","last_became_active":"2024-09-21T13:59:32.203767+0000","last_became_peered":"2024-09-21T13:59:32.203767+0000","last_unstale":"2024-09-21T14:04:23.215409+0000","last_undegraded":"2024-09-21T14:04:23.215409+0000","last_fullsized":"2024-09-21T14:04:23.215409+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:36.371747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:02:36.371747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"412'8022","reported_seq":8095,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:35.780605+0000","last_change":"2024-09-21T14:02:56.564166+0000","last_active":"2024-09-21T14:04:35.780605+0000","last_peered":"2024-09-21T14:04:35.780605+0000","last_clean":"2024-09-21T14:04:35.780605+0000","last_became_active":"2024-09-21T14:02:56.563788+0000","last_became_peered":"2024-09-21T14:02:56.563788+0000","last_unstale":"2024-09-21T14:04:35.780605+0000","last_undegraded":"2024-09-21T14:04:35.780605+0000","last_fullsized":"2024-09-21T14:04:35.780605+0000","mapping_epoch":381,"log_start":"393'3137","ondisk_log_start":"393'3137","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":4885,"log_dups_size":0,"ondisk_log_size":4885,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":525598720,"num_objects":8020,"num_object_clones":0,"num_object_copies":24060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8022,"num_write_kb":513344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"290'27","reported_seq":788,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.348389+0000","last_change":"2024-09-21T14:03:31.360992+0000","last_active":"2024-09-21T14:04:32.348389+0000","last_peered":"2024-09-21T14:04:32.348389+0000","last_clean":"2024-09-21T14:04:32.348389+0000","last_became_active":"2024-09-21T14:01:41.192481+0000","last_became_peered":"2024-09-21T14:01:41.192481+0000","last_unstale":"2024-09-21T14:04:32.348389+0000","last_undegraded":"2024-09-21T14:04:32.348389+0000","last_fullsized":"2024-09-21T14:04:32.348389+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'27","last_scrub_stamp":"2024-09-21T14:03:31.360906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:03:31.360906+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:41.422161+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":672,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T14:04:23.212981+0000","last_change":"2024-09-21T14:04:21.218814+0000","last_active":"2024-09-21T14:04:23.212981+0000","last_peered":"2024-09-21T14:04:23.212981+0000","last_clean":"2024-09-21T14:04:23.212981+0000","last_became_active":"2024-09-21T14:04:21.201644+0000","last_became_peered":"2024-09-21T14:04:21.201644+0000","last_unstale":"2024-09-21T14:04:23.212981+0000","last_undegraded":"2024-09-21T14:04:23.212981+0000","last_fullsized":"2024-09-21T14:04:23.212981+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T14:03:04.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:03:04.953658+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:27.366051+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"300'14","reported_seq":885,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T14:04:23.314065+0000","last_change":"2024-09-21T13:59:32.220898+0000","last_active":"2024-09-21T14:04:23.314065+0000","last_peered":"2024-09-21T14:04:23.314065+0000","last_clean":"2024-09-21T14:04:23.314065+0000","last_became_active":"2024-09-21T13:59:32.206956+0000","last_became_peered":"2024-09-21T13:59:32.206956+0000","last_unstale":"2024-09-21T14:04:23.314065+0000","last_undegraded":"2024-09-21T14:04:23.314065+0000","last_fullsized":"2024-09-21T14:04:23.314065+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'11","last_scrub_stamp":"2024-09-21T13:56:31.616003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:56:31.616003+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:27.198182+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":728,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.348119+0000","last_change":"2024-09-21T14:01:41.248408+0000","last_active":"2024-09-21T14:04:32.348119+0000","last_peered":"2024-09-21T14:04:32.348119+0000","last_clean":"2024-09-21T14:04:32.348119+0000","last_became_active":"2024-09-21T14:01:41.246106+0000","last_became_peered":"2024-09-21T14:01:41.246106+0000","last_unstale":"2024-09-21T14:04:32.348119+0000","last_undegraded":"2024-09-21T14:04:32.348119+0000","last_fullsized":"2024-09-21T14:04:32.348119+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:39.258659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:39.258659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":324,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.354962+0000","last_change":"2024-09-21T14:04:21.198859+0000","last_active":"2024-09-21T14:04:32.354962+0000","last_peered":"2024-09-21T14:04:32.354962+0000","last_clean":"2024-09-21T14:04:32.354962+0000","last_became_active":"2024-09-21T14:04:21.197402+0000","last_became_peered":"2024-09-21T14:04:21.197402+0000","last_unstale":"2024-09-21T14:04:32.354962+0000","last_undegraded":"2024-09-21T14:04:32.354962+0000","last_fullsized":"2024-09-21T14:04:32.354962+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:50.733007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:01:50.733007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:31.764576+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"290'12","reported_seq":700,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T14:04:23.216103+0000","last_change":"2024-09-21T13:59:32.247422+0000","last_active":"2024-09-21T14:04:23.216103+0000","last_peered":"2024-09-21T14:04:23.216103+0000","last_clean":"2024-09-21T14:04:23.216103+0000","last_became_active":"2024-09-21T13:59:32.211306+0000","last_became_peered":"2024-09-21T13:59:32.211306+0000","last_unstale":"2024-09-21T14:04:23.216103+0000","last_undegraded":"2024-09-21T14:04:23.216103+0000","last_fullsized":"2024-09-21T14:04:23.216103+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'6","last_scrub_stamp":"2024-09-21T13:55:12.066873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:55:12.066873+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:56.265399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":660,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.349863+0000","last_change":"2024-09-21T14:01:41.199742+0000","last_active":"2024-09-21T14:04:32.349863+0000","last_peered":"2024-09-21T14:04:32.349863+0000","last_clean":"2024-09-21T14:04:32.349863+0000","last_became_active":"2024-09-21T14:01:41.199321+0000","last_became_peered":"2024-09-21T14:01:41.199321+0000","last_unstale":"2024-09-21T14:04:32.349863+0000","last_undegraded":"2024-09-21T14:04:32.349863+0000","last_fullsized":"2024-09-21T14:04:32.349863+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:37.106238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:37.106238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:17.707656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":705,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T14:04:23.211511+0000","last_change":"2024-09-21T14:03:01.872986+0000","last_active":"2024-09-21T14:04:23.211511+0000","last_peered":"2024-09-21T14:04:23.211511+0000","last_clean":"2024-09-21T14:04:23.211511+0000","last_became_active":"2024-09-21T13:59:32.219525+0000","last_became_peered":"2024-09-21T13:59:32.219525+0000","last_unstale":"2024-09-21T14:04:23.211511+0000","last_undegraded":"2024-09-21T14:04:23.211511+0000","last_fullsized":"2024-09-21T14:04:23.211511+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:01.872928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:01.872928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"290'15","reported_seq":680,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T14:04:23.215525+0000","last_change":"2024-09-21T14:04:21.228883+0000","last_active":"2024-09-21T14:04:23.215525+0000","last_peered":"2024-09-21T14:04:23.215525+0000","last_clean":"2024-09-21T14:04:23.215525+0000","last_became_active":"2024-09-21T14:04:21.208398+0000","last_became_peered":"2024-09-21T14:04:21.208398+0000","last_unstale":"2024-09-21T14:04:23.215525+0000","last_undegraded":"2024-09-21T14:04:23.215525+0000","last_fullsized":"2024-09-21T14:04:23.215525+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:55:04.990842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:55:04.990842+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"61'1","reported_seq":723,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T14:04:23.211338+0000","last_change":"2024-09-21T14:04:04.012140+0000","last_active":"2024-09-21T14:04:23.211338+0000","last_peered":"2024-09-21T14:04:23.211338+0000","last_clean":"2024-09-21T14:04:23.211338+0000","last_became_active":"2024-09-21T13:59:32.219515+0000","last_became_peered":"2024-09-21T13:59:32.219515+0000","last_unstale":"2024-09-21T14:04:23.211338+0000","last_undegraded":"2024-09-21T14:04:23.211338+0000","last_fullsized":"2024-09-21T14:04:23.211338+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T14:04:04.012006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:04.012006+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:20.215213+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":656,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T14:04:23.215783+0000","last_change":"2024-09-21T14:03:33.309440+0000","last_active":"2024-09-21T14:04:23.215783+0000","last_peered":"2024-09-21T14:04:23.215783+0000","last_clean":"2024-09-21T14:04:23.215783+0000","last_became_active":"2024-09-21T13:59:32.210586+0000","last_became_peered":"2024-09-21T13:59:32.210586+0000","last_unstale":"2024-09-21T14:04:23.215783+0000","last_undegraded":"2024-09-21T14:04:23.215783+0000","last_fullsized":"2024-09-21T14:04:23.215783+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:33.309248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:33.309248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:57.005556+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":708,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.354050+0000","last_change":"2024-09-21T14:02:29.731635+0000","last_active":"2024-09-21T14:04:32.354050+0000","last_peered":"2024-09-21T14:04:32.354050+0000","last_clean":"2024-09-21T14:04:32.354050+0000","last_became_active":"2024-09-21T13:59:32.205803+0000","last_became_peered":"2024-09-21T13:59:32.205803+0000","last_unstale":"2024-09-21T14:04:32.354050+0000","last_undegraded":"2024-09-21T14:04:32.354050+0000","last_fullsized":"2024-09-21T14:04:32.354050+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:29.731565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:02:29.731565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:56.723726+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"290'15","reported_seq":705,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.349537+0000","last_change":"2024-09-21T14:01:41.189676+0000","last_active":"2024-09-21T14:04:32.349537+0000","last_peered":"2024-09-21T14:04:32.349537+0000","last_clean":"2024-09-21T14:04:32.349537+0000","last_became_active":"2024-09-21T14:01:41.187904+0000","last_became_peered":"2024-09-21T14:01:41.187904+0000","last_unstale":"2024-09-21T14:04:32.349537+0000","last_undegraded":"2024-09-21T14:04:32.349537+0000","last_fullsized":"2024-09-21T14:04:32.349537+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:57:24.124604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:57:24.124604+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:30.424236+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":584,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T14:04:23.211187+0000","last_change":"2024-09-21T13:59:32.212325+0000","last_active":"2024-09-21T14:04:23.211187+0000","last_peered":"2024-09-21T14:04:23.211187+0000","last_clean":"2024-09-21T14:04:23.211187+0000","last_became_active":"2024-09-21T13:59:32.208749+0000","last_became_peered":"2024-09-21T13:59:32.208749+0000","last_unstale":"2024-09-21T14:04:23.211187+0000","last_undegraded":"2024-09-21T14:04:23.211187+0000","last_fullsized":"2024-09-21T14:04:23.211187+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:20.574745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:20.574745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"300'14","reported_seq":818,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T14:04:23.313348+0000","last_change":"2024-09-21T13:59:32.208253+0000","last_active":"2024-09-21T14:04:23.313348+0000","last_peered":"2024-09-21T14:04:23.313348+0000","last_clean":"2024-09-21T14:04:23.313348+0000","last_became_active":"2024-09-21T13:59:32.206926+0000","last_became_peered":"2024-09-21T13:59:32.206926+0000","last_unstale":"2024-09-21T14:04:23.313348+0000","last_undegraded":"2024-09-21T14:04:23.313348+0000","last_fullsized":"2024-09-21T14:04:23.313348+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'10","last_scrub_stamp":"2024-09-21T13:54:20.946782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:20.946782+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:20.221431+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":752,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.353941+0000","last_change":"2024-09-21T14:01:41.200564+0000","last_active":"2024-09-21T14:04:32.353941+0000","last_peered":"2024-09-21T14:04:32.353941+0000","last_clean":"2024-09-21T14:04:32.353941+0000","last_became_active":"2024-09-21T14:01:41.200113+0000","last_became_peered":"2024-09-21T14:01:41.200113+0000","last_unstale":"2024-09-21T14:04:32.353941+0000","last_undegraded":"2024-09-21T14:04:32.353941+0000","last_fullsized":"2024-09-21T14:04:32.353941+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:54.805619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:54.805619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:50.990023+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"290'20","reported_seq":707,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T14:04:23.215075+0000","last_change":"2024-09-21T14:04:04.192280+0000","last_active":"2024-09-21T14:04:23.215075+0000","last_peered":"2024-09-21T14:04:23.215075+0000","last_clean":"2024-09-21T14:04:23.215075+0000","last_became_active":"2024-09-21T13:59:32.208332+0000","last_became_peered":"2024-09-21T13:59:32.208332+0000","last_unstale":"2024-09-21T14:04:23.215075+0000","last_undegraded":"2024-09-21T14:04:23.215075+0000","last_fullsized":"2024-09-21T14:04:23.215075+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'20","last_scrub_stamp":"2024-09-21T14:04:04.192193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:04:04.192193+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:06.970216+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":691,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T14:04:23.211821+0000","last_change":"2024-09-21T13:59:32.212737+0000","last_active":"2024-09-21T14:04:23.211821+0000","last_peered":"2024-09-21T14:04:23.211821+0000","last_clean":"2024-09-21T14:04:23.211821+0000","last_became_active":"2024-09-21T13:59:32.209804+0000","last_became_peered":"2024-09-21T13:59:32.209804+0000","last_unstale":"2024-09-21T14:04:23.211821+0000","last_undegraded":"2024-09-21T14:04:23.211821+0000","last_fullsized":"2024-09-21T14:04:23.211821+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:25.571256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:25.571256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"346'10","reported_seq":829,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.349485+0000","last_change":"2024-09-21T14:03:34.365517+0000","last_active":"2024-09-21T14:04:32.349485+0000","last_peered":"2024-09-21T14:04:32.349485+0000","last_clean":"2024-09-21T14:04:32.349485+0000","last_became_active":"2024-09-21T14:01:41.206303+0000","last_became_peered":"2024-09-21T14:01:41.206303+0000","last_unstale":"2024-09-21T14:04:32.349485+0000","last_undegraded":"2024-09-21T14:04:32.349485+0000","last_fullsized":"2024-09-21T14:04:32.349485+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'10","last_scrub_stamp":"2024-09-21T14:03:34.365428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:34.365428+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:00.856228+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":603,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T14:04:23.212537+0000","last_change":"2024-09-21T14:04:21.202582+0000","last_active":"2024-09-21T14:04:23.212537+0000","last_peered":"2024-09-21T14:04:23.212537+0000","last_clean":"2024-09-21T14:04:23.212537+0000","last_became_active":"2024-09-21T14:04:21.201277+0000","last_became_peered":"2024-09-21T14:04:21.201277+0000","last_unstale":"2024-09-21T14:04:23.212537+0000","last_undegraded":"2024-09-21T14:04:23.212537+0000","last_fullsized":"2024-09-21T14:04:23.212537+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:27.961544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:27.961544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:48.519808+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"290'19","reported_seq":823,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.353068+0000","last_change":"2024-09-21T14:04:05.139089+0000","last_active":"2024-09-21T14:04:32.353068+0000","last_peered":"2024-09-21T14:04:32.353068+0000","last_clean":"2024-09-21T14:04:32.353068+0000","last_became_active":"2024-09-21T13:58:17.771497+0000","last_became_peered":"2024-09-21T13:58:17.771497+0000","last_unstale":"2024-09-21T14:04:32.353068+0000","last_undegraded":"2024-09-21T14:04:32.353068+0000","last_fullsized":"2024-09-21T14:04:32.353068+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'19","last_scrub_stamp":"2024-09-21T14:04:05.138989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:04:05.138989+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:14.903630+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":333,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T14:04:23.212001+0000","last_change":"2024-09-21T14:02:57.807136+0000","last_active":"2024-09-21T14:04:23.212001+0000","last_peered":"2024-09-21T14:04:23.212001+0000","last_clean":"2024-09-21T14:04:23.212001+0000","last_became_active":"2024-09-21T13:59:32.201568+0000","last_became_peered":"2024-09-21T13:59:32.201568+0000","last_unstale":"2024-09-21T14:04:23.212001+0000","last_undegraded":"2024-09-21T14:04:23.212001+0000","last_fullsized":"2024-09-21T14:04:23.212001+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:57.807059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:02:57.807059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":765,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.348462+0000","last_change":"2024-09-21T14:01:41.185588+0000","last_active":"2024-09-21T14:04:32.348462+0000","last_peered":"2024-09-21T14:04:32.348462+0000","last_clean":"2024-09-21T14:04:32.348462+0000","last_became_active":"2024-09-21T14:01:41.185293+0000","last_became_peered":"2024-09-21T14:01:41.185293+0000","last_unstale":"2024-09-21T14:04:32.348462+0000","last_undegraded":"2024-09-21T14:04:32.348462+0000","last_fullsized":"2024-09-21T14:04:32.348462+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:35.409237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:35.409237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":671,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T14:04:23.214341+0000","last_change":"2024-09-21T14:04:21.209188+0000","last_active":"2024-09-21T14:04:23.214341+0000","last_peered":"2024-09-21T14:04:23.214341+0000","last_clean":"2024-09-21T14:04:23.214341+0000","last_became_active":"2024-09-21T14:04:21.208329+0000","last_became_peered":"2024-09-21T14:04:21.208329+0000","last_unstale":"2024-09-21T14:04:23.214341+0000","last_undegraded":"2024-09-21T14:04:23.214341+0000","last_fullsized":"2024-09-21T14:04:23.214341+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:57.225602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:02:57.225602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"290'15","reported_seq":825,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.348685+0000","last_change":"2024-09-21T14:04:12.139406+0000","last_active":"2024-09-21T14:04:32.348685+0000","last_peered":"2024-09-21T14:04:32.348685+0000","last_clean":"2024-09-21T14:04:32.348685+0000","last_became_active":"2024-09-21T13:58:17.764120+0000","last_became_peered":"2024-09-21T13:58:17.764120+0000","last_unstale":"2024-09-21T14:04:32.348685+0000","last_undegraded":"2024-09-21T14:04:32.348685+0000","last_fullsized":"2024-09-21T14:04:32.348685+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'15","last_scrub_stamp":"2024-09-21T14:04:12.139190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:04:12.139190+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:40.181606+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":703,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T14:04:23.211392+0000","last_change":"2024-09-21T14:04:07.038058+0000","last_active":"2024-09-21T14:04:23.211392+0000","last_peered":"2024-09-21T14:04:23.211392+0000","last_clean":"2024-09-21T14:04:23.211392+0000","last_became_active":"2024-09-21T13:59:32.221854+0000","last_became_peered":"2024-09-21T13:59:32.221854+0000","last_unstale":"2024-09-21T14:04:23.211392+0000","last_undegraded":"2024-09-21T14:04:23.211392+0000","last_fullsized":"2024-09-21T14:04:23.211392+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:07.037989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:07.037989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:34.939994+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":639,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T14:04:23.214962+0000","last_change":"2024-09-21T14:03:32.262147+0000","last_active":"2024-09-21T14:04:23.214962+0000","last_peered":"2024-09-21T14:04:23.214962+0000","last_clean":"2024-09-21T14:04:23.214962+0000","last_became_active":"2024-09-21T13:57:12.703441+0000","last_became_peered":"2024-09-21T13:57:12.703441+0000","last_unstale":"2024-09-21T14:04:23.214962+0000","last_undegraded":"2024-09-21T14:04:23.214962+0000","last_fullsized":"2024-09-21T14:04:23.214962+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:32.262067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:32.262067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:00.749089+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":563,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.348137+0000","last_change":"2024-09-21T14:03:29.345499+0000","last_active":"2024-09-21T14:04:32.348137+0000","last_peered":"2024-09-21T14:04:32.348137+0000","last_clean":"2024-09-21T14:04:32.348137+0000","last_became_active":"2024-09-21T14:01:41.186934+0000","last_became_peered":"2024-09-21T14:01:41.186934+0000","last_unstale":"2024-09-21T14:04:32.348137+0000","last_undegraded":"2024-09-21T14:04:32.348137+0000","last_fullsized":"2024-09-21T14:04:32.348137+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:29.345427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:29.345427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:45.842768+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"290'19","reported_seq":775,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T14:04:23.214739+0000","last_change":"2024-09-21T13:59:32.277742+0000","last_active":"2024-09-21T14:04:23.214739+0000","last_peered":"2024-09-21T14:04:23.214739+0000","last_clean":"2024-09-21T14:04:23.214739+0000","last_became_active":"2024-09-21T13:59:32.231567+0000","last_became_peered":"2024-09-21T13:59:32.231567+0000","last_unstale":"2024-09-21T14:04:23.214739+0000","last_undegraded":"2024-09-21T14:04:23.214739+0000","last_fullsized":"2024-09-21T14:04:23.214739+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:54:22.978267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:22.978267+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":665,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.349932+0000","last_change":"2024-09-21T14:03:01.448170+0000","last_active":"2024-09-21T14:04:32.349932+0000","last_peered":"2024-09-21T14:04:32.349932+0000","last_clean":"2024-09-21T14:04:32.349932+0000","last_became_active":"2024-09-21T14:01:41.190760+0000","last_became_peered":"2024-09-21T14:01:41.190760+0000","last_unstale":"2024-09-21T14:04:32.349932+0000","last_undegraded":"2024-09-21T14:04:32.349932+0000","last_fullsized":"2024-09-21T14:04:32.349932+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:01.448109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:03:01.448109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":701,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T14:04:23.212201+0000","last_change":"2024-09-21T13:59:32.219508+0000","last_active":"2024-09-21T14:04:23.212201+0000","last_peered":"2024-09-21T14:04:23.212201+0000","last_clean":"2024-09-21T14:04:23.212201+0000","last_became_active":"2024-09-21T13:59:32.214004+0000","last_became_peered":"2024-09-21T13:59:32.214004+0000","last_unstale":"2024-09-21T14:04:23.212201+0000","last_undegraded":"2024-09-21T14:04:23.212201+0000","last_fullsized":"2024-09-21T14:04:23.212201+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:36.525417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:36.525417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:39.980347+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":704,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T14:04:23.215130+0000","last_change":"2024-09-21T14:04:21.214685+0000","last_active":"2024-09-21T14:04:23.215130+0000","last_peered":"2024-09-21T14:04:23.215130+0000","last_clean":"2024-09-21T14:04:23.215130+0000","last_became_active":"2024-09-21T14:04:21.213307+0000","last_became_peered":"2024-09-21T14:04:21.213307+0000","last_unstale":"2024-09-21T14:04:23.215130+0000","last_undegraded":"2024-09-21T14:04:23.215130+0000","last_fullsized":"2024-09-21T14:04:23.215130+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:28.124296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:28.124296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"290'8","reported_seq":721,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T14:04:23.214224+0000","last_change":"2024-09-21T13:59:26.034457+0000","last_active":"2024-09-21T14:04:23.214224+0000","last_peered":"2024-09-21T14:04:23.214224+0000","last_clean":"2024-09-21T14:04:23.214224+0000","last_became_active":"2024-09-21T13:58:17.775079+0000","last_became_peered":"2024-09-21T13:58:17.775079+0000","last_unstale":"2024-09-21T14:04:23.214224+0000","last_undegraded":"2024-09-21T14:04:23.214224+0000","last_fullsized":"2024-09-21T14:04:23.214224+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'8","last_scrub_stamp":"2024-09-21T13:59:26.034372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:26.034372+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":633,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:34.387435+0000","last_change":"2024-09-21T14:04:34.387435+0000","last_active":"2024-09-21T14:04:34.387435+0000","last_peered":"2024-09-21T14:04:34.387435+0000","last_clean":"2024-09-21T14:04:34.387435+0000","last_became_active":"2024-09-21T14:01:41.195989+0000","last_became_peered":"2024-09-21T14:01:41.195989+0000","last_unstale":"2024-09-21T14:04:34.387435+0000","last_undegraded":"2024-09-21T14:04:34.387435+0000","last_fullsized":"2024-09-21T14:04:34.387435+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:34.387367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:34.387367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:00.076717+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":332,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.354610+0000","last_change":"2024-09-21T14:04:21.205254+0000","last_active":"2024-09-21T14:04:32.354610+0000","last_peered":"2024-09-21T14:04:32.354610+0000","last_clean":"2024-09-21T14:04:32.354610+0000","last_became_active":"2024-09-21T14:04:21.204954+0000","last_became_peered":"2024-09-21T14:04:21.204954+0000","last_unstale":"2024-09-21T14:04:32.354610+0000","last_undegraded":"2024-09-21T14:04:32.354610+0000","last_fullsized":"2024-09-21T14:04:32.354610+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:30.727972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:30.727972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:40.626223+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":642,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T14:04:23.211109+0000","last_change":"2024-09-21T14:04:15.135536+0000","last_active":"2024-09-21T14:04:23.211109+0000","last_peered":"2024-09-21T14:04:23.211109+0000","last_clean":"2024-09-21T14:04:23.211109+0000","last_became_active":"2024-09-21T13:59:32.210134+0000","last_became_peered":"2024-09-21T13:59:32.210134+0000","last_unstale":"2024-09-21T14:04:23.211109+0000","last_undegraded":"2024-09-21T14:04:23.211109+0000","last_fullsized":"2024-09-21T14:04:23.211109+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:15.135463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:04:15.135463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:37.791722+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":750,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.348010+0000","last_change":"2024-09-21T14:01:41.187792+0000","last_active":"2024-09-21T14:04:32.348010+0000","last_peered":"2024-09-21T14:04:32.348010+0000","last_clean":"2024-09-21T14:04:32.348010+0000","last_became_active":"2024-09-21T14:01:41.187438+0000","last_became_peered":"2024-09-21T14:01:41.187438+0000","last_unstale":"2024-09-21T14:04:32.348010+0000","last_undegraded":"2024-09-21T14:04:32.348010+0000","last_fullsized":"2024-09-21T14:04:32.348010+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:17.706325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:17.706325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:45.351033+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":684,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T14:04:23.214526+0000","last_change":"2024-09-21T14:04:21.208524+0000","last_active":"2024-09-21T14:04:23.214526+0000","last_peered":"2024-09-21T14:04:23.214526+0000","last_clean":"2024-09-21T14:04:23.214526+0000","last_became_active":"2024-09-21T14:04:21.208121+0000","last_became_peered":"2024-09-21T14:04:21.208121+0000","last_unstale":"2024-09-21T14:04:23.214526+0000","last_undegraded":"2024-09-21T14:04:23.214526+0000","last_fullsized":"2024-09-21T14:04:23.214526+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:13.776572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:13.776572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"290'19","reported_seq":358,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-09-21T14:04:32.356230+0000","last_change":"2024-09-21T14:03:31.762681+0000","last_active":"2024-09-21T14:04:32.356230+0000","last_peered":"2024-09-21T14:04:32.356230+0000","last_clean":"2024-09-21T14:04:32.356230+0000","last_became_active":"2024-09-21T14:01:41.194606+0000","last_became_peered":"2024-09-21T14:01:41.194606+0000","last_unstale":"2024-09-21T14:04:32.356230+0000","last_undegraded":"2024-09-21T14:04:32.356230+0000","last_fullsized":"2024-09-21T14:04:32.356230+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'19","last_scrub_stamp":"2024-09-21T14:03:31.762605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:03:31.762605+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:58.699819+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":603,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-09-21T14:04:23.212499+0000","last_change":"2024-09-21T13:59:32.203045+0000","last_active":"2024-09-21T14:04:23.212499+0000","last_peered":"2024-09-21T14:04:23.212499+0000","last_clean":"2024-09-21T14:04:23.212499+0000","last_became_active":"2024-09-21T13:59:32.202225+0000","last_became_peered":"2024-09-21T13:59:32.202225+0000","last_unstale":"2024-09-21T14:04:23.212499+0000","last_undegraded":"2024-09-21T14:04:23.212499+0000","last_fullsized":"2024-09-21T14:04:23.212499+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:40.650832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:40.650832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:45.037251+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":8206155776,"num_objects":125216,"num_object_clones":0,"num_object_copies":375648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":29496,"num_objects_unfound":0,"num_objects_dirty":125216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":125218,"num_write_kb":8013888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1580,"num_bytes_recovered":103546880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24711856128,"data_stored":24711856128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80396,"ondisk_log_size":80396,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101,"ondisk_log_size":101,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1107,"num_read_kb":864,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":202,"num_bytes_recovered":2794,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":687,"ondisk_log_size":687,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":305,"num_read_kb":300,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":322019,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":62,"num_read_kb":119,"num_write":63,"num_write_kb":888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":977379,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":298,"seq":1279900254269,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2325972,"kb_used_data":1959620,"kb_used_omap":4,"kb_used_meta":366331,"kb_avail":91419180,"statfs":{"total":95995035648,"available":93613240320,"internally_reserved":0,"allocated":2006650880,"data_stored":2005944583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":375123624},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463109,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3879240,"kb_used_data":3478592,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":89865912,"statfs":{"total":95995035648,"available":92022693888,"internally_reserved":0,"allocated":3562078208,"data_stored":3561453621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":410057158},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":345,"seq":1481763717156,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2701968,"kb_used_data":2539012,"kb_used_omap":4,"kb_used_meta":162939,"kb_avail":91043184,"statfs":{"total":95995035648,"available":93228220416,"internally_reserved":0,"allocated":2599948288,"data_stored":2599321019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":166850211},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920978,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4475600,"kb_used_data":3979144,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":89269552,"statfs":{"total":95995035648,"available":91412021248,"internally_reserved":0,"allocated":4074643456,"data_stored":4074010763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":508164549},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,1,3,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":240,"seq":1030792151129,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2855448,"kb_used_data":2462216,"kb_used_omap":4,"kb_used_meta":393211,"kb_avail":90889704,"statfs":{"total":95995035648,"available":93071056896,"internally_reserved":0,"allocated":2521309184,"data_stored":2520599322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4443,"internal_metadata":402648741},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":130,"seq":558345748631,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3595352,"kb_used_data":2977288,"kb_used_omap":4,"kb_used_meta":618043,"kb_avail":90149800,"statfs":{"total":95995035648,"available":92313395200,"internally_reserved":0,"allocated":3048742912,"data_stored":3048111569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4441,"internal_metadata":632876711},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575076,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2274444,"kb_used_data":1629444,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":91470708,"statfs":{"total":95995035648,"available":93666004992,"internally_reserved":0,"allocated":1668550656,"data_stored":1667857845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":660273605},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738602,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5662640,"kb_used_data":5031528,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":88082512,"statfs":{"total":95995035648,"available":90196492288,"internally_reserved":0,"allocated":5152284672,"data_stored":5151684065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":646052293},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":2448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5166465024,"data_stored":5166465024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1674379264,"data_stored":1674379264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3056992256,"data_stored":3056992256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2520645632,"data_stored":2520645632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4090298368,"data_stored":4090298368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2608922624,"data_stored":2608922624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3587244032,"data_stored":3587244032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2006908928,"data_stored":2006908928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T14:04:38.237 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd pool set unique_pool_3 pgp_num 16 2024-09-21T14:04:38.238 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:38.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:38.305 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:38.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:38.406 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:04:38.442 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:38.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:38.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:04:38.477 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:38 smithi049 ceph-mon[40489]: from='client.25951 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:04:38.477 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:38 smithi049 ceph-mon[40489]: osdmap e413: 8 total, 8 up, 8 in 2024-09-21T14:04:38.477 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:38 smithi049 ceph-mon[40489]: 10.7 scrub starts 2024-09-21T14:04:38.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:38 smithi049 ceph-mon[30080]: from='client.25951 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:04:38.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:38 smithi049 ceph-mon[30080]: osdmap e413: 8 total, 8 up, 8 in 2024-09-21T14:04:38.478 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:38 smithi049 ceph-mon[30080]: 10.7 scrub starts 2024-09-21T14:04:38.526 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T14:04:38.526 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-09-21T14:04:38.526 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-09-21T14:04:38.526 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-09-21T14:04:38.526 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.526 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.15207.0:232030 8.2 e239/183 8:49ac6d95:::benchmark_data_smithi049_109311_object105511:head v 239'14230, mlcod=239'14227)", 2024-09-21T14:04:38.527 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.654181+0000", 2024-09-21T14:04:38.527 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 446.86543820600002, 2024-09-21T14:04:38.527 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.227698072, 2024-09-21T14:04:38.527 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:38.527 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:38.527 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:38.527 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:38.527 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.527 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:38.527 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.654181+0000", 2024-09-21T14:04:38.527 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:38.527 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.527 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.528 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:38.528 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.654181+0000", 2024-09-21T14:04:38.528 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:38.528 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.528 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.528 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:38.528 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.654181+0000", 2024-09-21T14:04:38.528 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.44e-07 2024-09-21T14:04:38.528 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.528 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.528 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:38.528 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.654188+0000", 2024-09-21T14:04:38.528 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1080000000000004e-06 2024-09-21T14:04:38.528 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.529 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.529 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:38.529 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.654202+0000", 2024-09-21T14:04:38.529 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3283e-05 2024-09-21T14:04:38.529 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.529 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.529 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:38.529 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.654208+0000", 2024-09-21T14:04:38.529 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1179999999999997e-06 2024-09-21T14:04:38.529 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.529 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.529 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:38.529 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.654644+0000", 2024-09-21T14:04:38.530 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000436191 2024-09-21T14:04:38.530 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.530 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.530 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:04:38.530 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.654661+0000", 2024-09-21T14:04:38.530 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6623e-05 2024-09-21T14:04:38.530 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.530 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.530 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:04:38.530 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.881820+0000", 2024-09-21T14:04:38.530 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22715939499999999 2024-09-21T14:04:38.530 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.530 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.530 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:38.531 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.881879+0000", 2024-09-21T14:04:38.531 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8709999999999999e-05 2024-09-21T14:04:38.531 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:38.531 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:38.531 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:38.531 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.531 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.531 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232032 8.e 8:755f5137:::benchmark_data_smithi049_109311_object105513:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T14:04:38.531 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.656185+0000", 2024-09-21T14:04:38.531 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 446.86343415200002, 2024-09-21T14:04:38.531 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22610913499999999, 2024-09-21T14:04:38.531 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:38.531 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:38.532 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:38.532 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:04:38.532 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T14:04:38.532 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:04:38.532 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232032 2024-09-21T14:04:38.532 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.532 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:38.532 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.532 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:38.532 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.656185+0000", 2024-09-21T14:04:38.532 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:38.532 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.532 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.532 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:38.533 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.656185+0000", 2024-09-21T14:04:38.533 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:38.533 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.533 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.533 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:38.533 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.656187+0000", 2024-09-21T14:04:38.533 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.215e-06 2024-09-21T14:04:38.533 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.533 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.533 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:38.533 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.656193+0000", 2024-09-21T14:04:38.533 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8900000000000004e-06 2024-09-21T14:04:38.533 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.533 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.533 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:38.534 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.656207+0000", 2024-09-21T14:04:38.534 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3886999999999999e-05 2024-09-21T14:04:38.534 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.534 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.534 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:38.534 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.656214+0000", 2024-09-21T14:04:38.534 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6210000000000004e-06 2024-09-21T14:04:38.534 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.534 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.534 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:38.534 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.658622+0000", 2024-09-21T14:04:38.534 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0024080999999999998 2024-09-21T14:04:38.534 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.534 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.534 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:04:38.535 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.658725+0000", 2024-09-21T14:04:38.535 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000102497 2024-09-21T14:04:38.535 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.535 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.535 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,4", 2024-09-21T14:04:38.535 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.658827+0000", 2024-09-21T14:04:38.535 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000101846 2024-09-21T14:04:38.535 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.535 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.535 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:38.535 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.666078+0000", 2024-09-21T14:04:38.535 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0072517420000000003 2024-09-21T14:04:38.535 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.535 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.535 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:04:38.536 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.666142+0000", 2024-09-21T14:04:38.536 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3538000000000001e-05 2024-09-21T14:04:38.536 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.536 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.536 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:38.536 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.882193+0000", 2024-09-21T14:04:38.536 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21605080600000001 2024-09-21T14:04:38.536 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.536 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.536 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:04:38.536 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.882250+0000", 2024-09-21T14:04:38.536 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6773e-05 2024-09-21T14:04:38.536 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.536 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.536 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:38.537 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.882294+0000", 2024-09-21T14:04:38.537 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.422e-05 2024-09-21T14:04:38.537 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:38.537 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:38.537 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:38.537 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.537 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.537 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.15207.0:232036 8.2 e239/183 8:4507ec93:::benchmark_data_smithi049_109311_object105517:head v 239'14231, mlcod=239'14227)", 2024-09-21T14:04:38.537 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.656873+0000", 2024-09-21T14:04:38.537 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 446.86274563000001, 2024-09-21T14:04:38.537 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22504153599999999, 2024-09-21T14:04:38.537 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:38.537 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:38.537 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:38.537 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:38.537 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.538 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:38.538 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.656873+0000", 2024-09-21T14:04:38.538 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:38.538 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.538 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.538 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:38.538 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.656873+0000", 2024-09-21T14:04:38.538 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:38.538 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.538 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.538 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:38.538 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.656874+0000", 2024-09-21T14:04:38.538 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7e-07 2024-09-21T14:04:38.538 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.538 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.539 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:38.539 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.656877+0000", 2024-09-21T14:04:38.539 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4489999999999999e-06 2024-09-21T14:04:38.539 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.539 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.539 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:38.539 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.656887+0000", 2024-09-21T14:04:38.539 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9550000000000004e-06 2024-09-21T14:04:38.539 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.539 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.539 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:38.539 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.656892+0000", 2024-09-21T14:04:38.539 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6630000000000001e-06 2024-09-21T14:04:38.539 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.540 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.540 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:38.540 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.670143+0000", 2024-09-21T14:04:38.540 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.013251447 2024-09-21T14:04:38.540 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.540 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.540 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:04:38.540 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.670163+0000", 2024-09-21T14:04:38.540 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0185000000000001e-05 2024-09-21T14:04:38.540 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.540 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.540 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:04:38.540 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.881884+0000", 2024-09-21T14:04:38.541 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21172070400000001 2024-09-21T14:04:38.541 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.541 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.541 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:38.541 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.881915+0000", 2024-09-21T14:04:38.541 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.0762999999999997e-05 2024-09-21T14:04:38.541 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:38.541 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:38.541 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:38.541 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.541 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.541 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.15207.0:232032 8.e e240/183 ondisk, result = 0)", 2024-09-21T14:04:38.541 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.685964+0000", 2024-09-21T14:04:38.541 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 446.833654685, 2024-09-21T14:04:38.541 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.19633603699999999, 2024-09-21T14:04:38.542 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:38.542 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:38.542 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T14:04:38.542 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:38.542 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.542 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:38.542 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.685964+0000", 2024-09-21T14:04:38.542 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:38.542 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.542 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.542 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:38.542 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.685964+0000", 2024-09-21T14:04:38.542 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:38.542 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.542 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.543 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:38.543 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.685965+0000", 2024-09-21T14:04:38.543 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2890000000000001e-06 2024-09-21T14:04:38.543 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.543 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.543 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:38.543 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.685971+0000", 2024-09-21T14:04:38.543 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4280000000000004e-06 2024-09-21T14:04:38.543 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.543 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.543 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:38.543 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.881522+0000", 2024-09-21T14:04:38.543 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.19555102499999999 2024-09-21T14:04:38.543 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.544 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.544 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:38.544 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.881535+0000", 2024-09-21T14:04:38.544 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3157e-05 2024-09-21T14:04:38.544 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.544 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.544 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:38.544 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.882178+0000", 2024-09-21T14:04:38.544 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00064299300000000002 2024-09-21T14:04:38.544 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.544 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.544 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:04:38.544 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.882187+0000", 2024-09-21T14:04:38.544 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2790000000000004e-06 2024-09-21T14:04:38.545 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.545 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.545 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:38.545 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.882300+0000", 2024-09-21T14:04:38.545 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000112866 2024-09-21T14:04:38.545 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:38.545 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:38.545 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:38.545 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.545 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.545 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25421.0:113894 9.b e296/278 9:ddec8d1f:::benchmark_data_smithi049_138504_object113893:head v 296'7127, mlcod=296'7123)", 2024-09-21T14:04:38.545 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:59:25.591505+0000", 2024-09-21T14:04:38.545 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 312.92811356800001, 2024-09-21T14:04:38.545 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.34647377800000001, 2024-09-21T14:04:38.546 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:38.546 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:38.546 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:38.546 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:38.546 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.546 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:38.546 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.591505+0000", 2024-09-21T14:04:38.546 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:38.546 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.546 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.546 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:38.546 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.591505+0000", 2024-09-21T14:04:38.546 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:38.546 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.546 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.547 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:38.547 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.591507+0000", 2024-09-21T14:04:38.547 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3319999999999999e-06 2024-09-21T14:04:38.547 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.547 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.547 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:38.547 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.591549+0000", 2024-09-21T14:04:38.547 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1995e-05 2024-09-21T14:04:38.547 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.547 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.547 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:38.547 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.591563+0000", 2024-09-21T14:04:38.547 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4436999999999999e-05 2024-09-21T14:04:38.547 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.548 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.548 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:38.548 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.591571+0000", 2024-09-21T14:04:38.548 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3920000000000003e-06 2024-09-21T14:04:38.548 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.548 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.548 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:38.548 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.592738+0000", 2024-09-21T14:04:38.548 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001166818 2024-09-21T14:04:38.548 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.548 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.548 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:04:38.548 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.592758+0000", 2024-09-21T14:04:38.548 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9273e-05 2024-09-21T14:04:38.548 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.549 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.549 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:04:38.549 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.596266+0000", 2024-09-21T14:04:38.549 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0035082199999999998 2024-09-21T14:04:38.549 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.549 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.549 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:38.549 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.937979+0000", 2024-09-21T14:04:38.549 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.34171331100000002 2024-09-21T14:04:38.549 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:38.549 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:38.549 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:38.549 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.549 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.549 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25421.0:113904 9.5 e296/273 9:a9988855:::benchmark_data_smithi049_138504_object113903:head v 296'7353, mlcod=296'7351)", 2024-09-21T14:04:38.550 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:59:25.594163+0000", 2024-09-21T14:04:38.550 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 312.925455795, 2024-09-21T14:04:38.550 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.34383867899999998, 2024-09-21T14:04:38.550 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:38.550 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:38.550 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:38.550 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:38.550 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.550 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:38.550 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.594163+0000", 2024-09-21T14:04:38.550 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:38.550 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.550 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.551 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:38.551 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.594163+0000", 2024-09-21T14:04:38.551 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:38.551 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.551 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.551 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:38.551 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.594165+0000", 2024-09-21T14:04:38.551 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.468e-06 2024-09-21T14:04:38.551 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.551 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.551 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:38.551 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.594212+0000", 2024-09-21T14:04:38.551 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7679999999999998e-05 2024-09-21T14:04:38.551 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.552 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.552 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:38.552 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.594227+0000", 2024-09-21T14:04:38.552 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.483e-05 2024-09-21T14:04:38.552 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.552 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.552 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:38.552 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.594235+0000", 2024-09-21T14:04:38.552 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6020000000000002e-06 2024-09-21T14:04:38.552 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.552 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.552 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:38.552 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.594307+0000", 2024-09-21T14:04:38.552 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2090999999999998e-05 2024-09-21T14:04:38.553 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.553 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.553 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:04:38.553 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.594328+0000", 2024-09-21T14:04:38.553 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1406000000000001e-05 2024-09-21T14:04:38.553 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.553 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.553 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:04:38.553 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.937961+0000", 2024-09-21T14:04:38.553 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.343633148 2024-09-21T14:04:38.553 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.553 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.553 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:38.553 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.938002+0000", 2024-09-21T14:04:38.554 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0454000000000003e-05 2024-09-21T14:04:38.554 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:38.554 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:38.554 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:38.554 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.554 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.554 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25421.0:113910 9.5 e296/273 9:a516f056:::benchmark_data_smithi049_138504_object113909:head v 296'7354, mlcod=296'7351)", 2024-09-21T14:04:38.554 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:59:25.597837+0000", 2024-09-21T14:04:38.554 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 312.92178205099998, 2024-09-21T14:04:38.554 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.346102413, 2024-09-21T14:04:38.554 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:38.554 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:38.554 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:38.554 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:38.555 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.555 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:38.555 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.597837+0000", 2024-09-21T14:04:38.555 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:38.555 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.555 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.555 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:38.555 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.597837+0000", 2024-09-21T14:04:38.555 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:38.555 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.555 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.555 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:38.555 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.597838+0000", 2024-09-21T14:04:38.555 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3909999999999999e-06 2024-09-21T14:04:38.556 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.556 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.556 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:38.556 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.597870+0000", 2024-09-21T14:04:38.556 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2209000000000003e-05 2024-09-21T14:04:38.556 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.556 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.556 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:38.556 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.937999+0000", 2024-09-21T14:04:38.556 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.340128284 2024-09-21T14:04:38.556 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.556 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.556 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:38.556 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.938005+0000", 2024-09-21T14:04:38.556 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2870000000000001e-06 2024-09-21T14:04:38.557 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.557 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.557 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:38.557 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.938058+0000", 2024-09-21T14:04:38.557 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2608000000000002e-05 2024-09-21T14:04:38.557 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.557 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.557 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:04:38.557 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.938086+0000", 2024-09-21T14:04:38.557 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.8699000000000001e-05 2024-09-21T14:04:38.557 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.557 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.557 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:04:38.558 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.943789+0000", 2024-09-21T14:04:38.558 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0057023810000000003 2024-09-21T14:04:38.558 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.558 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.558 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:38.558 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.943939+0000", 2024-09-21T14:04:38.558 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00015055400000000001 2024-09-21T14:04:38.558 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:38.558 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:38.558 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:38.558 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.558 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.558 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25421.0:113916 9.8 e296/260 9:1f612498:::benchmark_data_smithi049_138504_object113915:head v 296'7254, mlcod=296'7252)", 2024-09-21T14:04:38.558 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:59:25.603665+0000", 2024-09-21T14:04:38.559 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 312.91595408000001, 2024-09-21T14:04:38.559 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.33941505399999999, 2024-09-21T14:04:38.559 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:38.559 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:38.559 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:38.559 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:38.559 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.559 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:38.559 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.603665+0000", 2024-09-21T14:04:38.559 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:38.559 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.559 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.559 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:38.559 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.603665+0000", 2024-09-21T14:04:38.560 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:38.560 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.560 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.560 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:38.560 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.603666+0000", 2024-09-21T14:04:38.560 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5829999999999999e-06 2024-09-21T14:04:38.560 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.560 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.560 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:38.560 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.603694+0000", 2024-09-21T14:04:38.560 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.7767999999999999e-05 2024-09-21T14:04:38.560 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.560 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.560 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:38.561 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.937900+0000", 2024-09-21T14:04:38.561 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.33420539999999999 2024-09-21T14:04:38.561 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.561 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.561 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:38.561 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.937917+0000", 2024-09-21T14:04:38.561 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7156999999999999e-05 2024-09-21T14:04:38.561 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.561 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.561 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:38.561 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.938061+0000", 2024-09-21T14:04:38.561 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014396600000000001 2024-09-21T14:04:38.561 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.561 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.562 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:04:38.562 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.938085+0000", 2024-09-21T14:04:38.562 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4607e-05 2024-09-21T14:04:38.562 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.562 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.562 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:04:38.562 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.942769+0000", 2024-09-21T14:04:38.562 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0046839270000000001 2024-09-21T14:04:38.562 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.562 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.562 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:38.562 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.943080+0000", 2024-09-21T14:04:38.562 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00031064599999999998 2024-09-21T14:04:38.563 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:38.563 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:38.563 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:38.563 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.563 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.563 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25421.0:113922 9.b e296/278 9:db1a22c4:::benchmark_data_smithi049_138504_object113921:head v 296'7128, mlcod=296'7125)", 2024-09-21T14:04:38.563 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:59:25.605556+0000", 2024-09-21T14:04:38.563 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 312.91406281500002, 2024-09-21T14:04:38.563 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.34451127500000001, 2024-09-21T14:04:38.563 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:38.563 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:38.563 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:38.563 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:38.563 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.563 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:38.564 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.605556+0000", 2024-09-21T14:04:38.564 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:38.564 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.564 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.564 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:38.564 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.605556+0000", 2024-09-21T14:04:38.564 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:38.564 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.564 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.564 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:38.564 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.605557+0000", 2024-09-21T14:04:38.564 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.184e-06 2024-09-21T14:04:38.565 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.565 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.565 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:38.565 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.605595+0000", 2024-09-21T14:04:38.565 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8139000000000002e-05 2024-09-21T14:04:38.565 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.565 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.565 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:38.565 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.937938+0000", 2024-09-21T14:04:38.565 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.33234267899999997 2024-09-21T14:04:38.565 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.565 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.565 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:38.565 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.937946+0000", 2024-09-21T14:04:38.566 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9240000000000007e-06 2024-09-21T14:04:38.566 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.566 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.566 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:38.566 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.938185+0000", 2024-09-21T14:04:38.566 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000239108 2024-09-21T14:04:38.566 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.566 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.566 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:04:38.566 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.938206+0000", 2024-09-21T14:04:38.566 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0693999999999999e-05 2024-09-21T14:04:38.566 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.566 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.566 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:04:38.567 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.949931+0000", 2024-09-21T14:04:38.567 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.011725279 2024-09-21T14:04:38.567 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.567 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.567 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:38.567 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.950067+0000", 2024-09-21T14:04:38.567 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013626799999999999 2024-09-21T14:04:38.567 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:38.567 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:38.567 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:38.567 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.567 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.567 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25421.0:170760 9.9 9:9867205a:::benchmark_data_smithi049_138504_object49352:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-09-21T14:04:38.568 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:38.569 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:00:31.314976+0000", 2024-09-21T14:04:38.569 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 247.20464279199999, 2024-09-21T14:04:38.569 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.13146185399999999, 2024-09-21T14:04:38.569 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:38.569 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:38.569 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:38.569 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:04:38.569 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25421", 2024-09-21T14:04:38.569 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:04:38.569 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 170760 2024-09-21T14:04:38.569 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.569 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:38.570 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.570 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:38.570 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:31.314976+0000", 2024-09-21T14:04:38.570 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:38.570 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.570 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.570 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:38.570 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:31.314976+0000", 2024-09-21T14:04:38.570 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:38.570 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.570 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.570 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:38.570 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:31.314977+0000", 2024-09-21T14:04:38.570 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2200000000000002e-07 2024-09-21T14:04:38.571 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.571 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.571 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:38.571 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:31.314982+0000", 2024-09-21T14:04:38.571 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7099999999999998e-06 2024-09-21T14:04:38.571 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.571 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.571 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:38.571 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:31.314990+0000", 2024-09-21T14:04:38.571 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6589999999999996e-06 2024-09-21T14:04:38.571 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.571 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.571 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:38.571 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:31.314995+0000", 2024-09-21T14:04:38.572 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9980000000000003e-06 2024-09-21T14:04:38.572 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.572 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.572 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:38.572 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:31.426242+0000", 2024-09-21T14:04:38.572 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.111246288 2024-09-21T14:04:38.572 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.572 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.572 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:04:38.572 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:31.426415+0000", 2024-09-21T14:04:38.572 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00017327 2024-09-21T14:04:38.572 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.572 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.572 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,7", 2024-09-21T14:04:38.572 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:31.426597+0000", 2024-09-21T14:04:38.573 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00018237099999999999 2024-09-21T14:04:38.573 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.573 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.573 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:04:38.573 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:31.432628+0000", 2024-09-21T14:04:38.573 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0060311790000000002 2024-09-21T14:04:38.573 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.573 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.573 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:38.573 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:31.444042+0000", 2024-09-21T14:04:38.573 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.011413510999999999 2024-09-21T14:04:38.573 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.573 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.573 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:38.573 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:31.444090+0000", 2024-09-21T14:04:38.574 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7905999999999998e-05 2024-09-21T14:04:38.574 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.574 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.574 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:04:38.574 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:31.444150+0000", 2024-09-21T14:04:38.574 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0177000000000003e-05 2024-09-21T14:04:38.574 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.574 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.574 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:38.574 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:31.446438+0000", 2024-09-21T14:04:38.574 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0022878629999999998 2024-09-21T14:04:38.574 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:38.574 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:38.574 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:38.575 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.575 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.575 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.16068.0:70919 10.5 e398/381 10:af948525:::benchmark_data_smithi049_176811_object70918:head v 398'4522, mlcod=398'4519)", 2024-09-21T14:04:38.575 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:03:44.036638+0000", 2024-09-21T14:04:38.575 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 54.482980972999997, 2024-09-21T14:04:38.575 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.299326869, 2024-09-21T14:04:38.575 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:38.575 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:38.575 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:38.575 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:38.575 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.575 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:38.576 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.036638+0000", 2024-09-21T14:04:38.576 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:38.576 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.576 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.576 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:38.576 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.036638+0000", 2024-09-21T14:04:38.576 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:38.576 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.576 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.576 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:38.576 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.036639+0000", 2024-09-21T14:04:38.576 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6599999999999995e-07 2024-09-21T14:04:38.576 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.577 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.577 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:38.577 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.036714+0000", 2024-09-21T14:04:38.577 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4894000000000005e-05 2024-09-21T14:04:38.577 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.577 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.577 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:38.577 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.036724+0000", 2024-09-21T14:04:38.577 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0002e-05 2024-09-21T14:04:38.577 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.577 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.577 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:38.577 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.036730+0000", 2024-09-21T14:04:38.577 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7050000000000002e-06 2024-09-21T14:04:38.578 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.578 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.578 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:38.578 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.038425+0000", 2024-09-21T14:04:38.578 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0016944970000000001 2024-09-21T14:04:38.578 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.578 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.578 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:04:38.578 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.038462+0000", 2024-09-21T14:04:38.578 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7218999999999997e-05 2024-09-21T14:04:38.578 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.578 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.578 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:04:38.578 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.335918+0000", 2024-09-21T14:04:38.578 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.297456108 2024-09-21T14:04:38.579 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.579 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.579 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:38.579 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.335965+0000", 2024-09-21T14:04:38.579 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6678000000000003e-05 2024-09-21T14:04:38.579 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:38.579 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:38.579 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:38.579 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.579 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.579 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.16068.0:70926 10.5 e398/381 10:a9932ee5:::benchmark_data_smithi049_176811_object70925:head v 398'4523, mlcod=398'4519)", 2024-09-21T14:04:38.579 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:03:44.042122+0000", 2024-09-21T14:04:38.579 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 54.477497057999997, 2024-09-21T14:04:38.579 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.29753411899999999, 2024-09-21T14:04:38.580 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:38.580 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:38.580 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:38.580 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:38.580 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.580 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:38.580 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.042122+0000", 2024-09-21T14:04:38.580 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:38.580 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.580 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.580 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:38.580 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.042122+0000", 2024-09-21T14:04:38.580 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:38.580 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.580 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.580 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:38.581 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.042123+0000", 2024-09-21T14:04:38.581 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0189999999999999e-06 2024-09-21T14:04:38.581 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.581 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.581 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:38.581 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.042146+0000", 2024-09-21T14:04:38.581 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3646999999999999e-05 2024-09-21T14:04:38.581 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.581 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.581 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:38.581 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.042156+0000", 2024-09-21T14:04:38.581 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2529999999999993e-06 2024-09-21T14:04:38.581 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.581 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.582 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:38.582 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.042162+0000", 2024-09-21T14:04:38.582 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7039999999999997e-06 2024-09-21T14:04:38.582 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.582 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.582 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:38.582 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044662+0000", 2024-09-21T14:04:38.582 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0024995030000000001 2024-09-21T14:04:38.582 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.582 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.582 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:04:38.582 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044684+0000", 2024-09-21T14:04:38.582 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2158e-05 2024-09-21T14:04:38.582 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.583 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.583 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:04:38.583 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.339619+0000", 2024-09-21T14:04:38.583 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.29493498899999998 2024-09-21T14:04:38.583 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.583 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.583 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:38.583 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.339656+0000", 2024-09-21T14:04:38.583 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6845999999999999e-05 2024-09-21T14:04:38.583 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:38.583 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:38.583 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:38.583 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.583 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.584 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.16068.0:70930 10.6 e398/381 10:67bb9b1d:::benchmark_data_smithi049_176811_object70929:head v 398'4447, mlcod=398'4444)", 2024-09-21T14:04:38.584 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:03:44.042229+0000", 2024-09-21T14:04:38.584 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 54.477390145999998, 2024-09-21T14:04:38.584 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.29373393599999997, 2024-09-21T14:04:38.584 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:38.584 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:38.584 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:38.584 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:38.584 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.584 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:38.584 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.042229+0000", 2024-09-21T14:04:38.584 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:38.584 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.584 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.585 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:38.585 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.042229+0000", 2024-09-21T14:04:38.585 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:38.585 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.585 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.585 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:38.585 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.042229+0000", 2024-09-21T14:04:38.585 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6699999999999999e-07 2024-09-21T14:04:38.585 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.585 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.585 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:38.585 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.042257+0000", 2024-09-21T14:04:38.585 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.8115000000000001e-05 2024-09-21T14:04:38.585 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.585 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.585 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:38.586 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.042266+0000", 2024-09-21T14:04:38.586 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.7090000000000008e-06 2024-09-21T14:04:38.586 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.586 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.586 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:38.586 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.042272+0000", 2024-09-21T14:04:38.586 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.604e-06 2024-09-21T14:04:38.586 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.586 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.586 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:38.586 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.042333+0000", 2024-09-21T14:04:38.586 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1248999999999996e-05 2024-09-21T14:04:38.586 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.587 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.587 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:04:38.587 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.042350+0000", 2024-09-21T14:04:38.587 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7209000000000001e-05 2024-09-21T14:04:38.587 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.587 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.587 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:04:38.587 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.048124+0000", 2024-09-21T14:04:38.587 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0057740329999999996 2024-09-21T14:04:38.587 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.587 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.587 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:38.587 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.335963+0000", 2024-09-21T14:04:38.587 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.28783855000000003 2024-09-21T14:04:38.588 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:38.588 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:38.588 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:38.588 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.588 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.588 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.16068.0:70932 10.8 e398/381 10:1f467600:::benchmark_data_smithi049_176811_object70931:head v 398'4447, mlcod=398'4445)", 2024-09-21T14:04:38.588 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:03:44.045179+0000", 2024-09-21T14:04:38.588 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 54.474439787000001, 2024-09-21T14:04:38.588 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.29080314400000001, 2024-09-21T14:04:38.588 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:38.588 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:38.588 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:38.588 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:38.588 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.588 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:38.589 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.045179+0000", 2024-09-21T14:04:38.589 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:38.589 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.589 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.589 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:38.589 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.045179+0000", 2024-09-21T14:04:38.589 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:38.589 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.589 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.589 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:38.589 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.045180+0000", 2024-09-21T14:04:38.589 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.879999999999999e-07 2024-09-21T14:04:38.589 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.589 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.590 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:38.590 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.045223+0000", 2024-09-21T14:04:38.590 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2972999999999997e-05 2024-09-21T14:04:38.590 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.590 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.590 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:38.590 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.045234+0000", 2024-09-21T14:04:38.590 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1106e-05 2024-09-21T14:04:38.590 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.590 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.590 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:38.590 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.045241+0000", 2024-09-21T14:04:38.590 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9319999999999999e-06 2024-09-21T14:04:38.590 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.591 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.591 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:38.591 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.045392+0000", 2024-09-21T14:04:38.591 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000151244 2024-09-21T14:04:38.591 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.591 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.591 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:04:38.591 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.045411+0000", 2024-09-21T14:04:38.591 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8811e-05 2024-09-21T14:04:38.591 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.591 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.591 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:04:38.591 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.335942+0000", 2024-09-21T14:04:38.591 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.29053121199999998 2024-09-21T14:04:38.592 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.592 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.592 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:38.592 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.335982+0000", 2024-09-21T14:04:38.592 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9878e-05 2024-09-21T14:04:38.592 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:38.592 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:38.592 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:38.592 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.592 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.592 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.16068.0:70929 10.5 e398/381 10:a6d68c9a:::benchmark_data_smithi049_176811_object70928:head v 398'4524, mlcod=398'4519)", 2024-09-21T14:04:38.592 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:03:44.045264+0000", 2024-09-21T14:04:38.592 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 54.474354474000002, 2024-09-21T14:04:38.592 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.29697228799999997, 2024-09-21T14:04:38.593 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:38.593 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:38.593 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:38.593 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:38.593 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.593 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:38.593 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.045264+0000", 2024-09-21T14:04:38.593 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:38.593 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.593 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.593 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:38.593 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.045264+0000", 2024-09-21T14:04:38.593 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:38.593 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.593 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.594 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:38.594 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.045265+0000", 2024-09-21T14:04:38.594 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8500000000000002e-07 2024-09-21T14:04:38.594 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.594 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.594 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:38.594 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.045286+0000", 2024-09-21T14:04:38.594 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1155e-05 2024-09-21T14:04:38.594 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.594 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.594 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:38.594 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.045292+0000", 2024-09-21T14:04:38.594 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.596e-06 2024-09-21T14:04:38.594 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.595 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.595 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:38.595 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.045298+0000", 2024-09-21T14:04:38.595 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1830000000000002e-06 2024-09-21T14:04:38.595 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.595 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.595 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:38.595 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.047376+0000", 2024-09-21T14:04:38.595 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0020778419999999999 2024-09-21T14:04:38.595 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.595 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.595 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:04:38.595 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.047552+0000", 2024-09-21T14:04:38.595 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000176613 2024-09-21T14:04:38.596 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.596 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.596 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:04:38.596 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.342198+0000", 2024-09-21T14:04:38.596 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.29464633800000001 2024-09-21T14:04:38.596 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.596 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.596 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:38.596 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.342237+0000", 2024-09-21T14:04:38.596 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8176000000000002e-05 2024-09-21T14:04:38.596 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:38.596 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:38.596 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:38.597 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.597 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.597 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.16068.0:70938 10.b e398/381 10:dcded94b:::benchmark_data_smithi049_176811_object70937:head v 398'4329, mlcod=398'4327)", 2024-09-21T14:04:38.597 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:03:44.046615+0000", 2024-09-21T14:04:38.597 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 54.473004007, 2024-09-21T14:04:38.597 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.28946017299999999, 2024-09-21T14:04:38.597 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:38.597 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:38.597 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:38.597 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:38.597 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.597 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:38.597 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.046615+0000", 2024-09-21T14:04:38.597 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:38.597 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.598 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.598 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:38.598 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.046615+0000", 2024-09-21T14:04:38.598 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:38.598 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.598 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.598 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:38.598 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.046616+0000", 2024-09-21T14:04:38.598 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4099999999999998e-07 2024-09-21T14:04:38.598 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.598 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.598 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:38.598 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.046649+0000", 2024-09-21T14:04:38.599 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3068000000000003e-05 2024-09-21T14:04:38.599 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.599 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.599 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:38.599 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.046661+0000", 2024-09-21T14:04:38.599 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2731999999999999e-05 2024-09-21T14:04:38.599 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.599 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.599 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:38.599 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.046668+0000", 2024-09-21T14:04:38.599 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0330000000000003e-06 2024-09-21T14:04:38.599 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.599 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.599 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:38.600 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.047195+0000", 2024-09-21T14:04:38.600 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00052707299999999997 2024-09-21T14:04:38.600 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.600 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.600 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:04:38.600 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.047214+0000", 2024-09-21T14:04:38.600 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8437e-05 2024-09-21T14:04:38.600 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.600 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.600 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:04:38.600 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.336022+0000", 2024-09-21T14:04:38.600 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.28880810499999998 2024-09-21T14:04:38.600 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.600 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.601 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:38.601 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.336075+0000", 2024-09-21T14:04:38.601 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2984e-05 2024-09-21T14:04:38.601 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:38.601 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:38.601 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:38.601 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.601 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.601 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.16068.0:70936 10.4 e398/381 10:2fa37e00:::benchmark_data_smithi049_176811_object70935:head v 398'4409, mlcod=398'4407)", 2024-09-21T14:04:38.601 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:03:44.046702+0000", 2024-09-21T14:04:38.601 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 54.472916910000002, 2024-09-21T14:04:38.601 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.29141013199999999, 2024-09-21T14:04:38.601 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:38.601 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:38.602 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:38.602 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:38.602 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.602 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:38.602 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.046702+0000", 2024-09-21T14:04:38.602 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:38.602 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.602 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.602 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:38.602 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.046702+0000", 2024-09-21T14:04:38.602 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:38.602 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.602 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.603 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:38.603 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.046702+0000", 2024-09-21T14:04:38.603 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.32e-07 2024-09-21T14:04:38.603 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.603 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.603 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:38.603 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.046739+0000", 2024-09-21T14:04:38.603 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6627e-05 2024-09-21T14:04:38.603 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.603 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.603 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:38.603 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.046747+0000", 2024-09-21T14:04:38.603 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5549999999999997e-06 2024-09-21T14:04:38.603 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.603 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.604 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:38.604 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.046751+0000", 2024-09-21T14:04:38.604 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9250000000000003e-06 2024-09-21T14:04:38.604 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.604 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.604 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:38.604 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.046881+0000", 2024-09-21T14:04:38.604 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012939200000000001 2024-09-21T14:04:38.604 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.604 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.604 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:04:38.604 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.046900+0000", 2024-09-21T14:04:38.604 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8694000000000001e-05 2024-09-21T14:04:38.604 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.604 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.605 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:04:38.605 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.337972+0000", 2024-09-21T14:04:38.605 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.29107263999999999 2024-09-21T14:04:38.605 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.605 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.605 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:38.605 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.338112+0000", 2024-09-21T14:04:38.605 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013986700000000001 2024-09-21T14:04:38.605 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:38.605 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:38.605 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:38.605 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.605 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.605 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.16068.0:70937 10.1 e398/381 10:88bdcf7f:::benchmark_data_smithi049_176811_object70936:head v 398'4600, mlcod=398'4598)", 2024-09-21T14:04:38.605 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:03:44.046765+0000", 2024-09-21T14:04:38.605 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 54.472853438000001, 2024-09-21T14:04:38.606 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.293462367, 2024-09-21T14:04:38.606 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:38.606 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:38.606 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:38.606 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:38.606 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.606 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:38.606 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.046765+0000", 2024-09-21T14:04:38.606 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:38.606 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.606 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.606 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:38.606 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.046765+0000", 2024-09-21T14:04:38.607 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:38.607 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.607 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.607 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:38.607 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.046766+0000", 2024-09-21T14:04:38.607 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.8799999999999998e-07 2024-09-21T14:04:38.607 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.607 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.607 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:38.607 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.046798+0000", 2024-09-21T14:04:38.607 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2019e-05 2024-09-21T14:04:38.607 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.607 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.607 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:38.608 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.046804+0000", 2024-09-21T14:04:38.608 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1639999999999998e-06 2024-09-21T14:04:38.608 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.608 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.608 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:38.608 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.046808+0000", 2024-09-21T14:04:38.608 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4870000000000004e-06 2024-09-21T14:04:38.608 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.608 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.608 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:38.608 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.336616+0000", 2024-09-21T14:04:38.608 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.28980800699999998 2024-09-21T14:04:38.608 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.608 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.608 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:04:38.608 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.336635+0000", 2024-09-21T14:04:38.609 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8669e-05 2024-09-21T14:04:38.609 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.609 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.609 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:04:38.609 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.340183+0000", 2024-09-21T14:04:38.609 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0035481050000000002 2024-09-21T14:04:38.609 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.609 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.609 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:38.609 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.340228+0000", 2024-09-21T14:04:38.609 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4628e-05 2024-09-21T14:04:38.609 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:38.609 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:38.609 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:38.609 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.609 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.610 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.16068.0:70933 10.5 e398/381 10:a4f74f69:::benchmark_data_smithi049_176811_object70932:head v 398'4525, mlcod=398'4520)", 2024-09-21T14:04:38.610 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:03:44.048683+0000", 2024-09-21T14:04:38.610 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 54.470935627999999, 2024-09-21T14:04:38.610 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.29601093899999997, 2024-09-21T14:04:38.610 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:38.610 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:38.610 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:38.611 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:38.611 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.611 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:38.611 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.048683+0000", 2024-09-21T14:04:38.611 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:38.611 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.611 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.611 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:38.611 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.048683+0000", 2024-09-21T14:04:38.611 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:38.611 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.611 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.611 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:38.611 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.048684+0000", 2024-09-21T14:04:38.612 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.11e-06 2024-09-21T14:04:38.612 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.612 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.612 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:38.612 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.048709+0000", 2024-09-21T14:04:38.612 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4241999999999999e-05 2024-09-21T14:04:38.612 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.612 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.612 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:38.612 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.335916+0000", 2024-09-21T14:04:38.612 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.28720726600000002 2024-09-21T14:04:38.612 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.612 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.612 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:38.613 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.335929+0000", 2024-09-21T14:04:38.613 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3569e-05 2024-09-21T14:04:38.613 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.613 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.613 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:38.613 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.335995+0000", 2024-09-21T14:04:38.613 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5368000000000002e-05 2024-09-21T14:04:38.613 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.613 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.613 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:04:38.613 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.336020+0000", 2024-09-21T14:04:38.613 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5616e-05 2024-09-21T14:04:38.613 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.613 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.614 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:04:38.614 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.344629+0000", 2024-09-21T14:04:38.614 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0086085529999999997 2024-09-21T14:04:38.614 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.614 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.614 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:38.614 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.344694+0000", 2024-09-21T14:04:38.614 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5215000000000005e-05 2024-09-21T14:04:38.614 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:38.614 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:38.614 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:38.614 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.614 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.614 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.16068.0:70940 10.6 e398/381 10:6be6d6c4:::benchmark_data_smithi049_176811_object70939:head v 398'4448, mlcod=398'4445)", 2024-09-21T14:04:38.614 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:03:44.048795+0000", 2024-09-21T14:04:38.615 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 54.470824108999999, 2024-09-21T14:04:38.615 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.29301090000000002, 2024-09-21T14:04:38.615 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:38.615 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:38.615 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:38.615 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:38.615 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.615 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:38.615 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.048795+0000", 2024-09-21T14:04:38.615 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:38.615 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.615 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.615 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:38.615 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.048795+0000", 2024-09-21T14:04:38.616 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:38.616 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.616 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.616 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:38.616 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.048796+0000", 2024-09-21T14:04:38.616 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0279999999999999e-06 2024-09-21T14:04:38.616 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.616 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.616 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:38.616 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.048850+0000", 2024-09-21T14:04:38.616 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3940000000000002e-05 2024-09-21T14:04:38.616 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.616 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.616 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:38.616 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.335907+0000", 2024-09-21T14:04:38.617 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.28705735599999999 2024-09-21T14:04:38.617 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.617 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.617 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:38.617 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.335929+0000", 2024-09-21T14:04:38.617 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2294999999999999e-05 2024-09-21T14:04:38.617 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.617 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.617 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:38.617 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.335994+0000", 2024-09-21T14:04:38.617 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4984e-05 2024-09-21T14:04:38.617 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.617 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.618 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:04:38.618 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.336063+0000", 2024-09-21T14:04:38.618 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8738000000000005e-05 2024-09-21T14:04:38.618 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.618 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.618 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:04:38.618 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.341711+0000", 2024-09-21T14:04:38.618 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.005647961 2024-09-21T14:04:38.618 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:38.618 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:38.618 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:38.618 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.341806+0000", 2024-09-21T14:04:38.618 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4598000000000005e-05 2024-09-21T14:04:38.618 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:38.619 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:38.619 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:38.619 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:38.619 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:38.619 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T14:04:38.645 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:38.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:38.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:38.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:38 smithi187 ceph-mon[33032]: from='client.25951 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:04:38.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:38 smithi187 ceph-mon[33032]: osdmap e413: 8 total, 8 up, 8 in 2024-09-21T14:04:38.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:38 smithi187 ceph-mon[33032]: 10.7 scrub starts 2024-09-21T14:04:38.747 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:04:38.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:38.785 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:38.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:04:38.886 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:04:38.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:38.919 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:38.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:04:39.020 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:04:39.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:39.056 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:39.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:04:39.157 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:04:39.199 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:39.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:39.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:04:39.301 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:04:39.334 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:39.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:39.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:04:39.437 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:04:39.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:39.477 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:39.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:04:39.578 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:04:39.612 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:39.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:39.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:04:39.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:39 smithi187 ceph-mon[33032]: pgmap v797: 158 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 154 active+clean; 7.7 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 57 MiB/s wr, 964 op/s; 29496/377682 objects misplaced (7.810%); 5.1 MiB/s, 82 objects/s recovering 2024-09-21T14:04:39.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:39 smithi187 ceph-mon[33032]: 1.7 scrub starts 2024-09-21T14:04:39.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:39 smithi187 ceph-mon[33032]: 1.7 scrub ok 2024-09-21T14:04:39.715 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:04:39.751 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:39.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:39.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:04:39.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:39 smithi049 ceph-mon[40489]: pgmap v797: 158 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 154 active+clean; 7.7 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 57 MiB/s wr, 964 op/s; 29496/377682 objects misplaced (7.810%); 5.1 MiB/s, 82 objects/s recovering 2024-09-21T14:04:39.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:39 smithi049 ceph-mon[40489]: 1.7 scrub starts 2024-09-21T14:04:39.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:39 smithi049 ceph-mon[40489]: 1.7 scrub ok 2024-09-21T14:04:39.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:39 smithi049 ceph-mon[30080]: pgmap v797: 158 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 154 active+clean; 7.7 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 57 MiB/s wr, 964 op/s; 29496/377682 objects misplaced (7.810%); 5.1 MiB/s, 82 objects/s recovering 2024-09-21T14:04:39.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:39 smithi049 ceph-mon[30080]: 1.7 scrub starts 2024-09-21T14:04:39.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:39 smithi049 ceph-mon[30080]: 1.7 scrub ok 2024-09-21T14:04:39.853 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:04:39.874 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:39.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:39.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:04:39.976 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:04:40.003 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:40.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:40.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:04:40.106 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:04:40.143 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:40.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:40.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:04:40.194 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:04:39 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:04:40.244 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:04:40.278 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:40.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:40.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:04:40.380 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:04:40.415 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:40.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:40.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:04:40.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:40 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T14:04:40.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:40 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:04:40.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:40 smithi049 ceph-mon[40489]: 10.f scrub starts 2024-09-21T14:04:40.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:40 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T14:04:40.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:40 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:04:40.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:40 smithi049 ceph-mon[30080]: 10.f scrub starts 2024-09-21T14:04:40.517 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:04:40.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:40.564 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:40.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:04:40.665 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:04:40.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:40 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T14:04:40.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:40 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:04:40.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:40 smithi187 ceph-mon[33032]: 10.f scrub starts 2024-09-21T14:04:40.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:40.702 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:40.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:04:40.803 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:04:40.847 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:40.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:40.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:04:40.949 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:40.987 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:40.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:40.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:41.004 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:04:40 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:04:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:04:41.089 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:04:41.129 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:41.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:41.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:04:41.162 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T14:04:41.221 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:04:41.230 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:04:41.231 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:41.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:41.309 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:41.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:41.321 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:04:41.411 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:04:41.444 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:41.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:41.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:04:41.545 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:04:41.589 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:41.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:41.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:04:41.684 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:41 smithi049 ceph-mon[30080]: 4.d scrub starts 2024-09-21T14:04:41.684 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:41 smithi049 ceph-mon[30080]: 4.d scrub ok 2024-09-21T14:04:41.684 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:41 smithi049 ceph-mon[30080]: pgmap v798: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 153 active+clean; 7.6 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 767 B/s rd, 59 MiB/s wr, 1.26k op/s; 28662/375987 objects misplaced (7.623%); 10 MiB/s, 167 objects/s recovering 2024-09-21T14:04:41.684 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:41 smithi049 ceph-mon[30080]: 10.7 scrub ok 2024-09-21T14:04:41.684 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:41 smithi049 ceph-mon[30080]: 5.10 scrub starts 2024-09-21T14:04:41.684 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:41 smithi049 ceph-mon[30080]: 5.10 scrub ok 2024-09-21T14:04:41.684 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:41 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:04:41.684 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:41 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:04:41.688 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:41 smithi049 ceph-mon[40489]: 4.d scrub starts 2024-09-21T14:04:41.688 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:41 smithi049 ceph-mon[40489]: 4.d scrub ok 2024-09-21T14:04:41.688 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:41 smithi049 ceph-mon[40489]: pgmap v798: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 153 active+clean; 7.6 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 767 B/s rd, 59 MiB/s wr, 1.26k op/s; 28662/375987 objects misplaced (7.623%); 10 MiB/s, 167 objects/s recovering 2024-09-21T14:04:41.688 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:41 smithi049 ceph-mon[40489]: 10.7 scrub ok 2024-09-21T14:04:41.688 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:41 smithi049 ceph-mon[40489]: 5.10 scrub starts 2024-09-21T14:04:41.689 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:41 smithi049 ceph-mon[40489]: 5.10 scrub ok 2024-09-21T14:04:41.689 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:41 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:04:41.689 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:41 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:04:41.692 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:04:41.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:41 smithi187 ceph-mon[33032]: 4.d scrub starts 2024-09-21T14:04:41.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:41 smithi187 ceph-mon[33032]: 4.d scrub ok 2024-09-21T14:04:41.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:41 smithi187 ceph-mon[33032]: pgmap v798: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 153 active+clean; 7.6 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 767 B/s rd, 59 MiB/s wr, 1.26k op/s; 28662/375987 objects misplaced (7.623%); 10 MiB/s, 167 objects/s recovering 2024-09-21T14:04:41.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:41 smithi187 ceph-mon[33032]: 10.7 scrub ok 2024-09-21T14:04:41.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:41 smithi187 ceph-mon[33032]: 5.10 scrub starts 2024-09-21T14:04:41.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:41 smithi187 ceph-mon[33032]: 5.10 scrub ok 2024-09-21T14:04:41.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:41 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:04:41.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:41 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:04:41.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:41.765 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:41.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:04:41.866 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:04:41.897 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:41.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:41.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:04:41.999 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:42.041 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:42.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:42.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:42.143 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:04:42.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:42.204 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:42.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:04:42.305 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:04:42.364 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:42.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:42.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:04:42.451 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:04:42.451 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:04:42.451 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:04:42.452 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:04:42.452 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:04:42.452 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:04:42.466 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:04:42.485 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:42 smithi049 ceph-mon[30080]: 6.0 scrub starts 2024-09-21T14:04:42.485 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:42 smithi049 ceph-mon[30080]: 6.0 scrub ok 2024-09-21T14:04:42.485 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:42 smithi049 ceph-mon[30080]: 5.11 scrub starts 2024-09-21T14:04:42.485 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:42 smithi049 ceph-mon[30080]: 5.11 scrub ok 2024-09-21T14:04:42.486 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:42 smithi049 ceph-mon[30080]: 4.1e scrub starts 2024-09-21T14:04:42.486 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:42 smithi049 ceph-mon[30080]: 4.1e scrub ok 2024-09-21T14:04:42.486 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:42 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T14:04:42.486 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:42 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T14:04:42.486 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:42 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:04:42.486 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:42 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3415198225' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:04:42.486 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:42 smithi049 ceph-mon[40489]: 6.0 scrub starts 2024-09-21T14:04:42.486 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:42 smithi049 ceph-mon[40489]: 6.0 scrub ok 2024-09-21T14:04:42.486 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:42 smithi049 ceph-mon[40489]: 5.11 scrub starts 2024-09-21T14:04:42.486 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:42 smithi049 ceph-mon[40489]: 5.11 scrub ok 2024-09-21T14:04:42.487 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:42 smithi049 ceph-mon[40489]: 4.1e scrub starts 2024-09-21T14:04:42.487 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:42 smithi049 ceph-mon[40489]: 4.1e scrub ok 2024-09-21T14:04:42.487 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:42 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T14:04:42.487 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:42 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T14:04:42.487 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:42 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:04:42.487 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:42 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3415198225' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:04:42.487 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:04:42.487 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:04:42.525 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:04:42.525 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:04:42.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:42.532 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:42.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:04:42.556 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:04:42.556 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:04:42.593 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:04:42.593 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:04:42.633 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:04:42.634 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:04:42.635 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:04:42.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:42 smithi187 ceph-mon[33032]: 6.0 scrub starts 2024-09-21T14:04:42.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:42 smithi187 ceph-mon[33032]: 6.0 scrub ok 2024-09-21T14:04:42.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:42 smithi187 ceph-mon[33032]: 5.11 scrub starts 2024-09-21T14:04:42.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:42 smithi187 ceph-mon[33032]: 5.11 scrub ok 2024-09-21T14:04:42.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:42 smithi187 ceph-mon[33032]: 4.1e scrub starts 2024-09-21T14:04:42.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:42 smithi187 ceph-mon[33032]: 4.1e scrub ok 2024-09-21T14:04:42.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:42 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T14:04:42.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:42 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T14:04:42.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:42 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:04:42.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:42 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3415198225' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:04:42.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:42.696 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:42.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:04:42.797 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:04:42.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:42.832 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:42.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:04:42.888 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:04:42.933 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:04:42.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:42.976 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:42.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:04:43.077 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:04:43.109 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:43.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:43.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:04:43.210 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:43.244 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:43.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:43.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:43.345 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:04:43.387 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:43.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:43.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:04:43.453 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-09-21T14:04:43.490 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:04:43.525 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:43.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:43.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:04:43.626 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:04:43.653 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:43.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:43.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:04:43.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:43 smithi187 ceph-mon[33032]: pgmap v799: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 153 active+clean; 7.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.1 KiB/s rd, 49 MiB/s wr, 1.14k op/s; 28662/373062 objects misplaced (7.683%); 8.7 MiB/s, 138 objects/s recovering 2024-09-21T14:04:43.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:43 smithi187 ceph-mon[33032]: 3.2 scrub starts 2024-09-21T14:04:43.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:43 smithi187 ceph-mon[33032]: 3.2 scrub ok 2024-09-21T14:04:43.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:43 smithi187 ceph-mon[33032]: 6.a scrub starts 2024-09-21T14:04:43.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:43 smithi187 ceph-mon[33032]: 6.a scrub ok 2024-09-21T14:04:43.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:43 smithi187 ceph-mon[33032]: 10.f scrub ok 2024-09-21T14:04:43.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:43 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2570110754' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:04:43.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:43 smithi049 ceph-mon[40489]: pgmap v799: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 153 active+clean; 7.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.1 KiB/s rd, 49 MiB/s wr, 1.14k op/s; 28662/373062 objects misplaced (7.683%); 8.7 MiB/s, 138 objects/s recovering 2024-09-21T14:04:43.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:43 smithi049 ceph-mon[40489]: 3.2 scrub starts 2024-09-21T14:04:43.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:43 smithi049 ceph-mon[40489]: 3.2 scrub ok 2024-09-21T14:04:43.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:43 smithi049 ceph-mon[40489]: 6.a scrub starts 2024-09-21T14:04:43.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:43 smithi049 ceph-mon[40489]: 6.a scrub ok 2024-09-21T14:04:43.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:43 smithi049 ceph-mon[40489]: 10.f scrub ok 2024-09-21T14:04:43.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:43 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2570110754' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:04:43.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:43 smithi049 ceph-mon[30080]: pgmap v799: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 153 active+clean; 7.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.1 KiB/s rd, 49 MiB/s wr, 1.14k op/s; 28662/373062 objects misplaced (7.683%); 8.7 MiB/s, 138 objects/s recovering 2024-09-21T14:04:43.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:43 smithi049 ceph-mon[30080]: 3.2 scrub starts 2024-09-21T14:04:43.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:43 smithi049 ceph-mon[30080]: 3.2 scrub ok 2024-09-21T14:04:43.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:43 smithi049 ceph-mon[30080]: 6.a scrub starts 2024-09-21T14:04:43.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:43 smithi049 ceph-mon[30080]: 6.a scrub ok 2024-09-21T14:04:43.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:43 smithi049 ceph-mon[30080]: 10.f scrub ok 2024-09-21T14:04:43.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:43 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2570110754' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:04:43.756 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:04:43.801 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:43.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:43.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:04:43.903 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:04:43.939 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:43.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:43.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:04:44.042 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:04:44.073 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:44.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:44.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:04:44.175 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:04:44.216 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:44.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:44.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:04:44.318 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:04:44.352 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:44.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:44.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:04:44.454 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:04:44.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:44.484 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:44.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:04:44.585 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:04:44.620 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:44.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:44.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:04:44.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:44 smithi187 ceph-mon[33032]: 4.2 scrub starts 2024-09-21T14:04:44.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:44 smithi187 ceph-mon[33032]: 4.2 scrub ok 2024-09-21T14:04:44.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:44 smithi187 ceph-mon[33032]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T14:04:44.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:44 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2570110754' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T14:04:44.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:44 smithi187 ceph-mon[33032]: osdmap e414: 8 total, 8 up, 8 in 2024-09-21T14:04:44.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:44 smithi187 ceph-mon[33032]: 5.3 scrub starts 2024-09-21T14:04:44.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:44 smithi187 ceph-mon[33032]: 5.3 scrub ok 2024-09-21T14:04:44.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:44 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:04:44.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:44 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:04:44.722 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:04:44.734 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:44 smithi049 ceph-mon[40489]: 4.2 scrub starts 2024-09-21T14:04:44.734 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:44 smithi049 ceph-mon[40489]: 4.2 scrub ok 2024-09-21T14:04:44.734 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:44 smithi049 ceph-mon[40489]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T14:04:44.734 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:44 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2570110754' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T14:04:44.734 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:44 smithi049 ceph-mon[40489]: osdmap e414: 8 total, 8 up, 8 in 2024-09-21T14:04:44.734 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:44 smithi049 ceph-mon[40489]: 5.3 scrub starts 2024-09-21T14:04:44.734 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:44 smithi049 ceph-mon[40489]: 5.3 scrub ok 2024-09-21T14:04:44.735 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:44 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:04:44.735 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:44 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:04:44.735 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:44 smithi049 ceph-mon[30080]: 4.2 scrub starts 2024-09-21T14:04:44.735 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:44 smithi049 ceph-mon[30080]: 4.2 scrub ok 2024-09-21T14:04:44.735 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:44 smithi049 ceph-mon[30080]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T14:04:44.735 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:44 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2570110754' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T14:04:44.735 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:44 smithi049 ceph-mon[30080]: osdmap e414: 8 total, 8 up, 8 in 2024-09-21T14:04:44.735 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:44 smithi049 ceph-mon[30080]: 5.3 scrub starts 2024-09-21T14:04:44.735 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:44 smithi049 ceph-mon[30080]: 5.3 scrub ok 2024-09-21T14:04:44.735 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:44 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:04:44.736 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:44 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:04:44.752 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:44.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:44.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:04:44.853 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:04:44.884 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:44.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:44.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:04:44.986 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:04:45.017 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:45.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:45.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:04:45.119 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:04:45.151 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:45.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:45.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:04:45.220 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T14:04:45.220 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:04:45.253 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:04:45.282 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:45.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:45.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:04:45.384 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:04:45.423 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:45.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:45.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:04:45.472 INFO:teuthology.orchestra.run.smithi049.stderr:set pool 10 pgp_num to 16 2024-09-21T14:04:45.524 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:45.561 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:45.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:45.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:45.663 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:04:45.706 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:45.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:45.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:04:45.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:45 smithi049 ceph-mon[40489]: pgmap v801: 158 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 154 active+clean; 7.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.4 KiB/s rd, 18 MiB/s wr, 1.19k op/s; 26700/361878 objects misplaced (7.378%); 11 MiB/s, 182 objects/s recovering 2024-09-21T14:04:45.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:45 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3479180581' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"} : dispatch 2024-09-21T14:04:45.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:45 smithi049 ceph-mon[40489]: 1.6 scrub starts 2024-09-21T14:04:45.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:45 smithi049 ceph-mon[40489]: 1.6 scrub ok 2024-09-21T14:04:45.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:45 smithi049 ceph-mon[40489]: 6.3 scrub starts 2024-09-21T14:04:45.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:45 smithi049 ceph-mon[40489]: 6.3 scrub ok 2024-09-21T14:04:45.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:45 smithi049 ceph-mon[30080]: pgmap v801: 158 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 154 active+clean; 7.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.4 KiB/s rd, 18 MiB/s wr, 1.19k op/s; 26700/361878 objects misplaced (7.378%); 11 MiB/s, 182 objects/s recovering 2024-09-21T14:04:45.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:45 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3479180581' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"} : dispatch 2024-09-21T14:04:45.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:45 smithi049 ceph-mon[30080]: 1.6 scrub starts 2024-09-21T14:04:45.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:45 smithi049 ceph-mon[30080]: 1.6 scrub ok 2024-09-21T14:04:45.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:45 smithi049 ceph-mon[30080]: 6.3 scrub starts 2024-09-21T14:04:45.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:45 smithi049 ceph-mon[30080]: 6.3 scrub ok 2024-09-21T14:04:45.809 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:04:45.841 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:45.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:45.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:04:45.942 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:04:45.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:45 smithi187 ceph-mon[33032]: pgmap v801: 158 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 154 active+clean; 7.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.4 KiB/s rd, 18 MiB/s wr, 1.19k op/s; 26700/361878 objects misplaced (7.378%); 11 MiB/s, 182 objects/s recovering 2024-09-21T14:04:45.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:45 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3479180581' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"} : dispatch 2024-09-21T14:04:45.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:45 smithi187 ceph-mon[33032]: 1.6 scrub starts 2024-09-21T14:04:45.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:45 smithi187 ceph-mon[33032]: 1.6 scrub ok 2024-09-21T14:04:45.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:45 smithi187 ceph-mon[33032]: 6.3 scrub starts 2024-09-21T14:04:45.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:45 smithi187 ceph-mon[33032]: 6.3 scrub ok 2024-09-21T14:04:45.981 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:45.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:45.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:04:46.083 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:46.104 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T14:04:46.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:46.123 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:46.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:46.164 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T14:04:46.224 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:04:46.264 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:46.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:46.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:04:46.366 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:46.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:46.422 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:46.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:46.523 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:04:46.566 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:46.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:46.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:04:46.668 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:04:46.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:46.711 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:46.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:04:46.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:46 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3479180581' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]': finished 2024-09-21T14:04:46.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:46 smithi049 ceph-mon[40489]: osdmap e415: 8 total, 8 up, 8 in 2024-09-21T14:04:46.757 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:46 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3479180581' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]': finished 2024-09-21T14:04:46.758 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:46 smithi049 ceph-mon[30080]: osdmap e415: 8 total, 8 up, 8 in 2024-09-21T14:04:46.813 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:04:46.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:46.870 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:46.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:04:46.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:46 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3479180581' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]': finished 2024-09-21T14:04:46.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:46 smithi187 ceph-mon[33032]: osdmap e415: 8 total, 8 up, 8 in 2024-09-21T14:04:46.972 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:47.027 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:47.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:47.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:47.129 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:04:47.170 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:47.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:47.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:04:47.274 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:04:47.308 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:47.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:47.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:04:47.409 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:04:47.442 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:47.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:47.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:04:47.447 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:04:47.458 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:04:47.544 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:04:47.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:47.586 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:47.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:04:47.609 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:47 smithi049 ceph-mon[30080]: 10.5 scrub starts 2024-09-21T14:04:47.609 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:47 smithi049 ceph-mon[30080]: pgmap v803: 158 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 154 active+clean; 7.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 767 B/s rd, 11 MiB/s wr, 1.14k op/s; 26700/358830 objects misplaced (7.441%); 11 MiB/s, 182 objects/s recovering 2024-09-21T14:04:47.610 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:47 smithi049 ceph-mon[30080]: 6.19 scrub starts 2024-09-21T14:04:47.610 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:47 smithi049 ceph-mon[30080]: 6.19 scrub ok 2024-09-21T14:04:47.610 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:47 smithi049 ceph-mon[40489]: 10.5 scrub starts 2024-09-21T14:04:47.610 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:47 smithi049 ceph-mon[40489]: pgmap v803: 158 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 154 active+clean; 7.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 767 B/s rd, 11 MiB/s wr, 1.14k op/s; 26700/358830 objects misplaced (7.441%); 11 MiB/s, 182 objects/s recovering 2024-09-21T14:04:47.610 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:47 smithi049 ceph-mon[40489]: 6.19 scrub starts 2024-09-21T14:04:47.610 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:47 smithi049 ceph-mon[40489]: 6.19 scrub ok 2024-09-21T14:04:47.687 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:04:47.728 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:47.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:47.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:04:47.829 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:04:47.869 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:47.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:47.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:04:47.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:47 smithi187 ceph-mon[33032]: 10.5 scrub starts 2024-09-21T14:04:47.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:47 smithi187 ceph-mon[33032]: pgmap v803: 158 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 154 active+clean; 7.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 767 B/s rd, 11 MiB/s wr, 1.14k op/s; 26700/358830 objects misplaced (7.441%); 11 MiB/s, 182 objects/s recovering 2024-09-21T14:04:47.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:47 smithi187 ceph-mon[33032]: 6.19 scrub starts 2024-09-21T14:04:47.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:47 smithi187 ceph-mon[33032]: 6.19 scrub ok 2024-09-21T14:04:47.971 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:04:48.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:48.011 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:48.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:04:48.112 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:48.154 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:48.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:48.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:48.256 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:04:48.290 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:48.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:48.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:04:48.392 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:04:48.430 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:04:48.430 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:04:48.431 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:48.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:48.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:04:48.467 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:04:48.468 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:04:48.508 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:04:48.509 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:04:48.534 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:04:48.557 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:04:48.557 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:04:48.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:48.561 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:48.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:04:48.601 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:04:48.601 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:04:48.634 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T14:04:48.634 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-09-21T14:04:48.634 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-09-21T14:04:48.634 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-09-21T14:04:48.634 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T14:04:48.649 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:04:48.649 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:04:48.661 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:04:48.694 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:04:48.695 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:04:48.704 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:48.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:48.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:04:48.729 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:04:48.729 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:04:48.807 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:48.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:48.845 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:48.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:48.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:48 smithi187 ceph-mon[33032]: 6.d scrub starts 2024-09-21T14:04:48.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:48 smithi187 ceph-mon[33032]: 6.d scrub ok 2024-09-21T14:04:48.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:48 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2481976905' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:04:48.946 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:48.981 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:48.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:48.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:49.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:48 smithi049 ceph-mon[40489]: 6.d scrub starts 2024-09-21T14:04:49.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:48 smithi049 ceph-mon[40489]: 6.d scrub ok 2024-09-21T14:04:49.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:48 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2481976905' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:04:49.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:48 smithi049 ceph-mon[30080]: 6.d scrub starts 2024-09-21T14:04:49.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:48 smithi049 ceph-mon[30080]: 6.d scrub ok 2024-09-21T14:04:49.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:48 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2481976905' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:04:49.083 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:04:49.119 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:49.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:49.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:04:49.221 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:04:49.251 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:49.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:49.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:04:49.353 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:04:49.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:49.383 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:49.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:04:49.415 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T14:04:49.485 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:04:49.527 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:49.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:49.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:04:49.629 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:04:49.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:49.676 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:49.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:04:49.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:49 smithi049 ceph-mon[40489]: pgmap v804: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 153 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 888 op/s; 26700/354675 objects misplaced (7.528%); 6.1 MiB/s, 97 objects/s recovering 2024-09-21T14:04:49.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:49 smithi049 ceph-mon[40489]: 3.5 scrub starts 2024-09-21T14:04:49.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:49 smithi049 ceph-mon[40489]: 3.5 scrub ok 2024-09-21T14:04:49.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:49 smithi049 ceph-mon[30080]: pgmap v804: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 153 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 888 op/s; 26700/354675 objects misplaced (7.528%); 6.1 MiB/s, 97 objects/s recovering 2024-09-21T14:04:49.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:49 smithi049 ceph-mon[30080]: 3.5 scrub starts 2024-09-21T14:04:49.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:49 smithi049 ceph-mon[30080]: 3.5 scrub ok 2024-09-21T14:04:49.777 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:04:49.821 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:49.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:49.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:04:49.902 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:49 smithi187 ceph-mon[33032]: pgmap v804: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 153 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 888 op/s; 26700/354675 objects misplaced (7.528%); 6.1 MiB/s, 97 objects/s recovering 2024-09-21T14:04:49.902 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:49 smithi187 ceph-mon[33032]: 3.5 scrub starts 2024-09-21T14:04:49.902 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:49 smithi187 ceph-mon[33032]: 3.5 scrub ok 2024-09-21T14:04:49.923 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:04:49.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:49.958 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:49.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:04:50.059 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:04:50.079 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:50.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:50.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:04:50.181 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:04:50.194 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:04:49 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:04:50.203 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:50.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:50.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:04:50.305 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:04:50.327 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:50.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:50.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:04:50.428 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:50.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:50.466 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:50.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:50.567 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:04:50.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:50.607 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:50.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:04:50.708 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:04:50.739 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:50.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:50.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:04:50.841 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:04:50.879 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:50.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:50.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:04:50.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:50 smithi187 ceph-mon[33032]: 10.5 scrub ok 2024-09-21T14:04:50.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:50 smithi187 ceph-mon[33032]: 5.b scrub starts 2024-09-21T14:04:50.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:50 smithi187 ceph-mon[33032]: 5.b scrub ok 2024-09-21T14:04:50.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:50 smithi187 ceph-mon[33032]: 1.4 scrub starts 2024-09-21T14:04:50.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:50 smithi187 ceph-mon[33032]: 1.4 scrub ok 2024-09-21T14:04:50.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:50 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:04:50.981 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:04:50.982 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:50 smithi049 ceph-mon[40489]: 10.5 scrub ok 2024-09-21T14:04:50.983 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:50 smithi049 ceph-mon[40489]: 5.b scrub starts 2024-09-21T14:04:50.983 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:50 smithi049 ceph-mon[40489]: 5.b scrub ok 2024-09-21T14:04:50.983 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:50 smithi049 ceph-mon[40489]: 1.4 scrub starts 2024-09-21T14:04:50.983 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:50 smithi049 ceph-mon[40489]: 1.4 scrub ok 2024-09-21T14:04:50.983 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:50 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:04:50.983 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:04:50 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:04:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:04:50.983 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:50 smithi049 ceph-mon[30080]: 10.5 scrub ok 2024-09-21T14:04:50.984 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:50 smithi049 ceph-mon[30080]: 5.b scrub starts 2024-09-21T14:04:50.984 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:50 smithi049 ceph-mon[30080]: 5.b scrub ok 2024-09-21T14:04:50.984 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:50 smithi049 ceph-mon[30080]: 1.4 scrub starts 2024-09-21T14:04:50.984 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:50 smithi049 ceph-mon[30080]: 1.4 scrub ok 2024-09-21T14:04:50.984 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:50 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:04:51.013 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:51.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:51.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:04:51.115 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:04:51.146 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:51.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:51.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:04:51.248 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:51.287 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:51.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:51.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:51.389 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:04:51.432 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:51.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:51.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:04:51.534 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:04:51.576 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:51.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:51.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:04:51.678 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:04:51.713 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:51.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:51.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:04:51.815 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:04:51.850 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:51.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:51.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:04:51.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:51 smithi187 ceph-mon[33032]: 10.3 scrub starts 2024-09-21T14:04:51.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:51 smithi187 ceph-mon[33032]: pgmap v805: 158 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 153 active+clean; 7.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.21k op/s; 24808/344097 objects misplaced (7.210%); 11 MiB/s, 173 objects/s recovering 2024-09-21T14:04:51.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:51 smithi187 ceph-mon[33032]: 4.1d scrub starts 2024-09-21T14:04:51.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:51 smithi187 ceph-mon[33032]: 4.1d scrub ok 2024-09-21T14:04:51.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:51 smithi187 ceph-mon[33032]: 10.6 scrub starts 2024-09-21T14:04:51.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:51 smithi187 ceph-mon[33032]: 5.1e scrub starts 2024-09-21T14:04:51.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:51 smithi187 ceph-mon[33032]: 3.a scrub starts 2024-09-21T14:04:51.952 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:04:51.978 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:51.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:51.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:04:51.992 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 4, 6, 2, 3, 7, 5] 2024-09-21T14:04:51.992 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T14:04:51.992 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-21T14:04:51.993 INFO:tasks.thrashosds.thrasher:Setting osd 0 primary_affinity to 1.000000 2024-09-21T14:04:51.993 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 0 1 2024-09-21T14:04:52.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:51 smithi049 ceph-mon[40489]: 10.3 scrub starts 2024-09-21T14:04:52.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:51 smithi049 ceph-mon[40489]: pgmap v805: 158 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 153 active+clean; 7.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.21k op/s; 24808/344097 objects misplaced (7.210%); 11 MiB/s, 173 objects/s recovering 2024-09-21T14:04:52.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:51 smithi049 ceph-mon[40489]: 4.1d scrub starts 2024-09-21T14:04:52.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:51 smithi049 ceph-mon[40489]: 4.1d scrub ok 2024-09-21T14:04:52.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:51 smithi049 ceph-mon[40489]: 10.6 scrub starts 2024-09-21T14:04:52.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:51 smithi049 ceph-mon[40489]: 5.1e scrub starts 2024-09-21T14:04:52.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:51 smithi049 ceph-mon[40489]: 3.a scrub starts 2024-09-21T14:04:52.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:51 smithi049 ceph-mon[30080]: 10.3 scrub starts 2024-09-21T14:04:52.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:51 smithi049 ceph-mon[30080]: pgmap v805: 158 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 153 active+clean; 7.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.21k op/s; 24808/344097 objects misplaced (7.210%); 11 MiB/s, 173 objects/s recovering 2024-09-21T14:04:52.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:51 smithi049 ceph-mon[30080]: 4.1d scrub starts 2024-09-21T14:04:52.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:51 smithi049 ceph-mon[30080]: 4.1d scrub ok 2024-09-21T14:04:52.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:51 smithi049 ceph-mon[30080]: 10.6 scrub starts 2024-09-21T14:04:52.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:51 smithi049 ceph-mon[30080]: 5.1e scrub starts 2024-09-21T14:04:52.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:51 smithi049 ceph-mon[30080]: 3.a scrub starts 2024-09-21T14:04:52.079 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:04:52.166 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:52.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:52.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:04:52.268 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:04:52.288 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:52.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:52.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:04:52.390 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:04:52.421 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:52.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:52.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:04:52.523 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:52.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:52.565 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:52.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:52.666 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:04:52.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:52.701 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:52.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:04:52.802 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:52.838 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:52.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:52.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:52.940 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:04:52.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:52 smithi187 ceph-mon[33032]: 5.1e scrub ok 2024-09-21T14:04:52.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:52 smithi187 ceph-mon[33032]: 3.a scrub ok 2024-09-21T14:04:52.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:52 smithi187 ceph-mon[33032]: 1.8 scrub starts 2024-09-21T14:04:52.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:52 smithi187 ceph-mon[33032]: 1.8 scrub ok 2024-09-21T14:04:52.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:52 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:04:52.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:52 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:04:52.989 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:52.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:52.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:04:53.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:52 smithi049 ceph-mon[40489]: 5.1e scrub ok 2024-09-21T14:04:53.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:52 smithi049 ceph-mon[40489]: 3.a scrub ok 2024-09-21T14:04:53.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:52 smithi049 ceph-mon[40489]: 1.8 scrub starts 2024-09-21T14:04:53.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:52 smithi049 ceph-mon[40489]: 1.8 scrub ok 2024-09-21T14:04:53.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:52 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:04:53.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:52 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:04:53.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:52 smithi049 ceph-mon[30080]: 5.1e scrub ok 2024-09-21T14:04:53.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:52 smithi049 ceph-mon[30080]: 3.a scrub ok 2024-09-21T14:04:53.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:52 smithi049 ceph-mon[30080]: 1.8 scrub starts 2024-09-21T14:04:53.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:52 smithi049 ceph-mon[30080]: 1.8 scrub ok 2024-09-21T14:04:53.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:52 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:04:53.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:52 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:04:53.090 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:04:53.123 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:53.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:53.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:04:53.225 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:04:53.255 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:53.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:53.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:04:53.357 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:04:53.392 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:53.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:53.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:04:53.494 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:04:53.531 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:53.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:53.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:04:53.633 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:04:53.668 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:53.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:53.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:04:53.770 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:04:53.771 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:04:53.782 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T14:04:53.803 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:53.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:53.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:04:53.843 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:04:53.849 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:53 smithi049 ceph-mon[40489]: pgmap v806: 158 pgs: 1 active+remapped+backfilling, 2 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 152 active+clean; 6.9 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s; 24808/341295 objects misplaced (7.269%); 10 MiB/s, 161 objects/s recovering 2024-09-21T14:04:53.849 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:53 smithi049 ceph-mon[40489]: 10.6 scrub ok 2024-09-21T14:04:53.850 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:53 smithi049 ceph-mon[40489]: 10.3 scrub ok 2024-09-21T14:04:53.850 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:53 smithi049 ceph-mon[40489]: osdmap e416: 8 total, 8 up, 8 in 2024-09-21T14:04:53.850 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:53 smithi049 ceph-mon[30080]: pgmap v806: 158 pgs: 1 active+remapped+backfilling, 2 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 152 active+clean; 6.9 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s; 24808/341295 objects misplaced (7.269%); 10 MiB/s, 161 objects/s recovering 2024-09-21T14:04:53.850 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:53 smithi049 ceph-mon[30080]: 10.6 scrub ok 2024-09-21T14:04:53.850 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:53 smithi049 ceph-mon[30080]: 10.3 scrub ok 2024-09-21T14:04:53.850 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:53 smithi049 ceph-mon[30080]: osdmap e416: 8 total, 8 up, 8 in 2024-09-21T14:04:53.905 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:53.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:53 smithi187 ceph-mon[33032]: pgmap v806: 158 pgs: 1 active+remapped+backfilling, 2 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 152 active+clean; 6.9 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s; 24808/341295 objects misplaced (7.269%); 10 MiB/s, 161 objects/s recovering 2024-09-21T14:04:53.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:53 smithi187 ceph-mon[33032]: 10.6 scrub ok 2024-09-21T14:04:53.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:53 smithi187 ceph-mon[33032]: 10.3 scrub ok 2024-09-21T14:04:53.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:53 smithi187 ceph-mon[33032]: osdmap e416: 8 total, 8 up, 8 in 2024-09-21T14:04:53.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:53.945 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:53.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:54.046 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:04:54.095 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:54.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:54.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:04:54.197 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:04:54.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:54.241 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:54.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:04:54.342 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:04:54.379 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:54.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:54.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:04:54.480 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:54.520 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:54.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:54.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:54.621 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:04:54.651 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:54.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:54.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:04:54.753 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:04:54.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:54 smithi049 ceph-mon[30080]: 4.3 scrub starts 2024-09-21T14:04:54.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:54 smithi049 ceph-mon[30080]: 4.3 scrub ok 2024-09-21T14:04:54.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:54 smithi049 ceph-mon[40489]: 4.3 scrub starts 2024-09-21T14:04:54.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:54 smithi049 ceph-mon[40489]: 4.3 scrub ok 2024-09-21T14:04:54.780 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:54.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:54.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:04:54.882 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:54.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:54.932 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:54.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:54.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:54 smithi187 ceph-mon[33032]: 4.3 scrub starts 2024-09-21T14:04:54.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:54 smithi187 ceph-mon[33032]: 4.3 scrub ok 2024-09-21T14:04:55.023 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:04:55.033 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:04:55.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:55.090 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:55.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:04:55.191 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:04:55.226 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:55.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:55.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:04:55.235 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T14:04:55.327 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:55.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:55.368 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:55.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:55.470 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:04:55.519 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:55.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:55.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:04:55.567 INFO:teuthology.orchestra.run.smithi049.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-21T14:04:55.571 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-09-21T14:04:55.620 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:04:55.657 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:55.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:55.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:04:55.759 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:04:55.788 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:55.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:55.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:04:55.890 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:04:55.913 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:55.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:55.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:04:55.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:55 smithi187 ceph-mon[33032]: pgmap v808: 158 pgs: 4 active+remapped+backfilling, 1 active+clean+scrubbing, 153 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 1.20k op/s; 22475/329943 objects misplaced (6.812%); 13 MiB/s, 211 objects/s recovering 2024-09-21T14:04:55.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:55 smithi187 ceph-mon[33032]: 5.f scrub starts 2024-09-21T14:04:55.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:55 smithi187 ceph-mon[33032]: 5.f scrub ok 2024-09-21T14:04:55.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:55 smithi187 ceph-mon[33032]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull) 2024-09-21T14:04:55.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:55 smithi187 ceph-mon[33032]: Cluster is now healthy 2024-09-21T14:04:55.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:55 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1241837807' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-21T14:04:55.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:55 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-21T14:04:55.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:55 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2973038888' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:04:55.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:55 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:04:55.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:55 smithi187 ceph-mon[33032]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:04:56.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:55 smithi049 ceph-mon[40489]: pgmap v808: 158 pgs: 4 active+remapped+backfilling, 1 active+clean+scrubbing, 153 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 1.20k op/s; 22475/329943 objects misplaced (6.812%); 13 MiB/s, 211 objects/s recovering 2024-09-21T14:04:56.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:55 smithi049 ceph-mon[40489]: 5.f scrub starts 2024-09-21T14:04:56.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:55 smithi049 ceph-mon[40489]: 5.f scrub ok 2024-09-21T14:04:56.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:55 smithi049 ceph-mon[40489]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull) 2024-09-21T14:04:56.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:55 smithi049 ceph-mon[40489]: Cluster is now healthy 2024-09-21T14:04:56.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:55 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1241837807' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-21T14:04:56.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:55 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-21T14:04:56.007 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:55 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2973038888' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:04:56.007 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:55 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:04:56.007 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:55 smithi049 ceph-mon[40489]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:04:56.007 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:55 smithi049 ceph-mon[30080]: pgmap v808: 158 pgs: 4 active+remapped+backfilling, 1 active+clean+scrubbing, 153 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 1.20k op/s; 22475/329943 objects misplaced (6.812%); 13 MiB/s, 211 objects/s recovering 2024-09-21T14:04:56.007 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:55 smithi049 ceph-mon[30080]: 5.f scrub starts 2024-09-21T14:04:56.007 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:55 smithi049 ceph-mon[30080]: 5.f scrub ok 2024-09-21T14:04:56.007 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:55 smithi049 ceph-mon[30080]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull) 2024-09-21T14:04:56.007 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:55 smithi049 ceph-mon[30080]: Cluster is now healthy 2024-09-21T14:04:56.007 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:55 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1241837807' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-21T14:04:56.007 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:55 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-21T14:04:56.007 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:55 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2973038888' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:04:56.007 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:55 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:04:56.008 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:55 smithi049 ceph-mon[30080]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:04:56.016 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:04:56.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:56.043 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:56.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:04:56.144 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:56.169 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:56.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:56.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:56.271 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:04:56.296 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:56.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:56.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:04:56.398 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:04:56.424 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:56.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:56.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:04:56.527 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:04:56.554 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:56.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:56.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:04:56.656 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:04:56.693 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:56.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:56.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:04:56.724 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:56 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-21T14:04:56.724 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:56 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T14:04:56.724 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:56 smithi049 ceph-mon[40489]: osdmap e417: 8 total, 8 up, 8 in 2024-09-21T14:04:56.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:56 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-21T14:04:56.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:56 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T14:04:56.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:56 smithi049 ceph-mon[30080]: osdmap e417: 8 total, 8 up, 8 in 2024-09-21T14:04:56.795 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:04:56.833 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:56.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:56.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:04:56.935 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:04:56.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:56 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-21T14:04:56.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:56 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T14:04:56.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:56 smithi187 ceph-mon[33032]: osdmap e417: 8 total, 8 up, 8 in 2024-09-21T14:04:56.979 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:56.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:56.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:04:57.080 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:04:57.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:57.119 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:57.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:04:57.220 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:04:57.259 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:57.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:57.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:04:57.360 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:04:57.400 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:57.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:57.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:04:57.502 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:04:57.549 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:57.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:57.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:04:57.652 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:04:57.699 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:57.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:57.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:04:57.801 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:04:57.816 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:57 smithi049 ceph-mon[30080]: pgmap v810: 158 pgs: 4 active+remapped+backfilling, 154 active+clean; 6.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s; 22475/327048 objects misplaced (6.872%); 14 MiB/s, 226 objects/s recovering 2024-09-21T14:04:57.816 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:57 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:04:57.816 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:57 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:04:57.817 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:57 smithi049 ceph-mon[30080]: osdmap e418: 8 total, 8 up, 8 in 2024-09-21T14:04:57.817 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T14:04:57.817 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-09-21T14:04:57.817 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-09-21T14:04:57.817 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-09-21T14:04:57.817 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.817 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24365.0:1780 4.4 4:22d26bf9:::data_loggenerations_metadata:head [watch reconnect cookie 94025139211520 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-09-21T14:04:57.817 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:55:40.878748+0000", 2024-09-21T14:04:57.817 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 556.93447930699995, 2024-09-21T14:04:57.818 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.39793440499999999, 2024-09-21T14:04:57.818 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:57.818 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:57.818 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T14:04:57.818 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:04:57.818 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24365", 2024-09-21T14:04:57.818 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3404764347", 2024-09-21T14:04:57.818 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 1780 2024-09-21T14:04:57.818 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.818 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:57.818 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.818 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:57.819 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:40.878748+0000", 2024-09-21T14:04:57.819 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:57.819 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.819 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.819 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:57.819 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:40.878748+0000", 2024-09-21T14:04:57.819 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:57.819 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.819 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.819 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:57.819 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:40.878750+0000", 2024-09-21T14:04:57.819 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.962e-06 2024-09-21T14:04:57.820 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.820 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.820 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:57.820 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:40.878756+0000", 2024-09-21T14:04:57.820 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3080000000000004e-06 2024-09-21T14:04:57.820 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.820 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.820 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:57.820 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:40.878768+0000", 2024-09-21T14:04:57.820 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.203e-05 2024-09-21T14:04:57.820 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.820 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.820 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:57.820 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:40.878776+0000", 2024-09-21T14:04:57.821 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4849999999999992e-06 2024-09-21T14:04:57.821 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.821 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.821 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:57.821 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:40.878849+0000", 2024-09-21T14:04:57.821 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2670999999999996e-05 2024-09-21T14:04:57.821 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.821 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.821 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:04:57.821 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:40.878859+0000", 2024-09-21T14:04:57.821 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6099999999999995e-06 2024-09-21T14:04:57.822 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.822 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.822 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:57.822 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:41.271074+0000", 2024-09-21T14:04:57.822 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.39221556499999999 2024-09-21T14:04:57.822 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.822 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.822 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:04:57.822 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:41.271224+0000", 2024-09-21T14:04:57.822 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00015018400000000001 2024-09-21T14:04:57.822 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.822 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.823 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:57.823 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:41.276682+0000", 2024-09-21T14:04:57.823 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00545759 2024-09-21T14:04:57.823 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:57.823 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:57.823 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:57.823 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.823 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.823 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24365.0:1781 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 94025139203456 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-09-21T14:04:57.824 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:55:40.878794+0000", 2024-09-21T14:04:57.824 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 556.934432851, 2024-09-21T14:04:57.824 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.39819017699999998, 2024-09-21T14:04:57.824 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:57.824 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:57.824 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T14:04:57.824 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:04:57.824 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24365", 2024-09-21T14:04:57.824 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3404764347", 2024-09-21T14:04:57.825 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 1781 2024-09-21T14:04:57.825 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.825 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:57.825 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.825 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:57.825 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:40.878794+0000", 2024-09-21T14:04:57.825 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:57.825 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.825 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.825 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:57.825 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:40.878794+0000", 2024-09-21T14:04:57.825 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:57.826 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.826 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.826 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:57.826 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:40.878795+0000", 2024-09-21T14:04:57.826 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9299999999999998e-07 2024-09-21T14:04:57.826 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.826 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.826 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:57.826 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:40.878798+0000", 2024-09-21T14:04:57.827 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.0120000000000001e-06 2024-09-21T14:04:57.827 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.827 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.827 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:57.827 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:40.878805+0000", 2024-09-21T14:04:57.827 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6830000000000008e-06 2024-09-21T14:04:57.827 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.827 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.827 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:57.827 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:40.878811+0000", 2024-09-21T14:04:57.827 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9009999999999999e-06 2024-09-21T14:04:57.827 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.827 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.828 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:57.828 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:40.878873+0000", 2024-09-21T14:04:57.828 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1412000000000001e-05 2024-09-21T14:04:57.828 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.828 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.828 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:04:57.828 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:40.878882+0000", 2024-09-21T14:04:57.828 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2289999999999992e-06 2024-09-21T14:04:57.828 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.828 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.828 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:57.829 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:41.274347+0000", 2024-09-21T14:04:57.829 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.39546476200000003 2024-09-21T14:04:57.829 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.829 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.829 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:04:57.829 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:41.274501+0000", 2024-09-21T14:04:57.829 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00015485499999999999 2024-09-21T14:04:57.829 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.829 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.829 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:57.829 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:55:41.276984+0000", 2024-09-21T14:04:57.829 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0024828300000000001 2024-09-21T14:04:57.829 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:57.830 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:57.830 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:57.830 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.830 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.830 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232045 8.7 8:e258b742:::benchmark_data_smithi049_109311_object105526:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T14:04:57.830 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.681733+0000", 2024-09-21T14:04:57.830 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 466.13149434799999, 2024-09-21T14:04:57.830 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0897055980000001, 2024-09-21T14:04:57.830 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:57.830 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:57.830 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:57.830 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:04:57.831 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T14:04:57.831 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:04:57.831 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232045 2024-09-21T14:04:57.831 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.831 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:57.831 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.831 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:57.831 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.681733+0000", 2024-09-21T14:04:57.831 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:57.831 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.831 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.831 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:57.832 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.681733+0000", 2024-09-21T14:04:57.832 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:57.832 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.832 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.832 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:57.832 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.681734+0000", 2024-09-21T14:04:57.832 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6020000000000001e-06 2024-09-21T14:04:57.832 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.832 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.832 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:57.832 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.681740+0000", 2024-09-21T14:04:57.832 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.592e-06 2024-09-21T14:04:57.833 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.833 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.833 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:57.833 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.681751+0000", 2024-09-21T14:04:57.833 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1640999999999999e-05 2024-09-21T14:04:57.833 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.833 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.833 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:57.833 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.681759+0000", 2024-09-21T14:04:57.833 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5399999999999998e-06 2024-09-21T14:04:57.833 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.833 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.833 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:57.833 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.686829+0000", 2024-09-21T14:04:57.834 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0050702300000000002 2024-09-21T14:04:57.834 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.834 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.834 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:04:57.834 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.686838+0000", 2024-09-21T14:04:57.834 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.884e-06 2024-09-21T14:04:57.834 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.834 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.834 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:57.834 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.767863+0000", 2024-09-21T14:04:57.835 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.081025358 2024-09-21T14:04:57.835 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.835 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.835 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:04:57.835 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.768022+0000", 2024-09-21T14:04:57.835 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00015892599999999999 2024-09-21T14:04:57.835 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.835 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.835 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,3,6", 2024-09-21T14:04:57.835 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.768190+0000", 2024-09-21T14:04:57.835 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016811899999999999 2024-09-21T14:04:57.835 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.836 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.836 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:04:57.836 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.770833+0000", 2024-09-21T14:04:57.836 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0026427410000000001 2024-09-21T14:04:57.836 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.836 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.836 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:57.836 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.771163+0000", 2024-09-21T14:04:57.836 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000329618 2024-09-21T14:04:57.836 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.836 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.836 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:57.837 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.771221+0000", 2024-09-21T14:04:57.837 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8322000000000002e-05 2024-09-21T14:04:57.837 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.837 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.837 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:57.837 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.771281+0000", 2024-09-21T14:04:57.837 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0053999999999998e-05 2024-09-21T14:04:57.837 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.837 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.837 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:04:57.837 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.771351+0000", 2024-09-21T14:04:57.837 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9828999999999998e-05 2024-09-21T14:04:57.838 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.838 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.838 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:57.838 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.771438+0000", 2024-09-21T14:04:57.838 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.7142e-05 2024-09-21T14:04:57.838 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:57.838 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:57.838 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:57.838 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.838 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.838 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232047 8.7 8:e0ec053c:::benchmark_data_smithi049_109311_object105528:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T14:04:57.838 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.686294+0000", 2024-09-21T14:04:57.839 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 466.12693292900002, 2024-09-21T14:04:57.839 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0967370940000001, 2024-09-21T14:04:57.839 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:57.839 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:57.839 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:57.839 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:04:57.839 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T14:04:57.839 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:04:57.839 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232047 2024-09-21T14:04:57.839 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.839 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:57.840 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.840 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:57.840 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.686294+0000", 2024-09-21T14:04:57.840 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:57.840 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.840 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.840 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:57.840 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.686294+0000", 2024-09-21T14:04:57.840 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:57.840 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.840 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.840 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:57.840 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.686295+0000", 2024-09-21T14:04:57.841 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2890000000000001e-06 2024-09-21T14:04:57.841 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.841 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.841 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:57.841 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.686300+0000", 2024-09-21T14:04:57.841 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1900000000000003e-06 2024-09-21T14:04:57.841 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.841 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.841 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:57.841 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.686332+0000", 2024-09-21T14:04:57.841 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1986000000000002e-05 2024-09-21T14:04:57.841 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.842 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.842 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:57.842 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.686341+0000", 2024-09-21T14:04:57.842 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1869999999999997e-06 2024-09-21T14:04:57.842 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.842 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.842 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:57.842 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.686860+0000", 2024-09-21T14:04:57.842 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00051973299999999998 2024-09-21T14:04:57.842 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.842 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.842 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:04:57.843 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.686870+0000", 2024-09-21T14:04:57.843 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2250000000000006e-06 2024-09-21T14:04:57.843 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.843 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.843 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:57.843 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.771686+0000", 2024-09-21T14:04:57.843 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.084816461 2024-09-21T14:04:57.843 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.843 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.843 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:04:57.843 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.771804+0000", 2024-09-21T14:04:57.843 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000117829 2024-09-21T14:04:57.844 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.844 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.844 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,3,6", 2024-09-21T14:04:57.844 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.771895+0000", 2024-09-21T14:04:57.844 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0903999999999996e-05 2024-09-21T14:04:57.844 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.844 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.844 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:57.844 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.777261+0000", 2024-09-21T14:04:57.844 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.005366081 2024-09-21T14:04:57.844 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.844 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.845 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:57.845 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.777313+0000", 2024-09-21T14:04:57.845 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1749000000000002e-05 2024-09-21T14:04:57.845 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.845 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.845 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:57.845 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.777359+0000", 2024-09-21T14:04:57.845 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6221999999999999e-05 2024-09-21T14:04:57.845 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.845 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.845 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:04:57.846 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.782926+0000", 2024-09-21T14:04:57.846 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0055667060000000003 2024-09-21T14:04:57.846 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.846 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.846 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:04:57.846 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.782961+0000", 2024-09-21T14:04:57.846 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5394e-05 2024-09-21T14:04:57.846 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.846 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.846 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:57.846 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.783031+0000", 2024-09-21T14:04:57.846 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0137999999999998e-05 2024-09-21T14:04:57.847 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:57.847 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:57.847 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:57.847 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.847 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.847 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232056 8.7 8:e847aa11:::benchmark_data_smithi049_109311_object105537:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T14:04:57.847 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.884592+0000", 2024-09-21T14:04:57.847 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 465.92863503000001, 2024-09-21T14:04:57.847 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.89863333199999995, 2024-09-21T14:04:57.847 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:57.847 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:57.847 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:57.848 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:04:57.848 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T14:04:57.848 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:04:57.848 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232056 2024-09-21T14:04:57.849 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.849 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:57.849 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.850 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:57.850 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.884592+0000", 2024-09-21T14:04:57.850 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:57.850 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.850 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.850 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:57.850 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.884592+0000", 2024-09-21T14:04:57.850 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:57.850 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.850 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.850 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:57.850 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.884594+0000", 2024-09-21T14:04:57.851 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1619999999999998e-06 2024-09-21T14:04:57.851 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.851 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.851 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:57.851 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:57 smithi049 ceph-mon[40489]: pgmap v810: 158 pgs: 4 active+remapped+backfilling, 154 active+clean; 6.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s; 22475/327048 objects misplaced (6.872%); 14 MiB/s, 226 objects/s recovering 2024-09-21T14:04:57.851 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:57 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:04:57.851 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:57 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:04:57.851 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:57 smithi049 ceph-mon[40489]: osdmap e418: 8 total, 8 up, 8 in 2024-09-21T14:04:57.852 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.884602+0000", 2024-09-21T14:04:57.852 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4649999999999994e-06 2024-09-21T14:04:57.852 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.852 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.852 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:57.852 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.884625+0000", 2024-09-21T14:04:57.852 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2904e-05 2024-09-21T14:04:57.853 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.853 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.853 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:57.853 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.884635+0000", 2024-09-21T14:04:57.853 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8190000000000003e-06 2024-09-21T14:04:57.853 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.853 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.853 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:57.853 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.884711+0000", 2024-09-21T14:04:57.853 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5407000000000005e-05 2024-09-21T14:04:57.853 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.854 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.854 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:04:57.854 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.884721+0000", 2024-09-21T14:04:57.854 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0631999999999999e-05 2024-09-21T14:04:57.854 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.854 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.854 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:57.854 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.774310+0000", 2024-09-21T14:04:57.854 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.88958867699999999 2024-09-21T14:04:57.854 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.854 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.855 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:04:57.855 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.774481+0000", 2024-09-21T14:04:57.855 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00017151699999999999 2024-09-21T14:04:57.855 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.855 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.855 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,3,6", 2024-09-21T14:04:57.855 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.774664+0000", 2024-09-21T14:04:57.855 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00018272000000000001 2024-09-21T14:04:57.855 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.855 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.855 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:57.855 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.782843+0000", 2024-09-21T14:04:57.856 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0081790089999999992 2024-09-21T14:04:57.856 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.856 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.856 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:57.856 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.782904+0000", 2024-09-21T14:04:57.856 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0591999999999999e-05 2024-09-21T14:04:57.856 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.856 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.856 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:04:57.856 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.783040+0000", 2024-09-21T14:04:57.856 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013623300000000001 2024-09-21T14:04:57.857 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.857 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.857 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:57.857 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.783085+0000", 2024-09-21T14:04:57.857 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4948999999999999e-05 2024-09-21T14:04:57.857 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.857 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.857 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:04:57.857 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.783122+0000", 2024-09-21T14:04:57.857 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7268000000000003e-05 2024-09-21T14:04:57.858 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.858 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.858 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:57.858 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.783225+0000", 2024-09-21T14:04:57.858 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000102978 2024-09-21T14:04:57.858 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:57.858 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:57.858 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:57.858 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.858 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.858 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25421.0:172314 9.4 9:2b749f7e:::benchmark_data_smithi049_138504_object50906:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-09-21T14:04:57.859 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:00:33.178595+0000", 2024-09-21T14:04:57.859 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 264.63463174899999, 2024-09-21T14:04:57.859 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.342539818, 2024-09-21T14:04:57.859 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:57.859 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:57.859 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:57.859 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:04:57.859 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25421", 2024-09-21T14:04:57.859 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:04:57.859 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 172314 2024-09-21T14:04:57.859 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.859 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:57.860 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.860 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:57.860 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.178595+0000", 2024-09-21T14:04:57.860 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:57.860 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.860 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.860 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:57.860 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.178595+0000", 2024-09-21T14:04:57.860 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:57.860 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.860 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.860 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:57.861 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.178596+0000", 2024-09-21T14:04:57.861 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5599999999999995e-07 2024-09-21T14:04:57.861 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.861 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.861 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:57.861 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.178599+0000", 2024-09-21T14:04:57.861 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3919999999999999e-06 2024-09-21T14:04:57.861 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.861 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.861 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:57.861 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.178607+0000", 2024-09-21T14:04:57.861 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3780000000000001e-06 2024-09-21T14:04:57.862 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.862 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.862 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:57.862 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.178613+0000", 2024-09-21T14:04:57.862 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8629999999999996e-06 2024-09-21T14:04:57.862 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.862 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.862 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:57.862 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.178862+0000", 2024-09-21T14:04:57.862 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00024909299999999998 2024-09-21T14:04:57.862 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.862 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.863 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:04:57.863 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.178965+0000", 2024-09-21T14:04:57.863 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000102204 2024-09-21T14:04:57.863 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.863 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.863 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,4", 2024-09-21T14:04:57.863 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.179255+0000", 2024-09-21T14:04:57.863 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00029060400000000003 2024-09-21T14:04:57.863 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.863 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.863 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:04:57.863 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.181536+0000", 2024-09-21T14:04:57.864 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0022811630000000001 2024-09-21T14:04:57.864 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.864 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.864 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:57.864 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.187746+0000", 2024-09-21T14:04:57.864 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0062093790000000001 2024-09-21T14:04:57.864 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.864 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.864 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:57.864 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.521042+0000", 2024-09-21T14:04:57.864 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.33329632300000001 2024-09-21T14:04:57.865 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.865 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.865 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:04:57.865 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.521095+0000", 2024-09-21T14:04:57.865 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3365999999999997e-05 2024-09-21T14:04:57.865 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.865 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.865 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:57.865 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.521135+0000", 2024-09-21T14:04:57.865 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9496999999999998e-05 2024-09-21T14:04:57.865 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:57.865 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:57.865 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:57.866 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.866 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.866 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25421.0:172319 9.4 9:21e132db:::benchmark_data_smithi049_138504_object50911:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-09-21T14:04:57.866 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:00:33.184640+0000", 2024-09-21T14:04:57.866 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 264.62858641700001, 2024-09-21T14:04:57.866 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.338985429, 2024-09-21T14:04:57.866 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:57.866 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:57.866 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:57.866 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:04:57.866 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25421", 2024-09-21T14:04:57.867 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:04:57.867 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 172319 2024-09-21T14:04:57.867 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.867 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:57.867 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.867 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:57.867 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.184640+0000", 2024-09-21T14:04:57.867 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:57.867 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.867 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.867 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:57.867 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.184640+0000", 2024-09-21T14:04:57.868 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:57.868 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.868 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.868 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:57.868 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.184642+0000", 2024-09-21T14:04:57.868 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.488e-06 2024-09-21T14:04:57.868 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.868 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.868 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:57.868 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.184647+0000", 2024-09-21T14:04:57.868 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4e-06 2024-09-21T14:04:57.868 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.869 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.869 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:57.869 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.184679+0000", 2024-09-21T14:04:57.869 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1482000000000001e-05 2024-09-21T14:04:57.869 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.869 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.869 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:57.869 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.184688+0000", 2024-09-21T14:04:57.869 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0380000000000005e-06 2024-09-21T14:04:57.869 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.869 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.870 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:57.870 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.184742+0000", 2024-09-21T14:04:57.870 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3888999999999999e-05 2024-09-21T14:04:57.870 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.870 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.870 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:04:57.870 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.184861+0000", 2024-09-21T14:04:57.870 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011920499999999999 2024-09-21T14:04:57.870 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.870 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.870 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,4", 2024-09-21T14:04:57.870 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.185030+0000", 2024-09-21T14:04:57.871 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016910499999999999 2024-09-21T14:04:57.871 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.871 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.871 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:04:57.871 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.187701+0000", 2024-09-21T14:04:57.871 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0026706659999999999 2024-09-21T14:04:57.871 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.871 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.871 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:57.871 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.188554+0000", 2024-09-21T14:04:57.871 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00085298600000000002 2024-09-21T14:04:57.871 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.872 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.872 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:57.872 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.523534+0000", 2024-09-21T14:04:57.872 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.33498041099999998 2024-09-21T14:04:57.872 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.872 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.872 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:04:57.872 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.523587+0000", 2024-09-21T14:04:57.872 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2604e-05 2024-09-21T14:04:57.872 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.872 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.872 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:57.873 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.523626+0000", 2024-09-21T14:04:57.873 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9155e-05 2024-09-21T14:04:57.873 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:57.873 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:57.873 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:57.873 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.873 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.873 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25421.0:172326 9.4 9:22a8fc8a:::benchmark_data_smithi049_138504_object50918:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-09-21T14:04:57.873 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:00:33.254009+0000", 2024-09-21T14:04:57.873 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 264.55921811399998, 2024-09-21T14:04:57.873 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.269760204, 2024-09-21T14:04:57.873 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:57.874 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:57.874 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:57.874 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:04:57.874 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25421", 2024-09-21T14:04:57.874 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:04:57.874 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 172326 2024-09-21T14:04:57.874 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.874 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:57.874 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.874 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:57.874 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254009+0000", 2024-09-21T14:04:57.874 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:57.875 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.875 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.875 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:57.875 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254009+0000", 2024-09-21T14:04:57.875 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:57.875 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.875 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.875 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:57.875 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254010+0000", 2024-09-21T14:04:57.875 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.42e-06 2024-09-21T14:04:57.875 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.875 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.876 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:57.876 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254014+0000", 2024-09-21T14:04:57.876 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9550000000000002e-06 2024-09-21T14:04:57.876 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.876 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.876 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:57.876 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254023+0000", 2024-09-21T14:04:57.876 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3330000000000003e-06 2024-09-21T14:04:57.876 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.876 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.876 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:57.876 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254030+0000", 2024-09-21T14:04:57.877 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5719999999999999e-06 2024-09-21T14:04:57.877 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.877 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.877 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:57.877 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254298+0000", 2024-09-21T14:04:57.877 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00026834 2024-09-21T14:04:57.877 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.877 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.877 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:04:57.877 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254412+0000", 2024-09-21T14:04:57.877 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011395000000000001 2024-09-21T14:04:57.878 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.878 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.878 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,4", 2024-09-21T14:04:57.878 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254540+0000", 2024-09-21T14:04:57.878 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012807199999999999 2024-09-21T14:04:57.878 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.878 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.878 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:57.878 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.259890+0000", 2024-09-21T14:04:57.878 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0053499230000000004 2024-09-21T14:04:57.878 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.878 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.879 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:04:57.879 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.259900+0000", 2024-09-21T14:04:57.879 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8919999999999995e-06 2024-09-21T14:04:57.879 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.879 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.879 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:57.879 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.523677+0000", 2024-09-21T14:04:57.879 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26377665300000003 2024-09-21T14:04:57.879 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.879 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.879 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:04:57.879 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.523732+0000", 2024-09-21T14:04:57.879 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5605e-05 2024-09-21T14:04:57.880 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.880 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.880 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:57.880 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.523769+0000", 2024-09-21T14:04:57.880 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6489000000000002e-05 2024-09-21T14:04:57.880 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:57.880 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:57.880 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:57.880 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.880 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.880 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25421.0:172330 9.4 9:2882ed62:::benchmark_data_smithi049_138504_object50922:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-09-21T14:04:57.880 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:00:33.254698+0000", 2024-09-21T14:04:57.881 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 264.55852859100003, 2024-09-21T14:04:57.881 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26919505599999999, 2024-09-21T14:04:57.881 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:57.881 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:57.881 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:57.881 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:04:57.881 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25421", 2024-09-21T14:04:57.881 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:04:57.881 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 172330 2024-09-21T14:04:57.881 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.881 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:57.881 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.882 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:57.882 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254698+0000", 2024-09-21T14:04:57.882 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:57.882 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.882 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.882 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:57.882 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254698+0000", 2024-09-21T14:04:57.882 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:57.882 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.882 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.882 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:57.883 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254699+0000", 2024-09-21T14:04:57.883 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3799999999999998e-07 2024-09-21T14:04:57.883 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.883 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.883 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:57.883 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254701+0000", 2024-09-21T14:04:57.883 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5409999999999999e-06 2024-09-21T14:04:57.883 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.883 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.883 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:57.883 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254708+0000", 2024-09-21T14:04:57.883 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6089999999999997e-06 2024-09-21T14:04:57.884 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.884 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.884 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:57.884 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254713+0000", 2024-09-21T14:04:57.884 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2449999999999998e-06 2024-09-21T14:04:57.884 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.884 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.884 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:57.884 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.257115+0000", 2024-09-21T14:04:57.884 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0024021030000000001 2024-09-21T14:04:57.884 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.884 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.884 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:04:57.885 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.257220+0000", 2024-09-21T14:04:57.885 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010508 2024-09-21T14:04:57.885 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.885 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.885 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,4", 2024-09-21T14:04:57.885 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.257330+0000", 2024-09-21T14:04:57.885 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000110165 2024-09-21T14:04:57.885 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.885 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.885 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:04:57.885 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.260045+0000", 2024-09-21T14:04:57.885 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0027141890000000001 2024-09-21T14:04:57.886 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.886 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.886 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:57.886 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.274124+0000", 2024-09-21T14:04:57.886 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.014079473 2024-09-21T14:04:57.886 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.886 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.886 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:57.886 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.523815+0000", 2024-09-21T14:04:57.886 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.249690836 2024-09-21T14:04:57.886 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.887 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.887 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:04:57.887 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.523860+0000", 2024-09-21T14:04:57.887 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4935999999999998e-05 2024-09-21T14:04:57.887 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.887 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.887 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:57.887 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.523893+0000", 2024-09-21T14:04:57.887 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3441000000000001e-05 2024-09-21T14:04:57.887 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:57.887 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:57.887 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:57.888 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.888 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.888 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25421.0:172334 9.4 9:24fa2bd3:::benchmark_data_smithi049_138504_object50926:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-09-21T14:04:57.888 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:00:33.260571+0000", 2024-09-21T14:04:57.888 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 264.55265543500002, 2024-09-21T14:04:57.888 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26343834999999999, 2024-09-21T14:04:57.888 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:57.888 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:57.888 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:57.888 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:04:57.888 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25421", 2024-09-21T14:04:57.888 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:04:57.888 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 172334 2024-09-21T14:04:57.889 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.889 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:57.889 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.889 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:57.889 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.260571+0000", 2024-09-21T14:04:57.889 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:57.889 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.889 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.889 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:57.889 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.260571+0000", 2024-09-21T14:04:57.889 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:57.890 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.890 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.890 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:57.890 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.260573+0000", 2024-09-21T14:04:57.890 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2839999999999999e-06 2024-09-21T14:04:57.890 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.890 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.890 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:57.890 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.260577+0000", 2024-09-21T14:04:57.890 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5940000000000004e-06 2024-09-21T14:04:57.890 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.891 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.891 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:57.891 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.260588+0000", 2024-09-21T14:04:57.891 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1163e-05 2024-09-21T14:04:57.891 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.891 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.891 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:57.891 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.260596+0000", 2024-09-21T14:04:57.891 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0520000000000004e-06 2024-09-21T14:04:57.891 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.891 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.891 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:57.891 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.260646+0000", 2024-09-21T14:04:57.892 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0593999999999997e-05 2024-09-21T14:04:57.892 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.892 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.892 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:04:57.892 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.260743+0000", 2024-09-21T14:04:57.892 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6998999999999997e-05 2024-09-21T14:04:57.892 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.892 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.892 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,4", 2024-09-21T14:04:57.892 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.260842+0000", 2024-09-21T14:04:57.892 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9185999999999995e-05 2024-09-21T14:04:57.893 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.893 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.893 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:04:57.893 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.274135+0000", 2024-09-21T14:04:57.893 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.013293142000000001 2024-09-21T14:04:57.893 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.893 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.893 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:57.893 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.274165+0000", 2024-09-21T14:04:57.893 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9277000000000001e-05 2024-09-21T14:04:57.893 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.893 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.893 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:57.894 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.523933+0000", 2024-09-21T14:04:57.894 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.249768038 2024-09-21T14:04:57.894 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.894 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.894 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:04:57.894 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.523976+0000", 2024-09-21T14:04:57.894 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3120000000000001e-05 2024-09-21T14:04:57.894 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.894 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.894 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:57.894 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.524010+0000", 2024-09-21T14:04:57.894 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3900999999999999e-05 2024-09-21T14:04:57.895 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:57.895 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:57.895 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:57.895 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.895 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.895 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25421.0:172337 9.3 9:c4d16778:::benchmark_data_smithi049_138504_object50929:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-09-21T14:04:57.895 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:00:33.262471+0000", 2024-09-21T14:04:57.895 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 264.55075635899999, 2024-09-21T14:04:57.895 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26159086300000001, 2024-09-21T14:04:57.895 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:57.895 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:57.896 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:57.896 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:04:57.896 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25421", 2024-09-21T14:04:57.896 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:04:57.896 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 172337 2024-09-21T14:04:57.896 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.896 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:57.896 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.896 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:57.896 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.262471+0000", 2024-09-21T14:04:57.896 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:57.896 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.896 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.897 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:57.897 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.262471+0000", 2024-09-21T14:04:57.897 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:57.897 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.897 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.897 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:57.897 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.262471+0000", 2024-09-21T14:04:57.897 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2500000000000004e-07 2024-09-21T14:04:57.897 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.897 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.897 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:57.897 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.262835+0000", 2024-09-21T14:04:57.898 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00036389300000000001 2024-09-21T14:04:57.898 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.898 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.898 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:57.898 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.262844+0000", 2024-09-21T14:04:57.898 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0299999999999999e-06 2024-09-21T14:04:57.898 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.898 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.898 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:57.898 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.262851+0000", 2024-09-21T14:04:57.898 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3600000000000001e-06 2024-09-21T14:04:57.899 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.899 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.899 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:57.899 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.263740+0000", 2024-09-21T14:04:57.899 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00088968700000000001 2024-09-21T14:04:57.899 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.899 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.899 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:04:57.899 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.273507+0000", 2024-09-21T14:04:57.899 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0097663909999999993 2024-09-21T14:04:57.899 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.899 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.900 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,7", 2024-09-21T14:04:57.900 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.273631+0000", 2024-09-21T14:04:57.900 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012441500000000001 2024-09-21T14:04:57.900 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.900 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.900 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:04:57.900 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.279244+0000", 2024-09-21T14:04:57.900 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0056131870000000004 2024-09-21T14:04:57.900 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:57.902 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:57.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:04:57.903 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.903 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:57.903 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.283719+0000", 2024-09-21T14:04:57.903 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0044745640000000003 2024-09-21T14:04:57.903 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.903 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.903 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:57.904 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.523913+0000", 2024-09-21T14:04:57.904 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.24019366 2024-09-21T14:04:57.904 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.904 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.904 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:04:57.904 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.523964+0000", 2024-09-21T14:04:57.904 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1039999999999999e-05 2024-09-21T14:04:57.904 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.904 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.904 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:57.904 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.524061+0000", 2024-09-21T14:04:57.904 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7711e-05 2024-09-21T14:04:57.905 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:57.905 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:57.905 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:57.905 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.905 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.905 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25421.0:172338 9.3 9:cb49996e:::benchmark_data_smithi049_138504_object50930:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-09-21T14:04:57.905 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:00:33.262872+0000", 2024-09-21T14:04:57.905 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 264.55035485399998, 2024-09-21T14:04:57.905 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26143280099999999, 2024-09-21T14:04:57.905 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:57.905 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:57.905 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:57.906 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:04:57.906 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25421", 2024-09-21T14:04:57.906 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:04:57.906 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 172338 2024-09-21T14:04:57.906 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.906 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:57.906 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.906 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:57.906 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.262872+0000", 2024-09-21T14:04:57.906 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:57.906 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.907 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.907 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:57.907 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.262872+0000", 2024-09-21T14:04:57.907 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:57.907 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.907 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.907 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:57.907 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.262873+0000", 2024-09-21T14:04:57.907 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5300000000000004e-07 2024-09-21T14:04:57.907 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.907 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.908 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:57.908 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.262875+0000", 2024-09-21T14:04:57.908 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.706e-06 2024-09-21T14:04:57.908 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.908 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.908 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:57.908 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.262882+0000", 2024-09-21T14:04:57.908 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4989999999999999e-06 2024-09-21T14:04:57.908 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.908 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.908 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:57.908 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.262887+0000", 2024-09-21T14:04:57.909 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9509999999999997e-06 2024-09-21T14:04:57.909 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.909 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.909 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:57.909 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.276120+0000", 2024-09-21T14:04:57.909 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.013233300999999999 2024-09-21T14:04:57.909 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.909 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.909 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:04:57.909 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.276230+0000", 2024-09-21T14:04:57.909 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000110229 2024-09-21T14:04:57.909 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.910 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.910 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,7", 2024-09-21T14:04:57.910 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.276349+0000", 2024-09-21T14:04:57.910 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011883900000000001 2024-09-21T14:04:57.910 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.910 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.910 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:04:57.910 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.283223+0000", 2024-09-21T14:04:57.910 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.006873891 2024-09-21T14:04:57.910 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.911 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.911 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:57.911 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.283857+0000", 2024-09-21T14:04:57.911 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00063396899999999996 2024-09-21T14:04:57.911 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.911 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.911 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:57.911 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.524123+0000", 2024-09-21T14:04:57.911 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.24026641700000001 2024-09-21T14:04:57.911 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.911 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.912 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:04:57.912 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.524197+0000", 2024-09-21T14:04:57.912 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3915e-05 2024-09-21T14:04:57.912 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.912 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.912 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:57.912 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.524305+0000", 2024-09-21T14:04:57.912 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000107431 2024-09-21T14:04:57.912 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:57.912 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:57.912 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:57.912 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.913 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.913 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25421.0:187674 9.d 9:b1fcb922:::benchmark_data_smithi049_138504_object66266:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e350)", 2024-09-21T14:04:57.913 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:01:45.364290+0000", 2024-09-21T14:04:57.913 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 192.44893701300001, 2024-09-21T14:04:57.913 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.23297105300000001, 2024-09-21T14:04:57.913 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:57.913 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:57.913 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:57.913 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:04:57.913 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25421", 2024-09-21T14:04:57.913 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:04:57.914 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 187674 2024-09-21T14:04:57.914 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.914 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:57.914 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.914 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:57.914 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:01:45.364290+0000", 2024-09-21T14:04:57.914 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:57.914 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.914 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.914 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:57.914 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:01:45.364290+0000", 2024-09-21T14:04:57.915 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:57.915 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.915 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.915 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:57.915 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:01:45.364292+0000", 2024-09-21T14:04:57.915 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5069999999999999e-06 2024-09-21T14:04:57.915 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.915 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.915 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:57.915 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:01:45.364302+0000", 2024-09-21T14:04:57.915 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.764e-06 2024-09-21T14:04:57.916 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.916 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.916 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:57.916 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:01:45.364325+0000", 2024-09-21T14:04:57.916 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2481e-05 2024-09-21T14:04:57.916 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.916 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.916 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:57.916 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:01:45.364335+0000", 2024-09-21T14:04:57.916 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0591000000000001e-05 2024-09-21T14:04:57.916 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.917 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.917 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:57.917 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:01:45.364417+0000", 2024-09-21T14:04:57.917 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2250999999999996e-05 2024-09-21T14:04:57.917 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.917 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.917 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:04:57.917 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:01:45.364547+0000", 2024-09-21T14:04:57.917 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012936100000000001 2024-09-21T14:04:57.917 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.917 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.918 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,7", 2024-09-21T14:04:57.918 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:01:45.364720+0000", 2024-09-21T14:04:57.918 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000173139 2024-09-21T14:04:57.918 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.918 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.918 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:04:57.918 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:01:45.367514+0000", 2024-09-21T14:04:57.918 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0027938400000000001 2024-09-21T14:04:57.918 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.918 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.918 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:57.918 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:01:45.367749+0000", 2024-09-21T14:04:57.919 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000234792 2024-09-21T14:04:57.919 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.919 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.919 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:57.919 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:01:45.597049+0000", 2024-09-21T14:04:57.919 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22929991799999999 2024-09-21T14:04:57.919 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.919 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.919 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:04:57.919 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:01:45.597148+0000", 2024-09-21T14:04:57.919 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9321000000000007e-05 2024-09-21T14:04:57.919 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.919 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.920 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:57.920 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:01:45.597261+0000", 2024-09-21T14:04:57.920 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000113088 2024-09-21T14:04:57.920 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:57.920 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:57.920 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:57.920 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.920 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.920 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:131799 10.2 10:464db636:::benchmark_data_smithi049_176811_object4222:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-09-21T14:04:57.920 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:04:39.757991+0000", 2024-09-21T14:04:57.920 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 18.055235788000001, 2024-09-21T14:04:57.921 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27366854600000001, 2024-09-21T14:04:57.921 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:57.921 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:57.921 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:57.921 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:04:57.921 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:04:57.921 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:04:57.921 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 131799 2024-09-21T14:04:57.921 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.921 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:57.921 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.921 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:57.921 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:39.757991+0000", 2024-09-21T14:04:57.922 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:57.922 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.922 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.922 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:57.922 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:39.757991+0000", 2024-09-21T14:04:57.922 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:57.922 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.922 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.922 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:57.922 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:39.757993+0000", 2024-09-21T14:04:57.922 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7749999999999999e-06 2024-09-21T14:04:57.922 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.923 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.923 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:57.923 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:39.757999+0000", 2024-09-21T14:04:57.923 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9510000000000003e-06 2024-09-21T14:04:57.923 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.923 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.923 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:57.923 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:39.758011+0000", 2024-09-21T14:04:57.923 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2663000000000001e-05 2024-09-21T14:04:57.923 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.923 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.923 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:57.924 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:39.758019+0000", 2024-09-21T14:04:57.924 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1189999999999999e-06 2024-09-21T14:04:57.924 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.924 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.924 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:57.924 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:39.758078+0000", 2024-09-21T14:04:57.924 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9091000000000001e-05 2024-09-21T14:04:57.924 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.924 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.924 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:04:57.924 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:39.758290+0000", 2024-09-21T14:04:57.924 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000212258 2024-09-21T14:04:57.925 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.925 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.925 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,4", 2024-09-21T14:04:57.925 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:39.758402+0000", 2024-09-21T14:04:57.925 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000111629 2024-09-21T14:04:57.925 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.925 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.925 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:04:57.925 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:39.763486+0000", 2024-09-21T14:04:57.925 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0050847310000000003 2024-09-21T14:04:57.925 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.925 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.926 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:57.926 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:39.764304+0000", 2024-09-21T14:04:57.926 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00081737699999999999 2024-09-21T14:04:57.926 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.926 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.926 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:57.926 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:40.031555+0000", 2024-09-21T14:04:57.926 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26725161200000003 2024-09-21T14:04:57.926 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.926 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.926 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:04:57.927 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:40.031612+0000", 2024-09-21T14:04:57.927 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7058000000000002e-05 2024-09-21T14:04:57.927 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.927 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.927 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:57.927 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:40.031660+0000", 2024-09-21T14:04:57.927 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7281999999999999e-05 2024-09-21T14:04:57.927 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:57.927 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:57.927 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:57.927 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.927 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.928 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.16068.0:144704 10.4 e415/409 10:2120b6ff:::benchmark_data_smithi049_176811_object17127:head v 415'8973, mlcod=415'8971)", 2024-09-21T14:04:57.928 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:04:52.184045+0000", 2024-09-21T14:04:57.928 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 5.6291819470000002, 2024-09-21T14:04:57.928 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.17891684199999999, 2024-09-21T14:04:57.928 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:57.928 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:57.928 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:57.928 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:57.928 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.928 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:57.928 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.184045+0000", 2024-09-21T14:04:57.928 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:57.929 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.929 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.929 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:57.929 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.184045+0000", 2024-09-21T14:04:57.929 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:57.929 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.929 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.929 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:57.929 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.184046+0000", 2024-09-21T14:04:57.929 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4300000000000002e-07 2024-09-21T14:04:57.929 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.929 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.930 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:57.930 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.184051+0000", 2024-09-21T14:04:57.930 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3170000000000001e-06 2024-09-21T14:04:57.930 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.930 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.930 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:57.930 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.184061+0000", 2024-09-21T14:04:57.930 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.5810000000000003e-06 2024-09-21T14:04:57.930 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.930 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.930 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:57.930 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.184067+0000", 2024-09-21T14:04:57.931 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.883e-06 2024-09-21T14:04:57.931 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.931 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.931 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:57.931 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.184126+0000", 2024-09-21T14:04:57.931 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8319000000000002e-05 2024-09-21T14:04:57.931 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.931 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.931 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:04:57.931 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.184143+0000", 2024-09-21T14:04:57.931 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7506000000000001e-05 2024-09-21T14:04:57.931 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.932 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.932 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:04:57.932 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.362932+0000", 2024-09-21T14:04:57.932 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.178788318 2024-09-21T14:04:57.932 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.932 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.932 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:57.932 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.362962+0000", 2024-09-21T14:04:57.932 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.0175000000000002e-05 2024-09-21T14:04:57.932 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:57.932 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:57.932 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:57.933 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.933 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.933 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.16068.0:144700 10.0 e415/409 10:01ec5c76:::benchmark_data_smithi049_176811_object17123:head v 415'9086, mlcod=415'9084)", 2024-09-21T14:04:57.933 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:04:52.184533+0000", 2024-09-21T14:04:57.933 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 5.6286943730000001, 2024-09-21T14:04:57.933 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.185675167, 2024-09-21T14:04:57.933 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:57.933 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:57.933 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:57.933 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:57.933 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.933 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:57.933 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.184533+0000", 2024-09-21T14:04:57.934 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:57.934 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.934 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.934 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:57.934 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.184533+0000", 2024-09-21T14:04:57.934 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:57.934 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.934 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.934 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:57.934 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.184533+0000", 2024-09-21T14:04:57.934 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7700000000000004e-07 2024-09-21T14:04:57.935 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.935 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.935 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:57.935 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.184538+0000", 2024-09-21T14:04:57.935 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9899999999999997e-06 2024-09-21T14:04:57.935 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.935 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.935 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:57.935 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.184547+0000", 2024-09-21T14:04:57.935 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1519999999999998e-06 2024-09-21T14:04:57.935 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.936 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.936 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:57.936 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.184553+0000", 2024-09-21T14:04:57.936 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8139999999999997e-06 2024-09-21T14:04:57.936 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.936 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.936 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:57.936 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.185885+0000", 2024-09-21T14:04:57.936 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0013316300000000001 2024-09-21T14:04:57.936 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.936 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.936 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:04:57.937 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.185908+0000", 2024-09-21T14:04:57.937 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2931000000000001e-05 2024-09-21T14:04:57.937 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.937 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.937 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:04:57.937 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.370166+0000", 2024-09-21T14:04:57.937 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.18425854799999999 2024-09-21T14:04:57.937 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.937 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.937 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:57.937 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.370208+0000", 2024-09-21T14:04:57.938 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1325000000000002e-05 2024-09-21T14:04:57.938 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:57.938 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:57.938 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:57.938 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.938 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.938 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:144709 10.2 10:41650bcc:::benchmark_data_smithi049_176811_object17132:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-09-21T14:04:57.938 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:04:52.185400+0000", 2024-09-21T14:04:57.938 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 5.6278269910000001, 2024-09-21T14:04:57.938 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.18154635399999999, 2024-09-21T14:04:57.938 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:57.938 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:57.939 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:57.939 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:04:57.939 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:04:57.939 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:04:57.939 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 144709 2024-09-21T14:04:57.939 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.939 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:57.939 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.939 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:57.939 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.185400+0000", 2024-09-21T14:04:57.939 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:57.940 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.940 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.940 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:57.940 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.185400+0000", 2024-09-21T14:04:57.940 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:57.940 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.940 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.940 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:57.940 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.185402+0000", 2024-09-21T14:04:57.940 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6789999999999999e-06 2024-09-21T14:04:57.940 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.940 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.941 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:57.941 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.185408+0000", 2024-09-21T14:04:57.941 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9800000000000003e-06 2024-09-21T14:04:57.941 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.941 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.941 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:57.941 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.185422+0000", 2024-09-21T14:04:57.941 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4616e-05 2024-09-21T14:04:57.941 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.941 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.941 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:57.942 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.185430+0000", 2024-09-21T14:04:57.942 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3159999999999997e-06 2024-09-21T14:04:57.942 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.942 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.942 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:57.942 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.185502+0000", 2024-09-21T14:04:57.942 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1159000000000001e-05 2024-09-21T14:04:57.942 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.942 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.942 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:04:57.942 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.185636+0000", 2024-09-21T14:04:57.942 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013425000000000001 2024-09-21T14:04:57.943 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.943 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.943 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,4", 2024-09-21T14:04:57.943 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.185751+0000", 2024-09-21T14:04:57.943 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000115553 2024-09-21T14:04:57.943 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.943 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.943 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:57.943 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.365854+0000", 2024-09-21T14:04:57.943 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.180102969 2024-09-21T14:04:57.943 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.944 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.944 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:04:57.944 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.366438+0000", 2024-09-21T14:04:57.944 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000584028 2024-09-21T14:04:57.944 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.944 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.944 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:57.944 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.366480+0000", 2024-09-21T14:04:57.944 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1151e-05 2024-09-21T14:04:57.944 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.944 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.944 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:04:57.945 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.366893+0000", 2024-09-21T14:04:57.945 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00041380099999999999 2024-09-21T14:04:57.945 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.945 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.945 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:57.945 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.366946+0000", 2024-09-21T14:04:57.945 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2852000000000002e-05 2024-09-21T14:04:57.945 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:57.945 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:57.945 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:57.945 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.945 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.946 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.16068.0:144711 10.b e415/381 10:dffdfab5:::benchmark_data_smithi049_176811_object17134:head v 415'8996, mlcod=415'8994)", 2024-09-21T14:04:57.946 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:04:52.188100+0000", 2024-09-21T14:04:57.946 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 5.6251267049999996, 2024-09-21T14:04:57.946 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.18037874700000001, 2024-09-21T14:04:57.946 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:57.946 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:57.946 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:57.946 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:57.946 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.946 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:57.946 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.188100+0000", 2024-09-21T14:04:57.946 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:57.947 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.947 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.947 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:57.947 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.188100+0000", 2024-09-21T14:04:57.947 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:57.947 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.947 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.947 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:57.947 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.188101+0000", 2024-09-21T14:04:57.947 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.201e-06 2024-09-21T14:04:57.947 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.947 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.948 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:57.948 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.188108+0000", 2024-09-21T14:04:57.948 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4949999999999996e-06 2024-09-21T14:04:57.948 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.948 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.948 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:57.948 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.362265+0000", 2024-09-21T14:04:57.948 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.17415722 2024-09-21T14:04:57.948 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.948 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.948 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:57.948 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.362276+0000", 2024-09-21T14:04:57.949 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1001e-05 2024-09-21T14:04:57.949 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.949 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.949 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:57.949 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.363070+0000", 2024-09-21T14:04:57.949 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00079424599999999997 2024-09-21T14:04:57.949 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.949 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.949 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:04:57.949 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.363092+0000", 2024-09-21T14:04:57.949 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1348e-05 2024-09-21T14:04:57.949 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.950 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.950 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:04:57.950 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.368438+0000", 2024-09-21T14:04:57.950 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0053467059999999997 2024-09-21T14:04:57.950 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.950 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.950 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:57.950 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.368479+0000", 2024-09-21T14:04:57.950 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.053e-05 2024-09-21T14:04:57.950 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:57.950 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:57.950 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:57.951 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.951 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.951 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.16068.0:144709 10.2 e415/381 ondisk, result = 0)", 2024-09-21T14:04:57.951 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:04:52.198323+0000", 2024-09-21T14:04:57.951 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 5.6149034870000003, 2024-09-21T14:04:57.951 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.16863062400000001, 2024-09-21T14:04:57.951 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:57.951 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:57.951 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T14:04:57.951 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:57.951 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.951 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:57.952 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.198323+0000", 2024-09-21T14:04:57.952 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:57.952 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.952 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.952 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:57.952 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.198323+0000", 2024-09-21T14:04:57.952 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:57.952 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.952 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.952 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:57.952 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.198325+0000", 2024-09-21T14:04:57.953 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3850000000000001e-06 2024-09-21T14:04:57.953 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.953 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.953 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:57.953 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.198333+0000", 2024-09-21T14:04:57.953 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5890000000000003e-06 2024-09-21T14:04:57.953 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.953 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.953 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:57.953 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.362520+0000", 2024-09-21T14:04:57.953 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.164186419 2024-09-21T14:04:57.954 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.954 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.954 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:57.954 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.362525+0000", 2024-09-21T14:04:57.954 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4450000000000004e-06 2024-09-21T14:04:57.954 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.954 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.954 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:57.954 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.366468+0000", 2024-09-21T14:04:57.954 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0039423280000000002 2024-09-21T14:04:57.955 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.955 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.955 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:04:57.955 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.366474+0000", 2024-09-21T14:04:57.955 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.861e-06 2024-09-21T14:04:57.955 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.955 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.955 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:57.955 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.366954+0000", 2024-09-21T14:04:57.955 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00047959700000000001 2024-09-21T14:04:57.955 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:57.955 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:57.956 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:57.956 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.956 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.956 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:144719 10.2 10:4ec97677:::benchmark_data_smithi049_176811_object17142:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-09-21T14:04:57.956 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:04:52.198581+0000", 2024-09-21T14:04:57.956 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 5.6146463439999996, 2024-09-21T14:04:57.956 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.179241806, 2024-09-21T14:04:57.956 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:04:57.956 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:04:57.956 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:04:57.956 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:04:57.956 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:04:57.957 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:04:57.957 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 144719 2024-09-21T14:04:57.957 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.957 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:04:57.957 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.957 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:04:57.957 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.198581+0000", 2024-09-21T14:04:57.957 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:57.957 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.957 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.957 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:04:57.957 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.198581+0000", 2024-09-21T14:04:57.958 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:04:57.958 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.958 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.958 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:04:57.958 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.198582+0000", 2024-09-21T14:04:57.958 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.827e-06 2024-09-21T14:04:57.958 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.958 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.958 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:04:57.958 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.198589+0000", 2024-09-21T14:04:57.958 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5679999999999996e-06 2024-09-21T14:04:57.958 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.959 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.959 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:04:57.959 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.363090+0000", 2024-09-21T14:04:57.959 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.16450070899999999 2024-09-21T14:04:57.959 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.959 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.959 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:04:57.959 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.363102+0000", 2024-09-21T14:04:57.959 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1868999999999999e-05 2024-09-21T14:04:57.959 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.959 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.959 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:04:57.960 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.366982+0000", 2024-09-21T14:04:57.960 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.003880278 2024-09-21T14:04:57.960 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.960 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.960 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:04:57.960 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.367079+0000", 2024-09-21T14:04:57.960 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7411999999999995e-05 2024-09-21T14:04:57.960 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.960 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.960 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,4", 2024-09-21T14:04:57.960 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.367178+0000", 2024-09-21T14:04:57.960 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8770999999999993e-05 2024-09-21T14:04:57.960 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.961 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.961 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:57.961 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.377632+0000", 2024-09-21T14:04:57.961 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.010454296 2024-09-21T14:04:57.961 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.961 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.961 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:04:57.961 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.377644+0000", 2024-09-21T14:04:57.961 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1721e-05 2024-09-21T14:04:57.961 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.961 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.961 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:04:57.962 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.377674+0000", 2024-09-21T14:04:57.962 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.0428000000000001e-05 2024-09-21T14:04:57.962 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.962 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.962 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:04:57.962 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.377776+0000", 2024-09-21T14:04:57.962 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000101686 2024-09-21T14:04:57.962 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:04:57.962 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:04:57.962 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:04:57.962 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:52.377822+0000", 2024-09-21T14:04:57.963 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6241e-05 2024-09-21T14:04:57.963 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:57.963 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:57.963 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:57.963 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:04:57.963 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:04:57.963 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T14:04:57.964 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:57 smithi187 ceph-mon[33032]: pgmap v810: 158 pgs: 4 active+remapped+backfilling, 154 active+clean; 6.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s; 22475/327048 objects misplaced (6.872%); 14 MiB/s, 226 objects/s recovering 2024-09-21T14:04:57.964 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:57 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:04:57.964 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:57 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:04:57.964 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:57 smithi187 ceph-mon[33032]: osdmap e418: 8 total, 8 up, 8 in 2024-09-21T14:04:58.003 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:04:58.039 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:58.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:58.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:04:58.141 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:04:58.184 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:58.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:58.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:04:58.286 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:04:58.325 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:58.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:58.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:04:58.426 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:04:58.461 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:58.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:58.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:04:58.562 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:04:58.603 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:58.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:58.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:04:58.705 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:04:58.752 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:58.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:58.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:04:58.854 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:04:58.893 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:58.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:58.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:04:58.952 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:04:58.995 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:04:59.040 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:59.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:59.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:04:59.142 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:04:59.183 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:59.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:59.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:04:59.284 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:04:59.332 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:59.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:59.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:04:59.434 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:04:59.471 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:59.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:59.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:04:59.573 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:04:59.608 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:04:59.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:59.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:04:59.611 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T14:04:59.711 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:04:59.747 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:59.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:59.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:04:59.849 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:04:59.896 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:04:59.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:04:59.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:04:59.913 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:59 smithi187 ceph-mon[33032]: pgmap v812: 158 pgs: 4 active+remapped+backfilling, 154 active+clean; 6.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 935 op/s; 22475/324423 objects misplaced (6.928%); 13 MiB/s, 200 objects/s recovering 2024-09-21T14:04:59.913 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:59 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:04:59.913 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:04:59 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:04:59.915 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:59 smithi049 ceph-mon[40489]: pgmap v812: 158 pgs: 4 active+remapped+backfilling, 154 active+clean; 6.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 935 op/s; 22475/324423 objects misplaced (6.928%); 13 MiB/s, 200 objects/s recovering 2024-09-21T14:04:59.915 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:59 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:04:59.916 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:04:59 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:04:59.916 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:59 smithi049 ceph-mon[30080]: pgmap v812: 158 pgs: 4 active+remapped+backfilling, 154 active+clean; 6.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 935 op/s; 22475/324423 objects misplaced (6.928%); 13 MiB/s, 200 objects/s recovering 2024-09-21T14:04:59.916 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:59 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:04:59.916 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:04:59 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:04:59.998 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:00.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:00.054 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:00.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:00.087 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:05:00.155 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:00.193 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:00.194 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:04:59 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:05:00.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:00.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:00.295 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:05:00.306 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:05:00.306 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:05:00.330 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:00.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:00.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:05:00.336 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:05:00.336 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:05:00.371 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:05:00.371 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:05:00.405 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:05:00.406 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:05:00.432 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:00.444 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:05:00.445 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:05:00.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:00.475 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:00.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:00.479 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:05:00.480 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:05:00.519 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:05:00.520 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:05:00.555 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:05:00.555 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:05:00.577 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:05:00.619 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:00.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:00.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:05:00.653 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T14:05:00.721 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:05:00.765 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:00.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:00.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:05:00.774 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T14:05:00.774 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:05:00.867 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:00.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:00.905 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:00.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:00.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:00 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:05:00.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:00 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/665726726' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:05:00.973 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:00 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:05:00.973 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:00 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/665726726' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:05:00.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:00 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:05:00.973 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:00 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/665726726' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:05:00.974 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:05:00 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:05:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:05:01.006 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:05:01.045 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:01.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:01.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:05:01.147 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:01.184 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:01.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:01.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:01.286 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:05:01.324 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:01.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:01.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:05:01.426 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:05:01.466 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:01.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:01.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:05:01.568 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:01.607 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:01.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:01.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:01.709 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:05:01.754 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:01.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:01.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:05:01.764 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T14:05:01.856 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:01.862 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:05:01.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:01.891 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:01.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:01.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:01 smithi187 ceph-mon[33032]: pgmap v813: 158 pgs: 4 active+remapped+backfilling, 154 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.18k op/s; 17769/314595 objects misplaced (5.648%); 42 MiB/s, 678 objects/s recovering 2024-09-21T14:05:01.991 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:01 smithi049 ceph-mon[40489]: pgmap v813: 158 pgs: 4 active+remapped+backfilling, 154 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.18k op/s; 17769/314595 objects misplaced (5.648%); 42 MiB/s, 678 objects/s recovering 2024-09-21T14:05:01.991 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:01 smithi049 ceph-mon[30080]: pgmap v813: 158 pgs: 4 active+remapped+backfilling, 154 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.18k op/s; 17769/314595 objects misplaced (5.648%); 42 MiB/s, 678 objects/s recovering 2024-09-21T14:05:01.993 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:05:02.030 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:02.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:02.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:05:02.131 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:05:02.197 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:02.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:02.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:05:02.198 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:05:02.299 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:05:02.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:02.346 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:02.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:05:02.448 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:05:02.494 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:02.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:02.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:05:02.566 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 4, 6, 2, 3, 7, 5] 2024-09-21T14:05:02.566 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T14:05:02.567 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-09-21T14:05:02.567 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-09-21T14:05:02.567 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T14:05:02.595 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:05:02.672 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:02.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:02.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:05:02.775 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:05:02.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:02.857 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:02.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:05:02.958 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:02.998 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:02.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:02.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:03.061 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:05:03.072 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T14:05:03.072 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:05:03.099 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:05:03.147 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:03.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:03.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:05:03.249 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:05:03.289 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:03.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:03.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:05:03.391 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:03.424 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:03.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:03.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:03.526 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:05:03.565 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:03.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:03.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:05:03.622 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-09-21T14:05:03.667 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:05:03.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:03 smithi187 ceph-mon[33032]: pgmap v814: 158 pgs: 4 active+remapped+backfilling, 154 active+clean; 6.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 759 op/s; 17769/311691 objects misplaced (5.701%); 30 MiB/s, 485 objects/s recovering 2024-09-21T14:05:03.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:03 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2422689468' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:05:03.720 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:03.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:03.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:05:03.752 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T14:05:03.822 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:05:03.874 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:03.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:03.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:05:03.882 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T14:05:03.975 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:04.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:03 smithi049 ceph-mon[40489]: pgmap v814: 158 pgs: 4 active+remapped+backfilling, 154 active+clean; 6.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 759 op/s; 17769/311691 objects misplaced (5.701%); 30 MiB/s, 485 objects/s recovering 2024-09-21T14:05:04.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:03 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2422689468' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:05:04.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:03 smithi049 ceph-mon[30080]: pgmap v814: 158 pgs: 4 active+remapped+backfilling, 154 active+clean; 6.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 759 op/s; 17769/311691 objects misplaced (5.701%); 30 MiB/s, 485 objects/s recovering 2024-09-21T14:05:04.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:03 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2422689468' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:05:04.009 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:04.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:04.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:04.111 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:05:04.118 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T14:05:04.123 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-09-21T14:05:04.149 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:04.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:04.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:05:04.251 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:05:04.305 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:05:04.306 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:04.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:04.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:05:04.408 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:05:04.453 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:04.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:04.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:05:04.554 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:04.594 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:04.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:04.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:04.696 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:04.732 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:04.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:04.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:04.780 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:04 smithi187 ceph-mon[33032]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:05:04.780 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:04 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2422689468' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T14:05:04.780 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:04 smithi187 ceph-mon[33032]: osdmap e419: 8 total, 8 up, 8 in 2024-09-21T14:05:04.835 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:04.870 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:04.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:04.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:04.922 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:04 smithi049 ceph-mon[30080]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:05:04.922 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:04 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2422689468' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T14:05:04.922 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:04 smithi049 ceph-mon[30080]: osdmap e419: 8 total, 8 up, 8 in 2024-09-21T14:05:04.923 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:04 smithi049 ceph-mon[40489]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:05:04.923 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:04 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2422689468' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T14:05:04.923 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:04 smithi049 ceph-mon[40489]: osdmap e419: 8 total, 8 up, 8 in 2024-09-21T14:05:04.971 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:05:05.008 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:05.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:05.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:05:05.109 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:05:05.139 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:05.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:05.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:05:05.240 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:05.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:05.284 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:05.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:05.319 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T14:05:05.319 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:05:05.385 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:05:05.408 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":816,"stamp":"2024-09-21T14:05:04.046551+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6543728842,"num_objects":100039,"num_object_clones":0,"num_object_copies":300117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":14403,"num_objects_unfound":0,"num_objects_dirty":100039,"num_whiteouts":0,"num_read":1485,"num_read_kb":1274,"num_write":156077,"num_write_kb":8165374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11436,"num_bytes_recovered":736121401,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83064,"ondisk_log_size":83064,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":478,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":23879092,"kb_used_data":20055752,"kb_used_omap":23,"kb_used_meta":3822632,"kb_avail":726082124,"statfs":{"total":767960285184,"available":743508094976,"internally_reserved":0,"allocated":20537090048,"data_stored":20534631841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24148,"internal_metadata":3914375596},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,4,2,0,1,0,0,12,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-588316672,"num_objects":-8977,"num_object_clones":0,"num_object_copies":-26931,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-8072,"num_objects_unfound":0,"num_objects_dirty":-8977,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":8977,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6364,"num_bytes_recovered":417464320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002375"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":668,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.586996+0000","last_change":"2024-09-21T14:04:21.204685+0000","last_active":"2024-09-21T14:04:56.586996+0000","last_peered":"2024-09-21T14:04:56.586996+0000","last_clean":"2024-09-21T14:04:56.586996+0000","last_became_active":"2024-09-21T14:04:21.204299+0000","last_became_peered":"2024-09-21T14:04:21.204299+0000","last_unstale":"2024-09-21T14:04:56.586996+0000","last_undegraded":"2024-09-21T14:04:56.586996+0000","last_fullsized":"2024-09-21T14:04:56.586996+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:56.458125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:02:56.458125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"290'15","reported_seq":830,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-21T14:05:03.631130+0000","last_change":"2024-09-21T14:01:34.785322+0000","last_active":"2024-09-21T14:05:03.631130+0000","last_peered":"2024-09-21T14:05:03.631130+0000","last_clean":"2024-09-21T14:05:03.631130+0000","last_became_active":"2024-09-21T13:58:17.771220+0000","last_became_peered":"2024-09-21T13:58:17.771220+0000","last_unstale":"2024-09-21T14:05:03.631130+0000","last_undegraded":"2024-09-21T14:05:03.631130+0000","last_fullsized":"2024-09-21T14:05:03.631130+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'15","last_scrub_stamp":"2024-09-21T14:01:34.785267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:34.785267+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":777,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.579142+0000","last_change":"2024-09-21T14:03:33.301558+0000","last_active":"2024-09-21T14:04:56.579142+0000","last_peered":"2024-09-21T14:04:56.579142+0000","last_clean":"2024-09-21T14:04:56.579142+0000","last_became_active":"2024-09-21T14:01:41.202168+0000","last_became_peered":"2024-09-21T14:01:41.202168+0000","last_unstale":"2024-09-21T14:04:56.579142+0000","last_undegraded":"2024-09-21T14:04:56.579142+0000","last_fullsized":"2024-09-21T14:04:56.579142+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:33.301487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:33.301487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":747,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-21T14:05:03.631901+0000","last_change":"2024-09-21T14:01:41.203762+0000","last_active":"2024-09-21T14:05:03.631901+0000","last_peered":"2024-09-21T14:05:03.631901+0000","last_clean":"2024-09-21T14:05:03.631901+0000","last_became_active":"2024-09-21T14:01:41.203275+0000","last_became_peered":"2024-09-21T14:01:41.203275+0000","last_unstale":"2024-09-21T14:05:03.631901+0000","last_undegraded":"2024-09-21T14:05:03.631901+0000","last_fullsized":"2024-09-21T14:05:03.631901+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:00.320510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:00.320510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":723,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.584653+0000","last_change":"2024-09-21T14:04:00.241774+0000","last_active":"2024-09-21T14:04:56.584653+0000","last_peered":"2024-09-21T14:04:56.584653+0000","last_clean":"2024-09-21T14:04:56.584653+0000","last_became_active":"2024-09-21T13:59:32.203340+0000","last_became_peered":"2024-09-21T13:59:32.203340+0000","last_unstale":"2024-09-21T14:04:56.584653+0000","last_undegraded":"2024-09-21T14:04:56.584653+0000","last_fullsized":"2024-09-21T14:04:56.584653+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:00.241562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:04:00.241562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:19.693680+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":727,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.585268+0000","last_change":"2024-09-21T13:58:17.778322+0000","last_active":"2024-09-21T14:04:56.585268+0000","last_peered":"2024-09-21T14:04:56.585268+0000","last_clean":"2024-09-21T14:04:56.585268+0000","last_became_active":"2024-09-21T13:58:17.777893+0000","last_became_peered":"2024-09-21T13:58:17.777893+0000","last_unstale":"2024-09-21T14:04:56.585268+0000","last_undegraded":"2024-09-21T14:04:56.585268+0000","last_fullsized":"2024-09-21T14:04:56.585268+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:07.001135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:55:07.001135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"290'27","reported_seq":709,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.585055+0000","last_change":"2024-09-21T14:04:08.320458+0000","last_active":"2024-09-21T14:04:56.585055+0000","last_peered":"2024-09-21T14:04:56.585055+0000","last_clean":"2024-09-21T14:04:56.585055+0000","last_became_active":"2024-09-21T13:57:12.703469+0000","last_became_peered":"2024-09-21T13:57:12.703469+0000","last_unstale":"2024-09-21T14:04:56.585055+0000","last_undegraded":"2024-09-21T14:04:56.585055+0000","last_fullsized":"2024-09-21T14:04:56.585055+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'27","last_scrub_stamp":"2024-09-21T14:04:08.320339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:04:08.320339+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:32.005889+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":484,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-21T14:05:03.637085+0000","last_change":"2024-09-21T14:04:35.907676+0000","last_active":"2024-09-21T14:05:03.637085+0000","last_peered":"2024-09-21T14:05:03.637085+0000","last_clean":"2024-09-21T14:05:03.637085+0000","last_became_active":"2024-09-21T14:04:21.208237+0000","last_became_peered":"2024-09-21T14:04:21.208237+0000","last_unstale":"2024-09-21T14:05:03.637085+0000","last_undegraded":"2024-09-21T14:05:03.637085+0000","last_fullsized":"2024-09-21T14:05:03.637085+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:35.907611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:04:35.907611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:39.685760+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":728,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.586360+0000","last_change":"2024-09-21T14:04:11.718218+0000","last_active":"2024-09-21T14:04:56.586360+0000","last_peered":"2024-09-21T14:04:56.586360+0000","last_clean":"2024-09-21T14:04:56.586360+0000","last_became_active":"2024-09-21T14:01:41.194517+0000","last_became_peered":"2024-09-21T14:01:41.194517+0000","last_unstale":"2024-09-21T14:04:56.586360+0000","last_undegraded":"2024-09-21T14:04:56.586360+0000","last_fullsized":"2024-09-21T14:04:56.586360+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:11.718143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:11.718143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:19.307011+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"290'15","reported_seq":567,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.584136+0000","last_change":"2024-09-21T14:04:21.228412+0000","last_active":"2024-09-21T14:04:56.584136+0000","last_peered":"2024-09-21T14:04:56.584136+0000","last_clean":"2024-09-21T14:04:56.584136+0000","last_became_active":"2024-09-21T14:04:21.208244+0000","last_became_peered":"2024-09-21T14:04:21.208244+0000","last_unstale":"2024-09-21T14:04:56.584136+0000","last_undegraded":"2024-09-21T14:04:56.584136+0000","last_fullsized":"2024-09-21T14:04:56.584136+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'15","last_scrub_stamp":"2024-09-21T14:00:02.475661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:00:02.475661+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":669,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.586655+0000","last_change":"2024-09-21T14:03:04.516642+0000","last_active":"2024-09-21T14:04:56.586655+0000","last_peered":"2024-09-21T14:04:56.586655+0000","last_clean":"2024-09-21T14:04:56.586655+0000","last_became_active":"2024-09-21T14:01:41.188248+0000","last_became_peered":"2024-09-21T14:01:41.188248+0000","last_unstale":"2024-09-21T14:04:56.586655+0000","last_undegraded":"2024-09-21T14:04:56.586655+0000","last_fullsized":"2024-09-21T14:04:56.586655+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:04.516583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:04.516583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":705,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.582194+0000","last_change":"2024-09-21T14:04:21.205796+0000","last_active":"2024-09-21T14:04:56.582194+0000","last_peered":"2024-09-21T14:04:56.582194+0000","last_clean":"2024-09-21T14:04:56.582194+0000","last_became_active":"2024-09-21T14:04:21.205416+0000","last_became_peered":"2024-09-21T14:04:21.205416+0000","last_unstale":"2024-09-21T14:04:56.582194+0000","last_undegraded":"2024-09-21T14:04:56.582194+0000","last_fullsized":"2024-09-21T14:04:56.582194+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:25.792985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:54:25.792985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":705,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.586041+0000","last_change":"2024-09-21T14:04:10.722881+0000","last_active":"2024-09-21T14:04:56.586041+0000","last_peered":"2024-09-21T14:04:56.586041+0000","last_clean":"2024-09-21T14:04:56.586041+0000","last_became_active":"2024-09-21T14:01:41.189373+0000","last_became_peered":"2024-09-21T14:01:41.189373+0000","last_unstale":"2024-09-21T14:04:56.586041+0000","last_undegraded":"2024-09-21T14:04:56.586041+0000","last_fullsized":"2024-09-21T14:04:56.586041+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:10.722799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:10.722799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:25.238119+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"290'7","reported_seq":567,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.584761+0000","last_change":"2024-09-21T14:04:21.231643+0000","last_active":"2024-09-21T14:04:56.584761+0000","last_peered":"2024-09-21T14:04:56.584761+0000","last_clean":"2024-09-21T14:04:56.584761+0000","last_became_active":"2024-09-21T14:04:21.213102+0000","last_became_peered":"2024-09-21T14:04:21.213102+0000","last_unstale":"2024-09-21T14:04:56.584761+0000","last_undegraded":"2024-09-21T14:04:56.584761+0000","last_fullsized":"2024-09-21T14:04:56.584761+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'7","last_scrub_stamp":"2024-09-21T14:01:57.563184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:57.563184+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":690,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.584299+0000","last_change":"2024-09-21T13:59:32.204565+0000","last_active":"2024-09-21T14:04:56.584299+0000","last_peered":"2024-09-21T14:04:56.584299+0000","last_clean":"2024-09-21T14:04:56.584299+0000","last_became_active":"2024-09-21T13:59:32.204174+0000","last_became_peered":"2024-09-21T13:59:32.204174+0000","last_unstale":"2024-09-21T14:04:56.584299+0000","last_undegraded":"2024-09-21T14:04:56.584299+0000","last_fullsized":"2024-09-21T14:04:56.584299+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:06.722293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:06.722293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":739,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.585850+0000","last_change":"2024-09-21T14:04:46.471869+0000","last_active":"2024-09-21T14:04:56.585850+0000","last_peered":"2024-09-21T14:04:56.585850+0000","last_clean":"2024-09-21T14:04:56.585850+0000","last_became_active":"2024-09-21T14:01:41.198672+0000","last_became_peered":"2024-09-21T14:01:41.198672+0000","last_unstale":"2024-09-21T14:04:56.585850+0000","last_undegraded":"2024-09-21T14:04:56.585850+0000","last_fullsized":"2024-09-21T14:04:56.585850+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:46.471817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:04:46.471817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:01.005314+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":797,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-21T14:05:03.632154+0000","last_change":"2024-09-21T14:03:36.018910+0000","last_active":"2024-09-21T14:05:03.632154+0000","last_peered":"2024-09-21T14:05:03.632154+0000","last_clean":"2024-09-21T14:05:03.632154+0000","last_became_active":"2024-09-21T14:01:41.200360+0000","last_became_peered":"2024-09-21T14:01:41.200360+0000","last_unstale":"2024-09-21T14:05:03.632154+0000","last_undegraded":"2024-09-21T14:05:03.632154+0000","last_fullsized":"2024-09-21T14:05:03.632154+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:36.018851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:36.018851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:05.142781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":714,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-21T14:05:03.631215+0000","last_change":"2024-09-21T14:04:21.202574+0000","last_active":"2024-09-21T14:05:03.631215+0000","last_peered":"2024-09-21T14:05:03.631215+0000","last_clean":"2024-09-21T14:05:03.631215+0000","last_became_active":"2024-09-21T14:04:21.202178+0000","last_became_peered":"2024-09-21T14:04:21.202178+0000","last_unstale":"2024-09-21T14:05:03.631215+0000","last_undegraded":"2024-09-21T14:05:03.631215+0000","last_fullsized":"2024-09-21T14:05:03.631215+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:36.679321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:02:36.679321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"290'27","reported_seq":778,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.582023+0000","last_change":"2024-09-21T14:04:01.621388+0000","last_active":"2024-09-21T14:04:56.582023+0000","last_peered":"2024-09-21T14:04:56.582023+0000","last_clean":"2024-09-21T14:04:56.582023+0000","last_became_active":"2024-09-21T13:57:12.680783+0000","last_became_peered":"2024-09-21T13:57:12.680783+0000","last_unstale":"2024-09-21T14:04:56.582023+0000","last_undegraded":"2024-09-21T14:04:56.582023+0000","last_fullsized":"2024-09-21T14:04:56.582023+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'27","last_scrub_stamp":"2024-09-21T14:04:01.621303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:04:01.621303+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:23.163153+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":770,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.580460+0000","last_change":"2024-09-21T14:04:21.204283+0000","last_active":"2024-09-21T14:04:56.580460+0000","last_peered":"2024-09-21T14:04:56.580460+0000","last_clean":"2024-09-21T14:04:56.580460+0000","last_became_active":"2024-09-21T14:04:21.203937+0000","last_became_peered":"2024-09-21T14:04:21.203937+0000","last_unstale":"2024-09-21T14:04:56.580460+0000","last_undegraded":"2024-09-21T14:04:56.580460+0000","last_fullsized":"2024-09-21T14:04:56.580460+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:08.299252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:08.299252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":707,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.585578+0000","last_change":"2024-09-21T14:01:41.191486+0000","last_active":"2024-09-21T14:04:56.585578+0000","last_peered":"2024-09-21T14:04:56.585578+0000","last_clean":"2024-09-21T14:04:56.585578+0000","last_became_active":"2024-09-21T14:01:41.191116+0000","last_became_peered":"2024-09-21T14:01:41.191116+0000","last_unstale":"2024-09-21T14:04:56.585578+0000","last_undegraded":"2024-09-21T14:04:56.585578+0000","last_fullsized":"2024-09-21T14:04:56.585578+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:57.962266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:57.962266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:18.210525+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":767,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-21T14:05:03.632248+0000","last_change":"2024-09-21T14:00:51.495433+0000","last_active":"2024-09-21T14:05:03.632248+0000","last_peered":"2024-09-21T14:05:03.632248+0000","last_clean":"2024-09-21T14:05:03.632248+0000","last_became_active":"2024-09-21T13:52:04.316016+0000","last_became_peered":"2024-09-21T13:52:04.316016+0000","last_unstale":"2024-09-21T14:05:03.632248+0000","last_undegraded":"2024-09-21T14:05:03.632248+0000","last_fullsized":"2024-09-21T14:05:03.632248+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:00:51.495368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:00:51.495368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"290'18","reported_seq":721,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.585635+0000","last_change":"2024-09-21T14:04:50.393599+0000","last_active":"2024-09-21T14:04:56.585635+0000","last_peered":"2024-09-21T14:04:56.585635+0000","last_clean":"2024-09-21T14:04:56.585635+0000","last_became_active":"2024-09-21T13:57:12.684454+0000","last_became_peered":"2024-09-21T13:57:12.684454+0000","last_unstale":"2024-09-21T14:04:56.585635+0000","last_undegraded":"2024-09-21T14:04:56.585635+0000","last_fullsized":"2024-09-21T14:04:56.585635+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'18","last_scrub_stamp":"2024-09-21T14:04:50.393462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:04:50.393462+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:00.435535+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":762,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-21T14:05:03.633008+0000","last_change":"2024-09-21T14:03:38.065257+0000","last_active":"2024-09-21T14:05:03.633008+0000","last_peered":"2024-09-21T14:05:03.633008+0000","last_clean":"2024-09-21T14:05:03.633008+0000","last_became_active":"2024-09-21T13:57:12.681897+0000","last_became_peered":"2024-09-21T13:57:12.681897+0000","last_unstale":"2024-09-21T14:05:03.633008+0000","last_undegraded":"2024-09-21T14:05:03.633008+0000","last_fullsized":"2024-09-21T14:05:03.633008+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:38.065196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:38.065196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":682,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.586112+0000","last_change":"2024-09-21T14:03:28.933216+0000","last_active":"2024-09-21T14:04:56.586112+0000","last_peered":"2024-09-21T14:04:56.586112+0000","last_clean":"2024-09-21T14:04:56.586112+0000","last_became_active":"2024-09-21T14:01:41.181946+0000","last_became_peered":"2024-09-21T14:01:41.181946+0000","last_unstale":"2024-09-21T14:04:56.586112+0000","last_undegraded":"2024-09-21T14:04:56.586112+0000","last_fullsized":"2024-09-21T14:04:56.586112+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T14:03:28.933143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:28.933143+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":786,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.579274+0000","last_change":"2024-09-21T14:04:21.215322+0000","last_active":"2024-09-21T14:04:56.579274+0000","last_peered":"2024-09-21T14:04:56.579274+0000","last_clean":"2024-09-21T14:04:56.579274+0000","last_became_active":"2024-09-21T14:04:21.200889+0000","last_became_peered":"2024-09-21T14:04:21.200889+0000","last_unstale":"2024-09-21T14:04:56.579274+0000","last_undegraded":"2024-09-21T14:04:56.579274+0000","last_fullsized":"2024-09-21T14:04:56.579274+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T14:01:58.994811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:01:58.994811+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"290'20","reported_seq":103,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.590045+0000","last_change":"2024-09-21T14:04:56.589169+0000","last_active":"2024-09-21T14:04:56.590045+0000","last_peered":"2024-09-21T14:04:56.590045+0000","last_clean":"2024-09-21T14:04:56.590045+0000","last_became_active":"2024-09-21T14:04:56.588859+0000","last_became_peered":"2024-09-21T14:04:56.588859+0000","last_unstale":"2024-09-21T14:04:56.590045+0000","last_undegraded":"2024-09-21T14:04:56.590045+0000","last_fullsized":"2024-09-21T14:04:56.590045+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'20","last_scrub_stamp":"2024-09-21T14:04:42.048082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:04:42.048082+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:51.310282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"409'13","reported_seq":432,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-21T14:05:03.637615+0000","last_change":"2024-09-21T14:04:21.222657+0000","last_active":"2024-09-21T14:05:03.637615+0000","last_peered":"2024-09-21T14:05:03.637615+0000","last_clean":"2024-09-21T14:05:03.637615+0000","last_became_active":"2024-09-21T14:04:21.206136+0000","last_became_peered":"2024-09-21T14:04:21.206136+0000","last_unstale":"2024-09-21T14:05:03.637615+0000","last_undegraded":"2024-09-21T14:05:03.637615+0000","last_fullsized":"2024-09-21T14:05:03.637615+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'12","last_scrub_stamp":"2024-09-21T13:59:36.012693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:36.012693+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:27.834286+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"419'9970","reported_seq":10131,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-21T14:05:03.768224+0000","last_change":"2024-09-21T14:04:09.971242+0000","last_active":"2024-09-21T14:05:03.768224+0000","last_peered":"2024-09-21T14:05:03.768224+0000","last_clean":"2024-09-21T14:05:03.768224+0000","last_became_active":"2024-09-21T14:02:56.559536+0000","last_became_peered":"2024-09-21T14:02:56.559536+0000","last_unstale":"2024-09-21T14:05:03.768224+0000","last_undegraded":"2024-09-21T14:05:03.768224+0000","last_fullsized":"2024-09-21T14:05:03.768224+0000","mapping_epoch":381,"log_start":"398'4500","ondisk_log_start":"398'4500","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"406'6478","last_scrub_stamp":"2024-09-21T14:04:09.971039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:04:09.971039+0000","objects_scrubbed":6417,"log_size":5470,"log_dups_size":0,"ondisk_log_size":5470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:17.197608+0000","scrub_duration":1835,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":410255360,"num_objects":6260,"num_object_clones":0,"num_object_copies":18780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9970,"num_write_kb":519360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"75'79","reported_seq":178,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.586172+0000","last_change":"2024-09-21T14:03:59.776621+0000","last_active":"2024-09-21T14:04:56.586172+0000","last_peered":"2024-09-21T14:04:56.586172+0000","last_clean":"2024-09-21T14:04:56.586172+0000","last_became_active":"2024-09-21T14:01:41.246888+0000","last_became_peered":"2024-09-21T14:01:41.246888+0000","last_unstale":"2024-09-21T14:04:56.586172+0000","last_undegraded":"2024-09-21T14:04:56.586172+0000","last_fullsized":"2024-09-21T14:04:56.586172+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:03:59.776552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:03:59.776552+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:21.450064+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"290'27","reported_seq":830,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.580811+0000","last_change":"2024-09-21T14:04:03.259647+0000","last_active":"2024-09-21T14:04:56.580811+0000","last_peered":"2024-09-21T14:04:56.580811+0000","last_clean":"2024-09-21T14:04:56.580811+0000","last_became_active":"2024-09-21T13:57:12.695609+0000","last_became_peered":"2024-09-21T13:57:12.695609+0000","last_unstale":"2024-09-21T14:04:56.580811+0000","last_undegraded":"2024-09-21T14:04:56.580811+0000","last_fullsized":"2024-09-21T14:04:56.580811+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'27","last_scrub_stamp":"2024-09-21T14:04:03.259554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:04:03.259554+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:20.677105+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":707,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.584274+0000","last_change":"2024-09-21T14:04:01.183193+0000","last_active":"2024-09-21T14:04:56.584274+0000","last_peered":"2024-09-21T14:04:56.584274+0000","last_clean":"2024-09-21T14:04:56.584274+0000","last_became_active":"2024-09-21T13:59:32.205350+0000","last_became_peered":"2024-09-21T13:59:32.205350+0000","last_unstale":"2024-09-21T14:04:56.584274+0000","last_undegraded":"2024-09-21T14:04:56.584274+0000","last_fullsized":"2024-09-21T14:04:56.584274+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:01.183123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:01.183123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:13.677443+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"409'11","reported_seq":812,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.586606+0000","last_change":"2024-09-21T14:04:21.211284+0000","last_active":"2024-09-21T14:04:56.586606+0000","last_peered":"2024-09-21T14:04:56.586606+0000","last_clean":"2024-09-21T14:04:56.586606+0000","last_became_active":"2024-09-21T14:04:21.206029+0000","last_became_peered":"2024-09-21T14:04:21.206029+0000","last_unstale":"2024-09-21T14:04:56.586606+0000","last_undegraded":"2024-09-21T14:04:56.586606+0000","last_fullsized":"2024-09-21T14:04:56.586606+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'10","last_scrub_stamp":"2024-09-21T14:02:59.831128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:02:59.831128+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":641,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.585824+0000","last_change":"2024-09-21T14:04:47.464272+0000","last_active":"2024-09-21T14:04:56.585824+0000","last_peered":"2024-09-21T14:04:56.585824+0000","last_clean":"2024-09-21T14:04:56.585824+0000","last_became_active":"2024-09-21T14:04:21.203505+0000","last_became_peered":"2024-09-21T14:04:21.203505+0000","last_unstale":"2024-09-21T14:04:56.585824+0000","last_undegraded":"2024-09-21T14:04:56.585824+0000","last_fullsized":"2024-09-21T14:04:56.585824+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:47.464207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:04:47.464207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:48.380565+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"419'9507","reported_seq":9664,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-21T14:05:03.777584+0000","last_change":"2024-09-21T14:04:08.894792+0000","last_active":"2024-09-21T14:05:03.777584+0000","last_peered":"2024-09-21T14:05:03.777584+0000","last_clean":"2024-09-21T14:05:03.777584+0000","last_became_active":"2024-09-21T14:02:56.562524+0000","last_became_peered":"2024-09-21T14:02:56.562524+0000","last_unstale":"2024-09-21T14:05:03.777584+0000","last_undegraded":"2024-09-21T14:05:03.777584+0000","last_fullsized":"2024-09-21T14:05:03.777584+0000","mapping_epoch":381,"log_start":"397'4000","ondisk_log_start":"397'4000","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"406'6081","last_scrub_stamp":"2024-09-21T14:04:08.894653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:04:08.894653+0000","objects_scrubbed":6008,"log_size":5507,"log_dups_size":0,"ondisk_log_size":5507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:26.881293+0000","scrub_duration":1756,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":399966208,"num_objects":6103,"num_object_clones":0,"num_object_copies":18309,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9507,"num_write_kb":499520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"75'79","reported_seq":226,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-21T14:05:03.631784+0000","last_change":"2024-09-21T14:02:06.395083+0000","last_active":"2024-09-21T14:05:03.631784+0000","last_peered":"2024-09-21T14:05:03.631784+0000","last_clean":"2024-09-21T14:05:03.631784+0000","last_became_active":"2024-09-21T14:00:41.137756+0000","last_became_peered":"2024-09-21T14:00:41.137756+0000","last_unstale":"2024-09-21T14:05:03.631784+0000","last_undegraded":"2024-09-21T14:05:03.631784+0000","last_fullsized":"2024-09-21T14:05:03.631784+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:02:06.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:02:06.395011+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"290'30","reported_seq":748,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.584379+0000","last_change":"2024-09-21T14:04:21.228589+0000","last_active":"2024-09-21T14:04:56.584379+0000","last_peered":"2024-09-21T14:04:56.584379+0000","last_clean":"2024-09-21T14:04:56.584379+0000","last_became_active":"2024-09-21T14:04:21.208982+0000","last_became_peered":"2024-09-21T14:04:21.208982+0000","last_unstale":"2024-09-21T14:04:56.584379+0000","last_undegraded":"2024-09-21T14:04:56.584379+0000","last_fullsized":"2024-09-21T14:04:56.584379+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'18","last_scrub_stamp":"2024-09-21T13:54:19.001030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:19.001030+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:23.192760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":659,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.585883+0000","last_change":"2024-09-21T14:04:02.214555+0000","last_active":"2024-09-21T14:04:56.585883+0000","last_peered":"2024-09-21T14:04:56.585883+0000","last_clean":"2024-09-21T14:04:56.585883+0000","last_became_active":"2024-09-21T13:59:32.211004+0000","last_became_peered":"2024-09-21T13:59:32.211004+0000","last_unstale":"2024-09-21T14:04:56.585883+0000","last_undegraded":"2024-09-21T14:04:56.585883+0000","last_fullsized":"2024-09-21T14:04:56.585883+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:02.214464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:02.214464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:04.300176+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":771,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-21T14:05:03.631800+0000","last_change":"2024-09-21T13:59:34.293330+0000","last_active":"2024-09-21T14:05:03.631800+0000","last_peered":"2024-09-21T14:05:03.631800+0000","last_clean":"2024-09-21T14:05:03.631800+0000","last_became_active":"2024-09-21T13:59:32.220304+0000","last_became_peered":"2024-09-21T13:59:32.220304+0000","last_unstale":"2024-09-21T14:05:03.631800+0000","last_undegraded":"2024-09-21T14:05:03.631800+0000","last_fullsized":"2024-09-21T14:05:03.631800+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:34.292892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:34.292892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":405,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.584240+0000","last_change":"2024-09-21T14:04:21.210131+0000","last_active":"2024-09-21T14:04:56.584240+0000","last_peered":"2024-09-21T14:04:56.584240+0000","last_clean":"2024-09-21T14:04:56.584240+0000","last_became_active":"2024-09-21T14:04:21.207782+0000","last_became_peered":"2024-09-21T14:04:21.207782+0000","last_unstale":"2024-09-21T14:04:56.584240+0000","last_undegraded":"2024-09-21T14:04:56.584240+0000","last_fullsized":"2024-09-21T14:04:56.584240+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:23.334162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:23.334162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"419'9928","reported_seq":10101,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-21T14:05:03.681484+0000","last_change":"2024-09-21T14:04:42.544112+0000","last_active":"2024-09-21T14:05:03.681484+0000","last_peered":"2024-09-21T14:05:03.681484+0000","last_clean":"2024-09-21T14:05:03.681484+0000","last_became_active":"2024-09-21T14:02:56.557055+0000","last_became_peered":"2024-09-21T14:02:56.557055+0000","last_unstale":"2024-09-21T14:05:03.681484+0000","last_undegraded":"2024-09-21T14:05:03.681484+0000","last_fullsized":"2024-09-21T14:05:03.681484+0000","mapping_epoch":381,"log_start":"398'4810","ondisk_log_start":"398'4810","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'8494","last_scrub_stamp":"2024-09-21T14:04:42.543986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:04:42.543986+0000","objects_scrubbed":7689,"log_size":5118,"log_dups_size":0,"ondisk_log_size":5118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:45.816257+0000","scrub_duration":2665,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404488192,"num_objects":6172,"num_object_clones":0,"num_object_copies":18516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9928,"num_write_kb":515200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"75'79","reported_seq":235,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.584119+0000","last_change":"2024-09-21T14:04:49.393518+0000","last_active":"2024-09-21T14:04:56.584119+0000","last_peered":"2024-09-21T14:04:56.584119+0000","last_clean":"2024-09-21T14:04:56.584119+0000","last_became_active":"2024-09-21T14:01:42.173042+0000","last_became_peered":"2024-09-21T14:01:42.173042+0000","last_unstale":"2024-09-21T14:04:56.584119+0000","last_undegraded":"2024-09-21T14:04:56.584119+0000","last_fullsized":"2024-09-21T14:04:56.584119+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:04:49.393415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:04:49.393415+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:56.538057+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"290'28","reported_seq":868,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-21T14:05:03.631894+0000","last_change":"2024-09-21T14:03:27.047111+0000","last_active":"2024-09-21T14:05:03.631894+0000","last_peered":"2024-09-21T14:05:03.631894+0000","last_clean":"2024-09-21T14:05:03.631894+0000","last_became_active":"2024-09-21T14:01:41.200859+0000","last_became_peered":"2024-09-21T14:01:41.200859+0000","last_unstale":"2024-09-21T14:05:03.631894+0000","last_undegraded":"2024-09-21T14:05:03.631894+0000","last_fullsized":"2024-09-21T14:05:03.631894+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'28","last_scrub_stamp":"2024-09-21T14:03:27.046906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:03:27.046906+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":44,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.580793+0000","last_change":"2024-09-21T14:04:21.216890+0000","last_active":"2024-09-21T14:04:56.580793+0000","last_peered":"2024-09-21T14:04:56.580793+0000","last_clean":"2024-09-21T14:04:56.580793+0000","last_became_active":"2024-09-21T14:04:21.205354+0000","last_became_peered":"2024-09-21T14:04:21.205354+0000","last_unstale":"2024-09-21T14:04:56.580793+0000","last_undegraded":"2024-09-21T14:04:56.580793+0000","last_fullsized":"2024-09-21T14:04:56.580793+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:57:38.046627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:38.046627+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"241'10","reported_seq":833,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.584609+0000","last_change":"2024-09-21T14:03:03.228551+0000","last_active":"2024-09-21T14:04:56.584609+0000","last_peered":"2024-09-21T14:04:56.584609+0000","last_clean":"2024-09-21T14:04:56.584609+0000","last_became_active":"2024-09-21T13:57:12.686651+0000","last_became_peered":"2024-09-21T13:57:12.686651+0000","last_unstale":"2024-09-21T14:04:56.584609+0000","last_undegraded":"2024-09-21T14:04:56.584609+0000","last_fullsized":"2024-09-21T14:04:56.584609+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'10","last_scrub_stamp":"2024-09-21T14:03:03.228443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:03.228443+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":723,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.584954+0000","last_change":"2024-09-21T14:04:45.055770+0000","last_active":"2024-09-21T14:04:56.584954+0000","last_peered":"2024-09-21T14:04:56.584954+0000","last_clean":"2024-09-21T14:04:56.584954+0000","last_became_active":"2024-09-21T13:59:32.209615+0000","last_became_peered":"2024-09-21T13:59:32.209615+0000","last_unstale":"2024-09-21T14:04:56.584954+0000","last_undegraded":"2024-09-21T14:04:56.584954+0000","last_fullsized":"2024-09-21T14:04:56.584954+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:45.055707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:04:45.055707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:03.598442+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.c","version":"418'9426","reported_seq":9509,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:05:01.250992+0000","last_change":"2024-09-21T14:02:56.563139+0000","last_active":"2024-09-21T14:05:01.250992+0000","last_peered":"2024-09-21T14:05:01.250992+0000","last_clean":"2024-09-21T14:05:01.250992+0000","last_became_active":"2024-09-21T14:02:56.562806+0000","last_became_peered":"2024-09-21T14:02:56.562806+0000","last_unstale":"2024-09-21T14:05:01.250992+0000","last_undegraded":"2024-09-21T14:05:01.250992+0000","last_fullsized":"2024-09-21T14:05:01.250992+0000","mapping_epoch":381,"log_start":"400'4909","ondisk_log_start":"400'4909","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":4517,"log_dups_size":0,"ondisk_log_size":4517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":406585344,"num_objects":6204,"num_object_clones":0,"num_object_copies":18612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9426,"num_write_kb":500160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"75'79","reported_seq":807,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-21T14:05:03.637289+0000","last_change":"2024-09-21T14:04:38.868411+0000","last_active":"2024-09-21T14:05:03.637289+0000","last_peered":"2024-09-21T14:05:03.637289+0000","last_clean":"2024-09-21T14:05:03.637289+0000","last_became_active":"2024-09-21T14:00:41.083425+0000","last_became_peered":"2024-09-21T14:00:41.083425+0000","last_unstale":"2024-09-21T14:05:03.637289+0000","last_undegraded":"2024-09-21T14:05:03.637289+0000","last_fullsized":"2024-09-21T14:05:03.637289+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:04:38.868343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:04:38.868343+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:49.409572+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"290'20","reported_seq":90,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.581505+0000","last_change":"2024-09-21T14:04:42.776698+0000","last_active":"2024-09-21T14:04:56.581505+0000","last_peered":"2024-09-21T14:04:56.581505+0000","last_clean":"2024-09-21T14:04:56.581505+0000","last_became_active":"2024-09-21T14:03:29.158218+0000","last_became_peered":"2024-09-21T14:03:29.158218+0000","last_unstale":"2024-09-21T14:04:56.581505+0000","last_undegraded":"2024-09-21T14:04:56.581505+0000","last_fullsized":"2024-09-21T14:04:56.581505+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'20","last_scrub_stamp":"2024-09-21T14:04:42.776603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:04:42.776603+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:58.993778+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"61'1","reported_seq":696,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.585937+0000","last_change":"2024-09-21T14:04:48.425745+0000","last_active":"2024-09-21T14:04:56.585937+0000","last_peered":"2024-09-21T14:04:56.585937+0000","last_clean":"2024-09-21T14:04:56.585937+0000","last_became_active":"2024-09-21T14:01:41.195268+0000","last_became_peered":"2024-09-21T14:01:41.195268+0000","last_unstale":"2024-09-21T14:04:56.585937+0000","last_undegraded":"2024-09-21T14:04:56.585937+0000","last_fullsized":"2024-09-21T14:04:56.585937+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T14:04:48.425665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:48.425665+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:14.101369+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"418'16","reported_seq":108,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:58.323724+0000","last_change":"2024-09-21T14:04:56.590159+0000","last_active":"2024-09-21T14:04:58.323724+0000","last_peered":"2024-09-21T14:04:58.323724+0000","last_clean":"2024-09-21T14:04:58.323724+0000","last_became_active":"2024-09-21T14:04:56.589828+0000","last_became_peered":"2024-09-21T14:04:56.589828+0000","last_unstale":"2024-09-21T14:04:58.323724+0000","last_undegraded":"2024-09-21T14:04:58.323724+0000","last_fullsized":"2024-09-21T14:04:58.323724+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'15","last_scrub_stamp":"2024-09-21T14:04:43.936597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:04:43.936597+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:47.698681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":207,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.601639+0000","last_change":"2024-09-21T14:04:56.587813+0000","last_active":"2024-09-21T14:04:56.601639+0000","last_peered":"2024-09-21T14:04:56.601639+0000","last_clean":"2024-09-21T14:04:56.601639+0000","last_became_active":"2024-09-21T14:04:56.586891+0000","last_became_peered":"2024-09-21T14:04:56.586891+0000","last_unstale":"2024-09-21T14:04:56.601639+0000","last_undegraded":"2024-09-21T14:04:56.601639+0000","last_fullsized":"2024-09-21T14:04:56.601639+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:41.400346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:04:41.400346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:41.941073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.d","version":"419'9792","reported_seq":12847,"reported_epoch":419,"state":"active+remapped+backfilling","last_fresh":"2024-09-21T14:05:03.732984+0000","last_change":"2024-09-21T14:04:52.308462+0000","last_active":"2024-09-21T14:05:03.732984+0000","last_peered":"2024-09-21T14:05:03.732984+0000","last_clean":"2024-09-21T14:04:20.166030+0000","last_became_active":"2024-09-21T14:04:22.281826+0000","last_became_peered":"2024-09-21T14:04:22.281826+0000","last_unstale":"2024-09-21T14:05:03.732984+0000","last_undegraded":"2024-09-21T14:05:03.732984+0000","last_fullsized":"2024-09-21T14:05:03.732984+0000","mapping_epoch":409,"log_start":"398'4668","ondisk_log_start":"398'4668","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"406'6403","last_scrub_stamp":"2024-09-21T14:04:11.019087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:04:11.019087+0000","objects_scrubbed":6346,"log_size":5124,"log_dups_size":0,"ondisk_log_size":5124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1942,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":405471256,"num_objects":6188,"num_object_clones":0,"num_object_copies":18564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4828,"num_objects_unfound":0,"num_objects_dirty":6188,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9792,"num_write_kb":511297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1444,"num_bytes_recovered":94633984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,0],"avail_no_missing":["6","0","4"],"object_location_counts":[{"shards":"0,4,6","objects":6188}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"75'79","reported_seq":815,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-21T14:05:03.636127+0000","last_change":"2024-09-21T14:04:44.986902+0000","last_active":"2024-09-21T14:05:03.636127+0000","last_peered":"2024-09-21T14:05:03.636127+0000","last_clean":"2024-09-21T14:05:03.636127+0000","last_became_active":"2024-09-21T14:00:41.073047+0000","last_became_peered":"2024-09-21T14:00:41.073047+0000","last_unstale":"2024-09-21T14:05:03.636127+0000","last_undegraded":"2024-09-21T14:05:03.636127+0000","last_fullsized":"2024-09-21T14:05:03.636127+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:04:44.986816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:04:44.986816+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:58.169458+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"290'29","reported_seq":508,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.590529+0000","last_change":"2024-09-21T14:04:56.589405+0000","last_active":"2024-09-21T14:04:56.590529+0000","last_peered":"2024-09-21T14:04:56.590529+0000","last_clean":"2024-09-21T14:04:56.590529+0000","last_became_active":"2024-09-21T14:04:56.589018+0000","last_became_peered":"2024-09-21T14:04:56.589018+0000","last_unstale":"2024-09-21T14:04:56.590529+0000","last_undegraded":"2024-09-21T14:04:56.590529+0000","last_fullsized":"2024-09-21T14:04:56.590529+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'29","last_scrub_stamp":"2024-09-21T14:04:53.477766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:04:53.477766+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:12.011852+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":770,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.580609+0000","last_change":"2024-09-21T14:01:41.185706+0000","last_active":"2024-09-21T14:04:56.580609+0000","last_peered":"2024-09-21T14:04:56.580609+0000","last_clean":"2024-09-21T14:04:56.580609+0000","last_became_active":"2024-09-21T14:01:41.185400+0000","last_became_peered":"2024-09-21T14:01:41.185400+0000","last_unstale":"2024-09-21T14:04:56.580609+0000","last_undegraded":"2024-09-21T14:04:56.580609+0000","last_fullsized":"2024-09-21T14:04:56.580609+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:16.712990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:16.712990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":356,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-21T14:05:03.636733+0000","last_change":"2024-09-21T14:04:03.070260+0000","last_active":"2024-09-21T14:05:03.636733+0000","last_peered":"2024-09-21T14:05:03.636733+0000","last_clean":"2024-09-21T14:05:03.636733+0000","last_became_active":"2024-09-21T14:01:41.193973+0000","last_became_peered":"2024-09-21T14:01:41.193973+0000","last_unstale":"2024-09-21T14:05:03.636733+0000","last_undegraded":"2024-09-21T14:05:03.636733+0000","last_fullsized":"2024-09-21T14:05:03.636733+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:03.069973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:04:03.069973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:26.954137+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":761,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.579058+0000","last_change":"2024-09-21T13:58:43.952014+0000","last_active":"2024-09-21T14:04:56.579058+0000","last_peered":"2024-09-21T14:04:56.579058+0000","last_clean":"2024-09-21T14:04:56.579058+0000","last_became_active":"2024-09-21T13:58:17.757628+0000","last_became_peered":"2024-09-21T13:58:17.757628+0000","last_unstale":"2024-09-21T14:04:56.579058+0000","last_undegraded":"2024-09-21T14:04:56.579058+0000","last_fullsized":"2024-09-21T14:04:56.579058+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:43.951941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:43.951941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"419'9735","reported_seq":12935,"reported_epoch":419,"state":"active+remapped+backfilling","last_fresh":"2024-09-21T14:05:03.731659+0000","last_change":"2024-09-21T14:04:52.389607+0000","last_active":"2024-09-21T14:05:03.731659+0000","last_peered":"2024-09-21T14:05:03.731659+0000","last_clean":"2024-09-21T14:04:20.165921+0000","last_became_active":"2024-09-21T14:04:22.361054+0000","last_became_peered":"2024-09-21T14:04:22.361054+0000","last_unstale":"2024-09-21T14:05:03.731659+0000","last_undegraded":"2024-09-21T14:05:03.731659+0000","last_fullsized":"2024-09-21T14:05:03.731659+0000","mapping_epoch":409,"log_start":"398'4648","ondisk_log_start":"398'4648","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":3149,"log_size":5087,"log_dups_size":0,"ondisk_log_size":5087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":397869056,"num_objects":6071,"num_object_clones":0,"num_object_copies":18213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4636,"num_objects_unfound":0,"num_objects_dirty":6071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9735,"num_write_kb":505792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1534,"num_bytes_recovered":100532224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,0],"avail_no_missing":["6","0","5"],"object_location_counts":[{"shards":"0,5,6","objects":6071}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"75'79","reported_seq":170,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.586010+0000","last_change":"2024-09-21T14:01:41.199377+0000","last_active":"2024-09-21T14:04:56.586010+0000","last_peered":"2024-09-21T14:04:56.586010+0000","last_clean":"2024-09-21T14:04:56.586010+0000","last_became_active":"2024-09-21T14:01:41.198750+0000","last_became_peered":"2024-09-21T14:01:41.198750+0000","last_unstale":"2024-09-21T14:04:56.586010+0000","last_undegraded":"2024-09-21T14:04:56.586010+0000","last_fullsized":"2024-09-21T14:04:56.586010+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"290'44","reported_seq":1037,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.580522+0000","last_change":"2024-09-21T14:02:57.316829+0000","last_active":"2024-09-21T14:04:56.580522+0000","last_peered":"2024-09-21T14:04:56.580522+0000","last_clean":"2024-09-21T14:04:56.580522+0000","last_became_active":"2024-09-21T13:57:12.684690+0000","last_became_peered":"2024-09-21T13:57:12.684690+0000","last_unstale":"2024-09-21T14:04:56.580522+0000","last_undegraded":"2024-09-21T14:04:56.580522+0000","last_fullsized":"2024-09-21T14:04:56.580522+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'44","last_scrub_stamp":"2024-09-21T14:02:57.316763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:57.316763+0000","objects_scrubbed":10,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":787,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-21T14:05:03.631256+0000","last_change":"2024-09-21T14:04:11.158776+0000","last_active":"2024-09-21T14:05:03.631256+0000","last_peered":"2024-09-21T14:05:03.631256+0000","last_clean":"2024-09-21T14:05:03.631256+0000","last_became_active":"2024-09-21T13:58:17.770937+0000","last_became_peered":"2024-09-21T13:58:17.770937+0000","last_unstale":"2024-09-21T14:05:03.631256+0000","last_undegraded":"2024-09-21T14:05:03.631256+0000","last_fullsized":"2024-09-21T14:05:03.631256+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:11.158703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:11.158703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:26.630567+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":715,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.586044+0000","last_change":"2024-09-21T14:03:30.390161+0000","last_active":"2024-09-21T14:04:56.586044+0000","last_peered":"2024-09-21T14:04:56.586044+0000","last_clean":"2024-09-21T14:04:56.586044+0000","last_became_active":"2024-09-21T14:01:41.247642+0000","last_became_peered":"2024-09-21T14:01:41.247642+0000","last_unstale":"2024-09-21T14:04:56.586044+0000","last_undegraded":"2024-09-21T14:04:56.586044+0000","last_fullsized":"2024-09-21T14:04:56.586044+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-21T14:03:30.390086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T14:03:30.390086+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":695,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.580374+0000","last_change":"2024-09-21T14:04:21.204705+0000","last_active":"2024-09-21T14:04:56.580374+0000","last_peered":"2024-09-21T14:04:56.580374+0000","last_clean":"2024-09-21T14:04:56.580374+0000","last_became_active":"2024-09-21T14:04:21.204315+0000","last_became_peered":"2024-09-21T14:04:21.204315+0000","last_unstale":"2024-09-21T14:04:56.580374+0000","last_undegraded":"2024-09-21T14:04:56.580374+0000","last_fullsized":"2024-09-21T14:04:56.580374+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:07.021059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:55:07.021059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":686,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.585867+0000","last_change":"2024-09-21T14:03:02.238409+0000","last_active":"2024-09-21T14:04:56.585867+0000","last_peered":"2024-09-21T14:04:56.585867+0000","last_clean":"2024-09-21T14:04:56.585867+0000","last_became_active":"2024-09-21T13:59:32.208980+0000","last_became_peered":"2024-09-21T13:59:32.208980+0000","last_unstale":"2024-09-21T14:04:56.585867+0000","last_undegraded":"2024-09-21T14:04:56.585867+0000","last_fullsized":"2024-09-21T14:04:56.585867+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:02.238357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:02.238357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"418'9474","reported_seq":9627,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:05:00.915592+0000","last_change":"2024-09-21T14:04:06.243298+0000","last_active":"2024-09-21T14:05:00.915592+0000","last_peered":"2024-09-21T14:05:00.915592+0000","last_clean":"2024-09-21T14:05:00.915592+0000","last_became_active":"2024-09-21T14:02:56.558773+0000","last_became_peered":"2024-09-21T14:02:56.558773+0000","last_unstale":"2024-09-21T14:05:00.915592+0000","last_undegraded":"2024-09-21T14:05:00.915592+0000","last_fullsized":"2024-09-21T14:05:00.915592+0000","mapping_epoch":381,"log_start":"398'4569","ondisk_log_start":"398'4569","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'5948","last_scrub_stamp":"2024-09-21T14:04:06.243212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:04:06.243212+0000","objects_scrubbed":5906,"log_size":4905,"log_dups_size":0,"ondisk_log_size":4905,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:14.772338+0000","scrub_duration":1458,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":409206784,"num_objects":6244,"num_object_clones":0,"num_object_copies":18732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9474,"num_write_kb":502976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"75'79","reported_seq":810,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.584190+0000","last_change":"2024-09-21T14:04:21.206885+0000","last_active":"2024-09-21T14:04:56.584190+0000","last_peered":"2024-09-21T14:04:56.584190+0000","last_clean":"2024-09-21T14:04:56.584190+0000","last_became_active":"2024-09-21T14:04:21.206472+0000","last_became_peered":"2024-09-21T14:04:21.206472+0000","last_unstale":"2024-09-21T14:04:56.584190+0000","last_undegraded":"2024-09-21T14:04:56.584190+0000","last_fullsized":"2024-09-21T14:04:56.584190+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:05:23.132366+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"290'17","reported_seq":815,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.580898+0000","last_change":"2024-09-21T14:04:21.224066+0000","last_active":"2024-09-21T14:04:56.580898+0000","last_peered":"2024-09-21T14:04:56.580898+0000","last_clean":"2024-09-21T14:04:56.580898+0000","last_became_active":"2024-09-21T14:04:21.205738+0000","last_became_peered":"2024-09-21T14:04:21.205738+0000","last_unstale":"2024-09-21T14:04:56.580898+0000","last_undegraded":"2024-09-21T14:04:56.580898+0000","last_fullsized":"2024-09-21T14:04:56.580898+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'13","last_scrub_stamp":"2024-09-21T13:58:11.619848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:11.619848+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":813,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.580874+0000","last_change":"2024-09-21T14:04:10.193524+0000","last_active":"2024-09-21T14:04:56.580874+0000","last_peered":"2024-09-21T14:04:56.580874+0000","last_clean":"2024-09-21T14:04:56.580874+0000","last_became_active":"2024-09-21T13:58:17.759671+0000","last_became_peered":"2024-09-21T13:58:17.759671+0000","last_unstale":"2024-09-21T14:04:56.580874+0000","last_undegraded":"2024-09-21T14:04:56.580874+0000","last_fullsized":"2024-09-21T14:04:56.580874+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:10.193318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:10.193318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:25.269486+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":687,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.581345+0000","last_change":"2024-09-21T13:58:41.644876+0000","last_active":"2024-09-21T14:04:56.581345+0000","last_peered":"2024-09-21T14:04:56.581345+0000","last_clean":"2024-09-21T14:04:56.581345+0000","last_became_active":"2024-09-21T13:57:12.682147+0000","last_became_peered":"2024-09-21T13:57:12.682147+0000","last_unstale":"2024-09-21T14:04:56.581345+0000","last_undegraded":"2024-09-21T14:04:56.581345+0000","last_fullsized":"2024-09-21T14:04:56.581345+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:58:41.644789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:58:41.644789+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:17.567651+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":666,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.584202+0000","last_change":"2024-09-21T14:01:50.880840+0000","last_active":"2024-09-21T14:04:56.584202+0000","last_peered":"2024-09-21T14:04:56.584202+0000","last_clean":"2024-09-21T14:04:56.584202+0000","last_became_active":"2024-09-21T14:01:41.182000+0000","last_became_peered":"2024-09-21T14:01:41.182000+0000","last_unstale":"2024-09-21T14:04:56.584202+0000","last_undegraded":"2024-09-21T14:04:56.584202+0000","last_fullsized":"2024-09-21T14:04:56.584202+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:50.880471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:01:50.880471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:20.548012+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":723,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.585247+0000","last_change":"2024-09-21T13:59:32.222791+0000","last_active":"2024-09-21T14:04:56.585247+0000","last_peered":"2024-09-21T14:04:56.585247+0000","last_clean":"2024-09-21T14:04:56.585247+0000","last_became_active":"2024-09-21T13:59:32.221502+0000","last_became_peered":"2024-09-21T13:59:32.221502+0000","last_unstale":"2024-09-21T14:04:56.585247+0000","last_undegraded":"2024-09-21T14:04:56.585247+0000","last_fullsized":"2024-09-21T14:04:56.585247+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:14.673582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:14.673582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"418'9551","reported_seq":9709,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:59.491182+0000","last_change":"2024-09-21T14:04:56.601777+0000","last_active":"2024-09-21T14:04:59.491182+0000","last_peered":"2024-09-21T14:04:59.491182+0000","last_clean":"2024-09-21T14:04:59.491182+0000","last_became_active":"2024-09-21T14:04:56.586861+0000","last_became_peered":"2024-09-21T14:04:56.586861+0000","last_unstale":"2024-09-21T14:04:59.491182+0000","last_undegraded":"2024-09-21T14:04:59.491182+0000","last_fullsized":"2024-09-21T14:04:59.491182+0000","mapping_epoch":417,"log_start":"398'4382","ondisk_log_start":"398'4382","created":381,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'6143","last_scrub_stamp":"2024-09-21T14:04:06.996122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:04:06.996122+0000","objects_scrubbed":6071,"log_size":5169,"log_dups_size":0,"ondisk_log_size":5169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:36.829006+0000","scrub_duration":2407,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":429588480,"num_objects":6555,"num_object_clones":0,"num_object_copies":19665,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9551,"num_write_kb":515392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"75'79","reported_seq":215,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.579898+0000","last_change":"2024-09-21T14:04:21.211219+0000","last_active":"2024-09-21T14:04:56.579898+0000","last_peered":"2024-09-21T14:04:56.579898+0000","last_clean":"2024-09-21T14:04:56.579898+0000","last_became_active":"2024-09-21T14:04:21.210586+0000","last_became_peered":"2024-09-21T14:04:21.210586+0000","last_unstale":"2024-09-21T14:04:56.579898+0000","last_undegraded":"2024-09-21T14:04:56.579898+0000","last_fullsized":"2024-09-21T14:04:56.579898+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"290'24","reported_seq":612,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.586688+0000","last_change":"2024-09-21T14:04:21.221906+0000","last_active":"2024-09-21T14:04:56.586688+0000","last_peered":"2024-09-21T14:04:56.586688+0000","last_clean":"2024-09-21T14:04:56.586688+0000","last_became_active":"2024-09-21T14:04:21.206881+0000","last_became_peered":"2024-09-21T14:04:21.206881+0000","last_unstale":"2024-09-21T14:04:56.586688+0000","last_undegraded":"2024-09-21T14:04:56.586688+0000","last_fullsized":"2024-09-21T14:04:56.586688+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'24","last_scrub_stamp":"2024-09-21T14:02:57.559936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:57.559936+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":331,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.590025+0000","last_change":"2024-09-21T14:04:56.588103+0000","last_active":"2024-09-21T14:04:56.590025+0000","last_peered":"2024-09-21T14:04:56.590025+0000","last_clean":"2024-09-21T14:04:56.590025+0000","last_became_active":"2024-09-21T14:04:56.587384+0000","last_became_peered":"2024-09-21T14:04:56.587384+0000","last_unstale":"2024-09-21T14:04:56.590025+0000","last_undegraded":"2024-09-21T14:04:56.590025+0000","last_fullsized":"2024-09-21T14:04:56.590025+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:59.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:59:59.450546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:58.054530+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":1586,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:59.742227+0000","last_change":"2024-09-21T13:59:32.218796+0000","last_active":"2024-09-21T14:04:59.742227+0000","last_peered":"2024-09-21T14:04:59.742227+0000","last_clean":"2024-09-21T14:04:59.742227+0000","last_became_active":"2024-09-21T13:59:32.203361+0000","last_became_peered":"2024-09-21T13:59:32.203361+0000","last_unstale":"2024-09-21T14:04:59.742227+0000","last_undegraded":"2024-09-21T14:04:59.742227+0000","last_fullsized":"2024-09-21T14:04:59.742227+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-21T13:56:29.617143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:56:29.617143+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:31.086108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047048000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":335,"num_read_kb":330,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":686,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.587037+0000","last_change":"2024-09-21T14:03:06.470855+0000","last_active":"2024-09-21T14:04:56.587037+0000","last_peered":"2024-09-21T14:04:56.587037+0000","last_clean":"2024-09-21T14:04:56.587037+0000","last_became_active":"2024-09-21T14:01:41.200531+0000","last_became_peered":"2024-09-21T14:01:41.200531+0000","last_unstale":"2024-09-21T14:04:56.587037+0000","last_undegraded":"2024-09-21T14:04:56.587037+0000","last_fullsized":"2024-09-21T14:04:56.587037+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:06.470705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:06.470705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":790,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.580160+0000","last_change":"2024-09-21T14:03:04.357026+0000","last_active":"2024-09-21T14:04:56.580160+0000","last_peered":"2024-09-21T14:04:56.580160+0000","last_clean":"2024-09-21T14:04:56.580160+0000","last_became_active":"2024-09-21T14:01:41.184664+0000","last_became_peered":"2024-09-21T14:01:41.184664+0000","last_unstale":"2024-09-21T14:04:56.580160+0000","last_undegraded":"2024-09-21T14:04:56.580160+0000","last_fullsized":"2024-09-21T14:04:56.580160+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:04.356979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:04.356979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"419'9707","reported_seq":9864,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-21T14:05:03.719123+0000","last_change":"2024-09-21T14:04:08.011633+0000","last_active":"2024-09-21T14:05:03.719123+0000","last_peered":"2024-09-21T14:05:03.719123+0000","last_clean":"2024-09-21T14:05:03.719123+0000","last_became_active":"2024-09-21T14:02:56.557263+0000","last_became_peered":"2024-09-21T14:02:56.557263+0000","last_unstale":"2024-09-21T14:05:03.719123+0000","last_undegraded":"2024-09-21T14:05:03.719123+0000","last_fullsized":"2024-09-21T14:05:03.719123+0000","mapping_epoch":381,"log_start":"398'4619","ondisk_log_start":"398'4619","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"406'6147","last_scrub_stamp":"2024-09-21T14:04:08.011556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:04:08.011556+0000","objects_scrubbed":6074,"log_size":5088,"log_dups_size":0,"ondisk_log_size":5088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:08.505662+0000","scrub_duration":1868,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":406913024,"num_objects":6209,"num_object_clones":0,"num_object_copies":18627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9707,"num_write_kb":509312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"300'32","reported_seq":466,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-21T14:05:03.637425+0000","last_change":"2024-09-21T14:00:47.130089+0000","last_active":"2024-09-21T14:05:03.637425+0000","last_peered":"2024-09-21T14:05:03.637425+0000","last_clean":"2024-09-21T14:05:03.637425+0000","last_became_active":"2024-09-21T13:59:32.204624+0000","last_became_peered":"2024-09-21T13:59:32.204624+0000","last_unstale":"2024-09-21T14:05:03.637425+0000","last_undegraded":"2024-09-21T14:05:03.637425+0000","last_fullsized":"2024-09-21T14:05:03.637425+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'32","last_scrub_stamp":"2024-09-21T14:00:47.129789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:00:47.129789+0000","objects_scrubbed":8,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:20.900768+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":744,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.584039+0000","last_change":"2024-09-21T14:04:42.390182+0000","last_active":"2024-09-21T14:04:56.584039+0000","last_peered":"2024-09-21T14:04:56.584039+0000","last_clean":"2024-09-21T14:04:56.584039+0000","last_became_active":"2024-09-21T14:01:41.190210+0000","last_became_peered":"2024-09-21T14:01:41.190210+0000","last_unstale":"2024-09-21T14:04:56.584039+0000","last_undegraded":"2024-09-21T14:04:56.584039+0000","last_fullsized":"2024-09-21T14:04:56.584039+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:42.390102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:42.390102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:10.628828+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"75'79","reported_seq":804,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.584442+0000","last_change":"2024-09-21T14:04:21.205378+0000","last_active":"2024-09-21T14:04:56.584442+0000","last_peered":"2024-09-21T14:04:56.584442+0000","last_clean":"2024-09-21T14:04:56.584442+0000","last_became_active":"2024-09-21T14:04:21.204537+0000","last_became_peered":"2024-09-21T14:04:21.204537+0000","last_unstale":"2024-09-21T14:04:56.584442+0000","last_undegraded":"2024-09-21T14:04:56.584442+0000","last_fullsized":"2024-09-21T14:04:56.584442+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:05:22.117825+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":19,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53669,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":711,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.586117+0000","last_change":"2024-09-21T14:01:41.182186+0000","last_active":"2024-09-21T14:04:56.586117+0000","last_peered":"2024-09-21T14:04:56.586117+0000","last_clean":"2024-09-21T14:04:56.586117+0000","last_became_active":"2024-09-21T14:01:41.181899+0000","last_became_peered":"2024-09-21T14:01:41.181899+0000","last_unstale":"2024-09-21T14:04:56.586117+0000","last_undegraded":"2024-09-21T14:04:56.586117+0000","last_fullsized":"2024-09-21T14:04:56.586117+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:15.700409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:15.700409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":651,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.586806+0000","last_change":"2024-09-21T14:04:01.760941+0000","last_active":"2024-09-21T14:04:56.586806+0000","last_peered":"2024-09-21T14:04:56.586806+0000","last_clean":"2024-09-21T14:04:56.586806+0000","last_became_active":"2024-09-21T14:03:57.346453+0000","last_became_peered":"2024-09-21T14:03:57.346453+0000","last_unstale":"2024-09-21T14:04:56.586806+0000","last_undegraded":"2024-09-21T14:04:56.586806+0000","last_fullsized":"2024-09-21T14:04:56.586806+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:01.760755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:04:01.760755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:08.954955+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"419'9820","reported_seq":14778,"reported_epoch":419,"state":"active+remapped+backfilling","last_fresh":"2024-09-21T14:05:03.731980+0000","last_change":"2024-09-21T14:04:52.362441+0000","last_active":"2024-09-21T14:05:03.731980+0000","last_peered":"2024-09-21T14:05:03.731980+0000","last_clean":"2024-09-21T14:04:20.179742+0000","last_became_active":"2024-09-21T14:04:22.332419+0000","last_became_peered":"2024-09-21T14:04:22.332419+0000","last_unstale":"2024-09-21T14:05:03.731980+0000","last_undegraded":"2024-09-21T14:05:03.731980+0000","last_fullsized":"2024-09-21T14:05:03.731980+0000","mapping_epoch":409,"log_start":"398'4636","ondisk_log_start":"398'4636","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"406'6598","last_scrub_stamp":"2024-09-21T14:04:14.205496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:04:14.205496+0000","objects_scrubbed":6523,"log_size":5184,"log_dups_size":0,"ondisk_log_size":5184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402784256,"num_objects":6146,"num_object_clones":0,"num_object_copies":18438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3892,"num_objects_unfound":0,"num_objects_dirty":6146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9820,"num_write_kb":510912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2393,"num_bytes_recovered":156827648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,4],"avail_no_missing":["6","1","4"],"object_location_counts":[{"shards":"1,4,6","objects":6146}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"290'19","reported_seq":825,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-21T14:05:03.632236+0000","last_change":"2024-09-21T14:02:28.553549+0000","last_active":"2024-09-21T14:05:03.632236+0000","last_peered":"2024-09-21T14:05:03.632236+0000","last_clean":"2024-09-21T14:05:03.632236+0000","last_became_active":"2024-09-21T13:58:17.776310+0000","last_became_peered":"2024-09-21T13:58:17.776310+0000","last_unstale":"2024-09-21T14:05:03.632236+0000","last_undegraded":"2024-09-21T14:05:03.632236+0000","last_fullsized":"2024-09-21T14:05:03.632236+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'19","last_scrub_stamp":"2024-09-21T14:02:28.553065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:28.553065+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":795,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-21T14:05:03.632049+0000","last_change":"2024-09-21T14:03:32.957705+0000","last_active":"2024-09-21T14:05:03.632049+0000","last_peered":"2024-09-21T14:05:03.632049+0000","last_clean":"2024-09-21T14:05:03.632049+0000","last_became_active":"2024-09-21T13:59:32.223247+0000","last_became_peered":"2024-09-21T13:59:32.223247+0000","last_unstale":"2024-09-21T14:05:03.632049+0000","last_undegraded":"2024-09-21T14:05:03.632049+0000","last_fullsized":"2024-09-21T14:05:03.632049+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:32.957649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:03:32.957649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":720,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.586787+0000","last_change":"2024-09-21T14:04:54.483668+0000","last_active":"2024-09-21T14:04:56.586787+0000","last_peered":"2024-09-21T14:04:56.586787+0000","last_clean":"2024-09-21T14:04:56.586787+0000","last_became_active":"2024-09-21T14:01:41.193941+0000","last_became_peered":"2024-09-21T14:01:41.193941+0000","last_unstale":"2024-09-21T14:04:56.586787+0000","last_undegraded":"2024-09-21T14:04:56.586787+0000","last_fullsized":"2024-09-21T14:04:56.586787+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:54.483438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:04:54.483438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:21.139691+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":722,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.585467+0000","last_change":"2024-09-21T14:03:36.302360+0000","last_active":"2024-09-21T14:04:56.585467+0000","last_peered":"2024-09-21T14:04:56.585467+0000","last_clean":"2024-09-21T14:04:56.585467+0000","last_became_active":"2024-09-21T14:01:41.190192+0000","last_became_peered":"2024-09-21T14:01:41.190192+0000","last_unstale":"2024-09-21T14:04:56.585467+0000","last_undegraded":"2024-09-21T14:04:56.585467+0000","last_fullsized":"2024-09-21T14:04:56.585467+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:36.302281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:36.302281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"418'9695","reported_seq":9857,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:05:03.001720+0000","last_change":"2024-09-21T14:04:52.445772+0000","last_active":"2024-09-21T14:05:03.001720+0000","last_peered":"2024-09-21T14:05:03.001720+0000","last_clean":"2024-09-21T14:05:03.001720+0000","last_became_active":"2024-09-21T14:02:56.560070+0000","last_became_peered":"2024-09-21T14:02:56.560070+0000","last_unstale":"2024-09-21T14:05:03.001720+0000","last_undegraded":"2024-09-21T14:05:03.001720+0000","last_fullsized":"2024-09-21T14:05:03.001720+0000","mapping_epoch":381,"log_start":"398'4687","ondisk_log_start":"398'4687","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'9024","last_scrub_stamp":"2024-09-21T14:04:52.445618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:04:52.445618+0000","objects_scrubbed":6949,"log_size":5008,"log_dups_size":0,"ondisk_log_size":5008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:01.306398+0000","scrub_duration":2824,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":406126592,"num_objects":6197,"num_object_clones":0,"num_object_copies":18591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9695,"num_write_kb":508544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"75'79","reported_seq":185,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.585987+0000","last_change":"2024-09-21T14:04:51.454570+0000","last_active":"2024-09-21T14:04:56.585987+0000","last_peered":"2024-09-21T14:04:56.585987+0000","last_clean":"2024-09-21T14:04:56.585987+0000","last_became_active":"2024-09-21T14:01:41.188685+0000","last_became_peered":"2024-09-21T14:01:41.188685+0000","last_unstale":"2024-09-21T14:04:56.585987+0000","last_undegraded":"2024-09-21T14:04:56.585987+0000","last_fullsized":"2024-09-21T14:04:56.585987+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:04:51.454483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:04:51.454483+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:17.391460+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"290'31","reported_seq":599,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.582391+0000","last_change":"2024-09-21T14:04:39.741288+0000","last_active":"2024-09-21T14:04:56.582391+0000","last_peered":"2024-09-21T14:04:56.582391+0000","last_clean":"2024-09-21T14:04:56.582391+0000","last_became_active":"2024-09-21T14:04:21.204164+0000","last_became_peered":"2024-09-21T14:04:21.204164+0000","last_unstale":"2024-09-21T14:04:56.582391+0000","last_undegraded":"2024-09-21T14:04:56.582391+0000","last_fullsized":"2024-09-21T14:04:56.582391+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'31","last_scrub_stamp":"2024-09-21T14:04:39.740794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:04:39.740794+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:54.328250+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":337,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-21T14:05:03.636649+0000","last_change":"2024-09-21T14:04:51.139211+0000","last_active":"2024-09-21T14:05:03.636649+0000","last_peered":"2024-09-21T14:05:03.636649+0000","last_clean":"2024-09-21T14:05:03.636649+0000","last_became_active":"2024-09-21T13:58:17.762520+0000","last_became_peered":"2024-09-21T13:58:17.762520+0000","last_unstale":"2024-09-21T14:05:03.636649+0000","last_undegraded":"2024-09-21T14:05:03.636649+0000","last_fullsized":"2024-09-21T14:05:03.636649+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:51.139123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:51.139123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:51.167153+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":740,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.580259+0000","last_change":"2024-09-21T13:58:52.030800+0000","last_active":"2024-09-21T14:04:56.580259+0000","last_peered":"2024-09-21T14:04:56.580259+0000","last_clean":"2024-09-21T14:04:56.580259+0000","last_became_active":"2024-09-21T13:49:02.067472+0000","last_became_peered":"2024-09-21T13:49:02.067472+0000","last_unstale":"2024-09-21T14:04:56.580259+0000","last_undegraded":"2024-09-21T14:04:56.580259+0000","last_fullsized":"2024-09-21T14:04:56.580259+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:52.030718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:52.030718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":622,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.582589+0000","last_change":"2024-09-21T13:57:12.683321+0000","last_active":"2024-09-21T14:04:56.582589+0000","last_peered":"2024-09-21T14:04:56.582589+0000","last_clean":"2024-09-21T14:04:56.582589+0000","last_became_active":"2024-09-21T13:57:12.682979+0000","last_became_peered":"2024-09-21T13:57:12.682979+0000","last_unstale":"2024-09-21T14:04:56.582589+0000","last_undegraded":"2024-09-21T14:04:56.582589+0000","last_fullsized":"2024-09-21T14:04:56.582589+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:18.027944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:18.027944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:24.573702+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2","version":"418'9563","reported_seq":9649,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:59.774548+0000","last_change":"2024-09-21T14:02:56.559657+0000","last_active":"2024-09-21T14:04:59.774548+0000","last_peered":"2024-09-21T14:04:59.774548+0000","last_clean":"2024-09-21T14:04:59.774548+0000","last_became_active":"2024-09-21T14:02:56.559368+0000","last_became_peered":"2024-09-21T14:02:56.559368+0000","last_unstale":"2024-09-21T14:04:59.774548+0000","last_undegraded":"2024-09-21T14:04:59.774548+0000","last_fullsized":"2024-09-21T14:04:59.774548+0000","mapping_epoch":381,"log_start":"398'4842","ondisk_log_start":"398'4842","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":4721,"log_dups_size":0,"ondisk_log_size":4721,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":419364864,"num_objects":6399,"num_object_clones":0,"num_object_copies":19197,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9563,"num_write_kb":510784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"75'79","reported_seq":811,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.584736+0000","last_change":"2024-09-21T14:02:55.806160+0000","last_active":"2024-09-21T14:04:56.584736+0000","last_peered":"2024-09-21T14:04:56.584736+0000","last_clean":"2024-09-21T14:04:56.584736+0000","last_became_active":"2024-09-21T14:00:41.139315+0000","last_became_peered":"2024-09-21T14:00:41.139315+0000","last_unstale":"2024-09-21T14:04:56.584736+0000","last_undegraded":"2024-09-21T14:04:56.584736+0000","last_fullsized":"2024-09-21T14:04:56.584736+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:02:55.806077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.806077+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"290'20","reported_seq":832,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-21T14:05:03.633152+0000","last_change":"2024-09-21T13:58:17.789407+0000","last_active":"2024-09-21T14:05:03.633152+0000","last_peered":"2024-09-21T14:05:03.633152+0000","last_clean":"2024-09-21T14:05:03.633152+0000","last_became_active":"2024-09-21T13:58:17.765427+0000","last_became_peered":"2024-09-21T13:58:17.765427+0000","last_unstale":"2024-09-21T14:05:03.633152+0000","last_undegraded":"2024-09-21T14:05:03.633152+0000","last_fullsized":"2024-09-21T14:05:03.633152+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:58:13.464196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:13.464196+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":686,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.584516+0000","last_change":"2024-09-21T14:04:21.215419+0000","last_active":"2024-09-21T14:04:56.584516+0000","last_peered":"2024-09-21T14:04:56.584516+0000","last_clean":"2024-09-21T14:04:56.584516+0000","last_became_active":"2024-09-21T14:04:21.215102+0000","last_became_peered":"2024-09-21T14:04:21.215102+0000","last_unstale":"2024-09-21T14:04:56.584516+0000","last_undegraded":"2024-09-21T14:04:56.584516+0000","last_fullsized":"2024-09-21T14:04:56.584516+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:58.236645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:02:58.236645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"346'14","reported_seq":828,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.582601+0000","last_change":"2024-09-21T14:01:41.186960+0000","last_active":"2024-09-21T14:04:56.582601+0000","last_peered":"2024-09-21T14:04:56.582601+0000","last_clean":"2024-09-21T14:04:56.582601+0000","last_became_active":"2024-09-21T14:01:41.186564+0000","last_became_peered":"2024-09-21T14:01:41.186564+0000","last_unstale":"2024-09-21T14:04:56.582601+0000","last_undegraded":"2024-09-21T14:04:56.582601+0000","last_fullsized":"2024-09-21T14:04:56.582601+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'12","last_scrub_stamp":"2024-09-21T13:59:38.738120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:38.738120+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"68'1","reported_seq":760,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-21T14:05:03.632085+0000","last_change":"2024-09-21T13:59:31.392186+0000","last_active":"2024-09-21T14:05:03.632085+0000","last_peered":"2024-09-21T14:05:03.632085+0000","last_clean":"2024-09-21T14:05:03.632085+0000","last_became_active":"2024-09-21T13:52:04.313210+0000","last_became_peered":"2024-09-21T13:52:04.313210+0000","last_unstale":"2024-09-21T14:05:03.632085+0000","last_undegraded":"2024-09-21T14:05:03.632085+0000","last_fullsized":"2024-09-21T14:05:03.632085+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T13:59:31.392105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:31.392105+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.5","version":"418'9919","reported_seq":10089,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:05:03.007152+0000","last_change":"2024-09-21T14:04:48.634887+0000","last_active":"2024-09-21T14:05:03.007152+0000","last_peered":"2024-09-21T14:05:03.007152+0000","last_clean":"2024-09-21T14:05:03.007152+0000","last_became_active":"2024-09-21T14:02:56.556786+0000","last_became_peered":"2024-09-21T14:02:56.556786+0000","last_unstale":"2024-09-21T14:05:03.007152+0000","last_undegraded":"2024-09-21T14:05:03.007152+0000","last_fullsized":"2024-09-21T14:05:03.007152+0000","mapping_epoch":381,"log_start":"398'4925","ondisk_log_start":"398'4925","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'8991","last_scrub_stamp":"2024-09-21T14:04:48.631935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:04:48.631935+0000","objects_scrubbed":7324,"log_size":4994,"log_dups_size":0,"ondisk_log_size":4994,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:05.417795+0000","scrub_duration":2904,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":413859840,"num_objects":6315,"num_object_clones":0,"num_object_copies":18945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9919,"num_write_kb":519488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"290'15","reported_seq":785,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.583473+0000","last_change":"2024-09-21T13:58:50.498597+0000","last_active":"2024-09-21T14:04:56.583473+0000","last_peered":"2024-09-21T14:04:56.583473+0000","last_clean":"2024-09-21T14:04:56.583473+0000","last_became_active":"2024-09-21T13:49:00.053016+0000","last_became_peered":"2024-09-21T13:49:00.053016+0000","last_unstale":"2024-09-21T14:04:56.583473+0000","last_undegraded":"2024-09-21T14:04:56.583473+0000","last_fullsized":"2024-09-21T14:04:56.583473+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:58:50.498369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:50.498369+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:58.890531+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":724,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.586231+0000","last_change":"2024-09-21T14:04:00.767591+0000","last_active":"2024-09-21T14:04:56.586231+0000","last_peered":"2024-09-21T14:04:56.586231+0000","last_clean":"2024-09-21T14:04:56.586231+0000","last_became_active":"2024-09-21T14:01:41.192443+0000","last_became_peered":"2024-09-21T14:01:41.192443+0000","last_unstale":"2024-09-21T14:04:56.586231+0000","last_undegraded":"2024-09-21T14:04:56.586231+0000","last_fullsized":"2024-09-21T14:04:56.586231+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:00.767519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:00.767519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:08.722933+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":706,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-21T14:05:03.631994+0000","last_change":"2024-09-21T14:02:56.796917+0000","last_active":"2024-09-21T14:05:03.631994+0000","last_peered":"2024-09-21T14:05:03.631994+0000","last_clean":"2024-09-21T14:05:03.631994+0000","last_became_active":"2024-09-21T13:57:12.680978+0000","last_became_peered":"2024-09-21T13:57:12.680978+0000","last_unstale":"2024-09-21T14:05:03.631994+0000","last_undegraded":"2024-09-21T14:05:03.631994+0000","last_fullsized":"2024-09-21T14:05:03.631994+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:56.796861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:02:56.796861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":653,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.584190+0000","last_change":"2024-09-21T14:04:21.210096+0000","last_active":"2024-09-21T14:04:56.584190+0000","last_peered":"2024-09-21T14:04:56.584190+0000","last_clean":"2024-09-21T14:04:56.584190+0000","last_became_active":"2024-09-21T14:04:21.207534+0000","last_became_peered":"2024-09-21T14:04:21.207534+0000","last_unstale":"2024-09-21T14:04:56.584190+0000","last_undegraded":"2024-09-21T14:04:56.584190+0000","last_fullsized":"2024-09-21T14:04:56.584190+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:13.698091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:13.698091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"419'9720","reported_seq":21530,"reported_epoch":419,"state":"active+remapped+backfilling","last_fresh":"2024-09-21T14:05:03.789864+0000","last_change":"2024-09-21T14:04:22.282881+0000","last_active":"2024-09-21T14:05:03.789864+0000","last_peered":"2024-09-21T14:05:03.789864+0000","last_clean":"2024-09-21T14:04:20.165463+0000","last_became_active":"2024-09-21T14:04:22.266519+0000","last_became_peered":"2024-09-21T14:04:22.266519+0000","last_unstale":"2024-09-21T14:05:03.789864+0000","last_undegraded":"2024-09-21T14:05:03.789864+0000","last_fullsized":"2024-09-21T14:05:03.789864+0000","mapping_epoch":409,"log_start":"398'4200","ondisk_log_start":"398'4200","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":5520,"log_dups_size":0,"ondisk_log_size":5520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":396754944,"num_objects":6054,"num_object_clones":0,"num_object_copies":18162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1047,"num_objects_unfound":0,"num_objects_dirty":6054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9720,"num_write_kb":504768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5858,"num_bytes_recovered":383909888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,0],"avail_no_missing":["4","0","3"],"object_location_counts":[{"shards":"0,3,4","objects":6054}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"290'35","reported_seq":395,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-21T14:05:03.636569+0000","last_change":"2024-09-21T14:01:02.006619+0000","last_active":"2024-09-21T14:05:03.636569+0000","last_peered":"2024-09-21T14:05:03.636569+0000","last_clean":"2024-09-21T14:05:03.636569+0000","last_became_active":"2024-09-21T13:59:32.204184+0000","last_became_peered":"2024-09-21T13:59:32.204184+0000","last_unstale":"2024-09-21T14:05:03.636569+0000","last_undegraded":"2024-09-21T14:05:03.636569+0000","last_fullsized":"2024-09-21T14:05:03.636569+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'35","last_scrub_stamp":"2024-09-21T14:01:02.006504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:02.006504+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:29.829621+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":746,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.585593+0000","last_change":"2024-09-21T14:03:31.882264+0000","last_active":"2024-09-21T14:04:56.585593+0000","last_peered":"2024-09-21T14:04:56.585593+0000","last_clean":"2024-09-21T14:04:56.585593+0000","last_became_active":"2024-09-21T14:01:41.184567+0000","last_became_peered":"2024-09-21T14:01:41.184567+0000","last_unstale":"2024-09-21T14:04:56.585593+0000","last_undegraded":"2024-09-21T14:04:56.585593+0000","last_fullsized":"2024-09-21T14:04:56.585593+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:31.882215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:03:31.882215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":724,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.581771+0000","last_change":"2024-09-21T14:04:48.685009+0000","last_active":"2024-09-21T14:04:56.581771+0000","last_peered":"2024-09-21T14:04:56.581771+0000","last_clean":"2024-09-21T14:04:56.581771+0000","last_became_active":"2024-09-21T14:01:41.181911+0000","last_became_peered":"2024-09-21T14:01:41.181911+0000","last_unstale":"2024-09-21T14:04:56.581771+0000","last_undegraded":"2024-09-21T14:04:56.581771+0000","last_fullsized":"2024-09-21T14:04:56.581771+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:48.684941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:04:48.684941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:08.622354+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":673,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.584321+0000","last_change":"2024-09-21T14:03:33.893158+0000","last_active":"2024-09-21T14:04:56.584321+0000","last_peered":"2024-09-21T14:04:56.584321+0000","last_clean":"2024-09-21T14:04:56.584321+0000","last_became_active":"2024-09-21T13:59:32.219332+0000","last_became_peered":"2024-09-21T13:59:32.219332+0000","last_unstale":"2024-09-21T14:04:56.584321+0000","last_undegraded":"2024-09-21T14:04:56.584321+0000","last_fullsized":"2024-09-21T14:04:56.584321+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:33.893072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:33.893072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"419'9899","reported_seq":10076,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-21T14:05:03.715211+0000","last_change":"2024-09-21T14:04:40.683339+0000","last_active":"2024-09-21T14:05:03.715211+0000","last_peered":"2024-09-21T14:05:03.715211+0000","last_clean":"2024-09-21T14:05:03.715211+0000","last_became_active":"2024-09-21T14:02:56.559331+0000","last_became_peered":"2024-09-21T14:02:56.559331+0000","last_unstale":"2024-09-21T14:05:03.715211+0000","last_undegraded":"2024-09-21T14:05:03.715211+0000","last_fullsized":"2024-09-21T14:05:03.715211+0000","mapping_epoch":381,"log_start":"398'4790","ondisk_log_start":"398'4790","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'8371","last_scrub_stamp":"2024-09-21T14:04:40.683117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:04:40.683117+0000","objects_scrubbed":7905,"log_size":5109,"log_dups_size":0,"ondisk_log_size":5109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:47.784341+0000","scrub_duration":2828,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":411893760,"num_objects":6285,"num_object_clones":0,"num_object_copies":18855,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9899,"num_write_kb":517888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"290'18","reported_seq":804,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-21T14:05:03.632010+0000","last_change":"2024-09-21T13:59:28.340494+0000","last_active":"2024-09-21T14:05:03.632010+0000","last_peered":"2024-09-21T14:05:03.632010+0000","last_clean":"2024-09-21T14:05:03.632010+0000","last_became_active":"2024-09-21T13:57:12.680627+0000","last_became_peered":"2024-09-21T13:57:12.680627+0000","last_unstale":"2024-09-21T14:05:03.632010+0000","last_undegraded":"2024-09-21T14:05:03.632010+0000","last_fullsized":"2024-09-21T14:05:03.632010+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'18","last_scrub_stamp":"2024-09-21T13:59:28.340415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:28.340415+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":700,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.587610+0000","last_change":"2024-09-21T14:04:21.206034+0000","last_active":"2024-09-21T14:04:56.587610+0000","last_peered":"2024-09-21T14:04:56.587610+0000","last_clean":"2024-09-21T14:04:56.587610+0000","last_became_active":"2024-09-21T14:04:21.205699+0000","last_became_peered":"2024-09-21T14:04:21.205699+0000","last_unstale":"2024-09-21T14:04:56.587610+0000","last_undegraded":"2024-09-21T14:04:56.587610+0000","last_fullsized":"2024-09-21T14:04:56.587610+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:21.582325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:21.582325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:25.153959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":679,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.580173+0000","last_change":"2024-09-21T14:02:27.791041+0000","last_active":"2024-09-21T14:04:56.580173+0000","last_peered":"2024-09-21T14:04:56.580173+0000","last_clean":"2024-09-21T14:04:56.580173+0000","last_became_active":"2024-09-21T13:52:04.322147+0000","last_became_peered":"2024-09-21T13:52:04.322147+0000","last_unstale":"2024-09-21T14:04:56.580173+0000","last_undegraded":"2024-09-21T14:04:56.580173+0000","last_fullsized":"2024-09-21T14:04:56.580173+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:27.790994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:02:27.790994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":687,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.584791+0000","last_change":"2024-09-21T14:02:36.371955+0000","last_active":"2024-09-21T14:04:56.584791+0000","last_peered":"2024-09-21T14:04:56.584791+0000","last_clean":"2024-09-21T14:04:56.584791+0000","last_became_active":"2024-09-21T13:59:32.203767+0000","last_became_peered":"2024-09-21T13:59:32.203767+0000","last_unstale":"2024-09-21T14:04:56.584791+0000","last_undegraded":"2024-09-21T14:04:56.584791+0000","last_fullsized":"2024-09-21T14:04:56.584791+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:36.371747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:02:36.371747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"418'9606","reported_seq":9772,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:05:00.900023+0000","last_change":"2024-09-21T14:04:52.260111+0000","last_active":"2024-09-21T14:05:00.900023+0000","last_peered":"2024-09-21T14:05:00.900023+0000","last_clean":"2024-09-21T14:05:00.900023+0000","last_became_active":"2024-09-21T14:02:56.563788+0000","last_became_peered":"2024-09-21T14:02:56.563788+0000","last_unstale":"2024-09-21T14:05:00.900023+0000","last_undegraded":"2024-09-21T14:05:00.900023+0000","last_fullsized":"2024-09-21T14:05:00.900023+0000","mapping_epoch":381,"log_start":"398'4737","ondisk_log_start":"398'4737","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'9041","last_scrub_stamp":"2024-09-21T14:04:52.260025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:04:52.260025+0000","objects_scrubbed":7053,"log_size":4869,"log_dups_size":0,"ondisk_log_size":4869,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:57.305832+0000","scrub_duration":1833,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":421789696,"num_objects":6436,"num_object_clones":0,"num_object_copies":19308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9606,"num_write_kb":513344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"290'27","reported_seq":800,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.586151+0000","last_change":"2024-09-21T14:03:31.360992+0000","last_active":"2024-09-21T14:04:56.586151+0000","last_peered":"2024-09-21T14:04:56.586151+0000","last_clean":"2024-09-21T14:04:56.586151+0000","last_became_active":"2024-09-21T14:01:41.192481+0000","last_became_peered":"2024-09-21T14:01:41.192481+0000","last_unstale":"2024-09-21T14:04:56.586151+0000","last_undegraded":"2024-09-21T14:04:56.586151+0000","last_fullsized":"2024-09-21T14:04:56.586151+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'27","last_scrub_stamp":"2024-09-21T14:03:31.360906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:03:31.360906+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":686,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.585448+0000","last_change":"2024-09-21T14:04:21.218814+0000","last_active":"2024-09-21T14:04:56.585448+0000","last_peered":"2024-09-21T14:04:56.585448+0000","last_clean":"2024-09-21T14:04:56.585448+0000","last_became_active":"2024-09-21T14:04:21.201644+0000","last_became_peered":"2024-09-21T14:04:21.201644+0000","last_unstale":"2024-09-21T14:04:56.585448+0000","last_undegraded":"2024-09-21T14:04:56.585448+0000","last_fullsized":"2024-09-21T14:04:56.585448+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T14:03:04.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:03:04.953658+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"300'14","reported_seq":905,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.584100+0000","last_change":"2024-09-21T13:59:32.220898+0000","last_active":"2024-09-21T14:04:56.584100+0000","last_peered":"2024-09-21T14:04:56.584100+0000","last_clean":"2024-09-21T14:04:56.584100+0000","last_became_active":"2024-09-21T13:59:32.206956+0000","last_became_peered":"2024-09-21T13:59:32.206956+0000","last_unstale":"2024-09-21T14:04:56.584100+0000","last_undegraded":"2024-09-21T14:04:56.584100+0000","last_fullsized":"2024-09-21T14:04:56.584100+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'11","last_scrub_stamp":"2024-09-21T13:56:31.616003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:56:31.616003+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:27.198182+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":748,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.585799+0000","last_change":"2024-09-21T14:04:42.433805+0000","last_active":"2024-09-21T14:04:56.585799+0000","last_peered":"2024-09-21T14:04:56.585799+0000","last_clean":"2024-09-21T14:04:56.585799+0000","last_became_active":"2024-09-21T14:01:41.246106+0000","last_became_peered":"2024-09-21T14:01:41.246106+0000","last_unstale":"2024-09-21T14:04:56.585799+0000","last_undegraded":"2024-09-21T14:04:56.585799+0000","last_fullsized":"2024-09-21T14:04:56.585799+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:42.433729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:04:42.433729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:04.041353+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":338,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-21T14:05:03.637173+0000","last_change":"2024-09-21T14:04:21.198859+0000","last_active":"2024-09-21T14:05:03.637173+0000","last_peered":"2024-09-21T14:05:03.637173+0000","last_clean":"2024-09-21T14:05:03.637173+0000","last_became_active":"2024-09-21T14:04:21.197402+0000","last_became_peered":"2024-09-21T14:04:21.197402+0000","last_unstale":"2024-09-21T14:05:03.637173+0000","last_undegraded":"2024-09-21T14:05:03.637173+0000","last_fullsized":"2024-09-21T14:05:03.637173+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:50.733007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:01:50.733007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:31.764576+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"290'12","reported_seq":714,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.586039+0000","last_change":"2024-09-21T13:59:32.247422+0000","last_active":"2024-09-21T14:04:56.586039+0000","last_peered":"2024-09-21T14:04:56.586039+0000","last_clean":"2024-09-21T14:04:56.586039+0000","last_became_active":"2024-09-21T13:59:32.211306+0000","last_became_peered":"2024-09-21T13:59:32.211306+0000","last_unstale":"2024-09-21T14:04:56.586039+0000","last_undegraded":"2024-09-21T14:04:56.586039+0000","last_fullsized":"2024-09-21T14:04:56.586039+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'6","last_scrub_stamp":"2024-09-21T13:55:12.066873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:55:12.066873+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":672,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.586352+0000","last_change":"2024-09-21T14:01:41.199742+0000","last_active":"2024-09-21T14:04:56.586352+0000","last_peered":"2024-09-21T14:04:56.586352+0000","last_clean":"2024-09-21T14:04:56.586352+0000","last_became_active":"2024-09-21T14:01:41.199321+0000","last_became_peered":"2024-09-21T14:01:41.199321+0000","last_unstale":"2024-09-21T14:04:56.586352+0000","last_undegraded":"2024-09-21T14:04:56.586352+0000","last_fullsized":"2024-09-21T14:04:56.586352+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:37.106238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:37.106238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:17.707656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":719,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.585469+0000","last_change":"2024-09-21T14:03:01.872986+0000","last_active":"2024-09-21T14:04:56.585469+0000","last_peered":"2024-09-21T14:04:56.585469+0000","last_clean":"2024-09-21T14:04:56.585469+0000","last_became_active":"2024-09-21T13:59:32.219525+0000","last_became_peered":"2024-09-21T13:59:32.219525+0000","last_unstale":"2024-09-21T14:04:56.585469+0000","last_undegraded":"2024-09-21T14:04:56.585469+0000","last_fullsized":"2024-09-21T14:04:56.585469+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:01.872928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:01.872928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"290'15","reported_seq":694,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.585768+0000","last_change":"2024-09-21T14:04:21.228883+0000","last_active":"2024-09-21T14:04:56.585768+0000","last_peered":"2024-09-21T14:04:56.585768+0000","last_clean":"2024-09-21T14:04:56.585768+0000","last_became_active":"2024-09-21T14:04:21.208398+0000","last_became_peered":"2024-09-21T14:04:21.208398+0000","last_unstale":"2024-09-21T14:04:56.585768+0000","last_undegraded":"2024-09-21T14:04:56.585768+0000","last_fullsized":"2024-09-21T14:04:56.585768+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:55:04.990842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:55:04.990842+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:29.202675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"61'1","reported_seq":737,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.584581+0000","last_change":"2024-09-21T14:04:04.012140+0000","last_active":"2024-09-21T14:04:56.584581+0000","last_peered":"2024-09-21T14:04:56.584581+0000","last_clean":"2024-09-21T14:04:56.584581+0000","last_became_active":"2024-09-21T13:59:32.219515+0000","last_became_peered":"2024-09-21T13:59:32.219515+0000","last_unstale":"2024-09-21T14:04:56.584581+0000","last_undegraded":"2024-09-21T14:04:56.584581+0000","last_fullsized":"2024-09-21T14:04:56.584581+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T14:04:04.012006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:04.012006+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:20.215213+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":670,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.585782+0000","last_change":"2024-09-21T14:03:33.309440+0000","last_active":"2024-09-21T14:04:56.585782+0000","last_peered":"2024-09-21T14:04:56.585782+0000","last_clean":"2024-09-21T14:04:56.585782+0000","last_became_active":"2024-09-21T13:59:32.210586+0000","last_became_peered":"2024-09-21T13:59:32.210586+0000","last_unstale":"2024-09-21T14:04:56.585782+0000","last_undegraded":"2024-09-21T14:04:56.585782+0000","last_fullsized":"2024-09-21T14:04:56.585782+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:33.309248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:33.309248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:57.005556+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":720,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.581700+0000","last_change":"2024-09-21T14:02:29.731635+0000","last_active":"2024-09-21T14:04:56.581700+0000","last_peered":"2024-09-21T14:04:56.581700+0000","last_clean":"2024-09-21T14:04:56.581700+0000","last_became_active":"2024-09-21T13:59:32.205803+0000","last_became_peered":"2024-09-21T13:59:32.205803+0000","last_unstale":"2024-09-21T14:04:56.581700+0000","last_undegraded":"2024-09-21T14:04:56.581700+0000","last_fullsized":"2024-09-21T14:04:56.581700+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:29.731565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:02:29.731565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:04:56.723726+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"290'15","reported_seq":717,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.586273+0000","last_change":"2024-09-21T14:01:41.189676+0000","last_active":"2024-09-21T14:04:56.586273+0000","last_peered":"2024-09-21T14:04:56.586273+0000","last_clean":"2024-09-21T14:04:56.586273+0000","last_became_active":"2024-09-21T14:01:41.187904+0000","last_became_peered":"2024-09-21T14:01:41.187904+0000","last_unstale":"2024-09-21T14:04:56.586273+0000","last_undegraded":"2024-09-21T14:04:56.586273+0000","last_fullsized":"2024-09-21T14:04:56.586273+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:57:24.124604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:57:24.124604+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:30.424236+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":361,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.590010+0000","last_change":"2024-09-21T14:04:56.588256+0000","last_active":"2024-09-21T14:04:56.590010+0000","last_peered":"2024-09-21T14:04:56.590010+0000","last_clean":"2024-09-21T14:04:56.590010+0000","last_became_active":"2024-09-21T14:04:56.587953+0000","last_became_peered":"2024-09-21T14:04:56.587953+0000","last_unstale":"2024-09-21T14:04:56.590010+0000","last_undegraded":"2024-09-21T14:04:56.590010+0000","last_fullsized":"2024-09-21T14:04:56.590010+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:37.078458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:37.078458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:52.347012+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"300'14","reported_seq":838,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.585611+0000","last_change":"2024-09-21T13:59:32.208253+0000","last_active":"2024-09-21T14:04:56.585611+0000","last_peered":"2024-09-21T14:04:56.585611+0000","last_clean":"2024-09-21T14:04:56.585611+0000","last_became_active":"2024-09-21T13:59:32.206926+0000","last_became_peered":"2024-09-21T13:59:32.206926+0000","last_unstale":"2024-09-21T14:04:56.585611+0000","last_undegraded":"2024-09-21T14:04:56.585611+0000","last_fullsized":"2024-09-21T14:04:56.585611+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'10","last_scrub_stamp":"2024-09-21T13:54:20.946782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:20.946782+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:20.221431+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":766,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-21T14:05:03.631585+0000","last_change":"2024-09-21T14:01:41.200564+0000","last_active":"2024-09-21T14:05:03.631585+0000","last_peered":"2024-09-21T14:05:03.631585+0000","last_clean":"2024-09-21T14:05:03.631585+0000","last_became_active":"2024-09-21T14:01:41.200113+0000","last_became_peered":"2024-09-21T14:01:41.200113+0000","last_unstale":"2024-09-21T14:05:03.631585+0000","last_undegraded":"2024-09-21T14:05:03.631585+0000","last_fullsized":"2024-09-21T14:05:03.631585+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:54.805619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:54.805619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"290'20","reported_seq":721,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.585371+0000","last_change":"2024-09-21T14:04:04.192280+0000","last_active":"2024-09-21T14:04:56.585371+0000","last_peered":"2024-09-21T14:04:56.585371+0000","last_clean":"2024-09-21T14:04:56.585371+0000","last_became_active":"2024-09-21T13:59:32.208332+0000","last_became_peered":"2024-09-21T13:59:32.208332+0000","last_unstale":"2024-09-21T14:04:56.585371+0000","last_undegraded":"2024-09-21T14:04:56.585371+0000","last_fullsized":"2024-09-21T14:04:56.585371+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'20","last_scrub_stamp":"2024-09-21T14:04:04.192193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:04:04.192193+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:06.970216+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":705,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.584870+0000","last_change":"2024-09-21T13:59:32.212737+0000","last_active":"2024-09-21T14:04:56.584870+0000","last_peered":"2024-09-21T14:04:56.584870+0000","last_clean":"2024-09-21T14:04:56.584870+0000","last_became_active":"2024-09-21T13:59:32.209804+0000","last_became_peered":"2024-09-21T13:59:32.209804+0000","last_unstale":"2024-09-21T14:04:56.584870+0000","last_undegraded":"2024-09-21T14:04:56.584870+0000","last_fullsized":"2024-09-21T14:04:56.584870+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:25.571256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:25.571256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:29.134483+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"346'10","reported_seq":846,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.586454+0000","last_change":"2024-09-21T14:03:34.365517+0000","last_active":"2024-09-21T14:04:56.586454+0000","last_peered":"2024-09-21T14:04:56.586454+0000","last_clean":"2024-09-21T14:04:56.586454+0000","last_became_active":"2024-09-21T14:01:41.206303+0000","last_became_peered":"2024-09-21T14:01:41.206303+0000","last_unstale":"2024-09-21T14:04:56.586454+0000","last_undegraded":"2024-09-21T14:04:56.586454+0000","last_fullsized":"2024-09-21T14:04:56.586454+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'10","last_scrub_stamp":"2024-09-21T14:03:34.365428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:34.365428+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:00.856228+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":617,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.587490+0000","last_change":"2024-09-21T14:04:21.202582+0000","last_active":"2024-09-21T14:04:56.587490+0000","last_peered":"2024-09-21T14:04:56.587490+0000","last_clean":"2024-09-21T14:04:56.587490+0000","last_became_active":"2024-09-21T14:04:21.201277+0000","last_became_peered":"2024-09-21T14:04:21.201277+0000","last_unstale":"2024-09-21T14:04:56.587490+0000","last_undegraded":"2024-09-21T14:04:56.587490+0000","last_fullsized":"2024-09-21T14:04:56.587490+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:27.961544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:27.961544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"290'19","reported_seq":837,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-21T14:05:03.631110+0000","last_change":"2024-09-21T14:04:05.139089+0000","last_active":"2024-09-21T14:05:03.631110+0000","last_peered":"2024-09-21T14:05:03.631110+0000","last_clean":"2024-09-21T14:05:03.631110+0000","last_became_active":"2024-09-21T13:58:17.771497+0000","last_became_peered":"2024-09-21T13:58:17.771497+0000","last_unstale":"2024-09-21T14:05:03.631110+0000","last_undegraded":"2024-09-21T14:05:03.631110+0000","last_fullsized":"2024-09-21T14:05:03.631110+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'19","last_scrub_stamp":"2024-09-21T14:04:05.138989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:04:05.138989+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:14.903630+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":347,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.586218+0000","last_change":"2024-09-21T14:02:57.807136+0000","last_active":"2024-09-21T14:04:56.586218+0000","last_peered":"2024-09-21T14:04:56.586218+0000","last_clean":"2024-09-21T14:04:56.586218+0000","last_became_active":"2024-09-21T13:59:32.201568+0000","last_became_peered":"2024-09-21T13:59:32.201568+0000","last_unstale":"2024-09-21T14:04:56.586218+0000","last_undegraded":"2024-09-21T14:04:56.586218+0000","last_fullsized":"2024-09-21T14:04:56.586218+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:57.807059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:02:57.807059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":777,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.579487+0000","last_change":"2024-09-21T14:01:41.185588+0000","last_active":"2024-09-21T14:04:56.579487+0000","last_peered":"2024-09-21T14:04:56.579487+0000","last_clean":"2024-09-21T14:04:56.579487+0000","last_became_active":"2024-09-21T14:01:41.185293+0000","last_became_peered":"2024-09-21T14:01:41.185293+0000","last_unstale":"2024-09-21T14:04:56.579487+0000","last_undegraded":"2024-09-21T14:04:56.579487+0000","last_fullsized":"2024-09-21T14:04:56.579487+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:35.409237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:35.409237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":685,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.584531+0000","last_change":"2024-09-21T14:04:21.209188+0000","last_active":"2024-09-21T14:04:56.584531+0000","last_peered":"2024-09-21T14:04:56.584531+0000","last_clean":"2024-09-21T14:04:56.584531+0000","last_became_active":"2024-09-21T14:04:21.208329+0000","last_became_peered":"2024-09-21T14:04:21.208329+0000","last_unstale":"2024-09-21T14:04:56.584531+0000","last_undegraded":"2024-09-21T14:04:56.584531+0000","last_fullsized":"2024-09-21T14:04:56.584531+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:57.225602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:02:57.225602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"290'15","reported_seq":837,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.579700+0000","last_change":"2024-09-21T14:04:12.139406+0000","last_active":"2024-09-21T14:04:56.579700+0000","last_peered":"2024-09-21T14:04:56.579700+0000","last_clean":"2024-09-21T14:04:56.579700+0000","last_became_active":"2024-09-21T13:58:17.764120+0000","last_became_peered":"2024-09-21T13:58:17.764120+0000","last_unstale":"2024-09-21T14:04:56.579700+0000","last_undegraded":"2024-09-21T14:04:56.579700+0000","last_fullsized":"2024-09-21T14:04:56.579700+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'15","last_scrub_stamp":"2024-09-21T14:04:12.139190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:04:12.139190+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:40.181606+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":717,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.585327+0000","last_change":"2024-09-21T14:04:07.038058+0000","last_active":"2024-09-21T14:04:56.585327+0000","last_peered":"2024-09-21T14:04:56.585327+0000","last_clean":"2024-09-21T14:04:56.585327+0000","last_became_active":"2024-09-21T13:59:32.221854+0000","last_became_peered":"2024-09-21T13:59:32.221854+0000","last_unstale":"2024-09-21T14:04:56.585327+0000","last_undegraded":"2024-09-21T14:04:56.585327+0000","last_fullsized":"2024-09-21T14:04:56.585327+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:07.037989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:07.037989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:34.939994+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":653,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.584411+0000","last_change":"2024-09-21T14:03:32.262147+0000","last_active":"2024-09-21T14:04:56.584411+0000","last_peered":"2024-09-21T14:04:56.584411+0000","last_clean":"2024-09-21T14:04:56.584411+0000","last_became_active":"2024-09-21T13:57:12.703441+0000","last_became_peered":"2024-09-21T13:57:12.703441+0000","last_unstale":"2024-09-21T14:04:56.584411+0000","last_undegraded":"2024-09-21T14:04:56.584411+0000","last_fullsized":"2024-09-21T14:04:56.584411+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:32.262067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:32.262067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:00.749089+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":425,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.601661+0000","last_change":"2024-09-21T14:04:56.588351+0000","last_active":"2024-09-21T14:04:56.601661+0000","last_peered":"2024-09-21T14:04:56.601661+0000","last_clean":"2024-09-21T14:04:56.601661+0000","last_became_active":"2024-09-21T14:04:56.587503+0000","last_became_peered":"2024-09-21T14:04:56.587503+0000","last_unstale":"2024-09-21T14:04:56.601661+0000","last_undegraded":"2024-09-21T14:04:56.601661+0000","last_fullsized":"2024-09-21T14:04:56.601661+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:29.345427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:29.345427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"290'19","reported_seq":789,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.584809+0000","last_change":"2024-09-21T13:59:32.277742+0000","last_active":"2024-09-21T14:04:56.584809+0000","last_peered":"2024-09-21T14:04:56.584809+0000","last_clean":"2024-09-21T14:04:56.584809+0000","last_became_active":"2024-09-21T13:59:32.231567+0000","last_became_peered":"2024-09-21T13:59:32.231567+0000","last_unstale":"2024-09-21T14:04:56.584809+0000","last_undegraded":"2024-09-21T14:04:56.584809+0000","last_fullsized":"2024-09-21T14:04:56.584809+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:54:22.978267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:22.978267+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:24.229733+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":677,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.586445+0000","last_change":"2024-09-21T14:03:01.448170+0000","last_active":"2024-09-21T14:04:56.586445+0000","last_peered":"2024-09-21T14:04:56.586445+0000","last_clean":"2024-09-21T14:04:56.586445+0000","last_became_active":"2024-09-21T14:01:41.190760+0000","last_became_peered":"2024-09-21T14:01:41.190760+0000","last_unstale":"2024-09-21T14:04:56.586445+0000","last_undegraded":"2024-09-21T14:04:56.586445+0000","last_fullsized":"2024-09-21T14:04:56.586445+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:01.448109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:03:01.448109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":723,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.584528+0000","last_change":"2024-09-21T14:04:41.091577+0000","last_active":"2024-09-21T14:04:56.584528+0000","last_peered":"2024-09-21T14:04:56.584528+0000","last_clean":"2024-09-21T14:04:56.584528+0000","last_became_active":"2024-09-21T13:59:32.214004+0000","last_became_peered":"2024-09-21T13:59:32.214004+0000","last_unstale":"2024-09-21T14:04:56.584528+0000","last_undegraded":"2024-09-21T14:04:56.584528+0000","last_fullsized":"2024-09-21T14:04:56.584528+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:41.091525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:04:41.091525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:06.917055+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":718,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.584609+0000","last_change":"2024-09-21T14:04:21.214685+0000","last_active":"2024-09-21T14:04:56.584609+0000","last_peered":"2024-09-21T14:04:56.584609+0000","last_clean":"2024-09-21T14:04:56.584609+0000","last_became_active":"2024-09-21T14:04:21.213307+0000","last_became_peered":"2024-09-21T14:04:21.213307+0000","last_unstale":"2024-09-21T14:04:56.584609+0000","last_undegraded":"2024-09-21T14:04:56.584609+0000","last_fullsized":"2024-09-21T14:04:56.584609+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:28.124296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:28.124296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"290'8","reported_seq":735,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.584879+0000","last_change":"2024-09-21T13:59:26.034457+0000","last_active":"2024-09-21T14:04:56.584879+0000","last_peered":"2024-09-21T14:04:56.584879+0000","last_clean":"2024-09-21T14:04:56.584879+0000","last_became_active":"2024-09-21T13:58:17.775079+0000","last_became_peered":"2024-09-21T13:58:17.775079+0000","last_unstale":"2024-09-21T14:04:56.584879+0000","last_undegraded":"2024-09-21T14:04:56.584879+0000","last_fullsized":"2024-09-21T14:04:56.584879+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'8","last_scrub_stamp":"2024-09-21T13:59:26.034372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:26.034372+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":496,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.589993+0000","last_change":"2024-09-21T14:04:56.587839+0000","last_active":"2024-09-21T14:04:56.589993+0000","last_peered":"2024-09-21T14:04:56.589993+0000","last_clean":"2024-09-21T14:04:56.589993+0000","last_became_active":"2024-09-21T14:04:56.587555+0000","last_became_peered":"2024-09-21T14:04:56.587555+0000","last_unstale":"2024-09-21T14:04:56.589993+0000","last_undegraded":"2024-09-21T14:04:56.589993+0000","last_fullsized":"2024-09-21T14:04:56.589993+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:34.387367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:34.387367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:38.961058+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":354,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-21T14:05:03.636352+0000","last_change":"2024-09-21T14:04:41.932290+0000","last_active":"2024-09-21T14:05:03.636352+0000","last_peered":"2024-09-21T14:05:03.636352+0000","last_clean":"2024-09-21T14:05:03.636352+0000","last_became_active":"2024-09-21T14:04:21.204954+0000","last_became_peered":"2024-09-21T14:04:21.204954+0000","last_unstale":"2024-09-21T14:05:03.636352+0000","last_undegraded":"2024-09-21T14:05:03.636352+0000","last_fullsized":"2024-09-21T14:05:03.636352+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:41.932233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:04:41.932233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:03.486134+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":656,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.584360+0000","last_change":"2024-09-21T14:04:15.135536+0000","last_active":"2024-09-21T14:04:56.584360+0000","last_peered":"2024-09-21T14:04:56.584360+0000","last_clean":"2024-09-21T14:04:56.584360+0000","last_became_active":"2024-09-21T13:59:32.210134+0000","last_became_peered":"2024-09-21T13:59:32.210134+0000","last_unstale":"2024-09-21T14:04:56.584360+0000","last_undegraded":"2024-09-21T14:04:56.584360+0000","last_fullsized":"2024-09-21T14:04:56.584360+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:15.135463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:04:15.135463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:37.791722+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":762,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.580279+0000","last_change":"2024-09-21T14:01:41.187792+0000","last_active":"2024-09-21T14:04:56.580279+0000","last_peered":"2024-09-21T14:04:56.580279+0000","last_clean":"2024-09-21T14:04:56.580279+0000","last_became_active":"2024-09-21T14:01:41.187438+0000","last_became_peered":"2024-09-21T14:01:41.187438+0000","last_unstale":"2024-09-21T14:04:56.580279+0000","last_undegraded":"2024-09-21T14:04:56.580279+0000","last_fullsized":"2024-09-21T14:04:56.580279+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:17.706325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:17.706325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":706,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.584701+0000","last_change":"2024-09-21T14:04:33.252571+0000","last_active":"2024-09-21T14:04:56.584701+0000","last_peered":"2024-09-21T14:04:56.584701+0000","last_clean":"2024-09-21T14:04:56.584701+0000","last_became_active":"2024-09-21T14:04:21.208121+0000","last_became_peered":"2024-09-21T14:04:21.208121+0000","last_unstale":"2024-09-21T14:04:56.584701+0000","last_undegraded":"2024-09-21T14:04:56.584701+0000","last_fullsized":"2024-09-21T14:04:56.584701+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:33.252481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:33.252481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:48.348987+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"290'19","reported_seq":372,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-21T14:05:03.637430+0000","last_change":"2024-09-21T14:03:31.762681+0000","last_active":"2024-09-21T14:05:03.637430+0000","last_peered":"2024-09-21T14:05:03.637430+0000","last_clean":"2024-09-21T14:05:03.637430+0000","last_became_active":"2024-09-21T14:01:41.194606+0000","last_became_peered":"2024-09-21T14:01:41.194606+0000","last_unstale":"2024-09-21T14:05:03.637430+0000","last_undegraded":"2024-09-21T14:05:03.637430+0000","last_fullsized":"2024-09-21T14:05:03.637430+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'19","last_scrub_stamp":"2024-09-21T14:03:31.762605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:03:31.762605+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":380,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-21T14:04:56.591469+0000","last_change":"2024-09-21T14:04:56.591137+0000","last_active":"2024-09-21T14:04:56.591469+0000","last_peered":"2024-09-21T14:04:56.591469+0000","last_clean":"2024-09-21T14:04:56.591469+0000","last_became_active":"2024-09-21T14:04:56.590829+0000","last_became_peered":"2024-09-21T14:04:56.590829+0000","last_unstale":"2024-09-21T14:04:56.591469+0000","last_undegraded":"2024-09-21T14:04:56.591469+0000","last_fullsized":"2024-09-21T14:04:56.591469+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:51.121361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:04:51.121361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:59.958121+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":6542917656,"num_objects":99838,"num_object_clones":0,"num_object_copies":299514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":14403,"num_objects_unfound":0,"num_objects_dirty":99838,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155312,"num_write_kb":8164737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11229,"num_bytes_recovered":735903744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20419981312,"data_stored":20419969096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81390,"ondisk_log_size":81390,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102,"ondisk_log_size":102,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1107,"num_read_kb":864,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":197,"num_bytes_recovered":2794,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":687,"ondisk_log_size":687,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":335,"num_read_kb":330,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":805047,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":42,"num_read_kb":79,"num_write":43,"num_write_kb":592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":214679,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":298,"seq":1279900254275,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2131388,"kb_used_data":1746988,"kb_used_omap":4,"kb_used_meta":384379,"kb_avail":91613764,"statfs":{"total":95995035648,"available":93812494336,"internally_reserved":0,"allocated":1788915712,"data_stored":1788597607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":393604776},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463115,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3209060,"kb_used_data":2790420,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":90536092,"statfs":{"total":95995035648,"available":92708958208,"internally_reserved":0,"allocated":2857390080,"data_stored":2857095270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":428538310},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":345,"seq":1481763717161,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2222880,"kb_used_data":2046036,"kb_used_omap":4,"kb_used_meta":176827,"kb_avail":91522272,"statfs":{"total":95995035648,"available":93718806528,"internally_reserved":0,"allocated":2095140864,"data_stored":2094847444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":181071523},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920984,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3684908,"kb_used_data":3162204,"kb_used_omap":1,"kb_used_meta":522622,"kb_avail":90060244,"statfs":{"total":95995035648,"available":92221689856,"internally_reserved":0,"allocated":3238096896,"data_stored":3237793980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":535165381},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":240,"seq":1030792151135,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2500992,"kb_used_data":2124400,"kb_used_omap":4,"kb_used_meta":376571,"kb_avail":91244160,"statfs":{"total":95995035648,"available":93434019840,"internally_reserved":0,"allocated":2175385600,"data_stored":2175063930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4443,"internal_metadata":385609381},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":130,"seq":558345748637,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3112232,"kb_used_data":2484440,"kb_used_omap":4,"kb_used_meta":627771,"kb_avail":90632920,"statfs":{"total":95995035648,"available":92808110080,"internally_reserved":0,"allocated":2544066560,"data_stored":2543769066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4441,"internal_metadata":642838183},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575081,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2189404,"kb_used_data":1544404,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":91555748,"statfs":{"total":95995035648,"available":93753085952,"internally_reserved":0,"allocated":1581469696,"data_stored":1581110734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":660273605},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738607,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4828228,"kb_used_data":4156860,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":88916924,"statfs":{"total":95995035648,"available":91050930176,"internally_reserved":0,"allocated":4256624640,"data_stored":4256353810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":687274437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1,0,0,12,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":2448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4233236480,"data_stored":4233232408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1576468480,"data_stored":1576468480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2537619456,"data_stored":2537619456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2158493696,"data_stored":2158493696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3221884928,"data_stored":3221880856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2083061760,"data_stored":2083061760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2829520896,"data_stored":2829516824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1779695616,"data_stored":1779695616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T14:05:05.415 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 1, currently 11 2024-09-21T14:05:05.437 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:05.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:05.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:05:05.475 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:05:05.539 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:05:05.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:05.580 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:05.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:05:05.681 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:05:05.716 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:05.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:05.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:05:05.818 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:05:05.865 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:05 smithi049 ceph-mon[40489]: pgmap v816: 158 pgs: 4 active+remapped+backfilling, 154 active+clean; 6.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s; 14403/300117 objects misplaced (4.799%); 50 MiB/s, 795 objects/s recovering 2024-09-21T14:05:05.865 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:05 smithi049 ceph-mon[40489]: from='client.26189 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:05:05.865 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:05 smithi049 ceph-mon[30080]: pgmap v816: 158 pgs: 4 active+remapped+backfilling, 154 active+clean; 6.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s; 14403/300117 objects misplaced (4.799%); 50 MiB/s, 795 objects/s recovering 2024-09-21T14:05:05.865 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:05 smithi049 ceph-mon[30080]: from='client.26189 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:05:05.868 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:05.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:05.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:05:05.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:05 smithi187 ceph-mon[33032]: pgmap v816: 158 pgs: 4 active+remapped+backfilling, 154 active+clean; 6.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s; 14403/300117 objects misplaced (4.799%); 50 MiB/s, 795 objects/s recovering 2024-09-21T14:05:05.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:05 smithi187 ceph-mon[33032]: from='client.26189 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:05:05.969 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:05:06.004 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:06.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:06.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:05:06.106 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:06.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:06.139 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:06.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:06.240 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:05:06.284 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:06.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:06.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:05:06.386 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:06.419 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:06.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:06.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:06.520 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:05:06.555 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:06.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:06.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:05:06.575 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T14:05:06.578 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:05:06.578 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:05:06.578 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:05:06.578 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:05:06.578 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:05:06.578 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:05:06.578 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:05:06.578 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:05:06.578 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:05:06.578 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:05:06.578 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:05:06.579 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:05:06.579 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:05:06.579 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:05:06.579 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:05:06.579 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:05:06.657 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:06.704 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T14:05:06.707 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:06.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:06.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:06.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:06 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/4285675642' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:05:06.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:06 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/4285675642' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:05:06.808 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:06.846 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:06.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:06.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:06.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:06 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/4285675642' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:05:06.948 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:05:06.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:06.989 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:06.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:05:07.090 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:07.137 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:07.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:07.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:07.159 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:05:07.239 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:05:07.277 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:07.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:07.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:05:07.379 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:05:07.416 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:07.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:07.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:05:07.518 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:05:07.544 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:07.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:07.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:05:07.645 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:07.680 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:07.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:07.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:07.782 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:05:07.808 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:07.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:07.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:05:07.910 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:07.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:07 smithi187 ceph-mon[33032]: pgmap v817: 158 pgs: 4 active+remapped+backfilling, 154 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 540 B/s rd, 0 B/s wr, 1.06k op/s; 14403/296079 objects misplaced (4.865%); 42 MiB/s, 672 objects/s recovering 2024-09-21T14:05:07.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:07.947 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:07.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:08.001 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:07 smithi049 ceph-mon[30080]: pgmap v817: 158 pgs: 4 active+remapped+backfilling, 154 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 540 B/s rd, 0 B/s wr, 1.06k op/s; 14403/296079 objects misplaced (4.865%); 42 MiB/s, 672 objects/s recovering 2024-09-21T14:05:08.001 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:07 smithi049 ceph-mon[40489]: pgmap v817: 158 pgs: 4 active+remapped+backfilling, 154 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 540 B/s rd, 0 B/s wr, 1.06k op/s; 14403/296079 objects misplaced (4.865%); 42 MiB/s, 672 objects/s recovering 2024-09-21T14:05:08.015 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T14:05:08.015 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-21T14:05:08.015 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.015 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:160755 10.7 10:e6cf48f8:::benchmark_data_smithi049_176811_object33178:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e420)", 2024-09-21T14:05:08.015 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:05:08.004861+0000", 2024-09-21T14:05:08.016 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0099422060000000003, 2024-09-21T14:05:08.016 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0099715250000000002, 2024-09-21T14:05:08.016 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:05:08.016 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:05:08.016 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-21T14:05:08.016 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:05:08.016 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:05:08.016 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:05:08.016 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 160755 2024-09-21T14:05:08.016 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.016 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:05:08.016 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.016 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:05:08.016 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.004861+0000", 2024-09-21T14:05:08.017 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:08.017 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.017 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.017 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:05:08.017 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.004861+0000", 2024-09-21T14:05:08.017 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:08.017 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.017 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.017 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:05:08.017 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.004862+0000", 2024-09-21T14:05:08.017 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2890000000000001e-06 2024-09-21T14:05:08.017 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.017 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.018 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:05:08.018 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.004866+0000", 2024-09-21T14:05:08.018 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.157e-06 2024-09-21T14:05:08.018 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.018 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.018 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:05:08.018 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.004874+0000", 2024-09-21T14:05:08.018 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9119999999999998e-06 2024-09-21T14:05:08.018 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.018 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.018 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:05:08.018 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.004879+0000", 2024-09-21T14:05:08.018 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8980000000000001e-06 2024-09-21T14:05:08.018 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.019 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.019 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:05:08.019 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.007342+0000", 2024-09-21T14:05:08.019 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.002462171 2024-09-21T14:05:08.019 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.019 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.019 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:05:08.019 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.007469+0000", 2024-09-21T14:05:08.019 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000127303 2024-09-21T14:05:08.019 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.019 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.019 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,5", 2024-09-21T14:05:08.019 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.008303+0000", 2024-09-21T14:05:08.019 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00083424400000000004 2024-09-21T14:05:08.020 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:08.020 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:05:08.020 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:08.020 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.020 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.020 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:160759 10.a 10:584e4933:::benchmark_data_smithi049_176811_object33182:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e420)", 2024-09-21T14:05:08.020 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:05:08.005953+0000", 2024-09-21T14:05:08.020 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0088506609999999993, 2024-09-21T14:05:08.020 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0089547689999999996, 2024-09-21T14:05:08.020 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:05:08.020 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:05:08.020 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-21T14:05:08.020 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:05:08.021 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:05:08.021 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:05:08.021 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 160759 2024-09-21T14:05:08.021 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.021 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:05:08.021 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.021 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:05:08.021 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.005953+0000", 2024-09-21T14:05:08.021 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:08.021 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.021 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.021 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:05:08.021 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.005953+0000", 2024-09-21T14:05:08.021 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:08.022 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.022 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.022 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:05:08.022 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.005954+0000", 2024-09-21T14:05:08.022 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.5000000000000001e-07 2024-09-21T14:05:08.022 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.022 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.022 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:05:08.022 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.005958+0000", 2024-09-21T14:05:08.022 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5190000000000003e-06 2024-09-21T14:05:08.022 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.022 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.022 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:05:08.022 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.005970+0000", 2024-09-21T14:05:08.023 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1409999999999999e-05 2024-09-21T14:05:08.023 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.023 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.023 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:05:08.023 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.005976+0000", 2024-09-21T14:05:08.023 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8600000000000004e-06 2024-09-21T14:05:08.023 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.023 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.023 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:05:08.023 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.006133+0000", 2024-09-21T14:05:08.023 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000156785 2024-09-21T14:05:08.023 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.023 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.024 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:05:08.024 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.006242+0000", 2024-09-21T14:05:08.024 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000108943 2024-09-21T14:05:08.024 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.024 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.024 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2,5", 2024-09-21T14:05:08.024 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.006361+0000", 2024-09-21T14:05:08.024 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000118374 2024-09-21T14:05:08.024 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:08.024 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:05:08.024 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:08.024 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.024 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.025 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:160760 10.f 10:f94f5bbc:::benchmark_data_smithi049_176811_object33183:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e420)", 2024-09-21T14:05:08.025 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:05:08.006758+0000", 2024-09-21T14:05:08.025 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0080453699999999996, 2024-09-21T14:05:08.025 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0082079340000000001, 2024-09-21T14:05:08.025 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:05:08.025 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:05:08.025 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-21T14:05:08.025 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:05:08.025 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:05:08.025 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:05:08.025 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 160760 2024-09-21T14:05:08.025 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.025 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:05:08.026 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.026 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:05:08.026 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.006758+0000", 2024-09-21T14:05:08.026 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:08.026 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.026 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.026 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:05:08.026 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.006758+0000", 2024-09-21T14:05:08.026 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:08.026 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.026 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.027 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:05:08.027 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.006759+0000", 2024-09-21T14:05:08.027 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7000000000000003e-07 2024-09-21T14:05:08.027 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.027 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.027 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:05:08.027 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.006763+0000", 2024-09-21T14:05:08.027 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3100000000000002e-06 2024-09-21T14:05:08.027 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.027 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.027 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:05:08.027 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.006772+0000", 2024-09-21T14:05:08.027 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5490000000000006e-06 2024-09-21T14:05:08.027 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.028 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.028 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:05:08.028 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.006778+0000", 2024-09-21T14:05:08.028 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6379999999999998e-06 2024-09-21T14:05:08.028 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.028 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.028 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:05:08.028 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.011538+0000", 2024-09-21T14:05:08.028 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0047595149999999998 2024-09-21T14:05:08.028 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.028 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.028 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:05:08.028 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.011645+0000", 2024-09-21T14:05:08.028 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000106678 2024-09-21T14:05:08.029 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.029 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.029 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,3", 2024-09-21T14:05:08.029 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.011750+0000", 2024-09-21T14:05:08.029 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000105111 2024-09-21T14:05:08.029 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:08.029 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:05:08.029 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:08.029 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.029 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.029 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:160762 10.b 10:dd063a31:::benchmark_data_smithi049_176811_object33185:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e420)", 2024-09-21T14:05:08.030 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:05:08.007771+0000", 2024-09-21T14:05:08.030 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0070323649999999996, 2024-09-21T14:05:08.030 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00725284, 2024-09-21T14:05:08.030 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:05:08.030 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:05:08.030 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-21T14:05:08.030 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:05:08.030 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:05:08.030 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:05:08.030 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 160762 2024-09-21T14:05:08.030 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.031 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:05:08.031 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.031 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:05:08.031 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.007771+0000", 2024-09-21T14:05:08.031 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:08.031 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.031 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.031 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:05:08.031 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.007771+0000", 2024-09-21T14:05:08.031 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:08.031 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.031 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.031 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:05:08.032 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.007772+0000", 2024-09-21T14:05:08.032 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0189999999999999e-06 2024-09-21T14:05:08.032 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.032 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.032 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:05:08.032 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.007776+0000", 2024-09-21T14:05:08.032 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9720000000000003e-06 2024-09-21T14:05:08.032 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.032 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.032 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:05:08.032 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.007788+0000", 2024-09-21T14:05:08.032 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1874e-05 2024-09-21T14:05:08.032 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.032 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.033 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:05:08.033 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.007795+0000", 2024-09-21T14:05:08.033 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3810000000000001e-06 2024-09-21T14:05:08.033 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.033 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.033 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:05:08.033 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.007844+0000", 2024-09-21T14:05:08.033 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8918000000000002e-05 2024-09-21T14:05:08.033 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.033 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.033 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:05:08.033 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.007950+0000", 2024-09-21T14:05:08.033 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000106011 2024-09-21T14:05:08.033 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.034 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.034 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-21T14:05:08.034 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.008058+0000", 2024-09-21T14:05:08.034 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010802999999999999 2024-09-21T14:05:08.034 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.034 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.034 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:05:08.034 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.010976+0000", 2024-09-21T14:05:08.034 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0029174510000000002 2024-09-21T14:05:08.034 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.034 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.034 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:05:08.034 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.012174+0000", 2024-09-21T14:05:08.034 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001198263 2024-09-21T14:05:08.035 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:08.035 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:05:08.035 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:08.035 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.035 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.035 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.16068.0:160759 10.a e420/409)", 2024-09-21T14:05:08.035 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:05:08.010404+0000", 2024-09-21T14:05:08.035 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.004399722, 2024-09-21T14:05:08.035 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0046802550000000004, 2024-09-21T14:05:08.035 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:05:08.035 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:05:08.035 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "queued for pg", 2024-09-21T14:05:08.035 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:05:08.035 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.036 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:05:08.036 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.010404+0000", 2024-09-21T14:05:08.036 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:08.036 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.036 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.036 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:05:08.036 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.010404+0000", 2024-09-21T14:05:08.036 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:08.036 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.036 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.036 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:05:08.036 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.010405+0000", 2024-09-21T14:05:08.036 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.9500000000000001e-07 2024-09-21T14:05:08.036 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.037 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.037 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:05:08.037 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.010410+0000", 2024-09-21T14:05:08.037 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0549999999999999e-06 2024-09-21T14:05:08.037 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.037 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.037 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:05:08.037 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.010420+0000", 2024-09-21T14:05:08.037 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0576e-05 2024-09-21T14:05:08.037 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.037 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.037 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:05:08.037 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.010427+0000", 2024-09-21T14:05:08.037 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9909999999999996e-06 2024-09-21T14:05:08.038 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:08.038 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:05:08.038 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:08.038 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.038 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.038 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "MOSDPGPushReply(10.d 420/409 [PushReplyOp(10:b509d717:::benchmark_data_smithi049_176811_object42787:head)])", 2024-09-21T14:05:08.038 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:05:08.014109+0000", 2024-09-21T14:05:08.038 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.00069407099999999999, 2024-09-21T14:05:08.038 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001009872, 2024-09-21T14:05:08.038 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:05:08.038 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:05:08.038 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-21T14:05:08.038 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:05:08.038 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.039 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:05:08.039 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.014109+0000", 2024-09-21T14:05:08.039 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:08.039 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.039 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.039 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:05:08.039 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.014109+0000", 2024-09-21T14:05:08.039 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:08.039 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.039 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.039 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:05:08.039 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.014110+0000", 2024-09-21T14:05:08.039 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5400000000000002e-07 2024-09-21T14:05:08.039 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.039 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.040 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:05:08.040 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.014114+0000", 2024-09-21T14:05:08.040 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0759999999999996e-06 2024-09-21T14:05:08.040 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.040 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.040 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:05:08.040 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.014124+0000", 2024-09-21T14:05:08.040 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0558e-05 2024-09-21T14:05:08.040 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.040 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.040 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:05:08.040 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.014131+0000", 2024-09-21T14:05:08.040 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8519999999999998e-06 2024-09-21T14:05:08.040 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.041 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.041 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:05:08.041 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.015018+0000", 2024-09-21T14:05:08.041 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00088655799999999996 2024-09-21T14:05:08.041 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:08.041 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:05:08.041 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:08.041 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.041 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.041 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.16068.0:160759 10.a e420/409)", 2024-09-21T14:05:08.041 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:05:08.014190+0000", 2024-09-21T14:05:08.041 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.00061343400000000005, 2024-09-21T14:05:08.041 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00096836399999999999, 2024-09-21T14:05:08.041 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:05:08.042 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:05:08.042 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "queued for pg", 2024-09-21T14:05:08.042 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:05:08.042 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.042 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:05:08.042 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.014190+0000", 2024-09-21T14:05:08.042 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:08.042 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.042 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.042 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:05:08.042 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.014190+0000", 2024-09-21T14:05:08.042 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:08.042 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.042 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.043 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:05:08.043 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.014191+0000", 2024-09-21T14:05:08.043 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5199999999999997e-07 2024-09-21T14:05:08.043 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.043 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.043 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:05:08.043 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.014193+0000", 2024-09-21T14:05:08.043 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0770000000000002e-06 2024-09-21T14:05:08.043 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.043 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.043 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:05:08.043 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.014199+0000", 2024-09-21T14:05:08.043 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2280000000000003e-06 2024-09-21T14:05:08.043 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.043 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.044 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:05:08.044 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.014203+0000", 2024-09-21T14:05:08.044 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9339999999999999e-06 2024-09-21T14:05:08.044 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:08.044 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:05:08.044 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:08.044 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.044 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.044 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:160764 10.b 10:de1b77da:::benchmark_data_smithi049_176811_object33187:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e420)", 2024-09-21T14:05:08.044 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:05:08.014258+0000", 2024-09-21T14:05:08.044 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.00054566800000000002, 2024-09-21T14:05:08.044 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00094097699999999996, 2024-09-21T14:05:08.044 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:05:08.044 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:05:08.045 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-21T14:05:08.045 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:05:08.045 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:05:08.045 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:05:08.045 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 160764 2024-09-21T14:05:08.045 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.045 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:05:08.045 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.045 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:05:08.045 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.014258+0000", 2024-09-21T14:05:08.045 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:08.045 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.045 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.045 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:05:08.045 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.014258+0000", 2024-09-21T14:05:08.046 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:08.046 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.046 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.046 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:05:08.046 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.014259+0000", 2024-09-21T14:05:08.046 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.482e-06 2024-09-21T14:05:08.046 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.046 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.046 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:05:08.046 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.014263+0000", 2024-09-21T14:05:08.046 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4529999999999998e-06 2024-09-21T14:05:08.046 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.046 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.047 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:05:08.047 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.014271+0000", 2024-09-21T14:05:08.047 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5860000000000005e-06 2024-09-21T14:05:08.047 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.047 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.047 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:05:08.047 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.014277+0000", 2024-09-21T14:05:08.047 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4530000000000001e-06 2024-09-21T14:05:08.047 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.047 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.047 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:05:08.047 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.014614+0000", 2024-09-21T14:05:08.047 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00033714500000000001 2024-09-21T14:05:08.047 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.047 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.048 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:05:08.048 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.014717+0000", 2024-09-21T14:05:08.048 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000103414 2024-09-21T14:05:08.048 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.048 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.048 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-21T14:05:08.048 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.014822+0000", 2024-09-21T14:05:08.048 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010516100000000001 2024-09-21T14:05:08.048 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:08.048 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:05:08.048 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:08.048 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.048 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.048 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:160765 10.d 10.d69727dd (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e420)", 2024-09-21T14:05:08.049 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:05:08.014285+0000", 2024-09-21T14:05:08.049 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.00051793, 2024-09-21T14:05:08.049 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00096708299999999998, 2024-09-21T14:05:08.049 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:05:08.049 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:05:08.049 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "queued for pg", 2024-09-21T14:05:08.049 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:05:08.049 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:05:08.049 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:05:08.049 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 160765 2024-09-21T14:05:08.049 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.049 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:05:08.049 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.049 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:05:08.050 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.014285+0000", 2024-09-21T14:05:08.050 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:08.050 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.050 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.050 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:05:08.050 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.014285+0000", 2024-09-21T14:05:08.050 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:08.050 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.050 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.050 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:05:08.050 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.014286+0000", 2024-09-21T14:05:08.050 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9400000000000001e-07 2024-09-21T14:05:08.050 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.051 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.051 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:05:08.051 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.014288+0000", 2024-09-21T14:05:08.051 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.829e-06 2024-09-21T14:05:08.051 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.051 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.051 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:05:08.051 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.014293+0000", 2024-09-21T14:05:08.051 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1200000000000001e-06 2024-09-21T14:05:08.051 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.051 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.051 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:05:08.051 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.014298+0000", 2024-09-21T14:05:08.051 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9439999999999996e-06 2024-09-21T14:05:08.051 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:08.052 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:05:08.052 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:08.052 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.052 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.052 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:160750 10.0 10:021a79dc:::benchmark_data_smithi049_176811_object33173:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e420)", 2024-09-21T14:05:08.052 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:05:08.000004+0000", 2024-09-21T14:05:08.052 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.014799089, 2024-09-21T14:05:08.052 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.015291856, 2024-09-21T14:05:08.052 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:05:08.052 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:05:08.052 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-21T14:05:08.052 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:05:08.052 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:05:08.052 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:05:08.053 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 160750 2024-09-21T14:05:08.053 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.053 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:05:08.053 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.053 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:05:08.053 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.000004+0000", 2024-09-21T14:05:08.053 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:08.053 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.053 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.053 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:05:08.053 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.000004+0000", 2024-09-21T14:05:08.053 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:08.053 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.053 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.054 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:05:08.054 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.000006+0000", 2024-09-21T14:05:08.054 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3170000000000001e-06 2024-09-21T14:05:08.054 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.054 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.054 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:05:08.054 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.000010+0000", 2024-09-21T14:05:08.054 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6190000000000002e-06 2024-09-21T14:05:08.054 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.054 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.054 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:05:08.054 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.000021+0000", 2024-09-21T14:05:08.054 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0803e-05 2024-09-21T14:05:08.054 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.055 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.055 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:05:08.055 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.000028+0000", 2024-09-21T14:05:08.055 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9090000000000003e-06 2024-09-21T14:05:08.055 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.055 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.055 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:05:08.055 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.002740+0000", 2024-09-21T14:05:08.055 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0027122940000000001 2024-09-21T14:05:08.055 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.055 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.055 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:05:08.055 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.002872+0000", 2024-09-21T14:05:08.055 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013138700000000001 2024-09-21T14:05:08.055 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.056 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.056 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,4,7", 2024-09-21T14:05:08.056 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.003041+0000", 2024-09-21T14:05:08.056 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00016974000000000001 2024-09-21T14:05:08.056 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.056 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.056 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:05:08.056 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.011417+0000", 2024-09-21T14:05:08.056 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0083754759999999998 2024-09-21T14:05:08.056 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.056 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.056 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:05:08.056 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.011534+0000", 2024-09-21T14:05:08.056 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000116765 2024-09-21T14:05:08.057 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:08.057 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:08.057 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:05:08.057 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:08.013860+0000", 2024-09-21T14:05:08.057 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0023259919999999998 2024-09-21T14:05:08.057 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:08.057 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:05:08.057 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:08.057 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:08.057 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2024-09-21T14:05:08.057 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 10 2024-09-21T14:05:08.057 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T14:05:08.058 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:05:08.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:08.095 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:08.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:05:08.195 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:08.233 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:08.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:08.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:08.334 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:05:08.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:08.403 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:08.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:05:08.412 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:05:08.504 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:05:08.554 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:08.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:08.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:05:08.658 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:05:08.693 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:08.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:08.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:05:08.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:08 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:05:08.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:08 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:05:08.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:08 smithi187 ceph-mon[33032]: osdmap e420: 8 total, 8 up, 8 in 2024-09-21T14:05:08.795 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:05:08.843 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:08.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:08.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:05:08.923 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:08 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:05:08.923 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:08 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:05:08.923 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:08 smithi049 ceph-mon[40489]: osdmap e420: 8 total, 8 up, 8 in 2024-09-21T14:05:08.923 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:08 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:05:08.923 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:08 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:05:08.923 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:08 smithi049 ceph-mon[30080]: osdmap e420: 8 total, 8 up, 8 in 2024-09-21T14:05:08.945 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:08.980 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:08.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:08.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:09.057 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T14:05:09.082 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:05:09.118 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:09.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:09.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:05:09.220 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:09.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:09.265 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:09.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:09.366 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:05:09.399 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:09.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:09.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:05:09.500 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:05:09.500 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:09.572 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:09.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:09.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:09.665 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-09-21T14:05:09.673 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:09.704 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:09.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:09.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:09.806 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:05:09.841 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:09.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:09.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:05:09.923 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:09 smithi187 ceph-mon[33032]: pgmap v819: 158 pgs: 4 active+remapped+backfilling, 154 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 891 op/s; 14403/293211 objects misplaced (4.912%); 19 MiB/s, 309 objects/s recovering 2024-09-21T14:05:09.924 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:09 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/29082191' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:05:09.924 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:09 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:05:09.943 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:09.993 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:09.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:09.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:10.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:09 smithi049 ceph-mon[40489]: pgmap v819: 158 pgs: 4 active+remapped+backfilling, 154 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 891 op/s; 14403/293211 objects misplaced (4.912%); 19 MiB/s, 309 objects/s recovering 2024-09-21T14:05:10.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:09 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/29082191' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:05:10.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:09 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:05:10.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:09 smithi049 ceph-mon[30080]: pgmap v819: 158 pgs: 4 active+remapped+backfilling, 154 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 891 op/s; 14403/293211 objects misplaced (4.912%); 19 MiB/s, 309 objects/s recovering 2024-09-21T14:05:10.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:09 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/29082191' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:05:10.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:09 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:05:10.095 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:05:10.130 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:10.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:10.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:05:10.194 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:05:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:05:10.231 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:05:10.272 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:10.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:10.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:05:10.367 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T14:05:10.374 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:10.405 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:10.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:10.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:10.415 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 4, 6, 2, 3, 7, 5] 2024-09-21T14:05:10.416 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T14:05:10.416 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-21T14:05:10.416 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd dump -f json-pretty 2024-09-21T14:05:10.506 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:05:10.523 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T14:05:10.523 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-21T14:05:10.523 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-21T14:05:10.523 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-21T14:05:10.523 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T14:05:10.538 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:10.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:10.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:05:10.639 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:05:10.670 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:10.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:10.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:05:10.695 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:10 smithi049 ceph-mon[30080]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:05:10.695 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:10 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T14:05:10.695 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:10 smithi049 ceph-mon[30080]: osdmap e421: 8 total, 8 up, 8 in 2024-09-21T14:05:10.695 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:10 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:05:10.695 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:05:10 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:05:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:05:10.772 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:05:10.826 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:10.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:10.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:05:10.854 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:05:10.857 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:05:10.928 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:05:10.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:10 smithi187 ceph-mon[33032]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:05:10.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:10 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T14:05:10.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:10 smithi187 ceph-mon[33032]: osdmap e421: 8 total, 8 up, 8 in 2024-09-21T14:05:10.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:10 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:05:10.969 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:10.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:10.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:05:10.977 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:10 smithi049 ceph-mon[40489]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:05:10.977 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:10 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T14:05:10.977 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:10 smithi049 ceph-mon[40489]: osdmap e421: 8 total, 8 up, 8 in 2024-09-21T14:05:10.977 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:10 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:05:11.071 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:05:11.110 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:11.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:11.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:05:11.212 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:11.246 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:11.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:11.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:11.347 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:05:11.390 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:11.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:11.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:05:11.444 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 14:05:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=sqlstore.transactions t=2024-09-21T14:05:10.945781434Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-21T14:05:11.444 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 14:05:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=sqlstore.transactions t=2024-09-21T14:05:10.957119642Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-09-21T14:05:11.444 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 14:05:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=sqlstore.transactions t=2024-09-21T14:05:10.968048777Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=2 code="database is locked" 2024-09-21T14:05:11.445 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 14:05:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=sqlstore.transactions t=2024-09-21T14:05:10.979245706Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=3 code="database is locked" 2024-09-21T14:05:11.445 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 14:05:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=sqlstore.transactions t=2024-09-21T14:05:10.990777774Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=4 code="database is locked" 2024-09-21T14:05:11.492 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:11.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:11.535 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:11.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:11.636 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:05:11.668 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:11.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:11.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:05:11.774 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:05:11.789 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:05:11.789 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:05:11.812 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:11.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:11.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:05:11.830 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:05:11.830 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:05:11.861 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:05:11.862 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:05:11.881 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:05:11.881 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:05:11.914 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:05:11.924 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:05:11.924 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:05:11.951 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:11.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:11.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:05:11.961 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:05:11.961 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:05:11.961 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:11 smithi049 ceph-mon[40489]: pgmap v821: 158 pgs: 4 active+remapped+backfilling, 154 active+clean; 5.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 797 B/s rd, 0 B/s wr, 1.40k op/s; 10983/282669 objects misplaced (3.885%); 51 MiB/s, 810 objects/s recovering 2024-09-21T14:05:11.961 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:11 smithi049 ceph-mon[40489]: 3.1f scrub starts 2024-09-21T14:05:11.961 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:11 smithi049 ceph-mon[40489]: 3.1f scrub ok 2024-09-21T14:05:11.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:11 smithi049 ceph-mon[30080]: pgmap v821: 158 pgs: 4 active+remapped+backfilling, 154 active+clean; 5.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 797 B/s rd, 0 B/s wr, 1.40k op/s; 10983/282669 objects misplaced (3.885%); 51 MiB/s, 810 objects/s recovering 2024-09-21T14:05:11.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:11 smithi049 ceph-mon[30080]: 3.1f scrub starts 2024-09-21T14:05:11.962 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:11 smithi049 ceph-mon[30080]: 3.1f scrub ok 2024-09-21T14:05:11.996 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:05:11.996 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:05:12.050 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:05:12.050 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:05:12.054 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:12.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:12.089 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:12.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:12.186 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T14:05:12.186 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T14:05:12.186 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 421, 2024-09-21T14:05:12.186 INFO:teuthology.orchestra.run.smithi049.stdout: "fsid": "40a3d2e8-781f-11ef-baf6-efdc52797490", 2024-09-21T14:05:12.187 INFO:teuthology.orchestra.run.smithi049.stdout: "created": "2024-09-21T13:42:57.838275+0000", 2024-09-21T14:05:12.187 INFO:teuthology.orchestra.run.smithi049.stdout: "modified": "2024-09-21T14:05:09.657918+0000", 2024-09-21T14:05:12.187 INFO:teuthology.orchestra.run.smithi049.stdout: "last_up_change": "2024-09-21T14:01:40.158454+0000", 2024-09-21T14:05:12.187 INFO:teuthology.orchestra.run.smithi049.stdout: "last_in_change": "2024-09-21T13:58:16.730861+0000", 2024-09-21T14:05:12.187 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-21T14:05:12.187 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_num": 5804032, 2024-09-21T14:05:12.187 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_set": [ 2024-09-21T14:05:12.187 INFO:teuthology.orchestra.run.smithi049.stdout: "nodeep-scrub", 2024-09-21T14:05:12.187 INFO:teuthology.orchestra.run.smithi049.stdout: "pglog_hardlimit", 2024-09-21T14:05:12.187 INFO:teuthology.orchestra.run.smithi049.stdout: "purged_snapdirs", 2024-09-21T14:05:12.187 INFO:teuthology.orchestra.run.smithi049.stdout: "recovery_deletes", 2024-09-21T14:05:12.187 INFO:teuthology.orchestra.run.smithi049.stdout: "sortbitwise" 2024-09-21T14:05:12.187 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T14:05:12.188 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_version": 22, 2024-09-21T14:05:12.188 INFO:teuthology.orchestra.run.smithi049.stdout: "full_ratio": 0.94999998807907104, 2024-09-21T14:05:12.188 INFO:teuthology.orchestra.run.smithi049.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-21T14:05:12.188 INFO:teuthology.orchestra.run.smithi049.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-21T14:05:12.188 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_snapshot": "", 2024-09-21T14:05:12.188 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_max": 10, 2024-09-21T14:05:12.188 INFO:teuthology.orchestra.run.smithi049.stdout: "max_osd": 8, 2024-09-21T14:05:12.188 INFO:teuthology.orchestra.run.smithi049.stdout: "require_min_compat_client": "luminous", 2024-09-21T14:05:12.188 INFO:teuthology.orchestra.run.smithi049.stdout: "min_compat_client": "luminous", 2024-09-21T14:05:12.188 INFO:teuthology.orchestra.run.smithi049.stdout: "require_osd_release": "squid", 2024-09-21T14:05:12.188 INFO:teuthology.orchestra.run.smithi049.stdout: "allow_crimson": false, 2024-09-21T14:05:12.188 INFO:teuthology.orchestra.run.smithi049.stdout: "pools": [ 2024-09-21T14:05:12.188 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.189 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 1, 2024-09-21T14:05:12.189 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": ".mgr", 2024-09-21T14:05:12.189 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:46:21.096244+0000", 2024-09-21T14:05:12.189 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T14:05:12.189 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T14:05:12.189 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T14:05:12.189 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T14:05:12.189 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T14:05:12.189 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T14:05:12.189 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T14:05:12.189 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T14:05:12.189 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T14:05:12.189 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T14:05:12.189 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T14:05:12.189 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T14:05:12.190 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T14:05:12.190 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 11, 2024-09-21T14:05:12.190 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 11, 2024-09-21T14:05:12.190 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 11, 2024-09-21T14:05:12.190 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 11, 2024-09-21T14:05:12.190 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 11, 2024-09-21T14:05:12.190 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T14:05:12.190 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T14:05:12.190 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T14:05:12.190 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T14:05:12.190 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T14:05:12.190 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T14:05:12.190 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T14:05:12.191 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.191 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "320", 2024-09-21T14:05:12.191 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T14:05:12.191 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T14:05:12.191 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "318", 2024-09-21T14:05:12.191 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T14:05:12.191 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T14:05:12.191 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T14:05:12.191 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T14:05:12.191 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T14:05:12.191 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T14:05:12.191 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T14:05:12.191 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T14:05:12.192 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T14:05:12.192 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T14:05:12.192 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T14:05:12.192 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T14:05:12.192 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T14:05:12.192 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T14:05:12.192 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T14:05:12.192 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T14:05:12.192 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T14:05:12.192 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T14:05:12.192 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T14:05:12.192 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T14:05:12.192 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T14:05:12.193 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T14:05:12.193 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T14:05:12.193 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.193 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T14:05:12.193 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T14:05:12.193 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T14:05:12.193 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T14:05:12.193 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T14:05:12.193 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T14:05:12.193 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T14:05:12.193 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T14:05:12.193 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T14:05:12.193 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T14:05:12.193 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T14:05:12.194 INFO:teuthology.orchestra.run.smithi049.stdout: "options": { 2024-09-21T14:05:12.194 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_max": 32, 2024-09-21T14:05:12.194 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_min": 1 2024-09-21T14:05:12.194 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.194 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T14:05:12.194 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr": {} 2024-09-21T14:05:12.194 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.194 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T14:05:12.194 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T14:05:12.194 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2.1800000667572021, 2024-09-21T14:05:12.194 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2.1800000667572021, 2024-09-21T14:05:12.194 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1, 2024-09-21T14:05:12.194 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2.1800000667572021, 2024-09-21T14:05:12.195 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2.1800000667572021, 2024-09-21T14:05:12.195 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.97835731506347656, 2024-09-21T14:05:12.195 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.98000001907348633, 2024-09-21T14:05:12.195 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.98000001907348633 2024-09-21T14:05:12.195 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:12.195 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.195 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.195 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 2, 2024-09-21T14:05:12.195 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "datapool", 2024-09-21T14:05:12.195 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:48:55.540712+0000", 2024-09-21T14:05:12.195 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 8193, 2024-09-21T14:05:12.195 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T14:05:12.195 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T14:05:12.195 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T14:05:12.196 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T14:05:12.196 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T14:05:12.196 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T14:05:12.196 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T14:05:12.196 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T14:05:12.196 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T14:05:12.196 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T14:05:12.196 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T14:05:12.196 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T14:05:12.196 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 3, 2024-09-21T14:05:12.196 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 3, 2024-09-21T14:05:12.196 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 3, 2024-09-21T14:05:12.196 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 3, 2024-09-21T14:05:12.196 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 3, 2024-09-21T14:05:12.197 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T14:05:12.197 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T14:05:12.197 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T14:05:12.197 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T14:05:12.197 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T14:05:12.197 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T14:05:12.197 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T14:05:12.197 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.197 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "63", 2024-09-21T14:05:12.197 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T14:05:12.197 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T14:05:12.197 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T14:05:12.197 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T14:05:12.197 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T14:05:12.198 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 2, 2024-09-21T14:05:12.198 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 63, 2024-09-21T14:05:12.198 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T14:05:12.198 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T14:05:12.198 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T14:05:12.198 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T14:05:12.198 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T14:05:12.198 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T14:05:12.198 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T14:05:12.198 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T14:05:12.198 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T14:05:12.198 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T14:05:12.198 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T14:05:12.199 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T14:05:12.199 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T14:05:12.199 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T14:05:12.199 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T14:05:12.199 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T14:05:12.199 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T14:05:12.199 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T14:05:12.199 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T14:05:12.199 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.199 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T14:05:12.199 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T14:05:12.199 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T14:05:12.199 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T14:05:12.199 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T14:05:12.200 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T14:05:12.200 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T14:05:12.200 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T14:05:12.200 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T14:05:12.200 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T14:05:12.200 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T14:05:12.200 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T14:05:12.200 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T14:05:12.200 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd": {} 2024-09-21T14:05:12.200 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.200 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T14:05:12.200 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T14:05:12.200 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2.6500000953674316, 2024-09-21T14:05:12.200 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2.6500000953674316, 2024-09-21T14:05:12.200 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 0.87999999523162842, 2024-09-21T14:05:12.201 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2.3299999237060547, 2024-09-21T14:05:12.201 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2.3299999237060547, 2024-09-21T14:05:12.201 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.97835731506347656, 2024-09-21T14:05:12.201 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.98000001907348633, 2024-09-21T14:05:12.201 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.98000001907348633 2024-09-21T14:05:12.201 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:12.201 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.201 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.201 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 3, 2024-09-21T14:05:12.201 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": ".rgw.root", 2024-09-21T14:05:12.201 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:48:56.417456+0000", 2024-09-21T14:05:12.201 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T14:05:12.201 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T14:05:12.201 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T14:05:12.202 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T14:05:12.202 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T14:05:12.202 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T14:05:12.202 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T14:05:12.202 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T14:05:12.202 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T14:05:12.202 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T14:05:12.202 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T14:05:12.202 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T14:05:12.202 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T14:05:12.202 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 32, 2024-09-21T14:05:12.202 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 32, 2024-09-21T14:05:12.202 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 32, 2024-09-21T14:05:12.203 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 32, 2024-09-21T14:05:12.203 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 32, 2024-09-21T14:05:12.203 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T14:05:12.203 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:12.204 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:11 smithi187 ceph-mon[33032]: pgmap v821: 158 pgs: 4 active+remapped+backfilling, 154 active+clean; 5.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 797 B/s rd, 0 B/s wr, 1.40k op/s; 10983/282669 objects misplaced (3.885%); 51 MiB/s, 810 objects/s recovering 2024-09-21T14:05:12.204 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:11 smithi187 ceph-mon[33032]: 3.1f scrub starts 2024-09-21T14:05:12.204 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:11 smithi187 ceph-mon[33032]: 3.1f scrub ok 2024-09-21T14:05:12.205 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T14:05:12.205 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T14:05:12.205 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T14:05:12.205 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T14:05:12.205 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T14:05:12.205 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T14:05:12.205 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.205 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "62", 2024-09-21T14:05:12.205 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T14:05:12.205 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T14:05:12.206 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T14:05:12.206 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T14:05:12.206 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T14:05:12.206 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T14:05:12.206 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T14:05:12.206 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T14:05:12.206 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T14:05:12.206 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T14:05:12.206 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T14:05:12.206 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T14:05:12.206 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T14:05:12.206 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T14:05:12.206 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T14:05:12.207 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T14:05:12.207 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T14:05:12.207 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T14:05:12.207 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T14:05:12.207 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T14:05:12.207 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T14:05:12.207 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T14:05:12.207 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T14:05:12.207 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T14:05:12.207 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T14:05:12.207 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T14:05:12.207 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.207 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T14:05:12.208 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T14:05:12.208 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T14:05:12.208 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T14:05:12.208 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T14:05:12.208 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T14:05:12.208 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T14:05:12.208 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T14:05:12.208 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T14:05:12.208 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T14:05:12.208 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T14:05:12.208 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T14:05:12.208 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T14:05:12.208 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw": {} 2024-09-21T14:05:12.209 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.209 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T14:05:12.209 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T14:05:12.209 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 1.75, 2024-09-21T14:05:12.209 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 1.75, 2024-09-21T14:05:12.209 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1, 2024-09-21T14:05:12.209 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 1.75, 2024-09-21T14:05:12.209 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 1.75, 2024-09-21T14:05:12.209 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.97835731506347656, 2024-09-21T14:05:12.209 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.98000001907348633, 2024-09-21T14:05:12.209 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.98000001907348633 2024-09-21T14:05:12.209 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:12.209 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.209 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.210 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 4, 2024-09-21T14:05:12.210 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "default.rgw.log", 2024-09-21T14:05:12.210 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:48:58.128455+0000", 2024-09-21T14:05:12.210 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T14:05:12.210 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T14:05:12.210 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T14:05:12.210 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T14:05:12.210 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T14:05:12.210 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T14:05:12.210 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T14:05:12.210 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T14:05:12.210 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T14:05:12.210 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T14:05:12.210 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T14:05:12.210 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T14:05:12.211 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T14:05:12.211 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 32, 2024-09-21T14:05:12.211 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 32, 2024-09-21T14:05:12.211 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 32, 2024-09-21T14:05:12.211 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 32, 2024-09-21T14:05:12.211 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 32, 2024-09-21T14:05:12.211 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T14:05:12.211 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T14:05:12.211 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T14:05:12.211 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T14:05:12.211 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T14:05:12.211 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T14:05:12.211 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T14:05:12.212 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.212 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "64", 2024-09-21T14:05:12.212 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T14:05:12.212 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T14:05:12.212 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T14:05:12.212 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T14:05:12.212 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T14:05:12.212 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T14:05:12.212 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T14:05:12.212 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T14:05:12.213 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T14:05:12.213 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T14:05:12.213 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T14:05:12.213 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T14:05:12.213 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T14:05:12.213 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T14:05:12.213 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T14:05:12.213 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T14:05:12.213 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T14:05:12.213 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T14:05:12.213 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T14:05:12.213 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T14:05:12.213 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T14:05:12.214 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T14:05:12.214 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T14:05:12.214 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T14:05:12.214 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T14:05:12.214 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T14:05:12.214 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.214 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T14:05:12.214 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T14:05:12.214 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T14:05:12.214 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T14:05:12.214 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T14:05:12.214 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T14:05:12.214 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T14:05:12.215 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T14:05:12.215 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T14:05:12.215 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T14:05:12.215 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T14:05:12.215 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T14:05:12.215 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T14:05:12.215 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw": {} 2024-09-21T14:05:12.215 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.215 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T14:05:12.215 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T14:05:12.215 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2.5, 2024-09-21T14:05:12.215 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2.5, 2024-09-21T14:05:12.215 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1, 2024-09-21T14:05:12.215 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2.5, 2024-09-21T14:05:12.216 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2.5, 2024-09-21T14:05:12.216 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.97835731506347656, 2024-09-21T14:05:12.216 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.98000001907348633, 2024-09-21T14:05:12.216 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.98000001907348633 2024-09-21T14:05:12.216 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:12.216 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.216 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.216 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 5, 2024-09-21T14:05:12.216 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "default.rgw.control", 2024-09-21T14:05:12.216 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:49:00.069361+0000", 2024-09-21T14:05:12.216 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T14:05:12.216 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T14:05:12.216 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T14:05:12.216 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T14:05:12.217 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T14:05:12.217 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T14:05:12.217 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T14:05:12.217 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T14:05:12.217 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T14:05:12.217 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T14:05:12.217 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T14:05:12.217 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T14:05:12.217 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T14:05:12.217 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 32, 2024-09-21T14:05:12.217 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 32, 2024-09-21T14:05:12.217 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 32, 2024-09-21T14:05:12.217 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 32, 2024-09-21T14:05:12.217 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 32, 2024-09-21T14:05:12.217 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T14:05:12.218 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T14:05:12.218 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T14:05:12.218 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T14:05:12.218 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T14:05:12.218 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T14:05:12.218 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T14:05:12.218 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.218 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "66", 2024-09-21T14:05:12.218 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T14:05:12.218 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T14:05:12.218 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T14:05:12.218 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T14:05:12.218 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T14:05:12.219 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T14:05:12.219 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T14:05:12.219 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T14:05:12.219 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T14:05:12.219 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T14:05:12.219 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T14:05:12.219 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T14:05:12.219 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T14:05:12.219 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T14:05:12.219 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T14:05:12.219 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T14:05:12.219 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T14:05:12.219 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T14:05:12.219 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T14:05:12.220 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T14:05:12.220 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T14:05:12.220 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T14:05:12.220 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T14:05:12.220 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T14:05:12.220 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T14:05:12.220 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T14:05:12.220 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.220 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T14:05:12.220 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T14:05:12.220 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T14:05:12.220 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T14:05:12.220 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T14:05:12.220 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T14:05:12.221 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T14:05:12.221 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T14:05:12.221 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T14:05:12.221 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T14:05:12.221 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T14:05:12.221 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T14:05:12.221 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T14:05:12.221 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw": {} 2024-09-21T14:05:12.221 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.221 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T14:05:12.221 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T14:05:12.221 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 1.5, 2024-09-21T14:05:12.221 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 1.5, 2024-09-21T14:05:12.222 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1, 2024-09-21T14:05:12.222 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 1.5, 2024-09-21T14:05:12.222 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 1.5, 2024-09-21T14:05:12.222 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.97835731506347656, 2024-09-21T14:05:12.222 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.98000001907348633, 2024-09-21T14:05:12.222 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.98000001907348633 2024-09-21T14:05:12.222 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:12.222 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.222 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.222 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 6, 2024-09-21T14:05:12.222 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "default.rgw.meta", 2024-09-21T14:05:12.222 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:49:02.127399+0000", 2024-09-21T14:05:12.223 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T14:05:12.223 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T14:05:12.223 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T14:05:12.223 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T14:05:12.223 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T14:05:12.223 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T14:05:12.223 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T14:05:12.223 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T14:05:12.223 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T14:05:12.223 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T14:05:12.223 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T14:05:12.223 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T14:05:12.223 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T14:05:12.223 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 32, 2024-09-21T14:05:12.224 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 32, 2024-09-21T14:05:12.224 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 32, 2024-09-21T14:05:12.224 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 32, 2024-09-21T14:05:12.224 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 32, 2024-09-21T14:05:12.224 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T14:05:12.224 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T14:05:12.224 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T14:05:12.224 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T14:05:12.224 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T14:05:12.224 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T14:05:12.224 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T14:05:12.224 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.224 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "68", 2024-09-21T14:05:12.225 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T14:05:12.225 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T14:05:12.225 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T14:05:12.225 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T14:05:12.225 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T14:05:12.225 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T14:05:12.225 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T14:05:12.225 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T14:05:12.225 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T14:05:12.225 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T14:05:12.225 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T14:05:12.225 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T14:05:12.225 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T14:05:12.225 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T14:05:12.226 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T14:05:12.226 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T14:05:12.226 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T14:05:12.226 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T14:05:12.226 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T14:05:12.226 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T14:05:12.226 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T14:05:12.226 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T14:05:12.226 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T14:05:12.226 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T14:05:12.226 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T14:05:12.226 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T14:05:12.226 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.226 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T14:05:12.227 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T14:05:12.227 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T14:05:12.227 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T14:05:12.227 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T14:05:12.227 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T14:05:12.227 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T14:05:12.227 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T14:05:12.227 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T14:05:12.227 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T14:05:12.227 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T14:05:12.227 INFO:teuthology.orchestra.run.smithi049.stdout: "options": { 2024-09-21T14:05:12.227 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_bias": 4 2024-09-21T14:05:12.227 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.227 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T14:05:12.228 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw": {} 2024-09-21T14:05:12.228 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.228 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T14:05:12.228 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T14:05:12.228 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2, 2024-09-21T14:05:12.228 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2, 2024-09-21T14:05:12.228 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1, 2024-09-21T14:05:12.228 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2, 2024-09-21T14:05:12.228 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2, 2024-09-21T14:05:12.228 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.97835731506347656, 2024-09-21T14:05:12.228 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.98000001907348633, 2024-09-21T14:05:12.228 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.98000001907348633 2024-09-21T14:05:12.228 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:12.228 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.229 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.229 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 10, 2024-09-21T14:05:12.229 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "unique_pool_3", 2024-09-21T14:05:12.229 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T14:02:55.287153+0000", 2024-09-21T14:05:12.229 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T14:05:12.229 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T14:05:12.229 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T14:05:12.229 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T14:05:12.229 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T14:05:12.229 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T14:05:12.229 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T14:05:12.229 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T14:05:12.229 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T14:05:12.229 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T14:05:12.230 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T14:05:12.230 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T14:05:12.230 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T14:05:12.230 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 16, 2024-09-21T14:05:12.230 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 16, 2024-09-21T14:05:12.230 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 16, 2024-09-21T14:05:12.230 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 16, 2024-09-21T14:05:12.230 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 16, 2024-09-21T14:05:12.230 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T14:05:12.230 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T14:05:12.230 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T14:05:12.230 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T14:05:12.230 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T14:05:12.231 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T14:05:12.231 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T14:05:12.231 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.231 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "415", 2024-09-21T14:05:12.231 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T14:05:12.231 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T14:05:12.231 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T14:05:12.231 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T14:05:12.231 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T14:05:12.231 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T14:05:12.231 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T14:05:12.231 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T14:05:12.232 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T14:05:12.232 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T14:05:12.232 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T14:05:12.232 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T14:05:12.232 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T14:05:12.232 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T14:05:12.232 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T14:05:12.232 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T14:05:12.232 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T14:05:12.232 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T14:05:12.232 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T14:05:12.233 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T14:05:12.233 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T14:05:12.233 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T14:05:12.233 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T14:05:12.233 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T14:05:12.233 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T14:05:12.233 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T14:05:12.233 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.233 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T14:05:12.233 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T14:05:12.233 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T14:05:12.233 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T14:05:12.233 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T14:05:12.233 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T14:05:12.234 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T14:05:12.234 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T14:05:12.234 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T14:05:12.234 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T14:05:12.234 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T14:05:12.234 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T14:05:12.234 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T14:05:12.234 INFO:teuthology.orchestra.run.smithi049.stdout: "rados": {} 2024-09-21T14:05:12.234 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.234 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T14:05:12.234 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T14:05:12.234 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 3, 2024-09-21T14:05:12.234 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 3, 2024-09-21T14:05:12.234 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1, 2024-09-21T14:05:12.235 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 3, 2024-09-21T14:05:12.235 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 3, 2024-09-21T14:05:12.235 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.97835731506347656, 2024-09-21T14:05:12.235 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.98000001907348633, 2024-09-21T14:05:12.235 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.98000001907348633 2024-09-21T14:05:12.235 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:12.235 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:12.235 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T14:05:12.235 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-09-21T14:05:12.235 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.235 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 0, 2024-09-21T14:05:12.235 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "2595dca9-6219-4e0c-8728-751eb102e73a", 2024-09-21T14:05:12.235 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T14:05:12.235 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T14:05:12.236 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 0.6176910400390625, 2024-09-21T14:05:12.236 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T14:05:12.236 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T14:05:12.236 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T14:05:12.236 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 8, 2024-09-21T14:05:12.236 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 417, 2024-09-21T14:05:12.236 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T14:05:12.236 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T14:05:12.236 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T14:05:12.236 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:12.236 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.236 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:12.236 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6802", 2024-09-21T14:05:12.236 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T14:05:12.237 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.237 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.237 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:12.237 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6803", 2024-09-21T14:05:12.237 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T14:05:12.237 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:12.237 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:12.237 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.237 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T14:05:12.237 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:12.237 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.237 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:12.237 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6804", 2024-09-21T14:05:12.237 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T14:05:12.238 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.238 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.238 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:12.238 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6805", 2024-09-21T14:05:12.238 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T14:05:12.238 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:12.238 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:12.238 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.238 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T14:05:12.238 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:12.238 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.238 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:12.238 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6808", 2024-09-21T14:05:12.238 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T14:05:12.239 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.239 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.239 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:12.239 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6809", 2024-09-21T14:05:12.239 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T14:05:12.239 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:12.239 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:12.239 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.239 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T14:05:12.239 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:12.239 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.239 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:12.239 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6806", 2024-09-21T14:05:12.240 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T14:05:12.240 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.240 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.240 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:12.240 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6807", 2024-09-21T14:05:12.240 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T14:05:12.240 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:12.240 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:12.240 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.240 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6803/2095561412", 2024-09-21T14:05:12.240 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6805/2095561412", 2024-09-21T14:05:12.240 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6809/2095561412", 2024-09-21T14:05:12.240 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6807/2095561412", 2024-09-21T14:05:12.241 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T14:05:12.241 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T14:05:12.241 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T14:05:12.241 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:12.241 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.241 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.241 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 1, 2024-09-21T14:05:12.241 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "0ba327f9-5524-4a40-abcc-729ded0b1917", 2024-09-21T14:05:12.241 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T14:05:12.241 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T14:05:12.241 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T14:05:12.241 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T14:05:12.241 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T14:05:12.241 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T14:05:12.241 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 13, 2024-09-21T14:05:12.242 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 408, 2024-09-21T14:05:12.242 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T14:05:12.242 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T14:05:12.242 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T14:05:12.242 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:12.242 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.242 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:12.242 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6810", 2024-09-21T14:05:12.242 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T14:05:12.242 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.242 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.242 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:12.243 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6811", 2024-09-21T14:05:12.243 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T14:05:12.243 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:12.243 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:12.243 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.243 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T14:05:12.243 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:12.243 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.243 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:12.243 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6812", 2024-09-21T14:05:12.243 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T14:05:12.243 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.243 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.243 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:12.244 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6813", 2024-09-21T14:05:12.244 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T14:05:12.244 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:12.244 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:12.244 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.244 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T14:05:12.244 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:12.244 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.244 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:12.244 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6816", 2024-09-21T14:05:12.244 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T14:05:12.244 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.245 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.245 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:12.245 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6817", 2024-09-21T14:05:12.245 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T14:05:12.245 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:12.245 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:12.245 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.245 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T14:05:12.245 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:12.245 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.245 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:12.245 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6814", 2024-09-21T14:05:12.245 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T14:05:12.245 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.246 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.246 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:12.246 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6815", 2024-09-21T14:05:12.246 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T14:05:12.246 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:12.246 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:12.246 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.246 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6811/1382840527", 2024-09-21T14:05:12.246 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6813/1382840527", 2024-09-21T14:05:12.246 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6817/1382840527", 2024-09-21T14:05:12.246 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6815/1382840527", 2024-09-21T14:05:12.246 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T14:05:12.246 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T14:05:12.247 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T14:05:12.247 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:12.247 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.247 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.247 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 2, 2024-09-21T14:05:12.247 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "27a45750-204d-48fd-bb8f-ebb01273343d", 2024-09-21T14:05:12.247 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T14:05:12.247 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T14:05:12.247 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T14:05:12.247 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 0.8268585205078125, 2024-09-21T14:05:12.247 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 18, 2024-09-21T14:05:12.247 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 100, 2024-09-21T14:05:12.247 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 130, 2024-09-21T14:05:12.248 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 408, 2024-09-21T14:05:12.248 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 101, 2024-09-21T14:05:12.248 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T14:05:12.248 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T14:05:12.248 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:12.248 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.248 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:12.248 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6818", 2024-09-21T14:05:12.248 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T14:05:12.248 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.248 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.248 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:12.248 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6819", 2024-09-21T14:05:12.248 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T14:05:12.249 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:12.249 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:12.249 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.249 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T14:05:12.249 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:12.249 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.249 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:12.249 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6820", 2024-09-21T14:05:12.249 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T14:05:12.249 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.249 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.249 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:12.249 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6821", 2024-09-21T14:05:12.249 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T14:05:12.250 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:12.250 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:12.250 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.250 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T14:05:12.250 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:12.250 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.250 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:12.250 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6824", 2024-09-21T14:05:12.250 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T14:05:12.250 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.250 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.250 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:12.250 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6825", 2024-09-21T14:05:12.251 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T14:05:12.251 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:12.251 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:12.251 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.251 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T14:05:12.251 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:12.251 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.251 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:12.251 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6822", 2024-09-21T14:05:12.251 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T14:05:12.251 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.251 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.251 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:12.251 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6823", 2024-09-21T14:05:12.252 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T14:05:12.252 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:12.252 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:12.252 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.252 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6819/1945314741", 2024-09-21T14:05:12.252 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6821/1945314741", 2024-09-21T14:05:12.252 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6825/1945314741", 2024-09-21T14:05:12.252 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6823/1945314741", 2024-09-21T14:05:12.252 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T14:05:12.252 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T14:05:12.252 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T14:05:12.252 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:12.252 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.252 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.253 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 3, 2024-09-21T14:05:12.253 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "528bc390-3dc9-4f3b-ab75-d7412c9625db", 2024-09-21T14:05:12.253 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T14:05:12.253 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T14:05:12.253 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T14:05:12.253 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T14:05:12.253 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 27, 2024-09-21T14:05:12.253 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 214, 2024-09-21T14:05:12.253 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 240, 2024-09-21T14:05:12.253 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 408, 2024-09-21T14:05:12.253 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 215, 2024-09-21T14:05:12.253 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T14:05:12.253 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T14:05:12.253 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:12.254 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.254 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:12.254 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6826", 2024-09-21T14:05:12.254 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T14:05:12.254 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.254 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.254 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:12.254 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6827", 2024-09-21T14:05:12.254 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T14:05:12.254 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:12.254 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:12.254 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.254 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T14:05:12.255 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:12.255 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.255 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:12.255 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6828", 2024-09-21T14:05:12.255 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T14:05:12.255 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.255 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.255 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:12.255 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6829", 2024-09-21T14:05:12.255 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T14:05:12.255 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:12.255 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:12.255 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.256 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T14:05:12.256 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:12.256 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.256 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:12.256 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6832", 2024-09-21T14:05:12.256 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T14:05:12.256 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.256 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.256 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:12.256 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6833", 2024-09-21T14:05:12.256 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T14:05:12.256 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:12.256 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:12.256 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.257 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T14:05:12.257 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:12.257 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.257 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:12.257 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6830", 2024-09-21T14:05:12.257 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T14:05:12.257 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.257 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.257 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:12.257 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6831", 2024-09-21T14:05:12.257 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T14:05:12.257 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:12.257 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:12.257 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.258 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6827/886735721", 2024-09-21T14:05:12.258 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6829/886735721", 2024-09-21T14:05:12.258 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6833/886735721", 2024-09-21T14:05:12.258 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6831/886735721", 2024-09-21T14:05:12.258 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T14:05:12.258 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T14:05:12.258 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T14:05:12.258 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:12.258 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.258 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.258 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 4, 2024-09-21T14:05:12.258 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "6fc64698-2d37-4db0-bf3d-c7e612c45163", 2024-09-21T14:05:12.258 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T14:05:12.259 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T14:05:12.259 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 0.7888946533203125, 2024-09-21T14:05:12.259 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T14:05:12.259 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T14:05:12.259 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T14:05:12.259 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 33, 2024-09-21T14:05:12.259 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 409, 2024-09-21T14:05:12.259 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T14:05:12.259 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T14:05:12.259 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T14:05:12.259 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:12.259 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.259 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:12.259 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6800", 2024-09-21T14:05:12.259 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T14:05:12.260 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.260 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.260 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:12.260 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6801", 2024-09-21T14:05:12.260 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T14:05:12.260 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:12.260 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:12.260 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.260 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T14:05:12.260 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:12.260 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.260 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:12.260 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6802", 2024-09-21T14:05:12.260 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T14:05:12.261 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.261 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.261 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:12.261 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6803", 2024-09-21T14:05:12.261 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T14:05:12.261 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:12.261 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:12.261 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.261 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T14:05:12.261 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:12.261 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.261 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:12.261 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6806", 2024-09-21T14:05:12.261 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T14:05:12.262 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.262 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.262 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:12.262 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6807", 2024-09-21T14:05:12.262 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T14:05:12.262 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:12.262 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:12.262 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.262 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T14:05:12.262 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:12.262 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.262 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:12.262 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6804", 2024-09-21T14:05:12.263 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T14:05:12.263 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.263 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.263 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:12.263 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6805", 2024-09-21T14:05:12.263 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T14:05:12.263 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:12.263 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:12.263 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.263 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.187:6801/1702447776", 2024-09-21T14:05:12.263 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.187:6803/1702447776", 2024-09-21T14:05:12.263 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.187:6807/1702447776", 2024-09-21T14:05:12.263 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.187:6805/1702447776", 2024-09-21T14:05:12.263 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T14:05:12.264 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T14:05:12.264 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T14:05:12.264 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:12.264 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.264 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.264 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 5, 2024-09-21T14:05:12.264 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "8d7f62a1-7e92-4e06-9cad-86b96be514d8", 2024-09-21T14:05:12.264 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T14:05:12.264 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T14:05:12.264 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T14:05:12.264 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T14:05:12.264 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 39, 2024-09-21T14:05:12.264 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 324, 2024-09-21T14:05:12.264 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 345, 2024-09-21T14:05:12.265 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 408, 2024-09-21T14:05:12.265 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 325, 2024-09-21T14:05:12.265 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T14:05:12.265 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T14:05:12.265 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:12.265 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.265 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:12.265 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6808", 2024-09-21T14:05:12.265 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1595353965 2024-09-21T14:05:12.265 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.265 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.265 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:12.265 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6809", 2024-09-21T14:05:12.265 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1595353965 2024-09-21T14:05:12.266 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:12.266 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:12.266 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.266 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T14:05:12.266 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:12.266 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.266 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:12.266 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6810", 2024-09-21T14:05:12.266 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1595353965 2024-09-21T14:05:12.266 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.266 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.266 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:12.266 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6811", 2024-09-21T14:05:12.267 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1595353965 2024-09-21T14:05:12.267 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:12.267 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:12.267 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.267 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T14:05:12.267 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:12.267 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.267 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:12.267 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6814", 2024-09-21T14:05:12.267 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1595353965 2024-09-21T14:05:12.267 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.267 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.267 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:12.267 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6815", 2024-09-21T14:05:12.267 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1595353965 2024-09-21T14:05:12.268 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:12.268 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:12.268 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.268 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T14:05:12.268 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:12.268 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.268 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:12.268 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6812", 2024-09-21T14:05:12.268 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1595353965 2024-09-21T14:05:12.268 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.268 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.268 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:12.268 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6813", 2024-09-21T14:05:12.268 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1595353965 2024-09-21T14:05:12.269 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:12.269 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:12.269 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.269 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.187:6809/1595353965", 2024-09-21T14:05:12.269 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.187:6811/1595353965", 2024-09-21T14:05:12.269 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.187:6815/1595353965", 2024-09-21T14:05:12.269 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.187:6813/1595353965", 2024-09-21T14:05:12.269 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T14:05:12.269 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T14:05:12.269 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T14:05:12.269 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:12.269 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.269 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.269 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 6, 2024-09-21T14:05:12.270 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "3a7afd13-26ce-444c-9cc0-81ce2eab6ce5", 2024-09-21T14:05:12.270 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T14:05:12.270 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T14:05:12.270 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 0.9076080322265625, 2024-09-21T14:05:12.270 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T14:05:12.270 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T14:05:12.270 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T14:05:12.270 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 47, 2024-09-21T14:05:12.270 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 409, 2024-09-21T14:05:12.270 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T14:05:12.270 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T14:05:12.270 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T14:05:12.270 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:12.271 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.271 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:12.271 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6816", 2024-09-21T14:05:12.271 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T14:05:12.271 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.271 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.271 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:12.271 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6817", 2024-09-21T14:05:12.271 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T14:05:12.271 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:12.271 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:12.271 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.271 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T14:05:12.271 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:12.272 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.272 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:12.272 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6818", 2024-09-21T14:05:12.272 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T14:05:12.272 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.272 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.272 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:12.272 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6819", 2024-09-21T14:05:12.272 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T14:05:12.272 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:12.272 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:12.272 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.272 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T14:05:12.272 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:12.273 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.273 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:12.273 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6822", 2024-09-21T14:05:12.273 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T14:05:12.273 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.273 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.273 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:12.273 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6823", 2024-09-21T14:05:12.273 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T14:05:12.273 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:12.273 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:12.273 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.273 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T14:05:12.273 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:12.273 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.274 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:12.274 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6820", 2024-09-21T14:05:12.274 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T14:05:12.274 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.274 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.274 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:12.274 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6821", 2024-09-21T14:05:12.274 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T14:05:12.274 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:12.274 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:12.274 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.274 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.187:6817/2728426525", 2024-09-21T14:05:12.274 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.187:6819/2728426525", 2024-09-21T14:05:12.275 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.187:6823/2728426525", 2024-09-21T14:05:12.275 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.187:6821/2728426525", 2024-09-21T14:05:12.275 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T14:05:12.275 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T14:05:12.275 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T14:05:12.275 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:12.275 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.275 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.275 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 7, 2024-09-21T14:05:12.275 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "1e08d891-8747-402e-b4c2-aa3802ce8537", 2024-09-21T14:05:12.275 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T14:05:12.276 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T14:05:12.276 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T14:05:12.276 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T14:05:12.277 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 54, 2024-09-21T14:05:12.277 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 272, 2024-09-21T14:05:12.277 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 298, 2024-09-21T14:05:12.277 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 408, 2024-09-21T14:05:12.277 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 273, 2024-09-21T14:05:12.277 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T14:05:12.277 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T14:05:12.277 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:12.277 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.277 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:12.277 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6824", 2024-09-21T14:05:12.277 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 13894964 2024-09-21T14:05:12.277 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.278 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.278 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:12.278 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6825", 2024-09-21T14:05:12.278 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 13894964 2024-09-21T14:05:12.278 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:12.278 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:12.278 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.278 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T14:05:12.278 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:12.278 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.278 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:12.278 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6826", 2024-09-21T14:05:12.278 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 13894964 2024-09-21T14:05:12.278 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.279 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.279 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:12.279 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6827", 2024-09-21T14:05:12.279 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 13894964 2024-09-21T14:05:12.279 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:12.279 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:12.279 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.279 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T14:05:12.279 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:12.279 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.279 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:12.279 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6830", 2024-09-21T14:05:12.279 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 13894964 2024-09-21T14:05:12.279 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.280 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.280 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:12.280 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6831", 2024-09-21T14:05:12.280 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 13894964 2024-09-21T14:05:12.280 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:12.280 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:12.280 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.280 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T14:05:12.280 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:12.280 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.280 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:12.280 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6828", 2024-09-21T14:05:12.280 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 13894964 2024-09-21T14:05:12.281 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.281 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.281 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:12.281 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6829", 2024-09-21T14:05:12.281 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 13894964 2024-09-21T14:05:12.281 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:12.281 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:12.281 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.281 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.187:6825/13894964", 2024-09-21T14:05:12.281 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.187:6827/13894964", 2024-09-21T14:05:12.281 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.187:6831/13894964", 2024-09-21T14:05:12.281 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.187:6829/13894964", 2024-09-21T14:05:12.281 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T14:05:12.282 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T14:05:12.282 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T14:05:12.282 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:12.282 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:12.282 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T14:05:12.282 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_xinfo": [ 2024-09-21T14:05:12.282 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.282 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 0, 2024-09-21T14:05:12.282 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T14:05:12.282 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T14:05:12.282 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T14:05:12.282 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T14:05:12.282 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T14:05:12.282 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T14:04:20.834953+0000", 2024-09-21T14:05:12.283 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T14:05:12.283 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.283 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.283 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 1, 2024-09-21T14:05:12.283 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T14:05:12.283 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T14:05:12.283 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T14:05:12.283 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T14:05:12.283 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T14:05:12.283 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T14:05:07.323212+0000", 2024-09-21T14:05:12.283 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T14:05:12.283 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.283 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.283 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 2, 2024-09-21T14:05:12.284 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T14:05:12.284 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T14:05:12.284 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T14:05:12.284 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T14:05:12.284 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T14:05:12.284 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T14:04:07.105378+0000", 2024-09-21T14:05:12.284 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 100 2024-09-21T14:05:12.284 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.284 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.284 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 3, 2024-09-21T14:05:12.284 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T14:05:12.284 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T14:05:12.285 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T14:05:12.285 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T14:05:12.285 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T14:05:12.285 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T14:04:36.699677+0000", 2024-09-21T14:05:12.285 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 214 2024-09-21T14:05:12.285 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.285 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.285 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 4, 2024-09-21T14:05:12.285 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T14:05:12.285 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T14:05:12.285 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T14:05:12.285 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T14:05:12.285 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T14:05:12.286 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T14:04:23.066811+0000", 2024-09-21T14:05:12.286 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T14:05:12.286 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.286 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.286 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 5, 2024-09-21T14:05:12.286 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T14:05:12.286 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T14:05:12.286 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T14:05:12.286 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T14:05:12.286 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T14:05:12.286 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T14:04:21.949899+0000", 2024-09-21T14:05:12.286 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 324 2024-09-21T14:05:12.286 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.287 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.287 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 6, 2024-09-21T14:05:12.287 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T14:05:12.287 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T14:05:12.287 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T14:05:12.287 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T14:05:12.287 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T14:05:12.287 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T14:04:52.103632+0000", 2024-09-21T14:05:12.287 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T14:05:12.287 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.287 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.287 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 7, 2024-09-21T14:05:12.287 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T14:05:12.287 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T14:05:12.288 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T14:05:12.288 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T14:05:12.288 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T14:05:12.288 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T14:04:56.482050+0000", 2024-09-21T14:05:12.288 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 272 2024-09-21T14:05:12.288 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:12.288 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T14:05:12.288 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap": [ 2024-09-21T14:05:12.288 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.288 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "3.14", 2024-09-21T14:05:12.288 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-09-21T14:05:12.288 INFO:teuthology.orchestra.run.smithi049.stdout: 7, 2024-09-21T14:05:12.288 INFO:teuthology.orchestra.run.smithi049.stdout: 6, 2024-09-21T14:05:12.288 INFO:teuthology.orchestra.run.smithi049.stdout: 0 2024-09-21T14:05:12.289 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:12.289 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:12.289 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T14:05:12.289 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap_items": [ 2024-09-21T14:05:12.289 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.289 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "4.2", 2024-09-21T14:05:12.289 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-09-21T14:05:12.289 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.289 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 1, 2024-09-21T14:05:12.289 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 2 2024-09-21T14:05:12.289 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:12.289 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:12.289 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.289 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.289 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "4.6", 2024-09-21T14:05:12.290 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-09-21T14:05:12.290 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.290 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 1, 2024-09-21T14:05:12.290 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 5 2024-09-21T14:05:12.290 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:12.290 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:12.290 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.290 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.290 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "4.d", 2024-09-21T14:05:12.290 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-09-21T14:05:12.290 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.290 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 1, 2024-09-21T14:05:12.290 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 5 2024-09-21T14:05:12.290 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:12.291 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:12.291 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.291 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.291 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "6.7", 2024-09-21T14:05:12.291 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-09-21T14:05:12.291 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.291 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 3, 2024-09-21T14:05:12.291 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 7 2024-09-21T14:05:12.291 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:12.291 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:12.291 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:12.291 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T14:05:12.291 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap_primaries": [], 2024-09-21T14:05:12.291 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_temp": [ 2024-09-21T14:05:12.292 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.292 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "10.0", 2024-09-21T14:05:12.292 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-09-21T14:05:12.292 INFO:teuthology.orchestra.run.smithi049.stdout: 6, 2024-09-21T14:05:12.292 INFO:teuthology.orchestra.run.smithi049.stdout: 1, 2024-09-21T14:05:12.292 INFO:teuthology.orchestra.run.smithi049.stdout: 4 2024-09-21T14:05:12.292 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:12.292 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.292 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.292 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "10.4", 2024-09-21T14:05:12.292 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-09-21T14:05:12.292 INFO:teuthology.orchestra.run.smithi049.stdout: 4, 2024-09-21T14:05:12.292 INFO:teuthology.orchestra.run.smithi049.stdout: 3, 2024-09-21T14:05:12.292 INFO:teuthology.orchestra.run.smithi049.stdout: 0 2024-09-21T14:05:12.293 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:12.293 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.293 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.293 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "10.a", 2024-09-21T14:05:12.293 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-09-21T14:05:12.293 INFO:teuthology.orchestra.run.smithi049.stdout: 6, 2024-09-21T14:05:12.293 INFO:teuthology.orchestra.run.smithi049.stdout: 5, 2024-09-21T14:05:12.293 INFO:teuthology.orchestra.run.smithi049.stdout: 0 2024-09-21T14:05:12.293 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:12.293 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.293 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:12.293 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "10.d", 2024-09-21T14:05:12.293 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-09-21T14:05:12.294 INFO:teuthology.orchestra.run.smithi049.stdout: 6, 2024-09-21T14:05:12.294 INFO:teuthology.orchestra.run.smithi049.stdout: 4, 2024-09-21T14:05:12.294 INFO:teuthology.orchestra.run.smithi049.stdout: 0 2024-09-21T14:05:12.294 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:12.294 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:12.294 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T14:05:12.294 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_temp": [], 2024-09-21T14:05:12.294 INFO:teuthology.orchestra.run.smithi049.stdout: "blocklist": { 2024-09-21T14:05:12.294 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/2731793917": "2024-09-22T13:49:27.311313+0000", 2024-09-21T14:05:12.294 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/799831425": "2024-09-22T13:49:27.311313+0000", 2024-09-21T14:05:12.294 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/2518197280": "2024-09-22T13:43:28.110499+0000", 2024-09-21T14:05:12.294 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3556829486": "2024-09-22T13:43:28.110499+0000", 2024-09-21T14:05:12.294 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/1249926115": "2024-09-22T13:49:27.311313+0000", 2024-09-21T14:05:12.295 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/231530138": "2024-09-22T13:43:28.110499+0000", 2024-09-21T14:05:12.295 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/4220781134": "2024-09-22T13:49:27.311313+0000", 2024-09-21T14:05:12.295 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/202325455": "2024-09-22T13:43:28.110499+0000", 2024-09-21T14:05:12.295 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/4129636473": "2024-09-22T13:43:56.633723+0000", 2024-09-21T14:05:12.295 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/2518197280": "2024-09-22T13:43:28.110499+0000", 2024-09-21T14:05:12.295 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3514741273": "2024-09-22T13:43:56.633723+0000", 2024-09-21T14:05:12.295 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/4129636473": "2024-09-22T13:43:56.633723+0000", 2024-09-21T14:05:12.295 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3125743776": "2024-09-22T13:43:56.633723+0000", 2024-09-21T14:05:12.295 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3600977363": "2024-09-22T13:43:56.633723+0000", 2024-09-21T14:05:12.295 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/3453571598": "2024-09-22T13:49:27.311313+0000", 2024-09-21T14:05:12.295 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/3453571598": "2024-09-22T13:49:27.311313+0000", 2024-09-21T14:05:12.295 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3553734432": "2024-09-22T13:49:27.311313+0000" 2024-09-21T14:05:12.295 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.295 INFO:teuthology.orchestra.run.smithi049.stdout: "range_blocklist": {}, 2024-09-21T14:05:12.295 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profiles": { 2024-09-21T14:05:12.296 INFO:teuthology.orchestra.run.smithi049.stdout: "default": { 2024-09-21T14:05:12.296 INFO:teuthology.orchestra.run.smithi049.stdout: "crush-failure-domain": "osd", 2024-09-21T14:05:12.296 INFO:teuthology.orchestra.run.smithi049.stdout: "k": "2", 2024-09-21T14:05:12.296 INFO:teuthology.orchestra.run.smithi049.stdout: "m": "1", 2024-09-21T14:05:12.296 INFO:teuthology.orchestra.run.smithi049.stdout: "plugin": "jerasure", 2024-09-21T14:05:12.296 INFO:teuthology.orchestra.run.smithi049.stdout: "technique": "reed_sol_van" 2024-09-21T14:05:12.296 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:12.296 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:12.296 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps_queue": [], 2024-09-21T14:05:12.296 INFO:teuthology.orchestra.run.smithi049.stdout: "new_removed_snaps": [], 2024-09-21T14:05:12.296 INFO:teuthology.orchestra.run.smithi049.stdout: "new_purged_snaps": [], 2024-09-21T14:05:12.296 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_node_flags": {}, 2024-09-21T14:05:12.296 INFO:teuthology.orchestra.run.smithi049.stdout: "device_class_flags": {}, 2024-09-21T14:05:12.297 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode": { 2024-09-21T14:05:12.297 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode_enabled": false, 2024-09-21T14:05:12.297 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_bucket_count": 0, 2024-09-21T14:05:12.297 INFO:teuthology.orchestra.run.smithi049.stdout: "degraded_stretch_mode": 0, 2024-09-21T14:05:12.297 INFO:teuthology.orchestra.run.smithi049.stdout: "recovering_stretch_mode": 0, 2024-09-21T14:05:12.297 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode_bucket": 0 2024-09-21T14:05:12.297 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:12.297 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T14:05:12.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:12.298 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:12.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:12.399 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:12.418 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T14:05:12.434 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:12.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:12.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:12.536 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:05:12.567 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:12.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:12.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:05:12.669 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:12.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:12.695 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:12.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:12.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:12 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1842138076' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:05:12.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:12 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3831829690' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T14:05:12.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:12 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1842138076' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:05:12.757 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:12 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3831829690' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T14:05:12.796 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:12.832 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:12.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:12.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:12.934 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:12.962 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:12.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:12.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:13.064 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:05:13.087 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:13.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:13.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:05:13.189 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:05:13.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:12 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1842138076' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:05:13.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:12 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3831829690' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T14:05:13.215 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:13.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:13.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:05:13.316 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:13.341 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:13.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:13.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:13.443 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:13.469 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:13.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:13.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:13.571 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:05:13.595 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:13.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:13.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:05:13.697 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:05:13.719 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:13.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:13.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:05:13.821 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:05:13.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:13.857 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:13.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:05:13.958 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:13.991 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:13.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:13.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:14.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:13 smithi049 ceph-mon[40489]: pgmap v822: 158 pgs: 4 active+remapped+backfilling, 154 active+clean; 5.7 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 840 op/s; 10983/279975 objects misplaced (3.923%); 21 MiB/s, 340 objects/s recovering 2024-09-21T14:05:14.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:13 smithi049 ceph-mon[40489]: 5.1b scrub starts 2024-09-21T14:05:14.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:13 smithi049 ceph-mon[40489]: 5.1b scrub ok 2024-09-21T14:05:14.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:13 smithi049 ceph-mon[40489]: osdmap e422: 8 total, 8 up, 8 in 2024-09-21T14:05:14.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:13 smithi049 ceph-mon[30080]: pgmap v822: 158 pgs: 4 active+remapped+backfilling, 154 active+clean; 5.7 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 840 op/s; 10983/279975 objects misplaced (3.923%); 21 MiB/s, 340 objects/s recovering 2024-09-21T14:05:14.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:13 smithi049 ceph-mon[30080]: 5.1b scrub starts 2024-09-21T14:05:14.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:13 smithi049 ceph-mon[30080]: 5.1b scrub ok 2024-09-21T14:05:14.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:13 smithi049 ceph-mon[30080]: osdmap e422: 8 total, 8 up, 8 in 2024-09-21T14:05:14.093 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:05:14.114 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:14.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:14.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:05:14.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:13 smithi187 ceph-mon[33032]: pgmap v822: 158 pgs: 4 active+remapped+backfilling, 154 active+clean; 5.7 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 840 op/s; 10983/279975 objects misplaced (3.923%); 21 MiB/s, 340 objects/s recovering 2024-09-21T14:05:14.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:13 smithi187 ceph-mon[33032]: 5.1b scrub starts 2024-09-21T14:05:14.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:13 smithi187 ceph-mon[33032]: 5.1b scrub ok 2024-09-21T14:05:14.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:13 smithi187 ceph-mon[33032]: osdmap e422: 8 total, 8 up, 8 in 2024-09-21T14:05:14.216 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:05:14.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:14.252 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:14.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:05:14.353 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:14.392 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:14.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:14.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:14.494 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:05:14.518 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:14.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:14.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:05:14.620 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:05:14.658 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:14.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:14.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:05:14.760 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:14.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:14.783 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:14.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:14.884 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:05:14.919 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:14.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:14.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:05:15.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:14 smithi049 ceph-mon[40489]: 3.16 scrub starts 2024-09-21T14:05:15.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:14 smithi049 ceph-mon[40489]: 3.16 scrub ok 2024-09-21T14:05:15.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:14 smithi049 ceph-mon[40489]: 1.3 scrub starts 2024-09-21T14:05:15.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:14 smithi049 ceph-mon[40489]: 1.3 scrub ok 2024-09-21T14:05:15.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:14 smithi049 ceph-mon[40489]: osdmap e423: 8 total, 8 up, 8 in 2024-09-21T14:05:15.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:14 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:05:15.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:14 smithi049 ceph-mon[30080]: 3.16 scrub starts 2024-09-21T14:05:15.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:14 smithi049 ceph-mon[30080]: 3.16 scrub ok 2024-09-21T14:05:15.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:14 smithi049 ceph-mon[30080]: 1.3 scrub starts 2024-09-21T14:05:15.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:14 smithi049 ceph-mon[30080]: 1.3 scrub ok 2024-09-21T14:05:15.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:14 smithi049 ceph-mon[30080]: osdmap e423: 8 total, 8 up, 8 in 2024-09-21T14:05:15.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:14 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:05:15.021 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:05:15.054 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:15.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:15.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:05:15.156 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:15.183 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:15.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:15.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:15.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:14 smithi187 ceph-mon[33032]: 3.16 scrub starts 2024-09-21T14:05:15.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:14 smithi187 ceph-mon[33032]: 3.16 scrub ok 2024-09-21T14:05:15.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:14 smithi187 ceph-mon[33032]: 1.3 scrub starts 2024-09-21T14:05:15.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:14 smithi187 ceph-mon[33032]: 1.3 scrub ok 2024-09-21T14:05:15.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:14 smithi187 ceph-mon[33032]: osdmap e423: 8 total, 8 up, 8 in 2024-09-21T14:05:15.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:14 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:05:15.284 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:15.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:15.311 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:15.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:15.411 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:15.448 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:15.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:15.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:15.549 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:05:15.581 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:15.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:15.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:05:15.682 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:05:15.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:15.725 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:15.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:05:15.826 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:05:15.846 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:15.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:15.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:05:15.949 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:05:15.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:15.992 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:15.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:05:16.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:15 smithi049 ceph-mon[40489]: pgmap v825: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.21k op/s; 7641/271470 objects misplaced (2.815%); 59 MiB/s, 937 objects/s recovering 2024-09-21T14:05:16.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:15 smithi049 ceph-mon[40489]: 5.7 scrub starts 2024-09-21T14:05:16.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:15 smithi049 ceph-mon[40489]: 6.1a scrub starts 2024-09-21T14:05:16.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:15 smithi049 ceph-mon[40489]: 5.7 scrub ok 2024-09-21T14:05:16.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:15 smithi049 ceph-mon[40489]: 6.1a scrub ok 2024-09-21T14:05:16.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:15 smithi049 ceph-mon[30080]: pgmap v825: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.21k op/s; 7641/271470 objects misplaced (2.815%); 59 MiB/s, 937 objects/s recovering 2024-09-21T14:05:16.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:15 smithi049 ceph-mon[30080]: 5.7 scrub starts 2024-09-21T14:05:16.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:15 smithi049 ceph-mon[30080]: 6.1a scrub starts 2024-09-21T14:05:16.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:15 smithi049 ceph-mon[30080]: 5.7 scrub ok 2024-09-21T14:05:16.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:15 smithi049 ceph-mon[30080]: 6.1a scrub ok 2024-09-21T14:05:16.093 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:16.132 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:16.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:16.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:16.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:15 smithi187 ceph-mon[33032]: pgmap v825: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.21k op/s; 7641/271470 objects misplaced (2.815%); 59 MiB/s, 937 objects/s recovering 2024-09-21T14:05:16.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:15 smithi187 ceph-mon[33032]: 5.7 scrub starts 2024-09-21T14:05:16.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:15 smithi187 ceph-mon[33032]: 6.1a scrub starts 2024-09-21T14:05:16.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:15 smithi187 ceph-mon[33032]: 5.7 scrub ok 2024-09-21T14:05:16.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:15 smithi187 ceph-mon[33032]: 6.1a scrub ok 2024-09-21T14:05:16.235 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:16.265 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:16.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:16.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:16.367 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:05:16.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:16.403 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:16.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:05:16.441 INFO:tasks.thrashosds.thrasher:j is {'epoch': 421, 'fsid': '40a3d2e8-781f-11ef-baf6-efdc52797490', 'created': '2024-09-21T13:42:57.838275+0000', 'modified': '2024-09-21T14:05:09.657918+0000', 'last_up_change': '2024-09-21T14:01:40.158454+0000', 'last_in_change': '2024-09-21T13:58:16.730861+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 22, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 10, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T13:46:21.096244+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '320', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '318', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.180000066757202, 'score_stable': 2.180000066757202, 'optimal_score': 1, 'raw_score_acting': 2.180000066757202, 'raw_score_stable': 2.180000066757202, 'primary_affinity_weighted': 0.9783573150634766, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.9800000190734863}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-21T13:48:55.540712+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 63, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 0.9783573150634766, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.9800000190734863}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-21T13:48:56.417456+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.9783573150634766, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.9800000190734863}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-21T13:48:58.128455+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 0.9783573150634766, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.9800000190734863}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-21T13:49:00.069361+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9783573150634766, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.9800000190734863}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-21T13:49:02.127399+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9783573150634766, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.9800000190734863}}, {'pool': 10, 'pool_name': 'unique_pool_3', 'create_time': '2024-09-21T14:02:55.287153+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '415', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3, 'score_stable': 3, 'optimal_score': 1, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9783573150634766, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.9800000190734863}}], 'osds': [{'osd': 0, 'uuid': '2595dca9-6219-4e0c-8728-751eb102e73a', 'up': 1, 'in': 1, 'weight': 0.6176910400390625, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 417, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6802', 'nonce': 2095561412}, {'type': 'v1', 'addr': '172.21.15.49:6803', 'nonce': 2095561412}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6804', 'nonce': 2095561412}, {'type': 'v1', 'addr': '172.21.15.49:6805', 'nonce': 2095561412}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6808', 'nonce': 2095561412}, {'type': 'v1', 'addr': '172.21.15.49:6809', 'nonce': 2095561412}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6806', 'nonce': 2095561412}, {'type': 'v1', 'addr': '172.21.15.49:6807', 'nonce': 2095561412}]}, 'public_addr': '172.21.15.49:6803/2095561412', 'cluster_addr': '172.21.15.49:6805/2095561412', 'heartbeat_back_addr': '172.21.15.49:6809/2095561412', 'heartbeat_front_addr': '172.21.15.49:6807/2095561412', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '0ba327f9-5524-4a40-abcc-729ded0b1917', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 408, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6810', 'nonce': 1382840527}, {'type': 'v1', 'addr': '172.21.15.49:6811', 'nonce': 1382840527}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6812', 'nonce': 1382840527}, {'type': 'v1', 'addr': '172.21.15.49:6813', 'nonce': 1382840527}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6816', 'nonce': 1382840527}, {'type': 'v1', 'addr': '172.21.15.49:6817', 'nonce': 1382840527}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6814', 'nonce': 1382840527}, {'type': 'v1', 'addr': '172.21.15.49:6815', 'nonce': 1382840527}]}, 'public_addr': '172.21.15.49:6811/1382840527', 'cluster_addr': '172.21.15.49:6813/1382840527', 'heartbeat_back_addr': '172.21.15.49:6817/1382840527', 'heartbeat_front_addr': '172.21.15.49:6815/1382840527', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '27a45750-204d-48fd-bb8f-ebb01273343d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.8268585205078125, 'last_clean_begin': 18, 'last_clean_end': 100, 'up_from': 130, 'up_thru': 408, 'down_at': 101, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6818', 'nonce': 1945314741}, {'type': 'v1', 'addr': '172.21.15.49:6819', 'nonce': 1945314741}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6820', 'nonce': 1945314741}, {'type': 'v1', 'addr': '172.21.15.49:6821', 'nonce': 1945314741}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6824', 'nonce': 1945314741}, {'type': 'v1', 'addr': '172.21.15.49:6825', 'nonce': 1945314741}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6822', 'nonce': 1945314741}, {'type': 'v1', 'addr': '172.21.15.49:6823', 'nonce': 1945314741}]}, 'public_addr': '172.21.15.49:6819/1945314741', 'cluster_addr': '172.21.15.49:6821/1945314741', 'heartbeat_back_addr': '172.21.15.49:6825/1945314741', 'heartbeat_front_addr': '172.21.15.49:6823/1945314741', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '528bc390-3dc9-4f3b-ab75-d7412c9625db', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 27, 'last_clean_end': 214, 'up_from': 240, 'up_thru': 408, 'down_at': 215, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6826', 'nonce': 886735721}, {'type': 'v1', 'addr': '172.21.15.49:6827', 'nonce': 886735721}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6828', 'nonce': 886735721}, {'type': 'v1', 'addr': '172.21.15.49:6829', 'nonce': 886735721}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6832', 'nonce': 886735721}, {'type': 'v1', 'addr': '172.21.15.49:6833', 'nonce': 886735721}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6830', 'nonce': 886735721}, {'type': 'v1', 'addr': '172.21.15.49:6831', 'nonce': 886735721}]}, 'public_addr': '172.21.15.49:6827/886735721', 'cluster_addr': '172.21.15.49:6829/886735721', 'heartbeat_back_addr': '172.21.15.49:6833/886735721', 'heartbeat_front_addr': '172.21.15.49:6831/886735721', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '6fc64698-2d37-4db0-bf3d-c7e612c45163', 'up': 1, 'in': 1, 'weight': 0.7888946533203125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 409, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6800', 'nonce': 1702447776}, {'type': 'v1', 'addr': '172.21.15.187:6801', 'nonce': 1702447776}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6802', 'nonce': 1702447776}, {'type': 'v1', 'addr': '172.21.15.187:6803', 'nonce': 1702447776}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6806', 'nonce': 1702447776}, {'type': 'v1', 'addr': '172.21.15.187:6807', 'nonce': 1702447776}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6804', 'nonce': 1702447776}, {'type': 'v1', 'addr': '172.21.15.187:6805', 'nonce': 1702447776}]}, 'public_addr': '172.21.15.187:6801/1702447776', 'cluster_addr': '172.21.15.187:6803/1702447776', 'heartbeat_back_addr': '172.21.15.187:6807/1702447776', 'heartbeat_front_addr': '172.21.15.187:6805/1702447776', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '8d7f62a1-7e92-4e06-9cad-86b96be514d8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 39, 'last_clean_end': 324, 'up_from': 345, 'up_thru': 408, 'down_at': 325, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6808', 'nonce': 1595353965}, {'type': 'v1', 'addr': '172.21.15.187:6809', 'nonce': 1595353965}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6810', 'nonce': 1595353965}, {'type': 'v1', 'addr': '172.21.15.187:6811', 'nonce': 1595353965}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6814', 'nonce': 1595353965}, {'type': 'v1', 'addr': '172.21.15.187:6815', 'nonce': 1595353965}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6812', 'nonce': 1595353965}, {'type': 'v1', 'addr': '172.21.15.187:6813', 'nonce': 1595353965}]}, 'public_addr': '172.21.15.187:6809/1595353965', 'cluster_addr': '172.21.15.187:6811/1595353965', 'heartbeat_back_addr': '172.21.15.187:6815/1595353965', 'heartbeat_front_addr': '172.21.15.187:6813/1595353965', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '3a7afd13-26ce-444c-9cc0-81ce2eab6ce5', 'up': 1, 'in': 1, 'weight': 0.9076080322265625, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 409, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6816', 'nonce': 2728426525}, {'type': 'v1', 'addr': '172.21.15.187:6817', 'nonce': 2728426525}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6818', 'nonce': 2728426525}, {'type': 'v1', 'addr': '172.21.15.187:6819', 'nonce': 2728426525}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6822', 'nonce': 2728426525}, {'type': 'v1', 'addr': '172.21.15.187:6823', 'nonce': 2728426525}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6820', 'nonce': 2728426525}, {'type': 'v1', 'addr': '172.21.15.187:6821', 'nonce': 2728426525}]}, 'public_addr': '172.21.15.187:6817/2728426525', 'cluster_addr': '172.21.15.187:6819/2728426525', 'heartbeat_back_addr': '172.21.15.187:6823/2728426525', 'heartbeat_front_addr': '172.21.15.187:6821/2728426525', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1e08d891-8747-402e-b4c2-aa3802ce8537', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 54, 'last_clean_end': 272, 'up_from': 298, 'up_thru': 408, 'down_at': 273, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 13894964}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 13894964}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 13894964}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 13894964}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6830', 'nonce': 13894964}, {'type': 'v1', 'addr': '172.21.15.187:6831', 'nonce': 13894964}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6828', 'nonce': 13894964}, {'type': 'v1', 'addr': '172.21.15.187:6829', 'nonce': 13894964}]}, 'public_addr': '172.21.15.187:6825/13894964', 'cluster_addr': '172.21.15.187:6827/13894964', 'heartbeat_back_addr': '172.21.15.187:6831/13894964', 'heartbeat_front_addr': '172.21.15.187:6829/13894964', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T14:04:20.834953+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T14:05:07.323212+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T14:04:07.105378+0000', 'dead_epoch': 100}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T14:04:36.699677+0000', 'dead_epoch': 214}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T14:04:23.066811+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T14:04:21.949899+0000', 'dead_epoch': 324}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T14:04:52.103632+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T14:04:56.482050+0000', 'dead_epoch': 272}], 'pg_upmap': [{'pgid': '3.14', 'osds': [7, 6, 0]}], 'pg_upmap_items': [{'pgid': '4.2', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.6', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.d', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '6.7', 'mappings': [{'from': 3, 'to': 7}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '10.0', 'osds': [6, 1, 4]}, {'pgid': '10.4', 'osds': [4, 3, 0]}, {'pgid': '10.a', 'osds': [6, 5, 0]}, {'pgid': '10.d', 'osds': [6, 4, 0]}], 'primary_temp': [], 'blocklist': {'172.21.15.49:0/2731793917': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:0/799831425': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:6801/2518197280': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:0/3556829486': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:0/1249926115': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:0/231530138': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:0/4220781134': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:0/202325455': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:6800/4129636473': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:6800/2518197280': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:0/3514741273': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:6801/4129636473': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:0/3125743776': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:0/3600977363': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:6801/3453571598': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:6800/3453571598': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:0/3553734432': '2024-09-22T13:49:27.311313+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-21T14:05:16.441 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T14:05:16.464 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:05:16.505 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:05:16.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:16.540 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:16.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:05:16.641 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:05:16.671 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:16.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:16.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:05:16.731 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:16 smithi049 ceph-mon[30080]: 2.2 scrub starts 2024-09-21T14:05:16.731 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:16 smithi049 ceph-mon[30080]: 2.2 scrub ok 2024-09-21T14:05:16.731 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:16 smithi049 ceph-mon[30080]: 4.c scrub starts 2024-09-21T14:05:16.731 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:16 smithi049 ceph-mon[30080]: 4.c scrub ok 2024-09-21T14:05:16.773 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:05:16.804 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:16.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:16.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:05:16.905 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:16.941 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:16.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:16.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:17.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:16 smithi049 ceph-mon[40489]: 2.2 scrub starts 2024-09-21T14:05:17.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:16 smithi049 ceph-mon[40489]: 2.2 scrub ok 2024-09-21T14:05:17.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:16 smithi049 ceph-mon[40489]: 4.c scrub starts 2024-09-21T14:05:17.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:16 smithi049 ceph-mon[40489]: 4.c scrub ok 2024-09-21T14:05:17.043 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:05:17.102 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:17.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:17.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:05:17.134 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:05:17.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:16 smithi187 ceph-mon[33032]: 2.2 scrub starts 2024-09-21T14:05:17.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:16 smithi187 ceph-mon[33032]: 2.2 scrub ok 2024-09-21T14:05:17.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:16 smithi187 ceph-mon[33032]: 4.c scrub starts 2024-09-21T14:05:17.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:16 smithi187 ceph-mon[33032]: 4.c scrub ok 2024-09-21T14:05:17.205 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:05:17.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:17.252 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:17.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:05:17.353 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:17.394 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:17.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:17.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:17.495 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:05:17.537 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:17.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:17.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:05:17.579 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T14:05:17.640 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:05:17.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:17.687 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:17.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:05:17.752 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-09-21T14:05:17.788 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:17.822 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:17.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:17.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:17.923 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:17.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:17.959 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:17.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:18.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:17 smithi049 ceph-mon[40489]: pgmap v826: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 787 op/s; 7641/268512 objects misplaced (2.846%); 28 MiB/s, 454 objects/s recovering 2024-09-21T14:05:18.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:17 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1525881571' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:05:18.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:17 smithi049 ceph-mon[30080]: pgmap v826: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 787 op/s; 7641/268512 objects misplaced (2.846%); 28 MiB/s, 454 objects/s recovering 2024-09-21T14:05:18.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:17 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1525881571' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:05:18.060 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:05:18.091 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:18.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:18.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:05:18.160 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T14:05:18.163 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-09-21T14:05:18.193 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:05:18.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:17 smithi187 ceph-mon[33032]: pgmap v826: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 787 op/s; 7641/268512 objects misplaced (2.846%); 28 MiB/s, 454 objects/s recovering 2024-09-21T14:05:18.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:17 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1525881571' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:05:18.235 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:18.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:18.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:05:18.337 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:05:18.374 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:18.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:18.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:05:18.475 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:05:18.515 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:18.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:18.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:05:18.617 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:05:18.651 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:18.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:18.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:05:18.753 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:18.784 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:18.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:18.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:18.886 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:05:18.895 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T14:05:18.926 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:18.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:18.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:05:19.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:18 smithi049 ceph-mon[30080]: 4.8 scrub starts 2024-09-21T14:05:19.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:18 smithi049 ceph-mon[30080]: 4.8 scrub ok 2024-09-21T14:05:19.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:18 smithi049 ceph-mon[30080]: 6.18 scrub starts 2024-09-21T14:05:19.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:18 smithi049 ceph-mon[30080]: 6.18 scrub ok 2024-09-21T14:05:19.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:18 smithi049 ceph-mon[30080]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T14:05:19.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:18 smithi049 ceph-mon[30080]: Cluster is now healthy 2024-09-21T14:05:19.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:18 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1525881571' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T14:05:19.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:18 smithi049 ceph-mon[30080]: osdmap e424: 8 total, 8 up, 8 in 2024-09-21T14:05:19.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:18 smithi049 ceph-mon[40489]: 4.8 scrub starts 2024-09-21T14:05:19.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:18 smithi049 ceph-mon[40489]: 4.8 scrub ok 2024-09-21T14:05:19.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:18 smithi049 ceph-mon[40489]: 6.18 scrub starts 2024-09-21T14:05:19.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:18 smithi049 ceph-mon[40489]: 6.18 scrub ok 2024-09-21T14:05:19.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:18 smithi049 ceph-mon[40489]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T14:05:19.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:18 smithi049 ceph-mon[40489]: Cluster is now healthy 2024-09-21T14:05:19.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:18 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1525881571' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T14:05:19.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:18 smithi049 ceph-mon[40489]: osdmap e424: 8 total, 8 up, 8 in 2024-09-21T14:05:19.029 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:05:19.064 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:19.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:19.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:05:19.166 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:19.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:18 smithi187 ceph-mon[33032]: 4.8 scrub starts 2024-09-21T14:05:19.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:18 smithi187 ceph-mon[33032]: 4.8 scrub ok 2024-09-21T14:05:19.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:18 smithi187 ceph-mon[33032]: 6.18 scrub starts 2024-09-21T14:05:19.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:18 smithi187 ceph-mon[33032]: 6.18 scrub ok 2024-09-21T14:05:19.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:18 smithi187 ceph-mon[33032]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T14:05:19.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:18 smithi187 ceph-mon[33032]: Cluster is now healthy 2024-09-21T14:05:19.196 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:18 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1525881571' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T14:05:19.196 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:18 smithi187 ceph-mon[33032]: osdmap e424: 8 total, 8 up, 8 in 2024-09-21T14:05:19.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:19.220 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:19.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:19.276 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:05:19.321 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:19.360 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:19.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:19.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:19.462 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:05:19.496 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:19.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:19.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:05:19.560 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":828,"stamp":"2024-09-21T14:05:18.052241+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5805412132,"num_objects":88774,"num_object_clones":0,"num_object_copies":266322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7641,"num_objects_unfound":0,"num_objects_dirty":88774,"num_whiteouts":0,"num_read":1490,"num_read_kb":1269,"num_write":167331,"num_write_kb":8165226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17061,"num_bytes_recovered":1104707731,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84029,"ondisk_log_size":84029,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":478,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":21931912,"kb_used_data":17965936,"kb_used_omap":23,"kb_used_meta":3965608,"kb_avail":728029304,"statfs":{"total":767960285184,"available":745502007296,"internally_reserved":0,"allocated":18397118464,"data_stored":18394542029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24148,"internal_metadata":4060783020},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,0,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-298308006,"num_objects":-4551,"num_object_clones":0,"num_object_copies":-13653,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-3342,"num_objects_unfound":0,"num_objects_dirty":-4551,"num_whiteouts":0,"num_read":-5,"num_read_kb":-15,"num_write":4540,"num_write_kb":-148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2901,"num_bytes_recovered":190066266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002684"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":686,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.834605+0000","last_change":"2024-09-21T14:05:10.397918+0000","last_active":"2024-09-21T14:05:13.834605+0000","last_peered":"2024-09-21T14:05:13.834605+0000","last_clean":"2024-09-21T14:05:13.834605+0000","last_became_active":"2024-09-21T14:04:21.204299+0000","last_became_peered":"2024-09-21T14:04:21.204299+0000","last_unstale":"2024-09-21T14:05:13.834605+0000","last_undegraded":"2024-09-21T14:05:13.834605+0000","last_fullsized":"2024-09-21T14:05:13.834605+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:10.397765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:05:10.397765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:40.202681+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"290'15","reported_seq":838,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.706635+0000","last_change":"2024-09-21T14:01:34.785322+0000","last_active":"2024-09-21T14:05:13.706635+0000","last_peered":"2024-09-21T14:05:13.706635+0000","last_clean":"2024-09-21T14:05:13.706635+0000","last_became_active":"2024-09-21T13:58:17.771220+0000","last_became_peered":"2024-09-21T13:58:17.771220+0000","last_unstale":"2024-09-21T14:05:13.706635+0000","last_undegraded":"2024-09-21T14:05:13.706635+0000","last_fullsized":"2024-09-21T14:05:13.706635+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'15","last_scrub_stamp":"2024-09-21T14:01:34.785267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:34.785267+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":787,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.704902+0000","last_change":"2024-09-21T14:03:33.301558+0000","last_active":"2024-09-21T14:05:13.704902+0000","last_peered":"2024-09-21T14:05:13.704902+0000","last_clean":"2024-09-21T14:05:13.704902+0000","last_became_active":"2024-09-21T14:01:41.202168+0000","last_became_peered":"2024-09-21T14:01:41.202168+0000","last_unstale":"2024-09-21T14:05:13.704902+0000","last_undegraded":"2024-09-21T14:05:13.704902+0000","last_fullsized":"2024-09-21T14:05:13.704902+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:33.301487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:33.301487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":755,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.706812+0000","last_change":"2024-09-21T14:01:41.203762+0000","last_active":"2024-09-21T14:05:13.706812+0000","last_peered":"2024-09-21T14:05:13.706812+0000","last_clean":"2024-09-21T14:05:13.706812+0000","last_became_active":"2024-09-21T14:01:41.203275+0000","last_became_peered":"2024-09-21T14:01:41.203275+0000","last_unstale":"2024-09-21T14:05:13.706812+0000","last_undegraded":"2024-09-21T14:05:13.706812+0000","last_fullsized":"2024-09-21T14:05:13.706812+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:00.320510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:57:00.320510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":733,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.708233+0000","last_change":"2024-09-21T14:04:00.241774+0000","last_active":"2024-09-21T14:05:13.708233+0000","last_peered":"2024-09-21T14:05:13.708233+0000","last_clean":"2024-09-21T14:05:13.708233+0000","last_became_active":"2024-09-21T13:59:32.203340+0000","last_became_peered":"2024-09-21T13:59:32.203340+0000","last_unstale":"2024-09-21T14:05:13.708233+0000","last_undegraded":"2024-09-21T14:05:13.708233+0000","last_fullsized":"2024-09-21T14:05:13.708233+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:00.241562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:04:00.241562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:19.693680+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":738,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.708411+0000","last_change":"2024-09-21T13:58:17.778322+0000","last_active":"2024-09-21T14:05:13.708411+0000","last_peered":"2024-09-21T14:05:13.708411+0000","last_clean":"2024-09-21T14:05:13.708411+0000","last_became_active":"2024-09-21T13:58:17.777893+0000","last_became_peered":"2024-09-21T13:58:17.777893+0000","last_unstale":"2024-09-21T14:05:13.708411+0000","last_undegraded":"2024-09-21T14:05:13.708411+0000","last_fullsized":"2024-09-21T14:05:13.708411+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:07.001135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:55:07.001135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"290'27","reported_seq":719,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.708082+0000","last_change":"2024-09-21T14:04:08.320458+0000","last_active":"2024-09-21T14:05:13.708082+0000","last_peered":"2024-09-21T14:05:13.708082+0000","last_clean":"2024-09-21T14:05:13.708082+0000","last_became_active":"2024-09-21T13:57:12.703469+0000","last_became_peered":"2024-09-21T13:57:12.703469+0000","last_unstale":"2024-09-21T14:05:13.708082+0000","last_undegraded":"2024-09-21T14:05:13.708082+0000","last_fullsized":"2024-09-21T14:05:13.708082+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'27","last_scrub_stamp":"2024-09-21T14:04:08.320339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:04:08.320339+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:32.005889+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":490,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-21T14:05:12.709332+0000","last_change":"2024-09-21T14:04:35.907676+0000","last_active":"2024-09-21T14:05:12.709332+0000","last_peered":"2024-09-21T14:05:12.709332+0000","last_clean":"2024-09-21T14:05:12.709332+0000","last_became_active":"2024-09-21T14:04:21.208237+0000","last_became_peered":"2024-09-21T14:04:21.208237+0000","last_unstale":"2024-09-21T14:05:12.709332+0000","last_undegraded":"2024-09-21T14:05:12.709332+0000","last_fullsized":"2024-09-21T14:05:12.709332+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:35.907611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:04:35.907611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:39.685760+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":738,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.834141+0000","last_change":"2024-09-21T14:04:11.718218+0000","last_active":"2024-09-21T14:05:13.834141+0000","last_peered":"2024-09-21T14:05:13.834141+0000","last_clean":"2024-09-21T14:05:13.834141+0000","last_became_active":"2024-09-21T14:01:41.194517+0000","last_became_peered":"2024-09-21T14:01:41.194517+0000","last_unstale":"2024-09-21T14:05:13.834141+0000","last_undegraded":"2024-09-21T14:05:13.834141+0000","last_fullsized":"2024-09-21T14:05:13.834141+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:11.718143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:11.718143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:19.307011+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"290'15","reported_seq":577,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.708165+0000","last_change":"2024-09-21T14:04:21.228412+0000","last_active":"2024-09-21T14:05:13.708165+0000","last_peered":"2024-09-21T14:05:13.708165+0000","last_clean":"2024-09-21T14:05:13.708165+0000","last_became_active":"2024-09-21T14:04:21.208244+0000","last_became_peered":"2024-09-21T14:04:21.208244+0000","last_unstale":"2024-09-21T14:05:13.708165+0000","last_undegraded":"2024-09-21T14:05:13.708165+0000","last_fullsized":"2024-09-21T14:05:13.708165+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'15","last_scrub_stamp":"2024-09-21T14:00:02.475661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:00:02.475661+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":687,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.833409+0000","last_change":"2024-09-21T14:05:12.369047+0000","last_active":"2024-09-21T14:05:13.833409+0000","last_peered":"2024-09-21T14:05:13.833409+0000","last_clean":"2024-09-21T14:05:13.833409+0000","last_became_active":"2024-09-21T14:01:41.188248+0000","last_became_peered":"2024-09-21T14:01:41.188248+0000","last_unstale":"2024-09-21T14:05:13.833409+0000","last_undegraded":"2024-09-21T14:05:13.833409+0000","last_fullsized":"2024-09-21T14:05:13.833409+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:12.368984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:05:12.368984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:27.784175+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":716,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.705196+0000","last_change":"2024-09-21T14:04:21.205796+0000","last_active":"2024-09-21T14:05:13.705196+0000","last_peered":"2024-09-21T14:05:13.705196+0000","last_clean":"2024-09-21T14:05:13.705196+0000","last_became_active":"2024-09-21T14:04:21.205416+0000","last_became_peered":"2024-09-21T14:04:21.205416+0000","last_unstale":"2024-09-21T14:05:13.705196+0000","last_undegraded":"2024-09-21T14:05:13.705196+0000","last_fullsized":"2024-09-21T14:05:13.705196+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:54:25.792985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:54:25.792985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":715,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.833900+0000","last_change":"2024-09-21T14:04:10.722881+0000","last_active":"2024-09-21T14:05:13.833900+0000","last_peered":"2024-09-21T14:05:13.833900+0000","last_clean":"2024-09-21T14:05:13.833900+0000","last_became_active":"2024-09-21T14:01:41.189373+0000","last_became_peered":"2024-09-21T14:01:41.189373+0000","last_unstale":"2024-09-21T14:05:13.833900+0000","last_undegraded":"2024-09-21T14:05:13.833900+0000","last_fullsized":"2024-09-21T14:05:13.833900+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:10.722799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:10.722799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:25.238119+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"290'7","reported_seq":577,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.708353+0000","last_change":"2024-09-21T14:04:21.231643+0000","last_active":"2024-09-21T14:05:13.708353+0000","last_peered":"2024-09-21T14:05:13.708353+0000","last_clean":"2024-09-21T14:05:13.708353+0000","last_became_active":"2024-09-21T14:04:21.213102+0000","last_became_peered":"2024-09-21T14:04:21.213102+0000","last_unstale":"2024-09-21T14:05:13.708353+0000","last_undegraded":"2024-09-21T14:05:13.708353+0000","last_fullsized":"2024-09-21T14:05:13.708353+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'7","last_scrub_stamp":"2024-09-21T14:01:57.563184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:57.563184+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":700,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.791959+0000","last_change":"2024-09-21T13:59:32.204565+0000","last_active":"2024-09-21T14:05:13.791959+0000","last_peered":"2024-09-21T14:05:13.791959+0000","last_clean":"2024-09-21T14:05:13.791959+0000","last_became_active":"2024-09-21T13:59:32.204174+0000","last_became_peered":"2024-09-21T13:59:32.204174+0000","last_unstale":"2024-09-21T14:05:13.791959+0000","last_undegraded":"2024-09-21T14:05:13.791959+0000","last_fullsized":"2024-09-21T14:05:13.791959+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:06.722293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:06.722293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":749,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.833193+0000","last_change":"2024-09-21T14:04:46.471869+0000","last_active":"2024-09-21T14:05:13.833193+0000","last_peered":"2024-09-21T14:05:13.833193+0000","last_clean":"2024-09-21T14:05:13.833193+0000","last_became_active":"2024-09-21T14:01:41.198672+0000","last_became_peered":"2024-09-21T14:01:41.198672+0000","last_unstale":"2024-09-21T14:05:13.833193+0000","last_undegraded":"2024-09-21T14:05:13.833193+0000","last_fullsized":"2024-09-21T14:05:13.833193+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:46.471817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:04:46.471817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:01.005314+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":806,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.788320+0000","last_change":"2024-09-21T14:03:36.018910+0000","last_active":"2024-09-21T14:05:13.788320+0000","last_peered":"2024-09-21T14:05:13.788320+0000","last_clean":"2024-09-21T14:05:13.788320+0000","last_became_active":"2024-09-21T14:01:41.200360+0000","last_became_peered":"2024-09-21T14:01:41.200360+0000","last_unstale":"2024-09-21T14:05:13.788320+0000","last_undegraded":"2024-09-21T14:05:13.788320+0000","last_fullsized":"2024-09-21T14:05:13.788320+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:36.018851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:36.018851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":722,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.706994+0000","last_change":"2024-09-21T14:04:21.202574+0000","last_active":"2024-09-21T14:05:13.706994+0000","last_peered":"2024-09-21T14:05:13.706994+0000","last_clean":"2024-09-21T14:05:13.706994+0000","last_became_active":"2024-09-21T14:04:21.202178+0000","last_became_peered":"2024-09-21T14:04:21.202178+0000","last_unstale":"2024-09-21T14:05:13.706994+0000","last_undegraded":"2024-09-21T14:05:13.706994+0000","last_fullsized":"2024-09-21T14:05:13.706994+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:36.679321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:02:36.679321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"290'27","reported_seq":790,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T14:05:17.776915+0000","last_change":"2024-09-21T14:04:01.621388+0000","last_active":"2024-09-21T14:05:17.776915+0000","last_peered":"2024-09-21T14:05:17.776915+0000","last_clean":"2024-09-21T14:05:17.776915+0000","last_became_active":"2024-09-21T13:57:12.680783+0000","last_became_peered":"2024-09-21T13:57:12.680783+0000","last_unstale":"2024-09-21T14:05:17.776915+0000","last_undegraded":"2024-09-21T14:05:17.776915+0000","last_fullsized":"2024-09-21T14:05:17.776915+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'27","last_scrub_stamp":"2024-09-21T14:04:01.621303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:04:01.621303+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:23.163153+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":780,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.706003+0000","last_change":"2024-09-21T14:04:21.204283+0000","last_active":"2024-09-21T14:05:13.706003+0000","last_peered":"2024-09-21T14:05:13.706003+0000","last_clean":"2024-09-21T14:05:13.706003+0000","last_became_active":"2024-09-21T14:04:21.203937+0000","last_became_peered":"2024-09-21T14:04:21.203937+0000","last_unstale":"2024-09-21T14:05:13.706003+0000","last_undegraded":"2024-09-21T14:05:13.706003+0000","last_fullsized":"2024-09-21T14:05:13.706003+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:08.299252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:08.299252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":717,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.708554+0000","last_change":"2024-09-21T14:01:41.191486+0000","last_active":"2024-09-21T14:05:13.708554+0000","last_peered":"2024-09-21T14:05:13.708554+0000","last_clean":"2024-09-21T14:05:13.708554+0000","last_became_active":"2024-09-21T14:01:41.191116+0000","last_became_peered":"2024-09-21T14:01:41.191116+0000","last_unstale":"2024-09-21T14:05:13.708554+0000","last_undegraded":"2024-09-21T14:05:13.708554+0000","last_fullsized":"2024-09-21T14:05:13.708554+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:57.962266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:57.962266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:18.210525+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":775,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.788047+0000","last_change":"2024-09-21T14:00:51.495433+0000","last_active":"2024-09-21T14:05:13.788047+0000","last_peered":"2024-09-21T14:05:13.788047+0000","last_clean":"2024-09-21T14:05:13.788047+0000","last_became_active":"2024-09-21T13:52:04.316016+0000","last_became_peered":"2024-09-21T13:52:04.316016+0000","last_unstale":"2024-09-21T14:05:13.788047+0000","last_undegraded":"2024-09-21T14:05:13.788047+0000","last_fullsized":"2024-09-21T14:05:13.788047+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:00:51.495368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:00:51.495368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"290'18","reported_seq":731,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.708657+0000","last_change":"2024-09-21T14:04:50.393599+0000","last_active":"2024-09-21T14:05:13.708657+0000","last_peered":"2024-09-21T14:05:13.708657+0000","last_clean":"2024-09-21T14:05:13.708657+0000","last_became_active":"2024-09-21T13:57:12.684454+0000","last_became_peered":"2024-09-21T13:57:12.684454+0000","last_unstale":"2024-09-21T14:05:13.708657+0000","last_undegraded":"2024-09-21T14:05:13.708657+0000","last_fullsized":"2024-09-21T14:05:13.708657+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'18","last_scrub_stamp":"2024-09-21T14:04:50.393462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:04:50.393462+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:00.435535+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":770,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.788375+0000","last_change":"2024-09-21T14:03:38.065257+0000","last_active":"2024-09-21T14:05:13.788375+0000","last_peered":"2024-09-21T14:05:13.788375+0000","last_clean":"2024-09-21T14:05:13.788375+0000","last_became_active":"2024-09-21T13:57:12.681897+0000","last_became_peered":"2024-09-21T13:57:12.681897+0000","last_unstale":"2024-09-21T14:05:13.788375+0000","last_undegraded":"2024-09-21T14:05:13.788375+0000","last_fullsized":"2024-09-21T14:05:13.788375+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:38.065196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:38.065196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":692,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.792141+0000","last_change":"2024-09-21T14:03:28.933216+0000","last_active":"2024-09-21T14:05:13.792141+0000","last_peered":"2024-09-21T14:05:13.792141+0000","last_clean":"2024-09-21T14:05:13.792141+0000","last_became_active":"2024-09-21T14:01:41.181946+0000","last_became_peered":"2024-09-21T14:01:41.181946+0000","last_unstale":"2024-09-21T14:05:13.792141+0000","last_undegraded":"2024-09-21T14:05:13.792141+0000","last_fullsized":"2024-09-21T14:05:13.792141+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T14:03:28.933143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:28.933143+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":796,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.705039+0000","last_change":"2024-09-21T14:04:21.215322+0000","last_active":"2024-09-21T14:05:13.705039+0000","last_peered":"2024-09-21T14:05:13.705039+0000","last_clean":"2024-09-21T14:05:13.705039+0000","last_became_active":"2024-09-21T14:04:21.200889+0000","last_became_peered":"2024-09-21T14:04:21.200889+0000","last_unstale":"2024-09-21T14:05:13.705039+0000","last_undegraded":"2024-09-21T14:05:13.705039+0000","last_fullsized":"2024-09-21T14:05:13.705039+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T14:01:58.994811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:01:58.994811+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"290'20","reported_seq":113,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.706144+0000","last_change":"2024-09-21T14:04:56.589169+0000","last_active":"2024-09-21T14:05:13.706144+0000","last_peered":"2024-09-21T14:05:13.706144+0000","last_clean":"2024-09-21T14:05:13.706144+0000","last_became_active":"2024-09-21T14:04:56.588859+0000","last_became_peered":"2024-09-21T14:04:56.588859+0000","last_unstale":"2024-09-21T14:05:13.706144+0000","last_undegraded":"2024-09-21T14:05:13.706144+0000","last_fullsized":"2024-09-21T14:05:13.706144+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'20","last_scrub_stamp":"2024-09-21T14:04:42.048082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:04:42.048082+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:51.310282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"409'13","reported_seq":440,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-21T14:05:12.717484+0000","last_change":"2024-09-21T14:04:21.222657+0000","last_active":"2024-09-21T14:05:12.717484+0000","last_peered":"2024-09-21T14:05:12.717484+0000","last_clean":"2024-09-21T14:05:12.717484+0000","last_became_active":"2024-09-21T14:04:21.206136+0000","last_became_peered":"2024-09-21T14:04:21.206136+0000","last_unstale":"2024-09-21T14:05:12.717484+0000","last_undegraded":"2024-09-21T14:05:12.717484+0000","last_fullsized":"2024-09-21T14:05:12.717484+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'12","last_scrub_stamp":"2024-09-21T13:59:36.012693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:36.012693+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:27.834286+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"422'10523","reported_seq":10692,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.706608+0000","last_change":"2024-09-21T14:04:09.971242+0000","last_active":"2024-09-21T14:05:13.706608+0000","last_peered":"2024-09-21T14:05:13.706608+0000","last_clean":"2024-09-21T14:05:13.706608+0000","last_became_active":"2024-09-21T14:02:56.559536+0000","last_became_peered":"2024-09-21T14:02:56.559536+0000","last_unstale":"2024-09-21T14:05:13.706608+0000","last_undegraded":"2024-09-21T14:05:13.706608+0000","last_fullsized":"2024-09-21T14:05:13.706608+0000","mapping_epoch":381,"log_start":"399'5000","ondisk_log_start":"399'5000","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"406'6478","last_scrub_stamp":"2024-09-21T14:04:09.971039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:04:09.971039+0000","objects_scrubbed":6417,"log_size":5523,"log_dups_size":0,"ondisk_log_size":5523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:17.197608+0000","scrub_duration":1835,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":374013952,"num_objects":5707,"num_object_clones":0,"num_object_copies":17121,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10523,"num_write_kb":519360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"75'79","reported_seq":188,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.835709+0000","last_change":"2024-09-21T14:03:59.776621+0000","last_active":"2024-09-21T14:05:13.835709+0000","last_peered":"2024-09-21T14:05:13.835709+0000","last_clean":"2024-09-21T14:05:13.835709+0000","last_became_active":"2024-09-21T14:01:41.246888+0000","last_became_peered":"2024-09-21T14:01:41.246888+0000","last_unstale":"2024-09-21T14:05:13.835709+0000","last_undegraded":"2024-09-21T14:05:13.835709+0000","last_fullsized":"2024-09-21T14:05:13.835709+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:03:59.776552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:03:59.776552+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:21.450064+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"290'27","reported_seq":840,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.706467+0000","last_change":"2024-09-21T14:04:03.259647+0000","last_active":"2024-09-21T14:05:13.706467+0000","last_peered":"2024-09-21T14:05:13.706467+0000","last_clean":"2024-09-21T14:05:13.706467+0000","last_became_active":"2024-09-21T13:57:12.695609+0000","last_became_peered":"2024-09-21T13:57:12.695609+0000","last_unstale":"2024-09-21T14:05:13.706467+0000","last_undegraded":"2024-09-21T14:05:13.706467+0000","last_fullsized":"2024-09-21T14:05:13.706467+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'27","last_scrub_stamp":"2024-09-21T14:04:03.259554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:04:03.259554+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:20.677105+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":717,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.707637+0000","last_change":"2024-09-21T14:04:01.183193+0000","last_active":"2024-09-21T14:05:13.707637+0000","last_peered":"2024-09-21T14:05:13.707637+0000","last_clean":"2024-09-21T14:05:13.707637+0000","last_became_active":"2024-09-21T13:59:32.205350+0000","last_became_peered":"2024-09-21T13:59:32.205350+0000","last_unstale":"2024-09-21T14:05:13.707637+0000","last_undegraded":"2024-09-21T14:05:13.707637+0000","last_fullsized":"2024-09-21T14:05:13.707637+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:01.183123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:01.183123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"409'11","reported_seq":825,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.834397+0000","last_change":"2024-09-21T14:04:21.211284+0000","last_active":"2024-09-21T14:05:13.834397+0000","last_peered":"2024-09-21T14:05:13.834397+0000","last_clean":"2024-09-21T14:05:13.834397+0000","last_became_active":"2024-09-21T14:04:21.206029+0000","last_became_peered":"2024-09-21T14:04:21.206029+0000","last_unstale":"2024-09-21T14:05:13.834397+0000","last_undegraded":"2024-09-21T14:05:13.834397+0000","last_fullsized":"2024-09-21T14:05:13.834397+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'10","last_scrub_stamp":"2024-09-21T14:02:59.831128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:02:59.831128+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:07.398197+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":651,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.833673+0000","last_change":"2024-09-21T14:04:47.464272+0000","last_active":"2024-09-21T14:05:13.833673+0000","last_peered":"2024-09-21T14:05:13.833673+0000","last_clean":"2024-09-21T14:05:13.833673+0000","last_became_active":"2024-09-21T14:04:21.203505+0000","last_became_peered":"2024-09-21T14:04:21.203505+0000","last_unstale":"2024-09-21T14:05:13.833673+0000","last_undegraded":"2024-09-21T14:05:13.833673+0000","last_fullsized":"2024-09-21T14:05:13.833673+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:47.464207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:04:47.464207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:48.380565+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"422'10067","reported_seq":10232,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.787922+0000","last_change":"2024-09-21T14:04:08.894792+0000","last_active":"2024-09-21T14:05:13.787922+0000","last_peered":"2024-09-21T14:05:13.787922+0000","last_clean":"2024-09-21T14:05:13.787922+0000","last_became_active":"2024-09-21T14:02:56.562524+0000","last_became_peered":"2024-09-21T14:02:56.562524+0000","last_unstale":"2024-09-21T14:05:13.787922+0000","last_undegraded":"2024-09-21T14:05:13.787922+0000","last_fullsized":"2024-09-21T14:05:13.787922+0000","mapping_epoch":381,"log_start":"398'4600","ondisk_log_start":"398'4600","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"406'6081","last_scrub_stamp":"2024-09-21T14:04:08.894653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:04:08.894653+0000","objects_scrubbed":6008,"log_size":5467,"log_dups_size":0,"ondisk_log_size":5467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:26.881293+0000","scrub_duration":1756,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":363266048,"num_objects":5543,"num_object_clones":0,"num_object_copies":16629,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10067,"num_write_kb":499520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"75'79","reported_seq":234,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.788696+0000","last_change":"2024-09-21T14:02:06.395083+0000","last_active":"2024-09-21T14:05:13.788696+0000","last_peered":"2024-09-21T14:05:13.788696+0000","last_clean":"2024-09-21T14:05:13.788696+0000","last_became_active":"2024-09-21T14:00:41.137756+0000","last_became_peered":"2024-09-21T14:00:41.137756+0000","last_unstale":"2024-09-21T14:05:13.788696+0000","last_undegraded":"2024-09-21T14:05:13.788696+0000","last_fullsized":"2024-09-21T14:05:13.788696+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:02:06.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:02:06.395011+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"290'30","reported_seq":759,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.707653+0000","last_change":"2024-09-21T14:04:21.228589+0000","last_active":"2024-09-21T14:05:13.707653+0000","last_peered":"2024-09-21T14:05:13.707653+0000","last_clean":"2024-09-21T14:05:13.707653+0000","last_became_active":"2024-09-21T14:04:21.208982+0000","last_became_peered":"2024-09-21T14:04:21.208982+0000","last_unstale":"2024-09-21T14:05:13.707653+0000","last_undegraded":"2024-09-21T14:05:13.707653+0000","last_fullsized":"2024-09-21T14:05:13.707653+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'18","last_scrub_stamp":"2024-09-21T13:54:19.001030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:19.001030+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:23.192760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":669,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.708671+0000","last_change":"2024-09-21T14:04:02.214555+0000","last_active":"2024-09-21T14:05:13.708671+0000","last_peered":"2024-09-21T14:05:13.708671+0000","last_clean":"2024-09-21T14:05:13.708671+0000","last_became_active":"2024-09-21T13:59:32.211004+0000","last_became_peered":"2024-09-21T13:59:32.211004+0000","last_unstale":"2024-09-21T14:05:13.708671+0000","last_undegraded":"2024-09-21T14:05:13.708671+0000","last_fullsized":"2024-09-21T14:05:13.708671+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:02.214464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:02.214464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":779,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.707141+0000","last_change":"2024-09-21T13:59:34.293330+0000","last_active":"2024-09-21T14:05:13.707141+0000","last_peered":"2024-09-21T14:05:13.707141+0000","last_clean":"2024-09-21T14:05:13.707141+0000","last_became_active":"2024-09-21T13:59:32.220304+0000","last_became_peered":"2024-09-21T13:59:32.220304+0000","last_unstale":"2024-09-21T14:05:13.707141+0000","last_undegraded":"2024-09-21T14:05:13.707141+0000","last_fullsized":"2024-09-21T14:05:13.707141+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:34.292892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:34.292892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":415,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.708273+0000","last_change":"2024-09-21T14:04:21.210131+0000","last_active":"2024-09-21T14:05:13.708273+0000","last_peered":"2024-09-21T14:05:13.708273+0000","last_clean":"2024-09-21T14:05:13.708273+0000","last_became_active":"2024-09-21T14:04:21.207782+0000","last_became_peered":"2024-09-21T14:04:21.207782+0000","last_unstale":"2024-09-21T14:05:13.708273+0000","last_undegraded":"2024-09-21T14:05:13.708273+0000","last_fullsized":"2024-09-21T14:05:13.708273+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:23.334162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:23.334162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"422'10504","reported_seq":10684,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-21T14:05:12.934694+0000","last_change":"2024-09-21T14:04:42.544112+0000","last_active":"2024-09-21T14:05:12.934694+0000","last_peered":"2024-09-21T14:05:12.934694+0000","last_clean":"2024-09-21T14:05:12.934694+0000","last_became_active":"2024-09-21T14:02:56.557055+0000","last_became_peered":"2024-09-21T14:02:56.557055+0000","last_unstale":"2024-09-21T14:05:12.934694+0000","last_undegraded":"2024-09-21T14:05:12.934694+0000","last_fullsized":"2024-09-21T14:05:12.934694+0000","mapping_epoch":381,"log_start":"401'5410","ondisk_log_start":"401'5410","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'8494","last_scrub_stamp":"2024-09-21T14:04:42.543986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:04:42.543986+0000","objects_scrubbed":7689,"log_size":5094,"log_dups_size":0,"ondisk_log_size":5094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:45.816257+0000","scrub_duration":2665,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":366739456,"num_objects":5596,"num_object_clones":0,"num_object_copies":16788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10504,"num_write_kb":515200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"75'79","reported_seq":245,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.708465+0000","last_change":"2024-09-21T14:04:49.393518+0000","last_active":"2024-09-21T14:05:13.708465+0000","last_peered":"2024-09-21T14:05:13.708465+0000","last_clean":"2024-09-21T14:05:13.708465+0000","last_became_active":"2024-09-21T14:01:42.173042+0000","last_became_peered":"2024-09-21T14:01:42.173042+0000","last_unstale":"2024-09-21T14:05:13.708465+0000","last_undegraded":"2024-09-21T14:05:13.708465+0000","last_fullsized":"2024-09-21T14:05:13.708465+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:04:49.393415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:04:49.393415+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:56.538057+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"290'28","reported_seq":876,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.706755+0000","last_change":"2024-09-21T14:03:27.047111+0000","last_active":"2024-09-21T14:05:13.706755+0000","last_peered":"2024-09-21T14:05:13.706755+0000","last_clean":"2024-09-21T14:05:13.706755+0000","last_became_active":"2024-09-21T14:01:41.200859+0000","last_became_peered":"2024-09-21T14:01:41.200859+0000","last_unstale":"2024-09-21T14:05:13.706755+0000","last_undegraded":"2024-09-21T14:05:13.706755+0000","last_fullsized":"2024-09-21T14:05:13.706755+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'28","last_scrub_stamp":"2024-09-21T14:03:27.046906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:03:27.046906+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":54,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.706398+0000","last_change":"2024-09-21T14:04:21.216890+0000","last_active":"2024-09-21T14:05:13.706398+0000","last_peered":"2024-09-21T14:05:13.706398+0000","last_clean":"2024-09-21T14:05:13.706398+0000","last_became_active":"2024-09-21T14:04:21.205354+0000","last_became_peered":"2024-09-21T14:04:21.205354+0000","last_unstale":"2024-09-21T14:05:13.706398+0000","last_undegraded":"2024-09-21T14:05:13.706398+0000","last_fullsized":"2024-09-21T14:05:13.706398+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:57:38.046627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:38.046627+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"241'10","reported_seq":847,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.707835+0000","last_change":"2024-09-21T14:03:03.228551+0000","last_active":"2024-09-21T14:05:13.707835+0000","last_peered":"2024-09-21T14:05:13.707835+0000","last_clean":"2024-09-21T14:05:13.707835+0000","last_became_active":"2024-09-21T13:57:12.686651+0000","last_became_peered":"2024-09-21T13:57:12.686651+0000","last_unstale":"2024-09-21T14:05:13.707835+0000","last_undegraded":"2024-09-21T14:05:13.707835+0000","last_fullsized":"2024-09-21T14:05:13.707835+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'10","last_scrub_stamp":"2024-09-21T14:03:03.228443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:03.228443+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":733,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.792524+0000","last_change":"2024-09-21T14:04:45.055770+0000","last_active":"2024-09-21T14:05:13.792524+0000","last_peered":"2024-09-21T14:05:13.792524+0000","last_clean":"2024-09-21T14:05:13.792524+0000","last_became_active":"2024-09-21T13:59:32.209615+0000","last_became_peered":"2024-09-21T13:59:32.209615+0000","last_unstale":"2024-09-21T14:05:13.792524+0000","last_undegraded":"2024-09-21T14:05:13.792524+0000","last_fullsized":"2024-09-21T14:05:13.792524+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:45.055707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:04:45.055707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:03.598442+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.c","version":"423'10267","reported_seq":10364,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:16.120438+0000","last_change":"2024-09-21T14:02:56.563139+0000","last_active":"2024-09-21T14:05:16.120438+0000","last_peered":"2024-09-21T14:05:16.120438+0000","last_clean":"2024-09-21T14:05:16.120438+0000","last_became_active":"2024-09-21T14:02:56.562806+0000","last_became_peered":"2024-09-21T14:02:56.562806+0000","last_unstale":"2024-09-21T14:05:16.120438+0000","last_undegraded":"2024-09-21T14:05:16.120438+0000","last_fullsized":"2024-09-21T14:05:16.120438+0000","mapping_epoch":381,"log_start":"404'5709","ondisk_log_start":"404'5709","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":4558,"log_dups_size":0,"ondisk_log_size":4558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":351469568,"num_objects":5363,"num_object_clones":0,"num_object_copies":16089,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10267,"num_write_kb":500160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"75'79","reported_seq":813,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-21T14:05:12.717019+0000","last_change":"2024-09-21T14:04:38.868411+0000","last_active":"2024-09-21T14:05:12.717019+0000","last_peered":"2024-09-21T14:05:12.717019+0000","last_clean":"2024-09-21T14:05:12.717019+0000","last_became_active":"2024-09-21T14:00:41.083425+0000","last_became_peered":"2024-09-21T14:00:41.083425+0000","last_unstale":"2024-09-21T14:05:12.717019+0000","last_undegraded":"2024-09-21T14:05:12.717019+0000","last_fullsized":"2024-09-21T14:05:12.717019+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:04:38.868343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:04:38.868343+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:49.409572+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"290'20","reported_seq":102,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T14:05:17.780753+0000","last_change":"2024-09-21T14:04:42.776698+0000","last_active":"2024-09-21T14:05:17.780753+0000","last_peered":"2024-09-21T14:05:17.780753+0000","last_clean":"2024-09-21T14:05:17.780753+0000","last_became_active":"2024-09-21T14:03:29.158218+0000","last_became_peered":"2024-09-21T14:03:29.158218+0000","last_unstale":"2024-09-21T14:05:17.780753+0000","last_undegraded":"2024-09-21T14:05:17.780753+0000","last_fullsized":"2024-09-21T14:05:17.780753+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'20","last_scrub_stamp":"2024-09-21T14:04:42.776603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:04:42.776603+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:58.993778+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"61'1","reported_seq":706,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.833753+0000","last_change":"2024-09-21T14:04:48.425745+0000","last_active":"2024-09-21T14:05:13.833753+0000","last_peered":"2024-09-21T14:05:13.833753+0000","last_clean":"2024-09-21T14:05:13.833753+0000","last_became_active":"2024-09-21T14:01:41.195268+0000","last_became_peered":"2024-09-21T14:01:41.195268+0000","last_unstale":"2024-09-21T14:05:13.833753+0000","last_undegraded":"2024-09-21T14:05:13.833753+0000","last_fullsized":"2024-09-21T14:05:13.833753+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T14:04:48.425665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:48.425665+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:14.101369+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"418'16","reported_seq":121,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.705564+0000","last_change":"2024-09-21T14:04:56.590159+0000","last_active":"2024-09-21T14:05:13.705564+0000","last_peered":"2024-09-21T14:05:13.705564+0000","last_clean":"2024-09-21T14:05:13.705564+0000","last_became_active":"2024-09-21T14:04:56.589828+0000","last_became_peered":"2024-09-21T14:04:56.589828+0000","last_unstale":"2024-09-21T14:05:13.705564+0000","last_undegraded":"2024-09-21T14:05:13.705564+0000","last_fullsized":"2024-09-21T14:05:13.705564+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'15","last_scrub_stamp":"2024-09-21T14:04:43.936597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:04:43.936597+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:47.698681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":217,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.704944+0000","last_change":"2024-09-21T14:04:56.587813+0000","last_active":"2024-09-21T14:05:13.704944+0000","last_peered":"2024-09-21T14:05:13.704944+0000","last_clean":"2024-09-21T14:05:13.704944+0000","last_became_active":"2024-09-21T14:04:56.586891+0000","last_became_peered":"2024-09-21T14:04:56.586891+0000","last_unstale":"2024-09-21T14:05:13.704944+0000","last_undegraded":"2024-09-21T14:05:13.704944+0000","last_fullsized":"2024-09-21T14:05:13.704944+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:41.400346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:04:41.400346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:41.941073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.d","version":"422'10371","reported_seq":16194,"reported_epoch":422,"state":"active+remapped+backfilling","last_fresh":"2024-09-21T14:05:13.788216+0000","last_change":"2024-09-21T14:04:52.308462+0000","last_active":"2024-09-21T14:05:13.788216+0000","last_peered":"2024-09-21T14:05:13.788216+0000","last_clean":"2024-09-21T14:04:20.166030+0000","last_became_active":"2024-09-21T14:04:22.281826+0000","last_became_peered":"2024-09-21T14:04:22.281826+0000","last_unstale":"2024-09-21T14:05:13.788216+0000","last_undegraded":"2024-09-21T14:05:13.788216+0000","last_fullsized":"2024-09-21T14:05:13.788216+0000","mapping_epoch":409,"log_start":"400'5268","ondisk_log_start":"400'5268","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"406'6403","last_scrub_stamp":"2024-09-21T14:04:11.019087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:04:11.019087+0000","objects_scrubbed":6346,"log_size":5103,"log_dups_size":0,"ondisk_log_size":5103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1942,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":367525912,"num_objects":5609,"num_object_clones":0,"num_object_copies":16827,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3045,"num_objects_unfound":0,"num_objects_dirty":5609,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10371,"num_write_kb":511297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2823,"num_bytes_recovered":185008128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,0],"avail_no_missing":["6","0","4"],"object_location_counts":[{"shards":"0,4,6","objects":5609}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"75'79","reported_seq":821,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-21T14:05:12.710487+0000","last_change":"2024-09-21T14:04:44.986902+0000","last_active":"2024-09-21T14:05:12.710487+0000","last_peered":"2024-09-21T14:05:12.710487+0000","last_clean":"2024-09-21T14:05:12.710487+0000","last_became_active":"2024-09-21T14:00:41.073047+0000","last_became_peered":"2024-09-21T14:00:41.073047+0000","last_unstale":"2024-09-21T14:05:12.710487+0000","last_undegraded":"2024-09-21T14:05:12.710487+0000","last_fullsized":"2024-09-21T14:05:12.710487+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:04:44.986816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:04:44.986816+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:58.169458+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"290'29","reported_seq":518,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.705771+0000","last_change":"2024-09-21T14:04:56.589405+0000","last_active":"2024-09-21T14:05:13.705771+0000","last_peered":"2024-09-21T14:05:13.705771+0000","last_clean":"2024-09-21T14:05:13.705771+0000","last_became_active":"2024-09-21T14:04:56.589018+0000","last_became_peered":"2024-09-21T14:04:56.589018+0000","last_unstale":"2024-09-21T14:05:13.705771+0000","last_undegraded":"2024-09-21T14:05:13.705771+0000","last_fullsized":"2024-09-21T14:05:13.705771+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'29","last_scrub_stamp":"2024-09-21T14:04:53.477766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:04:53.477766+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:12.011852+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":780,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.706021+0000","last_change":"2024-09-21T14:01:41.185706+0000","last_active":"2024-09-21T14:05:13.706021+0000","last_peered":"2024-09-21T14:05:13.706021+0000","last_clean":"2024-09-21T14:05:13.706021+0000","last_became_active":"2024-09-21T14:01:41.185400+0000","last_became_peered":"2024-09-21T14:01:41.185400+0000","last_unstale":"2024-09-21T14:05:13.706021+0000","last_undegraded":"2024-09-21T14:05:13.706021+0000","last_fullsized":"2024-09-21T14:05:13.706021+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:16.712990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:16.712990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":362,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-21T14:05:12.715626+0000","last_change":"2024-09-21T14:04:03.070260+0000","last_active":"2024-09-21T14:05:12.715626+0000","last_peered":"2024-09-21T14:05:12.715626+0000","last_clean":"2024-09-21T14:05:12.715626+0000","last_became_active":"2024-09-21T14:01:41.193973+0000","last_became_peered":"2024-09-21T14:01:41.193973+0000","last_unstale":"2024-09-21T14:05:12.715626+0000","last_undegraded":"2024-09-21T14:05:12.715626+0000","last_fullsized":"2024-09-21T14:05:12.715626+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:03.069973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:04:03.069973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:26.954137+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":771,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.704796+0000","last_change":"2024-09-21T13:58:43.952014+0000","last_active":"2024-09-21T14:05:13.704796+0000","last_peered":"2024-09-21T14:05:13.704796+0000","last_clean":"2024-09-21T14:05:13.704796+0000","last_became_active":"2024-09-21T13:58:17.757628+0000","last_became_peered":"2024-09-21T13:58:17.757628+0000","last_unstale":"2024-09-21T14:05:13.704796+0000","last_undegraded":"2024-09-21T14:05:13.704796+0000","last_fullsized":"2024-09-21T14:05:13.704796+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:43.951941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:43.951941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"422'10310","reported_seq":16237,"reported_epoch":422,"state":"active+remapped+backfilling","last_fresh":"2024-09-21T14:05:13.788993+0000","last_change":"2024-09-21T14:04:52.389607+0000","last_active":"2024-09-21T14:05:13.788993+0000","last_peered":"2024-09-21T14:05:13.788993+0000","last_clean":"2024-09-21T14:04:20.165921+0000","last_became_active":"2024-09-21T14:04:22.361054+0000","last_became_peered":"2024-09-21T14:04:22.361054+0000","last_unstale":"2024-09-21T14:05:13.788993+0000","last_undegraded":"2024-09-21T14:05:13.788993+0000","last_fullsized":"2024-09-21T14:05:13.788993+0000","mapping_epoch":409,"log_start":"400'5148","ondisk_log_start":"400'5148","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":3149,"log_size":5162,"log_dups_size":0,"ondisk_log_size":5162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":360185856,"num_objects":5496,"num_object_clones":0,"num_object_copies":16488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2873,"num_objects_unfound":0,"num_objects_dirty":5496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10310,"num_write_kb":505792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2894,"num_bytes_recovered":189661184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,0],"avail_no_missing":["6","0","5"],"object_location_counts":[{"shards":"0,5,6","objects":5496}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"75'79","reported_seq":180,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.833270+0000","last_change":"2024-09-21T14:01:41.199377+0000","last_active":"2024-09-21T14:05:13.833270+0000","last_peered":"2024-09-21T14:05:13.833270+0000","last_clean":"2024-09-21T14:05:13.833270+0000","last_became_active":"2024-09-21T14:01:41.198750+0000","last_became_peered":"2024-09-21T14:01:41.198750+0000","last_unstale":"2024-09-21T14:05:13.833270+0000","last_undegraded":"2024-09-21T14:05:13.833270+0000","last_fullsized":"2024-09-21T14:05:13.833270+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:05:59.493308+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"290'44","reported_seq":1050,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.705829+0000","last_change":"2024-09-21T14:02:57.316829+0000","last_active":"2024-09-21T14:05:13.705829+0000","last_peered":"2024-09-21T14:05:13.705829+0000","last_clean":"2024-09-21T14:05:13.705829+0000","last_became_active":"2024-09-21T13:57:12.684690+0000","last_became_peered":"2024-09-21T13:57:12.684690+0000","last_unstale":"2024-09-21T14:05:13.705829+0000","last_undegraded":"2024-09-21T14:05:13.705829+0000","last_fullsized":"2024-09-21T14:05:13.705829+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'44","last_scrub_stamp":"2024-09-21T14:02:57.316763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:57.316763+0000","objects_scrubbed":10,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":795,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.707041+0000","last_change":"2024-09-21T14:04:11.158776+0000","last_active":"2024-09-21T14:05:13.707041+0000","last_peered":"2024-09-21T14:05:13.707041+0000","last_clean":"2024-09-21T14:05:13.707041+0000","last_became_active":"2024-09-21T13:58:17.770937+0000","last_became_peered":"2024-09-21T13:58:17.770937+0000","last_unstale":"2024-09-21T14:05:13.707041+0000","last_undegraded":"2024-09-21T14:05:13.707041+0000","last_fullsized":"2024-09-21T14:05:13.707041+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:11.158703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:11.158703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:26.630567+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":733,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:15.377389+0000","last_change":"2024-09-21T14:05:15.377389+0000","last_active":"2024-09-21T14:05:15.377389+0000","last_peered":"2024-09-21T14:05:15.377389+0000","last_clean":"2024-09-21T14:05:15.377389+0000","last_became_active":"2024-09-21T14:01:41.247642+0000","last_became_peered":"2024-09-21T14:01:41.247642+0000","last_unstale":"2024-09-21T14:05:15.377389+0000","last_undegraded":"2024-09-21T14:05:15.377389+0000","last_fullsized":"2024-09-21T14:05:15.377389+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-21T14:05:15.377198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T14:05:15.377198+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:31.018201+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":706,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.705790+0000","last_change":"2024-09-21T14:04:21.204705+0000","last_active":"2024-09-21T14:05:13.705790+0000","last_peered":"2024-09-21T14:05:13.705790+0000","last_clean":"2024-09-21T14:05:13.705790+0000","last_became_active":"2024-09-21T14:04:21.204315+0000","last_became_peered":"2024-09-21T14:04:21.204315+0000","last_unstale":"2024-09-21T14:05:13.705790+0000","last_undegraded":"2024-09-21T14:05:13.705790+0000","last_fullsized":"2024-09-21T14:05:13.705790+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:07.021059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:55:07.021059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":696,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.708728+0000","last_change":"2024-09-21T14:03:02.238409+0000","last_active":"2024-09-21T14:05:13.708728+0000","last_peered":"2024-09-21T14:05:13.708728+0000","last_clean":"2024-09-21T14:05:13.708728+0000","last_became_active":"2024-09-21T13:59:32.208980+0000","last_became_peered":"2024-09-21T13:59:32.208980+0000","last_unstale":"2024-09-21T14:05:13.708728+0000","last_undegraded":"2024-09-21T14:05:13.708728+0000","last_fullsized":"2024-09-21T14:05:13.708728+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:02.238357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:02.238357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"423'10352","reported_seq":10517,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:15.929448+0000","last_change":"2024-09-21T14:04:06.243298+0000","last_active":"2024-09-21T14:05:15.929448+0000","last_peered":"2024-09-21T14:05:15.929448+0000","last_clean":"2024-09-21T14:05:15.929448+0000","last_became_active":"2024-09-21T14:02:56.558773+0000","last_became_peered":"2024-09-21T14:02:56.558773+0000","last_unstale":"2024-09-21T14:05:15.929448+0000","last_undegraded":"2024-09-21T14:05:15.929448+0000","last_fullsized":"2024-09-21T14:05:15.929448+0000","mapping_epoch":381,"log_start":"403'5469","ondisk_log_start":"403'5469","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'5948","last_scrub_stamp":"2024-09-21T14:04:06.243212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:04:06.243212+0000","objects_scrubbed":5906,"log_size":4883,"log_dups_size":0,"ondisk_log_size":4883,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1458,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":351666176,"num_objects":5366,"num_object_clones":0,"num_object_copies":16098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10352,"num_write_kb":502976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"75'79","reported_seq":820,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.791708+0000","last_change":"2024-09-21T14:04:21.206885+0000","last_active":"2024-09-21T14:05:13.791708+0000","last_peered":"2024-09-21T14:05:13.791708+0000","last_clean":"2024-09-21T14:05:13.791708+0000","last_became_active":"2024-09-21T14:04:21.206472+0000","last_became_peered":"2024-09-21T14:04:21.206472+0000","last_unstale":"2024-09-21T14:05:13.791708+0000","last_undegraded":"2024-09-21T14:05:13.791708+0000","last_fullsized":"2024-09-21T14:05:13.791708+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:05:23.132366+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"290'17","reported_seq":825,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.706551+0000","last_change":"2024-09-21T14:04:21.224066+0000","last_active":"2024-09-21T14:05:13.706551+0000","last_peered":"2024-09-21T14:05:13.706551+0000","last_clean":"2024-09-21T14:05:13.706551+0000","last_became_active":"2024-09-21T14:04:21.205738+0000","last_became_peered":"2024-09-21T14:04:21.205738+0000","last_unstale":"2024-09-21T14:05:13.706551+0000","last_undegraded":"2024-09-21T14:05:13.706551+0000","last_fullsized":"2024-09-21T14:05:13.706551+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'13","last_scrub_stamp":"2024-09-21T13:58:11.619848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:11.619848+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":824,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.704675+0000","last_change":"2024-09-21T14:04:10.193524+0000","last_active":"2024-09-21T14:05:13.704675+0000","last_peered":"2024-09-21T14:05:13.704675+0000","last_clean":"2024-09-21T14:05:13.704675+0000","last_became_active":"2024-09-21T13:58:17.759671+0000","last_became_peered":"2024-09-21T13:58:17.759671+0000","last_unstale":"2024-09-21T14:05:13.704675+0000","last_undegraded":"2024-09-21T14:05:13.704675+0000","last_fullsized":"2024-09-21T14:05:13.704675+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:10.193318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:10.193318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:25.269486+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":699,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T14:05:17.772637+0000","last_change":"2024-09-21T13:58:41.644876+0000","last_active":"2024-09-21T14:05:17.772637+0000","last_peered":"2024-09-21T14:05:17.772637+0000","last_clean":"2024-09-21T14:05:17.772637+0000","last_became_active":"2024-09-21T13:57:12.682147+0000","last_became_peered":"2024-09-21T13:57:12.682147+0000","last_unstale":"2024-09-21T14:05:17.772637+0000","last_undegraded":"2024-09-21T14:05:17.772637+0000","last_fullsized":"2024-09-21T14:05:17.772637+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:58:41.644789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:58:41.644789+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":678,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T14:05:17.779149+0000","last_change":"2024-09-21T14:01:50.880840+0000","last_active":"2024-09-21T14:05:17.779149+0000","last_peered":"2024-09-21T14:05:17.779149+0000","last_clean":"2024-09-21T14:05:17.779149+0000","last_became_active":"2024-09-21T14:01:41.182000+0000","last_became_peered":"2024-09-21T14:01:41.182000+0000","last_unstale":"2024-09-21T14:05:17.779149+0000","last_undegraded":"2024-09-21T14:05:17.779149+0000","last_fullsized":"2024-09-21T14:05:17.779149+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:50.880471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:01:50.880471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:20.548012+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":733,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.792508+0000","last_change":"2024-09-21T13:59:32.222791+0000","last_active":"2024-09-21T14:05:13.792508+0000","last_peered":"2024-09-21T14:05:13.792508+0000","last_clean":"2024-09-21T14:05:13.792508+0000","last_became_active":"2024-09-21T13:59:32.221502+0000","last_became_peered":"2024-09-21T13:59:32.221502+0000","last_unstale":"2024-09-21T14:05:13.792508+0000","last_undegraded":"2024-09-21T14:05:13.792508+0000","last_fullsized":"2024-09-21T14:05:13.792508+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:14.673582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:14.673582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"423'10492","reported_seq":10662,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:14.495297+0000","last_change":"2024-09-21T14:04:56.601777+0000","last_active":"2024-09-21T14:05:14.495297+0000","last_peered":"2024-09-21T14:05:14.495297+0000","last_clean":"2024-09-21T14:05:14.495297+0000","last_became_active":"2024-09-21T14:04:56.586861+0000","last_became_peered":"2024-09-21T14:04:56.586861+0000","last_unstale":"2024-09-21T14:05:14.495297+0000","last_undegraded":"2024-09-21T14:05:14.495297+0000","last_fullsized":"2024-09-21T14:05:14.495297+0000","mapping_epoch":417,"log_start":"398'4382","ondisk_log_start":"398'4382","created":381,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'6143","last_scrub_stamp":"2024-09-21T14:04:06.996122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:04:06.996122+0000","objects_scrubbed":6071,"log_size":6110,"log_dups_size":0,"ondisk_log_size":6110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:36.829006+0000","scrub_duration":2407,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":367919104,"num_objects":5614,"num_object_clones":0,"num_object_copies":16842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10492,"num_write_kb":515392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"75'79","reported_seq":233,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.706685+0000","last_change":"2024-09-21T14:05:13.503769+0000","last_active":"2024-09-21T14:05:13.706685+0000","last_peered":"2024-09-21T14:05:13.706685+0000","last_clean":"2024-09-21T14:05:13.706685+0000","last_became_active":"2024-09-21T14:04:21.210586+0000","last_became_peered":"2024-09-21T14:04:21.210586+0000","last_unstale":"2024-09-21T14:05:13.706685+0000","last_undegraded":"2024-09-21T14:05:13.706685+0000","last_fullsized":"2024-09-21T14:05:13.706685+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:05:13.503720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:05:13.503720+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:33.310357+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"290'24","reported_seq":622,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.834522+0000","last_change":"2024-09-21T14:04:21.221906+0000","last_active":"2024-09-21T14:05:13.834522+0000","last_peered":"2024-09-21T14:05:13.834522+0000","last_clean":"2024-09-21T14:05:13.834522+0000","last_became_active":"2024-09-21T14:04:21.206881+0000","last_became_peered":"2024-09-21T14:04:21.206881+0000","last_unstale":"2024-09-21T14:05:13.834522+0000","last_undegraded":"2024-09-21T14:05:13.834522+0000","last_fullsized":"2024-09-21T14:05:13.834522+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'24","last_scrub_stamp":"2024-09-21T14:02:57.559936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:57.559936+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:06.366044+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":341,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.705317+0000","last_change":"2024-09-21T14:04:56.588103+0000","last_active":"2024-09-21T14:05:13.705317+0000","last_peered":"2024-09-21T14:05:13.705317+0000","last_clean":"2024-09-21T14:05:13.705317+0000","last_became_active":"2024-09-21T14:04:56.587384+0000","last_became_peered":"2024-09-21T14:04:56.587384+0000","last_unstale":"2024-09-21T14:05:13.705317+0000","last_undegraded":"2024-09-21T14:05:13.705317+0000","last_fullsized":"2024-09-21T14:05:13.705317+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:59.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:59:59.450546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:58.054530+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":1611,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:14.783342+0000","last_change":"2024-09-21T13:59:32.218796+0000","last_active":"2024-09-21T14:05:14.783342+0000","last_peered":"2024-09-21T14:05:14.783342+0000","last_clean":"2024-09-21T14:05:14.783342+0000","last_became_active":"2024-09-21T13:59:32.203361+0000","last_became_peered":"2024-09-21T13:59:32.203361+0000","last_unstale":"2024-09-21T14:05:14.783342+0000","last_undegraded":"2024-09-21T14:05:14.783342+0000","last_fullsized":"2024-09-21T14:05:14.783342+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-21T13:56:29.617143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:56:29.617143+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:31.086108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047048000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":350,"num_read_kb":345,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":704,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:14.344869+0000","last_change":"2024-09-21T14:05:14.344869+0000","last_active":"2024-09-21T14:05:14.344869+0000","last_peered":"2024-09-21T14:05:14.344869+0000","last_clean":"2024-09-21T14:05:14.344869+0000","last_became_active":"2024-09-21T14:01:41.200531+0000","last_became_peered":"2024-09-21T14:01:41.200531+0000","last_unstale":"2024-09-21T14:05:14.344869+0000","last_undegraded":"2024-09-21T14:05:14.344869+0000","last_fullsized":"2024-09-21T14:05:14.344869+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:14.344795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:05:14.344795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:33.382674+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":800,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.705529+0000","last_change":"2024-09-21T14:03:04.357026+0000","last_active":"2024-09-21T14:05:13.705529+0000","last_peered":"2024-09-21T14:05:13.705529+0000","last_clean":"2024-09-21T14:05:13.705529+0000","last_became_active":"2024-09-21T14:01:41.184664+0000","last_became_peered":"2024-09-21T14:01:41.184664+0000","last_unstale":"2024-09-21T14:05:13.705529+0000","last_undegraded":"2024-09-21T14:05:13.705529+0000","last_fullsized":"2024-09-21T14:05:13.705529+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:04.356979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:04.356979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"422'10276","reported_seq":10440,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-21T14:05:12.925044+0000","last_change":"2024-09-21T14:04:08.011633+0000","last_active":"2024-09-21T14:05:12.925044+0000","last_peered":"2024-09-21T14:05:12.925044+0000","last_clean":"2024-09-21T14:05:12.925044+0000","last_became_active":"2024-09-21T14:02:56.557263+0000","last_became_peered":"2024-09-21T14:02:56.557263+0000","last_unstale":"2024-09-21T14:05:12.925044+0000","last_undegraded":"2024-09-21T14:05:12.925044+0000","last_fullsized":"2024-09-21T14:05:12.925044+0000","mapping_epoch":381,"log_start":"400'5119","ondisk_log_start":"400'5119","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"406'6147","last_scrub_stamp":"2024-09-21T14:04:08.011556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:04:08.011556+0000","objects_scrubbed":6074,"log_size":5157,"log_dups_size":0,"ondisk_log_size":5157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1868,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369623040,"num_objects":5640,"num_object_clones":0,"num_object_copies":16920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10276,"num_write_kb":509312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"300'32","reported_seq":474,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-21T14:05:12.712198+0000","last_change":"2024-09-21T14:00:47.130089+0000","last_active":"2024-09-21T14:05:12.712198+0000","last_peered":"2024-09-21T14:05:12.712198+0000","last_clean":"2024-09-21T14:05:12.712198+0000","last_became_active":"2024-09-21T13:59:32.204624+0000","last_became_peered":"2024-09-21T13:59:32.204624+0000","last_unstale":"2024-09-21T14:05:12.712198+0000","last_undegraded":"2024-09-21T14:05:12.712198+0000","last_fullsized":"2024-09-21T14:05:12.712198+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'32","last_scrub_stamp":"2024-09-21T14:00:47.129789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:00:47.129789+0000","objects_scrubbed":8,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:20.900768+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":754,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.708060+0000","last_change":"2024-09-21T14:04:42.390182+0000","last_active":"2024-09-21T14:05:13.708060+0000","last_peered":"2024-09-21T14:05:13.708060+0000","last_clean":"2024-09-21T14:05:13.708060+0000","last_became_active":"2024-09-21T14:01:41.190210+0000","last_became_peered":"2024-09-21T14:01:41.190210+0000","last_unstale":"2024-09-21T14:05:13.708060+0000","last_undegraded":"2024-09-21T14:05:13.708060+0000","last_fullsized":"2024-09-21T14:05:13.708060+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:42.390102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:42.390102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:10.628828+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"75'79","reported_seq":814,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.792733+0000","last_change":"2024-09-21T14:04:21.205378+0000","last_active":"2024-09-21T14:05:13.792733+0000","last_peered":"2024-09-21T14:05:13.792733+0000","last_clean":"2024-09-21T14:05:13.792733+0000","last_became_active":"2024-09-21T14:04:21.204537+0000","last_became_peered":"2024-09-21T14:04:21.204537+0000","last_unstale":"2024-09-21T14:05:13.792733+0000","last_undegraded":"2024-09-21T14:05:13.792733+0000","last_fullsized":"2024-09-21T14:05:13.792733+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:05:22.117825+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":19,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53669,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":721,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.792228+0000","last_change":"2024-09-21T14:01:41.182186+0000","last_active":"2024-09-21T14:05:13.792228+0000","last_peered":"2024-09-21T14:05:13.792228+0000","last_clean":"2024-09-21T14:05:13.792228+0000","last_became_active":"2024-09-21T14:01:41.181899+0000","last_became_peered":"2024-09-21T14:01:41.181899+0000","last_unstale":"2024-09-21T14:05:13.792228+0000","last_undegraded":"2024-09-21T14:05:13.792228+0000","last_fullsized":"2024-09-21T14:05:13.792228+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:15.700409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:15.700409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":661,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.834514+0000","last_change":"2024-09-21T14:04:01.760941+0000","last_active":"2024-09-21T14:05:13.834514+0000","last_peered":"2024-09-21T14:05:13.834514+0000","last_clean":"2024-09-21T14:05:13.834514+0000","last_became_active":"2024-09-21T14:03:57.346453+0000","last_became_peered":"2024-09-21T14:03:57.346453+0000","last_unstale":"2024-09-21T14:05:13.834514+0000","last_undegraded":"2024-09-21T14:05:13.834514+0000","last_fullsized":"2024-09-21T14:05:13.834514+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:01.760755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:04:01.760755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"422'10400","reported_seq":19128,"reported_epoch":422,"state":"active+remapped+backfilling","last_fresh":"2024-09-21T14:05:13.669283+0000","last_change":"2024-09-21T14:04:52.362441+0000","last_active":"2024-09-21T14:05:13.669283+0000","last_peered":"2024-09-21T14:05:13.669283+0000","last_clean":"2024-09-21T14:04:20.179742+0000","last_became_active":"2024-09-21T14:04:22.332419+0000","last_became_peered":"2024-09-21T14:04:22.332419+0000","last_unstale":"2024-09-21T14:05:13.669283+0000","last_undegraded":"2024-09-21T14:05:13.669283+0000","last_fullsized":"2024-09-21T14:05:13.669283+0000","mapping_epoch":409,"log_start":"400'5236","ondisk_log_start":"400'5236","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"406'6598","last_scrub_stamp":"2024-09-21T14:04:14.205496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:04:14.205496+0000","objects_scrubbed":6523,"log_size":5164,"log_dups_size":0,"ondisk_log_size":5164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":364773376,"num_objects":5566,"num_object_clones":0,"num_object_copies":16698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1723,"num_objects_unfound":0,"num_objects_dirty":5566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10400,"num_write_kb":510912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4275,"num_bytes_recovered":280166400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,4],"avail_no_missing":["6","1","4"],"object_location_counts":[{"shards":"1,4,6","objects":5566}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"290'19","reported_seq":833,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.788410+0000","last_change":"2024-09-21T14:02:28.553549+0000","last_active":"2024-09-21T14:05:13.788410+0000","last_peered":"2024-09-21T14:05:13.788410+0000","last_clean":"2024-09-21T14:05:13.788410+0000","last_became_active":"2024-09-21T13:58:17.776310+0000","last_became_peered":"2024-09-21T13:58:17.776310+0000","last_unstale":"2024-09-21T14:05:13.788410+0000","last_undegraded":"2024-09-21T14:05:13.788410+0000","last_fullsized":"2024-09-21T14:05:13.788410+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'19","last_scrub_stamp":"2024-09-21T14:02:28.553065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:28.553065+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":803,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.788117+0000","last_change":"2024-09-21T14:03:32.957705+0000","last_active":"2024-09-21T14:05:13.788117+0000","last_peered":"2024-09-21T14:05:13.788117+0000","last_clean":"2024-09-21T14:05:13.788117+0000","last_became_active":"2024-09-21T13:59:32.223247+0000","last_became_peered":"2024-09-21T13:59:32.223247+0000","last_unstale":"2024-09-21T14:05:13.788117+0000","last_undegraded":"2024-09-21T14:05:13.788117+0000","last_fullsized":"2024-09-21T14:05:13.788117+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:32.957649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:03:32.957649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":730,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.834468+0000","last_change":"2024-09-21T14:04:54.483668+0000","last_active":"2024-09-21T14:05:13.834468+0000","last_peered":"2024-09-21T14:05:13.834468+0000","last_clean":"2024-09-21T14:05:13.834468+0000","last_became_active":"2024-09-21T14:01:41.193941+0000","last_became_peered":"2024-09-21T14:01:41.193941+0000","last_unstale":"2024-09-21T14:05:13.834468+0000","last_undegraded":"2024-09-21T14:05:13.834468+0000","last_fullsized":"2024-09-21T14:05:13.834468+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:54.483438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:04:54.483438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:21.139691+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":732,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.708901+0000","last_change":"2024-09-21T14:03:36.302360+0000","last_active":"2024-09-21T14:05:13.708901+0000","last_peered":"2024-09-21T14:05:13.708901+0000","last_clean":"2024-09-21T14:05:13.708901+0000","last_became_active":"2024-09-21T14:01:41.190192+0000","last_became_peered":"2024-09-21T14:01:41.190192+0000","last_unstale":"2024-09-21T14:05:13.708901+0000","last_undegraded":"2024-09-21T14:05:13.708901+0000","last_fullsized":"2024-09-21T14:05:13.708901+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:36.302281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:36.302281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"424'10558","reported_seq":10734,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T14:05:18.017110+0000","last_change":"2024-09-21T14:04:52.445772+0000","last_active":"2024-09-21T14:05:18.017110+0000","last_peered":"2024-09-21T14:05:18.017110+0000","last_clean":"2024-09-21T14:05:18.017110+0000","last_became_active":"2024-09-21T14:02:56.560070+0000","last_became_peered":"2024-09-21T14:02:56.560070+0000","last_unstale":"2024-09-21T14:05:18.017110+0000","last_undegraded":"2024-09-21T14:05:18.017110+0000","last_fullsized":"2024-09-21T14:05:18.017110+0000","mapping_epoch":381,"log_start":"404'5587","ondisk_log_start":"404'5587","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'9024","last_scrub_stamp":"2024-09-21T14:04:52.445618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:04:52.445618+0000","objects_scrubbed":6949,"log_size":4971,"log_dups_size":0,"ondisk_log_size":4971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:01.306398+0000","scrub_duration":2824,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":349569024,"num_objects":5334,"num_object_clones":0,"num_object_copies":16002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10558,"num_write_kb":508544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"75'79","reported_seq":195,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.832896+0000","last_change":"2024-09-21T14:04:51.454570+0000","last_active":"2024-09-21T14:05:13.832896+0000","last_peered":"2024-09-21T14:05:13.832896+0000","last_clean":"2024-09-21T14:05:13.832896+0000","last_became_active":"2024-09-21T14:01:41.188685+0000","last_became_peered":"2024-09-21T14:01:41.188685+0000","last_unstale":"2024-09-21T14:05:13.832896+0000","last_undegraded":"2024-09-21T14:05:13.832896+0000","last_fullsized":"2024-09-21T14:05:13.832896+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:04:51.454483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:04:51.454483+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:17.391460+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"290'31","reported_seq":611,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T14:05:17.790800+0000","last_change":"2024-09-21T14:04:39.741288+0000","last_active":"2024-09-21T14:05:17.790800+0000","last_peered":"2024-09-21T14:05:17.790800+0000","last_clean":"2024-09-21T14:05:17.790800+0000","last_became_active":"2024-09-21T14:04:21.204164+0000","last_became_peered":"2024-09-21T14:04:21.204164+0000","last_unstale":"2024-09-21T14:05:17.790800+0000","last_undegraded":"2024-09-21T14:05:17.790800+0000","last_fullsized":"2024-09-21T14:05:17.790800+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'31","last_scrub_stamp":"2024-09-21T14:04:39.740794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:04:39.740794+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:54.328250+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":343,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-21T14:05:12.715434+0000","last_change":"2024-09-21T14:04:51.139211+0000","last_active":"2024-09-21T14:05:12.715434+0000","last_peered":"2024-09-21T14:05:12.715434+0000","last_clean":"2024-09-21T14:05:12.715434+0000","last_became_active":"2024-09-21T13:58:17.762520+0000","last_became_peered":"2024-09-21T13:58:17.762520+0000","last_unstale":"2024-09-21T14:05:12.715434+0000","last_undegraded":"2024-09-21T14:05:12.715434+0000","last_fullsized":"2024-09-21T14:05:12.715434+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:51.139123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:51.139123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:51.167153+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":750,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.705627+0000","last_change":"2024-09-21T13:58:52.030800+0000","last_active":"2024-09-21T14:05:13.705627+0000","last_peered":"2024-09-21T14:05:13.705627+0000","last_clean":"2024-09-21T14:05:13.705627+0000","last_became_active":"2024-09-21T13:49:02.067472+0000","last_became_peered":"2024-09-21T13:49:02.067472+0000","last_unstale":"2024-09-21T14:05:13.705627+0000","last_undegraded":"2024-09-21T14:05:13.705627+0000","last_fullsized":"2024-09-21T14:05:13.705627+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:52.030718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:52.030718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":634,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T14:05:17.777669+0000","last_change":"2024-09-21T13:57:12.683321+0000","last_active":"2024-09-21T14:05:17.777669+0000","last_peered":"2024-09-21T14:05:17.777669+0000","last_clean":"2024-09-21T14:05:17.777669+0000","last_became_active":"2024-09-21T13:57:12.682979+0000","last_became_peered":"2024-09-21T13:57:12.682979+0000","last_unstale":"2024-09-21T14:05:17.777669+0000","last_undegraded":"2024-09-21T14:05:17.777669+0000","last_fullsized":"2024-09-21T14:05:17.777669+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:18.027944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:18.027944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:24.573702+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2","version":"423'10458","reported_seq":10555,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:14.732120+0000","last_change":"2024-09-21T14:02:56.559657+0000","last_active":"2024-09-21T14:05:14.732120+0000","last_peered":"2024-09-21T14:05:14.732120+0000","last_clean":"2024-09-21T14:05:14.732120+0000","last_became_active":"2024-09-21T14:02:56.559368+0000","last_became_peered":"2024-09-21T14:02:56.559368+0000","last_unstale":"2024-09-21T14:05:14.732120+0000","last_undegraded":"2024-09-21T14:05:14.732120+0000","last_fullsized":"2024-09-21T14:05:14.732120+0000","mapping_epoch":381,"log_start":"404'5742","ondisk_log_start":"404'5742","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":4716,"log_dups_size":0,"ondisk_log_size":4716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":360710144,"num_objects":5504,"num_object_clones":0,"num_object_copies":16512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10458,"num_write_kb":510784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"75'79","reported_seq":821,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.792025+0000","last_change":"2024-09-21T14:02:55.806160+0000","last_active":"2024-09-21T14:05:13.792025+0000","last_peered":"2024-09-21T14:05:13.792025+0000","last_clean":"2024-09-21T14:05:13.792025+0000","last_became_active":"2024-09-21T14:00:41.139315+0000","last_became_peered":"2024-09-21T14:00:41.139315+0000","last_unstale":"2024-09-21T14:05:13.792025+0000","last_undegraded":"2024-09-21T14:05:13.792025+0000","last_fullsized":"2024-09-21T14:05:13.792025+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:02:55.806077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.806077+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"290'20","reported_seq":840,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.788429+0000","last_change":"2024-09-21T13:58:17.789407+0000","last_active":"2024-09-21T14:05:13.788429+0000","last_peered":"2024-09-21T14:05:13.788429+0000","last_clean":"2024-09-21T14:05:13.788429+0000","last_became_active":"2024-09-21T13:58:17.765427+0000","last_became_peered":"2024-09-21T13:58:17.765427+0000","last_unstale":"2024-09-21T14:05:13.788429+0000","last_undegraded":"2024-09-21T14:05:13.788429+0000","last_fullsized":"2024-09-21T14:05:13.788429+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-21T13:58:13.464196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:13.464196+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":696,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.708122+0000","last_change":"2024-09-21T14:04:21.215419+0000","last_active":"2024-09-21T14:05:13.708122+0000","last_peered":"2024-09-21T14:05:13.708122+0000","last_clean":"2024-09-21T14:05:13.708122+0000","last_became_active":"2024-09-21T14:04:21.215102+0000","last_became_peered":"2024-09-21T14:04:21.215102+0000","last_unstale":"2024-09-21T14:05:13.708122+0000","last_undegraded":"2024-09-21T14:05:13.708122+0000","last_fullsized":"2024-09-21T14:05:13.708122+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:58.236645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:02:58.236645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"346'14","reported_seq":843,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T14:05:17.790902+0000","last_change":"2024-09-21T14:01:41.186960+0000","last_active":"2024-09-21T14:05:17.790902+0000","last_peered":"2024-09-21T14:05:17.790902+0000","last_clean":"2024-09-21T14:05:17.790902+0000","last_became_active":"2024-09-21T14:01:41.186564+0000","last_became_peered":"2024-09-21T14:01:41.186564+0000","last_unstale":"2024-09-21T14:05:17.790902+0000","last_undegraded":"2024-09-21T14:05:17.790902+0000","last_fullsized":"2024-09-21T14:05:17.790902+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'12","last_scrub_stamp":"2024-09-21T13:59:38.738120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:38.738120+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"68'1","reported_seq":768,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.788050+0000","last_change":"2024-09-21T13:59:31.392186+0000","last_active":"2024-09-21T14:05:13.788050+0000","last_peered":"2024-09-21T14:05:13.788050+0000","last_clean":"2024-09-21T14:05:13.788050+0000","last_became_active":"2024-09-21T13:52:04.313210+0000","last_became_peered":"2024-09-21T13:52:04.313210+0000","last_unstale":"2024-09-21T14:05:13.788050+0000","last_undegraded":"2024-09-21T14:05:13.788050+0000","last_fullsized":"2024-09-21T14:05:13.788050+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T13:59:31.392105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:31.392105+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.5","version":"424'10796","reported_seq":10978,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T14:05:18.007088+0000","last_change":"2024-09-21T14:04:48.634887+0000","last_active":"2024-09-21T14:05:18.007088+0000","last_peered":"2024-09-21T14:05:18.007088+0000","last_clean":"2024-09-21T14:05:18.007088+0000","last_became_active":"2024-09-21T14:02:56.556786+0000","last_became_peered":"2024-09-21T14:02:56.556786+0000","last_unstale":"2024-09-21T14:05:18.007088+0000","last_undegraded":"2024-09-21T14:05:18.007088+0000","last_fullsized":"2024-09-21T14:05:18.007088+0000","mapping_epoch":381,"log_start":"404'5825","ondisk_log_start":"404'5825","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'8991","last_scrub_stamp":"2024-09-21T14:04:48.631935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:04:48.631935+0000","objects_scrubbed":7324,"log_size":4971,"log_dups_size":0,"ondisk_log_size":4971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:05.417795+0000","scrub_duration":2904,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":356384768,"num_objects":5438,"num_object_clones":0,"num_object_copies":16314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10796,"num_write_kb":519488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"290'15","reported_seq":795,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.705663+0000","last_change":"2024-09-21T13:58:50.498597+0000","last_active":"2024-09-21T14:05:13.705663+0000","last_peered":"2024-09-21T14:05:13.705663+0000","last_clean":"2024-09-21T14:05:13.705663+0000","last_became_active":"2024-09-21T13:49:00.053016+0000","last_became_peered":"2024-09-21T13:49:00.053016+0000","last_unstale":"2024-09-21T14:05:13.705663+0000","last_undegraded":"2024-09-21T14:05:13.705663+0000","last_fullsized":"2024-09-21T14:05:13.705663+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:58:50.498369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:50.498369+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:59.036946+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":734,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.834022+0000","last_change":"2024-09-21T14:04:00.767591+0000","last_active":"2024-09-21T14:05:13.834022+0000","last_peered":"2024-09-21T14:05:13.834022+0000","last_clean":"2024-09-21T14:05:13.834022+0000","last_became_active":"2024-09-21T14:01:41.192443+0000","last_became_peered":"2024-09-21T14:01:41.192443+0000","last_unstale":"2024-09-21T14:05:13.834022+0000","last_undegraded":"2024-09-21T14:05:13.834022+0000","last_fullsized":"2024-09-21T14:05:13.834022+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:00.767519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:00.767519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":714,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.788153+0000","last_change":"2024-09-21T14:02:56.796917+0000","last_active":"2024-09-21T14:05:13.788153+0000","last_peered":"2024-09-21T14:05:13.788153+0000","last_clean":"2024-09-21T14:05:13.788153+0000","last_became_active":"2024-09-21T13:57:12.680978+0000","last_became_peered":"2024-09-21T13:57:12.680978+0000","last_unstale":"2024-09-21T14:05:13.788153+0000","last_undegraded":"2024-09-21T14:05:13.788153+0000","last_fullsized":"2024-09-21T14:05:13.788153+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:56.796861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:02:56.796861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":663,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.791758+0000","last_change":"2024-09-21T14:04:21.210096+0000","last_active":"2024-09-21T14:05:13.791758+0000","last_peered":"2024-09-21T14:05:13.791758+0000","last_clean":"2024-09-21T14:05:13.791758+0000","last_became_active":"2024-09-21T14:04:21.207534+0000","last_became_peered":"2024-09-21T14:04:21.207534+0000","last_unstale":"2024-09-21T14:05:13.791758+0000","last_undegraded":"2024-09-21T14:05:13.791758+0000","last_fullsized":"2024-09-21T14:05:13.791758+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:13.698091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:13.698091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"423'10276","reported_seq":24111,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.815252+0000","last_change":"2024-09-21T14:05:13.815252+0000","last_active":"2024-09-21T14:05:13.815252+0000","last_peered":"2024-09-21T14:05:13.815252+0000","last_clean":"2024-09-21T14:05:13.815252+0000","last_became_active":"2024-09-21T14:05:13.789148+0000","last_became_peered":"2024-09-21T14:05:13.789148+0000","last_unstale":"2024-09-21T14:05:13.815252+0000","last_undegraded":"2024-09-21T14:05:13.815252+0000","last_fullsized":"2024-09-21T14:05:13.815252+0000","mapping_epoch":422,"log_start":"398'4800","ondisk_log_start":"398'4800","created":381,"last_epoch_clean":423,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":5476,"log_dups_size":0,"ondisk_log_size":5476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":360316928,"num_objects":5498,"num_object_clones":0,"num_object_copies":16494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10276,"num_write_kb":504768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6862,"num_bytes_recovered":449708032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"290'35","reported_seq":401,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-21T14:05:12.715547+0000","last_change":"2024-09-21T14:01:02.006619+0000","last_active":"2024-09-21T14:05:12.715547+0000","last_peered":"2024-09-21T14:05:12.715547+0000","last_clean":"2024-09-21T14:05:12.715547+0000","last_became_active":"2024-09-21T13:59:32.204184+0000","last_became_peered":"2024-09-21T13:59:32.204184+0000","last_unstale":"2024-09-21T14:05:12.715547+0000","last_undegraded":"2024-09-21T14:05:12.715547+0000","last_fullsized":"2024-09-21T14:05:12.715547+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'35","last_scrub_stamp":"2024-09-21T14:01:02.006504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:02.006504+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:29.829621+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":756,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.793856+0000","last_change":"2024-09-21T14:03:31.882264+0000","last_active":"2024-09-21T14:05:13.793856+0000","last_peered":"2024-09-21T14:05:13.793856+0000","last_clean":"2024-09-21T14:05:13.793856+0000","last_became_active":"2024-09-21T14:01:41.184567+0000","last_became_peered":"2024-09-21T14:01:41.184567+0000","last_unstale":"2024-09-21T14:05:13.793856+0000","last_undegraded":"2024-09-21T14:05:13.793856+0000","last_fullsized":"2024-09-21T14:05:13.793856+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:31.882215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:03:31.882215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":736,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T14:05:17.779613+0000","last_change":"2024-09-21T14:04:48.685009+0000","last_active":"2024-09-21T14:05:17.779613+0000","last_peered":"2024-09-21T14:05:17.779613+0000","last_clean":"2024-09-21T14:05:17.779613+0000","last_became_active":"2024-09-21T14:01:41.181911+0000","last_became_peered":"2024-09-21T14:01:41.181911+0000","last_unstale":"2024-09-21T14:05:17.779613+0000","last_undegraded":"2024-09-21T14:05:17.779613+0000","last_fullsized":"2024-09-21T14:05:17.779613+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:48.684941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:04:48.684941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:08.622354+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":683,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.792727+0000","last_change":"2024-09-21T14:03:33.893158+0000","last_active":"2024-09-21T14:05:13.792727+0000","last_peered":"2024-09-21T14:05:13.792727+0000","last_clean":"2024-09-21T14:05:13.792727+0000","last_became_active":"2024-09-21T13:59:32.219332+0000","last_became_peered":"2024-09-21T13:59:32.219332+0000","last_unstale":"2024-09-21T14:05:13.792727+0000","last_undegraded":"2024-09-21T14:05:13.792727+0000","last_fullsized":"2024-09-21T14:05:13.792727+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:33.893072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:33.893072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"422'10441","reported_seq":10624,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-21T14:05:12.817024+0000","last_change":"2024-09-21T14:04:40.683339+0000","last_active":"2024-09-21T14:05:12.817024+0000","last_peered":"2024-09-21T14:05:12.817024+0000","last_clean":"2024-09-21T14:05:12.817024+0000","last_became_active":"2024-09-21T14:02:56.559331+0000","last_became_peered":"2024-09-21T14:02:56.559331+0000","last_unstale":"2024-09-21T14:05:12.817024+0000","last_undegraded":"2024-09-21T14:05:12.817024+0000","last_fullsized":"2024-09-21T14:05:12.817024+0000","mapping_epoch":381,"log_start":"400'5290","ondisk_log_start":"400'5290","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'8371","last_scrub_stamp":"2024-09-21T14:04:40.683117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:04:40.683117+0000","objects_scrubbed":7905,"log_size":5151,"log_dups_size":0,"ondisk_log_size":5151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:47.784341+0000","scrub_duration":2828,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376373248,"num_objects":5743,"num_object_clones":0,"num_object_copies":17229,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10441,"num_write_kb":517888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"290'18","reported_seq":812,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.706986+0000","last_change":"2024-09-21T13:59:28.340494+0000","last_active":"2024-09-21T14:05:13.706986+0000","last_peered":"2024-09-21T14:05:13.706986+0000","last_clean":"2024-09-21T14:05:13.706986+0000","last_became_active":"2024-09-21T13:57:12.680627+0000","last_became_peered":"2024-09-21T13:57:12.680627+0000","last_unstale":"2024-09-21T14:05:13.706986+0000","last_undegraded":"2024-09-21T14:05:13.706986+0000","last_fullsized":"2024-09-21T14:05:13.706986+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'18","last_scrub_stamp":"2024-09-21T13:59:28.340415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:28.340415+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":710,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.792775+0000","last_change":"2024-09-21T14:04:21.206034+0000","last_active":"2024-09-21T14:05:13.792775+0000","last_peered":"2024-09-21T14:05:13.792775+0000","last_clean":"2024-09-21T14:05:13.792775+0000","last_became_active":"2024-09-21T14:04:21.205699+0000","last_became_peered":"2024-09-21T14:04:21.205699+0000","last_unstale":"2024-09-21T14:05:13.792775+0000","last_undegraded":"2024-09-21T14:05:13.792775+0000","last_fullsized":"2024-09-21T14:05:13.792775+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:21.582325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:21.582325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:25.153959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":691,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T14:05:17.772255+0000","last_change":"2024-09-21T14:02:27.791041+0000","last_active":"2024-09-21T14:05:17.772255+0000","last_peered":"2024-09-21T14:05:17.772255+0000","last_clean":"2024-09-21T14:05:17.772255+0000","last_became_active":"2024-09-21T13:52:04.322147+0000","last_became_peered":"2024-09-21T13:52:04.322147+0000","last_unstale":"2024-09-21T14:05:17.772255+0000","last_undegraded":"2024-09-21T14:05:17.772255+0000","last_fullsized":"2024-09-21T14:05:17.772255+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:27.790994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:02:27.790994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":697,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.708391+0000","last_change":"2024-09-21T14:02:36.371955+0000","last_active":"2024-09-21T14:05:13.708391+0000","last_peered":"2024-09-21T14:05:13.708391+0000","last_clean":"2024-09-21T14:05:13.708391+0000","last_became_active":"2024-09-21T13:59:32.203767+0000","last_became_peered":"2024-09-21T13:59:32.203767+0000","last_unstale":"2024-09-21T14:05:13.708391+0000","last_undegraded":"2024-09-21T14:05:13.708391+0000","last_fullsized":"2024-09-21T14:05:13.708391+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:36.371747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:02:36.371747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"423'10486","reported_seq":10662,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:15.939422+0000","last_change":"2024-09-21T14:04:52.260111+0000","last_active":"2024-09-21T14:05:15.939422+0000","last_peered":"2024-09-21T14:05:15.939422+0000","last_clean":"2024-09-21T14:05:15.939422+0000","last_became_active":"2024-09-21T14:02:56.563788+0000","last_became_peered":"2024-09-21T14:02:56.563788+0000","last_unstale":"2024-09-21T14:05:15.939422+0000","last_undegraded":"2024-09-21T14:05:15.939422+0000","last_fullsized":"2024-09-21T14:05:15.939422+0000","mapping_epoch":381,"log_start":"403'5637","ondisk_log_start":"403'5637","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'9041","last_scrub_stamp":"2024-09-21T14:04:52.260025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:04:52.260025+0000","objects_scrubbed":7053,"log_size":4849,"log_dups_size":0,"ondisk_log_size":4849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:57.305832+0000","scrub_duration":1833,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":364118016,"num_objects":5556,"num_object_clones":0,"num_object_copies":16668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10486,"num_write_kb":513344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"290'27","reported_seq":810,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.833030+0000","last_change":"2024-09-21T14:03:31.360992+0000","last_active":"2024-09-21T14:05:13.833030+0000","last_peered":"2024-09-21T14:05:13.833030+0000","last_clean":"2024-09-21T14:05:13.833030+0000","last_became_active":"2024-09-21T14:01:41.192481+0000","last_became_peered":"2024-09-21T14:01:41.192481+0000","last_unstale":"2024-09-21T14:05:13.833030+0000","last_undegraded":"2024-09-21T14:05:13.833030+0000","last_fullsized":"2024-09-21T14:05:13.833030+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'27","last_scrub_stamp":"2024-09-21T14:03:31.360906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:03:31.360906+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":696,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.792466+0000","last_change":"2024-09-21T14:04:21.218814+0000","last_active":"2024-09-21T14:05:13.792466+0000","last_peered":"2024-09-21T14:05:13.792466+0000","last_clean":"2024-09-21T14:05:13.792466+0000","last_became_active":"2024-09-21T14:04:21.201644+0000","last_became_peered":"2024-09-21T14:04:21.201644+0000","last_unstale":"2024-09-21T14:05:13.792466+0000","last_undegraded":"2024-09-21T14:05:13.792466+0000","last_fullsized":"2024-09-21T14:05:13.792466+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T14:03:04.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:03:04.953658+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"300'14","reported_seq":918,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.791734+0000","last_change":"2024-09-21T13:59:32.220898+0000","last_active":"2024-09-21T14:05:13.791734+0000","last_peered":"2024-09-21T14:05:13.791734+0000","last_clean":"2024-09-21T14:05:13.791734+0000","last_became_active":"2024-09-21T13:59:32.206956+0000","last_became_peered":"2024-09-21T13:59:32.206956+0000","last_unstale":"2024-09-21T14:05:13.791734+0000","last_undegraded":"2024-09-21T14:05:13.791734+0000","last_fullsized":"2024-09-21T14:05:13.791734+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'11","last_scrub_stamp":"2024-09-21T13:56:31.616003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:56:31.616003+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:27.198182+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":758,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.833535+0000","last_change":"2024-09-21T14:04:42.433805+0000","last_active":"2024-09-21T14:05:13.833535+0000","last_peered":"2024-09-21T14:05:13.833535+0000","last_clean":"2024-09-21T14:05:13.833535+0000","last_became_active":"2024-09-21T14:01:41.246106+0000","last_became_peered":"2024-09-21T14:01:41.246106+0000","last_unstale":"2024-09-21T14:05:13.833535+0000","last_undegraded":"2024-09-21T14:05:13.833535+0000","last_fullsized":"2024-09-21T14:05:13.833535+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:42.433729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:04:42.433729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:04.041353+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":344,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-21T14:05:12.709467+0000","last_change":"2024-09-21T14:04:21.198859+0000","last_active":"2024-09-21T14:05:12.709467+0000","last_peered":"2024-09-21T14:05:12.709467+0000","last_clean":"2024-09-21T14:05:12.709467+0000","last_became_active":"2024-09-21T14:04:21.197402+0000","last_became_peered":"2024-09-21T14:04:21.197402+0000","last_unstale":"2024-09-21T14:05:12.709467+0000","last_undegraded":"2024-09-21T14:05:12.709467+0000","last_fullsized":"2024-09-21T14:05:12.709467+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:50.733007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:01:50.733007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:31.764576+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"290'12","reported_seq":724,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.708880+0000","last_change":"2024-09-21T13:59:32.247422+0000","last_active":"2024-09-21T14:05:13.708880+0000","last_peered":"2024-09-21T14:05:13.708880+0000","last_clean":"2024-09-21T14:05:13.708880+0000","last_became_active":"2024-09-21T13:59:32.211306+0000","last_became_peered":"2024-09-21T13:59:32.211306+0000","last_unstale":"2024-09-21T14:05:13.708880+0000","last_undegraded":"2024-09-21T14:05:13.708880+0000","last_fullsized":"2024-09-21T14:05:13.708880+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'6","last_scrub_stamp":"2024-09-21T13:55:12.066873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:55:12.066873+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":682,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.834276+0000","last_change":"2024-09-21T14:01:41.199742+0000","last_active":"2024-09-21T14:05:13.834276+0000","last_peered":"2024-09-21T14:05:13.834276+0000","last_clean":"2024-09-21T14:05:13.834276+0000","last_became_active":"2024-09-21T14:01:41.199321+0000","last_became_peered":"2024-09-21T14:01:41.199321+0000","last_unstale":"2024-09-21T14:05:13.834276+0000","last_undegraded":"2024-09-21T14:05:13.834276+0000","last_fullsized":"2024-09-21T14:05:13.834276+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:37.106238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:37.106238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:17.707656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":729,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.792811+0000","last_change":"2024-09-21T14:03:01.872986+0000","last_active":"2024-09-21T14:05:13.792811+0000","last_peered":"2024-09-21T14:05:13.792811+0000","last_clean":"2024-09-21T14:05:13.792811+0000","last_became_active":"2024-09-21T13:59:32.219525+0000","last_became_peered":"2024-09-21T13:59:32.219525+0000","last_unstale":"2024-09-21T14:05:13.792811+0000","last_undegraded":"2024-09-21T14:05:13.792811+0000","last_fullsized":"2024-09-21T14:05:13.792811+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:01.872928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:01.872928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"290'15","reported_seq":704,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.708570+0000","last_change":"2024-09-21T14:04:21.228883+0000","last_active":"2024-09-21T14:05:13.708570+0000","last_peered":"2024-09-21T14:05:13.708570+0000","last_clean":"2024-09-21T14:05:13.708570+0000","last_became_active":"2024-09-21T14:04:21.208398+0000","last_became_peered":"2024-09-21T14:04:21.208398+0000","last_unstale":"2024-09-21T14:05:13.708570+0000","last_undegraded":"2024-09-21T14:05:13.708570+0000","last_fullsized":"2024-09-21T14:05:13.708570+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:55:04.990842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:55:04.990842+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:29.202675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"61'1","reported_seq":747,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.791938+0000","last_change":"2024-09-21T14:04:04.012140+0000","last_active":"2024-09-21T14:05:13.791938+0000","last_peered":"2024-09-21T14:05:13.791938+0000","last_clean":"2024-09-21T14:05:13.791938+0000","last_became_active":"2024-09-21T13:59:32.219515+0000","last_became_peered":"2024-09-21T13:59:32.219515+0000","last_unstale":"2024-09-21T14:05:13.791938+0000","last_undegraded":"2024-09-21T14:05:13.791938+0000","last_fullsized":"2024-09-21T14:05:13.791938+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T14:04:04.012006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:04.012006+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:20.215213+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":680,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.708662+0000","last_change":"2024-09-21T14:03:33.309440+0000","last_active":"2024-09-21T14:05:13.708662+0000","last_peered":"2024-09-21T14:05:13.708662+0000","last_clean":"2024-09-21T14:05:13.708662+0000","last_became_active":"2024-09-21T13:59:32.210586+0000","last_became_peered":"2024-09-21T13:59:32.210586+0000","last_unstale":"2024-09-21T14:05:13.708662+0000","last_undegraded":"2024-09-21T14:05:13.708662+0000","last_fullsized":"2024-09-21T14:05:13.708662+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:33.309248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:33.309248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":732,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T14:05:17.772369+0000","last_change":"2024-09-21T14:02:29.731635+0000","last_active":"2024-09-21T14:05:17.772369+0000","last_peered":"2024-09-21T14:05:17.772369+0000","last_clean":"2024-09-21T14:05:17.772369+0000","last_became_active":"2024-09-21T13:59:32.205803+0000","last_became_peered":"2024-09-21T13:59:32.205803+0000","last_unstale":"2024-09-21T14:05:17.772369+0000","last_undegraded":"2024-09-21T14:05:17.772369+0000","last_fullsized":"2024-09-21T14:05:17.772369+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:29.731565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:02:29.731565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"290'15","reported_seq":727,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.834060+0000","last_change":"2024-09-21T14:01:41.189676+0000","last_active":"2024-09-21T14:05:13.834060+0000","last_peered":"2024-09-21T14:05:13.834060+0000","last_clean":"2024-09-21T14:05:13.834060+0000","last_became_active":"2024-09-21T14:01:41.187904+0000","last_became_peered":"2024-09-21T14:01:41.187904+0000","last_unstale":"2024-09-21T14:05:13.834060+0000","last_undegraded":"2024-09-21T14:05:13.834060+0000","last_fullsized":"2024-09-21T14:05:13.834060+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:57:24.124604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:57:24.124604+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:30.424236+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":371,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.705977+0000","last_change":"2024-09-21T14:04:56.588256+0000","last_active":"2024-09-21T14:05:13.705977+0000","last_peered":"2024-09-21T14:05:13.705977+0000","last_clean":"2024-09-21T14:05:13.705977+0000","last_became_active":"2024-09-21T14:04:56.587953+0000","last_became_peered":"2024-09-21T14:04:56.587953+0000","last_unstale":"2024-09-21T14:05:13.705977+0000","last_undegraded":"2024-09-21T14:05:13.705977+0000","last_fullsized":"2024-09-21T14:05:13.705977+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:37.078458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:37.078458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:52.347012+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"300'14","reported_seq":851,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.708961+0000","last_change":"2024-09-21T13:59:32.208253+0000","last_active":"2024-09-21T14:05:13.708961+0000","last_peered":"2024-09-21T14:05:13.708961+0000","last_clean":"2024-09-21T14:05:13.708961+0000","last_became_active":"2024-09-21T13:59:32.206926+0000","last_became_peered":"2024-09-21T13:59:32.206926+0000","last_unstale":"2024-09-21T14:05:13.708961+0000","last_undegraded":"2024-09-21T14:05:13.708961+0000","last_fullsized":"2024-09-21T14:05:13.708961+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'10","last_scrub_stamp":"2024-09-21T13:54:20.946782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:20.946782+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:20.221431+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":774,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.707078+0000","last_change":"2024-09-21T14:01:41.200564+0000","last_active":"2024-09-21T14:05:13.707078+0000","last_peered":"2024-09-21T14:05:13.707078+0000","last_clean":"2024-09-21T14:05:13.707078+0000","last_became_active":"2024-09-21T14:01:41.200113+0000","last_became_peered":"2024-09-21T14:01:41.200113+0000","last_unstale":"2024-09-21T14:05:13.707078+0000","last_undegraded":"2024-09-21T14:05:13.707078+0000","last_fullsized":"2024-09-21T14:05:13.707078+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:54.805619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:54.805619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"290'20","reported_seq":731,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.708851+0000","last_change":"2024-09-21T14:04:04.192280+0000","last_active":"2024-09-21T14:05:13.708851+0000","last_peered":"2024-09-21T14:05:13.708851+0000","last_clean":"2024-09-21T14:05:13.708851+0000","last_became_active":"2024-09-21T13:59:32.208332+0000","last_became_peered":"2024-09-21T13:59:32.208332+0000","last_unstale":"2024-09-21T14:05:13.708851+0000","last_undegraded":"2024-09-21T14:05:13.708851+0000","last_fullsized":"2024-09-21T14:05:13.708851+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'20","last_scrub_stamp":"2024-09-21T14:04:04.192193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:04:04.192193+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":715,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.792438+0000","last_change":"2024-09-21T13:59:32.212737+0000","last_active":"2024-09-21T14:05:13.792438+0000","last_peered":"2024-09-21T14:05:13.792438+0000","last_clean":"2024-09-21T14:05:13.792438+0000","last_became_active":"2024-09-21T13:59:32.209804+0000","last_became_peered":"2024-09-21T13:59:32.209804+0000","last_unstale":"2024-09-21T14:05:13.792438+0000","last_undegraded":"2024-09-21T14:05:13.792438+0000","last_fullsized":"2024-09-21T14:05:13.792438+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:25.571256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:25.571256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:29.134483+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"346'10","reported_seq":859,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.833977+0000","last_change":"2024-09-21T14:03:34.365517+0000","last_active":"2024-09-21T14:05:13.833977+0000","last_peered":"2024-09-21T14:05:13.833977+0000","last_clean":"2024-09-21T14:05:13.833977+0000","last_became_active":"2024-09-21T14:01:41.206303+0000","last_became_peered":"2024-09-21T14:01:41.206303+0000","last_unstale":"2024-09-21T14:05:13.833977+0000","last_undegraded":"2024-09-21T14:05:13.833977+0000","last_fullsized":"2024-09-21T14:05:13.833977+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'10","last_scrub_stamp":"2024-09-21T14:03:34.365428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:34.365428+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":627,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.792653+0000","last_change":"2024-09-21T14:04:21.202582+0000","last_active":"2024-09-21T14:05:13.792653+0000","last_peered":"2024-09-21T14:05:13.792653+0000","last_clean":"2024-09-21T14:05:13.792653+0000","last_became_active":"2024-09-21T14:04:21.201277+0000","last_became_peered":"2024-09-21T14:04:21.201277+0000","last_unstale":"2024-09-21T14:05:13.792653+0000","last_undegraded":"2024-09-21T14:05:13.792653+0000","last_fullsized":"2024-09-21T14:05:13.792653+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:27.961544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:27.961544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"290'19","reported_seq":845,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.706861+0000","last_change":"2024-09-21T14:04:05.139089+0000","last_active":"2024-09-21T14:05:13.706861+0000","last_peered":"2024-09-21T14:05:13.706861+0000","last_clean":"2024-09-21T14:05:13.706861+0000","last_became_active":"2024-09-21T13:58:17.771497+0000","last_became_peered":"2024-09-21T13:58:17.771497+0000","last_unstale":"2024-09-21T14:05:13.706861+0000","last_undegraded":"2024-09-21T14:05:13.706861+0000","last_fullsized":"2024-09-21T14:05:13.706861+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'19","last_scrub_stamp":"2024-09-21T14:04:05.138989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:04:05.138989+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:14.903630+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":357,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.792629+0000","last_change":"2024-09-21T14:02:57.807136+0000","last_active":"2024-09-21T14:05:13.792629+0000","last_peered":"2024-09-21T14:05:13.792629+0000","last_clean":"2024-09-21T14:05:13.792629+0000","last_became_active":"2024-09-21T13:59:32.201568+0000","last_became_peered":"2024-09-21T13:59:32.201568+0000","last_unstale":"2024-09-21T14:05:13.792629+0000","last_undegraded":"2024-09-21T14:05:13.792629+0000","last_fullsized":"2024-09-21T14:05:13.792629+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:57.807059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:02:57.807059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":787,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.705229+0000","last_change":"2024-09-21T14:01:41.185588+0000","last_active":"2024-09-21T14:05:13.705229+0000","last_peered":"2024-09-21T14:05:13.705229+0000","last_clean":"2024-09-21T14:05:13.705229+0000","last_became_active":"2024-09-21T14:01:41.185293+0000","last_became_peered":"2024-09-21T14:01:41.185293+0000","last_unstale":"2024-09-21T14:05:13.705229+0000","last_undegraded":"2024-09-21T14:05:13.705229+0000","last_fullsized":"2024-09-21T14:05:13.705229+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:57:35.409237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:57:35.409237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":695,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.707799+0000","last_change":"2024-09-21T14:04:21.209188+0000","last_active":"2024-09-21T14:05:13.707799+0000","last_peered":"2024-09-21T14:05:13.707799+0000","last_clean":"2024-09-21T14:05:13.707799+0000","last_became_active":"2024-09-21T14:04:21.208329+0000","last_became_peered":"2024-09-21T14:04:21.208329+0000","last_unstale":"2024-09-21T14:05:13.707799+0000","last_undegraded":"2024-09-21T14:05:13.707799+0000","last_fullsized":"2024-09-21T14:05:13.707799+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:57.225602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:02:57.225602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"290'15","reported_seq":847,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.705428+0000","last_change":"2024-09-21T14:04:12.139406+0000","last_active":"2024-09-21T14:05:13.705428+0000","last_peered":"2024-09-21T14:05:13.705428+0000","last_clean":"2024-09-21T14:05:13.705428+0000","last_became_active":"2024-09-21T13:58:17.764120+0000","last_became_peered":"2024-09-21T13:58:17.764120+0000","last_unstale":"2024-09-21T14:05:13.705428+0000","last_undegraded":"2024-09-21T14:05:13.705428+0000","last_fullsized":"2024-09-21T14:05:13.705428+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'15","last_scrub_stamp":"2024-09-21T14:04:12.139190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:04:12.139190+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:40.181606+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":727,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.792617+0000","last_change":"2024-09-21T14:04:07.038058+0000","last_active":"2024-09-21T14:05:13.792617+0000","last_peered":"2024-09-21T14:05:13.792617+0000","last_clean":"2024-09-21T14:05:13.792617+0000","last_became_active":"2024-09-21T13:59:32.221854+0000","last_became_peered":"2024-09-21T13:59:32.221854+0000","last_unstale":"2024-09-21T14:05:13.792617+0000","last_undegraded":"2024-09-21T14:05:13.792617+0000","last_fullsized":"2024-09-21T14:05:13.792617+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:07.037989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:07.037989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:34.939994+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":663,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.708096+0000","last_change":"2024-09-21T14:03:32.262147+0000","last_active":"2024-09-21T14:05:13.708096+0000","last_peered":"2024-09-21T14:05:13.708096+0000","last_clean":"2024-09-21T14:05:13.708096+0000","last_became_active":"2024-09-21T13:57:12.703441+0000","last_became_peered":"2024-09-21T13:57:12.703441+0000","last_unstale":"2024-09-21T14:05:13.708096+0000","last_undegraded":"2024-09-21T14:05:13.708096+0000","last_fullsized":"2024-09-21T14:05:13.708096+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:32.262067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:32.262067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":435,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.705401+0000","last_change":"2024-09-21T14:04:56.588351+0000","last_active":"2024-09-21T14:05:13.705401+0000","last_peered":"2024-09-21T14:05:13.705401+0000","last_clean":"2024-09-21T14:05:13.705401+0000","last_became_active":"2024-09-21T14:04:56.587503+0000","last_became_peered":"2024-09-21T14:04:56.587503+0000","last_unstale":"2024-09-21T14:05:13.705401+0000","last_undegraded":"2024-09-21T14:05:13.705401+0000","last_fullsized":"2024-09-21T14:05:13.705401+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:29.345427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:29.345427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:05.186916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"290'19","reported_seq":799,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.707923+0000","last_change":"2024-09-21T13:59:32.277742+0000","last_active":"2024-09-21T14:05:13.707923+0000","last_peered":"2024-09-21T14:05:13.707923+0000","last_clean":"2024-09-21T14:05:13.707923+0000","last_became_active":"2024-09-21T13:59:32.231567+0000","last_became_peered":"2024-09-21T13:59:32.231567+0000","last_unstale":"2024-09-21T14:05:13.707923+0000","last_undegraded":"2024-09-21T14:05:13.707923+0000","last_fullsized":"2024-09-21T14:05:13.707923+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:54:22.978267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:22.978267+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:24.229733+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":695,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.834389+0000","last_change":"2024-09-21T14:05:13.355232+0000","last_active":"2024-09-21T14:05:13.834389+0000","last_peered":"2024-09-21T14:05:13.834389+0000","last_clean":"2024-09-21T14:05:13.834389+0000","last_became_active":"2024-09-21T14:01:41.190760+0000","last_became_peered":"2024-09-21T14:01:41.190760+0000","last_unstale":"2024-09-21T14:05:13.834389+0000","last_undegraded":"2024-09-21T14:05:13.834389+0000","last_fullsized":"2024-09-21T14:05:13.834389+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:13.355183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:05:13.355183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:37.299500+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":733,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.793717+0000","last_change":"2024-09-21T14:04:41.091577+0000","last_active":"2024-09-21T14:05:13.793717+0000","last_peered":"2024-09-21T14:05:13.793717+0000","last_clean":"2024-09-21T14:05:13.793717+0000","last_became_active":"2024-09-21T13:59:32.214004+0000","last_became_peered":"2024-09-21T13:59:32.214004+0000","last_unstale":"2024-09-21T14:05:13.793717+0000","last_undegraded":"2024-09-21T14:05:13.793717+0000","last_fullsized":"2024-09-21T14:05:13.793717+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:41.091525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:04:41.091525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:06.917055+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":728,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.708207+0000","last_change":"2024-09-21T14:04:21.214685+0000","last_active":"2024-09-21T14:05:13.708207+0000","last_peered":"2024-09-21T14:05:13.708207+0000","last_clean":"2024-09-21T14:05:13.708207+0000","last_became_active":"2024-09-21T14:04:21.213307+0000","last_became_peered":"2024-09-21T14:04:21.213307+0000","last_unstale":"2024-09-21T14:05:13.708207+0000","last_undegraded":"2024-09-21T14:05:13.708207+0000","last_fullsized":"2024-09-21T14:05:13.708207+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:28.124296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:28.124296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"290'8","reported_seq":745,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.707916+0000","last_change":"2024-09-21T13:59:26.034457+0000","last_active":"2024-09-21T14:05:13.707916+0000","last_peered":"2024-09-21T14:05:13.707916+0000","last_clean":"2024-09-21T14:05:13.707916+0000","last_became_active":"2024-09-21T13:58:17.775079+0000","last_became_peered":"2024-09-21T13:58:17.775079+0000","last_unstale":"2024-09-21T14:05:13.707916+0000","last_undegraded":"2024-09-21T14:05:13.707916+0000","last_fullsized":"2024-09-21T14:05:13.707916+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'8","last_scrub_stamp":"2024-09-21T13:59:26.034372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:26.034372+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":506,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.705730+0000","last_change":"2024-09-21T14:04:56.587839+0000","last_active":"2024-09-21T14:05:13.705730+0000","last_peered":"2024-09-21T14:05:13.705730+0000","last_clean":"2024-09-21T14:05:13.705730+0000","last_became_active":"2024-09-21T14:04:56.587555+0000","last_became_peered":"2024-09-21T14:04:56.587555+0000","last_unstale":"2024-09-21T14:05:13.705730+0000","last_undegraded":"2024-09-21T14:05:13.705730+0000","last_fullsized":"2024-09-21T14:05:13.705730+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:34.387367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:34.387367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:38.961058+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":360,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-21T14:05:12.709149+0000","last_change":"2024-09-21T14:04:41.932290+0000","last_active":"2024-09-21T14:05:12.709149+0000","last_peered":"2024-09-21T14:05:12.709149+0000","last_clean":"2024-09-21T14:05:12.709149+0000","last_became_active":"2024-09-21T14:04:21.204954+0000","last_became_peered":"2024-09-21T14:04:21.204954+0000","last_unstale":"2024-09-21T14:05:12.709149+0000","last_undegraded":"2024-09-21T14:05:12.709149+0000","last_fullsized":"2024-09-21T14:05:12.709149+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:41.932233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:04:41.932233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:03.486134+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":666,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.792048+0000","last_change":"2024-09-21T14:04:15.135536+0000","last_active":"2024-09-21T14:05:13.792048+0000","last_peered":"2024-09-21T14:05:13.792048+0000","last_clean":"2024-09-21T14:05:13.792048+0000","last_became_active":"2024-09-21T13:59:32.210134+0000","last_became_peered":"2024-09-21T13:59:32.210134+0000","last_unstale":"2024-09-21T14:05:13.792048+0000","last_undegraded":"2024-09-21T14:05:13.792048+0000","last_fullsized":"2024-09-21T14:05:13.792048+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:15.135463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:04:15.135463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:37.791722+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":772,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.705391+0000","last_change":"2024-09-21T14:01:41.187792+0000","last_active":"2024-09-21T14:05:13.705391+0000","last_peered":"2024-09-21T14:05:13.705391+0000","last_clean":"2024-09-21T14:05:13.705391+0000","last_became_active":"2024-09-21T14:01:41.187438+0000","last_became_peered":"2024-09-21T14:01:41.187438+0000","last_unstale":"2024-09-21T14:05:13.705391+0000","last_undegraded":"2024-09-21T14:05:13.705391+0000","last_fullsized":"2024-09-21T14:05:13.705391+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:17.706325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:17.706325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":716,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.708057+0000","last_change":"2024-09-21T14:04:33.252571+0000","last_active":"2024-09-21T14:05:13.708057+0000","last_peered":"2024-09-21T14:05:13.708057+0000","last_clean":"2024-09-21T14:05:13.708057+0000","last_became_active":"2024-09-21T14:04:21.208121+0000","last_became_peered":"2024-09-21T14:04:21.208121+0000","last_unstale":"2024-09-21T14:05:13.708057+0000","last_undegraded":"2024-09-21T14:05:13.708057+0000","last_fullsized":"2024-09-21T14:05:13.708057+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:33.252481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:33.252481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:48.348987+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"290'19","reported_seq":378,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-21T14:05:12.717346+0000","last_change":"2024-09-21T14:03:31.762681+0000","last_active":"2024-09-21T14:05:12.717346+0000","last_peered":"2024-09-21T14:05:12.717346+0000","last_clean":"2024-09-21T14:05:12.717346+0000","last_became_active":"2024-09-21T14:01:41.194606+0000","last_became_peered":"2024-09-21T14:01:41.194606+0000","last_unstale":"2024-09-21T14:05:12.717346+0000","last_undegraded":"2024-09-21T14:05:12.717346+0000","last_fullsized":"2024-09-21T14:05:12.717346+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'19","last_scrub_stamp":"2024-09-21T14:03:31.762605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:03:31.762605+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":390,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-21T14:05:13.706057+0000","last_change":"2024-09-21T14:04:56.591137+0000","last_active":"2024-09-21T14:05:13.706057+0000","last_peered":"2024-09-21T14:05:13.706057+0000","last_clean":"2024-09-21T14:05:13.706057+0000","last_became_active":"2024-09-21T14:04:56.590829+0000","last_became_peered":"2024-09-21T14:04:56.590829+0000","last_unstale":"2024-09-21T14:05:13.706057+0000","last_undegraded":"2024-09-21T14:05:13.706057+0000","last_fullsized":"2024-09-21T14:05:13.706057+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:51.121361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:04:51.121361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:59.958121+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":5804654616,"num_objects":88573,"num_object_clones":0,"num_object_copies":265719,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7641,"num_objects_unfound":0,"num_objects_dirty":88573,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166577,"num_write_kb":8164737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16854,"num_bytes_recovered":1104543744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18337378304,"data_stored":18337366088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82355,"ondisk_log_size":82355,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102,"ondisk_log_size":102,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1107,"num_read_kb":864,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":197,"num_bytes_recovered":2794,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":687,"ondisk_log_size":687,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":350,"num_read_kb":345,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":751377,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":32,"num_read_kb":59,"num_write":32,"num_write_kb":444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":298,"seq":1279900254278,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2042988,"kb_used_data":1654428,"kb_used_omap":4,"kb_used_meta":388539,"kb_avail":91702164,"statfs":{"total":95995035648,"available":93903015936,"internally_reserved":0,"allocated":1694134272,"data_stored":1693800385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":397864616},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463117,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2928780,"kb_used_data":2510264,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":90816372,"statfs":{"total":95995035648,"available":92995964928,"internally_reserved":0,"allocated":2570510336,"data_stored":2570203758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":428538310},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,0,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":345,"seq":1481763717164,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1943952,"kb_used_data":1767044,"kb_used_omap":4,"kb_used_meta":176827,"kb_avail":91801200,"statfs":{"total":95995035648,"available":94004428800,"internally_reserved":0,"allocated":1809453056,"data_stored":1809143854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":181071523},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920986,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3382292,"kb_used_data":2859648,"kb_used_omap":1,"kb_used_meta":522622,"kb_avail":90362860,"statfs":{"total":95995035648,"available":92531568640,"internally_reserved":0,"allocated":2928279552,"data_stored":2927964868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":535165381},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":240,"seq":1030792151138,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2354292,"kb_used_data":1941600,"kb_used_omap":4,"kb_used_meta":412667,"kb_avail":91390860,"statfs":{"total":95995035648,"available":93584240640,"internally_reserved":0,"allocated":1988198400,"data_stored":1987860948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4443,"internal_metadata":422571685},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":130,"seq":558345748640,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2929880,"kb_used_data":2243784,"kb_used_omap":4,"kb_used_meta":686075,"kb_avail":90815272,"statfs":{"total":95995035648,"available":92994838528,"internally_reserved":0,"allocated":2297634816,"data_stored":2297321540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4441,"internal_metadata":702541479},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575084,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2104860,"kb_used_data":1415556,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":91640292,"statfs":{"total":95995035648,"available":93839659008,"internally_reserved":0,"allocated":1449529344,"data_stored":1449154600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738610,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4244868,"kb_used_data":3573612,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":89500284,"statfs":{"total":95995035648,"available":91648290816,"internally_reserved":0,"allocated":3659378688,"data_stored":3659092076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":687274437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":2448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3644198912,"data_stored":3644194840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1444544512,"data_stored":1444544512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2290614272,"data_stored":2290614272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1974534144,"data_stored":1974534144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2927431680,"data_stored":2927427608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1794768896,"data_stored":1794768896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2568491008,"data_stored":2568486936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1692794880,"data_stored":1692794880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T14:05:19.567 INFO:tasks.thrashosds.thrasher:Setting 6.b to [4, 0, 5, 6, 3, 1] 2024-09-21T14:05:19.567 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '6.b', '4', '0', '5', '6', '3', '1'] 2024-09-21T14:05:19.568 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd pg-upmap-items 6.b 4 0 5 6 3 1 2024-09-21T14:05:19.598 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:05:19.620 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:05:19.635 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:19.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:19.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:05:19.736 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:19.778 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:19.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:19.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:19.879 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:19.912 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:19.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:19.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:20.014 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:05:20.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:20.052 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:20.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:05:20.104 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:19 smithi049 ceph-mon[30080]: pgmap v828: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 5.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 756 op/s; 7641/266322 objects misplaced (2.869%); 30 MiB/s, 483 objects/s recovering 2024-09-21T14:05:20.105 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:19 smithi049 ceph-mon[30080]: from='client.16305 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:05:20.105 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:19 smithi049 ceph-mon[30080]: 5.5 scrub starts 2024-09-21T14:05:20.105 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:19 smithi049 ceph-mon[30080]: 5.5 scrub ok 2024-09-21T14:05:20.105 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:19 smithi049 ceph-mon[40489]: pgmap v828: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 5.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 756 op/s; 7641/266322 objects misplaced (2.869%); 30 MiB/s, 483 objects/s recovering 2024-09-21T14:05:20.105 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:19 smithi049 ceph-mon[40489]: from='client.16305 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:05:20.105 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:19 smithi049 ceph-mon[40489]: 5.5 scrub starts 2024-09-21T14:05:20.105 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:19 smithi049 ceph-mon[40489]: 5.5 scrub ok 2024-09-21T14:05:20.153 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:05:20.188 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:20.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:20.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:05:20.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:19 smithi187 ceph-mon[33032]: pgmap v828: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 5.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 756 op/s; 7641/266322 objects misplaced (2.869%); 30 MiB/s, 483 objects/s recovering 2024-09-21T14:05:20.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:19 smithi187 ceph-mon[33032]: from='client.16305 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:05:20.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:19 smithi187 ceph-mon[33032]: 5.5 scrub starts 2024-09-21T14:05:20.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:19 smithi187 ceph-mon[33032]: 5.5 scrub ok 2024-09-21T14:05:20.194 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:05:19 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:05:20.287 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:05:20.287 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:05:20.290 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:05:20.303 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:05:20.324 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:20.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:20.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:05:20.325 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:05:20.325 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:05:20.362 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:05:20.363 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:05:20.401 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:05:20.401 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:05:20.425 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:05:20.435 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:05:20.435 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:05:20.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:20.469 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:20.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:05:20.474 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:05:20.474 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:05:20.514 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:05:20.514 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:05:20.554 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:05:20.554 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:05:20.570 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:20.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:20.601 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:20.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:20.702 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:05:20.703 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:05:20 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:05:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:05:20.742 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:20.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:20.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:05:20.844 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:05:20.885 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:20.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:20.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:05:20.987 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:21.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:20 smithi049 ceph-mon[40489]: 5.12 scrub starts 2024-09-21T14:05:21.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:20 smithi049 ceph-mon[40489]: 5.12 scrub ok 2024-09-21T14:05:21.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:20 smithi049 ceph-mon[40489]: 5.16 scrub starts 2024-09-21T14:05:21.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:20 smithi049 ceph-mon[40489]: 5.16 scrub ok 2024-09-21T14:05:21.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:20 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:05:21.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:20 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/891635194' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:05:21.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:20 smithi049 ceph-mon[30080]: 5.12 scrub starts 2024-09-21T14:05:21.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:20 smithi049 ceph-mon[30080]: 5.12 scrub ok 2024-09-21T14:05:21.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:20 smithi049 ceph-mon[30080]: 5.16 scrub starts 2024-09-21T14:05:21.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:20 smithi049 ceph-mon[30080]: 5.16 scrub ok 2024-09-21T14:05:21.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:20 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:05:21.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:20 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/891635194' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:05:21.020 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:21.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:21.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:21.052 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T14:05:21.122 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:21.153 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:21.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:21.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:21.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:20 smithi187 ceph-mon[33032]: 5.12 scrub starts 2024-09-21T14:05:21.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:20 smithi187 ceph-mon[33032]: 5.12 scrub ok 2024-09-21T14:05:21.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:20 smithi187 ceph-mon[33032]: 5.16 scrub starts 2024-09-21T14:05:21.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:20 smithi187 ceph-mon[33032]: 5.16 scrub ok 2024-09-21T14:05:21.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:20 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:05:21.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:20 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/891635194' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:05:21.255 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:05:21.306 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:21.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:21.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:05:21.408 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:05:21.430 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:21.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:21.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:05:21.531 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:05:21.565 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:21.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:21.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:05:21.667 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:05:21.696 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:21.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:21.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:05:21.798 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:05:21.821 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:21.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:21.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:05:21.884 INFO:teuthology.orchestra.run.smithi049.stderr:set 6.b pg_upmap_items mapping to [4->0,5->6,3->1] 2024-09-21T14:05:21.924 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:21.958 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:21.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:21.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:22.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:21 smithi049 ceph-mon[30080]: pgmap v829: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 5.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s; 5553/256083 objects misplaced (2.168%); 39 MiB/s, 628 objects/s recovering 2024-09-21T14:05:22.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:21 smithi049 ceph-mon[30080]: 6.17 scrub starts 2024-09-21T14:05:22.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:21 smithi049 ceph-mon[30080]: 6.17 scrub ok 2024-09-21T14:05:22.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:21 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1049601577' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "6.b", "id": [4, 0, 5, 6, 3, 1]} : dispatch 2024-09-21T14:05:22.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:21 smithi049 ceph-mon[40489]: pgmap v829: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 5.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s; 5553/256083 objects misplaced (2.168%); 39 MiB/s, 628 objects/s recovering 2024-09-21T14:05:22.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:21 smithi049 ceph-mon[40489]: 6.17 scrub starts 2024-09-21T14:05:22.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:21 smithi049 ceph-mon[40489]: 6.17 scrub ok 2024-09-21T14:05:22.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:21 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1049601577' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "6.b", "id": [4, 0, 5, 6, 3, 1]} : dispatch 2024-09-21T14:05:22.059 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:05:22.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:22.099 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:22.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:05:22.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:21 smithi187 ceph-mon[33032]: pgmap v829: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 5.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s; 5553/256083 objects misplaced (2.168%); 39 MiB/s, 628 objects/s recovering 2024-09-21T14:05:22.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:21 smithi187 ceph-mon[33032]: 6.17 scrub starts 2024-09-21T14:05:22.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:21 smithi187 ceph-mon[33032]: 6.17 scrub ok 2024-09-21T14:05:22.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:21 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1049601577' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "6.b", "id": [4, 0, 5, 6, 3, 1]} : dispatch 2024-09-21T14:05:22.200 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:05:22.247 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:22.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:22.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:05:22.349 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:22.388 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:22.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:22.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:22.490 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:22.524 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:22.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:22.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:22.626 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:05:22.647 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:22.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:22.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:05:22.683 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:05:22.749 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:05:22.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:22.792 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:22.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:05:22.893 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:22.915 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:22.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:22.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:23.017 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:23.051 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:23.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:23.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:23.089 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:22 smithi049 ceph-mon[40489]: 5.15 scrub starts 2024-09-21T14:05:23.289 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:22 smithi049 ceph-mon[40489]: 5.15 scrub ok 2024-09-21T14:05:23.289 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:22 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1049601577' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.b", "id": [4, 0, 5, 6, 3, 1]}]': finished 2024-09-21T14:05:23.289 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:22 smithi049 ceph-mon[40489]: osdmap e425: 8 total, 8 up, 8 in 2024-09-21T14:05:23.289 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:22 smithi049 ceph-mon[30080]: 5.15 scrub starts 2024-09-21T14:05:23.289 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:22 smithi049 ceph-mon[30080]: 5.15 scrub ok 2024-09-21T14:05:23.289 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:22 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1049601577' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.b", "id": [4, 0, 5, 6, 3, 1]}]': finished 2024-09-21T14:05:23.290 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:22 smithi049 ceph-mon[30080]: osdmap e425: 8 total, 8 up, 8 in 2024-09-21T14:05:23.291 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:05:23.291 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:22 smithi187 ceph-mon[33032]: 5.15 scrub starts 2024-09-21T14:05:23.291 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:22 smithi187 ceph-mon[33032]: 5.15 scrub ok 2024-09-21T14:05:23.292 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:22 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1049601577' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.b", "id": [4, 0, 5, 6, 3, 1]}]': finished 2024-09-21T14:05:23.292 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:22 smithi187 ceph-mon[33032]: osdmap e425: 8 total, 8 up, 8 in 2024-09-21T14:05:23.326 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:23.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:23.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:05:23.428 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:23.464 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:23.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:23.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:23.494 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T14:05:23.494 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-21T14:05:23.495 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-21T14:05:23.495 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-21T14:05:23.495 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.495 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:29977 9.b 9.ed124ccb (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e269)", 2024-09-21T14:05:23.495 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:58:16.771091+0000", 2024-09-21T14:05:23.495 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 425.89503809199999, 2024-09-21T14:05:23.495 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.97926761900000003, 2024-09-21T14:05:23.495 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:05:23.495 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:05:23.495 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-21T14:05:23.495 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:05:23.495 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:05:23.495 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:05:23.496 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 29977 2024-09-21T14:05:23.496 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.496 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:05:23.496 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.496 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:05:23.496 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.771091+0000", 2024-09-21T14:05:23.496 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:23.496 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.496 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.496 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:05:23.496 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.771091+0000", 2024-09-21T14:05:23.496 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:23.496 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.497 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.497 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:05:23.497 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.771093+0000", 2024-09-21T14:05:23.497 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.606e-06 2024-09-21T14:05:23.497 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.497 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.497 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:05:23.497 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.771152+0000", 2024-09-21T14:05:23.497 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9827000000000002e-05 2024-09-21T14:05:23.497 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.497 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.497 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:05:23.497 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.771162+0000", 2024-09-21T14:05:23.497 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0010000000000001e-05 2024-09-21T14:05:23.498 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.498 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.498 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:05:23.498 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.771171+0000", 2024-09-21T14:05:23.498 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2970000000000004e-06 2024-09-21T14:05:23.498 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.498 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.498 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:05:23.498 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.771209+0000", 2024-09-21T14:05:23.498 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8182000000000001e-05 2024-09-21T14:05:23.498 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.498 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.498 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:05:23.499 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.771215+0000", 2024-09-21T14:05:23.499 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9599999999999997e-06 2024-09-21T14:05:23.499 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.499 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.499 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:05:23.499 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:17.750348+0000", 2024-09-21T14:05:23.499 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.97913300400000003 2024-09-21T14:05:23.499 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.499 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.499 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:05:23.499 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:17.750359+0000", 2024-09-21T14:05:23.499 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0733e-05 2024-09-21T14:05:23.499 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:23.500 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:05:23.500 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:23.500 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.500 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.500 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:29992 9.b 9.6daaec5b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-09-21T14:05:23.500 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:58:16.780680+0000", 2024-09-21T14:05:23.500 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 425.885449381, 2024-09-21T14:05:23.500 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.96971144600000003, 2024-09-21T14:05:23.500 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:05:23.500 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:05:23.500 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-21T14:05:23.500 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:05:23.500 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:05:23.500 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:05:23.501 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 29992 2024-09-21T14:05:23.501 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.501 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:05:23.501 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.501 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:05:23.501 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.780680+0000", 2024-09-21T14:05:23.501 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:23.501 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.501 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.501 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:05:23.501 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.780680+0000", 2024-09-21T14:05:23.501 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:23.501 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.501 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.502 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:05:23.502 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.780681+0000", 2024-09-21T14:05:23.502 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6509999999999999e-06 2024-09-21T14:05:23.502 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.502 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.502 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:05:23.502 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.780750+0000", 2024-09-21T14:05:23.502 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8570000000000002e-05 2024-09-21T14:05:23.502 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.502 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.502 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:05:23.502 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.780762+0000", 2024-09-21T14:05:23.502 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2109999999999999e-05 2024-09-21T14:05:23.502 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.503 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.503 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:05:23.503 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.780770+0000", 2024-09-21T14:05:23.503 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3690000000000007e-06 2024-09-21T14:05:23.503 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.503 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.503 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:05:23.503 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.780828+0000", 2024-09-21T14:05:23.503 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.749e-05 2024-09-21T14:05:23.503 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.503 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.503 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:05:23.503 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.780836+0000", 2024-09-21T14:05:23.503 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1219999999999995e-06 2024-09-21T14:05:23.504 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.504 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.504 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:05:23.504 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:17.750382+0000", 2024-09-21T14:05:23.504 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.96954600400000002 2024-09-21T14:05:23.504 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.504 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.504 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:05:23.504 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:17.750391+0000", 2024-09-21T14:05:23.504 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1300000000000007e-06 2024-09-21T14:05:23.504 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:23.504 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:05:23.504 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:23.504 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.504 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.505 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:29996 9.b 9.d0040c7b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-09-21T14:05:23.505 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:58:16.785408+0000", 2024-09-21T14:05:23.505 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 425.88072105200001, 2024-09-21T14:05:23.505 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.96553488099999996, 2024-09-21T14:05:23.505 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:05:23.505 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:05:23.505 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-21T14:05:23.505 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:05:23.505 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:05:23.505 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:05:23.505 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 29996 2024-09-21T14:05:23.505 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.505 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:05:23.505 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.506 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:05:23.506 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.785408+0000", 2024-09-21T14:05:23.506 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:23.506 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.506 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.506 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:05:23.506 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.785408+0000", 2024-09-21T14:05:23.506 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:23.506 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.506 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.506 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:05:23.506 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.785410+0000", 2024-09-21T14:05:23.506 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5030000000000001e-06 2024-09-21T14:05:23.506 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.507 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.507 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:05:23.507 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.785466+0000", 2024-09-21T14:05:23.507 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6938999999999998e-05 2024-09-21T14:05:23.507 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.507 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.507 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:05:23.507 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.785477+0000", 2024-09-21T14:05:23.507 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0543000000000001e-05 2024-09-21T14:05:23.507 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.507 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.507 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:05:23.507 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.785483+0000", 2024-09-21T14:05:23.507 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5999999999999997e-06 2024-09-21T14:05:23.508 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.508 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.508 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:05:23.508 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.785528+0000", 2024-09-21T14:05:23.508 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5286e-05 2024-09-21T14:05:23.508 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.508 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.508 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:05:23.508 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.785537+0000", 2024-09-21T14:05:23.508 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3600000000000002e-06 2024-09-21T14:05:23.508 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.508 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.508 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:05:23.508 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:17.750935+0000", 2024-09-21T14:05:23.509 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.96539792099999999 2024-09-21T14:05:23.509 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.509 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.509 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:05:23.509 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:17.750943+0000", 2024-09-21T14:05:23.509 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.729e-06 2024-09-21T14:05:23.509 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:23.509 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:05:23.509 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:23.509 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.509 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.509 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:30003 9.b 9.b7b5b7bb (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-09-21T14:05:23.509 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:58:16.789071+0000", 2024-09-21T14:05:23.509 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 425.87705833199999, 2024-09-21T14:05:23.510 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.96190145699999996, 2024-09-21T14:05:23.510 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:05:23.510 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:05:23.510 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-21T14:05:23.510 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:05:23.510 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:05:23.510 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:05:23.510 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 30003 2024-09-21T14:05:23.510 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.510 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:05:23.510 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.510 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:05:23.510 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.789071+0000", 2024-09-21T14:05:23.510 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:23.511 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.511 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.511 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:05:23.511 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.789071+0000", 2024-09-21T14:05:23.511 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:23.511 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.511 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.511 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:05:23.511 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.789074+0000", 2024-09-21T14:05:23.511 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8430000000000001e-06 2024-09-21T14:05:23.511 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.511 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.511 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:05:23.511 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.789182+0000", 2024-09-21T14:05:23.512 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000108019 2024-09-21T14:05:23.512 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.512 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.512 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:05:23.512 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.789210+0000", 2024-09-21T14:05:23.512 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8503000000000001e-05 2024-09-21T14:05:23.512 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.512 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.512 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:05:23.512 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.789226+0000", 2024-09-21T14:05:23.512 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5403000000000001e-05 2024-09-21T14:05:23.512 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.512 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.512 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:05:23.512 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.789287+0000", 2024-09-21T14:05:23.513 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1326000000000002e-05 2024-09-21T14:05:23.513 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.513 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.513 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:05:23.513 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.789293+0000", 2024-09-21T14:05:23.513 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3509999999999998e-06 2024-09-21T14:05:23.513 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.513 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.513 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:05:23.513 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:17.750965+0000", 2024-09-21T14:05:23.513 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.96167175199999999 2024-09-21T14:05:23.513 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.513 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.513 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:05:23.514 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:17.750972+0000", 2024-09-21T14:05:23.514 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2599999999999999e-06 2024-09-21T14:05:23.514 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:23.514 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:05:23.514 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:23.514 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.514 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.514 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:30009 9.b 9.13ae813b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-09-21T14:05:23.514 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:58:16.793372+0000", 2024-09-21T14:05:23.514 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 425.87275681800003, 2024-09-21T14:05:23.514 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.95778215200000005, 2024-09-21T14:05:23.514 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:05:23.514 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:05:23.514 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-21T14:05:23.515 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:05:23.515 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:05:23.515 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:05:23.515 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 30009 2024-09-21T14:05:23.515 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.515 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:05:23.515 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.515 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:05:23.515 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.793372+0000", 2024-09-21T14:05:23.515 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:23.515 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.515 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.515 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:05:23.515 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.793372+0000", 2024-09-21T14:05:23.516 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:23.516 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.516 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.516 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:05:23.516 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.793374+0000", 2024-09-21T14:05:23.516 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0190000000000001e-06 2024-09-21T14:05:23.516 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.516 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.516 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:05:23.516 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.793447+0000", 2024-09-21T14:05:23.516 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2768000000000005e-05 2024-09-21T14:05:23.516 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.516 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.516 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:05:23.517 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.793464+0000", 2024-09-21T14:05:23.517 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6612e-05 2024-09-21T14:05:23.517 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.517 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.517 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:05:23.517 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.793472+0000", 2024-09-21T14:05:23.517 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.276e-06 2024-09-21T14:05:23.517 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.517 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.517 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:05:23.517 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.793527+0000", 2024-09-21T14:05:23.517 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4752e-05 2024-09-21T14:05:23.517 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.517 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.517 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:05:23.518 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.793535+0000", 2024-09-21T14:05:23.518 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3890000000000005e-06 2024-09-21T14:05:23.518 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.518 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.518 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:05:23.518 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:17.751138+0000", 2024-09-21T14:05:23.518 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.95760284799999995 2024-09-21T14:05:23.518 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.518 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.518 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:05:23.518 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:17.751154+0000", 2024-09-21T14:05:23.518 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6487999999999999e-05 2024-09-21T14:05:23.518 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:23.519 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:05:23.519 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:23.519 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.519 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.519 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:30021 9.b 9.9be4c56b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-09-21T14:05:23.519 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:58:16.803723+0000", 2024-09-21T14:05:23.519 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 425.86240574300001, 2024-09-21T14:05:23.519 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.94748764900000004, 2024-09-21T14:05:23.519 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:05:23.519 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:05:23.519 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-21T14:05:23.519 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:05:23.519 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:05:23.519 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:05:23.520 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 30021 2024-09-21T14:05:23.520 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.520 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:05:23.520 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.520 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:05:23.520 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.803723+0000", 2024-09-21T14:05:23.520 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:23.520 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.520 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.520 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:05:23.520 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.803723+0000", 2024-09-21T14:05:23.520 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:23.520 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.520 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.520 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:05:23.521 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.803725+0000", 2024-09-21T14:05:23.521 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.973e-06 2024-09-21T14:05:23.521 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.521 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.521 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:05:23.521 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.803794+0000", 2024-09-21T14:05:23.521 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9092999999999997e-05 2024-09-21T14:05:23.521 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.521 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.521 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:05:23.521 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.803814+0000", 2024-09-21T14:05:23.521 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9618000000000001e-05 2024-09-21T14:05:23.521 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.521 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.521 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:05:23.522 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.803825+0000", 2024-09-21T14:05:23.522 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0788000000000001e-05 2024-09-21T14:05:23.522 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.522 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.522 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:05:23.522 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.803887+0000", 2024-09-21T14:05:23.522 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2618000000000003e-05 2024-09-21T14:05:23.522 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.522 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.522 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:05:23.522 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:16.803893+0000", 2024-09-21T14:05:23.522 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.874e-06 2024-09-21T14:05:23.522 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.522 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.523 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:05:23.524 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:17.751204+0000", 2024-09-21T14:05:23.524 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.94731062300000002 2024-09-21T14:05:23.524 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.524 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.524 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:05:23.524 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:58:17.751211+0000", 2024-09-21T14:05:23.524 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.0620000000000003e-06 2024-09-21T14:05:23.524 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:23.524 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:05:23.524 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:23.524 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.525 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.525 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:113894 9.b 9:ddec8d1f:::benchmark_data_smithi049_138504_object113893:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-21T14:05:23.525 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:59:25.585009+0000", 2024-09-21T14:05:23.525 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 357.08111968200001, 2024-09-21T14:05:23.525 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.35427235800000001, 2024-09-21T14:05:23.525 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:05:23.525 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:05:23.525 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:05:23.525 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:05:23.525 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:05:23.525 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:05:23.525 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 113894 2024-09-21T14:05:23.525 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.525 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:05:23.526 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.526 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:05:23.526 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.585009+0000", 2024-09-21T14:05:23.526 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:23.526 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.526 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.526 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:05:23.526 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.585009+0000", 2024-09-21T14:05:23.526 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:23.526 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.526 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.526 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:05:23.526 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.585011+0000", 2024-09-21T14:05:23.526 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1659999999999999e-06 2024-09-21T14:05:23.527 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.527 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.527 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:05:23.527 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.585112+0000", 2024-09-21T14:05:23.527 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000101773 2024-09-21T14:05:23.527 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.527 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.527 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:05:23.527 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.585130+0000", 2024-09-21T14:05:23.527 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7668000000000001e-05 2024-09-21T14:05:23.527 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.527 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.527 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:05:23.527 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.585138+0000", 2024-09-21T14:05:23.528 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0020000000000006e-06 2024-09-21T14:05:23.528 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.528 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.528 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:05:23.528 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.590959+0000", 2024-09-21T14:05:23.528 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0058207110000000001 2024-09-21T14:05:23.528 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.528 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.528 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:05:23.528 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.591076+0000", 2024-09-21T14:05:23.528 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000116903 2024-09-21T14:05:23.528 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.528 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.528 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,5", 2024-09-21T14:05:23.528 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.591201+0000", 2024-09-21T14:05:23.529 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012490099999999999 2024-09-21T14:05:23.529 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.529 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.529 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:05:23.529 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.593938+0000", 2024-09-21T14:05:23.529 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.002737644 2024-09-21T14:05:23.529 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.529 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.529 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:05:23.529 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.595211+0000", 2024-09-21T14:05:23.529 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00127315 2024-09-21T14:05:23.529 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.529 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.529 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:05:23.530 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.939188+0000", 2024-09-21T14:05:23.530 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.34397667799999998 2024-09-21T14:05:23.530 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.530 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.530 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:05:23.530 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.939242+0000", 2024-09-21T14:05:23.530 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3783999999999999e-05 2024-09-21T14:05:23.530 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.530 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.530 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:05:23.530 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.939282+0000", 2024-09-21T14:05:23.530 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9978000000000002e-05 2024-09-21T14:05:23.530 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:23.530 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:05:23.531 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:23.531 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.531 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.531 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:113922 9.b 9:db1a22c4:::benchmark_data_smithi049_138504_object113921:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-21T14:05:23.531 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:59:25.604822+0000", 2024-09-21T14:05:23.531 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 357.06130759000001, 2024-09-21T14:05:23.531 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.34532540099999998, 2024-09-21T14:05:23.531 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:05:23.531 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:05:23.531 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:05:23.531 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:05:23.531 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:05:23.531 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:05:23.531 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 113922 2024-09-21T14:05:23.531 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.532 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:05:23.532 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.532 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:05:23.532 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.604822+0000", 2024-09-21T14:05:23.532 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:23.532 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.532 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.532 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:05:23.532 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.604822+0000", 2024-09-21T14:05:23.532 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:23.532 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.532 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.532 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:05:23.532 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.604823+0000", 2024-09-21T14:05:23.533 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.871e-06 2024-09-21T14:05:23.533 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.533 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.533 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:05:23.533 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.604877+0000", 2024-09-21T14:05:23.533 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4112e-05 2024-09-21T14:05:23.533 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.533 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.533 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:05:23.533 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.604896+0000", 2024-09-21T14:05:23.533 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8569000000000001e-05 2024-09-21T14:05:23.533 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.533 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.533 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:05:23.533 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.604906+0000", 2024-09-21T14:05:23.534 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.8889999999999997e-06 2024-09-21T14:05:23.534 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.534 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.534 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:05:23.534 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.604971+0000", 2024-09-21T14:05:23.534 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5406000000000004e-05 2024-09-21T14:05:23.534 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.534 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.534 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:05:23.534 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.605106+0000", 2024-09-21T14:05:23.534 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013487199999999999 2024-09-21T14:05:23.534 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.534 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.534 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,5", 2024-09-21T14:05:23.535 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.605248+0000", 2024-09-21T14:05:23.535 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014186299999999999 2024-09-21T14:05:23.535 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.535 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.535 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:05:23.535 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.608440+0000", 2024-09-21T14:05:23.535 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0031917349999999998 2024-09-21T14:05:23.535 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.535 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.535 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:05:23.535 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.608968+0000", 2024-09-21T14:05:23.535 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00052824000000000002 2024-09-21T14:05:23.535 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.535 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.536 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:05:23.536 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.950032+0000", 2024-09-21T14:05:23.536 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.34106423200000002 2024-09-21T14:05:23.536 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.536 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.536 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:05:23.536 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.950093+0000", 2024-09-21T14:05:23.536 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1123000000000004e-05 2024-09-21T14:05:23.536 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.536 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.536 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:05:23.536 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:59:25.950147+0000", 2024-09-21T14:05:23.536 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3489000000000002e-05 2024-09-21T14:05:23.536 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:23.536 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:05:23.537 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:23.537 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.537 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.537 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24365.0:3299 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94025139202304 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-21T14:05:23.537 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:04:20.181187+0000", 2024-09-21T14:05:23.537 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 62.484941855999999, 2024-09-21T14:05:23.537 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.036669439, 2024-09-21T14:05:23.537 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:05:23.537 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:05:23.537 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T14:05:23.537 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:05:23.537 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24365", 2024-09-21T14:05:23.537 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3404764347", 2024-09-21T14:05:23.537 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 3299 2024-09-21T14:05:23.538 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.538 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:05:23.538 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.538 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:05:23.538 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.181187+0000", 2024-09-21T14:05:23.538 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:23.538 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.538 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.538 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:05:23.538 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.181187+0000", 2024-09-21T14:05:23.538 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:23.538 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.538 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.538 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:05:23.539 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.181189+0000", 2024-09-21T14:05:23.539 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3170000000000001e-06 2024-09-21T14:05:23.539 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.539 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.539 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:05:23.539 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.181194+0000", 2024-09-21T14:05:23.539 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2129999999999999e-06 2024-09-21T14:05:23.539 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.539 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.539 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:05:23.539 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.181210+0000", 2024-09-21T14:05:23.539 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6302000000000001e-05 2024-09-21T14:05:23.539 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.539 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.540 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:05:23.540 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.181218+0000", 2024-09-21T14:05:23.540 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4449999999999999e-06 2024-09-21T14:05:23.540 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.540 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.540 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:05:23.540 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.181297+0000", 2024-09-21T14:05:23.540 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9095000000000001e-05 2024-09-21T14:05:23.540 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.540 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.540 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op must wait for map", 2024-09-21T14:05:23.540 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.181303+0000", 2024-09-21T14:05:23.540 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9050000000000002e-06 2024-09-21T14:05:23.540 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.541 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.541 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:05:23.541 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.190356+0000", 2024-09-21T14:05:23.541 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0090531729999999994 2024-09-21T14:05:23.541 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.541 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.541 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:05:23.541 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.190365+0000", 2024-09-21T14:05:23.541 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.9670000000000007e-06 2024-09-21T14:05:23.541 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.541 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.541 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:05:23.541 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.207076+0000", 2024-09-21T14:05:23.541 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0167116089999999 2024-09-21T14:05:23.541 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.542 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.542 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for degraded object", 2024-09-21T14:05:23.542 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.208617+0000", 2024-09-21T14:05:23.542 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001540249 2024-09-21T14:05:23.542 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.542 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.542 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:05:23.542 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.214326+0000", 2024-09-21T14:05:23.542 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0057091950000000002 2024-09-21T14:05:23.542 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.542 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.542 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:05:23.542 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.214408+0000", 2024-09-21T14:05:23.543 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2770999999999997e-05 2024-09-21T14:05:23.543 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.543 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.543 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:05:23.543 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.217857+0000", 2024-09-21T14:05:23.543 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003448198 2024-09-21T14:05:23.543 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:23.543 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:05:23.543 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:23.543 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.543 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.543 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:112204 10.a 10:5c740623:::benchmark_data_smithi049_176811_object112203:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-21T14:05:23.543 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:04:20.206510+0000", 2024-09-21T14:05:23.543 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 62.459618849000002, 2024-09-21T14:05:23.544 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1818630240000001, 2024-09-21T14:05:23.544 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:05:23.544 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:05:23.544 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:05:23.544 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:05:23.544 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:05:23.544 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:05:23.544 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 112204 2024-09-21T14:05:23.544 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.544 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:05:23.544 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.544 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:05:23.544 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.206510+0000", 2024-09-21T14:05:23.544 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:23.545 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.545 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.545 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:05:23.545 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.206510+0000", 2024-09-21T14:05:23.545 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:23.545 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.545 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.545 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:05:23.545 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.206511+0000", 2024-09-21T14:05:23.545 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0900000000000001e-07 2024-09-21T14:05:23.545 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.545 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.545 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:05:23.545 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.206541+0000", 2024-09-21T14:05:23.546 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0082000000000001e-05 2024-09-21T14:05:23.546 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.546 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.546 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:05:23.546 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.206548+0000", 2024-09-21T14:05:23.546 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.0110000000000003e-06 2024-09-21T14:05:23.546 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.546 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.546 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:05:23.546 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.206553+0000", 2024-09-21T14:05:23.546 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1529999999999996e-06 2024-09-21T14:05:23.546 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.546 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.546 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:05:23.547 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.206601+0000", 2024-09-21T14:05:23.547 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8393000000000002e-05 2024-09-21T14:05:23.547 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.547 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.547 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:05:23.547 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.206610+0000", 2024-09-21T14:05:23.547 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.507e-06 2024-09-21T14:05:23.547 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.547 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.547 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:05:23.547 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.188308+0000", 2024-09-21T14:05:23.547 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.98169809900000005 2024-09-21T14:05:23.547 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.547 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.547 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:05:23.548 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.188318+0000", 2024-09-21T14:05:23.548 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0179e-05 2024-09-21T14:05:23.548 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.548 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.548 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:05:23.548 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.370141+0000", 2024-09-21T14:05:23.548 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1818224210000001 2024-09-21T14:05:23.548 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.548 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.548 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:05:23.548 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.370277+0000", 2024-09-21T14:05:23.548 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013640399999999999 2024-09-21T14:05:23.548 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.548 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.549 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2,5", 2024-09-21T14:05:23.549 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.370711+0000", 2024-09-21T14:05:23.549 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000433826 2024-09-21T14:05:23.549 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.549 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.549 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:05:23.549 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.377442+0000", 2024-09-21T14:05:23.549 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0067311239999999998 2024-09-21T14:05:23.549 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.549 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.549 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:05:23.549 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.380659+0000", 2024-09-21T14:05:23.549 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0032168869999999999 2024-09-21T14:05:23.549 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.549 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.550 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:05:23.550 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.388142+0000", 2024-09-21T14:05:23.550 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0074832570000000001 2024-09-21T14:05:23.550 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.550 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.550 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:05:23.550 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.388212+0000", 2024-09-21T14:05:23.550 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9850000000000004e-05 2024-09-21T14:05:23.550 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.550 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.550 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:05:23.550 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.388269+0000", 2024-09-21T14:05:23.550 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7224e-05 2024-09-21T14:05:23.551 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.551 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.551 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:05:23.551 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.388373+0000", 2024-09-21T14:05:23.551 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000103998 2024-09-21T14:05:23.551 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:23.551 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:05:23.551 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:23.551 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.551 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.551 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:112222 10.d 10:bb42e81a:::benchmark_data_smithi049_176811_object112221:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-21T14:05:23.551 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:04:20.235680+0000", 2024-09-21T14:05:23.551 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 62.430449388, 2024-09-21T14:05:23.551 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0718219699999998, 2024-09-21T14:05:23.552 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:05:23.552 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:05:23.552 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:05:23.552 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:05:23.552 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:05:23.552 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:05:23.552 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 112222 2024-09-21T14:05:23.552 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.552 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:05:23.552 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.552 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:05:23.552 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235680+0000", 2024-09-21T14:05:23.552 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:23.552 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.552 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.553 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:05:23.553 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235680+0000", 2024-09-21T14:05:23.553 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:23.553 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.553 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.553 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:05:23.553 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235681+0000", 2024-09-21T14:05:23.553 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.285e-06 2024-09-21T14:05:23.553 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.553 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.553 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:05:23.553 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235734+0000", 2024-09-21T14:05:23.553 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2580000000000001e-05 2024-09-21T14:05:23.553 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.554 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.554 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:05:23.554 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235744+0000", 2024-09-21T14:05:23.554 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0197000000000001e-05 2024-09-21T14:05:23.554 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.554 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.554 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:05:23.554 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235749+0000", 2024-09-21T14:05:23.554 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3809999999999998e-06 2024-09-21T14:05:23.554 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.554 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.554 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:05:23.554 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235804+0000", 2024-09-21T14:05:23.554 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4592999999999997e-05 2024-09-21T14:05:23.555 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.555 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.555 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:05:23.555 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235814+0000", 2024-09-21T14:05:23.555 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0545e-05 2024-09-21T14:05:23.555 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.555 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.555 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:05:23.555 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.187322+0000", 2024-09-21T14:05:23.555 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.95150779399999996 2024-09-21T14:05:23.555 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.555 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.555 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:05:23.555 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.187331+0000", 2024-09-21T14:05:23.555 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.6929999999999996e-06 2024-09-21T14:05:23.556 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.556 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.556 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:05:23.556 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.285689+0000", 2024-09-21T14:05:23.556 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.098357961 2024-09-21T14:05:23.556 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.556 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.556 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:05:23.556 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.285795+0000", 2024-09-21T14:05:23.556 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010604599999999999 2024-09-21T14:05:23.556 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.556 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.556 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3,4", 2024-09-21T14:05:23.557 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.286188+0000", 2024-09-21T14:05:23.557 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00039307900000000001 2024-09-21T14:05:23.557 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.557 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.557 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:05:23.557 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.297975+0000", 2024-09-21T14:05:23.557 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.011787446 2024-09-21T14:05:23.557 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.557 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.557 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:05:23.557 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.298048+0000", 2024-09-21T14:05:23.557 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3146999999999996e-05 2024-09-21T14:05:23.557 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.557 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.558 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:05:23.558 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.298141+0000", 2024-09-21T14:05:23.558 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.2391000000000004e-05 2024-09-21T14:05:23.558 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.558 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.558 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:05:23.558 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.307285+0000", 2024-09-21T14:05:23.558 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0091442880000000004 2024-09-21T14:05:23.558 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.558 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.558 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:05:23.558 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.307366+0000", 2024-09-21T14:05:23.558 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1155999999999993e-05 2024-09-21T14:05:23.558 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.559 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.559 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:05:23.559 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.307502+0000", 2024-09-21T14:05:23.559 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013538800000000001 2024-09-21T14:05:23.559 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:23.559 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:05:23.559 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:23.559 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.559 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.559 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:112223 10.0 10:05563d3b:::benchmark_data_smithi049_176811_object112222:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-21T14:05:23.559 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:04:20.235762+0000", 2024-09-21T14:05:23.559 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 62.430367515999997, 2024-09-21T14:05:23.559 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1264867669999998, 2024-09-21T14:05:23.559 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:05:23.560 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:05:23.560 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:05:23.560 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:05:23.560 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:05:23.560 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:05:23.560 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 112223 2024-09-21T14:05:23.560 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.560 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:05:23.560 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.560 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:05:23.560 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235762+0000", 2024-09-21T14:05:23.560 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:23.560 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.560 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.560 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:05:23.561 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235762+0000", 2024-09-21T14:05:23.561 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:23.561 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.561 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.561 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:05:23.561 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235762+0000", 2024-09-21T14:05:23.561 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2799999999999996e-07 2024-09-21T14:05:23.561 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.561 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.561 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:05:23.561 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235796+0000", 2024-09-21T14:05:23.561 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3402000000000003e-05 2024-09-21T14:05:23.561 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.561 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.562 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:05:23.562 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235803+0000", 2024-09-21T14:05:23.562 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9929999999999998e-06 2024-09-21T14:05:23.562 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.562 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.562 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:05:23.562 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235808+0000", 2024-09-21T14:05:23.562 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9699999999999998e-06 2024-09-21T14:05:23.562 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.562 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.562 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:05:23.562 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235863+0000", 2024-09-21T14:05:23.562 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5877000000000001e-05 2024-09-21T14:05:23.562 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.562 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.563 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:05:23.563 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235870+0000", 2024-09-21T14:05:23.563 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.007e-06 2024-09-21T14:05:23.563 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.563 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.563 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:05:23.563 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.186804+0000", 2024-09-21T14:05:23.563 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.95093328600000004 2024-09-21T14:05:23.563 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.563 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.563 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:05:23.563 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.186815+0000", 2024-09-21T14:05:23.563 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1265e-05 2024-09-21T14:05:23.563 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.564 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.564 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:05:23.564 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.336944+0000", 2024-09-21T14:05:23.564 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1501292430000001 2024-09-21T14:05:23.564 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.564 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.564 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:05:23.564 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.337058+0000", 2024-09-21T14:05:23.564 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000113881 2024-09-21T14:05:23.564 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.564 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.564 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,4,7", 2024-09-21T14:05:23.564 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.337547+0000", 2024-09-21T14:05:23.564 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00048919999999999996 2024-09-21T14:05:23.564 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.565 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.565 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:05:23.565 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.346097+0000", 2024-09-21T14:05:23.565 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0085500969999999996 2024-09-21T14:05:23.565 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.565 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.565 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:05:23.565 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.346145+0000", 2024-09-21T14:05:23.565 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7700000000000001e-05 2024-09-21T14:05:23.565 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.565 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.565 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:05:23.565 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.362046+0000", 2024-09-21T14:05:23.565 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.015900935000000001 2024-09-21T14:05:23.566 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.566 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.566 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:05:23.566 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.362138+0000", 2024-09-21T14:05:23.566 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1525000000000002e-05 2024-09-21T14:05:23.566 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.566 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.566 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:05:23.566 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.362175+0000", 2024-09-21T14:05:23.566 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6928999999999998e-05 2024-09-21T14:05:23.566 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.566 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.566 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:05:23.566 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.362248+0000", 2024-09-21T14:05:23.567 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3829000000000001e-05 2024-09-21T14:05:23.567 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:23.567 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:05:23.567 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:23.567 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.567 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.567 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:112226 10.0 10:0f5734d7:::benchmark_data_smithi049_176811_object112225:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-21T14:05:23.567 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:04:20.235905+0000", 2024-09-21T14:05:23.567 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 62.430224045999999, 2024-09-21T14:05:23.567 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1266882329999999, 2024-09-21T14:05:23.567 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:05:23.567 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:05:23.567 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:05:23.567 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:05:23.568 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:05:23.568 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:05:23.568 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 112226 2024-09-21T14:05:23.568 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.568 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:05:23.568 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.568 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:05:23.568 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235905+0000", 2024-09-21T14:05:23.568 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:23.568 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.568 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.568 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:05:23.568 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235905+0000", 2024-09-21T14:05:23.568 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:23.568 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.569 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.569 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:05:23.569 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235906+0000", 2024-09-21T14:05:23.569 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4e-07 2024-09-21T14:05:23.569 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.569 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.569 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:05:23.569 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235975+0000", 2024-09-21T14:05:23.569 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9052000000000002e-05 2024-09-21T14:05:23.569 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.569 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.569 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:05:23.569 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235985+0000", 2024-09-21T14:05:23.569 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.9690000000000006e-06 2024-09-21T14:05:23.570 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.570 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.570 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:05:23.570 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235991+0000", 2024-09-21T14:05:23.570 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9939999999999997e-06 2024-09-21T14:05:23.570 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.570 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.570 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:05:23.570 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.236048+0000", 2024-09-21T14:05:23.570 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6943e-05 2024-09-21T14:05:23.570 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.570 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.570 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:05:23.570 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.236057+0000", 2024-09-21T14:05:23.571 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.4790000000000002e-06 2024-09-21T14:05:23.571 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.571 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.571 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:05:23.571 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.187660+0000", 2024-09-21T14:05:23.571 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.95160278700000001 2024-09-21T14:05:23.571 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.571 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.571 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:05:23.571 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.187669+0000", 2024-09-21T14:05:23.571 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.2979999999999997e-06 2024-09-21T14:05:23.571 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.571 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.571 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:05:23.571 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.346204+0000", 2024-09-21T14:05:23.572 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1585341869999999 2024-09-21T14:05:23.572 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.572 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.572 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:05:23.572 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.346382+0000", 2024-09-21T14:05:23.572 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00017858500000000001 2024-09-21T14:05:23.572 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.572 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.572 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,4,7", 2024-09-21T14:05:23.572 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.346555+0000", 2024-09-21T14:05:23.572 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0001728 2024-09-21T14:05:23.572 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.572 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.572 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:05:23.573 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.353773+0000", 2024-09-21T14:05:23.573 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0072175240000000003 2024-09-21T14:05:23.573 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.573 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.573 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:05:23.573 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.362086+0000", 2024-09-21T14:05:23.573 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0083139189999999995 2024-09-21T14:05:23.573 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.573 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.573 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:05:23.573 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.362257+0000", 2024-09-21T14:05:23.573 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00017090200000000001 2024-09-21T14:05:23.573 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.573 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.573 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:05:23.574 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.362405+0000", 2024-09-21T14:05:23.574 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014748800000000001 2024-09-21T14:05:23.574 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.574 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.574 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:05:23.574 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.362477+0000", 2024-09-21T14:05:23.574 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2214999999999999e-05 2024-09-21T14:05:23.574 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.574 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.574 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:05:23.574 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.362593+0000", 2024-09-21T14:05:23.574 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000116251 2024-09-21T14:05:23.574 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:23.574 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:05:23.575 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:23.575 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.575 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.575 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:112230 10.d 10:b278ca69:::benchmark_data_smithi049_176811_object112229:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-21T14:05:23.575 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:04:20.239727+0000", 2024-09-21T14:05:23.575 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 62.426401798999997, 2024-09-21T14:05:23.575 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0797783679999999, 2024-09-21T14:05:23.575 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:05:23.575 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:05:23.575 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:05:23.575 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:05:23.575 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:05:23.575 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:05:23.575 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 112230 2024-09-21T14:05:23.575 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.576 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:05:23.576 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.576 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:05:23.576 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.239727+0000", 2024-09-21T14:05:23.576 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:23.576 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.576 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.576 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:05:23.576 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.239727+0000", 2024-09-21T14:05:23.576 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:23.576 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.576 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.576 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:05:23.576 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.239728+0000", 2024-09-21T14:05:23.577 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8199999999999999e-07 2024-09-21T14:05:23.577 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.577 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.577 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:05:23.577 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.239763+0000", 2024-09-21T14:05:23.577 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4799999999999999e-05 2024-09-21T14:05:23.577 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.577 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.577 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:05:23.577 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.239772+0000", 2024-09-21T14:05:23.577 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1099999999999992e-06 2024-09-21T14:05:23.577 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.577 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.577 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:05:23.578 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.239778+0000", 2024-09-21T14:05:23.578 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0920000000000003e-06 2024-09-21T14:05:23.578 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.578 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.578 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:05:23.578 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.239835+0000", 2024-09-21T14:05:23.578 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6758000000000001e-05 2024-09-21T14:05:23.578 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.578 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.578 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:05:23.579 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.239861+0000", 2024-09-21T14:05:23.579 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.6290000000000001e-05 2024-09-21T14:05:23.579 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.579 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.579 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:05:23.579 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.187397+0000", 2024-09-21T14:05:23.579 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.94753606300000004 2024-09-21T14:05:23.579 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.579 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.579 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:05:23.579 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.187407+0000", 2024-09-21T14:05:23.579 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6600000000000007e-06 2024-09-21T14:05:23.579 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.579 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.580 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:05:23.580 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.290801+0000", 2024-09-21T14:05:23.580 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1033946459999999 2024-09-21T14:05:23.580 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.580 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.580 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:05:23.580 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.290936+0000", 2024-09-21T14:05:23.580 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013488899999999999 2024-09-21T14:05:23.580 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.580 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.580 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3,4", 2024-09-21T14:05:23.580 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.291093+0000", 2024-09-21T14:05:23.580 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015686300000000001 2024-09-21T14:05:23.580 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.581 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.581 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:05:23.581 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.298120+0000", 2024-09-21T14:05:23.581 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0070268209999999999 2024-09-21T14:05:23.581 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.581 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.581 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:05:23.581 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.298149+0000", 2024-09-21T14:05:23.581 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8609999999999999e-05 2024-09-21T14:05:23.581 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.581 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.581 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:05:23.581 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.298207+0000", 2024-09-21T14:05:23.581 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7859000000000003e-05 2024-09-21T14:05:23.582 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.582 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.582 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:05:23.582 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.319309+0000", 2024-09-21T14:05:23.582 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.021102584000000001 2024-09-21T14:05:23.582 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.582 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.582 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:05:23.582 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.319379+0000", 2024-09-21T14:05:23.582 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9858999999999996e-05 2024-09-21T14:05:23.582 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.582 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.582 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:05:23.582 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.319506+0000", 2024-09-21T14:05:23.583 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000126682 2024-09-21T14:05:23.583 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:23.583 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:05:23.583 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:23.583 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.583 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.583 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:112234 10.a 10:5cc5be6b:::benchmark_data_smithi049_176811_object112233:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-21T14:05:23.583 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:04:20.240298+0000", 2024-09-21T14:05:23.583 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 62.425831322000001, 2024-09-21T14:05:23.583 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.148374665, 2024-09-21T14:05:23.583 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:05:23.583 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:05:23.583 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:05:23.583 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:05:23.584 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:05:23.584 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:05:23.584 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 112234 2024-09-21T14:05:23.584 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.584 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:05:23.584 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.584 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:05:23.584 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.240298+0000", 2024-09-21T14:05:23.584 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:23.584 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.584 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.584 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:05:23.584 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.240298+0000", 2024-09-21T14:05:23.584 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:23.585 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.585 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.585 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:05:23.585 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.240299+0000", 2024-09-21T14:05:23.585 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4199999999999995e-07 2024-09-21T14:05:23.585 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.585 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.585 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:05:23.585 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.240385+0000", 2024-09-21T14:05:23.585 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.6550000000000003e-05 2024-09-21T14:05:23.585 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.585 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.585 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:05:23.585 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.240395+0000", 2024-09-21T14:05:23.586 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.8579999999999995e-06 2024-09-21T14:05:23.586 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.586 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.586 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:05:23.586 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.240400+0000", 2024-09-21T14:05:23.586 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3040000000000004e-06 2024-09-21T14:05:23.586 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.586 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.586 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:05:23.586 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.240444+0000", 2024-09-21T14:05:23.586 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3594000000000003e-05 2024-09-21T14:05:23.586 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.586 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.586 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:05:23.587 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.240451+0000", 2024-09-21T14:05:23.587 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.0319999999999998e-06 2024-09-21T14:05:23.587 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.587 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.587 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:05:23.587 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.188345+0000", 2024-09-21T14:05:23.587 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.94789378400000002 2024-09-21T14:05:23.587 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.587 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.587 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:05:23.587 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.188351+0000", 2024-09-21T14:05:23.587 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7619999999999998e-06 2024-09-21T14:05:23.587 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.587 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.587 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:05:23.588 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.377491+0000", 2024-09-21T14:05:23.588 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.189139642 2024-09-21T14:05:23.588 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.588 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.588 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:05:23.588 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.377612+0000", 2024-09-21T14:05:23.588 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000120528 2024-09-21T14:05:23.588 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.588 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.588 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2,5", 2024-09-21T14:05:23.588 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.377713+0000", 2024-09-21T14:05:23.588 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010192 2024-09-21T14:05:23.588 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.588 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.589 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:05:23.589 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.388079+0000", 2024-09-21T14:05:23.589 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.010365298 2024-09-21T14:05:23.589 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.589 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.589 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:05:23.589 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.388158+0000", 2024-09-21T14:05:23.589 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8726000000000005e-05 2024-09-21T14:05:23.589 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.589 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.589 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:05:23.589 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.388482+0000", 2024-09-21T14:05:23.589 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00032423 2024-09-21T14:05:23.589 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.589 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.590 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:05:23.590 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.388524+0000", 2024-09-21T14:05:23.590 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1782000000000001e-05 2024-09-21T14:05:23.590 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.590 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.590 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:05:23.590 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.388577+0000", 2024-09-21T14:05:23.590 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3965999999999998e-05 2024-09-21T14:05:23.590 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.590 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.590 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:05:23.590 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.388672+0000", 2024-09-21T14:05:23.590 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.4946999999999997e-05 2024-09-21T14:05:23.590 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:23.591 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:05:23.591 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:23.591 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.591 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.591 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:112240 10.a 10:59634543:::benchmark_data_smithi049_176811_object112239:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-21T14:05:23.591 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:04:20.245943+0000", 2024-09-21T14:05:23.591 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 62.420186065999999, 2024-09-21T14:05:23.591 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1573033979999998, 2024-09-21T14:05:23.591 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:05:23.591 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:05:23.591 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:05:23.591 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:05:23.591 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:05:23.591 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:05:23.592 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 112240 2024-09-21T14:05:23.592 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.592 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:05:23.592 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.592 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:05:23.592 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.245943+0000", 2024-09-21T14:05:23.592 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:23.592 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.592 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.592 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:05:23.592 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.245943+0000", 2024-09-21T14:05:23.592 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:23.592 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.592 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.592 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:05:23.593 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.245944+0000", 2024-09-21T14:05:23.593 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3540000000000001e-06 2024-09-21T14:05:23.593 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.593 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.593 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:05:23.593 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.246007+0000", 2024-09-21T14:05:23.593 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2155000000000004e-05 2024-09-21T14:05:23.593 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.593 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.593 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:05:23.593 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.246019+0000", 2024-09-21T14:05:23.593 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2814e-05 2024-09-21T14:05:23.593 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.593 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.594 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:05:23.594 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.246026+0000", 2024-09-21T14:05:23.594 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1470000000000004e-06 2024-09-21T14:05:23.594 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.594 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.594 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:05:23.594 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.246084+0000", 2024-09-21T14:05:23.594 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7731e-05 2024-09-21T14:05:23.594 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.594 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.594 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:05:23.594 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.246095+0000", 2024-09-21T14:05:23.594 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0964e-05 2024-09-21T14:05:23.595 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.595 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.595 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:05:23.595 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.188376+0000", 2024-09-21T14:05:23.595 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.94228104199999996 2024-09-21T14:05:23.595 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.595 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.595 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:05:23.595 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.188383+0000", 2024-09-21T14:05:23.595 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8979999999999999e-06 2024-09-21T14:05:23.595 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.595 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.595 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:05:23.595 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.380685+0000", 2024-09-21T14:05:23.595 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.192302239 2024-09-21T14:05:23.596 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.596 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.596 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:05:23.596 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.381343+0000", 2024-09-21T14:05:23.596 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00065746299999999995 2024-09-21T14:05:23.596 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.596 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.596 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2,5", 2024-09-21T14:05:23.596 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.381520+0000", 2024-09-21T14:05:23.596 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00017690300000000001 2024-09-21T14:05:23.596 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.596 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.596 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:05:23.597 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.388164+0000", 2024-09-21T14:05:23.597 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0066440359999999999 2024-09-21T14:05:23.597 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.597 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.597 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:05:23.597 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.388424+0000", 2024-09-21T14:05:23.597 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00026048999999999999 2024-09-21T14:05:23.597 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.597 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.597 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:05:23.597 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.388788+0000", 2024-09-21T14:05:23.597 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00036423599999999999 2024-09-21T14:05:23.597 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.597 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.597 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:05:23.598 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.403084+0000", 2024-09-21T14:05:23.598 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.014295491 2024-09-21T14:05:23.598 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.598 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.598 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:05:23.598 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.403141+0000", 2024-09-21T14:05:23.598 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7290999999999997e-05 2024-09-21T14:05:23.598 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.598 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.598 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:05:23.598 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.403246+0000", 2024-09-21T14:05:23.598 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010514399999999999 2024-09-21T14:05:23.598 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:23.599 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:05:23.599 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:23.599 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.599 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.599 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:112251 10.a 10:59ed2c1b:::benchmark_data_smithi049_176811_object112250:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-21T14:05:23.599 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:04:20.255435+0000", 2024-09-21T14:05:23.599 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 62.410693983000002, 2024-09-21T14:05:23.599 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1536580829999998, 2024-09-21T14:05:23.599 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:05:23.599 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:05:23.599 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:05:23.599 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:05:23.599 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:05:23.599 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:05:23.599 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 112251 2024-09-21T14:05:23.600 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.600 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:05:23.600 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.600 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:05:23.600 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.255435+0000", 2024-09-21T14:05:23.600 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:23.600 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.600 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.600 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:05:23.600 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.255435+0000", 2024-09-21T14:05:23.600 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:23.600 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.600 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.600 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:05:23.601 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.255436+0000", 2024-09-21T14:05:23.601 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.184e-06 2024-09-21T14:05:23.601 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.601 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.601 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:05:23.601 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.255522+0000", 2024-09-21T14:05:23.601 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5778000000000004e-05 2024-09-21T14:05:23.601 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.601 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.601 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:05:23.601 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.255532+0000", 2024-09-21T14:05:23.601 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.8199999999999992e-06 2024-09-21T14:05:23.601 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.601 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.602 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:05:23.602 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.255538+0000", 2024-09-21T14:05:23.602 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8710000000000002e-06 2024-09-21T14:05:23.602 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.602 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.602 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:05:23.602 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.255589+0000", 2024-09-21T14:05:23.602 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1644000000000002e-05 2024-09-21T14:05:23.602 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.602 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.602 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:05:23.602 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.255596+0000", 2024-09-21T14:05:23.602 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6499999999999999e-06 2024-09-21T14:05:23.603 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.603 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.603 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:05:23.603 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.188406+0000", 2024-09-21T14:05:23.603 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.932810425 2024-09-21T14:05:23.603 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.603 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.603 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:05:23.603 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.188413+0000", 2024-09-21T14:05:23.603 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9859999999999997e-06 2024-09-21T14:05:23.603 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.603 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.603 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:05:23.603 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.384443+0000", 2024-09-21T14:05:23.604 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.196029257 2024-09-21T14:05:23.604 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.604 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.604 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:05:23.604 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.384560+0000", 2024-09-21T14:05:23.604 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000117766 2024-09-21T14:05:23.604 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.604 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.604 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2,5", 2024-09-21T14:05:23.604 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.384711+0000", 2024-09-21T14:05:23.604 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015052899999999999 2024-09-21T14:05:23.604 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.604 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.604 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:05:23.605 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.388438+0000", 2024-09-21T14:05:23.605 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00372669 2024-09-21T14:05:23.605 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.605 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.605 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:05:23.605 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.388729+0000", 2024-09-21T14:05:23.605 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00029102899999999999 2024-09-21T14:05:23.605 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.605 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.605 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:05:23.605 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.397078+0000", 2024-09-21T14:05:23.605 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0083490219999999993 2024-09-21T14:05:23.605 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.605 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.606 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:05:23.606 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.408920+0000", 2024-09-21T14:05:23.606 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.011842065000000001 2024-09-21T14:05:23.606 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.606 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.606 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:05:23.606 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.408981+0000", 2024-09-21T14:05:23.606 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1569999999999995e-05 2024-09-21T14:05:23.606 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.606 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.606 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:05:23.606 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.409093+0000", 2024-09-21T14:05:23.606 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000111797 2024-09-21T14:05:23.607 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:23.607 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:05:23.607 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:23.607 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.607 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.607 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:112253 10.d 10:be7afa8e:::benchmark_data_smithi049_176811_object112252:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-21T14:05:23.607 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:04:20.263232+0000", 2024-09-21T14:05:23.607 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 62.402897207999999, 2024-09-21T14:05:23.607 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0592742639999999, 2024-09-21T14:05:23.607 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:05:23.607 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:05:23.607 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:05:23.607 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:05:23.607 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:05:23.607 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:05:23.608 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 112253 2024-09-21T14:05:23.608 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.608 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:05:23.608 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.608 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:05:23.608 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.263232+0000", 2024-09-21T14:05:23.608 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:23.608 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.608 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.608 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:05:23.608 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.263232+0000", 2024-09-21T14:05:23.608 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:23.608 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.608 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.609 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:05:23.609 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.263233+0000", 2024-09-21T14:05:23.609 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.488e-06 2024-09-21T14:05:23.609 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.609 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.609 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:05:23.609 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.263305+0000", 2024-09-21T14:05:23.609 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1552999999999997e-05 2024-09-21T14:05:23.609 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.609 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.609 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:05:23.609 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.263319+0000", 2024-09-21T14:05:23.609 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4233000000000001e-05 2024-09-21T14:05:23.609 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.609 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.610 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:05:23.610 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.263325+0000", 2024-09-21T14:05:23.610 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9950000000000002e-06 2024-09-21T14:05:23.610 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.610 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.610 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:05:23.610 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.263389+0000", 2024-09-21T14:05:23.610 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3567999999999999e-05 2024-09-21T14:05:23.610 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.610 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.610 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:05:23.610 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.263400+0000", 2024-09-21T14:05:23.610 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1206000000000001e-05 2024-09-21T14:05:23.611 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.611 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.611 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:05:23.611 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.187432+0000", 2024-09-21T14:05:23.611 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.92403234099999998 2024-09-21T14:05:23.611 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.611 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.611 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:05:23.611 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.187440+0000", 2024-09-21T14:05:23.611 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0260000000000007e-06 2024-09-21T14:05:23.611 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.611 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.611 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:05:23.611 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.295005+0000", 2024-09-21T14:05:23.612 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1075643630000001 2024-09-21T14:05:23.612 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.612 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.612 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:05:23.612 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.295169+0000", 2024-09-21T14:05:23.612 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000163859 2024-09-21T14:05:23.612 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.612 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.612 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3,4", 2024-09-21T14:05:23.612 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.295394+0000", 2024-09-21T14:05:23.612 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000225353 2024-09-21T14:05:23.612 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.612 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.612 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:05:23.612 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.298164+0000", 2024-09-21T14:05:23.613 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0027705170000000001 2024-09-21T14:05:23.613 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.613 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.613 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:05:23.613 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.298919+0000", 2024-09-21T14:05:23.613 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00075427899999999995 2024-09-21T14:05:23.613 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.613 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.613 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:05:23.613 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.316063+0000", 2024-09-21T14:05:23.613 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.017144309 2024-09-21T14:05:23.613 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.613 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.613 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:05:23.614 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.322394+0000", 2024-09-21T14:05:23.614 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0063310720000000001 2024-09-21T14:05:23.614 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.614 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.614 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:05:23.614 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.322436+0000", 2024-09-21T14:05:23.614 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1897999999999998e-05 2024-09-21T14:05:23.614 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.614 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.614 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:05:23.614 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.322506+0000", 2024-09-21T14:05:23.614 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.0203999999999994e-05 2024-09-21T14:05:23.614 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:23.614 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:05:23.615 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:23.615 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.615 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.615 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:112256 10.a 10:5949a1c7:::benchmark_data_smithi049_176811_object112255:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-21T14:05:23.615 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:04:20.267531+0000", 2024-09-21T14:05:23.615 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 62.398598548000002, 2024-09-21T14:05:23.615 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.141982547, 2024-09-21T14:05:23.615 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:05:23.615 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:05:23.615 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:05:23.615 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:05:23.615 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:05:23.615 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:05:23.615 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 112256 2024-09-21T14:05:23.616 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.616 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:05:23.616 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.616 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:05:23.616 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.267531+0000", 2024-09-21T14:05:23.616 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:23.616 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.616 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.616 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:05:23.616 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.267531+0000", 2024-09-21T14:05:23.616 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:23.616 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.616 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.616 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:05:23.617 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.267531+0000", 2024-09-21T14:05:23.617 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1999999999999998e-07 2024-09-21T14:05:23.617 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.617 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.617 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:05:23.617 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.267570+0000", 2024-09-21T14:05:23.617 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8336e-05 2024-09-21T14:05:23.617 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.617 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.617 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:05:23.617 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.267579+0000", 2024-09-21T14:05:23.617 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.9930000000000001e-06 2024-09-21T14:05:23.617 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.617 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.617 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:05:23.618 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.267583+0000", 2024-09-21T14:05:23.618 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3830000000000002e-06 2024-09-21T14:05:23.618 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.618 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.618 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:05:23.618 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.267632+0000", 2024-09-21T14:05:23.618 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8822000000000001e-05 2024-09-21T14:05:23.618 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.618 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.618 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:05:23.618 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.267640+0000", 2024-09-21T14:05:23.618 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2300000000000008e-06 2024-09-21T14:05:23.618 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.619 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.619 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:05:23.619 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.188439+0000", 2024-09-21T14:05:23.619 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.92079873300000004 2024-09-21T14:05:23.619 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.619 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.619 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:05:23.619 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.188446+0000", 2024-09-21T14:05:23.619 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.255e-06 2024-09-21T14:05:23.619 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.619 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.619 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:05:23.619 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.388852+0000", 2024-09-21T14:05:23.619 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.200405897 2024-09-21T14:05:23.619 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.620 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.620 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:05:23.620 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.389001+0000", 2024-09-21T14:05:23.620 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000148641 2024-09-21T14:05:23.620 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.620 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.620 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2,5", 2024-09-21T14:05:23.620 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.389171+0000", 2024-09-21T14:05:23.620 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00017001399999999999 2024-09-21T14:05:23.620 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.620 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.620 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:05:23.620 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.396980+0000", 2024-09-21T14:05:23.620 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0078089149999999996 2024-09-21T14:05:23.621 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.621 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.621 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:05:23.621 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.397128+0000", 2024-09-21T14:05:23.621 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014819799999999999 2024-09-21T14:05:23.621 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.621 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.621 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:05:23.621 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.399986+0000", 2024-09-21T14:05:23.621 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0028577210000000001 2024-09-21T14:05:23.621 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.621 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.621 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:05:23.621 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.409327+0000", 2024-09-21T14:05:23.622 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0093417110000000008 2024-09-21T14:05:23.622 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.622 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.622 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:05:23.622 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.409390+0000", 2024-09-21T14:05:23.622 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2636000000000001e-05 2024-09-21T14:05:23.622 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.622 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.622 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:05:23.622 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.409513+0000", 2024-09-21T14:05:23.622 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012314199999999999 2024-09-21T14:05:23.622 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:23.622 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:05:23.623 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:23.623 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.623 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.623 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:112262 10.a 10:5144907d:::benchmark_data_smithi049_176811_object112261:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-21T14:05:23.623 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:04:20.285420+0000", 2024-09-21T14:05:23.623 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 62.380708789000003, 2024-09-21T14:05:23.623 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1243977709999999, 2024-09-21T14:05:23.623 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:05:23.623 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:05:23.623 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:05:23.623 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:05:23.623 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:05:23.623 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:05:23.623 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 112262 2024-09-21T14:05:23.623 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.624 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:05:23.624 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.624 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:05:23.624 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.285420+0000", 2024-09-21T14:05:23.624 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:23.624 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.624 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.624 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:05:23.624 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.285420+0000", 2024-09-21T14:05:23.624 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:05:23.624 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.624 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.624 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:05:23.624 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.285422+0000", 2024-09-21T14:05:23.625 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4449999999999999e-06 2024-09-21T14:05:23.625 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.625 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.625 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:05:23.625 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.285450+0000", 2024-09-21T14:05:23.625 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7985999999999999e-05 2024-09-21T14:05:23.625 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.625 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.625 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:05:23.625 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.285462+0000", 2024-09-21T14:05:23.625 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.185e-05 2024-09-21T14:05:23.625 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.625 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.625 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:05:23.626 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.285468+0000", 2024-09-21T14:05:23.626 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3570000000000003e-06 2024-09-21T14:05:23.626 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.626 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.626 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:05:23.626 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.285519+0000", 2024-09-21T14:05:23.626 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0912999999999999e-05 2024-09-21T14:05:23.626 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.626 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.626 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:05:23.626 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.285526+0000", 2024-09-21T14:05:23.626 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1069999999999999e-06 2024-09-21T14:05:23.626 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.626 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.627 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:05:23.627 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.188522+0000", 2024-09-21T14:05:23.627 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.90299566200000003 2024-09-21T14:05:23.627 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.627 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.627 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:05:23.627 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.188533+0000", 2024-09-21T14:05:23.627 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0903999999999999e-05 2024-09-21T14:05:23.627 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.627 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.627 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:05:23.627 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.391763+0000", 2024-09-21T14:05:23.627 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2032303799999999 2024-09-21T14:05:23.627 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.628 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.628 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:05:23.628 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.391895+0000", 2024-09-21T14:05:23.628 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013225300000000001 2024-09-21T14:05:23.628 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.628 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.628 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2,5", 2024-09-21T14:05:23.628 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.392058+0000", 2024-09-21T14:05:23.628 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000162619 2024-09-21T14:05:23.628 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.628 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.628 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:05:23.628 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.397025+0000", 2024-09-21T14:05:23.628 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0049667569999999996 2024-09-21T14:05:23.629 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.629 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.629 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:05:23.629 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.402990+0000", 2024-09-21T14:05:23.629 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0059658200000000002 2024-09-21T14:05:23.629 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.629 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.629 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:05:23.629 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.403310+0000", 2024-09-21T14:05:23.629 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00031964599999999998 2024-09-21T14:05:23.629 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.629 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.629 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:05:23.629 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.409700+0000", 2024-09-21T14:05:23.629 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0063900240000000002 2024-09-21T14:05:23.630 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.630 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.630 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:05:23.630 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.409743+0000", 2024-09-21T14:05:23.630 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2670000000000003e-05 2024-09-21T14:05:23.630 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:05:23.630 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:05:23.630 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:05:23.630 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.409818+0000", 2024-09-21T14:05:23.630 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5377999999999996e-05 2024-09-21T14:05:23.630 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:23.630 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:05:23.630 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:23.630 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:05:23.631 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:05:23.631 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T14:05:23.631 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:05:23.682 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:23.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:23.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:05:23.784 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:05:23.815 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:23.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:23.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:05:23.917 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:05:23.955 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:23.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:23.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:05:24.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:23 smithi049 ceph-mon[30080]: pgmap v831: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 762 op/s; 5553/253122 objects misplaced (2.194%); 13 MiB/s, 215 objects/s recovering 2024-09-21T14:05:24.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:23 smithi049 ceph-mon[30080]: 3.c scrub starts 2024-09-21T14:05:24.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:23 smithi049 ceph-mon[30080]: 3.c scrub ok 2024-09-21T14:05:24.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:23 smithi049 ceph-mon[40489]: pgmap v831: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 762 op/s; 5553/253122 objects misplaced (2.194%); 13 MiB/s, 215 objects/s recovering 2024-09-21T14:05:24.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:23 smithi049 ceph-mon[40489]: 3.c scrub starts 2024-09-21T14:05:24.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:23 smithi049 ceph-mon[40489]: 3.c scrub ok 2024-09-21T14:05:24.056 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:24.091 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:24.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:24.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:24.193 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:05:24.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:23 smithi187 ceph-mon[33032]: pgmap v831: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 762 op/s; 5553/253122 objects misplaced (2.194%); 13 MiB/s, 215 objects/s recovering 2024-09-21T14:05:24.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:23 smithi187 ceph-mon[33032]: 3.c scrub starts 2024-09-21T14:05:24.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:23 smithi187 ceph-mon[33032]: 3.c scrub ok 2024-09-21T14:05:24.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:24.226 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:24.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:05:24.327 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:05:24.361 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:24.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:24.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:05:24.462 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:24.471 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T14:05:24.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:24.501 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:24.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:24.602 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:05:24.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:24.642 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:24.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:05:24.743 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:05:24.776 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:24.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:24.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:05:24.877 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:05:24.878 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:05:24.910 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-09-21T14:05:24.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:24.921 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:24.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:05:25.022 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:05:25.065 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:25.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:25.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:05:25.167 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:25.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:24 smithi187 ceph-mon[33032]: 4.9 scrub starts 2024-09-21T14:05:25.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:24 smithi187 ceph-mon[33032]: 4.9 scrub ok 2024-09-21T14:05:25.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:24 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/471138688' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:05:25.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:24 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:05:25.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:24 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:05:25.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:24 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:05:25.205 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:25.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:25.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:25.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:24 smithi049 ceph-mon[40489]: 4.9 scrub starts 2024-09-21T14:05:25.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:24 smithi049 ceph-mon[40489]: 4.9 scrub ok 2024-09-21T14:05:25.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:24 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/471138688' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:05:25.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:24 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:05:25.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:24 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:05:25.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:24 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:05:25.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:24 smithi049 ceph-mon[30080]: 4.9 scrub starts 2024-09-21T14:05:25.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:24 smithi049 ceph-mon[30080]: 4.9 scrub ok 2024-09-21T14:05:25.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:24 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/471138688' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:05:25.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:24 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:05:25.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:24 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:05:25.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:24 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:05:25.307 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:05:25.347 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:25.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:25.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:05:25.449 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:05:25.478 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:25.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:25.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:05:25.580 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:05:25.605 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:25.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:25.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:05:25.706 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:05:25.726 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:05:25.726 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:05:25.732 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:25.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:25.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:05:25.746 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:05:25.747 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:05:25.766 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:05:25.766 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:05:25.785 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:05:25.785 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:05:25.805 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:05:25.805 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:05:25.824 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:05:25.824 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:05:25.834 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:05:25.844 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:05:25.844 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:05:25.858 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:25.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:25.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:05:25.864 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:05:25.864 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:05:25.960 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:05:25.984 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:25.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:25.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:05:26.086 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:05:26.108 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:26.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:26.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:05:26.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:25 smithi187 ceph-mon[33032]: pgmap v832: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 4.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s; 3715/242505 objects misplaced (1.532%); 26 MiB/s, 410 objects/s recovering 2024-09-21T14:05:26.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:25 smithi187 ceph-mon[33032]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:05:26.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:25 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T14:05:26.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:25 smithi187 ceph-mon[33032]: osdmap e426: 8 total, 8 up, 8 in 2024-09-21T14:05:26.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:25 smithi187 ceph-mon[33032]: osdmap e427: 8 total, 8 up, 8 in 2024-09-21T14:05:26.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:25 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/154466258' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:05:26.211 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:05:26.234 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:26.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:26.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:05:26.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:25 smithi049 ceph-mon[40489]: pgmap v832: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 4.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s; 3715/242505 objects misplaced (1.532%); 26 MiB/s, 410 objects/s recovering 2024-09-21T14:05:26.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:25 smithi049 ceph-mon[40489]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:05:26.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:25 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T14:05:26.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:25 smithi049 ceph-mon[40489]: osdmap e426: 8 total, 8 up, 8 in 2024-09-21T14:05:26.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:25 smithi049 ceph-mon[40489]: osdmap e427: 8 total, 8 up, 8 in 2024-09-21T14:05:26.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:25 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/154466258' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:05:26.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:25 smithi049 ceph-mon[30080]: pgmap v832: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 4.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s; 3715/242505 objects misplaced (1.532%); 26 MiB/s, 410 objects/s recovering 2024-09-21T14:05:26.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:25 smithi049 ceph-mon[30080]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:05:26.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:25 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T14:05:26.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:25 smithi049 ceph-mon[30080]: osdmap e426: 8 total, 8 up, 8 in 2024-09-21T14:05:26.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:25 smithi049 ceph-mon[30080]: osdmap e427: 8 total, 8 up, 8 in 2024-09-21T14:05:26.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:25 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/154466258' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:05:26.336 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:26.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:26.369 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:26.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:26.470 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:26.507 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:26.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:26.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:26.609 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:05:26.644 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:26.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:26.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:05:26.746 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:05:26.777 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:26.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:26.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:05:26.879 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:05:26.919 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:26.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:26.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:05:27.020 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:27.059 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:27.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:27.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:27.161 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:27.192 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:27.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:27.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:27.294 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:05:27.338 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:27.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:27.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:05:27.440 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:05:27.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:27 smithi187 ceph-mon[33032]: pgmap v835: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 4.9 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 950 op/s; 3715/238986 objects misplaced (1.554%); 16 MiB/s, 260 objects/s recovering 2024-09-21T14:05:27.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:27 smithi187 ceph-mon[33032]: osdmap e428: 8 total, 8 up, 8 in 2024-09-21T14:05:27.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:27.490 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:27.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:05:27.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:27 smithi049 ceph-mon[40489]: pgmap v835: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 4.9 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 950 op/s; 3715/238986 objects misplaced (1.554%); 16 MiB/s, 260 objects/s recovering 2024-09-21T14:05:27.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:27 smithi049 ceph-mon[40489]: osdmap e428: 8 total, 8 up, 8 in 2024-09-21T14:05:27.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:27 smithi049 ceph-mon[30080]: pgmap v835: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 4.9 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 950 op/s; 3715/238986 objects misplaced (1.554%); 16 MiB/s, 260 objects/s recovering 2024-09-21T14:05:27.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:27 smithi049 ceph-mon[30080]: osdmap e428: 8 total, 8 up, 8 in 2024-09-21T14:05:27.592 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:05:27.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:27.627 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:27.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:05:27.728 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:05:27.759 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:27.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:27.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:05:27.860 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:05:27.884 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:27.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:27.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:05:27.986 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:05:28.022 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:28.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:28.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:05:28.124 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:05:28.156 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:28.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:28.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:05:28.257 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:28.291 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 4, 6, 2, 3, 7, 5] 2024-09-21T14:05:28.291 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T14:05:28.291 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-21T14:05:28.292 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T14:05:28.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:28.296 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:28.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:28.393 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T14:05:28.396 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:05:28.423 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:28.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:28.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:05:28.525 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:05:28.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:28.590 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:28.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:05:28.691 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:05:28.741 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:28.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:28.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:05:28.761 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:05:28.843 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:28.879 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:28.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:28.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:28.981 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:05:29.025 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:29.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:29.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:05:29.127 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:05:29.163 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:29.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:29.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:05:29.264 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:29.301 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:29.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:29.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:29.403 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:05:29.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:29.442 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:29.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:05:29.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:29 smithi187 ceph-mon[33032]: pgmap v837: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 925 op/s; 3715/236466 objects misplaced (1.571%); 16 MiB/s, 260 objects/s recovering 2024-09-21T14:05:29.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:29 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.b", "id": [3, 1, 7, 6]} : dispatch 2024-09-21T14:05:29.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:29 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1f", "id": [3, 0]} : dispatch 2024-09-21T14:05:29.453 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T14:05:29.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:29 smithi049 ceph-mon[40489]: pgmap v837: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 925 op/s; 3715/236466 objects misplaced (1.571%); 16 MiB/s, 260 objects/s recovering 2024-09-21T14:05:29.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:29 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.b", "id": [3, 1, 7, 6]} : dispatch 2024-09-21T14:05:29.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:29 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1f", "id": [3, 0]} : dispatch 2024-09-21T14:05:29.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:29 smithi049 ceph-mon[30080]: pgmap v837: 158 pgs: 3 active+remapped+backfilling, 155 active+clean; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 925 op/s; 3715/236466 objects misplaced (1.571%); 16 MiB/s, 260 objects/s recovering 2024-09-21T14:05:29.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:29 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.b", "id": [3, 1, 7, 6]} : dispatch 2024-09-21T14:05:29.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:29 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1f", "id": [3, 0]} : dispatch 2024-09-21T14:05:29.543 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:05:29.576 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:29.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:29.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:05:29.677 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:05:29.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:29.712 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:29.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:05:29.813 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:05:29.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:29.868 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:29.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:05:29.969 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:05:29.999 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:30.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:30.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:05:30.101 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:05:30.111 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:05:30.155 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:30.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:30.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:05:30.188 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:05:30.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:05:29 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:05:30.229 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T14:05:30.260 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:05:30.290 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:30.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:30.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:05:30.392 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:05:30.457 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:30 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.b", "id": [3, 1, 7, 6]}]': finished 2024-09-21T14:05:30.457 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:30 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1f", "id": [3, 0]}]': finished 2024-09-21T14:05:30.457 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:30 smithi049 ceph-mon[30080]: osdmap e429: 8 total, 8 up, 8 in 2024-09-21T14:05:30.457 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:30 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:05:30.458 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:30 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.b", "id": [3, 1, 7, 6]}]': finished 2024-09-21T14:05:30.458 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:30 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1f", "id": [3, 0]}]': finished 2024-09-21T14:05:30.458 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:30 smithi049 ceph-mon[40489]: osdmap e429: 8 total, 8 up, 8 in 2024-09-21T14:05:30.458 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:30 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:05:30.494 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:30.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:30.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:05:30.527 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T14:05:30.529 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-09-21T14:05:30.595 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:30.632 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:30.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:30.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:30.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:30 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.b", "id": [3, 1, 7, 6]}]': finished 2024-09-21T14:05:30.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:30 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1f", "id": [3, 0]}]': finished 2024-09-21T14:05:30.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:30 smithi187 ceph-mon[33032]: osdmap e429: 8 total, 8 up, 8 in 2024-09-21T14:05:30.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:30 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:05:30.734 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:05:30.754 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:05:30 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:05:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:05:30.775 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:30.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:30.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:05:30.877 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:05:30.911 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:30.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:30.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:05:31.013 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:05:31.049 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:31.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:31.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:05:31.151 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:31.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:31.182 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:31.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:31.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:31 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:05:31.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:31 smithi049 ceph-mon[30080]: pgmap v839: 158 pgs: 1 peering, 2 active+remapped+backfilling, 155 active+clean; 4.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 993 B/s rd, 0 B/s wr, 902 op/s; 1579/226614 objects misplaced (0.697%); 24 MiB/s, 386 objects/s recovering 2024-09-21T14:05:31.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:31 smithi049 ceph-mon[30080]: osdmap e430: 8 total, 8 up, 8 in 2024-09-21T14:05:31.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:31 smithi049 ceph-mon[30080]: from='client.26020 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:05:31.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:31 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:05:31.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:31 smithi049 ceph-mon[40489]: pgmap v839: 158 pgs: 1 peering, 2 active+remapped+backfilling, 155 active+clean; 4.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 993 B/s rd, 0 B/s wr, 902 op/s; 1579/226614 objects misplaced (0.697%); 24 MiB/s, 386 objects/s recovering 2024-09-21T14:05:31.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:31 smithi049 ceph-mon[40489]: osdmap e430: 8 total, 8 up, 8 in 2024-09-21T14:05:31.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:31 smithi049 ceph-mon[40489]: from='client.26020 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:05:31.283 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:31.313 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:31.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:31.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:31.414 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:05:31.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:31.454 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:31.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:05:31.555 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:05:31.618 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:31.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:31.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:05:31.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:31 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:05:31.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:31 smithi187 ceph-mon[33032]: pgmap v839: 158 pgs: 1 peering, 2 active+remapped+backfilling, 155 active+clean; 4.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 993 B/s rd, 0 B/s wr, 902 op/s; 1579/226614 objects misplaced (0.697%); 24 MiB/s, 386 objects/s recovering 2024-09-21T14:05:31.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:31 smithi187 ceph-mon[33032]: osdmap e430: 8 total, 8 up, 8 in 2024-09-21T14:05:31.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:31 smithi187 ceph-mon[33032]: from='client.26020 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:05:31.719 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:05:31.765 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:31.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:31.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:05:31.867 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:05:31.899 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:31.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:31.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:05:31.935 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T14:05:31.935 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:05:32.001 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:32.035 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:32.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:32.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:32.137 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:05:32.172 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:32.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:32.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:05:32.256 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-09-21T14:05:32.274 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:05:32.318 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:32.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:32.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:05:32.421 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:05:32.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:32.470 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:32.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:05:32.497 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:32 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3542378480' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:05:32.497 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:32 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:05:32.497 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:32 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3542378480' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:05:32.497 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:32 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:05:32.498 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":840,"stamp":"2024-09-21T14:05:30.223308+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4937977636,"num_objects":75538,"num_object_clones":0,"num_object_copies":226614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1579,"num_objects_unfound":0,"num_objects_dirty":75538,"num_whiteouts":0,"num_read":1500,"num_read_kb":1279,"num_write":180567,"num_write_kb":8165226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22343,"num_bytes_recovered":1450803371,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85065,"ondisk_log_size":85065,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":479,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":19723784,"kb_used_data":15694124,"kb_used_omap":23,"kb_used_meta":4029480,"kb_avail":730237432,"statfs":{"total":767960285184,"available":747763130368,"internally_reserved":0,"allocated":16070782976,"data_stored":16068035079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24148,"internal_metadata":4126187948},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,5],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-304414720,"num_objects":-4645,"num_object_clones":0,"num_object_copies":-13935,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2136,"num_objects_unfound":0,"num_objects_dirty":-4645,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4645,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1993,"num_bytes_recovered":130613248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.067888"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":694,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T14:05:25.163658+0000","last_change":"2024-09-21T14:05:10.397918+0000","last_active":"2024-09-21T14:05:25.163658+0000","last_peered":"2024-09-21T14:05:25.163658+0000","last_clean":"2024-09-21T14:05:25.163658+0000","last_became_active":"2024-09-21T14:04:21.204299+0000","last_became_peered":"2024-09-21T14:04:21.204299+0000","last_unstale":"2024-09-21T14:05:25.163658+0000","last_undegraded":"2024-09-21T14:05:25.163658+0000","last_fullsized":"2024-09-21T14:05:25.163658+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:10.397765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:05:10.397765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:40.202681+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"290'15","reported_seq":848,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.248259+0000","last_change":"2024-09-21T14:01:34.785322+0000","last_active":"2024-09-21T14:05:26.248259+0000","last_peered":"2024-09-21T14:05:26.248259+0000","last_clean":"2024-09-21T14:05:26.248259+0000","last_became_active":"2024-09-21T13:58:17.771220+0000","last_became_peered":"2024-09-21T13:58:17.771220+0000","last_unstale":"2024-09-21T14:05:26.248259+0000","last_undegraded":"2024-09-21T14:05:26.248259+0000","last_fullsized":"2024-09-21T14:05:26.248259+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'15","last_scrub_stamp":"2024-09-21T14:01:34.785267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:34.785267+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":799,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-09-21T14:05:29.212090+0000","last_change":"2024-09-21T14:03:33.301558+0000","last_active":"2024-09-21T14:05:29.212090+0000","last_peered":"2024-09-21T14:05:29.212090+0000","last_clean":"2024-09-21T14:05:29.212090+0000","last_became_active":"2024-09-21T14:01:41.202168+0000","last_became_peered":"2024-09-21T14:01:41.202168+0000","last_unstale":"2024-09-21T14:05:29.212090+0000","last_undegraded":"2024-09-21T14:05:29.212090+0000","last_fullsized":"2024-09-21T14:05:29.212090+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:33.301487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:33.301487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":773,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.248257+0000","last_change":"2024-09-21T14:05:14.348636+0000","last_active":"2024-09-21T14:05:26.248257+0000","last_peered":"2024-09-21T14:05:26.248257+0000","last_clean":"2024-09-21T14:05:26.248257+0000","last_became_active":"2024-09-21T14:01:41.203275+0000","last_became_peered":"2024-09-21T14:01:41.203275+0000","last_unstale":"2024-09-21T14:05:26.248257+0000","last_undegraded":"2024-09-21T14:05:26.248257+0000","last_fullsized":"2024-09-21T14:05:26.248257+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:14.348563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:05:14.348563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:30.470326+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":743,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.169045+0000","last_change":"2024-09-21T14:04:00.241774+0000","last_active":"2024-09-21T14:05:26.169045+0000","last_peered":"2024-09-21T14:05:26.169045+0000","last_clean":"2024-09-21T14:05:26.169045+0000","last_became_active":"2024-09-21T13:59:32.203340+0000","last_became_peered":"2024-09-21T13:59:32.203340+0000","last_unstale":"2024-09-21T14:05:26.169045+0000","last_undegraded":"2024-09-21T14:05:26.169045+0000","last_fullsized":"2024-09-21T14:05:26.169045+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:00.241562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:04:00.241562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":748,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.169222+0000","last_change":"2024-09-21T13:58:17.778322+0000","last_active":"2024-09-21T14:05:26.169222+0000","last_peered":"2024-09-21T14:05:26.169222+0000","last_clean":"2024-09-21T14:05:26.169222+0000","last_became_active":"2024-09-21T13:58:17.777893+0000","last_became_peered":"2024-09-21T13:58:17.777893+0000","last_unstale":"2024-09-21T14:05:26.169222+0000","last_undegraded":"2024-09-21T14:05:26.169222+0000","last_fullsized":"2024-09-21T14:05:26.169222+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:07.001135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:55:07.001135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"290'27","reported_seq":729,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.168297+0000","last_change":"2024-09-21T14:04:08.320458+0000","last_active":"2024-09-21T14:05:26.168297+0000","last_peered":"2024-09-21T14:05:26.168297+0000","last_clean":"2024-09-21T14:05:26.168297+0000","last_became_active":"2024-09-21T13:57:12.703469+0000","last_became_peered":"2024-09-21T13:57:12.703469+0000","last_unstale":"2024-09-21T14:05:26.168297+0000","last_undegraded":"2024-09-21T14:05:26.168297+0000","last_fullsized":"2024-09-21T14:05:26.168297+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'27","last_scrub_stamp":"2024-09-21T14:04:08.320339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:04:08.320339+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:32.005889+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":502,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.190130+0000","last_change":"2024-09-21T14:04:35.907676+0000","last_active":"2024-09-21T14:05:26.190130+0000","last_peered":"2024-09-21T14:05:26.190130+0000","last_clean":"2024-09-21T14:05:26.190130+0000","last_became_active":"2024-09-21T14:04:21.208237+0000","last_became_peered":"2024-09-21T14:04:21.208237+0000","last_unstale":"2024-09-21T14:05:26.190130+0000","last_undegraded":"2024-09-21T14:05:26.190130+0000","last_fullsized":"2024-09-21T14:05:26.190130+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:35.907611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:04:35.907611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:39.685760+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":746,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T14:05:25.162055+0000","last_change":"2024-09-21T14:04:11.718218+0000","last_active":"2024-09-21T14:05:25.162055+0000","last_peered":"2024-09-21T14:05:25.162055+0000","last_clean":"2024-09-21T14:05:25.162055+0000","last_became_active":"2024-09-21T14:01:41.194517+0000","last_became_peered":"2024-09-21T14:01:41.194517+0000","last_unstale":"2024-09-21T14:05:25.162055+0000","last_undegraded":"2024-09-21T14:05:25.162055+0000","last_fullsized":"2024-09-21T14:05:25.162055+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:11.718143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:11.718143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"290'15","reported_seq":587,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.168362+0000","last_change":"2024-09-21T14:04:21.228412+0000","last_active":"2024-09-21T14:05:26.168362+0000","last_peered":"2024-09-21T14:05:26.168362+0000","last_clean":"2024-09-21T14:05:26.168362+0000","last_became_active":"2024-09-21T14:04:21.208244+0000","last_became_peered":"2024-09-21T14:04:21.208244+0000","last_unstale":"2024-09-21T14:05:26.168362+0000","last_undegraded":"2024-09-21T14:05:26.168362+0000","last_fullsized":"2024-09-21T14:05:26.168362+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'15","last_scrub_stamp":"2024-09-21T14:00:02.475661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:00:02.475661+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":695,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T14:05:25.160962+0000","last_change":"2024-09-21T14:05:12.369047+0000","last_active":"2024-09-21T14:05:25.160962+0000","last_peered":"2024-09-21T14:05:25.160962+0000","last_clean":"2024-09-21T14:05:25.160962+0000","last_became_active":"2024-09-21T14:01:41.188248+0000","last_became_peered":"2024-09-21T14:01:41.188248+0000","last_unstale":"2024-09-21T14:05:25.160962+0000","last_undegraded":"2024-09-21T14:05:25.160962+0000","last_fullsized":"2024-09-21T14:05:25.160962+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:12.368984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:05:12.368984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:27.784175+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":736,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-09-21T14:05:29.211957+0000","last_change":"2024-09-21T14:05:17.384328+0000","last_active":"2024-09-21T14:05:29.211957+0000","last_peered":"2024-09-21T14:05:29.211957+0000","last_clean":"2024-09-21T14:05:29.211957+0000","last_became_active":"2024-09-21T14:04:21.205416+0000","last_became_peered":"2024-09-21T14:04:21.205416+0000","last_unstale":"2024-09-21T14:05:29.211957+0000","last_undegraded":"2024-09-21T14:05:29.211957+0000","last_fullsized":"2024-09-21T14:05:29.211957+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:17.384142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:05:17.384142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:39.285417+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":723,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T14:05:25.160939+0000","last_change":"2024-09-21T14:04:10.722881+0000","last_active":"2024-09-21T14:05:25.160939+0000","last_peered":"2024-09-21T14:05:25.160939+0000","last_clean":"2024-09-21T14:05:25.160939+0000","last_became_active":"2024-09-21T14:01:41.189373+0000","last_became_peered":"2024-09-21T14:01:41.189373+0000","last_unstale":"2024-09-21T14:05:25.160939+0000","last_undegraded":"2024-09-21T14:05:25.160939+0000","last_fullsized":"2024-09-21T14:05:25.160939+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:10.722799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:10.722799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:25.238119+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"290'7","reported_seq":587,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.168862+0000","last_change":"2024-09-21T14:04:21.231643+0000","last_active":"2024-09-21T14:05:26.168862+0000","last_peered":"2024-09-21T14:05:26.168862+0000","last_clean":"2024-09-21T14:05:26.168862+0000","last_became_active":"2024-09-21T14:04:21.213102+0000","last_became_peered":"2024-09-21T14:04:21.213102+0000","last_unstale":"2024-09-21T14:05:26.168862+0000","last_undegraded":"2024-09-21T14:05:26.168862+0000","last_fullsized":"2024-09-21T14:05:26.168862+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'7","last_scrub_stamp":"2024-09-21T14:01:57.563184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:57.563184+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":710,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.190862+0000","last_change":"2024-09-21T13:59:32.204565+0000","last_active":"2024-09-21T14:05:26.190862+0000","last_peered":"2024-09-21T14:05:26.190862+0000","last_clean":"2024-09-21T14:05:26.190862+0000","last_became_active":"2024-09-21T13:59:32.204174+0000","last_became_peered":"2024-09-21T13:59:32.204174+0000","last_unstale":"2024-09-21T14:05:26.190862+0000","last_undegraded":"2024-09-21T14:05:26.190862+0000","last_fullsized":"2024-09-21T14:05:26.190862+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:06.722293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:06.722293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":757,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T14:05:25.160406+0000","last_change":"2024-09-21T14:04:46.471869+0000","last_active":"2024-09-21T14:05:25.160406+0000","last_peered":"2024-09-21T14:05:25.160406+0000","last_clean":"2024-09-21T14:05:25.160406+0000","last_became_active":"2024-09-21T14:01:41.198672+0000","last_became_peered":"2024-09-21T14:01:41.198672+0000","last_unstale":"2024-09-21T14:05:25.160406+0000","last_undegraded":"2024-09-21T14:05:25.160406+0000","last_fullsized":"2024-09-21T14:05:25.160406+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:46.471817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:04:46.471817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:01.005314+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":816,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.248909+0000","last_change":"2024-09-21T14:03:36.018910+0000","last_active":"2024-09-21T14:05:26.248909+0000","last_peered":"2024-09-21T14:05:26.248909+0000","last_clean":"2024-09-21T14:05:26.248909+0000","last_became_active":"2024-09-21T14:01:41.200360+0000","last_became_peered":"2024-09-21T14:01:41.200360+0000","last_unstale":"2024-09-21T14:05:26.248909+0000","last_undegraded":"2024-09-21T14:05:26.248909+0000","last_fullsized":"2024-09-21T14:05:26.248909+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:36.018851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:36.018851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":732,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.248035+0000","last_change":"2024-09-21T14:04:21.202574+0000","last_active":"2024-09-21T14:05:26.248035+0000","last_peered":"2024-09-21T14:05:26.248035+0000","last_clean":"2024-09-21T14:05:26.248035+0000","last_became_active":"2024-09-21T14:04:21.202178+0000","last_became_peered":"2024-09-21T14:04:21.202178+0000","last_unstale":"2024-09-21T14:05:26.248035+0000","last_undegraded":"2024-09-21T14:05:26.248035+0000","last_fullsized":"2024-09-21T14:05:26.248035+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:36.679321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:02:36.679321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"290'27","reported_seq":798,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.191509+0000","last_change":"2024-09-21T14:04:01.621388+0000","last_active":"2024-09-21T14:05:26.191509+0000","last_peered":"2024-09-21T14:05:26.191509+0000","last_clean":"2024-09-21T14:05:26.191509+0000","last_became_active":"2024-09-21T13:57:12.680783+0000","last_became_peered":"2024-09-21T13:57:12.680783+0000","last_unstale":"2024-09-21T14:05:26.191509+0000","last_undegraded":"2024-09-21T14:05:26.191509+0000","last_fullsized":"2024-09-21T14:05:26.191509+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'27","last_scrub_stamp":"2024-09-21T14:04:01.621303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:04:01.621303+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":792,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-09-21T14:05:29.212648+0000","last_change":"2024-09-21T14:04:21.204283+0000","last_active":"2024-09-21T14:05:29.212648+0000","last_peered":"2024-09-21T14:05:29.212648+0000","last_clean":"2024-09-21T14:05:29.212648+0000","last_became_active":"2024-09-21T14:04:21.203937+0000","last_became_peered":"2024-09-21T14:04:21.203937+0000","last_unstale":"2024-09-21T14:05:29.212648+0000","last_undegraded":"2024-09-21T14:05:29.212648+0000","last_fullsized":"2024-09-21T14:05:29.212648+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:08.299252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:08.299252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":2,"reported_epoch":429,"state":"peering","last_fresh":"2024-09-21T14:05:26.168916+0000","last_change":"2024-09-21T14:05:29.546886+0000","last_active":"2024-09-21T14:05:29.546886+0000","last_peered":"2024-09-21T14:05:26.168916+0000","last_clean":"2024-09-21T14:05:26.168916+0000","last_became_active":"2024-09-21T14:01:41.191116+0000","last_became_peered":"2024-09-21T14:01:41.191116+0000","last_unstale":"2024-09-21T14:05:26.168916+0000","last_undegraded":"2024-09-21T14:05:26.168916+0000","last_fullsized":"2024-09-21T14:05:26.168916+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:57.962266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:57.962266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":785,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.249190+0000","last_change":"2024-09-21T14:00:51.495433+0000","last_active":"2024-09-21T14:05:26.249190+0000","last_peered":"2024-09-21T14:05:26.249190+0000","last_clean":"2024-09-21T14:05:26.249190+0000","last_became_active":"2024-09-21T13:52:04.316016+0000","last_became_peered":"2024-09-21T13:52:04.316016+0000","last_unstale":"2024-09-21T14:05:26.249190+0000","last_undegraded":"2024-09-21T14:05:26.249190+0000","last_fullsized":"2024-09-21T14:05:26.249190+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:00:51.495368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:00:51.495368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"290'18","reported_seq":741,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.168259+0000","last_change":"2024-09-21T14:04:50.393599+0000","last_active":"2024-09-21T14:05:26.168259+0000","last_peered":"2024-09-21T14:05:26.168259+0000","last_clean":"2024-09-21T14:05:26.168259+0000","last_became_active":"2024-09-21T13:57:12.684454+0000","last_became_peered":"2024-09-21T13:57:12.684454+0000","last_unstale":"2024-09-21T14:05:26.168259+0000","last_undegraded":"2024-09-21T14:05:26.168259+0000","last_fullsized":"2024-09-21T14:05:26.168259+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'18","last_scrub_stamp":"2024-09-21T14:04:50.393462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:04:50.393462+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:00.435535+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":780,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.248622+0000","last_change":"2024-09-21T14:03:38.065257+0000","last_active":"2024-09-21T14:05:26.248622+0000","last_peered":"2024-09-21T14:05:26.248622+0000","last_clean":"2024-09-21T14:05:26.248622+0000","last_became_active":"2024-09-21T13:57:12.681897+0000","last_became_peered":"2024-09-21T13:57:12.681897+0000","last_unstale":"2024-09-21T14:05:26.248622+0000","last_undegraded":"2024-09-21T14:05:26.248622+0000","last_fullsized":"2024-09-21T14:05:26.248622+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:38.065196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:38.065196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":702,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.191180+0000","last_change":"2024-09-21T14:03:28.933216+0000","last_active":"2024-09-21T14:05:26.191180+0000","last_peered":"2024-09-21T14:05:26.191180+0000","last_clean":"2024-09-21T14:05:26.191180+0000","last_became_active":"2024-09-21T14:01:41.181946+0000","last_became_peered":"2024-09-21T14:01:41.181946+0000","last_unstale":"2024-09-21T14:05:26.191180+0000","last_undegraded":"2024-09-21T14:05:26.191180+0000","last_fullsized":"2024-09-21T14:05:26.191180+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T14:03:28.933143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:28.933143+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":808,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-09-21T14:05:29.212187+0000","last_change":"2024-09-21T14:04:21.215322+0000","last_active":"2024-09-21T14:05:29.212187+0000","last_peered":"2024-09-21T14:05:29.212187+0000","last_clean":"2024-09-21T14:05:29.212187+0000","last_became_active":"2024-09-21T14:04:21.200889+0000","last_became_peered":"2024-09-21T14:04:21.200889+0000","last_unstale":"2024-09-21T14:05:29.212187+0000","last_undegraded":"2024-09-21T14:05:29.212187+0000","last_fullsized":"2024-09-21T14:05:29.212187+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T14:01:58.994811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:01:58.994811+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"290'20","reported_seq":125,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-09-21T14:05:29.214265+0000","last_change":"2024-09-21T14:04:56.589169+0000","last_active":"2024-09-21T14:05:29.214265+0000","last_peered":"2024-09-21T14:05:29.214265+0000","last_clean":"2024-09-21T14:05:29.214265+0000","last_became_active":"2024-09-21T14:04:56.588859+0000","last_became_peered":"2024-09-21T14:04:56.588859+0000","last_unstale":"2024-09-21T14:05:29.214265+0000","last_undegraded":"2024-09-21T14:05:29.214265+0000","last_fullsized":"2024-09-21T14:05:29.214265+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'20","last_scrub_stamp":"2024-09-21T14:04:42.048082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:04:42.048082+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:51.310282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"409'13","reported_seq":455,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.215270+0000","last_change":"2024-09-21T14:04:21.222657+0000","last_active":"2024-09-21T14:05:26.215270+0000","last_peered":"2024-09-21T14:05:26.215270+0000","last_clean":"2024-09-21T14:05:26.215270+0000","last_became_active":"2024-09-21T14:04:21.206136+0000","last_became_peered":"2024-09-21T14:04:21.206136+0000","last_unstale":"2024-09-21T14:05:26.215270+0000","last_undegraded":"2024-09-21T14:05:26.215270+0000","last_fullsized":"2024-09-21T14:05:26.215270+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'12","last_scrub_stamp":"2024-09-21T13:59:36.012693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:36.012693+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"428'11444","reported_seq":11621,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:28.800901+0000","last_change":"2024-09-21T14:04:09.971242+0000","last_active":"2024-09-21T14:05:28.800901+0000","last_peered":"2024-09-21T14:05:28.800901+0000","last_clean":"2024-09-21T14:05:28.800901+0000","last_became_active":"2024-09-21T14:02:56.559536+0000","last_became_peered":"2024-09-21T14:02:56.559536+0000","last_unstale":"2024-09-21T14:05:28.800901+0000","last_undegraded":"2024-09-21T14:05:28.800901+0000","last_fullsized":"2024-09-21T14:05:28.800901+0000","mapping_epoch":381,"log_start":"404'5900","ondisk_log_start":"404'5900","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"406'6478","last_scrub_stamp":"2024-09-21T14:04:09.971039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:04:09.971039+0000","objects_scrubbed":6417,"log_size":5544,"log_dups_size":0,"ondisk_log_size":5544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1835,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":313655296,"num_objects":4786,"num_object_clones":0,"num_object_copies":14358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11444,"num_write_kb":519360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"75'79","reported_seq":196,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T14:05:25.162323+0000","last_change":"2024-09-21T14:03:59.776621+0000","last_active":"2024-09-21T14:05:25.162323+0000","last_peered":"2024-09-21T14:05:25.162323+0000","last_clean":"2024-09-21T14:05:25.162323+0000","last_became_active":"2024-09-21T14:01:41.246888+0000","last_became_peered":"2024-09-21T14:01:41.246888+0000","last_unstale":"2024-09-21T14:05:25.162323+0000","last_undegraded":"2024-09-21T14:05:25.162323+0000","last_fullsized":"2024-09-21T14:05:25.162323+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:03:59.776552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:03:59.776552+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"290'27","reported_seq":852,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-09-21T14:05:29.213111+0000","last_change":"2024-09-21T14:04:03.259647+0000","last_active":"2024-09-21T14:05:29.213111+0000","last_peered":"2024-09-21T14:05:29.213111+0000","last_clean":"2024-09-21T14:05:29.213111+0000","last_became_active":"2024-09-21T13:57:12.695609+0000","last_became_peered":"2024-09-21T13:57:12.695609+0000","last_unstale":"2024-09-21T14:05:29.213111+0000","last_undegraded":"2024-09-21T14:05:29.213111+0000","last_fullsized":"2024-09-21T14:05:29.213111+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'27","last_scrub_stamp":"2024-09-21T14:04:03.259554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:04:03.259554+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":727,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.167503+0000","last_change":"2024-09-21T14:04:01.183193+0000","last_active":"2024-09-21T14:05:26.167503+0000","last_peered":"2024-09-21T14:05:26.167503+0000","last_clean":"2024-09-21T14:05:26.167503+0000","last_became_active":"2024-09-21T13:59:32.205350+0000","last_became_peered":"2024-09-21T13:59:32.205350+0000","last_unstale":"2024-09-21T14:05:26.167503+0000","last_undegraded":"2024-09-21T14:05:26.167503+0000","last_fullsized":"2024-09-21T14:05:26.167503+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:01.183123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:01.183123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"409'11","reported_seq":835,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T14:05:25.161624+0000","last_change":"2024-09-21T14:04:21.211284+0000","last_active":"2024-09-21T14:05:25.161624+0000","last_peered":"2024-09-21T14:05:25.161624+0000","last_clean":"2024-09-21T14:05:25.161624+0000","last_became_active":"2024-09-21T14:04:21.206029+0000","last_became_peered":"2024-09-21T14:04:21.206029+0000","last_unstale":"2024-09-21T14:05:25.161624+0000","last_undegraded":"2024-09-21T14:05:25.161624+0000","last_fullsized":"2024-09-21T14:05:25.161624+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'10","last_scrub_stamp":"2024-09-21T14:02:59.831128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:02:59.831128+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:07.398197+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":659,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T14:05:25.161394+0000","last_change":"2024-09-21T14:04:47.464272+0000","last_active":"2024-09-21T14:05:25.161394+0000","last_peered":"2024-09-21T14:05:25.161394+0000","last_clean":"2024-09-21T14:05:25.161394+0000","last_became_active":"2024-09-21T14:04:21.203505+0000","last_became_peered":"2024-09-21T14:04:21.203505+0000","last_unstale":"2024-09-21T14:05:25.161394+0000","last_undegraded":"2024-09-21T14:05:25.161394+0000","last_fullsized":"2024-09-21T14:05:25.161394+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:47.464207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:04:47.464207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:48.380565+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"428'10996","reported_seq":11171,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:28.807065+0000","last_change":"2024-09-21T14:04:08.894792+0000","last_active":"2024-09-21T14:05:28.807065+0000","last_peered":"2024-09-21T14:05:28.807065+0000","last_clean":"2024-09-21T14:05:28.807065+0000","last_became_active":"2024-09-21T14:02:56.562524+0000","last_became_peered":"2024-09-21T14:02:56.562524+0000","last_unstale":"2024-09-21T14:05:28.807065+0000","last_undegraded":"2024-09-21T14:05:28.807065+0000","last_fullsized":"2024-09-21T14:05:28.807065+0000","mapping_epoch":381,"log_start":"404'5500","ondisk_log_start":"404'5500","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"406'6081","last_scrub_stamp":"2024-09-21T14:04:08.894653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:04:08.894653+0000","objects_scrubbed":6008,"log_size":5496,"log_dups_size":0,"ondisk_log_size":5496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1756,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":302383104,"num_objects":4614,"num_object_clones":0,"num_object_copies":13842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10996,"num_write_kb":499520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"75'79","reported_seq":244,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.249318+0000","last_change":"2024-09-21T14:02:06.395083+0000","last_active":"2024-09-21T14:05:26.249318+0000","last_peered":"2024-09-21T14:05:26.249318+0000","last_clean":"2024-09-21T14:05:26.249318+0000","last_became_active":"2024-09-21T14:00:41.137756+0000","last_became_peered":"2024-09-21T14:00:41.137756+0000","last_unstale":"2024-09-21T14:05:26.249318+0000","last_undegraded":"2024-09-21T14:05:26.249318+0000","last_fullsized":"2024-09-21T14:05:26.249318+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:02:06.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:02:06.395011+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"290'30","reported_seq":769,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.167603+0000","last_change":"2024-09-21T14:04:21.228589+0000","last_active":"2024-09-21T14:05:26.167603+0000","last_peered":"2024-09-21T14:05:26.167603+0000","last_clean":"2024-09-21T14:05:26.167603+0000","last_became_active":"2024-09-21T14:04:21.208982+0000","last_became_peered":"2024-09-21T14:04:21.208982+0000","last_unstale":"2024-09-21T14:05:26.167603+0000","last_undegraded":"2024-09-21T14:05:26.167603+0000","last_fullsized":"2024-09-21T14:05:26.167603+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'18","last_scrub_stamp":"2024-09-21T13:54:19.001030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:19.001030+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":679,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.169080+0000","last_change":"2024-09-21T14:04:02.214555+0000","last_active":"2024-09-21T14:05:26.169080+0000","last_peered":"2024-09-21T14:05:26.169080+0000","last_clean":"2024-09-21T14:05:26.169080+0000","last_became_active":"2024-09-21T13:59:32.211004+0000","last_became_peered":"2024-09-21T13:59:32.211004+0000","last_unstale":"2024-09-21T14:05:26.169080+0000","last_undegraded":"2024-09-21T14:05:26.169080+0000","last_fullsized":"2024-09-21T14:05:26.169080+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:02.214464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:02.214464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":789,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.247814+0000","last_change":"2024-09-21T13:59:34.293330+0000","last_active":"2024-09-21T14:05:26.247814+0000","last_peered":"2024-09-21T14:05:26.247814+0000","last_clean":"2024-09-21T14:05:26.247814+0000","last_became_active":"2024-09-21T13:59:32.220304+0000","last_became_peered":"2024-09-21T13:59:32.220304+0000","last_unstale":"2024-09-21T14:05:26.247814+0000","last_undegraded":"2024-09-21T14:05:26.247814+0000","last_fullsized":"2024-09-21T14:05:26.247814+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:34.292892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:34.292892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:25.376399+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":425,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.168464+0000","last_change":"2024-09-21T14:04:21.210131+0000","last_active":"2024-09-21T14:05:26.168464+0000","last_peered":"2024-09-21T14:05:26.168464+0000","last_clean":"2024-09-21T14:05:26.168464+0000","last_became_active":"2024-09-21T14:04:21.207782+0000","last_became_peered":"2024-09-21T14:04:21.207782+0000","last_unstale":"2024-09-21T14:05:26.168464+0000","last_undegraded":"2024-09-21T14:05:26.168464+0000","last_fullsized":"2024-09-21T14:05:26.168464+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:23.334162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:23.334162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"428'11448","reported_seq":11640,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:28.797914+0000","last_change":"2024-09-21T14:04:42.544112+0000","last_active":"2024-09-21T14:05:28.797914+0000","last_peered":"2024-09-21T14:05:28.797914+0000","last_clean":"2024-09-21T14:05:28.797914+0000","last_became_active":"2024-09-21T14:02:56.557055+0000","last_became_peered":"2024-09-21T14:02:56.557055+0000","last_unstale":"2024-09-21T14:05:28.797914+0000","last_undegraded":"2024-09-21T14:05:28.797914+0000","last_fullsized":"2024-09-21T14:05:28.797914+0000","mapping_epoch":381,"log_start":"406'6310","ondisk_log_start":"406'6310","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'8494","last_scrub_stamp":"2024-09-21T14:04:42.543986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:04:42.543986+0000","objects_scrubbed":7689,"log_size":5138,"log_dups_size":0,"ondisk_log_size":5138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:45.816257+0000","scrub_duration":2665,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":304873472,"num_objects":4652,"num_object_clones":0,"num_object_copies":13956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11448,"num_write_kb":515200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"75'79","reported_seq":255,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.168574+0000","last_change":"2024-09-21T14:04:49.393518+0000","last_active":"2024-09-21T14:05:26.168574+0000","last_peered":"2024-09-21T14:05:26.168574+0000","last_clean":"2024-09-21T14:05:26.168574+0000","last_became_active":"2024-09-21T14:01:42.173042+0000","last_became_peered":"2024-09-21T14:01:42.173042+0000","last_unstale":"2024-09-21T14:05:26.168574+0000","last_undegraded":"2024-09-21T14:05:26.168574+0000","last_fullsized":"2024-09-21T14:05:26.168574+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:04:49.393415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:04:49.393415+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:56.538057+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"290'28","reported_seq":886,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.247908+0000","last_change":"2024-09-21T14:03:27.047111+0000","last_active":"2024-09-21T14:05:26.247908+0000","last_peered":"2024-09-21T14:05:26.247908+0000","last_clean":"2024-09-21T14:05:26.247908+0000","last_became_active":"2024-09-21T14:01:41.200859+0000","last_became_peered":"2024-09-21T14:01:41.200859+0000","last_unstale":"2024-09-21T14:05:26.247908+0000","last_undegraded":"2024-09-21T14:05:26.247908+0000","last_fullsized":"2024-09-21T14:05:26.247908+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'28","last_scrub_stamp":"2024-09-21T14:03:27.046906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:03:27.046906+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":66,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-09-21T14:05:29.212913+0000","last_change":"2024-09-21T14:04:21.216890+0000","last_active":"2024-09-21T14:05:29.212913+0000","last_peered":"2024-09-21T14:05:29.212913+0000","last_clean":"2024-09-21T14:05:29.212913+0000","last_became_active":"2024-09-21T14:04:21.205354+0000","last_became_peered":"2024-09-21T14:04:21.205354+0000","last_unstale":"2024-09-21T14:05:29.212913+0000","last_undegraded":"2024-09-21T14:05:29.212913+0000","last_fullsized":"2024-09-21T14:05:29.212913+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:57:38.046627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:38.046627+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:25.341767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"241'10","reported_seq":859,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.167816+0000","last_change":"2024-09-21T14:03:03.228551+0000","last_active":"2024-09-21T14:05:26.167816+0000","last_peered":"2024-09-21T14:05:26.167816+0000","last_clean":"2024-09-21T14:05:26.167816+0000","last_became_active":"2024-09-21T13:57:12.686651+0000","last_became_peered":"2024-09-21T13:57:12.686651+0000","last_unstale":"2024-09-21T14:05:26.167816+0000","last_undegraded":"2024-09-21T14:05:26.167816+0000","last_fullsized":"2024-09-21T14:05:26.167816+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'10","last_scrub_stamp":"2024-09-21T14:03:03.228443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:03.228443+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":743,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.214435+0000","last_change":"2024-09-21T14:04:45.055770+0000","last_active":"2024-09-21T14:05:26.214435+0000","last_peered":"2024-09-21T14:05:26.214435+0000","last_clean":"2024-09-21T14:05:26.214435+0000","last_became_active":"2024-09-21T13:59:32.209615+0000","last_became_peered":"2024-09-21T13:59:32.209615+0000","last_unstale":"2024-09-21T14:05:26.214435+0000","last_undegraded":"2024-09-21T14:05:26.214435+0000","last_fullsized":"2024-09-21T14:05:26.214435+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:45.055707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:04:45.055707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:03.598442+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.c","version":"428'10822","reported_seq":10929,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.191111+0000","last_change":"2024-09-21T14:02:56.563139+0000","last_active":"2024-09-21T14:05:26.191111+0000","last_peered":"2024-09-21T14:05:26.191111+0000","last_clean":"2024-09-21T14:05:26.191111+0000","last_became_active":"2024-09-21T14:02:56.562806+0000","last_became_peered":"2024-09-21T14:02:56.562806+0000","last_unstale":"2024-09-21T14:05:26.191111+0000","last_undegraded":"2024-09-21T14:05:26.191111+0000","last_fullsized":"2024-09-21T14:05:26.191111+0000","mapping_epoch":381,"log_start":"406'6309","ondisk_log_start":"406'6309","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":4513,"log_dups_size":0,"ondisk_log_size":4513,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":315097088,"num_objects":4808,"num_object_clones":0,"num_object_copies":14424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10822,"num_write_kb":500160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"75'79","reported_seq":825,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.215068+0000","last_change":"2024-09-21T14:04:38.868411+0000","last_active":"2024-09-21T14:05:26.215068+0000","last_peered":"2024-09-21T14:05:26.215068+0000","last_clean":"2024-09-21T14:05:26.215068+0000","last_became_active":"2024-09-21T14:00:41.083425+0000","last_became_peered":"2024-09-21T14:00:41.083425+0000","last_unstale":"2024-09-21T14:05:26.215068+0000","last_undegraded":"2024-09-21T14:05:26.215068+0000","last_fullsized":"2024-09-21T14:05:26.215068+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:04:38.868343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:04:38.868343+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:49.409572+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"290'20","reported_seq":110,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.191066+0000","last_change":"2024-09-21T14:04:42.776698+0000","last_active":"2024-09-21T14:05:26.191066+0000","last_peered":"2024-09-21T14:05:26.191066+0000","last_clean":"2024-09-21T14:05:26.191066+0000","last_became_active":"2024-09-21T14:03:29.158218+0000","last_became_peered":"2024-09-21T14:03:29.158218+0000","last_unstale":"2024-09-21T14:05:26.191066+0000","last_undegraded":"2024-09-21T14:05:26.191066+0000","last_fullsized":"2024-09-21T14:05:26.191066+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'20","last_scrub_stamp":"2024-09-21T14:04:42.776603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:04:42.776603+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:58.993778+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"61'1","reported_seq":714,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T14:05:25.161498+0000","last_change":"2024-09-21T14:04:48.425745+0000","last_active":"2024-09-21T14:05:25.161498+0000","last_peered":"2024-09-21T14:05:25.161498+0000","last_clean":"2024-09-21T14:05:25.161498+0000","last_became_active":"2024-09-21T14:01:41.195268+0000","last_became_peered":"2024-09-21T14:01:41.195268+0000","last_unstale":"2024-09-21T14:05:25.161498+0000","last_undegraded":"2024-09-21T14:05:25.161498+0000","last_fullsized":"2024-09-21T14:05:25.161498+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T14:04:48.425665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:48.425665+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:14.101369+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"418'16","reported_seq":136,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-09-21T14:05:29.214361+0000","last_change":"2024-09-21T14:04:56.590159+0000","last_active":"2024-09-21T14:05:29.214361+0000","last_peered":"2024-09-21T14:05:29.214361+0000","last_clean":"2024-09-21T14:05:29.214361+0000","last_became_active":"2024-09-21T14:04:56.589828+0000","last_became_peered":"2024-09-21T14:04:56.589828+0000","last_unstale":"2024-09-21T14:05:29.214361+0000","last_undegraded":"2024-09-21T14:05:29.214361+0000","last_fullsized":"2024-09-21T14:05:29.214361+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'15","last_scrub_stamp":"2024-09-21T14:04:43.936597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:04:43.936597+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:47.698681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":229,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-09-21T14:05:29.214657+0000","last_change":"2024-09-21T14:04:56.587813+0000","last_active":"2024-09-21T14:05:29.214657+0000","last_peered":"2024-09-21T14:05:29.214657+0000","last_clean":"2024-09-21T14:05:29.214657+0000","last_became_active":"2024-09-21T14:04:56.586891+0000","last_became_peered":"2024-09-21T14:04:56.586891+0000","last_unstale":"2024-09-21T14:05:29.214657+0000","last_undegraded":"2024-09-21T14:05:29.214657+0000","last_fullsized":"2024-09-21T14:05:29.214657+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:41.400346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:04:41.400346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:41.941073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.d","version":"428'11315","reported_seq":20728,"reported_epoch":428,"state":"active+remapped+backfilling","last_fresh":"2024-09-21T14:05:28.825587+0000","last_change":"2024-09-21T14:04:52.308462+0000","last_active":"2024-09-21T14:05:28.825587+0000","last_peered":"2024-09-21T14:05:28.825587+0000","last_clean":"2024-09-21T14:04:20.166030+0000","last_became_active":"2024-09-21T14:04:22.281826+0000","last_became_peered":"2024-09-21T14:04:22.281826+0000","last_unstale":"2024-09-21T14:05:28.825587+0000","last_undegraded":"2024-09-21T14:05:28.825587+0000","last_fullsized":"2024-09-21T14:05:28.825587+0000","mapping_epoch":409,"log_start":"406'6168","ondisk_log_start":"406'6168","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"406'6403","last_scrub_stamp":"2024-09-21T14:04:11.019087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:04:11.019087+0000","objects_scrubbed":6346,"log_size":5147,"log_dups_size":0,"ondisk_log_size":5147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1942,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":305659928,"num_objects":4665,"num_object_clones":0,"num_object_copies":13995,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":889,"num_objects_unfound":0,"num_objects_dirty":4665,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11315,"num_write_kb":511297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4612,"num_bytes_recovered":302186520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,0],"avail_no_missing":["6","0","4"],"object_location_counts":[{"shards":"0,4,6","objects":4665}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"75'79","reported_seq":833,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.190291+0000","last_change":"2024-09-21T14:04:44.986902+0000","last_active":"2024-09-21T14:05:26.190291+0000","last_peered":"2024-09-21T14:05:26.190291+0000","last_clean":"2024-09-21T14:05:26.190291+0000","last_became_active":"2024-09-21T14:00:41.073047+0000","last_became_peered":"2024-09-21T14:00:41.073047+0000","last_unstale":"2024-09-21T14:05:26.190291+0000","last_undegraded":"2024-09-21T14:05:26.190291+0000","last_fullsized":"2024-09-21T14:05:26.190291+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:04:44.986816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:04:44.986816+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:58.169458+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"290'29","reported_seq":530,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-09-21T14:05:29.214469+0000","last_change":"2024-09-21T14:04:56.589405+0000","last_active":"2024-09-21T14:05:29.214469+0000","last_peered":"2024-09-21T14:05:29.214469+0000","last_clean":"2024-09-21T14:05:29.214469+0000","last_became_active":"2024-09-21T14:04:56.589018+0000","last_became_peered":"2024-09-21T14:04:56.589018+0000","last_unstale":"2024-09-21T14:05:29.214469+0000","last_undegraded":"2024-09-21T14:05:29.214469+0000","last_fullsized":"2024-09-21T14:05:29.214469+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'29","last_scrub_stamp":"2024-09-21T14:04:53.477766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:04:53.477766+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:12.011852+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":792,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-09-21T14:05:29.212814+0000","last_change":"2024-09-21T14:01:41.185706+0000","last_active":"2024-09-21T14:05:29.212814+0000","last_peered":"2024-09-21T14:05:29.212814+0000","last_clean":"2024-09-21T14:05:29.212814+0000","last_became_active":"2024-09-21T14:01:41.185400+0000","last_became_peered":"2024-09-21T14:01:41.185400+0000","last_unstale":"2024-09-21T14:05:29.212814+0000","last_undegraded":"2024-09-21T14:05:29.212814+0000","last_fullsized":"2024-09-21T14:05:29.212814+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:16.712990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:16.712990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:26.352049+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":374,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.214801+0000","last_change":"2024-09-21T14:04:03.070260+0000","last_active":"2024-09-21T14:05:26.214801+0000","last_peered":"2024-09-21T14:05:26.214801+0000","last_clean":"2024-09-21T14:05:26.214801+0000","last_became_active":"2024-09-21T14:01:41.193973+0000","last_became_peered":"2024-09-21T14:01:41.193973+0000","last_unstale":"2024-09-21T14:05:26.214801+0000","last_undegraded":"2024-09-21T14:05:26.214801+0000","last_fullsized":"2024-09-21T14:05:26.214801+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:03.069973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:04:03.069973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:26.954137+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":783,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-09-21T14:05:29.211845+0000","last_change":"2024-09-21T13:58:43.952014+0000","last_active":"2024-09-21T14:05:29.211845+0000","last_peered":"2024-09-21T14:05:29.211845+0000","last_clean":"2024-09-21T14:05:29.211845+0000","last_became_active":"2024-09-21T13:58:17.757628+0000","last_became_peered":"2024-09-21T13:58:17.757628+0000","last_unstale":"2024-09-21T14:05:29.211845+0000","last_undegraded":"2024-09-21T14:05:29.211845+0000","last_fullsized":"2024-09-21T14:05:29.211845+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:43.951941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:43.951941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"428'11177","reported_seq":20893,"reported_epoch":428,"state":"active+remapped+backfilling","last_fresh":"2024-09-21T14:05:28.823803+0000","last_change":"2024-09-21T14:04:52.389607+0000","last_active":"2024-09-21T14:05:28.823803+0000","last_peered":"2024-09-21T14:05:28.823803+0000","last_clean":"2024-09-21T14:04:20.165921+0000","last_became_active":"2024-09-21T14:04:22.361054+0000","last_became_peered":"2024-09-21T14:04:22.361054+0000","last_unstale":"2024-09-21T14:05:28.823803+0000","last_undegraded":"2024-09-21T14:05:28.823803+0000","last_fullsized":"2024-09-21T14:05:28.823803+0000","mapping_epoch":409,"log_start":"405'6048","ondisk_log_start":"405'6048","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":3149,"log_size":5129,"log_dups_size":0,"ondisk_log_size":5129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":303366144,"num_objects":4629,"num_object_clones":0,"num_object_copies":13887,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":690,"num_objects_unfound":0,"num_objects_dirty":4629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11177,"num_write_kb":505792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4783,"num_bytes_recovered":313458688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,0],"avail_no_missing":["6","0","5"],"object_location_counts":[{"shards":"0,5,6","objects":4629}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"75'79","reported_seq":188,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T14:05:25.160569+0000","last_change":"2024-09-21T14:01:41.199377+0000","last_active":"2024-09-21T14:05:25.160569+0000","last_peered":"2024-09-21T14:05:25.160569+0000","last_clean":"2024-09-21T14:05:25.160569+0000","last_became_active":"2024-09-21T14:01:41.198750+0000","last_became_peered":"2024-09-21T14:01:41.198750+0000","last_unstale":"2024-09-21T14:05:25.160569+0000","last_undegraded":"2024-09-21T14:05:25.160569+0000","last_fullsized":"2024-09-21T14:05:25.160569+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:05:59.493308+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"290'44","reported_seq":1065,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-09-21T14:05:29.212708+0000","last_change":"2024-09-21T14:02:57.316829+0000","last_active":"2024-09-21T14:05:29.212708+0000","last_peered":"2024-09-21T14:05:29.212708+0000","last_clean":"2024-09-21T14:05:29.212708+0000","last_became_active":"2024-09-21T13:57:12.684690+0000","last_became_peered":"2024-09-21T13:57:12.684690+0000","last_unstale":"2024-09-21T14:05:29.212708+0000","last_undegraded":"2024-09-21T14:05:29.212708+0000","last_fullsized":"2024-09-21T14:05:29.212708+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'44","last_scrub_stamp":"2024-09-21T14:02:57.316763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:57.316763+0000","objects_scrubbed":10,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":805,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.248114+0000","last_change":"2024-09-21T14:04:11.158776+0000","last_active":"2024-09-21T14:05:26.248114+0000","last_peered":"2024-09-21T14:05:26.248114+0000","last_clean":"2024-09-21T14:05:26.248114+0000","last_became_active":"2024-09-21T13:58:17.770937+0000","last_became_peered":"2024-09-21T13:58:17.770937+0000","last_unstale":"2024-09-21T14:05:26.248114+0000","last_undegraded":"2024-09-21T14:05:26.248114+0000","last_fullsized":"2024-09-21T14:05:26.248114+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:11.158703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:11.158703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:26.630567+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":741,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T14:05:25.162413+0000","last_change":"2024-09-21T14:05:15.377389+0000","last_active":"2024-09-21T14:05:25.162413+0000","last_peered":"2024-09-21T14:05:25.162413+0000","last_clean":"2024-09-21T14:05:25.162413+0000","last_became_active":"2024-09-21T14:01:41.247642+0000","last_became_peered":"2024-09-21T14:01:41.247642+0000","last_unstale":"2024-09-21T14:05:25.162413+0000","last_undegraded":"2024-09-21T14:05:25.162413+0000","last_fullsized":"2024-09-21T14:05:25.162413+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-21T14:05:15.377198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T14:05:15.377198+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:31.018201+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":726,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-09-21T14:05:29.212436+0000","last_change":"2024-09-21T14:05:18.354476+0000","last_active":"2024-09-21T14:05:29.212436+0000","last_peered":"2024-09-21T14:05:29.212436+0000","last_clean":"2024-09-21T14:05:29.212436+0000","last_became_active":"2024-09-21T14:04:21.204315+0000","last_became_peered":"2024-09-21T14:04:21.204315+0000","last_unstale":"2024-09-21T14:05:29.212436+0000","last_undegraded":"2024-09-21T14:05:29.212436+0000","last_fullsized":"2024-09-21T14:05:29.212436+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:18.354390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:05:18.354390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:19.770158+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":706,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.169142+0000","last_change":"2024-09-21T14:03:02.238409+0000","last_active":"2024-09-21T14:05:26.169142+0000","last_peered":"2024-09-21T14:05:26.169142+0000","last_clean":"2024-09-21T14:05:26.169142+0000","last_became_active":"2024-09-21T13:59:32.208980+0000","last_became_peered":"2024-09-21T13:59:32.208980+0000","last_unstale":"2024-09-21T14:05:26.169142+0000","last_undegraded":"2024-09-21T14:05:26.169142+0000","last_fullsized":"2024-09-21T14:05:26.169142+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:02.238357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:02.238357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"427'10937","reported_seq":11110,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T14:05:25.358608+0000","last_change":"2024-09-21T14:04:06.243298+0000","last_active":"2024-09-21T14:05:25.358608+0000","last_peered":"2024-09-21T14:05:25.358608+0000","last_clean":"2024-09-21T14:05:25.358608+0000","last_became_active":"2024-09-21T14:02:56.558773+0000","last_became_peered":"2024-09-21T14:02:56.558773+0000","last_unstale":"2024-09-21T14:05:25.358608+0000","last_undegraded":"2024-09-21T14:05:25.358608+0000","last_fullsized":"2024-09-21T14:05:25.358608+0000","mapping_epoch":381,"log_start":"406'6069","ondisk_log_start":"406'6069","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'5948","last_scrub_stamp":"2024-09-21T14:04:06.243212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:04:06.243212+0000","objects_scrubbed":5906,"log_size":4868,"log_dups_size":0,"ondisk_log_size":4868,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1458,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":313327616,"num_objects":4781,"num_object_clones":0,"num_object_copies":14343,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10937,"num_write_kb":502976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"75'79","reported_seq":830,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.190848+0000","last_change":"2024-09-21T14:04:21.206885+0000","last_active":"2024-09-21T14:05:26.190848+0000","last_peered":"2024-09-21T14:05:26.190848+0000","last_clean":"2024-09-21T14:05:26.190848+0000","last_became_active":"2024-09-21T14:04:21.206472+0000","last_became_peered":"2024-09-21T14:04:21.206472+0000","last_unstale":"2024-09-21T14:05:26.190848+0000","last_undegraded":"2024-09-21T14:05:26.190848+0000","last_fullsized":"2024-09-21T14:05:26.190848+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"290'17","reported_seq":837,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-09-21T14:05:29.213094+0000","last_change":"2024-09-21T14:04:21.224066+0000","last_active":"2024-09-21T14:05:29.213094+0000","last_peered":"2024-09-21T14:05:29.213094+0000","last_clean":"2024-09-21T14:05:29.213094+0000","last_became_active":"2024-09-21T14:04:21.205738+0000","last_became_peered":"2024-09-21T14:04:21.205738+0000","last_unstale":"2024-09-21T14:05:29.213094+0000","last_undegraded":"2024-09-21T14:05:29.213094+0000","last_fullsized":"2024-09-21T14:05:29.213094+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'13","last_scrub_stamp":"2024-09-21T13:58:11.619848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:11.619848+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":836,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-09-21T14:05:29.211446+0000","last_change":"2024-09-21T14:04:10.193524+0000","last_active":"2024-09-21T14:05:29.211446+0000","last_peered":"2024-09-21T14:05:29.211446+0000","last_clean":"2024-09-21T14:05:29.211446+0000","last_became_active":"2024-09-21T13:58:17.759671+0000","last_became_peered":"2024-09-21T13:58:17.759671+0000","last_unstale":"2024-09-21T14:05:29.211446+0000","last_undegraded":"2024-09-21T14:05:29.211446+0000","last_fullsized":"2024-09-21T14:05:29.211446+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:10.193318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:10.193318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":707,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.191136+0000","last_change":"2024-09-21T13:58:41.644876+0000","last_active":"2024-09-21T14:05:26.191136+0000","last_peered":"2024-09-21T14:05:26.191136+0000","last_clean":"2024-09-21T14:05:26.191136+0000","last_became_active":"2024-09-21T13:57:12.682147+0000","last_became_peered":"2024-09-21T13:57:12.682147+0000","last_unstale":"2024-09-21T14:05:26.191136+0000","last_undegraded":"2024-09-21T14:05:26.191136+0000","last_fullsized":"2024-09-21T14:05:26.191136+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:58:41.644789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:58:41.644789+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":686,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.191603+0000","last_change":"2024-09-21T14:01:50.880840+0000","last_active":"2024-09-21T14:05:26.191603+0000","last_peered":"2024-09-21T14:05:26.191603+0000","last_clean":"2024-09-21T14:05:26.191603+0000","last_became_active":"2024-09-21T14:01:41.182000+0000","last_became_peered":"2024-09-21T14:01:41.182000+0000","last_unstale":"2024-09-21T14:05:26.191603+0000","last_undegraded":"2024-09-21T14:05:26.191603+0000","last_fullsized":"2024-09-21T14:05:26.191603+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:50.880471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:01:50.880471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":743,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.213726+0000","last_change":"2024-09-21T13:59:32.222791+0000","last_active":"2024-09-21T14:05:26.213726+0000","last_peered":"2024-09-21T14:05:26.213726+0000","last_clean":"2024-09-21T14:05:26.213726+0000","last_became_active":"2024-09-21T13:59:32.221502+0000","last_became_peered":"2024-09-21T13:59:32.221502+0000","last_unstale":"2024-09-21T14:05:26.213726+0000","last_undegraded":"2024-09-21T14:05:26.213726+0000","last_fullsized":"2024-09-21T14:05:26.213726+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:14.673582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:14.673582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"429'11432","reported_seq":11614,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-09-21T14:05:29.497322+0000","last_change":"2024-09-21T14:04:56.601777+0000","last_active":"2024-09-21T14:05:29.497322+0000","last_peered":"2024-09-21T14:05:29.497322+0000","last_clean":"2024-09-21T14:05:29.497322+0000","last_became_active":"2024-09-21T14:04:56.586861+0000","last_became_peered":"2024-09-21T14:04:56.586861+0000","last_unstale":"2024-09-21T14:05:29.497322+0000","last_undegraded":"2024-09-21T14:05:29.497322+0000","last_fullsized":"2024-09-21T14:05:29.497322+0000","mapping_epoch":417,"log_start":"398'4382","ondisk_log_start":"398'4382","created":381,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'6143","last_scrub_stamp":"2024-09-21T14:04:06.996122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:04:06.996122+0000","objects_scrubbed":6071,"log_size":7050,"log_dups_size":0,"ondisk_log_size":7050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:36.829006+0000","scrub_duration":2407,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306315264,"num_objects":4674,"num_object_clones":0,"num_object_copies":14022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11432,"num_write_kb":515392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"75'79","reported_seq":245,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-09-21T14:05:29.213782+0000","last_change":"2024-09-21T14:05:13.503769+0000","last_active":"2024-09-21T14:05:29.213782+0000","last_peered":"2024-09-21T14:05:29.213782+0000","last_clean":"2024-09-21T14:05:29.213782+0000","last_became_active":"2024-09-21T14:04:21.210586+0000","last_became_peered":"2024-09-21T14:04:21.210586+0000","last_unstale":"2024-09-21T14:05:29.213782+0000","last_undegraded":"2024-09-21T14:05:29.213782+0000","last_fullsized":"2024-09-21T14:05:29.213782+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:05:13.503720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:05:13.503720+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:33.310357+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"290'24","reported_seq":630,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T14:05:25.162335+0000","last_change":"2024-09-21T14:04:21.221906+0000","last_active":"2024-09-21T14:05:25.162335+0000","last_peered":"2024-09-21T14:05:25.162335+0000","last_clean":"2024-09-21T14:05:25.162335+0000","last_became_active":"2024-09-21T14:04:21.206881+0000","last_became_peered":"2024-09-21T14:04:21.206881+0000","last_unstale":"2024-09-21T14:05:25.162335+0000","last_undegraded":"2024-09-21T14:05:25.162335+0000","last_fullsized":"2024-09-21T14:05:25.162335+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'24","last_scrub_stamp":"2024-09-21T14:02:57.559936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:57.559936+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:06.366044+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":353,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-09-21T14:05:29.214670+0000","last_change":"2024-09-21T14:04:56.588103+0000","last_active":"2024-09-21T14:05:29.214670+0000","last_peered":"2024-09-21T14:05:29.214670+0000","last_clean":"2024-09-21T14:05:29.214670+0000","last_became_active":"2024-09-21T14:04:56.587384+0000","last_became_peered":"2024-09-21T14:04:56.587384+0000","last_unstale":"2024-09-21T14:05:29.214670+0000","last_undegraded":"2024-09-21T14:05:29.214670+0000","last_fullsized":"2024-09-21T14:05:29.214670+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:59.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:59:59.450546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:58.054530+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":1631,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.191119+0000","last_change":"2024-09-21T13:59:32.218796+0000","last_active":"2024-09-21T14:05:26.191119+0000","last_peered":"2024-09-21T14:05:26.191119+0000","last_clean":"2024-09-21T14:05:26.191119+0000","last_became_active":"2024-09-21T13:59:32.203361+0000","last_became_peered":"2024-09-21T13:59:32.203361+0000","last_unstale":"2024-09-21T14:05:26.191119+0000","last_undegraded":"2024-09-21T14:05:26.191119+0000","last_fullsized":"2024-09-21T14:05:26.191119+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-21T13:56:29.617143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:56:29.617143+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:31.086108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047048000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":360,"num_read_kb":355,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":712,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T14:05:25.163727+0000","last_change":"2024-09-21T14:05:14.344869+0000","last_active":"2024-09-21T14:05:25.163727+0000","last_peered":"2024-09-21T14:05:25.163727+0000","last_clean":"2024-09-21T14:05:25.163727+0000","last_became_active":"2024-09-21T14:01:41.200531+0000","last_became_peered":"2024-09-21T14:01:41.200531+0000","last_unstale":"2024-09-21T14:05:25.163727+0000","last_undegraded":"2024-09-21T14:05:25.163727+0000","last_fullsized":"2024-09-21T14:05:25.163727+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:14.344795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:05:14.344795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:33.382674+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":812,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-09-21T14:05:29.212400+0000","last_change":"2024-09-21T14:03:04.357026+0000","last_active":"2024-09-21T14:05:29.212400+0000","last_peered":"2024-09-21T14:05:29.212400+0000","last_clean":"2024-09-21T14:05:29.212400+0000","last_became_active":"2024-09-21T14:01:41.184664+0000","last_became_peered":"2024-09-21T14:01:41.184664+0000","last_unstale":"2024-09-21T14:05:29.212400+0000","last_undegraded":"2024-09-21T14:05:29.212400+0000","last_fullsized":"2024-09-21T14:05:29.212400+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:04.356979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:04.356979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"428'11193","reported_seq":11369,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:28.794349+0000","last_change":"2024-09-21T14:04:08.011633+0000","last_active":"2024-09-21T14:05:28.794349+0000","last_peered":"2024-09-21T14:05:28.794349+0000","last_clean":"2024-09-21T14:05:28.794349+0000","last_became_active":"2024-09-21T14:02:56.557263+0000","last_became_peered":"2024-09-21T14:02:56.557263+0000","last_unstale":"2024-09-21T14:05:28.794349+0000","last_undegraded":"2024-09-21T14:05:28.794349+0000","last_fullsized":"2024-09-21T14:05:28.794349+0000","mapping_epoch":381,"log_start":"404'6019","ondisk_log_start":"404'6019","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"406'6147","last_scrub_stamp":"2024-09-21T14:04:08.011556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:04:08.011556+0000","objects_scrubbed":6074,"log_size":5174,"log_dups_size":0,"ondisk_log_size":5174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1868,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":309526528,"num_objects":4723,"num_object_clones":0,"num_object_copies":14169,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11193,"num_write_kb":509312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"300'32","reported_seq":489,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.216381+0000","last_change":"2024-09-21T14:00:47.130089+0000","last_active":"2024-09-21T14:05:26.216381+0000","last_peered":"2024-09-21T14:05:26.216381+0000","last_clean":"2024-09-21T14:05:26.216381+0000","last_became_active":"2024-09-21T13:59:32.204624+0000","last_became_peered":"2024-09-21T13:59:32.204624+0000","last_unstale":"2024-09-21T14:05:26.216381+0000","last_undegraded":"2024-09-21T14:05:26.216381+0000","last_fullsized":"2024-09-21T14:05:26.216381+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'32","last_scrub_stamp":"2024-09-21T14:00:47.129789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:00:47.129789+0000","objects_scrubbed":8,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":764,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.168273+0000","last_change":"2024-09-21T14:04:42.390182+0000","last_active":"2024-09-21T14:05:26.168273+0000","last_peered":"2024-09-21T14:05:26.168273+0000","last_clean":"2024-09-21T14:05:26.168273+0000","last_became_active":"2024-09-21T14:01:41.190210+0000","last_became_peered":"2024-09-21T14:01:41.190210+0000","last_unstale":"2024-09-21T14:05:26.168273+0000","last_undegraded":"2024-09-21T14:05:26.168273+0000","last_fullsized":"2024-09-21T14:05:26.168273+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:42.390102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:42.390102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:10.628828+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"75'79","reported_seq":824,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.213913+0000","last_change":"2024-09-21T14:04:21.205378+0000","last_active":"2024-09-21T14:05:26.213913+0000","last_peered":"2024-09-21T14:05:26.213913+0000","last_clean":"2024-09-21T14:05:26.213913+0000","last_became_active":"2024-09-21T14:04:21.204537+0000","last_became_peered":"2024-09-21T14:04:21.204537+0000","last_unstale":"2024-09-21T14:05:26.213913+0000","last_undegraded":"2024-09-21T14:05:26.213913+0000","last_fullsized":"2024-09-21T14:05:26.213913+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":19,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53669,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":731,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.191189+0000","last_change":"2024-09-21T14:01:41.182186+0000","last_active":"2024-09-21T14:05:26.191189+0000","last_peered":"2024-09-21T14:05:26.191189+0000","last_clean":"2024-09-21T14:05:26.191189+0000","last_became_active":"2024-09-21T14:01:41.181899+0000","last_became_peered":"2024-09-21T14:01:41.181899+0000","last_unstale":"2024-09-21T14:05:26.191189+0000","last_undegraded":"2024-09-21T14:05:26.191189+0000","last_fullsized":"2024-09-21T14:05:26.191189+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:15.700409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:15.700409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":669,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T14:05:25.163486+0000","last_change":"2024-09-21T14:04:01.760941+0000","last_active":"2024-09-21T14:05:25.163486+0000","last_peered":"2024-09-21T14:05:25.163486+0000","last_clean":"2024-09-21T14:05:25.163486+0000","last_became_active":"2024-09-21T14:03:57.346453+0000","last_became_peered":"2024-09-21T14:03:57.346453+0000","last_unstale":"2024-09-21T14:05:25.163486+0000","last_undegraded":"2024-09-21T14:05:25.163486+0000","last_fullsized":"2024-09-21T14:05:25.163486+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:01.760755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:04:01.760755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"428'11367","reported_seq":23325,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:28.806181+0000","last_change":"2024-09-21T14:05:26.237178+0000","last_active":"2024-09-21T14:05:28.806181+0000","last_peered":"2024-09-21T14:05:28.806181+0000","last_clean":"2024-09-21T14:05:28.806181+0000","last_became_active":"2024-09-21T14:05:26.215569+0000","last_became_peered":"2024-09-21T14:05:26.215569+0000","last_unstale":"2024-09-21T14:05:28.806181+0000","last_undegraded":"2024-09-21T14:05:28.806181+0000","last_fullsized":"2024-09-21T14:05:28.806181+0000","mapping_epoch":427,"log_start":"406'6236","ondisk_log_start":"406'6236","created":381,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"406'6598","last_scrub_stamp":"2024-09-21T14:04:14.205496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:04:14.205496+0000","objects_scrubbed":6523,"log_size":5131,"log_dups_size":0,"ondisk_log_size":5131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:30.032839+0000","scrub_duration":2142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301400064,"num_objects":4599,"num_object_clones":0,"num_object_copies":13797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11367,"num_write_kb":510912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5879,"num_bytes_recovered":385286144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"290'19","reported_seq":843,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.248959+0000","last_change":"2024-09-21T14:02:28.553549+0000","last_active":"2024-09-21T14:05:26.248959+0000","last_peered":"2024-09-21T14:05:26.248959+0000","last_clean":"2024-09-21T14:05:26.248959+0000","last_became_active":"2024-09-21T13:58:17.776310+0000","last_became_peered":"2024-09-21T13:58:17.776310+0000","last_unstale":"2024-09-21T14:05:26.248959+0000","last_undegraded":"2024-09-21T14:05:26.248959+0000","last_fullsized":"2024-09-21T14:05:26.248959+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'19","last_scrub_stamp":"2024-09-21T14:02:28.553065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:28.553065+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":813,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.248135+0000","last_change":"2024-09-21T14:03:32.957705+0000","last_active":"2024-09-21T14:05:26.248135+0000","last_peered":"2024-09-21T14:05:26.248135+0000","last_clean":"2024-09-21T14:05:26.248135+0000","last_became_active":"2024-09-21T13:59:32.223247+0000","last_became_peered":"2024-09-21T13:59:32.223247+0000","last_unstale":"2024-09-21T14:05:26.248135+0000","last_undegraded":"2024-09-21T14:05:26.248135+0000","last_fullsized":"2024-09-21T14:05:26.248135+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:32.957649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:03:32.957649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":738,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T14:05:25.163409+0000","last_change":"2024-09-21T14:04:54.483668+0000","last_active":"2024-09-21T14:05:25.163409+0000","last_peered":"2024-09-21T14:05:25.163409+0000","last_clean":"2024-09-21T14:05:25.163409+0000","last_became_active":"2024-09-21T14:01:41.193941+0000","last_became_peered":"2024-09-21T14:01:41.193941+0000","last_unstale":"2024-09-21T14:05:25.163409+0000","last_undegraded":"2024-09-21T14:05:25.163409+0000","last_fullsized":"2024-09-21T14:05:25.163409+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:54.483438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:04:54.483438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:21.139691+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":742,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.169284+0000","last_change":"2024-09-21T14:03:36.302360+0000","last_active":"2024-09-21T14:05:26.169284+0000","last_peered":"2024-09-21T14:05:26.169284+0000","last_clean":"2024-09-21T14:05:26.169284+0000","last_became_active":"2024-09-21T14:01:41.190192+0000","last_became_peered":"2024-09-21T14:01:41.190192+0000","last_unstale":"2024-09-21T14:05:26.169284+0000","last_undegraded":"2024-09-21T14:05:26.169284+0000","last_fullsized":"2024-09-21T14:05:26.169284+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:36.302281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:36.302281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"428'11142","reported_seq":11326,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:28.026828+0000","last_change":"2024-09-21T14:04:52.445772+0000","last_active":"2024-09-21T14:05:28.026828+0000","last_peered":"2024-09-21T14:05:28.026828+0000","last_clean":"2024-09-21T14:05:28.026828+0000","last_became_active":"2024-09-21T14:02:56.560070+0000","last_became_peered":"2024-09-21T14:02:56.560070+0000","last_unstale":"2024-09-21T14:05:28.026828+0000","last_undegraded":"2024-09-21T14:05:28.026828+0000","last_fullsized":"2024-09-21T14:05:28.026828+0000","mapping_epoch":381,"log_start":"406'6187","ondisk_log_start":"406'6187","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'9024","last_scrub_stamp":"2024-09-21T14:04:52.445618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:04:52.445618+0000","objects_scrubbed":6949,"log_size":4955,"log_dups_size":0,"ondisk_log_size":4955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:01.306398+0000","scrub_duration":2824,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":311296000,"num_objects":4750,"num_object_clones":0,"num_object_copies":14250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11142,"num_write_kb":508544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"75'79","reported_seq":203,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T14:05:25.160340+0000","last_change":"2024-09-21T14:04:51.454570+0000","last_active":"2024-09-21T14:05:25.160340+0000","last_peered":"2024-09-21T14:05:25.160340+0000","last_clean":"2024-09-21T14:05:25.160340+0000","last_became_active":"2024-09-21T14:01:41.188685+0000","last_became_peered":"2024-09-21T14:01:41.188685+0000","last_unstale":"2024-09-21T14:05:25.160340+0000","last_undegraded":"2024-09-21T14:05:25.160340+0000","last_fullsized":"2024-09-21T14:05:25.160340+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:04:51.454483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:04:51.454483+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:17.391460+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"290'31","reported_seq":619,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.191378+0000","last_change":"2024-09-21T14:04:39.741288+0000","last_active":"2024-09-21T14:05:26.191378+0000","last_peered":"2024-09-21T14:05:26.191378+0000","last_clean":"2024-09-21T14:05:26.191378+0000","last_became_active":"2024-09-21T14:04:21.204164+0000","last_became_peered":"2024-09-21T14:04:21.204164+0000","last_unstale":"2024-09-21T14:05:26.191378+0000","last_undegraded":"2024-09-21T14:05:26.191378+0000","last_fullsized":"2024-09-21T14:05:26.191378+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'31","last_scrub_stamp":"2024-09-21T14:04:39.740794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:04:39.740794+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:54.328250+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":355,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.190205+0000","last_change":"2024-09-21T14:04:51.139211+0000","last_active":"2024-09-21T14:05:26.190205+0000","last_peered":"2024-09-21T14:05:26.190205+0000","last_clean":"2024-09-21T14:05:26.190205+0000","last_became_active":"2024-09-21T13:58:17.762520+0000","last_became_peered":"2024-09-21T13:58:17.762520+0000","last_unstale":"2024-09-21T14:05:26.190205+0000","last_undegraded":"2024-09-21T14:05:26.190205+0000","last_fullsized":"2024-09-21T14:05:26.190205+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:51.139123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:51.139123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:51.167153+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":762,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-09-21T14:05:29.212508+0000","last_change":"2024-09-21T13:58:52.030800+0000","last_active":"2024-09-21T14:05:29.212508+0000","last_peered":"2024-09-21T14:05:29.212508+0000","last_clean":"2024-09-21T14:05:29.212508+0000","last_became_active":"2024-09-21T13:49:02.067472+0000","last_became_peered":"2024-09-21T13:49:02.067472+0000","last_unstale":"2024-09-21T14:05:29.212508+0000","last_undegraded":"2024-09-21T14:05:29.212508+0000","last_fullsized":"2024-09-21T14:05:29.212508+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:52.030718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:52.030718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":642,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.191654+0000","last_change":"2024-09-21T13:57:12.683321+0000","last_active":"2024-09-21T14:05:26.191654+0000","last_peered":"2024-09-21T14:05:26.191654+0000","last_clean":"2024-09-21T14:05:26.191654+0000","last_became_active":"2024-09-21T13:57:12.682979+0000","last_became_peered":"2024-09-21T13:57:12.682979+0000","last_unstale":"2024-09-21T14:05:26.191654+0000","last_undegraded":"2024-09-21T14:05:26.191654+0000","last_fullsized":"2024-09-21T14:05:26.191654+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:18.027944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:18.027944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2","version":"429'11375","reported_seq":11483,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-09-21T14:05:29.759123+0000","last_change":"2024-09-21T14:02:56.559657+0000","last_active":"2024-09-21T14:05:29.759123+0000","last_peered":"2024-09-21T14:05:29.759123+0000","last_clean":"2024-09-21T14:05:29.759123+0000","last_became_active":"2024-09-21T14:02:56.559368+0000","last_became_peered":"2024-09-21T14:02:56.559368+0000","last_unstale":"2024-09-21T14:05:29.759123+0000","last_undegraded":"2024-09-21T14:05:29.759123+0000","last_fullsized":"2024-09-21T14:05:29.759123+0000","mapping_epoch":381,"log_start":"406'6642","ondisk_log_start":"406'6642","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":4733,"log_dups_size":0,"ondisk_log_size":4733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":300613632,"num_objects":4587,"num_object_clones":0,"num_object_copies":13761,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11375,"num_write_kb":510784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"75'79","reported_seq":831,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.213555+0000","last_change":"2024-09-21T14:02:55.806160+0000","last_active":"2024-09-21T14:05:26.213555+0000","last_peered":"2024-09-21T14:05:26.213555+0000","last_clean":"2024-09-21T14:05:26.213555+0000","last_became_active":"2024-09-21T14:00:41.139315+0000","last_became_peered":"2024-09-21T14:00:41.139315+0000","last_unstale":"2024-09-21T14:05:26.213555+0000","last_undegraded":"2024-09-21T14:05:26.213555+0000","last_fullsized":"2024-09-21T14:05:26.213555+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:02:55.806077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.806077+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"290'20","reported_seq":858,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.248438+0000","last_change":"2024-09-21T14:05:15.387788+0000","last_active":"2024-09-21T14:05:26.248438+0000","last_peered":"2024-09-21T14:05:26.248438+0000","last_clean":"2024-09-21T14:05:26.248438+0000","last_became_active":"2024-09-21T13:58:17.765427+0000","last_became_peered":"2024-09-21T13:58:17.765427+0000","last_unstale":"2024-09-21T14:05:26.248438+0000","last_undegraded":"2024-09-21T14:05:26.248438+0000","last_fullsized":"2024-09-21T14:05:26.248438+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'20","last_scrub_stamp":"2024-09-21T14:05:15.387706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:05:15.387706+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:39.269627+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":706,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.168450+0000","last_change":"2024-09-21T14:04:21.215419+0000","last_active":"2024-09-21T14:05:26.168450+0000","last_peered":"2024-09-21T14:05:26.168450+0000","last_clean":"2024-09-21T14:05:26.168450+0000","last_became_active":"2024-09-21T14:04:21.215102+0000","last_became_peered":"2024-09-21T14:04:21.215102+0000","last_unstale":"2024-09-21T14:05:26.168450+0000","last_undegraded":"2024-09-21T14:05:26.168450+0000","last_fullsized":"2024-09-21T14:05:26.168450+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:58.236645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:02:58.236645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"346'14","reported_seq":853,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.191488+0000","last_change":"2024-09-21T14:01:41.186960+0000","last_active":"2024-09-21T14:05:26.191488+0000","last_peered":"2024-09-21T14:05:26.191488+0000","last_clean":"2024-09-21T14:05:26.191488+0000","last_became_active":"2024-09-21T14:01:41.186564+0000","last_became_peered":"2024-09-21T14:01:41.186564+0000","last_unstale":"2024-09-21T14:05:26.191488+0000","last_undegraded":"2024-09-21T14:05:26.191488+0000","last_fullsized":"2024-09-21T14:05:26.191488+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'12","last_scrub_stamp":"2024-09-21T13:59:38.738120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:38.738120+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"68'1","reported_seq":778,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.248852+0000","last_change":"2024-09-21T13:59:31.392186+0000","last_active":"2024-09-21T14:05:26.248852+0000","last_peered":"2024-09-21T14:05:26.248852+0000","last_clean":"2024-09-21T14:05:26.248852+0000","last_became_active":"2024-09-21T13:52:04.313210+0000","last_became_peered":"2024-09-21T13:52:04.313210+0000","last_unstale":"2024-09-21T14:05:26.248852+0000","last_undegraded":"2024-09-21T14:05:26.248852+0000","last_fullsized":"2024-09-21T14:05:26.248852+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T13:59:31.392105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:31.392105+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.5","version":"428'11441","reported_seq":11631,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:27.982471+0000","last_change":"2024-09-21T14:04:48.634887+0000","last_active":"2024-09-21T14:05:27.982471+0000","last_peered":"2024-09-21T14:05:27.982471+0000","last_clean":"2024-09-21T14:05:27.982471+0000","last_became_active":"2024-09-21T14:02:56.556786+0000","last_became_peered":"2024-09-21T14:02:56.556786+0000","last_unstale":"2024-09-21T14:05:27.982471+0000","last_undegraded":"2024-09-21T14:05:27.982471+0000","last_fullsized":"2024-09-21T14:05:27.982471+0000","mapping_epoch":381,"log_start":"406'6425","ondisk_log_start":"406'6425","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'8991","last_scrub_stamp":"2024-09-21T14:04:48.631935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:04:48.631935+0000","objects_scrubbed":7324,"log_size":5016,"log_dups_size":0,"ondisk_log_size":5016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:05.417795+0000","scrub_duration":2904,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314114048,"num_objects":4793,"num_object_clones":0,"num_object_copies":14379,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11441,"num_write_kb":519488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"290'15","reported_seq":807,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-09-21T14:05:29.214365+0000","last_change":"2024-09-21T13:58:50.498597+0000","last_active":"2024-09-21T14:05:29.214365+0000","last_peered":"2024-09-21T14:05:29.214365+0000","last_clean":"2024-09-21T14:05:29.214365+0000","last_became_active":"2024-09-21T13:49:00.053016+0000","last_became_peered":"2024-09-21T13:49:00.053016+0000","last_unstale":"2024-09-21T14:05:29.214365+0000","last_undegraded":"2024-09-21T14:05:29.214365+0000","last_fullsized":"2024-09-21T14:05:29.214365+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:58:50.498369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:50.498369+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:59.036946+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":750,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T14:05:25.161877+0000","last_change":"2024-09-21T14:05:22.402246+0000","last_active":"2024-09-21T14:05:25.161877+0000","last_peered":"2024-09-21T14:05:25.161877+0000","last_clean":"2024-09-21T14:05:25.161877+0000","last_became_active":"2024-09-21T14:01:41.192443+0000","last_became_peered":"2024-09-21T14:01:41.192443+0000","last_unstale":"2024-09-21T14:05:25.161877+0000","last_undegraded":"2024-09-21T14:05:25.161877+0000","last_fullsized":"2024-09-21T14:05:25.161877+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:22.402182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:05:22.402182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:48.287514+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":724,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.249012+0000","last_change":"2024-09-21T14:02:56.796917+0000","last_active":"2024-09-21T14:05:26.249012+0000","last_peered":"2024-09-21T14:05:26.249012+0000","last_clean":"2024-09-21T14:05:26.249012+0000","last_became_active":"2024-09-21T13:57:12.680978+0000","last_became_peered":"2024-09-21T13:57:12.680978+0000","last_unstale":"2024-09-21T14:05:26.249012+0000","last_undegraded":"2024-09-21T14:05:26.249012+0000","last_fullsized":"2024-09-21T14:05:26.249012+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:56.796861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:02:56.796861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":673,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.191142+0000","last_change":"2024-09-21T14:04:21.210096+0000","last_active":"2024-09-21T14:05:26.191142+0000","last_peered":"2024-09-21T14:05:26.191142+0000","last_clean":"2024-09-21T14:05:26.191142+0000","last_became_active":"2024-09-21T14:04:21.207534+0000","last_became_peered":"2024-09-21T14:04:21.207534+0000","last_unstale":"2024-09-21T14:05:26.191142+0000","last_undegraded":"2024-09-21T14:05:26.191142+0000","last_fullsized":"2024-09-21T14:05:26.191142+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:13.698091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:13.698091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"428'11231","reported_seq":25078,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:28.783468+0000","last_change":"2024-09-21T14:05:13.815252+0000","last_active":"2024-09-21T14:05:28.783468+0000","last_peered":"2024-09-21T14:05:28.783468+0000","last_clean":"2024-09-21T14:05:28.783468+0000","last_became_active":"2024-09-21T14:05:13.789148+0000","last_became_peered":"2024-09-21T14:05:13.789148+0000","last_unstale":"2024-09-21T14:05:28.783468+0000","last_undegraded":"2024-09-21T14:05:28.783468+0000","last_fullsized":"2024-09-21T14:05:28.783468+0000","mapping_epoch":422,"log_start":"404'5700","ondisk_log_start":"404'5700","created":381,"last_epoch_clean":423,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":5531,"log_dups_size":0,"ondisk_log_size":5531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297730048,"num_objects":4543,"num_object_clones":0,"num_object_copies":13629,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11231,"num_write_kb":504768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6862,"num_bytes_recovered":449708032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"290'35","reported_seq":413,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.214690+0000","last_change":"2024-09-21T14:01:02.006619+0000","last_active":"2024-09-21T14:05:26.214690+0000","last_peered":"2024-09-21T14:05:26.214690+0000","last_clean":"2024-09-21T14:05:26.214690+0000","last_became_active":"2024-09-21T13:59:32.204184+0000","last_became_peered":"2024-09-21T13:59:32.204184+0000","last_unstale":"2024-09-21T14:05:26.214690+0000","last_undegraded":"2024-09-21T14:05:26.214690+0000","last_fullsized":"2024-09-21T14:05:26.214690+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'35","last_scrub_stamp":"2024-09-21T14:01:02.006504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:02.006504+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:29.829621+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":766,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.214394+0000","last_change":"2024-09-21T14:03:31.882264+0000","last_active":"2024-09-21T14:05:26.214394+0000","last_peered":"2024-09-21T14:05:26.214394+0000","last_clean":"2024-09-21T14:05:26.214394+0000","last_became_active":"2024-09-21T14:01:41.184567+0000","last_became_peered":"2024-09-21T14:01:41.184567+0000","last_unstale":"2024-09-21T14:05:26.214394+0000","last_undegraded":"2024-09-21T14:05:26.214394+0000","last_fullsized":"2024-09-21T14:05:26.214394+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:31.882215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:03:31.882215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":744,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.191205+0000","last_change":"2024-09-21T14:04:48.685009+0000","last_active":"2024-09-21T14:05:26.191205+0000","last_peered":"2024-09-21T14:05:26.191205+0000","last_clean":"2024-09-21T14:05:26.191205+0000","last_became_active":"2024-09-21T14:01:41.181911+0000","last_became_peered":"2024-09-21T14:01:41.181911+0000","last_unstale":"2024-09-21T14:05:26.191205+0000","last_undegraded":"2024-09-21T14:05:26.191205+0000","last_fullsized":"2024-09-21T14:05:26.191205+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:48.684941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:04:48.684941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:08.622354+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":693,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.191154+0000","last_change":"2024-09-21T14:03:33.893158+0000","last_active":"2024-09-21T14:05:26.191154+0000","last_peered":"2024-09-21T14:05:26.191154+0000","last_clean":"2024-09-21T14:05:26.191154+0000","last_became_active":"2024-09-21T13:59:32.219332+0000","last_became_peered":"2024-09-21T13:59:32.219332+0000","last_unstale":"2024-09-21T14:05:26.191154+0000","last_undegraded":"2024-09-21T14:05:26.191154+0000","last_fullsized":"2024-09-21T14:05:26.191154+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:33.893072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:33.893072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"428'11397","reported_seq":11590,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:28.798123+0000","last_change":"2024-09-21T14:04:40.683339+0000","last_active":"2024-09-21T14:05:28.798123+0000","last_peered":"2024-09-21T14:05:28.798123+0000","last_clean":"2024-09-21T14:05:28.798123+0000","last_became_active":"2024-09-21T14:02:56.559331+0000","last_became_peered":"2024-09-21T14:02:56.559331+0000","last_unstale":"2024-09-21T14:05:28.798123+0000","last_undegraded":"2024-09-21T14:05:28.798123+0000","last_fullsized":"2024-09-21T14:05:28.798123+0000","mapping_epoch":381,"log_start":"406'6290","ondisk_log_start":"406'6290","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'8371","last_scrub_stamp":"2024-09-21T14:04:40.683117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:04:40.683117+0000","objects_scrubbed":7905,"log_size":5107,"log_dups_size":0,"ondisk_log_size":5107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:47.784341+0000","scrub_duration":2828,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":313720832,"num_objects":4787,"num_object_clones":0,"num_object_copies":14361,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11397,"num_write_kb":517888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"290'18","reported_seq":830,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.248051+0000","last_change":"2024-09-21T14:05:23.423056+0000","last_active":"2024-09-21T14:05:26.248051+0000","last_peered":"2024-09-21T14:05:26.248051+0000","last_clean":"2024-09-21T14:05:26.248051+0000","last_became_active":"2024-09-21T13:57:12.680627+0000","last_became_peered":"2024-09-21T13:57:12.680627+0000","last_unstale":"2024-09-21T14:05:26.248051+0000","last_undegraded":"2024-09-21T14:05:26.248051+0000","last_fullsized":"2024-09-21T14:05:26.248051+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'18","last_scrub_stamp":"2024-09-21T14:05:23.420582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:05:23.420582+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:45.787887+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":720,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.214599+0000","last_change":"2024-09-21T14:04:21.206034+0000","last_active":"2024-09-21T14:05:26.214599+0000","last_peered":"2024-09-21T14:05:26.214599+0000","last_clean":"2024-09-21T14:05:26.214599+0000","last_became_active":"2024-09-21T14:04:21.205699+0000","last_became_peered":"2024-09-21T14:04:21.205699+0000","last_unstale":"2024-09-21T14:05:26.214599+0000","last_undegraded":"2024-09-21T14:05:26.214599+0000","last_fullsized":"2024-09-21T14:05:26.214599+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:21.582325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:21.582325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":699,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.190929+0000","last_change":"2024-09-21T14:02:27.791041+0000","last_active":"2024-09-21T14:05:26.190929+0000","last_peered":"2024-09-21T14:05:26.190929+0000","last_clean":"2024-09-21T14:05:26.190929+0000","last_became_active":"2024-09-21T13:52:04.322147+0000","last_became_peered":"2024-09-21T13:52:04.322147+0000","last_unstale":"2024-09-21T14:05:26.190929+0000","last_undegraded":"2024-09-21T14:05:26.190929+0000","last_fullsized":"2024-09-21T14:05:26.190929+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:27.790994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:02:27.790994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":707,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.168962+0000","last_change":"2024-09-21T14:02:36.371955+0000","last_active":"2024-09-21T14:05:26.168962+0000","last_peered":"2024-09-21T14:05:26.168962+0000","last_clean":"2024-09-21T14:05:26.168962+0000","last_became_active":"2024-09-21T13:59:32.203767+0000","last_became_peered":"2024-09-21T13:59:32.203767+0000","last_unstale":"2024-09-21T14:05:26.168962+0000","last_undegraded":"2024-09-21T14:05:26.168962+0000","last_fullsized":"2024-09-21T14:05:26.168962+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:36.371747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:02:36.371747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"427'11096","reported_seq":11280,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T14:05:25.485056+0000","last_change":"2024-09-21T14:04:52.260111+0000","last_active":"2024-09-21T14:05:25.485056+0000","last_peered":"2024-09-21T14:05:25.485056+0000","last_clean":"2024-09-21T14:05:25.485056+0000","last_became_active":"2024-09-21T14:02:56.563788+0000","last_became_peered":"2024-09-21T14:02:56.563788+0000","last_unstale":"2024-09-21T14:05:25.485056+0000","last_undegraded":"2024-09-21T14:05:25.485056+0000","last_fullsized":"2024-09-21T14:05:25.485056+0000","mapping_epoch":381,"log_start":"406'6237","ondisk_log_start":"406'6237","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'9041","last_scrub_stamp":"2024-09-21T14:04:52.260025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:04:52.260025+0000","objects_scrubbed":7053,"log_size":4859,"log_dups_size":0,"ondisk_log_size":4859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:57.305832+0000","scrub_duration":1833,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324141056,"num_objects":4946,"num_object_clones":0,"num_object_copies":14838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11096,"num_write_kb":513344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"290'27","reported_seq":826,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T14:05:25.160536+0000","last_change":"2024-09-21T14:05:17.381123+0000","last_active":"2024-09-21T14:05:25.160536+0000","last_peered":"2024-09-21T14:05:25.160536+0000","last_clean":"2024-09-21T14:05:25.160536+0000","last_became_active":"2024-09-21T14:01:41.192481+0000","last_became_peered":"2024-09-21T14:01:41.192481+0000","last_unstale":"2024-09-21T14:05:25.160536+0000","last_undegraded":"2024-09-21T14:05:25.160536+0000","last_fullsized":"2024-09-21T14:05:25.160536+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'27","last_scrub_stamp":"2024-09-21T14:05:17.381049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:05:17.381049+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:32.455335+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":706,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.214200+0000","last_change":"2024-09-21T14:04:21.218814+0000","last_active":"2024-09-21T14:05:26.214200+0000","last_peered":"2024-09-21T14:05:26.214200+0000","last_clean":"2024-09-21T14:05:26.214200+0000","last_became_active":"2024-09-21T14:04:21.201644+0000","last_became_peered":"2024-09-21T14:04:21.201644+0000","last_unstale":"2024-09-21T14:05:26.214200+0000","last_undegraded":"2024-09-21T14:05:26.214200+0000","last_fullsized":"2024-09-21T14:05:26.214200+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T14:03:04.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:03:04.953658+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"300'14","reported_seq":930,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.190953+0000","last_change":"2024-09-21T13:59:32.220898+0000","last_active":"2024-09-21T14:05:26.190953+0000","last_peered":"2024-09-21T14:05:26.190953+0000","last_clean":"2024-09-21T14:05:26.190953+0000","last_became_active":"2024-09-21T13:59:32.206956+0000","last_became_peered":"2024-09-21T13:59:32.206956+0000","last_unstale":"2024-09-21T14:05:26.190953+0000","last_undegraded":"2024-09-21T14:05:26.190953+0000","last_fullsized":"2024-09-21T14:05:26.190953+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'11","last_scrub_stamp":"2024-09-21T13:56:31.616003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:56:31.616003+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:27.198182+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":766,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T14:05:25.162106+0000","last_change":"2024-09-21T14:04:42.433805+0000","last_active":"2024-09-21T14:05:25.162106+0000","last_peered":"2024-09-21T14:05:25.162106+0000","last_clean":"2024-09-21T14:05:25.162106+0000","last_became_active":"2024-09-21T14:01:41.246106+0000","last_became_peered":"2024-09-21T14:01:41.246106+0000","last_unstale":"2024-09-21T14:05:25.162106+0000","last_undegraded":"2024-09-21T14:05:25.162106+0000","last_fullsized":"2024-09-21T14:05:25.162106+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:42.433729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:04:42.433729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:04.041353+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":357,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.190228+0000","last_change":"2024-09-21T14:04:21.198859+0000","last_active":"2024-09-21T14:05:26.190228+0000","last_peered":"2024-09-21T14:05:26.190228+0000","last_clean":"2024-09-21T14:05:26.190228+0000","last_became_active":"2024-09-21T14:04:21.197402+0000","last_became_peered":"2024-09-21T14:04:21.197402+0000","last_unstale":"2024-09-21T14:05:26.190228+0000","last_undegraded":"2024-09-21T14:05:26.190228+0000","last_fullsized":"2024-09-21T14:05:26.190228+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:50.733007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:01:50.733007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:31.764576+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"290'12","reported_seq":734,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.169360+0000","last_change":"2024-09-21T13:59:32.247422+0000","last_active":"2024-09-21T14:05:26.169360+0000","last_peered":"2024-09-21T14:05:26.169360+0000","last_clean":"2024-09-21T14:05:26.169360+0000","last_became_active":"2024-09-21T13:59:32.211306+0000","last_became_peered":"2024-09-21T13:59:32.211306+0000","last_unstale":"2024-09-21T14:05:26.169360+0000","last_undegraded":"2024-09-21T14:05:26.169360+0000","last_fullsized":"2024-09-21T14:05:26.169360+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'6","last_scrub_stamp":"2024-09-21T13:55:12.066873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:55:12.066873+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":698,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T14:05:25.161451+0000","last_change":"2024-09-21T14:05:19.378454+0000","last_active":"2024-09-21T14:05:25.161451+0000","last_peered":"2024-09-21T14:05:25.161451+0000","last_clean":"2024-09-21T14:05:25.161451+0000","last_became_active":"2024-09-21T14:01:41.199321+0000","last_became_peered":"2024-09-21T14:01:41.199321+0000","last_unstale":"2024-09-21T14:05:25.161451+0000","last_undegraded":"2024-09-21T14:05:25.161451+0000","last_fullsized":"2024-09-21T14:05:25.161451+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:19.378393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:05:19.378393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:35.761731+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":739,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.214230+0000","last_change":"2024-09-21T14:03:01.872986+0000","last_active":"2024-09-21T14:05:26.214230+0000","last_peered":"2024-09-21T14:05:26.214230+0000","last_clean":"2024-09-21T14:05:26.214230+0000","last_became_active":"2024-09-21T13:59:32.219525+0000","last_became_peered":"2024-09-21T13:59:32.219525+0000","last_unstale":"2024-09-21T14:05:26.214230+0000","last_undegraded":"2024-09-21T14:05:26.214230+0000","last_fullsized":"2024-09-21T14:05:26.214230+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:01.872928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:01.872928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"290'15","reported_seq":714,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.168918+0000","last_change":"2024-09-21T14:04:21.228883+0000","last_active":"2024-09-21T14:05:26.168918+0000","last_peered":"2024-09-21T14:05:26.168918+0000","last_clean":"2024-09-21T14:05:26.168918+0000","last_became_active":"2024-09-21T14:04:21.208398+0000","last_became_peered":"2024-09-21T14:04:21.208398+0000","last_unstale":"2024-09-21T14:05:26.168918+0000","last_undegraded":"2024-09-21T14:05:26.168918+0000","last_fullsized":"2024-09-21T14:05:26.168918+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:55:04.990842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:55:04.990842+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:29.202675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"61'1","reported_seq":757,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.213454+0000","last_change":"2024-09-21T14:04:04.012140+0000","last_active":"2024-09-21T14:05:26.213454+0000","last_peered":"2024-09-21T14:05:26.213454+0000","last_clean":"2024-09-21T14:05:26.213454+0000","last_became_active":"2024-09-21T13:59:32.219515+0000","last_became_peered":"2024-09-21T13:59:32.219515+0000","last_unstale":"2024-09-21T14:05:26.213454+0000","last_undegraded":"2024-09-21T14:05:26.213454+0000","last_fullsized":"2024-09-21T14:05:26.213454+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T14:04:04.012006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:04.012006+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":690,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.169001+0000","last_change":"2024-09-21T14:03:33.309440+0000","last_active":"2024-09-21T14:05:26.169001+0000","last_peered":"2024-09-21T14:05:26.169001+0000","last_clean":"2024-09-21T14:05:26.169001+0000","last_became_active":"2024-09-21T13:59:32.210586+0000","last_became_peered":"2024-09-21T13:59:32.210586+0000","last_unstale":"2024-09-21T14:05:26.169001+0000","last_undegraded":"2024-09-21T14:05:26.169001+0000","last_fullsized":"2024-09-21T14:05:26.169001+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:33.309248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:33.309248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":740,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.191144+0000","last_change":"2024-09-21T14:02:29.731635+0000","last_active":"2024-09-21T14:05:26.191144+0000","last_peered":"2024-09-21T14:05:26.191144+0000","last_clean":"2024-09-21T14:05:26.191144+0000","last_became_active":"2024-09-21T13:59:32.205803+0000","last_became_peered":"2024-09-21T13:59:32.205803+0000","last_unstale":"2024-09-21T14:05:26.191144+0000","last_undegraded":"2024-09-21T14:05:26.191144+0000","last_fullsized":"2024-09-21T14:05:26.191144+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:29.731565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:02:29.731565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"290'15","reported_seq":735,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T14:05:25.161971+0000","last_change":"2024-09-21T14:01:41.189676+0000","last_active":"2024-09-21T14:05:25.161971+0000","last_peered":"2024-09-21T14:05:25.161971+0000","last_clean":"2024-09-21T14:05:25.161971+0000","last_became_active":"2024-09-21T14:01:41.187904+0000","last_became_peered":"2024-09-21T14:01:41.187904+0000","last_unstale":"2024-09-21T14:05:25.161971+0000","last_undegraded":"2024-09-21T14:05:25.161971+0000","last_fullsized":"2024-09-21T14:05:25.161971+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:57:24.124604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:57:24.124604+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:30.424236+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":383,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-09-21T14:05:29.215288+0000","last_change":"2024-09-21T14:04:56.588256+0000","last_active":"2024-09-21T14:05:29.215288+0000","last_peered":"2024-09-21T14:05:29.215288+0000","last_clean":"2024-09-21T14:05:29.215288+0000","last_became_active":"2024-09-21T14:04:56.587953+0000","last_became_peered":"2024-09-21T14:04:56.587953+0000","last_unstale":"2024-09-21T14:05:29.215288+0000","last_undegraded":"2024-09-21T14:05:29.215288+0000","last_fullsized":"2024-09-21T14:05:29.215288+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:37.078458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:37.078458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:52.347012+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"300'14","reported_seq":863,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.169397+0000","last_change":"2024-09-21T13:59:32.208253+0000","last_active":"2024-09-21T14:05:26.169397+0000","last_peered":"2024-09-21T14:05:26.169397+0000","last_clean":"2024-09-21T14:05:26.169397+0000","last_became_active":"2024-09-21T13:59:32.206926+0000","last_became_peered":"2024-09-21T13:59:32.206926+0000","last_unstale":"2024-09-21T14:05:26.169397+0000","last_undegraded":"2024-09-21T14:05:26.169397+0000","last_fullsized":"2024-09-21T14:05:26.169397+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'10","last_scrub_stamp":"2024-09-21T13:54:20.946782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:20.946782+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":792,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.248770+0000","last_change":"2024-09-21T14:05:20.368393+0000","last_active":"2024-09-21T14:05:26.248770+0000","last_peered":"2024-09-21T14:05:26.248770+0000","last_clean":"2024-09-21T14:05:26.248770+0000","last_became_active":"2024-09-21T14:01:41.200113+0000","last_became_peered":"2024-09-21T14:01:41.200113+0000","last_unstale":"2024-09-21T14:05:26.248770+0000","last_undegraded":"2024-09-21T14:05:26.248770+0000","last_fullsized":"2024-09-21T14:05:26.248770+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:20.368325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:05:20.368325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:41.310306+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"290'20","reported_seq":741,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.169192+0000","last_change":"2024-09-21T14:04:04.192280+0000","last_active":"2024-09-21T14:05:26.169192+0000","last_peered":"2024-09-21T14:05:26.169192+0000","last_clean":"2024-09-21T14:05:26.169192+0000","last_became_active":"2024-09-21T13:59:32.208332+0000","last_became_peered":"2024-09-21T13:59:32.208332+0000","last_unstale":"2024-09-21T14:05:26.169192+0000","last_undegraded":"2024-09-21T14:05:26.169192+0000","last_fullsized":"2024-09-21T14:05:26.169192+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'20","last_scrub_stamp":"2024-09-21T14:04:04.192193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:04:04.192193+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":725,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.191174+0000","last_change":"2024-09-21T13:59:32.212737+0000","last_active":"2024-09-21T14:05:26.191174+0000","last_peered":"2024-09-21T14:05:26.191174+0000","last_clean":"2024-09-21T14:05:26.191174+0000","last_became_active":"2024-09-21T13:59:32.209804+0000","last_became_peered":"2024-09-21T13:59:32.209804+0000","last_unstale":"2024-09-21T14:05:26.191174+0000","last_undegraded":"2024-09-21T14:05:26.191174+0000","last_fullsized":"2024-09-21T14:05:26.191174+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:25.571256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:25.571256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:29.134483+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"346'10","reported_seq":877,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T14:05:25.161869+0000","last_change":"2024-09-21T14:05:21.397197+0000","last_active":"2024-09-21T14:05:25.161869+0000","last_peered":"2024-09-21T14:05:25.161869+0000","last_clean":"2024-09-21T14:05:25.161869+0000","last_became_active":"2024-09-21T14:01:41.206303+0000","last_became_peered":"2024-09-21T14:01:41.206303+0000","last_unstale":"2024-09-21T14:05:25.161869+0000","last_undegraded":"2024-09-21T14:05:25.161869+0000","last_fullsized":"2024-09-21T14:05:25.161869+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'10","last_scrub_stamp":"2024-09-21T14:05:21.397143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:05:21.397143+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:36.172088+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":637,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.214460+0000","last_change":"2024-09-21T14:04:21.202582+0000","last_active":"2024-09-21T14:05:26.214460+0000","last_peered":"2024-09-21T14:05:26.214460+0000","last_clean":"2024-09-21T14:05:26.214460+0000","last_became_active":"2024-09-21T14:04:21.201277+0000","last_became_peered":"2024-09-21T14:04:21.201277+0000","last_unstale":"2024-09-21T14:05:26.214460+0000","last_undegraded":"2024-09-21T14:05:26.214460+0000","last_fullsized":"2024-09-21T14:05:26.214460+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:27.961544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:27.961544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"290'19","reported_seq":856,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.247804+0000","last_change":"2024-09-21T14:04:05.139089+0000","last_active":"2024-09-21T14:05:26.247804+0000","last_peered":"2024-09-21T14:05:26.247804+0000","last_clean":"2024-09-21T14:05:26.247804+0000","last_became_active":"2024-09-21T13:58:17.771497+0000","last_became_peered":"2024-09-21T13:58:17.771497+0000","last_unstale":"2024-09-21T14:05:26.247804+0000","last_undegraded":"2024-09-21T14:05:26.247804+0000","last_fullsized":"2024-09-21T14:05:26.247804+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'19","last_scrub_stamp":"2024-09-21T14:04:05.138989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:04:05.138989+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":367,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.213692+0000","last_change":"2024-09-21T14:02:57.807136+0000","last_active":"2024-09-21T14:05:26.213692+0000","last_peered":"2024-09-21T14:05:26.213692+0000","last_clean":"2024-09-21T14:05:26.213692+0000","last_became_active":"2024-09-21T13:59:32.201568+0000","last_became_peered":"2024-09-21T13:59:32.201568+0000","last_unstale":"2024-09-21T14:05:26.213692+0000","last_undegraded":"2024-09-21T14:05:26.213692+0000","last_fullsized":"2024-09-21T14:05:26.213692+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:57.807059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:02:57.807059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":808,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-09-21T14:05:29.212400+0000","last_change":"2024-09-21T14:05:19.308488+0000","last_active":"2024-09-21T14:05:29.212400+0000","last_peered":"2024-09-21T14:05:29.212400+0000","last_clean":"2024-09-21T14:05:29.212400+0000","last_became_active":"2024-09-21T14:01:41.185293+0000","last_became_peered":"2024-09-21T14:01:41.185293+0000","last_unstale":"2024-09-21T14:05:29.212400+0000","last_undegraded":"2024-09-21T14:05:29.212400+0000","last_fullsized":"2024-09-21T14:05:29.212400+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:19.308311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:05:19.308311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:42.164269+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":705,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.167823+0000","last_change":"2024-09-21T14:04:21.209188+0000","last_active":"2024-09-21T14:05:26.167823+0000","last_peered":"2024-09-21T14:05:26.167823+0000","last_clean":"2024-09-21T14:05:26.167823+0000","last_became_active":"2024-09-21T14:04:21.208329+0000","last_became_peered":"2024-09-21T14:04:21.208329+0000","last_unstale":"2024-09-21T14:05:26.167823+0000","last_undegraded":"2024-09-21T14:05:26.167823+0000","last_fullsized":"2024-09-21T14:05:26.167823+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:57.225602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:02:57.225602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"290'15","reported_seq":860,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-09-21T14:05:29.212376+0000","last_change":"2024-09-21T14:04:12.139406+0000","last_active":"2024-09-21T14:05:29.212376+0000","last_peered":"2024-09-21T14:05:29.212376+0000","last_clean":"2024-09-21T14:05:29.212376+0000","last_became_active":"2024-09-21T13:58:17.764120+0000","last_became_peered":"2024-09-21T13:58:17.764120+0000","last_unstale":"2024-09-21T14:05:29.212376+0000","last_undegraded":"2024-09-21T14:05:29.212376+0000","last_fullsized":"2024-09-21T14:05:29.212376+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'15","last_scrub_stamp":"2024-09-21T14:04:12.139190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:04:12.139190+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:40.181606+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":737,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.214067+0000","last_change":"2024-09-21T14:04:07.038058+0000","last_active":"2024-09-21T14:05:26.214067+0000","last_peered":"2024-09-21T14:05:26.214067+0000","last_clean":"2024-09-21T14:05:26.214067+0000","last_became_active":"2024-09-21T13:59:32.221854+0000","last_became_peered":"2024-09-21T13:59:32.221854+0000","last_unstale":"2024-09-21T14:05:26.214067+0000","last_undegraded":"2024-09-21T14:05:26.214067+0000","last_fullsized":"2024-09-21T14:05:26.214067+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:07.037989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:07.037989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:34.939994+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":673,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.168338+0000","last_change":"2024-09-21T14:03:32.262147+0000","last_active":"2024-09-21T14:05:26.168338+0000","last_peered":"2024-09-21T14:05:26.168338+0000","last_clean":"2024-09-21T14:05:26.168338+0000","last_became_active":"2024-09-21T13:57:12.703441+0000","last_became_peered":"2024-09-21T13:57:12.703441+0000","last_unstale":"2024-09-21T14:05:26.168338+0000","last_undegraded":"2024-09-21T14:05:26.168338+0000","last_fullsized":"2024-09-21T14:05:26.168338+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:32.262067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:32.262067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":447,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-09-21T14:05:29.214549+0000","last_change":"2024-09-21T14:04:56.588351+0000","last_active":"2024-09-21T14:05:29.214549+0000","last_peered":"2024-09-21T14:05:29.214549+0000","last_clean":"2024-09-21T14:05:29.214549+0000","last_became_active":"2024-09-21T14:04:56.587503+0000","last_became_peered":"2024-09-21T14:04:56.587503+0000","last_unstale":"2024-09-21T14:05:29.214549+0000","last_undegraded":"2024-09-21T14:05:29.214549+0000","last_fullsized":"2024-09-21T14:05:29.214549+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:29.345427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:29.345427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:05.186916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"290'19","reported_seq":809,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.168088+0000","last_change":"2024-09-21T13:59:32.277742+0000","last_active":"2024-09-21T14:05:26.168088+0000","last_peered":"2024-09-21T14:05:26.168088+0000","last_clean":"2024-09-21T14:05:26.168088+0000","last_became_active":"2024-09-21T13:59:32.231567+0000","last_became_peered":"2024-09-21T13:59:32.231567+0000","last_unstale":"2024-09-21T14:05:26.168088+0000","last_undegraded":"2024-09-21T14:05:26.168088+0000","last_fullsized":"2024-09-21T14:05:26.168088+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:54:22.978267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:22.978267+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":703,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-09-21T14:05:25.161535+0000","last_change":"2024-09-21T14:05:13.355232+0000","last_active":"2024-09-21T14:05:25.161535+0000","last_peered":"2024-09-21T14:05:25.161535+0000","last_clean":"2024-09-21T14:05:25.161535+0000","last_became_active":"2024-09-21T14:01:41.190760+0000","last_became_peered":"2024-09-21T14:01:41.190760+0000","last_unstale":"2024-09-21T14:05:25.161535+0000","last_undegraded":"2024-09-21T14:05:25.161535+0000","last_fullsized":"2024-09-21T14:05:25.161535+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:13.355183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:05:13.355183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:37.299500+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":743,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.213442+0000","last_change":"2024-09-21T14:04:41.091577+0000","last_active":"2024-09-21T14:05:26.213442+0000","last_peered":"2024-09-21T14:05:26.213442+0000","last_clean":"2024-09-21T14:05:26.213442+0000","last_became_active":"2024-09-21T13:59:32.214004+0000","last_became_peered":"2024-09-21T13:59:32.214004+0000","last_unstale":"2024-09-21T14:05:26.213442+0000","last_undegraded":"2024-09-21T14:05:26.213442+0000","last_fullsized":"2024-09-21T14:05:26.213442+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:41.091525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:04:41.091525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:06.917055+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":738,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.168554+0000","last_change":"2024-09-21T14:04:21.214685+0000","last_active":"2024-09-21T14:05:26.168554+0000","last_peered":"2024-09-21T14:05:26.168554+0000","last_clean":"2024-09-21T14:05:26.168554+0000","last_became_active":"2024-09-21T14:04:21.213307+0000","last_became_peered":"2024-09-21T14:04:21.213307+0000","last_unstale":"2024-09-21T14:05:26.168554+0000","last_undegraded":"2024-09-21T14:05:26.168554+0000","last_fullsized":"2024-09-21T14:05:26.168554+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:28.124296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:28.124296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"290'8","reported_seq":755,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.167923+0000","last_change":"2024-09-21T13:59:26.034457+0000","last_active":"2024-09-21T14:05:26.167923+0000","last_peered":"2024-09-21T14:05:26.167923+0000","last_clean":"2024-09-21T14:05:26.167923+0000","last_became_active":"2024-09-21T13:58:17.775079+0000","last_became_peered":"2024-09-21T13:58:17.775079+0000","last_unstale":"2024-09-21T14:05:26.167923+0000","last_undegraded":"2024-09-21T14:05:26.167923+0000","last_fullsized":"2024-09-21T14:05:26.167923+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'8","last_scrub_stamp":"2024-09-21T13:59:26.034372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:26.034372+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":518,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-09-21T14:05:29.214150+0000","last_change":"2024-09-21T14:04:56.587839+0000","last_active":"2024-09-21T14:05:29.214150+0000","last_peered":"2024-09-21T14:05:29.214150+0000","last_clean":"2024-09-21T14:05:29.214150+0000","last_became_active":"2024-09-21T14:04:56.587555+0000","last_became_peered":"2024-09-21T14:04:56.587555+0000","last_unstale":"2024-09-21T14:05:29.214150+0000","last_undegraded":"2024-09-21T14:05:29.214150+0000","last_fullsized":"2024-09-21T14:05:29.214150+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:34.387367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:34.387367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:38.961058+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":373,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.190046+0000","last_change":"2024-09-21T14:04:41.932290+0000","last_active":"2024-09-21T14:05:26.190046+0000","last_peered":"2024-09-21T14:05:26.190046+0000","last_clean":"2024-09-21T14:05:26.190046+0000","last_became_active":"2024-09-21T14:04:21.204954+0000","last_became_peered":"2024-09-21T14:04:21.204954+0000","last_unstale":"2024-09-21T14:05:26.190046+0000","last_undegraded":"2024-09-21T14:05:26.190046+0000","last_fullsized":"2024-09-21T14:05:26.190046+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:41.932233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:04:41.932233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:03.486134+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":676,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.190959+0000","last_change":"2024-09-21T14:04:15.135536+0000","last_active":"2024-09-21T14:05:26.190959+0000","last_peered":"2024-09-21T14:05:26.190959+0000","last_clean":"2024-09-21T14:05:26.190959+0000","last_became_active":"2024-09-21T13:59:32.210134+0000","last_became_peered":"2024-09-21T13:59:32.210134+0000","last_unstale":"2024-09-21T14:05:26.190959+0000","last_undegraded":"2024-09-21T14:05:26.190959+0000","last_fullsized":"2024-09-21T14:05:26.190959+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:15.135463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:04:15.135463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:37.791722+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":784,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-09-21T14:05:29.212345+0000","last_change":"2024-09-21T14:01:41.187792+0000","last_active":"2024-09-21T14:05:29.212345+0000","last_peered":"2024-09-21T14:05:29.212345+0000","last_clean":"2024-09-21T14:05:29.212345+0000","last_became_active":"2024-09-21T14:01:41.187438+0000","last_became_peered":"2024-09-21T14:01:41.187438+0000","last_unstale":"2024-09-21T14:05:29.212345+0000","last_undegraded":"2024-09-21T14:05:29.212345+0000","last_fullsized":"2024-09-21T14:05:29.212345+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:17.706325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:17.706325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":726,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.168017+0000","last_change":"2024-09-21T14:04:33.252571+0000","last_active":"2024-09-21T14:05:26.168017+0000","last_peered":"2024-09-21T14:05:26.168017+0000","last_clean":"2024-09-21T14:05:26.168017+0000","last_became_active":"2024-09-21T14:04:21.208121+0000","last_became_peered":"2024-09-21T14:04:21.208121+0000","last_unstale":"2024-09-21T14:05:26.168017+0000","last_undegraded":"2024-09-21T14:05:26.168017+0000","last_fullsized":"2024-09-21T14:05:26.168017+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:33.252481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:33.252481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:48.348987+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"290'19","reported_seq":390,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-21T14:05:26.215096+0000","last_change":"2024-09-21T14:03:31.762681+0000","last_active":"2024-09-21T14:05:26.215096+0000","last_peered":"2024-09-21T14:05:26.215096+0000","last_clean":"2024-09-21T14:05:26.215096+0000","last_became_active":"2024-09-21T14:01:41.194606+0000","last_became_peered":"2024-09-21T14:01:41.194606+0000","last_unstale":"2024-09-21T14:05:26.215096+0000","last_undegraded":"2024-09-21T14:05:26.215096+0000","last_fullsized":"2024-09-21T14:05:26.215096+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'19","last_scrub_stamp":"2024-09-21T14:03:31.762605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:03:31.762605+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":402,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-09-21T14:05:29.214184+0000","last_change":"2024-09-21T14:04:56.591137+0000","last_active":"2024-09-21T14:05:29.214184+0000","last_peered":"2024-09-21T14:05:29.214184+0000","last_clean":"2024-09-21T14:05:29.214184+0000","last_became_active":"2024-09-21T14:04:56.590829+0000","last_became_peered":"2024-09-21T14:04:56.590829+0000","last_unstale":"2024-09-21T14:05:29.214184+0000","last_undegraded":"2024-09-21T14:05:29.214184+0000","last_fullsized":"2024-09-21T14:05:29.214184+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:51.121361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:04:51.121361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:59.958121+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":4937220120,"num_objects":75337,"num_object_clones":0,"num_object_copies":226011,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1579,"num_objects_unfound":0,"num_objects_dirty":75337,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":179813,"num_write_kb":8164737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22136,"num_bytes_recovered":1450639384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16000565248,"data_stored":16000548960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83391,"ondisk_log_size":83391,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102,"ondisk_log_size":102,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1107,"num_read_kb":864,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":197,"num_bytes_recovered":2794,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":687,"ondisk_log_size":687,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":360,"num_read_kb":355,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":751377,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":32,"num_read_kb":59,"num_write":32,"num_write_kb":444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":298,"seq":1279900254280,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1967652,"kb_used_data":1555468,"kb_used_omap":4,"kb_used_meta":412155,"kb_avail":91777500,"statfs":{"total":95995035648,"available":93980160000,"internally_reserved":0,"allocated":1592799232,"data_stored":1592449419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":422047400},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463120,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2540048,"kb_used_data":2115968,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":91205104,"statfs":{"total":95995035648,"available":93394026496,"internally_reserved":0,"allocated":2166751232,"data_stored":2166420696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":434239942},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,5],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":345,"seq":1481763717166,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1751112,"kb_used_data":1561780,"kb_used_omap":4,"kb_used_meta":189307,"kb_avail":91994040,"statfs":{"total":95995035648,"available":94201896960,"internally_reserved":0,"allocated":1599262720,"data_stored":1598937592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":193851043},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920989,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2916888,"kb_used_data":2402568,"kb_used_omap":1,"kb_used_meta":514302,"kb_avail":90828264,"statfs":{"total":95995035648,"available":93008142336,"internally_reserved":0,"allocated":2460229632,"data_stored":2459890990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":526645701},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":240,"seq":1030792151140,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2215928,"kb_used_data":1794912,"kb_used_omap":4,"kb_used_meta":420987,"kb_avail":91529224,"statfs":{"total":95995035648,"available":93725925376,"internally_reserved":0,"allocated":1837989888,"data_stored":1837628420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4443,"internal_metadata":431091365},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":130,"seq":558345748642,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2784532,"kb_used_data":2072068,"kb_used_omap":4,"kb_used_meta":712443,"kb_avail":90960620,"statfs":{"total":95995035648,"available":93143674880,"internally_reserved":0,"allocated":2121797632,"data_stored":2121464412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4441,"internal_metadata":729542311},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575087,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1867744,"kb_used_data":1178508,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":91877408,"statfs":{"total":95995035648,"available":94082465792,"internally_reserved":0,"allocated":1206792192,"data_stored":1206393581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738613,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3679880,"kb_used_data":3012852,"kb_used_omap":1,"kb_used_meta":667006,"kb_avail":90065272,"statfs":{"total":95995035648,"available":92226838528,"internally_reserved":0,"allocated":3085160448,"data_stored":3084849969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":683014597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":2448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3077771264,"data_stored":3077767192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1198260224,"data_stored":1198260224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2113077248,"data_stored":2113077248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1835929600,"data_stored":1835925528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2447904768,"data_stored":2447900696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1597177856,"data_stored":1597177856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2140803072,"data_stored":2140799000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1589641216,"data_stored":1589641216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T14:05:32.504 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-21T14:05:32.505 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd pool set .mgr pg_num 21 2024-09-21T14:05:32.571 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:05:32.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:32.627 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:32.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:05:32.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:32 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3542378480' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:05:32.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:32 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:05:32.727 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:05:32.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:32.771 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:32.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:05:32.872 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:32.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:32.906 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:32.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:33.007 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:33.045 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:33.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:33.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:33.148 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:05:33.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:33.179 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:33.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:05:33.280 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:05:33.324 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:33.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:33.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:05:33.369 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:33 smithi049 ceph-mon[30080]: pgmap v841: 158 pgs: 1 peering, 2 active+remapped+backfilling, 155 active+clean; 4.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 840 op/s; 1579/223884 objects misplaced (0.705%); 21 MiB/s, 331 objects/s recovering 2024-09-21T14:05:33.369 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:33 smithi049 ceph-mon[30080]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:05:33.369 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:33 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T14:05:33.369 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:33 smithi049 ceph-mon[30080]: osdmap e431: 8 total, 8 up, 8 in 2024-09-21T14:05:33.371 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:33 smithi049 ceph-mon[40489]: pgmap v841: 158 pgs: 1 peering, 2 active+remapped+backfilling, 155 active+clean; 4.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 840 op/s; 1579/223884 objects misplaced (0.705%); 21 MiB/s, 331 objects/s recovering 2024-09-21T14:05:33.371 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:33 smithi049 ceph-mon[40489]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:05:33.371 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:33 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T14:05:33.371 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:33 smithi049 ceph-mon[40489]: osdmap e431: 8 total, 8 up, 8 in 2024-09-21T14:05:33.426 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:33.455 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:33.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:33.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:33.557 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:33.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:33.593 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:33.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:33.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:33 smithi187 ceph-mon[33032]: pgmap v841: 158 pgs: 1 peering, 2 active+remapped+backfilling, 155 active+clean; 4.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 840 op/s; 1579/223884 objects misplaced (0.705%); 21 MiB/s, 331 objects/s recovering 2024-09-21T14:05:33.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:33 smithi187 ceph-mon[33032]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:05:33.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:33 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T14:05:33.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:33 smithi187 ceph-mon[33032]: osdmap e431: 8 total, 8 up, 8 in 2024-09-21T14:05:33.695 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:05:33.729 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:33.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:33.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:05:33.831 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:05:33.874 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:33.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:33.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:05:33.951 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:05:33.974 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T14:05:33.976 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:05:34.015 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:05:34.036 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:34.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:34.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:05:34.138 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:34.166 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:34.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:34.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:34.267 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:34.294 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:34.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:34.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:34.396 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:05:34.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:34.432 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:34.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:05:34.533 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:05:34.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:34.575 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:34.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:05:34.676 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:05:34.707 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:34.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:34.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:05:34.791 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:05:34.791 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:05:34.809 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:05:34.826 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:05:34.827 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:05:34.858 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:34.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:34.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:05:34.862 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:05:34.863 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:05:34.895 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:05:34.895 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:05:34.934 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:05:34.934 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:05:34.959 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:05:34.975 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:05:34.975 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:05:35.002 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:35.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:35.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:05:35.019 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:05:35.020 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:05:35.053 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:05:35.053 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:05:35.104 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:05:35.145 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:35.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:35.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:05:35.246 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:35.274 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:35.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:35.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:35.375 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:05:35.402 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:35.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:35.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:05:35.422 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T14:05:35.503 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:05:35.525 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:35.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:35.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:05:35.627 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:05:35.651 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:35.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:35.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:05:35.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:35 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:05:35.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:35 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:05:35.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:35 smithi187 ceph-mon[33032]: pgmap v843: 158 pgs: 1 peering, 2 active+remapped+backfilling, 155 active+clean; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.33k op/s; 402/212505 objects misplaced (0.189%); 32 MiB/s, 513 objects/s recovering 2024-09-21T14:05:35.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:35 smithi187 ceph-mon[33032]: osdmap e432: 8 total, 8 up, 8 in 2024-09-21T14:05:35.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:35 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2298066462' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:05:35.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:35 smithi187 ceph-mon[33032]: osdmap e433: 8 total, 8 up, 8 in 2024-09-21T14:05:35.752 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:05:35.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:35 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:05:35.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:35 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:05:35.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:35 smithi049 ceph-mon[30080]: pgmap v843: 158 pgs: 1 peering, 2 active+remapped+backfilling, 155 active+clean; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.33k op/s; 402/212505 objects misplaced (0.189%); 32 MiB/s, 513 objects/s recovering 2024-09-21T14:05:35.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:35 smithi049 ceph-mon[30080]: osdmap e432: 8 total, 8 up, 8 in 2024-09-21T14:05:35.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:35 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2298066462' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:05:35.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:35 smithi049 ceph-mon[30080]: osdmap e433: 8 total, 8 up, 8 in 2024-09-21T14:05:35.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:35 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:05:35.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:35 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:05:35.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:35 smithi049 ceph-mon[40489]: pgmap v843: 158 pgs: 1 peering, 2 active+remapped+backfilling, 155 active+clean; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.33k op/s; 402/212505 objects misplaced (0.189%); 32 MiB/s, 513 objects/s recovering 2024-09-21T14:05:35.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:35 smithi049 ceph-mon[40489]: osdmap e432: 8 total, 8 up, 8 in 2024-09-21T14:05:35.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:35 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2298066462' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:05:35.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:35 smithi049 ceph-mon[40489]: osdmap e433: 8 total, 8 up, 8 in 2024-09-21T14:05:35.774 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:35.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:35.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:05:35.857 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:05:35.860 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:05:35.876 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:05:35.899 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:35.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:35.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:05:36.001 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:36.027 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:36.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:36.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:36.129 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:36.154 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:36.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:36.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:36.162 INFO:teuthology.orchestra.run.smithi049.stderr:set pool 1 pg_num to 21 2024-09-21T14:05:36.256 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:36.293 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:36.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:36.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:36.395 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:05:36.421 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:36.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:36.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:05:36.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:36 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1791855841' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-21T14:05:36.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:36 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1791855841' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-09-21T14:05:36.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:36 smithi049 ceph-mon[30080]: osdmap e434: 8 total, 8 up, 8 in 2024-09-21T14:05:36.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:36 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1791855841' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-21T14:05:36.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:36 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1791855841' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-09-21T14:05:36.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:36 smithi049 ceph-mon[40489]: osdmap e434: 8 total, 8 up, 8 in 2024-09-21T14:05:36.523 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:05:36.545 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:36.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:36.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:05:36.647 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:05:36.668 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:36.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:36.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:05:36.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:36 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1791855841' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-21T14:05:36.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:36 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1791855841' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-09-21T14:05:36.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:36 smithi187 ceph-mon[33032]: osdmap e434: 8 total, 8 up, 8 in 2024-09-21T14:05:36.770 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:36.790 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:36.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:36.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:36.892 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:05:36.913 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:36.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:36.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:05:37.014 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:05:37.035 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:37.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:37.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:05:37.137 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:05:37.160 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:37.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:37.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:05:37.261 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:05:37.276 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:37 smithi049 ceph-mon[30080]: pgmap v846: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 4.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 877 B/s rd, 0 B/s wr, 1.04k op/s; 402/208359 objects misplaced (0.193%); 12 MiB/s, 186 objects/s recovering 2024-09-21T14:05:37.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:37.309 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:37.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:05:37.410 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:05:37.452 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:37.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:37.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:05:37.554 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:05:37.585 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:37.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:37.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:05:37.687 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:37.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:37 smithi187 ceph-mon[33032]: pgmap v846: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 4.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 877 B/s rd, 0 B/s wr, 1.04k op/s; 402/208359 objects misplaced (0.193%); 12 MiB/s, 186 objects/s recovering 2024-09-21T14:05:37.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:37 smithi187 ceph-mon[33032]: osdmap e435: 8 total, 8 up, 8 in 2024-09-21T14:05:37.718 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:37.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:37.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:37.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:37 smithi049 ceph-mon[30080]: osdmap e435: 8 total, 8 up, 8 in 2024-09-21T14:05:37.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:37 smithi049 ceph-mon[40489]: pgmap v846: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 4.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 877 B/s rd, 0 B/s wr, 1.04k op/s; 402/208359 objects misplaced (0.193%); 12 MiB/s, 186 objects/s recovering 2024-09-21T14:05:37.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:37 smithi049 ceph-mon[40489]: osdmap e435: 8 total, 8 up, 8 in 2024-09-21T14:05:37.812 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T14:05:37.820 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:37.847 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:37.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:37.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:37.949 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:05:37.978 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:37.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:37.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:05:38.080 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:05:38.112 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:38.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:38.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:05:38.214 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:05:38.246 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:38.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:38.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:05:38.257 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T14:05:38.257 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-09-21T14:05:38.257 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-09-21T14:05:38.257 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-09-21T14:05:38.257 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T14:05:38.287 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-09-21T14:05:38.348 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:05:38.383 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:38.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:38.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:05:38.484 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:05:38.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:38 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/945832145' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:05:38.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:38 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:05:38.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:38 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-21T14:05:38.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:38 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/945832145' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:05:38.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:38 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:05:38.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:38 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-21T14:05:38.516 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:38.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:38.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:05:38.619 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:05:38.656 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:38.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:38.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:05:38.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:38 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/945832145' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:05:38.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:38 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:05:38.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:38 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-21T14:05:38.758 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:05:38.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:38.802 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:38.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:05:38.903 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:05:38.939 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:38.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:38.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:05:39.041 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:05:39.077 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:39.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:39.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:05:39.179 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:05:39.223 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:39.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:39.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:05:39.325 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:05:39.361 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:39.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:39.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:05:39.463 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:05:39.495 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:39.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:39.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:05:39.595 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:39 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:05:39.595 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:39 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:05:39.595 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:39 smithi049 ceph-mon[30080]: pgmap v849: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 4.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 418 op/s; 402/207501 objects misplaced (0.194%) 2024-09-21T14:05:39.595 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:39 smithi049 ceph-mon[30080]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:05:39.595 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:39 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T14:05:39.595 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:39 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-21T14:05:39.595 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:39 smithi049 ceph-mon[30080]: osdmap e436: 8 total, 8 up, 8 in 2024-09-21T14:05:39.595 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:39 smithi049 ceph-mon[30080]: 1.a scrub starts 2024-09-21T14:05:39.595 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:39 smithi049 ceph-mon[30080]: 1.a scrub ok 2024-09-21T14:05:39.595 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:39 smithi049 ceph-mon[30080]: 4.4 scrub starts 2024-09-21T14:05:39.595 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:39 smithi049 ceph-mon[30080]: 4.4 scrub ok 2024-09-21T14:05:39.596 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:39 smithi049 ceph-mon[30080]: 3.13 scrub starts 2024-09-21T14:05:39.596 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:39 smithi049 ceph-mon[30080]: 3.13 scrub ok 2024-09-21T14:05:39.596 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:39 smithi049 ceph-mon[30080]: 4.a scrub starts 2024-09-21T14:05:39.596 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:39 smithi049 ceph-mon[30080]: 4.a scrub ok 2024-09-21T14:05:39.596 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:39 smithi049 ceph-mon[30080]: osdmap e437: 8 total, 8 up, 8 in 2024-09-21T14:05:39.596 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:39 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:05:39.596 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:39 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:05:39.596 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:39 smithi049 ceph-mon[40489]: pgmap v849: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 4.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 418 op/s; 402/207501 objects misplaced (0.194%) 2024-09-21T14:05:39.596 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:39 smithi049 ceph-mon[40489]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:05:39.596 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:39 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T14:05:39.597 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:39 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-21T14:05:39.597 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:39 smithi049 ceph-mon[40489]: osdmap e436: 8 total, 8 up, 8 in 2024-09-21T14:05:39.597 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:39 smithi049 ceph-mon[40489]: 1.a scrub starts 2024-09-21T14:05:39.597 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:39 smithi049 ceph-mon[40489]: 1.a scrub ok 2024-09-21T14:05:39.597 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:39 smithi049 ceph-mon[40489]: 4.4 scrub starts 2024-09-21T14:05:39.597 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:39 smithi049 ceph-mon[40489]: 4.4 scrub ok 2024-09-21T14:05:39.597 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:39 smithi049 ceph-mon[40489]: 3.13 scrub starts 2024-09-21T14:05:39.597 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:39 smithi049 ceph-mon[40489]: 3.13 scrub ok 2024-09-21T14:05:39.597 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:39 smithi049 ceph-mon[40489]: 4.a scrub starts 2024-09-21T14:05:39.597 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:39 smithi049 ceph-mon[40489]: 4.a scrub ok 2024-09-21T14:05:39.597 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:39 smithi049 ceph-mon[40489]: osdmap e437: 8 total, 8 up, 8 in 2024-09-21T14:05:39.597 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:39.626 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:39.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:39.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:39.659 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T14:05:39.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:39 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:05:39.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:39 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:05:39.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:39 smithi187 ceph-mon[33032]: pgmap v849: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 4.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 418 op/s; 402/207501 objects misplaced (0.194%) 2024-09-21T14:05:39.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:39 smithi187 ceph-mon[33032]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:05:39.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:39 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T14:05:39.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:39 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-21T14:05:39.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:39 smithi187 ceph-mon[33032]: osdmap e436: 8 total, 8 up, 8 in 2024-09-21T14:05:39.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:39 smithi187 ceph-mon[33032]: 1.a scrub starts 2024-09-21T14:05:39.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:39 smithi187 ceph-mon[33032]: 1.a scrub ok 2024-09-21T14:05:39.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:39 smithi187 ceph-mon[33032]: 4.4 scrub starts 2024-09-21T14:05:39.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:39 smithi187 ceph-mon[33032]: 4.4 scrub ok 2024-09-21T14:05:39.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:39 smithi187 ceph-mon[33032]: 3.13 scrub starts 2024-09-21T14:05:39.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:39 smithi187 ceph-mon[33032]: 3.13 scrub ok 2024-09-21T14:05:39.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:39 smithi187 ceph-mon[33032]: 4.a scrub starts 2024-09-21T14:05:39.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:39 smithi187 ceph-mon[33032]: 4.a scrub ok 2024-09-21T14:05:39.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:39 smithi187 ceph-mon[33032]: osdmap e437: 8 total, 8 up, 8 in 2024-09-21T14:05:39.728 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:39.762 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:39.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:39.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:39.864 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:39.896 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:39.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:39.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:39.998 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:40.029 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:40.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:40.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:40.131 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:05:40.166 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:40.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:40.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:05:40.267 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:40.295 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:40.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:40.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:40.306 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:05:39 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:05:40.330 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:05:40.373 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:05:40.397 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:05:40.434 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:40.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:40.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:05:40.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:40 smithi049 ceph-mon[40489]: 5.d scrub starts 2024-09-21T14:05:40.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:40 smithi049 ceph-mon[40489]: 5.d scrub ok 2024-09-21T14:05:40.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:40 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:05:40.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:40 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:05:40.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:40 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-21T14:05:40.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:40 smithi049 ceph-mon[30080]: 5.d scrub starts 2024-09-21T14:05:40.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:40 smithi049 ceph-mon[30080]: 5.d scrub ok 2024-09-21T14:05:40.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:40 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:05:40.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:40 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:05:40.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:40 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-21T14:05:40.536 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:05:40.569 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:40.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:40.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:05:40.570 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:40 smithi187 ceph-mon[33032]: 5.d scrub starts 2024-09-21T14:05:40.570 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:40 smithi187 ceph-mon[33032]: 5.d scrub ok 2024-09-21T14:05:40.570 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:40 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:05:40.570 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:40 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:05:40.570 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:40 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-21T14:05:40.670 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:05:40.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:40.723 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:40.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:05:40.824 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:05:40.840 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:05:40 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:05:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:05:40.872 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:40.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:40.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:05:40.974 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:05:41.012 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:41.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:41.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:05:41.078 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:05:41.078 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:05:41.115 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:05:41.116 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:05:41.116 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:05:41.154 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:41.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:41.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:05:41.160 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:05:41.160 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:05:41.197 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:05:41.197 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:05:41.229 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:05:41.229 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:05:41.258 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:05:41.262 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:05:41.263 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:05:41.289 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:41.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:41.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:05:41.300 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:05:41.300 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:05:41.335 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:05:41.335 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:05:41.391 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:05:41.401 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:41 smithi049 ceph-mon[30080]: 5.6 scrub starts 2024-09-21T14:05:41.401 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:41 smithi049 ceph-mon[30080]: 5.6 scrub ok 2024-09-21T14:05:41.401 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:41 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:05:41.401 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:41 smithi049 ceph-mon[30080]: pgmap v852: 168 pgs: 3 peering, 8 unknown, 157 active+clean; 4.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 910 op/s; 6.2 MiB/s, 98 objects/s recovering 2024-09-21T14:05:41.401 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:41 smithi049 ceph-mon[30080]: 1.4 scrub starts 2024-09-21T14:05:41.401 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:41 smithi049 ceph-mon[30080]: 1.4 scrub ok 2024-09-21T14:05:41.401 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:41 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-21T14:05:41.402 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:41 smithi049 ceph-mon[30080]: 1.1 scrub starts 2024-09-21T14:05:41.402 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:41 smithi049 ceph-mon[30080]: osdmap e438: 8 total, 8 up, 8 in 2024-09-21T14:05:41.402 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:41 smithi049 ceph-mon[30080]: 1.1 scrub ok 2024-09-21T14:05:41.402 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:41 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/44235249' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:05:41.402 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:41 smithi049 ceph-mon[40489]: 5.6 scrub starts 2024-09-21T14:05:41.402 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:41 smithi049 ceph-mon[40489]: 5.6 scrub ok 2024-09-21T14:05:41.402 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:41 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:05:41.402 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:41 smithi049 ceph-mon[40489]: pgmap v852: 168 pgs: 3 peering, 8 unknown, 157 active+clean; 4.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 910 op/s; 6.2 MiB/s, 98 objects/s recovering 2024-09-21T14:05:41.402 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:41 smithi049 ceph-mon[40489]: 1.4 scrub starts 2024-09-21T14:05:41.402 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:41 smithi049 ceph-mon[40489]: 1.4 scrub ok 2024-09-21T14:05:41.403 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:41 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-21T14:05:41.403 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:41 smithi049 ceph-mon[40489]: 1.1 scrub starts 2024-09-21T14:05:41.403 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:41 smithi049 ceph-mon[40489]: osdmap e438: 8 total, 8 up, 8 in 2024-09-21T14:05:41.403 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:41 smithi049 ceph-mon[40489]: 1.1 scrub ok 2024-09-21T14:05:41.403 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:41 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/44235249' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:05:41.434 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:41.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:41.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:05:41.536 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:05:41.576 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:41.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:41.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:05:41.678 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:41.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:41 smithi187 ceph-mon[33032]: 5.6 scrub starts 2024-09-21T14:05:41.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:41 smithi187 ceph-mon[33032]: 5.6 scrub ok 2024-09-21T14:05:41.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:41 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:05:41.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:41 smithi187 ceph-mon[33032]: pgmap v852: 168 pgs: 3 peering, 8 unknown, 157 active+clean; 4.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 910 op/s; 6.2 MiB/s, 98 objects/s recovering 2024-09-21T14:05:41.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:41 smithi187 ceph-mon[33032]: 1.4 scrub starts 2024-09-21T14:05:41.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:41 smithi187 ceph-mon[33032]: 1.4 scrub ok 2024-09-21T14:05:41.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:41 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-21T14:05:41.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:41 smithi187 ceph-mon[33032]: 1.1 scrub starts 2024-09-21T14:05:41.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:41 smithi187 ceph-mon[33032]: osdmap e438: 8 total, 8 up, 8 in 2024-09-21T14:05:41.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:41 smithi187 ceph-mon[33032]: 1.1 scrub ok 2024-09-21T14:05:41.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:41 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/44235249' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:05:41.709 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:41.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:41.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:41.811 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:05:41.845 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:41.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:41.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:05:41.947 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:05:41.984 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:41.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:41.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:05:42.016 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T14:05:42.016 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:05:42.086 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:42.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:42.119 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:42.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:42.220 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:05:42.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:42.262 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:42.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:05:42.312 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T14:05:42.363 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:05:42.415 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:42.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:42.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:05:42.442 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:42 smithi187 ceph-mon[33032]: osdmap e439: 8 total, 8 up, 8 in 2024-09-21T14:05:42.442 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:42 smithi187 ceph-mon[33032]: 1.3 scrub starts 2024-09-21T14:05:42.442 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:42 smithi187 ceph-mon[33032]: 1.3 scrub ok 2024-09-21T14:05:42.442 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:42 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:05:42.442 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:42 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:05:42.442 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:42 smithi187 ceph-mon[33032]: 1.6 scrub starts 2024-09-21T14:05:42.442 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:42 smithi187 ceph-mon[33032]: 1.6 scrub ok 2024-09-21T14:05:42.517 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:05:42.556 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:42.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:42.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:05:42.658 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:05:42.704 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:42.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:42.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:05:42.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:42 smithi049 ceph-mon[40489]: osdmap e439: 8 total, 8 up, 8 in 2024-09-21T14:05:42.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:42 smithi049 ceph-mon[40489]: 1.3 scrub starts 2024-09-21T14:05:42.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:42 smithi049 ceph-mon[40489]: 1.3 scrub ok 2024-09-21T14:05:42.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:42 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:05:42.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:42 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:05:42.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:42 smithi049 ceph-mon[40489]: 1.6 scrub starts 2024-09-21T14:05:42.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:42 smithi049 ceph-mon[40489]: 1.6 scrub ok 2024-09-21T14:05:42.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:42 smithi049 ceph-mon[30080]: osdmap e439: 8 total, 8 up, 8 in 2024-09-21T14:05:42.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:42 smithi049 ceph-mon[30080]: 1.3 scrub starts 2024-09-21T14:05:42.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:42 smithi049 ceph-mon[30080]: 1.3 scrub ok 2024-09-21T14:05:42.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:42 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:05:42.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:42 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:05:42.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:42 smithi049 ceph-mon[30080]: 1.6 scrub starts 2024-09-21T14:05:42.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:42 smithi049 ceph-mon[30080]: 1.6 scrub ok 2024-09-21T14:05:42.806 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:05:42.841 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:42.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:42.842 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T14:05:42.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:05:42.944 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:05:42.986 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:42.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:42.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:05:43.020 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 4, 6, 2, 3, 7, 5] 2024-09-21T14:05:43.021 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T14:05:43.021 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-21T14:05:43.021 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd dump -f json-pretty 2024-09-21T14:05:43.088 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:05:43.139 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:43.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:43.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:05:43.240 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:05:43.290 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:43.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:43.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:05:43.392 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:05:43.464 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:05:43.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:43.494 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:43.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:05:43.595 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:43.596 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:43 smithi187 ceph-mon[33032]: pgmap v855: 168 pgs: 9 peering, 2 unknown, 157 active+clean; 3.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 3.7 KiB/s rd, 18 KiB/s wr, 1.10k op/s; 6.2 MiB/s, 98 objects/s recovering 2024-09-21T14:05:43.596 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:43 smithi187 ceph-mon[33032]: 1.5 scrub starts 2024-09-21T14:05:43.596 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:43 smithi187 ceph-mon[33032]: 1.10 scrub starts 2024-09-21T14:05:43.596 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:43 smithi187 ceph-mon[33032]: 1.5 scrub ok 2024-09-21T14:05:43.597 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:43 smithi187 ceph-mon[33032]: 1.10 scrub ok 2024-09-21T14:05:43.597 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:43 smithi187 ceph-mon[33032]: Health check failed: Reduced data availability: 2 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-09-21T14:05:43.597 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:43 smithi187 ceph-mon[33032]: osdmap e440: 8 total, 8 up, 8 in 2024-09-21T14:05:43.597 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:43 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T14:05:43.632 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:43.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:43.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:43.640 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:43 smithi049 ceph-mon[40489]: pgmap v855: 168 pgs: 9 peering, 2 unknown, 157 active+clean; 3.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 3.7 KiB/s rd, 18 KiB/s wr, 1.10k op/s; 6.2 MiB/s, 98 objects/s recovering 2024-09-21T14:05:43.640 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:43 smithi049 ceph-mon[40489]: 1.5 scrub starts 2024-09-21T14:05:43.640 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:43 smithi049 ceph-mon[40489]: 1.10 scrub starts 2024-09-21T14:05:43.640 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:43 smithi049 ceph-mon[40489]: 1.5 scrub ok 2024-09-21T14:05:43.640 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:43 smithi049 ceph-mon[40489]: 1.10 scrub ok 2024-09-21T14:05:43.640 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:43 smithi049 ceph-mon[40489]: Health check failed: Reduced data availability: 2 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-09-21T14:05:43.640 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:43 smithi049 ceph-mon[40489]: osdmap e440: 8 total, 8 up, 8 in 2024-09-21T14:05:43.640 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:43 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T14:05:43.641 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:43 smithi049 ceph-mon[30080]: pgmap v855: 168 pgs: 9 peering, 2 unknown, 157 active+clean; 3.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 3.7 KiB/s rd, 18 KiB/s wr, 1.10k op/s; 6.2 MiB/s, 98 objects/s recovering 2024-09-21T14:05:43.641 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:43 smithi049 ceph-mon[30080]: 1.5 scrub starts 2024-09-21T14:05:43.641 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:43 smithi049 ceph-mon[30080]: 1.10 scrub starts 2024-09-21T14:05:43.641 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:43 smithi049 ceph-mon[30080]: 1.5 scrub ok 2024-09-21T14:05:43.641 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:43 smithi049 ceph-mon[30080]: 1.10 scrub ok 2024-09-21T14:05:43.641 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:43 smithi049 ceph-mon[30080]: Health check failed: Reduced data availability: 2 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-09-21T14:05:43.641 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:43 smithi049 ceph-mon[30080]: osdmap e440: 8 total, 8 up, 8 in 2024-09-21T14:05:43.641 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:43 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T14:05:43.734 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:43.766 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:43.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:43.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:43.868 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:05:43.917 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:43.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:43.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:05:44.018 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:05:44.058 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:44.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:44.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:05:44.163 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:05:44.215 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:44.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:44.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:05:44.224 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:05:44.244 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:05:44.317 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:05:44.343 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:44.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:44.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:05:44.445 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:44.473 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:44.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:44.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:44.575 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:05:44.600 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:44.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:44.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:05:44.627 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:44 smithi049 ceph-mon[40489]: 1.11 scrub starts 2024-09-21T14:05:44.627 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:44 smithi049 ceph-mon[40489]: 1.11 scrub ok 2024-09-21T14:05:44.627 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:44 smithi049 ceph-mon[40489]: 10.8 scrub starts 2024-09-21T14:05:44.627 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:44 smithi049 ceph-mon[40489]: 1.b scrub starts 2024-09-21T14:05:44.627 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:44 smithi049 ceph-mon[40489]: 1.b scrub ok 2024-09-21T14:05:44.627 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:44 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:05:44.627 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:44 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:05:44.628 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:44 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T14:05:44.628 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:44 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T14:05:44.628 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:44 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:05:44.628 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:44 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:05:44.628 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:44 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:05:44.628 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:44 smithi049 ceph-mon[30080]: 1.11 scrub starts 2024-09-21T14:05:44.628 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:44 smithi049 ceph-mon[30080]: 1.11 scrub ok 2024-09-21T14:05:44.629 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:44 smithi049 ceph-mon[30080]: 10.8 scrub starts 2024-09-21T14:05:44.629 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:44 smithi049 ceph-mon[30080]: 1.b scrub starts 2024-09-21T14:05:44.629 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:44 smithi049 ceph-mon[30080]: 1.b scrub ok 2024-09-21T14:05:44.629 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:44 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:05:44.629 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:44 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:05:44.629 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:44 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T14:05:44.629 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:44 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T14:05:44.629 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:44 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:05:44.629 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:44 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:05:44.629 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:44 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:05:44.703 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:05:44.743 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:44.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:44.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:05:44.845 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:44.878 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:44.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:44.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:44.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:44 smithi187 ceph-mon[33032]: 1.11 scrub starts 2024-09-21T14:05:44.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:44 smithi187 ceph-mon[33032]: 1.11 scrub ok 2024-09-21T14:05:44.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:44 smithi187 ceph-mon[33032]: 10.8 scrub starts 2024-09-21T14:05:44.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:44 smithi187 ceph-mon[33032]: 1.b scrub starts 2024-09-21T14:05:44.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:44 smithi187 ceph-mon[33032]: 1.b scrub ok 2024-09-21T14:05:44.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:44 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:05:44.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:44 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:05:44.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:44 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T14:05:44.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:44 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T14:05:44.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:44 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:05:44.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:44 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:05:44.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:44 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:05:44.980 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:05:45.006 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:45.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:45.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:05:45.108 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:05:45.135 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:45.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:45.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:05:45.237 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:05:45.290 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:45.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:45.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:05:45.294 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T14:05:45.295 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T14:05:45.295 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 440, 2024-09-21T14:05:45.295 INFO:teuthology.orchestra.run.smithi049.stdout: "fsid": "40a3d2e8-781f-11ef-baf6-efdc52797490", 2024-09-21T14:05:45.295 INFO:teuthology.orchestra.run.smithi049.stdout: "created": "2024-09-21T13:42:57.838275+0000", 2024-09-21T14:05:45.295 INFO:teuthology.orchestra.run.smithi049.stdout: "modified": "2024-09-21T14:05:42.338359+0000", 2024-09-21T14:05:45.295 INFO:teuthology.orchestra.run.smithi049.stdout: "last_up_change": "2024-09-21T14:01:40.158454+0000", 2024-09-21T14:05:45.295 INFO:teuthology.orchestra.run.smithi049.stdout: "last_in_change": "2024-09-21T13:58:16.730861+0000", 2024-09-21T14:05:45.295 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-21T14:05:45.295 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_num": 5804032, 2024-09-21T14:05:45.295 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_set": [ 2024-09-21T14:05:45.295 INFO:teuthology.orchestra.run.smithi049.stdout: "nodeep-scrub", 2024-09-21T14:05:45.296 INFO:teuthology.orchestra.run.smithi049.stdout: "pglog_hardlimit", 2024-09-21T14:05:45.296 INFO:teuthology.orchestra.run.smithi049.stdout: "purged_snapdirs", 2024-09-21T14:05:45.296 INFO:teuthology.orchestra.run.smithi049.stdout: "recovery_deletes", 2024-09-21T14:05:45.296 INFO:teuthology.orchestra.run.smithi049.stdout: "sortbitwise" 2024-09-21T14:05:45.296 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T14:05:45.296 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_version": 22, 2024-09-21T14:05:45.296 INFO:teuthology.orchestra.run.smithi049.stdout: "full_ratio": 0.94999998807907104, 2024-09-21T14:05:45.296 INFO:teuthology.orchestra.run.smithi049.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-21T14:05:45.296 INFO:teuthology.orchestra.run.smithi049.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-21T14:05:45.296 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_snapshot": "", 2024-09-21T14:05:45.296 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_max": 10, 2024-09-21T14:05:45.296 INFO:teuthology.orchestra.run.smithi049.stdout: "max_osd": 8, 2024-09-21T14:05:45.296 INFO:teuthology.orchestra.run.smithi049.stdout: "require_min_compat_client": "luminous", 2024-09-21T14:05:45.297 INFO:teuthology.orchestra.run.smithi049.stdout: "min_compat_client": "luminous", 2024-09-21T14:05:45.297 INFO:teuthology.orchestra.run.smithi049.stdout: "require_osd_release": "squid", 2024-09-21T14:05:45.297 INFO:teuthology.orchestra.run.smithi049.stdout: "allow_crimson": false, 2024-09-21T14:05:45.297 INFO:teuthology.orchestra.run.smithi049.stdout: "pools": [ 2024-09-21T14:05:45.297 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.297 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 1, 2024-09-21T14:05:45.297 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": ".mgr", 2024-09-21T14:05:45.297 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:46:21.096244+0000", 2024-09-21T14:05:45.297 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T14:05:45.297 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T14:05:45.297 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T14:05:45.297 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T14:05:45.297 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T14:05:45.297 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T14:05:45.297 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T14:05:45.298 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T14:05:45.298 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T14:05:45.298 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T14:05:45.298 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T14:05:45.298 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T14:05:45.298 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T14:05:45.298 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 21, 2024-09-21T14:05:45.298 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 21, 2024-09-21T14:05:45.298 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 21, 2024-09-21T14:05:45.298 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 21, 2024-09-21T14:05:45.298 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 21, 2024-09-21T14:05:45.298 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T14:05:45.298 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T14:05:45.298 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T14:05:45.299 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T14:05:45.299 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T14:05:45.299 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T14:05:45.299 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T14:05:45.299 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.299 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "438", 2024-09-21T14:05:45.299 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T14:05:45.299 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T14:05:45.299 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "436", 2024-09-21T14:05:45.299 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T14:05:45.299 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T14:05:45.299 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T14:05:45.299 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T14:05:45.299 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T14:05:45.300 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T14:05:45.300 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T14:05:45.300 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T14:05:45.300 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T14:05:45.300 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T14:05:45.300 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T14:05:45.300 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T14:05:45.300 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T14:05:45.300 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T14:05:45.300 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T14:05:45.300 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T14:05:45.300 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T14:05:45.300 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T14:05:45.301 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T14:05:45.301 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T14:05:45.301 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T14:05:45.301 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T14:05:45.301 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T14:05:45.301 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.301 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T14:05:45.301 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T14:05:45.301 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T14:05:45.301 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T14:05:45.301 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T14:05:45.301 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T14:05:45.301 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T14:05:45.301 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T14:05:45.302 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T14:05:45.302 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T14:05:45.302 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T14:05:45.302 INFO:teuthology.orchestra.run.smithi049.stdout: "options": { 2024-09-21T14:05:45.302 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_max": 32, 2024-09-21T14:05:45.302 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_min": 1 2024-09-21T14:05:45.302 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.302 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T14:05:45.302 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr": {} 2024-09-21T14:05:45.302 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.303 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T14:05:45.303 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T14:05:45.303 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2.2899999618530273, 2024-09-21T14:05:45.303 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2.2899999618530273, 2024-09-21T14:05:45.303 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1, 2024-09-21T14:05:45.303 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2.2899999618530273, 2024-09-21T14:05:45.303 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2.2899999618530273, 2024-09-21T14:05:45.303 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.97835731506347656, 2024-09-21T14:05:45.303 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.98000001907348633, 2024-09-21T14:05:45.303 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.98000001907348633 2024-09-21T14:05:45.303 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:45.303 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.303 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.304 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 2, 2024-09-21T14:05:45.304 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "datapool", 2024-09-21T14:05:45.304 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:48:55.540712+0000", 2024-09-21T14:05:45.304 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 8193, 2024-09-21T14:05:45.304 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T14:05:45.304 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T14:05:45.304 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T14:05:45.304 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T14:05:45.304 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T14:05:45.304 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T14:05:45.304 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T14:05:45.304 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T14:05:45.304 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T14:05:45.305 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T14:05:45.305 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T14:05:45.305 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T14:05:45.305 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 3, 2024-09-21T14:05:45.305 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 3, 2024-09-21T14:05:45.305 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 3, 2024-09-21T14:05:45.305 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 3, 2024-09-21T14:05:45.305 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 3, 2024-09-21T14:05:45.305 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T14:05:45.305 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T14:05:45.305 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T14:05:45.306 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T14:05:45.306 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T14:05:45.306 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T14:05:45.306 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T14:05:45.306 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.306 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "63", 2024-09-21T14:05:45.306 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T14:05:45.306 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T14:05:45.306 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T14:05:45.306 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T14:05:45.306 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T14:05:45.306 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 2, 2024-09-21T14:05:45.307 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 63, 2024-09-21T14:05:45.307 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T14:05:45.307 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T14:05:45.307 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T14:05:45.307 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T14:05:45.307 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T14:05:45.307 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T14:05:45.307 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T14:05:45.307 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T14:05:45.307 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T14:05:45.307 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T14:05:45.307 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T14:05:45.307 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T14:05:45.308 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T14:05:45.308 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T14:05:45.308 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T14:05:45.308 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T14:05:45.308 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T14:05:45.308 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T14:05:45.308 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T14:05:45.308 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.308 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T14:05:45.308 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T14:05:45.308 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T14:05:45.309 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T14:05:45.309 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T14:05:45.309 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T14:05:45.309 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T14:05:45.309 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T14:05:45.309 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T14:05:45.309 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T14:05:45.309 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T14:05:45.309 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T14:05:45.309 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T14:05:45.309 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd": {} 2024-09-21T14:05:45.309 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.309 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T14:05:45.310 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T14:05:45.310 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2.6500000953674316, 2024-09-21T14:05:45.310 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2.6500000953674316, 2024-09-21T14:05:45.310 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 0.87999999523162842, 2024-09-21T14:05:45.310 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2.3299999237060547, 2024-09-21T14:05:45.310 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2.3299999237060547, 2024-09-21T14:05:45.310 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.97835731506347656, 2024-09-21T14:05:45.310 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.98000001907348633, 2024-09-21T14:05:45.310 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.98000001907348633 2024-09-21T14:05:45.311 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:45.311 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.311 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.311 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 3, 2024-09-21T14:05:45.311 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": ".rgw.root", 2024-09-21T14:05:45.311 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:48:56.417456+0000", 2024-09-21T14:05:45.311 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T14:05:45.311 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T14:05:45.311 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T14:05:45.311 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T14:05:45.312 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T14:05:45.312 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T14:05:45.312 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T14:05:45.312 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T14:05:45.312 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T14:05:45.312 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T14:05:45.312 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T14:05:45.312 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T14:05:45.312 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T14:05:45.313 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 32, 2024-09-21T14:05:45.313 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 32, 2024-09-21T14:05:45.313 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 32, 2024-09-21T14:05:45.313 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 32, 2024-09-21T14:05:45.313 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 32, 2024-09-21T14:05:45.313 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T14:05:45.314 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T14:05:45.314 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T14:05:45.314 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T14:05:45.314 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T14:05:45.314 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T14:05:45.314 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T14:05:45.314 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.314 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "62", 2024-09-21T14:05:45.314 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T14:05:45.315 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T14:05:45.315 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T14:05:45.315 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T14:05:45.315 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T14:05:45.315 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T14:05:45.315 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T14:05:45.315 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T14:05:45.315 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T14:05:45.315 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T14:05:45.315 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T14:05:45.315 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T14:05:45.315 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T14:05:45.315 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T14:05:45.316 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T14:05:45.316 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T14:05:45.316 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T14:05:45.316 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T14:05:45.316 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T14:05:45.316 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T14:05:45.316 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T14:05:45.316 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T14:05:45.316 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T14:05:45.316 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T14:05:45.316 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T14:05:45.316 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T14:05:45.316 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.316 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T14:05:45.317 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T14:05:45.317 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T14:05:45.317 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T14:05:45.317 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T14:05:45.317 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T14:05:45.317 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T14:05:45.317 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T14:05:45.317 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T14:05:45.317 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T14:05:45.317 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T14:05:45.317 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T14:05:45.317 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T14:05:45.317 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw": {} 2024-09-21T14:05:45.318 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.318 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T14:05:45.318 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T14:05:45.318 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 1.75, 2024-09-21T14:05:45.318 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 1.75, 2024-09-21T14:05:45.318 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1, 2024-09-21T14:05:45.318 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 1.75, 2024-09-21T14:05:45.318 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 1.75, 2024-09-21T14:05:45.318 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.97835731506347656, 2024-09-21T14:05:45.318 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.98000001907348633, 2024-09-21T14:05:45.318 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.98000001907348633 2024-09-21T14:05:45.318 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:45.318 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.318 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.319 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 4, 2024-09-21T14:05:45.319 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "default.rgw.log", 2024-09-21T14:05:45.319 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:48:58.128455+0000", 2024-09-21T14:05:45.319 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T14:05:45.319 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T14:05:45.319 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T14:05:45.319 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T14:05:45.319 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T14:05:45.319 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T14:05:45.319 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T14:05:45.319 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T14:05:45.319 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T14:05:45.319 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T14:05:45.320 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T14:05:45.320 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T14:05:45.320 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T14:05:45.320 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 32, 2024-09-21T14:05:45.320 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 32, 2024-09-21T14:05:45.320 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 32, 2024-09-21T14:05:45.320 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 32, 2024-09-21T14:05:45.320 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 32, 2024-09-21T14:05:45.320 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T14:05:45.320 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T14:05:45.320 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T14:05:45.320 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T14:05:45.320 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T14:05:45.320 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T14:05:45.321 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T14:05:45.321 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.321 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "64", 2024-09-21T14:05:45.321 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T14:05:45.321 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T14:05:45.321 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T14:05:45.321 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T14:05:45.321 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T14:05:45.321 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T14:05:45.321 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T14:05:45.321 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T14:05:45.321 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T14:05:45.321 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T14:05:45.321 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T14:05:45.321 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T14:05:45.322 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T14:05:45.322 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T14:05:45.322 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T14:05:45.322 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T14:05:45.322 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T14:05:45.322 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T14:05:45.322 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T14:05:45.322 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T14:05:45.322 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T14:05:45.322 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T14:05:45.322 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T14:05:45.322 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T14:05:45.322 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T14:05:45.322 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T14:05:45.323 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.323 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T14:05:45.323 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T14:05:45.323 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T14:05:45.323 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T14:05:45.323 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T14:05:45.323 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T14:05:45.323 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T14:05:45.323 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T14:05:45.323 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T14:05:45.323 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T14:05:45.323 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T14:05:45.323 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T14:05:45.323 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T14:05:45.323 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw": {} 2024-09-21T14:05:45.324 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.324 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T14:05:45.324 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T14:05:45.324 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2.5, 2024-09-21T14:05:45.324 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2.5, 2024-09-21T14:05:45.324 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1, 2024-09-21T14:05:45.324 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2.5, 2024-09-21T14:05:45.324 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2.5, 2024-09-21T14:05:45.324 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.97835731506347656, 2024-09-21T14:05:45.324 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.98000001907348633, 2024-09-21T14:05:45.324 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.98000001907348633 2024-09-21T14:05:45.324 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:45.324 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.324 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.325 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 5, 2024-09-21T14:05:45.325 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "default.rgw.control", 2024-09-21T14:05:45.325 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:49:00.069361+0000", 2024-09-21T14:05:45.325 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T14:05:45.325 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T14:05:45.325 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T14:05:45.325 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T14:05:45.325 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T14:05:45.325 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T14:05:45.325 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T14:05:45.325 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T14:05:45.325 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T14:05:45.325 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T14:05:45.325 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T14:05:45.325 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T14:05:45.326 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T14:05:45.326 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 32, 2024-09-21T14:05:45.326 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 32, 2024-09-21T14:05:45.326 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 32, 2024-09-21T14:05:45.326 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 32, 2024-09-21T14:05:45.326 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 32, 2024-09-21T14:05:45.326 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T14:05:45.326 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T14:05:45.326 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T14:05:45.326 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T14:05:45.326 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T14:05:45.326 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T14:05:45.326 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T14:05:45.326 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.327 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "66", 2024-09-21T14:05:45.327 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T14:05:45.327 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T14:05:45.327 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T14:05:45.327 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T14:05:45.327 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T14:05:45.327 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T14:05:45.327 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T14:05:45.327 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T14:05:45.327 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T14:05:45.327 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T14:05:45.327 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T14:05:45.327 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T14:05:45.327 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T14:05:45.327 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T14:05:45.328 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T14:05:45.328 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T14:05:45.328 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T14:05:45.328 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T14:05:45.328 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T14:05:45.328 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T14:05:45.328 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T14:05:45.328 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T14:05:45.328 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T14:05:45.328 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T14:05:45.328 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T14:05:45.328 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T14:05:45.328 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.328 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T14:05:45.328 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T14:05:45.329 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T14:05:45.329 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T14:05:45.329 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T14:05:45.329 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T14:05:45.329 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T14:05:45.329 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T14:05:45.329 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T14:05:45.329 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T14:05:45.329 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T14:05:45.329 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T14:05:45.329 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T14:05:45.329 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw": {} 2024-09-21T14:05:45.329 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.329 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T14:05:45.330 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T14:05:45.330 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 1.5, 2024-09-21T14:05:45.330 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 1.5, 2024-09-21T14:05:45.330 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1, 2024-09-21T14:05:45.330 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 1.5, 2024-09-21T14:05:45.330 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 1.5, 2024-09-21T14:05:45.330 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.97835731506347656, 2024-09-21T14:05:45.330 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.98000001907348633, 2024-09-21T14:05:45.330 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.98000001907348633 2024-09-21T14:05:45.330 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:45.330 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.330 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.330 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 6, 2024-09-21T14:05:45.330 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "default.rgw.meta", 2024-09-21T14:05:45.331 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:49:02.127399+0000", 2024-09-21T14:05:45.331 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T14:05:45.331 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T14:05:45.331 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T14:05:45.331 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T14:05:45.331 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T14:05:45.331 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T14:05:45.331 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T14:05:45.331 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T14:05:45.331 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T14:05:45.331 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T14:05:45.331 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T14:05:45.331 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T14:05:45.331 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T14:05:45.331 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 32, 2024-09-21T14:05:45.332 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 32, 2024-09-21T14:05:45.332 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 32, 2024-09-21T14:05:45.332 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 32, 2024-09-21T14:05:45.332 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 32, 2024-09-21T14:05:45.332 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T14:05:45.332 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T14:05:45.332 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T14:05:45.332 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T14:05:45.332 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T14:05:45.332 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T14:05:45.332 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T14:05:45.332 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.332 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "68", 2024-09-21T14:05:45.332 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T14:05:45.333 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T14:05:45.333 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T14:05:45.333 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T14:05:45.333 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T14:05:45.333 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T14:05:45.333 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T14:05:45.333 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T14:05:45.333 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T14:05:45.333 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T14:05:45.333 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T14:05:45.333 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T14:05:45.333 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T14:05:45.333 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T14:05:45.333 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T14:05:45.333 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T14:05:45.334 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T14:05:45.334 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T14:05:45.334 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T14:05:45.334 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T14:05:45.334 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T14:05:45.334 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T14:05:45.334 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T14:05:45.334 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T14:05:45.334 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T14:05:45.334 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T14:05:45.334 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.334 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T14:05:45.334 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T14:05:45.335 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T14:05:45.335 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T14:05:45.335 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T14:05:45.335 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T14:05:45.335 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T14:05:45.335 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T14:05:45.335 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T14:05:45.335 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T14:05:45.335 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T14:05:45.335 INFO:teuthology.orchestra.run.smithi049.stdout: "options": { 2024-09-21T14:05:45.336 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_bias": 4 2024-09-21T14:05:45.336 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.336 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T14:05:45.336 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw": {} 2024-09-21T14:05:45.336 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.336 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T14:05:45.336 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T14:05:45.336 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2, 2024-09-21T14:05:45.336 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2, 2024-09-21T14:05:45.336 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1, 2024-09-21T14:05:45.336 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2, 2024-09-21T14:05:45.336 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2, 2024-09-21T14:05:45.336 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.97835731506347656, 2024-09-21T14:05:45.336 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.98000001907348633, 2024-09-21T14:05:45.337 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.98000001907348633 2024-09-21T14:05:45.337 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:45.337 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.337 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.337 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 10, 2024-09-21T14:05:45.337 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "unique_pool_3", 2024-09-21T14:05:45.337 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T14:02:55.287153+0000", 2024-09-21T14:05:45.337 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T14:05:45.337 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T14:05:45.338 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T14:05:45.338 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T14:05:45.338 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T14:05:45.338 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T14:05:45.338 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T14:05:45.338 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T14:05:45.338 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T14:05:45.338 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T14:05:45.338 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T14:05:45.338 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T14:05:45.338 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T14:05:45.338 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 16, 2024-09-21T14:05:45.338 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 16, 2024-09-21T14:05:45.339 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 16, 2024-09-21T14:05:45.339 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 16, 2024-09-21T14:05:45.339 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 16, 2024-09-21T14:05:45.339 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T14:05:45.339 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T14:05:45.339 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T14:05:45.339 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T14:05:45.339 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T14:05:45.339 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T14:05:45.339 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T14:05:45.339 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.340 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "415", 2024-09-21T14:05:45.340 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T14:05:45.340 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T14:05:45.340 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T14:05:45.340 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T14:05:45.340 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T14:05:45.341 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T14:05:45.341 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T14:05:45.341 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T14:05:45.342 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T14:05:45.342 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T14:05:45.342 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T14:05:45.342 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T14:05:45.342 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T14:05:45.342 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T14:05:45.342 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T14:05:45.342 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T14:05:45.342 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T14:05:45.342 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T14:05:45.342 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T14:05:45.342 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T14:05:45.342 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T14:05:45.343 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T14:05:45.343 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T14:05:45.343 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T14:05:45.343 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T14:05:45.343 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T14:05:45.343 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.343 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T14:05:45.343 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T14:05:45.343 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T14:05:45.343 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T14:05:45.343 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T14:05:45.343 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T14:05:45.343 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T14:05:45.343 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T14:05:45.343 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T14:05:45.344 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T14:05:45.344 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T14:05:45.344 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T14:05:45.344 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T14:05:45.344 INFO:teuthology.orchestra.run.smithi049.stdout: "rados": {} 2024-09-21T14:05:45.344 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.344 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T14:05:45.344 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T14:05:45.344 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 3, 2024-09-21T14:05:45.344 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 3, 2024-09-21T14:05:45.344 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1, 2024-09-21T14:05:45.344 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 3, 2024-09-21T14:05:45.344 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 3, 2024-09-21T14:05:45.344 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.97835731506347656, 2024-09-21T14:05:45.345 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.98000001907348633, 2024-09-21T14:05:45.345 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.98000001907348633 2024-09-21T14:05:45.345 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:45.345 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:45.345 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T14:05:45.345 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-09-21T14:05:45.345 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.345 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 0, 2024-09-21T14:05:45.345 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "2595dca9-6219-4e0c-8728-751eb102e73a", 2024-09-21T14:05:45.345 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T14:05:45.345 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T14:05:45.345 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 0.6176910400390625, 2024-09-21T14:05:45.345 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T14:05:45.345 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T14:05:45.346 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T14:05:45.346 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 8, 2024-09-21T14:05:45.346 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 429, 2024-09-21T14:05:45.346 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T14:05:45.346 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T14:05:45.346 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T14:05:45.346 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:45.346 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.346 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:45.346 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6802", 2024-09-21T14:05:45.346 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T14:05:45.347 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.347 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.347 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:45.347 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6803", 2024-09-21T14:05:45.347 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T14:05:45.347 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:45.347 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:45.347 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.347 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T14:05:45.347 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:45.347 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.347 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:45.347 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6804", 2024-09-21T14:05:45.348 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T14:05:45.348 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.348 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.348 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:45.348 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6805", 2024-09-21T14:05:45.348 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T14:05:45.348 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:45.348 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:45.348 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.348 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T14:05:45.348 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:45.349 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.349 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:45.349 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6808", 2024-09-21T14:05:45.349 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T14:05:45.349 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.349 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.349 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:45.350 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6809", 2024-09-21T14:05:45.350 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T14:05:45.350 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:45.351 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:45.351 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.351 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T14:05:45.351 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:45.351 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.351 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:45.351 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6806", 2024-09-21T14:05:45.351 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T14:05:45.351 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.351 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.351 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:45.352 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6807", 2024-09-21T14:05:45.352 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T14:05:45.352 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:45.352 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:45.352 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.352 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6803/2095561412", 2024-09-21T14:05:45.352 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6805/2095561412", 2024-09-21T14:05:45.352 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6809/2095561412", 2024-09-21T14:05:45.352 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6807/2095561412", 2024-09-21T14:05:45.352 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T14:05:45.352 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T14:05:45.352 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T14:05:45.352 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:45.352 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.353 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.353 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 1, 2024-09-21T14:05:45.353 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "0ba327f9-5524-4a40-abcc-729ded0b1917", 2024-09-21T14:05:45.353 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T14:05:45.353 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T14:05:45.353 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T14:05:45.353 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T14:05:45.353 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T14:05:45.353 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T14:05:45.353 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 13, 2024-09-21T14:05:45.353 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 438, 2024-09-21T14:05:45.353 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T14:05:45.353 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T14:05:45.354 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T14:05:45.354 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:45.354 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.354 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:45.354 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6810", 2024-09-21T14:05:45.354 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T14:05:45.354 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.354 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.354 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:45.354 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6811", 2024-09-21T14:05:45.354 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T14:05:45.354 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:45.354 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:45.355 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.355 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T14:05:45.355 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:45.355 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.355 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:45.355 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6812", 2024-09-21T14:05:45.355 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T14:05:45.355 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.355 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.355 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:45.355 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6813", 2024-09-21T14:05:45.355 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T14:05:45.355 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:45.355 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:45.355 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.356 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T14:05:45.356 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:45.356 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.356 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:45.356 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6816", 2024-09-21T14:05:45.356 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T14:05:45.356 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.356 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.356 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:45.356 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6817", 2024-09-21T14:05:45.356 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T14:05:45.356 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:45.356 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:45.356 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.357 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T14:05:45.357 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:45.357 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.357 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:45.357 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6814", 2024-09-21T14:05:45.357 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T14:05:45.357 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.357 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.357 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:45.357 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6815", 2024-09-21T14:05:45.357 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T14:05:45.357 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:45.357 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:45.357 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.358 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6811/1382840527", 2024-09-21T14:05:45.358 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6813/1382840527", 2024-09-21T14:05:45.358 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6817/1382840527", 2024-09-21T14:05:45.358 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6815/1382840527", 2024-09-21T14:05:45.358 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T14:05:45.358 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T14:05:45.358 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T14:05:45.358 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:45.358 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.358 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.358 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 2, 2024-09-21T14:05:45.358 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "27a45750-204d-48fd-bb8f-ebb01273343d", 2024-09-21T14:05:45.358 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T14:05:45.358 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T14:05:45.359 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T14:05:45.359 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 0.8268585205078125, 2024-09-21T14:05:45.359 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 18, 2024-09-21T14:05:45.359 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 100, 2024-09-21T14:05:45.359 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 130, 2024-09-21T14:05:45.359 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 438, 2024-09-21T14:05:45.359 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 101, 2024-09-21T14:05:45.359 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T14:05:45.359 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T14:05:45.359 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:45.359 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.359 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:45.359 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6818", 2024-09-21T14:05:45.359 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T14:05:45.360 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.360 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.360 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:45.360 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6819", 2024-09-21T14:05:45.360 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T14:05:45.360 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:45.360 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:45.360 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.360 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T14:05:45.360 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:45.360 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.360 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:45.360 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6820", 2024-09-21T14:05:45.360 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T14:05:45.361 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.361 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.361 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:45.361 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6821", 2024-09-21T14:05:45.361 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T14:05:45.361 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:45.361 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:45.361 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.361 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T14:05:45.361 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:45.361 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.361 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:45.361 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6824", 2024-09-21T14:05:45.361 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T14:05:45.361 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.362 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.362 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:45.362 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6825", 2024-09-21T14:05:45.362 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T14:05:45.362 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:45.362 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:45.362 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.362 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T14:05:45.362 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:45.362 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.362 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:45.362 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6822", 2024-09-21T14:05:45.362 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T14:05:45.362 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.363 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.363 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:45.363 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6823", 2024-09-21T14:05:45.363 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T14:05:45.363 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:45.363 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:45.363 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.363 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6819/1945314741", 2024-09-21T14:05:45.363 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6821/1945314741", 2024-09-21T14:05:45.363 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6825/1945314741", 2024-09-21T14:05:45.363 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6823/1945314741", 2024-09-21T14:05:45.363 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T14:05:45.363 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T14:05:45.363 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T14:05:45.363 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:45.364 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.364 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.364 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 3, 2024-09-21T14:05:45.364 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "528bc390-3dc9-4f3b-ab75-d7412c9625db", 2024-09-21T14:05:45.364 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T14:05:45.364 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T14:05:45.364 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T14:05:45.364 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T14:05:45.364 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 27, 2024-09-21T14:05:45.364 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 214, 2024-09-21T14:05:45.364 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 240, 2024-09-21T14:05:45.364 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 438, 2024-09-21T14:05:45.364 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 215, 2024-09-21T14:05:45.364 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T14:05:45.365 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T14:05:45.365 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:45.365 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.365 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:45.365 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6826", 2024-09-21T14:05:45.365 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T14:05:45.365 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.365 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.365 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:45.365 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6827", 2024-09-21T14:05:45.365 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T14:05:45.365 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:45.365 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:45.365 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.365 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T14:05:45.366 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:45.366 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.366 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:45.366 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6828", 2024-09-21T14:05:45.366 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T14:05:45.366 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.366 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.366 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:45.366 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6829", 2024-09-21T14:05:45.366 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T14:05:45.366 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:45.366 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:45.366 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.366 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T14:05:45.367 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:45.367 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.367 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:45.367 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6832", 2024-09-21T14:05:45.367 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T14:05:45.367 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.367 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.367 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:45.367 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6833", 2024-09-21T14:05:45.367 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T14:05:45.367 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:45.367 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:45.367 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.367 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T14:05:45.367 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:45.367 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.368 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:45.368 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6830", 2024-09-21T14:05:45.368 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T14:05:45.368 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.368 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.368 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:45.368 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6831", 2024-09-21T14:05:45.368 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T14:05:45.368 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:45.368 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:45.368 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.368 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6827/886735721", 2024-09-21T14:05:45.368 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6829/886735721", 2024-09-21T14:05:45.369 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6833/886735721", 2024-09-21T14:05:45.369 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6831/886735721", 2024-09-21T14:05:45.369 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T14:05:45.369 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T14:05:45.369 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T14:05:45.369 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:45.369 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.369 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.369 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 4, 2024-09-21T14:05:45.369 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "6fc64698-2d37-4db0-bf3d-c7e612c45163", 2024-09-21T14:05:45.369 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T14:05:45.369 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T14:05:45.369 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 0.7888946533203125, 2024-09-21T14:05:45.369 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T14:05:45.369 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T14:05:45.370 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T14:05:45.370 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 33, 2024-09-21T14:05:45.370 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 436, 2024-09-21T14:05:45.370 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T14:05:45.370 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T14:05:45.370 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T14:05:45.370 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:45.370 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.370 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:45.370 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6800", 2024-09-21T14:05:45.370 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T14:05:45.370 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.370 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.370 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:45.371 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6801", 2024-09-21T14:05:45.371 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T14:05:45.371 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:45.371 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:45.371 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.371 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T14:05:45.371 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:45.371 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.371 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:45.371 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6802", 2024-09-21T14:05:45.371 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T14:05:45.372 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.372 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.372 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:45.372 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6803", 2024-09-21T14:05:45.372 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T14:05:45.372 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:45.372 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:45.372 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.372 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T14:05:45.372 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:45.372 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.372 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:45.372 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6806", 2024-09-21T14:05:45.373 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T14:05:45.373 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.373 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.373 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:45.373 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6807", 2024-09-21T14:05:45.373 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T14:05:45.373 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:45.373 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:45.373 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.373 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T14:05:45.374 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:45.374 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.374 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:45.374 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6804", 2024-09-21T14:05:45.374 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T14:05:45.374 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.374 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.374 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:45.374 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6805", 2024-09-21T14:05:45.374 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T14:05:45.374 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:45.374 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:45.374 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.374 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.187:6801/1702447776", 2024-09-21T14:05:45.375 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.187:6803/1702447776", 2024-09-21T14:05:45.375 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.187:6807/1702447776", 2024-09-21T14:05:45.375 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.187:6805/1702447776", 2024-09-21T14:05:45.375 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T14:05:45.375 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T14:05:45.375 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T14:05:45.375 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:45.375 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.375 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.375 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 5, 2024-09-21T14:05:45.375 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "8d7f62a1-7e92-4e06-9cad-86b96be514d8", 2024-09-21T14:05:45.376 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T14:05:45.376 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T14:05:45.376 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T14:05:45.376 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T14:05:45.376 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 39, 2024-09-21T14:05:45.376 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 324, 2024-09-21T14:05:45.376 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 345, 2024-09-21T14:05:45.376 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 438, 2024-09-21T14:05:45.376 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 325, 2024-09-21T14:05:45.376 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T14:05:45.376 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T14:05:45.376 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:45.376 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.376 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:45.377 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6808", 2024-09-21T14:05:45.377 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1595353965 2024-09-21T14:05:45.377 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.377 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.377 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:45.377 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6809", 2024-09-21T14:05:45.377 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1595353965 2024-09-21T14:05:45.377 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:45.377 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:45.377 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.377 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T14:05:45.378 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:45.378 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.378 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:45.378 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6810", 2024-09-21T14:05:45.378 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1595353965 2024-09-21T14:05:45.378 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.378 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.378 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:45.378 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6811", 2024-09-21T14:05:45.378 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1595353965 2024-09-21T14:05:45.378 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:45.378 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:45.378 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.379 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T14:05:45.379 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:45.379 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.379 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:45.379 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6814", 2024-09-21T14:05:45.379 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1595353965 2024-09-21T14:05:45.379 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.379 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.379 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:45.379 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6815", 2024-09-21T14:05:45.379 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1595353965 2024-09-21T14:05:45.380 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:45.380 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:45.380 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.380 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T14:05:45.380 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:45.380 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.380 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:45.380 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6812", 2024-09-21T14:05:45.380 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1595353965 2024-09-21T14:05:45.380 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.380 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.380 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:45.380 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6813", 2024-09-21T14:05:45.381 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1595353965 2024-09-21T14:05:45.381 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:45.381 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:45.381 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.381 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.187:6809/1595353965", 2024-09-21T14:05:45.381 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.187:6811/1595353965", 2024-09-21T14:05:45.381 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.187:6815/1595353965", 2024-09-21T14:05:45.381 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.187:6813/1595353965", 2024-09-21T14:05:45.381 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T14:05:45.381 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T14:05:45.381 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T14:05:45.381 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:45.382 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.382 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.382 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 6, 2024-09-21T14:05:45.382 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "3a7afd13-26ce-444c-9cc0-81ce2eab6ce5", 2024-09-21T14:05:45.382 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T14:05:45.382 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T14:05:45.382 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 0.9076080322265625, 2024-09-21T14:05:45.382 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T14:05:45.382 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T14:05:45.382 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T14:05:45.383 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 47, 2024-09-21T14:05:45.383 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 438, 2024-09-21T14:05:45.383 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T14:05:45.383 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T14:05:45.383 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T14:05:45.383 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:45.383 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.383 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:45.383 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6816", 2024-09-21T14:05:45.383 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T14:05:45.384 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.384 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.384 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:45.384 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6817", 2024-09-21T14:05:45.384 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T14:05:45.384 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:45.384 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:45.384 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.384 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T14:05:45.384 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:45.384 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.385 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:45.385 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6818", 2024-09-21T14:05:45.385 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T14:05:45.385 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.385 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.385 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:45.385 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6819", 2024-09-21T14:05:45.385 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T14:05:45.385 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:45.385 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:45.385 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.385 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T14:05:45.385 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:45.385 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.386 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:45.386 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6822", 2024-09-21T14:05:45.386 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T14:05:45.386 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.386 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.386 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:45.386 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6823", 2024-09-21T14:05:45.386 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T14:05:45.386 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:45.386 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:45.386 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.386 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T14:05:45.386 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:45.387 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.387 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:45.387 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6820", 2024-09-21T14:05:45.387 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T14:05:45.387 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.387 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.387 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:45.387 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6821", 2024-09-21T14:05:45.387 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T14:05:45.387 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:45.387 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:45.387 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.387 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.187:6817/2728426525", 2024-09-21T14:05:45.388 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.187:6819/2728426525", 2024-09-21T14:05:45.388 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.187:6823/2728426525", 2024-09-21T14:05:45.388 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.187:6821/2728426525", 2024-09-21T14:05:45.388 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T14:05:45.388 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T14:05:45.388 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T14:05:45.388 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:45.388 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.388 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.388 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 7, 2024-09-21T14:05:45.388 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "1e08d891-8747-402e-b4c2-aa3802ce8537", 2024-09-21T14:05:45.388 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T14:05:45.389 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T14:05:45.389 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T14:05:45.389 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T14:05:45.389 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 54, 2024-09-21T14:05:45.389 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 272, 2024-09-21T14:05:45.389 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 298, 2024-09-21T14:05:45.389 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 438, 2024-09-21T14:05:45.389 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 273, 2024-09-21T14:05:45.390 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T14:05:45.390 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T14:05:45.390 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:45.390 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.390 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:45.390 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6824", 2024-09-21T14:05:45.390 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 13894964 2024-09-21T14:05:45.390 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.390 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.390 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:45.390 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6825", 2024-09-21T14:05:45.390 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 13894964 2024-09-21T14:05:45.390 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:45.390 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:45.391 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.391 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T14:05:45.391 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:45.391 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.391 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:45.391 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6826", 2024-09-21T14:05:45.391 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 13894964 2024-09-21T14:05:45.391 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.391 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.391 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:45.391 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6827", 2024-09-21T14:05:45.391 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 13894964 2024-09-21T14:05:45.391 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:45.391 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:45.391 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.392 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T14:05:45.392 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:45.392 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.392 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:45.392 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6830", 2024-09-21T14:05:45.392 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 13894964 2024-09-21T14:05:45.392 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.392 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.392 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:45.392 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6831", 2024-09-21T14:05:45.392 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 13894964 2024-09-21T14:05:45.392 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:45.392 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:45.392 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.393 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T14:05:45.393 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:05:45.393 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.393 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:05:45.393 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6828", 2024-09-21T14:05:45.393 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 13894964 2024-09-21T14:05:45.393 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.393 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.393 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:05:45.393 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6829", 2024-09-21T14:05:45.393 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 13894964 2024-09-21T14:05:45.393 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:45.393 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:45.393 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.394 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.187:6825/13894964", 2024-09-21T14:05:45.394 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.187:6827/13894964", 2024-09-21T14:05:45.394 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.187:6831/13894964", 2024-09-21T14:05:45.394 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.187:6829/13894964", 2024-09-21T14:05:45.394 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T14:05:45.394 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T14:05:45.394 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T14:05:45.394 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:45.394 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:45.394 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T14:05:45.394 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_xinfo": [ 2024-09-21T14:05:45.394 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.394 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 0, 2024-09-21T14:05:45.395 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T14:05:45.395 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T14:05:45.395 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T14:05:45.395 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T14:05:45.395 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T14:05:45.395 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T14:05:33.900762+0000", 2024-09-21T14:05:45.395 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T14:05:45.395 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.395 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.395 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 1, 2024-09-21T14:05:45.395 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T14:05:45.395 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T14:05:45.395 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T14:05:45.395 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T14:05:45.396 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T14:05:45.396 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T14:05:07.323212+0000", 2024-09-21T14:05:45.396 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T14:05:45.396 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.396 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.396 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 2, 2024-09-21T14:05:45.396 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T14:05:45.396 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T14:05:45.396 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T14:05:45.396 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T14:05:45.396 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T14:05:45.396 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T14:05:24.517116+0000", 2024-09-21T14:05:45.396 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 100 2024-09-21T14:05:45.396 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.396 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.397 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 3, 2024-09-21T14:05:45.397 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T14:05:45.397 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T14:05:45.397 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T14:05:45.397 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T14:05:45.397 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T14:05:45.397 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T14:05:41.501105+0000", 2024-09-21T14:05:45.397 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 214 2024-09-21T14:05:45.397 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.397 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.397 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 4, 2024-09-21T14:05:45.397 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T14:05:45.397 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T14:05:45.397 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T14:05:45.397 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T14:05:45.398 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T14:05:45.398 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T14:05:37.539402+0000", 2024-09-21T14:05:45.398 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T14:05:45.398 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.398 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.398 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 5, 2024-09-21T14:05:45.398 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T14:05:45.398 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T14:05:45.398 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T14:05:45.398 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T14:05:45.398 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T14:05:45.398 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T14:05:39.824748+0000", 2024-09-21T14:05:45.398 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 324 2024-09-21T14:05:45.399 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.399 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.399 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 6, 2024-09-21T14:05:45.399 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T14:05:45.399 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T14:05:45.399 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T14:05:45.399 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T14:05:45.399 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T14:05:45.399 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T14:04:52.103632+0000", 2024-09-21T14:05:45.399 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T14:05:45.399 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.399 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.399 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 7, 2024-09-21T14:05:45.399 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T14:05:45.399 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T14:05:45.400 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T14:05:45.400 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T14:05:45.400 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T14:05:45.400 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T14:04:56.482050+0000", 2024-09-21T14:05:45.400 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 272 2024-09-21T14:05:45.400 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:45.400 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T14:05:45.400 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap": [ 2024-09-21T14:05:45.400 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.400 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "3.14", 2024-09-21T14:05:45.400 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-09-21T14:05:45.400 INFO:teuthology.orchestra.run.smithi049.stdout: 7, 2024-09-21T14:05:45.400 INFO:teuthology.orchestra.run.smithi049.stdout: 6, 2024-09-21T14:05:45.400 INFO:teuthology.orchestra.run.smithi049.stdout: 0 2024-09-21T14:05:45.400 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:45.401 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:45.401 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T14:05:45.401 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap_items": [ 2024-09-21T14:05:45.401 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.401 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "4.2", 2024-09-21T14:05:45.401 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-09-21T14:05:45.401 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.401 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 1, 2024-09-21T14:05:45.401 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 2 2024-09-21T14:05:45.401 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:45.401 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:45.401 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.401 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.401 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "4.6", 2024-09-21T14:05:45.402 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-09-21T14:05:45.402 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.402 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 1, 2024-09-21T14:05:45.402 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 5 2024-09-21T14:05:45.402 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:45.402 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:45.402 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.402 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.402 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "4.d", 2024-09-21T14:05:45.402 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-09-21T14:05:45.402 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.402 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 1, 2024-09-21T14:05:45.402 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 5 2024-09-21T14:05:45.402 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:45.402 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:45.403 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.403 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.403 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "6.7", 2024-09-21T14:05:45.403 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-09-21T14:05:45.403 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.403 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 3, 2024-09-21T14:05:45.403 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 7 2024-09-21T14:05:45.403 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:45.403 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:45.403 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.403 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.403 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "6.b", 2024-09-21T14:05:45.403 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-09-21T14:05:45.403 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.403 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 3, 2024-09-21T14:05:45.404 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 1 2024-09-21T14:05:45.404 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.404 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.404 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 7, 2024-09-21T14:05:45.404 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 6 2024-09-21T14:05:45.404 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:45.404 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:45.404 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.404 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.404 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "6.1f", 2024-09-21T14:05:45.404 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-09-21T14:05:45.404 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:05:45.404 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 3, 2024-09-21T14:05:45.404 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 0 2024-09-21T14:05:45.404 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:45.405 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:05:45.405 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:45.405 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T14:05:45.405 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap_primaries": [], 2024-09-21T14:05:45.405 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_temp": [], 2024-09-21T14:05:45.405 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_temp": [], 2024-09-21T14:05:45.405 INFO:teuthology.orchestra.run.smithi049.stdout: "blocklist": { 2024-09-21T14:05:45.405 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/2731793917": "2024-09-22T13:49:27.311313+0000", 2024-09-21T14:05:45.405 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/799831425": "2024-09-22T13:49:27.311313+0000", 2024-09-21T14:05:45.405 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/2518197280": "2024-09-22T13:43:28.110499+0000", 2024-09-21T14:05:45.405 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3556829486": "2024-09-22T13:43:28.110499+0000", 2024-09-21T14:05:45.405 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/1249926115": "2024-09-22T13:49:27.311313+0000", 2024-09-21T14:05:45.405 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/231530138": "2024-09-22T13:43:28.110499+0000", 2024-09-21T14:05:45.406 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/4220781134": "2024-09-22T13:49:27.311313+0000", 2024-09-21T14:05:45.406 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/202325455": "2024-09-22T13:43:28.110499+0000", 2024-09-21T14:05:45.406 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/4129636473": "2024-09-22T13:43:56.633723+0000", 2024-09-21T14:05:45.406 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/2518197280": "2024-09-22T13:43:28.110499+0000", 2024-09-21T14:05:45.406 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3514741273": "2024-09-22T13:43:56.633723+0000", 2024-09-21T14:05:45.406 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/4129636473": "2024-09-22T13:43:56.633723+0000", 2024-09-21T14:05:45.406 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3125743776": "2024-09-22T13:43:56.633723+0000", 2024-09-21T14:05:45.406 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3600977363": "2024-09-22T13:43:56.633723+0000", 2024-09-21T14:05:45.406 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/3453571598": "2024-09-22T13:49:27.311313+0000", 2024-09-21T14:05:45.406 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/3453571598": "2024-09-22T13:49:27.311313+0000", 2024-09-21T14:05:45.406 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3553734432": "2024-09-22T13:49:27.311313+0000" 2024-09-21T14:05:45.407 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.407 INFO:teuthology.orchestra.run.smithi049.stdout: "range_blocklist": {}, 2024-09-21T14:05:45.407 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profiles": { 2024-09-21T14:05:45.407 INFO:teuthology.orchestra.run.smithi049.stdout: "default": { 2024-09-21T14:05:45.407 INFO:teuthology.orchestra.run.smithi049.stdout: "crush-failure-domain": "osd", 2024-09-21T14:05:45.407 INFO:teuthology.orchestra.run.smithi049.stdout: "k": "2", 2024-09-21T14:05:45.407 INFO:teuthology.orchestra.run.smithi049.stdout: "m": "1", 2024-09-21T14:05:45.407 INFO:teuthology.orchestra.run.smithi049.stdout: "plugin": "jerasure", 2024-09-21T14:05:45.407 INFO:teuthology.orchestra.run.smithi049.stdout: "technique": "reed_sol_van" 2024-09-21T14:05:45.407 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:45.407 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:05:45.407 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps_queue": [], 2024-09-21T14:05:45.407 INFO:teuthology.orchestra.run.smithi049.stdout: "new_removed_snaps": [], 2024-09-21T14:05:45.407 INFO:teuthology.orchestra.run.smithi049.stdout: "new_purged_snaps": [], 2024-09-21T14:05:45.407 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_node_flags": {}, 2024-09-21T14:05:45.408 INFO:teuthology.orchestra.run.smithi049.stdout: "device_class_flags": {}, 2024-09-21T14:05:45.408 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode": { 2024-09-21T14:05:45.408 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode_enabled": false, 2024-09-21T14:05:45.408 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_bucket_count": 0, 2024-09-21T14:05:45.408 INFO:teuthology.orchestra.run.smithi049.stdout: "degraded_stretch_mode": 0, 2024-09-21T14:05:45.408 INFO:teuthology.orchestra.run.smithi049.stdout: "recovering_stretch_mode": 0, 2024-09-21T14:05:45.408 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode_bucket": 0 2024-09-21T14:05:45.408 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:05:45.408 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T14:05:45.409 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:45.432 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T14:05:45.433 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:05:45.443 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:45.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:45.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:45.465 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T14:05:45.500 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-09-21T14:05:45.545 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:05:45.581 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:45.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:45.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:05:45.683 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:45.721 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:45.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:45.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:45.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:45 smithi049 ceph-mon[30080]: pgmap v857: 168 pgs: 6 peering, 162 active+clean; 3.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 3.1 KiB/s rd, 15 KiB/s wr, 1.21k op/s; 5.5 KiB/s, 0 objects/s recovering 2024-09-21T14:05:45.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:45 smithi049 ceph-mon[30080]: 1.12 scrub starts 2024-09-21T14:05:45.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:45 smithi049 ceph-mon[30080]: 1.12 scrub ok 2024-09-21T14:05:45.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:45 smithi049 ceph-mon[30080]: 3.17 scrub starts 2024-09-21T14:05:45.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:45 smithi049 ceph-mon[30080]: 3.17 scrub ok 2024-09-21T14:05:45.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:45 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1192720238' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:05:45.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:45 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3364326922' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T14:05:45.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:45 smithi049 ceph-mon[40489]: pgmap v857: 168 pgs: 6 peering, 162 active+clean; 3.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 3.1 KiB/s rd, 15 KiB/s wr, 1.21k op/s; 5.5 KiB/s, 0 objects/s recovering 2024-09-21T14:05:45.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:45 smithi049 ceph-mon[40489]: 1.12 scrub starts 2024-09-21T14:05:45.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:45 smithi049 ceph-mon[40489]: 1.12 scrub ok 2024-09-21T14:05:45.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:45 smithi049 ceph-mon[40489]: 3.17 scrub starts 2024-09-21T14:05:45.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:45 smithi049 ceph-mon[40489]: 3.17 scrub ok 2024-09-21T14:05:45.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:45 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1192720238' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:05:45.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:45 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3364326922' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T14:05:45.823 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:05:45.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:45.868 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:45.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:05:45.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:45 smithi187 ceph-mon[33032]: pgmap v857: 168 pgs: 6 peering, 162 active+clean; 3.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 3.1 KiB/s rd, 15 KiB/s wr, 1.21k op/s; 5.5 KiB/s, 0 objects/s recovering 2024-09-21T14:05:45.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:45 smithi187 ceph-mon[33032]: 1.12 scrub starts 2024-09-21T14:05:45.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:45 smithi187 ceph-mon[33032]: 1.12 scrub ok 2024-09-21T14:05:45.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:45 smithi187 ceph-mon[33032]: 3.17 scrub starts 2024-09-21T14:05:45.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:45 smithi187 ceph-mon[33032]: 3.17 scrub ok 2024-09-21T14:05:45.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:45 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1192720238' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:05:45.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:45 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3364326922' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T14:05:45.969 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:46.001 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:46.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:46.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:46.103 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:05:46.137 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:46.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:46.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:05:46.239 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:05:46.274 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:46.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:46.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:05:46.376 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:46.410 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:46.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:46.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:46.512 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:05:46.550 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:46.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:46.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:05:46.652 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:05:46.689 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:46.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:46.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:05:46.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:46 smithi049 ceph-mon[40489]: 1.14 scrub starts 2024-09-21T14:05:46.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:46 smithi049 ceph-mon[40489]: 1.14 scrub ok 2024-09-21T14:05:46.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:46 smithi049 ceph-mon[40489]: 10.8 scrub ok 2024-09-21T14:05:46.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:46 smithi049 ceph-mon[40489]: 3.1d scrub starts 2024-09-21T14:05:46.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:46 smithi049 ceph-mon[40489]: 3.1d scrub ok 2024-09-21T14:05:46.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:46 smithi049 ceph-mon[40489]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T14:05:46.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:46 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1192720238' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T14:05:46.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:46 smithi049 ceph-mon[40489]: osdmap e441: 8 total, 8 up, 8 in 2024-09-21T14:05:46.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:46 smithi049 ceph-mon[40489]: 1.7 scrub starts 2024-09-21T14:05:46.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:46 smithi049 ceph-mon[40489]: 1.7 scrub ok 2024-09-21T14:05:46.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:46 smithi049 ceph-mon[30080]: 1.14 scrub starts 2024-09-21T14:05:46.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:46 smithi049 ceph-mon[30080]: 1.14 scrub ok 2024-09-21T14:05:46.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:46 smithi049 ceph-mon[30080]: 10.8 scrub ok 2024-09-21T14:05:46.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:46 smithi049 ceph-mon[30080]: 3.1d scrub starts 2024-09-21T14:05:46.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:46 smithi049 ceph-mon[30080]: 3.1d scrub ok 2024-09-21T14:05:46.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:46 smithi049 ceph-mon[30080]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T14:05:46.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:46 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1192720238' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T14:05:46.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:46 smithi049 ceph-mon[30080]: osdmap e441: 8 total, 8 up, 8 in 2024-09-21T14:05:46.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:46 smithi049 ceph-mon[30080]: 1.7 scrub starts 2024-09-21T14:05:46.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:46 smithi049 ceph-mon[30080]: 1.7 scrub ok 2024-09-21T14:05:46.791 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:46.824 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:46.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:46.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:46.926 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:05:46.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:46 smithi187 ceph-mon[33032]: 1.14 scrub starts 2024-09-21T14:05:46.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:46 smithi187 ceph-mon[33032]: 1.14 scrub ok 2024-09-21T14:05:46.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:46 smithi187 ceph-mon[33032]: 10.8 scrub ok 2024-09-21T14:05:46.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:46 smithi187 ceph-mon[33032]: 3.1d scrub starts 2024-09-21T14:05:46.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:46 smithi187 ceph-mon[33032]: 3.1d scrub ok 2024-09-21T14:05:46.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:46 smithi187 ceph-mon[33032]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T14:05:46.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:46 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1192720238' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T14:05:46.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:46 smithi187 ceph-mon[33032]: osdmap e441: 8 total, 8 up, 8 in 2024-09-21T14:05:46.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:46 smithi187 ceph-mon[33032]: 1.7 scrub starts 2024-09-21T14:05:46.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:46 smithi187 ceph-mon[33032]: 1.7 scrub ok 2024-09-21T14:05:46.962 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:46.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:46.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:05:47.066 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:05:47.100 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:47.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:47.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:05:47.202 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:05:47.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:47.240 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:47.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:05:47.341 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:05:47.377 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:47.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:47.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:05:47.478 INFO:tasks.thrashosds.thrasher:j is {'epoch': 440, 'fsid': '40a3d2e8-781f-11ef-baf6-efdc52797490', 'created': '2024-09-21T13:42:57.838275+0000', 'modified': '2024-09-21T14:05:42.338359+0000', 'last_up_change': '2024-09-21T14:01:40.158454+0000', 'last_in_change': '2024-09-21T13:58:16.730861+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 22, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 10, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T13:46:21.096244+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 21, 'pg_placement_num': 21, 'pg_placement_num_target': 21, 'pg_num_target': 21, 'pg_num_pending': 21, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '438', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '436', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.2899999618530273, 'score_stable': 2.2899999618530273, 'optimal_score': 1, 'raw_score_acting': 2.2899999618530273, 'raw_score_stable': 2.2899999618530273, 'primary_affinity_weighted': 0.9783573150634766, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.9800000190734863}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-21T13:48:55.540712+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 63, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 0.9783573150634766, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.9800000190734863}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-21T13:48:56.417456+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.9783573150634766, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.9800000190734863}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-21T13:48:58.128455+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 0.9783573150634766, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.9800000190734863}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-21T13:49:00.069361+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9783573150634766, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.9800000190734863}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-21T13:49:02.127399+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9783573150634766, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.9800000190734863}}, {'pool': 10, 'pool_name': 'unique_pool_3', 'create_time': '2024-09-21T14:02:55.287153+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '415', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3, 'score_stable': 3, 'optimal_score': 1, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9783573150634766, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.9800000190734863}}], 'osds': [{'osd': 0, 'uuid': '2595dca9-6219-4e0c-8728-751eb102e73a', 'up': 1, 'in': 1, 'weight': 0.6176910400390625, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 429, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6802', 'nonce': 2095561412}, {'type': 'v1', 'addr': '172.21.15.49:6803', 'nonce': 2095561412}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6804', 'nonce': 2095561412}, {'type': 'v1', 'addr': '172.21.15.49:6805', 'nonce': 2095561412}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6808', 'nonce': 2095561412}, {'type': 'v1', 'addr': '172.21.15.49:6809', 'nonce': 2095561412}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6806', 'nonce': 2095561412}, {'type': 'v1', 'addr': '172.21.15.49:6807', 'nonce': 2095561412}]}, 'public_addr': '172.21.15.49:6803/2095561412', 'cluster_addr': '172.21.15.49:6805/2095561412', 'heartbeat_back_addr': '172.21.15.49:6809/2095561412', 'heartbeat_front_addr': '172.21.15.49:6807/2095561412', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '0ba327f9-5524-4a40-abcc-729ded0b1917', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 438, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6810', 'nonce': 1382840527}, {'type': 'v1', 'addr': '172.21.15.49:6811', 'nonce': 1382840527}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6812', 'nonce': 1382840527}, {'type': 'v1', 'addr': '172.21.15.49:6813', 'nonce': 1382840527}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6816', 'nonce': 1382840527}, {'type': 'v1', 'addr': '172.21.15.49:6817', 'nonce': 1382840527}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6814', 'nonce': 1382840527}, {'type': 'v1', 'addr': '172.21.15.49:6815', 'nonce': 1382840527}]}, 'public_addr': '172.21.15.49:6811/1382840527', 'cluster_addr': '172.21.15.49:6813/1382840527', 'heartbeat_back_addr': '172.21.15.49:6817/1382840527', 'heartbeat_front_addr': '172.21.15.49:6815/1382840527', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '27a45750-204d-48fd-bb8f-ebb01273343d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.8268585205078125, 'last_clean_begin': 18, 'last_clean_end': 100, 'up_from': 130, 'up_thru': 438, 'down_at': 101, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6818', 'nonce': 1945314741}, {'type': 'v1', 'addr': '172.21.15.49:6819', 'nonce': 1945314741}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6820', 'nonce': 1945314741}, {'type': 'v1', 'addr': '172.21.15.49:6821', 'nonce': 1945314741}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6824', 'nonce': 1945314741}, {'type': 'v1', 'addr': '172.21.15.49:6825', 'nonce': 1945314741}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6822', 'nonce': 1945314741}, {'type': 'v1', 'addr': '172.21.15.49:6823', 'nonce': 1945314741}]}, 'public_addr': '172.21.15.49:6819/1945314741', 'cluster_addr': '172.21.15.49:6821/1945314741', 'heartbeat_back_addr': '172.21.15.49:6825/1945314741', 'heartbeat_front_addr': '172.21.15.49:6823/1945314741', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '528bc390-3dc9-4f3b-ab75-d7412c9625db', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 27, 'last_clean_end': 214, 'up_from': 240, 'up_thru': 438, 'down_at': 215, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6826', 'nonce': 886735721}, {'type': 'v1', 'addr': '172.21.15.49:6827', 'nonce': 886735721}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6828', 'nonce': 886735721}, {'type': 'v1', 'addr': '172.21.15.49:6829', 'nonce': 886735721}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6832', 'nonce': 886735721}, {'type': 'v1', 'addr': '172.21.15.49:6833', 'nonce': 886735721}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6830', 'nonce': 886735721}, {'type': 'v1', 'addr': '172.21.15.49:6831', 'nonce': 886735721}]}, 'public_addr': '172.21.15.49:6827/886735721', 'cluster_addr': '172.21.15.49:6829/886735721', 'heartbeat_back_addr': '172.21.15.49:6833/886735721', 'heartbeat_front_addr': '172.21.15.49:6831/886735721', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '6fc64698-2d37-4db0-bf3d-c7e612c45163', 'up': 1, 'in': 1, 'weight': 0.7888946533203125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 436, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6800', 'nonce': 1702447776}, {'type': 'v1', 'addr': '172.21.15.187:6801', 'nonce': 1702447776}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6802', 'nonce': 1702447776}, {'type': 'v1', 'addr': '172.21.15.187:6803', 'nonce': 1702447776}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6806', 'nonce': 1702447776}, {'type': 'v1', 'addr': '172.21.15.187:6807', 'nonce': 1702447776}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6804', 'nonce': 1702447776}, {'type': 'v1', 'addr': '172.21.15.187:6805', 'nonce': 1702447776}]}, 'public_addr': '172.21.15.187:6801/1702447776', 'cluster_addr': '172.21.15.187:6803/1702447776', 'heartbeat_back_addr': '172.21.15.187:6807/1702447776', 'heartbeat_front_addr': '172.21.15.187:6805/1702447776', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '8d7f62a1-7e92-4e06-9cad-86b96be514d8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 39, 'last_clean_end': 324, 'up_from': 345, 'up_thru': 438, 'down_at': 325, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6808', 'nonce': 1595353965}, {'type': 'v1', 'addr': '172.21.15.187:6809', 'nonce': 1595353965}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6810', 'nonce': 1595353965}, {'type': 'v1', 'addr': '172.21.15.187:6811', 'nonce': 1595353965}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6814', 'nonce': 1595353965}, {'type': 'v1', 'addr': '172.21.15.187:6815', 'nonce': 1595353965}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6812', 'nonce': 1595353965}, {'type': 'v1', 'addr': '172.21.15.187:6813', 'nonce': 1595353965}]}, 'public_addr': '172.21.15.187:6809/1595353965', 'cluster_addr': '172.21.15.187:6811/1595353965', 'heartbeat_back_addr': '172.21.15.187:6815/1595353965', 'heartbeat_front_addr': '172.21.15.187:6813/1595353965', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '3a7afd13-26ce-444c-9cc0-81ce2eab6ce5', 'up': 1, 'in': 1, 'weight': 0.9076080322265625, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 438, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6816', 'nonce': 2728426525}, {'type': 'v1', 'addr': '172.21.15.187:6817', 'nonce': 2728426525}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6818', 'nonce': 2728426525}, {'type': 'v1', 'addr': '172.21.15.187:6819', 'nonce': 2728426525}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6822', 'nonce': 2728426525}, {'type': 'v1', 'addr': '172.21.15.187:6823', 'nonce': 2728426525}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6820', 'nonce': 2728426525}, {'type': 'v1', 'addr': '172.21.15.187:6821', 'nonce': 2728426525}]}, 'public_addr': '172.21.15.187:6817/2728426525', 'cluster_addr': '172.21.15.187:6819/2728426525', 'heartbeat_back_addr': '172.21.15.187:6823/2728426525', 'heartbeat_front_addr': '172.21.15.187:6821/2728426525', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1e08d891-8747-402e-b4c2-aa3802ce8537', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 54, 'last_clean_end': 272, 'up_from': 298, 'up_thru': 438, 'down_at': 273, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 13894964}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 13894964}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 13894964}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 13894964}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6830', 'nonce': 13894964}, {'type': 'v1', 'addr': '172.21.15.187:6831', 'nonce': 13894964}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6828', 'nonce': 13894964}, {'type': 'v1', 'addr': '172.21.15.187:6829', 'nonce': 13894964}]}, 'public_addr': '172.21.15.187:6825/13894964', 'cluster_addr': '172.21.15.187:6827/13894964', 'heartbeat_back_addr': '172.21.15.187:6831/13894964', 'heartbeat_front_addr': '172.21.15.187:6829/13894964', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T14:05:33.900762+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T14:05:07.323212+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T14:05:24.517116+0000', 'dead_epoch': 100}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T14:05:41.501105+0000', 'dead_epoch': 214}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T14:05:37.539402+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T14:05:39.824748+0000', 'dead_epoch': 324}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T14:04:52.103632+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T14:04:56.482050+0000', 'dead_epoch': 272}], 'pg_upmap': [{'pgid': '3.14', 'osds': [7, 6, 0]}], 'pg_upmap_items': [{'pgid': '4.2', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.6', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.d', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '6.7', 'mappings': [{'from': 3, 'to': 7}]}, {'pgid': '6.b', 'mappings': [{'from': 3, 'to': 1}, {'from': 7, 'to': 6}]}, {'pgid': '6.1f', 'mappings': [{'from': 3, 'to': 0}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.49:0/2731793917': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:0/799831425': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:6801/2518197280': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:0/3556829486': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:0/1249926115': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:0/231530138': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:0/4220781134': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:0/202325455': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:6800/4129636473': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:6800/2518197280': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:0/3514741273': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:6801/4129636473': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:0/3125743776': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:0/3600977363': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:6801/3453571598': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:6800/3453571598': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:0/3553734432': '2024-09-22T13:49:27.311313+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-21T14:05:47.479 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T14:05:47.479 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:05:47.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:47.524 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:47.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:05:47.625 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:05:47.692 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:47.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:47.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:05:47.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:47 smithi049 ceph-mon[30080]: pgmap v859: 168 pgs: 1 active+clean+scrubbing, 3 peering, 164 active+clean; 3.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s 2024-09-21T14:05:47.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:47 smithi049 ceph-mon[30080]: 3.1c scrub starts 2024-09-21T14:05:47.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:47 smithi049 ceph-mon[30080]: 3.1c scrub ok 2024-09-21T14:05:47.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:47 smithi049 ceph-mon[30080]: 6.0 scrub starts 2024-09-21T14:05:47.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:47 smithi049 ceph-mon[30080]: 6.0 scrub ok 2024-09-21T14:05:47.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:47 smithi049 ceph-mon[30080]: 1.13 scrub starts 2024-09-21T14:05:47.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:47 smithi049 ceph-mon[30080]: 1.13 scrub ok 2024-09-21T14:05:47.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:47 smithi049 ceph-mon[40489]: pgmap v859: 168 pgs: 1 active+clean+scrubbing, 3 peering, 164 active+clean; 3.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s 2024-09-21T14:05:47.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:47 smithi049 ceph-mon[40489]: 3.1c scrub starts 2024-09-21T14:05:47.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:47 smithi049 ceph-mon[40489]: 3.1c scrub ok 2024-09-21T14:05:47.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:47 smithi049 ceph-mon[40489]: 6.0 scrub starts 2024-09-21T14:05:47.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:47 smithi049 ceph-mon[40489]: 6.0 scrub ok 2024-09-21T14:05:47.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:47 smithi049 ceph-mon[40489]: 1.13 scrub starts 2024-09-21T14:05:47.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:47 smithi049 ceph-mon[40489]: 1.13 scrub ok 2024-09-21T14:05:47.794 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:05:47.841 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:47.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:47.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:05:47.943 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:05:47.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:47 smithi187 ceph-mon[33032]: pgmap v859: 168 pgs: 1 active+clean+scrubbing, 3 peering, 164 active+clean; 3.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s 2024-09-21T14:05:47.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:47 smithi187 ceph-mon[33032]: 3.1c scrub starts 2024-09-21T14:05:47.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:47 smithi187 ceph-mon[33032]: 3.1c scrub ok 2024-09-21T14:05:47.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:47 smithi187 ceph-mon[33032]: 6.0 scrub starts 2024-09-21T14:05:47.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:47 smithi187 ceph-mon[33032]: 6.0 scrub ok 2024-09-21T14:05:47.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:47 smithi187 ceph-mon[33032]: 1.13 scrub starts 2024-09-21T14:05:47.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:47 smithi187 ceph-mon[33032]: 1.13 scrub ok 2024-09-21T14:05:47.984 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:47.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:47.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:05:48.055 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T14:05:48.086 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:48.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:48.119 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:48.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:48.220 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:05:48.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:48.263 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:48.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:05:48.364 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:48.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:48.396 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:48.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:48.497 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:05:48.539 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:48.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:48.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:05:48.641 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:48.643 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:05:48.666 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:48.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:48.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:48.671 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:05:48.759 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:05:48.768 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:05:48.810 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:48 smithi049 ceph-mon[30080]: 1.0 scrub starts 2024-09-21T14:05:48.811 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:48 smithi049 ceph-mon[30080]: 1.0 scrub ok 2024-09-21T14:05:48.811 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:48 smithi049 ceph-mon[30080]: 4.e scrub starts 2024-09-21T14:05:48.811 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:48 smithi049 ceph-mon[30080]: 4.e scrub ok 2024-09-21T14:05:48.811 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:48 smithi049 ceph-mon[40489]: 1.0 scrub starts 2024-09-21T14:05:48.811 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:48 smithi049 ceph-mon[40489]: 1.0 scrub ok 2024-09-21T14:05:48.811 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:48 smithi049 ceph-mon[40489]: 4.e scrub starts 2024-09-21T14:05:48.811 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:48 smithi049 ceph-mon[40489]: 4.e scrub ok 2024-09-21T14:05:48.812 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:48.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:48.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:05:48.914 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:05:48.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:48 smithi187 ceph-mon[33032]: 1.0 scrub starts 2024-09-21T14:05:48.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:48 smithi187 ceph-mon[33032]: 1.0 scrub ok 2024-09-21T14:05:48.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:48 smithi187 ceph-mon[33032]: 4.e scrub starts 2024-09-21T14:05:48.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:48 smithi187 ceph-mon[33032]: 4.e scrub ok 2024-09-21T14:05:48.963 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:48.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:48.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:05:49.065 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:05:49.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:49.095 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:49.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:05:49.196 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:05:49.239 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:49.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:49.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:05:49.341 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:49.367 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:49.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:49.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:49.469 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:49.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:49.502 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:49.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:49.603 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:05:49.645 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:49.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:49.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:05:49.736 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T14:05:49.736 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:05:49.736 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:49 smithi049 ceph-mon[40489]: pgmap v860: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 3.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 980 op/s; 36 KiB/s, 0 objects/s recovering 2024-09-21T14:05:49.736 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:49 smithi049 ceph-mon[40489]: 1.c scrub starts 2024-09-21T14:05:49.737 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:49 smithi049 ceph-mon[40489]: 1.c scrub ok 2024-09-21T14:05:49.737 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:49 smithi049 ceph-mon[40489]: 6.d scrub starts 2024-09-21T14:05:49.737 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:49 smithi049 ceph-mon[40489]: 6.d scrub ok 2024-09-21T14:05:49.737 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:49 smithi049 ceph-mon[40489]: 10.2 scrub starts 2024-09-21T14:05:49.737 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:49 smithi049 ceph-mon[40489]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-09-21T14:05:49.737 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:49 smithi049 ceph-mon[40489]: Cluster is now healthy 2024-09-21T14:05:49.737 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:49 smithi049 ceph-mon[30080]: pgmap v860: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 3.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 980 op/s; 36 KiB/s, 0 objects/s recovering 2024-09-21T14:05:49.737 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:49 smithi049 ceph-mon[30080]: 1.c scrub starts 2024-09-21T14:05:49.737 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:49 smithi049 ceph-mon[30080]: 1.c scrub ok 2024-09-21T14:05:49.737 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:49 smithi049 ceph-mon[30080]: 6.d scrub starts 2024-09-21T14:05:49.737 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:49 smithi049 ceph-mon[30080]: 6.d scrub ok 2024-09-21T14:05:49.737 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:49 smithi049 ceph-mon[30080]: 10.2 scrub starts 2024-09-21T14:05:49.738 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:49 smithi049 ceph-mon[30080]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-09-21T14:05:49.738 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:49 smithi049 ceph-mon[30080]: Cluster is now healthy 2024-09-21T14:05:49.747 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:49.775 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:49.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:49.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:49.876 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:05:49.911 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:49.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:49.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:05:49.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:49 smithi187 ceph-mon[33032]: pgmap v860: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 3.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 980 op/s; 36 KiB/s, 0 objects/s recovering 2024-09-21T14:05:49.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:49 smithi187 ceph-mon[33032]: 1.c scrub starts 2024-09-21T14:05:49.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:49 smithi187 ceph-mon[33032]: 1.c scrub ok 2024-09-21T14:05:49.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:49 smithi187 ceph-mon[33032]: 6.d scrub starts 2024-09-21T14:05:49.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:49 smithi187 ceph-mon[33032]: 6.d scrub ok 2024-09-21T14:05:49.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:49 smithi187 ceph-mon[33032]: 10.2 scrub starts 2024-09-21T14:05:49.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:49 smithi187 ceph-mon[33032]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-09-21T14:05:49.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:49 smithi187 ceph-mon[33032]: Cluster is now healthy 2024-09-21T14:05:50.013 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:05:50.040 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:50.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:50.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:05:50.141 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:50.178 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:50.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:50.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:50.280 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:05:50.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:50.313 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:50.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:05:50.417 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T14:05:50.418 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:05:50.421 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-09-21T14:05:50.444 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:05:49 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:05:50.501 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:50.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:50.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:05:50.573 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:50 smithi049 ceph-mon[40489]: 4.0 scrub starts 2024-09-21T14:05:50.573 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:50 smithi049 ceph-mon[40489]: 4.0 scrub ok 2024-09-21T14:05:50.573 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:50 smithi049 ceph-mon[40489]: 1.2 scrub starts 2024-09-21T14:05:50.573 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:50 smithi049 ceph-mon[40489]: 1.2 scrub ok 2024-09-21T14:05:50.573 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:50 smithi049 ceph-mon[40489]: 10.2 scrub ok 2024-09-21T14:05:50.573 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:50 smithi049 ceph-mon[40489]: 1.f scrub starts 2024-09-21T14:05:50.573 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:50 smithi049 ceph-mon[40489]: 1.f scrub ok 2024-09-21T14:05:50.574 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:50 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/644375712' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:05:50.574 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:50 smithi049 ceph-mon[30080]: 4.0 scrub starts 2024-09-21T14:05:50.574 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:50 smithi049 ceph-mon[30080]: 4.0 scrub ok 2024-09-21T14:05:50.574 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:50 smithi049 ceph-mon[30080]: 1.2 scrub starts 2024-09-21T14:05:50.574 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:50 smithi049 ceph-mon[30080]: 1.2 scrub ok 2024-09-21T14:05:50.574 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:50 smithi049 ceph-mon[30080]: 10.2 scrub ok 2024-09-21T14:05:50.574 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:50 smithi049 ceph-mon[30080]: 1.f scrub starts 2024-09-21T14:05:50.574 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:50 smithi049 ceph-mon[30080]: 1.f scrub ok 2024-09-21T14:05:50.574 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:50 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/644375712' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:05:50.578 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:05:50.578 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:05:50.578 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:05:50.578 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:05:50.578 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:05:50.578 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:05:50.603 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:05:50.606 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:05:50.606 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:05:50.640 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:50.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:50.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:05:50.642 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:05:50.642 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:05:50.644 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T14:05:50.675 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:05:50.676 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:05:50.711 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:05:50.712 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:05:50.742 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:05:50.747 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:05:50.747 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:05:50.812 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:50.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:50.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:05:50.914 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:05:50.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:50 smithi187 ceph-mon[33032]: 4.0 scrub starts 2024-09-21T14:05:50.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:50 smithi187 ceph-mon[33032]: 4.0 scrub ok 2024-09-21T14:05:50.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:50 smithi187 ceph-mon[33032]: 1.2 scrub starts 2024-09-21T14:05:50.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:50 smithi187 ceph-mon[33032]: 1.2 scrub ok 2024-09-21T14:05:50.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:50 smithi187 ceph-mon[33032]: 10.2 scrub ok 2024-09-21T14:05:50.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:50 smithi187 ceph-mon[33032]: 1.f scrub starts 2024-09-21T14:05:50.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:50 smithi187 ceph-mon[33032]: 1.f scrub ok 2024-09-21T14:05:50.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:50 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/644375712' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:05:50.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:50.955 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:50.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:05:50.963 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:05:50 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:05:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:05:51.057 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:05:51.093 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:51.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:51.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:05:51.195 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:51.226 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:51.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:51.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:51.327 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:05:51.364 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:51.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:51.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:05:51.466 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:05:51.499 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:51.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:51.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:05:51.601 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:05:51.636 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:51.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:51.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:05:51.652 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:51 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:05:51.652 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:51 smithi049 ceph-mon[30080]: from='client.16368 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:05:51.652 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:51 smithi049 ceph-mon[30080]: pgmap v861: 168 pgs: 168 active+clean; 3.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.36k op/s; 30 KiB/s, 0 objects/s recovering 2024-09-21T14:05:51.652 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:51 smithi049 ceph-mon[30080]: 3.1b scrub starts 2024-09-21T14:05:51.652 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:51 smithi049 ceph-mon[30080]: 3.1b scrub ok 2024-09-21T14:05:51.652 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:51 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:05:51.652 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:51 smithi049 ceph-mon[40489]: from='client.16368 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:05:51.652 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:51 smithi049 ceph-mon[40489]: pgmap v861: 168 pgs: 168 active+clean; 3.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.36k op/s; 30 KiB/s, 0 objects/s recovering 2024-09-21T14:05:51.652 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:51 smithi049 ceph-mon[40489]: 3.1b scrub starts 2024-09-21T14:05:51.653 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:51 smithi049 ceph-mon[40489]: 3.1b scrub ok 2024-09-21T14:05:51.705 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":860,"stamp":"2024-09-21T14:05:48.075505+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3797837865,"num_objects":58143,"num_object_clones":0,"num_object_copies":174429,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58143,"num_whiteouts":0,"num_read":1508,"num_read_kb":1279,"num_write":197948,"num_write_kb":8165078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23827,"num_bytes_recovered":1548398341,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80143,"ondisk_log_size":80143,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":505,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":16344604,"kb_used_data":12228632,"kb_used_omap":23,"kb_used_meta":4115560,"kb_avail":733616612,"statfs":{"total":767960285184,"available":751223410688,"internally_reserved":0,"allocated":12522119168,"data_stored":12519424682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24148,"internal_metadata":4214333868},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-435326926,"num_objects":-6640,"num_object_clones":0,"num_object_copies":-19920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6640,"num_whiteouts":0,"num_read":-12,"num_read_kb":-25,"num_write":6621,"num_write_kb":-222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":247175,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.751041"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":723,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.501732+0000","last_change":"2024-09-21T14:05:10.397918+0000","last_active":"2024-09-21T14:05:45.501732+0000","last_peered":"2024-09-21T14:05:45.501732+0000","last_clean":"2024-09-21T14:05:45.501732+0000","last_became_active":"2024-09-21T14:04:21.204299+0000","last_became_peered":"2024-09-21T14:04:21.204299+0000","last_unstale":"2024-09-21T14:05:45.501732+0000","last_undegraded":"2024-09-21T14:05:45.501732+0000","last_fullsized":"2024-09-21T14:05:45.501732+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:10.397765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:05:10.397765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:40.202681+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"290'15","reported_seq":872,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:42.378521+0000","last_change":"2024-09-21T14:01:34.785322+0000","last_active":"2024-09-21T14:05:42.378521+0000","last_peered":"2024-09-21T14:05:42.378521+0000","last_clean":"2024-09-21T14:05:42.378521+0000","last_became_active":"2024-09-21T13:58:17.771220+0000","last_became_peered":"2024-09-21T13:58:17.771220+0000","last_unstale":"2024-09-21T14:05:42.378521+0000","last_undegraded":"2024-09-21T14:05:42.378521+0000","last_fullsized":"2024-09-21T14:05:42.378521+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'15","last_scrub_stamp":"2024-09-21T14:01:34.785267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:34.785267+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:33.309104+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":821,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:42.384454+0000","last_change":"2024-09-21T14:03:33.301558+0000","last_active":"2024-09-21T14:05:42.384454+0000","last_peered":"2024-09-21T14:05:42.384454+0000","last_clean":"2024-09-21T14:05:42.384454+0000","last_became_active":"2024-09-21T14:01:41.202168+0000","last_became_peered":"2024-09-21T14:01:41.202168+0000","last_unstale":"2024-09-21T14:05:42.384454+0000","last_undegraded":"2024-09-21T14:05:42.384454+0000","last_fullsized":"2024-09-21T14:05:42.384454+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:33.301487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:33.301487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":797,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:42.380011+0000","last_change":"2024-09-21T14:05:14.348636+0000","last_active":"2024-09-21T14:05:42.380011+0000","last_peered":"2024-09-21T14:05:42.380011+0000","last_clean":"2024-09-21T14:05:42.380011+0000","last_became_active":"2024-09-21T14:01:41.203275+0000","last_became_peered":"2024-09-21T14:01:41.203275+0000","last_unstale":"2024-09-21T14:05:42.380011+0000","last_undegraded":"2024-09-21T14:05:42.380011+0000","last_fullsized":"2024-09-21T14:05:42.380011+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:14.348563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:05:14.348563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:30.470326+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":769,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.525891+0000","last_change":"2024-09-21T14:04:00.241774+0000","last_active":"2024-09-21T14:05:45.525891+0000","last_peered":"2024-09-21T14:05:45.525891+0000","last_clean":"2024-09-21T14:05:45.525891+0000","last_became_active":"2024-09-21T13:59:32.203340+0000","last_became_peered":"2024-09-21T13:59:32.203340+0000","last_unstale":"2024-09-21T14:05:45.525891+0000","last_undegraded":"2024-09-21T14:05:45.525891+0000","last_fullsized":"2024-09-21T14:05:45.525891+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:00.241562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:04:00.241562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":774,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.526420+0000","last_change":"2024-09-21T13:58:17.778322+0000","last_active":"2024-09-21T14:05:45.526420+0000","last_peered":"2024-09-21T14:05:45.526420+0000","last_clean":"2024-09-21T14:05:45.526420+0000","last_became_active":"2024-09-21T13:58:17.777893+0000","last_became_peered":"2024-09-21T13:58:17.777893+0000","last_unstale":"2024-09-21T14:05:45.526420+0000","last_undegraded":"2024-09-21T14:05:45.526420+0000","last_fullsized":"2024-09-21T14:05:45.526420+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:55:07.001135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:55:07.001135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"290'27","reported_seq":755,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.525696+0000","last_change":"2024-09-21T14:04:08.320458+0000","last_active":"2024-09-21T14:05:45.525696+0000","last_peered":"2024-09-21T14:05:45.525696+0000","last_clean":"2024-09-21T14:05:45.525696+0000","last_became_active":"2024-09-21T13:57:12.703469+0000","last_became_peered":"2024-09-21T13:57:12.703469+0000","last_unstale":"2024-09-21T14:05:45.525696+0000","last_undegraded":"2024-09-21T14:05:45.525696+0000","last_fullsized":"2024-09-21T14:05:45.525696+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'27","last_scrub_stamp":"2024-09-21T14:04:08.320339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:04:08.320339+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":526,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:42.381974+0000","last_change":"2024-09-21T14:04:35.907676+0000","last_active":"2024-09-21T14:05:42.381974+0000","last_peered":"2024-09-21T14:05:42.381974+0000","last_clean":"2024-09-21T14:05:42.381974+0000","last_became_active":"2024-09-21T14:04:21.208237+0000","last_became_peered":"2024-09-21T14:04:21.208237+0000","last_unstale":"2024-09-21T14:05:42.381974+0000","last_undegraded":"2024-09-21T14:05:42.381974+0000","last_fullsized":"2024-09-21T14:05:42.381974+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:35.907611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:04:35.907611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":782,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.501538+0000","last_change":"2024-09-21T14:05:45.271378+0000","last_active":"2024-09-21T14:05:45.501538+0000","last_peered":"2024-09-21T14:05:45.501538+0000","last_clean":"2024-09-21T14:05:45.501538+0000","last_became_active":"2024-09-21T14:01:41.194517+0000","last_became_peered":"2024-09-21T14:01:41.194517+0000","last_unstale":"2024-09-21T14:05:45.501538+0000","last_undegraded":"2024-09-21T14:05:45.501538+0000","last_fullsized":"2024-09-21T14:05:45.501538+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:45.271320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:05:45.271320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:06.487120+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"290'15","reported_seq":613,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.526088+0000","last_change":"2024-09-21T14:04:21.228412+0000","last_active":"2024-09-21T14:05:45.526088+0000","last_peered":"2024-09-21T14:05:45.526088+0000","last_clean":"2024-09-21T14:05:45.526088+0000","last_became_active":"2024-09-21T14:04:21.208244+0000","last_became_peered":"2024-09-21T14:04:21.208244+0000","last_unstale":"2024-09-21T14:05:45.526088+0000","last_undegraded":"2024-09-21T14:05:45.526088+0000","last_fullsized":"2024-09-21T14:05:45.526088+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'15","last_scrub_stamp":"2024-09-21T14:00:02.475661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:00:02.475661+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":723,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.501271+0000","last_change":"2024-09-21T14:05:12.369047+0000","last_active":"2024-09-21T14:05:45.501271+0000","last_peered":"2024-09-21T14:05:45.501271+0000","last_clean":"2024-09-21T14:05:45.501271+0000","last_became_active":"2024-09-21T14:01:41.188248+0000","last_became_peered":"2024-09-21T14:01:41.188248+0000","last_unstale":"2024-09-21T14:05:45.501271+0000","last_undegraded":"2024-09-21T14:05:45.501271+0000","last_fullsized":"2024-09-21T14:05:45.501271+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:12.368984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:05:12.368984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:27.784175+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":758,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:42.387721+0000","last_change":"2024-09-21T14:05:17.384328+0000","last_active":"2024-09-21T14:05:42.387721+0000","last_peered":"2024-09-21T14:05:42.387721+0000","last_clean":"2024-09-21T14:05:42.387721+0000","last_became_active":"2024-09-21T14:04:21.205416+0000","last_became_peered":"2024-09-21T14:04:21.205416+0000","last_unstale":"2024-09-21T14:05:42.387721+0000","last_undegraded":"2024-09-21T14:05:42.387721+0000","last_fullsized":"2024-09-21T14:05:42.387721+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:17.384142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:05:17.384142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:39.285417+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":751,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.500968+0000","last_change":"2024-09-21T14:04:10.722881+0000","last_active":"2024-09-21T14:05:45.500968+0000","last_peered":"2024-09-21T14:05:45.500968+0000","last_clean":"2024-09-21T14:05:45.500968+0000","last_became_active":"2024-09-21T14:01:41.189373+0000","last_became_peered":"2024-09-21T14:01:41.189373+0000","last_unstale":"2024-09-21T14:05:45.500968+0000","last_undegraded":"2024-09-21T14:05:45.500968+0000","last_fullsized":"2024-09-21T14:05:45.500968+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:10.722799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:10.722799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"290'7","reported_seq":613,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.526040+0000","last_change":"2024-09-21T14:04:21.231643+0000","last_active":"2024-09-21T14:05:45.526040+0000","last_peered":"2024-09-21T14:05:45.526040+0000","last_clean":"2024-09-21T14:05:45.526040+0000","last_became_active":"2024-09-21T14:04:21.213102+0000","last_became_peered":"2024-09-21T14:04:21.213102+0000","last_unstale":"2024-09-21T14:05:45.526040+0000","last_undegraded":"2024-09-21T14:05:45.526040+0000","last_fullsized":"2024-09-21T14:05:45.526040+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'7","last_scrub_stamp":"2024-09-21T14:01:57.563184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:57.563184+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":736,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.504334+0000","last_change":"2024-09-21T13:59:32.204565+0000","last_active":"2024-09-21T14:05:45.504334+0000","last_peered":"2024-09-21T14:05:45.504334+0000","last_clean":"2024-09-21T14:05:45.504334+0000","last_became_active":"2024-09-21T13:59:32.204174+0000","last_became_peered":"2024-09-21T13:59:32.204174+0000","last_unstale":"2024-09-21T14:05:45.504334+0000","last_undegraded":"2024-09-21T14:05:45.504334+0000","last_fullsized":"2024-09-21T14:05:45.504334+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:06.722293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:06.722293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":785,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.500721+0000","last_change":"2024-09-21T14:04:46.471869+0000","last_active":"2024-09-21T14:05:45.500721+0000","last_peered":"2024-09-21T14:05:45.500721+0000","last_clean":"2024-09-21T14:05:45.500721+0000","last_became_active":"2024-09-21T14:01:41.198672+0000","last_became_peered":"2024-09-21T14:01:41.198672+0000","last_unstale":"2024-09-21T14:05:45.500721+0000","last_undegraded":"2024-09-21T14:05:45.500721+0000","last_fullsized":"2024-09-21T14:05:45.500721+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:46.471817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:04:46.471817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:01.005314+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":840,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:42.379302+0000","last_change":"2024-09-21T14:03:36.018910+0000","last_active":"2024-09-21T14:05:42.379302+0000","last_peered":"2024-09-21T14:05:42.379302+0000","last_clean":"2024-09-21T14:05:42.379302+0000","last_became_active":"2024-09-21T14:01:41.200360+0000","last_became_peered":"2024-09-21T14:01:41.200360+0000","last_unstale":"2024-09-21T14:05:42.379302+0000","last_undegraded":"2024-09-21T14:05:42.379302+0000","last_fullsized":"2024-09-21T14:05:42.379302+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:36.018851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:36.018851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":757,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:42.378832+0000","last_change":"2024-09-21T14:04:21.202574+0000","last_active":"2024-09-21T14:05:42.378832+0000","last_peered":"2024-09-21T14:05:42.378832+0000","last_clean":"2024-09-21T14:05:42.378832+0000","last_became_active":"2024-09-21T14:04:21.202178+0000","last_became_peered":"2024-09-21T14:04:21.202178+0000","last_unstale":"2024-09-21T14:05:42.378832+0000","last_undegraded":"2024-09-21T14:05:42.378832+0000","last_fullsized":"2024-09-21T14:05:42.378832+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:36.679321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:02:36.679321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"290'27","reported_seq":824,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.505158+0000","last_change":"2024-09-21T14:04:01.621388+0000","last_active":"2024-09-21T14:05:45.505158+0000","last_peered":"2024-09-21T14:05:45.505158+0000","last_clean":"2024-09-21T14:05:45.505158+0000","last_became_active":"2024-09-21T13:57:12.680783+0000","last_became_peered":"2024-09-21T13:57:12.680783+0000","last_unstale":"2024-09-21T14:05:45.505158+0000","last_undegraded":"2024-09-21T14:05:45.505158+0000","last_fullsized":"2024-09-21T14:05:45.505158+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'27","last_scrub_stamp":"2024-09-21T14:04:01.621303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:04:01.621303+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":814,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:42.385367+0000","last_change":"2024-09-21T14:04:21.204283+0000","last_active":"2024-09-21T14:05:42.385367+0000","last_peered":"2024-09-21T14:05:42.385367+0000","last_clean":"2024-09-21T14:05:42.385367+0000","last_became_active":"2024-09-21T14:04:21.203937+0000","last_became_peered":"2024-09-21T14:04:21.203937+0000","last_unstale":"2024-09-21T14:05:42.385367+0000","last_undegraded":"2024-09-21T14:05:42.385367+0000","last_fullsized":"2024-09-21T14:05:42.385367+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:08.299252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:08.299252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":32,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:42.395064+0000","last_change":"2024-09-21T14:05:30.242243+0000","last_active":"2024-09-21T14:05:42.395064+0000","last_peered":"2024-09-21T14:05:42.395064+0000","last_clean":"2024-09-21T14:05:42.395064+0000","last_became_active":"2024-09-21T14:05:30.241942+0000","last_became_peered":"2024-09-21T14:05:30.241942+0000","last_unstale":"2024-09-21T14:05:42.395064+0000","last_undegraded":"2024-09-21T14:05:42.395064+0000","last_fullsized":"2024-09-21T14:05:42.395064+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:57.962266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:57.962266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:31.344923+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":809,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:42.380293+0000","last_change":"2024-09-21T14:00:51.495433+0000","last_active":"2024-09-21T14:05:42.380293+0000","last_peered":"2024-09-21T14:05:42.380293+0000","last_clean":"2024-09-21T14:05:42.380293+0000","last_became_active":"2024-09-21T13:52:04.316016+0000","last_became_peered":"2024-09-21T13:52:04.316016+0000","last_unstale":"2024-09-21T14:05:42.380293+0000","last_undegraded":"2024-09-21T14:05:42.380293+0000","last_fullsized":"2024-09-21T14:05:42.380293+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:00:51.495368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:00:51.495368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:30.355078+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"290'18","reported_seq":767,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.530040+0000","last_change":"2024-09-21T14:04:50.393599+0000","last_active":"2024-09-21T14:05:45.530040+0000","last_peered":"2024-09-21T14:05:45.530040+0000","last_clean":"2024-09-21T14:05:45.530040+0000","last_became_active":"2024-09-21T13:57:12.684454+0000","last_became_peered":"2024-09-21T13:57:12.684454+0000","last_unstale":"2024-09-21T14:05:45.530040+0000","last_undegraded":"2024-09-21T14:05:45.530040+0000","last_fullsized":"2024-09-21T14:05:45.530040+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'18","last_scrub_stamp":"2024-09-21T14:04:50.393462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:04:50.393462+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:00.435535+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":804,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:42.379744+0000","last_change":"2024-09-21T14:03:38.065257+0000","last_active":"2024-09-21T14:05:42.379744+0000","last_peered":"2024-09-21T14:05:42.379744+0000","last_clean":"2024-09-21T14:05:42.379744+0000","last_became_active":"2024-09-21T13:57:12.681897+0000","last_became_peered":"2024-09-21T13:57:12.681897+0000","last_unstale":"2024-09-21T14:05:42.379744+0000","last_undegraded":"2024-09-21T14:05:42.379744+0000","last_fullsized":"2024-09-21T14:05:42.379744+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:38.065196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:38.065196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":728,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.504957+0000","last_change":"2024-09-21T14:03:28.933216+0000","last_active":"2024-09-21T14:05:45.504957+0000","last_peered":"2024-09-21T14:05:45.504957+0000","last_clean":"2024-09-21T14:05:45.504957+0000","last_became_active":"2024-09-21T14:01:41.181946+0000","last_became_peered":"2024-09-21T14:01:41.181946+0000","last_unstale":"2024-09-21T14:05:45.504957+0000","last_undegraded":"2024-09-21T14:05:45.504957+0000","last_fullsized":"2024-09-21T14:05:45.504957+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T14:03:28.933143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:28.933143+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":830,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:42.384621+0000","last_change":"2024-09-21T14:04:21.215322+0000","last_active":"2024-09-21T14:05:42.384621+0000","last_peered":"2024-09-21T14:05:42.384621+0000","last_clean":"2024-09-21T14:05:42.384621+0000","last_became_active":"2024-09-21T14:04:21.200889+0000","last_became_peered":"2024-09-21T14:04:21.200889+0000","last_unstale":"2024-09-21T14:05:42.384621+0000","last_undegraded":"2024-09-21T14:05:42.384621+0000","last_fullsized":"2024-09-21T14:05:42.384621+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T14:01:58.994811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:01:58.994811+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:37.343845+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"290'20","reported_seq":147,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:42.395893+0000","last_change":"2024-09-21T14:04:56.589169+0000","last_active":"2024-09-21T14:05:42.395893+0000","last_peered":"2024-09-21T14:05:42.395893+0000","last_clean":"2024-09-21T14:05:42.395893+0000","last_became_active":"2024-09-21T14:04:56.588859+0000","last_became_peered":"2024-09-21T14:04:56.588859+0000","last_unstale":"2024-09-21T14:05:42.395893+0000","last_undegraded":"2024-09-21T14:05:42.395893+0000","last_fullsized":"2024-09-21T14:05:42.395893+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'20","last_scrub_stamp":"2024-09-21T14:04:42.048082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:04:42.048082+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:51.310282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"409'13","reported_seq":482,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:42.383444+0000","last_change":"2024-09-21T14:04:21.222657+0000","last_active":"2024-09-21T14:05:42.383444+0000","last_peered":"2024-09-21T14:05:42.383444+0000","last_clean":"2024-09-21T14:05:42.383444+0000","last_became_active":"2024-09-21T14:04:21.206136+0000","last_became_peered":"2024-09-21T14:04:21.206136+0000","last_unstale":"2024-09-21T14:05:42.383444+0000","last_undegraded":"2024-09-21T14:05:42.383444+0000","last_fullsized":"2024-09-21T14:05:42.383444+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'12","last_scrub_stamp":"2024-09-21T13:59:36.012693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:36.012693+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:36.927975+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"440'12524","reported_seq":12727,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:43.808062+0000","last_change":"2024-09-21T14:04:09.971242+0000","last_active":"2024-09-21T14:05:43.808062+0000","last_peered":"2024-09-21T14:05:43.808062+0000","last_clean":"2024-09-21T14:05:43.808062+0000","last_became_active":"2024-09-21T14:02:56.559536+0000","last_became_peered":"2024-09-21T14:02:56.559536+0000","last_unstale":"2024-09-21T14:05:43.808062+0000","last_undegraded":"2024-09-21T14:05:43.808062+0000","last_fullsized":"2024-09-21T14:05:43.808062+0000","mapping_epoch":381,"log_start":"410'7236","ondisk_log_start":"410'7236","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"406'6478","last_scrub_stamp":"2024-09-21T14:04:09.971039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:04:09.971039+0000","objects_scrubbed":6417,"log_size":5288,"log_dups_size":0,"ondisk_log_size":5288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1835,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":242876416,"num_objects":3706,"num_object_clones":0,"num_object_copies":11118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12524,"num_write_kb":519360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"75'79","reported_seq":235,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.501157+0000","last_change":"2024-09-21T14:05:38.314512+0000","last_active":"2024-09-21T14:05:45.501157+0000","last_peered":"2024-09-21T14:05:45.501157+0000","last_clean":"2024-09-21T14:05:45.501157+0000","last_became_active":"2024-09-21T14:01:41.246888+0000","last_became_peered":"2024-09-21T14:01:41.246888+0000","last_unstale":"2024-09-21T14:05:45.501157+0000","last_undegraded":"2024-09-21T14:05:45.501157+0000","last_fullsized":"2024-09-21T14:05:45.501157+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:05:38.314431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:05:38.314431+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:43.366780+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"290'27","reported_seq":874,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:42.388877+0000","last_change":"2024-09-21T14:04:03.259647+0000","last_active":"2024-09-21T14:05:42.388877+0000","last_peered":"2024-09-21T14:05:42.388877+0000","last_clean":"2024-09-21T14:05:42.388877+0000","last_became_active":"2024-09-21T13:57:12.695609+0000","last_became_peered":"2024-09-21T13:57:12.695609+0000","last_unstale":"2024-09-21T14:05:42.388877+0000","last_undegraded":"2024-09-21T14:05:42.388877+0000","last_fullsized":"2024-09-21T14:05:42.388877+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'27","last_scrub_stamp":"2024-09-21T14:04:03.259554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:04:03.259554+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":753,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.524443+0000","last_change":"2024-09-21T14:04:01.183193+0000","last_active":"2024-09-21T14:05:45.524443+0000","last_peered":"2024-09-21T14:05:45.524443+0000","last_clean":"2024-09-21T14:05:45.524443+0000","last_became_active":"2024-09-21T13:59:32.205350+0000","last_became_peered":"2024-09-21T13:59:32.205350+0000","last_unstale":"2024-09-21T14:05:45.524443+0000","last_undegraded":"2024-09-21T14:05:45.524443+0000","last_fullsized":"2024-09-21T14:05:45.524443+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:01.183123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:01.183123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"409'11","reported_seq":867,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.501523+0000","last_change":"2024-09-21T14:04:21.211284+0000","last_active":"2024-09-21T14:05:45.501523+0000","last_peered":"2024-09-21T14:05:45.501523+0000","last_clean":"2024-09-21T14:05:45.501523+0000","last_became_active":"2024-09-21T14:04:21.206029+0000","last_became_peered":"2024-09-21T14:04:21.206029+0000","last_unstale":"2024-09-21T14:05:45.501523+0000","last_undegraded":"2024-09-21T14:05:45.501523+0000","last_fullsized":"2024-09-21T14:05:45.501523+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'10","last_scrub_stamp":"2024-09-21T14:02:59.831128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:02:59.831128+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:07.398197+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":687,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.501119+0000","last_change":"2024-09-21T14:04:47.464272+0000","last_active":"2024-09-21T14:05:45.501119+0000","last_peered":"2024-09-21T14:05:45.501119+0000","last_clean":"2024-09-21T14:05:45.501119+0000","last_became_active":"2024-09-21T14:04:21.203505+0000","last_became_peered":"2024-09-21T14:04:21.203505+0000","last_unstale":"2024-09-21T14:05:45.501119+0000","last_undegraded":"2024-09-21T14:05:45.501119+0000","last_fullsized":"2024-09-21T14:05:45.501119+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:47.464207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:04:47.464207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:48.380565+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"440'11973","reported_seq":12170,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:43.814508+0000","last_change":"2024-09-21T14:04:08.894792+0000","last_active":"2024-09-21T14:05:43.814508+0000","last_peered":"2024-09-21T14:05:43.814508+0000","last_clean":"2024-09-21T14:05:43.814508+0000","last_became_active":"2024-09-21T14:02:56.562524+0000","last_became_peered":"2024-09-21T14:02:56.562524+0000","last_unstale":"2024-09-21T14:05:43.814508+0000","last_undegraded":"2024-09-21T14:05:43.814508+0000","last_fullsized":"2024-09-21T14:05:43.814508+0000","mapping_epoch":381,"log_start":"407'6721","ondisk_log_start":"407'6721","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"406'6081","last_scrub_stamp":"2024-09-21T14:04:08.894653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:04:08.894653+0000","objects_scrubbed":6008,"log_size":5252,"log_dups_size":0,"ondisk_log_size":5252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1756,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238354432,"num_objects":3637,"num_object_clones":0,"num_object_copies":10911,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11973,"num_write_kb":499520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"75'79","reported_seq":285,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:42.380798+0000","last_change":"2024-09-21T14:05:42.239605+0000","last_active":"2024-09-21T14:05:42.380798+0000","last_peered":"2024-09-21T14:05:42.380798+0000","last_clean":"2024-09-21T14:05:42.380798+0000","last_became_active":"2024-09-21T14:05:39.296209+0000","last_became_peered":"2024-09-21T14:05:39.296209+0000","last_unstale":"2024-09-21T14:05:42.380798+0000","last_undegraded":"2024-09-21T14:05:42.380798+0000","last_fullsized":"2024-09-21T14:05:42.380798+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:05:42.239373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:05:42.239373+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:44.080609+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"290'30","reported_seq":795,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.524545+0000","last_change":"2024-09-21T14:04:21.228589+0000","last_active":"2024-09-21T14:05:45.524545+0000","last_peered":"2024-09-21T14:05:45.524545+0000","last_clean":"2024-09-21T14:05:45.524545+0000","last_became_active":"2024-09-21T14:04:21.208982+0000","last_became_peered":"2024-09-21T14:04:21.208982+0000","last_unstale":"2024-09-21T14:05:45.524545+0000","last_undegraded":"2024-09-21T14:05:45.524545+0000","last_fullsized":"2024-09-21T14:05:45.524545+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'18","last_scrub_stamp":"2024-09-21T13:54:19.001030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:19.001030+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":705,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.527013+0000","last_change":"2024-09-21T14:04:02.214555+0000","last_active":"2024-09-21T14:05:45.527013+0000","last_peered":"2024-09-21T14:05:45.527013+0000","last_clean":"2024-09-21T14:05:45.527013+0000","last_became_active":"2024-09-21T13:59:32.211004+0000","last_became_peered":"2024-09-21T13:59:32.211004+0000","last_unstale":"2024-09-21T14:05:45.527013+0000","last_undegraded":"2024-09-21T14:05:45.527013+0000","last_fullsized":"2024-09-21T14:05:45.527013+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:02.214464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:02.214464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":814,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:42.378927+0000","last_change":"2024-09-21T13:59:34.293330+0000","last_active":"2024-09-21T14:05:42.378927+0000","last_peered":"2024-09-21T14:05:42.378927+0000","last_clean":"2024-09-21T14:05:42.378927+0000","last_became_active":"2024-09-21T13:59:32.220304+0000","last_became_peered":"2024-09-21T13:59:32.220304+0000","last_unstale":"2024-09-21T14:05:42.378927+0000","last_undegraded":"2024-09-21T14:05:42.378927+0000","last_fullsized":"2024-09-21T14:05:42.378927+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:34.292892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:34.292892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:25.376399+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":451,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.526175+0000","last_change":"2024-09-21T14:04:21.210131+0000","last_active":"2024-09-21T14:05:45.526175+0000","last_peered":"2024-09-21T14:05:45.526175+0000","last_clean":"2024-09-21T14:05:45.526175+0000","last_became_active":"2024-09-21T14:04:21.207782+0000","last_became_peered":"2024-09-21T14:04:21.207782+0000","last_unstale":"2024-09-21T14:05:45.526175+0000","last_undegraded":"2024-09-21T14:05:45.526175+0000","last_fullsized":"2024-09-21T14:05:45.526175+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:23.334162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:23.334162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"440'12426","reported_seq":12642,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:43.832845+0000","last_change":"2024-09-21T14:04:42.544112+0000","last_active":"2024-09-21T14:05:43.832845+0000","last_peered":"2024-09-21T14:05:43.832845+0000","last_clean":"2024-09-21T14:05:43.832845+0000","last_became_active":"2024-09-21T14:02:56.557055+0000","last_became_peered":"2024-09-21T14:02:56.557055+0000","last_unstale":"2024-09-21T14:05:43.832845+0000","last_undegraded":"2024-09-21T14:05:43.832845+0000","last_fullsized":"2024-09-21T14:05:43.832845+0000","mapping_epoch":381,"log_start":"411'7723","ondisk_log_start":"411'7723","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'8494","last_scrub_stamp":"2024-09-21T14:04:42.543986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:04:42.543986+0000","objects_scrubbed":7689,"log_size":4703,"log_dups_size":0,"ondisk_log_size":4703,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:45.816257+0000","scrub_duration":2665,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":240779264,"num_objects":3674,"num_object_clones":0,"num_object_copies":11022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12426,"num_write_kb":515200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"75'79","reported_seq":298,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.524418+0000","last_change":"2024-09-21T14:05:40.261679+0000","last_active":"2024-09-21T14:05:45.524418+0000","last_peered":"2024-09-21T14:05:45.524418+0000","last_clean":"2024-09-21T14:05:45.524418+0000","last_became_active":"2024-09-21T14:05:39.301665+0000","last_became_peered":"2024-09-21T14:05:39.301665+0000","last_unstale":"2024-09-21T14:05:45.524418+0000","last_undegraded":"2024-09-21T14:05:45.524418+0000","last_fullsized":"2024-09-21T14:05:45.524418+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:05:40.261587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:05:40.261587+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:51.523914+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"290'28","reported_seq":911,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:42.379013+0000","last_change":"2024-09-21T14:03:27.047111+0000","last_active":"2024-09-21T14:05:42.379013+0000","last_peered":"2024-09-21T14:05:42.379013+0000","last_clean":"2024-09-21T14:05:42.379013+0000","last_became_active":"2024-09-21T14:01:41.200859+0000","last_became_peered":"2024-09-21T14:01:41.200859+0000","last_unstale":"2024-09-21T14:05:42.379013+0000","last_undegraded":"2024-09-21T14:05:42.379013+0000","last_fullsized":"2024-09-21T14:05:42.379013+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'28","last_scrub_stamp":"2024-09-21T14:03:27.046906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:03:27.046906+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":88,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:42.388382+0000","last_change":"2024-09-21T14:04:21.216890+0000","last_active":"2024-09-21T14:05:42.388382+0000","last_peered":"2024-09-21T14:05:42.388382+0000","last_clean":"2024-09-21T14:05:42.388382+0000","last_became_active":"2024-09-21T14:04:21.205354+0000","last_became_peered":"2024-09-21T14:04:21.205354+0000","last_unstale":"2024-09-21T14:05:42.388382+0000","last_undegraded":"2024-09-21T14:05:42.388382+0000","last_fullsized":"2024-09-21T14:05:42.388382+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:57:38.046627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:57:38.046627+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:25.341767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"241'10","reported_seq":889,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.524772+0000","last_change":"2024-09-21T14:03:03.228551+0000","last_active":"2024-09-21T14:05:45.524772+0000","last_peered":"2024-09-21T14:05:45.524772+0000","last_clean":"2024-09-21T14:05:45.524772+0000","last_became_active":"2024-09-21T13:57:12.686651+0000","last_became_peered":"2024-09-21T13:57:12.686651+0000","last_unstale":"2024-09-21T14:05:45.524772+0000","last_undegraded":"2024-09-21T14:05:45.524772+0000","last_fullsized":"2024-09-21T14:05:45.524772+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'10","last_scrub_stamp":"2024-09-21T14:03:03.228443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:03.228443+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":770,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.504968+0000","last_change":"2024-09-21T14:04:45.055770+0000","last_active":"2024-09-21T14:05:45.504968+0000","last_peered":"2024-09-21T14:05:45.504968+0000","last_clean":"2024-09-21T14:05:45.504968+0000","last_became_active":"2024-09-21T13:59:32.209615+0000","last_became_peered":"2024-09-21T13:59:32.209615+0000","last_unstale":"2024-09-21T14:05:45.504968+0000","last_undegraded":"2024-09-21T14:05:45.504968+0000","last_fullsized":"2024-09-21T14:05:45.504968+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:45.055707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:04:45.055707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:03.598442+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.c","version":"441'12106","reported_seq":12238,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:46.258728+0000","last_change":"2024-09-21T14:02:56.563139+0000","last_active":"2024-09-21T14:05:46.258728+0000","last_peered":"2024-09-21T14:05:46.258728+0000","last_clean":"2024-09-21T14:05:46.258728+0000","last_became_active":"2024-09-21T14:02:56.562806+0000","last_became_peered":"2024-09-21T14:02:56.562806+0000","last_unstale":"2024-09-21T14:05:46.258728+0000","last_undegraded":"2024-09-21T14:05:46.258728+0000","last_fullsized":"2024-09-21T14:05:46.258728+0000","mapping_epoch":381,"log_start":"412'7800","ondisk_log_start":"412'7800","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":4306,"log_dups_size":0,"ondisk_log_size":4306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230948864,"num_objects":3524,"num_object_clones":0,"num_object_copies":10572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12106,"num_write_kb":500160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"75'79","reported_seq":858,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:42.382796+0000","last_change":"2024-09-21T14:05:39.299082+0000","last_active":"2024-09-21T14:05:42.382796+0000","last_peered":"2024-09-21T14:05:42.382796+0000","last_clean":"2024-09-21T14:05:42.382796+0000","last_became_active":"2024-09-21T14:05:39.298700+0000","last_became_peered":"2024-09-21T14:05:39.298700+0000","last_unstale":"2024-09-21T14:05:42.382796+0000","last_undegraded":"2024-09-21T14:05:42.382796+0000","last_fullsized":"2024-09-21T14:05:42.382796+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:04:38.868343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:04:38.868343+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"290'20","reported_seq":137,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.505107+0000","last_change":"2024-09-21T14:04:42.776698+0000","last_active":"2024-09-21T14:05:45.505107+0000","last_peered":"2024-09-21T14:05:45.505107+0000","last_clean":"2024-09-21T14:05:45.505107+0000","last_became_active":"2024-09-21T14:03:29.158218+0000","last_became_peered":"2024-09-21T14:03:29.158218+0000","last_unstale":"2024-09-21T14:05:45.505107+0000","last_undegraded":"2024-09-21T14:05:45.505107+0000","last_fullsized":"2024-09-21T14:05:45.505107+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'20","last_scrub_stamp":"2024-09-21T14:04:42.776603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:04:42.776603+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:58.993778+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"61'1","reported_seq":742,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.501362+0000","last_change":"2024-09-21T14:04:48.425745+0000","last_active":"2024-09-21T14:05:45.501362+0000","last_peered":"2024-09-21T14:05:45.501362+0000","last_clean":"2024-09-21T14:05:45.501362+0000","last_became_active":"2024-09-21T14:01:41.195268+0000","last_became_peered":"2024-09-21T14:01:41.195268+0000","last_unstale":"2024-09-21T14:05:45.501362+0000","last_undegraded":"2024-09-21T14:05:45.501362+0000","last_fullsized":"2024-09-21T14:05:45.501362+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T14:04:48.425665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:48.425665+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:14.101369+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"418'16","reported_seq":161,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:42.403881+0000","last_change":"2024-09-21T14:04:56.590159+0000","last_active":"2024-09-21T14:05:42.403881+0000","last_peered":"2024-09-21T14:05:42.403881+0000","last_clean":"2024-09-21T14:05:42.403881+0000","last_became_active":"2024-09-21T14:04:56.589828+0000","last_became_peered":"2024-09-21T14:04:56.589828+0000","last_unstale":"2024-09-21T14:05:42.403881+0000","last_undegraded":"2024-09-21T14:05:42.403881+0000","last_fullsized":"2024-09-21T14:05:42.403881+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'15","last_scrub_stamp":"2024-09-21T14:04:43.936597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:04:43.936597+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:47.698681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":251,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:42.396145+0000","last_change":"2024-09-21T14:04:56.587813+0000","last_active":"2024-09-21T14:05:42.396145+0000","last_peered":"2024-09-21T14:05:42.396145+0000","last_clean":"2024-09-21T14:05:42.396145+0000","last_became_active":"2024-09-21T14:04:56.586891+0000","last_became_peered":"2024-09-21T14:04:56.586891+0000","last_unstale":"2024-09-21T14:05:42.396145+0000","last_undegraded":"2024-09-21T14:05:42.396145+0000","last_fullsized":"2024-09-21T14:05:42.396145+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:41.400346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:04:41.400346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.d","version":"440'12288","reported_seq":23402,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:43.807345+0000","last_change":"2024-09-21T14:05:37.191906+0000","last_active":"2024-09-21T14:05:43.807345+0000","last_peered":"2024-09-21T14:05:43.807345+0000","last_clean":"2024-09-21T14:05:43.807345+0000","last_became_active":"2024-09-21T14:05:37.173042+0000","last_became_peered":"2024-09-21T14:05:37.173042+0000","last_unstale":"2024-09-21T14:05:43.807345+0000","last_undegraded":"2024-09-21T14:05:43.807345+0000","last_fullsized":"2024-09-21T14:05:43.807345+0000","mapping_epoch":434,"log_start":"411'7590","ondisk_log_start":"411'7590","created":381,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":0,"last_scrub":"406'6403","last_scrub_stamp":"2024-09-21T14:04:11.019087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:04:11.019087+0000","objects_scrubbed":6346,"log_size":4698,"log_dups_size":0,"ondisk_log_size":4698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1942,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":241893400,"num_objects":3692,"num_object_clones":0,"num_object_copies":11076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3692,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12288,"num_write_kb":511297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5445,"num_bytes_recovered":356778008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"75'79","reported_seq":874,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:42.382260+0000","last_change":"2024-09-21T14:05:41.888291+0000","last_active":"2024-09-21T14:05:42.382260+0000","last_peered":"2024-09-21T14:05:42.382260+0000","last_clean":"2024-09-21T14:05:42.382260+0000","last_became_active":"2024-09-21T14:05:39.309814+0000","last_became_peered":"2024-09-21T14:05:39.309814+0000","last_unstale":"2024-09-21T14:05:42.382260+0000","last_undegraded":"2024-09-21T14:05:42.382260+0000","last_fullsized":"2024-09-21T14:05:42.382260+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:05:41.888228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:05:41.888228+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:43.660673+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"290'29","reported_seq":552,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:42.404008+0000","last_change":"2024-09-21T14:04:56.589405+0000","last_active":"2024-09-21T14:05:42.404008+0000","last_peered":"2024-09-21T14:05:42.404008+0000","last_clean":"2024-09-21T14:05:42.404008+0000","last_became_active":"2024-09-21T14:04:56.589018+0000","last_became_peered":"2024-09-21T14:04:56.589018+0000","last_unstale":"2024-09-21T14:05:42.404008+0000","last_undegraded":"2024-09-21T14:05:42.404008+0000","last_fullsized":"2024-09-21T14:05:42.404008+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'29","last_scrub_stamp":"2024-09-21T14:04:53.477766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:04:53.477766+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:12.011852+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":815,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:42.388557+0000","last_change":"2024-09-21T14:01:41.185706+0000","last_active":"2024-09-21T14:05:42.388557+0000","last_peered":"2024-09-21T14:05:42.388557+0000","last_clean":"2024-09-21T14:05:42.388557+0000","last_became_active":"2024-09-21T14:01:41.185400+0000","last_became_peered":"2024-09-21T14:01:41.185400+0000","last_unstale":"2024-09-21T14:05:42.388557+0000","last_undegraded":"2024-09-21T14:05:42.388557+0000","last_fullsized":"2024-09-21T14:05:42.388557+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:16.712990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:58:16.712990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:26.352049+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":398,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:42.382912+0000","last_change":"2024-09-21T14:04:03.070260+0000","last_active":"2024-09-21T14:05:42.382912+0000","last_peered":"2024-09-21T14:05:42.382912+0000","last_clean":"2024-09-21T14:05:42.382912+0000","last_became_active":"2024-09-21T14:01:41.193973+0000","last_became_peered":"2024-09-21T14:01:41.193973+0000","last_unstale":"2024-09-21T14:05:42.382912+0000","last_undegraded":"2024-09-21T14:05:42.382912+0000","last_fullsized":"2024-09-21T14:05:42.382912+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:03.069973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:04:03.069973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":806,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:42.384383+0000","last_change":"2024-09-21T13:58:43.952014+0000","last_active":"2024-09-21T14:05:42.384383+0000","last_peered":"2024-09-21T14:05:42.384383+0000","last_clean":"2024-09-21T14:05:42.384383+0000","last_became_active":"2024-09-21T13:58:17.757628+0000","last_became_peered":"2024-09-21T13:58:17.757628+0000","last_unstale":"2024-09-21T14:05:42.384383+0000","last_undegraded":"2024-09-21T14:05:42.384383+0000","last_fullsized":"2024-09-21T14:05:42.384383+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:43.951941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:43.951941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:34.384802+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"440'12136","reported_seq":23192,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:43.804862+0000","last_change":"2024-09-21T14:05:36.203139+0000","last_active":"2024-09-21T14:05:43.804862+0000","last_peered":"2024-09-21T14:05:43.804862+0000","last_clean":"2024-09-21T14:05:43.804862+0000","last_became_active":"2024-09-21T14:05:36.169834+0000","last_became_peered":"2024-09-21T14:05:36.169834+0000","last_unstale":"2024-09-21T14:05:43.804862+0000","last_undegraded":"2024-09-21T14:05:43.804862+0000","last_fullsized":"2024-09-21T14:05:43.804862+0000","mapping_epoch":433,"log_start":"411'7431","ondisk_log_start":"411'7431","created":381,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":3149,"log_size":4705,"log_dups_size":0,"ondisk_log_size":4705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":240517120,"num_objects":3670,"num_object_clones":0,"num_object_copies":11010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12136,"num_write_kb":505792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5435,"num_bytes_recovered":356188160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"75'79","reported_seq":233,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.500868+0000","last_change":"2024-09-21T14:05:40.322410+0000","last_active":"2024-09-21T14:05:45.500868+0000","last_peered":"2024-09-21T14:05:45.500868+0000","last_clean":"2024-09-21T14:05:45.500868+0000","last_became_active":"2024-09-21T14:05:39.310615+0000","last_became_peered":"2024-09-21T14:05:39.310615+0000","last_unstale":"2024-09-21T14:05:45.500868+0000","last_undegraded":"2024-09-21T14:05:45.500868+0000","last_fullsized":"2024-09-21T14:05:45.500868+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:05:40.322344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:05:40.322344+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:59.230708+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"290'44","reported_seq":1099,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:43.327250+0000","last_change":"2024-09-21T14:05:38.387094+0000","last_active":"2024-09-21T14:05:43.327250+0000","last_peered":"2024-09-21T14:05:43.327250+0000","last_clean":"2024-09-21T14:05:43.327250+0000","last_became_active":"2024-09-21T13:57:12.684690+0000","last_became_peered":"2024-09-21T13:57:12.684690+0000","last_unstale":"2024-09-21T14:05:43.327250+0000","last_undegraded":"2024-09-21T14:05:43.327250+0000","last_fullsized":"2024-09-21T14:05:43.327250+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'44","last_scrub_stamp":"2024-09-21T14:05:38.386161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:05:38.386161+0000","objects_scrubbed":10,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:56.392096+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":830,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:42.378968+0000","last_change":"2024-09-21T14:04:11.158776+0000","last_active":"2024-09-21T14:05:42.378968+0000","last_peered":"2024-09-21T14:05:42.378968+0000","last_clean":"2024-09-21T14:05:42.378968+0000","last_became_active":"2024-09-21T13:58:17.770937+0000","last_became_peered":"2024-09-21T13:58:17.770937+0000","last_unstale":"2024-09-21T14:05:42.378968+0000","last_undegraded":"2024-09-21T14:05:42.378968+0000","last_fullsized":"2024-09-21T14:05:42.378968+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:11.158703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:11.158703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":770,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.501126+0000","last_change":"2024-09-21T14:05:15.377389+0000","last_active":"2024-09-21T14:05:45.501126+0000","last_peered":"2024-09-21T14:05:45.501126+0000","last_clean":"2024-09-21T14:05:45.501126+0000","last_became_active":"2024-09-21T14:01:41.247642+0000","last_became_peered":"2024-09-21T14:01:41.247642+0000","last_unstale":"2024-09-21T14:05:45.501126+0000","last_undegraded":"2024-09-21T14:05:45.501126+0000","last_fullsized":"2024-09-21T14:05:45.501126+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-21T14:05:15.377198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T14:05:15.377198+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:31.018201+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":748,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:42.385170+0000","last_change":"2024-09-21T14:05:18.354476+0000","last_active":"2024-09-21T14:05:42.385170+0000","last_peered":"2024-09-21T14:05:42.385170+0000","last_clean":"2024-09-21T14:05:42.385170+0000","last_became_active":"2024-09-21T14:04:21.204315+0000","last_became_peered":"2024-09-21T14:04:21.204315+0000","last_unstale":"2024-09-21T14:05:42.385170+0000","last_undegraded":"2024-09-21T14:05:42.385170+0000","last_fullsized":"2024-09-21T14:05:42.385170+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:18.354390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:05:18.354390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:19.770158+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":732,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.526744+0000","last_change":"2024-09-21T14:03:02.238409+0000","last_active":"2024-09-21T14:05:45.526744+0000","last_peered":"2024-09-21T14:05:45.526744+0000","last_clean":"2024-09-21T14:05:45.526744+0000","last_became_active":"2024-09-21T13:59:32.208980+0000","last_became_peered":"2024-09-21T13:59:32.208980+0000","last_unstale":"2024-09-21T14:05:45.526744+0000","last_undegraded":"2024-09-21T14:05:45.526744+0000","last_fullsized":"2024-09-21T14:05:45.526744+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:02.238357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:02.238357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"441'12196","reported_seq":12397,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.968894+0000","last_change":"2024-09-21T14:04:06.243298+0000","last_active":"2024-09-21T14:05:45.968894+0000","last_peered":"2024-09-21T14:05:45.968894+0000","last_clean":"2024-09-21T14:05:45.968894+0000","last_became_active":"2024-09-21T14:02:56.558773+0000","last_became_peered":"2024-09-21T14:02:56.558773+0000","last_unstale":"2024-09-21T14:05:45.968894+0000","last_undegraded":"2024-09-21T14:05:45.968894+0000","last_fullsized":"2024-09-21T14:05:45.968894+0000","mapping_epoch":381,"log_start":"412'7711","ondisk_log_start":"412'7711","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'5948","last_scrub_stamp":"2024-09-21T14:04:06.243212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:04:06.243212+0000","objects_scrubbed":5906,"log_size":4485,"log_dups_size":0,"ondisk_log_size":4485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:35.308518+0000","scrub_duration":1458,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230817792,"num_objects":3522,"num_object_clones":0,"num_object_copies":10566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12196,"num_write_kb":502976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"75'79","reported_seq":865,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.504232+0000","last_change":"2024-09-21T14:05:39.303986+0000","last_active":"2024-09-21T14:05:45.504232+0000","last_peered":"2024-09-21T14:05:45.504232+0000","last_clean":"2024-09-21T14:05:45.504232+0000","last_became_active":"2024-09-21T14:05:39.303625+0000","last_became_peered":"2024-09-21T14:05:39.303625+0000","last_unstale":"2024-09-21T14:05:45.504232+0000","last_undegraded":"2024-09-21T14:05:45.504232+0000","last_fullsized":"2024-09-21T14:05:45.504232+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":10,"num_write":5,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"290'17","reported_seq":859,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:42.388963+0000","last_change":"2024-09-21T14:04:21.224066+0000","last_active":"2024-09-21T14:05:42.388963+0000","last_peered":"2024-09-21T14:05:42.388963+0000","last_clean":"2024-09-21T14:05:42.388963+0000","last_became_active":"2024-09-21T14:04:21.205738+0000","last_became_peered":"2024-09-21T14:04:21.205738+0000","last_unstale":"2024-09-21T14:05:42.388963+0000","last_undegraded":"2024-09-21T14:05:42.388963+0000","last_fullsized":"2024-09-21T14:05:42.388963+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'13","last_scrub_stamp":"2024-09-21T13:58:11.619848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:11.619848+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:30.397396+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":858,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:42.387125+0000","last_change":"2024-09-21T14:04:10.193524+0000","last_active":"2024-09-21T14:05:42.387125+0000","last_peered":"2024-09-21T14:05:42.387125+0000","last_clean":"2024-09-21T14:05:42.387125+0000","last_became_active":"2024-09-21T13:58:17.759671+0000","last_became_peered":"2024-09-21T13:58:17.759671+0000","last_unstale":"2024-09-21T14:05:42.387125+0000","last_undegraded":"2024-09-21T14:05:42.387125+0000","last_fullsized":"2024-09-21T14:05:42.387125+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:10.193318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:10.193318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":733,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.504979+0000","last_change":"2024-09-21T13:58:41.644876+0000","last_active":"2024-09-21T14:05:45.504979+0000","last_peered":"2024-09-21T14:05:45.504979+0000","last_clean":"2024-09-21T14:05:45.504979+0000","last_became_active":"2024-09-21T13:57:12.682147+0000","last_became_peered":"2024-09-21T13:57:12.682147+0000","last_unstale":"2024-09-21T14:05:45.504979+0000","last_undegraded":"2024-09-21T14:05:45.504979+0000","last_fullsized":"2024-09-21T14:05:45.504979+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:58:41.644789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:58:41.644789+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:36.808348+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":720,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.507406+0000","last_change":"2024-09-21T14:05:39.821227+0000","last_active":"2024-09-21T14:05:45.507406+0000","last_peered":"2024-09-21T14:05:45.507406+0000","last_clean":"2024-09-21T14:05:45.507406+0000","last_became_active":"2024-09-21T14:01:41.182000+0000","last_became_peered":"2024-09-21T14:01:41.182000+0000","last_unstale":"2024-09-21T14:05:45.507406+0000","last_undegraded":"2024-09-21T14:05:45.507406+0000","last_fullsized":"2024-09-21T14:05:45.507406+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:39.821154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:05:39.821154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:53.732833+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":770,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.505238+0000","last_change":"2024-09-21T13:59:32.222791+0000","last_active":"2024-09-21T14:05:45.505238+0000","last_peered":"2024-09-21T14:05:45.505238+0000","last_clean":"2024-09-21T14:05:45.505238+0000","last_became_active":"2024-09-21T13:59:32.221502+0000","last_became_peered":"2024-09-21T13:59:32.221502+0000","last_unstale":"2024-09-21T14:05:45.505238+0000","last_undegraded":"2024-09-21T14:05:45.505238+0000","last_fullsized":"2024-09-21T14:05:45.505238+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:14.673582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:14.673582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"440'12394","reported_seq":12624,"reported_epoch":440,"state":"active+clean+scrubbing","last_fresh":"2024-09-21T14:05:44.567518+0000","last_change":"2024-09-21T14:05:43.376537+0000","last_active":"2024-09-21T14:05:44.567518+0000","last_peered":"2024-09-21T14:05:44.567518+0000","last_clean":"2024-09-21T14:05:44.567518+0000","last_became_active":"2024-09-21T14:04:56.586861+0000","last_became_peered":"2024-09-21T14:04:56.586861+0000","last_unstale":"2024-09-21T14:05:44.567518+0000","last_undegraded":"2024-09-21T14:05:44.567518+0000","last_fullsized":"2024-09-21T14:05:44.567518+0000","mapping_epoch":417,"log_start":"404'5916","ondisk_log_start":"404'5916","created":381,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'6143","last_scrub_stamp":"2024-09-21T14:04:06.996122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:04:06.996122+0000","objects_scrubbed":2399,"log_size":6478,"log_dups_size":0,"ondisk_log_size":6478,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"scrubbing for 1s","scrub_duration":2407,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243269632,"num_objects":3712,"num_object_clones":0,"num_object_copies":11136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12394,"num_write_kb":515392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"75'79","reported_seq":284,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:42.393779+0000","last_change":"2024-09-21T14:05:41.396162+0000","last_active":"2024-09-21T14:05:42.393779+0000","last_peered":"2024-09-21T14:05:42.393779+0000","last_clean":"2024-09-21T14:05:42.393779+0000","last_became_active":"2024-09-21T14:05:39.311460+0000","last_became_peered":"2024-09-21T14:05:39.311460+0000","last_unstale":"2024-09-21T14:05:42.393779+0000","last_undegraded":"2024-09-21T14:05:42.393779+0000","last_fullsized":"2024-09-21T14:05:42.393779+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:05:41.395817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:05:41.395817+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:50.181830+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"290'24","reported_seq":658,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.501644+0000","last_change":"2024-09-21T14:04:21.221906+0000","last_active":"2024-09-21T14:05:45.501644+0000","last_peered":"2024-09-21T14:05:45.501644+0000","last_clean":"2024-09-21T14:05:45.501644+0000","last_became_active":"2024-09-21T14:04:21.206881+0000","last_became_peered":"2024-09-21T14:04:21.206881+0000","last_unstale":"2024-09-21T14:05:45.501644+0000","last_undegraded":"2024-09-21T14:05:45.501644+0000","last_fullsized":"2024-09-21T14:05:45.501644+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'24","last_scrub_stamp":"2024-09-21T14:02:57.559936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:57.559936+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:06.366044+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":375,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:42.395600+0000","last_change":"2024-09-21T14:04:56.588103+0000","last_active":"2024-09-21T14:05:42.395600+0000","last_peered":"2024-09-21T14:05:42.395600+0000","last_clean":"2024-09-21T14:05:42.395600+0000","last_became_active":"2024-09-21T14:04:56.587384+0000","last_became_peered":"2024-09-21T14:04:56.587384+0000","last_unstale":"2024-09-21T14:05:42.395600+0000","last_undegraded":"2024-09-21T14:05:42.395600+0000","last_fullsized":"2024-09-21T14:05:42.395600+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:59.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:59:59.450546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:58.054530+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":1678,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.504486+0000","last_change":"2024-09-21T13:59:32.218796+0000","last_active":"2024-09-21T14:05:45.504486+0000","last_peered":"2024-09-21T14:05:45.504486+0000","last_clean":"2024-09-21T14:05:45.504486+0000","last_became_active":"2024-09-21T13:59:32.203361+0000","last_became_peered":"2024-09-21T13:59:32.203361+0000","last_unstale":"2024-09-21T14:05:45.504486+0000","last_undegraded":"2024-09-21T14:05:45.504486+0000","last_fullsized":"2024-09-21T14:05:45.504486+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-21T13:56:29.617143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:56:29.617143+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047048000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":380,"num_read_kb":375,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":740,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.501701+0000","last_change":"2024-09-21T14:05:14.344869+0000","last_active":"2024-09-21T14:05:45.501701+0000","last_peered":"2024-09-21T14:05:45.501701+0000","last_clean":"2024-09-21T14:05:45.501701+0000","last_became_active":"2024-09-21T14:01:41.200531+0000","last_became_peered":"2024-09-21T14:01:41.200531+0000","last_unstale":"2024-09-21T14:05:45.501701+0000","last_undegraded":"2024-09-21T14:05:45.501701+0000","last_fullsized":"2024-09-21T14:05:45.501701+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:14.344795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:05:14.344795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:33.382674+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":835,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:42.388148+0000","last_change":"2024-09-21T14:03:04.357026+0000","last_active":"2024-09-21T14:05:42.388148+0000","last_peered":"2024-09-21T14:05:42.388148+0000","last_clean":"2024-09-21T14:05:42.388148+0000","last_became_active":"2024-09-21T14:01:41.184664+0000","last_became_peered":"2024-09-21T14:01:41.184664+0000","last_unstale":"2024-09-21T14:05:42.388148+0000","last_undegraded":"2024-09-21T14:05:42.388148+0000","last_fullsized":"2024-09-21T14:05:42.388148+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:04.356979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:04.356979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"440'12135","reported_seq":12335,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:43.819307+0000","last_change":"2024-09-21T14:04:08.011633+0000","last_active":"2024-09-21T14:05:43.819307+0000","last_peered":"2024-09-21T14:05:43.819307+0000","last_clean":"2024-09-21T14:05:43.819307+0000","last_became_active":"2024-09-21T14:02:56.557263+0000","last_became_peered":"2024-09-21T14:02:56.557263+0000","last_unstale":"2024-09-21T14:05:43.819307+0000","last_undegraded":"2024-09-21T14:05:43.819307+0000","last_fullsized":"2024-09-21T14:05:43.819307+0000","mapping_epoch":381,"log_start":"411'7439","ondisk_log_start":"411'7439","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"406'6147","last_scrub_stamp":"2024-09-21T14:04:08.011556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:04:08.011556+0000","objects_scrubbed":6074,"log_size":4696,"log_dups_size":0,"ondisk_log_size":4696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1868,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247791616,"num_objects":3781,"num_object_clones":0,"num_object_copies":11343,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12135,"num_write_kb":509312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"300'32","reported_seq":516,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:42.382687+0000","last_change":"2024-09-21T14:00:47.130089+0000","last_active":"2024-09-21T14:05:42.382687+0000","last_peered":"2024-09-21T14:05:42.382687+0000","last_clean":"2024-09-21T14:05:42.382687+0000","last_became_active":"2024-09-21T13:59:32.204624+0000","last_became_peered":"2024-09-21T13:59:32.204624+0000","last_unstale":"2024-09-21T14:05:42.382687+0000","last_undegraded":"2024-09-21T14:05:42.382687+0000","last_fullsized":"2024-09-21T14:05:42.382687+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'32","last_scrub_stamp":"2024-09-21T14:00:47.129789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:00:47.129789+0000","objects_scrubbed":8,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:37.908980+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":791,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.525999+0000","last_change":"2024-09-21T14:04:42.390182+0000","last_active":"2024-09-21T14:05:45.525999+0000","last_peered":"2024-09-21T14:05:45.525999+0000","last_clean":"2024-09-21T14:05:45.525999+0000","last_became_active":"2024-09-21T14:01:41.190210+0000","last_became_peered":"2024-09-21T14:01:41.190210+0000","last_unstale":"2024-09-21T14:05:45.525999+0000","last_undegraded":"2024-09-21T14:05:45.525999+0000","last_fullsized":"2024-09-21T14:05:45.525999+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:42.390102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:42.390102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:10.628828+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"75'79","reported_seq":860,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.504605+0000","last_change":"2024-09-21T14:05:39.305521+0000","last_active":"2024-09-21T14:05:45.504605+0000","last_peered":"2024-09-21T14:05:45.504605+0000","last_clean":"2024-09-21T14:05:45.504605+0000","last_became_active":"2024-09-21T14:05:39.304753+0000","last_became_peered":"2024-09-21T14:05:39.304753+0000","last_unstale":"2024-09-21T14:05:45.504605+0000","last_undegraded":"2024-09-21T14:05:45.504605+0000","last_fullsized":"2024-09-21T14:05:45.504605+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T13:57:29.501122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T13:57:29.501122+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26834,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":9,"num_write":5,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26834,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":758,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.505078+0000","last_change":"2024-09-21T14:01:41.182186+0000","last_active":"2024-09-21T14:05:45.505078+0000","last_peered":"2024-09-21T14:05:45.505078+0000","last_clean":"2024-09-21T14:05:45.505078+0000","last_became_active":"2024-09-21T14:01:41.181899+0000","last_became_peered":"2024-09-21T14:01:41.181899+0000","last_unstale":"2024-09-21T14:05:45.505078+0000","last_undegraded":"2024-09-21T14:05:45.505078+0000","last_fullsized":"2024-09-21T14:05:45.505078+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:15.700409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:15.700409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":697,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.501549+0000","last_change":"2024-09-21T14:04:01.760941+0000","last_active":"2024-09-21T14:05:45.501549+0000","last_peered":"2024-09-21T14:05:45.501549+0000","last_clean":"2024-09-21T14:05:45.501549+0000","last_became_active":"2024-09-21T14:03:57.346453+0000","last_became_peered":"2024-09-21T14:03:57.346453+0000","last_unstale":"2024-09-21T14:05:45.501549+0000","last_undegraded":"2024-09-21T14:05:45.501549+0000","last_fullsized":"2024-09-21T14:05:45.501549+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:01.760755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:04:01.760755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:33.350016+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"75'79","reported_seq":290,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:43.453954+0000","last_change":"2024-09-21T14:05:43.453954+0000","last_active":"2024-09-21T14:05:43.453954+0000","last_peered":"2024-09-21T14:05:43.453954+0000","last_clean":"2024-09-21T14:05:43.453954+0000","last_became_active":"2024-09-21T14:05:41.345392+0000","last_became_peered":"2024-09-21T14:05:41.345392+0000","last_unstale":"2024-09-21T14:05:43.453954+0000","last_undegraded":"2024-09-21T14:05:43.453954+0000","last_fullsized":"2024-09-21T14:05:43.453954+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:05:43.453553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:05:43.453553+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:11.518687+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"440'12328","reported_seq":24311,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:43.841908+0000","last_change":"2024-09-21T14:05:26.237178+0000","last_active":"2024-09-21T14:05:43.841908+0000","last_peered":"2024-09-21T14:05:43.841908+0000","last_clean":"2024-09-21T14:05:43.841908+0000","last_became_active":"2024-09-21T14:05:26.215569+0000","last_became_peered":"2024-09-21T14:05:26.215569+0000","last_unstale":"2024-09-21T14:05:43.841908+0000","last_undegraded":"2024-09-21T14:05:43.841908+0000","last_fullsized":"2024-09-21T14:05:43.841908+0000","mapping_epoch":427,"log_start":"411'7630","ondisk_log_start":"411'7630","created":381,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"406'6598","last_scrub_stamp":"2024-09-21T14:04:14.205496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:04:14.205496+0000","objects_scrubbed":6523,"log_size":4698,"log_dups_size":0,"ondisk_log_size":4698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238419968,"num_objects":3638,"num_object_clones":0,"num_object_copies":10914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12328,"num_write_kb":510912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5879,"num_bytes_recovered":385286144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"290'19","reported_seq":867,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:42.379376+0000","last_change":"2024-09-21T14:02:28.553549+0000","last_active":"2024-09-21T14:05:42.379376+0000","last_peered":"2024-09-21T14:05:42.379376+0000","last_clean":"2024-09-21T14:05:42.379376+0000","last_became_active":"2024-09-21T13:58:17.776310+0000","last_became_peered":"2024-09-21T13:58:17.776310+0000","last_unstale":"2024-09-21T14:05:42.379376+0000","last_undegraded":"2024-09-21T14:05:42.379376+0000","last_fullsized":"2024-09-21T14:05:42.379376+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'19","last_scrub_stamp":"2024-09-21T14:02:28.553065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:28.553065+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":837,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:42.379246+0000","last_change":"2024-09-21T14:03:32.957705+0000","last_active":"2024-09-21T14:05:42.379246+0000","last_peered":"2024-09-21T14:05:42.379246+0000","last_clean":"2024-09-21T14:05:42.379246+0000","last_became_active":"2024-09-21T13:59:32.223247+0000","last_became_peered":"2024-09-21T13:59:32.223247+0000","last_unstale":"2024-09-21T14:05:42.379246+0000","last_undegraded":"2024-09-21T14:05:42.379246+0000","last_fullsized":"2024-09-21T14:05:42.379246+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:32.957649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:03:32.957649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":766,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.501419+0000","last_change":"2024-09-21T14:04:54.483668+0000","last_active":"2024-09-21T14:05:45.501419+0000","last_peered":"2024-09-21T14:05:45.501419+0000","last_clean":"2024-09-21T14:05:45.501419+0000","last_became_active":"2024-09-21T14:01:41.193941+0000","last_became_peered":"2024-09-21T14:01:41.193941+0000","last_unstale":"2024-09-21T14:05:45.501419+0000","last_undegraded":"2024-09-21T14:05:45.501419+0000","last_fullsized":"2024-09-21T14:05:45.501419+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:54.483438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:04:54.483438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:21.139691+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":768,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.525027+0000","last_change":"2024-09-21T14:03:36.302360+0000","last_active":"2024-09-21T14:05:45.525027+0000","last_peered":"2024-09-21T14:05:45.525027+0000","last_clean":"2024-09-21T14:05:45.525027+0000","last_became_active":"2024-09-21T14:01:41.190192+0000","last_became_peered":"2024-09-21T14:01:41.190192+0000","last_unstale":"2024-09-21T14:05:45.525027+0000","last_undegraded":"2024-09-21T14:05:45.525027+0000","last_fullsized":"2024-09-21T14:05:45.525027+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:36.302281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:36.302281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"441'12501","reported_seq":12711,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:48.026868+0000","last_change":"2024-09-21T14:04:52.445772+0000","last_active":"2024-09-21T14:05:48.026868+0000","last_peered":"2024-09-21T14:05:48.026868+0000","last_clean":"2024-09-21T14:05:48.026868+0000","last_became_active":"2024-09-21T14:02:56.560070+0000","last_became_peered":"2024-09-21T14:02:56.560070+0000","last_unstale":"2024-09-21T14:05:48.026868+0000","last_undegraded":"2024-09-21T14:05:48.026868+0000","last_fullsized":"2024-09-21T14:05:48.026868+0000","mapping_epoch":381,"log_start":"412'7722","ondisk_log_start":"412'7722","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'9024","last_scrub_stamp":"2024-09-21T14:04:52.445618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:04:52.445618+0000","objects_scrubbed":6949,"log_size":4779,"log_dups_size":0,"ondisk_log_size":4779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:01.306398+0000","scrub_duration":2824,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222232576,"num_objects":3391,"num_object_clones":0,"num_object_copies":10173,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12501,"num_write_kb":508544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"75'79","reported_seq":234,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.500620+0000","last_change":"2024-09-21T14:04:51.454570+0000","last_active":"2024-09-21T14:05:45.500620+0000","last_peered":"2024-09-21T14:05:45.500620+0000","last_clean":"2024-09-21T14:05:45.500620+0000","last_became_active":"2024-09-21T14:01:41.188685+0000","last_became_peered":"2024-09-21T14:01:41.188685+0000","last_unstale":"2024-09-21T14:05:45.500620+0000","last_undegraded":"2024-09-21T14:05:45.500620+0000","last_fullsized":"2024-09-21T14:05:45.500620+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:04:51.454483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:04:51.454483+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:11.924533+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"290'31","reported_seq":645,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.505865+0000","last_change":"2024-09-21T14:04:39.741288+0000","last_active":"2024-09-21T14:05:45.505865+0000","last_peered":"2024-09-21T14:05:45.505865+0000","last_clean":"2024-09-21T14:05:45.505865+0000","last_became_active":"2024-09-21T14:04:21.204164+0000","last_became_peered":"2024-09-21T14:04:21.204164+0000","last_unstale":"2024-09-21T14:05:45.505865+0000","last_undegraded":"2024-09-21T14:05:45.505865+0000","last_fullsized":"2024-09-21T14:05:45.505865+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'31","last_scrub_stamp":"2024-09-21T14:04:39.740794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:04:39.740794+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:54.328250+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":379,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:42.382606+0000","last_change":"2024-09-21T14:04:51.139211+0000","last_active":"2024-09-21T14:05:42.382606+0000","last_peered":"2024-09-21T14:05:42.382606+0000","last_clean":"2024-09-21T14:05:42.382606+0000","last_became_active":"2024-09-21T13:58:17.762520+0000","last_became_peered":"2024-09-21T13:58:17.762520+0000","last_unstale":"2024-09-21T14:05:42.382606+0000","last_undegraded":"2024-09-21T14:05:42.382606+0000","last_fullsized":"2024-09-21T14:05:42.382606+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:51.139123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:51.139123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:51.167153+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":784,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:42.388207+0000","last_change":"2024-09-21T13:58:52.030800+0000","last_active":"2024-09-21T14:05:42.388207+0000","last_peered":"2024-09-21T14:05:42.388207+0000","last_clean":"2024-09-21T14:05:42.388207+0000","last_became_active":"2024-09-21T13:49:02.067472+0000","last_became_peered":"2024-09-21T13:49:02.067472+0000","last_unstale":"2024-09-21T14:05:42.388207+0000","last_undegraded":"2024-09-21T14:05:42.388207+0000","last_fullsized":"2024-09-21T14:05:42.388207+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:52.030718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:52.030718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:32.403049+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":669,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.505620+0000","last_change":"2024-09-21T13:57:12.683321+0000","last_active":"2024-09-21T14:05:45.505620+0000","last_peered":"2024-09-21T14:05:45.505620+0000","last_clean":"2024-09-21T14:05:45.505620+0000","last_became_active":"2024-09-21T13:57:12.682979+0000","last_became_peered":"2024-09-21T13:57:12.682979+0000","last_unstale":"2024-09-21T14:05:45.505620+0000","last_undegraded":"2024-09-21T14:05:45.505620+0000","last_fullsized":"2024-09-21T14:05:45.505620+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:18.027944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:18.027944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:34.834418+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2","version":"440'12374","reported_seq":12505,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:44.792837+0000","last_change":"2024-09-21T14:02:56.559657+0000","last_active":"2024-09-21T14:05:44.792837+0000","last_peered":"2024-09-21T14:05:44.792837+0000","last_clean":"2024-09-21T14:05:44.792837+0000","last_became_active":"2024-09-21T14:02:56.559368+0000","last_became_peered":"2024-09-21T14:02:56.559368+0000","last_unstale":"2024-09-21T14:05:44.792837+0000","last_undegraded":"2024-09-21T14:05:44.792837+0000","last_fullsized":"2024-09-21T14:05:44.792837+0000","mapping_epoch":381,"log_start":"412'8045","ondisk_log_start":"412'8045","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":4329,"log_dups_size":0,"ondisk_log_size":4329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235143168,"num_objects":3588,"num_object_clones":0,"num_object_copies":10764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12374,"num_write_kb":510784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"75'79","reported_seq":860,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.509207+0000","last_change":"2024-09-21T14:02:55.806160+0000","last_active":"2024-09-21T14:05:45.509207+0000","last_peered":"2024-09-21T14:05:45.509207+0000","last_clean":"2024-09-21T14:05:45.509207+0000","last_became_active":"2024-09-21T14:00:41.139315+0000","last_became_peered":"2024-09-21T14:00:41.139315+0000","last_unstale":"2024-09-21T14:05:45.509207+0000","last_undegraded":"2024-09-21T14:05:45.509207+0000","last_fullsized":"2024-09-21T14:05:45.509207+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:02:55.806077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.806077+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"290'20","reported_seq":882,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:42.379797+0000","last_change":"2024-09-21T14:05:15.387788+0000","last_active":"2024-09-21T14:05:42.379797+0000","last_peered":"2024-09-21T14:05:42.379797+0000","last_clean":"2024-09-21T14:05:42.379797+0000","last_became_active":"2024-09-21T13:58:17.765427+0000","last_became_peered":"2024-09-21T13:58:17.765427+0000","last_unstale":"2024-09-21T14:05:42.379797+0000","last_undegraded":"2024-09-21T14:05:42.379797+0000","last_fullsized":"2024-09-21T14:05:42.379797+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'20","last_scrub_stamp":"2024-09-21T14:05:15.387706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:05:15.387706+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:39.269627+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":732,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.525711+0000","last_change":"2024-09-21T14:04:21.215419+0000","last_active":"2024-09-21T14:05:45.525711+0000","last_peered":"2024-09-21T14:05:45.525711+0000","last_clean":"2024-09-21T14:05:45.525711+0000","last_became_active":"2024-09-21T14:04:21.215102+0000","last_became_peered":"2024-09-21T14:04:21.215102+0000","last_unstale":"2024-09-21T14:05:45.525711+0000","last_undegraded":"2024-09-21T14:05:45.525711+0000","last_fullsized":"2024-09-21T14:05:45.525711+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:58.236645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:02:58.236645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"346'14","reported_seq":892,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.505952+0000","last_change":"2024-09-21T14:05:38.806902+0000","last_active":"2024-09-21T14:05:45.505952+0000","last_peered":"2024-09-21T14:05:45.505952+0000","last_clean":"2024-09-21T14:05:45.505952+0000","last_became_active":"2024-09-21T14:01:41.186564+0000","last_became_peered":"2024-09-21T14:01:41.186564+0000","last_unstale":"2024-09-21T14:05:45.505952+0000","last_undegraded":"2024-09-21T14:05:45.505952+0000","last_fullsized":"2024-09-21T14:05:45.505952+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'14","last_scrub_stamp":"2024-09-21T14:05:38.806818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:05:38.806818+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:45.465117+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"68'1","reported_seq":802,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:42.379220+0000","last_change":"2024-09-21T13:59:31.392186+0000","last_active":"2024-09-21T14:05:42.379220+0000","last_peered":"2024-09-21T14:05:42.379220+0000","last_clean":"2024-09-21T14:05:42.379220+0000","last_became_active":"2024-09-21T13:52:04.313210+0000","last_became_peered":"2024-09-21T13:52:04.313210+0000","last_unstale":"2024-09-21T14:05:42.379220+0000","last_undegraded":"2024-09-21T14:05:42.379220+0000","last_fullsized":"2024-09-21T14:05:42.379220+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T13:59:31.392105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:31.392105+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:26.357121+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.e","version":"75'79","reported_seq":16,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.505709+0000","last_change":"2024-09-21T14:05:41.349652+0000","last_active":"2024-09-21T14:05:45.505709+0000","last_peered":"2024-09-21T14:05:45.505709+0000","last_clean":"2024-09-21T14:05:45.505709+0000","last_became_active":"2024-09-21T14:05:41.349297+0000","last_became_peered":"2024-09-21T14:05:41.349297+0000","last_unstale":"2024-09-21T14:05:45.505709+0000","last_undegraded":"2024-09-21T14:05:45.505709+0000","last_fullsized":"2024-09-21T14:05:45.505709+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:04:44.986816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:04:44.986816+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":10,"num_write":5,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"441'12818","reported_seq":13034,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:48.022139+0000","last_change":"2024-09-21T14:04:48.634887+0000","last_active":"2024-09-21T14:05:48.022139+0000","last_peered":"2024-09-21T14:05:48.022139+0000","last_clean":"2024-09-21T14:05:48.022139+0000","last_became_active":"2024-09-21T14:02:56.556786+0000","last_became_peered":"2024-09-21T14:02:56.556786+0000","last_unstale":"2024-09-21T14:05:48.022139+0000","last_undegraded":"2024-09-21T14:05:48.022139+0000","last_fullsized":"2024-09-21T14:05:48.022139+0000","mapping_epoch":381,"log_start":"412'8043","ondisk_log_start":"412'8043","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'8991","last_scrub_stamp":"2024-09-21T14:04:48.631935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:04:48.631935+0000","objects_scrubbed":7324,"log_size":4775,"log_dups_size":0,"ondisk_log_size":4775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:05.417795+0000","scrub_duration":2904,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":223870976,"num_objects":3416,"num_object_clones":0,"num_object_copies":10248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12818,"num_write_kb":519488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"290'15","reported_seq":829,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:42.403927+0000","last_change":"2024-09-21T13:58:50.498597+0000","last_active":"2024-09-21T14:05:42.403927+0000","last_peered":"2024-09-21T14:05:42.403927+0000","last_clean":"2024-09-21T14:05:42.403927+0000","last_became_active":"2024-09-21T13:49:00.053016+0000","last_became_peered":"2024-09-21T13:49:00.053016+0000","last_unstale":"2024-09-21T14:05:42.403927+0000","last_undegraded":"2024-09-21T14:05:42.403927+0000","last_fullsized":"2024-09-21T14:05:42.403927+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:58:50.498369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:50.498369+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:59.036946+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":778,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.501116+0000","last_change":"2024-09-21T14:05:22.402246+0000","last_active":"2024-09-21T14:05:45.501116+0000","last_peered":"2024-09-21T14:05:45.501116+0000","last_clean":"2024-09-21T14:05:45.501116+0000","last_became_active":"2024-09-21T14:01:41.192443+0000","last_became_peered":"2024-09-21T14:01:41.192443+0000","last_unstale":"2024-09-21T14:05:45.501116+0000","last_undegraded":"2024-09-21T14:05:45.501116+0000","last_fullsized":"2024-09-21T14:05:45.501116+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:22.402182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:05:22.402182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:48.287514+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":748,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:42.380055+0000","last_change":"2024-09-21T14:02:56.796917+0000","last_active":"2024-09-21T14:05:42.380055+0000","last_peered":"2024-09-21T14:05:42.380055+0000","last_clean":"2024-09-21T14:05:42.380055+0000","last_became_active":"2024-09-21T13:57:12.680978+0000","last_became_peered":"2024-09-21T13:57:12.680978+0000","last_unstale":"2024-09-21T14:05:42.380055+0000","last_undegraded":"2024-09-21T14:05:42.380055+0000","last_fullsized":"2024-09-21T14:05:42.380055+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:56.796861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:02:56.796861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":699,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.508901+0000","last_change":"2024-09-21T14:04:21.210096+0000","last_active":"2024-09-21T14:05:45.508901+0000","last_peered":"2024-09-21T14:05:45.508901+0000","last_clean":"2024-09-21T14:05:45.508901+0000","last_became_active":"2024-09-21T14:04:21.207534+0000","last_became_peered":"2024-09-21T14:04:21.207534+0000","last_unstale":"2024-09-21T14:05:45.508901+0000","last_undegraded":"2024-09-21T14:05:45.508901+0000","last_fullsized":"2024-09-21T14:05:45.508901+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:13.698091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:13.698091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"75'79","reported_seq":864,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:42.383159+0000","last_change":"2024-09-21T14:05:41.328159+0000","last_active":"2024-09-21T14:05:42.383159+0000","last_peered":"2024-09-21T14:05:42.383159+0000","last_clean":"2024-09-21T14:05:42.383159+0000","last_became_active":"2024-09-21T14:05:41.327757+0000","last_became_peered":"2024-09-21T14:05:41.327757+0000","last_unstale":"2024-09-21T14:05:42.383159+0000","last_undegraded":"2024-09-21T14:05:42.383159+0000","last_fullsized":"2024-09-21T14:05:42.383159+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:04:38.868343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:04:38.868343+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.4","version":"440'12137","reported_seq":26008,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:43.828754+0000","last_change":"2024-09-21T14:05:13.815252+0000","last_active":"2024-09-21T14:05:43.828754+0000","last_peered":"2024-09-21T14:05:43.828754+0000","last_clean":"2024-09-21T14:05:43.828754+0000","last_became_active":"2024-09-21T14:05:13.789148+0000","last_became_peered":"2024-09-21T14:05:13.789148+0000","last_unstale":"2024-09-21T14:05:43.828754+0000","last_undegraded":"2024-09-21T14:05:43.828754+0000","last_fullsized":"2024-09-21T14:05:43.828754+0000","mapping_epoch":422,"log_start":"407'6884","ondisk_log_start":"407'6884","created":381,"last_epoch_clean":423,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.537905+0000","objects_scrubbed":0,"log_size":5253,"log_dups_size":0,"ondisk_log_size":5253,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238354432,"num_objects":3637,"num_object_clones":0,"num_object_copies":10911,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12137,"num_write_kb":504768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6862,"num_bytes_recovered":449708032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"290'35","reported_seq":445,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:42.382801+0000","last_change":"2024-09-21T14:05:38.897778+0000","last_active":"2024-09-21T14:05:42.382801+0000","last_peered":"2024-09-21T14:05:42.382801+0000","last_clean":"2024-09-21T14:05:42.382801+0000","last_became_active":"2024-09-21T13:59:32.204184+0000","last_became_peered":"2024-09-21T13:59:32.204184+0000","last_unstale":"2024-09-21T14:05:42.382801+0000","last_undegraded":"2024-09-21T14:05:42.382801+0000","last_fullsized":"2024-09-21T14:05:42.382801+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'35","last_scrub_stamp":"2024-09-21T14:05:38.897722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:05:38.897722+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:03.684912+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":792,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.505558+0000","last_change":"2024-09-21T14:03:31.882264+0000","last_active":"2024-09-21T14:05:45.505558+0000","last_peered":"2024-09-21T14:05:45.505558+0000","last_clean":"2024-09-21T14:05:45.505558+0000","last_became_active":"2024-09-21T14:01:41.184567+0000","last_became_peered":"2024-09-21T14:01:41.184567+0000","last_unstale":"2024-09-21T14:05:45.505558+0000","last_undegraded":"2024-09-21T14:05:45.505558+0000","last_fullsized":"2024-09-21T14:05:45.505558+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:31.882215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:03:31.882215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":771,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.508663+0000","last_change":"2024-09-21T14:04:48.685009+0000","last_active":"2024-09-21T14:05:45.508663+0000","last_peered":"2024-09-21T14:05:45.508663+0000","last_clean":"2024-09-21T14:05:45.508663+0000","last_became_active":"2024-09-21T14:01:41.181911+0000","last_became_peered":"2024-09-21T14:01:41.181911+0000","last_unstale":"2024-09-21T14:05:45.508663+0000","last_undegraded":"2024-09-21T14:05:45.508663+0000","last_fullsized":"2024-09-21T14:05:45.508663+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:48.684941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:04:48.684941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:08.622354+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":719,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.504529+0000","last_change":"2024-09-21T14:03:33.893158+0000","last_active":"2024-09-21T14:05:45.504529+0000","last_peered":"2024-09-21T14:05:45.504529+0000","last_clean":"2024-09-21T14:05:45.504529+0000","last_became_active":"2024-09-21T13:59:32.219332+0000","last_became_peered":"2024-09-21T13:59:32.219332+0000","last_unstale":"2024-09-21T14:05:45.504529+0000","last_undegraded":"2024-09-21T14:05:45.504529+0000","last_fullsized":"2024-09-21T14:05:45.504529+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:33.893072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:33.893072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"75'79","reported_seq":302,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.524568+0000","last_change":"2024-09-21T14:05:41.372774+0000","last_active":"2024-09-21T14:05:45.524568+0000","last_peered":"2024-09-21T14:05:45.524568+0000","last_clean":"2024-09-21T14:05:45.524568+0000","last_became_active":"2024-09-21T14:05:41.340686+0000","last_became_peered":"2024-09-21T14:05:41.340686+0000","last_unstale":"2024-09-21T14:05:45.524568+0000","last_undegraded":"2024-09-21T14:05:45.524568+0000","last_fullsized":"2024-09-21T14:05:45.524568+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:04:49.393415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:04:49.393415+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109227,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.7","version":"440'12416","reported_seq":12635,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:43.832959+0000","last_change":"2024-09-21T14:04:40.683339+0000","last_active":"2024-09-21T14:05:43.832959+0000","last_peered":"2024-09-21T14:05:43.832959+0000","last_clean":"2024-09-21T14:05:43.832959+0000","last_became_active":"2024-09-21T14:02:56.559331+0000","last_became_peered":"2024-09-21T14:02:56.559331+0000","last_unstale":"2024-09-21T14:05:43.832959+0000","last_undegraded":"2024-09-21T14:05:43.832959+0000","last_fullsized":"2024-09-21T14:05:43.832959+0000","mapping_epoch":381,"log_start":"411'7703","ondisk_log_start":"411'7703","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'8371","last_scrub_stamp":"2024-09-21T14:04:40.683117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:04:40.683117+0000","objects_scrubbed":7905,"log_size":4713,"log_dups_size":0,"ondisk_log_size":4713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:47.784341+0000","scrub_duration":2828,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":246939648,"num_objects":3768,"num_object_clones":0,"num_object_copies":11304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12416,"num_write_kb":517888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"290'18","reported_seq":854,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:42.379157+0000","last_change":"2024-09-21T14:05:23.423056+0000","last_active":"2024-09-21T14:05:42.379157+0000","last_peered":"2024-09-21T14:05:42.379157+0000","last_clean":"2024-09-21T14:05:42.379157+0000","last_became_active":"2024-09-21T13:57:12.680627+0000","last_became_peered":"2024-09-21T13:57:12.680627+0000","last_unstale":"2024-09-21T14:05:42.379157+0000","last_undegraded":"2024-09-21T14:05:42.379157+0000","last_fullsized":"2024-09-21T14:05:42.379157+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'18","last_scrub_stamp":"2024-09-21T14:05:23.420582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:05:23.420582+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:45.787887+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":746,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.506202+0000","last_change":"2024-09-21T14:04:21.206034+0000","last_active":"2024-09-21T14:05:45.506202+0000","last_peered":"2024-09-21T14:05:45.506202+0000","last_clean":"2024-09-21T14:05:45.506202+0000","last_became_active":"2024-09-21T14:04:21.205699+0000","last_became_peered":"2024-09-21T14:04:21.205699+0000","last_unstale":"2024-09-21T14:05:45.506202+0000","last_undegraded":"2024-09-21T14:05:45.506202+0000","last_fullsized":"2024-09-21T14:05:45.506202+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:21.582325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:21.582325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:30.406016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":725,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.506747+0000","last_change":"2024-09-21T14:02:27.791041+0000","last_active":"2024-09-21T14:05:45.506747+0000","last_peered":"2024-09-21T14:05:45.506747+0000","last_clean":"2024-09-21T14:05:45.506747+0000","last_became_active":"2024-09-21T13:52:04.322147+0000","last_became_peered":"2024-09-21T13:52:04.322147+0000","last_unstale":"2024-09-21T14:05:45.506747+0000","last_undegraded":"2024-09-21T14:05:45.506747+0000","last_fullsized":"2024-09-21T14:05:45.506747+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:27.790994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:02:27.790994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":740,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.526133+0000","last_change":"2024-09-21T14:05:30.243329+0000","last_active":"2024-09-21T14:05:45.526133+0000","last_peered":"2024-09-21T14:05:45.526133+0000","last_clean":"2024-09-21T14:05:45.526133+0000","last_became_active":"2024-09-21T14:05:30.243052+0000","last_became_peered":"2024-09-21T14:05:30.243052+0000","last_unstale":"2024-09-21T14:05:45.526133+0000","last_undegraded":"2024-09-21T14:05:45.526133+0000","last_fullsized":"2024-09-21T14:05:45.526133+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:36.371747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:02:36.371747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"75'79","reported_seq":16,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.505041+0000","last_change":"2024-09-21T14:05:41.345101+0000","last_active":"2024-09-21T14:05:45.505041+0000","last_peered":"2024-09-21T14:05:45.505041+0000","last_clean":"2024-09-21T14:05:45.505041+0000","last_became_active":"2024-09-21T14:05:41.344754+0000","last_became_peered":"2024-09-21T14:05:41.344754+0000","last_unstale":"2024-09-21T14:05:45.505041+0000","last_undegraded":"2024-09-21T14:05:45.505041+0000","last_fullsized":"2024-09-21T14:05:45.505041+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:02:06.395011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:02:06.395011+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":10,"num_write":5,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.6","version":"441'12454","reported_seq":12666,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.949628+0000","last_change":"2024-09-21T14:04:52.260111+0000","last_active":"2024-09-21T14:05:45.949628+0000","last_peered":"2024-09-21T14:05:45.949628+0000","last_clean":"2024-09-21T14:05:45.949628+0000","last_became_active":"2024-09-21T14:02:56.563788+0000","last_became_peered":"2024-09-21T14:02:56.563788+0000","last_unstale":"2024-09-21T14:05:45.949628+0000","last_undegraded":"2024-09-21T14:05:45.949628+0000","last_fullsized":"2024-09-21T14:05:45.949628+0000","mapping_epoch":381,"log_start":"412'7933","ondisk_log_start":"412'7933","created":381,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'9041","last_scrub_stamp":"2024-09-21T14:04:52.260025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:02:55.537905+0000","last_clean_scrub_stamp":"2024-09-21T14:04:52.260025+0000","objects_scrubbed":7053,"log_size":4521,"log_dups_size":0,"ondisk_log_size":4521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:57.305832+0000","scrub_duration":1833,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235143168,"num_objects":3588,"num_object_clones":0,"num_object_copies":10764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12454,"num_write_kb":513344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"290'27","reported_seq":854,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.500718+0000","last_change":"2024-09-21T14:05:17.381123+0000","last_active":"2024-09-21T14:05:45.500718+0000","last_peered":"2024-09-21T14:05:45.500718+0000","last_clean":"2024-09-21T14:05:45.500718+0000","last_became_active":"2024-09-21T14:01:41.192481+0000","last_became_peered":"2024-09-21T14:01:41.192481+0000","last_unstale":"2024-09-21T14:05:45.500718+0000","last_undegraded":"2024-09-21T14:05:45.500718+0000","last_fullsized":"2024-09-21T14:05:45.500718+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'27","last_scrub_stamp":"2024-09-21T14:05:17.381049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:05:17.381049+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:32.455335+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":732,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.506551+0000","last_change":"2024-09-21T14:04:21.218814+0000","last_active":"2024-09-21T14:05:45.506551+0000","last_peered":"2024-09-21T14:05:45.506551+0000","last_clean":"2024-09-21T14:05:45.506551+0000","last_became_active":"2024-09-21T14:04:21.201644+0000","last_became_peered":"2024-09-21T14:04:21.201644+0000","last_unstale":"2024-09-21T14:05:45.506551+0000","last_undegraded":"2024-09-21T14:05:45.506551+0000","last_fullsized":"2024-09-21T14:05:45.506551+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T14:03:04.953658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:03:04.953658+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"300'14","reported_seq":960,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.508747+0000","last_change":"2024-09-21T13:59:32.220898+0000","last_active":"2024-09-21T14:05:45.508747+0000","last_peered":"2024-09-21T14:05:45.508747+0000","last_clean":"2024-09-21T14:05:45.508747+0000","last_became_active":"2024-09-21T13:59:32.206956+0000","last_became_peered":"2024-09-21T13:59:32.206956+0000","last_unstale":"2024-09-21T14:05:45.508747+0000","last_undegraded":"2024-09-21T14:05:45.508747+0000","last_fullsized":"2024-09-21T14:05:45.508747+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'11","last_scrub_stamp":"2024-09-21T13:56:31.616003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:56:31.616003+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:35.423474+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":794,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.500966+0000","last_change":"2024-09-21T14:04:42.433805+0000","last_active":"2024-09-21T14:05:45.500966+0000","last_peered":"2024-09-21T14:05:45.500966+0000","last_clean":"2024-09-21T14:05:45.500966+0000","last_became_active":"2024-09-21T14:01:41.246106+0000","last_became_peered":"2024-09-21T14:01:41.246106+0000","last_unstale":"2024-09-21T14:05:45.500966+0000","last_undegraded":"2024-09-21T14:05:45.500966+0000","last_fullsized":"2024-09-21T14:05:45.500966+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:42.433729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:04:42.433729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:04.041353+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"75'79","reported_seq":24,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.500747+0000","last_change":"2024-09-21T14:05:44.254364+0000","last_active":"2024-09-21T14:05:45.500747+0000","last_peered":"2024-09-21T14:05:45.500747+0000","last_clean":"2024-09-21T14:05:45.500747+0000","last_became_active":"2024-09-21T14:05:41.352416+0000","last_became_peered":"2024-09-21T14:05:41.352416+0000","last_unstale":"2024-09-21T14:05:45.500747+0000","last_undegraded":"2024-09-21T14:05:45.500747+0000","last_fullsized":"2024-09-21T14:05:45.500747+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":5,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:05:44.254297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:05:44.254297+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:12.352853+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":381,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:42.382058+0000","last_change":"2024-09-21T14:04:21.198859+0000","last_active":"2024-09-21T14:05:42.382058+0000","last_peered":"2024-09-21T14:05:42.382058+0000","last_clean":"2024-09-21T14:05:42.382058+0000","last_became_active":"2024-09-21T14:04:21.197402+0000","last_became_peered":"2024-09-21T14:04:21.197402+0000","last_unstale":"2024-09-21T14:05:42.382058+0000","last_undegraded":"2024-09-21T14:05:42.382058+0000","last_fullsized":"2024-09-21T14:05:42.382058+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:01:50.733007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:01:50.733007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"290'12","reported_seq":760,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.530545+0000","last_change":"2024-09-21T13:59:32.247422+0000","last_active":"2024-09-21T14:05:45.530545+0000","last_peered":"2024-09-21T14:05:45.530545+0000","last_clean":"2024-09-21T14:05:45.530545+0000","last_became_active":"2024-09-21T13:59:32.211306+0000","last_became_peered":"2024-09-21T13:59:32.211306+0000","last_unstale":"2024-09-21T14:05:45.530545+0000","last_undegraded":"2024-09-21T14:05:45.530545+0000","last_fullsized":"2024-09-21T14:05:45.530545+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'6","last_scrub_stamp":"2024-09-21T13:55:12.066873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:55:12.066873+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":726,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.501412+0000","last_change":"2024-09-21T14:05:19.378454+0000","last_active":"2024-09-21T14:05:45.501412+0000","last_peered":"2024-09-21T14:05:45.501412+0000","last_clean":"2024-09-21T14:05:45.501412+0000","last_became_active":"2024-09-21T14:01:41.199321+0000","last_became_peered":"2024-09-21T14:01:41.199321+0000","last_unstale":"2024-09-21T14:05:45.501412+0000","last_undegraded":"2024-09-21T14:05:45.501412+0000","last_fullsized":"2024-09-21T14:05:45.501412+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:19.378393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:05:19.378393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:35.761731+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":765,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.505420+0000","last_change":"2024-09-21T14:03:01.872986+0000","last_active":"2024-09-21T14:05:45.505420+0000","last_peered":"2024-09-21T14:05:45.505420+0000","last_clean":"2024-09-21T14:05:45.505420+0000","last_became_active":"2024-09-21T13:59:32.219525+0000","last_became_peered":"2024-09-21T13:59:32.219525+0000","last_unstale":"2024-09-21T14:05:45.505420+0000","last_undegraded":"2024-09-21T14:05:45.505420+0000","last_fullsized":"2024-09-21T14:05:45.505420+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:01.872928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:01.872928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"75'79","reported_seq":282,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:42.393194+0000","last_change":"2024-09-21T14:05:41.346564+0000","last_active":"2024-09-21T14:05:42.393194+0000","last_peered":"2024-09-21T14:05:42.393194+0000","last_clean":"2024-09-21T14:05:42.393194+0000","last_became_active":"2024-09-21T14:05:41.345778+0000","last_became_peered":"2024-09-21T14:05:41.345778+0000","last_unstale":"2024-09-21T14:05:42.393194+0000","last_undegraded":"2024-09-21T14:05:42.393194+0000","last_fullsized":"2024-09-21T14:05:42.393194+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":5,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:05:13.503720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:05:13.503720+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"290'15","reported_seq":740,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.526562+0000","last_change":"2024-09-21T14:04:21.228883+0000","last_active":"2024-09-21T14:05:45.526562+0000","last_peered":"2024-09-21T14:05:45.526562+0000","last_clean":"2024-09-21T14:05:45.526562+0000","last_became_active":"2024-09-21T14:04:21.208398+0000","last_became_peered":"2024-09-21T14:04:21.208398+0000","last_unstale":"2024-09-21T14:05:45.526562+0000","last_undegraded":"2024-09-21T14:05:45.526562+0000","last_fullsized":"2024-09-21T14:05:45.526562+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:55:04.990842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:55:04.990842+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"61'1","reported_seq":783,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.509118+0000","last_change":"2024-09-21T14:04:04.012140+0000","last_active":"2024-09-21T14:05:45.509118+0000","last_peered":"2024-09-21T14:05:45.509118+0000","last_clean":"2024-09-21T14:05:45.509118+0000","last_became_active":"2024-09-21T13:59:32.219515+0000","last_became_peered":"2024-09-21T13:59:32.219515+0000","last_unstale":"2024-09-21T14:05:45.509118+0000","last_undegraded":"2024-09-21T14:05:45.509118+0000","last_fullsized":"2024-09-21T14:05:45.509118+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T14:04:04.012006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:04.012006+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":716,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.526939+0000","last_change":"2024-09-21T14:03:33.309440+0000","last_active":"2024-09-21T14:05:45.526939+0000","last_peered":"2024-09-21T14:05:45.526939+0000","last_clean":"2024-09-21T14:05:45.526939+0000","last_became_active":"2024-09-21T13:59:32.210586+0000","last_became_peered":"2024-09-21T13:59:32.210586+0000","last_unstale":"2024-09-21T14:05:45.526939+0000","last_undegraded":"2024-09-21T14:05:45.526939+0000","last_fullsized":"2024-09-21T14:05:45.526939+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:33.309248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:33.309248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":766,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.504871+0000","last_change":"2024-09-21T14:02:29.731635+0000","last_active":"2024-09-21T14:05:45.504871+0000","last_peered":"2024-09-21T14:05:45.504871+0000","last_clean":"2024-09-21T14:05:45.504871+0000","last_became_active":"2024-09-21T13:59:32.205803+0000","last_became_peered":"2024-09-21T13:59:32.205803+0000","last_unstale":"2024-09-21T14:05:45.504871+0000","last_undegraded":"2024-09-21T14:05:45.504871+0000","last_fullsized":"2024-09-21T14:05:45.504871+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:29.731565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:02:29.731565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"75'79","reported_seq":24,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.500539+0000","last_change":"2024-09-21T14:05:42.242477+0000","last_active":"2024-09-21T14:05:45.500539+0000","last_peered":"2024-09-21T14:05:45.500539+0000","last_clean":"2024-09-21T14:05:45.500539+0000","last_became_active":"2024-09-21T14:05:41.325618+0000","last_became_peered":"2024-09-21T14:05:41.325618+0000","last_unstale":"2024-09-21T14:05:45.500539+0000","last_undegraded":"2024-09-21T14:05:45.500539+0000","last_fullsized":"2024-09-21T14:05:45.500539+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":5,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:05:42.242405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:05:42.242405+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:50.588885+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"290'15","reported_seq":763,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.501477+0000","last_change":"2024-09-21T14:01:41.189676+0000","last_active":"2024-09-21T14:05:45.501477+0000","last_peered":"2024-09-21T14:05:45.501477+0000","last_clean":"2024-09-21T14:05:45.501477+0000","last_became_active":"2024-09-21T14:01:41.187904+0000","last_became_peered":"2024-09-21T14:01:41.187904+0000","last_unstale":"2024-09-21T14:05:45.501477+0000","last_undegraded":"2024-09-21T14:05:45.501477+0000","last_fullsized":"2024-09-21T14:05:45.501477+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:57:24.124604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:57:24.124604+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:32.389481+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":405,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:42.400217+0000","last_change":"2024-09-21T14:04:56.588256+0000","last_active":"2024-09-21T14:05:42.400217+0000","last_peered":"2024-09-21T14:05:42.400217+0000","last_clean":"2024-09-21T14:05:42.400217+0000","last_became_active":"2024-09-21T14:04:56.587953+0000","last_became_peered":"2024-09-21T14:04:56.587953+0000","last_unstale":"2024-09-21T14:05:42.400217+0000","last_undegraded":"2024-09-21T14:05:42.400217+0000","last_fullsized":"2024-09-21T14:05:42.400217+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:37.078458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:37.078458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:52.347012+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"300'14","reported_seq":893,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.525265+0000","last_change":"2024-09-21T13:59:32.208253+0000","last_active":"2024-09-21T14:05:45.525265+0000","last_peered":"2024-09-21T14:05:45.525265+0000","last_clean":"2024-09-21T14:05:45.525265+0000","last_became_active":"2024-09-21T13:59:32.206926+0000","last_became_peered":"2024-09-21T13:59:32.206926+0000","last_unstale":"2024-09-21T14:05:45.525265+0000","last_undegraded":"2024-09-21T14:05:45.525265+0000","last_fullsized":"2024-09-21T14:05:45.525265+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'10","last_scrub_stamp":"2024-09-21T13:54:20.946782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:54:20.946782+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":816,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:42.378978+0000","last_change":"2024-09-21T14:05:20.368393+0000","last_active":"2024-09-21T14:05:42.378978+0000","last_peered":"2024-09-21T14:05:42.378978+0000","last_clean":"2024-09-21T14:05:42.378978+0000","last_became_active":"2024-09-21T14:01:41.200113+0000","last_became_peered":"2024-09-21T14:01:41.200113+0000","last_unstale":"2024-09-21T14:05:42.378978+0000","last_undegraded":"2024-09-21T14:05:42.378978+0000","last_fullsized":"2024-09-21T14:05:42.378978+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:20.368325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:05:20.368325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:41.310306+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"75'79","reported_seq":239,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.500575+0000","last_change":"2024-09-21T14:05:43.226598+0000","last_active":"2024-09-21T14:05:45.500575+0000","last_peered":"2024-09-21T14:05:45.500575+0000","last_clean":"2024-09-21T14:05:45.500575+0000","last_became_active":"2024-09-21T14:05:41.344418+0000","last_became_peered":"2024-09-21T14:05:41.344418+0000","last_unstale":"2024-09-21T14:05:45.500575+0000","last_undegraded":"2024-09-21T14:05:45.500575+0000","last_fullsized":"2024-09-21T14:05:45.500575+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":5,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:05:43.226395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:05:43.226395+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:46.272654+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"290'20","reported_seq":767,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.524939+0000","last_change":"2024-09-21T14:04:04.192280+0000","last_active":"2024-09-21T14:05:45.524939+0000","last_peered":"2024-09-21T14:05:45.524939+0000","last_clean":"2024-09-21T14:05:45.524939+0000","last_became_active":"2024-09-21T13:59:32.208332+0000","last_became_peered":"2024-09-21T13:59:32.208332+0000","last_unstale":"2024-09-21T14:05:45.524939+0000","last_undegraded":"2024-09-21T14:05:45.524939+0000","last_fullsized":"2024-09-21T14:05:45.524939+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'20","last_scrub_stamp":"2024-09-21T14:04:04.192193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:04:04.192193+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":759,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.504746+0000","last_change":"2024-09-21T14:05:38.445596+0000","last_active":"2024-09-21T14:05:45.504746+0000","last_peered":"2024-09-21T14:05:45.504746+0000","last_clean":"2024-09-21T14:05:45.504746+0000","last_became_active":"2024-09-21T13:59:32.209804+0000","last_became_peered":"2024-09-21T13:59:32.209804+0000","last_unstale":"2024-09-21T14:05:45.504746+0000","last_undegraded":"2024-09-21T14:05:45.504746+0000","last_fullsized":"2024-09-21T14:05:45.504746+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:38.445529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:05:38.445529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:05.619808+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"346'10","reported_seq":909,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.501401+0000","last_change":"2024-09-21T14:05:21.397197+0000","last_active":"2024-09-21T14:05:45.501401+0000","last_peered":"2024-09-21T14:05:45.501401+0000","last_clean":"2024-09-21T14:05:45.501401+0000","last_became_active":"2024-09-21T14:01:41.206303+0000","last_became_peered":"2024-09-21T14:01:41.206303+0000","last_unstale":"2024-09-21T14:05:45.501401+0000","last_undegraded":"2024-09-21T14:05:45.501401+0000","last_fullsized":"2024-09-21T14:05:45.501401+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'10","last_scrub_stamp":"2024-09-21T14:05:21.397143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:05:21.397143+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:36.172088+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":663,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.506034+0000","last_change":"2024-09-21T14:04:21.202582+0000","last_active":"2024-09-21T14:05:45.506034+0000","last_peered":"2024-09-21T14:05:45.506034+0000","last_clean":"2024-09-21T14:05:45.506034+0000","last_became_active":"2024-09-21T14:04:21.201277+0000","last_became_peered":"2024-09-21T14:04:21.201277+0000","last_unstale":"2024-09-21T14:05:45.506034+0000","last_undegraded":"2024-09-21T14:05:45.506034+0000","last_fullsized":"2024-09-21T14:05:45.506034+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:27.961544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:27.961544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"290'19","reported_seq":880,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:42.378674+0000","last_change":"2024-09-21T14:04:05.139089+0000","last_active":"2024-09-21T14:05:42.378674+0000","last_peered":"2024-09-21T14:05:42.378674+0000","last_clean":"2024-09-21T14:05:42.378674+0000","last_became_active":"2024-09-21T13:58:17.771497+0000","last_became_peered":"2024-09-21T13:58:17.771497+0000","last_unstale":"2024-09-21T14:05:42.378674+0000","last_undegraded":"2024-09-21T14:05:42.378674+0000","last_fullsized":"2024-09-21T14:05:42.378674+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'19","last_scrub_stamp":"2024-09-21T14:04:05.138989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:04:05.138989+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":393,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.505121+0000","last_change":"2024-09-21T14:02:57.807136+0000","last_active":"2024-09-21T14:05:45.505121+0000","last_peered":"2024-09-21T14:05:45.505121+0000","last_clean":"2024-09-21T14:05:45.505121+0000","last_became_active":"2024-09-21T13:59:32.201568+0000","last_became_peered":"2024-09-21T13:59:32.201568+0000","last_unstale":"2024-09-21T14:05:45.505121+0000","last_undegraded":"2024-09-21T14:05:45.505121+0000","last_fullsized":"2024-09-21T14:05:45.505121+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:57.807059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:02:57.807059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":830,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:42.387935+0000","last_change":"2024-09-21T14:05:19.308488+0000","last_active":"2024-09-21T14:05:42.387935+0000","last_peered":"2024-09-21T14:05:42.387935+0000","last_clean":"2024-09-21T14:05:42.387935+0000","last_became_active":"2024-09-21T14:01:41.185293+0000","last_became_peered":"2024-09-21T14:01:41.185293+0000","last_unstale":"2024-09-21T14:05:42.387935+0000","last_undegraded":"2024-09-21T14:05:42.387935+0000","last_fullsized":"2024-09-21T14:05:42.387935+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:19.308311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:05:19.308311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:42.164269+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":731,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.525204+0000","last_change":"2024-09-21T14:04:21.209188+0000","last_active":"2024-09-21T14:05:45.525204+0000","last_peered":"2024-09-21T14:05:45.525204+0000","last_clean":"2024-09-21T14:05:45.525204+0000","last_became_active":"2024-09-21T14:04:21.208329+0000","last_became_peered":"2024-09-21T14:04:21.208329+0000","last_unstale":"2024-09-21T14:05:45.525204+0000","last_undegraded":"2024-09-21T14:05:45.525204+0000","last_fullsized":"2024-09-21T14:05:45.525204+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:57.225602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:02:57.225602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"290'15","reported_seq":882,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:42.388092+0000","last_change":"2024-09-21T14:04:12.139406+0000","last_active":"2024-09-21T14:05:42.388092+0000","last_peered":"2024-09-21T14:05:42.388092+0000","last_clean":"2024-09-21T14:05:42.388092+0000","last_became_active":"2024-09-21T13:58:17.764120+0000","last_became_peered":"2024-09-21T13:58:17.764120+0000","last_unstale":"2024-09-21T14:05:42.388092+0000","last_undegraded":"2024-09-21T14:05:42.388092+0000","last_fullsized":"2024-09-21T14:05:42.388092+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'15","last_scrub_stamp":"2024-09-21T14:04:12.139190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:04:12.139190+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":763,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.505285+0000","last_change":"2024-09-21T14:04:07.038058+0000","last_active":"2024-09-21T14:05:45.505285+0000","last_peered":"2024-09-21T14:05:45.505285+0000","last_clean":"2024-09-21T14:05:45.505285+0000","last_became_active":"2024-09-21T13:59:32.221854+0000","last_became_peered":"2024-09-21T13:59:32.221854+0000","last_unstale":"2024-09-21T14:05:45.505285+0000","last_undegraded":"2024-09-21T14:05:45.505285+0000","last_fullsized":"2024-09-21T14:05:45.505285+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:07.037989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:07.037989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":699,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.525627+0000","last_change":"2024-09-21T14:03:32.262147+0000","last_active":"2024-09-21T14:05:45.525627+0000","last_peered":"2024-09-21T14:05:45.525627+0000","last_clean":"2024-09-21T14:05:45.525627+0000","last_became_active":"2024-09-21T13:57:12.703441+0000","last_became_peered":"2024-09-21T13:57:12.703441+0000","last_unstale":"2024-09-21T14:05:45.525627+0000","last_undegraded":"2024-09-21T14:05:45.525627+0000","last_fullsized":"2024-09-21T14:05:45.525627+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:32.262067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:32.262067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":469,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:42.396005+0000","last_change":"2024-09-21T14:04:56.588351+0000","last_active":"2024-09-21T14:05:42.396005+0000","last_peered":"2024-09-21T14:05:42.396005+0000","last_clean":"2024-09-21T14:05:42.396005+0000","last_became_active":"2024-09-21T14:04:56.587503+0000","last_became_peered":"2024-09-21T14:04:56.587503+0000","last_unstale":"2024-09-21T14:05:42.396005+0000","last_undegraded":"2024-09-21T14:05:42.396005+0000","last_fullsized":"2024-09-21T14:05:42.396005+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:29.345427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:29.345427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:05.186916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"75'79","reported_seq":24,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.504580+0000","last_change":"2024-09-21T14:05:44.794897+0000","last_active":"2024-09-21T14:05:45.504580+0000","last_peered":"2024-09-21T14:05:45.504580+0000","last_clean":"2024-09-21T14:05:45.504580+0000","last_became_active":"2024-09-21T14:05:41.358002+0000","last_became_peered":"2024-09-21T14:05:41.358002+0000","last_unstale":"2024-09-21T14:05:45.504580+0000","last_undegraded":"2024-09-21T14:05:45.504580+0000","last_fullsized":"2024-09-21T14:05:45.504580+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":5,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:05:44.794826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:05:44.794826+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:05.143030+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"290'19","reported_seq":835,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.525446+0000","last_change":"2024-09-21T13:59:32.277742+0000","last_active":"2024-09-21T14:05:45.525446+0000","last_peered":"2024-09-21T14:05:45.525446+0000","last_clean":"2024-09-21T14:05:45.525446+0000","last_became_active":"2024-09-21T13:59:32.231567+0000","last_became_peered":"2024-09-21T13:59:32.231567+0000","last_unstale":"2024-09-21T14:05:45.525446+0000","last_undegraded":"2024-09-21T14:05:45.525446+0000","last_fullsized":"2024-09-21T14:05:45.525446+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-21T13:54:22.978267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:54:22.978267+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":731,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.501475+0000","last_change":"2024-09-21T14:05:13.355232+0000","last_active":"2024-09-21T14:05:45.501475+0000","last_peered":"2024-09-21T14:05:45.501475+0000","last_clean":"2024-09-21T14:05:45.501475+0000","last_became_active":"2024-09-21T14:01:41.190760+0000","last_became_peered":"2024-09-21T14:01:41.190760+0000","last_unstale":"2024-09-21T14:05:45.501475+0000","last_undegraded":"2024-09-21T14:05:45.501475+0000","last_fullsized":"2024-09-21T14:05:45.501475+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:13.355183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:05:13.355183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:37.299500+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":769,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.504687+0000","last_change":"2024-09-21T14:04:41.091577+0000","last_active":"2024-09-21T14:05:45.504687+0000","last_peered":"2024-09-21T14:05:45.504687+0000","last_clean":"2024-09-21T14:05:45.504687+0000","last_became_active":"2024-09-21T13:59:32.214004+0000","last_became_peered":"2024-09-21T13:59:32.214004+0000","last_unstale":"2024-09-21T14:05:45.504687+0000","last_undegraded":"2024-09-21T14:05:45.504687+0000","last_fullsized":"2024-09-21T14:05:45.504687+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:41.091525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:04:41.091525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:06.917055+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":764,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.525789+0000","last_change":"2024-09-21T14:04:21.214685+0000","last_active":"2024-09-21T14:05:45.525789+0000","last_peered":"2024-09-21T14:05:45.525789+0000","last_clean":"2024-09-21T14:05:45.525789+0000","last_became_active":"2024-09-21T14:04:21.213307+0000","last_became_peered":"2024-09-21T14:04:21.213307+0000","last_unstale":"2024-09-21T14:05:45.525789+0000","last_undegraded":"2024-09-21T14:05:45.525789+0000","last_fullsized":"2024-09-21T14:05:45.525789+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:28.124296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:28.124296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"290'8","reported_seq":781,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.524887+0000","last_change":"2024-09-21T13:59:26.034457+0000","last_active":"2024-09-21T14:05:45.524887+0000","last_peered":"2024-09-21T14:05:45.524887+0000","last_clean":"2024-09-21T14:05:45.524887+0000","last_became_active":"2024-09-21T13:58:17.775079+0000","last_became_peered":"2024-09-21T13:58:17.775079+0000","last_unstale":"2024-09-21T14:05:45.524887+0000","last_undegraded":"2024-09-21T14:05:45.524887+0000","last_fullsized":"2024-09-21T14:05:45.524887+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'8","last_scrub_stamp":"2024-09-21T13:59:26.034372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:26.034372+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":548,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:44.390926+0000","last_change":"2024-09-21T14:05:44.390926+0000","last_active":"2024-09-21T14:05:44.390926+0000","last_peered":"2024-09-21T14:05:44.390926+0000","last_clean":"2024-09-21T14:05:44.390926+0000","last_became_active":"2024-09-21T14:04:56.587555+0000","last_became_peered":"2024-09-21T14:04:56.587555+0000","last_unstale":"2024-09-21T14:05:44.390926+0000","last_undegraded":"2024-09-21T14:05:44.390926+0000","last_fullsized":"2024-09-21T14:05:44.390926+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:44.390852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:05:44.390852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:51.855429+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":397,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:42.382096+0000","last_change":"2024-09-21T14:04:41.932290+0000","last_active":"2024-09-21T14:05:42.382096+0000","last_peered":"2024-09-21T14:05:42.382096+0000","last_clean":"2024-09-21T14:05:42.382096+0000","last_became_active":"2024-09-21T14:04:21.204954+0000","last_became_peered":"2024-09-21T14:04:21.204954+0000","last_unstale":"2024-09-21T14:05:42.382096+0000","last_undegraded":"2024-09-21T14:05:42.382096+0000","last_fullsized":"2024-09-21T14:05:42.382096+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:41.932233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:04:41.932233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:03.486134+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":702,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.504410+0000","last_change":"2024-09-21T14:04:15.135536+0000","last_active":"2024-09-21T14:05:45.504410+0000","last_peered":"2024-09-21T14:05:45.504410+0000","last_clean":"2024-09-21T14:05:45.504410+0000","last_became_active":"2024-09-21T13:59:32.210134+0000","last_became_peered":"2024-09-21T13:59:32.210134+0000","last_unstale":"2024-09-21T14:05:45.504410+0000","last_undegraded":"2024-09-21T14:05:45.504410+0000","last_fullsized":"2024-09-21T14:05:45.504410+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:15.135463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:04:15.135463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":806,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:42.385068+0000","last_change":"2024-09-21T14:01:41.187792+0000","last_active":"2024-09-21T14:05:42.385068+0000","last_peered":"2024-09-21T14:05:42.385068+0000","last_clean":"2024-09-21T14:05:42.385068+0000","last_became_active":"2024-09-21T14:01:41.187438+0000","last_became_peered":"2024-09-21T14:01:41.187438+0000","last_unstale":"2024-09-21T14:05:42.385068+0000","last_undegraded":"2024-09-21T14:05:42.385068+0000","last_fullsized":"2024-09-21T14:05:42.385068+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:17.706325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:17.706325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:31.441904+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":752,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-21T14:05:45.524983+0000","last_change":"2024-09-21T14:04:33.252571+0000","last_active":"2024-09-21T14:05:45.524983+0000","last_peered":"2024-09-21T14:05:45.524983+0000","last_clean":"2024-09-21T14:05:45.524983+0000","last_became_active":"2024-09-21T14:04:21.208121+0000","last_became_peered":"2024-09-21T14:04:21.208121+0000","last_unstale":"2024-09-21T14:05:45.524983+0000","last_undegraded":"2024-09-21T14:05:45.524983+0000","last_fullsized":"2024-09-21T14:05:45.524983+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:33.252481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:33.252481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:48.348987+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"290'19","reported_seq":414,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:42.383246+0000","last_change":"2024-09-21T14:03:31.762681+0000","last_active":"2024-09-21T14:05:42.383246+0000","last_peered":"2024-09-21T14:05:42.383246+0000","last_clean":"2024-09-21T14:05:42.383246+0000","last_became_active":"2024-09-21T14:01:41.194606+0000","last_became_peered":"2024-09-21T14:01:41.194606+0000","last_unstale":"2024-09-21T14:05:42.383246+0000","last_undegraded":"2024-09-21T14:05:42.383246+0000","last_fullsized":"2024-09-21T14:05:42.383246+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'19","last_scrub_stamp":"2024-09-21T14:03:31.762605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:03:31.762605+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":424,"reported_epoch":440,"state":"active+clean","last_fresh":"2024-09-21T14:05:42.395798+0000","last_change":"2024-09-21T14:04:56.591137+0000","last_active":"2024-09-21T14:05:42.395798+0000","last_peered":"2024-09-21T14:05:42.395798+0000","last_clean":"2024-09-21T14:05:42.395798+0000","last_became_active":"2024-09-21T14:04:56.590829+0000","last_became_peered":"2024-09-21T14:04:56.590829+0000","last_unstale":"2024-09-21T14:05:42.395798+0000","last_undegraded":"2024-09-21T14:05:42.395798+0000","last_fullsized":"2024-09-21T14:05:42.395798+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:51.121361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:04:51.121361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:05:59.958121+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":3797352472,"num_objects":57944,"num_object_clones":0,"num_object_copies":173832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57944,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":197206,"num_write_kb":8164737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23621,"num_bytes_recovered":1547960344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12368953344,"data_stored":12368937056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77679,"ondisk_log_size":77679,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102,"ondisk_log_size":102,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1107,"num_read_kb":864,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":197,"num_bytes_recovered":2794,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":687,"ondisk_log_size":687,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":380,"num_read_kb":375,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":479254,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":20,"num_read_kb":39,"num_write":20,"num_write_kb":296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":435019,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1659,"ondisk_log_size":1659,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":298,"seq":1279900254284,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1570012,"kb_used_data":1149516,"kb_used_omap":4,"kb_used_meta":420475,"kb_avail":92175140,"statfs":{"total":95995035648,"available":94387343360,"internally_reserved":0,"allocated":1177104384,"data_stored":1176757760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":430567080},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463123,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2116292,"kb_used_data":1683764,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":91628860,"statfs":{"total":95995035648,"available":93827952640,"internally_reserved":0,"allocated":1724174336,"data_stored":1723851007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":442759622},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":345,"seq":1481763717170,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1339008,"kb_used_data":1158004,"kb_used_omap":4,"kb_used_meta":180987,"kb_avail":92406144,"statfs":{"total":95995035648,"available":94623891456,"internally_reserved":0,"allocated":1185796096,"data_stored":1185474157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":185331363},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920992,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2481100,"kb_used_data":1940412,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":91264052,"statfs":{"total":95995035648,"available":93454389248,"internally_reserved":0,"allocated":1986981888,"data_stored":1986650453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":553646533},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":240,"seq":1030792151144,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1843044,"kb_used_data":1385940,"kb_used_omap":4,"kb_used_meta":457083,"kb_avail":91902108,"statfs":{"total":95995035648,"available":94107758592,"internally_reserved":0,"allocated":1419202560,"data_stored":1418848299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4443,"internal_metadata":468053669},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":130,"seq":558345748646,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2243476,"kb_used_data":1525444,"kb_used_omap":4,"kb_used_meta":718011,"kb_avail":91501676,"statfs":{"total":95995035648,"available":93697716224,"internally_reserved":0,"allocated":1562054656,"data_stored":1561724588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4441,"internal_metadata":735243943},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575090,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1618376,"kb_used_data":933300,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":92126776,"statfs":{"total":95995035648,"available":94337818624,"internally_reserved":0,"allocated":955699200,"data_stored":955311719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":701495749},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738616,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3133296,"kb_used_data":2452252,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":90611856,"statfs":{"total":95995035648,"available":92786540544,"internally_reserved":0,"allocated":2511106048,"data_stored":2510806699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":697235909},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":2448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2503610368,"data_stored":2503606296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":948436992,"data_stored":948436992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1550712832,"data_stored":1550712832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1383927808,"data_stored":1383923736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1967525888,"data_stored":1967521816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1166540800,"data_stored":1166540800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1694502912,"data_stored":1694498840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1153695744,"data_stored":1153695744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T14:05:51.711 INFO:tasks.thrashosds.thrasher:Setting 10.b to [4, 6, 7, 5, 0, 1] 2024-09-21T14:05:51.711 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '10.b', '4', '6', '7', '5', '0', '1'] 2024-09-21T14:05:51.711 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd pg-upmap-items 10.b 4 6 7 5 0 1 2024-09-21T14:05:51.737 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:05:51.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:51.806 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:51.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:05:51.907 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:05:51.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:51 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:05:51.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:51 smithi187 ceph-mon[33032]: from='client.16368 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:05:51.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:51 smithi187 ceph-mon[33032]: pgmap v861: 168 pgs: 168 active+clean; 3.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.36k op/s; 30 KiB/s, 0 objects/s recovering 2024-09-21T14:05:51.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:51 smithi187 ceph-mon[33032]: 3.1b scrub starts 2024-09-21T14:05:51.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:51 smithi187 ceph-mon[33032]: 3.1b scrub ok 2024-09-21T14:05:51.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:51.955 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:51.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:05:52.055 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:52.077 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:52.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:52.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:52.178 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:05:52.213 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:52.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:52.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:05:52.315 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:05:52.352 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:52.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:52.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:05:52.453 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:52.485 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:52.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:52.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:52.587 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:05:52.621 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:52.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:52.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:05:52.722 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:05:52.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:52.759 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:52.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:05:52.837 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:52 smithi187 ceph-mon[33032]: 5.8 scrub starts 2024-09-21T14:05:52.837 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:52 smithi187 ceph-mon[33032]: 5.8 scrub ok 2024-09-21T14:05:52.860 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:52.892 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:52.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:52.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:52.924 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:52 smithi049 ceph-mon[40489]: 5.8 scrub starts 2024-09-21T14:05:52.924 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:52 smithi049 ceph-mon[40489]: 5.8 scrub ok 2024-09-21T14:05:52.925 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:52 smithi049 ceph-mon[30080]: 5.8 scrub starts 2024-09-21T14:05:52.925 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:52 smithi049 ceph-mon[30080]: 5.8 scrub ok 2024-09-21T14:05:52.993 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:05:53.034 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:53.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:53.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:05:53.135 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:05:53.170 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:53.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:53.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:05:53.274 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:05:53.309 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:53.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:53.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:05:53.411 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:05:53.447 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:53.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:53.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:05:53.549 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:53.583 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:53.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:53.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:53.685 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:05:53.718 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:53.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:53.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:05:53.820 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:05:53.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:53.859 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:53.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:05:53.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:53 smithi187 ceph-mon[33032]: pgmap v862: 168 pgs: 168 active+clean; 3.2 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.24k op/s 2024-09-21T14:05:53.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:53 smithi187 ceph-mon[33032]: 5.14 scrub starts 2024-09-21T14:05:53.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:53 smithi187 ceph-mon[33032]: 5.14 scrub ok 2024-09-21T14:05:53.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:53 smithi187 ceph-mon[33032]: 10.4 scrub starts 2024-09-21T14:05:53.960 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:53.996 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:53.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:53.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:54.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:53 smithi049 ceph-mon[40489]: pgmap v862: 168 pgs: 168 active+clean; 3.2 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.24k op/s 2024-09-21T14:05:54.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:53 smithi049 ceph-mon[40489]: 5.14 scrub starts 2024-09-21T14:05:54.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:53 smithi049 ceph-mon[40489]: 5.14 scrub ok 2024-09-21T14:05:54.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:53 smithi049 ceph-mon[40489]: 10.4 scrub starts 2024-09-21T14:05:54.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:53 smithi049 ceph-mon[30080]: pgmap v862: 168 pgs: 168 active+clean; 3.2 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.24k op/s 2024-09-21T14:05:54.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:53 smithi049 ceph-mon[30080]: 5.14 scrub starts 2024-09-21T14:05:54.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:53 smithi049 ceph-mon[30080]: 5.14 scrub ok 2024-09-21T14:05:54.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:53 smithi049 ceph-mon[30080]: 10.4 scrub starts 2024-09-21T14:05:54.097 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:05:54.119 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:54.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:54.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:05:54.221 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:05:54.242 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:54.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:54.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:05:54.345 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:05:54.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:54.664 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:54.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:05:54.766 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:54.800 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:54.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:54.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:54.901 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:05:54.937 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:54.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:54.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:05:54.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:54 smithi187 ceph-mon[33032]: 6.14 scrub starts 2024-09-21T14:05:54.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:54 smithi187 ceph-mon[33032]: 6.14 scrub ok 2024-09-21T14:05:54.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:54 smithi187 ceph-mon[33032]: 5.3 scrub starts 2024-09-21T14:05:54.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:54 smithi187 ceph-mon[33032]: 5.3 scrub ok 2024-09-21T14:05:54.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:54 smithi187 ceph-mon[33032]: 3.10 scrub starts 2024-09-21T14:05:54.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:54 smithi187 ceph-mon[33032]: 3.10 scrub ok 2024-09-21T14:05:55.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:54 smithi049 ceph-mon[30080]: 6.14 scrub starts 2024-09-21T14:05:55.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:54 smithi049 ceph-mon[30080]: 6.14 scrub ok 2024-09-21T14:05:55.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:54 smithi049 ceph-mon[30080]: 5.3 scrub starts 2024-09-21T14:05:55.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:54 smithi049 ceph-mon[30080]: 5.3 scrub ok 2024-09-21T14:05:55.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:54 smithi049 ceph-mon[30080]: 3.10 scrub starts 2024-09-21T14:05:55.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:54 smithi049 ceph-mon[30080]: 3.10 scrub ok 2024-09-21T14:05:55.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:54 smithi049 ceph-mon[40489]: 6.14 scrub starts 2024-09-21T14:05:55.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:54 smithi049 ceph-mon[40489]: 6.14 scrub ok 2024-09-21T14:05:55.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:54 smithi049 ceph-mon[40489]: 5.3 scrub starts 2024-09-21T14:05:55.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:54 smithi049 ceph-mon[40489]: 5.3 scrub ok 2024-09-21T14:05:55.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:54 smithi049 ceph-mon[40489]: 3.10 scrub starts 2024-09-21T14:05:55.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:54 smithi049 ceph-mon[40489]: 3.10 scrub ok 2024-09-21T14:05:55.038 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:55.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:55.074 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:55.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:55.175 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:05:55.207 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:55.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:55.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:05:55.309 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:55.345 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:55.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:55.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:55.447 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:55.484 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:55.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:55.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:55.586 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:55.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:55.623 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:55.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:55.723 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:05:55.758 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:55.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:55.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:05:55.860 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:05:55.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:55.900 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:55.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:05:55.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:55 smithi187 ceph-mon[33032]: pgmap v863: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 2.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.23k op/s 2024-09-21T14:05:55.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:55 smithi187 ceph-mon[33032]: 4.11 scrub starts 2024-09-21T14:05:55.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:55 smithi187 ceph-mon[33032]: 4.11 scrub ok 2024-09-21T14:05:55.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:55 smithi187 ceph-mon[33032]: 1.d scrub starts 2024-09-21T14:05:55.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:55 smithi187 ceph-mon[33032]: 1.d scrub ok 2024-09-21T14:05:55.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:55 smithi187 ceph-mon[33032]: 10.4 scrub ok 2024-09-21T14:05:55.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:55 smithi187 ceph-mon[33032]: 10.a scrub starts 2024-09-21T14:05:55.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:55 smithi187 ceph-mon[33032]: 5.a scrub starts 2024-09-21T14:05:55.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:55 smithi187 ceph-mon[33032]: 5.a scrub ok 2024-09-21T14:05:55.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:55 smithi187 ceph-mon[33032]: 4.16 scrub starts 2024-09-21T14:05:56.001 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:56.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:55 smithi049 ceph-mon[40489]: pgmap v863: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 2.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.23k op/s 2024-09-21T14:05:56.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:55 smithi049 ceph-mon[40489]: 4.11 scrub starts 2024-09-21T14:05:56.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:55 smithi049 ceph-mon[40489]: 4.11 scrub ok 2024-09-21T14:05:56.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:55 smithi049 ceph-mon[40489]: 1.d scrub starts 2024-09-21T14:05:56.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:55 smithi049 ceph-mon[40489]: 1.d scrub ok 2024-09-21T14:05:56.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:55 smithi049 ceph-mon[40489]: 10.4 scrub ok 2024-09-21T14:05:56.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:55 smithi049 ceph-mon[40489]: 10.a scrub starts 2024-09-21T14:05:56.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:55 smithi049 ceph-mon[40489]: 5.a scrub starts 2024-09-21T14:05:56.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:55 smithi049 ceph-mon[40489]: 5.a scrub ok 2024-09-21T14:05:56.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:55 smithi049 ceph-mon[40489]: 4.16 scrub starts 2024-09-21T14:05:56.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:55 smithi049 ceph-mon[30080]: pgmap v863: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 2.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.23k op/s 2024-09-21T14:05:56.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:55 smithi049 ceph-mon[30080]: 4.11 scrub starts 2024-09-21T14:05:56.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:55 smithi049 ceph-mon[30080]: 4.11 scrub ok 2024-09-21T14:05:56.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:55 smithi049 ceph-mon[30080]: 1.d scrub starts 2024-09-21T14:05:56.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:55 smithi049 ceph-mon[30080]: 1.d scrub ok 2024-09-21T14:05:56.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:55 smithi049 ceph-mon[30080]: 10.4 scrub ok 2024-09-21T14:05:56.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:55 smithi049 ceph-mon[30080]: 10.a scrub starts 2024-09-21T14:05:56.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:55 smithi049 ceph-mon[30080]: 5.a scrub starts 2024-09-21T14:05:56.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:55 smithi049 ceph-mon[30080]: 5.a scrub ok 2024-09-21T14:05:56.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:55 smithi049 ceph-mon[30080]: 4.16 scrub starts 2024-09-21T14:05:56.037 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:56.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:56.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:56.139 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:05:56.178 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:56.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:56.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:05:56.280 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:05:56.322 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:56.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:56.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:05:56.424 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:56.457 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:56.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:56.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:56.559 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:56.593 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:56.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:56.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:56.695 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:56.718 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:56.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:56.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:56.820 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:05:56.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:56.860 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:56.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:05:56.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:56 smithi187 ceph-mon[33032]: 4.16 scrub ok 2024-09-21T14:05:56.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:56 smithi187 ceph-mon[33032]: 10.a scrub ok 2024-09-21T14:05:56.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:56 smithi187 ceph-mon[33032]: 4.1f scrub starts 2024-09-21T14:05:56.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:56 smithi187 ceph-mon[33032]: 4.1f scrub ok 2024-09-21T14:05:56.961 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:56.995 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:56.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:56.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:57.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:56 smithi049 ceph-mon[40489]: 4.16 scrub ok 2024-09-21T14:05:57.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:56 smithi049 ceph-mon[40489]: 10.a scrub ok 2024-09-21T14:05:57.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:56 smithi049 ceph-mon[40489]: 4.1f scrub starts 2024-09-21T14:05:57.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:56 smithi049 ceph-mon[40489]: 4.1f scrub ok 2024-09-21T14:05:57.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:56 smithi049 ceph-mon[30080]: 4.16 scrub ok 2024-09-21T14:05:57.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:56 smithi049 ceph-mon[30080]: 10.a scrub ok 2024-09-21T14:05:57.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:56 smithi049 ceph-mon[30080]: 4.1f scrub starts 2024-09-21T14:05:57.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:56 smithi049 ceph-mon[30080]: 4.1f scrub ok 2024-09-21T14:05:57.097 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:57.128 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:57.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:57.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:57.230 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:05:57.262 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:57.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:57.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:05:57.364 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:05:57.397 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:57.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:57.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:05:57.498 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:05:57.537 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:57.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:57.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:05:57.639 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:05:57.669 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:57.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:57.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:05:57.753 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T14:05:57.772 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:05:57.798 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:57.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:57.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:05:57.811 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:05:57.813 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:05:57.900 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:05:57.931 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:57.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:57.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:05:57.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:57 smithi187 ceph-mon[33032]: 4.1c scrub starts 2024-09-21T14:05:57.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:57 smithi187 ceph-mon[33032]: 4.1c scrub ok 2024-09-21T14:05:57.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:57 smithi187 ceph-mon[33032]: pgmap v864: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 2.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.24k op/s 2024-09-21T14:05:57.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:57 smithi187 ceph-mon[33032]: 4.1e scrub starts 2024-09-21T14:05:57.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:57 smithi187 ceph-mon[33032]: 4.1e scrub ok 2024-09-21T14:05:57.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:57 smithi187 ceph-mon[33032]: 3.1e scrub starts 2024-09-21T14:05:57.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:57 smithi187 ceph-mon[33032]: 3.1e scrub ok 2024-09-21T14:05:57.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:57 smithi187 ceph-mon[33032]: 4.1 scrub starts 2024-09-21T14:05:57.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:57 smithi187 ceph-mon[33032]: 4.1 scrub ok 2024-09-21T14:05:57.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:57 smithi049 ceph-mon[30080]: 4.1c scrub starts 2024-09-21T14:05:57.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:57 smithi049 ceph-mon[30080]: 4.1c scrub ok 2024-09-21T14:05:57.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:57 smithi049 ceph-mon[30080]: pgmap v864: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 2.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.24k op/s 2024-09-21T14:05:57.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:57 smithi049 ceph-mon[30080]: 4.1e scrub starts 2024-09-21T14:05:57.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:57 smithi049 ceph-mon[30080]: 4.1e scrub ok 2024-09-21T14:05:57.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:57 smithi049 ceph-mon[30080]: 3.1e scrub starts 2024-09-21T14:05:57.958 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:57 smithi049 ceph-mon[30080]: 3.1e scrub ok 2024-09-21T14:05:57.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:57 smithi049 ceph-mon[30080]: 4.1 scrub starts 2024-09-21T14:05:57.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:57 smithi049 ceph-mon[30080]: 4.1 scrub ok 2024-09-21T14:05:57.959 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:57 smithi049 ceph-mon[40489]: 4.1c scrub starts 2024-09-21T14:05:57.959 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:57 smithi049 ceph-mon[40489]: 4.1c scrub ok 2024-09-21T14:05:57.959 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:57 smithi049 ceph-mon[40489]: pgmap v864: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 2.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.24k op/s 2024-09-21T14:05:57.959 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:57 smithi049 ceph-mon[40489]: 4.1e scrub starts 2024-09-21T14:05:57.959 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:57 smithi049 ceph-mon[40489]: 4.1e scrub ok 2024-09-21T14:05:57.959 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:57 smithi049 ceph-mon[40489]: 3.1e scrub starts 2024-09-21T14:05:57.959 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:57 smithi049 ceph-mon[40489]: 3.1e scrub ok 2024-09-21T14:05:57.959 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:57 smithi049 ceph-mon[40489]: 4.1 scrub starts 2024-09-21T14:05:57.959 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:57 smithi049 ceph-mon[40489]: 4.1 scrub ok 2024-09-21T14:05:58.033 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:05:58.064 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:58.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:58.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:05:58.166 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:05:58.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:58.198 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:58.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:05:58.298 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:05:58.348 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:58.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:58.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:05:58.449 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:05:58.490 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:58.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:58.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:05:58.592 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:05:58.629 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:58.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:58.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:05:58.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:58 smithi049 ceph-mon[30080]: 1.e scrub starts 2024-09-21T14:05:58.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:58 smithi049 ceph-mon[30080]: 1.e scrub ok 2024-09-21T14:05:58.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:58 smithi049 ceph-mon[30080]: 6.4 scrub starts 2024-09-21T14:05:58.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:58 smithi049 ceph-mon[30080]: 6.4 scrub ok 2024-09-21T14:05:58.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:58 smithi049 ceph-mon[30080]: 10.b scrub starts 2024-09-21T14:05:58.706 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:58 smithi049 ceph-mon[40489]: 1.e scrub starts 2024-09-21T14:05:58.706 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:58 smithi049 ceph-mon[40489]: 1.e scrub ok 2024-09-21T14:05:58.706 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:58 smithi049 ceph-mon[40489]: 6.4 scrub starts 2024-09-21T14:05:58.706 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:58 smithi049 ceph-mon[40489]: 6.4 scrub ok 2024-09-21T14:05:58.706 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:58 smithi049 ceph-mon[40489]: 10.b scrub starts 2024-09-21T14:05:58.730 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:58.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:58.768 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:58.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:58.869 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:05:58.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:58.917 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:58.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:05:58.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:58 smithi187 ceph-mon[33032]: 1.e scrub starts 2024-09-21T14:05:58.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:58 smithi187 ceph-mon[33032]: 1.e scrub ok 2024-09-21T14:05:58.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:58 smithi187 ceph-mon[33032]: 6.4 scrub starts 2024-09-21T14:05:58.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:58 smithi187 ceph-mon[33032]: 6.4 scrub ok 2024-09-21T14:05:58.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:58 smithi187 ceph-mon[33032]: 10.b scrub starts 2024-09-21T14:05:59.019 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:05:59.060 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:59.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:59.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:05:59.133 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:05:59.153 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T14:05:59.162 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:59.186 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:59.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:59.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:59.288 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:05:59.318 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:59.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:59.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:05:59.419 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:05:59.447 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:59.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:59.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:05:59.550 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:05:59.589 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:59.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:59.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:05:59.605 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-09-21T14:05:59.610 INFO:teuthology.orchestra.run.smithi049.stderr:set 10.b pg_upmap_items mapping to [4->6,7->5,0->1] 2024-09-21T14:05:59.690 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:05:59.725 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:05:59.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:59.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:05:59.826 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:05:59.843 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:59 smithi049 ceph-mon[30080]: pgmap v865: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 2.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s 2024-09-21T14:05:59.843 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:59 smithi049 ceph-mon[30080]: 10.6 scrub starts 2024-09-21T14:05:59.843 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:59 smithi049 ceph-mon[30080]: 2.0 scrub starts 2024-09-21T14:05:59.843 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:59 smithi049 ceph-mon[30080]: 2.0 scrub ok 2024-09-21T14:05:59.843 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:59 smithi049 ceph-mon[30080]: 10.b scrub ok 2024-09-21T14:05:59.843 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:59 smithi049 ceph-mon[30080]: 10.6 scrub ok 2024-09-21T14:05:59.843 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:59 smithi049 ceph-mon[30080]: 3.9 scrub starts 2024-09-21T14:05:59.843 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:59 smithi049 ceph-mon[30080]: 3.9 scrub ok 2024-09-21T14:05:59.843 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:59 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/136009295' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "10.b", "id": [4, 6, 7, 5, 0, 1]} : dispatch 2024-09-21T14:05:59.843 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:59 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "10.b", "id": [4, 6, 7, 5, 0, 1]} : dispatch 2024-09-21T14:05:59.844 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:59 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:05:59.844 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:59 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:05:59.844 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:05:59 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1015995849' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:05:59.844 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:59 smithi049 ceph-mon[40489]: pgmap v865: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 2.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s 2024-09-21T14:05:59.844 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:59 smithi049 ceph-mon[40489]: 10.6 scrub starts 2024-09-21T14:05:59.844 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:59 smithi049 ceph-mon[40489]: 2.0 scrub starts 2024-09-21T14:05:59.844 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:59 smithi049 ceph-mon[40489]: 2.0 scrub ok 2024-09-21T14:05:59.844 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:59 smithi049 ceph-mon[40489]: 10.b scrub ok 2024-09-21T14:05:59.844 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:59 smithi049 ceph-mon[40489]: 10.6 scrub ok 2024-09-21T14:05:59.844 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:59 smithi049 ceph-mon[40489]: 3.9 scrub starts 2024-09-21T14:05:59.844 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:59 smithi049 ceph-mon[40489]: 3.9 scrub ok 2024-09-21T14:05:59.845 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:59 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/136009295' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "10.b", "id": [4, 6, 7, 5, 0, 1]} : dispatch 2024-09-21T14:05:59.845 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:59 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "10.b", "id": [4, 6, 7, 5, 0, 1]} : dispatch 2024-09-21T14:05:59.845 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:59 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:05:59.845 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:59 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:05:59.845 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:05:59 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1015995849' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:05:59.872 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:05:59.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:05:59.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:05:59.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:59 smithi187 ceph-mon[33032]: pgmap v865: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 2.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s 2024-09-21T14:05:59.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:59 smithi187 ceph-mon[33032]: 10.6 scrub starts 2024-09-21T14:05:59.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:59 smithi187 ceph-mon[33032]: 2.0 scrub starts 2024-09-21T14:05:59.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:59 smithi187 ceph-mon[33032]: 2.0 scrub ok 2024-09-21T14:05:59.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:59 smithi187 ceph-mon[33032]: 10.b scrub ok 2024-09-21T14:05:59.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:59 smithi187 ceph-mon[33032]: 10.6 scrub ok 2024-09-21T14:05:59.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:59 smithi187 ceph-mon[33032]: 3.9 scrub starts 2024-09-21T14:05:59.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:59 smithi187 ceph-mon[33032]: 3.9 scrub ok 2024-09-21T14:05:59.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:59 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/136009295' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "10.b", "id": [4, 6, 7, 5, 0, 1]} : dispatch 2024-09-21T14:05:59.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:59 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "10.b", "id": [4, 6, 7, 5, 0, 1]} : dispatch 2024-09-21T14:05:59.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:59 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:05:59.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:59 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:05:59.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:05:59 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1015995849' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:05:59.973 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:00.006 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:00.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:00.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:00.108 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:06:00.143 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:00.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:00.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:06:00.245 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:00.284 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:00.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:00.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:00.386 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:06:00.417 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:00.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:00.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:06:00.444 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:05:59 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:06:00.519 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:00.555 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:00.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:00.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:00.657 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:06:00.703 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:00.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:00.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:06:00.710 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T14:06:00.710 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-09-21T14:06:00.711 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-09-21T14:06:00.711 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-09-21T14:06:00.711 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.711 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232022 8.f 8.d7b7a40f (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T14:06:00.711 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.700289+0000", 2024-09-21T14:06:00.711 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 528.13333081400003, 2024-09-21T14:06:00.711 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97395471300000003, 2024-09-21T14:06:00.711 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:06:00.711 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:06:00.711 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T14:06:00.711 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:06:00.711 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T14:06:00.712 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:06:00.712 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232022 2024-09-21T14:06:00.712 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.712 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:06:00.712 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.712 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:06:00.712 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700289+0000", 2024-09-21T14:06:00.712 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:00.712 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.712 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.712 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:06:00.712 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700289+0000", 2024-09-21T14:06:00.712 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:00.712 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.712 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.713 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:06:00.713 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700291+0000", 2024-09-21T14:06:00.713 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5850000000000001e-06 2024-09-21T14:06:00.713 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.713 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.713 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:06:00.713 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700296+0000", 2024-09-21T14:06:00.713 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9749999999999998e-06 2024-09-21T14:06:00.713 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.713 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.713 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:06:00.713 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700306+0000", 2024-09-21T14:06:00.713 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9310000000000003e-06 2024-09-21T14:06:00.713 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.714 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.714 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:06:00.714 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700317+0000", 2024-09-21T14:06:00.714 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1199000000000001e-05 2024-09-21T14:06:00.714 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.714 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.714 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:00.714 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.711626+0000", 2024-09-21T14:06:00.714 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.011309009 2024-09-21T14:06:00.714 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.714 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.714 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:06:00.715 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.711637+0000", 2024-09-21T14:06:00.715 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1182000000000001e-05 2024-09-21T14:06:00.715 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.715 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.715 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:00.715 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674230+0000", 2024-09-21T14:06:00.715 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.962593535 2024-09-21T14:06:00.715 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.715 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.715 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:06:00.715 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674244+0000", 2024-09-21T14:06:00.715 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3297e-05 2024-09-21T14:06:00.715 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:00.715 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:00.716 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:00.716 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.716 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.716 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232025 8.f 8.4e5a668f (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T14:06:00.716 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.700329+0000", 2024-09-21T14:06:00.716 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 528.13329101600004, 2024-09-21T14:06:00.716 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97395089300000004, 2024-09-21T14:06:00.716 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:06:00.716 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:06:00.716 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T14:06:00.716 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:06:00.716 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T14:06:00.716 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:06:00.716 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232025 2024-09-21T14:06:00.717 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.717 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:06:00.717 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.717 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:06:00.717 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700329+0000", 2024-09-21T14:06:00.717 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:00.717 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.717 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.717 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:06:00.717 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700329+0000", 2024-09-21T14:06:00.717 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:00.717 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.717 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.717 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:06:00.718 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700329+0000", 2024-09-21T14:06:00.718 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7100000000000002e-07 2024-09-21T14:06:00.718 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.718 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.718 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:06:00.718 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700332+0000", 2024-09-21T14:06:00.718 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6589999999999999e-06 2024-09-21T14:06:00.718 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.718 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.718 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:06:00.718 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700339+0000", 2024-09-21T14:06:00.719 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8940000000000004e-06 2024-09-21T14:06:00.719 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.719 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.719 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:06:00.719 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700345+0000", 2024-09-21T14:06:00.719 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6219999999999997e-06 2024-09-21T14:06:00.719 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.719 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.719 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:00.719 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.711662+0000", 2024-09-21T14:06:00.719 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.011317522 2024-09-21T14:06:00.719 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.719 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.719 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:06:00.719 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.711669+0000", 2024-09-21T14:06:00.720 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2570000000000001e-06 2024-09-21T14:06:00.720 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.720 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.720 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:00.720 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674270+0000", 2024-09-21T14:06:00.720 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.96260044300000003 2024-09-21T14:06:00.720 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.720 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.720 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:06:00.720 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674280+0000", 2024-09-21T14:06:00.720 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0025e-05 2024-09-21T14:06:00.720 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:00.720 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:00.720 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:00.721 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.721 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.721 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232033 8.f 8.d949c02f (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T14:06:00.721 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.700352+0000", 2024-09-21T14:06:00.721 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 528.13326761600001, 2024-09-21T14:06:00.721 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97396006099999999, 2024-09-21T14:06:00.721 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:06:00.721 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:06:00.721 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T14:06:00.721 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:06:00.721 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T14:06:00.721 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:06:00.721 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232033 2024-09-21T14:06:00.722 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.722 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:06:00.722 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.722 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:06:00.722 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700352+0000", 2024-09-21T14:06:00.722 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:00.722 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.722 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.722 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:06:00.722 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700352+0000", 2024-09-21T14:06:00.722 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:00.722 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.723 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.723 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:06:00.723 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700353+0000", 2024-09-21T14:06:00.723 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.89e-07 2024-09-21T14:06:00.723 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.723 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.723 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:06:00.723 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700355+0000", 2024-09-21T14:06:00.723 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1840000000000002e-06 2024-09-21T14:06:00.723 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.723 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.723 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:06:00.723 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700364+0000", 2024-09-21T14:06:00.723 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4919999999999999e-06 2024-09-21T14:06:00.724 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.724 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.724 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:06:00.724 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700372+0000", 2024-09-21T14:06:00.724 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9659999999999996e-06 2024-09-21T14:06:00.724 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.724 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.724 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:00.724 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.711693+0000", 2024-09-21T14:06:00.724 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.011321059 2024-09-21T14:06:00.724 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.724 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.724 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:06:00.724 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.711701+0000", 2024-09-21T14:06:00.725 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3640000000000001e-06 2024-09-21T14:06:00.725 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.725 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.725 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:00.725 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674303+0000", 2024-09-21T14:06:00.725 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.96260253100000004 2024-09-21T14:06:00.725 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.725 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.725 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:06:00.725 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674312+0000", 2024-09-21T14:06:00.725 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0759999999999992e-06 2024-09-21T14:06:00.725 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:00.725 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:00.726 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:00.726 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.726 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.726 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232051 8.0 8.85c8f300 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T14:06:00.726 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.700380+0000", 2024-09-21T14:06:00.726 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 528.13323991300001, 2024-09-21T14:06:00.726 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97451152799999996, 2024-09-21T14:06:00.726 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:06:00.726 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:06:00.726 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T14:06:00.726 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:06:00.726 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T14:06:00.726 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:06:00.726 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232051 2024-09-21T14:06:00.727 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.727 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:06:00.727 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.727 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:06:00.727 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700380+0000", 2024-09-21T14:06:00.727 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:00.727 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.727 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.727 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:06:00.727 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700380+0000", 2024-09-21T14:06:00.727 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:00.727 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.727 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.727 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:06:00.728 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700380+0000", 2024-09-21T14:06:00.728 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1699999999999999e-07 2024-09-21T14:06:00.728 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.728 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.728 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:06:00.728 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700383+0000", 2024-09-21T14:06:00.728 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1600000000000001e-06 2024-09-21T14:06:00.728 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.728 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.728 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:06:00.728 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700389+0000", 2024-09-21T14:06:00.728 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6080000000000001e-06 2024-09-21T14:06:00.728 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.728 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.729 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:06:00.729 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.700394+0000", 2024-09-21T14:06:00.729 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9810000000000003e-06 2024-09-21T14:06:00.729 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.729 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.729 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:00.729 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.715050+0000", 2024-09-21T14:06:00.729 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.014656269 2024-09-21T14:06:00.729 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.729 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.729 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:06:00.729 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.715056+0000", 2024-09-21T14:06:00.729 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0129999999999999e-06 2024-09-21T14:06:00.729 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.730 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.730 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:00.730 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674876+0000", 2024-09-21T14:06:00.730 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.95981962700000001 2024-09-21T14:06:00.730 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.730 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.730 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:06:00.730 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674892+0000", 2024-09-21T14:06:00.730 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5452999999999999e-05 2024-09-21T14:06:00.730 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:00.730 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:00.730 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:00.731 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.731 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.731 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232059 8.f 8.b757f3bf (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T14:06:00.731 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.889250+0000", 2024-09-21T14:06:00.731 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 527.94437010000001, 2024-09-21T14:06:00.731 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.78509475900000003, 2024-09-21T14:06:00.731 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:06:00.731 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:06:00.731 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T14:06:00.731 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:06:00.732 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T14:06:00.732 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:06:00.732 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232059 2024-09-21T14:06:00.732 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.732 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:06:00.732 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.732 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:06:00.732 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889250+0000", 2024-09-21T14:06:00.732 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:00.732 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.732 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.732 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:06:00.732 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889250+0000", 2024-09-21T14:06:00.733 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:00.733 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.733 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.733 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:06:00.733 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889252+0000", 2024-09-21T14:06:00.733 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9130000000000001e-06 2024-09-21T14:06:00.733 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.733 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.733 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:06:00.733 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889258+0000", 2024-09-21T14:06:00.733 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7410000000000003e-06 2024-09-21T14:06:00.733 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.733 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.734 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:06:00.734 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889360+0000", 2024-09-21T14:06:00.734 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000101382 2024-09-21T14:06:00.734 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.734 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.734 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:06:00.734 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889391+0000", 2024-09-21T14:06:00.734 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1032999999999999e-05 2024-09-21T14:06:00.734 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.734 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.734 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:00.735 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889500+0000", 2024-09-21T14:06:00.735 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000109194 2024-09-21T14:06:00.735 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.735 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.735 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:06:00.735 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889512+0000", 2024-09-21T14:06:00.735 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2368000000000001e-05 2024-09-21T14:06:00.735 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.735 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.735 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:00.735 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674335+0000", 2024-09-21T14:06:00.736 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.78482304000000003 2024-09-21T14:06:00.736 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.736 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.736 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:06:00.737 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674345+0000", 2024-09-21T14:06:00.737 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0880000000000001e-06 2024-09-21T14:06:00.737 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:00.737 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:00.737 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:00.737 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.737 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.737 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232060 8.b 8.6c14963b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T14:06:00.737 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.889442+0000", 2024-09-21T14:06:00.737 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 527.94417791199999, 2024-09-21T14:06:00.738 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.78518956100000004, 2024-09-21T14:06:00.738 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:06:00.738 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:06:00.738 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T14:06:00.738 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:06:00.738 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T14:06:00.738 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:06:00.738 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232060 2024-09-21T14:06:00.738 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.738 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:06:00.738 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.738 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:06:00.739 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889442+0000", 2024-09-21T14:06:00.739 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:00.739 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.739 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.739 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:06:00.739 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889442+0000", 2024-09-21T14:06:00.739 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:00.739 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.739 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.739 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:06:00.739 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889443+0000", 2024-09-21T14:06:00.739 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9100000000000003e-07 2024-09-21T14:06:00.739 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.739 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.740 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:06:00.740 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889447+0000", 2024-09-21T14:06:00.740 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4939999999999997e-06 2024-09-21T14:06:00.740 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.740 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.740 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:06:00.740 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889457+0000", 2024-09-21T14:06:00.740 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0145e-05 2024-09-21T14:06:00.740 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.740 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.740 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:06:00.740 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889465+0000", 2024-09-21T14:06:00.740 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2749999999999998e-06 2024-09-21T14:06:00.740 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.740 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.741 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:00.741 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889550+0000", 2024-09-21T14:06:00.741 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5000999999999995e-05 2024-09-21T14:06:00.741 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.741 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.741 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:06:00.741 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.889561+0000", 2024-09-21T14:06:00.741 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.129e-05 2024-09-21T14:06:00.741 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.741 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.741 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:00.741 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674619+0000", 2024-09-21T14:06:00.741 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.785057703 2024-09-21T14:06:00.741 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.742 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.742 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:06:00.742 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674632+0000", 2024-09-21T14:06:00.742 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2962000000000001e-05 2024-09-21T14:06:00.742 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:00.742 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:00.742 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:00.742 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.742 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.742 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232068 8.f 8.d5b3687f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T14:06:00.742 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.894747+0000", 2024-09-21T14:06:00.742 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 527.93887322399996, 2024-09-21T14:06:00.742 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.77963143999999995, 2024-09-21T14:06:00.743 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:06:00.743 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:06:00.743 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T14:06:00.743 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:06:00.743 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T14:06:00.743 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:06:00.743 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232068 2024-09-21T14:06:00.743 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.743 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:06:00.743 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.743 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:06:00.743 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.894747+0000", 2024-09-21T14:06:00.743 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:00.743 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.744 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.744 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:06:00.744 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.894747+0000", 2024-09-21T14:06:00.744 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:00.744 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.744 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.744 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:06:00.744 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.894749+0000", 2024-09-21T14:06:00.744 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4849999999999999e-06 2024-09-21T14:06:00.744 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.744 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.744 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:06:00.744 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.894761+0000", 2024-09-21T14:06:00.744 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1809e-05 2024-09-21T14:06:00.744 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.745 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.745 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:06:00.745 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.894780+0000", 2024-09-21T14:06:00.745 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8593999999999998e-05 2024-09-21T14:06:00.745 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.745 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.745 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:06:00.745 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.894789+0000", 2024-09-21T14:06:00.745 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2839999999999995e-06 2024-09-21T14:06:00.745 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.745 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.745 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:00.745 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.894865+0000", 2024-09-21T14:06:00.745 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6150999999999996e-05 2024-09-21T14:06:00.745 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.746 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.746 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:06:00.746 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.894874+0000", 2024-09-21T14:06:00.746 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2450000000000004e-06 2024-09-21T14:06:00.746 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.746 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.746 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:00.746 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674368+0000", 2024-09-21T14:06:00.746 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.77949425999999999 2024-09-21T14:06:00.746 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.746 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.746 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:06:00.746 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674378+0000", 2024-09-21T14:06:00.746 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6120000000000005e-06 2024-09-21T14:06:00.747 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:00.747 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:00.747 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:00.747 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.747 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.747 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232074 8.0 8.57810 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T14:06:00.747 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.901181+0000", 2024-09-21T14:06:00.747 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 527.93243871100003, 2024-09-21T14:06:00.747 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.77382758200000001, 2024-09-21T14:06:00.747 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:06:00.747 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:06:00.747 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T14:06:00.747 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:06:00.747 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T14:06:00.747 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:06:00.748 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232074 2024-09-21T14:06:00.748 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.748 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:06:00.748 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.748 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:06:00.748 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.901181+0000", 2024-09-21T14:06:00.748 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:00.748 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.748 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.748 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:06:00.748 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.901181+0000", 2024-09-21T14:06:00.748 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:00.748 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.748 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.749 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:06:00.749 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.901184+0000", 2024-09-21T14:06:00.749 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.8660000000000002e-06 2024-09-21T14:06:00.749 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.749 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.749 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:06:00.749 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.901193+0000", 2024-09-21T14:06:00.749 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1549999999999996e-06 2024-09-21T14:06:00.749 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.749 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.749 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:06:00.749 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.901215+0000", 2024-09-21T14:06:00.749 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2272e-05 2024-09-21T14:06:00.749 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.750 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.750 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:06:00.750 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.901227+0000", 2024-09-21T14:06:00.750 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1116e-05 2024-09-21T14:06:00.750 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.750 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.750 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:00.750 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.901351+0000", 2024-09-21T14:06:00.750 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000124172 2024-09-21T14:06:00.750 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.750 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.750 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:06:00.750 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.901368+0000", 2024-09-21T14:06:00.750 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7008999999999999e-05 2024-09-21T14:06:00.750 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.751 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.751 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:00.751 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674970+0000", 2024-09-21T14:06:00.751 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.77360225199999999 2024-09-21T14:06:00.751 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.751 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.751 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:06:00.751 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.675009+0000", 2024-09-21T14:06:00.751 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8739999999999998e-05 2024-09-21T14:06:00.751 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:00.751 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:00.751 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:00.751 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.751 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.751 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232078 8.b 8.7f49789b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T14:06:00.752 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.904662+0000", 2024-09-21T14:06:00.752 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 527.92895830600003, 2024-09-21T14:06:00.752 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.77001060399999999, 2024-09-21T14:06:00.752 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:06:00.752 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:06:00.752 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T14:06:00.752 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:06:00.752 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T14:06:00.752 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:06:00.752 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232078 2024-09-21T14:06:00.752 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.752 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:06:00.752 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.753 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:06:00.753 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904662+0000", 2024-09-21T14:06:00.753 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:00.753 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.753 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.753 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:06:00.753 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904662+0000", 2024-09-21T14:06:00.753 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:00.753 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.753 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.753 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:06:00.753 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904663+0000", 2024-09-21T14:06:00.753 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.548e-06 2024-09-21T14:06:00.753 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.753 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.753 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:06:00.754 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904669+0000", 2024-09-21T14:06:00.754 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9619999999999998e-06 2024-09-21T14:06:00.754 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.754 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.754 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:06:00.754 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904684+0000", 2024-09-21T14:06:00.754 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4718e-05 2024-09-21T14:06:00.754 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.754 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.754 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:06:00.754 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904691+0000", 2024-09-21T14:06:00.754 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.861e-06 2024-09-21T14:06:00.754 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.754 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.755 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:00.755 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904750+0000", 2024-09-21T14:06:00.755 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9533000000000001e-05 2024-09-21T14:06:00.755 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.755 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.755 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:06:00.755 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904757+0000", 2024-09-21T14:06:00.755 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0369999999999997e-06 2024-09-21T14:06:00.755 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.755 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.755 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:00.755 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674661+0000", 2024-09-21T14:06:00.755 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.769903754 2024-09-21T14:06:00.755 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.756 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.756 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:06:00.756 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674672+0000", 2024-09-21T14:06:00.756 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1191e-05 2024-09-21T14:06:00.756 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:00.756 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:00.756 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:00.756 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.756 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.756 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.15207.0:232079 8.b 8.5c3e90b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T14:06:00.756 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:57:11.904710+0000", 2024-09-21T14:06:00.756 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 527.92890995100004, 2024-09-21T14:06:00.756 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.76999876, 2024-09-21T14:06:00.756 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:06:00.757 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:06:00.757 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T14:06:00.757 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:06:00.757 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.15207", 2024-09-21T14:06:00.757 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:06:00.757 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 232079 2024-09-21T14:06:00.757 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.757 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:06:00.757 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.757 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:06:00.757 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904710+0000", 2024-09-21T14:06:00.757 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:00.757 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.757 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.757 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:06:00.758 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904710+0000", 2024-09-21T14:06:00.758 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:00.758 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.758 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.758 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:06:00.758 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904711+0000", 2024-09-21T14:06:00.758 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8199999999999999e-07 2024-09-21T14:06:00.758 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.758 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.758 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:06:00.758 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904714+0000", 2024-09-21T14:06:00.758 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9079999999999999e-06 2024-09-21T14:06:00.758 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.758 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.759 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:06:00.759 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904726+0000", 2024-09-21T14:06:00.759 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2514000000000001e-05 2024-09-21T14:06:00.759 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.759 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.759 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:06:00.759 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904733+0000", 2024-09-21T14:06:00.759 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2690000000000002e-06 2024-09-21T14:06:00.759 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.759 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.759 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:00.759 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904784+0000", 2024-09-21T14:06:00.759 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1007000000000001e-05 2024-09-21T14:06:00.759 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.760 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.760 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:06:00.760 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:11.904794+0000", 2024-09-21T14:06:00.760 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7419999999999992e-06 2024-09-21T14:06:00.760 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.760 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.760 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:00.760 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674699+0000", 2024-09-21T14:06:00.760 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.76990482900000001 2024-09-21T14:06:00.760 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.760 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.760 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:06:00.760 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:57:12.674709+0000", 2024-09-21T14:06:00.760 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8090000000000004e-06 2024-09-21T14:06:00.761 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:00.761 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:00.761 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:00.761 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.761 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.761 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25421.0:113916 9.8 9:1f612498:::benchmark_data_smithi049_138504_object113915:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-21T14:06:00.761 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:59:25.602805+0000", 2024-09-21T14:06:00.761 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 394.23081451899998, 2024-09-21T14:06:00.761 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.34059980099999998, 2024-09-21T14:06:00.761 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:06:00.761 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:06:00.761 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:00.761 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:06:00.761 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25421", 2024-09-21T14:06:00.762 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:06:00.762 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 113916 2024-09-21T14:06:00.762 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.762 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:06:00.762 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.762 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:06:00.762 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.602805+0000", 2024-09-21T14:06:00.762 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:00.762 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.762 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.762 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:06:00.762 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.602805+0000", 2024-09-21T14:06:00.762 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:00.762 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.763 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.763 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:06:00.763 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.602807+0000", 2024-09-21T14:06:00.763 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7099999999999999e-06 2024-09-21T14:06:00.763 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.763 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.763 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:06:00.763 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.602830+0000", 2024-09-21T14:06:00.763 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2758000000000001e-05 2024-09-21T14:06:00.763 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.763 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.763 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:06:00.763 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.602847+0000", 2024-09-21T14:06:00.763 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6721000000000001e-05 2024-09-21T14:06:00.763 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.764 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.764 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:06:00.764 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.602854+0000", 2024-09-21T14:06:00.764 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9100000000000005e-06 2024-09-21T14:06:00.764 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.764 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.764 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:00.764 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.602925+0000", 2024-09-21T14:06:00.764 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0088999999999999e-05 2024-09-21T14:06:00.764 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.764 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.764 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:06:00.764 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.603043+0000", 2024-09-21T14:06:00.764 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011804200000000001 2024-09-21T14:06:00.765 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.765 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.765 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,5", 2024-09-21T14:06:00.765 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.603177+0000", 2024-09-21T14:06:00.765 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000134821 2024-09-21T14:06:00.765 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.765 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.765 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:06:00.765 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.606802+0000", 2024-09-21T14:06:00.765 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0036243880000000001 2024-09-21T14:06:00.765 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.765 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.765 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:00.765 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.608461+0000", 2024-09-21T14:06:00.766 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0016595710000000001 2024-09-21T14:06:00.766 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.766 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.766 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:00.766 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.943300+0000", 2024-09-21T14:06:00.766 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.33483894400000003 2024-09-21T14:06:00.766 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.766 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.766 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:06:00.766 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.943360+0000", 2024-09-21T14:06:00.766 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9401999999999998e-05 2024-09-21T14:06:00.766 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.766 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.766 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:06:00.767 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.943405+0000", 2024-09-21T14:06:00.767 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5445000000000003e-05 2024-09-21T14:06:00.767 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:00.767 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:00.767 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:00.767 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.767 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.767 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25421.0:113943 9.8 9:1e81a0a3:::benchmark_data_smithi049_138504_object113942:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-21T14:06:00.767 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:59:25.623062+0000", 2024-09-21T14:06:00.767 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 394.21055775600001, 2024-09-21T14:06:00.767 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.32046552900000003, 2024-09-21T14:06:00.767 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:06:00.767 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:06:00.767 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:00.768 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:06:00.768 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25421", 2024-09-21T14:06:00.768 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:06:00.768 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 113943 2024-09-21T14:06:00.768 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.768 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:06:00.768 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.768 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:06:00.768 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.623062+0000", 2024-09-21T14:06:00.768 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:00.768 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.768 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.768 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:06:00.768 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.623062+0000", 2024-09-21T14:06:00.769 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:00.769 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.769 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.769 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:06:00.769 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.623063+0000", 2024-09-21T14:06:00.769 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0210000000000001e-06 2024-09-21T14:06:00.769 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.769 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.769 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:06:00.769 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.623089+0000", 2024-09-21T14:06:00.769 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.567e-05 2024-09-21T14:06:00.769 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.769 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.769 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:06:00.770 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.623103+0000", 2024-09-21T14:06:00.770 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4248e-05 2024-09-21T14:06:00.770 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.770 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.770 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:06:00.770 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.623109+0000", 2024-09-21T14:06:00.770 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5949999999999998e-06 2024-09-21T14:06:00.770 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.770 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.770 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:00.770 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.623150+0000", 2024-09-21T14:06:00.770 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0868999999999998e-05 2024-09-21T14:06:00.770 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.770 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.771 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:06:00.771 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.623240+0000", 2024-09-21T14:06:00.771 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0562000000000005e-05 2024-09-21T14:06:00.771 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.771 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.771 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,5", 2024-09-21T14:06:00.771 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.623353+0000", 2024-09-21T14:06:00.771 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000113249 2024-09-21T14:06:00.771 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.771 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.771 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:06:00.771 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.625438+0000", 2024-09-21T14:06:00.771 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0020847750000000001 2024-09-21T14:06:00.771 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.772 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.772 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:00.772 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.626378+0000", 2024-09-21T14:06:00.772 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00093971199999999999 2024-09-21T14:06:00.772 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.772 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.772 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:00.772 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.943444+0000", 2024-09-21T14:06:00.772 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.317066615 2024-09-21T14:06:00.772 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.772 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.772 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:06:00.772 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.943487+0000", 2024-09-21T14:06:00.773 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2933999999999999e-05 2024-09-21T14:06:00.773 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.773 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.773 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:06:00.773 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.943528+0000", 2024-09-21T14:06:00.773 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0278999999999998e-05 2024-09-21T14:06:00.773 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:00.773 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:00.773 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:00.773 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.773 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.773 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25421.0:113948 9.8 9:1fea698d:::benchmark_data_smithi049_138504_object113947:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-21T14:06:00.773 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:59:25.630511+0000", 2024-09-21T14:06:00.773 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 394.20310847000002, 2024-09-21T14:06:00.773 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.31617158400000001, 2024-09-21T14:06:00.774 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:06:00.774 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:06:00.774 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:00.774 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:06:00.774 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25421", 2024-09-21T14:06:00.774 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:06:00.774 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 113948 2024-09-21T14:06:00.774 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.774 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:06:00.774 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.774 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:06:00.774 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.630511+0000", 2024-09-21T14:06:00.774 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:00.775 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.775 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.775 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:06:00.775 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.630511+0000", 2024-09-21T14:06:00.775 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:00.775 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.775 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.775 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:06:00.775 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.630512+0000", 2024-09-21T14:06:00.775 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.16e-07 2024-09-21T14:06:00.775 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.775 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.775 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:06:00.775 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.630531+0000", 2024-09-21T14:06:00.775 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8845999999999999e-05 2024-09-21T14:06:00.776 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.776 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.776 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:06:00.776 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.630542+0000", 2024-09-21T14:06:00.776 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0651e-05 2024-09-21T14:06:00.776 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.776 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.776 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:06:00.776 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.630547+0000", 2024-09-21T14:06:00.776 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8770000000000001e-06 2024-09-21T14:06:00.776 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.776 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.776 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:00.776 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.630585+0000", 2024-09-21T14:06:00.776 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8488999999999997e-05 2024-09-21T14:06:00.777 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.777 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.777 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:06:00.777 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.630660+0000", 2024-09-21T14:06:00.777 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5290000000000006e-05 2024-09-21T14:06:00.777 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.777 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.777 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,5", 2024-09-21T14:06:00.777 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.630760+0000", 2024-09-21T14:06:00.777 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9834000000000007e-05 2024-09-21T14:06:00.777 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.777 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.777 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:06:00.777 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.632607+0000", 2024-09-21T14:06:00.778 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001847174 2024-09-21T14:06:00.778 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.778 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.778 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:00.778 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.633151+0000", 2024-09-21T14:06:00.778 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00054377600000000005 2024-09-21T14:06:00.778 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.778 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.778 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:00.778 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.946575+0000", 2024-09-21T14:06:00.778 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.31342341099999999 2024-09-21T14:06:00.778 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.778 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.778 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:06:00.779 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.946632+0000", 2024-09-21T14:06:00.779 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7055999999999997e-05 2024-09-21T14:06:00.779 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.779 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.779 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:06:00.779 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.946683+0000", 2024-09-21T14:06:00.779 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1264000000000002e-05 2024-09-21T14:06:00.779 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:00.779 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:00.779 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:00.779 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.779 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.779 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25421.0:172314 9.4 e315/260 9:2b749f7e:::benchmark_data_smithi049_138504_object50906:head v 315'10563, mlcod=315'10559)", 2024-09-21T14:06:00.779 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:00:33.250136+0000", 2024-09-21T14:06:00.780 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 326.58348383800001, 2024-09-21T14:06:00.780 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.270468655, 2024-09-21T14:06:00.780 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:06:00.780 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:06:00.780 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:00.780 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:06:00.780 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.780 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:06:00.780 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250136+0000", 2024-09-21T14:06:00.780 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:00.780 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.780 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.780 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:06:00.780 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250136+0000", 2024-09-21T14:06:00.781 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:00.781 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.781 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.781 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:06:00.781 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250137+0000", 2024-09-21T14:06:00.781 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1999999999999999e-07 2024-09-21T14:06:00.781 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.781 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.781 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:06:00.781 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250142+0000", 2024-09-21T14:06:00.781 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5609999999999998e-06 2024-09-21T14:06:00.781 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.781 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.781 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:06:00.782 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250150+0000", 2024-09-21T14:06:00.782 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1310000000000006e-06 2024-09-21T14:06:00.782 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.782 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.782 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:06:00.782 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250156+0000", 2024-09-21T14:06:00.782 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7740000000000001e-06 2024-09-21T14:06:00.782 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.782 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.782 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:00.782 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.253391+0000", 2024-09-21T14:06:00.782 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0032349969999999999 2024-09-21T14:06:00.782 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.782 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.783 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:06:00.783 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.253410+0000", 2024-09-21T14:06:00.783 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9097e-05 2024-09-21T14:06:00.783 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.783 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.783 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:06:00.783 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.520555+0000", 2024-09-21T14:06:00.783 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26714502600000001 2024-09-21T14:06:00.783 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.783 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.783 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:06:00.783 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.520605+0000", 2024-09-21T14:06:00.783 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9449e-05 2024-09-21T14:06:00.783 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:00.784 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:00.784 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:00.784 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.784 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.784 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25421.0:172319 9.4 e315/260 9:21e132db:::benchmark_data_smithi049_138504_object50911:head v 315'10564, mlcod=315'10559)", 2024-09-21T14:06:00.784 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:00:33.250191+0000", 2024-09-21T14:06:00.784 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 326.58342885399998, 2024-09-21T14:06:00.784 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.270657121, 2024-09-21T14:06:00.784 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:06:00.784 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:06:00.784 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:00.784 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:06:00.784 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.784 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:06:00.785 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250191+0000", 2024-09-21T14:06:00.785 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:00.785 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.785 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.785 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:06:00.785 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250191+0000", 2024-09-21T14:06:00.785 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:00.785 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.785 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.785 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:06:00.785 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250191+0000", 2024-09-21T14:06:00.785 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8799999999999998e-07 2024-09-21T14:06:00.785 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.785 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.786 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:06:00.786 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250195+0000", 2024-09-21T14:06:00.786 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.647e-06 2024-09-21T14:06:00.786 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.786 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.786 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:06:00.786 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250201+0000", 2024-09-21T14:06:00.786 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4200000000000004e-06 2024-09-21T14:06:00.786 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.786 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.786 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:06:00.786 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250206+0000", 2024-09-21T14:06:00.786 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.994e-06 2024-09-21T14:06:00.786 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.787 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.787 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:00.787 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.256110+0000", 2024-09-21T14:06:00.787 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0059039840000000001 2024-09-21T14:06:00.787 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.787 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.787 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:06:00.787 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.256130+0000", 2024-09-21T14:06:00.787 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.98e-05 2024-09-21T14:06:00.787 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.787 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.787 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:06:00.787 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.520816+0000", 2024-09-21T14:06:00.787 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.264685528 2024-09-21T14:06:00.788 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.788 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.788 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:06:00.788 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.520848+0000", 2024-09-21T14:06:00.788 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2360000000000002e-05 2024-09-21T14:06:00.788 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:00.788 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:00.788 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:00.788 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.788 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.788 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25421.0:172326 9.4 e315/260 9:22a8fc8a:::benchmark_data_smithi049_138504_object50918:head v 315'10565, mlcod=315'10560)", 2024-09-21T14:06:00.788 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:00:33.254759+0000", 2024-09-21T14:06:00.788 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 326.57886056400002, 2024-09-21T14:06:00.788 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26611502300000001, 2024-09-21T14:06:00.789 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:06:00.789 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:06:00.789 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:00.789 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:06:00.789 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.789 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:06:00.789 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254759+0000", 2024-09-21T14:06:00.789 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:00.789 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.789 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.789 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:06:00.789 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254759+0000", 2024-09-21T14:06:00.789 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:00.789 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.789 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.790 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:06:00.790 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254760+0000", 2024-09-21T14:06:00.790 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6400000000000001e-07 2024-09-21T14:06:00.790 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.790 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.790 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:06:00.790 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254854+0000", 2024-09-21T14:06:00.790 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4320999999999994e-05 2024-09-21T14:06:00.790 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.790 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.790 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:06:00.790 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254865+0000", 2024-09-21T14:06:00.790 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.101e-05 2024-09-21T14:06:00.790 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.791 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.791 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:06:00.791 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254872+0000", 2024-09-21T14:06:00.791 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2539999999999998e-06 2024-09-21T14:06:00.791 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.791 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.791 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:00.791 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.258679+0000", 2024-09-21T14:06:00.791 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.003807342 2024-09-21T14:06:00.791 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.791 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.791 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:06:00.791 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.258699+0000", 2024-09-21T14:06:00.791 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.039e-05 2024-09-21T14:06:00.792 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.792 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.792 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:06:00.792 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.520851+0000", 2024-09-21T14:06:00.792 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26215118900000001 2024-09-21T14:06:00.792 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.792 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.792 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:06:00.792 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.520874+0000", 2024-09-21T14:06:00.792 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3753000000000001e-05 2024-09-21T14:06:00.792 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:00.792 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:00.792 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:00.792 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.793 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.793 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25421.0:172331 9.f e315/260 9:f6449e5b:::benchmark_data_smithi049_138504_object50923:head v 315'10871, mlcod=315'10867)", 2024-09-21T14:06:00.793 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:00:33.254942+0000", 2024-09-21T14:06:00.793 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 326.57867818699998, 2024-09-21T14:06:00.793 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.274133181, 2024-09-21T14:06:00.793 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:06:00.793 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:06:00.793 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:00.793 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:06:00.793 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.793 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:06:00.793 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254942+0000", 2024-09-21T14:06:00.793 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:00.794 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.794 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.794 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:06:00.794 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254942+0000", 2024-09-21T14:06:00.794 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:00.794 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.794 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.794 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:06:00.794 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254942+0000", 2024-09-21T14:06:00.794 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5700000000000002e-07 2024-09-21T14:06:00.794 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.794 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.794 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:06:00.795 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254949+0000", 2024-09-21T14:06:00.795 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.024e-06 2024-09-21T14:06:00.795 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.795 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.795 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:06:00.795 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254959+0000", 2024-09-21T14:06:00.795 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6390000000000004e-06 2024-09-21T14:06:00.795 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.795 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.795 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:06:00.795 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254965+0000", 2024-09-21T14:06:00.795 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6420000000000004e-06 2024-09-21T14:06:00.795 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.795 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.795 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:00.796 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.259407+0000", 2024-09-21T14:06:00.796 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0044420140000000002 2024-09-21T14:06:00.796 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.796 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.796 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:06:00.796 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.259425+0000", 2024-09-21T14:06:00.796 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7937000000000001e-05 2024-09-21T14:06:00.796 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.796 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.796 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:06:00.796 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.529033+0000", 2024-09-21T14:06:00.796 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26960846599999999 2024-09-21T14:06:00.796 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.797 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.797 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:06:00.797 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.529075+0000", 2024-09-21T14:06:00.797 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1801999999999997e-05 2024-09-21T14:06:00.797 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:00.797 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:00.797 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:00.797 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.797 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.797 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25421.0:172333 9.2 9:461458cc:::benchmark_data_smithi049_138504_object50925:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-09-21T14:06:00.797 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:00:33.258960+0000", 2024-09-21T14:06:00.797 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 326.57466005200001, 2024-09-21T14:06:00.797 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.261926308, 2024-09-21T14:06:00.797 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:06:00.797 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:06:00.798 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:00.798 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:06:00.798 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25421", 2024-09-21T14:06:00.798 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:06:00.798 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 172333 2024-09-21T14:06:00.798 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.798 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:06:00.798 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.798 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:06:00.798 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.258960+0000", 2024-09-21T14:06:00.798 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:00.798 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.798 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.798 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:06:00.799 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.258960+0000", 2024-09-21T14:06:00.799 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:00.799 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.799 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.799 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:06:00.799 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.258961+0000", 2024-09-21T14:06:00.799 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5629999999999999e-06 2024-09-21T14:06:00.799 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.799 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.799 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:06:00.799 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.258967+0000", 2024-09-21T14:06:00.799 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5079999999999996e-06 2024-09-21T14:06:00.799 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.799 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.799 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:06:00.800 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.258979+0000", 2024-09-21T14:06:00.800 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1897e-05 2024-09-21T14:06:00.800 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.800 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.800 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:06:00.800 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.258986+0000", 2024-09-21T14:06:00.800 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8739999999999997e-06 2024-09-21T14:06:00.800 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.800 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.800 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:00.800 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.259036+0000", 2024-09-21T14:06:00.800 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0695000000000002e-05 2024-09-21T14:06:00.800 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.800 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.801 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:06:00.801 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.259144+0000", 2024-09-21T14:06:00.801 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000107462 2024-09-21T14:06:00.801 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.801 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.801 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,2", 2024-09-21T14:06:00.801 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.259252+0000", 2024-09-21T14:06:00.801 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000107667 2024-09-21T14:06:00.801 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.801 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.801 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:06:00.801 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.517843+0000", 2024-09-21T14:06:00.801 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.25859111400000001 2024-09-21T14:06:00.801 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.801 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.802 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:00.802 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.517994+0000", 2024-09-21T14:06:00.802 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00015150299999999999 2024-09-21T14:06:00.802 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.802 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.802 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:00.802 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.520805+0000", 2024-09-21T14:06:00.802 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0028113489999999999 2024-09-21T14:06:00.802 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.802 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.802 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:06:00.802 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.520850+0000", 2024-09-21T14:06:00.802 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4333999999999999e-05 2024-09-21T14:06:00.803 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.803 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.803 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:06:00.803 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.520886+0000", 2024-09-21T14:06:00.803 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6341999999999999e-05 2024-09-21T14:06:00.803 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:00.803 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:00.803 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:00.803 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.803 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.803 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.16068.0:208862 10.d e441/434 10:b40252c3:::benchmark_data_smithi049_176811_object81285:head v 441'13126, mlcod=441'13122)", 2024-09-21T14:06:00.803 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:05:54.047328+0000", 2024-09-21T14:06:00.803 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 5.7862917449999998, 2024-09-21T14:06:00.803 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.31640931900000002, 2024-09-21T14:06:00.804 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:06:00.804 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:06:00.804 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:00.804 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:06:00.804 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.804 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:06:00.804 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047328+0000", 2024-09-21T14:06:00.804 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:00.804 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.804 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.804 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:06:00.804 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047328+0000", 2024-09-21T14:06:00.804 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:00.804 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.804 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.805 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:06:00.805 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047329+0000", 2024-09-21T14:06:00.805 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0410000000000001e-06 2024-09-21T14:06:00.805 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.805 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.805 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:06:00.805 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047338+0000", 2024-09-21T14:06:00.805 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6160000000000002e-06 2024-09-21T14:06:00.805 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.805 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.805 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:06:00.805 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047348+0000", 2024-09-21T14:06:00.805 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0688e-05 2024-09-21T14:06:00.805 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.806 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.806 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:06:00.806 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047356+0000", 2024-09-21T14:06:00.806 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3350000000000001e-06 2024-09-21T14:06:00.806 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.806 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.806 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:00.806 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047399+0000", 2024-09-21T14:06:00.806 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2833000000000001e-05 2024-09-21T14:06:00.806 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.806 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.806 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:06:00.806 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047415+0000", 2024-09-21T14:06:00.807 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6662000000000001e-05 2024-09-21T14:06:00.807 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.807 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.807 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:06:00.807 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.363700+0000", 2024-09-21T14:06:00.807 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.31628446799999999 2024-09-21T14:06:00.807 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.807 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.807 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:06:00.807 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.363737+0000", 2024-09-21T14:06:00.807 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7676000000000003e-05 2024-09-21T14:06:00.807 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:00.807 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:00.807 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:00.808 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.808 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.808 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.16068.0:208866 10.3 e441/381 10:c8242cfb:::benchmark_data_smithi049_176811_object81289:head v 441'12975, mlcod=441'12973)", 2024-09-21T14:06:00.808 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:05:54.047688+0000", 2024-09-21T14:06:00.808 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 5.7859321929999998, 2024-09-21T14:06:00.808 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.31391443499999999, 2024-09-21T14:06:00.808 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:06:00.808 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:06:00.808 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:00.808 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:06:00.808 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.808 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:06:00.808 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047688+0000", 2024-09-21T14:06:00.808 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:00.809 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.809 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.809 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:06:00.809 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047688+0000", 2024-09-21T14:06:00.809 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:00.809 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.809 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.809 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:06:00.809 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047688+0000", 2024-09-21T14:06:00.809 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9999999999999997e-07 2024-09-21T14:06:00.809 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.809 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.809 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:06:00.809 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047693+0000", 2024-09-21T14:06:00.809 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7759999999999997e-06 2024-09-21T14:06:00.810 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.810 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.810 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:06:00.810 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.357055+0000", 2024-09-21T14:06:00.810 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.309361422 2024-09-21T14:06:00.810 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.810 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.810 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:06:00.810 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.357070+0000", 2024-09-21T14:06:00.810 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5311e-05 2024-09-21T14:06:00.810 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.810 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.810 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:00.810 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.357133+0000", 2024-09-21T14:06:00.811 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2611000000000001e-05 2024-09-21T14:06:00.811 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.811 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.811 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:06:00.811 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.357154+0000", 2024-09-21T14:06:00.811 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1376e-05 2024-09-21T14:06:00.811 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.811 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.811 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:06:00.811 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.361561+0000", 2024-09-21T14:06:00.811 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0044073599999999999 2024-09-21T14:06:00.811 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:00.811 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:00.811 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:06:00.812 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.361602+0000", 2024-09-21T14:06:00.812 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0878999999999999e-05 2024-09-21T14:06:00.812 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:00.812 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:00.812 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:00.812 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:00.812 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:00.812 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T14:06:00.812 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:00.849 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:00.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:00.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:00.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:00 smithi187 ceph-mon[33032]: 5.1d scrub starts 2024-09-21T14:06:00.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:00 smithi187 ceph-mon[33032]: 5.1a scrub starts 2024-09-21T14:06:00.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:00 smithi187 ceph-mon[33032]: 5.1d scrub ok 2024-09-21T14:06:00.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:00 smithi187 ceph-mon[33032]: 5.1a scrub ok 2024-09-21T14:06:00.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:00 smithi187 ceph-mon[33032]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:06:00.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:00 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "10.b", "id": [4, 6, 7, 5, 0, 1]}]': finished 2024-09-21T14:06:00.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:00 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1015995849' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T14:06:00.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:00 smithi187 ceph-mon[33032]: osdmap e442: 8 total, 8 up, 8 in 2024-09-21T14:06:00.951 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:00.989 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:00.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:00.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:01.004 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:06:00 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:06:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:06:01.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:00 smithi049 ceph-mon[40489]: 5.1d scrub starts 2024-09-21T14:06:01.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:00 smithi049 ceph-mon[40489]: 5.1a scrub starts 2024-09-21T14:06:01.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:00 smithi049 ceph-mon[40489]: 5.1d scrub ok 2024-09-21T14:06:01.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:00 smithi049 ceph-mon[40489]: 5.1a scrub ok 2024-09-21T14:06:01.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:00 smithi049 ceph-mon[40489]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:06:01.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:00 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "10.b", "id": [4, 6, 7, 5, 0, 1]}]': finished 2024-09-21T14:06:01.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:00 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1015995849' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T14:06:01.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:00 smithi049 ceph-mon[40489]: osdmap e442: 8 total, 8 up, 8 in 2024-09-21T14:06:01.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:00 smithi049 ceph-mon[30080]: 5.1d scrub starts 2024-09-21T14:06:01.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:00 smithi049 ceph-mon[30080]: 5.1a scrub starts 2024-09-21T14:06:01.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:00 smithi049 ceph-mon[30080]: 5.1d scrub ok 2024-09-21T14:06:01.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:00 smithi049 ceph-mon[30080]: 5.1a scrub ok 2024-09-21T14:06:01.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:00 smithi049 ceph-mon[30080]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:06:01.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:00 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "10.b", "id": [4, 6, 7, 5, 0, 1]}]': finished 2024-09-21T14:06:01.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:00 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1015995849' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T14:06:01.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:00 smithi049 ceph-mon[30080]: osdmap e442: 8 total, 8 up, 8 in 2024-09-21T14:06:01.091 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:06:01.121 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:01.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:01.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:06:01.223 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:06:01.254 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:01.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:01.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:06:01.356 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:06:01.384 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T14:06:01.400 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:01.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:01.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:06:01.502 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:01.534 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:01.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:01.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:01.636 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:06:01.690 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:01.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:01.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:06:01.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:01 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:06:01.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:01 smithi187 ceph-mon[33032]: pgmap v867: 168 pgs: 168 active+clean; 2.5 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.26k op/s 2024-09-21T14:06:01.791 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:01.815 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:06:01.826 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:01.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:01.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:01.928 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:06:01.961 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:01.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:01.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:06:02.001 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:01 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:06:02.001 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:01 smithi049 ceph-mon[40489]: pgmap v867: 168 pgs: 168 active+clean; 2.5 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.26k op/s 2024-09-21T14:06:02.001 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:01 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:06:02.001 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:01 smithi049 ceph-mon[30080]: pgmap v867: 168 pgs: 168 active+clean; 2.5 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.26k op/s 2024-09-21T14:06:02.062 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:02.106 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:02.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:02.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:02.207 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:06:02.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:02.246 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:02.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:06:02.347 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:02.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:02.379 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:02.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:02.450 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T14:06:02.450 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-21T14:06:02.450 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:02.450 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.16068.0:219288 10.8 e442/417 10:1daee8c6:::benchmark_data_smithi049_176811_object91711:head v 442'13822, mlcod=442'13819)", 2024-09-21T14:06:02.450 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:06:02.448132+0000", 2024-09-21T14:06:02.451 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0014766410000000001, 2024-09-21T14:06:02.451 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0015028870000000001, 2024-09-21T14:06:02.451 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:06:02.451 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:06:02.451 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T14:06:02.451 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:06:02.451 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:02.451 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:06:02.451 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:06:02.448132+0000", 2024-09-21T14:06:02.451 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:02.451 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:02.451 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:02.451 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:06:02.451 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:06:02.448132+0000", 2024-09-21T14:06:02.452 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:02.452 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:02.452 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:02.452 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:06:02.452 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:06:02.448133+0000", 2024-09-21T14:06:02.452 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8000000000000004e-07 2024-09-21T14:06:02.452 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:02.452 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:02.452 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:06:02.452 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:06:02.448139+0000", 2024-09-21T14:06:02.452 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4400000000000002e-06 2024-09-21T14:06:02.452 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:02.452 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:02.452 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:06:02.452 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:06:02.448153+0000", 2024-09-21T14:06:02.453 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.362e-05 2024-09-21T14:06:02.453 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:02.453 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:02.453 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:06:02.453 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:06:02.448160+0000", 2024-09-21T14:06:02.453 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5569999999999998e-06 2024-09-21T14:06:02.453 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:02.453 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:02.453 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:02.453 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:06:02.448208+0000", 2024-09-21T14:06:02.453 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7679999999999998e-05 2024-09-21T14:06:02.453 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:02.453 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:02.453 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:06:02.453 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:06:02.448228+0000", 2024-09-21T14:06:02.454 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0378999999999999e-05 2024-09-21T14:06:02.454 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:02.454 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:06:02.454 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:02.454 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:02.454 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2024-09-21T14:06:02.454 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 1 2024-09-21T14:06:02.454 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T14:06:02.481 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:06:02.508 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:06:02.517 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:02.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:02.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:06:02.618 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:06:02.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:02.656 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:02.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:06:02.757 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:06:02.793 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:02.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:02.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:06:02.895 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:02.935 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:02.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:02.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:03.037 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:06:03.084 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:03.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:03.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:06:03.186 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:06:03.214 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:03.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:03.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:06:03.229 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:06:03.229 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:06:03.266 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:06:03.267 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:06:03.304 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:06:03.304 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:06:03.316 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:03.341 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:06:03.342 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:06:03.360 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:03.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:03.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:03.379 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:06:03.379 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:06:03.414 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:06:03.414 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:06:03.449 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:06:03.450 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:06:03.462 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:03.484 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:06:03.485 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:06:03.501 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:03.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:03.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:03.504 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T14:06:03.603 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:06:03.640 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:03.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:03.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:06:03.742 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:06:03.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:03 smithi049 ceph-mon[40489]: pgmap v868: 168 pgs: 168 active+clean; 2.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.26k op/s 2024-09-21T14:06:03.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:03 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/992997645' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:06:03.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:03 smithi049 ceph-mon[30080]: pgmap v868: 168 pgs: 168 active+clean; 2.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.26k op/s 2024-09-21T14:06:03.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:03 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/992997645' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:06:03.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:03.785 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:03.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:06:03.886 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:03.914 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:03.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:03.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:03.943 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:06:03.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:03 smithi187 ceph-mon[33032]: pgmap v868: 168 pgs: 168 active+clean; 2.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.26k op/s 2024-09-21T14:06:03.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:03 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/992997645' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:06:04.016 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:06:04.055 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:04.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:04.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:06:04.060 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T14:06:04.157 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:04.206 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:04.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:04.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:04.308 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:06:04.339 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:04.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:04.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:06:04.441 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:04.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:04.498 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:04.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:04.551 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:06:04.562 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T14:06:04.563 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:06:04.600 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:06:04.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:04.633 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:04.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:06:04.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:04 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:06:04.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:04 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:06:04.734 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:06:04.769 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:04.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:04.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:06:04.871 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:06:04.881 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:04 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:06:04.881 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:04 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:06:04.881 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:04 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:06:04.881 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:04 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:06:04.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:04.919 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:04.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:06:05.020 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:05.071 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:05.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:05.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:05.173 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:06:05.204 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:05.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:05.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:06:05.299 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T14:06:05.306 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:06:05.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:05.403 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:05.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:06:05.504 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:06:05.544 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:05.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:05.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:06:05.646 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:05.681 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:05.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:05.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:05.723 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:06:05.783 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:06:05.821 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:05.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:05.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:06:05.885 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:05 smithi187 ceph-mon[33032]: pgmap v869: 168 pgs: 168 active+clean; 2.2 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.25k op/s 2024-09-21T14:06:05.885 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:05 smithi187 ceph-mon[33032]: osdmap e443: 8 total, 8 up, 8 in 2024-09-21T14:06:05.885 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:05 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/309304540' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:06:05.885 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:05 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:06:05.923 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:06:05.960 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:05.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:05.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:06:06.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:05 smithi049 ceph-mon[30080]: pgmap v869: 168 pgs: 168 active+clean; 2.2 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.25k op/s 2024-09-21T14:06:06.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:05 smithi049 ceph-mon[30080]: osdmap e443: 8 total, 8 up, 8 in 2024-09-21T14:06:06.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:05 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/309304540' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:06:06.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:05 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:06:06.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:05 smithi049 ceph-mon[40489]: pgmap v869: 168 pgs: 168 active+clean; 2.2 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.25k op/s 2024-09-21T14:06:06.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:05 smithi049 ceph-mon[40489]: osdmap e443: 8 total, 8 up, 8 in 2024-09-21T14:06:06.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:05 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/309304540' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:06:06.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:05 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:06:06.062 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:06.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:06.098 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:06.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:06.179 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-09-21T14:06:06.199 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:06:06.238 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:06.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:06.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:06:06.242 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T14:06:06.340 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:06.340 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T14:06:06.340 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:06:06.382 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:06.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:06.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:06.484 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:06:06.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:06.519 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:06.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:06:06.620 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:06.664 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:06.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:06.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:06.766 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:06.808 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:06.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:06.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:06.910 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:06:06.944 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:06.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:06 smithi187 ceph-mon[33032]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:06:06.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:06 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T14:06:06.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:06 smithi187 ceph-mon[33032]: osdmap e444: 8 total, 8 up, 8 in 2024-09-21T14:06:06.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:06.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:06:07.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:06 smithi049 ceph-mon[30080]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:06:07.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:06 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T14:06:07.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:06 smithi049 ceph-mon[30080]: osdmap e444: 8 total, 8 up, 8 in 2024-09-21T14:06:07.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:06 smithi049 ceph-mon[40489]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:06:07.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:06 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T14:06:07.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:06 smithi049 ceph-mon[40489]: osdmap e444: 8 total, 8 up, 8 in 2024-09-21T14:06:07.046 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:06:07.081 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:07.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:07.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:06:07.171 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T14:06:07.183 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:06:07.225 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:06:07.228 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:07.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:07.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:06:07.330 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:06:07.373 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:07.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:07.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:06:07.475 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:07.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:07.512 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:07.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:07.532 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 4, 6, 2, 3, 7, 5] 2024-09-21T14:06:07.533 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T14:06:07.533 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-09-21T14:06:07.533 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph tell osd.0 reset_purged_snaps_last 2024-09-21T14:06:07.581 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:06:07.613 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:07.670 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:07.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:07.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:07.772 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:06:07.803 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:07.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:07.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:06:07.899 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:07 smithi187 ceph-mon[33032]: pgmap v871: 168 pgs: 168 active+clean; 2.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.53k op/s 2024-09-21T14:06:07.899 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:07 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:06:07.899 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:07 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:06:07.905 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:06:07.905 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:06:07.906 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:06:07.906 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:07 smithi049 ceph-mon[30080]: pgmap v871: 168 pgs: 168 active+clean; 2.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.53k op/s 2024-09-21T14:06:07.907 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:07 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:06:07.907 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:07 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:06:07.907 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:07 smithi049 ceph-mon[40489]: pgmap v871: 168 pgs: 168 active+clean; 2.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.53k op/s 2024-09-21T14:06:07.907 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:07 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:06:07.907 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:07 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:06:07.939 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:07.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:07.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:06:07.941 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:06:07.941 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:06:07.975 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:06:07.976 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:06:07.993 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:06:07.994 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:06:08.008 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:06:08.041 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:06:08.071 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:06:08.071 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:06:08.073 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:08.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:08.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:06:08.104 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:06:08.105 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:06:08.142 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:06:08.142 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:06:08.173 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:06:08.173 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:06:08.175 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:06:08.224 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:08.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:08.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:06:08.326 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:06:08.364 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:08.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:08.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:06:08.466 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:06:08.485 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T14:06:08.485 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-21T14:06:08.485 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:08.485 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.16068.0:227004 10.a e445/433 10:518a12de:::benchmark_data_smithi049_176811_object99427:head v 445'14079, mlcod=445'14077)", 2024-09-21T14:06:08.485 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:06:08.463899+0000", 2024-09-21T14:06:08.485 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0027081829999999999, 2024-09-21T14:06:08.485 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0027367149999999998, 2024-09-21T14:06:08.485 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:06:08.485 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:06:08.485 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T14:06:08.485 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:06:08.485 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:08.485 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:06:08.485 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:06:08.463899+0000", 2024-09-21T14:06:08.486 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:08.486 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:08.486 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:08.486 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:06:08.486 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:06:08.463899+0000", 2024-09-21T14:06:08.486 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:08.486 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:08.486 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:08.486 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:06:08.486 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:06:08.463900+0000", 2024-09-21T14:06:08.486 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.012e-06 2024-09-21T14:06:08.486 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:08.486 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:08.486 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:06:08.486 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:06:08.463907+0000", 2024-09-21T14:06:08.487 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6569999999999997e-06 2024-09-21T14:06:08.487 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:08.487 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:08.487 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:06:08.487 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:06:08.463925+0000", 2024-09-21T14:06:08.487 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7748e-05 2024-09-21T14:06:08.487 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:08.487 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:08.487 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:06:08.487 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:06:08.463934+0000", 2024-09-21T14:06:08.487 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.422e-06 2024-09-21T14:06:08.487 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:08.487 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:08.487 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:08.487 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:06:08.464016+0000", 2024-09-21T14:06:08.488 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1946000000000005e-05 2024-09-21T14:06:08.488 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:08.488 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:08.488 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:06:08.488 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:06:08.464037+0000", 2024-09-21T14:06:08.488 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1212999999999998e-05 2024-09-21T14:06:08.488 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:08.488 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:06:08.488 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:08.488 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:08.488 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2024-09-21T14:06:08.488 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 1 2024-09-21T14:06:08.488 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T14:06:08.502 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:08.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:08.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:06:08.604 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:06:08.641 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:08.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:08.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:06:08.743 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:08.780 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:08.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:08.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:08.883 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:08.916 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:08.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:08.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:08.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:08 smithi187 ceph-mon[33032]: osdmap e445: 8 total, 8 up, 8 in 2024-09-21T14:06:08.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:08 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3057915401' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:06:09.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:08 smithi049 ceph-mon[30080]: osdmap e445: 8 total, 8 up, 8 in 2024-09-21T14:06:09.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:08 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3057915401' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:06:09.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:08 smithi049 ceph-mon[40489]: osdmap e445: 8 total, 8 up, 8 in 2024-09-21T14:06:09.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:08 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3057915401' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:06:09.018 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:06:09.051 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:09.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:09.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:06:09.056 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T14:06:09.152 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:06:09.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:09.181 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:09.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:06:09.281 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:09.319 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:09.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:09.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:09.421 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:06:09.449 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:09.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:09.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:06:09.550 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:06:09.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:09.584 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:09.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:06:09.685 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:09.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:09.723 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:09.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:09.824 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:06:09.859 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:09.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:09.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:06:09.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:09 smithi187 ceph-mon[33032]: pgmap v874: 168 pgs: 168 active+clean; 2.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.07k op/s 2024-09-21T14:06:09.961 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:06:09.983 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:09 smithi049 ceph-mon[40489]: pgmap v874: 168 pgs: 168 active+clean; 2.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.07k op/s 2024-09-21T14:06:09.983 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:09 smithi049 ceph-mon[30080]: pgmap v874: 168 pgs: 168 active+clean; 2.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.07k op/s 2024-09-21T14:06:09.997 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:09.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:09.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:06:10.072 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:06:10.098 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:06:10.123 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:10.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:10.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:06:10.225 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:06:10.259 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:10.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:10.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:06:10.360 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:10.400 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:10.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:10.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:10.401 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:06:09 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:06:10.447 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T14:06:10.502 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:10.538 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:10.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:10.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:10.640 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:10.677 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:10.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:10.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:10.779 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:06:10.819 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:10.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:10.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:06:10.861 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph tell osd.1 reset_purged_snaps_last 2024-09-21T14:06:10.861 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:06:10 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:06:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:06:10.872 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:06:10.921 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:06:10.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:10.962 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:10.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:06:11.063 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:06:11.100 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:11.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:11.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:06:11.202 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:06:11.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:11.235 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:11.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:06:11.336 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:06:11.371 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:11 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:06:11.371 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:11 smithi187 ceph-mon[33032]: pgmap v875: 168 pgs: 168 active+clean; 1.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.20k op/s 2024-09-21T14:06:11.371 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:11 smithi187 ceph-mon[33032]: osdmap e446: 8 total, 8 up, 8 in 2024-09-21T14:06:11.373 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:11.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:11.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:06:11.404 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T14:06:11.436 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:11 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:06:11.436 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:11 smithi049 ceph-mon[30080]: pgmap v875: 168 pgs: 168 active+clean; 1.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.20k op/s 2024-09-21T14:06:11.436 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:11 smithi049 ceph-mon[30080]: osdmap e446: 8 total, 8 up, 8 in 2024-09-21T14:06:11.436 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:11 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:06:11.436 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:11 smithi049 ceph-mon[40489]: pgmap v875: 168 pgs: 168 active+clean; 1.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.20k op/s 2024-09-21T14:06:11.437 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:11 smithi049 ceph-mon[40489]: osdmap e446: 8 total, 8 up, 8 in 2024-09-21T14:06:11.475 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:06:11.508 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:11.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:11.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:06:11.539 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T14:06:11.539 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-21T14:06:11.539 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-21T14:06:11.539 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-21T14:06:11.539 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T14:06:11.544 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:06:11.610 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:11.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:11.652 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:11.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:11.753 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:06:11.791 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:11.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:11.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:06:11.893 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:11.940 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:11.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:11.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:12.041 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:06:12.078 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:12.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:12.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:06:12.180 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:12.207 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-09-21T14:06:12.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:12.234 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:12.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:12.320 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T14:06:12.334 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:12.366 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:12.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:12.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:12.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:12 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/134983576' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:06:12.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:12 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:06:12.453 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:12 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/134983576' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:06:12.453 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:12 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:06:12.453 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:12 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/134983576' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:06:12.454 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:12 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:06:12.468 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:06:12.499 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:12.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:12.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:06:12.601 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:12.635 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:12.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:12.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:12.736 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:06:12.760 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:06:12.787 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:12.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:12.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:06:12.889 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:12.930 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:12.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:12.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:13.032 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:06:13.077 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:13.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:13.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:06:13.179 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:13.232 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:13.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:13.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:13.334 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:06:13.368 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:13.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:13.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:06:13.438 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T14:06:13.439 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-21T14:06:13.439 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-21T14:06:13.439 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-21T14:06:13.439 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.439 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187592 9.5 9:a5cf25e0:::benchmark_data_smithi049_138504_object66184:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:06:13.439 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:41.268428+0000", 2024-09-21T14:06:13.439 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 272.16622650400001, 2024-09-21T14:06:13.439 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3375785200000001, 2024-09-21T14:06:13.439 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:06:13.439 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:06:13.439 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:13.439 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:06:13.440 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:06:13.440 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:06:13.440 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187592 2024-09-21T14:06:13.440 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.440 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:06:13.440 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.440 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:06:13.440 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268428+0000", 2024-09-21T14:06:13.440 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:13.440 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.440 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.441 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:06:13.441 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268428+0000", 2024-09-21T14:06:13.441 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:13.441 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.441 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.441 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:06:13.441 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268429+0000", 2024-09-21T14:06:13.441 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.466e-06 2024-09-21T14:06:13.441 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.441 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.441 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:06:13.441 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268435+0000", 2024-09-21T14:06:13.441 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5539999999999997e-06 2024-09-21T14:06:13.441 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.442 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.442 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:06:13.442 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268445+0000", 2024-09-21T14:06:13.442 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0366999999999999e-05 2024-09-21T14:06:13.442 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.442 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.442 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:06:13.442 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268452+0000", 2024-09-21T14:06:13.442 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8870000000000003e-06 2024-09-21T14:06:13.442 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.442 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.442 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:13.442 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.289931+0000", 2024-09-21T14:06:13.442 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.021479166000000001 2024-09-21T14:06:13.443 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.443 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.443 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:06:13.443 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.289937+0000", 2024-09-21T14:06:13.443 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4140000000000002e-06 2024-09-21T14:06:13.443 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.443 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.443 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:13.443 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.566606+0000", 2024-09-21T14:06:13.443 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2766695319999997 2024-09-21T14:06:13.444 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.444 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.444 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:06:13.444 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.569571+0000", 2024-09-21T14:06:13.444 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0029643489999999998 2024-09-21T14:06:13.444 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.444 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.444 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:06:13.444 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.573938+0000", 2024-09-21T14:06:13.444 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0043675099999999998 2024-09-21T14:06:13.444 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.444 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.444 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:06:13.445 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.589876+0000", 2024-09-21T14:06:13.445 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.015938249000000002 2024-09-21T14:06:13.445 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.445 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.445 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:13.445 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.597274+0000", 2024-09-21T14:06:13.445 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0073979509999999998 2024-09-21T14:06:13.445 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.445 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.445 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:13.445 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.598324+0000", 2024-09-21T14:06:13.445 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001050156 2024-09-21T14:06:13.445 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.445 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.446 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:06:13.446 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.599807+0000", 2024-09-21T14:06:13.446 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0014825439999999999 2024-09-21T14:06:13.446 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.446 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.446 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:06:13.446 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.606006+0000", 2024-09-21T14:06:13.446 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.006199375 2024-09-21T14:06:13.446 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:13.446 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:06:13.446 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:13.446 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.446 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.447 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187594 9.5 9:a4584f0e:::benchmark_data_smithi049_138504_object66186:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:06:13.447 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:41.268527+0000", 2024-09-21T14:06:13.447 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 272.16612698900002, 2024-09-21T14:06:13.447 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3614060480000001, 2024-09-21T14:06:13.447 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:06:13.447 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:06:13.447 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:13.447 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:06:13.447 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:06:13.447 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:06:13.447 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187594 2024-09-21T14:06:13.447 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.447 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:06:13.447 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.448 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:06:13.448 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268527+0000", 2024-09-21T14:06:13.448 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:13.448 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.448 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.448 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:06:13.448 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268527+0000", 2024-09-21T14:06:13.448 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:13.448 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.448 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.448 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:06:13.449 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268528+0000", 2024-09-21T14:06:13.449 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5299999999999999e-07 2024-09-21T14:06:13.449 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.449 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.449 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:06:13.449 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268531+0000", 2024-09-21T14:06:13.449 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.157e-06 2024-09-21T14:06:13.449 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.449 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.449 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:06:13.449 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268538+0000", 2024-09-21T14:06:13.449 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9290000000000001e-06 2024-09-21T14:06:13.449 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.449 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.450 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:06:13.450 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268543+0000", 2024-09-21T14:06:13.450 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0409999999999997e-06 2024-09-21T14:06:13.450 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.450 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.450 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:13.450 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.289953+0000", 2024-09-21T14:06:13.450 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.021409811000000001 2024-09-21T14:06:13.450 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.450 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.450 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:06:13.450 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.289957+0000", 2024-09-21T14:06:13.450 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9489999999999998e-06 2024-09-21T14:06:13.450 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.451 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.451 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:13.451 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.606351+0000", 2024-09-21T14:06:13.451 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3163947599999997 2024-09-21T14:06:13.451 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.451 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.451 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:06:13.451 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.610368+0000", 2024-09-21T14:06:13.451 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0040163289999999999 2024-09-21T14:06:13.451 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.451 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.451 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:06:13.451 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.615560+0000", 2024-09-21T14:06:13.451 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0051918939999999998 2024-09-21T14:06:13.452 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.452 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.452 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:13.452 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.622425+0000", 2024-09-21T14:06:13.452 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0068655959999999999 2024-09-21T14:06:13.452 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.452 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.452 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:13.452 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.623509+0000", 2024-09-21T14:06:13.452 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001083669 2024-09-21T14:06:13.452 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.452 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.452 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:06:13.452 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.624009+0000", 2024-09-21T14:06:13.453 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00050001600000000005 2024-09-21T14:06:13.453 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.453 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.453 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:06:13.453 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.625282+0000", 2024-09-21T14:06:13.453 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0012730599999999999 2024-09-21T14:06:13.453 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.453 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.453 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:06:13.453 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.629933+0000", 2024-09-21T14:06:13.453 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0046513839999999997 2024-09-21T14:06:13.453 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:13.453 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:06:13.453 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:13.454 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.454 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.454 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187607 9.7 9:ea996e28:::benchmark_data_smithi049_138504_object66199:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:06:13.454 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:41.286116+0000", 2024-09-21T14:06:13.454 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 272.14853833900003, 2024-09-21T14:06:13.454 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0827087349999998, 2024-09-21T14:06:13.454 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:06:13.454 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:06:13.454 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:13.454 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:06:13.454 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:06:13.454 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:06:13.454 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187607 2024-09-21T14:06:13.454 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.455 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:06:13.455 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.455 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:06:13.455 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.286116+0000", 2024-09-21T14:06:13.455 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:13.455 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.455 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.455 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:06:13.455 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.286116+0000", 2024-09-21T14:06:13.455 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:13.455 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.455 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.455 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:06:13.455 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.286117+0000", 2024-09-21T14:06:13.456 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.097e-06 2024-09-21T14:06:13.456 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.456 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.456 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:06:13.456 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.286120+0000", 2024-09-21T14:06:13.456 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.224e-06 2024-09-21T14:06:13.456 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.456 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.456 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:06:13.456 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.286129+0000", 2024-09-21T14:06:13.456 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3659999999999992e-06 2024-09-21T14:06:13.456 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.456 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.456 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:06:13.457 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.286134+0000", 2024-09-21T14:06:13.457 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6130000000000003e-06 2024-09-21T14:06:13.457 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.457 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.457 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:13.457 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.286627+0000", 2024-09-21T14:06:13.457 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00049266000000000002 2024-09-21T14:06:13.457 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.457 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.457 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:06:13.457 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.286663+0000", 2024-09-21T14:06:13.457 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6164999999999997e-05 2024-09-21T14:06:13.457 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.457 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.458 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:13.458 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.330333+0000", 2024-09-21T14:06:13.458 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0436700029999999 2024-09-21T14:06:13.458 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.458 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.458 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:06:13.458 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.333462+0000", 2024-09-21T14:06:13.458 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003128881 2024-09-21T14:06:13.458 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.458 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.458 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,7", 2024-09-21T14:06:13.458 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.337950+0000", 2024-09-21T14:06:13.458 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0044878089999999997 2024-09-21T14:06:13.459 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.459 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.459 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:06:13.459 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353871+0000", 2024-09-21T14:06:13.459 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.015921270000000001 2024-09-21T14:06:13.459 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.459 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.459 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:13.459 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.360705+0000", 2024-09-21T14:06:13.459 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0068337110000000001 2024-09-21T14:06:13.459 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.459 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.459 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:13.460 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.362017+0000", 2024-09-21T14:06:13.460 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0013119749999999999 2024-09-21T14:06:13.460 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.460 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.460 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:06:13.460 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.364166+0000", 2024-09-21T14:06:13.460 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0021495030000000001 2024-09-21T14:06:13.460 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.461 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.461 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:06:13.461 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.368825+0000", 2024-09-21T14:06:13.461 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0046584579999999999 2024-09-21T14:06:13.461 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:13.461 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:06:13.461 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:13.461 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.461 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.462 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187609 9.5 9:a0a81d79:::benchmark_data_smithi049_138504_object66201:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:06:13.462 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:41.287003+0000", 2024-09-21T14:06:13.462 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 272.14765172800003, 2024-09-21T14:06:13.462 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3857519199999997, 2024-09-21T14:06:13.462 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:06:13.462 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:06:13.462 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:13.462 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:06:13.463 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:06:13.463 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:06:13.463 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187609 2024-09-21T14:06:13.463 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.463 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:06:13.463 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.463 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:06:13.463 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.287003+0000", 2024-09-21T14:06:13.463 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:13.464 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.464 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.464 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:06:13.464 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.287003+0000", 2024-09-21T14:06:13.464 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:13.464 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.464 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.464 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:06:13.465 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.287003+0000", 2024-09-21T14:06:13.465 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8400000000000003e-07 2024-09-21T14:06:13.465 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.465 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.465 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:06:13.465 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.287017+0000", 2024-09-21T14:06:13.465 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3831e-05 2024-09-21T14:06:13.465 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.465 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.466 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:06:13.466 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.287026+0000", 2024-09-21T14:06:13.466 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8389999999999996e-06 2024-09-21T14:06:13.466 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.466 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.466 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:06:13.466 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.287032+0000", 2024-09-21T14:06:13.466 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3509999999999998e-06 2024-09-21T14:06:13.466 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.467 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.467 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:13.467 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.289969+0000", 2024-09-21T14:06:13.467 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0029364399999999998 2024-09-21T14:06:13.467 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.467 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.467 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:06:13.467 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.289973+0000", 2024-09-21T14:06:13.468 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6339999999999999e-06 2024-09-21T14:06:13.468 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.468 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.468 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:13.468 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.631003+0000", 2024-09-21T14:06:13.468 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3410304320000002 2024-09-21T14:06:13.468 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.468 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.468 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:06:13.469 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.634397+0000", 2024-09-21T14:06:13.469 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0033938369999999998 2024-09-21T14:06:13.469 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.469 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.469 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:06:13.469 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.639017+0000", 2024-09-21T14:06:13.469 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0046205889999999996 2024-09-21T14:06:13.469 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.469 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.470 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:06:13.470 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.662137+0000", 2024-09-21T14:06:13.470 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.023119475 2024-09-21T14:06:13.470 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.470 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.470 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:13.470 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.663533+0000", 2024-09-21T14:06:13.471 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0013956159999999999 2024-09-21T14:06:13.471 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.471 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.471 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:13.471 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.664730+0000", 2024-09-21T14:06:13.471 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0011970139999999999 2024-09-21T14:06:13.471 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.471 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.471 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:06:13.472 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:06:13.473 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:13 smithi187 ceph-mon[33032]: pgmap v877: 168 pgs: 168 active+clean; 1.7 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.11k op/s 2024-09-21T14:06:13.473 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:13 smithi187 ceph-mon[33032]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:06:13.473 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:13 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T14:06:13.473 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:13 smithi187 ceph-mon[33032]: osdmap e447: 8 total, 8 up, 8 in 2024-09-21T14:06:13.473 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:13 smithi187 ceph-mon[33032]: 6.a scrub starts 2024-09-21T14:06:13.473 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:13 smithi187 ceph-mon[33032]: 6.a scrub ok 2024-09-21T14:06:13.474 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.666523+0000", 2024-09-21T14:06:13.474 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001793046 2024-09-21T14:06:13.474 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.474 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.474 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:06:13.474 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.672754+0000", 2024-09-21T14:06:13.474 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0062319319999999999 2024-09-21T14:06:13.475 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:13.475 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:06:13.475 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:13.475 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.475 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.475 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187621 9.5 9:a3e366d7:::benchmark_data_smithi049_138504_object66213:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:06:13.475 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:41.300495+0000", 2024-09-21T14:06:13.475 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 272.13415908000002, 2024-09-21T14:06:13.475 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3801383449999998, 2024-09-21T14:06:13.475 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:06:13.475 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:06:13.475 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:13.476 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:06:13.476 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:06:13.476 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:06:13.476 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187621 2024-09-21T14:06:13.476 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.476 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:06:13.476 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.476 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:06:13.476 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300495+0000", 2024-09-21T14:06:13.476 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:13.476 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.476 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.477 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:06:13.477 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300495+0000", 2024-09-21T14:06:13.477 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:13.477 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.477 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.477 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:06:13.477 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300497+0000", 2024-09-21T14:06:13.477 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3710000000000001e-06 2024-09-21T14:06:13.477 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.477 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.477 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:06:13.477 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300502+0000", 2024-09-21T14:06:13.477 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3569999999999997e-06 2024-09-21T14:06:13.477 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.478 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.478 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:06:13.478 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300511+0000", 2024-09-21T14:06:13.478 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.9509999999999995e-06 2024-09-21T14:06:13.478 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.478 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.478 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:06:13.478 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300517+0000", 2024-09-21T14:06:13.478 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2489999999999998e-06 2024-09-21T14:06:13.478 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.478 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.478 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:13.479 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300606+0000", 2024-09-21T14:06:13.479 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8794000000000004e-05 2024-09-21T14:06:13.479 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.479 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.479 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:06:13.479 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300616+0000", 2024-09-21T14:06:13.479 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0338e-05 2024-09-21T14:06:13.479 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.480 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.480 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:13.480 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.643702+0000", 2024-09-21T14:06:13.480 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.343085759 2024-09-21T14:06:13.480 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.480 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.480 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:06:13.480 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.648030+0000", 2024-09-21T14:06:13.480 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0043275859999999996 2024-09-21T14:06:13.481 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.481 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.481 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:06:13.481 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.653318+0000", 2024-09-21T14:06:13.481 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.005287936 2024-09-21T14:06:13.481 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.482 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.482 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:06:13.482 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.662422+0000", 2024-09-21T14:06:13.482 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0091040340000000004 2024-09-21T14:06:13.482 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.482 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.482 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:13.482 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.673842+0000", 2024-09-21T14:06:13.483 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.011420382999999999 2024-09-21T14:06:13.483 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.483 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.483 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:13.483 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.675052+0000", 2024-09-21T14:06:13.483 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0012104290000000001 2024-09-21T14:06:13.483 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.483 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.483 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:06:13.484 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.676489+0000", 2024-09-21T14:06:13.484 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0014364309999999999 2024-09-21T14:06:13.484 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.484 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.484 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:06:13.484 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.680634+0000", 2024-09-21T14:06:13.484 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.004144727 2024-09-21T14:06:13.484 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:13.484 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:06:13.484 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:13.484 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.484 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.485 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187623 9.7 9:e79a683b:::benchmark_data_smithi049_138504_object66215:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:06:13.485 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:41.300818+0000", 2024-09-21T14:06:13.485 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 272.13383667599999, 2024-09-21T14:06:13.485 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1014428919999997, 2024-09-21T14:06:13.485 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:06:13.485 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:06:13.485 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:13.485 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:06:13.485 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:06:13.485 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:06:13.485 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187623 2024-09-21T14:06:13.486 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.486 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:06:13.486 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.486 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:06:13.486 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300818+0000", 2024-09-21T14:06:13.486 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:13.486 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.486 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.486 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:06:13.486 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300818+0000", 2024-09-21T14:06:13.486 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:13.487 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.487 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.487 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:06:13.487 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300818+0000", 2024-09-21T14:06:13.487 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.54e-07 2024-09-21T14:06:13.487 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.487 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.487 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:06:13.487 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300822+0000", 2024-09-21T14:06:13.487 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6770000000000001e-06 2024-09-21T14:06:13.487 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.487 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.488 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:06:13.488 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300832+0000", 2024-09-21T14:06:13.488 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0202e-05 2024-09-21T14:06:13.488 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.488 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.488 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:06:13.488 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300847+0000", 2024-09-21T14:06:13.488 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4992e-05 2024-09-21T14:06:13.488 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.488 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.488 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:13.488 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300931+0000", 2024-09-21T14:06:13.489 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3577000000000003e-05 2024-09-21T14:06:13.489 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.489 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.489 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:06:13.489 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300940+0000", 2024-09-21T14:06:13.489 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8009999999999993e-06 2024-09-21T14:06:13.489 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.489 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.489 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:13.489 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.370531+0000", 2024-09-21T14:06:13.489 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0695910870000001 2024-09-21T14:06:13.489 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.489 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.490 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:06:13.490 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.373842+0000", 2024-09-21T14:06:13.490 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0033110259999999999 2024-09-21T14:06:13.490 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.490 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.490 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,7", 2024-09-21T14:06:13.490 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.378337+0000", 2024-09-21T14:06:13.490 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0044951100000000001 2024-09-21T14:06:13.490 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.490 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.490 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:13.490 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.394514+0000", 2024-09-21T14:06:13.490 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.016176877999999999 2024-09-21T14:06:13.491 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.491 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.491 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:06:13.491 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.395030+0000", 2024-09-21T14:06:13.491 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00051615000000000005 2024-09-21T14:06:13.491 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.491 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.491 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:13.491 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.395803+0000", 2024-09-21T14:06:13.492 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00077358799999999997 2024-09-21T14:06:13.492 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.492 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.492 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:06:13.492 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.397305+0000", 2024-09-21T14:06:13.492 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001501811 2024-09-21T14:06:13.492 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.492 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.492 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:06:13.492 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.402261+0000", 2024-09-21T14:06:13.493 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0049552390000000002 2024-09-21T14:06:13.493 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:13.493 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:06:13.493 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:13.493 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.493 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.493 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187624 9.5 9:a181e4d0:::benchmark_data_smithi049_138504_object66216:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:06:13.493 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:41.300871+0000", 2024-09-21T14:06:13.493 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 272.13378367299998, 2024-09-21T14:06:13.493 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4164464030000001, 2024-09-21T14:06:13.493 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:06:13.494 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:06:13.494 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:13.494 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:06:13.494 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:06:13.494 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:06:13.494 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187624 2024-09-21T14:06:13.494 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.494 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:06:13.494 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.494 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:06:13.495 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300871+0000", 2024-09-21T14:06:13.495 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:13.495 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.495 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.495 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:06:13.495 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300871+0000", 2024-09-21T14:06:13.495 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:13.495 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.495 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.495 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:06:13.495 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300871+0000", 2024-09-21T14:06:13.496 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8599999999999998e-07 2024-09-21T14:06:13.496 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.496 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.496 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:06:13.496 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300875+0000", 2024-09-21T14:06:13.496 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7720000000000001e-06 2024-09-21T14:06:13.496 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.496 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.496 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:06:13.496 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300886+0000", 2024-09-21T14:06:13.496 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1058e-05 2024-09-21T14:06:13.497 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.497 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.497 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:06:13.497 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300892+0000", 2024-09-21T14:06:13.497 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.161e-06 2024-09-21T14:06:13.497 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.497 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.497 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:13.497 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.301287+0000", 2024-09-21T14:06:13.498 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00039522800000000002 2024-09-21T14:06:13.498 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.498 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.498 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:06:13.498 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.301297+0000", 2024-09-21T14:06:13.498 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6209999999999999e-06 2024-09-21T14:06:13.498 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.498 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.498 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:13.499 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.681021+0000", 2024-09-21T14:06:13.499 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3797241769999999 2024-09-21T14:06:13.499 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.499 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.499 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:06:13.499 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.685054+0000", 2024-09-21T14:06:13.499 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0040323909999999998 2024-09-21T14:06:13.499 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.499 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.500 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:06:13.500 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.689479+0000", 2024-09-21T14:06:13.500 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0044253740000000001 2024-09-21T14:06:13.500 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.500 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.500 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:13.500 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.710060+0000", 2024-09-21T14:06:13.500 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.020581051999999999 2024-09-21T14:06:13.500 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.500 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.501 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:06:13.501 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.710641+0000", 2024-09-21T14:06:13.501 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00058055800000000001 2024-09-21T14:06:13.501 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.501 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.501 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:13.501 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.711399+0000", 2024-09-21T14:06:13.501 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00075836299999999996 2024-09-21T14:06:13.501 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.501 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.501 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:06:13.501 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.712857+0000", 2024-09-21T14:06:13.502 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001458373 2024-09-21T14:06:13.502 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.502 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.502 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:06:13.502 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.717317+0000", 2024-09-21T14:06:13.502 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0044596890000000002 2024-09-21T14:06:13.502 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:13.502 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:06:13.502 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:13.502 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.502 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.503 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187626 9.7 9:e3df99cb:::benchmark_data_smithi049_138504_object66218:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:06:13.503 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:41.304965+0000", 2024-09-21T14:06:13.503 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 272.12968953699999, 2024-09-21T14:06:13.503 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1061633610000001, 2024-09-21T14:06:13.503 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:06:13.503 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:06:13.503 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:13.503 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:06:13.503 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:06:13.504 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:06:13.504 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187626 2024-09-21T14:06:13.504 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.504 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:06:13.504 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.504 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:06:13.504 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.304965+0000", 2024-09-21T14:06:13.504 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:13.504 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.504 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.504 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:06:13.505 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.304965+0000", 2024-09-21T14:06:13.505 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:13.505 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.505 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.505 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:06:13.505 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.304966+0000", 2024-09-21T14:06:13.505 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6619999999999999e-06 2024-09-21T14:06:13.505 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.505 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.505 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:06:13.505 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.304973+0000", 2024-09-21T14:06:13.505 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1759999999999998e-06 2024-09-21T14:06:13.505 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.506 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.506 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:06:13.506 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.304987+0000", 2024-09-21T14:06:13.506 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4372999999999999e-05 2024-09-21T14:06:13.506 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.506 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.506 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:06:13.506 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.304995+0000", 2024-09-21T14:06:13.506 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4109999999999996e-06 2024-09-21T14:06:13.506 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.506 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.506 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:13.507 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.305067+0000", 2024-09-21T14:06:13.507 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1833000000000002e-05 2024-09-21T14:06:13.507 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.507 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.507 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:06:13.507 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.305077+0000", 2024-09-21T14:06:13.507 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0161e-05 2024-09-21T14:06:13.507 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.507 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.507 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:13.507 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.382147+0000", 2024-09-21T14:06:13.507 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0770691709999998 2024-09-21T14:06:13.507 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.508 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.508 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:06:13.508 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.385209+0000", 2024-09-21T14:06:13.508 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0030624139999999998 2024-09-21T14:06:13.508 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.508 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.508 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,7", 2024-09-21T14:06:13.508 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.389597+0000", 2024-09-21T14:06:13.508 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.004388477 2024-09-21T14:06:13.508 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.508 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.508 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:13.509 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.403080+0000", 2024-09-21T14:06:13.509 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.013482984 2024-09-21T14:06:13.509 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.509 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.509 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:06:13.509 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.403583+0000", 2024-09-21T14:06:13.509 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00050300099999999999 2024-09-21T14:06:13.509 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.509 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.509 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:13.509 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.404464+0000", 2024-09-21T14:06:13.510 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00088034599999999995 2024-09-21T14:06:13.510 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.510 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.510 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:06:13.510 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.406114+0000", 2024-09-21T14:06:13.510 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0016504499999999999 2024-09-21T14:06:13.510 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.510 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.510 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:06:13.510 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.411128+0000", 2024-09-21T14:06:13.511 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0050139019999999998 2024-09-21T14:06:13.511 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:13.511 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:06:13.511 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:13.511 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.511 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.511 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187630 9.5 9:afb0e788:::benchmark_data_smithi049_138504_object66222:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:06:13.511 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:41.309934+0000", 2024-09-21T14:06:13.511 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 272.12471995599998, 2024-09-21T14:06:13.511 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4164949509999998, 2024-09-21T14:06:13.512 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:06:13.512 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:06:13.512 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:13.512 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:06:13.512 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:06:13.512 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:06:13.512 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187630 2024-09-21T14:06:13.512 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.512 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:06:13.512 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.513 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:06:13.513 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.309934+0000", 2024-09-21T14:06:13.513 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:13.513 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.513 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.513 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:06:13.513 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.309934+0000", 2024-09-21T14:06:13.513 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:13.513 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.513 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.513 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:06:13.513 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.309936+0000", 2024-09-21T14:06:13.514 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7460000000000001e-06 2024-09-21T14:06:13.514 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.514 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.514 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:06:13.514 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.309943+0000", 2024-09-21T14:06:13.514 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6579999999999996e-06 2024-09-21T14:06:13.514 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.514 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.514 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:06:13.514 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.309957+0000", 2024-09-21T14:06:13.514 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4496e-05 2024-09-21T14:06:13.515 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.515 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.515 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:06:13.515 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.309964+0000", 2024-09-21T14:06:13.515 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2219999999999996e-06 2024-09-21T14:06:13.515 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.515 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.515 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:13.515 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.310043+0000", 2024-09-21T14:06:13.515 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8801999999999995e-05 2024-09-21T14:06:13.515 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.516 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.516 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:06:13.516 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.310055+0000", 2024-09-21T14:06:13.516 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1637999999999999e-05 2024-09-21T14:06:13.516 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.516 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.516 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:13.516 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.693253+0000", 2024-09-21T14:06:13.516 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3831976189999997 2024-09-21T14:06:13.516 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.516 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.517 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:06:13.517 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.697465+0000", 2024-09-21T14:06:13.517 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0042119770000000004 2024-09-21T14:06:13.517 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.517 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.517 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:06:13.517 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.703657+0000", 2024-09-21T14:06:13.517 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0061928870000000002 2024-09-21T14:06:13.517 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.517 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.517 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:06:13.517 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.717616+0000", 2024-09-21T14:06:13.517 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.013958435999999999 2024-09-21T14:06:13.518 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.518 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.518 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:13.518 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.718555+0000", 2024-09-21T14:06:13.518 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00093888999999999995 2024-09-21T14:06:13.518 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.518 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.518 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:13.518 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.719633+0000", 2024-09-21T14:06:13.518 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0010787150000000001 2024-09-21T14:06:13.518 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.518 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.519 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:06:13.519 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.721631+0000", 2024-09-21T14:06:13.519 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0019973220000000002 2024-09-21T14:06:13.519 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.519 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.519 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:06:13.519 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.726429+0000", 2024-09-21T14:06:13.519 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0047985429999999997 2024-09-21T14:06:13.519 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:13.519 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:06:13.519 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:13.519 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.519 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.520 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187631 9.7 9:eb4378cd:::benchmark_data_smithi049_138504_object66223:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:06:13.520 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:41.310203+0000", 2024-09-21T14:06:13.520 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 272.124451448, 2024-09-21T14:06:13.520 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1434362220000001, 2024-09-21T14:06:13.520 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:06:13.520 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:06:13.520 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:13.520 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:06:13.520 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:06:13.520 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:06:13.520 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187631 2024-09-21T14:06:13.521 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.521 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:06:13.521 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.521 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:06:13.521 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.310203+0000", 2024-09-21T14:06:13.521 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:13.521 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.521 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.521 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:06:13.521 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.310203+0000", 2024-09-21T14:06:13.521 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:13.521 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.522 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.522 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:06:13.522 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.310204+0000", 2024-09-21T14:06:13.522 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3699999999999999e-07 2024-09-21T14:06:13.522 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.522 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.522 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:06:13.522 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.310209+0000", 2024-09-21T14:06:13.522 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7039999999999999e-06 2024-09-21T14:06:13.522 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.522 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.523 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:06:13.523 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.310219+0000", 2024-09-21T14:06:13.523 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7319999999999993e-06 2024-09-21T14:06:13.523 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.523 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.523 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:06:13.523 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.310226+0000", 2024-09-21T14:06:13.523 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4239999999999998e-06 2024-09-21T14:06:13.523 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.523 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.523 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:13.523 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.310279+0000", 2024-09-21T14:06:13.523 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3124999999999997e-05 2024-09-21T14:06:13.524 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.524 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.524 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:06:13.524 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.310288+0000", 2024-09-21T14:06:13.524 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3149999999999998e-06 2024-09-21T14:06:13.524 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.524 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.524 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:13.524 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.416040+0000", 2024-09-21T14:06:13.524 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1057518990000004 2024-09-21T14:06:13.524 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.524 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.525 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:06:13.525 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.419285+0000", 2024-09-21T14:06:13.525 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0032447399999999999 2024-09-21T14:06:13.525 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.525 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.525 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,7", 2024-09-21T14:06:13.525 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.423815+0000", 2024-09-21T14:06:13.525 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0045301509999999996 2024-09-21T14:06:13.525 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.525 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.525 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:13.525 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.439415+0000", 2024-09-21T14:06:13.525 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.015600334 2024-09-21T14:06:13.526 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.526 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:13 smithi049 ceph-mon[40489]: pgmap v877: 168 pgs: 168 active+clean; 1.7 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.11k op/s 2024-09-21T14:06:13.527 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:13 smithi049 ceph-mon[40489]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:06:13.527 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:13 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T14:06:13.527 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:13 smithi049 ceph-mon[40489]: osdmap e447: 8 total, 8 up, 8 in 2024-09-21T14:06:13.527 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:13 smithi049 ceph-mon[40489]: 6.a scrub starts 2024-09-21T14:06:13.527 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:13 smithi049 ceph-mon[40489]: 6.a scrub ok 2024-09-21T14:06:13.527 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:13 smithi049 ceph-mon[30080]: pgmap v877: 168 pgs: 168 active+clean; 1.7 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.11k op/s 2024-09-21T14:06:13.527 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:13 smithi049 ceph-mon[30080]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:06:13.528 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:13 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T14:06:13.528 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:13 smithi049 ceph-mon[30080]: osdmap e447: 8 total, 8 up, 8 in 2024-09-21T14:06:13.528 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:13 smithi049 ceph-mon[30080]: 6.a scrub starts 2024-09-21T14:06:13.528 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:13 smithi049 ceph-mon[30080]: 6.a scrub ok 2024-09-21T14:06:13.528 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.529 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:13.529 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.440566+0000", 2024-09-21T14:06:13.529 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0011505129999999999 2024-09-21T14:06:13.529 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.529 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.529 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:06:13.529 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.447676+0000", 2024-09-21T14:06:13.529 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0071103659999999999 2024-09-21T14:06:13.529 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.529 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.530 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:06:13.530 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.448927+0000", 2024-09-21T14:06:13.530 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0012508759999999999 2024-09-21T14:06:13.530 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.530 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.530 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:06:13.530 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.453639+0000", 2024-09-21T14:06:13.530 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0047121059999999998 2024-09-21T14:06:13.530 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:13.530 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:06:13.530 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:13.531 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.531 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.531 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187650 9.5 9:ab0f6d84:::benchmark_data_smithi049_138504_object66242:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:06:13.531 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:41.359224+0000", 2024-09-21T14:06:13.531 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 272.07542994699998, 2024-09-21T14:06:13.531 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4044763060000003, 2024-09-21T14:06:13.531 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:06:13.531 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:06:13.531 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:13.531 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:06:13.531 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:06:13.531 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:06:13.532 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187650 2024-09-21T14:06:13.532 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.532 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:06:13.532 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.532 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:06:13.532 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.359224+0000", 2024-09-21T14:06:13.532 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:13.532 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.532 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.532 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:06:13.532 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.359224+0000", 2024-09-21T14:06:13.533 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:13.533 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.533 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.533 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:06:13.533 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.359226+0000", 2024-09-21T14:06:13.533 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.423e-06 2024-09-21T14:06:13.533 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.533 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.533 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:06:13.533 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.359231+0000", 2024-09-21T14:06:13.533 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0980000000000001e-06 2024-09-21T14:06:13.534 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.534 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.534 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:06:13.534 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.359243+0000", 2024-09-21T14:06:13.534 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2316e-05 2024-09-21T14:06:13.534 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.534 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.534 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:06:13.534 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.359250+0000", 2024-09-21T14:06:13.534 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8820000000000003e-06 2024-09-21T14:06:13.534 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.535 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.535 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:13.535 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.359329+0000", 2024-09-21T14:06:13.535 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8541000000000006e-05 2024-09-21T14:06:13.535 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.535 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.535 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:06:13.535 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.359341+0000", 2024-09-21T14:06:13.535 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2238e-05 2024-09-21T14:06:13.535 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.535 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.536 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:13.536 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.726850+0000", 2024-09-21T14:06:13.536 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3675093619999998 2024-09-21T14:06:13.536 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.536 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.536 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:06:13.536 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.731153+0000", 2024-09-21T14:06:13.536 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0043027860000000003 2024-09-21T14:06:13.536 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.536 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.536 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:06:13.537 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.737448+0000", 2024-09-21T14:06:13.537 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0062951559999999997 2024-09-21T14:06:13.537 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.537 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.537 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:13.537 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.754776+0000", 2024-09-21T14:06:13.537 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.017328300000000001 2024-09-21T14:06:13.537 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.537 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.537 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:13.537 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.756004+0000", 2024-09-21T14:06:13.537 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001227337 2024-09-21T14:06:13.537 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.538 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.538 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:06:13.538 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.757990+0000", 2024-09-21T14:06:13.538 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0019857809999999998 2024-09-21T14:06:13.538 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.538 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.538 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:06:13.538 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.759321+0000", 2024-09-21T14:06:13.538 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001331203 2024-09-21T14:06:13.538 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.538 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.539 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:06:13.539 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.763701+0000", 2024-09-21T14:06:13.539 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0043798830000000002 2024-09-21T14:06:13.539 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:13.539 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:06:13.539 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:13.539 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.539 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.539 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187651 9.f 9:fed2c87a:::benchmark_data_smithi049_138504_object66243:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e347)", 2024-09-21T14:06:13.539 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:42.222057+0000", 2024-09-21T14:06:13.539 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 271.21259681800001, 2024-09-21T14:06:13.539 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1315866940000001, 2024-09-21T14:06:13.539 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:06:13.540 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:06:13.540 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:13.540 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:06:13.540 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:06:13.540 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:06:13.540 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187651 2024-09-21T14:06:13.540 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.540 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:06:13.540 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.540 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:06:13.540 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.222057+0000", 2024-09-21T14:06:13.541 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:13.541 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.541 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.541 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:06:13.541 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.222057+0000", 2024-09-21T14:06:13.541 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:13.541 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.541 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.541 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:06:13.541 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.222059+0000", 2024-09-21T14:06:13.541 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8029999999999999e-06 2024-09-21T14:06:13.541 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.541 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.542 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:06:13.542 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.222066+0000", 2024-09-21T14:06:13.542 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2670000000000003e-06 2024-09-21T14:06:13.542 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.542 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.542 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:06:13.542 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.222078+0000", 2024-09-21T14:06:13.542 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2899e-05 2024-09-21T14:06:13.542 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.542 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.542 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:06:13.543 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.222086+0000", 2024-09-21T14:06:13.543 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8080000000000005e-06 2024-09-21T14:06:13.543 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.543 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.543 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:13.543 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.233188+0000", 2024-09-21T14:06:13.543 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0111019990000001 2024-09-21T14:06:13.543 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.543 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.543 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:06:13.543 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.233317+0000", 2024-09-21T14:06:13.544 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012857999999999999 2024-09-21T14:06:13.544 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.544 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.544 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3", 2024-09-21T14:06:13.544 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.233386+0000", 2024-09-21T14:06:13.544 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8901999999999999e-05 2024-09-21T14:06:13.544 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.544 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.544 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:06:13.544 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.342687+0000", 2024-09-21T14:06:13.544 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.109301519 2024-09-21T14:06:13.544 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.545 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.545 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:13.545 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.342738+0000", 2024-09-21T14:06:13.545 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0756e-05 2024-09-21T14:06:13.545 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.545 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.545 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:13.545 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353571+0000", 2024-09-21T14:06:13.545 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.01083253 2024-09-21T14:06:13.545 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.546 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.546 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:06:13.546 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353614+0000", 2024-09-21T14:06:13.546 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3129e-05 2024-09-21T14:06:13.546 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.546 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.546 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:06:13.546 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353644+0000", 2024-09-21T14:06:13.546 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0502e-05 2024-09-21T14:06:13.546 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:13.546 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:06:13.547 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:13.547 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.547 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.547 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187654 9.5 9:a73ea609:::benchmark_data_smithi049_138504_object66246:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e347)", 2024-09-21T14:06:13.547 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:42.234021+0000", 2024-09-21T14:06:13.547 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 271.20063296299998, 2024-09-21T14:06:13.547 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5379708970000001, 2024-09-21T14:06:13.547 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:06:13.547 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:06:13.547 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:13.547 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:06:13.548 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:06:13.548 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:06:13.548 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187654 2024-09-21T14:06:13.548 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.548 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:06:13.548 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.548 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:06:13.548 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.234021+0000", 2024-09-21T14:06:13.548 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:13.548 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.548 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.548 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:06:13.549 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.234021+0000", 2024-09-21T14:06:13.549 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:13.549 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.549 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.549 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:06:13.549 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.234023+0000", 2024-09-21T14:06:13.549 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5460000000000001e-06 2024-09-21T14:06:13.549 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.549 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.550 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:06:13.550 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.234029+0000", 2024-09-21T14:06:13.550 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8069999999999996e-06 2024-09-21T14:06:13.550 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.550 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.550 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:06:13.550 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.234041+0000", 2024-09-21T14:06:13.550 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2189e-05 2024-09-21T14:06:13.550 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.550 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.551 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:06:13.551 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.234052+0000", 2024-09-21T14:06:13.551 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0713999999999999e-05 2024-09-21T14:06:13.551 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.551 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.551 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:13.551 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.743160+0000", 2024-09-21T14:06:13.551 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5091079629999999 2024-09-21T14:06:13.551 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.552 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.552 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:06:13.552 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.746100+0000", 2024-09-21T14:06:13.552 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0029400559999999999 2024-09-21T14:06:13.552 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.552 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.552 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:06:13.552 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.750393+0000", 2024-09-21T14:06:13.552 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0042938280000000004 2024-09-21T14:06:13.553 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.553 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.553 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:13.553 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.757289+0000", 2024-09-21T14:06:13.553 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0068951739999999996 2024-09-21T14:06:13.553 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.553 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.553 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:13.553 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.764759+0000", 2024-09-21T14:06:13.553 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0074705689999999998 2024-09-21T14:06:13.553 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.554 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.554 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:06:13.554 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.765446+0000", 2024-09-21T14:06:13.554 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00068706799999999999 2024-09-21T14:06:13.554 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.554 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.554 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:06:13.554 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.766945+0000", 2024-09-21T14:06:13.554 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0014990509999999999 2024-09-21T14:06:13.554 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.554 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.555 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:06:13.555 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.771992+0000", 2024-09-21T14:06:13.555 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0050469319999999996 2024-09-21T14:06:13.555 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:13.555 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:06:13.555 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:13.555 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.555 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.555 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187662 9.f 9:f2269e64:::benchmark_data_smithi049_138504_object66254:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e347)", 2024-09-21T14:06:13.555 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:42.244949+0000", 2024-09-21T14:06:13.555 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 271.18970517399998, 2024-09-21T14:06:13.555 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1087973170000001, 2024-09-21T14:06:13.556 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:06:13.556 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:06:13.556 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:13.556 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:06:13.556 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:06:13.556 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:06:13.556 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187662 2024-09-21T14:06:13.556 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.556 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:06:13.556 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.556 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:06:13.556 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.244949+0000", 2024-09-21T14:06:13.557 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:13.557 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.557 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.557 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:06:13.557 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.244949+0000", 2024-09-21T14:06:13.557 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:13.557 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.557 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.557 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:06:13.557 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.244951+0000", 2024-09-21T14:06:13.557 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.962e-06 2024-09-21T14:06:13.557 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.557 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.558 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:06:13.558 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.244957+0000", 2024-09-21T14:06:13.558 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8170000000000004e-06 2024-09-21T14:06:13.558 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.558 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.558 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:06:13.558 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.244972+0000", 2024-09-21T14:06:13.558 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5204999999999999e-05 2024-09-21T14:06:13.558 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.558 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.558 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:06:13.559 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.244981+0000", 2024-09-21T14:06:13.559 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5990000000000002e-06 2024-09-21T14:06:13.559 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.559 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.559 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:13.559 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.234563+0000", 2024-09-21T14:06:13.559 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.989582548 2024-09-21T14:06:13.559 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.559 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.559 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:06:13.559 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.234696+0000", 2024-09-21T14:06:13.559 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013280200000000001 2024-09-21T14:06:13.560 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.560 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.560 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3", 2024-09-21T14:06:13.560 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.234783+0000", 2024-09-21T14:06:13.560 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.6990999999999994e-05 2024-09-21T14:06:13.560 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.560 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.560 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:06:13.560 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.342749+0000", 2024-09-21T14:06:13.560 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1079658480000001 2024-09-21T14:06:13.560 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.560 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.561 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:13.561 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353673+0000", 2024-09-21T14:06:13.561 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.010924473000000001 2024-09-21T14:06:13.561 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.561 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.561 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:13.561 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353696+0000", 2024-09-21T14:06:13.561 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2792000000000001e-05 2024-09-21T14:06:13.561 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.561 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.561 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:06:13.561 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353726+0000", 2024-09-21T14:06:13.562 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9509000000000002e-05 2024-09-21T14:06:13.562 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.562 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.562 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:06:13.562 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353746+0000", 2024-09-21T14:06:13.562 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0771000000000002e-05 2024-09-21T14:06:13.562 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:13.562 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:06:13.562 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:13.562 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.562 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.563 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187665 9.f 9:fde6db3c:::benchmark_data_smithi049_138504_object66257:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e347)", 2024-09-21T14:06:13.563 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:42.254849+0000", 2024-09-21T14:06:13.563 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 271.17980531199998, 2024-09-21T14:06:13.563 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.186045939, 2024-09-21T14:06:13.563 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:06:13.563 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:06:13.563 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:13.563 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:06:13.563 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:06:13.563 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:06:13.564 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187665 2024-09-21T14:06:13.564 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.564 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:06:13.564 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.564 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:06:13.564 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.254849+0000", 2024-09-21T14:06:13.564 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:13.564 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.564 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.564 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:06:13.564 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.254849+0000", 2024-09-21T14:06:13.565 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:13.565 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.565 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.565 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:06:13.565 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.254851+0000", 2024-09-21T14:06:13.565 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9209999999999999e-06 2024-09-21T14:06:13.565 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.565 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.565 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:06:13.565 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.254857+0000", 2024-09-21T14:06:13.565 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3659999999999997e-06 2024-09-21T14:06:13.565 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.566 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.566 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:06:13.566 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.254870+0000", 2024-09-21T14:06:13.566 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2892e-05 2024-09-21T14:06:13.566 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.566 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.566 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:06:13.566 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.254882+0000", 2024-09-21T14:06:13.566 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1647999999999999e-05 2024-09-21T14:06:13.566 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.566 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.567 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:13.567 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.415623+0000", 2024-09-21T14:06:13.567 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.160741582 2024-09-21T14:06:13.567 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.567 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.567 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:06:13.567 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.415771+0000", 2024-09-21T14:06:13.567 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014724000000000001 2024-09-21T14:06:13.567 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.567 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.567 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3", 2024-09-21T14:06:13.567 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.415898+0000", 2024-09-21T14:06:13.568 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012708399999999999 2024-09-21T14:06:13.568 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.568 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.568 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:13.568 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.427284+0000", 2024-09-21T14:06:13.568 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.011386624 2024-09-21T14:06:13.568 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.568 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.568 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:13.568 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.427305+0000", 2024-09-21T14:06:13.568 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0956000000000001e-05 2024-09-21T14:06:13.569 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.569 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.569 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:06:13.569 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.440820+0000", 2024-09-21T14:06:13.569 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.013514511999999999 2024-09-21T14:06:13.569 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.569 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.569 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:06:13.569 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.440860+0000", 2024-09-21T14:06:13.569 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0448000000000003e-05 2024-09-21T14:06:13.569 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.569 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.570 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:06:13.570 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.440895+0000", 2024-09-21T14:06:13.570 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4666000000000003e-05 2024-09-21T14:06:13.570 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:13.570 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:06:13.570 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:13.570 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.570 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.570 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25421.0:187664 9.d e347/345 9:bfa1ff03:::benchmark_data_smithi049_138504_object66256:head v 347'11740, mlcod=346'11738)", 2024-09-21T14:06:13.570 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:42.255288+0000", 2024-09-21T14:06:13.570 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 271.17936643299998, 2024-09-21T14:06:13.571 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2055402320000002, 2024-09-21T14:06:13.571 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:06:13.571 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:06:13.571 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:13.571 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:06:13.571 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.571 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:06:13.571 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.255288+0000", 2024-09-21T14:06:13.571 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:13.571 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.571 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.571 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:06:13.572 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.255288+0000", 2024-09-21T14:06:13.572 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:13.572 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.572 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.572 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:06:13.572 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.255289+0000", 2024-09-21T14:06:13.572 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2200000000000003e-07 2024-09-21T14:06:13.572 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.572 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.572 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:06:13.572 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.255296+0000", 2024-09-21T14:06:13.573 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3440000000000003e-06 2024-09-21T14:06:13.573 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.573 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.573 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:06:13.573 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.255312+0000", 2024-09-21T14:06:13.573 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5968999999999999e-05 2024-09-21T14:06:13.573 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.573 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.573 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:06:13.573 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.255319+0000", 2024-09-21T14:06:13.574 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9199999999999998e-06 2024-09-21T14:06:13.574 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.574 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.574 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:13.574 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.459355+0000", 2024-09-21T14:06:13.574 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2040357249999998 2024-09-21T14:06:13.574 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.574 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.574 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:06:13.574 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.459378+0000", 2024-09-21T14:06:13.575 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3757e-05 2024-09-21T14:06:13.575 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.575 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.575 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:06:13.575 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.460796+0000", 2024-09-21T14:06:13.575 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001417721 2024-09-21T14:06:13.575 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.575 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.575 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:06:13.575 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.460828+0000", 2024-09-21T14:06:13.575 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1974000000000003e-05 2024-09-21T14:06:13.576 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:13.576 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:06:13.576 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:13.576 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.576 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.576 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24365.0:2978 5.15 5:a93a5511:::notify.2:head [watch ping cookie 94025139204608 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:06:13.576 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:43.276775+0000", 2024-09-21T14:06:13.576 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 270.15787891100001, 2024-09-21T14:06:13.576 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4885784160000002, 2024-09-21T14:06:13.576 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:06:13.576 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:06:13.577 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T14:06:13.577 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:06:13.577 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24365", 2024-09-21T14:06:13.577 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3404764347", 2024-09-21T14:06:13.577 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 2978 2024-09-21T14:06:13.577 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.577 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:06:13.577 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.577 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:06:13.577 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:43.276775+0000", 2024-09-21T14:06:13.577 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:13.578 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.578 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.578 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:06:13.578 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:43.276775+0000", 2024-09-21T14:06:13.578 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:13.578 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.578 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.578 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:06:13.578 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:43.276777+0000", 2024-09-21T14:06:13.578 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9630000000000001e-06 2024-09-21T14:06:13.578 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.578 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.579 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:06:13.579 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:43.276784+0000", 2024-09-21T14:06:13.579 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1219999999999997e-06 2024-09-21T14:06:13.579 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.579 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.579 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:06:13.579 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:43.276802+0000", 2024-09-21T14:06:13.579 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7096999999999999e-05 2024-09-21T14:06:13.579 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.579 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.579 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:06:13.579 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:43.276811+0000", 2024-09-21T14:06:13.579 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7780000000000002e-06 2024-09-21T14:06:13.580 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.580 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.580 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:13.580 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.765168+0000", 2024-09-21T14:06:13.580 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4883561849999998 2024-09-21T14:06:13.580 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.580 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.580 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:06:13.580 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.765252+0000", 2024-09-21T14:06:13.580 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4201000000000002e-05 2024-09-21T14:06:13.580 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.581 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.581 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:06:13.581 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.765354+0000", 2024-09-21T14:06:13.581 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010207 2024-09-21T14:06:13.581 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:13.581 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:06:13.581 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:13.581 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.581 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.581 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.25421.0:187651 9.f e349/345 ondisk, result = 0)", 2024-09-21T14:06:13.581 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:44.238795+0000", 2024-09-21T14:06:13.581 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 269.19585962399998, 2024-09-21T14:06:13.582 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.115079242, 2024-09-21T14:06:13.582 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:06:13.582 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:06:13.582 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T14:06:13.582 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:06:13.582 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.582 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:06:13.582 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238795+0000", 2024-09-21T14:06:13.582 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:13.582 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.583 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.583 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:06:13.583 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238795+0000", 2024-09-21T14:06:13.583 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:13.583 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.583 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.583 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:06:13.583 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238795+0000", 2024-09-21T14:06:13.583 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9800000000000003e-07 2024-09-21T14:06:13.583 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.583 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.584 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:06:13.584 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238799+0000", 2024-09-21T14:06:13.584 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9920000000000001e-06 2024-09-21T14:06:13.584 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.584 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.584 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:06:13.584 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238809+0000", 2024-09-21T14:06:13.584 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6120000000000005e-06 2024-09-21T14:06:13.584 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.584 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.584 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:06:13.584 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238816+0000", 2024-09-21T14:06:13.585 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6229999999999999e-06 2024-09-21T14:06:13.585 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.585 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.585 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:13.585 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.342725+0000", 2024-09-21T14:06:13.585 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1039093099999999 2024-09-21T14:06:13.585 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.585 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.585 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:06:13.585 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.342733+0000", 2024-09-21T14:06:13.585 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9189999999999999e-06 2024-09-21T14:06:13.585 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.586 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.586 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:06:13.586 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353874+0000", 2024-09-21T14:06:13.586 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.011141088 2024-09-21T14:06:13.586 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:13.586 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:06:13.586 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:13.586 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.586 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.586 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.25421.0:187651 9.f e349/345 ondisk, result = 0)", 2024-09-21T14:06:13.586 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:44.238813+0000", 2024-09-21T14:06:13.587 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 269.195841282, 2024-09-21T14:06:13.587 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1148360479999999, 2024-09-21T14:06:13.587 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:06:13.587 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:06:13.587 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T14:06:13.587 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:06:13.587 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.587 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:06:13.587 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238813+0000", 2024-09-21T14:06:13.587 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:13.587 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.587 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.588 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:06:13.588 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238813+0000", 2024-09-21T14:06:13.588 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:13.588 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.588 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.588 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:06:13.588 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238814+0000", 2024-09-21T14:06:13.588 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4700000000000001e-07 2024-09-21T14:06:13.588 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.588 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.588 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:06:13.588 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238817+0000", 2024-09-21T14:06:13.589 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3469999999999999e-06 2024-09-21T14:06:13.589 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.589 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.589 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:06:13.589 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238827+0000", 2024-09-21T14:06:13.589 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0011e-05 2024-09-21T14:06:13.589 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.589 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.589 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:06:13.589 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238832+0000", 2024-09-21T14:06:13.589 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1139999999999997e-06 2024-09-21T14:06:13.589 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.590 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.590 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:13.590 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353559+0000", 2024-09-21T14:06:13.590 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1147265909999999 2024-09-21T14:06:13.590 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.590 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.590 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:06:13.590 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353566+0000", 2024-09-21T14:06:13.590 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8669999999999994e-06 2024-09-21T14:06:13.590 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.590 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.591 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:06:13.591 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353649+0000", 2024-09-21T14:06:13.591 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2571000000000006e-05 2024-09-21T14:06:13.591 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:13.591 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:06:13.591 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:13.591 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.591 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.591 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187668 9.5 9:acffd95c:::benchmark_data_smithi049_138504_object66260:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e349)", 2024-09-21T14:06:13.591 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:44.468347+0000", 2024-09-21T14:06:13.591 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 268.96630778500003, 2024-09-21T14:06:13.591 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3419073799999999, 2024-09-21T14:06:13.592 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:06:13.592 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:06:13.592 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:13.592 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:06:13.592 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:06:13.592 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:06:13.592 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187668 2024-09-21T14:06:13.592 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.592 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:06:13.592 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.592 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:06:13.592 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.468347+0000", 2024-09-21T14:06:13.593 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:13.593 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.593 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.593 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:06:13.593 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.468347+0000", 2024-09-21T14:06:13.593 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:13.593 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.593 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.593 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:06:13.593 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.468347+0000", 2024-09-21T14:06:13.593 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2399999999999997e-07 2024-09-21T14:06:13.594 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.594 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.594 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:06:13.594 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.468351+0000", 2024-09-21T14:06:13.594 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6049999999999998e-06 2024-09-21T14:06:13.594 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.594 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.594 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:06:13.594 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.468358+0000", 2024-09-21T14:06:13.594 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4510000000000003e-06 2024-09-21T14:06:13.594 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.595 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.595 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:06:13.595 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.468366+0000", 2024-09-21T14:06:13.595 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6140000000000003e-06 2024-09-21T14:06:13.595 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.595 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.595 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:13.595 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.772366+0000", 2024-09-21T14:06:13.595 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3040005530000001 2024-09-21T14:06:13.595 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.595 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.595 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:06:13.596 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.775717+0000", 2024-09-21T14:06:13.596 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0033501080000000001 2024-09-21T14:06:13.596 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.596 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.596 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:06:13.596 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.781090+0000", 2024-09-21T14:06:13.596 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0053731990000000004 2024-09-21T14:06:13.596 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.596 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.596 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:13.596 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.798846+0000", 2024-09-21T14:06:13.596 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.017756042999999999 2024-09-21T14:06:13.596 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.597 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.597 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:13.597 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.800017+0000", 2024-09-21T14:06:13.597 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0011714430000000001 2024-09-21T14:06:13.597 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.597 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.597 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:06:13.597 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.800552+0000", 2024-09-21T14:06:13.597 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00053442500000000005 2024-09-21T14:06:13.597 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.597 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.597 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:06:13.598 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.801930+0000", 2024-09-21T14:06:13.598 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0013783109999999999 2024-09-21T14:06:13.598 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:13.598 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:13.598 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:06:13.598 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.810254+0000", 2024-09-21T14:06:13.598 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.008323904 2024-09-21T14:06:13.598 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:13.598 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:06:13.598 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:13.598 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:13.599 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:06:13.599 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T14:06:13.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:13.601 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:13.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:06:13.651 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:06:13.692 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph tell osd.2 reset_purged_snaps_last 2024-09-21T14:06:13.702 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:06:13.733 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:13.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:13.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:06:13.835 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:06:13.865 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:13.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:13.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:06:13.967 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:06:13.993 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:13.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:13.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:06:14.095 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:06:14.126 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:14.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:14.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:06:14.227 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:14.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:14 smithi049 ceph-mon[40489]: 5.11 scrub starts 2024-09-21T14:06:14.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:14 smithi049 ceph-mon[40489]: 5.11 scrub ok 2024-09-21T14:06:14.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:14 smithi049 ceph-mon[40489]: 4.12 scrub starts 2024-09-21T14:06:14.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:14 smithi049 ceph-mon[40489]: 4.12 scrub ok 2024-09-21T14:06:14.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:14 smithi049 ceph-mon[30080]: 5.11 scrub starts 2024-09-21T14:06:14.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:14 smithi049 ceph-mon[30080]: 5.11 scrub ok 2024-09-21T14:06:14.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:14 smithi049 ceph-mon[30080]: 4.12 scrub starts 2024-09-21T14:06:14.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:14 smithi049 ceph-mon[30080]: 4.12 scrub ok 2024-09-21T14:06:14.271 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:14.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:14.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:14.372 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:06:14.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:14.414 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:14.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:06:14.515 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:06:14.549 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:14.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:14.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:06:14.651 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:06:14.680 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:14.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:14.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:06:14.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:14 smithi187 ceph-mon[33032]: 5.11 scrub starts 2024-09-21T14:06:14.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:14 smithi187 ceph-mon[33032]: 5.11 scrub ok 2024-09-21T14:06:14.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:14 smithi187 ceph-mon[33032]: 4.12 scrub starts 2024-09-21T14:06:14.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:14 smithi187 ceph-mon[33032]: 4.12 scrub ok 2024-09-21T14:06:14.782 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:14.810 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:14.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:14.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:14.912 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:14.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:14.946 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:14.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:15.047 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:06:15.082 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:15.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:15.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:06:15.183 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:06:15.217 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:15.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:15.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:06:15.219 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T14:06:15.319 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:06:15.328 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:06:15.329 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:06:15.329 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:06:15.329 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:06:15.329 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:06:15.329 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:06:15.329 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:06:15.329 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:06:15.329 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:06:15.329 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:06:15.329 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:06:15.329 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:06:15.329 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:06:15.330 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:06:15.330 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:06:15.330 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:06:15.360 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:15.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:15.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:06:15.462 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:15.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:15.500 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:15.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:15.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:15 smithi049 ceph-mon[30080]: pgmap v879: 168 pgs: 168 active+clean; 1.4 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 796 B/s rd, 0 B/s wr, 1.59k op/s 2024-09-21T14:06:15.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:15 smithi049 ceph-mon[30080]: 4.1a scrub starts 2024-09-21T14:06:15.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:15 smithi049 ceph-mon[30080]: 4.1a scrub ok 2024-09-21T14:06:15.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:15 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2911017950' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:06:15.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:15 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:06:15.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:15 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:06:15.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:15 smithi049 ceph-mon[30080]: 3.f scrub starts 2024-09-21T14:06:15.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:15 smithi049 ceph-mon[30080]: 3.f scrub ok 2024-09-21T14:06:15.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:15 smithi049 ceph-mon[30080]: osdmap e448: 8 total, 8 up, 8 in 2024-09-21T14:06:15.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:15 smithi049 ceph-mon[40489]: pgmap v879: 168 pgs: 168 active+clean; 1.4 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 796 B/s rd, 0 B/s wr, 1.59k op/s 2024-09-21T14:06:15.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:15 smithi049 ceph-mon[40489]: 4.1a scrub starts 2024-09-21T14:06:15.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:15 smithi049 ceph-mon[40489]: 4.1a scrub ok 2024-09-21T14:06:15.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:15 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2911017950' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:06:15.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:15 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:06:15.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:15 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:06:15.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:15 smithi049 ceph-mon[40489]: 3.f scrub starts 2024-09-21T14:06:15.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:15 smithi049 ceph-mon[40489]: 3.f scrub ok 2024-09-21T14:06:15.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:15 smithi049 ceph-mon[40489]: osdmap e448: 8 total, 8 up, 8 in 2024-09-21T14:06:15.601 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:06:15.624 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:15.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:15.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:06:15.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:15 smithi187 ceph-mon[33032]: pgmap v879: 168 pgs: 168 active+clean; 1.4 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 796 B/s rd, 0 B/s wr, 1.59k op/s 2024-09-21T14:06:15.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:15 smithi187 ceph-mon[33032]: 4.1a scrub starts 2024-09-21T14:06:15.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:15 smithi187 ceph-mon[33032]: 4.1a scrub ok 2024-09-21T14:06:15.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:15 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2911017950' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:06:15.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:15 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:06:15.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:15 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:06:15.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:15 smithi187 ceph-mon[33032]: 3.f scrub starts 2024-09-21T14:06:15.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:15 smithi187 ceph-mon[33032]: 3.f scrub ok 2024-09-21T14:06:15.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:15 smithi187 ceph-mon[33032]: osdmap e448: 8 total, 8 up, 8 in 2024-09-21T14:06:15.725 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:15.748 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:15.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:15.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:15.849 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:06:15.871 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:15.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:15.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:06:15.972 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:15.996 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:15.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:15.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:16.098 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:06:16.128 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:16.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:16.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:06:16.230 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:16.264 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:16.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:16.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:16.365 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:16.395 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:16.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:16.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:16.497 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:16.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:16 smithi049 ceph-mon[40489]: 4.3 scrub starts 2024-09-21T14:06:16.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:16 smithi049 ceph-mon[40489]: 4.3 scrub ok 2024-09-21T14:06:16.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:16 smithi049 ceph-mon[30080]: 4.3 scrub starts 2024-09-21T14:06:16.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:16 smithi049 ceph-mon[30080]: 4.3 scrub ok 2024-09-21T14:06:16.520 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:16.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:16.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:16.622 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:16.658 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:16.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:16.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:16.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:16 smithi187 ceph-mon[33032]: 4.3 scrub starts 2024-09-21T14:06:16.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:16 smithi187 ceph-mon[33032]: 4.3 scrub ok 2024-09-21T14:06:16.760 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:16.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:16.797 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:16.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:16.898 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:16.938 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:16.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:16.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:17.039 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:06:17.068 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:17.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:17.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:06:17.169 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:17.204 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:17.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:17.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:17.306 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:06:17.336 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:17.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:17.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:06:17.439 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:06:17.470 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:17.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:17.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:06:17.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:17 smithi049 ceph-mon[40489]: pgmap v881: 168 pgs: 168 active+clean; 1.3 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.10k op/s 2024-09-21T14:06:17.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:17 smithi049 ceph-mon[30080]: pgmap v881: 168 pgs: 168 active+clean; 1.3 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.10k op/s 2024-09-21T14:06:17.572 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:17.604 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:17.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:17.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:17.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:17 smithi187 ceph-mon[33032]: pgmap v881: 168 pgs: 168 active+clean; 1.3 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.10k op/s 2024-09-21T14:06:17.705 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:06:17.726 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:17.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:17.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:06:17.828 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:06:17.864 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:17.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:17.865 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:06:17.966 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:06:17.995 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:17.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:17.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:06:18.097 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:06:18.127 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:18.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:18.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:06:18.229 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:06:18.265 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:18.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:18.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:06:18.366 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:18.370 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T14:06:18.405 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:18.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:18.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:18.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:18 smithi049 ceph-mon[40489]: 1.8 scrub starts 2024-09-21T14:06:18.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:18 smithi049 ceph-mon[40489]: 1.8 scrub ok 2024-09-21T14:06:18.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:18 smithi049 ceph-mon[30080]: 1.8 scrub starts 2024-09-21T14:06:18.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:18 smithi049 ceph-mon[30080]: 1.8 scrub ok 2024-09-21T14:06:18.507 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:18.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:18.554 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:18.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:18.655 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:18.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:18 smithi187 ceph-mon[33032]: 1.8 scrub starts 2024-09-21T14:06:18.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:18 smithi187 ceph-mon[33032]: 1.8 scrub ok 2024-09-21T14:06:18.699 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:18.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:18.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:18.802 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:18.848 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:18.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:18.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:18.889 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:06:18.900 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:06:18.944 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:06:18.955 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:06:19.010 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:19.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:19.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:06:19.113 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:19.157 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:19.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:19.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:19.258 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:06:19.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:19.297 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:19.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:06:19.335 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:19 smithi049 ceph-mon[30080]: pgmap v882: 168 pgs: 168 active+clean; 1.3 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 875 op/s 2024-09-21T14:06:19.335 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:19 smithi049 ceph-mon[30080]: 3.5 scrub starts 2024-09-21T14:06:19.335 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:19 smithi049 ceph-mon[30080]: 3.5 scrub ok 2024-09-21T14:06:19.335 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:19 smithi049 ceph-mon[30080]: 3.d scrub starts 2024-09-21T14:06:19.335 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:19 smithi049 ceph-mon[30080]: 3.d scrub ok 2024-09-21T14:06:19.335 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:19 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:06:19.335 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:19 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:06:19.336 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:19 smithi049 ceph-mon[40489]: pgmap v882: 168 pgs: 168 active+clean; 1.3 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 875 op/s 2024-09-21T14:06:19.336 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:19 smithi049 ceph-mon[40489]: 3.5 scrub starts 2024-09-21T14:06:19.336 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:19 smithi049 ceph-mon[40489]: 3.5 scrub ok 2024-09-21T14:06:19.336 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:19 smithi049 ceph-mon[40489]: 3.d scrub starts 2024-09-21T14:06:19.336 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:19 smithi049 ceph-mon[40489]: 3.d scrub ok 2024-09-21T14:06:19.336 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:19 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:06:19.336 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:19 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:06:19.397 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:06:19.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:19.438 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:19.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:06:19.539 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:19.577 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:19.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:19.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:19.679 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:19.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:19 smithi187 ceph-mon[33032]: pgmap v882: 168 pgs: 168 active+clean; 1.3 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 875 op/s 2024-09-21T14:06:19.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:19 smithi187 ceph-mon[33032]: 3.5 scrub starts 2024-09-21T14:06:19.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:19 smithi187 ceph-mon[33032]: 3.5 scrub ok 2024-09-21T14:06:19.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:19 smithi187 ceph-mon[33032]: 3.d scrub starts 2024-09-21T14:06:19.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:19 smithi187 ceph-mon[33032]: 3.d scrub ok 2024-09-21T14:06:19.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:19 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:06:19.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:19 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:06:19.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:19.721 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:19.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:19.822 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:19.858 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:19.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:19.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:19.960 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:06:19.989 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:19.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:19.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:06:20.091 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:20.132 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:20.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:20.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:20.202 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T14:06:20.203 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:06:20.233 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:20.263 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:06:20 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:06:20.277 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:20.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:20.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:20.379 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:06:20.413 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:20.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:20.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:06:20.515 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:20.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:20.552 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:20.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:20.579 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:20 smithi049 ceph-mon[30080]: 3.3 scrub starts 2024-09-21T14:06:20.579 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:20 smithi049 ceph-mon[30080]: 3.3 scrub ok 2024-09-21T14:06:20.579 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:20 smithi049 ceph-mon[30080]: osdmap e449: 8 total, 8 up, 8 in 2024-09-21T14:06:20.579 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:20 smithi049 ceph-mon[30080]: 4.1b scrub starts 2024-09-21T14:06:20.580 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:20 smithi049 ceph-mon[30080]: 4.1b scrub ok 2024-09-21T14:06:20.580 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:20 smithi049 ceph-mon[30080]: 6.8 scrub starts 2024-09-21T14:06:20.580 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:20 smithi049 ceph-mon[30080]: 6.8 scrub ok 2024-09-21T14:06:20.580 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:20 smithi049 ceph-mon[30080]: osdmap e450: 8 total, 8 up, 8 in 2024-09-21T14:06:20.580 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:20 smithi049 ceph-mon[40489]: 3.3 scrub starts 2024-09-21T14:06:20.580 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:20 smithi049 ceph-mon[40489]: 3.3 scrub ok 2024-09-21T14:06:20.580 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:20 smithi049 ceph-mon[40489]: osdmap e449: 8 total, 8 up, 8 in 2024-09-21T14:06:20.580 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:20 smithi049 ceph-mon[40489]: 4.1b scrub starts 2024-09-21T14:06:20.580 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:20 smithi049 ceph-mon[40489]: 4.1b scrub ok 2024-09-21T14:06:20.580 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:20 smithi049 ceph-mon[40489]: 6.8 scrub starts 2024-09-21T14:06:20.580 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:20 smithi049 ceph-mon[40489]: 6.8 scrub ok 2024-09-21T14:06:20.580 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:20 smithi049 ceph-mon[40489]: osdmap e450: 8 total, 8 up, 8 in 2024-09-21T14:06:20.653 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:06:20.686 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:20.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:20.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:06:20.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:20 smithi187 ceph-mon[33032]: 3.3 scrub starts 2024-09-21T14:06:20.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:20 smithi187 ceph-mon[33032]: 3.3 scrub ok 2024-09-21T14:06:20.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:20 smithi187 ceph-mon[33032]: osdmap e449: 8 total, 8 up, 8 in 2024-09-21T14:06:20.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:20 smithi187 ceph-mon[33032]: 4.1b scrub starts 2024-09-21T14:06:20.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:20 smithi187 ceph-mon[33032]: 4.1b scrub ok 2024-09-21T14:06:20.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:20 smithi187 ceph-mon[33032]: 6.8 scrub starts 2024-09-21T14:06:20.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:20 smithi187 ceph-mon[33032]: 6.8 scrub ok 2024-09-21T14:06:20.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:20 smithi187 ceph-mon[33032]: osdmap e450: 8 total, 8 up, 8 in 2024-09-21T14:06:20.777 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T14:06:20.788 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:06:20.818 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:20.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:20.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:06:20.920 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:06:20.942 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:06:20 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:06:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:06:20.949 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:20.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:20.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:06:21.051 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:06:21.085 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:21.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:21.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:06:21.187 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:21.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:21.241 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:21.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:21.342 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:21.376 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:21.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:21.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:21.478 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:06:21.514 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:21.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:21.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:06:21.547 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:21 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:06:21.548 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:21 smithi049 ceph-mon[30080]: pgmap v884: 168 pgs: 168 active+clean; 1.0 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 648 B/s rd, 0 B/s wr, 1.30k op/s 2024-09-21T14:06:21.548 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:21 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:06:21.548 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:21 smithi049 ceph-mon[40489]: pgmap v884: 168 pgs: 168 active+clean; 1.0 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 648 B/s rd, 0 B/s wr, 1.30k op/s 2024-09-21T14:06:21.598 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph tell osd.3 reset_purged_snaps_last 2024-09-21T14:06:21.616 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:06:21.647 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:21.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:21.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:06:21.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:21 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:06:21.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:21 smithi187 ceph-mon[33032]: pgmap v884: 168 pgs: 168 active+clean; 1.0 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 648 B/s rd, 0 B/s wr, 1.30k op/s 2024-09-21T14:06:21.749 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:06:21.782 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:21.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:21.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:06:21.884 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:06:21.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:21.921 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:21.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:06:22.022 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:22.055 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:22.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:22.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:22.133 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T14:06:22.157 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:06:22.184 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:22.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:22.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:06:22.260 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:06:22.277 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:06:22.287 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:06:22.290 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-09-21T14:06:22.315 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:22.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:22.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:06:22.417 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:06:22.455 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:22.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:22.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:06:22.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:22 smithi049 ceph-mon[30080]: 6.b scrub starts 2024-09-21T14:06:22.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:22 smithi049 ceph-mon[30080]: 6.b scrub ok 2024-09-21T14:06:22.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:22 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1716151472' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:06:22.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:22 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:06:22.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:22 smithi049 ceph-mon[30080]: 5.5 scrub starts 2024-09-21T14:06:22.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:22 smithi049 ceph-mon[30080]: 5.5 scrub ok 2024-09-21T14:06:22.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:22 smithi049 ceph-mon[40489]: 6.b scrub starts 2024-09-21T14:06:22.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:22 smithi049 ceph-mon[40489]: 6.b scrub ok 2024-09-21T14:06:22.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:22 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1716151472' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:06:22.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:22 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:06:22.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:22 smithi049 ceph-mon[40489]: 5.5 scrub starts 2024-09-21T14:06:22.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:22 smithi049 ceph-mon[40489]: 5.5 scrub ok 2024-09-21T14:06:22.528 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:22 smithi187 ceph-mon[33032]: 6.b scrub starts 2024-09-21T14:06:22.528 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:22 smithi187 ceph-mon[33032]: 6.b scrub ok 2024-09-21T14:06:22.528 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:22 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1716151472' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:06:22.528 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:22 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:06:22.528 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:22 smithi187 ceph-mon[33032]: 5.5 scrub starts 2024-09-21T14:06:22.528 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:22 smithi187 ceph-mon[33032]: 5.5 scrub ok 2024-09-21T14:06:22.556 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:06:22.589 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:22.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:22.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:06:22.690 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:06:22.729 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:22.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:22.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:06:22.831 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:06:22.865 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:22.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:22.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:06:22.969 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:23.011 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:23.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:23.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:23.112 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:23.152 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:23.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:23.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:23.254 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:23.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:23.298 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:23.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:23.399 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:23.446 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:23.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:23.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:23.548 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:23.595 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:23.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:23.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:23.679 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:23 smithi049 ceph-mon[40489]: pgmap v886: 168 pgs: 168 active+clean; 989 MiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 966 op/s 2024-09-21T14:06:23.680 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:23 smithi049 ceph-mon[40489]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T14:06:23.680 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:23 smithi049 ceph-mon[40489]: Cluster is now healthy 2024-09-21T14:06:23.680 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:23 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T14:06:23.680 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:23 smithi049 ceph-mon[40489]: osdmap e451: 8 total, 8 up, 8 in 2024-09-21T14:06:23.680 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:23 smithi049 ceph-mon[40489]: 6.1c scrub starts 2024-09-21T14:06:23.680 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:23 smithi049 ceph-mon[40489]: 6.1c scrub ok 2024-09-21T14:06:23.680 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:23 smithi049 ceph-mon[30080]: pgmap v886: 168 pgs: 168 active+clean; 989 MiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 966 op/s 2024-09-21T14:06:23.680 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:23 smithi049 ceph-mon[30080]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T14:06:23.680 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:23 smithi049 ceph-mon[30080]: Cluster is now healthy 2024-09-21T14:06:23.681 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:23 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T14:06:23.681 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:23 smithi049 ceph-mon[30080]: osdmap e451: 8 total, 8 up, 8 in 2024-09-21T14:06:23.681 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:23 smithi049 ceph-mon[30080]: 6.1c scrub starts 2024-09-21T14:06:23.681 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:23 smithi049 ceph-mon[30080]: 6.1c scrub ok 2024-09-21T14:06:23.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:23 smithi187 ceph-mon[33032]: pgmap v886: 168 pgs: 168 active+clean; 989 MiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 966 op/s 2024-09-21T14:06:23.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:23 smithi187 ceph-mon[33032]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T14:06:23.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:23 smithi187 ceph-mon[33032]: Cluster is now healthy 2024-09-21T14:06:23.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:23 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T14:06:23.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:23 smithi187 ceph-mon[33032]: osdmap e451: 8 total, 8 up, 8 in 2024-09-21T14:06:23.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:23 smithi187 ceph-mon[33032]: 6.1c scrub starts 2024-09-21T14:06:23.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:23 smithi187 ceph-mon[33032]: 6.1c scrub ok 2024-09-21T14:06:23.697 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:23.740 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:23.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:23.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:23.842 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:06:23.876 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:23.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:23.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:06:23.942 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:06:23.942 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:06:23.942 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:06:23.942 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:06:23.943 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:06:23.943 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:06:23.943 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:06:23.943 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:06:23.944 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:06:23.944 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:06:23.944 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:06:23.944 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:06:23.945 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:06:23.945 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:06:23.948 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:06:23.949 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:06:23.978 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:24.017 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:24.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:24.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:24.119 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:06:24.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:24.159 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:24.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:06:24.260 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:06:24.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:24.311 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:24.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:06:24.412 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:06:24.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:24.455 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:24.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:06:24.556 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:06:24.590 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:24.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:24.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:06:24.692 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:06:24.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:24 smithi187 ceph-mon[33032]: 10.e scrub starts 2024-09-21T14:06:24.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:24 smithi187 ceph-mon[33032]: 3.b scrub starts 2024-09-21T14:06:24.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:24 smithi187 ceph-mon[33032]: 3.b scrub ok 2024-09-21T14:06:24.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:24 smithi187 ceph-mon[33032]: 10.e scrub ok 2024-09-21T14:06:24.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:24 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2522520109' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:06:24.726 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:24.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:24.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:06:24.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:24 smithi049 ceph-mon[40489]: 10.e scrub starts 2024-09-21T14:06:24.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:24 smithi049 ceph-mon[40489]: 3.b scrub starts 2024-09-21T14:06:24.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:24 smithi049 ceph-mon[40489]: 3.b scrub ok 2024-09-21T14:06:24.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:24 smithi049 ceph-mon[40489]: 10.e scrub ok 2024-09-21T14:06:24.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:24 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2522520109' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:06:24.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:24 smithi049 ceph-mon[30080]: 10.e scrub starts 2024-09-21T14:06:24.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:24 smithi049 ceph-mon[30080]: 3.b scrub starts 2024-09-21T14:06:24.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:24 smithi049 ceph-mon[30080]: 3.b scrub ok 2024-09-21T14:06:24.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:24 smithi049 ceph-mon[30080]: 10.e scrub ok 2024-09-21T14:06:24.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:24 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2522520109' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:06:24.828 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:24.836 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph tell osd.4 reset_purged_snaps_last 2024-09-21T14:06:24.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:24.877 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:24.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:24.978 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:06:25.016 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:25.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:25.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:06:25.118 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:06:25.153 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:25.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:25.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:06:25.255 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:06:25.289 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:25.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:25.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:06:25.390 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:06:25.419 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:06:25.424 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:25.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:25.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:06:25.525 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:25.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:25 smithi187 ceph-mon[33032]: pgmap v888: 168 pgs: 168 active+clean; 718 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.66k op/s 2024-09-21T14:06:25.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:25 smithi187 ceph-mon[33032]: 5.f scrub starts 2024-09-21T14:06:25.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:25 smithi187 ceph-mon[33032]: 5.f scrub ok 2024-09-21T14:06:25.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:25 smithi187 ceph-mon[33032]: osdmap e452: 8 total, 8 up, 8 in 2024-09-21T14:06:25.700 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:25.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:25.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:25.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:25 smithi049 ceph-mon[30080]: pgmap v888: 168 pgs: 168 active+clean; 718 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.66k op/s 2024-09-21T14:06:25.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:25 smithi049 ceph-mon[30080]: 5.f scrub starts 2024-09-21T14:06:25.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:25 smithi049 ceph-mon[30080]: 5.f scrub ok 2024-09-21T14:06:25.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:25 smithi049 ceph-mon[30080]: osdmap e452: 8 total, 8 up, 8 in 2024-09-21T14:06:25.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:25 smithi049 ceph-mon[40489]: pgmap v888: 168 pgs: 168 active+clean; 718 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.66k op/s 2024-09-21T14:06:25.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:25 smithi049 ceph-mon[40489]: 5.f scrub starts 2024-09-21T14:06:25.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:25 smithi049 ceph-mon[40489]: 5.f scrub ok 2024-09-21T14:06:25.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:25 smithi049 ceph-mon[40489]: osdmap e452: 8 total, 8 up, 8 in 2024-09-21T14:06:25.803 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:06:25.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:25.836 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:25.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:06:25.937 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:06:25.995 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:25.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:25.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:06:26.097 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:26.135 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:26.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:26.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:26.237 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:06:26.275 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:26.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:26.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:06:26.376 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:06:26.419 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:26.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:26.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:06:26.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:26 smithi049 ceph-mon[30080]: 6.6 scrub starts 2024-09-21T14:06:26.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:26 smithi049 ceph-mon[30080]: 6.6 scrub ok 2024-09-21T14:06:26.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:26 smithi049 ceph-mon[30080]: 3.6 scrub starts 2024-09-21T14:06:26.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:26 smithi049 ceph-mon[30080]: 3.6 scrub ok 2024-09-21T14:06:26.469 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:26 smithi049 ceph-mon[40489]: 6.6 scrub starts 2024-09-21T14:06:26.469 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:26 smithi049 ceph-mon[40489]: 6.6 scrub ok 2024-09-21T14:06:26.469 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:26 smithi049 ceph-mon[40489]: 3.6 scrub starts 2024-09-21T14:06:26.469 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:26 smithi049 ceph-mon[40489]: 3.6 scrub ok 2024-09-21T14:06:26.472 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:06:26.521 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:26.532 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T14:06:26.532 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-09-21T14:06:26.532 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-09-21T14:06:26.532 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-09-21T14:06:26.532 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T14:06:26.576 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:26.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:26.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:26.680 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:06:26.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:26 smithi187 ceph-mon[33032]: 6.6 scrub starts 2024-09-21T14:06:26.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:26 smithi187 ceph-mon[33032]: 6.6 scrub ok 2024-09-21T14:06:26.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:26 smithi187 ceph-mon[33032]: 3.6 scrub starts 2024-09-21T14:06:26.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:26 smithi187 ceph-mon[33032]: 3.6 scrub ok 2024-09-21T14:06:26.718 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:26.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:26.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:06:26.820 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:26.859 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:26.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:26.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:26.961 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:27.001 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:27.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:27.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:27.103 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:27.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:27.142 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:27.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:27.243 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:27.286 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:27.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:27.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:27.388 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:06:27.405 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T14:06:27.423 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:27.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:27.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:06:27.438 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:27 smithi049 ceph-mon[30080]: pgmap v890: 168 pgs: 168 active+clean; 617 MiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.18k op/s 2024-09-21T14:06:27.438 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:27 smithi049 ceph-mon[30080]: 6.16 scrub starts 2024-09-21T14:06:27.438 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:27 smithi049 ceph-mon[30080]: 6.16 scrub ok 2024-09-21T14:06:27.438 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:27 smithi049 ceph-mon[40489]: pgmap v890: 168 pgs: 168 active+clean; 617 MiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.18k op/s 2024-09-21T14:06:27.438 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:27 smithi049 ceph-mon[40489]: 6.16 scrub starts 2024-09-21T14:06:27.438 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:27 smithi049 ceph-mon[40489]: 6.16 scrub ok 2024-09-21T14:06:27.525 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:06:27.558 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:27.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:27.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:06:27.660 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:06:27.692 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:27.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:27.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:06:27.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:27 smithi187 ceph-mon[33032]: pgmap v890: 168 pgs: 168 active+clean; 617 MiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.18k op/s 2024-09-21T14:06:27.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:27 smithi187 ceph-mon[33032]: 6.16 scrub starts 2024-09-21T14:06:27.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:27 smithi187 ceph-mon[33032]: 6.16 scrub ok 2024-09-21T14:06:27.795 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:27.832 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:27.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:27.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:27.874 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T14:06:27.934 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:27.947 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T14:06:28.005 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:28.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:28.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:28.106 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:28.144 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:28.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:28.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:28.246 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:06:28.276 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:28.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:28.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:06:28.378 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:06:28.399 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:28.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:28.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:06:28.501 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:06:28.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:28 smithi049 ceph-mon[40489]: 6.11 scrub starts 2024-09-21T14:06:28.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:28 smithi049 ceph-mon[40489]: 6.11 scrub ok 2024-09-21T14:06:28.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:28 smithi049 ceph-mon[40489]: 3.4 scrub starts 2024-09-21T14:06:28.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:28 smithi049 ceph-mon[40489]: 3.4 scrub ok 2024-09-21T14:06:28.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:28 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:06:28.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:28 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:06:28.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:28 smithi049 ceph-mon[30080]: 6.11 scrub starts 2024-09-21T14:06:28.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:28 smithi049 ceph-mon[30080]: 6.11 scrub ok 2024-09-21T14:06:28.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:28 smithi049 ceph-mon[30080]: 3.4 scrub starts 2024-09-21T14:06:28.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:28 smithi049 ceph-mon[30080]: 3.4 scrub ok 2024-09-21T14:06:28.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:28 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:06:28.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:28 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:06:28.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:28.535 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:28.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:06:28.636 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:06:28.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:28.669 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:28.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:06:28.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:28 smithi187 ceph-mon[33032]: 6.11 scrub starts 2024-09-21T14:06:28.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:28 smithi187 ceph-mon[33032]: 6.11 scrub ok 2024-09-21T14:06:28.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:28 smithi187 ceph-mon[33032]: 3.4 scrub starts 2024-09-21T14:06:28.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:28 smithi187 ceph-mon[33032]: 3.4 scrub ok 2024-09-21T14:06:28.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:28 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:06:28.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:28 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:06:28.771 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:06:28.794 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:28.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:28.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:06:28.896 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:06:28.926 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:28.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:28.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:06:29.028 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:06:29.053 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:29.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:29.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:06:29.155 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:06:29.201 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:29.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:29.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:06:29.303 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:29.343 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:29.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:29.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:29.445 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:29.482 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:29.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:29.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:29.584 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:06:29.617 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:29.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:29.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:06:29.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:29 smithi187 ceph-mon[33032]: pgmap v891: 168 pgs: 168 active+clean; 591 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 948 op/s 2024-09-21T14:06:29.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:29 smithi187 ceph-mon[33032]: osdmap e453: 8 total, 8 up, 8 in 2024-09-21T14:06:29.718 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:06:29.750 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:29.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:29.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:06:29.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:29 smithi049 ceph-mon[40489]: pgmap v891: 168 pgs: 168 active+clean; 591 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 948 op/s 2024-09-21T14:06:29.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:29 smithi049 ceph-mon[40489]: osdmap e453: 8 total, 8 up, 8 in 2024-09-21T14:06:29.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:29 smithi049 ceph-mon[30080]: pgmap v891: 168 pgs: 168 active+clean; 591 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 948 op/s 2024-09-21T14:06:29.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:29 smithi049 ceph-mon[30080]: osdmap e453: 8 total, 8 up, 8 in 2024-09-21T14:06:29.834 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph tell osd.5 reset_purged_snaps_last 2024-09-21T14:06:29.852 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:06:29.881 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:29.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:29.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:06:29.958 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:06:29.983 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:06:30.006 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:06:30.009 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:30.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:30.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:06:30.040 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:06:30.110 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:30.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:30.169 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:30.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:30.270 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:30.312 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:30.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:30.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:30.315 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:06:30 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:06:30.390 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:30 smithi049 ceph-mon[30080]: 5.1 scrub starts 2024-09-21T14:06:30.390 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:30 smithi049 ceph-mon[30080]: 5.1 scrub ok 2024-09-21T14:06:30.390 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:30 smithi049 ceph-mon[30080]: 5.0 scrub starts 2024-09-21T14:06:30.390 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:30 smithi049 ceph-mon[30080]: 5.0 scrub ok 2024-09-21T14:06:30.390 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:30 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:06:30.391 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:30 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:06:30.391 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:30 smithi049 ceph-mon[30080]: osdmap e454: 8 total, 8 up, 8 in 2024-09-21T14:06:30.391 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:30 smithi049 ceph-mon[40489]: 5.1 scrub starts 2024-09-21T14:06:30.391 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:30 smithi049 ceph-mon[40489]: 5.1 scrub ok 2024-09-21T14:06:30.391 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:30 smithi049 ceph-mon[40489]: 5.0 scrub starts 2024-09-21T14:06:30.391 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:30 smithi049 ceph-mon[40489]: 5.0 scrub ok 2024-09-21T14:06:30.391 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:30 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:06:30.391 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:30 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:06:30.391 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:30 smithi049 ceph-mon[40489]: osdmap e454: 8 total, 8 up, 8 in 2024-09-21T14:06:30.414 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:30.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:30.451 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:30.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:30.552 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:06:30.591 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:30.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:30.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:06:30.693 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:06:30.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:30 smithi187 ceph-mon[33032]: 5.1 scrub starts 2024-09-21T14:06:30.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:30 smithi187 ceph-mon[33032]: 5.1 scrub ok 2024-09-21T14:06:30.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:30 smithi187 ceph-mon[33032]: 5.0 scrub starts 2024-09-21T14:06:30.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:30 smithi187 ceph-mon[33032]: 5.0 scrub ok 2024-09-21T14:06:30.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:30 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:06:30.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:30 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:06:30.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:30 smithi187 ceph-mon[33032]: osdmap e454: 8 total, 8 up, 8 in 2024-09-21T14:06:30.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:30.721 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:30.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:06:30.754 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:06:30 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:06:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:06:30.768 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T14:06:30.768 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-09-21T14:06:30.768 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-09-21T14:06:30.768 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-09-21T14:06:30.768 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.768 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25421.0:113894 9.b e296/278 9:ddec8d1f:::benchmark_data_smithi049_138504_object113893:head v 296'7127, mlcod=296'7123)", 2024-09-21T14:06:30.768 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:59:25.591505+0000", 2024-09-21T14:06:30.768 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 425.17246648100001, 2024-09-21T14:06:30.768 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.34647377800000001, 2024-09-21T14:06:30.768 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:06:30.769 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:06:30.769 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:30.769 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:06:30.769 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.769 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:06:30.769 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.591505+0000", 2024-09-21T14:06:30.769 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:30.769 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.769 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.769 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:06:30.769 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.591505+0000", 2024-09-21T14:06:30.769 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:30.769 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.769 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.769 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:06:30.770 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.591507+0000", 2024-09-21T14:06:30.770 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3319999999999999e-06 2024-09-21T14:06:30.770 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.770 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.770 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:06:30.770 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.591549+0000", 2024-09-21T14:06:30.770 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1995e-05 2024-09-21T14:06:30.770 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.770 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.770 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:06:30.770 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.591563+0000", 2024-09-21T14:06:30.770 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4436999999999999e-05 2024-09-21T14:06:30.770 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.771 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.771 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:06:30.771 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.591571+0000", 2024-09-21T14:06:30.771 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3920000000000003e-06 2024-09-21T14:06:30.771 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.771 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.771 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:30.771 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.592738+0000", 2024-09-21T14:06:30.771 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001166818 2024-09-21T14:06:30.771 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.771 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.771 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:06:30.771 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.592758+0000", 2024-09-21T14:06:30.771 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9273e-05 2024-09-21T14:06:30.771 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.772 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.772 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:06:30.772 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.596266+0000", 2024-09-21T14:06:30.772 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0035082199999999998 2024-09-21T14:06:30.772 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.772 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.772 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:06:30.772 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.937979+0000", 2024-09-21T14:06:30.772 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.34171331100000002 2024-09-21T14:06:30.772 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:30.772 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:30.772 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:30.772 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.773 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.773 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25421.0:113904 9.5 e296/273 9:a9988855:::benchmark_data_smithi049_138504_object113903:head v 296'7353, mlcod=296'7351)", 2024-09-21T14:06:30.773 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:59:25.594163+0000", 2024-09-21T14:06:30.773 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 425.16980870800001, 2024-09-21T14:06:30.773 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.34383867899999998, 2024-09-21T14:06:30.773 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:06:30.773 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:06:30.773 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:30.773 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:06:30.773 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.773 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:06:30.773 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.594163+0000", 2024-09-21T14:06:30.773 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:30.773 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.773 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.774 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:06:30.774 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.594163+0000", 2024-09-21T14:06:30.774 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:30.774 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.774 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.774 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:06:30.774 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.594165+0000", 2024-09-21T14:06:30.774 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.468e-06 2024-09-21T14:06:30.774 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.774 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.774 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:06:30.774 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.594212+0000", 2024-09-21T14:06:30.774 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7679999999999998e-05 2024-09-21T14:06:30.775 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.775 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.775 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:06:30.775 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.594227+0000", 2024-09-21T14:06:30.775 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.483e-05 2024-09-21T14:06:30.775 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.775 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.775 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:06:30.775 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.594235+0000", 2024-09-21T14:06:30.775 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6020000000000002e-06 2024-09-21T14:06:30.775 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.775 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.775 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:30.775 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.594307+0000", 2024-09-21T14:06:30.775 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2090999999999998e-05 2024-09-21T14:06:30.776 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.776 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.776 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:06:30.776 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.594328+0000", 2024-09-21T14:06:30.776 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1406000000000001e-05 2024-09-21T14:06:30.776 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.776 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.776 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:06:30.776 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.937961+0000", 2024-09-21T14:06:30.776 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.343633148 2024-09-21T14:06:30.776 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.776 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.776 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:06:30.776 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.938002+0000", 2024-09-21T14:06:30.777 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0454000000000003e-05 2024-09-21T14:06:30.777 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:30.777 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:30.777 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:30.777 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.777 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.777 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25421.0:113910 9.5 e296/273 9:a516f056:::benchmark_data_smithi049_138504_object113909:head v 296'7354, mlcod=296'7351)", 2024-09-21T14:06:30.777 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:59:25.597837+0000", 2024-09-21T14:06:30.777 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 425.16613496399998, 2024-09-21T14:06:30.777 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.346102413, 2024-09-21T14:06:30.777 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:06:30.777 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:06:30.777 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:30.778 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:06:30.778 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.778 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:06:30.778 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.597837+0000", 2024-09-21T14:06:30.778 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:30.778 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.778 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.778 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:06:30.778 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.597837+0000", 2024-09-21T14:06:30.778 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:30.778 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.778 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.778 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:06:30.779 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.597838+0000", 2024-09-21T14:06:30.779 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3909999999999999e-06 2024-09-21T14:06:30.779 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.779 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.779 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:06:30.779 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.597870+0000", 2024-09-21T14:06:30.779 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2209000000000003e-05 2024-09-21T14:06:30.779 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.779 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.779 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:06:30.779 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.937999+0000", 2024-09-21T14:06:30.779 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.340128284 2024-09-21T14:06:30.779 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.779 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.779 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:06:30.780 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.938005+0000", 2024-09-21T14:06:30.780 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2870000000000001e-06 2024-09-21T14:06:30.780 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.780 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.780 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:30.780 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.938058+0000", 2024-09-21T14:06:30.780 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2608000000000002e-05 2024-09-21T14:06:30.780 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.780 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.780 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:06:30.780 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.938086+0000", 2024-09-21T14:06:30.780 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.8699000000000001e-05 2024-09-21T14:06:30.780 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.780 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.780 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:06:30.781 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.943789+0000", 2024-09-21T14:06:30.781 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0057023810000000003 2024-09-21T14:06:30.781 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.781 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.781 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:06:30.781 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.943939+0000", 2024-09-21T14:06:30.781 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00015055400000000001 2024-09-21T14:06:30.781 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:30.781 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:30.781 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:30.781 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.781 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.781 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25421.0:113922 9.b e296/278 9:db1a22c4:::benchmark_data_smithi049_138504_object113921:head v 296'7128, mlcod=296'7125)", 2024-09-21T14:06:30.781 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T13:59:25.605556+0000", 2024-09-21T14:06:30.782 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 425.15841572800002, 2024-09-21T14:06:30.782 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.34451127500000001, 2024-09-21T14:06:30.782 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:06:30.782 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:06:30.782 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:30.782 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:06:30.782 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.782 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:06:30.782 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.605556+0000", 2024-09-21T14:06:30.782 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:30.782 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.782 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.782 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:06:30.783 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.605556+0000", 2024-09-21T14:06:30.783 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:30.783 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.783 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.783 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:06:30.783 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.605557+0000", 2024-09-21T14:06:30.783 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.184e-06 2024-09-21T14:06:30.783 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.783 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.783 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:06:30.783 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.605595+0000", 2024-09-21T14:06:30.783 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8139000000000002e-05 2024-09-21T14:06:30.783 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.783 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.783 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:06:30.784 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.937938+0000", 2024-09-21T14:06:30.784 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.33234267899999997 2024-09-21T14:06:30.784 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.784 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.784 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:06:30.784 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.937946+0000", 2024-09-21T14:06:30.784 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9240000000000007e-06 2024-09-21T14:06:30.784 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.784 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.784 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:30.784 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.938185+0000", 2024-09-21T14:06:30.784 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000239108 2024-09-21T14:06:30.784 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.784 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.785 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:06:30.785 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.938206+0000", 2024-09-21T14:06:30.785 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0693999999999999e-05 2024-09-21T14:06:30.785 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.785 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.785 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:06:30.785 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.949931+0000", 2024-09-21T14:06:30.785 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.011725279 2024-09-21T14:06:30.785 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.785 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.785 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:06:30.785 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T13:59:25.950067+0000", 2024-09-21T14:06:30.785 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013626799999999999 2024-09-21T14:06:30.785 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:30.785 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:30.786 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:30.786 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.786 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.786 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:148200 10.8 10:11441144:::benchmark_data_smithi049_176811_object20623:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-09-21T14:06:30.786 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:04:55.587007+0000", 2024-09-21T14:06:30.786 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 95.176965031999998, 2024-09-21T14:06:30.786 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.019040647, 2024-09-21T14:06:30.786 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:06:30.786 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:06:30.786 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:30.786 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:06:30.786 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:06:30.786 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:06:30.787 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 148200 2024-09-21T14:06:30.787 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.787 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:06:30.787 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.787 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:06:30.787 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.587007+0000", 2024-09-21T14:06:30.787 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:30.787 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.787 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.787 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:06:30.787 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.587007+0000", 2024-09-21T14:06:30.787 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:30.787 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.788 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.788 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:06:30.788 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.587008+0000", 2024-09-21T14:06:30.788 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.649e-06 2024-09-21T14:06:30.788 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.788 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.788 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:06:30.788 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.587014+0000", 2024-09-21T14:06:30.788 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5849999999999999e-06 2024-09-21T14:06:30.788 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.788 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.788 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:06:30.788 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.587031+0000", 2024-09-21T14:06:30.789 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6756e-05 2024-09-21T14:06:30.789 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.789 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.789 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:06:30.789 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.587040+0000", 2024-09-21T14:06:30.789 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3510000000000008e-06 2024-09-21T14:06:30.789 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.789 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.789 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:30.789 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.587135+0000", 2024-09-21T14:06:30.789 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4471999999999999e-05 2024-09-21T14:06:30.789 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.790 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.790 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:06:30.790 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.587145+0000", 2024-09-21T14:06:30.790 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0186e-05 2024-09-21T14:06:30.790 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.790 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.790 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:30.790 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.598177+0000", 2024-09-21T14:06:30.790 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0110321339999999 2024-09-21T14:06:30.790 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.790 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.790 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:06:30.790 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.599612+0000", 2024-09-21T14:06:30.790 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001434828 2024-09-21T14:06:30.791 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.791 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.791 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,7", 2024-09-21T14:06:30.791 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.599704+0000", 2024-09-21T14:06:30.791 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2164999999999996e-05 2024-09-21T14:06:30.791 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.791 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.791 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:06:30.791 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.601928+0000", 2024-09-21T14:06:30.791 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0022241869999999999 2024-09-21T14:06:30.791 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.791 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.791 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:30.791 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.605855+0000", 2024-09-21T14:06:30.792 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0039272839999999996 2024-09-21T14:06:30.792 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.792 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.792 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:30.792 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.605933+0000", 2024-09-21T14:06:30.792 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7194999999999995e-05 2024-09-21T14:06:30.792 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.792 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.792 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:06:30.792 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.605997+0000", 2024-09-21T14:06:30.792 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4182000000000003e-05 2024-09-21T14:06:30.792 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.792 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.792 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:06:30.792 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.606047+0000", 2024-09-21T14:06:30.793 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0673000000000001e-05 2024-09-21T14:06:30.793 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:30.793 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:30.793 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:30.793 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.793 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.793 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:148204 10.8 10:173acc92:::benchmark_data_smithi049_176811_object20627:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-09-21T14:06:30.793 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:04:55.598766+0000", 2024-09-21T14:06:30.793 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 95.165206108999996, 2024-09-21T14:06:30.793 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.012530269, 2024-09-21T14:06:30.793 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:06:30.793 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:06:30.793 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:30.793 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:06:30.793 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:06:30.794 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:06:30.794 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 148204 2024-09-21T14:06:30.794 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.794 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:06:30.794 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.794 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:06:30.794 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.598766+0000", 2024-09-21T14:06:30.794 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:30.794 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.794 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.794 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:06:30.794 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.598766+0000", 2024-09-21T14:06:30.794 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:30.794 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.795 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.795 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:06:30.795 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.598768+0000", 2024-09-21T14:06:30.795 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.951e-06 2024-09-21T14:06:30.795 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.795 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.795 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:06:30.795 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.598775+0000", 2024-09-21T14:06:30.795 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8450000000000003e-06 2024-09-21T14:06:30.795 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.795 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.795 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:06:30.795 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.598793+0000", 2024-09-21T14:06:30.795 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7348999999999999e-05 2024-09-21T14:06:30.796 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.796 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.796 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:06:30.796 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.598803+0000", 2024-09-21T14:06:30.796 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8120000000000002e-06 2024-09-21T14:06:30.796 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.796 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.796 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:30.797 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.598888+0000", 2024-09-21T14:06:30.797 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4975999999999994e-05 2024-09-21T14:06:30.797 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.797 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.797 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:06:30.797 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.598898+0000", 2024-09-21T14:06:30.797 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0899e-05 2024-09-21T14:06:30.797 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.797 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.797 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:30.797 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.601975+0000", 2024-09-21T14:06:30.797 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0030761989999999 2024-09-21T14:06:30.797 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.797 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.797 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:06:30.798 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.602095+0000", 2024-09-21T14:06:30.798 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012031699999999999 2024-09-21T14:06:30.798 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.798 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.798 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,7", 2024-09-21T14:06:30.798 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.602214+0000", 2024-09-21T14:06:30.798 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011874700000000001 2024-09-21T14:06:30.798 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.798 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.798 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:06:30.798 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.606106+0000", 2024-09-21T14:06:30.798 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0038918389999999998 2024-09-21T14:06:30.798 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.799 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.799 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:30.799 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.606149+0000", 2024-09-21T14:06:30.799 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3593000000000001e-05 2024-09-21T14:06:30.799 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.799 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.799 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:30.799 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.611191+0000", 2024-09-21T14:06:30.799 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0050419280000000002 2024-09-21T14:06:30.799 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.799 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.799 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:06:30.799 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.611250+0000", 2024-09-21T14:06:30.799 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8956000000000003e-05 2024-09-21T14:06:30.800 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.800 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.800 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:06:30.800 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.611296+0000", 2024-09-21T14:06:30.800 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5858e-05 2024-09-21T14:06:30.800 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:30.800 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:30.800 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:30.800 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.800 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.800 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:148276 10.8 10:1259e2cf:::benchmark_data_smithi049_176811_object20699:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-09-21T14:06:30.800 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:04:55.670175+0000", 2024-09-21T14:06:30.800 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 95.093796558999998, 2024-09-21T14:06:30.800 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.94134552400000004, 2024-09-21T14:06:30.800 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:06:30.800 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:06:30.801 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:30.801 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:06:30.801 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:06:30.801 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:06:30.801 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 148276 2024-09-21T14:06:30.801 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.801 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:06:30.801 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.801 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:06:30.801 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.670175+0000", 2024-09-21T14:06:30.801 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:30.801 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.801 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.801 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:06:30.801 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.670175+0000", 2024-09-21T14:06:30.802 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:30.802 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.802 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.802 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:06:30.802 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.670177+0000", 2024-09-21T14:06:30.802 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.911e-06 2024-09-21T14:06:30.802 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.802 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.802 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:06:30.802 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.670183+0000", 2024-09-21T14:06:30.802 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1190000000000002e-06 2024-09-21T14:06:30.802 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.802 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.802 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:06:30.802 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.670198+0000", 2024-09-21T14:06:30.803 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5204e-05 2024-09-21T14:06:30.803 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.803 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.803 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:06:30.803 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.670207+0000", 2024-09-21T14:06:30.803 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1710000000000002e-06 2024-09-21T14:06:30.803 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.803 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.803 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:30.803 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.670292+0000", 2024-09-21T14:06:30.803 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5241999999999995e-05 2024-09-21T14:06:30.803 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.803 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.803 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:06:30.804 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.670302+0000", 2024-09-21T14:06:30.804 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.009e-05 2024-09-21T14:06:30.804 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.804 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.804 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:30.804 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.603863+0000", 2024-09-21T14:06:30.804 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.93356139299999996 2024-09-21T14:06:30.804 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.804 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.804 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:06:30.804 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.603979+0000", 2024-09-21T14:06:30.804 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000116057 2024-09-21T14:06:30.804 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.804 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.805 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,7", 2024-09-21T14:06:30.805 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.604090+0000", 2024-09-21T14:06:30.805 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011069900000000001 2024-09-21T14:06:30.805 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.805 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.805 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:06:30.805 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.606163+0000", 2024-09-21T14:06:30.805 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0020731040000000001 2024-09-21T14:06:30.805 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.805 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.805 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:30.805 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.606226+0000", 2024-09-21T14:06:30.805 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3025000000000001e-05 2024-09-21T14:06:30.805 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.806 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.806 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:30.806 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.611418+0000", 2024-09-21T14:06:30.806 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0051918659999999998 2024-09-21T14:06:30.806 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.806 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.806 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:06:30.806 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.611466+0000", 2024-09-21T14:06:30.806 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8284000000000001e-05 2024-09-21T14:06:30.806 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.806 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.806 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:06:30.806 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.611521+0000", 2024-09-21T14:06:30.806 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4358999999999999e-05 2024-09-21T14:06:30.807 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:30.807 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:30.807 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:30.807 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.807 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.807 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:148281 10.8 10:11e48ab0:::benchmark_data_smithi049_176811_object20704:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-09-21T14:06:30.807 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:04:55.671888+0000", 2024-09-21T14:06:30.807 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 95.092083291999998, 2024-09-21T14:06:30.807 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.93981524800000005, 2024-09-21T14:06:30.807 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:06:30.807 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:06:30.807 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:30.807 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:06:30.807 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:06:30.808 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:06:30.808 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 148281 2024-09-21T14:06:30.808 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.808 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:06:30.808 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.808 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:06:30.808 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.671888+0000", 2024-09-21T14:06:30.808 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:30.808 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.808 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.808 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:06:30.808 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.671888+0000", 2024-09-21T14:06:30.808 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:30.808 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.808 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.809 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:06:30.809 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.671889+0000", 2024-09-21T14:06:30.809 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.009e-06 2024-09-21T14:06:30.809 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.809 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.809 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:06:30.809 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.671894+0000", 2024-09-21T14:06:30.809 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2459999999999997e-06 2024-09-21T14:06:30.809 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.809 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.809 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:06:30.809 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.671904+0000", 2024-09-21T14:06:30.809 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0505e-05 2024-09-21T14:06:30.809 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.809 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.809 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:06:30.810 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.671910+0000", 2024-09-21T14:06:30.810 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3329999999999996e-06 2024-09-21T14:06:30.810 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.810 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.810 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:30.810 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.671956+0000", 2024-09-21T14:06:30.810 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6570000000000003e-05 2024-09-21T14:06:30.810 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.810 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.810 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:06:30.810 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.671962+0000", 2024-09-21T14:06:30.810 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6659999999999996e-06 2024-09-21T14:06:30.810 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.810 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.811 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:30.811 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.606281+0000", 2024-09-21T14:06:30.811 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.93431893200000005 2024-09-21T14:06:30.811 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.811 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.811 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:06:30.811 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.606381+0000", 2024-09-21T14:06:30.811 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010040200000000001 2024-09-21T14:06:30.811 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.811 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.812 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,7", 2024-09-21T14:06:30.812 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.606453+0000", 2024-09-21T14:06:30.812 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1423999999999999e-05 2024-09-21T14:06:30.812 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.812 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.812 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:30.812 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.611355+0000", 2024-09-21T14:06:30.812 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0049022880000000003 2024-09-21T14:06:30.812 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.812 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.812 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:06:30.812 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.611534+0000", 2024-09-21T14:06:30.812 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000179116 2024-09-21T14:06:30.812 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.813 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.813 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:30.813 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.611636+0000", 2024-09-21T14:06:30.813 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000101909 2024-09-21T14:06:30.813 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.813 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.813 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:06:30.813 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.611666+0000", 2024-09-21T14:06:30.813 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.0468e-05 2024-09-21T14:06:30.813 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.813 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.813 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:06:30.814 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.611704+0000", 2024-09-21T14:06:30.814 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7379999999999998e-05 2024-09-21T14:06:30.814 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:30.814 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:30.814 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:30.814 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.814 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.814 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:148288 10.8 10:1610cc8e:::benchmark_data_smithi049_176811_object20711:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-09-21T14:06:30.814 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:04:55.675295+0000", 2024-09-21T14:06:30.814 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 95.088676319000001, 2024-09-21T14:06:30.815 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.94425190999999997, 2024-09-21T14:06:30.815 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:06:30.815 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:06:30.815 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:30.815 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:06:30.815 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:06:30.815 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:06:30.815 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 148288 2024-09-21T14:06:30.815 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.815 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:06:30.815 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.815 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:06:30.815 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.675295+0000", 2024-09-21T14:06:30.815 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:30.816 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.816 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.816 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:06:30.816 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.675295+0000", 2024-09-21T14:06:30.816 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:30.816 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.816 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.816 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:06:30.816 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.675297+0000", 2024-09-21T14:06:30.816 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4890000000000001e-06 2024-09-21T14:06:30.816 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.816 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.817 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:06:30.817 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.675302+0000", 2024-09-21T14:06:30.817 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8250000000000004e-06 2024-09-21T14:06:30.817 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.817 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.817 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:06:30.817 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.675321+0000", 2024-09-21T14:06:30.817 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8981999999999999e-05 2024-09-21T14:06:30.817 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.817 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.817 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:06:30.817 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.675329+0000", 2024-09-21T14:06:30.817 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2940000000000006e-06 2024-09-21T14:06:30.817 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.818 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.818 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:30.818 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.675411+0000", 2024-09-21T14:06:30.818 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1927000000000004e-05 2024-09-21T14:06:30.818 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.818 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.818 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:06:30.818 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.675421+0000", 2024-09-21T14:06:30.818 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9389999999999992e-06 2024-09-21T14:06:30.818 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.818 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.818 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:30.818 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.608831+0000", 2024-09-21T14:06:30.819 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.93340988899999999 2024-09-21T14:06:30.819 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.819 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.819 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:06:30.819 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.608931+0000", 2024-09-21T14:06:30.819 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000100715 2024-09-21T14:06:30.819 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.819 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.819 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,7", 2024-09-21T14:06:30.819 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.609052+0000", 2024-09-21T14:06:30.819 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000120139 2024-09-21T14:06:30.819 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.820 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.820 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:06:30.820 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.611541+0000", 2024-09-21T14:06:30.820 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0024891079999999999 2024-09-21T14:06:30.820 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.820 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.820 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:30.820 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.611824+0000", 2024-09-21T14:06:30.820 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00028299899999999998 2024-09-21T14:06:30.820 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.820 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.820 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:30.820 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.619454+0000", 2024-09-21T14:06:30.820 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.007630375 2024-09-21T14:06:30.820 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.821 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.821 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:06:30.821 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.619487+0000", 2024-09-21T14:06:30.821 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2790000000000003e-05 2024-09-21T14:06:30.821 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.821 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.821 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:06:30.821 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.619547+0000", 2024-09-21T14:06:30.821 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0439000000000002e-05 2024-09-21T14:06:30.821 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:30.821 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:30.821 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:30.821 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.821 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.821 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:148289 10.8 10:19cc1202:::benchmark_data_smithi049_176811_object20712:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-09-21T14:06:30.822 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:04:55.675487+0000", 2024-09-21T14:06:30.822 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 95.088485008999996, 2024-09-21T14:06:30.822 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.95131562199999997, 2024-09-21T14:06:30.822 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:06:30.822 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:06:30.822 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:30.822 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:06:30.822 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:06:30.822 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:06:30.822 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 148289 2024-09-21T14:06:30.822 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.822 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:06:30.822 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.822 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:06:30.823 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.675487+0000", 2024-09-21T14:06:30.823 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:30.823 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.823 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.823 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:06:30.823 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.675487+0000", 2024-09-21T14:06:30.823 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:30.823 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.823 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.823 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:06:30.823 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.675488+0000", 2024-09-21T14:06:30.823 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4200000000000005e-07 2024-09-21T14:06:30.823 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.823 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.823 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:06:30.824 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:06:30.825 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.675507+0000", 2024-09-21T14:06:30.825 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9219999999999999e-05 2024-09-21T14:06:30.825 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.825 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.825 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:06:30.825 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.675521+0000", 2024-09-21T14:06:30.825 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4688999999999999e-05 2024-09-21T14:06:30.825 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.825 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.825 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:06:30.825 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.675529+0000", 2024-09-21T14:06:30.825 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2599999999999999e-06 2024-09-21T14:06:30.826 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.826 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.826 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:30.826 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.675588+0000", 2024-09-21T14:06:30.826 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8936e-05 2024-09-21T14:06:30.826 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.826 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.826 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:06:30.826 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.675597+0000", 2024-09-21T14:06:30.826 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6889999999999999e-06 2024-09-21T14:06:30.826 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.826 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.826 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:30.827 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.611876+0000", 2024-09-21T14:06:30.827 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.93627835500000001 2024-09-21T14:06:30.827 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.827 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.827 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:06:30.827 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.611995+0000", 2024-09-21T14:06:30.827 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000119517 2024-09-21T14:06:30.827 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.827 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.827 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,7", 2024-09-21T14:06:30.827 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.612090+0000", 2024-09-21T14:06:30.827 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4495999999999998e-05 2024-09-21T14:06:30.827 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.827 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.827 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:30.828 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.619609+0000", 2024-09-21T14:06:30.828 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0075196880000000001 2024-09-21T14:06:30.828 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.828 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.828 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:06:30.828 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.619628+0000", 2024-09-21T14:06:30.828 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8425000000000001e-05 2024-09-21T14:06:30.828 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.828 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.828 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:30.828 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.626706+0000", 2024-09-21T14:06:30.828 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0070784200000000002 2024-09-21T14:06:30.828 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.828 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.828 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:06:30.828 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.626757+0000", 2024-09-21T14:06:30.829 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0977999999999999e-05 2024-09-21T14:06:30.829 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.829 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.829 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:06:30.829 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.626802+0000", 2024-09-21T14:06:30.829 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5107e-05 2024-09-21T14:06:30.829 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:30.829 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:30.829 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:30.829 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.829 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.829 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:148298 10.8 10:1042f94c:::benchmark_data_smithi049_176811_object20721:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-09-21T14:06:30.829 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:04:55.686237+0000", 2024-09-21T14:06:30.829 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 95.077734880999998, 2024-09-21T14:06:30.829 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.94072119499999995, 2024-09-21T14:06:30.829 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:06:30.830 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:06:30.830 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:30.830 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:06:30.830 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:06:30.830 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:06:30.830 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 148298 2024-09-21T14:06:30.830 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.830 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:06:30.830 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.830 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:06:30.830 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.686237+0000", 2024-09-21T14:06:30.830 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:30.830 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.830 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.831 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:06:30.831 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.686237+0000", 2024-09-21T14:06:30.831 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:30.831 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.831 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.831 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:06:30.831 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.686238+0000", 2024-09-21T14:06:30.831 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5519999999999999e-06 2024-09-21T14:06:30.831 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.831 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.831 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:06:30.831 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.686245+0000", 2024-09-21T14:06:30.831 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1449999999999996e-06 2024-09-21T14:06:30.831 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.831 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.832 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:06:30.832 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.686262+0000", 2024-09-21T14:06:30.832 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7258999999999999e-05 2024-09-21T14:06:30.832 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.832 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.832 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:06:30.832 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.686270+0000", 2024-09-21T14:06:30.832 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6820000000000003e-06 2024-09-21T14:06:30.832 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.832 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.832 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:30.832 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.686328+0000", 2024-09-21T14:06:30.832 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8081999999999997e-05 2024-09-21T14:06:30.832 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.832 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.832 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:06:30.833 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.686337+0000", 2024-09-21T14:06:30.833 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7159999999999998e-06 2024-09-21T14:06:30.833 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.833 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.833 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:30.833 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.614591+0000", 2024-09-21T14:06:30.833 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.92825406499999996 2024-09-21T14:06:30.833 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.833 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.833 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:06:30.833 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.614697+0000", 2024-09-21T14:06:30.833 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010538400000000001 2024-09-21T14:06:30.833 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.833 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.834 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,7", 2024-09-21T14:06:30.834 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.614812+0000", 2024-09-21T14:06:30.834 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000115492 2024-09-21T14:06:30.834 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.834 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.834 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:06:30.834 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.619633+0000", 2024-09-21T14:06:30.834 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0048206730000000001 2024-09-21T14:06:30.834 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.834 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.834 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:30.834 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.619684+0000", 2024-09-21T14:06:30.834 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1159000000000003e-05 2024-09-21T14:06:30.834 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.835 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.835 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:30.835 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.626880+0000", 2024-09-21T14:06:30.835 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0071960069999999999 2024-09-21T14:06:30.835 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.835 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.835 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:06:30.835 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.626926+0000", 2024-09-21T14:06:30.835 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5794999999999998e-05 2024-09-21T14:06:30.835 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.835 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.835 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:06:30.835 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.626958+0000", 2024-09-21T14:06:30.835 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2184000000000002e-05 2024-09-21T14:06:30.835 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:30.836 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:30.836 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:30.836 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.836 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.836 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:148326 10.8 10:190bfd8a:::benchmark_data_smithi049_176811_object20749:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-09-21T14:06:30.836 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:04:55.712544+0000", 2024-09-21T14:06:30.836 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 95.051427454999995, 2024-09-21T14:06:30.836 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.91455890500000003, 2024-09-21T14:06:30.836 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:06:30.836 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:06:30.836 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:30.836 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:06:30.836 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:06:30.836 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:06:30.836 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 148326 2024-09-21T14:06:30.836 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.837 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:06:30.837 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.837 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:06:30.837 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.712544+0000", 2024-09-21T14:06:30.837 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:30.837 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.837 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.837 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:06:30.837 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.712544+0000", 2024-09-21T14:06:30.837 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:30.837 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.837 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.837 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:06:30.837 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.712546+0000", 2024-09-21T14:06:30.838 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4079999999999999e-06 2024-09-21T14:06:30.838 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.838 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.838 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:06:30.838 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.712551+0000", 2024-09-21T14:06:30.838 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4890000000000003e-06 2024-09-21T14:06:30.838 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.838 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.838 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:06:30.838 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.712566+0000", 2024-09-21T14:06:30.838 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5203e-05 2024-09-21T14:06:30.838 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.838 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.838 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:06:30.839 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.712573+0000", 2024-09-21T14:06:30.839 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3670000000000002e-06 2024-09-21T14:06:30.839 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.839 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.839 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:30.839 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.712633+0000", 2024-09-21T14:06:30.839 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0538000000000002e-05 2024-09-21T14:06:30.839 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.839 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.839 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:06:30.839 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.712643+0000", 2024-09-21T14:06:30.839 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0032e-05 2024-09-21T14:06:30.839 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.839 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.840 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:30.840 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.617061+0000", 2024-09-21T14:06:30.840 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.90441751699999995 2024-09-21T14:06:30.840 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.840 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.840 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:06:30.840 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.617158+0000", 2024-09-21T14:06:30.840 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6871000000000001e-05 2024-09-21T14:06:30.840 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.840 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.840 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,7", 2024-09-21T14:06:30.840 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.617226+0000", 2024-09-21T14:06:30.840 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8147999999999999e-05 2024-09-21T14:06:30.840 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.840 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.841 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:06:30.841 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.623899+0000", 2024-09-21T14:06:30.841 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0066735259999999999 2024-09-21T14:06:30.841 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.841 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.841 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:30.841 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.626846+0000", 2024-09-21T14:06:30.841 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002946204 2024-09-21T14:06:30.841 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.841 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.841 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:30.841 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.627031+0000", 2024-09-21T14:06:30.841 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00018589100000000001 2024-09-21T14:06:30.842 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.842 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.842 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:06:30.842 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.627071+0000", 2024-09-21T14:06:30.842 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9215999999999999e-05 2024-09-21T14:06:30.842 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.842 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.842 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:06:30.842 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.627103+0000", 2024-09-21T14:06:30.842 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2495e-05 2024-09-21T14:06:30.842 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:30.842 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:30.842 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:30.842 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.842 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.843 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:148345 10.8 10:1ca635b1:::benchmark_data_smithi049_176811_object20768:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-09-21T14:06:30.843 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:04:55.730454+0000", 2024-09-21T14:06:30.843 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 95.033517665000005, 2024-09-21T14:06:30.843 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.90786199499999998, 2024-09-21T14:06:30.843 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:06:30.843 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:06:30.843 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:30.843 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:06:30.843 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:06:30.843 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:06:30.843 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 148345 2024-09-21T14:06:30.843 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.843 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:06:30.843 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.844 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:06:30.844 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.730454+0000", 2024-09-21T14:06:30.844 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:30.844 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.844 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.844 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:06:30.844 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.730454+0000", 2024-09-21T14:06:30.844 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:30.844 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.844 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.844 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:06:30.844 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.730456+0000", 2024-09-21T14:06:30.844 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.668e-06 2024-09-21T14:06:30.844 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.844 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.844 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:06:30.845 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.730463+0000", 2024-09-21T14:06:30.845 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8240000000000002e-06 2024-09-21T14:06:30.845 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.845 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.845 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:06:30.845 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.730478+0000", 2024-09-21T14:06:30.845 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5364999999999999e-05 2024-09-21T14:06:30.845 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.845 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.845 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:06:30.845 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.730486+0000", 2024-09-21T14:06:30.845 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.225e-06 2024-09-21T14:06:30.845 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.845 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.845 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:30.845 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.730567+0000", 2024-09-21T14:06:30.846 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0895000000000004e-05 2024-09-21T14:06:30.846 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.846 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.846 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:06:30.846 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.730577+0000", 2024-09-21T14:06:30.846 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7569999999999999e-06 2024-09-21T14:06:30.846 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.846 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.846 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:30.846 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.619717+0000", 2024-09-21T14:06:30.846 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.88914048199999995 2024-09-21T14:06:30.846 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.846 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.846 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:06:30.846 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.619824+0000", 2024-09-21T14:06:30.847 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000107046 2024-09-21T14:06:30.847 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.847 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.847 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,7", 2024-09-21T14:06:30.847 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.619921+0000", 2024-09-21T14:06:30.847 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6459000000000006e-05 2024-09-21T14:06:30.847 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.847 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.847 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:30.847 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.626996+0000", 2024-09-21T14:06:30.847 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0070750450000000003 2024-09-21T14:06:30.847 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.847 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.847 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:30.848 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.632227+0000", 2024-09-21T14:06:30.848 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0052307669999999999 2024-09-21T14:06:30.848 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.848 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.848 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:06:30.848 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.638223+0000", 2024-09-21T14:06:30.848 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.005996159 2024-09-21T14:06:30.848 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.848 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.848 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:06:30.848 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.638272+0000", 2024-09-21T14:06:30.848 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8837e-05 2024-09-21T14:06:30.848 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.848 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.849 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:06:30.849 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.638316+0000", 2024-09-21T14:06:30.849 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4465999999999997e-05 2024-09-21T14:06:30.849 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:30.849 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:30.849 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:30.849 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.849 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.849 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:148362 10.8 10:1e3ef663:::benchmark_data_smithi049_176811_object20785:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-09-21T14:06:30.849 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:04:55.747088+0000", 2024-09-21T14:06:30.849 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 95.016883624000002, 2024-09-21T14:06:30.849 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.89129988000000004, 2024-09-21T14:06:30.850 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:06:30.850 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:06:30.850 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:30.850 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:06:30.850 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:06:30.850 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:06:30.850 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 148362 2024-09-21T14:06:30.850 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.850 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:06:30.850 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.850 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:06:30.850 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.747088+0000", 2024-09-21T14:06:30.850 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:30.850 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.851 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.851 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:06:30.851 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.747088+0000", 2024-09-21T14:06:30.852 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:30.852 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.852 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.852 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:06:30.852 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.747090+0000", 2024-09-21T14:06:30.852 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1409999999999999e-06 2024-09-21T14:06:30.852 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.852 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.853 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:06:30.853 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.747097+0000", 2024-09-21T14:06:30.853 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.409e-06 2024-09-21T14:06:30.853 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.853 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.853 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:06:30.853 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.747117+0000", 2024-09-21T14:06:30.853 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0125000000000001e-05 2024-09-21T14:06:30.853 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.853 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.853 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:06:30.853 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.747125+0000", 2024-09-21T14:06:30.853 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0059999999999992e-06 2024-09-21T14:06:30.853 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.854 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.854 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:30.854 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.747263+0000", 2024-09-21T14:06:30.854 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013833000000000001 2024-09-21T14:06:30.854 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.854 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.854 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:06:30.854 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.747274+0000", 2024-09-21T14:06:30.854 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1207e-05 2024-09-21T14:06:30.854 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.854 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.854 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:30.854 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.623995+0000", 2024-09-21T14:06:30.854 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.87672050400000001 2024-09-21T14:06:30.855 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.855 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.855 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:06:30.855 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.624107+0000", 2024-09-21T14:06:30.855 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011234300000000001 2024-09-21T14:06:30.855 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.855 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.855 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,7", 2024-09-21T14:06:30.855 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.624240+0000", 2024-09-21T14:06:30.855 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000132922 2024-09-21T14:06:30.855 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.855 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.855 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:30.855 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.632186+0000", 2024-09-21T14:06:30.856 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0079461110000000005 2024-09-21T14:06:30.856 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.856 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.856 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:30.856 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.632268+0000", 2024-09-21T14:06:30.856 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1803000000000003e-05 2024-09-21T14:06:30.856 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.856 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.856 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:06:30.856 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.638322+0000", 2024-09-21T14:06:30.856 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0060544509999999998 2024-09-21T14:06:30.856 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.856 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.856 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:06:30.857 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.638358+0000", 2024-09-21T14:06:30.857 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5599999999999998e-05 2024-09-21T14:06:30.857 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.857 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.857 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:06:30.857 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.638388+0000", 2024-09-21T14:06:30.857 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9927999999999999e-05 2024-09-21T14:06:30.857 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:30.857 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:30.857 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:30.857 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.857 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.857 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:148377 10.8 10:167b8c70:::benchmark_data_smithi049_176811_object20800:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-09-21T14:06:30.857 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:04:55.764470+0000", 2024-09-21T14:06:30.858 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 94.999501554000005, 2024-09-21T14:06:30.858 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.87398584000000001, 2024-09-21T14:06:30.858 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:06:30.858 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:06:30.858 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:30.858 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:06:30.858 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:06:30.858 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:06:30.858 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 148377 2024-09-21T14:06:30.858 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.858 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:06:30.858 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.858 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:06:30.858 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.764470+0000", 2024-09-21T14:06:30.858 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:30.859 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.859 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.859 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:06:30.859 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.764470+0000", 2024-09-21T14:06:30.859 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:30.859 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.859 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.859 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:06:30.859 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.764472+0000", 2024-09-21T14:06:30.859 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.663e-06 2024-09-21T14:06:30.859 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.860 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.860 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:06:30.860 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.764479+0000", 2024-09-21T14:06:30.860 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0849999999999999e-06 2024-09-21T14:06:30.860 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.860 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.860 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:06:30.860 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.764506+0000", 2024-09-21T14:06:30.860 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6900999999999999e-05 2024-09-21T14:06:30.860 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.860 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.860 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:06:30.860 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.764515+0000", 2024-09-21T14:06:30.861 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1360000000000003e-06 2024-09-21T14:06:30.861 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.861 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.861 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:30.861 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.764593+0000", 2024-09-21T14:06:30.861 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8506999999999999e-05 2024-09-21T14:06:30.861 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.861 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.861 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:06:30.861 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.764605+0000", 2024-09-21T14:06:30.861 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1321000000000001e-05 2024-09-21T14:06:30.861 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.861 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.861 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:30.861 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.627131+0000", 2024-09-21T14:06:30.862 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.862526499 2024-09-21T14:06:30.862 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.862 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.862 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:06:30.862 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.627233+0000", 2024-09-21T14:06:30.862 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010123 2024-09-21T14:06:30.862 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.862 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.862 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,7", 2024-09-21T14:06:30.862 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.627334+0000", 2024-09-21T14:06:30.862 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000101437 2024-09-21T14:06:30.862 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.862 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.862 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:30.863 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.638176+0000", 2024-09-21T14:06:30.863 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.010841991 2024-09-21T14:06:30.863 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.863 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.863 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:30.863 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.638213+0000", 2024-09-21T14:06:30.863 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6758999999999998e-05 2024-09-21T14:06:30.863 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.863 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.863 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:06:30.863 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.638393+0000", 2024-09-21T14:06:30.863 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00018035999999999999 2024-09-21T14:06:30.863 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.863 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.863 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:06:30.864 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.638427+0000", 2024-09-21T14:06:30.864 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4155000000000001e-05 2024-09-21T14:06:30.864 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.864 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.864 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:06:30.864 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.638456+0000", 2024-09-21T14:06:30.864 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.8796e-05 2024-09-21T14:06:30.864 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:30.864 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:30.864 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:30.864 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.864 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.864 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:148380 10.8 10:184ea8ff:::benchmark_data_smithi049_176811_object20803:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-09-21T14:06:30.864 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:04:55.765967+0000", 2024-09-21T14:06:30.864 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 94.998005157999998, 2024-09-21T14:06:30.865 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.88603139099999995, 2024-09-21T14:06:30.865 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:06:30.865 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:06:30.865 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:30.865 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:06:30.865 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:06:30.865 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:06:30.865 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 148380 2024-09-21T14:06:30.865 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.865 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:06:30.865 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.865 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:06:30.865 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.765967+0000", 2024-09-21T14:06:30.865 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:30.865 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.866 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.866 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:06:30.866 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.765967+0000", 2024-09-21T14:06:30.866 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:30.866 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.866 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.866 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:06:30.866 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.765968+0000", 2024-09-21T14:06:30.866 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3030000000000001e-06 2024-09-21T14:06:30.866 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.866 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.867 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:06:30.867 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.765974+0000", 2024-09-21T14:06:30.867 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3999999999999997e-06 2024-09-21T14:06:30.867 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.867 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.867 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:06:30.867 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.765995+0000", 2024-09-21T14:06:30.867 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0267000000000001e-05 2024-09-21T14:06:30.867 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.867 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.867 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:06:30.867 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.766004+0000", 2024-09-21T14:06:30.867 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4730000000000006e-06 2024-09-21T14:06:30.867 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.867 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.868 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:30.868 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.766078+0000", 2024-09-21T14:06:30.868 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3850000000000006e-05 2024-09-21T14:06:30.868 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.868 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.868 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:06:30.868 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.766089+0000", 2024-09-21T14:06:30.868 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.119e-05 2024-09-21T14:06:30.868 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.868 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.868 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:30.868 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.629790+0000", 2024-09-21T14:06:30.868 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.86370065100000004 2024-09-21T14:06:30.868 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.869 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.869 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:06:30.869 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.629921+0000", 2024-09-21T14:06:30.869 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000131521 2024-09-21T14:06:30.869 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.869 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.869 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,7", 2024-09-21T14:06:30.869 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.630025+0000", 2024-09-21T14:06:30.869 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000103646 2024-09-21T14:06:30.869 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.869 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.869 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:06:30.869 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.638466+0000", 2024-09-21T14:06:30.870 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0084406240000000007 2024-09-21T14:06:30.870 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.870 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.870 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:30.870 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.638521+0000", 2024-09-21T14:06:30.870 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5426000000000001e-05 2024-09-21T14:06:30.870 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.870 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.870 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:30.870 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.651885+0000", 2024-09-21T14:06:30.870 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.013364407 2024-09-21T14:06:30.870 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.870 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.870 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:06:30.871 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.651942+0000", 2024-09-21T14:06:30.871 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6944000000000002e-05 2024-09-21T14:06:30.871 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.871 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.871 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:06:30.871 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.651998+0000", 2024-09-21T14:06:30.871 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5689000000000002e-05 2024-09-21T14:06:30.871 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:30.871 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:30.871 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:30.871 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.871 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.871 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:148382 10.8 10:17a8a665:::benchmark_data_smithi049_176811_object20805:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-09-21T14:06:30.871 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:04:55.771284+0000", 2024-09-21T14:06:30.871 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 94.992687853000007, 2024-09-21T14:06:30.872 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.88084546799999996, 2024-09-21T14:06:30.872 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:06:30.872 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:06:30.872 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:30.872 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:06:30.872 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:06:30.872 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:06:30.872 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 148382 2024-09-21T14:06:30.872 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.872 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:06:30.872 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.872 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:06:30.872 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.771284+0000", 2024-09-21T14:06:30.872 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:30.872 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.873 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.873 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:06:30.873 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.771284+0000", 2024-09-21T14:06:30.873 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:30.873 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.873 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.873 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:06:30.873 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.771286+0000", 2024-09-21T14:06:30.873 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.849e-06 2024-09-21T14:06:30.873 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.873 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.873 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:06:30.873 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.771291+0000", 2024-09-21T14:06:30.873 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5729999999999998e-06 2024-09-21T14:06:30.874 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.874 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.874 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:06:30.874 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.771308+0000", 2024-09-21T14:06:30.874 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6929999999999999e-05 2024-09-21T14:06:30.874 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.874 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.874 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:06:30.874 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.771316+0000", 2024-09-21T14:06:30.874 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5950000000000001e-06 2024-09-21T14:06:30.874 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.874 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.874 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:30.874 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.771378+0000", 2024-09-21T14:06:30.874 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2574000000000001e-05 2024-09-21T14:06:30.875 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.875 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.875 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:06:30.875 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.771388+0000", 2024-09-21T14:06:30.875 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0039e-05 2024-09-21T14:06:30.875 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.875 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.875 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:30.875 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.632299+0000", 2024-09-21T14:06:30.875 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.860910587 2024-09-21T14:06:30.875 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.875 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.875 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:06:30.875 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.632403+0000", 2024-09-21T14:06:30.876 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000103688 2024-09-21T14:06:30.876 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.876 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.876 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,7", 2024-09-21T14:06:30.876 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.632833+0000", 2024-09-21T14:06:30.876 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00042987599999999999 2024-09-21T14:06:30.876 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.876 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.876 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:06:30.876 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.638471+0000", 2024-09-21T14:06:30.876 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0056379120000000001 2024-09-21T14:06:30.876 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.876 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.876 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:30.876 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.638556+0000", 2024-09-21T14:06:30.877 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5357000000000003e-05 2024-09-21T14:06:30.877 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.877 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.877 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:30.877 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.652041+0000", 2024-09-21T14:06:30.877 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.01348482 2024-09-21T14:06:30.877 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.877 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.877 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:06:30.877 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.652091+0000", 2024-09-21T14:06:30.877 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0151999999999997e-05 2024-09-21T14:06:30.877 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.877 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.878 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:06:30.878 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.652129+0000", 2024-09-21T14:06:30.878 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8516000000000002e-05 2024-09-21T14:06:30.878 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:30.878 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:30.878 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:30.878 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.878 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.878 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:148383 10.8 10:1452c5c4:::benchmark_data_smithi049_176811_object20806:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-09-21T14:06:30.878 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:04:55.771736+0000", 2024-09-21T14:06:30.878 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 94.992235827000002, 2024-09-21T14:06:30.878 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.88066859799999997, 2024-09-21T14:06:30.878 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:06:30.878 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:06:30.878 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:30.879 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:06:30.879 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:06:30.879 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:06:30.879 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 148383 2024-09-21T14:06:30.879 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.879 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:06:30.879 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:30.880 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:30.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:06:30.881 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:06:30.881 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.771736+0000", 2024-09-21T14:06:30.881 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:30.881 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.881 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.881 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:06:30.881 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.771736+0000", 2024-09-21T14:06:30.881 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:30.881 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.881 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.881 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:06:30.881 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.771737+0000", 2024-09-21T14:06:30.881 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3099999999999996e-07 2024-09-21T14:06:30.882 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.882 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.882 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:06:30.882 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.771741+0000", 2024-09-21T14:06:30.882 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.019e-06 2024-09-21T14:06:30.882 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.882 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.882 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:06:30.882 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.771750+0000", 2024-09-21T14:06:30.882 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4550000000000001e-06 2024-09-21T14:06:30.882 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.882 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.882 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:06:30.882 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.771756+0000", 2024-09-21T14:06:30.883 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2379999999999997e-06 2024-09-21T14:06:30.883 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.883 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.883 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:30.883 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.771868+0000", 2024-09-21T14:06:30.883 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000112321 2024-09-21T14:06:30.883 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.883 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.883 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:06:30.883 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.771878+0000", 2024-09-21T14:06:30.883 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7240000000000004e-06 2024-09-21T14:06:30.883 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.883 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.883 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:30.883 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.635580+0000", 2024-09-21T14:06:30.884 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.86370200600000002 2024-09-21T14:06:30.884 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.884 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.884 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:06:30.884 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.635677+0000", 2024-09-21T14:06:30.884 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7850999999999995e-05 2024-09-21T14:06:30.884 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.884 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.884 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,7", 2024-09-21T14:06:30.884 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.635805+0000", 2024-09-21T14:06:30.884 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000127856 2024-09-21T14:06:30.884 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.884 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.884 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:30.884 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.641202+0000", 2024-09-21T14:06:30.885 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0053963860000000004 2024-09-21T14:06:30.885 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.885 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.885 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:06:30.885 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.644478+0000", 2024-09-21T14:06:30.885 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0032760039999999999 2024-09-21T14:06:30.885 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.885 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.885 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:30.885 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.652260+0000", 2024-09-21T14:06:30.885 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0077823520000000002 2024-09-21T14:06:30.885 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.885 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.885 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:06:30.886 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.652367+0000", 2024-09-21T14:06:30.886 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000107144 2024-09-21T14:06:30.886 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.886 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.886 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:06:30.886 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.652405+0000", 2024-09-21T14:06:30.886 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7311000000000002e-05 2024-09-21T14:06:30.886 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:30.886 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:30.886 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:30.886 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.886 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.886 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:148397 10.8 10:15efa11b:::benchmark_data_smithi049_176811_object20820:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-09-21T14:06:30.886 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:04:55.795257+0000", 2024-09-21T14:06:30.887 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 94.968714392999999, 2024-09-21T14:06:30.887 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.85728244799999997, 2024-09-21T14:06:30.887 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:06:30.887 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:06:30.887 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:30.887 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:06:30.887 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:06:30.887 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:06:30.887 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 148397 2024-09-21T14:06:30.887 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.887 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:06:30.887 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.887 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:06:30.887 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.795257+0000", 2024-09-21T14:06:30.887 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:30.888 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.888 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.888 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:06:30.888 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.795257+0000", 2024-09-21T14:06:30.888 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:30.888 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.888 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.888 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:06:30.888 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.795259+0000", 2024-09-21T14:06:30.888 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6780000000000001e-06 2024-09-21T14:06:30.888 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.888 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.888 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:06:30.888 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.795265+0000", 2024-09-21T14:06:30.889 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1850000000000001e-06 2024-09-21T14:06:30.889 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.889 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.889 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:06:30.889 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.795279+0000", 2024-09-21T14:06:30.889 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3546999999999999e-05 2024-09-21T14:06:30.889 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.889 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.889 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:06:30.889 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.795287+0000", 2024-09-21T14:06:30.889 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4740000000000005e-06 2024-09-21T14:06:30.889 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.890 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.890 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:30.890 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.795357+0000", 2024-09-21T14:06:30.890 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9783000000000006e-05 2024-09-21T14:06:30.890 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.890 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.890 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:06:30.890 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.795367+0000", 2024-09-21T14:06:30.890 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0434999999999999e-05 2024-09-21T14:06:30.890 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.890 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.890 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:30.891 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.638583+0000", 2024-09-21T14:06:30.891 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.84321592499999998 2024-09-21T14:06:30.891 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.891 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.891 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:06:30.891 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.638685+0000", 2024-09-21T14:06:30.891 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000101498 2024-09-21T14:06:30.891 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.891 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.891 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,7", 2024-09-21T14:06:30.891 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.638799+0000", 2024-09-21T14:06:30.891 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000113959 2024-09-21T14:06:30.891 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.891 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.892 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:06:30.892 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.644483+0000", 2024-09-21T14:06:30.892 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0056838629999999999 2024-09-21T14:06:30.892 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.892 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.892 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:30.892 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.645051+0000", 2024-09-21T14:06:30.892 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00056823599999999996 2024-09-21T14:06:30.892 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.892 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.892 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:30.892 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.652450+0000", 2024-09-21T14:06:30.892 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0073992889999999999 2024-09-21T14:06:30.892 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.892 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.893 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:06:30.893 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.652504+0000", 2024-09-21T14:06:30.893 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4051999999999997e-05 2024-09-21T14:06:30.893 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.893 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.893 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:06:30.893 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.652540+0000", 2024-09-21T14:06:30.893 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5524000000000001e-05 2024-09-21T14:06:30.893 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:30.893 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:30.893 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:30.893 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.893 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.893 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:148428 10.8 10:161326c5:::benchmark_data_smithi049_176811_object20851:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-09-21T14:06:30.893 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:04:56.144786+0000", 2024-09-21T14:06:30.894 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 94.619185310999995, 2024-09-21T14:06:30.894 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.50787033299999995, 2024-09-21T14:06:30.894 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:06:30.894 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:06:30.894 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:30.894 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:06:30.894 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:06:30.894 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:06:30.894 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 148428 2024-09-21T14:06:30.894 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.894 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:06:30.894 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.894 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:06:30.895 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.144786+0000", 2024-09-21T14:06:30.895 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:30.895 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.895 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.895 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:06:30.895 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.144786+0000", 2024-09-21T14:06:30.895 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:06:30.895 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.895 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.895 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:06:30.895 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.144788+0000", 2024-09-21T14:06:30.895 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.297e-06 2024-09-21T14:06:30.895 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.895 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.895 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:06:30.896 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.144793+0000", 2024-09-21T14:06:30.896 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.989e-06 2024-09-21T14:06:30.896 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.896 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.896 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:06:30.896 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.144805+0000", 2024-09-21T14:06:30.896 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2544000000000001e-05 2024-09-21T14:06:30.896 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.896 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.896 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:06:30.896 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.144811+0000", 2024-09-21T14:06:30.896 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6409999999999999e-06 2024-09-21T14:06:30.896 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.896 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.896 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:30.897 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.144880+0000", 2024-09-21T14:06:30.897 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9581999999999999e-05 2024-09-21T14:06:30.897 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.897 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.897 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:06:30.897 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.144894+0000", 2024-09-21T14:06:30.897 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.349e-05 2024-09-21T14:06:30.897 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.897 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.897 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:06:30.897 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.641231+0000", 2024-09-21T14:06:30.897 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.49633749599999999 2024-09-21T14:06:30.897 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.897 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.898 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:06:30.898 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.641337+0000", 2024-09-21T14:06:30.898 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000105263 2024-09-21T14:06:30.898 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.898 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.898 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,7", 2024-09-21T14:06:30.898 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.641438+0000", 2024-09-21T14:06:30.898 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010134899999999999 2024-09-21T14:06:30.898 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.898 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.898 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:30.898 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.645183+0000", 2024-09-21T14:06:30.898 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0037453109999999999 2024-09-21T14:06:30.898 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.898 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.899 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:06:30.899 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.651845+0000", 2024-09-21T14:06:30.899 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0066612399999999997 2024-09-21T14:06:30.899 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.899 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.899 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:30.899 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.652580+0000", 2024-09-21T14:06:30.899 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00073548299999999999 2024-09-21T14:06:30.899 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.899 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.899 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:06:30.899 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.652623+0000", 2024-09-21T14:06:30.899 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3241999999999997e-05 2024-09-21T14:06:30.899 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:30.900 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:30.900 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:06:30.900 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.652657+0000", 2024-09-21T14:06:30.900 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3405999999999998e-05 2024-09-21T14:06:30.900 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:30.900 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:30.900 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:30.900 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:30.900 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:30.900 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T14:06:30.981 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:06:31.015 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:31.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:31.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:06:31.117 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:06:31.152 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:31.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:31.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:06:31.254 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:31.294 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:31.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:31.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:31.396 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:06:31.426 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:31.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:31.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:06:31.527 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:06:31.558 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:31.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:31.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:06:31.600 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:31 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:06:31.601 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:31 smithi049 ceph-mon[30080]: pgmap v893: 168 pgs: 168 active+clean; 262 MiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 655 B/s rd, 0 B/s wr, 1.49k op/s 2024-09-21T14:06:31.601 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:31 smithi049 ceph-mon[30080]: 6.e scrub starts 2024-09-21T14:06:31.601 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:31 smithi049 ceph-mon[30080]: 6.e scrub ok 2024-09-21T14:06:31.601 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:31 smithi049 ceph-mon[30080]: 6.c scrub starts 2024-09-21T14:06:31.601 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:31 smithi049 ceph-mon[30080]: 6.c scrub ok 2024-09-21T14:06:31.601 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:31 smithi049 ceph-mon[30080]: 5.1b scrub starts 2024-09-21T14:06:31.601 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:31 smithi049 ceph-mon[30080]: 5.1b scrub ok 2024-09-21T14:06:31.603 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:31 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:06:31.603 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:31 smithi049 ceph-mon[40489]: pgmap v893: 168 pgs: 168 active+clean; 262 MiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 655 B/s rd, 0 B/s wr, 1.49k op/s 2024-09-21T14:06:31.604 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:31 smithi049 ceph-mon[40489]: 6.e scrub starts 2024-09-21T14:06:31.604 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:31 smithi049 ceph-mon[40489]: 6.e scrub ok 2024-09-21T14:06:31.604 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:31 smithi049 ceph-mon[40489]: 6.c scrub starts 2024-09-21T14:06:31.604 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:31 smithi049 ceph-mon[40489]: 6.c scrub ok 2024-09-21T14:06:31.604 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:31 smithi049 ceph-mon[40489]: 5.1b scrub starts 2024-09-21T14:06:31.604 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:31 smithi049 ceph-mon[40489]: 5.1b scrub ok 2024-09-21T14:06:31.660 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:06:31.689 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:31.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:31.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:06:31.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:31 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:06:31.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:31 smithi187 ceph-mon[33032]: pgmap v893: 168 pgs: 168 active+clean; 262 MiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 655 B/s rd, 0 B/s wr, 1.49k op/s 2024-09-21T14:06:31.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:31 smithi187 ceph-mon[33032]: 6.e scrub starts 2024-09-21T14:06:31.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:31 smithi187 ceph-mon[33032]: 6.e scrub ok 2024-09-21T14:06:31.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:31 smithi187 ceph-mon[33032]: 6.c scrub starts 2024-09-21T14:06:31.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:31 smithi187 ceph-mon[33032]: 6.c scrub ok 2024-09-21T14:06:31.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:31 smithi187 ceph-mon[33032]: 5.1b scrub starts 2024-09-21T14:06:31.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:31 smithi187 ceph-mon[33032]: 5.1b scrub ok 2024-09-21T14:06:31.791 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:31.815 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Removed 127574 objects 2024-09-21T14:06:31.815 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Clean up completed and total clean up time :115.924 2024-09-21T14:06:31.824 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:31.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:31.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:31.926 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:31.948 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:31.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:31.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:32.050 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:06:32.071 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:32.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:32.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:06:32.173 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:06:32.195 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:32.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:32.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:06:32.297 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:32.322 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-09-21T14:06:32.323 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:32.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:32.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:32.425 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:32.448 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:32.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:32.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:32.549 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:06:32.572 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:32.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:32.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:06:32.576 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:32 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/788225582' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:06:32.577 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:32 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/788225582' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:06:32.673 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:32.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:32 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/788225582' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:06:32.698 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:32.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:32.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:32.799 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:32.822 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:32.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:32.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:32.923 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:32.945 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:32.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:32.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:33.046 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:06:33.069 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:33.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:33.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:06:33.170 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:33.188 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-09-21T14:06:33.189 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-09-21T14:06:33.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:33.193 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:33.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:33.294 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:33.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:33.319 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:33.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:33.401 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:06:33.401 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:06:33.401 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:06:33.401 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:06:33.401 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:06:33.401 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:06:33.401 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:06:33.402 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:06:33.402 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:06:33.403 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:06:33.420 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:06:33.421 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:06:33.421 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:06:33.440 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:06:33.440 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:06:33.444 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:33.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:33.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:06:33.458 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:06:33.458 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:06:33.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:33 smithi049 ceph-mon[30080]: pgmap v895: 168 pgs: 168 active+clean; 192 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.10k op/s 2024-09-21T14:06:33.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:33 smithi049 ceph-mon[30080]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:06:33.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:33 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/788225582' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T14:06:33.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:33 smithi049 ceph-mon[30080]: osdmap e455: 8 total, 8 up, 8 in 2024-09-21T14:06:33.468 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:33 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2813175168' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:06:33.468 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:33 smithi049 ceph-mon[40489]: pgmap v895: 168 pgs: 168 active+clean; 192 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.10k op/s 2024-09-21T14:06:33.468 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:33 smithi049 ceph-mon[40489]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:06:33.468 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:33 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/788225582' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T14:06:33.468 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:33 smithi049 ceph-mon[40489]: osdmap e455: 8 total, 8 up, 8 in 2024-09-21T14:06:33.468 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:33 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2813175168' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:06:33.545 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:33.567 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:33.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:33.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:33.669 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:06:33.693 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:33.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:33.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:06:33.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:33 smithi187 ceph-mon[33032]: pgmap v895: 168 pgs: 168 active+clean; 192 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.10k op/s 2024-09-21T14:06:33.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:33 smithi187 ceph-mon[33032]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:06:33.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:33 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/788225582' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T14:06:33.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:33 smithi187 ceph-mon[33032]: osdmap e455: 8 total, 8 up, 8 in 2024-09-21T14:06:33.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:33 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2813175168' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:06:33.795 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:06:33.818 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:33.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:33.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:06:33.920 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:33.941 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:33.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:33.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:34.043 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:06:34.069 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:34.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:34.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:06:34.171 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:34.193 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:34.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:34.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:34.295 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:34.318 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:34.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:34.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:34.419 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:06:34.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:34.442 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:34.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:06:34.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:34 smithi049 ceph-mon[40489]: 5.17 scrub starts 2024-09-21T14:06:34.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:34 smithi049 ceph-mon[40489]: 5.17 scrub ok 2024-09-21T14:06:34.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:34 smithi049 ceph-mon[30080]: 5.17 scrub starts 2024-09-21T14:06:34.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:34 smithi049 ceph-mon[30080]: 5.17 scrub ok 2024-09-21T14:06:34.543 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:06:34.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:34.569 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:34.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:06:34.670 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:34.693 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:06:34.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:34 smithi187 ceph-mon[33032]: 5.17 scrub starts 2024-09-21T14:06:34.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:34 smithi187 ceph-mon[33032]: 5.17 scrub ok 2024-09-21T14:06:34.699 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:34.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:34.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:34.801 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:06:34.823 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:34.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:34.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:06:34.925 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:34.948 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:34.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:34.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:35.049 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:06:35.075 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:35.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:35.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:06:35.177 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:35.179 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:06:35.213 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:35.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:35.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:35.218 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T14:06:35.315 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:35.337 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:35.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:35.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:35.435 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:35 smithi049 ceph-mon[40489]: pgmap v897: 168 pgs: 168 active+clean; 30 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.50k op/s 2024-09-21T14:06:35.435 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:35 smithi049 ceph-mon[40489]: osdmap e456: 8 total, 8 up, 8 in 2024-09-21T14:06:35.436 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:35 smithi049 ceph-mon[30080]: pgmap v897: 168 pgs: 168 active+clean; 30 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.50k op/s 2024-09-21T14:06:35.436 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:35 smithi049 ceph-mon[30080]: osdmap e456: 8 total, 8 up, 8 in 2024-09-21T14:06:35.439 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:06:35.465 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:35.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:35.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:06:35.566 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:06:35.592 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:35.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:35.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:06:35.694 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:06:35.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:35 smithi187 ceph-mon[33032]: pgmap v897: 168 pgs: 168 active+clean; 30 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.50k op/s 2024-09-21T14:06:35.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:35 smithi187 ceph-mon[33032]: osdmap e456: 8 total, 8 up, 8 in 2024-09-21T14:06:35.719 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:35.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:35.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:06:35.821 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:06:35.845 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:35.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:35.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:06:35.946 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:35.969 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:35.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:35.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:36.071 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:06:36.093 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:36.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:36.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:06:36.195 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:36.219 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:36.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:36.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:36.224 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T14:06:36.321 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:06:36.351 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:36.351 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'unique_pool_3' removed 2024-09-21T14:06:36.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:36.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:06:36.453 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:36.481 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:36.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:36.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:36.583 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:36.605 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:36.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:36.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:36.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:36 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1482150440' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T14:06:36.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:36 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T14:06:36.707 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:06:36.716 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:36 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1482150440' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T14:06:36.716 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:36 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T14:06:36.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:36 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1482150440' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T14:06:36.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:36 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T14:06:36.724 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T14:06:36.734 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:36.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:36.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:06:36.836 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:06:36.860 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:36.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:36.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:06:36.961 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:36.983 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:36.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:36.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:37.083 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-21T14:06:37.083 INFO:tasks.radosbench:Beginning radosbench... 2024-09-21T14:06:37.083 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-09-21T14:06:37.084 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd pool create unique_pool_4 16 2024-09-21T14:06:37.085 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:06:37.106 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:37.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:37.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:06:37.208 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:06:37.229 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:37.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:37.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:06:37.331 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:06:37.354 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:37.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:37.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:06:37.456 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:06:37.478 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:37.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:37.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:06:37.580 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:06:37.601 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:37.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:37.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:06:37.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:37 smithi187 ceph-mon[33032]: pgmap v899: 168 pgs: 168 active+clean; 2.8 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 692 op/s 2024-09-21T14:06:37.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:37 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T14:06:37.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:37 smithi187 ceph-mon[33032]: osdmap e457: 8 total, 8 up, 8 in 2024-09-21T14:06:37.703 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:06:37.726 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:37.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:37.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:06:37.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:37 smithi049 ceph-mon[40489]: pgmap v899: 168 pgs: 168 active+clean; 2.8 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 692 op/s 2024-09-21T14:06:37.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:37 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T14:06:37.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:37 smithi049 ceph-mon[40489]: osdmap e457: 8 total, 8 up, 8 in 2024-09-21T14:06:37.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:37 smithi049 ceph-mon[30080]: pgmap v899: 168 pgs: 168 active+clean; 2.8 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 692 op/s 2024-09-21T14:06:37.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:37 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T14:06:37.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:37 smithi049 ceph-mon[30080]: osdmap e457: 8 total, 8 up, 8 in 2024-09-21T14:06:37.827 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:37.850 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:37.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:37.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:37.951 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:06:37.973 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:37.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:37.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:06:38.075 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:38.098 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:38.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:38.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:38.199 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:06:38.221 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:38.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:38.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:06:38.323 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:06:38.345 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:38.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:38.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:06:38.447 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:06:38.469 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:38.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:38.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:06:38.570 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:38.597 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:38.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:38.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:38.698 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:06:38.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:38.721 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:38.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:06:38.822 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:38.845 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:38.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:38.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:38.946 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:06:38.970 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:38.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:38.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:06:39.072 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:39.076 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph tell osd.7 reset_purged_snaps_last 2024-09-21T14:06:39.095 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:39.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:39.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:39.109 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:06:39.112 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:06:39.124 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:06:39.196 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:39.219 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:39.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:39.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:39.320 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:39.345 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:39.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:39.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:39.447 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:06:39.473 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:39.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:39.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:06:39.574 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:39.598 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:39.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:39.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:39.630 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:39 smithi049 ceph-mon[30080]: pgmap v901: 152 pgs: 152 active+clean; 583 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-21T14:06:39.630 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:39 smithi049 ceph-mon[40489]: pgmap v901: 152 pgs: 152 active+clean; 583 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-21T14:06:39.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:39 smithi187 ceph-mon[33032]: pgmap v901: 152 pgs: 152 active+clean; 583 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-21T14:06:39.699 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:39.724 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:39.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:39.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:39.770 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:06:39.826 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:06:39.851 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:39.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:39.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:06:39.953 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:39.974 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:39.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:39.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:40.076 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:06:40.098 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:40.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:40.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:06:40.199 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:40.221 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:40.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:40.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:40.323 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:06:40.348 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:40.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:40.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:06:40.353 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-09-21T14:06:40.356 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:06:40 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:06:40.450 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:40.475 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:40.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:40.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:40.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:40 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:06:40.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:40 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:06:40.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:40 smithi049 ceph-mon[40489]: osdmap e458: 8 total, 8 up, 8 in 2024-09-21T14:06:40.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:40 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2846408384' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:06:40.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:40 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:06:40.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:40 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:06:40.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:40 smithi049 ceph-mon[30080]: osdmap e458: 8 total, 8 up, 8 in 2024-09-21T14:06:40.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:40 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2846408384' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:06:40.552 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T14:06:40.552 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:06:40.577 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:06:40.599 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:40.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:40.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:06:40.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:40 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:06:40.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:40 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:06:40.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:40 smithi187 ceph-mon[33032]: osdmap e458: 8 total, 8 up, 8 in 2024-09-21T14:06:40.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:40 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2846408384' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:06:40.701 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:40.723 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:40.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:40.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:40.798 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:06:40 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:06:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:06:40.825 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:40.847 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:40.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:40.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:40.949 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:06:40.974 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:40.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:40.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:06:41.076 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:41.098 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:41.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:41.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:41.200 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:06:41.224 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:41.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:41.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:06:41.326 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:06:41.349 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:41.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:41.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:06:41.451 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:06:41.475 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:41.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:41.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:06:41.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:41 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:06:41.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:41 smithi049 ceph-mon[40489]: pgmap v903: 152 pgs: 152 active+clean; 583 KiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-21T14:06:41.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:41 smithi049 ceph-mon[40489]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:06:41.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:41 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2846408384' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T14:06:41.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:41 smithi049 ceph-mon[40489]: osdmap e459: 8 total, 8 up, 8 in 2024-09-21T14:06:41.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:41 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:06:41.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:41 smithi049 ceph-mon[30080]: pgmap v903: 152 pgs: 152 active+clean; 583 KiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-21T14:06:41.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:41 smithi049 ceph-mon[30080]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:06:41.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:41 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2846408384' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T14:06:41.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:41 smithi049 ceph-mon[30080]: osdmap e459: 8 total, 8 up, 8 in 2024-09-21T14:06:41.577 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:06:41.599 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:41.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:41.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:06:41.607 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:41 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:06:41.607 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:41 smithi187 ceph-mon[33032]: pgmap v903: 152 pgs: 152 active+clean; 583 KiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-21T14:06:41.607 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:41 smithi187 ceph-mon[33032]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:06:41.607 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:41 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2846408384' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T14:06:41.607 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:41 smithi187 ceph-mon[33032]: osdmap e459: 8 total, 8 up, 8 in 2024-09-21T14:06:41.702 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:41.725 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:41.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:41.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:41.827 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:06:41.849 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:41.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:41.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:06:41.951 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:06:41.966 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T14:06:41.971 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:41.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:41.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:06:42.073 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:42.096 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:42.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:42.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:42.197 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:06:42.219 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:42.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:42.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:06:42.321 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:42.344 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:42.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:42.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:42.441 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:06:42.441 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:06:42.446 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:42.448 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:06:42.456 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:06:42.456 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:06:42.468 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:42.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:42.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:42.474 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:06:42.474 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:06:42.493 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:06:42.493 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:06:42.494 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:42 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:06:42.494 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:42 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:06:42.494 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:42 smithi049 ceph-mon[40489]: osdmap e460: 8 total, 8 up, 8 in 2024-09-21T14:06:42.494 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:42 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:06:42.494 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:42 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:06:42.494 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:42 smithi049 ceph-mon[30080]: osdmap e460: 8 total, 8 up, 8 in 2024-09-21T14:06:42.510 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:06:42.510 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:06:42.527 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:06:42.527 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:06:42.543 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:06:42.544 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:06:42.561 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:06:42.561 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:06:42.571 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:06:42.593 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:42.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:42.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:06:42.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:42 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:06:42.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:42 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:06:42.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:42 smithi187 ceph-mon[33032]: osdmap e460: 8 total, 8 up, 8 in 2024-09-21T14:06:42.695 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:06:42.716 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:42.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:42.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:06:42.817 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:42.840 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:42.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:42.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:42.941 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:42.963 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:42.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:42.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:43.065 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:06:43.087 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:43.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:43.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:06:43.189 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:43.210 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:43.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:43.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:43.312 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:06:43.334 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:43.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:43.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:06:43.435 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:06:43.457 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:43.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:43.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:06:43.556 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T14:06:43.559 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:06:43.580 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:43.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:43.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:06:43.682 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:06:43.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:43 smithi187 ceph-mon[33032]: pgmap v906: 152 pgs: 152 active+clean; 583 KiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 889 B/s rd, 0 op/s 2024-09-21T14:06:43.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:43 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3411684770' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:06:43.703 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:43.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:43.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:06:43.726 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:43 smithi049 ceph-mon[40489]: pgmap v906: 152 pgs: 152 active+clean; 583 KiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 889 B/s rd, 0 op/s 2024-09-21T14:06:43.726 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:43 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3411684770' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:06:43.726 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:43 smithi049 ceph-mon[30080]: pgmap v906: 152 pgs: 152 active+clean; 583 KiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 889 B/s rd, 0 op/s 2024-09-21T14:06:43.726 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:43 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3411684770' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:06:43.805 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:06:43.826 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:43.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:43.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:06:43.928 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:06:43.949 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:43.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:43.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:06:44.051 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:06:44.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:44.073 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:44.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:06:44.174 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:44.196 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:44.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:44.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:44.297 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:44.320 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:44.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:44.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:44.369 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'unique_pool_4' created 2024-09-21T14:06:44.422 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:44.444 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:44.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:44.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:44.546 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:44.568 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:44.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:44.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:44.669 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:44.691 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:44.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:44.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:44.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:44 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/922201954' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-21T14:06:44.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:44 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/922201954' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-21T14:06:44.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:44 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/922201954' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-21T14:06:44.793 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:06:44.810 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph tell osd.6 reset_purged_snaps_last 2024-09-21T14:06:44.816 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:44.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:44.817 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:06:44.917 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:06:44.939 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:44.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:44.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:06:45.041 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:06:45.063 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:06:45.064 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:06:45.065 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:45.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:45.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:06:45.167 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:06:45.189 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:45.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:45.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:06:45.290 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:45.315 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:45.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:45.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:45.417 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:06:45.439 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:45.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:45.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:06:45.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:45 smithi049 ceph-mon[40489]: pgmap v907: 152 pgs: 152 active+clean; 583 KiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T14:06:45.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:45 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/922201954' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-21T14:06:45.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:45 smithi049 ceph-mon[40489]: osdmap e461: 8 total, 8 up, 8 in 2024-09-21T14:06:45.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:45 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:06:45.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:45 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:06:45.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:45 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T14:06:45.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:45 smithi049 ceph-mon[40489]: osdmap e462: 8 total, 8 up, 8 in 2024-09-21T14:06:45.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:45 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T14:06:45.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:45 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T14:06:45.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:45 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:06:45.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:45 smithi049 ceph-mon[30080]: pgmap v907: 152 pgs: 152 active+clean; 583 KiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T14:06:45.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:45 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/922201954' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-21T14:06:45.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:45 smithi049 ceph-mon[30080]: osdmap e461: 8 total, 8 up, 8 in 2024-09-21T14:06:45.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:45 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:06:45.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:45 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:06:45.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:45 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T14:06:45.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:45 smithi049 ceph-mon[30080]: osdmap e462: 8 total, 8 up, 8 in 2024-09-21T14:06:45.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:45 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T14:06:45.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:45 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T14:06:45.507 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:45 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:06:45.542 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:45.564 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:45.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:45.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:45.666 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:45.671 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:06:45.690 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:45.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:45.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:45.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:45 smithi187 ceph-mon[33032]: pgmap v907: 152 pgs: 152 active+clean; 583 KiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T14:06:45.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:45 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/922201954' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-21T14:06:45.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:45 smithi187 ceph-mon[33032]: osdmap e461: 8 total, 8 up, 8 in 2024-09-21T14:06:45.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:45 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:06:45.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:45 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:06:45.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:45 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T14:06:45.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:45 smithi187 ceph-mon[33032]: osdmap e462: 8 total, 8 up, 8 in 2024-09-21T14:06:45.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:45 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T14:06:45.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:45 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T14:06:45.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:45 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:06:45.760 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T14:06:45.791 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:45.827 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:45.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:45.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:45.928 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:45.954 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:45.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:45.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:46.056 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:06:46.079 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:46.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:46.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:06:46.181 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:06:46.193 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-09-21T14:06:46.205 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:46.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:46.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:06:46.306 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:46.329 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:46.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:46.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:46.430 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:06:46.456 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:46.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:46.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:06:46.557 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:06:46.582 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:46.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:46.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:06:46.616 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-09-21T14:06:46.638 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:46 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1093058737' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:06:46.638 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:46 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:06:46.639 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:46 smithi049 ceph-mon[40489]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:06:46.639 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:46 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T14:06:46.639 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:46 smithi049 ceph-mon[40489]: osdmap e463: 8 total, 8 up, 8 in 2024-09-21T14:06:46.639 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:46 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1093058737' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:06:46.639 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:46 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:06:46.639 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:46 smithi049 ceph-mon[30080]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:06:46.639 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:46 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T14:06:46.639 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:46 smithi049 ceph-mon[30080]: osdmap e463: 8 total, 8 up, 8 in 2024-09-21T14:06:46.684 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:46.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:46 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1093058737' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:06:46.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:46 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:06:46.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:46 smithi187 ceph-mon[33032]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:06:46.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:46 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T14:06:46.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:46 smithi187 ceph-mon[33032]: osdmap e463: 8 total, 8 up, 8 in 2024-09-21T14:06:46.708 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:46.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:46.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:46.810 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:46.833 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:46.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:46.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:46.934 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:06:46.961 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:46.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:46.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:06:47.062 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:06:47.088 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:47.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:47.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:06:47.173 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T14:06:47.173 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-09-21T14:06:47.173 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-09-21T14:06:47.173 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-09-21T14:06:47.173 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T14:06:47.190 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:06:47.216 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:47.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:47.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:06:47.317 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:47.341 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:47.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:47.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:47.436 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:47 smithi049 ceph-mon[40489]: pgmap v910: 168 pgs: 3 creating+peering, 12 unknown, 153 active+clean; 583 KiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 891 B/s rd, 0 op/s 2024-09-21T14:06:47.436 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:47 smithi049 ceph-mon[40489]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T14:06:47.436 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:47 smithi049 ceph-mon[40489]: 4.8 scrub starts 2024-09-21T14:06:47.437 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:47 smithi049 ceph-mon[40489]: 4.8 scrub ok 2024-09-21T14:06:47.437 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:47 smithi049 ceph-mon[40489]: 3.19 scrub starts 2024-09-21T14:06:47.437 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:47 smithi049 ceph-mon[40489]: 3.19 scrub ok 2024-09-21T14:06:47.437 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:47 smithi049 ceph-mon[30080]: pgmap v910: 168 pgs: 3 creating+peering, 12 unknown, 153 active+clean; 583 KiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 891 B/s rd, 0 op/s 2024-09-21T14:06:47.437 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:47 smithi049 ceph-mon[30080]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T14:06:47.437 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:47 smithi049 ceph-mon[30080]: 4.8 scrub starts 2024-09-21T14:06:47.437 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:47 smithi049 ceph-mon[30080]: 4.8 scrub ok 2024-09-21T14:06:47.437 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:47 smithi049 ceph-mon[30080]: 3.19 scrub starts 2024-09-21T14:06:47.437 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:47 smithi049 ceph-mon[30080]: 3.19 scrub ok 2024-09-21T14:06:47.443 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:47.464 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:47.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:47.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:47.566 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:06:47.592 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:47.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:47.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:06:47.694 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:06:47.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:47 smithi187 ceph-mon[33032]: pgmap v910: 168 pgs: 3 creating+peering, 12 unknown, 153 active+clean; 583 KiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 891 B/s rd, 0 op/s 2024-09-21T14:06:47.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:47 smithi187 ceph-mon[33032]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T14:06:47.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:47 smithi187 ceph-mon[33032]: 4.8 scrub starts 2024-09-21T14:06:47.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:47 smithi187 ceph-mon[33032]: 4.8 scrub ok 2024-09-21T14:06:47.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:47 smithi187 ceph-mon[33032]: 3.19 scrub starts 2024-09-21T14:06:47.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:47 smithi187 ceph-mon[33032]: 3.19 scrub ok 2024-09-21T14:06:47.720 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:47.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:47.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:06:47.821 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:06:47.847 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:47.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:47.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:06:47.949 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:47.974 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:47.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:47.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:48.076 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:06:48.099 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:48.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:48.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:06:48.201 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:48.225 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:48.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:48.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:48.327 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:06:48.352 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:48.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:48.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:06:48.454 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:48.478 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:48.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:48.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:48.580 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:48.604 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:48.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:48.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:48.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:48 smithi187 ceph-mon[33032]: 5.d scrub starts 2024-09-21T14:06:48.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:48 smithi187 ceph-mon[33032]: 5.d scrub ok 2024-09-21T14:06:48.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:48 smithi187 ceph-mon[33032]: 6.17 scrub starts 2024-09-21T14:06:48.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:48 smithi187 ceph-mon[33032]: 6.17 scrub ok 2024-09-21T14:06:48.696 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:48 smithi049 ceph-mon[30080]: 5.d scrub starts 2024-09-21T14:06:48.696 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:48 smithi049 ceph-mon[30080]: 5.d scrub ok 2024-09-21T14:06:48.696 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:48 smithi049 ceph-mon[30080]: 6.17 scrub starts 2024-09-21T14:06:48.696 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:48 smithi049 ceph-mon[30080]: 6.17 scrub ok 2024-09-21T14:06:48.696 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:48 smithi049 ceph-mon[40489]: 5.d scrub starts 2024-09-21T14:06:48.696 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:48 smithi049 ceph-mon[40489]: 5.d scrub ok 2024-09-21T14:06:48.696 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:48 smithi049 ceph-mon[40489]: 6.17 scrub starts 2024-09-21T14:06:48.696 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:48 smithi049 ceph-mon[40489]: 6.17 scrub ok 2024-09-21T14:06:48.706 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:48.730 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:48.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:48.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:48.833 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:06:48.859 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:48.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:48.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:06:48.961 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:48.987 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:48.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:48.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:49.088 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:49.114 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:49.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:49.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:49.206 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T14:06:49.216 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:06:49.242 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:49.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:49.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:06:49.344 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:06:49.369 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:49.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:49.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:06:49.425 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:49 smithi049 ceph-mon[30080]: 3.18 scrub starts 2024-09-21T14:06:49.425 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:49 smithi049 ceph-mon[30080]: 3.18 scrub ok 2024-09-21T14:06:49.425 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:49 smithi049 ceph-mon[30080]: pgmap v912: 168 pgs: 3 creating+peering, 4 unknown, 161 active+clean; 583 KiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T14:06:49.425 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:49 smithi049 ceph-mon[30080]: 1.5 scrub starts 2024-09-21T14:06:49.426 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:49 smithi049 ceph-mon[30080]: 1.5 scrub ok 2024-09-21T14:06:49.426 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:49 smithi049 ceph-mon[40489]: 3.18 scrub starts 2024-09-21T14:06:49.426 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:49 smithi049 ceph-mon[40489]: 3.18 scrub ok 2024-09-21T14:06:49.426 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:49 smithi049 ceph-mon[40489]: pgmap v912: 168 pgs: 3 creating+peering, 4 unknown, 161 active+clean; 583 KiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T14:06:49.426 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:49 smithi049 ceph-mon[40489]: 1.5 scrub starts 2024-09-21T14:06:49.426 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:49 smithi049 ceph-mon[40489]: 1.5 scrub ok 2024-09-21T14:06:49.427 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:06:49.427 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:06:49.471 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:06:49.495 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:49.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:49.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:06:49.597 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:49.619 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:49.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:49.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:49.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:49 smithi187 ceph-mon[33032]: 3.18 scrub starts 2024-09-21T14:06:49.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:49 smithi187 ceph-mon[33032]: 3.18 scrub ok 2024-09-21T14:06:49.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:49 smithi187 ceph-mon[33032]: pgmap v912: 168 pgs: 3 creating+peering, 4 unknown, 161 active+clean; 583 KiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T14:06:49.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:49 smithi187 ceph-mon[33032]: 1.5 scrub starts 2024-09-21T14:06:49.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:49 smithi187 ceph-mon[33032]: 1.5 scrub ok 2024-09-21T14:06:49.721 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:49.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:49.745 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:49.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:49.845 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:06:49.872 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:49.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:49.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:06:49.973 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:49.995 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:49.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:49.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:50.097 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:50.119 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:50.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:50.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:50.221 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:50.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:50.243 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:50.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:50.344 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:06:50.371 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:50.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:50.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:06:50.387 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:06:50 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:06:50.473 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:50.496 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:50.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:50.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:50.505 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:06:50.513 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:06:50.513 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:06:50.532 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:06:50.532 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:06:50.549 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:06:50.549 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:06:50.565 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:06:50.565 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:06:50.583 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:06:50.584 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:06:50.599 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:50.601 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:06:50.602 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:06:50.620 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:06:50.620 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:06:50.620 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:06:50 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:06:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:06:50.621 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:50 smithi049 ceph-mon[30080]: 4.1d scrub starts 2024-09-21T14:06:50.621 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:50 smithi049 ceph-mon[30080]: 4.1d scrub ok 2024-09-21T14:06:50.621 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:50 smithi049 ceph-mon[30080]: 5.10 scrub starts 2024-09-21T14:06:50.621 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:50 smithi049 ceph-mon[30080]: 5.10 scrub ok 2024-09-21T14:06:50.621 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:50 smithi049 ceph-mon[30080]: 5.7 scrub starts 2024-09-21T14:06:50.621 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:50 smithi049 ceph-mon[30080]: 5.7 scrub ok 2024-09-21T14:06:50.621 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:50 smithi049 ceph-mon[30080]: osdmap e464: 8 total, 8 up, 8 in 2024-09-21T14:06:50.621 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:50 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1483006174' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T14:06:50.622 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:50 smithi049 ceph-mon[40489]: 4.1d scrub starts 2024-09-21T14:06:50.622 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:50 smithi049 ceph-mon[40489]: 4.1d scrub ok 2024-09-21T14:06:50.622 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:50 smithi049 ceph-mon[40489]: 5.10 scrub starts 2024-09-21T14:06:50.622 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:50 smithi049 ceph-mon[40489]: 5.10 scrub ok 2024-09-21T14:06:50.622 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:50 smithi049 ceph-mon[40489]: 5.7 scrub starts 2024-09-21T14:06:50.622 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:50 smithi049 ceph-mon[40489]: 5.7 scrub ok 2024-09-21T14:06:50.622 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:50 smithi049 ceph-mon[40489]: osdmap e464: 8 total, 8 up, 8 in 2024-09-21T14:06:50.622 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:50 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1483006174' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T14:06:50.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:50.623 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:50.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:50.637 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:06:50.637 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:06:50.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:50 smithi187 ceph-mon[33032]: 4.1d scrub starts 2024-09-21T14:06:50.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:50 smithi187 ceph-mon[33032]: 4.1d scrub ok 2024-09-21T14:06:50.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:50 smithi187 ceph-mon[33032]: 5.10 scrub starts 2024-09-21T14:06:50.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:50 smithi187 ceph-mon[33032]: 5.10 scrub ok 2024-09-21T14:06:50.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:50 smithi187 ceph-mon[33032]: 5.7 scrub starts 2024-09-21T14:06:50.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:50 smithi187 ceph-mon[33032]: 5.7 scrub ok 2024-09-21T14:06:50.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:50 smithi187 ceph-mon[33032]: osdmap e464: 8 total, 8 up, 8 in 2024-09-21T14:06:50.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:50 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1483006174' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T14:06:50.724 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:50.747 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:50.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:50.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:50.849 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:50.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:50.872 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:50.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:50.973 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:06:50.999 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:50.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:51.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:06:51.100 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:51.123 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:51.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:51.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:51.198 INFO:teuthology.orchestra.run.smithi049.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-09-21T14:06:51.226 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:06:51.250 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:51.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:51.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:06:51.352 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:06:51.376 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:51.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:51.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:06:51.428 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T14:06:51.477 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:51.499 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:51.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:51.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:51.601 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:06:51.624 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:51.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:51.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:06:51.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:51 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:06:51.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:51 smithi187 ceph-mon[33032]: pgmap v913: 168 pgs: 168 active+clean; 583 KiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T14:06:51.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:51 smithi187 ceph-mon[33032]: 3.2 scrub starts 2024-09-21T14:06:51.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:51 smithi187 ceph-mon[33032]: 3.2 scrub ok 2024-09-21T14:06:51.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:51 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/278274999' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:06:51.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:51 smithi187 ceph-mon[33032]: 6.18 scrub starts 2024-09-21T14:06:51.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:51 smithi187 ceph-mon[33032]: 6.18 scrub ok 2024-09-21T14:06:51.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:51 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1483006174' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T14:06:51.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:51 smithi187 ceph-mon[33032]: osdmap e465: 8 total, 8 up, 8 in 2024-09-21T14:06:51.725 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:51.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:51.748 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:51.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:51.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:51 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:06:51.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:51 smithi049 ceph-mon[30080]: pgmap v913: 168 pgs: 168 active+clean; 583 KiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T14:06:51.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:51 smithi049 ceph-mon[30080]: 3.2 scrub starts 2024-09-21T14:06:51.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:51 smithi049 ceph-mon[30080]: 3.2 scrub ok 2024-09-21T14:06:51.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:51 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/278274999' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:06:51.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:51 smithi049 ceph-mon[30080]: 6.18 scrub starts 2024-09-21T14:06:51.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:51 smithi049 ceph-mon[30080]: 6.18 scrub ok 2024-09-21T14:06:51.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:51 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1483006174' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T14:06:51.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:51 smithi049 ceph-mon[30080]: osdmap e465: 8 total, 8 up, 8 in 2024-09-21T14:06:51.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:51 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:06:51.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:51 smithi049 ceph-mon[40489]: pgmap v913: 168 pgs: 168 active+clean; 583 KiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T14:06:51.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:51 smithi049 ceph-mon[40489]: 3.2 scrub starts 2024-09-21T14:06:51.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:51 smithi049 ceph-mon[40489]: 3.2 scrub ok 2024-09-21T14:06:51.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:51 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/278274999' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:06:51.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:51 smithi049 ceph-mon[40489]: 6.18 scrub starts 2024-09-21T14:06:51.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:51 smithi049 ceph-mon[40489]: 6.18 scrub ok 2024-09-21T14:06:51.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:51 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1483006174' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T14:06:51.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:51 smithi049 ceph-mon[40489]: osdmap e465: 8 total, 8 up, 8 in 2024-09-21T14:06:51.849 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:51.872 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:51.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:51.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:51.973 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:06:51.999 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:52.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:52.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:06:52.101 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:52.124 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:52.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:52.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:52.225 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:06:52.248 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:52.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:52.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:06:52.350 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:52.374 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:52.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:52.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:52.477 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:06:52.488 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T14:06:52.488 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:06:52.501 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:52.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:52.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:06:52.602 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:06:52.625 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:52.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:52.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:06:52.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:52 smithi187 ceph-mon[33032]: 5.12 scrub starts 2024-09-21T14:06:52.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:52 smithi187 ceph-mon[33032]: 5.12 scrub ok 2024-09-21T14:06:52.728 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:06:52.750 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:52.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:52.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:06:52.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:52 smithi049 ceph-mon[40489]: 5.12 scrub starts 2024-09-21T14:06:52.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:52 smithi049 ceph-mon[40489]: 5.12 scrub ok 2024-09-21T14:06:52.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:52 smithi049 ceph-mon[30080]: 5.12 scrub starts 2024-09-21T14:06:52.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:52 smithi049 ceph-mon[30080]: 5.12 scrub ok 2024-09-21T14:06:52.852 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:06:52.875 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:52.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:52.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:06:52.977 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:53.002 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:53.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:53.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:53.104 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:53.127 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:53.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:53.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:53.172 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T14:06:53.186 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:06:53.229 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:53.251 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:53.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:53.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:53.352 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:06:53.374 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:53.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:53.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:06:53.395 DEBUG:teuthology.orchestra.run.smithi049:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-09-21T14:06:53.418 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-21T14:06:53.418 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-21T14:06:53.426 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:06:53.475 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:06:53.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:53.497 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:53.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:06:53.503 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:hints = 1 2024-09-21T14:06:53.503 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-21T14:06:53.503 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Object prefix: benchmark_data_smithi049_206132 2024-09-21T14:06:53.503 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T14:06:53.503 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 0 0 0 0 0 0 - 0 2024-09-21T14:06:53.547 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:53 smithi187 ceph-mon[33032]: pgmap v916: 168 pgs: 168 active+clean; 583 KiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T14:06:53.547 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:53 smithi187 ceph-mon[33032]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T14:06:53.547 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:53 smithi187 ceph-mon[33032]: 1.3 scrub starts 2024-09-21T14:06:53.547 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:53 smithi187 ceph-mon[33032]: 1.3 scrub ok 2024-09-21T14:06:53.563 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T14:06:53.599 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:06:53.630 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:53.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:53.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:06:53.673 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:53 smithi049 ceph-mon[40489]: pgmap v916: 168 pgs: 168 active+clean; 583 KiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T14:06:53.673 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:53 smithi049 ceph-mon[40489]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T14:06:53.673 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:53 smithi049 ceph-mon[40489]: 1.3 scrub starts 2024-09-21T14:06:53.673 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:53 smithi049 ceph-mon[40489]: 1.3 scrub ok 2024-09-21T14:06:53.674 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:53 smithi049 ceph-mon[30080]: pgmap v916: 168 pgs: 168 active+clean; 583 KiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-21T14:06:53.674 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:53 smithi049 ceph-mon[30080]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T14:06:53.675 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:53 smithi049 ceph-mon[30080]: 1.3 scrub starts 2024-09-21T14:06:53.675 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:53 smithi049 ceph-mon[30080]: 1.3 scrub ok 2024-09-21T14:06:53.732 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:06:53.733 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 4, 6, 2, 3, 7, 5] 2024-09-21T14:06:53.733 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T14:06:53.733 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-21T14:06:53.733 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd dump -f json-pretty 2024-09-21T14:06:53.763 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:53.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:53.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:06:53.865 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:06:53.911 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:53.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:53.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:06:54.013 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:54.038 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:54.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:54.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:54.090 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:06:54.140 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:06:54.166 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:54.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:54.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:06:54.249 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T14:06:54.249 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:06:54.269 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:54.296 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:54.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:54.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:54.398 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:54.420 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-09-21T14:06:54.454 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:54.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:54.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:54.503 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 1 15 2641 2626 164.115 164.125 0.00406079 0.00607128 2024-09-21T14:06:54.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:54 smithi049 ceph-mon[40489]: 1.4 scrub starts 2024-09-21T14:06:54.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:54 smithi049 ceph-mon[40489]: 1.4 scrub ok 2024-09-21T14:06:54.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:54 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1030243059' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:06:54.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:54 smithi049 ceph-mon[30080]: 1.4 scrub starts 2024-09-21T14:06:54.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:54 smithi049 ceph-mon[30080]: 1.4 scrub ok 2024-09-21T14:06:54.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:54 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1030243059' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:06:54.557 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:06:54.584 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:54.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:54.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:06:54.686 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:54.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:54 smithi187 ceph-mon[33032]: 1.4 scrub starts 2024-09-21T14:06:54.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:54 smithi187 ceph-mon[33032]: 1.4 scrub ok 2024-09-21T14:06:54.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:54 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1030243059' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:06:54.725 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:54.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:54.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:54.827 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:54.871 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:54.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:54.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:54.973 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:06:54.998 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:06:54.998 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:06:55.002 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:55.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:55.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:06:55.036 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:06:55.036 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:06:55.073 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:06:55.073 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:06:55.104 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:06:55.107 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:06:55.107 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:06:55.135 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:55.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:55.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:06:55.142 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:06:55.143 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:06:55.177 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:06:55.177 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:06:55.212 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:06:55.212 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:06:55.236 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:06:55.251 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:06:55.251 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:06:55.277 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:55.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:55.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:06:55.379 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:55.419 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:55.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:55.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:55.503 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 2 15 5523 5508 172.112 180.125 0.0040851 0.00579798 2024-09-21T14:06:55.521 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:06:55.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:55.553 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:55.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:06:55.654 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:55.692 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:55.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:55.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:55.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:55 smithi187 ceph-mon[33032]: pgmap v917: 168 pgs: 168 active+clean; 9.9 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.2 MiB/s wr, 20 op/s 2024-09-21T14:06:55.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:55 smithi187 ceph-mon[33032]: 3.17 scrub starts 2024-09-21T14:06:55.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:55 smithi187 ceph-mon[33032]: 3.17 scrub ok 2024-09-21T14:06:55.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:55 smithi187 ceph-mon[33032]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T14:06:55.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:55 smithi187 ceph-mon[33032]: Cluster is now healthy 2024-09-21T14:06:55.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:55 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1030243059' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T14:06:55.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:55 smithi187 ceph-mon[33032]: osdmap e466: 8 total, 8 up, 8 in 2024-09-21T14:06:55.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:55 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2157423321' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:06:55.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:55 smithi049 ceph-mon[40489]: pgmap v917: 168 pgs: 168 active+clean; 9.9 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.2 MiB/s wr, 20 op/s 2024-09-21T14:06:55.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:55 smithi049 ceph-mon[40489]: 3.17 scrub starts 2024-09-21T14:06:55.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:55 smithi049 ceph-mon[40489]: 3.17 scrub ok 2024-09-21T14:06:55.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:55 smithi049 ceph-mon[40489]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T14:06:55.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:55 smithi049 ceph-mon[40489]: Cluster is now healthy 2024-09-21T14:06:55.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:55 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1030243059' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T14:06:55.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:55 smithi049 ceph-mon[40489]: osdmap e466: 8 total, 8 up, 8 in 2024-09-21T14:06:55.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:55 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2157423321' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:06:55.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:55 smithi049 ceph-mon[30080]: pgmap v917: 168 pgs: 168 active+clean; 9.9 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.2 MiB/s wr, 20 op/s 2024-09-21T14:06:55.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:55 smithi049 ceph-mon[30080]: 3.17 scrub starts 2024-09-21T14:06:55.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:55 smithi049 ceph-mon[30080]: 3.17 scrub ok 2024-09-21T14:06:55.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:55 smithi049 ceph-mon[30080]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T14:06:55.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:55 smithi049 ceph-mon[30080]: Cluster is now healthy 2024-09-21T14:06:55.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:55 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1030243059' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T14:06:55.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:55 smithi049 ceph-mon[30080]: osdmap e466: 8 total, 8 up, 8 in 2024-09-21T14:06:55.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:55 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2157423321' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:06:55.795 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:06:55.820 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:06:55.825 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:55.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:55.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:06:55.926 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:55.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:55.968 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:55.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:56.068 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:56.075 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T14:06:56.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:56.111 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:56.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:56.212 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:06:56.244 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:56.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:56.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:06:56.471 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:06:56.486 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:56.504 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 3 16 8577 8561 178.303 190.812 0.0046925 0.00559848 2024-09-21T14:06:56.524 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:56.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:56.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:56.626 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:06:56.662 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:56.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:56.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:06:56.764 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:56.807 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:56.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:56.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:56.909 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:06:56.942 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:56.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:56.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:06:57.032 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T14:06:57.033 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-21T14:06:57.033 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-21T14:06:57.033 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-21T14:06:57.033 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T14:06:57.044 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:06:57.084 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:57.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:57.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:06:57.187 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:06:57.228 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:57.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:57.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:06:57.329 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:57.336 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T14:06:57.339 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T14:06:57.339 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 466, 2024-09-21T14:06:57.339 INFO:teuthology.orchestra.run.smithi049.stdout: "fsid": "40a3d2e8-781f-11ef-baf6-efdc52797490", 2024-09-21T14:06:57.339 INFO:teuthology.orchestra.run.smithi049.stdout: "created": "2024-09-21T13:42:57.838275+0000", 2024-09-21T14:06:57.339 INFO:teuthology.orchestra.run.smithi049.stdout: "modified": "2024-09-21T14:06:54.397986+0000", 2024-09-21T14:06:57.339 INFO:teuthology.orchestra.run.smithi049.stdout: "last_up_change": "2024-09-21T14:01:40.158454+0000", 2024-09-21T14:06:57.340 INFO:teuthology.orchestra.run.smithi049.stdout: "last_in_change": "2024-09-21T13:58:16.730861+0000", 2024-09-21T14:06:57.340 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-21T14:06:57.340 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_num": 5799936, 2024-09-21T14:06:57.340 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_set": [ 2024-09-21T14:06:57.340 INFO:teuthology.orchestra.run.smithi049.stdout: "pglog_hardlimit", 2024-09-21T14:06:57.340 INFO:teuthology.orchestra.run.smithi049.stdout: "purged_snapdirs", 2024-09-21T14:06:57.340 INFO:teuthology.orchestra.run.smithi049.stdout: "recovery_deletes", 2024-09-21T14:06:57.340 INFO:teuthology.orchestra.run.smithi049.stdout: "sortbitwise" 2024-09-21T14:06:57.340 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T14:06:57.340 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_version": 22, 2024-09-21T14:06:57.340 INFO:teuthology.orchestra.run.smithi049.stdout: "full_ratio": 0.94999998807907104, 2024-09-21T14:06:57.340 INFO:teuthology.orchestra.run.smithi049.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-21T14:06:57.341 INFO:teuthology.orchestra.run.smithi049.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-21T14:06:57.341 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_snapshot": "", 2024-09-21T14:06:57.341 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_max": 11, 2024-09-21T14:06:57.341 INFO:teuthology.orchestra.run.smithi049.stdout: "max_osd": 8, 2024-09-21T14:06:57.341 INFO:teuthology.orchestra.run.smithi049.stdout: "require_min_compat_client": "luminous", 2024-09-21T14:06:57.341 INFO:teuthology.orchestra.run.smithi049.stdout: "min_compat_client": "luminous", 2024-09-21T14:06:57.341 INFO:teuthology.orchestra.run.smithi049.stdout: "require_osd_release": "squid", 2024-09-21T14:06:57.341 INFO:teuthology.orchestra.run.smithi049.stdout: "allow_crimson": false, 2024-09-21T14:06:57.341 INFO:teuthology.orchestra.run.smithi049.stdout: "pools": [ 2024-09-21T14:06:57.341 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.341 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 1, 2024-09-21T14:06:57.341 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": ".mgr", 2024-09-21T14:06:57.341 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:46:21.096244+0000", 2024-09-21T14:06:57.341 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T14:06:57.341 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T14:06:57.342 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T14:06:57.342 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T14:06:57.342 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T14:06:57.342 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T14:06:57.342 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T14:06:57.342 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T14:06:57.342 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T14:06:57.342 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T14:06:57.342 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T14:06:57.342 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T14:06:57.342 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T14:06:57.342 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 21, 2024-09-21T14:06:57.342 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 21, 2024-09-21T14:06:57.342 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 21, 2024-09-21T14:06:57.343 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 21, 2024-09-21T14:06:57.343 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 21, 2024-09-21T14:06:57.343 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T14:06:57.343 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T14:06:57.343 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T14:06:57.343 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T14:06:57.343 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T14:06:57.343 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T14:06:57.343 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T14:06:57.343 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.343 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "438", 2024-09-21T14:06:57.343 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T14:06:57.344 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T14:06:57.344 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "436", 2024-09-21T14:06:57.344 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T14:06:57.344 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T14:06:57.344 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T14:06:57.344 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T14:06:57.344 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T14:06:57.344 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T14:06:57.344 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T14:06:57.344 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T14:06:57.344 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T14:06:57.344 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T14:06:57.344 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T14:06:57.344 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T14:06:57.345 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T14:06:57.345 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T14:06:57.345 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T14:06:57.345 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T14:06:57.345 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T14:06:57.345 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T14:06:57.345 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T14:06:57.345 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T14:06:57.345 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T14:06:57.345 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T14:06:57.345 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T14:06:57.345 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.345 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T14:06:57.345 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T14:06:57.345 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T14:06:57.346 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T14:06:57.346 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T14:06:57.346 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T14:06:57.346 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T14:06:57.346 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T14:06:57.346 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T14:06:57.346 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T14:06:57.346 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T14:06:57.346 INFO:teuthology.orchestra.run.smithi049.stdout: "options": { 2024-09-21T14:06:57.346 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_max": 32, 2024-09-21T14:06:57.346 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_min": 1 2024-09-21T14:06:57.346 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.346 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T14:06:57.346 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr": {} 2024-09-21T14:06:57.347 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.347 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T14:06:57.347 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T14:06:57.347 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2.2899999618530273, 2024-09-21T14:06:57.347 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2.2899999618530273, 2024-09-21T14:06:57.347 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1, 2024-09-21T14:06:57.347 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2.2899999618530273, 2024-09-21T14:06:57.347 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2.2899999618530273, 2024-09-21T14:06:57.347 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.97835731506347656, 2024-09-21T14:06:57.347 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.98000001907348633, 2024-09-21T14:06:57.347 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.98000001907348633 2024-09-21T14:06:57.347 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:57.347 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.348 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.348 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 2, 2024-09-21T14:06:57.348 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "datapool", 2024-09-21T14:06:57.348 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:48:55.540712+0000", 2024-09-21T14:06:57.348 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 8193, 2024-09-21T14:06:57.348 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T14:06:57.348 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T14:06:57.348 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T14:06:57.348 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T14:06:57.348 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T14:06:57.348 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T14:06:57.348 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T14:06:57.348 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T14:06:57.348 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T14:06:57.348 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T14:06:57.348 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T14:06:57.349 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T14:06:57.349 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 3, 2024-09-21T14:06:57.349 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 3, 2024-09-21T14:06:57.349 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 3, 2024-09-21T14:06:57.349 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 3, 2024-09-21T14:06:57.349 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 3, 2024-09-21T14:06:57.349 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T14:06:57.349 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T14:06:57.349 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T14:06:57.349 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T14:06:57.349 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T14:06:57.349 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T14:06:57.349 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T14:06:57.349 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.349 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "63", 2024-09-21T14:06:57.350 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T14:06:57.350 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T14:06:57.350 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T14:06:57.350 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T14:06:57.350 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T14:06:57.350 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 2, 2024-09-21T14:06:57.350 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 63, 2024-09-21T14:06:57.350 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T14:06:57.350 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T14:06:57.350 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T14:06:57.350 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T14:06:57.350 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T14:06:57.351 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T14:06:57.351 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T14:06:57.351 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T14:06:57.351 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T14:06:57.351 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T14:06:57.351 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T14:06:57.351 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T14:06:57.351 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T14:06:57.351 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T14:06:57.351 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T14:06:57.351 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T14:06:57.351 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T14:06:57.351 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T14:06:57.351 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T14:06:57.351 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.352 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T14:06:57.352 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T14:06:57.352 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T14:06:57.352 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T14:06:57.352 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T14:06:57.352 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T14:06:57.352 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T14:06:57.352 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T14:06:57.352 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T14:06:57.352 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T14:06:57.352 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T14:06:57.352 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T14:06:57.352 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T14:06:57.352 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd": {} 2024-09-21T14:06:57.352 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.352 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T14:06:57.353 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T14:06:57.353 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2.6500000953674316, 2024-09-21T14:06:57.353 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2.6500000953674316, 2024-09-21T14:06:57.353 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 0.87999999523162842, 2024-09-21T14:06:57.353 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2.3299999237060547, 2024-09-21T14:06:57.353 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2.3299999237060547, 2024-09-21T14:06:57.353 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.97835731506347656, 2024-09-21T14:06:57.353 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.98000001907348633, 2024-09-21T14:06:57.353 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.98000001907348633 2024-09-21T14:06:57.353 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:57.353 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.353 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.353 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 3, 2024-09-21T14:06:57.354 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": ".rgw.root", 2024-09-21T14:06:57.354 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:48:56.417456+0000", 2024-09-21T14:06:57.354 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T14:06:57.354 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T14:06:57.354 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T14:06:57.354 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T14:06:57.354 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T14:06:57.354 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T14:06:57.354 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T14:06:57.354 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T14:06:57.354 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T14:06:57.354 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T14:06:57.354 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T14:06:57.354 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T14:06:57.355 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T14:06:57.355 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 32, 2024-09-21T14:06:57.355 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 32, 2024-09-21T14:06:57.355 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 32, 2024-09-21T14:06:57.355 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 32, 2024-09-21T14:06:57.355 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 32, 2024-09-21T14:06:57.355 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T14:06:57.355 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T14:06:57.355 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T14:06:57.355 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T14:06:57.355 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T14:06:57.355 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T14:06:57.355 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T14:06:57.355 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.356 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "62", 2024-09-21T14:06:57.356 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T14:06:57.356 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T14:06:57.356 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T14:06:57.356 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T14:06:57.356 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T14:06:57.356 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T14:06:57.356 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T14:06:57.356 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T14:06:57.356 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T14:06:57.356 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T14:06:57.356 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T14:06:57.356 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T14:06:57.356 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T14:06:57.356 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T14:06:57.357 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T14:06:57.357 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T14:06:57.357 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T14:06:57.357 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T14:06:57.357 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T14:06:57.357 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T14:06:57.357 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T14:06:57.357 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T14:06:57.357 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T14:06:57.357 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T14:06:57.357 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T14:06:57.357 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T14:06:57.357 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.357 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T14:06:57.357 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T14:06:57.357 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T14:06:57.358 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T14:06:57.358 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T14:06:57.358 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T14:06:57.358 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T14:06:57.358 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T14:06:57.358 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T14:06:57.358 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T14:06:57.358 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T14:06:57.358 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T14:06:57.358 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T14:06:57.358 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw": {} 2024-09-21T14:06:57.358 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.359 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T14:06:57.359 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T14:06:57.359 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 1.75, 2024-09-21T14:06:57.359 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 1.75, 2024-09-21T14:06:57.359 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1, 2024-09-21T14:06:57.359 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 1.75, 2024-09-21T14:06:57.359 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 1.75, 2024-09-21T14:06:57.359 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.97835731506347656, 2024-09-21T14:06:57.359 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.98000001907348633, 2024-09-21T14:06:57.359 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.98000001907348633 2024-09-21T14:06:57.359 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:57.359 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.359 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.359 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 4, 2024-09-21T14:06:57.359 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "default.rgw.log", 2024-09-21T14:06:57.360 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:48:58.128455+0000", 2024-09-21T14:06:57.360 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T14:06:57.360 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T14:06:57.360 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T14:06:57.360 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T14:06:57.360 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T14:06:57.360 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T14:06:57.360 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T14:06:57.360 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T14:06:57.360 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T14:06:57.360 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T14:06:57.360 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T14:06:57.360 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T14:06:57.360 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T14:06:57.360 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 32, 2024-09-21T14:06:57.360 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 32, 2024-09-21T14:06:57.361 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 32, 2024-09-21T14:06:57.361 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 32, 2024-09-21T14:06:57.361 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 32, 2024-09-21T14:06:57.361 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T14:06:57.361 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T14:06:57.361 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T14:06:57.361 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T14:06:57.361 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T14:06:57.361 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T14:06:57.361 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T14:06:57.361 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.361 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "64", 2024-09-21T14:06:57.361 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T14:06:57.361 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T14:06:57.362 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T14:06:57.362 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T14:06:57.362 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T14:06:57.362 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T14:06:57.362 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T14:06:57.362 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T14:06:57.362 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T14:06:57.362 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T14:06:57.362 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T14:06:57.362 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T14:06:57.362 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T14:06:57.362 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T14:06:57.362 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T14:06:57.362 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T14:06:57.362 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T14:06:57.363 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T14:06:57.363 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T14:06:57.363 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T14:06:57.363 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T14:06:57.363 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T14:06:57.363 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T14:06:57.363 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T14:06:57.363 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T14:06:57.363 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T14:06:57.363 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.363 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T14:06:57.363 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T14:06:57.363 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T14:06:57.365 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T14:06:57.365 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T14:06:57.366 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T14:06:57.366 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T14:06:57.366 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T14:06:57.366 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T14:06:57.366 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T14:06:57.366 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T14:06:57.366 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T14:06:57.366 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T14:06:57.366 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw": {} 2024-09-21T14:06:57.366 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.366 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T14:06:57.367 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T14:06:57.367 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2.5, 2024-09-21T14:06:57.367 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2.5, 2024-09-21T14:06:57.367 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1, 2024-09-21T14:06:57.367 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2.5, 2024-09-21T14:06:57.367 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2.5, 2024-09-21T14:06:57.367 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.97835731506347656, 2024-09-21T14:06:57.367 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.98000001907348633, 2024-09-21T14:06:57.367 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.98000001907348633 2024-09-21T14:06:57.367 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:57.368 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.368 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.368 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 5, 2024-09-21T14:06:57.368 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "default.rgw.control", 2024-09-21T14:06:57.368 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:49:00.069361+0000", 2024-09-21T14:06:57.368 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T14:06:57.368 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T14:06:57.368 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T14:06:57.368 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T14:06:57.368 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T14:06:57.368 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T14:06:57.368 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T14:06:57.369 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T14:06:57.369 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T14:06:57.369 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T14:06:57.369 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T14:06:57.369 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T14:06:57.369 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T14:06:57.369 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 32, 2024-09-21T14:06:57.369 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 32, 2024-09-21T14:06:57.369 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 32, 2024-09-21T14:06:57.369 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 32, 2024-09-21T14:06:57.370 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 32, 2024-09-21T14:06:57.370 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T14:06:57.370 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T14:06:57.370 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T14:06:57.370 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T14:06:57.370 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T14:06:57.370 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T14:06:57.370 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T14:06:57.370 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.370 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "66", 2024-09-21T14:06:57.370 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T14:06:57.371 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T14:06:57.371 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T14:06:57.371 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T14:06:57.371 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T14:06:57.371 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T14:06:57.371 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T14:06:57.371 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T14:06:57.371 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T14:06:57.371 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T14:06:57.371 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T14:06:57.372 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T14:06:57.372 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T14:06:57.372 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T14:06:57.372 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T14:06:57.372 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T14:06:57.372 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T14:06:57.372 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T14:06:57.372 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T14:06:57.372 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T14:06:57.373 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T14:06:57.373 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T14:06:57.373 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T14:06:57.373 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T14:06:57.373 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T14:06:57.373 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T14:06:57.373 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.373 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T14:06:57.373 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T14:06:57.373 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T14:06:57.373 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T14:06:57.374 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T14:06:57.374 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T14:06:57.374 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T14:06:57.374 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T14:06:57.374 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T14:06:57.374 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T14:06:57.374 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T14:06:57.374 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T14:06:57.374 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T14:06:57.374 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw": {} 2024-09-21T14:06:57.375 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.375 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T14:06:57.375 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T14:06:57.375 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 1.5, 2024-09-21T14:06:57.375 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 1.5, 2024-09-21T14:06:57.375 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1, 2024-09-21T14:06:57.375 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 1.5, 2024-09-21T14:06:57.376 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 1.5, 2024-09-21T14:06:57.376 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.97835731506347656, 2024-09-21T14:06:57.376 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.98000001907348633, 2024-09-21T14:06:57.376 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.98000001907348633 2024-09-21T14:06:57.376 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:57.376 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.376 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.376 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 6, 2024-09-21T14:06:57.376 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "default.rgw.meta", 2024-09-21T14:06:57.377 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:49:02.127399+0000", 2024-09-21T14:06:57.377 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T14:06:57.377 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T14:06:57.377 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T14:06:57.377 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T14:06:57.377 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T14:06:57.377 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T14:06:57.377 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T14:06:57.377 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T14:06:57.377 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T14:06:57.377 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T14:06:57.377 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T14:06:57.377 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T14:06:57.378 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T14:06:57.378 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 32, 2024-09-21T14:06:57.378 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 32, 2024-09-21T14:06:57.378 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 32, 2024-09-21T14:06:57.378 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 32, 2024-09-21T14:06:57.378 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 32, 2024-09-21T14:06:57.378 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T14:06:57.378 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T14:06:57.378 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T14:06:57.378 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T14:06:57.378 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T14:06:57.378 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T14:06:57.378 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T14:06:57.379 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.379 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "68", 2024-09-21T14:06:57.379 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T14:06:57.379 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T14:06:57.379 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T14:06:57.379 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T14:06:57.379 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T14:06:57.379 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T14:06:57.379 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T14:06:57.379 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T14:06:57.379 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T14:06:57.379 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T14:06:57.380 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T14:06:57.380 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T14:06:57.380 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T14:06:57.380 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T14:06:57.380 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T14:06:57.380 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T14:06:57.380 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T14:06:57.380 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T14:06:57.380 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T14:06:57.380 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T14:06:57.380 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T14:06:57.380 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T14:06:57.381 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T14:06:57.381 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T14:06:57.381 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T14:06:57.381 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T14:06:57.381 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.381 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T14:06:57.381 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T14:06:57.381 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T14:06:57.381 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T14:06:57.381 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T14:06:57.381 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T14:06:57.381 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T14:06:57.381 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T14:06:57.382 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T14:06:57.382 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T14:06:57.382 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T14:06:57.382 INFO:teuthology.orchestra.run.smithi049.stdout: "options": { 2024-09-21T14:06:57.382 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_bias": 4 2024-09-21T14:06:57.382 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.382 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T14:06:57.382 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw": {} 2024-09-21T14:06:57.382 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.382 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T14:06:57.382 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T14:06:57.382 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2, 2024-09-21T14:06:57.382 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2, 2024-09-21T14:06:57.383 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1, 2024-09-21T14:06:57.383 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2, 2024-09-21T14:06:57.383 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2, 2024-09-21T14:06:57.383 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.97835731506347656, 2024-09-21T14:06:57.383 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.98000001907348633, 2024-09-21T14:06:57.383 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.98000001907348633 2024-09-21T14:06:57.383 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:57.383 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.383 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.383 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 11, 2024-09-21T14:06:57.384 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "unique_pool_4", 2024-09-21T14:06:57.384 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T14:06:43.392407+0000", 2024-09-21T14:06:57.384 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T14:06:57.384 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T14:06:57.384 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T14:06:57.384 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T14:06:57.384 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T14:06:57.384 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T14:06:57.384 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T14:06:57.384 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T14:06:57.384 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T14:06:57.385 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T14:06:57.385 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T14:06:57.385 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T14:06:57.385 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T14:06:57.385 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 16, 2024-09-21T14:06:57.385 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 16, 2024-09-21T14:06:57.385 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 16, 2024-09-21T14:06:57.385 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 16, 2024-09-21T14:06:57.385 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 16, 2024-09-21T14:06:57.385 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T14:06:57.385 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T14:06:57.386 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T14:06:57.386 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T14:06:57.386 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T14:06:57.386 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T14:06:57.386 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T14:06:57.386 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.386 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "465", 2024-09-21T14:06:57.386 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T14:06:57.386 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T14:06:57.386 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T14:06:57.386 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T14:06:57.386 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T14:06:57.387 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T14:06:57.387 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T14:06:57.387 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T14:06:57.387 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T14:06:57.387 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T14:06:57.387 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T14:06:57.387 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T14:06:57.387 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T14:06:57.387 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T14:06:57.387 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T14:06:57.387 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T14:06:57.387 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T14:06:57.387 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T14:06:57.388 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T14:06:57.388 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T14:06:57.388 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T14:06:57.388 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T14:06:57.388 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T14:06:57.388 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T14:06:57.388 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T14:06:57.388 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T14:06:57.388 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.388 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T14:06:57.388 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T14:06:57.388 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T14:06:57.388 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T14:06:57.389 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T14:06:57.389 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T14:06:57.389 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T14:06:57.389 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T14:06:57.389 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T14:06:57.389 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T14:06:57.389 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T14:06:57.389 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T14:06:57.389 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T14:06:57.389 INFO:teuthology.orchestra.run.smithi049.stdout: "rados": {} 2024-09-21T14:06:57.389 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.389 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T14:06:57.389 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T14:06:57.390 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2.5, 2024-09-21T14:06:57.390 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2.5, 2024-09-21T14:06:57.390 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1, 2024-09-21T14:06:57.390 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2.5, 2024-09-21T14:06:57.390 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2.5, 2024-09-21T14:06:57.390 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.97835731506347656, 2024-09-21T14:06:57.390 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.98000001907348633, 2024-09-21T14:06:57.390 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.98000001907348633 2024-09-21T14:06:57.390 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:57.390 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:57.390 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T14:06:57.391 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-09-21T14:06:57.391 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.391 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 0, 2024-09-21T14:06:57.391 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "2595dca9-6219-4e0c-8728-751eb102e73a", 2024-09-21T14:06:57.391 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T14:06:57.391 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T14:06:57.391 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 0.6176910400390625, 2024-09-21T14:06:57.391 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T14:06:57.391 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T14:06:57.391 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T14:06:57.391 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 8, 2024-09-21T14:06:57.391 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 429, 2024-09-21T14:06:57.391 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T14:06:57.392 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T14:06:57.392 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T14:06:57.392 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:06:57.392 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.392 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:06:57.392 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6802", 2024-09-21T14:06:57.392 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T14:06:57.392 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.392 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.392 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:06:57.392 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6803", 2024-09-21T14:06:57.392 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T14:06:57.392 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:57.393 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:57.393 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.393 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T14:06:57.393 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:06:57.393 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.393 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:06:57.393 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6804", 2024-09-21T14:06:57.393 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T14:06:57.393 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.393 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.393 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:06:57.393 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6805", 2024-09-21T14:06:57.393 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T14:06:57.394 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:57.394 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:57.394 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.394 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T14:06:57.394 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:06:57.394 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.394 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:06:57.394 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6808", 2024-09-21T14:06:57.394 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T14:06:57.394 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.394 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.394 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:06:57.395 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6809", 2024-09-21T14:06:57.395 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T14:06:57.395 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:57.395 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:57.395 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.395 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T14:06:57.395 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:06:57.395 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.395 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:06:57.395 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6806", 2024-09-21T14:06:57.395 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T14:06:57.395 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.395 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.396 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:06:57.396 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6807", 2024-09-21T14:06:57.396 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T14:06:57.396 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:57.396 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:57.396 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.396 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6803/2095561412", 2024-09-21T14:06:57.396 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6805/2095561412", 2024-09-21T14:06:57.396 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6809/2095561412", 2024-09-21T14:06:57.396 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6807/2095561412", 2024-09-21T14:06:57.396 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T14:06:57.396 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T14:06:57.397 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T14:06:57.397 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:57.397 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.397 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.397 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 1, 2024-09-21T14:06:57.397 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "0ba327f9-5524-4a40-abcc-729ded0b1917", 2024-09-21T14:06:57.397 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T14:06:57.397 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T14:06:57.397 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T14:06:57.397 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T14:06:57.397 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T14:06:57.397 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T14:06:57.397 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 13, 2024-09-21T14:06:57.398 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 461, 2024-09-21T14:06:57.398 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T14:06:57.398 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T14:06:57.398 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T14:06:57.398 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:06:57.398 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.398 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:06:57.398 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6810", 2024-09-21T14:06:57.398 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T14:06:57.398 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.398 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.398 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:06:57.399 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6811", 2024-09-21T14:06:57.399 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T14:06:57.399 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:57.399 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:57.399 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.399 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T14:06:57.399 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:06:57.399 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.399 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:06:57.399 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6812", 2024-09-21T14:06:57.399 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T14:06:57.399 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.400 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.400 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:06:57.400 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6813", 2024-09-21T14:06:57.400 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T14:06:57.400 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:57.400 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:57.400 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.400 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T14:06:57.400 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:06:57.400 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.400 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:06:57.400 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6816", 2024-09-21T14:06:57.400 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T14:06:57.400 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.401 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.401 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:06:57.401 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6817", 2024-09-21T14:06:57.401 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T14:06:57.401 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:57.401 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:57.401 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.401 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T14:06:57.401 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:06:57.401 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.401 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:06:57.402 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6814", 2024-09-21T14:06:57.402 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T14:06:57.402 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.402 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.402 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:06:57.402 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6815", 2024-09-21T14:06:57.402 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T14:06:57.402 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:57.402 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:57.402 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.403 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6811/1382840527", 2024-09-21T14:06:57.403 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6813/1382840527", 2024-09-21T14:06:57.403 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6817/1382840527", 2024-09-21T14:06:57.403 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6815/1382840527", 2024-09-21T14:06:57.403 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T14:06:57.403 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T14:06:57.403 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T14:06:57.403 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:57.403 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.403 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.403 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 2, 2024-09-21T14:06:57.403 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "27a45750-204d-48fd-bb8f-ebb01273343d", 2024-09-21T14:06:57.403 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T14:06:57.404 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T14:06:57.404 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T14:06:57.404 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 0.8268585205078125, 2024-09-21T14:06:57.404 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 18, 2024-09-21T14:06:57.404 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 100, 2024-09-21T14:06:57.404 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 130, 2024-09-21T14:06:57.404 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 461, 2024-09-21T14:06:57.404 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 101, 2024-09-21T14:06:57.404 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T14:06:57.404 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T14:06:57.404 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:06:57.404 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.405 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:06:57.405 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6818", 2024-09-21T14:06:57.405 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T14:06:57.405 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.405 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.405 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:06:57.405 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6819", 2024-09-21T14:06:57.405 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T14:06:57.405 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:57.405 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:57.405 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.405 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T14:06:57.405 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:06:57.406 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.406 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:06:57.406 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6820", 2024-09-21T14:06:57.406 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T14:06:57.406 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.406 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.406 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:06:57.406 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6821", 2024-09-21T14:06:57.406 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T14:06:57.406 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:57.406 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:57.406 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.406 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T14:06:57.406 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:06:57.407 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.407 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:06:57.407 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6824", 2024-09-21T14:06:57.407 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T14:06:57.407 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.407 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.407 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:06:57.407 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6825", 2024-09-21T14:06:57.407 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T14:06:57.407 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:57.407 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:57.407 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.407 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T14:06:57.408 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:06:57.408 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.408 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:06:57.408 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6822", 2024-09-21T14:06:57.408 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T14:06:57.408 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.408 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.408 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:06:57.408 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6823", 2024-09-21T14:06:57.408 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T14:06:57.408 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:57.408 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:57.408 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.409 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6819/1945314741", 2024-09-21T14:06:57.409 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6821/1945314741", 2024-09-21T14:06:57.409 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6825/1945314741", 2024-09-21T14:06:57.409 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6823/1945314741", 2024-09-21T14:06:57.409 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T14:06:57.409 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T14:06:57.409 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T14:06:57.409 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:57.409 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.409 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.409 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 3, 2024-09-21T14:06:57.409 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "528bc390-3dc9-4f3b-ab75-d7412c9625db", 2024-09-21T14:06:57.409 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T14:06:57.410 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T14:06:57.410 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T14:06:57.410 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T14:06:57.410 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 27, 2024-09-21T14:06:57.410 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 214, 2024-09-21T14:06:57.410 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 240, 2024-09-21T14:06:57.410 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 461, 2024-09-21T14:06:57.410 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 215, 2024-09-21T14:06:57.410 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T14:06:57.410 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T14:06:57.410 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:06:57.410 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.410 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:06:57.411 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6826", 2024-09-21T14:06:57.411 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T14:06:57.411 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.411 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.411 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:06:57.411 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6827", 2024-09-21T14:06:57.411 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T14:06:57.411 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:57.411 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:57.411 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.411 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T14:06:57.411 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:06:57.411 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.412 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:06:57.412 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6828", 2024-09-21T14:06:57.412 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T14:06:57.412 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.412 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.412 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:06:57.412 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6829", 2024-09-21T14:06:57.412 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T14:06:57.412 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:57.412 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:57.413 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.413 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T14:06:57.413 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:06:57.413 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.413 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:06:57.413 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6832", 2024-09-21T14:06:57.413 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T14:06:57.413 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.413 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.413 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:06:57.413 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6833", 2024-09-21T14:06:57.413 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T14:06:57.413 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:57.414 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:57.414 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.414 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T14:06:57.414 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:06:57.414 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.414 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:06:57.414 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6830", 2024-09-21T14:06:57.414 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T14:06:57.414 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.414 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.414 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:06:57.414 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6831", 2024-09-21T14:06:57.415 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T14:06:57.415 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:57.415 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:57.415 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:57.416 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:57.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:57.417 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6827/886735721", 2024-09-21T14:06:57.417 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6829/886735721", 2024-09-21T14:06:57.417 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6833/886735721", 2024-09-21T14:06:57.417 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6831/886735721", 2024-09-21T14:06:57.417 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T14:06:57.417 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T14:06:57.417 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T14:06:57.417 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:57.417 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.417 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.417 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 4, 2024-09-21T14:06:57.417 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "6fc64698-2d37-4db0-bf3d-c7e612c45163", 2024-09-21T14:06:57.417 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T14:06:57.418 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T14:06:57.418 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 0.7888946533203125, 2024-09-21T14:06:57.418 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T14:06:57.418 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T14:06:57.418 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T14:06:57.418 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 33, 2024-09-21T14:06:57.418 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 436, 2024-09-21T14:06:57.418 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T14:06:57.418 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T14:06:57.418 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T14:06:57.418 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:06:57.418 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.419 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:06:57.419 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6800", 2024-09-21T14:06:57.419 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T14:06:57.419 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.419 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.419 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:06:57.419 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6801", 2024-09-21T14:06:57.419 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T14:06:57.419 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:57.419 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:57.419 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.420 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T14:06:57.420 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:06:57.420 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.420 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:06:57.420 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6802", 2024-09-21T14:06:57.420 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T14:06:57.420 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.420 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.420 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:06:57.420 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6803", 2024-09-21T14:06:57.420 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T14:06:57.420 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:57.421 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:57.421 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.421 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T14:06:57.421 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:06:57.421 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.421 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:06:57.421 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6806", 2024-09-21T14:06:57.421 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T14:06:57.421 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.421 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.421 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:06:57.421 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6807", 2024-09-21T14:06:57.421 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T14:06:57.422 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:57.422 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:57.422 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.422 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T14:06:57.422 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:06:57.422 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.422 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:06:57.422 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6804", 2024-09-21T14:06:57.422 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T14:06:57.422 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.422 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.422 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:06:57.423 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6805", 2024-09-21T14:06:57.423 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T14:06:57.423 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:57.423 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:57.423 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.423 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.187:6801/1702447776", 2024-09-21T14:06:57.423 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.187:6803/1702447776", 2024-09-21T14:06:57.423 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.187:6807/1702447776", 2024-09-21T14:06:57.423 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.187:6805/1702447776", 2024-09-21T14:06:57.423 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T14:06:57.423 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T14:06:57.423 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T14:06:57.424 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:57.424 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.424 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.424 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 5, 2024-09-21T14:06:57.424 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "8d7f62a1-7e92-4e06-9cad-86b96be514d8", 2024-09-21T14:06:57.424 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T14:06:57.424 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T14:06:57.424 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T14:06:57.424 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T14:06:57.424 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 39, 2024-09-21T14:06:57.424 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 324, 2024-09-21T14:06:57.424 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 345, 2024-09-21T14:06:57.425 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 461, 2024-09-21T14:06:57.425 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 325, 2024-09-21T14:06:57.425 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T14:06:57.425 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T14:06:57.425 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:06:57.425 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.425 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:06:57.425 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6808", 2024-09-21T14:06:57.425 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1595353965 2024-09-21T14:06:57.425 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.425 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.425 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:06:57.425 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6809", 2024-09-21T14:06:57.426 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1595353965 2024-09-21T14:06:57.426 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:57.426 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:57.426 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.426 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T14:06:57.426 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:06:57.426 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.426 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:06:57.426 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6810", 2024-09-21T14:06:57.426 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1595353965 2024-09-21T14:06:57.426 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.426 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.426 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:06:57.427 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6811", 2024-09-21T14:06:57.427 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1595353965 2024-09-21T14:06:57.427 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:57.427 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:57.427 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.427 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T14:06:57.427 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:06:57.427 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.427 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:06:57.427 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6814", 2024-09-21T14:06:57.427 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1595353965 2024-09-21T14:06:57.428 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.428 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.428 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:06:57.428 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6815", 2024-09-21T14:06:57.428 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1595353965 2024-09-21T14:06:57.428 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:57.428 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:57.428 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.428 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T14:06:57.428 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:06:57.428 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.428 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:06:57.428 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6812", 2024-09-21T14:06:57.429 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1595353965 2024-09-21T14:06:57.429 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.429 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.429 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:06:57.429 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6813", 2024-09-21T14:06:57.429 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1595353965 2024-09-21T14:06:57.429 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:57.429 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:57.429 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.429 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.187:6809/1595353965", 2024-09-21T14:06:57.429 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.187:6811/1595353965", 2024-09-21T14:06:57.429 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.187:6815/1595353965", 2024-09-21T14:06:57.429 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.187:6813/1595353965", 2024-09-21T14:06:57.430 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T14:06:57.430 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T14:06:57.430 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T14:06:57.430 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:57.430 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.430 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.430 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 6, 2024-09-21T14:06:57.430 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "3a7afd13-26ce-444c-9cc0-81ce2eab6ce5", 2024-09-21T14:06:57.430 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T14:06:57.430 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T14:06:57.430 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 0.9076080322265625, 2024-09-21T14:06:57.430 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T14:06:57.431 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T14:06:57.431 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T14:06:57.431 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 47, 2024-09-21T14:06:57.431 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 461, 2024-09-21T14:06:57.431 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T14:06:57.431 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T14:06:57.431 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T14:06:57.431 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:06:57.431 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.431 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:06:57.431 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6816", 2024-09-21T14:06:57.431 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T14:06:57.432 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.432 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.432 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:06:57.432 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6817", 2024-09-21T14:06:57.432 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T14:06:57.432 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:57.432 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:57.432 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.432 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T14:06:57.432 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:06:57.432 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.432 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:06:57.433 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6818", 2024-09-21T14:06:57.433 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T14:06:57.433 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.433 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.433 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:06:57.433 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6819", 2024-09-21T14:06:57.433 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T14:06:57.433 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:57.433 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:57.433 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.433 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T14:06:57.433 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:06:57.433 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.433 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:06:57.434 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6822", 2024-09-21T14:06:57.434 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T14:06:57.434 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.434 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.434 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:06:57.434 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6823", 2024-09-21T14:06:57.434 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T14:06:57.434 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:57.434 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:57.434 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.434 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T14:06:57.435 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:06:57.435 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.435 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:06:57.435 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6820", 2024-09-21T14:06:57.435 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T14:06:57.435 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.435 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.435 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:06:57.435 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6821", 2024-09-21T14:06:57.435 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T14:06:57.435 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:57.435 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:57.435 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.435 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.187:6817/2728426525", 2024-09-21T14:06:57.436 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.187:6819/2728426525", 2024-09-21T14:06:57.436 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.187:6823/2728426525", 2024-09-21T14:06:57.436 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.187:6821/2728426525", 2024-09-21T14:06:57.436 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T14:06:57.436 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T14:06:57.436 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T14:06:57.436 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:57.436 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.436 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.436 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 7, 2024-09-21T14:06:57.436 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "1e08d891-8747-402e-b4c2-aa3802ce8537", 2024-09-21T14:06:57.436 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T14:06:57.436 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T14:06:57.437 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T14:06:57.437 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T14:06:57.437 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 54, 2024-09-21T14:06:57.437 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 272, 2024-09-21T14:06:57.437 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 298, 2024-09-21T14:06:57.437 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 461, 2024-09-21T14:06:57.437 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 273, 2024-09-21T14:06:57.437 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T14:06:57.437 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T14:06:57.437 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:06:57.437 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.437 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:06:57.437 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6824", 2024-09-21T14:06:57.438 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 13894964 2024-09-21T14:06:57.438 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.438 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.438 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:06:57.438 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6825", 2024-09-21T14:06:57.438 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 13894964 2024-09-21T14:06:57.438 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:57.438 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:57.438 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.438 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T14:06:57.438 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:06:57.438 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.439 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:06:57.439 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6826", 2024-09-21T14:06:57.439 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 13894964 2024-09-21T14:06:57.439 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.439 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.439 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:06:57.439 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6827", 2024-09-21T14:06:57.439 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 13894964 2024-09-21T14:06:57.439 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:57.439 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:57.439 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.439 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T14:06:57.440 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:06:57.440 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.440 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:06:57.440 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6830", 2024-09-21T14:06:57.440 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 13894964 2024-09-21T14:06:57.440 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.440 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.440 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:06:57.440 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6831", 2024-09-21T14:06:57.440 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 13894964 2024-09-21T14:06:57.440 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:57.441 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:57.441 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.441 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T14:06:57.441 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:06:57.441 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.441 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:06:57.441 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6828", 2024-09-21T14:06:57.441 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 13894964 2024-09-21T14:06:57.441 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.441 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.441 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:06:57.441 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6829", 2024-09-21T14:06:57.441 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 13894964 2024-09-21T14:06:57.442 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:57.442 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:57.442 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.442 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.187:6825/13894964", 2024-09-21T14:06:57.442 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.187:6827/13894964", 2024-09-21T14:06:57.442 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.187:6831/13894964", 2024-09-21T14:06:57.442 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.187:6829/13894964", 2024-09-21T14:06:57.442 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T14:06:57.442 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T14:06:57.442 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T14:06:57.442 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:57.442 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:57.443 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T14:06:57.443 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_xinfo": [ 2024-09-21T14:06:57.443 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.443 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 0, 2024-09-21T14:06:57.443 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T14:06:57.443 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T14:06:57.443 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T14:06:57.443 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T14:06:57.443 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T14:06:57.443 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T14:06:38.896732+0000", 2024-09-21T14:06:57.443 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T14:06:57.444 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.444 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.444 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 1, 2024-09-21T14:06:57.444 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T14:06:57.444 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T14:06:57.444 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T14:06:57.444 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T14:06:57.444 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T14:06:57.444 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T14:06:18.855219+0000", 2024-09-21T14:06:57.444 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T14:06:57.444 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.444 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.445 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 2, 2024-09-21T14:06:57.445 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T14:06:57.445 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T14:06:57.445 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T14:06:57.445 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T14:06:57.445 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T14:06:57.445 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T14:06:27.643288+0000", 2024-09-21T14:06:57.445 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 100 2024-09-21T14:06:57.445 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.445 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.445 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 3, 2024-09-21T14:06:57.445 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T14:06:57.445 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T14:06:57.446 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T14:06:57.446 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T14:06:57.446 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T14:06:57.446 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T14:05:41.501105+0000", 2024-09-21T14:06:57.446 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 214 2024-09-21T14:06:57.446 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.446 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.446 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 4, 2024-09-21T14:06:57.446 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T14:06:57.446 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T14:06:57.446 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T14:06:57.446 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T14:06:57.446 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T14:06:57.447 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T14:06:40.662638+0000", 2024-09-21T14:06:57.447 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T14:06:57.447 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.447 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.447 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 5, 2024-09-21T14:06:57.447 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T14:06:57.447 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T14:06:57.447 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T14:06:57.447 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T14:06:57.447 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T14:06:57.447 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T14:05:39.824748+0000", 2024-09-21T14:06:57.447 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 324 2024-09-21T14:06:57.447 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.447 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.448 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 6, 2024-09-21T14:06:57.448 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T14:06:57.448 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T14:06:57.448 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T14:06:57.448 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T14:06:57.448 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T14:06:57.448 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T14:06:06.798526+0000", 2024-09-21T14:06:57.448 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T14:06:57.448 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.448 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.448 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 7, 2024-09-21T14:06:57.449 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T14:06:57.449 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T14:06:57.449 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T14:06:57.449 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T14:06:57.449 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T14:06:57.449 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T14:06:03.637111+0000", 2024-09-21T14:06:57.449 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 272 2024-09-21T14:06:57.449 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:57.449 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T14:06:57.449 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap": [ 2024-09-21T14:06:57.449 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.449 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "3.14", 2024-09-21T14:06:57.449 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-09-21T14:06:57.450 INFO:teuthology.orchestra.run.smithi049.stdout: 7, 2024-09-21T14:06:57.450 INFO:teuthology.orchestra.run.smithi049.stdout: 6, 2024-09-21T14:06:57.450 INFO:teuthology.orchestra.run.smithi049.stdout: 0 2024-09-21T14:06:57.450 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:57.450 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:57.450 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T14:06:57.450 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap_items": [ 2024-09-21T14:06:57.450 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.451 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "4.2", 2024-09-21T14:06:57.451 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-09-21T14:06:57.451 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.451 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 1, 2024-09-21T14:06:57.451 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 2 2024-09-21T14:06:57.451 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:57.451 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:57.451 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.451 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.451 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "4.6", 2024-09-21T14:06:57.451 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-09-21T14:06:57.451 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.451 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 1, 2024-09-21T14:06:57.452 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 5 2024-09-21T14:06:57.452 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:57.452 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:57.452 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.452 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.452 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "4.d", 2024-09-21T14:06:57.452 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-09-21T14:06:57.452 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.452 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 1, 2024-09-21T14:06:57.452 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 5 2024-09-21T14:06:57.452 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:57.452 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:57.452 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.452 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.453 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "6.7", 2024-09-21T14:06:57.453 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-09-21T14:06:57.453 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.453 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 3, 2024-09-21T14:06:57.453 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 7 2024-09-21T14:06:57.453 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:57.453 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:57.453 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.453 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.453 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "6.b", 2024-09-21T14:06:57.453 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-09-21T14:06:57.453 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.453 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 3, 2024-09-21T14:06:57.453 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 1 2024-09-21T14:06:57.454 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.454 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.454 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 7, 2024-09-21T14:06:57.454 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 6 2024-09-21T14:06:57.454 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:57.454 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:57.454 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.454 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.454 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "6.1f", 2024-09-21T14:06:57.455 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-09-21T14:06:57.455 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:06:57.455 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 3, 2024-09-21T14:06:57.455 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 0 2024-09-21T14:06:57.455 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:57.455 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:06:57.455 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:57.455 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T14:06:57.455 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap_primaries": [], 2024-09-21T14:06:57.455 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_temp": [], 2024-09-21T14:06:57.455 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_temp": [], 2024-09-21T14:06:57.456 INFO:teuthology.orchestra.run.smithi049.stdout: "blocklist": { 2024-09-21T14:06:57.456 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/2731793917": "2024-09-22T13:49:27.311313+0000", 2024-09-21T14:06:57.456 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/799831425": "2024-09-22T13:49:27.311313+0000", 2024-09-21T14:06:57.456 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/2518197280": "2024-09-22T13:43:28.110499+0000", 2024-09-21T14:06:57.456 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3556829486": "2024-09-22T13:43:28.110499+0000", 2024-09-21T14:06:57.456 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/1249926115": "2024-09-22T13:49:27.311313+0000", 2024-09-21T14:06:57.456 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/231530138": "2024-09-22T13:43:28.110499+0000", 2024-09-21T14:06:57.456 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/4220781134": "2024-09-22T13:49:27.311313+0000", 2024-09-21T14:06:57.456 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/202325455": "2024-09-22T13:43:28.110499+0000", 2024-09-21T14:06:57.456 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/4129636473": "2024-09-22T13:43:56.633723+0000", 2024-09-21T14:06:57.456 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/2518197280": "2024-09-22T13:43:28.110499+0000", 2024-09-21T14:06:57.456 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3514741273": "2024-09-22T13:43:56.633723+0000", 2024-09-21T14:06:57.457 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/4129636473": "2024-09-22T13:43:56.633723+0000", 2024-09-21T14:06:57.457 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3125743776": "2024-09-22T13:43:56.633723+0000", 2024-09-21T14:06:57.457 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3600977363": "2024-09-22T13:43:56.633723+0000", 2024-09-21T14:06:57.457 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/3453571598": "2024-09-22T13:49:27.311313+0000", 2024-09-21T14:06:57.457 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/3453571598": "2024-09-22T13:49:27.311313+0000", 2024-09-21T14:06:57.457 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3553734432": "2024-09-22T13:49:27.311313+0000" 2024-09-21T14:06:57.457 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.457 INFO:teuthology.orchestra.run.smithi049.stdout: "range_blocklist": {}, 2024-09-21T14:06:57.457 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profiles": { 2024-09-21T14:06:57.457 INFO:teuthology.orchestra.run.smithi049.stdout: "default": { 2024-09-21T14:06:57.457 INFO:teuthology.orchestra.run.smithi049.stdout: "crush-failure-domain": "osd", 2024-09-21T14:06:57.457 INFO:teuthology.orchestra.run.smithi049.stdout: "k": "2", 2024-09-21T14:06:57.457 INFO:teuthology.orchestra.run.smithi049.stdout: "m": "1", 2024-09-21T14:06:57.457 INFO:teuthology.orchestra.run.smithi049.stdout: "plugin": "jerasure", 2024-09-21T14:06:57.458 INFO:teuthology.orchestra.run.smithi049.stdout: "technique": "reed_sol_van" 2024-09-21T14:06:57.458 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:57.458 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:06:57.458 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps_queue": [], 2024-09-21T14:06:57.458 INFO:teuthology.orchestra.run.smithi049.stdout: "new_removed_snaps": [], 2024-09-21T14:06:57.458 INFO:teuthology.orchestra.run.smithi049.stdout: "new_purged_snaps": [], 2024-09-21T14:06:57.458 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_node_flags": {}, 2024-09-21T14:06:57.458 INFO:teuthology.orchestra.run.smithi049.stdout: "device_class_flags": {}, 2024-09-21T14:06:57.458 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode": { 2024-09-21T14:06:57.458 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode_enabled": false, 2024-09-21T14:06:57.458 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_bucket_count": 0, 2024-09-21T14:06:57.458 INFO:teuthology.orchestra.run.smithi049.stdout: "degraded_stretch_mode": 0, 2024-09-21T14:06:57.459 INFO:teuthology.orchestra.run.smithi049.stdout: "recovering_stretch_mode": 0, 2024-09-21T14:06:57.459 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode_bucket": 0 2024-09-21T14:06:57.459 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:06:57.459 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T14:06:57.504 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 4 16 11231 11215 175.193 165.875 0.00319961 0.00569897 2024-09-21T14:06:57.518 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:57.558 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:57.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:57.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:57.628 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T14:06:57.659 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:06:57.670 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:57 smithi049 ceph-mon[40489]: pgmap v919: 168 pgs: 168 active+clean; 78 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 13 MiB/s wr, 207 op/s 2024-09-21T14:06:57.670 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:57 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:06:57.671 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:57 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:06:57.671 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:57 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1682006021' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T14:06:57.671 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:57 smithi049 ceph-mon[30080]: pgmap v919: 168 pgs: 168 active+clean; 78 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 13 MiB/s wr, 207 op/s 2024-09-21T14:06:57.671 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:57 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:06:57.671 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:57 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:06:57.671 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:57 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1682006021' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T14:06:57.689 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:57.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:57.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:06:57.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:57 smithi187 ceph-mon[33032]: pgmap v919: 168 pgs: 168 active+clean; 78 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 13 MiB/s wr, 207 op/s 2024-09-21T14:06:57.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:57 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:06:57.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:57 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:06:57.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:57 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1682006021' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T14:06:57.776 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T14:06:57.791 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:57.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:57.833 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:57.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:57.934 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:06:57.965 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:57.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:57.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:06:58.066 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:06:58.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:58.107 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:58.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:06:58.177 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:06:58.207 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:58.246 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:58.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:58.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:58.348 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:06:58.369 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:58.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:58.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:06:58.387 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T14:06:58.471 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:06:58.505 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 5 16 13565 13549 169.307 145.875 0.0085797 0.00589771 2024-09-21T14:06:58.506 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:58.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:58.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:06:58.607 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:06:58.648 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:58.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:58.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:06:58.656 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:58 smithi187 ceph-mon[33032]: 5.6 scrub starts 2024-09-21T14:06:58.656 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:58 smithi187 ceph-mon[33032]: 5.6 scrub ok 2024-09-21T14:06:58.656 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:58 smithi187 ceph-mon[33032]: osdmap e467: 8 total, 8 up, 8 in 2024-09-21T14:06:58.656 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:58 smithi187 ceph-mon[33032]: 5.14 scrub starts 2024-09-21T14:06:58.656 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:58 smithi187 ceph-mon[33032]: 5.14 scrub ok 2024-09-21T14:06:58.656 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:58 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:06:58.656 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:58 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:06:58.750 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:06:58.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:58 smithi049 ceph-mon[30080]: 5.6 scrub starts 2024-09-21T14:06:58.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:58 smithi049 ceph-mon[30080]: 5.6 scrub ok 2024-09-21T14:06:58.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:58 smithi049 ceph-mon[30080]: osdmap e467: 8 total, 8 up, 8 in 2024-09-21T14:06:58.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:58 smithi049 ceph-mon[30080]: 5.14 scrub starts 2024-09-21T14:06:58.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:58 smithi049 ceph-mon[30080]: 5.14 scrub ok 2024-09-21T14:06:58.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:58 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:06:58.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:58 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:06:58.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:58 smithi049 ceph-mon[40489]: 5.6 scrub starts 2024-09-21T14:06:58.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:58 smithi049 ceph-mon[40489]: 5.6 scrub ok 2024-09-21T14:06:58.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:58 smithi049 ceph-mon[40489]: osdmap e467: 8 total, 8 up, 8 in 2024-09-21T14:06:58.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:58 smithi049 ceph-mon[40489]: 5.14 scrub starts 2024-09-21T14:06:58.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:58 smithi049 ceph-mon[40489]: 5.14 scrub ok 2024-09-21T14:06:58.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:58 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:06:58.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:58 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:06:58.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:58.787 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:58.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:06:58.804 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T14:06:58.804 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-21T14:06:58.804 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-21T14:06:58.804 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-21T14:06:58.804 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.804 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.15207.0:232030 8.2 8:49ac6d95:::benchmark_data_smithi049_109311_object105511:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T14:06:58.805 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:57:11.653701+0000", 2024-09-21T14:06:58.805 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 587.14848830799997, 2024-09-21T14:06:58.805 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.22905246600000001, 2024-09-21T14:06:58.805 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:06:58.805 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:06:58.805 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:58.805 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:06:58.805 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.15207", 2024-09-21T14:06:58.805 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:06:58.805 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 232030 2024-09-21T14:06:58.805 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.805 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:06:58.805 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.805 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:06:58.805 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.653701+0000", 2024-09-21T14:06:58.806 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:58.806 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.806 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.806 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:06:58.806 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.653701+0000", 2024-09-21T14:06:58.806 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:58.806 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.806 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.806 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:06:58.806 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.653702+0000", 2024-09-21T14:06:58.806 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3600000000000002e-07 2024-09-21T14:06:58.806 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.806 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.807 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:06:58.807 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.653706+0000", 2024-09-21T14:06:58.807 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6490000000000001e-06 2024-09-21T14:06:58.807 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.807 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.807 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:06:58.807 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.653716+0000", 2024-09-21T14:06:58.807 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0509999999999999e-05 2024-09-21T14:06:58.807 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.807 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.807 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:06:58.807 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.653721+0000", 2024-09-21T14:06:58.807 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1279999999999999e-06 2024-09-21T14:06:58.807 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.808 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.808 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:58.808 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.653764+0000", 2024-09-21T14:06:58.808 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2531999999999999e-05 2024-09-21T14:06:58.808 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.808 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.808 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:06:58.808 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.653848+0000", 2024-09-21T14:06:58.808 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4091000000000005e-05 2024-09-21T14:06:58.808 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.808 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.808 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,5", 2024-09-21T14:06:58.808 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.653914+0000", 2024-09-21T14:06:58.808 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5977999999999997e-05 2024-09-21T14:06:58.808 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.809 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.809 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:06:58.809 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.656207+0000", 2024-09-21T14:06:58.809 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0022932690000000001 2024-09-21T14:06:58.809 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.809 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.809 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:58.809 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.659356+0000", 2024-09-21T14:06:58.809 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0031493160000000001 2024-09-21T14:06:58.809 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.809 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.809 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:58.809 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.882645+0000", 2024-09-21T14:06:58.809 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.22328900700000001 2024-09-21T14:06:58.809 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.810 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.810 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:06:58.810 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.882706+0000", 2024-09-21T14:06:58.810 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0279999999999999e-05 2024-09-21T14:06:58.810 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.810 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.810 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:06:58.810 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.882754+0000", 2024-09-21T14:06:58.810 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7870000000000001e-05 2024-09-21T14:06:58.810 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:58.810 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:06:58.810 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:58.810 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.810 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.811 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.15207.0:232036 8.2 8:4507ec93:::benchmark_data_smithi049_109311_object105517:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T14:06:58.811 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:57:11.656238+0000", 2024-09-21T14:06:58.811 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 587.14595137900005, 2024-09-21T14:06:58.811 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.22665437799999999, 2024-09-21T14:06:58.811 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:06:58.811 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:06:58.811 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:58.811 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:06:58.811 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.15207", 2024-09-21T14:06:58.811 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:06:58.811 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 232036 2024-09-21T14:06:58.811 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.811 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:06:58.811 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.812 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:06:58.812 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.656238+0000", 2024-09-21T14:06:58.812 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:58.812 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.812 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.812 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:06:58.812 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.656238+0000", 2024-09-21T14:06:58.812 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:58.812 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.812 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.812 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:06:58.812 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.656239+0000", 2024-09-21T14:06:58.812 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9100000000000003e-07 2024-09-21T14:06:58.812 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.812 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.812 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:06:58.813 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.656243+0000", 2024-09-21T14:06:58.813 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1230000000000002e-06 2024-09-21T14:06:58.813 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.813 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.813 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:06:58.813 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.656252+0000", 2024-09-21T14:06:58.813 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7490000000000004e-06 2024-09-21T14:06:58.813 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.813 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.813 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:06:58.813 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.656259+0000", 2024-09-21T14:06:58.813 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7559999999999999e-06 2024-09-21T14:06:58.813 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.813 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.813 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:58.813 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.656310+0000", 2024-09-21T14:06:58.814 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0203e-05 2024-09-21T14:06:58.814 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.814 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.814 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:06:58.814 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.656440+0000", 2024-09-21T14:06:58.814 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000130376 2024-09-21T14:06:58.814 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.814 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.814 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,5", 2024-09-21T14:06:58.814 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.656550+0000", 2024-09-21T14:06:58.814 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000109812 2024-09-21T14:06:58.814 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.814 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.814 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:06:58.814 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.658548+0000", 2024-09-21T14:06:58.815 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0019982649999999999 2024-09-21T14:06:58.815 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.815 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.815 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:58.815 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.659437+0000", 2024-09-21T14:06:58.815 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000888743 2024-09-21T14:06:58.815 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.815 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.815 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:58.815 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.882792+0000", 2024-09-21T14:06:58.815 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.22335466200000001 2024-09-21T14:06:58.815 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.815 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.815 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:06:58.815 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.882858+0000", 2024-09-21T14:06:58.816 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6512000000000003e-05 2024-09-21T14:06:58.816 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.816 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.816 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:06:58.816 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.882892+0000", 2024-09-21T14:06:58.816 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4385999999999999e-05 2024-09-21T14:06:58.816 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:58.816 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:06:58.816 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:58.816 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.816 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.816 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.15207.0:232041 8.2 8:42fefff6:::benchmark_data_smithi049_109311_object105522:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T14:06:58.816 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:57:11.674073+0000", 2024-09-21T14:06:58.816 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 587.128116611, 2024-09-21T14:06:58.816 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21533840400000001, 2024-09-21T14:06:58.817 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:06:58.817 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:06:58.817 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:58.817 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:06:58.817 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.15207", 2024-09-21T14:06:58.817 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:06:58.818 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 232041 2024-09-21T14:06:58.818 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.818 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:06:58.818 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.818 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:06:58.818 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.674073+0000", 2024-09-21T14:06:58.818 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:58.818 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.818 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.818 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:06:58.818 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.674073+0000", 2024-09-21T14:06:58.818 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:58.818 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.818 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.819 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:06:58.819 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.674074+0000", 2024-09-21T14:06:58.819 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5889999999999999e-06 2024-09-21T14:06:58.819 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.819 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.819 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:06:58.819 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.674080+0000", 2024-09-21T14:06:58.819 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1290000000000004e-06 2024-09-21T14:06:58.819 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.819 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.819 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:06:58.819 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.674102+0000", 2024-09-21T14:06:58.819 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2541e-05 2024-09-21T14:06:58.819 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.819 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.820 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:06:58.820 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.674112+0000", 2024-09-21T14:06:58.820 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.5310000000000007e-06 2024-09-21T14:06:58.820 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.820 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.820 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:58.820 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.674234+0000", 2024-09-21T14:06:58.820 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012247000000000001 2024-09-21T14:06:58.820 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.820 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.820 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:06:58.820 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.674365+0000", 2024-09-21T14:06:58.820 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013083300000000001 2024-09-21T14:06:58.820 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.820 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.820 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,5", 2024-09-21T14:06:58.821 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.674489+0000", 2024-09-21T14:06:58.821 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000124079 2024-09-21T14:06:58.821 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.821 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.821 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:06:58.821 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.680300+0000", 2024-09-21T14:06:58.821 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0058106709999999999 2024-09-21T14:06:58.821 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.821 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.821 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:58.821 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.681669+0000", 2024-09-21T14:06:58.821 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0013690460000000001 2024-09-21T14:06:58.821 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.821 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.821 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:58.821 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.889300+0000", 2024-09-21T14:06:58.822 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20763156999999999 2024-09-21T14:06:58.822 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.822 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.822 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:06:58.822 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.889364+0000", 2024-09-21T14:06:58.822 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3979000000000005e-05 2024-09-21T14:06:58.822 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.822 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.822 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:06:58.822 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.889411+0000", 2024-09-21T14:06:58.822 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6965999999999997e-05 2024-09-21T14:06:58.822 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:58.822 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:06:58.822 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:58.823 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.823 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.823 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.15207.0:232052 8.5 8:aec88863:::benchmark_data_smithi049_109311_object105533:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T14:06:58.823 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T13:57:11.686395+0000", 2024-09-21T14:06:58.823 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 587.11579440800006, 2024-09-21T14:06:58.823 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20789333199999999, 2024-09-21T14:06:58.823 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:06:58.823 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:06:58.823 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:58.823 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:06:58.823 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.15207", 2024-09-21T14:06:58.823 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3169849279", 2024-09-21T14:06:58.823 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 232052 2024-09-21T14:06:58.824 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.824 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:06:58.824 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.824 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:06:58.824 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.686395+0000", 2024-09-21T14:06:58.824 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:58.824 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.824 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.824 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:06:58.824 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.686395+0000", 2024-09-21T14:06:58.824 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:58.824 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.824 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.824 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:06:58.825 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.686396+0000", 2024-09-21T14:06:58.825 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4600000000000004e-07 2024-09-21T14:06:58.825 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.825 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.825 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:06:58.825 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.686399+0000", 2024-09-21T14:06:58.825 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8329999999999998e-06 2024-09-21T14:06:58.825 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.825 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.825 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:06:58.825 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.686404+0000", 2024-09-21T14:06:58.825 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8220000000000003e-06 2024-09-21T14:06:58.825 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.825 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.826 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:06:58.826 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.686408+0000", 2024-09-21T14:06:58.826 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0040000000000001e-06 2024-09-21T14:06:58.826 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.826 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.826 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:58.826 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.688420+0000", 2024-09-21T14:06:58.826 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.002011896 2024-09-21T14:06:58.826 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.826 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.826 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:06:58.826 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.688526+0000", 2024-09-21T14:06:58.826 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000106031 2024-09-21T14:06:58.826 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.827 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.827 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:06:58.827 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.688626+0000", 2024-09-21T14:06:58.827 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000100185 2024-09-21T14:06:58.827 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.827 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.827 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:06:58.827 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.690125+0000", 2024-09-21T14:06:58.827 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0014986820000000001 2024-09-21T14:06:58.827 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.827 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.827 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:58.827 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.690262+0000", 2024-09-21T14:06:58.827 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000136444 2024-09-21T14:06:58.828 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.828 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.828 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:58.828 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.894191+0000", 2024-09-21T14:06:58.828 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.203929104 2024-09-21T14:06:58.828 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.828 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.828 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:06:58.828 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.894251+0000", 2024-09-21T14:06:58.828 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9868999999999999e-05 2024-09-21T14:06:58.828 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.828 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.828 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:06:58.828 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T13:57:11.894288+0000", 2024-09-21T14:06:58.828 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7815999999999998e-05 2024-09-21T14:06:58.829 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:58.829 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:06:58.829 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:58.829 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.829 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.829 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:70936 10.4 10:2fa37e00:::benchmark_data_smithi049_176811_object70935:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-09-21T14:06:58.829 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:03:44.044727+0000", 2024-09-21T14:06:58.829 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 194.75746287000001, 2024-09-21T14:06:58.829 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.29569119599999999, 2024-09-21T14:06:58.829 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:06:58.829 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:06:58.829 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:58.829 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:06:58.829 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:06:58.829 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:06:58.830 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 70936 2024-09-21T14:06:58.830 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.830 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:06:58.830 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.830 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:06:58.830 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:44.044727+0000", 2024-09-21T14:06:58.830 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:58.830 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.830 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.830 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:06:58.830 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:44.044727+0000", 2024-09-21T14:06:58.830 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:58.830 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.830 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.831 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:06:58.831 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:44.044728+0000", 2024-09-21T14:06:58.831 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2890000000000001e-06 2024-09-21T14:06:58.831 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.831 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.831 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:06:58.831 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:44.044788+0000", 2024-09-21T14:06:58.831 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9682999999999998e-05 2024-09-21T14:06:58.831 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.831 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.831 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:06:58.831 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:44.044804+0000", 2024-09-21T14:06:58.831 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6577000000000001e-05 2024-09-21T14:06:58.831 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.831 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.831 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:06:58.832 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:44.044821+0000", 2024-09-21T14:06:58.832 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7037e-05 2024-09-21T14:06:58.832 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.832 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.832 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:58.832 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:44.044885+0000", 2024-09-21T14:06:58.832 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3363999999999998e-05 2024-09-21T14:06:58.832 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.832 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.832 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:06:58.832 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:44.045037+0000", 2024-09-21T14:06:58.832 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015252500000000001 2024-09-21T14:06:58.832 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.832 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.832 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3", 2024-09-21T14:06:58.832 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:44.045212+0000", 2024-09-21T14:06:58.833 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00017447 2024-09-21T14:06:58.833 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.833 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.833 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:06:58.833 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:44.047913+0000", 2024-09-21T14:06:58.833 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0027016539999999999 2024-09-21T14:06:58.833 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.833 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.833 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:58.833 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:44.048904+0000", 2024-09-21T14:06:58.833 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00099064300000000004 2024-09-21T14:06:58.833 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.833 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.833 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:58.833 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:44.340346+0000", 2024-09-21T14:06:58.834 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.29144245499999999 2024-09-21T14:06:58.834 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.834 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.834 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:06:58.834 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:44.340391+0000", 2024-09-21T14:06:58.834 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4906e-05 2024-09-21T14:06:58.834 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.834 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.834 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:06:58.834 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:44.340418+0000", 2024-09-21T14:06:58.834 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.6593000000000001e-05 2024-09-21T14:06:58.834 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:58.834 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:06:58.834 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:58.835 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.835 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.835 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:70937 10.1 10:88bdcf7f:::benchmark_data_smithi049_176811_object70936:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-09-21T14:06:58.835 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:03:44.044844+0000", 2024-09-21T14:06:58.835 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 194.75734565900001, 2024-09-21T14:06:58.835 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.29562298300000001, 2024-09-21T14:06:58.835 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:06:58.835 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:06:58.835 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:58.835 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:06:58.835 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:06:58.835 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:06:58.835 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 70937 2024-09-21T14:06:58.835 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.835 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:06:58.836 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.836 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:06:58.836 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:44.044844+0000", 2024-09-21T14:06:58.836 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:58.836 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.836 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.836 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:06:58.836 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:44.044844+0000", 2024-09-21T14:06:58.836 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:58.836 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.836 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.836 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:06:58.836 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:44.044845+0000", 2024-09-21T14:06:58.836 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0500000000000002e-07 2024-09-21T14:06:58.836 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.837 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.837 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:06:58.837 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:44.044878+0000", 2024-09-21T14:06:58.837 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3105000000000002e-05 2024-09-21T14:06:58.837 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.837 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.837 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:06:58.837 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:44.044886+0000", 2024-09-21T14:06:58.837 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.6789999999999994e-06 2024-09-21T14:06:58.837 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.837 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.837 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:06:58.837 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:44.044893+0000", 2024-09-21T14:06:58.837 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6980000000000001e-06 2024-09-21T14:06:58.837 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.838 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.838 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:58.838 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:44.044954+0000", 2024-09-21T14:06:58.838 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1221000000000002e-05 2024-09-21T14:06:58.838 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.838 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.838 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:06:58.838 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:44.045080+0000", 2024-09-21T14:06:58.838 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000126045 2024-09-21T14:06:58.838 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.838 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.838 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:06:58.838 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:44.045232+0000", 2024-09-21T14:06:58.838 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000151941 2024-09-21T14:06:58.838 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.839 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.839 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:06:58.839 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:44.047861+0000", 2024-09-21T14:06:58.839 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.002628773 2024-09-21T14:06:58.839 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.839 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.839 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:58.839 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:44.047903+0000", 2024-09-21T14:06:58.839 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2221000000000001e-05 2024-09-21T14:06:58.839 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.839 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.839 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:58.839 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:44.340370+0000", 2024-09-21T14:06:58.839 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.29246707700000002 2024-09-21T14:06:58.839 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.840 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.840 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:06:58.840 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:44.340425+0000", 2024-09-21T14:06:58.840 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4187000000000001e-05 2024-09-21T14:06:58.840 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.840 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.840 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:06:58.840 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:03:44.340467+0000", 2024-09-21T14:06:58.840 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2131e-05 2024-09-21T14:06:58.840 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:58.840 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:06:58.840 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:58.840 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.840 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.841 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:112229 10.4 10:2381a1a9:::benchmark_data_smithi049_176811_object112228:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-21T14:06:58.841 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:04:20.239436+0000", 2024-09-21T14:06:58.841 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 158.562753813, 2024-09-21T14:06:58.841 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.048690772, 2024-09-21T14:06:58.841 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:06:58.841 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:06:58.841 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:58.841 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:06:58.841 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:06:58.841 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:06:58.841 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 112229 2024-09-21T14:06:58.841 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.841 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:06:58.841 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.841 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:06:58.841 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.239436+0000", 2024-09-21T14:06:58.842 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:58.842 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.842 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.842 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:06:58.842 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.239436+0000", 2024-09-21T14:06:58.842 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:58.842 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.842 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.842 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:06:58.842 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.239437+0000", 2024-09-21T14:06:58.842 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.668e-06 2024-09-21T14:06:58.842 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.842 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.842 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:06:58.842 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.239558+0000", 2024-09-21T14:06:58.843 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012067599999999999 2024-09-21T14:06:58.843 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.843 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.843 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:06:58.843 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.239570+0000", 2024-09-21T14:06:58.843 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1581e-05 2024-09-21T14:06:58.843 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.843 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.843 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:06:58.843 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.239585+0000", 2024-09-21T14:06:58.843 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5241e-05 2024-09-21T14:06:58.843 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.843 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.843 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:58.844 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.239656+0000", 2024-09-21T14:06:58.844 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1248000000000006e-05 2024-09-21T14:06:58.844 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.844 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.844 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:06:58.844 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.239666+0000", 2024-09-21T14:06:58.844 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0068000000000001e-05 2024-09-21T14:06:58.844 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.844 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.844 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:58.844 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.194584+0000", 2024-09-21T14:06:58.844 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.95491765200000001 2024-09-21T14:06:58.844 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.844 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.844 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:06:58.844 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.194597+0000", 2024-09-21T14:06:58.845 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3377000000000001e-05 2024-09-21T14:06:58.845 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.845 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.845 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:58.845 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.269588+0000", 2024-09-21T14:06:58.845 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0749913129999999 2024-09-21T14:06:58.845 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.845 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.845 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:06:58.845 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.269689+0000", 2024-09-21T14:06:58.845 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000101006 2024-09-21T14:06:58.845 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.845 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.845 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1,3", 2024-09-21T14:06:58.845 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.269788+0000", 2024-09-21T14:06:58.846 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.8152000000000005e-05 2024-09-21T14:06:58.846 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.846 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.846 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:06:58.846 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.276727+0000", 2024-09-21T14:06:58.846 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0069397479999999999 2024-09-21T14:06:58.846 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.846 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.846 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:58.846 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.278656+0000", 2024-09-21T14:06:58.846 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001929123 2024-09-21T14:06:58.846 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.846 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.846 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:58.847 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.279569+0000", 2024-09-21T14:06:58.847 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00091213100000000005 2024-09-21T14:06:58.847 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.847 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.847 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:58.847 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.288003+0000", 2024-09-21T14:06:58.847 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0084348429999999992 2024-09-21T14:06:58.847 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.847 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.847 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:06:58.847 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.288045+0000", 2024-09-21T14:06:58.847 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1553e-05 2024-09-21T14:06:58.847 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.847 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.847 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:06:58.848 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.288126+0000", 2024-09-21T14:06:58.848 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1391999999999996e-05 2024-09-21T14:06:58.848 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:58.848 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:06:58.848 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:58.848 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.848 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.848 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:112231 10.4 10:2e3f0834:::benchmark_data_smithi049_176811_object112230:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-21T14:06:58.848 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:04:20.239610+0000", 2024-09-21T14:06:58.848 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 158.562579484, 2024-09-21T14:06:58.848 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0486707210000001, 2024-09-21T14:06:58.848 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:06:58.848 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:06:58.848 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:58.849 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:06:58.849 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:06:58.849 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:06:58.849 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 112231 2024-09-21T14:06:58.849 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.849 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:06:58.849 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.849 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:06:58.849 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.239610+0000", 2024-09-21T14:06:58.849 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:58.849 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.849 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.849 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:06:58.849 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.239610+0000", 2024-09-21T14:06:58.849 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:58.850 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.850 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.850 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:06:58.850 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.239611+0000", 2024-09-21T14:06:58.850 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3500000000000005e-07 2024-09-21T14:06:58.850 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.850 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.850 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:06:58.850 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.239723+0000", 2024-09-21T14:06:58.850 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000112341 2024-09-21T14:06:58.850 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.850 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.850 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:06:58.850 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.239733+0000", 2024-09-21T14:06:58.850 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0118999999999999e-05 2024-09-21T14:06:58.851 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.851 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.851 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:06:58.851 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.239740+0000", 2024-09-21T14:06:58.851 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6069999999999996e-06 2024-09-21T14:06:58.851 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.851 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.851 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:58.851 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.239792+0000", 2024-09-21T14:06:58.851 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2629e-05 2024-09-21T14:06:58.851 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.851 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.851 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:06:58.851 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.239801+0000", 2024-09-21T14:06:58.852 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.6500000000000002e-06 2024-09-21T14:06:58.852 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.852 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.852 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:58.852 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.194627+0000", 2024-09-21T14:06:58.852 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.95482541300000001 2024-09-21T14:06:58.852 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.852 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.852 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:06:58.852 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.194635+0000", 2024-09-21T14:06:58.852 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9899999999999997e-06 2024-09-21T14:06:58.852 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.852 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.852 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:58.852 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.271698+0000", 2024-09-21T14:06:58.853 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.07706378 2024-09-21T14:06:58.853 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.853 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.853 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:06:58.853 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.271785+0000", 2024-09-21T14:06:58.853 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7182000000000007e-05 2024-09-21T14:06:58.853 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.853 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.853 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1,3", 2024-09-21T14:06:58.853 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.271892+0000", 2024-09-21T14:06:58.853 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000106346 2024-09-21T14:06:58.853 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.853 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.853 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:06:58.853 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.276736+0000", 2024-09-21T14:06:58.854 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0048441659999999996 2024-09-21T14:06:58.854 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.854 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.854 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:58.854 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.279711+0000", 2024-09-21T14:06:58.854 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0029745090000000002 2024-09-21T14:06:58.854 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.854 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.854 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:58.854 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.288164+0000", 2024-09-21T14:06:58.854 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0084534599999999994 2024-09-21T14:06:58.854 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.854 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.855 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:58.855 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.288197+0000", 2024-09-21T14:06:58.855 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2592000000000002e-05 2024-09-21T14:06:58.855 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.855 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.855 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:06:58.855 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.288228+0000", 2024-09-21T14:06:58.855 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1581999999999997e-05 2024-09-21T14:06:58.855 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.855 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.855 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:06:58.855 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.288281+0000", 2024-09-21T14:06:58.855 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2519999999999999e-05 2024-09-21T14:06:58.855 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:58.855 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:06:58.856 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:58.856 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.856 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.856 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:112244 10.4 10:228aeb9f:::benchmark_data_smithi049_176811_object112243:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-21T14:06:58.856 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:04:20.250628+0000", 2024-09-21T14:06:58.856 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 158.55156144099999, 2024-09-21T14:06:58.856 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0378127940000001, 2024-09-21T14:06:58.857 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:06:58.857 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:06:58.857 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:58.857 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:06:58.857 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:06:58.857 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:06:58.857 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 112244 2024-09-21T14:06:58.857 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.857 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:06:58.857 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.857 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:06:58.857 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.250628+0000", 2024-09-21T14:06:58.857 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:58.857 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.858 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.858 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:06:58.858 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.250628+0000", 2024-09-21T14:06:58.858 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:58.858 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.858 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.858 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:06:58.858 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.250630+0000", 2024-09-21T14:06:58.858 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.841e-06 2024-09-21T14:06:58.858 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.858 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.858 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:06:58.858 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.250735+0000", 2024-09-21T14:06:58.859 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000105393 2024-09-21T14:06:58.859 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.859 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.859 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:06:58.859 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.250751+0000", 2024-09-21T14:06:58.859 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5763000000000001e-05 2024-09-21T14:06:58.859 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.859 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.859 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:06:58.859 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.250759+0000", 2024-09-21T14:06:58.859 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2400000000000007e-06 2024-09-21T14:06:58.859 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.859 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.859 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:58.859 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.250829+0000", 2024-09-21T14:06:58.860 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9852999999999997e-05 2024-09-21T14:06:58.860 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.860 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.860 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:06:58.860 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.250846+0000", 2024-09-21T14:06:58.860 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7099e-05 2024-09-21T14:06:58.860 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.860 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.860 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:58.860 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.194667+0000", 2024-09-21T14:06:58.860 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.94382083999999999 2024-09-21T14:06:58.860 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.860 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.860 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:06:58.860 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.194675+0000", 2024-09-21T14:06:58.861 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7230000000000004e-06 2024-09-21T14:06:58.861 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.861 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.861 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:58.861 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.273226+0000", 2024-09-21T14:06:58.861 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0785516289999999 2024-09-21T14:06:58.861 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.861 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.861 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:06:58.861 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.273299+0000", 2024-09-21T14:06:58.861 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2143000000000004e-05 2024-09-21T14:06:58.861 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.861 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.861 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1,3", 2024-09-21T14:06:58.862 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.273396+0000", 2024-09-21T14:06:58.862 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7665999999999996e-05 2024-09-21T14:06:58.862 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.862 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.862 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:06:58.862 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.276742+0000", 2024-09-21T14:06:58.862 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003345621 2024-09-21T14:06:58.862 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.862 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.862 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:58.862 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.282756+0000", 2024-09-21T14:06:58.862 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0060141680000000003 2024-09-21T14:06:58.862 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.862 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.863 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:58.863 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.288320+0000", 2024-09-21T14:06:58.863 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0055642080000000002 2024-09-21T14:06:58.863 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.863 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.863 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:58.863 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.288351+0000", 2024-09-21T14:06:58.863 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0511e-05 2024-09-21T14:06:58.863 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.863 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.863 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:06:58.863 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.288381+0000", 2024-09-21T14:06:58.863 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0270999999999999e-05 2024-09-21T14:06:58.863 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.863 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.863 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:06:58.864 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.288441+0000", 2024-09-21T14:06:58.864 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9824999999999997e-05 2024-09-21T14:06:58.864 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:58.864 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:06:58.864 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:58.864 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.864 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.864 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:112248 10.4 10:2865fdef:::benchmark_data_smithi049_176811_object112247:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-21T14:06:58.864 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:04:20.253079+0000", 2024-09-21T14:06:58.864 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 158.54911038399999, 2024-09-21T14:06:58.864 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0355127999999998, 2024-09-21T14:06:58.864 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:06:58.864 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:06:58.864 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:58.864 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:06:58.864 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:06:58.865 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:06:58.865 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 112248 2024-09-21T14:06:58.865 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.865 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:06:58.865 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.865 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:06:58.865 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.253079+0000", 2024-09-21T14:06:58.865 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:58.865 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.865 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.865 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:06:58.865 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.253079+0000", 2024-09-21T14:06:58.865 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:58.865 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.865 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.865 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:06:58.866 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.253081+0000", 2024-09-21T14:06:58.866 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7799999999999999e-06 2024-09-21T14:06:58.866 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.866 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.866 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:06:58.866 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.253199+0000", 2024-09-21T14:06:58.866 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011827 2024-09-21T14:06:58.866 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.866 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.866 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:06:58.866 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.253215+0000", 2024-09-21T14:06:58.866 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5525000000000001e-05 2024-09-21T14:06:58.866 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.866 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.866 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:06:58.867 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.253223+0000", 2024-09-21T14:06:58.867 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9969999999999998e-06 2024-09-21T14:06:58.867 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.867 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.867 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:58.867 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.253298+0000", 2024-09-21T14:06:58.867 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5427999999999997e-05 2024-09-21T14:06:58.867 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.867 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.867 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:06:58.867 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.253307+0000", 2024-09-21T14:06:58.867 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0229999999999998e-06 2024-09-21T14:06:58.867 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.867 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.868 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:58.868 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.194706+0000", 2024-09-21T14:06:58.868 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.94139865 2024-09-21T14:06:58.868 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.868 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.868 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:06:58.868 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.194713+0000", 2024-09-21T14:06:58.868 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5129999999999999e-06 2024-09-21T14:06:58.868 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.868 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.868 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:58.868 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.274436+0000", 2024-09-21T14:06:58.868 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0797226369999999 2024-09-21T14:06:58.868 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.869 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.869 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:06:58.869 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.274522+0000", 2024-09-21T14:06:58.869 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5945000000000005e-05 2024-09-21T14:06:58.869 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.869 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.869 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1,3", 2024-09-21T14:06:58.869 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.274669+0000", 2024-09-21T14:06:58.869 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000147245 2024-09-21T14:06:58.869 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.869 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.869 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:06:58.869 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.278678+0000", 2024-09-21T14:06:58.869 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0040084509999999997 2024-09-21T14:06:58.870 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.870 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.870 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:58.870 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.282792+0000", 2024-09-21T14:06:58.870 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0041142560000000002 2024-09-21T14:06:58.870 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.870 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.870 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:58.870 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.288475+0000", 2024-09-21T14:06:58.870 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0056830379999999996 2024-09-21T14:06:58.870 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.870 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.870 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:58.871 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.288506+0000", 2024-09-21T14:06:58.871 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1068000000000001e-05 2024-09-21T14:06:58.871 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.871 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.871 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:06:58.871 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.288535+0000", 2024-09-21T14:06:58.871 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9464000000000001e-05 2024-09-21T14:06:58.871 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.871 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.871 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:06:58.871 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.288592+0000", 2024-09-21T14:06:58.871 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6509999999999999e-05 2024-09-21T14:06:58.871 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:58.872 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:06:58.872 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:58.872 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.872 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.872 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:112252 10.4 10:25607d1e:::benchmark_data_smithi049_176811_object112251:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-21T14:06:58.872 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:04:20.256669+0000", 2024-09-21T14:06:58.872 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 158.545519998, 2024-09-21T14:06:58.872 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.032077707, 2024-09-21T14:06:58.872 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:06:58.872 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:06:58.872 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:58.872 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:06:58.872 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:06:58.872 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:06:58.872 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 112252 2024-09-21T14:06:58.873 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.873 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:06:58.873 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.873 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:06:58.873 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.256669+0000", 2024-09-21T14:06:58.873 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:58.873 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.873 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.873 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:06:58.873 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.256669+0000", 2024-09-21T14:06:58.873 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:58.873 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.873 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.873 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:06:58.874 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.256671+0000", 2024-09-21T14:06:58.874 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4920000000000001e-06 2024-09-21T14:06:58.874 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.874 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.874 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:06:58.874 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.256771+0000", 2024-09-21T14:06:58.874 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010003 2024-09-21T14:06:58.874 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.874 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.874 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:06:58.874 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.256784+0000", 2024-09-21T14:06:58.874 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3375e-05 2024-09-21T14:06:58.874 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.875 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.875 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:06:58.875 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.256790+0000", 2024-09-21T14:06:58.875 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.049e-06 2024-09-21T14:06:58.875 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.875 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.875 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:58.875 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.256848+0000", 2024-09-21T14:06:58.875 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7862000000000003e-05 2024-09-21T14:06:58.875 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.875 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.875 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:06:58.875 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.256856+0000", 2024-09-21T14:06:58.875 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0129999999999993e-06 2024-09-21T14:06:58.876 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.876 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.876 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:58.876 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.194751+0000", 2024-09-21T14:06:58.876 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.93789514600000001 2024-09-21T14:06:58.876 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.876 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.876 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:06:58.876 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.194852+0000", 2024-09-21T14:06:58.876 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000100234 2024-09-21T14:06:58.876 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.876 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.876 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:58.876 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.276785+0000", 2024-09-21T14:06:58.876 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0819329390000001 2024-09-21T14:06:58.877 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.877 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.877 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:06:58.877 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.276924+0000", 2024-09-21T14:06:58.877 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013901699999999999 2024-09-21T14:06:58.877 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.877 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.877 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1,3", 2024-09-21T14:06:58.877 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.277028+0000", 2024-09-21T14:06:58.877 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010436 2024-09-21T14:06:58.877 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.877 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.877 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:06:58.877 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.278898+0000", 2024-09-21T14:06:58.877 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0018704069999999999 2024-09-21T14:06:58.878 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.878 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.878 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:58.878 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.282825+0000", 2024-09-21T14:06:58.878 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0039263520000000001 2024-09-21T14:06:58.878 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.878 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.878 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:58.878 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.288628+0000", 2024-09-21T14:06:58.878 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0058029259999999999 2024-09-21T14:06:58.878 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.878 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.878 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:58.878 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.288661+0000", 2024-09-21T14:06:58.879 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3759e-05 2024-09-21T14:06:58.879 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.879 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.879 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:06:58.879 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.288693+0000", 2024-09-21T14:06:58.879 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1838000000000002e-05 2024-09-21T14:06:58.879 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.879 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.879 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:06:58.879 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.288747+0000", 2024-09-21T14:06:58.879 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3908e-05 2024-09-21T14:06:58.879 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:58.879 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:06:58.880 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:58.880 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.880 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.880 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:131791 10.4 10:2cbd491a:::benchmark_data_smithi049_176811_object4214:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-09-21T14:06:58.880 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:04:39.751761+0000", 2024-09-21T14:06:58.880 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 139.05042802400001, 2024-09-21T14:06:58.880 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.27974941399999997, 2024-09-21T14:06:58.880 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:06:58.880 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:06:58.880 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:58.880 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:06:58.880 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:06:58.880 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:06:58.880 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 131791 2024-09-21T14:06:58.880 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.881 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:06:58.881 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.881 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:06:58.881 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:39.751761+0000", 2024-09-21T14:06:58.881 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:58.881 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.881 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.881 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:06:58.881 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:39.751761+0000", 2024-09-21T14:06:58.881 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:58.881 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.881 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.881 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:06:58.881 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:39.751763+0000", 2024-09-21T14:06:58.882 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3289999999999999e-06 2024-09-21T14:06:58.882 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.882 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.882 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:06:58.882 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:39.751767+0000", 2024-09-21T14:06:58.882 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2560000000000004e-06 2024-09-21T14:06:58.882 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.882 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.882 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:06:58.882 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:39.751776+0000", 2024-09-21T14:06:58.882 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.9719999999999998e-06 2024-09-21T14:06:58.882 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.882 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.882 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:06:58.883 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:39.751782+0000", 2024-09-21T14:06:58.883 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9630000000000003e-06 2024-09-21T14:06:58.883 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.883 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.883 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:58.883 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:39.752100+0000", 2024-09-21T14:06:58.883 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00031850299999999998 2024-09-21T14:06:58.883 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.883 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.883 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:06:58.883 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:39.752227+0000", 2024-09-21T14:06:58.883 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012691699999999999 2024-09-21T14:06:58.883 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.883 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.883 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1,3", 2024-09-21T14:06:58.884 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:39.752351+0000", 2024-09-21T14:06:58.884 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000123421 2024-09-21T14:06:58.884 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.884 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.884 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:58.884 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:39.763506+0000", 2024-09-21T14:06:58.884 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.011155234999999999 2024-09-21T14:06:58.884 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.884 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.884 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:58.884 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:39.763853+0000", 2024-09-21T14:06:58.884 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000346967 2024-09-21T14:06:58.884 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.884 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.884 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:06:58.885 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:39.763867+0000", 2024-09-21T14:06:58.885 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4469e-05 2024-09-21T14:06:58.885 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.885 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.885 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:58.885 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:40.031335+0000", 2024-09-21T14:06:58.885 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.26746768799999998 2024-09-21T14:06:58.885 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.885 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.885 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:06:58.885 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:40.031401+0000", 2024-09-21T14:06:58.885 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6360999999999997e-05 2024-09-21T14:06:58.885 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.885 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.885 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:06:58.886 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:40.031511+0000", 2024-09-21T14:06:58.886 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010933300000000001 2024-09-21T14:06:58.886 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:58.886 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:06:58.886 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:58.886 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.886 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.886 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:131797 10.4 10:20ab94a6:::benchmark_data_smithi049_176811_object4220:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-09-21T14:06:58.886 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:04:39.755526+0000", 2024-09-21T14:06:58.886 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 139.04666355500001, 2024-09-21T14:06:58.886 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.27810648500000001, 2024-09-21T14:06:58.886 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:06:58.886 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:06:58.887 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:58.887 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:06:58.887 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:06:58.887 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:06:58.887 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 131797 2024-09-21T14:06:58.887 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.887 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:06:58.887 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.887 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:06:58.887 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:39.755526+0000", 2024-09-21T14:06:58.887 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:58.887 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.887 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.887 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:06:58.887 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:39.755526+0000", 2024-09-21T14:06:58.888 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:58.888 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.888 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.888 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:06:58.888 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:39.755527+0000", 2024-09-21T14:06:58.888 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.5300000000000002e-07 2024-09-21T14:06:58.888 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.888 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.888 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:06:58.888 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:39.755530+0000", 2024-09-21T14:06:58.888 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.597e-06 2024-09-21T14:06:58.888 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.888 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.888 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:06:58.888 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:39.755541+0000", 2024-09-21T14:06:58.889 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0478e-05 2024-09-21T14:06:58.889 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.889 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.889 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:06:58.889 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:39.755548+0000", 2024-09-21T14:06:58.889 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.362e-06 2024-09-21T14:06:58.889 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.889 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.889 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:58.889 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:39.756417+0000", 2024-09-21T14:06:58.889 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00086912199999999995 2024-09-21T14:06:58.889 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.889 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.889 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:06:58.890 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:39.756525+0000", 2024-09-21T14:06:58.890 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000107797 2024-09-21T14:06:58.890 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.890 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.890 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1,3", 2024-09-21T14:06:58.890 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:39.756648+0000", 2024-09-21T14:06:58.890 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012269200000000001 2024-09-21T14:06:58.890 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.890 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.890 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:58.890 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:39.763544+0000", 2024-09-21T14:06:58.890 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0068958070000000003 2024-09-21T14:06:58.890 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.890 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.891 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:58.891 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:39.765924+0000", 2024-09-21T14:06:58.891 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0023804809999999998 2024-09-21T14:06:58.891 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.891 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.891 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:06:58.891 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:39.769118+0000", 2024-09-21T14:06:58.891 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0031936769999999998 2024-09-21T14:06:58.891 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.891 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.891 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:58.891 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:40.033457+0000", 2024-09-21T14:06:58.891 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.26433934799999997 2024-09-21T14:06:58.891 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.892 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.892 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:06:58.892 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:40.033513+0000", 2024-09-21T14:06:58.892 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6097999999999997e-05 2024-09-21T14:06:58.892 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.892 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.892 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:06:58.892 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:40.033632+0000", 2024-09-21T14:06:58.892 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000119073 2024-09-21T14:06:58.892 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:58.892 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:06:58.892 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:58.892 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.892 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.892 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:131813 10.1 10:8cc73110:::benchmark_data_smithi049_176811_object4236:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-09-21T14:06:58.893 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:04:39.775543+0000", 2024-09-21T14:06:58.893 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 139.02664687999999, 2024-09-21T14:06:58.893 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.27433647999999999, 2024-09-21T14:06:58.893 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:06:58.893 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:06:58.893 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:58.893 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:06:58.893 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:06:58.893 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:06:58.893 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 131813 2024-09-21T14:06:58.893 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.893 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:06:58.893 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.893 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:06:58.893 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:39.775543+0000", 2024-09-21T14:06:58.893 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:58.894 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.894 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.894 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:06:58.894 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:39.775543+0000", 2024-09-21T14:06:58.894 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:58.894 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.894 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.894 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:06:58.894 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:39.775544+0000", 2024-09-21T14:06:58.894 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.062e-06 2024-09-21T14:06:58.894 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.894 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.894 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:06:58.894 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:39.775547+0000", 2024-09-21T14:06:58.894 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7069999999999998e-06 2024-09-21T14:06:58.895 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.895 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.895 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:06:58.895 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:39.775556+0000", 2024-09-21T14:06:58.895 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4449999999999996e-06 2024-09-21T14:06:58.895 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.895 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.895 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:06:58.895 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:39.775560+0000", 2024-09-21T14:06:58.895 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3279999999999999e-06 2024-09-21T14:06:58.895 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.895 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.895 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:58.895 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:39.775596+0000", 2024-09-21T14:06:58.895 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6260999999999997e-05 2024-09-21T14:06:58.895 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.895 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.896 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:06:58.896 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:39.775683+0000", 2024-09-21T14:06:58.896 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.6778000000000002e-05 2024-09-21T14:06:58.896 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.896 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:06:58.897 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.897 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:06:58.897 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:39.775783+0000", 2024-09-21T14:06:58.897 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010007500000000001 2024-09-21T14:06:58.897 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.897 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.897 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:06:58.897 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:39.778105+0000", 2024-09-21T14:06:58.897 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0023222080000000001 2024-09-21T14:06:58.897 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.898 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.898 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:58.898 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:39.778346+0000", 2024-09-21T14:06:58.898 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000240246 2024-09-21T14:06:58.898 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.898 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.898 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:58.898 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:40.049794+0000", 2024-09-21T14:06:58.898 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.27144866200000001 2024-09-21T14:06:58.898 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.898 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.898 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:06:58.898 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:40.049842+0000", 2024-09-21T14:06:58.899 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7206000000000002e-05 2024-09-21T14:06:58.899 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.899 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.899 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:06:58.899 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:40.049879+0000", 2024-09-21T14:06:58.899 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7502000000000001e-05 2024-09-21T14:06:58.899 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:58.899 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:06:58.899 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:58.899 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.899 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.899 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:160591 10.4 10:242883d3:::benchmark_data_smithi049_176811_object33014:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-09-21T14:06:58.899 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:05:07.645387+0000", 2024-09-21T14:06:58.899 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 111.156802685, 2024-09-21T14:06:58.900 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.23109671100000001, 2024-09-21T14:06:58.900 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:06:58.900 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:06:58.900 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:58.900 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:06:58.900 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:06:58.900 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:06:58.900 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 160591 2024-09-21T14:06:58.900 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.900 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:06:58.900 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.900 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:06:58.900 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:07.645387+0000", 2024-09-21T14:06:58.900 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:58.900 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.901 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.901 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:06:58.901 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:07.645387+0000", 2024-09-21T14:06:58.901 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:58.901 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.901 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.901 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:06:58.901 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:07.645388+0000", 2024-09-21T14:06:58.901 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4389999999999999e-06 2024-09-21T14:06:58.901 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.901 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.901 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:06:58.901 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:07.645392+0000", 2024-09-21T14:06:58.901 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.692e-06 2024-09-21T14:06:58.901 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.902 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.902 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:06:58.902 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:07.645401+0000", 2024-09-21T14:06:58.902 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8040000000000007e-06 2024-09-21T14:06:58.902 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.902 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.902 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:06:58.902 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:07.645407+0000", 2024-09-21T14:06:58.902 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9859999999999999e-06 2024-09-21T14:06:58.902 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.902 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.902 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:58.902 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:07.646132+0000", 2024-09-21T14:06:58.903 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00072542400000000001 2024-09-21T14:06:58.903 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.903 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.903 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:06:58.903 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:07.646261+0000", 2024-09-21T14:06:58.903 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000128703 2024-09-21T14:06:58.903 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.903 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.903 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1,3", 2024-09-21T14:06:58.903 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:07.646410+0000", 2024-09-21T14:06:58.903 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014885500000000001 2024-09-21T14:06:58.903 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.903 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.903 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:58.904 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:07.650716+0000", 2024-09-21T14:06:58.904 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0043061089999999998 2024-09-21T14:06:58.904 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.904 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.904 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:58.904 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:07.650752+0000", 2024-09-21T14:06:58.904 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6241000000000001e-05 2024-09-21T14:06:58.904 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.904 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.904 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:06:58.904 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:07.653393+0000", 2024-09-21T14:06:58.904 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0026409670000000001 2024-09-21T14:06:58.904 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.904 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.905 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:58.905 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:07.876302+0000", 2024-09-21T14:06:58.905 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.222908789 2024-09-21T14:06:58.905 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.905 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.905 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:06:58.905 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:07.876363+0000", 2024-09-21T14:06:58.905 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0949000000000002e-05 2024-09-21T14:06:58.905 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.905 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.905 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:06:58.905 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:07.876483+0000", 2024-09-21T14:06:58.905 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000120753 2024-09-21T14:06:58.905 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:58.905 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:06:58.906 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:58.906 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.906 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.906 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:160608 10.4 10:2c9d7c5e:::benchmark_data_smithi049_176811_object33031:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e420)", 2024-09-21T14:06:58.906 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:05:07.667704+0000", 2024-09-21T14:06:58.906 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 111.134485571, 2024-09-21T14:06:58.906 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.209286309, 2024-09-21T14:06:58.906 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:06:58.906 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:06:58.906 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:58.906 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:06:58.906 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:06:58.906 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:06:58.907 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 160608 2024-09-21T14:06:58.907 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.907 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:06:58.907 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.907 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:06:58.907 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:07.667704+0000", 2024-09-21T14:06:58.907 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:58.907 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.907 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.907 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:06:58.907 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:07.667704+0000", 2024-09-21T14:06:58.907 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:58.907 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.907 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.908 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:06:58.908 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:07.667705+0000", 2024-09-21T14:06:58.908 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.311e-06 2024-09-21T14:06:58.908 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.908 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.908 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:06:58.908 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:07.667710+0000", 2024-09-21T14:06:58.908 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7859999999999996e-06 2024-09-21T14:06:58.908 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.908 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.908 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:06:58.908 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:07.667725+0000", 2024-09-21T14:06:58.908 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4955e-05 2024-09-21T14:06:58.908 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.908 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.909 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:06:58.909 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:07.667733+0000", 2024-09-21T14:06:58.909 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3590000000000008e-06 2024-09-21T14:06:58.909 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.909 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.909 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:58.909 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:07.668305+0000", 2024-09-21T14:06:58.909 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00057156600000000002 2024-09-21T14:06:58.909 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.909 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.909 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:06:58.909 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:07.668441+0000", 2024-09-21T14:06:58.909 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013566500000000001 2024-09-21T14:06:58.909 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.910 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.910 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1,3", 2024-09-21T14:06:58.910 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:07.668587+0000", 2024-09-21T14:06:58.910 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014675300000000001 2024-09-21T14:06:58.910 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.910 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.910 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:58.910 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:07.671887+0000", 2024-09-21T14:06:58.910 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0032992830000000001 2024-09-21T14:06:58.910 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.910 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.910 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:58.910 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:07.671922+0000", 2024-09-21T14:06:58.910 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5710999999999997e-05 2024-09-21T14:06:58.911 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.911 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.911 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:06:58.911 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:07.673252+0000", 2024-09-21T14:06:58.911 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0013298240000000001 2024-09-21T14:06:58.911 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.911 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.911 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:58.911 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:07.876799+0000", 2024-09-21T14:06:58.911 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20354714700000001 2024-09-21T14:06:58.911 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.911 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.911 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:06:58.911 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:07.876866+0000", 2024-09-21T14:06:58.911 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7207999999999998e-05 2024-09-21T14:06:58.912 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.912 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.912 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:06:58.912 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:07.876990+0000", 2024-09-21T14:06:58.912 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012374099999999999 2024-09-21T14:06:58.912 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:58.912 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:06:58.912 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:58.912 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.912 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.912 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:208868 10.4 10:2074335c:::benchmark_data_smithi049_176811_object81291:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e441)", 2024-09-21T14:06:58.912 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:05:54.050594+0000", 2024-09-21T14:06:58.912 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 64.751595542000004, 2024-09-21T14:06:58.912 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.31457637199999999, 2024-09-21T14:06:58.912 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:06:58.912 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:06:58.913 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:58.913 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:06:58.913 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:06:58.913 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:06:58.913 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 208868 2024-09-21T14:06:58.913 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.913 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:06:58.913 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.913 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:06:58.913 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:54.050594+0000", 2024-09-21T14:06:58.913 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:58.913 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.913 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.913 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:06:58.913 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:54.050594+0000", 2024-09-21T14:06:58.913 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:58.914 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.914 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.914 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:06:58.914 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:54.050596+0000", 2024-09-21T14:06:58.914 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.635e-06 2024-09-21T14:06:58.914 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.914 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.914 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:06:58.914 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:54.050601+0000", 2024-09-21T14:06:58.914 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.626e-06 2024-09-21T14:06:58.914 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.914 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.914 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:06:58.914 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:54.050613+0000", 2024-09-21T14:06:58.914 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1817000000000001e-05 2024-09-21T14:06:58.915 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.915 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.915 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:06:58.915 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:54.050620+0000", 2024-09-21T14:06:58.915 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1539999999999996e-06 2024-09-21T14:06:58.915 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.915 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.915 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:58.915 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:54.050699+0000", 2024-09-21T14:06:58.915 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8957999999999999e-05 2024-09-21T14:06:58.915 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.915 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.915 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:06:58.915 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:54.050803+0000", 2024-09-21T14:06:58.915 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000103757 2024-09-21T14:06:58.915 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.916 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.916 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3", 2024-09-21T14:06:58.916 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:54.050926+0000", 2024-09-21T14:06:58.916 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000123051 2024-09-21T14:06:58.916 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.916 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.916 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:06:58.916 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:54.054282+0000", 2024-09-21T14:06:58.916 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003355902 2024-09-21T14:06:58.916 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.916 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.916 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:58.916 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:54.059483+0000", 2024-09-21T14:06:58.916 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0052016040000000003 2024-09-21T14:06:58.916 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.916 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.917 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:58.917 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:54.365059+0000", 2024-09-21T14:06:58.917 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.30557575100000001 2024-09-21T14:06:58.917 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.917 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.917 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:06:58.917 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:54.365113+0000", 2024-09-21T14:06:58.917 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3854000000000003e-05 2024-09-21T14:06:58.917 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.917 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.917 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:06:58.917 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:54.365170+0000", 2024-09-21T14:06:58.917 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7262999999999997e-05 2024-09-21T14:06:58.917 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:58.917 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:06:58.917 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:58.918 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.918 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.918 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:208878 10.4 10:22db5967:::benchmark_data_smithi049_176811_object81301:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e441)", 2024-09-21T14:06:58.918 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:05:54.056337+0000", 2024-09-21T14:06:58.918 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 64.745852084000006, 2024-09-21T14:06:58.918 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.31213438999999998, 2024-09-21T14:06:58.918 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:06:58.918 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:06:58.918 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:58.918 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:06:58.918 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:06:58.918 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:06:58.918 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 208878 2024-09-21T14:06:58.918 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.918 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:06:58.919 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.919 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:06:58.919 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:54.056337+0000", 2024-09-21T14:06:58.919 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:58.919 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.919 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.919 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:06:58.919 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:54.056337+0000", 2024-09-21T14:06:58.919 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:58.919 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.919 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.919 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:06:58.919 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:54.056339+0000", 2024-09-21T14:06:58.919 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.361e-06 2024-09-21T14:06:58.919 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.920 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.920 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:06:58.920 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:54.056344+0000", 2024-09-21T14:06:58.920 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1499999999999998e-06 2024-09-21T14:06:58.920 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.920 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.920 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:06:58.920 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:54.056358+0000", 2024-09-21T14:06:58.920 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3878e-05 2024-09-21T14:06:58.920 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.920 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.920 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:06:58.920 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:54.056365+0000", 2024-09-21T14:06:58.920 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1690000000000003e-06 2024-09-21T14:06:58.920 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.920 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.921 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:58.921 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:54.056405+0000", 2024-09-21T14:06:58.921 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9891000000000001e-05 2024-09-21T14:06:58.921 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.921 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.921 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:06:58.921 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:54.056508+0000", 2024-09-21T14:06:58.921 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010338100000000001 2024-09-21T14:06:58.921 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.921 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.921 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3", 2024-09-21T14:06:58.921 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:54.056630+0000", 2024-09-21T14:06:58.921 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000121345 2024-09-21T14:06:58.921 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.921 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.921 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:58.922 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:54.059579+0000", 2024-09-21T14:06:58.922 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0029496290000000001 2024-09-21T14:06:58.922 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.922 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.922 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:06:58.922 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:54.064657+0000", 2024-09-21T14:06:58.922 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0050777620000000004 2024-09-21T14:06:58.922 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.922 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.922 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:58.922 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:54.368370+0000", 2024-09-21T14:06:58.922 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.30371298099999999 2024-09-21T14:06:58.922 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.922 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.922 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:06:58.923 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:54.368422+0000", 2024-09-21T14:06:58.923 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1606e-05 2024-09-21T14:06:58.923 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.923 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.923 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:06:58.923 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:54.368472+0000", 2024-09-21T14:06:58.923 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0237e-05 2024-09-21T14:06:58.923 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:58.923 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:06:58.923 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:58.923 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.923 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.923 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:208882 10.4 10:2992e84b:::benchmark_data_smithi049_176811_object81305:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e441)", 2024-09-21T14:06:58.923 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:05:54.058536+0000", 2024-09-21T14:06:58.923 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 64.743653159000004, 2024-09-21T14:06:58.924 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.31062339700000002, 2024-09-21T14:06:58.924 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:06:58.924 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:06:58.924 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:58.924 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:06:58.924 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:06:58.924 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:06:58.924 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 208882 2024-09-21T14:06:58.924 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.924 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:06:58.924 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.924 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:06:58.924 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:54.058536+0000", 2024-09-21T14:06:58.924 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:58.924 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.925 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.925 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:06:58.925 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:54.058536+0000", 2024-09-21T14:06:58.925 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:58.925 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.925 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.925 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:06:58.925 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:54.058537+0000", 2024-09-21T14:06:58.925 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.64e-07 2024-09-21T14:06:58.925 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.925 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.925 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:06:58.925 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:54.058541+0000", 2024-09-21T14:06:58.925 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.924e-06 2024-09-21T14:06:58.926 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.926 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.926 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:06:58.926 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:54.058552+0000", 2024-09-21T14:06:58.926 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0947999999999999e-05 2024-09-21T14:06:58.926 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.926 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.926 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:06:58.926 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:54.058559+0000", 2024-09-21T14:06:58.926 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6000000000000003e-06 2024-09-21T14:06:58.926 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.926 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.926 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:58.926 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:54.059611+0000", 2024-09-21T14:06:58.927 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001052568 2024-09-21T14:06:58.927 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.927 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.927 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:06:58.927 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:54.059719+0000", 2024-09-21T14:06:58.927 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000107534 2024-09-21T14:06:58.927 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.927 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.927 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3", 2024-09-21T14:06:58.927 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:54.059848+0000", 2024-09-21T14:06:58.927 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012904999999999999 2024-09-21T14:06:58.927 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.927 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.927 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:58.927 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:54.064648+0000", 2024-09-21T14:06:58.928 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.004800599 2024-09-21T14:06:58.928 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.928 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.928 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:06:58.928 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:54.064662+0000", 2024-09-21T14:06:58.928 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4017e-05 2024-09-21T14:06:58.928 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.928 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.928 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:58.928 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:54.369027+0000", 2024-09-21T14:06:58.928 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.30436491399999999 2024-09-21T14:06:58.928 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.928 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.928 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:06:58.929 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:54.369110+0000", 2024-09-21T14:06:58.929 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2520000000000003e-05 2024-09-21T14:06:58.929 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.929 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.929 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:06:58.929 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:54.369160+0000", 2024-09-21T14:06:58.929 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9759000000000003e-05 2024-09-21T14:06:58.929 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:58.929 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:06:58.929 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:58.929 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.929 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.929 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:208883 10.4 10:2bd14f13:::benchmark_data_smithi049_176811_object81306:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e441)", 2024-09-21T14:06:58.929 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:05:54.061308+0000", 2024-09-21T14:06:58.929 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 64.740881379000001, 2024-09-21T14:06:58.929 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.31954796400000002, 2024-09-21T14:06:58.930 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:06:58.930 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:06:58.930 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:06:58.930 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:06:58.930 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:06:58.930 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:06:58.930 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 208883 2024-09-21T14:06:58.930 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.930 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:06:58.930 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.930 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:06:58.930 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:54.061308+0000", 2024-09-21T14:06:58.930 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:58.930 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.930 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.931 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:06:58.931 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:54.061308+0000", 2024-09-21T14:06:58.931 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:06:58.931 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.931 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.931 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:06:58.931 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:54.061309+0000", 2024-09-21T14:06:58.931 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7900000000000007e-07 2024-09-21T14:06:58.931 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.931 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.931 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:06:58.931 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:54.061313+0000", 2024-09-21T14:06:58.931 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2980000000000002e-06 2024-09-21T14:06:58.931 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.931 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.931 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:06:58.932 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:54.061326+0000", 2024-09-21T14:06:58.932 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2174999999999999e-05 2024-09-21T14:06:58.932 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.932 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.932 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:06:58.932 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:54.061333+0000", 2024-09-21T14:06:58.932 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.345e-06 2024-09-21T14:06:58.932 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.932 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.932 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:06:58.932 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:54.062736+0000", 2024-09-21T14:06:58.932 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001402754 2024-09-21T14:06:58.932 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.932 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.932 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:06:58.932 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:54.062809+0000", 2024-09-21T14:06:58.933 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3077999999999994e-05 2024-09-21T14:06:58.933 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.933 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.933 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3", 2024-09-21T14:06:58.933 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:54.062883+0000", 2024-09-21T14:06:58.933 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4709999999999995e-05 2024-09-21T14:06:58.933 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.933 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.933 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:58.933 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:54.064694+0000", 2024-09-21T14:06:58.933 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001810231 2024-09-21T14:06:58.933 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.933 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.933 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:06:58.933 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:54.064742+0000", 2024-09-21T14:06:58.934 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8678999999999999e-05 2024-09-21T14:06:58.934 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.934 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.934 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:06:58.934 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:54.380733+0000", 2024-09-21T14:06:58.934 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.315990822 2024-09-21T14:06:58.934 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.934 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.934 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:06:58.934 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:54.380793+0000", 2024-09-21T14:06:58.934 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0087999999999998e-05 2024-09-21T14:06:58.934 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:06:58.934 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:06:58.934 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:06:58.934 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:05:54.380856+0000", 2024-09-21T14:06:58.935 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2805000000000006e-05 2024-09-21T14:06:58.935 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:58.935 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:06:58.935 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:58.935 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:06:58.935 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:06:58.935 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T14:06:58.967 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:58.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:58.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:06:59.069 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:06:59.094 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:59.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:59.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:06:59.195 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:06:59.228 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:59.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:59.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:06:59.329 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:06:59.350 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:59.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:59.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:06:59.452 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:59.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:59.495 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:59.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:59.505 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 6 16 15835 15819 164.734 141.875 0.00505256 0.00606356 2024-09-21T14:06:59.596 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:06:59.617 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:59.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:59.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:06:59.719 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:06:59.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:59 smithi049 ceph-mon[30080]: pgmap v921: 168 pgs: 168 active+clean; 335 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1.4 KiB/s rd, 48 MiB/s wr, 776 op/s 2024-09-21T14:06:59.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:59 smithi049 ceph-mon[30080]: osdmap e468: 8 total, 8 up, 8 in 2024-09-21T14:06:59.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:59 smithi049 ceph-mon[30080]: 5.16 scrub starts 2024-09-21T14:06:59.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:59 smithi049 ceph-mon[30080]: 5.16 scrub ok 2024-09-21T14:06:59.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:59 smithi049 ceph-mon[30080]: 4.4 scrub starts 2024-09-21T14:06:59.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:59 smithi049 ceph-mon[30080]: 4.4 scrub ok 2024-09-21T14:06:59.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:59 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:06:59.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:06:59 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:06:59.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:59 smithi049 ceph-mon[40489]: pgmap v921: 168 pgs: 168 active+clean; 335 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1.4 KiB/s rd, 48 MiB/s wr, 776 op/s 2024-09-21T14:06:59.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:59 smithi049 ceph-mon[40489]: osdmap e468: 8 total, 8 up, 8 in 2024-09-21T14:06:59.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:59 smithi049 ceph-mon[40489]: 5.16 scrub starts 2024-09-21T14:06:59.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:59 smithi049 ceph-mon[40489]: 5.16 scrub ok 2024-09-21T14:06:59.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:59 smithi049 ceph-mon[40489]: 4.4 scrub starts 2024-09-21T14:06:59.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:59 smithi049 ceph-mon[40489]: 4.4 scrub ok 2024-09-21T14:06:59.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:59 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:06:59.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:06:59 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:06:59.757 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:06:59.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:59.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:06:59.859 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:06:59.891 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:06:59.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:06:59.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:06:59.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:59 smithi187 ceph-mon[33032]: pgmap v921: 168 pgs: 168 active+clean; 335 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1.4 KiB/s rd, 48 MiB/s wr, 776 op/s 2024-09-21T14:06:59.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:59 smithi187 ceph-mon[33032]: osdmap e468: 8 total, 8 up, 8 in 2024-09-21T14:06:59.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:59 smithi187 ceph-mon[33032]: 5.16 scrub starts 2024-09-21T14:06:59.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:59 smithi187 ceph-mon[33032]: 5.16 scrub ok 2024-09-21T14:06:59.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:59 smithi187 ceph-mon[33032]: 4.4 scrub starts 2024-09-21T14:06:59.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:59 smithi187 ceph-mon[33032]: 4.4 scrub ok 2024-09-21T14:06:59.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:59 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:06:59.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:06:59 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:06:59.994 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:00.033 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:00.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:00.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:00.136 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:07:00.156 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:00.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:00.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:07:00.258 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:07:00.291 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:00.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:00.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:07:00.392 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:07:00.413 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:00.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:00.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:07:00.444 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:07:00 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:07:00.505 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 7 16 17834 17818 159.048 124.938 0.00386064 0.00628036 2024-09-21T14:07:00.514 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:07:00.548 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:00.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:00.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:07:00.650 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:00.680 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:00.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:00.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:00.782 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:07:00.810 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:00.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:00.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:07:00.912 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:07:00.944 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:00.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:00.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:07:01.004 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:07:00 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:07:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:07:01.046 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:07:01.066 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:01.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:01.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:07:01.168 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:01.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:01.190 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:01.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:01.291 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:07:01.314 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:01.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:01.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:07:01.416 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:01.454 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:01.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:01.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:01.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:01 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:07:01.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:01 smithi049 ceph-mon[40489]: pgmap v923: 168 pgs: 168 active+clean; 702 MiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 115 MiB/s wr, 1.85k op/s 2024-09-21T14:07:01.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:01 smithi049 ceph-mon[40489]: 4.1c scrub starts 2024-09-21T14:07:01.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:01 smithi049 ceph-mon[40489]: 4.1c scrub ok 2024-09-21T14:07:01.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:01 smithi049 ceph-mon[40489]: 4.9 scrub starts 2024-09-21T14:07:01.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:01 smithi049 ceph-mon[40489]: 4.9 scrub ok 2024-09-21T14:07:01.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:01 smithi049 ceph-mon[40489]: 5.15 scrub starts 2024-09-21T14:07:01.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:01 smithi049 ceph-mon[40489]: 5.15 scrub ok 2024-09-21T14:07:01.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:01 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:07:01.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:01 smithi049 ceph-mon[30080]: pgmap v923: 168 pgs: 168 active+clean; 702 MiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 115 MiB/s wr, 1.85k op/s 2024-09-21T14:07:01.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:01 smithi049 ceph-mon[30080]: 4.1c scrub starts 2024-09-21T14:07:01.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:01 smithi049 ceph-mon[30080]: 4.1c scrub ok 2024-09-21T14:07:01.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:01 smithi049 ceph-mon[30080]: 4.9 scrub starts 2024-09-21T14:07:01.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:01 smithi049 ceph-mon[30080]: 4.9 scrub ok 2024-09-21T14:07:01.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:01 smithi049 ceph-mon[30080]: 5.15 scrub starts 2024-09-21T14:07:01.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:01 smithi049 ceph-mon[30080]: 5.15 scrub ok 2024-09-21T14:07:01.506 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 8 16 19607 19591 153.018 110.812 0.00348447 0.0065287 2024-09-21T14:07:01.556 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:07:01.581 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:01.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:01.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:07:01.683 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:07:01.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:01 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:07:01.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:01 smithi187 ceph-mon[33032]: pgmap v923: 168 pgs: 168 active+clean; 702 MiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 115 MiB/s wr, 1.85k op/s 2024-09-21T14:07:01.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:01 smithi187 ceph-mon[33032]: 4.1c scrub starts 2024-09-21T14:07:01.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:01 smithi187 ceph-mon[33032]: 4.1c scrub ok 2024-09-21T14:07:01.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:01 smithi187 ceph-mon[33032]: 4.9 scrub starts 2024-09-21T14:07:01.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:01 smithi187 ceph-mon[33032]: 4.9 scrub ok 2024-09-21T14:07:01.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:01 smithi187 ceph-mon[33032]: 5.15 scrub starts 2024-09-21T14:07:01.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:01 smithi187 ceph-mon[33032]: 5.15 scrub ok 2024-09-21T14:07:01.715 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:01.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:01.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:07:01.817 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:07:01.847 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:01.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:01.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:07:01.949 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:07:01.988 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:01.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:01.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:07:02.090 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:07:02.127 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:02.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:02.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:07:02.229 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:07:02.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:02.260 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:02.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:07:02.361 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:07:02.399 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:02.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:02.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:07:02.500 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:07:02.505 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 9 16 21673 21657 150.363 129.125 0.00473147 0.00664425 2024-09-21T14:07:02.530 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:02.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:02.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:07:02.623 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T14:07:02.632 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:07:02.699 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:02.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:02.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:07:02.802 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:07:02.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:02.833 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:02.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:07:02.934 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:07:02.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:02.974 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:02.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:07:03.026 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:07:03.075 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:07:03.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:03.113 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:03.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:07:03.214 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:07:03.236 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:03.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:03.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:07:03.338 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:07:03.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:03.369 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:03.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:07:03.400 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:03 smithi187 ceph-mon[33032]: 1.f scrub starts 2024-09-21T14:07:03.401 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:03 smithi187 ceph-mon[33032]: 1.f scrub ok 2024-09-21T14:07:03.401 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:03 smithi187 ceph-mon[33032]: pgmap v924: 168 pgs: 168 active+clean; 1.1 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 141 MiB/s wr, 2.26k op/s 2024-09-21T14:07:03.401 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:03 smithi187 ceph-mon[33032]: 4.d scrub starts 2024-09-21T14:07:03.401 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:03 smithi187 ceph-mon[33032]: 4.d scrub ok 2024-09-21T14:07:03.470 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:07:03.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:03 smithi049 ceph-mon[40489]: 1.f scrub starts 2024-09-21T14:07:03.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:03 smithi049 ceph-mon[40489]: 1.f scrub ok 2024-09-21T14:07:03.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:03 smithi049 ceph-mon[40489]: pgmap v924: 168 pgs: 168 active+clean; 1.1 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 141 MiB/s wr, 2.26k op/s 2024-09-21T14:07:03.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:03 smithi049 ceph-mon[40489]: 4.d scrub starts 2024-09-21T14:07:03.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:03 smithi049 ceph-mon[40489]: 4.d scrub ok 2024-09-21T14:07:03.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:03 smithi049 ceph-mon[30080]: 1.f scrub starts 2024-09-21T14:07:03.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:03 smithi049 ceph-mon[30080]: 1.f scrub ok 2024-09-21T14:07:03.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:03 smithi049 ceph-mon[30080]: pgmap v924: 168 pgs: 168 active+clean; 1.1 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 141 MiB/s wr, 2.26k op/s 2024-09-21T14:07:03.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:03 smithi049 ceph-mon[30080]: 4.d scrub starts 2024-09-21T14:07:03.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:03 smithi049 ceph-mon[30080]: 4.d scrub ok 2024-09-21T14:07:03.505 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 10 16 23845 23829 148.9 135.75 0.00713379 0.00670829 2024-09-21T14:07:03.510 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:03.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:03.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:07:03.612 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:07:03.647 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:03.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:03.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:07:03.749 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:07:03.786 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:03.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:03.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:07:03.888 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:07:03.926 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T14:07:03.926 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-21T14:07:03.926 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:03.927 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26396.0:24732 11.3 11:cd2e4034:::benchmark_data_smithi049_206132_object24731:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e468)", 2024-09-21T14:07:03.927 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:07:03.922858+0000", 2024-09-21T14:07:03.927 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0030026369999999998, 2024-09-21T14:07:03.927 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0030362470000000002, 2024-09-21T14:07:03.927 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:07:03.927 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:07:03.927 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-21T14:07:03.927 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:07:03.927 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26396", 2024-09-21T14:07:03.927 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:07:03.927 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 24732 2024-09-21T14:07:03.927 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:03.927 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:07:03.927 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:03.928 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:07:03.928 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:03.922858+0000", 2024-09-21T14:07:03.928 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:07:03.928 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:03.928 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:03.928 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:07:03.928 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:03.922858+0000", 2024-09-21T14:07:03.928 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:07:03.928 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:03.928 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:03.928 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:07:03.928 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:03.922860+0000", 2024-09-21T14:07:03.928 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5889999999999999e-06 2024-09-21T14:07:03.928 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:03.928 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:03.928 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:07:03.929 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:03.922902+0000", 2024-09-21T14:07:03.929 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1700999999999999e-05 2024-09-21T14:07:03.929 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:03.929 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:03.929 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:07:03.929 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:03.922914+0000", 2024-09-21T14:07:03.929 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.26e-05 2024-09-21T14:07:03.929 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:03.929 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:03.929 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:07:03.929 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:03.922922+0000", 2024-09-21T14:07:03.929 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9359999999999999e-06 2024-09-21T14:07:03.929 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:03.929 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:03.929 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:07:03.930 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:03.922992+0000", 2024-09-21T14:07:03.930 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9791999999999999e-05 2024-09-21T14:07:03.930 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:03.930 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:03.930 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:07:03.930 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:03.923126+0000", 2024-09-21T14:07:03.930 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000133714 2024-09-21T14:07:03.930 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:03.930 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:03.930 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,7", 2024-09-21T14:07:03.930 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:03.923242+0000", 2024-09-21T14:07:03.930 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011620000000000001 2024-09-21T14:07:03.930 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:03.930 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:07:03.931 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:03.931 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:03.931 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:03.931 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.26396.0:24732 11.3 e468/461)", 2024-09-21T14:07:03.931 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:07:03.924960+0000", 2024-09-21T14:07:03.931 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.00090054100000000004, 2024-09-21T14:07:03.931 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0010030690000000001, 2024-09-21T14:07:03.931 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:07:03.931 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:07:03.931 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "queued for pg", 2024-09-21T14:07:03.931 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:07:03.931 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:03.931 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:07:03.931 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:03.924960+0000", 2024-09-21T14:07:03.931 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:07:03.931 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:03.932 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:03.932 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:07:03.932 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:03.924960+0000", 2024-09-21T14:07:03.932 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:07:03.932 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:03.932 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:03.932 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:07:03.932 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:03.924961+0000", 2024-09-21T14:07:03.932 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.85e-07 2024-09-21T14:07:03.932 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:03.932 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:03.932 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:07:03.932 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:03.924965+0000", 2024-09-21T14:07:03.932 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1810000000000003e-06 2024-09-21T14:07:03.932 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:03.933 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:03.933 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:07:03.933 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:03.924978+0000", 2024-09-21T14:07:03.933 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2288e-05 2024-09-21T14:07:03.933 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:03.933 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:03.933 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:07:03.933 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:03.924984+0000", 2024-09-21T14:07:03.933 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1820000000000003e-06 2024-09-21T14:07:03.933 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:03.933 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:07:03.933 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:03.933 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:03.933 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2024-09-21T14:07:03.933 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 2 2024-09-21T14:07:03.934 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T14:07:03.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:03.934 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:03.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:07:04.035 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:07:04.068 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:04.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:04.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:07:04.139 INFO:tasks.thrashosds.thrasher:j is {'epoch': 466, 'fsid': '40a3d2e8-781f-11ef-baf6-efdc52797490', 'created': '2024-09-21T13:42:57.838275+0000', 'modified': '2024-09-21T14:06:54.397986+0000', 'last_up_change': '2024-09-21T14:01:40.158454+0000', 'last_in_change': '2024-09-21T13:58:16.730861+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 22, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 11, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T13:46:21.096244+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 21, 'pg_placement_num': 21, 'pg_placement_num_target': 21, 'pg_num_target': 21, 'pg_num_pending': 21, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '438', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '436', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.2899999618530273, 'score_stable': 2.2899999618530273, 'optimal_score': 1, 'raw_score_acting': 2.2899999618530273, 'raw_score_stable': 2.2899999618530273, 'primary_affinity_weighted': 0.9783573150634766, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.9800000190734863}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-21T13:48:55.540712+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 63, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 0.9783573150634766, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.9800000190734863}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-21T13:48:56.417456+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.9783573150634766, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.9800000190734863}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-21T13:48:58.128455+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 0.9783573150634766, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.9800000190734863}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-21T13:49:00.069361+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9783573150634766, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.9800000190734863}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-21T13:49:02.127399+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9783573150634766, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.9800000190734863}}, {'pool': 11, 'pool_name': 'unique_pool_4', 'create_time': '2024-09-21T14:06:43.392407+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '465', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 0.9783573150634766, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.9800000190734863}}], 'osds': [{'osd': 0, 'uuid': '2595dca9-6219-4e0c-8728-751eb102e73a', 'up': 1, 'in': 1, 'weight': 0.6176910400390625, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 429, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6802', 'nonce': 2095561412}, {'type': 'v1', 'addr': '172.21.15.49:6803', 'nonce': 2095561412}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6804', 'nonce': 2095561412}, {'type': 'v1', 'addr': '172.21.15.49:6805', 'nonce': 2095561412}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6808', 'nonce': 2095561412}, {'type': 'v1', 'addr': '172.21.15.49:6809', 'nonce': 2095561412}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6806', 'nonce': 2095561412}, {'type': 'v1', 'addr': '172.21.15.49:6807', 'nonce': 2095561412}]}, 'public_addr': '172.21.15.49:6803/2095561412', 'cluster_addr': '172.21.15.49:6805/2095561412', 'heartbeat_back_addr': '172.21.15.49:6809/2095561412', 'heartbeat_front_addr': '172.21.15.49:6807/2095561412', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '0ba327f9-5524-4a40-abcc-729ded0b1917', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 461, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6810', 'nonce': 1382840527}, {'type': 'v1', 'addr': '172.21.15.49:6811', 'nonce': 1382840527}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6812', 'nonce': 1382840527}, {'type': 'v1', 'addr': '172.21.15.49:6813', 'nonce': 1382840527}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6816', 'nonce': 1382840527}, {'type': 'v1', 'addr': '172.21.15.49:6817', 'nonce': 1382840527}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6814', 'nonce': 1382840527}, {'type': 'v1', 'addr': '172.21.15.49:6815', 'nonce': 1382840527}]}, 'public_addr': '172.21.15.49:6811/1382840527', 'cluster_addr': '172.21.15.49:6813/1382840527', 'heartbeat_back_addr': '172.21.15.49:6817/1382840527', 'heartbeat_front_addr': '172.21.15.49:6815/1382840527', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '27a45750-204d-48fd-bb8f-ebb01273343d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.8268585205078125, 'last_clean_begin': 18, 'last_clean_end': 100, 'up_from': 130, 'up_thru': 461, 'down_at': 101, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6818', 'nonce': 1945314741}, {'type': 'v1', 'addr': '172.21.15.49:6819', 'nonce': 1945314741}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6820', 'nonce': 1945314741}, {'type': 'v1', 'addr': '172.21.15.49:6821', 'nonce': 1945314741}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6824', 'nonce': 1945314741}, {'type': 'v1', 'addr': '172.21.15.49:6825', 'nonce': 1945314741}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6822', 'nonce': 1945314741}, {'type': 'v1', 'addr': '172.21.15.49:6823', 'nonce': 1945314741}]}, 'public_addr': '172.21.15.49:6819/1945314741', 'cluster_addr': '172.21.15.49:6821/1945314741', 'heartbeat_back_addr': '172.21.15.49:6825/1945314741', 'heartbeat_front_addr': '172.21.15.49:6823/1945314741', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '528bc390-3dc9-4f3b-ab75-d7412c9625db', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 27, 'last_clean_end': 214, 'up_from': 240, 'up_thru': 461, 'down_at': 215, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6826', 'nonce': 886735721}, {'type': 'v1', 'addr': '172.21.15.49:6827', 'nonce': 886735721}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6828', 'nonce': 886735721}, {'type': 'v1', 'addr': '172.21.15.49:6829', 'nonce': 886735721}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6832', 'nonce': 886735721}, {'type': 'v1', 'addr': '172.21.15.49:6833', 'nonce': 886735721}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6830', 'nonce': 886735721}, {'type': 'v1', 'addr': '172.21.15.49:6831', 'nonce': 886735721}]}, 'public_addr': '172.21.15.49:6827/886735721', 'cluster_addr': '172.21.15.49:6829/886735721', 'heartbeat_back_addr': '172.21.15.49:6833/886735721', 'heartbeat_front_addr': '172.21.15.49:6831/886735721', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '6fc64698-2d37-4db0-bf3d-c7e612c45163', 'up': 1, 'in': 1, 'weight': 0.7888946533203125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 436, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6800', 'nonce': 1702447776}, {'type': 'v1', 'addr': '172.21.15.187:6801', 'nonce': 1702447776}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6802', 'nonce': 1702447776}, {'type': 'v1', 'addr': '172.21.15.187:6803', 'nonce': 1702447776}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6806', 'nonce': 1702447776}, {'type': 'v1', 'addr': '172.21.15.187:6807', 'nonce': 1702447776}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6804', 'nonce': 1702447776}, {'type': 'v1', 'addr': '172.21.15.187:6805', 'nonce': 1702447776}]}, 'public_addr': '172.21.15.187:6801/1702447776', 'cluster_addr': '172.21.15.187:6803/1702447776', 'heartbeat_back_addr': '172.21.15.187:6807/1702447776', 'heartbeat_front_addr': '172.21.15.187:6805/1702447776', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '8d7f62a1-7e92-4e06-9cad-86b96be514d8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 39, 'last_clean_end': 324, 'up_from': 345, 'up_thru': 461, 'down_at': 325, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6808', 'nonce': 1595353965}, {'type': 'v1', 'addr': '172.21.15.187:6809', 'nonce': 1595353965}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6810', 'nonce': 1595353965}, {'type': 'v1', 'addr': '172.21.15.187:6811', 'nonce': 1595353965}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6814', 'nonce': 1595353965}, {'type': 'v1', 'addr': '172.21.15.187:6815', 'nonce': 1595353965}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6812', 'nonce': 1595353965}, {'type': 'v1', 'addr': '172.21.15.187:6813', 'nonce': 1595353965}]}, 'public_addr': '172.21.15.187:6809/1595353965', 'cluster_addr': '172.21.15.187:6811/1595353965', 'heartbeat_back_addr': '172.21.15.187:6815/1595353965', 'heartbeat_front_addr': '172.21.15.187:6813/1595353965', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '3a7afd13-26ce-444c-9cc0-81ce2eab6ce5', 'up': 1, 'in': 1, 'weight': 0.9076080322265625, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 461, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6816', 'nonce': 2728426525}, {'type': 'v1', 'addr': '172.21.15.187:6817', 'nonce': 2728426525}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6818', 'nonce': 2728426525}, {'type': 'v1', 'addr': '172.21.15.187:6819', 'nonce': 2728426525}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6822', 'nonce': 2728426525}, {'type': 'v1', 'addr': '172.21.15.187:6823', 'nonce': 2728426525}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6820', 'nonce': 2728426525}, {'type': 'v1', 'addr': '172.21.15.187:6821', 'nonce': 2728426525}]}, 'public_addr': '172.21.15.187:6817/2728426525', 'cluster_addr': '172.21.15.187:6819/2728426525', 'heartbeat_back_addr': '172.21.15.187:6823/2728426525', 'heartbeat_front_addr': '172.21.15.187:6821/2728426525', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1e08d891-8747-402e-b4c2-aa3802ce8537', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 54, 'last_clean_end': 272, 'up_from': 298, 'up_thru': 461, 'down_at': 273, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 13894964}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 13894964}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 13894964}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 13894964}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6830', 'nonce': 13894964}, {'type': 'v1', 'addr': '172.21.15.187:6831', 'nonce': 13894964}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6828', 'nonce': 13894964}, {'type': 'v1', 'addr': '172.21.15.187:6829', 'nonce': 13894964}]}, 'public_addr': '172.21.15.187:6825/13894964', 'cluster_addr': '172.21.15.187:6827/13894964', 'heartbeat_back_addr': '172.21.15.187:6831/13894964', 'heartbeat_front_addr': '172.21.15.187:6829/13894964', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T14:06:38.896732+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T14:06:18.855219+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T14:06:27.643288+0000', 'dead_epoch': 100}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T14:05:41.501105+0000', 'dead_epoch': 214}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T14:06:40.662638+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T14:05:39.824748+0000', 'dead_epoch': 324}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T14:06:06.798526+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T14:06:03.637111+0000', 'dead_epoch': 272}], 'pg_upmap': [{'pgid': '3.14', 'osds': [7, 6, 0]}], 'pg_upmap_items': [{'pgid': '4.2', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.6', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.d', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '6.7', 'mappings': [{'from': 3, 'to': 7}]}, {'pgid': '6.b', 'mappings': [{'from': 3, 'to': 1}, {'from': 7, 'to': 6}]}, {'pgid': '6.1f', 'mappings': [{'from': 3, 'to': 0}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.49:0/2731793917': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:0/799831425': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:6801/2518197280': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:0/3556829486': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:0/1249926115': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:0/231530138': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:0/4220781134': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:0/202325455': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:6800/4129636473': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:6800/2518197280': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:0/3514741273': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:6801/4129636473': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:0/3125743776': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:0/3600977363': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:6801/3453571598': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:6800/3453571598': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:0/3553734432': '2024-09-22T13:49:27.311313+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-21T14:07:04.139 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T14:07:04.169 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:07:04.196 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:04.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:04.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:07:04.198 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:07:04.253 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:07:04.297 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:04.335 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:04.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:04.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:04.341 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:04 smithi049 ceph-mon[30080]: 5.1c scrub starts 2024-09-21T14:07:04.341 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:04 smithi049 ceph-mon[30080]: 5.1c scrub ok 2024-09-21T14:07:04.341 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:04 smithi049 ceph-mon[30080]: 6.5 scrub starts 2024-09-21T14:07:04.341 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:04 smithi049 ceph-mon[30080]: 6.5 scrub ok 2024-09-21T14:07:04.341 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:04 smithi049 ceph-mon[40489]: 5.1c scrub starts 2024-09-21T14:07:04.341 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:04 smithi049 ceph-mon[40489]: 5.1c scrub ok 2024-09-21T14:07:04.341 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:04 smithi049 ceph-mon[40489]: 6.5 scrub starts 2024-09-21T14:07:04.341 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:04 smithi049 ceph-mon[40489]: 6.5 scrub ok 2024-09-21T14:07:04.437 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:04.487 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:04.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:04.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:04.505 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 11 16 25867 25851 146.852 126.375 0.00832794 0.00680321 2024-09-21T14:07:04.589 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:07:04.616 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:04 smithi187 ceph-mon[33032]: 5.1c scrub starts 2024-09-21T14:07:04.616 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:04 smithi187 ceph-mon[33032]: 5.1c scrub ok 2024-09-21T14:07:04.616 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:04 smithi187 ceph-mon[33032]: 6.5 scrub starts 2024-09-21T14:07:04.616 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:04 smithi187 ceph-mon[33032]: 6.5 scrub ok 2024-09-21T14:07:04.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:04.640 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:04.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:07:04.666 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T14:07:04.741 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:07:04.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:04.785 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:04.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:07:04.886 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:07:04.919 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:04.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:04.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:07:05.020 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:07:05.038 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:07:05.055 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:05.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:05.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:07:05.157 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:07:05.209 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:05.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:05.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:07:05.311 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:07:05.341 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:05.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:05.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:07:05.389 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:05 smithi049 ceph-mon[40489]: pgmap v925: 168 pgs: 168 active+clean; 1.2 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 146 MiB/s wr, 2.35k op/s 2024-09-21T14:07:05.389 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:05 smithi049 ceph-mon[40489]: 4.2 scrub starts 2024-09-21T14:07:05.389 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:05 smithi049 ceph-mon[40489]: 4.2 scrub ok 2024-09-21T14:07:05.389 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:05 smithi049 ceph-mon[40489]: 4.e scrub starts 2024-09-21T14:07:05.389 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:05 smithi049 ceph-mon[40489]: 4.e scrub ok 2024-09-21T14:07:05.389 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:05 smithi049 ceph-mon[40489]: 6.19 scrub starts 2024-09-21T14:07:05.389 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:05 smithi049 ceph-mon[40489]: 6.19 scrub ok 2024-09-21T14:07:05.391 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:05 smithi049 ceph-mon[30080]: pgmap v925: 168 pgs: 168 active+clean; 1.2 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 146 MiB/s wr, 2.35k op/s 2024-09-21T14:07:05.391 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:05 smithi049 ceph-mon[30080]: 4.2 scrub starts 2024-09-21T14:07:05.391 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:05 smithi049 ceph-mon[30080]: 4.2 scrub ok 2024-09-21T14:07:05.391 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:05 smithi049 ceph-mon[30080]: 4.e scrub starts 2024-09-21T14:07:05.391 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:05 smithi049 ceph-mon[30080]: 4.e scrub ok 2024-09-21T14:07:05.391 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:05 smithi049 ceph-mon[30080]: 6.19 scrub starts 2024-09-21T14:07:05.391 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:05 smithi049 ceph-mon[30080]: 6.19 scrub ok 2024-09-21T14:07:05.405 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:07:05.443 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:07:05.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:05 smithi187 ceph-mon[33032]: pgmap v925: 168 pgs: 168 active+clean; 1.2 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 146 MiB/s wr, 2.35k op/s 2024-09-21T14:07:05.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:05 smithi187 ceph-mon[33032]: 4.2 scrub starts 2024-09-21T14:07:05.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:05 smithi187 ceph-mon[33032]: 4.2 scrub ok 2024-09-21T14:07:05.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:05 smithi187 ceph-mon[33032]: 4.e scrub starts 2024-09-21T14:07:05.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:05 smithi187 ceph-mon[33032]: 4.e scrub ok 2024-09-21T14:07:05.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:05 smithi187 ceph-mon[33032]: 6.19 scrub starts 2024-09-21T14:07:05.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:05 smithi187 ceph-mon[33032]: 6.19 scrub ok 2024-09-21T14:07:05.473 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:05.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:05.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:07:05.505 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 12 15 27781 27766 144.587 119.688 0.00561896 0.00691033 2024-09-21T14:07:05.575 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:07:05.586 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:07:05.586 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:07:05.615 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:05.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:05.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:07:05.631 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:07:05.631 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:07:05.665 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:07:05.666 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:07:05.693 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T14:07:05.693 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-21T14:07:05.693 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-21T14:07:05.693 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-21T14:07:05.693 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T14:07:05.701 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:07:05.701 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:07:05.717 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:07:05.736 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:07:05.736 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:07:05.751 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:05.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:05.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:07:05.772 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:07:05.772 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:07:05.817 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:07:05.817 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:07:05.849 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:07:05.849 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:07:05.852 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:07:05.893 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:05.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:05.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:07:05.994 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:07:06.033 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:06.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:06.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:07:06.135 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:07:06.175 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:06.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:06.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:07:06.259 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-09-21T14:07:06.277 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:06.319 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:06.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:06.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:06.421 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:07:06.462 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:06.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:06.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:07:06.487 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T14:07:06.490 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-09-21T14:07:06.505 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 13 16 29612 29596 142.263 114.375 0.00692818 0.00702297 2024-09-21T14:07:06.509 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:06 smithi049 ceph-mon[40489]: 5.2 scrub starts 2024-09-21T14:07:06.509 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:06 smithi049 ceph-mon[40489]: 5.2 scrub ok 2024-09-21T14:07:06.509 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:06 smithi049 ceph-mon[40489]: 4.17 scrub starts 2024-09-21T14:07:06.509 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:06 smithi049 ceph-mon[40489]: 4.17 scrub ok 2024-09-21T14:07:06.509 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:06 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1859272631' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:07:06.509 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:06 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/4016532633' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:07:06.510 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:06 smithi049 ceph-mon[30080]: 5.2 scrub starts 2024-09-21T14:07:06.510 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:06 smithi049 ceph-mon[30080]: 5.2 scrub ok 2024-09-21T14:07:06.510 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:06 smithi049 ceph-mon[30080]: 4.17 scrub starts 2024-09-21T14:07:06.510 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:06 smithi049 ceph-mon[30080]: 4.17 scrub ok 2024-09-21T14:07:06.510 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:06 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1859272631' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:07:06.510 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:06 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/4016532633' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:07:06.565 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:07:06.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:06.600 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:06.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:07:06.678 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:06 smithi187 ceph-mon[33032]: 5.2 scrub starts 2024-09-21T14:07:06.678 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:06 smithi187 ceph-mon[33032]: 5.2 scrub ok 2024-09-21T14:07:06.678 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:06 smithi187 ceph-mon[33032]: 4.17 scrub starts 2024-09-21T14:07:06.679 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:06 smithi187 ceph-mon[33032]: 4.17 scrub ok 2024-09-21T14:07:06.679 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:06 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1859272631' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:07:06.679 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:06 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/4016532633' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:07:06.701 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:06.737 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:06.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:06.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:06.838 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:07:06.865 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:06.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:06.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:07:06.967 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:07:07.002 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:07.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:07.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:07:07.045 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T14:07:07.104 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:07:07.141 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:07.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:07.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:07:07.243 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:07:07.276 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:07.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:07.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:07:07.377 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:07:07.409 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:07:07.417 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:07.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:07.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:07:07.506 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 14 15 31591 31576 140.94 123.75 0.00531661 0.00709002 2024-09-21T14:07:07.518 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:07:07.538 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:07 smithi187 ceph-mon[33032]: pgmap v926: 168 pgs: 168 active+clean; 1.4 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 590 B/s rd, 123 MiB/s wr, 1.97k op/s 2024-09-21T14:07:07.538 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:07 smithi187 ceph-mon[33032]: 1.14 scrub starts 2024-09-21T14:07:07.538 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:07 smithi187 ceph-mon[33032]: 1.14 scrub ok 2024-09-21T14:07:07.538 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:07 smithi187 ceph-mon[33032]: 6.1e scrub starts 2024-09-21T14:07:07.538 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:07 smithi187 ceph-mon[33032]: 6.1e scrub ok 2024-09-21T14:07:07.538 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:07 smithi187 ceph-mon[33032]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:07:07.538 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:07 smithi187 ceph-mon[33032]: 4.b scrub starts 2024-09-21T14:07:07.538 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:07 smithi187 ceph-mon[33032]: 4.b scrub ok 2024-09-21T14:07:07.539 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:07 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1859272631' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T14:07:07.539 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:07 smithi187 ceph-mon[33032]: osdmap e469: 8 total, 8 up, 8 in 2024-09-21T14:07:07.539 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:07 smithi187 ceph-mon[33032]: from='client.16530 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:07:07.540 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:07.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:07.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:07:07.641 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:07:07.685 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:07.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:07.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:07:07.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:07 smithi049 ceph-mon[40489]: pgmap v926: 168 pgs: 168 active+clean; 1.4 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 590 B/s rd, 123 MiB/s wr, 1.97k op/s 2024-09-21T14:07:07.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:07 smithi049 ceph-mon[40489]: 1.14 scrub starts 2024-09-21T14:07:07.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:07 smithi049 ceph-mon[40489]: 1.14 scrub ok 2024-09-21T14:07:07.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:07 smithi049 ceph-mon[40489]: 6.1e scrub starts 2024-09-21T14:07:07.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:07 smithi049 ceph-mon[40489]: 6.1e scrub ok 2024-09-21T14:07:07.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:07 smithi049 ceph-mon[40489]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:07:07.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:07 smithi049 ceph-mon[40489]: 4.b scrub starts 2024-09-21T14:07:07.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:07 smithi049 ceph-mon[40489]: 4.b scrub ok 2024-09-21T14:07:07.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:07 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1859272631' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T14:07:07.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:07 smithi049 ceph-mon[40489]: osdmap e469: 8 total, 8 up, 8 in 2024-09-21T14:07:07.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:07 smithi049 ceph-mon[40489]: from='client.16530 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:07:07.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:07 smithi049 ceph-mon[30080]: pgmap v926: 168 pgs: 168 active+clean; 1.4 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 590 B/s rd, 123 MiB/s wr, 1.97k op/s 2024-09-21T14:07:07.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:07 smithi049 ceph-mon[30080]: 1.14 scrub starts 2024-09-21T14:07:07.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:07 smithi049 ceph-mon[30080]: 1.14 scrub ok 2024-09-21T14:07:07.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:07 smithi049 ceph-mon[30080]: 6.1e scrub starts 2024-09-21T14:07:07.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:07 smithi049 ceph-mon[30080]: 6.1e scrub ok 2024-09-21T14:07:07.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:07 smithi049 ceph-mon[30080]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:07:07.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:07 smithi049 ceph-mon[30080]: 4.b scrub starts 2024-09-21T14:07:07.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:07 smithi049 ceph-mon[30080]: 4.b scrub ok 2024-09-21T14:07:07.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:07 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1859272631' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T14:07:07.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:07 smithi049 ceph-mon[30080]: osdmap e469: 8 total, 8 up, 8 in 2024-09-21T14:07:07.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:07 smithi049 ceph-mon[30080]: from='client.16530 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:07:07.787 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:07:07.816 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:07.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:07.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:07:07.918 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:07:07.955 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:07.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:07.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:07:08.007 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T14:07:08.007 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:07:08.057 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:07:08.079 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:08.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:08.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:07:08.181 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:07:08.221 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:08.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:08.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:07:08.323 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:07:08.363 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:08.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:08.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:07:08.465 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:07:08.504 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:08.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:08.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:07:08.506 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 15 16 33350 33334 138.868 109.875 0.00882191 0.00719478 2024-09-21T14:07:08.605 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:07:08.634 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:08.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:08.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:07:08.736 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:07:08.767 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:08.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:08.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:07:08.869 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:07:08.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:08.911 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:08.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:07:09.012 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:07:09.038 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:09.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:09.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:07:09.139 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:07:09.170 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:09.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:09.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:07:09.272 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:07:09.301 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:09.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:09.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:07:09.402 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:09.443 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:09.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:09.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:09.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:09 smithi049 ceph-mon[40489]: pgmap v928: 168 pgs: 168 active+clean; 1.7 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 1.0 KiB/s rd, 133 MiB/s wr, 2.12k op/s 2024-09-21T14:07:09.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:09 smithi049 ceph-mon[30080]: pgmap v928: 168 pgs: 168 active+clean; 1.7 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 1.0 KiB/s rd, 133 MiB/s wr, 2.12k op/s 2024-09-21T14:07:09.506 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 16 16 34962 34946 136.486 100.75 0.00333145 0.00729747 2024-09-21T14:07:09.545 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:07:09.570 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:09.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:09.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:07:09.672 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:07:09.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:09 smithi187 ceph-mon[33032]: pgmap v928: 168 pgs: 168 active+clean; 1.7 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 1.0 KiB/s rd, 133 MiB/s wr, 2.12k op/s 2024-09-21T14:07:09.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:09.710 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:09.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:07:09.810 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:07:09.830 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":927,"stamp":"2024-09-21T14:07:06.257987+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1471683099,"num_objects":22646,"num_object_clones":0,"num_object_copies":67938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22646,"num_whiteouts":0,"num_read":1562,"num_read_kb":1314,"num_write":23169,"num_write_kb":1436653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":205,"num_bytes_recovered":2978,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24911,"ondisk_log_size":24911,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":508,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":9308376,"kb_used_data":4881692,"kb_used_omap":23,"kb_used_meta":4426536,"kb_avail":740652840,"statfs":{"total":767960285184,"available":758428508160,"internally_reserved":0,"allocated":4998852608,"data_stored":4995244009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24148,"internal_metadata":4532773292},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,5,5,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1120468992,"num_objects":17097,"num_object_clones":0,"num_object_copies":51291,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17097,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":17097,"num_write_kb":1094208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.155630"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":776,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.456596+0000","last_change":"2024-09-21T14:05:10.397918+0000","last_active":"2024-09-21T14:06:58.456596+0000","last_peered":"2024-09-21T14:06:58.456596+0000","last_clean":"2024-09-21T14:06:58.456596+0000","last_became_active":"2024-09-21T14:04:21.204299+0000","last_became_peered":"2024-09-21T14:04:21.204299+0000","last_unstale":"2024-09-21T14:06:58.456596+0000","last_undegraded":"2024-09-21T14:06:58.456596+0000","last_fullsized":"2024-09-21T14:06:58.456596+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:10.397765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:05:10.397765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"290'15","reported_seq":929,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.462046+0000","last_change":"2024-09-21T14:01:34.785322+0000","last_active":"2024-09-21T14:06:58.462046+0000","last_peered":"2024-09-21T14:06:58.462046+0000","last_clean":"2024-09-21T14:06:58.462046+0000","last_became_active":"2024-09-21T13:58:17.771220+0000","last_became_peered":"2024-09-21T13:58:17.771220+0000","last_unstale":"2024-09-21T14:06:58.462046+0000","last_undegraded":"2024-09-21T14:06:58.462046+0000","last_fullsized":"2024-09-21T14:06:58.462046+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'15","last_scrub_stamp":"2024-09-21T14:01:34.785267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:01:34.785267+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:36.212273+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":878,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.463911+0000","last_change":"2024-09-21T14:03:33.301558+0000","last_active":"2024-09-21T14:06:58.463911+0000","last_peered":"2024-09-21T14:06:58.463911+0000","last_clean":"2024-09-21T14:06:58.463911+0000","last_became_active":"2024-09-21T14:01:41.202168+0000","last_became_peered":"2024-09-21T14:01:41.202168+0000","last_unstale":"2024-09-21T14:06:58.463911+0000","last_undegraded":"2024-09-21T14:06:58.463911+0000","last_fullsized":"2024-09-21T14:06:58.463911+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:33.301487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:33.301487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:03.434291+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":854,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.462182+0000","last_change":"2024-09-21T14:05:14.348636+0000","last_active":"2024-09-21T14:06:58.462182+0000","last_peered":"2024-09-21T14:06:58.462182+0000","last_clean":"2024-09-21T14:06:58.462182+0000","last_became_active":"2024-09-21T14:01:41.203275+0000","last_became_peered":"2024-09-21T14:01:41.203275+0000","last_unstale":"2024-09-21T14:06:58.462182+0000","last_undegraded":"2024-09-21T14:06:58.462182+0000","last_fullsized":"2024-09-21T14:06:58.462182+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:14.348563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:05:14.348563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:44.131432+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":823,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.464619+0000","last_change":"2024-09-21T14:04:00.241774+0000","last_active":"2024-09-21T14:06:58.464619+0000","last_peered":"2024-09-21T14:06:58.464619+0000","last_clean":"2024-09-21T14:06:58.464619+0000","last_became_active":"2024-09-21T13:59:32.203340+0000","last_became_peered":"2024-09-21T13:59:32.203340+0000","last_unstale":"2024-09-21T14:06:58.464619+0000","last_undegraded":"2024-09-21T14:06:58.464619+0000","last_fullsized":"2024-09-21T14:06:58.464619+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:00.241562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:04:00.241562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:43.498216+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":836,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.466539+0000","last_change":"2024-09-21T14:05:56.406305+0000","last_active":"2024-09-21T14:06:58.466539+0000","last_peered":"2024-09-21T14:06:58.466539+0000","last_clean":"2024-09-21T14:06:58.466539+0000","last_became_active":"2024-09-21T13:58:17.777893+0000","last_became_peered":"2024-09-21T13:58:17.777893+0000","last_unstale":"2024-09-21T14:06:58.466539+0000","last_undegraded":"2024-09-21T14:06:58.466539+0000","last_fullsized":"2024-09-21T14:06:58.466539+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:56.406233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:05:56.406233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:16.764993+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"290'27","reported_seq":810,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.461766+0000","last_change":"2024-09-21T14:04:08.320458+0000","last_active":"2024-09-21T14:06:58.461766+0000","last_peered":"2024-09-21T14:06:58.461766+0000","last_clean":"2024-09-21T14:06:58.461766+0000","last_became_active":"2024-09-21T13:57:12.703469+0000","last_became_peered":"2024-09-21T13:57:12.703469+0000","last_unstale":"2024-09-21T14:06:58.461766+0000","last_undegraded":"2024-09-21T14:06:58.461766+0000","last_fullsized":"2024-09-21T14:06:58.461766+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'27","last_scrub_stamp":"2024-09-21T14:04:08.320339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:04:08.320339+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:44.483186+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":583,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.459783+0000","last_change":"2024-09-21T14:04:35.907676+0000","last_active":"2024-09-21T14:06:58.459783+0000","last_peered":"2024-09-21T14:06:58.459783+0000","last_clean":"2024-09-21T14:06:58.459783+0000","last_became_active":"2024-09-21T14:04:21.208237+0000","last_became_peered":"2024-09-21T14:04:21.208237+0000","last_unstale":"2024-09-21T14:06:58.459783+0000","last_undegraded":"2024-09-21T14:06:58.459783+0000","last_fullsized":"2024-09-21T14:06:58.459783+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:35.907611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:04:35.907611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:07.939563+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":836,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.456422+0000","last_change":"2024-09-21T14:05:45.271378+0000","last_active":"2024-09-21T14:06:58.456422+0000","last_peered":"2024-09-21T14:06:58.456422+0000","last_clean":"2024-09-21T14:06:58.456422+0000","last_became_active":"2024-09-21T14:01:41.194517+0000","last_became_peered":"2024-09-21T14:01:41.194517+0000","last_unstale":"2024-09-21T14:06:58.456422+0000","last_undegraded":"2024-09-21T14:06:58.456422+0000","last_fullsized":"2024-09-21T14:06:58.456422+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:45.271320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:05:45.271320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:06.487120+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"290'15","reported_seq":676,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.466741+0000","last_change":"2024-09-21T14:06:14.293576+0000","last_active":"2024-09-21T14:06:58.466741+0000","last_peered":"2024-09-21T14:06:58.466741+0000","last_clean":"2024-09-21T14:06:58.466741+0000","last_became_active":"2024-09-21T14:04:21.208244+0000","last_became_peered":"2024-09-21T14:04:21.208244+0000","last_unstale":"2024-09-21T14:06:58.466741+0000","last_undegraded":"2024-09-21T14:06:58.466741+0000","last_fullsized":"2024-09-21T14:06:58.466741+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'15","last_scrub_stamp":"2024-09-21T14:06:14.293466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:06:14.293466+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:26.384373+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":785,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.455729+0000","last_change":"2024-09-21T14:06:30.375160+0000","last_active":"2024-09-21T14:06:58.455729+0000","last_peered":"2024-09-21T14:06:58.455729+0000","last_clean":"2024-09-21T14:06:58.455729+0000","last_became_active":"2024-09-21T14:01:41.188248+0000","last_became_peered":"2024-09-21T14:01:41.188248+0000","last_unstale":"2024-09-21T14:06:58.455729+0000","last_undegraded":"2024-09-21T14:06:58.455729+0000","last_fullsized":"2024-09-21T14:06:58.455729+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:30.375083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:06:30.375083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:31.395157+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":822,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.466578+0000","last_change":"2024-09-21T14:06:50.577626+0000","last_active":"2024-09-21T14:06:58.466578+0000","last_peered":"2024-09-21T14:06:58.466578+0000","last_clean":"2024-09-21T14:06:58.466578+0000","last_became_active":"2024-09-21T14:04:21.205416+0000","last_became_peered":"2024-09-21T14:04:21.205416+0000","last_unstale":"2024-09-21T14:06:58.466578+0000","last_undegraded":"2024-09-21T14:06:58.466578+0000","last_fullsized":"2024-09-21T14:06:58.466578+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:50.577560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:06:50.577560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:16.585406+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":813,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.455978+0000","last_change":"2024-09-21T14:05:46.334192+0000","last_active":"2024-09-21T14:06:58.455978+0000","last_peered":"2024-09-21T14:06:58.455978+0000","last_clean":"2024-09-21T14:06:58.455978+0000","last_became_active":"2024-09-21T14:01:41.189373+0000","last_became_peered":"2024-09-21T14:01:41.189373+0000","last_unstale":"2024-09-21T14:06:58.455978+0000","last_undegraded":"2024-09-21T14:06:58.455978+0000","last_fullsized":"2024-09-21T14:06:58.455978+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:46.334117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:05:46.334117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"290'7","reported_seq":676,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.464790+0000","last_change":"2024-09-21T14:06:19.344847+0000","last_active":"2024-09-21T14:06:58.464790+0000","last_peered":"2024-09-21T14:06:58.464790+0000","last_clean":"2024-09-21T14:06:58.464790+0000","last_became_active":"2024-09-21T14:04:21.213102+0000","last_became_peered":"2024-09-21T14:04:21.213102+0000","last_unstale":"2024-09-21T14:06:58.464790+0000","last_undegraded":"2024-09-21T14:06:58.464790+0000","last_fullsized":"2024-09-21T14:06:58.464790+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'7","last_scrub_stamp":"2024-09-21T14:06:19.344638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:06:19.344638+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:20.906218+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":798,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.457743+0000","last_change":"2024-09-21T14:05:59.400943+0000","last_active":"2024-09-21T14:06:58.457743+0000","last_peered":"2024-09-21T14:06:58.457743+0000","last_clean":"2024-09-21T14:06:58.457743+0000","last_became_active":"2024-09-21T13:59:32.204174+0000","last_became_peered":"2024-09-21T13:59:32.204174+0000","last_unstale":"2024-09-21T14:06:58.457743+0000","last_undegraded":"2024-09-21T14:06:58.457743+0000","last_fullsized":"2024-09-21T14:06:58.457743+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:59.400878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:05:59.400878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:22.062555+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":847,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:07:04.491944+0000","last_change":"2024-09-21T14:07:04.491944+0000","last_active":"2024-09-21T14:07:04.491944+0000","last_peered":"2024-09-21T14:07:04.491944+0000","last_clean":"2024-09-21T14:07:04.491944+0000","last_became_active":"2024-09-21T14:01:41.198672+0000","last_became_peered":"2024-09-21T14:01:41.198672+0000","last_unstale":"2024-09-21T14:07:04.491944+0000","last_undegraded":"2024-09-21T14:07:04.491944+0000","last_fullsized":"2024-09-21T14:07:04.491944+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:07:04.491870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:07:04.491870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:16.215308+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":896,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.463017+0000","last_change":"2024-09-21T14:03:36.018910+0000","last_active":"2024-09-21T14:06:58.463017+0000","last_peered":"2024-09-21T14:06:58.463017+0000","last_clean":"2024-09-21T14:06:58.463017+0000","last_became_active":"2024-09-21T14:01:41.200360+0000","last_became_peered":"2024-09-21T14:01:41.200360+0000","last_unstale":"2024-09-21T14:06:58.463017+0000","last_undegraded":"2024-09-21T14:06:58.463017+0000","last_fullsized":"2024-09-21T14:06:58.463017+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:36.018851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:36.018851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:06.204862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":821,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.462743+0000","last_change":"2024-09-21T14:05:51.215203+0000","last_active":"2024-09-21T14:06:58.462743+0000","last_peered":"2024-09-21T14:06:58.462743+0000","last_clean":"2024-09-21T14:06:58.462743+0000","last_became_active":"2024-09-21T14:04:21.202178+0000","last_became_peered":"2024-09-21T14:04:21.202178+0000","last_unstale":"2024-09-21T14:06:58.462743+0000","last_undegraded":"2024-09-21T14:06:58.462743+0000","last_fullsized":"2024-09-21T14:06:58.462743+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:51.215139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:05:51.215139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:08.727453+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"290'27","reported_seq":895,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:07:00.227575+0000","last_change":"2024-09-21T14:07:00.227428+0000","last_active":"2024-09-21T14:07:00.227575+0000","last_peered":"2024-09-21T14:07:00.227575+0000","last_clean":"2024-09-21T14:07:00.227575+0000","last_became_active":"2024-09-21T13:57:12.680783+0000","last_became_peered":"2024-09-21T13:57:12.680783+0000","last_unstale":"2024-09-21T14:07:00.227575+0000","last_undegraded":"2024-09-21T14:07:00.227575+0000","last_fullsized":"2024-09-21T14:07:00.227575+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'27","last_scrub_stamp":"2024-09-21T14:07:00.227334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:07:00.227334+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:20.501281+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":879,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.464884+0000","last_change":"2024-09-21T14:05:59.399621+0000","last_active":"2024-09-21T14:06:58.464884+0000","last_peered":"2024-09-21T14:06:58.464884+0000","last_clean":"2024-09-21T14:06:58.464884+0000","last_became_active":"2024-09-21T14:04:21.203937+0000","last_became_peered":"2024-09-21T14:04:21.203937+0000","last_unstale":"2024-09-21T14:06:58.464884+0000","last_undegraded":"2024-09-21T14:06:58.464884+0000","last_fullsized":"2024-09-21T14:06:58.464884+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:59.399552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:05:59.399552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:18.892698+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":88,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.462278+0000","last_change":"2024-09-21T14:05:30.242243+0000","last_active":"2024-09-21T14:06:58.462278+0000","last_peered":"2024-09-21T14:06:58.462278+0000","last_clean":"2024-09-21T14:06:58.462278+0000","last_became_active":"2024-09-21T14:05:30.241942+0000","last_became_peered":"2024-09-21T14:05:30.241942+0000","last_unstale":"2024-09-21T14:06:58.462278+0000","last_undegraded":"2024-09-21T14:06:58.462278+0000","last_fullsized":"2024-09-21T14:06:58.462278+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:57.962266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:57.962266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:34.269427+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":866,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.462849+0000","last_change":"2024-09-21T14:00:51.495433+0000","last_active":"2024-09-21T14:06:58.462849+0000","last_peered":"2024-09-21T14:06:58.462849+0000","last_clean":"2024-09-21T14:06:58.462849+0000","last_became_active":"2024-09-21T13:52:04.316016+0000","last_became_peered":"2024-09-21T13:52:04.316016+0000","last_unstale":"2024-09-21T14:06:58.462849+0000","last_undegraded":"2024-09-21T14:06:58.462849+0000","last_fullsized":"2024-09-21T14:06:58.462849+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:00:51.495368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:00:51.495368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:34.201254+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"290'18","reported_seq":829,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.464558+0000","last_change":"2024-09-21T14:06:49.375317+0000","last_active":"2024-09-21T14:06:58.464558+0000","last_peered":"2024-09-21T14:06:58.464558+0000","last_clean":"2024-09-21T14:06:58.464558+0000","last_became_active":"2024-09-21T13:57:12.684454+0000","last_became_peered":"2024-09-21T13:57:12.684454+0000","last_unstale":"2024-09-21T14:06:58.464558+0000","last_undegraded":"2024-09-21T14:06:58.464558+0000","last_fullsized":"2024-09-21T14:06:58.464558+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'18","last_scrub_stamp":"2024-09-21T14:06:49.375262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:06:49.375262+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:10.954563+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":869,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:07:03.254086+0000","last_change":"2024-09-21T14:07:03.254086+0000","last_active":"2024-09-21T14:07:03.254086+0000","last_peered":"2024-09-21T14:07:03.254086+0000","last_clean":"2024-09-21T14:07:03.254086+0000","last_became_active":"2024-09-21T13:57:12.681897+0000","last_became_peered":"2024-09-21T13:57:12.681897+0000","last_unstale":"2024-09-21T14:07:03.254086+0000","last_undegraded":"2024-09-21T14:07:03.254086+0000","last_fullsized":"2024-09-21T14:07:03.254086+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:07:03.254025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:07:03.254025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:27.756000+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":790,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.458428+0000","last_change":"2024-09-21T14:06:22.700164+0000","last_active":"2024-09-21T14:06:58.458428+0000","last_peered":"2024-09-21T14:06:58.458428+0000","last_clean":"2024-09-21T14:06:58.458428+0000","last_became_active":"2024-09-21T14:01:41.181946+0000","last_became_peered":"2024-09-21T14:01:41.181946+0000","last_unstale":"2024-09-21T14:06:58.458428+0000","last_undegraded":"2024-09-21T14:06:58.458428+0000","last_fullsized":"2024-09-21T14:06:58.458428+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T14:06:22.700098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:06:22.700098+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:46.623483+0000","scrub_duration":225,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":895,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.464084+0000","last_change":"2024-09-21T14:06:46.580150+0000","last_active":"2024-09-21T14:06:58.464084+0000","last_peered":"2024-09-21T14:06:58.464084+0000","last_clean":"2024-09-21T14:06:58.464084+0000","last_became_active":"2024-09-21T14:04:21.200889+0000","last_became_peered":"2024-09-21T14:04:21.200889+0000","last_unstale":"2024-09-21T14:06:58.464084+0000","last_undegraded":"2024-09-21T14:06:58.464084+0000","last_fullsized":"2024-09-21T14:06:58.464084+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T14:06:46.580099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:06:46.580099+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:05.603264+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"290'20","reported_seq":212,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.462666+0000","last_change":"2024-09-21T14:05:56.301994+0000","last_active":"2024-09-21T14:06:58.462666+0000","last_peered":"2024-09-21T14:06:58.462666+0000","last_clean":"2024-09-21T14:06:58.462666+0000","last_became_active":"2024-09-21T14:04:56.588859+0000","last_became_peered":"2024-09-21T14:04:56.588859+0000","last_unstale":"2024-09-21T14:06:58.462666+0000","last_undegraded":"2024-09-21T14:06:58.462666+0000","last_fullsized":"2024-09-21T14:06:58.462666+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'20","last_scrub_stamp":"2024-09-21T14:05:56.301903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:05:56.301903+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:11.468696+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"409'13","reported_seq":555,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.460758+0000","last_change":"2024-09-21T14:04:21.222657+0000","last_active":"2024-09-21T14:06:58.460758+0000","last_peered":"2024-09-21T14:06:58.460758+0000","last_clean":"2024-09-21T14:06:58.460758+0000","last_became_active":"2024-09-21T14:04:21.206136+0000","last_became_peered":"2024-09-21T14:04:21.206136+0000","last_unstale":"2024-09-21T14:06:58.460758+0000","last_undegraded":"2024-09-21T14:06:58.460758+0000","last_fullsized":"2024-09-21T14:06:58.460758+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'12","last_scrub_stamp":"2024-09-21T13:59:36.012693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:59:36.012693+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:39.856589+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.0","version":"468'1651","reported_seq":1676,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:07:04.834680+0000","last_change":"2024-09-21T14:06:45.198465+0000","last_active":"2024-09-21T14:07:04.834680+0000","last_peered":"2024-09-21T14:07:04.834680+0000","last_clean":"2024-09-21T14:07:04.834680+0000","last_became_active":"2024-09-21T14:06:45.198161+0000","last_became_peered":"2024-09-21T14:06:45.198161+0000","last_unstale":"2024-09-21T14:07:04.834680+0000","last_undegraded":"2024-09-21T14:07:04.834680+0000","last_fullsized":"2024-09-21T14:07:04.834680+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_clean_scrub_stamp":"2024-09-21T14:06:44.365469+0000","objects_scrubbed":0,"log_size":1651,"log_dups_size":0,"ondisk_log_size":1651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:10.360593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108199936,"num_objects":1651,"num_object_clones":0,"num_object_copies":4953,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1651,"num_write_kb":105664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"75'79","reported_seq":289,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.456188+0000","last_change":"2024-09-21T14:05:38.314512+0000","last_active":"2024-09-21T14:06:58.456188+0000","last_peered":"2024-09-21T14:06:58.456188+0000","last_clean":"2024-09-21T14:06:58.456188+0000","last_became_active":"2024-09-21T14:01:41.246888+0000","last_became_peered":"2024-09-21T14:01:41.246888+0000","last_unstale":"2024-09-21T14:06:58.456188+0000","last_undegraded":"2024-09-21T14:06:58.456188+0000","last_fullsized":"2024-09-21T14:06:58.456188+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:05:38.314431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:05:38.314431+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"290'27","reported_seq":930,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.467000+0000","last_change":"2024-09-21T14:04:03.259647+0000","last_active":"2024-09-21T14:06:58.467000+0000","last_peered":"2024-09-21T14:06:58.467000+0000","last_clean":"2024-09-21T14:06:58.467000+0000","last_became_active":"2024-09-21T13:57:12.695609+0000","last_became_peered":"2024-09-21T13:57:12.695609+0000","last_unstale":"2024-09-21T14:06:58.467000+0000","last_undegraded":"2024-09-21T14:06:58.467000+0000","last_fullsized":"2024-09-21T14:06:58.467000+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'27","last_scrub_stamp":"2024-09-21T14:04:03.259554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:04:03.259554+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:05.467439+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":808,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.461773+0000","last_change":"2024-09-21T14:04:01.183193+0000","last_active":"2024-09-21T14:06:58.461773+0000","last_peered":"2024-09-21T14:06:58.461773+0000","last_clean":"2024-09-21T14:06:58.461773+0000","last_became_active":"2024-09-21T13:59:32.205350+0000","last_became_peered":"2024-09-21T13:59:32.205350+0000","last_unstale":"2024-09-21T14:06:58.461773+0000","last_undegraded":"2024-09-21T14:06:58.461773+0000","last_fullsized":"2024-09-21T14:06:58.461773+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:01.183123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:01.183123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:39.482977+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"409'11","reported_seq":937,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.456156+0000","last_change":"2024-09-21T14:04:21.211284+0000","last_active":"2024-09-21T14:06:58.456156+0000","last_peered":"2024-09-21T14:06:58.456156+0000","last_clean":"2024-09-21T14:06:58.456156+0000","last_became_active":"2024-09-21T14:04:21.206029+0000","last_became_peered":"2024-09-21T14:04:21.206029+0000","last_unstale":"2024-09-21T14:06:58.456156+0000","last_undegraded":"2024-09-21T14:06:58.456156+0000","last_fullsized":"2024-09-21T14:06:58.456156+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'10","last_scrub_stamp":"2024-09-21T14:02:59.831128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:02:59.831128+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:11.439269+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":749,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.455956+0000","last_change":"2024-09-21T14:05:48.393515+0000","last_active":"2024-09-21T14:06:58.455956+0000","last_peered":"2024-09-21T14:06:58.455956+0000","last_clean":"2024-09-21T14:06:58.455956+0000","last_became_active":"2024-09-21T14:04:21.203505+0000","last_became_peered":"2024-09-21T14:04:21.203505+0000","last_unstale":"2024-09-21T14:06:58.455956+0000","last_undegraded":"2024-09-21T14:06:58.455956+0000","last_fullsized":"2024-09-21T14:06:58.455956+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:48.393427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:05:48.393427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"468'1267","reported_seq":1292,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:07:01.279670+0000","last_change":"2024-09-21T14:06:45.201708+0000","last_active":"2024-09-21T14:07:01.279670+0000","last_peered":"2024-09-21T14:07:01.279670+0000","last_clean":"2024-09-21T14:07:01.279670+0000","last_became_active":"2024-09-21T14:06:45.201471+0000","last_became_peered":"2024-09-21T14:06:45.201471+0000","last_unstale":"2024-09-21T14:07:01.279670+0000","last_undegraded":"2024-09-21T14:07:01.279670+0000","last_fullsized":"2024-09-21T14:07:01.279670+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_clean_scrub_stamp":"2024-09-21T14:06:44.365469+0000","objects_scrubbed":0,"log_size":1267,"log_dups_size":0,"ondisk_log_size":1267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:02.283284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83034112,"num_objects":1267,"num_object_clones":0,"num_object_copies":3801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1267,"num_write_kb":81088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"75'79","reported_seq":350,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.463095+0000","last_change":"2024-09-21T14:06:49.267268+0000","last_active":"2024-09-21T14:06:58.463095+0000","last_peered":"2024-09-21T14:06:58.463095+0000","last_clean":"2024-09-21T14:06:58.463095+0000","last_became_active":"2024-09-21T14:05:39.296209+0000","last_became_peered":"2024-09-21T14:05:39.296209+0000","last_unstale":"2024-09-21T14:06:58.463095+0000","last_undegraded":"2024-09-21T14:06:58.463095+0000","last_fullsized":"2024-09-21T14:06:58.463095+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:06:49.267210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:06:49.267210+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:14.671886+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"290'30","reported_seq":857,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.462112+0000","last_change":"2024-09-21T14:05:49.298373+0000","last_active":"2024-09-21T14:06:58.462112+0000","last_peered":"2024-09-21T14:06:58.462112+0000","last_clean":"2024-09-21T14:06:58.462112+0000","last_became_active":"2024-09-21T14:04:21.208982+0000","last_became_peered":"2024-09-21T14:04:21.208982+0000","last_unstale":"2024-09-21T14:06:58.462112+0000","last_undegraded":"2024-09-21T14:06:58.462112+0000","last_fullsized":"2024-09-21T14:06:58.462112+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'30","last_scrub_stamp":"2024-09-21T14:05:49.298270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:05:49.298270+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:17.897748+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":760,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.467028+0000","last_change":"2024-09-21T14:04:02.214555+0000","last_active":"2024-09-21T14:06:58.467028+0000","last_peered":"2024-09-21T14:06:58.467028+0000","last_clean":"2024-09-21T14:06:58.467028+0000","last_became_active":"2024-09-21T13:59:32.211004+0000","last_became_peered":"2024-09-21T13:59:32.211004+0000","last_unstale":"2024-09-21T14:06:58.467028+0000","last_undegraded":"2024-09-21T14:06:58.467028+0000","last_fullsized":"2024-09-21T14:06:58.467028+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:02.214464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:02.214464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:34.351763+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":878,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.462257+0000","last_change":"2024-09-21T14:06:29.295088+0000","last_active":"2024-09-21T14:06:58.462257+0000","last_peered":"2024-09-21T14:06:58.462257+0000","last_clean":"2024-09-21T14:06:58.462257+0000","last_became_active":"2024-09-21T13:59:32.220304+0000","last_became_peered":"2024-09-21T13:59:32.220304+0000","last_unstale":"2024-09-21T14:06:58.462257+0000","last_undegraded":"2024-09-21T14:06:58.462257+0000","last_fullsized":"2024-09-21T14:06:58.462257+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:29.295016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:06:29.295016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:52.033998+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":506,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.466845+0000","last_change":"2024-09-21T14:04:21.210131+0000","last_active":"2024-09-21T14:06:58.466845+0000","last_peered":"2024-09-21T14:06:58.466845+0000","last_clean":"2024-09-21T14:06:58.466845+0000","last_became_active":"2024-09-21T14:04:21.207782+0000","last_became_peered":"2024-09-21T14:04:21.207782+0000","last_unstale":"2024-09-21T14:06:58.466845+0000","last_undegraded":"2024-09-21T14:06:58.466845+0000","last_fullsized":"2024-09-21T14:06:58.466845+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:23.334162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:23.334162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:10.441367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"468'1713","reported_seq":1738,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:07:04.859744+0000","last_change":"2024-09-21T14:06:45.309299+0000","last_active":"2024-09-21T14:07:04.859744+0000","last_peered":"2024-09-21T14:07:04.859744+0000","last_clean":"2024-09-21T14:07:04.859744+0000","last_became_active":"2024-09-21T14:06:45.308994+0000","last_became_peered":"2024-09-21T14:06:45.308994+0000","last_unstale":"2024-09-21T14:07:04.859744+0000","last_undegraded":"2024-09-21T14:07:04.859744+0000","last_fullsized":"2024-09-21T14:07:04.859744+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_clean_scrub_stamp":"2024-09-21T14:06:44.365469+0000","objects_scrubbed":0,"log_size":1713,"log_dups_size":0,"ondisk_log_size":1713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:45.933129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112263168,"num_objects":1713,"num_object_clones":0,"num_object_copies":5139,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1713,"num_write_kb":109632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"75'79","reported_seq":361,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.464214+0000","last_change":"2024-09-21T14:06:53.322034+0000","last_active":"2024-09-21T14:06:58.464214+0000","last_peered":"2024-09-21T14:06:58.464214+0000","last_clean":"2024-09-21T14:06:58.464214+0000","last_became_active":"2024-09-21T14:05:39.301665+0000","last_became_peered":"2024-09-21T14:05:39.301665+0000","last_unstale":"2024-09-21T14:06:58.464214+0000","last_undegraded":"2024-09-21T14:06:58.464214+0000","last_fullsized":"2024-09-21T14:06:58.464214+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:06:53.321984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:06:53.321984+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:04.064078+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"290'28","reported_seq":975,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.462354+0000","last_change":"2024-09-21T14:05:57.295752+0000","last_active":"2024-09-21T14:06:58.462354+0000","last_peered":"2024-09-21T14:06:58.462354+0000","last_clean":"2024-09-21T14:06:58.462354+0000","last_became_active":"2024-09-21T14:01:41.200859+0000","last_became_peered":"2024-09-21T14:01:41.200859+0000","last_unstale":"2024-09-21T14:06:58.462354+0000","last_undegraded":"2024-09-21T14:06:58.462354+0000","last_fullsized":"2024-09-21T14:06:58.462354+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'28","last_scrub_stamp":"2024-09-21T14:05:57.295299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:05:57.295299+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:59.469011+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":153,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.470088+0000","last_change":"2024-09-21T14:06:25.406121+0000","last_active":"2024-09-21T14:06:58.470088+0000","last_peered":"2024-09-21T14:06:58.470088+0000","last_clean":"2024-09-21T14:06:58.470088+0000","last_became_active":"2024-09-21T14:04:21.205354+0000","last_became_peered":"2024-09-21T14:04:21.205354+0000","last_unstale":"2024-09-21T14:06:58.470088+0000","last_undegraded":"2024-09-21T14:06:58.470088+0000","last_fullsized":"2024-09-21T14:06:58.470088+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T14:06:25.406035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:06:25.406035+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:33.677665+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"241'10","reported_seq":966,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.461969+0000","last_change":"2024-09-21T14:06:29.326529+0000","last_active":"2024-09-21T14:06:58.461969+0000","last_peered":"2024-09-21T14:06:58.461969+0000","last_clean":"2024-09-21T14:06:58.461969+0000","last_became_active":"2024-09-21T13:57:12.686651+0000","last_became_peered":"2024-09-21T13:57:12.686651+0000","last_unstale":"2024-09-21T14:06:58.461969+0000","last_undegraded":"2024-09-21T14:06:58.461969+0000","last_fullsized":"2024-09-21T14:06:58.461969+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'10","last_scrub_stamp":"2024-09-21T14:06:29.326448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:06:29.326448+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:47.781933+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":823,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.457989+0000","last_change":"2024-09-21T14:04:45.055770+0000","last_active":"2024-09-21T14:06:58.457989+0000","last_peered":"2024-09-21T14:06:58.457989+0000","last_clean":"2024-09-21T14:06:58.457989+0000","last_became_active":"2024-09-21T13:59:32.209615+0000","last_became_peered":"2024-09-21T13:59:32.209615+0000","last_unstale":"2024-09-21T14:06:58.457989+0000","last_undegraded":"2024-09-21T14:06:58.457989+0000","last_fullsized":"2024-09-21T14:06:58.457989+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:45.055707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:04:45.055707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:42.450896+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.d","version":"468'1417","reported_seq":1442,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:07:03.092854+0000","last_change":"2024-09-21T14:06:45.307126+0000","last_active":"2024-09-21T14:07:03.092854+0000","last_peered":"2024-09-21T14:07:03.092854+0000","last_clean":"2024-09-21T14:07:03.092854+0000","last_became_active":"2024-09-21T14:06:45.306827+0000","last_became_peered":"2024-09-21T14:06:45.306827+0000","last_unstale":"2024-09-21T14:07:03.092854+0000","last_undegraded":"2024-09-21T14:07:03.092854+0000","last_fullsized":"2024-09-21T14:07:03.092854+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_clean_scrub_stamp":"2024-09-21T14:06:44.365469+0000","objects_scrubbed":0,"log_size":1417,"log_dups_size":0,"ondisk_log_size":1417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:48.540029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92864512,"num_objects":1417,"num_object_clones":0,"num_object_copies":4251,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1417,"num_write_kb":90688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"75'79","reported_seq":922,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.460459+0000","last_change":"2024-09-21T14:05:45.878618+0000","last_active":"2024-09-21T14:06:58.460459+0000","last_peered":"2024-09-21T14:06:58.460459+0000","last_clean":"2024-09-21T14:06:58.460459+0000","last_became_active":"2024-09-21T14:05:39.298700+0000","last_became_peered":"2024-09-21T14:05:39.298700+0000","last_unstale":"2024-09-21T14:06:58.460459+0000","last_undegraded":"2024-09-21T14:06:58.460459+0000","last_fullsized":"2024-09-21T14:06:58.460459+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:05:45.878553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:05:45.878553+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:00.537504+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"290'20","reported_seq":191,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.458720+0000","last_change":"2024-09-21T14:04:42.776698+0000","last_active":"2024-09-21T14:06:58.458720+0000","last_peered":"2024-09-21T14:06:58.458720+0000","last_clean":"2024-09-21T14:06:58.458720+0000","last_became_active":"2024-09-21T14:03:29.158218+0000","last_became_peered":"2024-09-21T14:03:29.158218+0000","last_unstale":"2024-09-21T14:06:58.458720+0000","last_undegraded":"2024-09-21T14:06:58.458720+0000","last_fullsized":"2024-09-21T14:06:58.458720+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'20","last_scrub_stamp":"2024-09-21T14:04:42.776603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:04:42.776603+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:03.753526+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"61'1","reported_seq":804,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.456031+0000","last_change":"2024-09-21T14:06:18.400004+0000","last_active":"2024-09-21T14:06:58.456031+0000","last_peered":"2024-09-21T14:06:58.456031+0000","last_clean":"2024-09-21T14:06:58.456031+0000","last_became_active":"2024-09-21T14:01:41.195268+0000","last_became_peered":"2024-09-21T14:01:41.195268+0000","last_unstale":"2024-09-21T14:06:58.456031+0000","last_undegraded":"2024-09-21T14:06:58.456031+0000","last_fullsized":"2024-09-21T14:06:58.456031+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T14:06:18.399922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:06:18.399922+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:29.876068+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"418'16","reported_seq":242,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:07:03.344444+0000","last_change":"2024-09-21T14:05:53.307706+0000","last_active":"2024-09-21T14:07:03.344444+0000","last_peered":"2024-09-21T14:07:03.344444+0000","last_clean":"2024-09-21T14:07:03.344444+0000","last_became_active":"2024-09-21T14:04:56.589828+0000","last_became_peered":"2024-09-21T14:04:56.589828+0000","last_unstale":"2024-09-21T14:07:03.344444+0000","last_undegraded":"2024-09-21T14:07:03.344444+0000","last_fullsized":"2024-09-21T14:07:03.344444+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"418'16","last_scrub_stamp":"2024-09-21T14:05:53.307631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:05:53.307631+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:19.796274+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":316,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.462100+0000","last_change":"2024-09-21T14:05:46.372772+0000","last_active":"2024-09-21T14:06:58.462100+0000","last_peered":"2024-09-21T14:06:58.462100+0000","last_clean":"2024-09-21T14:06:58.462100+0000","last_became_active":"2024-09-21T14:04:56.586891+0000","last_became_peered":"2024-09-21T14:04:56.586891+0000","last_unstale":"2024-09-21T14:06:58.462100+0000","last_undegraded":"2024-09-21T14:06:58.462100+0000","last_fullsized":"2024-09-21T14:06:58.462100+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:46.372709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:05:46.372709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:08.643754+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"468'1673","reported_seq":1698,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:07:04.838394+0000","last_change":"2024-09-21T14:06:45.306909+0000","last_active":"2024-09-21T14:07:04.838394+0000","last_peered":"2024-09-21T14:07:04.838394+0000","last_clean":"2024-09-21T14:07:04.838394+0000","last_became_active":"2024-09-21T14:06:45.306632+0000","last_became_peered":"2024-09-21T14:06:45.306632+0000","last_unstale":"2024-09-21T14:07:04.838394+0000","last_undegraded":"2024-09-21T14:07:04.838394+0000","last_fullsized":"2024-09-21T14:07:04.838394+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_clean_scrub_stamp":"2024-09-21T14:06:44.365469+0000","objects_scrubbed":0,"log_size":1673,"log_dups_size":0,"ondisk_log_size":1673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:54.580260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109641728,"num_objects":1673,"num_object_clones":0,"num_object_copies":5019,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1673,"num_write_kb":107072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"75'79","reported_seq":930,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.460085+0000","last_change":"2024-09-21T14:05:41.888291+0000","last_active":"2024-09-21T14:06:58.460085+0000","last_peered":"2024-09-21T14:06:58.460085+0000","last_clean":"2024-09-21T14:06:58.460085+0000","last_became_active":"2024-09-21T14:05:39.309814+0000","last_became_peered":"2024-09-21T14:05:39.309814+0000","last_unstale":"2024-09-21T14:06:58.460085+0000","last_undegraded":"2024-09-21T14:06:58.460085+0000","last_fullsized":"2024-09-21T14:06:58.460085+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:05:41.888228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:05:41.888228+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:44.753436+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"290'29","reported_seq":617,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.462991+0000","last_change":"2024-09-21T14:06:15.301878+0000","last_active":"2024-09-21T14:06:58.462991+0000","last_peered":"2024-09-21T14:06:58.462991+0000","last_clean":"2024-09-21T14:06:58.462991+0000","last_became_active":"2024-09-21T14:04:56.589018+0000","last_became_peered":"2024-09-21T14:04:56.589018+0000","last_unstale":"2024-09-21T14:06:58.462991+0000","last_undegraded":"2024-09-21T14:06:58.462991+0000","last_fullsized":"2024-09-21T14:06:58.462991+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'29","last_scrub_stamp":"2024-09-21T14:06:15.301794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:06:15.301794+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:29.443136+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":879,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.465152+0000","last_change":"2024-09-21T14:06:27.457531+0000","last_active":"2024-09-21T14:06:58.465152+0000","last_peered":"2024-09-21T14:06:58.465152+0000","last_clean":"2024-09-21T14:06:58.465152+0000","last_became_active":"2024-09-21T14:01:41.185400+0000","last_became_peered":"2024-09-21T14:01:41.185400+0000","last_unstale":"2024-09-21T14:06:58.465152+0000","last_undegraded":"2024-09-21T14:06:58.465152+0000","last_fullsized":"2024-09-21T14:06:58.465152+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:27.457452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:06:27.457452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:49.107109+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":455,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.460376+0000","last_change":"2024-09-21T14:04:03.070260+0000","last_active":"2024-09-21T14:06:58.460376+0000","last_peered":"2024-09-21T14:06:58.460376+0000","last_clean":"2024-09-21T14:06:58.460376+0000","last_became_active":"2024-09-21T14:01:41.193973+0000","last_became_peered":"2024-09-21T14:01:41.193973+0000","last_unstale":"2024-09-21T14:06:58.460376+0000","last_undegraded":"2024-09-21T14:06:58.460376+0000","last_fullsized":"2024-09-21T14:06:58.460376+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:03.069973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:04:03.069973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:02.030696+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":862,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.463771+0000","last_change":"2024-09-21T13:58:43.952014+0000","last_active":"2024-09-21T14:06:58.463771+0000","last_peered":"2024-09-21T14:06:58.463771+0000","last_clean":"2024-09-21T14:06:58.463771+0000","last_became_active":"2024-09-21T13:58:17.757628+0000","last_became_peered":"2024-09-21T13:58:17.757628+0000","last_unstale":"2024-09-21T14:06:58.463771+0000","last_undegraded":"2024-09-21T14:06:58.463771+0000","last_fullsized":"2024-09-21T14:06:58.463771+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:43.951941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:43.951941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:41.500702+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.b","version":"468'1170","reported_seq":1195,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:07:01.476160+0000","last_change":"2024-09-21T14:06:45.309354+0000","last_active":"2024-09-21T14:07:01.476160+0000","last_peered":"2024-09-21T14:07:01.476160+0000","last_clean":"2024-09-21T14:07:01.476160+0000","last_became_active":"2024-09-21T14:06:45.309027+0000","last_became_peered":"2024-09-21T14:06:45.309027+0000","last_unstale":"2024-09-21T14:07:01.476160+0000","last_undegraded":"2024-09-21T14:07:01.476160+0000","last_fullsized":"2024-09-21T14:07:01.476160+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_clean_scrub_stamp":"2024-09-21T14:06:44.365469+0000","objects_scrubbed":0,"log_size":1170,"log_dups_size":0,"ondisk_log_size":1170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:07.075781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76677120,"num_objects":1170,"num_object_clones":0,"num_object_copies":3510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1170,"num_write_kb":74880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"75'79","reported_seq":286,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.455696+0000","last_change":"2024-09-21T14:05:40.322410+0000","last_active":"2024-09-21T14:06:58.455696+0000","last_peered":"2024-09-21T14:06:58.455696+0000","last_clean":"2024-09-21T14:06:58.455696+0000","last_became_active":"2024-09-21T14:05:39.310615+0000","last_became_peered":"2024-09-21T14:05:39.310615+0000","last_unstale":"2024-09-21T14:06:58.455696+0000","last_undegraded":"2024-09-21T14:06:58.455696+0000","last_fullsized":"2024-09-21T14:06:58.455696+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:05:40.322344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:05:40.322344+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:59.230708+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"290'44","reported_seq":1179,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.701679+0000","last_change":"2024-09-21T14:06:58.701515+0000","last_active":"2024-09-21T14:06:58.701679+0000","last_peered":"2024-09-21T14:06:58.701679+0000","last_clean":"2024-09-21T14:06:58.701679+0000","last_became_active":"2024-09-21T13:57:12.684690+0000","last_became_peered":"2024-09-21T13:57:12.684690+0000","last_unstale":"2024-09-21T14:06:58.701679+0000","last_undegraded":"2024-09-21T14:06:58.701679+0000","last_fullsized":"2024-09-21T14:06:58.701679+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'44","last_scrub_stamp":"2024-09-21T14:06:58.701390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:06:58.701390+0000","objects_scrubbed":10,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:01.036756+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":894,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.462753+0000","last_change":"2024-09-21T14:06:19.213603+0000","last_active":"2024-09-21T14:06:58.462753+0000","last_peered":"2024-09-21T14:06:58.462753+0000","last_clean":"2024-09-21T14:06:58.462753+0000","last_became_active":"2024-09-21T13:58:17.770937+0000","last_became_peered":"2024-09-21T13:58:17.770937+0000","last_unstale":"2024-09-21T14:06:58.462753+0000","last_undegraded":"2024-09-21T14:06:58.462753+0000","last_fullsized":"2024-09-21T14:06:58.462753+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:19.213554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:06:19.213554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:20.328797+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":823,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.456093+0000","last_change":"2024-09-21T14:05:15.377389+0000","last_active":"2024-09-21T14:06:58.456093+0000","last_peered":"2024-09-21T14:06:58.456093+0000","last_clean":"2024-09-21T14:06:58.456093+0000","last_became_active":"2024-09-21T14:01:41.247642+0000","last_became_peered":"2024-09-21T14:01:41.247642+0000","last_unstale":"2024-09-21T14:06:58.456093+0000","last_undegraded":"2024-09-21T14:06:58.456093+0000","last_fullsized":"2024-09-21T14:06:58.456093+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-21T14:05:15.377198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T14:05:15.377198+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:45.581805+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":812,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.464689+0000","last_change":"2024-09-21T14:06:21.386168+0000","last_active":"2024-09-21T14:06:58.464689+0000","last_peered":"2024-09-21T14:06:58.464689+0000","last_clean":"2024-09-21T14:06:58.464689+0000","last_became_active":"2024-09-21T14:04:21.204315+0000","last_became_peered":"2024-09-21T14:04:21.204315+0000","last_unstale":"2024-09-21T14:06:58.464689+0000","last_undegraded":"2024-09-21T14:06:58.464689+0000","last_fullsized":"2024-09-21T14:06:58.464689+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:21.386103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:06:21.386103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:23.983959+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":795,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.466917+0000","last_change":"2024-09-21T14:06:25.273212+0000","last_active":"2024-09-21T14:06:58.466917+0000","last_peered":"2024-09-21T14:06:58.466917+0000","last_clean":"2024-09-21T14:06:58.466917+0000","last_became_active":"2024-09-21T13:59:32.208980+0000","last_became_peered":"2024-09-21T13:59:32.208980+0000","last_unstale":"2024-09-21T14:06:58.466917+0000","last_undegraded":"2024-09-21T14:06:58.466917+0000","last_fullsized":"2024-09-21T14:06:58.466917+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:25.273146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:06:25.273146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:42.805356+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"468'1238","reported_seq":1263,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:07:01.465699+0000","last_change":"2024-09-21T14:06:45.200125+0000","last_active":"2024-09-21T14:07:01.465699+0000","last_peered":"2024-09-21T14:07:01.465699+0000","last_clean":"2024-09-21T14:07:01.465699+0000","last_became_active":"2024-09-21T14:06:45.199758+0000","last_became_peered":"2024-09-21T14:06:45.199758+0000","last_unstale":"2024-09-21T14:07:01.465699+0000","last_undegraded":"2024-09-21T14:07:01.465699+0000","last_fullsized":"2024-09-21T14:07:01.465699+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_clean_scrub_stamp":"2024-09-21T14:06:44.365469+0000","objects_scrubbed":0,"log_size":1238,"log_dups_size":0,"ondisk_log_size":1238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:11.671431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81133568,"num_objects":1238,"num_object_clones":0,"num_object_copies":3714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1238,"num_write_kb":79232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"75'79","reported_seq":927,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.457610+0000","last_change":"2024-09-21T14:05:49.356107+0000","last_active":"2024-09-21T14:06:58.457610+0000","last_peered":"2024-09-21T14:06:58.457610+0000","last_clean":"2024-09-21T14:06:58.457610+0000","last_became_active":"2024-09-21T14:05:39.303625+0000","last_became_peered":"2024-09-21T14:05:39.303625+0000","last_unstale":"2024-09-21T14:06:58.457610+0000","last_undegraded":"2024-09-21T14:06:58.457610+0000","last_fullsized":"2024-09-21T14:06:58.457610+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:05:49.356026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:05:49.356026+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:15.665558+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"290'17","reported_seq":916,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.467099+0000","last_change":"2024-09-21T14:04:21.224066+0000","last_active":"2024-09-21T14:06:58.467099+0000","last_peered":"2024-09-21T14:06:58.467099+0000","last_clean":"2024-09-21T14:06:58.467099+0000","last_became_active":"2024-09-21T14:04:21.205738+0000","last_became_peered":"2024-09-21T14:04:21.205738+0000","last_unstale":"2024-09-21T14:06:58.467099+0000","last_undegraded":"2024-09-21T14:06:58.467099+0000","last_fullsized":"2024-09-21T14:06:58.467099+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'13","last_scrub_stamp":"2024-09-21T13:58:11.619848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:11.619848+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:32.462295+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":915,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.463810+0000","last_change":"2024-09-21T14:04:10.193524+0000","last_active":"2024-09-21T14:06:58.463810+0000","last_peered":"2024-09-21T14:06:58.463810+0000","last_clean":"2024-09-21T14:06:58.463810+0000","last_became_active":"2024-09-21T13:58:17.759671+0000","last_became_peered":"2024-09-21T13:58:17.759671+0000","last_unstale":"2024-09-21T14:06:58.463810+0000","last_undegraded":"2024-09-21T14:06:58.463810+0000","last_fullsized":"2024-09-21T14:06:58.463810+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:10.193318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:10.193318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:07.426514+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":788,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.453894+0000","last_change":"2024-09-21T13:58:41.644876+0000","last_active":"2024-09-21T14:06:58.453894+0000","last_peered":"2024-09-21T14:06:58.453894+0000","last_clean":"2024-09-21T14:06:58.453894+0000","last_became_active":"2024-09-21T13:57:12.682147+0000","last_became_peered":"2024-09-21T13:57:12.682147+0000","last_unstale":"2024-09-21T14:06:58.453894+0000","last_undegraded":"2024-09-21T14:06:58.453894+0000","last_fullsized":"2024-09-21T14:06:58.453894+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T13:58:41.644789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T13:58:41.644789+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:39.967186+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":782,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.455312+0000","last_change":"2024-09-21T14:06:57.211334+0000","last_active":"2024-09-21T14:06:58.455312+0000","last_peered":"2024-09-21T14:06:58.455312+0000","last_clean":"2024-09-21T14:06:58.455312+0000","last_became_active":"2024-09-21T14:01:41.182000+0000","last_became_peered":"2024-09-21T14:01:41.182000+0000","last_unstale":"2024-09-21T14:06:58.455312+0000","last_undegraded":"2024-09-21T14:06:58.455312+0000","last_fullsized":"2024-09-21T14:06:58.455312+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:57.211269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:06:57.211269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:58.902061+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":823,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.458194+0000","last_change":"2024-09-21T13:59:32.222791+0000","last_active":"2024-09-21T14:06:58.458194+0000","last_peered":"2024-09-21T14:06:58.458194+0000","last_clean":"2024-09-21T14:06:58.458194+0000","last_became_active":"2024-09-21T13:59:32.221502+0000","last_became_peered":"2024-09-21T13:59:32.221502+0000","last_unstale":"2024-09-21T14:06:58.458194+0000","last_undegraded":"2024-09-21T14:06:58.458194+0000","last_fullsized":"2024-09-21T14:06:58.458194+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:14.673582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:14.673582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:00.357786+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"468'1219","reported_seq":1244,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:07:01.458869+0000","last_change":"2024-09-21T14:06:45.199524+0000","last_active":"2024-09-21T14:07:01.458869+0000","last_peered":"2024-09-21T14:07:01.458869+0000","last_clean":"2024-09-21T14:07:01.458869+0000","last_became_active":"2024-09-21T14:06:45.199164+0000","last_became_peered":"2024-09-21T14:06:45.199164+0000","last_unstale":"2024-09-21T14:07:01.458869+0000","last_undegraded":"2024-09-21T14:07:01.458869+0000","last_fullsized":"2024-09-21T14:07:01.458869+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_clean_scrub_stamp":"2024-09-21T14:06:44.365469+0000","objects_scrubbed":0,"log_size":1219,"log_dups_size":0,"ondisk_log_size":1219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:03.031145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79888384,"num_objects":1219,"num_object_clones":0,"num_object_copies":3657,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1219,"num_write_kb":78016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"75'79","reported_seq":349,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.465433+0000","last_change":"2024-09-21T14:06:52.656799+0000","last_active":"2024-09-21T14:06:58.465433+0000","last_peered":"2024-09-21T14:06:58.465433+0000","last_clean":"2024-09-21T14:06:58.465433+0000","last_became_active":"2024-09-21T14:05:39.311460+0000","last_became_peered":"2024-09-21T14:05:39.311460+0000","last_unstale":"2024-09-21T14:06:58.465433+0000","last_undegraded":"2024-09-21T14:06:58.465433+0000","last_fullsized":"2024-09-21T14:06:58.465433+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:06:52.656733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:06:52.656733+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:06.411252+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"290'24","reported_seq":712,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.456282+0000","last_change":"2024-09-21T14:04:21.221906+0000","last_active":"2024-09-21T14:06:58.456282+0000","last_peered":"2024-09-21T14:06:58.456282+0000","last_clean":"2024-09-21T14:06:58.456282+0000","last_became_active":"2024-09-21T14:04:21.206881+0000","last_became_peered":"2024-09-21T14:04:21.206881+0000","last_unstale":"2024-09-21T14:06:58.456282+0000","last_undegraded":"2024-09-21T14:06:58.456282+0000","last_fullsized":"2024-09-21T14:06:58.456282+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'24","last_scrub_stamp":"2024-09-21T14:02:57.559936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:57.559936+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:10.415071+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":432,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.462230+0000","last_change":"2024-09-21T14:04:56.588103+0000","last_active":"2024-09-21T14:06:58.462230+0000","last_peered":"2024-09-21T14:06:58.462230+0000","last_clean":"2024-09-21T14:06:58.462230+0000","last_became_active":"2024-09-21T14:04:56.587384+0000","last_became_peered":"2024-09-21T14:04:56.587384+0000","last_unstale":"2024-09-21T14:06:58.462230+0000","last_undegraded":"2024-09-21T14:06:58.462230+0000","last_fullsized":"2024-09-21T14:06:58.462230+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:59.450546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:59:59.450546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":1814,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:59.964347+0000","last_change":"2024-09-21T14:05:58.405306+0000","last_active":"2024-09-21T14:06:59.964347+0000","last_peered":"2024-09-21T14:06:59.964347+0000","last_clean":"2024-09-21T14:06:59.964347+0000","last_became_active":"2024-09-21T13:59:32.203361+0000","last_became_peered":"2024-09-21T13:59:32.203361+0000","last_unstale":"2024-09-21T14:06:59.964347+0000","last_undegraded":"2024-09-21T14:06:59.964347+0000","last_fullsized":"2024-09-21T14:06:59.964347+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-21T14:05:58.405190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T14:05:58.405190+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:04.336005+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0.00047048000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":455,"num_read_kb":450,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":802,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.456655+0000","last_change":"2024-09-21T14:06:49.591004+0000","last_active":"2024-09-21T14:06:58.456655+0000","last_peered":"2024-09-21T14:06:58.456655+0000","last_clean":"2024-09-21T14:06:58.456655+0000","last_became_active":"2024-09-21T14:01:41.200531+0000","last_became_peered":"2024-09-21T14:01:41.200531+0000","last_unstale":"2024-09-21T14:06:58.456655+0000","last_undegraded":"2024-09-21T14:06:58.456655+0000","last_fullsized":"2024-09-21T14:06:58.456655+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:49.590942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:06:49.590942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:04.361193+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":899,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.464630+0000","last_change":"2024-09-21T14:05:57.451365+0000","last_active":"2024-09-21T14:06:58.464630+0000","last_peered":"2024-09-21T14:06:58.464630+0000","last_clean":"2024-09-21T14:06:58.464630+0000","last_became_active":"2024-09-21T14:01:41.184664+0000","last_became_peered":"2024-09-21T14:01:41.184664+0000","last_unstale":"2024-09-21T14:06:58.464630+0000","last_undegraded":"2024-09-21T14:06:58.464630+0000","last_fullsized":"2024-09-21T14:06:58.464630+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:57.451303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:05:57.451303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:21.808214+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"468'1274","reported_seq":1299,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:07:01.449167+0000","last_change":"2024-09-21T14:06:45.200009+0000","last_active":"2024-09-21T14:07:01.449167+0000","last_peered":"2024-09-21T14:07:01.449167+0000","last_clean":"2024-09-21T14:07:01.449167+0000","last_became_active":"2024-09-21T14:06:45.199722+0000","last_became_peered":"2024-09-21T14:06:45.199722+0000","last_unstale":"2024-09-21T14:07:01.449167+0000","last_undegraded":"2024-09-21T14:07:01.449167+0000","last_fullsized":"2024-09-21T14:07:01.449167+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_clean_scrub_stamp":"2024-09-21T14:06:44.365469+0000","objects_scrubbed":0,"log_size":1274,"log_dups_size":0,"ondisk_log_size":1274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:05.107055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83492864,"num_objects":1274,"num_object_clones":0,"num_object_copies":3822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1274,"num_write_kb":81536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"300'32","reported_seq":589,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.460587+0000","last_change":"2024-09-21T14:00:47.130089+0000","last_active":"2024-09-21T14:06:58.460587+0000","last_peered":"2024-09-21T14:06:58.460587+0000","last_clean":"2024-09-21T14:06:58.460587+0000","last_became_active":"2024-09-21T13:59:32.204624+0000","last_became_peered":"2024-09-21T13:59:32.204624+0000","last_unstale":"2024-09-21T14:06:58.460587+0000","last_undegraded":"2024-09-21T14:06:58.460587+0000","last_fullsized":"2024-09-21T14:06:58.460587+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'32","last_scrub_stamp":"2024-09-21T14:00:47.129789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:00:47.129789+0000","objects_scrubbed":8,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:41.792320+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":853,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.466628+0000","last_change":"2024-09-21T14:06:50.360740+0000","last_active":"2024-09-21T14:06:58.466628+0000","last_peered":"2024-09-21T14:06:58.466628+0000","last_clean":"2024-09-21T14:06:58.466628+0000","last_became_active":"2024-09-21T14:01:41.190210+0000","last_became_peered":"2024-09-21T14:01:41.190210+0000","last_unstale":"2024-09-21T14:06:58.466628+0000","last_undegraded":"2024-09-21T14:06:58.466628+0000","last_fullsized":"2024-09-21T14:06:58.466628+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:50.360693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:06:50.360693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:57.577149+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"75'79","reported_seq":921,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.457602+0000","last_change":"2024-09-21T14:05:47.436025+0000","last_active":"2024-09-21T14:06:58.457602+0000","last_peered":"2024-09-21T14:06:58.457602+0000","last_clean":"2024-09-21T14:06:58.457602+0000","last_became_active":"2024-09-21T14:05:39.304753+0000","last_became_peered":"2024-09-21T14:05:39.304753+0000","last_unstale":"2024-09-21T14:06:58.457602+0000","last_undegraded":"2024-09-21T14:06:58.457602+0000","last_fullsized":"2024-09-21T14:06:58.457602+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:05:47.435952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:05:47.435952+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:08.812648+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":811,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.458592+0000","last_change":"2024-09-21T14:01:41.182186+0000","last_active":"2024-09-21T14:06:58.458592+0000","last_peered":"2024-09-21T14:06:58.458592+0000","last_clean":"2024-09-21T14:06:58.458592+0000","last_became_active":"2024-09-21T14:01:41.181899+0000","last_became_peered":"2024-09-21T14:01:41.181899+0000","last_unstale":"2024-09-21T14:06:58.458592+0000","last_undegraded":"2024-09-21T14:06:58.458592+0000","last_fullsized":"2024-09-21T14:06:58.458592+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:15.700409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:15.700409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:01.384802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":751,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.456492+0000","last_change":"2024-09-21T14:04:01.760941+0000","last_active":"2024-09-21T14:06:58.456492+0000","last_peered":"2024-09-21T14:06:58.456492+0000","last_clean":"2024-09-21T14:06:58.456492+0000","last_became_active":"2024-09-21T14:03:57.346453+0000","last_became_peered":"2024-09-21T14:03:57.346453+0000","last_unstale":"2024-09-21T14:06:58.456492+0000","last_undegraded":"2024-09-21T14:06:58.456492+0000","last_fullsized":"2024-09-21T14:06:58.456492+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:01.760755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:04:01.760755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:37.406663+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"468'1564","reported_seq":1589,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:07:03.890147+0000","last_change":"2024-09-21T14:06:45.309302+0000","last_active":"2024-09-21T14:07:03.890147+0000","last_peered":"2024-09-21T14:07:03.890147+0000","last_clean":"2024-09-21T14:07:03.890147+0000","last_became_active":"2024-09-21T14:06:45.309054+0000","last_became_peered":"2024-09-21T14:06:45.309054+0000","last_unstale":"2024-09-21T14:07:03.890147+0000","last_undegraded":"2024-09-21T14:07:03.890147+0000","last_fullsized":"2024-09-21T14:07:03.890147+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_clean_scrub_stamp":"2024-09-21T14:06:44.365469+0000","objects_scrubbed":0,"log_size":1564,"log_dups_size":0,"ondisk_log_size":1564,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:02.565167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102498304,"num_objects":1564,"num_object_clones":0,"num_object_copies":4692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1564,"num_write_kb":100096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"75'79","reported_seq":346,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.465234+0000","last_change":"2024-09-21T14:05:43.453954+0000","last_active":"2024-09-21T14:06:58.465234+0000","last_peered":"2024-09-21T14:06:58.465234+0000","last_clean":"2024-09-21T14:06:58.465234+0000","last_became_active":"2024-09-21T14:05:41.345392+0000","last_became_peered":"2024-09-21T14:05:41.345392+0000","last_unstale":"2024-09-21T14:06:58.465234+0000","last_undegraded":"2024-09-21T14:06:58.465234+0000","last_fullsized":"2024-09-21T14:06:58.465234+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:05:43.453553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:05:43.453553+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:11.518687+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"290'19","reported_seq":932,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.463137+0000","last_change":"2024-09-21T14:05:48.260237+0000","last_active":"2024-09-21T14:06:58.463137+0000","last_peered":"2024-09-21T14:06:58.463137+0000","last_clean":"2024-09-21T14:06:58.463137+0000","last_became_active":"2024-09-21T13:58:17.776310+0000","last_became_peered":"2024-09-21T13:58:17.776310+0000","last_unstale":"2024-09-21T14:06:58.463137+0000","last_undegraded":"2024-09-21T14:06:58.463137+0000","last_fullsized":"2024-09-21T14:06:58.463137+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'19","last_scrub_stamp":"2024-09-21T14:05:48.260160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:05:48.260160+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":902,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.462870+0000","last_change":"2024-09-21T14:05:59.261563+0000","last_active":"2024-09-21T14:06:58.462870+0000","last_peered":"2024-09-21T14:06:58.462870+0000","last_clean":"2024-09-21T14:06:58.462870+0000","last_became_active":"2024-09-21T13:59:32.223247+0000","last_became_peered":"2024-09-21T13:59:32.223247+0000","last_unstale":"2024-09-21T14:06:58.462870+0000","last_undegraded":"2024-09-21T14:06:58.462870+0000","last_fullsized":"2024-09-21T14:06:58.462870+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:59.261503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:05:59.261503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:05.771739+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":828,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.456372+0000","last_change":"2024-09-21T14:06:24.464033+0000","last_active":"2024-09-21T14:06:58.456372+0000","last_peered":"2024-09-21T14:06:58.456372+0000","last_clean":"2024-09-21T14:06:58.456372+0000","last_became_active":"2024-09-21T14:01:41.193941+0000","last_became_peered":"2024-09-21T14:01:41.193941+0000","last_unstale":"2024-09-21T14:06:58.456372+0000","last_undegraded":"2024-09-21T14:06:58.456372+0000","last_fullsized":"2024-09-21T14:06:58.456372+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:24.463961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:06:24.463961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:53.947232+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":831,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.465613+0000","last_change":"2024-09-21T14:06:30.364931+0000","last_active":"2024-09-21T14:06:58.465613+0000","last_peered":"2024-09-21T14:06:58.465613+0000","last_clean":"2024-09-21T14:06:58.465613+0000","last_became_active":"2024-09-21T14:01:41.190192+0000","last_became_peered":"2024-09-21T14:01:41.190192+0000","last_unstale":"2024-09-21T14:06:58.465613+0000","last_undegraded":"2024-09-21T14:06:58.465613+0000","last_fullsized":"2024-09-21T14:06:58.465613+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:30.364645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:06:30.364645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:55.510384+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"468'1138","reported_seq":1163,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:07:01.468828+0000","last_change":"2024-09-21T14:06:45.309676+0000","last_active":"2024-09-21T14:07:01.468828+0000","last_peered":"2024-09-21T14:07:01.468828+0000","last_clean":"2024-09-21T14:07:01.468828+0000","last_became_active":"2024-09-21T14:06:45.309049+0000","last_became_peered":"2024-09-21T14:06:45.309049+0000","last_unstale":"2024-09-21T14:07:01.468828+0000","last_undegraded":"2024-09-21T14:07:01.468828+0000","last_fullsized":"2024-09-21T14:07:01.468828+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_clean_scrub_stamp":"2024-09-21T14:06:44.365469+0000","objects_scrubbed":0,"log_size":1138,"log_dups_size":0,"ondisk_log_size":1138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:54.166378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74579968,"num_objects":1138,"num_object_clones":0,"num_object_copies":3414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1138,"num_write_kb":72832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"75'79","reported_seq":296,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.455883+0000","last_change":"2024-09-21T14:06:17.355357+0000","last_active":"2024-09-21T14:06:58.455883+0000","last_peered":"2024-09-21T14:06:58.455883+0000","last_clean":"2024-09-21T14:06:58.455883+0000","last_became_active":"2024-09-21T14:01:41.188685+0000","last_became_peered":"2024-09-21T14:01:41.188685+0000","last_unstale":"2024-09-21T14:06:58.455883+0000","last_undegraded":"2024-09-21T14:06:58.455883+0000","last_fullsized":"2024-09-21T14:06:58.455883+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:06:17.355284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:06:17.355284+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:22.633906+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"290'31","reported_seq":708,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:07:02.226348+0000","last_change":"2024-09-21T14:07:02.226348+0000","last_active":"2024-09-21T14:07:02.226348+0000","last_peered":"2024-09-21T14:07:02.226348+0000","last_clean":"2024-09-21T14:07:02.226348+0000","last_became_active":"2024-09-21T14:04:21.204164+0000","last_became_peered":"2024-09-21T14:04:21.204164+0000","last_unstale":"2024-09-21T14:07:02.226348+0000","last_undegraded":"2024-09-21T14:07:02.226348+0000","last_fullsized":"2024-09-21T14:07:02.226348+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'31","last_scrub_stamp":"2024-09-21T14:07:02.225989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:07:02.225989+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:11.028179+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":436,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.460162+0000","last_change":"2024-09-21T14:04:51.139211+0000","last_active":"2024-09-21T14:06:58.460162+0000","last_peered":"2024-09-21T14:06:58.460162+0000","last_clean":"2024-09-21T14:06:58.460162+0000","last_became_active":"2024-09-21T13:58:17.762520+0000","last_became_peered":"2024-09-21T13:58:17.762520+0000","last_unstale":"2024-09-21T14:06:58.460162+0000","last_undegraded":"2024-09-21T14:06:58.460162+0000","last_fullsized":"2024-09-21T14:06:58.460162+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:51.139123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:51.139123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:10.952034+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":841,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.464750+0000","last_change":"2024-09-21T13:58:52.030800+0000","last_active":"2024-09-21T14:06:58.464750+0000","last_peered":"2024-09-21T14:06:58.464750+0000","last_clean":"2024-09-21T14:06:58.464750+0000","last_became_active":"2024-09-21T13:49:02.067472+0000","last_became_peered":"2024-09-21T13:49:02.067472+0000","last_unstale":"2024-09-21T14:06:58.464750+0000","last_undegraded":"2024-09-21T14:06:58.464750+0000","last_fullsized":"2024-09-21T14:06:58.464750+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:52.030718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:58:52.030718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:39.486985+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":723,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.454781+0000","last_change":"2024-09-21T13:57:12.683321+0000","last_active":"2024-09-21T14:06:58.454781+0000","last_peered":"2024-09-21T14:06:58.454781+0000","last_clean":"2024-09-21T14:06:58.454781+0000","last_became_active":"2024-09-21T13:57:12.682979+0000","last_became_peered":"2024-09-21T13:57:12.682979+0000","last_unstale":"2024-09-21T14:06:58.454781+0000","last_undegraded":"2024-09-21T14:06:58.454781+0000","last_fullsized":"2024-09-21T14:06:58.454781+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:53:18.027944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:53:18.027944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:36.953349+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"468'1546","reported_seq":1571,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:07:03.907061+0000","last_change":"2024-09-21T14:06:45.197312+0000","last_active":"2024-09-21T14:07:03.907061+0000","last_peered":"2024-09-21T14:07:03.907061+0000","last_clean":"2024-09-21T14:07:03.907061+0000","last_became_active":"2024-09-21T14:06:45.197043+0000","last_became_peered":"2024-09-21T14:06:45.197043+0000","last_unstale":"2024-09-21T14:07:03.907061+0000","last_undegraded":"2024-09-21T14:07:03.907061+0000","last_fullsized":"2024-09-21T14:07:03.907061+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_clean_scrub_stamp":"2024-09-21T14:06:44.365469+0000","objects_scrubbed":0,"log_size":1546,"log_dups_size":0,"ondisk_log_size":1546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:06.059596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101318656,"num_objects":1546,"num_object_clones":0,"num_object_copies":4638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1546,"num_write_kb":98944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"75'79","reported_seq":913,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.458154+0000","last_change":"2024-09-21T14:02:55.806160+0000","last_active":"2024-09-21T14:06:58.458154+0000","last_peered":"2024-09-21T14:06:58.458154+0000","last_clean":"2024-09-21T14:06:58.458154+0000","last_became_active":"2024-09-21T14:00:41.139315+0000","last_became_peered":"2024-09-21T14:00:41.139315+0000","last_unstale":"2024-09-21T14:06:58.458154+0000","last_undegraded":"2024-09-21T14:06:58.458154+0000","last_fullsized":"2024-09-21T14:06:58.458154+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:02:55.806077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:02:55.806077+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:04.340771+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"290'20","reported_seq":939,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.463379+0000","last_change":"2024-09-21T14:05:15.387788+0000","last_active":"2024-09-21T14:06:58.463379+0000","last_peered":"2024-09-21T14:06:58.463379+0000","last_clean":"2024-09-21T14:06:58.463379+0000","last_became_active":"2024-09-21T13:58:17.765427+0000","last_became_peered":"2024-09-21T13:58:17.765427+0000","last_unstale":"2024-09-21T14:06:58.463379+0000","last_undegraded":"2024-09-21T14:06:58.463379+0000","last_fullsized":"2024-09-21T14:06:58.463379+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'20","last_scrub_stamp":"2024-09-21T14:05:15.387706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:05:15.387706+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:46.186077+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":795,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.464438+0000","last_change":"2024-09-21T14:06:23.286825+0000","last_active":"2024-09-21T14:06:58.464438+0000","last_peered":"2024-09-21T14:06:58.464438+0000","last_clean":"2024-09-21T14:06:58.464438+0000","last_became_active":"2024-09-21T14:04:21.215102+0000","last_became_peered":"2024-09-21T14:04:21.215102+0000","last_unstale":"2024-09-21T14:06:58.464438+0000","last_undegraded":"2024-09-21T14:06:58.464438+0000","last_fullsized":"2024-09-21T14:06:58.464438+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:23.286766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:06:23.286766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:38.569162+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"346'14","reported_seq":969,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.453937+0000","last_change":"2024-09-21T14:06:47.037635+0000","last_active":"2024-09-21T14:06:58.453937+0000","last_peered":"2024-09-21T14:06:58.453937+0000","last_clean":"2024-09-21T14:06:58.453937+0000","last_became_active":"2024-09-21T14:01:41.186564+0000","last_became_peered":"2024-09-21T14:01:41.186564+0000","last_unstale":"2024-09-21T14:06:58.453937+0000","last_undegraded":"2024-09-21T14:06:58.453937+0000","last_fullsized":"2024-09-21T14:06:58.453937+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'14","last_scrub_stamp":"2024-09-21T14:06:47.037583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:06:47.037583+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:49.696423+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"68'1","reported_seq":866,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.462927+0000","last_change":"2024-09-21T14:06:30.258612+0000","last_active":"2024-09-21T14:06:58.462927+0000","last_peered":"2024-09-21T14:06:58.462927+0000","last_clean":"2024-09-21T14:06:58.462927+0000","last_became_active":"2024-09-21T13:52:04.313210+0000","last_became_peered":"2024-09-21T13:52:04.313210+0000","last_unstale":"2024-09-21T14:06:58.462927+0000","last_undegraded":"2024-09-21T14:06:58.462927+0000","last_fullsized":"2024-09-21T14:06:58.462927+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T14:06:30.258539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:06:30.258539+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:42.732350+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.4","version":"468'1191","reported_seq":1216,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:07:01.283856+0000","last_change":"2024-09-21T14:06:45.200466+0000","last_active":"2024-09-21T14:07:01.283856+0000","last_peered":"2024-09-21T14:07:01.283856+0000","last_clean":"2024-09-21T14:07:01.283856+0000","last_became_active":"2024-09-21T14:06:45.200015+0000","last_became_peered":"2024-09-21T14:06:45.200015+0000","last_unstale":"2024-09-21T14:07:01.283856+0000","last_undegraded":"2024-09-21T14:07:01.283856+0000","last_fullsized":"2024-09-21T14:07:01.283856+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_clean_scrub_stamp":"2024-09-21T14:06:44.365469+0000","objects_scrubbed":0,"log_size":1191,"log_dups_size":0,"ondisk_log_size":1191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:50.017549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78053376,"num_objects":1191,"num_object_clones":0,"num_object_copies":3573,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1191,"num_write_kb":76224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.e","version":"75'79","reported_seq":77,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.458169+0000","last_change":"2024-09-21T14:05:57.367735+0000","last_active":"2024-09-21T14:06:58.458169+0000","last_peered":"2024-09-21T14:06:58.458169+0000","last_clean":"2024-09-21T14:06:58.458169+0000","last_became_active":"2024-09-21T14:05:41.349297+0000","last_became_peered":"2024-09-21T14:05:41.349297+0000","last_unstale":"2024-09-21T14:06:58.458169+0000","last_undegraded":"2024-09-21T14:06:58.458169+0000","last_fullsized":"2024-09-21T14:06:58.458169+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:05:57.367668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:05:57.367668+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:00.039266+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"290'15","reported_seq":886,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.462939+0000","last_change":"2024-09-21T13:58:50.498597+0000","last_active":"2024-09-21T14:06:58.462939+0000","last_peered":"2024-09-21T14:06:58.462939+0000","last_clean":"2024-09-21T14:06:58.462939+0000","last_became_active":"2024-09-21T13:49:00.053016+0000","last_became_peered":"2024-09-21T13:49:00.053016+0000","last_unstale":"2024-09-21T14:06:58.462939+0000","last_undegraded":"2024-09-21T14:06:58.462939+0000","last_fullsized":"2024-09-21T14:06:58.462939+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:58:50.498369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:58:50.498369+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:03.361593+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":832,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.456052+0000","last_change":"2024-09-21T14:05:22.402246+0000","last_active":"2024-09-21T14:06:58.456052+0000","last_peered":"2024-09-21T14:06:58.456052+0000","last_clean":"2024-09-21T14:06:58.456052+0000","last_became_active":"2024-09-21T14:01:41.192443+0000","last_became_peered":"2024-09-21T14:01:41.192443+0000","last_unstale":"2024-09-21T14:06:58.456052+0000","last_undegraded":"2024-09-21T14:06:58.456052+0000","last_fullsized":"2024-09-21T14:06:58.456052+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:22.402182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:05:22.402182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":813,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.462271+0000","last_change":"2024-09-21T14:05:55.318522+0000","last_active":"2024-09-21T14:06:58.462271+0000","last_peered":"2024-09-21T14:06:58.462271+0000","last_clean":"2024-09-21T14:06:58.462271+0000","last_became_active":"2024-09-21T13:57:12.680978+0000","last_became_peered":"2024-09-21T13:57:12.680978+0000","last_unstale":"2024-09-21T14:06:58.462271+0000","last_undegraded":"2024-09-21T14:06:58.462271+0000","last_fullsized":"2024-09-21T14:06:58.462271+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:55.318451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:05:55.318451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:06:59.905530+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":753,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.457576+0000","last_change":"2024-09-21T14:04:21.210096+0000","last_active":"2024-09-21T14:06:58.457576+0000","last_peered":"2024-09-21T14:06:58.457576+0000","last_clean":"2024-09-21T14:06:58.457576+0000","last_became_active":"2024-09-21T14:04:21.207534+0000","last_became_peered":"2024-09-21T14:04:21.207534+0000","last_unstale":"2024-09-21T14:06:58.457576+0000","last_undegraded":"2024-09-21T14:06:58.457576+0000","last_fullsized":"2024-09-21T14:06:58.457576+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:13.698091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:13.698091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:02.342745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.5","version":"468'1190","reported_seq":1215,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:07:01.277309+0000","last_change":"2024-09-21T14:06:45.199711+0000","last_active":"2024-09-21T14:07:01.277309+0000","last_peered":"2024-09-21T14:07:01.277309+0000","last_clean":"2024-09-21T14:07:01.277309+0000","last_became_active":"2024-09-21T14:06:45.199355+0000","last_became_peered":"2024-09-21T14:06:45.199355+0000","last_unstale":"2024-09-21T14:07:01.277309+0000","last_undegraded":"2024-09-21T14:07:01.277309+0000","last_fullsized":"2024-09-21T14:07:01.277309+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_clean_scrub_stamp":"2024-09-21T14:06:44.365469+0000","objects_scrubbed":0,"log_size":1190,"log_dups_size":0,"ondisk_log_size":1190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:06.819268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77987840,"num_objects":1190,"num_object_clones":0,"num_object_copies":3570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1190,"num_write_kb":76160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"75'79","reported_seq":936,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:07:01.780758+0000","last_change":"2024-09-21T14:07:01.780758+0000","last_active":"2024-09-21T14:07:01.780758+0000","last_peered":"2024-09-21T14:07:01.780758+0000","last_clean":"2024-09-21T14:07:01.780758+0000","last_became_active":"2024-09-21T14:05:41.327757+0000","last_became_peered":"2024-09-21T14:05:41.327757+0000","last_unstale":"2024-09-21T14:07:01.780758+0000","last_undegraded":"2024-09-21T14:07:01.780758+0000","last_fullsized":"2024-09-21T14:07:01.780758+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:07:01.780652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:07:01.780652+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:24.482569+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"290'35","reported_seq":502,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.460348+0000","last_change":"2024-09-21T14:05:38.897778+0000","last_active":"2024-09-21T14:06:58.460348+0000","last_peered":"2024-09-21T14:06:58.460348+0000","last_clean":"2024-09-21T14:06:58.460348+0000","last_became_active":"2024-09-21T13:59:32.204184+0000","last_became_peered":"2024-09-21T13:59:32.204184+0000","last_unstale":"2024-09-21T14:06:58.460348+0000","last_undegraded":"2024-09-21T14:06:58.460348+0000","last_fullsized":"2024-09-21T14:06:58.460348+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'35","last_scrub_stamp":"2024-09-21T14:05:38.897722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:05:38.897722+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:03.684912+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":854,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.458562+0000","last_change":"2024-09-21T14:06:18.491573+0000","last_active":"2024-09-21T14:06:58.458562+0000","last_peered":"2024-09-21T14:06:58.458562+0000","last_clean":"2024-09-21T14:06:58.458562+0000","last_became_active":"2024-09-21T14:01:41.184567+0000","last_became_peered":"2024-09-21T14:01:41.184567+0000","last_unstale":"2024-09-21T14:06:58.458562+0000","last_undegraded":"2024-09-21T14:06:58.458562+0000","last_fullsized":"2024-09-21T14:06:58.458562+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:18.491509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:06:18.491509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:33.917647+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":825,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.454233+0000","last_change":"2024-09-21T14:04:48.685009+0000","last_active":"2024-09-21T14:06:58.454233+0000","last_peered":"2024-09-21T14:06:58.454233+0000","last_clean":"2024-09-21T14:06:58.454233+0000","last_became_active":"2024-09-21T14:01:41.181911+0000","last_became_peered":"2024-09-21T14:01:41.181911+0000","last_unstale":"2024-09-21T14:06:58.454233+0000","last_undegraded":"2024-09-21T14:06:58.454233+0000","last_fullsized":"2024-09-21T14:06:58.454233+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:48.684941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:04:48.684941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:08.846613+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":781,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.457501+0000","last_change":"2024-09-21T14:06:19.468000+0000","last_active":"2024-09-21T14:06:58.457501+0000","last_peered":"2024-09-21T14:06:58.457501+0000","last_clean":"2024-09-21T14:06:58.457501+0000","last_became_active":"2024-09-21T13:59:32.219332+0000","last_became_peered":"2024-09-21T13:59:32.219332+0000","last_unstale":"2024-09-21T14:06:58.457501+0000","last_undegraded":"2024-09-21T14:06:58.457501+0000","last_fullsized":"2024-09-21T14:06:58.457501+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:19.467928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:06:19.467928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:24.239618+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"468'1474","reported_seq":1499,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:07:03.066178+0000","last_change":"2024-09-21T14:06:45.307813+0000","last_active":"2024-09-21T14:07:03.066178+0000","last_peered":"2024-09-21T14:07:03.066178+0000","last_clean":"2024-09-21T14:07:03.066178+0000","last_became_active":"2024-09-21T14:06:45.307066+0000","last_became_peered":"2024-09-21T14:06:45.307066+0000","last_unstale":"2024-09-21T14:07:03.066178+0000","last_undegraded":"2024-09-21T14:07:03.066178+0000","last_fullsized":"2024-09-21T14:07:03.066178+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_clean_scrub_stamp":"2024-09-21T14:06:44.365469+0000","objects_scrubbed":0,"log_size":1474,"log_dups_size":0,"ondisk_log_size":1474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:01.821180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96600064,"num_objects":1474,"num_object_clones":0,"num_object_copies":4422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1474,"num_write_kb":94336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"75'79","reported_seq":364,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.464976+0000","last_change":"2024-09-21T14:05:48.355758+0000","last_active":"2024-09-21T14:06:58.464976+0000","last_peered":"2024-09-21T14:06:58.464976+0000","last_clean":"2024-09-21T14:06:58.464976+0000","last_became_active":"2024-09-21T14:05:41.340686+0000","last_became_peered":"2024-09-21T14:05:41.340686+0000","last_unstale":"2024-09-21T14:06:58.464976+0000","last_undegraded":"2024-09-21T14:06:58.464976+0000","last_fullsized":"2024-09-21T14:06:58.464976+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:05:48.355659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:05:48.355659+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:13.452284+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"290'18","reported_seq":919,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:07:00.294624+0000","last_change":"2024-09-21T14:07:00.294624+0000","last_active":"2024-09-21T14:07:00.294624+0000","last_peered":"2024-09-21T14:07:00.294624+0000","last_clean":"2024-09-21T14:07:00.294624+0000","last_became_active":"2024-09-21T13:57:12.680627+0000","last_became_peered":"2024-09-21T13:57:12.680627+0000","last_unstale":"2024-09-21T14:07:00.294624+0000","last_undegraded":"2024-09-21T14:07:00.294624+0000","last_fullsized":"2024-09-21T14:07:00.294624+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'18","last_scrub_stamp":"2024-09-21T14:07:00.294535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:07:00.294535+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:19.072614+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":800,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.458563+0000","last_change":"2024-09-21T14:04:21.206034+0000","last_active":"2024-09-21T14:06:58.458563+0000","last_peered":"2024-09-21T14:06:58.458563+0000","last_clean":"2024-09-21T14:06:58.458563+0000","last_became_active":"2024-09-21T14:04:21.205699+0000","last_became_peered":"2024-09-21T14:04:21.205699+0000","last_unstale":"2024-09-21T14:06:58.458563+0000","last_undegraded":"2024-09-21T14:06:58.458563+0000","last_fullsized":"2024-09-21T14:06:58.458563+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:56:21.582325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T13:56:21.582325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:34.501832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":787,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.453313+0000","last_change":"2024-09-21T14:05:50.814761+0000","last_active":"2024-09-21T14:06:58.453313+0000","last_peered":"2024-09-21T14:06:58.453313+0000","last_clean":"2024-09-21T14:06:58.453313+0000","last_became_active":"2024-09-21T13:52:04.322147+0000","last_became_peered":"2024-09-21T13:52:04.322147+0000","last_unstale":"2024-09-21T14:06:58.453313+0000","last_undegraded":"2024-09-21T14:06:58.453313+0000","last_fullsized":"2024-09-21T14:06:58.453313+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:50.814700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:05:50.814700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:07.860682+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":803,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.465000+0000","last_change":"2024-09-21T14:06:21.331650+0000","last_active":"2024-09-21T14:06:58.465000+0000","last_peered":"2024-09-21T14:06:58.465000+0000","last_clean":"2024-09-21T14:06:58.465000+0000","last_became_active":"2024-09-21T14:05:30.243052+0000","last_became_peered":"2024-09-21T14:05:30.243052+0000","last_unstale":"2024-09-21T14:06:58.465000+0000","last_undegraded":"2024-09-21T14:06:58.465000+0000","last_fullsized":"2024-09-21T14:06:58.465000+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:21.331572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:06:21.331572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:50.275757+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"468'1722","reported_seq":1747,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:07:05.988325+0000","last_change":"2024-09-21T14:06:45.307096+0000","last_active":"2024-09-21T14:07:05.988325+0000","last_peered":"2024-09-21T14:07:05.988325+0000","last_clean":"2024-09-21T14:07:05.988325+0000","last_became_active":"2024-09-21T14:06:45.306744+0000","last_became_peered":"2024-09-21T14:06:45.306744+0000","last_unstale":"2024-09-21T14:07:05.988325+0000","last_undegraded":"2024-09-21T14:07:05.988325+0000","last_fullsized":"2024-09-21T14:07:05.988325+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_clean_scrub_stamp":"2024-09-21T14:06:44.365469+0000","objects_scrubbed":0,"log_size":1722,"log_dups_size":0,"ondisk_log_size":1722,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:07.108410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112852992,"num_objects":1722,"num_object_clones":0,"num_object_copies":5166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1722,"num_write_kb":110208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"75'79","reported_seq":77,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.458114+0000","last_change":"2024-09-21T14:05:54.369941+0000","last_active":"2024-09-21T14:06:58.458114+0000","last_peered":"2024-09-21T14:06:58.458114+0000","last_clean":"2024-09-21T14:06:58.458114+0000","last_became_active":"2024-09-21T14:05:41.344754+0000","last_became_peered":"2024-09-21T14:05:41.344754+0000","last_unstale":"2024-09-21T14:06:58.458114+0000","last_undegraded":"2024-09-21T14:06:58.458114+0000","last_fullsized":"2024-09-21T14:06:58.458114+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:05:54.369875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:05:54.369875+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:19.330054+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"290'27","reported_seq":916,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.455635+0000","last_change":"2024-09-21T14:06:46.555735+0000","last_active":"2024-09-21T14:06:58.455635+0000","last_peered":"2024-09-21T14:06:58.455635+0000","last_clean":"2024-09-21T14:06:58.455635+0000","last_became_active":"2024-09-21T14:01:41.192481+0000","last_became_peered":"2024-09-21T14:01:41.192481+0000","last_unstale":"2024-09-21T14:06:58.455635+0000","last_undegraded":"2024-09-21T14:06:58.455635+0000","last_fullsized":"2024-09-21T14:06:58.455635+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'27","last_scrub_stamp":"2024-09-21T14:06:46.555656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:06:46.555656+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:03.289195+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":794,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.458556+0000","last_change":"2024-09-21T14:06:14.644137+0000","last_active":"2024-09-21T14:06:58.458556+0000","last_peered":"2024-09-21T14:06:58.458556+0000","last_clean":"2024-09-21T14:06:58.458556+0000","last_became_active":"2024-09-21T14:04:21.201644+0000","last_became_peered":"2024-09-21T14:04:21.201644+0000","last_unstale":"2024-09-21T14:06:58.458556+0000","last_undegraded":"2024-09-21T14:06:58.458556+0000","last_fullsized":"2024-09-21T14:06:58.458556+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T14:06:14.644055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:06:14.644055+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:20.565507+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"300'14","reported_seq":1029,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.457429+0000","last_change":"2024-09-21T13:59:32.220898+0000","last_active":"2024-09-21T14:06:58.457429+0000","last_peered":"2024-09-21T14:06:58.457429+0000","last_clean":"2024-09-21T14:06:58.457429+0000","last_became_active":"2024-09-21T13:59:32.206956+0000","last_became_peered":"2024-09-21T13:59:32.206956+0000","last_unstale":"2024-09-21T14:06:58.457429+0000","last_undegraded":"2024-09-21T14:06:58.457429+0000","last_fullsized":"2024-09-21T14:06:58.457429+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'11","last_scrub_stamp":"2024-09-21T13:56:31.616003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T13:56:31.616003+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:35.470428+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":856,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.455900+0000","last_change":"2024-09-21T14:06:12.456231+0000","last_active":"2024-09-21T14:06:58.455900+0000","last_peered":"2024-09-21T14:06:58.455900+0000","last_clean":"2024-09-21T14:06:58.455900+0000","last_became_active":"2024-09-21T14:01:41.246106+0000","last_became_peered":"2024-09-21T14:01:41.246106+0000","last_unstale":"2024-09-21T14:06:58.455900+0000","last_undegraded":"2024-09-21T14:06:58.455900+0000","last_fullsized":"2024-09-21T14:06:58.455900+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:12.455977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:06:12.455977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:35.271731+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"75'79","reported_seq":77,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.455719+0000","last_change":"2024-09-21T14:05:44.254364+0000","last_active":"2024-09-21T14:06:58.455719+0000","last_peered":"2024-09-21T14:06:58.455719+0000","last_clean":"2024-09-21T14:06:58.455719+0000","last_became_active":"2024-09-21T14:05:41.352416+0000","last_became_peered":"2024-09-21T14:05:41.352416+0000","last_unstale":"2024-09-21T14:06:58.455719+0000","last_undegraded":"2024-09-21T14:06:58.455719+0000","last_fullsized":"2024-09-21T14:06:58.455719+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":5,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:05:44.254297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:05:44.254297+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:12.352853+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":445,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.459870+0000","last_change":"2024-09-21T14:05:53.922454+0000","last_active":"2024-09-21T14:06:58.459870+0000","last_peered":"2024-09-21T14:06:58.459870+0000","last_clean":"2024-09-21T14:06:58.459870+0000","last_became_active":"2024-09-21T14:04:21.197402+0000","last_became_peered":"2024-09-21T14:04:21.197402+0000","last_unstale":"2024-09-21T14:06:58.459870+0000","last_undegraded":"2024-09-21T14:06:58.459870+0000","last_fullsized":"2024-09-21T14:06:58.459870+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:53.922394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:05:53.922394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:11.574573+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"290'12","reported_seq":815,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.467291+0000","last_change":"2024-09-21T13:59:32.247422+0000","last_active":"2024-09-21T14:06:58.467291+0000","last_peered":"2024-09-21T14:06:58.467291+0000","last_clean":"2024-09-21T14:06:58.467291+0000","last_became_active":"2024-09-21T13:59:32.211306+0000","last_became_peered":"2024-09-21T13:59:32.211306+0000","last_unstale":"2024-09-21T14:06:58.467291+0000","last_undegraded":"2024-09-21T14:06:58.467291+0000","last_fullsized":"2024-09-21T14:06:58.467291+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'6","last_scrub_stamp":"2024-09-21T13:55:12.066873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:55:12.066873+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:00.495495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":788,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.528749+0000","last_change":"2024-09-21T14:06:58.528749+0000","last_active":"2024-09-21T14:06:58.528749+0000","last_peered":"2024-09-21T14:06:58.528749+0000","last_clean":"2024-09-21T14:06:58.528749+0000","last_became_active":"2024-09-21T14:01:41.199321+0000","last_became_peered":"2024-09-21T14:01:41.199321+0000","last_unstale":"2024-09-21T14:06:58.528749+0000","last_undegraded":"2024-09-21T14:06:58.528749+0000","last_fullsized":"2024-09-21T14:06:58.528749+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:58.528683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:06:58.528683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:18.738755+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":819,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.458409+0000","last_change":"2024-09-21T14:03:01.872986+0000","last_active":"2024-09-21T14:06:58.458409+0000","last_peered":"2024-09-21T14:06:58.458409+0000","last_clean":"2024-09-21T14:06:58.458409+0000","last_became_active":"2024-09-21T13:59:32.219525+0000","last_became_peered":"2024-09-21T13:59:32.219525+0000","last_unstale":"2024-09-21T14:06:58.458409+0000","last_undegraded":"2024-09-21T14:06:58.458409+0000","last_fullsized":"2024-09-21T14:06:58.458409+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:01.872928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:01.872928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:07.399094+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"75'79","reported_seq":346,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.463951+0000","last_change":"2024-09-21T14:05:46.414866+0000","last_active":"2024-09-21T14:06:58.463951+0000","last_peered":"2024-09-21T14:06:58.463951+0000","last_clean":"2024-09-21T14:06:58.463951+0000","last_became_active":"2024-09-21T14:05:41.345778+0000","last_became_peered":"2024-09-21T14:05:41.345778+0000","last_unstale":"2024-09-21T14:06:58.463951+0000","last_undegraded":"2024-09-21T14:06:58.463951+0000","last_fullsized":"2024-09-21T14:06:58.463951+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":5,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:05:46.414817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:05:46.414817+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:04.286589+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"290'15","reported_seq":803,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.466697+0000","last_change":"2024-09-21T14:05:55.357872+0000","last_active":"2024-09-21T14:06:58.466697+0000","last_peered":"2024-09-21T14:06:58.466697+0000","last_clean":"2024-09-21T14:06:58.466697+0000","last_became_active":"2024-09-21T14:04:21.208398+0000","last_became_peered":"2024-09-21T14:04:21.208398+0000","last_unstale":"2024-09-21T14:06:58.466697+0000","last_undegraded":"2024-09-21T14:06:58.466697+0000","last_fullsized":"2024-09-21T14:06:58.466697+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'15","last_scrub_stamp":"2024-09-21T14:05:55.357788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:05:55.357788+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:07.895999+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"61'1","reported_seq":837,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.457933+0000","last_change":"2024-09-21T14:04:04.012140+0000","last_active":"2024-09-21T14:06:58.457933+0000","last_peered":"2024-09-21T14:06:58.457933+0000","last_clean":"2024-09-21T14:06:58.457933+0000","last_became_active":"2024-09-21T13:59:32.219515+0000","last_became_peered":"2024-09-21T13:59:32.219515+0000","last_unstale":"2024-09-21T14:06:58.457933+0000","last_undegraded":"2024-09-21T14:06:58.457933+0000","last_fullsized":"2024-09-21T14:06:58.457933+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T14:04:04.012006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:04.012006+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:37.396557+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":779,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.466400+0000","last_change":"2024-09-21T14:06:32.413433+0000","last_active":"2024-09-21T14:06:58.466400+0000","last_peered":"2024-09-21T14:06:58.466400+0000","last_clean":"2024-09-21T14:06:58.466400+0000","last_became_active":"2024-09-21T13:59:32.210586+0000","last_became_peered":"2024-09-21T13:59:32.210586+0000","last_unstale":"2024-09-21T14:06:58.466400+0000","last_undegraded":"2024-09-21T14:06:58.466400+0000","last_fullsized":"2024-09-21T14:06:58.466400+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:32.413388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:06:32.413388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:46.029225+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":829,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.455660+0000","last_change":"2024-09-21T14:05:52.751448+0000","last_active":"2024-09-21T14:06:58.455660+0000","last_peered":"2024-09-21T14:06:58.455660+0000","last_clean":"2024-09-21T14:06:58.455660+0000","last_became_active":"2024-09-21T13:59:32.205803+0000","last_became_peered":"2024-09-21T13:59:32.205803+0000","last_unstale":"2024-09-21T14:06:58.455660+0000","last_undegraded":"2024-09-21T14:06:58.455660+0000","last_fullsized":"2024-09-21T14:06:58.455660+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:52.751387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:05:52.751387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:05.267663+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"75'79","reported_seq":77,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.455454+0000","last_change":"2024-09-21T14:05:42.242477+0000","last_active":"2024-09-21T14:06:58.455454+0000","last_peered":"2024-09-21T14:06:58.455454+0000","last_clean":"2024-09-21T14:06:58.455454+0000","last_became_active":"2024-09-21T14:05:41.325618+0000","last_became_peered":"2024-09-21T14:05:41.325618+0000","last_unstale":"2024-09-21T14:06:58.455454+0000","last_undegraded":"2024-09-21T14:06:58.455454+0000","last_fullsized":"2024-09-21T14:06:58.455454+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":5,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:05:42.242405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:05:42.242405+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"290'15","reported_seq":817,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.456367+0000","last_change":"2024-09-21T14:01:41.189676+0000","last_active":"2024-09-21T14:06:58.456367+0000","last_peered":"2024-09-21T14:06:58.456367+0000","last_clean":"2024-09-21T14:06:58.456367+0000","last_became_active":"2024-09-21T14:01:41.187904+0000","last_became_peered":"2024-09-21T14:01:41.187904+0000","last_unstale":"2024-09-21T14:06:58.456367+0000","last_undegraded":"2024-09-21T14:06:58.456367+0000","last_fullsized":"2024-09-21T14:06:58.456367+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'9","last_scrub_stamp":"2024-09-21T13:57:24.124604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:57:24.124604+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:33.399682+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":461,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.462952+0000","last_change":"2024-09-21T14:04:56.588256+0000","last_active":"2024-09-21T14:06:58.462952+0000","last_peered":"2024-09-21T14:06:58.462952+0000","last_clean":"2024-09-21T14:06:58.462952+0000","last_became_active":"2024-09-21T14:04:56.587953+0000","last_became_peered":"2024-09-21T14:04:56.587953+0000","last_unstale":"2024-09-21T14:06:58.462952+0000","last_undegraded":"2024-09-21T14:06:58.462952+0000","last_fullsized":"2024-09-21T14:06:58.462952+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:37.078458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:37.078458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:07.251801+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"300'14","reported_seq":979,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.465696+0000","last_change":"2024-09-21T14:06:58.253918+0000","last_active":"2024-09-21T14:06:58.465696+0000","last_peered":"2024-09-21T14:06:58.465696+0000","last_clean":"2024-09-21T14:06:58.465696+0000","last_became_active":"2024-09-21T13:59:32.206926+0000","last_became_peered":"2024-09-21T13:59:32.206926+0000","last_unstale":"2024-09-21T14:06:58.465696+0000","last_undegraded":"2024-09-21T14:06:58.465696+0000","last_fullsized":"2024-09-21T14:06:58.465696+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'14","last_scrub_stamp":"2024-09-21T14:06:58.253842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:06:58.253842+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:09.539580+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":881,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.463549+0000","last_change":"2024-09-21T14:06:48.260420+0000","last_active":"2024-09-21T14:06:58.463549+0000","last_peered":"2024-09-21T14:06:58.463549+0000","last_clean":"2024-09-21T14:06:58.463549+0000","last_became_active":"2024-09-21T14:01:41.200113+0000","last_became_peered":"2024-09-21T14:01:41.200113+0000","last_unstale":"2024-09-21T14:06:58.463549+0000","last_undegraded":"2024-09-21T14:06:58.463549+0000","last_fullsized":"2024-09-21T14:06:58.463549+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:48.260355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:06:48.260355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:48.683271+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"75'79","reported_seq":292,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.455485+0000","last_change":"2024-09-21T14:05:43.226598+0000","last_active":"2024-09-21T14:06:58.455485+0000","last_peered":"2024-09-21T14:06:58.455485+0000","last_clean":"2024-09-21T14:06:58.455485+0000","last_became_active":"2024-09-21T14:05:41.344418+0000","last_became_peered":"2024-09-21T14:05:41.344418+0000","last_unstale":"2024-09-21T14:06:58.455485+0000","last_undegraded":"2024-09-21T14:06:58.455485+0000","last_fullsized":"2024-09-21T14:06:58.455485+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":5,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:05:43.226395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:05:43.226395+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"290'20","reported_seq":822,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.465487+0000","last_change":"2024-09-21T14:04:04.192280+0000","last_active":"2024-09-21T14:06:58.465487+0000","last_peered":"2024-09-21T14:06:58.465487+0000","last_clean":"2024-09-21T14:06:58.465487+0000","last_became_active":"2024-09-21T13:59:32.208332+0000","last_became_peered":"2024-09-21T13:59:32.208332+0000","last_unstale":"2024-09-21T14:06:58.465487+0000","last_undegraded":"2024-09-21T14:06:58.465487+0000","last_fullsized":"2024-09-21T14:06:58.465487+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'20","last_scrub_stamp":"2024-09-21T14:04:04.192193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:04:04.192193+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:35.399480+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":813,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.457921+0000","last_change":"2024-09-21T14:05:38.445596+0000","last_active":"2024-09-21T14:06:58.457921+0000","last_peered":"2024-09-21T14:06:58.457921+0000","last_clean":"2024-09-21T14:06:58.457921+0000","last_became_active":"2024-09-21T13:59:32.209804+0000","last_became_peered":"2024-09-21T13:59:32.209804+0000","last_unstale":"2024-09-21T14:06:58.457921+0000","last_undegraded":"2024-09-21T14:06:58.457921+0000","last_fullsized":"2024-09-21T14:06:58.457921+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:38.445529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:05:38.445529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:05.619808+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"346'10","reported_seq":987,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:07:00.541065+0000","last_change":"2024-09-21T14:07:00.540992+0000","last_active":"2024-09-21T14:07:00.541065+0000","last_peered":"2024-09-21T14:07:00.541065+0000","last_clean":"2024-09-21T14:07:00.541065+0000","last_became_active":"2024-09-21T14:01:41.206303+0000","last_became_peered":"2024-09-21T14:01:41.206303+0000","last_unstale":"2024-09-21T14:07:00.541065+0000","last_undegraded":"2024-09-21T14:07:00.541065+0000","last_fullsized":"2024-09-21T14:07:00.541065+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'10","last_scrub_stamp":"2024-09-21T14:07:00.540922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:07:00.540922+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:01.078917+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":725,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.458399+0000","last_change":"2024-09-21T14:06:26.530560+0000","last_active":"2024-09-21T14:06:58.458399+0000","last_peered":"2024-09-21T14:06:58.458399+0000","last_clean":"2024-09-21T14:06:58.458399+0000","last_became_active":"2024-09-21T14:04:21.201277+0000","last_became_peered":"2024-09-21T14:04:21.201277+0000","last_unstale":"2024-09-21T14:06:58.458399+0000","last_undegraded":"2024-09-21T14:06:58.458399+0000","last_fullsized":"2024-09-21T14:06:58.458399+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:26.530508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:06:26.530508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:43.948605+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"290'19","reported_seq":936,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.462367+0000","last_change":"2024-09-21T14:04:05.139089+0000","last_active":"2024-09-21T14:06:58.462367+0000","last_peered":"2024-09-21T14:06:58.462367+0000","last_clean":"2024-09-21T14:06:58.462367+0000","last_became_active":"2024-09-21T13:58:17.771497+0000","last_became_peered":"2024-09-21T13:58:17.771497+0000","last_unstale":"2024-09-21T14:06:58.462367+0000","last_undegraded":"2024-09-21T14:06:58.462367+0000","last_fullsized":"2024-09-21T14:06:58.462367+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'19","last_scrub_stamp":"2024-09-21T14:04:05.138989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:04:05.138989+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:07.188963+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":447,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.458723+0000","last_change":"2024-09-21T14:02:57.807136+0000","last_active":"2024-09-21T14:06:58.458723+0000","last_peered":"2024-09-21T14:06:58.458723+0000","last_clean":"2024-09-21T14:06:58.458723+0000","last_became_active":"2024-09-21T13:59:32.201568+0000","last_became_peered":"2024-09-21T13:59:32.201568+0000","last_unstale":"2024-09-21T14:06:58.458723+0000","last_undegraded":"2024-09-21T14:06:58.458723+0000","last_fullsized":"2024-09-21T14:06:58.458723+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:57.807059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:02:57.807059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:06.379506+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":894,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.463843+0000","last_change":"2024-09-21T14:06:51.607443+0000","last_active":"2024-09-21T14:06:58.463843+0000","last_peered":"2024-09-21T14:06:58.463843+0000","last_clean":"2024-09-21T14:06:58.463843+0000","last_became_active":"2024-09-21T14:01:41.185293+0000","last_became_peered":"2024-09-21T14:01:41.185293+0000","last_unstale":"2024-09-21T14:06:58.463843+0000","last_undegraded":"2024-09-21T14:06:58.463843+0000","last_fullsized":"2024-09-21T14:06:58.463843+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:51.607390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:06:51.607390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:20.636041+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":794,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.462280+0000","last_change":"2024-09-21T14:06:27.243814+0000","last_active":"2024-09-21T14:06:58.462280+0000","last_peered":"2024-09-21T14:06:58.462280+0000","last_clean":"2024-09-21T14:06:58.462280+0000","last_became_active":"2024-09-21T14:04:21.208329+0000","last_became_peered":"2024-09-21T14:04:21.208329+0000","last_unstale":"2024-09-21T14:06:58.462280+0000","last_undegraded":"2024-09-21T14:06:58.462280+0000","last_fullsized":"2024-09-21T14:06:58.462280+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:27.243740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:06:27.243740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:31.206095+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"290'15","reported_seq":946,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.464008+0000","last_change":"2024-09-21T14:06:13.312821+0000","last_active":"2024-09-21T14:06:58.464008+0000","last_peered":"2024-09-21T14:06:58.464008+0000","last_clean":"2024-09-21T14:06:58.464008+0000","last_became_active":"2024-09-21T13:58:17.764120+0000","last_became_peered":"2024-09-21T13:58:17.764120+0000","last_unstale":"2024-09-21T14:06:58.464008+0000","last_undegraded":"2024-09-21T14:06:58.464008+0000","last_fullsized":"2024-09-21T14:06:58.464008+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'15","last_scrub_stamp":"2024-09-21T14:06:13.312732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:06:13.312732+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:30.948235+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":817,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.458223+0000","last_change":"2024-09-21T14:04:07.038058+0000","last_active":"2024-09-21T14:06:58.458223+0000","last_peered":"2024-09-21T14:06:58.458223+0000","last_clean":"2024-09-21T14:06:58.458223+0000","last_became_active":"2024-09-21T13:59:32.221854+0000","last_became_peered":"2024-09-21T13:59:32.221854+0000","last_unstale":"2024-09-21T14:06:58.458223+0000","last_undegraded":"2024-09-21T14:06:58.458223+0000","last_fullsized":"2024-09-21T14:06:58.458223+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:07.037989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:07.037989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:38.436141+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":754,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.463621+0000","last_change":"2024-09-21T14:03:32.262147+0000","last_active":"2024-09-21T14:06:58.463621+0000","last_peered":"2024-09-21T14:06:58.463621+0000","last_clean":"2024-09-21T14:06:58.463621+0000","last_became_active":"2024-09-21T13:57:12.703441+0000","last_became_peered":"2024-09-21T13:57:12.703441+0000","last_unstale":"2024-09-21T14:06:58.463621+0000","last_undegraded":"2024-09-21T14:06:58.463621+0000","last_fullsized":"2024-09-21T14:06:58.463621+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:32.262067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:03:32.262067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:33.379100+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":525,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.461888+0000","last_change":"2024-09-21T14:04:56.588351+0000","last_active":"2024-09-21T14:06:58.461888+0000","last_peered":"2024-09-21T14:06:58.461888+0000","last_clean":"2024-09-21T14:06:58.461888+0000","last_became_active":"2024-09-21T14:04:56.587503+0000","last_became_peered":"2024-09-21T14:04:56.587503+0000","last_unstale":"2024-09-21T14:06:58.461888+0000","last_undegraded":"2024-09-21T14:06:58.461888+0000","last_fullsized":"2024-09-21T14:06:58.461888+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:29.345427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:29.345427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:06.294143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"75'79","reported_seq":78,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.455561+0000","last_change":"2024-09-21T14:05:44.794897+0000","last_active":"2024-09-21T14:06:58.455561+0000","last_peered":"2024-09-21T14:06:58.455561+0000","last_clean":"2024-09-21T14:06:58.455561+0000","last_became_active":"2024-09-21T14:05:41.358002+0000","last_became_peered":"2024-09-21T14:05:41.358002+0000","last_unstale":"2024-09-21T14:06:58.455561+0000","last_undegraded":"2024-09-21T14:06:58.455561+0000","last_fullsized":"2024-09-21T14:06:58.455561+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":5,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:05:44.794826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:05:44.794826+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:05.143030+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"290'19","reported_seq":898,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.461942+0000","last_change":"2024-09-21T14:05:54.360856+0000","last_active":"2024-09-21T14:06:58.461942+0000","last_peered":"2024-09-21T14:06:58.461942+0000","last_clean":"2024-09-21T14:06:58.461942+0000","last_became_active":"2024-09-21T13:59:32.231567+0000","last_became_peered":"2024-09-21T13:59:32.231567+0000","last_unstale":"2024-09-21T14:06:58.461942+0000","last_undegraded":"2024-09-21T14:06:58.461942+0000","last_fullsized":"2024-09-21T14:06:58.461942+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'19","last_scrub_stamp":"2024-09-21T14:05:54.360786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:05:54.360786+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:01.151650+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":785,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.456085+0000","last_change":"2024-09-21T14:05:13.355232+0000","last_active":"2024-09-21T14:06:58.456085+0000","last_peered":"2024-09-21T14:06:58.456085+0000","last_clean":"2024-09-21T14:06:58.456085+0000","last_became_active":"2024-09-21T14:01:41.190760+0000","last_became_peered":"2024-09-21T14:01:41.190760+0000","last_unstale":"2024-09-21T14:06:58.456085+0000","last_undegraded":"2024-09-21T14:06:58.456085+0000","last_fullsized":"2024-09-21T14:06:58.456085+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:13.355183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:05:13.355183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":831,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.457601+0000","last_change":"2024-09-21T14:06:49.450760+0000","last_active":"2024-09-21T14:06:58.457601+0000","last_peered":"2024-09-21T14:06:58.457601+0000","last_clean":"2024-09-21T14:06:58.457601+0000","last_became_active":"2024-09-21T13:59:32.214004+0000","last_became_peered":"2024-09-21T13:59:32.214004+0000","last_unstale":"2024-09-21T14:06:58.457601+0000","last_undegraded":"2024-09-21T14:06:58.457601+0000","last_fullsized":"2024-09-21T14:06:58.457601+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:49.450699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:06:49.450699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:01.179394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":819,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.464483+0000","last_change":"2024-09-21T14:04:21.214685+0000","last_active":"2024-09-21T14:06:58.464483+0000","last_peered":"2024-09-21T14:06:58.464483+0000","last_clean":"2024-09-21T14:06:58.464483+0000","last_became_active":"2024-09-21T14:04:21.213307+0000","last_became_peered":"2024-09-21T14:04:21.213307+0000","last_unstale":"2024-09-21T14:06:58.464483+0000","last_undegraded":"2024-09-21T14:06:58.464483+0000","last_fullsized":"2024-09-21T14:06:58.464483+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:28.124296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:28.124296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:05.443644+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"290'8","reported_seq":836,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.462064+0000","last_change":"2024-09-21T13:59:26.034457+0000","last_active":"2024-09-21T14:06:58.462064+0000","last_peered":"2024-09-21T14:06:58.462064+0000","last_clean":"2024-09-21T14:06:58.462064+0000","last_became_active":"2024-09-21T13:58:17.775079+0000","last_became_peered":"2024-09-21T13:58:17.775079+0000","last_unstale":"2024-09-21T14:06:58.462064+0000","last_undegraded":"2024-09-21T14:06:58.462064+0000","last_fullsized":"2024-09-21T14:06:58.462064+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'8","last_scrub_stamp":"2024-09-21T13:59:26.034372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T13:59:26.034372+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:09.460453+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":612,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.462479+0000","last_change":"2024-09-21T14:06:54.291008+0000","last_active":"2024-09-21T14:06:58.462479+0000","last_peered":"2024-09-21T14:06:58.462479+0000","last_clean":"2024-09-21T14:06:58.462479+0000","last_became_active":"2024-09-21T14:04:56.587555+0000","last_became_peered":"2024-09-21T14:04:56.587555+0000","last_unstale":"2024-09-21T14:06:58.462479+0000","last_undegraded":"2024-09-21T14:06:58.462479+0000","last_fullsized":"2024-09-21T14:06:58.462479+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:54.290961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:06:54.290961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:02.798065+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":461,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.460056+0000","last_change":"2024-09-21T14:06:12.938710+0000","last_active":"2024-09-21T14:06:58.460056+0000","last_peered":"2024-09-21T14:06:58.460056+0000","last_clean":"2024-09-21T14:06:58.460056+0000","last_became_active":"2024-09-21T14:04:21.204954+0000","last_became_peered":"2024-09-21T14:04:21.204954+0000","last_unstale":"2024-09-21T14:06:58.460056+0000","last_undegraded":"2024-09-21T14:06:58.460056+0000","last_fullsized":"2024-09-21T14:06:58.460056+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:12.938632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:06:12.938632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:18.519522+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":756,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.458033+0000","last_change":"2024-09-21T14:04:15.135536+0000","last_active":"2024-09-21T14:06:58.458033+0000","last_peered":"2024-09-21T14:06:58.458033+0000","last_clean":"2024-09-21T14:06:58.458033+0000","last_became_active":"2024-09-21T13:59:32.210134+0000","last_became_peered":"2024-09-21T13:59:32.210134+0000","last_unstale":"2024-09-21T14:06:58.458033+0000","last_undegraded":"2024-09-21T14:06:58.458033+0000","last_fullsized":"2024-09-21T14:06:58.458033+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:15.135463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:04:15.135463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:41.430879+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":863,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.464536+0000","last_change":"2024-09-21T14:01:41.187792+0000","last_active":"2024-09-21T14:06:58.464536+0000","last_peered":"2024-09-21T14:06:58.464536+0000","last_clean":"2024-09-21T14:06:58.464536+0000","last_became_active":"2024-09-21T14:01:41.187438+0000","last_became_peered":"2024-09-21T14:01:41.187438+0000","last_unstale":"2024-09-21T14:06:58.464536+0000","last_undegraded":"2024-09-21T14:06:58.464536+0000","last_fullsized":"2024-09-21T14:06:58.464536+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:58:17.706325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:58:17.706325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:34.451704+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":815,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.462163+0000","last_change":"2024-09-21T14:06:47.440308+0000","last_active":"2024-09-21T14:06:58.462163+0000","last_peered":"2024-09-21T14:06:58.462163+0000","last_clean":"2024-09-21T14:06:58.462163+0000","last_became_active":"2024-09-21T14:04:21.208121+0000","last_became_peered":"2024-09-21T14:04:21.208121+0000","last_unstale":"2024-09-21T14:06:58.462163+0000","last_undegraded":"2024-09-21T14:06:58.462163+0000","last_fullsized":"2024-09-21T14:06:58.462163+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:47.440233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:06:47.440233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:13.873794+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"290'19","reported_seq":479,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.460632+0000","last_change":"2024-09-21T14:05:55.994009+0000","last_active":"2024-09-21T14:06:58.460632+0000","last_peered":"2024-09-21T14:06:58.460632+0000","last_clean":"2024-09-21T14:06:58.460632+0000","last_became_active":"2024-09-21T14:01:41.194606+0000","last_became_peered":"2024-09-21T14:01:41.194606+0000","last_unstale":"2024-09-21T14:06:58.460632+0000","last_undegraded":"2024-09-21T14:06:58.460632+0000","last_fullsized":"2024-09-21T14:06:58.460632+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'19","last_scrub_stamp":"2024-09-21T14:05:55.993943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:05:55.993943+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:10.350997+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":481,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-21T14:06:58.462555+0000","last_change":"2024-09-21T14:04:56.591137+0000","last_active":"2024-09-21T14:06:58.462555+0000","last_peered":"2024-09-21T14:06:58.462555+0000","last_clean":"2024-09-21T14:06:58.462555+0000","last_became_active":"2024-09-21T14:04:56.590829+0000","last_became_peered":"2024-09-21T14:04:56.590829+0000","last_unstale":"2024-09-21T14:06:58.462555+0000","last_undegraded":"2024-09-21T14:06:58.462555+0000","last_fullsized":"2024-09-21T14:06:58.462555+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:51.121361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:04:51.121361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:07:08.256194+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":1471086592,"num_objects":22447,"num_object_clones":0,"num_object_copies":67341,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22447,"num_write_kb":1436608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4695916544,"data_stored":4695916544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22447,"ondisk_log_size":22447,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102,"ondisk_log_size":102,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1107,"num_read_kb":864,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":197,"num_bytes_recovered":2794,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":687,"ondisk_log_size":687,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":455,"num_read_kb":450,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1659,"ondisk_log_size":1659,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":298,"seq":1279900254299,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":919472,"kb_used_data":430944,"kb_used_omap":4,"kb_used_meta":488507,"kb_avail":92825680,"statfs":{"total":95995035648,"available":95053496320,"internally_reserved":0,"allocated":441286656,"data_stored":440823225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":500231848},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463139,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1296724,"kb_used_data":782404,"kb_used_omap":1,"kb_used_meta":514302,"kb_avail":92448428,"statfs":{"total":95995035648,"available":94667190272,"internally_reserved":0,"allocated":801181696,"data_stored":800745632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":526645702},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":345,"seq":1481763717186,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":657172,"kb_used_data":440072,"kb_used_omap":4,"kb_used_meta":217083,"kb_avail":93087980,"statfs":{"total":95995035648,"available":95322091520,"internally_reserved":0,"allocated":450633728,"data_stored":450194982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":222293667},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921008,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1060572,"kb_used_data":493516,"kb_used_omap":1,"kb_used_meta":567038,"kb_avail":92684580,"statfs":{"total":95995035648,"available":94909009920,"internally_reserved":0,"allocated":505360384,"data_stored":504916214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":580647365},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":240,"seq":1030792151159,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":997044,"kb_used_data":530212,"kb_used_omap":4,"kb_used_meta":466811,"kb_avail":92748108,"statfs":{"total":95995035648,"available":94974062592,"internally_reserved":0,"allocated":542937088,"data_stored":542470092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4443,"internal_metadata":478015141},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":130,"seq":558345748661,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1254684,"kb_used_data":506124,"kb_used_omap":4,"kb_used_meta":748539,"kb_avail":92490468,"statfs":{"total":95995035648,"available":94710239232,"internally_reserved":0,"allocated":518270976,"data_stored":517828156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4441,"internal_metadata":766504615},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575106,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1654808,"kb_used_data":951688,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":92090344,"statfs":{"total":95995035648,"available":94300512256,"internally_reserved":0,"allocated":974528512,"data_stored":974024224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738632,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1467900,"kb_used_data":746732,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":92277252,"statfs":{"total":95995035648,"available":94491906048,"internally_reserved":0,"allocated":764653568,"data_stored":764241484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":2448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":425132032,"data_stored":425132032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":985333760,"data_stored":985333760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":568459264,"data_stored":568459264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":555745280,"data_stored":555745280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":402718720,"data_stored":402718720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":467664896,"data_stored":467664896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":810352640,"data_stored":810352640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":480509952,"data_stored":480509952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T14:07:09.837 INFO:tasks.thrashosds.thrasher:Setting 5.16 to [0, 7, 1, 6, 3, 5] 2024-09-21T14:07:09.837 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '5.16', '0', '7', '1', '6', '3', '5'] 2024-09-21T14:07:09.837 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd pg-upmap-items 5.16 0 7 1 6 3 5 2024-09-21T14:07:09.846 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:09.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:09.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:07:09.948 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:07:09.975 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:09.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:09.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:07:10.077 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:07:10.106 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:10.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:10.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:07:10.208 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:07:10.236 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:10.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:10.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:07:10.311 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:07:10.338 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:07:10.380 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:10.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:10.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:07:10.444 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:07:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:07:10.481 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:07:10.506 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 17 16 36514 36498 134.163 97 0.00508059 0.00744758 2024-09-21T14:07:10.511 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:10.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:10.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:07:10.569 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T14:07:10.612 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:07:10.639 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:10.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:10.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:07:10.740 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:07:10.755 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:07:10 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:07:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:07:10.773 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:10.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:10.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:07:10.832 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T14:07:10.875 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:07:10.899 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:10.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:10.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:07:11.001 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:07:11.064 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:11.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:11.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:07:11.109 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:07:11.166 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:07:11.205 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:11 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:07:11.205 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:11 smithi049 ceph-mon[30080]: pgmap v929: 168 pgs: 168 active+clean; 1.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 127 MiB/s wr, 2.03k op/s 2024-09-21T14:07:11.205 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:11 smithi049 ceph-mon[30080]: osdmap e470: 8 total, 8 up, 8 in 2024-09-21T14:07:11.217 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T14:07:11.226 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:11.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:11.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:07:11.327 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:11.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:11.383 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:11.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:11.484 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:07:11.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:11 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:07:11.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:11 smithi049 ceph-mon[40489]: pgmap v929: 168 pgs: 168 active+clean; 1.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 127 MiB/s wr, 2.03k op/s 2024-09-21T14:07:11.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:11 smithi049 ceph-mon[40489]: osdmap e470: 8 total, 8 up, 8 in 2024-09-21T14:07:11.506 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 18 15 37858 37843 131.379 84.0625 0.0095079 0.00760559 2024-09-21T14:07:11.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:11.523 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:11.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:07:11.624 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:07:11.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:11.656 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:11.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:07:11.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:11 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:07:11.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:11 smithi187 ceph-mon[33032]: pgmap v929: 168 pgs: 168 active+clean; 1.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 127 MiB/s wr, 2.03k op/s 2024-09-21T14:07:11.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:11 smithi187 ceph-mon[33032]: osdmap e470: 8 total, 8 up, 8 in 2024-09-21T14:07:11.758 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:07:11.789 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:11.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:11.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:07:11.891 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:11.930 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:11.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:11.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:12.032 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:07:12.071 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:12.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:12.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:07:12.174 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:07:12.211 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:12.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:12.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:07:12.228 INFO:teuthology.orchestra.run.smithi049.stderr:set 5.16 pg_upmap_items mapping to [0->7,1->6,3->5] 2024-09-21T14:07:12.313 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:07:12.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:12.342 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:12.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:07:12.442 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:07:12.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:12 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1393625062' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "5.16", "id": [0, 7, 1, 6, 3, 5]} : dispatch 2024-09-21T14:07:12.485 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:12.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:12.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:07:12.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:12 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1393625062' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "5.16", "id": [0, 7, 1, 6, 3, 5]} : dispatch 2024-09-21T14:07:12.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:12 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1393625062' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "5.16", "id": [0, 7, 1, 6, 3, 5]} : dispatch 2024-09-21T14:07:12.506 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 19 16 39512 39496 129.901 103.312 0.00442867 0.0076928 2024-09-21T14:07:12.586 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:07:12.624 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:12.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:12.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:07:12.725 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:07:12.764 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:12.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:12.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:07:12.866 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:07:12.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:12.906 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:12.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:07:13.006 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:07:13.037 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:13.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:13.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:07:13.139 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:07:13.172 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:13.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:13.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:07:13.273 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:07:13.301 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:13.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:13.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:07:13.403 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:07:13.427 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:13.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:13.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:07:13.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:13 smithi049 ceph-mon[40489]: pgmap v931: 168 pgs: 168 active+clean; 2.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 126 MiB/s wr, 2.02k op/s 2024-09-21T14:07:13.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:13 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1393625062' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "5.16", "id": [0, 7, 1, 6, 3, 5]}]': finished 2024-09-21T14:07:13.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:13 smithi049 ceph-mon[40489]: osdmap e471: 8 total, 8 up, 8 in 2024-09-21T14:07:13.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:13 smithi049 ceph-mon[30080]: pgmap v931: 168 pgs: 168 active+clean; 2.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 126 MiB/s wr, 2.02k op/s 2024-09-21T14:07:13.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:13 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1393625062' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "5.16", "id": [0, 7, 1, 6, 3, 5]}]': finished 2024-09-21T14:07:13.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:13 smithi049 ceph-mon[30080]: osdmap e471: 8 total, 8 up, 8 in 2024-09-21T14:07:13.506 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:2024-09-21T14:07:13.507665+0000 min lat: 0.00119458 max lat: 0.116551 avg lat: 0.00777019 2024-09-21T14:07:13.506 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T14:07:13.506 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 20 16 41178 41162 128.612 104.125 0.0116159 0.00777019 2024-09-21T14:07:13.529 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:07:13.564 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:13.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:13.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:07:13.634 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:13 smithi187 ceph-mon[33032]: pgmap v931: 168 pgs: 168 active+clean; 2.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 126 MiB/s wr, 2.02k op/s 2024-09-21T14:07:13.634 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:13 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1393625062' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "5.16", "id": [0, 7, 1, 6, 3, 5]}]': finished 2024-09-21T14:07:13.634 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:13 smithi187 ceph-mon[33032]: osdmap e471: 8 total, 8 up, 8 in 2024-09-21T14:07:13.665 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:07:13.693 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:13.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:13.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:07:13.795 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:07:13.825 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:13.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:13.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:07:13.919 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T14:07:13.919 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:07:13.927 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:07:13.960 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:13.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:13.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:07:14.061 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:07:14.104 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:14.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:14.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:07:14.206 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:07:14.234 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:14.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:14.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:07:14.335 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:07:14.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:14.365 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:14.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:07:14.466 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:07:14.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:14 smithi049 ceph-mon[30080]: osdmap e472: 8 total, 8 up, 8 in 2024-09-21T14:07:14.504 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:14.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:14 smithi049 ceph-mon[40489]: osdmap e472: 8 total, 8 up, 8 in 2024-09-21T14:07:14.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:14.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:07:14.506 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 21 16 42872 42856 127.529 105.875 0.0116106 0.00783551 2024-09-21T14:07:14.606 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:07:14.644 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:14.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:14.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:07:14.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:14 smithi187 ceph-mon[33032]: osdmap e472: 8 total, 8 up, 8 in 2024-09-21T14:07:14.745 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:07:14.771 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:14.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:14.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:07:14.775 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T14:07:14.873 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:07:14.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:14.916 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:14.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:07:15.017 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:07:15.045 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:15.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:15.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:07:15.147 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:15.193 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:15.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:15.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:15.295 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:07:15.337 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:15.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:15.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:07:15.439 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:07:15.468 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:15.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:15.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:07:15.506 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 22 15 44504 44489 126.371 102.062 0.00503691 0.00790688 2024-09-21T14:07:15.570 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:07:15.603 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:07:15.618 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:07:15.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:15.651 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:15.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:07:15.750 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:15 smithi049 ceph-mon[40489]: pgmap v934: 168 pgs: 168 active+clean; 2.3 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 110 MiB/s wr, 1.77k op/s 2024-09-21T14:07:15.750 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:15 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:07:15.750 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:15 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:07:15.750 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:15 smithi049 ceph-mon[30080]: pgmap v934: 168 pgs: 168 active+clean; 2.3 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 110 MiB/s wr, 1.77k op/s 2024-09-21T14:07:15.750 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:15 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:07:15.750 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:15 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:07:15.753 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:07:15.797 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:15.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:15.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:07:15.899 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:07:15.930 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:15.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:15.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:07:15.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:15 smithi187 ceph-mon[33032]: pgmap v934: 168 pgs: 168 active+clean; 2.3 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 110 MiB/s wr, 1.77k op/s 2024-09-21T14:07:15.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:15 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:07:15.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:15 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:07:16.031 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:07:16.032 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:07:16.058 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:16.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:16.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:07:16.159 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:07:16.191 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:16.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:16.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:07:16.292 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:07:16.330 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:16.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:16.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:07:16.432 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:07:16.459 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:16.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:16.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:07:16.506 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 23 15 46124 46109 125.279 101.25 0.015153 0.00797668 2024-09-21T14:07:16.561 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:07:16.594 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:16.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:16.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:07:16.696 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:07:16.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:16.741 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:16.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:07:16.842 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:07:16.879 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:16.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:16.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:07:16.982 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:17.029 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:17.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:17.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:17.131 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:07:17.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:17.171 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:17.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:07:17.272 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:07:17.306 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:17.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:17.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:07:17.408 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:07:17.435 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:17.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:17.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:07:17.506 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 24 16 47613 47597 123.933 93 0.0154809 0.00806313 2024-09-21T14:07:17.537 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:07:17.578 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:17.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:17.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:07:17.680 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:07:17.713 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:17.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:17.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:07:17.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:17 smithi049 ceph-mon[30080]: pgmap v935: 168 pgs: 168 active+clean; 2.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 90 MiB/s wr, 1.44k op/s 2024-09-21T14:07:17.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:17 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:07:17.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:17 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:07:17.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:17 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1310948352' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:07:17.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:17 smithi049 ceph-mon[40489]: pgmap v935: 168 pgs: 168 active+clean; 2.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 90 MiB/s wr, 1.44k op/s 2024-09-21T14:07:17.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:17 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:07:17.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:17 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:07:17.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:17 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1310948352' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:07:17.814 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:07:17.856 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:17.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:17.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:07:17.936 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:07:17.937 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:07:17.937 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:07:17.937 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:07:17.937 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:07:17.937 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:07:17.937 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:07:17.937 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:07:17.937 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:07:17.938 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:07:17.938 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:07:17.938 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:07:17.938 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:07:17.938 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:07:17.938 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:07:17.938 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:07:17.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:17 smithi187 ceph-mon[33032]: pgmap v935: 168 pgs: 168 active+clean; 2.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 90 MiB/s wr, 1.44k op/s 2024-09-21T14:07:17.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:17 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:07:17.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:17 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:07:17.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:17 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1310948352' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:07:17.958 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:07:17.997 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:17.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:17.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:07:18.100 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:07:18.128 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:18.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:18.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:07:18.230 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:07:18.272 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:18.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:18.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:07:18.374 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:07:18.404 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:18.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:18.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:07:18.505 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-09-21T14:07:18.506 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:07:18.507 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 25 16 49236 49220 123.033 101.438 0.017965 0.00811995 2024-09-21T14:07:18.535 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:18.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:18.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:07:18.636 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:18 smithi049 ceph-mon[40489]: osdmap e473: 8 total, 8 up, 8 in 2024-09-21T14:07:18.636 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:18 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1299832559' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:07:18.636 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:18 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:07:18.636 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:18 smithi049 ceph-mon[30080]: osdmap e473: 8 total, 8 up, 8 in 2024-09-21T14:07:18.636 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:18 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1299832559' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:07:18.636 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:18 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:07:18.638 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:07:18.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:18.686 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:18.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:07:18.787 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:07:18.818 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:18.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:18.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:07:18.920 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:18.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:18 smithi187 ceph-mon[33032]: osdmap e473: 8 total, 8 up, 8 in 2024-09-21T14:07:18.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:18 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1299832559' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:07:18.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:18 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:07:18.960 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:18.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:18.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:18.979 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T14:07:18.979 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-09-21T14:07:18.979 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-09-21T14:07:18.979 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-09-21T14:07:18.979 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T14:07:19.062 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:07:19.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:19.096 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:19.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:07:19.197 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:07:19.239 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:19.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:19.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:07:19.344 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:07:19.375 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:19.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:19.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:07:19.477 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:07:19.506 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:19.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:19.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:07:19.507 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 26 15 50739 50724 121.916 94 0.0131727 0.00819611 2024-09-21T14:07:19.608 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:19.649 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:19.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:19.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:19.751 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:07:19.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:19 smithi049 ceph-mon[30080]: pgmap v937: 168 pgs: 168 active+clean; 2.7 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 85 MiB/s wr, 1.36k op/s 2024-09-21T14:07:19.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:19 smithi049 ceph-mon[30080]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:07:19.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:19 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T14:07:19.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:19 smithi049 ceph-mon[30080]: osdmap e474: 8 total, 8 up, 8 in 2024-09-21T14:07:19.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:19 smithi049 ceph-mon[40489]: pgmap v937: 168 pgs: 168 active+clean; 2.7 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 85 MiB/s wr, 1.36k op/s 2024-09-21T14:07:19.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:19 smithi049 ceph-mon[40489]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:07:19.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:19 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T14:07:19.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:19 smithi049 ceph-mon[40489]: osdmap e474: 8 total, 8 up, 8 in 2024-09-21T14:07:19.784 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:19.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:19.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:07:19.886 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:07:19.923 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:19.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:19.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:07:19.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:19 smithi187 ceph-mon[33032]: pgmap v937: 168 pgs: 168 active+clean; 2.7 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 85 MiB/s wr, 1.36k op/s 2024-09-21T14:07:19.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:19 smithi187 ceph-mon[33032]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:07:19.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:19 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T14:07:19.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:19 smithi187 ceph-mon[33032]: osdmap e474: 8 total, 8 up, 8 in 2024-09-21T14:07:20.024 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:20.066 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:20.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:20.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:20.168 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:07:20.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:20.210 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:20.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:07:20.310 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:07:20.355 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:20.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:20.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:07:20.444 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:07:20 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:07:20.457 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:07:20.489 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:20.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:20.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:07:20.507 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 27 16 52308 52292 121.03 98 0.0207633 0.00825644 2024-09-21T14:07:20.590 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:07:20.593 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 4, 6, 2, 3, 7, 5] 2024-09-21T14:07:20.593 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T14:07:20.593 INFO:tasks.thrashosds.thrasher:Setting full ratio to .001 2024-09-21T14:07:20.593 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set-full-ratio .001 2024-09-21T14:07:20.633 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:20.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:20.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:07:20.734 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:20.782 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:20.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:20.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:20.835 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:07:20 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:07:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:07:20.885 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:07:20.895 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T14:07:20.915 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:20.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:20.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:07:21.016 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:07:21.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:21.066 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:21.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:07:21.166 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:21.206 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:21.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:21.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:21.308 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:07:21.339 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:21.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:21.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:07:21.441 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:21.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:21.507 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:21.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:21.511 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 28 16 53705 53689 119.823 87.3125 0.00726862 0.00833907 2024-09-21T14:07:21.562 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:07:21.579 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:07:21.608 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:07:21.661 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:21.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:21.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:07:21.711 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:21 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:07:21.711 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:21 smithi049 ceph-mon[40489]: pgmap v939: 168 pgs: 168 active+clean; 2.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 743 B/s rd, 97 MiB/s wr, 1.55k op/s 2024-09-21T14:07:21.711 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:21 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:07:21.711 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:21 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:07:21.711 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:21 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:07:21.711 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:21 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:07:21.711 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:21 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:07:21.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:21 smithi049 ceph-mon[30080]: pgmap v939: 168 pgs: 168 active+clean; 2.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 743 B/s rd, 97 MiB/s wr, 1.55k op/s 2024-09-21T14:07:21.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:21 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:07:21.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:21 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:07:21.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:21 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:07:21.712 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:21 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:07:21.763 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:07:21.806 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:21.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:21.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:07:21.908 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:07:21.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:21 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:07:21.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:21 smithi187 ceph-mon[33032]: pgmap v939: 168 pgs: 168 active+clean; 2.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 743 B/s rd, 97 MiB/s wr, 1.55k op/s 2024-09-21T14:07:21.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:21 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:07:21.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:21 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:07:21.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:21 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:07:21.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:21 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:07:21.951 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:21.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:21.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:07:22.053 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:07:22.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:22.096 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:22.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:07:22.197 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:22.237 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:22.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:22.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:22.339 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:07:22.366 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:22.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:22.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:07:22.467 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:07:22.507 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 29 15 55121 55106 118.745 88.5625 0.0129349 0.00841527 2024-09-21T14:07:22.512 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:22.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:22.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:07:22.614 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:07:22.642 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:22.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:22.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:07:22.691 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T14:07:22.733 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:22 smithi049 ceph-mon[40489]: osdmap e475: 8 total, 8 up, 8 in 2024-09-21T14:07:22.733 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:22 smithi049 ceph-mon[30080]: osdmap e475: 8 total, 8 up, 8 in 2024-09-21T14:07:22.745 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:07:22.770 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:22.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:22.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:07:22.812 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T14:07:22.812 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-09-21T14:07:22.812 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-09-21T14:07:22.812 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-09-21T14:07:22.812 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.812 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:70919 10.5 10:af948525:::benchmark_data_smithi049_176811_object70918:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-09-21T14:07:22.812 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:03:44.034223+0000", 2024-09-21T14:07:22.812 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 218.77405338400001, 2024-09-21T14:07:22.813 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.30270100300000002, 2024-09-21T14:07:22.813 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:07:22.813 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:07:22.813 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:07:22.813 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:07:22.813 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:07:22.813 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:07:22.813 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 70919 2024-09-21T14:07:22.813 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.813 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:07:22.813 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.813 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:07:22.813 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.034223+0000", 2024-09-21T14:07:22.813 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:07:22.814 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.814 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.814 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:07:22.814 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.034223+0000", 2024-09-21T14:07:22.814 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:07:22.814 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.814 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.814 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:07:22.814 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.034224+0000", 2024-09-21T14:07:22.814 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.525e-06 2024-09-21T14:07:22.814 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.814 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.814 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:07:22.814 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.034324+0000", 2024-09-21T14:07:22.815 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9767999999999998e-05 2024-09-21T14:07:22.815 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.815 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.815 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:07:22.815 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.034335+0000", 2024-09-21T14:07:22.815 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0789e-05 2024-09-21T14:07:22.815 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.815 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.815 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:07:22.815 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.034342+0000", 2024-09-21T14:07:22.815 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.08e-06 2024-09-21T14:07:22.815 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.815 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.815 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:07:22.815 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.035390+0000", 2024-09-21T14:07:22.816 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0010480789999999999 2024-09-21T14:07:22.816 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.816 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.816 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:07:22.816 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.035523+0000", 2024-09-21T14:07:22.816 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000132464 2024-09-21T14:07:22.816 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.816 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.816 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,3", 2024-09-21T14:07:22.816 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.035674+0000", 2024-09-21T14:07:22.816 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00015162300000000001 2024-09-21T14:07:22.816 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.816 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.816 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:22.817 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.040923+0000", 2024-09-21T14:07:22.817 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0052487920000000004 2024-09-21T14:07:22.817 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.817 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.817 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:07:22.817 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.040935+0000", 2024-09-21T14:07:22.817 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2167000000000001e-05 2024-09-21T14:07:22.817 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.817 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.817 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:22.817 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.336462+0000", 2024-09-21T14:07:22.817 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.29552672000000002 2024-09-21T14:07:22.817 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.817 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.818 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:07:22.818 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.336875+0000", 2024-09-21T14:07:22.818 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00041344400000000001 2024-09-21T14:07:22.818 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.818 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.818 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:07:22.818 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.336924+0000", 2024-09-21T14:07:22.818 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8551999999999999e-05 2024-09-21T14:07:22.818 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:22.818 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:22.818 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:22.818 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.818 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.818 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:70926 10.5 10:a9932ee5:::benchmark_data_smithi049_176811_object70925:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-09-21T14:07:22.819 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:03:44.039120+0000", 2024-09-21T14:07:22.819 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 218.76915592399999, 2024-09-21T14:07:22.819 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.30408621800000002, 2024-09-21T14:07:22.819 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:07:22.819 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:07:22.819 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:07:22.819 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:07:22.819 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:07:22.819 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:07:22.819 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 70926 2024-09-21T14:07:22.819 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.819 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:07:22.819 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.819 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:07:22.819 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.039120+0000", 2024-09-21T14:07:22.820 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:07:22.820 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.820 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.820 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:07:22.820 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.039120+0000", 2024-09-21T14:07:22.820 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:07:22.820 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.820 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.820 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:07:22.820 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.039121+0000", 2024-09-21T14:07:22.820 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.06e-07 2024-09-21T14:07:22.820 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.820 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.820 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:07:22.820 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.039843+0000", 2024-09-21T14:07:22.820 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00072204099999999998 2024-09-21T14:07:22.821 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.821 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.821 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:07:22.821 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.039869+0000", 2024-09-21T14:07:22.821 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6027999999999999e-05 2024-09-21T14:07:22.821 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.821 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.821 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:07:22.821 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.039876+0000", 2024-09-21T14:07:22.821 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9829999999999999e-06 2024-09-21T14:07:22.821 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.821 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.821 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:07:22.821 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.041641+0000", 2024-09-21T14:07:22.822 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0017650890000000001 2024-09-21T14:07:22.822 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.822 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.822 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:07:22.822 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.041772+0000", 2024-09-21T14:07:22.822 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000131066 2024-09-21T14:07:22.822 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.822 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.822 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,3", 2024-09-21T14:07:22.822 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.041907+0000", 2024-09-21T14:07:22.822 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013487800000000001 2024-09-21T14:07:22.822 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.822 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.822 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:07:22.823 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.053687+0000", 2024-09-21T14:07:22.823 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.011779947000000001 2024-09-21T14:07:22.823 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.823 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.823 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:22.823 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.056240+0000", 2024-09-21T14:07:22.823 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0025530079999999998 2024-09-21T14:07:22.823 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.823 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.823 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:22.823 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.343014+0000", 2024-09-21T14:07:22.823 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.286773575 2024-09-21T14:07:22.823 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.823 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.823 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:07:22.823 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.343155+0000", 2024-09-21T14:07:22.824 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014166299999999999 2024-09-21T14:07:22.824 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.824 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.824 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:07:22.824 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.343207+0000", 2024-09-21T14:07:22.824 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1334e-05 2024-09-21T14:07:22.824 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:22.824 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:22.824 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:22.824 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.824 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.824 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:70929 10.5 10:a6d68c9a:::benchmark_data_smithi049_176811_object70928:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-09-21T14:07:22.824 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:03:44.041499+0000", 2024-09-21T14:07:22.825 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 218.766777792, 2024-09-21T14:07:22.825 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.30182954299999998, 2024-09-21T14:07:22.825 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:07:22.825 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:07:22.825 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:07:22.825 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:07:22.825 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:07:22.825 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:07:22.825 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 70929 2024-09-21T14:07:22.825 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.825 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:07:22.825 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.825 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:07:22.825 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.041499+0000", 2024-09-21T14:07:22.826 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:07:22.826 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.826 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.826 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:07:22.826 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.041499+0000", 2024-09-21T14:07:22.826 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:07:22.826 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.826 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.826 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:07:22.826 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.041500+0000", 2024-09-21T14:07:22.826 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.778e-06 2024-09-21T14:07:22.826 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.826 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.827 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:07:22.827 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.041554+0000", 2024-09-21T14:07:22.827 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3501999999999997e-05 2024-09-21T14:07:22.827 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.827 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.827 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:07:22.827 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.041573+0000", 2024-09-21T14:07:22.827 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9210000000000001e-05 2024-09-21T14:07:22.827 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.827 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.827 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:07:22.827 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.041581+0000", 2024-09-21T14:07:22.827 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.276e-06 2024-09-21T14:07:22.827 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.828 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.828 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:07:22.828 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044856+0000", 2024-09-21T14:07:22.828 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0032744570000000001 2024-09-21T14:07:22.828 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.828 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.828 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:07:22.828 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044987+0000", 2024-09-21T14:07:22.828 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0001317 2024-09-21T14:07:22.828 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.828 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.828 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,3", 2024-09-21T14:07:22.828 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.045127+0000", 2024-09-21T14:07:22.828 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013962800000000001 2024-09-21T14:07:22.828 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.829 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.829 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:07:22.829 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.053696+0000", 2024-09-21T14:07:22.829 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0085692059999999994 2024-09-21T14:07:22.829 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.829 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.829 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:22.829 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.056300+0000", 2024-09-21T14:07:22.829 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0026036190000000002 2024-09-21T14:07:22.829 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.829 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.829 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:22.829 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.343249+0000", 2024-09-21T14:07:22.829 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.28694936300000001 2024-09-21T14:07:22.829 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.830 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.830 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:07:22.830 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.343296+0000", 2024-09-21T14:07:22.830 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6414e-05 2024-09-21T14:07:22.830 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.830 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.830 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:07:22.830 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.343328+0000", 2024-09-21T14:07:22.830 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.239e-05 2024-09-21T14:07:22.830 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:22.830 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:22.830 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:22.830 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.830 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.831 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:70932 10.8 10:1f467600:::benchmark_data_smithi049_176811_object70931:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-09-21T14:07:22.831 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:03:44.044077+0000", 2024-09-21T14:07:22.831 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 218.76419893400001, 2024-09-21T14:07:22.831 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.29252181500000002, 2024-09-21T14:07:22.831 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:07:22.831 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:07:22.831 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:07:22.831 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:07:22.831 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:07:22.831 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:07:22.831 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 70932 2024-09-21T14:07:22.831 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.831 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:07:22.831 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.832 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:07:22.832 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044077+0000", 2024-09-21T14:07:22.832 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:07:22.832 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.832 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.832 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:07:22.832 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044077+0000", 2024-09-21T14:07:22.832 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:07:22.832 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.832 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.832 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:07:22.832 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044079+0000", 2024-09-21T14:07:22.832 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3650000000000001e-06 2024-09-21T14:07:22.832 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.833 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.833 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:07:22.833 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044103+0000", 2024-09-21T14:07:22.833 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4343000000000001e-05 2024-09-21T14:07:22.833 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.833 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.833 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:07:22.833 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044120+0000", 2024-09-21T14:07:22.833 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6867999999999998e-05 2024-09-21T14:07:22.833 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.833 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.833 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:07:22.833 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044129+0000", 2024-09-21T14:07:22.833 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0180000000000007e-06 2024-09-21T14:07:22.833 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.834 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.834 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:07:22.834 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044198+0000", 2024-09-21T14:07:22.834 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9282999999999994e-05 2024-09-21T14:07:22.834 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.834 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.834 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:07:22.834 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044374+0000", 2024-09-21T14:07:22.834 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00017556900000000001 2024-09-21T14:07:22.834 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.834 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.834 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:07:22.834 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044544+0000", 2024-09-21T14:07:22.834 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000169977 2024-09-21T14:07:22.835 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.835 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.835 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:22.835 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.049643+0000", 2024-09-21T14:07:22.835 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.005099213 2024-09-21T14:07:22.835 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.835 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.835 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:07:22.835 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.049656+0000", 2024-09-21T14:07:22.835 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2839e-05 2024-09-21T14:07:22.835 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.835 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.835 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:22.835 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.336459+0000", 2024-09-21T14:07:22.836 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.28680339399999999 2024-09-21T14:07:22.836 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.836 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.836 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:07:22.836 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.336550+0000", 2024-09-21T14:07:22.836 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0919000000000001e-05 2024-09-21T14:07:22.836 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.836 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.836 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:07:22.836 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.336599+0000", 2024-09-21T14:07:22.836 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9026999999999997e-05 2024-09-21T14:07:22.836 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:22.836 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:22.836 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:22.836 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.836 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.837 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:70933 10.5 10:a4f74f69:::benchmark_data_smithi049_176811_object70932:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-09-21T14:07:22.837 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:03:44.044479+0000", 2024-09-21T14:07:22.837 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 218.76379759899999, 2024-09-21T14:07:22.837 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.311219415, 2024-09-21T14:07:22.837 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:07:22.837 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:07:22.837 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:07:22.837 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:07:22.837 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:07:22.837 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:07:22.837 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 70933 2024-09-21T14:07:22.837 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.837 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:07:22.837 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.837 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:07:22.838 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044479+0000", 2024-09-21T14:07:22.838 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:07:22.838 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.838 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.838 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:07:22.838 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044479+0000", 2024-09-21T14:07:22.838 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:07:22.838 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.838 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.839 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:07:22.839 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044479+0000", 2024-09-21T14:07:22.839 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6000000000000003e-07 2024-09-21T14:07:22.839 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.839 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.839 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:07:22.839 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044512+0000", 2024-09-21T14:07:22.839 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2669999999999997e-05 2024-09-21T14:07:22.839 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.839 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.839 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:07:22.840 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044521+0000", 2024-09-21T14:07:22.840 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2639999999999997e-06 2024-09-21T14:07:22.840 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.840 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.840 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:07:22.840 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044528+0000", 2024-09-21T14:07:22.840 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3400000000000003e-06 2024-09-21T14:07:22.840 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.840 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.840 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:07:22.840 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.048058+0000", 2024-09-21T14:07:22.840 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0035303330000000001 2024-09-21T14:07:22.840 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.840 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.840 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:07:22.841 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.048196+0000", 2024-09-21T14:07:22.841 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013794600000000001 2024-09-21T14:07:22.841 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.841 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.841 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,3", 2024-09-21T14:07:22.841 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.048338+0000", 2024-09-21T14:07:22.841 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014244099999999999 2024-09-21T14:07:22.841 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.841 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.841 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:07:22.841 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.056197+0000", 2024-09-21T14:07:22.841 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0078581610000000007 2024-09-21T14:07:22.841 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.841 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.842 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:22.842 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.059674+0000", 2024-09-21T14:07:22.842 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.003477258 2024-09-21T14:07:22.842 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.842 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.842 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:22.842 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.353631+0000", 2024-09-21T14:07:22.842 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.29395722400000002 2024-09-21T14:07:22.842 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.842 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.842 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:07:22.842 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.355638+0000", 2024-09-21T14:07:22.842 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0020068260000000002 2024-09-21T14:07:22.842 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.843 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.843 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:07:22.843 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.355698+0000", 2024-09-21T14:07:22.843 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0192000000000002e-05 2024-09-21T14:07:22.843 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:22.843 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:22.843 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:22.843 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.843 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.843 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:70934 10.5 10:af8b5e6c:::benchmark_data_smithi049_176811_object70933:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-09-21T14:07:22.843 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:03:44.044765+0000", 2024-09-21T14:07:22.843 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 218.76351107299999, 2024-09-21T14:07:22.843 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.311083428, 2024-09-21T14:07:22.843 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:07:22.843 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:07:22.843 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:07:22.844 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:07:22.844 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:07:22.844 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:07:22.844 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 70934 2024-09-21T14:07:22.844 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.844 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:07:22.844 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.844 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:07:22.844 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044765+0000", 2024-09-21T14:07:22.844 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:07:22.844 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.844 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.844 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:07:22.844 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044765+0000", 2024-09-21T14:07:22.844 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:07:22.845 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.845 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.845 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:07:22.845 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044766+0000", 2024-09-21T14:07:22.845 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9800000000000003e-07 2024-09-21T14:07:22.845 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.845 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.845 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:07:22.845 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044788+0000", 2024-09-21T14:07:22.845 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2226999999999999e-05 2024-09-21T14:07:22.845 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.845 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.845 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:07:22.845 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044798+0000", 2024-09-21T14:07:22.845 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4429999999999992e-06 2024-09-21T14:07:22.846 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.846 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.846 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:07:22.846 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044804+0000", 2024-09-21T14:07:22.846 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4320000000000004e-06 2024-09-21T14:07:22.846 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.846 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.846 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:07:22.846 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.050867+0000", 2024-09-21T14:07:22.846 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0060626739999999997 2024-09-21T14:07:22.846 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.846 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.846 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:07:22.846 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.051028+0000", 2024-09-21T14:07:22.847 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016143200000000001 2024-09-21T14:07:22.847 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.847 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.847 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,3", 2024-09-21T14:07:22.847 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.051335+0000", 2024-09-21T14:07:22.847 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00030696199999999998 2024-09-21T14:07:22.847 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.847 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.847 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:07:22.847 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.056260+0000", 2024-09-21T14:07:22.847 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0049247759999999996 2024-09-21T14:07:22.847 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.847 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.847 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:22.847 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.059715+0000", 2024-09-21T14:07:22.848 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0034552739999999999 2024-09-21T14:07:22.848 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.848 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.848 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:22.848 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.355755+0000", 2024-09-21T14:07:22.848 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.29603972699999997 2024-09-21T14:07:22.848 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.848 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.848 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:07:22.848 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.355812+0000", 2024-09-21T14:07:22.848 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7420000000000003e-05 2024-09-21T14:07:22.848 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.848 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.848 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:07:22.848 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.355849+0000", 2024-09-21T14:07:22.848 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6362999999999997e-05 2024-09-21T14:07:22.849 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:22.849 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:22.849 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:22.849 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.849 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.849 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:70935 10.5 10:a1fdc64d:::benchmark_data_smithi049_176811_object70934:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-09-21T14:07:22.849 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:03:44.044851+0000", 2024-09-21T14:07:22.849 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 218.763424843, 2024-09-21T14:07:22.849 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.31126390999999998, 2024-09-21T14:07:22.849 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:07:22.849 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:07:22.849 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:07:22.849 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:07:22.849 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:07:22.849 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:07:22.850 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 70935 2024-09-21T14:07:22.850 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.850 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:07:22.850 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.850 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:07:22.850 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044851+0000", 2024-09-21T14:07:22.850 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:07:22.850 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.850 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.850 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:07:22.850 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044851+0000", 2024-09-21T14:07:22.850 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:07:22.850 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.850 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.850 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:07:22.851 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044852+0000", 2024-09-21T14:07:22.851 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0800000000000004e-07 2024-09-21T14:07:22.851 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.851 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.851 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:07:22.851 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044873+0000", 2024-09-21T14:07:22.851 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0999e-05 2024-09-21T14:07:22.851 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.851 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.851 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:07:22.851 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044882+0000", 2024-09-21T14:07:22.851 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6950000000000006e-06 2024-09-21T14:07:22.851 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.851 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.852 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:07:22.852 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044888+0000", 2024-09-21T14:07:22.852 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2770000000000002e-06 2024-09-21T14:07:22.852 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.852 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.852 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:07:22.852 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.053728+0000", 2024-09-21T14:07:22.852 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.008840311 2024-09-21T14:07:22.852 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.852 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.852 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:07:22.852 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.053855+0000", 2024-09-21T14:07:22.852 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000126174 2024-09-21T14:07:22.852 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.852 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.853 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,3", 2024-09-21T14:07:22.853 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.053941+0000", 2024-09-21T14:07:22.853 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6009999999999998e-05 2024-09-21T14:07:22.853 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.853 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.853 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:07:22.853 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.056428+0000", 2024-09-21T14:07:22.853 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0024874340000000002 2024-09-21T14:07:22.853 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.853 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.853 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:22.853 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.060518+0000", 2024-09-21T14:07:22.853 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.004089606 2024-09-21T14:07:22.853 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.853 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.853 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:22.854 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.356032+0000", 2024-09-21T14:07:22.854 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.29551429600000001 2024-09-21T14:07:22.854 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.854 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.854 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:07:22.854 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.356082+0000", 2024-09-21T14:07:22.854 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9679000000000003e-05 2024-09-21T14:07:22.854 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.854 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.854 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:07:22.854 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.356115+0000", 2024-09-21T14:07:22.854 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3821e-05 2024-09-21T14:07:22.854 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:22.855 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:22.855 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:22.855 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.855 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.855 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:70945 10.8 10:1428b9f2:::benchmark_data_smithi049_176811_object70944:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-09-21T14:07:22.855 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:03:44.055643+0000", 2024-09-21T14:07:22.855 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 218.752633721, 2024-09-21T14:07:22.855 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.28525528100000003, 2024-09-21T14:07:22.855 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:07:22.855 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:07:22.855 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:07:22.855 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:07:22.855 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:07:22.855 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:07:22.856 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 70945 2024-09-21T14:07:22.856 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.856 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:07:22.856 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.856 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:07:22.856 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.055643+0000", 2024-09-21T14:07:22.856 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:07:22.856 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.856 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.856 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:07:22.856 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.055643+0000", 2024-09-21T14:07:22.856 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:07:22.856 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.856 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.856 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:07:22.857 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.055644+0000", 2024-09-21T14:07:22.857 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5209999999999999e-06 2024-09-21T14:07:22.857 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.857 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.857 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:07:22.857 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.055666+0000", 2024-09-21T14:07:22.857 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1977999999999999e-05 2024-09-21T14:07:22.857 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.857 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.857 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:07:22.857 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.055682+0000", 2024-09-21T14:07:22.857 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6005999999999999e-05 2024-09-21T14:07:22.857 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.857 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.857 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:07:22.858 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.055690+0000", 2024-09-21T14:07:22.858 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.763e-06 2024-09-21T14:07:22.858 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.858 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.858 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:07:22.858 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.055754+0000", 2024-09-21T14:07:22.858 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3637000000000001e-05 2024-09-21T14:07:22.858 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.858 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.858 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:07:22.858 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.055891+0000", 2024-09-21T14:07:22.858 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013709900000000001 2024-09-21T14:07:22.858 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.858 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.858 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:07:22.859 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.056021+0000", 2024-09-21T14:07:22.859 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013076599999999999 2024-09-21T14:07:22.859 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.859 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.859 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:07:22.859 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.058112+0000", 2024-09-21T14:07:22.859 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002090887 2024-09-21T14:07:22.859 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.859 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.859 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:22.859 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.058366+0000", 2024-09-21T14:07:22.859 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00025399500000000002 2024-09-21T14:07:22.859 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.859 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.859 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:22.860 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.340797+0000", 2024-09-21T14:07:22.860 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.28243062699999999 2024-09-21T14:07:22.860 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.860 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.860 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:07:22.860 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.340853+0000", 2024-09-21T14:07:22.860 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6184000000000003e-05 2024-09-21T14:07:22.860 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.860 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.860 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:07:22.860 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.340898+0000", 2024-09-21T14:07:22.860 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4817999999999997e-05 2024-09-21T14:07:22.860 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:22.860 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:22.860 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:22.860 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.861 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.861 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:70947 10.5 10:a00a68cc:::benchmark_data_smithi049_176811_object70946:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-09-21T14:07:22.861 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:03:44.057433+0000", 2024-09-21T14:07:22.861 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 218.750842879, 2024-09-21T14:07:22.861 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.30024145400000002, 2024-09-21T14:07:22.861 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:07:22.861 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:07:22.861 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:07:22.861 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:07:22.861 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:07:22.861 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:07:22.861 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 70947 2024-09-21T14:07:22.861 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.861 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:07:22.861 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.862 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:07:22.862 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.057433+0000", 2024-09-21T14:07:22.862 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:07:22.862 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.862 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.862 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:07:22.862 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.057433+0000", 2024-09-21T14:07:22.862 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:07:22.862 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.862 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.862 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:07:22.862 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.057435+0000", 2024-09-21T14:07:22.862 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1680000000000001e-06 2024-09-21T14:07:22.862 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.863 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.863 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:07:22.863 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.057462+0000", 2024-09-21T14:07:22.863 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.7415000000000001e-05 2024-09-21T14:07:22.863 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.863 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.863 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:07:22.863 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.057477+0000", 2024-09-21T14:07:22.863 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5150000000000001e-05 2024-09-21T14:07:22.863 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.863 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.863 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:07:22.863 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.057486+0000", 2024-09-21T14:07:22.863 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1770000000000004e-06 2024-09-21T14:07:22.864 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.864 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.864 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:07:22.864 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.057543+0000", 2024-09-21T14:07:22.864 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7122e-05 2024-09-21T14:07:22.864 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.864 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.864 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:07:22.864 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.057640+0000", 2024-09-21T14:07:22.864 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6124000000000003e-05 2024-09-21T14:07:22.864 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.864 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.864 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,3", 2024-09-21T14:07:22.864 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.057762+0000", 2024-09-21T14:07:22.865 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012260600000000001 2024-09-21T14:07:22.865 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.865 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.865 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:07:22.865 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.059813+0000", 2024-09-21T14:07:22.865 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002050666 2024-09-21T14:07:22.865 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.865 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.865 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:22.865 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.060700+0000", 2024-09-21T14:07:22.865 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00088741999999999998 2024-09-21T14:07:22.865 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.865 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.865 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:22.866 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.357580+0000", 2024-09-21T14:07:22.866 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.29687936700000001 2024-09-21T14:07:22.866 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.866 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.866 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:07:22.866 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.357635+0000", 2024-09-21T14:07:22.866 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5053999999999998e-05 2024-09-21T14:07:22.866 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.866 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.866 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:07:22.866 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.357675+0000", 2024-09-21T14:07:22.866 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0185000000000002e-05 2024-09-21T14:07:22.866 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:22.866 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:22.867 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:22.867 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.867 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.867 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:131812 10.5 10:abb38f76:::benchmark_data_smithi049_176811_object4235:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-09-21T14:07:22.867 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:04:39.769460+0000", 2024-09-21T14:07:22.867 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 163.038816204, 2024-09-21T14:07:22.867 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.270613783, 2024-09-21T14:07:22.867 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:07:22.867 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:07:22.867 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:07:22.867 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:07:22.867 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:07:22.867 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:07:22.867 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 131812 2024-09-21T14:07:22.867 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.868 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:07:22.868 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.868 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:07:22.868 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:39.769460+0000", 2024-09-21T14:07:22.868 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:07:22.868 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.868 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.868 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:07:22.868 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:39.769460+0000", 2024-09-21T14:07:22.868 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:07:22.868 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.868 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.868 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:07:22.868 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:39.769462+0000", 2024-09-21T14:07:22.868 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6130000000000001e-06 2024-09-21T14:07:22.869 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.869 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.869 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:07:22.869 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:39.769467+0000", 2024-09-21T14:07:22.869 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.775e-06 2024-09-21T14:07:22.869 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.869 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.869 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:07:22.869 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:39.769480+0000", 2024-09-21T14:07:22.869 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3108000000000001e-05 2024-09-21T14:07:22.869 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.869 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.869 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:07:22.869 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:39.769487+0000", 2024-09-21T14:07:22.869 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9569999999999996e-06 2024-09-21T14:07:22.870 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.870 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.870 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:07:22.870 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:39.769555+0000", 2024-09-21T14:07:22.870 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8573999999999998e-05 2024-09-21T14:07:22.870 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.870 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.870 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:07:22.870 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:39.769645+0000", 2024-09-21T14:07:22.870 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9464999999999998e-05 2024-09-21T14:07:22.870 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.870 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.870 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,3", 2024-09-21T14:07:22.870 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:39.769725+0000", 2024-09-21T14:07:22.871 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9913999999999994e-05 2024-09-21T14:07:22.871 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.871 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.871 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:07:22.871 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:39.771831+0000", 2024-09-21T14:07:22.871 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0021061220000000002 2024-09-21T14:07:22.871 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.871 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.871 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:22.871 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:39.922948+0000", 2024-09-21T14:07:22.871 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.15111724900000001 2024-09-21T14:07:22.871 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.871 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.871 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:22.872 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:40.039982+0000", 2024-09-21T14:07:22.872 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.11703383000000001 2024-09-21T14:07:22.872 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.872 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.872 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:07:22.872 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:40.040030+0000", 2024-09-21T14:07:22.872 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8377000000000001e-05 2024-09-21T14:07:22.872 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.872 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.872 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:07:22.872 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:40.040074+0000", 2024-09-21T14:07:22.872 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3798999999999999e-05 2024-09-21T14:07:22.872 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:22.872 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:22.872 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:22.873 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.873 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.873 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:142774 10.3 10:cb227b01:::benchmark_data_smithi049_176811_object15197:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-09-21T14:07:22.873 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:04:50.151661+0000", 2024-09-21T14:07:22.873 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 152.65661494700001, 2024-09-21T14:07:22.873 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.228470856, 2024-09-21T14:07:22.873 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:07:22.873 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:07:22.873 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:07:22.873 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:07:22.873 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:07:22.873 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:07:22.873 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 142774 2024-09-21T14:07:22.873 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.873 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:07:22.874 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.874 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:07:22.874 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:50.151661+0000", 2024-09-21T14:07:22.874 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:07:22.874 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.874 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.874 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:07:22.874 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:50.151661+0000", 2024-09-21T14:07:22.874 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:07:22.874 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.874 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.874 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:07:22.874 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:50.151663+0000", 2024-09-21T14:07:22.874 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.511e-06 2024-09-21T14:07:22.875 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.875 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.875 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:07:22.875 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:50.151668+0000", 2024-09-21T14:07:22.875 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6349999999999997e-06 2024-09-21T14:07:22.875 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.875 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.875 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:07:22.875 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:50.151702+0000", 2024-09-21T14:07:22.875 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4686e-05 2024-09-21T14:07:22.875 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.875 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.875 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:07:22.875 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:50.151721+0000", 2024-09-21T14:07:22.875 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8785000000000001e-05 2024-09-21T14:07:22.876 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.876 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.876 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:07:22.876 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:50.163370+0000", 2024-09-21T14:07:22.876 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.011648708000000001 2024-09-21T14:07:22.876 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.876 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.876 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:07:22.876 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:50.163496+0000", 2024-09-21T14:07:22.876 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000126673 2024-09-21T14:07:22.876 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.876 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.876 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,5", 2024-09-21T14:07:22.876 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:50.163614+0000", 2024-09-21T14:07:22.876 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000117205 2024-09-21T14:07:22.877 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.877 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.877 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:22.877 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:50.173683+0000", 2024-09-21T14:07:22.877 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.010069793 2024-09-21T14:07:22.877 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.877 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.877 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:07:22.877 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:50.178515+0000", 2024-09-21T14:07:22.877 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0048316590000000003 2024-09-21T14:07:22.877 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:07:22.878 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.879 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.879 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:22.879 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:50.379832+0000", 2024-09-21T14:07:22.879 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.20131663699999999 2024-09-21T14:07:22.879 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.879 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.879 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:07:22.879 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:50.380072+0000", 2024-09-21T14:07:22.879 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000240196 2024-09-21T14:07:22.879 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.879 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.879 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:07:22.879 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:50.380132+0000", 2024-09-21T14:07:22.880 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0368000000000002e-05 2024-09-21T14:07:22.880 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:22.880 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:22.880 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:22.880 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.880 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.880 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:142786 10.5 10:a51788d0:::benchmark_data_smithi049_176811_object15209:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-09-21T14:07:22.880 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:04:50.158570+0000", 2024-09-21T14:07:22.880 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 152.64970669300001, 2024-09-21T14:07:22.880 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.221318974, 2024-09-21T14:07:22.880 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:07:22.880 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:07:22.880 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:07:22.880 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:07:22.880 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:07:22.881 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:07:22.881 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 142786 2024-09-21T14:07:22.881 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.881 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:07:22.881 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.881 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:07:22.881 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:50.158570+0000", 2024-09-21T14:07:22.881 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:07:22.881 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.881 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.881 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:07:22.881 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:50.158570+0000", 2024-09-21T14:07:22.881 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:07:22.881 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.882 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.882 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:07:22.882 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:50.158571+0000", 2024-09-21T14:07:22.882 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.655e-06 2024-09-21T14:07:22.882 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.882 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.882 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:07:22.882 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:50.158576+0000", 2024-09-21T14:07:22.882 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9849999999999997e-06 2024-09-21T14:07:22.882 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.882 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.882 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:07:22.882 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:50.158590+0000", 2024-09-21T14:07:22.882 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3837e-05 2024-09-21T14:07:22.883 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.883 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.883 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:07:22.883 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:50.158599+0000", 2024-09-21T14:07:22.883 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1470000000000007e-06 2024-09-21T14:07:22.883 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.883 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.883 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:07:22.883 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:50.158747+0000", 2024-09-21T14:07:22.883 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014780300000000001 2024-09-21T14:07:22.883 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.883 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.883 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:07:22.883 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:50.158892+0000", 2024-09-21T14:07:22.884 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014486699999999999 2024-09-21T14:07:22.884 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.884 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.884 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,3", 2024-09-21T14:07:22.884 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:50.159001+0000", 2024-09-21T14:07:22.884 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000108856 2024-09-21T14:07:22.884 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.884 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.884 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:22.884 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:50.171233+0000", 2024-09-21T14:07:22.884 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.012231766 2024-09-21T14:07:22.884 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.884 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.884 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:07:22.884 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:50.171244+0000", 2024-09-21T14:07:22.885 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1663e-05 2024-09-21T14:07:22.885 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.885 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.885 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:22.885 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:50.379778+0000", 2024-09-21T14:07:22.885 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.208534205 2024-09-21T14:07:22.885 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.885 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.885 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:07:22.885 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:50.379842+0000", 2024-09-21T14:07:22.885 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3875999999999996e-05 2024-09-21T14:07:22.885 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.885 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.885 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:07:22.886 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:50.379889+0000", 2024-09-21T14:07:22.886 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6313999999999997e-05 2024-09-21T14:07:22.886 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:22.886 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:22.886 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:22.886 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.886 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.886 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:142799 10.3 10:c1ab7d87:::benchmark_data_smithi049_176811_object15222:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-09-21T14:07:22.886 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:04:50.170716+0000", 2024-09-21T14:07:22.886 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 152.63756023400001, 2024-09-21T14:07:22.886 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.227021425, 2024-09-21T14:07:22.886 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:07:22.886 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:07:22.886 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:07:22.887 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:07:22.887 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:07:22.887 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:07:22.887 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 142799 2024-09-21T14:07:22.887 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.887 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:07:22.887 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.887 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:07:22.887 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:50.170716+0000", 2024-09-21T14:07:22.887 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:07:22.887 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.887 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.887 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:07:22.887 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:50.170716+0000", 2024-09-21T14:07:22.887 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:07:22.887 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.888 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.888 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:07:22.888 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:50.170718+0000", 2024-09-21T14:07:22.888 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.483e-06 2024-09-21T14:07:22.888 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.888 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.888 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:07:22.888 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:50.170721+0000", 2024-09-21T14:07:22.888 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8580000000000002e-06 2024-09-21T14:07:22.888 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.888 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.888 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:07:22.888 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:50.170730+0000", 2024-09-21T14:07:22.888 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0000000000000002e-06 2024-09-21T14:07:22.888 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.888 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.889 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:07:22.889 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:50.170736+0000", 2024-09-21T14:07:22.889 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.959e-06 2024-09-21T14:07:22.889 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.889 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.889 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:07:22.889 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:50.173722+0000", 2024-09-21T14:07:22.889 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0029853369999999998 2024-09-21T14:07:22.889 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.889 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.889 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:07:22.889 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:50.173847+0000", 2024-09-21T14:07:22.889 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012479400000000001 2024-09-21T14:07:22.889 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.890 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.890 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,5", 2024-09-21T14:07:22.890 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:50.173982+0000", 2024-09-21T14:07:22.890 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000135644 2024-09-21T14:07:22.890 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.890 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.890 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:22.890 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:50.178554+0000", 2024-09-21T14:07:22.890 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.004571945 2024-09-21T14:07:22.890 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.890 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.890 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:07:22.890 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:50.178567+0000", 2024-09-21T14:07:22.890 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3047000000000001e-05 2024-09-21T14:07:22.891 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.891 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.891 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:22.891 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:50.397624+0000", 2024-09-21T14:07:22.891 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21905727 2024-09-21T14:07:22.891 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.891 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.891 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:07:22.891 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:50.397680+0000", 2024-09-21T14:07:22.891 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5461000000000003e-05 2024-09-21T14:07:22.891 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.891 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.891 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:07:22.891 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:50.397738+0000", 2024-09-21T14:07:22.891 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7627000000000002e-05 2024-09-21T14:07:22.892 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:22.892 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:22.892 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:22.892 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.892 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.892 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:142802 10.5 10:a7e18be4:::benchmark_data_smithi049_176811_object15225:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-09-21T14:07:22.892 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:04:50.174809+0000", 2024-09-21T14:07:22.892 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 152.63346687500001, 2024-09-21T14:07:22.892 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21682669199999999, 2024-09-21T14:07:22.892 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:07:22.892 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:07:22.892 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:07:22.892 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:07:22.892 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:07:22.892 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:07:22.893 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 142802 2024-09-21T14:07:22.893 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.893 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:07:22.893 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.893 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:07:22.893 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:50.174809+0000", 2024-09-21T14:07:22.893 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:07:22.893 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.893 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.893 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:07:22.893 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:50.174809+0000", 2024-09-21T14:07:22.893 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:07:22.893 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.894 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.894 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:07:22.894 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:50.174811+0000", 2024-09-21T14:07:22.894 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.725e-06 2024-09-21T14:07:22.894 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.894 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.894 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:07:22.894 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:50.174817+0000", 2024-09-21T14:07:22.894 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1129999999999997e-06 2024-09-21T14:07:22.894 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.894 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.894 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:07:22.894 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:50.174836+0000", 2024-09-21T14:07:22.895 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8437999999999999e-05 2024-09-21T14:07:22.895 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.895 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.895 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:07:22.895 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:50.174845+0000", 2024-09-21T14:07:22.895 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.7739999999999993e-06 2024-09-21T14:07:22.895 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.895 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.895 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:07:22.895 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:50.175344+0000", 2024-09-21T14:07:22.895 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00049940100000000001 2024-09-21T14:07:22.895 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.895 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.895 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:07:22.896 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:50.175465+0000", 2024-09-21T14:07:22.896 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000121298 2024-09-21T14:07:22.896 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.896 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.896 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,3", 2024-09-21T14:07:22.896 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:50.175603+0000", 2024-09-21T14:07:22.896 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013778600000000001 2024-09-21T14:07:22.896 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.896 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.896 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:07:22.896 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:50.178307+0000", 2024-09-21T14:07:22.896 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0027041349999999999 2024-09-21T14:07:22.896 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.896 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.897 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:22.897 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:50.179047+0000", 2024-09-21T14:07:22.897 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00074017699999999996 2024-09-21T14:07:22.897 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.897 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.897 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:22.897 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:50.391381+0000", 2024-09-21T14:07:22.897 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21233366100000001 2024-09-21T14:07:22.897 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.897 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.898 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:07:22.898 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:50.391588+0000", 2024-09-21T14:07:22.898 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00020716699999999999 2024-09-21T14:07:22.898 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.898 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.898 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:07:22.898 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:50.391636+0000", 2024-09-21T14:07:22.898 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8016999999999998e-05 2024-09-21T14:07:22.898 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:22.898 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:22.898 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:22.899 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.899 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.899 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:160586 10.5 10:a28ec32d:::benchmark_data_smithi049_176811_object33009:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-09-21T14:07:22.899 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:05:07.638714+0000", 2024-09-21T14:07:22.899 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 135.16956268600001, 2024-09-21T14:07:22.899 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21612146400000001, 2024-09-21T14:07:22.899 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:07:22.899 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:07:22.899 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:07:22.899 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:07:22.899 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:07:22.899 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:07:22.899 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 160586 2024-09-21T14:07:22.899 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.900 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:07:22.900 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.900 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:07:22.900 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:07.638714+0000", 2024-09-21T14:07:22.900 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:07:22.900 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.900 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.900 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:07:22.900 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:07.638714+0000", 2024-09-21T14:07:22.900 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:07:22.900 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.900 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.900 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:07:22.900 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:07.638715+0000", 2024-09-21T14:07:22.900 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.793e-06 2024-09-21T14:07:22.901 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.901 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.901 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:07:22.901 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:07.638722+0000", 2024-09-21T14:07:22.901 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5470000000000002e-06 2024-09-21T14:07:22.901 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.901 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.901 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:07:22.901 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:07.638739+0000", 2024-09-21T14:07:22.901 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7367000000000002e-05 2024-09-21T14:07:22.901 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.901 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.901 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:07:22.901 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:07.640589+0000", 2024-09-21T14:07:22.902 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001849299 2024-09-21T14:07:22.902 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.902 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.902 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:07:22.902 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:07.640648+0000", 2024-09-21T14:07:22.902 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9704000000000003e-05 2024-09-21T14:07:22.902 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.902 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.902 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:07:22.902 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:07.640877+0000", 2024-09-21T14:07:22.902 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000229077 2024-09-21T14:07:22.902 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.902 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.903 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,3", 2024-09-21T14:07:22.903 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:07.640982+0000", 2024-09-21T14:07:22.903 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010490800000000001 2024-09-21T14:07:22.903 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.903 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.903 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:07:22.903 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:07.643798+0000", 2024-09-21T14:07:22.903 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0028159550000000002 2024-09-21T14:07:22.903 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.903 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.903 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:22.903 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:07.649027+0000", 2024-09-21T14:07:22.903 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0052288949999999999 2024-09-21T14:07:22.903 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.904 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.904 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:22.904 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:07.854735+0000", 2024-09-21T14:07:22.904 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.20570813399999999 2024-09-21T14:07:22.904 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.904 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.904 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:07:22.904 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:07.854792+0000", 2024-09-21T14:07:22.904 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7132999999999997e-05 2024-09-21T14:07:22.904 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.904 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.904 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:07:22.905 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:07.854835+0000", 2024-09-21T14:07:22.905 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2651999999999998e-05 2024-09-21T14:07:22.905 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:22.905 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:22.905 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:22.905 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.905 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.905 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:160603 10.5 10:ac8372c9:::benchmark_data_smithi049_176811_object33026:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-09-21T14:07:22.905 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:05:07.653064+0000", 2024-09-21T14:07:22.905 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 135.15521266299999, 2024-09-21T14:07:22.905 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22185257799999999, 2024-09-21T14:07:22.905 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:07:22.905 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:07:22.906 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:07:22.906 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:07:22.906 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:07:22.906 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:07:22.906 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 160603 2024-09-21T14:07:22.906 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.906 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:07:22.906 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.906 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:07:22.906 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:07.653064+0000", 2024-09-21T14:07:22.906 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:07:22.906 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.906 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.907 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:07:22.907 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:07.653064+0000", 2024-09-21T14:07:22.907 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:07:22.907 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.907 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.907 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:07:22.907 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:07.653066+0000", 2024-09-21T14:07:22.907 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1430000000000001e-06 2024-09-21T14:07:22.907 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.907 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.907 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:07:22.907 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:07.653072+0000", 2024-09-21T14:07:22.907 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6699999999999997e-06 2024-09-21T14:07:22.907 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.908 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.908 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:07:22.908 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:07.653095+0000", 2024-09-21T14:07:22.908 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3008000000000001e-05 2024-09-21T14:07:22.908 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.908 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.908 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:07:22.908 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:07.653105+0000", 2024-09-21T14:07:22.908 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3000000000000007e-06 2024-09-21T14:07:22.908 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.908 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.908 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:07:22.908 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:07.653205+0000", 2024-09-21T14:07:22.908 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000100262 2024-09-21T14:07:22.909 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.909 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.909 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:07:22.909 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:07.653371+0000", 2024-09-21T14:07:22.909 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016623899999999999 2024-09-21T14:07:22.909 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.909 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.909 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,3", 2024-09-21T14:07:22.909 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:07.653515+0000", 2024-09-21T14:07:22.909 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014332599999999999 2024-09-21T14:07:22.909 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.909 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.909 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:07:22.909 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:07.656082+0000", 2024-09-21T14:07:22.909 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0025671840000000001 2024-09-21T14:07:22.910 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.910 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.910 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:22.910 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:07.656327+0000", 2024-09-21T14:07:22.910 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00024490699999999999 2024-09-21T14:07:22.910 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.910 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.910 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:22.910 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:07.874815+0000", 2024-09-21T14:07:22.910 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21848846799999999 2024-09-21T14:07:22.910 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.910 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.910 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:07:22.911 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:07.874870+0000", 2024-09-21T14:07:22.911 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4920000000000003e-05 2024-09-21T14:07:22.911 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.911 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.911 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:07:22.911 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:07.874916+0000", 2024-09-21T14:07:22.911 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6150999999999999e-05 2024-09-21T14:07:22.911 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:22.911 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:22.911 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:22.911 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.911 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.911 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:208866 10.3 10:c8242cfb:::benchmark_data_smithi049_176811_object81289:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e441)", 2024-09-21T14:07:22.912 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:05:54.047026+0000", 2024-09-21T14:07:22.912 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 88.761250403000005, 2024-09-21T14:07:22.912 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.31484621499999998, 2024-09-21T14:07:22.912 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:07:22.912 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:07:22.912 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:07:22.912 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:07:22.912 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:07:22.912 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:07:22.912 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 208866 2024-09-21T14:07:22.912 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.912 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:07:22.912 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.912 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:07:22.913 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047026+0000", 2024-09-21T14:07:22.913 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:07:22.913 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.913 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.913 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:07:22.913 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047026+0000", 2024-09-21T14:07:22.913 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:07:22.913 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.913 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.913 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:07:22.913 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047027+0000", 2024-09-21T14:07:22.913 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3820000000000001e-06 2024-09-21T14:07:22.913 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.913 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.914 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:07:22.914 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047033+0000", 2024-09-21T14:07:22.914 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3859999999999998e-06 2024-09-21T14:07:22.914 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.914 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.914 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:07:22.914 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047046+0000", 2024-09-21T14:07:22.914 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3482999999999999e-05 2024-09-21T14:07:22.914 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.914 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.914 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:07:22.914 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047053+0000", 2024-09-21T14:07:22.914 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.25e-06 2024-09-21T14:07:22.914 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.915 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.915 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:07:22.915 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047124+0000", 2024-09-21T14:07:22.915 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0309999999999996e-05 2024-09-21T14:07:22.915 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.915 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.915 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:07:22.915 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047243+0000", 2024-09-21T14:07:22.915 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011944599999999999 2024-09-21T14:07:22.915 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.915 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.915 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,5", 2024-09-21T14:07:22.915 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047366+0000", 2024-09-21T14:07:22.915 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012313100000000001 2024-09-21T14:07:22.916 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.916 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.916 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:07:22.916 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.052472+0000", 2024-09-21T14:07:22.916 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0051057760000000002 2024-09-21T14:07:22.916 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.916 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.916 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:22.916 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.055019+0000", 2024-09-21T14:07:22.916 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0025469709999999999 2024-09-21T14:07:22.916 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.916 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.916 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:22.916 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.361767+0000", 2024-09-21T14:07:22.917 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.30674835499999997 2024-09-21T14:07:22.917 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.917 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.917 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:07:22.917 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.361826+0000", 2024-09-21T14:07:22.917 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8084999999999997e-05 2024-09-21T14:07:22.917 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.917 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.917 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:07:22.917 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.361872+0000", 2024-09-21T14:07:22.917 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6640000000000001e-05 2024-09-21T14:07:22.917 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:22.917 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:22.917 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:22.917 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.917 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.918 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:208867 10.3 10:c6755fb4:::benchmark_data_smithi049_176811_object81290:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e441)", 2024-09-21T14:07:22.918 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:05:54.047593+0000", 2024-09-21T14:07:22.918 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 88.760683155999999, 2024-09-21T14:07:22.918 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.31790776100000001, 2024-09-21T14:07:22.918 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:07:22.918 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:07:22.918 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:07:22.918 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:07:22.918 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:07:22.918 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:07:22.918 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 208867 2024-09-21T14:07:22.918 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.918 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:07:22.918 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.918 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:07:22.919 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047593+0000", 2024-09-21T14:07:22.919 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:07:22.919 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.919 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.919 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:07:22.919 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047593+0000", 2024-09-21T14:07:22.919 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:07:22.919 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.919 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.919 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:07:22.919 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047594+0000", 2024-09-21T14:07:22.919 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9100000000000003e-07 2024-09-21T14:07:22.919 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.919 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.919 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:07:22.921 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047598+0000", 2024-09-21T14:07:22.921 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0199999999999996e-06 2024-09-21T14:07:22.921 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.921 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.921 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:07:22.921 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047609+0000", 2024-09-21T14:07:22.921 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0858e-05 2024-09-21T14:07:22.921 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.921 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.921 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:07:22.921 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047616+0000", 2024-09-21T14:07:22.921 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3629999999999996e-06 2024-09-21T14:07:22.922 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.922 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.922 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:07:22.922 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.050406+0000", 2024-09-21T14:07:22.922 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0027899639999999998 2024-09-21T14:07:22.922 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.922 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.922 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:07:22.922 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.050525+0000", 2024-09-21T14:07:22.922 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011861 2024-09-21T14:07:22.922 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.922 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.922 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,5", 2024-09-21T14:07:22.923 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.050650+0000", 2024-09-21T14:07:22.923 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000124881 2024-09-21T14:07:22.923 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.923 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.923 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:07:22.923 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.052609+0000", 2024-09-21T14:07:22.923 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0019588499999999998 2024-09-21T14:07:22.923 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.923 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.923 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:22.923 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.057528+0000", 2024-09-21T14:07:22.923 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0049191590000000002 2024-09-21T14:07:22.923 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.923 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.923 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:22.924 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.365390+0000", 2024-09-21T14:07:22.924 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.30786228399999999 2024-09-21T14:07:22.924 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.924 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.925 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:07:22.925 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.365449+0000", 2024-09-21T14:07:22.925 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9497999999999999e-05 2024-09-21T14:07:22.925 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.925 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.926 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:07:22.926 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.365501+0000", 2024-09-21T14:07:22.926 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1483000000000002e-05 2024-09-21T14:07:22.926 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:22.926 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:22.926 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:22.926 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.926 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.926 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:208869 10.5 10:a17fb6ee:::benchmark_data_smithi049_176811_object81292:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e441)", 2024-09-21T14:07:22.926 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:05:54.048229+0000", 2024-09-21T14:07:22.926 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 88.760047463999996, 2024-09-21T14:07:22.926 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.31834309799999999, 2024-09-21T14:07:22.927 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:07:22.927 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:07:22.927 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:07:22.927 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:07:22.927 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:07:22.927 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:07:22.927 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 208869 2024-09-21T14:07:22.927 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.927 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:07:22.927 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.927 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:07:22.927 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.048229+0000", 2024-09-21T14:07:22.927 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:07:22.928 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.928 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.928 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:07:22.928 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.048229+0000", 2024-09-21T14:07:22.928 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:07:22.928 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.928 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.928 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:07:22.928 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.048230+0000", 2024-09-21T14:07:22.928 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3799999999999996e-07 2024-09-21T14:07:22.928 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.928 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.928 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:07:22.928 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.048233+0000", 2024-09-21T14:07:22.928 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.602e-06 2024-09-21T14:07:22.929 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.929 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.929 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:07:22.929 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.048242+0000", 2024-09-21T14:07:22.929 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0389999999999993e-06 2024-09-21T14:07:22.929 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.929 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.929 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:07:22.929 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.048249+0000", 2024-09-21T14:07:22.929 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7599999999999997e-06 2024-09-21T14:07:22.929 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.929 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.929 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:07:22.929 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.048305+0000", 2024-09-21T14:07:22.929 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6085999999999998e-05 2024-09-21T14:07:22.930 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.930 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.930 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:07:22.930 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.048397+0000", 2024-09-21T14:07:22.930 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2069000000000002e-05 2024-09-21T14:07:22.930 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.930 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.930 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,3", 2024-09-21T14:07:22.930 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.048465+0000", 2024-09-21T14:07:22.930 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7998999999999997e-05 2024-09-21T14:07:22.930 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.930 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.930 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:07:22.930 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.050592+0000", 2024-09-21T14:07:22.930 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0021271509999999999 2024-09-21T14:07:22.931 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.931 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.931 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:22.931 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.051296+0000", 2024-09-21T14:07:22.931 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00070396900000000004 2024-09-21T14:07:22.931 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.931 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.931 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:22.931 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.366467+0000", 2024-09-21T14:07:22.931 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.31517033700000002 2024-09-21T14:07:22.931 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.931 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.931 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:07:22.932 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.366534+0000", 2024-09-21T14:07:22.932 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7045999999999995e-05 2024-09-21T14:07:22.932 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.932 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.932 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:07:22.932 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.366572+0000", 2024-09-21T14:07:22.932 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8201999999999998e-05 2024-09-21T14:07:22.932 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:22.932 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:22.932 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:22.932 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.932 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.932 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:208877 10.3 10:cae1fc76:::benchmark_data_smithi049_176811_object81300:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e441)", 2024-09-21T14:07:22.932 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:05:54.054998+0000", 2024-09-21T14:07:22.932 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 88.753277937999997, 2024-09-21T14:07:22.933 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.310795918, 2024-09-21T14:07:22.933 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:07:22.933 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:07:22.933 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:07:22.933 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:07:22.933 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:07:22.933 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:07:22.933 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 208877 2024-09-21T14:07:22.933 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.933 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:07:22.933 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.933 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:07:22.933 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.054998+0000", 2024-09-21T14:07:22.933 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:07:22.934 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.934 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.934 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:07:22.934 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.054998+0000", 2024-09-21T14:07:22.934 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:07:22.934 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.934 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.934 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:07:22.934 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.055000+0000", 2024-09-21T14:07:22.934 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.821e-06 2024-09-21T14:07:22.934 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.934 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.934 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:07:22.934 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.055004+0000", 2024-09-21T14:07:22.935 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1540000000000004e-06 2024-09-21T14:07:22.935 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.935 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.935 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:07:22.935 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.055015+0000", 2024-09-21T14:07:22.935 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0594999999999999e-05 2024-09-21T14:07:22.935 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.935 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.935 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:07:22.935 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.055022+0000", 2024-09-21T14:07:22.935 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9380000000000003e-06 2024-09-21T14:07:22.935 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.935 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.935 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:07:22.935 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.055127+0000", 2024-09-21T14:07:22.936 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010461199999999999 2024-09-21T14:07:22.936 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.936 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.936 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:07:22.936 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.055242+0000", 2024-09-21T14:07:22.936 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000115377 2024-09-21T14:07:22.936 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.936 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.936 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,5", 2024-09-21T14:07:22.936 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.055379+0000", 2024-09-21T14:07:22.936 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000136627 2024-09-21T14:07:22.936 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.936 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.936 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:07:22.937 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.057617+0000", 2024-09-21T14:07:22.937 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00223865 2024-09-21T14:07:22.937 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.937 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.937 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:22.937 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.058057+0000", 2024-09-21T14:07:22.937 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00044032700000000001 2024-09-21T14:07:22.937 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.937 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.937 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:22.937 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.365711+0000", 2024-09-21T14:07:22.937 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.30765357999999998 2024-09-21T14:07:22.937 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.937 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.937 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:07:22.938 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.365757+0000", 2024-09-21T14:07:22.938 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5985000000000001e-05 2024-09-21T14:07:22.938 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:22.938 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:22.938 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:07:22.938 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.365794+0000", 2024-09-21T14:07:22.938 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7252000000000002e-05 2024-09-21T14:07:22.938 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:22.938 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:22.938 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:22.938 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:22.938 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:22.938 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T14:07:22.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:22.939 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:22.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:07:22.940 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:22 smithi187 ceph-mon[33032]: osdmap e475: 8 total, 8 up, 8 in 2024-09-21T14:07:23.040 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:07:23.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:23.102 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:23.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:07:23.202 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:07:23.253 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:23.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:23.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:07:23.355 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:07:23.387 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:23.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:23.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:07:23.461 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T14:07:23.488 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:07:23.508 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 30 16 56439 56423 117.528 82.3125 0.0134957 0.00850255 2024-09-21T14:07:23.514 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:23.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:23.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:07:23.546 INFO:teuthology.orchestra.run.smithi049.stderr:osd set-full-ratio 0.001 2024-09-21T14:07:23.616 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:07:23.669 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:23.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:23.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:07:23.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:23 smithi049 ceph-mon[40489]: pgmap v941: 168 pgs: 168 active+clean; 3.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 123 MiB/s wr, 1.97k op/s 2024-09-21T14:07:23.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:23 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3710607373' entity='client.admin' cmd={"prefix": "osd set-full-ratio", "ratio": 0.001} : dispatch 2024-09-21T14:07:23.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:23 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mon-a[30057]: 2024-09-21T14:07:23.529+0000 7fc8b9851640 -1 log_channel(cluster) log [ERR] : Health check failed: full ratio(s) out of order (OSD_OUT_OF_ORDER_FULL) 2024-09-21T14:07:23.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:23 smithi049 ceph-mon[30080]: pgmap v941: 168 pgs: 168 active+clean; 3.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 123 MiB/s wr, 1.97k op/s 2024-09-21T14:07:23.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:23 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3710607373' entity='client.admin' cmd={"prefix": "osd set-full-ratio", "ratio": 0.001} : dispatch 2024-09-21T14:07:23.771 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:07:23.798 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:23.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:23.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:07:23.900 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:07:23.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:23.930 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:23.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:07:23.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:23 smithi187 ceph-mon[33032]: pgmap v941: 168 pgs: 168 active+clean; 3.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 123 MiB/s wr, 1.97k op/s 2024-09-21T14:07:23.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:23 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3710607373' entity='client.admin' cmd={"prefix": "osd set-full-ratio", "ratio": 0.001} : dispatch 2024-09-21T14:07:24.032 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:07:24.060 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:24.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:24.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:07:24.162 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:07:24.190 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:24.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:24.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:07:24.208 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T14:07:24.291 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:24.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:24.347 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:24.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:24.448 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:07:24.500 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:24.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:24.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:07:24.509 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 31 16 57829 57813 116.537 86.875 0.00814872 0.00857451 2024-09-21T14:07:24.603 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:07:24.618 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:07:24.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:24.670 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:24.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:07:24.771 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:07:24.787 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:24 smithi049 ceph-mon[30080]: Health check failed: full ratio(s) out of order (OSD_OUT_OF_ORDER_FULL) 2024-09-21T14:07:24.787 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:24 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3710607373' entity='client.admin' cmd='[{"prefix": "osd set-full-ratio", "ratio": 0.001}]': finished 2024-09-21T14:07:24.787 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:24 smithi049 ceph-mon[30080]: osdmap e476: 8 total, 8 up, 8 in 2024-09-21T14:07:24.787 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:24 smithi049 ceph-mon[40489]: Health check failed: full ratio(s) out of order (OSD_OUT_OF_ORDER_FULL) 2024-09-21T14:07:24.787 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:24 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3710607373' entity='client.admin' cmd='[{"prefix": "osd set-full-ratio", "ratio": 0.001}]': finished 2024-09-21T14:07:24.787 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:24 smithi049 ceph-mon[40489]: osdmap e476: 8 total, 8 up, 8 in 2024-09-21T14:07:24.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:24.818 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:24.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:07:24.919 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:07:24.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:24 smithi187 ceph-mon[33032]: Health check failed: full ratio(s) out of order (OSD_OUT_OF_ORDER_FULL) 2024-09-21T14:07:24.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:24 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3710607373' entity='client.admin' cmd='[{"prefix": "osd set-full-ratio", "ratio": 0.001}]': finished 2024-09-21T14:07:24.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:24 smithi187 ceph-mon[33032]: osdmap e476: 8 total, 8 up, 8 in 2024-09-21T14:07:24.956 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:24.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:24.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:07:25.057 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:07:25.090 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:25.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:25.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:07:25.191 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:07:25.226 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:25.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:25.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:07:25.328 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:25.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:25.369 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:25.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:25.470 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:25.509 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 32 16 59223 59207 115.616 87.125 0.0127117 0.0086432 2024-09-21T14:07:25.511 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:25.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:25.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:25.612 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:07:25.646 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:25.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:25.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:07:25.748 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:07:25.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:25 smithi049 ceph-mon[40489]: pgmap v943: 168 pgs: 168 active+clean; 3.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 99 MiB/s wr, 1.58k op/s 2024-09-21T14:07:25.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:25 smithi049 ceph-mon[30080]: pgmap v943: 168 pgs: 168 active+clean; 3.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 99 MiB/s wr, 1.58k op/s 2024-09-21T14:07:25.779 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:25.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:25.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:07:25.880 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:07:25.915 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:25.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:25.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:07:25.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:25 smithi187 ceph-mon[33032]: pgmap v943: 168 pgs: 168 active+clean; 3.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 99 MiB/s wr, 1.58k op/s 2024-09-21T14:07:26.016 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:07:26.059 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:26.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:26.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:07:26.160 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:07:26.192 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:26.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:26.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:07:26.294 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:07:26.322 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:26.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:26.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:07:26.424 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:07:26.457 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:26.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:26.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:07:26.510 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 33 16 60510 60494 114.55 80.4375 0.0173407 0.00872372 2024-09-21T14:07:26.559 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:07:26.603 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:26.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:26.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:07:26.707 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:07:26.728 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:26.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:26.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:07:26.830 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:26.867 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:26.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:26.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:26.969 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:27.123 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:27.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:27.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:27.225 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:07:27.262 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:27.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:27.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:07:27.364 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:07:27.404 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:27.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:27.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:07:27.506 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:07:27.510 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 34 16 61773 61757 113.502 78.9375 0.062237 0.008804 2024-09-21T14:07:27.535 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:27.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:27.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:07:27.637 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:07:27.658 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:27.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:27.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:07:27.761 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:07:27.792 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:27.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:27.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:07:27.848 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:27 smithi049 ceph-mon[40489]: pgmap v944: 168 pgs: 168 active+clean; 3.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 672 B/s rd, 84 MiB/s wr, 1.35k op/s 2024-09-21T14:07:27.849 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:27 smithi049 ceph-mon[30080]: pgmap v944: 168 pgs: 168 active+clean; 3.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 672 B/s rd, 84 MiB/s wr, 1.35k op/s 2024-09-21T14:07:27.894 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:07:27.924 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:27.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:27.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:07:27.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:27 smithi187 ceph-mon[33032]: pgmap v944: 168 pgs: 168 active+clean; 3.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 672 B/s rd, 84 MiB/s wr, 1.35k op/s 2024-09-21T14:07:28.025 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:07:28.053 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:28.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:28.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:07:28.155 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:07:28.181 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:28.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:28.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:07:28.283 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:07:28.314 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:28.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:28.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:07:28.416 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:07:28.455 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:28.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:28.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:07:28.510 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 35 16 62752 62736 112.007 61.1875 0.00979094 0.00892272 2024-09-21T14:07:28.556 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:07:28.587 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:28.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:28.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:07:28.689 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:28.726 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:28.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:28.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:28.829 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:07:28.877 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:07:28.881 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:28.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:28.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:07:28.983 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:29.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:29.026 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:29.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:29.127 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:07:29.162 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:29.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:29.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:07:29.264 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:07:29.311 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:29.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:29.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:07:29.412 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:07:29.465 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:29.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:29.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:07:29.510 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 36 16 64130 64114 111.288 86.125 0.00773238 0.00898024 2024-09-21T14:07:29.568 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:07:29.598 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:29.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:29.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:07:29.631 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:07:29.699 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:07:29.716 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:29 smithi049 ceph-mon[40489]: pgmap v945: 168 pgs: 168 active+clean; 3.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 88 MiB/s wr, 1.40k op/s 2024-09-21T14:07:29.716 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:29 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:07:29.716 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:29 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:07:29.717 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:07:29.717 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:07:29.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:29 smithi049 ceph-mon[30080]: pgmap v945: 168 pgs: 168 active+clean; 3.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 88 MiB/s wr, 1.40k op/s 2024-09-21T14:07:29.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:29 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:07:29.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:29 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:07:29.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:29.755 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:29.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:07:29.758 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:07:29.758 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:07:29.774 INFO:tasks.thrashosds.thrasher:Setting full ratio back to .95 2024-09-21T14:07:29.775 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set-full-ratio .95 2024-09-21T14:07:29.794 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:07:29.794 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:07:29.827 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:07:29.827 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:07:29.857 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:07:29.868 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:07:29.868 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:07:29.888 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:29.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:29.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:07:29.898 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:07:29.898 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:07:29.932 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:07:29.933 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:07:29.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:29 smithi187 ceph-mon[33032]: pgmap v945: 168 pgs: 168 active+clean; 3.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 88 MiB/s wr, 1.40k op/s 2024-09-21T14:07:29.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:29 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:07:29.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:29 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:07:29.967 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:07:29.967 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:07:29.989 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:07:30.016 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:30.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:30.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:07:30.118 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:07:30.153 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:30.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:30.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:07:30.255 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:07:30.298 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:30.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:30.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:07:30.400 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:07:30.421 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:30.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:30.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:07:30.444 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:07:30 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:07:30.510 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 37 16 65340 65324 110.324 75.625 0.0310787 0.00905884 2024-09-21T14:07:30.522 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:07:30.549 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:30.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:30.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:07:30.578 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:30 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1334425100' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:07:30.578 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:30 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1334425100' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:07:30.651 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:07:30.680 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:30.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:30.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:07:30.782 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:07:30.830 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:30.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:30.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:07:30.932 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:07:30.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:30 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1334425100' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:07:30.961 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:30.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:30.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:07:30.995 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:07:30 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:07:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:07:31.062 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:07:31.103 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:31.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:31.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:07:31.205 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:07:31.229 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:31.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:31.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:07:31.330 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:07:31.358 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:31.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:31.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:07:31.460 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:07:31.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:31.493 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:31.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:07:31.510 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 38 16 66418 66402 109.194 67.375 0.00843081 0.00915282 2024-09-21T14:07:31.578 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-09-21T14:07:31.593 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:07:31.621 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:31.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:31.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:07:31.722 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:07:31.753 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:31.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:31.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:07:31.855 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:07:31.900 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:31.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:31.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:07:31.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:31 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:07:31.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:31 smithi187 ceph-mon[33032]: pgmap v946: 168 pgs: 168 active+clean; 3.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 595 B/s rd, 88 MiB/s wr, 1.40k op/s 2024-09-21T14:07:31.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:31 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1231707132' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:07:32.002 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:07:32.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:31 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:07:32.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:31 smithi049 ceph-mon[40489]: pgmap v946: 168 pgs: 168 active+clean; 3.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 595 B/s rd, 88 MiB/s wr, 1.40k op/s 2024-09-21T14:07:32.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:31 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1231707132' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:07:32.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:31 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:07:32.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:31 smithi049 ceph-mon[30080]: pgmap v946: 168 pgs: 168 active+clean; 3.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 595 B/s rd, 88 MiB/s wr, 1.40k op/s 2024-09-21T14:07:32.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:31 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1231707132' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:07:32.032 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:32.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:32.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:07:32.136 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:07:32.167 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:32.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:32.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:07:32.269 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:07:32.308 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:32.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:32.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:07:32.410 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:07:32.448 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:32.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:32.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:07:32.510 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 39 16 67790 67774 108.593 85.75 0.00915237 0.00920257 2024-09-21T14:07:32.550 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:07:32.582 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:32.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:32.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:07:32.684 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:07:32.718 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:32.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:32.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:07:32.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:32 smithi049 ceph-mon[30080]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:07:32.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:32 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1231707132' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T14:07:32.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:32 smithi049 ceph-mon[30080]: osdmap e477: 8 total, 8 up, 8 in 2024-09-21T14:07:32.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:32 smithi049 ceph-mon[40489]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:07:32.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:32 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1231707132' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T14:07:32.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:32 smithi049 ceph-mon[40489]: osdmap e477: 8 total, 8 up, 8 in 2024-09-21T14:07:32.820 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:07:32.857 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:32.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:32.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:07:32.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:32 smithi187 ceph-mon[33032]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:07:32.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:32 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1231707132' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T14:07:32.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:32 smithi187 ceph-mon[33032]: osdmap e477: 8 total, 8 up, 8 in 2024-09-21T14:07:32.960 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:07:33.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:33.010 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:33.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:07:33.112 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:33.165 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:33.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:33.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:33.267 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:33.274 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T14:07:33.274 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:07:33.326 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:33.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:33.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:33.428 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:07:33.459 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:33.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:33.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:07:33.510 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:2024-09-21T14:07:33.511788+0000 min lat: 0.00119458 max lat: 0.215614 avg lat: 0.00925726 2024-09-21T14:07:33.510 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T14:07:33.510 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 40 16 69127 69111 107.967 83.5625 0.0181019 0.00925726 2024-09-21T14:07:33.560 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:07:33.588 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:33.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:33.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:07:33.690 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:33.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:33.730 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:33.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:33.831 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:07:33.863 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:33.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:33.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:07:33.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:33 smithi187 ceph-mon[33032]: pgmap v948: 168 pgs: 168 active+clean; 4.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 89 MiB/s wr, 1.43k op/s 2024-09-21T14:07:33.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:33 smithi187 ceph-mon[33032]: 4.3 scrub starts 2024-09-21T14:07:33.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:33 smithi187 ceph-mon[33032]: 4.3 scrub ok 2024-09-21T14:07:33.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:33 smithi187 ceph-mon[33032]: 4.7 scrub starts 2024-09-21T14:07:33.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:33 smithi187 ceph-mon[33032]: 4.7 scrub ok 2024-09-21T14:07:33.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:33 smithi187 ceph-mon[33032]: 3.1e scrub starts 2024-09-21T14:07:33.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:33 smithi187 ceph-mon[33032]: 3.1e scrub ok 2024-09-21T14:07:33.965 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:07:34.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:34.006 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:34.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:07:34.007 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:33 smithi049 ceph-mon[30080]: pgmap v948: 168 pgs: 168 active+clean; 4.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 89 MiB/s wr, 1.43k op/s 2024-09-21T14:07:34.007 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:33 smithi049 ceph-mon[30080]: 4.3 scrub starts 2024-09-21T14:07:34.007 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:33 smithi049 ceph-mon[30080]: 4.3 scrub ok 2024-09-21T14:07:34.007 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:33 smithi049 ceph-mon[30080]: 4.7 scrub starts 2024-09-21T14:07:34.007 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:33 smithi049 ceph-mon[30080]: 4.7 scrub ok 2024-09-21T14:07:34.007 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:33 smithi049 ceph-mon[30080]: 3.1e scrub starts 2024-09-21T14:07:34.007 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:33 smithi049 ceph-mon[30080]: 3.1e scrub ok 2024-09-21T14:07:34.008 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:33 smithi049 ceph-mon[40489]: pgmap v948: 168 pgs: 168 active+clean; 4.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 89 MiB/s wr, 1.43k op/s 2024-09-21T14:07:34.008 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:33 smithi049 ceph-mon[40489]: 4.3 scrub starts 2024-09-21T14:07:34.008 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:33 smithi049 ceph-mon[40489]: 4.3 scrub ok 2024-09-21T14:07:34.008 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:33 smithi049 ceph-mon[40489]: 4.7 scrub starts 2024-09-21T14:07:34.008 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:33 smithi049 ceph-mon[40489]: 4.7 scrub ok 2024-09-21T14:07:34.008 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:33 smithi049 ceph-mon[40489]: 3.1e scrub starts 2024-09-21T14:07:34.008 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:33 smithi049 ceph-mon[40489]: 3.1e scrub ok 2024-09-21T14:07:34.108 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:07:34.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:34.147 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:34.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:07:34.163 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:07:34.187 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T14:07:34.249 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:07:34.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:34.325 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:34.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:07:34.426 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:07:34.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:34.482 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:34.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:07:34.510 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 41 16 70430 70414 107.32 81.4375 0.0168328 0.00931218 2024-09-21T14:07:34.541 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T14:07:34.583 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:07:34.638 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:34.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:34.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:07:34.740 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:07:34.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:34 smithi049 ceph-mon[30080]: 4.15 scrub starts 2024-09-21T14:07:34.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:34 smithi049 ceph-mon[30080]: 4.15 scrub ok 2024-09-21T14:07:34.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:34 smithi049 ceph-mon[40489]: 4.15 scrub starts 2024-09-21T14:07:34.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:34 smithi049 ceph-mon[40489]: 4.15 scrub ok 2024-09-21T14:07:34.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:34.805 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:34.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:07:34.906 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:07:34.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:34 smithi187 ceph-mon[33032]: 4.15 scrub starts 2024-09-21T14:07:34.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:34 smithi187 ceph-mon[33032]: 4.15 scrub ok 2024-09-21T14:07:34.973 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:34.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:34.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:07:34.996 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:07:35.014 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T14:07:35.078 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:07:35.106 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:35.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:35.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:07:35.208 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:07:35.232 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:35.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:35.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:07:35.313 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:07:35.334 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:07:35.358 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:35.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:35.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:07:35.460 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:07:35.490 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:35.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:35.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:07:35.512 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 42 16 71448 71432 106.275 63.625 0.00833574 0.00940388 2024-09-21T14:07:35.591 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:35.634 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:35.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:35.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:35.736 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:07:35.767 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:35.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:35.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:07:35.868 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:07:35.898 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:35.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:35.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:07:35.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:35 smithi187 ceph-mon[33032]: pgmap v949: 168 pgs: 168 active+clean; 4.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 80 MiB/s wr, 1.29k op/s 2024-09-21T14:07:35.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:35 smithi187 ceph-mon[33032]: 1.12 scrub starts 2024-09-21T14:07:35.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:35 smithi187 ceph-mon[33032]: 1.12 scrub ok 2024-09-21T14:07:35.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:35 smithi187 ceph-mon[33032]: 6.1f scrub starts 2024-09-21T14:07:35.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:35 smithi187 ceph-mon[33032]: 6.1f scrub ok 2024-09-21T14:07:35.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:35 smithi187 ceph-mon[33032]: osdmap e478: 8 total, 8 up, 8 in 2024-09-21T14:07:35.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:35 smithi187 ceph-mon[33032]: 5.13 scrub starts 2024-09-21T14:07:35.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:35 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/4214999441' entity='client.admin' cmd={"prefix": "osd set-full-ratio", "ratio": 0.95} : dispatch 2024-09-21T14:07:35.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:35 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set-full-ratio", "ratio": 0.95} : dispatch 2024-09-21T14:07:35.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:35 smithi187 ceph-mon[33032]: 5.13 scrub ok 2024-09-21T14:07:35.999 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:07:36.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:35 smithi049 ceph-mon[40489]: pgmap v949: 168 pgs: 168 active+clean; 4.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 80 MiB/s wr, 1.29k op/s 2024-09-21T14:07:36.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:35 smithi049 ceph-mon[40489]: 1.12 scrub starts 2024-09-21T14:07:36.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:35 smithi049 ceph-mon[40489]: 1.12 scrub ok 2024-09-21T14:07:36.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:35 smithi049 ceph-mon[40489]: 6.1f scrub starts 2024-09-21T14:07:36.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:35 smithi049 ceph-mon[40489]: 6.1f scrub ok 2024-09-21T14:07:36.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:35 smithi049 ceph-mon[40489]: osdmap e478: 8 total, 8 up, 8 in 2024-09-21T14:07:36.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:35 smithi049 ceph-mon[40489]: 5.13 scrub starts 2024-09-21T14:07:36.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:35 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/4214999441' entity='client.admin' cmd={"prefix": "osd set-full-ratio", "ratio": 0.95} : dispatch 2024-09-21T14:07:36.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:35 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set-full-ratio", "ratio": 0.95} : dispatch 2024-09-21T14:07:36.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:35 smithi049 ceph-mon[40489]: 5.13 scrub ok 2024-09-21T14:07:36.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:35 smithi049 ceph-mon[30080]: pgmap v949: 168 pgs: 168 active+clean; 4.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 80 MiB/s wr, 1.29k op/s 2024-09-21T14:07:36.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:35 smithi049 ceph-mon[30080]: 1.12 scrub starts 2024-09-21T14:07:36.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:35 smithi049 ceph-mon[30080]: 1.12 scrub ok 2024-09-21T14:07:36.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:35 smithi049 ceph-mon[30080]: 6.1f scrub starts 2024-09-21T14:07:36.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:35 smithi049 ceph-mon[30080]: 6.1f scrub ok 2024-09-21T14:07:36.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:35 smithi049 ceph-mon[30080]: osdmap e478: 8 total, 8 up, 8 in 2024-09-21T14:07:36.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:35 smithi049 ceph-mon[30080]: 5.13 scrub starts 2024-09-21T14:07:36.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:35 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/4214999441' entity='client.admin' cmd={"prefix": "osd set-full-ratio", "ratio": 0.95} : dispatch 2024-09-21T14:07:36.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:35 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set-full-ratio", "ratio": 0.95} : dispatch 2024-09-21T14:07:36.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:35 smithi049 ceph-mon[30080]: 5.13 scrub ok 2024-09-21T14:07:36.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:36.021 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:36.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:07:36.122 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:07:36.151 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:36.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:36.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:07:36.229 INFO:teuthology.orchestra.run.smithi049.stderr:osd set-full-ratio 0.95 2024-09-21T14:07:36.252 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:07:36.275 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:36.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:36.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:07:36.377 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:07:36.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:36.411 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:36.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:07:36.512 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:36.513 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 43 16 72646 72630 105.545 74.875 0.0144319 0.00946904 2024-09-21T14:07:36.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:36.555 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:36.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:36.656 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:07:36.691 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:36.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:36.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:07:36.792 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:36.833 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:36.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:36.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:36.879 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:36 smithi049 ceph-mon[40489]: 3.e scrub starts 2024-09-21T14:07:36.879 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:36 smithi049 ceph-mon[40489]: 3.e scrub ok 2024-09-21T14:07:36.879 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:36 smithi049 ceph-mon[40489]: 1.8 scrub starts 2024-09-21T14:07:36.879 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:36 smithi049 ceph-mon[40489]: 1.8 scrub ok 2024-09-21T14:07:36.879 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:36 smithi049 ceph-mon[40489]: 3.1a scrub starts 2024-09-21T14:07:36.879 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:36 smithi049 ceph-mon[40489]: 3.1a scrub ok 2024-09-21T14:07:36.879 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:36 smithi049 ceph-mon[40489]: Health check cleared: OSD_OUT_OF_ORDER_FULL (was: full ratio(s) out of order) 2024-09-21T14:07:36.879 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:36 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set-full-ratio", "ratio": 0.95}]': finished 2024-09-21T14:07:36.879 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:36 smithi049 ceph-mon[40489]: osdmap e479: 8 total, 8 up, 8 in 2024-09-21T14:07:36.880 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:36 smithi049 ceph-mon[30080]: 3.e scrub starts 2024-09-21T14:07:36.880 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:36 smithi049 ceph-mon[30080]: 3.e scrub ok 2024-09-21T14:07:36.880 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:36 smithi049 ceph-mon[30080]: 1.8 scrub starts 2024-09-21T14:07:36.880 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:36 smithi049 ceph-mon[30080]: 1.8 scrub ok 2024-09-21T14:07:36.880 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:36 smithi049 ceph-mon[30080]: 3.1a scrub starts 2024-09-21T14:07:36.880 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:36 smithi049 ceph-mon[30080]: 3.1a scrub ok 2024-09-21T14:07:36.880 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:36 smithi049 ceph-mon[30080]: Health check cleared: OSD_OUT_OF_ORDER_FULL (was: full ratio(s) out of order) 2024-09-21T14:07:36.880 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:36 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set-full-ratio", "ratio": 0.95}]': finished 2024-09-21T14:07:36.880 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:36 smithi049 ceph-mon[30080]: osdmap e479: 8 total, 8 up, 8 in 2024-09-21T14:07:36.936 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:07:36.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:36 smithi187 ceph-mon[33032]: 3.e scrub starts 2024-09-21T14:07:36.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:36 smithi187 ceph-mon[33032]: 3.e scrub ok 2024-09-21T14:07:36.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:36 smithi187 ceph-mon[33032]: 1.8 scrub starts 2024-09-21T14:07:36.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:36 smithi187 ceph-mon[33032]: 1.8 scrub ok 2024-09-21T14:07:36.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:36 smithi187 ceph-mon[33032]: 3.1a scrub starts 2024-09-21T14:07:36.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:36 smithi187 ceph-mon[33032]: 3.1a scrub ok 2024-09-21T14:07:36.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:36 smithi187 ceph-mon[33032]: Health check cleared: OSD_OUT_OF_ORDER_FULL (was: full ratio(s) out of order) 2024-09-21T14:07:36.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:36 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set-full-ratio", "ratio": 0.95}]': finished 2024-09-21T14:07:36.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:36 smithi187 ceph-mon[33032]: osdmap e479: 8 total, 8 up, 8 in 2024-09-21T14:07:36.974 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:36.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:36.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:07:37.076 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:07:37.106 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:37.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:37.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:07:37.208 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:07:37.228 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:37.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:37.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:07:37.330 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:07:37.351 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:37.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:37.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:07:37.453 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:07:37.484 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:37.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:37.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:07:37.512 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 44 16 73579 73563 104.471 58.3125 0.00944974 0.0095671 2024-09-21T14:07:37.587 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:07:37.611 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:37.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:37.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:07:37.713 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:07:37.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:37.752 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:37.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:07:37.852 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:07:37.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:37.892 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:37.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:07:37.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:37 smithi187 ceph-mon[33032]: pgmap v951: 168 pgs: 168 active+clean; 4.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-09-21T14:07:37.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:37 smithi187 ceph-mon[33032]: 5.9 scrub starts 2024-09-21T14:07:37.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:37 smithi187 ceph-mon[33032]: 5.9 scrub ok 2024-09-21T14:07:37.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:37 smithi187 ceph-mon[33032]: 4.18 scrub starts 2024-09-21T14:07:37.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:37 smithi187 ceph-mon[33032]: 4.18 scrub ok 2024-09-21T14:07:37.993 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:07:38.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:37 smithi049 ceph-mon[30080]: pgmap v951: 168 pgs: 168 active+clean; 4.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-09-21T14:07:38.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:37 smithi049 ceph-mon[30080]: 5.9 scrub starts 2024-09-21T14:07:38.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:37 smithi049 ceph-mon[30080]: 5.9 scrub ok 2024-09-21T14:07:38.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:37 smithi049 ceph-mon[30080]: 4.18 scrub starts 2024-09-21T14:07:38.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:37 smithi049 ceph-mon[30080]: 4.18 scrub ok 2024-09-21T14:07:38.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:37 smithi049 ceph-mon[40489]: pgmap v951: 168 pgs: 168 active+clean; 4.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-09-21T14:07:38.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:37 smithi049 ceph-mon[40489]: 5.9 scrub starts 2024-09-21T14:07:38.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:37 smithi049 ceph-mon[40489]: 5.9 scrub ok 2024-09-21T14:07:38.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:37 smithi049 ceph-mon[40489]: 4.18 scrub starts 2024-09-21T14:07:38.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:37 smithi049 ceph-mon[40489]: 4.18 scrub ok 2024-09-21T14:07:38.033 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:38.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:38.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:07:38.135 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:07:38.162 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T14:07:38.162 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:07:38.176 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:38.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:38.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:07:38.278 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:07:38.309 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:38.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:38.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:07:38.411 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:38.458 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:38.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:38.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:38.513 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 45 16 74753 74737 103.779 73.375 0.0182299 0.00963073 2024-09-21T14:07:38.560 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:38.601 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:38.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:38.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:38.703 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:07:38.738 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:38.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:38.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:07:38.839 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:07:38.871 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:38.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:38.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:07:38.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:38 smithi187 ceph-mon[33032]: 3.7 scrub starts 2024-09-21T14:07:38.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:38 smithi187 ceph-mon[33032]: 3.7 scrub ok 2024-09-21T14:07:38.972 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:07:39.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:38 smithi049 ceph-mon[30080]: 3.7 scrub starts 2024-09-21T14:07:39.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:38 smithi049 ceph-mon[30080]: 3.7 scrub ok 2024-09-21T14:07:39.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:38 smithi049 ceph-mon[40489]: 3.7 scrub starts 2024-09-21T14:07:39.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:38 smithi049 ceph-mon[40489]: 3.7 scrub ok 2024-09-21T14:07:39.006 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:39.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:39.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:07:39.108 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:07:39.148 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:39.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:39.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:07:39.250 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:07:39.293 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:39.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:39.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:07:39.395 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:07:39.423 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:39.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:39.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:07:39.513 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 46 16 75939 75923 103.134 74.125 0.0114708 0.00969103 2024-09-21T14:07:39.524 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:39.556 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T14:07:39.569 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:39.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:39.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:39.673 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:07:39.703 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:39.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:39.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:07:39.746 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:39 smithi049 ceph-mon[40489]: pgmap v953: 168 pgs: 168 active+clean; 4.4 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 782 B/s rd, 59 MiB/s wr, 948 op/s 2024-09-21T14:07:39.747 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:39 smithi049 ceph-mon[30080]: pgmap v953: 168 pgs: 168 active+clean; 4.4 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 782 B/s rd, 59 MiB/s wr, 948 op/s 2024-09-21T14:07:39.805 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:07:39.833 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:39.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:39.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:07:39.934 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:07:39.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:39 smithi187 ceph-mon[33032]: pgmap v953: 168 pgs: 168 active+clean; 4.4 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 782 B/s rd, 59 MiB/s wr, 948 op/s 2024-09-21T14:07:39.966 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:39.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:39.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:07:40.067 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:07:40.092 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:40.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:40.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:07:40.194 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:40.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:40.246 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:40.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:40.347 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:07:40.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:40.395 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:40.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:07:40.444 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:07:40 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:07:40.496 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:07:40.514 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 47 16 77072 77056 102.445 70.8125 0.029616 0.0097548 2024-09-21T14:07:40.518 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:40.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:40.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:07:40.585 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:07:40.600 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:07:40.622 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:07:40.645 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:40.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:40.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:07:40.735 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:07:40 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:07:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:07:40.736 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:40 smithi049 ceph-mon[40489]: 6.7 scrub starts 2024-09-21T14:07:40.736 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:40 smithi049 ceph-mon[40489]: 6.7 scrub ok 2024-09-21T14:07:40.736 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:40 smithi049 ceph-mon[40489]: osdmap e480: 8 total, 8 up, 8 in 2024-09-21T14:07:40.736 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:40 smithi049 ceph-mon[40489]: 4.14 scrub starts 2024-09-21T14:07:40.736 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:40 smithi049 ceph-mon[40489]: 4.14 scrub ok 2024-09-21T14:07:40.736 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:40 smithi049 ceph-mon[30080]: 6.7 scrub starts 2024-09-21T14:07:40.736 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:40 smithi049 ceph-mon[30080]: 6.7 scrub ok 2024-09-21T14:07:40.736 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:40 smithi049 ceph-mon[30080]: osdmap e480: 8 total, 8 up, 8 in 2024-09-21T14:07:40.736 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:40 smithi049 ceph-mon[30080]: 4.14 scrub starts 2024-09-21T14:07:40.737 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:40 smithi049 ceph-mon[30080]: 4.14 scrub ok 2024-09-21T14:07:40.747 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:07:40.775 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:40.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:40.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:07:40.877 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:40.916 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:40.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:40.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:40.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:40 smithi187 ceph-mon[33032]: 6.7 scrub starts 2024-09-21T14:07:40.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:40 smithi187 ceph-mon[33032]: 6.7 scrub ok 2024-09-21T14:07:40.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:40 smithi187 ceph-mon[33032]: osdmap e480: 8 total, 8 up, 8 in 2024-09-21T14:07:40.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:40 smithi187 ceph-mon[33032]: 4.14 scrub starts 2024-09-21T14:07:40.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:40 smithi187 ceph-mon[33032]: 4.14 scrub ok 2024-09-21T14:07:41.018 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:07:41.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:41.045 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:41.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:07:41.145 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:07:41.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:41.183 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:41.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:07:41.284 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:07:41.311 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:41.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:41.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:07:41.412 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:07:41.456 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:41.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:41.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:07:41.514 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 48 16 78230 78214 101.818 72.375 0.00597092 0.0098161 2024-09-21T14:07:41.559 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:07:41.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:41.604 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:41.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:07:41.678 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-09-21T14:07:41.706 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:07:41.732 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:41.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:41.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:07:41.805 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:07:41.805 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:07:41.812 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T14:07:41.813 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:07:41.834 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:07:41.856 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:07:41.856 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:07:41.856 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:41 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:07:41.856 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:41 smithi049 ceph-mon[40489]: pgmap v954: 168 pgs: 168 active+clean; 4.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 68 MiB/s wr, 1.08k op/s 2024-09-21T14:07:41.856 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:41 smithi049 ceph-mon[40489]: 3.11 scrub starts 2024-09-21T14:07:41.856 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:41 smithi049 ceph-mon[40489]: 3.11 scrub ok 2024-09-21T14:07:41.856 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:41 smithi049 ceph-mon[40489]: 3.5 scrub starts 2024-09-21T14:07:41.856 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:41 smithi049 ceph-mon[40489]: 3.5 scrub ok 2024-09-21T14:07:41.857 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:41 smithi049 ceph-mon[40489]: 6.1d scrub starts 2024-09-21T14:07:41.857 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:41 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3173986294' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:07:41.857 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:41 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:07:41.857 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:41 smithi049 ceph-mon[40489]: 6.1d scrub ok 2024-09-21T14:07:41.857 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:41 smithi049 ceph-mon[40489]: 3.8 scrub starts 2024-09-21T14:07:41.857 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:41 smithi049 ceph-mon[40489]: 3.8 scrub ok 2024-09-21T14:07:41.857 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:41 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:07:41.857 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:41 smithi049 ceph-mon[30080]: pgmap v954: 168 pgs: 168 active+clean; 4.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 68 MiB/s wr, 1.08k op/s 2024-09-21T14:07:41.857 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:41 smithi049 ceph-mon[30080]: 3.11 scrub starts 2024-09-21T14:07:41.857 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:41 smithi049 ceph-mon[30080]: 3.11 scrub ok 2024-09-21T14:07:41.857 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:41 smithi049 ceph-mon[30080]: 3.5 scrub starts 2024-09-21T14:07:41.858 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:41 smithi049 ceph-mon[30080]: 3.5 scrub ok 2024-09-21T14:07:41.858 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:41 smithi049 ceph-mon[30080]: 6.1d scrub starts 2024-09-21T14:07:41.858 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:41 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3173986294' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:07:41.858 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:41 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:07:41.858 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:41 smithi049 ceph-mon[30080]: 6.1d scrub ok 2024-09-21T14:07:41.858 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:41 smithi049 ceph-mon[30080]: 3.8 scrub starts 2024-09-21T14:07:41.858 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:41 smithi049 ceph-mon[30080]: 3.8 scrub ok 2024-09-21T14:07:41.862 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:41.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:41.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:07:41.890 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:07:41.890 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:07:41.924 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:07:41.924 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:07:41.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:41 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:07:41.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:41 smithi187 ceph-mon[33032]: pgmap v954: 168 pgs: 168 active+clean; 4.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 68 MiB/s wr, 1.08k op/s 2024-09-21T14:07:41.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:41 smithi187 ceph-mon[33032]: 3.11 scrub starts 2024-09-21T14:07:41.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:41 smithi187 ceph-mon[33032]: 3.11 scrub ok 2024-09-21T14:07:41.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:41 smithi187 ceph-mon[33032]: 3.5 scrub starts 2024-09-21T14:07:41.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:41 smithi187 ceph-mon[33032]: 3.5 scrub ok 2024-09-21T14:07:41.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:41 smithi187 ceph-mon[33032]: 6.1d scrub starts 2024-09-21T14:07:41.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:41 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3173986294' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:07:41.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:41 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:07:41.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:41 smithi187 ceph-mon[33032]: 6.1d scrub ok 2024-09-21T14:07:41.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:41 smithi187 ceph-mon[33032]: 3.8 scrub starts 2024-09-21T14:07:41.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:41 smithi187 ceph-mon[33032]: 3.8 scrub ok 2024-09-21T14:07:41.960 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:07:41.961 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:07:41.964 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:07:41.994 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:07:41.994 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:07:41.995 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:41.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:41.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:07:42.031 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:07:42.031 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:07:42.068 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:07:42.068 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:07:42.097 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:07:42.122 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:42.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:42.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:07:42.223 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:07:42.255 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:42.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:42.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:07:42.357 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:07:42.391 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:42.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:42.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:07:42.493 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:07:42.514 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 49 16 79415 79399 101.252 74.0625 0.020207 0.00987097 2024-09-21T14:07:42.520 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:42.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:42.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:07:42.621 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:07:42.663 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:42.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:42.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:07:42.764 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:42.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:42.802 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:42.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:42.903 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:07:42.932 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:42.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:42.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:07:42.993 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:42 smithi049 ceph-mon[30080]: 6.f scrub starts 2024-09-21T14:07:42.993 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:42 smithi049 ceph-mon[30080]: 6.f scrub ok 2024-09-21T14:07:42.993 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:42 smithi049 ceph-mon[30080]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T14:07:42.993 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:42 smithi049 ceph-mon[30080]: Cluster is now healthy 2024-09-21T14:07:42.993 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:42 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T14:07:42.993 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:42 smithi049 ceph-mon[30080]: osdmap e481: 8 total, 8 up, 8 in 2024-09-21T14:07:42.993 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:42 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2553239399' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:07:42.993 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:42 smithi049 ceph-mon[30080]: 5.c scrub starts 2024-09-21T14:07:42.993 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:42 smithi049 ceph-mon[30080]: 5.c scrub ok 2024-09-21T14:07:42.993 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:42 smithi049 ceph-mon[30080]: 4.0 scrub starts 2024-09-21T14:07:42.993 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:42 smithi049 ceph-mon[30080]: 4.0 scrub ok 2024-09-21T14:07:42.994 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:42 smithi049 ceph-mon[40489]: 6.f scrub starts 2024-09-21T14:07:42.994 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:42 smithi049 ceph-mon[40489]: 6.f scrub ok 2024-09-21T14:07:42.994 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:42 smithi049 ceph-mon[40489]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T14:07:42.994 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:42 smithi049 ceph-mon[40489]: Cluster is now healthy 2024-09-21T14:07:42.994 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:42 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T14:07:42.994 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:42 smithi049 ceph-mon[40489]: osdmap e481: 8 total, 8 up, 8 in 2024-09-21T14:07:42.994 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:42 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2553239399' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:07:42.994 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:42 smithi049 ceph-mon[40489]: 5.c scrub starts 2024-09-21T14:07:42.994 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:42 smithi049 ceph-mon[40489]: 5.c scrub ok 2024-09-21T14:07:42.994 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:42 smithi049 ceph-mon[40489]: 4.0 scrub starts 2024-09-21T14:07:42.994 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:42 smithi049 ceph-mon[40489]: 4.0 scrub ok 2024-09-21T14:07:43.034 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:07:43.073 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:43.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:43.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:07:43.175 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:07:43.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:42 smithi187 ceph-mon[33032]: 6.f scrub starts 2024-09-21T14:07:43.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:42 smithi187 ceph-mon[33032]: 6.f scrub ok 2024-09-21T14:07:43.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:42 smithi187 ceph-mon[33032]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T14:07:43.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:42 smithi187 ceph-mon[33032]: Cluster is now healthy 2024-09-21T14:07:43.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:42 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T14:07:43.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:42 smithi187 ceph-mon[33032]: osdmap e481: 8 total, 8 up, 8 in 2024-09-21T14:07:43.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:42 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2553239399' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:07:43.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:42 smithi187 ceph-mon[33032]: 5.c scrub starts 2024-09-21T14:07:43.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:42 smithi187 ceph-mon[33032]: 5.c scrub ok 2024-09-21T14:07:43.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:42 smithi187 ceph-mon[33032]: 4.0 scrub starts 2024-09-21T14:07:43.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:42 smithi187 ceph-mon[33032]: 4.0 scrub ok 2024-09-21T14:07:43.216 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:43.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:43.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:07:43.319 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:07:43.348 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:43.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:43.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:07:43.450 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:07:43.497 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:43.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:43.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:07:43.514 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 50 16 80696 80680 100.828 80.0625 0.0102906 0.00991281 2024-09-21T14:07:43.599 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:43.639 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:43.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:43.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:43.741 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:07:43.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:43.771 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:43.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:07:43.872 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:07:43.901 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:43.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:43.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:07:44.003 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:07:44.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:43 smithi049 ceph-mon[40489]: pgmap v957: 168 pgs: 168 active+clean; 4.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 91 MiB/s wr, 1.45k op/s 2024-09-21T14:07:44.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:43 smithi049 ceph-mon[40489]: 3.15 scrub starts 2024-09-21T14:07:44.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:43 smithi049 ceph-mon[40489]: 3.15 scrub ok 2024-09-21T14:07:44.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:43 smithi049 ceph-mon[40489]: 5.1b scrub starts 2024-09-21T14:07:44.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:43 smithi049 ceph-mon[40489]: 5.1b scrub ok 2024-09-21T14:07:44.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:43 smithi049 ceph-mon[40489]: 6.1 scrub starts 2024-09-21T14:07:44.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:43 smithi049 ceph-mon[40489]: 6.1 scrub ok 2024-09-21T14:07:44.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:43 smithi049 ceph-mon[30080]: pgmap v957: 168 pgs: 168 active+clean; 4.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 91 MiB/s wr, 1.45k op/s 2024-09-21T14:07:44.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:43 smithi049 ceph-mon[30080]: 3.15 scrub starts 2024-09-21T14:07:44.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:43 smithi049 ceph-mon[30080]: 3.15 scrub ok 2024-09-21T14:07:44.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:43 smithi049 ceph-mon[30080]: 5.1b scrub starts 2024-09-21T14:07:44.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:43 smithi049 ceph-mon[30080]: 5.1b scrub ok 2024-09-21T14:07:44.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:43 smithi049 ceph-mon[30080]: 6.1 scrub starts 2024-09-21T14:07:44.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:43 smithi049 ceph-mon[30080]: 6.1 scrub ok 2024-09-21T14:07:44.040 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:44.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:44.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:07:44.142 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:07:44.175 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:44.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:44.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:07:44.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:43 smithi187 ceph-mon[33032]: pgmap v957: 168 pgs: 168 active+clean; 4.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 91 MiB/s wr, 1.45k op/s 2024-09-21T14:07:44.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:43 smithi187 ceph-mon[33032]: 3.15 scrub starts 2024-09-21T14:07:44.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:43 smithi187 ceph-mon[33032]: 3.15 scrub ok 2024-09-21T14:07:44.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:43 smithi187 ceph-mon[33032]: 5.1b scrub starts 2024-09-21T14:07:44.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:43 smithi187 ceph-mon[33032]: 5.1b scrub ok 2024-09-21T14:07:44.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:43 smithi187 ceph-mon[33032]: 6.1 scrub starts 2024-09-21T14:07:44.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:43 smithi187 ceph-mon[33032]: 6.1 scrub ok 2024-09-21T14:07:44.277 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:07:44.305 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:44.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:44.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:07:44.343 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T14:07:44.407 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:44.446 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:44.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:44.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:44.514 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 51 16 82062 82046 100.525 85.375 0.0090249 0.00994237 2024-09-21T14:07:44.548 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:07:44.587 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:44.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:44.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:07:44.689 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:07:44.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:44.730 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:44.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:07:44.771 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:07:44.831 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:07:44.866 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:44.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:44.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:07:44.967 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:07:45.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:44 smithi049 ceph-mon[40489]: 2.1 scrub starts 2024-09-21T14:07:45.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:44 smithi049 ceph-mon[40489]: 2.1 scrub ok 2024-09-21T14:07:45.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:44 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:07:45.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:44 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:07:45.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:44 smithi049 ceph-mon[30080]: 2.1 scrub starts 2024-09-21T14:07:45.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:44 smithi049 ceph-mon[30080]: 2.1 scrub ok 2024-09-21T14:07:45.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:44 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:07:45.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:44 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:07:45.005 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:45.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:45.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:07:45.045 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T14:07:45.066 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:44 smithi187 ceph-mon[33032]: 2.1 scrub starts 2024-09-21T14:07:45.066 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:44 smithi187 ceph-mon[33032]: 2.1 scrub ok 2024-09-21T14:07:45.066 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:44 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:07:45.066 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:44 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:07:45.108 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:45.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:45.155 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:45.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:45.156 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 4, 6, 2, 3, 7, 5] 2024-09-21T14:07:45.156 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T14:07:45.156 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-21T14:07:45.156 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd dump -f json-pretty 2024-09-21T14:07:45.256 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:45.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:45.303 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:45.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:45.404 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:07:45.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:45.435 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:45.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:07:45.515 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 52 16 83274 83258 100.047 75.75 0.00657606 0.00998991 2024-09-21T14:07:45.520 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:07:45.536 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:07:45.563 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:45.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:45.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:07:45.665 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:07:45.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:45.714 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:45.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:07:45.802 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T14:07:45.814 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:45.894 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:45.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:45.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:45.996 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:46.011 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:45 smithi049 ceph-mon[40489]: pgmap v958: 168 pgs: 168 active+clean; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 80 MiB/s wr, 1.29k op/s 2024-09-21T14:07:46.011 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:45 smithi049 ceph-mon[40489]: 6.1a scrub starts 2024-09-21T14:07:46.011 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:45 smithi049 ceph-mon[40489]: 6.1a scrub ok 2024-09-21T14:07:46.011 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:45 smithi049 ceph-mon[40489]: osdmap e482: 8 total, 8 up, 8 in 2024-09-21T14:07:46.011 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:45 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T14:07:46.011 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:45 smithi049 ceph-mon[30080]: pgmap v958: 168 pgs: 168 active+clean; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 80 MiB/s wr, 1.29k op/s 2024-09-21T14:07:46.012 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:45 smithi049 ceph-mon[30080]: 6.1a scrub starts 2024-09-21T14:07:46.012 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:45 smithi049 ceph-mon[30080]: 6.1a scrub ok 2024-09-21T14:07:46.012 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:45 smithi049 ceph-mon[30080]: osdmap e482: 8 total, 8 up, 8 in 2024-09-21T14:07:46.012 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:45 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T14:07:46.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:46.077 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:46.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:46.081 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:45 smithi187 ceph-mon[33032]: pgmap v958: 168 pgs: 168 active+clean; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 80 MiB/s wr, 1.29k op/s 2024-09-21T14:07:46.081 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:45 smithi187 ceph-mon[33032]: 6.1a scrub starts 2024-09-21T14:07:46.081 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:45 smithi187 ceph-mon[33032]: 6.1a scrub ok 2024-09-21T14:07:46.081 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:45 smithi187 ceph-mon[33032]: osdmap e482: 8 total, 8 up, 8 in 2024-09-21T14:07:46.081 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:45 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T14:07:46.178 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:07:46.202 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:46.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:46.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:07:46.304 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:07:46.336 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:46.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:46.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:07:46.381 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T14:07:46.381 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-21T14:07:46.381 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.381 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26396.0:83930 11.5 11:af0384bc:::benchmark_data_smithi049_206132_object83929:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e482)", 2024-09-21T14:07:46.381 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:07:46.145387+0000", 2024-09-21T14:07:46.382 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.053570628000000002, 2024-09-21T14:07:46.382 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.053605071999999997, 2024-09-21T14:07:46.382 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:07:46.382 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:07:46.382 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-21T14:07:46.382 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:07:46.382 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26396", 2024-09-21T14:07:46.382 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:07:46.382 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 83930 2024-09-21T14:07:46.382 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.382 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:07:46.382 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.382 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:07:46.382 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.145387+0000", 2024-09-21T14:07:46.382 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:07:46.383 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.383 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.383 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:07:46.383 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.145387+0000", 2024-09-21T14:07:46.383 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:07:46.383 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.383 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.383 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:07:46.383 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.145389+0000", 2024-09-21T14:07:46.383 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4619999999999999e-06 2024-09-21T14:07:46.383 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.383 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.383 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:07:46.383 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.145421+0000", 2024-09-21T14:07:46.383 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2146e-05 2024-09-21T14:07:46.384 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.384 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.384 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:07:46.384 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.145432+0000", 2024-09-21T14:07:46.384 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0599e-05 2024-09-21T14:07:46.384 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.384 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.384 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:07:46.384 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.145438+0000", 2024-09-21T14:07:46.384 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6340000000000003e-06 2024-09-21T14:07:46.384 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.384 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.384 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:07:46.384 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.145505+0000", 2024-09-21T14:07:46.385 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7156999999999994e-05 2024-09-21T14:07:46.385 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.385 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.385 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:07:46.385 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.145647+0000", 2024-09-21T14:07:46.385 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000141936 2024-09-21T14:07:46.385 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.385 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.385 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 5,6", 2024-09-21T14:07:46.385 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.145789+0000", 2024-09-21T14:07:46.385 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000142072 2024-09-21T14:07:46.385 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.385 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.386 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:46.386 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.181948+0000", 2024-09-21T14:07:46.386 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.036158539000000003 2024-09-21T14:07:46.386 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.386 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.386 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:07:46.386 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.182004+0000", 2024-09-21T14:07:46.386 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5989000000000002e-05 2024-09-21T14:07:46.386 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:46.386 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:07:46.386 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:46.386 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.386 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.386 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26396.0:83931 11.5 11:ad7b19c4:::benchmark_data_smithi049_206132_object83930:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e482)", 2024-09-21T14:07:46.386 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:07:46.145488+0000", 2024-09-21T14:07:46.387 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.053470050999999998, 2024-09-21T14:07:46.387 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.053557625999999997, 2024-09-21T14:07:46.387 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:07:46.387 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:07:46.387 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-21T14:07:46.387 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:07:46.387 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26396", 2024-09-21T14:07:46.387 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:07:46.387 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 83931 2024-09-21T14:07:46.387 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.387 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:07:46.387 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.387 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:07:46.387 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.145488+0000", 2024-09-21T14:07:46.387 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:07:46.388 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.388 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.388 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:07:46.388 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.145488+0000", 2024-09-21T14:07:46.388 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:07:46.388 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.388 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.388 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:07:46.388 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.145489+0000", 2024-09-21T14:07:46.388 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1009999999999999e-06 2024-09-21T14:07:46.388 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.388 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.388 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:07:46.388 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.145534+0000", 2024-09-21T14:07:46.388 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4845999999999997e-05 2024-09-21T14:07:46.389 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.389 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.389 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:07:46.389 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.145543+0000", 2024-09-21T14:07:46.389 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0250000000000008e-06 2024-09-21T14:07:46.389 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.389 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.389 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:07:46.389 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.145548+0000", 2024-09-21T14:07:46.389 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5620000000000003e-06 2024-09-21T14:07:46.389 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.389 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.389 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:07:46.389 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.148169+0000", 2024-09-21T14:07:46.389 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0026206179999999999 2024-09-21T14:07:46.390 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.390 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.390 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:07:46.390 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.148322+0000", 2024-09-21T14:07:46.390 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015294199999999999 2024-09-21T14:07:46.390 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.390 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.390 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 5,6", 2024-09-21T14:07:46.390 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.148478+0000", 2024-09-21T14:07:46.390 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015584299999999999 2024-09-21T14:07:46.390 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.390 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.390 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:46.390 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.181993+0000", 2024-09-21T14:07:46.390 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.033514926 2024-09-21T14:07:46.391 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.391 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.391 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:07:46.391 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.182008+0000", 2024-09-21T14:07:46.391 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5446000000000001e-05 2024-09-21T14:07:46.391 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:46.391 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:07:46.391 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:46.391 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.391 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.391 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26396.0:83933 11.4 11:28bd7f1c:::benchmark_data_smithi049_206132_object83932:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e482)", 2024-09-21T14:07:46.391 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:07:46.148423+0000", 2024-09-21T14:07:46.391 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.050534552000000003, 2024-09-21T14:07:46.391 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.050662604999999999, 2024-09-21T14:07:46.391 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:07:46.391 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:07:46.392 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-21T14:07:46.392 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:07:46.392 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26396", 2024-09-21T14:07:46.392 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:07:46.392 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 83933 2024-09-21T14:07:46.392 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.392 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:07:46.392 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.392 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:07:46.392 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.148423+0000", 2024-09-21T14:07:46.392 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:07:46.392 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.392 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.392 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:07:46.392 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.148423+0000", 2024-09-21T14:07:46.392 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:07:46.393 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.393 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.393 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:07:46.393 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.148425+0000", 2024-09-21T14:07:46.393 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.685e-06 2024-09-21T14:07:46.393 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.393 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.393 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:07:46.393 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.148489+0000", 2024-09-21T14:07:46.393 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3874000000000005e-05 2024-09-21T14:07:46.393 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.393 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.393 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:07:46.393 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.148499+0000", 2024-09-21T14:07:46.393 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0103e-05 2024-09-21T14:07:46.393 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.394 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.394 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:07:46.394 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.148505+0000", 2024-09-21T14:07:46.394 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6609999999999997e-06 2024-09-21T14:07:46.394 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.394 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.394 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:07:46.394 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.148560+0000", 2024-09-21T14:07:46.394 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5053999999999998e-05 2024-09-21T14:07:46.394 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.394 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.394 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:07:46.394 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.148679+0000", 2024-09-21T14:07:46.394 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000119061 2024-09-21T14:07:46.394 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.395 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.395 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,6", 2024-09-21T14:07:46.395 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.148815+0000", 2024-09-21T14:07:46.395 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000136172 2024-09-21T14:07:46.395 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.395 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.395 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:07:46.395 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.183903+0000", 2024-09-21T14:07:46.395 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.035087972000000002 2024-09-21T14:07:46.395 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.395 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.395 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:46.395 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.183958+0000", 2024-09-21T14:07:46.395 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4803999999999999e-05 2024-09-21T14:07:46.396 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:46.396 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:07:46.396 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:46.396 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.396 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.396 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26396.0:83934 11.4 11:2050172c:::benchmark_data_smithi049_206132_object83933:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e482)", 2024-09-21T14:07:46.396 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:07:46.149170+0000", 2024-09-21T14:07:46.396 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.049787527999999998, 2024-09-21T14:07:46.396 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.049955962999999999, 2024-09-21T14:07:46.396 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:07:46.396 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:07:46.396 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-21T14:07:46.396 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:07:46.396 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26396", 2024-09-21T14:07:46.396 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:07:46.396 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 83934 2024-09-21T14:07:46.397 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.397 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:07:46.397 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.397 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:07:46.397 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.149170+0000", 2024-09-21T14:07:46.397 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:07:46.397 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.397 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.397 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:07:46.397 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.149170+0000", 2024-09-21T14:07:46.397 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:07:46.397 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.397 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.397 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:07:46.397 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.149171+0000", 2024-09-21T14:07:46.398 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6599999999999996e-07 2024-09-21T14:07:46.398 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.398 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.398 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:07:46.398 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.149230+0000", 2024-09-21T14:07:46.398 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9110000000000002e-05 2024-09-21T14:07:46.398 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.398 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.398 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:07:46.398 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.149239+0000", 2024-09-21T14:07:46.398 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1630000000000002e-06 2024-09-21T14:07:46.398 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.398 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.398 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:07:46.398 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.149245+0000", 2024-09-21T14:07:46.399 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1250000000000001e-06 2024-09-21T14:07:46.399 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.399 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.399 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:07:46.399 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.181865+0000", 2024-09-21T14:07:46.399 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.032620623000000001 2024-09-21T14:07:46.399 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.399 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.399 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:07:46.399 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.181983+0000", 2024-09-21T14:07:46.399 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011765900000000001 2024-09-21T14:07:46.399 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.399 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.399 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,6", 2024-09-21T14:07:46.399 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.182074+0000", 2024-09-21T14:07:46.399 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.077e-05 2024-09-21T14:07:46.400 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.400 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.400 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:07:46.400 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.184146+0000", 2024-09-21T14:07:46.400 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0020728740000000002 2024-09-21T14:07:46.400 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.400 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.400 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:46.400 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.184297+0000", 2024-09-21T14:07:46.400 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000150245 2024-09-21T14:07:46.400 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:46.400 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:07:46.400 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:46.400 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.400 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.400 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26396.0:83944 11.5 11:ab18421a:::benchmark_data_smithi049_206132_object83943:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e482)", 2024-09-21T14:07:46.401 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:07:46.184327+0000", 2024-09-21T14:07:46.401 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.014630674, 2024-09-21T14:07:46.401 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.014837688999999999, 2024-09-21T14:07:46.401 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:07:46.401 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:07:46.401 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-21T14:07:46.401 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:07:46.401 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26396", 2024-09-21T14:07:46.401 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:07:46.401 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 83944 2024-09-21T14:07:46.401 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.401 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:07:46.401 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.401 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:07:46.401 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.184327+0000", 2024-09-21T14:07:46.401 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:07:46.402 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.402 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.402 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:07:46.402 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.184327+0000", 2024-09-21T14:07:46.402 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:07:46.402 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.402 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.402 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:07:46.402 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.184329+0000", 2024-09-21T14:07:46.402 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2950000000000001e-06 2024-09-21T14:07:46.402 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.402 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.402 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:07:46.402 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.184385+0000", 2024-09-21T14:07:46.402 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6224999999999998e-05 2024-09-21T14:07:46.402 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.403 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.403 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:07:46.403 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.184396+0000", 2024-09-21T14:07:46.403 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1117999999999999e-05 2024-09-21T14:07:46.403 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.403 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.403 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:07:46.403 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.184403+0000", 2024-09-21T14:07:46.403 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2810000000000003e-06 2024-09-21T14:07:46.403 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.403 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.403 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:07:46.403 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.184450+0000", 2024-09-21T14:07:46.403 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6983000000000001e-05 2024-09-21T14:07:46.404 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.404 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.404 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:07:46.404 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.184537+0000", 2024-09-21T14:07:46.404 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7161000000000001e-05 2024-09-21T14:07:46.404 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.404 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.404 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 5,6", 2024-09-21T14:07:46.404 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.184626+0000", 2024-09-21T14:07:46.404 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8926999999999997e-05 2024-09-21T14:07:46.404 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.404 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.404 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:07:46.404 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.186629+0000", 2024-09-21T14:07:46.405 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0020021840000000002 2024-09-21T14:07:46.405 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.405 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.405 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:46.405 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.186890+0000", 2024-09-21T14:07:46.405 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00026146600000000002 2024-09-21T14:07:46.405 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:46.405 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:07:46.405 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:46.405 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.405 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.405 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26396.0:83946 11.4 11:2dcbb1c1:::benchmark_data_smithi049_206132_object83945:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e482)", 2024-09-21T14:07:46.405 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:07:46.188341+0000", 2024-09-21T14:07:46.405 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.010616755, 2024-09-21T14:07:46.405 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.010860959, 2024-09-21T14:07:46.405 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:07:46.406 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:07:46.406 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-21T14:07:46.406 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:07:46.406 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26396", 2024-09-21T14:07:46.406 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:07:46.406 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 83946 2024-09-21T14:07:46.406 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.406 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:07:46.406 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.406 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:07:46.406 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.188341+0000", 2024-09-21T14:07:46.406 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:07:46.406 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.406 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.406 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:07:46.407 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.188341+0000", 2024-09-21T14:07:46.407 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:07:46.407 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.407 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.407 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:07:46.407 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.188343+0000", 2024-09-21T14:07:46.407 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.759e-06 2024-09-21T14:07:46.407 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.407 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.407 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:07:46.407 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.188395+0000", 2024-09-21T14:07:46.407 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2292e-05 2024-09-21T14:07:46.407 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.407 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.407 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:07:46.407 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.188407+0000", 2024-09-21T14:07:46.408 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1615e-05 2024-09-21T14:07:46.408 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.408 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.408 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:07:46.408 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.188412+0000", 2024-09-21T14:07:46.408 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3789999999999997e-06 2024-09-21T14:07:46.408 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.408 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.408 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:07:46.408 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.188471+0000", 2024-09-21T14:07:46.408 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8570999999999999e-05 2024-09-21T14:07:46.408 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.408 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.408 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:07:46.408 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.188629+0000", 2024-09-21T14:07:46.408 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015849899999999999 2024-09-21T14:07:46.409 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.409 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.409 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,6", 2024-09-21T14:07:46.409 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.188790+0000", 2024-09-21T14:07:46.409 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00016022899999999999 2024-09-21T14:07:46.409 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.409 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.409 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:46.409 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.191190+0000", 2024-09-21T14:07:46.409 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0024002060000000002 2024-09-21T14:07:46.409 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.409 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.409 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:07:46.409 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.191350+0000", 2024-09-21T14:07:46.409 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00016044799999999999 2024-09-21T14:07:46.410 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:46.410 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:07:46.410 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:46.410 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.410 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.410 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26396.0:83921 11.5 11:ae0b81c4:::benchmark_data_smithi049_206132_object83920:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e482)", 2024-09-21T14:07:46.410 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:07:46.140213+0000", 2024-09-21T14:07:46.410 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.058744728000000003, 2024-09-21T14:07:46.410 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.059028659999999997, 2024-09-21T14:07:46.410 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:07:46.410 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:07:46.410 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-21T14:07:46.410 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:07:46.410 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26396", 2024-09-21T14:07:46.411 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:07:46.411 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 83921 2024-09-21T14:07:46.411 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.411 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:07:46.411 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.411 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:07:46.411 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.140213+0000", 2024-09-21T14:07:46.411 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:07:46.411 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.411 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.411 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:07:46.411 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.140213+0000", 2024-09-21T14:07:46.411 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:07:46.411 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.412 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.412 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:07:46.412 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.140214+0000", 2024-09-21T14:07:46.412 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.161e-06 2024-09-21T14:07:46.412 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.412 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.412 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:07:46.412 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.140332+0000", 2024-09-21T14:07:46.412 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000117646 2024-09-21T14:07:46.412 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.412 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.412 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:07:46.412 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.140344+0000", 2024-09-21T14:07:46.412 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1474000000000001e-05 2024-09-21T14:07:46.412 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.413 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.413 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:07:46.413 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.140349+0000", 2024-09-21T14:07:46.413 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2079999999999999e-06 2024-09-21T14:07:46.413 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.413 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.413 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:07:46.413 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.140396+0000", 2024-09-21T14:07:46.413 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7138000000000002e-05 2024-09-21T14:07:46.413 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.413 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.413 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:07:46.413 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.140627+0000", 2024-09-21T14:07:46.413 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000231239 2024-09-21T14:07:46.414 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.414 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.414 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 5,6", 2024-09-21T14:07:46.414 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.140760+0000", 2024-09-21T14:07:46.414 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013264400000000001 2024-09-21T14:07:46.414 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.414 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.414 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:07:46.414 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.143676+0000", 2024-09-21T14:07:46.414 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0029162509999999999 2024-09-21T14:07:46.414 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:46.414 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:46.414 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:46.415 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:07:46.145398+0000", 2024-09-21T14:07:46.415 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0017217090000000001 2024-09-21T14:07:46.415 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:46.415 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:07:46.415 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:46.415 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:46.415 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2024-09-21T14:07:46.415 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 7 2024-09-21T14:07:46.415 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T14:07:46.437 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:07:46.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:46.492 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:46.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:07:46.515 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 53 16 84285 84269 99.3515 63.1875 0.007415 0.0100604 2024-09-21T14:07:46.594 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:07:46.621 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:46.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:46.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:07:46.629 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:07:46.641 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:07:46.722 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:07:46.733 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:46 smithi049 ceph-mon[30080]: 6.a scrub starts 2024-09-21T14:07:46.751 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:46.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:46.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:07:46.853 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:07:46.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:46.891 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:46.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:07:46.992 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:07:47.001 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:46 smithi049 ceph-mon[30080]: 6.a scrub ok 2024-09-21T14:07:47.001 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:46 smithi049 ceph-mon[30080]: 5.5 scrub starts 2024-09-21T14:07:47.001 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:46 smithi049 ceph-mon[30080]: 5.5 scrub ok 2024-09-21T14:07:47.001 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:46 smithi049 ceph-mon[40489]: 6.a scrub starts 2024-09-21T14:07:47.001 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:46 smithi049 ceph-mon[40489]: 6.a scrub ok 2024-09-21T14:07:47.001 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:46 smithi049 ceph-mon[40489]: 5.5 scrub starts 2024-09-21T14:07:47.001 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:46 smithi049 ceph-mon[40489]: 5.5 scrub ok 2024-09-21T14:07:47.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:47.048 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:47.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:07:47.149 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:07:47.178 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:47.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:47.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:07:47.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:46 smithi187 ceph-mon[33032]: 6.a scrub starts 2024-09-21T14:07:47.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:46 smithi187 ceph-mon[33032]: 6.a scrub ok 2024-09-21T14:07:47.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:46 smithi187 ceph-mon[33032]: 5.5 scrub starts 2024-09-21T14:07:47.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:46 smithi187 ceph-mon[33032]: 5.5 scrub ok 2024-09-21T14:07:47.281 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:07:47.320 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:47.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:47.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:07:47.353 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T14:07:47.422 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:47.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:47.459 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:47.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:47.515 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 54 16 85353 85337 98.7477 66.75 0.00884096 0.0101218 2024-09-21T14:07:47.559 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:07:47.605 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:47.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:47.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:07:47.706 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:47.719 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:07:47.748 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:47.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:47.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:47.758 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T14:07:47.758 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T14:07:47.759 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 482, 2024-09-21T14:07:47.759 INFO:teuthology.orchestra.run.smithi049.stdout: "fsid": "40a3d2e8-781f-11ef-baf6-efdc52797490", 2024-09-21T14:07:47.759 INFO:teuthology.orchestra.run.smithi049.stdout: "created": "2024-09-21T13:42:57.838275+0000", 2024-09-21T14:07:47.759 INFO:teuthology.orchestra.run.smithi049.stdout: "modified": "2024-09-21T14:07:45.208083+0000", 2024-09-21T14:07:47.759 INFO:teuthology.orchestra.run.smithi049.stdout: "last_up_change": "2024-09-21T14:01:40.158454+0000", 2024-09-21T14:07:47.759 INFO:teuthology.orchestra.run.smithi049.stdout: "last_in_change": "2024-09-21T13:58:16.730861+0000", 2024-09-21T14:07:47.759 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-21T14:07:47.759 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_num": 5799936, 2024-09-21T14:07:47.759 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_set": [ 2024-09-21T14:07:47.759 INFO:teuthology.orchestra.run.smithi049.stdout: "pglog_hardlimit", 2024-09-21T14:07:47.759 INFO:teuthology.orchestra.run.smithi049.stdout: "purged_snapdirs", 2024-09-21T14:07:47.760 INFO:teuthology.orchestra.run.smithi049.stdout: "recovery_deletes", 2024-09-21T14:07:47.760 INFO:teuthology.orchestra.run.smithi049.stdout: "sortbitwise" 2024-09-21T14:07:47.760 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T14:07:47.760 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_version": 22, 2024-09-21T14:07:47.760 INFO:teuthology.orchestra.run.smithi049.stdout: "full_ratio": 0.94999998807907104, 2024-09-21T14:07:47.760 INFO:teuthology.orchestra.run.smithi049.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-21T14:07:47.760 INFO:teuthology.orchestra.run.smithi049.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-21T14:07:47.760 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_snapshot": "", 2024-09-21T14:07:47.760 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_max": 11, 2024-09-21T14:07:47.760 INFO:teuthology.orchestra.run.smithi049.stdout: "max_osd": 8, 2024-09-21T14:07:47.760 INFO:teuthology.orchestra.run.smithi049.stdout: "require_min_compat_client": "luminous", 2024-09-21T14:07:47.760 INFO:teuthology.orchestra.run.smithi049.stdout: "min_compat_client": "luminous", 2024-09-21T14:07:47.760 INFO:teuthology.orchestra.run.smithi049.stdout: "require_osd_release": "squid", 2024-09-21T14:07:47.761 INFO:teuthology.orchestra.run.smithi049.stdout: "allow_crimson": false, 2024-09-21T14:07:47.761 INFO:teuthology.orchestra.run.smithi049.stdout: "pools": [ 2024-09-21T14:07:47.761 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.761 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 1, 2024-09-21T14:07:47.761 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": ".mgr", 2024-09-21T14:07:47.761 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:46:21.096244+0000", 2024-09-21T14:07:47.761 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T14:07:47.761 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T14:07:47.761 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T14:07:47.761 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T14:07:47.761 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T14:07:47.761 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T14:07:47.761 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T14:07:47.762 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T14:07:47.762 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T14:07:47.762 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T14:07:47.762 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T14:07:47.762 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T14:07:47.762 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T14:07:47.762 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 21, 2024-09-21T14:07:47.762 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 21, 2024-09-21T14:07:47.762 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 21, 2024-09-21T14:07:47.762 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 21, 2024-09-21T14:07:47.762 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 21, 2024-09-21T14:07:47.762 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T14:07:47.762 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T14:07:47.763 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T14:07:47.763 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T14:07:47.763 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T14:07:47.763 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T14:07:47.763 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T14:07:47.763 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.763 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "438", 2024-09-21T14:07:47.763 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T14:07:47.763 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T14:07:47.763 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "436", 2024-09-21T14:07:47.763 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T14:07:47.763 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T14:07:47.764 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T14:07:47.764 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T14:07:47.764 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T14:07:47.764 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T14:07:47.764 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T14:07:47.764 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T14:07:47.764 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T14:07:47.764 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T14:07:47.764 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T14:07:47.764 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T14:07:47.764 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T14:07:47.764 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T14:07:47.764 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T14:07:47.765 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T14:07:47.765 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T14:07:47.765 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T14:07:47.765 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T14:07:47.765 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T14:07:47.765 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T14:07:47.765 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T14:07:47.765 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T14:07:47.765 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.765 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T14:07:47.765 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T14:07:47.765 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T14:07:47.765 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T14:07:47.766 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T14:07:47.766 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T14:07:47.766 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T14:07:47.766 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T14:07:47.766 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T14:07:47.766 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T14:07:47.766 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T14:07:47.766 INFO:teuthology.orchestra.run.smithi049.stdout: "options": { 2024-09-21T14:07:47.766 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_max": 32, 2024-09-21T14:07:47.766 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_min": 1 2024-09-21T14:07:47.766 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.766 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T14:07:47.766 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr": {} 2024-09-21T14:07:47.766 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.767 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T14:07:47.767 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T14:07:47.767 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2.2899999618530273, 2024-09-21T14:07:47.767 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2.2899999618530273, 2024-09-21T14:07:47.767 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1, 2024-09-21T14:07:47.767 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2.2899999618530273, 2024-09-21T14:07:47.767 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2.2899999618530273, 2024-09-21T14:07:47.767 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.97835731506347656, 2024-09-21T14:07:47.767 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.98000001907348633, 2024-09-21T14:07:47.767 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.98000001907348633 2024-09-21T14:07:47.767 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:47.767 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.767 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.767 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 2, 2024-09-21T14:07:47.767 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "datapool", 2024-09-21T14:07:47.768 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:48:55.540712+0000", 2024-09-21T14:07:47.768 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 8193, 2024-09-21T14:07:47.768 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T14:07:47.768 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T14:07:47.768 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T14:07:47.768 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T14:07:47.768 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T14:07:47.768 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T14:07:47.768 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T14:07:47.768 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T14:07:47.768 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T14:07:47.768 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T14:07:47.768 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T14:07:47.768 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T14:07:47.768 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 3, 2024-09-21T14:07:47.768 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 3, 2024-09-21T14:07:47.769 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 3, 2024-09-21T14:07:47.769 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 3, 2024-09-21T14:07:47.769 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 3, 2024-09-21T14:07:47.769 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T14:07:47.769 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T14:07:47.769 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T14:07:47.769 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T14:07:47.769 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T14:07:47.769 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T14:07:47.769 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T14:07:47.769 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.769 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "63", 2024-09-21T14:07:47.769 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T14:07:47.770 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T14:07:47.770 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T14:07:47.770 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T14:07:47.770 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T14:07:47.770 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 2, 2024-09-21T14:07:47.770 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 63, 2024-09-21T14:07:47.770 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T14:07:47.770 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T14:07:47.770 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T14:07:47.770 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T14:07:47.770 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T14:07:47.770 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T14:07:47.770 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T14:07:47.770 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T14:07:47.770 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T14:07:47.771 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T14:07:47.771 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T14:07:47.771 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T14:07:47.771 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T14:07:47.771 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T14:07:47.771 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T14:07:47.771 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T14:07:47.771 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T14:07:47.771 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T14:07:47.771 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T14:07:47.771 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.771 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T14:07:47.771 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T14:07:47.771 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T14:07:47.771 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T14:07:47.772 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T14:07:47.772 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T14:07:47.772 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T14:07:47.772 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T14:07:47.772 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T14:07:47.772 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T14:07:47.772 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T14:07:47.772 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T14:07:47.772 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T14:07:47.772 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd": {} 2024-09-21T14:07:47.772 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.772 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T14:07:47.772 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T14:07:47.772 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2.6500000953674316, 2024-09-21T14:07:47.772 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2.6500000953674316, 2024-09-21T14:07:47.772 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 0.87999999523162842, 2024-09-21T14:07:47.773 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2.3299999237060547, 2024-09-21T14:07:47.773 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2.3299999237060547, 2024-09-21T14:07:47.773 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.97835731506347656, 2024-09-21T14:07:47.773 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.98000001907348633, 2024-09-21T14:07:47.773 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.98000001907348633 2024-09-21T14:07:47.773 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:47.773 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.773 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.773 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 3, 2024-09-21T14:07:47.773 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": ".rgw.root", 2024-09-21T14:07:47.773 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:48:56.417456+0000", 2024-09-21T14:07:47.773 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T14:07:47.773 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T14:07:47.773 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T14:07:47.773 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T14:07:47.774 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T14:07:47.774 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T14:07:47.774 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T14:07:47.774 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T14:07:47.774 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T14:07:47.774 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T14:07:47.774 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T14:07:47.774 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T14:07:47.774 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T14:07:47.774 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 32, 2024-09-21T14:07:47.774 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 32, 2024-09-21T14:07:47.774 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 32, 2024-09-21T14:07:47.774 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 32, 2024-09-21T14:07:47.774 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 32, 2024-09-21T14:07:47.775 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T14:07:47.775 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T14:07:47.775 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T14:07:47.775 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T14:07:47.775 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T14:07:47.776 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T14:07:47.776 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T14:07:47.776 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.776 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "62", 2024-09-21T14:07:47.776 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T14:07:47.776 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T14:07:47.776 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T14:07:47.776 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T14:07:47.776 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T14:07:47.776 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T14:07:47.776 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T14:07:47.776 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T14:07:47.776 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T14:07:47.776 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T14:07:47.777 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T14:07:47.777 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T14:07:47.777 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T14:07:47.777 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T14:07:47.777 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T14:07:47.777 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T14:07:47.777 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T14:07:47.777 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T14:07:47.777 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T14:07:47.777 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T14:07:47.777 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T14:07:47.777 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T14:07:47.777 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T14:07:47.778 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T14:07:47.778 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T14:07:47.778 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T14:07:47.778 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.778 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T14:07:47.778 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T14:07:47.778 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T14:07:47.778 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T14:07:47.778 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T14:07:47.778 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T14:07:47.778 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T14:07:47.778 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T14:07:47.778 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T14:07:47.778 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T14:07:47.778 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T14:07:47.779 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T14:07:47.779 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T14:07:47.779 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw": {} 2024-09-21T14:07:47.779 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.779 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T14:07:47.779 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T14:07:47.779 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 1.75, 2024-09-21T14:07:47.779 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 1.75, 2024-09-21T14:07:47.779 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1, 2024-09-21T14:07:47.779 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 1.75, 2024-09-21T14:07:47.779 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 1.75, 2024-09-21T14:07:47.779 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.97835731506347656, 2024-09-21T14:07:47.779 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.98000001907348633, 2024-09-21T14:07:47.779 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.98000001907348633 2024-09-21T14:07:47.779 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:47.780 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.780 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.780 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 4, 2024-09-21T14:07:47.780 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "default.rgw.log", 2024-09-21T14:07:47.780 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:48:58.128455+0000", 2024-09-21T14:07:47.780 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T14:07:47.780 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T14:07:47.780 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T14:07:47.780 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T14:07:47.780 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T14:07:47.780 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T14:07:47.780 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T14:07:47.780 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T14:07:47.780 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T14:07:47.780 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T14:07:47.781 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T14:07:47.781 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T14:07:47.781 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T14:07:47.781 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 32, 2024-09-21T14:07:47.781 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 32, 2024-09-21T14:07:47.781 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 32, 2024-09-21T14:07:47.781 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 32, 2024-09-21T14:07:47.781 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 32, 2024-09-21T14:07:47.781 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T14:07:47.781 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T14:07:47.781 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T14:07:47.781 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T14:07:47.781 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T14:07:47.782 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T14:07:47.782 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T14:07:47.782 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.782 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "64", 2024-09-21T14:07:47.782 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T14:07:47.782 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T14:07:47.782 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T14:07:47.782 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T14:07:47.782 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T14:07:47.782 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T14:07:47.782 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T14:07:47.782 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T14:07:47.782 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T14:07:47.783 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T14:07:47.783 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T14:07:47.783 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T14:07:47.783 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T14:07:47.783 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T14:07:47.783 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T14:07:47.783 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T14:07:47.783 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T14:07:47.783 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T14:07:47.783 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T14:07:47.783 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T14:07:47.783 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T14:07:47.784 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T14:07:47.784 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T14:07:47.784 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T14:07:47.784 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T14:07:47.784 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T14:07:47.784 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.784 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T14:07:47.784 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T14:07:47.784 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T14:07:47.784 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T14:07:47.784 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T14:07:47.784 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T14:07:47.784 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T14:07:47.784 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T14:07:47.785 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T14:07:47.785 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T14:07:47.785 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T14:07:47.785 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T14:07:47.785 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T14:07:47.785 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw": {} 2024-09-21T14:07:47.785 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.785 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T14:07:47.785 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T14:07:47.785 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2.5, 2024-09-21T14:07:47.785 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2.5, 2024-09-21T14:07:47.785 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1, 2024-09-21T14:07:47.785 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2.5, 2024-09-21T14:07:47.785 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2.5, 2024-09-21T14:07:47.785 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.97835731506347656, 2024-09-21T14:07:47.786 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.98000001907348633, 2024-09-21T14:07:47.786 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.98000001907348633 2024-09-21T14:07:47.786 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:47.786 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.786 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.786 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 5, 2024-09-21T14:07:47.786 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "default.rgw.control", 2024-09-21T14:07:47.786 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:49:00.069361+0000", 2024-09-21T14:07:47.786 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T14:07:47.786 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T14:07:47.786 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T14:07:47.786 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T14:07:47.786 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T14:07:47.787 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T14:07:47.787 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T14:07:47.787 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T14:07:47.787 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T14:07:47.787 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T14:07:47.787 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T14:07:47.787 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T14:07:47.787 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T14:07:47.787 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 32, 2024-09-21T14:07:47.787 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 32, 2024-09-21T14:07:47.787 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 32, 2024-09-21T14:07:47.787 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 32, 2024-09-21T14:07:47.787 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 32, 2024-09-21T14:07:47.788 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T14:07:47.788 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T14:07:47.788 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T14:07:47.788 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T14:07:47.788 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T14:07:47.788 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T14:07:47.788 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T14:07:47.788 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.788 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "66", 2024-09-21T14:07:47.788 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T14:07:47.788 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T14:07:47.788 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T14:07:47.788 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T14:07:47.788 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T14:07:47.788 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T14:07:47.789 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T14:07:47.789 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T14:07:47.789 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T14:07:47.789 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T14:07:47.789 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T14:07:47.789 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T14:07:47.789 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T14:07:47.789 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T14:07:47.789 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T14:07:47.789 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T14:07:47.789 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T14:07:47.789 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T14:07:47.789 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T14:07:47.789 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T14:07:47.790 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T14:07:47.790 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T14:07:47.790 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T14:07:47.790 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T14:07:47.790 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T14:07:47.790 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T14:07:47.790 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.790 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T14:07:47.790 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T14:07:47.790 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T14:07:47.790 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T14:07:47.790 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T14:07:47.790 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T14:07:47.790 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T14:07:47.791 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T14:07:47.791 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T14:07:47.791 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T14:07:47.791 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T14:07:47.791 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T14:07:47.791 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T14:07:47.791 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw": {} 2024-09-21T14:07:47.791 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.791 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T14:07:47.791 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T14:07:47.791 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 1.5, 2024-09-21T14:07:47.791 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 1.5, 2024-09-21T14:07:47.791 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1, 2024-09-21T14:07:47.791 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 1.5, 2024-09-21T14:07:47.791 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 1.5, 2024-09-21T14:07:47.792 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.97835731506347656, 2024-09-21T14:07:47.792 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.98000001907348633, 2024-09-21T14:07:47.792 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.98000001907348633 2024-09-21T14:07:47.792 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:47.792 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.792 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.792 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 6, 2024-09-21T14:07:47.792 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "default.rgw.meta", 2024-09-21T14:07:47.792 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:49:02.127399+0000", 2024-09-21T14:07:47.792 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T14:07:47.792 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T14:07:47.792 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T14:07:47.792 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T14:07:47.792 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T14:07:47.792 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T14:07:47.793 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T14:07:47.793 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T14:07:47.793 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T14:07:47.793 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T14:07:47.793 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T14:07:47.793 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T14:07:47.793 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T14:07:47.793 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 32, 2024-09-21T14:07:47.793 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 32, 2024-09-21T14:07:47.793 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 32, 2024-09-21T14:07:47.793 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 32, 2024-09-21T14:07:47.793 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 32, 2024-09-21T14:07:47.793 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T14:07:47.794 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T14:07:47.794 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T14:07:47.794 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T14:07:47.794 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T14:07:47.794 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T14:07:47.794 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T14:07:47.794 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.794 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "68", 2024-09-21T14:07:47.794 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T14:07:47.794 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T14:07:47.794 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T14:07:47.794 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T14:07:47.794 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T14:07:47.795 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T14:07:47.795 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T14:07:47.795 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T14:07:47.795 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T14:07:47.795 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T14:07:47.795 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T14:07:47.795 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T14:07:47.795 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T14:07:47.795 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T14:07:47.795 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T14:07:47.795 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T14:07:47.795 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T14:07:47.795 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T14:07:47.795 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T14:07:47.796 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T14:07:47.796 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T14:07:47.796 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T14:07:47.796 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T14:07:47.796 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T14:07:47.796 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T14:07:47.796 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T14:07:47.796 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.796 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T14:07:47.796 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T14:07:47.796 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T14:07:47.796 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T14:07:47.796 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T14:07:47.797 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T14:07:47.797 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T14:07:47.797 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T14:07:47.797 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T14:07:47.797 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T14:07:47.797 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T14:07:47.797 INFO:teuthology.orchestra.run.smithi049.stdout: "options": { 2024-09-21T14:07:47.797 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_bias": 4 2024-09-21T14:07:47.797 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.797 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T14:07:47.797 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw": {} 2024-09-21T14:07:47.797 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.797 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T14:07:47.797 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T14:07:47.798 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2, 2024-09-21T14:07:47.798 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2, 2024-09-21T14:07:47.798 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1, 2024-09-21T14:07:47.798 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2, 2024-09-21T14:07:47.798 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2, 2024-09-21T14:07:47.798 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.97835731506347656, 2024-09-21T14:07:47.798 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.98000001907348633, 2024-09-21T14:07:47.798 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.98000001907348633 2024-09-21T14:07:47.798 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:47.798 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.798 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.798 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 11, 2024-09-21T14:07:47.798 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "unique_pool_4", 2024-09-21T14:07:47.799 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T14:06:43.392407+0000", 2024-09-21T14:07:47.799 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T14:07:47.799 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T14:07:47.799 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T14:07:47.799 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T14:07:47.799 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T14:07:47.799 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T14:07:47.799 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T14:07:47.799 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T14:07:47.799 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T14:07:47.799 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T14:07:47.799 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T14:07:47.799 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T14:07:47.799 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T14:07:47.800 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 16, 2024-09-21T14:07:47.800 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 16, 2024-09-21T14:07:47.800 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 16, 2024-09-21T14:07:47.800 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 16, 2024-09-21T14:07:47.800 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 16, 2024-09-21T14:07:47.800 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T14:07:47.800 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T14:07:47.800 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T14:07:47.800 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T14:07:47.800 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T14:07:47.800 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T14:07:47.800 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T14:07:47.801 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.801 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "465", 2024-09-21T14:07:47.801 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T14:07:47.801 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T14:07:47.801 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T14:07:47.801 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T14:07:47.801 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T14:07:47.801 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T14:07:47.801 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T14:07:47.801 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T14:07:47.801 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T14:07:47.801 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T14:07:47.801 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T14:07:47.801 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T14:07:47.801 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T14:07:47.802 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T14:07:47.802 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T14:07:47.802 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T14:07:47.802 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T14:07:47.802 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T14:07:47.802 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T14:07:47.802 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T14:07:47.802 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T14:07:47.802 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T14:07:47.802 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T14:07:47.802 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T14:07:47.802 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T14:07:47.802 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T14:07:47.803 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.803 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T14:07:47.803 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T14:07:47.803 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T14:07:47.803 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T14:07:47.803 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T14:07:47.803 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T14:07:47.803 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T14:07:47.803 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T14:07:47.803 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T14:07:47.803 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T14:07:47.803 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T14:07:47.803 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T14:07:47.803 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T14:07:47.804 INFO:teuthology.orchestra.run.smithi049.stdout: "rados": {} 2024-09-21T14:07:47.804 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.804 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T14:07:47.804 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T14:07:47.804 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2.5, 2024-09-21T14:07:47.804 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2.5, 2024-09-21T14:07:47.804 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1, 2024-09-21T14:07:47.804 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2.5, 2024-09-21T14:07:47.804 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2.5, 2024-09-21T14:07:47.804 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.97835731506347656, 2024-09-21T14:07:47.804 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.98000001907348633, 2024-09-21T14:07:47.804 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.98000001907348633 2024-09-21T14:07:47.804 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:47.805 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:47.805 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T14:07:47.805 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-09-21T14:07:47.805 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.805 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 0, 2024-09-21T14:07:47.805 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "2595dca9-6219-4e0c-8728-751eb102e73a", 2024-09-21T14:07:47.805 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T14:07:47.805 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T14:07:47.805 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 0.6176910400390625, 2024-09-21T14:07:47.805 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T14:07:47.805 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T14:07:47.805 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T14:07:47.805 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 8, 2024-09-21T14:07:47.805 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 429, 2024-09-21T14:07:47.805 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T14:07:47.806 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T14:07:47.806 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T14:07:47.806 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:07:47.806 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.806 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:07:47.806 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6802", 2024-09-21T14:07:47.806 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T14:07:47.806 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.806 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.806 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:07:47.806 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6803", 2024-09-21T14:07:47.806 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T14:07:47.806 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:47.807 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:47.807 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.807 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T14:07:47.807 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:07:47.807 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.807 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:07:47.807 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6804", 2024-09-21T14:07:47.807 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T14:07:47.807 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.807 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.807 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:07:47.807 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6805", 2024-09-21T14:07:47.807 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T14:07:47.807 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:47.808 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:47.808 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.808 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T14:07:47.808 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:07:47.808 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.808 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:07:47.808 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6808", 2024-09-21T14:07:47.808 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T14:07:47.808 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.808 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.808 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:07:47.808 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6809", 2024-09-21T14:07:47.809 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:47 smithi049 ceph-mon[30080]: pgmap v960: 168 pgs: 168 active+clean; 4.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.03k op/s 2024-09-21T14:07:47.809 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:47 smithi049 ceph-mon[30080]: 2.2 scrub starts 2024-09-21T14:07:47.809 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:47 smithi049 ceph-mon[30080]: 2.2 scrub ok 2024-09-21T14:07:47.809 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:47 smithi049 ceph-mon[30080]: 5.1f scrub starts 2024-09-21T14:07:47.809 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:47 smithi049 ceph-mon[30080]: 5.1f scrub ok 2024-09-21T14:07:47.809 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:47 smithi049 ceph-mon[30080]: 4.12 scrub starts 2024-09-21T14:07:47.809 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:47 smithi049 ceph-mon[30080]: 4.12 scrub ok 2024-09-21T14:07:47.809 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:47 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T14:07:47.809 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:47 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T14:07:47.809 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:47 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:07:47.809 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:47 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3337271436' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:07:47.810 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:47 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:07:47.810 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:47 smithi049 ceph-mon[40489]: pgmap v960: 168 pgs: 168 active+clean; 4.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.03k op/s 2024-09-21T14:07:47.810 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:47 smithi049 ceph-mon[40489]: 2.2 scrub starts 2024-09-21T14:07:47.810 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:47 smithi049 ceph-mon[40489]: 2.2 scrub ok 2024-09-21T14:07:47.810 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:47 smithi049 ceph-mon[40489]: 5.1f scrub starts 2024-09-21T14:07:47.810 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:47 smithi049 ceph-mon[40489]: 5.1f scrub ok 2024-09-21T14:07:47.810 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:47 smithi049 ceph-mon[40489]: 4.12 scrub starts 2024-09-21T14:07:47.810 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:47 smithi049 ceph-mon[40489]: 4.12 scrub ok 2024-09-21T14:07:47.810 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:47 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T14:07:47.810 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:47 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T14:07:47.811 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:47 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:07:47.811 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:47 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3337271436' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:07:47.811 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:47 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:07:47.811 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T14:07:47.811 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:47.811 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:47.811 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.811 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T14:07:47.812 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:07:47.812 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.812 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:07:47.812 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6806", 2024-09-21T14:07:47.812 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T14:07:47.812 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.812 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.812 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:07:47.812 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6807", 2024-09-21T14:07:47.812 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T14:07:47.812 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:47.812 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:47.812 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.812 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6803/2095561412", 2024-09-21T14:07:47.812 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6805/2095561412", 2024-09-21T14:07:47.813 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6809/2095561412", 2024-09-21T14:07:47.813 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6807/2095561412", 2024-09-21T14:07:47.813 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T14:07:47.813 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T14:07:47.813 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T14:07:47.813 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:47.813 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.813 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.813 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 1, 2024-09-21T14:07:47.813 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "0ba327f9-5524-4a40-abcc-729ded0b1917", 2024-09-21T14:07:47.813 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T14:07:47.813 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T14:07:47.813 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T14:07:47.814 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T14:07:47.814 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T14:07:47.814 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T14:07:47.814 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 13, 2024-09-21T14:07:47.814 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 461, 2024-09-21T14:07:47.814 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T14:07:47.814 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T14:07:47.814 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T14:07:47.814 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:07:47.814 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.814 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:07:47.814 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6810", 2024-09-21T14:07:47.814 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T14:07:47.814 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.815 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.815 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:07:47.815 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6811", 2024-09-21T14:07:47.815 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T14:07:47.815 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:47.815 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:47.815 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.815 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T14:07:47.815 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:07:47.815 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.815 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:07:47.815 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6812", 2024-09-21T14:07:47.815 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T14:07:47.815 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.816 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.816 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:07:47.816 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6813", 2024-09-21T14:07:47.816 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T14:07:47.816 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:47.816 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:47.816 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.816 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T14:07:47.816 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:07:47.816 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.816 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:07:47.816 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6816", 2024-09-21T14:07:47.816 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T14:07:47.816 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.817 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.817 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:07:47.817 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6817", 2024-09-21T14:07:47.817 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T14:07:47.817 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:47.817 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:47.817 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.817 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T14:07:47.817 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:07:47.817 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.817 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:07:47.817 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6814", 2024-09-21T14:07:47.817 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T14:07:47.818 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.818 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.818 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:07:47.818 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6815", 2024-09-21T14:07:47.818 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T14:07:47.818 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:47.818 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:47.818 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.818 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6811/1382840527", 2024-09-21T14:07:47.818 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6813/1382840527", 2024-09-21T14:07:47.818 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6817/1382840527", 2024-09-21T14:07:47.818 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6815/1382840527", 2024-09-21T14:07:47.818 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T14:07:47.818 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T14:07:47.819 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T14:07:47.819 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:47.819 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.819 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.819 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 2, 2024-09-21T14:07:47.819 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "27a45750-204d-48fd-bb8f-ebb01273343d", 2024-09-21T14:07:47.819 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T14:07:47.819 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T14:07:47.819 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T14:07:47.819 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 0.8268585205078125, 2024-09-21T14:07:47.819 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 18, 2024-09-21T14:07:47.819 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 100, 2024-09-21T14:07:47.819 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 130, 2024-09-21T14:07:47.819 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 461, 2024-09-21T14:07:47.819 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 101, 2024-09-21T14:07:47.819 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T14:07:47.820 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T14:07:47.820 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:07:47.820 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.820 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:07:47.820 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6818", 2024-09-21T14:07:47.820 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T14:07:47.820 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.820 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.820 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:07:47.820 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6819", 2024-09-21T14:07:47.820 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T14:07:47.820 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:47.820 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:47.820 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.820 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T14:07:47.820 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:07:47.821 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.821 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:07:47.821 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6820", 2024-09-21T14:07:47.821 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T14:07:47.821 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.821 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.821 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:07:47.821 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6821", 2024-09-21T14:07:47.821 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T14:07:47.821 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:47.821 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:47.821 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.821 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T14:07:47.821 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:07:47.821 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.822 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:07:47.822 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6824", 2024-09-21T14:07:47.822 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T14:07:47.822 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.822 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.822 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:07:47.822 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6825", 2024-09-21T14:07:47.822 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T14:07:47.822 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:47.822 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:47.822 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.822 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T14:07:47.822 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:07:47.822 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.823 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:07:47.823 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6822", 2024-09-21T14:07:47.823 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T14:07:47.823 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.823 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.823 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:07:47.823 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6823", 2024-09-21T14:07:47.823 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T14:07:47.823 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:47.823 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:47.823 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.823 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6819/1945314741", 2024-09-21T14:07:47.824 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6821/1945314741", 2024-09-21T14:07:47.824 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6825/1945314741", 2024-09-21T14:07:47.824 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6823/1945314741", 2024-09-21T14:07:47.824 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T14:07:47.824 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T14:07:47.824 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T14:07:47.824 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:47.824 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.824 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.824 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 3, 2024-09-21T14:07:47.824 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "528bc390-3dc9-4f3b-ab75-d7412c9625db", 2024-09-21T14:07:47.824 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T14:07:47.824 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T14:07:47.824 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T14:07:47.825 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T14:07:47.825 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 27, 2024-09-21T14:07:47.825 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 214, 2024-09-21T14:07:47.825 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 240, 2024-09-21T14:07:47.825 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 461, 2024-09-21T14:07:47.825 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 215, 2024-09-21T14:07:47.825 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T14:07:47.825 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T14:07:47.825 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:07:47.825 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.825 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:07:47.825 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6826", 2024-09-21T14:07:47.825 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T14:07:47.825 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.826 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.826 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:07:47.826 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6827", 2024-09-21T14:07:47.826 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T14:07:47.826 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:47.826 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:47.826 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.826 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T14:07:47.826 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:07:47.826 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.826 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:07:47.826 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6828", 2024-09-21T14:07:47.826 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T14:07:47.827 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.827 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.827 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:07:47.827 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6829", 2024-09-21T14:07:47.827 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T14:07:47.827 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:47.827 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:47.827 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.827 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T14:07:47.827 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:07:47.827 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.827 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:07:47.828 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6832", 2024-09-21T14:07:47.828 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T14:07:47.828 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.828 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.828 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:07:47.828 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6833", 2024-09-21T14:07:47.828 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T14:07:47.828 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:47.828 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:47.828 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.828 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T14:07:47.828 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:07:47.828 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.828 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:07:47.828 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6830", 2024-09-21T14:07:47.829 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T14:07:47.829 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.829 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.829 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:07:47.829 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6831", 2024-09-21T14:07:47.829 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T14:07:47.829 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:47.829 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:47.829 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.829 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6827/886735721", 2024-09-21T14:07:47.829 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6829/886735721", 2024-09-21T14:07:47.829 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6833/886735721", 2024-09-21T14:07:47.830 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6831/886735721", 2024-09-21T14:07:47.830 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T14:07:47.830 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T14:07:47.830 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T14:07:47.830 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:47.830 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.830 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.830 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 4, 2024-09-21T14:07:47.830 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "6fc64698-2d37-4db0-bf3d-c7e612c45163", 2024-09-21T14:07:47.830 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T14:07:47.830 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T14:07:47.830 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 0.7888946533203125, 2024-09-21T14:07:47.830 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T14:07:47.830 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T14:07:47.831 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T14:07:47.831 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 33, 2024-09-21T14:07:47.831 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 436, 2024-09-21T14:07:47.831 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T14:07:47.831 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T14:07:47.831 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T14:07:47.831 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:07:47.831 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.831 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:07:47.831 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6800", 2024-09-21T14:07:47.831 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T14:07:47.831 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.831 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.831 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:07:47.832 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6801", 2024-09-21T14:07:47.832 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T14:07:47.832 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:47.832 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:47.832 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.832 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T14:07:47.832 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:07:47.832 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.832 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:07:47.832 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6802", 2024-09-21T14:07:47.832 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T14:07:47.832 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.832 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.832 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:07:47.833 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6803", 2024-09-21T14:07:47.833 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T14:07:47.833 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:47.833 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:47.833 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.833 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T14:07:47.833 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:07:47.833 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.833 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:07:47.833 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6806", 2024-09-21T14:07:47.833 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T14:07:47.833 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.833 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.833 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:07:47.833 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6807", 2024-09-21T14:07:47.833 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T14:07:47.833 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:47.834 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:47.834 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.834 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T14:07:47.834 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:07:47.834 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.834 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:07:47.834 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6804", 2024-09-21T14:07:47.834 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T14:07:47.834 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.834 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.834 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:07:47.834 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6805", 2024-09-21T14:07:47.834 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T14:07:47.834 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:47.835 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:47.835 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.835 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.187:6801/1702447776", 2024-09-21T14:07:47.835 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.187:6803/1702447776", 2024-09-21T14:07:47.835 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.187:6807/1702447776", 2024-09-21T14:07:47.835 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.187:6805/1702447776", 2024-09-21T14:07:47.835 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T14:07:47.835 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T14:07:47.835 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T14:07:47.835 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:47.835 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.835 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.835 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 5, 2024-09-21T14:07:47.835 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "8d7f62a1-7e92-4e06-9cad-86b96be514d8", 2024-09-21T14:07:47.835 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T14:07:47.836 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T14:07:47.836 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T14:07:47.836 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T14:07:47.836 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 39, 2024-09-21T14:07:47.836 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 324, 2024-09-21T14:07:47.836 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 345, 2024-09-21T14:07:47.836 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 471, 2024-09-21T14:07:47.836 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 325, 2024-09-21T14:07:47.836 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T14:07:47.836 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T14:07:47.836 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:07:47.836 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.837 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:07:47.837 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6808", 2024-09-21T14:07:47.837 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1595353965 2024-09-21T14:07:47.837 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.837 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.837 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:07:47.837 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6809", 2024-09-21T14:07:47.837 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1595353965 2024-09-21T14:07:47.837 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:47.837 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:47.837 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.837 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T14:07:47.837 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:07:47.838 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.838 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:07:47.838 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6810", 2024-09-21T14:07:47.838 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1595353965 2024-09-21T14:07:47.838 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.838 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.838 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:07:47.838 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6811", 2024-09-21T14:07:47.838 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1595353965 2024-09-21T14:07:47.838 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:47.838 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:47.838 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.838 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T14:07:47.839 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:07:47.839 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.839 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:07:47.839 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6814", 2024-09-21T14:07:47.839 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1595353965 2024-09-21T14:07:47.839 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.839 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.839 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:07:47.839 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6815", 2024-09-21T14:07:47.839 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1595353965 2024-09-21T14:07:47.839 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:47.839 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:47.839 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.839 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T14:07:47.840 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:07:47.840 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.840 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:07:47.840 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6812", 2024-09-21T14:07:47.840 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1595353965 2024-09-21T14:07:47.840 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.840 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.840 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:07:47.840 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6813", 2024-09-21T14:07:47.840 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1595353965 2024-09-21T14:07:47.840 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:47.840 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:47.840 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.840 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.187:6809/1595353965", 2024-09-21T14:07:47.841 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.187:6811/1595353965", 2024-09-21T14:07:47.841 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.187:6815/1595353965", 2024-09-21T14:07:47.841 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.187:6813/1595353965", 2024-09-21T14:07:47.841 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T14:07:47.841 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T14:07:47.841 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T14:07:47.841 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:47.841 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.841 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.841 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 6, 2024-09-21T14:07:47.841 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "3a7afd13-26ce-444c-9cc0-81ce2eab6ce5", 2024-09-21T14:07:47.841 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T14:07:47.841 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T14:07:47.841 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 0.9076080322265625, 2024-09-21T14:07:47.842 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T14:07:47.842 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T14:07:47.842 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T14:07:47.842 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 47, 2024-09-21T14:07:47.842 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 461, 2024-09-21T14:07:47.842 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T14:07:47.842 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T14:07:47.842 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T14:07:47.842 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:07:47.842 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.842 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:07:47.842 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6816", 2024-09-21T14:07:47.842 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T14:07:47.843 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.843 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.843 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:07:47.843 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6817", 2024-09-21T14:07:47.843 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T14:07:47.843 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:47.843 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:47.843 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.843 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T14:07:47.843 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:07:47.843 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.843 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:07:47.843 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6818", 2024-09-21T14:07:47.843 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T14:07:47.843 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.844 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.844 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:07:47.844 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6819", 2024-09-21T14:07:47.844 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T14:07:47.844 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:47.844 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:47.844 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.844 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T14:07:47.844 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:07:47.844 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.844 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:07:47.844 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6822", 2024-09-21T14:07:47.845 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T14:07:47.845 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.845 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.845 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:07:47.845 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6823", 2024-09-21T14:07:47.845 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T14:07:47.845 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:47.845 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:47.845 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.845 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T14:07:47.845 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:07:47.845 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.845 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:07:47.845 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6820", 2024-09-21T14:07:47.846 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T14:07:47.846 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.846 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.846 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:07:47.846 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6821", 2024-09-21T14:07:47.846 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T14:07:47.846 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:47.846 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:47.846 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.846 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.187:6817/2728426525", 2024-09-21T14:07:47.846 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.187:6819/2728426525", 2024-09-21T14:07:47.846 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.187:6823/2728426525", 2024-09-21T14:07:47.846 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.187:6821/2728426525", 2024-09-21T14:07:47.846 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T14:07:47.846 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T14:07:47.847 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T14:07:47.847 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:47.847 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.847 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.847 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 7, 2024-09-21T14:07:47.847 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "1e08d891-8747-402e-b4c2-aa3802ce8537", 2024-09-21T14:07:47.847 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T14:07:47.847 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T14:07:47.847 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T14:07:47.848 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T14:07:47.848 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 54, 2024-09-21T14:07:47.848 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 272, 2024-09-21T14:07:47.848 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 298, 2024-09-21T14:07:47.848 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 461, 2024-09-21T14:07:47.848 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 273, 2024-09-21T14:07:47.848 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T14:07:47.848 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T14:07:47.848 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:07:47.848 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.848 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:07:47.848 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6824", 2024-09-21T14:07:47.848 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 13894964 2024-09-21T14:07:47.848 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.849 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.849 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:07:47.849 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6825", 2024-09-21T14:07:47.849 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 13894964 2024-09-21T14:07:47.849 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:47.849 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:47.849 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.849 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T14:07:47.849 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:07:47.849 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.849 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:07:47.849 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6826", 2024-09-21T14:07:47.849 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 13894964 2024-09-21T14:07:47.849 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.849 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.849 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:07:47.850 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6827", 2024-09-21T14:07:47.850 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 13894964 2024-09-21T14:07:47.850 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:47.850 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:47.850 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.850 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T14:07:47.850 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:07:47.850 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.850 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:07:47.850 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6830", 2024-09-21T14:07:47.850 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 13894964 2024-09-21T14:07:47.850 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.850 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.850 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:07:47.850 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6831", 2024-09-21T14:07:47.851 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 13894964 2024-09-21T14:07:47.851 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:47.851 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:47.851 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.851 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T14:07:47.851 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:07:47.851 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.851 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:07:47.851 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6828", 2024-09-21T14:07:47.851 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 13894964 2024-09-21T14:07:47.851 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.851 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.851 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:07:47.851 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6829", 2024-09-21T14:07:47.851 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 13894964 2024-09-21T14:07:47.851 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:47.852 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:47.852 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.852 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.187:6825/13894964", 2024-09-21T14:07:47.852 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.187:6827/13894964", 2024-09-21T14:07:47.852 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.187:6831/13894964", 2024-09-21T14:07:47.852 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.187:6829/13894964", 2024-09-21T14:07:47.852 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T14:07:47.852 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T14:07:47.852 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T14:07:47.852 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:47.852 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:47.852 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T14:07:47.852 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_xinfo": [ 2024-09-21T14:07:47.852 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.853 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 0, 2024-09-21T14:07:47.853 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T14:07:47.853 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T14:07:47.853 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T14:07:47.853 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T14:07:47.853 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T14:07:47.853 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T14:06:38.896732+0000", 2024-09-21T14:07:47.853 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T14:07:47.853 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.853 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.853 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 1, 2024-09-21T14:07:47.853 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T14:07:47.853 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T14:07:47.853 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T14:07:47.853 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T14:07:47.853 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T14:07:47.854 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T14:07:20.759253+0000", 2024-09-21T14:07:47.854 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T14:07:47.854 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.854 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.854 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 2, 2024-09-21T14:07:47.854 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T14:07:47.854 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T14:07:47.854 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T14:07:47.854 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T14:07:47.854 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T14:07:47.854 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T14:06:27.643288+0000", 2024-09-21T14:07:47.854 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 100 2024-09-21T14:07:47.854 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.854 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.854 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 3, 2024-09-21T14:07:47.855 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T14:07:47.855 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T14:07:47.855 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T14:07:47.855 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T14:07:47.855 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T14:07:47.855 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T14:06:58.391048+0000", 2024-09-21T14:07:47.855 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 214 2024-09-21T14:07:47.855 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.855 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.855 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 4, 2024-09-21T14:07:47.855 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T14:07:47.855 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T14:07:47.855 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T14:07:47.855 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T14:07:47.855 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T14:07:47.856 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T14:06:40.662638+0000", 2024-09-21T14:07:47.856 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T14:07:47.856 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.856 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.856 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 5, 2024-09-21T14:07:47.856 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T14:07:47.856 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T14:07:47.856 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T14:07:47.856 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T14:07:47.856 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T14:07:47.856 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T14:06:56.536868+0000", 2024-09-21T14:07:47.856 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 324 2024-09-21T14:07:47.856 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.856 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.857 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 6, 2024-09-21T14:07:47.857 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T14:07:47.857 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T14:07:47.857 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T14:07:47.857 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T14:07:47.857 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T14:07:47.857 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T14:07:16.644516+0000", 2024-09-21T14:07:47.857 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T14:07:47.857 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.857 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.857 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 7, 2024-09-21T14:07:47.857 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T14:07:47.857 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T14:07:47.857 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T14:07:47.857 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T14:07:47.858 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T14:07:47.858 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T14:07:20.884792+0000", 2024-09-21T14:07:47.858 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 272 2024-09-21T14:07:47.858 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:47.858 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T14:07:47.858 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap": [ 2024-09-21T14:07:47.858 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.858 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "3.14", 2024-09-21T14:07:47.858 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-09-21T14:07:47.858 INFO:teuthology.orchestra.run.smithi049.stdout: 7, 2024-09-21T14:07:47.858 INFO:teuthology.orchestra.run.smithi049.stdout: 6, 2024-09-21T14:07:47.858 INFO:teuthology.orchestra.run.smithi049.stdout: 0 2024-09-21T14:07:47.858 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:47.858 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:47.858 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T14:07:47.859 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap_items": [ 2024-09-21T14:07:47.859 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.859 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "4.2", 2024-09-21T14:07:47.859 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-09-21T14:07:47.859 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.859 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 1, 2024-09-21T14:07:47.859 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 2 2024-09-21T14:07:47.859 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:47.859 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:47.859 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.859 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.859 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "4.6", 2024-09-21T14:07:47.859 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-09-21T14:07:47.859 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.860 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 1, 2024-09-21T14:07:47.860 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 5 2024-09-21T14:07:47.860 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:47.860 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:47.860 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.860 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.860 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "4.d", 2024-09-21T14:07:47.860 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-09-21T14:07:47.860 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.860 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 1, 2024-09-21T14:07:47.860 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 5 2024-09-21T14:07:47.860 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:47.860 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:47.860 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.860 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.860 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "5.16", 2024-09-21T14:07:47.861 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-09-21T14:07:47.861 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.861 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 1, 2024-09-21T14:07:47.861 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 6 2024-09-21T14:07:47.861 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.861 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.861 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 3, 2024-09-21T14:07:47.861 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 5 2024-09-21T14:07:47.861 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:47.861 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:47.861 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.861 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.861 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "6.7", 2024-09-21T14:07:47.861 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-09-21T14:07:47.862 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.862 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 3, 2024-09-21T14:07:47.862 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 7 2024-09-21T14:07:47.862 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:47.862 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:47.862 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.862 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.862 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "6.b", 2024-09-21T14:07:47.862 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-09-21T14:07:47.862 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.862 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 3, 2024-09-21T14:07:47.862 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 1 2024-09-21T14:07:47.862 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.862 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.862 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 7, 2024-09-21T14:07:47.862 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 6 2024-09-21T14:07:47.863 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:47.863 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:47.863 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.863 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.863 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "6.1f", 2024-09-21T14:07:47.863 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-09-21T14:07:47.863 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:07:47.863 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 3, 2024-09-21T14:07:47.863 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 0 2024-09-21T14:07:47.863 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:47.863 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:07:47.863 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:47.863 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T14:07:47.863 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap_primaries": [], 2024-09-21T14:07:47.863 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_temp": [], 2024-09-21T14:07:47.864 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_temp": [], 2024-09-21T14:07:47.864 INFO:teuthology.orchestra.run.smithi049.stdout: "blocklist": { 2024-09-21T14:07:47.864 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/2731793917": "2024-09-22T13:49:27.311313+0000", 2024-09-21T14:07:47.864 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/799831425": "2024-09-22T13:49:27.311313+0000", 2024-09-21T14:07:47.864 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/2518197280": "2024-09-22T13:43:28.110499+0000", 2024-09-21T14:07:47.864 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3556829486": "2024-09-22T13:43:28.110499+0000", 2024-09-21T14:07:47.864 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/1249926115": "2024-09-22T13:49:27.311313+0000", 2024-09-21T14:07:47.864 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/231530138": "2024-09-22T13:43:28.110499+0000", 2024-09-21T14:07:47.864 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/4220781134": "2024-09-22T13:49:27.311313+0000", 2024-09-21T14:07:47.864 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/202325455": "2024-09-22T13:43:28.110499+0000", 2024-09-21T14:07:47.864 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/4129636473": "2024-09-22T13:43:56.633723+0000", 2024-09-21T14:07:47.864 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/2518197280": "2024-09-22T13:43:28.110499+0000", 2024-09-21T14:07:47.864 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3514741273": "2024-09-22T13:43:56.633723+0000", 2024-09-21T14:07:47.864 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/4129636473": "2024-09-22T13:43:56.633723+0000", 2024-09-21T14:07:47.864 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3125743776": "2024-09-22T13:43:56.633723+0000", 2024-09-21T14:07:47.865 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3600977363": "2024-09-22T13:43:56.633723+0000", 2024-09-21T14:07:47.865 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/3453571598": "2024-09-22T13:49:27.311313+0000", 2024-09-21T14:07:47.865 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/3453571598": "2024-09-22T13:49:27.311313+0000", 2024-09-21T14:07:47.865 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3553734432": "2024-09-22T13:49:27.311313+0000" 2024-09-21T14:07:47.865 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.865 INFO:teuthology.orchestra.run.smithi049.stdout: "range_blocklist": {}, 2024-09-21T14:07:47.865 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profiles": { 2024-09-21T14:07:47.865 INFO:teuthology.orchestra.run.smithi049.stdout: "default": { 2024-09-21T14:07:47.865 INFO:teuthology.orchestra.run.smithi049.stdout: "crush-failure-domain": "osd", 2024-09-21T14:07:47.865 INFO:teuthology.orchestra.run.smithi049.stdout: "k": "2", 2024-09-21T14:07:47.865 INFO:teuthology.orchestra.run.smithi049.stdout: "m": "1", 2024-09-21T14:07:47.865 INFO:teuthology.orchestra.run.smithi049.stdout: "plugin": "jerasure", 2024-09-21T14:07:47.865 INFO:teuthology.orchestra.run.smithi049.stdout: "technique": "reed_sol_van" 2024-09-21T14:07:47.865 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:47.865 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:07:47.865 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps_queue": [], 2024-09-21T14:07:47.866 INFO:teuthology.orchestra.run.smithi049.stdout: "new_removed_snaps": [], 2024-09-21T14:07:47.866 INFO:teuthology.orchestra.run.smithi049.stdout: "new_purged_snaps": [], 2024-09-21T14:07:47.866 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_node_flags": {}, 2024-09-21T14:07:47.866 INFO:teuthology.orchestra.run.smithi049.stdout: "device_class_flags": {}, 2024-09-21T14:07:47.866 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode": { 2024-09-21T14:07:47.866 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode_enabled": false, 2024-09-21T14:07:47.866 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_bucket_count": 0, 2024-09-21T14:07:47.866 INFO:teuthology.orchestra.run.smithi049.stdout: "degraded_stretch_mode": 0, 2024-09-21T14:07:47.866 INFO:teuthology.orchestra.run.smithi049.stdout: "recovering_stretch_mode": 0, 2024-09-21T14:07:47.866 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode_bucket": 0 2024-09-21T14:07:47.866 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:07:47.866 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T14:07:47.866 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:07:47.906 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:47.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:47.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:07:47.938 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:07:47.938 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:07:47.975 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:07:47.975 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:07:48.007 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-09-21T14:07:48.008 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:07:48.009 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:07:48.009 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:07:48.010 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:47 smithi187 ceph-mon[33032]: pgmap v960: 168 pgs: 168 active+clean; 4.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.03k op/s 2024-09-21T14:07:48.010 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:47 smithi187 ceph-mon[33032]: 2.2 scrub starts 2024-09-21T14:07:48.010 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:47 smithi187 ceph-mon[33032]: 2.2 scrub ok 2024-09-21T14:07:48.010 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:47 smithi187 ceph-mon[33032]: 5.1f scrub starts 2024-09-21T14:07:48.010 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:47 smithi187 ceph-mon[33032]: 5.1f scrub ok 2024-09-21T14:07:48.010 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:47 smithi187 ceph-mon[33032]: 4.12 scrub starts 2024-09-21T14:07:48.010 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:47 smithi187 ceph-mon[33032]: 4.12 scrub ok 2024-09-21T14:07:48.010 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:47 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T14:07:48.011 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:47 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T14:07:48.011 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:47 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:07:48.011 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:47 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3337271436' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:07:48.011 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:47 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:07:48.043 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:48.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:48.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:07:48.067 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:07:48.067 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:07:48.104 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:07:48.104 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:07:48.138 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:07:48.138 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:07:48.145 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:48.178 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:07:48.179 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:07:48.182 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:48.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:48.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:48.216 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:07:48.217 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:07:48.274 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T14:07:48.274 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:07:48.285 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:07:48.317 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:48.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:48.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:07:48.419 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:07:48.449 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:48.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:48.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:07:48.515 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 55 15 86409 86394 98.1534 66.0625 0.018509 0.0101831 2024-09-21T14:07:48.551 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:07:48.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:48.581 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:48.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:07:48.682 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:07:48.725 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:48.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:48.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:07:48.827 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:07:48.866 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:48.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:48.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:07:48.926 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:48 smithi187 ceph-mon[33032]: 6.12 scrub starts 2024-09-21T14:07:48.926 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:48 smithi187 ceph-mon[33032]: 6.12 scrub ok 2024-09-21T14:07:48.926 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:48 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2406389403' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T14:07:48.927 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:48 smithi187 ceph-mon[33032]: 3.6 scrub starts 2024-09-21T14:07:48.927 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:48 smithi187 ceph-mon[33032]: 3.6 scrub ok 2024-09-21T14:07:48.927 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:48 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/4003115069' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:07:48.927 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:48 smithi187 ceph-mon[33032]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:07:48.927 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:48 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T14:07:48.927 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:48 smithi187 ceph-mon[33032]: osdmap e483: 8 total, 8 up, 8 in 2024-09-21T14:07:48.968 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:48.980 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T14:07:49.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:49.003 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:49.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:49.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:48 smithi049 ceph-mon[30080]: 6.12 scrub starts 2024-09-21T14:07:49.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:48 smithi049 ceph-mon[30080]: 6.12 scrub ok 2024-09-21T14:07:49.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:48 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2406389403' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T14:07:49.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:48 smithi049 ceph-mon[30080]: 3.6 scrub starts 2024-09-21T14:07:49.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:48 smithi049 ceph-mon[30080]: 3.6 scrub ok 2024-09-21T14:07:49.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:48 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/4003115069' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:07:49.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:48 smithi049 ceph-mon[30080]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:07:49.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:48 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T14:07:49.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:48 smithi049 ceph-mon[30080]: osdmap e483: 8 total, 8 up, 8 in 2024-09-21T14:07:49.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:48 smithi049 ceph-mon[40489]: 6.12 scrub starts 2024-09-21T14:07:49.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:48 smithi049 ceph-mon[40489]: 6.12 scrub ok 2024-09-21T14:07:49.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:48 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2406389403' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T14:07:49.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:48 smithi049 ceph-mon[40489]: 3.6 scrub starts 2024-09-21T14:07:49.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:48 smithi049 ceph-mon[40489]: 3.6 scrub ok 2024-09-21T14:07:49.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:48 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/4003115069' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:07:49.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:48 smithi049 ceph-mon[40489]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:07:49.007 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:48 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T14:07:49.007 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:48 smithi049 ceph-mon[40489]: osdmap e483: 8 total, 8 up, 8 in 2024-09-21T14:07:49.104 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:07:49.138 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:49.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:49.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:07:49.240 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:07:49.275 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:49.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:49.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:07:49.377 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:07:49.385 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:07:49.414 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:49.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:49.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:07:49.515 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:07:49.516 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 56 16 87756 87740 97.9028 84.125 0.00456011 0.0102093 2024-09-21T14:07:49.547 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:49.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:49.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:07:49.649 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:07:49.683 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:49.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:49.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:07:49.785 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:07:49.841 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:49.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:49.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:07:49.943 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:49.984 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:49.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:49.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:49.991 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T14:07:49.991 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:07:50.086 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:07:50.102 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:50 smithi187 ceph-mon[33032]: pgmap v962: 168 pgs: 168 active+clean; 5.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 793 B/s rd, 59 MiB/s wr, 939 op/s 2024-09-21T14:07:50.128 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:50.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:50.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:07:50.206 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:50 smithi049 ceph-mon[40489]: pgmap v962: 168 pgs: 168 active+clean; 5.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 793 B/s rd, 59 MiB/s wr, 939 op/s 2024-09-21T14:07:50.206 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:50 smithi049 ceph-mon[30080]: pgmap v962: 168 pgs: 168 active+clean; 5.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 793 B/s rd, 59 MiB/s wr, 939 op/s 2024-09-21T14:07:50.230 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:07:50.258 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:50.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:50.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:07:50.359 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:50.365 INFO:tasks.thrashosds.thrasher:j is {'epoch': 482, 'fsid': '40a3d2e8-781f-11ef-baf6-efdc52797490', 'created': '2024-09-21T13:42:57.838275+0000', 'modified': '2024-09-21T14:07:45.208083+0000', 'last_up_change': '2024-09-21T14:01:40.158454+0000', 'last_in_change': '2024-09-21T13:58:16.730861+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 22, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 11, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T13:46:21.096244+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 21, 'pg_placement_num': 21, 'pg_placement_num_target': 21, 'pg_num_target': 21, 'pg_num_pending': 21, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '438', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '436', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.2899999618530273, 'score_stable': 2.2899999618530273, 'optimal_score': 1, 'raw_score_acting': 2.2899999618530273, 'raw_score_stable': 2.2899999618530273, 'primary_affinity_weighted': 0.9783573150634766, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.9800000190734863}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-21T13:48:55.540712+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 63, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 0.9783573150634766, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.9800000190734863}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-21T13:48:56.417456+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.9783573150634766, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.9800000190734863}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-21T13:48:58.128455+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 0.9783573150634766, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.9800000190734863}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-21T13:49:00.069361+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9783573150634766, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.9800000190734863}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-21T13:49:02.127399+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9783573150634766, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.9800000190734863}}, {'pool': 11, 'pool_name': 'unique_pool_4', 'create_time': '2024-09-21T14:06:43.392407+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '465', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 0.9783573150634766, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.9800000190734863}}], 'osds': [{'osd': 0, 'uuid': '2595dca9-6219-4e0c-8728-751eb102e73a', 'up': 1, 'in': 1, 'weight': 0.6176910400390625, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 429, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6802', 'nonce': 2095561412}, {'type': 'v1', 'addr': '172.21.15.49:6803', 'nonce': 2095561412}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6804', 'nonce': 2095561412}, {'type': 'v1', 'addr': '172.21.15.49:6805', 'nonce': 2095561412}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6808', 'nonce': 2095561412}, {'type': 'v1', 'addr': '172.21.15.49:6809', 'nonce': 2095561412}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6806', 'nonce': 2095561412}, {'type': 'v1', 'addr': '172.21.15.49:6807', 'nonce': 2095561412}]}, 'public_addr': '172.21.15.49:6803/2095561412', 'cluster_addr': '172.21.15.49:6805/2095561412', 'heartbeat_back_addr': '172.21.15.49:6809/2095561412', 'heartbeat_front_addr': '172.21.15.49:6807/2095561412', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '0ba327f9-5524-4a40-abcc-729ded0b1917', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 461, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6810', 'nonce': 1382840527}, {'type': 'v1', 'addr': '172.21.15.49:6811', 'nonce': 1382840527}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6812', 'nonce': 1382840527}, {'type': 'v1', 'addr': '172.21.15.49:6813', 'nonce': 1382840527}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6816', 'nonce': 1382840527}, {'type': 'v1', 'addr': '172.21.15.49:6817', 'nonce': 1382840527}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6814', 'nonce': 1382840527}, {'type': 'v1', 'addr': '172.21.15.49:6815', 'nonce': 1382840527}]}, 'public_addr': '172.21.15.49:6811/1382840527', 'cluster_addr': '172.21.15.49:6813/1382840527', 'heartbeat_back_addr': '172.21.15.49:6817/1382840527', 'heartbeat_front_addr': '172.21.15.49:6815/1382840527', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '27a45750-204d-48fd-bb8f-ebb01273343d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.8268585205078125, 'last_clean_begin': 18, 'last_clean_end': 100, 'up_from': 130, 'up_thru': 461, 'down_at': 101, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6818', 'nonce': 1945314741}, {'type': 'v1', 'addr': '172.21.15.49:6819', 'nonce': 1945314741}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6820', 'nonce': 1945314741}, {'type': 'v1', 'addr': '172.21.15.49:6821', 'nonce': 1945314741}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6824', 'nonce': 1945314741}, {'type': 'v1', 'addr': '172.21.15.49:6825', 'nonce': 1945314741}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6822', 'nonce': 1945314741}, {'type': 'v1', 'addr': '172.21.15.49:6823', 'nonce': 1945314741}]}, 'public_addr': '172.21.15.49:6819/1945314741', 'cluster_addr': '172.21.15.49:6821/1945314741', 'heartbeat_back_addr': '172.21.15.49:6825/1945314741', 'heartbeat_front_addr': '172.21.15.49:6823/1945314741', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '528bc390-3dc9-4f3b-ab75-d7412c9625db', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 27, 'last_clean_end': 214, 'up_from': 240, 'up_thru': 461, 'down_at': 215, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6826', 'nonce': 886735721}, {'type': 'v1', 'addr': '172.21.15.49:6827', 'nonce': 886735721}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6828', 'nonce': 886735721}, {'type': 'v1', 'addr': '172.21.15.49:6829', 'nonce': 886735721}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6832', 'nonce': 886735721}, {'type': 'v1', 'addr': '172.21.15.49:6833', 'nonce': 886735721}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6830', 'nonce': 886735721}, {'type': 'v1', 'addr': '172.21.15.49:6831', 'nonce': 886735721}]}, 'public_addr': '172.21.15.49:6827/886735721', 'cluster_addr': '172.21.15.49:6829/886735721', 'heartbeat_back_addr': '172.21.15.49:6833/886735721', 'heartbeat_front_addr': '172.21.15.49:6831/886735721', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '6fc64698-2d37-4db0-bf3d-c7e612c45163', 'up': 1, 'in': 1, 'weight': 0.7888946533203125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 436, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6800', 'nonce': 1702447776}, {'type': 'v1', 'addr': '172.21.15.187:6801', 'nonce': 1702447776}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6802', 'nonce': 1702447776}, {'type': 'v1', 'addr': '172.21.15.187:6803', 'nonce': 1702447776}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6806', 'nonce': 1702447776}, {'type': 'v1', 'addr': '172.21.15.187:6807', 'nonce': 1702447776}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6804', 'nonce': 1702447776}, {'type': 'v1', 'addr': '172.21.15.187:6805', 'nonce': 1702447776}]}, 'public_addr': '172.21.15.187:6801/1702447776', 'cluster_addr': '172.21.15.187:6803/1702447776', 'heartbeat_back_addr': '172.21.15.187:6807/1702447776', 'heartbeat_front_addr': '172.21.15.187:6805/1702447776', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '8d7f62a1-7e92-4e06-9cad-86b96be514d8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 39, 'last_clean_end': 324, 'up_from': 345, 'up_thru': 471, 'down_at': 325, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6808', 'nonce': 1595353965}, {'type': 'v1', 'addr': '172.21.15.187:6809', 'nonce': 1595353965}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6810', 'nonce': 1595353965}, {'type': 'v1', 'addr': '172.21.15.187:6811', 'nonce': 1595353965}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6814', 'nonce': 1595353965}, {'type': 'v1', 'addr': '172.21.15.187:6815', 'nonce': 1595353965}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6812', 'nonce': 1595353965}, {'type': 'v1', 'addr': '172.21.15.187:6813', 'nonce': 1595353965}]}, 'public_addr': '172.21.15.187:6809/1595353965', 'cluster_addr': '172.21.15.187:6811/1595353965', 'heartbeat_back_addr': '172.21.15.187:6815/1595353965', 'heartbeat_front_addr': '172.21.15.187:6813/1595353965', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '3a7afd13-26ce-444c-9cc0-81ce2eab6ce5', 'up': 1, 'in': 1, 'weight': 0.9076080322265625, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 461, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6816', 'nonce': 2728426525}, {'type': 'v1', 'addr': '172.21.15.187:6817', 'nonce': 2728426525}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6818', 'nonce': 2728426525}, {'type': 'v1', 'addr': '172.21.15.187:6819', 'nonce': 2728426525}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6822', 'nonce': 2728426525}, {'type': 'v1', 'addr': '172.21.15.187:6823', 'nonce': 2728426525}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6820', 'nonce': 2728426525}, {'type': 'v1', 'addr': '172.21.15.187:6821', 'nonce': 2728426525}]}, 'public_addr': '172.21.15.187:6817/2728426525', 'cluster_addr': '172.21.15.187:6819/2728426525', 'heartbeat_back_addr': '172.21.15.187:6823/2728426525', 'heartbeat_front_addr': '172.21.15.187:6821/2728426525', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1e08d891-8747-402e-b4c2-aa3802ce8537', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 54, 'last_clean_end': 272, 'up_from': 298, 'up_thru': 461, 'down_at': 273, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 13894964}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 13894964}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 13894964}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 13894964}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6830', 'nonce': 13894964}, {'type': 'v1', 'addr': '172.21.15.187:6831', 'nonce': 13894964}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6828', 'nonce': 13894964}, {'type': 'v1', 'addr': '172.21.15.187:6829', 'nonce': 13894964}]}, 'public_addr': '172.21.15.187:6825/13894964', 'cluster_addr': '172.21.15.187:6827/13894964', 'heartbeat_back_addr': '172.21.15.187:6831/13894964', 'heartbeat_front_addr': '172.21.15.187:6829/13894964', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T14:06:38.896732+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T14:07:20.759253+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T14:06:27.643288+0000', 'dead_epoch': 100}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T14:06:58.391048+0000', 'dead_epoch': 214}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T14:06:40.662638+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T14:06:56.536868+0000', 'dead_epoch': 324}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T14:07:16.644516+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T14:07:20.884792+0000', 'dead_epoch': 272}], 'pg_upmap': [{'pgid': '3.14', 'osds': [7, 6, 0]}], 'pg_upmap_items': [{'pgid': '4.2', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.6', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.d', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '5.16', 'mappings': [{'from': 1, 'to': 6}, {'from': 3, 'to': 5}]}, {'pgid': '6.7', 'mappings': [{'from': 3, 'to': 7}]}, {'pgid': '6.b', 'mappings': [{'from': 3, 'to': 1}, {'from': 7, 'to': 6}]}, {'pgid': '6.1f', 'mappings': [{'from': 3, 'to': 0}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.49:0/2731793917': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:0/799831425': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:6801/2518197280': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:0/3556829486': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:0/1249926115': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:0/231530138': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:0/4220781134': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:0/202325455': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:6800/4129636473': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:6800/2518197280': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:0/3514741273': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:6801/4129636473': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:0/3125743776': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:0/3600977363': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:6801/3453571598': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:6800/3453571598': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:0/3553734432': '2024-09-22T13:49:27.311313+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-21T14:07:50.365 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 4.d 2024-09-21T14:07:50.365 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd rm-pg-upmap-items 4.d 2024-09-21T14:07:50.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:50.405 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:50.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:50.444 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:07:50 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:07:50.506 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:07:50.516 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 57 16 89055 89039 97.6091 81.1875 0.00550268 0.0102398 2024-09-21T14:07:50.537 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:50.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:50.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:07:50.639 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:07:50.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:50.679 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:50.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:07:50.694 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T14:07:50.736 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:07:50 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:07:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:07:50.780 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:07:50.813 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:50.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:50.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:07:50.915 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:50.926 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:07:50.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:50.966 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:50.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:51.067 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:51.112 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:51.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:51.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:51.116 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:07:51.214 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:07:51.242 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:51.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:51.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:07:51.344 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:07:51.374 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:51.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:51.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:07:51.425 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:51 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:07:51.425 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:51 smithi049 ceph-mon[40489]: pgmap v963: 168 pgs: 168 active+clean; 5.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-09-21T14:07:51.425 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:51 smithi049 ceph-mon[40489]: osdmap e484: 8 total, 8 up, 8 in 2024-09-21T14:07:51.425 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:51 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:07:51.425 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:51 smithi049 ceph-mon[30080]: pgmap v963: 168 pgs: 168 active+clean; 5.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-09-21T14:07:51.425 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:51 smithi049 ceph-mon[30080]: osdmap e484: 8 total, 8 up, 8 in 2024-09-21T14:07:51.475 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:07:51.501 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:51.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:51.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:07:51.516 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 58 15 90307 90292 97.2764 78.3125 0.0186895 0.0102744 2024-09-21T14:07:51.603 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:07:51.632 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:51.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:51.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:07:51.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:51 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:07:51.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:51 smithi187 ceph-mon[33032]: pgmap v963: 168 pgs: 168 active+clean; 5.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-09-21T14:07:51.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:51 smithi187 ceph-mon[33032]: osdmap e484: 8 total, 8 up, 8 in 2024-09-21T14:07:51.734 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:07:51.765 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:51.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:51.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:07:51.844 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T14:07:51.867 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:07:51.895 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:51.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:51.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:07:51.996 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:07:52.013 INFO:teuthology.orchestra.run.smithi049.stderr:clear 4.d pg_upmap_items mapping 2024-09-21T14:07:52.034 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:52.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:52.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:07:52.136 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:07:52.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:52.186 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:52.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:07:52.287 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:07:52.332 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:52.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:52.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:07:52.351 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T14:07:52.351 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:07:52.435 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:07:52.465 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:52.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:52.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:07:52.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:52 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:07:52.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:52 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:07:52.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:52 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:07:52.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:52 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:07:52.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:52 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1123602140' entity='client.admin' cmd={"prefix": "osd rm-pg-upmap-items", "pgid": "4.d"} : dispatch 2024-09-21T14:07:52.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:52 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1123602140' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.d"}]': finished 2024-09-21T14:07:52.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:52 smithi049 ceph-mon[40489]: osdmap e485: 8 total, 8 up, 8 in 2024-09-21T14:07:52.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:52 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:07:52.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:52 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:07:52.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:52 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:07:52.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:52 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:07:52.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:52 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1123602140' entity='client.admin' cmd={"prefix": "osd rm-pg-upmap-items", "pgid": "4.d"} : dispatch 2024-09-21T14:07:52.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:52 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1123602140' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.d"}]': finished 2024-09-21T14:07:52.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:52 smithi049 ceph-mon[30080]: osdmap e485: 8 total, 8 up, 8 in 2024-09-21T14:07:52.516 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 59 16 91461 91445 96.849 72.0625 0.00750702 0.0103206 2024-09-21T14:07:52.566 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:07:52.602 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:52.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:52.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:07:52.690 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:52 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:07:52.691 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:52 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:07:52.691 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:52 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:07:52.691 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:52 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:07:52.691 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:52 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1123602140' entity='client.admin' cmd={"prefix": "osd rm-pg-upmap-items", "pgid": "4.d"} : dispatch 2024-09-21T14:07:52.691 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:52 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1123602140' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.d"}]': finished 2024-09-21T14:07:52.691 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:52 smithi187 ceph-mon[33032]: osdmap e485: 8 total, 8 up, 8 in 2024-09-21T14:07:52.704 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:07:52.739 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:52.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:52.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:07:52.811 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T14:07:52.841 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:07:52.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:52.929 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:52.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:07:52.939 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:07:53.030 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:07:53.067 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:53.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:53.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:07:53.078 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T14:07:53.169 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:07:53.211 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:53.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:53.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:07:53.312 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:07:53.345 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:53.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:53.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:07:53.347 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:53 smithi049 ceph-mon[30080]: pgmap v966: 168 pgs: 168 active+clean; 5.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.7 KiB/s rd, 93 MiB/s wr, 1.50k op/s 2024-09-21T14:07:53.347 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:53 smithi049 ceph-mon[30080]: osdmap e486: 8 total, 8 up, 8 in 2024-09-21T14:07:53.347 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:53 smithi049 ceph-mon[40489]: pgmap v966: 168 pgs: 168 active+clean; 5.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.7 KiB/s rd, 93 MiB/s wr, 1.50k op/s 2024-09-21T14:07:53.347 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:53 smithi049 ceph-mon[40489]: osdmap e486: 8 total, 8 up, 8 in 2024-09-21T14:07:53.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:53 smithi187 ceph-mon[33032]: pgmap v966: 168 pgs: 168 active+clean; 5.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.7 KiB/s rd, 93 MiB/s wr, 1.50k op/s 2024-09-21T14:07:53.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:53 smithi187 ceph-mon[33032]: osdmap e486: 8 total, 8 up, 8 in 2024-09-21T14:07:53.447 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:07:53.481 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:07:53.495 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:53.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:53.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:07:53.517 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:2024-09-21T14:07:53.518450+0000 min lat: 0.00119458 max lat: 0.27709 avg lat: 0.0103627 2024-09-21T14:07:53.518 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T14:07:53.518 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 60 16 92632 92616 96.4529 73.1875 0.0100384 0.0103627 2024-09-21T14:07:53.597 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:07:53.652 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:53.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:53.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:07:53.669 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:07:53.755 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:07:53.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:53.791 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:53.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:07:53.892 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:53.935 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:53.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:53.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:54.037 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:07:54.067 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:54.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:54.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:07:54.168 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:07:54.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:54.211 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:54.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:07:54.237 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-09-21T14:07:54.312 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:07:54.347 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:54.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:54.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:07:54.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:54 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1523163303' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:07:54.448 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:54.487 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:54.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:54.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:54.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:54 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1523163303' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:07:54.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:54 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1523163303' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:07:54.517 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 61 16 93828 93812 96.097 74.75 0.00383984 0.0104012 2024-09-21T14:07:54.589 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:07:54.596 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T14:07:54.596 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:07:54.621 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:54.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:54.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:07:54.627 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:07:54.627 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:07:54.627 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:07:54.627 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:07:54.628 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:07:54.628 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:07:54.628 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:07:54.628 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:07:54.628 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:07:54.628 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:07:54.645 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:07:54.645 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:07:54.685 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:07:54.685 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:07:54.722 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:07:54.722 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:07:54.723 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:07:54.763 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:54.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:54.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:07:54.865 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:07:54.903 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:54.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:54.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:07:55.004 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:07:55.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:55.043 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:55.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:07:55.144 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:07:55.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:55.186 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:55.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:07:55.287 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:07:55.304 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:55 smithi187 ceph-mon[33032]: pgmap v968: 168 pgs: 168 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 77 MiB/s wr, 1.24k op/s; 18 B/s, 1 objects/s recovering 2024-09-21T14:07:55.304 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:55 smithi187 ceph-mon[33032]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:07:55.304 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:55 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1523163303' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T14:07:55.304 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:55 smithi187 ceph-mon[33032]: osdmap e487: 8 total, 8 up, 8 in 2024-09-21T14:07:55.304 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:55 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3933692940' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:07:55.304 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:55 smithi187 ceph-mon[33032]: osdmap e488: 8 total, 8 up, 8 in 2024-09-21T14:07:55.319 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:55.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:55.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:07:55.356 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T14:07:55.421 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:07:55.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:55.468 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:55.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:07:55.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:55 smithi049 ceph-mon[40489]: pgmap v968: 168 pgs: 168 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 77 MiB/s wr, 1.24k op/s; 18 B/s, 1 objects/s recovering 2024-09-21T14:07:55.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:55 smithi049 ceph-mon[40489]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:07:55.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:55 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1523163303' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T14:07:55.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:55 smithi049 ceph-mon[40489]: osdmap e487: 8 total, 8 up, 8 in 2024-09-21T14:07:55.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:55 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3933692940' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:07:55.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:55 smithi049 ceph-mon[40489]: osdmap e488: 8 total, 8 up, 8 in 2024-09-21T14:07:55.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:55 smithi049 ceph-mon[30080]: pgmap v968: 168 pgs: 168 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 77 MiB/s wr, 1.24k op/s; 18 B/s, 1 objects/s recovering 2024-09-21T14:07:55.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:55 smithi049 ceph-mon[30080]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:07:55.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:55 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1523163303' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T14:07:55.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:55 smithi049 ceph-mon[30080]: osdmap e487: 8 total, 8 up, 8 in 2024-09-21T14:07:55.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:55 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3933692940' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:07:55.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:55 smithi049 ceph-mon[30080]: osdmap e488: 8 total, 8 up, 8 in 2024-09-21T14:07:55.517 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 62 16 95102 95086 95.8312 79.625 0.0137517 0.0104298 2024-09-21T14:07:55.569 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:07:55.609 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:55.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:55.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:07:55.712 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:07:55.746 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:55.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:55.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:07:55.787 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:07:55.848 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:55.883 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:55.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:55.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:55.986 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:07:56.023 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:56.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:56.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:07:56.125 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:07:56.158 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:56.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:56.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:07:56.260 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:07:56.299 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:56.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:56.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:07:56.401 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:07:56.402 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T14:07:56.402 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-21T14:07:56.402 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-21T14:07:56.403 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-21T14:07:56.403 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.403 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187592 9.5 9:a5cf25e0:::benchmark_data_smithi049_138504_object66184:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:07:56.403 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:41.268428+0000", 2024-09-21T14:07:56.403 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 375.131002868, 2024-09-21T14:07:56.403 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3375785200000001, 2024-09-21T14:07:56.403 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:07:56.403 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:07:56.403 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:07:56.404 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:07:56.404 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:07:56.404 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:07:56.404 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187592 2024-09-21T14:07:56.404 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.404 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:07:56.404 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.404 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:07:56.405 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268428+0000", 2024-09-21T14:07:56.405 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:07:56.405 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.405 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.405 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:07:56.405 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268428+0000", 2024-09-21T14:07:56.405 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:07:56.405 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.405 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.406 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:07:56.406 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268429+0000", 2024-09-21T14:07:56.406 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.466e-06 2024-09-21T14:07:56.406 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.406 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.406 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:07:56.406 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268435+0000", 2024-09-21T14:07:56.406 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5539999999999997e-06 2024-09-21T14:07:56.407 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.407 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.407 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:07:56.407 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268445+0000", 2024-09-21T14:07:56.407 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0366999999999999e-05 2024-09-21T14:07:56.407 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.407 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.407 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:07:56.408 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268452+0000", 2024-09-21T14:07:56.408 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8870000000000003e-06 2024-09-21T14:07:56.408 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.408 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.408 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:07:56.408 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.289931+0000", 2024-09-21T14:07:56.408 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.021479166000000001 2024-09-21T14:07:56.408 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.408 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.409 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:07:56.409 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.289937+0000", 2024-09-21T14:07:56.409 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4140000000000002e-06 2024-09-21T14:07:56.409 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.409 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.409 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:07:56.409 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.566606+0000", 2024-09-21T14:07:56.409 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2766695319999997 2024-09-21T14:07:56.409 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.410 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.410 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:07:56.410 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.569571+0000", 2024-09-21T14:07:56.410 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0029643489999999998 2024-09-21T14:07:56.410 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.410 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.410 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:07:56.410 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.573938+0000", 2024-09-21T14:07:56.411 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0043675099999999998 2024-09-21T14:07:56.411 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.411 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.411 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:07:56.411 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.589876+0000", 2024-09-21T14:07:56.411 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.015938249000000002 2024-09-21T14:07:56.411 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.411 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.411 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:56.412 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.597274+0000", 2024-09-21T14:07:56.412 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0073979509999999998 2024-09-21T14:07:56.412 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.412 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.412 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:56.412 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.598324+0000", 2024-09-21T14:07:56.412 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001050156 2024-09-21T14:07:56.412 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.412 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.413 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:07:56.413 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.599807+0000", 2024-09-21T14:07:56.413 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0014825439999999999 2024-09-21T14:07:56.413 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.413 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.413 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:07:56.413 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.606006+0000", 2024-09-21T14:07:56.413 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.006199375 2024-09-21T14:07:56.414 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:56.414 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:07:56.414 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:56.414 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.414 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.414 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187594 9.5 9:a4584f0e:::benchmark_data_smithi049_138504_object66186:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:07:56.414 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:41.268527+0000", 2024-09-21T14:07:56.414 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 375.13090335300001, 2024-09-21T14:07:56.414 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3614060480000001, 2024-09-21T14:07:56.415 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:07:56.415 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:07:56.415 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:07:56.415 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:07:56.415 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:07:56.415 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:07:56.415 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187594 2024-09-21T14:07:56.415 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.415 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:07:56.416 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.416 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:07:56.416 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268527+0000", 2024-09-21T14:07:56.416 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:07:56.416 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.416 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.416 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:07:56.416 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268527+0000", 2024-09-21T14:07:56.416 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:07:56.416 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.416 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.417 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:07:56.417 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268528+0000", 2024-09-21T14:07:56.417 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5299999999999999e-07 2024-09-21T14:07:56.417 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.417 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.417 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:07:56.417 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268531+0000", 2024-09-21T14:07:56.417 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.157e-06 2024-09-21T14:07:56.417 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.417 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.417 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:07:56.417 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268538+0000", 2024-09-21T14:07:56.417 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9290000000000001e-06 2024-09-21T14:07:56.417 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.418 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.418 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:07:56.418 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268543+0000", 2024-09-21T14:07:56.418 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0409999999999997e-06 2024-09-21T14:07:56.418 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.418 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.418 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:07:56.418 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.289953+0000", 2024-09-21T14:07:56.418 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.021409811000000001 2024-09-21T14:07:56.418 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.418 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.419 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:07:56.419 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.289957+0000", 2024-09-21T14:07:56.419 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9489999999999998e-06 2024-09-21T14:07:56.419 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.419 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.419 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:07:56.419 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.606351+0000", 2024-09-21T14:07:56.419 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3163947599999997 2024-09-21T14:07:56.419 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.419 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.419 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:07:56.419 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.610368+0000", 2024-09-21T14:07:56.419 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0040163289999999999 2024-09-21T14:07:56.419 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.419 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.420 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:07:56.420 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.615560+0000", 2024-09-21T14:07:56.420 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0051918939999999998 2024-09-21T14:07:56.420 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.420 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.420 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:56.420 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.622425+0000", 2024-09-21T14:07:56.420 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0068655959999999999 2024-09-21T14:07:56.420 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.420 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.420 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:56.420 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.623509+0000", 2024-09-21T14:07:56.420 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001083669 2024-09-21T14:07:56.420 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.420 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.421 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:07:56.421 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.624009+0000", 2024-09-21T14:07:56.421 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00050001600000000005 2024-09-21T14:07:56.421 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.421 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.421 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:07:56.421 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.625282+0000", 2024-09-21T14:07:56.421 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0012730599999999999 2024-09-21T14:07:56.421 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.421 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.421 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:07:56.421 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.629933+0000", 2024-09-21T14:07:56.421 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0046513839999999997 2024-09-21T14:07:56.422 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:56.422 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:07:56.422 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:56.422 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.422 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.422 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187607 9.7 9:ea996e28:::benchmark_data_smithi049_138504_object66199:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:07:56.422 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:41.286116+0000", 2024-09-21T14:07:56.422 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 375.11331470300001, 2024-09-21T14:07:56.422 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0827087349999998, 2024-09-21T14:07:56.422 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:07:56.422 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:07:56.422 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:07:56.422 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:07:56.422 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:07:56.423 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:07:56.423 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187607 2024-09-21T14:07:56.423 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.423 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:07:56.423 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.423 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:07:56.423 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.286116+0000", 2024-09-21T14:07:56.423 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:07:56.423 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.423 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.423 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:07:56.423 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.286116+0000", 2024-09-21T14:07:56.423 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:07:56.423 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.423 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.423 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:07:56.424 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.286117+0000", 2024-09-21T14:07:56.424 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.097e-06 2024-09-21T14:07:56.424 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.424 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.424 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:07:56.424 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.286120+0000", 2024-09-21T14:07:56.424 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.224e-06 2024-09-21T14:07:56.424 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.424 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.424 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:07:56.424 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.286129+0000", 2024-09-21T14:07:56.424 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3659999999999992e-06 2024-09-21T14:07:56.424 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.424 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.424 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:07:56.424 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.286134+0000", 2024-09-21T14:07:56.425 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6130000000000003e-06 2024-09-21T14:07:56.425 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.425 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.425 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:07:56.425 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.286627+0000", 2024-09-21T14:07:56.425 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00049266000000000002 2024-09-21T14:07:56.425 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.425 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.425 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:07:56.425 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.286663+0000", 2024-09-21T14:07:56.425 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6164999999999997e-05 2024-09-21T14:07:56.425 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.425 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.425 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:07:56.426 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.330333+0000", 2024-09-21T14:07:56.426 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0436700029999999 2024-09-21T14:07:56.426 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.426 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.426 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:07:56.426 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.333462+0000", 2024-09-21T14:07:56.426 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003128881 2024-09-21T14:07:56.426 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.426 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.426 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,7", 2024-09-21T14:07:56.426 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.337950+0000", 2024-09-21T14:07:56.426 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0044878089999999997 2024-09-21T14:07:56.426 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.426 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.427 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:07:56.427 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353871+0000", 2024-09-21T14:07:56.427 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.015921270000000001 2024-09-21T14:07:56.427 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.427 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.427 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:56.427 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.360705+0000", 2024-09-21T14:07:56.427 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0068337110000000001 2024-09-21T14:07:56.427 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.427 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.427 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:56.427 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.362017+0000", 2024-09-21T14:07:56.427 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0013119749999999999 2024-09-21T14:07:56.427 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.427 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.428 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:07:56.428 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.364166+0000", 2024-09-21T14:07:56.428 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0021495030000000001 2024-09-21T14:07:56.428 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.428 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.428 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:07:56.428 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.368825+0000", 2024-09-21T14:07:56.428 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0046584579999999999 2024-09-21T14:07:56.428 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:56.428 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:07:56.428 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:56.428 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.428 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.429 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187609 9.5 9:a0a81d79:::benchmark_data_smithi049_138504_object66201:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:07:56.429 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:41.287003+0000", 2024-09-21T14:07:56.429 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 375.11242809200002, 2024-09-21T14:07:56.429 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3857519199999997, 2024-09-21T14:07:56.429 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:07:56.429 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:07:56.429 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:07:56.429 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:07:56.429 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:07:56.429 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:07:56.429 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187609 2024-09-21T14:07:56.429 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.429 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:07:56.429 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.430 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:07:56.430 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.287003+0000", 2024-09-21T14:07:56.430 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:07:56.430 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.430 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.430 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:07:56.430 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.287003+0000", 2024-09-21T14:07:56.430 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:07:56.430 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.430 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.430 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:07:56.430 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.287003+0000", 2024-09-21T14:07:56.430 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8400000000000003e-07 2024-09-21T14:07:56.430 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.431 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.431 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:07:56.431 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.287017+0000", 2024-09-21T14:07:56.431 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3831e-05 2024-09-21T14:07:56.431 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.431 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.431 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:07:56.431 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.287026+0000", 2024-09-21T14:07:56.431 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8389999999999996e-06 2024-09-21T14:07:56.431 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.431 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.431 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:07:56.431 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.287032+0000", 2024-09-21T14:07:56.431 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3509999999999998e-06 2024-09-21T14:07:56.432 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.432 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.432 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:07:56.432 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.289969+0000", 2024-09-21T14:07:56.432 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0029364399999999998 2024-09-21T14:07:56.432 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.432 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.432 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:07:56.432 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.289973+0000", 2024-09-21T14:07:56.432 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6339999999999999e-06 2024-09-21T14:07:56.432 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.432 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.432 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:07:56.432 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.631003+0000", 2024-09-21T14:07:56.432 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3410304320000002 2024-09-21T14:07:56.432 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.433 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.433 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:07:56.433 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.634397+0000", 2024-09-21T14:07:56.433 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0033938369999999998 2024-09-21T14:07:56.433 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.433 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.433 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:07:56.433 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.639017+0000", 2024-09-21T14:07:56.433 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0046205889999999996 2024-09-21T14:07:56.433 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.433 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.433 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:07:56.433 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.662137+0000", 2024-09-21T14:07:56.433 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.023119475 2024-09-21T14:07:56.434 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.434 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.434 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:56.434 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.663533+0000", 2024-09-21T14:07:56.434 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0013956159999999999 2024-09-21T14:07:56.434 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.434 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.434 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:56.434 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.664730+0000", 2024-09-21T14:07:56.434 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0011970139999999999 2024-09-21T14:07:56.434 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.434 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.435 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:07:56.435 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.666523+0000", 2024-09-21T14:07:56.435 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001793046 2024-09-21T14:07:56.435 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.436 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.436 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:07:56.436 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.672754+0000", 2024-09-21T14:07:56.436 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0062319319999999999 2024-09-21T14:07:56.436 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:56.436 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:07:56.436 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:56.436 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.436 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.436 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187621 9.5 9:a3e366d7:::benchmark_data_smithi049_138504_object66213:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:07:56.436 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:41.300495+0000", 2024-09-21T14:07:56.436 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 375.09893544400001, 2024-09-21T14:07:56.436 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3801383449999998, 2024-09-21T14:07:56.437 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:07:56.437 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:07:56.437 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:07:56.437 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:07:56.437 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:07:56.437 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:07:56.437 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187621 2024-09-21T14:07:56.437 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.437 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:07:56.437 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.437 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:07:56.437 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300495+0000", 2024-09-21T14:07:56.437 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:07:56.438 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.438 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.438 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:07:56.438 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300495+0000", 2024-09-21T14:07:56.438 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:07:56.438 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.438 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.438 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:07:56.438 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300497+0000", 2024-09-21T14:07:56.438 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3710000000000001e-06 2024-09-21T14:07:56.438 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.438 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.438 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:07:56.438 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300502+0000", 2024-09-21T14:07:56.439 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3569999999999997e-06 2024-09-21T14:07:56.439 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.439 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.439 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:07:56.439 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300511+0000", 2024-09-21T14:07:56.439 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.9509999999999995e-06 2024-09-21T14:07:56.439 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.439 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.439 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:07:56.439 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300517+0000", 2024-09-21T14:07:56.439 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2489999999999998e-06 2024-09-21T14:07:56.439 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.439 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.439 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:07:56.440 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300606+0000", 2024-09-21T14:07:56.440 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8794000000000004e-05 2024-09-21T14:07:56.440 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.440 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.440 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:07:56.440 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300616+0000", 2024-09-21T14:07:56.440 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0338e-05 2024-09-21T14:07:56.440 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.440 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.440 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:07:56.440 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.643702+0000", 2024-09-21T14:07:56.440 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.343085759 2024-09-21T14:07:56.440 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.440 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.441 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:07:56.441 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.648030+0000", 2024-09-21T14:07:56.441 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0043275859999999996 2024-09-21T14:07:56.441 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.441 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.441 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:07:56.441 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.653318+0000", 2024-09-21T14:07:56.441 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.005287936 2024-09-21T14:07:56.441 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.441 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.441 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:07:56.441 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.662422+0000", 2024-09-21T14:07:56.441 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0091040340000000004 2024-09-21T14:07:56.441 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.442 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.442 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:56.442 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.673842+0000", 2024-09-21T14:07:56.442 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.011420382999999999 2024-09-21T14:07:56.442 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.442 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.442 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:56.442 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.675052+0000", 2024-09-21T14:07:56.442 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0012104290000000001 2024-09-21T14:07:56.442 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.442 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.442 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:07:56.443 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.676489+0000", 2024-09-21T14:07:56.443 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0014364309999999999 2024-09-21T14:07:56.443 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.443 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.443 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:07:56.443 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.680634+0000", 2024-09-21T14:07:56.443 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.004144727 2024-09-21T14:07:56.443 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:56.443 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:07:56.443 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:56.443 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.443 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.443 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187623 9.7 9:e79a683b:::benchmark_data_smithi049_138504_object66215:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:07:56.444 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:41.300818+0000", 2024-09-21T14:07:56.444 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 375.09861303999998, 2024-09-21T14:07:56.444 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1014428919999997, 2024-09-21T14:07:56.444 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:07:56.444 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:07:56.444 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:07:56.444 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:07:56.444 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:07:56.444 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:07:56.445 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187623 2024-09-21T14:07:56.445 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.445 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:07:56.445 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.445 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:07:56.445 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300818+0000", 2024-09-21T14:07:56.445 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:07:56.445 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.445 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.446 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:07:56.446 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300818+0000", 2024-09-21T14:07:56.446 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:07:56.446 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.446 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.446 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:07:56.446 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300818+0000", 2024-09-21T14:07:56.446 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.54e-07 2024-09-21T14:07:56.446 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.447 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.447 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:07:56.447 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300822+0000", 2024-09-21T14:07:56.447 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6770000000000001e-06 2024-09-21T14:07:56.447 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.447 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.447 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:07:56.447 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300832+0000", 2024-09-21T14:07:56.447 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0202e-05 2024-09-21T14:07:56.448 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.448 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.448 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:07:56.448 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300847+0000", 2024-09-21T14:07:56.448 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4992e-05 2024-09-21T14:07:56.448 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.448 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.448 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:07:56.448 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300931+0000", 2024-09-21T14:07:56.448 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3577000000000003e-05 2024-09-21T14:07:56.449 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.449 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.449 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:07:56.449 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300940+0000", 2024-09-21T14:07:56.449 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8009999999999993e-06 2024-09-21T14:07:56.449 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.449 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.449 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:07:56.450 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.370531+0000", 2024-09-21T14:07:56.450 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0695910870000001 2024-09-21T14:07:56.450 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.450 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.450 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:07:56.450 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.373842+0000", 2024-09-21T14:07:56.450 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0033110259999999999 2024-09-21T14:07:56.450 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.450 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.451 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,7", 2024-09-21T14:07:56.451 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.378337+0000", 2024-09-21T14:07:56.451 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0044951100000000001 2024-09-21T14:07:56.451 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.451 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.451 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:56.451 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.394514+0000", 2024-09-21T14:07:56.451 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.016176877999999999 2024-09-21T14:07:56.451 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.452 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.452 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:07:56.452 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.395030+0000", 2024-09-21T14:07:56.452 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00051615000000000005 2024-09-21T14:07:56.452 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.452 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.452 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:56.452 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.395803+0000", 2024-09-21T14:07:56.453 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00077358799999999997 2024-09-21T14:07:56.453 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.453 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.453 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:07:56.453 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.397305+0000", 2024-09-21T14:07:56.453 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001501811 2024-09-21T14:07:56.453 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.453 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.453 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:07:56.454 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.402261+0000", 2024-09-21T14:07:56.454 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0049552390000000002 2024-09-21T14:07:56.454 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:56.454 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:07:56.454 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:56.454 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.454 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.454 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187624 9.5 9:a181e4d0:::benchmark_data_smithi049_138504_object66216:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:07:56.454 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:41.300871+0000", 2024-09-21T14:07:56.455 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 375.09856003700003, 2024-09-21T14:07:56.455 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4164464030000001, 2024-09-21T14:07:56.455 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:07:56.455 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:07:56.455 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:07:56.455 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:07:56.455 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:07:56.455 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:07:56.455 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187624 2024-09-21T14:07:56.456 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.456 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:07:56.456 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.456 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:07:56.456 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300871+0000", 2024-09-21T14:07:56.456 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:07:56.456 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.456 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.456 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:07:56.457 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300871+0000", 2024-09-21T14:07:56.457 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:07:56.457 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.457 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.457 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:07:56.457 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300871+0000", 2024-09-21T14:07:56.457 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8599999999999998e-07 2024-09-21T14:07:56.457 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.458 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.458 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:07:56.458 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300875+0000", 2024-09-21T14:07:56.458 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7720000000000001e-06 2024-09-21T14:07:56.458 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.458 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.458 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:07:56.458 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300886+0000", 2024-09-21T14:07:56.459 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1058e-05 2024-09-21T14:07:56.459 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.459 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.459 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:07:56.459 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300892+0000", 2024-09-21T14:07:56.459 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.161e-06 2024-09-21T14:07:56.459 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.459 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.459 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:07:56.459 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.301287+0000", 2024-09-21T14:07:56.459 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00039522800000000002 2024-09-21T14:07:56.459 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.459 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.460 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:07:56.460 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.301297+0000", 2024-09-21T14:07:56.460 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6209999999999999e-06 2024-09-21T14:07:56.460 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.460 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.460 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:07:56.460 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.681021+0000", 2024-09-21T14:07:56.460 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3797241769999999 2024-09-21T14:07:56.460 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.460 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.460 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:07:56.460 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.685054+0000", 2024-09-21T14:07:56.460 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0040323909999999998 2024-09-21T14:07:56.460 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.461 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.461 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:07:56.461 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.689479+0000", 2024-09-21T14:07:56.461 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0044253740000000001 2024-09-21T14:07:56.461 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.461 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.461 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:56.461 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.710060+0000", 2024-09-21T14:07:56.461 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.020581051999999999 2024-09-21T14:07:56.461 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.461 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.461 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:07:56.461 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.710641+0000", 2024-09-21T14:07:56.461 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00058055800000000001 2024-09-21T14:07:56.462 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.462 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.462 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:56.462 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.711399+0000", 2024-09-21T14:07:56.462 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00075836299999999996 2024-09-21T14:07:56.462 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.462 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.462 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:07:56.462 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.712857+0000", 2024-09-21T14:07:56.462 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001458373 2024-09-21T14:07:56.462 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.462 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.462 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:07:56.463 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.717317+0000", 2024-09-21T14:07:56.463 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0044596890000000002 2024-09-21T14:07:56.463 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:56.463 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:07:56.463 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:56.463 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.463 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.463 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187626 9.7 9:e3df99cb:::benchmark_data_smithi049_138504_object66218:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:07:56.463 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:41.304965+0000", 2024-09-21T14:07:56.463 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 375.09446590099998, 2024-09-21T14:07:56.463 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1061633610000001, 2024-09-21T14:07:56.463 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:07:56.463 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:07:56.463 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:07:56.463 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:07:56.464 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:07:56.464 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:07:56.464 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187626 2024-09-21T14:07:56.464 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.464 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:07:56.464 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.464 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:07:56.464 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.304965+0000", 2024-09-21T14:07:56.464 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:07:56.464 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.464 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.464 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:07:56.465 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.304965+0000", 2024-09-21T14:07:56.465 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:07:56.465 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.465 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.465 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:07:56.465 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.304966+0000", 2024-09-21T14:07:56.465 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6619999999999999e-06 2024-09-21T14:07:56.465 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.465 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.465 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:07:56.465 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.304973+0000", 2024-09-21T14:07:56.465 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1759999999999998e-06 2024-09-21T14:07:56.465 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.465 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.465 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:07:56.465 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.304987+0000", 2024-09-21T14:07:56.465 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4372999999999999e-05 2024-09-21T14:07:56.466 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.466 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.466 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:07:56.466 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.304995+0000", 2024-09-21T14:07:56.466 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4109999999999996e-06 2024-09-21T14:07:56.466 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.466 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.466 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:07:56.466 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.305067+0000", 2024-09-21T14:07:56.466 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1833000000000002e-05 2024-09-21T14:07:56.466 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.466 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.466 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:07:56.466 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.305077+0000", 2024-09-21T14:07:56.466 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0161e-05 2024-09-21T14:07:56.466 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.467 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.467 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:07:56.467 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.382147+0000", 2024-09-21T14:07:56.467 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0770691709999998 2024-09-21T14:07:56.467 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.467 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.467 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:07:56.467 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.385209+0000", 2024-09-21T14:07:56.467 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0030624139999999998 2024-09-21T14:07:56.467 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.467 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.467 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,7", 2024-09-21T14:07:56.467 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.389597+0000", 2024-09-21T14:07:56.467 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.004388477 2024-09-21T14:07:56.467 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.467 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.468 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:56.468 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.403080+0000", 2024-09-21T14:07:56.468 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.013482984 2024-09-21T14:07:56.468 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.468 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.468 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:07:56.468 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.403583+0000", 2024-09-21T14:07:56.468 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00050300099999999999 2024-09-21T14:07:56.468 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.468 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.468 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:56.468 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.404464+0000", 2024-09-21T14:07:56.468 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00088034599999999995 2024-09-21T14:07:56.468 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.469 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.469 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:07:56.469 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.406114+0000", 2024-09-21T14:07:56.469 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0016504499999999999 2024-09-21T14:07:56.469 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.469 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.469 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:07:56.469 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.411128+0000", 2024-09-21T14:07:56.469 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0050139019999999998 2024-09-21T14:07:56.469 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:56.469 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:07:56.469 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:56.469 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.469 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.469 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187630 9.5 9:afb0e788:::benchmark_data_smithi049_138504_object66222:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:07:56.469 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:41.309934+0000", 2024-09-21T14:07:56.470 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 375.08949632000002, 2024-09-21T14:07:56.470 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4164949509999998, 2024-09-21T14:07:56.470 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:07:56.470 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:07:56.470 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:07:56.470 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:07:56.470 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:07:56.470 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:07:56.470 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187630 2024-09-21T14:07:56.470 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.470 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:07:56.470 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.470 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:07:56.470 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.309934+0000", 2024-09-21T14:07:56.470 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:07:56.471 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.471 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.471 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:07:56.471 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.309934+0000", 2024-09-21T14:07:56.471 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:07:56.471 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.471 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.471 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:07:56.471 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.309936+0000", 2024-09-21T14:07:56.471 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7460000000000001e-06 2024-09-21T14:07:56.471 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.471 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.471 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:07:56.471 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.309943+0000", 2024-09-21T14:07:56.471 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6579999999999996e-06 2024-09-21T14:07:56.472 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.472 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.472 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:07:56.472 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.309957+0000", 2024-09-21T14:07:56.472 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4496e-05 2024-09-21T14:07:56.472 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.472 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.472 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:07:56.472 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.309964+0000", 2024-09-21T14:07:56.472 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2219999999999996e-06 2024-09-21T14:07:56.472 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.472 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.473 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:07:56.473 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.310043+0000", 2024-09-21T14:07:56.473 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8801999999999995e-05 2024-09-21T14:07:56.473 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.473 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.473 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:07:56.473 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.310055+0000", 2024-09-21T14:07:56.473 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1637999999999999e-05 2024-09-21T14:07:56.473 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.473 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.473 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:07:56.473 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.693253+0000", 2024-09-21T14:07:56.473 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3831976189999997 2024-09-21T14:07:56.474 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.474 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.474 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:07:56.474 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.697465+0000", 2024-09-21T14:07:56.474 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0042119770000000004 2024-09-21T14:07:56.474 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.474 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.474 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:07:56.474 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.703657+0000", 2024-09-21T14:07:56.474 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0061928870000000002 2024-09-21T14:07:56.474 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.474 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.474 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:07:56.475 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.717616+0000", 2024-09-21T14:07:56.475 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.013958435999999999 2024-09-21T14:07:56.475 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.475 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.475 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:56.475 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.718555+0000", 2024-09-21T14:07:56.475 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00093888999999999995 2024-09-21T14:07:56.475 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.475 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.475 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:56.475 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.719633+0000", 2024-09-21T14:07:56.475 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0010787150000000001 2024-09-21T14:07:56.475 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.475 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.475 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:07:56.476 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.721631+0000", 2024-09-21T14:07:56.476 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0019973220000000002 2024-09-21T14:07:56.476 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.476 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.476 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:07:56.476 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.726429+0000", 2024-09-21T14:07:56.476 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0047985429999999997 2024-09-21T14:07:56.476 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:56.476 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:07:56.476 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:56.476 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.476 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.476 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187631 9.7 9:eb4378cd:::benchmark_data_smithi049_138504_object66223:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:07:56.477 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:41.310203+0000", 2024-09-21T14:07:56.477 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 375.08922781199999, 2024-09-21T14:07:56.477 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1434362220000001, 2024-09-21T14:07:56.477 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:07:56.477 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:07:56.477 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:07:56.477 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:07:56.477 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:07:56.477 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:07:56.477 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187631 2024-09-21T14:07:56.477 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.477 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:07:56.477 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.477 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:07:56.478 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.310203+0000", 2024-09-21T14:07:56.478 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:07:56.478 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.478 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.478 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:07:56.478 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.310203+0000", 2024-09-21T14:07:56.478 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:07:56.478 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.478 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.478 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:07:56.478 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.310204+0000", 2024-09-21T14:07:56.478 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3699999999999999e-07 2024-09-21T14:07:56.478 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.478 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.478 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:07:56.479 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.310209+0000", 2024-09-21T14:07:56.479 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7039999999999999e-06 2024-09-21T14:07:56.479 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.479 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.479 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:07:56.479 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.310219+0000", 2024-09-21T14:07:56.479 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7319999999999993e-06 2024-09-21T14:07:56.479 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.479 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.479 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:07:56.479 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.310226+0000", 2024-09-21T14:07:56.479 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4239999999999998e-06 2024-09-21T14:07:56.479 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.480 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.480 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:07:56.480 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.310279+0000", 2024-09-21T14:07:56.480 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3124999999999997e-05 2024-09-21T14:07:56.480 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.480 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.480 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:07:56.480 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.310288+0000", 2024-09-21T14:07:56.480 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3149999999999998e-06 2024-09-21T14:07:56.480 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.480 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.480 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:07:56.480 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.416040+0000", 2024-09-21T14:07:56.480 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1057518990000004 2024-09-21T14:07:56.481 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.481 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.481 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:07:56.481 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.419285+0000", 2024-09-21T14:07:56.481 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0032447399999999999 2024-09-21T14:07:56.481 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.481 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.481 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,7", 2024-09-21T14:07:56.481 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.423815+0000", 2024-09-21T14:07:56.481 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0045301509999999996 2024-09-21T14:07:56.481 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.481 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.482 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:56.482 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.439415+0000", 2024-09-21T14:07:56.482 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.015600334 2024-09-21T14:07:56.482 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.483 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.483 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:56.484 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.440566+0000", 2024-09-21T14:07:56.484 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0011505129999999999 2024-09-21T14:07:56.484 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.484 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.484 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:07:56.484 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.447676+0000", 2024-09-21T14:07:56.484 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0071103659999999999 2024-09-21T14:07:56.484 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.484 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.484 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:07:56.484 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.448927+0000", 2024-09-21T14:07:56.484 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0012508759999999999 2024-09-21T14:07:56.484 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.485 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.485 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:07:56.485 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.453639+0000", 2024-09-21T14:07:56.485 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0047121059999999998 2024-09-21T14:07:56.485 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:56.485 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:07:56.485 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:56.485 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.485 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.485 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187650 9.5 9:ab0f6d84:::benchmark_data_smithi049_138504_object66242:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:07:56.485 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:41.359224+0000", 2024-09-21T14:07:56.485 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 375.04020631100002, 2024-09-21T14:07:56.486 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4044763060000003, 2024-09-21T14:07:56.486 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:07:56.486 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:07:56.486 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:07:56.486 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:07:56.486 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:07:56.486 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:07:56.486 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187650 2024-09-21T14:07:56.486 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.486 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:07:56.486 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.486 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:07:56.486 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.359224+0000", 2024-09-21T14:07:56.486 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:07:56.487 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.487 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.487 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:07:56.487 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.359224+0000", 2024-09-21T14:07:56.487 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:07:56.487 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.487 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.487 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:07:56.487 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.359226+0000", 2024-09-21T14:07:56.487 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.423e-06 2024-09-21T14:07:56.487 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.487 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.487 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:07:56.487 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.359231+0000", 2024-09-21T14:07:56.487 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0980000000000001e-06 2024-09-21T14:07:56.488 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.488 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.488 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:07:56.488 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.359243+0000", 2024-09-21T14:07:56.488 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2316e-05 2024-09-21T14:07:56.488 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.488 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.488 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:07:56.488 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.359250+0000", 2024-09-21T14:07:56.488 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8820000000000003e-06 2024-09-21T14:07:56.488 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.488 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.488 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:07:56.488 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.359329+0000", 2024-09-21T14:07:56.489 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8541000000000006e-05 2024-09-21T14:07:56.489 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.489 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.489 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:07:56.489 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.359341+0000", 2024-09-21T14:07:56.489 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2238e-05 2024-09-21T14:07:56.489 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.489 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.489 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:07:56.489 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.726850+0000", 2024-09-21T14:07:56.489 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3675093619999998 2024-09-21T14:07:56.489 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.489 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.489 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:07:56.490 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.731153+0000", 2024-09-21T14:07:56.490 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0043027860000000003 2024-09-21T14:07:56.490 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.490 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.490 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:07:56.490 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.737448+0000", 2024-09-21T14:07:56.490 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0062951559999999997 2024-09-21T14:07:56.490 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.490 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.490 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:56.490 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.754776+0000", 2024-09-21T14:07:56.490 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.017328300000000001 2024-09-21T14:07:56.490 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.491 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.491 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:56.491 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.756004+0000", 2024-09-21T14:07:56.491 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001227337 2024-09-21T14:07:56.491 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.491 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.491 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:07:56.491 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.757990+0000", 2024-09-21T14:07:56.491 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0019857809999999998 2024-09-21T14:07:56.491 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.491 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.491 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:07:56.491 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.759321+0000", 2024-09-21T14:07:56.491 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001331203 2024-09-21T14:07:56.491 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.492 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.492 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:07:56.492 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.763701+0000", 2024-09-21T14:07:56.492 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0043798830000000002 2024-09-21T14:07:56.492 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:56.492 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:07:56.492 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:56.492 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.492 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.492 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187651 9.f 9:fed2c87a:::benchmark_data_smithi049_138504_object66243:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e347)", 2024-09-21T14:07:56.492 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:42.222057+0000", 2024-09-21T14:07:56.492 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 374.177373182, 2024-09-21T14:07:56.492 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1315866940000001, 2024-09-21T14:07:56.492 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:07:56.492 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:07:56.493 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:07:56.493 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:07:56.493 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:07:56.493 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:07:56.493 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187651 2024-09-21T14:07:56.493 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.493 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:07:56.493 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.493 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:07:56.493 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.222057+0000", 2024-09-21T14:07:56.493 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:07:56.493 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.493 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.493 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:07:56.494 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.222057+0000", 2024-09-21T14:07:56.494 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:07:56.494 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.494 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.494 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:07:56.494 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.222059+0000", 2024-09-21T14:07:56.494 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8029999999999999e-06 2024-09-21T14:07:56.494 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.494 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.494 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:07:56.494 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.222066+0000", 2024-09-21T14:07:56.494 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2670000000000003e-06 2024-09-21T14:07:56.494 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.495 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.495 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:07:56.495 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.222078+0000", 2024-09-21T14:07:56.495 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2899e-05 2024-09-21T14:07:56.495 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.495 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.495 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:07:56.495 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.222086+0000", 2024-09-21T14:07:56.495 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8080000000000005e-06 2024-09-21T14:07:56.495 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.495 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.495 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:07:56.495 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.233188+0000", 2024-09-21T14:07:56.495 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0111019990000001 2024-09-21T14:07:56.496 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.496 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.496 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:07:56.496 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.233317+0000", 2024-09-21T14:07:56.496 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012857999999999999 2024-09-21T14:07:56.496 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.496 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.496 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3", 2024-09-21T14:07:56.496 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.233386+0000", 2024-09-21T14:07:56.496 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8901999999999999e-05 2024-09-21T14:07:56.496 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.496 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.496 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:07:56.496 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.342687+0000", 2024-09-21T14:07:56.496 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.109301519 2024-09-21T14:07:56.497 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.497 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.497 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:56.497 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.342738+0000", 2024-09-21T14:07:56.497 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0756e-05 2024-09-21T14:07:56.497 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.497 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.497 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:56.497 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353571+0000", 2024-09-21T14:07:56.497 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.01083253 2024-09-21T14:07:56.497 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.497 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.497 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:07:56.497 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353614+0000", 2024-09-21T14:07:56.498 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3129e-05 2024-09-21T14:07:56.498 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.498 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.498 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:07:56.498 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353644+0000", 2024-09-21T14:07:56.498 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0502e-05 2024-09-21T14:07:56.498 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:56.498 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:07:56.498 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:56.498 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.498 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.498 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187654 9.5 9:a73ea609:::benchmark_data_smithi049_138504_object66246:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e347)", 2024-09-21T14:07:56.498 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:42.234021+0000", 2024-09-21T14:07:56.499 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 374.16540932700002, 2024-09-21T14:07:56.499 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5379708970000001, 2024-09-21T14:07:56.499 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:07:56.499 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:07:56.499 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:07:56.499 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:07:56.499 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:07:56.499 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:07:56.499 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187654 2024-09-21T14:07:56.499 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.499 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:07:56.499 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.499 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:07:56.499 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.234021+0000", 2024-09-21T14:07:56.499 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:07:56.500 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.500 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.500 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:07:56.500 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.234021+0000", 2024-09-21T14:07:56.500 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:07:56.500 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.500 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.500 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:07:56.500 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.234023+0000", 2024-09-21T14:07:56.500 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5460000000000001e-06 2024-09-21T14:07:56.500 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.500 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.500 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:07:56.501 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.234029+0000", 2024-09-21T14:07:56.501 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8069999999999996e-06 2024-09-21T14:07:56.501 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.501 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.501 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:07:56.501 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.234041+0000", 2024-09-21T14:07:56.501 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2189e-05 2024-09-21T14:07:56.502 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.502 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.502 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:07:56.502 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.234052+0000", 2024-09-21T14:07:56.502 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0713999999999999e-05 2024-09-21T14:07:56.502 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.502 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.503 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:07:56.503 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.743160+0000", 2024-09-21T14:07:56.503 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5091079629999999 2024-09-21T14:07:56.503 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.503 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.503 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:07:56.503 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.746100+0000", 2024-09-21T14:07:56.503 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0029400559999999999 2024-09-21T14:07:56.503 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.503 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.504 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:07:56.504 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.750393+0000", 2024-09-21T14:07:56.504 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0042938280000000004 2024-09-21T14:07:56.504 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.504 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.504 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:56.504 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.757289+0000", 2024-09-21T14:07:56.504 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0068951739999999996 2024-09-21T14:07:56.504 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.505 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.505 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:56.505 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.764759+0000", 2024-09-21T14:07:56.505 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0074705689999999998 2024-09-21T14:07:56.505 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.505 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.505 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:07:56.505 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.765446+0000", 2024-09-21T14:07:56.505 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00068706799999999999 2024-09-21T14:07:56.505 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.505 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.505 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:07:56.505 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.766945+0000", 2024-09-21T14:07:56.505 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0014990509999999999 2024-09-21T14:07:56.505 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.506 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.506 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:07:56.506 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.771992+0000", 2024-09-21T14:07:56.506 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0050469319999999996 2024-09-21T14:07:56.506 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:56.506 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:07:56.506 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:56.506 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.506 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.506 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187662 9.f 9:f2269e64:::benchmark_data_smithi049_138504_object66254:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e347)", 2024-09-21T14:07:56.506 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:42.244949+0000", 2024-09-21T14:07:56.507 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 374.15448153800003, 2024-09-21T14:07:56.507 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1087973170000001, 2024-09-21T14:07:56.507 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:07:56.507 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:07:56.507 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:07:56.507 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:07:56.507 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:07:56.507 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:07:56.507 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187662 2024-09-21T14:07:56.507 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.507 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:07:56.507 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.507 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:07:56.507 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.244949+0000", 2024-09-21T14:07:56.508 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:07:56.508 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.508 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.508 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:07:56.508 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.244949+0000", 2024-09-21T14:07:56.508 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:07:56.508 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.508 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.508 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:07:56.508 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.244951+0000", 2024-09-21T14:07:56.508 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.962e-06 2024-09-21T14:07:56.508 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.508 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.508 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:07:56.508 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.244957+0000", 2024-09-21T14:07:56.508 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8170000000000004e-06 2024-09-21T14:07:56.509 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.509 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.509 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:07:56.509 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.244972+0000", 2024-09-21T14:07:56.509 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5204999999999999e-05 2024-09-21T14:07:56.509 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.509 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.509 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:07:56.509 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.244981+0000", 2024-09-21T14:07:56.509 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5990000000000002e-06 2024-09-21T14:07:56.509 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.509 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.509 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:07:56.509 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.234563+0000", 2024-09-21T14:07:56.509 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.989582548 2024-09-21T14:07:56.510 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.510 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.510 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:07:56.510 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.234696+0000", 2024-09-21T14:07:56.510 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013280200000000001 2024-09-21T14:07:56.510 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.510 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.510 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3", 2024-09-21T14:07:56.510 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.234783+0000", 2024-09-21T14:07:56.510 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.6990999999999994e-05 2024-09-21T14:07:56.510 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.510 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.510 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:07:56.510 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.342749+0000", 2024-09-21T14:07:56.511 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1079658480000001 2024-09-21T14:07:56.511 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.511 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.511 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:56.511 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353673+0000", 2024-09-21T14:07:56.511 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.010924473000000001 2024-09-21T14:07:56.511 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.511 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.511 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:56.511 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353696+0000", 2024-09-21T14:07:56.511 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2792000000000001e-05 2024-09-21T14:07:56.511 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.511 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.511 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:07:56.511 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353726+0000", 2024-09-21T14:07:56.511 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9509000000000002e-05 2024-09-21T14:07:56.512 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.512 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.512 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:07:56.512 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353746+0000", 2024-09-21T14:07:56.512 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0771000000000002e-05 2024-09-21T14:07:56.512 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:56.512 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:07:56.512 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:56.512 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.512 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.512 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187665 9.f 9:fde6db3c:::benchmark_data_smithi049_138504_object66257:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e347)", 2024-09-21T14:07:56.512 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:42.254849+0000", 2024-09-21T14:07:56.512 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 374.14458167599997, 2024-09-21T14:07:56.512 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.186045939, 2024-09-21T14:07:56.513 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:07:56.513 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:07:56.513 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:07:56.513 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:07:56.513 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:07:56.513 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:07:56.513 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187665 2024-09-21T14:07:56.513 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.513 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:07:56.513 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.513 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:07:56.513 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.254849+0000", 2024-09-21T14:07:56.513 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:07:56.513 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.514 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.514 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:07:56.514 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.254849+0000", 2024-09-21T14:07:56.514 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:07:56.514 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.514 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.514 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:07:56.514 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.254851+0000", 2024-09-21T14:07:56.514 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9209999999999999e-06 2024-09-21T14:07:56.514 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.514 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.514 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:07:56.514 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.254857+0000", 2024-09-21T14:07:56.514 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3659999999999997e-06 2024-09-21T14:07:56.515 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.515 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.515 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:07:56.515 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.254870+0000", 2024-09-21T14:07:56.515 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2892e-05 2024-09-21T14:07:56.515 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.515 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.515 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:07:56.515 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.254882+0000", 2024-09-21T14:07:56.515 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1647999999999999e-05 2024-09-21T14:07:56.515 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.515 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.515 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:07:56.515 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.415623+0000", 2024-09-21T14:07:56.516 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.160741582 2024-09-21T14:07:56.516 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.516 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.516 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:07:56.516 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.415771+0000", 2024-09-21T14:07:56.516 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014724000000000001 2024-09-21T14:07:56.516 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.516 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.516 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3", 2024-09-21T14:07:56.516 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.415898+0000", 2024-09-21T14:07:56.516 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012708399999999999 2024-09-21T14:07:56.516 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.516 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.516 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:56.516 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.427284+0000", 2024-09-21T14:07:56.517 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.011386624 2024-09-21T14:07:56.517 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.517 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.517 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:56.517 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.427305+0000", 2024-09-21T14:07:56.517 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0956000000000001e-05 2024-09-21T14:07:56.517 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.517 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.517 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:07:56.517 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.440820+0000", 2024-09-21T14:07:56.517 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.013514511999999999 2024-09-21T14:07:56.517 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.517 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.518 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:07:56.518 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.440860+0000", 2024-09-21T14:07:56.518 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0448000000000003e-05 2024-09-21T14:07:56.518 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.518 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.518 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:07:56.518 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.440895+0000", 2024-09-21T14:07:56.518 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4666000000000003e-05 2024-09-21T14:07:56.518 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:56.518 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:07:56.518 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:56.518 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.518 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.518 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25421.0:187664 9.d e347/345 9:bfa1ff03:::benchmark_data_smithi049_138504_object66256:head v 347'11740, mlcod=346'11738)", 2024-09-21T14:07:56.519 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:42.255288+0000", 2024-09-21T14:07:56.519 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 374.14414279699997, 2024-09-21T14:07:56.519 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2055402320000002, 2024-09-21T14:07:56.519 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:07:56.519 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:07:56.519 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:07:56.519 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:07:56.519 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.519 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:07:56.519 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.255288+0000", 2024-09-21T14:07:56.519 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:07:56.519 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.519 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.519 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:07:56.519 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.255288+0000", 2024-09-21T14:07:56.519 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:07:56.520 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.520 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.520 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:07:56.520 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.255289+0000", 2024-09-21T14:07:56.520 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2200000000000003e-07 2024-09-21T14:07:56.520 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.520 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.520 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:07:56.520 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.255296+0000", 2024-09-21T14:07:56.520 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3440000000000003e-06 2024-09-21T14:07:56.520 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.520 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.520 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:07:56.520 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.255312+0000", 2024-09-21T14:07:56.520 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5968999999999999e-05 2024-09-21T14:07:56.520 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.521 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.521 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:07:56.521 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.255319+0000", 2024-09-21T14:07:56.521 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9199999999999998e-06 2024-09-21T14:07:56.521 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.521 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.521 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:07:56.521 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.459355+0000", 2024-09-21T14:07:56.521 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2040357249999998 2024-09-21T14:07:56.521 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.521 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.521 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:07:56.521 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.459378+0000", 2024-09-21T14:07:56.521 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3757e-05 2024-09-21T14:07:56.521 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.521 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.521 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:07:56.522 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.460796+0000", 2024-09-21T14:07:56.522 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001417721 2024-09-21T14:07:56.522 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.522 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.522 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:07:56.522 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.460828+0000", 2024-09-21T14:07:56.522 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1974000000000003e-05 2024-09-21T14:07:56.522 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:56.522 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:07:56.522 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:56.522 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.522 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.523 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24365.0:2978 5.15 5:a93a5511:::notify.2:head [watch ping cookie 94025139204608 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:07:56.523 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:43.276775+0000", 2024-09-21T14:07:56.523 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 373.122655275, 2024-09-21T14:07:56.523 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4885784160000002, 2024-09-21T14:07:56.523 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:07:56.523 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:07:56.523 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T14:07:56.523 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:07:56.523 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24365", 2024-09-21T14:07:56.523 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3404764347", 2024-09-21T14:07:56.523 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 2978 2024-09-21T14:07:56.523 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.523 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:07:56.523 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.523 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:07:56.523 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:43.276775+0000", 2024-09-21T14:07:56.524 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:07:56.524 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.524 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.524 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:07:56.524 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:43.276775+0000", 2024-09-21T14:07:56.524 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:07:56.524 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.524 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.524 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:07:56.524 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:43.276777+0000", 2024-09-21T14:07:56.524 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9630000000000001e-06 2024-09-21T14:07:56.524 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.524 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.524 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:07:56.524 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:43.276784+0000", 2024-09-21T14:07:56.525 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1219999999999997e-06 2024-09-21T14:07:56.525 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.525 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.525 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:07:56.525 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:43.276802+0000", 2024-09-21T14:07:56.525 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7096999999999999e-05 2024-09-21T14:07:56.525 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.525 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.525 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:07:56.526 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:43.276811+0000", 2024-09-21T14:07:56.526 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7780000000000002e-06 2024-09-21T14:07:56.526 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:56.527 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:56.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:07:56.527 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.527 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:07:56.527 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.765168+0000", 2024-09-21T14:07:56.528 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4883561849999998 2024-09-21T14:07:56.528 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.528 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.528 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:07:56.528 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.765252+0000", 2024-09-21T14:07:56.528 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4201000000000002e-05 2024-09-21T14:07:56.528 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.528 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.528 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:07:56.528 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.765354+0000", 2024-09-21T14:07:56.528 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010207 2024-09-21T14:07:56.528 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:56.528 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:07:56.528 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:56.528 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.528 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.529 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.25421.0:187651 9.f e349/345 ondisk, result = 0)", 2024-09-21T14:07:56.529 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:44.238795+0000", 2024-09-21T14:07:56.529 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 372.16063598800002, 2024-09-21T14:07:56.529 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.115079242, 2024-09-21T14:07:56.529 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:07:56.529 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:07:56.529 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T14:07:56.529 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:07:56.529 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.529 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:07:56.529 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238795+0000", 2024-09-21T14:07:56.529 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:07:56.529 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.529 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.529 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:07:56.529 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238795+0000", 2024-09-21T14:07:56.530 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:07:56.530 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.530 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.530 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:07:56.530 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238795+0000", 2024-09-21T14:07:56.530 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9800000000000003e-07 2024-09-21T14:07:56.530 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.530 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.530 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:07:56.530 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238799+0000", 2024-09-21T14:07:56.530 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9920000000000001e-06 2024-09-21T14:07:56.530 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.531 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.531 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:07:56.531 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238809+0000", 2024-09-21T14:07:56.531 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6120000000000005e-06 2024-09-21T14:07:56.531 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.531 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.531 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:07:56.531 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238816+0000", 2024-09-21T14:07:56.531 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6229999999999999e-06 2024-09-21T14:07:56.531 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.531 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.531 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:07:56.531 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.342725+0000", 2024-09-21T14:07:56.531 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1039093099999999 2024-09-21T14:07:56.532 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.532 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.532 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:07:56.532 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.342733+0000", 2024-09-21T14:07:56.532 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9189999999999999e-06 2024-09-21T14:07:56.532 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.532 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.532 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:07:56.532 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353874+0000", 2024-09-21T14:07:56.532 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.011141088 2024-09-21T14:07:56.532 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:56.532 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:07:56.532 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:56.532 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.532 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.532 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.25421.0:187651 9.f e349/345 ondisk, result = 0)", 2024-09-21T14:07:56.533 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:44.238813+0000", 2024-09-21T14:07:56.533 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 372.16061764599999, 2024-09-21T14:07:56.533 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1148360479999999, 2024-09-21T14:07:56.533 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:07:56.533 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:07:56.533 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T14:07:56.533 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:07:56.533 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.533 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:07:56.533 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238813+0000", 2024-09-21T14:07:56.533 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:07:56.533 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.533 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.533 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:07:56.533 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238813+0000", 2024-09-21T14:07:56.534 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:07:56.534 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.534 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.534 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:07:56.534 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238814+0000", 2024-09-21T14:07:56.534 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4700000000000001e-07 2024-09-21T14:07:56.534 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.534 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.534 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:07:56.534 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238817+0000", 2024-09-21T14:07:56.534 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3469999999999999e-06 2024-09-21T14:07:56.534 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.534 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.534 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:07:56.534 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238827+0000", 2024-09-21T14:07:56.535 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0011e-05 2024-09-21T14:07:56.535 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.535 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.535 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:07:56.535 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238832+0000", 2024-09-21T14:07:56.535 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1139999999999997e-06 2024-09-21T14:07:56.535 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.535 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.535 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:07:56.535 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353559+0000", 2024-09-21T14:07:56.535 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1147265909999999 2024-09-21T14:07:56.535 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.535 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.535 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:07:56.536 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353566+0000", 2024-09-21T14:07:56.536 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8669999999999994e-06 2024-09-21T14:07:56.536 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.536 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.536 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:07:56.536 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353649+0000", 2024-09-21T14:07:56.536 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2571000000000006e-05 2024-09-21T14:07:56.536 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:56.536 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:07:56.536 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:56.536 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.536 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.536 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187668 9.5 9:acffd95c:::benchmark_data_smithi049_138504_object66260:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e349)", 2024-09-21T14:07:56.536 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:44.468347+0000", 2024-09-21T14:07:56.536 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 371.93108414900001, 2024-09-21T14:07:56.537 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3419073799999999, 2024-09-21T14:07:56.537 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:07:56.537 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:07:56.537 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:07:56.537 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:07:56.537 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:07:56.537 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:07:56.537 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187668 2024-09-21T14:07:56.537 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.537 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:07:56.537 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.537 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:07:56.537 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.468347+0000", 2024-09-21T14:07:56.537 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:07:56.538 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.538 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.538 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:07:56.538 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.468347+0000", 2024-09-21T14:07:56.538 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:07:56.538 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.538 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.538 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:07:56.538 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.468347+0000", 2024-09-21T14:07:56.538 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2399999999999997e-07 2024-09-21T14:07:56.538 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.538 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.538 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:07:56.539 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.468351+0000", 2024-09-21T14:07:56.539 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6049999999999998e-06 2024-09-21T14:07:56.539 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.539 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.539 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:07:56.539 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.468358+0000", 2024-09-21T14:07:56.539 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4510000000000003e-06 2024-09-21T14:07:56.539 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.539 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.539 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:07:56.539 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.468366+0000", 2024-09-21T14:07:56.539 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6140000000000003e-06 2024-09-21T14:07:56.539 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.539 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.540 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:07:56.540 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.772366+0000", 2024-09-21T14:07:56.540 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3040005530000001 2024-09-21T14:07:56.540 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.540 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.540 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:07:56.540 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.775717+0000", 2024-09-21T14:07:56.540 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0033501080000000001 2024-09-21T14:07:56.540 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.540 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.540 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:07:56.540 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.781090+0000", 2024-09-21T14:07:56.540 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0053731990000000004 2024-09-21T14:07:56.540 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.540 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.540 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:56.541 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.798846+0000", 2024-09-21T14:07:56.541 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.017756042999999999 2024-09-21T14:07:56.541 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.541 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.541 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:07:56.541 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.800017+0000", 2024-09-21T14:07:56.541 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0011714430000000001 2024-09-21T14:07:56.541 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.541 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.541 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:07:56.541 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.800552+0000", 2024-09-21T14:07:56.541 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00053442500000000005 2024-09-21T14:07:56.541 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.541 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.541 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:07:56.541 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.801930+0000", 2024-09-21T14:07:56.542 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0013783109999999999 2024-09-21T14:07:56.542 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:07:56.542 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:07:56.542 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:07:56.542 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.810254+0000", 2024-09-21T14:07:56.542 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.008323904 2024-09-21T14:07:56.542 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:56.542 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:07:56.542 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:56.542 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:07:56.542 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:07:56.542 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T14:07:56.543 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 63 16 96477 96461 95.6741 85.9375 0.00889755 0.0104476 2024-09-21T14:07:56.628 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:07:56.668 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:56.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:56.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:07:56.769 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:07:56.800 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:56.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:56.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:07:56.901 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:07:56.924 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:56.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:56.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:07:57.026 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:57.062 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:57.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:57.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:57.164 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:07:57.198 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:57.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:57.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:07:57.300 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:07:57.339 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:57.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:57.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:07:57.441 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:07:57.461 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:57.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:57.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:07:57.472 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T14:07:57.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:57 smithi049 ceph-mon[40489]: pgmap v971: 168 pgs: 168 active+clean; 5.6 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 46 MiB/s wr, 730 op/s; 26 B/s, 1 objects/s recovering 2024-09-21T14:07:57.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:57 smithi049 ceph-mon[30080]: pgmap v971: 168 pgs: 168 active+clean; 5.6 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 46 MiB/s wr, 730 op/s; 26 B/s, 1 objects/s recovering 2024-09-21T14:07:57.517 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 64 16 97467 97451 95.1457 61.875 0.0152133 0.0105055 2024-09-21T14:07:57.563 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:07:57.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:57.618 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:57.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:07:57.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:57 smithi187 ceph-mon[33032]: pgmap v971: 168 pgs: 168 active+clean; 5.6 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 46 MiB/s wr, 730 op/s; 26 B/s, 1 objects/s recovering 2024-09-21T14:07:57.758 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:07:57.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:57.816 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:57.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:07:57.873 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 4, 6, 2, 3, 7, 5] 2024-09-21T14:07:57.873 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T14:07:57.873 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-21T14:07:57.874 INFO:tasks.thrashosds.thrasher:Setting osd 7 primary_affinity to 0.000000 2024-09-21T14:07:57.874 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 7 0 2024-09-21T14:07:57.881 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:07:57.917 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:07:57.945 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:57.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:57.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:07:57.971 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T14:07:58.048 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:07:58.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:58.118 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:58.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:07:58.219 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:07:58.316 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:58.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:58.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:07:58.418 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:07:58.449 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:58.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:58.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:07:58.517 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 65 16 98482 98466 94.6579 63.4375 0.0190726 0.0105592 2024-09-21T14:07:58.551 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:07:58.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:58.625 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:58.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:07:58.670 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:07:58.725 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:07:58.741 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:07:58.752 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:58.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:58.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:07:58.853 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:07:58.878 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:58.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:58.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:07:58.980 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:07:59.026 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:59.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:59.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:07:59.089 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T14:07:59.129 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:59.185 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:59.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:59.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:59.246 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-09-21T14:07:59.287 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:07:59.313 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:59.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:59.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:07:59.415 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:07:59.430 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:59 smithi049 ceph-mon[40489]: pgmap v972: 168 pgs: 168 active+clean; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.03k op/s; 18 B/s, 1 objects/s recovering 2024-09-21T14:07:59.430 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:07:59 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3990963808' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:07:59.431 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:59 smithi049 ceph-mon[30080]: pgmap v972: 168 pgs: 168 active+clean; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.03k op/s; 18 B/s, 1 objects/s recovering 2024-09-21T14:07:59.431 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:07:59 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3990963808' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:07:59.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:59 smithi187 ceph-mon[33032]: pgmap v972: 168 pgs: 168 active+clean; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.03k op/s; 18 B/s, 1 objects/s recovering 2024-09-21T14:07:59.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:07:59 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3990963808' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:07:59.467 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:59.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:59.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:07:59.518 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 66 16 99626 99610 94.3067 71.5 0.0129439 0.0105991 2024-09-21T14:07:59.569 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:07:59.592 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:07:59.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:59.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:07:59.694 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:07:59.732 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:59.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:59.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:07:59.834 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:07:59.880 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:07:59.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:07:59.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:07:59.981 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:08:00.022 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:00.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:00.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:08:00.124 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:00.150 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:00.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:00.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:00.252 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:00.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:00.299 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:00.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:00.400 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:08:00.439 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:00.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:00.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:08:00.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:00 smithi187 ceph-mon[33032]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:08:00.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:00 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3990963808' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T14:08:00.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:00 smithi187 ceph-mon[33032]: osdmap e489: 8 total, 8 up, 8 in 2024-09-21T14:08:00.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:00 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:08:00.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:00 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:08:00.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:00 smithi187 ceph-mon[33032]: osdmap e490: 8 total, 8 up, 8 in 2024-09-21T14:08:00.444 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:08:00 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:08:00.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:00 smithi049 ceph-mon[40489]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:08:00.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:00 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3990963808' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T14:08:00.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:00 smithi049 ceph-mon[40489]: osdmap e489: 8 total, 8 up, 8 in 2024-09-21T14:08:00.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:00 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:08:00.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:00 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:08:00.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:00 smithi049 ceph-mon[40489]: osdmap e490: 8 total, 8 up, 8 in 2024-09-21T14:08:00.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:00 smithi049 ceph-mon[30080]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:08:00.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:00 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3990963808' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T14:08:00.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:00 smithi049 ceph-mon[30080]: osdmap e489: 8 total, 8 up, 8 in 2024-09-21T14:08:00.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:00 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:08:00.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:00 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:08:00.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:00 smithi049 ceph-mon[30080]: osdmap e490: 8 total, 8 up, 8 in 2024-09-21T14:08:00.518 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 67 16 100809 100793 94.0026 73.9375 0.0140952 0.0106333 2024-09-21T14:08:00.541 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:08:00.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:00.583 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:00.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:08:00.684 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:08:00.712 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:00.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:00.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:08:00.814 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:00.856 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:00.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:00.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:00.958 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:00.989 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:00.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:00.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:01.004 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:08:00 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:08:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:08:01.090 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:01.121 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:01.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:01.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:01.223 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:01.254 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:01.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:01.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:01.356 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:08:01.397 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:01.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:01.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:08:01.499 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:01.518 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 68 16 102092 102076 93.7992 80.1875 0.0123815 0.010656 2024-09-21T14:08:01.540 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:01.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:01.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:01.642 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:08:01.677 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:01.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:01.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:08:01.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:01 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:08:01.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:01 smithi187 ceph-mon[33032]: pgmap v974: 168 pgs: 168 active+clean; 6.0 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-09-21T14:08:01.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:01 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3326336954' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:08:01.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:01 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:08:01.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:01 smithi049 ceph-mon[40489]: pgmap v974: 168 pgs: 168 active+clean; 6.0 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-09-21T14:08:01.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:01 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3326336954' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:08:01.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:01 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:08:01.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:01 smithi049 ceph-mon[30080]: pgmap v974: 168 pgs: 168 active+clean; 6.0 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-09-21T14:08:01.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:01 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3326336954' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:08:01.780 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:08:01.819 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:01.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:01.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:08:01.921 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:08:01.962 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:01.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:01.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:08:02.064 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:08:02.100 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:02.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:02.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:08:02.202 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:08:02.241 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:02.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:02.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:08:02.343 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:02.372 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:02.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:02.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:02.473 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:02.516 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:02.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:02.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:02.518 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 69 16 103381 103365 93.6073 80.5625 0.0214018 0.0106779 2024-09-21T14:08:02.618 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:02.643 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:02.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:02.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:02.688 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:08:02.689 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:08:02.689 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:08:02.689 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:08:02.689 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:08:02.689 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:08:02.689 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:08:02.689 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:08:02.690 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:08:02.690 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:08:02.690 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:08:02.690 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:08:02.690 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:08:02.690 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:08:02.690 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:08:02.690 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:08:02.744 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:08:02.784 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:02.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:02.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:08:02.886 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:08:02.922 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:02.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:02.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:08:03.024 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:08:03.051 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:03.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:03.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:08:03.153 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:08:03.182 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:03.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:03.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:08:03.284 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:03.294 INFO:teuthology.orchestra.run.smithi049.stderr:set osd.7 primary-affinity to 0 (802) 2024-09-21T14:08:03.316 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:03.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:03.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:03.418 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:08:03.441 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:03.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:03.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:08:03.518 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 70 16 104364 104348 93.1477 61.4375 0.00351042 0.0107011 2024-09-21T14:08:03.543 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:03.566 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:03.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:03.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:03.669 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:08:03.673 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:03 smithi049 ceph-mon[30080]: 5.6 scrub starts 2024-09-21T14:08:03.673 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:03 smithi049 ceph-mon[30080]: 5.6 scrub ok 2024-09-21T14:08:03.673 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:03 smithi049 ceph-mon[30080]: pgmap v976: 168 pgs: 168 active+clean; 6.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.4 KiB/s rd, 83 MiB/s wr, 1.33k op/s 2024-09-21T14:08:03.673 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:03 smithi049 ceph-mon[30080]: 5.15 scrub starts 2024-09-21T14:08:03.673 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:03 smithi049 ceph-mon[30080]: 5.15 scrub ok 2024-09-21T14:08:03.673 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:03 smithi049 ceph-mon[30080]: 11.1 scrub starts 2024-09-21T14:08:03.673 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:03 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1830641631' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.0} : dispatch 2024-09-21T14:08:03.674 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:03 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.0} : dispatch 2024-09-21T14:08:03.674 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:03 smithi049 ceph-mon[40489]: 5.6 scrub starts 2024-09-21T14:08:03.674 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:03 smithi049 ceph-mon[40489]: 5.6 scrub ok 2024-09-21T14:08:03.674 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:03 smithi049 ceph-mon[40489]: pgmap v976: 168 pgs: 168 active+clean; 6.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.4 KiB/s rd, 83 MiB/s wr, 1.33k op/s 2024-09-21T14:08:03.674 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:03 smithi049 ceph-mon[40489]: 5.15 scrub starts 2024-09-21T14:08:03.674 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:03 smithi049 ceph-mon[40489]: 5.15 scrub ok 2024-09-21T14:08:03.674 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:03 smithi049 ceph-mon[40489]: 11.1 scrub starts 2024-09-21T14:08:03.675 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:03 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1830641631' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.0} : dispatch 2024-09-21T14:08:03.675 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:03 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.0} : dispatch 2024-09-21T14:08:03.680 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:08:03.690 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:03.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:03.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:08:03.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:03 smithi187 ceph-mon[33032]: 5.6 scrub starts 2024-09-21T14:08:03.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:03 smithi187 ceph-mon[33032]: 5.6 scrub ok 2024-09-21T14:08:03.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:03 smithi187 ceph-mon[33032]: pgmap v976: 168 pgs: 168 active+clean; 6.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.4 KiB/s rd, 83 MiB/s wr, 1.33k op/s 2024-09-21T14:08:03.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:03 smithi187 ceph-mon[33032]: 5.15 scrub starts 2024-09-21T14:08:03.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:03 smithi187 ceph-mon[33032]: 5.15 scrub ok 2024-09-21T14:08:03.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:03 smithi187 ceph-mon[33032]: 11.1 scrub starts 2024-09-21T14:08:03.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:03 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1830641631' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.0} : dispatch 2024-09-21T14:08:03.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:03 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.0} : dispatch 2024-09-21T14:08:03.792 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:03.815 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:03.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:03.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:03.917 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:08:03.939 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:03.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:03.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:08:04.041 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:04.062 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:04.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:04.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:04.164 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:04.185 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:04.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:04.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:04.287 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:08:04.320 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:04.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:04.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:08:04.359 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T14:08:04.403 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:04 smithi049 ceph-mon[40489]: 4.1a scrub starts 2024-09-21T14:08:04.404 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:04 smithi049 ceph-mon[40489]: 4.1a scrub ok 2024-09-21T14:08:04.404 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:04 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.0}]': finished 2024-09-21T14:08:04.404 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:04 smithi049 ceph-mon[40489]: osdmap e491: 8 total, 8 up, 8 in 2024-09-21T14:08:04.404 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:04 smithi049 ceph-mon[40489]: 11.1 scrub ok 2024-09-21T14:08:04.404 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:04 smithi049 ceph-mon[40489]: osdmap e492: 8 total, 8 up, 8 in 2024-09-21T14:08:04.404 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:04 smithi049 ceph-mon[30080]: 4.1a scrub starts 2024-09-21T14:08:04.404 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:04 smithi049 ceph-mon[30080]: 4.1a scrub ok 2024-09-21T14:08:04.404 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:04 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.0}]': finished 2024-09-21T14:08:04.404 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:04 smithi049 ceph-mon[30080]: osdmap e491: 8 total, 8 up, 8 in 2024-09-21T14:08:04.404 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:04 smithi049 ceph-mon[30080]: 11.1 scrub ok 2024-09-21T14:08:04.404 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:04 smithi049 ceph-mon[30080]: osdmap e492: 8 total, 8 up, 8 in 2024-09-21T14:08:04.422 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:04.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:04.474 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:04.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:04.518 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 71 16 104596 104580 92.0401 14.5 0.00978376 0.01086 2024-09-21T14:08:04.575 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:08:04.602 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:04.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:04.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:08:04.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:04 smithi187 ceph-mon[33032]: 4.1a scrub starts 2024-09-21T14:08:04.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:04 smithi187 ceph-mon[33032]: 4.1a scrub ok 2024-09-21T14:08:04.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:04 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.0}]': finished 2024-09-21T14:08:04.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:04 smithi187 ceph-mon[33032]: osdmap e491: 8 total, 8 up, 8 in 2024-09-21T14:08:04.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:04 smithi187 ceph-mon[33032]: 11.1 scrub ok 2024-09-21T14:08:04.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:04 smithi187 ceph-mon[33032]: osdmap e492: 8 total, 8 up, 8 in 2024-09-21T14:08:04.704 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:08:04.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:04.777 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:04.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:08:04.865 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T14:08:04.865 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:08:04.877 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:04.907 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:04.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:04.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:05.008 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:08:05.052 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:05.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:05.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:08:05.138 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:08:05.155 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:05.203 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:05.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:05.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:05.304 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:08:05.346 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:05.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:05.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:08:05.447 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:08:05.478 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:05.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:05.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:08:05.518 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 72 16 105732 105716 91.7479 71 0.0133887 0.0108949 2024-09-21T14:08:05.580 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:05.614 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:05.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:05.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:05.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:05 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:08:05.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:05 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:08:05.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:05 smithi187 ceph-mon[33032]: 6.11 scrub starts 2024-09-21T14:08:05.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:05 smithi187 ceph-mon[33032]: 6.11 scrub ok 2024-09-21T14:08:05.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:05 smithi187 ceph-mon[33032]: 11.6 scrub starts 2024-09-21T14:08:05.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:05 smithi187 ceph-mon[33032]: pgmap v978: 168 pgs: 14 peering, 154 active+clean; 6.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 2.5 KiB/s rd, 76 MiB/s wr, 1.22k op/s; 8.7 KiB/s, 0 objects/s recovering 2024-09-21T14:08:05.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:05 smithi187 ceph-mon[33032]: Health check failed: Reduced data availability: 8 pgs inactive, 13 pgs peering (PG_AVAILABILITY) 2024-09-21T14:08:05.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:05 smithi187 ceph-mon[33032]: 6.1c scrub starts 2024-09-21T14:08:05.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:05 smithi187 ceph-mon[33032]: 6.1c scrub ok 2024-09-21T14:08:05.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:05 smithi187 ceph-mon[33032]: 1.2 scrub starts 2024-09-21T14:08:05.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:05 smithi187 ceph-mon[33032]: 1.2 scrub ok 2024-09-21T14:08:05.712 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T14:08:05.716 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:08:05.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:05 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:08:05.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:05 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:08:05.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:05 smithi049 ceph-mon[30080]: 6.11 scrub starts 2024-09-21T14:08:05.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:05 smithi049 ceph-mon[30080]: 6.11 scrub ok 2024-09-21T14:08:05.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:05 smithi049 ceph-mon[30080]: 11.6 scrub starts 2024-09-21T14:08:05.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:05 smithi049 ceph-mon[30080]: pgmap v978: 168 pgs: 14 peering, 154 active+clean; 6.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 2.5 KiB/s rd, 76 MiB/s wr, 1.22k op/s; 8.7 KiB/s, 0 objects/s recovering 2024-09-21T14:08:05.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:05 smithi049 ceph-mon[30080]: Health check failed: Reduced data availability: 8 pgs inactive, 13 pgs peering (PG_AVAILABILITY) 2024-09-21T14:08:05.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:05 smithi049 ceph-mon[30080]: 6.1c scrub starts 2024-09-21T14:08:05.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:05 smithi049 ceph-mon[30080]: 6.1c scrub ok 2024-09-21T14:08:05.741 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:05 smithi049 ceph-mon[30080]: 1.2 scrub starts 2024-09-21T14:08:05.742 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:05 smithi049 ceph-mon[30080]: 1.2 scrub ok 2024-09-21T14:08:05.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:05.742 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:05.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:08:05.744 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:05 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:08:05.744 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:05 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:08:05.744 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:05 smithi049 ceph-mon[40489]: 6.11 scrub starts 2024-09-21T14:08:05.744 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:05 smithi049 ceph-mon[40489]: 6.11 scrub ok 2024-09-21T14:08:05.744 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:05 smithi049 ceph-mon[40489]: 11.6 scrub starts 2024-09-21T14:08:05.744 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:05 smithi049 ceph-mon[40489]: pgmap v978: 168 pgs: 14 peering, 154 active+clean; 6.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 2.5 KiB/s rd, 76 MiB/s wr, 1.22k op/s; 8.7 KiB/s, 0 objects/s recovering 2024-09-21T14:08:05.744 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:05 smithi049 ceph-mon[40489]: Health check failed: Reduced data availability: 8 pgs inactive, 13 pgs peering (PG_AVAILABILITY) 2024-09-21T14:08:05.744 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:05 smithi049 ceph-mon[40489]: 6.1c scrub starts 2024-09-21T14:08:05.745 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:05 smithi049 ceph-mon[40489]: 6.1c scrub ok 2024-09-21T14:08:05.745 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:05 smithi049 ceph-mon[40489]: 1.2 scrub starts 2024-09-21T14:08:05.745 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:05 smithi049 ceph-mon[40489]: 1.2 scrub ok 2024-09-21T14:08:05.803 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T14:08:05.843 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:05.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:05.893 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:05.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:05.994 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:06.020 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:06.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:06.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:06.121 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:06.150 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:06.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:06.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:06.252 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:06.301 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:06.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:06.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:06.403 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:08:06.452 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:06.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:06.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:08:06.518 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 73 16 106849 106833 91.4473 69.8125 0.0122412 0.0109307 2024-09-21T14:08:06.554 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:08:06.586 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:06.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:06.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:08:06.688 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:08:06.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:06 smithi187 ceph-mon[33032]: 1.d scrub starts 2024-09-21T14:08:06.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:06 smithi187 ceph-mon[33032]: 1.d scrub ok 2024-09-21T14:08:06.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:06 smithi187 ceph-mon[33032]: 3.17 scrub starts 2024-09-21T14:08:06.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:06 smithi187 ceph-mon[33032]: 3.17 scrub ok 2024-09-21T14:08:06.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:06 smithi187 ceph-mon[33032]: 1.0 scrub starts 2024-09-21T14:08:06.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:06 smithi187 ceph-mon[33032]: 1.0 scrub ok 2024-09-21T14:08:06.709 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:06.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:06.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:08:06.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:06 smithi049 ceph-mon[40489]: 1.d scrub starts 2024-09-21T14:08:06.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:06 smithi049 ceph-mon[40489]: 1.d scrub ok 2024-09-21T14:08:06.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:06 smithi049 ceph-mon[40489]: 3.17 scrub starts 2024-09-21T14:08:06.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:06 smithi049 ceph-mon[40489]: 3.17 scrub ok 2024-09-21T14:08:06.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:06 smithi049 ceph-mon[40489]: 1.0 scrub starts 2024-09-21T14:08:06.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:06 smithi049 ceph-mon[40489]: 1.0 scrub ok 2024-09-21T14:08:06.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:06 smithi049 ceph-mon[30080]: 1.d scrub starts 2024-09-21T14:08:06.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:06 smithi049 ceph-mon[30080]: 1.d scrub ok 2024-09-21T14:08:06.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:06 smithi049 ceph-mon[30080]: 3.17 scrub starts 2024-09-21T14:08:06.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:06 smithi049 ceph-mon[30080]: 3.17 scrub ok 2024-09-21T14:08:06.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:06 smithi049 ceph-mon[30080]: 1.0 scrub starts 2024-09-21T14:08:06.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:06 smithi049 ceph-mon[30080]: 1.0 scrub ok 2024-09-21T14:08:06.810 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:06.845 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:06.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:06.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:06.947 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:06.973 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:06.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:06.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:07.075 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:08:07.107 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:07.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:07.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:08:07.209 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:07.249 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:07.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:07.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:07.351 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:08:07.380 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:07.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:07.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:08:07.481 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:08:07.519 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 74 16 108013 107997 91.1946 72.75 0.00984617 0.0109608 2024-09-21T14:08:07.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:07.520 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:07.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:08:07.620 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:08:07.650 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:07.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:07.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:08:07.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:07 smithi187 ceph-mon[33032]: 11.6 scrub ok 2024-09-21T14:08:07.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:07 smithi187 ceph-mon[33032]: pgmap v980: 168 pgs: 14 peering, 154 active+clean; 6.4 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s; 8.7 KiB/s, 0 objects/s recovering 2024-09-21T14:08:07.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:07 smithi187 ceph-mon[33032]: 3.13 scrub starts 2024-09-21T14:08:07.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:07 smithi187 ceph-mon[33032]: 3.13 scrub ok 2024-09-21T14:08:07.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:07 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:08:07.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:07 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:08:07.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:07 smithi187 ceph-mon[33032]: 3.d scrub starts 2024-09-21T14:08:07.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:07 smithi187 ceph-mon[33032]: 3.d scrub ok 2024-09-21T14:08:07.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:07 smithi187 ceph-mon[33032]: 2.0 scrub starts 2024-09-21T14:08:07.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:07 smithi187 ceph-mon[33032]: 2.0 scrub ok 2024-09-21T14:08:07.751 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:07.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:07 smithi049 ceph-mon[40489]: 11.6 scrub ok 2024-09-21T14:08:07.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:07 smithi049 ceph-mon[40489]: pgmap v980: 168 pgs: 14 peering, 154 active+clean; 6.4 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s; 8.7 KiB/s, 0 objects/s recovering 2024-09-21T14:08:07.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:07 smithi049 ceph-mon[40489]: 3.13 scrub starts 2024-09-21T14:08:07.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:07 smithi049 ceph-mon[40489]: 3.13 scrub ok 2024-09-21T14:08:07.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:07 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:08:07.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:07 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:08:07.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:07 smithi049 ceph-mon[40489]: 3.d scrub starts 2024-09-21T14:08:07.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:07 smithi049 ceph-mon[40489]: 3.d scrub ok 2024-09-21T14:08:07.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:07 smithi049 ceph-mon[40489]: 2.0 scrub starts 2024-09-21T14:08:07.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:07 smithi049 ceph-mon[40489]: 2.0 scrub ok 2024-09-21T14:08:07.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:07 smithi049 ceph-mon[30080]: 11.6 scrub ok 2024-09-21T14:08:07.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:07 smithi049 ceph-mon[30080]: pgmap v980: 168 pgs: 14 peering, 154 active+clean; 6.4 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s; 8.7 KiB/s, 0 objects/s recovering 2024-09-21T14:08:07.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:07 smithi049 ceph-mon[30080]: 3.13 scrub starts 2024-09-21T14:08:07.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:07 smithi049 ceph-mon[30080]: 3.13 scrub ok 2024-09-21T14:08:07.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:07 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:08:07.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:07 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:08:07.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:07 smithi049 ceph-mon[30080]: 3.d scrub starts 2024-09-21T14:08:07.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:07 smithi049 ceph-mon[30080]: 3.d scrub ok 2024-09-21T14:08:07.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:07 smithi049 ceph-mon[30080]: 2.0 scrub starts 2024-09-21T14:08:07.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:07 smithi049 ceph-mon[30080]: 2.0 scrub ok 2024-09-21T14:08:07.782 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:07.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:07.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:07.883 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:08:07.922 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:07.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:07.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:08:08.024 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:08:08.067 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:08.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:08.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:08:08.168 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:08:08.194 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:08.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:08.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:08:08.295 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:08.327 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:08.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:08.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:08.429 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:08:08.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:08.453 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:08.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:08:08.519 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 75 16 109206 109190 90.9728 74.5625 0.00804898 0.0109873 2024-09-21T14:08:08.554 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:08:08.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:08.593 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:08.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:08:08.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:08 smithi187 ceph-mon[33032]: 5.4 scrub starts 2024-09-21T14:08:08.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:08 smithi187 ceph-mon[33032]: 5.4 scrub ok 2024-09-21T14:08:08.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:08 smithi187 ceph-mon[33032]: osdmap e493: 8 total, 8 up, 8 in 2024-09-21T14:08:08.694 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:08:08.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:08.718 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:08.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:08:08.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:08 smithi049 ceph-mon[40489]: 5.4 scrub starts 2024-09-21T14:08:08.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:08 smithi049 ceph-mon[40489]: 5.4 scrub ok 2024-09-21T14:08:08.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:08 smithi049 ceph-mon[40489]: osdmap e493: 8 total, 8 up, 8 in 2024-09-21T14:08:08.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:08 smithi049 ceph-mon[30080]: 5.4 scrub starts 2024-09-21T14:08:08.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:08 smithi049 ceph-mon[30080]: 5.4 scrub ok 2024-09-21T14:08:08.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:08 smithi049 ceph-mon[30080]: osdmap e493: 8 total, 8 up, 8 in 2024-09-21T14:08:08.819 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:08.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:08.847 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:08.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:08.948 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:08.982 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:08.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:08.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:09.083 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:09.107 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:09.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:09.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:09.172 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 4, 6, 2, 3, 7, 5] 2024-09-21T14:08:09.173 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T14:08:09.173 INFO:tasks.thrashosds.thrasher:Reweighting osd 3 to 0.5022322683272371 2024-09-21T14:08:09.173 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd reweight 3 0.5022322683272371 2024-09-21T14:08:09.208 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:08:09.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:09.266 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:09.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:08:09.366 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:08:09.394 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:09.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:09.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:08:09.498 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:08:09.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:09 smithi049 ceph-mon[30080]: pgmap v982: 168 pgs: 14 peering, 154 active+clean; 6.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 50 MiB/s wr, 798 op/s; 26 KiB/s, 0 objects/s recovering 2024-09-21T14:08:09.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:09 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:08:09.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:09 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:08:09.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:09 smithi049 ceph-mon[30080]: 5.19 scrub starts 2024-09-21T14:08:09.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:09 smithi049 ceph-mon[30080]: 5.19 scrub ok 2024-09-21T14:08:09.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:09 smithi049 ceph-mon[40489]: pgmap v982: 168 pgs: 14 peering, 154 active+clean; 6.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 50 MiB/s wr, 798 op/s; 26 KiB/s, 0 objects/s recovering 2024-09-21T14:08:09.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:09 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:08:09.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:09 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:08:09.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:09 smithi049 ceph-mon[40489]: 5.19 scrub starts 2024-09-21T14:08:09.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:09 smithi049 ceph-mon[40489]: 5.19 scrub ok 2024-09-21T14:08:09.519 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 76 16 110350 110334 90.7165 71.5 0.0220039 0.011018 2024-09-21T14:08:09.524 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:09.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:09.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:08:09.626 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:08:09.652 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:09.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:09.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:08:09.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:09 smithi187 ceph-mon[33032]: pgmap v982: 168 pgs: 14 peering, 154 active+clean; 6.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 50 MiB/s wr, 798 op/s; 26 KiB/s, 0 objects/s recovering 2024-09-21T14:08:09.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:09 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:08:09.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:09 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:08:09.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:09 smithi187 ceph-mon[33032]: 5.19 scrub starts 2024-09-21T14:08:09.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:09 smithi187 ceph-mon[33032]: 5.19 scrub ok 2024-09-21T14:08:09.755 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:09.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:09.802 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:09.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:09.903 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:08:09.942 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:09.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:09.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:08:10.044 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:10.072 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:10.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:10.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:10.174 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:10.233 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:08:10.243 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:08:10.254 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:10.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:10.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:10.298 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:08:10.357 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:10.386 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:10.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:10.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:10.389 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-09-21T14:08:10.444 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:08:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:08:10.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:10 smithi187 ceph-mon[33032]: osdmap e494: 8 total, 8 up, 8 in 2024-09-21T14:08:10.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:10 smithi187 ceph-mon[33032]: 3.1 scrub starts 2024-09-21T14:08:10.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:10 smithi187 ceph-mon[33032]: 3.1 scrub ok 2024-09-21T14:08:10.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:10 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/383041312' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:08:10.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:10 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:08:10.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:10 smithi187 ceph-mon[33032]: 4.4 scrub starts 2024-09-21T14:08:10.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:10 smithi187 ceph-mon[33032]: 4.4 scrub ok 2024-09-21T14:08:10.488 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:10.519 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 77 16 111477 111461 90.4529 70.4375 0.00662267 0.0110506 2024-09-21T14:08:10.530 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:10.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:10.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:10.631 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:08:10.671 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:10.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:10.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:08:10.754 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:08:10 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:08:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:08:10.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:10 smithi049 ceph-mon[40489]: osdmap e494: 8 total, 8 up, 8 in 2024-09-21T14:08:10.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:10 smithi049 ceph-mon[40489]: 3.1 scrub starts 2024-09-21T14:08:10.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:10 smithi049 ceph-mon[40489]: 3.1 scrub ok 2024-09-21T14:08:10.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:10 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/383041312' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:08:10.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:10 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:08:10.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:10 smithi049 ceph-mon[40489]: 4.4 scrub starts 2024-09-21T14:08:10.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:10 smithi049 ceph-mon[40489]: 4.4 scrub ok 2024-09-21T14:08:10.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:10 smithi049 ceph-mon[30080]: osdmap e494: 8 total, 8 up, 8 in 2024-09-21T14:08:10.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:10 smithi049 ceph-mon[30080]: 3.1 scrub starts 2024-09-21T14:08:10.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:10 smithi049 ceph-mon[30080]: 3.1 scrub ok 2024-09-21T14:08:10.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:10 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/383041312' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:08:10.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:10 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:08:10.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:10 smithi049 ceph-mon[30080]: 4.4 scrub starts 2024-09-21T14:08:10.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:10 smithi049 ceph-mon[30080]: 4.4 scrub ok 2024-09-21T14:08:10.773 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:08:10.809 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:10.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:10.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:08:10.910 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:10.961 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:10.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:10.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:11.063 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:08:11.108 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:11.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:11.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:08:11.211 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:08:11.245 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:11.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:11.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:08:11.347 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:11.375 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:11.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:11.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:11.477 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:08:11.519 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:11.519 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 78 16 112681 112665 90.2579 75.25 0.00866902 0.0110744 2024-09-21T14:08:11.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:11.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:08:11.621 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:11.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:11.646 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:11.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:11.687 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:11 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:08:11.687 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:11 smithi049 ceph-mon[40489]: pgmap v984: 168 pgs: 168 active+clean; 6.6 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s; 8.7 KiB/s, 0 objects/s recovering 2024-09-21T14:08:11.687 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:11 smithi049 ceph-mon[40489]: 6.9 scrub starts 2024-09-21T14:08:11.687 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:11 smithi049 ceph-mon[40489]: 6.9 scrub ok 2024-09-21T14:08:11.687 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:11 smithi049 ceph-mon[40489]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T14:08:11.687 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:11 smithi049 ceph-mon[40489]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs inactive, 13 pgs peering) 2024-09-21T14:08:11.688 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:11 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T14:08:11.688 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:11 smithi049 ceph-mon[40489]: osdmap e495: 8 total, 8 up, 8 in 2024-09-21T14:08:11.688 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:11 smithi049 ceph-mon[40489]: 11.f scrub starts 2024-09-21T14:08:11.688 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:11 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:08:11.688 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:11 smithi049 ceph-mon[30080]: pgmap v984: 168 pgs: 168 active+clean; 6.6 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s; 8.7 KiB/s, 0 objects/s recovering 2024-09-21T14:08:11.688 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:11 smithi049 ceph-mon[30080]: 6.9 scrub starts 2024-09-21T14:08:11.688 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:11 smithi049 ceph-mon[30080]: 6.9 scrub ok 2024-09-21T14:08:11.688 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:11 smithi049 ceph-mon[30080]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T14:08:11.688 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:11 smithi049 ceph-mon[30080]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs inactive, 13 pgs peering) 2024-09-21T14:08:11.688 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:11 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T14:08:11.688 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:11 smithi049 ceph-mon[30080]: osdmap e495: 8 total, 8 up, 8 in 2024-09-21T14:08:11.688 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:11 smithi049 ceph-mon[30080]: 11.f scrub starts 2024-09-21T14:08:11.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:11 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:08:11.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:11 smithi187 ceph-mon[33032]: pgmap v984: 168 pgs: 168 active+clean; 6.6 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s; 8.7 KiB/s, 0 objects/s recovering 2024-09-21T14:08:11.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:11 smithi187 ceph-mon[33032]: 6.9 scrub starts 2024-09-21T14:08:11.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:11 smithi187 ceph-mon[33032]: 6.9 scrub ok 2024-09-21T14:08:11.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:11 smithi187 ceph-mon[33032]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T14:08:11.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:11 smithi187 ceph-mon[33032]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs inactive, 13 pgs peering) 2024-09-21T14:08:11.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:11 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T14:08:11.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:11 smithi187 ceph-mon[33032]: osdmap e495: 8 total, 8 up, 8 in 2024-09-21T14:08:11.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:11 smithi187 ceph-mon[33032]: 11.f scrub starts 2024-09-21T14:08:11.748 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:11.776 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:11.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:11.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:11.878 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:08:11.904 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:11.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:11.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:08:12.006 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:12.034 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:12.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:12.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:12.136 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:08:12.184 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:12.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:12.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:08:12.286 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:08:12.321 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:12.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:12.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:08:12.423 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:12.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:12 smithi187 ceph-mon[33032]: 4.13 scrub starts 2024-09-21T14:08:12.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:12 smithi187 ceph-mon[33032]: 4.13 scrub ok 2024-09-21T14:08:12.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:12 smithi187 ceph-mon[33032]: 1.9 scrub starts 2024-09-21T14:08:12.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:12 smithi187 ceph-mon[33032]: 1.9 scrub ok 2024-09-21T14:08:12.468 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:12.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:12.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:12.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:12 smithi049 ceph-mon[40489]: 4.13 scrub starts 2024-09-21T14:08:12.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:12 smithi049 ceph-mon[40489]: 4.13 scrub ok 2024-09-21T14:08:12.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:12 smithi049 ceph-mon[40489]: 1.9 scrub starts 2024-09-21T14:08:12.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:12 smithi049 ceph-mon[40489]: 1.9 scrub ok 2024-09-21T14:08:12.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:12 smithi049 ceph-mon[30080]: 4.13 scrub starts 2024-09-21T14:08:12.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:12 smithi049 ceph-mon[30080]: 4.13 scrub ok 2024-09-21T14:08:12.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:12 smithi049 ceph-mon[30080]: 1.9 scrub starts 2024-09-21T14:08:12.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:12 smithi049 ceph-mon[30080]: 1.9 scrub ok 2024-09-21T14:08:12.521 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 79 16 113736 113720 89.9498 65.9375 0.00411021 0.0111124 2024-09-21T14:08:12.570 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:08:12.614 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:12.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:12.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:08:12.716 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:08:12.754 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:12.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:12.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:08:12.856 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:12.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:12.902 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:12.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:13.003 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:13.031 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:13.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:13.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:13.133 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:13.165 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:13.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:13.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:13.267 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:08:13.293 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:13.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:13.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:08:13.388 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.3 to 0.502232 (8092) 2024-09-21T14:08:13.394 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:08:13.440 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:13.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:13.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:08:13.470 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T14:08:13.470 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:08:13.519 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:2024-09-21T14:08:13.521037+0000 min lat: 0.00119458 max lat: 1.03552 avg lat: 0.0111487 2024-09-21T14:08:13.520 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T14:08:13.520 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 80 16 114677 114661 89.5606 58.8125 0.00303539 0.0111487 2024-09-21T14:08:13.542 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:13.563 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:13.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:13.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:13.665 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:13.678 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:13 smithi187 ceph-mon[33032]: 1.e scrub starts 2024-09-21T14:08:13.678 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:13 smithi187 ceph-mon[33032]: 1.e scrub ok 2024-09-21T14:08:13.678 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:13 smithi187 ceph-mon[33032]: 5.b scrub starts 2024-09-21T14:08:13.678 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:13 smithi187 ceph-mon[33032]: 5.b scrub ok 2024-09-21T14:08:13.678 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:13 smithi187 ceph-mon[33032]: pgmap v986: 168 pgs: 168 active+clean; 6.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.18k op/s; 0 B/s, 0 objects/s recovering 2024-09-21T14:08:13.678 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:13 smithi187 ceph-mon[33032]: 3.16 scrub starts 2024-09-21T14:08:13.678 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:13 smithi187 ceph-mon[33032]: 3.16 scrub ok 2024-09-21T14:08:13.678 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:13 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3034766589' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:08:13.678 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:13 smithi187 ceph-mon[33032]: 4.f scrub starts 2024-09-21T14:08:13.679 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:13 smithi187 ceph-mon[33032]: 4.f scrub ok 2024-09-21T14:08:13.679 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:13 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3142109721' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 0.5022322683272371} : dispatch 2024-09-21T14:08:13.679 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:13 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 0.5022322683272371} : dispatch 2024-09-21T14:08:13.686 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:13.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:13.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:13.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:13 smithi049 ceph-mon[40489]: 1.e scrub starts 2024-09-21T14:08:13.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:13 smithi049 ceph-mon[40489]: 1.e scrub ok 2024-09-21T14:08:13.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:13 smithi049 ceph-mon[40489]: 5.b scrub starts 2024-09-21T14:08:13.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:13 smithi049 ceph-mon[40489]: 5.b scrub ok 2024-09-21T14:08:13.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:13 smithi049 ceph-mon[40489]: pgmap v986: 168 pgs: 168 active+clean; 6.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.18k op/s; 0 B/s, 0 objects/s recovering 2024-09-21T14:08:13.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:13 smithi049 ceph-mon[40489]: 3.16 scrub starts 2024-09-21T14:08:13.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:13 smithi049 ceph-mon[40489]: 3.16 scrub ok 2024-09-21T14:08:13.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:13 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3034766589' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:08:13.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:13 smithi049 ceph-mon[40489]: 4.f scrub starts 2024-09-21T14:08:13.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:13 smithi049 ceph-mon[40489]: 4.f scrub ok 2024-09-21T14:08:13.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:13 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3142109721' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 0.5022322683272371} : dispatch 2024-09-21T14:08:13.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:13 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 0.5022322683272371} : dispatch 2024-09-21T14:08:13.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:13 smithi049 ceph-mon[30080]: 1.e scrub starts 2024-09-21T14:08:13.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:13 smithi049 ceph-mon[30080]: 1.e scrub ok 2024-09-21T14:08:13.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:13 smithi049 ceph-mon[30080]: 5.b scrub starts 2024-09-21T14:08:13.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:13 smithi049 ceph-mon[30080]: 5.b scrub ok 2024-09-21T14:08:13.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:13 smithi049 ceph-mon[30080]: pgmap v986: 168 pgs: 168 active+clean; 6.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.18k op/s; 0 B/s, 0 objects/s recovering 2024-09-21T14:08:13.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:13 smithi049 ceph-mon[30080]: 3.16 scrub starts 2024-09-21T14:08:13.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:13 smithi049 ceph-mon[30080]: 3.16 scrub ok 2024-09-21T14:08:13.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:13 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3034766589' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:08:13.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:13 smithi049 ceph-mon[30080]: 4.f scrub starts 2024-09-21T14:08:13.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:13 smithi049 ceph-mon[30080]: 4.f scrub ok 2024-09-21T14:08:13.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:13 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3142109721' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 0.5022322683272371} : dispatch 2024-09-21T14:08:13.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:13 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 0.5022322683272371} : dispatch 2024-09-21T14:08:13.788 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:13.808 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:13.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:13.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:13.910 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:08:13.931 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:13.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:13.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:08:14.033 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:14.055 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:14.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:14.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:14.157 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:08:14.172 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T14:08:14.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:14.180 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:14.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:08:14.281 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:08:14.303 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:14.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:14.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:08:14.407 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:08:14.414 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:14 smithi049 ceph-mon[30080]: 6.14 scrub starts 2024-09-21T14:08:14.414 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:14 smithi049 ceph-mon[30080]: 6.14 scrub ok 2024-09-21T14:08:14.433 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:14.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:14.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:08:14.510 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:08:14.510 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:08:14.510 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:08:14.510 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:08:14.511 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:08:14.511 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:08:14.511 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:08:14.511 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:08:14.511 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:08:14.511 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:08:14.511 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:08:14.511 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:08:14.512 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:08:14.512 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:08:14.512 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:08:14.512 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:08:14.519 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 81 16 114677 114661 88.455 0 - 0.0111487 2024-09-21T14:08:14.535 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:08:14.557 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:14.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:14.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:08:14.658 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:08:14.684 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:14.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:14.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:08:14.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:14 smithi187 ceph-mon[33032]: 6.14 scrub starts 2024-09-21T14:08:14.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:14 smithi187 ceph-mon[33032]: 6.14 scrub ok 2024-09-21T14:08:14.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:14 smithi187 ceph-mon[33032]: 4.1 scrub starts 2024-09-21T14:08:14.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:14 smithi187 ceph-mon[33032]: 4.1 scrub ok 2024-09-21T14:08:14.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:14 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 0.5022322683272371}]': finished 2024-09-21T14:08:14.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:14 smithi187 ceph-mon[33032]: osdmap e496: 8 total, 8 up, 8 in 2024-09-21T14:08:14.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:14 smithi187 ceph-mon[33032]: 11.f scrub ok 2024-09-21T14:08:14.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:14 smithi187 ceph-mon[33032]: 6.15 scrub starts 2024-09-21T14:08:14.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:14 smithi187 ceph-mon[33032]: 6.15 scrub ok 2024-09-21T14:08:14.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:14 smithi187 ceph-mon[33032]: osdmap e497: 8 total, 8 up, 8 in 2024-09-21T14:08:14.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:14 smithi049 ceph-mon[30080]: 4.1 scrub starts 2024-09-21T14:08:14.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:14 smithi049 ceph-mon[30080]: 4.1 scrub ok 2024-09-21T14:08:14.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:14 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 0.5022322683272371}]': finished 2024-09-21T14:08:14.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:14 smithi049 ceph-mon[30080]: osdmap e496: 8 total, 8 up, 8 in 2024-09-21T14:08:14.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:14 smithi049 ceph-mon[30080]: 11.f scrub ok 2024-09-21T14:08:14.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:14 smithi049 ceph-mon[30080]: 6.15 scrub starts 2024-09-21T14:08:14.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:14 smithi049 ceph-mon[30080]: 6.15 scrub ok 2024-09-21T14:08:14.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:14 smithi049 ceph-mon[30080]: osdmap e497: 8 total, 8 up, 8 in 2024-09-21T14:08:14.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:14 smithi049 ceph-mon[40489]: 6.14 scrub starts 2024-09-21T14:08:14.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:14 smithi049 ceph-mon[40489]: 6.14 scrub ok 2024-09-21T14:08:14.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:14 smithi049 ceph-mon[40489]: 4.1 scrub starts 2024-09-21T14:08:14.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:14 smithi049 ceph-mon[40489]: 4.1 scrub ok 2024-09-21T14:08:14.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:14 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 0.5022322683272371}]': finished 2024-09-21T14:08:14.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:14 smithi049 ceph-mon[40489]: osdmap e496: 8 total, 8 up, 8 in 2024-09-21T14:08:14.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:14 smithi049 ceph-mon[40489]: 11.f scrub ok 2024-09-21T14:08:14.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:14 smithi049 ceph-mon[40489]: 6.15 scrub starts 2024-09-21T14:08:14.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:14 smithi049 ceph-mon[40489]: 6.15 scrub ok 2024-09-21T14:08:14.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:14 smithi049 ceph-mon[40489]: osdmap e497: 8 total, 8 up, 8 in 2024-09-21T14:08:14.786 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:08:14.808 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:14.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:14.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:08:14.909 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:08:14.931 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:14.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:14.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:08:15.033 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:08:15.059 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:15.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:15.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:08:15.161 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:08:15.186 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:15.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:15.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:08:15.288 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:15.313 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:15.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:15.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:15.414 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:15.422 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T14:08:15.438 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:15.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:15.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:15.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:15 smithi049 ceph-mon[30080]: pgmap v988: 168 pgs: 2 unknown, 13 peering, 153 active+clean; 6.9 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-21T14:08:15.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:15 smithi049 ceph-mon[30080]: 4.10 scrub starts 2024-09-21T14:08:15.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:15 smithi049 ceph-mon[30080]: 4.10 scrub ok 2024-09-21T14:08:15.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:15 smithi049 ceph-mon[30080]: Health check failed: Reduced data availability: 6 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-09-21T14:08:15.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:15 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:08:15.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:15 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:08:15.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:15 smithi049 ceph-mon[30080]: 6.13 scrub starts 2024-09-21T14:08:15.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:15 smithi049 ceph-mon[30080]: 6.13 scrub ok 2024-09-21T14:08:15.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:15 smithi049 ceph-mon[30080]: 5.a scrub starts 2024-09-21T14:08:15.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:15 smithi049 ceph-mon[30080]: 5.a scrub ok 2024-09-21T14:08:15.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:15 smithi049 ceph-mon[30080]: osdmap e498: 8 total, 8 up, 8 in 2024-09-21T14:08:15.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:15 smithi049 ceph-mon[40489]: pgmap v988: 168 pgs: 2 unknown, 13 peering, 153 active+clean; 6.9 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-21T14:08:15.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:15 smithi049 ceph-mon[40489]: 4.10 scrub starts 2024-09-21T14:08:15.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:15 smithi049 ceph-mon[40489]: 4.10 scrub ok 2024-09-21T14:08:15.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:15 smithi049 ceph-mon[40489]: Health check failed: Reduced data availability: 6 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-09-21T14:08:15.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:15 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:08:15.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:15 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:08:15.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:15 smithi049 ceph-mon[40489]: 6.13 scrub starts 2024-09-21T14:08:15.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:15 smithi049 ceph-mon[40489]: 6.13 scrub ok 2024-09-21T14:08:15.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:15 smithi049 ceph-mon[40489]: 5.a scrub starts 2024-09-21T14:08:15.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:15 smithi049 ceph-mon[40489]: 5.a scrub ok 2024-09-21T14:08:15.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:15 smithi049 ceph-mon[40489]: osdmap e498: 8 total, 8 up, 8 in 2024-09-21T14:08:15.520 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 82 16 114706 114690 87.3985 0.90625 0.0158119 0.0114177 2024-09-21T14:08:15.540 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:15.573 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:15.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:15.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:15.675 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:08:15.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:15 smithi187 ceph-mon[33032]: pgmap v988: 168 pgs: 2 unknown, 13 peering, 153 active+clean; 6.9 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-21T14:08:15.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:15 smithi187 ceph-mon[33032]: 4.10 scrub starts 2024-09-21T14:08:15.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:15 smithi187 ceph-mon[33032]: 4.10 scrub ok 2024-09-21T14:08:15.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:15 smithi187 ceph-mon[33032]: Health check failed: Reduced data availability: 6 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-09-21T14:08:15.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:15 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:08:15.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:15 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:08:15.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:15 smithi187 ceph-mon[33032]: 6.13 scrub starts 2024-09-21T14:08:15.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:15 smithi187 ceph-mon[33032]: 6.13 scrub ok 2024-09-21T14:08:15.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:15 smithi187 ceph-mon[33032]: 5.a scrub starts 2024-09-21T14:08:15.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:15 smithi187 ceph-mon[33032]: 5.a scrub ok 2024-09-21T14:08:15.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:15 smithi187 ceph-mon[33032]: osdmap e498: 8 total, 8 up, 8 in 2024-09-21T14:08:15.724 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:15.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:15.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:08:15.801 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:08:15.826 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:15.857 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:15.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:15.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:15.958 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:08:16.011 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:16.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:16.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:08:16.113 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:08:16.147 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:16.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:16.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:08:16.249 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:16.280 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:16.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:16.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:16.381 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:08:16.426 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:16.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:16.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:08:16.520 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 83 16 115794 115778 87.1648 68 0.0109608 0.0114669 2024-09-21T14:08:16.528 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:16.575 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:16.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:16.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:16.677 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:08:16.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:16 smithi187 ceph-mon[33032]: 3.b scrub starts 2024-09-21T14:08:16.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:16 smithi187 ceph-mon[33032]: 3.b scrub ok 2024-09-21T14:08:16.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:16 smithi187 ceph-mon[33032]: 11.2 starting backfill to osd.7 from (0'0,0'0] MAX to 495'7095 2024-09-21T14:08:16.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:16 smithi187 ceph-mon[33032]: 11.1 starting backfill to osd.2 from (0'0,0'0] MAX to 495'7064 2024-09-21T14:08:16.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:16 smithi187 ceph-mon[33032]: 11.8 starting backfill to osd.2 from (0'0,0'0] MAX to 495'7032 2024-09-21T14:08:16.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:16 smithi187 ceph-mon[33032]: 11.0 starting backfill to osd.6 from (0'0,0'0] MAX to 495'7147 2024-09-21T14:08:16.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:16 smithi187 ceph-mon[33032]: 11.9 starting backfill to osd.7 from (0'0,0'0] MAX to 495'7185 2024-09-21T14:08:16.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:16 smithi187 ceph-mon[33032]: 3.9 scrub starts 2024-09-21T14:08:16.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:16 smithi187 ceph-mon[33032]: 3.9 scrub ok 2024-09-21T14:08:16.716 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:16 smithi049 ceph-mon[40489]: 3.b scrub starts 2024-09-21T14:08:16.716 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:16 smithi049 ceph-mon[40489]: 3.b scrub ok 2024-09-21T14:08:16.716 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:16 smithi049 ceph-mon[40489]: 11.2 starting backfill to osd.7 from (0'0,0'0] MAX to 495'7095 2024-09-21T14:08:16.716 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:16 smithi049 ceph-mon[40489]: 11.1 starting backfill to osd.2 from (0'0,0'0] MAX to 495'7064 2024-09-21T14:08:16.716 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:16 smithi049 ceph-mon[40489]: 11.8 starting backfill to osd.2 from (0'0,0'0] MAX to 495'7032 2024-09-21T14:08:16.716 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:16 smithi049 ceph-mon[40489]: 11.0 starting backfill to osd.6 from (0'0,0'0] MAX to 495'7147 2024-09-21T14:08:16.716 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:16 smithi049 ceph-mon[40489]: 11.9 starting backfill to osd.7 from (0'0,0'0] MAX to 495'7185 2024-09-21T14:08:16.717 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:16 smithi049 ceph-mon[40489]: 3.9 scrub starts 2024-09-21T14:08:16.717 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:16 smithi049 ceph-mon[40489]: 3.9 scrub ok 2024-09-21T14:08:16.717 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:16.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:16 smithi049 ceph-mon[30080]: 3.b scrub starts 2024-09-21T14:08:16.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:16 smithi049 ceph-mon[30080]: 3.b scrub ok 2024-09-21T14:08:16.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:16 smithi049 ceph-mon[30080]: 11.2 starting backfill to osd.7 from (0'0,0'0] MAX to 495'7095 2024-09-21T14:08:16.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:16 smithi049 ceph-mon[30080]: 11.1 starting backfill to osd.2 from (0'0,0'0] MAX to 495'7064 2024-09-21T14:08:16.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:16 smithi049 ceph-mon[30080]: 11.8 starting backfill to osd.2 from (0'0,0'0] MAX to 495'7032 2024-09-21T14:08:16.717 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:16 smithi049 ceph-mon[30080]: 11.0 starting backfill to osd.6 from (0'0,0'0] MAX to 495'7147 2024-09-21T14:08:16.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:16 smithi049 ceph-mon[30080]: 11.9 starting backfill to osd.7 from (0'0,0'0] MAX to 495'7185 2024-09-21T14:08:16.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:16 smithi049 ceph-mon[30080]: 3.9 scrub starts 2024-09-21T14:08:16.718 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:16 smithi049 ceph-mon[30080]: 3.9 scrub ok 2024-09-21T14:08:16.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:16.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:08:16.819 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:08:16.851 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:16.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:16.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:08:16.899 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T14:08:16.899 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-09-21T14:08:16.899 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-09-21T14:08:16.900 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-09-21T14:08:16.900 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.900 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:148200 10.8 10:11441144:::benchmark_data_smithi049_176811_object20623:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-09-21T14:08:16.900 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:04:55.587007+0000", 2024-09-21T14:08:16.900 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 201.30928857399999, 2024-09-21T14:08:16.900 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.019040647, 2024-09-21T14:08:16.900 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:08:16.900 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:08:16.900 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:08:16.900 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:08:16.900 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:08:16.900 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:08:16.900 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 148200 2024-09-21T14:08:16.901 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.901 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:08:16.901 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.901 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:08:16.901 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.587007+0000", 2024-09-21T14:08:16.901 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:16.901 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.901 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.901 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:08:16.901 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.587007+0000", 2024-09-21T14:08:16.901 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:16.901 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.902 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.902 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:08:16.902 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.587008+0000", 2024-09-21T14:08:16.902 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.649e-06 2024-09-21T14:08:16.902 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.902 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.902 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:08:16.902 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.587014+0000", 2024-09-21T14:08:16.902 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5849999999999999e-06 2024-09-21T14:08:16.903 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.903 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.903 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:08:16.903 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.587031+0000", 2024-09-21T14:08:16.903 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6756e-05 2024-09-21T14:08:16.903 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.903 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.903 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:08:16.903 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.587040+0000", 2024-09-21T14:08:16.903 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3510000000000008e-06 2024-09-21T14:08:16.903 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.903 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.903 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:16.904 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.587135+0000", 2024-09-21T14:08:16.904 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4471999999999999e-05 2024-09-21T14:08:16.904 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.904 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.904 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:08:16.904 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.587145+0000", 2024-09-21T14:08:16.904 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0186e-05 2024-09-21T14:08:16.904 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.904 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.904 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:16.904 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.598177+0000", 2024-09-21T14:08:16.904 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0110321339999999 2024-09-21T14:08:16.904 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.905 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.905 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:08:16.905 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.599612+0000", 2024-09-21T14:08:16.905 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001434828 2024-09-21T14:08:16.905 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.905 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.905 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,7", 2024-09-21T14:08:16.905 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.599704+0000", 2024-09-21T14:08:16.905 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2164999999999996e-05 2024-09-21T14:08:16.905 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.905 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.905 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:08:16.905 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.601928+0000", 2024-09-21T14:08:16.905 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0022241869999999999 2024-09-21T14:08:16.905 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.906 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.906 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:16.906 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.605855+0000", 2024-09-21T14:08:16.906 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0039272839999999996 2024-09-21T14:08:16.906 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.906 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.906 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:16.906 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.605933+0000", 2024-09-21T14:08:16.906 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7194999999999995e-05 2024-09-21T14:08:16.906 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.906 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.906 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:08:16.906 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.605997+0000", 2024-09-21T14:08:16.907 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4182000000000003e-05 2024-09-21T14:08:16.907 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.907 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.907 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:08:16.907 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.606047+0000", 2024-09-21T14:08:16.907 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0673000000000001e-05 2024-09-21T14:08:16.907 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:16.907 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:16.907 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:16.907 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.908 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.908 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:148204 10.8 10:173acc92:::benchmark_data_smithi049_176811_object20627:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-09-21T14:08:16.908 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:04:55.598766+0000", 2024-09-21T14:08:16.908 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 201.29752965099999, 2024-09-21T14:08:16.908 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.012530269, 2024-09-21T14:08:16.908 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:08:16.908 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:08:16.908 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:08:16.908 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:08:16.908 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:08:16.908 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:08:16.908 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 148204 2024-09-21T14:08:16.908 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.908 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:08:16.909 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.909 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:08:16.909 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.598766+0000", 2024-09-21T14:08:16.909 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:16.909 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.909 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.909 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:08:16.909 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.598766+0000", 2024-09-21T14:08:16.909 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:16.909 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.909 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.909 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:08:16.909 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.598768+0000", 2024-09-21T14:08:16.910 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.951e-06 2024-09-21T14:08:16.910 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.910 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.910 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:08:16.910 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.598775+0000", 2024-09-21T14:08:16.910 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8450000000000003e-06 2024-09-21T14:08:16.910 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.910 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.910 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:08:16.910 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.598793+0000", 2024-09-21T14:08:16.910 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7348999999999999e-05 2024-09-21T14:08:16.910 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.911 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.911 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:08:16.911 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.598803+0000", 2024-09-21T14:08:16.911 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8120000000000002e-06 2024-09-21T14:08:16.911 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.911 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.911 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:16.911 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.598888+0000", 2024-09-21T14:08:16.911 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4975999999999994e-05 2024-09-21T14:08:16.911 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.911 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.911 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:08:16.911 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.598898+0000", 2024-09-21T14:08:16.911 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0899e-05 2024-09-21T14:08:16.912 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.912 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.912 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:16.912 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.601975+0000", 2024-09-21T14:08:16.912 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0030761989999999 2024-09-21T14:08:16.912 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.912 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.912 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:08:16.912 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.602095+0000", 2024-09-21T14:08:16.912 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012031699999999999 2024-09-21T14:08:16.912 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.912 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.912 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,7", 2024-09-21T14:08:16.912 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.602214+0000", 2024-09-21T14:08:16.913 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011874700000000001 2024-09-21T14:08:16.913 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.913 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.913 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:08:16.913 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.606106+0000", 2024-09-21T14:08:16.913 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0038918389999999998 2024-09-21T14:08:16.913 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.913 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.913 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:16.913 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.606149+0000", 2024-09-21T14:08:16.914 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3593000000000001e-05 2024-09-21T14:08:16.914 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.914 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.914 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:16.914 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.611191+0000", 2024-09-21T14:08:16.914 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0050419280000000002 2024-09-21T14:08:16.914 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.914 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.914 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:08:16.914 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.611250+0000", 2024-09-21T14:08:16.914 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8956000000000003e-05 2024-09-21T14:08:16.914 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.914 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.915 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:08:16.915 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.611296+0000", 2024-09-21T14:08:16.915 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5858e-05 2024-09-21T14:08:16.915 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:16.915 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:16.915 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:16.916 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.916 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.916 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:148276 10.8 10:1259e2cf:::benchmark_data_smithi049_176811_object20699:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-09-21T14:08:16.916 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:04:55.670175+0000", 2024-09-21T14:08:16.916 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 201.22612010099999, 2024-09-21T14:08:16.916 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.94134552400000004, 2024-09-21T14:08:16.916 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:08:16.916 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:08:16.916 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:08:16.916 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:08:16.916 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:08:16.916 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:08:16.917 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 148276 2024-09-21T14:08:16.917 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.917 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:08:16.917 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.917 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:08:16.917 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.670175+0000", 2024-09-21T14:08:16.917 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:16.917 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.917 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.917 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:08:16.918 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.670175+0000", 2024-09-21T14:08:16.918 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:16.918 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.918 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.918 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:08:16.918 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.670177+0000", 2024-09-21T14:08:16.918 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.911e-06 2024-09-21T14:08:16.918 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.918 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.918 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:08:16.918 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.670183+0000", 2024-09-21T14:08:16.919 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1190000000000002e-06 2024-09-21T14:08:16.919 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.919 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.919 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:08:16.919 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.670198+0000", 2024-09-21T14:08:16.919 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5204e-05 2024-09-21T14:08:16.919 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.919 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.919 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:08:16.919 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.670207+0000", 2024-09-21T14:08:16.919 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1710000000000002e-06 2024-09-21T14:08:16.919 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.919 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.919 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:16.920 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.670292+0000", 2024-09-21T14:08:16.920 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5241999999999995e-05 2024-09-21T14:08:16.920 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.920 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.920 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:08:16.920 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.670302+0000", 2024-09-21T14:08:16.920 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.009e-05 2024-09-21T14:08:16.920 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.920 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.920 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:16.920 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.603863+0000", 2024-09-21T14:08:16.920 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.93356139299999996 2024-09-21T14:08:16.921 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.921 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.921 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:08:16.921 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.603979+0000", 2024-09-21T14:08:16.921 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000116057 2024-09-21T14:08:16.921 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.921 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.921 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,7", 2024-09-21T14:08:16.921 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.604090+0000", 2024-09-21T14:08:16.921 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011069900000000001 2024-09-21T14:08:16.921 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.921 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.921 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:08:16.921 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.606163+0000", 2024-09-21T14:08:16.921 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0020731040000000001 2024-09-21T14:08:16.922 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.922 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.922 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:16.922 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.606226+0000", 2024-09-21T14:08:16.922 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3025000000000001e-05 2024-09-21T14:08:16.922 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.922 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.922 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:16.922 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.611418+0000", 2024-09-21T14:08:16.922 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0051918659999999998 2024-09-21T14:08:16.922 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.922 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.923 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:08:16.923 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.611466+0000", 2024-09-21T14:08:16.923 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8284000000000001e-05 2024-09-21T14:08:16.923 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.923 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.923 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:08:16.923 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.611521+0000", 2024-09-21T14:08:16.923 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4358999999999999e-05 2024-09-21T14:08:16.923 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:16.923 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:16.923 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:16.924 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.924 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.924 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:148281 10.8 10:11e48ab0:::benchmark_data_smithi049_176811_object20704:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-09-21T14:08:16.924 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:04:55.671888+0000", 2024-09-21T14:08:16.924 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 201.22440683400001, 2024-09-21T14:08:16.924 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.93981524800000005, 2024-09-21T14:08:16.924 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:08:16.924 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:08:16.924 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:08:16.924 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:08:16.924 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:08:16.924 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:08:16.924 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 148281 2024-09-21T14:08:16.924 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.925 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:08:16.925 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.925 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:08:16.925 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.671888+0000", 2024-09-21T14:08:16.925 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:16.925 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.925 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.925 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:08:16.925 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.671888+0000", 2024-09-21T14:08:16.925 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:16.925 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.925 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.925 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:08:16.925 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.671889+0000", 2024-09-21T14:08:16.925 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.009e-06 2024-09-21T14:08:16.926 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.926 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.926 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:08:16.926 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.671894+0000", 2024-09-21T14:08:16.926 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2459999999999997e-06 2024-09-21T14:08:16.926 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.926 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.926 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:08:16.926 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.671904+0000", 2024-09-21T14:08:16.926 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0505e-05 2024-09-21T14:08:16.926 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.926 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.927 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:08:16.927 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.671910+0000", 2024-09-21T14:08:16.927 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3329999999999996e-06 2024-09-21T14:08:16.927 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.927 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.927 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:16.927 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.671956+0000", 2024-09-21T14:08:16.927 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6570000000000003e-05 2024-09-21T14:08:16.927 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.927 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.927 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:08:16.927 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.671962+0000", 2024-09-21T14:08:16.928 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6659999999999996e-06 2024-09-21T14:08:16.928 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.928 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.928 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:16.928 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.606281+0000", 2024-09-21T14:08:16.928 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.93431893200000005 2024-09-21T14:08:16.928 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.928 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.928 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:08:16.929 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.606381+0000", 2024-09-21T14:08:16.929 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010040200000000001 2024-09-21T14:08:16.929 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.929 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.929 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,7", 2024-09-21T14:08:16.929 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.606453+0000", 2024-09-21T14:08:16.929 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1423999999999999e-05 2024-09-21T14:08:16.929 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.929 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.929 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:16.929 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.611355+0000", 2024-09-21T14:08:16.929 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0049022880000000003 2024-09-21T14:08:16.930 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.930 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.930 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:08:16.930 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.611534+0000", 2024-09-21T14:08:16.930 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000179116 2024-09-21T14:08:16.930 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.930 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.930 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:16.930 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.611636+0000", 2024-09-21T14:08:16.931 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000101909 2024-09-21T14:08:16.931 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.931 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.931 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:08:16.931 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.611666+0000", 2024-09-21T14:08:16.931 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.0468e-05 2024-09-21T14:08:16.931 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.931 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.931 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:08:16.931 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.611704+0000", 2024-09-21T14:08:16.931 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7379999999999998e-05 2024-09-21T14:08:16.931 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:16.932 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:16.932 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:16.932 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.932 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.932 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:148288 10.8 10:1610cc8e:::benchmark_data_smithi049_176811_object20711:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-09-21T14:08:16.932 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:04:55.675295+0000", 2024-09-21T14:08:16.932 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 201.220999861, 2024-09-21T14:08:16.932 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.94425190999999997, 2024-09-21T14:08:16.932 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:08:16.932 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:08:16.932 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:08:16.933 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:08:16.933 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:08:16.933 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:08:16.933 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 148288 2024-09-21T14:08:16.933 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.933 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:08:16.933 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.933 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:08:16.933 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.675295+0000", 2024-09-21T14:08:16.933 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:16.933 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.934 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.934 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:08:16.934 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.675295+0000", 2024-09-21T14:08:16.934 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:16.934 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.934 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.934 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:08:16.934 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.675297+0000", 2024-09-21T14:08:16.934 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4890000000000001e-06 2024-09-21T14:08:16.935 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.935 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.935 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:08:16.935 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.675302+0000", 2024-09-21T14:08:16.935 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8250000000000004e-06 2024-09-21T14:08:16.935 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.935 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.935 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:08:16.935 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.675321+0000", 2024-09-21T14:08:16.935 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8981999999999999e-05 2024-09-21T14:08:16.935 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.935 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.935 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:08:16.935 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.675329+0000", 2024-09-21T14:08:16.936 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2940000000000006e-06 2024-09-21T14:08:16.936 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.936 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.936 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:16.936 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.675411+0000", 2024-09-21T14:08:16.936 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1927000000000004e-05 2024-09-21T14:08:16.936 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.936 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.936 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:08:16.936 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.675421+0000", 2024-09-21T14:08:16.936 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9389999999999992e-06 2024-09-21T14:08:16.936 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.937 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.937 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:16.937 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.608831+0000", 2024-09-21T14:08:16.937 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.93340988899999999 2024-09-21T14:08:16.937 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.937 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.937 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:08:16.937 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.608931+0000", 2024-09-21T14:08:16.937 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000100715 2024-09-21T14:08:16.937 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.937 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.937 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,7", 2024-09-21T14:08:16.938 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.609052+0000", 2024-09-21T14:08:16.938 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000120139 2024-09-21T14:08:16.938 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.938 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.938 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:08:16.938 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.611541+0000", 2024-09-21T14:08:16.938 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0024891079999999999 2024-09-21T14:08:16.938 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.938 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.939 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:16.939 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.611824+0000", 2024-09-21T14:08:16.939 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00028299899999999998 2024-09-21T14:08:16.939 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.939 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.939 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:16.939 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.619454+0000", 2024-09-21T14:08:16.939 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.007630375 2024-09-21T14:08:16.939 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.939 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.940 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:08:16.940 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.619487+0000", 2024-09-21T14:08:16.940 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2790000000000003e-05 2024-09-21T14:08:16.940 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.940 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.940 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:08:16.940 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.619547+0000", 2024-09-21T14:08:16.940 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0439000000000002e-05 2024-09-21T14:08:16.940 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:16.940 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:16.940 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:16.941 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.941 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.941 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:148289 10.8 10:19cc1202:::benchmark_data_smithi049_176811_object20712:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-09-21T14:08:16.941 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:04:55.675487+0000", 2024-09-21T14:08:16.941 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 201.220808551, 2024-09-21T14:08:16.941 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.95131562199999997, 2024-09-21T14:08:16.941 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:08:16.941 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:08:16.941 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:08:16.942 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:08:16.942 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:08:16.942 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:08:16.942 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 148289 2024-09-21T14:08:16.942 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.942 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:08:16.942 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.942 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:08:16.942 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.675487+0000", 2024-09-21T14:08:16.942 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:16.942 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.942 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.942 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:08:16.942 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.675487+0000", 2024-09-21T14:08:16.943 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:16.943 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.943 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.943 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:08:16.943 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.675488+0000", 2024-09-21T14:08:16.943 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4200000000000005e-07 2024-09-21T14:08:16.943 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.943 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.943 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:08:16.943 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.675507+0000", 2024-09-21T14:08:16.943 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9219999999999999e-05 2024-09-21T14:08:16.943 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.943 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.943 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:08:16.944 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.675521+0000", 2024-09-21T14:08:16.944 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4688999999999999e-05 2024-09-21T14:08:16.944 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.944 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.944 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:08:16.944 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.675529+0000", 2024-09-21T14:08:16.944 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2599999999999999e-06 2024-09-21T14:08:16.944 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.944 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.944 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:16.944 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.675588+0000", 2024-09-21T14:08:16.945 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8936e-05 2024-09-21T14:08:16.945 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.945 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.945 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:08:16.945 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.675597+0000", 2024-09-21T14:08:16.945 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6889999999999999e-06 2024-09-21T14:08:16.945 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.945 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.945 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:16.945 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.611876+0000", 2024-09-21T14:08:16.945 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.93627835500000001 2024-09-21T14:08:16.945 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.945 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.945 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:08:16.946 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.611995+0000", 2024-09-21T14:08:16.946 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000119517 2024-09-21T14:08:16.946 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.946 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.946 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,7", 2024-09-21T14:08:16.946 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.612090+0000", 2024-09-21T14:08:16.946 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4495999999999998e-05 2024-09-21T14:08:16.946 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.946 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.946 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:16.946 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.619609+0000", 2024-09-21T14:08:16.946 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0075196880000000001 2024-09-21T14:08:16.946 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.946 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.947 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:08:16.947 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.619628+0000", 2024-09-21T14:08:16.947 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8425000000000001e-05 2024-09-21T14:08:16.947 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.947 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.947 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:16.947 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.626706+0000", 2024-09-21T14:08:16.947 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:08:16.948 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0070784200000000002 2024-09-21T14:08:16.948 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.948 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.948 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:08:16.948 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.626757+0000", 2024-09-21T14:08:16.949 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0977999999999999e-05 2024-09-21T14:08:16.949 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.949 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.949 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:08:16.949 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.626802+0000", 2024-09-21T14:08:16.949 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5107e-05 2024-09-21T14:08:16.949 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:16.949 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:16.949 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:16.949 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.950 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.950 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:148298 10.8 10:1042f94c:::benchmark_data_smithi049_176811_object20721:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-09-21T14:08:16.950 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:04:55.686237+0000", 2024-09-21T14:08:16.950 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 201.21005842299999, 2024-09-21T14:08:16.950 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.94072119499999995, 2024-09-21T14:08:16.950 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:08:16.950 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:08:16.950 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:08:16.950 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:08:16.950 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:08:16.950 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:08:16.950 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 148298 2024-09-21T14:08:16.951 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.951 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:08:16.951 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.951 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:08:16.951 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.686237+0000", 2024-09-21T14:08:16.951 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:16.951 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.951 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.951 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:08:16.951 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.686237+0000", 2024-09-21T14:08:16.951 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:16.952 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.952 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.952 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:08:16.952 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.686238+0000", 2024-09-21T14:08:16.952 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5519999999999999e-06 2024-09-21T14:08:16.952 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.952 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.952 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:08:16.952 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.686245+0000", 2024-09-21T14:08:16.952 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1449999999999996e-06 2024-09-21T14:08:16.952 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.952 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.952 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:08:16.952 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.686262+0000", 2024-09-21T14:08:16.953 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7258999999999999e-05 2024-09-21T14:08:16.953 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.953 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.953 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:08:16.953 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.686270+0000", 2024-09-21T14:08:16.953 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6820000000000003e-06 2024-09-21T14:08:16.953 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.953 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.953 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:16.953 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.686328+0000", 2024-09-21T14:08:16.953 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8081999999999997e-05 2024-09-21T14:08:16.953 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.953 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.953 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:08:16.954 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.686337+0000", 2024-09-21T14:08:16.954 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7159999999999998e-06 2024-09-21T14:08:16.954 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.954 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.954 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:16.954 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.614591+0000", 2024-09-21T14:08:16.954 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.92825406499999996 2024-09-21T14:08:16.954 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.954 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.954 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:08:16.954 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.614697+0000", 2024-09-21T14:08:16.954 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010538400000000001 2024-09-21T14:08:16.955 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.955 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.955 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,7", 2024-09-21T14:08:16.955 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.614812+0000", 2024-09-21T14:08:16.955 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000115492 2024-09-21T14:08:16.955 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.955 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.955 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:08:16.955 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.619633+0000", 2024-09-21T14:08:16.956 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0048206730000000001 2024-09-21T14:08:16.956 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.956 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.956 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:16.956 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.619684+0000", 2024-09-21T14:08:16.956 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1159000000000003e-05 2024-09-21T14:08:16.956 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.956 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.956 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:16.956 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.626880+0000", 2024-09-21T14:08:16.956 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0071960069999999999 2024-09-21T14:08:16.956 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.956 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.956 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:08:16.956 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.626926+0000", 2024-09-21T14:08:16.957 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5794999999999998e-05 2024-09-21T14:08:16.957 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.957 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.957 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:08:16.957 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.626958+0000", 2024-09-21T14:08:16.957 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2184000000000002e-05 2024-09-21T14:08:16.957 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:16.957 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:16.957 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:16.957 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.957 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.957 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:148326 10.8 10:190bfd8a:::benchmark_data_smithi049_176811_object20749:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-09-21T14:08:16.957 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:04:55.712544+0000", 2024-09-21T14:08:16.957 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 201.183750997, 2024-09-21T14:08:16.958 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.91455890500000003, 2024-09-21T14:08:16.958 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:08:16.958 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:08:16.958 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:08:16.958 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:08:16.958 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:08:16.958 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:08:16.958 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 148326 2024-09-21T14:08:16.958 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.958 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:08:16.958 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.958 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:08:16.958 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.712544+0000", 2024-09-21T14:08:16.959 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:16.959 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.959 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.959 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:08:16.959 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.712544+0000", 2024-09-21T14:08:16.959 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:16.959 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.959 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.959 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:08:16.959 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.712546+0000", 2024-09-21T14:08:16.959 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4079999999999999e-06 2024-09-21T14:08:16.959 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.960 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.960 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:08:16.960 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.712551+0000", 2024-09-21T14:08:16.960 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4890000000000003e-06 2024-09-21T14:08:16.960 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.960 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.960 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:08:16.960 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.712566+0000", 2024-09-21T14:08:16.960 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5203e-05 2024-09-21T14:08:16.960 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.961 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.961 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:08:16.961 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.712573+0000", 2024-09-21T14:08:16.961 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3670000000000002e-06 2024-09-21T14:08:16.961 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.961 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.961 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:16.961 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.712633+0000", 2024-09-21T14:08:16.961 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0538000000000002e-05 2024-09-21T14:08:16.961 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.961 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.961 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:08:16.961 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.712643+0000", 2024-09-21T14:08:16.962 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0032e-05 2024-09-21T14:08:16.962 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.962 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.962 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:16.962 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.617061+0000", 2024-09-21T14:08:16.962 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.90441751699999995 2024-09-21T14:08:16.962 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.962 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.962 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:08:16.962 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.617158+0000", 2024-09-21T14:08:16.962 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6871000000000001e-05 2024-09-21T14:08:16.963 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.963 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.963 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,7", 2024-09-21T14:08:16.963 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.617226+0000", 2024-09-21T14:08:16.963 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8147999999999999e-05 2024-09-21T14:08:16.963 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.963 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.965 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:08:16.965 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.623899+0000", 2024-09-21T14:08:16.966 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0066735259999999999 2024-09-21T14:08:16.966 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.966 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.966 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:16.966 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.626846+0000", 2024-09-21T14:08:16.966 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002946204 2024-09-21T14:08:16.966 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.966 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.966 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:16.966 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.627031+0000", 2024-09-21T14:08:16.966 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00018589100000000001 2024-09-21T14:08:16.966 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.967 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.967 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:08:16.967 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.627071+0000", 2024-09-21T14:08:16.967 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9215999999999999e-05 2024-09-21T14:08:16.967 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.967 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.967 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:08:16.967 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.627103+0000", 2024-09-21T14:08:16.968 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2495e-05 2024-09-21T14:08:16.968 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:16.968 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:16.968 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:16.968 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.968 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.968 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:148345 10.8 10:1ca635b1:::benchmark_data_smithi049_176811_object20768:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-09-21T14:08:16.968 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:04:55.730454+0000", 2024-09-21T14:08:16.968 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 201.165841207, 2024-09-21T14:08:16.968 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.90786199499999998, 2024-09-21T14:08:16.968 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:08:16.968 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:08:16.968 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:08:16.968 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:08:16.968 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:08:16.969 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:08:16.969 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 148345 2024-09-21T14:08:16.969 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.969 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:08:16.969 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.969 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:08:16.969 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.730454+0000", 2024-09-21T14:08:16.969 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:16.969 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.969 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.969 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:08:16.969 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.730454+0000", 2024-09-21T14:08:16.969 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:16.969 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.970 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.970 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:08:16.970 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.730456+0000", 2024-09-21T14:08:16.970 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.668e-06 2024-09-21T14:08:16.970 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.970 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.970 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:08:16.970 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.730463+0000", 2024-09-21T14:08:16.970 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8240000000000002e-06 2024-09-21T14:08:16.970 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.970 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.970 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:08:16.970 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.730478+0000", 2024-09-21T14:08:16.971 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5364999999999999e-05 2024-09-21T14:08:16.971 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.971 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.971 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:08:16.971 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.730486+0000", 2024-09-21T14:08:16.971 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.225e-06 2024-09-21T14:08:16.971 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.971 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.971 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:16.971 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.730567+0000", 2024-09-21T14:08:16.971 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0895000000000004e-05 2024-09-21T14:08:16.972 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.972 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.972 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:08:16.972 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.730577+0000", 2024-09-21T14:08:16.972 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7569999999999999e-06 2024-09-21T14:08:16.972 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.972 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.972 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:16.972 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.619717+0000", 2024-09-21T14:08:16.972 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.88914048199999995 2024-09-21T14:08:16.972 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.972 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.973 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:08:16.973 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.619824+0000", 2024-09-21T14:08:16.973 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000107046 2024-09-21T14:08:16.973 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.973 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.973 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,7", 2024-09-21T14:08:16.973 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.619921+0000", 2024-09-21T14:08:16.973 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6459000000000006e-05 2024-09-21T14:08:16.973 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.973 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.973 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:16.973 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.626996+0000", 2024-09-21T14:08:16.973 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0070750450000000003 2024-09-21T14:08:16.973 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.973 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.974 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:16.974 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.632227+0000", 2024-09-21T14:08:16.974 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0052307669999999999 2024-09-21T14:08:16.974 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.974 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.974 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:08:16.974 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.638223+0000", 2024-09-21T14:08:16.974 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.005996159 2024-09-21T14:08:16.974 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.974 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.974 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:08:16.974 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.638272+0000", 2024-09-21T14:08:16.974 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8837e-05 2024-09-21T14:08:16.975 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.975 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.975 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:08:16.975 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.638316+0000", 2024-09-21T14:08:16.975 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4465999999999997e-05 2024-09-21T14:08:16.975 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:16.975 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:16.975 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:16.975 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.975 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.975 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:148362 10.8 10:1e3ef663:::benchmark_data_smithi049_176811_object20785:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-09-21T14:08:16.975 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:04:55.747088+0000", 2024-09-21T14:08:16.975 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 201.149207166, 2024-09-21T14:08:16.976 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.89129988000000004, 2024-09-21T14:08:16.976 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:08:16.976 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:08:16.976 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:08:16.976 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:08:16.976 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:08:16.976 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:08:16.976 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 148362 2024-09-21T14:08:16.976 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.976 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:08:16.976 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.976 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:08:16.976 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.747088+0000", 2024-09-21T14:08:16.976 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:16.977 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.977 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.977 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:08:16.977 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.747088+0000", 2024-09-21T14:08:16.977 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:16.977 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.977 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.977 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:08:16.977 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.747090+0000", 2024-09-21T14:08:16.977 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1409999999999999e-06 2024-09-21T14:08:16.977 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.977 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.977 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:08:16.977 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.747097+0000", 2024-09-21T14:08:16.978 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.409e-06 2024-09-21T14:08:16.978 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.978 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.978 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:08:16.978 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.747117+0000", 2024-09-21T14:08:16.978 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0125000000000001e-05 2024-09-21T14:08:16.978 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.978 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.978 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:08:16.978 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.747125+0000", 2024-09-21T14:08:16.978 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0059999999999992e-06 2024-09-21T14:08:16.978 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.978 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.978 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:16.979 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.747263+0000", 2024-09-21T14:08:16.979 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013833000000000001 2024-09-21T14:08:16.979 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.979 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.979 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:08:16.979 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.747274+0000", 2024-09-21T14:08:16.979 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1207e-05 2024-09-21T14:08:16.979 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.979 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.979 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:16.979 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.623995+0000", 2024-09-21T14:08:16.979 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.87672050400000001 2024-09-21T14:08:16.979 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.979 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.979 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:08:16.980 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.624107+0000", 2024-09-21T14:08:16.980 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011234300000000001 2024-09-21T14:08:16.980 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.980 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.980 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,7", 2024-09-21T14:08:16.980 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.624240+0000", 2024-09-21T14:08:16.980 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000132922 2024-09-21T14:08:16.980 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.980 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.980 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:16.980 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.632186+0000", 2024-09-21T14:08:16.980 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0079461110000000005 2024-09-21T14:08:16.980 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.980 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.980 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:16.981 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.632268+0000", 2024-09-21T14:08:16.981 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1803000000000003e-05 2024-09-21T14:08:16.981 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.981 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.981 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:08:16.981 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.638322+0000", 2024-09-21T14:08:16.981 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0060544509999999998 2024-09-21T14:08:16.981 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.981 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.982 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:08:16.982 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.638358+0000", 2024-09-21T14:08:16.982 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5599999999999998e-05 2024-09-21T14:08:16.982 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.982 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.982 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:08:16.982 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.638388+0000", 2024-09-21T14:08:16.982 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9927999999999999e-05 2024-09-21T14:08:16.982 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:16.982 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:16.982 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:16.983 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.983 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.983 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:148377 10.8 10:167b8c70:::benchmark_data_smithi049_176811_object20800:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-09-21T14:08:16.983 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:04:55.764470+0000", 2024-09-21T14:08:16.983 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 201.131825096, 2024-09-21T14:08:16.983 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.87398584000000001, 2024-09-21T14:08:16.983 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:08:16.983 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:08:16.983 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:08:16.983 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:08:16.983 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:08:16.983 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:08:16.983 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 148377 2024-09-21T14:08:16.984 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.984 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:08:16.984 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.984 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:08:16.984 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.764470+0000", 2024-09-21T14:08:16.984 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:16.984 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.984 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.984 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:08:16.984 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.764470+0000", 2024-09-21T14:08:16.984 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:16.984 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.984 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.984 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:08:16.984 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.764472+0000", 2024-09-21T14:08:16.985 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.663e-06 2024-09-21T14:08:16.985 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.985 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.985 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:08:16.985 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.764479+0000", 2024-09-21T14:08:16.985 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0849999999999999e-06 2024-09-21T14:08:16.985 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.985 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.985 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:08:16.985 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.764506+0000", 2024-09-21T14:08:16.985 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6900999999999999e-05 2024-09-21T14:08:16.985 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.985 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.985 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:08:16.986 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.764515+0000", 2024-09-21T14:08:16.986 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1360000000000003e-06 2024-09-21T14:08:16.986 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.986 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.986 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:16.986 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.764593+0000", 2024-09-21T14:08:16.986 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8506999999999999e-05 2024-09-21T14:08:16.986 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.986 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.986 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:08:16.986 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.764605+0000", 2024-09-21T14:08:16.986 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1321000000000001e-05 2024-09-21T14:08:16.986 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.986 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.987 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:16.987 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.627131+0000", 2024-09-21T14:08:16.987 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.862526499 2024-09-21T14:08:16.987 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.987 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.987 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:08:16.987 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.627233+0000", 2024-09-21T14:08:16.987 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010123 2024-09-21T14:08:16.987 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.987 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.987 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,7", 2024-09-21T14:08:16.987 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.627334+0000", 2024-09-21T14:08:16.987 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000101437 2024-09-21T14:08:16.987 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.988 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.988 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:16.988 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.638176+0000", 2024-09-21T14:08:16.988 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.010841991 2024-09-21T14:08:16.988 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.988 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.988 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:16.988 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.638213+0000", 2024-09-21T14:08:16.988 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6758999999999998e-05 2024-09-21T14:08:16.988 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.988 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.988 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:08:16.988 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.638393+0000", 2024-09-21T14:08:16.988 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00018035999999999999 2024-09-21T14:08:16.989 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.989 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.989 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:08:16.989 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.638427+0000", 2024-09-21T14:08:16.989 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4155000000000001e-05 2024-09-21T14:08:16.989 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.989 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.989 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:08:16.989 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.638456+0000", 2024-09-21T14:08:16.989 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.8796e-05 2024-09-21T14:08:16.989 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:16.989 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:16.989 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:16.989 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.990 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.990 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:148380 10.8 10:184ea8ff:::benchmark_data_smithi049_176811_object20803:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-09-21T14:08:16.990 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:04:55.765967+0000", 2024-09-21T14:08:16.990 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 201.13032870000001, 2024-09-21T14:08:16.990 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.88603139099999995, 2024-09-21T14:08:16.990 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:08:16.990 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:08:16.990 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:08:16.990 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:08:16.990 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:08:16.990 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:08:16.990 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 148380 2024-09-21T14:08:16.991 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.991 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:08:16.991 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.991 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:08:16.991 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.765967+0000", 2024-09-21T14:08:16.991 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:16.991 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.991 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.991 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:08:16.991 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.765967+0000", 2024-09-21T14:08:16.991 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:16.991 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.991 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.991 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:08:16.991 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.765968+0000", 2024-09-21T14:08:16.992 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3030000000000001e-06 2024-09-21T14:08:16.992 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.992 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.992 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:08:16.992 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.765974+0000", 2024-09-21T14:08:16.992 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3999999999999997e-06 2024-09-21T14:08:16.992 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.992 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.992 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:08:16.993 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.765995+0000", 2024-09-21T14:08:16.993 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0267000000000001e-05 2024-09-21T14:08:16.993 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.993 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.993 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:08:16.993 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.766004+0000", 2024-09-21T14:08:16.993 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4730000000000006e-06 2024-09-21T14:08:16.993 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.993 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.993 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:16.993 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.766078+0000", 2024-09-21T14:08:16.994 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3850000000000006e-05 2024-09-21T14:08:16.994 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.994 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.994 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:08:16.994 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.766089+0000", 2024-09-21T14:08:16.994 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.119e-05 2024-09-21T14:08:16.994 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.994 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.994 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:16.994 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.629790+0000", 2024-09-21T14:08:16.994 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.86370065100000004 2024-09-21T14:08:16.994 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.994 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.995 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:08:16.995 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.629921+0000", 2024-09-21T14:08:16.995 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000131521 2024-09-21T14:08:16.995 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.995 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.995 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,7", 2024-09-21T14:08:16.995 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.630025+0000", 2024-09-21T14:08:16.995 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000103646 2024-09-21T14:08:16.995 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.995 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.995 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:08:16.995 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.638466+0000", 2024-09-21T14:08:16.996 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:08:16.997 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0084406240000000007 2024-09-21T14:08:16.997 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.997 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.997 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:16.997 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.638521+0000", 2024-09-21T14:08:16.997 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5426000000000001e-05 2024-09-21T14:08:16.997 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.997 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.997 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:16.997 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.651885+0000", 2024-09-21T14:08:16.998 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.013364407 2024-09-21T14:08:16.998 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.998 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.998 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:08:16.998 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.651942+0000", 2024-09-21T14:08:16.998 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6944000000000002e-05 2024-09-21T14:08:16.998 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.998 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.998 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:08:16.998 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.651998+0000", 2024-09-21T14:08:16.998 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5689000000000002e-05 2024-09-21T14:08:16.998 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:16.999 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:16.999 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:16.999 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:16.999 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:16.999 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:148382 10.8 10:17a8a665:::benchmark_data_smithi049_176811_object20805:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-09-21T14:08:16.999 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:04:55.771284+0000", 2024-09-21T14:08:16.999 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 201.125011395, 2024-09-21T14:08:16.999 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.88084546799999996, 2024-09-21T14:08:16.999 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:08:16.999 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:08:16.999 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:08:16.999 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:08:16.999 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:08:17.000 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:08:17.000 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 148382 2024-09-21T14:08:17.000 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.000 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:08:17.000 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.000 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:08:17.000 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.771284+0000", 2024-09-21T14:08:17.000 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:17.000 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.000 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.000 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:08:17.000 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.771284+0000", 2024-09-21T14:08:17.000 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:17.000 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.001 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.001 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:08:17.001 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.771286+0000", 2024-09-21T14:08:17.001 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.849e-06 2024-09-21T14:08:17.001 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.001 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.001 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:08:17.001 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.771291+0000", 2024-09-21T14:08:17.001 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5729999999999998e-06 2024-09-21T14:08:17.001 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.001 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.001 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:08:17.001 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.771308+0000", 2024-09-21T14:08:17.001 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6929999999999999e-05 2024-09-21T14:08:17.001 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.002 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.002 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:08:17.002 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.771316+0000", 2024-09-21T14:08:17.002 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5950000000000001e-06 2024-09-21T14:08:17.002 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.002 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.002 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:17.002 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.771378+0000", 2024-09-21T14:08:17.002 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2574000000000001e-05 2024-09-21T14:08:17.002 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.002 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.003 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:08:17.003 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.771388+0000", 2024-09-21T14:08:17.003 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0039e-05 2024-09-21T14:08:17.003 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.003 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.003 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:17.003 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.632299+0000", 2024-09-21T14:08:17.003 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.860910587 2024-09-21T14:08:17.003 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.003 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.004 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:08:17.004 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.632403+0000", 2024-09-21T14:08:17.004 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000103688 2024-09-21T14:08:17.004 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.004 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.004 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,7", 2024-09-21T14:08:17.004 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.632833+0000", 2024-09-21T14:08:17.004 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00042987599999999999 2024-09-21T14:08:17.004 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.004 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.004 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:08:17.004 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.638471+0000", 2024-09-21T14:08:17.004 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0056379120000000001 2024-09-21T14:08:17.004 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.004 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.005 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:17.005 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.638556+0000", 2024-09-21T14:08:17.005 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5357000000000003e-05 2024-09-21T14:08:17.005 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.005 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.005 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:17.005 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.652041+0000", 2024-09-21T14:08:17.005 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.01348482 2024-09-21T14:08:17.005 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.005 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.005 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:08:17.005 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.652091+0000", 2024-09-21T14:08:17.005 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0151999999999997e-05 2024-09-21T14:08:17.005 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.006 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.006 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:08:17.006 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.652129+0000", 2024-09-21T14:08:17.006 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8516000000000002e-05 2024-09-21T14:08:17.006 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:17.006 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:17.006 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:17.006 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.006 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.006 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:148383 10.8 10:1452c5c4:::benchmark_data_smithi049_176811_object20806:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-09-21T14:08:17.006 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:04:55.771736+0000", 2024-09-21T14:08:17.006 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 201.124559369, 2024-09-21T14:08:17.006 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.88066859799999997, 2024-09-21T14:08:17.006 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:08:17.007 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:08:17.007 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:08:17.007 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:08:17.007 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:08:17.007 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:08:17.007 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 148383 2024-09-21T14:08:17.007 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.007 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:08:17.007 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.007 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:08:17.007 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.771736+0000", 2024-09-21T14:08:17.007 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:17.007 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.007 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.008 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:08:17.008 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.771736+0000", 2024-09-21T14:08:17.008 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:17.008 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.008 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.008 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:08:17.008 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.771737+0000", 2024-09-21T14:08:17.008 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3099999999999996e-07 2024-09-21T14:08:17.008 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.008 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.008 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:08:17.008 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.771741+0000", 2024-09-21T14:08:17.008 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.019e-06 2024-09-21T14:08:17.008 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.009 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.009 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:08:17.009 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.771750+0000", 2024-09-21T14:08:17.009 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4550000000000001e-06 2024-09-21T14:08:17.009 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.009 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.009 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:08:17.009 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.771756+0000", 2024-09-21T14:08:17.009 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2379999999999997e-06 2024-09-21T14:08:17.009 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.009 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.009 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:17.009 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.771868+0000", 2024-09-21T14:08:17.009 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000112321 2024-09-21T14:08:17.009 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.010 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.010 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:08:17.010 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.771878+0000", 2024-09-21T14:08:17.010 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7240000000000004e-06 2024-09-21T14:08:17.010 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.010 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.010 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:17.010 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.635580+0000", 2024-09-21T14:08:17.010 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.86370200600000002 2024-09-21T14:08:17.010 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.010 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.010 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:08:17.010 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.635677+0000", 2024-09-21T14:08:17.010 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7850999999999995e-05 2024-09-21T14:08:17.011 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.011 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.011 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,7", 2024-09-21T14:08:17.011 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.635805+0000", 2024-09-21T14:08:17.011 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000127856 2024-09-21T14:08:17.011 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.011 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.011 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:17.011 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.641202+0000", 2024-09-21T14:08:17.011 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0053963860000000004 2024-09-21T14:08:17.011 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.011 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.012 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:08:17.012 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.644478+0000", 2024-09-21T14:08:17.012 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0032760039999999999 2024-09-21T14:08:17.012 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.012 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.012 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:17.012 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.652260+0000", 2024-09-21T14:08:17.012 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0077823520000000002 2024-09-21T14:08:17.012 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.012 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.012 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:08:17.012 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.652367+0000", 2024-09-21T14:08:17.012 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000107144 2024-09-21T14:08:17.012 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.013 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.013 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:08:17.013 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.652405+0000", 2024-09-21T14:08:17.013 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7311000000000002e-05 2024-09-21T14:08:17.013 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:17.013 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:17.013 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:17.013 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.013 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.014 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:148397 10.8 10:15efa11b:::benchmark_data_smithi049_176811_object20820:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-09-21T14:08:17.014 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:04:55.795257+0000", 2024-09-21T14:08:17.014 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 201.10103793499999, 2024-09-21T14:08:17.014 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.85728244799999997, 2024-09-21T14:08:17.014 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:08:17.014 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:08:17.014 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:08:17.014 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:08:17.014 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:08:17.014 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:08:17.014 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 148397 2024-09-21T14:08:17.015 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.015 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:08:17.015 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.015 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:08:17.015 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.795257+0000", 2024-09-21T14:08:17.015 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:17.015 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.015 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.015 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:08:17.015 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.795257+0000", 2024-09-21T14:08:17.015 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:17.015 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.015 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.015 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:08:17.015 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.795259+0000", 2024-09-21T14:08:17.016 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6780000000000001e-06 2024-09-21T14:08:17.016 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.016 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.016 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:08:17.016 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.795265+0000", 2024-09-21T14:08:17.016 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1850000000000001e-06 2024-09-21T14:08:17.016 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.016 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.016 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:08:17.016 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.795279+0000", 2024-09-21T14:08:17.016 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3546999999999999e-05 2024-09-21T14:08:17.016 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.016 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.016 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:08:17.016 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.795287+0000", 2024-09-21T14:08:17.017 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4740000000000005e-06 2024-09-21T14:08:17.017 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.017 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.017 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:17.017 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.795357+0000", 2024-09-21T14:08:17.017 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9783000000000006e-05 2024-09-21T14:08:17.017 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.017 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.017 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:08:17.017 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.795367+0000", 2024-09-21T14:08:17.017 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0434999999999999e-05 2024-09-21T14:08:17.017 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.017 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.017 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:17.018 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.638583+0000", 2024-09-21T14:08:17.018 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.84321592499999998 2024-09-21T14:08:17.018 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.018 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.018 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:08:17.018 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.638685+0000", 2024-09-21T14:08:17.018 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000101498 2024-09-21T14:08:17.018 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.018 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.018 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,7", 2024-09-21T14:08:17.018 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.638799+0000", 2024-09-21T14:08:17.018 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000113959 2024-09-21T14:08:17.018 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.018 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.019 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:08:17.019 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.644483+0000", 2024-09-21T14:08:17.019 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0056838629999999999 2024-09-21T14:08:17.019 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.019 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.019 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:17.019 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.645051+0000", 2024-09-21T14:08:17.019 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00056823599999999996 2024-09-21T14:08:17.019 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.019 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.019 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:17.019 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.652450+0000", 2024-09-21T14:08:17.019 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0073992889999999999 2024-09-21T14:08:17.019 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.019 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.020 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:08:17.020 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.652504+0000", 2024-09-21T14:08:17.020 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4051999999999997e-05 2024-09-21T14:08:17.020 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.020 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.020 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:08:17.020 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.652540+0000", 2024-09-21T14:08:17.020 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5524000000000001e-05 2024-09-21T14:08:17.020 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:17.020 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:17.020 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:17.020 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.020 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.021 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:148428 10.8 10:161326c5:::benchmark_data_smithi049_176811_object20851:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-09-21T14:08:17.021 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:04:56.144786+0000", 2024-09-21T14:08:17.021 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 200.75150885299999, 2024-09-21T14:08:17.021 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.50787033299999995, 2024-09-21T14:08:17.021 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:08:17.021 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:08:17.021 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:08:17.021 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:08:17.021 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:08:17.021 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:08:17.021 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 148428 2024-09-21T14:08:17.021 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.021 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:08:17.021 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.022 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:08:17.022 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.144786+0000", 2024-09-21T14:08:17.022 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:17.022 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.022 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.022 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:08:17.022 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.144786+0000", 2024-09-21T14:08:17.022 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:17.022 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.022 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.022 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:08:17.022 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.144788+0000", 2024-09-21T14:08:17.022 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.297e-06 2024-09-21T14:08:17.023 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.023 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.023 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:08:17.023 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.144793+0000", 2024-09-21T14:08:17.023 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.989e-06 2024-09-21T14:08:17.023 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.023 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.023 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:08:17.023 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.144805+0000", 2024-09-21T14:08:17.023 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2544000000000001e-05 2024-09-21T14:08:17.023 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.023 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.023 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:08:17.024 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.144811+0000", 2024-09-21T14:08:17.024 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6409999999999999e-06 2024-09-21T14:08:17.024 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.024 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.024 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:17.024 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.144880+0000", 2024-09-21T14:08:17.024 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9581999999999999e-05 2024-09-21T14:08:17.024 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.024 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.024 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:08:17.025 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.144894+0000", 2024-09-21T14:08:17.025 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.349e-05 2024-09-21T14:08:17.025 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.025 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.025 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:17.025 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.641231+0000", 2024-09-21T14:08:17.025 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.49633749599999999 2024-09-21T14:08:17.025 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.025 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.025 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:08:17.025 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.641337+0000", 2024-09-21T14:08:17.025 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000105263 2024-09-21T14:08:17.025 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.025 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.025 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,7", 2024-09-21T14:08:17.026 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.641438+0000", 2024-09-21T14:08:17.026 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010134899999999999 2024-09-21T14:08:17.026 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.026 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.026 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:17.026 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.645183+0000", 2024-09-21T14:08:17.026 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0037453109999999999 2024-09-21T14:08:17.026 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.026 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.026 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:08:17.026 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.651845+0000", 2024-09-21T14:08:17.026 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0066612399999999997 2024-09-21T14:08:17.026 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.026 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.026 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:17.027 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.652580+0000", 2024-09-21T14:08:17.027 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00073548299999999999 2024-09-21T14:08:17.027 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.027 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.027 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:08:17.027 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.652623+0000", 2024-09-21T14:08:17.027 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3241999999999997e-05 2024-09-21T14:08:17.027 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.027 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.027 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:08:17.027 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.652657+0000", 2024-09-21T14:08:17.027 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3405999999999998e-05 2024-09-21T14:08:17.028 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:17.028 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:17.028 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:17.028 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.028 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.028 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:114620 11.2 11:476bf97d:::benchmark_data_smithi049_206132_object114619:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e496)", 2024-09-21T14:08:17.028 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:08:13.397846+0000", 2024-09-21T14:08:17.028 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 3.498448931, 2024-09-21T14:08:17.028 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.072462561, 2024-09-21T14:08:17.028 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:08:17.028 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:08:17.028 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T14:08:17.028 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:08:17.028 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:08:17.028 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:08:17.029 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 114620 2024-09-21T14:08:17.029 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.029 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:08:17.029 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.029 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:08:17.029 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.397846+0000", 2024-09-21T14:08:17.029 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:17.029 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.029 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.029 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:08:17.029 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.397846+0000", 2024-09-21T14:08:17.029 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:17.029 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.030 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.030 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:08:17.030 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.397850+0000", 2024-09-21T14:08:17.030 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3340000000000002e-06 2024-09-21T14:08:17.030 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.030 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.030 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:08:17.030 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.397885+0000", 2024-09-21T14:08:17.030 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5642999999999997e-05 2024-09-21T14:08:17.030 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.030 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.030 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:08:17.030 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.397911+0000", 2024-09-21T14:08:17.030 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6035000000000001e-05 2024-09-21T14:08:17.030 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.031 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.031 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:08:17.031 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.397924+0000", 2024-09-21T14:08:17.031 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2772000000000001e-05 2024-09-21T14:08:17.031 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.031 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.031 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:17.031 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.398113+0000", 2024-09-21T14:08:17.031 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00018851499999999999 2024-09-21T14:08:17.031 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.031 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.031 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:08:17.031 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.398128+0000", 2024-09-21T14:08:17.031 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5727999999999999e-05 2024-09-21T14:08:17.032 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.032 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.032 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:17.032 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:14.392816+0000", 2024-09-21T14:08:17.032 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.99468794699999996 2024-09-21T14:08:17.032 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.032 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.032 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:08:17.032 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:14.392829+0000", 2024-09-21T14:08:17.032 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3132999999999999e-05 2024-09-21T14:08:17.032 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.032 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.032 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:17.032 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.469000+0000", 2024-09-21T14:08:17.032 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0761709930000001 2024-09-21T14:08:17.033 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.033 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.033 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:08:17.033 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.470309+0000", 2024-09-21T14:08:17.033 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0013084609999999999 2024-09-21T14:08:17.033 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:17.033 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:17.033 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:17.033 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.033 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.033 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:114639 11.2 11:40d27929:::benchmark_data_smithi049_206132_object114638:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e496)", 2024-09-21T14:08:17.033 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:08:13.414322+0000", 2024-09-21T14:08:17.033 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 3.4819727930000002, 2024-09-21T14:08:17.033 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0772069179999999, 2024-09-21T14:08:17.034 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:08:17.034 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:08:17.034 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:08:17.034 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:08:17.034 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:08:17.034 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:08:17.034 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 114639 2024-09-21T14:08:17.034 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.034 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:08:17.034 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.035 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:08:17.035 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.414322+0000", 2024-09-21T14:08:17.035 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:17.035 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.035 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.035 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:08:17.035 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.414322+0000", 2024-09-21T14:08:17.035 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:17.035 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.035 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.035 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:08:17.035 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.414325+0000", 2024-09-21T14:08:17.035 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.137e-06 2024-09-21T14:08:17.036 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.036 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.036 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:08:17.036 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.414357+0000", 2024-09-21T14:08:17.036 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2638000000000002e-05 2024-09-21T14:08:17.036 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.036 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.036 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:08:17.036 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.414380+0000", 2024-09-21T14:08:17.036 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2813999999999999e-05 2024-09-21T14:08:17.036 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.036 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.036 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:08:17.036 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.414391+0000", 2024-09-21T14:08:17.037 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0817e-05 2024-09-21T14:08:17.037 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.037 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.037 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:17.037 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.416468+0000", 2024-09-21T14:08:17.037 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0020774190000000001 2024-09-21T14:08:17.037 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.037 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.037 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:08:17.037 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.416508+0000", 2024-09-21T14:08:17.037 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9650999999999997e-05 2024-09-21T14:08:17.037 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.037 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.038 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:17.038 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:14.392856+0000", 2024-09-21T14:08:17.038 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.976347721 2024-09-21T14:08:17.038 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.038 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.038 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:08:17.038 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:14.392864+0000", 2024-09-21T14:08:17.038 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8720000000000002e-06 2024-09-21T14:08:17.038 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.038 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.038 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:17.038 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.470610+0000", 2024-09-21T14:08:17.038 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0777461859999999 2024-09-21T14:08:17.039 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.039 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.039 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:08:17.039 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.471012+0000", 2024-09-21T14:08:17.039 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00040198799999999999 2024-09-21T14:08:17.039 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.039 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.039 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,3,7", 2024-09-21T14:08:17.039 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.471118+0000", 2024-09-21T14:08:17.039 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000106474 2024-09-21T14:08:17.040 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.040 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.041 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:17.041 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.476674+0000", 2024-09-21T14:08:17.041 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0055553030000000002 2024-09-21T14:08:17.041 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.041 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.041 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:08:17.042 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.480688+0000", 2024-09-21T14:08:17.042 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0040147519999999999 2024-09-21T14:08:17.042 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.042 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.042 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:17.042 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.480737+0000", 2024-09-21T14:08:17.042 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9024999999999999e-05 2024-09-21T14:08:17.043 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.043 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.043 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:17.043 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.491326+0000", 2024-09-21T14:08:17.043 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.010588444000000001 2024-09-21T14:08:17.043 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.043 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.044 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:08:17.044 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.491396+0000", 2024-09-21T14:08:17.044 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9957999999999997e-05 2024-09-21T14:08:17.044 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.044 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.044 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:08:17.045 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.491529+0000", 2024-09-21T14:08:17.045 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000133719 2024-09-21T14:08:17.045 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:17.045 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:17.045 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:17.045 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.045 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.046 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:114643 11.2 11:4163170e:::benchmark_data_smithi049_206132_object114642:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e496)", 2024-09-21T14:08:17.046 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:08:13.424426+0000", 2024-09-21T14:08:17.046 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 3.4718693190000001, 2024-09-21T14:08:17.046 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0720125490000001, 2024-09-21T14:08:17.046 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:08:17.046 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:08:17.047 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:08:17.047 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:08:17.047 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:08:17.047 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:08:17.047 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 114643 2024-09-21T14:08:17.047 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.048 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:08:17.048 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.048 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:08:17.048 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.424426+0000", 2024-09-21T14:08:17.048 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:17.048 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.049 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.049 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:08:17.049 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.424426+0000", 2024-09-21T14:08:17.049 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:17.049 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.049 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.049 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:08:17.050 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.424428+0000", 2024-09-21T14:08:17.050 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9300000000000002e-06 2024-09-21T14:08:17.050 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.050 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.050 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:08:17.050 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.424454+0000", 2024-09-21T14:08:17.051 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6415999999999999e-05 2024-09-21T14:08:17.051 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.051 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.051 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:08:17.051 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.424468+0000", 2024-09-21T14:08:17.051 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4059e-05 2024-09-21T14:08:17.051 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.052 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.052 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:08:17.052 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.424477+0000", 2024-09-21T14:08:17.052 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.789e-06 2024-09-21T14:08:17.052 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.052 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.052 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:17.053 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.424745+0000", 2024-09-21T14:08:17.053 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00026768400000000002 2024-09-21T14:08:17.053 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.053 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.053 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:08:17.053 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.424757+0000", 2024-09-21T14:08:17.054 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1827999999999999e-05 2024-09-21T14:08:17.054 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.054 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.054 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:17.054 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:14.392888+0000", 2024-09-21T14:08:17.054 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.96813083499999997 2024-09-21T14:08:17.054 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.055 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.055 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:08:17.055 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:14.392895+0000", 2024-09-21T14:08:17.055 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8019999999999992e-06 2024-09-21T14:08:17.055 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.055 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.056 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:17.056 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.473705+0000", 2024-09-21T14:08:17.056 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0808099389999999 2024-09-21T14:08:17.056 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.056 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.056 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:08:17.056 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.473854+0000", 2024-09-21T14:08:17.057 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014908 2024-09-21T14:08:17.057 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.057 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.057 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,3,7", 2024-09-21T14:08:17.057 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.474013+0000", 2024-09-21T14:08:17.057 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000158448 2024-09-21T14:08:17.058 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.058 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.058 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:17.058 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.476717+0000", 2024-09-21T14:08:17.058 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0027038740000000002 2024-09-21T14:08:17.058 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.058 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.059 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:08:17.059 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.480695+0000", 2024-09-21T14:08:17.059 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0039782050000000003 2024-09-21T14:08:17.059 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.059 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.059 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:17.060 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.491683+0000", 2024-09-21T14:08:17.060 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.010987702 2024-09-21T14:08:17.060 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.060 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.060 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:17.060 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.496198+0000", 2024-09-21T14:08:17.060 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0045154449999999999 2024-09-21T14:08:17.061 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.061 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.061 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:08:17.061 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.496322+0000", 2024-09-21T14:08:17.061 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012440899999999999 2024-09-21T14:08:17.061 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.062 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.062 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:08:17.062 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.496439+0000", 2024-09-21T14:08:17.062 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000116104 2024-09-21T14:08:17.062 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:17.062 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:17.063 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:17.063 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.063 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.063 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:114648 11.2 11:4b2f3ec4:::benchmark_data_smithi049_206132_object114647:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e496)", 2024-09-21T14:08:17.063 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:08:13.432129+0000", 2024-09-21T14:08:17.063 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 3.4641658959999999, 2024-09-21T14:08:17.063 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0645723679999999, 2024-09-21T14:08:17.064 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:08:17.064 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:08:17.064 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:08:17.064 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:08:17.064 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:08:17.064 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:08:17.065 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 114648 2024-09-21T14:08:17.065 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.065 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:08:17.065 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.065 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:08:17.065 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.432129+0000", 2024-09-21T14:08:17.065 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:17.066 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.066 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.066 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:08:17.066 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.432129+0000", 2024-09-21T14:08:17.066 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:17.066 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.067 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.067 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:08:17.067 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.432132+0000", 2024-09-21T14:08:17.067 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2369999999999999e-06 2024-09-21T14:08:17.067 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.067 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.067 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:08:17.068 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.432159+0000", 2024-09-21T14:08:17.068 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6917e-05 2024-09-21T14:08:17.068 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.068 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.068 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:08:17.068 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.432183+0000", 2024-09-21T14:08:17.069 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4331000000000001e-05 2024-09-21T14:08:17.069 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.069 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.069 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:08:17.069 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.432193+0000", 2024-09-21T14:08:17.069 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0302999999999999e-05 2024-09-21T14:08:17.069 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.070 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.070 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:17.070 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.433016+0000", 2024-09-21T14:08:17.070 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000823189 2024-09-21T14:08:17.070 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.070 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.071 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:08:17.071 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.433070+0000", 2024-09-21T14:08:17.071 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3266000000000001e-05 2024-09-21T14:08:17.071 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.071 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.071 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:17.072 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:14.392920+0000", 2024-09-21T14:08:17.072 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.95985063800000003 2024-09-21T14:08:17.072 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.072 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.072 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:08:17.072 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:14.392928+0000", 2024-09-21T14:08:17.072 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6120000000000001e-06 2024-09-21T14:08:17.073 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.073 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.073 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:17.073 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.476748+0000", 2024-09-21T14:08:17.073 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.083820094 2024-09-21T14:08:17.073 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.074 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.074 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:08:17.074 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.476929+0000", 2024-09-21T14:08:17.074 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000181435 2024-09-21T14:08:17.074 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.074 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.074 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,3,7", 2024-09-21T14:08:17.075 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.477085+0000", 2024-09-21T14:08:17.075 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00015513899999999999 2024-09-21T14:08:17.075 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.075 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.075 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:08:17.076 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.480749+0000", 2024-09-21T14:08:17.076 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0036649040000000001 2024-09-21T14:08:17.076 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.076 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.076 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:17.076 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.491737+0000", 2024-09-21T14:08:17.076 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.010987215 2024-09-21T14:08:17.077 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.077 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.077 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:17.077 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.491777+0000", 2024-09-21T14:08:17.077 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9966000000000003e-05 2024-09-21T14:08:17.077 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.077 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.078 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:17.078 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.496480+0000", 2024-09-21T14:08:17.078 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0047029139999999999 2024-09-21T14:08:17.078 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.078 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.078 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:08:17.078 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.496583+0000", 2024-09-21T14:08:17.079 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000103933 2024-09-21T14:08:17.079 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:17.079 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:17.079 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:08:17.079 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.496702+0000", 2024-09-21T14:08:17.079 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000118275 2024-09-21T14:08:17.079 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:17.079 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:17.080 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:17.080 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:17.080 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:17.080 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T14:08:17.084 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:17.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:17.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:08:17.186 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:17.221 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:17.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:17.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:17.322 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:17.357 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:17.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:17.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:17.458 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:08:17.490 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:17.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:17.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:08:17.520 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 84 16 116932 116916 86.9738 71.125 0.00678168 0.0114933 2024-09-21T14:08:17.592 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:08:17.626 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:17.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:17.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:08:17.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:17 smithi187 ceph-mon[33032]: pgmap v991: 168 pgs: 2 remapped+peering, 2 unknown, 13 peering, 151 active+clean; 7.0 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s; 94 KiB/s, 4 objects/s recovering 2024-09-21T14:08:17.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:17 smithi187 ceph-mon[33032]: 1.4 scrub starts 2024-09-21T14:08:17.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:17 smithi187 ceph-mon[33032]: 1.4 scrub ok 2024-09-21T14:08:17.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:17 smithi187 ceph-mon[33032]: 5.1a scrub starts 2024-09-21T14:08:17.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:17 smithi187 ceph-mon[33032]: 5.1a scrub ok 2024-09-21T14:08:17.727 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:08:17.737 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:17 smithi049 ceph-mon[30080]: pgmap v991: 168 pgs: 2 remapped+peering, 2 unknown, 13 peering, 151 active+clean; 7.0 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s; 94 KiB/s, 4 objects/s recovering 2024-09-21T14:08:17.738 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:17 smithi049 ceph-mon[30080]: 1.4 scrub starts 2024-09-21T14:08:17.738 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:17 smithi049 ceph-mon[30080]: 1.4 scrub ok 2024-09-21T14:08:17.738 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:17 smithi049 ceph-mon[30080]: 5.1a scrub starts 2024-09-21T14:08:17.738 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:17 smithi049 ceph-mon[30080]: 5.1a scrub ok 2024-09-21T14:08:17.738 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:17 smithi049 ceph-mon[40489]: pgmap v991: 168 pgs: 2 remapped+peering, 2 unknown, 13 peering, 151 active+clean; 7.0 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s; 94 KiB/s, 4 objects/s recovering 2024-09-21T14:08:17.738 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:17 smithi049 ceph-mon[40489]: 1.4 scrub starts 2024-09-21T14:08:17.738 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:17 smithi049 ceph-mon[40489]: 1.4 scrub ok 2024-09-21T14:08:17.738 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:17 smithi049 ceph-mon[40489]: 5.1a scrub starts 2024-09-21T14:08:17.738 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:17 smithi049 ceph-mon[40489]: 5.1a scrub ok 2024-09-21T14:08:17.771 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T14:08:17.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:17.773 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:17.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:08:17.874 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:17.924 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:17.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:17.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:18.026 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:08:18.082 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:18.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:18.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:08:18.183 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:08:18.297 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:18.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:18.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:08:18.399 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:18.434 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:18.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:18.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:18.520 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 85 16 117940 117924 86.6917 63 0.00934224 0.0115303 2024-09-21T14:08:18.536 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:08:18.582 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:18.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:18.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:08:18.684 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:08:18.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:18 smithi187 ceph-mon[33032]: 3.1b scrub starts 2024-09-21T14:08:18.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:18 smithi187 ceph-mon[33032]: 3.1b scrub ok 2024-09-21T14:08:18.717 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:18.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:18.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:08:18.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:18 smithi049 ceph-mon[30080]: 3.1b scrub starts 2024-09-21T14:08:18.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:18 smithi049 ceph-mon[30080]: 3.1b scrub ok 2024-09-21T14:08:18.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:18 smithi049 ceph-mon[40489]: 3.1b scrub starts 2024-09-21T14:08:18.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:18 smithi049 ceph-mon[40489]: 3.1b scrub ok 2024-09-21T14:08:18.819 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:18.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:18.857 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:18.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:18.959 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:08:18.989 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:18.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:18.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:08:19.090 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:19.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:19.130 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:19.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:19.231 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:19.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:19.274 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:19.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:19.375 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:08:19.422 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:19.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:19.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:08:19.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:19 smithi049 ceph-mon[40489]: pgmap v992: 168 pgs: 2 remapped+peering, 2 unknown, 13 peering, 151 active+clean; 7.0 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 38 MiB/s wr, 614 op/s; 90 KiB/s, 3 objects/s recovering 2024-09-21T14:08:19.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:19 smithi049 ceph-mon[40489]: 4.11 scrub starts 2024-09-21T14:08:19.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:19 smithi049 ceph-mon[40489]: 4.11 scrub ok 2024-09-21T14:08:19.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:19 smithi049 ceph-mon[30080]: pgmap v992: 168 pgs: 2 remapped+peering, 2 unknown, 13 peering, 151 active+clean; 7.0 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 38 MiB/s wr, 614 op/s; 90 KiB/s, 3 objects/s recovering 2024-09-21T14:08:19.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:19 smithi049 ceph-mon[30080]: 4.11 scrub starts 2024-09-21T14:08:19.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:19 smithi049 ceph-mon[30080]: 4.11 scrub ok 2024-09-21T14:08:19.520 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 86 16 119095 119079 86.523 72.1875 0.00964414 0.0115528 2024-09-21T14:08:19.524 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:08:19.567 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:19.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:19.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:08:19.669 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:08:19.707 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:19.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:19.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:08:19.809 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:19.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:19.845 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:19.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:19.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:19 smithi187 ceph-mon[33032]: pgmap v992: 168 pgs: 2 remapped+peering, 2 unknown, 13 peering, 151 active+clean; 7.0 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 38 MiB/s wr, 614 op/s; 90 KiB/s, 3 objects/s recovering 2024-09-21T14:08:19.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:19 smithi187 ceph-mon[33032]: 4.11 scrub starts 2024-09-21T14:08:19.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:19 smithi187 ceph-mon[33032]: 4.11 scrub ok 2024-09-21T14:08:19.946 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:19.967 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:19.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:19.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:20.069 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:08:20.104 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:20.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:20.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:08:20.206 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:08:20.238 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:20.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:20.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:08:20.282 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 4, 6, 2, 3, 7, 5] 2024-09-21T14:08:20.283 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T14:08:20.283 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-21T14:08:20.283 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd dump -f json-pretty 2024-09-21T14:08:20.340 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:20.396 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:20.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:20.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:20.445 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:08:20 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:08:20.468 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-09-21T14:08:20.499 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:08:20.520 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 87 15 120028 120013 86.1994 58.375 0.0105673 0.0115964 2024-09-21T14:08:20.534 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:20.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:20.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:08:20.579 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:20 smithi049 ceph-mon[40489]: 6.b scrub starts 2024-09-21T14:08:20.579 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:20 smithi049 ceph-mon[40489]: 6.b scrub ok 2024-09-21T14:08:20.579 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:20 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2406957944' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:08:20.579 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:20 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:08:20.579 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:20 smithi049 ceph-mon[40489]: 3.f scrub starts 2024-09-21T14:08:20.580 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:20 smithi049 ceph-mon[40489]: 3.f scrub ok 2024-09-21T14:08:20.580 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:20 smithi049 ceph-mon[30080]: 6.b scrub starts 2024-09-21T14:08:20.580 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:20 smithi049 ceph-mon[30080]: 6.b scrub ok 2024-09-21T14:08:20.580 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:20 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2406957944' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:08:20.580 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:20 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:08:20.580 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:20 smithi049 ceph-mon[30080]: 3.f scrub starts 2024-09-21T14:08:20.580 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:20 smithi049 ceph-mon[30080]: 3.f scrub ok 2024-09-21T14:08:20.637 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:08:20.670 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:20.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:20.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:08:20.772 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:08:20.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:20.820 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:20.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:08:20.921 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:20.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:20 smithi187 ceph-mon[33032]: 6.b scrub starts 2024-09-21T14:08:20.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:20 smithi187 ceph-mon[33032]: 6.b scrub ok 2024-09-21T14:08:20.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:20 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2406957944' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:08:20.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:20 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:08:20.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:20 smithi187 ceph-mon[33032]: 3.f scrub starts 2024-09-21T14:08:20.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:20 smithi187 ceph-mon[33032]: 3.f scrub ok 2024-09-21T14:08:20.953 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:20.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:20.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:21.004 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:08:20 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:08:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:08:21.055 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:08:21.096 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:21.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:21.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:08:21.197 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:08:21.240 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:21.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:21.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:08:21.342 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:08:21.399 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:21.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:21.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:08:21.501 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:21.520 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 88 16 121068 121052 85.9578 64.9375 0.0203829 0.0116291 2024-09-21T14:08:21.534 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:21.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:21.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:21.636 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:21.681 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:21.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:21.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:21.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:21 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:08:21.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:21 smithi049 ceph-mon[40489]: pgmap v993: 168 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 7.2 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 302 B/s rd, 56 MiB/s wr, 897 op/s; 34238/354414 objects misplaced (9.660%); 24 MiB/s, 384 objects/s recovering 2024-09-21T14:08:21.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:21 smithi049 ceph-mon[40489]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T14:08:21.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:21 smithi049 ceph-mon[40489]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 6 pgs peering) 2024-09-21T14:08:21.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:21 smithi049 ceph-mon[40489]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:08:21.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:21 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T14:08:21.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:21 smithi049 ceph-mon[40489]: osdmap e499: 8 total, 8 up, 8 in 2024-09-21T14:08:21.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:21 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:08:21.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:21 smithi049 ceph-mon[30080]: pgmap v993: 168 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 7.2 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 302 B/s rd, 56 MiB/s wr, 897 op/s; 34238/354414 objects misplaced (9.660%); 24 MiB/s, 384 objects/s recovering 2024-09-21T14:08:21.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:21 smithi049 ceph-mon[30080]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T14:08:21.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:21 smithi049 ceph-mon[30080]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 6 pgs peering) 2024-09-21T14:08:21.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:21 smithi049 ceph-mon[30080]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:08:21.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:21 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T14:08:21.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:21 smithi049 ceph-mon[30080]: osdmap e499: 8 total, 8 up, 8 in 2024-09-21T14:08:21.782 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:08:21.821 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:21.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:21.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:08:21.923 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:08:21.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:21 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:08:21.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:21 smithi187 ceph-mon[33032]: pgmap v993: 168 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 7.2 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 302 B/s rd, 56 MiB/s wr, 897 op/s; 34238/354414 objects misplaced (9.660%); 24 MiB/s, 384 objects/s recovering 2024-09-21T14:08:21.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:21 smithi187 ceph-mon[33032]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T14:08:21.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:21 smithi187 ceph-mon[33032]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 6 pgs peering) 2024-09-21T14:08:21.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:21 smithi187 ceph-mon[33032]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:08:21.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:21 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T14:08:21.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:21 smithi187 ceph-mon[33032]: osdmap e499: 8 total, 8 up, 8 in 2024-09-21T14:08:21.963 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:21.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:21.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:08:22.065 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:22.097 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:22.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:22.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:22.199 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:08:22.241 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:22.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:22.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:08:22.343 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:22.376 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:22.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:22.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:22.478 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:22.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:22.519 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:22.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:22.521 INFO:tasks.radosbench.radosbench.0.smithi049.stdout: 89 16 122262 122246 85.8304 74.625 0.0163753 0.0116459 2024-09-21T14:08:22.619 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:08:22.650 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:22.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:22.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:08:22.752 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:08:22.788 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:22.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:22.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:08:22.889 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:08:22.934 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:22.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:22.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:08:23.001 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T14:08:23.036 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:08:23.112 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:23.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:23.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:08:23.213 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:08:23.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:23.269 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:23.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:08:23.370 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:23.400 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:23.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:23.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:23.502 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:08:23.520 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Total time run: 90.0072 2024-09-21T14:08:23.520 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Total writes made: 123376 2024-09-21T14:08:23.521 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Write size: 65536 2024-09-21T14:08:23.521 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Object size: 65536 2024-09-21T14:08:23.521 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Bandwidth (MB/sec): 85.671 2024-09-21T14:08:23.521 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Stddev Bandwidth: 31.2389 2024-09-21T14:08:23.521 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Max bandwidth (MB/sec): 190.812 2024-09-21T14:08:23.521 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Min bandwidth (MB/sec): 0 2024-09-21T14:08:23.521 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Average IOPS: 1370 2024-09-21T14:08:23.521 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Stddev IOPS: 499.837 2024-09-21T14:08:23.521 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Max IOPS: 3053 2024-09-21T14:08:23.521 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Min IOPS: 0 2024-09-21T14:08:23.521 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Average Latency(s): 0.0116684 2024-09-21T14:08:23.521 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Stddev Latency(s): 0.0274702 2024-09-21T14:08:23.521 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Max latency(s): 2.10904 2024-09-21T14:08:23.521 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Min latency(s): 0.00119458 2024-09-21T14:08:23.525 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Cleaning up (deleting benchmark objects) 2024-09-21T14:08:23.531 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:23.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:23.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:08:23.607 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:08:23.633 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:08:23.664 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:08:23.699 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:23.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:23.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:08:23.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:23 smithi049 ceph-mon[40489]: pgmap v995: 168 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 7.3 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 0 B/s rd, 50 MiB/s wr, 800 op/s; 34238/358392 objects misplaced (9.553%); 21 MiB/s, 334 objects/s recovering 2024-09-21T14:08:23.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:23 smithi049 ceph-mon[30080]: pgmap v995: 168 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 7.3 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 0 B/s rd, 50 MiB/s wr, 800 op/s; 34238/358392 objects misplaced (9.553%); 21 MiB/s, 334 objects/s recovering 2024-09-21T14:08:23.782 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:23 smithi187 ceph-mon[33032]: pgmap v995: 168 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 7.3 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 0 B/s rd, 50 MiB/s wr, 800 op/s; 34238/358392 objects misplaced (9.553%); 21 MiB/s, 334 objects/s recovering 2024-09-21T14:08:23.801 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:23.836 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:23.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:23.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:23.938 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:08:24.001 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:24.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:24.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:08:24.103 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:08:24.149 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:24.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:24.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:08:24.251 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:08:24.294 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:24.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:24.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:08:24.396 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:08:24.439 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:24.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:24.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:08:24.516 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:08:24.541 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:08:24.594 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:24.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:24.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:08:24.696 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:08:24.724 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:24.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:24.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:08:24.826 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:24.842 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:08:24.842 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:08:24.855 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:24.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:24.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:24.891 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:08:24.891 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:08:24.942 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:08:24.943 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:08:24.943 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T14:08:24.943 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T14:08:24.943 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 499, 2024-09-21T14:08:24.943 INFO:teuthology.orchestra.run.smithi049.stdout: "fsid": "40a3d2e8-781f-11ef-baf6-efdc52797490", 2024-09-21T14:08:24.943 INFO:teuthology.orchestra.run.smithi049.stdout: "created": "2024-09-21T13:42:57.838275+0000", 2024-09-21T14:08:24.944 INFO:teuthology.orchestra.run.smithi049.stdout: "modified": "2024-09-21T14:08:20.447518+0000", 2024-09-21T14:08:24.944 INFO:teuthology.orchestra.run.smithi049.stdout: "last_up_change": "2024-09-21T14:01:40.158454+0000", 2024-09-21T14:08:24.944 INFO:teuthology.orchestra.run.smithi049.stdout: "last_in_change": "2024-09-21T13:58:16.730861+0000", 2024-09-21T14:08:24.944 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-21T14:08:24.944 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_num": 5801984, 2024-09-21T14:08:24.944 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_set": [ 2024-09-21T14:08:24.944 INFO:teuthology.orchestra.run.smithi049.stdout: "noscrub", 2024-09-21T14:08:24.944 INFO:teuthology.orchestra.run.smithi049.stdout: "pglog_hardlimit", 2024-09-21T14:08:24.944 INFO:teuthology.orchestra.run.smithi049.stdout: "purged_snapdirs", 2024-09-21T14:08:24.944 INFO:teuthology.orchestra.run.smithi049.stdout: "recovery_deletes", 2024-09-21T14:08:24.944 INFO:teuthology.orchestra.run.smithi049.stdout: "sortbitwise" 2024-09-21T14:08:24.944 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T14:08:24.944 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_version": 22, 2024-09-21T14:08:24.944 INFO:teuthology.orchestra.run.smithi049.stdout: "full_ratio": 0.94999998807907104, 2024-09-21T14:08:24.945 INFO:teuthology.orchestra.run.smithi049.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-21T14:08:24.945 INFO:teuthology.orchestra.run.smithi049.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-21T14:08:24.945 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_snapshot": "", 2024-09-21T14:08:24.945 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_max": 11, 2024-09-21T14:08:24.945 INFO:teuthology.orchestra.run.smithi049.stdout: "max_osd": 8, 2024-09-21T14:08:24.945 INFO:teuthology.orchestra.run.smithi049.stdout: "require_min_compat_client": "luminous", 2024-09-21T14:08:24.945 INFO:teuthology.orchestra.run.smithi049.stdout: "min_compat_client": "luminous", 2024-09-21T14:08:24.945 INFO:teuthology.orchestra.run.smithi049.stdout: "require_osd_release": "squid", 2024-09-21T14:08:24.945 INFO:teuthology.orchestra.run.smithi049.stdout: "allow_crimson": false, 2024-09-21T14:08:24.945 INFO:teuthology.orchestra.run.smithi049.stdout: "pools": [ 2024-09-21T14:08:24.945 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:24.945 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 1, 2024-09-21T14:08:24.945 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": ".mgr", 2024-09-21T14:08:24.945 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:46:21.096244+0000", 2024-09-21T14:08:24.946 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T14:08:24.946 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T14:08:24.946 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T14:08:24.946 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T14:08:24.946 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T14:08:24.946 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T14:08:24.946 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T14:08:24.946 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T14:08:24.946 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T14:08:24.946 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T14:08:24.946 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T14:08:24.946 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T14:08:24.946 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T14:08:24.947 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 21, 2024-09-21T14:08:24.947 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 21, 2024-09-21T14:08:24.947 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 21, 2024-09-21T14:08:24.947 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 21, 2024-09-21T14:08:24.947 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 21, 2024-09-21T14:08:24.947 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T14:08:24.947 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T14:08:24.947 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T14:08:24.947 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T14:08:24.947 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T14:08:24.947 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T14:08:24.947 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T14:08:24.947 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:24.947 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "438", 2024-09-21T14:08:24.948 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T14:08:24.948 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T14:08:24.948 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "436", 2024-09-21T14:08:24.948 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T14:08:24.948 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T14:08:24.948 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T14:08:24.948 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T14:08:24.948 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T14:08:24.948 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T14:08:24.948 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T14:08:24.948 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T14:08:24.948 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T14:08:24.948 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T14:08:24.948 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T14:08:24.949 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T14:08:24.949 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T14:08:24.949 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T14:08:24.949 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T14:08:24.949 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T14:08:24.949 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T14:08:24.949 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T14:08:24.949 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T14:08:24.949 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T14:08:24.949 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T14:08:24.949 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T14:08:24.949 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T14:08:24.949 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:24.949 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T14:08:24.950 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T14:08:24.950 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T14:08:24.950 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T14:08:24.950 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T14:08:24.950 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T14:08:24.950 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T14:08:24.950 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T14:08:24.950 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T14:08:24.950 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T14:08:24.950 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T14:08:24.950 INFO:teuthology.orchestra.run.smithi049.stdout: "options": { 2024-09-21T14:08:24.950 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_max": 32, 2024-09-21T14:08:24.950 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_min": 1 2024-09-21T14:08:24.950 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:24.951 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T14:08:24.951 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr": {} 2024-09-21T14:08:24.951 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:24.951 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T14:08:24.951 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T14:08:24.951 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2.3399999141693115, 2024-09-21T14:08:24.951 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2.3399999141693115, 2024-09-21T14:08:24.951 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1.1399999856948853, 2024-09-21T14:08:24.951 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2.6700000762939453, 2024-09-21T14:08:24.951 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2.6700000762939453, 2024-09-21T14:08:24.951 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.85335731506347656, 2024-09-21T14:08:24.951 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.98000001907348633, 2024-09-21T14:08:24.951 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.85000002384185791 2024-09-21T14:08:24.951 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:24.952 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:24.952 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:24.952 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 2, 2024-09-21T14:08:24.952 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "datapool", 2024-09-21T14:08:24.952 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:48:55.540712+0000", 2024-09-21T14:08:24.952 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 8193, 2024-09-21T14:08:24.952 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T14:08:24.952 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T14:08:24.952 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T14:08:24.952 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T14:08:24.952 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T14:08:24.952 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T14:08:24.952 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T14:08:24.952 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T14:08:24.953 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T14:08:24.953 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T14:08:24.953 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T14:08:24.953 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T14:08:24.953 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 3, 2024-09-21T14:08:24.953 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 3, 2024-09-21T14:08:24.953 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 3, 2024-09-21T14:08:24.953 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 3, 2024-09-21T14:08:24.953 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 3, 2024-09-21T14:08:24.953 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T14:08:24.953 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T14:08:24.953 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T14:08:24.953 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T14:08:24.954 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T14:08:24.954 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T14:08:24.954 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T14:08:24.954 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:24.954 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "63", 2024-09-21T14:08:24.954 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T14:08:24.954 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T14:08:24.954 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T14:08:24.954 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T14:08:24.954 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T14:08:24.954 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 2, 2024-09-21T14:08:24.954 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 63, 2024-09-21T14:08:24.954 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T14:08:24.955 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T14:08:24.955 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T14:08:24.955 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T14:08:24.955 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T14:08:24.955 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T14:08:24.955 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T14:08:24.955 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T14:08:24.955 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T14:08:24.955 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T14:08:24.955 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T14:08:24.955 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T14:08:24.955 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T14:08:24.955 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T14:08:24.955 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T14:08:24.955 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T14:08:24.956 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T14:08:24.956 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T14:08:24.956 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T14:08:24.956 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:24.956 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T14:08:24.956 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T14:08:24.956 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T14:08:24.956 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T14:08:24.956 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T14:08:24.956 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T14:08:24.956 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T14:08:24.956 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T14:08:24.956 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T14:08:24.956 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T14:08:24.957 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T14:08:24.957 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T14:08:24.957 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T14:08:24.957 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd": {} 2024-09-21T14:08:24.957 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:24.957 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T14:08:24.957 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T14:08:24.957 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2.3299999237060547, 2024-09-21T14:08:24.957 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2.3299999237060547, 2024-09-21T14:08:24.957 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 0.86000001430511475, 2024-09-21T14:08:24.957 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2, 2024-09-21T14:08:24.957 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2, 2024-09-21T14:08:24.957 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.85335731506347656, 2024-09-21T14:08:24.957 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.98000001907348633, 2024-09-21T14:08:24.957 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.85000002384185791 2024-09-21T14:08:24.958 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:24.958 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:24.958 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:24.958 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 3, 2024-09-21T14:08:24.958 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": ".rgw.root", 2024-09-21T14:08:24.958 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:48:56.417456+0000", 2024-09-21T14:08:24.958 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T14:08:24.958 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T14:08:24.958 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T14:08:24.958 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T14:08:24.958 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T14:08:24.958 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T14:08:24.958 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T14:08:24.959 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T14:08:24.959 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T14:08:24.959 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T14:08:24.959 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T14:08:24.959 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T14:08:24.959 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T14:08:24.959 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 32, 2024-09-21T14:08:24.959 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 32, 2024-09-21T14:08:24.959 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 32, 2024-09-21T14:08:24.959 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 32, 2024-09-21T14:08:24.959 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 32, 2024-09-21T14:08:24.959 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T14:08:24.959 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:08:24.960 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T14:08:24.961 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T14:08:24.961 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T14:08:24.961 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T14:08:24.961 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T14:08:24.961 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T14:08:24.961 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:24.961 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "62", 2024-09-21T14:08:24.961 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T14:08:24.961 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T14:08:24.961 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T14:08:24.961 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T14:08:24.961 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T14:08:24.961 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T14:08:24.962 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T14:08:24.962 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T14:08:24.962 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T14:08:24.962 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T14:08:24.962 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T14:08:24.962 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T14:08:24.962 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T14:08:24.962 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T14:08:24.962 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T14:08:24.962 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T14:08:24.962 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T14:08:24.962 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T14:08:24.962 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T14:08:24.962 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T14:08:24.963 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T14:08:24.963 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T14:08:24.963 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T14:08:24.963 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T14:08:24.963 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T14:08:24.963 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T14:08:24.963 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:24.963 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T14:08:24.963 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T14:08:24.963 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T14:08:24.963 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T14:08:24.963 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T14:08:24.963 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T14:08:24.963 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T14:08:24.964 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T14:08:24.964 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T14:08:24.964 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T14:08:24.964 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T14:08:24.964 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T14:08:24.964 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T14:08:24.964 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw": {} 2024-09-21T14:08:24.964 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:24.964 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T14:08:24.964 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T14:08:24.964 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 1.75, 2024-09-21T14:08:24.964 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 1.75, 2024-09-21T14:08:24.964 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1.1399999856948853, 2024-09-21T14:08:24.964 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2, 2024-09-21T14:08:24.965 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2, 2024-09-21T14:08:24.965 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.85335731506347656, 2024-09-21T14:08:24.965 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.98000001907348633, 2024-09-21T14:08:24.965 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.85000002384185791 2024-09-21T14:08:24.965 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:24.965 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:24.965 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:24.965 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 4, 2024-09-21T14:08:24.965 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "default.rgw.log", 2024-09-21T14:08:24.965 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:48:58.128455+0000", 2024-09-21T14:08:24.965 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T14:08:24.965 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T14:08:24.965 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T14:08:24.965 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T14:08:24.965 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T14:08:24.966 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T14:08:24.966 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T14:08:24.966 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T14:08:24.966 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T14:08:24.966 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T14:08:24.966 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T14:08:24.966 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T14:08:24.966 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T14:08:24.966 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 32, 2024-09-21T14:08:24.966 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 32, 2024-09-21T14:08:24.966 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 32, 2024-09-21T14:08:24.966 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 32, 2024-09-21T14:08:24.966 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 32, 2024-09-21T14:08:24.966 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T14:08:24.967 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T14:08:24.967 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T14:08:24.967 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T14:08:24.967 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T14:08:24.967 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T14:08:24.967 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T14:08:24.967 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:24.967 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "64", 2024-09-21T14:08:24.967 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T14:08:24.967 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T14:08:24.967 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T14:08:24.967 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T14:08:24.967 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T14:08:24.967 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T14:08:24.968 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T14:08:24.968 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T14:08:24.968 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T14:08:24.968 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T14:08:24.968 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T14:08:24.968 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T14:08:24.968 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T14:08:24.968 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T14:08:24.968 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T14:08:24.968 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T14:08:24.968 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T14:08:24.968 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T14:08:24.968 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T14:08:24.968 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T14:08:24.969 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T14:08:24.969 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T14:08:24.969 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T14:08:24.969 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T14:08:24.969 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T14:08:24.969 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T14:08:24.969 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:24.969 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T14:08:24.969 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T14:08:24.969 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T14:08:24.969 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T14:08:24.969 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T14:08:24.969 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T14:08:24.969 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T14:08:24.969 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T14:08:24.970 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T14:08:24.970 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T14:08:24.970 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T14:08:24.970 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T14:08:24.970 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T14:08:24.970 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw": {} 2024-09-21T14:08:24.970 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:24.970 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T14:08:24.970 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T14:08:24.970 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 1.5399999618530273, 2024-09-21T14:08:24.970 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 1.5399999618530273, 2024-09-21T14:08:24.970 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1.1399999856948853, 2024-09-21T14:08:24.970 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 1.75, 2024-09-21T14:08:24.970 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 1.75, 2024-09-21T14:08:24.971 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.85335731506347656, 2024-09-21T14:08:24.971 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.98000001907348633, 2024-09-21T14:08:24.971 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.85000002384185791 2024-09-21T14:08:24.971 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:24.971 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:24.971 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:24.971 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 5, 2024-09-21T14:08:24.971 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "default.rgw.control", 2024-09-21T14:08:24.971 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:49:00.069361+0000", 2024-09-21T14:08:24.971 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T14:08:24.971 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T14:08:24.971 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T14:08:24.971 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T14:08:24.971 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T14:08:24.972 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T14:08:24.972 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T14:08:24.972 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T14:08:24.972 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T14:08:24.972 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T14:08:24.972 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T14:08:24.972 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T14:08:24.972 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T14:08:24.972 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 32, 2024-09-21T14:08:24.972 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 32, 2024-09-21T14:08:24.972 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 32, 2024-09-21T14:08:24.972 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 32, 2024-09-21T14:08:24.972 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 32, 2024-09-21T14:08:24.972 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T14:08:24.973 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T14:08:24.973 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T14:08:24.973 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T14:08:24.973 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T14:08:24.973 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T14:08:24.973 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T14:08:24.973 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:24.973 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "66", 2024-09-21T14:08:24.973 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T14:08:24.973 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T14:08:24.973 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T14:08:24.973 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T14:08:24.973 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T14:08:24.973 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T14:08:24.974 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T14:08:24.974 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T14:08:24.974 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T14:08:24.974 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T14:08:24.974 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T14:08:24.974 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T14:08:24.974 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T14:08:24.974 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T14:08:24.974 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T14:08:24.974 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T14:08:24.974 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T14:08:24.974 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T14:08:24.974 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T14:08:24.974 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T14:08:24.975 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T14:08:24.975 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T14:08:24.975 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T14:08:24.975 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T14:08:24.975 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T14:08:24.975 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T14:08:24.975 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:24.975 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T14:08:24.975 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T14:08:24.975 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T14:08:24.975 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T14:08:24.975 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T14:08:24.975 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T14:08:24.975 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T14:08:24.976 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T14:08:24.976 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T14:08:24.976 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T14:08:24.976 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T14:08:24.976 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T14:08:24.976 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T14:08:24.976 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw": {} 2024-09-21T14:08:24.976 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:24.976 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T14:08:24.976 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T14:08:24.976 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 1.5399999618530273, 2024-09-21T14:08:24.976 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 1.5399999618530273, 2024-09-21T14:08:24.976 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1.1399999856948853, 2024-09-21T14:08:24.976 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 1.75, 2024-09-21T14:08:24.977 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 1.75, 2024-09-21T14:08:24.977 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.85335731506347656, 2024-09-21T14:08:24.977 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.98000001907348633, 2024-09-21T14:08:24.977 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.85000002384185791 2024-09-21T14:08:24.977 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:24.977 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:24.977 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:24.977 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 6, 2024-09-21T14:08:24.977 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "default.rgw.meta", 2024-09-21T14:08:24.977 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:49:02.127399+0000", 2024-09-21T14:08:24.977 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T14:08:24.977 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T14:08:24.977 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T14:08:24.977 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T14:08:24.978 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T14:08:24.978 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T14:08:24.978 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T14:08:24.978 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T14:08:24.978 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T14:08:24.978 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T14:08:24.978 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T14:08:24.978 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T14:08:24.978 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T14:08:24.978 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 32, 2024-09-21T14:08:24.978 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 32, 2024-09-21T14:08:24.978 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 32, 2024-09-21T14:08:24.978 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 32, 2024-09-21T14:08:24.978 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 32, 2024-09-21T14:08:24.979 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T14:08:24.979 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T14:08:24.979 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T14:08:24.979 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T14:08:24.979 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T14:08:24.979 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T14:08:24.979 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T14:08:24.979 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:24.979 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "68", 2024-09-21T14:08:24.979 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T14:08:24.979 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T14:08:24.979 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T14:08:24.979 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T14:08:24.979 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T14:08:24.979 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T14:08:24.980 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T14:08:24.980 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T14:08:24.980 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T14:08:24.980 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T14:08:24.980 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T14:08:24.980 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T14:08:24.980 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T14:08:24.980 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T14:08:24.980 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T14:08:24.980 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T14:08:24.980 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T14:08:24.980 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T14:08:24.980 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T14:08:24.980 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T14:08:24.981 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T14:08:24.981 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T14:08:24.981 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T14:08:24.981 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T14:08:24.981 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T14:08:24.981 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T14:08:24.981 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:24.981 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T14:08:24.981 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T14:08:24.981 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T14:08:24.981 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T14:08:24.981 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T14:08:24.981 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T14:08:24.981 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T14:08:24.982 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T14:08:24.982 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T14:08:24.982 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T14:08:24.982 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T14:08:24.982 INFO:teuthology.orchestra.run.smithi049.stdout: "options": { 2024-09-21T14:08:24.982 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_bias": 4 2024-09-21T14:08:24.982 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:24.982 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T14:08:24.982 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw": {} 2024-09-21T14:08:24.982 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:24.982 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T14:08:24.982 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T14:08:24.982 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 1.5399999618530273, 2024-09-21T14:08:24.982 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 1.5399999618530273, 2024-09-21T14:08:24.983 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1.1399999856948853, 2024-09-21T14:08:24.983 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 1.75, 2024-09-21T14:08:24.983 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 1.75, 2024-09-21T14:08:24.983 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.85335731506347656, 2024-09-21T14:08:24.983 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.98000001907348633, 2024-09-21T14:08:24.983 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.85000002384185791 2024-09-21T14:08:24.983 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:24.983 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:24.983 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:24.983 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 11, 2024-09-21T14:08:24.983 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "unique_pool_4", 2024-09-21T14:08:24.983 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T14:06:43.392407+0000", 2024-09-21T14:08:24.983 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T14:08:24.983 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T14:08:24.983 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T14:08:24.984 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T14:08:24.984 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T14:08:24.984 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T14:08:24.984 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T14:08:24.984 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T14:08:24.984 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T14:08:24.984 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T14:08:24.984 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T14:08:24.984 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T14:08:24.984 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T14:08:24.984 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 16, 2024-09-21T14:08:24.984 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 16, 2024-09-21T14:08:24.984 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 16, 2024-09-21T14:08:24.984 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 16, 2024-09-21T14:08:24.984 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 16, 2024-09-21T14:08:24.985 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T14:08:24.985 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T14:08:24.985 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T14:08:24.985 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T14:08:24.985 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T14:08:24.985 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T14:08:24.985 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T14:08:24.985 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:24.985 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "465", 2024-09-21T14:08:24.985 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T14:08:24.985 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T14:08:24.985 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T14:08:24.985 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T14:08:24.985 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T14:08:24.986 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T14:08:24.986 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T14:08:24.986 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T14:08:24.986 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T14:08:24.986 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T14:08:24.986 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T14:08:24.986 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T14:08:24.986 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T14:08:24.986 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T14:08:24.986 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T14:08:24.986 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T14:08:24.986 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T14:08:24.986 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T14:08:24.986 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T14:08:24.986 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T14:08:24.987 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T14:08:24.987 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T14:08:24.987 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T14:08:24.987 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T14:08:24.987 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T14:08:24.987 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T14:08:24.987 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:24.987 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T14:08:24.987 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T14:08:24.987 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T14:08:24.987 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T14:08:24.987 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T14:08:24.987 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T14:08:24.987 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T14:08:24.988 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T14:08:24.988 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T14:08:24.988 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T14:08:24.988 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T14:08:24.988 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T14:08:24.988 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T14:08:24.988 INFO:teuthology.orchestra.run.smithi049.stdout: "rados": {} 2024-09-21T14:08:24.988 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:24.988 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T14:08:24.988 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T14:08:24.988 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2.190000057220459, 2024-09-21T14:08:24.988 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2.190000057220459, 2024-09-21T14:08:24.988 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1.1399999856948853, 2024-09-21T14:08:24.988 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2.5, 2024-09-21T14:08:24.989 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2.5, 2024-09-21T14:08:24.989 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.85335731506347656, 2024-09-21T14:08:24.989 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.98000001907348633, 2024-09-21T14:08:24.989 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.85000002384185791 2024-09-21T14:08:24.989 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:24.989 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:24.989 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T14:08:24.989 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-09-21T14:08:24.989 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:24.989 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 0, 2024-09-21T14:08:24.989 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "2595dca9-6219-4e0c-8728-751eb102e73a", 2024-09-21T14:08:24.989 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T14:08:24.989 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T14:08:24.989 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 0.6176910400390625, 2024-09-21T14:08:24.989 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T14:08:24.990 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T14:08:24.990 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T14:08:24.990 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 8, 2024-09-21T14:08:24.990 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 497, 2024-09-21T14:08:24.990 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T14:08:24.990 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T14:08:24.990 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T14:08:24.990 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:08:24.990 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:24.990 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:08:24.990 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6802", 2024-09-21T14:08:24.990 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T14:08:24.990 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:24.990 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:24.990 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:08:24.991 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6803", 2024-09-21T14:08:24.991 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T14:08:24.991 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:24.991 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:24.991 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:24.991 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T14:08:24.991 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:08:24.991 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:24.991 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:08:24.991 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6804", 2024-09-21T14:08:24.991 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T14:08:24.991 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:24.991 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:24.991 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:08:24.992 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6805", 2024-09-21T14:08:24.992 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T14:08:24.992 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:24.992 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:24.992 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:24.992 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T14:08:24.992 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:08:24.992 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:24.992 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:08:24.993 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6808", 2024-09-21T14:08:24.993 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T14:08:24.993 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:24.993 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:24.993 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:08:24.993 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6809", 2024-09-21T14:08:24.993 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T14:08:24.993 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:24.993 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:24.993 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:24.993 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T14:08:24.994 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:08:24.994 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:24.994 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:08:24.994 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6806", 2024-09-21T14:08:24.994 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T14:08:24.994 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:24.994 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:24.994 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:08:24.994 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6807", 2024-09-21T14:08:24.994 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T14:08:24.994 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:24.994 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:24.994 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:24.994 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6803/2095561412", 2024-09-21T14:08:24.995 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6805/2095561412", 2024-09-21T14:08:24.995 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6809/2095561412", 2024-09-21T14:08:24.995 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6807/2095561412", 2024-09-21T14:08:24.995 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T14:08:24.995 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T14:08:24.995 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T14:08:24.995 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:24.995 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:24.995 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:24.995 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 1, 2024-09-21T14:08:24.995 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "0ba327f9-5524-4a40-abcc-729ded0b1917", 2024-09-21T14:08:24.995 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T14:08:24.995 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T14:08:24.995 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T14:08:24.996 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T14:08:24.996 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T14:08:24.996 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T14:08:24.996 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 13, 2024-09-21T14:08:24.996 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 497, 2024-09-21T14:08:24.996 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T14:08:24.996 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T14:08:24.996 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T14:08:24.996 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:08:24.996 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:24.996 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:08:24.996 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6810", 2024-09-21T14:08:24.996 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T14:08:24.996 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:24.997 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:24.997 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:08:24.997 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6811", 2024-09-21T14:08:24.997 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T14:08:24.997 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:24.997 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:24.997 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:24.997 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T14:08:24.997 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:08:24.997 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:24.997 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:08:24.997 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6812", 2024-09-21T14:08:24.997 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T14:08:24.997 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:24.998 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:24.998 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:08:24.998 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6813", 2024-09-21T14:08:24.998 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T14:08:24.998 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:24.998 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:24.998 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:24.998 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T14:08:24.998 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:08:24.998 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:24.998 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:08:24.998 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6816", 2024-09-21T14:08:24.998 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T14:08:24.998 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:24.999 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:24.999 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:08:24.999 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6817", 2024-09-21T14:08:24.999 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T14:08:24.999 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:24.999 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:24.999 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:24.999 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T14:08:24.999 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:08:24.999 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:24.999 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:08:24.999 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6814", 2024-09-21T14:08:24.999 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T14:08:24.999 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.000 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.000 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:08:25.000 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6815", 2024-09-21T14:08:25.000 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T14:08:25.000 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:25.000 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:25.000 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.000 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6811/1382840527", 2024-09-21T14:08:25.000 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6813/1382840527", 2024-09-21T14:08:25.000 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6817/1382840527", 2024-09-21T14:08:25.000 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6815/1382840527", 2024-09-21T14:08:25.000 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T14:08:25.000 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T14:08:25.000 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T14:08:25.001 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:25.001 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.001 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.001 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 2, 2024-09-21T14:08:25.001 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "27a45750-204d-48fd-bb8f-ebb01273343d", 2024-09-21T14:08:25.001 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T14:08:25.001 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T14:08:25.001 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T14:08:25.001 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 0.8268585205078125, 2024-09-21T14:08:25.001 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 18, 2024-09-21T14:08:25.001 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 100, 2024-09-21T14:08:25.001 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 130, 2024-09-21T14:08:25.001 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 496, 2024-09-21T14:08:25.001 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 101, 2024-09-21T14:08:25.001 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T14:08:25.002 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T14:08:25.002 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:08:25.002 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.002 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:08:25.002 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6818", 2024-09-21T14:08:25.002 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T14:08:25.002 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.002 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.002 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:08:25.002 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6819", 2024-09-21T14:08:25.002 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T14:08:25.002 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:25.002 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:25.002 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.003 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T14:08:25.003 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:08:25.003 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.003 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:08:25.003 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6820", 2024-09-21T14:08:25.003 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T14:08:25.003 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.003 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.003 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:08:25.003 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6821", 2024-09-21T14:08:25.003 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T14:08:25.003 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:25.003 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:25.003 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.004 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T14:08:25.004 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:08:25.004 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.004 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:08:25.004 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6824", 2024-09-21T14:08:25.004 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T14:08:25.004 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.004 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.004 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:08:25.004 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6825", 2024-09-21T14:08:25.004 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T14:08:25.004 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:25.004 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:25.004 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.004 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T14:08:25.005 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:08:25.005 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.005 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:08:25.005 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6822", 2024-09-21T14:08:25.005 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T14:08:25.005 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.005 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.005 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:08:25.005 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6823", 2024-09-21T14:08:25.005 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T14:08:25.005 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:25.005 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:25.005 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.005 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6819/1945314741", 2024-09-21T14:08:25.006 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6821/1945314741", 2024-09-21T14:08:25.006 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6825/1945314741", 2024-09-21T14:08:25.006 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6823/1945314741", 2024-09-21T14:08:25.006 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T14:08:25.006 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T14:08:25.006 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T14:08:25.006 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:25.006 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.006 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.006 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 3, 2024-09-21T14:08:25.006 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "528bc390-3dc9-4f3b-ab75-d7412c9625db", 2024-09-21T14:08:25.006 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T14:08:25.006 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T14:08:25.006 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 0.502227783203125, 2024-09-21T14:08:25.007 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T14:08:25.007 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 27, 2024-09-21T14:08:25.007 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 214, 2024-09-21T14:08:25.007 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 240, 2024-09-21T14:08:25.007 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 491, 2024-09-21T14:08:25.007 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 215, 2024-09-21T14:08:25.007 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T14:08:25.007 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T14:08:25.007 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:08:25.007 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.007 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:08:25.007 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6826", 2024-09-21T14:08:25.007 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T14:08:25.007 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.007 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.008 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:08:25.008 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6827", 2024-09-21T14:08:25.008 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T14:08:25.008 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:25.008 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:25.008 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.008 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T14:08:25.008 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:08:25.008 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.008 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:08:25.008 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6828", 2024-09-21T14:08:25.008 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T14:08:25.008 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.008 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.009 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:08:25.009 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6829", 2024-09-21T14:08:25.009 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T14:08:25.009 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:25.009 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:25.009 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.009 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T14:08:25.009 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:08:25.009 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.009 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:08:25.009 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6832", 2024-09-21T14:08:25.009 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T14:08:25.009 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.009 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.010 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:08:25.010 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6833", 2024-09-21T14:08:25.010 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T14:08:25.010 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:25.010 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:25.010 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.010 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T14:08:25.010 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:08:25.010 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.010 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:08:25.010 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6830", 2024-09-21T14:08:25.010 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T14:08:25.010 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.010 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.011 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:08:25.011 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6831", 2024-09-21T14:08:25.011 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T14:08:25.011 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:25.011 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:25.011 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.011 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6827/886735721", 2024-09-21T14:08:25.011 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6829/886735721", 2024-09-21T14:08:25.011 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6833/886735721", 2024-09-21T14:08:25.011 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6831/886735721", 2024-09-21T14:08:25.011 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T14:08:25.011 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T14:08:25.011 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T14:08:25.011 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:25.011 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.012 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.012 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 4, 2024-09-21T14:08:25.012 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "6fc64698-2d37-4db0-bf3d-c7e612c45163", 2024-09-21T14:08:25.012 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T14:08:25.012 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T14:08:25.012 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 0.7888946533203125, 2024-09-21T14:08:25.012 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T14:08:25.012 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T14:08:25.012 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T14:08:25.012 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 33, 2024-09-21T14:08:25.012 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 496, 2024-09-21T14:08:25.012 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T14:08:25.012 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T14:08:25.012 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T14:08:25.012 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:08:25.013 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.013 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:08:25.013 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6800", 2024-09-21T14:08:25.013 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T14:08:25.013 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.013 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.013 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:08:25.013 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6801", 2024-09-21T14:08:25.013 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T14:08:25.013 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:25.013 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:25.013 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.013 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T14:08:25.013 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:08:25.014 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.014 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:08:25.014 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6802", 2024-09-21T14:08:25.014 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T14:08:25.014 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.014 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.014 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:08:25.014 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6803", 2024-09-21T14:08:25.014 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T14:08:25.014 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:25.014 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:25.014 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.014 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T14:08:25.014 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:08:25.015 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.015 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:08:25.015 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6806", 2024-09-21T14:08:25.015 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T14:08:25.015 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.015 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.015 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:08:25.015 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6807", 2024-09-21T14:08:25.015 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T14:08:25.015 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:25.015 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:25.015 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.015 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T14:08:25.015 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:08:25.015 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.016 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:08:25.016 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6804", 2024-09-21T14:08:25.016 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T14:08:25.016 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.016 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.016 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:08:25.016 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6805", 2024-09-21T14:08:25.016 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T14:08:25.016 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:25.016 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:25.016 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.016 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.187:6801/1702447776", 2024-09-21T14:08:25.016 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.187:6803/1702447776", 2024-09-21T14:08:25.016 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.187:6807/1702447776", 2024-09-21T14:08:25.017 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.187:6805/1702447776", 2024-09-21T14:08:25.017 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T14:08:25.017 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T14:08:25.017 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T14:08:25.017 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:25.017 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.017 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.017 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 5, 2024-09-21T14:08:25.017 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "8d7f62a1-7e92-4e06-9cad-86b96be514d8", 2024-09-21T14:08:25.017 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T14:08:25.017 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T14:08:25.017 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T14:08:25.017 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T14:08:25.018 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 39, 2024-09-21T14:08:25.018 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 324, 2024-09-21T14:08:25.018 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 345, 2024-09-21T14:08:25.018 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 496, 2024-09-21T14:08:25.018 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 325, 2024-09-21T14:08:25.018 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T14:08:25.018 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T14:08:25.018 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:08:25.018 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.018 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:08:25.018 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6808", 2024-09-21T14:08:25.018 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1595353965 2024-09-21T14:08:25.018 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.018 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.019 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:08:25.019 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6809", 2024-09-21T14:08:25.019 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1595353965 2024-09-21T14:08:25.019 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:25.019 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:25.019 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.019 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T14:08:25.019 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:08:25.019 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.019 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:08:25.019 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6810", 2024-09-21T14:08:25.019 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1595353965 2024-09-21T14:08:25.019 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.019 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.020 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:08:25.020 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6811", 2024-09-21T14:08:25.020 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1595353965 2024-09-21T14:08:25.020 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:25.020 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:25.020 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.020 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T14:08:25.020 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:08:25.020 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.020 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:08:25.020 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6814", 2024-09-21T14:08:25.020 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1595353965 2024-09-21T14:08:25.020 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.020 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.021 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:08:25.021 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6815", 2024-09-21T14:08:25.021 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1595353965 2024-09-21T14:08:25.021 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:25.021 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:25.021 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.021 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T14:08:25.021 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:08:25.021 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.021 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:08:25.021 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6812", 2024-09-21T14:08:25.021 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1595353965 2024-09-21T14:08:25.021 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.021 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.022 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:08:25.022 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6813", 2024-09-21T14:08:25.022 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1595353965 2024-09-21T14:08:25.022 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:25.022 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:25.022 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.022 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.187:6809/1595353965", 2024-09-21T14:08:25.022 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.187:6811/1595353965", 2024-09-21T14:08:25.022 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.187:6815/1595353965", 2024-09-21T14:08:25.022 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.187:6813/1595353965", 2024-09-21T14:08:25.022 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T14:08:25.022 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T14:08:25.022 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T14:08:25.022 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:25.022 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.023 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.023 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 6, 2024-09-21T14:08:25.023 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "3a7afd13-26ce-444c-9cc0-81ce2eab6ce5", 2024-09-21T14:08:25.023 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T14:08:25.023 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T14:08:25.023 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 0.9076080322265625, 2024-09-21T14:08:25.023 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T14:08:25.023 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T14:08:25.023 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T14:08:25.023 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 47, 2024-09-21T14:08:25.023 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 497, 2024-09-21T14:08:25.023 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T14:08:25.023 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T14:08:25.023 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T14:08:25.024 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:08:25.024 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.024 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:08:25.024 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6816", 2024-09-21T14:08:25.024 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T14:08:25.024 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.024 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.024 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:08:25.024 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6817", 2024-09-21T14:08:25.024 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T14:08:25.024 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:25.024 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:25.024 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.024 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T14:08:25.024 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:08:25.025 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.025 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:08:25.025 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6818", 2024-09-21T14:08:25.025 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T14:08:25.025 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.025 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.025 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:08:25.025 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6819", 2024-09-21T14:08:25.025 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T14:08:25.025 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:25.025 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:25.025 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.025 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T14:08:25.025 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:08:25.026 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.026 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:08:25.026 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6822", 2024-09-21T14:08:25.026 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T14:08:25.026 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.026 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.026 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:08:25.026 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6823", 2024-09-21T14:08:25.026 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T14:08:25.026 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:25.026 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:25.026 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.026 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T14:08:25.026 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:08:25.027 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.027 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:08:25.027 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6820", 2024-09-21T14:08:25.027 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T14:08:25.027 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.027 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.027 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:08:25.027 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6821", 2024-09-21T14:08:25.027 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T14:08:25.027 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:25.027 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:25.027 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.027 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.187:6817/2728426525", 2024-09-21T14:08:25.027 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.187:6819/2728426525", 2024-09-21T14:08:25.028 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.187:6823/2728426525", 2024-09-21T14:08:25.028 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.187:6821/2728426525", 2024-09-21T14:08:25.028 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:08:25.029 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:08:25.029 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T14:08:25.029 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T14:08:25.029 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T14:08:25.029 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:25.030 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.030 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.030 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 7, 2024-09-21T14:08:25.030 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "1e08d891-8747-402e-b4c2-aa3802ce8537", 2024-09-21T14:08:25.030 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T14:08:25.030 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T14:08:25.030 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T14:08:25.030 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 0, 2024-09-21T14:08:25.030 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 54, 2024-09-21T14:08:25.030 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 272, 2024-09-21T14:08:25.030 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 298, 2024-09-21T14:08:25.030 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 461, 2024-09-21T14:08:25.030 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 273, 2024-09-21T14:08:25.030 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T14:08:25.031 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T14:08:25.031 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:08:25.031 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.031 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:08:25.031 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6824", 2024-09-21T14:08:25.031 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 13894964 2024-09-21T14:08:25.031 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.031 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.031 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:08:25.031 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6825", 2024-09-21T14:08:25.031 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 13894964 2024-09-21T14:08:25.031 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:25.031 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:25.031 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.032 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T14:08:25.032 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:08:25.032 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.032 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:08:25.032 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6826", 2024-09-21T14:08:25.032 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 13894964 2024-09-21T14:08:25.032 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.032 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.032 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:08:25.032 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6827", 2024-09-21T14:08:25.032 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 13894964 2024-09-21T14:08:25.032 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:25.032 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:25.032 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.033 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T14:08:25.033 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:08:25.033 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.033 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:08:25.033 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6830", 2024-09-21T14:08:25.033 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 13894964 2024-09-21T14:08:25.033 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.033 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.033 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:08:25.033 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6831", 2024-09-21T14:08:25.033 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 13894964 2024-09-21T14:08:25.033 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:25.033 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:25.033 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.034 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T14:08:25.034 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:08:25.034 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.034 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:08:25.034 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6828", 2024-09-21T14:08:25.034 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 13894964 2024-09-21T14:08:25.034 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.034 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.034 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:08:25.034 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6829", 2024-09-21T14:08:25.034 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 13894964 2024-09-21T14:08:25.034 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:25.034 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:25.034 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.035 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.187:6825/13894964", 2024-09-21T14:08:25.035 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.187:6827/13894964", 2024-09-21T14:08:25.035 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.187:6831/13894964", 2024-09-21T14:08:25.035 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.187:6829/13894964", 2024-09-21T14:08:25.035 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T14:08:25.035 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T14:08:25.035 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T14:08:25.035 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:25.035 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:25.035 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T14:08:25.035 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_xinfo": [ 2024-09-21T14:08:25.035 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.035 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 0, 2024-09-21T14:08:25.035 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T14:08:25.035 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T14:08:25.036 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T14:08:25.036 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T14:08:25.036 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T14:08:25.036 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T14:07:51.070593+0000", 2024-09-21T14:08:25.036 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T14:08:25.036 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.036 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.036 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 1, 2024-09-21T14:08:25.036 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T14:08:25.036 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T14:08:25.036 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T14:08:25.036 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T14:08:25.036 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T14:08:25.036 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T14:07:20.759253+0000", 2024-09-21T14:08:25.037 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T14:08:25.037 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.037 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.037 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 2, 2024-09-21T14:08:25.037 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T14:08:25.037 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T14:08:25.037 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T14:08:25.037 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T14:08:25.037 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T14:08:25.037 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T14:07:51.524983+0000", 2024-09-21T14:08:25.037 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 100 2024-09-21T14:08:25.037 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.037 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.037 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 3, 2024-09-21T14:08:25.038 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T14:08:25.038 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T14:08:25.038 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T14:08:25.038 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T14:08:25.038 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T14:08:25.038 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T14:08:08.471904+0000", 2024-09-21T14:08:25.038 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 214 2024-09-21T14:08:25.038 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.038 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.038 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 4, 2024-09-21T14:08:25.038 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T14:08:25.038 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T14:08:25.038 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T14:08:25.038 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T14:08:25.039 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T14:08:25.039 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T14:08:03.561026+0000", 2024-09-21T14:08:25.039 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T14:08:25.039 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.039 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.039 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 5, 2024-09-21T14:08:25.039 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T14:08:25.039 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T14:08:25.039 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T14:08:25.039 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T14:08:25.039 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T14:08:25.039 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T14:08:06.355614+0000", 2024-09-21T14:08:25.039 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 324 2024-09-21T14:08:25.039 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.040 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.040 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 6, 2024-09-21T14:08:25.040 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T14:08:25.040 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T14:08:25.040 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T14:08:25.040 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T14:08:25.040 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T14:08:25.040 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T14:07:16.644516+0000", 2024-09-21T14:08:25.040 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T14:08:25.040 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.040 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.040 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 7, 2024-09-21T14:08:25.040 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T14:08:25.040 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T14:08:25.041 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T14:08:25.041 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T14:08:25.041 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T14:08:25.041 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T14:07:20.884792+0000", 2024-09-21T14:08:25.041 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 272 2024-09-21T14:08:25.041 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:25.041 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T14:08:25.041 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap": [ 2024-09-21T14:08:25.041 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.041 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "3.14", 2024-09-21T14:08:25.041 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-09-21T14:08:25.042 INFO:teuthology.orchestra.run.smithi049.stdout: 7, 2024-09-21T14:08:25.042 INFO:teuthology.orchestra.run.smithi049.stdout: 6, 2024-09-21T14:08:25.042 INFO:teuthology.orchestra.run.smithi049.stdout: 0 2024-09-21T14:08:25.042 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:25.042 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:25.042 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T14:08:25.042 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap_items": [ 2024-09-21T14:08:25.042 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.042 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "4.2", 2024-09-21T14:08:25.042 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-09-21T14:08:25.042 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.042 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 1, 2024-09-21T14:08:25.042 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 2 2024-09-21T14:08:25.042 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:25.043 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:25.043 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.043 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.043 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "4.6", 2024-09-21T14:08:25.043 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-09-21T14:08:25.043 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.043 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 1, 2024-09-21T14:08:25.043 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 5 2024-09-21T14:08:25.043 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:25.043 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:25.043 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.043 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.043 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "5.16", 2024-09-21T14:08:25.043 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-09-21T14:08:25.044 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.044 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 1, 2024-09-21T14:08:25.044 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 6 2024-09-21T14:08:25.044 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.044 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.044 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 3, 2024-09-21T14:08:25.044 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 5 2024-09-21T14:08:25.044 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:25.044 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:25.044 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.044 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.044 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "6.7", 2024-09-21T14:08:25.044 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-09-21T14:08:25.044 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.044 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 3, 2024-09-21T14:08:25.045 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 7 2024-09-21T14:08:25.045 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:25.045 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:25.045 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.045 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.045 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "6.b", 2024-09-21T14:08:25.045 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-09-21T14:08:25.045 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.045 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 3, 2024-09-21T14:08:25.045 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 1 2024-09-21T14:08:25.045 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.045 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.045 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 7, 2024-09-21T14:08:25.045 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 6 2024-09-21T14:08:25.046 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:25.046 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:25.046 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.046 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.046 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "6.1f", 2024-09-21T14:08:25.046 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-09-21T14:08:25.046 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.046 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 3, 2024-09-21T14:08:25.046 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 0 2024-09-21T14:08:25.046 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:25.046 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:25.046 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:25.046 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T14:08:25.047 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap_primaries": [], 2024-09-21T14:08:25.047 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_temp": [ 2024-09-21T14:08:25.047 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.047 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "11.0", 2024-09-21T14:08:25.047 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-09-21T14:08:25.047 INFO:teuthology.orchestra.run.smithi049.stdout: 1, 2024-09-21T14:08:25.047 INFO:teuthology.orchestra.run.smithi049.stdout: 7, 2024-09-21T14:08:25.047 INFO:teuthology.orchestra.run.smithi049.stdout: 3 2024-09-21T14:08:25.047 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:25.047 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.047 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.047 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "11.1", 2024-09-21T14:08:25.047 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-09-21T14:08:25.047 INFO:teuthology.orchestra.run.smithi049.stdout: 6, 2024-09-21T14:08:25.048 INFO:teuthology.orchestra.run.smithi049.stdout: 0, 2024-09-21T14:08:25.048 INFO:teuthology.orchestra.run.smithi049.stdout: 3 2024-09-21T14:08:25.048 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:25.048 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.048 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.048 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "11.2", 2024-09-21T14:08:25.048 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-09-21T14:08:25.048 INFO:teuthology.orchestra.run.smithi049.stdout: 0, 2024-09-21T14:08:25.048 INFO:teuthology.orchestra.run.smithi049.stdout: 2, 2024-09-21T14:08:25.048 INFO:teuthology.orchestra.run.smithi049.stdout: 3 2024-09-21T14:08:25.048 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:25.048 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.048 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.048 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "11.8", 2024-09-21T14:08:25.049 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-09-21T14:08:25.049 INFO:teuthology.orchestra.run.smithi049.stdout: 6, 2024-09-21T14:08:25.049 INFO:teuthology.orchestra.run.smithi049.stdout: 5, 2024-09-21T14:08:25.049 INFO:teuthology.orchestra.run.smithi049.stdout: 3 2024-09-21T14:08:25.049 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:25.049 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.049 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:25.049 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "11.9", 2024-09-21T14:08:25.049 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-09-21T14:08:25.049 INFO:teuthology.orchestra.run.smithi049.stdout: 6, 2024-09-21T14:08:25.049 INFO:teuthology.orchestra.run.smithi049.stdout: 1, 2024-09-21T14:08:25.049 INFO:teuthology.orchestra.run.smithi049.stdout: 3 2024-09-21T14:08:25.049 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:25.049 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:25.049 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T14:08:25.050 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_temp": [], 2024-09-21T14:08:25.050 INFO:teuthology.orchestra.run.smithi049.stdout: "blocklist": { 2024-09-21T14:08:25.050 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/2731793917": "2024-09-22T13:49:27.311313+0000", 2024-09-21T14:08:25.050 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/799831425": "2024-09-22T13:49:27.311313+0000", 2024-09-21T14:08:25.050 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/2518197280": "2024-09-22T13:43:28.110499+0000", 2024-09-21T14:08:25.050 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3556829486": "2024-09-22T13:43:28.110499+0000", 2024-09-21T14:08:25.050 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/1249926115": "2024-09-22T13:49:27.311313+0000", 2024-09-21T14:08:25.050 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/231530138": "2024-09-22T13:43:28.110499+0000", 2024-09-21T14:08:25.050 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/4220781134": "2024-09-22T13:49:27.311313+0000", 2024-09-21T14:08:25.050 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/202325455": "2024-09-22T13:43:28.110499+0000", 2024-09-21T14:08:25.050 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/4129636473": "2024-09-22T13:43:56.633723+0000", 2024-09-21T14:08:25.050 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/2518197280": "2024-09-22T13:43:28.110499+0000", 2024-09-21T14:08:25.050 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3514741273": "2024-09-22T13:43:56.633723+0000", 2024-09-21T14:08:25.051 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/4129636473": "2024-09-22T13:43:56.633723+0000", 2024-09-21T14:08:25.051 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3125743776": "2024-09-22T13:43:56.633723+0000", 2024-09-21T14:08:25.051 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3600977363": "2024-09-22T13:43:56.633723+0000", 2024-09-21T14:08:25.051 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/3453571598": "2024-09-22T13:49:27.311313+0000", 2024-09-21T14:08:25.051 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/3453571598": "2024-09-22T13:49:27.311313+0000", 2024-09-21T14:08:25.051 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3553734432": "2024-09-22T13:49:27.311313+0000" 2024-09-21T14:08:25.051 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.051 INFO:teuthology.orchestra.run.smithi049.stdout: "range_blocklist": {}, 2024-09-21T14:08:25.051 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profiles": { 2024-09-21T14:08:25.051 INFO:teuthology.orchestra.run.smithi049.stdout: "default": { 2024-09-21T14:08:25.051 INFO:teuthology.orchestra.run.smithi049.stdout: "crush-failure-domain": "osd", 2024-09-21T14:08:25.051 INFO:teuthology.orchestra.run.smithi049.stdout: "k": "2", 2024-09-21T14:08:25.051 INFO:teuthology.orchestra.run.smithi049.stdout: "m": "1", 2024-09-21T14:08:25.051 INFO:teuthology.orchestra.run.smithi049.stdout: "plugin": "jerasure", 2024-09-21T14:08:25.051 INFO:teuthology.orchestra.run.smithi049.stdout: "technique": "reed_sol_van" 2024-09-21T14:08:25.052 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:25.052 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:25.052 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps_queue": [], 2024-09-21T14:08:25.052 INFO:teuthology.orchestra.run.smithi049.stdout: "new_removed_snaps": [], 2024-09-21T14:08:25.052 INFO:teuthology.orchestra.run.smithi049.stdout: "new_purged_snaps": [], 2024-09-21T14:08:25.052 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_node_flags": {}, 2024-09-21T14:08:25.052 INFO:teuthology.orchestra.run.smithi049.stdout: "device_class_flags": {}, 2024-09-21T14:08:25.052 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode": { 2024-09-21T14:08:25.052 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode_enabled": false, 2024-09-21T14:08:25.052 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_bucket_count": 0, 2024-09-21T14:08:25.052 INFO:teuthology.orchestra.run.smithi049.stdout: "degraded_stretch_mode": 0, 2024-09-21T14:08:25.052 INFO:teuthology.orchestra.run.smithi049.stdout: "recovering_stretch_mode": 0, 2024-09-21T14:08:25.052 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode_bucket": 0 2024-09-21T14:08:25.052 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:25.052 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T14:08:25.053 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:08:25.053 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:08:25.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:25.054 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:25.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:08:25.062 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:08:25.063 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:08:25.098 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:08:25.099 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:08:25.135 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:08:25.135 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:08:25.155 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:25.186 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:25.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:25.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:25.288 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:08:25.332 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:25.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:25.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:08:25.434 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:25.465 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:25.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:25.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:25.567 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:25.603 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T14:08:25.607 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:25.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:25.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:25.710 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:08:25.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:25 smithi049 ceph-mon[30080]: pgmap v996: 168 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 7.4 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 53 MiB/s wr, 864 op/s; 33126/364629 objects misplaced (9.085%); 32 MiB/s, 511 objects/s recovering 2024-09-21T14:08:25.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:25 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3837101568' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:08:25.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:25 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/4022958561' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T14:08:25.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:25 smithi049 ceph-mon[40489]: pgmap v996: 168 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 7.4 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 53 MiB/s wr, 864 op/s; 33126/364629 objects misplaced (9.085%); 32 MiB/s, 511 objects/s recovering 2024-09-21T14:08:25.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:25 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3837101568' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:08:25.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:25 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/4022958561' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T14:08:25.780 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:25.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:25.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:08:25.881 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:08:25.920 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:25.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:25.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:08:25.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:25 smithi187 ceph-mon[33032]: pgmap v996: 168 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 7.4 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 53 MiB/s wr, 864 op/s; 33126/364629 objects misplaced (9.085%); 32 MiB/s, 511 objects/s recovering 2024-09-21T14:08:25.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:25 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3837101568' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:08:25.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:25 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/4022958561' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T14:08:26.021 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:26.056 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:26.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:26.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:26.157 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:26.187 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:26.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:26.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:26.289 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:08:26.315 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:26.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:26.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:08:26.417 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:08:26.453 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:26.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:26.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:08:26.555 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:26.584 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:26.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:26.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:26.686 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:08:26.720 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:26.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:26.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:08:26.822 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:08:26.865 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:26.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:26.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:08:26.967 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:26.998 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:26.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:26.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:27.100 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:08:27.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:27.147 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:27.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:08:27.248 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:08:27.286 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:27.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:27.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:08:27.388 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:08:27.415 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:27.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:27.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:08:27.517 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:08:27.557 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:27.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:27.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:08:27.659 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:27.693 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:27.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:27.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:27.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:27 smithi049 ceph-mon[40489]: pgmap v997: 168 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 7.5 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 54 MiB/s wr, 959 op/s; 32006/366768 objects misplaced (8.726%); 37 MiB/s, 589 objects/s recovering 2024-09-21T14:08:27.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:27 smithi049 ceph-mon[30080]: pgmap v997: 168 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 7.5 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 54 MiB/s wr, 959 op/s; 32006/366768 objects misplaced (8.726%); 37 MiB/s, 589 objects/s recovering 2024-09-21T14:08:27.794 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:08:27.824 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:27.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:27.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:08:27.925 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:08:27.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:27 smithi187 ceph-mon[33032]: pgmap v997: 168 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 7.5 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 54 MiB/s wr, 959 op/s; 32006/366768 objects misplaced (8.726%); 37 MiB/s, 589 objects/s recovering 2024-09-21T14:08:27.962 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:27.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:27.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:08:28.064 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:28.104 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:28.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:28.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:28.206 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:28.239 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:28.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:28.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:28.343 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:28.375 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:28.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:28.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:28.477 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:08:28.521 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:28.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:28.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:08:28.623 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:28.667 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:28.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:28.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:28.769 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:28.814 INFO:tasks.thrashosds.thrasher:j is {'epoch': 499, 'fsid': '40a3d2e8-781f-11ef-baf6-efdc52797490', 'created': '2024-09-21T13:42:57.838275+0000', 'modified': '2024-09-21T14:08:20.447518+0000', 'last_up_change': '2024-09-21T14:01:40.158454+0000', 'last_in_change': '2024-09-21T13:58:16.730861+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 22, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 11, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T13:46:21.096244+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 21, 'pg_placement_num': 21, 'pg_placement_num_target': 21, 'pg_num_target': 21, 'pg_num_pending': 21, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '438', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '436', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.3399999141693115, 'score_stable': 2.3399999141693115, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 2.6700000762939453, 'raw_score_stable': 2.6700000762939453, 'primary_affinity_weighted': 0.8533573150634766, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.8500000238418579}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-21T13:48:55.540712+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 63, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.3299999237060547, 'score_stable': 2.3299999237060547, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8533573150634766, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.8500000238418579}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-21T13:48:56.417456+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8533573150634766, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.8500000238418579}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-21T13:48:58.128455+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5399999618530273, 'score_stable': 1.5399999618530273, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8533573150634766, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.8500000238418579}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-21T13:49:00.069361+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5399999618530273, 'score_stable': 1.5399999618530273, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8533573150634766, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.8500000238418579}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-21T13:49:02.127399+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5399999618530273, 'score_stable': 1.5399999618530273, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8533573150634766, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.8500000238418579}}, {'pool': 11, 'pool_name': 'unique_pool_4', 'create_time': '2024-09-21T14:06:43.392407+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '465', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.190000057220459, 'score_stable': 2.190000057220459, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 0.8533573150634766, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.8500000238418579}}], 'osds': [{'osd': 0, 'uuid': '2595dca9-6219-4e0c-8728-751eb102e73a', 'up': 1, 'in': 1, 'weight': 0.6176910400390625, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 497, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6802', 'nonce': 2095561412}, {'type': 'v1', 'addr': '172.21.15.49:6803', 'nonce': 2095561412}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6804', 'nonce': 2095561412}, {'type': 'v1', 'addr': '172.21.15.49:6805', 'nonce': 2095561412}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6808', 'nonce': 2095561412}, {'type': 'v1', 'addr': '172.21.15.49:6809', 'nonce': 2095561412}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6806', 'nonce': 2095561412}, {'type': 'v1', 'addr': '172.21.15.49:6807', 'nonce': 2095561412}]}, 'public_addr': '172.21.15.49:6803/2095561412', 'cluster_addr': '172.21.15.49:6805/2095561412', 'heartbeat_back_addr': '172.21.15.49:6809/2095561412', 'heartbeat_front_addr': '172.21.15.49:6807/2095561412', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '0ba327f9-5524-4a40-abcc-729ded0b1917', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 497, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6810', 'nonce': 1382840527}, {'type': 'v1', 'addr': '172.21.15.49:6811', 'nonce': 1382840527}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6812', 'nonce': 1382840527}, {'type': 'v1', 'addr': '172.21.15.49:6813', 'nonce': 1382840527}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6816', 'nonce': 1382840527}, {'type': 'v1', 'addr': '172.21.15.49:6817', 'nonce': 1382840527}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6814', 'nonce': 1382840527}, {'type': 'v1', 'addr': '172.21.15.49:6815', 'nonce': 1382840527}]}, 'public_addr': '172.21.15.49:6811/1382840527', 'cluster_addr': '172.21.15.49:6813/1382840527', 'heartbeat_back_addr': '172.21.15.49:6817/1382840527', 'heartbeat_front_addr': '172.21.15.49:6815/1382840527', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '27a45750-204d-48fd-bb8f-ebb01273343d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.8268585205078125, 'last_clean_begin': 18, 'last_clean_end': 100, 'up_from': 130, 'up_thru': 496, 'down_at': 101, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6818', 'nonce': 1945314741}, {'type': 'v1', 'addr': '172.21.15.49:6819', 'nonce': 1945314741}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6820', 'nonce': 1945314741}, {'type': 'v1', 'addr': '172.21.15.49:6821', 'nonce': 1945314741}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6824', 'nonce': 1945314741}, {'type': 'v1', 'addr': '172.21.15.49:6825', 'nonce': 1945314741}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6822', 'nonce': 1945314741}, {'type': 'v1', 'addr': '172.21.15.49:6823', 'nonce': 1945314741}]}, 'public_addr': '172.21.15.49:6819/1945314741', 'cluster_addr': '172.21.15.49:6821/1945314741', 'heartbeat_back_addr': '172.21.15.49:6825/1945314741', 'heartbeat_front_addr': '172.21.15.49:6823/1945314741', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '528bc390-3dc9-4f3b-ab75-d7412c9625db', 'up': 1, 'in': 1, 'weight': 0.502227783203125, 'primary_affinity': 1, 'last_clean_begin': 27, 'last_clean_end': 214, 'up_from': 240, 'up_thru': 491, 'down_at': 215, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6826', 'nonce': 886735721}, {'type': 'v1', 'addr': '172.21.15.49:6827', 'nonce': 886735721}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6828', 'nonce': 886735721}, {'type': 'v1', 'addr': '172.21.15.49:6829', 'nonce': 886735721}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6832', 'nonce': 886735721}, {'type': 'v1', 'addr': '172.21.15.49:6833', 'nonce': 886735721}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6830', 'nonce': 886735721}, {'type': 'v1', 'addr': '172.21.15.49:6831', 'nonce': 886735721}]}, 'public_addr': '172.21.15.49:6827/886735721', 'cluster_addr': '172.21.15.49:6829/886735721', 'heartbeat_back_addr': '172.21.15.49:6833/886735721', 'heartbeat_front_addr': '172.21.15.49:6831/886735721', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '6fc64698-2d37-4db0-bf3d-c7e612c45163', 'up': 1, 'in': 1, 'weight': 0.7888946533203125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 496, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6800', 'nonce': 1702447776}, {'type': 'v1', 'addr': '172.21.15.187:6801', 'nonce': 1702447776}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6802', 'nonce': 1702447776}, {'type': 'v1', 'addr': '172.21.15.187:6803', 'nonce': 1702447776}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6806', 'nonce': 1702447776}, {'type': 'v1', 'addr': '172.21.15.187:6807', 'nonce': 1702447776}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6804', 'nonce': 1702447776}, {'type': 'v1', 'addr': '172.21.15.187:6805', 'nonce': 1702447776}]}, 'public_addr': '172.21.15.187:6801/1702447776', 'cluster_addr': '172.21.15.187:6803/1702447776', 'heartbeat_back_addr': '172.21.15.187:6807/1702447776', 'heartbeat_front_addr': '172.21.15.187:6805/1702447776', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '8d7f62a1-7e92-4e06-9cad-86b96be514d8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 39, 'last_clean_end': 324, 'up_from': 345, 'up_thru': 496, 'down_at': 325, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6808', 'nonce': 1595353965}, {'type': 'v1', 'addr': '172.21.15.187:6809', 'nonce': 1595353965}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6810', 'nonce': 1595353965}, {'type': 'v1', 'addr': '172.21.15.187:6811', 'nonce': 1595353965}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6814', 'nonce': 1595353965}, {'type': 'v1', 'addr': '172.21.15.187:6815', 'nonce': 1595353965}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6812', 'nonce': 1595353965}, {'type': 'v1', 'addr': '172.21.15.187:6813', 'nonce': 1595353965}]}, 'public_addr': '172.21.15.187:6809/1595353965', 'cluster_addr': '172.21.15.187:6811/1595353965', 'heartbeat_back_addr': '172.21.15.187:6815/1595353965', 'heartbeat_front_addr': '172.21.15.187:6813/1595353965', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '3a7afd13-26ce-444c-9cc0-81ce2eab6ce5', 'up': 1, 'in': 1, 'weight': 0.9076080322265625, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 497, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6816', 'nonce': 2728426525}, {'type': 'v1', 'addr': '172.21.15.187:6817', 'nonce': 2728426525}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6818', 'nonce': 2728426525}, {'type': 'v1', 'addr': '172.21.15.187:6819', 'nonce': 2728426525}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6822', 'nonce': 2728426525}, {'type': 'v1', 'addr': '172.21.15.187:6823', 'nonce': 2728426525}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6820', 'nonce': 2728426525}, {'type': 'v1', 'addr': '172.21.15.187:6821', 'nonce': 2728426525}]}, 'public_addr': '172.21.15.187:6817/2728426525', 'cluster_addr': '172.21.15.187:6819/2728426525', 'heartbeat_back_addr': '172.21.15.187:6823/2728426525', 'heartbeat_front_addr': '172.21.15.187:6821/2728426525', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1e08d891-8747-402e-b4c2-aa3802ce8537', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 54, 'last_clean_end': 272, 'up_from': 298, 'up_thru': 461, 'down_at': 273, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 13894964}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 13894964}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 13894964}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 13894964}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6830', 'nonce': 13894964}, {'type': 'v1', 'addr': '172.21.15.187:6831', 'nonce': 13894964}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6828', 'nonce': 13894964}, {'type': 'v1', 'addr': '172.21.15.187:6829', 'nonce': 13894964}]}, 'public_addr': '172.21.15.187:6825/13894964', 'cluster_addr': '172.21.15.187:6827/13894964', 'heartbeat_back_addr': '172.21.15.187:6831/13894964', 'heartbeat_front_addr': '172.21.15.187:6829/13894964', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T14:07:51.070593+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T14:07:20.759253+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T14:07:51.524983+0000', 'dead_epoch': 100}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T14:08:08.471904+0000', 'dead_epoch': 214}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T14:08:03.561026+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T14:08:06.355614+0000', 'dead_epoch': 324}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T14:07:16.644516+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T14:07:20.884792+0000', 'dead_epoch': 272}], 'pg_upmap': [{'pgid': '3.14', 'osds': [7, 6, 0]}], 'pg_upmap_items': [{'pgid': '4.2', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.6', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '5.16', 'mappings': [{'from': 1, 'to': 6}, {'from': 3, 'to': 5}]}, {'pgid': '6.7', 'mappings': [{'from': 3, 'to': 7}]}, {'pgid': '6.b', 'mappings': [{'from': 3, 'to': 1}, {'from': 7, 'to': 6}]}, {'pgid': '6.1f', 'mappings': [{'from': 3, 'to': 0}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '11.0', 'osds': [1, 7, 3]}, {'pgid': '11.1', 'osds': [6, 0, 3]}, {'pgid': '11.2', 'osds': [0, 2, 3]}, {'pgid': '11.8', 'osds': [6, 5, 3]}, {'pgid': '11.9', 'osds': [6, 1, 3]}], 'primary_temp': [], 'blocklist': {'172.21.15.49:0/2731793917': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:0/799831425': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:6801/2518197280': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:0/3556829486': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:0/1249926115': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:0/231530138': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:0/4220781134': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:0/202325455': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:6800/4129636473': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:6800/2518197280': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:0/3514741273': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:6801/4129636473': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:0/3125743776': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:0/3600977363': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:6801/3453571598': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:6800/3453571598': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:0/3553734432': '2024-09-22T13:49:27.311313+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-21T14:08:28.814 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T14:08:28.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:28.824 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:28.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:28.925 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:28.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:28.979 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:28.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:29.080 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:08:29.135 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:29.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:29.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:08:29.237 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:08:29.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:29.292 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:29.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:08:29.393 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:29.436 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:29.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:29.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:29.538 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:29.569 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:29.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:29.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:29.670 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:08:29.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:29.706 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:29.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:08:29.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:29 smithi049 ceph-mon[30080]: pgmap v998: 168 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 7.4 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 307 B/s rd, 53 MiB/s wr, 990 op/s; 32006/366387 objects misplaced (8.736%); 37 MiB/s, 589 objects/s recovering 2024-09-21T14:08:29.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:29 smithi049 ceph-mon[40489]: pgmap v998: 168 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 7.4 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 307 B/s rd, 53 MiB/s wr, 990 op/s; 32006/366387 objects misplaced (8.736%); 37 MiB/s, 589 objects/s recovering 2024-09-21T14:08:29.781 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T14:08:29.781 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:08:29.807 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:08:29.849 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:29.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:29.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:08:29.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:29 smithi187 ceph-mon[33032]: pgmap v998: 168 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 7.4 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 307 B/s rd, 53 MiB/s wr, 990 op/s; 32006/366387 objects misplaced (8.736%); 37 MiB/s, 589 objects/s recovering 2024-09-21T14:08:29.951 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:29.992 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:29.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:29.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:30.080 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:08:30.094 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:08:30.123 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:30.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:30.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:08:30.225 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:30.255 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:30.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:30.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:30.360 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:08:30.401 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:30.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:30.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:08:30.444 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:08:30 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:08:30.502 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:08:30.544 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:30.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:30.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:08:30.575 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:08:30.642 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T14:08:30.646 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:30.679 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:30.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:30.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:30.781 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:30.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:30.828 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:30.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:30.874 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:30 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:08:30.875 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:30 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:08:30.875 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:30 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:08:30.875 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:30 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:08:30.875 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:08:30 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:08:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:08:30.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:30 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:08:30.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:30 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:08:30.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:30 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:08:30.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:30 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:08:30.929 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:08:30.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:30 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:08:30.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:30 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:08:30.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:30 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:08:30.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:30 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:08:30.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:30.979 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:30.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:08:31.080 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:31.108 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:31.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:31.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:31.210 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:08:31.262 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:31.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:31.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:08:31.364 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:08:31.407 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:31.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:31.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:08:31.509 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:08:31.551 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:31.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:31.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:08:31.652 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:31.668 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T14:08:31.685 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:31.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:31.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:31.787 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:08:31.851 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:31.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:31.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:08:31.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:31 smithi187 ceph-mon[33032]: pgmap v999: 168 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 7.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 511 B/s rd, 34 MiB/s wr, 1.11k op/s; 27044/353706 objects misplaced (7.646%); 43 MiB/s, 690 objects/s recovering 2024-09-21T14:08:31.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:31 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:08:31.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:31 smithi187 ceph-mon[33032]: osdmap e500: 8 total, 8 up, 8 in 2024-09-21T14:08:31.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:31 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:08:31.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:31 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:08:31.953 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:32.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:31 smithi049 ceph-mon[30080]: pgmap v999: 168 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 7.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 511 B/s rd, 34 MiB/s wr, 1.11k op/s; 27044/353706 objects misplaced (7.646%); 43 MiB/s, 690 objects/s recovering 2024-09-21T14:08:32.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:31 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:08:32.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:31 smithi049 ceph-mon[30080]: osdmap e500: 8 total, 8 up, 8 in 2024-09-21T14:08:32.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:31 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:08:32.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:31 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:08:32.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:31 smithi049 ceph-mon[40489]: pgmap v999: 168 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 7.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 511 B/s rd, 34 MiB/s wr, 1.11k op/s; 27044/353706 objects misplaced (7.646%); 43 MiB/s, 690 objects/s recovering 2024-09-21T14:08:32.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:31 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:08:32.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:31 smithi049 ceph-mon[40489]: osdmap e500: 8 total, 8 up, 8 in 2024-09-21T14:08:32.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:31 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:08:32.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:31 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:08:32.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:32.007 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:32.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:32.107 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:32.156 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:32.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:32.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:32.258 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:08:32.295 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:32.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:32.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:08:32.397 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:32.433 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:32.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:32.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:32.535 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:08:32.558 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:32.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:32.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:08:32.659 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:32.692 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:32.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:32.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:32.793 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:32.834 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:32.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:32.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:32.934 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:32 smithi049 ceph-mon[40489]: osdmap e501: 8 total, 8 up, 8 in 2024-09-21T14:08:32.935 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:32 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/4239793478' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:08:32.935 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:32 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:08:32.935 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:32 smithi049 ceph-mon[30080]: osdmap e501: 8 total, 8 up, 8 in 2024-09-21T14:08:32.935 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:32 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/4239793478' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:08:32.935 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:32 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:08:32.935 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:08:32.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:32 smithi187 ceph-mon[33032]: osdmap e501: 8 total, 8 up, 8 in 2024-09-21T14:08:32.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:32 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/4239793478' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:08:32.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:32 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:08:32.958 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:32.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:32.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:08:33.060 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:08:33.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:33.098 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:33.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:08:33.199 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:08:33.234 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:33.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:33.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:08:33.337 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:08:33.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:33.368 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:33.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:08:33.469 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:08:33.496 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:33.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:33.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:08:33.598 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:33.624 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:33.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:33.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:33.726 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:08:33.760 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:33.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:33.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:08:33.862 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:33.901 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T14:08:33.902 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:33 smithi049 ceph-mon[40489]: pgmap v1002: 168 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 7.1 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.4 KiB/s rd, 15 MiB/s wr, 1.10k op/s; 27044/349494 objects misplaced (7.738%); 39 MiB/s, 625 objects/s recovering 2024-09-21T14:08:33.902 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:33 smithi049 ceph-mon[40489]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:08:33.902 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:33 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T14:08:33.902 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:33 smithi049 ceph-mon[40489]: osdmap e502: 8 total, 8 up, 8 in 2024-09-21T14:08:33.902 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:33 smithi049 ceph-mon[40489]: from='client.26531 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:08:33.903 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:33 smithi049 ceph-mon[30080]: pgmap v1002: 168 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 7.1 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.4 KiB/s rd, 15 MiB/s wr, 1.10k op/s; 27044/349494 objects misplaced (7.738%); 39 MiB/s, 625 objects/s recovering 2024-09-21T14:08:33.904 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:33 smithi049 ceph-mon[30080]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:08:33.904 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:33 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T14:08:33.904 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:33 smithi049 ceph-mon[30080]: osdmap e502: 8 total, 8 up, 8 in 2024-09-21T14:08:33.904 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:33 smithi049 ceph-mon[30080]: from='client.26531 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:08:33.907 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:33.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:33.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:33.911 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-09-21T14:08:33.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:33 smithi187 ceph-mon[33032]: pgmap v1002: 168 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 7.1 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.4 KiB/s rd, 15 MiB/s wr, 1.10k op/s; 27044/349494 objects misplaced (7.738%); 39 MiB/s, 625 objects/s recovering 2024-09-21T14:08:33.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:33 smithi187 ceph-mon[33032]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:08:33.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:33 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T14:08:33.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:33 smithi187 ceph-mon[33032]: osdmap e502: 8 total, 8 up, 8 in 2024-09-21T14:08:33.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:33 smithi187 ceph-mon[33032]: from='client.26531 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:08:34.010 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:08:34.033 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:34.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:34.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:08:34.135 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:34.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:34.175 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:34.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:34.276 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:08:34.313 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:34.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:34.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:08:34.415 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:34.446 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:34.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:34.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:34.548 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:34.582 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:34.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:34.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:34.684 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:08:34.718 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:34.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:34.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:08:34.820 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:34.851 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:34.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:34.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:34.954 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:08:34.987 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:34.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:34.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:08:35.089 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:35.117 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:35.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:35.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:35.219 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:35.253 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:35.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:35.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:35.355 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:35.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:35.387 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:35.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:35.488 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:35.524 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:35.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:35.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:35.626 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:08:35.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:35.668 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:35.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:08:35.769 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:08:35.799 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:35.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:35.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:08:35.901 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:35.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:35.937 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:35.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:35.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:35 smithi187 ceph-mon[33032]: pgmap v1004: 168 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 7.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1023 B/s rd, 427 KiB/s wr, 1.33k op/s; 24084/342753 objects misplaced (7.027%); 61 MiB/s, 978 objects/s recovering 2024-09-21T14:08:36.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:35 smithi049 ceph-mon[40489]: pgmap v1004: 168 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 7.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1023 B/s rd, 427 KiB/s wr, 1.33k op/s; 24084/342753 objects misplaced (7.027%); 61 MiB/s, 978 objects/s recovering 2024-09-21T14:08:36.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:35 smithi049 ceph-mon[30080]: pgmap v1004: 168 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 7.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1023 B/s rd, 427 KiB/s wr, 1.33k op/s; 24084/342753 objects misplaced (7.027%); 61 MiB/s, 978 objects/s recovering 2024-09-21T14:08:36.038 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:36.062 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:36.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:36.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:36.164 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:08:36.200 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:36.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:36.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:08:36.247 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-09-21T14:08:36.303 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:36.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:36.346 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:36.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:36.447 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:08:36.485 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:36.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:36.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:08:36.587 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:08:36.631 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:36.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:36.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:08:36.733 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:36.764 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:36.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:36.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:36.866 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:08:36.903 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:36.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:36.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:08:37.005 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:37.036 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:37.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:37.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:37.138 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:08:37.173 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:37.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:37.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:08:37.275 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:37.305 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:37.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:37.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:37.407 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:08:37.455 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:37.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:37.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:08:37.557 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:37.601 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:37.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:37.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:37.704 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:37.734 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:37.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:37.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:37.835 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:37.858 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:37 smithi049 ceph-mon[40489]: pgmap v1005: 168 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 6.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 965 op/s; 22029/336339 objects misplaced (6.550%); 41 MiB/s, 654 objects/s recovering 2024-09-21T14:08:37.859 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:37 smithi049 ceph-mon[30080]: pgmap v1005: 168 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 6.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 965 op/s; 22029/336339 objects misplaced (6.550%); 41 MiB/s, 654 objects/s recovering 2024-09-21T14:08:37.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:37.872 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:37.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:37.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:37 smithi187 ceph-mon[33032]: pgmap v1005: 168 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 6.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 965 op/s; 22029/336339 objects misplaced (6.550%); 41 MiB/s, 654 objects/s recovering 2024-09-21T14:08:37.974 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:38.015 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:38.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:38.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:38.117 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:38.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:38.163 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:38.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:38.248 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":1003,"stamp":"2024-09-21T14:08:32.651531+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7622083806,"num_objects":116498,"num_object_clones":0,"num_object_copies":349494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":27044,"num_objects_unfound":0,"num_objects_dirty":116498,"num_whiteouts":0,"num_read":1203,"num_read_kb":955,"num_write":131177,"num_write_kb":7896110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9722,"num_bytes_recovered":620959151,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70362,"ondisk_log_size":70362,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":509,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":28027712,"kb_used_data":22971712,"kb_used_omap":23,"kb_used_meta":5055464,"kb_avail":721933504,"statfs":{"total":767960285184,"available":739259908096,"internally_reserved":0,"allocated":23523033088,"data_stored":23520446577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24148,"internal_metadata":5176795564},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-377421800,"num_objects":-5758,"num_object_clones":0,"num_object_copies":-17274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-4962,"num_objects_unfound":0,"num_objects_dirty":-5758,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":6406,"num_write_kb":20673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3587,"num_bytes_recovered":235077632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.507238"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":841,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-21T14:08:30.622888+0000","last_change":"2024-09-21T14:05:10.397918+0000","last_active":"2024-09-21T14:08:30.622888+0000","last_peered":"2024-09-21T14:08:30.622888+0000","last_clean":"2024-09-21T14:08:30.622888+0000","last_became_active":"2024-09-21T14:04:21.204299+0000","last_became_peered":"2024-09-21T14:04:21.204299+0000","last_unstale":"2024-09-21T14:08:30.622888+0000","last_undegraded":"2024-09-21T14:08:30.622888+0000","last_fullsized":"2024-09-21T14:08:30.622888+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:10.397765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:05:10.397765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:28.238296+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"290'15","reported_seq":1000,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.462403+0000","last_change":"2024-09-21T14:07:37.119071+0000","last_active":"2024-09-21T14:08:20.462403+0000","last_peered":"2024-09-21T14:08:20.462403+0000","last_clean":"2024-09-21T14:08:20.462403+0000","last_became_active":"2024-09-21T13:58:17.771220+0000","last_became_peered":"2024-09-21T13:58:17.771220+0000","last_unstale":"2024-09-21T14:08:20.462403+0000","last_undegraded":"2024-09-21T14:08:20.462403+0000","last_fullsized":"2024-09-21T14:08:20.462403+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'15","last_scrub_stamp":"2024-09-21T14:07:37.118971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:07:37.118971+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:54.190946+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":949,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.482861+0000","last_change":"2024-09-21T14:08:09.021013+0000","last_active":"2024-09-21T14:08:20.482861+0000","last_peered":"2024-09-21T14:08:20.482861+0000","last_clean":"2024-09-21T14:08:20.482861+0000","last_became_active":"2024-09-21T14:01:41.202168+0000","last_became_peered":"2024-09-21T14:01:41.202168+0000","last_unstale":"2024-09-21T14:08:20.482861+0000","last_undegraded":"2024-09-21T14:08:20.482861+0000","last_fullsized":"2024-09-21T14:08:20.482861+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:08:09.020943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:08:09.020943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:36.782820+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":924,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.462494+0000","last_change":"2024-09-21T14:07:45.174462+0000","last_active":"2024-09-21T14:08:20.462494+0000","last_peered":"2024-09-21T14:08:20.462494+0000","last_clean":"2024-09-21T14:08:20.462494+0000","last_became_active":"2024-09-21T14:01:41.203275+0000","last_became_peered":"2024-09-21T14:01:41.203275+0000","last_unstale":"2024-09-21T14:08:20.462494+0000","last_undegraded":"2024-09-21T14:08:20.462494+0000","last_fullsized":"2024-09-21T14:08:20.462494+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:07:45.174406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:07:45.174406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:01.573461+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":481,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.470946+0000","last_change":"2024-09-21T14:08:14.411821+0000","last_active":"2024-09-21T14:08:20.470946+0000","last_peered":"2024-09-21T14:08:20.470946+0000","last_clean":"2024-09-21T14:08:20.470946+0000","last_became_active":"2024-09-21T14:08:14.411266+0000","last_became_peered":"2024-09-21T14:08:14.411266+0000","last_unstale":"2024-09-21T14:08:20.470946+0000","last_undegraded":"2024-09-21T14:08:20.470946+0000","last_fullsized":"2024-09-21T14:08:20.470946+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:00.241562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:04:00.241562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":435,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.471847+0000","last_change":"2024-09-21T14:08:14.411332+0000","last_active":"2024-09-21T14:08:20.471847+0000","last_peered":"2024-09-21T14:08:20.471847+0000","last_clean":"2024-09-21T14:08:20.471847+0000","last_became_active":"2024-09-21T14:08:14.411010+0000","last_became_peered":"2024-09-21T14:08:14.411010+0000","last_unstale":"2024-09-21T14:08:20.471847+0000","last_undegraded":"2024-09-21T14:08:20.471847+0000","last_fullsized":"2024-09-21T14:08:20.471847+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:07:33.294212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:07:33.294212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:49.264157+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"290'27","reported_seq":473,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.466656+0000","last_change":"2024-09-21T14:08:14.433447+0000","last_active":"2024-09-21T14:08:20.466656+0000","last_peered":"2024-09-21T14:08:20.466656+0000","last_clean":"2024-09-21T14:08:20.466656+0000","last_became_active":"2024-09-21T14:08:14.415832+0000","last_became_peered":"2024-09-21T14:08:14.415832+0000","last_unstale":"2024-09-21T14:08:20.466656+0000","last_undegraded":"2024-09-21T14:08:20.466656+0000","last_fullsized":"2024-09-21T14:08:20.466656+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'27","last_scrub_stamp":"2024-09-21T14:04:08.320339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:04:08.320339+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":645,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.470520+0000","last_change":"2024-09-21T14:04:35.907676+0000","last_active":"2024-09-21T14:08:20.470520+0000","last_peered":"2024-09-21T14:08:20.470520+0000","last_clean":"2024-09-21T14:08:20.470520+0000","last_became_active":"2024-09-21T14:04:21.208237+0000","last_became_peered":"2024-09-21T14:04:21.208237+0000","last_unstale":"2024-09-21T14:08:20.470520+0000","last_undegraded":"2024-09-21T14:08:20.470520+0000","last_fullsized":"2024-09-21T14:08:20.470520+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:35.907611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:04:35.907611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":900,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-21T14:08:30.622431+0000","last_change":"2024-09-21T14:05:45.271378+0000","last_active":"2024-09-21T14:08:30.622431+0000","last_peered":"2024-09-21T14:08:30.622431+0000","last_clean":"2024-09-21T14:08:30.622431+0000","last_became_active":"2024-09-21T14:01:41.194517+0000","last_became_peered":"2024-09-21T14:01:41.194517+0000","last_unstale":"2024-09-21T14:08:30.622431+0000","last_undegraded":"2024-09-21T14:08:30.622431+0000","last_fullsized":"2024-09-21T14:08:30.622431+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:45.271320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:05:45.271320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:31.504720+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"290'15","reported_seq":748,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-21T14:08:30.628932+0000","last_change":"2024-09-21T14:08:03.123287+0000","last_active":"2024-09-21T14:08:30.628932+0000","last_peered":"2024-09-21T14:08:30.628932+0000","last_clean":"2024-09-21T14:08:30.628932+0000","last_became_active":"2024-09-21T14:04:21.208244+0000","last_became_peered":"2024-09-21T14:04:21.208244+0000","last_unstale":"2024-09-21T14:08:30.628932+0000","last_undegraded":"2024-09-21T14:08:30.628932+0000","last_fullsized":"2024-09-21T14:08:30.628932+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'15","last_scrub_stamp":"2024-09-21T14:08:03.123210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:08:03.123210+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:05.027710+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":857,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-21T14:08:30.621571+0000","last_change":"2024-09-21T14:07:42.488397+0000","last_active":"2024-09-21T14:08:30.621571+0000","last_peered":"2024-09-21T14:08:30.621571+0000","last_clean":"2024-09-21T14:08:30.621571+0000","last_became_active":"2024-09-21T14:01:41.188248+0000","last_became_peered":"2024-09-21T14:01:41.188248+0000","last_unstale":"2024-09-21T14:08:30.621571+0000","last_undegraded":"2024-09-21T14:08:30.621571+0000","last_fullsized":"2024-09-21T14:08:30.621571+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:07:42.488329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:07:42.488329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:45.682775+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":885,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.488871+0000","last_change":"2024-09-21T14:06:50.577626+0000","last_active":"2024-09-21T14:08:20.488871+0000","last_peered":"2024-09-21T14:08:20.488871+0000","last_clean":"2024-09-21T14:08:20.488871+0000","last_became_active":"2024-09-21T14:04:21.205416+0000","last_became_peered":"2024-09-21T14:04:21.205416+0000","last_unstale":"2024-09-21T14:08:20.488871+0000","last_undegraded":"2024-09-21T14:08:20.488871+0000","last_fullsized":"2024-09-21T14:08:20.488871+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:50.577560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:06:50.577560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":877,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-21T14:08:30.621606+0000","last_change":"2024-09-21T14:05:46.334192+0000","last_active":"2024-09-21T14:08:30.621606+0000","last_peered":"2024-09-21T14:08:30.621606+0000","last_clean":"2024-09-21T14:08:30.621606+0000","last_became_active":"2024-09-21T14:01:41.189373+0000","last_became_peered":"2024-09-21T14:01:41.189373+0000","last_unstale":"2024-09-21T14:08:30.621606+0000","last_undegraded":"2024-09-21T14:08:30.621606+0000","last_fullsized":"2024-09-21T14:08:30.621606+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:46.334117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:05:46.334117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"290'7","reported_seq":740,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-21T14:08:30.627768+0000","last_change":"2024-09-21T14:06:19.344847+0000","last_active":"2024-09-21T14:08:30.627768+0000","last_peered":"2024-09-21T14:08:30.627768+0000","last_clean":"2024-09-21T14:08:30.627768+0000","last_became_active":"2024-09-21T14:04:21.213102+0000","last_became_peered":"2024-09-21T14:04:21.213102+0000","last_unstale":"2024-09-21T14:08:30.627768+0000","last_undegraded":"2024-09-21T14:08:30.627768+0000","last_fullsized":"2024-09-21T14:08:30.627768+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'7","last_scrub_stamp":"2024-09-21T14:06:19.344638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:06:19.344638+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:59.120217+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":475,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.462595+0000","last_change":"2024-09-21T14:08:17.147423+0000","last_active":"2024-09-21T14:08:20.462595+0000","last_peered":"2024-09-21T14:08:20.462595+0000","last_clean":"2024-09-21T14:08:20.462595+0000","last_became_active":"2024-09-21T14:08:04.295165+0000","last_became_peered":"2024-09-21T14:08:04.295165+0000","last_unstale":"2024-09-21T14:08:20.462595+0000","last_undegraded":"2024-09-21T14:08:20.462595+0000","last_fullsized":"2024-09-21T14:08:20.462595+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:08:17.147364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:08:17.147364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:39.953976+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":918,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-21T14:08:30.621009+0000","last_change":"2024-09-21T14:08:14.419726+0000","last_active":"2024-09-21T14:08:30.621009+0000","last_peered":"2024-09-21T14:08:30.621009+0000","last_clean":"2024-09-21T14:08:30.621009+0000","last_became_active":"2024-09-21T14:08:14.419373+0000","last_became_peered":"2024-09-21T14:08:14.419373+0000","last_unstale":"2024-09-21T14:08:30.621009+0000","last_undegraded":"2024-09-21T14:08:30.621009+0000","last_fullsized":"2024-09-21T14:08:30.621009+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:07:04.491870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:07:04.491870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":967,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.463602+0000","last_change":"2024-09-21T14:07:06.241357+0000","last_active":"2024-09-21T14:08:20.463602+0000","last_peered":"2024-09-21T14:08:20.463602+0000","last_clean":"2024-09-21T14:08:20.463602+0000","last_became_active":"2024-09-21T14:01:41.200360+0000","last_became_peered":"2024-09-21T14:01:41.200360+0000","last_unstale":"2024-09-21T14:08:20.463602+0000","last_undegraded":"2024-09-21T14:08:20.463602+0000","last_fullsized":"2024-09-21T14:08:20.463602+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:07:06.241295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:07:06.241295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:22.706664+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":892,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.462824+0000","last_change":"2024-09-21T14:08:18.118600+0000","last_active":"2024-09-21T14:08:20.462824+0000","last_peered":"2024-09-21T14:08:20.462824+0000","last_clean":"2024-09-21T14:08:20.462824+0000","last_became_active":"2024-09-21T14:04:21.202178+0000","last_became_peered":"2024-09-21T14:04:21.202178+0000","last_unstale":"2024-09-21T14:08:20.462824+0000","last_undegraded":"2024-09-21T14:08:20.462824+0000","last_fullsized":"2024-09-21T14:08:20.462824+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:08:18.118531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:08:18.118531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:44.324536+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"290'27","reported_seq":980,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.476261+0000","last_change":"2024-09-21T14:08:14.436854+0000","last_active":"2024-09-21T14:08:20.476261+0000","last_peered":"2024-09-21T14:08:20.476261+0000","last_clean":"2024-09-21T14:08:20.476261+0000","last_became_active":"2024-09-21T14:08:14.424393+0000","last_became_peered":"2024-09-21T14:08:14.424393+0000","last_unstale":"2024-09-21T14:08:20.476261+0000","last_undegraded":"2024-09-21T14:08:20.476261+0000","last_fullsized":"2024-09-21T14:08:20.476261+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'27","last_scrub_stamp":"2024-09-21T14:07:00.227334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:07:00.227334+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":941,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.479403+0000","last_change":"2024-09-21T14:05:59.399621+0000","last_active":"2024-09-21T14:08:20.479403+0000","last_peered":"2024-09-21T14:08:20.479403+0000","last_clean":"2024-09-21T14:08:20.479403+0000","last_became_active":"2024-09-21T14:04:21.203937+0000","last_became_peered":"2024-09-21T14:04:21.203937+0000","last_unstale":"2024-09-21T14:08:20.479403+0000","last_undegraded":"2024-09-21T14:08:20.479403+0000","last_fullsized":"2024-09-21T14:08:20.479403+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:59.399552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:05:59.399552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:23.657050+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":159,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.482832+0000","last_change":"2024-09-21T14:07:34.561619+0000","last_active":"2024-09-21T14:08:20.482832+0000","last_peered":"2024-09-21T14:08:20.482832+0000","last_clean":"2024-09-21T14:08:20.482832+0000","last_became_active":"2024-09-21T14:05:30.241942+0000","last_became_peered":"2024-09-21T14:05:30.241942+0000","last_unstale":"2024-09-21T14:08:20.482832+0000","last_undegraded":"2024-09-21T14:08:20.482832+0000","last_fullsized":"2024-09-21T14:08:20.482832+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:07:34.561444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:07:34.561444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:38.250854+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":936,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.462786+0000","last_change":"2024-09-21T14:07:36.165505+0000","last_active":"2024-09-21T14:08:20.462786+0000","last_peered":"2024-09-21T14:08:20.462786+0000","last_clean":"2024-09-21T14:08:20.462786+0000","last_became_active":"2024-09-21T13:52:04.316016+0000","last_became_peered":"2024-09-21T13:52:04.316016+0000","last_unstale":"2024-09-21T14:08:20.462786+0000","last_undegraded":"2024-09-21T14:08:20.462786+0000","last_fullsized":"2024-09-21T14:08:20.462786+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:07:36.165453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:07:36.165453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:52.814512+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"290'18","reported_seq":473,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.479577+0000","last_change":"2024-09-21T14:08:14.427628+0000","last_active":"2024-09-21T14:08:20.479577+0000","last_peered":"2024-09-21T14:08:20.479577+0000","last_clean":"2024-09-21T14:08:20.479577+0000","last_became_active":"2024-09-21T14:08:14.407320+0000","last_became_peered":"2024-09-21T14:08:14.407320+0000","last_unstale":"2024-09-21T14:08:20.479577+0000","last_undegraded":"2024-09-21T14:08:20.479577+0000","last_fullsized":"2024-09-21T14:08:20.479577+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'18","last_scrub_stamp":"2024-09-21T14:06:49.375262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:06:49.375262+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":23,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":931,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.463069+0000","last_change":"2024-09-21T14:07:03.254086+0000","last_active":"2024-09-21T14:08:20.463069+0000","last_peered":"2024-09-21T14:08:20.463069+0000","last_clean":"2024-09-21T14:08:20.463069+0000","last_became_active":"2024-09-21T13:57:12.681897+0000","last_became_peered":"2024-09-21T13:57:12.681897+0000","last_unstale":"2024-09-21T14:08:20.463069+0000","last_undegraded":"2024-09-21T14:08:20.463069+0000","last_fullsized":"2024-09-21T14:08:20.463069+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:07:03.254025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:07:03.254025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:27.756000+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":104,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-21T14:08:30.621461+0000","last_change":"2024-09-21T14:08:04.415897+0000","last_active":"2024-09-21T14:08:30.621461+0000","last_peered":"2024-09-21T14:08:30.621461+0000","last_clean":"2024-09-21T14:08:30.621461+0000","last_became_active":"2024-09-21T14:08:04.288485+0000","last_became_peered":"2024-09-21T14:08:04.288485+0000","last_unstale":"2024-09-21T14:08:30.621461+0000","last_undegraded":"2024-09-21T14:08:30.621461+0000","last_fullsized":"2024-09-21T14:08:30.621461+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T14:08:04.415821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:08:04.415821+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:17.328368+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":958,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.482997+0000","last_change":"2024-09-21T14:06:46.580150+0000","last_active":"2024-09-21T14:08:20.482997+0000","last_peered":"2024-09-21T14:08:20.482997+0000","last_clean":"2024-09-21T14:08:20.482997+0000","last_became_active":"2024-09-21T14:04:21.200889+0000","last_became_peered":"2024-09-21T14:04:21.200889+0000","last_unstale":"2024-09-21T14:08:20.482997+0000","last_undegraded":"2024-09-21T14:08:20.482997+0000","last_fullsized":"2024-09-21T14:08:20.482997+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T14:06:46.580099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:06:46.580099+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"290'20","reported_seq":293,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.483381+0000","last_change":"2024-09-21T14:08:14.432462+0000","last_active":"2024-09-21T14:08:20.483381+0000","last_peered":"2024-09-21T14:08:20.483381+0000","last_clean":"2024-09-21T14:08:20.483381+0000","last_became_active":"2024-09-21T14:08:14.411798+0000","last_became_peered":"2024-09-21T14:08:14.411798+0000","last_unstale":"2024-09-21T14:08:20.483381+0000","last_undegraded":"2024-09-21T14:08:20.483381+0000","last_fullsized":"2024-09-21T14:08:20.483381+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'20","last_scrub_stamp":"2024-09-21T14:05:56.301903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:05:56.301903+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"409'13","reported_seq":642,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:23.359047+0000","last_change":"2024-09-21T14:07:46.640741+0000","last_active":"2024-09-21T14:08:23.359047+0000","last_peered":"2024-09-21T14:08:23.359047+0000","last_clean":"2024-09-21T14:08:23.359047+0000","last_became_active":"2024-09-21T14:04:21.206136+0000","last_became_peered":"2024-09-21T14:04:21.206136+0000","last_unstale":"2024-09-21T14:08:23.359047+0000","last_undegraded":"2024-09-21T14:08:23.359047+0000","last_fullsized":"2024-09-21T14:08:23.359047+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"409'13","last_scrub_stamp":"2024-09-21T14:07:46.640665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:07:46.640665+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:08.548272+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.0","version":"499'8144","reported_seq":11641,"reported_epoch":499,"state":"active+remapped+backfilling","last_fresh":"2024-09-21T14:08:30.089539+0000","last_change":"2024-09-21T14:08:15.458122+0000","last_active":"2024-09-21T14:08:30.089539+0000","last_peered":"2024-09-21T14:08:30.089539+0000","last_clean":"2024-09-21T14:08:13.356726+0000","last_became_active":"2024-09-21T14:08:15.445581+0000","last_became_peered":"2024-09-21T14:08:15.445581+0000","last_unstale":"2024-09-21T14:08:30.089539+0000","last_undegraded":"2024-09-21T14:08:30.089539+0000","last_fullsized":"2024-09-21T14:08:30.089539+0000","mapping_epoch":497,"log_start":"477'4389","ondisk_log_start":"477'4389","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_clean_scrub_stamp":"2024-09-21T14:06:44.365469+0000","objects_scrubbed":0,"log_size":3755,"log_dups_size":0,"ondisk_log_size":3755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":472776704,"num_objects":7214,"num_object_clones":0,"num_object_copies":21642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5574,"num_objects_unfound":0,"num_objects_dirty":7214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8144,"num_write_kb":491456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1702,"num_bytes_recovered":111542272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,3],"avail_no_missing":["1","3","7"],"object_location_counts":[{"shards":"1,3,7","objects":7214}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"75'79","reported_seq":354,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-21T14:08:30.622541+0000","last_change":"2024-09-21T14:05:38.314512+0000","last_active":"2024-09-21T14:08:30.622541+0000","last_peered":"2024-09-21T14:08:30.622541+0000","last_clean":"2024-09-21T14:08:30.622541+0000","last_became_active":"2024-09-21T14:01:41.246888+0000","last_became_peered":"2024-09-21T14:01:41.246888+0000","last_unstale":"2024-09-21T14:08:30.622541+0000","last_undegraded":"2024-09-21T14:08:30.622541+0000","last_fullsized":"2024-09-21T14:08:30.622541+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:05:38.314431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:05:38.314431+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:29.287611+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"290'27","reported_seq":1001,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.479858+0000","last_change":"2024-09-21T14:08:12.954432+0000","last_active":"2024-09-21T14:08:20.479858+0000","last_peered":"2024-09-21T14:08:20.479858+0000","last_clean":"2024-09-21T14:08:20.479858+0000","last_became_active":"2024-09-21T13:57:12.695609+0000","last_became_peered":"2024-09-21T13:57:12.695609+0000","last_unstale":"2024-09-21T14:08:20.479858+0000","last_undegraded":"2024-09-21T14:08:20.479858+0000","last_fullsized":"2024-09-21T14:08:20.479858+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'27","last_scrub_stamp":"2024-09-21T14:08:12.954351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:08:12.954351+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:14.415512+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":466,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.488377+0000","last_change":"2024-09-21T14:08:14.411502+0000","last_active":"2024-09-21T14:08:20.488377+0000","last_peered":"2024-09-21T14:08:20.488377+0000","last_clean":"2024-09-21T14:08:20.488377+0000","last_became_active":"2024-09-21T14:08:14.410580+0000","last_became_peered":"2024-09-21T14:08:14.410580+0000","last_unstale":"2024-09-21T14:08:20.488377+0000","last_undegraded":"2024-09-21T14:08:20.488377+0000","last_fullsized":"2024-09-21T14:08:20.488377+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:07:41.211607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:07:41.211607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:02.183006+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"409'11","reported_seq":1018,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-21T14:08:30.622060+0000","last_change":"2024-09-21T14:04:21.211284+0000","last_active":"2024-09-21T14:08:30.622060+0000","last_peered":"2024-09-21T14:08:30.622060+0000","last_clean":"2024-09-21T14:08:30.622060+0000","last_became_active":"2024-09-21T14:04:21.206029+0000","last_became_peered":"2024-09-21T14:04:21.206029+0000","last_unstale":"2024-09-21T14:08:30.622060+0000","last_undegraded":"2024-09-21T14:08:30.622060+0000","last_fullsized":"2024-09-21T14:08:30.622060+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'10","last_scrub_stamp":"2024-09-21T14:02:59.831128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:02:59.831128+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:22.227561+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":814,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-21T14:08:30.621478+0000","last_change":"2024-09-21T14:05:48.393515+0000","last_active":"2024-09-21T14:08:30.621478+0000","last_peered":"2024-09-21T14:08:30.621478+0000","last_clean":"2024-09-21T14:08:30.621478+0000","last_became_active":"2024-09-21T14:04:21.203505+0000","last_became_peered":"2024-09-21T14:04:21.203505+0000","last_unstale":"2024-09-21T14:08:30.621478+0000","last_undegraded":"2024-09-21T14:08:30.621478+0000","last_fullsized":"2024-09-21T14:08:30.621478+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:48.393427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:05:48.393427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"499'8238","reported_seq":8410,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:30.068391+0000","last_change":"2024-09-21T14:08:13.669043+0000","last_active":"2024-09-21T14:08:30.068391+0000","last_peered":"2024-09-21T14:08:30.068391+0000","last_clean":"2024-09-21T14:08:30.068391+0000","last_became_active":"2024-09-21T14:08:04.291231+0000","last_became_peered":"2024-09-21T14:08:04.291231+0000","last_unstale":"2024-09-21T14:08:30.068391+0000","last_undegraded":"2024-09-21T14:08:30.068391+0000","last_fullsized":"2024-09-21T14:08:30.068391+0000","mapping_epoch":491,"log_start":"478'4562","ondisk_log_start":"478'4562","created":461,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"496'7220","last_scrub_stamp":"2024-09-21T14:08:13.668911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_clean_scrub_stamp":"2024-09-21T14:08:13.668911+0000","objects_scrubbed":7144,"log_size":3676,"log_dups_size":0,"ondisk_log_size":3676,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:15.744514+0000","scrub_duration":2668,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":471990272,"num_objects":7202,"num_object_clones":0,"num_object_copies":21606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8238,"num_write_kb":494080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"75'79","reported_seq":412,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.463449+0000","last_change":"2024-09-21T14:06:49.267268+0000","last_active":"2024-09-21T14:08:20.463449+0000","last_peered":"2024-09-21T14:08:20.463449+0000","last_clean":"2024-09-21T14:08:20.463449+0000","last_became_active":"2024-09-21T14:05:39.296209+0000","last_became_peered":"2024-09-21T14:05:39.296209+0000","last_unstale":"2024-09-21T14:08:20.463449+0000","last_undegraded":"2024-09-21T14:08:20.463449+0000","last_fullsized":"2024-09-21T14:08:20.463449+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:06:49.267210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:06:49.267210+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"290'30","reported_seq":737,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.483644+0000","last_change":"2024-09-21T14:08:14.441516+0000","last_active":"2024-09-21T14:08:20.483644+0000","last_peered":"2024-09-21T14:08:20.483644+0000","last_clean":"2024-09-21T14:08:20.483644+0000","last_became_active":"2024-09-21T14:08:14.420066+0000","last_became_peered":"2024-09-21T14:08:14.420066+0000","last_unstale":"2024-09-21T14:08:20.483644+0000","last_undegraded":"2024-09-21T14:08:20.483644+0000","last_fullsized":"2024-09-21T14:08:20.483644+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'30","last_scrub_stamp":"2024-09-21T14:07:42.181176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:07:42.181176+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:52.244662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":833,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-21T14:08:30.626267+0000","last_change":"2024-09-21T14:07:38.210925+0000","last_active":"2024-09-21T14:08:30.626267+0000","last_peered":"2024-09-21T14:08:30.626267+0000","last_clean":"2024-09-21T14:08:30.626267+0000","last_became_active":"2024-09-21T13:59:32.211004+0000","last_became_peered":"2024-09-21T13:59:32.211004+0000","last_unstale":"2024-09-21T14:08:30.626267+0000","last_undegraded":"2024-09-21T14:08:30.626267+0000","last_fullsized":"2024-09-21T14:08:30.626267+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:07:38.210848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:07:38.210848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:57.985606+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":948,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.462347+0000","last_change":"2024-09-21T14:08:14.395163+0000","last_active":"2024-09-21T14:08:20.462347+0000","last_peered":"2024-09-21T14:08:20.462347+0000","last_clean":"2024-09-21T14:08:20.462347+0000","last_became_active":"2024-09-21T14:08:14.394855+0000","last_became_peered":"2024-09-21T14:08:14.394855+0000","last_unstale":"2024-09-21T14:08:20.462347+0000","last_undegraded":"2024-09-21T14:08:20.462347+0000","last_fullsized":"2024-09-21T14:08:20.462347+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:29.295016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:06:29.295016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":393,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.473016+0000","last_change":"2024-09-21T14:08:14.409150+0000","last_active":"2024-09-21T14:08:20.473016+0000","last_peered":"2024-09-21T14:08:20.473016+0000","last_clean":"2024-09-21T14:08:20.473016+0000","last_became_active":"2024-09-21T14:08:14.408836+0000","last_became_peered":"2024-09-21T14:08:14.408836+0000","last_unstale":"2024-09-21T14:08:20.473016+0000","last_undegraded":"2024-09-21T14:08:20.473016+0000","last_fullsized":"2024-09-21T14:08:20.473016+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T13:59:23.334162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T13:59:23.334162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"499'8229","reported_seq":8316,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:30.064798+0000","last_change":"2024-09-21T14:06:45.309299+0000","last_active":"2024-09-21T14:08:30.064798+0000","last_peered":"2024-09-21T14:08:30.064798+0000","last_clean":"2024-09-21T14:08:30.064798+0000","last_became_active":"2024-09-21T14:06:45.308994+0000","last_became_peered":"2024-09-21T14:06:45.308994+0000","last_unstale":"2024-09-21T14:08:30.064798+0000","last_undegraded":"2024-09-21T14:08:30.064798+0000","last_fullsized":"2024-09-21T14:08:30.064798+0000","mapping_epoch":461,"log_start":"477'4494","ondisk_log_start":"477'4494","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_clean_scrub_stamp":"2024-09-21T14:06:44.365469+0000","objects_scrubbed":0,"log_size":3735,"log_dups_size":0,"ondisk_log_size":3735,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:49.854728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":479002624,"num_objects":7309,"num_object_clones":0,"num_object_copies":21927,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8229,"num_write_kb":497216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"75'79","reported_seq":29,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-21T14:08:30.621657+0000","last_change":"2024-09-21T14:08:16.400437+0000","last_active":"2024-09-21T14:08:30.621657+0000","last_peered":"2024-09-21T14:08:30.621657+0000","last_clean":"2024-09-21T14:08:30.621657+0000","last_became_active":"2024-09-21T14:08:14.395761+0000","last_became_peered":"2024-09-21T14:08:14.395761+0000","last_unstale":"2024-09-21T14:08:30.621657+0000","last_undegraded":"2024-09-21T14:08:30.621657+0000","last_fullsized":"2024-09-21T14:08:30.621657+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:08:16.400353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:08:16.400353+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:22.806664+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"290'28","reported_seq":1046,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.462444+0000","last_change":"2024-09-21T14:08:13.140559+0000","last_active":"2024-09-21T14:08:20.462444+0000","last_peered":"2024-09-21T14:08:20.462444+0000","last_clean":"2024-09-21T14:08:20.462444+0000","last_became_active":"2024-09-21T14:01:41.200859+0000","last_became_peered":"2024-09-21T14:01:41.200859+0000","last_unstale":"2024-09-21T14:08:20.462444+0000","last_undegraded":"2024-09-21T14:08:20.462444+0000","last_fullsized":"2024-09-21T14:08:20.462444+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'28","last_scrub_stamp":"2024-09-21T14:08:13.140470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:08:13.140470+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:39.840606+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":223,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.479239+0000","last_change":"2024-09-21T14:07:47.859129+0000","last_active":"2024-09-21T14:08:20.479239+0000","last_peered":"2024-09-21T14:08:20.479239+0000","last_clean":"2024-09-21T14:08:20.479239+0000","last_became_active":"2024-09-21T14:04:21.205354+0000","last_became_peered":"2024-09-21T14:04:21.205354+0000","last_unstale":"2024-09-21T14:08:20.479239+0000","last_undegraded":"2024-09-21T14:08:20.479239+0000","last_fullsized":"2024-09-21T14:08:20.479239+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T14:07:47.858840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:07:47.858840+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:11.047145+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"498'11","reported_seq":125,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:23.360449+0000","last_change":"2024-09-21T14:08:14.426325+0000","last_active":"2024-09-21T14:08:23.360449+0000","last_peered":"2024-09-21T14:08:23.360449+0000","last_clean":"2024-09-21T14:08:23.360449+0000","last_became_active":"2024-09-21T14:08:14.411976+0000","last_became_peered":"2024-09-21T14:08:14.411976+0000","last_unstale":"2024-09-21T14:08:23.360449+0000","last_undegraded":"2024-09-21T14:08:23.360449+0000","last_fullsized":"2024-09-21T14:08:23.360449+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'10","last_scrub_stamp":"2024-09-21T14:06:29.326448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:06:29.326448+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":533,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.471303+0000","last_change":"2024-09-21T14:08:04.293706+0000","last_active":"2024-09-21T14:08:20.471303+0000","last_peered":"2024-09-21T14:08:20.471303+0000","last_clean":"2024-09-21T14:08:20.471303+0000","last_became_active":"2024-09-21T14:08:04.292483+0000","last_became_peered":"2024-09-21T14:08:04.292483+0000","last_unstale":"2024-09-21T14:08:20.471303+0000","last_undegraded":"2024-09-21T14:08:20.471303+0000","last_fullsized":"2024-09-21T14:08:20.471303+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:45.055707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:04:45.055707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"499'8027","reported_seq":8115,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:28.081085+0000","last_change":"2024-09-21T14:06:45.307126+0000","last_active":"2024-09-21T14:08:28.081085+0000","last_peered":"2024-09-21T14:08:28.081085+0000","last_clean":"2024-09-21T14:08:28.081085+0000","last_became_active":"2024-09-21T14:06:45.306827+0000","last_became_peered":"2024-09-21T14:06:45.306827+0000","last_unstale":"2024-09-21T14:08:28.081085+0000","last_undegraded":"2024-09-21T14:08:28.081085+0000","last_fullsized":"2024-09-21T14:08:28.081085+0000","mapping_epoch":461,"log_start":"476'3610","ondisk_log_start":"476'3610","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_clean_scrub_stamp":"2024-09-21T14:06:44.365469+0000","objects_scrubbed":0,"log_size":4417,"log_dups_size":0,"ondisk_log_size":4417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:48.959999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":481296408,"num_objects":7345,"num_object_clones":0,"num_object_copies":22035,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7345,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8027,"num_write_kb":491841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"75'79","reported_seq":992,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.471885+0000","last_change":"2024-09-21T14:08:14.395143+0000","last_active":"2024-09-21T14:08:20.471885+0000","last_peered":"2024-09-21T14:08:20.471885+0000","last_clean":"2024-09-21T14:08:20.471885+0000","last_became_active":"2024-09-21T14:08:14.394536+0000","last_became_peered":"2024-09-21T14:08:14.394536+0000","last_unstale":"2024-09-21T14:08:20.471885+0000","last_undegraded":"2024-09-21T14:08:20.471885+0000","last_fullsized":"2024-09-21T14:08:20.471885+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:05:45.878553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:05:45.878553+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"290'20","reported_seq":262,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.472192+0000","last_change":"2024-09-21T14:07:04.213903+0000","last_active":"2024-09-21T14:08:20.472192+0000","last_peered":"2024-09-21T14:08:20.472192+0000","last_clean":"2024-09-21T14:08:20.472192+0000","last_became_active":"2024-09-21T14:03:29.158218+0000","last_became_peered":"2024-09-21T14:03:29.158218+0000","last_unstale":"2024-09-21T14:08:20.472192+0000","last_undegraded":"2024-09-21T14:08:20.472192+0000","last_fullsized":"2024-09-21T14:08:20.472192+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'20","last_scrub_stamp":"2024-09-21T14:07:04.212370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:07:04.212370+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"61'1","reported_seq":888,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-21T14:08:30.621572+0000","last_change":"2024-09-21T14:08:14.409954+0000","last_active":"2024-09-21T14:08:30.621572+0000","last_peered":"2024-09-21T14:08:30.621572+0000","last_clean":"2024-09-21T14:08:30.621572+0000","last_became_active":"2024-09-21T14:08:14.398397+0000","last_became_peered":"2024-09-21T14:08:14.398397+0000","last_unstale":"2024-09-21T14:08:30.621572+0000","last_undegraded":"2024-09-21T14:08:30.621572+0000","last_fullsized":"2024-09-21T14:08:30.621572+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T14:07:40.545199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:07:40.545199+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:53.850119+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":993,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"418'16","reported_seq":321,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:23.359326+0000","last_change":"2024-09-21T14:05:53.307706+0000","last_active":"2024-09-21T14:08:23.359326+0000","last_peered":"2024-09-21T14:08:23.359326+0000","last_clean":"2024-09-21T14:08:23.359326+0000","last_became_active":"2024-09-21T14:04:56.589828+0000","last_became_peered":"2024-09-21T14:04:56.589828+0000","last_unstale":"2024-09-21T14:08:23.359326+0000","last_undegraded":"2024-09-21T14:08:23.359326+0000","last_fullsized":"2024-09-21T14:08:23.359326+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"418'16","last_scrub_stamp":"2024-09-21T14:05:53.307631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:05:53.307631+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:30.507565+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":378,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.482409+0000","last_change":"2024-09-21T14:05:46.372772+0000","last_active":"2024-09-21T14:08:20.482409+0000","last_peered":"2024-09-21T14:08:20.482409+0000","last_clean":"2024-09-21T14:08:20.482409+0000","last_became_active":"2024-09-21T14:04:56.586891+0000","last_became_peered":"2024-09-21T14:04:56.586891+0000","last_unstale":"2024-09-21T14:08:20.482409+0000","last_undegraded":"2024-09-21T14:08:20.482409+0000","last_fullsized":"2024-09-21T14:08:20.482409+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:46.372709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:05:46.372709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:23.400233+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"499'8245","reported_seq":8330,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:30.037085+0000","last_change":"2024-09-21T14:06:45.306909+0000","last_active":"2024-09-21T14:08:30.037085+0000","last_peered":"2024-09-21T14:08:30.037085+0000","last_clean":"2024-09-21T14:08:30.037085+0000","last_became_active":"2024-09-21T14:06:45.306632+0000","last_became_peered":"2024-09-21T14:06:45.306632+0000","last_unstale":"2024-09-21T14:08:30.037085+0000","last_undegraded":"2024-09-21T14:08:30.037085+0000","last_fullsized":"2024-09-21T14:08:30.037085+0000","mapping_epoch":461,"log_start":"479'4542","ondisk_log_start":"479'4542","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_clean_scrub_stamp":"2024-09-21T14:06:44.365469+0000","objects_scrubbed":0,"log_size":3703,"log_dups_size":0,"ondisk_log_size":3703,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:54.833561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":476905472,"num_objects":7277,"num_object_clones":0,"num_object_copies":21831,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8245,"num_write_kb":496704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"75'79","reported_seq":1000,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.471739+0000","last_change":"2024-09-21T14:08:14.402716+0000","last_active":"2024-09-21T14:08:20.471739+0000","last_peered":"2024-09-21T14:08:20.471739+0000","last_clean":"2024-09-21T14:08:20.471739+0000","last_became_active":"2024-09-21T14:08:14.402297+0000","last_became_peered":"2024-09-21T14:08:14.402297+0000","last_unstale":"2024-09-21T14:08:20.471739+0000","last_undegraded":"2024-09-21T14:08:20.471739+0000","last_fullsized":"2024-09-21T14:08:20.471739+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:05:41.888228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:05:41.888228+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"290'29","reported_seq":687,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.485858+0000","last_change":"2024-09-21T14:07:32.546696+0000","last_active":"2024-09-21T14:08:20.485858+0000","last_peered":"2024-09-21T14:08:20.485858+0000","last_clean":"2024-09-21T14:08:20.485858+0000","last_became_active":"2024-09-21T14:04:56.589018+0000","last_became_peered":"2024-09-21T14:04:56.589018+0000","last_unstale":"2024-09-21T14:08:20.485858+0000","last_undegraded":"2024-09-21T14:08:20.485858+0000","last_fullsized":"2024-09-21T14:08:20.485858+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'29","last_scrub_stamp":"2024-09-21T14:07:32.545961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:07:32.545961+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:35.743009+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":942,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.481180+0000","last_change":"2024-09-21T14:06:27.457531+0000","last_active":"2024-09-21T14:08:20.481180+0000","last_peered":"2024-09-21T14:08:20.481180+0000","last_clean":"2024-09-21T14:08:20.481180+0000","last_became_active":"2024-09-21T14:01:41.185400+0000","last_became_peered":"2024-09-21T14:01:41.185400+0000","last_unstale":"2024-09-21T14:08:20.481180+0000","last_undegraded":"2024-09-21T14:08:20.481180+0000","last_fullsized":"2024-09-21T14:08:20.481180+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:27.457452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:06:27.457452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:50.837651+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":525,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.470416+0000","last_change":"2024-09-21T14:07:04.801076+0000","last_active":"2024-09-21T14:08:20.470416+0000","last_peered":"2024-09-21T14:08:20.470416+0000","last_clean":"2024-09-21T14:08:20.470416+0000","last_became_active":"2024-09-21T14:01:41.193973+0000","last_became_peered":"2024-09-21T14:01:41.193973+0000","last_unstale":"2024-09-21T14:08:20.470416+0000","last_undegraded":"2024-09-21T14:08:20.470416+0000","last_fullsized":"2024-09-21T14:08:20.470416+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:07:04.801001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:07:04.801001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:24.469012+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":933,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.482440+0000","last_change":"2024-09-21T14:07:42.868224+0000","last_active":"2024-09-21T14:08:20.482440+0000","last_peered":"2024-09-21T14:08:20.482440+0000","last_clean":"2024-09-21T14:08:20.482440+0000","last_became_active":"2024-09-21T13:58:17.757628+0000","last_became_peered":"2024-09-21T13:58:17.757628+0000","last_unstale":"2024-09-21T14:08:20.482440+0000","last_undegraded":"2024-09-21T14:08:20.482440+0000","last_fullsized":"2024-09-21T14:08:20.482440+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:07:42.868151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:07:42.868151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:49.702699+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.b","version":"500'8154","reported_seq":8243,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-21T14:08:31.616192+0000","last_change":"2024-09-21T14:06:45.309354+0000","last_active":"2024-09-21T14:08:31.616192+0000","last_peered":"2024-09-21T14:08:31.616192+0000","last_clean":"2024-09-21T14:08:31.616192+0000","last_became_active":"2024-09-21T14:06:45.309027+0000","last_became_peered":"2024-09-21T14:06:45.309027+0000","last_unstale":"2024-09-21T14:08:31.616192+0000","last_undegraded":"2024-09-21T14:08:31.616192+0000","last_fullsized":"2024-09-21T14:08:31.616192+0000","mapping_epoch":461,"log_start":"472'2900","ondisk_log_start":"472'2900","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_clean_scrub_stamp":"2024-09-21T14:06:44.365469+0000","objects_scrubbed":0,"log_size":5254,"log_dups_size":0,"ondisk_log_size":5254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":462028800,"num_objects":7050,"num_object_clones":0,"num_object_copies":21150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8154,"num_write_kb":486528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"75'79","reported_seq":351,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-21T14:08:30.621149+0000","last_change":"2024-09-21T14:05:40.322410+0000","last_active":"2024-09-21T14:08:30.621149+0000","last_peered":"2024-09-21T14:08:30.621149+0000","last_clean":"2024-09-21T14:08:30.621149+0000","last_became_active":"2024-09-21T14:05:39.310615+0000","last_became_peered":"2024-09-21T14:05:39.310615+0000","last_unstale":"2024-09-21T14:08:30.621149+0000","last_undegraded":"2024-09-21T14:08:30.621149+0000","last_fullsized":"2024-09-21T14:08:30.621149+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:05:40.322344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:05:40.322344+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"498'45","reported_seq":1297,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:23.359705+0000","last_change":"2024-09-21T14:08:14.440676+0000","last_active":"2024-09-21T14:08:23.359705+0000","last_peered":"2024-09-21T14:08:23.359705+0000","last_clean":"2024-09-21T14:08:23.359705+0000","last_became_active":"2024-09-21T14:08:14.418714+0000","last_became_peered":"2024-09-21T14:08:14.418714+0000","last_unstale":"2024-09-21T14:08:23.359705+0000","last_undegraded":"2024-09-21T14:08:23.359705+0000","last_fullsized":"2024-09-21T14:08:23.359705+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'44","last_scrub_stamp":"2024-09-21T14:08:10.016212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:08:10.016212+0000","objects_scrubbed":10,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:24.904135+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":957,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.462888+0000","last_change":"2024-09-21T14:06:19.213603+0000","last_active":"2024-09-21T14:08:20.462888+0000","last_peered":"2024-09-21T14:08:20.462888+0000","last_clean":"2024-09-21T14:08:20.462888+0000","last_became_active":"2024-09-21T13:58:17.770937+0000","last_became_peered":"2024-09-21T13:58:17.770937+0000","last_unstale":"2024-09-21T14:08:20.462888+0000","last_undegraded":"2024-09-21T14:08:20.462888+0000","last_fullsized":"2024-09-21T14:08:20.462888+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:19.213554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:06:19.213554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:21.167065+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":896,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-21T14:08:30.622515+0000","last_change":"2024-09-21T14:07:46.426604+0000","last_active":"2024-09-21T14:08:30.622515+0000","last_peered":"2024-09-21T14:08:30.622515+0000","last_clean":"2024-09-21T14:08:30.622515+0000","last_became_active":"2024-09-21T14:01:41.247642+0000","last_became_peered":"2024-09-21T14:01:41.247642+0000","last_unstale":"2024-09-21T14:08:30.622515+0000","last_undegraded":"2024-09-21T14:08:30.622515+0000","last_fullsized":"2024-09-21T14:08:30.622515+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-21T14:07:46.426553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T14:07:46.426553+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:11.667708+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":883,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.479160+0000","last_change":"2024-09-21T14:07:45.837323+0000","last_active":"2024-09-21T14:08:20.479160+0000","last_peered":"2024-09-21T14:08:20.479160+0000","last_clean":"2024-09-21T14:08:20.479160+0000","last_became_active":"2024-09-21T14:04:21.204315+0000","last_became_peered":"2024-09-21T14:04:21.204315+0000","last_unstale":"2024-09-21T14:08:20.479160+0000","last_undegraded":"2024-09-21T14:08:20.479160+0000","last_fullsized":"2024-09-21T14:08:20.479160+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:07:45.837251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:07:45.837251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:51.700449+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":445,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.463752+0000","last_change":"2024-09-21T14:08:14.395014+0000","last_active":"2024-09-21T14:08:20.463752+0000","last_peered":"2024-09-21T14:08:20.463752+0000","last_clean":"2024-09-21T14:08:20.463752+0000","last_became_active":"2024-09-21T14:08:14.394400+0000","last_became_peered":"2024-09-21T14:08:14.394400+0000","last_unstale":"2024-09-21T14:08:20.463752+0000","last_undegraded":"2024-09-21T14:08:20.463752+0000","last_fullsized":"2024-09-21T14:08:20.463752+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:25.273146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:06:25.273146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.8","version":"499'7983","reported_seq":11352,"reported_epoch":499,"state":"active+remapped+backfilling","last_fresh":"2024-09-21T14:08:28.948060+0000","last_change":"2024-09-21T14:08:15.492037+0000","last_active":"2024-09-21T14:08:28.948060+0000","last_peered":"2024-09-21T14:08:28.948060+0000","last_clean":"2024-09-21T14:08:13.363497+0000","last_became_active":"2024-09-21T14:08:15.466425+0000","last_became_peered":"2024-09-21T14:08:15.466425+0000","last_unstale":"2024-09-21T14:08:28.948060+0000","last_undegraded":"2024-09-21T14:08:28.948060+0000","last_fullsized":"2024-09-21T14:08:28.948060+0000","mapping_epoch":497,"log_start":"476'3985","ondisk_log_start":"476'3985","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_clean_scrub_stamp":"2024-09-21T14:06:44.365469+0000","objects_scrubbed":0,"log_size":3998,"log_dups_size":0,"ondisk_log_size":3998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":472055808,"num_objects":7203,"num_object_clones":0,"num_object_copies":21609,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5604,"num_objects_unfound":0,"num_objects_dirty":7203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7983,"num_write_kb":485952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1638,"num_bytes_recovered":107347968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,3],"avail_no_missing":["6","3","5"],"object_location_counts":[{"shards":"3,5,6","objects":7203}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"75'79","reported_seq":122,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.485880+0000","last_change":"2024-09-21T14:08:04.449324+0000","last_active":"2024-09-21T14:08:20.485880+0000","last_peered":"2024-09-21T14:08:20.485880+0000","last_clean":"2024-09-21T14:08:20.485880+0000","last_became_active":"2024-09-21T14:08:04.293050+0000","last_became_peered":"2024-09-21T14:08:04.293050+0000","last_unstale":"2024-09-21T14:08:20.485880+0000","last_undegraded":"2024-09-21T14:08:20.485880+0000","last_fullsized":"2024-09-21T14:08:20.485880+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:08:04.448969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:08:04.448969+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:14.142346+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"290'17","reported_seq":986,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.479943+0000","last_change":"2024-09-21T14:07:32.781619+0000","last_active":"2024-09-21T14:08:20.479943+0000","last_peered":"2024-09-21T14:08:20.479943+0000","last_clean":"2024-09-21T14:08:20.479943+0000","last_became_active":"2024-09-21T14:04:21.205738+0000","last_became_peered":"2024-09-21T14:04:21.205738+0000","last_unstale":"2024-09-21T14:08:20.479943+0000","last_undegraded":"2024-09-21T14:08:20.479943+0000","last_fullsized":"2024-09-21T14:08:20.479943+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'17","last_scrub_stamp":"2024-09-21T14:07:32.780439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:07:32.780439+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:02.157262+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":977,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.484637+0000","last_change":"2024-09-21T14:04:10.193524+0000","last_active":"2024-09-21T14:08:20.484637+0000","last_peered":"2024-09-21T14:08:20.484637+0000","last_clean":"2024-09-21T14:08:20.484637+0000","last_became_active":"2024-09-21T13:58:17.759671+0000","last_became_peered":"2024-09-21T13:58:17.759671+0000","last_unstale":"2024-09-21T14:08:20.484637+0000","last_undegraded":"2024-09-21T14:08:20.484637+0000","last_fullsized":"2024-09-21T14:08:20.484637+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:10.193318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:10.193318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":869,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.466622+0000","last_change":"2024-09-21T14:08:14.427269+0000","last_active":"2024-09-21T14:08:20.466622+0000","last_peered":"2024-09-21T14:08:20.466622+0000","last_clean":"2024-09-21T14:08:20.466622+0000","last_became_active":"2024-09-21T14:08:14.415335+0000","last_became_peered":"2024-09-21T14:08:14.415335+0000","last_unstale":"2024-09-21T14:08:20.466622+0000","last_undegraded":"2024-09-21T14:08:20.466622+0000","last_fullsized":"2024-09-21T14:08:20.466622+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T14:07:43.046486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T14:07:43.046486+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:56.424914+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":853,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.468280+0000","last_change":"2024-09-21T14:08:02.089974+0000","last_active":"2024-09-21T14:08:20.468280+0000","last_peered":"2024-09-21T14:08:20.468280+0000","last_clean":"2024-09-21T14:08:20.468280+0000","last_became_active":"2024-09-21T14:01:41.182000+0000","last_became_peered":"2024-09-21T14:01:41.182000+0000","last_unstale":"2024-09-21T14:08:20.468280+0000","last_undegraded":"2024-09-21T14:08:20.468280+0000","last_fullsized":"2024-09-21T14:08:20.468280+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:08:02.089791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:08:02.089791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:28.458845+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":541,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.471575+0000","last_change":"2024-09-21T14:08:04.290668+0000","last_active":"2024-09-21T14:08:20.471575+0000","last_peered":"2024-09-21T14:08:20.471575+0000","last_clean":"2024-09-21T14:08:20.471575+0000","last_became_active":"2024-09-21T14:08:04.290357+0000","last_became_peered":"2024-09-21T14:08:04.290357+0000","last_unstale":"2024-09-21T14:08:20.471575+0000","last_undegraded":"2024-09-21T14:08:20.471575+0000","last_fullsized":"2024-09-21T14:08:20.471575+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:07:03.383502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:07:03.383502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:27.533360+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.9","version":"499'8121","reported_seq":15605,"reported_epoch":499,"state":"active+remapped+backfilling","last_fresh":"2024-09-21T14:08:28.957453+0000","last_change":"2024-09-21T14:08:15.485353+0000","last_active":"2024-09-21T14:08:28.957453+0000","last_peered":"2024-09-21T14:08:28.957453+0000","last_clean":"2024-09-21T14:08:13.371656+0000","last_became_active":"2024-09-21T14:08:15.474001+0000","last_became_peered":"2024-09-21T14:08:15.474001+0000","last_unstale":"2024-09-21T14:08:28.957453+0000","last_undegraded":"2024-09-21T14:08:28.957453+0000","last_fullsized":"2024-09-21T14:08:28.957453+0000","mapping_epoch":497,"log_start":"477'4138","ondisk_log_start":"477'4138","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_clean_scrub_stamp":"2024-09-21T14:06:44.365469+0000","objects_scrubbed":0,"log_size":3983,"log_dups_size":0,"ondisk_log_size":3983,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483590144,"num_objects":7379,"num_object_clones":0,"num_object_copies":22137,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3741,"num_objects_unfound":0,"num_objects_dirty":7379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8121,"num_write_kb":496000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3696,"num_bytes_recovered":242221056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,3],"avail_no_missing":["6","1","3"],"object_location_counts":[{"shards":"1,3,6","objects":7379}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"75'79","reported_seq":411,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.484405+0000","last_change":"2024-09-21T14:06:52.656799+0000","last_active":"2024-09-21T14:08:20.484405+0000","last_peered":"2024-09-21T14:08:20.484405+0000","last_clean":"2024-09-21T14:08:20.484405+0000","last_became_active":"2024-09-21T14:05:39.311460+0000","last_became_peered":"2024-09-21T14:05:39.311460+0000","last_unstale":"2024-09-21T14:08:20.484405+0000","last_undegraded":"2024-09-21T14:08:20.484405+0000","last_fullsized":"2024-09-21T14:08:20.484405+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:06:52.656733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:06:52.656733+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"290'24","reported_seq":776,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-21T14:08:30.622299+0000","last_change":"2024-09-21T14:04:21.221906+0000","last_active":"2024-09-21T14:08:30.622299+0000","last_peered":"2024-09-21T14:08:30.622299+0000","last_clean":"2024-09-21T14:08:30.622299+0000","last_became_active":"2024-09-21T14:04:21.206881+0000","last_became_peered":"2024-09-21T14:04:21.206881+0000","last_unstale":"2024-09-21T14:08:30.622299+0000","last_undegraded":"2024-09-21T14:08:30.622299+0000","last_fullsized":"2024-09-21T14:08:30.622299+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'24","last_scrub_stamp":"2024-09-21T14:02:57.559936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:02:57.559936+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:21.219094+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":509,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.482975+0000","last_change":"2024-09-21T14:08:14.402465+0000","last_active":"2024-09-21T14:08:20.482975+0000","last_peered":"2024-09-21T14:08:20.482975+0000","last_clean":"2024-09-21T14:08:20.482975+0000","last_became_active":"2024-09-21T14:08:14.402133+0000","last_became_peered":"2024-09-21T14:08:14.402133+0000","last_unstale":"2024-09-21T14:08:20.482975+0000","last_undegraded":"2024-09-21T14:08:20.482975+0000","last_fullsized":"2024-09-21T14:08:20.482975+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:08:09.550409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:08:09.550409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:37.986542+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":224,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:28.397516+0000","last_change":"2024-09-21T14:08:07.018283+0000","last_active":"2024-09-21T14:08:28.397516+0000","last_peered":"2024-09-21T14:08:28.397516+0000","last_clean":"2024-09-21T14:08:28.397516+0000","last_became_active":"2024-09-21T14:08:04.289119+0000","last_became_peered":"2024-09-21T14:08:04.289119+0000","last_unstale":"2024-09-21T14:08:28.397516+0000","last_undegraded":"2024-09-21T14:08:28.397516+0000","last_fullsized":"2024-09-21T14:08:28.397516+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-21T14:08:07.018200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T14:08:07.018200+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:35.403254+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0.00047048000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":98,"num_read_kb":93,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":867,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-21T14:08:30.622972+0000","last_change":"2024-09-21T14:06:49.591004+0000","last_active":"2024-09-21T14:08:30.622972+0000","last_peered":"2024-09-21T14:08:30.622972+0000","last_clean":"2024-09-21T14:08:30.622972+0000","last_became_active":"2024-09-21T14:01:41.200531+0000","last_became_peered":"2024-09-21T14:01:41.200531+0000","last_unstale":"2024-09-21T14:08:30.622972+0000","last_undegraded":"2024-09-21T14:08:30.622972+0000","last_fullsized":"2024-09-21T14:08:30.622972+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:49.590942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:06:49.590942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":969,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.480527+0000","last_change":"2024-09-21T14:08:14.418231+0000","last_active":"2024-09-21T14:08:20.480527+0000","last_peered":"2024-09-21T14:08:20.480527+0000","last_clean":"2024-09-21T14:08:20.480527+0000","last_became_active":"2024-09-21T14:08:14.410113+0000","last_became_peered":"2024-09-21T14:08:14.410113+0000","last_unstale":"2024-09-21T14:08:20.480527+0000","last_undegraded":"2024-09-21T14:08:20.480527+0000","last_fullsized":"2024-09-21T14:08:20.480527+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:57.451303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:05:57.451303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"500'8387","reported_seq":8476,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-21T14:08:31.587848+0000","last_change":"2024-09-21T14:06:45.200009+0000","last_active":"2024-09-21T14:08:31.587848+0000","last_peered":"2024-09-21T14:08:31.587848+0000","last_clean":"2024-09-21T14:08:31.587848+0000","last_became_active":"2024-09-21T14:06:45.199722+0000","last_became_peered":"2024-09-21T14:06:45.199722+0000","last_unstale":"2024-09-21T14:08:31.587848+0000","last_undegraded":"2024-09-21T14:08:31.587848+0000","last_fullsized":"2024-09-21T14:08:31.587848+0000","mapping_epoch":461,"log_start":"473'3000","ondisk_log_start":"473'3000","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_clean_scrub_stamp":"2024-09-21T14:06:44.365469+0000","objects_scrubbed":0,"log_size":5387,"log_dups_size":0,"ondisk_log_size":5387,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:29.358757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":473628672,"num_objects":7227,"num_object_clones":0,"num_object_copies":21681,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8387,"num_write_kb":499648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"300'32","reported_seq":668,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:23.359224+0000","last_change":"2024-09-21T14:00:47.130089+0000","last_active":"2024-09-21T14:08:23.359224+0000","last_peered":"2024-09-21T14:08:23.359224+0000","last_clean":"2024-09-21T14:08:23.359224+0000","last_became_active":"2024-09-21T13:59:32.204624+0000","last_became_peered":"2024-09-21T13:59:32.204624+0000","last_unstale":"2024-09-21T14:08:23.359224+0000","last_undegraded":"2024-09-21T14:08:23.359224+0000","last_fullsized":"2024-09-21T14:08:23.359224+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'32","last_scrub_stamp":"2024-09-21T14:00:47.129789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:00:47.129789+0000","objects_scrubbed":8,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:48.727112+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":600,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-21T14:08:30.622438+0000","last_change":"2024-09-21T14:08:14.394325+0000","last_active":"2024-09-21T14:08:30.622438+0000","last_peered":"2024-09-21T14:08:30.622438+0000","last_clean":"2024-09-21T14:08:30.622438+0000","last_became_active":"2024-09-21T14:08:14.393535+0000","last_became_peered":"2024-09-21T14:08:14.393535+0000","last_unstale":"2024-09-21T14:08:30.622438+0000","last_undegraded":"2024-09-21T14:08:30.622438+0000","last_fullsized":"2024-09-21T14:08:30.622438+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:50.360693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:06:50.360693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"75'79","reported_seq":607,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.469432+0000","last_change":"2024-09-21T14:08:06.009831+0000","last_active":"2024-09-21T14:08:20.469432+0000","last_peered":"2024-09-21T14:08:20.469432+0000","last_clean":"2024-09-21T14:08:20.469432+0000","last_became_active":"2024-09-21T14:08:04.298203+0000","last_became_peered":"2024-09-21T14:08:04.298203+0000","last_unstale":"2024-09-21T14:08:20.469432+0000","last_undegraded":"2024-09-21T14:08:20.469432+0000","last_fullsized":"2024-09-21T14:08:20.469432+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:08:06.009757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:08:06.009757+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:18.239351+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":591,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.475912+0000","last_change":"2024-09-21T14:08:07.199530+0000","last_active":"2024-09-21T14:08:20.475912+0000","last_peered":"2024-09-21T14:08:20.475912+0000","last_clean":"2024-09-21T14:08:20.475912+0000","last_became_active":"2024-09-21T14:08:04.288211+0000","last_became_peered":"2024-09-21T14:08:04.288211+0000","last_unstale":"2024-09-21T14:08:20.475912+0000","last_undegraded":"2024-09-21T14:08:20.475912+0000","last_fullsized":"2024-09-21T14:08:20.475912+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:08:07.199297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:08:07.199297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:36.314372+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":824,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-21T14:08:30.622675+0000","last_change":"2024-09-21T14:07:39.519394+0000","last_active":"2024-09-21T14:08:30.622675+0000","last_peered":"2024-09-21T14:08:30.622675+0000","last_clean":"2024-09-21T14:08:30.622675+0000","last_became_active":"2024-09-21T14:03:57.346453+0000","last_became_peered":"2024-09-21T14:03:57.346453+0000","last_unstale":"2024-09-21T14:08:30.622675+0000","last_undegraded":"2024-09-21T14:08:30.622675+0000","last_fullsized":"2024-09-21T14:08:30.622675+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:07:39.519327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:07:39.519327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:01.491612+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"499'8018","reported_seq":8184,"reported_epoch":499,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-21T14:08:28.915069+0000","last_change":"2024-09-21T14:08:15.496174+0000","last_active":"2024-09-21T14:08:28.915069+0000","last_peered":"2024-09-21T14:08:28.915069+0000","last_clean":"2024-09-21T14:08:13.362646+0000","last_became_active":"2024-09-21T14:08:15.478487+0000","last_became_peered":"2024-09-21T14:08:15.478487+0000","last_unstale":"2024-09-21T14:08:28.915069+0000","last_undegraded":"2024-09-21T14:08:28.915069+0000","last_fullsized":"2024-09-21T14:08:28.915069+0000","mapping_epoch":497,"log_start":"476'4017","ondisk_log_start":"476'4017","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"491'6444","last_scrub_stamp":"2024-09-21T14:08:03.828589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_clean_scrub_stamp":"2024-09-21T14:08:03.828589+0000","objects_scrubbed":6440,"log_size":4001,"log_dups_size":0,"ondisk_log_size":4001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":890,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":476184576,"num_objects":7266,"num_object_clones":0,"num_object_copies":21798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7266,"num_objects_unfound":0,"num_objects_dirty":7266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8018,"num_write_kb":489088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,3],"avail_no_missing":["6","0","3"],"object_location_counts":[{"shards":"0,3,6","objects":7266}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"75'79","reported_seq":409,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.484207+0000","last_change":"2024-09-21T14:05:43.453954+0000","last_active":"2024-09-21T14:08:20.484207+0000","last_peered":"2024-09-21T14:08:20.484207+0000","last_clean":"2024-09-21T14:08:20.484207+0000","last_became_active":"2024-09-21T14:05:41.345392+0000","last_became_peered":"2024-09-21T14:05:41.345392+0000","last_unstale":"2024-09-21T14:08:20.484207+0000","last_undegraded":"2024-09-21T14:08:20.484207+0000","last_fullsized":"2024-09-21T14:08:20.484207+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:05:43.453553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:05:43.453553+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"290'19","reported_seq":1002,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.463790+0000","last_change":"2024-09-21T14:07:04.229852+0000","last_active":"2024-09-21T14:08:20.463790+0000","last_peered":"2024-09-21T14:08:20.463790+0000","last_clean":"2024-09-21T14:08:20.463790+0000","last_became_active":"2024-09-21T13:58:17.776310+0000","last_became_peered":"2024-09-21T13:58:17.776310+0000","last_unstale":"2024-09-21T14:08:20.463790+0000","last_undegraded":"2024-09-21T14:08:20.463790+0000","last_fullsized":"2024-09-21T14:08:20.463790+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'19","last_scrub_stamp":"2024-09-21T14:07:04.229772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:07:04.229772+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:22.757657+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":972,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.463075+0000","last_change":"2024-09-21T14:08:16.097026+0000","last_active":"2024-09-21T14:08:20.463075+0000","last_peered":"2024-09-21T14:08:20.463075+0000","last_clean":"2024-09-21T14:08:20.463075+0000","last_became_active":"2024-09-21T13:59:32.223247+0000","last_became_peered":"2024-09-21T13:59:32.223247+0000","last_unstale":"2024-09-21T14:08:20.463075+0000","last_undegraded":"2024-09-21T14:08:20.463075+0000","last_fullsized":"2024-09-21T14:08:20.463075+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:08:16.096946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:08:16.096946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:33.820562+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":893,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-21T14:08:30.622528+0000","last_change":"2024-09-21T14:06:24.464033+0000","last_active":"2024-09-21T14:08:30.622528+0000","last_peered":"2024-09-21T14:08:30.622528+0000","last_clean":"2024-09-21T14:08:30.622528+0000","last_became_active":"2024-09-21T14:01:41.193941+0000","last_became_peered":"2024-09-21T14:01:41.193941+0000","last_unstale":"2024-09-21T14:08:30.622528+0000","last_undegraded":"2024-09-21T14:08:30.622528+0000","last_fullsized":"2024-09-21T14:08:30.622528+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:24.463961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:06:24.463961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:54.335716+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":576,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.473514+0000","last_change":"2024-09-21T14:08:14.399584+0000","last_active":"2024-09-21T14:08:20.473514+0000","last_peered":"2024-09-21T14:08:20.473514+0000","last_clean":"2024-09-21T14:08:20.473514+0000","last_became_active":"2024-09-21T14:08:14.398067+0000","last_became_peered":"2024-09-21T14:08:14.398067+0000","last_unstale":"2024-09-21T14:08:20.473514+0000","last_undegraded":"2024-09-21T14:08:20.473514+0000","last_fullsized":"2024-09-21T14:08:20.473514+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:30.364645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:06:30.364645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"499'8054","reported_seq":13007,"reported_epoch":499,"state":"active+remapped+backfilling","last_fresh":"2024-09-21T14:08:29.675427+0000","last_change":"2024-09-21T14:08:15.480359+0000","last_active":"2024-09-21T14:08:29.675427+0000","last_peered":"2024-09-21T14:08:29.675427+0000","last_clean":"2024-09-21T14:08:13.362425+0000","last_became_active":"2024-09-21T14:08:15.464698+0000","last_became_peered":"2024-09-21T14:08:15.464698+0000","last_unstale":"2024-09-21T14:08:29.675427+0000","last_undegraded":"2024-09-21T14:08:29.675427+0000","last_fullsized":"2024-09-21T14:08:29.675427+0000","mapping_epoch":497,"log_start":"472'2900","ondisk_log_start":"472'2900","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_clean_scrub_stamp":"2024-09-21T14:06:44.365469+0000","objects_scrubbed":0,"log_size":5154,"log_dups_size":0,"ondisk_log_size":5154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":473563136,"num_objects":7226,"num_object_clones":0,"num_object_copies":21678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4859,"num_objects_unfound":0,"num_objects_dirty":7226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8054,"num_write_kb":488960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2430,"num_bytes_recovered":159252480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,3],"avail_no_missing":["0","2","3"],"object_location_counts":[{"shards":"0,2,3","objects":7226}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.8","version":"75'79","reported_seq":380,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-21T14:08:30.620768+0000","last_change":"2024-09-21T14:08:14.413907+0000","last_active":"2024-09-21T14:08:30.620768+0000","last_peered":"2024-09-21T14:08:30.620768+0000","last_clean":"2024-09-21T14:08:30.620768+0000","last_became_active":"2024-09-21T14:08:14.401513+0000","last_became_peered":"2024-09-21T14:08:14.401513+0000","last_unstale":"2024-09-21T14:08:30.620768+0000","last_undegraded":"2024-09-21T14:08:30.620768+0000","last_fullsized":"2024-09-21T14:08:30.620768+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:07:35.587194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:07:35.587194+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:04.978991+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"290'31","reported_seq":795,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.466973+0000","last_change":"2024-09-21T14:07:53.076972+0000","last_active":"2024-09-21T14:08:20.466973+0000","last_peered":"2024-09-21T14:08:20.466973+0000","last_clean":"2024-09-21T14:08:20.466973+0000","last_became_active":"2024-09-21T14:07:53.044656+0000","last_became_peered":"2024-09-21T14:07:53.044656+0000","last_unstale":"2024-09-21T14:08:20.466973+0000","last_undegraded":"2024-09-21T14:08:20.466973+0000","last_fullsized":"2024-09-21T14:08:20.466973+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'31","last_scrub_stamp":"2024-09-21T14:07:02.225989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:07:02.225989+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:31.904961+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":498,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.470122+0000","last_change":"2024-09-21T14:04:51.139211+0000","last_active":"2024-09-21T14:08:20.470122+0000","last_peered":"2024-09-21T14:08:20.470122+0000","last_clean":"2024-09-21T14:08:20.470122+0000","last_became_active":"2024-09-21T13:58:17.762520+0000","last_became_peered":"2024-09-21T13:58:17.762520+0000","last_unstale":"2024-09-21T14:08:20.470122+0000","last_undegraded":"2024-09-21T14:08:20.470122+0000","last_fullsized":"2024-09-21T14:08:20.470122+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:51.139123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:51.139123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":911,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.480608+0000","last_change":"2024-09-21T14:07:41.870428+0000","last_active":"2024-09-21T14:08:20.480608+0000","last_peered":"2024-09-21T14:08:20.480608+0000","last_clean":"2024-09-21T14:08:20.480608+0000","last_became_active":"2024-09-21T13:49:02.067472+0000","last_became_peered":"2024-09-21T13:49:02.067472+0000","last_unstale":"2024-09-21T14:08:20.480608+0000","last_undegraded":"2024-09-21T14:08:20.480608+0000","last_fullsized":"2024-09-21T14:08:20.480608+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:07:41.870349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:07:41.870349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:00.551048+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":801,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.475359+0000","last_change":"2024-09-21T14:08:14.411279+0000","last_active":"2024-09-21T14:08:20.475359+0000","last_peered":"2024-09-21T14:08:20.475359+0000","last_clean":"2024-09-21T14:08:20.475359+0000","last_became_active":"2024-09-21T14:08:14.410821+0000","last_became_peered":"2024-09-21T14:08:14.410821+0000","last_unstale":"2024-09-21T14:08:20.475359+0000","last_undegraded":"2024-09-21T14:08:20.475359+0000","last_fullsized":"2024-09-21T14:08:20.475359+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:07:41.059862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:07:41.059862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:06.899174+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"499'8206","reported_seq":8293,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:28.906431+0000","last_change":"2024-09-21T14:06:45.197312+0000","last_active":"2024-09-21T14:08:28.906431+0000","last_peered":"2024-09-21T14:08:28.906431+0000","last_clean":"2024-09-21T14:08:28.906431+0000","last_became_active":"2024-09-21T14:06:45.197043+0000","last_became_peered":"2024-09-21T14:06:45.197043+0000","last_unstale":"2024-09-21T14:08:28.906431+0000","last_undegraded":"2024-09-21T14:08:28.906431+0000","last_fullsized":"2024-09-21T14:08:28.906431+0000","mapping_epoch":461,"log_start":"476'4190","ondisk_log_start":"476'4190","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_clean_scrub_stamp":"2024-09-21T14:06:44.365469+0000","objects_scrubbed":0,"log_size":4016,"log_dups_size":0,"ondisk_log_size":4016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":486539264,"num_objects":7424,"num_object_clones":0,"num_object_copies":22272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8206,"num_write_kb":500160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"75'79","reported_seq":634,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-21T14:08:30.625817+0000","last_change":"2024-09-21T14:08:11.167381+0000","last_active":"2024-09-21T14:08:30.625817+0000","last_peered":"2024-09-21T14:08:30.625817+0000","last_clean":"2024-09-21T14:08:30.625817+0000","last_became_active":"2024-09-21T14:08:04.290319+0000","last_became_peered":"2024-09-21T14:08:04.290319+0000","last_unstale":"2024-09-21T14:08:30.625817+0000","last_undegraded":"2024-09-21T14:08:30.625817+0000","last_fullsized":"2024-09-21T14:08:30.625817+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:08:11.166966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:08:11.166966+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:16.592245+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"290'20","reported_seq":1001,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.463161+0000","last_change":"2024-09-21T14:05:15.387788+0000","last_active":"2024-09-21T14:08:20.463161+0000","last_peered":"2024-09-21T14:08:20.463161+0000","last_clean":"2024-09-21T14:08:20.463161+0000","last_became_active":"2024-09-21T13:58:17.765427+0000","last_became_peered":"2024-09-21T13:58:17.765427+0000","last_unstale":"2024-09-21T14:08:20.463161+0000","last_undegraded":"2024-09-21T14:08:20.463161+0000","last_fullsized":"2024-09-21T14:08:20.463161+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'20","last_scrub_stamp":"2024-09-21T14:05:15.387706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:05:15.387706+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:48.159851+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":868,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-21T14:08:30.627687+0000","last_change":"2024-09-21T14:08:15.127205+0000","last_active":"2024-09-21T14:08:30.627687+0000","last_peered":"2024-09-21T14:08:30.627687+0000","last_clean":"2024-09-21T14:08:30.627687+0000","last_became_active":"2024-09-21T14:04:21.215102+0000","last_became_peered":"2024-09-21T14:04:21.215102+0000","last_unstale":"2024-09-21T14:08:30.627687+0000","last_undegraded":"2024-09-21T14:08:30.627687+0000","last_fullsized":"2024-09-21T14:08:30.627687+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:08:15.127159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:08:15.127159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:42.021723+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"346'14","reported_seq":1049,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.467503+0000","last_change":"2024-09-21T14:06:47.037635+0000","last_active":"2024-09-21T14:08:20.467503+0000","last_peered":"2024-09-21T14:08:20.467503+0000","last_clean":"2024-09-21T14:08:20.467503+0000","last_became_active":"2024-09-21T14:01:41.186564+0000","last_became_peered":"2024-09-21T14:01:41.186564+0000","last_unstale":"2024-09-21T14:08:20.467503+0000","last_undegraded":"2024-09-21T14:08:20.467503+0000","last_fullsized":"2024-09-21T14:08:20.467503+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'14","last_scrub_stamp":"2024-09-21T14:06:47.037583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:06:47.037583+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:51.013411+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"68'1","reported_seq":929,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.463489+0000","last_change":"2024-09-21T14:06:30.258612+0000","last_active":"2024-09-21T14:08:20.463489+0000","last_peered":"2024-09-21T14:08:20.463489+0000","last_clean":"2024-09-21T14:08:20.463489+0000","last_became_active":"2024-09-21T13:52:04.313210+0000","last_became_peered":"2024-09-21T13:52:04.313210+0000","last_unstale":"2024-09-21T14:08:20.463489+0000","last_undegraded":"2024-09-21T14:08:20.463489+0000","last_fullsized":"2024-09-21T14:08:20.463489+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T14:06:30.258539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:06:30.258539+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:49.200508+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.4","version":"499'8168","reported_seq":8261,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:28.925847+0000","last_change":"2024-09-21T14:08:04.295353+0000","last_active":"2024-09-21T14:08:28.925847+0000","last_peered":"2024-09-21T14:08:28.925847+0000","last_clean":"2024-09-21T14:08:28.925847+0000","last_became_active":"2024-09-21T14:08:04.289424+0000","last_became_peered":"2024-09-21T14:08:04.289424+0000","last_unstale":"2024-09-21T14:08:28.925847+0000","last_undegraded":"2024-09-21T14:08:28.925847+0000","last_fullsized":"2024-09-21T14:08:28.925847+0000","mapping_epoch":491,"log_start":"476'4127","ondisk_log_start":"476'4127","created":461,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_clean_scrub_stamp":"2024-09-21T14:06:44.365469+0000","objects_scrubbed":0,"log_size":4041,"log_dups_size":0,"ondisk_log_size":4041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":486408192,"num_objects":7422,"num_object_clones":0,"num_object_copies":22266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8168,"num_write_kb":498880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"75'79","reported_seq":611,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-21T14:08:30.626031+0000","last_change":"2024-09-21T14:08:12.129261+0000","last_active":"2024-09-21T14:08:30.626031+0000","last_peered":"2024-09-21T14:08:30.626031+0000","last_clean":"2024-09-21T14:08:30.626031+0000","last_became_active":"2024-09-21T14:08:04.295682+0000","last_became_peered":"2024-09-21T14:08:04.295682+0000","last_unstale":"2024-09-21T14:08:30.626031+0000","last_undegraded":"2024-09-21T14:08:30.626031+0000","last_fullsized":"2024-09-21T14:08:30.626031+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:08:12.129198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:08:12.129198+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:30.158919+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"290'15","reported_seq":956,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.485756+0000","last_change":"2024-09-21T14:07:06.252753+0000","last_active":"2024-09-21T14:08:20.485756+0000","last_peered":"2024-09-21T14:08:20.485756+0000","last_clean":"2024-09-21T14:08:20.485756+0000","last_became_active":"2024-09-21T13:49:00.053016+0000","last_became_peered":"2024-09-21T13:49:00.053016+0000","last_unstale":"2024-09-21T14:08:20.485756+0000","last_undegraded":"2024-09-21T14:08:20.485756+0000","last_fullsized":"2024-09-21T14:08:20.485756+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'15","last_scrub_stamp":"2024-09-21T14:07:06.252672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:07:06.252672+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:25.861995+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":904,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-21T14:08:30.621683+0000","last_change":"2024-09-21T14:08:14.410591+0000","last_active":"2024-09-21T14:08:30.621683+0000","last_peered":"2024-09-21T14:08:30.621683+0000","last_clean":"2024-09-21T14:08:30.621683+0000","last_became_active":"2024-09-21T14:08:14.410232+0000","last_became_peered":"2024-09-21T14:08:14.410232+0000","last_unstale":"2024-09-21T14:08:30.621683+0000","last_undegraded":"2024-09-21T14:08:30.621683+0000","last_fullsized":"2024-09-21T14:08:30.621683+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:22.402182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:05:22.402182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:25.206906+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":891,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.462562+0000","last_change":"2024-09-21T14:08:15.110491+0000","last_active":"2024-09-21T14:08:20.462562+0000","last_peered":"2024-09-21T14:08:20.462562+0000","last_clean":"2024-09-21T14:08:20.462562+0000","last_became_active":"2024-09-21T14:08:14.407697+0000","last_became_peered":"2024-09-21T14:08:14.407697+0000","last_unstale":"2024-09-21T14:08:20.462562+0000","last_undegraded":"2024-09-21T14:08:20.462562+0000","last_fullsized":"2024-09-21T14:08:20.462562+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:08:15.110439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:08:15.110439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:25.067150+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":660,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.466759+0000","last_change":"2024-09-21T14:08:10.164973+0000","last_active":"2024-09-21T14:08:20.466759+0000","last_peered":"2024-09-21T14:08:20.466759+0000","last_clean":"2024-09-21T14:08:20.466759+0000","last_became_active":"2024-09-21T14:08:04.295557+0000","last_became_peered":"2024-09-21T14:08:04.295557+0000","last_unstale":"2024-09-21T14:08:20.466759+0000","last_undegraded":"2024-09-21T14:08:20.466759+0000","last_fullsized":"2024-09-21T14:08:20.466759+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:08:10.164899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:08:10.164899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:14.206148+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"500'8309","reported_seq":8404,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-21T14:08:30.976905+0000","last_change":"2024-09-21T14:08:04.311473+0000","last_active":"2024-09-21T14:08:30.976905+0000","last_peered":"2024-09-21T14:08:30.976905+0000","last_clean":"2024-09-21T14:08:30.976905+0000","last_became_active":"2024-09-21T14:08:04.300912+0000","last_became_peered":"2024-09-21T14:08:04.300912+0000","last_unstale":"2024-09-21T14:08:30.976905+0000","last_undegraded":"2024-09-21T14:08:30.976905+0000","last_fullsized":"2024-09-21T14:08:30.976905+0000","mapping_epoch":491,"log_start":"476'4122","ondisk_log_start":"476'4122","created":461,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_clean_scrub_stamp":"2024-09-21T14:06:44.365469+0000","objects_scrubbed":0,"log_size":4187,"log_dups_size":0,"ondisk_log_size":4187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":479133696,"num_objects":7311,"num_object_clones":0,"num_object_copies":21933,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8309,"num_write_kb":499840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.f","version":"75'79","reported_seq":999,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.472047+0000","last_change":"2024-09-21T14:07:01.780758+0000","last_active":"2024-09-21T14:08:20.472047+0000","last_peered":"2024-09-21T14:08:20.472047+0000","last_clean":"2024-09-21T14:08:20.472047+0000","last_became_active":"2024-09-21T14:05:41.327757+0000","last_became_peered":"2024-09-21T14:05:41.327757+0000","last_unstale":"2024-09-21T14:08:20.472047+0000","last_undegraded":"2024-09-21T14:08:20.472047+0000","last_fullsized":"2024-09-21T14:08:20.472047+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:07:01.780652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:07:01.780652+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:24.482569+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"290'35","reported_seq":564,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.470324+0000","last_change":"2024-09-21T14:05:38.897778+0000","last_active":"2024-09-21T14:08:20.470324+0000","last_peered":"2024-09-21T14:08:20.470324+0000","last_clean":"2024-09-21T14:08:20.470324+0000","last_became_active":"2024-09-21T13:59:32.204184+0000","last_became_peered":"2024-09-21T13:59:32.204184+0000","last_unstale":"2024-09-21T14:08:20.470324+0000","last_undegraded":"2024-09-21T14:08:20.470324+0000","last_fullsized":"2024-09-21T14:08:20.470324+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'35","last_scrub_stamp":"2024-09-21T14:05:38.897722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:05:38.897722+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":619,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-21T14:08:30.621833+0000","last_change":"2024-09-21T14:08:06.409193+0000","last_active":"2024-09-21T14:08:30.621833+0000","last_peered":"2024-09-21T14:08:30.621833+0000","last_clean":"2024-09-21T14:08:30.621833+0000","last_became_active":"2024-09-21T14:08:04.300236+0000","last_became_peered":"2024-09-21T14:08:04.300236+0000","last_unstale":"2024-09-21T14:08:30.621833+0000","last_undegraded":"2024-09-21T14:08:30.621833+0000","last_fullsized":"2024-09-21T14:08:30.621833+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:08:06.409140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:08:06.409140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:09.403902+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":896,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.466390+0000","last_change":"2024-09-21T14:08:12.134422+0000","last_active":"2024-09-21T14:08:20.466390+0000","last_peered":"2024-09-21T14:08:20.466390+0000","last_clean":"2024-09-21T14:08:20.466390+0000","last_became_active":"2024-09-21T14:01:41.181911+0000","last_became_peered":"2024-09-21T14:01:41.181911+0000","last_unstale":"2024-09-21T14:08:20.466390+0000","last_undegraded":"2024-09-21T14:08:20.466390+0000","last_fullsized":"2024-09-21T14:08:20.466390+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:08:12.134107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:08:12.134107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:20.854162+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":472,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.467215+0000","last_change":"2024-09-21T14:08:04.291203+0000","last_active":"2024-09-21T14:08:20.467215+0000","last_peered":"2024-09-21T14:08:20.467215+0000","last_clean":"2024-09-21T14:08:20.467215+0000","last_became_active":"2024-09-21T14:08:04.290897+0000","last_became_peered":"2024-09-21T14:08:04.290897+0000","last_unstale":"2024-09-21T14:08:20.467215+0000","last_undegraded":"2024-09-21T14:08:20.467215+0000","last_fullsized":"2024-09-21T14:08:20.467215+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:19.467928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:06:19.467928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.6","version":"499'8082","reported_seq":8243,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:28.131639+0000","last_change":"2024-09-21T14:08:06.099180+0000","last_active":"2024-09-21T14:08:28.131639+0000","last_peered":"2024-09-21T14:08:28.131639+0000","last_clean":"2024-09-21T14:08:28.131639+0000","last_became_active":"2024-09-21T14:06:45.307066+0000","last_became_peered":"2024-09-21T14:06:45.307066+0000","last_unstale":"2024-09-21T14:08:28.131639+0000","last_undegraded":"2024-09-21T14:08:28.131639+0000","last_fullsized":"2024-09-21T14:08:28.131639+0000","mapping_epoch":461,"log_start":"476'3643","ondisk_log_start":"476'3643","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"492'6638","last_scrub_stamp":"2024-09-21T14:08:06.099093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_clean_scrub_stamp":"2024-09-21T14:08:06.099093+0000","objects_scrubbed":6553,"log_size":4439,"log_dups_size":0,"ondisk_log_size":4439,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:30.771133+0000","scrub_duration":1978,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484179968,"num_objects":7388,"num_object_clones":0,"num_object_copies":22164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8082,"num_write_kb":495040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"75'79","reported_seq":304,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.482483+0000","last_change":"2024-09-21T14:08:14.407580+0000","last_active":"2024-09-21T14:08:20.482483+0000","last_peered":"2024-09-21T14:08:20.482483+0000","last_clean":"2024-09-21T14:08:20.482483+0000","last_became_active":"2024-09-21T14:08:14.397469+0000","last_became_peered":"2024-09-21T14:08:14.397469+0000","last_unstale":"2024-09-21T14:08:20.482483+0000","last_undegraded":"2024-09-21T14:08:20.482483+0000","last_fullsized":"2024-09-21T14:08:20.482483+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:05:48.355659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:05:48.355659+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109227,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"290'18","reported_seq":1000,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.462981+0000","last_change":"2024-09-21T14:08:14.418654+0000","last_active":"2024-09-21T14:08:20.462981+0000","last_peered":"2024-09-21T14:08:20.462981+0000","last_clean":"2024-09-21T14:08:20.462981+0000","last_became_active":"2024-09-21T14:08:14.411011+0000","last_became_peered":"2024-09-21T14:08:14.411011+0000","last_unstale":"2024-09-21T14:08:20.462981+0000","last_undegraded":"2024-09-21T14:08:20.462981+0000","last_fullsized":"2024-09-21T14:08:20.462981+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'18","last_scrub_stamp":"2024-09-21T14:07:00.294535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:07:00.294535+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":699,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.479040+0000","last_change":"2024-09-21T14:08:04.291256+0000","last_active":"2024-09-21T14:08:20.479040+0000","last_peered":"2024-09-21T14:08:20.479040+0000","last_clean":"2024-09-21T14:08:20.479040+0000","last_became_active":"2024-09-21T14:08:04.290876+0000","last_became_peered":"2024-09-21T14:08:04.290876+0000","last_unstale":"2024-09-21T14:08:20.479040+0000","last_undegraded":"2024-09-21T14:08:20.479040+0000","last_fullsized":"2024-09-21T14:08:20.479040+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:07:35.401092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:07:35.401092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:55.971922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":850,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.467000+0000","last_change":"2024-09-21T14:05:50.814761+0000","last_active":"2024-09-21T14:08:20.467000+0000","last_peered":"2024-09-21T14:08:20.467000+0000","last_clean":"2024-09-21T14:08:20.467000+0000","last_became_active":"2024-09-21T13:52:04.322147+0000","last_became_peered":"2024-09-21T13:52:04.322147+0000","last_unstale":"2024-09-21T14:08:20.467000+0000","last_undegraded":"2024-09-21T14:08:20.467000+0000","last_fullsized":"2024-09-21T14:08:20.467000+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:50.814700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:05:50.814700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":876,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-21T14:08:30.627936+0000","last_change":"2024-09-21T14:08:19.227253+0000","last_active":"2024-09-21T14:08:30.627936+0000","last_peered":"2024-09-21T14:08:30.627936+0000","last_clean":"2024-09-21T14:08:30.627936+0000","last_became_active":"2024-09-21T14:05:30.243052+0000","last_became_peered":"2024-09-21T14:05:30.243052+0000","last_unstale":"2024-09-21T14:08:30.627936+0000","last_undegraded":"2024-09-21T14:08:30.627936+0000","last_fullsized":"2024-09-21T14:08:30.627936+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:08:19.227163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:08:19.227163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:48.648382+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"500'8090","reported_seq":8179,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-21T14:08:31.016673+0000","last_change":"2024-09-21T14:06:45.307096+0000","last_active":"2024-09-21T14:08:31.016673+0000","last_peered":"2024-09-21T14:08:31.016673+0000","last_clean":"2024-09-21T14:08:31.016673+0000","last_became_active":"2024-09-21T14:06:45.306744+0000","last_became_peered":"2024-09-21T14:06:45.306744+0000","last_unstale":"2024-09-21T14:08:31.016673+0000","last_undegraded":"2024-09-21T14:08:31.016673+0000","last_fullsized":"2024-09-21T14:08:31.016673+0000","mapping_epoch":461,"log_start":"476'3942","ondisk_log_start":"476'3942","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_clean_scrub_stamp":"2024-09-21T14:06:44.365469+0000","objects_scrubbed":0,"log_size":4148,"log_dups_size":0,"ondisk_log_size":4148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":462422016,"num_objects":7056,"num_object_clones":0,"num_object_copies":21168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8090,"num_write_kb":484672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"75'79","reported_seq":304,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.463242+0000","last_change":"2024-09-21T14:08:05.154262+0000","last_active":"2024-09-21T14:08:20.463242+0000","last_peered":"2024-09-21T14:08:20.463242+0000","last_clean":"2024-09-21T14:08:20.463242+0000","last_became_active":"2024-09-21T14:08:04.296887+0000","last_became_peered":"2024-09-21T14:08:04.296887+0000","last_unstale":"2024-09-21T14:08:20.463242+0000","last_undegraded":"2024-09-21T14:08:20.463242+0000","last_fullsized":"2024-09-21T14:08:20.463242+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:08:05.154200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:08:05.154200+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:28.720504+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"290'27","reported_seq":981,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-21T14:08:30.620952+0000","last_change":"2024-09-21T14:06:46.555735+0000","last_active":"2024-09-21T14:08:30.620952+0000","last_peered":"2024-09-21T14:08:30.620952+0000","last_clean":"2024-09-21T14:08:30.620952+0000","last_became_active":"2024-09-21T14:01:41.192481+0000","last_became_peered":"2024-09-21T14:01:41.192481+0000","last_unstale":"2024-09-21T14:08:30.620952+0000","last_undegraded":"2024-09-21T14:08:30.620952+0000","last_fullsized":"2024-09-21T14:08:30.620952+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'27","last_scrub_stamp":"2024-09-21T14:06:46.555656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:06:46.555656+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":100,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.463653+0000","last_change":"2024-09-21T14:08:20.092422+0000","last_active":"2024-09-21T14:08:20.463653+0000","last_peered":"2024-09-21T14:08:20.463653+0000","last_clean":"2024-09-21T14:08:20.463653+0000","last_became_active":"2024-09-21T14:08:04.294477+0000","last_became_peered":"2024-09-21T14:08:04.294477+0000","last_unstale":"2024-09-21T14:08:20.463653+0000","last_undegraded":"2024-09-21T14:08:20.463653+0000","last_fullsized":"2024-09-21T14:08:20.463653+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T14:08:20.092342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:08:20.092342+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:34.306126+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"492'15","reported_seq":672,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:23.360818+0000","last_change":"2024-09-21T14:08:04.299992+0000","last_active":"2024-09-21T14:08:23.360818+0000","last_peered":"2024-09-21T14:08:23.360818+0000","last_clean":"2024-09-21T14:08:23.360818+0000","last_became_active":"2024-09-21T14:08:04.299624+0000","last_became_peered":"2024-09-21T14:08:04.299624+0000","last_unstale":"2024-09-21T14:08:23.360818+0000","last_undegraded":"2024-09-21T14:08:23.360818+0000","last_fullsized":"2024-09-21T14:08:23.360818+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'14","last_scrub_stamp":"2024-09-21T14:07:36.427806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:07:36.427806+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:41.114010+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":929,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-21T14:08:30.622295+0000","last_change":"2024-09-21T14:07:45.480987+0000","last_active":"2024-09-21T14:08:30.622295+0000","last_peered":"2024-09-21T14:08:30.622295+0000","last_clean":"2024-09-21T14:08:30.622295+0000","last_became_active":"2024-09-21T14:01:41.246106+0000","last_became_peered":"2024-09-21T14:01:41.246106+0000","last_unstale":"2024-09-21T14:08:30.622295+0000","last_undegraded":"2024-09-21T14:08:30.622295+0000","last_fullsized":"2024-09-21T14:08:30.622295+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:07:45.480939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:07:45.480939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:04.805742+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"75'79","reported_seq":150,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-21T14:08:30.622084+0000","last_change":"2024-09-21T14:07:34.549041+0000","last_active":"2024-09-21T14:08:30.622084+0000","last_peered":"2024-09-21T14:08:30.622084+0000","last_clean":"2024-09-21T14:08:30.622084+0000","last_became_active":"2024-09-21T14:05:41.352416+0000","last_became_peered":"2024-09-21T14:05:41.352416+0000","last_unstale":"2024-09-21T14:08:30.622084+0000","last_undegraded":"2024-09-21T14:08:30.622084+0000","last_fullsized":"2024-09-21T14:08:30.622084+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":5,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:07:34.548995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:07:34.548995+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:50.500812+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":515,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.470655+0000","last_change":"2024-09-21T14:08:14.399712+0000","last_active":"2024-09-21T14:08:20.470655+0000","last_peered":"2024-09-21T14:08:20.470655+0000","last_clean":"2024-09-21T14:08:20.470655+0000","last_became_active":"2024-09-21T14:08:14.398906+0000","last_became_peered":"2024-09-21T14:08:14.398906+0000","last_unstale":"2024-09-21T14:08:20.470655+0000","last_undegraded":"2024-09-21T14:08:20.470655+0000","last_fullsized":"2024-09-21T14:08:20.470655+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:53.922394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:05:53.922394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"290'12","reported_seq":462,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.479776+0000","last_change":"2024-09-21T14:08:14.430792+0000","last_active":"2024-09-21T14:08:20.479776+0000","last_peered":"2024-09-21T14:08:20.479776+0000","last_clean":"2024-09-21T14:08:20.479776+0000","last_became_active":"2024-09-21T14:08:14.413439+0000","last_became_peered":"2024-09-21T14:08:14.413439+0000","last_unstale":"2024-09-21T14:08:20.479776+0000","last_undegraded":"2024-09-21T14:08:20.479776+0000","last_fullsized":"2024-09-21T14:08:20.479776+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'12","last_scrub_stamp":"2024-09-21T14:07:05.134142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:07:05.134142+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:31.178874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":13,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":859,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-21T14:08:30.621884+0000","last_change":"2024-09-21T14:07:13.246488+0000","last_active":"2024-09-21T14:08:30.621884+0000","last_peered":"2024-09-21T14:08:30.621884+0000","last_clean":"2024-09-21T14:08:30.621884+0000","last_became_active":"2024-09-21T14:07:13.246170+0000","last_became_peered":"2024-09-21T14:07:13.246170+0000","last_unstale":"2024-09-21T14:08:30.621884+0000","last_undegraded":"2024-09-21T14:08:30.621884+0000","last_fullsized":"2024-09-21T14:08:30.621884+0000","mapping_epoch":471,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:58.528683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:06:58.528683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":537,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.471670+0000","last_change":"2024-09-21T14:08:13.932501+0000","last_active":"2024-09-21T14:08:20.471670+0000","last_peered":"2024-09-21T14:08:20.471670+0000","last_clean":"2024-09-21T14:08:20.471670+0000","last_became_active":"2024-09-21T14:08:04.300053+0000","last_became_peered":"2024-09-21T14:08:04.300053+0000","last_unstale":"2024-09-21T14:08:20.471670+0000","last_undegraded":"2024-09-21T14:08:20.471670+0000","last_fullsized":"2024-09-21T14:08:20.471670+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:08:13.932452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:08:13.932452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:24.922195+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.13","version":"75'79","reported_seq":416,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.481282+0000","last_change":"2024-09-21T14:08:14.414805+0000","last_active":"2024-09-21T14:08:20.481282+0000","last_peered":"2024-09-21T14:08:20.481282+0000","last_clean":"2024-09-21T14:08:20.481282+0000","last_became_active":"2024-09-21T14:08:14.414455+0000","last_became_peered":"2024-09-21T14:08:14.414455+0000","last_unstale":"2024-09-21T14:08:20.481282+0000","last_undegraded":"2024-09-21T14:08:20.481282+0000","last_fullsized":"2024-09-21T14:08:20.481282+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":5,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:05:46.414817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:05:46.414817+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"290'15","reported_seq":867,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-21T14:08:30.626119+0000","last_change":"2024-09-21T14:05:55.357872+0000","last_active":"2024-09-21T14:08:30.626119+0000","last_peered":"2024-09-21T14:08:30.626119+0000","last_clean":"2024-09-21T14:08:30.626119+0000","last_became_active":"2024-09-21T14:04:21.208398+0000","last_became_peered":"2024-09-21T14:04:21.208398+0000","last_unstale":"2024-09-21T14:08:30.626119+0000","last_undegraded":"2024-09-21T14:08:30.626119+0000","last_fullsized":"2024-09-21T14:08:30.626119+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'15","last_scrub_stamp":"2024-09-21T14:05:55.357788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:05:55.357788+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:26.363117+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"61'1","reported_seq":91,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.462890+0000","last_change":"2024-09-21T14:08:04.301917+0000","last_active":"2024-09-21T14:08:20.462890+0000","last_peered":"2024-09-21T14:08:20.462890+0000","last_clean":"2024-09-21T14:08:20.462890+0000","last_became_active":"2024-09-21T14:08:04.301462+0000","last_became_peered":"2024-09-21T14:08:04.301462+0000","last_unstale":"2024-09-21T14:08:20.462890+0000","last_undegraded":"2024-09-21T14:08:20.462890+0000","last_fullsized":"2024-09-21T14:08:20.462890+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T14:07:40.388705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:07:40.388705+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:49.530341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":428,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.479642+0000","last_change":"2024-09-21T14:08:14.420307+0000","last_active":"2024-09-21T14:08:20.479642+0000","last_peered":"2024-09-21T14:08:20.479642+0000","last_clean":"2024-09-21T14:08:20.479642+0000","last_became_active":"2024-09-21T14:08:14.419869+0000","last_became_peered":"2024-09-21T14:08:14.419869+0000","last_unstale":"2024-09-21T14:08:20.479642+0000","last_undegraded":"2024-09-21T14:08:20.479642+0000","last_fullsized":"2024-09-21T14:08:20.479642+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:32.413388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:06:32.413388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":900,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.475869+0000","last_change":"2024-09-21T14:08:13.097305+0000","last_active":"2024-09-21T14:08:20.475869+0000","last_peered":"2024-09-21T14:08:20.475869+0000","last_clean":"2024-09-21T14:08:20.475869+0000","last_became_active":"2024-09-21T13:59:32.205803+0000","last_became_peered":"2024-09-21T13:59:32.205803+0000","last_unstale":"2024-09-21T14:08:20.475869+0000","last_undegraded":"2024-09-21T14:08:20.475869+0000","last_fullsized":"2024-09-21T14:08:20.475869+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:08:13.096578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:08:13.096578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:41.252075+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"75'79","reported_seq":142,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-21T14:08:30.620649+0000","last_change":"2024-09-21T14:05:42.242477+0000","last_active":"2024-09-21T14:08:30.620649+0000","last_peered":"2024-09-21T14:08:30.620649+0000","last_clean":"2024-09-21T14:08:30.620649+0000","last_became_active":"2024-09-21T14:05:41.325618+0000","last_became_peered":"2024-09-21T14:05:41.325618+0000","last_unstale":"2024-09-21T14:08:30.620649+0000","last_undegraded":"2024-09-21T14:08:30.620649+0000","last_fullsized":"2024-09-21T14:08:30.620649+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":5,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:05:42.242405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:05:42.242405+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"290'15","reported_seq":889,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-21T14:08:30.622010+0000","last_change":"2024-09-21T14:07:33.588871+0000","last_active":"2024-09-21T14:08:30.622010+0000","last_peered":"2024-09-21T14:08:30.622010+0000","last_clean":"2024-09-21T14:08:30.622010+0000","last_became_active":"2024-09-21T14:01:41.187904+0000","last_became_peered":"2024-09-21T14:01:41.187904+0000","last_unstale":"2024-09-21T14:08:30.622010+0000","last_undegraded":"2024-09-21T14:08:30.622010+0000","last_fullsized":"2024-09-21T14:08:30.622010+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'15","last_scrub_stamp":"2024-09-21T14:07:33.588707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:07:33.588707+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:34.869303+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":531,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.486164+0000","last_change":"2024-09-21T14:08:14.424827+0000","last_active":"2024-09-21T14:08:20.486164+0000","last_peered":"2024-09-21T14:08:20.486164+0000","last_clean":"2024-09-21T14:08:20.486164+0000","last_became_active":"2024-09-21T14:08:14.422865+0000","last_became_peered":"2024-09-21T14:08:14.422865+0000","last_unstale":"2024-09-21T14:08:20.486164+0000","last_undegraded":"2024-09-21T14:08:20.486164+0000","last_fullsized":"2024-09-21T14:08:20.486164+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:37.078458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:37.078458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"498'15","reported_seq":225,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.476073+0000","last_change":"2024-09-21T14:08:14.433250+0000","last_active":"2024-09-21T14:08:20.476073+0000","last_peered":"2024-09-21T14:08:20.476073+0000","last_clean":"2024-09-21T14:08:20.476073+0000","last_became_active":"2024-09-21T14:08:14.425036+0000","last_became_peered":"2024-09-21T14:08:14.425036+0000","last_unstale":"2024-09-21T14:08:20.476073+0000","last_undegraded":"2024-09-21T14:08:20.476073+0000","last_fullsized":"2024-09-21T14:08:20.476073+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'14","last_scrub_stamp":"2024-09-21T14:06:58.253842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:06:58.253842+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":944,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.463574+0000","last_change":"2024-09-21T14:06:48.260420+0000","last_active":"2024-09-21T14:08:20.463574+0000","last_peered":"2024-09-21T14:08:20.463574+0000","last_clean":"2024-09-21T14:08:20.463574+0000","last_became_active":"2024-09-21T14:01:41.200113+0000","last_became_peered":"2024-09-21T14:01:41.200113+0000","last_unstale":"2024-09-21T14:08:20.463574+0000","last_undegraded":"2024-09-21T14:08:20.463574+0000","last_fullsized":"2024-09-21T14:08:20.463574+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:48.260355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:06:48.260355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:50.211247+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"75'79","reported_seq":364,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-21T14:08:30.620873+0000","last_change":"2024-09-21T14:08:14.419675+0000","last_active":"2024-09-21T14:08:30.620873+0000","last_peered":"2024-09-21T14:08:30.620873+0000","last_clean":"2024-09-21T14:08:30.620873+0000","last_became_active":"2024-09-21T14:08:14.419335+0000","last_became_peered":"2024-09-21T14:08:14.419335+0000","last_unstale":"2024-09-21T14:08:30.620873+0000","last_undegraded":"2024-09-21T14:08:30.620873+0000","last_fullsized":"2024-09-21T14:08:30.620873+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":5,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:05:43.226395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:05:43.226395+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"290'20","reported_seq":894,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-21T14:08:30.626415+0000","last_change":"2024-09-21T14:07:40.228945+0000","last_active":"2024-09-21T14:08:30.626415+0000","last_peered":"2024-09-21T14:08:30.626415+0000","last_clean":"2024-09-21T14:08:30.626415+0000","last_became_active":"2024-09-21T13:59:32.208332+0000","last_became_peered":"2024-09-21T13:59:32.208332+0000","last_unstale":"2024-09-21T14:08:30.626415+0000","last_undegraded":"2024-09-21T14:08:30.626415+0000","last_fullsized":"2024-09-21T14:08:30.626415+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'20","last_scrub_stamp":"2024-09-21T14:07:40.228846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:07:40.228846+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:07.604865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":490,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.462729+0000","last_change":"2024-09-21T14:08:06.140232+0000","last_active":"2024-09-21T14:08:20.462729+0000","last_peered":"2024-09-21T14:08:20.462729+0000","last_clean":"2024-09-21T14:08:20.462729+0000","last_became_active":"2024-09-21T14:08:04.294307+0000","last_became_peered":"2024-09-21T14:08:04.294307+0000","last_unstale":"2024-09-21T14:08:20.462729+0000","last_undegraded":"2024-09-21T14:08:20.462729+0000","last_fullsized":"2024-09-21T14:08:20.462729+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:08:06.140163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:08:06.140163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:16.355527+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"346'10","reported_seq":1076,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-21T14:08:30.621921+0000","last_change":"2024-09-21T14:08:02.389592+0000","last_active":"2024-09-21T14:08:30.621921+0000","last_peered":"2024-09-21T14:08:30.621921+0000","last_clean":"2024-09-21T14:08:30.621921+0000","last_became_active":"2024-09-21T14:01:41.206303+0000","last_became_peered":"2024-09-21T14:01:41.206303+0000","last_unstale":"2024-09-21T14:08:30.621921+0000","last_undegraded":"2024-09-21T14:08:30.621921+0000","last_fullsized":"2024-09-21T14:08:30.621921+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'10","last_scrub_stamp":"2024-09-21T14:08:02.389168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:08:02.389168+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:19.828016+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":607,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.463448+0000","last_change":"2024-09-21T14:08:14.392434+0000","last_active":"2024-09-21T14:08:20.463448+0000","last_peered":"2024-09-21T14:08:20.463448+0000","last_clean":"2024-09-21T14:08:20.463448+0000","last_became_active":"2024-09-21T14:08:14.391953+0000","last_became_peered":"2024-09-21T14:08:14.391953+0000","last_unstale":"2024-09-21T14:08:20.463448+0000","last_undegraded":"2024-09-21T14:08:20.463448+0000","last_fullsized":"2024-09-21T14:08:20.463448+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:26.530508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:06:26.530508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"290'19","reported_seq":1007,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.462623+0000","last_change":"2024-09-21T14:08:11.081708+0000","last_active":"2024-09-21T14:08:20.462623+0000","last_peered":"2024-09-21T14:08:20.462623+0000","last_clean":"2024-09-21T14:08:20.462623+0000","last_became_active":"2024-09-21T13:58:17.771497+0000","last_became_peered":"2024-09-21T13:58:17.771497+0000","last_unstale":"2024-09-21T14:08:20.462623+0000","last_undegraded":"2024-09-21T14:08:20.462623+0000","last_fullsized":"2024-09-21T14:08:20.462623+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'19","last_scrub_stamp":"2024-09-21T14:08:11.081620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:08:11.081620+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:23.248711+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":157,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.473516+0000","last_change":"2024-09-21T14:08:04.290156+0000","last_active":"2024-09-21T14:08:20.473516+0000","last_peered":"2024-09-21T14:08:20.473516+0000","last_clean":"2024-09-21T14:08:20.473516+0000","last_became_active":"2024-09-21T14:08:04.289905+0000","last_became_peered":"2024-09-21T14:08:04.289905+0000","last_unstale":"2024-09-21T14:08:20.473516+0000","last_undegraded":"2024-09-21T14:08:20.473516+0000","last_fullsized":"2024-09-21T14:08:20.473516+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:02:57.807059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:02:57.807059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":964,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.478583+0000","last_change":"2024-09-21T14:08:14.415030+0000","last_active":"2024-09-21T14:08:20.478583+0000","last_peered":"2024-09-21T14:08:20.478583+0000","last_clean":"2024-09-21T14:08:20.478583+0000","last_became_active":"2024-09-21T14:08:14.414701+0000","last_became_peered":"2024-09-21T14:08:14.414701+0000","last_unstale":"2024-09-21T14:08:20.478583+0000","last_undegraded":"2024-09-21T14:08:20.478583+0000","last_fullsized":"2024-09-21T14:08:20.478583+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:51.607390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:06:51.607390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":676,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-21T14:08:30.620921+0000","last_change":"2024-09-21T14:08:14.397091+0000","last_active":"2024-09-21T14:08:30.620921+0000","last_peered":"2024-09-21T14:08:30.620921+0000","last_clean":"2024-09-21T14:08:30.620921+0000","last_became_active":"2024-09-21T14:08:14.396749+0000","last_became_peered":"2024-09-21T14:08:14.396749+0000","last_unstale":"2024-09-21T14:08:30.620921+0000","last_undegraded":"2024-09-21T14:08:30.620921+0000","last_fullsized":"2024-09-21T14:08:30.620921+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:08:04.070079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:08:04.070079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:07.128763+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"290'15","reported_seq":1017,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.478748+0000","last_change":"2024-09-21T14:07:46.841902+0000","last_active":"2024-09-21T14:08:20.478748+0000","last_peered":"2024-09-21T14:08:20.478748+0000","last_clean":"2024-09-21T14:08:20.478748+0000","last_became_active":"2024-09-21T13:58:17.764120+0000","last_became_peered":"2024-09-21T13:58:17.764120+0000","last_unstale":"2024-09-21T14:08:20.478748+0000","last_undegraded":"2024-09-21T14:08:20.478748+0000","last_fullsized":"2024-09-21T14:08:20.478748+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'15","last_scrub_stamp":"2024-09-21T14:07:46.841824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:07:46.841824+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:01.677107+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":488,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-21T14:08:30.625872+0000","last_change":"2024-09-21T14:08:04.295006+0000","last_active":"2024-09-21T14:08:30.625872+0000","last_peered":"2024-09-21T14:08:30.625872+0000","last_clean":"2024-09-21T14:08:30.625872+0000","last_became_active":"2024-09-21T14:08:04.294705+0000","last_became_peered":"2024-09-21T14:08:04.294705+0000","last_unstale":"2024-09-21T14:08:30.625872+0000","last_undegraded":"2024-09-21T14:08:30.625872+0000","last_fullsized":"2024-09-21T14:08:30.625872+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:07:42.393697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:07:42.393697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:00.349426+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":826,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-21T14:08:30.627596+0000","last_change":"2024-09-21T14:07:35.284133+0000","last_active":"2024-09-21T14:08:30.627596+0000","last_peered":"2024-09-21T14:08:30.627596+0000","last_clean":"2024-09-21T14:08:30.627596+0000","last_became_active":"2024-09-21T13:57:12.703441+0000","last_became_peered":"2024-09-21T13:57:12.703441+0000","last_unstale":"2024-09-21T14:08:30.627596+0000","last_undegraded":"2024-09-21T14:08:30.627596+0000","last_fullsized":"2024-09-21T14:08:30.627596+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:07:35.284064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:07:35.284064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:56.362795+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":588,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.482237+0000","last_change":"2024-09-21T14:04:56.588351+0000","last_active":"2024-09-21T14:08:20.482237+0000","last_peered":"2024-09-21T14:08:20.482237+0000","last_clean":"2024-09-21T14:08:20.482237+0000","last_became_active":"2024-09-21T14:04:56.587503+0000","last_became_peered":"2024-09-21T14:04:56.587503+0000","last_unstale":"2024-09-21T14:08:20.482237+0000","last_undegraded":"2024-09-21T14:08:20.482237+0000","last_fullsized":"2024-09-21T14:08:20.482237+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:03:29.345427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:03:29.345427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"75'79","reported_seq":149,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.475578+0000","last_change":"2024-09-21T14:07:06.186092+0000","last_active":"2024-09-21T14:08:20.475578+0000","last_peered":"2024-09-21T14:08:20.475578+0000","last_clean":"2024-09-21T14:08:20.475578+0000","last_became_active":"2024-09-21T14:05:41.358002+0000","last_became_peered":"2024-09-21T14:05:41.358002+0000","last_unstale":"2024-09-21T14:08:20.475578+0000","last_undegraded":"2024-09-21T14:08:20.475578+0000","last_fullsized":"2024-09-21T14:08:20.475578+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":5,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:07:06.185811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:07:06.185811+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"290'19","reported_seq":970,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-21T14:08:30.625713+0000","last_change":"2024-09-21T14:08:18.240004+0000","last_active":"2024-09-21T14:08:30.625713+0000","last_peered":"2024-09-21T14:08:30.625713+0000","last_clean":"2024-09-21T14:08:30.625713+0000","last_became_active":"2024-09-21T13:59:32.231567+0000","last_became_peered":"2024-09-21T13:59:32.231567+0000","last_unstale":"2024-09-21T14:08:30.625713+0000","last_undegraded":"2024-09-21T14:08:30.625713+0000","last_fullsized":"2024-09-21T14:08:30.625713+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'19","last_scrub_stamp":"2024-09-21T14:08:18.239931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:08:18.239931+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:39.588202+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":857,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-21T14:08:30.621976+0000","last_change":"2024-09-21T14:08:12.391143+0000","last_active":"2024-09-21T14:08:30.621976+0000","last_peered":"2024-09-21T14:08:30.621976+0000","last_clean":"2024-09-21T14:08:30.621976+0000","last_became_active":"2024-09-21T14:01:41.190760+0000","last_became_peered":"2024-09-21T14:01:41.190760+0000","last_unstale":"2024-09-21T14:08:30.621976+0000","last_undegraded":"2024-09-21T14:08:30.621976+0000","last_fullsized":"2024-09-21T14:08:30.621976+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:08:12.391069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:08:12.391069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:18.210905+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":533,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.462331+0000","last_change":"2024-09-21T14:08:04.300371+0000","last_active":"2024-09-21T14:08:20.462331+0000","last_peered":"2024-09-21T14:08:20.462331+0000","last_clean":"2024-09-21T14:08:20.462331+0000","last_became_active":"2024-09-21T14:08:04.300081+0000","last_became_peered":"2024-09-21T14:08:04.300081+0000","last_unstale":"2024-09-21T14:08:20.462331+0000","last_undegraded":"2024-09-21T14:08:20.462331+0000","last_fullsized":"2024-09-21T14:08:20.462331+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:49.450699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:06:49.450699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":715,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.470835+0000","last_change":"2024-09-21T14:08:14.971093+0000","last_active":"2024-09-21T14:08:20.470835+0000","last_peered":"2024-09-21T14:08:20.470835+0000","last_clean":"2024-09-21T14:08:20.470835+0000","last_became_active":"2024-09-21T14:08:14.399115+0000","last_became_peered":"2024-09-21T14:08:14.399115+0000","last_unstale":"2024-09-21T14:08:20.470835+0000","last_undegraded":"2024-09-21T14:08:20.470835+0000","last_fullsized":"2024-09-21T14:08:20.470835+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:08:14.971033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:08:14.971033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:17.753210+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"290'8","reported_seq":908,"reported_epoch":500,"state":"active+clean","last_fresh":"2024-09-21T14:08:30.628943+0000","last_change":"2024-09-21T14:08:14.166256+0000","last_active":"2024-09-21T14:08:30.628943+0000","last_peered":"2024-09-21T14:08:30.628943+0000","last_clean":"2024-09-21T14:08:30.628943+0000","last_became_active":"2024-09-21T13:58:17.775079+0000","last_became_peered":"2024-09-21T13:58:17.775079+0000","last_unstale":"2024-09-21T14:08:30.628943+0000","last_undegraded":"2024-09-21T14:08:30.628943+0000","last_fullsized":"2024-09-21T14:08:30.628943+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'8","last_scrub_stamp":"2024-09-21T14:08:14.166184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:08:14.166184+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:15.818117+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":690,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.483022+0000","last_change":"2024-09-21T14:08:14.400813+0000","last_active":"2024-09-21T14:08:20.483022+0000","last_peered":"2024-09-21T14:08:20.483022+0000","last_clean":"2024-09-21T14:08:20.483022+0000","last_became_active":"2024-09-21T14:08:14.400477+0000","last_became_peered":"2024-09-21T14:08:14.400477+0000","last_unstale":"2024-09-21T14:08:20.483022+0000","last_undegraded":"2024-09-21T14:08:20.483022+0000","last_fullsized":"2024-09-21T14:08:20.483022+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:08:05.499112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:08:05.499112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:20.449372+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":524,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.470255+0000","last_change":"2024-09-21T14:06:12.938710+0000","last_active":"2024-09-21T14:08:20.470255+0000","last_peered":"2024-09-21T14:08:20.470255+0000","last_clean":"2024-09-21T14:08:20.470255+0000","last_became_active":"2024-09-21T14:04:21.204954+0000","last_became_peered":"2024-09-21T14:04:21.204954+0000","last_unstale":"2024-09-21T14:08:20.470255+0000","last_undegraded":"2024-09-21T14:08:20.470255+0000","last_fullsized":"2024-09-21T14:08:20.470255+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:12.938632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:06:12.938632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":425,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.462466+0000","last_change":"2024-09-21T14:08:04.295718+0000","last_active":"2024-09-21T14:08:20.462466+0000","last_peered":"2024-09-21T14:08:20.462466+0000","last_clean":"2024-09-21T14:08:20.462466+0000","last_became_active":"2024-09-21T14:08:04.295387+0000","last_became_peered":"2024-09-21T14:08:04.295387+0000","last_unstale":"2024-09-21T14:08:20.462466+0000","last_undegraded":"2024-09-21T14:08:20.462466+0000","last_fullsized":"2024-09-21T14:08:20.462466+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:07:47.329279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:07:47.329279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:08.809604+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":933,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.479060+0000","last_change":"2024-09-21T14:07:40.894675+0000","last_active":"2024-09-21T14:08:20.479060+0000","last_peered":"2024-09-21T14:08:20.479060+0000","last_clean":"2024-09-21T14:08:20.479060+0000","last_became_active":"2024-09-21T14:01:41.187438+0000","last_became_peered":"2024-09-21T14:01:41.187438+0000","last_unstale":"2024-09-21T14:08:20.479060+0000","last_undegraded":"2024-09-21T14:08:20.479060+0000","last_fullsized":"2024-09-21T14:08:20.479060+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:07:40.894592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:07:40.894592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:09:00.717804+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":686,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.488460+0000","last_change":"2024-09-21T14:08:14.412412+0000","last_active":"2024-09-21T14:08:20.488460+0000","last_peered":"2024-09-21T14:08:20.488460+0000","last_clean":"2024-09-21T14:08:20.488460+0000","last_became_active":"2024-09-21T14:08:14.411173+0000","last_became_peered":"2024-09-21T14:08:14.411173+0000","last_unstale":"2024-09-21T14:08:20.488460+0000","last_undegraded":"2024-09-21T14:08:20.488460+0000","last_fullsized":"2024-09-21T14:08:20.488460+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:47.440233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:06:47.440233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"290'19","reported_seq":541,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.472093+0000","last_change":"2024-09-21T14:05:55.994009+0000","last_active":"2024-09-21T14:08:20.472093+0000","last_peered":"2024-09-21T14:08:20.472093+0000","last_clean":"2024-09-21T14:08:20.472093+0000","last_became_active":"2024-09-21T14:01:41.194606+0000","last_became_peered":"2024-09-21T14:01:41.194606+0000","last_unstale":"2024-09-21T14:08:20.472093+0000","last_undegraded":"2024-09-21T14:08:20.472093+0000","last_fullsized":"2024-09-21T14:08:20.472093+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'19","last_scrub_stamp":"2024-09-21T14:05:55.993943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:05:55.993943+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":543,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-21T14:08:20.483303+0000","last_change":"2024-09-21T14:04:56.591137+0000","last_active":"2024-09-21T14:08:20.483303+0000","last_peered":"2024-09-21T14:08:20.483303+0000","last_clean":"2024-09-21T14:08:20.483303+0000","last_became_active":"2024-09-21T14:04:56.590829+0000","last_became_peered":"2024-09-21T14:04:56.590829+0000","last_unstale":"2024-09-21T14:08:20.483303+0000","last_undegraded":"2024-09-21T14:08:20.483303+0000","last_fullsized":"2024-09-21T14:08:20.483303+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:51.121361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:04:51.121361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:08:21.368858+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":7621705752,"num_objects":116299,"num_object_clones":0,"num_object_copies":348897,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":27044,"num_objects_unfound":0,"num_objects_dirty":116299,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":130455,"num_write_kb":7896065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9466,"num_bytes_recovered":620363776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":23475400704,"data_stored":23475388488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67894,"ondisk_log_size":67894,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":105,"ondisk_log_size":105,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1104,"num_read_kb":861,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":244,"num_bytes_recovered":3922,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":688,"ondisk_log_size":688,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1085,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":98,"num_read_kb":93,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":371915,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1659,"ondisk_log_size":1659,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":298,"seq":1279900254317,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3807824,"kb_used_data":3211004,"kb_used_omap":4,"kb_used_meta":596795,"kb_avail":89937328,"statfs":{"total":95995035648,"available":92095823872,"internally_reserved":0,"allocated":3288068096,"data_stored":3287715053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":611118760},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463156,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4453552,"kb_used_data":3867040,"kb_used_omap":1,"kb_used_meta":586494,"kb_avail":89291600,"statfs":{"total":95995035648,"available":91434598400,"internally_reserved":0,"allocated":3959848960,"data_stored":3959516471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":600570310},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":345,"seq":1481763717203,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2107100,"kb_used_data":1849744,"kb_used_omap":4,"kb_used_meta":257339,"kb_avail":91638052,"statfs":{"total":95995035648,"available":93837365248,"internally_reserved":0,"allocated":1894137856,"data_stored":1893821384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":263515811},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921025,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2474724,"kb_used_data":1884052,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":91270428,"statfs":{"total":95995035648,"available":93460918272,"internally_reserved":0,"allocated":1929269248,"data_stored":1928947266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":604830149},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":240,"seq":1030792151177,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3784520,"kb_used_data":3191352,"kb_used_omap":4,"kb_used_meta":593147,"kb_avail":89960632,"statfs":{"total":95995035648,"available":92119687168,"internally_reserved":0,"allocated":3267944448,"data_stored":3267658435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4443,"internal_metadata":607383205},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":130,"seq":558345748678,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3750496,"kb_used_data":2937936,"kb_used_omap":4,"kb_used_meta":812411,"kb_avail":89994656,"statfs":{"total":95995035648,"available":92154527744,"internally_reserved":0,"allocated":3008446464,"data_stored":3008129818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4441,"internal_metadata":831909543},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575123,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5018076,"kb_used_data":4174540,"kb_used_omap":1,"kb_used_meta":843326,"kb_avail":88727076,"statfs":{"total":95995035648,"available":90856525824,"internally_reserved":0,"allocated":4274728960,"data_stored":4274350932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":863566277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738649,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2631420,"kb_used_data":1856044,"kb_used_omap":1,"kb_used_meta":775294,"kb_avail":91113732,"statfs":{"total":95995035648,"available":93300461568,"internally_reserved":0,"allocated":1900589056,"data_stored":1900307218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":793901509},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1898905600,"data_stored":1898905600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4271833088,"data_stored":4271833088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3006009344,"data_stored":3006005272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3261136896,"data_stored":3261136896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1913196544,"data_stored":1913192472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1880686592,"data_stored":1880686592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3958247424,"data_stored":3958243352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3285385216,"data_stored":3285385216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T14:08:38.255 INFO:tasks.thrashosds.thrasher:Setting 1.9 to [2, 3, 0] 2024-09-21T14:08:38.255 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '1.9', '2', '3', '0'] 2024-09-21T14:08:38.255 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd pg-upmap 1.9 2 3 0 2024-09-21T14:08:38.264 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:38.268 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:08:38.273 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:08:38.324 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:38.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:38.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:38.426 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:08:38.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:38.477 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:38.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:08:38.578 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:08:38.625 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:38.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:38.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:08:38.726 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:08:38.774 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:38.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:38.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:08:38.876 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:38.919 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:38.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:38.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:39.021 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:08:39.049 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:39.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:39.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:08:39.151 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:08:39.185 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:39.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:39.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:08:39.288 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:39.319 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:39.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:39.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:39.421 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:08:39.431 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:08:39.431 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:08:39.445 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:39.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:39.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:08:39.453 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:08:39.453 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:08:39.546 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:08:39.555 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:08:39.556 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:08:39.590 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:08:39.590 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:08:39.595 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:39.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:39.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:08:39.624 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:08:39.624 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:08:39.642 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T14:08:39.661 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:08:39.661 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:08:39.697 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:08:39.697 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:08:39.697 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:08:39.725 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:39.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:39.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:08:39.730 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:08:39.730 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:08:39.827 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:39.859 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:39.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:39.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:39.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:39 smithi187 ceph-mon[33032]: pgmap v1006: 168 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 6.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 679 B/s rd, 0 B/s wr, 848 op/s; 22029/334548 objects misplaced (6.585%); 33 MiB/s, 521 objects/s recovering 2024-09-21T14:08:39.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:39 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3222666045' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:08:39.961 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:39.992 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:39.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:39.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:40.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:39 smithi049 ceph-mon[40489]: pgmap v1006: 168 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 6.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 679 B/s rd, 0 B/s wr, 848 op/s; 22029/334548 objects misplaced (6.585%); 33 MiB/s, 521 objects/s recovering 2024-09-21T14:08:40.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:39 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3222666045' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:08:40.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:39 smithi049 ceph-mon[30080]: pgmap v1006: 168 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 6.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 679 B/s rd, 0 B/s wr, 848 op/s; 22029/334548 objects misplaced (6.585%); 33 MiB/s, 521 objects/s recovering 2024-09-21T14:08:40.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:39 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3222666045' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:08:40.095 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:40.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:40.130 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:40.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:40.231 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:08:40.253 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:40.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:40.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:08:40.354 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:40.388 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:40.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:40.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:40.444 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:08:40 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:08:40.490 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:08:40.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:40.544 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:40.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:08:40.645 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:40.670 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:40.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:40.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:40.772 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:40.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:40.804 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:40.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:40.905 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:08:40.927 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:40.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:40.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:08:40.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:40 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:08:40.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:40 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:08:41.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:40 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:08:41.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:40 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:08:41.004 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:08:40 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:08:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:08:41.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:40 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:08:41.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:40 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:08:41.029 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:08:41.064 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:41.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:41.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:08:41.166 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:41.195 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:41.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:41.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:41.297 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:08:41.319 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:41.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:41.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:08:41.421 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:08:41.446 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:41.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:41.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:08:41.548 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:08:41.572 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:41.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:41.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:08:41.674 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:41.699 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:41.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:41.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:41.751 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T14:08:41.752 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-09-21T14:08:41.752 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-09-21T14:08:41.752 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-09-21T14:08:41.752 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T14:08:41.800 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:08:41.825 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:41.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:41.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:08:41.927 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:08:41.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:41 smithi187 ceph-mon[33032]: pgmap v1007: 168 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 6.5 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.07k op/s; 17530/322293 objects misplaced (5.439%); 55 MiB/s, 877 objects/s recovering 2024-09-21T14:08:41.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:41 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:08:41.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:41 smithi187 ceph-mon[33032]: osdmap e503: 8 total, 8 up, 8 in 2024-09-21T14:08:41.956 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:41.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:41.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:08:42.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:41 smithi049 ceph-mon[40489]: pgmap v1007: 168 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 6.5 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.07k op/s; 17530/322293 objects misplaced (5.439%); 55 MiB/s, 877 objects/s recovering 2024-09-21T14:08:42.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:41 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:08:42.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:41 smithi049 ceph-mon[40489]: osdmap e503: 8 total, 8 up, 8 in 2024-09-21T14:08:42.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:41 smithi049 ceph-mon[30080]: pgmap v1007: 168 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 6.5 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.07k op/s; 17530/322293 objects misplaced (5.439%); 55 MiB/s, 877 objects/s recovering 2024-09-21T14:08:42.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:41 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:08:42.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:41 smithi049 ceph-mon[30080]: osdmap e503: 8 total, 8 up, 8 in 2024-09-21T14:08:42.058 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:08:42.088 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:42.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:42.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:08:42.190 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:08:42.230 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:42.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:42.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:08:42.332 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:42.364 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:42.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:42.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:42.466 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:08:42.502 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:42.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:42.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:08:42.604 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:42.640 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:42.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:42.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:42.742 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:42.771 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:42.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:42.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:42.873 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:42.916 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:42.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:42.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:43.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:42 smithi049 ceph-mon[40489]: osdmap e504: 8 total, 8 up, 8 in 2024-09-21T14:08:43.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:42 smithi049 ceph-mon[30080]: osdmap e504: 8 total, 8 up, 8 in 2024-09-21T14:08:43.018 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:08:43.053 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:43.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:43.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:08:43.155 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:08:43.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:42 smithi187 ceph-mon[33032]: osdmap e504: 8 total, 8 up, 8 in 2024-09-21T14:08:43.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:43.198 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:43.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:08:43.299 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:43.356 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:43.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:43.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:43.458 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:08:43.498 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:43.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:43.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:08:43.600 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:43.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:43.632 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:43.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:43.733 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:43.764 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:43.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:43.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:43.865 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:08:43.896 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T14:08:43.914 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:43.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:43.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:08:43.952 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:43 smithi049 ceph-mon[40489]: pgmap v1010: 168 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.00k op/s; 17530/318642 objects misplaced (5.501%); 40 MiB/s, 647 objects/s recovering 2024-09-21T14:08:43.953 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:43 smithi049 ceph-mon[30080]: pgmap v1010: 168 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.00k op/s; 17530/318642 objects misplaced (5.501%); 40 MiB/s, 647 objects/s recovering 2024-09-21T14:08:44.016 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:44.046 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:44.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:44.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:44.147 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:08:44.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:43 smithi187 ceph-mon[33032]: pgmap v1010: 168 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.00k op/s; 17530/318642 objects misplaced (5.501%); 40 MiB/s, 647 objects/s recovering 2024-09-21T14:08:44.203 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:44.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:44.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:08:44.305 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:08:44.335 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:44.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:44.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:08:44.436 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:44.464 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:44.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:44.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:44.565 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:44.604 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:44.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:44.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:44.706 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:44.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:44.735 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:44.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:44.788 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:08:44.836 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:08:44.865 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:44.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:44.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:08:44.967 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:45.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:44 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:08:45.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:44 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:08:45.020 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:45.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:45.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:45.122 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:08:45.130 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:08:45.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:45.187 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:45.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:08:45.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:44 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:08:45.288 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:45.327 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:45.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:45.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:45.429 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:45.458 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:45.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:45.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:45.478 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:08:45.560 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:45.617 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:45.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:45.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:45.718 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:08:45.758 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:45.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:45.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:08:45.860 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:45.879 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:45 smithi049 ceph-mon[30080]: pgmap v1011: 168 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 164 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 981 op/s; 15791/312795 objects misplaced (5.048%); 37 MiB/s, 599 objects/s recovering 2024-09-21T14:08:45.879 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:45 smithi049 ceph-mon[40489]: pgmap v1011: 168 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 164 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 981 op/s; 15791/312795 objects misplaced (5.048%); 37 MiB/s, 599 objects/s recovering 2024-09-21T14:08:45.890 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:45.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:45.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:45.992 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:46.042 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:46.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:46.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:46.144 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:46.171 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:46.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:46.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:46.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:45 smithi187 ceph-mon[33032]: pgmap v1011: 168 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 164 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 981 op/s; 15791/312795 objects misplaced (5.048%); 37 MiB/s, 599 objects/s recovering 2024-09-21T14:08:46.272 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:46.308 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:46.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:46.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:46.409 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:08:46.454 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:46.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:46.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:08:46.535 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T14:08:46.535 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-09-21T14:08:46.535 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-09-21T14:08:46.535 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-09-21T14:08:46.536 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.536 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25421.0:172314 9.4 9:2b749f7e:::benchmark_data_smithi049_138504_object50906:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-09-21T14:08:46.536 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:00:33.178595+0000", 2024-09-21T14:08:46.536 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 493.35454344800002, 2024-09-21T14:08:46.536 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.342539818, 2024-09-21T14:08:46.536 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:08:46.536 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:08:46.536 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:08:46.536 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:08:46.536 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25421", 2024-09-21T14:08:46.536 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:08:46.536 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 172314 2024-09-21T14:08:46.536 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.536 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:08:46.537 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.537 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:08:46.537 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.178595+0000", 2024-09-21T14:08:46.537 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:46.537 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.537 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.537 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:08:46.537 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.178595+0000", 2024-09-21T14:08:46.537 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:46.537 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.537 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.537 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:08:46.537 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.178596+0000", 2024-09-21T14:08:46.537 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5599999999999995e-07 2024-09-21T14:08:46.537 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.538 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.538 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:08:46.538 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.178599+0000", 2024-09-21T14:08:46.538 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3919999999999999e-06 2024-09-21T14:08:46.538 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.538 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.538 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:08:46.538 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.178607+0000", 2024-09-21T14:08:46.538 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3780000000000001e-06 2024-09-21T14:08:46.538 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.538 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.538 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:08:46.538 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.178613+0000", 2024-09-21T14:08:46.539 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8629999999999996e-06 2024-09-21T14:08:46.539 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.539 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.539 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:46.539 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.178862+0000", 2024-09-21T14:08:46.539 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00024909299999999998 2024-09-21T14:08:46.539 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.539 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.539 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:08:46.539 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.178965+0000", 2024-09-21T14:08:46.539 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000102204 2024-09-21T14:08:46.539 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.539 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.539 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,4", 2024-09-21T14:08:46.539 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.179255+0000", 2024-09-21T14:08:46.540 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00029060400000000003 2024-09-21T14:08:46.540 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.540 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.540 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:08:46.540 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.181536+0000", 2024-09-21T14:08:46.540 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0022811630000000001 2024-09-21T14:08:46.540 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.540 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.540 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:46.540 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.187746+0000", 2024-09-21T14:08:46.540 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0062093790000000001 2024-09-21T14:08:46.540 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.540 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.540 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:46.541 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.521042+0000", 2024-09-21T14:08:46.541 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.33329632300000001 2024-09-21T14:08:46.541 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.541 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.541 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:08:46.541 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.521095+0000", 2024-09-21T14:08:46.541 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3365999999999997e-05 2024-09-21T14:08:46.541 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.541 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.541 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:08:46.541 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.521135+0000", 2024-09-21T14:08:46.541 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9496999999999998e-05 2024-09-21T14:08:46.541 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:46.541 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:46.541 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:46.542 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.542 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.542 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25421.0:172319 9.4 9:21e132db:::benchmark_data_smithi049_138504_object50911:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-09-21T14:08:46.542 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:00:33.184640+0000", 2024-09-21T14:08:46.542 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 493.34849811599997, 2024-09-21T14:08:46.542 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.338985429, 2024-09-21T14:08:46.542 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:08:46.542 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:08:46.542 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:08:46.542 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:08:46.542 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25421", 2024-09-21T14:08:46.542 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:08:46.542 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 172319 2024-09-21T14:08:46.542 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.543 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:08:46.543 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.543 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:08:46.543 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.184640+0000", 2024-09-21T14:08:46.543 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:46.543 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.543 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.543 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:08:46.543 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.184640+0000", 2024-09-21T14:08:46.543 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:46.543 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.543 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.543 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:08:46.543 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.184642+0000", 2024-09-21T14:08:46.543 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.488e-06 2024-09-21T14:08:46.544 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.544 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.544 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:08:46.544 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.184647+0000", 2024-09-21T14:08:46.544 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4e-06 2024-09-21T14:08:46.544 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.544 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.544 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:08:46.544 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.184679+0000", 2024-09-21T14:08:46.544 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1482000000000001e-05 2024-09-21T14:08:46.544 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.544 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.544 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:08:46.544 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.184688+0000", 2024-09-21T14:08:46.544 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0380000000000005e-06 2024-09-21T14:08:46.544 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.544 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.545 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:46.545 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.184742+0000", 2024-09-21T14:08:46.545 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3888999999999999e-05 2024-09-21T14:08:46.545 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.545 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.545 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:08:46.545 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.184861+0000", 2024-09-21T14:08:46.545 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011920499999999999 2024-09-21T14:08:46.545 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.545 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.545 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,4", 2024-09-21T14:08:46.545 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.185030+0000", 2024-09-21T14:08:46.545 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016910499999999999 2024-09-21T14:08:46.545 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.546 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.546 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:08:46.546 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.187701+0000", 2024-09-21T14:08:46.546 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0026706659999999999 2024-09-21T14:08:46.546 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.546 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.546 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:46.546 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.188554+0000", 2024-09-21T14:08:46.546 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00085298600000000002 2024-09-21T14:08:46.546 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.546 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.546 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:46.546 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.523534+0000", 2024-09-21T14:08:46.546 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.33498041099999998 2024-09-21T14:08:46.547 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.547 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.547 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:08:46.547 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.523587+0000", 2024-09-21T14:08:46.547 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2604e-05 2024-09-21T14:08:46.547 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.547 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.547 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:08:46.547 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.523626+0000", 2024-09-21T14:08:46.547 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9155e-05 2024-09-21T14:08:46.547 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:46.547 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:46.547 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:46.547 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.547 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.548 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25421.0:172326 9.4 9:22a8fc8a:::benchmark_data_smithi049_138504_object50918:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-09-21T14:08:46.548 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:00:33.254009+0000", 2024-09-21T14:08:46.548 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 493.279129813, 2024-09-21T14:08:46.548 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.269760204, 2024-09-21T14:08:46.548 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:08:46.548 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:08:46.548 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:08:46.548 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:08:46.548 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25421", 2024-09-21T14:08:46.548 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:08:46.548 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 172326 2024-09-21T14:08:46.548 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.549 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:08:46.549 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.549 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:08:46.549 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254009+0000", 2024-09-21T14:08:46.549 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:46.549 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.549 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.549 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:08:46.549 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254009+0000", 2024-09-21T14:08:46.549 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:46.549 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.549 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.549 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:08:46.550 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254010+0000", 2024-09-21T14:08:46.550 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.42e-06 2024-09-21T14:08:46.550 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.550 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.550 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:08:46.550 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254014+0000", 2024-09-21T14:08:46.550 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9550000000000002e-06 2024-09-21T14:08:46.550 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.550 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.550 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:08:46.550 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254023+0000", 2024-09-21T14:08:46.551 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3330000000000003e-06 2024-09-21T14:08:46.551 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.551 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.551 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:08:46.551 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254030+0000", 2024-09-21T14:08:46.551 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5719999999999999e-06 2024-09-21T14:08:46.551 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.551 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.551 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:46.551 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254298+0000", 2024-09-21T14:08:46.551 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00026834 2024-09-21T14:08:46.552 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.552 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.552 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:08:46.552 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254412+0000", 2024-09-21T14:08:46.552 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011395000000000001 2024-09-21T14:08:46.552 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.552 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.552 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,4", 2024-09-21T14:08:46.552 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254540+0000", 2024-09-21T14:08:46.552 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012807199999999999 2024-09-21T14:08:46.552 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.552 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.553 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:46.553 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.259890+0000", 2024-09-21T14:08:46.553 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0053499230000000004 2024-09-21T14:08:46.553 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.553 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.553 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:08:46.553 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.259900+0000", 2024-09-21T14:08:46.553 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8919999999999995e-06 2024-09-21T14:08:46.553 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.553 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.553 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:46.553 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.523677+0000", 2024-09-21T14:08:46.553 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26377665300000003 2024-09-21T14:08:46.553 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.554 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.554 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:08:46.554 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.523732+0000", 2024-09-21T14:08:46.554 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5605e-05 2024-09-21T14:08:46.554 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.554 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.554 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:08:46.554 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.523769+0000", 2024-09-21T14:08:46.554 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6489000000000002e-05 2024-09-21T14:08:46.554 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:46.554 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:46.554 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:46.554 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.555 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.555 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25421.0:172330 9.4 9:2882ed62:::benchmark_data_smithi049_138504_object50922:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-09-21T14:08:46.555 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:00:33.254698+0000", 2024-09-21T14:08:46.555 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 493.27844028999999, 2024-09-21T14:08:46.555 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26919505599999999, 2024-09-21T14:08:46.555 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:08:46.555 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:08:46.555 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:08:46.555 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:08:46.555 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25421", 2024-09-21T14:08:46.555 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:08:46.555 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 172330 2024-09-21T14:08:46.555 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.555 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:08:46.556 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.556 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:08:46.556 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254698+0000", 2024-09-21T14:08:46.556 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:46.556 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.556 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.556 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:08:46.556 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254698+0000", 2024-09-21T14:08:46.556 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:46.556 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.556 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.556 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:08:46.556 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254699+0000", 2024-09-21T14:08:46.556 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3799999999999998e-07 2024-09-21T14:08:46.556 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.557 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.557 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:08:46.557 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254701+0000", 2024-09-21T14:08:46.557 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5409999999999999e-06 2024-09-21T14:08:46.557 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.557 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.557 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:08:46.557 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254708+0000", 2024-09-21T14:08:46.557 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6089999999999997e-06 2024-09-21T14:08:46.557 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.557 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.557 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:08:46.557 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254713+0000", 2024-09-21T14:08:46.557 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2449999999999998e-06 2024-09-21T14:08:46.558 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.558 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.558 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:46.558 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.257115+0000", 2024-09-21T14:08:46.558 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0024021030000000001 2024-09-21T14:08:46.558 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.558 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.558 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:08:46.558 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.257220+0000", 2024-09-21T14:08:46.558 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010508 2024-09-21T14:08:46.558 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.558 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.558 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,4", 2024-09-21T14:08:46.558 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.257330+0000", 2024-09-21T14:08:46.559 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000110165 2024-09-21T14:08:46.559 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.559 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.559 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:08:46.559 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.260045+0000", 2024-09-21T14:08:46.559 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0027141890000000001 2024-09-21T14:08:46.559 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.559 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.559 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:46.559 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.274124+0000", 2024-09-21T14:08:46.559 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.014079473 2024-09-21T14:08:46.559 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.559 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.559 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:46.559 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.523815+0000", 2024-09-21T14:08:46.559 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.249690836 2024-09-21T14:08:46.560 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.560 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.560 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:08:46.560 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.523860+0000", 2024-09-21T14:08:46.560 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4935999999999998e-05 2024-09-21T14:08:46.560 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.560 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.560 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:08:46.560 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.523893+0000", 2024-09-21T14:08:46.560 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3441000000000001e-05 2024-09-21T14:08:46.560 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:46.560 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:46.560 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:46.560 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.560 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.561 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25421.0:172334 9.4 9:24fa2bd3:::benchmark_data_smithi049_138504_object50926:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-09-21T14:08:46.561 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:00:33.260571+0000", 2024-09-21T14:08:46.561 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 493.27256713399998, 2024-09-21T14:08:46.561 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26343834999999999, 2024-09-21T14:08:46.561 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:08:46.561 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:08:46.561 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:08:46.561 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:08:46.561 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25421", 2024-09-21T14:08:46.561 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:08:46.561 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 172334 2024-09-21T14:08:46.562 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.562 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:08:46.562 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.562 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:08:46.562 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.260571+0000", 2024-09-21T14:08:46.562 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:46.562 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.562 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.563 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:08:46.563 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.260571+0000", 2024-09-21T14:08:46.563 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:46.563 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.563 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.563 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:08:46.563 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.260573+0000", 2024-09-21T14:08:46.564 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2839999999999999e-06 2024-09-21T14:08:46.564 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.564 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.564 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:08:46.564 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.260577+0000", 2024-09-21T14:08:46.564 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5940000000000004e-06 2024-09-21T14:08:46.564 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.565 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.565 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:08:46.565 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.260588+0000", 2024-09-21T14:08:46.565 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1163e-05 2024-09-21T14:08:46.565 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.565 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.565 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:08:46.566 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.260596+0000", 2024-09-21T14:08:46.566 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0520000000000004e-06 2024-09-21T14:08:46.566 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.566 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.566 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:46.566 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.260646+0000", 2024-09-21T14:08:46.566 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0593999999999997e-05 2024-09-21T14:08:46.567 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.567 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.567 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:08:46.567 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.260743+0000", 2024-09-21T14:08:46.567 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6998999999999997e-05 2024-09-21T14:08:46.567 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.567 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.567 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,4", 2024-09-21T14:08:46.567 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.260842+0000", 2024-09-21T14:08:46.567 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9185999999999995e-05 2024-09-21T14:08:46.567 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.567 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.568 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:08:46.568 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.274135+0000", 2024-09-21T14:08:46.568 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.013293142000000001 2024-09-21T14:08:46.568 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.568 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.568 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:46.568 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.274165+0000", 2024-09-21T14:08:46.569 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9277000000000001e-05 2024-09-21T14:08:46.569 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.569 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.569 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:46.569 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.523933+0000", 2024-09-21T14:08:46.569 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.249768038 2024-09-21T14:08:46.569 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.569 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.569 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:08:46.569 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.523976+0000", 2024-09-21T14:08:46.569 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3120000000000001e-05 2024-09-21T14:08:46.570 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.570 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.570 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:08:46.570 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.524010+0000", 2024-09-21T14:08:46.570 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3900999999999999e-05 2024-09-21T14:08:46.570 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:46.570 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:46.570 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:46.570 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.570 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.570 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25421.0:172337 9.3 9:c4d16778:::benchmark_data_smithi049_138504_object50929:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-09-21T14:08:46.570 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:00:33.262471+0000", 2024-09-21T14:08:46.570 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 493.27066805800001, 2024-09-21T14:08:46.571 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26159086300000001, 2024-09-21T14:08:46.571 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:08:46.571 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:08:46.571 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:08:46.571 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:08:46.571 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25421", 2024-09-21T14:08:46.571 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:08:46.571 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 172337 2024-09-21T14:08:46.571 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.571 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:08:46.571 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.571 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:08:46.571 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.262471+0000", 2024-09-21T14:08:46.571 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:46.571 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.572 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.572 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:08:46.572 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.262471+0000", 2024-09-21T14:08:46.572 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:46.572 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.572 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.572 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:08:46.572 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.262471+0000", 2024-09-21T14:08:46.572 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2500000000000004e-07 2024-09-21T14:08:46.572 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.572 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.572 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:08:46.572 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.262835+0000", 2024-09-21T14:08:46.572 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00036389300000000001 2024-09-21T14:08:46.573 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.573 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.573 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:08:46.573 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.262844+0000", 2024-09-21T14:08:46.573 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0299999999999999e-06 2024-09-21T14:08:46.573 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.573 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.573 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:08:46.573 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.262851+0000", 2024-09-21T14:08:46.573 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3600000000000001e-06 2024-09-21T14:08:46.573 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.573 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.573 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:46.573 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.263740+0000", 2024-09-21T14:08:46.573 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00088968700000000001 2024-09-21T14:08:46.574 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.574 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.574 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:08:46.574 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.273507+0000", 2024-09-21T14:08:46.574 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0097663909999999993 2024-09-21T14:08:46.574 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.574 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.574 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,7", 2024-09-21T14:08:46.574 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.273631+0000", 2024-09-21T14:08:46.574 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012441500000000001 2024-09-21T14:08:46.574 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.574 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.575 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:08:46.575 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.279244+0000", 2024-09-21T14:08:46.575 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0056131870000000004 2024-09-21T14:08:46.575 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.575 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.575 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:46.575 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.283719+0000", 2024-09-21T14:08:46.575 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0044745640000000003 2024-09-21T14:08:46.575 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.575 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.575 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:46.575 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.523913+0000", 2024-09-21T14:08:46.575 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.24019366 2024-09-21T14:08:46.575 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.576 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.576 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:08:46.576 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.523964+0000", 2024-09-21T14:08:46.576 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1039999999999999e-05 2024-09-21T14:08:46.576 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.576 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.576 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:08:46.576 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.524061+0000", 2024-09-21T14:08:46.576 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7711e-05 2024-09-21T14:08:46.576 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:46.576 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:46.576 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:46.577 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.577 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.577 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25421.0:172338 9.3 9:cb49996e:::benchmark_data_smithi049_138504_object50930:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-09-21T14:08:46.577 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:00:33.262872+0000", 2024-09-21T14:08:46.577 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 493.270266553, 2024-09-21T14:08:46.577 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26143280099999999, 2024-09-21T14:08:46.577 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:08:46.577 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:08:46.577 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:08:46.577 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:08:46.577 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25421", 2024-09-21T14:08:46.577 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:08:46.577 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 172338 2024-09-21T14:08:46.577 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.578 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:08:46.578 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.578 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:08:46.578 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.262872+0000", 2024-09-21T14:08:46.578 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:46.578 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.578 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.578 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:08:46.578 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.262872+0000", 2024-09-21T14:08:46.578 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:46.578 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.578 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.578 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:08:46.578 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.262873+0000", 2024-09-21T14:08:46.579 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5300000000000004e-07 2024-09-21T14:08:46.579 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.579 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.579 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:08:46.579 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.262875+0000", 2024-09-21T14:08:46.579 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.706e-06 2024-09-21T14:08:46.579 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.579 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.579 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:08:46.579 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.262882+0000", 2024-09-21T14:08:46.579 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4989999999999999e-06 2024-09-21T14:08:46.579 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.579 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.579 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:08:46.580 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.262887+0000", 2024-09-21T14:08:46.580 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9509999999999997e-06 2024-09-21T14:08:46.580 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.580 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.580 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:46.580 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:46.581 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.276120+0000", 2024-09-21T14:08:46.581 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.013233300999999999 2024-09-21T14:08:46.581 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.581 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.581 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:08:46.581 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.276230+0000", 2024-09-21T14:08:46.581 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000110229 2024-09-21T14:08:46.581 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.582 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.582 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,7", 2024-09-21T14:08:46.582 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.276349+0000", 2024-09-21T14:08:46.582 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011883900000000001 2024-09-21T14:08:46.582 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.582 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.582 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:08:46.582 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.283223+0000", 2024-09-21T14:08:46.582 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.006873891 2024-09-21T14:08:46.582 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.582 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.582 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:46.582 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.283857+0000", 2024-09-21T14:08:46.582 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00063396899999999996 2024-09-21T14:08:46.583 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.583 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.583 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:46.583 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.524123+0000", 2024-09-21T14:08:46.583 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.24026641700000001 2024-09-21T14:08:46.583 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.583 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.583 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:08:46.583 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.524197+0000", 2024-09-21T14:08:46.583 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3915e-05 2024-09-21T14:08:46.583 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.583 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.583 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:08:46.583 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.524305+0000", 2024-09-21T14:08:46.584 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000107431 2024-09-21T14:08:46.584 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:46.584 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:46.584 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:46.584 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.584 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.584 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25421.0:187674 9.d 9:b1fcb922:::benchmark_data_smithi049_138504_object66266:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e350)", 2024-09-21T14:08:46.584 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:01:45.364290+0000", 2024-09-21T14:08:46.584 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 421.168848712, 2024-09-21T14:08:46.584 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.23297105300000001, 2024-09-21T14:08:46.584 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:08:46.584 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:08:46.584 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:08:46.584 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:08:46.584 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25421", 2024-09-21T14:08:46.585 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:08:46.585 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 187674 2024-09-21T14:08:46.585 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.585 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:08:46.585 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.585 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:08:46.585 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:01:45.364290+0000", 2024-09-21T14:08:46.585 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:46.585 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.585 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.585 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:08:46.585 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:01:45.364290+0000", 2024-09-21T14:08:46.585 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:46.585 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.585 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.586 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:08:46.586 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:01:45.364292+0000", 2024-09-21T14:08:46.586 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5069999999999999e-06 2024-09-21T14:08:46.586 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.586 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.586 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:08:46.586 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:01:45.364302+0000", 2024-09-21T14:08:46.586 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.764e-06 2024-09-21T14:08:46.586 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.586 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.586 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:08:46.586 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:01:45.364325+0000", 2024-09-21T14:08:46.586 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2481e-05 2024-09-21T14:08:46.586 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.587 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.587 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:08:46.587 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:01:45.364335+0000", 2024-09-21T14:08:46.587 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0591000000000001e-05 2024-09-21T14:08:46.587 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.587 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.587 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:46.587 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:01:45.364417+0000", 2024-09-21T14:08:46.587 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2250999999999996e-05 2024-09-21T14:08:46.587 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.587 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.587 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:08:46.587 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:01:45.364547+0000", 2024-09-21T14:08:46.587 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012936100000000001 2024-09-21T14:08:46.588 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.588 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.588 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,7", 2024-09-21T14:08:46.588 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:01:45.364720+0000", 2024-09-21T14:08:46.588 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000173139 2024-09-21T14:08:46.588 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.588 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.588 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:08:46.588 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:01:45.367514+0000", 2024-09-21T14:08:46.588 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0027938400000000001 2024-09-21T14:08:46.588 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.588 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.588 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:46.588 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:01:45.367749+0000", 2024-09-21T14:08:46.589 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000234792 2024-09-21T14:08:46.589 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.589 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.589 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:46.589 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:01:45.597049+0000", 2024-09-21T14:08:46.589 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22929991799999999 2024-09-21T14:08:46.589 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.589 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.589 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:08:46.589 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:01:45.597148+0000", 2024-09-21T14:08:46.589 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9321000000000007e-05 2024-09-21T14:08:46.589 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.589 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.590 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:08:46.590 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:01:45.597261+0000", 2024-09-21T14:08:46.590 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000113088 2024-09-21T14:08:46.590 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:46.590 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:46.590 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:46.590 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.590 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.590 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:131799 10.2 10:464db636:::benchmark_data_smithi049_176811_object4222:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-09-21T14:08:46.590 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:04:39.757991+0000", 2024-09-21T14:08:46.590 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 246.775147487, 2024-09-21T14:08:46.590 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27366854600000001, 2024-09-21T14:08:46.591 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:08:46.591 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:08:46.591 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:08:46.591 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:08:46.591 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:08:46.591 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:08:46.591 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 131799 2024-09-21T14:08:46.591 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.591 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:08:46.591 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.591 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:08:46.591 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:39.757991+0000", 2024-09-21T14:08:46.591 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:46.591 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.592 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.592 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:08:46.592 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:39.757991+0000", 2024-09-21T14:08:46.592 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:46.592 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.592 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.592 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:08:46.592 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:39.757993+0000", 2024-09-21T14:08:46.592 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7749999999999999e-06 2024-09-21T14:08:46.592 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.592 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.592 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:08:46.592 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:39.757999+0000", 2024-09-21T14:08:46.592 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9510000000000003e-06 2024-09-21T14:08:46.592 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.593 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.593 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:08:46.593 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:39.758011+0000", 2024-09-21T14:08:46.593 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2663000000000001e-05 2024-09-21T14:08:46.593 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.593 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.593 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:08:46.593 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:39.758019+0000", 2024-09-21T14:08:46.593 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1189999999999999e-06 2024-09-21T14:08:46.593 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.593 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.593 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:46.594 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:39.758078+0000", 2024-09-21T14:08:46.594 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9091000000000001e-05 2024-09-21T14:08:46.594 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.594 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.594 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:08:46.594 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:39.758290+0000", 2024-09-21T14:08:46.594 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000212258 2024-09-21T14:08:46.594 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.594 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.594 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,4", 2024-09-21T14:08:46.594 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:39.758402+0000", 2024-09-21T14:08:46.594 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000111629 2024-09-21T14:08:46.594 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.595 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.595 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:08:46.595 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:39.763486+0000", 2024-09-21T14:08:46.595 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0050847310000000003 2024-09-21T14:08:46.595 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.595 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.595 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:46.595 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:39.764304+0000", 2024-09-21T14:08:46.595 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00081737699999999999 2024-09-21T14:08:46.595 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.595 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.595 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:46.595 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:40.031555+0000", 2024-09-21T14:08:46.595 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26725161200000003 2024-09-21T14:08:46.596 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.596 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.596 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:08:46.596 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:40.031612+0000", 2024-09-21T14:08:46.596 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7058000000000002e-05 2024-09-21T14:08:46.596 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.596 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.596 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:08:46.596 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:40.031660+0000", 2024-09-21T14:08:46.596 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7281999999999999e-05 2024-09-21T14:08:46.596 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:46.596 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:46.597 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:46.597 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.597 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.597 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:160592 10.2 10:49fe2980:::benchmark_data_smithi049_176811_object33015:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-09-21T14:08:46.597 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:05:07.645804+0000", 2024-09-21T14:08:46.597 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 218.88733438700001, 2024-09-21T14:08:46.597 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.226679192, 2024-09-21T14:08:46.597 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:08:46.597 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:08:46.597 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:08:46.597 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:08:46.597 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:08:46.597 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:08:46.597 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 160592 2024-09-21T14:08:46.598 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.598 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:08:46.598 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.598 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:08:46.598 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:07.645804+0000", 2024-09-21T14:08:46.598 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:46.598 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.598 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.598 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:08:46.598 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:07.645804+0000", 2024-09-21T14:08:46.598 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:46.598 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.598 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.598 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:08:46.599 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:07.645806+0000", 2024-09-21T14:08:46.599 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7600000000000001e-06 2024-09-21T14:08:46.599 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.599 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.599 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:08:46.599 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:07.645812+0000", 2024-09-21T14:08:46.599 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4620000000000001e-06 2024-09-21T14:08:46.599 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.599 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.599 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:08:46.599 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:07.645826+0000", 2024-09-21T14:08:46.599 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3645999999999999e-05 2024-09-21T14:08:46.599 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.599 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.599 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:08:46.600 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:07.645833+0000", 2024-09-21T14:08:46.600 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2420000000000003e-06 2024-09-21T14:08:46.600 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.600 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.600 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:46.600 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:07.645888+0000", 2024-09-21T14:08:46.600 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4339999999999998e-05 2024-09-21T14:08:46.600 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.600 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.600 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:08:46.600 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:07.646029+0000", 2024-09-21T14:08:46.600 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000141328 2024-09-21T14:08:46.600 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.600 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.601 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,4", 2024-09-21T14:08:46.601 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:07.646146+0000", 2024-09-21T14:08:46.601 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000116995 2024-09-21T14:08:46.601 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.601 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.601 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:08:46.601 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:07.648981+0000", 2024-09-21T14:08:46.601 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002835148 2024-09-21T14:08:46.601 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.601 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.601 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:46.601 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:07.654162+0000", 2024-09-21T14:08:46.601 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0051807959999999997 2024-09-21T14:08:46.601 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.601 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.602 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:46.602 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:07.872102+0000", 2024-09-21T14:08:46.602 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21793960200000001 2024-09-21T14:08:46.602 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.602 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.602 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:08:46.602 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:07.872429+0000", 2024-09-21T14:08:46.602 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00032782800000000001 2024-09-21T14:08:46.602 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.602 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.602 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:08:46.602 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:07.872483+0000", 2024-09-21T14:08:46.602 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4045000000000002e-05 2024-09-21T14:08:46.603 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:46.603 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:46.603 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:46.603 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.603 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.603 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26396.0:61786 11.7 e476/461 11:ec5bd234:::benchmark_data_smithi049_206132_object61785:head v 476'3804, mlcod=476'3801)", 2024-09-21T14:08:46.603 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:07:27.524950+0000", 2024-09-21T14:08:46.603 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 79.008188595999997, 2024-09-21T14:08:46.603 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.206763382, 2024-09-21T14:08:46.603 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:08:46.603 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:08:46.603 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:08:46.603 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:08:46.603 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.603 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:08:46.604 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.524950+0000", 2024-09-21T14:08:46.604 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:46.604 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.604 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.604 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:08:46.604 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.524950+0000", 2024-09-21T14:08:46.604 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:46.604 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.604 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.604 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:08:46.604 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.524951+0000", 2024-09-21T14:08:46.604 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9999999999999996e-07 2024-09-21T14:08:46.604 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.605 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.605 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:08:46.605 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.525152+0000", 2024-09-21T14:08:46.605 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000200946 2024-09-21T14:08:46.605 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.605 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.605 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:08:46.605 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.525163+0000", 2024-09-21T14:08:46.605 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1225e-05 2024-09-21T14:08:46.605 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.605 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.605 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:08:46.605 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.525169+0000", 2024-09-21T14:08:46.605 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1800000000000001e-06 2024-09-21T14:08:46.605 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.606 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.606 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:46.606 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.527418+0000", 2024-09-21T14:08:46.606 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002248426 2024-09-21T14:08:46.606 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.606 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.606 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:08:46.606 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.527435+0000", 2024-09-21T14:08:46.606 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7496999999999999e-05 2024-09-21T14:08:46.606 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.606 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.606 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:08:46.606 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.731665+0000", 2024-09-21T14:08:46.607 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.20422974099999999 2024-09-21T14:08:46.607 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.607 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.607 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:08:46.607 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.731713+0000", 2024-09-21T14:08:46.607 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8566999999999998e-05 2024-09-21T14:08:46.607 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:46.607 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:46.607 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:46.607 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.607 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.607 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26396.0:61790 11.b e476/461 11:d012fa8a:::benchmark_data_smithi049_206132_object61789:head v 476'3841, mlcod=476'3837)", 2024-09-21T14:08:46.607 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:07:27.528299+0000", 2024-09-21T14:08:46.608 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 79.004839660000002, 2024-09-21T14:08:46.608 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.20577252800000001, 2024-09-21T14:08:46.608 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:08:46.608 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:08:46.608 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:08:46.608 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:08:46.608 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.608 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:08:46.608 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.528299+0000", 2024-09-21T14:08:46.608 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:46.608 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.608 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.608 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:08:46.609 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.528299+0000", 2024-09-21T14:08:46.609 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:46.609 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.609 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.609 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:08:46.610 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.528300+0000", 2024-09-21T14:08:46.610 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9400000000000004e-07 2024-09-21T14:08:46.610 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.610 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.610 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:08:46.610 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.528323+0000", 2024-09-21T14:08:46.610 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3632999999999999e-05 2024-09-21T14:08:46.611 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.611 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.611 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:08:46.611 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.528332+0000", 2024-09-21T14:08:46.611 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4889999999999995e-06 2024-09-21T14:08:46.611 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.611 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.611 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:08:46.611 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.528339+0000", 2024-09-21T14:08:46.611 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5710000000000003e-06 2024-09-21T14:08:46.611 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.611 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.611 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:46.611 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.529418+0000", 2024-09-21T14:08:46.612 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0010798330000000001 2024-09-21T14:08:46.612 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.612 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.612 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:08:46.612 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.529436+0000", 2024-09-21T14:08:46.612 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.747e-05 2024-09-21T14:08:46.612 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.612 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.612 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:08:46.612 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.734038+0000", 2024-09-21T14:08:46.612 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.20460215900000001 2024-09-21T14:08:46.612 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.612 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.613 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:08:46.613 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.734071+0000", 2024-09-21T14:08:46.613 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3479000000000003e-05 2024-09-21T14:08:46.613 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:46.613 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:46.613 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:46.613 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.613 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.613 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:61805 11.0 11:027baeac:::benchmark_data_smithi049_206132_object61804:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e476)", 2024-09-21T14:08:46.613 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:07:27.533085+0000", 2024-09-21T14:08:46.613 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 79.000053284000003, 2024-09-21T14:08:46.613 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.207077343, 2024-09-21T14:08:46.613 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:08:46.614 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:08:46.614 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:08:46.614 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:08:46.614 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:08:46.614 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:08:46.614 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 61805 2024-09-21T14:08:46.614 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.614 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:08:46.614 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.614 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:08:46.614 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.533085+0000", 2024-09-21T14:08:46.614 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:46.615 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.615 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.615 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:08:46.615 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.533085+0000", 2024-09-21T14:08:46.615 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:46.615 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.615 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.615 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:08:46.615 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.533087+0000", 2024-09-21T14:08:46.615 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0190000000000001e-06 2024-09-21T14:08:46.615 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.615 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.616 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:08:46.616 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.533112+0000", 2024-09-21T14:08:46.616 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4450000000000001e-05 2024-09-21T14:08:46.616 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.616 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.616 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:08:46.616 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.533124+0000", 2024-09-21T14:08:46.616 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.271e-05 2024-09-21T14:08:46.616 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.616 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.616 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:08:46.616 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.533132+0000", 2024-09-21T14:08:46.616 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.418e-06 2024-09-21T14:08:46.616 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.617 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.617 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:46.617 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.533198+0000", 2024-09-21T14:08:46.617 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6434999999999997e-05 2024-09-21T14:08:46.617 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.617 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.617 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:08:46.617 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.533334+0000", 2024-09-21T14:08:46.617 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000135811 2024-09-21T14:08:46.617 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.617 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.617 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,7", 2024-09-21T14:08:46.617 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.731777+0000", 2024-09-21T14:08:46.617 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.198442919 2024-09-21T14:08:46.618 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.618 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.618 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:08:46.618 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.739939+0000", 2024-09-21T14:08:46.618 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0081617180000000001 2024-09-21T14:08:46.618 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.618 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.618 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:46.618 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.739982+0000", 2024-09-21T14:08:46.618 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2898000000000002e-05 2024-09-21T14:08:46.618 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.618 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.618 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:46.619 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.740042+0000", 2024-09-21T14:08:46.619 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0609999999999997e-05 2024-09-21T14:08:46.619 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.619 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.619 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:08:46.619 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.740103+0000", 2024-09-21T14:08:46.619 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0526999999999998e-05 2024-09-21T14:08:46.619 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.619 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.619 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:08:46.619 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.740163+0000", 2024-09-21T14:08:46.619 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9827999999999997e-05 2024-09-21T14:08:46.619 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:46.620 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:46.620 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:46.620 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.620 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.620 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:96825 11.c 11:3fd589d7:::benchmark_data_smithi049_206132_object96824:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e488)", 2024-09-21T14:08:46.620 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:07:56.787373+0000", 2024-09-21T14:08:46.620 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 49.745765165999998, 2024-09-21T14:08:46.620 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22893764899999999, 2024-09-21T14:08:46.620 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:08:46.620 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:08:46.620 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:08:46.620 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:08:46.621 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:08:46.621 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:08:46.621 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 96825 2024-09-21T14:08:46.621 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.621 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:08:46.621 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.621 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:08:46.621 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:56.787373+0000", 2024-09-21T14:08:46.621 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:46.621 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.621 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.621 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:08:46.621 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:56.787373+0000", 2024-09-21T14:08:46.621 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:46.621 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.622 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.622 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:08:46.622 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:56.787374+0000", 2024-09-21T14:08:46.622 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.004e-06 2024-09-21T14:08:46.622 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.622 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.622 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:08:46.622 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:56.787394+0000", 2024-09-21T14:08:46.622 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9409e-05 2024-09-21T14:08:46.622 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.622 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.622 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:08:46.623 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:56.787407+0000", 2024-09-21T14:08:46.623 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3396e-05 2024-09-21T14:08:46.623 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.623 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.623 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:08:46.623 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:56.787414+0000", 2024-09-21T14:08:46.623 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7290000000000003e-06 2024-09-21T14:08:46.623 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.623 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.623 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:46.623 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:56.787482+0000", 2024-09-21T14:08:46.623 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8089999999999994e-05 2024-09-21T14:08:46.623 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.623 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.624 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:08:46.624 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:56.787609+0000", 2024-09-21T14:08:46.624 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012669100000000001 2024-09-21T14:08:46.624 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.624 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.624 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,4", 2024-09-21T14:08:46.624 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:56.787754+0000", 2024-09-21T14:08:46.624 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014529899999999999 2024-09-21T14:08:46.624 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.624 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.624 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:08:46.624 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:56.790113+0000", 2024-09-21T14:08:46.624 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0023590130000000001 2024-09-21T14:08:46.624 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.625 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.625 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:46.625 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:56.795554+0000", 2024-09-21T14:08:46.625 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0054409280000000003 2024-09-21T14:08:46.625 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.625 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.625 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:46.625 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:57.016171+0000", 2024-09-21T14:08:46.625 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.220616909 2024-09-21T14:08:46.625 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.625 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.625 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:08:46.625 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:57.016266+0000", 2024-09-21T14:08:46.625 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4871999999999996e-05 2024-09-21T14:08:46.626 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.626 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.626 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:08:46.626 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:57.016311+0000", 2024-09-21T14:08:46.626 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5309000000000003e-05 2024-09-21T14:08:46.626 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:46.626 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:46.626 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:46.626 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.626 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.626 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:96826 11.c 11:31dc2e28:::benchmark_data_smithi049_206132_object96825:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e488)", 2024-09-21T14:08:46.626 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:07:56.790257+0000", 2024-09-21T14:08:46.626 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 49.742881830000002, 2024-09-21T14:08:46.626 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22620938800000001, 2024-09-21T14:08:46.627 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:08:46.627 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:08:46.627 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:08:46.627 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:08:46.627 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:08:46.627 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:08:46.627 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 96826 2024-09-21T14:08:46.627 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.627 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:08:46.627 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.627 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:08:46.627 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:56.790257+0000", 2024-09-21T14:08:46.627 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:46.627 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.628 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.628 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:08:46.628 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:56.790257+0000", 2024-09-21T14:08:46.628 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:46.628 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.628 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.628 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:08:46.628 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:56.790259+0000", 2024-09-21T14:08:46.628 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8470000000000001e-06 2024-09-21T14:08:46.628 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.628 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.628 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:08:46.628 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:56.790278+0000", 2024-09-21T14:08:46.628 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8958999999999999e-05 2024-09-21T14:08:46.628 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.629 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.629 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:08:46.629 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:56.790291+0000", 2024-09-21T14:08:46.629 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3413e-05 2024-09-21T14:08:46.629 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.629 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.629 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:08:46.629 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:56.790296+0000", 2024-09-21T14:08:46.629 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8980000000000004e-06 2024-09-21T14:08:46.629 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.629 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.629 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:46.629 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:56.790338+0000", 2024-09-21T14:08:46.630 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1857999999999998e-05 2024-09-21T14:08:46.630 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.630 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.630 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:08:46.630 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:56.790474+0000", 2024-09-21T14:08:46.630 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013657 2024-09-21T14:08:46.630 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.630 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.630 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,4", 2024-09-21T14:08:46.630 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:56.790581+0000", 2024-09-21T14:08:46.630 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000106912 2024-09-21T14:08:46.630 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.631 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.631 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:46.631 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:56.795607+0000", 2024-09-21T14:08:46.631 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0050257460000000002 2024-09-21T14:08:46.631 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.631 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.631 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:08:46.631 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:56.801685+0000", 2024-09-21T14:08:46.631 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0060778810000000003 2024-09-21T14:08:46.631 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.631 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.631 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:46.631 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:57.016348+0000", 2024-09-21T14:08:46.631 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21466318300000001 2024-09-21T14:08:46.632 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.632 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.632 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:08:46.632 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:57.016430+0000", 2024-09-21T14:08:46.632 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2387999999999998e-05 2024-09-21T14:08:46.632 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.632 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.632 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:08:46.632 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:57.016466+0000", 2024-09-21T14:08:46.632 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5732999999999998e-05 2024-09-21T14:08:46.632 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:46.632 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:46.633 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:46.633 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.633 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.633 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:96832 11.c 11:3d11df7f:::benchmark_data_smithi049_206132_object96831:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e488)", 2024-09-21T14:08:46.633 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:07:56.791201+0000", 2024-09-21T14:08:46.633 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 49.741937366999998, 2024-09-21T14:08:46.633 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22994894599999999, 2024-09-21T14:08:46.633 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:08:46.633 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:08:46.633 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:08:46.634 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:08:46.634 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:08:46.634 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:08:46.634 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 96832 2024-09-21T14:08:46.634 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.634 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:08:46.634 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.634 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:08:46.635 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:56.791201+0000", 2024-09-21T14:08:46.635 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:46.635 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.635 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.635 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:08:46.635 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:56.791201+0000", 2024-09-21T14:08:46.635 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:46.636 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.636 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.636 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:08:46.636 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:56.791202+0000", 2024-09-21T14:08:46.636 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.018e-06 2024-09-21T14:08:46.636 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.637 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.637 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:08:46.637 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:56.791223+0000", 2024-09-21T14:08:46.637 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0358999999999999e-05 2024-09-21T14:08:46.637 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.637 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.637 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:08:46.638 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:56.791246+0000", 2024-09-21T14:08:46.638 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3621e-05 2024-09-21T14:08:46.638 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.638 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.638 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:08:46.638 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:56.791254+0000", 2024-09-21T14:08:46.638 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2989999999999999e-06 2024-09-21T14:08:46.639 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.639 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.639 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:46.639 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:56.795642+0000", 2024-09-21T14:08:46.639 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0043881110000000001 2024-09-21T14:08:46.639 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.640 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.640 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:08:46.640 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:56.795772+0000", 2024-09-21T14:08:46.640 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000130345 2024-09-21T14:08:46.640 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.640 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.640 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,4", 2024-09-21T14:08:46.641 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:56.795911+0000", 2024-09-21T14:08:46.641 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000139281 2024-09-21T14:08:46.641 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.641 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.641 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:46.641 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:56.801721+0000", 2024-09-21T14:08:46.641 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0058096409999999999 2024-09-21T14:08:46.642 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.642 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.642 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:08:46.642 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:56.801731+0000", 2024-09-21T14:08:46.642 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0604999999999999e-05 2024-09-21T14:08:46.642 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.642 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.643 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:46.643 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:57.020848+0000", 2024-09-21T14:08:46.643 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.2191167 2024-09-21T14:08:46.643 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.643 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.643 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:08:46.644 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:57.021104+0000", 2024-09-21T14:08:46.644 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00025535 2024-09-21T14:08:46.644 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.644 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.644 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:08:46.644 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:57.021150+0000", 2024-09-21T14:08:46.644 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6616000000000002e-05 2024-09-21T14:08:46.645 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:46.645 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:46.645 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:46.645 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.645 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.645 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:96839 11.c 11:325f3c89:::benchmark_data_smithi049_206132_object96838:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e488)", 2024-09-21T14:08:46.646 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:07:56.797327+0000", 2024-09-21T14:08:46.646 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 49.735811292000001, 2024-09-21T14:08:46.646 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.22403147500000001, 2024-09-21T14:08:46.646 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:08:46.646 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:08:46.646 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:08:46.647 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:08:46.647 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:08:46.647 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:08:46.647 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 96839 2024-09-21T14:08:46.647 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.647 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:08:46.647 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.648 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:08:46.648 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:56.797327+0000", 2024-09-21T14:08:46.648 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:46.648 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.648 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.648 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:08:46.648 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:56.797327+0000", 2024-09-21T14:08:46.649 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:46.649 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.649 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.649 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:08:46.649 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:56.797328+0000", 2024-09-21T14:08:46.649 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0079999999999999e-06 2024-09-21T14:08:46.649 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.650 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.650 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:08:46.650 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:56.797361+0000", 2024-09-21T14:08:46.650 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2218999999999998e-05 2024-09-21T14:08:46.650 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.650 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.650 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:08:46.650 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:56.797372+0000", 2024-09-21T14:08:46.650 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.186e-05 2024-09-21T14:08:46.651 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.651 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.651 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:08:46.651 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:56.797380+0000", 2024-09-21T14:08:46.651 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7200000000000006e-06 2024-09-21T14:08:46.651 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.651 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.651 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:46.651 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:56.799136+0000", 2024-09-21T14:08:46.651 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001756149 2024-09-21T14:08:46.651 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.651 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.652 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:08:46.652 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:56.799266+0000", 2024-09-21T14:08:46.652 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012969199999999999 2024-09-21T14:08:46.652 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.652 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.652 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,4", 2024-09-21T14:08:46.652 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:56.799424+0000", 2024-09-21T14:08:46.652 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00015774499999999999 2024-09-21T14:08:46.652 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.652 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.652 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:08:46.652 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:56.802257+0000", 2024-09-21T14:08:46.653 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002833159 2024-09-21T14:08:46.653 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.653 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.653 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:46.653 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:56.805101+0000", 2024-09-21T14:08:46.653 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002844273 2024-09-21T14:08:46.653 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.653 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.653 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:46.653 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:57.021261+0000", 2024-09-21T14:08:46.653 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21615975300000001 2024-09-21T14:08:46.653 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.653 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.653 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:08:46.653 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:57.021319+0000", 2024-09-21T14:08:46.654 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8318e-05 2024-09-21T14:08:46.654 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.654 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.654 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:08:46.654 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:57.021359+0000", 2024-09-21T14:08:46.654 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9579000000000002e-05 2024-09-21T14:08:46.654 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:46.654 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:46.654 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:46.654 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.654 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.654 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:96856 11.c 11:300c7686:::benchmark_data_smithi049_206132_object96855:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e488)", 2024-09-21T14:08:46.654 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:07:56.813792+0000", 2024-09-21T14:08:46.655 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 49.719346889999997, 2024-09-21T14:08:46.655 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.20771561599999999, 2024-09-21T14:08:46.655 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:08:46.655 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:08:46.655 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:08:46.655 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:08:46.655 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:08:46.655 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:08:46.655 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 96856 2024-09-21T14:08:46.655 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.655 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:08:46.655 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.655 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:08:46.655 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:56.813792+0000", 2024-09-21T14:08:46.656 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:46.656 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.656 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.656 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:08:46.656 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:56.813792+0000", 2024-09-21T14:08:46.656 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:46.656 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.656 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.656 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:08:46.656 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:56.813793+0000", 2024-09-21T14:08:46.656 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2550000000000001e-06 2024-09-21T14:08:46.656 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.656 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.656 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:08:46.656 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:56.813814+0000", 2024-09-21T14:08:46.657 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1033e-05 2024-09-21T14:08:46.657 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.657 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.657 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:08:46.657 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:56.813828+0000", 2024-09-21T14:08:46.657 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3937999999999999e-05 2024-09-21T14:08:46.657 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.657 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.657 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:08:46.657 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:56.813836+0000", 2024-09-21T14:08:46.657 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9889999999999992e-06 2024-09-21T14:08:46.657 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.657 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.657 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:46.657 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:56.813892+0000", 2024-09-21T14:08:46.658 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6579000000000002e-05 2024-09-21T14:08:46.658 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.658 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.658 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:08:46.658 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:56.814001+0000", 2024-09-21T14:08:46.658 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010901 2024-09-21T14:08:46.658 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.658 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.658 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,4", 2024-09-21T14:08:46.658 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:56.814095+0000", 2024-09-21T14:08:46.658 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3572999999999993e-05 2024-09-21T14:08:46.658 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.658 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.658 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:08:46.658 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:56.816029+0000", 2024-09-21T14:08:46.659 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0019343850000000001 2024-09-21T14:08:46.659 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.659 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.659 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:46.659 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:56.816381+0000", 2024-09-21T14:08:46.659 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00035193000000000003 2024-09-21T14:08:46.659 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.659 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.659 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:46.659 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:57.021398+0000", 2024-09-21T14:08:46.659 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.205016532 2024-09-21T14:08:46.659 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.659 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.659 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:08:46.659 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:57.021446+0000", 2024-09-21T14:08:46.660 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7639999999999998e-05 2024-09-21T14:08:46.660 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.660 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.660 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:08:46.660 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:57.021507+0000", 2024-09-21T14:08:46.660 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1752000000000001e-05 2024-09-21T14:08:46.660 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:46.660 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:46.660 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:46.660 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.660 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.661 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:114649 11.0 11:068fefea:::benchmark_data_smithi049_206132_object114648:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e496)", 2024-09-21T14:08:46.661 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:08:13.432904+0000", 2024-09-21T14:08:46.661 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 33.100234241000003, 2024-09-21T14:08:46.661 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0255742080000001, 2024-09-21T14:08:46.662 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:08:46.662 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:08:46.662 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:08:46.662 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:08:46.662 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:08:46.662 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:08:46.662 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 114649 2024-09-21T14:08:46.662 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.662 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:08:46.662 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.662 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:08:46.662 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.432904+0000", 2024-09-21T14:08:46.663 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:46.663 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.663 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.663 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:08:46.663 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.432904+0000", 2024-09-21T14:08:46.663 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:46.663 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.663 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.663 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:08:46.663 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.432906+0000", 2024-09-21T14:08:46.663 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9489999999999999e-06 2024-09-21T14:08:46.663 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.663 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.663 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:08:46.663 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.432929+0000", 2024-09-21T14:08:46.664 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3127000000000001e-05 2024-09-21T14:08:46.664 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.664 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.664 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:08:46.664 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.432944+0000", 2024-09-21T14:08:46.664 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4337e-05 2024-09-21T14:08:46.664 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.664 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.664 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:08:46.664 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.432952+0000", 2024-09-21T14:08:46.664 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.028e-06 2024-09-21T14:08:46.664 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.664 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.664 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:46.665 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.433019+0000", 2024-09-21T14:08:46.665 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7089999999999996e-05 2024-09-21T14:08:46.665 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.665 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.665 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:08:46.665 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.433028+0000", 2024-09-21T14:08:46.665 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1479999999999995e-06 2024-09-21T14:08:46.665 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.665 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.665 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:46.665 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:14.399063+0000", 2024-09-21T14:08:46.665 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.96603520200000004 2024-09-21T14:08:46.665 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.665 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.666 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:08:46.666 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:14.399076+0000", 2024-09-21T14:08:46.666 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3125999999999999e-05 2024-09-21T14:08:46.666 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.666 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.666 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:46.666 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.450194+0000", 2024-09-21T14:08:46.666 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0511174190000001 2024-09-21T14:08:46.666 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.666 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.666 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:08:46.666 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.450385+0000", 2024-09-21T14:08:46.666 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000191545 2024-09-21T14:08:46.667 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.667 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.667 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,6,7", 2024-09-21T14:08:46.667 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.450862+0000", 2024-09-21T14:08:46.667 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00047710299999999999 2024-09-21T14:08:46.667 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.667 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.667 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:46.667 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.457471+0000", 2024-09-21T14:08:46.667 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0066087819999999997 2024-09-21T14:08:46.667 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.667 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.667 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:08:46.667 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.457506+0000", 2024-09-21T14:08:46.668 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4357000000000003e-05 2024-09-21T14:08:46.668 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.668 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.668 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:46.668 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.457770+0000", 2024-09-21T14:08:46.668 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00026468299999999999 2024-09-21T14:08:46.668 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.668 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.668 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:46.668 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.458276+0000", 2024-09-21T14:08:46.668 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00050568800000000001 2024-09-21T14:08:46.668 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.668 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.668 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:08:46.668 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.458352+0000", 2024-09-21T14:08:46.669 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5975000000000004e-05 2024-09-21T14:08:46.669 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.669 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.669 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:08:46.669 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.458479+0000", 2024-09-21T14:08:46.669 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000126649 2024-09-21T14:08:46.669 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:46.669 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:46.669 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:46.670 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.670 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.670 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:114668 11.0 11:073d9639:::benchmark_data_smithi049_206132_object114667:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e496)", 2024-09-21T14:08:46.670 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:08:13.450326+0000", 2024-09-21T14:08:46.670 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 33.082812400000002, 2024-09-21T14:08:46.670 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0164301259999999, 2024-09-21T14:08:46.670 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:08:46.670 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:08:46.670 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:08:46.671 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:08:46.671 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:08:46.671 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:08:46.671 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 114668 2024-09-21T14:08:46.671 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.671 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:08:46.671 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.671 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:08:46.671 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.450326+0000", 2024-09-21T14:08:46.672 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:46.672 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.672 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.672 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:08:46.672 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.450326+0000", 2024-09-21T14:08:46.672 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:08:46.672 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.672 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.672 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:08:46.673 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.450328+0000", 2024-09-21T14:08:46.673 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.556e-06 2024-09-21T14:08:46.673 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.673 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.673 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:08:46.673 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.450354+0000", 2024-09-21T14:08:46.673 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6658000000000001e-05 2024-09-21T14:08:46.673 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.673 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.673 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:08:46.674 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.450367+0000", 2024-09-21T14:08:46.674 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2686e-05 2024-09-21T14:08:46.674 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.674 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.674 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:08:46.674 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.450375+0000", 2024-09-21T14:08:46.674 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0069999999999997e-06 2024-09-21T14:08:46.674 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.675 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.675 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:46.675 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.450451+0000", 2024-09-21T14:08:46.675 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6359999999999994e-05 2024-09-21T14:08:46.675 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.675 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.675 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:08:46.675 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.450460+0000", 2024-09-21T14:08:46.676 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1880000000000002e-06 2024-09-21T14:08:46.676 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.676 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.676 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:46.676 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:14.399105+0000", 2024-09-21T14:08:46.676 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.94864543700000004 2024-09-21T14:08:46.676 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.676 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.676 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:08:46.676 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:14.399114+0000", 2024-09-21T14:08:46.677 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4640000000000006e-06 2024-09-21T14:08:46.677 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.677 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.677 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:08:46.677 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.454780+0000", 2024-09-21T14:08:46.677 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.055666666 2024-09-21T14:08:46.677 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.677 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.678 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:08:46.678 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.454970+0000", 2024-09-21T14:08:46.678 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00018995700000000001 2024-09-21T14:08:46.678 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.678 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.678 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,6,7", 2024-09-21T14:08:46.678 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.455174+0000", 2024-09-21T14:08:46.678 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00020342900000000001 2024-09-21T14:08:46.678 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.679 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.679 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:46.679 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.457569+0000", 2024-09-21T14:08:46.679 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0023951160000000001 2024-09-21T14:08:46.679 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.679 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.679 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:08:46.679 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.457709+0000", 2024-09-21T14:08:46.679 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014006000000000001 2024-09-21T14:08:46.680 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.680 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.680 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:46.680 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.457809+0000", 2024-09-21T14:08:46.680 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000100488 2024-09-21T14:08:46.680 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.680 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.680 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:08:46.680 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.466564+0000", 2024-09-21T14:08:46.681 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0087546389999999998 2024-09-21T14:08:46.681 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.681 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.681 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:08:46.681 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.466636+0000", 2024-09-21T14:08:46.681 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2305999999999995e-05 2024-09-21T14:08:46.681 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:08:46.681 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:08:46.681 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:08:46.682 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.466756+0000", 2024-09-21T14:08:46.682 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000120109 2024-09-21T14:08:46.682 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:46.682 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:46.682 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:46.682 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:08:46.682 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:08:46.682 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T14:08:46.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:46.683 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:46.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:46.738 INFO:teuthology.orchestra.run.smithi049.stderr:set 1.9 pg_upmap mapping to [2,3,0] 2024-09-21T14:08:46.752 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T14:08:46.784 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:46.843 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:46.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:46.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:46.946 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:08:46.999 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:47.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:47.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:08:47.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:46 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2583249065' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "1.9", "id": [2, 3, 0]} : dispatch 2024-09-21T14:08:47.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:46 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "1.9", "id": [2, 3, 0]} : dispatch 2024-09-21T14:08:47.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:46 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2583249065' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "1.9", "id": [2, 3, 0]} : dispatch 2024-09-21T14:08:47.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:46 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "1.9", "id": [2, 3, 0]} : dispatch 2024-09-21T14:08:47.101 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:08:47.160 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:47.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:47.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:08:47.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:46 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2583249065' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "1.9", "id": [2, 3, 0]} : dispatch 2024-09-21T14:08:47.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:46 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "1.9", "id": [2, 3, 0]} : dispatch 2024-09-21T14:08:47.262 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:08:47.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:47.301 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:47.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:08:47.403 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:08:47.449 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:47.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:47.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:08:47.551 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:47.593 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:47.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:47.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:47.694 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:08:47.729 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:47.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:47.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:08:47.752 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-09-21T14:08:47.830 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:47.861 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:47.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:47.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:47.963 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:48.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:47 smithi049 ceph-mon[30080]: pgmap v1012: 168 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 164 active+clean; 6.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.16k op/s; 13740/306771 objects misplaced (4.479%); 52 MiB/s, 830 objects/s recovering 2024-09-21T14:08:48.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:47 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.9", "id": [2, 3, 0]}]': finished 2024-09-21T14:08:48.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:47 smithi049 ceph-mon[30080]: osdmap e505: 8 total, 8 up, 8 in 2024-09-21T14:08:48.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:47 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2739533598' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:08:48.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:47 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:08:48.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:47 smithi049 ceph-mon[40489]: pgmap v1012: 168 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 164 active+clean; 6.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.16k op/s; 13740/306771 objects misplaced (4.479%); 52 MiB/s, 830 objects/s recovering 2024-09-21T14:08:48.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:47 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.9", "id": [2, 3, 0]}]': finished 2024-09-21T14:08:48.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:47 smithi049 ceph-mon[40489]: osdmap e505: 8 total, 8 up, 8 in 2024-09-21T14:08:48.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:47 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2739533598' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:08:48.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:47 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:08:48.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:48.006 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:48.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:48.107 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:08:48.160 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:48.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:48.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:08:48.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:47 smithi187 ceph-mon[33032]: pgmap v1012: 168 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 164 active+clean; 6.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.16k op/s; 13740/306771 objects misplaced (4.479%); 52 MiB/s, 830 objects/s recovering 2024-09-21T14:08:48.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:47 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.9", "id": [2, 3, 0]}]': finished 2024-09-21T14:08:48.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:47 smithi187 ceph-mon[33032]: osdmap e505: 8 total, 8 up, 8 in 2024-09-21T14:08:48.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:47 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2739533598' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:08:48.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:47 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:08:48.263 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:08:48.304 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:48.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:48.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:08:48.406 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:08:48.444 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:48.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:48.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:08:48.546 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:08:48.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:48.577 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:48.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:08:48.678 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:08:48.703 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:48.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:48.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:08:48.805 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:08:48.844 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:48.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:48.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:08:48.946 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:08:48.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:48.977 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:48.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:08:49.078 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:08:49.112 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:49.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:49.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:08:49.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:48 smithi187 ceph-mon[33032]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:08:49.201 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:48 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T14:08:49.201 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:48 smithi187 ceph-mon[33032]: osdmap e506: 8 total, 8 up, 8 in 2024-09-21T14:08:49.201 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:48 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T14:08:49.214 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:08:49.244 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:49.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:49.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:08:49.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:48 smithi049 ceph-mon[40489]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:08:49.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:48 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T14:08:49.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:48 smithi049 ceph-mon[40489]: osdmap e506: 8 total, 8 up, 8 in 2024-09-21T14:08:49.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:48 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T14:08:49.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:48 smithi049 ceph-mon[30080]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:08:49.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:48 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T14:08:49.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:48 smithi049 ceph-mon[30080]: osdmap e506: 8 total, 8 up, 8 in 2024-09-21T14:08:49.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:48 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T14:08:49.346 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:49.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:49.388 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:49.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:49.489 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:49.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:49.513 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:49.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:49.614 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:08:49.657 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:49.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:49.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:08:49.758 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:08:49.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:49.798 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:49.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:08:49.899 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:49.935 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:49.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:49.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:50.037 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:08:50.070 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:50.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:50.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:08:50.169 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:49 smithi187 ceph-mon[33032]: pgmap v1015: 168 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 164 active+clean; 6.2 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 791 B/s rd, 0 B/s wr, 711 op/s; 13740/304866 objects misplaced (4.507%); 30 MiB/s, 480 objects/s recovering 2024-09-21T14:08:50.169 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:49 smithi187 ceph-mon[33032]: 1.1 scrub starts 2024-09-21T14:08:50.169 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:49 smithi187 ceph-mon[33032]: 1.1 scrub ok 2024-09-21T14:08:50.169 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:49 smithi187 ceph-mon[33032]: 4.c scrub starts 2024-09-21T14:08:50.169 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:49 smithi187 ceph-mon[33032]: 4.c scrub ok 2024-09-21T14:08:50.172 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:08:50.205 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:50.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:50.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:08:50.208 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:49 smithi049 ceph-mon[40489]: pgmap v1015: 168 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 164 active+clean; 6.2 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 791 B/s rd, 0 B/s wr, 711 op/s; 13740/304866 objects misplaced (4.507%); 30 MiB/s, 480 objects/s recovering 2024-09-21T14:08:50.208 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:49 smithi049 ceph-mon[40489]: 1.1 scrub starts 2024-09-21T14:08:50.208 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:49 smithi049 ceph-mon[40489]: 1.1 scrub ok 2024-09-21T14:08:50.208 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:49 smithi049 ceph-mon[40489]: 4.c scrub starts 2024-09-21T14:08:50.209 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:49 smithi049 ceph-mon[40489]: 4.c scrub ok 2024-09-21T14:08:50.209 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:49 smithi049 ceph-mon[30080]: pgmap v1015: 168 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 164 active+clean; 6.2 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 791 B/s rd, 0 B/s wr, 711 op/s; 13740/304866 objects misplaced (4.507%); 30 MiB/s, 480 objects/s recovering 2024-09-21T14:08:50.209 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:49 smithi049 ceph-mon[30080]: 1.1 scrub starts 2024-09-21T14:08:50.209 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:49 smithi049 ceph-mon[30080]: 1.1 scrub ok 2024-09-21T14:08:50.209 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:49 smithi049 ceph-mon[30080]: 4.c scrub starts 2024-09-21T14:08:50.209 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:49 smithi049 ceph-mon[30080]: 4.c scrub ok 2024-09-21T14:08:50.308 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:50.342 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:50.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:50.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:50.444 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:08:50.444 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:08:50 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:08:50.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:50.487 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:50.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:08:50.588 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:08:50.634 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:50.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:50.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:08:50.735 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:08:50.780 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:50.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:50.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:08:50.881 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:50.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:50.919 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:50.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:50.958 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:50 smithi049 ceph-mon[40489]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-21T14:08:50.958 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:08:50 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:08:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:08:50.959 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:50 smithi049 ceph-mon[30080]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-21T14:08:51.019 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:51.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:51.053 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:51.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:51.154 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:08:51.191 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:51.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:51.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:08:51.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:50 smithi187 ceph-mon[33032]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-21T14:08:51.294 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:51.336 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:51.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:51.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:51.438 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:51.465 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:51.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:51.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:51.566 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:08:51.592 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:51.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:51.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:08:51.693 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:08:51.725 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:51.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:51.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:08:51.827 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:08:51.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:51.860 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:51.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:08:51.961 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:08:52.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:52.008 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:52.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:08:52.092 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:51 smithi049 ceph-mon[40489]: pgmap v1016: 168 pgs: 4 active+remapped+backfilling, 164 active+clean; 6.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 3.1 KiB/s rd, 18 KiB/s wr, 904 op/s; 11548/297000 objects misplaced (3.888%); 37 MiB/s, 596 objects/s recovering 2024-09-21T14:08:52.092 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:51 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:08:52.092 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:51 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T14:08:52.092 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:51 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T14:08:52.092 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:51 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:08:52.093 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T14:08:52.094 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:51 smithi049 ceph-mon[30080]: pgmap v1016: 168 pgs: 4 active+remapped+backfilling, 164 active+clean; 6.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 3.1 KiB/s rd, 18 KiB/s wr, 904 op/s; 11548/297000 objects misplaced (3.888%); 37 MiB/s, 596 objects/s recovering 2024-09-21T14:08:52.094 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:51 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:08:52.094 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:51 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T14:08:52.094 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:51 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T14:08:52.094 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:51 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:08:52.109 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:52.113 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:08:52.121 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:51 smithi187 ceph-mon[33032]: pgmap v1016: 168 pgs: 4 active+remapped+backfilling, 164 active+clean; 6.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 3.1 KiB/s rd, 18 KiB/s wr, 904 op/s; 11548/297000 objects misplaced (3.888%); 37 MiB/s, 596 objects/s recovering 2024-09-21T14:08:52.121 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:51 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:08:52.122 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:51 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T14:08:52.122 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:51 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T14:08:52.122 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:51 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:08:52.151 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:52.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:52.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:52.253 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:08:52.296 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:52.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:52.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:08:52.398 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:52.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:52.435 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:52.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:52.536 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:08:52.572 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:52.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:52.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:08:52.601 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:08:52.673 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:52.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:52.701 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:52.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:52.764 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:08:52.765 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:08:52.801 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:08:52.801 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:08:52.802 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:08:52.840 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:08:52.840 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:08:52.850 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:52.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:52.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:08:52.878 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:08:52.878 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:08:52.910 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:08:52.910 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:08:52.944 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:08:52.945 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:08:52.952 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:52.981 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:08:52.981 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:08:52.990 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:52.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:52.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:53.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:52 smithi049 ceph-mon[30080]: 3.1d scrub starts 2024-09-21T14:08:53.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:52 smithi049 ceph-mon[30080]: 3.1d scrub ok 2024-09-21T14:08:53.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:52 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/798282166' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:08:53.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:52 smithi049 ceph-mon[40489]: 3.1d scrub starts 2024-09-21T14:08:53.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:52 smithi049 ceph-mon[40489]: 3.1d scrub ok 2024-09-21T14:08:53.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:52 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/798282166' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:08:53.016 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:08:53.016 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:08:53.092 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:53.123 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:53.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:53.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:53.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:52 smithi187 ceph-mon[33032]: 3.1d scrub starts 2024-09-21T14:08:53.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:52 smithi187 ceph-mon[33032]: 3.1d scrub ok 2024-09-21T14:08:53.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:52 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/798282166' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:08:53.225 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:08:53.264 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:53.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:53.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:08:53.366 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:08:53.394 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:53.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:53.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:08:53.454 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T14:08:53.495 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:08:53.519 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:53.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:53.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:08:53.621 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:53.651 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:53.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:53.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:53.752 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:08:53.804 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:53.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:53.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:08:53.905 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:08:53.931 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:53.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:53.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:08:54.033 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:08:54.072 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:08:54.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:54.102 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:54.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:08:54.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:53 smithi187 ceph-mon[33032]: pgmap v1017: 168 pgs: 4 active+remapped+backfilling, 164 active+clean; 5.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 3.7 KiB/s rd, 18 KiB/s wr, 987 op/s; 10805/289161 objects misplaced (3.737%); 32 MiB/s, 516 objects/s recovering 2024-09-21T14:08:54.203 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:08:54.213 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:53 smithi049 ceph-mon[40489]: pgmap v1017: 168 pgs: 4 active+remapped+backfilling, 164 active+clean; 5.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 3.7 KiB/s rd, 18 KiB/s wr, 987 op/s; 10805/289161 objects misplaced (3.737%); 32 MiB/s, 516 objects/s recovering 2024-09-21T14:08:54.214 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:53 smithi049 ceph-mon[30080]: pgmap v1017: 168 pgs: 4 active+remapped+backfilling, 164 active+clean; 5.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 3.7 KiB/s rd, 18 KiB/s wr, 987 op/s; 10805/289161 objects misplaced (3.737%); 32 MiB/s, 516 objects/s recovering 2024-09-21T14:08:54.234 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:54.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:54.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:08:54.336 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:54.365 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:54.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:54.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:54.467 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:08:54.490 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:54.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:54.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:08:54.592 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:54.620 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:54.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:54.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:54.722 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:54.754 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:54.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:54.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:54.857 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:08:54.901 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:54.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:54.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:08:55.003 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:08:55.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:54 smithi049 ceph-mon[30080]: 6.e scrub starts 2024-09-21T14:08:55.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:54 smithi049 ceph-mon[30080]: 6.e scrub ok 2024-09-21T14:08:55.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:54 smithi049 ceph-mon[40489]: 6.e scrub starts 2024-09-21T14:08:55.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:54 smithi049 ceph-mon[40489]: 6.e scrub ok 2024-09-21T14:08:55.033 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:55.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:55.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:08:55.135 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:55.157 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:55.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:55.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:55.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:54 smithi187 ceph-mon[33032]: 6.e scrub starts 2024-09-21T14:08:55.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:54 smithi187 ceph-mon[33032]: 6.e scrub ok 2024-09-21T14:08:55.259 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:08:55.283 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:55.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:55.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:08:55.385 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:08:55.413 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:55.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:55.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:08:55.514 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:08:55.540 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:55.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:55.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:08:55.642 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:08:55.671 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:55.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:55.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:08:55.773 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:08:55.795 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T14:08:55.795 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:08:55.797 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:55.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:55.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:08:55.851 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-09-21T14:08:55.899 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:08:55.921 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:55.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:55.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:08:55.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:55 smithi187 ceph-mon[33032]: pgmap v1018: 168 pgs: 4 active+remapped+backfilling, 164 active+clean; 5.7 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 3.1 KiB/s rd, 18 KiB/s wr, 1.01k op/s; 9129/282483 objects misplaced (3.232%); 28 MiB/s, 451 objects/s recovering 2024-09-21T14:08:55.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:55 smithi187 ceph-mon[33032]: osdmap e507: 8 total, 8 up, 8 in 2024-09-21T14:08:55.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:55 smithi187 ceph-mon[33032]: 1.c scrub starts 2024-09-21T14:08:55.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:55 smithi187 ceph-mon[33032]: 1.c scrub ok 2024-09-21T14:08:55.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:55 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3861585039' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:08:55.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:55 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:08:55.946 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T14:08:56.023 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:08:56.068 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:56.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:56.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:08:56.169 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:08:56.219 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:56.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:56.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:08:56.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:55 smithi049 ceph-mon[40489]: pgmap v1018: 168 pgs: 4 active+remapped+backfilling, 164 active+clean; 5.7 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 3.1 KiB/s rd, 18 KiB/s wr, 1.01k op/s; 9129/282483 objects misplaced (3.232%); 28 MiB/s, 451 objects/s recovering 2024-09-21T14:08:56.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:55 smithi049 ceph-mon[40489]: osdmap e507: 8 total, 8 up, 8 in 2024-09-21T14:08:56.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:55 smithi049 ceph-mon[40489]: 1.c scrub starts 2024-09-21T14:08:56.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:55 smithi049 ceph-mon[40489]: 1.c scrub ok 2024-09-21T14:08:56.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:55 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3861585039' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:08:56.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:55 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:08:56.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:55 smithi049 ceph-mon[30080]: pgmap v1018: 168 pgs: 4 active+remapped+backfilling, 164 active+clean; 5.7 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 3.1 KiB/s rd, 18 KiB/s wr, 1.01k op/s; 9129/282483 objects misplaced (3.232%); 28 MiB/s, 451 objects/s recovering 2024-09-21T14:08:56.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:55 smithi049 ceph-mon[30080]: osdmap e507: 8 total, 8 up, 8 in 2024-09-21T14:08:56.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:55 smithi049 ceph-mon[30080]: 1.c scrub starts 2024-09-21T14:08:56.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:55 smithi049 ceph-mon[30080]: 1.c scrub ok 2024-09-21T14:08:56.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:55 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3861585039' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:08:56.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:55 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:08:56.321 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:08:56.355 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:56.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:56.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:08:56.457 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:08:56.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:56.491 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:56.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:08:56.592 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:56.601 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 4, 6, 2, 3, 7, 5] 2024-09-21T14:08:56.601 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T14:08:56.601 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-21T14:08:56.602 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd reweight-by-utilization 1000 3.0 2024-09-21T14:08:56.622 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:56.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:56.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:56.670 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T14:08:56.725 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:08:56.760 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:56.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:56.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:08:56.862 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:56.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:56.898 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:56.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:56.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:56 smithi187 ceph-mon[33032]: 11.5 scrub starts 2024-09-21T14:08:56.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:56 smithi187 ceph-mon[33032]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T14:08:56.946 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:56 smithi187 ceph-mon[33032]: Cluster is now healthy 2024-09-21T14:08:56.947 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:56 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T14:08:56.947 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:56 smithi187 ceph-mon[33032]: osdmap e508: 8 total, 8 up, 8 in 2024-09-21T14:08:56.947 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:56 smithi187 ceph-mon[33032]: 11.5 scrub ok 2024-09-21T14:08:56.999 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:08:57.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:56 smithi049 ceph-mon[40489]: 11.5 scrub starts 2024-09-21T14:08:57.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:56 smithi049 ceph-mon[40489]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T14:08:57.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:56 smithi049 ceph-mon[40489]: Cluster is now healthy 2024-09-21T14:08:57.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:56 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T14:08:57.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:56 smithi049 ceph-mon[40489]: osdmap e508: 8 total, 8 up, 8 in 2024-09-21T14:08:57.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:56 smithi049 ceph-mon[40489]: 11.5 scrub ok 2024-09-21T14:08:57.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:56 smithi049 ceph-mon[30080]: 11.5 scrub starts 2024-09-21T14:08:57.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:56 smithi049 ceph-mon[30080]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T14:08:57.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:56 smithi049 ceph-mon[30080]: Cluster is now healthy 2024-09-21T14:08:57.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:56 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T14:08:57.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:56 smithi049 ceph-mon[30080]: osdmap e508: 8 total, 8 up, 8 in 2024-09-21T14:08:57.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:56 smithi049 ceph-mon[30080]: 11.5 scrub ok 2024-09-21T14:08:57.038 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:57.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:57.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:08:57.054 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:08:57.141 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:57.163 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:57.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:57.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:57.264 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:08:57.305 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:57.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:57.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:08:57.406 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:08:57.443 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:57.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:57.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:08:57.544 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:57.582 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:57.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:57.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:57.612 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T14:08:57.612 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:08:57.684 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:08:57.710 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:57.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:57.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:08:57.811 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:08:57.836 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:57.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:57.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:08:57.939 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:08:57.972 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:57.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:57.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:08:58.074 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:58.102 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:58.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:58.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:58.173 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:57 smithi049 ceph-mon[30080]: pgmap v1021: 168 pgs: 1 active+clean+scrubbing, 1 active+clean+remapped, 3 active+remapped+backfilling, 163 active+clean; 5.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 3.7 KiB/s rd, 18 KiB/s wr, 1.22k op/s; 7578/275769 objects misplaced (2.748%); 39 MiB/s, 629 objects/s recovering 2024-09-21T14:08:58.173 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:57 smithi049 ceph-mon[30080]: 5.f scrub starts 2024-09-21T14:08:58.173 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:57 smithi049 ceph-mon[30080]: 5.f scrub ok 2024-09-21T14:08:58.173 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:57 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:08:58.174 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:57 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:08:58.174 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:57 smithi049 ceph-mon[30080]: osdmap e509: 8 total, 8 up, 8 in 2024-09-21T14:08:58.174 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:57 smithi049 ceph-mon[30080]: 6.10 scrub starts 2024-09-21T14:08:58.174 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:57 smithi049 ceph-mon[30080]: 6.10 scrub ok 2024-09-21T14:08:58.174 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:57 smithi049 ceph-mon[30080]: 6.17 scrub starts 2024-09-21T14:08:58.174 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:57 smithi049 ceph-mon[30080]: 6.17 scrub ok 2024-09-21T14:08:58.174 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:57 smithi049 ceph-mon[40489]: pgmap v1021: 168 pgs: 1 active+clean+scrubbing, 1 active+clean+remapped, 3 active+remapped+backfilling, 163 active+clean; 5.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 3.7 KiB/s rd, 18 KiB/s wr, 1.22k op/s; 7578/275769 objects misplaced (2.748%); 39 MiB/s, 629 objects/s recovering 2024-09-21T14:08:58.174 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:57 smithi049 ceph-mon[40489]: 5.f scrub starts 2024-09-21T14:08:58.174 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:57 smithi049 ceph-mon[40489]: 5.f scrub ok 2024-09-21T14:08:58.174 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:57 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:08:58.174 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:57 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:08:58.175 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:57 smithi049 ceph-mon[40489]: osdmap e509: 8 total, 8 up, 8 in 2024-09-21T14:08:58.175 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:57 smithi049 ceph-mon[40489]: 6.10 scrub starts 2024-09-21T14:08:58.175 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:57 smithi049 ceph-mon[40489]: 6.10 scrub ok 2024-09-21T14:08:58.175 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:57 smithi049 ceph-mon[40489]: 6.17 scrub starts 2024-09-21T14:08:58.175 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:57 smithi049 ceph-mon[40489]: 6.17 scrub ok 2024-09-21T14:08:58.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:57 smithi187 ceph-mon[33032]: pgmap v1021: 168 pgs: 1 active+clean+scrubbing, 1 active+clean+remapped, 3 active+remapped+backfilling, 163 active+clean; 5.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 3.7 KiB/s rd, 18 KiB/s wr, 1.22k op/s; 7578/275769 objects misplaced (2.748%); 39 MiB/s, 629 objects/s recovering 2024-09-21T14:08:58.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:57 smithi187 ceph-mon[33032]: 5.f scrub starts 2024-09-21T14:08:58.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:57 smithi187 ceph-mon[33032]: 5.f scrub ok 2024-09-21T14:08:58.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:57 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:08:58.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:57 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:08:58.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:57 smithi187 ceph-mon[33032]: osdmap e509: 8 total, 8 up, 8 in 2024-09-21T14:08:58.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:57 smithi187 ceph-mon[33032]: 6.10 scrub starts 2024-09-21T14:08:58.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:57 smithi187 ceph-mon[33032]: 6.10 scrub ok 2024-09-21T14:08:58.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:57 smithi187 ceph-mon[33032]: 6.17 scrub starts 2024-09-21T14:08:58.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:57 smithi187 ceph-mon[33032]: 6.17 scrub ok 2024-09-21T14:08:58.203 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:08:58.235 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:58.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:58.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:08:58.337 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:08:58.355 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T14:08:58.380 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:58.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:58.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:08:58.482 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:08:58.527 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:58.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:58.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:08:58.629 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:58.652 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:08:58.661 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:58.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:58.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:58.715 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:08:58.731 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:08:58.763 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:08:58.812 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:58.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:58.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:08:58.913 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:58.948 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:58.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:58.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:59.050 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:08:59.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:59.089 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:08:59.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:08:59.137 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:58 smithi049 ceph-mon[40489]: 3.11 scrub starts 2024-09-21T14:08:59.137 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:58 smithi049 ceph-mon[40489]: 3.11 scrub ok 2024-09-21T14:08:59.137 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:58 smithi049 ceph-mon[40489]: 5.13 scrub starts 2024-09-21T14:08:59.137 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:58 smithi049 ceph-mon[40489]: 5.13 scrub ok 2024-09-21T14:08:59.137 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:58 smithi049 ceph-mon[30080]: 3.11 scrub starts 2024-09-21T14:08:59.137 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:58 smithi049 ceph-mon[30080]: 3.11 scrub ok 2024-09-21T14:08:59.137 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:58 smithi049 ceph-mon[30080]: 5.13 scrub starts 2024-09-21T14:08:59.138 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:58 smithi049 ceph-mon[30080]: 5.13 scrub ok 2024-09-21T14:08:59.139 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:58 smithi187 ceph-mon[33032]: 3.11 scrub starts 2024-09-21T14:08:59.139 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:58 smithi187 ceph-mon[33032]: 3.11 scrub ok 2024-09-21T14:08:59.139 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:58 smithi187 ceph-mon[33032]: 5.13 scrub starts 2024-09-21T14:08:59.139 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:58 smithi187 ceph-mon[33032]: 5.13 scrub ok 2024-09-21T14:08:59.191 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:59.221 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:59.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:59.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:59.323 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:59.366 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:59.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:59.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:59.436 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T14:08:59.436 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:08:59.469 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:08:59.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:59.499 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:59.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:08:59.600 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:08:59.624 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:59.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:59.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:08:59.725 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:59.746 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:59.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:59.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:59.848 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:08:59.881 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:08:59.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:08:59.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:08:59.892 INFO:teuthology.orchestra.run.smithi049.stdout:moved 31 / 504 (6.15079%) 2024-09-21T14:08:59.893 INFO:teuthology.orchestra.run.smithi049.stdout:avg 63 2024-09-21T14:08:59.893 INFO:teuthology.orchestra.run.smithi049.stdout:stddev 18.5742 -> 16.7631 (expected baseline 7.42462) 2024-09-21T14:08:59.893 INFO:teuthology.orchestra.run.smithi049.stdout:min osd.3 with 25 -> 24 pgs (0.396825 -> 0.380952 * mean) 2024-09-21T14:08:59.893 INFO:teuthology.orchestra.run.smithi049.stdout:max osd.1 with 82 -> 78 pgs (1.30159 -> 1.2381 * mean) 2024-09-21T14:08:59.893 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T14:08:59.893 INFO:teuthology.orchestra.run.smithi049.stdout:oload 1000 2024-09-21T14:08:59.893 INFO:teuthology.orchestra.run.smithi049.stdout:max_change 3 2024-09-21T14:08:59.893 INFO:teuthology.orchestra.run.smithi049.stdout:max_change_osds 4 2024-09-21T14:08:59.893 INFO:teuthology.orchestra.run.smithi049.stdout:average_utilization 0.0322 2024-09-21T14:08:59.893 INFO:teuthology.orchestra.run.smithi049.stdout:overload_utilization 0.3220 2024-09-21T14:08:59.893 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4 weight 0.7889 -> 1.0000 2024-09-21T14:08:59.893 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0 weight 0.6177 -> 0.8247 2024-09-21T14:08:59.983 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:09:00.008 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:00.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:00.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:09:00.082 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:59 smithi187 ceph-mon[33032]: pgmap v1023: 168 pgs: 1 active+clean+scrubbing, 1 active+clean+remapped, 3 active+remapped+backfilling, 163 active+clean; 5.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 832 op/s; 7578/274185 objects misplaced (2.764%); 29 MiB/s, 457 objects/s recovering 2024-09-21T14:09:00.082 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:59 smithi187 ceph-mon[33032]: 3.1a scrub starts 2024-09-21T14:09:00.082 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:59 smithi187 ceph-mon[33032]: 3.1a scrub ok 2024-09-21T14:09:00.082 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:59 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:09:00.082 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:59 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:09:00.083 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:08:59 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'54047','4':'65536'}"} : dispatch 2024-09-21T14:09:00.110 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:09:00.117 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T14:09:00.135 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:00.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:00.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:09:00.237 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:09:00.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:59 smithi049 ceph-mon[40489]: pgmap v1023: 168 pgs: 1 active+clean+scrubbing, 1 active+clean+remapped, 3 active+remapped+backfilling, 163 active+clean; 5.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 832 op/s; 7578/274185 objects misplaced (2.764%); 29 MiB/s, 457 objects/s recovering 2024-09-21T14:09:00.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:59 smithi049 ceph-mon[40489]: 3.1a scrub starts 2024-09-21T14:09:00.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:59 smithi049 ceph-mon[40489]: 3.1a scrub ok 2024-09-21T14:09:00.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:59 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:09:00.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:59 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:09:00.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:08:59 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'54047','4':'65536'}"} : dispatch 2024-09-21T14:09:00.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:59 smithi049 ceph-mon[30080]: pgmap v1023: 168 pgs: 1 active+clean+scrubbing, 1 active+clean+remapped, 3 active+remapped+backfilling, 163 active+clean; 5.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 832 op/s; 7578/274185 objects misplaced (2.764%); 29 MiB/s, 457 objects/s recovering 2024-09-21T14:09:00.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:59 smithi049 ceph-mon[30080]: 3.1a scrub starts 2024-09-21T14:09:00.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:59 smithi049 ceph-mon[30080]: 3.1a scrub ok 2024-09-21T14:09:00.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:59 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:09:00.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:59 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:09:00.256 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:08:59 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'54047','4':'65536'}"} : dispatch 2024-09-21T14:09:00.273 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:00.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:00.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:09:00.375 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:00.400 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:00.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:00.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:00.401 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:09:00.401 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:09:00.420 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:09:00.420 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:09:00.432 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:09:00.437 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:09:00.438 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:09:00.444 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:09:00 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:09:00.456 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:09:00.457 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:09:00.478 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:09:00.479 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:09:00.498 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:09:00.498 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:09:00.502 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:09:00.518 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:09:00.518 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:09:00.526 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:00.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:00.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:09:00.537 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:09:00.537 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:09:00.629 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:09:00.654 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:00.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:00.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:09:00.680 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T14:09:00.755 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:09:00.781 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:00.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:00.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:09:00.883 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:09:00.893 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:09:00 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:09:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:09:00.909 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:00.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:00.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:09:01.012 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:09:01.035 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:01.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:01.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:09:01.137 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:01.158 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:01.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:01.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:01.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:00 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:09:01.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:00 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:09:01.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:00 smithi187 ceph-mon[33032]: from='client.26558 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T14:09:01.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:00 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'54047','4':'65536'}"}]': finished 2024-09-21T14:09:01.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:00 smithi187 ceph-mon[33032]: osdmap e510: 8 total, 8 up, 8 in 2024-09-21T14:09:01.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:00 smithi187 ceph-mon[33032]: 4.18 scrub starts 2024-09-21T14:09:01.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:00 smithi187 ceph-mon[33032]: 4.18 scrub ok 2024-09-21T14:09:01.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:00 smithi187 ceph-mon[33032]: osdmap e511: 8 total, 8 up, 8 in 2024-09-21T14:09:01.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:00 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/364342646' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:09:01.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:00 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:09:01.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:00 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:09:01.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:00 smithi049 ceph-mon[40489]: from='client.26558 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T14:09:01.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:00 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'54047','4':'65536'}"}]': finished 2024-09-21T14:09:01.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:00 smithi049 ceph-mon[40489]: osdmap e510: 8 total, 8 up, 8 in 2024-09-21T14:09:01.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:00 smithi049 ceph-mon[40489]: 4.18 scrub starts 2024-09-21T14:09:01.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:00 smithi049 ceph-mon[40489]: 4.18 scrub ok 2024-09-21T14:09:01.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:00 smithi049 ceph-mon[40489]: osdmap e511: 8 total, 8 up, 8 in 2024-09-21T14:09:01.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:00 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/364342646' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:09:01.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:00 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:09:01.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:00 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:09:01.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:00 smithi049 ceph-mon[30080]: from='client.26558 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T14:09:01.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:00 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'54047','4':'65536'}"}]': finished 2024-09-21T14:09:01.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:00 smithi049 ceph-mon[30080]: osdmap e510: 8 total, 8 up, 8 in 2024-09-21T14:09:01.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:00 smithi049 ceph-mon[30080]: 4.18 scrub starts 2024-09-21T14:09:01.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:00 smithi049 ceph-mon[30080]: 4.18 scrub ok 2024-09-21T14:09:01.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:00 smithi049 ceph-mon[30080]: osdmap e511: 8 total, 8 up, 8 in 2024-09-21T14:09:01.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:00 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/364342646' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:09:01.261 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:09:01.288 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:01.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:01.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:09:01.390 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:09:01.421 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:01.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:01.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:09:01.523 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:09:01.550 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:01.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:01.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:09:01.652 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:09:01.681 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:01.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:01.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:09:01.692 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T14:09:01.692 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-21T14:09:01.692 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.692 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26396.0:160737 11.8 11.e6ca85e8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e512)", 2024-09-21T14:09:01.692 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:09:01.519915+0000", 2024-09-21T14:09:01.692 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.17164969899999999, 2024-09-21T14:09:01.693 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.17167489899999999, 2024-09-21T14:09:01.693 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:09:01.693 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:09:01.693 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for peered", 2024-09-21T14:09:01.693 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:09:01.693 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26396", 2024-09-21T14:09:01.693 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:09:01.693 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 160737 2024-09-21T14:09:01.693 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.693 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:09:01.693 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.693 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:09:01.693 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.519915+0000", 2024-09-21T14:09:01.693 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:01.694 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.694 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.694 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:09:01.694 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.519915+0000", 2024-09-21T14:09:01.694 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:01.694 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.694 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.694 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:09:01.694 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.519917+0000", 2024-09-21T14:09:01.694 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1790000000000001e-06 2024-09-21T14:09:01.694 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.694 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.694 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:09:01.694 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.519921+0000", 2024-09-21T14:09:01.695 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1459999999999998e-06 2024-09-21T14:09:01.695 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.695 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.695 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:09:01.695 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.519930+0000", 2024-09-21T14:09:01.695 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3300000000000005e-06 2024-09-21T14:09:01.695 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.695 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.695 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:09:01.695 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.519938+0000", 2024-09-21T14:09:01.695 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6149999999999999e-06 2024-09-21T14:09:01.695 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.695 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.695 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:01.696 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.519988+0000", 2024-09-21T14:09:01.696 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.02e-05 2024-09-21T14:09:01.696 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.696 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.696 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:09:01.696 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.519994+0000", 2024-09-21T14:09:01.696 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9460000000000003e-06 2024-09-21T14:09:01.696 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:01.696 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:09:01.696 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:01.696 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.696 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.696 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26396.0:160632 11.8 11.953004c8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e512)", 2024-09-21T14:09:01.696 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:09:01.412828+0000", 2024-09-21T14:09:01.696 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.27873668699999998, 2024-09-21T14:09:01.697 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.27880651899999997, 2024-09-21T14:09:01.697 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:09:01.697 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:09:01.697 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for peered", 2024-09-21T14:09:01.697 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:09:01.697 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26396", 2024-09-21T14:09:01.697 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:09:01.697 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 160632 2024-09-21T14:09:01.697 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.697 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:09:01.697 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.697 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:09:01.697 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.412828+0000", 2024-09-21T14:09:01.697 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:01.698 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.698 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.698 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:09:01.698 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.412828+0000", 2024-09-21T14:09:01.698 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:01.698 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.698 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.698 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:09:01.698 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.412830+0000", 2024-09-21T14:09:01.698 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.201e-06 2024-09-21T14:09:01.698 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.698 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.698 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:09:01.698 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.412835+0000", 2024-09-21T14:09:01.698 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.989e-06 2024-09-21T14:09:01.699 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.699 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.699 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:09:01.699 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.412858+0000", 2024-09-21T14:09:01.699 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3513e-05 2024-09-21T14:09:01.699 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.699 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.699 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:09:01.699 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.412865+0000", 2024-09-21T14:09:01.699 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.272e-06 2024-09-21T14:09:01.699 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.699 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.699 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:01.699 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.412922+0000", 2024-09-21T14:09:01.700 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6910000000000002e-05 2024-09-21T14:09:01.700 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.700 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.700 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:09:01.700 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.412930+0000", 2024-09-21T14:09:01.700 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9729999999999997e-06 2024-09-21T14:09:01.700 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:01.700 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:09:01.700 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:01.700 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.700 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.700 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26396.0:160652 11.8 11.106cf4e8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e512)", 2024-09-21T14:09:01.700 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:09:01.426899+0000", 2024-09-21T14:09:01.700 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.264666136, 2024-09-21T14:09:01.700 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.26477593599999999, 2024-09-21T14:09:01.701 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:09:01.701 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:09:01.701 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for peered", 2024-09-21T14:09:01.701 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:09:01.701 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26396", 2024-09-21T14:09:01.701 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:09:01.701 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 160652 2024-09-21T14:09:01.701 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.701 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:09:01.701 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.701 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:09:01.701 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.426899+0000", 2024-09-21T14:09:01.701 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:01.701 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.702 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.702 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:09:01.702 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.426899+0000", 2024-09-21T14:09:01.702 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:01.702 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.702 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.702 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:09:01.702 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.426900+0000", 2024-09-21T14:09:01.702 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7000000000000003e-07 2024-09-21T14:09:01.702 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.702 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.702 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:09:01.702 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.426904+0000", 2024-09-21T14:09:01.703 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2980000000000002e-06 2024-09-21T14:09:01.703 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.703 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.703 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:09:01.703 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.426913+0000", 2024-09-21T14:09:01.703 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1400000000000006e-06 2024-09-21T14:09:01.703 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.703 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.703 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:09:01.703 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.426921+0000", 2024-09-21T14:09:01.703 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0409999999999998e-06 2024-09-21T14:09:01.703 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.703 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.703 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:01.703 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.426957+0000", 2024-09-21T14:09:01.704 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5989999999999999e-05 2024-09-21T14:09:01.704 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.704 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.704 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:09:01.704 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.426962+0000", 2024-09-21T14:09:01.704 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8770000000000001e-06 2024-09-21T14:09:01.704 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:01.704 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:09:01.704 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:01.704 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.704 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.704 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26396.0:160756 11.8 11.4dc6a838 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e512)", 2024-09-21T14:09:01.704 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:09:01.543967+0000", 2024-09-21T14:09:01.704 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.14759820300000001, 2024-09-21T14:09:01.705 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.14773797599999999, 2024-09-21T14:09:01.705 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:09:01.705 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:09:01.705 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for peered", 2024-09-21T14:09:01.705 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:09:01.705 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26396", 2024-09-21T14:09:01.705 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:09:01.705 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 160756 2024-09-21T14:09:01.705 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.705 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:09:01.705 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.705 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:09:01.705 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.543967+0000", 2024-09-21T14:09:01.705 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:01.705 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.706 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.706 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:09:01.706 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.543967+0000", 2024-09-21T14:09:01.706 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:01.706 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.706 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.707 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:09:01.707 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.543968+0000", 2024-09-21T14:09:01.707 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.542e-06 2024-09-21T14:09:01.707 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.707 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.707 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:09:01.707 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.543974+0000", 2024-09-21T14:09:01.707 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5350000000000004e-06 2024-09-21T14:09:01.707 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.707 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.707 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:09:01.707 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.543984+0000", 2024-09-21T14:09:01.707 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0394000000000001e-05 2024-09-21T14:09:01.708 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.708 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.708 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:09:01.708 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.543991+0000", 2024-09-21T14:09:01.708 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8580000000000002e-06 2024-09-21T14:09:01.708 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.708 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.708 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:01.708 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.544055+0000", 2024-09-21T14:09:01.708 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4048999999999996e-05 2024-09-21T14:09:01.708 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.708 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.708 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:09:01.708 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.544064+0000", 2024-09-21T14:09:01.709 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.918e-06 2024-09-21T14:09:01.709 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:01.709 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:09:01.709 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:01.709 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.709 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.709 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26396.0:160635 11.8 11.451b1638 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e512)", 2024-09-21T14:09:01.709 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:09:01.414558+0000", 2024-09-21T14:09:01.709 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.27700717600000002, 2024-09-21T14:09:01.709 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.27717756700000001, 2024-09-21T14:09:01.709 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:09:01.709 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:09:01.709 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for peered", 2024-09-21T14:09:01.709 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:09:01.710 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26396", 2024-09-21T14:09:01.710 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:09:01.710 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 160635 2024-09-21T14:09:01.710 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.710 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:09:01.710 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.710 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:09:01.710 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.414558+0000", 2024-09-21T14:09:01.710 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:01.710 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.710 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.710 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:09:01.710 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.414558+0000", 2024-09-21T14:09:01.710 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:01.711 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.711 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.711 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:09:01.711 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.414559+0000", 2024-09-21T14:09:01.711 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.237e-06 2024-09-21T14:09:01.711 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.711 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.711 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:09:01.711 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.414564+0000", 2024-09-21T14:09:01.711 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8720000000000001e-06 2024-09-21T14:09:01.711 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.711 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.711 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:09:01.711 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.414577+0000", 2024-09-21T14:09:01.712 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2826999999999999e-05 2024-09-21T14:09:01.712 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.712 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.712 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:09:01.712 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.414584+0000", 2024-09-21T14:09:01.712 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4889999999999998e-06 2024-09-21T14:09:01.712 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.712 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.712 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:01.712 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.414635+0000", 2024-09-21T14:09:01.712 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1088999999999998e-05 2024-09-21T14:09:01.712 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.712 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.712 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:09:01.713 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.414643+0000", 2024-09-21T14:09:01.713 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4749999999999996e-06 2024-09-21T14:09:01.713 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:01.713 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:09:01.713 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:01.713 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.713 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.713 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "MOSDPGPushReply(11.9 512/511 [PushReplyOp(11:904cdb2f:::benchmark_data_smithi049_206132_object44099:head)])", 2024-09-21T14:09:01.713 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:09:01.691601+0000", 2024-09-21T14:09:01.713 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 4294967295.9999642, 2024-09-21T14:09:01.713 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000165394, 2024-09-21T14:09:01.713 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:09:01.713 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:09:01.713 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-21T14:09:01.713 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:09:01.714 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.714 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:09:01.714 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.691601+0000", 2024-09-21T14:09:01.714 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:01.714 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.714 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.714 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:09:01.714 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.691601+0000", 2024-09-21T14:09:01.714 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:01.714 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.714 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.714 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:09:01.714 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.691601+0000", 2024-09-21T14:09:01.714 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1699999999999999e-07 2024-09-21T14:09:01.714 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.715 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.715 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:09:01.715 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.691604+0000", 2024-09-21T14:09:01.715 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.835e-06 2024-09-21T14:09:01.715 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.715 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.715 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:09:01.715 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.691611+0000", 2024-09-21T14:09:01.715 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2949999999999996e-06 2024-09-21T14:09:01.715 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.715 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.715 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:09:01.715 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.691618+0000", 2024-09-21T14:09:01.715 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.002e-06 2024-09-21T14:09:01.716 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.716 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.716 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:01.716 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.691645+0000", 2024-09-21T14:09:01.716 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.6650000000000001e-05 2024-09-21T14:09:01.716 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:01.716 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:09:01.716 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:01.716 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.716 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.716 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26396.0:160636 11.8 11.53af8618 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e512)", 2024-09-21T14:09:01.716 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:09:01.414628+0000", 2024-09-21T14:09:01.716 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.27693670199999998, 2024-09-21T14:09:01.716 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.27716281999999998, 2024-09-21T14:09:01.716 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:09:01.716 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:09:01.717 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for peered", 2024-09-21T14:09:01.717 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:09:01.717 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26396", 2024-09-21T14:09:01.717 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:09:01.717 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 160636 2024-09-21T14:09:01.717 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.717 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:09:01.717 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.717 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:09:01.717 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.414628+0000", 2024-09-21T14:09:01.717 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:01.717 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.717 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.717 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:09:01.717 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.414628+0000", 2024-09-21T14:09:01.717 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:01.718 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.718 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.718 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:09:01.718 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.414629+0000", 2024-09-21T14:09:01.718 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6199999999999997e-07 2024-09-21T14:09:01.718 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.718 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.718 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:09:01.718 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.414632+0000", 2024-09-21T14:09:01.718 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.728e-06 2024-09-21T14:09:01.718 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.718 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.718 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:09:01.718 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.414639+0000", 2024-09-21T14:09:01.719 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8600000000000004e-06 2024-09-21T14:09:01.719 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.719 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.719 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:09:01.719 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.414644+0000", 2024-09-21T14:09:01.719 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4049999999999999e-06 2024-09-21T14:09:01.719 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.719 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.719 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:01.719 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.414699+0000", 2024-09-21T14:09:01.719 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5458000000000003e-05 2024-09-21T14:09:01.719 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.719 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.719 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:09:01.719 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.414710+0000", 2024-09-21T14:09:01.719 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0900000000000001e-05 2024-09-21T14:09:01.720 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:01.720 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:09:01.720 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:01.720 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.720 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.720 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26396.0:160595 11.8 11.f9354758 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e512)", 2024-09-21T14:09:01.720 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:09:01.382695+0000", 2024-09-21T14:09:01.720 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.30886955599999999, 2024-09-21T14:09:01.720 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.30916181700000001, 2024-09-21T14:09:01.720 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:09:01.720 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:09:01.720 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for peered", 2024-09-21T14:09:01.720 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:09:01.720 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26396", 2024-09-21T14:09:01.721 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:09:01.721 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 160595 2024-09-21T14:09:01.721 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.721 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:09:01.721 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.721 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:09:01.721 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.382695+0000", 2024-09-21T14:09:01.721 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:01.721 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.721 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.721 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:09:01.721 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.382695+0000", 2024-09-21T14:09:01.721 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:01.721 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.722 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.722 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:09:01.722 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.382697+0000", 2024-09-21T14:09:01.722 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.249e-06 2024-09-21T14:09:01.722 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.722 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.722 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:09:01.722 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.382701+0000", 2024-09-21T14:09:01.722 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.087e-06 2024-09-21T14:09:01.722 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.722 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.722 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:09:01.722 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.382711+0000", 2024-09-21T14:09:01.723 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0548e-05 2024-09-21T14:09:01.723 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.723 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.723 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:09:01.723 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.382719+0000", 2024-09-21T14:09:01.723 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1890000000000001e-06 2024-09-21T14:09:01.723 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.723 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.723 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:01.723 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.382764+0000", 2024-09-21T14:09:01.723 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5136000000000003e-05 2024-09-21T14:09:01.723 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.723 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.723 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:09:01.723 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.382772+0000", 2024-09-21T14:09:01.724 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3950000000000001e-06 2024-09-21T14:09:01.724 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:01.724 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:09:01.724 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:01.724 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.724 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.724 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26396.0:160638 11.8 11.d6104348 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e512)", 2024-09-21T14:09:01.724 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:09:01.415896+0000", 2024-09-21T14:09:01.724 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.27566875499999999, 2024-09-21T14:09:01.724 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.276017756, 2024-09-21T14:09:01.724 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:09:01.724 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:09:01.724 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for peered", 2024-09-21T14:09:01.724 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:09:01.725 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26396", 2024-09-21T14:09:01.725 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:09:01.725 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 160638 2024-09-21T14:09:01.725 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.725 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:09:01.725 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.725 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:09:01.725 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.415896+0000", 2024-09-21T14:09:01.725 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:01.725 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.725 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.725 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:09:01.725 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.415896+0000", 2024-09-21T14:09:01.725 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:01.725 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.725 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.726 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:09:01.726 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.415897+0000", 2024-09-21T14:09:01.726 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.026e-06 2024-09-21T14:09:01.726 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.726 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.726 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:09:01.726 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.415906+0000", 2024-09-21T14:09:01.726 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5550000000000003e-06 2024-09-21T14:09:01.726 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.726 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.726 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:09:01.726 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.415919+0000", 2024-09-21T14:09:01.726 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2978e-05 2024-09-21T14:09:01.726 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.727 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.727 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:09:01.727 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.415926+0000", 2024-09-21T14:09:01.727 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3719999999999999e-06 2024-09-21T14:09:01.727 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.727 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.727 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:01.727 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.415975+0000", 2024-09-21T14:09:01.727 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8974999999999998e-05 2024-09-21T14:09:01.727 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.727 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.727 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:09:01.727 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.415982+0000", 2024-09-21T14:09:01.727 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5440000000000004e-06 2024-09-21T14:09:01.727 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:01.728 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:09:01.728 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:01.728 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.728 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.728 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26396.0:160798 11.8 11.343a9688 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e512)", 2024-09-21T14:09:01.728 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:09:01.631966+0000", 2024-09-21T14:09:01.728 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.059599196, 2024-09-21T14:09:01.728 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.059979180999999999, 2024-09-21T14:09:01.728 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:09:01.728 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:09:01.728 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for peered", 2024-09-21T14:09:01.728 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:09:01.728 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26396", 2024-09-21T14:09:01.728 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:09:01.729 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 160798 2024-09-21T14:09:01.729 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.729 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:09:01.729 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.729 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:09:01.729 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.631966+0000", 2024-09-21T14:09:01.729 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:01.729 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.729 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.729 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:09:01.729 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.631966+0000", 2024-09-21T14:09:01.729 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:01.729 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.729 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.729 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:09:01.730 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.631967+0000", 2024-09-21T14:09:01.730 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2869999999999999e-06 2024-09-21T14:09:01.730 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.730 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.730 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:09:01.730 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.631972+0000", 2024-09-21T14:09:01.730 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7500000000000003e-06 2024-09-21T14:09:01.730 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.730 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.730 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:09:01.730 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.631979+0000", 2024-09-21T14:09:01.730 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3529999999999997e-06 2024-09-21T14:09:01.730 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.730 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.731 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:09:01.731 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.631984+0000", 2024-09-21T14:09:01.731 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5990000000000004e-06 2024-09-21T14:09:01.731 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.731 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.731 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:01.731 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.632035+0000", 2024-09-21T14:09:01.731 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1156000000000003e-05 2024-09-21T14:09:01.731 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.731 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.731 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:09:01.731 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.632044+0000", 2024-09-21T14:09:01.731 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1430000000000004e-06 2024-09-21T14:09:01.731 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:01.731 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:09:01.731 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:01.732 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.732 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.732 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26396.0:160745 11.8 11.b7ace858 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e512)", 2024-09-21T14:09:01.732 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:09:01.527788+0000", 2024-09-21T14:09:01.732 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.16377735800000001, 2024-09-21T14:09:01.732 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.164189744, 2024-09-21T14:09:01.732 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:09:01.732 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:09:01.732 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for peered", 2024-09-21T14:09:01.732 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:09:01.732 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26396", 2024-09-21T14:09:01.732 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:09:01.732 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 160745 2024-09-21T14:09:01.732 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.732 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:09:01.733 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.733 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:09:01.733 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.527788+0000", 2024-09-21T14:09:01.733 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:01.733 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.733 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.733 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:09:01.733 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.527788+0000", 2024-09-21T14:09:01.733 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:01.733 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.733 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.733 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:09:01.733 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.527788+0000", 2024-09-21T14:09:01.733 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7100000000000001e-07 2024-09-21T14:09:01.733 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.734 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.734 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:09:01.734 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.527791+0000", 2024-09-21T14:09:01.734 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9399999999999998e-06 2024-09-21T14:09:01.734 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.734 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.734 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:09:01.734 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.527801+0000", 2024-09-21T14:09:01.734 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7280000000000007e-06 2024-09-21T14:09:01.734 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.734 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.734 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:09:01.734 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.527806+0000", 2024-09-21T14:09:01.734 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2599999999999996e-06 2024-09-21T14:09:01.735 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.735 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.735 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:01.735 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.527853+0000", 2024-09-21T14:09:01.735 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6582000000000002e-05 2024-09-21T14:09:01.735 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.735 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.735 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:09:01.735 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.527858+0000", 2024-09-21T14:09:01.735 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.356e-06 2024-09-21T14:09:01.735 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:01.735 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:09:01.735 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:01.736 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.736 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.736 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26396.0:160746 11.8 11.31a799f8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e512)", 2024-09-21T14:09:01.736 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:09:01.527876+0000", 2024-09-21T14:09:01.736 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.163689209, 2024-09-21T14:09:01.736 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.16414324199999999, 2024-09-21T14:09:01.736 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:09:01.736 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:09:01.736 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for peered", 2024-09-21T14:09:01.736 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:09:01.736 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26396", 2024-09-21T14:09:01.736 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:09:01.736 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 160746 2024-09-21T14:09:01.736 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.736 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:09:01.736 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.737 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:09:01.737 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.527876+0000", 2024-09-21T14:09:01.737 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:01.737 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.737 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.737 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:09:01.737 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.527876+0000", 2024-09-21T14:09:01.737 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:01.737 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.737 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.737 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:09:01.737 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.527877+0000", 2024-09-21T14:09:01.737 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.99e-07 2024-09-21T14:09:01.737 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.737 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.738 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:09:01.738 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.527880+0000", 2024-09-21T14:09:01.738 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.269e-06 2024-09-21T14:09:01.738 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.738 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.738 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:09:01.738 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.527888+0000", 2024-09-21T14:09:01.738 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7710000000000006e-06 2024-09-21T14:09:01.738 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.738 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.738 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:09:01.738 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.527893+0000", 2024-09-21T14:09:01.738 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2809999999999999e-06 2024-09-21T14:09:01.738 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.738 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.739 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:01.739 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.527916+0000", 2024-09-21T14:09:01.739 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2773999999999999e-05 2024-09-21T14:09:01.739 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.739 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.739 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:09:01.739 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.527919+0000", 2024-09-21T14:09:01.739 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3639999999999999e-06 2024-09-21T14:09:01.739 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:01.739 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:09:01.739 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:01.739 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.739 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.739 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26396.0:160558 11.8 11.64182808 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e512)", 2024-09-21T14:09:01.739 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:09:01.337475+0000", 2024-09-21T14:09:01.740 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.35409001299999998, 2024-09-21T14:09:01.740 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.35458759499999998, 2024-09-21T14:09:01.740 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:09:01.740 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:09:01.740 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for peered", 2024-09-21T14:09:01.740 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:09:01.740 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26396", 2024-09-21T14:09:01.740 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:09:01.740 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 160558 2024-09-21T14:09:01.740 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.740 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:09:01.740 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.740 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:09:01.740 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.337475+0000", 2024-09-21T14:09:01.741 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:01.741 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.741 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.741 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:09:01.741 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.337475+0000", 2024-09-21T14:09:01.741 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:01.741 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.741 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.741 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:09:01.741 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.337476+0000", 2024-09-21T14:09:01.741 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3699999999999999e-07 2024-09-21T14:09:01.741 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.741 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.741 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:09:01.741 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.337480+0000", 2024-09-21T14:09:01.742 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.76e-06 2024-09-21T14:09:01.742 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.742 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.742 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:09:01.742 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.337491+0000", 2024-09-21T14:09:01.742 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1007e-05 2024-09-21T14:09:01.742 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.742 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.742 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:09:01.742 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.337497+0000", 2024-09-21T14:09:01.742 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6900000000000003e-06 2024-09-21T14:09:01.742 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.742 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.742 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:01.743 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.337551+0000", 2024-09-21T14:09:01.743 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3653000000000003e-05 2024-09-21T14:09:01.743 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.743 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.743 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:09:01.743 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.337560+0000", 2024-09-21T14:09:01.743 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4519999999999997e-06 2024-09-21T14:09:01.743 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:01.743 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:09:01.743 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:01.743 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.743 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.743 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26396.0:160800 11.8 11.15300748 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e512)", 2024-09-21T14:09:01.743 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:09:01.636072+0000", 2024-09-21T14:09:01.744 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.055493204999999997, 2024-09-21T14:09:01.744 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.056024727000000003, 2024-09-21T14:09:01.744 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:09:01.744 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:09:01.744 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for peered", 2024-09-21T14:09:01.744 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:09:01.744 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26396", 2024-09-21T14:09:01.744 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:09:01.744 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 160800 2024-09-21T14:09:01.744 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.744 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:09:01.744 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.744 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:09:01.744 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.636072+0000", 2024-09-21T14:09:01.744 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:01.744 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.745 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.745 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:09:01.745 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.636072+0000", 2024-09-21T14:09:01.745 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:01.745 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.745 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.745 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:09:01.745 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.636073+0000", 2024-09-21T14:09:01.745 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.575e-06 2024-09-21T14:09:01.745 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.745 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.745 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:09:01.745 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.636079+0000", 2024-09-21T14:09:01.745 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0469999999999999e-06 2024-09-21T14:09:01.746 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.746 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.746 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:09:01.746 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.636094+0000", 2024-09-21T14:09:01.746 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4450000000000001e-05 2024-09-21T14:09:01.746 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.746 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.746 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:09:01.746 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.636101+0000", 2024-09-21T14:09:01.746 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5020000000000003e-06 2024-09-21T14:09:01.746 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.746 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.746 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:01.746 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.636154+0000", 2024-09-21T14:09:01.746 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2627000000000003e-05 2024-09-21T14:09:01.747 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.747 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.747 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:09:01.747 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.636160+0000", 2024-09-21T14:09:01.747 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3829999999999997e-06 2024-09-21T14:09:01.747 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:01.747 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:09:01.747 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:01.747 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.747 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.748 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26396.0:160641 11.8 11.4afe8998 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e512)", 2024-09-21T14:09:01.748 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:09:01.417920+0000", 2024-09-21T14:09:01.748 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.27364514600000001, 2024-09-21T14:09:01.748 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.27421687700000003, 2024-09-21T14:09:01.748 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:09:01.748 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:09:01.748 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for peered", 2024-09-21T14:09:01.748 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:09:01.748 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26396", 2024-09-21T14:09:01.748 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:09:01.748 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 160641 2024-09-21T14:09:01.748 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.749 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:09:01.749 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.749 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:09:01.749 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.417920+0000", 2024-09-21T14:09:01.749 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:01.749 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.749 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.749 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:09:01.749 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.417920+0000", 2024-09-21T14:09:01.749 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:01.749 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.749 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.749 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:09:01.749 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.417921+0000", 2024-09-21T14:09:01.750 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.02e-07 2024-09-21T14:09:01.750 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.750 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.750 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:09:01.750 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.417925+0000", 2024-09-21T14:09:01.750 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0110000000000002e-06 2024-09-21T14:09:01.750 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.750 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.750 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:09:01.750 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.417940+0000", 2024-09-21T14:09:01.750 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5617e-05 2024-09-21T14:09:01.750 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.750 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.750 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:09:01.751 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.417946+0000", 2024-09-21T14:09:01.751 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2939999999999996e-06 2024-09-21T14:09:01.751 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.751 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.751 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:01.751 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.418304+0000", 2024-09-21T14:09:01.751 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00035818999999999998 2024-09-21T14:09:01.751 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.751 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.751 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:09:01.751 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.418311+0000", 2024-09-21T14:09:01.751 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5490000000000001e-06 2024-09-21T14:09:01.751 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:01.751 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:09:01.752 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:01.752 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.752 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.752 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26396.0:160648 11.8 11.a9746d58 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e512)", 2024-09-21T14:09:01.752 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:09:01.423439+0000", 2024-09-21T14:09:01.752 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.268126321, 2024-09-21T14:09:01.752 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.268729145, 2024-09-21T14:09:01.752 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:09:01.752 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:09:01.752 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for peered", 2024-09-21T14:09:01.752 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:09:01.752 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26396", 2024-09-21T14:09:01.752 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:09:01.752 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 160648 2024-09-21T14:09:01.753 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.753 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:09:01.753 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.753 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:09:01.753 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.423439+0000", 2024-09-21T14:09:01.753 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:01.753 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.753 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.753 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:09:01.753 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.423439+0000", 2024-09-21T14:09:01.753 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:01.753 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.753 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.753 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:09:01.753 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.423440+0000", 2024-09-21T14:09:01.754 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3880000000000001e-06 2024-09-21T14:09:01.754 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.754 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.754 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:09:01.754 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.423444+0000", 2024-09-21T14:09:01.754 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0300000000000004e-06 2024-09-21T14:09:01.754 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.754 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.754 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:09:01.754 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.423454+0000", 2024-09-21T14:09:01.754 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3940000000000002e-06 2024-09-21T14:09:01.754 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.754 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.755 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:09:01.755 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.423459+0000", 2024-09-21T14:09:01.755 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8370000000000002e-06 2024-09-21T14:09:01.755 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.755 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.755 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:01.755 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.423512+0000", 2024-09-21T14:09:01.755 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3114e-05 2024-09-21T14:09:01.755 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.755 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.755 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:09:01.755 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.423521+0000", 2024-09-21T14:09:01.755 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.9940000000000006e-06 2024-09-21T14:09:01.755 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:01.755 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:09:01.756 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:01.756 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.756 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.756 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26396.0:160679 11.8 11.f1c2e6f8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e512)", 2024-09-21T14:09:01.756 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:09:01.455076+0000", 2024-09-21T14:09:01.756 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.236488796, 2024-09-21T14:09:01.756 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.23712871999999999, 2024-09-21T14:09:01.756 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:09:01.756 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:09:01.756 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for peered", 2024-09-21T14:09:01.756 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:09:01.756 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26396", 2024-09-21T14:09:01.756 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:09:01.756 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 160679 2024-09-21T14:09:01.757 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.757 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:09:01.757 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.757 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:09:01.757 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.455076+0000", 2024-09-21T14:09:01.757 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:01.757 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.757 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.757 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:09:01.757 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.455076+0000", 2024-09-21T14:09:01.757 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:01.757 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.757 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.757 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:09:01.757 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.455078+0000", 2024-09-21T14:09:01.758 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2759999999999999e-06 2024-09-21T14:09:01.758 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.758 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.758 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:09:01.758 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.455083+0000", 2024-09-21T14:09:01.758 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.006e-06 2024-09-21T14:09:01.758 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.758 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.758 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:09:01.758 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.455094+0000", 2024-09-21T14:09:01.758 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1946e-05 2024-09-21T14:09:01.758 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.758 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.758 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:09:01.759 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.455103+0000", 2024-09-21T14:09:01.759 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0579999999999998e-06 2024-09-21T14:09:01.759 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.759 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.759 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:01.759 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.455164+0000", 2024-09-21T14:09:01.759 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1723999999999994e-05 2024-09-21T14:09:01.759 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:01.759 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:01.759 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:09:01.759 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:01.455173+0000", 2024-09-21T14:09:01.759 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.9979999999999992e-06 2024-09-21T14:09:01.759 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:01.759 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:09:01.759 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:01.760 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:01.760 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2024-09-21T14:09:01.760 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 17 2024-09-21T14:09:01.760 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T14:09:01.783 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:09:01.807 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:01.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:01.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:09:01.909 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:09:01.933 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:01.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:01.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:09:02.035 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:09:02.066 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:02.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:02.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:09:02.099 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-21T14:09:02.099 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd reweight-by-utilization 1000 0.05 2024-09-21T14:09:02.168 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:02.169 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:01 smithi049 ceph-mon[40489]: pgmap v1025: 168 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfilling, 164 active+clean; 5.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 963 B/s rd, 0 B/s wr, 887 op/s; 5667/267249 objects misplaced (2.120%); 27 MiB/s, 436 objects/s recovering 2024-09-21T14:09:02.169 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:01 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:09:02.169 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:01 smithi049 ceph-mon[40489]: 4.8 scrub starts 2024-09-21T14:09:02.169 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:01 smithi049 ceph-mon[40489]: 4.8 scrub ok 2024-09-21T14:09:02.169 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:01 smithi049 ceph-mon[40489]: 1.11 scrub starts 2024-09-21T14:09:02.170 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:01 smithi049 ceph-mon[40489]: 1.11 scrub ok 2024-09-21T14:09:02.170 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:01 smithi049 ceph-mon[40489]: 11.9 starting backfill to osd.0 from (0'0,0'0] MAX to 509'10073 2024-09-21T14:09:02.170 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:01 smithi049 ceph-mon[40489]: osdmap e512: 8 total, 8 up, 8 in 2024-09-21T14:09:02.170 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:01 smithi049 ceph-mon[40489]: 3.7 scrub starts 2024-09-21T14:09:02.170 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:01 smithi049 ceph-mon[40489]: 3.7 scrub ok 2024-09-21T14:09:02.170 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:01 smithi049 ceph-mon[30080]: pgmap v1025: 168 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfilling, 164 active+clean; 5.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 963 B/s rd, 0 B/s wr, 887 op/s; 5667/267249 objects misplaced (2.120%); 27 MiB/s, 436 objects/s recovering 2024-09-21T14:09:02.170 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:01 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:09:02.170 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:01 smithi049 ceph-mon[30080]: 4.8 scrub starts 2024-09-21T14:09:02.170 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:01 smithi049 ceph-mon[30080]: 4.8 scrub ok 2024-09-21T14:09:02.170 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:01 smithi049 ceph-mon[30080]: 1.11 scrub starts 2024-09-21T14:09:02.170 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:01 smithi049 ceph-mon[30080]: 1.11 scrub ok 2024-09-21T14:09:02.171 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:01 smithi049 ceph-mon[30080]: 11.9 starting backfill to osd.0 from (0'0,0'0] MAX to 509'10073 2024-09-21T14:09:02.171 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:01 smithi049 ceph-mon[30080]: osdmap e512: 8 total, 8 up, 8 in 2024-09-21T14:09:02.171 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:01 smithi049 ceph-mon[30080]: 3.7 scrub starts 2024-09-21T14:09:02.171 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:01 smithi049 ceph-mon[30080]: 3.7 scrub ok 2024-09-21T14:09:02.177 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:09:02.194 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:02.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:01 smithi187 ceph-mon[33032]: pgmap v1025: 168 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfilling, 164 active+clean; 5.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 963 B/s rd, 0 B/s wr, 887 op/s; 5667/267249 objects misplaced (2.120%); 27 MiB/s, 436 objects/s recovering 2024-09-21T14:09:02.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:01 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:09:02.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:01 smithi187 ceph-mon[33032]: 4.8 scrub starts 2024-09-21T14:09:02.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:01 smithi187 ceph-mon[33032]: 4.8 scrub ok 2024-09-21T14:09:02.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:01 smithi187 ceph-mon[33032]: 1.11 scrub starts 2024-09-21T14:09:02.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:01 smithi187 ceph-mon[33032]: 1.11 scrub ok 2024-09-21T14:09:02.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:01 smithi187 ceph-mon[33032]: 11.9 starting backfill to osd.0 from (0'0,0'0] MAX to 509'10073 2024-09-21T14:09:02.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:01 smithi187 ceph-mon[33032]: osdmap e512: 8 total, 8 up, 8 in 2024-09-21T14:09:02.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:01 smithi187 ceph-mon[33032]: 3.7 scrub starts 2024-09-21T14:09:02.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:01 smithi187 ceph-mon[33032]: 3.7 scrub ok 2024-09-21T14:09:02.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:02.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:02.296 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:02.326 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:02.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:02.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:02.428 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:09:02.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:02.458 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:02.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:09:02.545 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T14:09:02.560 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:09:02.608 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:02.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:02.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:09:02.710 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:09:02.752 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:02.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:02.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:09:02.854 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:02.899 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:02.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:02.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:02.908 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:02 smithi187 ceph-mon[33032]: 11.e starting backfill to osd.4 from (0'0,0'0] MAX to 509'10146 2024-09-21T14:09:02.909 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:02 smithi187 ceph-mon[33032]: 11.6 starting backfill to osd.0 from (0'0,0'0] MAX to 509'10101 2024-09-21T14:09:02.909 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:02 smithi187 ceph-mon[33032]: osdmap e513: 8 total, 8 up, 8 in 2024-09-21T14:09:02.909 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:02 smithi187 ceph-mon[33032]: 3.15 scrub starts 2024-09-21T14:09:02.909 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:02 smithi187 ceph-mon[33032]: 3.15 scrub ok 2024-09-21T14:09:02.909 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:09:02.918 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:09:03.001 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:09:03.049 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:03.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:03.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:09:03.095 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:02 smithi049 ceph-mon[40489]: 11.e starting backfill to osd.4 from (0'0,0'0] MAX to 509'10146 2024-09-21T14:09:03.095 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:02 smithi049 ceph-mon[40489]: 11.6 starting backfill to osd.0 from (0'0,0'0] MAX to 509'10101 2024-09-21T14:09:03.095 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:02 smithi049 ceph-mon[40489]: osdmap e513: 8 total, 8 up, 8 in 2024-09-21T14:09:03.095 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:02 smithi049 ceph-mon[40489]: 3.15 scrub starts 2024-09-21T14:09:03.095 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:02 smithi049 ceph-mon[40489]: 3.15 scrub ok 2024-09-21T14:09:03.096 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:02 smithi049 ceph-mon[30080]: 11.e starting backfill to osd.4 from (0'0,0'0] MAX to 509'10146 2024-09-21T14:09:03.096 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:02 smithi049 ceph-mon[30080]: 11.6 starting backfill to osd.0 from (0'0,0'0] MAX to 509'10101 2024-09-21T14:09:03.096 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:02 smithi049 ceph-mon[30080]: osdmap e513: 8 total, 8 up, 8 in 2024-09-21T14:09:03.096 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:02 smithi049 ceph-mon[30080]: 3.15 scrub starts 2024-09-21T14:09:03.096 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:02 smithi049 ceph-mon[30080]: 3.15 scrub ok 2024-09-21T14:09:03.151 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:09:03.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:03.199 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:03.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:09:03.300 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:09:03.341 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:03.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:03.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:09:03.443 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:09:03.483 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:03.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:03.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:09:03.494 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T14:09:03.494 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-21T14:09:03.494 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-21T14:09:03.494 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-21T14:09:03.494 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T14:09:03.561 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T14:09:03.585 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:03.611 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:03.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:03.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:03.713 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:03.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:03.770 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:03.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:03.871 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:09:03.918 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-09-21T14:09:03.919 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:03.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:03.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:09:04.021 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:09:04.060 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:04.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:04.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:09:04.162 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:09:04.165 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:03 smithi187 ceph-mon[33032]: pgmap v1028: 168 pgs: 5 peering, 3 active+remapped+backfilling, 160 active+clean; 5.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 966 B/s rd, 0 B/s wr, 787 op/s; 4571/261681 objects misplaced (1.747%); 32 MiB/s, 515 objects/s recovering 2024-09-21T14:09:04.165 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:03 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3369730859' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:09:04.165 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:03 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:09:04.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:04.213 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:04.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:09:04.218 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T14:09:04.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:03 smithi049 ceph-mon[40489]: pgmap v1028: 168 pgs: 5 peering, 3 active+remapped+backfilling, 160 active+clean; 5.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 966 B/s rd, 0 B/s wr, 787 op/s; 4571/261681 objects misplaced (1.747%); 32 MiB/s, 515 objects/s recovering 2024-09-21T14:09:04.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:03 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3369730859' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:09:04.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:03 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:09:04.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:03 smithi049 ceph-mon[30080]: pgmap v1028: 168 pgs: 5 peering, 3 active+remapped+backfilling, 160 active+clean; 5.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 966 B/s rd, 0 B/s wr, 787 op/s; 4571/261681 objects misplaced (1.747%); 32 MiB/s, 515 objects/s recovering 2024-09-21T14:09:04.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:03 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3369730859' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:09:04.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:03 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:09:04.313 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:09:04.351 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:04.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:04.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:09:04.453 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:09:04.496 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:04.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:04.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:09:04.598 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:09:04.633 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:04.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:04.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:09:04.666 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:09:04.734 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:09:04.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:04.771 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:04.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:09:04.872 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:09:04.919 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:09:04.922 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:04.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:04.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:09:04.946 INFO:teuthology.orchestra.run.smithi049.stdout:moved 2 / 504 (0.396825%) 2024-09-21T14:09:04.946 INFO:teuthology.orchestra.run.smithi049.stdout:avg 63 2024-09-21T14:09:04.946 INFO:teuthology.orchestra.run.smithi049.stdout:stddev 16.7631 -> 16.5151 (expected baseline 7.42462) 2024-09-21T14:09:04.946 INFO:teuthology.orchestra.run.smithi049.stdout:min osd.3 with 24 -> 24 pgs (0.380952 -> 0.380952 * mean) 2024-09-21T14:09:04.946 INFO:teuthology.orchestra.run.smithi049.stdout:max osd.1 with 78 -> 78 pgs (1.2381 -> 1.2381 * mean) 2024-09-21T14:09:04.946 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T14:09:04.946 INFO:teuthology.orchestra.run.smithi049.stdout:oload 1000 2024-09-21T14:09:04.946 INFO:teuthology.orchestra.run.smithi049.stdout:max_change 0.05 2024-09-21T14:09:04.947 INFO:teuthology.orchestra.run.smithi049.stdout:max_change_osds 4 2024-09-21T14:09:04.947 INFO:teuthology.orchestra.run.smithi049.stdout:average_utilization 0.0313 2024-09-21T14:09:04.947 INFO:teuthology.orchestra.run.smithi049.stdout:overload_utilization 0.3134 2024-09-21T14:09:04.947 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0 weight 0.8247 -> 0.8747 2024-09-21T14:09:05.025 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:09:05.057 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:05.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:05.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:09:05.136 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:04 smithi187 ceph-mon[33032]: 5.7 scrub starts 2024-09-21T14:09:05.137 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:04 smithi187 ceph-mon[33032]: 5.7 scrub ok 2024-09-21T14:09:05.137 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:04 smithi187 ceph-mon[33032]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:09:05.137 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:04 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T14:09:05.137 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:04 smithi187 ceph-mon[33032]: osdmap e514: 8 total, 8 up, 8 in 2024-09-21T14:09:05.137 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:04 smithi187 ceph-mon[33032]: from='client.16689 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T14:09:05.137 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:04 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'57323'}"} : dispatch 2024-09-21T14:09:05.159 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:05.199 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:05.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:05.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:05.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:04 smithi049 ceph-mon[30080]: 5.7 scrub starts 2024-09-21T14:09:05.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:04 smithi049 ceph-mon[30080]: 5.7 scrub ok 2024-09-21T14:09:05.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:04 smithi049 ceph-mon[30080]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:09:05.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:04 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T14:09:05.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:04 smithi049 ceph-mon[30080]: osdmap e514: 8 total, 8 up, 8 in 2024-09-21T14:09:05.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:04 smithi049 ceph-mon[30080]: from='client.16689 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T14:09:05.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:04 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'57323'}"} : dispatch 2024-09-21T14:09:05.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:04 smithi049 ceph-mon[40489]: 5.7 scrub starts 2024-09-21T14:09:05.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:04 smithi049 ceph-mon[40489]: 5.7 scrub ok 2024-09-21T14:09:05.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:04 smithi049 ceph-mon[40489]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:09:05.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:04 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T14:09:05.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:04 smithi049 ceph-mon[40489]: osdmap e514: 8 total, 8 up, 8 in 2024-09-21T14:09:05.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:04 smithi049 ceph-mon[40489]: from='client.16689 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T14:09:05.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:04 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'57323'}"} : dispatch 2024-09-21T14:09:05.296 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T14:09:05.297 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-21T14:09:05.297 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-21T14:09:05.297 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-21T14:09:05.297 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.297 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:112204 10.a 10:5c740623:::benchmark_data_smithi049_176811_object112203:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-21T14:09:05.297 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:04:20.206510+0000", 2024-09-21T14:09:05.297 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 285.08734674099998, 2024-09-21T14:09:05.297 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1818630240000001, 2024-09-21T14:09:05.297 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:09:05.297 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:09:05.297 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:09:05.298 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:09:05.298 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:09:05.298 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:09:05.298 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 112204 2024-09-21T14:09:05.298 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.298 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:09:05.298 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.298 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:09:05.298 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.206510+0000", 2024-09-21T14:09:05.298 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:05.298 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.298 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.298 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:09:05.299 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.206510+0000", 2024-09-21T14:09:05.299 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:05.299 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.299 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.299 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:09:05.299 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.206511+0000", 2024-09-21T14:09:05.299 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0900000000000001e-07 2024-09-21T14:09:05.299 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.299 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.299 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:09:05.299 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.206541+0000", 2024-09-21T14:09:05.299 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0082000000000001e-05 2024-09-21T14:09:05.299 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.299 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.300 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:09:05.300 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.206548+0000", 2024-09-21T14:09:05.300 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.0110000000000003e-06 2024-09-21T14:09:05.300 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.300 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.300 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:09:05.300 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.206553+0000", 2024-09-21T14:09:05.300 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1529999999999996e-06 2024-09-21T14:09:05.300 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.300 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.300 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.300 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.206601+0000", 2024-09-21T14:09:05.300 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8393000000000002e-05 2024-09-21T14:09:05.300 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.301 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.301 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:09:05.301 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.206610+0000", 2024-09-21T14:09:05.301 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.507e-06 2024-09-21T14:09:05.301 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.301 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.301 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.301 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.188308+0000", 2024-09-21T14:09:05.301 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.98169809900000005 2024-09-21T14:09:05.301 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.301 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.301 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:09:05.301 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.188318+0000", 2024-09-21T14:09:05.301 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0179e-05 2024-09-21T14:09:05.301 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.302 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.302 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.302 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.370141+0000", 2024-09-21T14:09:05.302 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1818224210000001 2024-09-21T14:09:05.302 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.302 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.302 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:09:05.302 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.370277+0000", 2024-09-21T14:09:05.302 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013640399999999999 2024-09-21T14:09:05.302 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.302 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.302 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2,5", 2024-09-21T14:09:05.302 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.370711+0000", 2024-09-21T14:09:05.302 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000433826 2024-09-21T14:09:05.303 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.303 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.303 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:05.303 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.377442+0000", 2024-09-21T14:09:05.303 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0067311239999999998 2024-09-21T14:09:05.303 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.303 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.303 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:09:05.303 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.380659+0000", 2024-09-21T14:09:05.303 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0032168869999999999 2024-09-21T14:09:05.303 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.303 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.303 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:05.304 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.388142+0000", 2024-09-21T14:09:05.304 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0074832570000000001 2024-09-21T14:09:05.304 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.304 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.304 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:05.304 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.388212+0000", 2024-09-21T14:09:05.304 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9850000000000004e-05 2024-09-21T14:09:05.304 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.304 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.304 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:09:05.304 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.388269+0000", 2024-09-21T14:09:05.304 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7224e-05 2024-09-21T14:09:05.304 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.305 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.305 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:09:05.305 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.388373+0000", 2024-09-21T14:09:05.305 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000103998 2024-09-21T14:09:05.305 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:05.305 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:09:05.305 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:05.305 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.305 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.305 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:112222 10.d 10:bb42e81a:::benchmark_data_smithi049_176811_object112221:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-21T14:09:05.305 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:04:20.235680+0000", 2024-09-21T14:09:05.305 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 285.05817728, 2024-09-21T14:09:05.305 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0718219699999998, 2024-09-21T14:09:05.305 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:09:05.305 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:09:05.306 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:09:05.306 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:09:05.306 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:09:05.306 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:09:05.306 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 112222 2024-09-21T14:09:05.306 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.306 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:09:05.306 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.306 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:09:05.306 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235680+0000", 2024-09-21T14:09:05.306 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:05.306 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.306 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.307 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:09:05.307 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235680+0000", 2024-09-21T14:09:05.307 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:05.307 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.307 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.307 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:09:05.307 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235681+0000", 2024-09-21T14:09:05.307 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.285e-06 2024-09-21T14:09:05.307 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.307 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.307 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:09:05.307 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235734+0000", 2024-09-21T14:09:05.307 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2580000000000001e-05 2024-09-21T14:09:05.307 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.307 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.308 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:09:05.308 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235744+0000", 2024-09-21T14:09:05.308 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0197000000000001e-05 2024-09-21T14:09:05.308 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.308 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.308 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:09:05.308 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235749+0000", 2024-09-21T14:09:05.308 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3809999999999998e-06 2024-09-21T14:09:05.308 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.308 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.308 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.308 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235804+0000", 2024-09-21T14:09:05.308 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4592999999999997e-05 2024-09-21T14:09:05.309 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.309 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.309 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:09:05.309 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235814+0000", 2024-09-21T14:09:05.309 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0545e-05 2024-09-21T14:09:05.309 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.309 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.309 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.309 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.187322+0000", 2024-09-21T14:09:05.309 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.95150779399999996 2024-09-21T14:09:05.309 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.309 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.309 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:09:05.309 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.187331+0000", 2024-09-21T14:09:05.309 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.6929999999999996e-06 2024-09-21T14:09:05.310 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.310 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.310 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.310 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.285689+0000", 2024-09-21T14:09:05.310 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.098357961 2024-09-21T14:09:05.310 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.310 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.310 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:09:05.310 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.285795+0000", 2024-09-21T14:09:05.310 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:09:05.311 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010604599999999999 2024-09-21T14:09:05.311 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.311 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.312 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3,4", 2024-09-21T14:09:05.312 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.286188+0000", 2024-09-21T14:09:05.312 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00039307900000000001 2024-09-21T14:09:05.312 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.312 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.312 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:05.312 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.297975+0000", 2024-09-21T14:09:05.312 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.011787446 2024-09-21T14:09:05.312 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.312 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.312 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:05.312 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.298048+0000", 2024-09-21T14:09:05.312 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3146999999999996e-05 2024-09-21T14:09:05.313 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.313 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.313 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:09:05.313 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.298141+0000", 2024-09-21T14:09:05.313 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.2391000000000004e-05 2024-09-21T14:09:05.313 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.313 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.313 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:05.313 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.307285+0000", 2024-09-21T14:09:05.313 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0091442880000000004 2024-09-21T14:09:05.313 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.313 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.313 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:09:05.313 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.307366+0000", 2024-09-21T14:09:05.314 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1155999999999993e-05 2024-09-21T14:09:05.314 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.314 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.314 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:09:05.314 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.307502+0000", 2024-09-21T14:09:05.314 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013538800000000001 2024-09-21T14:09:05.314 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:05.314 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:09:05.314 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:05.314 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.314 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.314 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:112223 10.0 10:05563d3b:::benchmark_data_smithi049_176811_object112222:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-21T14:09:05.314 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:04:20.235762+0000", 2024-09-21T14:09:05.314 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 285.05809540799999, 2024-09-21T14:09:05.315 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1264867669999998, 2024-09-21T14:09:05.315 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:09:05.315 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:09:05.315 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:09:05.315 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:09:05.315 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:09:05.315 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:09:05.315 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 112223 2024-09-21T14:09:05.315 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.315 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:09:05.315 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.315 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:09:05.315 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235762+0000", 2024-09-21T14:09:05.316 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:05.316 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.316 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.316 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:09:05.316 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235762+0000", 2024-09-21T14:09:05.316 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:05.316 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.316 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.316 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:09:05.316 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235762+0000", 2024-09-21T14:09:05.316 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2799999999999996e-07 2024-09-21T14:09:05.316 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.316 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.316 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:09:05.317 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235796+0000", 2024-09-21T14:09:05.317 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3402000000000003e-05 2024-09-21T14:09:05.317 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.317 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.317 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:09:05.317 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235803+0000", 2024-09-21T14:09:05.317 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9929999999999998e-06 2024-09-21T14:09:05.317 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.317 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.317 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:09:05.317 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235808+0000", 2024-09-21T14:09:05.317 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9699999999999998e-06 2024-09-21T14:09:05.317 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.317 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.317 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.318 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235863+0000", 2024-09-21T14:09:05.318 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5877000000000001e-05 2024-09-21T14:09:05.318 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.318 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.318 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:09:05.318 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235870+0000", 2024-09-21T14:09:05.318 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.007e-06 2024-09-21T14:09:05.318 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.318 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.318 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.318 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.186804+0000", 2024-09-21T14:09:05.319 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.95093328600000004 2024-09-21T14:09:05.319 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.319 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.319 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:09:05.319 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.186815+0000", 2024-09-21T14:09:05.319 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1265e-05 2024-09-21T14:09:05.319 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.319 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.319 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.319 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.336944+0000", 2024-09-21T14:09:05.319 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1501292430000001 2024-09-21T14:09:05.319 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.319 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.319 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:09:05.320 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.337058+0000", 2024-09-21T14:09:05.320 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000113881 2024-09-21T14:09:05.320 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.320 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.320 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,4,7", 2024-09-21T14:09:05.320 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.337547+0000", 2024-09-21T14:09:05.320 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00048919999999999996 2024-09-21T14:09:05.320 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.320 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.320 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:05.320 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.346097+0000", 2024-09-21T14:09:05.320 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0085500969999999996 2024-09-21T14:09:05.320 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.320 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.321 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:05.321 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.346145+0000", 2024-09-21T14:09:05.321 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7700000000000001e-05 2024-09-21T14:09:05.321 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.321 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.321 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:09:05.321 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.362046+0000", 2024-09-21T14:09:05.321 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.015900935000000001 2024-09-21T14:09:05.321 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.321 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.321 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:05.321 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.362138+0000", 2024-09-21T14:09:05.321 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1525000000000002e-05 2024-09-21T14:09:05.321 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.321 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.322 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:09:05.322 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.362175+0000", 2024-09-21T14:09:05.322 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6928999999999998e-05 2024-09-21T14:09:05.322 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.322 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.322 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:09:05.322 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.362248+0000", 2024-09-21T14:09:05.322 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3829000000000001e-05 2024-09-21T14:09:05.322 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:05.322 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:09:05.322 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:05.322 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.322 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.322 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:112226 10.0 10:0f5734d7:::benchmark_data_smithi049_176811_object112225:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-21T14:09:05.323 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:04:20.235905+0000", 2024-09-21T14:09:05.323 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 285.05795193799997, 2024-09-21T14:09:05.323 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1266882329999999, 2024-09-21T14:09:05.323 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:09:05.323 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:09:05.323 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:09:05.323 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:09:05.323 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:09:05.323 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:09:05.323 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 112226 2024-09-21T14:09:05.323 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.323 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:09:05.323 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.323 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:09:05.323 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235905+0000", 2024-09-21T14:09:05.324 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:05.324 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.324 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.324 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:09:05.324 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235905+0000", 2024-09-21T14:09:05.324 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:05.324 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.324 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.324 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:09:05.324 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235906+0000", 2024-09-21T14:09:05.324 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4e-07 2024-09-21T14:09:05.324 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.324 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.324 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:09:05.324 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235975+0000", 2024-09-21T14:09:05.324 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9052000000000002e-05 2024-09-21T14:09:05.325 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.325 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.325 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:09:05.325 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235985+0000", 2024-09-21T14:09:05.325 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.9690000000000006e-06 2024-09-21T14:09:05.325 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.325 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.325 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:09:05.325 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235991+0000", 2024-09-21T14:09:05.325 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9939999999999997e-06 2024-09-21T14:09:05.325 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.325 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.325 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.325 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.236048+0000", 2024-09-21T14:09:05.326 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6943e-05 2024-09-21T14:09:05.326 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.326 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.326 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:09:05.326 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.236057+0000", 2024-09-21T14:09:05.326 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.4790000000000002e-06 2024-09-21T14:09:05.326 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.326 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.326 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.326 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.187660+0000", 2024-09-21T14:09:05.326 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.95160278700000001 2024-09-21T14:09:05.326 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.326 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.326 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:09:05.327 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.187669+0000", 2024-09-21T14:09:05.327 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.2979999999999997e-06 2024-09-21T14:09:05.327 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.327 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.327 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.327 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.346204+0000", 2024-09-21T14:09:05.327 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1585341869999999 2024-09-21T14:09:05.327 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.327 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.327 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:09:05.327 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.346382+0000", 2024-09-21T14:09:05.327 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00017858500000000001 2024-09-21T14:09:05.327 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.327 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.327 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,4,7", 2024-09-21T14:09:05.328 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.346555+0000", 2024-09-21T14:09:05.328 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0001728 2024-09-21T14:09:05.328 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.328 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.328 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:05.328 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.353773+0000", 2024-09-21T14:09:05.328 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0072175240000000003 2024-09-21T14:09:05.328 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.328 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.328 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:05.328 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.362086+0000", 2024-09-21T14:09:05.328 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0083139189999999995 2024-09-21T14:09:05.329 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.329 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.329 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:09:05.329 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.362257+0000", 2024-09-21T14:09:05.329 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00017090200000000001 2024-09-21T14:09:05.329 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.329 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.329 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:05.329 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.362405+0000", 2024-09-21T14:09:05.329 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014748800000000001 2024-09-21T14:09:05.329 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.329 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.329 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:09:05.329 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.362477+0000", 2024-09-21T14:09:05.329 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2214999999999999e-05 2024-09-21T14:09:05.330 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.330 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.330 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:09:05.330 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.362593+0000", 2024-09-21T14:09:05.330 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000116251 2024-09-21T14:09:05.330 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:05.330 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:09:05.330 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:05.330 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.330 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.330 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:112230 10.d 10:b278ca69:::benchmark_data_smithi049_176811_object112229:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-21T14:09:05.330 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:04:20.239727+0000", 2024-09-21T14:09:05.330 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 285.05412969100001, 2024-09-21T14:09:05.330 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0797783679999999, 2024-09-21T14:09:05.331 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:09:05.331 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:09:05.331 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:09:05.331 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:09:05.331 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:09:05.331 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:09:05.331 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 112230 2024-09-21T14:09:05.331 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.331 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:09:05.331 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.331 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:09:05.331 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.239727+0000", 2024-09-21T14:09:05.331 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:05.331 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.332 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.332 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:09:05.332 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.239727+0000", 2024-09-21T14:09:05.332 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:05.332 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.332 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.332 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:09:05.332 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.239728+0000", 2024-09-21T14:09:05.332 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8199999999999999e-07 2024-09-21T14:09:05.332 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.332 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.332 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:09:05.332 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.239763+0000", 2024-09-21T14:09:05.332 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4799999999999999e-05 2024-09-21T14:09:05.332 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.333 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.333 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:09:05.333 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.239772+0000", 2024-09-21T14:09:05.333 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1099999999999992e-06 2024-09-21T14:09:05.333 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.333 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.333 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:09:05.333 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.239778+0000", 2024-09-21T14:09:05.333 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0920000000000003e-06 2024-09-21T14:09:05.333 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.333 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.333 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.333 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.239835+0000", 2024-09-21T14:09:05.333 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6758000000000001e-05 2024-09-21T14:09:05.333 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.334 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.334 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:09:05.334 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.239861+0000", 2024-09-21T14:09:05.334 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.6290000000000001e-05 2024-09-21T14:09:05.334 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.334 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.334 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.334 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.187397+0000", 2024-09-21T14:09:05.334 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.94753606300000004 2024-09-21T14:09:05.334 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.334 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.334 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:09:05.334 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.187407+0000", 2024-09-21T14:09:05.334 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6600000000000007e-06 2024-09-21T14:09:05.335 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.335 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.335 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.335 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.290801+0000", 2024-09-21T14:09:05.335 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1033946459999999 2024-09-21T14:09:05.335 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.335 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.335 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:09:05.335 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.290936+0000", 2024-09-21T14:09:05.335 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013488899999999999 2024-09-21T14:09:05.335 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.335 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.335 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3,4", 2024-09-21T14:09:05.336 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.291093+0000", 2024-09-21T14:09:05.336 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015686300000000001 2024-09-21T14:09:05.336 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.336 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.336 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:05.336 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.298120+0000", 2024-09-21T14:09:05.336 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0070268209999999999 2024-09-21T14:09:05.336 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.336 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.336 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:09:05.336 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.298149+0000", 2024-09-21T14:09:05.336 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8609999999999999e-05 2024-09-21T14:09:05.336 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.336 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.336 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:05.337 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.298207+0000", 2024-09-21T14:09:05.337 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7859000000000003e-05 2024-09-21T14:09:05.337 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.337 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.337 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:05.337 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.319309+0000", 2024-09-21T14:09:05.337 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.021102584000000001 2024-09-21T14:09:05.337 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.337 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.337 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:09:05.337 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.319379+0000", 2024-09-21T14:09:05.337 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9858999999999996e-05 2024-09-21T14:09:05.337 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.337 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.338 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:09:05.338 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.319506+0000", 2024-09-21T14:09:05.338 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000126682 2024-09-21T14:09:05.338 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:05.338 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:09:05.338 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:05.338 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.338 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.338 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:112234 10.a 10:5cc5be6b:::benchmark_data_smithi049_176811_object112233:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-21T14:09:05.338 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:04:20.240298+0000", 2024-09-21T14:09:05.339 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 285.05355921400002, 2024-09-21T14:09:05.339 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.148374665, 2024-09-21T14:09:05.339 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:09:05.339 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:09:05.339 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:09:05.339 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:09:05.339 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:09:05.339 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:09:05.339 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 112234 2024-09-21T14:09:05.339 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.339 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:09:05.339 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.339 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:09:05.339 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.240298+0000", 2024-09-21T14:09:05.339 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:05.340 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.340 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.340 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:09:05.340 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.240298+0000", 2024-09-21T14:09:05.340 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:05.340 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.340 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.340 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:09:05.340 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.240299+0000", 2024-09-21T14:09:05.340 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4199999999999995e-07 2024-09-21T14:09:05.340 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.340 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.340 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:09:05.340 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.240385+0000", 2024-09-21T14:09:05.340 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.6550000000000003e-05 2024-09-21T14:09:05.341 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.341 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.341 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:09:05.341 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.240395+0000", 2024-09-21T14:09:05.341 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.8579999999999995e-06 2024-09-21T14:09:05.341 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.341 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.341 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:09:05.341 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.240400+0000", 2024-09-21T14:09:05.341 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3040000000000004e-06 2024-09-21T14:09:05.341 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.341 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.341 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.341 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.240444+0000", 2024-09-21T14:09:05.341 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3594000000000003e-05 2024-09-21T14:09:05.342 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.342 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.342 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:09:05.342 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.240451+0000", 2024-09-21T14:09:05.342 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.0319999999999998e-06 2024-09-21T14:09:05.342 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.342 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.342 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.342 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.188345+0000", 2024-09-21T14:09:05.342 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.94789378400000002 2024-09-21T14:09:05.342 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.342 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.342 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:09:05.342 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.188351+0000", 2024-09-21T14:09:05.343 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7619999999999998e-06 2024-09-21T14:09:05.343 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.343 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.343 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.343 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.377491+0000", 2024-09-21T14:09:05.343 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.189139642 2024-09-21T14:09:05.343 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.343 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.343 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:09:05.343 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.377612+0000", 2024-09-21T14:09:05.343 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000120528 2024-09-21T14:09:05.343 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.343 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.343 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2,5", 2024-09-21T14:09:05.344 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.377713+0000", 2024-09-21T14:09:05.344 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010192 2024-09-21T14:09:05.344 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.344 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.344 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:05.344 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.388079+0000", 2024-09-21T14:09:05.344 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.010365298 2024-09-21T14:09:05.344 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.345 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.345 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:09:05.345 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.388158+0000", 2024-09-21T14:09:05.345 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8726000000000005e-05 2024-09-21T14:09:05.345 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.345 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.345 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:05.345 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.388482+0000", 2024-09-21T14:09:05.345 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00032423 2024-09-21T14:09:05.345 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.345 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.345 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:05.345 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.388524+0000", 2024-09-21T14:09:05.346 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1782000000000001e-05 2024-09-21T14:09:05.346 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.346 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.346 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:09:05.346 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.388577+0000", 2024-09-21T14:09:05.346 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3965999999999998e-05 2024-09-21T14:09:05.346 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.346 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.346 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:09:05.346 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.388672+0000", 2024-09-21T14:09:05.346 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.4946999999999997e-05 2024-09-21T14:09:05.346 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:05.346 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:09:05.347 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:05.347 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.347 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.347 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:112240 10.a 10:59634543:::benchmark_data_smithi049_176811_object112239:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-21T14:09:05.347 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:04:20.245943+0000", 2024-09-21T14:09:05.347 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 285.04791395799998, 2024-09-21T14:09:05.347 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1573033979999998, 2024-09-21T14:09:05.347 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:09:05.347 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:09:05.347 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:09:05.347 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:09:05.347 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:09:05.347 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:09:05.348 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 112240 2024-09-21T14:09:05.348 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.348 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:09:05.348 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.348 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:09:05.348 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.245943+0000", 2024-09-21T14:09:05.348 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:05.348 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.348 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.348 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:09:05.348 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.245943+0000", 2024-09-21T14:09:05.348 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:05.348 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.349 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.349 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:09:05.349 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.245944+0000", 2024-09-21T14:09:05.349 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3540000000000001e-06 2024-09-21T14:09:05.349 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.349 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.349 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:09:05.349 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.246007+0000", 2024-09-21T14:09:05.349 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2155000000000004e-05 2024-09-21T14:09:05.349 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.349 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.349 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:09:05.349 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.246019+0000", 2024-09-21T14:09:05.349 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2814e-05 2024-09-21T14:09:05.350 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.350 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.350 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:09:05.350 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.246026+0000", 2024-09-21T14:09:05.350 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1470000000000004e-06 2024-09-21T14:09:05.350 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.350 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.350 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.350 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.246084+0000", 2024-09-21T14:09:05.350 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7731e-05 2024-09-21T14:09:05.350 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.350 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.350 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:09:05.351 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.246095+0000", 2024-09-21T14:09:05.351 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0964e-05 2024-09-21T14:09:05.351 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.351 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.351 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.351 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.188376+0000", 2024-09-21T14:09:05.351 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.94228104199999996 2024-09-21T14:09:05.351 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.351 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.351 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:09:05.351 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.188383+0000", 2024-09-21T14:09:05.351 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8979999999999999e-06 2024-09-21T14:09:05.351 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.351 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.352 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.352 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.380685+0000", 2024-09-21T14:09:05.352 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.192302239 2024-09-21T14:09:05.352 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.352 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.352 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:09:05.352 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.381343+0000", 2024-09-21T14:09:05.352 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00065746299999999995 2024-09-21T14:09:05.352 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.353 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.354 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2,5", 2024-09-21T14:09:05.354 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.381520+0000", 2024-09-21T14:09:05.354 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00017690300000000001 2024-09-21T14:09:05.354 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.354 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.354 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:09:05.354 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.388164+0000", 2024-09-21T14:09:05.354 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0066440359999999999 2024-09-21T14:09:05.354 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.354 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.354 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:05.354 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.388424+0000", 2024-09-21T14:09:05.354 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00026048999999999999 2024-09-21T14:09:05.355 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.355 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.355 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:05.355 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.388788+0000", 2024-09-21T14:09:05.355 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00036423599999999999 2024-09-21T14:09:05.355 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.355 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.355 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:05.355 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.403084+0000", 2024-09-21T14:09:05.355 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.014295491 2024-09-21T14:09:05.355 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.355 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.355 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:09:05.355 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.403141+0000", 2024-09-21T14:09:05.355 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7290999999999997e-05 2024-09-21T14:09:05.356 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.356 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.356 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:09:05.356 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.403246+0000", 2024-09-21T14:09:05.356 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010514399999999999 2024-09-21T14:09:05.356 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:05.356 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:09:05.356 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:05.356 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.356 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.356 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:112251 10.a 10:59ed2c1b:::benchmark_data_smithi049_176811_object112250:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-21T14:09:05.356 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:04:20.255435+0000", 2024-09-21T14:09:05.356 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 285.03842187499998, 2024-09-21T14:09:05.356 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1536580829999998, 2024-09-21T14:09:05.357 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:09:05.357 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:09:05.357 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:09:05.357 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:09:05.357 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:09:05.357 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:09:05.357 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 112251 2024-09-21T14:09:05.357 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.357 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:09:05.357 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.357 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:09:05.357 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.255435+0000", 2024-09-21T14:09:05.357 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:05.357 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.357 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.358 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:09:05.358 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.255435+0000", 2024-09-21T14:09:05.358 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:05.358 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.358 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.358 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:09:05.358 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.255436+0000", 2024-09-21T14:09:05.358 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.184e-06 2024-09-21T14:09:05.358 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.358 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.358 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:09:05.358 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.255522+0000", 2024-09-21T14:09:05.358 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5778000000000004e-05 2024-09-21T14:09:05.359 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.359 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.359 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:09:05.359 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.255532+0000", 2024-09-21T14:09:05.359 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.8199999999999992e-06 2024-09-21T14:09:05.359 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.359 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.359 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:09:05.359 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.255538+0000", 2024-09-21T14:09:05.359 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8710000000000002e-06 2024-09-21T14:09:05.359 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.359 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.359 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.360 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.255589+0000", 2024-09-21T14:09:05.360 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1644000000000002e-05 2024-09-21T14:09:05.360 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.360 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.360 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:09:05.360 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.255596+0000", 2024-09-21T14:09:05.360 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6499999999999999e-06 2024-09-21T14:09:05.360 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.360 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.360 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.360 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.188406+0000", 2024-09-21T14:09:05.360 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.932810425 2024-09-21T14:09:05.360 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.360 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.360 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:09:05.361 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.188413+0000", 2024-09-21T14:09:05.361 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9859999999999997e-06 2024-09-21T14:09:05.361 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.361 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.361 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.361 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.384443+0000", 2024-09-21T14:09:05.361 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.196029257 2024-09-21T14:09:05.361 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.361 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.361 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:09:05.361 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.384560+0000", 2024-09-21T14:09:05.361 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000117766 2024-09-21T14:09:05.361 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.361 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.362 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2,5", 2024-09-21T14:09:05.362 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.384711+0000", 2024-09-21T14:09:05.362 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015052899999999999 2024-09-21T14:09:05.362 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.362 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.362 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:09:05.362 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.388438+0000", 2024-09-21T14:09:05.362 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00372669 2024-09-21T14:09:05.362 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.362 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.362 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:05.362 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.388729+0000", 2024-09-21T14:09:05.362 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00029102899999999999 2024-09-21T14:09:05.362 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.363 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.363 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:05.363 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.397078+0000", 2024-09-21T14:09:05.363 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0083490219999999993 2024-09-21T14:09:05.363 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.363 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.363 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:05.363 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.408920+0000", 2024-09-21T14:09:05.363 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.011842065000000001 2024-09-21T14:09:05.363 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.363 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.363 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:09:05.363 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.408981+0000", 2024-09-21T14:09:05.363 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1569999999999995e-05 2024-09-21T14:09:05.364 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.364 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.364 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:09:05.364 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.409093+0000", 2024-09-21T14:09:05.364 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000111797 2024-09-21T14:09:05.364 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:05.364 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:09:05.364 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:05.364 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.364 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.364 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:112253 10.d 10:be7afa8e:::benchmark_data_smithi049_176811_object112252:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-21T14:09:05.364 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:04:20.263232+0000", 2024-09-21T14:09:05.364 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 285.03062510000001, 2024-09-21T14:09:05.364 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0592742639999999, 2024-09-21T14:09:05.364 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:09:05.365 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:09:05.365 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:09:05.365 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:09:05.365 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:09:05.365 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:09:05.365 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 112253 2024-09-21T14:09:05.365 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.365 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:09:05.365 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.365 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:09:05.365 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.263232+0000", 2024-09-21T14:09:05.365 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:05.365 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.365 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.365 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:09:05.365 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.263232+0000", 2024-09-21T14:09:05.366 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:05.366 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.366 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.366 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:09:05.366 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.263233+0000", 2024-09-21T14:09:05.366 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.488e-06 2024-09-21T14:09:05.366 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.366 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.366 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:09:05.366 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.263305+0000", 2024-09-21T14:09:05.366 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1552999999999997e-05 2024-09-21T14:09:05.366 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.366 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.366 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:09:05.367 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.263319+0000", 2024-09-21T14:09:05.367 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4233000000000001e-05 2024-09-21T14:09:05.367 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.367 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.367 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:09:05.367 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.263325+0000", 2024-09-21T14:09:05.367 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9950000000000002e-06 2024-09-21T14:09:05.367 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.367 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.367 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.367 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.263389+0000", 2024-09-21T14:09:05.367 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3567999999999999e-05 2024-09-21T14:09:05.367 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.367 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.368 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:09:05.368 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.263400+0000", 2024-09-21T14:09:05.368 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1206000000000001e-05 2024-09-21T14:09:05.368 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.368 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.368 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.368 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.187432+0000", 2024-09-21T14:09:05.368 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.92403234099999998 2024-09-21T14:09:05.368 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.368 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.368 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:09:05.368 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.187440+0000", 2024-09-21T14:09:05.368 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0260000000000007e-06 2024-09-21T14:09:05.368 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.368 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.369 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.369 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.295005+0000", 2024-09-21T14:09:05.369 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1075643630000001 2024-09-21T14:09:05.369 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.369 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.369 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:09:05.369 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.295169+0000", 2024-09-21T14:09:05.369 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000163859 2024-09-21T14:09:05.369 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.369 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.369 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3,4", 2024-09-21T14:09:05.369 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.295394+0000", 2024-09-21T14:09:05.369 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000225353 2024-09-21T14:09:05.369 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.369 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.369 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:09:05.370 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.298164+0000", 2024-09-21T14:09:05.370 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0027705170000000001 2024-09-21T14:09:05.370 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.370 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.370 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:05.370 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.298919+0000", 2024-09-21T14:09:05.370 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00075427899999999995 2024-09-21T14:09:05.370 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.370 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.370 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:05.370 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.316063+0000", 2024-09-21T14:09:05.370 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.017144309 2024-09-21T14:09:05.370 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.370 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.371 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:05.371 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.322394+0000", 2024-09-21T14:09:05.371 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0063310720000000001 2024-09-21T14:09:05.371 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.371 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.371 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:09:05.371 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.322436+0000", 2024-09-21T14:09:05.371 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1897999999999998e-05 2024-09-21T14:09:05.371 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.371 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.371 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:09:05.371 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.322506+0000", 2024-09-21T14:09:05.371 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.0203999999999994e-05 2024-09-21T14:09:05.371 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:05.372 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:09:05.372 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:05.372 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.372 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.372 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:112256 10.a 10:5949a1c7:::benchmark_data_smithi049_176811_object112255:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-21T14:09:05.372 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:04:20.267531+0000", 2024-09-21T14:09:05.372 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 285.02632643999999, 2024-09-21T14:09:05.372 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.141982547, 2024-09-21T14:09:05.372 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:09:05.372 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:09:05.372 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:09:05.372 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:09:05.372 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:09:05.372 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:09:05.372 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 112256 2024-09-21T14:09:05.372 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.373 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:09:05.373 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.373 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:09:05.373 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.267531+0000", 2024-09-21T14:09:05.373 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:05.373 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.373 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.373 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:09:05.373 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.267531+0000", 2024-09-21T14:09:05.373 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:05.373 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.373 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.373 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:09:05.373 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.267531+0000", 2024-09-21T14:09:05.374 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1999999999999998e-07 2024-09-21T14:09:05.374 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.374 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.374 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:09:05.374 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.267570+0000", 2024-09-21T14:09:05.374 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8336e-05 2024-09-21T14:09:05.374 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.374 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.374 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:09:05.374 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.267579+0000", 2024-09-21T14:09:05.374 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.9930000000000001e-06 2024-09-21T14:09:05.374 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.374 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.374 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:09:05.374 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.267583+0000", 2024-09-21T14:09:05.375 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3830000000000002e-06 2024-09-21T14:09:05.375 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.375 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.375 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.375 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.267632+0000", 2024-09-21T14:09:05.375 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8822000000000001e-05 2024-09-21T14:09:05.375 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.375 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.375 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:09:05.375 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.267640+0000", 2024-09-21T14:09:05.375 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2300000000000008e-06 2024-09-21T14:09:05.375 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.375 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.375 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.376 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.188439+0000", 2024-09-21T14:09:05.376 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.92079873300000004 2024-09-21T14:09:05.376 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.376 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.376 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:09:05.376 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.188446+0000", 2024-09-21T14:09:05.376 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.255e-06 2024-09-21T14:09:05.376 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.376 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.376 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.376 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.388852+0000", 2024-09-21T14:09:05.376 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.200405897 2024-09-21T14:09:05.376 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.376 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.377 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:09:05.377 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.389001+0000", 2024-09-21T14:09:05.377 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000148641 2024-09-21T14:09:05.377 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.377 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.377 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2,5", 2024-09-21T14:09:05.377 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.389171+0000", 2024-09-21T14:09:05.377 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00017001399999999999 2024-09-21T14:09:05.377 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.377 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.377 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:05.377 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.396980+0000", 2024-09-21T14:09:05.377 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0078089149999999996 2024-09-21T14:09:05.377 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.377 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.377 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:05.378 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.397128+0000", 2024-09-21T14:09:05.378 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014819799999999999 2024-09-21T14:09:05.378 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.378 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.378 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:09:05.378 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.399986+0000", 2024-09-21T14:09:05.378 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0028577210000000001 2024-09-21T14:09:05.378 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.378 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.378 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:05.378 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.409327+0000", 2024-09-21T14:09:05.378 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0093417110000000008 2024-09-21T14:09:05.378 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.378 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.379 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:09:05.379 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.409390+0000", 2024-09-21T14:09:05.379 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2636000000000001e-05 2024-09-21T14:09:05.379 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.379 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.379 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:09:05.379 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.409513+0000", 2024-09-21T14:09:05.379 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012314199999999999 2024-09-21T14:09:05.379 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:05.379 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:09:05.379 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:05.379 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.379 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.379 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:112262 10.a 10:5144907d:::benchmark_data_smithi049_176811_object112261:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-21T14:09:05.380 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:04:20.285420+0000", 2024-09-21T14:09:05.380 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 285.00843668099998, 2024-09-21T14:09:05.380 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1243977709999999, 2024-09-21T14:09:05.380 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:09:05.380 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:09:05.380 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:09:05.380 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:09:05.380 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:09:05.380 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:09:05.380 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 112262 2024-09-21T14:09:05.380 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.380 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:09:05.380 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.381 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:09:05.381 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.285420+0000", 2024-09-21T14:09:05.381 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:05.381 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.381 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.381 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:09:05.381 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.285420+0000", 2024-09-21T14:09:05.381 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:05.381 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.381 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.381 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:09:05.381 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.285422+0000", 2024-09-21T14:09:05.381 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4449999999999999e-06 2024-09-21T14:09:05.381 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.381 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.382 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:09:05.382 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.285450+0000", 2024-09-21T14:09:05.382 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7985999999999999e-05 2024-09-21T14:09:05.382 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.382 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.382 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:09:05.382 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.285462+0000", 2024-09-21T14:09:05.382 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.185e-05 2024-09-21T14:09:05.382 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.382 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.382 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:09:05.382 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.285468+0000", 2024-09-21T14:09:05.383 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3570000000000003e-06 2024-09-21T14:09:05.383 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.383 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.383 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.383 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.285519+0000", 2024-09-21T14:09:05.383 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0912999999999999e-05 2024-09-21T14:09:05.383 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.383 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.383 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:09:05.383 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.285526+0000", 2024-09-21T14:09:05.383 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1069999999999999e-06 2024-09-21T14:09:05.383 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.383 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.383 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.384 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.188522+0000", 2024-09-21T14:09:05.384 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.90299566200000003 2024-09-21T14:09:05.384 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.384 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.384 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:09:05.384 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.188533+0000", 2024-09-21T14:09:05.384 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0903999999999999e-05 2024-09-21T14:09:05.384 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.384 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.384 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.384 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.391763+0000", 2024-09-21T14:09:05.384 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2032303799999999 2024-09-21T14:09:05.384 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.385 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.385 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:09:05.385 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.391895+0000", 2024-09-21T14:09:05.385 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013225300000000001 2024-09-21T14:09:05.385 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.385 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.385 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2,5", 2024-09-21T14:09:05.385 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.392058+0000", 2024-09-21T14:09:05.385 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000162619 2024-09-21T14:09:05.385 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.385 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.385 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:05.385 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.397025+0000", 2024-09-21T14:09:05.385 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0049667569999999996 2024-09-21T14:09:05.385 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.385 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.386 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:09:05.386 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.402990+0000", 2024-09-21T14:09:05.386 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0059658200000000002 2024-09-21T14:09:05.386 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.386 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.386 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:05.386 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.403310+0000", 2024-09-21T14:09:05.386 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00031964599999999998 2024-09-21T14:09:05.386 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.386 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.386 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:05.386 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.409700+0000", 2024-09-21T14:09:05.386 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0063900240000000002 2024-09-21T14:09:05.387 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.387 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.387 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:09:05.387 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.409743+0000", 2024-09-21T14:09:05.387 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2670000000000003e-05 2024-09-21T14:09:05.387 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.387 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.387 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:09:05.387 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.409818+0000", 2024-09-21T14:09:05.387 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5377999999999996e-05 2024-09-21T14:09:05.387 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:05.387 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:09:05.387 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:05.387 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.387 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.388 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26396.0:114618 11.9 11:93cd8964:::benchmark_data_smithi049_206132_object114617:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e496)", 2024-09-21T14:09:05.388 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:08:13.395853+0000", 2024-09-21T14:09:05.388 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 51.898004393000001, 2024-09-21T14:09:05.388 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.082294047, 2024-09-21T14:09:05.388 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:09:05.388 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:09:05.388 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-21T14:09:05.388 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:09:05.388 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26396", 2024-09-21T14:09:05.388 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:09:05.388 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 114618 2024-09-21T14:09:05.388 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.388 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:09:05.388 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.389 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:09:05.389 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.395853+0000", 2024-09-21T14:09:05.389 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:05.389 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.389 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.389 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:09:05.389 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.395853+0000", 2024-09-21T14:09:05.389 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:05.389 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.389 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.389 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:09:05.389 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.395854+0000", 2024-09-21T14:09:05.389 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5740000000000001e-06 2024-09-21T14:09:05.389 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.389 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.390 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:09:05.390 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.395890+0000", 2024-09-21T14:09:05.390 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5565999999999998e-05 2024-09-21T14:09:05.390 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.390 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.390 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:09:05.390 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.395913+0000", 2024-09-21T14:09:05.390 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3283000000000001e-05 2024-09-21T14:09:05.390 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.390 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.390 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:09:05.390 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.395921+0000", 2024-09-21T14:09:05.390 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3080000000000008e-06 2024-09-21T14:09:05.390 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.390 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.391 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.391 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.396057+0000", 2024-09-21T14:09:05.391 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013601600000000001 2024-09-21T14:09:05.391 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.391 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.391 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:09:05.391 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.396066+0000", 2024-09-21T14:09:05.391 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1260000000000004e-06 2024-09-21T14:09:05.391 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.391 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.391 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.391 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:14.382158+0000", 2024-09-21T14:09:05.391 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.98609193299999998 2024-09-21T14:09:05.391 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.391 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.392 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:09:05.392 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:14.382166+0000", 2024-09-21T14:09:05.392 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4420000000000001e-06 2024-09-21T14:09:05.392 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.392 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.392 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.392 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.477272+0000", 2024-09-21T14:09:05.392 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.095106286 2024-09-21T14:09:05.392 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.392 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.392 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:09:05.392 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.478147+0000", 2024-09-21T14:09:05.392 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00087451300000000005 2024-09-21T14:09:05.392 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:05.392 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:09:05.393 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:05.393 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.393 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.393 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26396.0:114621 11.9 11:908cded0:::benchmark_data_smithi049_206132_object114620:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e496)", 2024-09-21T14:09:05.393 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:08:13.395943+0000", 2024-09-21T14:09:05.393 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 51.897914182999997, 2024-09-21T14:09:05.393 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0826702849999998, 2024-09-21T14:09:05.393 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:09:05.393 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:09:05.393 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-21T14:09:05.393 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:09:05.393 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26396", 2024-09-21T14:09:05.393 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:09:05.394 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 114621 2024-09-21T14:09:05.394 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.394 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:09:05.394 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.394 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:09:05.394 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.395943+0000", 2024-09-21T14:09:05.394 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:05.394 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.394 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.394 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:09:05.394 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.395943+0000", 2024-09-21T14:09:05.394 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:05.394 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.395 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.395 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:09:05.395 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.395943+0000", 2024-09-21T14:09:05.395 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1900000000000002e-07 2024-09-21T14:09:05.395 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.395 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.395 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:09:05.395 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.395971+0000", 2024-09-21T14:09:05.395 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7152e-05 2024-09-21T14:09:05.395 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.395 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.395 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:09:05.395 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.395978+0000", 2024-09-21T14:09:05.395 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1269999999999997e-06 2024-09-21T14:09:05.395 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.396 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.396 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:09:05.396 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.395983+0000", 2024-09-21T14:09:05.396 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0640000000000001e-06 2024-09-21T14:09:05.396 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.396 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.396 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.396 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.396087+0000", 2024-09-21T14:09:05.396 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000104096 2024-09-21T14:09:05.396 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.396 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.396 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:09:05.396 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.396093+0000", 2024-09-21T14:09:05.396 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9209999999999997e-06 2024-09-21T14:09:05.397 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.397 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.397 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.397 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:14.382224+0000", 2024-09-21T14:09:05.397 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.98613127199999995 2024-09-21T14:09:05.397 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.397 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.397 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:09:05.397 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:14.382231+0000", 2024-09-21T14:09:05.397 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.951e-06 2024-09-21T14:09:05.397 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.397 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.397 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.397 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.478549+0000", 2024-09-21T14:09:05.397 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.096318106 2024-09-21T14:09:05.397 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.398 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.398 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:09:05.398 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.478613+0000", 2024-09-21T14:09:05.398 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3977000000000001e-05 2024-09-21T14:09:05.398 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:05.398 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:09:05.398 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:05.398 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.398 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.398 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26396.0:114641 11.1 11:877bfe6e:::benchmark_data_smithi049_206132_object114640:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e496)", 2024-09-21T14:09:05.398 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:08:13.423568+0000", 2024-09-21T14:09:05.398 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 51.870288909000003, 2024-09-21T14:09:05.398 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0805294160000001, 2024-09-21T14:09:05.399 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:09:05.399 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:09:05.399 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:09:05.399 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:09:05.399 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26396", 2024-09-21T14:09:05.399 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:09:05.399 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 114641 2024-09-21T14:09:05.399 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.399 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:09:05.399 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.399 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:09:05.399 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.423568+0000", 2024-09-21T14:09:05.399 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:05.399 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.400 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.400 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:09:05.400 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.423568+0000", 2024-09-21T14:09:05.400 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:05.400 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.400 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.400 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:09:05.400 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.423569+0000", 2024-09-21T14:09:05.400 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.142e-06 2024-09-21T14:09:05.400 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.400 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.400 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:09:05.400 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.423598+0000", 2024-09-21T14:09:05.400 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9127000000000001e-05 2024-09-21T14:09:05.401 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.401 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.401 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:09:05.401 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.423607+0000", 2024-09-21T14:09:05.401 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8610000000000004e-06 2024-09-21T14:09:05.401 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.401 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.401 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:09:05.401 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.423613+0000", 2024-09-21T14:09:05.401 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1850000000000001e-06 2024-09-21T14:09:05.401 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.401 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.401 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.401 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.423811+0000", 2024-09-21T14:09:05.401 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00019768300000000001 2024-09-21T14:09:05.401 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.402 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.402 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:09:05.402 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.423820+0000", 2024-09-21T14:09:05.402 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8689999999999993e-06 2024-09-21T14:09:05.402 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.402 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.402 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.402 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:14.382121+0000", 2024-09-21T14:09:05.402 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.95830109399999996 2024-09-21T14:09:05.402 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.402 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.402 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:09:05.402 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:14.382130+0000", 2024-09-21T14:09:05.403 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5939999999999994e-06 2024-09-21T14:09:05.403 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.403 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.403 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.403 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.485606+0000", 2024-09-21T14:09:05.403 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.103476015 2024-09-21T14:09:05.403 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.403 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.403 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:09:05.403 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.485724+0000", 2024-09-21T14:09:05.403 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000117935 2024-09-21T14:09:05.403 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.403 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.403 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2,3", 2024-09-21T14:09:05.403 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.486171+0000", 2024-09-21T14:09:05.404 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00044764300000000001 2024-09-21T14:09:05.404 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.404 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.404 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:09:05.404 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.503749+0000", 2024-09-21T14:09:05.404 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.017577406 2024-09-21T14:09:05.404 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.404 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.404 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:05.404 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.503794+0000", 2024-09-21T14:09:05.404 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5247999999999998e-05 2024-09-21T14:09:05.404 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.404 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.404 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:05.405 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.503863+0000", 2024-09-21T14:09:05.405 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9047000000000005e-05 2024-09-21T14:09:05.405 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.405 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.405 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:05.405 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.503900+0000", 2024-09-21T14:09:05.405 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6609000000000001e-05 2024-09-21T14:09:05.405 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.405 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.405 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:09:05.405 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.503962+0000", 2024-09-21T14:09:05.405 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2712999999999994e-05 2024-09-21T14:09:05.405 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.405 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.405 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:09:05.405 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.504097+0000", 2024-09-21T14:09:05.406 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013524500000000001 2024-09-21T14:09:05.406 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:05.406 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:09:05.406 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:05.408 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.408 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.408 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26396.0:114655 11.1 11:81711ddc:::benchmark_data_smithi049_206132_object114654:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e496)", 2024-09-21T14:09:05.408 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:08:13.438595+0000", 2024-09-21T14:09:05.408 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 51.855262265, 2024-09-21T14:09:05.408 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0658325770000001, 2024-09-21T14:09:05.408 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:09:05.408 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:09:05.408 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:09:05.408 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:09:05.408 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26396", 2024-09-21T14:09:05.408 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:09:05.409 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 114655 2024-09-21T14:09:05.409 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.409 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:09:05.409 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.409 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:09:05.409 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.438595+0000", 2024-09-21T14:09:05.409 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:05.409 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.409 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.409 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:09:05.409 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.438595+0000", 2024-09-21T14:09:05.409 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:05.409 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.409 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.409 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:09:05.410 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.438597+0000", 2024-09-21T14:09:05.410 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8279999999999999e-06 2024-09-21T14:09:05.410 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.410 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.410 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:09:05.410 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.438653+0000", 2024-09-21T14:09:05.410 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6721999999999997e-05 2024-09-21T14:09:05.410 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.410 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.410 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:09:05.410 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.438667+0000", 2024-09-21T14:09:05.410 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3458000000000001e-05 2024-09-21T14:09:05.410 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.410 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.411 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:09:05.411 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.438674+0000", 2024-09-21T14:09:05.411 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.661e-06 2024-09-21T14:09:05.411 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.411 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.411 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.411 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.438739+0000", 2024-09-21T14:09:05.411 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4886000000000002e-05 2024-09-21T14:09:05.411 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.411 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.411 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:09:05.411 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.438749+0000", 2024-09-21T14:09:05.412 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.285e-06 2024-09-21T14:09:05.412 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.412 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.412 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.412 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:14.382193+0000", 2024-09-21T14:09:05.412 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.94344481099999999 2024-09-21T14:09:05.412 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.412 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.412 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:09:05.412 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:14.382201+0000", 2024-09-21T14:09:05.412 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2729999999999997e-06 2024-09-21T14:09:05.412 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.412 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.412 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.413 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.490327+0000", 2024-09-21T14:09:05.413 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1081260070000001 2024-09-21T14:09:05.413 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.413 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.413 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:09:05.413 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.490433+0000", 2024-09-21T14:09:05.413 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000106208 2024-09-21T14:09:05.413 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.413 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.413 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2,3", 2024-09-21T14:09:05.413 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.490538+0000", 2024-09-21T14:09:05.413 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000105078 2024-09-21T14:09:05.413 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.413 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.413 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:09:05.414 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.503754+0000", 2024-09-21T14:09:05.414 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.013216462999999999 2024-09-21T14:09:05.414 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.414 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.414 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:05.414 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.504143+0000", 2024-09-21T14:09:05.414 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00038873100000000002 2024-09-21T14:09:05.414 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.414 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.414 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:05.414 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.504183+0000", 2024-09-21T14:09:05.414 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9845999999999998e-05 2024-09-21T14:09:05.414 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.414 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.414 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:05.415 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.504224+0000", 2024-09-21T14:09:05.415 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0930999999999998e-05 2024-09-21T14:09:05.415 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.415 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.415 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:09:05.415 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.504296+0000", 2024-09-21T14:09:05.415 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2177999999999999e-05 2024-09-21T14:09:05.415 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.415 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.415 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:09:05.415 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.504427+0000", 2024-09-21T14:09:05.415 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000131211 2024-09-21T14:09:05.415 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:05.415 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:09:05.415 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:05.416 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.416 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.416 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26396.0:114658 11.8 11:14301b05:::benchmark_data_smithi049_206132_object114657:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e496)", 2024-09-21T14:09:05.416 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:08:13.439803+0000", 2024-09-21T14:09:05.416 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 51.854054013000002, 2024-09-21T14:09:05.416 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0403675209999999, 2024-09-21T14:09:05.416 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:09:05.416 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:09:05.416 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:09:05.416 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:09:05.416 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26396", 2024-09-21T14:09:05.416 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:09:05.416 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 114658 2024-09-21T14:09:05.416 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.416 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:09:05.417 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.417 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:09:05.417 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.439803+0000", 2024-09-21T14:09:05.417 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:05.417 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.417 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.417 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:09:05.417 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.439803+0000", 2024-09-21T14:09:05.417 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:05.417 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.417 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.417 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:09:05.417 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.439804+0000", 2024-09-21T14:09:05.418 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0440000000000001e-06 2024-09-21T14:09:05.418 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.418 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.418 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:09:05.418 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.439976+0000", 2024-09-21T14:09:05.418 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000172246 2024-09-21T14:09:05.418 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.418 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.418 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:09:05.418 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.439995+0000", 2024-09-21T14:09:05.418 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8457999999999998e-05 2024-09-21T14:09:05.418 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.418 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.419 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:09:05.419 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.440003+0000", 2024-09-21T14:09:05.419 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3280000000000006e-06 2024-09-21T14:09:05.419 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.419 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.419 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.419 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.440069+0000", 2024-09-21T14:09:05.419 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5916999999999999e-05 2024-09-21T14:09:05.419 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.419 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.419 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:09:05.419 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.440079+0000", 2024-09-21T14:09:05.419 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6569999999999992e-06 2024-09-21T14:09:05.420 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.420 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.420 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.420 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:14.381910+0000", 2024-09-21T14:09:05.420 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.94183137100000003 2024-09-21T14:09:05.420 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.420 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.420 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:09:05.420 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:14.381921+0000", 2024-09-21T14:09:05.420 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1504e-05 2024-09-21T14:09:05.420 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.420 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.420 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.420 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.470599+0000", 2024-09-21T14:09:05.420 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0886771740000001 2024-09-21T14:09:05.420 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.421 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.421 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:09:05.421 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.472108+0000", 2024-09-21T14:09:05.421 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001509327 2024-09-21T14:09:05.421 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.421 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.421 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,3,5", 2024-09-21T14:09:05.421 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.472305+0000", 2024-09-21T14:09:05.421 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0001973 2024-09-21T14:09:05.421 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.421 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.421 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:09:05.421 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.478156+0000", 2024-09-21T14:09:05.421 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0058505679999999996 2024-09-21T14:09:05.421 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.422 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.422 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:05.422 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.478430+0000", 2024-09-21T14:09:05.422 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000274438 2024-09-21T14:09:05.422 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.422 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.422 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:05.422 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.478926+0000", 2024-09-21T14:09:05.422 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00049566400000000004 2024-09-21T14:09:05.422 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.422 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.422 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:05.422 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.479986+0000", 2024-09-21T14:09:05.422 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0010602039999999999 2024-09-21T14:09:05.423 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.423 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.423 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:09:05.423 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.480049+0000", 2024-09-21T14:09:05.423 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2459999999999995e-05 2024-09-21T14:09:05.423 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.423 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.423 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:09:05.423 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.480170+0000", 2024-09-21T14:09:05.423 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000121861 2024-09-21T14:09:05.423 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:05.423 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:09:05.423 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:05.423 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.424 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.424 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26396.0:114665 11.1 11:83a286ac:::benchmark_data_smithi049_206132_object114664:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e496)", 2024-09-21T14:09:05.424 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:08:13.447546+0000", 2024-09-21T14:09:05.424 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 51.846310715999998, 2024-09-21T14:09:05.424 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.057220273, 2024-09-21T14:09:05.424 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:09:05.424 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:09:05.424 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:09:05.424 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:09:05.424 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26396", 2024-09-21T14:09:05.424 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:09:05.424 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 114665 2024-09-21T14:09:05.424 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.424 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:09:05.424 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.425 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:09:05.425 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.447546+0000", 2024-09-21T14:09:05.425 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:05.425 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.425 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.425 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:09:05.425 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.447546+0000", 2024-09-21T14:09:05.425 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:05.425 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.425 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.425 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:09:05.425 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.447547+0000", 2024-09-21T14:09:05.425 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.057e-06 2024-09-21T14:09:05.425 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.426 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.426 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:09:05.426 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.447573+0000", 2024-09-21T14:09:05.426 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5749999999999999e-05 2024-09-21T14:09:05.426 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.426 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.426 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:09:05.426 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.447584+0000", 2024-09-21T14:09:05.426 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0672e-05 2024-09-21T14:09:05.426 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.426 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.426 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:09:05.426 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.447591+0000", 2024-09-21T14:09:05.426 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7800000000000003e-06 2024-09-21T14:09:05.427 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.427 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.427 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.427 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.447659+0000", 2024-09-21T14:09:05.427 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8356999999999996e-05 2024-09-21T14:09:05.427 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.427 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.427 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:09:05.427 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.447668+0000", 2024-09-21T14:09:05.427 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1670000000000005e-06 2024-09-21T14:09:05.427 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.427 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.427 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.427 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:14.382257+0000", 2024-09-21T14:09:05.428 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.93458886500000005 2024-09-21T14:09:05.428 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.428 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.428 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:09:05.428 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:14.382264+0000", 2024-09-21T14:09:05.428 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3209999999999999e-06 2024-09-21T14:09:05.428 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.428 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.428 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.428 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.492016+0000", 2024-09-21T14:09:05.428 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1097516160000001 2024-09-21T14:09:05.428 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.428 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.428 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:09:05.429 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.492143+0000", 2024-09-21T14:09:05.429 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012762400000000001 2024-09-21T14:09:05.429 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.429 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.429 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2,3", 2024-09-21T14:09:05.429 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.492248+0000", 2024-09-21T14:09:05.429 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010458699999999999 2024-09-21T14:09:05.429 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.429 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.429 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:09:05.429 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.503808+0000", 2024-09-21T14:09:05.429 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.011559764 2024-09-21T14:09:05.429 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.429 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.429 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:05.430 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.504471+0000", 2024-09-21T14:09:05.430 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00066317899999999996 2024-09-21T14:09:05.430 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.430 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.430 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:05.430 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.504522+0000", 2024-09-21T14:09:05.430 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0837999999999997e-05 2024-09-21T14:09:05.430 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.430 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.430 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:05.430 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.504580+0000", 2024-09-21T14:09:05.430 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8279999999999998e-05 2024-09-21T14:09:05.430 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.430 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.431 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:09:05.431 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.504654+0000", 2024-09-21T14:09:05.431 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4173999999999998e-05 2024-09-21T14:09:05.431 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.431 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.431 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:09:05.431 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.504767+0000", 2024-09-21T14:09:05.431 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000112242 2024-09-21T14:09:05.431 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:05.431 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:09:05.431 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:05.431 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.431 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.431 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26396.0:114667 11.9 11:9a028036:::benchmark_data_smithi049_206132_object114666:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e496)", 2024-09-21T14:09:05.432 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:08:13.448989+0000", 2024-09-21T14:09:05.432 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 51.844868439000003, 2024-09-21T14:09:05.432 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0554569699999998, 2024-09-21T14:09:05.432 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:09:05.432 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:09:05.432 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:09:05.432 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:09:05.432 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26396", 2024-09-21T14:09:05.432 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:09:05.432 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 114667 2024-09-21T14:09:05.432 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.432 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:09:05.432 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.432 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:09:05.433 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.448989+0000", 2024-09-21T14:09:05.433 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:05.433 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.433 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.433 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:09:05.433 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.448989+0000", 2024-09-21T14:09:05.433 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:05.433 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.433 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.433 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:09:05.433 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.448990+0000", 2024-09-21T14:09:05.433 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.048e-06 2024-09-21T14:09:05.433 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.433 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.434 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:09:05.434 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.449014+0000", 2024-09-21T14:09:05.434 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4239e-05 2024-09-21T14:09:05.434 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.434 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.434 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:09:05.434 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.449023+0000", 2024-09-21T14:09:05.434 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0019999999999995e-06 2024-09-21T14:09:05.434 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.434 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.434 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:09:05.434 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.449029+0000", 2024-09-21T14:09:05.434 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1539999999999999e-06 2024-09-21T14:09:05.434 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.435 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.435 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.435 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.449083+0000", 2024-09-21T14:09:05.435 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3826000000000003e-05 2024-09-21T14:09:05.435 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.435 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.435 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:09:05.435 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.449093+0000", 2024-09-21T14:09:05.435 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7289999999999995e-06 2024-09-21T14:09:05.435 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.435 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.435 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.435 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:14.382287+0000", 2024-09-21T14:09:05.435 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.93319489799999999 2024-09-21T14:09:05.435 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.435 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.436 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:09:05.436 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:14.382294+0000", 2024-09-21T14:09:05.436 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9720000000000003e-06 2024-09-21T14:09:05.436 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.436 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.436 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.436 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.478634+0000", 2024-09-21T14:09:05.436 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.096339365 2024-09-21T14:09:05.436 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.436 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.436 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:09:05.436 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.478925+0000", 2024-09-21T14:09:05.436 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00029151600000000002 2024-09-21T14:09:05.436 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.437 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.437 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3,7", 2024-09-21T14:09:05.437 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.479198+0000", 2024-09-21T14:09:05.437 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00027239499999999998 2024-09-21T14:09:05.437 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.437 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.437 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:05.437 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.485480+0000", 2024-09-21T14:09:05.437 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0062823159999999996 2024-09-21T14:09:05.437 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.437 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.437 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:09:05.437 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.490308+0000", 2024-09-21T14:09:05.437 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0048277040000000004 2024-09-21T14:09:05.438 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.438 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.438 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:05.438 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.491855+0000", 2024-09-21T14:09:05.438 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00154737 2024-09-21T14:09:05.438 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.438 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.438 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:05.438 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.504276+0000", 2024-09-21T14:09:05.438 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.012421129 2024-09-21T14:09:05.438 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.438 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.438 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:09:05.438 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.504340+0000", 2024-09-21T14:09:05.439 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3678999999999998e-05 2024-09-21T14:09:05.439 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.439 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.439 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:09:05.439 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.504446+0000", 2024-09-21T14:09:05.439 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000105628 2024-09-21T14:09:05.439 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:05.439 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:09:05.439 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:05.439 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.439 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.439 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26396.0:114671 11.9 11:95b30c50:::benchmark_data_smithi049_206132_object114670:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e496)", 2024-09-21T14:09:05.439 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:08:13.451969+0000", 2024-09-21T14:09:05.439 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 51.841888482000002, 2024-09-21T14:09:05.440 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.077318972, 2024-09-21T14:09:05.440 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:09:05.440 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:09:05.440 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:09:05.440 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:09:05.440 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26396", 2024-09-21T14:09:05.440 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:09:05.440 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 114671 2024-09-21T14:09:05.440 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.440 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:09:05.440 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.440 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:09:05.440 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.451969+0000", 2024-09-21T14:09:05.440 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:05.440 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.441 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.441 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:09:05.441 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.451969+0000", 2024-09-21T14:09:05.441 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:05.441 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.441 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.441 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:09:05.441 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.451969+0000", 2024-09-21T14:09:05.441 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7999999999999993e-07 2024-09-21T14:09:05.441 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.441 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.441 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:09:05.441 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.452029+0000", 2024-09-21T14:09:05.441 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9088000000000001e-05 2024-09-21T14:09:05.441 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.442 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.442 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:09:05.442 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.452039+0000", 2024-09-21T14:09:05.442 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.012e-05 2024-09-21T14:09:05.442 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.442 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.442 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:09:05.442 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.452046+0000", 2024-09-21T14:09:05.442 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.0870000000000001e-06 2024-09-21T14:09:05.442 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.442 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.442 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.442 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.452098+0000", 2024-09-21T14:09:05.442 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2012999999999998e-05 2024-09-21T14:09:05.443 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.443 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.443 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:09:05.443 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.452105+0000", 2024-09-21T14:09:05.443 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9929999999999998e-06 2024-09-21T14:09:05.443 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.443 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.443 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.443 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:14.382365+0000", 2024-09-21T14:09:05.443 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.93026017699999997 2024-09-21T14:09:05.443 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.443 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.443 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:09:05.443 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:14.382373+0000", 2024-09-21T14:09:05.443 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6260000000000003e-06 2024-09-21T14:09:05.444 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.444 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.444 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.444 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.505275+0000", 2024-09-21T14:09:05.444 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1229028670000001 2024-09-21T14:09:05.444 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.444 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.444 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:09:05.444 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.505402+0000", 2024-09-21T14:09:05.444 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012643299999999999 2024-09-21T14:09:05.444 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.444 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.444 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3,7", 2024-09-21T14:09:05.444 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.505573+0000", 2024-09-21T14:09:05.444 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00017150299999999999 2024-09-21T14:09:05.445 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.445 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.445 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:05.445 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.507545+0000", 2024-09-21T14:09:05.445 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0019714939999999999 2024-09-21T14:09:05.445 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.445 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.445 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:09:05.445 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.513978+0000", 2024-09-21T14:09:05.445 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0064327050000000004 2024-09-21T14:09:05.445 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.445 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.445 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:05.445 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.525537+0000", 2024-09-21T14:09:05.445 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.011559135 2024-09-21T14:09:05.446 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.446 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.446 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:05.446 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.529080+0000", 2024-09-21T14:09:05.446 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0035429580000000001 2024-09-21T14:09:05.446 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.446 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.446 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:09:05.446 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.529159+0000", 2024-09-21T14:09:05.446 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8903e-05 2024-09-21T14:09:05.446 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.446 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.446 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:09:05.446 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.529287+0000", 2024-09-21T14:09:05.446 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012888999999999999 2024-09-21T14:09:05.447 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:05.447 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:09:05.447 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:05.447 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.447 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.447 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26396.0:114673 11.1 11:8d58e7bd:::benchmark_data_smithi049_206132_object114672:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e496)", 2024-09-21T14:09:05.447 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:08:13.458726+0000", 2024-09-21T14:09:05.447 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 51.835131433000001, 2024-09-21T14:09:05.447 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0464583840000001, 2024-09-21T14:09:05.447 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:09:05.447 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:09:05.447 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:09:05.447 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:09:05.447 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26396", 2024-09-21T14:09:05.448 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:09:05.448 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 114673 2024-09-21T14:09:05.448 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.448 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:09:05.448 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.448 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:09:05.448 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.458726+0000", 2024-09-21T14:09:05.448 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:05.448 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.448 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.448 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:09:05.448 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.458726+0000", 2024-09-21T14:09:05.448 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:05.448 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.449 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.449 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:09:05.449 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.458732+0000", 2024-09-21T14:09:05.449 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6089999999999997e-06 2024-09-21T14:09:05.449 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.449 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.449 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:09:05.449 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.458762+0000", 2024-09-21T14:09:05.449 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0312e-05 2024-09-21T14:09:05.449 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.449 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.449 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:09:05.449 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.458777+0000", 2024-09-21T14:09:05.449 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4078e-05 2024-09-21T14:09:05.450 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.450 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.450 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:09:05.450 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.458784+0000", 2024-09-21T14:09:05.450 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6590000000000007e-06 2024-09-21T14:09:05.450 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.450 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.450 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.450 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.458844+0000", 2024-09-21T14:09:05.450 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0198000000000002e-05 2024-09-21T14:09:05.450 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.450 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.450 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:09:05.450 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.458855+0000", 2024-09-21T14:09:05.451 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0926e-05 2024-09-21T14:09:05.451 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.451 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.451 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.451 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:14.382334+0000", 2024-09-21T14:09:05.451 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.92347837899999996 2024-09-21T14:09:05.451 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.451 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.451 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:09:05.451 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:14.382342+0000", 2024-09-21T14:09:05.451 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5199999999999997e-06 2024-09-21T14:09:05.451 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.451 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.452 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:05.452 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.493596+0000", 2024-09-21T14:09:05.452 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1112538190000001 2024-09-21T14:09:05.452 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.452 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.452 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:09:05.452 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.493665+0000", 2024-09-21T14:09:05.452 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9076000000000001e-05 2024-09-21T14:09:05.452 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.452 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.452 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2,3", 2024-09-21T14:09:05.452 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.493770+0000", 2024-09-21T14:09:05.452 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000104502 2024-09-21T14:09:05.452 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.453 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.453 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:09:05.453 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.503813+0000", 2024-09-21T14:09:05.453 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.010043557999999999 2024-09-21T14:09:05.453 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.453 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.453 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:05.453 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.504820+0000", 2024-09-21T14:09:05.453 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0010069899999999999 2024-09-21T14:09:05.453 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.453 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.453 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:05.453 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.504875+0000", 2024-09-21T14:09:05.453 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5282999999999999e-05 2024-09-21T14:09:05.454 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.454 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.454 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:05.454 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.504974+0000", 2024-09-21T14:09:05.454 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.8987000000000006e-05 2024-09-21T14:09:05.454 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.454 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.454 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:09:05.454 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.505032+0000", 2024-09-21T14:09:05.454 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7970000000000002e-05 2024-09-21T14:09:05.454 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:05.454 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:05.454 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:09:05.454 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.505184+0000", 2024-09-21T14:09:05.455 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015151800000000001 2024-09-21T14:09:05.455 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:05.455 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:09:05.455 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:05.455 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:05.455 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:09:05.455 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T14:09:05.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:05.456 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:05.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:09:05.557 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:05.594 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:05.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:05.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:05.695 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:05.724 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:05.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:05.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:05.826 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:05.855 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:05.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:05.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:05.957 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:05.987 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:05.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:05.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:06.088 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:09:06.125 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:06.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:06.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:09:06.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:05 smithi187 ceph-mon[33032]: pgmap v1031: 168 pgs: 5 peering, 3 active+remapped+backfilling, 160 active+clean; 5.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 4.0 KiB/s rd, 18 KiB/s wr, 748 op/s; 7094/258309 objects misplaced (2.746%); 59 MiB/s, 937 objects/s recovering 2024-09-21T14:09:06.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:05 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'57323'}"}]': finished 2024-09-21T14:09:06.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:05 smithi187 ceph-mon[33032]: osdmap e515: 8 total, 8 up, 8 in 2024-09-21T14:09:06.226 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:09:06.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:05 smithi049 ceph-mon[40489]: pgmap v1031: 168 pgs: 5 peering, 3 active+remapped+backfilling, 160 active+clean; 5.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 4.0 KiB/s rd, 18 KiB/s wr, 748 op/s; 7094/258309 objects misplaced (2.746%); 59 MiB/s, 937 objects/s recovering 2024-09-21T14:09:06.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:05 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'57323'}"}]': finished 2024-09-21T14:09:06.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:05 smithi049 ceph-mon[40489]: osdmap e515: 8 total, 8 up, 8 in 2024-09-21T14:09:06.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:05 smithi049 ceph-mon[30080]: pgmap v1031: 168 pgs: 5 peering, 3 active+remapped+backfilling, 160 active+clean; 5.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 4.0 KiB/s rd, 18 KiB/s wr, 748 op/s; 7094/258309 objects misplaced (2.746%); 59 MiB/s, 937 objects/s recovering 2024-09-21T14:09:06.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:05 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'57323'}"}]': finished 2024-09-21T14:09:06.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:05 smithi049 ceph-mon[30080]: osdmap e515: 8 total, 8 up, 8 in 2024-09-21T14:09:06.270 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:06.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:06.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:09:06.372 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:06.393 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:06.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:06.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:06.495 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:06.521 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T14:09:06.536 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:06.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:06.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:06.638 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:06.673 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:06.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:06.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:06.775 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:09:06.828 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:06.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:06.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:09:06.833 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T14:09:06.930 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:09:06.963 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:06.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:06.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:09:07.065 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:09:07.096 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:07.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:07.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:09:07.198 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:07.223 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:07.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:07.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:07.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:06 smithi049 ceph-mon[40489]: osdmap e516: 8 total, 8 up, 8 in 2024-09-21T14:09:07.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:06 smithi049 ceph-mon[30080]: osdmap e516: 8 total, 8 up, 8 in 2024-09-21T14:09:07.324 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:07.345 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:07.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:07.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:07.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:06 smithi187 ceph-mon[33032]: osdmap e516: 8 total, 8 up, 8 in 2024-09-21T14:09:07.446 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:07.476 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:07.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:07.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:07.578 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:07.606 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:07.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:07.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:07.708 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:07.731 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:07.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:07.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:07.833 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:07.865 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:07.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:07.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:07.967 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:09:08.006 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:08.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:08.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:09:08.108 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:09:08.144 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:08.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:08.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:09:08.245 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:08.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:07 smithi049 ceph-mon[40489]: pgmap v1034: 168 pgs: 1 peering, 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 162 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 7.7 KiB/s rd, 37 KiB/s wr, 824 op/s; 15945/251895 objects misplaced (6.330%); 66 MiB/s, 1.06k objects/s recovering 2024-09-21T14:09:08.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:07 smithi049 ceph-mon[40489]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T14:09:08.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:07 smithi049 ceph-mon[30080]: pgmap v1034: 168 pgs: 1 peering, 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 162 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 7.7 KiB/s rd, 37 KiB/s wr, 824 op/s; 15945/251895 objects misplaced (6.330%); 66 MiB/s, 1.06k objects/s recovering 2024-09-21T14:09:08.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:07 smithi049 ceph-mon[30080]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T14:09:08.288 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:08.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:08.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:08.390 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:09:08.422 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:08.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:08.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:09:08.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:07 smithi187 ceph-mon[33032]: pgmap v1034: 168 pgs: 1 peering, 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 162 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 7.7 KiB/s rd, 37 KiB/s wr, 824 op/s; 15945/251895 objects misplaced (6.330%); 66 MiB/s, 1.06k objects/s recovering 2024-09-21T14:09:08.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:07 smithi187 ceph-mon[33032]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-21T14:09:08.523 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:09:08.557 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:08.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:08.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:09:08.659 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:09:08.679 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:08.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:08.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:09:08.781 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:08.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:08.810 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:08.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:08.911 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:08.940 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:08.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:08.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:09.041 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:09:09.062 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:09.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:09.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:09:09.164 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:09:09.192 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:09.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:09.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:09:09.294 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:09:09.335 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:09.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:09.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:09:09.437 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:09:09.466 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:09.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:09.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:09:09.568 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:09:09.609 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:09.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:09.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:09:09.711 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:09:09.771 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:09.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:09.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:09:09.873 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:09:09.909 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:09.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:09.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:09:10.011 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:09:10.044 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:10.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:10.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:09:10.145 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:09:10.186 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:10.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:10.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:09:10.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:09 smithi049 ceph-mon[40489]: pgmap v1035: 168 pgs: 1 peering, 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 162 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 7.3 KiB/s rd, 25 KiB/s wr, 654 op/s; 15945/250221 objects misplaced (6.372%); 45 MiB/s, 717 objects/s recovering 2024-09-21T14:09:10.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:09 smithi049 ceph-mon[30080]: pgmap v1035: 168 pgs: 1 peering, 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 162 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 7.3 KiB/s rd, 25 KiB/s wr, 654 op/s; 15945/250221 objects misplaced (6.372%); 45 MiB/s, 717 objects/s recovering 2024-09-21T14:09:10.288 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:09:10.322 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:10.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:10.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:09:10.423 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:09:10.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:09 smithi187 ceph-mon[33032]: pgmap v1035: 168 pgs: 1 peering, 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 162 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 7.3 KiB/s rd, 25 KiB/s wr, 654 op/s; 15945/250221 objects misplaced (6.372%); 45 MiB/s, 717 objects/s recovering 2024-09-21T14:09:10.444 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:09:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:09:10.464 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:10.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:10.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:09:10.566 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:10.610 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:10.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:10.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:10.712 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:10.754 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:09:10 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:09:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:09:10.757 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:10.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:10.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:10.859 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:09:10.891 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:10.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:10.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:09:10.992 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:09:11.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:11.042 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:11.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:09:11.143 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:11.182 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:11.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:11.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:11.284 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:09:11.306 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:11.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:11.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:09:11.408 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:09:11.440 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:11.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:11.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:09:11.542 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:09:11.584 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:11.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:11.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:09:11.686 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:09:11.717 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:11.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:11.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:09:11.819 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:09:11.849 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:11.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:11.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:09:11.951 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:09:11.980 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:11.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:11.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:09:12.082 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:09:12.118 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:12.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:12.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:09:12.220 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:12.247 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:11 smithi187 ceph-mon[33032]: pgmap v1036: 168 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 4.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 48 KiB/s rd, 24 KiB/s wr, 1.04k op/s; 13480/243078 objects misplaced (5.546%); 61 MiB/s, 978 objects/s recovering 2024-09-21T14:09:12.247 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:11 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:09:12.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:11 smithi049 ceph-mon[40489]: pgmap v1036: 168 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 4.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 48 KiB/s rd, 24 KiB/s wr, 1.04k op/s; 13480/243078 objects misplaced (5.546%); 61 MiB/s, 978 objects/s recovering 2024-09-21T14:09:12.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:12 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:09:12.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:11 smithi049 ceph-mon[30080]: pgmap v1036: 168 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 4.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 48 KiB/s rd, 24 KiB/s wr, 1.04k op/s; 13480/243078 objects misplaced (5.546%); 61 MiB/s, 978 objects/s recovering 2024-09-21T14:09:12.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:12 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:09:12.263 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:12.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:12.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:12.366 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:09:12.404 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:12.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:12.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:09:12.505 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:09:12.544 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:12.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:12.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:09:12.646 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:12.677 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:12.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:12.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:12.779 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:09:12.801 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:12.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:12.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:09:12.903 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:09:12.926 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:12.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:12.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:09:13.028 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:09:13.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:13.072 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:13.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:09:13.172 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:09:13.207 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:13.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:13.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:09:13.217 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-21T14:09:13.218 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd reweight-by-utilization 1000 1.0 2024-09-21T14:09:13.309 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:09:13.362 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:13.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:13.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:09:13.464 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:13.496 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:13.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:13.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:13.598 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:09:13.631 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:13.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:13.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:09:13.733 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:13.769 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:13.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:13.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:13.871 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:13.906 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:13.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:13.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:14.008 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:14.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:14.057 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:14.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:14.158 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:09:14.202 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:14.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:14.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:09:14.232 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:13 smithi049 ceph-mon[40489]: pgmap v1037: 168 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 4.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 55 KiB/s rd, 9.2 KiB/s wr, 1.01k op/s; 12583/236244 objects misplaced (5.326%); 32 MiB/s, 512 objects/s recovering 2024-09-21T14:09:14.232 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:13 smithi049 ceph-mon[30080]: pgmap v1037: 168 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 4.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 55 KiB/s rd, 9.2 KiB/s wr, 1.01k op/s; 12583/236244 objects misplaced (5.326%); 32 MiB/s, 512 objects/s recovering 2024-09-21T14:09:14.292 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:09:14.304 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:14.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:14.333 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:14.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:14.433 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:09:14.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:13 smithi187 ceph-mon[33032]: pgmap v1037: 168 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 4.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 55 KiB/s rd, 9.2 KiB/s wr, 1.01k op/s; 12583/236244 objects misplaced (5.326%); 32 MiB/s, 512 objects/s recovering 2024-09-21T14:09:14.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:14.490 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:14.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:09:14.591 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:09:14.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:14.635 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:14.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:09:14.657 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:09:14.657 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:09:14.701 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:09:14.701 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:09:14.737 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:09:14.748 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:09:14.749 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:09:14.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:14.787 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:14.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:09:14.790 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:09:14.790 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:09:14.824 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:09:14.824 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:09:14.857 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:09:14.858 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:09:14.888 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:14.895 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:09:14.896 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:09:14.935 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:09:14.935 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:09:14.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:14.948 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:14.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:15.049 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:09:15.098 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:15.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:15.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:09:15.201 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:09:15.230 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:15.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:15.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:09:15.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:14 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1791164281' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:09:15.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:14 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:09:15.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:14 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:09:15.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:14 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1791164281' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:09:15.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:14 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:09:15.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:14 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:09:15.333 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:09:15.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:15.378 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:15.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:09:15.418 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:09:15.419 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:09:15.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:14 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1791164281' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:09:15.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:14 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:09:15.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:14 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:09:15.479 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:15.509 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:15.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:15.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:15.611 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:09:15.655 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:15.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:15.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:09:15.757 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:15.796 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:15.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:15.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:15.898 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:15.928 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:15.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:15.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:15.996 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-09-21T14:09:16.030 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:09:16.060 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:16.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:16.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:09:16.162 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:16.200 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:16.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:16.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:16.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:16 smithi049 ceph-mon[30080]: pgmap v1038: 168 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 4.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 48 KiB/s rd, 0 B/s wr, 997 op/s; 11125/230877 objects misplaced (4.819%); 30 MiB/s, 485 objects/s recovering 2024-09-21T14:09:16.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:16 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3157652978' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:09:16.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:16 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:09:16.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:16 smithi049 ceph-mon[40489]: pgmap v1038: 168 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 4.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 48 KiB/s rd, 0 B/s wr, 997 op/s; 11125/230877 objects misplaced (4.819%); 30 MiB/s, 485 objects/s recovering 2024-09-21T14:09:16.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:16 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3157652978' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:09:16.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:16 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:09:16.302 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:16.329 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:16.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:16.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:16.431 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:16.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:16 smithi187 ceph-mon[33032]: pgmap v1038: 168 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 4.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 48 KiB/s rd, 0 B/s wr, 997 op/s; 11125/230877 objects misplaced (4.819%); 30 MiB/s, 485 objects/s recovering 2024-09-21T14:09:16.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:16 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3157652978' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:09:16.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:16 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:09:16.462 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:16.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:16.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:16.564 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:09:16.596 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:16.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:16.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:09:16.697 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:09:16.722 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:16.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:16.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:09:16.824 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:16.858 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:16.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:16.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:16.959 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:17.000 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:17.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:17.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:17.101 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:09:17.137 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:17.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:17.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:09:17.239 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:17.279 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:17.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:17.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:17.311 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:17 smithi049 ceph-mon[40489]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:09:17.311 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:17 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T14:09:17.311 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:17 smithi049 ceph-mon[40489]: osdmap e517: 8 total, 8 up, 8 in 2024-09-21T14:09:17.311 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:17 smithi049 ceph-mon[30080]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:09:17.311 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:17 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T14:09:17.312 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:17 smithi049 ceph-mon[30080]: osdmap e517: 8 total, 8 up, 8 in 2024-09-21T14:09:17.381 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:09:17.409 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:17.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:17.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:09:17.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:17 smithi187 ceph-mon[33032]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:09:17.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:17 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T14:09:17.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:17 smithi187 ceph-mon[33032]: osdmap e517: 8 total, 8 up, 8 in 2024-09-21T14:09:17.511 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:09:17.541 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:17.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:17.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:09:17.642 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:17.689 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:17.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:17.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:17.791 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:17.818 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:17.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:17.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:17.908 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T14:09:17.908 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:09:17.920 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:09:17.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:17.961 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:17.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:09:18.062 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:18.103 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:18.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:18.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:18.205 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:09:18.247 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:18.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:18.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:09:18.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:18 smithi049 ceph-mon[40489]: pgmap v1040: 168 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 4.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 43 KiB/s rd, 0 B/s wr, 1.01k op/s; 9698/223656 objects misplaced (4.336%); 29 MiB/s, 471 objects/s recovering 2024-09-21T14:09:18.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:18 smithi049 ceph-mon[30080]: pgmap v1040: 168 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 4.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 43 KiB/s rd, 0 B/s wr, 1.01k op/s; 9698/223656 objects misplaced (4.336%); 29 MiB/s, 471 objects/s recovering 2024-09-21T14:09:18.349 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:18.383 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:18.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:18.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:18.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:18 smithi187 ceph-mon[33032]: pgmap v1040: 168 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 4.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 43 KiB/s rd, 0 B/s wr, 1.01k op/s; 9698/223656 objects misplaced (4.336%); 29 MiB/s, 471 objects/s recovering 2024-09-21T14:09:18.486 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:09:18.526 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:18.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:18.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:09:18.628 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:18.663 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:18.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:18.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:18.767 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:09:18.776 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T14:09:18.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:18.807 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:18.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:09:18.908 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:18.936 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:18.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:18.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:19.033 INFO:teuthology.orchestra.run.smithi049.stdout:moved 12 / 504 (2.38095%) 2024-09-21T14:09:19.033 INFO:teuthology.orchestra.run.smithi049.stdout:avg 63 2024-09-21T14:09:19.033 INFO:teuthology.orchestra.run.smithi049.stdout:stddev 16.5151 -> 15.9687 (expected baseline 7.42462) 2024-09-21T14:09:19.033 INFO:teuthology.orchestra.run.smithi049.stdout:min osd.3 with 24 -> 24 pgs (0.380952 -> 0.380952 * mean) 2024-09-21T14:09:19.033 INFO:teuthology.orchestra.run.smithi049.stdout:max osd.1 with 78 -> 77 pgs (1.2381 -> 1.22222 * mean) 2024-09-21T14:09:19.033 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T14:09:19.033 INFO:teuthology.orchestra.run.smithi049.stdout:oload 1000 2024-09-21T14:09:19.033 INFO:teuthology.orchestra.run.smithi049.stdout:max_change 1 2024-09-21T14:09:19.033 INFO:teuthology.orchestra.run.smithi049.stdout:max_change_osds 4 2024-09-21T14:09:19.033 INFO:teuthology.orchestra.run.smithi049.stdout:average_utilization 0.0290 2024-09-21T14:09:19.033 INFO:teuthology.orchestra.run.smithi049.stdout:overload_utilization 0.2898 2024-09-21T14:09:19.033 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0 weight 0.8747 -> 1.0000 2024-09-21T14:09:19.038 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:19.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:19.105 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:19.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:19.206 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:09:19.244 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:19.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:19.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:09:19.279 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:19 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'65536'}"} : dispatch 2024-09-21T14:09:19.279 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:19 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'65536'}"} : dispatch 2024-09-21T14:09:19.347 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:09:19.396 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:19.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:19.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:09:19.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:19 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'65536'}"} : dispatch 2024-09-21T14:09:19.498 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:09:19.526 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:19.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:19.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:09:19.629 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:19.657 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:19.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:19.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:19.758 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:09:19.781 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:19.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:19.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:09:19.858 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T14:09:19.882 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:09:19.930 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:19.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:19.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:09:20.031 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:20.050 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-09-21T14:09:20.051 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd test-reweight-by-utilization 1000 3.0 2024-09-21T14:09:20.058 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:20.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:20.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:20.160 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:09:20.163 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:09:20.196 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:20.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:20.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:09:20.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:20 smithi049 ceph-mon[40489]: from='client.26582 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T14:09:20.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:20 smithi049 ceph-mon[40489]: pgmap v1041: 168 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 4.5 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 1.01k op/s; 9698/221841 objects misplaced (4.372%); 29 MiB/s, 471 objects/s recovering 2024-09-21T14:09:20.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:20 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'65536'}"}]': finished 2024-09-21T14:09:20.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:20 smithi049 ceph-mon[40489]: osdmap e518: 8 total, 8 up, 8 in 2024-09-21T14:09:20.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:20 smithi049 ceph-mon[30080]: from='client.26582 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T14:09:20.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:20 smithi049 ceph-mon[30080]: pgmap v1041: 168 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 4.5 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 1.01k op/s; 9698/221841 objects misplaced (4.372%); 29 MiB/s, 471 objects/s recovering 2024-09-21T14:09:20.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:20 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'65536'}"}]': finished 2024-09-21T14:09:20.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:20 smithi049 ceph-mon[30080]: osdmap e518: 8 total, 8 up, 8 in 2024-09-21T14:09:20.298 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:20.321 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:20.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:20.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:20.423 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:20.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:20 smithi187 ceph-mon[33032]: from='client.26582 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T14:09:20.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:20 smithi187 ceph-mon[33032]: pgmap v1041: 168 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 4.5 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 1.01k op/s; 9698/221841 objects misplaced (4.372%); 29 MiB/s, 471 objects/s recovering 2024-09-21T14:09:20.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:20 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'65536'}"}]': finished 2024-09-21T14:09:20.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:20 smithi187 ceph-mon[33032]: osdmap e518: 8 total, 8 up, 8 in 2024-09-21T14:09:20.444 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:09:20 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:09:20.446 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:20.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:20.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:20.550 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:09:20.580 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:20.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:20.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:09:20.682 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:20.705 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:20.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:20.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:20.791 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:09:20.807 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:09:20.844 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:20.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:20.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:09:20.848 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:09:20.871 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T14:09:20.871 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:09:20.871 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:09:20 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:09:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:09:20.946 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:09:20.975 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:20.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:20.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:09:21.077 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:21.097 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:21.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:21.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:21.199 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:21.230 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:21.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:21.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:21.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:21 smithi049 ceph-mon[30080]: osdmap e519: 8 total, 8 up, 8 in 2024-09-21T14:09:21.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:21 smithi049 ceph-mon[30080]: osdmap e520: 8 total, 8 up, 8 in 2024-09-21T14:09:21.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:21 smithi049 ceph-mon[40489]: osdmap e519: 8 total, 8 up, 8 in 2024-09-21T14:09:21.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:21 smithi049 ceph-mon[40489]: osdmap e520: 8 total, 8 up, 8 in 2024-09-21T14:09:21.327 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T14:09:21.332 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:09:21.365 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:21.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:21.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:09:21.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:21 smithi187 ceph-mon[33032]: osdmap e519: 8 total, 8 up, 8 in 2024-09-21T14:09:21.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:21 smithi187 ceph-mon[33032]: osdmap e520: 8 total, 8 up, 8 in 2024-09-21T14:09:21.470 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:09:21.509 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:21.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:21.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:09:21.610 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:09:21.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:21.643 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:21.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:09:21.743 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:21.781 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:21.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:21.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:21.883 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:09:21.941 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:21.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:21.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:09:22.043 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:22.086 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:22.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:22.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:22.188 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:22.228 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:22.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:22.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:22.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:22 smithi049 ceph-mon[40489]: pgmap v1044: 168 pgs: 8 peering, 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 155 active+clean; 4.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 682 B/s rd, 0 B/s wr, 919 op/s; 7373/214332 objects misplaced (3.440%); 31 MiB/s, 492 objects/s recovering 2024-09-21T14:09:22.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:22 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:09:22.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:22 smithi049 ceph-mon[40489]: 11.8 starting backfill to osd.0 from (0'0,0'0] MAX to 517'10900 2024-09-21T14:09:22.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:22 smithi049 ceph-mon[40489]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T14:09:22.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:22 smithi049 ceph-mon[30080]: pgmap v1044: 168 pgs: 8 peering, 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 155 active+clean; 4.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 682 B/s rd, 0 B/s wr, 919 op/s; 7373/214332 objects misplaced (3.440%); 31 MiB/s, 492 objects/s recovering 2024-09-21T14:09:22.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:22 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:09:22.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:22 smithi049 ceph-mon[30080]: 11.8 starting backfill to osd.0 from (0'0,0'0] MAX to 517'10900 2024-09-21T14:09:22.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:22 smithi049 ceph-mon[30080]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T14:09:22.330 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:22.366 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:22.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:22.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:22.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:22 smithi187 ceph-mon[33032]: pgmap v1044: 168 pgs: 8 peering, 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 155 active+clean; 4.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 682 B/s rd, 0 B/s wr, 919 op/s; 7373/214332 objects misplaced (3.440%); 31 MiB/s, 492 objects/s recovering 2024-09-21T14:09:22.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:22 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:09:22.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:22 smithi187 ceph-mon[33032]: 11.8 starting backfill to osd.0 from (0'0,0'0] MAX to 517'10900 2024-09-21T14:09:22.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:22 smithi187 ceph-mon[33032]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T14:09:22.468 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:09:22.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:22.495 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:22.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:09:22.595 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:09:22.623 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:22.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:22.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:09:22.725 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:22.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:22.758 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:22.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:22.859 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:22.896 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:22.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:22.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:22.998 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:23.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:23.031 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:23.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:23.132 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:09:23.167 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:23.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:23.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:09:23.269 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:23.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:23.300 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:23.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:23.367 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:23 smithi187 ceph-mon[33032]: pgmap v1046: 168 pgs: 8 peering, 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 155 active+clean; 4.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 682 B/s rd, 0 B/s wr, 844 op/s; 6334/208461 objects misplaced (3.038%); 29 MiB/s, 470 objects/s recovering 2024-09-21T14:09:23.401 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:23.431 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:23.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:23.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:23.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:23 smithi049 ceph-mon[40489]: pgmap v1046: 168 pgs: 8 peering, 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 155 active+clean; 4.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 682 B/s rd, 0 B/s wr, 844 op/s; 6334/208461 objects misplaced (3.038%); 29 MiB/s, 470 objects/s recovering 2024-09-21T14:09:23.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:23 smithi049 ceph-mon[30080]: pgmap v1046: 168 pgs: 8 peering, 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 155 active+clean; 4.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 682 B/s rd, 0 B/s wr, 844 op/s; 6334/208461 objects misplaced (3.038%); 29 MiB/s, 470 objects/s recovering 2024-09-21T14:09:23.533 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:09:23.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:23.569 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:23.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:09:23.670 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:09:23.710 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:23.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:23.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:09:23.812 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:09:23.832 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:23.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:23.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:09:23.934 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:09:23.975 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:23.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:23.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:09:24.077 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:09:24.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:24.114 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:24.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:09:24.215 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:24.253 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:24.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:24.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:24.356 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:09:24.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:24.398 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:24.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:09:24.499 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:09:24.527 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:24.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:24.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:09:24.629 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:09:24.661 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:24.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:24.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:09:24.763 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:09:24.783 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:24.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:24.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:09:24.884 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:24.928 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:24.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:24.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:25.030 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:25.070 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:25.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:25.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:25.172 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:25.201 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:25.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:25.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:25.303 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:25.329 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:25.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:25.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:25.431 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:25.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:25.461 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:25.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:25.494 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:25 smithi049 ceph-mon[40489]: pgmap v1047: 168 pgs: 8 peering, 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 155 active+clean; 4.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 682 B/s rd, 0 B/s wr, 940 op/s; 8714/204924 objects misplaced (4.252%); 47 MiB/s, 749 objects/s recovering 2024-09-21T14:09:25.494 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:25 smithi049 ceph-mon[30080]: pgmap v1047: 168 pgs: 8 peering, 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 155 active+clean; 4.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 682 B/s rd, 0 B/s wr, 940 op/s; 8714/204924 objects misplaced (4.252%); 47 MiB/s, 749 objects/s recovering 2024-09-21T14:09:25.562 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:09:25.582 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:25.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:25.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:09:25.684 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:09:25.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:25 smithi187 ceph-mon[33032]: pgmap v1047: 168 pgs: 8 peering, 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 155 active+clean; 4.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 682 B/s rd, 0 B/s wr, 940 op/s; 8714/204924 objects misplaced (4.252%); 47 MiB/s, 749 objects/s recovering 2024-09-21T14:09:25.725 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:25.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:25.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:09:25.827 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:09:25.849 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:25.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:25.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:09:25.950 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:25.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:25.983 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:25.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:26.084 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:26.118 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:26.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:26.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:26.219 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:26.264 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:26.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:26.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:26.366 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:26.399 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:26.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:26.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:26.501 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:09:26.539 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:26.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:26.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:09:26.592 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:26 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/4112826531' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:09:26.593 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:26 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/4112826531' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:09:26.640 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:09:26.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:26.682 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:26.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:09:26.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:26 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/4112826531' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:09:26.783 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:09:26.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:26.819 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:26.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:09:26.920 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:09:26.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:26.948 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:26.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:09:27.049 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:27.074 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:27.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:27.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:27.176 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:27.212 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:27.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:27.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:27.314 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:09:27.354 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:27.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:27.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:09:27.456 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:09:27.488 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:27.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:27.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:09:27.589 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:09:27.614 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:27 smithi049 ceph-mon[40489]: pgmap v1048: 168 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 4.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 809 op/s; 6417/198843 objects misplaced (3.227%); 49 MiB/s, 790 objects/s recovering 2024-09-21T14:09:27.614 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:27 smithi049 ceph-mon[40489]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-21T14:09:27.616 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:27 smithi049 ceph-mon[30080]: pgmap v1048: 168 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 4.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 809 op/s; 6417/198843 objects misplaced (3.227%); 49 MiB/s, 790 objects/s recovering 2024-09-21T14:09:27.616 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:27 smithi049 ceph-mon[30080]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-21T14:09:27.619 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:27.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:27.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:09:27.653 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T14:09:27.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:27 smithi187 ceph-mon[33032]: pgmap v1048: 168 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 4.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 809 op/s; 6417/198843 objects misplaced (3.227%); 49 MiB/s, 790 objects/s recovering 2024-09-21T14:09:27.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:27 smithi187 ceph-mon[33032]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-21T14:09:27.721 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:09:27.739 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:09:27.739 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:09:27.739 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:09:27.739 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:09:27.739 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:09:27.740 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:09:27.740 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:09:27.740 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:09:27.740 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:09:27.740 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:09:27.740 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:09:27.740 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:09:27.740 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:09:27.740 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:09:27.740 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:09:27.740 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:09:27.767 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:27.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:27.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:09:27.869 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:09:27.891 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:27.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:27.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:09:27.987 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:09:27.993 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:28.016 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:28.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:28.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:28.118 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:09:28.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:28.141 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:28.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:09:28.242 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:28.265 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:28.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:28.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:28.366 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:28.400 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:28.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:28.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:28.420 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:28 smithi049 ceph-mon[30080]: osdmap e521: 8 total, 8 up, 8 in 2024-09-21T14:09:28.420 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:28 smithi049 ceph-mon[40489]: osdmap e521: 8 total, 8 up, 8 in 2024-09-21T14:09:28.503 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:28.536 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:28.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:28.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:28.638 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:28.689 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:28.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:28.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:28.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:28 smithi187 ceph-mon[33032]: osdmap e521: 8 total, 8 up, 8 in 2024-09-21T14:09:28.791 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:09:28.831 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:28.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:28.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:09:28.933 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:28.964 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:28.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:28.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:29.066 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:29.097 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:29.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:29.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:29.198 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:09:29.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:29.248 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:29.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:09:29.349 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:09:29.393 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:29.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:29.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:09:29.481 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:29 smithi049 ceph-mon[30080]: pgmap v1050: 168 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 4.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 729 op/s; 6417/196854 objects misplaced (3.260%); 37 MiB/s, 593 objects/s recovering 2024-09-21T14:09:29.482 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:29 smithi049 ceph-mon[30080]: osdmap e522: 8 total, 8 up, 8 in 2024-09-21T14:09:29.482 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:29 smithi049 ceph-mon[40489]: pgmap v1050: 168 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 4.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 729 op/s; 6417/196854 objects misplaced (3.260%); 37 MiB/s, 593 objects/s recovering 2024-09-21T14:09:29.482 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:29 smithi049 ceph-mon[40489]: osdmap e522: 8 total, 8 up, 8 in 2024-09-21T14:09:29.494 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:09:29.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:29.523 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:29.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:09:29.624 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:09:29.671 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:29.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:29.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:09:29.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:29 smithi187 ceph-mon[33032]: pgmap v1050: 168 pgs: 4 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 4.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 729 op/s; 6417/196854 objects misplaced (3.260%); 37 MiB/s, 593 objects/s recovering 2024-09-21T14:09:29.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:29 smithi187 ceph-mon[33032]: osdmap e522: 8 total, 8 up, 8 in 2024-09-21T14:09:29.773 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:29.811 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:29.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:29.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:29.912 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:29.934 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:29.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:29.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:30.035 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:09:30.075 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:30.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:30.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:09:30.177 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:30.212 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:30.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:30.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:30.255 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-09-21T14:09:30.314 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:09:30.347 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:30.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:30.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:09:30.449 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:09:30.478 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:30.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:30.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:09:30.481 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T14:09:30.545 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:30 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:09:30.545 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:30 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:09:30.546 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:30 smithi049 ceph-mon[40489]: osdmap e523: 8 total, 8 up, 8 in 2024-09-21T14:09:30.546 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:30 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [4, 2]} : dispatch 2024-09-21T14:09:30.546 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:30 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]} : dispatch 2024-09-21T14:09:30.546 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:30 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]} : dispatch 2024-09-21T14:09:30.546 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:30 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]} : dispatch 2024-09-21T14:09:30.546 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:30 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:09:30.546 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:30 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:09:30.546 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:30 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:09:30.546 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:30 smithi049 ceph-mon[40489]: from='client.26329 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T14:09:30.546 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:30 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1291829892' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:09:30.546 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:30 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:09:30.546 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:30 smithi049 ceph-mon[40489]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:09:30.547 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:30 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [4, 2]}]': finished 2024-09-21T14:09:30.547 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:30 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-09-21T14:09:30.547 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:30 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]': finished 2024-09-21T14:09:30.547 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:30 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-09-21T14:09:30.547 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:30 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T14:09:30.547 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:30 smithi049 ceph-mon[40489]: osdmap e524: 8 total, 8 up, 8 in 2024-09-21T14:09:30.547 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:30 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:09:30.547 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:30 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:09:30.547 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:30 smithi049 ceph-mon[30080]: osdmap e523: 8 total, 8 up, 8 in 2024-09-21T14:09:30.547 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:30 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [4, 2]} : dispatch 2024-09-21T14:09:30.547 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:30 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]} : dispatch 2024-09-21T14:09:30.548 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:30 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]} : dispatch 2024-09-21T14:09:30.548 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:30 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]} : dispatch 2024-09-21T14:09:30.548 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:30 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:09:30.548 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:30 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:09:30.548 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:30 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:09:30.548 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:30 smithi049 ceph-mon[30080]: from='client.26329 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T14:09:30.548 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:30 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1291829892' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:09:30.548 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:30 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:09:30.548 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:30 smithi049 ceph-mon[30080]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:09:30.548 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:30 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [4, 2]}]': finished 2024-09-21T14:09:30.548 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:30 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-09-21T14:09:30.548 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:30 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]': finished 2024-09-21T14:09:30.548 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:30 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-09-21T14:09:30.548 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:30 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T14:09:30.549 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:30 smithi049 ceph-mon[30080]: osdmap e524: 8 total, 8 up, 8 in 2024-09-21T14:09:30.580 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:09:30.603 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:30.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:30.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:09:30.694 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:09:30 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:09:30.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:30 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:09:30.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:30 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:09:30.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:30 smithi187 ceph-mon[33032]: osdmap e523: 8 total, 8 up, 8 in 2024-09-21T14:09:30.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:30 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [4, 2]} : dispatch 2024-09-21T14:09:30.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:30 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]} : dispatch 2024-09-21T14:09:30.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:30 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]} : dispatch 2024-09-21T14:09:30.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:30 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]} : dispatch 2024-09-21T14:09:30.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:30 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:09:30.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:30 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:09:30.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:30 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:09:30.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:30 smithi187 ceph-mon[33032]: from='client.26329 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T14:09:30.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:30 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1291829892' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:09:30.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:30 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:09:30.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:30 smithi187 ceph-mon[33032]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:09:30.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:30 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [4, 2]}]': finished 2024-09-21T14:09:30.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:30 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-09-21T14:09:30.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:30 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]': finished 2024-09-21T14:09:30.696 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:30 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-09-21T14:09:30.696 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:30 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T14:09:30.696 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:30 smithi187 ceph-mon[33032]: osdmap e524: 8 total, 8 up, 8 in 2024-09-21T14:09:30.705 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:30.727 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:30.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:30.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:30.829 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:30.852 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:30.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:30.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:30.922 INFO:teuthology.orchestra.run.smithi049.stdout:moved 0 / 504 (0%) 2024-09-21T14:09:30.923 INFO:teuthology.orchestra.run.smithi049.stdout:avg 63 2024-09-21T14:09:30.923 INFO:teuthology.orchestra.run.smithi049.stdout:stddev 15.9687 -> 15.9687 (expected baseline 7.42462) 2024-09-21T14:09:30.923 INFO:teuthology.orchestra.run.smithi049.stdout:min osd.3 with 24 -> 24 pgs (0.380952 -> 0.380952 * mean) 2024-09-21T14:09:30.923 INFO:teuthology.orchestra.run.smithi049.stdout:max osd.1 with 77 -> 77 pgs (1.22222 -> 1.22222 * mean) 2024-09-21T14:09:30.923 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T14:09:30.923 INFO:teuthology.orchestra.run.smithi049.stdout:oload 1000 2024-09-21T14:09:30.923 INFO:teuthology.orchestra.run.smithi049.stdout:max_change 3 2024-09-21T14:09:30.923 INFO:teuthology.orchestra.run.smithi049.stdout:max_change_osds 4 2024-09-21T14:09:30.923 INFO:teuthology.orchestra.run.smithi049.stdout:average_utilization 0.0267 2024-09-21T14:09:30.923 INFO:teuthology.orchestra.run.smithi049.stdout:overload_utilization 0.2666 2024-09-21T14:09:30.924 INFO:teuthology.orchestra.run.smithi049.stderr:no change 2024-09-21T14:09:30.924 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:09:30 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:09:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:09:30.954 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:09:30.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:30.977 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:30.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:09:31.078 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:31.101 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:31.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:31.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:31.202 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:09:31.225 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:31.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:31.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:09:31.326 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:31.355 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:31.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:31.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:31.433 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:09:31.456 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:09:31.486 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:31.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:31.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:09:31.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:31 smithi049 ceph-mon[30080]: pgmap v1053: 168 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 164 active+clean; 3.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 810 op/s; 4056/190353 objects misplaced (2.131%); 44 MiB/s, 709 objects/s recovering 2024-09-21T14:09:31.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:31 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:09:31.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:31 smithi049 ceph-mon[30080]: 3.0 scrub starts 2024-09-21T14:09:31.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:31 smithi049 ceph-mon[30080]: 3.0 scrub ok 2024-09-21T14:09:31.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:31 smithi049 ceph-mon[30080]: osdmap e525: 8 total, 8 up, 8 in 2024-09-21T14:09:31.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:31 smithi049 ceph-mon[40489]: pgmap v1053: 168 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 164 active+clean; 3.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 810 op/s; 4056/190353 objects misplaced (2.131%); 44 MiB/s, 709 objects/s recovering 2024-09-21T14:09:31.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:31 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:09:31.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:31 smithi049 ceph-mon[40489]: 3.0 scrub starts 2024-09-21T14:09:31.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:31 smithi049 ceph-mon[40489]: 3.0 scrub ok 2024-09-21T14:09:31.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:31 smithi049 ceph-mon[40489]: osdmap e525: 8 total, 8 up, 8 in 2024-09-21T14:09:31.589 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:31.611 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:31.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:31.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:31.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:31 smithi187 ceph-mon[33032]: pgmap v1053: 168 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 164 active+clean; 3.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 810 op/s; 4056/190353 objects misplaced (2.131%); 44 MiB/s, 709 objects/s recovering 2024-09-21T14:09:31.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:31 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:09:31.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:31 smithi187 ceph-mon[33032]: 3.0 scrub starts 2024-09-21T14:09:31.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:31 smithi187 ceph-mon[33032]: 3.0 scrub ok 2024-09-21T14:09:31.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:31 smithi187 ceph-mon[33032]: osdmap e525: 8 total, 8 up, 8 in 2024-09-21T14:09:31.713 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:31.738 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:31.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:31.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:31.839 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:31.862 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:31.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:31.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:31.964 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:09:31.986 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:31.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:31.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:09:32.029 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-09-21T14:09:32.029 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd test-reweight-by-utilization 1000 0.05 2024-09-21T14:09:32.088 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:09:32.110 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:32.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:32.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:09:32.212 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:09:32.236 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:32.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:32.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:09:32.338 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:09:32.373 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:32.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:32.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:09:32.474 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:09:32.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:32.531 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:32.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:09:32.566 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:09:32.632 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:09:32.672 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:32.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:32.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:09:32.687 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:32 smithi049 ceph-mon[40489]: 11.a scrub starts 2024-09-21T14:09:32.687 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:32 smithi049 ceph-mon[40489]: 4.19 scrub starts 2024-09-21T14:09:32.687 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:32 smithi049 ceph-mon[40489]: 4.19 scrub ok 2024-09-21T14:09:32.687 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:32 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:09:32.688 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:32 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:09:32.688 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:32 smithi049 ceph-mon[40489]: osdmap e526: 8 total, 8 up, 8 in 2024-09-21T14:09:32.688 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:32 smithi049 ceph-mon[30080]: 11.a scrub starts 2024-09-21T14:09:32.688 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:32 smithi049 ceph-mon[30080]: 4.19 scrub starts 2024-09-21T14:09:32.688 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:32 smithi049 ceph-mon[30080]: 4.19 scrub ok 2024-09-21T14:09:32.688 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:32 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:09:32.688 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:32 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:09:32.688 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:32 smithi049 ceph-mon[30080]: osdmap e526: 8 total, 8 up, 8 in 2024-09-21T14:09:32.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:32 smithi187 ceph-mon[33032]: 11.a scrub starts 2024-09-21T14:09:32.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:32 smithi187 ceph-mon[33032]: 4.19 scrub starts 2024-09-21T14:09:32.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:32 smithi187 ceph-mon[33032]: 4.19 scrub ok 2024-09-21T14:09:32.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:32 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:09:32.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:32 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:09:32.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:32 smithi187 ceph-mon[33032]: osdmap e526: 8 total, 8 up, 8 in 2024-09-21T14:09:32.774 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:09:32.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:32.819 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:32.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:09:32.833 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:09:32.920 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:09:32.964 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:32.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:32.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:09:33.066 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:09:33.102 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:33.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:33.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:09:33.204 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:33.239 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:33.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:33.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:33.341 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:09:33.385 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:33.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:33.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:09:33.487 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:09:33.500 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T14:09:33.500 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-09-21T14:09:33.500 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-09-21T14:09:33.500 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-09-21T14:09:33.501 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.501 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:70919 10.5 10:af948525:::benchmark_data_smithi049_176811_object70918:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-09-21T14:09:33.501 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:03:44.034223+0000", 2024-09-21T14:09:33.501 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 349.46270586600002, 2024-09-21T14:09:33.501 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.30270100300000002, 2024-09-21T14:09:33.501 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:09:33.501 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:09:33.501 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:09:33.501 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:09:33.501 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:09:33.501 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:09:33.501 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 70919 2024-09-21T14:09:33.501 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.502 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:09:33.502 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.502 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:09:33.502 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.034223+0000", 2024-09-21T14:09:33.502 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:33.502 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.502 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.502 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:09:33.502 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.034223+0000", 2024-09-21T14:09:33.502 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:33.502 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.502 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.502 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:09:33.502 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.034224+0000", 2024-09-21T14:09:33.503 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.525e-06 2024-09-21T14:09:33.503 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.503 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.503 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:09:33.503 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.034324+0000", 2024-09-21T14:09:33.503 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9767999999999998e-05 2024-09-21T14:09:33.503 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.503 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.503 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:09:33.503 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.034335+0000", 2024-09-21T14:09:33.503 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0789e-05 2024-09-21T14:09:33.503 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.503 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.504 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:09:33.504 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.034342+0000", 2024-09-21T14:09:33.504 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.08e-06 2024-09-21T14:09:33.504 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.504 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.504 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:09:33.504 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.035390+0000", 2024-09-21T14:09:33.504 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0010480789999999999 2024-09-21T14:09:33.504 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.504 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.504 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:09:33.504 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.035523+0000", 2024-09-21T14:09:33.504 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000132464 2024-09-21T14:09:33.504 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.505 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.505 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,3", 2024-09-21T14:09:33.505 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.035674+0000", 2024-09-21T14:09:33.505 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00015162300000000001 2024-09-21T14:09:33.505 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.505 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.505 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:33.505 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.040923+0000", 2024-09-21T14:09:33.505 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0052487920000000004 2024-09-21T14:09:33.505 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.505 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.505 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:09:33.505 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.040935+0000", 2024-09-21T14:09:33.505 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2167000000000001e-05 2024-09-21T14:09:33.505 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.505 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.506 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:33.506 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.336462+0000", 2024-09-21T14:09:33.506 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.29552672000000002 2024-09-21T14:09:33.506 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.506 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.506 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:09:33.506 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.336875+0000", 2024-09-21T14:09:33.506 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00041344400000000001 2024-09-21T14:09:33.506 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.506 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.506 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:09:33.506 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.336924+0000", 2024-09-21T14:09:33.506 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8551999999999999e-05 2024-09-21T14:09:33.506 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:33.507 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:09:33.507 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:33.507 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.507 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.507 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:70926 10.5 10:a9932ee5:::benchmark_data_smithi049_176811_object70925:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-09-21T14:09:33.507 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:03:44.039120+0000", 2024-09-21T14:09:33.507 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 349.45780840600003, 2024-09-21T14:09:33.507 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.30408621800000002, 2024-09-21T14:09:33.507 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:09:33.507 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:09:33.507 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:09:33.507 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:09:33.507 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:09:33.507 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:09:33.508 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 70926 2024-09-21T14:09:33.508 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.508 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:09:33.508 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.508 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:09:33.508 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.039120+0000", 2024-09-21T14:09:33.508 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:33.508 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.508 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.508 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:09:33.508 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.039120+0000", 2024-09-21T14:09:33.508 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:33.508 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.509 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.509 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:09:33.509 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.039121+0000", 2024-09-21T14:09:33.509 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.06e-07 2024-09-21T14:09:33.509 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.509 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.509 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:09:33.509 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.039843+0000", 2024-09-21T14:09:33.509 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00072204099999999998 2024-09-21T14:09:33.509 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.509 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.509 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:09:33.509 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.039869+0000", 2024-09-21T14:09:33.509 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6027999999999999e-05 2024-09-21T14:09:33.510 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.510 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.510 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:09:33.510 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.039876+0000", 2024-09-21T14:09:33.510 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9829999999999999e-06 2024-09-21T14:09:33.510 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.510 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.510 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:09:33.510 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.041641+0000", 2024-09-21T14:09:33.510 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0017650890000000001 2024-09-21T14:09:33.510 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.510 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.510 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:09:33.510 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.041772+0000", 2024-09-21T14:09:33.511 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000131066 2024-09-21T14:09:33.511 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.511 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.511 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,3", 2024-09-21T14:09:33.511 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.041907+0000", 2024-09-21T14:09:33.511 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013487800000000001 2024-09-21T14:09:33.511 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.511 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.511 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:09:33.511 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.053687+0000", 2024-09-21T14:09:33.511 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.011779947000000001 2024-09-21T14:09:33.511 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.511 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.512 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:33.512 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.056240+0000", 2024-09-21T14:09:33.512 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0025530079999999998 2024-09-21T14:09:33.512 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.512 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.512 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:33.512 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.343014+0000", 2024-09-21T14:09:33.512 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.286773575 2024-09-21T14:09:33.512 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.512 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.512 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:09:33.512 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.343155+0000", 2024-09-21T14:09:33.512 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014166299999999999 2024-09-21T14:09:33.513 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.513 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.513 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:09:33.513 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.343207+0000", 2024-09-21T14:09:33.513 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1334e-05 2024-09-21T14:09:33.513 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:33.513 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:09:33.513 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:33.513 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.513 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.513 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:70929 10.5 10:a6d68c9a:::benchmark_data_smithi049_176811_object70928:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-09-21T14:09:33.513 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:03:44.041499+0000", 2024-09-21T14:09:33.513 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 349.45543027399998, 2024-09-21T14:09:33.513 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.30182954299999998, 2024-09-21T14:09:33.514 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:09:33.514 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:09:33.514 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:09:33.514 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:09:33.514 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:09:33.514 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:09:33.514 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 70929 2024-09-21T14:09:33.514 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.514 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:09:33.514 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.514 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:09:33.514 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.041499+0000", 2024-09-21T14:09:33.514 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:33.515 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.515 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.515 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:09:33.515 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.041499+0000", 2024-09-21T14:09:33.515 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:33.515 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.515 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.515 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:09:33.515 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.041500+0000", 2024-09-21T14:09:33.515 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.778e-06 2024-09-21T14:09:33.515 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.515 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.515 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:09:33.515 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.041554+0000", 2024-09-21T14:09:33.516 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3501999999999997e-05 2024-09-21T14:09:33.516 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.516 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.516 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:09:33.516 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.041573+0000", 2024-09-21T14:09:33.516 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9210000000000001e-05 2024-09-21T14:09:33.516 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.516 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.516 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:09:33.516 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.041581+0000", 2024-09-21T14:09:33.516 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.276e-06 2024-09-21T14:09:33.516 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.516 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.516 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:09:33.517 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044856+0000", 2024-09-21T14:09:33.517 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0032744570000000001 2024-09-21T14:09:33.517 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.517 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.517 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:09:33.517 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044987+0000", 2024-09-21T14:09:33.517 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0001317 2024-09-21T14:09:33.517 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.517 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.517 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,3", 2024-09-21T14:09:33.517 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.045127+0000", 2024-09-21T14:09:33.517 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013962800000000001 2024-09-21T14:09:33.517 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.517 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.517 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:09:33.518 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.053696+0000", 2024-09-21T14:09:33.518 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0085692059999999994 2024-09-21T14:09:33.518 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.518 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.518 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:33.518 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.056300+0000", 2024-09-21T14:09:33.518 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0026036190000000002 2024-09-21T14:09:33.518 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.518 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.518 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:33.518 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.343249+0000", 2024-09-21T14:09:33.518 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.28694936300000001 2024-09-21T14:09:33.518 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.518 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.519 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:09:33.519 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.343296+0000", 2024-09-21T14:09:33.519 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6414e-05 2024-09-21T14:09:33.519 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.519 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.519 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:09:33.519 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.343328+0000", 2024-09-21T14:09:33.519 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.239e-05 2024-09-21T14:09:33.519 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:33.519 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:09:33.519 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:33.519 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.519 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.519 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:70932 10.8 10:1f467600:::benchmark_data_smithi049_176811_object70931:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-09-21T14:09:33.520 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:03:44.044077+0000", 2024-09-21T14:09:33.520 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 349.45285141599999, 2024-09-21T14:09:33.520 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.29252181500000002, 2024-09-21T14:09:33.520 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:09:33.520 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:09:33.520 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:09:33.520 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:09:33.520 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:09:33.520 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:09:33.520 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 70932 2024-09-21T14:09:33.520 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.520 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:09:33.520 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.520 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:09:33.520 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044077+0000", 2024-09-21T14:09:33.521 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:33.521 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.521 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.521 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:09:33.521 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044077+0000", 2024-09-21T14:09:33.521 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:33.521 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.521 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.521 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:09:33.521 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044079+0000", 2024-09-21T14:09:33.521 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3650000000000001e-06 2024-09-21T14:09:33.521 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.521 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.521 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:09:33.521 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044103+0000", 2024-09-21T14:09:33.522 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4343000000000001e-05 2024-09-21T14:09:33.522 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.522 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.522 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:09:33.522 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044120+0000", 2024-09-21T14:09:33.522 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6867999999999998e-05 2024-09-21T14:09:33.522 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.522 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.522 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:09:33.522 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044129+0000", 2024-09-21T14:09:33.522 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0180000000000007e-06 2024-09-21T14:09:33.522 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.522 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.522 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:09:33.522 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044198+0000", 2024-09-21T14:09:33.523 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9282999999999994e-05 2024-09-21T14:09:33.523 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.523 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.523 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:09:33.523 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044374+0000", 2024-09-21T14:09:33.523 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00017556900000000001 2024-09-21T14:09:33.523 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.523 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.523 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:09:33.523 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044544+0000", 2024-09-21T14:09:33.523 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000169977 2024-09-21T14:09:33.523 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.523 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.523 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:33.523 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.049643+0000", 2024-09-21T14:09:33.524 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.005099213 2024-09-21T14:09:33.524 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.524 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.524 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:09:33.524 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.049656+0000", 2024-09-21T14:09:33.524 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2839e-05 2024-09-21T14:09:33.524 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.524 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.524 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:33.524 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.336459+0000", 2024-09-21T14:09:33.524 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.28680339399999999 2024-09-21T14:09:33.524 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.524 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.524 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:09:33.525 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.336550+0000", 2024-09-21T14:09:33.525 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0919000000000001e-05 2024-09-21T14:09:33.525 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.525 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.525 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:09:33.525 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.336599+0000", 2024-09-21T14:09:33.525 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9026999999999997e-05 2024-09-21T14:09:33.525 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:33.525 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:09:33.525 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:33.525 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.525 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.526 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:70933 10.5 10:a4f74f69:::benchmark_data_smithi049_176811_object70932:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-09-21T14:09:33.526 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:03:44.044479+0000", 2024-09-21T14:09:33.526 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 349.45245008099999, 2024-09-21T14:09:33.526 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.311219415, 2024-09-21T14:09:33.526 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:09:33.526 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:09:33.526 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:09:33.526 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:09:33.526 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:09:33.526 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:09:33.526 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 70933 2024-09-21T14:09:33.526 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.526 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:09:33.527 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.527 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:09:33.527 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044479+0000", 2024-09-21T14:09:33.527 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:33.527 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.527 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.527 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:09:33.527 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044479+0000", 2024-09-21T14:09:33.527 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:33.527 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.527 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.528 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:09:33.528 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044479+0000", 2024-09-21T14:09:33.528 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6000000000000003e-07 2024-09-21T14:09:33.528 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.528 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.528 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:09:33.528 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044512+0000", 2024-09-21T14:09:33.528 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2669999999999997e-05 2024-09-21T14:09:33.528 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.528 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.529 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:09:33.529 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044521+0000", 2024-09-21T14:09:33.529 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2639999999999997e-06 2024-09-21T14:09:33.529 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.529 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.529 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:09:33.529 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044528+0000", 2024-09-21T14:09:33.529 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3400000000000003e-06 2024-09-21T14:09:33.529 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.529 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.529 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:09:33.529 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.048058+0000", 2024-09-21T14:09:33.529 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0035303330000000001 2024-09-21T14:09:33.529 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.530 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.530 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:09:33.530 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.048196+0000", 2024-09-21T14:09:33.530 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013794600000000001 2024-09-21T14:09:33.530 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.530 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.530 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,3", 2024-09-21T14:09:33.530 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.048338+0000", 2024-09-21T14:09:33.530 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014244099999999999 2024-09-21T14:09:33.530 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.530 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.530 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:09:33.530 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.056197+0000", 2024-09-21T14:09:33.530 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0078581610000000007 2024-09-21T14:09:33.531 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.531 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.531 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:33.531 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.059674+0000", 2024-09-21T14:09:33.531 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.003477258 2024-09-21T14:09:33.531 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.531 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.531 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:33.531 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.353631+0000", 2024-09-21T14:09:33.531 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.29395722400000002 2024-09-21T14:09:33.531 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.531 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.531 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:09:33.532 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.355638+0000", 2024-09-21T14:09:33.532 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0020068260000000002 2024-09-21T14:09:33.532 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.532 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.532 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:09:33.532 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.355698+0000", 2024-09-21T14:09:33.532 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0192000000000002e-05 2024-09-21T14:09:33.532 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:33.532 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:09:33.532 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:33.532 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.532 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.532 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:70934 10.5 10:af8b5e6c:::benchmark_data_smithi049_176811_object70933:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-09-21T14:09:33.532 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:03:44.044765+0000", 2024-09-21T14:09:33.532 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 349.45216355500003, 2024-09-21T14:09:33.533 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.311083428, 2024-09-21T14:09:33.533 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:09:33.533 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:09:33.533 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:09:33.533 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:09:33.533 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:09:33.533 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:09:33.533 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 70934 2024-09-21T14:09:33.533 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.533 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:09:33.533 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.533 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:09:33.533 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044765+0000", 2024-09-21T14:09:33.534 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:33.534 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.534 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.534 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:09:33.534 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044765+0000", 2024-09-21T14:09:33.534 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:33.534 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.534 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.534 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:09:33.534 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044766+0000", 2024-09-21T14:09:33.534 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9800000000000003e-07 2024-09-21T14:09:33.534 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.534 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.535 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:09:33.535 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044788+0000", 2024-09-21T14:09:33.535 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2226999999999999e-05 2024-09-21T14:09:33.535 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.535 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.535 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:09:33.535 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044798+0000", 2024-09-21T14:09:33.535 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4429999999999992e-06 2024-09-21T14:09:33.535 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.535 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.535 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:09:33.535 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044804+0000", 2024-09-21T14:09:33.535 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4320000000000004e-06 2024-09-21T14:09:33.536 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.536 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.536 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:09:33.536 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.050867+0000", 2024-09-21T14:09:33.536 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0060626739999999997 2024-09-21T14:09:33.536 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.536 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.536 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:09:33.536 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.051028+0000", 2024-09-21T14:09:33.536 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016143200000000001 2024-09-21T14:09:33.536 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.536 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.536 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,3", 2024-09-21T14:09:33.536 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.051335+0000", 2024-09-21T14:09:33.537 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00030696199999999998 2024-09-21T14:09:33.537 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.537 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.537 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:09:33.537 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.056260+0000", 2024-09-21T14:09:33.537 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0049247759999999996 2024-09-21T14:09:33.537 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.537 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.537 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:33.537 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.059715+0000", 2024-09-21T14:09:33.537 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0034552739999999999 2024-09-21T14:09:33.537 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.538 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.538 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:33.538 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.355755+0000", 2024-09-21T14:09:33.538 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.29603972699999997 2024-09-21T14:09:33.538 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.538 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.538 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:09:33.538 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.355812+0000", 2024-09-21T14:09:33.538 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7420000000000003e-05 2024-09-21T14:09:33.538 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.538 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.538 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:09:33.539 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.355849+0000", 2024-09-21T14:09:33.539 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6362999999999997e-05 2024-09-21T14:09:33.539 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:33.539 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:09:33.539 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:33.539 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.539 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.539 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:70935 10.5 10:a1fdc64d:::benchmark_data_smithi049_176811_object70934:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-09-21T14:09:33.539 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:03:44.044851+0000", 2024-09-21T14:09:33.539 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 349.452077325, 2024-09-21T14:09:33.539 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.31126390999999998, 2024-09-21T14:09:33.539 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:09:33.539 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:09:33.539 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:09:33.540 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:09:33.540 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:09:33.540 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:09:33.540 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 70935 2024-09-21T14:09:33.540 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.540 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:09:33.540 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.540 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:09:33.540 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044851+0000", 2024-09-21T14:09:33.540 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:33.540 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.540 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.540 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:09:33.540 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044851+0000", 2024-09-21T14:09:33.540 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:33.541 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.541 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.541 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:09:33.541 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044852+0000", 2024-09-21T14:09:33.541 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0800000000000004e-07 2024-09-21T14:09:33.541 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.541 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.541 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:09:33.541 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044873+0000", 2024-09-21T14:09:33.541 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0999e-05 2024-09-21T14:09:33.541 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.541 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.541 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:09:33.541 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044882+0000", 2024-09-21T14:09:33.542 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:33 smithi049 ceph-mon[40489]: 11.a scrub ok 2024-09-21T14:09:33.542 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:33 smithi049 ceph-mon[40489]: pgmap v1056: 168 pgs: 1 active+clean+scrubbing, 1 peering, 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 162 active+clean; 3.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 969 op/s; 4043/185232 objects misplaced (2.183%); 32 MiB/s, 513 objects/s recovering 2024-09-21T14:09:33.542 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:33 smithi049 ceph-mon[40489]: 4.10 scrub starts 2024-09-21T14:09:33.542 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:33 smithi049 ceph-mon[40489]: 4.10 scrub ok 2024-09-21T14:09:33.542 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:33 smithi049 ceph-mon[40489]: 1.a scrub starts 2024-09-21T14:09:33.542 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:33 smithi049 ceph-mon[40489]: 1.a scrub ok 2024-09-21T14:09:33.542 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:33 smithi049 ceph-mon[40489]: 5.8 scrub starts 2024-09-21T14:09:33.542 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:33 smithi049 ceph-mon[40489]: 5.8 scrub ok 2024-09-21T14:09:33.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:33.543 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:33 smithi049 ceph-mon[30080]: 11.a scrub ok 2024-09-21T14:09:33.543 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:33 smithi049 ceph-mon[30080]: pgmap v1056: 168 pgs: 1 active+clean+scrubbing, 1 peering, 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 162 active+clean; 3.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 969 op/s; 4043/185232 objects misplaced (2.183%); 32 MiB/s, 513 objects/s recovering 2024-09-21T14:09:33.543 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:33 smithi049 ceph-mon[30080]: 4.10 scrub starts 2024-09-21T14:09:33.543 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:33 smithi049 ceph-mon[30080]: 4.10 scrub ok 2024-09-21T14:09:33.543 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:33 smithi049 ceph-mon[30080]: 1.a scrub starts 2024-09-21T14:09:33.543 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:33 smithi049 ceph-mon[30080]: 1.a scrub ok 2024-09-21T14:09:33.543 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:33 smithi049 ceph-mon[30080]: 5.8 scrub starts 2024-09-21T14:09:33.543 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:33 smithi049 ceph-mon[30080]: 5.8 scrub ok 2024-09-21T14:09:33.544 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:33.544 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6950000000000006e-06 2024-09-21T14:09:33.544 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.544 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.545 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:09:33.545 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044888+0000", 2024-09-21T14:09:33.545 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2770000000000002e-06 2024-09-21T14:09:33.545 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.545 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.545 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:09:33.545 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.053728+0000", 2024-09-21T14:09:33.545 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.008840311 2024-09-21T14:09:33.545 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.545 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.545 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:09:33.545 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.053855+0000", 2024-09-21T14:09:33.545 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000126174 2024-09-21T14:09:33.545 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.546 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.546 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,3", 2024-09-21T14:09:33.546 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.053941+0000", 2024-09-21T14:09:33.546 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6009999999999998e-05 2024-09-21T14:09:33.546 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.546 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.546 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:09:33.546 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.056428+0000", 2024-09-21T14:09:33.546 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0024874340000000002 2024-09-21T14:09:33.546 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.546 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.546 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:33.546 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.060518+0000", 2024-09-21T14:09:33.547 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.004089606 2024-09-21T14:09:33.547 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.547 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.547 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:33.547 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.356032+0000", 2024-09-21T14:09:33.547 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.29551429600000001 2024-09-21T14:09:33.547 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.547 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.547 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:09:33.547 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.356082+0000", 2024-09-21T14:09:33.547 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9679000000000003e-05 2024-09-21T14:09:33.547 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.547 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.549 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:09:33.549 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.356115+0000", 2024-09-21T14:09:33.549 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3821e-05 2024-09-21T14:09:33.549 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:33.549 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:09:33.550 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:33.550 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.550 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.550 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:70945 10.8 10:1428b9f2:::benchmark_data_smithi049_176811_object70944:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-09-21T14:09:33.550 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:03:44.055643+0000", 2024-09-21T14:09:33.550 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 349.441286203, 2024-09-21T14:09:33.550 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.28525528100000003, 2024-09-21T14:09:33.550 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:09:33.550 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:09:33.550 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:09:33.550 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:09:33.550 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:09:33.550 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:09:33.550 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 70945 2024-09-21T14:09:33.551 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.551 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:09:33.551 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.551 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:09:33.551 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.055643+0000", 2024-09-21T14:09:33.551 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:33.551 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.551 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.551 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:09:33.551 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.055643+0000", 2024-09-21T14:09:33.551 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:33.551 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.551 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.551 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:09:33.551 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.055644+0000", 2024-09-21T14:09:33.552 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5209999999999999e-06 2024-09-21T14:09:33.552 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.552 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.552 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:09:33.552 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.055666+0000", 2024-09-21T14:09:33.552 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1977999999999999e-05 2024-09-21T14:09:33.552 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.552 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.552 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:09:33.552 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.055682+0000", 2024-09-21T14:09:33.552 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6005999999999999e-05 2024-09-21T14:09:33.552 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.552 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.552 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:09:33.553 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.055690+0000", 2024-09-21T14:09:33.553 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.763e-06 2024-09-21T14:09:33.553 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.553 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.553 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:09:33.553 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.055754+0000", 2024-09-21T14:09:33.553 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3637000000000001e-05 2024-09-21T14:09:33.553 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.553 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.553 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:09:33.553 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.055891+0000", 2024-09-21T14:09:33.553 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013709900000000001 2024-09-21T14:09:33.553 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.553 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.554 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:09:33.554 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.056021+0000", 2024-09-21T14:09:33.554 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013076599999999999 2024-09-21T14:09:33.554 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.554 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.554 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:09:33.554 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.058112+0000", 2024-09-21T14:09:33.554 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002090887 2024-09-21T14:09:33.554 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.554 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.554 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:33.554 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.058366+0000", 2024-09-21T14:09:33.554 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00025399500000000002 2024-09-21T14:09:33.555 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.555 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.555 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:33.555 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.340797+0000", 2024-09-21T14:09:33.555 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.28243062699999999 2024-09-21T14:09:33.555 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.555 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.555 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:09:33.555 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.340853+0000", 2024-09-21T14:09:33.555 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6184000000000003e-05 2024-09-21T14:09:33.555 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.555 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.555 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:09:33.555 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.340898+0000", 2024-09-21T14:09:33.556 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4817999999999997e-05 2024-09-21T14:09:33.556 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:33.556 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:09:33.556 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:33.556 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.556 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.556 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:70947 10.5 10:a00a68cc:::benchmark_data_smithi049_176811_object70946:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-09-21T14:09:33.556 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:03:44.057433+0000", 2024-09-21T14:09:33.556 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 349.43949536100001, 2024-09-21T14:09:33.556 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.30024145400000002, 2024-09-21T14:09:33.556 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:09:33.556 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:09:33.556 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:09:33.557 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:09:33.557 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:09:33.557 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:09:33.557 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 70947 2024-09-21T14:09:33.557 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.557 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:09:33.557 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.557 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:09:33.557 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.057433+0000", 2024-09-21T14:09:33.557 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:33.557 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.557 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.557 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:09:33.558 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.057433+0000", 2024-09-21T14:09:33.558 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:33.558 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.558 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.558 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:09:33.558 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.057435+0000", 2024-09-21T14:09:33.558 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1680000000000001e-06 2024-09-21T14:09:33.558 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.558 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.558 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:09:33.558 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.057462+0000", 2024-09-21T14:09:33.558 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.7415000000000001e-05 2024-09-21T14:09:33.558 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.558 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.559 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:09:33.559 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.057477+0000", 2024-09-21T14:09:33.559 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5150000000000001e-05 2024-09-21T14:09:33.559 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.559 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.559 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:09:33.559 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.057486+0000", 2024-09-21T14:09:33.559 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1770000000000004e-06 2024-09-21T14:09:33.559 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.559 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.559 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:09:33.559 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.057543+0000", 2024-09-21T14:09:33.559 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7122e-05 2024-09-21T14:09:33.559 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.560 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.560 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:09:33.560 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.057640+0000", 2024-09-21T14:09:33.560 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6124000000000003e-05 2024-09-21T14:09:33.560 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.560 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.560 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,3", 2024-09-21T14:09:33.560 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.057762+0000", 2024-09-21T14:09:33.560 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012260600000000001 2024-09-21T14:09:33.560 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.560 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.560 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:09:33.560 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.059813+0000", 2024-09-21T14:09:33.560 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002050666 2024-09-21T14:09:33.561 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.561 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.561 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:33.561 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.060700+0000", 2024-09-21T14:09:33.561 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00088741999999999998 2024-09-21T14:09:33.561 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.561 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.561 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:33.561 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.357580+0000", 2024-09-21T14:09:33.561 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.29687936700000001 2024-09-21T14:09:33.561 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.561 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.561 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:09:33.561 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.357635+0000", 2024-09-21T14:09:33.562 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5053999999999998e-05 2024-09-21T14:09:33.562 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.562 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.562 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:09:33.562 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.357675+0000", 2024-09-21T14:09:33.562 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0185000000000002e-05 2024-09-21T14:09:33.562 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:33.562 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:09:33.562 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:33.562 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.562 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.562 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:208866 10.3 10:c8242cfb:::benchmark_data_smithi049_176811_object81289:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e441)", 2024-09-21T14:09:33.562 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:05:54.047026+0000", 2024-09-21T14:09:33.562 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 219.449902885, 2024-09-21T14:09:33.562 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.31484621499999998, 2024-09-21T14:09:33.563 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:09:33.563 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:09:33.563 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:09:33.563 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:09:33.563 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:09:33.563 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:09:33.563 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 208866 2024-09-21T14:09:33.563 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.563 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:09:33.563 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.563 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:09:33.563 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047026+0000", 2024-09-21T14:09:33.563 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:33.563 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.564 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.564 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:09:33.564 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047026+0000", 2024-09-21T14:09:33.564 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:33.564 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.564 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.564 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:09:33.564 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047027+0000", 2024-09-21T14:09:33.564 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3820000000000001e-06 2024-09-21T14:09:33.564 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.564 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.564 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:09:33.564 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047033+0000", 2024-09-21T14:09:33.565 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3859999999999998e-06 2024-09-21T14:09:33.565 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.565 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.565 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:09:33.565 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047046+0000", 2024-09-21T14:09:33.565 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3482999999999999e-05 2024-09-21T14:09:33.565 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.565 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.565 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:09:33.565 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047053+0000", 2024-09-21T14:09:33.565 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.25e-06 2024-09-21T14:09:33.565 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.565 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.565 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:09:33.566 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047124+0000", 2024-09-21T14:09:33.566 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0309999999999996e-05 2024-09-21T14:09:33.566 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.566 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.566 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:09:33.566 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047243+0000", 2024-09-21T14:09:33.566 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011944599999999999 2024-09-21T14:09:33.566 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.566 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.566 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,5", 2024-09-21T14:09:33.566 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047366+0000", 2024-09-21T14:09:33.566 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012313100000000001 2024-09-21T14:09:33.566 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.566 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.567 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:09:33.567 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.052472+0000", 2024-09-21T14:09:33.567 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0051057760000000002 2024-09-21T14:09:33.567 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.567 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.567 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:33.567 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.055019+0000", 2024-09-21T14:09:33.567 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0025469709999999999 2024-09-21T14:09:33.567 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.567 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.567 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:33.567 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.361767+0000", 2024-09-21T14:09:33.567 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.30674835499999997 2024-09-21T14:09:33.567 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.568 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.568 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:09:33.568 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.361826+0000", 2024-09-21T14:09:33.568 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8084999999999997e-05 2024-09-21T14:09:33.568 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.568 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.568 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:09:33.568 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.361872+0000", 2024-09-21T14:09:33.568 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6640000000000001e-05 2024-09-21T14:09:33.568 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:33.568 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:09:33.568 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:33.568 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.568 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.568 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:208867 10.3 10:c6755fb4:::benchmark_data_smithi049_176811_object81290:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e441)", 2024-09-21T14:09:33.569 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:05:54.047593+0000", 2024-09-21T14:09:33.569 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 219.44933563800001, 2024-09-21T14:09:33.569 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.31790776100000001, 2024-09-21T14:09:33.569 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:09:33.569 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:09:33.569 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:09:33.569 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:09:33.569 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:09:33.569 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:09:33.569 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 208867 2024-09-21T14:09:33.569 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.569 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:09:33.569 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.570 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:09:33.570 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047593+0000", 2024-09-21T14:09:33.570 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:33.570 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.570 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.570 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:09:33.570 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047593+0000", 2024-09-21T14:09:33.570 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:33.570 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.570 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.570 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:09:33.570 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047594+0000", 2024-09-21T14:09:33.570 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9100000000000003e-07 2024-09-21T14:09:33.570 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.571 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.571 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:09:33.571 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047598+0000", 2024-09-21T14:09:33.571 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0199999999999996e-06 2024-09-21T14:09:33.571 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.571 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.571 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:09:33.571 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047609+0000", 2024-09-21T14:09:33.571 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0858e-05 2024-09-21T14:09:33.571 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.571 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.571 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:09:33.571 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047616+0000", 2024-09-21T14:09:33.571 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3629999999999996e-06 2024-09-21T14:09:33.572 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.572 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.572 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:09:33.572 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.050406+0000", 2024-09-21T14:09:33.572 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0027899639999999998 2024-09-21T14:09:33.572 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.572 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.572 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:09:33.572 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.050525+0000", 2024-09-21T14:09:33.572 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011861 2024-09-21T14:09:33.572 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.572 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.572 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,5", 2024-09-21T14:09:33.572 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.050650+0000", 2024-09-21T14:09:33.572 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000124881 2024-09-21T14:09:33.573 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.573 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.573 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:09:33.573 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.052609+0000", 2024-09-21T14:09:33.573 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0019588499999999998 2024-09-21T14:09:33.573 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.573 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.573 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:33.573 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.057528+0000", 2024-09-21T14:09:33.573 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0049191590000000002 2024-09-21T14:09:33.573 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.573 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.573 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:33.574 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.365390+0000", 2024-09-21T14:09:33.574 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.30786228399999999 2024-09-21T14:09:33.574 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.574 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.574 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:09:33.574 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.365449+0000", 2024-09-21T14:09:33.574 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9497999999999999e-05 2024-09-21T14:09:33.574 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.574 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.574 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:09:33.574 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.365501+0000", 2024-09-21T14:09:33.574 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1483000000000002e-05 2024-09-21T14:09:33.574 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:33.574 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:09:33.575 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:33.575 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.575 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.575 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:208869 10.5 10:a17fb6ee:::benchmark_data_smithi049_176811_object81292:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e441)", 2024-09-21T14:09:33.575 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:05:54.048229+0000", 2024-09-21T14:09:33.575 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 219.448699946, 2024-09-21T14:09:33.575 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.31834309799999999, 2024-09-21T14:09:33.575 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:09:33.575 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:09:33.575 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:09:33.575 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:09:33.575 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:09:33.575 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:09:33.575 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 208869 2024-09-21T14:09:33.576 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.576 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:09:33.576 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.576 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:09:33.576 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.048229+0000", 2024-09-21T14:09:33.576 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:33.576 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.576 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.576 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:09:33.576 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.048229+0000", 2024-09-21T14:09:33.576 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:33.576 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.576 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.577 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:09:33.577 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.048230+0000", 2024-09-21T14:09:33.577 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3799999999999996e-07 2024-09-21T14:09:33.577 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.577 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.577 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:09:33.577 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.048233+0000", 2024-09-21T14:09:33.577 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.602e-06 2024-09-21T14:09:33.577 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.577 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.577 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:09:33.577 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.048242+0000", 2024-09-21T14:09:33.577 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0389999999999993e-06 2024-09-21T14:09:33.577 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.578 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.578 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:09:33.578 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.048249+0000", 2024-09-21T14:09:33.578 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7599999999999997e-06 2024-09-21T14:09:33.578 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.578 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.578 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:09:33.578 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.048305+0000", 2024-09-21T14:09:33.578 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6085999999999998e-05 2024-09-21T14:09:33.578 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.578 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.578 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:09:33.578 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.048397+0000", 2024-09-21T14:09:33.578 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2069000000000002e-05 2024-09-21T14:09:33.579 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.579 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.579 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,3", 2024-09-21T14:09:33.579 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.048465+0000", 2024-09-21T14:09:33.579 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7998999999999997e-05 2024-09-21T14:09:33.579 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.579 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.579 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:09:33.579 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.050592+0000", 2024-09-21T14:09:33.579 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0021271509999999999 2024-09-21T14:09:33.579 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.579 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.579 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:33.580 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.051296+0000", 2024-09-21T14:09:33.580 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00070396900000000004 2024-09-21T14:09:33.580 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.580 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.580 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:33.580 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.366467+0000", 2024-09-21T14:09:33.580 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.31517033700000002 2024-09-21T14:09:33.580 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.580 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.580 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:09:33.580 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.366534+0000", 2024-09-21T14:09:33.580 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7045999999999995e-05 2024-09-21T14:09:33.580 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.580 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.581 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:09:33.581 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.366572+0000", 2024-09-21T14:09:33.581 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8201999999999998e-05 2024-09-21T14:09:33.581 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:33.581 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:09:33.581 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:33.581 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.581 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.581 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:208877 10.3 10:cae1fc76:::benchmark_data_smithi049_176811_object81300:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e441)", 2024-09-21T14:09:33.581 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:05:54.054998+0000", 2024-09-21T14:09:33.581 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 219.44193042000001, 2024-09-21T14:09:33.581 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.310795918, 2024-09-21T14:09:33.581 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:09:33.582 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:09:33.582 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:09:33.582 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:09:33.582 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:09:33.582 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:09:33.582 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 208877 2024-09-21T14:09:33.582 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.582 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:09:33.582 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.582 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:09:33.582 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.054998+0000", 2024-09-21T14:09:33.582 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:33.582 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.582 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.583 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:09:33.583 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.054998+0000", 2024-09-21T14:09:33.583 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:33.583 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.583 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.583 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:09:33.583 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.055000+0000", 2024-09-21T14:09:33.583 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.821e-06 2024-09-21T14:09:33.583 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.583 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.583 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:09:33.583 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.055004+0000", 2024-09-21T14:09:33.583 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1540000000000004e-06 2024-09-21T14:09:33.583 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.584 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.584 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:09:33.584 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.055015+0000", 2024-09-21T14:09:33.584 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0594999999999999e-05 2024-09-21T14:09:33.584 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.584 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.584 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:09:33.584 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.055022+0000", 2024-09-21T14:09:33.584 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9380000000000003e-06 2024-09-21T14:09:33.584 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.584 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.584 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:09:33.584 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.055127+0000", 2024-09-21T14:09:33.584 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010461199999999999 2024-09-21T14:09:33.585 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.585 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.585 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:09:33.585 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.055242+0000", 2024-09-21T14:09:33.585 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000115377 2024-09-21T14:09:33.585 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.585 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.585 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,5", 2024-09-21T14:09:33.585 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.055379+0000", 2024-09-21T14:09:33.585 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000136627 2024-09-21T14:09:33.585 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.585 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.585 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:09:33.585 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.057617+0000", 2024-09-21T14:09:33.586 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00223865 2024-09-21T14:09:33.586 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.586 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.586 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:33.586 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.058057+0000", 2024-09-21T14:09:33.586 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00044032700000000001 2024-09-21T14:09:33.586 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.586 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.586 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:33.586 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.365711+0000", 2024-09-21T14:09:33.586 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.30765357999999998 2024-09-21T14:09:33.586 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.586 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.587 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:09:33.587 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.365757+0000", 2024-09-21T14:09:33.587 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5985000000000001e-05 2024-09-21T14:09:33.587 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.587 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.587 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:09:33.587 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.365794+0000", 2024-09-21T14:09:33.587 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7252000000000002e-05 2024-09-21T14:09:33.587 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:33.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:09:33.588 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:09:33.588 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:33.588 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.588 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.588 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:165899 11.d 11:b5b2ff73:::benchmark_data_smithi049_206132_object42520:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e516)", 2024-09-21T14:09:33.588 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:09:07.435391+0000", 2024-09-21T14:09:33.588 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 26.061537484999999, 2024-09-21T14:09:33.589 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.29003483800000002, 2024-09-21T14:09:33.589 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:09:33.589 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:09:33.589 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:09:33.589 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:09:33.589 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:09:33.589 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:09:33.589 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 165899 2024-09-21T14:09:33.589 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.589 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:09:33.589 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.589 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:09:33.589 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.435391+0000", 2024-09-21T14:09:33.589 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:33.590 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.590 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.590 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:09:33.590 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.435391+0000", 2024-09-21T14:09:33.590 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:33.590 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.590 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.590 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:09:33.590 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.435393+0000", 2024-09-21T14:09:33.590 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9400000000000001e-06 2024-09-21T14:09:33.590 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.590 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.590 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:09:33.591 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.435399+0000", 2024-09-21T14:09:33.591 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8420000000000001e-06 2024-09-21T14:09:33.591 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.591 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.591 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:09:33.591 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.435411+0000", 2024-09-21T14:09:33.591 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1766000000000001e-05 2024-09-21T14:09:33.591 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.591 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.591 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:09:33.591 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.435419+0000", 2024-09-21T14:09:33.591 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.303e-06 2024-09-21T14:09:33.591 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.591 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.592 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:09:33.592 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.435703+0000", 2024-09-21T14:09:33.592 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00028423499999999998 2024-09-21T14:09:33.592 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.592 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.592 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:09:33.592 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.435808+0000", 2024-09-21T14:09:33.592 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010479700000000001 2024-09-21T14:09:33.592 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.592 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.592 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 4,6", 2024-09-21T14:09:33.592 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.435906+0000", 2024-09-21T14:09:33.592 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7822e-05 2024-09-21T14:09:33.592 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.593 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.593 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:33.593 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.439396+0000", 2024-09-21T14:09:33.593 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0034903529999999999 2024-09-21T14:09:33.593 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.593 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.593 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:09:33.593 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.440181+0000", 2024-09-21T14:09:33.593 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00078445100000000003 2024-09-21T14:09:33.593 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.593 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.593 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:33.593 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.443229+0000", 2024-09-21T14:09:33.593 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00304785 2024-09-21T14:09:33.594 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.594 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.594 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:09:33.594 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.725376+0000", 2024-09-21T14:09:33.594 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.28214762500000001 2024-09-21T14:09:33.594 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.594 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.594 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:09:33.594 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.725426+0000", 2024-09-21T14:09:33.594 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9854000000000001e-05 2024-09-21T14:09:33.594 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:33.594 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:09:33.594 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:33.594 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.595 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.595 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop_reply(client.26396.0:165899 11.d e516/461 ondisk, result = 0)", 2024-09-21T14:09:33.595 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:09:07.443133+0000", 2024-09-21T14:09:33.595 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 26.053795584, 2024-09-21T14:09:33.595 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.28229947500000002, 2024-09-21T14:09:33.595 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:09:33.595 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:09:33.595 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T14:09:33.595 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:09:33.595 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.595 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:09:33.595 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.443133+0000", 2024-09-21T14:09:33.595 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:33.595 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.595 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.596 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:09:33.596 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.443133+0000", 2024-09-21T14:09:33.596 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:33.596 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.596 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.596 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:09:33.596 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.443134+0000", 2024-09-21T14:09:33.596 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.009e-06 2024-09-21T14:09:33.596 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.596 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.596 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:09:33.596 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.443139+0000", 2024-09-21T14:09:33.596 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9590000000000003e-06 2024-09-21T14:09:33.596 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.597 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.597 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:09:33.597 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.443151+0000", 2024-09-21T14:09:33.597 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1341e-05 2024-09-21T14:09:33.597 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.597 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.597 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:09:33.597 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.443158+0000", 2024-09-21T14:09:33.597 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.109e-06 2024-09-21T14:09:33.597 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.597 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.597 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:09:33.597 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.443214+0000", 2024-09-21T14:09:33.597 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6329000000000002e-05 2024-09-21T14:09:33.598 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.598 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.598 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:09:33.598 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.443223+0000", 2024-09-21T14:09:33.598 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5439999999999999e-06 2024-09-21T14:09:33.598 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.598 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.598 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:09:33.598 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.725433+0000", 2024-09-21T14:09:33.598 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.28221018399999998 2024-09-21T14:09:33.598 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:33.598 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:09:33.598 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:33.598 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.599 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.599 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26396.0:165905 11.4 e516/491 11:24b5e356:::benchmark_data_smithi049_206132_object42526:head v 516'10459, mlcod=516'10457)", 2024-09-21T14:09:33.599 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:09:07.443183+0000", 2024-09-21T14:09:33.599 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 26.053746160999999, 2024-09-21T14:09:33.599 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.290709253, 2024-09-21T14:09:33.599 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:09:33.599 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:09:33.599 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:09:33.599 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:09:33.599 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.599 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:09:33.599 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.443183+0000", 2024-09-21T14:09:33.599 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:33.599 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.600 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.600 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:09:33.600 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.443183+0000", 2024-09-21T14:09:33.600 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:33.600 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.600 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.600 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:09:33.600 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.443183+0000", 2024-09-21T14:09:33.600 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1399999999999997e-07 2024-09-21T14:09:33.600 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.600 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.600 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:09:33.600 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.443196+0000", 2024-09-21T14:09:33.600 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3145999999999999e-05 2024-09-21T14:09:33.601 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.601 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.601 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:09:33.601 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.443208+0000", 2024-09-21T14:09:33.601 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1884e-05 2024-09-21T14:09:33.601 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.601 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.601 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:09:33.601 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.443214+0000", 2024-09-21T14:09:33.601 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4709999999999998e-06 2024-09-21T14:09:33.601 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.601 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.601 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:09:33.601 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.725666+0000", 2024-09-21T14:09:33.601 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.28245240799999999 2024-09-21T14:09:33.602 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.602 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.602 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:09:33.602 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.725685+0000", 2024-09-21T14:09:33.602 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9130000000000001e-05 2024-09-21T14:09:33.602 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.602 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.602 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:09:33.602 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.733850+0000", 2024-09-21T14:09:33.602 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00816448 2024-09-21T14:09:33.602 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.602 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.602 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:09:33.603 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.733892+0000", 2024-09-21T14:09:33.603 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2320000000000001e-05 2024-09-21T14:09:33.603 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:33.603 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:09:33.603 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:33.603 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.603 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.603 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26396.0:165906 11.8 e516/512 11:10b4dc38:::benchmark_data_smithi049_206132_object42527:head v 516'10211, mlcod=516'10208)", 2024-09-21T14:09:33.603 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:09:07.443231+0000", 2024-09-21T14:09:33.603 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 26.053697591999999, 2024-09-21T14:09:33.603 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.282090375, 2024-09-21T14:09:33.603 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:09:33.603 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:09:33.603 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:09:33.604 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:09:33.604 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.604 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:09:33.604 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.443231+0000", 2024-09-21T14:09:33.604 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:33.604 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.604 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.604 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:09:33.604 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.443231+0000", 2024-09-21T14:09:33.604 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:33.604 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.604 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.604 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:09:33.604 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.443232+0000", 2024-09-21T14:09:33.605 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6899999999999998e-07 2024-09-21T14:09:33.605 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.605 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.605 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:09:33.605 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.443235+0000", 2024-09-21T14:09:33.605 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7859999999999998e-06 2024-09-21T14:09:33.605 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.605 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.605 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:09:33.605 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.443242+0000", 2024-09-21T14:09:33.605 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.764e-06 2024-09-21T14:09:33.605 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.605 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.605 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:09:33.605 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.443247+0000", 2024-09-21T14:09:33.606 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9749999999999998e-06 2024-09-21T14:09:33.606 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.606 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.606 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:09:33.606 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.443291+0000", 2024-09-21T14:09:33.606 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.367e-05 2024-09-21T14:09:33.606 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.606 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.606 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:09:33.606 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.443310+0000", 2024-09-21T14:09:33.606 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9497999999999999e-05 2024-09-21T14:09:33.606 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.606 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.606 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:09:33.607 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.725273+0000", 2024-09-21T14:09:33.607 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.28196235400000003 2024-09-21T14:09:33.607 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.607 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.607 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:09:33.607 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.725322+0000", 2024-09-21T14:09:33.607 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8959000000000003e-05 2024-09-21T14:09:33.607 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:33.607 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:09:33.607 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:33.607 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.607 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.607 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26396.0:165901 11.6 e516/511 11:69bce1b9:::benchmark_data_smithi049_206132_object42522:head v 516'10409, mlcod=516'10407)", 2024-09-21T14:09:33.608 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:09:07.443293+0000", 2024-09-21T14:09:33.608 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 26.053635762999999, 2024-09-21T14:09:33.608 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.29079608299999998, 2024-09-21T14:09:33.608 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:09:33.608 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:09:33.608 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:09:33.608 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:09:33.608 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.608 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:09:33.608 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.443293+0000", 2024-09-21T14:09:33.608 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:33.608 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.608 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.608 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:09:33.609 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.443293+0000", 2024-09-21T14:09:33.609 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:33.609 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.609 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.609 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:09:33.609 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.443294+0000", 2024-09-21T14:09:33.609 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7599999999999997e-07 2024-09-21T14:09:33.609 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.609 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.609 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:09:33.609 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.443298+0000", 2024-09-21T14:09:33.609 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1869999999999999e-06 2024-09-21T14:09:33.609 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.609 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.610 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:09:33.610 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.443305+0000", 2024-09-21T14:09:33.610 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7249999999999997e-06 2024-09-21T14:09:33.610 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.610 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.610 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:09:33.610 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.443311+0000", 2024-09-21T14:09:33.610 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5400000000000003e-06 2024-09-21T14:09:33.610 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.610 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.610 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:09:33.610 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.443361+0000", 2024-09-21T14:09:33.610 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9805000000000002e-05 2024-09-21T14:09:33.610 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.611 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.611 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:09:33.611 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.443383+0000", 2024-09-21T14:09:33.611 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1928000000000001e-05 2024-09-21T14:09:33.611 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.611 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.611 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:09:33.611 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.734050+0000", 2024-09-21T14:09:33.611 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.29066679099999998 2024-09-21T14:09:33.611 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.611 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.611 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:09:33.611 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.734089+0000", 2024-09-21T14:09:33.611 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9631000000000001e-05 2024-09-21T14:09:33.612 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:33.612 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:09:33.612 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:33.612 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.612 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.612 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26396.0:165915 11.6 e516/511 11:61b3ff1d:::benchmark_data_smithi049_206132_object42536:head v 516'10410, mlcod=516'10407)", 2024-09-21T14:09:33.612 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:09:07.445961+0000", 2024-09-21T14:09:33.612 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 26.050968126000001, 2024-09-21T14:09:33.612 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.288154303, 2024-09-21T14:09:33.612 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:09:33.612 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:09:33.612 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:09:33.612 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:09:33.613 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.613 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:09:33.613 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.445961+0000", 2024-09-21T14:09:33.613 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:33.613 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.613 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.613 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:09:33.613 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.445961+0000", 2024-09-21T14:09:33.613 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:33.613 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.613 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.613 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:09:33.613 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.445962+0000", 2024-09-21T14:09:33.613 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.187e-06 2024-09-21T14:09:33.613 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.614 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.614 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:09:33.614 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.445970+0000", 2024-09-21T14:09:33.614 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8930000000000005e-06 2024-09-21T14:09:33.614 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.614 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.614 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:09:33.614 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.725273+0000", 2024-09-21T14:09:33.614 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27930311200000002 2024-09-21T14:09:33.614 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.614 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.614 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:09:33.614 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.725285+0000", 2024-09-21T14:09:33.615 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.218e-05 2024-09-21T14:09:33.615 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.615 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.615 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:09:33.615 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.728103+0000", 2024-09-21T14:09:33.615 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0028179720000000002 2024-09-21T14:09:33.615 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.615 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.615 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:09:33.615 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.728129+0000", 2024-09-21T14:09:33.615 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5922e-05 2024-09-21T14:09:33.615 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.615 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.615 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:09:33.616 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.734092+0000", 2024-09-21T14:09:33.616 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0059626890000000002 2024-09-21T14:09:33.616 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.616 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.616 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:09:33.616 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.734115+0000", 2024-09-21T14:09:33.616 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3348000000000001e-05 2024-09-21T14:09:33.616 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:33.616 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:09:33.616 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:33.616 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.616 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.616 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "pg_backfill(progress 11.1 e 516/516 lb 11:8ad20b0b:::benchmark_data_smithi049_206132_object52887:head)", 2024-09-21T14:09:33.616 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:09:07.446103+0000", 2024-09-21T14:09:33.616 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 26.050826139000002, 2024-09-21T14:09:33.617 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27948231400000001, 2024-09-21T14:09:33.617 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:09:33.617 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:09:33.617 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T14:09:33.617 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:09:33.617 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.617 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:09:33.617 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.446103+0000", 2024-09-21T14:09:33.617 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:33.617 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.617 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.617 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:09:33.617 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.446103+0000", 2024-09-21T14:09:33.618 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:33.618 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.618 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.618 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:09:33.618 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.446103+0000", 2024-09-21T14:09:33.618 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.61e-07 2024-09-21T14:09:33.618 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.618 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.618 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:09:33.618 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.446116+0000", 2024-09-21T14:09:33.618 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2291e-05 2024-09-21T14:09:33.618 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.618 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.618 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:09:33.619 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.725298+0000", 2024-09-21T14:09:33.619 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.27918275399999998 2024-09-21T14:09:33.619 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.619 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.619 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:09:33.619 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.725310+0000", 2024-09-21T14:09:33.619 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1423999999999999e-05 2024-09-21T14:09:33.619 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.619 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.619 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:09:33.619 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.725404+0000", 2024-09-21T14:09:33.619 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4000999999999997e-05 2024-09-21T14:09:33.619 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.619 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.619 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:09:33.620 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.725419+0000", 2024-09-21T14:09:33.620 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4725e-05 2024-09-21T14:09:33.620 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:33.620 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:33.620 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:09:33.620 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.725585+0000", 2024-09-21T14:09:33.620 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000166358 2024-09-21T14:09:33.620 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:33.620 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:09:33.620 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:33.620 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:33.620 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:09:33.620 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T14:09:33.671 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:09:33.671 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:09:33.671 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:09:33.671 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:09:33.671 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:09:33.672 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:09:33.688 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:09:33.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:33 smithi187 ceph-mon[33032]: 11.a scrub ok 2024-09-21T14:09:33.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:33 smithi187 ceph-mon[33032]: pgmap v1056: 168 pgs: 1 active+clean+scrubbing, 1 peering, 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 162 active+clean; 3.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 969 op/s; 4043/185232 objects misplaced (2.183%); 32 MiB/s, 513 objects/s recovering 2024-09-21T14:09:33.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:33 smithi187 ceph-mon[33032]: 4.10 scrub starts 2024-09-21T14:09:33.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:33 smithi187 ceph-mon[33032]: 4.10 scrub ok 2024-09-21T14:09:33.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:33 smithi187 ceph-mon[33032]: 1.a scrub starts 2024-09-21T14:09:33.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:33 smithi187 ceph-mon[33032]: 1.a scrub ok 2024-09-21T14:09:33.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:33 smithi187 ceph-mon[33032]: 5.8 scrub starts 2024-09-21T14:09:33.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:33 smithi187 ceph-mon[33032]: 5.8 scrub ok 2024-09-21T14:09:33.706 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:09:33.706 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:09:33.716 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:33.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:33.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:09:33.739 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:09:33.739 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:09:33.783 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:09:33.783 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:09:33.812 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:09:33.812 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:09:33.818 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:09:33.845 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:09:33.845 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:09:33.866 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:33.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:33.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:09:33.968 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:09:33.996 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:33.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:33.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:09:34.097 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:34.134 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:34.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:34.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:34.236 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:09:34.280 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:34.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:34.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:09:34.382 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:34.411 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:34.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:34.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:34.509 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T14:09:34.513 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:34.597 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:34.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:34.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:34.630 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:34 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1126775617' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:09:34.630 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:34 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:09:34.630 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:34 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:09:34.631 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:34 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1126775617' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:09:34.631 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:34 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:09:34.631 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:34 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:09:34.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:34 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1126775617' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:09:34.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:34 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:09:34.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:34 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:09:34.698 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:09:34.728 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:34.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:34.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:09:34.829 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:34.855 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:34.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:34.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:34.957 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:09:34.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:34.996 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:34.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:09:35.097 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:35.143 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:35.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:35.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:35.244 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:35.275 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:35.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:35.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:35.377 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:35.403 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:35.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:35.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:35.505 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:09:35.540 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:35.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:35.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:09:35.642 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:09:35.686 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:35.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:35.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:09:35.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:35 smithi187 ceph-mon[33032]: pgmap v1058: 168 pgs: 1 active+clean+scrubbing, 1 peering, 2 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 3.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 732 op/s; 2198/180486 objects misplaced (1.218%); 22 MiB/s, 358 objects/s recovering 2024-09-21T14:09:35.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:35 smithi187 ceph-mon[33032]: 11.b scrub starts 2024-09-21T14:09:35.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:35 smithi187 ceph-mon[33032]: 3.1c scrub starts 2024-09-21T14:09:35.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:35 smithi187 ceph-mon[33032]: 3.1c scrub ok 2024-09-21T14:09:35.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:35 smithi187 ceph-mon[33032]: osdmap e527: 8 total, 8 up, 8 in 2024-09-21T14:09:35.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:35 smithi187 ceph-mon[33032]: 6.8 scrub starts 2024-09-21T14:09:35.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:35 smithi187 ceph-mon[33032]: 6.8 scrub ok 2024-09-21T14:09:35.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:35 smithi187 ceph-mon[33032]: 1.13 scrub starts 2024-09-21T14:09:35.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:35 smithi187 ceph-mon[33032]: 1.13 scrub ok 2024-09-21T14:09:35.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:35 smithi049 ceph-mon[30080]: pgmap v1058: 168 pgs: 1 active+clean+scrubbing, 1 peering, 2 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 3.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 732 op/s; 2198/180486 objects misplaced (1.218%); 22 MiB/s, 358 objects/s recovering 2024-09-21T14:09:35.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:35 smithi049 ceph-mon[30080]: 11.b scrub starts 2024-09-21T14:09:35.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:35 smithi049 ceph-mon[30080]: 3.1c scrub starts 2024-09-21T14:09:35.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:35 smithi049 ceph-mon[30080]: 3.1c scrub ok 2024-09-21T14:09:35.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:35 smithi049 ceph-mon[30080]: osdmap e527: 8 total, 8 up, 8 in 2024-09-21T14:09:35.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:35 smithi049 ceph-mon[30080]: 6.8 scrub starts 2024-09-21T14:09:35.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:35 smithi049 ceph-mon[30080]: 6.8 scrub ok 2024-09-21T14:09:35.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:35 smithi049 ceph-mon[30080]: 1.13 scrub starts 2024-09-21T14:09:35.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:35 smithi049 ceph-mon[30080]: 1.13 scrub ok 2024-09-21T14:09:35.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:35 smithi049 ceph-mon[40489]: pgmap v1058: 168 pgs: 1 active+clean+scrubbing, 1 peering, 2 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 163 active+clean; 3.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 732 op/s; 2198/180486 objects misplaced (1.218%); 22 MiB/s, 358 objects/s recovering 2024-09-21T14:09:35.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:35 smithi049 ceph-mon[40489]: 11.b scrub starts 2024-09-21T14:09:35.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:35 smithi049 ceph-mon[40489]: 3.1c scrub starts 2024-09-21T14:09:35.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:35 smithi049 ceph-mon[40489]: 3.1c scrub ok 2024-09-21T14:09:35.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:35 smithi049 ceph-mon[40489]: osdmap e527: 8 total, 8 up, 8 in 2024-09-21T14:09:35.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:35 smithi049 ceph-mon[40489]: 6.8 scrub starts 2024-09-21T14:09:35.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:35 smithi049 ceph-mon[40489]: 6.8 scrub ok 2024-09-21T14:09:35.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:35 smithi049 ceph-mon[40489]: 1.13 scrub starts 2024-09-21T14:09:35.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:35 smithi049 ceph-mon[40489]: 1.13 scrub ok 2024-09-21T14:09:35.787 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:09:35.821 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:35.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:35.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:09:35.923 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:09:35.950 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:35.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:35.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:09:36.052 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:09:36.092 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:36.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:36.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:09:36.193 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:09:36.223 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:36.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:36.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:09:36.325 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:09:36.367 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:36.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:36.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:09:36.469 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:36.506 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:36.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:36.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:36.608 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:09:36.645 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:36.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:36.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:09:36.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:36 smithi187 ceph-mon[33032]: 6.2 scrub starts 2024-09-21T14:09:36.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:36 smithi187 ceph-mon[33032]: 6.2 scrub ok 2024-09-21T14:09:36.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:36 smithi187 ceph-mon[33032]: 3.b scrub starts 2024-09-21T14:09:36.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:36 smithi187 ceph-mon[33032]: 3.b scrub ok 2024-09-21T14:09:36.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:36 smithi187 ceph-mon[33032]: 5.e scrub starts 2024-09-21T14:09:36.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:36 smithi187 ceph-mon[33032]: 5.e scrub ok 2024-09-21T14:09:36.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:36 smithi187 ceph-mon[33032]: 1.b scrub starts 2024-09-21T14:09:36.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:36 smithi187 ceph-mon[33032]: 1.b scrub ok 2024-09-21T14:09:36.747 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:36.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:36 smithi049 ceph-mon[40489]: 6.2 scrub starts 2024-09-21T14:09:36.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:36 smithi049 ceph-mon[40489]: 6.2 scrub ok 2024-09-21T14:09:36.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:36 smithi049 ceph-mon[40489]: 3.b scrub starts 2024-09-21T14:09:36.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:36 smithi049 ceph-mon[40489]: 3.b scrub ok 2024-09-21T14:09:36.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:36 smithi049 ceph-mon[40489]: 5.e scrub starts 2024-09-21T14:09:36.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:36 smithi049 ceph-mon[40489]: 5.e scrub ok 2024-09-21T14:09:36.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:36 smithi049 ceph-mon[40489]: 1.b scrub starts 2024-09-21T14:09:36.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:36 smithi049 ceph-mon[40489]: 1.b scrub ok 2024-09-21T14:09:36.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:36 smithi049 ceph-mon[30080]: 6.2 scrub starts 2024-09-21T14:09:36.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:36 smithi049 ceph-mon[30080]: 6.2 scrub ok 2024-09-21T14:09:36.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:36 smithi049 ceph-mon[30080]: 3.b scrub starts 2024-09-21T14:09:36.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:36 smithi049 ceph-mon[30080]: 3.b scrub ok 2024-09-21T14:09:36.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:36 smithi049 ceph-mon[30080]: 5.e scrub starts 2024-09-21T14:09:36.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:36 smithi049 ceph-mon[30080]: 5.e scrub ok 2024-09-21T14:09:36.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:36 smithi049 ceph-mon[30080]: 1.b scrub starts 2024-09-21T14:09:36.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:36 smithi049 ceph-mon[30080]: 1.b scrub ok 2024-09-21T14:09:36.772 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:36.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:36.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:36.874 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:09:36.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:36.902 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:36.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:09:37.003 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:37.043 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:37.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:37.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:37.145 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:37.166 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:37.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:37.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:37.268 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:09:37.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:37.299 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:37.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:09:37.400 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:09:37.435 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:37.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:37.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:09:37.536 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:09:37.579 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:37.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:37.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:09:37.681 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:09:37.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:37 smithi187 ceph-mon[33032]: pgmap v1060: 168 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 165 active+clean; 3.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 892 op/s; 1152/174525 objects misplaced (0.660%); 29 MiB/s, 464 objects/s recovering 2024-09-21T14:09:37.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:37 smithi187 ceph-mon[33032]: 11.b scrub ok 2024-09-21T14:09:37.724 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:37 smithi049 ceph-mon[30080]: pgmap v1060: 168 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 165 active+clean; 3.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 892 op/s; 1152/174525 objects misplaced (0.660%); 29 MiB/s, 464 objects/s recovering 2024-09-21T14:09:37.725 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:37 smithi049 ceph-mon[30080]: 11.b scrub ok 2024-09-21T14:09:37.725 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:37 smithi049 ceph-mon[40489]: pgmap v1060: 168 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 165 active+clean; 3.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 892 op/s; 1152/174525 objects misplaced (0.660%); 29 MiB/s, 464 objects/s recovering 2024-09-21T14:09:37.725 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:37 smithi049 ceph-mon[40489]: 11.b scrub ok 2024-09-21T14:09:37.727 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:37.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:37.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:09:37.830 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:37.857 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:37.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:37.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:37.959 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:09:37.987 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:37.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:37.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:09:38.088 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:38.126 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:38.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:38.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:38.228 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:38.259 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:38.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:38.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:38.361 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:09:38.400 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:38.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:38.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:09:38.501 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:09:38.535 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:38.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:38.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:09:38.636 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:38 smithi049 ceph-mon[30080]: 5.18 scrub starts 2024-09-21T14:09:38.636 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:38 smithi049 ceph-mon[30080]: 5.18 scrub ok 2024-09-21T14:09:38.636 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:38 smithi049 ceph-mon[30080]: 6.7 scrub starts 2024-09-21T14:09:38.636 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:38 smithi049 ceph-mon[30080]: 6.7 scrub ok 2024-09-21T14:09:38.636 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:38 smithi049 ceph-mon[30080]: 6.4 scrub starts 2024-09-21T14:09:38.636 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:38 smithi049 ceph-mon[30080]: 6.4 scrub ok 2024-09-21T14:09:38.636 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:38.638 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:38 smithi049 ceph-mon[40489]: 5.18 scrub starts 2024-09-21T14:09:38.638 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:38 smithi049 ceph-mon[40489]: 5.18 scrub ok 2024-09-21T14:09:38.638 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:38 smithi049 ceph-mon[40489]: 6.7 scrub starts 2024-09-21T14:09:38.638 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:38 smithi049 ceph-mon[40489]: 6.7 scrub ok 2024-09-21T14:09:38.638 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:38 smithi049 ceph-mon[40489]: 6.4 scrub starts 2024-09-21T14:09:38.638 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:38 smithi049 ceph-mon[40489]: 6.4 scrub ok 2024-09-21T14:09:38.663 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:38.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:38.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:38.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:38 smithi187 ceph-mon[33032]: 5.18 scrub starts 2024-09-21T14:09:38.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:38 smithi187 ceph-mon[33032]: 5.18 scrub ok 2024-09-21T14:09:38.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:38 smithi187 ceph-mon[33032]: 6.7 scrub starts 2024-09-21T14:09:38.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:38 smithi187 ceph-mon[33032]: 6.7 scrub ok 2024-09-21T14:09:38.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:38 smithi187 ceph-mon[33032]: 6.4 scrub starts 2024-09-21T14:09:38.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:38 smithi187 ceph-mon[33032]: 6.4 scrub ok 2024-09-21T14:09:38.765 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:38.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:38.808 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:38.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:38.909 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:38.940 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:38.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:38.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:39.041 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:39.086 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:39.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:39.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:39.188 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:09:39.217 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:39.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:39.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:09:39.319 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:09:39.360 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:39.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:39.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:09:39.403 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T14:09:39.462 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:09:39.505 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:39.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:39.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:09:39.607 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:09:39.633 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:39.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:39.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:09:39.661 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:39 smithi049 ceph-mon[40489]: pgmap v1061: 168 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 166 active+clean; 3.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 739 B/s rd, 0 B/s wr, 665 op/s; 1152/172668 objects misplaced (0.667%); 25 MiB/s, 397 objects/s recovering 2024-09-21T14:09:39.661 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:39 smithi049 ceph-mon[40489]: 6.a scrub starts 2024-09-21T14:09:39.661 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:39 smithi049 ceph-mon[40489]: 6.a scrub ok 2024-09-21T14:09:39.661 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:39 smithi049 ceph-mon[40489]: 11.d scrub starts 2024-09-21T14:09:39.661 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:39 smithi049 ceph-mon[30080]: pgmap v1061: 168 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 166 active+clean; 3.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 739 B/s rd, 0 B/s wr, 665 op/s; 1152/172668 objects misplaced (0.667%); 25 MiB/s, 397 objects/s recovering 2024-09-21T14:09:39.661 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:39 smithi049 ceph-mon[30080]: 6.a scrub starts 2024-09-21T14:09:39.661 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:39 smithi049 ceph-mon[30080]: 6.a scrub ok 2024-09-21T14:09:39.661 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:39 smithi049 ceph-mon[30080]: 11.d scrub starts 2024-09-21T14:09:39.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:39 smithi187 ceph-mon[33032]: pgmap v1061: 168 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 166 active+clean; 3.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 739 B/s rd, 0 B/s wr, 665 op/s; 1152/172668 objects misplaced (0.667%); 25 MiB/s, 397 objects/s recovering 2024-09-21T14:09:39.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:39 smithi187 ceph-mon[33032]: 6.a scrub starts 2024-09-21T14:09:39.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:39 smithi187 ceph-mon[33032]: 6.a scrub ok 2024-09-21T14:09:39.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:39 smithi187 ceph-mon[33032]: 11.d scrub starts 2024-09-21T14:09:39.736 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:39.760 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:39.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:39.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:39.849 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:09:39.862 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:39.888 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:39.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:39.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:39.916 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:09:39.990 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:40.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:40.023 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:40.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:40.124 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:40.151 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:40.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:40.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:40.175 INFO:teuthology.orchestra.run.smithi049.stdout:moved 0 / 504 (0%) 2024-09-21T14:09:40.175 INFO:teuthology.orchestra.run.smithi049.stdout:avg 63 2024-09-21T14:09:40.175 INFO:teuthology.orchestra.run.smithi049.stdout:stddev 15.4596 -> 15.4596 (expected baseline 7.42462) 2024-09-21T14:09:40.176 INFO:teuthology.orchestra.run.smithi049.stdout:min osd.3 with 24 -> 24 pgs (0.380952 -> 0.380952 * mean) 2024-09-21T14:09:40.176 INFO:teuthology.orchestra.run.smithi049.stdout:max osd.1 with 74 -> 74 pgs (1.1746 -> 1.1746 * mean) 2024-09-21T14:09:40.176 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T14:09:40.176 INFO:teuthology.orchestra.run.smithi049.stdout:oload 1000 2024-09-21T14:09:40.176 INFO:teuthology.orchestra.run.smithi049.stdout:max_change 0.05 2024-09-21T14:09:40.176 INFO:teuthology.orchestra.run.smithi049.stdout:max_change_osds 4 2024-09-21T14:09:40.176 INFO:teuthology.orchestra.run.smithi049.stdout:average_utilization 0.0246 2024-09-21T14:09:40.176 INFO:teuthology.orchestra.run.smithi049.stdout:overload_utilization 0.2459 2024-09-21T14:09:40.176 INFO:teuthology.orchestra.run.smithi049.stderr:no change 2024-09-21T14:09:40.253 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:40.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:40.288 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:40.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:40.389 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:09:40.456 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:40.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:40.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:09:40.558 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:40.580 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:40 smithi049 ceph-mon[30080]: 3.3 scrub starts 2024-09-21T14:09:40.580 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:40 smithi049 ceph-mon[30080]: 3.3 scrub ok 2024-09-21T14:09:40.580 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:40 smithi049 ceph-mon[30080]: 5.d scrub starts 2024-09-21T14:09:40.580 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:40 smithi049 ceph-mon[30080]: 5.d scrub ok 2024-09-21T14:09:40.580 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:40 smithi049 ceph-mon[30080]: 11.d scrub ok 2024-09-21T14:09:40.580 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:40 smithi049 ceph-mon[40489]: 3.3 scrub starts 2024-09-21T14:09:40.580 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:40 smithi049 ceph-mon[40489]: 3.3 scrub ok 2024-09-21T14:09:40.580 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:40 smithi049 ceph-mon[40489]: 5.d scrub starts 2024-09-21T14:09:40.581 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:40 smithi049 ceph-mon[40489]: 5.d scrub ok 2024-09-21T14:09:40.581 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:40 smithi049 ceph-mon[40489]: 11.d scrub ok 2024-09-21T14:09:40.599 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:40.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:40.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:40.639 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T14:09:40.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:40 smithi187 ceph-mon[33032]: 3.3 scrub starts 2024-09-21T14:09:40.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:40 smithi187 ceph-mon[33032]: 3.3 scrub ok 2024-09-21T14:09:40.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:40 smithi187 ceph-mon[33032]: 5.d scrub starts 2024-09-21T14:09:40.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:40 smithi187 ceph-mon[33032]: 5.d scrub ok 2024-09-21T14:09:40.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:40 smithi187 ceph-mon[33032]: 11.d scrub ok 2024-09-21T14:09:40.695 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:09:40 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:09:40.701 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:09:40.735 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:40.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:40.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:09:40.837 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:09:40.914 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:09:40 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:09:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:09:40.923 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:40.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:40.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:09:41.025 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:41.053 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:41.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:41.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:41.155 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:09:41.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:41.194 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:41.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:09:41.295 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:09:41.330 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:41.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:41.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:09:41.432 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:09:41.465 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:41.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:41.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:09:41.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:41 smithi049 ceph-mon[30080]: pgmap v1062: 168 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 166 active+clean; 3.3 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 938 op/s; 366/162717 objects misplaced (0.225%); 27 MiB/s, 434 objects/s recovering 2024-09-21T14:09:41.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:41 smithi049 ceph-mon[30080]: from='client.16716 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T14:09:41.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:41 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:09:41.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:41 smithi049 ceph-mon[40489]: pgmap v1062: 168 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 166 active+clean; 3.3 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 938 op/s; 366/162717 objects misplaced (0.225%); 27 MiB/s, 434 objects/s recovering 2024-09-21T14:09:41.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:41 smithi049 ceph-mon[40489]: from='client.16716 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T14:09:41.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:41 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:09:41.567 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:41.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:41.588 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:41.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:41.689 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:09:41.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:41 smithi187 ceph-mon[33032]: pgmap v1062: 168 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 166 active+clean; 3.3 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 938 op/s; 366/162717 objects misplaced (0.225%); 27 MiB/s, 434 objects/s recovering 2024-09-21T14:09:41.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:41 smithi187 ceph-mon[33032]: from='client.16716 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T14:09:41.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:41 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:09:41.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:41.713 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:41.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:09:41.814 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:09:41.846 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:41.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:41.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:09:41.948 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:09:41.971 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:41.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:41.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:09:42.073 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:09:42.094 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:42.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:42.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:09:42.196 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:09:42.217 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:42.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:42.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:09:42.319 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:09:42.343 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:42.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:42.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:09:42.378 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-09-21T14:09:42.444 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:42.475 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:42.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:42.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:42.531 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T14:09:42.531 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:09:42.576 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:42.615 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:42.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:42.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:42.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:42 smithi187 ceph-mon[33032]: 6.6 scrub starts 2024-09-21T14:09:42.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:42 smithi187 ceph-mon[33032]: 6.6 scrub ok 2024-09-21T14:09:42.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:42 smithi187 ceph-mon[33032]: 5.14 scrub starts 2024-09-21T14:09:42.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:42 smithi187 ceph-mon[33032]: 5.14 scrub ok 2024-09-21T14:09:42.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:42 smithi187 ceph-mon[33032]: osdmap e528: 8 total, 8 up, 8 in 2024-09-21T14:09:42.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:42 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3756554277' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:09:42.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:42 smithi187 ceph-mon[33032]: 1.10 scrub starts 2024-09-21T14:09:42.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:42 smithi187 ceph-mon[33032]: 1.10 scrub ok 2024-09-21T14:09:42.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:42 smithi187 ceph-mon[33032]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T14:09:42.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:42 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3756554277' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T14:09:42.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:42 smithi187 ceph-mon[33032]: osdmap e529: 8 total, 8 up, 8 in 2024-09-21T14:09:42.716 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:42.750 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:42.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:42.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:42.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:42 smithi049 ceph-mon[30080]: 6.6 scrub starts 2024-09-21T14:09:42.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:42 smithi049 ceph-mon[30080]: 6.6 scrub ok 2024-09-21T14:09:42.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:42 smithi049 ceph-mon[30080]: 5.14 scrub starts 2024-09-21T14:09:42.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:42 smithi049 ceph-mon[30080]: 5.14 scrub ok 2024-09-21T14:09:42.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:42 smithi049 ceph-mon[30080]: osdmap e528: 8 total, 8 up, 8 in 2024-09-21T14:09:42.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:42 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3756554277' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:09:42.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:42 smithi049 ceph-mon[30080]: 1.10 scrub starts 2024-09-21T14:09:42.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:42 smithi049 ceph-mon[30080]: 1.10 scrub ok 2024-09-21T14:09:42.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:42 smithi049 ceph-mon[30080]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T14:09:42.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:42 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3756554277' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T14:09:42.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:42 smithi049 ceph-mon[30080]: osdmap e529: 8 total, 8 up, 8 in 2024-09-21T14:09:42.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:42 smithi049 ceph-mon[40489]: 6.6 scrub starts 2024-09-21T14:09:42.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:42 smithi049 ceph-mon[40489]: 6.6 scrub ok 2024-09-21T14:09:42.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:42 smithi049 ceph-mon[40489]: 5.14 scrub starts 2024-09-21T14:09:42.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:42 smithi049 ceph-mon[40489]: 5.14 scrub ok 2024-09-21T14:09:42.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:42 smithi049 ceph-mon[40489]: osdmap e528: 8 total, 8 up, 8 in 2024-09-21T14:09:42.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:42 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3756554277' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:09:42.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:42 smithi049 ceph-mon[40489]: 1.10 scrub starts 2024-09-21T14:09:42.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:42 smithi049 ceph-mon[40489]: 1.10 scrub ok 2024-09-21T14:09:42.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:42 smithi049 ceph-mon[40489]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T14:09:42.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:42 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3756554277' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T14:09:42.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:42 smithi049 ceph-mon[40489]: osdmap e529: 8 total, 8 up, 8 in 2024-09-21T14:09:42.852 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:09:42.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:42.887 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:42.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:09:42.988 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:09:43.014 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:43.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:43.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:09:43.116 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:43.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:43.151 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:43.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:43.252 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:43.292 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:43.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:43.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:43.394 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:09:43.423 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:43.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:43.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:09:43.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:43 smithi049 ceph-mon[40489]: 4.1b scrub starts 2024-09-21T14:09:43.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:43 smithi049 ceph-mon[40489]: 4.1b scrub ok 2024-09-21T14:09:43.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:43 smithi049 ceph-mon[40489]: 4.11 scrub starts 2024-09-21T14:09:43.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:43 smithi049 ceph-mon[40489]: 4.11 scrub ok 2024-09-21T14:09:43.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:43 smithi049 ceph-mon[40489]: pgmap v1064: 168 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 166 active+clean; 3.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.05k op/s; 352/155364 objects misplaced (0.227%); 14 MiB/s, 216 objects/s recovering 2024-09-21T14:09:43.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:43 smithi049 ceph-mon[40489]: 4.1d scrub starts 2024-09-21T14:09:43.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:43 smithi049 ceph-mon[40489]: 4.1d scrub ok 2024-09-21T14:09:43.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:43 smithi049 ceph-mon[30080]: 4.1b scrub starts 2024-09-21T14:09:43.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:43 smithi049 ceph-mon[30080]: 4.1b scrub ok 2024-09-21T14:09:43.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:43 smithi049 ceph-mon[30080]: 4.11 scrub starts 2024-09-21T14:09:43.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:43 smithi049 ceph-mon[30080]: 4.11 scrub ok 2024-09-21T14:09:43.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:43 smithi049 ceph-mon[30080]: pgmap v1064: 168 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 166 active+clean; 3.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.05k op/s; 352/155364 objects misplaced (0.227%); 14 MiB/s, 216 objects/s recovering 2024-09-21T14:09:43.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:43 smithi049 ceph-mon[30080]: 4.1d scrub starts 2024-09-21T14:09:43.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:43 smithi049 ceph-mon[30080]: 4.1d scrub ok 2024-09-21T14:09:43.525 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:09:43.558 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:43.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:43.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:09:43.660 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:43.691 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:43.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:43.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:43.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:43 smithi187 ceph-mon[33032]: 4.1b scrub starts 2024-09-21T14:09:43.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:43 smithi187 ceph-mon[33032]: 4.1b scrub ok 2024-09-21T14:09:43.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:43 smithi187 ceph-mon[33032]: 4.11 scrub starts 2024-09-21T14:09:43.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:43 smithi187 ceph-mon[33032]: 4.11 scrub ok 2024-09-21T14:09:43.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:43 smithi187 ceph-mon[33032]: pgmap v1064: 168 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 166 active+clean; 3.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.05k op/s; 352/155364 objects misplaced (0.227%); 14 MiB/s, 216 objects/s recovering 2024-09-21T14:09:43.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:43 smithi187 ceph-mon[33032]: 4.1d scrub starts 2024-09-21T14:09:43.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:43 smithi187 ceph-mon[33032]: 4.1d scrub ok 2024-09-21T14:09:43.793 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:43.819 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T14:09:43.823 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:43.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:43.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:43.925 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:43.953 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:43.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:43.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:44.054 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:44.085 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:44.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:44.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:44.186 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:44.235 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:44.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:44.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:44.337 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:09:44.383 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:44.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:44.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:09:44.484 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:44.522 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:44.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:44.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:44.623 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:09:44.661 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:44.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:44.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:09:44.762 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:44.790 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:44.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:44.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:44.794 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:09:44.829 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:09:44.891 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:44.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:44.923 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:44.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:45.024 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:45.063 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:45.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:45.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:45.165 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:45.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:45.206 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:45.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:45.307 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:45.354 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:45.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:45.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:45.455 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:09:45.497 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:45.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:45.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:09:45.577 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:09:45.577 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:09:45.599 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:09:45.604 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:45 smithi049 ceph-mon[30080]: 5.1 scrub starts 2024-09-21T14:09:45.604 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:45 smithi049 ceph-mon[30080]: 5.1 scrub ok 2024-09-21T14:09:45.604 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:45 smithi049 ceph-mon[30080]: pgmap v1066: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 3.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.07k op/s; 153/148785 objects misplaced (0.103%); 7.2 MiB/s, 114 objects/s recovering 2024-09-21T14:09:45.605 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:45 smithi049 ceph-mon[30080]: 1.8 scrub starts 2024-09-21T14:09:45.605 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:45 smithi049 ceph-mon[30080]: 1.8 scrub ok 2024-09-21T14:09:45.605 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:45 smithi049 ceph-mon[30080]: 4.2 scrub starts 2024-09-21T14:09:45.605 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:45 smithi049 ceph-mon[30080]: 4.2 scrub ok 2024-09-21T14:09:45.605 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:45 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:09:45.605 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:45 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:09:45.605 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:45 smithi049 ceph-mon[40489]: 5.1 scrub starts 2024-09-21T14:09:45.605 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:45 smithi049 ceph-mon[40489]: 5.1 scrub ok 2024-09-21T14:09:45.605 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:45 smithi049 ceph-mon[40489]: pgmap v1066: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 3.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.07k op/s; 153/148785 objects misplaced (0.103%); 7.2 MiB/s, 114 objects/s recovering 2024-09-21T14:09:45.606 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:45 smithi049 ceph-mon[40489]: 1.8 scrub starts 2024-09-21T14:09:45.606 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:45 smithi049 ceph-mon[40489]: 1.8 scrub ok 2024-09-21T14:09:45.606 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:45 smithi049 ceph-mon[40489]: 4.2 scrub starts 2024-09-21T14:09:45.606 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:45 smithi049 ceph-mon[40489]: 4.2 scrub ok 2024-09-21T14:09:45.606 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:45 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:09:45.606 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:45 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:09:45.611 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:09:45.611 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:09:45.638 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:45.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:45.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:09:45.647 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:09:45.648 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:09:45.685 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:09:45.685 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:09:45.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:45 smithi187 ceph-mon[33032]: 5.1 scrub starts 2024-09-21T14:09:45.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:45 smithi187 ceph-mon[33032]: 5.1 scrub ok 2024-09-21T14:09:45.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:45 smithi187 ceph-mon[33032]: pgmap v1066: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 3.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.07k op/s; 153/148785 objects misplaced (0.103%); 7.2 MiB/s, 114 objects/s recovering 2024-09-21T14:09:45.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:45 smithi187 ceph-mon[33032]: 1.8 scrub starts 2024-09-21T14:09:45.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:45 smithi187 ceph-mon[33032]: 1.8 scrub ok 2024-09-21T14:09:45.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:45 smithi187 ceph-mon[33032]: 4.2 scrub starts 2024-09-21T14:09:45.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:45 smithi187 ceph-mon[33032]: 4.2 scrub ok 2024-09-21T14:09:45.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:45 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:09:45.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:45 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:09:45.719 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:09:45.719 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:09:45.741 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:45.754 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:09:45.754 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:09:45.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:45.781 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:45.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:45.796 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:09:45.797 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:09:45.835 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:09:45.835 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:09:45.881 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:09:45.914 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:45.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:45.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:09:46.016 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:09:46.045 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:46.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:46.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:09:46.147 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:09:46.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:46.178 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:46.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:09:46.278 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:09:46.294 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T14:09:46.294 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-09-21T14:09:46.294 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-09-21T14:09:46.294 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-09-21T14:09:46.294 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T14:09:46.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:46.310 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:46.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:09:46.411 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:46.446 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:46.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:46.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:46.548 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:09:46.582 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:46.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:46.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:09:46.684 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:09:46.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:46 smithi187 ceph-mon[33032]: 5.10 scrub starts 2024-09-21T14:09:46.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:46 smithi187 ceph-mon[33032]: 5.10 scrub ok 2024-09-21T14:09:46.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:46 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/867506109' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:09:46.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:46 smithi187 ceph-mon[33032]: 4.5 scrub starts 2024-09-21T14:09:46.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:46 smithi187 ceph-mon[33032]: 4.5 scrub ok 2024-09-21T14:09:46.724 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:46.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:46.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:09:46.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:46 smithi049 ceph-mon[40489]: 5.10 scrub starts 2024-09-21T14:09:46.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:46 smithi049 ceph-mon[40489]: 5.10 scrub ok 2024-09-21T14:09:46.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:46 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/867506109' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:09:46.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:46 smithi049 ceph-mon[40489]: 4.5 scrub starts 2024-09-21T14:09:46.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:46 smithi049 ceph-mon[40489]: 4.5 scrub ok 2024-09-21T14:09:46.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:46 smithi049 ceph-mon[30080]: 5.10 scrub starts 2024-09-21T14:09:46.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:46 smithi049 ceph-mon[30080]: 5.10 scrub ok 2024-09-21T14:09:46.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:46 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/867506109' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:09:46.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:46 smithi049 ceph-mon[30080]: 4.5 scrub starts 2024-09-21T14:09:46.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:46 smithi049 ceph-mon[30080]: 4.5 scrub ok 2024-09-21T14:09:46.785 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T14:09:46.826 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:09:46.854 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:46.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:46.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:09:46.956 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:09:47.007 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:47.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:47.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:09:47.109 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:09:47.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:47.159 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:47.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:09:47.260 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:47.271 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T14:09:47.327 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:47.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:47.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:47.429 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:47.454 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:47.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:47.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:47.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:47 smithi049 ceph-mon[40489]: pgmap v1067: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 2.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.28k op/s; 138/141903 objects misplaced (0.097%); 7.2 MiB/s, 114 objects/s recovering 2024-09-21T14:09:47.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:47 smithi049 ceph-mon[40489]: 1.14 scrub starts 2024-09-21T14:09:47.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:47 smithi049 ceph-mon[40489]: 1.14 scrub ok 2024-09-21T14:09:47.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:47 smithi049 ceph-mon[40489]: 3.14 scrub starts 2024-09-21T14:09:47.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:47 smithi049 ceph-mon[40489]: 3.14 scrub ok 2024-09-21T14:09:47.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:47 smithi049 ceph-mon[40489]: 1.0 scrub starts 2024-09-21T14:09:47.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:47 smithi049 ceph-mon[40489]: 1.0 scrub ok 2024-09-21T14:09:47.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:47 smithi049 ceph-mon[30080]: pgmap v1067: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 2.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.28k op/s; 138/141903 objects misplaced (0.097%); 7.2 MiB/s, 114 objects/s recovering 2024-09-21T14:09:47.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:47 smithi049 ceph-mon[30080]: 1.14 scrub starts 2024-09-21T14:09:47.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:47 smithi049 ceph-mon[30080]: 1.14 scrub ok 2024-09-21T14:09:47.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:47 smithi049 ceph-mon[30080]: 3.14 scrub starts 2024-09-21T14:09:47.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:47 smithi049 ceph-mon[30080]: 3.14 scrub ok 2024-09-21T14:09:47.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:47 smithi049 ceph-mon[30080]: 1.0 scrub starts 2024-09-21T14:09:47.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:47 smithi049 ceph-mon[30080]: 1.0 scrub ok 2024-09-21T14:09:47.555 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:09:47.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:47.613 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:47.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:09:47.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:47 smithi187 ceph-mon[33032]: pgmap v1067: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 2.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.28k op/s; 138/141903 objects misplaced (0.097%); 7.2 MiB/s, 114 objects/s recovering 2024-09-21T14:09:47.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:47 smithi187 ceph-mon[33032]: 1.14 scrub starts 2024-09-21T14:09:47.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:47 smithi187 ceph-mon[33032]: 1.14 scrub ok 2024-09-21T14:09:47.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:47 smithi187 ceph-mon[33032]: 3.14 scrub starts 2024-09-21T14:09:47.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:47 smithi187 ceph-mon[33032]: 3.14 scrub ok 2024-09-21T14:09:47.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:47 smithi187 ceph-mon[33032]: 1.0 scrub starts 2024-09-21T14:09:47.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:47 smithi187 ceph-mon[33032]: 1.0 scrub ok 2024-09-21T14:09:47.714 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:47.744 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:47.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:47.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:47.846 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:47.877 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:47.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:47.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:47.978 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:48.003 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:48.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:48.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:48.104 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:48.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:48.143 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:48.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:48.244 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:48.275 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:48.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:48.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:48.377 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:09:48.404 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:48.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:48.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:09:48.506 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:09:48.539 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:48.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:48.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:09:48.641 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:09:48.643 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 4, 6, 2, 3, 7, 5] 2024-09-21T14:09:48.644 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T14:09:48.644 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-21T14:09:48.644 INFO:tasks.thrashosds.thrasher:Setting osd 0 primary_affinity to 0.000000 2024-09-21T14:09:48.644 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 0 0 2024-09-21T14:09:48.665 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:48.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:48.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:09:48.678 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:48 smithi049 ceph-mon[40489]: 3.d scrub starts 2024-09-21T14:09:48.678 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:48 smithi049 ceph-mon[40489]: 3.d scrub ok 2024-09-21T14:09:48.678 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:48 smithi049 ceph-mon[40489]: 4.1c scrub starts 2024-09-21T14:09:48.678 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:48 smithi049 ceph-mon[40489]: 4.1c scrub ok 2024-09-21T14:09:48.678 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:48 smithi049 ceph-mon[40489]: 4.6 scrub starts 2024-09-21T14:09:48.678 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:48 smithi049 ceph-mon[40489]: 4.6 scrub ok 2024-09-21T14:09:48.678 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:48 smithi049 ceph-mon[40489]: 5.1d scrub starts 2024-09-21T14:09:48.678 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:48 smithi049 ceph-mon[40489]: 5.1d scrub ok 2024-09-21T14:09:48.679 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:48 smithi049 ceph-mon[30080]: 3.d scrub starts 2024-09-21T14:09:48.679 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:48 smithi049 ceph-mon[30080]: 3.d scrub ok 2024-09-21T14:09:48.679 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:48 smithi049 ceph-mon[30080]: 4.1c scrub starts 2024-09-21T14:09:48.679 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:48 smithi049 ceph-mon[30080]: 4.1c scrub ok 2024-09-21T14:09:48.679 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:48 smithi049 ceph-mon[30080]: 4.6 scrub starts 2024-09-21T14:09:48.679 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:48 smithi049 ceph-mon[30080]: 4.6 scrub ok 2024-09-21T14:09:48.679 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:48 smithi049 ceph-mon[30080]: 5.1d scrub starts 2024-09-21T14:09:48.679 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:48 smithi049 ceph-mon[30080]: 5.1d scrub ok 2024-09-21T14:09:48.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:48 smithi187 ceph-mon[33032]: 3.d scrub starts 2024-09-21T14:09:48.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:48 smithi187 ceph-mon[33032]: 3.d scrub ok 2024-09-21T14:09:48.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:48 smithi187 ceph-mon[33032]: 4.1c scrub starts 2024-09-21T14:09:48.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:48 smithi187 ceph-mon[33032]: 4.1c scrub ok 2024-09-21T14:09:48.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:48 smithi187 ceph-mon[33032]: 4.6 scrub starts 2024-09-21T14:09:48.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:48 smithi187 ceph-mon[33032]: 4.6 scrub ok 2024-09-21T14:09:48.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:48 smithi187 ceph-mon[33032]: 5.1d scrub starts 2024-09-21T14:09:48.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:48 smithi187 ceph-mon[33032]: 5.1d scrub ok 2024-09-21T14:09:48.767 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:09:48.798 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:48.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:48.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:09:48.900 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:09:48.929 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:48.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:48.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:09:49.031 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:09:49.072 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:49.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:49.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:09:49.174 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:49.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:49.209 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:49.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:49.310 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:09:49.347 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:49.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:49.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:09:49.449 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:09:49.476 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:49.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:49.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:09:49.578 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:09:49.612 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:09:49.643 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:49.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:49.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:09:49.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:49 smithi187 ceph-mon[33032]: pgmap v1068: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 2.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 957 op/s; 138/139749 objects misplaced (0.099%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T14:09:49.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:49 smithi187 ceph-mon[33032]: 3.12 scrub starts 2024-09-21T14:09:49.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:49 smithi187 ceph-mon[33032]: 3.12 scrub ok 2024-09-21T14:09:49.745 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:09:49.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:49 smithi049 ceph-mon[40489]: pgmap v1068: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 2.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 957 op/s; 138/139749 objects misplaced (0.099%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T14:09:49.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:49 smithi049 ceph-mon[40489]: 3.12 scrub starts 2024-09-21T14:09:49.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:49 smithi049 ceph-mon[40489]: 3.12 scrub ok 2024-09-21T14:09:49.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:49 smithi049 ceph-mon[30080]: pgmap v1068: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 2.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 957 op/s; 138/139749 objects misplaced (0.099%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T14:09:49.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:49 smithi049 ceph-mon[30080]: 3.12 scrub starts 2024-09-21T14:09:49.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:49 smithi049 ceph-mon[30080]: 3.12 scrub ok 2024-09-21T14:09:49.790 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:49.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:49.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:09:49.891 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:09:49.935 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:49.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:49.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:09:50.037 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:50.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:50.067 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:50.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:50.168 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:09:50.208 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:50.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:50.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:09:50.309 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:09:50.309 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:50.335 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:50.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:50.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:50.383 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:09:50.438 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:50.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:50.509 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:50.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:50.579 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:50 smithi049 ceph-mon[40489]: 6.b scrub starts 2024-09-21T14:09:50.580 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:50 smithi049 ceph-mon[40489]: 6.b scrub ok 2024-09-21T14:09:50.580 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:50 smithi049 ceph-mon[40489]: 4.9 scrub starts 2024-09-21T14:09:50.580 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:50 smithi049 ceph-mon[40489]: 4.9 scrub ok 2024-09-21T14:09:50.580 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:50 smithi049 ceph-mon[40489]: 6.1b scrub starts 2024-09-21T14:09:50.580 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:50 smithi049 ceph-mon[40489]: 6.1b scrub ok 2024-09-21T14:09:50.580 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:50 smithi049 ceph-mon[30080]: 6.b scrub starts 2024-09-21T14:09:50.580 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:50 smithi049 ceph-mon[30080]: 6.b scrub ok 2024-09-21T14:09:50.581 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:50 smithi049 ceph-mon[30080]: 4.9 scrub starts 2024-09-21T14:09:50.581 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:50 smithi049 ceph-mon[30080]: 4.9 scrub ok 2024-09-21T14:09:50.581 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:50 smithi049 ceph-mon[30080]: 6.1b scrub starts 2024-09-21T14:09:50.581 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:50 smithi049 ceph-mon[30080]: 6.1b scrub ok 2024-09-21T14:09:50.610 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:50.639 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:50.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:50.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:50.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:50 smithi187 ceph-mon[33032]: 6.b scrub starts 2024-09-21T14:09:50.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:50 smithi187 ceph-mon[33032]: 6.b scrub ok 2024-09-21T14:09:50.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:50 smithi187 ceph-mon[33032]: 4.9 scrub starts 2024-09-21T14:09:50.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:50 smithi187 ceph-mon[33032]: 4.9 scrub ok 2024-09-21T14:09:50.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:50 smithi187 ceph-mon[33032]: 6.1b scrub starts 2024-09-21T14:09:50.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:50 smithi187 ceph-mon[33032]: 6.1b scrub ok 2024-09-21T14:09:50.695 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:09:50 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:09:50.741 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:09:50.785 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:50.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:50.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:09:50.886 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:50.894 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:09:50 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:09:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:09:50.920 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:50.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:50.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:51.022 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:09:51.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:51.051 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:51.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:09:51.152 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:09:51.181 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:51.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:51.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:09:51.286 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:51.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:51.319 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:51.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:51.420 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:09:51.460 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:51.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:51.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:09:51.465 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-09-21T14:09:51.563 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:51.590 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T14:09:51.592 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:51.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:51.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:51.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:51 smithi187 ceph-mon[33032]: pgmap v1069: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 2.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 138/130089 objects misplaced (0.106%); 1.4 MiB/s, 21 objects/s recovering 2024-09-21T14:09:51.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:51 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:09:51.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:51 smithi187 ceph-mon[33032]: 2.2 scrub starts 2024-09-21T14:09:51.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:51 smithi187 ceph-mon[33032]: 2.2 scrub ok 2024-09-21T14:09:51.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:51 smithi187 ceph-mon[33032]: 11.e scrub starts 2024-09-21T14:09:51.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:51 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3644641260' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:09:51.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:51 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:09:51.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:51 smithi187 ceph-mon[33032]: 11.e scrub ok 2024-09-21T14:09:51.695 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:09:51.720 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:51.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:51.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:09:51.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:51 smithi049 ceph-mon[40489]: pgmap v1069: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 2.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 138/130089 objects misplaced (0.106%); 1.4 MiB/s, 21 objects/s recovering 2024-09-21T14:09:51.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:51 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:09:51.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:51 smithi049 ceph-mon[40489]: 2.2 scrub starts 2024-09-21T14:09:51.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:51 smithi049 ceph-mon[40489]: 2.2 scrub ok 2024-09-21T14:09:51.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:51 smithi049 ceph-mon[40489]: 11.e scrub starts 2024-09-21T14:09:51.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:51 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3644641260' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:09:51.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:51 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:09:51.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:51 smithi049 ceph-mon[40489]: 11.e scrub ok 2024-09-21T14:09:51.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:51 smithi049 ceph-mon[30080]: pgmap v1069: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 2.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 138/130089 objects misplaced (0.106%); 1.4 MiB/s, 21 objects/s recovering 2024-09-21T14:09:51.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:51 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:09:51.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:51 smithi049 ceph-mon[30080]: 2.2 scrub starts 2024-09-21T14:09:51.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:51 smithi049 ceph-mon[30080]: 2.2 scrub ok 2024-09-21T14:09:51.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:51 smithi049 ceph-mon[30080]: 11.e scrub starts 2024-09-21T14:09:51.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:51 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3644641260' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:09:51.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:51 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:09:51.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:51 smithi049 ceph-mon[30080]: 11.e scrub ok 2024-09-21T14:09:51.822 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:51.851 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:51.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:51.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:51.953 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:09:51.982 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:51.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:51.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:09:52.084 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:09:52.109 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:52.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:52.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:09:52.211 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:09:52.238 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:52.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:52.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:09:52.339 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:52.371 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:52.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:52.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:52.459 INFO:teuthology.orchestra.run.smithi049.stderr:set osd.0 primary-affinity to 0 (802) 2024-09-21T14:09:52.472 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:09:52.499 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:52.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:52.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:09:52.600 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:09:52.624 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:52.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:52.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:09:52.725 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:09:52.746 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:52.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:52.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:09:52.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:52 smithi049 ceph-mon[30080]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:09:52.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:52 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T14:09:52.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:52 smithi049 ceph-mon[30080]: osdmap e530: 8 total, 8 up, 8 in 2024-09-21T14:09:52.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:52 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3558390995' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 0.0} : dispatch 2024-09-21T14:09:52.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:52 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T14:09:52.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:52 smithi049 ceph-mon[40489]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:09:52.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:52 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T14:09:52.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:52 smithi049 ceph-mon[40489]: osdmap e530: 8 total, 8 up, 8 in 2024-09-21T14:09:52.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:52 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3558390995' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 0.0} : dispatch 2024-09-21T14:09:52.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:52 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T14:09:52.847 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:09:52.869 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:52.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:52.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:09:52.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:52 smithi187 ceph-mon[33032]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:09:52.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:52 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T14:09:52.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:52 smithi187 ceph-mon[33032]: osdmap e530: 8 total, 8 up, 8 in 2024-09-21T14:09:52.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:52 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3558390995' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 0.0} : dispatch 2024-09-21T14:09:52.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:52 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T14:09:52.970 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:09:52.993 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:52.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:52.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:09:53.094 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:53.101 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T14:09:53.101 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-09-21T14:09:53.101 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-09-21T14:09:53.101 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-09-21T14:09:53.101 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.101 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25421.0:172314 9.4 e315/260 9:2b749f7e:::benchmark_data_smithi049_138504_object50906:head v 315'10563, mlcod=315'10559)", 2024-09-21T14:09:53.101 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:00:33.250136+0000", 2024-09-21T14:09:53.101 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 559.53385679899998, 2024-09-21T14:09:53.101 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.270468655, 2024-09-21T14:09:53.102 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:09:53.102 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:09:53.102 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:09:53.102 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:09:53.102 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.102 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:09:53.102 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250136+0000", 2024-09-21T14:09:53.102 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:53.102 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.102 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.102 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:09:53.102 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250136+0000", 2024-09-21T14:09:53.102 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:53.103 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.103 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.103 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:09:53.103 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250137+0000", 2024-09-21T14:09:53.103 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1999999999999999e-07 2024-09-21T14:09:53.103 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.103 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.103 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:09:53.103 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250142+0000", 2024-09-21T14:09:53.103 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5609999999999998e-06 2024-09-21T14:09:53.103 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.103 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.103 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:09:53.103 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250150+0000", 2024-09-21T14:09:53.104 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1310000000000006e-06 2024-09-21T14:09:53.104 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.104 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.104 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:09:53.104 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250156+0000", 2024-09-21T14:09:53.104 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7740000000000001e-06 2024-09-21T14:09:53.104 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.104 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.104 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:09:53.104 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.253391+0000", 2024-09-21T14:09:53.105 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0032349969999999999 2024-09-21T14:09:53.105 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.105 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.105 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:09:53.105 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.253410+0000", 2024-09-21T14:09:53.105 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9097e-05 2024-09-21T14:09:53.105 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.105 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.105 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:09:53.105 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.520555+0000", 2024-09-21T14:09:53.105 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26714502600000001 2024-09-21T14:09:53.105 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.105 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.106 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:09:53.106 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.520605+0000", 2024-09-21T14:09:53.106 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9449e-05 2024-09-21T14:09:53.106 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:53.106 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:09:53.106 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:53.106 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.106 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.106 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25421.0:172319 9.4 e315/260 9:21e132db:::benchmark_data_smithi049_138504_object50911:head v 315'10564, mlcod=315'10559)", 2024-09-21T14:09:53.106 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:00:33.250191+0000", 2024-09-21T14:09:53.106 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 559.53380181499995, 2024-09-21T14:09:53.106 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.270657121, 2024-09-21T14:09:53.106 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:09:53.107 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:09:53.107 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:09:53.107 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:09:53.107 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.107 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:09:53.107 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250191+0000", 2024-09-21T14:09:53.107 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:53.107 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.107 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.107 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:09:53.107 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250191+0000", 2024-09-21T14:09:53.107 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:53.107 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.107 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.108 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:09:53.108 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250191+0000", 2024-09-21T14:09:53.108 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8799999999999998e-07 2024-09-21T14:09:53.108 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.108 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.108 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:09:53.108 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250195+0000", 2024-09-21T14:09:53.108 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.647e-06 2024-09-21T14:09:53.108 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.108 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.108 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:09:53.108 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250201+0000", 2024-09-21T14:09:53.108 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4200000000000004e-06 2024-09-21T14:09:53.108 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.108 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.108 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:09:53.109 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.250206+0000", 2024-09-21T14:09:53.109 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.994e-06 2024-09-21T14:09:53.109 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.109 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.109 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:09:53.109 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.256110+0000", 2024-09-21T14:09:53.109 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0059039840000000001 2024-09-21T14:09:53.109 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.109 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.109 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:09:53.109 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.256130+0000", 2024-09-21T14:09:53.109 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.98e-05 2024-09-21T14:09:53.109 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.109 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.109 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:09:53.109 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.520816+0000", 2024-09-21T14:09:53.109 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.264685528 2024-09-21T14:09:53.110 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.110 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.110 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:09:53.110 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.520848+0000", 2024-09-21T14:09:53.110 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2360000000000002e-05 2024-09-21T14:09:53.110 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:53.110 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:09:53.110 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:53.110 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.110 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.110 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25421.0:172326 9.4 e315/260 9:22a8fc8a:::benchmark_data_smithi049_138504_object50918:head v 315'10565, mlcod=315'10560)", 2024-09-21T14:09:53.110 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:00:33.254759+0000", 2024-09-21T14:09:53.110 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 559.529233525, 2024-09-21T14:09:53.110 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26611502300000001, 2024-09-21T14:09:53.110 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:09:53.111 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:09:53.111 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:09:53.111 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:09:53.111 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.111 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:09:53.111 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254759+0000", 2024-09-21T14:09:53.111 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:53.111 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.111 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.111 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:09:53.111 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254759+0000", 2024-09-21T14:09:53.111 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:53.111 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.111 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.111 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:09:53.112 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254760+0000", 2024-09-21T14:09:53.112 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6400000000000001e-07 2024-09-21T14:09:53.112 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.112 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.112 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:09:53.112 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254854+0000", 2024-09-21T14:09:53.112 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4320999999999994e-05 2024-09-21T14:09:53.112 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.112 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.112 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:09:53.112 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254865+0000", 2024-09-21T14:09:53.112 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.101e-05 2024-09-21T14:09:53.112 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.112 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.112 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:09:53.112 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254872+0000", 2024-09-21T14:09:53.113 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2539999999999998e-06 2024-09-21T14:09:53.113 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.113 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.113 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:09:53.113 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.258679+0000", 2024-09-21T14:09:53.113 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.003807342 2024-09-21T14:09:53.113 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.113 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.113 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:09:53.113 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.258699+0000", 2024-09-21T14:09:53.113 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.039e-05 2024-09-21T14:09:53.113 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.113 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.113 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:09:53.113 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.520851+0000", 2024-09-21T14:09:53.114 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26215118900000001 2024-09-21T14:09:53.114 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.114 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.114 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:09:53.114 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.520874+0000", 2024-09-21T14:09:53.114 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3753000000000001e-05 2024-09-21T14:09:53.114 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:53.114 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:09:53.114 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:53.114 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.114 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.114 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.25421.0:172331 9.f e315/260 9:f6449e5b:::benchmark_data_smithi049_138504_object50923:head v 315'10871, mlcod=315'10867)", 2024-09-21T14:09:53.114 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:00:33.254942+0000", 2024-09-21T14:09:53.114 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 559.52905114800001, 2024-09-21T14:09:53.114 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.274133181, 2024-09-21T14:09:53.115 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:09:53.115 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:09:53.115 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:09:53.115 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:09:53.115 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.115 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:09:53.115 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254942+0000", 2024-09-21T14:09:53.115 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:53.115 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.115 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.115 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:09:53.115 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254942+0000", 2024-09-21T14:09:53.115 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:53.115 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.116 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.116 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:09:53.116 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254942+0000", 2024-09-21T14:09:53.116 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5700000000000002e-07 2024-09-21T14:09:53.116 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.116 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.116 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:09:53.116 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254949+0000", 2024-09-21T14:09:53.116 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.024e-06 2024-09-21T14:09:53.116 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.116 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.116 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:09:53.116 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254959+0000", 2024-09-21T14:09:53.116 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6390000000000004e-06 2024-09-21T14:09:53.117 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.117 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.117 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:09:53.117 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.254965+0000", 2024-09-21T14:09:53.117 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6420000000000004e-06 2024-09-21T14:09:53.117 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.117 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.117 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:09:53.117 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.259407+0000", 2024-09-21T14:09:53.117 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0044420140000000002 2024-09-21T14:09:53.117 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.117 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.117 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:09:53.117 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.259425+0000", 2024-09-21T14:09:53.117 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7937000000000001e-05 2024-09-21T14:09:53.118 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.118 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.118 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:09:53.118 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.529033+0000", 2024-09-21T14:09:53.118 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.26960846599999999 2024-09-21T14:09:53.118 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.118 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.118 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:09:53.118 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.529075+0000", 2024-09-21T14:09:53.118 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1801999999999997e-05 2024-09-21T14:09:53.118 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:53.118 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:09:53.118 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:53.119 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.119 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.119 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.25421.0:172333 9.2 9:461458cc:::benchmark_data_smithi049_138504_object50925:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-09-21T14:09:53.119 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:00:33.258960+0000", 2024-09-21T14:09:53.119 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 559.52503301299998, 2024-09-21T14:09:53.119 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.261926308, 2024-09-21T14:09:53.119 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:09:53.119 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:09:53.119 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:09:53.119 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:09:53.119 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.25421", 2024-09-21T14:09:53.119 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:09:53.119 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 172333 2024-09-21T14:09:53.119 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.119 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:09:53.119 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.120 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:09:53.120 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.258960+0000", 2024-09-21T14:09:53.120 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:53.120 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.120 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.120 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:09:53.120 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.258960+0000", 2024-09-21T14:09:53.120 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:53.120 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.120 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.120 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:09:53.120 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.258961+0000", 2024-09-21T14:09:53.120 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5629999999999999e-06 2024-09-21T14:09:53.120 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.120 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.121 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:09:53.121 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.258967+0000", 2024-09-21T14:09:53.121 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5079999999999996e-06 2024-09-21T14:09:53.121 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.121 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.121 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:09:53.121 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.258979+0000", 2024-09-21T14:09:53.121 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1897e-05 2024-09-21T14:09:53.121 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.121 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.121 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:09:53.121 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.258986+0000", 2024-09-21T14:09:53.121 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8739999999999997e-06 2024-09-21T14:09:53.121 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.122 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.122 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:09:53.122 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.259036+0000", 2024-09-21T14:09:53.122 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0695000000000002e-05 2024-09-21T14:09:53.122 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.122 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.122 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:09:53.122 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.259144+0000", 2024-09-21T14:09:53.122 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000107462 2024-09-21T14:09:53.122 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.122 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.122 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,2", 2024-09-21T14:09:53.122 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.259252+0000", 2024-09-21T14:09:53.122 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000107667 2024-09-21T14:09:53.123 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.123 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.123 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:09:53.123 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.517843+0000", 2024-09-21T14:09:53.123 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.25859111400000001 2024-09-21T14:09:53.123 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.123 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.123 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:53.123 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.517994+0000", 2024-09-21T14:09:53.123 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00015150299999999999 2024-09-21T14:09:53.123 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.123 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.123 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:53.123 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.520805+0000", 2024-09-21T14:09:53.124 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0028113489999999999 2024-09-21T14:09:53.124 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.124 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.124 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:09:53.124 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.520850+0000", 2024-09-21T14:09:53.124 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4333999999999999e-05 2024-09-21T14:09:53.124 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.124 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.124 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:09:53.124 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:00:33.520886+0000", 2024-09-21T14:09:53.124 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6341999999999999e-05 2024-09-21T14:09:53.124 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:53.124 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:09:53.124 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:53.124 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.125 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.125 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.16068.0:208862 10.d e441/434 10:b40252c3:::benchmark_data_smithi049_176811_object81285:head v 441'13126, mlcod=441'13122)", 2024-09-21T14:09:53.125 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:05:54.047328+0000", 2024-09-21T14:09:53.125 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 238.736664706, 2024-09-21T14:09:53.125 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.31640931900000002, 2024-09-21T14:09:53.125 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:09:53.125 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:09:53.125 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:09:53.125 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:09:53.125 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.125 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:09:53.125 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047328+0000", 2024-09-21T14:09:53.125 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:53.125 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.126 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.126 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:09:53.126 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047328+0000", 2024-09-21T14:09:53.126 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:53.126 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.126 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.126 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:09:53.126 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047329+0000", 2024-09-21T14:09:53.126 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0410000000000001e-06 2024-09-21T14:09:53.126 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.126 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.126 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:09:53.126 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047338+0000", 2024-09-21T14:09:53.126 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6160000000000002e-06 2024-09-21T14:09:53.127 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.127 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.127 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:09:53.127 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047348+0000", 2024-09-21T14:09:53.127 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0688e-05 2024-09-21T14:09:53.127 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.127 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.127 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:09:53.127 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047356+0000", 2024-09-21T14:09:53.127 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3350000000000001e-06 2024-09-21T14:09:53.127 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.127 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.127 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:09:53.127 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047399+0000", 2024-09-21T14:09:53.127 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2833000000000001e-05 2024-09-21T14:09:53.128 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.128 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.128 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:09:53.128 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047415+0000", 2024-09-21T14:09:53.128 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6662000000000001e-05 2024-09-21T14:09:53.128 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.128 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.128 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:09:53.128 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.363700+0000", 2024-09-21T14:09:53.128 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.31628446799999999 2024-09-21T14:09:53.128 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.128 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.128 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:09:53.128 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.363737+0000", 2024-09-21T14:09:53.129 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7676000000000003e-05 2024-09-21T14:09:53.129 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:53.129 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:09:53.129 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:53.129 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.129 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.129 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.16068.0:208866 10.3 e441/381 10:c8242cfb:::benchmark_data_smithi049_176811_object81289:head v 441'12975, mlcod=441'12973)", 2024-09-21T14:09:53.129 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:05:54.047688+0000", 2024-09-21T14:09:53.129 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 238.73630515400001, 2024-09-21T14:09:53.129 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.31391443499999999, 2024-09-21T14:09:53.129 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:09:53.129 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:09:53.129 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:09:53.129 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:09:53.130 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.130 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:09:53.130 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047688+0000", 2024-09-21T14:09:53.130 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:53.130 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.130 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.130 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:09:53.130 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047688+0000", 2024-09-21T14:09:53.130 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:53.130 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.130 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.130 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:09:53.130 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047688+0000", 2024-09-21T14:09:53.131 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9999999999999997e-07 2024-09-21T14:09:53.131 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.131 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.131 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:09:53.131 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047693+0000", 2024-09-21T14:09:53.131 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7759999999999997e-06 2024-09-21T14:09:53.131 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.131 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.131 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:09:53.131 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.357055+0000", 2024-09-21T14:09:53.131 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.309361422 2024-09-21T14:09:53.131 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.131 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.131 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:09:53.132 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.357070+0000", 2024-09-21T14:09:53.132 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5311e-05 2024-09-21T14:09:53.132 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.132 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.132 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:09:53.132 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.357133+0000", 2024-09-21T14:09:53.132 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2611000000000001e-05 2024-09-21T14:09:53.132 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.132 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.132 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:09:53.132 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.357154+0000", 2024-09-21T14:09:53.132 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1376e-05 2024-09-21T14:09:53.132 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.132 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.133 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:09:53.133 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.361561+0000", 2024-09-21T14:09:53.133 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0044073599999999999 2024-09-21T14:09:53.133 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.133 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.133 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:09:53.133 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.361602+0000", 2024-09-21T14:09:53.133 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0878999999999999e-05 2024-09-21T14:09:53.133 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:53.133 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:09:53.133 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:53.133 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.133 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.133 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:61790 11.b 11:d012fa8a:::benchmark_data_smithi049_206132_object61789:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e476)", 2024-09-21T14:09:53.134 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:07:27.526621+0000", 2024-09-21T14:09:53.134 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 145.25737171599999, 2024-09-21T14:09:53.134 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21106169299999999, 2024-09-21T14:09:53.134 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:09:53.134 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:09:53.134 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:09:53.134 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:09:53.134 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:09:53.134 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:09:53.134 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 61790 2024-09-21T14:09:53.134 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.134 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:09:53.134 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.134 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:09:53.135 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.526621+0000", 2024-09-21T14:09:53.135 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:53.135 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.135 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.135 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:09:53.135 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.526621+0000", 2024-09-21T14:09:53.135 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:53.135 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.135 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.135 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:09:53.135 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.526623+0000", 2024-09-21T14:09:53.135 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.725e-06 2024-09-21T14:09:53.135 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.135 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.136 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:09:53.136 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.526655+0000", 2024-09-21T14:09:53.136 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1717000000000001e-05 2024-09-21T14:09:53.136 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.136 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.136 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:09:53.136 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.526667+0000", 2024-09-21T14:09:53.136 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2121e-05 2024-09-21T14:09:53.136 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.136 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.136 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:09:53.136 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.526674+0000", 2024-09-21T14:09:53.136 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7330000000000003e-06 2024-09-21T14:09:53.136 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.137 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.137 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:09:53.137 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.527240+0000", 2024-09-21T14:09:53.137 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00056578599999999998 2024-09-21T14:09:53.137 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.137 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.137 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:09:53.137 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.527424+0000", 2024-09-21T14:09:53.137 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00018419199999999999 2024-09-21T14:09:53.137 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.137 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.137 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,1", 2024-09-21T14:09:53.137 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.527566+0000", 2024-09-21T14:09:53.137 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014196599999999999 2024-09-21T14:09:53.138 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.138 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.138 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:09:53.138 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.532651+0000", 2024-09-21T14:09:53.138 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0050843640000000001 2024-09-21T14:09:53.138 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.138 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.138 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:53.138 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.533057+0000", 2024-09-21T14:09:53.138 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00040586600000000001 2024-09-21T14:09:53.138 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.138 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.138 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:53.139 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.737591+0000", 2024-09-21T14:09:53.139 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.204534191 2024-09-21T14:09:53.139 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.139 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.139 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:09:53.139 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.737643+0000", 2024-09-21T14:09:53.139 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2555000000000001e-05 2024-09-21T14:09:53.139 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.139 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.139 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:09:53.139 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.737683+0000", 2024-09-21T14:09:53.139 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9477000000000002e-05 2024-09-21T14:09:53.139 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:53.140 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:09:53.140 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:53.140 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.140 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.140 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:61806 11.b 11:d5ce9cd2:::benchmark_data_smithi049_206132_object61805:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e476)", 2024-09-21T14:09:53.140 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:07:27.535767+0000", 2024-09-21T14:09:53.140 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 145.24822541399999, 2024-09-21T14:09:53.140 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.206897624, 2024-09-21T14:09:53.140 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:09:53.140 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:09:53.140 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:09:53.140 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:09:53.140 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:09:53.140 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:09:53.140 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 61806 2024-09-21T14:09:53.141 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.141 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:09:53.141 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.141 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:09:53.141 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.535767+0000", 2024-09-21T14:09:53.141 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:53.141 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.141 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.141 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:09:53.141 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.535767+0000", 2024-09-21T14:09:53.141 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:53.141 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.141 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.142 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:09:53.142 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.535769+0000", 2024-09-21T14:09:53.142 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2640000000000001e-06 2024-09-21T14:09:53.142 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.142 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.142 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:09:53.142 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.535792+0000", 2024-09-21T14:09:53.142 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3393000000000001e-05 2024-09-21T14:09:53.142 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.142 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.142 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:09:53.142 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.535808+0000", 2024-09-21T14:09:53.142 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5736e-05 2024-09-21T14:09:53.143 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.143 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.143 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:09:53.143 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.535817+0000", 2024-09-21T14:09:53.143 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4690000000000003e-06 2024-09-21T14:09:53.143 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.143 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.143 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:09:53.143 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.535874+0000", 2024-09-21T14:09:53.143 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6696000000000001e-05 2024-09-21T14:09:53.143 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.143 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.143 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:09:53.143 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.535985+0000", 2024-09-21T14:09:53.144 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000110908 2024-09-21T14:09:53.144 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.144 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.144 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,1", 2024-09-21T14:09:53.144 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.536105+0000", 2024-09-21T14:09:53.144 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000119753 2024-09-21T14:09:53.144 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.144 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.144 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:09:53.144 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.537905+0000", 2024-09-21T14:09:53.144 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001800779 2024-09-21T14:09:53.144 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.144 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.144 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:53.144 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.538385+0000", 2024-09-21T14:09:53.145 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00047917499999999998 2024-09-21T14:09:53.145 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.145 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.145 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:53.145 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.742566+0000", 2024-09-21T14:09:53.145 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.20418138899999999 2024-09-21T14:09:53.145 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.145 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.145 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:09:53.145 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.742621+0000", 2024-09-21T14:09:53.145 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4951999999999998e-05 2024-09-21T14:09:53.145 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.145 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.145 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:09:53.145 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.742665+0000", 2024-09-21T14:09:53.146 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4110000000000003e-05 2024-09-21T14:09:53.146 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:53.146 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:09:53.146 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:53.146 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.146 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.146 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:61815 11.9 11:9bfc68e2:::benchmark_data_smithi049_206132_object61814:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e476)", 2024-09-21T14:09:53.146 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:07:27.542890+0000", 2024-09-21T14:09:53.146 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 145.241102584, 2024-09-21T14:09:53.146 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.19371184399999999, 2024-09-21T14:09:53.146 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:09:53.146 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:09:53.146 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:09:53.147 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:09:53.147 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:09:53.147 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:09:53.147 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 61815 2024-09-21T14:09:53.147 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.147 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:09:53.147 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.147 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:09:53.147 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.542890+0000", 2024-09-21T14:09:53.147 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:53.147 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.147 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.147 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:09:53.147 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.542890+0000", 2024-09-21T14:09:53.147 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:53.147 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.148 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.148 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:09:53.148 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.542891+0000", 2024-09-21T14:09:53.148 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4300000000000001e-07 2024-09-21T14:09:53.148 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.148 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.148 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:09:53.148 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.542910+0000", 2024-09-21T14:09:53.148 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9029e-05 2024-09-21T14:09:53.148 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.148 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.148 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:09:53.148 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.542921+0000", 2024-09-21T14:09:53.148 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1037e-05 2024-09-21T14:09:53.148 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.148 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.149 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:09:53.149 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.542927+0000", 2024-09-21T14:09:53.149 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6879999999999996e-06 2024-09-21T14:09:53.149 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.149 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.149 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:09:53.149 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.542966+0000", 2024-09-21T14:09:53.149 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8954e-05 2024-09-21T14:09:53.149 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.149 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.149 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:09:53.149 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.543073+0000", 2024-09-21T14:09:53.149 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000107032 2024-09-21T14:09:53.149 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.149 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.150 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,6", 2024-09-21T14:09:53.150 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.543165+0000", 2024-09-21T14:09:53.150 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1959000000000005e-05 2024-09-21T14:09:53.150 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.150 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.150 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:09:53.150 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.544789+0000", 2024-09-21T14:09:53.150 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0016236499999999999 2024-09-21T14:09:53.150 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.150 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.150 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:53.150 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.545917+0000", 2024-09-21T14:09:53.150 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0011286169999999999 2024-09-21T14:09:53.150 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.150 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.151 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:53.151 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.736484+0000", 2024-09-21T14:09:53.151 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.19056731700000001 2024-09-21T14:09:53.151 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.151 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.151 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:09:53.151 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.736554+0000", 2024-09-21T14:09:53.151 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9183999999999994e-05 2024-09-21T14:09:53.151 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.151 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.151 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:09:53.151 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.736602+0000", 2024-09-21T14:09:53.151 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8433999999999997e-05 2024-09-21T14:09:53.151 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:53.152 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:09:53.152 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:53.152 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.152 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.152 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:61817 11.9 11:9eabf162:::benchmark_data_smithi049_206132_object61816:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e476)", 2024-09-21T14:09:53.152 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:07:27.548213+0000", 2024-09-21T14:09:53.152 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 145.23577977599999, 2024-09-21T14:09:53.152 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.19485654799999999, 2024-09-21T14:09:53.152 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:09:53.152 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:09:53.152 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:09:53.152 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:09:53.152 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:09:53.152 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:09:53.153 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 61817 2024-09-21T14:09:53.153 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.153 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:09:53.153 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.153 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:09:53.153 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.548213+0000", 2024-09-21T14:09:53.153 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:53.153 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.153 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.153 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:09:53.153 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.548213+0000", 2024-09-21T14:09:53.153 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:53.153 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.153 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.153 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:09:53.154 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.548214+0000", 2024-09-21T14:09:53.154 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0100000000000001e-06 2024-09-21T14:09:53.154 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.154 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.154 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:09:53.154 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.548233+0000", 2024-09-21T14:09:53.154 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9388000000000001e-05 2024-09-21T14:09:53.154 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.154 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.154 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:09:53.154 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.548244+0000", 2024-09-21T14:09:53.154 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0694e-05 2024-09-21T14:09:53.154 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.154 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.155 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:09:53.155 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.548251+0000", 2024-09-21T14:09:53.155 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3320000000000003e-06 2024-09-21T14:09:53.155 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.155 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.155 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:09:53.155 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.548302+0000", 2024-09-21T14:09:53.155 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0180999999999999e-05 2024-09-21T14:09:53.155 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.155 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.155 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:09:53.155 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.548384+0000", 2024-09-21T14:09:53.155 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2410000000000005e-05 2024-09-21T14:09:53.155 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.156 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.156 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,6", 2024-09-21T14:09:53.156 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.548470+0000", 2024-09-21T14:09:53.156 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5562000000000005e-05 2024-09-21T14:09:53.156 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.156 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.156 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:09:53.156 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.549773+0000", 2024-09-21T14:09:53.156 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001303331 2024-09-21T14:09:53.156 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.156 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.156 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:53.156 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.550545+0000", 2024-09-21T14:09:53.156 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00077198800000000004 2024-09-21T14:09:53.156 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.157 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.157 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:53.157 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.742904+0000", 2024-09-21T14:09:53.157 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.192359264 2024-09-21T14:09:53.157 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.157 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.157 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:09:53.157 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.743029+0000", 2024-09-21T14:09:53.157 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012438299999999999 2024-09-21T14:09:53.157 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.157 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.157 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:09:53.157 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.743070+0000", 2024-09-21T14:09:53.157 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1004999999999998e-05 2024-09-21T14:09:53.157 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:53.158 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:09:53.158 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:53.158 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.158 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.158 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:117730 11.b 11:d6b3af6b:::benchmark_data_smithi049_206132_object117729:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e498)", 2024-09-21T14:09:53.158 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:08:18.231024+0000", 2024-09-21T14:09:53.158 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 94.552968578000005, 2024-09-21T14:09:53.158 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.132113692, 2024-09-21T14:09:53.158 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:09:53.158 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:09:53.158 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:09:53.158 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:09:53.158 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:09:53.158 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:09:53.159 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 117730 2024-09-21T14:09:53.159 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.159 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:09:53.159 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.159 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:09:53.159 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:18.231024+0000", 2024-09-21T14:09:53.159 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:53.159 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.159 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.159 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:09:53.159 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:18.231024+0000", 2024-09-21T14:09:53.159 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:53.159 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.159 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.160 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:09:53.160 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:18.231027+0000", 2024-09-21T14:09:53.160 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2910000000000002e-06 2024-09-21T14:09:53.160 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.160 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.160 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:09:53.160 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:18.231053+0000", 2024-09-21T14:09:53.160 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6534999999999999e-05 2024-09-21T14:09:53.160 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.160 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.160 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:09:53.160 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:18.231067+0000", 2024-09-21T14:09:53.160 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3502e-05 2024-09-21T14:09:53.160 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.160 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.161 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:09:53.161 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:18.231074+0000", 2024-09-21T14:09:53.161 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6459999999999993e-06 2024-09-21T14:09:53.161 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.161 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.161 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:09:53.161 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:18.231159+0000", 2024-09-21T14:09:53.161 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4608e-05 2024-09-21T14:09:53.161 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.161 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.161 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:09:53.161 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:18.231304+0000", 2024-09-21T14:09:53.161 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000145196 2024-09-21T14:09:53.161 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.161 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.162 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,1", 2024-09-21T14:09:53.162 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:18.231450+0000", 2024-09-21T14:09:53.162 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014589599999999999 2024-09-21T14:09:53.162 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.162 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.162 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:09:53.162 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:18.234508+0000", 2024-09-21T14:09:53.162 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.003058032 2024-09-21T14:09:53.162 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.162 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.162 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:53.162 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:18.237611+0000", 2024-09-21T14:09:53.162 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0031031919999999998 2024-09-21T14:09:53.162 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.163 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.163 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:53.163 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:18.362984+0000", 2024-09-21T14:09:53.163 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.12537323 2024-09-21T14:09:53.163 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.163 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.163 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:09:53.163 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:18.363092+0000", 2024-09-21T14:09:53.163 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000107804 2024-09-21T14:09:53.163 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.163 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.163 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:09:53.163 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:18.363138+0000", 2024-09-21T14:09:53.163 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5760000000000002e-05 2024-09-21T14:09:53.163 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:53.164 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:09:53.164 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:53.164 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.164 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.164 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:150900 11.b 11:d3b5caa5:::benchmark_data_smithi049_206132_object27521:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e506)", 2024-09-21T14:09:53.164 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:08:50.049533+0000", 2024-09-21T14:09:53.164 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 62.734459796000003, 2024-09-21T14:09:53.164 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.138400934, 2024-09-21T14:09:53.164 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:09:53.164 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:09:53.164 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:09:53.164 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:09:53.164 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:09:53.164 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:09:53.164 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 150900 2024-09-21T14:09:53.165 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.165 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:09:53.165 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.165 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:09:53.165 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:50.049533+0000", 2024-09-21T14:09:53.165 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:53.165 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.165 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.165 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:09:53.165 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:50.049533+0000", 2024-09-21T14:09:53.165 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:53.165 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.165 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.165 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:09:53.165 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:50.049535+0000", 2024-09-21T14:09:53.166 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7069999999999999e-06 2024-09-21T14:09:53.166 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.166 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.166 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:09:53.166 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:50.049540+0000", 2024-09-21T14:09:53.166 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7060000000000001e-06 2024-09-21T14:09:53.166 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.166 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.166 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:09:53.166 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:50.049620+0000", 2024-09-21T14:09:53.166 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9747999999999996e-05 2024-09-21T14:09:53.166 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.166 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.166 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:09:53.167 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:50.049628+0000", 2024-09-21T14:09:53.167 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5000000000000002e-06 2024-09-21T14:09:53.167 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.167 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.167 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:09:53.167 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:50.049777+0000", 2024-09-21T14:09:53.167 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014942199999999999 2024-09-21T14:09:53.167 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.167 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.167 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:09:53.167 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:50.049882+0000", 2024-09-21T14:09:53.167 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000104586 2024-09-21T14:09:53.167 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.167 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.167 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,1", 2024-09-21T14:09:53.168 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:50.050005+0000", 2024-09-21T14:09:53.168 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012326399999999999 2024-09-21T14:09:53.168 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.168 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.168 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:09:53.168 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:50.053617+0000", 2024-09-21T14:09:53.168 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0036119070000000001 2024-09-21T14:09:53.168 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.168 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.168 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:53.168 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:50.054763+0000", 2024-09-21T14:09:53.168 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001146553 2024-09-21T14:09:53.168 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.168 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.169 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:53.169 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:50.187653+0000", 2024-09-21T14:09:53.169 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.13288966699999999 2024-09-21T14:09:53.169 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.169 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.169 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:09:53.169 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:50.187858+0000", 2024-09-21T14:09:53.169 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00020505799999999999 2024-09-21T14:09:53.169 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.169 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.169 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:09:53.169 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:50.187934+0000", 2024-09-21T14:09:53.169 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5815999999999994e-05 2024-09-21T14:09:53.169 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:53.170 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:09:53.170 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:53.170 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.170 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.170 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:165838 11.a 11:52af266b:::benchmark_data_smithi049_206132_object42459:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e516)", 2024-09-21T14:09:53.170 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:09:07.131235+0000", 2024-09-21T14:09:53.170 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 45.652757915999999, 2024-09-21T14:09:53.170 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.16957782099999999, 2024-09-21T14:09:53.170 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:09:53.170 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:09:53.170 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:09:53.170 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:09:53.170 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:09:53.170 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:09:53.171 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 165838 2024-09-21T14:09:53.171 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.171 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:09:53.171 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.171 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:09:53.171 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.131235+0000", 2024-09-21T14:09:53.171 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:53.171 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.171 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.171 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:09:53.171 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.131235+0000", 2024-09-21T14:09:53.171 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:53.171 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.171 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.172 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:09:53.172 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.131236+0000", 2024-09-21T14:09:53.172 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3859999999999999e-06 2024-09-21T14:09:53.172 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.172 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.172 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:09:53.172 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.131241+0000", 2024-09-21T14:09:53.172 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1100000000000002e-06 2024-09-21T14:09:53.172 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.172 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.172 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:09:53.172 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.131256+0000", 2024-09-21T14:09:53.172 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4250000000000001e-05 2024-09-21T14:09:53.172 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.173 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.173 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:09:53.173 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.131288+0000", 2024-09-21T14:09:53.173 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2449000000000001e-05 2024-09-21T14:09:53.173 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.173 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.173 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:09:53.173 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.131352+0000", 2024-09-21T14:09:53.173 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3961999999999995e-05 2024-09-21T14:09:53.173 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.173 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.173 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:09:53.173 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.131472+0000", 2024-09-21T14:09:53.173 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000119431 2024-09-21T14:09:53.174 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.174 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.174 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,6", 2024-09-21T14:09:53.174 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.131583+0000", 2024-09-21T14:09:53.174 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000111369 2024-09-21T14:09:53.174 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.174 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.174 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:09:53.174 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.135654+0000", 2024-09-21T14:09:53.174 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0040712389999999999 2024-09-21T14:09:53.174 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.174 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.174 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:53.174 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.136288+0000", 2024-09-21T14:09:53.175 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00063429000000000003 2024-09-21T14:09:53.175 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.175 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.175 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:53.175 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.300533+0000", 2024-09-21T14:09:53.175 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.164244749 2024-09-21T14:09:53.175 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.175 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.175 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:09:53.175 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.300763+0000", 2024-09-21T14:09:53.175 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00022982100000000001 2024-09-21T14:09:53.175 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.175 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.175 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:09:53.175 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.300813+0000", 2024-09-21T14:09:53.176 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9765000000000002e-05 2024-09-21T14:09:53.176 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:53.176 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:09:53.176 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:53.176 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.176 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.176 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:169603 11.b 11:d45540b4:::benchmark_data_smithi049_206132_object46224:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e516)", 2024-09-21T14:09:53.176 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:09:11.215200+0000", 2024-09-21T14:09:53.176 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 41.568793288000002, 2024-09-21T14:09:53.176 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.12844223599999999, 2024-09-21T14:09:53.176 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:09:53.176 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:09:53.176 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:09:53.176 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:09:53.176 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:09:53.177 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:09:53.177 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 169603 2024-09-21T14:09:53.177 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.177 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:09:53.177 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.177 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:09:53.177 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:11.215200+0000", 2024-09-21T14:09:53.177 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:53.177 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.177 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.177 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:09:53.177 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:11.215200+0000", 2024-09-21T14:09:53.177 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:53.177 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.177 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.178 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:09:53.178 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:11.215201+0000", 2024-09-21T14:09:53.178 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5269999999999999e-06 2024-09-21T14:09:53.178 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.178 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.178 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:09:53.178 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:11.215207+0000", 2024-09-21T14:09:53.178 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0279999999999997e-06 2024-09-21T14:09:53.178 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.178 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.178 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:09:53.178 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:11.215220+0000", 2024-09-21T14:09:53.179 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2659e-05 2024-09-21T14:09:53.179 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.179 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.179 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:09:53.179 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:11.215227+0000", 2024-09-21T14:09:53.179 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3980000000000002e-06 2024-09-21T14:09:53.179 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.179 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.179 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:09:53.179 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:11.216471+0000", 2024-09-21T14:09:53.179 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0012435160000000001 2024-09-21T14:09:53.179 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.179 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.179 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:09:53.179 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:11.216595+0000", 2024-09-21T14:09:53.180 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012395 2024-09-21T14:09:53.180 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.180 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.180 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,1", 2024-09-21T14:09:53.180 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:11.216702+0000", 2024-09-21T14:09:53.180 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010742399999999999 2024-09-21T14:09:53.180 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.180 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.180 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:09:53.180 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:11.221902+0000", 2024-09-21T14:09:53.180 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0051998929999999997 2024-09-21T14:09:53.180 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.180 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.181 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:53.181 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:11.231425+0000", 2024-09-21T14:09:53.181 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0095234840000000005 2024-09-21T14:09:53.181 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.181 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.181 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:53.181 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:11.343437+0000", 2024-09-21T14:09:53.181 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.11201127299999999 2024-09-21T14:09:53.181 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.181 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.181 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:09:53.181 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:11.343585+0000", 2024-09-21T14:09:53.181 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000147925 2024-09-21T14:09:53.181 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.182 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.182 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:09:53.182 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:11.343642+0000", 2024-09-21T14:09:53.182 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7158999999999999e-05 2024-09-21T14:09:53.182 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:53.182 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:09:53.182 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:53.182 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.182 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.182 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:169616 11.b 11:d3af3c10:::benchmark_data_smithi049_206132_object46237:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e516)", 2024-09-21T14:09:53.182 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:09:11.225545+0000", 2024-09-21T14:09:53.182 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 41.558447610000002, 2024-09-21T14:09:53.182 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.13182133200000001, 2024-09-21T14:09:53.182 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:09:53.183 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:09:53.183 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:09:53.183 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:09:53.183 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:09:53.183 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:09:53.183 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 169616 2024-09-21T14:09:53.183 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.183 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:09:53.183 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.183 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:09:53.183 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:11.225545+0000", 2024-09-21T14:09:53.183 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:53.183 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.183 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.184 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:09:53.184 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:11.225545+0000", 2024-09-21T14:09:53.184 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:53.184 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.184 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.184 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:09:53.184 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:11.225547+0000", 2024-09-21T14:09:53.184 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2010000000000002e-06 2024-09-21T14:09:53.184 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.184 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.184 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:09:53.184 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:11.225553+0000", 2024-09-21T14:09:53.184 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4979999999999997e-06 2024-09-21T14:09:53.184 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.184 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.185 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:09:53.185 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:11.225566+0000", 2024-09-21T14:09:53.185 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3022e-05 2024-09-21T14:09:53.185 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.185 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.185 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:09:53.185 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:11.225574+0000", 2024-09-21T14:09:53.185 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5669999999999997e-06 2024-09-21T14:09:53.185 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.185 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.185 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:09:53.185 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:11.225798+0000", 2024-09-21T14:09:53.185 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00022427699999999999 2024-09-21T14:09:53.185 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.186 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.186 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:09:53.186 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:11.225932+0000", 2024-09-21T14:09:53.186 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013460100000000001 2024-09-21T14:09:53.186 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.186 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.186 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,1", 2024-09-21T14:09:53.186 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:11.226125+0000", 2024-09-21T14:09:53.186 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00019216700000000001 2024-09-21T14:09:53.186 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.186 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.186 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:09:53.186 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:11.231001+0000", 2024-09-21T14:09:53.187 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0048763779999999998 2024-09-21T14:09:53.187 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.187 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.187 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:53.187 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:11.231670+0000", 2024-09-21T14:09:53.187 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00066916899999999995 2024-09-21T14:09:53.187 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.187 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.187 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:53.187 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:11.356888+0000", 2024-09-21T14:09:53.187 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.12521774099999999 2024-09-21T14:09:53.187 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.187 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.187 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:09:53.187 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:11.357042+0000", 2024-09-21T14:09:53.188 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000153981 2024-09-21T14:09:53.188 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.188 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.188 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:09:53.188 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:11.357367+0000", 2024-09-21T14:09:53.188 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00032473000000000002 2024-09-21T14:09:53.188 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:53.188 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:09:53.188 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:53.188 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.188 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.188 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:170987 11.b 11:db9999e3:::benchmark_data_smithi049_206132_object47608:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e516)", 2024-09-21T14:09:53.188 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:09:12.710859+0000", 2024-09-21T14:09:53.188 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 40.073134125000003, 2024-09-21T14:09:53.188 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.24559587799999999, 2024-09-21T14:09:53.189 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:09:53.189 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:09:53.189 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:09:53.189 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:09:53.189 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:09:53.189 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:09:53.189 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 170987 2024-09-21T14:09:53.189 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.189 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:09:53.189 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.189 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:09:53.189 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:12.710859+0000", 2024-09-21T14:09:53.189 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:53.189 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.189 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.190 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:09:53.190 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:12.710859+0000", 2024-09-21T14:09:53.190 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:53.190 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.190 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.190 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:09:53.190 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:12.710860+0000", 2024-09-21T14:09:53.190 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.466e-06 2024-09-21T14:09:53.190 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.190 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.190 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:09:53.190 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:12.710865+0000", 2024-09-21T14:09:53.190 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9320000000000004e-06 2024-09-21T14:09:53.190 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.191 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.191 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:09:53.191 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:12.710875+0000", 2024-09-21T14:09:53.191 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0236e-05 2024-09-21T14:09:53.191 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.191 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.191 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:09:53.191 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:12.710881+0000", 2024-09-21T14:09:53.191 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2630000000000003e-06 2024-09-21T14:09:53.191 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.191 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.191 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:09:53.191 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:12.710926+0000", 2024-09-21T14:09:53.191 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5389999999999997e-05 2024-09-21T14:09:53.191 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.192 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.192 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:09:53.192 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:12.711001+0000", 2024-09-21T14:09:53.192 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5327999999999994e-05 2024-09-21T14:09:53.192 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.192 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.192 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,1", 2024-09-21T14:09:53.192 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:12.711065+0000", 2024-09-21T14:09:53.192 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3565000000000006e-05 2024-09-21T14:09:53.192 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.192 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.192 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:09:53.192 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:12.714828+0000", 2024-09-21T14:09:53.192 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0037632579999999998 2024-09-21T14:09:53.192 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.193 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.193 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:53.193 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:12.715018+0000", 2024-09-21T14:09:53.193 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00019009300000000001 2024-09-21T14:09:53.193 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.193 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.193 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:53.193 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:12.956347+0000", 2024-09-21T14:09:53.193 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.24132851 2024-09-21T14:09:53.193 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.193 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.193 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:09:53.193 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:12.956408+0000", 2024-09-21T14:09:53.193 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1094999999999997e-05 2024-09-21T14:09:53.194 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.194 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.194 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:09:53.194 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:12.956455+0000", 2024-09-21T14:09:53.194 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6742000000000001e-05 2024-09-21T14:09:53.194 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:53.194 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:09:53.194 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:53.194 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.194 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.194 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:185609 11.b 11:dfa2366a:::benchmark_data_smithi049_206132_object62230:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e523)", 2024-09-21T14:09:53.194 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:09:29.843802+0000", 2024-09-21T14:09:53.194 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 22.940190607000002, 2024-09-21T14:09:53.194 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.12678909999999999, 2024-09-21T14:09:53.194 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:09:53.195 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:09:53.195 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:09:53.195 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:09:53.195 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:09:53.195 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:09:53.195 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 185609 2024-09-21T14:09:53.195 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.195 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:09:53.195 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.195 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:09:53.195 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.843802+0000", 2024-09-21T14:09:53.195 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:53.195 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.195 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.196 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:09:53.196 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.843802+0000", 2024-09-21T14:09:53.196 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:53.196 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.196 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.196 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:09:53.196 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.843804+0000", 2024-09-21T14:09:53.196 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.294e-06 2024-09-21T14:09:53.196 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.196 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.196 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:09:53.196 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.843809+0000", 2024-09-21T14:09:53.196 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3499999999999996e-06 2024-09-21T14:09:53.196 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.196 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.197 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:09:53.197 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.843819+0000", 2024-09-21T14:09:53.197 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0567000000000001e-05 2024-09-21T14:09:53.197 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.197 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.197 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:09:53.197 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.843826+0000", 2024-09-21T14:09:53.197 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9099999999999999e-06 2024-09-21T14:09:53.197 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.197 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.197 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:09:53.197 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.843876+0000", 2024-09-21T14:09:53.197 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9327999999999999e-05 2024-09-21T14:09:53.197 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.198 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.198 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:09:53.198 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.844008+0000", 2024-09-21T14:09:53.198 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013246500000000001 2024-09-21T14:09:53.198 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.198 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.198 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,1", 2024-09-21T14:09:53.198 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.844116+0000", 2024-09-21T14:09:53.198 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000107587 2024-09-21T14:09:53.198 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.198 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.198 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:09:53.198 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.863384+0000", 2024-09-21T14:09:53.199 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.019268596999999998 2024-09-21T14:09:53.199 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.199 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.199 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:53.199 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.868602+0000", 2024-09-21T14:09:53.199 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.005217345 2024-09-21T14:09:53.199 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.199 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.199 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:53.199 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.970396+0000", 2024-09-21T14:09:53.199 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.101794281 2024-09-21T14:09:53.199 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.199 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.199 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:09:53.199 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.970537+0000", 2024-09-21T14:09:53.200 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014088100000000001 2024-09-21T14:09:53.200 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.200 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.200 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:09:53.200 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.970591+0000", 2024-09-21T14:09:53.200 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4494999999999999e-05 2024-09-21T14:09:53.200 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:53.200 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:09:53.200 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:53.200 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.200 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.200 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:185618 11.b 11:d05bdf60:::benchmark_data_smithi049_206132_object62239:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e523)", 2024-09-21T14:09:53.200 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:09:29.848714+0000", 2024-09-21T14:09:53.200 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 22.935278841999999, 2024-09-21T14:09:53.200 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.137084123, 2024-09-21T14:09:53.201 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:09:53.201 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:09:53.201 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:09:53.201 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:09:53.201 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:09:53.201 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:09:53.201 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 185618 2024-09-21T14:09:53.201 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.201 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:09:53.201 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.201 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:09:53.201 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.848714+0000", 2024-09-21T14:09:53.201 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:53.201 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.202 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.202 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:09:53.202 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.848714+0000", 2024-09-21T14:09:53.202 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:53.202 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.202 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.202 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:09:53.202 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.848715+0000", 2024-09-21T14:09:53.202 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.389e-06 2024-09-21T14:09:53.202 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.202 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.202 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:09:53.202 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.848721+0000", 2024-09-21T14:09:53.202 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1919999999999996e-06 2024-09-21T14:09:53.202 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.203 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.203 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:09:53.203 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.848733+0000", 2024-09-21T14:09:53.203 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2347000000000001e-05 2024-09-21T14:09:53.203 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.203 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.203 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:09:53.203 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.848741+0000", 2024-09-21T14:09:53.203 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8429999999999993e-06 2024-09-21T14:09:53.203 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.203 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.203 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:09:53.203 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.853611+0000", 2024-09-21T14:09:53.203 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.004870259 2024-09-21T14:09:53.203 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.204 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.204 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:09:53.204 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.853736+0000", 2024-09-21T14:09:53.204 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012495900000000001 2024-09-21T14:09:53.204 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.204 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.204 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,1", 2024-09-21T14:09:53.204 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.853851+0000", 2024-09-21T14:09:53.204 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000115259 2024-09-21T14:09:53.204 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.204 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.204 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:09:53.204 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.863396+0000", 2024-09-21T14:09:53.204 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0095450810000000004 2024-09-21T14:09:53.205 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.205 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.205 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:53.205 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.952660+0000", 2024-09-21T14:09:53.205 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.089263268000000007 2024-09-21T14:09:53.205 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.205 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.205 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:53.205 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.985698+0000", 2024-09-21T14:09:53.205 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.033038784000000002 2024-09-21T14:09:53.205 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.205 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.205 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:09:53.205 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.985754+0000", 2024-09-21T14:09:53.206 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5130999999999998e-05 2024-09-21T14:09:53.206 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.206 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.206 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:09:53.206 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.985798+0000", 2024-09-21T14:09:53.206 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4610999999999997e-05 2024-09-21T14:09:53.206 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:53.206 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:09:53.206 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:53.206 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.206 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.206 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:185623 11.b 11:dece0c01:::benchmark_data_smithi049_206132_object62244:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e523)", 2024-09-21T14:09:53.206 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:09:29.854418+0000", 2024-09-21T14:09:53.206 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 22.929575341, 2024-09-21T14:09:53.207 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.13361891000000001, 2024-09-21T14:09:53.207 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:09:53.207 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:09:53.207 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:09:53.207 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:09:53.207 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:09:53.207 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:09:53.207 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 185623 2024-09-21T14:09:53.207 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.207 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:09:53.207 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.207 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:09:53.207 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.854418+0000", 2024-09-21T14:09:53.207 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:53.207 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.208 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.208 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:09:53.208 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.854418+0000", 2024-09-21T14:09:53.208 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:09:53.208 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.208 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.208 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:09:53.208 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.854418+0000", 2024-09-21T14:09:53.208 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5300000000000003e-07 2024-09-21T14:09:53.208 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.208 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.208 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:09:53.208 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.854422+0000", 2024-09-21T14:09:53.208 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9580000000000001e-06 2024-09-21T14:09:53.209 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.209 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.209 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:09:53.209 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.854431+0000", 2024-09-21T14:09:53.209 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.32e-06 2024-09-21T14:09:53.209 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.209 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.209 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:09:53.209 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.854437+0000", 2024-09-21T14:09:53.209 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4320000000000004e-06 2024-09-21T14:09:53.209 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.209 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.209 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:09:53.209 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.858487+0000", 2024-09-21T14:09:53.209 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0040498460000000002 2024-09-21T14:09:53.210 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.210 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.210 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:09:53.210 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.858611+0000", 2024-09-21T14:09:53.210 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012403499999999999 2024-09-21T14:09:53.210 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.210 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.210 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,1", 2024-09-21T14:09:53.210 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.858722+0000", 2024-09-21T14:09:53.210 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011086699999999999 2024-09-21T14:09:53.210 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.210 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.210 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:09:53.210 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.863710+0000", 2024-09-21T14:09:53.211 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00498804 2024-09-21T14:09:53.211 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.211 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.211 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:53.211 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.952709+0000", 2024-09-21T14:09:53.211 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.088999522999999997 2024-09-21T14:09:53.211 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.211 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.211 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:09:53.211 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.986072+0000", 2024-09-21T14:09:53.211 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.033363057000000002 2024-09-21T14:09:53.211 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.211 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.211 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:09:53.212 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.987984+0000", 2024-09-21T14:09:53.212 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001911825 2024-09-21T14:09:53.212 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:09:53.212 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:09:53.212 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:09:53.212 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.988036+0000", 2024-09-21T14:09:53.212 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2253999999999998e-05 2024-09-21T14:09:53.212 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:53.212 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:09:53.212 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:53.212 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:09:53.212 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:09:53.212 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T14:09:53.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:53.213 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:53.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:53.314 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:53.336 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:53.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:53.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:53.438 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:53.460 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:53.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:53.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:53.562 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:09:53.591 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:53.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:53.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:09:53.693 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:09:53.724 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:53.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:53.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:09:53.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:53 smithi049 ceph-mon[30080]: pgmap v1071: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 2.5 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 104 B/s rd, 0 B/s wr, 1.13k op/s; 122/122157 objects misplaced (0.100%); 1.2 MiB/s, 19 objects/s recovering 2024-09-21T14:09:53.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:53 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3558390995' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]': finished 2024-09-21T14:09:53.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:53 smithi049 ceph-mon[30080]: osdmap e531: 8 total, 8 up, 8 in 2024-09-21T14:09:53.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:53 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T14:09:53.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:53 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T14:09:53.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:53 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:09:53.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:53 smithi049 ceph-mon[40489]: pgmap v1071: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 2.5 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 104 B/s rd, 0 B/s wr, 1.13k op/s; 122/122157 objects misplaced (0.100%); 1.2 MiB/s, 19 objects/s recovering 2024-09-21T14:09:53.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:53 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3558390995' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]': finished 2024-09-21T14:09:53.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:53 smithi049 ceph-mon[40489]: osdmap e531: 8 total, 8 up, 8 in 2024-09-21T14:09:53.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:53 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T14:09:53.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:53 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T14:09:53.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:53 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:09:53.825 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:53.845 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T14:09:53.853 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:53.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:53.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:53.881 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:53 smithi187 ceph-mon[33032]: pgmap v1071: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 2.5 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 104 B/s rd, 0 B/s wr, 1.13k op/s; 122/122157 objects misplaced (0.100%); 1.2 MiB/s, 19 objects/s recovering 2024-09-21T14:09:53.881 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:53 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3558390995' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]': finished 2024-09-21T14:09:53.881 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:53 smithi187 ceph-mon[33032]: osdmap e531: 8 total, 8 up, 8 in 2024-09-21T14:09:53.881 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:53 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T14:09:53.881 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:53 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T14:09:53.882 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:53 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:09:53.955 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:09:53.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:53.999 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:54.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:09:54.100 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:09:54.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:54.138 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:54.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:09:54.238 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:09:54.240 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:09:54.275 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:54.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:54.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:09:54.346 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:09:54.377 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:09:54.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:54.410 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:54.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:09:54.510 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T14:09:54.510 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:09:54.555 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:54 smithi049 ceph-mon[30080]: osdmap e532: 8 total, 8 up, 8 in 2024-09-21T14:09:54.555 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:54 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:09:54.555 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:54 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:09:54.555 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:54 smithi049 ceph-mon[40489]: osdmap e532: 8 total, 8 up, 8 in 2024-09-21T14:09:54.555 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:54 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:09:54.556 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:54 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:09:54.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:54.564 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:54.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:09:54.665 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:09:54.670 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:54 smithi187 ceph-mon[33032]: osdmap e532: 8 total, 8 up, 8 in 2024-09-21T14:09:54.670 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:54 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:09:54.671 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:54 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:09:54.734 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:54.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:54.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:09:54.836 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:09:54.870 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:54.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:54.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:09:54.949 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T14:09:54.949 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-21T14:09:54.949 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:54.949 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.26396.0:210974 11.2 e533/531 11:49cc13b7:::benchmark_data_smithi049_206132_object87595:head v 533'13081, mlcod=533'13079)", 2024-09-21T14:09:54.949 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:09:54.946296+0000", 2024-09-21T14:09:54.949 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0025309970000000001, 2024-09-21T14:09:54.949 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.002558205, 2024-09-21T14:09:54.949 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:09:54.950 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:09:54.950 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T14:09:54.950 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:09:54.950 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:54.950 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:09:54.950 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:54.946296+0000", 2024-09-21T14:09:54.950 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:54.950 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:54.950 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:54.950 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:09:54.950 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:54.946296+0000", 2024-09-21T14:09:54.950 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:09:54.951 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:54.951 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:54.951 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:09:54.951 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:54.946297+0000", 2024-09-21T14:09:54.951 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8999999999999995e-07 2024-09-21T14:09:54.951 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:54.951 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:54.951 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:09:54.951 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:54.946304+0000", 2024-09-21T14:09:54.951 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4710000000000004e-06 2024-09-21T14:09:54.951 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:54.951 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:54.951 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:09:54.952 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:54.946319+0000", 2024-09-21T14:09:54.952 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.502e-05 2024-09-21T14:09:54.952 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:54.952 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:54.952 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:09:54.952 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:54.946326+0000", 2024-09-21T14:09:54.952 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3869999999999997e-06 2024-09-21T14:09:54.952 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:54.952 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:54.952 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:09:54.952 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:54.946393+0000", 2024-09-21T14:09:54.952 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6717999999999994e-05 2024-09-21T14:09:54.953 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:09:54.953 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:09:54.953 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:09:54.953 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:54.946418+0000", 2024-09-21T14:09:54.953 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5232000000000002e-05 2024-09-21T14:09:54.953 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:54.953 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:09:54.953 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:54.953 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:09:54.953 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2024-09-21T14:09:54.953 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 1 2024-09-21T14:09:54.953 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T14:09:54.973 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:09:55.000 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:55.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:55.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:09:55.093 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:09:55.094 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:09:55.102 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:09:55.142 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:09:55.142 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:09:55.158 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:55.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:55.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:09:55.176 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:09:55.176 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:09:55.183 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:09:55.207 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:09:55.207 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:09:55.240 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:09:55.241 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:09:55.260 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:09:55.278 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:09:55.278 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:09:55.312 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:55.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:55.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:09:55.314 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:09:55.314 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:09:55.346 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:09:55.346 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:09:55.415 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:09:55.462 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:55.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:55.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:09:55.563 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:55.592 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:55.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:55.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:55.694 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:55.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:55 smithi187 ceph-mon[33032]: pgmap v1074: 168 pgs: 3 peering, 1 active+remapped+backfill_toofull, 164 active+clean; 2.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.39k op/s; 122/114618 objects misplaced (0.106%); 0 B/s, 0 objects/s recovering 2024-09-21T14:09:55.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:55 smithi187 ceph-mon[33032]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-21T14:09:55.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:55 smithi187 ceph-mon[33032]: osdmap e533: 8 total, 8 up, 8 in 2024-09-21T14:09:55.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:55 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1728748807' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:09:55.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:55 smithi187 ceph-mon[33032]: osdmap e534: 8 total, 8 up, 8 in 2024-09-21T14:09:55.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:55.733 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:55.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:55.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:55 smithi049 ceph-mon[40489]: pgmap v1074: 168 pgs: 3 peering, 1 active+remapped+backfill_toofull, 164 active+clean; 2.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.39k op/s; 122/114618 objects misplaced (0.106%); 0 B/s, 0 objects/s recovering 2024-09-21T14:09:55.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:55 smithi049 ceph-mon[40489]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-21T14:09:55.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:55 smithi049 ceph-mon[40489]: osdmap e533: 8 total, 8 up, 8 in 2024-09-21T14:09:55.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:55 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1728748807' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:09:55.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:55 smithi049 ceph-mon[40489]: osdmap e534: 8 total, 8 up, 8 in 2024-09-21T14:09:55.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:55 smithi049 ceph-mon[30080]: pgmap v1074: 168 pgs: 3 peering, 1 active+remapped+backfill_toofull, 164 active+clean; 2.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.39k op/s; 122/114618 objects misplaced (0.106%); 0 B/s, 0 objects/s recovering 2024-09-21T14:09:55.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:55 smithi049 ceph-mon[30080]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-21T14:09:55.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:55 smithi049 ceph-mon[30080]: osdmap e533: 8 total, 8 up, 8 in 2024-09-21T14:09:55.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:55 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1728748807' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:09:55.757 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:55 smithi049 ceph-mon[30080]: osdmap e534: 8 total, 8 up, 8 in 2024-09-21T14:09:55.834 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:55.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:55.875 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:55.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:55.906 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T14:09:55.976 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:56.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:56.028 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:56.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:56.129 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:56.182 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:56.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:56.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:56.284 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:56.307 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:56.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:56.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:56.316 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:09:56.409 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:09:56.437 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:56.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:56.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:09:56.539 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:09:56.577 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:56.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:56.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:09:56.678 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:56.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:56 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3555012920' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:09:56.710 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:56.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:56.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:56.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:56 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3555012920' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:09:56.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:56 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3555012920' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:09:56.813 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:09:56.842 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:56.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:56.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:09:56.936 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T14:09:56.936 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:09:56.945 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:09:56.982 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:56.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:56.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:09:57.083 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:09:57.110 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:57.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:57.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:09:57.212 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:09:57.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:57.244 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:57.244 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:09:57.345 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:57.371 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:57.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:57.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:57.473 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:09:57.502 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:57.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:57.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:09:57.603 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:09:57.642 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:57.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:57.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:09:57.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:57 smithi187 ceph-mon[33032]: pgmap v1077: 168 pgs: 3 peering, 1 active+remapped+backfill_toofull, 164 active+clean; 2.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 767 B/s rd, 0 B/s wr, 971 op/s; 109/110508 objects misplaced (0.099%) 2024-09-21T14:09:57.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:57 smithi187 ceph-mon[33032]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:09:57.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:57 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3555012920' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T14:09:57.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:57 smithi187 ceph-mon[33032]: osdmap e535: 8 total, 8 up, 8 in 2024-09-21T14:09:57.744 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:57.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:57 smithi049 ceph-mon[40489]: pgmap v1077: 168 pgs: 3 peering, 1 active+remapped+backfill_toofull, 164 active+clean; 2.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 767 B/s rd, 0 B/s wr, 971 op/s; 109/110508 objects misplaced (0.099%) 2024-09-21T14:09:57.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:57 smithi049 ceph-mon[40489]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:09:57.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:57 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3555012920' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T14:09:57.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:57 smithi049 ceph-mon[40489]: osdmap e535: 8 total, 8 up, 8 in 2024-09-21T14:09:57.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:57 smithi049 ceph-mon[30080]: pgmap v1077: 168 pgs: 3 peering, 1 active+remapped+backfill_toofull, 164 active+clean; 2.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 767 B/s rd, 0 B/s wr, 971 op/s; 109/110508 objects misplaced (0.099%) 2024-09-21T14:09:57.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:57 smithi049 ceph-mon[30080]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:09:57.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:57 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3555012920' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T14:09:57.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:57 smithi049 ceph-mon[30080]: osdmap e535: 8 total, 8 up, 8 in 2024-09-21T14:09:57.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:57.770 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:57.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:57.871 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:09:57.893 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:57.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:57.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:09:57.995 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:09:58.026 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:58.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:58.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:09:58.074 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T14:09:58.127 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:09:58.161 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:58.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:58.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:09:58.263 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:09:58.303 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:58.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:58.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:09:58.405 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:58.437 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-09-21T14:09:58.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:58.455 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:58.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:58.487 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:09:58.556 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:09:58.593 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:58.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:58.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:09:58.695 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:09:58.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:58.731 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:58.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:09:58.832 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:58.860 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:58.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:58.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:58.961 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:59.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:59.004 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:59.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:59.075 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T14:09:59.075 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:09:59.104 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:09:59.145 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:59.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:59.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:09:59.248 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:59.276 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:59.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:59.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:59.342 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 4, 6, 2, 3, 7, 5] 2024-09-21T14:09:59.343 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T14:09:59.343 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-21T14:09:59.343 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd dump -f json-pretty 2024-09-21T14:09:59.377 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:59.406 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:59.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:59.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:59.508 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:09:59.538 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:59.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:59.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:09:59.640 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:59.669 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:59.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:59.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:59.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:09:59 smithi187 ceph-mon[33032]: pgmap v1079: 168 pgs: 3 peering, 1 active+remapped+backfill_toofull, 164 active+clean; 2.2 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 434 B/s rd, 0 B/s wr, 806 op/s; 109/108687 objects misplaced (0.100%) 2024-09-21T14:09:59.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:09:59 smithi049 ceph-mon[40489]: pgmap v1079: 168 pgs: 3 peering, 1 active+remapped+backfill_toofull, 164 active+clean; 2.2 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 434 B/s rd, 0 B/s wr, 806 op/s; 109/108687 objects misplaced (0.100%) 2024-09-21T14:09:59.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:09:59 smithi049 ceph-mon[30080]: pgmap v1079: 168 pgs: 3 peering, 1 active+remapped+backfill_toofull, 164 active+clean; 2.2 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 434 B/s rd, 0 B/s wr, 806 op/s; 109/108687 objects misplaced (0.100%) 2024-09-21T14:09:59.772 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:09:59.792 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:09:59.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:59.811 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:09:59.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:09:59.830 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T14:09:59.912 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:09:59.952 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:09:59.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:09:59.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:10:00.054 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:10:00.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:00.100 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:00.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:10:00.202 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:10:00.241 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:00.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:00.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:10:00.255 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:10:00.343 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:10:00.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:00.382 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:00.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:10:00.482 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:10:00.516 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:10:00 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:10:00.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:00.523 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:00.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:10:00.625 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:10:00.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:00.662 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:00.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:10:00.762 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:10:00.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:00.805 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:00.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:10:00.841 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T14:10:00.841 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T14:10:00.841 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 536, 2024-09-21T14:10:00.841 INFO:teuthology.orchestra.run.smithi049.stdout: "fsid": "40a3d2e8-781f-11ef-baf6-efdc52797490", 2024-09-21T14:10:00.841 INFO:teuthology.orchestra.run.smithi049.stdout: "created": "2024-09-21T13:42:57.838275+0000", 2024-09-21T14:10:00.841 INFO:teuthology.orchestra.run.smithi049.stdout: "modified": "2024-09-21T14:10:00.254985+0000", 2024-09-21T14:10:00.841 INFO:teuthology.orchestra.run.smithi049.stdout: "last_up_change": "2024-09-21T14:01:40.158454+0000", 2024-09-21T14:10:00.842 INFO:teuthology.orchestra.run.smithi049.stdout: "last_in_change": "2024-09-21T13:58:16.730861+0000", 2024-09-21T14:10:00.842 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-21T14:10:00.842 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_num": 5806080, 2024-09-21T14:10:00.842 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_set": [ 2024-09-21T14:10:00.842 INFO:teuthology.orchestra.run.smithi049.stdout: "nodeep-scrub", 2024-09-21T14:10:00.842 INFO:teuthology.orchestra.run.smithi049.stdout: "noscrub", 2024-09-21T14:10:00.842 INFO:teuthology.orchestra.run.smithi049.stdout: "pglog_hardlimit", 2024-09-21T14:10:00.842 INFO:teuthology.orchestra.run.smithi049.stdout: "purged_snapdirs", 2024-09-21T14:10:00.842 INFO:teuthology.orchestra.run.smithi049.stdout: "recovery_deletes", 2024-09-21T14:10:00.842 INFO:teuthology.orchestra.run.smithi049.stdout: "sortbitwise" 2024-09-21T14:10:00.842 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T14:10:00.842 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_version": 22, 2024-09-21T14:10:00.842 INFO:teuthology.orchestra.run.smithi049.stdout: "full_ratio": 0.94999998807907104, 2024-09-21T14:10:00.843 INFO:teuthology.orchestra.run.smithi049.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-21T14:10:00.843 INFO:teuthology.orchestra.run.smithi049.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-21T14:10:00.843 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_snapshot": "", 2024-09-21T14:10:00.843 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_max": 11, 2024-09-21T14:10:00.847 INFO:teuthology.orchestra.run.smithi049.stdout: "max_osd": 8, 2024-09-21T14:10:00.847 INFO:teuthology.orchestra.run.smithi049.stdout: "require_min_compat_client": "luminous", 2024-09-21T14:10:00.847 INFO:teuthology.orchestra.run.smithi049.stdout: "min_compat_client": "luminous", 2024-09-21T14:10:00.847 INFO:teuthology.orchestra.run.smithi049.stdout: "require_osd_release": "squid", 2024-09-21T14:10:00.847 INFO:teuthology.orchestra.run.smithi049.stdout: "allow_crimson": false, 2024-09-21T14:10:00.847 INFO:teuthology.orchestra.run.smithi049.stdout: "pools": [ 2024-09-21T14:10:00.847 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.847 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 1, 2024-09-21T14:10:00.847 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": ".mgr", 2024-09-21T14:10:00.848 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:46:21.096244+0000", 2024-09-21T14:10:00.848 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T14:10:00.848 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T14:10:00.848 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T14:10:00.848 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T14:10:00.848 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T14:10:00.848 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T14:10:00.848 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T14:10:00.848 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T14:10:00.848 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T14:10:00.848 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T14:10:00.848 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T14:10:00.848 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T14:10:00.848 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T14:10:00.849 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 21, 2024-09-21T14:10:00.849 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 21, 2024-09-21T14:10:00.849 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 21, 2024-09-21T14:10:00.849 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 21, 2024-09-21T14:10:00.849 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 21, 2024-09-21T14:10:00.849 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T14:10:00.849 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T14:10:00.849 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T14:10:00.849 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T14:10:00.849 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T14:10:00.849 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T14:10:00.849 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T14:10:00.849 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.849 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "438", 2024-09-21T14:10:00.850 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T14:10:00.850 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T14:10:00.850 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "436", 2024-09-21T14:10:00.850 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T14:10:00.850 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T14:10:00.850 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T14:10:00.850 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T14:10:00.850 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T14:10:00.850 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T14:10:00.850 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T14:10:00.850 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T14:10:00.850 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T14:10:00.850 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T14:10:00.851 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T14:10:00.851 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T14:10:00.851 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T14:10:00.851 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T14:10:00.851 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T14:10:00.851 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T14:10:00.851 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T14:10:00.851 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T14:10:00.851 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T14:10:00.851 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T14:10:00.851 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T14:10:00.851 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T14:10:00.851 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T14:10:00.851 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.852 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T14:10:00.852 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T14:10:00.852 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T14:10:00.852 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T14:10:00.852 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T14:10:00.852 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T14:10:00.852 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T14:10:00.852 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T14:10:00.852 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T14:10:00.852 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T14:10:00.852 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T14:10:00.852 INFO:teuthology.orchestra.run.smithi049.stdout: "options": { 2024-09-21T14:10:00.852 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_max": 32, 2024-09-21T14:10:00.852 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_min": 1 2024-09-21T14:10:00.853 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.853 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T14:10:00.853 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr": {} 2024-09-21T14:10:00.853 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.853 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T14:10:00.853 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T14:10:00.853 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2.0099999904632568, 2024-09-21T14:10:00.853 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2.0099999904632568, 2024-09-21T14:10:00.853 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1.3300000429153442, 2024-09-21T14:10:00.853 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2.6700000762939453, 2024-09-21T14:10:00.853 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2.6700000762939453, 2024-09-21T14:10:00.853 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.72835731506347656, 2024-09-21T14:10:00.853 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.97000002861022949, 2024-09-21T14:10:00.853 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.73000001907348633 2024-09-21T14:10:00.854 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:00.854 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.854 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.854 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 2, 2024-09-21T14:10:00.854 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "datapool", 2024-09-21T14:10:00.854 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:48:55.540712+0000", 2024-09-21T14:10:00.854 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 8193, 2024-09-21T14:10:00.854 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T14:10:00.854 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T14:10:00.854 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T14:10:00.854 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T14:10:00.854 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T14:10:00.854 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T14:10:00.854 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T14:10:00.855 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T14:10:00.855 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T14:10:00.855 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T14:10:00.855 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T14:10:00.855 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T14:10:00.855 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 3, 2024-09-21T14:10:00.855 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 3, 2024-09-21T14:10:00.855 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 3, 2024-09-21T14:10:00.855 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 3, 2024-09-21T14:10:00.855 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 3, 2024-09-21T14:10:00.855 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T14:10:00.855 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T14:10:00.855 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T14:10:00.855 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T14:10:00.855 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T14:10:00.856 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T14:10:00.856 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T14:10:00.856 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.856 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "63", 2024-09-21T14:10:00.856 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T14:10:00.856 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T14:10:00.856 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T14:10:00.856 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T14:10:00.856 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T14:10:00.856 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 2, 2024-09-21T14:10:00.856 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 63, 2024-09-21T14:10:00.856 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T14:10:00.856 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T14:10:00.857 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T14:10:00.857 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T14:10:00.857 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T14:10:00.857 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T14:10:00.857 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T14:10:00.857 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T14:10:00.857 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T14:10:00.857 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T14:10:00.857 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T14:10:00.857 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T14:10:00.857 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T14:10:00.857 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T14:10:00.857 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T14:10:00.857 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T14:10:00.858 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T14:10:00.858 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T14:10:00.858 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T14:10:00.858 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.858 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T14:10:00.858 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T14:10:00.858 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T14:10:00.858 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T14:10:00.858 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T14:10:00.858 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T14:10:00.858 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T14:10:00.859 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T14:10:00.859 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T14:10:00.859 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T14:10:00.859 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T14:10:00.859 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T14:10:00.859 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T14:10:00.859 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd": {} 2024-09-21T14:10:00.859 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.859 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T14:10:00.859 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T14:10:00.859 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 2, 2024-09-21T14:10:00.859 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 2, 2024-09-21T14:10:00.859 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1, 2024-09-21T14:10:00.859 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2, 2024-09-21T14:10:00.860 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2, 2024-09-21T14:10:00.860 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.72835731506347656, 2024-09-21T14:10:00.860 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.97000002861022949, 2024-09-21T14:10:00.860 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.73000001907348633 2024-09-21T14:10:00.860 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:00.860 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.860 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.860 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 3, 2024-09-21T14:10:00.860 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": ".rgw.root", 2024-09-21T14:10:00.860 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:48:56.417456+0000", 2024-09-21T14:10:00.860 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T14:10:00.860 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T14:10:00.860 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T14:10:00.860 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T14:10:00.861 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T14:10:00.861 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T14:10:00.861 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T14:10:00.861 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T14:10:00.861 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T14:10:00.861 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T14:10:00.861 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T14:10:00.861 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T14:10:00.861 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T14:10:00.861 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 32, 2024-09-21T14:10:00.861 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 32, 2024-09-21T14:10:00.861 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 32, 2024-09-21T14:10:00.861 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 32, 2024-09-21T14:10:00.861 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 32, 2024-09-21T14:10:00.861 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T14:10:00.862 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T14:10:00.862 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T14:10:00.862 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T14:10:00.862 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T14:10:00.862 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T14:10:00.862 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T14:10:00.862 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.862 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "62", 2024-09-21T14:10:00.862 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T14:10:00.863 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T14:10:00.863 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T14:10:00.863 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T14:10:00.863 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T14:10:00.863 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T14:10:00.863 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T14:10:00.863 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T14:10:00.863 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T14:10:00.863 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T14:10:00.863 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T14:10:00.863 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T14:10:00.863 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T14:10:00.863 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T14:10:00.863 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T14:10:00.864 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T14:10:00.864 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T14:10:00.864 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T14:10:00.864 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T14:10:00.864 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T14:10:00.864 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T14:10:00.864 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T14:10:00.864 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T14:10:00.864 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T14:10:00.864 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T14:10:00.865 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T14:10:00.865 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.865 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T14:10:00.866 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T14:10:00.866 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T14:10:00.866 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T14:10:00.866 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T14:10:00.866 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T14:10:00.866 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T14:10:00.866 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T14:10:00.866 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T14:10:00.866 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T14:10:00.866 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T14:10:00.866 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T14:10:00.866 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T14:10:00.866 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw": {} 2024-09-21T14:10:00.867 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.867 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T14:10:00.867 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T14:10:00.867 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 1.690000057220459, 2024-09-21T14:10:00.867 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 1.690000057220459, 2024-09-21T14:10:00.867 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1.3300000429153442, 2024-09-21T14:10:00.867 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2.25, 2024-09-21T14:10:00.867 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2.25, 2024-09-21T14:10:00.867 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.72835731506347656, 2024-09-21T14:10:00.867 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.97000002861022949, 2024-09-21T14:10:00.867 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.73000001907348633 2024-09-21T14:10:00.867 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:00.867 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.868 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.868 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 4, 2024-09-21T14:10:00.868 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "default.rgw.log", 2024-09-21T14:10:00.868 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:48:58.128455+0000", 2024-09-21T14:10:00.868 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T14:10:00.868 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T14:10:00.868 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T14:10:00.868 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T14:10:00.868 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T14:10:00.868 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T14:10:00.868 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T14:10:00.868 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T14:10:00.868 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T14:10:00.868 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T14:10:00.869 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T14:10:00.869 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T14:10:00.869 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T14:10:00.869 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 32, 2024-09-21T14:10:00.869 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 32, 2024-09-21T14:10:00.869 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 32, 2024-09-21T14:10:00.869 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 32, 2024-09-21T14:10:00.869 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 32, 2024-09-21T14:10:00.869 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T14:10:00.869 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T14:10:00.869 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T14:10:00.869 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T14:10:00.869 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T14:10:00.869 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T14:10:00.869 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T14:10:00.870 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.870 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "64", 2024-09-21T14:10:00.870 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T14:10:00.870 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T14:10:00.870 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T14:10:00.870 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T14:10:00.870 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T14:10:00.870 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T14:10:00.870 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T14:10:00.870 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T14:10:00.870 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T14:10:00.870 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T14:10:00.870 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T14:10:00.870 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T14:10:00.871 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T14:10:00.871 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T14:10:00.871 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T14:10:00.871 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T14:10:00.871 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T14:10:00.871 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T14:10:00.871 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T14:10:00.871 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T14:10:00.871 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T14:10:00.871 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T14:10:00.871 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T14:10:00.871 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T14:10:00.871 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T14:10:00.871 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T14:10:00.871 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.872 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T14:10:00.872 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T14:10:00.872 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T14:10:00.872 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T14:10:00.872 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T14:10:00.872 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T14:10:00.872 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T14:10:00.872 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T14:10:00.872 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T14:10:00.872 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T14:10:00.872 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T14:10:00.872 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T14:10:00.872 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T14:10:00.872 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw": {} 2024-09-21T14:10:00.873 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.873 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T14:10:00.873 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T14:10:00.873 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 1.690000057220459, 2024-09-21T14:10:00.873 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 1.690000057220459, 2024-09-21T14:10:00.873 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1.3300000429153442, 2024-09-21T14:10:00.873 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2.25, 2024-09-21T14:10:00.873 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2.25, 2024-09-21T14:10:00.873 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.72835731506347656, 2024-09-21T14:10:00.873 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.97000002861022949, 2024-09-21T14:10:00.873 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.73000001907348633 2024-09-21T14:10:00.873 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:00.873 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.873 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.873 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 5, 2024-09-21T14:10:00.874 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "default.rgw.control", 2024-09-21T14:10:00.874 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:49:00.069361+0000", 2024-09-21T14:10:00.874 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T14:10:00.874 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T14:10:00.874 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T14:10:00.874 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T14:10:00.874 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T14:10:00.874 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T14:10:00.874 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T14:10:00.874 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T14:10:00.874 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T14:10:00.874 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T14:10:00.874 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T14:10:00.874 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T14:10:00.875 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T14:10:00.875 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 32, 2024-09-21T14:10:00.875 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 32, 2024-09-21T14:10:00.875 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 32, 2024-09-21T14:10:00.875 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 32, 2024-09-21T14:10:00.875 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 32, 2024-09-21T14:10:00.875 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T14:10:00.875 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T14:10:00.875 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T14:10:00.875 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T14:10:00.875 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T14:10:00.875 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T14:10:00.875 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T14:10:00.875 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.876 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "66", 2024-09-21T14:10:00.876 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T14:10:00.876 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T14:10:00.876 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T14:10:00.876 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T14:10:00.876 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T14:10:00.876 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T14:10:00.876 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T14:10:00.876 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T14:10:00.876 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T14:10:00.876 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T14:10:00.876 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T14:10:00.876 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T14:10:00.876 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T14:10:00.877 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T14:10:00.877 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T14:10:00.877 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T14:10:00.877 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T14:10:00.877 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T14:10:00.877 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T14:10:00.877 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T14:10:00.877 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T14:10:00.877 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T14:10:00.877 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T14:10:00.877 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T14:10:00.877 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T14:10:00.877 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T14:10:00.877 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.877 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T14:10:00.878 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T14:10:00.878 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T14:10:00.878 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T14:10:00.878 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T14:10:00.878 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T14:10:00.878 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T14:10:00.878 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T14:10:00.878 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T14:10:00.878 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T14:10:00.878 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T14:10:00.878 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T14:10:00.878 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T14:10:00.878 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw": {} 2024-09-21T14:10:00.879 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.879 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T14:10:00.879 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T14:10:00.879 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 1.3200000524520874, 2024-09-21T14:10:00.879 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:00 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:10:00.879 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:00 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:10:00.879 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:00 smithi049 ceph-mon[40489]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; Reduced data availability: 1 pg inactive, 1 pg peering; Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull 2024-09-21T14:10:00.879 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:00 smithi049 ceph-mon[40489]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-09-21T14:10:00.879 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:00 smithi049 ceph-mon[40489]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive, 1 pg peering 2024-09-21T14:10:00.879 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:00 smithi049 ceph-mon[40489]: pg 5.1e is stuck peering for 17m, current state peering, last acting [2,0,7] 2024-09-21T14:10:00.880 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:00 smithi049 ceph-mon[40489]: [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull 2024-09-21T14:10:00.880 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:00 smithi049 ceph-mon[40489]: pg 11.0 is active+remapped+backfill_toofull, acting [1,7,3] 2024-09-21T14:10:00.880 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:00 smithi049 ceph-mon[40489]: osdmap e536: 8 total, 8 up, 8 in 2024-09-21T14:10:00.880 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:10:00 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:10:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:10:00.880 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:00 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:10:00.880 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:00 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:10:00.880 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:00 smithi049 ceph-mon[30080]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; Reduced data availability: 1 pg inactive, 1 pg peering; Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull 2024-09-21T14:10:00.880 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:00 smithi049 ceph-mon[30080]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-09-21T14:10:00.881 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:00 smithi049 ceph-mon[30080]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive, 1 pg peering 2024-09-21T14:10:00.881 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:00 smithi049 ceph-mon[30080]: pg 5.1e is stuck peering for 17m, current state peering, last acting [2,0,7] 2024-09-21T14:10:00.881 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:00 smithi049 ceph-mon[30080]: [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull 2024-09-21T14:10:00.881 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:00 smithi049 ceph-mon[30080]: pg 11.0 is active+remapped+backfill_toofull, acting [1,7,3] 2024-09-21T14:10:00.881 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:00 smithi049 ceph-mon[30080]: osdmap e536: 8 total, 8 up, 8 in 2024-09-21T14:10:00.881 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 1.3200000524520874, 2024-09-21T14:10:00.881 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1.3300000429153442, 2024-09-21T14:10:00.881 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 1.75, 2024-09-21T14:10:00.881 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 1.75, 2024-09-21T14:10:00.881 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.72835731506347656, 2024-09-21T14:10:00.882 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.97000002861022949, 2024-09-21T14:10:00.882 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.73000001907348633 2024-09-21T14:10:00.882 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:00.882 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.882 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.882 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 6, 2024-09-21T14:10:00.882 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "default.rgw.meta", 2024-09-21T14:10:00.882 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T13:49:02.127399+0000", 2024-09-21T14:10:00.882 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T14:10:00.882 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T14:10:00.882 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T14:10:00.882 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T14:10:00.882 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T14:10:00.883 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T14:10:00.883 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T14:10:00.883 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T14:10:00.883 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T14:10:00.883 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T14:10:00.883 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T14:10:00.883 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T14:10:00.883 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T14:10:00.883 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 32, 2024-09-21T14:10:00.883 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 32, 2024-09-21T14:10:00.883 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 32, 2024-09-21T14:10:00.883 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 32, 2024-09-21T14:10:00.883 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 32, 2024-09-21T14:10:00.883 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T14:10:00.883 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T14:10:00.884 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T14:10:00.884 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T14:10:00.884 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T14:10:00.884 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T14:10:00.884 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T14:10:00.884 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.884 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "68", 2024-09-21T14:10:00.884 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T14:10:00.884 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T14:10:00.884 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T14:10:00.884 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T14:10:00.884 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T14:10:00.884 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T14:10:00.884 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T14:10:00.885 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T14:10:00.885 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T14:10:00.885 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T14:10:00.885 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T14:10:00.885 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T14:10:00.885 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T14:10:00.885 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T14:10:00.885 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T14:10:00.885 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T14:10:00.885 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T14:10:00.885 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T14:10:00.885 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T14:10:00.885 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T14:10:00.885 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T14:10:00.886 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T14:10:00.886 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T14:10:00.886 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T14:10:00.886 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T14:10:00.886 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T14:10:00.886 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.886 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T14:10:00.886 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T14:10:00.886 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T14:10:00.886 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T14:10:00.886 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T14:10:00.886 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T14:10:00.886 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T14:10:00.886 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T14:10:00.887 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T14:10:00.887 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T14:10:00.887 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T14:10:00.887 INFO:teuthology.orchestra.run.smithi049.stdout: "options": { 2024-09-21T14:10:00.887 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_bias": 4 2024-09-21T14:10:00.887 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.887 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T14:10:00.887 INFO:teuthology.orchestra.run.smithi049.stdout: "rgw": {} 2024-09-21T14:10:00.887 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.887 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T14:10:00.887 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T14:10:00.887 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 1.5, 2024-09-21T14:10:00.887 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 1.5, 2024-09-21T14:10:00.887 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1.3300000429153442, 2024-09-21T14:10:00.888 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2, 2024-09-21T14:10:00.888 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2, 2024-09-21T14:10:00.888 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.72835731506347656, 2024-09-21T14:10:00.888 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.97000002861022949, 2024-09-21T14:10:00.888 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.73000001907348633 2024-09-21T14:10:00.888 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:00.888 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.888 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.888 INFO:teuthology.orchestra.run.smithi049.stdout: "pool": 11, 2024-09-21T14:10:00.888 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": "unique_pool_4", 2024-09-21T14:10:00.888 INFO:teuthology.orchestra.run.smithi049.stdout: "create_time": "2024-09-21T14:06:43.392407+0000", 2024-09-21T14:10:00.888 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-09-21T14:10:00.888 INFO:teuthology.orchestra.run.smithi049.stdout: "flags_names": "hashpspool", 2024-09-21T14:10:00.888 INFO:teuthology.orchestra.run.smithi049.stdout: "type": 1, 2024-09-21T14:10:00.888 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 3, 2024-09-21T14:10:00.889 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": 2, 2024-09-21T14:10:00.889 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_rule": 0, 2024-09-21T14:10:00.889 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_count": 0, 2024-09-21T14:10:00.889 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_target": 0, 2024-09-21T14:10:00.889 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T14:10:00.889 INFO:teuthology.orchestra.run.smithi049.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T14:10:00.889 INFO:teuthology.orchestra.run.smithi049.stdout: "is_stretch_pool": false, 2024-09-21T14:10:00.889 INFO:teuthology.orchestra.run.smithi049.stdout: "object_hash": 2, 2024-09-21T14:10:00.889 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscale_mode": "off", 2024-09-21T14:10:00.889 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": 16, 2024-09-21T14:10:00.889 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num": 16, 2024-09-21T14:10:00.889 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_placement_num_target": 16, 2024-09-21T14:10:00.889 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_target": 16, 2024-09-21T14:10:00.889 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num_pending": 16, 2024-09-21T14:10:00.890 INFO:teuthology.orchestra.run.smithi049.stdout: "last_pg_merge_meta": { 2024-09-21T14:10:00.890 INFO:teuthology.orchestra.run.smithi049.stdout: "source_pgid": "0.0", 2024-09-21T14:10:00.890 INFO:teuthology.orchestra.run.smithi049.stdout: "ready_epoch": 0, 2024-09-21T14:10:00.890 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_started": 0, 2024-09-21T14:10:00.890 INFO:teuthology.orchestra.run.smithi049.stdout: "last_epoch_clean": 0, 2024-09-21T14:10:00.890 INFO:teuthology.orchestra.run.smithi049.stdout: "source_version": "0'0", 2024-09-21T14:10:00.890 INFO:teuthology.orchestra.run.smithi049.stdout: "target_version": "0'0" 2024-09-21T14:10:00.890 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.890 INFO:teuthology.orchestra.run.smithi049.stdout: "last_change": "465", 2024-09-21T14:10:00.890 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend": "0", 2024-09-21T14:10:00.890 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T14:10:00.890 INFO:teuthology.orchestra.run.smithi049.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T14:10:00.890 INFO:teuthology.orchestra.run.smithi049.stdout: "auid": 0, 2024-09-21T14:10:00.890 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_mode": "selfmanaged", 2024-09-21T14:10:00.891 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_seq": 0, 2024-09-21T14:10:00.891 INFO:teuthology.orchestra.run.smithi049.stdout: "snap_epoch": 0, 2024-09-21T14:10:00.891 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_snaps": [], 2024-09-21T14:10:00.891 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps": "[]", 2024-09-21T14:10:00.891 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_bytes": 0, 2024-09-21T14:10:00.891 INFO:teuthology.orchestra.run.smithi049.stdout: "quota_max_objects": 0, 2024-09-21T14:10:00.891 INFO:teuthology.orchestra.run.smithi049.stdout: "tiers": [], 2024-09-21T14:10:00.891 INFO:teuthology.orchestra.run.smithi049.stdout: "tier_of": -1, 2024-09-21T14:10:00.891 INFO:teuthology.orchestra.run.smithi049.stdout: "read_tier": -1, 2024-09-21T14:10:00.891 INFO:teuthology.orchestra.run.smithi049.stdout: "write_tier": -1, 2024-09-21T14:10:00.891 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_mode": "none", 2024-09-21T14:10:00.891 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_bytes": 0, 2024-09-21T14:10:00.891 INFO:teuthology.orchestra.run.smithi049.stdout: "target_max_objects": 0, 2024-09-21T14:10:00.891 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T14:10:00.891 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T14:10:00.892 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T14:10:00.892 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_flush_age": 0, 2024-09-21T14:10:00.892 INFO:teuthology.orchestra.run.smithi049.stdout: "cache_min_evict_age": 0, 2024-09-21T14:10:00.892 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profile": "", 2024-09-21T14:10:00.892 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_params": { 2024-09-21T14:10:00.892 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "none" 2024-09-21T14:10:00.892 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.892 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_period": 0, 2024-09-21T14:10:00.892 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_count": 0, 2024-09-21T14:10:00.892 INFO:teuthology.orchestra.run.smithi049.stdout: "use_gmt_hitset": true, 2024-09-21T14:10:00.892 INFO:teuthology.orchestra.run.smithi049.stdout: "min_read_recency_for_promote": 0, 2024-09-21T14:10:00.892 INFO:teuthology.orchestra.run.smithi049.stdout: "min_write_recency_for_promote": 0, 2024-09-21T14:10:00.892 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T14:10:00.892 INFO:teuthology.orchestra.run.smithi049.stdout: "hit_set_search_last_n": 0, 2024-09-21T14:10:00.892 INFO:teuthology.orchestra.run.smithi049.stdout: "grade_table": [], 2024-09-21T14:10:00.893 INFO:teuthology.orchestra.run.smithi049.stdout: "stripe_width": 0, 2024-09-21T14:10:00.893 INFO:teuthology.orchestra.run.smithi049.stdout: "expected_num_objects": 0, 2024-09-21T14:10:00.893 INFO:teuthology.orchestra.run.smithi049.stdout: "fast_read": false, 2024-09-21T14:10:00.893 INFO:teuthology.orchestra.run.smithi049.stdout: "options": {}, 2024-09-21T14:10:00.893 INFO:teuthology.orchestra.run.smithi049.stdout: "application_metadata": { 2024-09-21T14:10:00.893 INFO:teuthology.orchestra.run.smithi049.stdout: "rados": {} 2024-09-21T14:10:00.893 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.893 INFO:teuthology.orchestra.run.smithi049.stdout: "read_balance": { 2024-09-21T14:10:00.893 INFO:teuthology.orchestra.run.smithi049.stdout: "score_type": "Fair distribution", 2024-09-21T14:10:00.893 INFO:teuthology.orchestra.run.smithi049.stdout: "score_acting": 1.8799999952316284, 2024-09-21T14:10:00.893 INFO:teuthology.orchestra.run.smithi049.stdout: "score_stable": 1.8799999952316284, 2024-09-21T14:10:00.893 INFO:teuthology.orchestra.run.smithi049.stdout: "optimal_score": 1.3300000429153442, 2024-09-21T14:10:00.893 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_acting": 2.5, 2024-09-21T14:10:00.893 INFO:teuthology.orchestra.run.smithi049.stdout: "raw_score_stable": 2.5, 2024-09-21T14:10:00.894 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity_weighted": 0.72835731506347656, 2024-09-21T14:10:00.894 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity": 0.97000002861022949, 2024-09-21T14:10:00.894 INFO:teuthology.orchestra.run.smithi049.stdout: "average_primary_affinity_weighted": 0.73000001907348633 2024-09-21T14:10:00.894 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:00.894 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:00.894 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T14:10:00.894 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-09-21T14:10:00.894 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.894 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 0, 2024-09-21T14:10:00.894 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "2595dca9-6219-4e0c-8728-751eb102e73a", 2024-09-21T14:10:00.894 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T14:10:00.894 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T14:10:00.894 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T14:10:00.895 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 0, 2024-09-21T14:10:00.895 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T14:10:00.895 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T14:10:00.895 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 8, 2024-09-21T14:10:00.895 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 524, 2024-09-21T14:10:00.895 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T14:10:00.895 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T14:10:00.895 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T14:10:00.895 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:10:00.895 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.895 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:10:00.895 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6802", 2024-09-21T14:10:00.895 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T14:10:00.895 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.896 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.896 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:10:00.896 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6803", 2024-09-21T14:10:00.896 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T14:10:00.896 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:00.896 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:00.896 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.896 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T14:10:00.896 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:10:00.896 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.896 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:10:00.896 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6804", 2024-09-21T14:10:00.896 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T14:10:00.896 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.896 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.897 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:10:00.897 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6805", 2024-09-21T14:10:00.897 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T14:10:00.897 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:00.897 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:00.897 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.897 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T14:10:00.897 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:10:00.897 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.897 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:10:00.897 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6808", 2024-09-21T14:10:00.897 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T14:10:00.897 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.897 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.898 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:10:00.898 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6809", 2024-09-21T14:10:00.898 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T14:10:00.898 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:00.898 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:00.898 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.898 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T14:10:00.898 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:10:00.898 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.898 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:10:00.898 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6806", 2024-09-21T14:10:00.898 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T14:10:00.898 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.898 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.899 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:10:00.899 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6807", 2024-09-21T14:10:00.899 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2095561412 2024-09-21T14:10:00.899 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:00.899 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:00.899 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.899 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6803/2095561412", 2024-09-21T14:10:00.899 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6805/2095561412", 2024-09-21T14:10:00.899 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6809/2095561412", 2024-09-21T14:10:00.899 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6807/2095561412", 2024-09-21T14:10:00.899 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T14:10:00.899 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T14:10:00.899 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T14:10:00.899 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:00.900 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.900 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.900 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 1, 2024-09-21T14:10:00.900 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "0ba327f9-5524-4a40-abcc-729ded0b1917", 2024-09-21T14:10:00.900 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T14:10:00.900 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T14:10:00.900 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T14:10:00.900 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T14:10:00.900 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T14:10:00.900 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T14:10:00.900 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 13, 2024-09-21T14:10:00.900 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 531, 2024-09-21T14:10:00.900 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T14:10:00.900 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T14:10:00.901 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T14:10:00.901 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:10:00.901 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.901 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:10:00.901 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6810", 2024-09-21T14:10:00.901 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T14:10:00.901 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.901 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.901 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:10:00.901 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6811", 2024-09-21T14:10:00.901 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T14:10:00.901 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:00.901 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:00.901 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.902 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T14:10:00.902 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:10:00.902 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.902 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:10:00.902 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6812", 2024-09-21T14:10:00.902 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T14:10:00.902 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.902 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.902 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:10:00.902 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6813", 2024-09-21T14:10:00.902 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T14:10:00.902 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:00.902 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:00.902 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.903 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T14:10:00.903 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:10:00.903 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.903 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:10:00.903 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6816", 2024-09-21T14:10:00.903 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T14:10:00.903 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.903 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.903 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:10:00.903 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6817", 2024-09-21T14:10:00.903 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T14:10:00.903 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:00.903 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:00.903 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.904 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T14:10:00.904 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:10:00.904 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.904 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:10:00.904 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6814", 2024-09-21T14:10:00.904 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T14:10:00.904 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.904 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.904 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:10:00.904 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6815", 2024-09-21T14:10:00.904 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1382840527 2024-09-21T14:10:00.904 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:00.904 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:00.904 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.904 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6811/1382840527", 2024-09-21T14:10:00.905 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6813/1382840527", 2024-09-21T14:10:00.905 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6817/1382840527", 2024-09-21T14:10:00.905 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6815/1382840527", 2024-09-21T14:10:00.905 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T14:10:00.905 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T14:10:00.905 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T14:10:00.905 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:00.905 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.905 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.905 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 2, 2024-09-21T14:10:00.905 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "27a45750-204d-48fd-bb8f-ebb01273343d", 2024-09-21T14:10:00.905 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T14:10:00.905 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T14:10:00.905 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T14:10:00.906 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 0.8268585205078125, 2024-09-21T14:10:00.906 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 18, 2024-09-21T14:10:00.906 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 100, 2024-09-21T14:10:00.906 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 130, 2024-09-21T14:10:00.906 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 531, 2024-09-21T14:10:00.906 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 101, 2024-09-21T14:10:00.906 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T14:10:00.906 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T14:10:00.906 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:10:00.906 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.906 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:10:00.906 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6818", 2024-09-21T14:10:00.906 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T14:10:00.906 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.907 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.907 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:10:00.907 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6819", 2024-09-21T14:10:00.907 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T14:10:00.907 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:00.907 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:00.907 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.907 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T14:10:00.907 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:10:00.907 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.907 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:10:00.907 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6820", 2024-09-21T14:10:00.907 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T14:10:00.907 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.908 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.908 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:10:00.908 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6821", 2024-09-21T14:10:00.908 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T14:10:00.908 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:00.908 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:00.908 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.908 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T14:10:00.908 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:10:00.908 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.908 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:10:00.908 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6824", 2024-09-21T14:10:00.908 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T14:10:00.908 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.909 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.909 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:10:00.909 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6825", 2024-09-21T14:10:00.909 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T14:10:00.909 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:00.909 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:00.909 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.909 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T14:10:00.909 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:10:00.909 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.909 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:10:00.909 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6822", 2024-09-21T14:10:00.909 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T14:10:00.909 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.909 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.910 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:10:00.910 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6823", 2024-09-21T14:10:00.910 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1945314741 2024-09-21T14:10:00.910 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:00.910 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:00.910 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.910 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6819/1945314741", 2024-09-21T14:10:00.910 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6821/1945314741", 2024-09-21T14:10:00.910 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6825/1945314741", 2024-09-21T14:10:00.910 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6823/1945314741", 2024-09-21T14:10:00.910 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T14:10:00.910 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T14:10:00.910 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T14:10:00.910 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:00.911 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.911 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.911 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 3, 2024-09-21T14:10:00.911 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "528bc390-3dc9-4f3b-ab75-d7412c9625db", 2024-09-21T14:10:00.911 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T14:10:00.911 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T14:10:00.911 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 0.502227783203125, 2024-09-21T14:10:00.911 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T14:10:00.911 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 27, 2024-09-21T14:10:00.911 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 214, 2024-09-21T14:10:00.911 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 240, 2024-09-21T14:10:00.911 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 531, 2024-09-21T14:10:00.911 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 215, 2024-09-21T14:10:00.911 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T14:10:00.912 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T14:10:00.912 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:10:00.912 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.912 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:10:00.912 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6826", 2024-09-21T14:10:00.912 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T14:10:00.912 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.912 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.912 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:10:00.912 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6827", 2024-09-21T14:10:00.912 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T14:10:00.912 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:00.912 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:00.912 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.913 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T14:10:00.913 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:10:00.913 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.913 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:10:00.913 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6828", 2024-09-21T14:10:00.913 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T14:10:00.913 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.913 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.913 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:10:00.913 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6829", 2024-09-21T14:10:00.913 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T14:10:00.913 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:00.913 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:00.913 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.913 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T14:10:00.914 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:10:00.914 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.914 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:10:00.914 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6832", 2024-09-21T14:10:00.914 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T14:10:00.914 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.914 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.914 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:10:00.914 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6833", 2024-09-21T14:10:00.914 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T14:10:00.914 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:00.914 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:00.914 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.914 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T14:10:00.915 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:10:00.915 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.915 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:10:00.915 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6830", 2024-09-21T14:10:00.915 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T14:10:00.915 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.915 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:10:00.916 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-21T14:10:00.916 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:10:00.917 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.917 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:10:00.917 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6831", 2024-09-21T14:10:00.917 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 886735721 2024-09-21T14:10:00.917 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:00.917 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:00.917 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.917 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.49:6827/886735721", 2024-09-21T14:10:00.917 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.49:6829/886735721", 2024-09-21T14:10:00.917 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.49:6833/886735721", 2024-09-21T14:10:00.917 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.49:6831/886735721", 2024-09-21T14:10:00.918 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T14:10:00.918 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T14:10:00.918 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T14:10:00.918 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:00.918 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.918 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.918 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 4, 2024-09-21T14:10:00.918 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "6fc64698-2d37-4db0-bf3d-c7e612c45163", 2024-09-21T14:10:00.918 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T14:10:00.918 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T14:10:00.918 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T14:10:00.918 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T14:10:00.918 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T14:10:00.918 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T14:10:00.919 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 33, 2024-09-21T14:10:00.919 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 531, 2024-09-21T14:10:00.919 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T14:10:00.919 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T14:10:00.919 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T14:10:00.919 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:10:00.919 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.919 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:10:00.919 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6800", 2024-09-21T14:10:00.919 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T14:10:00.919 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.919 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.919 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:10:00.919 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6801", 2024-09-21T14:10:00.920 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T14:10:00.920 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:00.920 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:00.920 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.920 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T14:10:00.920 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:10:00.920 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.920 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:10:00.920 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6802", 2024-09-21T14:10:00.920 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T14:10:00.920 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.920 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.920 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:10:00.920 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6803", 2024-09-21T14:10:00.921 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T14:10:00.921 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:00.921 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:00.921 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.921 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T14:10:00.921 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:10:00.921 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.921 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:10:00.921 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6806", 2024-09-21T14:10:00.921 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T14:10:00.921 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.921 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.921 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:10:00.921 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6807", 2024-09-21T14:10:00.922 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T14:10:00.922 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:00.922 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:00.922 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.922 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T14:10:00.922 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:10:00.922 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.922 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:10:00.922 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6804", 2024-09-21T14:10:00.922 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T14:10:00.922 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.922 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.922 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:10:00.923 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6805", 2024-09-21T14:10:00.923 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1702447776 2024-09-21T14:10:00.923 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:00.923 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:00.923 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.923 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.187:6801/1702447776", 2024-09-21T14:10:00.923 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.187:6803/1702447776", 2024-09-21T14:10:00.923 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.187:6807/1702447776", 2024-09-21T14:10:00.923 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.187:6805/1702447776", 2024-09-21T14:10:00.923 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T14:10:00.923 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T14:10:00.923 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T14:10:00.923 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:00.923 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.923 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.924 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 5, 2024-09-21T14:10:00.924 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "8d7f62a1-7e92-4e06-9cad-86b96be514d8", 2024-09-21T14:10:00.924 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T14:10:00.924 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T14:10:00.924 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T14:10:00.924 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T14:10:00.924 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 39, 2024-09-21T14:10:00.924 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 324, 2024-09-21T14:10:00.924 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 345, 2024-09-21T14:10:00.924 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 531, 2024-09-21T14:10:00.924 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 325, 2024-09-21T14:10:00.924 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T14:10:00.924 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T14:10:00.925 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:10:00.925 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.925 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:10:00.925 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6808", 2024-09-21T14:10:00.925 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1595353965 2024-09-21T14:10:00.925 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.925 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.925 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:10:00.925 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6809", 2024-09-21T14:10:00.925 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1595353965 2024-09-21T14:10:00.925 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:00.925 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:00.925 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.925 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T14:10:00.926 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:10:00.926 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.926 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:10:00.926 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6810", 2024-09-21T14:10:00.926 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1595353965 2024-09-21T14:10:00.926 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.926 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.926 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:10:00.926 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6811", 2024-09-21T14:10:00.926 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1595353965 2024-09-21T14:10:00.926 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:00.926 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:00.926 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.927 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T14:10:00.927 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:10:00.927 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.927 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:10:00.927 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6814", 2024-09-21T14:10:00.927 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1595353965 2024-09-21T14:10:00.927 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.927 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.927 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:10:00.927 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6815", 2024-09-21T14:10:00.927 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1595353965 2024-09-21T14:10:00.927 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:00.927 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:00.927 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.927 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T14:10:00.928 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:10:00.928 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.928 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:10:00.928 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6812", 2024-09-21T14:10:00.928 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1595353965 2024-09-21T14:10:00.928 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.928 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.928 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:10:00.928 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6813", 2024-09-21T14:10:00.928 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1595353965 2024-09-21T14:10:00.928 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:00.928 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:00.928 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.928 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.187:6809/1595353965", 2024-09-21T14:10:00.929 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.187:6811/1595353965", 2024-09-21T14:10:00.929 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.187:6815/1595353965", 2024-09-21T14:10:00.929 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.187:6813/1595353965", 2024-09-21T14:10:00.929 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T14:10:00.929 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T14:10:00.929 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T14:10:00.929 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:00.929 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.929 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.929 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 6, 2024-09-21T14:10:00.929 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "3a7afd13-26ce-444c-9cc0-81ce2eab6ce5", 2024-09-21T14:10:00.929 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T14:10:00.929 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T14:10:00.929 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 0.9076080322265625, 2024-09-21T14:10:00.930 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 1, 2024-09-21T14:10:00.930 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 0, 2024-09-21T14:10:00.930 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 0, 2024-09-21T14:10:00.930 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 47, 2024-09-21T14:10:00.930 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 531, 2024-09-21T14:10:00.930 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 0, 2024-09-21T14:10:00.930 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T14:10:00.930 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T14:10:00.930 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:10:00.930 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.930 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:10:00.930 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6816", 2024-09-21T14:10:00.930 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T14:10:00.930 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.930 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.931 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:10:00.931 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6817", 2024-09-21T14:10:00.931 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T14:10:00.931 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:00.931 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:00.931 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.931 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T14:10:00.931 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:10:00.931 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.931 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:10:00.931 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6818", 2024-09-21T14:10:00.931 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T14:10:00.931 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.931 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.932 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:10:00.932 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6819", 2024-09-21T14:10:00.932 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T14:10:00.932 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:00.932 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:00.932 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.932 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T14:10:00.932 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:10:00.932 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.932 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:10:00.932 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6822", 2024-09-21T14:10:00.932 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T14:10:00.932 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.932 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.932 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:10:00.933 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6823", 2024-09-21T14:10:00.933 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T14:10:00.933 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:00.933 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:00.933 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.933 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T14:10:00.933 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:10:00.933 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.933 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:10:00.933 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6820", 2024-09-21T14:10:00.933 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T14:10:00.933 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.933 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.933 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:10:00.934 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6821", 2024-09-21T14:10:00.934 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 2728426525 2024-09-21T14:10:00.934 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:00.934 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:00.934 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.934 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.187:6817/2728426525", 2024-09-21T14:10:00.934 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.187:6819/2728426525", 2024-09-21T14:10:00.934 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.187:6823/2728426525", 2024-09-21T14:10:00.934 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.187:6821/2728426525", 2024-09-21T14:10:00.934 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T14:10:00.934 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T14:10:00.934 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T14:10:00.934 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:00.934 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.935 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.935 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 7, 2024-09-21T14:10:00.935 INFO:teuthology.orchestra.run.smithi049.stdout: "uuid": "1e08d891-8747-402e-b4c2-aa3802ce8537", 2024-09-21T14:10:00.935 INFO:teuthology.orchestra.run.smithi049.stdout: "up": 1, 2024-09-21T14:10:00.935 INFO:teuthology.orchestra.run.smithi049.stdout: "in": 1, 2024-09-21T14:10:00.935 INFO:teuthology.orchestra.run.smithi049.stdout: "weight": 1, 2024-09-21T14:10:00.935 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_affinity": 0, 2024-09-21T14:10:00.935 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_begin": 54, 2024-09-21T14:10:00.935 INFO:teuthology.orchestra.run.smithi049.stdout: "last_clean_end": 272, 2024-09-21T14:10:00.935 INFO:teuthology.orchestra.run.smithi049.stdout: "up_from": 298, 2024-09-21T14:10:00.935 INFO:teuthology.orchestra.run.smithi049.stdout: "up_thru": 461, 2024-09-21T14:10:00.935 INFO:teuthology.orchestra.run.smithi049.stdout: "down_at": 273, 2024-09-21T14:10:00.935 INFO:teuthology.orchestra.run.smithi049.stdout: "lost_at": 0, 2024-09-21T14:10:00.935 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addrs": { 2024-09-21T14:10:00.936 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:10:00.936 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.936 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:10:00.936 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6824", 2024-09-21T14:10:00.936 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 13894964 2024-09-21T14:10:00.936 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.936 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.936 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:10:00.936 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6825", 2024-09-21T14:10:00.936 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 13894964 2024-09-21T14:10:00.936 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:00.936 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:00.936 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.936 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addrs": { 2024-09-21T14:10:00.936 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:10:00.937 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.937 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:10:00.937 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6826", 2024-09-21T14:10:00.937 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 13894964 2024-09-21T14:10:00.937 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.937 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.937 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:10:00.937 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6827", 2024-09-21T14:10:00.937 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 13894964 2024-09-21T14:10:00.937 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:00.937 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:00.937 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.937 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addrs": { 2024-09-21T14:10:00.937 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:10:00.938 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.938 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:10:00.938 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6830", 2024-09-21T14:10:00.938 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 13894964 2024-09-21T14:10:00.938 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.938 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.938 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:10:00.938 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6831", 2024-09-21T14:10:00.938 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 13894964 2024-09-21T14:10:00.938 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:00.938 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:00.938 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.938 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addrs": { 2024-09-21T14:10:00.938 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-09-21T14:10:00.939 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.939 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-09-21T14:10:00.939 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6828", 2024-09-21T14:10:00.939 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 13894964 2024-09-21T14:10:00.939 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.939 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.939 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-09-21T14:10:00.939 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.187:6829", 2024-09-21T14:10:00.939 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 13894964 2024-09-21T14:10:00.939 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:00.939 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:00.939 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.939 INFO:teuthology.orchestra.run.smithi049.stdout: "public_addr": "172.21.15.187:6825/13894964", 2024-09-21T14:10:00.940 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_addr": "172.21.15.187:6827/13894964", 2024-09-21T14:10:00.940 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_back_addr": "172.21.15.187:6831/13894964", 2024-09-21T14:10:00.940 INFO:teuthology.orchestra.run.smithi049.stdout: "heartbeat_front_addr": "172.21.15.187:6829/13894964", 2024-09-21T14:10:00.940 INFO:teuthology.orchestra.run.smithi049.stdout: "state": [ 2024-09-21T14:10:00.940 INFO:teuthology.orchestra.run.smithi049.stdout: "exists", 2024-09-21T14:10:00.940 INFO:teuthology.orchestra.run.smithi049.stdout: "up" 2024-09-21T14:10:00.940 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:00.940 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:00.940 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T14:10:00.940 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_xinfo": [ 2024-09-21T14:10:00.940 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.940 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 0, 2024-09-21T14:10:00.940 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T14:10:00.940 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T14:10:00.941 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T14:10:00.941 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T14:10:00.941 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T14:10:00.941 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T14:08:59.177843+0000", 2024-09-21T14:10:00.941 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T14:10:00.941 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.941 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.941 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 1, 2024-09-21T14:10:00.941 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T14:10:00.941 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T14:10:00.941 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T14:10:00.941 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T14:10:00.941 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T14:10:00.941 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T14:09:54.028052+0000", 2024-09-21T14:10:00.941 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T14:10:00.942 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.942 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.942 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 2, 2024-09-21T14:10:00.942 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T14:10:00.942 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T14:10:00.942 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T14:10:00.942 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T14:10:00.942 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T14:10:00.942 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T14:08:56.325350+0000", 2024-09-21T14:10:00.942 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 100 2024-09-21T14:10:00.942 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.942 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.942 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 3, 2024-09-21T14:10:00.942 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T14:10:00.943 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T14:10:00.943 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T14:10:00.943 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T14:10:00.943 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T14:10:00.943 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T14:09:31.746221+0000", 2024-09-21T14:10:00.943 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 214 2024-09-21T14:10:00.943 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.943 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.943 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 4, 2024-09-21T14:10:00.943 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T14:10:00.943 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T14:10:00.943 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T14:10:00.943 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T14:10:00.943 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T14:10:00.944 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T14:09:28.963827+0000", 2024-09-21T14:10:00.944 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T14:10:00.944 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.944 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.944 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 5, 2024-09-21T14:10:00.944 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T14:10:00.944 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T14:10:00.944 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T14:10:00.944 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T14:10:00.944 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T14:10:00.944 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T14:09:29.595445+0000", 2024-09-21T14:10:00.944 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 324 2024-09-21T14:10:00.944 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.944 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.944 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 6, 2024-09-21T14:10:00.945 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T14:10:00.945 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T14:10:00.945 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T14:10:00.945 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T14:10:00.945 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T14:10:00.945 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T14:08:39.866457+0000", 2024-09-21T14:10:00.945 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 0 2024-09-21T14:10:00.945 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.945 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.945 INFO:teuthology.orchestra.run.smithi049.stdout: "osd": 7, 2024-09-21T14:10:00.945 INFO:teuthology.orchestra.run.smithi049.stdout: "down_stamp": "0.000000", 2024-09-21T14:10:00.945 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_probability": 0, 2024-09-21T14:10:00.945 INFO:teuthology.orchestra.run.smithi049.stdout: "laggy_interval": 0, 2024-09-21T14:10:00.945 INFO:teuthology.orchestra.run.smithi049.stdout: "features": 4540719139924082687, 2024-09-21T14:10:00.946 INFO:teuthology.orchestra.run.smithi049.stdout: "old_weight": 0, 2024-09-21T14:10:00.946 INFO:teuthology.orchestra.run.smithi049.stdout: "last_purged_snaps_scrub": "2024-09-21T14:09:33.715526+0000", 2024-09-21T14:10:00.946 INFO:teuthology.orchestra.run.smithi049.stdout: "dead_epoch": 272 2024-09-21T14:10:00.946 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:00.946 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T14:10:00.946 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap": [ 2024-09-21T14:10:00.946 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.946 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "1.9", 2024-09-21T14:10:00.946 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-09-21T14:10:00.946 INFO:teuthology.orchestra.run.smithi049.stdout: 2, 2024-09-21T14:10:00.946 INFO:teuthology.orchestra.run.smithi049.stdout: 3, 2024-09-21T14:10:00.946 INFO:teuthology.orchestra.run.smithi049.stdout: 0 2024-09-21T14:10:00.946 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:00.947 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.947 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.947 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "3.14", 2024-09-21T14:10:00.947 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-09-21T14:10:00.947 INFO:teuthology.orchestra.run.smithi049.stdout: 7, 2024-09-21T14:10:00.947 INFO:teuthology.orchestra.run.smithi049.stdout: 6, 2024-09-21T14:10:00.947 INFO:teuthology.orchestra.run.smithi049.stdout: 0 2024-09-21T14:10:00.947 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:00.947 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:00.947 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T14:10:00.947 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap_items": [ 2024-09-21T14:10:00.947 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.947 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "3.1c", 2024-09-21T14:10:00.947 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-09-21T14:10:00.948 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.948 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 4, 2024-09-21T14:10:00.948 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 2 2024-09-21T14:10:00.948 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:00.948 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:00.948 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.948 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.948 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "4.2", 2024-09-21T14:10:00.948 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-09-21T14:10:00.948 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.948 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 1, 2024-09-21T14:10:00.948 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 2 2024-09-21T14:10:00.948 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:00.948 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:00.948 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.949 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.949 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "4.6", 2024-09-21T14:10:00.949 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-09-21T14:10:00.949 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.949 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 1, 2024-09-21T14:10:00.949 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 5 2024-09-21T14:10:00.949 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:00.949 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:00.949 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.949 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.949 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "4.10", 2024-09-21T14:10:00.949 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-09-21T14:10:00.949 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.949 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 1, 2024-09-21T14:10:00.949 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 2 2024-09-21T14:10:00.949 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:00.950 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:00.950 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.950 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.950 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "4.1c", 2024-09-21T14:10:00.950 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-09-21T14:10:00.950 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.950 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 1, 2024-09-21T14:10:00.950 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 5 2024-09-21T14:10:00.950 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:00.950 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:00.950 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.950 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.950 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "4.1d", 2024-09-21T14:10:00.950 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-09-21T14:10:00.951 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.951 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 1, 2024-09-21T14:10:00.951 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 2 2024-09-21T14:10:00.951 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:00.951 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:00.951 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.951 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.951 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "5.16", 2024-09-21T14:10:00.951 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-09-21T14:10:00.951 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.951 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 1, 2024-09-21T14:10:00.951 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 6 2024-09-21T14:10:00.951 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.952 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.952 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 3, 2024-09-21T14:10:00.952 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 5 2024-09-21T14:10:00.952 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:00.952 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:00.952 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.952 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.952 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "6.7", 2024-09-21T14:10:00.952 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-09-21T14:10:00.952 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.952 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 3, 2024-09-21T14:10:00.952 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 7 2024-09-21T14:10:00.952 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:00.952 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:00.952 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.953 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.953 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "6.b", 2024-09-21T14:10:00.953 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-09-21T14:10:00.953 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.953 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 3, 2024-09-21T14:10:00.953 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 1 2024-09-21T14:10:00.953 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.953 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.953 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 7, 2024-09-21T14:10:00.953 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 6 2024-09-21T14:10:00.953 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:00.953 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:00.953 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.953 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.954 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "6.1f", 2024-09-21T14:10:00.954 INFO:teuthology.orchestra.run.smithi049.stdout: "mappings": [ 2024-09-21T14:10:00.954 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.954 INFO:teuthology.orchestra.run.smithi049.stdout: "from": 3, 2024-09-21T14:10:00.954 INFO:teuthology.orchestra.run.smithi049.stdout: "to": 0 2024-09-21T14:10:00.954 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:00.954 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:00.954 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:00.954 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T14:10:00.954 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_upmap_primaries": [], 2024-09-21T14:10:00.954 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_temp": [ 2024-09-21T14:10:00.954 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:00.955 INFO:teuthology.orchestra.run.smithi049.stdout: "pgid": "11.0", 2024-09-21T14:10:00.955 INFO:teuthology.orchestra.run.smithi049.stdout: "osds": [ 2024-09-21T14:10:00.955 INFO:teuthology.orchestra.run.smithi049.stdout: 1, 2024-09-21T14:10:00.955 INFO:teuthology.orchestra.run.smithi049.stdout: 7, 2024-09-21T14:10:00.955 INFO:teuthology.orchestra.run.smithi049.stdout: 3 2024-09-21T14:10:00.955 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:00.955 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:00.955 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T14:10:00.955 INFO:teuthology.orchestra.run.smithi049.stdout: "primary_temp": [], 2024-09-21T14:10:00.955 INFO:teuthology.orchestra.run.smithi049.stdout: "blocklist": { 2024-09-21T14:10:00.955 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/2731793917": "2024-09-22T13:49:27.311313+0000", 2024-09-21T14:10:00.955 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/799831425": "2024-09-22T13:49:27.311313+0000", 2024-09-21T14:10:00.955 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/2518197280": "2024-09-22T13:43:28.110499+0000", 2024-09-21T14:10:00.955 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3556829486": "2024-09-22T13:43:28.110499+0000", 2024-09-21T14:10:00.956 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/1249926115": "2024-09-22T13:49:27.311313+0000", 2024-09-21T14:10:00.956 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/231530138": "2024-09-22T13:43:28.110499+0000", 2024-09-21T14:10:00.956 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/4220781134": "2024-09-22T13:49:27.311313+0000", 2024-09-21T14:10:00.956 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/202325455": "2024-09-22T13:43:28.110499+0000", 2024-09-21T14:10:00.956 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/4129636473": "2024-09-22T13:43:56.633723+0000", 2024-09-21T14:10:00.956 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/2518197280": "2024-09-22T13:43:28.110499+0000", 2024-09-21T14:10:00.956 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3514741273": "2024-09-22T13:43:56.633723+0000", 2024-09-21T14:10:00.956 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/4129636473": "2024-09-22T13:43:56.633723+0000", 2024-09-21T14:10:00.956 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3125743776": "2024-09-22T13:43:56.633723+0000", 2024-09-21T14:10:00.956 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3600977363": "2024-09-22T13:43:56.633723+0000", 2024-09-21T14:10:00.956 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6801/3453571598": "2024-09-22T13:49:27.311313+0000", 2024-09-21T14:10:00.956 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:6800/3453571598": "2024-09-22T13:49:27.311313+0000", 2024-09-21T14:10:00.956 INFO:teuthology.orchestra.run.smithi049.stdout: "172.21.15.49:0/3553734432": "2024-09-22T13:49:27.311313+0000" 2024-09-21T14:10:00.957 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.957 INFO:teuthology.orchestra.run.smithi049.stdout: "range_blocklist": {}, 2024-09-21T14:10:00.957 INFO:teuthology.orchestra.run.smithi049.stdout: "erasure_code_profiles": { 2024-09-21T14:10:00.957 INFO:teuthology.orchestra.run.smithi049.stdout: "default": { 2024-09-21T14:10:00.957 INFO:teuthology.orchestra.run.smithi049.stdout: "crush-failure-domain": "osd", 2024-09-21T14:10:00.957 INFO:teuthology.orchestra.run.smithi049.stdout: "k": "2", 2024-09-21T14:10:00.957 INFO:teuthology.orchestra.run.smithi049.stdout: "m": "1", 2024-09-21T14:10:00.957 INFO:teuthology.orchestra.run.smithi049.stdout: "plugin": "jerasure", 2024-09-21T14:10:00.957 INFO:teuthology.orchestra.run.smithi049.stdout: "technique": "reed_sol_van" 2024-09-21T14:10:00.957 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:00.957 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:00.957 INFO:teuthology.orchestra.run.smithi049.stdout: "removed_snaps_queue": [], 2024-09-21T14:10:00.957 INFO:teuthology.orchestra.run.smithi049.stdout: "new_removed_snaps": [], 2024-09-21T14:10:00.957 INFO:teuthology.orchestra.run.smithi049.stdout: "new_purged_snaps": [], 2024-09-21T14:10:00.958 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_node_flags": {}, 2024-09-21T14:10:00.958 INFO:teuthology.orchestra.run.smithi049.stdout: "device_class_flags": {}, 2024-09-21T14:10:00.958 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode": { 2024-09-21T14:10:00.958 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode_enabled": false, 2024-09-21T14:10:00.958 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_bucket_count": 0, 2024-09-21T14:10:00.958 INFO:teuthology.orchestra.run.smithi049.stdout: "degraded_stretch_mode": 0, 2024-09-21T14:10:00.958 INFO:teuthology.orchestra.run.smithi049.stdout: "recovering_stretch_mode": 0, 2024-09-21T14:10:00.958 INFO:teuthology.orchestra.run.smithi049.stdout: "stretch_mode_bucket": 0 2024-09-21T14:10:00.958 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:00.958 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T14:10:00.993 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:00.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:00.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:10:01.094 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:10:01.135 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:01.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:01.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:10:01.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:00 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:10:01.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:00 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:10:01.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:00 smithi187 ceph-mon[33032]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; Reduced data availability: 1 pg inactive, 1 pg peering; Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull 2024-09-21T14:10:01.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:00 smithi187 ceph-mon[33032]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-09-21T14:10:01.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:00 smithi187 ceph-mon[33032]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive, 1 pg peering 2024-09-21T14:10:01.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:00 smithi187 ceph-mon[33032]: pg 5.1e is stuck peering for 17m, current state peering, last acting [2,0,7] 2024-09-21T14:10:01.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:00 smithi187 ceph-mon[33032]: [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull 2024-09-21T14:10:01.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:00 smithi187 ceph-mon[33032]: pg 11.0 is active+remapped+backfill_toofull, acting [1,7,3] 2024-09-21T14:10:01.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:00 smithi187 ceph-mon[33032]: osdmap e536: 8 total, 8 up, 8 in 2024-09-21T14:10:01.236 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:10:01.237 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T14:10:01.278 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:01.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:01.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:10:01.380 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:10:01.408 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:01.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:01.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:10:01.430 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T14:10:01.510 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:10:01.538 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:01.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:01.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:10:01.640 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:10:01.664 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:01.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:01.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:10:01.766 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:10:01.795 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:01.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:01.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:10:01.849 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T14:10:01.897 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:10:01.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:01.934 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:01.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:10:02.035 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:10:02.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:02.073 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:02.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:10:02.174 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:10:02.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:01 smithi187 ceph-mon[33032]: pgmap v1080: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 2.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 341 B/s rd, 0 B/s wr, 858 op/s; 109/99168 objects misplaced (0.110%) 2024-09-21T14:10:02.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:01 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:10:02.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:01 smithi187 ceph-mon[33032]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-21T14:10:02.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:01 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1246017344' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T14:10:02.204 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:02.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:02.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:10:02.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:01 smithi049 ceph-mon[40489]: pgmap v1080: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 2.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 341 B/s rd, 0 B/s wr, 858 op/s; 109/99168 objects misplaced (0.110%) 2024-09-21T14:10:02.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:01 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:10:02.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:01 smithi049 ceph-mon[40489]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-21T14:10:02.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:01 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1246017344' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T14:10:02.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:01 smithi049 ceph-mon[30080]: pgmap v1080: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 2.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 341 B/s rd, 0 B/s wr, 858 op/s; 109/99168 objects misplaced (0.110%) 2024-09-21T14:10:02.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:01 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:10:02.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:01 smithi049 ceph-mon[30080]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-21T14:10:02.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:01 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1246017344' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-21T14:10:02.266 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:10:02.307 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:10:02.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:02.352 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:02.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:10:02.453 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:10:02.487 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:02.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:02.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:10:02.589 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:10:02.628 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:02.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:02.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:10:02.730 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:10:02.754 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:02.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:02.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:10:02.856 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:10:02.888 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:02.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:02.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:10:02.990 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:10:03.016 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:03.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:03.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:10:03.118 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:10:03.151 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:03.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:03.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:10:03.204 INFO:tasks.thrashosds.thrasher:j is {'epoch': 536, 'fsid': '40a3d2e8-781f-11ef-baf6-efdc52797490', 'created': '2024-09-21T13:42:57.838275+0000', 'modified': '2024-09-21T14:10:00.254985+0000', 'last_up_change': '2024-09-21T14:01:40.158454+0000', 'last_in_change': '2024-09-21T13:58:16.730861+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 22, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 11, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T13:46:21.096244+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 21, 'pg_placement_num': 21, 'pg_placement_num_target': 21, 'pg_num_target': 21, 'pg_num_pending': 21, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '438', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '436', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.009999990463257, 'score_stable': 2.009999990463257, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 2.6700000762939453, 'raw_score_stable': 2.6700000762939453, 'primary_affinity_weighted': 0.7283573150634766, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.7300000190734863}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-21T13:48:55.540712+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 63, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.7283573150634766, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.7300000190734863}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-21T13:48:56.417456+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.7283573150634766, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.7300000190734863}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-21T13:48:58.128455+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.7283573150634766, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.7300000190734863}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-21T13:49:00.069361+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3200000524520874, 'score_stable': 1.3200000524520874, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.7283573150634766, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.7300000190734863}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-21T13:49:02.127399+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.7283573150634766, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.7300000190734863}}, {'pool': 11, 'pool_name': 'unique_pool_4', 'create_time': '2024-09-21T14:06:43.392407+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '465', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 0.7283573150634766, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.7300000190734863}}], 'osds': [{'osd': 0, 'uuid': '2595dca9-6219-4e0c-8728-751eb102e73a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 524, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6802', 'nonce': 2095561412}, {'type': 'v1', 'addr': '172.21.15.49:6803', 'nonce': 2095561412}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6804', 'nonce': 2095561412}, {'type': 'v1', 'addr': '172.21.15.49:6805', 'nonce': 2095561412}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6808', 'nonce': 2095561412}, {'type': 'v1', 'addr': '172.21.15.49:6809', 'nonce': 2095561412}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6806', 'nonce': 2095561412}, {'type': 'v1', 'addr': '172.21.15.49:6807', 'nonce': 2095561412}]}, 'public_addr': '172.21.15.49:6803/2095561412', 'cluster_addr': '172.21.15.49:6805/2095561412', 'heartbeat_back_addr': '172.21.15.49:6809/2095561412', 'heartbeat_front_addr': '172.21.15.49:6807/2095561412', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '0ba327f9-5524-4a40-abcc-729ded0b1917', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 531, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6810', 'nonce': 1382840527}, {'type': 'v1', 'addr': '172.21.15.49:6811', 'nonce': 1382840527}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6812', 'nonce': 1382840527}, {'type': 'v1', 'addr': '172.21.15.49:6813', 'nonce': 1382840527}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6816', 'nonce': 1382840527}, {'type': 'v1', 'addr': '172.21.15.49:6817', 'nonce': 1382840527}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6814', 'nonce': 1382840527}, {'type': 'v1', 'addr': '172.21.15.49:6815', 'nonce': 1382840527}]}, 'public_addr': '172.21.15.49:6811/1382840527', 'cluster_addr': '172.21.15.49:6813/1382840527', 'heartbeat_back_addr': '172.21.15.49:6817/1382840527', 'heartbeat_front_addr': '172.21.15.49:6815/1382840527', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '27a45750-204d-48fd-bb8f-ebb01273343d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.8268585205078125, 'last_clean_begin': 18, 'last_clean_end': 100, 'up_from': 130, 'up_thru': 531, 'down_at': 101, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6818', 'nonce': 1945314741}, {'type': 'v1', 'addr': '172.21.15.49:6819', 'nonce': 1945314741}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6820', 'nonce': 1945314741}, {'type': 'v1', 'addr': '172.21.15.49:6821', 'nonce': 1945314741}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6824', 'nonce': 1945314741}, {'type': 'v1', 'addr': '172.21.15.49:6825', 'nonce': 1945314741}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6822', 'nonce': 1945314741}, {'type': 'v1', 'addr': '172.21.15.49:6823', 'nonce': 1945314741}]}, 'public_addr': '172.21.15.49:6819/1945314741', 'cluster_addr': '172.21.15.49:6821/1945314741', 'heartbeat_back_addr': '172.21.15.49:6825/1945314741', 'heartbeat_front_addr': '172.21.15.49:6823/1945314741', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '528bc390-3dc9-4f3b-ab75-d7412c9625db', 'up': 1, 'in': 1, 'weight': 0.502227783203125, 'primary_affinity': 1, 'last_clean_begin': 27, 'last_clean_end': 214, 'up_from': 240, 'up_thru': 531, 'down_at': 215, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6826', 'nonce': 886735721}, {'type': 'v1', 'addr': '172.21.15.49:6827', 'nonce': 886735721}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6828', 'nonce': 886735721}, {'type': 'v1', 'addr': '172.21.15.49:6829', 'nonce': 886735721}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6832', 'nonce': 886735721}, {'type': 'v1', 'addr': '172.21.15.49:6833', 'nonce': 886735721}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6830', 'nonce': 886735721}, {'type': 'v1', 'addr': '172.21.15.49:6831', 'nonce': 886735721}]}, 'public_addr': '172.21.15.49:6827/886735721', 'cluster_addr': '172.21.15.49:6829/886735721', 'heartbeat_back_addr': '172.21.15.49:6833/886735721', 'heartbeat_front_addr': '172.21.15.49:6831/886735721', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '6fc64698-2d37-4db0-bf3d-c7e612c45163', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 531, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6800', 'nonce': 1702447776}, {'type': 'v1', 'addr': '172.21.15.187:6801', 'nonce': 1702447776}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6802', 'nonce': 1702447776}, {'type': 'v1', 'addr': '172.21.15.187:6803', 'nonce': 1702447776}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6806', 'nonce': 1702447776}, {'type': 'v1', 'addr': '172.21.15.187:6807', 'nonce': 1702447776}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6804', 'nonce': 1702447776}, {'type': 'v1', 'addr': '172.21.15.187:6805', 'nonce': 1702447776}]}, 'public_addr': '172.21.15.187:6801/1702447776', 'cluster_addr': '172.21.15.187:6803/1702447776', 'heartbeat_back_addr': '172.21.15.187:6807/1702447776', 'heartbeat_front_addr': '172.21.15.187:6805/1702447776', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '8d7f62a1-7e92-4e06-9cad-86b96be514d8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 39, 'last_clean_end': 324, 'up_from': 345, 'up_thru': 531, 'down_at': 325, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6808', 'nonce': 1595353965}, {'type': 'v1', 'addr': '172.21.15.187:6809', 'nonce': 1595353965}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6810', 'nonce': 1595353965}, {'type': 'v1', 'addr': '172.21.15.187:6811', 'nonce': 1595353965}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6814', 'nonce': 1595353965}, {'type': 'v1', 'addr': '172.21.15.187:6815', 'nonce': 1595353965}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6812', 'nonce': 1595353965}, {'type': 'v1', 'addr': '172.21.15.187:6813', 'nonce': 1595353965}]}, 'public_addr': '172.21.15.187:6809/1595353965', 'cluster_addr': '172.21.15.187:6811/1595353965', 'heartbeat_back_addr': '172.21.15.187:6815/1595353965', 'heartbeat_front_addr': '172.21.15.187:6813/1595353965', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '3a7afd13-26ce-444c-9cc0-81ce2eab6ce5', 'up': 1, 'in': 1, 'weight': 0.9076080322265625, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 531, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6816', 'nonce': 2728426525}, {'type': 'v1', 'addr': '172.21.15.187:6817', 'nonce': 2728426525}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6818', 'nonce': 2728426525}, {'type': 'v1', 'addr': '172.21.15.187:6819', 'nonce': 2728426525}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6822', 'nonce': 2728426525}, {'type': 'v1', 'addr': '172.21.15.187:6823', 'nonce': 2728426525}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6820', 'nonce': 2728426525}, {'type': 'v1', 'addr': '172.21.15.187:6821', 'nonce': 2728426525}]}, 'public_addr': '172.21.15.187:6817/2728426525', 'cluster_addr': '172.21.15.187:6819/2728426525', 'heartbeat_back_addr': '172.21.15.187:6823/2728426525', 'heartbeat_front_addr': '172.21.15.187:6821/2728426525', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1e08d891-8747-402e-b4c2-aa3802ce8537', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 54, 'last_clean_end': 272, 'up_from': 298, 'up_thru': 461, 'down_at': 273, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 13894964}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 13894964}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 13894964}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 13894964}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6830', 'nonce': 13894964}, {'type': 'v1', 'addr': '172.21.15.187:6831', 'nonce': 13894964}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6828', 'nonce': 13894964}, {'type': 'v1', 'addr': '172.21.15.187:6829', 'nonce': 13894964}]}, 'public_addr': '172.21.15.187:6825/13894964', 'cluster_addr': '172.21.15.187:6827/13894964', 'heartbeat_back_addr': '172.21.15.187:6831/13894964', 'heartbeat_front_addr': '172.21.15.187:6829/13894964', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T14:08:59.177843+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T14:09:54.028052+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T14:08:56.325350+0000', 'dead_epoch': 100}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T14:09:31.746221+0000', 'dead_epoch': 214}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T14:09:28.963827+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T14:09:29.595445+0000', 'dead_epoch': 324}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T14:08:39.866457+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T14:09:33.715526+0000', 'dead_epoch': 272}], 'pg_upmap': [{'pgid': '1.9', 'osds': [2, 3, 0]}, {'pgid': '3.14', 'osds': [7, 6, 0]}], 'pg_upmap_items': [{'pgid': '3.1c', 'mappings': [{'from': 4, 'to': 2}]}, {'pgid': '4.2', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.6', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.10', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1c', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.1d', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '5.16', 'mappings': [{'from': 1, 'to': 6}, {'from': 3, 'to': 5}]}, {'pgid': '6.7', 'mappings': [{'from': 3, 'to': 7}]}, {'pgid': '6.b', 'mappings': [{'from': 3, 'to': 1}, {'from': 7, 'to': 6}]}, {'pgid': '6.1f', 'mappings': [{'from': 3, 'to': 0}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '11.0', 'osds': [1, 7, 3]}], 'primary_temp': [], 'blocklist': {'172.21.15.49:0/2731793917': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:0/799831425': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:6801/2518197280': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:0/3556829486': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:0/1249926115': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:0/231530138': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:0/4220781134': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:0/202325455': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:6800/4129636473': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:6800/2518197280': '2024-09-22T13:43:28.110499+0000', '172.21.15.49:0/3514741273': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:6801/4129636473': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:0/3125743776': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:0/3600977363': '2024-09-22T13:43:56.633723+0000', '172.21.15.49:6801/3453571598': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:6800/3453571598': '2024-09-22T13:49:27.311313+0000', '172.21.15.49:0/3553734432': '2024-09-22T13:49:27.311313+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-21T14:10:03.204 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T14:10:03.253 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:10:03.279 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:03.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:03.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:10:03.368 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:10:03.379 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:10:03.380 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:10:03.410 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:03.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:03.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:10:03.511 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:10:03.545 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:03.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:03.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:10:03.646 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:10:03.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:03.696 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:03.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:10:03.797 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:10:03.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:03.854 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:03.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:10:03.955 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:10:03.984 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:03.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:03.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:10:04.081 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:03 smithi049 ceph-mon[40489]: pgmap v1082: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 1.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.11k op/s; 85/91617 objects misplaced (0.093%) 2024-09-21T14:10:04.081 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:03 smithi049 ceph-mon[30080]: pgmap v1082: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 1.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.11k op/s; 85/91617 objects misplaced (0.093%) 2024-09-21T14:10:04.086 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:10:04.117 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:04.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:04.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:10:04.161 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:10:04.161 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:10:04.184 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:10:04.184 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:10:04.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:03 smithi187 ceph-mon[33032]: pgmap v1082: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 1.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.11k op/s; 85/91617 objects misplaced (0.093%) 2024-09-21T14:10:04.219 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:10:04.227 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:10:04.227 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:10:04.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:04.262 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:04.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:10:04.266 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:10:04.266 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:10:04.299 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:10:04.300 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:10:04.333 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:10:04.334 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:10:04.366 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:10:04.368 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:10:04.369 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:10:04.396 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:04.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:04.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:10:04.406 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:10:04.406 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:10:04.497 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:10:04.531 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:04.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:04.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:10:04.633 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:10:04.667 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:04.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:04.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:10:04.769 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:10:04.788 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T14:10:04.788 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-21T14:10:04.788 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-21T14:10:04.788 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-21T14:10:04.788 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T14:10:04.802 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:04.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:04.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:10:04.904 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:10:04.942 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:04.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:04.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:10:05.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:04 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:10:05.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:04 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:10:05.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:04 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3528609118' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:10:05.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:04 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:10:05.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:04 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:10:05.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:04 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3528609118' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:10:05.044 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:10:05.086 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:05.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:05.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:10:05.126 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:04 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:10:05.126 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:04 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:10:05.126 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:04 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3528609118' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:10:05.187 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:10:05.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:05.231 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:05.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:10:05.332 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:10:05.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:05.362 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:05.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:10:05.463 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:10:05.487 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:05.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:05.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:10:05.580 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T14:10:05.589 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:10:05.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:05.634 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:05.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:10:05.735 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:10:05.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:05.773 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:05.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:10:05.874 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:10:05.898 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:10:05.900 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:05.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:05.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:10:05.951 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:10:06.002 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:10:06.042 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:06.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:06.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:10:06.079 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:05 smithi187 ceph-mon[33032]: pgmap v1083: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 1.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 1.18k op/s; 85/82155 objects misplaced (0.103%) 2024-09-21T14:10:06.079 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:05 smithi187 ceph-mon[33032]: osdmap e537: 8 total, 8 up, 8 in 2024-09-21T14:10:06.079 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:05 smithi187 ceph-mon[33032]: osdmap e538: 8 total, 8 up, 8 in 2024-09-21T14:10:06.080 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:05 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1617721240' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:10:06.080 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:05 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:10:06.095 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:05 smithi049 ceph-mon[30080]: pgmap v1083: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 1.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 1.18k op/s; 85/82155 objects misplaced (0.103%) 2024-09-21T14:10:06.095 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:05 smithi049 ceph-mon[30080]: osdmap e537: 8 total, 8 up, 8 in 2024-09-21T14:10:06.095 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:05 smithi049 ceph-mon[30080]: osdmap e538: 8 total, 8 up, 8 in 2024-09-21T14:10:06.095 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:05 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1617721240' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:10:06.095 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:05 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:10:06.095 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:05 smithi049 ceph-mon[40489]: pgmap v1083: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 1.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 1.18k op/s; 85/82155 objects misplaced (0.103%) 2024-09-21T14:10:06.096 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:05 smithi049 ceph-mon[40489]: osdmap e537: 8 total, 8 up, 8 in 2024-09-21T14:10:06.096 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:05 smithi049 ceph-mon[40489]: osdmap e538: 8 total, 8 up, 8 in 2024-09-21T14:10:06.096 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:05 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1617721240' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:10:06.096 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:05 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:10:06.143 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:10:06.176 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:06.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:06.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:10:06.278 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:10:06.283 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-09-21T14:10:06.306 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:06.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:06.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:10:06.408 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:10:06.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:06.447 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:06.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:10:06.539 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T14:10:06.539 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-21T14:10:06.539 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-21T14:10:06.539 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-21T14:10:06.539 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.539 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187592 9.5 9:a5cf25e0:::benchmark_data_smithi049_138504_object66184:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:10:06.539 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:41.268428+0000", 2024-09-21T14:10:06.539 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 505.26808480800003, 2024-09-21T14:10:06.539 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3375785200000001, 2024-09-21T14:10:06.539 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:06.540 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:06.540 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:06.540 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:10:06.540 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:10:06.540 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:10:06.540 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187592 2024-09-21T14:10:06.540 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.540 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:06.540 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.540 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:06.540 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268428+0000", 2024-09-21T14:10:06.540 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:06.540 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.540 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.540 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:06.541 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268428+0000", 2024-09-21T14:10:06.541 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:06.541 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.541 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.541 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:06.541 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268429+0000", 2024-09-21T14:10:06.541 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.466e-06 2024-09-21T14:10:06.541 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.541 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.541 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:06.541 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268435+0000", 2024-09-21T14:10:06.541 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5539999999999997e-06 2024-09-21T14:10:06.541 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.541 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.542 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:06.542 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268445+0000", 2024-09-21T14:10:06.542 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0366999999999999e-05 2024-09-21T14:10:06.542 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.542 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.542 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:06.542 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268452+0000", 2024-09-21T14:10:06.542 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8870000000000003e-06 2024-09-21T14:10:06.542 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.542 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.542 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:06.542 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.289931+0000", 2024-09-21T14:10:06.542 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.021479166000000001 2024-09-21T14:10:06.543 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.543 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.543 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:06.543 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.289937+0000", 2024-09-21T14:10:06.543 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4140000000000002e-06 2024-09-21T14:10:06.543 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.543 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.543 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:06.543 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.566606+0000", 2024-09-21T14:10:06.543 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2766695319999997 2024-09-21T14:10:06.543 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.543 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.543 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:10:06.543 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.569571+0000", 2024-09-21T14:10:06.544 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0029643489999999998 2024-09-21T14:10:06.544 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.544 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.544 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:10:06.544 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.573938+0000", 2024-09-21T14:10:06.544 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0043675099999999998 2024-09-21T14:10:06.544 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.544 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.544 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:10:06.544 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.589876+0000", 2024-09-21T14:10:06.544 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.015938249000000002 2024-09-21T14:10:06.544 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.544 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.545 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:06.545 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.597274+0000", 2024-09-21T14:10:06.545 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0073979509999999998 2024-09-21T14:10:06.545 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.545 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.545 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:06.545 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.598324+0000", 2024-09-21T14:10:06.545 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001050156 2024-09-21T14:10:06.545 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.545 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.545 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:10:06.545 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.599807+0000", 2024-09-21T14:10:06.545 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0014825439999999999 2024-09-21T14:10:06.545 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.545 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.546 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:06.546 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.606006+0000", 2024-09-21T14:10:06.546 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.006199375 2024-09-21T14:10:06.546 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:06.546 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:06.546 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:06.546 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.546 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.546 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187594 9.5 9:a4584f0e:::benchmark_data_smithi049_138504_object66186:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:10:06.546 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:41.268527+0000", 2024-09-21T14:10:06.546 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 505.26798529299998, 2024-09-21T14:10:06.546 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3614060480000001, 2024-09-21T14:10:06.546 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:06.546 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:06.547 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:06.547 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:10:06.547 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:10:06.547 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:10:06.547 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187594 2024-09-21T14:10:06.547 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.547 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:06.547 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.547 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:06.547 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268527+0000", 2024-09-21T14:10:06.547 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:06.547 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.547 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.547 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:06.547 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268527+0000", 2024-09-21T14:10:06.548 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:06.548 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.548 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.548 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:06.548 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268528+0000", 2024-09-21T14:10:06.548 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5299999999999999e-07 2024-09-21T14:10:06.548 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.548 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.548 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:06.548 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268531+0000", 2024-09-21T14:10:06.548 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.157e-06 2024-09-21T14:10:06.548 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.548 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.548 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:06.548 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268538+0000", 2024-09-21T14:10:06.548 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9290000000000001e-06 2024-09-21T14:10:06.549 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.549 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.549 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:06.549 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268543+0000", 2024-09-21T14:10:06.549 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0409999999999997e-06 2024-09-21T14:10:06.549 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.549 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.549 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:06.549 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.289953+0000", 2024-09-21T14:10:06.549 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.021409811000000001 2024-09-21T14:10:06.549 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.549 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.549 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:06.549 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.289957+0000", 2024-09-21T14:10:06.549 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9489999999999998e-06 2024-09-21T14:10:06.550 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.550 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.550 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:06.550 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.606351+0000", 2024-09-21T14:10:06.550 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3163947599999997 2024-09-21T14:10:06.550 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.550 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.550 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:10:06.550 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.610368+0000", 2024-09-21T14:10:06.550 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0040163289999999999 2024-09-21T14:10:06.550 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.550 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.550 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:10:06.550 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.615560+0000", 2024-09-21T14:10:06.551 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0051918939999999998 2024-09-21T14:10:06.551 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.551 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.551 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:06.551 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.622425+0000", 2024-09-21T14:10:06.551 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0068655959999999999 2024-09-21T14:10:06.551 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.551 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.551 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:06.551 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.623509+0000", 2024-09-21T14:10:06.551 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001083669 2024-09-21T14:10:06.551 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.551 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.551 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:10:06.552 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.624009+0000", 2024-09-21T14:10:06.552 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00050001600000000005 2024-09-21T14:10:06.552 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.552 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.552 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:10:06.552 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.625282+0000", 2024-09-21T14:10:06.552 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0012730599999999999 2024-09-21T14:10:06.552 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.552 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.552 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:10:06.553 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:06.553 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.629933+0000", 2024-09-21T14:10:06.553 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0046513839999999997 2024-09-21T14:10:06.553 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:06.554 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:06.554 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:06.554 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.554 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.554 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187607 9.7 9:ea996e28:::benchmark_data_smithi049_138504_object66199:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:10:06.554 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:41.286116+0000", 2024-09-21T14:10:06.554 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 505.25039664299999, 2024-09-21T14:10:06.554 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0827087349999998, 2024-09-21T14:10:06.554 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:06.554 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:06.554 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:06.554 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:10:06.555 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:10:06.555 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:10:06.555 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187607 2024-09-21T14:10:06.555 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.555 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:06.555 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.555 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:06.555 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.286116+0000", 2024-09-21T14:10:06.555 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:06.555 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.555 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.555 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:06.555 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.286116+0000", 2024-09-21T14:10:06.556 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:06.556 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.556 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.556 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:06.556 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.286117+0000", 2024-09-21T14:10:06.556 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.097e-06 2024-09-21T14:10:06.556 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.556 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.556 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:06.556 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.286120+0000", 2024-09-21T14:10:06.556 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.224e-06 2024-09-21T14:10:06.556 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.556 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.556 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:06.557 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.286129+0000", 2024-09-21T14:10:06.557 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3659999999999992e-06 2024-09-21T14:10:06.557 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.557 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.557 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:06.557 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.286134+0000", 2024-09-21T14:10:06.557 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6130000000000003e-06 2024-09-21T14:10:06.557 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.557 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.557 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:06.557 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.286627+0000", 2024-09-21T14:10:06.557 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00049266000000000002 2024-09-21T14:10:06.557 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.558 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.558 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:06.558 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.286663+0000", 2024-09-21T14:10:06.558 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6164999999999997e-05 2024-09-21T14:10:06.558 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.558 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.558 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:06.558 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.330333+0000", 2024-09-21T14:10:06.558 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0436700029999999 2024-09-21T14:10:06.558 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.558 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.558 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:10:06.558 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.333462+0000", 2024-09-21T14:10:06.559 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003128881 2024-09-21T14:10:06.559 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.559 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.559 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,7", 2024-09-21T14:10:06.559 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.337950+0000", 2024-09-21T14:10:06.559 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0044878089999999997 2024-09-21T14:10:06.559 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.559 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.559 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:10:06.559 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353871+0000", 2024-09-21T14:10:06.559 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.015921270000000001 2024-09-21T14:10:06.559 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.559 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.559 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:06.560 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.360705+0000", 2024-09-21T14:10:06.560 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0068337110000000001 2024-09-21T14:10:06.560 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.560 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.560 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:06.560 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.362017+0000", 2024-09-21T14:10:06.560 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0013119749999999999 2024-09-21T14:10:06.560 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.560 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.560 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:10:06.560 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.364166+0000", 2024-09-21T14:10:06.560 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0021495030000000001 2024-09-21T14:10:06.560 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.560 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.561 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:06.561 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.368825+0000", 2024-09-21T14:10:06.561 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0046584579999999999 2024-09-21T14:10:06.561 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:06.561 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:06.561 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:06.561 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.561 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.561 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187609 9.5 9:a0a81d79:::benchmark_data_smithi049_138504_object66201:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:10:06.561 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:41.287003+0000", 2024-09-21T14:10:06.561 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 505.24951003199999, 2024-09-21T14:10:06.561 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3857519199999997, 2024-09-21T14:10:06.561 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:06.562 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:06.562 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:06.562 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:10:06.562 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:10:06.562 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:10:06.562 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187609 2024-09-21T14:10:06.562 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.562 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:06.562 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.562 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:06.562 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.287003+0000", 2024-09-21T14:10:06.562 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:06.562 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.563 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.563 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:06.563 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.287003+0000", 2024-09-21T14:10:06.563 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:06.563 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.563 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.563 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:06.563 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.287003+0000", 2024-09-21T14:10:06.563 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8400000000000003e-07 2024-09-21T14:10:06.563 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.563 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.563 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:06.563 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.287017+0000", 2024-09-21T14:10:06.563 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3831e-05 2024-09-21T14:10:06.564 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.564 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.564 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:06.564 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.287026+0000", 2024-09-21T14:10:06.564 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8389999999999996e-06 2024-09-21T14:10:06.564 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.564 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.564 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:06.564 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.287032+0000", 2024-09-21T14:10:06.564 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3509999999999998e-06 2024-09-21T14:10:06.564 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.564 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.564 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:06.564 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.289969+0000", 2024-09-21T14:10:06.565 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0029364399999999998 2024-09-21T14:10:06.565 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.565 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.565 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:06.565 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.289973+0000", 2024-09-21T14:10:06.565 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6339999999999999e-06 2024-09-21T14:10:06.565 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.565 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.565 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:06.565 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.631003+0000", 2024-09-21T14:10:06.565 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3410304320000002 2024-09-21T14:10:06.565 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.565 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.566 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:10:06.566 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.634397+0000", 2024-09-21T14:10:06.566 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0033938369999999998 2024-09-21T14:10:06.566 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.566 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.566 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:10:06.566 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.639017+0000", 2024-09-21T14:10:06.566 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0046205889999999996 2024-09-21T14:10:06.566 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.566 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.566 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:10:06.566 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.662137+0000", 2024-09-21T14:10:06.566 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.023119475 2024-09-21T14:10:06.567 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.567 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.567 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:06.567 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.663533+0000", 2024-09-21T14:10:06.567 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0013956159999999999 2024-09-21T14:10:06.567 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.567 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.567 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:06.567 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.664730+0000", 2024-09-21T14:10:06.567 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0011970139999999999 2024-09-21T14:10:06.567 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.567 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.567 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:10:06.567 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.666523+0000", 2024-09-21T14:10:06.568 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001793046 2024-09-21T14:10:06.568 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.568 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.568 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:06.568 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.672754+0000", 2024-09-21T14:10:06.568 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0062319319999999999 2024-09-21T14:10:06.568 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:06.568 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:06.568 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:06.568 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.568 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.568 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187621 9.5 9:a3e366d7:::benchmark_data_smithi049_138504_object66213:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:10:06.568 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:41.300495+0000", 2024-09-21T14:10:06.568 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 505.23601738399998, 2024-09-21T14:10:06.568 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3801383449999998, 2024-09-21T14:10:06.569 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:06.569 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:06.569 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:06.569 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:10:06.569 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:10:06.569 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:10:06.569 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187621 2024-09-21T14:10:06.569 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.569 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:06.569 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.569 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:06.569 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300495+0000", 2024-09-21T14:10:06.569 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:06.569 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.569 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.570 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:06.570 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300495+0000", 2024-09-21T14:10:06.570 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:06.570 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.570 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.570 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:06.570 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300497+0000", 2024-09-21T14:10:06.570 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3710000000000001e-06 2024-09-21T14:10:06.570 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.570 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.570 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:06.570 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300502+0000", 2024-09-21T14:10:06.570 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3569999999999997e-06 2024-09-21T14:10:06.571 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.571 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.571 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:06.571 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300511+0000", 2024-09-21T14:10:06.571 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.9509999999999995e-06 2024-09-21T14:10:06.571 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.571 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.571 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:06.571 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300517+0000", 2024-09-21T14:10:06.571 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2489999999999998e-06 2024-09-21T14:10:06.571 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.571 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.571 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:06.571 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300606+0000", 2024-09-21T14:10:06.572 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8794000000000004e-05 2024-09-21T14:10:06.572 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.572 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.572 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:06.572 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300616+0000", 2024-09-21T14:10:06.572 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0338e-05 2024-09-21T14:10:06.572 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.572 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.572 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:06.572 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.643702+0000", 2024-09-21T14:10:06.572 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.343085759 2024-09-21T14:10:06.572 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.572 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.572 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:10:06.573 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.648030+0000", 2024-09-21T14:10:06.573 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0043275859999999996 2024-09-21T14:10:06.573 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.573 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.573 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:10:06.573 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.653318+0000", 2024-09-21T14:10:06.573 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.005287936 2024-09-21T14:10:06.573 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.573 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.573 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:10:06.573 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.662422+0000", 2024-09-21T14:10:06.573 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0091040340000000004 2024-09-21T14:10:06.573 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.573 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.573 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:06.574 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.673842+0000", 2024-09-21T14:10:06.574 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.011420382999999999 2024-09-21T14:10:06.574 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.574 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.574 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:06.574 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.675052+0000", 2024-09-21T14:10:06.574 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0012104290000000001 2024-09-21T14:10:06.574 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.574 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.574 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:10:06.574 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.676489+0000", 2024-09-21T14:10:06.574 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0014364309999999999 2024-09-21T14:10:06.574 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.574 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.574 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:06.575 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.680634+0000", 2024-09-21T14:10:06.575 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.004144727 2024-09-21T14:10:06.575 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:06.575 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:06.575 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:06.575 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.575 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.575 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187623 9.7 9:e79a683b:::benchmark_data_smithi049_138504_object66215:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:10:06.575 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:41.300818+0000", 2024-09-21T14:10:06.575 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 505.23569498000001, 2024-09-21T14:10:06.575 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1014428919999997, 2024-09-21T14:10:06.575 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:06.575 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:06.575 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:06.576 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:10:06.576 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:10:06.576 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:10:06.576 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187623 2024-09-21T14:10:06.576 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.576 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:06.576 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.576 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:06.576 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300818+0000", 2024-09-21T14:10:06.576 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:06.576 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.576 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.576 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:06.576 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300818+0000", 2024-09-21T14:10:06.577 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:06.577 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.577 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.577 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:06.577 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300818+0000", 2024-09-21T14:10:06.577 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.54e-07 2024-09-21T14:10:06.577 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.577 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.577 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:06.577 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300822+0000", 2024-09-21T14:10:06.577 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6770000000000001e-06 2024-09-21T14:10:06.577 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.577 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.577 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:06.578 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300832+0000", 2024-09-21T14:10:06.578 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0202e-05 2024-09-21T14:10:06.578 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.578 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.578 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:06.578 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300847+0000", 2024-09-21T14:10:06.578 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4992e-05 2024-09-21T14:10:06.578 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.578 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.578 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:06.578 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300931+0000", 2024-09-21T14:10:06.578 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3577000000000003e-05 2024-09-21T14:10:06.578 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.578 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.578 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:06.579 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300940+0000", 2024-09-21T14:10:06.579 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8009999999999993e-06 2024-09-21T14:10:06.579 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.579 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.579 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:06.579 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.370531+0000", 2024-09-21T14:10:06.579 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0695910870000001 2024-09-21T14:10:06.579 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.579 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.580 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:10:06.580 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.373842+0000", 2024-09-21T14:10:06.580 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0033110259999999999 2024-09-21T14:10:06.580 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.580 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.580 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,7", 2024-09-21T14:10:06.580 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.378337+0000", 2024-09-21T14:10:06.580 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0044951100000000001 2024-09-21T14:10:06.580 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.580 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.580 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:06.580 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.394514+0000", 2024-09-21T14:10:06.580 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.016176877999999999 2024-09-21T14:10:06.580 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.581 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.581 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:10:06.581 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.395030+0000", 2024-09-21T14:10:06.581 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00051615000000000005 2024-09-21T14:10:06.581 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.581 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.581 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:06.581 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.395803+0000", 2024-09-21T14:10:06.581 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00077358799999999997 2024-09-21T14:10:06.581 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.581 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.581 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:10:06.581 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.397305+0000", 2024-09-21T14:10:06.582 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001501811 2024-09-21T14:10:06.582 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.582 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.582 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:06.582 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.402261+0000", 2024-09-21T14:10:06.582 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0049552390000000002 2024-09-21T14:10:06.582 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:06.582 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:06.582 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:06.582 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.582 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.582 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187624 9.5 9:a181e4d0:::benchmark_data_smithi049_138504_object66216:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:10:06.582 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:41.300871+0000", 2024-09-21T14:10:06.582 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 505.235641977, 2024-09-21T14:10:06.583 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4164464030000001, 2024-09-21T14:10:06.583 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:06.583 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:06.583 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:06.583 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:10:06.583 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:10:06.583 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:10:06.583 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187624 2024-09-21T14:10:06.583 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.583 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:06.583 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.583 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:06.583 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300871+0000", 2024-09-21T14:10:06.584 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:06.584 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.584 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.584 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:06.584 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300871+0000", 2024-09-21T14:10:06.584 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:06.584 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.584 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.584 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:06.584 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300871+0000", 2024-09-21T14:10:06.584 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8599999999999998e-07 2024-09-21T14:10:06.584 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.584 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.584 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:06.585 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300875+0000", 2024-09-21T14:10:06.585 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7720000000000001e-06 2024-09-21T14:10:06.585 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.585 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.585 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:06.585 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300886+0000", 2024-09-21T14:10:06.585 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1058e-05 2024-09-21T14:10:06.585 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.585 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.585 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:06.585 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300892+0000", 2024-09-21T14:10:06.585 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.161e-06 2024-09-21T14:10:06.585 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.586 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.586 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:06.586 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.301287+0000", 2024-09-21T14:10:06.586 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00039522800000000002 2024-09-21T14:10:06.586 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.586 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.586 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:06.586 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.301297+0000", 2024-09-21T14:10:06.586 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6209999999999999e-06 2024-09-21T14:10:06.586 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.586 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.586 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:06.586 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.681021+0000", 2024-09-21T14:10:06.586 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3797241769999999 2024-09-21T14:10:06.587 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.587 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.587 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:10:06.587 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.685054+0000", 2024-09-21T14:10:06.587 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0040323909999999998 2024-09-21T14:10:06.587 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.587 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.587 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:10:06.587 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.689479+0000", 2024-09-21T14:10:06.587 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0044253740000000001 2024-09-21T14:10:06.587 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.587 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.587 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:06.588 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.710060+0000", 2024-09-21T14:10:06.588 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.020581051999999999 2024-09-21T14:10:06.588 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.588 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.588 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:10:06.588 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.710641+0000", 2024-09-21T14:10:06.588 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00058055800000000001 2024-09-21T14:10:06.588 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.588 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.588 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:06.588 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.711399+0000", 2024-09-21T14:10:06.588 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00075836299999999996 2024-09-21T14:10:06.588 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.588 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.588 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:10:06.589 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.712857+0000", 2024-09-21T14:10:06.589 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001458373 2024-09-21T14:10:06.589 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.589 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.589 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:06.589 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.717317+0000", 2024-09-21T14:10:06.589 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0044596890000000002 2024-09-21T14:10:06.589 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:06.589 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:06.589 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:06.589 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.589 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.589 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187626 9.7 9:e3df99cb:::benchmark_data_smithi049_138504_object66218:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:10:06.589 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:41.304965+0000", 2024-09-21T14:10:06.589 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 505.23154784100001, 2024-09-21T14:10:06.590 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1061633610000001, 2024-09-21T14:10:06.590 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:06.590 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:06.590 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:06.590 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:10:06.590 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:10:06.590 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:10:06.590 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187626 2024-09-21T14:10:06.590 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.590 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:06.590 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.590 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:06.590 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.304965+0000", 2024-09-21T14:10:06.591 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:06.591 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.591 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.591 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:06.591 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.304965+0000", 2024-09-21T14:10:06.591 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:06.591 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.591 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.591 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:06.591 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.304966+0000", 2024-09-21T14:10:06.591 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6619999999999999e-06 2024-09-21T14:10:06.591 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.591 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.591 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:06.592 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.304973+0000", 2024-09-21T14:10:06.592 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1759999999999998e-06 2024-09-21T14:10:06.592 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.592 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.592 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:06.592 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.304987+0000", 2024-09-21T14:10:06.592 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4372999999999999e-05 2024-09-21T14:10:06.592 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.592 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.592 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:06.592 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.304995+0000", 2024-09-21T14:10:06.592 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4109999999999996e-06 2024-09-21T14:10:06.592 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.592 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.593 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:06.593 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.305067+0000", 2024-09-21T14:10:06.593 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1833000000000002e-05 2024-09-21T14:10:06.593 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.593 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.593 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:06.593 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.305077+0000", 2024-09-21T14:10:06.593 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0161e-05 2024-09-21T14:10:06.593 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.593 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.593 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:06.593 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.382147+0000", 2024-09-21T14:10:06.593 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0770691709999998 2024-09-21T14:10:06.593 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.594 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.594 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:10:06.594 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.385209+0000", 2024-09-21T14:10:06.594 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0030624139999999998 2024-09-21T14:10:06.594 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.594 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.594 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,7", 2024-09-21T14:10:06.594 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.389597+0000", 2024-09-21T14:10:06.594 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.004388477 2024-09-21T14:10:06.595 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.595 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.595 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:06.595 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.403080+0000", 2024-09-21T14:10:06.595 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.013482984 2024-09-21T14:10:06.595 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.595 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.595 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:10:06.595 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.403583+0000", 2024-09-21T14:10:06.595 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00050300099999999999 2024-09-21T14:10:06.595 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.596 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.596 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:06.597 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.404464+0000", 2024-09-21T14:10:06.597 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00088034599999999995 2024-09-21T14:10:06.597 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.597 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.597 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:10:06.597 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.406114+0000", 2024-09-21T14:10:06.597 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0016504499999999999 2024-09-21T14:10:06.597 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.597 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.597 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:06.597 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.411128+0000", 2024-09-21T14:10:06.597 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0050139019999999998 2024-09-21T14:10:06.597 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:06.598 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:06.598 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:06.598 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.598 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.598 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187630 9.5 9:afb0e788:::benchmark_data_smithi049_138504_object66222:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:10:06.598 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:41.309934+0000", 2024-09-21T14:10:06.598 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 505.22657826, 2024-09-21T14:10:06.598 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4164949509999998, 2024-09-21T14:10:06.598 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:06.598 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:06.599 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:06.599 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:10:06.599 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:10:06.599 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:10:06.599 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187630 2024-09-21T14:10:06.599 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.599 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:06.599 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.599 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:06.599 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.309934+0000", 2024-09-21T14:10:06.599 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:06.599 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.599 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.599 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:06.600 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.309934+0000", 2024-09-21T14:10:06.600 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:06.600 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.600 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.600 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:06.600 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.309936+0000", 2024-09-21T14:10:06.600 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7460000000000001e-06 2024-09-21T14:10:06.600 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.600 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.600 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:06.600 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.309943+0000", 2024-09-21T14:10:06.600 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6579999999999996e-06 2024-09-21T14:10:06.600 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.600 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.601 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:06.601 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.309957+0000", 2024-09-21T14:10:06.601 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4496e-05 2024-09-21T14:10:06.601 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.601 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.601 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:06.601 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.309964+0000", 2024-09-21T14:10:06.601 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2219999999999996e-06 2024-09-21T14:10:06.601 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.601 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.601 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:06.601 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.310043+0000", 2024-09-21T14:10:06.601 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8801999999999995e-05 2024-09-21T14:10:06.601 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.602 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.602 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:06.602 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.310055+0000", 2024-09-21T14:10:06.602 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1637999999999999e-05 2024-09-21T14:10:06.602 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.602 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.602 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:06.602 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.693253+0000", 2024-09-21T14:10:06.602 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3831976189999997 2024-09-21T14:10:06.602 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.602 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.602 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:10:06.602 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.697465+0000", 2024-09-21T14:10:06.602 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0042119770000000004 2024-09-21T14:10:06.603 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.603 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.603 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:10:06.603 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.703657+0000", 2024-09-21T14:10:06.603 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0061928870000000002 2024-09-21T14:10:06.603 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.603 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.603 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:10:06.603 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.717616+0000", 2024-09-21T14:10:06.603 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.013958435999999999 2024-09-21T14:10:06.603 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.603 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.603 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:06.603 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.718555+0000", 2024-09-21T14:10:06.604 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00093888999999999995 2024-09-21T14:10:06.604 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.604 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.604 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:06.604 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.719633+0000", 2024-09-21T14:10:06.604 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0010787150000000001 2024-09-21T14:10:06.604 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.604 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.604 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:10:06.604 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.721631+0000", 2024-09-21T14:10:06.604 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0019973220000000002 2024-09-21T14:10:06.604 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.604 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.604 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:06.605 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.726429+0000", 2024-09-21T14:10:06.605 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0047985429999999997 2024-09-21T14:10:06.605 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:06.605 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:06.605 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:06.605 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.605 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.605 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187631 9.7 9:eb4378cd:::benchmark_data_smithi049_138504_object66223:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:10:06.605 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:41.310203+0000", 2024-09-21T14:10:06.605 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 505.22630975200002, 2024-09-21T14:10:06.605 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1434362220000001, 2024-09-21T14:10:06.605 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:06.605 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:06.605 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:06.606 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:10:06.606 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:10:06.606 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:10:06.606 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187631 2024-09-21T14:10:06.606 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.606 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:06.606 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.606 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:06.606 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.310203+0000", 2024-09-21T14:10:06.606 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:06.606 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.606 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.606 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:06.606 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.310203+0000", 2024-09-21T14:10:06.606 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:06.607 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.607 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.607 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:06.607 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.310204+0000", 2024-09-21T14:10:06.607 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3699999999999999e-07 2024-09-21T14:10:06.607 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.607 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.607 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:06.607 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.310209+0000", 2024-09-21T14:10:06.607 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7039999999999999e-06 2024-09-21T14:10:06.607 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.607 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.607 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:06.608 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.310219+0000", 2024-09-21T14:10:06.608 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7319999999999993e-06 2024-09-21T14:10:06.608 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.608 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.608 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:06.608 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.310226+0000", 2024-09-21T14:10:06.608 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4239999999999998e-06 2024-09-21T14:10:06.608 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.608 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.608 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:06.608 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.310279+0000", 2024-09-21T14:10:06.608 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3124999999999997e-05 2024-09-21T14:10:06.608 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.608 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.608 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:06.609 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.310288+0000", 2024-09-21T14:10:06.609 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3149999999999998e-06 2024-09-21T14:10:06.609 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.609 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.609 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:06.609 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.416040+0000", 2024-09-21T14:10:06.609 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1057518990000004 2024-09-21T14:10:06.609 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.609 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.609 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:10:06.609 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.419285+0000", 2024-09-21T14:10:06.609 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0032447399999999999 2024-09-21T14:10:06.609 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.609 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.610 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,7", 2024-09-21T14:10:06.610 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.423815+0000", 2024-09-21T14:10:06.610 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0045301509999999996 2024-09-21T14:10:06.610 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.610 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.610 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:06.610 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.439415+0000", 2024-09-21T14:10:06.610 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.015600334 2024-09-21T14:10:06.610 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.610 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.610 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:06.610 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.440566+0000", 2024-09-21T14:10:06.611 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0011505129999999999 2024-09-21T14:10:06.611 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.611 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.611 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:10:06.611 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.447676+0000", 2024-09-21T14:10:06.611 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0071103659999999999 2024-09-21T14:10:06.611 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.611 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.611 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:10:06.611 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.448927+0000", 2024-09-21T14:10:06.611 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0012508759999999999 2024-09-21T14:10:06.611 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.611 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.611 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:06.611 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.453639+0000", 2024-09-21T14:10:06.612 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0047121059999999998 2024-09-21T14:10:06.612 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:06.612 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:06.612 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:06.612 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.612 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.612 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187650 9.5 9:ab0f6d84:::benchmark_data_smithi049_138504_object66242:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:10:06.612 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:41.359224+0000", 2024-09-21T14:10:06.612 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 505.17728825099999, 2024-09-21T14:10:06.612 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4044763060000003, 2024-09-21T14:10:06.612 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:06.612 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:06.612 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:06.612 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:10:06.613 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:10:06.613 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:10:06.613 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187650 2024-09-21T14:10:06.613 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.613 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:06.613 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.613 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:06.613 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.359224+0000", 2024-09-21T14:10:06.613 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:06.613 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.613 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.613 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:06.613 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.359224+0000", 2024-09-21T14:10:06.613 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:06.614 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.614 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.614 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:06.614 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.359226+0000", 2024-09-21T14:10:06.614 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.423e-06 2024-09-21T14:10:06.614 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.614 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.614 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:06.614 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.359231+0000", 2024-09-21T14:10:06.614 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0980000000000001e-06 2024-09-21T14:10:06.614 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.614 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.614 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:06.614 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.359243+0000", 2024-09-21T14:10:06.615 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2316e-05 2024-09-21T14:10:06.615 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.615 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.615 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:06.615 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.359250+0000", 2024-09-21T14:10:06.615 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8820000000000003e-06 2024-09-21T14:10:06.615 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.615 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.615 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:06.615 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.359329+0000", 2024-09-21T14:10:06.615 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8541000000000006e-05 2024-09-21T14:10:06.615 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.615 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.615 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:06.616 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.359341+0000", 2024-09-21T14:10:06.616 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2238e-05 2024-09-21T14:10:06.616 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.616 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.616 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:06.616 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.726850+0000", 2024-09-21T14:10:06.616 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3675093619999998 2024-09-21T14:10:06.616 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.616 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.616 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:10:06.616 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.731153+0000", 2024-09-21T14:10:06.616 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0043027860000000003 2024-09-21T14:10:06.616 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.616 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.617 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:10:06.617 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.737448+0000", 2024-09-21T14:10:06.617 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0062951559999999997 2024-09-21T14:10:06.617 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.617 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.617 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:06.617 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.754776+0000", 2024-09-21T14:10:06.617 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.017328300000000001 2024-09-21T14:10:06.617 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.617 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.617 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:06.617 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.756004+0000", 2024-09-21T14:10:06.617 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001227337 2024-09-21T14:10:06.617 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.617 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.618 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:10:06.618 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.757990+0000", 2024-09-21T14:10:06.618 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0019857809999999998 2024-09-21T14:10:06.618 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.618 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.618 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:10:06.618 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.759321+0000", 2024-09-21T14:10:06.618 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001331203 2024-09-21T14:10:06.618 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.618 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.618 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:06.618 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.763701+0000", 2024-09-21T14:10:06.618 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0043798830000000002 2024-09-21T14:10:06.618 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:06.618 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:06.619 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:06.619 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.619 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.619 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187651 9.f 9:fed2c87a:::benchmark_data_smithi049_138504_object66243:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e347)", 2024-09-21T14:10:06.619 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:42.222057+0000", 2024-09-21T14:10:06.619 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 504.31445512200003, 2024-09-21T14:10:06.619 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1315866940000001, 2024-09-21T14:10:06.619 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:06.619 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:06.619 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:06.619 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:10:06.620 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:10:06.620 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:10:06.620 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187651 2024-09-21T14:10:06.620 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.620 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:06.620 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.620 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:06.620 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.222057+0000", 2024-09-21T14:10:06.620 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:06.620 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.621 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.621 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:06.621 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.222057+0000", 2024-09-21T14:10:06.621 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:06.621 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.621 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.621 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:06.621 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.222059+0000", 2024-09-21T14:10:06.621 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8029999999999999e-06 2024-09-21T14:10:06.621 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.621 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.621 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:06.621 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.222066+0000", 2024-09-21T14:10:06.621 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2670000000000003e-06 2024-09-21T14:10:06.622 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.622 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.622 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:06.622 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.222078+0000", 2024-09-21T14:10:06.622 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2899e-05 2024-09-21T14:10:06.622 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.622 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.622 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:06.622 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.222086+0000", 2024-09-21T14:10:06.622 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8080000000000005e-06 2024-09-21T14:10:06.622 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.622 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.622 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:06.622 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.233188+0000", 2024-09-21T14:10:06.623 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0111019990000001 2024-09-21T14:10:06.623 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.623 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.623 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:10:06.623 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.233317+0000", 2024-09-21T14:10:06.623 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012857999999999999 2024-09-21T14:10:06.623 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.623 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.623 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3", 2024-09-21T14:10:06.623 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.233386+0000", 2024-09-21T14:10:06.623 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8901999999999999e-05 2024-09-21T14:10:06.623 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.623 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.623 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:10:06.624 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.342687+0000", 2024-09-21T14:10:06.624 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.109301519 2024-09-21T14:10:06.624 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.624 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.624 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:06.624 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.342738+0000", 2024-09-21T14:10:06.624 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0756e-05 2024-09-21T14:10:06.624 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.624 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.624 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:06.624 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353571+0000", 2024-09-21T14:10:06.624 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.01083253 2024-09-21T14:10:06.624 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.625 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.625 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:10:06.625 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353614+0000", 2024-09-21T14:10:06.625 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3129e-05 2024-09-21T14:10:06.625 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.625 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.625 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:06.625 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353644+0000", 2024-09-21T14:10:06.625 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0502e-05 2024-09-21T14:10:06.625 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:06.625 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:06.625 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:06.626 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.626 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.626 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187654 9.5 9:a73ea609:::benchmark_data_smithi049_138504_object66246:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e347)", 2024-09-21T14:10:06.626 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:42.234021+0000", 2024-09-21T14:10:06.626 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 504.30249126699999, 2024-09-21T14:10:06.626 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5379708970000001, 2024-09-21T14:10:06.626 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:06.626 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:06.626 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:06.626 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:10:06.626 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:10:06.626 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:10:06.626 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187654 2024-09-21T14:10:06.626 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.627 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:06.627 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.627 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:06.627 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.234021+0000", 2024-09-21T14:10:06.627 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:06.627 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.627 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.627 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:06.627 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.234021+0000", 2024-09-21T14:10:06.627 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:06.627 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.627 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.627 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:06.628 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.234023+0000", 2024-09-21T14:10:06.628 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5460000000000001e-06 2024-09-21T14:10:06.628 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.628 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.628 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:06.628 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.234029+0000", 2024-09-21T14:10:06.628 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8069999999999996e-06 2024-09-21T14:10:06.628 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.628 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.628 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:06.628 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.234041+0000", 2024-09-21T14:10:06.628 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2189e-05 2024-09-21T14:10:06.628 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.628 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.629 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:06.629 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.234052+0000", 2024-09-21T14:10:06.629 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0713999999999999e-05 2024-09-21T14:10:06.629 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.629 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.629 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:06.629 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.743160+0000", 2024-09-21T14:10:06.629 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5091079629999999 2024-09-21T14:10:06.629 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.629 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.629 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:10:06.629 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.746100+0000", 2024-09-21T14:10:06.629 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0029400559999999999 2024-09-21T14:10:06.629 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.630 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.630 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:10:06.630 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.750393+0000", 2024-09-21T14:10:06.630 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0042938280000000004 2024-09-21T14:10:06.630 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.630 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.630 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:06.630 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.757289+0000", 2024-09-21T14:10:06.630 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0068951739999999996 2024-09-21T14:10:06.630 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.630 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.630 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:06.630 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.764759+0000", 2024-09-21T14:10:06.631 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0074705689999999998 2024-09-21T14:10:06.631 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.631 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.631 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:10:06.631 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.765446+0000", 2024-09-21T14:10:06.631 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00068706799999999999 2024-09-21T14:10:06.631 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.631 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.631 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:10:06.631 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.766945+0000", 2024-09-21T14:10:06.631 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0014990509999999999 2024-09-21T14:10:06.631 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.631 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.631 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:06.632 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.771992+0000", 2024-09-21T14:10:06.632 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0050469319999999996 2024-09-21T14:10:06.632 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:06.632 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:06.632 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:06.632 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.632 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.632 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187662 9.f 9:f2269e64:::benchmark_data_smithi049_138504_object66254:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e347)", 2024-09-21T14:10:06.632 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:42.244949+0000", 2024-09-21T14:10:06.632 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 504.291563478, 2024-09-21T14:10:06.632 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1087973170000001, 2024-09-21T14:10:06.632 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:06.632 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:06.632 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:06.633 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:10:06.633 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:10:06.633 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:10:06.633 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187662 2024-09-21T14:10:06.633 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.633 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:06.633 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.633 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:06.633 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.244949+0000", 2024-09-21T14:10:06.633 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:06.633 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.633 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.633 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:06.633 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.244949+0000", 2024-09-21T14:10:06.633 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:06.634 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.634 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.634 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:06.634 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.244951+0000", 2024-09-21T14:10:06.634 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.962e-06 2024-09-21T14:10:06.634 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.634 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.634 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:06.634 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.244957+0000", 2024-09-21T14:10:06.634 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8170000000000004e-06 2024-09-21T14:10:06.634 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.634 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.634 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:06.635 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.244972+0000", 2024-09-21T14:10:06.635 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5204999999999999e-05 2024-09-21T14:10:06.635 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.635 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.635 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:06.635 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.244981+0000", 2024-09-21T14:10:06.635 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5990000000000002e-06 2024-09-21T14:10:06.635 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.635 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.635 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:06.635 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.234563+0000", 2024-09-21T14:10:06.635 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.989582548 2024-09-21T14:10:06.635 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.635 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.635 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:10:06.636 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.234696+0000", 2024-09-21T14:10:06.636 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013280200000000001 2024-09-21T14:10:06.636 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.636 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.636 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3", 2024-09-21T14:10:06.636 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.234783+0000", 2024-09-21T14:10:06.636 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.6990999999999994e-05 2024-09-21T14:10:06.636 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.636 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.636 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:10:06.636 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.342749+0000", 2024-09-21T14:10:06.636 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1079658480000001 2024-09-21T14:10:06.636 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.636 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.637 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:06.637 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353673+0000", 2024-09-21T14:10:06.637 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.010924473000000001 2024-09-21T14:10:06.637 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.637 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.637 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:06.637 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353696+0000", 2024-09-21T14:10:06.637 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2792000000000001e-05 2024-09-21T14:10:06.637 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.637 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.637 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:10:06.637 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353726+0000", 2024-09-21T14:10:06.637 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9509000000000002e-05 2024-09-21T14:10:06.637 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.637 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.638 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:06.638 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353746+0000", 2024-09-21T14:10:06.638 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0771000000000002e-05 2024-09-21T14:10:06.638 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:06.638 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:06.638 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:06.638 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.638 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.638 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187665 9.f 9:fde6db3c:::benchmark_data_smithi049_138504_object66257:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e347)", 2024-09-21T14:10:06.638 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:42.254849+0000", 2024-09-21T14:10:06.638 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 504.281663616, 2024-09-21T14:10:06.638 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.186045939, 2024-09-21T14:10:06.638 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:06.639 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:06.639 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:06.639 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:10:06.639 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:10:06.639 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:10:06.639 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187665 2024-09-21T14:10:06.639 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.639 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:06.639 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.639 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:06.639 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.254849+0000", 2024-09-21T14:10:06.639 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:06.639 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.639 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.639 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:06.640 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.254849+0000", 2024-09-21T14:10:06.640 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:06.640 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.640 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.640 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:06.640 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.254851+0000", 2024-09-21T14:10:06.640 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9209999999999999e-06 2024-09-21T14:10:06.640 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.640 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.640 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:06.640 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.254857+0000", 2024-09-21T14:10:06.640 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3659999999999997e-06 2024-09-21T14:10:06.640 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.640 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.641 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:06.641 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.254870+0000", 2024-09-21T14:10:06.641 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2892e-05 2024-09-21T14:10:06.641 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.641 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.641 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:06.641 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.254882+0000", 2024-09-21T14:10:06.641 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1647999999999999e-05 2024-09-21T14:10:06.641 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.641 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.641 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:06.641 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.415623+0000", 2024-09-21T14:10:06.641 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.160741582 2024-09-21T14:10:06.641 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.642 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.642 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:10:06.642 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.415771+0000", 2024-09-21T14:10:06.642 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014724000000000001 2024-09-21T14:10:06.642 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.642 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.642 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3", 2024-09-21T14:10:06.642 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.415898+0000", 2024-09-21T14:10:06.642 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012708399999999999 2024-09-21T14:10:06.642 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.642 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.642 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:06.642 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.427284+0000", 2024-09-21T14:10:06.642 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.011386624 2024-09-21T14:10:06.643 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.643 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.643 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:06.643 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.427305+0000", 2024-09-21T14:10:06.643 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0956000000000001e-05 2024-09-21T14:10:06.643 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.643 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.643 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:10:06.643 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.440820+0000", 2024-09-21T14:10:06.643 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.013514511999999999 2024-09-21T14:10:06.643 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.643 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.643 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:10:06.643 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.440860+0000", 2024-09-21T14:10:06.644 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0448000000000003e-05 2024-09-21T14:10:06.644 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.644 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.644 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:06.644 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.440895+0000", 2024-09-21T14:10:06.644 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4666000000000003e-05 2024-09-21T14:10:06.644 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:06.644 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:06.644 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:06.644 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.644 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.644 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25421.0:187664 9.d e347/345 9:bfa1ff03:::benchmark_data_smithi049_138504_object66256:head v 347'11740, mlcod=346'11738)", 2024-09-21T14:10:06.644 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:42.255288+0000", 2024-09-21T14:10:06.644 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 504.281224737, 2024-09-21T14:10:06.645 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2055402320000002, 2024-09-21T14:10:06.645 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:06.645 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:06.645 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:06.645 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:06.645 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.645 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:06.645 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.255288+0000", 2024-09-21T14:10:06.645 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:06.645 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.645 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.645 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:06.645 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.255288+0000", 2024-09-21T14:10:06.645 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:06.645 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.646 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.646 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:06.646 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.255289+0000", 2024-09-21T14:10:06.646 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2200000000000003e-07 2024-09-21T14:10:06.646 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.646 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.646 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:06.646 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.255296+0000", 2024-09-21T14:10:06.646 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3440000000000003e-06 2024-09-21T14:10:06.646 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.646 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.646 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:06.646 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.255312+0000", 2024-09-21T14:10:06.646 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5968999999999999e-05 2024-09-21T14:10:06.647 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.647 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.647 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:06.647 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.255319+0000", 2024-09-21T14:10:06.647 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9199999999999998e-06 2024-09-21T14:10:06.647 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.647 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.647 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:06.647 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.459355+0000", 2024-09-21T14:10:06.647 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2040357249999998 2024-09-21T14:10:06.647 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.647 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.647 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:10:06.647 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.459378+0000", 2024-09-21T14:10:06.648 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3757e-05 2024-09-21T14:10:06.648 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.648 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.648 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:10:06.648 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.460796+0000", 2024-09-21T14:10:06.648 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001417721 2024-09-21T14:10:06.648 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.648 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.648 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:06.648 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.460828+0000", 2024-09-21T14:10:06.648 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1974000000000003e-05 2024-09-21T14:10:06.648 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:06.648 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:06.649 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:06.649 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.649 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.649 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24365.0:2978 5.15 5:a93a5511:::notify.2:head [watch ping cookie 94025139204608 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:10:06.649 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:43.276775+0000", 2024-09-21T14:10:06.649 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 503.25973721499997, 2024-09-21T14:10:06.649 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4885784160000002, 2024-09-21T14:10:06.649 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:06.649 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:06.649 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T14:10:06.649 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:10:06.649 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24365", 2024-09-21T14:10:06.649 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3404764347", 2024-09-21T14:10:06.649 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 2978 2024-09-21T14:10:06.650 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.650 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:06.650 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.650 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:06.650 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:43.276775+0000", 2024-09-21T14:10:06.650 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:06.650 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.650 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.650 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:06.650 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:43.276775+0000", 2024-09-21T14:10:06.650 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:06.650 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.650 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.650 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:06.651 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:43.276777+0000", 2024-09-21T14:10:06.651 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9630000000000001e-06 2024-09-21T14:10:06.651 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.651 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.651 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:06.651 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:43.276784+0000", 2024-09-21T14:10:06.651 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1219999999999997e-06 2024-09-21T14:10:06.651 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.651 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.651 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:06.651 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:43.276802+0000", 2024-09-21T14:10:06.651 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7096999999999999e-05 2024-09-21T14:10:06.651 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.651 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.651 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:06.652 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:43.276811+0000", 2024-09-21T14:10:06.652 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7780000000000002e-06 2024-09-21T14:10:06.652 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.652 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.653 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:06.653 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.765168+0000", 2024-09-21T14:10:06.653 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4883561849999998 2024-09-21T14:10:06.653 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.653 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.653 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:10:06.653 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.765252+0000", 2024-09-21T14:10:06.653 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4201000000000002e-05 2024-09-21T14:10:06.654 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.654 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.654 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:06.654 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.765354+0000", 2024-09-21T14:10:06.654 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010207 2024-09-21T14:10:06.654 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:06.654 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:06.654 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:06.654 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.654 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.654 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.25421.0:187651 9.f e349/345 ondisk, result = 0)", 2024-09-21T14:10:06.654 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:44.238795+0000", 2024-09-21T14:10:06.654 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 502.297717928, 2024-09-21T14:10:06.655 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.115079242, 2024-09-21T14:10:06.655 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:06.655 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:06.655 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T14:10:06.655 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:06.655 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.655 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:06.655 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238795+0000", 2024-09-21T14:10:06.655 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:06.655 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.655 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.655 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:06.655 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238795+0000", 2024-09-21T14:10:06.655 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:06.656 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.656 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.656 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:06.656 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238795+0000", 2024-09-21T14:10:06.656 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9800000000000003e-07 2024-09-21T14:10:06.656 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.656 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.656 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:06.656 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238799+0000", 2024-09-21T14:10:06.656 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9920000000000001e-06 2024-09-21T14:10:06.656 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.656 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.656 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:06.656 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238809+0000", 2024-09-21T14:10:06.656 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6120000000000005e-06 2024-09-21T14:10:06.657 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.657 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.657 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:06.657 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238816+0000", 2024-09-21T14:10:06.657 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6229999999999999e-06 2024-09-21T14:10:06.657 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.657 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.657 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:06.657 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.342725+0000", 2024-09-21T14:10:06.657 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1039093099999999 2024-09-21T14:10:06.657 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.657 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.657 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:10:06.658 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.342733+0000", 2024-09-21T14:10:06.658 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9189999999999999e-06 2024-09-21T14:10:06.658 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.658 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.658 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:06.658 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353874+0000", 2024-09-21T14:10:06.658 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.011141088 2024-09-21T14:10:06.658 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:06.658 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:06.658 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:06.658 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.658 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.658 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.25421.0:187651 9.f e349/345 ondisk, result = 0)", 2024-09-21T14:10:06.659 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:44.238813+0000", 2024-09-21T14:10:06.659 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 502.29769958600002, 2024-09-21T14:10:06.659 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1148360479999999, 2024-09-21T14:10:06.659 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:06.659 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:06.659 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T14:10:06.659 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:06.659 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.659 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:06.659 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238813+0000", 2024-09-21T14:10:06.659 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:06.659 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.659 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.659 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:06.660 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238813+0000", 2024-09-21T14:10:06.660 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:06.660 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.660 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.660 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:06.660 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238814+0000", 2024-09-21T14:10:06.660 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4700000000000001e-07 2024-09-21T14:10:06.660 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.660 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.660 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:06.660 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238817+0000", 2024-09-21T14:10:06.660 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3469999999999999e-06 2024-09-21T14:10:06.660 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.660 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.660 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:06.661 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238827+0000", 2024-09-21T14:10:06.661 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0011e-05 2024-09-21T14:10:06.661 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.661 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.661 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:06.661 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238832+0000", 2024-09-21T14:10:06.661 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1139999999999997e-06 2024-09-21T14:10:06.661 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.661 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.661 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:06.661 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353559+0000", 2024-09-21T14:10:06.661 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1147265909999999 2024-09-21T14:10:06.661 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.661 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.662 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:10:06.662 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353566+0000", 2024-09-21T14:10:06.662 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8669999999999994e-06 2024-09-21T14:10:06.662 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.662 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.662 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:06.662 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353649+0000", 2024-09-21T14:10:06.662 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2571000000000006e-05 2024-09-21T14:10:06.662 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:06.662 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:06.662 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:06.662 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.662 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.662 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187668 9.5 9:acffd95c:::benchmark_data_smithi049_138504_object66260:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e349)", 2024-09-21T14:10:06.663 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:44.468347+0000", 2024-09-21T14:10:06.663 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 502.06816608899999, 2024-09-21T14:10:06.663 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3419073799999999, 2024-09-21T14:10:06.663 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:06.663 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:06.663 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:06.663 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:10:06.663 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:10:06.663 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:10:06.663 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187668 2024-09-21T14:10:06.663 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.663 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:06.663 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.663 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:06.664 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.468347+0000", 2024-09-21T14:10:06.664 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:06.664 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.664 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.664 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:06.664 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.468347+0000", 2024-09-21T14:10:06.664 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:06.664 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.664 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.664 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:06.664 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.468347+0000", 2024-09-21T14:10:06.664 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2399999999999997e-07 2024-09-21T14:10:06.664 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.665 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.665 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:06.665 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.468351+0000", 2024-09-21T14:10:06.665 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6049999999999998e-06 2024-09-21T14:10:06.665 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.665 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.665 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:06.665 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.468358+0000", 2024-09-21T14:10:06.665 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4510000000000003e-06 2024-09-21T14:10:06.665 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.665 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.665 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:06.665 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.468366+0000", 2024-09-21T14:10:06.665 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6140000000000003e-06 2024-09-21T14:10:06.665 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.666 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.666 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:06.666 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.772366+0000", 2024-09-21T14:10:06.666 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3040005530000001 2024-09-21T14:10:06.666 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.666 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.666 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:10:06.666 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.775717+0000", 2024-09-21T14:10:06.666 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0033501080000000001 2024-09-21T14:10:06.666 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.666 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.666 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:10:06.666 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.781090+0000", 2024-09-21T14:10:06.666 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0053731990000000004 2024-09-21T14:10:06.667 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.667 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.667 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:06.667 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.798846+0000", 2024-09-21T14:10:06.667 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.017756042999999999 2024-09-21T14:10:06.667 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.667 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.667 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:06.667 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.800017+0000", 2024-09-21T14:10:06.667 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0011714430000000001 2024-09-21T14:10:06.667 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.667 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.667 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:10:06.668 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.800552+0000", 2024-09-21T14:10:06.668 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00053442500000000005 2024-09-21T14:10:06.668 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.668 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.668 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:10:06.668 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.801930+0000", 2024-09-21T14:10:06.668 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0013783109999999999 2024-09-21T14:10:06.668 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:06.668 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:06.668 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:06.668 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.810254+0000", 2024-09-21T14:10:06.668 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.008323904 2024-09-21T14:10:06.668 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:06.668 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:06.669 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:06.669 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:06.669 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:06.669 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T14:10:06.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:06.670 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:06.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:10:06.771 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:10:06.810 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:06.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:06.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:10:06.912 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:10:06.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:06 smithi187 ceph-mon[33032]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:10:06.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:06 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T14:10:06.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:06 smithi187 ceph-mon[33032]: osdmap e539: 8 total, 8 up, 8 in 2024-09-21T14:10:06.955 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T14:10:06.957 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-09-21T14:10:06.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:06.960 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:06.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:10:07.061 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:10:07.097 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:07.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:07.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:10:07.198 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:10:07.234 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:07.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:07.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:10:07.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:06 smithi049 ceph-mon[40489]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:10:07.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:06 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T14:10:07.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:06 smithi049 ceph-mon[40489]: osdmap e539: 8 total, 8 up, 8 in 2024-09-21T14:10:07.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:06 smithi049 ceph-mon[30080]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:10:07.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:06 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T14:10:07.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:06 smithi049 ceph-mon[30080]: osdmap e539: 8 total, 8 up, 8 in 2024-09-21T14:10:07.336 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:10:07.365 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:07.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:07.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:10:07.467 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:10:07.500 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:07.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:07.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:10:07.602 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:10:07.644 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:07.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:07.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:10:07.746 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T14:10:07.747 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:10:07.797 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:07.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:07.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:10:07.899 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:10:07.923 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:07.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:07.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:10:07.927 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T14:10:07.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:07 smithi049 ceph-mon[30080]: pgmap v1086: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 1.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.25k op/s; 63/76758 objects misplaced (0.082%) 2024-09-21T14:10:07.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:07 smithi049 ceph-mon[30080]: 4.d scrub starts 2024-09-21T14:10:07.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:07 smithi049 ceph-mon[30080]: 4.d scrub ok 2024-09-21T14:10:07.968 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:07 smithi049 ceph-mon[30080]: from='client.26648 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:10:07.968 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:07 smithi049 ceph-mon[40489]: pgmap v1086: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 1.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.25k op/s; 63/76758 objects misplaced (0.082%) 2024-09-21T14:10:07.969 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:07 smithi049 ceph-mon[40489]: 4.d scrub starts 2024-09-21T14:10:07.969 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:07 smithi049 ceph-mon[40489]: 4.d scrub ok 2024-09-21T14:10:07.969 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:07 smithi049 ceph-mon[40489]: from='client.26648 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:10:08.025 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:10:08.068 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":1087,"stamp":"2024-09-21T14:10:06.277634+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1664155479,"num_objects":25586,"num_object_clones":0,"num_object_copies":76758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":63,"num_objects_unfound":0,"num_objects_dirty":25586,"num_whiteouts":0,"num_read":1739,"num_read_kb":1506,"num_write":222390,"num_write_kb":7896332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49443,"num_bytes_recovered":3221705652,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85048,"ondisk_log_size":85048,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":512,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":12385304,"kb_used_data":6732428,"kb_used_omap":23,"kb_used_meta":5652392,"kb_avail":737575912,"statfs":{"total":767960285184,"available":755277733888,"internally_reserved":0,"allocated":6894006272,"data_stored":6890576378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24148,"internal_metadata":5788049836},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-489553920,"num_objects":-7470,"num_object_clones":0,"num_object_copies":-22410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-46,"num_objects_unfound":0,"num_objects_dirty":-7470,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":7470,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.016327"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":899,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.275996+0000","last_change":"2024-09-21T14:09:53.481191+0000","last_active":"2024-09-21T14:10:05.275996+0000","last_peered":"2024-09-21T14:10:05.275996+0000","last_clean":"2024-09-21T14:10:05.275996+0000","last_became_active":"2024-09-21T14:09:53.480810+0000","last_became_peered":"2024-09-21T14:09:53.480810+0000","last_unstale":"2024-09-21T14:10:05.275996+0000","last_undegraded":"2024-09-21T14:10:05.275996+0000","last_fullsized":"2024-09-21T14:10:05.275996+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:10.397765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:05:10.397765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:57.228680+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"516'21","reported_seq":1091,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.275505+0000","last_change":"2024-09-21T14:09:00.018273+0000","last_active":"2024-09-21T14:10:00.275505+0000","last_peered":"2024-09-21T14:10:00.275505+0000","last_clean":"2024-09-21T14:10:00.275505+0000","last_became_active":"2024-09-21T13:58:17.771220+0000","last_became_peered":"2024-09-21T13:58:17.771220+0000","last_unstale":"2024-09-21T14:10:00.275505+0000","last_undegraded":"2024-09-21T14:10:00.275505+0000","last_fullsized":"2024-09-21T14:10:00.275505+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'15","last_scrub_stamp":"2024-09-21T14:09:00.018221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:09:00.018221+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:15.320984+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1027,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.285453+0000","last_change":"2024-09-21T14:08:09.021013+0000","last_active":"2024-09-21T14:10:05.285453+0000","last_peered":"2024-09-21T14:10:05.285453+0000","last_clean":"2024-09-21T14:10:05.285453+0000","last_became_active":"2024-09-21T14:01:41.202168+0000","last_became_peered":"2024-09-21T14:01:41.202168+0000","last_unstale":"2024-09-21T14:10:05.285453+0000","last_undegraded":"2024-09-21T14:10:05.285453+0000","last_fullsized":"2024-09-21T14:10:05.285453+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:08:09.020943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:08:09.020943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":999,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.276764+0000","last_change":"2024-09-21T14:07:45.174462+0000","last_active":"2024-09-21T14:10:00.276764+0000","last_peered":"2024-09-21T14:10:00.276764+0000","last_clean":"2024-09-21T14:10:00.276764+0000","last_became_active":"2024-09-21T14:01:41.203275+0000","last_became_peered":"2024-09-21T14:01:41.203275+0000","last_unstale":"2024-09-21T14:10:00.276764+0000","last_undegraded":"2024-09-21T14:10:00.276764+0000","last_fullsized":"2024-09-21T14:10:00.276764+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:07:45.174406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:07:45.174406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":563,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.268830+0000","last_change":"2024-09-21T14:09:49.798192+0000","last_active":"2024-09-21T14:10:00.268830+0000","last_peered":"2024-09-21T14:10:00.268830+0000","last_clean":"2024-09-21T14:10:00.268830+0000","last_became_active":"2024-09-21T14:08:14.411266+0000","last_became_peered":"2024-09-21T14:08:14.411266+0000","last_unstale":"2024-09-21T14:10:00.268830+0000","last_undegraded":"2024-09-21T14:10:00.268830+0000","last_fullsized":"2024-09-21T14:10:00.268830+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:09:49.798143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:09:49.798143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:11:15.070203+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":509,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.268694+0000","last_change":"2024-09-21T14:08:14.411332+0000","last_active":"2024-09-21T14:10:00.268694+0000","last_peered":"2024-09-21T14:10:00.268694+0000","last_clean":"2024-09-21T14:10:00.268694+0000","last_became_active":"2024-09-21T14:08:14.411010+0000","last_became_peered":"2024-09-21T14:08:14.411010+0000","last_unstale":"2024-09-21T14:10:00.268694+0000","last_undegraded":"2024-09-21T14:10:00.268694+0000","last_fullsized":"2024-09-21T14:10:00.268694+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:07:33.294212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:07:33.294212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"516'39","reported_seq":573,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.275433+0000","last_change":"2024-09-21T14:09:31.425375+0000","last_active":"2024-09-21T14:10:00.275433+0000","last_peered":"2024-09-21T14:10:00.275433+0000","last_clean":"2024-09-21T14:10:00.275433+0000","last_became_active":"2024-09-21T14:08:14.415832+0000","last_became_peered":"2024-09-21T14:08:14.415832+0000","last_unstale":"2024-09-21T14:10:00.275433+0000","last_undegraded":"2024-09-21T14:10:00.275433+0000","last_fullsized":"2024-09-21T14:10:00.275433+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'39","last_scrub_stamp":"2024-09-21T14:09:31.425147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:09:31.425147+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:45.105512+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":70,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.275389+0000","last_change":"2024-09-21T14:09:36.429717+0000","last_active":"2024-09-21T14:10:00.275389+0000","last_peered":"2024-09-21T14:10:00.275389+0000","last_clean":"2024-09-21T14:10:00.275389+0000","last_became_active":"2024-09-21T14:09:00.253451+0000","last_became_peered":"2024-09-21T14:09:00.253451+0000","last_unstale":"2024-09-21T14:10:00.275389+0000","last_undegraded":"2024-09-21T14:10:00.275389+0000","last_fullsized":"2024-09-21T14:10:00.275389+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:09:36.429653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:09:36.429653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:44.222652+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":985,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.275539+0000","last_change":"2024-09-21T14:08:51.319776+0000","last_active":"2024-09-21T14:10:05.275539+0000","last_peered":"2024-09-21T14:10:05.275539+0000","last_clean":"2024-09-21T14:10:05.275539+0000","last_became_active":"2024-09-21T14:01:41.194517+0000","last_became_peered":"2024-09-21T14:01:41.194517+0000","last_unstale":"2024-09-21T14:10:05.275539+0000","last_undegraded":"2024-09-21T14:10:05.275539+0000","last_fullsized":"2024-09-21T14:10:05.275539+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:08:51.319641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:08:51.319641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:20.835282+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"516'21","reported_seq":81,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.276519+0000","last_change":"2024-09-21T14:09:00.268550+0000","last_active":"2024-09-21T14:10:00.276519+0000","last_peered":"2024-09-21T14:10:00.276519+0000","last_clean":"2024-09-21T14:10:00.276519+0000","last_became_active":"2024-09-21T14:09:00.251527+0000","last_became_peered":"2024-09-21T14:09:00.251527+0000","last_unstale":"2024-09-21T14:10:00.276519+0000","last_undegraded":"2024-09-21T14:10:00.276519+0000","last_fullsized":"2024-09-21T14:10:00.276519+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'15","last_scrub_stamp":"2024-09-21T14:08:03.123210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:08:03.123210+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":934,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.274723+0000","last_change":"2024-09-21T14:07:42.488397+0000","last_active":"2024-09-21T14:10:05.274723+0000","last_peered":"2024-09-21T14:10:05.274723+0000","last_clean":"2024-09-21T14:10:05.274723+0000","last_became_active":"2024-09-21T14:01:41.188248+0000","last_became_peered":"2024-09-21T14:01:41.188248+0000","last_unstale":"2024-09-21T14:10:05.274723+0000","last_undegraded":"2024-09-21T14:10:05.274723+0000","last_fullsized":"2024-09-21T14:10:05.274723+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:07:42.488329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:07:42.488329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:23.280991+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":945,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.287323+0000","last_change":"2024-09-21T14:09:53.479302+0000","last_active":"2024-09-21T14:10:05.287323+0000","last_peered":"2024-09-21T14:10:05.287323+0000","last_clean":"2024-09-21T14:10:05.287323+0000","last_became_active":"2024-09-21T14:09:53.478710+0000","last_became_peered":"2024-09-21T14:09:53.478710+0000","last_unstale":"2024-09-21T14:10:05.287323+0000","last_undegraded":"2024-09-21T14:10:05.287323+0000","last_fullsized":"2024-09-21T14:10:05.287323+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:50.577560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:06:50.577560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":969,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.275098+0000","last_change":"2024-09-21T14:09:34.325823+0000","last_active":"2024-09-21T14:10:05.275098+0000","last_peered":"2024-09-21T14:10:05.275098+0000","last_clean":"2024-09-21T14:10:05.275098+0000","last_became_active":"2024-09-21T14:09:31.257075+0000","last_became_peered":"2024-09-21T14:09:31.257075+0000","last_unstale":"2024-09-21T14:10:05.275098+0000","last_undegraded":"2024-09-21T14:10:05.275098+0000","last_fullsized":"2024-09-21T14:10:05.275098+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:09:34.325751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:09:34.325751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:48.617481+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"516'9","reported_seq":79,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.276380+0000","last_change":"2024-09-21T14:09:41.805340+0000","last_active":"2024-09-21T14:10:00.276380+0000","last_peered":"2024-09-21T14:10:00.276380+0000","last_clean":"2024-09-21T14:10:00.276380+0000","last_became_active":"2024-09-21T14:09:00.253147+0000","last_became_peered":"2024-09-21T14:09:00.253147+0000","last_unstale":"2024-09-21T14:10:00.276380+0000","last_undegraded":"2024-09-21T14:10:00.276380+0000","last_fullsized":"2024-09-21T14:10:00.276380+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'9","last_scrub_stamp":"2024-09-21T14:09:41.805273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:09:41.805273+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:11:09.444395+0000","scrub_duration":291,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":549,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.276939+0000","last_change":"2024-09-21T14:08:17.147423+0000","last_active":"2024-09-21T14:10:00.276939+0000","last_peered":"2024-09-21T14:10:00.276939+0000","last_clean":"2024-09-21T14:10:00.276939+0000","last_became_active":"2024-09-21T14:08:04.295165+0000","last_became_peered":"2024-09-21T14:08:04.295165+0000","last_unstale":"2024-09-21T14:10:00.276939+0000","last_undegraded":"2024-09-21T14:10:00.276939+0000","last_fullsized":"2024-09-21T14:10:00.276939+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:08:17.147364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:08:17.147364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":995,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.274090+0000","last_change":"2024-09-21T14:08:14.419726+0000","last_active":"2024-09-21T14:10:05.274090+0000","last_peered":"2024-09-21T14:10:05.274090+0000","last_clean":"2024-09-21T14:10:05.274090+0000","last_became_active":"2024-09-21T14:08:14.419373+0000","last_became_peered":"2024-09-21T14:08:14.419373+0000","last_unstale":"2024-09-21T14:10:05.274090+0000","last_undegraded":"2024-09-21T14:10:05.274090+0000","last_fullsized":"2024-09-21T14:10:05.274090+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:07:04.491870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:07:04.491870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:14.395117+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1042,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.279086+0000","last_change":"2024-09-21T14:07:06.241357+0000","last_active":"2024-09-21T14:10:00.279086+0000","last_peered":"2024-09-21T14:10:00.279086+0000","last_clean":"2024-09-21T14:10:00.279086+0000","last_became_active":"2024-09-21T14:01:41.200360+0000","last_became_peered":"2024-09-21T14:01:41.200360+0000","last_unstale":"2024-09-21T14:10:00.279086+0000","last_undegraded":"2024-09-21T14:10:00.279086+0000","last_fullsized":"2024-09-21T14:10:00.279086+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:07:06.241295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:07:06.241295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":947,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.276608+0000","last_change":"2024-09-21T14:09:53.477205+0000","last_active":"2024-09-21T14:10:00.276608+0000","last_peered":"2024-09-21T14:10:00.276608+0000","last_clean":"2024-09-21T14:10:00.276608+0000","last_became_active":"2024-09-21T14:09:53.476467+0000","last_became_peered":"2024-09-21T14:09:53.476467+0000","last_unstale":"2024-09-21T14:10:00.276608+0000","last_undegraded":"2024-09-21T14:10:00.276608+0000","last_fullsized":"2024-09-21T14:10:00.276608+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:08:18.118531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:08:18.118531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"516'39","reported_seq":1127,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.277868+0000","last_change":"2024-09-21T14:09:47.338934+0000","last_active":"2024-09-21T14:10:00.277868+0000","last_peered":"2024-09-21T14:10:00.277868+0000","last_clean":"2024-09-21T14:10:00.277868+0000","last_became_active":"2024-09-21T14:09:31.260235+0000","last_became_peered":"2024-09-21T14:09:31.260235+0000","last_unstale":"2024-09-21T14:10:00.277868+0000","last_undegraded":"2024-09-21T14:10:00.277868+0000","last_fullsized":"2024-09-21T14:10:00.277868+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'39","last_scrub_stamp":"2024-09-21T14:09:47.338473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:09:47.338473+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:48.479017+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1035,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.284958+0000","last_change":"2024-09-21T14:09:48.156088+0000","last_active":"2024-09-21T14:10:05.284958+0000","last_peered":"2024-09-21T14:10:05.284958+0000","last_clean":"2024-09-21T14:10:05.284958+0000","last_became_active":"2024-09-21T14:09:00.254258+0000","last_became_peered":"2024-09-21T14:09:00.254258+0000","last_unstale":"2024-09-21T14:10:05.284958+0000","last_undegraded":"2024-09-21T14:10:05.284958+0000","last_fullsized":"2024-09-21T14:10:05.284958+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:09:48.156018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:09:48.156018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:49.616950+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":573,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.268755+0000","last_change":"2024-09-21T14:09:53.479210+0000","last_active":"2024-09-21T14:10:00.268755+0000","last_peered":"2024-09-21T14:10:00.268755+0000","last_clean":"2024-09-21T14:10:00.268755+0000","last_became_active":"2024-09-21T14:09:53.478621+0000","last_became_peered":"2024-09-21T14:09:53.478621+0000","last_unstale":"2024-09-21T14:10:00.268755+0000","last_undegraded":"2024-09-21T14:10:00.268755+0000","last_fullsized":"2024-09-21T14:10:00.268755+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:07:34.561444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:07:34.561444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":1019,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.277117+0000","last_change":"2024-09-21T14:08:59.040924+0000","last_active":"2024-09-21T14:10:00.277117+0000","last_peered":"2024-09-21T14:10:00.277117+0000","last_clean":"2024-09-21T14:10:00.277117+0000","last_became_active":"2024-09-21T13:52:04.316016+0000","last_became_peered":"2024-09-21T13:52:04.316016+0000","last_unstale":"2024-09-21T14:10:00.277117+0000","last_undegraded":"2024-09-21T14:10:00.277117+0000","last_fullsized":"2024-09-21T14:10:00.277117+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:08:59.040851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:08:59.040851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:06.606166+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"516'24","reported_seq":54,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.277003+0000","last_change":"2024-09-21T14:09:42.410247+0000","last_active":"2024-09-21T14:10:00.277003+0000","last_peered":"2024-09-21T14:10:00.277003+0000","last_clean":"2024-09-21T14:10:00.277003+0000","last_became_active":"2024-09-21T14:09:31.257652+0000","last_became_peered":"2024-09-21T14:09:31.257652+0000","last_unstale":"2024-09-21T14:10:00.277003+0000","last_undegraded":"2024-09-21T14:10:00.277003+0000","last_fullsized":"2024-09-21T14:10:00.277003+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'24","last_scrub_stamp":"2024-09-21T14:09:42.410188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:09:42.410188+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:44.946486+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1006,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.276043+0000","last_change":"2024-09-21T14:07:03.254086+0000","last_active":"2024-09-21T14:10:00.276043+0000","last_peered":"2024-09-21T14:10:00.276043+0000","last_clean":"2024-09-21T14:10:00.276043+0000","last_became_active":"2024-09-21T13:57:12.681897+0000","last_became_peered":"2024-09-21T13:57:12.681897+0000","last_unstale":"2024-09-21T14:10:00.276043+0000","last_undegraded":"2024-09-21T14:10:00.276043+0000","last_fullsized":"2024-09-21T14:10:00.276043+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:07:03.254025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:07:03.254025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":180,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.274970+0000","last_change":"2024-09-21T14:08:04.415897+0000","last_active":"2024-09-21T14:10:05.274970+0000","last_peered":"2024-09-21T14:10:05.274970+0000","last_clean":"2024-09-21T14:10:05.274970+0000","last_became_active":"2024-09-21T14:08:04.288485+0000","last_became_peered":"2024-09-21T14:08:04.288485+0000","last_unstale":"2024-09-21T14:10:05.274970+0000","last_undegraded":"2024-09-21T14:10:05.274970+0000","last_fullsized":"2024-09-21T14:10:05.274970+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T14:08:04.415821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:08:04.415821+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:52.214395+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":1047,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.285622+0000","last_change":"2024-09-21T14:09:00.267297+0000","last_active":"2024-09-21T14:10:05.285622+0000","last_peered":"2024-09-21T14:10:05.285622+0000","last_clean":"2024-09-21T14:10:05.285622+0000","last_became_active":"2024-09-21T14:09:00.257421+0000","last_became_peered":"2024-09-21T14:09:00.257421+0000","last_unstale":"2024-09-21T14:10:05.285622+0000","last_undegraded":"2024-09-21T14:10:05.285622+0000","last_fullsized":"2024-09-21T14:10:05.285622+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T14:06:46.580099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:06:46.580099+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:11:03.156200+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"516'30","reported_seq":366,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.284950+0000","last_change":"2024-09-21T14:09:53.477858+0000","last_active":"2024-09-21T14:10:05.284950+0000","last_peered":"2024-09-21T14:10:05.284950+0000","last_clean":"2024-09-21T14:10:05.284950+0000","last_became_active":"2024-09-21T14:09:53.476775+0000","last_became_peered":"2024-09-21T14:09:53.476775+0000","last_unstale":"2024-09-21T14:10:05.284950+0000","last_undegraded":"2024-09-21T14:10:05.284950+0000","last_fullsized":"2024-09-21T14:10:05.284950+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'20","last_scrub_stamp":"2024-09-21T14:05:56.301903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:05:56.301903+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:54.181837+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":37,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"513'14","reported_seq":747,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.268996+0000","last_change":"2024-09-21T14:09:00.264379+0000","last_active":"2024-09-21T14:10:00.268996+0000","last_peered":"2024-09-21T14:10:00.268996+0000","last_clean":"2024-09-21T14:10:00.268996+0000","last_became_active":"2024-09-21T14:09:00.258760+0000","last_became_peered":"2024-09-21T14:09:00.258760+0000","last_unstale":"2024-09-21T14:10:00.268996+0000","last_undegraded":"2024-09-21T14:10:00.268996+0000","last_fullsized":"2024-09-21T14:10:00.268996+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"409'13","last_scrub_stamp":"2024-09-21T14:07:46.640665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:07:46.640665+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.0","version":"538'13851","reported_seq":26426,"reported_epoch":538,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-21T14:10:05.355609+0000","last_change":"2024-09-21T14:10:00.286031+0000","last_active":"2024-09-21T14:10:05.355609+0000","last_peered":"2024-09-21T14:10:05.355609+0000","last_clean":"2024-09-21T14:08:13.356726+0000","last_became_active":"2024-09-21T14:08:15.445581+0000","last_became_peered":"2024-09-21T14:08:15.445581+0000","last_unstale":"2024-09-21T14:10:05.355609+0000","last_undegraded":"2024-09-21T14:10:05.355609+0000","last_fullsized":"2024-09-21T14:10:05.355609+0000","mapping_epoch":497,"log_start":"508'9789","ondisk_log_start":"508'9789","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_clean_scrub_stamp":"2024-09-21T14:06:44.365469+0000","objects_scrubbed":0,"log_size":4062,"log_dups_size":0,"ondisk_log_size":4062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98762752,"num_objects":1507,"num_object_clones":0,"num_object_copies":4521,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":63,"num_objects_unfound":0,"num_objects_dirty":1507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13851,"num_write_kb":491456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6195,"num_bytes_recovered":405995520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,3],"avail_no_missing":["1","3","7"],"object_location_counts":[{"shards":"1,3,7","objects":1507}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"75'79","reported_seq":438,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.278317+0000","last_change":"2024-09-21T14:09:32.389941+0000","last_active":"2024-09-21T14:10:05.278317+0000","last_peered":"2024-09-21T14:10:05.278317+0000","last_clean":"2024-09-21T14:10:05.278317+0000","last_became_active":"2024-09-21T14:01:41.246888+0000","last_became_peered":"2024-09-21T14:01:41.246888+0000","last_unstale":"2024-09-21T14:10:05.278317+0000","last_undegraded":"2024-09-21T14:10:05.278317+0000","last_fullsized":"2024-09-21T14:10:05.278317+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:09:32.389738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:09:32.389738+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:47.165827+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"516'39","reported_seq":1098,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.287941+0000","last_change":"2024-09-21T14:08:12.954432+0000","last_active":"2024-09-21T14:10:05.287941+0000","last_peered":"2024-09-21T14:10:05.287941+0000","last_clean":"2024-09-21T14:10:05.287941+0000","last_became_active":"2024-09-21T13:57:12.695609+0000","last_became_peered":"2024-09-21T13:57:12.695609+0000","last_unstale":"2024-09-21T14:10:05.287941+0000","last_undegraded":"2024-09-21T14:10:05.287941+0000","last_fullsized":"2024-09-21T14:10:05.287941+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'27","last_scrub_stamp":"2024-09-21T14:08:12.954351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:08:12.954351+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":544,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.286975+0000","last_change":"2024-09-21T14:08:14.411502+0000","last_active":"2024-09-21T14:10:05.286975+0000","last_peered":"2024-09-21T14:10:05.286975+0000","last_clean":"2024-09-21T14:10:05.286975+0000","last_became_active":"2024-09-21T14:08:14.410580+0000","last_became_peered":"2024-09-21T14:08:14.410580+0000","last_unstale":"2024-09-21T14:10:05.286975+0000","last_undegraded":"2024-09-21T14:10:05.286975+0000","last_fullsized":"2024-09-21T14:10:05.286975+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:07:41.211607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:07:41.211607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"513'12","reported_seq":87,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.275890+0000","last_change":"2024-09-21T14:09:35.393938+0000","last_active":"2024-09-21T14:10:00.275890+0000","last_peered":"2024-09-21T14:10:00.275890+0000","last_clean":"2024-09-21T14:10:00.275890+0000","last_became_active":"2024-09-21T14:09:00.253456+0000","last_became_peered":"2024-09-21T14:09:00.253456+0000","last_unstale":"2024-09-21T14:10:00.275890+0000","last_undegraded":"2024-09-21T14:10:00.275890+0000","last_fullsized":"2024-09-21T14:10:00.275890+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"513'12","last_scrub_stamp":"2024-09-21T14:09:35.393859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:09:35.393859+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:45.100127+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":897,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.274676+0000","last_change":"2024-09-21T14:09:20.049960+0000","last_active":"2024-09-21T14:10:05.274676+0000","last_peered":"2024-09-21T14:10:05.274676+0000","last_clean":"2024-09-21T14:10:05.274676+0000","last_became_active":"2024-09-21T14:09:20.049571+0000","last_became_peered":"2024-09-21T14:09:20.049571+0000","last_unstale":"2024-09-21T14:10:05.274676+0000","last_undegraded":"2024-09-21T14:10:05.274676+0000","last_fullsized":"2024-09-21T14:10:05.274676+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:48.393427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:05:48.393427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:20.292590+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"538'14021","reported_seq":14272,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.314477+0000","last_change":"2024-09-21T14:08:13.669043+0000","last_active":"2024-09-21T14:10:05.314477+0000","last_peered":"2024-09-21T14:10:05.314477+0000","last_clean":"2024-09-21T14:10:05.314477+0000","last_became_active":"2024-09-21T14:08:04.291231+0000","last_became_peered":"2024-09-21T14:08:04.291231+0000","last_unstale":"2024-09-21T14:10:05.314477+0000","last_undegraded":"2024-09-21T14:10:05.314477+0000","last_fullsized":"2024-09-21T14:10:05.314477+0000","mapping_epoch":491,"log_start":"509'9962","ondisk_log_start":"509'9962","created":461,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"496'7220","last_scrub_stamp":"2024-09-21T14:08:13.668911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_clean_scrub_stamp":"2024-09-21T14:08:13.668911+0000","objects_scrubbed":7144,"log_size":4059,"log_dups_size":0,"ondisk_log_size":4059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2668,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92995584,"num_objects":1419,"num_object_clones":0,"num_object_copies":4257,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14021,"num_write_kb":494080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"75'79","reported_seq":487,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.276310+0000","last_change":"2024-09-21T14:06:49.267268+0000","last_active":"2024-09-21T14:10:00.276310+0000","last_peered":"2024-09-21T14:10:00.276310+0000","last_clean":"2024-09-21T14:10:00.276310+0000","last_became_active":"2024-09-21T14:05:39.296209+0000","last_became_peered":"2024-09-21T14:05:39.296209+0000","last_unstale":"2024-09-21T14:10:00.276310+0000","last_undegraded":"2024-09-21T14:10:00.276310+0000","last_fullsized":"2024-09-21T14:10:00.276310+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:06:49.267210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:06:49.267210+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:58.494365+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"516'42","reported_seq":814,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.286637+0000","last_change":"2024-09-21T14:09:53.478836+0000","last_active":"2024-09-21T14:10:05.286637+0000","last_peered":"2024-09-21T14:10:05.286637+0000","last_clean":"2024-09-21T14:10:05.286637+0000","last_became_active":"2024-09-21T14:09:53.478280+0000","last_became_peered":"2024-09-21T14:09:53.478280+0000","last_unstale":"2024-09-21T14:10:05.286637+0000","last_undegraded":"2024-09-21T14:10:05.286637+0000","last_fullsized":"2024-09-21T14:10:05.286637+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'30","last_scrub_stamp":"2024-09-21T14:07:42.181176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:07:42.181176+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":913,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.267383+0000","last_change":"2024-09-21T14:09:01.342248+0000","last_active":"2024-09-21T14:10:00.267383+0000","last_peered":"2024-09-21T14:10:00.267383+0000","last_clean":"2024-09-21T14:10:00.267383+0000","last_became_active":"2024-09-21T13:59:32.211004+0000","last_became_peered":"2024-09-21T13:59:32.211004+0000","last_unstale":"2024-09-21T14:10:00.267383+0000","last_undegraded":"2024-09-21T14:10:00.267383+0000","last_fullsized":"2024-09-21T14:10:00.267383+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:09:01.342171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:09:01.342171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:20.730667+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1038,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.275743+0000","last_change":"2024-09-21T14:09:43.482457+0000","last_active":"2024-09-21T14:10:00.275743+0000","last_peered":"2024-09-21T14:10:00.275743+0000","last_clean":"2024-09-21T14:10:00.275743+0000","last_became_active":"2024-09-21T14:09:05.970776+0000","last_became_peered":"2024-09-21T14:09:05.970776+0000","last_unstale":"2024-09-21T14:10:00.275743+0000","last_undegraded":"2024-09-21T14:10:00.275743+0000","last_fullsized":"2024-09-21T14:10:00.275743+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:09:43.482393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:09:43.482393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:11:00.454321+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":70,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.276420+0000","last_change":"2024-09-21T14:09:34.433514+0000","last_active":"2024-09-21T14:10:00.276420+0000","last_peered":"2024-09-21T14:10:00.276420+0000","last_clean":"2024-09-21T14:10:00.276420+0000","last_became_active":"2024-09-21T14:09:00.253717+0000","last_became_peered":"2024-09-21T14:09:00.253717+0000","last_unstale":"2024-09-21T14:10:00.276420+0000","last_undegraded":"2024-09-21T14:10:00.276420+0000","last_fullsized":"2024-09-21T14:10:00.276420+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:09:34.433449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:09:34.433449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:54.181898+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"536'13988","reported_seq":23391,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:03.926169+0000","last_change":"2024-09-21T14:09:51.103765+0000","last_active":"2024-09-21T14:10:03.926169+0000","last_peered":"2024-09-21T14:10:03.926169+0000","last_clean":"2024-09-21T14:10:03.926169+0000","last_became_active":"2024-09-21T14:09:28.291814+0000","last_became_peered":"2024-09-21T14:09:28.291814+0000","last_unstale":"2024-09-21T14:10:03.926169+0000","last_undegraded":"2024-09-21T14:10:03.926169+0000","last_fullsized":"2024-09-21T14:10:03.926169+0000","mapping_epoch":521,"log_start":"507'9858","ondisk_log_start":"507'9858","created":461,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"529'13142","last_scrub_stamp":"2024-09-21T14:09:51.103521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_clean_scrub_stamp":"2024-09-21T14:09:51.103521+0000","objects_scrubbed":2423,"log_size":4130,"log_dups_size":0,"ondisk_log_size":4130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:11:00.526173+0000","scrub_duration":598,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101580800,"num_objects":1550,"num_object_clones":0,"num_object_copies":4650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13988,"num_write_kb":497216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4597,"num_bytes_recovered":301268992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"75'79","reported_seq":105,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.275181+0000","last_change":"2024-09-21T14:08:16.400437+0000","last_active":"2024-09-21T14:10:05.275181+0000","last_peered":"2024-09-21T14:10:05.275181+0000","last_clean":"2024-09-21T14:10:05.275181+0000","last_became_active":"2024-09-21T14:08:14.395761+0000","last_became_peered":"2024-09-21T14:08:14.395761+0000","last_unstale":"2024-09-21T14:10:05.275181+0000","last_undegraded":"2024-09-21T14:10:05.275181+0000","last_fullsized":"2024-09-21T14:10:05.275181+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:08:16.400353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:08:16.400353+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"516'42","reported_seq":1142,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.275821+0000","last_change":"2024-09-21T14:08:13.140559+0000","last_active":"2024-09-21T14:10:00.275821+0000","last_peered":"2024-09-21T14:10:00.275821+0000","last_clean":"2024-09-21T14:10:00.275821+0000","last_became_active":"2024-09-21T14:01:41.200859+0000","last_became_peered":"2024-09-21T14:01:41.200859+0000","last_unstale":"2024-09-21T14:10:00.275821+0000","last_undegraded":"2024-09-21T14:10:00.275821+0000","last_fullsized":"2024-09-21T14:10:00.275821+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'28","last_scrub_stamp":"2024-09-21T14:08:13.140470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:08:13.140470+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":267,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.285223+0000","last_change":"2024-09-21T14:09:53.481582+0000","last_active":"2024-09-21T14:10:05.285223+0000","last_peered":"2024-09-21T14:10:05.285223+0000","last_clean":"2024-09-21T14:10:05.285223+0000","last_became_active":"2024-09-21T14:09:53.481045+0000","last_became_peered":"2024-09-21T14:09:53.481045+0000","last_unstale":"2024-09-21T14:10:05.285223+0000","last_undegraded":"2024-09-21T14:10:05.285223+0000","last_fullsized":"2024-09-21T14:10:05.285223+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T14:07:47.858840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:07:47.858840+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"519'12","reported_seq":234,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.287221+0000","last_change":"2024-09-21T14:09:20.060665+0000","last_active":"2024-09-21T14:10:05.287221+0000","last_peered":"2024-09-21T14:10:05.287221+0000","last_clean":"2024-09-21T14:10:05.287221+0000","last_became_active":"2024-09-21T14:09:20.048106+0000","last_became_peered":"2024-09-21T14:09:20.048106+0000","last_unstale":"2024-09-21T14:10:05.287221+0000","last_undegraded":"2024-09-21T14:10:05.287221+0000","last_fullsized":"2024-09-21T14:10:05.287221+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'10","last_scrub_stamp":"2024-09-21T14:06:29.326448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:06:29.326448+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:55.147342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":607,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.268986+0000","last_change":"2024-09-21T14:08:04.293706+0000","last_active":"2024-09-21T14:10:00.268986+0000","last_peered":"2024-09-21T14:10:00.268986+0000","last_clean":"2024-09-21T14:10:00.268986+0000","last_became_active":"2024-09-21T14:08:04.292483+0000","last_became_peered":"2024-09-21T14:08:04.292483+0000","last_unstale":"2024-09-21T14:10:00.268986+0000","last_undegraded":"2024-09-21T14:10:00.268986+0000","last_fullsized":"2024-09-21T14:10:00.268986+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:45.055707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:04:45.055707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:59.968741+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"536'13704","reported_seq":13909,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:03.280388+0000","last_change":"2024-09-21T14:09:39.494469+0000","last_active":"2024-09-21T14:10:03.280388+0000","last_peered":"2024-09-21T14:10:03.280388+0000","last_clean":"2024-09-21T14:10:03.280388+0000","last_became_active":"2024-09-21T14:06:45.306827+0000","last_became_peered":"2024-09-21T14:06:45.306827+0000","last_unstale":"2024-09-21T14:10:03.280388+0000","last_undegraded":"2024-09-21T14:10:03.280388+0000","last_fullsized":"2024-09-21T14:10:03.280388+0000","mapping_epoch":461,"log_start":"502'8522","ondisk_log_start":"502'8522","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"527'12161","last_scrub_stamp":"2024-09-21T14:09:39.494341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_clean_scrub_stamp":"2024-09-21T14:09:39.494341+0000","objects_scrubbed":3245,"log_size":5182,"log_dups_size":0,"ondisk_log_size":5182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:11:05.593203+0000","scrub_duration":1112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109248536,"num_objects":1668,"num_object_clones":0,"num_object_copies":5004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1668,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13704,"num_write_kb":491841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"75'79","reported_seq":1067,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.268619+0000","last_change":"2024-09-21T14:08:14.395143+0000","last_active":"2024-09-21T14:10:00.268619+0000","last_peered":"2024-09-21T14:10:00.268619+0000","last_clean":"2024-09-21T14:10:00.268619+0000","last_became_active":"2024-09-21T14:08:14.394536+0000","last_became_peered":"2024-09-21T14:08:14.394536+0000","last_unstale":"2024-09-21T14:10:00.268619+0000","last_undegraded":"2024-09-21T14:10:00.268619+0000","last_fullsized":"2024-09-21T14:10:00.268619+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:05:45.878553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:05:45.878553+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"516'30","reported_seq":360,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.273806+0000","last_change":"2024-09-21T14:09:44.412826+0000","last_active":"2024-09-21T14:10:00.273806+0000","last_peered":"2024-09-21T14:10:00.273806+0000","last_clean":"2024-09-21T14:10:00.273806+0000","last_became_active":"2024-09-21T14:03:29.158218+0000","last_became_peered":"2024-09-21T14:03:29.158218+0000","last_unstale":"2024-09-21T14:10:00.273806+0000","last_undegraded":"2024-09-21T14:10:00.273806+0000","last_fullsized":"2024-09-21T14:10:00.273806+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'30","last_scrub_stamp":"2024-09-21T14:09:44.412609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:09:44.412609+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:54.002502+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"61'1","reported_seq":965,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.274768+0000","last_change":"2024-09-21T14:08:14.409954+0000","last_active":"2024-09-21T14:10:05.274768+0000","last_peered":"2024-09-21T14:10:05.274768+0000","last_clean":"2024-09-21T14:10:05.274768+0000","last_became_active":"2024-09-21T14:08:14.398397+0000","last_became_peered":"2024-09-21T14:08:14.398397+0000","last_unstale":"2024-09-21T14:10:05.274768+0000","last_undegraded":"2024-09-21T14:10:05.274768+0000","last_fullsized":"2024-09-21T14:10:05.274768+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T14:07:40.545199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:07:40.545199+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:28.310333+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":993,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"532'17","reported_seq":436,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.269681+0000","last_change":"2024-09-21T14:09:53.478808+0000","last_active":"2024-09-21T14:10:00.269681+0000","last_peered":"2024-09-21T14:10:00.269681+0000","last_clean":"2024-09-21T14:10:00.269681+0000","last_became_active":"2024-09-21T14:09:53.478308+0000","last_became_peered":"2024-09-21T14:09:53.478308+0000","last_unstale":"2024-09-21T14:10:00.269681+0000","last_undegraded":"2024-09-21T14:10:00.269681+0000","last_fullsized":"2024-09-21T14:10:00.269681+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"418'16","last_scrub_stamp":"2024-09-21T14:05:53.307631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:05:53.307631+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":593,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.265963+0000","last_change":"2024-09-21T14:09:53.477479+0000","last_active":"2024-09-21T14:10:00.265963+0000","last_peered":"2024-09-21T14:10:00.265963+0000","last_clean":"2024-09-21T14:10:00.265963+0000","last_became_active":"2024-09-21T14:09:53.476930+0000","last_became_peered":"2024-09-21T14:09:53.476930+0000","last_unstale":"2024-09-21T14:10:00.265963+0000","last_undegraded":"2024-09-21T14:10:00.265963+0000","last_fullsized":"2024-09-21T14:10:00.265963+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:46.372709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:05:46.372709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:56.154765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"538'14018","reported_seq":14181,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.355221+0000","last_change":"2024-09-21T14:06:45.306909+0000","last_active":"2024-09-21T14:10:05.355221+0000","last_peered":"2024-09-21T14:10:05.355221+0000","last_clean":"2024-09-21T14:10:05.355221+0000","last_became_active":"2024-09-21T14:06:45.306632+0000","last_became_peered":"2024-09-21T14:06:45.306632+0000","last_unstale":"2024-09-21T14:10:05.355221+0000","last_undegraded":"2024-09-21T14:10:05.355221+0000","last_fullsized":"2024-09-21T14:10:05.355221+0000","mapping_epoch":461,"log_start":"509'9942","ondisk_log_start":"509'9942","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_clean_scrub_stamp":"2024-09-21T14:06:44.365469+0000","objects_scrubbed":0,"log_size":4076,"log_dups_size":0,"ondisk_log_size":4076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98566144,"num_objects":1504,"num_object_clones":0,"num_object_copies":4512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14018,"num_write_kb":496704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"75'79","reported_seq":1075,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.268586+0000","last_change":"2024-09-21T14:08:14.402716+0000","last_active":"2024-09-21T14:10:00.268586+0000","last_peered":"2024-09-21T14:10:00.268586+0000","last_clean":"2024-09-21T14:10:00.268586+0000","last_became_active":"2024-09-21T14:08:14.402297+0000","last_became_peered":"2024-09-21T14:08:14.402297+0000","last_unstale":"2024-09-21T14:10:00.268586+0000","last_undegraded":"2024-09-21T14:10:00.268586+0000","last_fullsized":"2024-09-21T14:10:00.268586+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:05:41.888228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:05:41.888228+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"516'39","reported_seq":759,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.274940+0000","last_change":"2024-09-21T14:09:53.475448+0000","last_active":"2024-09-21T14:10:05.274940+0000","last_peered":"2024-09-21T14:10:05.274940+0000","last_clean":"2024-09-21T14:10:05.274940+0000","last_became_active":"2024-09-21T14:09:53.474895+0000","last_became_peered":"2024-09-21T14:09:53.474895+0000","last_unstale":"2024-09-21T14:10:05.274940+0000","last_undegraded":"2024-09-21T14:10:05.274940+0000","last_fullsized":"2024-09-21T14:10:05.274940+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'29","last_scrub_stamp":"2024-09-21T14:07:32.545961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:07:32.545961+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:11:01.269469+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":66,"num_read_kb":52,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1021,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.286097+0000","last_change":"2024-09-21T14:06:27.457531+0000","last_active":"2024-09-21T14:10:05.286097+0000","last_peered":"2024-09-21T14:10:05.286097+0000","last_clean":"2024-09-21T14:10:05.286097+0000","last_became_active":"2024-09-21T14:01:41.185400+0000","last_became_peered":"2024-09-21T14:01:41.185400+0000","last_unstale":"2024-09-21T14:10:05.286097+0000","last_undegraded":"2024-09-21T14:10:05.286097+0000","last_fullsized":"2024-09-21T14:10:05.286097+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:27.457452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:06:27.457452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:57.203039+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":600,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.269813+0000","last_change":"2024-09-21T14:07:04.801076+0000","last_active":"2024-09-21T14:10:00.269813+0000","last_peered":"2024-09-21T14:10:00.269813+0000","last_clean":"2024-09-21T14:10:00.269813+0000","last_became_active":"2024-09-21T14:01:41.193973+0000","last_became_peered":"2024-09-21T14:01:41.193973+0000","last_unstale":"2024-09-21T14:10:00.269813+0000","last_undegraded":"2024-09-21T14:10:00.269813+0000","last_fullsized":"2024-09-21T14:10:00.269813+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:07:04.801001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:07:04.801001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1012,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.285288+0000","last_change":"2024-09-21T14:07:42.868224+0000","last_active":"2024-09-21T14:10:05.285288+0000","last_peered":"2024-09-21T14:10:05.285288+0000","last_clean":"2024-09-21T14:10:05.285288+0000","last_became_active":"2024-09-21T13:58:17.757628+0000","last_became_peered":"2024-09-21T13:58:17.757628+0000","last_unstale":"2024-09-21T14:10:05.285288+0000","last_undegraded":"2024-09-21T14:10:05.285288+0000","last_fullsized":"2024-09-21T14:10:05.285288+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:07:42.868151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:07:42.868151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.b","version":"536'13532","reported_seq":13739,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:01.809876+0000","last_change":"2024-09-21T14:09:36.240676+0000","last_active":"2024-09-21T14:10:01.809876+0000","last_peered":"2024-09-21T14:10:01.809876+0000","last_clean":"2024-09-21T14:10:01.809876+0000","last_became_active":"2024-09-21T14:06:45.309027+0000","last_became_peered":"2024-09-21T14:06:45.309027+0000","last_unstale":"2024-09-21T14:10:01.809876+0000","last_undegraded":"2024-09-21T14:10:01.809876+0000","last_fullsized":"2024-09-21T14:10:01.809876+0000","mapping_epoch":461,"log_start":"476'3500","ondisk_log_start":"476'3500","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"527'11801","last_scrub_stamp":"2024-09-21T14:09:36.240503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_clean_scrub_stamp":"2024-09-21T14:09:36.240503+0000","objects_scrubbed":3487,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:11:02.800013+0000","scrub_duration":2037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109576192,"num_objects":1672,"num_object_clones":0,"num_object_copies":5016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13532,"num_write_kb":486528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"75'79","reported_seq":436,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.274286+0000","last_change":"2024-09-21T14:08:48.262540+0000","last_active":"2024-09-21T14:10:05.274286+0000","last_peered":"2024-09-21T14:10:05.274286+0000","last_clean":"2024-09-21T14:10:05.274286+0000","last_became_active":"2024-09-21T14:05:39.310615+0000","last_became_peered":"2024-09-21T14:05:39.310615+0000","last_unstale":"2024-09-21T14:10:05.274286+0000","last_undegraded":"2024-09-21T14:10:05.274286+0000","last_fullsized":"2024-09-21T14:10:05.274286+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:08:48.262482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:08:48.262482+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"516'57","reported_seq":1413,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.285840+0000","last_change":"2024-09-21T14:08:14.440676+0000","last_active":"2024-09-21T14:10:05.285840+0000","last_peered":"2024-09-21T14:10:05.285840+0000","last_clean":"2024-09-21T14:10:05.285840+0000","last_became_active":"2024-09-21T14:08:14.418714+0000","last_became_peered":"2024-09-21T14:08:14.418714+0000","last_unstale":"2024-09-21T14:10:05.285840+0000","last_undegraded":"2024-09-21T14:10:05.285840+0000","last_fullsized":"2024-09-21T14:10:05.285840+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'44","last_scrub_stamp":"2024-09-21T14:08:10.016212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:08:10.016212+0000","objects_scrubbed":10,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":84,"num_read_kb":69,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1040,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.276633+0000","last_change":"2024-09-21T14:09:38.447915+0000","last_active":"2024-09-21T14:10:00.276633+0000","last_peered":"2024-09-21T14:10:00.276633+0000","last_clean":"2024-09-21T14:10:00.276633+0000","last_became_active":"2024-09-21T13:58:17.770937+0000","last_became_peered":"2024-09-21T13:58:17.770937+0000","last_unstale":"2024-09-21T14:10:00.276633+0000","last_undegraded":"2024-09-21T14:10:00.276633+0000","last_fullsized":"2024-09-21T14:10:00.276633+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:09:38.447835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:09:38.447835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:50.593959+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":981,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.278227+0000","last_change":"2024-09-21T14:09:50.282494+0000","last_active":"2024-09-21T14:10:05.278227+0000","last_peered":"2024-09-21T14:10:05.278227+0000","last_clean":"2024-09-21T14:10:05.278227+0000","last_became_active":"2024-09-21T14:01:41.247642+0000","last_became_peered":"2024-09-21T14:01:41.247642+0000","last_unstale":"2024-09-21T14:10:05.278227+0000","last_undegraded":"2024-09-21T14:10:05.278227+0000","last_fullsized":"2024-09-21T14:10:05.278227+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-21T14:09:50.282440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T14:09:50.282440+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:11:19.768152+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":944,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.284779+0000","last_change":"2024-09-21T14:09:53.476470+0000","last_active":"2024-09-21T14:10:05.284779+0000","last_peered":"2024-09-21T14:10:05.284779+0000","last_clean":"2024-09-21T14:10:05.284779+0000","last_became_active":"2024-09-21T14:09:53.476184+0000","last_became_peered":"2024-09-21T14:09:53.476184+0000","last_unstale":"2024-09-21T14:10:05.284779+0000","last_undegraded":"2024-09-21T14:10:05.284779+0000","last_fullsized":"2024-09-21T14:10:05.284779+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:07:45.837251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:07:45.837251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":527,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.279418+0000","last_change":"2024-09-21T14:09:40.536403+0000","last_active":"2024-09-21T14:10:00.279418+0000","last_peered":"2024-09-21T14:10:00.279418+0000","last_clean":"2024-09-21T14:10:00.279418+0000","last_became_active":"2024-09-21T14:08:14.394400+0000","last_became_peered":"2024-09-21T14:08:14.394400+0000","last_unstale":"2024-09-21T14:10:00.279418+0000","last_undegraded":"2024-09-21T14:10:00.279418+0000","last_fullsized":"2024-09-21T14:10:00.279418+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:09:40.536328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:09:40.536328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:58.347403+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.8","version":"536'13590","reported_seq":33891,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:04.038702+0000","last_change":"2024-09-21T14:09:42.407127+0000","last_active":"2024-09-21T14:10:04.038702+0000","last_peered":"2024-09-21T14:10:04.038702+0000","last_clean":"2024-09-21T14:10:04.038702+0000","last_became_active":"2024-09-21T14:09:42.385441+0000","last_became_peered":"2024-09-21T14:09:42.385441+0000","last_unstale":"2024-09-21T14:10:04.038702+0000","last_undegraded":"2024-09-21T14:10:04.038702+0000","last_fullsized":"2024-09-21T14:10:04.038702+0000","mapping_epoch":528,"log_start":"506'9285","ondisk_log_start":"506'9285","created":461,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_clean_scrub_stamp":"2024-09-21T14:06:44.365469+0000","objects_scrubbed":0,"log_size":4305,"log_dups_size":0,"ondisk_log_size":4305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104595456,"num_objects":1596,"num_object_clones":0,"num_object_copies":4788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13590,"num_write_kb":485952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10053,"num_bytes_recovered":658833408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"75'79","reported_seq":164,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.276321+0000","last_change":"2024-09-21T14:09:53.476792+0000","last_active":"2024-09-21T14:10:00.276321+0000","last_peered":"2024-09-21T14:10:00.276321+0000","last_clean":"2024-09-21T14:10:00.276321+0000","last_became_active":"2024-09-21T14:09:53.476430+0000","last_became_peered":"2024-09-21T14:09:53.476430+0000","last_unstale":"2024-09-21T14:10:00.276321+0000","last_undegraded":"2024-09-21T14:10:00.276321+0000","last_fullsized":"2024-09-21T14:10:00.276321+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:08:04.448969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:08:04.448969+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"516'21","reported_seq":1090,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.288049+0000","last_change":"2024-09-21T14:09:00.271789+0000","last_active":"2024-09-21T14:10:05.288049+0000","last_peered":"2024-09-21T14:10:05.288049+0000","last_clean":"2024-09-21T14:10:05.288049+0000","last_became_active":"2024-09-21T14:09:00.258874+0000","last_became_peered":"2024-09-21T14:09:00.258874+0000","last_unstale":"2024-09-21T14:10:05.288049+0000","last_undegraded":"2024-09-21T14:10:05.288049+0000","last_fullsized":"2024-09-21T14:10:05.288049+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'17","last_scrub_stamp":"2024-09-21T14:07:32.780439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:07:32.780439+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1064,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.286575+0000","last_change":"2024-09-21T14:09:31.020309+0000","last_active":"2024-09-21T14:10:05.286575+0000","last_peered":"2024-09-21T14:10:05.286575+0000","last_clean":"2024-09-21T14:10:05.286575+0000","last_became_active":"2024-09-21T13:58:17.759671+0000","last_became_peered":"2024-09-21T13:58:17.759671+0000","last_unstale":"2024-09-21T14:10:05.286575+0000","last_undegraded":"2024-09-21T14:10:05.286575+0000","last_fullsized":"2024-09-21T14:10:05.286575+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:09:31.020250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:09:31.020250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:38.259257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":944,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.275343+0000","last_change":"2024-09-21T14:08:14.427269+0000","last_active":"2024-09-21T14:10:00.275343+0000","last_peered":"2024-09-21T14:10:00.275343+0000","last_clean":"2024-09-21T14:10:00.275343+0000","last_became_active":"2024-09-21T14:08:14.415335+0000","last_became_peered":"2024-09-21T14:08:14.415335+0000","last_unstale":"2024-09-21T14:10:00.275343+0000","last_undegraded":"2024-09-21T14:10:00.275343+0000","last_fullsized":"2024-09-21T14:10:00.275343+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T14:07:43.046486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T14:07:43.046486+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":928,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.279024+0000","last_change":"2024-09-21T14:08:02.089974+0000","last_active":"2024-09-21T14:10:00.279024+0000","last_peered":"2024-09-21T14:10:00.279024+0000","last_clean":"2024-09-21T14:10:00.279024+0000","last_became_active":"2024-09-21T14:01:41.182000+0000","last_became_peered":"2024-09-21T14:01:41.182000+0000","last_unstale":"2024-09-21T14:10:00.279024+0000","last_undegraded":"2024-09-21T14:10:00.279024+0000","last_fullsized":"2024-09-21T14:10:00.279024+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:08:02.089791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:08:02.089791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":615,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.269266+0000","last_change":"2024-09-21T14:08:04.290668+0000","last_active":"2024-09-21T14:10:00.269266+0000","last_peered":"2024-09-21T14:10:00.269266+0000","last_clean":"2024-09-21T14:10:00.269266+0000","last_became_active":"2024-09-21T14:08:04.290357+0000","last_became_peered":"2024-09-21T14:08:04.290357+0000","last_unstale":"2024-09-21T14:10:00.269266+0000","last_undegraded":"2024-09-21T14:10:00.269266+0000","last_fullsized":"2024-09-21T14:10:00.269266+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:07:03.383502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:07:03.383502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.9","version":"536'13874","reported_seq":37556,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:04.026320+0000","last_change":"2024-09-21T14:09:32.276151+0000","last_active":"2024-09-21T14:10:04.026320+0000","last_peered":"2024-09-21T14:10:04.026320+0000","last_clean":"2024-09-21T14:10:04.026320+0000","last_became_active":"2024-09-21T14:09:32.256387+0000","last_became_peered":"2024-09-21T14:09:32.256387+0000","last_unstale":"2024-09-21T14:10:04.026320+0000","last_undegraded":"2024-09-21T14:10:04.026320+0000","last_fullsized":"2024-09-21T14:10:04.026320+0000","mapping_epoch":525,"log_start":"506'9638","ondisk_log_start":"506'9638","created":461,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_clean_scrub_stamp":"2024-09-21T14:06:44.365469+0000","objects_scrubbed":0,"log_size":4236,"log_dups_size":0,"ondisk_log_size":4236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106561536,"num_objects":1626,"num_object_clones":0,"num_object_copies":4878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13874,"num_write_kb":496000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11745,"num_bytes_recovered":769720320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"75'79","reported_seq":490,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.288023+0000","last_change":"2024-09-21T14:06:52.656799+0000","last_active":"2024-09-21T14:10:05.288023+0000","last_peered":"2024-09-21T14:10:05.288023+0000","last_clean":"2024-09-21T14:10:05.288023+0000","last_became_active":"2024-09-21T14:05:39.311460+0000","last_became_peered":"2024-09-21T14:05:39.311460+0000","last_unstale":"2024-09-21T14:10:05.288023+0000","last_undegraded":"2024-09-21T14:10:05.288023+0000","last_fullsized":"2024-09-21T14:10:05.288023+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:06:52.656733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:06:52.656733+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"516'36","reported_seq":852,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.275420+0000","last_change":"2024-09-21T14:09:53.476027+0000","last_active":"2024-09-21T14:10:05.275420+0000","last_peered":"2024-09-21T14:10:05.275420+0000","last_clean":"2024-09-21T14:10:05.275420+0000","last_became_active":"2024-09-21T14:09:53.475544+0000","last_became_peered":"2024-09-21T14:09:53.475544+0000","last_unstale":"2024-09-21T14:10:05.275420+0000","last_undegraded":"2024-09-21T14:10:05.275420+0000","last_fullsized":"2024-09-21T14:10:05.275420+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'36","last_scrub_stamp":"2024-09-21T14:09:48.023313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:09:48.023313+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:11:02.147910+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":47,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":551,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.275615+0000","last_change":"2024-09-21T14:09:53.472387+0000","last_active":"2024-09-21T14:10:00.275615+0000","last_peered":"2024-09-21T14:10:00.275615+0000","last_clean":"2024-09-21T14:10:00.275615+0000","last_became_active":"2024-09-21T14:09:53.472120+0000","last_became_peered":"2024-09-21T14:09:53.472120+0000","last_unstale":"2024-09-21T14:10:00.275615+0000","last_undegraded":"2024-09-21T14:10:00.275615+0000","last_fullsized":"2024-09-21T14:10:00.275615+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:08:09.550409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:08:09.550409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":398,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.286873+0000","last_change":"2024-09-21T14:08:07.018283+0000","last_active":"2024-09-21T14:10:05.286873+0000","last_peered":"2024-09-21T14:10:05.286873+0000","last_clean":"2024-09-21T14:10:05.286873+0000","last_became_active":"2024-09-21T14:08:04.289119+0000","last_became_peered":"2024-09-21T14:08:04.289119+0000","last_unstale":"2024-09-21T14:10:05.286873+0000","last_undegraded":"2024-09-21T14:10:05.286873+0000","last_fullsized":"2024-09-21T14:10:05.286873+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-21T14:08:07.018200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T14:08:07.018200+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0.00047048000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":193,"num_read_kb":188,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":951,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.276068+0000","last_change":"2024-09-21T14:09:03.344780+0000","last_active":"2024-09-21T14:10:05.276068+0000","last_peered":"2024-09-21T14:10:05.276068+0000","last_clean":"2024-09-21T14:10:05.276068+0000","last_became_active":"2024-09-21T14:01:41.200531+0000","last_became_peered":"2024-09-21T14:01:41.200531+0000","last_unstale":"2024-09-21T14:10:05.276068+0000","last_undegraded":"2024-09-21T14:10:05.276068+0000","last_fullsized":"2024-09-21T14:10:05.276068+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:09:03.344612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:09:03.344612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:14.593720+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1056,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.285594+0000","last_change":"2024-09-21T14:09:38.077262+0000","last_active":"2024-09-21T14:10:05.285594+0000","last_peered":"2024-09-21T14:10:05.285594+0000","last_clean":"2024-09-21T14:10:05.285594+0000","last_became_active":"2024-09-21T14:08:14.410113+0000","last_became_peered":"2024-09-21T14:08:14.410113+0000","last_unstale":"2024-09-21T14:10:05.285594+0000","last_undegraded":"2024-09-21T14:10:05.285594+0000","last_fullsized":"2024-09-21T14:10:05.285594+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:09:38.077194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:09:38.077194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:11:05.590972+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"536'13842","reported_seq":14050,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:01.765727+0000","last_change":"2024-09-21T14:09:31.810433+0000","last_active":"2024-09-21T14:10:01.765727+0000","last_peered":"2024-09-21T14:10:01.765727+0000","last_clean":"2024-09-21T14:10:01.765727+0000","last_became_active":"2024-09-21T14:06:45.199722+0000","last_became_peered":"2024-09-21T14:06:45.199722+0000","last_unstale":"2024-09-21T14:10:01.765727+0000","last_undegraded":"2024-09-21T14:10:01.765727+0000","last_fullsized":"2024-09-21T14:10:01.765727+0000","mapping_epoch":461,"log_start":"476'3800","ondisk_log_start":"476'3800","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"525'11822","last_scrub_stamp":"2024-09-21T14:09:31.810207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_clean_scrub_stamp":"2024-09-21T14:09:31.810207+0000","objects_scrubbed":3794,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:32.498539+0000","scrub_duration":587,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116129792,"num_objects":1772,"num_object_clones":0,"num_object_copies":5316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13842,"num_write_kb":499648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"516'44","reported_seq":788,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.270433+0000","last_change":"2024-09-21T14:09:45.757846+0000","last_active":"2024-09-21T14:10:00.270433+0000","last_peered":"2024-09-21T14:10:00.270433+0000","last_clean":"2024-09-21T14:10:00.270433+0000","last_became_active":"2024-09-21T13:59:32.204624+0000","last_became_peered":"2024-09-21T13:59:32.204624+0000","last_unstale":"2024-09-21T14:10:00.270433+0000","last_undegraded":"2024-09-21T14:10:00.270433+0000","last_fullsized":"2024-09-21T14:10:00.270433+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'44","last_scrub_stamp":"2024-09-21T14:09:45.757763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:09:45.757763+0000","objects_scrubbed":8,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:11:06.476845+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":676,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.278109+0000","last_change":"2024-09-21T14:08:14.394325+0000","last_active":"2024-09-21T14:10:05.278109+0000","last_peered":"2024-09-21T14:10:05.278109+0000","last_clean":"2024-09-21T14:10:05.278109+0000","last_became_active":"2024-09-21T14:08:14.393535+0000","last_became_peered":"2024-09-21T14:08:14.393535+0000","last_unstale":"2024-09-21T14:10:05.278109+0000","last_undegraded":"2024-09-21T14:10:05.278109+0000","last_fullsized":"2024-09-21T14:10:05.278109+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:50.360693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:06:50.360693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:09.329173+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"75'79","reported_seq":647,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.268217+0000","last_change":"2024-09-21T14:09:53.482238+0000","last_active":"2024-09-21T14:10:00.268217+0000","last_peered":"2024-09-21T14:10:00.268217+0000","last_clean":"2024-09-21T14:10:00.268217+0000","last_became_active":"2024-09-21T14:09:53.481913+0000","last_became_peered":"2024-09-21T14:09:53.481913+0000","last_unstale":"2024-09-21T14:10:00.268217+0000","last_undegraded":"2024-09-21T14:10:00.268217+0000","last_fullsized":"2024-09-21T14:10:00.268217+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:09:47.053077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:09:47.053077+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:11:14.670344+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":665,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.277611+0000","last_change":"2024-09-21T14:08:07.199530+0000","last_active":"2024-09-21T14:10:00.277611+0000","last_peered":"2024-09-21T14:10:00.277611+0000","last_clean":"2024-09-21T14:10:00.277611+0000","last_became_active":"2024-09-21T14:08:04.288211+0000","last_became_peered":"2024-09-21T14:08:04.288211+0000","last_unstale":"2024-09-21T14:10:00.277611+0000","last_undegraded":"2024-09-21T14:10:00.277611+0000","last_fullsized":"2024-09-21T14:10:00.277611+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:08:07.199297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:08:07.199297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":908,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.276249+0000","last_change":"2024-09-21T14:09:37.303313+0000","last_active":"2024-09-21T14:10:05.276249+0000","last_peered":"2024-09-21T14:10:05.276249+0000","last_clean":"2024-09-21T14:10:05.276249+0000","last_became_active":"2024-09-21T14:03:57.346453+0000","last_became_peered":"2024-09-21T14:03:57.346453+0000","last_unstale":"2024-09-21T14:10:05.276249+0000","last_undegraded":"2024-09-21T14:10:05.276249+0000","last_fullsized":"2024-09-21T14:10:05.276249+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:09:37.303243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:09:37.303243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:55.239893+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"536'13709","reported_seq":24470,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:04.045186+0000","last_change":"2024-09-21T14:09:21.061867+0000","last_active":"2024-09-21T14:10:04.045186+0000","last_peered":"2024-09-21T14:10:04.045186+0000","last_clean":"2024-09-21T14:10:04.045186+0000","last_became_active":"2024-09-21T14:09:21.043454+0000","last_became_peered":"2024-09-21T14:09:21.043454+0000","last_unstale":"2024-09-21T14:10:04.045186+0000","last_undegraded":"2024-09-21T14:10:04.045186+0000","last_fullsized":"2024-09-21T14:10:04.045186+0000","mapping_epoch":519,"log_start":"506'9417","ondisk_log_start":"506'9417","created":461,"last_epoch_clean":520,"parent":"0.0","parent_split_bits":0,"last_scrub":"491'6444","last_scrub_stamp":"2024-09-21T14:08:03.828589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_clean_scrub_stamp":"2024-09-21T14:08:03.828589+0000","objects_scrubbed":6440,"log_size":4292,"log_dups_size":0,"ondisk_log_size":4292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":890,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103219200,"num_objects":1575,"num_object_clones":0,"num_object_copies":4725,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13709,"num_write_kb":489088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5251,"num_bytes_recovered":344129536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"75'79","reported_seq":495,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.287868+0000","last_change":"2024-09-21T14:09:36.116938+0000","last_active":"2024-09-21T14:10:05.287868+0000","last_peered":"2024-09-21T14:10:05.287868+0000","last_clean":"2024-09-21T14:10:05.287868+0000","last_became_active":"2024-09-21T14:05:41.345392+0000","last_became_peered":"2024-09-21T14:05:41.345392+0000","last_unstale":"2024-09-21T14:10:05.287868+0000","last_undegraded":"2024-09-21T14:10:05.287868+0000","last_fullsized":"2024-09-21T14:10:05.287868+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:09:36.116463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:09:36.116463+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:49.044308+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"516'27","reported_seq":1089,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.279184+0000","last_change":"2024-09-21T14:07:04.229852+0000","last_active":"2024-09-21T14:10:00.279184+0000","last_peered":"2024-09-21T14:10:00.279184+0000","last_clean":"2024-09-21T14:10:00.279184+0000","last_became_active":"2024-09-21T13:58:17.776310+0000","last_became_peered":"2024-09-21T13:58:17.776310+0000","last_unstale":"2024-09-21T14:10:00.279184+0000","last_undegraded":"2024-09-21T14:10:00.279184+0000","last_fullsized":"2024-09-21T14:10:00.279184+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'19","last_scrub_stamp":"2024-09-21T14:07:04.229772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:07:04.229772+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1047,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.276218+0000","last_change":"2024-09-21T14:08:16.097026+0000","last_active":"2024-09-21T14:10:00.276218+0000","last_peered":"2024-09-21T14:10:00.276218+0000","last_clean":"2024-09-21T14:10:00.276218+0000","last_became_active":"2024-09-21T13:59:32.223247+0000","last_became_peered":"2024-09-21T13:59:32.223247+0000","last_unstale":"2024-09-21T14:10:00.276218+0000","last_undegraded":"2024-09-21T14:10:00.276218+0000","last_fullsized":"2024-09-21T14:10:00.276218+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:08:16.096946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:08:16.096946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":977,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.275800+0000","last_change":"2024-09-21T14:08:56.305300+0000","last_active":"2024-09-21T14:10:05.275800+0000","last_peered":"2024-09-21T14:10:05.275800+0000","last_clean":"2024-09-21T14:10:05.275800+0000","last_became_active":"2024-09-21T14:01:41.193941+0000","last_became_peered":"2024-09-21T14:01:41.193941+0000","last_unstale":"2024-09-21T14:10:05.275800+0000","last_undegraded":"2024-09-21T14:10:05.275800+0000","last_fullsized":"2024-09-21T14:10:05.275800+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:08:56.305192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:08:56.305192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":650,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.270255+0000","last_change":"2024-09-21T14:08:14.399584+0000","last_active":"2024-09-21T14:10:00.270255+0000","last_peered":"2024-09-21T14:10:00.270255+0000","last_clean":"2024-09-21T14:10:00.270255+0000","last_became_active":"2024-09-21T14:08:14.398067+0000","last_became_peered":"2024-09-21T14:08:14.398067+0000","last_unstale":"2024-09-21T14:10:00.270255+0000","last_undegraded":"2024-09-21T14:10:00.270255+0000","last_fullsized":"2024-09-21T14:10:00.270255+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:30.364645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:06:30.364645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"536'13649","reported_seq":27436,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:03.291105+0000","last_change":"2024-09-21T14:09:53.516436+0000","last_active":"2024-09-21T14:10:03.291105+0000","last_peered":"2024-09-21T14:10:03.291105+0000","last_clean":"2024-09-21T14:10:03.291105+0000","last_became_active":"2024-09-21T14:09:53.481806+0000","last_became_peered":"2024-09-21T14:09:53.481806+0000","last_unstale":"2024-09-21T14:10:03.291105+0000","last_undegraded":"2024-09-21T14:10:03.291105+0000","last_fullsized":"2024-09-21T14:10:03.291105+0000","mapping_epoch":531,"log_start":"502'8488","ondisk_log_start":"502'8488","created":461,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_clean_scrub_stamp":"2024-09-21T14:06:44.365469+0000","objects_scrubbed":0,"log_size":5161,"log_dups_size":0,"ondisk_log_size":5161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:58.429669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106889216,"num_objects":1631,"num_object_clones":0,"num_object_copies":4893,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13649,"num_write_kb":488960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6801,"num_bytes_recovered":445710336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"75'79","reported_seq":464,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.273966+0000","last_change":"2024-09-21T14:09:44.277168+0000","last_active":"2024-09-21T14:10:05.273966+0000","last_peered":"2024-09-21T14:10:05.273966+0000","last_clean":"2024-09-21T14:10:05.273966+0000","last_became_active":"2024-09-21T14:08:14.401513+0000","last_became_peered":"2024-09-21T14:08:14.401513+0000","last_unstale":"2024-09-21T14:10:05.273966+0000","last_undegraded":"2024-09-21T14:10:05.273966+0000","last_fullsized":"2024-09-21T14:10:05.273966+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:09:44.277011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:09:44.277011+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:48.155726+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"516'45","reported_seq":101,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.276145+0000","last_change":"2024-09-21T14:09:00.277354+0000","last_active":"2024-09-21T14:10:00.276145+0000","last_peered":"2024-09-21T14:10:00.276145+0000","last_clean":"2024-09-21T14:10:00.276145+0000","last_became_active":"2024-09-21T14:09:00.257088+0000","last_became_peered":"2024-09-21T14:09:00.257088+0000","last_unstale":"2024-09-21T14:10:00.276145+0000","last_undegraded":"2024-09-21T14:10:00.276145+0000","last_fullsized":"2024-09-21T14:10:00.276145+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'31","last_scrub_stamp":"2024-09-21T14:07:02.225989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:07:02.225989+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":60,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":573,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.268508+0000","last_change":"2024-09-21T14:04:51.139211+0000","last_active":"2024-09-21T14:10:00.268508+0000","last_peered":"2024-09-21T14:10:00.268508+0000","last_clean":"2024-09-21T14:10:00.268508+0000","last_became_active":"2024-09-21T13:58:17.762520+0000","last_became_peered":"2024-09-21T13:58:17.762520+0000","last_unstale":"2024-09-21T14:10:00.268508+0000","last_undegraded":"2024-09-21T14:10:00.268508+0000","last_fullsized":"2024-09-21T14:10:00.268508+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:51.139123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:04:51.139123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:52.856680+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":990,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.285688+0000","last_change":"2024-09-21T14:07:41.870428+0000","last_active":"2024-09-21T14:10:05.285688+0000","last_peered":"2024-09-21T14:10:05.285688+0000","last_clean":"2024-09-21T14:10:05.285688+0000","last_became_active":"2024-09-21T13:49:02.067472+0000","last_became_peered":"2024-09-21T13:49:02.067472+0000","last_unstale":"2024-09-21T14:10:05.285688+0000","last_undegraded":"2024-09-21T14:10:05.285688+0000","last_fullsized":"2024-09-21T14:10:05.285688+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:07:41.870349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:07:41.870349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":876,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.278680+0000","last_change":"2024-09-21T14:08:14.411279+0000","last_active":"2024-09-21T14:10:00.278680+0000","last_peered":"2024-09-21T14:10:00.278680+0000","last_clean":"2024-09-21T14:10:00.278680+0000","last_became_active":"2024-09-21T14:08:14.410821+0000","last_became_peered":"2024-09-21T14:08:14.410821+0000","last_unstale":"2024-09-21T14:10:00.278680+0000","last_undegraded":"2024-09-21T14:10:00.278680+0000","last_fullsized":"2024-09-21T14:10:00.278680+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:07:41.059862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:07:41.059862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"536'13959","reported_seq":14120,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:04.068022+0000","last_change":"2024-09-21T14:06:45.197312+0000","last_active":"2024-09-21T14:10:04.068022+0000","last_peered":"2024-09-21T14:10:04.068022+0000","last_clean":"2024-09-21T14:10:04.068022+0000","last_became_active":"2024-09-21T14:06:45.197043+0000","last_became_peered":"2024-09-21T14:06:45.197043+0000","last_unstale":"2024-09-21T14:10:04.068022+0000","last_undegraded":"2024-09-21T14:10:04.068022+0000","last_fullsized":"2024-09-21T14:10:04.068022+0000","mapping_epoch":461,"log_start":"506'9690","ondisk_log_start":"506'9690","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_clean_scrub_stamp":"2024-09-21T14:06:44.365469+0000","objects_scrubbed":0,"log_size":4269,"log_dups_size":0,"ondisk_log_size":4269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109510656,"num_objects":1671,"num_object_clones":0,"num_object_copies":5013,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13959,"num_write_kb":500160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"75'79","reported_seq":670,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.275035+0000","last_change":"2024-09-21T14:08:47.765514+0000","last_active":"2024-09-21T14:10:00.275035+0000","last_peered":"2024-09-21T14:10:00.275035+0000","last_clean":"2024-09-21T14:10:00.275035+0000","last_became_active":"2024-09-21T14:08:47.765045+0000","last_became_peered":"2024-09-21T14:08:47.765045+0000","last_unstale":"2024-09-21T14:10:00.275035+0000","last_undegraded":"2024-09-21T14:10:00.275035+0000","last_fullsized":"2024-09-21T14:10:00.275035+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:08:11.166966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:08:11.166966+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:10:29.371941+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"516'30","reported_seq":1099,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.276185+0000","last_change":"2024-09-21T14:08:49.103034+0000","last_active":"2024-09-21T14:10:00.276185+0000","last_peered":"2024-09-21T14:10:00.276185+0000","last_clean":"2024-09-21T14:10:00.276185+0000","last_became_active":"2024-09-21T13:58:17.765427+0000","last_became_peered":"2024-09-21T13:58:17.765427+0000","last_unstale":"2024-09-21T14:10:00.276185+0000","last_undegraded":"2024-09-21T14:10:00.276185+0000","last_fullsized":"2024-09-21T14:10:00.276185+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'20","last_scrub_stamp":"2024-09-21T14:08:49.102928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:08:49.102928+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:00.929150+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":955,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.270270+0000","last_change":"2024-09-21T14:09:35.324158+0000","last_active":"2024-09-21T14:10:00.270270+0000","last_peered":"2024-09-21T14:10:00.270270+0000","last_clean":"2024-09-21T14:10:00.270270+0000","last_became_active":"2024-09-21T14:09:00.258570+0000","last_became_peered":"2024-09-21T14:09:00.258570+0000","last_unstale":"2024-09-21T14:10:00.270270+0000","last_undegraded":"2024-09-21T14:10:00.270270+0000","last_fullsized":"2024-09-21T14:10:00.270270+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:09:35.324097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:09:35.324097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:11:02.981711+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"346'14","reported_seq":1151,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.278430+0000","last_change":"2024-09-21T14:09:39.453380+0000","last_active":"2024-09-21T14:10:00.278430+0000","last_peered":"2024-09-21T14:10:00.278430+0000","last_clean":"2024-09-21T14:10:00.278430+0000","last_became_active":"2024-09-21T14:01:41.186564+0000","last_became_peered":"2024-09-21T14:01:41.186564+0000","last_unstale":"2024-09-21T14:10:00.278430+0000","last_undegraded":"2024-09-21T14:10:00.278430+0000","last_fullsized":"2024-09-21T14:10:00.278430+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'14","last_scrub_stamp":"2024-09-21T14:09:39.453307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:09:39.453307+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:46.016652+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"68'1","reported_seq":1012,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.275744+0000","last_change":"2024-09-21T14:08:54.081358+0000","last_active":"2024-09-21T14:10:00.275744+0000","last_peered":"2024-09-21T14:10:00.275744+0000","last_clean":"2024-09-21T14:10:00.275744+0000","last_became_active":"2024-09-21T13:52:04.313210+0000","last_became_peered":"2024-09-21T13:52:04.313210+0000","last_unstale":"2024-09-21T14:10:00.275744+0000","last_undegraded":"2024-09-21T14:10:00.275744+0000","last_fullsized":"2024-09-21T14:10:00.275744+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T14:08:54.081157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:08:54.081157+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:12.280674+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.4","version":"536'13973","reported_seq":14140,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:04.037383+0000","last_change":"2024-09-21T14:08:04.295353+0000","last_active":"2024-09-21T14:10:04.037383+0000","last_peered":"2024-09-21T14:10:04.037383+0000","last_clean":"2024-09-21T14:10:04.037383+0000","last_became_active":"2024-09-21T14:08:04.289424+0000","last_became_peered":"2024-09-21T14:08:04.289424+0000","last_unstale":"2024-09-21T14:10:04.037383+0000","last_undegraded":"2024-09-21T14:10:04.037383+0000","last_fullsized":"2024-09-21T14:10:04.037383+0000","mapping_epoch":491,"log_start":"506'9727","ondisk_log_start":"506'9727","created":461,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_clean_scrub_stamp":"2024-09-21T14:06:44.365469+0000","objects_scrubbed":0,"log_size":4246,"log_dups_size":0,"ondisk_log_size":4246,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105971712,"num_objects":1617,"num_object_clones":0,"num_object_copies":4851,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13973,"num_write_kb":498880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"75'79","reported_seq":62,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.275504+0000","last_change":"2024-09-21T14:09:00.257341+0000","last_active":"2024-09-21T14:10:00.275504+0000","last_peered":"2024-09-21T14:10:00.275504+0000","last_clean":"2024-09-21T14:10:00.275504+0000","last_became_active":"2024-09-21T14:09:00.255631+0000","last_became_peered":"2024-09-21T14:09:00.255631+0000","last_unstale":"2024-09-21T14:10:00.275504+0000","last_undegraded":"2024-09-21T14:10:00.275504+0000","last_fullsized":"2024-09-21T14:10:00.275504+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:08:12.129198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:08:12.129198+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:10:27.477410+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":10,"num_write":5,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"516'21","reported_seq":1022,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.288114+0000","last_change":"2024-09-21T14:09:53.482102+0000","last_active":"2024-09-21T14:10:05.288114+0000","last_peered":"2024-09-21T14:10:05.288114+0000","last_clean":"2024-09-21T14:10:05.288114+0000","last_became_active":"2024-09-21T14:09:53.480679+0000","last_became_peered":"2024-09-21T14:09:53.480679+0000","last_unstale":"2024-09-21T14:10:05.288114+0000","last_undegraded":"2024-09-21T14:10:05.288114+0000","last_fullsized":"2024-09-21T14:10:05.288114+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'15","last_scrub_stamp":"2024-09-21T14:07:06.252672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:07:06.252672+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":980,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.275265+0000","last_change":"2024-09-21T14:08:14.410591+0000","last_active":"2024-09-21T14:10:05.275265+0000","last_peered":"2024-09-21T14:10:05.275265+0000","last_clean":"2024-09-21T14:10:05.275265+0000","last_became_active":"2024-09-21T14:08:14.410232+0000","last_became_peered":"2024-09-21T14:08:14.410232+0000","last_unstale":"2024-09-21T14:10:05.275265+0000","last_undegraded":"2024-09-21T14:10:05.275265+0000","last_fullsized":"2024-09-21T14:10:05.275265+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:22.402182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:05:22.402182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:26.315412+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":965,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.276860+0000","last_change":"2024-09-21T14:08:15.110491+0000","last_active":"2024-09-21T14:10:00.276860+0000","last_peered":"2024-09-21T14:10:00.276860+0000","last_clean":"2024-09-21T14:10:00.276860+0000","last_became_active":"2024-09-21T14:08:14.407697+0000","last_became_peered":"2024-09-21T14:08:14.407697+0000","last_unstale":"2024-09-21T14:10:00.276860+0000","last_undegraded":"2024-09-21T14:10:00.276860+0000","last_fullsized":"2024-09-21T14:10:00.276860+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:08:15.110439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:08:15.110439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":699,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.275573+0000","last_change":"2024-09-21T14:09:53.480659+0000","last_active":"2024-09-21T14:10:00.275573+0000","last_peered":"2024-09-21T14:10:00.275573+0000","last_clean":"2024-09-21T14:10:00.275573+0000","last_became_active":"2024-09-21T14:09:53.480342+0000","last_became_peered":"2024-09-21T14:09:53.480342+0000","last_unstale":"2024-09-21T14:10:00.275573+0000","last_undegraded":"2024-09-21T14:10:00.275573+0000","last_fullsized":"2024-09-21T14:10:00.275573+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:08:10.164899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:08:10.164899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"538'14130","reported_seq":14367,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:06.066347+0000","last_change":"2024-09-21T14:08:56.098631+0000","last_active":"2024-09-21T14:10:06.066347+0000","last_peered":"2024-09-21T14:10:06.066347+0000","last_clean":"2024-09-21T14:10:06.066347+0000","last_became_active":"2024-09-21T14:08:04.300912+0000","last_became_peered":"2024-09-21T14:08:04.300912+0000","last_unstale":"2024-09-21T14:10:06.066347+0000","last_undegraded":"2024-09-21T14:10:06.066347+0000","last_fullsized":"2024-09-21T14:10:06.066347+0000","mapping_epoch":491,"log_start":"506'9638","ondisk_log_start":"506'9638","created":461,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"508'9825","last_scrub_stamp":"2024-09-21T14:08:56.098547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_clean_scrub_stamp":"2024-09-21T14:08:56.098547+0000","objects_scrubbed":5804,"log_size":4492,"log_dups_size":0,"ondisk_log_size":4492,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:15.273583+0000","scrub_duration":781,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97648640,"num_objects":1490,"num_object_clones":0,"num_object_copies":4470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14130,"num_write_kb":499840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.f","version":"75'79","reported_seq":1073,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.268849+0000","last_change":"2024-09-21T14:07:01.780758+0000","last_active":"2024-09-21T14:10:00.268849+0000","last_peered":"2024-09-21T14:10:00.268849+0000","last_clean":"2024-09-21T14:10:00.268849+0000","last_became_active":"2024-09-21T14:05:41.327757+0000","last_became_peered":"2024-09-21T14:05:41.327757+0000","last_unstale":"2024-09-21T14:10:00.268849+0000","last_undegraded":"2024-09-21T14:10:00.268849+0000","last_fullsized":"2024-09-21T14:10:00.268849+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:07:01.780652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:07:01.780652+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"516'51","reported_seq":663,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.269727+0000","last_change":"2024-09-21T14:05:38.897778+0000","last_active":"2024-09-21T14:10:00.269727+0000","last_peered":"2024-09-21T14:10:00.269727+0000","last_clean":"2024-09-21T14:10:00.269727+0000","last_became_active":"2024-09-21T13:59:32.204184+0000","last_became_peered":"2024-09-21T13:59:32.204184+0000","last_unstale":"2024-09-21T14:10:00.269727+0000","last_undegraded":"2024-09-21T14:10:00.269727+0000","last_fullsized":"2024-09-21T14:10:00.269727+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'35","last_scrub_stamp":"2024-09-21T14:05:38.897722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:05:38.897722+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":703,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.275162+0000","last_change":"2024-09-21T14:09:47.310450+0000","last_active":"2024-09-21T14:10:05.275162+0000","last_peered":"2024-09-21T14:10:05.275162+0000","last_clean":"2024-09-21T14:10:05.275162+0000","last_became_active":"2024-09-21T14:08:04.300236+0000","last_became_peered":"2024-09-21T14:08:04.300236+0000","last_unstale":"2024-09-21T14:10:05.275162+0000","last_undegraded":"2024-09-21T14:10:05.275162+0000","last_fullsized":"2024-09-21T14:10:05.275162+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:09:47.310387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:09:47.310387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:11:09.129475+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":971,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.276994+0000","last_change":"2024-09-21T14:08:12.134422+0000","last_active":"2024-09-21T14:10:00.276994+0000","last_peered":"2024-09-21T14:10:00.276994+0000","last_clean":"2024-09-21T14:10:00.276994+0000","last_became_active":"2024-09-21T14:01:41.181911+0000","last_became_peered":"2024-09-21T14:01:41.181911+0000","last_unstale":"2024-09-21T14:10:00.276994+0000","last_undegraded":"2024-09-21T14:10:00.276994+0000","last_fullsized":"2024-09-21T14:10:00.276994+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:08:12.134107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:08:12.134107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":554,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.274087+0000","last_change":"2024-09-21T14:09:34.400021+0000","last_active":"2024-09-21T14:10:00.274087+0000","last_peered":"2024-09-21T14:10:00.274087+0000","last_clean":"2024-09-21T14:10:00.274087+0000","last_became_active":"2024-09-21T14:08:04.290897+0000","last_became_peered":"2024-09-21T14:08:04.290897+0000","last_unstale":"2024-09-21T14:10:00.274087+0000","last_undegraded":"2024-09-21T14:10:00.274087+0000","last_fullsized":"2024-09-21T14:10:00.274087+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:09:34.399450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:09:34.399450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:11:04.023314+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.6","version":"536'13808","reported_seq":23097,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:03.317193+0000","last_change":"2024-09-21T14:09:53.516723+0000","last_active":"2024-09-21T14:10:03.317193+0000","last_peered":"2024-09-21T14:10:03.317193+0000","last_clean":"2024-09-21T14:10:03.317193+0000","last_became_active":"2024-09-21T14:09:53.479387+0000","last_became_peered":"2024-09-21T14:09:53.479387+0000","last_unstale":"2024-09-21T14:10:03.317193+0000","last_undegraded":"2024-09-21T14:10:03.317193+0000","last_fullsized":"2024-09-21T14:10:03.317193+0000","mapping_epoch":531,"log_start":"502'8655","ondisk_log_start":"502'8655","created":461,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"492'6638","last_scrub_stamp":"2024-09-21T14:08:06.099093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_clean_scrub_stamp":"2024-09-21T14:08:06.099093+0000","objects_scrubbed":6553,"log_size":5153,"log_dups_size":0,"ondisk_log_size":5153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1978,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108920832,"num_objects":1662,"num_object_clones":0,"num_object_copies":4986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13808,"num_write_kb":495040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4514,"num_bytes_recovered":295698432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"75'79","reported_seq":304,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.268436+0000","last_change":"2024-09-21T14:09:53.481166+0000","last_active":"2024-09-21T14:10:00.268436+0000","last_peered":"2024-09-21T14:10:00.268436+0000","last_clean":"2024-09-21T14:10:00.268436+0000","last_became_active":"2024-09-21T14:09:53.480800+0000","last_became_peered":"2024-09-21T14:09:53.480800+0000","last_unstale":"2024-09-21T14:10:00.268436+0000","last_undegraded":"2024-09-21T14:10:00.268436+0000","last_fullsized":"2024-09-21T14:10:00.268436+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:08:54.873214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:08:54.873214+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T14:10:57.966823+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109227,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"516'24","reported_seq":1092,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.276125+0000","last_change":"2024-09-21T14:09:49.533796+0000","last_active":"2024-09-21T14:10:00.276125+0000","last_peered":"2024-09-21T14:10:00.276125+0000","last_clean":"2024-09-21T14:10:00.276125+0000","last_became_active":"2024-09-21T14:08:14.411011+0000","last_became_peered":"2024-09-21T14:08:14.411011+0000","last_unstale":"2024-09-21T14:10:00.276125+0000","last_undegraded":"2024-09-21T14:10:00.276125+0000","last_fullsized":"2024-09-21T14:10:00.276125+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'24","last_scrub_stamp":"2024-09-21T14:09:49.533719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:09:49.533719+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:11:02.731079+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":62,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.279277+0000","last_change":"2024-09-21T14:09:00.256983+0000","last_active":"2024-09-21T14:10:00.279277+0000","last_peered":"2024-09-21T14:10:00.279277+0000","last_clean":"2024-09-21T14:10:00.279277+0000","last_became_active":"2024-09-21T14:09:00.256317+0000","last_became_peered":"2024-09-21T14:09:00.256317+0000","last_unstale":"2024-09-21T14:10:00.279277+0000","last_undegraded":"2024-09-21T14:10:00.279277+0000","last_fullsized":"2024-09-21T14:10:00.279277+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:07:35.401092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:07:35.401092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":933,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.273965+0000","last_change":"2024-09-21T14:09:32.433914+0000","last_active":"2024-09-21T14:10:00.273965+0000","last_peered":"2024-09-21T14:10:00.273965+0000","last_clean":"2024-09-21T14:10:00.273965+0000","last_became_active":"2024-09-21T13:52:04.322147+0000","last_became_peered":"2024-09-21T13:52:04.322147+0000","last_unstale":"2024-09-21T14:10:00.273965+0000","last_undegraded":"2024-09-21T14:10:00.273965+0000","last_fullsized":"2024-09-21T14:10:00.273965+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:09:32.433855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:09:32.433855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:53.688911+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":956,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.275595+0000","last_change":"2024-09-21T14:09:49.202617+0000","last_active":"2024-09-21T14:10:00.275595+0000","last_peered":"2024-09-21T14:10:00.275595+0000","last_clean":"2024-09-21T14:10:00.275595+0000","last_became_active":"2024-09-21T14:05:30.243052+0000","last_became_peered":"2024-09-21T14:05:30.243052+0000","last_unstale":"2024-09-21T14:10:00.275595+0000","last_undegraded":"2024-09-21T14:10:00.275595+0000","last_fullsized":"2024-09-21T14:10:00.275595+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:09:49.202551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:09:49.202551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:11:18.633889+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"538'13719","reported_seq":13885,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:06.064835+0000","last_change":"2024-09-21T14:06:45.307096+0000","last_active":"2024-09-21T14:10:06.064835+0000","last_peered":"2024-09-21T14:10:06.064835+0000","last_clean":"2024-09-21T14:10:06.064835+0000","last_became_active":"2024-09-21T14:06:45.306744+0000","last_became_peered":"2024-09-21T14:06:45.306744+0000","last_unstale":"2024-09-21T14:10:06.064835+0000","last_undegraded":"2024-09-21T14:10:06.064835+0000","last_fullsized":"2024-09-21T14:10:06.064835+0000","mapping_epoch":461,"log_start":"506'9166","ondisk_log_start":"506'9166","created":461,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T14:06:44.365469+0000","last_clean_scrub_stamp":"2024-09-21T14:06:44.365469+0000","objects_scrubbed":0,"log_size":4553,"log_dups_size":0,"ondisk_log_size":4553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:06.328251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93519872,"num_objects":1427,"num_object_clones":0,"num_object_copies":4281,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13719,"num_write_kb":484672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"75'79","reported_seq":378,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.275852+0000","last_change":"2024-09-21T14:08:05.154262+0000","last_active":"2024-09-21T14:10:00.275852+0000","last_peered":"2024-09-21T14:10:00.275852+0000","last_clean":"2024-09-21T14:10:00.275852+0000","last_became_active":"2024-09-21T14:08:04.296887+0000","last_became_peered":"2024-09-21T14:08:04.296887+0000","last_unstale":"2024-09-21T14:10:00.275852+0000","last_undegraded":"2024-09-21T14:10:00.275852+0000","last_fullsized":"2024-09-21T14:10:00.275852+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:08:05.154200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:08:05.154200+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"516'39","reported_seq":1083,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.274091+0000","last_change":"2024-09-21T14:09:00.330336+0000","last_active":"2024-09-21T14:10:05.274091+0000","last_peered":"2024-09-21T14:10:05.274091+0000","last_clean":"2024-09-21T14:10:05.274091+0000","last_became_active":"2024-09-21T14:01:41.192481+0000","last_became_peered":"2024-09-21T14:01:41.192481+0000","last_unstale":"2024-09-21T14:10:05.274091+0000","last_undegraded":"2024-09-21T14:10:05.274091+0000","last_fullsized":"2024-09-21T14:10:05.274091+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'27","last_scrub_stamp":"2024-09-21T14:09:00.330276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:09:00.330276+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:06.380654+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":187,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.277978+0000","last_change":"2024-09-21T14:09:00.270585+0000","last_active":"2024-09-21T14:10:00.277978+0000","last_peered":"2024-09-21T14:10:00.277978+0000","last_clean":"2024-09-21T14:10:00.277978+0000","last_became_active":"2024-09-21T14:09:00.256222+0000","last_became_peered":"2024-09-21T14:09:00.256222+0000","last_unstale":"2024-09-21T14:10:00.277978+0000","last_undegraded":"2024-09-21T14:10:00.277978+0000","last_fullsized":"2024-09-21T14:10:00.277978+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T14:08:20.092342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:08:20.092342+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"492'15","reported_seq":765,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.272032+0000","last_change":"2024-09-21T14:08:04.299992+0000","last_active":"2024-09-21T14:10:00.272032+0000","last_peered":"2024-09-21T14:10:00.272032+0000","last_clean":"2024-09-21T14:10:00.272032+0000","last_became_active":"2024-09-21T14:08:04.299624+0000","last_became_peered":"2024-09-21T14:08:04.299624+0000","last_unstale":"2024-09-21T14:10:00.272032+0000","last_undegraded":"2024-09-21T14:10:00.272032+0000","last_fullsized":"2024-09-21T14:10:00.272032+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'14","last_scrub_stamp":"2024-09-21T14:07:36.427806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:07:36.427806+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1013,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.274503+0000","last_change":"2024-09-21T14:09:38.274170+0000","last_active":"2024-09-21T14:10:05.274503+0000","last_peered":"2024-09-21T14:10:05.274503+0000","last_clean":"2024-09-21T14:10:05.274503+0000","last_became_active":"2024-09-21T14:01:41.246106+0000","last_became_peered":"2024-09-21T14:01:41.246106+0000","last_unstale":"2024-09-21T14:10:05.274503+0000","last_undegraded":"2024-09-21T14:10:05.274503+0000","last_fullsized":"2024-09-21T14:10:05.274503+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:09:38.274035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:09:38.274035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:43.801324+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"75'79","reported_seq":226,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.274315+0000","last_change":"2024-09-21T14:07:34.549041+0000","last_active":"2024-09-21T14:10:05.274315+0000","last_peered":"2024-09-21T14:10:05.274315+0000","last_clean":"2024-09-21T14:10:05.274315+0000","last_became_active":"2024-09-21T14:05:41.352416+0000","last_became_peered":"2024-09-21T14:05:41.352416+0000","last_unstale":"2024-09-21T14:10:05.274315+0000","last_undegraded":"2024-09-21T14:10:05.274315+0000","last_fullsized":"2024-09-21T14:10:05.274315+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":5,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:07:34.548995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:07:34.548995+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:27.275380+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":597,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.269037+0000","last_change":"2024-09-21T14:09:00.259096+0000","last_active":"2024-09-21T14:10:00.269037+0000","last_peered":"2024-09-21T14:10:00.269037+0000","last_clean":"2024-09-21T14:10:00.269037+0000","last_became_active":"2024-09-21T14:09:00.257598+0000","last_became_peered":"2024-09-21T14:09:00.257598+0000","last_unstale":"2024-09-21T14:10:00.269037+0000","last_undegraded":"2024-09-21T14:10:00.269037+0000","last_fullsized":"2024-09-21T14:10:00.269037+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:05:53.922394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:05:53.922394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"516'18","reported_seq":549,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.287825+0000","last_change":"2024-09-21T14:08:14.430792+0000","last_active":"2024-09-21T14:10:05.287825+0000","last_peered":"2024-09-21T14:10:05.287825+0000","last_clean":"2024-09-21T14:10:05.287825+0000","last_became_active":"2024-09-21T14:08:14.413439+0000","last_became_peered":"2024-09-21T14:08:14.413439+0000","last_unstale":"2024-09-21T14:10:05.287825+0000","last_undegraded":"2024-09-21T14:10:05.287825+0000","last_fullsized":"2024-09-21T14:10:05.287825+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'12","last_scrub_stamp":"2024-09-21T14:07:05.134142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:07:05.134142+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":22,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":936,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.275214+0000","last_change":"2024-09-21T14:07:13.246488+0000","last_active":"2024-09-21T14:10:05.275214+0000","last_peered":"2024-09-21T14:10:05.275214+0000","last_clean":"2024-09-21T14:10:05.275214+0000","last_became_active":"2024-09-21T14:07:13.246170+0000","last_became_peered":"2024-09-21T14:07:13.246170+0000","last_unstale":"2024-09-21T14:10:05.275214+0000","last_undegraded":"2024-09-21T14:10:05.275214+0000","last_fullsized":"2024-09-21T14:10:05.275214+0000","mapping_epoch":471,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:58.528683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:06:58.528683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:15.368188+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":611,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.270078+0000","last_change":"2024-09-21T14:08:13.932501+0000","last_active":"2024-09-21T14:10:00.270078+0000","last_peered":"2024-09-21T14:10:00.270078+0000","last_clean":"2024-09-21T14:10:00.270078+0000","last_became_active":"2024-09-21T14:08:04.300053+0000","last_became_peered":"2024-09-21T14:08:04.300053+0000","last_unstale":"2024-09-21T14:10:00.270078+0000","last_undegraded":"2024-09-21T14:10:00.270078+0000","last_fullsized":"2024-09-21T14:10:00.270078+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:08:13.932452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:08:13.932452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.13","version":"75'79","reported_seq":509,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.287564+0000","last_change":"2024-09-21T14:09:35.083536+0000","last_active":"2024-09-21T14:10:05.287564+0000","last_peered":"2024-09-21T14:10:05.287564+0000","last_clean":"2024-09-21T14:10:05.287564+0000","last_became_active":"2024-09-21T14:09:00.253144+0000","last_became_peered":"2024-09-21T14:09:00.253144+0000","last_unstale":"2024-09-21T14:10:05.287564+0000","last_undegraded":"2024-09-21T14:10:05.287564+0000","last_fullsized":"2024-09-21T14:10:05.287564+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":5,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:09:35.083450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:09:35.083450+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:43.178460+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"516'21","reported_seq":60,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.267223+0000","last_change":"2024-09-21T14:09:53.475612+0000","last_active":"2024-09-21T14:10:00.267223+0000","last_peered":"2024-09-21T14:10:00.267223+0000","last_clean":"2024-09-21T14:10:00.267223+0000","last_became_active":"2024-09-21T14:09:53.475166+0000","last_became_peered":"2024-09-21T14:09:53.475166+0000","last_unstale":"2024-09-21T14:10:00.267223+0000","last_undegraded":"2024-09-21T14:10:00.267223+0000","last_fullsized":"2024-09-21T14:10:00.267223+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'15","last_scrub_stamp":"2024-09-21T14:05:55.357788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:05:55.357788+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:58.193844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"61'1","reported_seq":173,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.275964+0000","last_change":"2024-09-21T14:08:58.071986+0000","last_active":"2024-09-21T14:10:00.275964+0000","last_peered":"2024-09-21T14:10:00.275964+0000","last_clean":"2024-09-21T14:10:00.275964+0000","last_became_active":"2024-09-21T14:08:04.301462+0000","last_became_peered":"2024-09-21T14:08:04.301462+0000","last_unstale":"2024-09-21T14:10:00.275964+0000","last_undegraded":"2024-09-21T14:10:00.275964+0000","last_fullsized":"2024-09-21T14:10:00.275964+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T14:08:58.071936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:08:58.071936+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":506,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.287324+0000","last_change":"2024-09-21T14:08:14.420307+0000","last_active":"2024-09-21T14:10:05.287324+0000","last_peered":"2024-09-21T14:10:05.287324+0000","last_clean":"2024-09-21T14:10:05.287324+0000","last_became_active":"2024-09-21T14:08:14.419869+0000","last_became_peered":"2024-09-21T14:08:14.419869+0000","last_unstale":"2024-09-21T14:10:05.287324+0000","last_undegraded":"2024-09-21T14:10:05.287324+0000","last_fullsized":"2024-09-21T14:10:05.287324+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:32.413388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:06:32.413388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:56.164632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":974,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.277474+0000","last_change":"2024-09-21T14:08:13.097305+0000","last_active":"2024-09-21T14:10:00.277474+0000","last_peered":"2024-09-21T14:10:00.277474+0000","last_clean":"2024-09-21T14:10:00.277474+0000","last_became_active":"2024-09-21T13:59:32.205803+0000","last_became_peered":"2024-09-21T13:59:32.205803+0000","last_unstale":"2024-09-21T14:10:00.277474+0000","last_undegraded":"2024-09-21T14:10:00.277474+0000","last_fullsized":"2024-09-21T14:10:00.277474+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:08:13.096578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:08:13.096578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"75'79","reported_seq":226,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.273766+0000","last_change":"2024-09-21T14:09:42.199233+0000","last_active":"2024-09-21T14:10:05.273766+0000","last_peered":"2024-09-21T14:10:05.273766+0000","last_clean":"2024-09-21T14:10:05.273766+0000","last_became_active":"2024-09-21T14:05:41.325618+0000","last_became_peered":"2024-09-21T14:05:41.325618+0000","last_unstale":"2024-09-21T14:10:05.273766+0000","last_undegraded":"2024-09-21T14:10:05.273766+0000","last_fullsized":"2024-09-21T14:10:05.273766+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":5,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:09:42.199185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:09:42.199185+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:11:06.293307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"516'21","reported_seq":975,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.275331+0000","last_change":"2024-09-21T14:07:33.588871+0000","last_active":"2024-09-21T14:10:05.275331+0000","last_peered":"2024-09-21T14:10:05.275331+0000","last_clean":"2024-09-21T14:10:05.275331+0000","last_became_active":"2024-09-21T14:01:41.187904+0000","last_became_peered":"2024-09-21T14:01:41.187904+0000","last_unstale":"2024-09-21T14:10:05.275331+0000","last_undegraded":"2024-09-21T14:10:05.275331+0000","last_fullsized":"2024-09-21T14:10:05.275331+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'15","last_scrub_stamp":"2024-09-21T14:07:33.588707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:07:33.588707+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:19.342298+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":542,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.284323+0000","last_change":"2024-09-21T14:09:53.477861+0000","last_active":"2024-09-21T14:10:05.284323+0000","last_peered":"2024-09-21T14:10:05.284323+0000","last_clean":"2024-09-21T14:10:05.284323+0000","last_became_active":"2024-09-21T14:09:53.477557+0000","last_became_peered":"2024-09-21T14:09:53.477557+0000","last_unstale":"2024-09-21T14:10:05.284323+0000","last_undegraded":"2024-09-21T14:10:05.284323+0000","last_fullsized":"2024-09-21T14:10:05.284323+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:09:49.069356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:09:49.069356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:49.870289+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.14","version":"498'15","reported_seq":326,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.277736+0000","last_change":"2024-09-21T14:09:41.360673+0000","last_active":"2024-09-21T14:10:00.277736+0000","last_peered":"2024-09-21T14:10:00.277736+0000","last_clean":"2024-09-21T14:10:00.277736+0000","last_became_active":"2024-09-21T14:08:14.425036+0000","last_became_peered":"2024-09-21T14:08:14.425036+0000","last_unstale":"2024-09-21T14:10:00.277736+0000","last_undegraded":"2024-09-21T14:10:00.277736+0000","last_fullsized":"2024-09-21T14:10:00.277736+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"498'15","last_scrub_stamp":"2024-09-21T14:09:41.360428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:09:41.360428+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:49.900594+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1026,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.277726+0000","last_change":"2024-09-21T14:08:57.105621+0000","last_active":"2024-09-21T14:10:00.277726+0000","last_peered":"2024-09-21T14:10:00.277726+0000","last_clean":"2024-09-21T14:10:00.277726+0000","last_became_active":"2024-09-21T14:01:41.200113+0000","last_became_peered":"2024-09-21T14:01:41.200113+0000","last_unstale":"2024-09-21T14:10:00.277726+0000","last_undegraded":"2024-09-21T14:10:00.277726+0000","last_fullsized":"2024-09-21T14:10:00.277726+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:08:57.105404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:08:57.105404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:11.548213+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"75'79","reported_seq":405,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.273917+0000","last_change":"2024-09-21T14:09:53.472418+0000","last_active":"2024-09-21T14:10:05.273917+0000","last_peered":"2024-09-21T14:10:05.273917+0000","last_clean":"2024-09-21T14:10:05.273917+0000","last_became_active":"2024-09-21T14:09:53.472067+0000","last_became_peered":"2024-09-21T14:09:53.472067+0000","last_unstale":"2024-09-21T14:10:05.273917+0000","last_undegraded":"2024-09-21T14:10:05.273917+0000","last_fullsized":"2024-09-21T14:10:05.273917+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":5,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:09:00.912443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:09:00.912443+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:18.669731+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"516'30","reported_seq":982,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.266819+0000","last_change":"2024-09-21T14:07:40.228945+0000","last_active":"2024-09-21T14:10:00.266819+0000","last_peered":"2024-09-21T14:10:00.266819+0000","last_clean":"2024-09-21T14:10:00.266819+0000","last_became_active":"2024-09-21T13:59:32.208332+0000","last_became_peered":"2024-09-21T13:59:32.208332+0000","last_unstale":"2024-09-21T14:10:00.266819+0000","last_undegraded":"2024-09-21T14:10:00.266819+0000","last_fullsized":"2024-09-21T14:10:00.266819+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'20","last_scrub_stamp":"2024-09-21T14:07:40.228846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:07:40.228846+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:08.214882+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":564,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.276529+0000","last_change":"2024-09-21T14:08:06.140232+0000","last_active":"2024-09-21T14:10:00.276529+0000","last_peered":"2024-09-21T14:10:00.276529+0000","last_clean":"2024-09-21T14:10:00.276529+0000","last_became_active":"2024-09-21T14:08:04.294307+0000","last_became_peered":"2024-09-21T14:08:04.294307+0000","last_unstale":"2024-09-21T14:10:00.276529+0000","last_undegraded":"2024-09-21T14:10:00.276529+0000","last_fullsized":"2024-09-21T14:10:00.276529+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:08:06.140163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:08:06.140163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"346'10","reported_seq":1172,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.275244+0000","last_change":"2024-09-21T14:08:02.389592+0000","last_active":"2024-09-21T14:10:05.275244+0000","last_peered":"2024-09-21T14:10:05.275244+0000","last_clean":"2024-09-21T14:10:05.275244+0000","last_became_active":"2024-09-21T14:01:41.206303+0000","last_became_peered":"2024-09-21T14:01:41.206303+0000","last_unstale":"2024-09-21T14:10:05.275244+0000","last_undegraded":"2024-09-21T14:10:05.275244+0000","last_fullsized":"2024-09-21T14:10:05.275244+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'10","last_scrub_stamp":"2024-09-21T14:08:02.389168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:08:02.389168+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":662,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.275652+0000","last_change":"2024-09-21T14:09:53.477156+0000","last_active":"2024-09-21T14:10:00.275652+0000","last_peered":"2024-09-21T14:10:00.275652+0000","last_clean":"2024-09-21T14:10:00.275652+0000","last_became_active":"2024-09-21T14:09:53.476628+0000","last_became_peered":"2024-09-21T14:09:53.476628+0000","last_unstale":"2024-09-21T14:10:00.275652+0000","last_undegraded":"2024-09-21T14:10:00.275652+0000","last_fullsized":"2024-09-21T14:10:00.275652+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:26.530508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:06:26.530508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:55.518638+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"516'27","reported_seq":1094,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.276541+0000","last_change":"2024-09-21T14:08:11.081708+0000","last_active":"2024-09-21T14:10:00.276541+0000","last_peered":"2024-09-21T14:10:00.276541+0000","last_clean":"2024-09-21T14:10:00.276541+0000","last_became_active":"2024-09-21T13:58:17.771497+0000","last_became_peered":"2024-09-21T13:58:17.771497+0000","last_unstale":"2024-09-21T14:10:00.276541+0000","last_undegraded":"2024-09-21T14:10:00.276541+0000","last_fullsized":"2024-09-21T14:10:00.276541+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'19","last_scrub_stamp":"2024-09-21T14:08:11.081620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:08:11.081620+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":239,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.270347+0000","last_change":"2024-09-21T14:09:46.721810+0000","last_active":"2024-09-21T14:10:00.270347+0000","last_peered":"2024-09-21T14:10:00.270347+0000","last_clean":"2024-09-21T14:10:00.270347+0000","last_became_active":"2024-09-21T14:08:04.289905+0000","last_became_peered":"2024-09-21T14:08:04.289905+0000","last_unstale":"2024-09-21T14:10:00.270347+0000","last_undegraded":"2024-09-21T14:10:00.270347+0000","last_fullsized":"2024-09-21T14:10:00.270347+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:09:46.721752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:09:46.721752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:49.152663+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1043,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.285242+0000","last_change":"2024-09-21T14:08:14.415030+0000","last_active":"2024-09-21T14:10:05.285242+0000","last_peered":"2024-09-21T14:10:05.285242+0000","last_clean":"2024-09-21T14:10:05.285242+0000","last_became_active":"2024-09-21T14:08:14.414701+0000","last_became_peered":"2024-09-21T14:08:14.414701+0000","last_unstale":"2024-09-21T14:10:05.285242+0000","last_undegraded":"2024-09-21T14:10:05.285242+0000","last_fullsized":"2024-09-21T14:10:05.285242+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:51.607390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:06:51.607390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":717,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.274008+0000","last_change":"2024-09-21T14:09:53.477865+0000","last_active":"2024-09-21T14:10:05.274008+0000","last_peered":"2024-09-21T14:10:05.274008+0000","last_clean":"2024-09-21T14:10:05.274008+0000","last_became_active":"2024-09-21T14:09:53.477172+0000","last_became_peered":"2024-09-21T14:09:53.477172+0000","last_unstale":"2024-09-21T14:10:05.274008+0000","last_undegraded":"2024-09-21T14:10:05.274008+0000","last_fullsized":"2024-09-21T14:10:05.274008+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:08:04.070079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:08:04.070079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"516'21","reported_seq":1105,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.285417+0000","last_change":"2024-09-21T14:07:46.841902+0000","last_active":"2024-09-21T14:10:05.285417+0000","last_peered":"2024-09-21T14:10:05.285417+0000","last_clean":"2024-09-21T14:10:05.285417+0000","last_became_active":"2024-09-21T13:58:17.764120+0000","last_became_peered":"2024-09-21T13:58:17.764120+0000","last_unstale":"2024-09-21T14:10:05.285417+0000","last_undegraded":"2024-09-21T14:10:05.285417+0000","last_fullsized":"2024-09-21T14:10:05.285417+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'15","last_scrub_stamp":"2024-09-21T14:07:46.841824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:07:46.841824+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":568,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.266998+0000","last_change":"2024-09-21T14:09:02.345725+0000","last_active":"2024-09-21T14:10:00.266998+0000","last_peered":"2024-09-21T14:10:00.266998+0000","last_clean":"2024-09-21T14:10:00.266998+0000","last_became_active":"2024-09-21T14:08:04.294705+0000","last_became_peered":"2024-09-21T14:08:04.294705+0000","last_unstale":"2024-09-21T14:10:00.266998+0000","last_undegraded":"2024-09-21T14:10:00.266998+0000","last_fullsized":"2024-09-21T14:10:00.266998+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:09:02.345643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:09:02.345643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:25.800683+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":907,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.270152+0000","last_change":"2024-09-21T14:08:58.343590+0000","last_active":"2024-09-21T14:10:00.270152+0000","last_peered":"2024-09-21T14:10:00.270152+0000","last_clean":"2024-09-21T14:10:00.270152+0000","last_became_active":"2024-09-21T13:57:12.703441+0000","last_became_peered":"2024-09-21T13:57:12.703441+0000","last_unstale":"2024-09-21T14:10:00.270152+0000","last_undegraded":"2024-09-21T14:10:00.270152+0000","last_fullsized":"2024-09-21T14:10:00.270152+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:08:58.343510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:08:58.343510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:20.418715+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":595,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.273877+0000","last_change":"2024-09-21T14:09:53.476477+0000","last_active":"2024-09-21T14:10:05.273877+0000","last_peered":"2024-09-21T14:10:05.273877+0000","last_clean":"2024-09-21T14:10:05.273877+0000","last_became_active":"2024-09-21T14:09:53.476094+0000","last_became_peered":"2024-09-21T14:09:53.476094+0000","last_unstale":"2024-09-21T14:10:05.273877+0000","last_undegraded":"2024-09-21T14:10:05.273877+0000","last_fullsized":"2024-09-21T14:10:05.273877+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:08:56.900069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:08:56.900069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"75'79","reported_seq":238,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.277046+0000","last_change":"2024-09-21T14:09:46.369378+0000","last_active":"2024-09-21T14:10:00.277046+0000","last_peered":"2024-09-21T14:10:00.277046+0000","last_clean":"2024-09-21T14:10:00.277046+0000","last_became_active":"2024-09-21T14:09:00.260220+0000","last_became_peered":"2024-09-21T14:09:00.260220+0000","last_unstale":"2024-09-21T14:10:00.277046+0000","last_undegraded":"2024-09-21T14:10:00.277046+0000","last_fullsized":"2024-09-21T14:10:00.277046+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":5,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:09:46.369300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:09:46.369300+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:49.325247+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"516'27","reported_seq":1063,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.266089+0000","last_change":"2024-09-21T14:09:42.038913+0000","last_active":"2024-09-21T14:10:00.266089+0000","last_peered":"2024-09-21T14:10:00.266089+0000","last_clean":"2024-09-21T14:10:00.266089+0000","last_became_active":"2024-09-21T13:59:32.231567+0000","last_became_peered":"2024-09-21T13:59:32.231567+0000","last_unstale":"2024-09-21T14:10:00.266089+0000","last_undegraded":"2024-09-21T14:10:00.266089+0000","last_fullsized":"2024-09-21T14:10:00.266089+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'27","last_scrub_stamp":"2024-09-21T14:09:42.038860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:09:42.038860+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:53.000884+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":934,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.275290+0000","last_change":"2024-09-21T14:08:12.391143+0000","last_active":"2024-09-21T14:10:05.275290+0000","last_peered":"2024-09-21T14:10:05.275290+0000","last_clean":"2024-09-21T14:10:05.275290+0000","last_became_active":"2024-09-21T14:01:41.190760+0000","last_became_peered":"2024-09-21T14:01:41.190760+0000","last_unstale":"2024-09-21T14:10:05.275290+0000","last_undegraded":"2024-09-21T14:10:05.275290+0000","last_fullsized":"2024-09-21T14:10:05.275290+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:08:12.391069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:08:12.391069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":615,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.277194+0000","last_change":"2024-09-21T14:09:44.503828+0000","last_active":"2024-09-21T14:10:00.277194+0000","last_peered":"2024-09-21T14:10:00.277194+0000","last_clean":"2024-09-21T14:10:00.277194+0000","last_became_active":"2024-09-21T14:08:04.300081+0000","last_became_peered":"2024-09-21T14:08:04.300081+0000","last_unstale":"2024-09-21T14:10:00.277194+0000","last_undegraded":"2024-09-21T14:10:00.277194+0000","last_fullsized":"2024-09-21T14:10:00.277194+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:09:44.503767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:09:44.503767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:11:00.237152+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":777,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.268796+0000","last_change":"2024-09-21T14:09:53.481691+0000","last_active":"2024-09-21T14:10:00.268796+0000","last_peered":"2024-09-21T14:10:00.268796+0000","last_clean":"2024-09-21T14:10:00.268796+0000","last_became_active":"2024-09-21T14:09:53.481315+0000","last_became_peered":"2024-09-21T14:09:53.481315+0000","last_unstale":"2024-09-21T14:10:00.268796+0000","last_undegraded":"2024-09-21T14:10:00.268796+0000","last_fullsized":"2024-09-21T14:10:00.268796+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:08:14.971033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:08:14.971033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"516'12","reported_seq":1008,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.266080+0000","last_change":"2024-09-21T14:09:32.244996+0000","last_active":"2024-09-21T14:10:00.266080+0000","last_peered":"2024-09-21T14:10:00.266080+0000","last_clean":"2024-09-21T14:10:00.266080+0000","last_became_active":"2024-09-21T14:09:31.261004+0000","last_became_peered":"2024-09-21T14:09:31.261004+0000","last_unstale":"2024-09-21T14:10:00.266080+0000","last_undegraded":"2024-09-21T14:10:00.266080+0000","last_fullsized":"2024-09-21T14:10:00.266080+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'12","last_scrub_stamp":"2024-09-21T14:09:32.244942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:09:32.244942+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:40.994777+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":701,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.275615+0000","last_change":"2024-09-21T14:09:53.478859+0000","last_active":"2024-09-21T14:10:05.275615+0000","last_peered":"2024-09-21T14:10:05.275615+0000","last_clean":"2024-09-21T14:10:05.275615+0000","last_became_active":"2024-09-21T14:09:53.478351+0000","last_became_peered":"2024-09-21T14:09:53.478351+0000","last_unstale":"2024-09-21T14:10:05.275615+0000","last_undegraded":"2024-09-21T14:10:05.275615+0000","last_fullsized":"2024-09-21T14:10:05.275615+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:08:05.499112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:08:05.499112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:11:04.312013+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":606,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.269928+0000","last_change":"2024-09-21T14:09:00.259234+0000","last_active":"2024-09-21T14:10:00.269928+0000","last_peered":"2024-09-21T14:10:00.269928+0000","last_clean":"2024-09-21T14:10:00.269928+0000","last_became_active":"2024-09-21T14:09:00.257432+0000","last_became_peered":"2024-09-21T14:09:00.257432+0000","last_unstale":"2024-09-21T14:10:00.269928+0000","last_undegraded":"2024-09-21T14:10:00.269928+0000","last_fullsized":"2024-09-21T14:10:00.269928+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:12.938632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:06:12.938632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":499,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.276675+0000","last_change":"2024-09-21T14:08:04.295718+0000","last_active":"2024-09-21T14:10:00.276675+0000","last_peered":"2024-09-21T14:10:00.276675+0000","last_clean":"2024-09-21T14:10:00.276675+0000","last_became_active":"2024-09-21T14:08:04.295387+0000","last_became_peered":"2024-09-21T14:08:04.295387+0000","last_unstale":"2024-09-21T14:10:00.276675+0000","last_undegraded":"2024-09-21T14:10:00.276675+0000","last_fullsized":"2024-09-21T14:10:00.276675+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:07:47.329279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:07:47.329279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1012,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.284692+0000","last_change":"2024-09-21T14:07:40.894675+0000","last_active":"2024-09-21T14:10:05.284692+0000","last_peered":"2024-09-21T14:10:05.284692+0000","last_clean":"2024-09-21T14:10:05.284692+0000","last_became_active":"2024-09-21T14:01:41.187438+0000","last_became_peered":"2024-09-21T14:01:41.187438+0000","last_unstale":"2024-09-21T14:10:05.284692+0000","last_undegraded":"2024-09-21T14:10:05.284692+0000","last_fullsized":"2024-09-21T14:10:05.284692+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:07:40.894592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:07:40.894592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":745,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-09-21T14:10:05.287111+0000","last_change":"2024-09-21T14:09:53.479011+0000","last_active":"2024-09-21T14:10:05.287111+0000","last_peered":"2024-09-21T14:10:05.287111+0000","last_clean":"2024-09-21T14:10:05.287111+0000","last_became_active":"2024-09-21T14:09:53.478446+0000","last_became_peered":"2024-09-21T14:09:53.478446+0000","last_unstale":"2024-09-21T14:10:05.287111+0000","last_undegraded":"2024-09-21T14:10:05.287111+0000","last_fullsized":"2024-09-21T14:10:05.287111+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:06:47.440233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:06:47.440233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"516'27","reported_seq":628,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.268947+0000","last_change":"2024-09-21T14:05:55.994009+0000","last_active":"2024-09-21T14:10:00.268947+0000","last_peered":"2024-09-21T14:10:00.268947+0000","last_clean":"2024-09-21T14:10:00.268947+0000","last_became_active":"2024-09-21T14:01:41.194606+0000","last_became_peered":"2024-09-21T14:01:41.194606+0000","last_unstale":"2024-09-21T14:10:00.268947+0000","last_undegraded":"2024-09-21T14:10:00.268947+0000","last_fullsized":"2024-09-21T14:10:00.268947+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'19","last_scrub_stamp":"2024-09-21T14:05:55.993943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:05:55.993943+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":390,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-09-21T14:10:00.278842+0000","last_change":"2024-09-21T14:09:53.478081+0000","last_active":"2024-09-21T14:10:00.278842+0000","last_peered":"2024-09-21T14:10:00.278842+0000","last_clean":"2024-09-21T14:10:00.278842+0000","last_became_active":"2024-09-21T14:09:53.477597+0000","last_became_peered":"2024-09-21T14:09:53.477597+0000","last_unstale":"2024-09-21T14:10:00.278842+0000","last_undegraded":"2024-09-21T14:10:00.278842+0000","last_fullsized":"2024-09-21T14:10:00.278842+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:04:51.121361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:04:51.121361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:10:57.399761+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":1663696920,"num_objects":25387,"num_object_clones":0,"num_object_copies":76161,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":63,"num_objects_unfound":0,"num_objects_dirty":25387,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":221367,"num_write_kb":7896065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49156,"num_bytes_recovered":3221356544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6745370624,"data_stored":6745358408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82290,"ondisk_log_size":82290,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109,"ondisk_log_size":109,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1530,"num_read_kb":1287,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":270,"num_bytes_recovered":4692,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":974,"ondisk_log_size":974,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1223,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":193,"num_read_kb":188,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":452420,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":30,"num_write":15,"num_write_kb":222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":343193,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1659,"ondisk_log_size":1659,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":298,"seq":1279900254335,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1708304,"kb_used_data":1012864,"kb_used_omap":4,"kb_used_meta":695355,"kb_avail":92036848,"statfs":{"total":95995035648,"available":94245732352,"internally_reserved":0,"allocated":1037172736,"data_stored":1036734589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":712044200},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463175,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1623112,"kb_used_data":933688,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":92122040,"statfs":{"total":95995035648,"available":94332968960,"internally_reserved":0,"allocated":956096512,"data_stored":955667115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":705755590},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":345,"seq":1481763717222,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":671512,"kb_used_data":387780,"kb_used_omap":4,"kb_used_meta":283707,"kb_avail":93073640,"statfs":{"total":95995035648,"available":95307407360,"internally_reserved":0,"allocated":397086720,"data_stored":396661610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":290516643},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921044,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1264784,"kb_used_data":601856,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":92480368,"statfs":{"total":95995035648,"available":94699896832,"internally_reserved":0,"allocated":616300544,"data_stored":615861596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":678754757},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":240,"seq":1030792151195,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1986144,"kb_used_data":1370764,"kb_used_omap":4,"kb_used_meta":615355,"kb_avail":91759008,"statfs":{"total":95995035648,"available":93961224192,"internally_reserved":0,"allocated":1403662336,"data_stored":1403275211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4443,"internal_metadata":630124197},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":130,"seq":558345748697,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1689852,"kb_used_data":764972,"kb_used_omap":4,"kb_used_meta":924859,"kb_avail":92055300,"statfs":{"total":95995035648,"available":94264627200,"internally_reserved":0,"allocated":783331328,"data_stored":782913724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4441,"internal_metadata":947056295},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575142,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1772408,"kb_used_data":866596,"kb_used_omap":1,"kb_used_meta":905790,"kb_avail":91972744,"statfs":{"total":95995035648,"available":94180089856,"internally_reserved":0,"allocated":887394304,"data_stored":886927575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":927529413},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738668,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1669188,"kb_used_data":793908,"kb_used_omap":1,"kb_used_meta":875262,"kb_avail":92075964,"statfs":{"total":95995035648,"available":94285787136,"internally_reserved":0,"allocated":812961792,"data_stored":812534958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":896268741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":783286272,"data_stored":783286272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":879558656,"data_stored":879558656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":753864704,"data_stored":753860632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1387397120,"data_stored":1387397120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":603787264,"data_stored":603783192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":382599168,"data_stored":382599168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":948244480,"data_stored":948240408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1006632960,"data_stored":1006632960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T14:10:08.075 INFO:tasks.thrashosds.thrasher:Setting 3.d to [3, 6, 5, 4, 0, 2] 2024-09-21T14:10:08.075 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '3.d', '3', '6', '5', '4', '0', '2'] 2024-09-21T14:10:08.076 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd pg-upmap-items 3.d 3 6 5 4 0 2 2024-09-21T14:10:08.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:08.116 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:08.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:10:08.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:07 smithi187 ceph-mon[33032]: pgmap v1086: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 1.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.25k op/s; 63/76758 objects misplaced (0.082%) 2024-09-21T14:10:08.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:07 smithi187 ceph-mon[33032]: 4.d scrub starts 2024-09-21T14:10:08.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:07 smithi187 ceph-mon[33032]: 4.d scrub ok 2024-09-21T14:10:08.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:07 smithi187 ceph-mon[33032]: from='client.26648 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:10:08.217 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:10:08.240 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:08.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:08.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:10:08.279 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:10:08.342 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:10:08.398 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:08.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:08.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:10:08.500 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:10:08.549 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:08.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:08.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:10:08.651 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:10:08.707 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:08.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:08.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:10:08.809 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:10:08.838 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:08.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:08.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:10:08.940 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:10:08.972 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:08.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:08.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:10:09.074 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:10:09.105 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:09.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:09.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:10:09.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:08 smithi187 ceph-mon[33032]: 1.6 scrub starts 2024-09-21T14:10:09.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:08 smithi187 ceph-mon[33032]: 1.6 scrub ok 2024-09-21T14:10:09.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:08 smithi187 ceph-mon[33032]: 4.14 scrub starts 2024-09-21T14:10:09.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:08 smithi187 ceph-mon[33032]: 4.14 scrub ok 2024-09-21T14:10:09.207 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:10:09.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:09.245 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:09.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:10:09.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:08 smithi049 ceph-mon[40489]: 1.6 scrub starts 2024-09-21T14:10:09.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:08 smithi049 ceph-mon[40489]: 1.6 scrub ok 2024-09-21T14:10:09.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:08 smithi049 ceph-mon[40489]: 4.14 scrub starts 2024-09-21T14:10:09.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:08 smithi049 ceph-mon[40489]: 4.14 scrub ok 2024-09-21T14:10:09.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:08 smithi049 ceph-mon[30080]: 1.6 scrub starts 2024-09-21T14:10:09.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:08 smithi049 ceph-mon[30080]: 1.6 scrub ok 2024-09-21T14:10:09.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:08 smithi049 ceph-mon[30080]: 4.14 scrub starts 2024-09-21T14:10:09.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:08 smithi049 ceph-mon[30080]: 4.14 scrub ok 2024-09-21T14:10:09.343 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T14:10:09.346 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:10:09.367 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:09.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:09.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:10:09.470 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:10:09.498 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:09.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:09.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:10:09.599 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:10:09.632 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:09.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:09.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:10:09.734 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:10:09.759 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:09.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:09.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:10:09.861 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:10:09.897 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:09.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:09.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:10:09.998 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:10:10.026 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:10.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:10.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:10:10.127 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:10:10.133 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:09 smithi049 ceph-mon[30080]: pgmap v1088: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 1.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 938 op/s; 63/74739 objects misplaced (0.084%) 2024-09-21T14:10:10.133 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:09 smithi049 ceph-mon[30080]: 5.1c scrub starts 2024-09-21T14:10:10.133 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:09 smithi049 ceph-mon[30080]: 5.1c scrub ok 2024-09-21T14:10:10.133 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:09 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:10:10.133 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:09 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:10:10.133 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:09 smithi049 ceph-mon[30080]: osdmap e540: 8 total, 8 up, 8 in 2024-09-21T14:10:10.134 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:09 smithi049 ceph-mon[40489]: pgmap v1088: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 1.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 938 op/s; 63/74739 objects misplaced (0.084%) 2024-09-21T14:10:10.134 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:09 smithi049 ceph-mon[40489]: 5.1c scrub starts 2024-09-21T14:10:10.134 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:09 smithi049 ceph-mon[40489]: 5.1c scrub ok 2024-09-21T14:10:10.134 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:09 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:10:10.134 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:09 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:10:10.134 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:09 smithi049 ceph-mon[40489]: osdmap e540: 8 total, 8 up, 8 in 2024-09-21T14:10:10.176 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:10.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:10.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:10:10.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:09 smithi187 ceph-mon[33032]: pgmap v1088: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 1.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 938 op/s; 63/74739 objects misplaced (0.084%) 2024-09-21T14:10:10.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:09 smithi187 ceph-mon[33032]: 5.1c scrub starts 2024-09-21T14:10:10.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:09 smithi187 ceph-mon[33032]: 5.1c scrub ok 2024-09-21T14:10:10.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:09 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:10:10.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:09 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:10:10.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:09 smithi187 ceph-mon[33032]: osdmap e540: 8 total, 8 up, 8 in 2024-09-21T14:10:10.278 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:10:10.308 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:10.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:10.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:10:10.410 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:10:10.438 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:10.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:10.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:10:10.540 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:10:10.560 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:10.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:10.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:10:10.662 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:10:10.690 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:10.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:10.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:10:10.694 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:10:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:10:10.791 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:10:10.824 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:10.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:10.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:10:10.862 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T14:10:10.862 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-09-21T14:10:10.862 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:10.862 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26396.0:228033 11.8 e540/528 11:13c94756:::benchmark_data_smithi049_206132_object104654:head v 540'14006, mlcod=540'14003)", 2024-09-21T14:10:10.862 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:10:10.128287+0000", 2024-09-21T14:10:10.862 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.0020301429999999999, 2024-09-21T14:10:10.862 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002057527, 2024-09-21T14:10:10.862 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:10.863 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:10.863 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T14:10:10.863 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:10.863 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:10.863 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:10.863 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:10.128287+0000", 2024-09-21T14:10:10.863 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:10.863 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:10.863 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:10.863 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:10.863 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:10.128287+0000", 2024-09-21T14:10:10.863 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:10.863 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:10.863 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:10.864 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:10.864 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:10.128288+0000", 2024-09-21T14:10:10.864 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4300000000000001e-07 2024-09-21T14:10:10.864 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:10.864 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:10.864 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:10.864 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:10.128295+0000", 2024-09-21T14:10:10.864 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6429999999999998e-06 2024-09-21T14:10:10.864 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:10.864 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:10.864 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:10.864 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:10.128308+0000", 2024-09-21T14:10:10.864 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3049e-05 2024-09-21T14:10:10.864 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:10.865 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:10.865 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:10.865 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:10.128316+0000", 2024-09-21T14:10:10.865 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7589999999999997e-06 2024-09-21T14:10:10.865 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:10.865 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:10.865 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:10.865 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:10.128403+0000", 2024-09-21T14:10:10.865 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.7526000000000002e-05 2024-09-21T14:10:10.865 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:10.865 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:10.865 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:10.865 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:10.128423+0000", 2024-09-21T14:10:10.865 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9809e-05 2024-09-21T14:10:10.866 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:10.866 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:10.866 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:10.866 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:10.866 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:10.866 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26396.0:228036 11.2 e540/531 11:42ccdc15:::benchmark_data_smithi049_206132_object104657:head v 540'14109, mlcod=540'14107)", 2024-09-21T14:10:10.866 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:10:10.128940+0000", 2024-09-21T14:10:10.866 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.0013778950000000001, 2024-09-21T14:10:10.866 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0014590440000000001, 2024-09-21T14:10:10.866 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:10.866 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:10.866 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T14:10:10.866 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:10.866 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:10.867 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:10.867 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:10.128940+0000", 2024-09-21T14:10:10.867 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:10.867 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:10.867 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:10.867 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:10.867 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:10.128940+0000", 2024-09-21T14:10:10.867 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:10.867 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:10.867 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:10.867 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:10.867 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:10.128940+0000", 2024-09-21T14:10:10.867 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5799999999999998e-07 2024-09-21T14:10:10.867 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:10.868 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:10.868 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:10.868 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:10.128946+0000", 2024-09-21T14:10:10.868 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6489999999999996e-06 2024-09-21T14:10:10.868 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:10.868 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:10.868 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:10.868 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:10.128961+0000", 2024-09-21T14:10:10.868 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5318000000000001e-05 2024-09-21T14:10:10.868 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:10.868 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:10.868 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:10.868 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:10.128968+0000", 2024-09-21T14:10:10.868 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3929999999999996e-06 2024-09-21T14:10:10.869 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:10.869 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:10.869 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:10.869 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:10.129135+0000", 2024-09-21T14:10:10.869 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016725400000000001 2024-09-21T14:10:10.869 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:10.869 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:10.869 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:10.869 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:10.129183+0000", 2024-09-21T14:10:10.869 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7679000000000002e-05 2024-09-21T14:10:10.869 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:10.869 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:10.869 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:10.869 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:10.870 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T14:10:10.870 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 2 2024-09-21T14:10:10.870 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T14:10:10.870 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:10:10 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:10:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:10:10.927 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:10:10.965 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:10.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:10.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:10:11.067 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:10:11.195 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:10:11.196 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:10 smithi187 ceph-mon[33032]: 3.e scrub starts 2024-09-21T14:10:11.196 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:10 smithi187 ceph-mon[33032]: 3.e scrub ok 2024-09-21T14:10:11.196 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:10 smithi187 ceph-mon[33032]: 11.c scrub starts 2024-09-21T14:10:11.196 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 14:10:11 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=cleanup t=2024-09-21T14:10:11.016201106Z level=info msg="Completed cleanup jobs" duration=45.216835ms 2024-09-21T14:10:11.196 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 14:10:11 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=plugins.update.checker t=2024-09-21T14:10:11.107319435Z level=info msg="Update check succeeded" duration=63.235347ms 2024-09-21T14:10:11.225 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:11.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:11.226 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:10:11.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:10 smithi049 ceph-mon[30080]: 3.e scrub starts 2024-09-21T14:10:11.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:10 smithi049 ceph-mon[30080]: 3.e scrub ok 2024-09-21T14:10:11.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:10 smithi049 ceph-mon[30080]: 11.c scrub starts 2024-09-21T14:10:11.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:10 smithi049 ceph-mon[40489]: 3.e scrub starts 2024-09-21T14:10:11.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:10 smithi049 ceph-mon[40489]: 3.e scrub ok 2024-09-21T14:10:11.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:10 smithi049 ceph-mon[40489]: 11.c scrub starts 2024-09-21T14:10:11.327 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:10:11.365 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:11.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:11.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:10:11.467 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:10:11.489 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:11.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:11.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:10:11.590 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:10:11.613 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:11.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:11.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:10:11.715 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:10:11.742 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:11.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:11.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:10:11.844 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:10:11.868 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:11.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:11.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:10:11.970 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:10:11.998 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:11.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:11.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:10:12.100 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:10:12.137 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:12.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:12.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:10:12.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:11 smithi187 ceph-mon[33032]: pgmap v1090: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 1.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 955 B/s rd, 0 B/s wr, 1.06k op/s; 63/65109 objects misplaced (0.097%) 2024-09-21T14:10:12.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:11 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:10:12.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:11 smithi187 ceph-mon[33032]: 11.c scrub ok 2024-09-21T14:10:12.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:11 smithi187 ceph-mon[33032]: 6.18 scrub starts 2024-09-21T14:10:12.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:11 smithi187 ceph-mon[33032]: 6.18 scrub ok 2024-09-21T14:10:12.204 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:11 smithi049 ceph-mon[30080]: pgmap v1090: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 1.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 955 B/s rd, 0 B/s wr, 1.06k op/s; 63/65109 objects misplaced (0.097%) 2024-09-21T14:10:12.204 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:11 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:10:12.204 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:11 smithi049 ceph-mon[30080]: 11.c scrub ok 2024-09-21T14:10:12.204 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:11 smithi049 ceph-mon[30080]: 6.18 scrub starts 2024-09-21T14:10:12.204 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:11 smithi049 ceph-mon[30080]: 6.18 scrub ok 2024-09-21T14:10:12.204 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:11 smithi049 ceph-mon[40489]: pgmap v1090: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 1.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 955 B/s rd, 0 B/s wr, 1.06k op/s; 63/65109 objects misplaced (0.097%) 2024-09-21T14:10:12.204 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:11 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:10:12.204 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:11 smithi049 ceph-mon[40489]: 11.c scrub ok 2024-09-21T14:10:12.205 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:11 smithi049 ceph-mon[40489]: 6.18 scrub starts 2024-09-21T14:10:12.205 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:11 smithi049 ceph-mon[40489]: 6.18 scrub ok 2024-09-21T14:10:12.229 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:10:12.239 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:10:12.258 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T14:10:12.265 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:12.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:12.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:10:12.367 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:10:12.406 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:12.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:12.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:10:12.508 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:10:12.540 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:12.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:12.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:10:12.642 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:10:12.673 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:12.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:12.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:10:12.775 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:10:12.813 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:12.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:12.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:10:12.915 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:10:12.951 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:12.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:12.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:10:13.053 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:10:13.083 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:13.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:13.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:10:13.144 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:12 smithi049 ceph-mon[30080]: 6.f scrub starts 2024-09-21T14:10:13.144 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:12 smithi049 ceph-mon[30080]: 6.f scrub ok 2024-09-21T14:10:13.144 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:12 smithi049 ceph-mon[30080]: 3.18 scrub starts 2024-09-21T14:10:13.144 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:12 smithi049 ceph-mon[30080]: 3.18 scrub ok 2024-09-21T14:10:13.145 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:12 smithi049 ceph-mon[40489]: 6.f scrub starts 2024-09-21T14:10:13.145 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:12 smithi049 ceph-mon[40489]: 6.f scrub ok 2024-09-21T14:10:13.145 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:12 smithi049 ceph-mon[40489]: 3.18 scrub starts 2024-09-21T14:10:13.145 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:12 smithi049 ceph-mon[40489]: 3.18 scrub ok 2024-09-21T14:10:13.155 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:12 smithi187 ceph-mon[33032]: 6.f scrub starts 2024-09-21T14:10:13.155 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:12 smithi187 ceph-mon[33032]: 6.f scrub ok 2024-09-21T14:10:13.155 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:12 smithi187 ceph-mon[33032]: 3.18 scrub starts 2024-09-21T14:10:13.155 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:12 smithi187 ceph-mon[33032]: 3.18 scrub ok 2024-09-21T14:10:13.184 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:10:13.226 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:13.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:13.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:10:13.259 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:10:13.259 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:10:13.267 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:10:13.317 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:10:13.317 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:10:13.328 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:10:13.366 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:10:13.367 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:10:13.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:13.397 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:13.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:10:13.417 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:10:13.417 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:10:13.454 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:10:13.454 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:10:13.487 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:10:13.488 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:10:13.498 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:10:13.526 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:10:13.526 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:10:13.533 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:13.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:13.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:10:13.565 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:10:13.565 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:10:13.635 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:10:13.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:13.671 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:13.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:10:13.772 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:10:13.810 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:13.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:13.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:10:13.889 INFO:teuthology.orchestra.run.smithi049.stderr:set 3.d pg_upmap_items mapping to [3->6,5->4,0->2] 2024-09-21T14:10:13.912 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:10:13.949 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:13.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:13.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:10:14.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:13 smithi049 ceph-mon[40489]: pgmap v1091: 168 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 166 active+clean; 1.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.16k op/s; 45/57996 objects misplaced (0.078%) 2024-09-21T14:10:14.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:13 smithi049 ceph-mon[40489]: 6.9 scrub starts 2024-09-21T14:10:14.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:13 smithi049 ceph-mon[40489]: 6.9 scrub ok 2024-09-21T14:10:14.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:13 smithi049 ceph-mon[40489]: 6.1a scrub starts 2024-09-21T14:10:14.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:13 smithi049 ceph-mon[40489]: 6.1a scrub ok 2024-09-21T14:10:14.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:13 smithi049 ceph-mon[40489]: 4.a scrub starts 2024-09-21T14:10:14.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:13 smithi049 ceph-mon[40489]: 4.a scrub ok 2024-09-21T14:10:14.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:13 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1572668952' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:10:14.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:13 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/4212889148' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "3.d", "id": [3, 6, 5, 4, 0, 2]} : dispatch 2024-09-21T14:10:14.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:13 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "3.d", "id": [3, 6, 5, 4, 0, 2]} : dispatch 2024-09-21T14:10:14.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:13 smithi049 ceph-mon[30080]: pgmap v1091: 168 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 166 active+clean; 1.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.16k op/s; 45/57996 objects misplaced (0.078%) 2024-09-21T14:10:14.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:13 smithi049 ceph-mon[30080]: 6.9 scrub starts 2024-09-21T14:10:14.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:13 smithi049 ceph-mon[30080]: 6.9 scrub ok 2024-09-21T14:10:14.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:13 smithi049 ceph-mon[30080]: 6.1a scrub starts 2024-09-21T14:10:14.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:13 smithi049 ceph-mon[30080]: 6.1a scrub ok 2024-09-21T14:10:14.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:13 smithi049 ceph-mon[30080]: 4.a scrub starts 2024-09-21T14:10:14.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:13 smithi049 ceph-mon[30080]: 4.a scrub ok 2024-09-21T14:10:14.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:13 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1572668952' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:10:14.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:13 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/4212889148' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "3.d", "id": [3, 6, 5, 4, 0, 2]} : dispatch 2024-09-21T14:10:14.006 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:13 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "3.d", "id": [3, 6, 5, 4, 0, 2]} : dispatch 2024-09-21T14:10:14.052 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:10:14.091 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:14.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:14.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:10:14.194 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:10:14.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:13 smithi187 ceph-mon[33032]: pgmap v1091: 168 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 166 active+clean; 1.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.16k op/s; 45/57996 objects misplaced (0.078%) 2024-09-21T14:10:14.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:13 smithi187 ceph-mon[33032]: 6.9 scrub starts 2024-09-21T14:10:14.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:13 smithi187 ceph-mon[33032]: 6.9 scrub ok 2024-09-21T14:10:14.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:13 smithi187 ceph-mon[33032]: 6.1a scrub starts 2024-09-21T14:10:14.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:13 smithi187 ceph-mon[33032]: 6.1a scrub ok 2024-09-21T14:10:14.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:13 smithi187 ceph-mon[33032]: 4.a scrub starts 2024-09-21T14:10:14.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:13 smithi187 ceph-mon[33032]: 4.a scrub ok 2024-09-21T14:10:14.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:13 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1572668952' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:10:14.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:13 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/4212889148' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "3.d", "id": [3, 6, 5, 4, 0, 2]} : dispatch 2024-09-21T14:10:14.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:13 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "3.d", "id": [3, 6, 5, 4, 0, 2]} : dispatch 2024-09-21T14:10:14.228 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:14.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:14.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:10:14.330 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:10:14.367 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:14.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:14.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:10:14.469 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:10:14.511 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:14.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:14.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:10:14.613 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:10:14.647 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:14.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:14.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:10:14.750 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:10:14.798 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:14.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:14.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:10:14.900 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:10:14.941 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:14.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:14.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:10:14.993 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:14 smithi049 ceph-mon[40489]: 4.1a scrub starts 2024-09-21T14:10:14.993 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:14 smithi049 ceph-mon[40489]: 4.1a scrub ok 2024-09-21T14:10:14.993 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:14 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "3.d", "id": [3, 6, 5, 4, 0, 2]}]': finished 2024-09-21T14:10:14.993 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:14 smithi049 ceph-mon[40489]: osdmap e541: 8 total, 8 up, 8 in 2024-09-21T14:10:14.994 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:14 smithi049 ceph-mon[40489]: 1.3 scrub starts 2024-09-21T14:10:14.994 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:14 smithi049 ceph-mon[40489]: 1.3 scrub ok 2024-09-21T14:10:14.994 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:14 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:10:14.994 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:14 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:10:14.994 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:14 smithi049 ceph-mon[30080]: 4.1a scrub starts 2024-09-21T14:10:14.994 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:14 smithi049 ceph-mon[30080]: 4.1a scrub ok 2024-09-21T14:10:14.994 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:14 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "3.d", "id": [3, 6, 5, 4, 0, 2]}]': finished 2024-09-21T14:10:14.994 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:14 smithi049 ceph-mon[30080]: osdmap e541: 8 total, 8 up, 8 in 2024-09-21T14:10:14.994 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:14 smithi049 ceph-mon[30080]: 1.3 scrub starts 2024-09-21T14:10:14.994 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:14 smithi049 ceph-mon[30080]: 1.3 scrub ok 2024-09-21T14:10:14.995 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:14 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:10:14.995 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:14 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:10:15.043 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:10:15.069 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:15.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:15.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:10:15.170 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:10:15.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:14 smithi187 ceph-mon[33032]: 4.1a scrub starts 2024-09-21T14:10:15.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:14 smithi187 ceph-mon[33032]: 4.1a scrub ok 2024-09-21T14:10:15.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:14 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "3.d", "id": [3, 6, 5, 4, 0, 2]}]': finished 2024-09-21T14:10:15.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:14 smithi187 ceph-mon[33032]: osdmap e541: 8 total, 8 up, 8 in 2024-09-21T14:10:15.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:14 smithi187 ceph-mon[33032]: 1.3 scrub starts 2024-09-21T14:10:15.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:14 smithi187 ceph-mon[33032]: 1.3 scrub ok 2024-09-21T14:10:15.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:14 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:10:15.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:14 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:10:15.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:15.199 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:15.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:10:15.299 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:10:15.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:15.343 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:15.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:10:15.444 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:10:15.470 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:15.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:15.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:10:15.499 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:10:15.572 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:10:15.613 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:15.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:15.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:10:15.715 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:10:15.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:15.752 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:15.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:10:15.853 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:10:15.888 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:15.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:15.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:10:15.987 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:15 smithi049 ceph-mon[40489]: pgmap v1093: 168 pgs: 1 peering, 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 165 active+clean; 1013 MiB data, 10 GiB used, 705 GiB / 715 GiB avail; 647 B/s rd, 0 B/s wr, 1.16k op/s; 45/49203 objects misplaced (0.091%) 2024-09-21T14:10:15.988 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:15 smithi049 ceph-mon[40489]: osdmap e542: 8 total, 8 up, 8 in 2024-09-21T14:10:15.988 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:15 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2693461240' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:10:15.988 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:15 smithi049 ceph-mon[30080]: pgmap v1093: 168 pgs: 1 peering, 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 165 active+clean; 1013 MiB data, 10 GiB used, 705 GiB / 715 GiB avail; 647 B/s rd, 0 B/s wr, 1.16k op/s; 45/49203 objects misplaced (0.091%) 2024-09-21T14:10:15.988 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:15 smithi049 ceph-mon[30080]: osdmap e542: 8 total, 8 up, 8 in 2024-09-21T14:10:15.988 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:15 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2693461240' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:10:15.989 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:10:16.024 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:16.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:16.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:10:16.126 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:10:16.153 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T14:10:16.153 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:10:16.164 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:16.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:16.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:10:16.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:15 smithi187 ceph-mon[33032]: pgmap v1093: 168 pgs: 1 peering, 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 165 active+clean; 1013 MiB data, 10 GiB used, 705 GiB / 715 GiB avail; 647 B/s rd, 0 B/s wr, 1.16k op/s; 45/49203 objects misplaced (0.091%) 2024-09-21T14:10:16.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:15 smithi187 ceph-mon[33032]: osdmap e542: 8 total, 8 up, 8 in 2024-09-21T14:10:16.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:15 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2693461240' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:10:16.265 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:10:16.288 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-09-21T14:10:16.294 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:16.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:16.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:10:16.396 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:10:16.430 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:16.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:16.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:10:16.532 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:10:16.559 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:16.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:16.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:10:16.661 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:10:16.693 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:16.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:16.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:10:16.769 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T14:10:16.795 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:10:16.819 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:16.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:16.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:10:16.921 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:10:16.949 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:16.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:16.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:10:17.050 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:10:17.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:17.093 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:17.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:10:17.194 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:10:17.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:16 smithi187 ceph-mon[33032]: 11.7 scrub starts 2024-09-21T14:10:17.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:16 smithi187 ceph-mon[33032]: 11.7 scrub ok 2024-09-21T14:10:17.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:16 smithi187 ceph-mon[33032]: 6.12 scrub starts 2024-09-21T14:10:17.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:16 smithi187 ceph-mon[33032]: 6.12 scrub ok 2024-09-21T14:10:17.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:16 smithi187 ceph-mon[33032]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T14:10:17.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:16 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2693461240' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T14:10:17.195 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:16 smithi187 ceph-mon[33032]: osdmap e543: 8 total, 8 up, 8 in 2024-09-21T14:10:17.236 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:17.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:17.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:10:17.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:16 smithi049 ceph-mon[40489]: 11.7 scrub starts 2024-09-21T14:10:17.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:16 smithi049 ceph-mon[40489]: 11.7 scrub ok 2024-09-21T14:10:17.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:16 smithi049 ceph-mon[40489]: 6.12 scrub starts 2024-09-21T14:10:17.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:16 smithi049 ceph-mon[40489]: 6.12 scrub ok 2024-09-21T14:10:17.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:16 smithi049 ceph-mon[40489]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T14:10:17.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:16 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2693461240' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T14:10:17.255 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:16 smithi049 ceph-mon[40489]: osdmap e543: 8 total, 8 up, 8 in 2024-09-21T14:10:17.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:16 smithi049 ceph-mon[30080]: 11.7 scrub starts 2024-09-21T14:10:17.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:16 smithi049 ceph-mon[30080]: 11.7 scrub ok 2024-09-21T14:10:17.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:16 smithi049 ceph-mon[30080]: 6.12 scrub starts 2024-09-21T14:10:17.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:16 smithi049 ceph-mon[30080]: 6.12 scrub ok 2024-09-21T14:10:17.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:16 smithi049 ceph-mon[30080]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T14:10:17.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:16 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2693461240' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T14:10:17.255 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:16 smithi049 ceph-mon[30080]: osdmap e543: 8 total, 8 up, 8 in 2024-09-21T14:10:17.338 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:10:17.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:17.377 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:17.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:10:17.478 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:10:17.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:17.508 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:17.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:10:17.609 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:10:17.642 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:17.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:17.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:10:17.744 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:10:17.770 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:17.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:17.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:10:17.872 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:10:17.900 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:17.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:17.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:10:18.002 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:10:18.033 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:18.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:18.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:10:18.134 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:10:18.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:18.164 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:18.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:10:18.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:17 smithi187 ceph-mon[33032]: pgmap v1095: 168 pgs: 1 peering, 1 active+remapped+backfill_toofull, 166 active+clean; 909 MiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.25k op/s; 33/44232 objects misplaced (0.075%) 2024-09-21T14:10:18.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:17 smithi187 ceph-mon[33032]: 5.12 scrub starts 2024-09-21T14:10:18.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:17 smithi187 ceph-mon[33032]: 5.12 scrub ok 2024-09-21T14:10:18.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:17 smithi049 ceph-mon[40489]: pgmap v1095: 168 pgs: 1 peering, 1 active+remapped+backfill_toofull, 166 active+clean; 909 MiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.25k op/s; 33/44232 objects misplaced (0.075%) 2024-09-21T14:10:18.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:17 smithi049 ceph-mon[40489]: 5.12 scrub starts 2024-09-21T14:10:18.254 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:17 smithi049 ceph-mon[40489]: 5.12 scrub ok 2024-09-21T14:10:18.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:17 smithi049 ceph-mon[30080]: pgmap v1095: 168 pgs: 1 peering, 1 active+remapped+backfill_toofull, 166 active+clean; 909 MiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.25k op/s; 33/44232 objects misplaced (0.075%) 2024-09-21T14:10:18.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:17 smithi049 ceph-mon[30080]: 5.12 scrub starts 2024-09-21T14:10:18.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:17 smithi049 ceph-mon[30080]: 5.12 scrub ok 2024-09-21T14:10:18.266 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:10:18.305 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:18.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:18.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:10:18.407 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:10:18.445 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:18.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:18.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:10:18.546 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:10:18.572 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T14:10:18.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:18.573 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:18.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:10:18.674 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:10:18.699 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:10:18.702 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:18.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:18.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:10:18.804 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:10:18.832 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:18.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:18.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:10:18.934 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:10:18.963 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:18.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:18.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:10:19.064 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:10:19.116 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:19.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:19.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:10:19.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:18 smithi187 ceph-mon[33032]: 5.b scrub starts 2024-09-21T14:10:19.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:18 smithi187 ceph-mon[33032]: 5.b scrub ok 2024-09-21T14:10:19.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:18 smithi187 ceph-mon[33032]: 3.2 scrub starts 2024-09-21T14:10:19.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:18 smithi187 ceph-mon[33032]: 3.2 scrub ok 2024-09-21T14:10:19.207 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:18 smithi049 ceph-mon[30080]: 5.b scrub starts 2024-09-21T14:10:19.207 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:18 smithi049 ceph-mon[30080]: 5.b scrub ok 2024-09-21T14:10:19.207 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:18 smithi049 ceph-mon[30080]: 3.2 scrub starts 2024-09-21T14:10:19.208 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:18 smithi049 ceph-mon[30080]: 3.2 scrub ok 2024-09-21T14:10:19.208 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:18 smithi049 ceph-mon[40489]: 5.b scrub starts 2024-09-21T14:10:19.208 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:18 smithi049 ceph-mon[40489]: 5.b scrub ok 2024-09-21T14:10:19.208 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:18 smithi049 ceph-mon[40489]: 3.2 scrub starts 2024-09-21T14:10:19.208 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:18 smithi049 ceph-mon[40489]: 3.2 scrub ok 2024-09-21T14:10:19.217 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:10:19.244 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:19.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:19.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:10:19.346 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:10:19.373 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:19.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:19.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:10:19.397 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:10:19.408 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:10:19.408 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:10:19.443 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:10:19.444 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:10:19.475 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:10:19.483 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:10:19.483 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:10:19.514 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:10:19.514 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:10:19.518 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:19.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:19.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:10:19.548 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:10:19.548 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:10:19.586 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:10:19.587 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:10:19.620 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:10:19.620 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:10:19.621 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:10:19.652 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:19.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:19.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:10:19.656 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:10:19.656 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:10:19.754 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:10:19.784 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:19.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:19.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:10:19.886 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:10:19.927 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:19.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:19.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:10:19.975 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T14:10:20.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:19 smithi049 ceph-mon[30080]: pgmap v1097: 168 pgs: 1 peering, 1 active+remapped+backfill_toofull, 166 active+clean; 866 MiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 881 op/s; 33/42138 objects misplaced (0.078%) 2024-09-21T14:10:20.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:19 smithi049 ceph-mon[30080]: 4.b scrub starts 2024-09-21T14:10:20.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:19 smithi049 ceph-mon[30080]: 4.b scrub ok 2024-09-21T14:10:20.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:19 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2350256952' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:10:20.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:19 smithi049 ceph-mon[40489]: pgmap v1097: 168 pgs: 1 peering, 1 active+remapped+backfill_toofull, 166 active+clean; 866 MiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 881 op/s; 33/42138 objects misplaced (0.078%) 2024-09-21T14:10:20.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:19 smithi049 ceph-mon[40489]: 4.b scrub starts 2024-09-21T14:10:20.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:19 smithi049 ceph-mon[40489]: 4.b scrub ok 2024-09-21T14:10:20.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:19 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2350256952' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:10:20.029 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:10:20.067 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:20.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:20.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:10:20.169 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:10:20.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:19 smithi187 ceph-mon[33032]: pgmap v1097: 168 pgs: 1 peering, 1 active+remapped+backfill_toofull, 166 active+clean; 866 MiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 881 op/s; 33/42138 objects misplaced (0.078%) 2024-09-21T14:10:20.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:19 smithi187 ceph-mon[33032]: 4.b scrub starts 2024-09-21T14:10:20.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:19 smithi187 ceph-mon[33032]: 4.b scrub ok 2024-09-21T14:10:20.194 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:19 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2350256952' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:10:20.198 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:20.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:20.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:10:20.299 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T14:10:20.299 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-09-21T14:10:20.299 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-09-21T14:10:20.300 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-09-21T14:10:20.300 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.300 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:148200 10.8 10:11441144:::benchmark_data_smithi049_176811_object20623:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-09-21T14:10:20.300 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:04:55.587007+0000", 2024-09-21T14:10:20.300 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 324.70928221399998, 2024-09-21T14:10:20.300 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.019040647, 2024-09-21T14:10:20.300 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:20.300 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:20.300 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:20.300 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:10:20.300 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:10:20.300 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:10:20.300 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 148200 2024-09-21T14:10:20.300 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.301 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:20.301 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.301 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:20.301 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.587007+0000", 2024-09-21T14:10:20.301 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:20.301 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.301 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.301 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:20.301 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.587007+0000", 2024-09-21T14:10:20.301 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:20.301 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.301 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.301 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:20.301 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.587008+0000", 2024-09-21T14:10:20.302 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.649e-06 2024-09-21T14:10:20.302 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.302 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.302 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:20.302 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.587014+0000", 2024-09-21T14:10:20.302 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5849999999999999e-06 2024-09-21T14:10:20.302 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.302 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.302 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:20.302 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.587031+0000", 2024-09-21T14:10:20.302 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6756e-05 2024-09-21T14:10:20.302 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.302 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.303 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:20.303 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.587040+0000", 2024-09-21T14:10:20.303 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3510000000000008e-06 2024-09-21T14:10:20.303 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.303 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.303 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:20.303 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.587135+0000", 2024-09-21T14:10:20.303 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4471999999999999e-05 2024-09-21T14:10:20.303 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.303 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.303 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:10:20.303 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.587145+0000", 2024-09-21T14:10:20.303 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0186e-05 2024-09-21T14:10:20.303 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.304 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.304 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:20.304 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.598177+0000", 2024-09-21T14:10:20.304 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0110321339999999 2024-09-21T14:10:20.304 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.304 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.304 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:20.304 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.599612+0000", 2024-09-21T14:10:20.304 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001434828 2024-09-21T14:10:20.304 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.304 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.305 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,7", 2024-09-21T14:10:20.305 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.599704+0000", 2024-09-21T14:10:20.305 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2164999999999996e-05 2024-09-21T14:10:20.305 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.305 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.305 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:10:20.305 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.601928+0000", 2024-09-21T14:10:20.305 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0022241869999999999 2024-09-21T14:10:20.305 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.305 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.305 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:20.305 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.605855+0000", 2024-09-21T14:10:20.305 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0039272839999999996 2024-09-21T14:10:20.305 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.305 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.306 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:20.306 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.605933+0000", 2024-09-21T14:10:20.306 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7194999999999995e-05 2024-09-21T14:10:20.306 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.306 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.306 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:10:20.306 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.605997+0000", 2024-09-21T14:10:20.306 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4182000000000003e-05 2024-09-21T14:10:20.306 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.307 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.307 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:20.307 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.606047+0000", 2024-09-21T14:10:20.307 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0673000000000001e-05 2024-09-21T14:10:20.307 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:20.307 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:20.307 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:20.307 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.307 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.307 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:148204 10.8 10:173acc92:::benchmark_data_smithi049_176811_object20627:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-09-21T14:10:20.307 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:04:55.598766+0000", 2024-09-21T14:10:20.307 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 324.69752329099998, 2024-09-21T14:10:20.308 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.012530269, 2024-09-21T14:10:20.308 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:20.308 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:20.308 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:20.308 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:10:20.308 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:10:20.308 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:10:20.308 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 148204 2024-09-21T14:10:20.308 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.308 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:20.308 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.308 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:20.308 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.598766+0000", 2024-09-21T14:10:20.308 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:20.309 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.309 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.309 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:20.309 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.598766+0000", 2024-09-21T14:10:20.309 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:20.309 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.309 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.309 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:20.309 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.598768+0000", 2024-09-21T14:10:20.309 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.951e-06 2024-09-21T14:10:20.309 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.309 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.309 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:20.309 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.598775+0000", 2024-09-21T14:10:20.310 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8450000000000003e-06 2024-09-21T14:10:20.310 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.310 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.310 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:20.310 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.598793+0000", 2024-09-21T14:10:20.310 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7348999999999999e-05 2024-09-21T14:10:20.310 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.310 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.310 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:20.310 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.598803+0000", 2024-09-21T14:10:20.310 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8120000000000002e-06 2024-09-21T14:10:20.310 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.310 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.310 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:20.311 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.598888+0000", 2024-09-21T14:10:20.311 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4975999999999994e-05 2024-09-21T14:10:20.311 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.311 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.311 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:10:20.311 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.598898+0000", 2024-09-21T14:10:20.311 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0899e-05 2024-09-21T14:10:20.311 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.311 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.311 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:20.311 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.601975+0000", 2024-09-21T14:10:20.311 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0030761989999999 2024-09-21T14:10:20.311 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.311 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.312 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:20.312 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.602095+0000", 2024-09-21T14:10:20.312 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012031699999999999 2024-09-21T14:10:20.312 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.312 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.312 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,7", 2024-09-21T14:10:20.312 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.602214+0000", 2024-09-21T14:10:20.312 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011874700000000001 2024-09-21T14:10:20.312 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.312 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.312 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:10:20.312 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.606106+0000", 2024-09-21T14:10:20.312 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0038918389999999998 2024-09-21T14:10:20.312 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.312 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.313 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:20.313 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.606149+0000", 2024-09-21T14:10:20.313 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3593000000000001e-05 2024-09-21T14:10:20.313 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.313 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.313 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:20.313 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.611191+0000", 2024-09-21T14:10:20.313 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0050419280000000002 2024-09-21T14:10:20.313 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.313 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.313 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:10:20.313 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.611250+0000", 2024-09-21T14:10:20.313 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8956000000000003e-05 2024-09-21T14:10:20.313 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.313 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.314 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:10:20.315 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:20.315 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.611296+0000", 2024-09-21T14:10:20.315 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5858e-05 2024-09-21T14:10:20.315 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:20.315 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:20.315 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:20.315 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.315 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.315 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:148288 10.8 10:1610cc8e:::benchmark_data_smithi049_176811_object20711:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-09-21T14:10:20.315 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:04:55.675295+0000", 2024-09-21T14:10:20.315 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 324.62099350099999, 2024-09-21T14:10:20.315 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.94425190999999997, 2024-09-21T14:10:20.316 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:20.316 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:20.316 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:20.316 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:10:20.316 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:10:20.316 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:10:20.316 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 148288 2024-09-21T14:10:20.316 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.316 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:20.316 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.316 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:20.316 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.675295+0000", 2024-09-21T14:10:20.316 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:20.316 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.317 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.317 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:20.317 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.675295+0000", 2024-09-21T14:10:20.317 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:20.317 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.317 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.317 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:20.317 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.675297+0000", 2024-09-21T14:10:20.317 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4890000000000001e-06 2024-09-21T14:10:20.317 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.317 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.317 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:20.317 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.675302+0000", 2024-09-21T14:10:20.317 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8250000000000004e-06 2024-09-21T14:10:20.318 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.318 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.318 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:20.318 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.675321+0000", 2024-09-21T14:10:20.318 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8981999999999999e-05 2024-09-21T14:10:20.318 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.318 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.318 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:20.318 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.675329+0000", 2024-09-21T14:10:20.318 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2940000000000006e-06 2024-09-21T14:10:20.318 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.318 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.318 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:20.318 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.675411+0000", 2024-09-21T14:10:20.319 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1927000000000004e-05 2024-09-21T14:10:20.319 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.319 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.319 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:10:20.319 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.675421+0000", 2024-09-21T14:10:20.319 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9389999999999992e-06 2024-09-21T14:10:20.319 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.319 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.319 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:20.319 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.608831+0000", 2024-09-21T14:10:20.319 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.93340988899999999 2024-09-21T14:10:20.319 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.319 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.320 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:20.320 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.608931+0000", 2024-09-21T14:10:20.320 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000100715 2024-09-21T14:10:20.320 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.320 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.320 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,7", 2024-09-21T14:10:20.320 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.609052+0000", 2024-09-21T14:10:20.320 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000120139 2024-09-21T14:10:20.320 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.320 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.320 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:10:20.320 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.611541+0000", 2024-09-21T14:10:20.320 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0024891079999999999 2024-09-21T14:10:20.321 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.321 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.321 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:20.321 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.611824+0000", 2024-09-21T14:10:20.321 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00028299899999999998 2024-09-21T14:10:20.321 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.321 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.321 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:20.321 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.619454+0000", 2024-09-21T14:10:20.321 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.007630375 2024-09-21T14:10:20.321 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.321 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.321 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:10:20.321 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.619487+0000", 2024-09-21T14:10:20.321 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2790000000000003e-05 2024-09-21T14:10:20.322 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.322 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.322 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:20.322 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.619547+0000", 2024-09-21T14:10:20.322 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0439000000000002e-05 2024-09-21T14:10:20.322 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:20.322 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:20.322 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:20.322 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.322 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.322 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:148289 10.8 10:19cc1202:::benchmark_data_smithi049_176811_object20712:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-09-21T14:10:20.322 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:04:55.675487+0000", 2024-09-21T14:10:20.323 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 324.620802191, 2024-09-21T14:10:20.323 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.95131562199999997, 2024-09-21T14:10:20.323 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:20.323 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:20.323 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:20.323 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:10:20.323 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:10:20.323 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:10:20.323 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 148289 2024-09-21T14:10:20.323 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.323 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:20.323 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.323 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:20.323 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.675487+0000", 2024-09-21T14:10:20.324 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:20.324 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.324 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.324 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:20.324 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.675487+0000", 2024-09-21T14:10:20.324 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:20.324 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.324 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.324 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:20.324 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.675488+0000", 2024-09-21T14:10:20.324 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4200000000000005e-07 2024-09-21T14:10:20.324 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.324 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.324 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:20.324 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.675507+0000", 2024-09-21T14:10:20.325 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9219999999999999e-05 2024-09-21T14:10:20.325 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.325 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.325 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:20.325 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.675521+0000", 2024-09-21T14:10:20.325 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4688999999999999e-05 2024-09-21T14:10:20.325 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.325 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.325 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:20.325 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.675529+0000", 2024-09-21T14:10:20.325 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2599999999999999e-06 2024-09-21T14:10:20.325 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.325 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.325 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:20.326 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.675588+0000", 2024-09-21T14:10:20.326 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8936e-05 2024-09-21T14:10:20.326 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.326 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.326 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:10:20.326 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:55.675597+0000", 2024-09-21T14:10:20.326 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6889999999999999e-06 2024-09-21T14:10:20.326 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.326 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.326 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:20.326 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.611876+0000", 2024-09-21T14:10:20.326 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.93627835500000001 2024-09-21T14:10:20.326 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.326 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.327 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:20.327 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.611995+0000", 2024-09-21T14:10:20.327 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000119517 2024-09-21T14:10:20.327 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.327 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.327 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,7", 2024-09-21T14:10:20.327 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.612090+0000", 2024-09-21T14:10:20.327 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4495999999999998e-05 2024-09-21T14:10:20.327 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.327 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.327 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:20.327 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.619609+0000", 2024-09-21T14:10:20.327 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0075196880000000001 2024-09-21T14:10:20.327 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.328 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.328 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:10:20.328 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.619628+0000", 2024-09-21T14:10:20.328 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8425000000000001e-05 2024-09-21T14:10:20.328 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.328 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.328 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:20.328 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.626706+0000", 2024-09-21T14:10:20.328 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0070784200000000002 2024-09-21T14:10:20.328 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.328 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.328 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:10:20.328 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.626757+0000", 2024-09-21T14:10:20.328 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0977999999999999e-05 2024-09-21T14:10:20.329 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.329 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.329 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:20.329 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:04:56.626802+0000", 2024-09-21T14:10:20.329 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5107e-05 2024-09-21T14:10:20.329 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:20.329 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:20.329 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:20.329 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.329 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.329 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:114620 11.2 11:476bf97d:::benchmark_data_smithi049_206132_object114619:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e496)", 2024-09-21T14:10:20.329 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:08:13.397846+0000", 2024-09-21T14:10:20.330 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 126.898442571, 2024-09-21T14:10:20.330 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.072462561, 2024-09-21T14:10:20.330 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:20.330 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:20.330 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T14:10:20.330 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:10:20.330 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:10:20.330 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:20.330 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 114620 2024-09-21T14:10:20.330 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.330 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:20.330 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.330 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:20.330 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.397846+0000", 2024-09-21T14:10:20.331 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:20.331 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.331 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.331 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:20.331 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.397846+0000", 2024-09-21T14:10:20.331 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:20.331 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.331 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.331 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:20.331 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.397850+0000", 2024-09-21T14:10:20.331 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3340000000000002e-06 2024-09-21T14:10:20.331 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.331 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.332 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:20.332 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.397885+0000", 2024-09-21T14:10:20.332 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5642999999999997e-05 2024-09-21T14:10:20.332 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.332 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.332 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:20.332 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.397911+0000", 2024-09-21T14:10:20.332 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6035000000000001e-05 2024-09-21T14:10:20.332 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.332 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.332 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:20.332 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.397924+0000", 2024-09-21T14:10:20.332 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2772000000000001e-05 2024-09-21T14:10:20.332 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.333 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.333 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:20.333 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.398113+0000", 2024-09-21T14:10:20.333 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00018851499999999999 2024-09-21T14:10:20.333 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.333 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.333 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:10:20.333 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.398128+0000", 2024-09-21T14:10:20.333 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5727999999999999e-05 2024-09-21T14:10:20.333 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.333 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.333 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:20.333 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:14.392816+0000", 2024-09-21T14:10:20.334 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.99468794699999996 2024-09-21T14:10:20.334 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.334 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.334 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:10:20.334 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:14.392829+0000", 2024-09-21T14:10:20.334 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3132999999999999e-05 2024-09-21T14:10:20.334 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.334 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.334 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:20.334 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.469000+0000", 2024-09-21T14:10:20.334 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0761709930000001 2024-09-21T14:10:20.334 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.334 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.334 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:20.335 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.470309+0000", 2024-09-21T14:10:20.335 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0013084609999999999 2024-09-21T14:10:20.335 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:20.335 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:20.335 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:20.335 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.335 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.335 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:114639 11.2 11:40d27929:::benchmark_data_smithi049_206132_object114638:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e496)", 2024-09-21T14:10:20.335 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:08:13.414322+0000", 2024-09-21T14:10:20.335 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 126.881966433, 2024-09-21T14:10:20.335 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0772069179999999, 2024-09-21T14:10:20.335 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:20.335 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:20.336 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:20.336 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:10:20.336 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:10:20.336 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:20.336 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 114639 2024-09-21T14:10:20.336 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.336 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:20.336 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.336 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:20.336 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.414322+0000", 2024-09-21T14:10:20.336 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:20.336 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.336 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.336 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:20.336 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.414322+0000", 2024-09-21T14:10:20.337 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:20.337 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.337 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.337 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:20.337 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.414325+0000", 2024-09-21T14:10:20.337 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.137e-06 2024-09-21T14:10:20.337 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.337 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.337 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:20.337 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.414357+0000", 2024-09-21T14:10:20.337 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2638000000000002e-05 2024-09-21T14:10:20.337 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.337 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.337 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:20.338 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.414380+0000", 2024-09-21T14:10:20.338 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2813999999999999e-05 2024-09-21T14:10:20.338 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.338 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.338 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:20.338 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.414391+0000", 2024-09-21T14:10:20.338 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0817e-05 2024-09-21T14:10:20.338 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.338 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.338 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:20.338 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.416468+0000", 2024-09-21T14:10:20.338 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0020774190000000001 2024-09-21T14:10:20.338 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.338 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.339 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:10:20.339 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.416508+0000", 2024-09-21T14:10:20.339 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9650999999999997e-05 2024-09-21T14:10:20.339 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.339 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.339 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:20.339 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:14.392856+0000", 2024-09-21T14:10:20.339 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.976347721 2024-09-21T14:10:20.339 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.339 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.339 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:10:20.339 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:14.392864+0000", 2024-09-21T14:10:20.339 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8720000000000002e-06 2024-09-21T14:10:20.339 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.340 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.340 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:20.340 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.470610+0000", 2024-09-21T14:10:20.340 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0777461859999999 2024-09-21T14:10:20.340 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.340 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.340 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:20.340 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.471012+0000", 2024-09-21T14:10:20.340 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00040198799999999999 2024-09-21T14:10:20.340 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.340 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.340 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,3,7", 2024-09-21T14:10:20.340 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.471118+0000", 2024-09-21T14:10:20.341 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000106474 2024-09-21T14:10:20.341 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.341 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.341 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:20.341 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.476674+0000", 2024-09-21T14:10:20.341 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0055553030000000002 2024-09-21T14:10:20.341 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.341 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.341 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:10:20.341 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.480688+0000", 2024-09-21T14:10:20.341 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0040147519999999999 2024-09-21T14:10:20.341 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.341 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.342 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:20.342 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.480737+0000", 2024-09-21T14:10:20.342 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9024999999999999e-05 2024-09-21T14:10:20.342 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.342 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.342 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:20.342 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.491326+0000", 2024-09-21T14:10:20.342 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.010588444000000001 2024-09-21T14:10:20.342 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.342 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.343 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:10:20.343 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.491396+0000", 2024-09-21T14:10:20.343 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9957999999999997e-05 2024-09-21T14:10:20.343 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.343 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.344 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:20.344 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.491529+0000", 2024-09-21T14:10:20.344 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000133719 2024-09-21T14:10:20.344 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:20.344 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:20.344 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:20.344 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.344 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.344 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:114643 11.2 11:4163170e:::benchmark_data_smithi049_206132_object114642:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e496)", 2024-09-21T14:10:20.344 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:08:13.424426+0000", 2024-09-21T14:10:20.344 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 126.871862959, 2024-09-21T14:10:20.344 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0720125490000001, 2024-09-21T14:10:20.344 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:20.345 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:20.345 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:20.345 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:10:20.345 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:10:20.345 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:20.345 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 114643 2024-09-21T14:10:20.345 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.345 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:20.345 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.345 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:20.345 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.424426+0000", 2024-09-21T14:10:20.345 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:20.345 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.345 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.346 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:20.346 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.424426+0000", 2024-09-21T14:10:20.346 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:20.346 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.346 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.346 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:20.346 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.424428+0000", 2024-09-21T14:10:20.346 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9300000000000002e-06 2024-09-21T14:10:20.346 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.346 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.346 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:20.346 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.424454+0000", 2024-09-21T14:10:20.347 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6415999999999999e-05 2024-09-21T14:10:20.347 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.347 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.347 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:20.347 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.424468+0000", 2024-09-21T14:10:20.347 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4059e-05 2024-09-21T14:10:20.347 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.347 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.347 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:20.347 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.424477+0000", 2024-09-21T14:10:20.347 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.789e-06 2024-09-21T14:10:20.347 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.347 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.348 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:20.348 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.424745+0000", 2024-09-21T14:10:20.348 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00026768400000000002 2024-09-21T14:10:20.348 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.348 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.348 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:10:20.348 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.424757+0000", 2024-09-21T14:10:20.348 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1827999999999999e-05 2024-09-21T14:10:20.348 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.348 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.348 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:20.348 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:14.392888+0000", 2024-09-21T14:10:20.348 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.96813083499999997 2024-09-21T14:10:20.349 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.349 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.349 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:10:20.349 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:14.392895+0000", 2024-09-21T14:10:20.349 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8019999999999992e-06 2024-09-21T14:10:20.349 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.349 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.349 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:20.349 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.473705+0000", 2024-09-21T14:10:20.349 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0808099389999999 2024-09-21T14:10:20.349 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.349 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.349 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:20.349 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.473854+0000", 2024-09-21T14:10:20.350 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014908 2024-09-21T14:10:20.350 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.350 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.350 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,3,7", 2024-09-21T14:10:20.350 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.474013+0000", 2024-09-21T14:10:20.350 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000158448 2024-09-21T14:10:20.350 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.350 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.350 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:20.350 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.476717+0000", 2024-09-21T14:10:20.350 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0027038740000000002 2024-09-21T14:10:20.350 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.350 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.351 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:10:20.351 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.480695+0000", 2024-09-21T14:10:20.351 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0039782050000000003 2024-09-21T14:10:20.351 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.351 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.351 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:20.351 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.491683+0000", 2024-09-21T14:10:20.351 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.010987702 2024-09-21T14:10:20.351 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.351 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.351 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:20.351 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.496198+0000", 2024-09-21T14:10:20.351 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0045154449999999999 2024-09-21T14:10:20.352 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.352 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.352 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:10:20.352 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.496322+0000", 2024-09-21T14:10:20.352 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012440899999999999 2024-09-21T14:10:20.352 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.352 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.352 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:20.352 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.496439+0000", 2024-09-21T14:10:20.352 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000116104 2024-09-21T14:10:20.352 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:20.352 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:20.352 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:20.352 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.353 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.353 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:114648 11.2 11:4b2f3ec4:::benchmark_data_smithi049_206132_object114647:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e496)", 2024-09-21T14:10:20.353 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:08:13.432129+0000", 2024-09-21T14:10:20.353 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 126.864159536, 2024-09-21T14:10:20.353 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0645723679999999, 2024-09-21T14:10:20.353 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:20.353 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:20.353 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:20.353 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:10:20.353 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:10:20.353 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:20.353 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 114648 2024-09-21T14:10:20.353 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.353 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:20.354 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.354 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:20.354 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.432129+0000", 2024-09-21T14:10:20.354 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:20.354 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.354 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.354 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:20.354 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.432129+0000", 2024-09-21T14:10:20.354 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:20.354 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.354 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.354 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:20.354 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.432132+0000", 2024-09-21T14:10:20.354 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2369999999999999e-06 2024-09-21T14:10:20.355 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.355 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.355 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:20.355 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.432159+0000", 2024-09-21T14:10:20.355 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.6917e-05 2024-09-21T14:10:20.355 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.355 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.355 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:20.355 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.432183+0000", 2024-09-21T14:10:20.355 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4331000000000001e-05 2024-09-21T14:10:20.355 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.355 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.355 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:20.355 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.432193+0000", 2024-09-21T14:10:20.356 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0302999999999999e-05 2024-09-21T14:10:20.356 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.356 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.356 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:20.356 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.433016+0000", 2024-09-21T14:10:20.356 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000823189 2024-09-21T14:10:20.356 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.356 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.356 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:10:20.356 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:13.433070+0000", 2024-09-21T14:10:20.356 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3266000000000001e-05 2024-09-21T14:10:20.356 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.356 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.356 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:20.356 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:14.392920+0000", 2024-09-21T14:10:20.357 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.95985063800000003 2024-09-21T14:10:20.357 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.357 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.357 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:10:20.357 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:14.392928+0000", 2024-09-21T14:10:20.357 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6120000000000001e-06 2024-09-21T14:10:20.357 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.357 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.357 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:20.357 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.476748+0000", 2024-09-21T14:10:20.357 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.083820094 2024-09-21T14:10:20.357 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.357 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.358 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:20.358 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.476929+0000", 2024-09-21T14:10:20.358 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000181435 2024-09-21T14:10:20.358 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.358 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.358 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,3,7", 2024-09-21T14:10:20.358 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.477085+0000", 2024-09-21T14:10:20.358 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00015513899999999999 2024-09-21T14:10:20.358 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.358 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.358 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:10:20.358 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.480749+0000", 2024-09-21T14:10:20.359 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0036649040000000001 2024-09-21T14:10:20.359 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.359 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.359 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:20.359 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.491737+0000", 2024-09-21T14:10:20.359 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.010987215 2024-09-21T14:10:20.359 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.359 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.359 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:20.359 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.491777+0000", 2024-09-21T14:10:20.359 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9966000000000003e-05 2024-09-21T14:10:20.359 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.359 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.359 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:20.359 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.496480+0000", 2024-09-21T14:10:20.360 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0047029139999999999 2024-09-21T14:10:20.360 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.360 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.360 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:10:20.360 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.496583+0000", 2024-09-21T14:10:20.360 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000103933 2024-09-21T14:10:20.360 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.360 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.360 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:20.360 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:15.496702+0000", 2024-09-21T14:10:20.360 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000118275 2024-09-21T14:10:20.360 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:20.360 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:20.361 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:20.361 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.361 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.361 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:185998 11.6 11:6b31a4c0:::benchmark_data_smithi049_206132_object62619:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e524)", 2024-09-21T14:10:20.361 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:09:30.261623+0000", 2024-09-21T14:10:20.361 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 50.034666004999998, 2024-09-21T14:10:20.361 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0612680320000001, 2024-09-21T14:10:20.361 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:20.361 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:20.361 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T14:10:20.361 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:10:20.361 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:10:20.362 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:20.362 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 185998 2024-09-21T14:10:20.362 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.362 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:20.362 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.362 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:20.362 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.261623+0000", 2024-09-21T14:10:20.362 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:20.362 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.362 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.362 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:20.362 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.261623+0000", 2024-09-21T14:10:20.362 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:20.362 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.363 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.363 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:20.363 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.261624+0000", 2024-09-21T14:10:20.363 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3659999999999999e-06 2024-09-21T14:10:20.363 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.363 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.363 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:20.363 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.261630+0000", 2024-09-21T14:10:20.363 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2240000000000003e-06 2024-09-21T14:10:20.363 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.363 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.363 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:20.363 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.261642+0000", 2024-09-21T14:10:20.363 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2482e-05 2024-09-21T14:10:20.363 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.364 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.364 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:20.364 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.261650+0000", 2024-09-21T14:10:20.364 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6769999999999995e-06 2024-09-21T14:10:20.364 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.364 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.364 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:20.364 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.273208+0000", 2024-09-21T14:10:20.364 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.011557878000000001 2024-09-21T14:10:20.364 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.364 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.364 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:10:20.364 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.273217+0000", 2024-09-21T14:10:20.364 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6199999999999994e-06 2024-09-21T14:10:20.365 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.365 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.365 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:20.365 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.315244+0000", 2024-09-21T14:10:20.365 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.042026559 2024-09-21T14:10:20.365 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.365 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.365 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for missing object", 2024-09-21T14:10:20.365 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.315337+0000", 2024-09-21T14:10:20.365 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.3014000000000001e-05 2024-09-21T14:10:20.365 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.365 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.365 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:20.365 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.322844+0000", 2024-09-21T14:10:20.366 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.007507657 2024-09-21T14:10:20.366 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.366 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.366 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:20.366 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.322891+0000", 2024-09-21T14:10:20.366 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6554999999999997e-05 2024-09-21T14:10:20.366 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:20.366 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:20.366 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:20.366 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.366 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.366 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:186001 11.6 11:68d2c960:::benchmark_data_smithi049_206132_object62622:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e524)", 2024-09-21T14:10:20.366 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:09:30.261670+0000", 2024-09-21T14:10:20.366 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 50.034618674000001, 2024-09-21T14:10:20.367 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0732705220000001, 2024-09-21T14:10:20.367 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:20.367 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:20.367 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T14:10:20.367 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:10:20.367 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:10:20.367 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:20.367 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 186001 2024-09-21T14:10:20.367 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.367 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:20.367 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.367 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:20.367 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.261670+0000", 2024-09-21T14:10:20.368 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:20.368 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.368 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.368 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:20.368 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.261670+0000", 2024-09-21T14:10:20.368 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:20.368 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.368 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.368 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:20.368 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.261671+0000", 2024-09-21T14:10:20.368 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9299999999999998e-07 2024-09-21T14:10:20.368 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.368 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.368 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:20.369 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.261674+0000", 2024-09-21T14:10:20.369 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.8100000000000002e-06 2024-09-21T14:10:20.369 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.369 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.369 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:20.369 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.261681+0000", 2024-09-21T14:10:20.369 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0149999999999998e-06 2024-09-21T14:10:20.369 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.369 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.369 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:20.369 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.261686+0000", 2024-09-21T14:10:20.369 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3820000000000003e-06 2024-09-21T14:10:20.369 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.369 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.369 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:20.370 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.273244+0000", 2024-09-21T14:10:20.370 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.011557883 2024-09-21T14:10:20.370 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.370 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.370 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:10:20.370 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.273250+0000", 2024-09-21T14:10:20.370 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0079999999999999e-06 2024-09-21T14:10:20.370 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.370 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.370 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:20.370 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.322905+0000", 2024-09-21T14:10:20.370 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0496556379999999 2024-09-21T14:10:20.370 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.370 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.370 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for missing object", 2024-09-21T14:10:20.371 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.322991+0000", 2024-09-21T14:10:20.371 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5061000000000004e-05 2024-09-21T14:10:20.371 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.371 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.371 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:20.372 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.334860+0000", 2024-09-21T14:10:20.372 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.011869773 2024-09-21T14:10:20.372 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.372 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.372 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:20.372 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.334941+0000", 2024-09-21T14:10:20.372 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0458999999999997e-05 2024-09-21T14:10:20.373 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:20.373 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:20.373 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:20.373 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.373 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.373 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:186007 11.6 11:6c046601:::benchmark_data_smithi049_206132_object62628:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e524)", 2024-09-21T14:10:20.373 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:09:30.261693+0000", 2024-09-21T14:10:20.373 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 50.034596215000001, 2024-09-21T14:10:20.373 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.072563368, 2024-09-21T14:10:20.373 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:20.373 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:20.373 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:20.373 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:10:20.374 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:10:20.374 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:20.374 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 186007 2024-09-21T14:10:20.374 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.374 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:20.374 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.374 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:20.374 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.261693+0000", 2024-09-21T14:10:20.374 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:20.374 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.374 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.374 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:20.374 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.261693+0000", 2024-09-21T14:10:20.374 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:20.375 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.375 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.375 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:20.375 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.261693+0000", 2024-09-21T14:10:20.375 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6899999999999998e-07 2024-09-21T14:10:20.375 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.375 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.375 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:20.375 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.261695+0000", 2024-09-21T14:10:20.375 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9829999999999999e-06 2024-09-21T14:10:20.375 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.375 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.376 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:20.376 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.261701+0000", 2024-09-21T14:10:20.376 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9200000000000001e-06 2024-09-21T14:10:20.376 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.376 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.376 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:20.376 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.261706+0000", 2024-09-21T14:10:20.376 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6059999999999996e-06 2024-09-21T14:10:20.376 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.376 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.376 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:20.376 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.273309+0000", 2024-09-21T14:10:20.376 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.011603082000000001 2024-09-21T14:10:20.376 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.377 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.377 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:10:20.377 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.273315+0000", 2024-09-21T14:10:20.377 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6100000000000002e-06 2024-09-21T14:10:20.377 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.377 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.377 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:20.377 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.323021+0000", 2024-09-21T14:10:20.377 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0497059230000001 2024-09-21T14:10:20.377 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.377 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.377 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:20.377 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.323138+0000", 2024-09-21T14:10:20.378 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011641099999999999 2024-09-21T14:10:20.378 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.378 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.378 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,2", 2024-09-21T14:10:20.378 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.323266+0000", 2024-09-21T14:10:20.378 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012815 2024-09-21T14:10:20.378 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.378 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.378 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:20.378 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.330018+0000", 2024-09-21T14:10:20.378 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0067520619999999996 2024-09-21T14:10:20.378 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.378 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.379 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:20.379 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.330114+0000", 2024-09-21T14:10:20.379 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.6204000000000003e-05 2024-09-21T14:10:20.379 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.379 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.379 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:10:20.379 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.334098+0000", 2024-09-21T14:10:20.379 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0039835540000000003 2024-09-21T14:10:20.379 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.379 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.379 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:10:20.379 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.334157+0000", 2024-09-21T14:10:20.379 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9855999999999997e-05 2024-09-21T14:10:20.379 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.379 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.380 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:20.380 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.334256+0000", 2024-09-21T14:10:20.380 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8638e-05 2024-09-21T14:10:20.380 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:20.380 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:20.380 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:20.380 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.380 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.380 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:186013 11.6 11:62b72c04:::benchmark_data_smithi049_206132_object62634:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e524)", 2024-09-21T14:10:20.380 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:09:30.274235+0000", 2024-09-21T14:10:20.380 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 50.022053544999999, 2024-09-21T14:10:20.380 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0608719790000001, 2024-09-21T14:10:20.380 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:20.380 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:20.381 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:20.381 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:10:20.381 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:10:20.381 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:20.381 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 186013 2024-09-21T14:10:20.381 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.381 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:20.381 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.381 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:20.381 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.274235+0000", 2024-09-21T14:10:20.381 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:20.381 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.381 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.382 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:20.382 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.274235+0000", 2024-09-21T14:10:20.382 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:20.382 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.382 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.382 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:20.382 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.274237+0000", 2024-09-21T14:10:20.382 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.575e-06 2024-09-21T14:10:20.382 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.382 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.382 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:20.382 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.274242+0000", 2024-09-21T14:10:20.382 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6439999999999999e-06 2024-09-21T14:10:20.383 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.383 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.383 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:20.383 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.274255+0000", 2024-09-21T14:10:20.383 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2907e-05 2024-09-21T14:10:20.383 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.383 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.383 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:20.383 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.274263+0000", 2024-09-21T14:10:20.383 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1950000000000003e-06 2024-09-21T14:10:20.383 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.383 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.383 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:20.383 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.274315+0000", 2024-09-21T14:10:20.384 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2633999999999998e-05 2024-09-21T14:10:20.384 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.384 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.384 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:10:20.384 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.274324+0000", 2024-09-21T14:10:20.384 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5040000000000002e-06 2024-09-21T14:10:20.384 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.384 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.384 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:20.384 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.325350+0000", 2024-09-21T14:10:20.384 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0510257670000001 2024-09-21T14:10:20.384 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.384 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.385 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:20.385 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.325486+0000", 2024-09-21T14:10:20.385 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000136062 2024-09-21T14:10:20.385 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.385 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.385 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,2", 2024-09-21T14:10:20.385 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.325628+0000", 2024-09-21T14:10:20.385 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000142516 2024-09-21T14:10:20.385 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.385 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.385 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:20.385 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.330065+0000", 2024-09-21T14:10:20.385 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0044365819999999997 2024-09-21T14:10:20.385 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.386 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.386 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:10:20.386 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.334360+0000", 2024-09-21T14:10:20.386 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0042952440000000001 2024-09-21T14:10:20.386 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.386 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.386 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:20.386 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.334979+0000", 2024-09-21T14:10:20.386 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00061849400000000003 2024-09-21T14:10:20.386 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.386 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.386 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:10:20.387 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.335023+0000", 2024-09-21T14:10:20.387 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4783000000000001e-05 2024-09-21T14:10:20.387 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.387 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.387 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:20.387 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.335107+0000", 2024-09-21T14:10:20.387 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4072000000000004e-05 2024-09-21T14:10:20.387 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:20.387 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:20.387 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:20.387 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.387 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.387 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:186028 11.6 11:61f42831:::benchmark_data_smithi049_206132_object62649:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e524)", 2024-09-21T14:10:20.387 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:09:30.289012+0000", 2024-09-21T14:10:20.388 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 50.007277094999999, 2024-09-21T14:10:20.388 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0463098550000001, 2024-09-21T14:10:20.388 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:20.388 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:20.388 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:20.388 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:10:20.388 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:10:20.388 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:20.388 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 186028 2024-09-21T14:10:20.388 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.388 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:20.388 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.388 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:20.388 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.289012+0000", 2024-09-21T14:10:20.389 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:20.389 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.389 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.389 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:20.389 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.289012+0000", 2024-09-21T14:10:20.389 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:20.389 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.389 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.389 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:20.389 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.289013+0000", 2024-09-21T14:10:20.389 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.485e-06 2024-09-21T14:10:20.389 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.389 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.390 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:20.390 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.289020+0000", 2024-09-21T14:10:20.390 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5899999999999996e-06 2024-09-21T14:10:20.390 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.390 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.390 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:20.390 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.289036+0000", 2024-09-21T14:10:20.390 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6141999999999999e-05 2024-09-21T14:10:20.390 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.390 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.390 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:20.390 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.289045+0000", 2024-09-21T14:10:20.390 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.67e-06 2024-09-21T14:10:20.390 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.391 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.391 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:20.391 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.289111+0000", 2024-09-21T14:10:20.391 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6180999999999995e-05 2024-09-21T14:10:20.391 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.391 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.391 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:10:20.391 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.289122+0000", 2024-09-21T14:10:20.391 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0766e-05 2024-09-21T14:10:20.391 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.391 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.391 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:20.391 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.327757+0000", 2024-09-21T14:10:20.391 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0386351869999999 2024-09-21T14:10:20.392 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.392 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.392 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:20.392 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.327884+0000", 2024-09-21T14:10:20.392 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012757999999999999 2024-09-21T14:10:20.392 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.392 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.392 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,2", 2024-09-21T14:10:20.392 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.328003+0000", 2024-09-21T14:10:20.392 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000118138 2024-09-21T14:10:20.392 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.392 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.392 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:10:20.392 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.334373+0000", 2024-09-21T14:10:20.393 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.006370728 2024-09-21T14:10:20.393 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.393 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.393 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:20.393 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.334431+0000", 2024-09-21T14:10:20.393 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7361000000000002e-05 2024-09-21T14:10:20.393 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.393 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.393 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:20.393 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.335186+0000", 2024-09-21T14:10:20.393 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00075530599999999997 2024-09-21T14:10:20.393 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.393 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.393 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:10:20.394 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.335237+0000", 2024-09-21T14:10:20.394 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0946999999999999e-05 2024-09-21T14:10:20.394 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.394 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.394 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:20.394 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.335322+0000", 2024-09-21T14:10:20.394 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4773999999999998e-05 2024-09-21T14:10:20.394 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:20.394 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:20.394 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:20.394 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.394 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.395 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:186035 11.6 11:6b7279cf:::benchmark_data_smithi049_206132_object62656:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e524)", 2024-09-21T14:10:20.395 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:09:30.294312+0000", 2024-09-21T14:10:20.395 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 50.001977013999998, 2024-09-21T14:10:20.395 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.041241064, 2024-09-21T14:10:20.395 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:20.395 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:20.395 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:20.395 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:10:20.395 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:10:20.395 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:20.395 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 186035 2024-09-21T14:10:20.395 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.395 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:20.395 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.395 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:20.396 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.294312+0000", 2024-09-21T14:10:20.396 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:20.396 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.396 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.396 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:20.396 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.294312+0000", 2024-09-21T14:10:20.396 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:20.396 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.396 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.396 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:20.396 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.294313+0000", 2024-09-21T14:10:20.396 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5629999999999999e-06 2024-09-21T14:10:20.396 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.396 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.396 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:20.397 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.294320+0000", 2024-09-21T14:10:20.397 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5050000000000004e-06 2024-09-21T14:10:20.397 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.397 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.397 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:20.397 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.294332+0000", 2024-09-21T14:10:20.397 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2004999999999999e-05 2024-09-21T14:10:20.397 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.397 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.397 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:20.397 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.294402+0000", 2024-09-21T14:10:20.397 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9966000000000001e-05 2024-09-21T14:10:20.397 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.397 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.398 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:20.398 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.298656+0000", 2024-09-21T14:10:20.398 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.004253639 2024-09-21T14:10:20.398 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.398 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.398 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:10:20.398 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.298664+0000", 2024-09-21T14:10:20.398 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4319999999999999e-06 2024-09-21T14:10:20.398 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.399 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.399 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:20.399 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.330149+0000", 2024-09-21T14:10:20.399 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0314849820000001 2024-09-21T14:10:20.399 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.399 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.399 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:20.399 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.330270+0000", 2024-09-21T14:10:20.399 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000121153 2024-09-21T14:10:20.399 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.399 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.399 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,2", 2024-09-21T14:10:20.399 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.330400+0000", 2024-09-21T14:10:20.399 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013010300000000001 2024-09-21T14:10:20.400 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.400 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.400 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:10:20.400 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.334381+0000", 2024-09-21T14:10:20.400 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.003980918 2024-09-21T14:10:20.400 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.400 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.400 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:20.400 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.335144+0000", 2024-09-21T14:10:20.400 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00076326300000000003 2024-09-21T14:10:20.400 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.401 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.401 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:20.401 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.335392+0000", 2024-09-21T14:10:20.401 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00024775900000000001 2024-09-21T14:10:20.401 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.401 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.401 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:10:20.401 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.335458+0000", 2024-09-21T14:10:20.401 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5697999999999993e-05 2024-09-21T14:10:20.401 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.402 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.402 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:20.402 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.335553+0000", 2024-09-21T14:10:20.402 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.5078e-05 2024-09-21T14:10:20.402 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:20.402 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:20.402 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:20.402 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.402 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.402 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:186042 11.6 11:62bd9e2e:::benchmark_data_smithi049_206132_object62663:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e524)", 2024-09-21T14:10:20.402 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:09:30.299806+0000", 2024-09-21T14:10:20.402 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 49.996483197000003, 2024-09-21T14:10:20.402 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.03619152, 2024-09-21T14:10:20.403 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:20.403 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:20.403 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:20.403 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:10:20.403 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:10:20.403 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:20.403 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 186042 2024-09-21T14:10:20.403 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.403 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:20.403 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.403 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:20.403 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.299806+0000", 2024-09-21T14:10:20.403 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:20.403 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.404 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.404 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:20.404 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.299806+0000", 2024-09-21T14:10:20.404 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:20.404 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.404 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.404 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:20.404 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.299808+0000", 2024-09-21T14:10:20.404 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3939999999999999e-06 2024-09-21T14:10:20.404 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.404 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.404 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:20.404 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.299815+0000", 2024-09-21T14:10:20.405 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1459999999999999e-06 2024-09-21T14:10:20.405 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.405 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.405 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:20.405 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.299835+0000", 2024-09-21T14:10:20.405 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9564000000000001e-05 2024-09-21T14:10:20.405 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.405 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.405 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:20.405 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.299844+0000", 2024-09-21T14:10:20.405 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.465e-06 2024-09-21T14:10:20.405 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.405 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.405 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:20.405 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.299925+0000", 2024-09-21T14:10:20.406 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0632999999999999e-05 2024-09-21T14:10:20.406 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.406 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.406 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:10:20.406 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.299938+0000", 2024-09-21T14:10:20.406 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3179999999999999e-05 2024-09-21T14:10:20.406 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.406 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.406 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:20.406 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.332342+0000", 2024-09-21T14:10:20.406 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0324041989999999 2024-09-21T14:10:20.406 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.406 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.407 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:20.407 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.332480+0000", 2024-09-21T14:10:20.407 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000137205 2024-09-21T14:10:20.407 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.407 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.407 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,2", 2024-09-21T14:10:20.407 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.332584+0000", 2024-09-21T14:10:20.407 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000104712 2024-09-21T14:10:20.407 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.407 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.407 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:10:20.407 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.334442+0000", 2024-09-21T14:10:20.407 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0018574679999999999 2024-09-21T14:10:20.407 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.408 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.408 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:20.408 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.335357+0000", 2024-09-21T14:10:20.408 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00091554400000000001 2024-09-21T14:10:20.408 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.408 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.408 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:20.408 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.335629+0000", 2024-09-21T14:10:20.408 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00027166799999999997 2024-09-21T14:10:20.408 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.408 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.408 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:10:20.408 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.335867+0000", 2024-09-21T14:10:20.408 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00023826199999999999 2024-09-21T14:10:20.408 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.409 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.409 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:20.409 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.335997+0000", 2024-09-21T14:10:20.409 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013008 2024-09-21T14:10:20.409 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:20.409 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:20.409 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:20.409 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.409 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.409 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:186104 11.6 11:69d22ff3:::benchmark_data_smithi049_206132_object62725:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e524)", 2024-09-21T14:10:20.409 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:09:30.389044+0000", 2024-09-21T14:10:20.409 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 49.907244532, 2024-09-21T14:10:20.409 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.96080045599999997, 2024-09-21T14:10:20.409 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:20.410 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:20.410 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:20.410 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:10:20.410 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:10:20.410 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:20.410 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 186104 2024-09-21T14:10:20.410 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.410 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:20.410 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.410 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:20.410 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.389044+0000", 2024-09-21T14:10:20.410 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:20.411 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.411 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.411 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:20.411 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.389044+0000", 2024-09-21T14:10:20.411 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:20.411 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.411 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.411 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:20.411 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.389046+0000", 2024-09-21T14:10:20.411 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.19e-06 2024-09-21T14:10:20.411 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.411 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.411 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:20.411 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.389050+0000", 2024-09-21T14:10:20.411 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8609999999999997e-06 2024-09-21T14:10:20.412 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.412 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.412 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:20.412 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.389065+0000", 2024-09-21T14:10:20.412 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4312e-05 2024-09-21T14:10:20.412 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.412 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.412 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:20.412 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.389073+0000", 2024-09-21T14:10:20.412 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7940000000000003e-06 2024-09-21T14:10:20.412 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.412 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.413 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:20.413 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.389135+0000", 2024-09-21T14:10:20.413 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2150999999999995e-05 2024-09-21T14:10:20.413 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.413 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.413 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:10:20.413 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.389144+0000", 2024-09-21T14:10:20.413 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2969999999999992e-06 2024-09-21T14:10:20.413 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.413 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.413 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:20.413 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.336191+0000", 2024-09-21T14:10:20.413 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.94704689200000003 2024-09-21T14:10:20.413 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.414 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.414 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:20.414 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.336293+0000", 2024-09-21T14:10:20.414 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000102185 2024-09-21T14:10:20.414 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.414 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.414 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,2", 2024-09-21T14:10:20.414 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.336394+0000", 2024-09-21T14:10:20.414 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000101272 2024-09-21T14:10:20.414 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.414 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.414 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:20.414 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.342351+0000", 2024-09-21T14:10:20.415 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0059565979999999996 2024-09-21T14:10:20.415 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.415 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.415 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:20.415 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:20.416 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.349708+0000", 2024-09-21T14:10:20.416 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0073572159999999998 2024-09-21T14:10:20.416 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.416 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.416 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:10:20.416 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.349759+0000", 2024-09-21T14:10:20.416 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0569000000000003e-05 2024-09-21T14:10:20.416 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.416 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.416 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:10:20.416 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.349808+0000", 2024-09-21T14:10:20.416 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9282e-05 2024-09-21T14:10:20.417 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.417 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.417 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:20.417 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.349845+0000", 2024-09-21T14:10:20.417 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6837e-05 2024-09-21T14:10:20.417 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:20.417 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:20.417 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:20.417 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.417 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.417 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:186106 11.6 11:620fa056:::benchmark_data_smithi049_206132_object62727:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e524)", 2024-09-21T14:10:20.417 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:09:30.390361+0000", 2024-09-21T14:10:20.417 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 49.905927595999998, 2024-09-21T14:10:20.418 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.95963843900000001, 2024-09-21T14:10:20.418 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:20.418 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:20.418 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:20.418 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:10:20.418 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:10:20.418 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:20.418 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 186106 2024-09-21T14:10:20.418 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.418 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:20.418 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.418 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:20.418 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.390361+0000", 2024-09-21T14:10:20.419 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:20.419 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.419 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.419 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:20.419 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.390361+0000", 2024-09-21T14:10:20.419 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:20.419 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.419 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.419 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:20.419 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.390362+0000", 2024-09-21T14:10:20.419 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1379999999999999e-06 2024-09-21T14:10:20.419 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.419 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.420 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:20.420 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.390367+0000", 2024-09-21T14:10:20.420 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8339999999999998e-06 2024-09-21T14:10:20.420 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.420 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.420 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:20.420 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.390383+0000", 2024-09-21T14:10:20.420 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5537999999999999e-05 2024-09-21T14:10:20.420 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.420 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.420 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:20.420 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.390390+0000", 2024-09-21T14:10:20.420 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.221e-06 2024-09-21T14:10:20.421 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.421 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.421 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:20.421 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.390441+0000", 2024-09-21T14:10:20.421 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0448999999999998e-05 2024-09-21T14:10:20.421 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.421 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.421 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:10:20.421 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.390448+0000", 2024-09-21T14:10:20.421 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1659999999999997e-06 2024-09-21T14:10:20.421 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.421 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.421 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:20.421 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.338891+0000", 2024-09-21T14:10:20.421 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.94844318599999999 2024-09-21T14:10:20.422 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.422 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.422 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:20.422 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.339014+0000", 2024-09-21T14:10:20.422 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000123363 2024-09-21T14:10:20.422 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.422 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.422 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,2", 2024-09-21T14:10:20.422 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.339131+0000", 2024-09-21T14:10:20.422 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000116839 2024-09-21T14:10:20.422 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.422 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.422 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:20.423 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.346015+0000", 2024-09-21T14:10:20.423 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0068834359999999997 2024-09-21T14:10:20.423 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.423 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.423 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:20.423 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.349749+0000", 2024-09-21T14:10:20.423 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0037340619999999998 2024-09-21T14:10:20.423 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.423 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.423 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:10:20.423 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.349925+0000", 2024-09-21T14:10:20.423 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00017689600000000001 2024-09-21T14:10:20.423 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.423 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.423 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:10:20.424 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.349964+0000", 2024-09-21T14:10:20.424 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9028e-05 2024-09-21T14:10:20.424 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.424 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.424 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:20.424 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.350000+0000", 2024-09-21T14:10:20.424 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5283000000000001e-05 2024-09-21T14:10:20.424 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:20.424 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:20.424 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:20.424 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.424 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.424 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:186123 11.6 11:66bb77f2:::benchmark_data_smithi049_206132_object62744:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e524)", 2024-09-21T14:10:20.425 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:09:30.402727+0000", 2024-09-21T14:10:20.425 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 49.893561781000002, 2024-09-21T14:10:20.425 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.94743138000000005, 2024-09-21T14:10:20.425 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:20.425 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:20.425 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:20.425 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:10:20.425 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:10:20.425 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:20.425 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 186123 2024-09-21T14:10:20.425 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.425 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:20.425 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.425 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:20.426 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.402727+0000", 2024-09-21T14:10:20.426 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:20.426 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.426 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.426 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:20.426 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.402727+0000", 2024-09-21T14:10:20.426 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:20.426 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.426 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.426 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:20.426 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.402728+0000", 2024-09-21T14:10:20.426 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2500000000000005e-07 2024-09-21T14:10:20.426 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.427 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.427 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:20.427 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.402731+0000", 2024-09-21T14:10:20.427 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.0290000000000001e-06 2024-09-21T14:10:20.427 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.427 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.427 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:20.427 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.402739+0000", 2024-09-21T14:10:20.427 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9359999999999999e-06 2024-09-21T14:10:20.427 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.427 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.427 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:20.427 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.402752+0000", 2024-09-21T14:10:20.427 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3078000000000001e-05 2024-09-21T14:10:20.428 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.428 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.428 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:20.428 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.402800+0000", 2024-09-21T14:10:20.428 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8544000000000001e-05 2024-09-21T14:10:20.428 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.428 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.428 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:10:20.428 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.402808+0000", 2024-09-21T14:10:20.428 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0630000000000006e-06 2024-09-21T14:10:20.428 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.428 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.428 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:20.428 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.340740+0000", 2024-09-21T14:10:20.428 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.93793204100000005 2024-09-21T14:10:20.429 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.429 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.429 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:20.429 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.340874+0000", 2024-09-21T14:10:20.429 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013398899999999999 2024-09-21T14:10:20.429 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.429 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.429 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,2", 2024-09-21T14:10:20.429 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.340965+0000", 2024-09-21T14:10:20.429 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0512000000000003e-05 2024-09-21T14:10:20.429 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.429 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.429 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:20.429 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.346064+0000", 2024-09-21T14:10:20.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:20.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:10:20.431 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0050994170000000002 2024-09-21T14:10:20.431 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.431 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.431 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:10:20.431 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.350007+0000", 2024-09-21T14:10:20.431 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0039422729999999996 2024-09-21T14:10:20.431 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.431 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.431 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:20.431 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.350072+0000", 2024-09-21T14:10:20.431 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5629000000000005e-05 2024-09-21T14:10:20.432 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.432 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.432 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:10:20.432 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.350121+0000", 2024-09-21T14:10:20.432 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9094000000000001e-05 2024-09-21T14:10:20.432 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.432 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.432 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:20.432 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.350159+0000", 2024-09-21T14:10:20.432 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7150000000000002e-05 2024-09-21T14:10:20.432 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:20.432 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:20.432 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:20.433 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.433 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.433 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:186128 11.6 11:646ab384:::benchmark_data_smithi049_206132_object62749:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e524)", 2024-09-21T14:10:20.433 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:09:30.407890+0000", 2024-09-21T14:10:20.433 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 49.888398954000003, 2024-09-21T14:10:20.433 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.95238523200000003, 2024-09-21T14:10:20.433 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:20.433 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:20.433 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:20.433 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:10:20.433 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:10:20.433 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:20.433 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 186128 2024-09-21T14:10:20.433 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.434 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:20.434 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.434 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:20.434 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.407890+0000", 2024-09-21T14:10:20.434 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:20.434 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.434 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.434 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:20.434 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.407890+0000", 2024-09-21T14:10:20.434 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:20.434 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.434 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.434 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:20.435 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.407891+0000", 2024-09-21T14:10:20.435 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.248e-06 2024-09-21T14:10:20.435 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.435 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.435 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:20.435 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.407896+0000", 2024-09-21T14:10:20.435 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.707e-06 2024-09-21T14:10:20.435 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.435 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.435 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:20.435 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.407908+0000", 2024-09-21T14:10:20.435 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2472e-05 2024-09-21T14:10:20.435 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.435 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.435 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:20.436 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.407916+0000", 2024-09-21T14:10:20.436 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2899999999999997e-06 2024-09-21T14:10:20.436 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.436 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.436 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:20.436 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.407978+0000", 2024-09-21T14:10:20.436 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2700999999999995e-05 2024-09-21T14:10:20.436 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.436 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.436 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:10:20.436 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.407987+0000", 2024-09-21T14:10:20.436 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.7779999999999996e-06 2024-09-21T14:10:20.436 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.436 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.437 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:20.437 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.342397+0000", 2024-09-21T14:10:20.437 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.93440970199999995 2024-09-21T14:10:20.437 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.437 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.437 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:20.437 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.342515+0000", 2024-09-21T14:10:20.437 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000117688 2024-09-21T14:10:20.437 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.437 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.437 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,2", 2024-09-21T14:10:20.437 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.342629+0000", 2024-09-21T14:10:20.437 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000114364 2024-09-21T14:10:20.437 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.438 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.438 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:10:20.438 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.350012+0000", 2024-09-21T14:10:20.438 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0073825770000000004 2024-09-21T14:10:20.438 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.438 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.438 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:20.438 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.353918+0000", 2024-09-21T14:10:20.438 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0039061790000000001 2024-09-21T14:10:20.438 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.438 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.438 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:20.438 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.360172+0000", 2024-09-21T14:10:20.438 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0062545719999999999 2024-09-21T14:10:20.439 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.439 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.439 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:10:20.439 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.360228+0000", 2024-09-21T14:10:20.439 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6022e-05 2024-09-21T14:10:20.439 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.439 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.439 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:20.439 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.360275+0000", 2024-09-21T14:10:20.439 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6931999999999997e-05 2024-09-21T14:10:20.439 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:20.439 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:20.440 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:20.440 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.440 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.440 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:186138 11.6 11:69f110c1:::benchmark_data_smithi049_206132_object62759:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e524)", 2024-09-21T14:10:20.440 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:09:30.417719+0000", 2024-09-21T14:10:20.440 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 49.878569491999997, 2024-09-21T14:10:20.440 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.94654681299999999, 2024-09-21T14:10:20.440 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:20.440 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:20.440 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:20.440 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:10:20.440 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:10:20.440 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:20.440 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 186138 2024-09-21T14:10:20.441 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.441 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:20.441 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.441 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:20.441 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.417719+0000", 2024-09-21T14:10:20.441 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:20.441 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.441 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.441 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:20.441 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.417719+0000", 2024-09-21T14:10:20.441 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:20.441 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.441 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.441 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:20.442 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.417721+0000", 2024-09-21T14:10:20.442 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.581e-06 2024-09-21T14:10:20.442 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.442 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.442 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:20.442 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.417726+0000", 2024-09-21T14:10:20.442 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1240000000000004e-06 2024-09-21T14:10:20.442 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.442 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.442 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:20.442 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.417740+0000", 2024-09-21T14:10:20.442 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4066000000000001e-05 2024-09-21T14:10:20.442 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.442 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.443 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:20.443 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.417748+0000", 2024-09-21T14:10:20.443 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8820000000000001e-06 2024-09-21T14:10:20.443 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.443 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.443 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:20.443 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.417793+0000", 2024-09-21T14:10:20.443 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5003000000000003e-05 2024-09-21T14:10:20.443 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.443 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.443 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:10:20.443 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:30.417798+0000", 2024-09-21T14:10:20.443 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1560000000000003e-06 2024-09-21T14:10:20.443 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.444 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.444 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:20.444 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.344040+0000", 2024-09-21T14:10:20.444 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.92624145099999999 2024-09-21T14:10:20.444 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.444 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.444 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:20.444 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.344128+0000", 2024-09-21T14:10:20.444 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.8497000000000003e-05 2024-09-21T14:10:20.444 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.444 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.444 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,2", 2024-09-21T14:10:20.444 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.344185+0000", 2024-09-21T14:10:20.444 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6412999999999997e-05 2024-09-21T14:10:20.444 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.444 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.445 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:10:20.445 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.350016+0000", 2024-09-21T14:10:20.445 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0058318609999999998 2024-09-21T14:10:20.445 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.445 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.445 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:20.445 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.353979+0000", 2024-09-21T14:10:20.445 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0039629130000000002 2024-09-21T14:10:20.445 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.445 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.445 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:20.445 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.364083+0000", 2024-09-21T14:10:20.445 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.01010405 2024-09-21T14:10:20.445 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.446 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.446 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:10:20.446 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.364218+0000", 2024-09-21T14:10:20.446 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013459999999999999 2024-09-21T14:10:20.446 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:20.446 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:20.446 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:20.446 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:31.364266+0000", 2024-09-21T14:10:20.446 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8216000000000001e-05 2024-09-21T14:10:20.446 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:20.446 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:20.446 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:20.446 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:20.446 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:20.447 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T14:10:20.475 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 4, 6, 2, 3, 7, 5] 2024-09-21T14:10:20.475 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T14:10:20.476 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-09-21T14:10:20.476 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T14:10:20.476 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T14:10:20.476 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.7 config set filestore_inject_stall 3 2024-09-21T14:10:20.531 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:10:20.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:20.569 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:20.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:10:20.670 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:10:20.694 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:10:20 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:10:20.710 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:20.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:20.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:10:20.812 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:10:20.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:20.844 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:20.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:10:20.918 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:10:20.945 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:10:20.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:20.991 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:20.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:10:21.004 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:10:20 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:10:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:10:21.092 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:10:21.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:21.146 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:21.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:10:21.247 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:10:21.282 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:21.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:21.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:10:21.334 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:21 smithi187 ceph-mon[33032]: 3.13 scrub starts 2024-09-21T14:10:21.334 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:21 smithi187 ceph-mon[33032]: 3.13 scrub ok 2024-09-21T14:10:21.334 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:21 smithi187 ceph-mon[33032]: 4.1f scrub starts 2024-09-21T14:10:21.334 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:21 smithi187 ceph-mon[33032]: 4.1f scrub ok 2024-09-21T14:10:21.334 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:21 smithi187 ceph-mon[33032]: 4.17 scrub starts 2024-09-21T14:10:21.334 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:21 smithi187 ceph-mon[33032]: 4.17 scrub ok 2024-09-21T14:10:21.334 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:21 smithi187 ceph-mon[33032]: pgmap v1098: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 656 MiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 811 B/s rd, 0 B/s wr, 1.21k op/s; 33/32067 objects misplaced (0.103%) 2024-09-21T14:10:21.334 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:21 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:10:21.334 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:21 smithi187 ceph-mon[33032]: osdmap e544: 8 total, 8 up, 8 in 2024-09-21T14:10:21.384 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:10:21.417 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:21.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:21.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:10:21.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:21 smithi049 ceph-mon[30080]: 3.13 scrub starts 2024-09-21T14:10:21.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:21 smithi049 ceph-mon[30080]: 3.13 scrub ok 2024-09-21T14:10:21.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:21 smithi049 ceph-mon[30080]: 4.1f scrub starts 2024-09-21T14:10:21.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:21 smithi049 ceph-mon[30080]: 4.1f scrub ok 2024-09-21T14:10:21.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:21 smithi049 ceph-mon[30080]: 4.17 scrub starts 2024-09-21T14:10:21.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:21 smithi049 ceph-mon[30080]: 4.17 scrub ok 2024-09-21T14:10:21.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:21 smithi049 ceph-mon[30080]: pgmap v1098: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 656 MiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 811 B/s rd, 0 B/s wr, 1.21k op/s; 33/32067 objects misplaced (0.103%) 2024-09-21T14:10:21.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:21 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:10:21.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:21 smithi049 ceph-mon[30080]: osdmap e544: 8 total, 8 up, 8 in 2024-09-21T14:10:21.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:21 smithi049 ceph-mon[40489]: 3.13 scrub starts 2024-09-21T14:10:21.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:21 smithi049 ceph-mon[40489]: 3.13 scrub ok 2024-09-21T14:10:21.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:21 smithi049 ceph-mon[40489]: 4.1f scrub starts 2024-09-21T14:10:21.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:21 smithi049 ceph-mon[40489]: 4.1f scrub ok 2024-09-21T14:10:21.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:21 smithi049 ceph-mon[40489]: 4.17 scrub starts 2024-09-21T14:10:21.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:21 smithi049 ceph-mon[40489]: 4.17 scrub ok 2024-09-21T14:10:21.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:21 smithi049 ceph-mon[40489]: pgmap v1098: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 656 MiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 811 B/s rd, 0 B/s wr, 1.21k op/s; 33/32067 objects misplaced (0.103%) 2024-09-21T14:10:21.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:21 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:10:21.506 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:21 smithi049 ceph-mon[40489]: osdmap e544: 8 total, 8 up, 8 in 2024-09-21T14:10:21.518 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:10:21.549 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:21.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:21.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:10:21.651 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:10:21.661 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T14:10:21.661 INFO:teuthology.orchestra.run.smithi187.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-21T14:10:21.661 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T14:10:21.688 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:21.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:21.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:10:21.790 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:10:21.830 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:21.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:21.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:10:21.931 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:10:21.960 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:21.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:21.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:10:21.968 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T14:10:21.988 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:10:22.061 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:10:22.130 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:22.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:22.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:10:22.232 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:10:22.262 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:22.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:22.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:10:22.364 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:10:22.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:22.418 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:22.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:10:22.433 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:22 smithi187 ceph-mon[33032]: 5.13 scrub starts 2024-09-21T14:10:22.433 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:22 smithi187 ceph-mon[33032]: 5.13 scrub ok 2024-09-21T14:10:22.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:22 smithi049 ceph-mon[30080]: 5.13 scrub starts 2024-09-21T14:10:22.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:22 smithi049 ceph-mon[30080]: 5.13 scrub ok 2024-09-21T14:10:22.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:22 smithi049 ceph-mon[40489]: 5.13 scrub starts 2024-09-21T14:10:22.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:22 smithi049 ceph-mon[40489]: 5.13 scrub ok 2024-09-21T14:10:22.520 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:10:22.552 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:22.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:22.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:10:22.654 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:10:22.696 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:22.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:22.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:10:22.798 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:10:22.803 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:10:22.826 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:22.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:22.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:10:22.928 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:10:22.954 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:22.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:22.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:10:23.056 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:10:23.098 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:23.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:23.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:10:23.105 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T14:10:23.200 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:10:23.226 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:23.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:23.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:10:23.295 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is set 2024-09-21T14:10:23.329 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:10:23.354 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:23.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:23.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:10:23.456 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:10:23.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:23.484 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:23.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:10:23.497 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T14:10:23.497 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-09-21T14:10:23.497 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:23.497 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26396.0:242981 11.b e545/461 11:daf1c768:::benchmark_data_smithi049_206132_object119602:head v 545'14990, mlcod=545'14987)", 2024-09-21T14:10:23.497 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:10:23.486547+0000", 2024-09-21T14:10:23.497 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.0093724610000000003, 2024-09-21T14:10:23.497 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0094023100000000005, 2024-09-21T14:10:23.497 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:23.497 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:23.497 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T14:10:23.497 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:23.497 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:23.497 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:23.498 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:23.486547+0000", 2024-09-21T14:10:23.498 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:23.498 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:23.498 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:23.498 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:23.498 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:23.486547+0000", 2024-09-21T14:10:23.498 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:23.498 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:23.498 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:23.498 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:23.498 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:23.486548+0000", 2024-09-21T14:10:23.498 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.206e-06 2024-09-21T14:10:23.498 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:23.499 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:23.499 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:23.499 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:23.486555+0000", 2024-09-21T14:10:23.499 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7800000000000003e-06 2024-09-21T14:10:23.499 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:23.499 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:23.499 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:23.499 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:23.486566+0000", 2024-09-21T14:10:23.499 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1389999999999999e-05 2024-09-21T14:10:23.499 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:23.499 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:23.499 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:23.499 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:23.486574+0000", 2024-09-21T14:10:23.499 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.216e-06 2024-09-21T14:10:23.500 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:23.500 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:23.500 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:23.500 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:23.487262+0000", 2024-09-21T14:10:23.500 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00068800100000000004 2024-09-21T14:10:23.500 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:23.500 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:23.500 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:23.500 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:23.487278+0000", 2024-09-21T14:10:23.500 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6409000000000002e-05 2024-09-21T14:10:23.500 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:23.500 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:23.500 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:23.501 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:23.501 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:23.501 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26396.0:242987 11.b e545/461 11:d892c254:::benchmark_data_smithi049_206132_object119608:head v 545'14991, mlcod=545'14987)", 2024-09-21T14:10:23.501 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:10:23.490255+0000", 2024-09-21T14:10:23.501 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.00566426, 2024-09-21T14:10:23.501 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0057496450000000003, 2024-09-21T14:10:23.501 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:23.501 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:23.501 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T14:10:23.501 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:23.501 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:23.501 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:23.501 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:23.490255+0000", 2024-09-21T14:10:23.501 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:23.502 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:23.502 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:23.502 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:23.502 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:23.490255+0000", 2024-09-21T14:10:23.502 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:23.502 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:23.502 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:23.502 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:23.502 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:23.490256+0000", 2024-09-21T14:10:23.502 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6899999999999998e-07 2024-09-21T14:10:23.502 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:23.502 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:23.502 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:23.502 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:23.490261+0000", 2024-09-21T14:10:23.503 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7400000000000001e-06 2024-09-21T14:10:23.503 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:23.503 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:23.503 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:23.503 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:23.490272+0000", 2024-09-21T14:10:23.503 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0256e-05 2024-09-21T14:10:23.503 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:23.503 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:23.503 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:23.503 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:23.490277+0000", 2024-09-21T14:10:23.503 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7440000000000004e-06 2024-09-21T14:10:23.503 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:23.503 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:23.503 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:23.503 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:23.492557+0000", 2024-09-21T14:10:23.504 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0022799610000000001 2024-09-21T14:10:23.504 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:23.504 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:23.504 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:23.504 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:23.492582+0000", 2024-09-21T14:10:23.504 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4890000000000001e-05 2024-09-21T14:10:23.504 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:23.504 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:23.504 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:23.504 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:23.504 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:23.504 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:242994 11.c 11:364ef552:::benchmark_data_smithi049_206132_object119615:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e545)", 2024-09-21T14:10:23.504 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:10:23.493706+0000", 2024-09-21T14:10:23.505 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.002213394, 2024-09-21T14:10:23.505 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002353862, 2024-09-21T14:10:23.505 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:23.505 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:23.505 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "waiting for sub ops", 2024-09-21T14:10:23.505 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:10:23.505 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:10:23.505 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:23.505 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 242994 2024-09-21T14:10:23.505 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:23.505 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:23.505 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:23.505 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:23.505 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:23.493706+0000", 2024-09-21T14:10:23.505 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:23.506 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:23.506 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:23.506 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:23.506 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:23.493706+0000", 2024-09-21T14:10:23.506 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:23.506 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:23.506 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:23.506 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:23.506 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:23.493708+0000", 2024-09-21T14:10:23.506 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7600000000000001e-06 2024-09-21T14:10:23.506 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:23.506 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:23.506 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:23.506 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:23.493713+0000", 2024-09-21T14:10:23.507 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3190000000000002e-06 2024-09-21T14:10:23.507 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:23.507 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:23.507 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:23.507 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:23.493727+0000", 2024-09-21T14:10:23.507 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3754000000000001e-05 2024-09-21T14:10:23.507 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:23.507 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:23.507 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:23.507 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:23.493736+0000", 2024-09-21T14:10:23.507 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.8079999999999994e-06 2024-09-21T14:10:23.507 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:23.507 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:23.507 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:23.507 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:23.493808+0000", 2024-09-21T14:10:23.508 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2491999999999996e-05 2024-09-21T14:10:23.508 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:23.508 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:23.508 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:23.508 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:23.493930+0000", 2024-09-21T14:10:23.508 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000121873 2024-09-21T14:10:23.508 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:23.508 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:23.508 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 2,4", 2024-09-21T14:10:23.508 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:23.494050+0000", 2024-09-21T14:10:23.508 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0001199 2024-09-21T14:10:23.508 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:23.508 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:23.508 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:23.509 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:23.509 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:23.509 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26396.0:242992 11.7 e545/461 11:e517176d:::benchmark_data_smithi049_206132_object119613:head v 545'14909, mlcod=545'14907)", 2024-09-21T14:10:23.509 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:10:23.494034+0000", 2024-09-21T14:10:23.509 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.001885814, 2024-09-21T14:10:23.509 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0020877360000000002, 2024-09-21T14:10:23.509 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:23.509 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:23.509 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T14:10:23.509 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:23.509 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:23.509 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:23.509 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:23.494034+0000", 2024-09-21T14:10:23.509 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:23.510 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:23.510 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:23.510 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:23.510 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:23.494034+0000", 2024-09-21T14:10:23.510 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:23.510 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:23.510 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:23.510 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:23.510 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:23.494034+0000", 2024-09-21T14:10:23.510 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5700000000000002e-07 2024-09-21T14:10:23.510 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:23.510 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:23.510 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:23.511 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:23.494044+0000", 2024-09-21T14:10:23.511 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2760000000000006e-06 2024-09-21T14:10:23.511 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:23.511 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:23.511 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:23.511 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:23.494054+0000", 2024-09-21T14:10:23.511 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0117e-05 2024-09-21T14:10:23.511 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:23.511 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:23.511 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:23.511 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:23.494062+0000", 2024-09-21T14:10:23.511 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1750000000000005e-06 2024-09-21T14:10:23.511 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:23.511 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:23.512 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:23.512 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:23.494110+0000", 2024-09-21T14:10:23.512 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8315000000000001e-05 2024-09-21T14:10:23.512 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:23.512 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:23.512 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:23.512 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:23.494130+0000", 2024-09-21T14:10:23.512 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9854999999999999e-05 2024-09-21T14:10:23.512 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:23.512 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:23.512 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:23.512 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:23.512 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:23.512 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26396.0:242975 11.b e545/461 11:d1eff80c:::benchmark_data_smithi049_206132_object119596:head v 545'14989, mlcod=545'14987)", 2024-09-21T14:10:23.513 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:10:23.479212+0000", 2024-09-21T14:10:23.513 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.016707632, 2024-09-21T14:10:23.513 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.016959663, 2024-09-21T14:10:23.513 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:23.513 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:23.513 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T14:10:23.513 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:23.513 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:23.513 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:23.513 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:23.479212+0000", 2024-09-21T14:10:23.513 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:23.513 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:23.513 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:23.513 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:23.513 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:23.479212+0000", 2024-09-21T14:10:23.514 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:23.514 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:23.514 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:23.514 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:23.514 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:23.479213+0000", 2024-09-21T14:10:23.514 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.097e-06 2024-09-21T14:10:23.514 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:23.514 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:23.514 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:23.514 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:23.479221+0000", 2024-09-21T14:10:23.514 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6990000000000003e-06 2024-09-21T14:10:23.514 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:23.514 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:23.514 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:23.515 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:23.479233+0000", 2024-09-21T14:10:23.515 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2523e-05 2024-09-21T14:10:23.515 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:23.515 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:23.515 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:23.515 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:23.479241+0000", 2024-09-21T14:10:23.515 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6450000000000005e-06 2024-09-21T14:10:23.515 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:23.515 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:23.515 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:23.515 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:23.479305+0000", 2024-09-21T14:10:23.515 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4188000000000003e-05 2024-09-21T14:10:23.515 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:23.515 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:23.515 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:23.515 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:23.479326+0000", 2024-09-21T14:10:23.516 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0460000000000001e-05 2024-09-21T14:10:23.516 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:23.516 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:23.516 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:23.516 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:23.516 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:23.516 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.26396.0:242984 11.3 e545/461 11:c21ce883:::benchmark_data_smithi049_206132_object119605:head v 545'15374, mlcod=545'15372)", 2024-09-21T14:10:23.516 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:10:23.483903+0000", 2024-09-21T14:10:23.516 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 0.012016169, 2024-09-21T14:10:23.516 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.012316024, 2024-09-21T14:10:23.516 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:23.516 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:23.516 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T14:10:23.516 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:23.516 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:23.517 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:23.517 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:23.483903+0000", 2024-09-21T14:10:23.517 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:23.517 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:23.517 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:23.517 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:23.517 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:23.483903+0000", 2024-09-21T14:10:23.517 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:23.517 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:23.517 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:23.517 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:23.517 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:23.483904+0000", 2024-09-21T14:10:23.517 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3099999999999999e-07 2024-09-21T14:10:23.517 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:23.517 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:23.518 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:23.518 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:23.483907+0000", 2024-09-21T14:10:23.518 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4869999999999998e-06 2024-09-21T14:10:23.518 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:23.518 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:23.518 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:23.518 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:23.483913+0000", 2024-09-21T14:10:23.518 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1249999999999998e-06 2024-09-21T14:10:23.518 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:23.518 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:23.518 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:23.518 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:23.483919+0000", 2024-09-21T14:10:23.518 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9780000000000002e-06 2024-09-21T14:10:23.518 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:23.519 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:23.519 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:23.519 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:23.484803+0000", 2024-09-21T14:10:23.519 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00088336399999999998 2024-09-21T14:10:23.519 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:23.519 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:23.519 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:23.519 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:23.484825+0000", 2024-09-21T14:10:23.519 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.2189e-05 2024-09-21T14:10:23.519 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:23.519 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:23.519 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:23.519 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:23.519 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-09-21T14:10:23.519 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 6 2024-09-21T14:10:23.520 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T14:10:23.585 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:10:23.615 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:23.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:23.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:10:23.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:23 smithi187 ceph-mon[33032]: 1.2 scrub starts 2024-09-21T14:10:23.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:23 smithi187 ceph-mon[33032]: 1.2 scrub ok 2024-09-21T14:10:23.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:23 smithi187 ceph-mon[33032]: 5.3 scrub starts 2024-09-21T14:10:23.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:23 smithi187 ceph-mon[33032]: 5.3 scrub ok 2024-09-21T14:10:23.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:23 smithi187 ceph-mon[33032]: pgmap v1100: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 497 MiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.13k op/s; 20/24426 objects misplaced (0.082%) 2024-09-21T14:10:23.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:23 smithi187 ceph-mon[33032]: 3.7 scrub starts 2024-09-21T14:10:23.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:23 smithi187 ceph-mon[33032]: 3.7 scrub ok 2024-09-21T14:10:23.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:23 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1299032469' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:10:23.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:23 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:10:23.717 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:10:23.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:23 smithi049 ceph-mon[30080]: 1.2 scrub starts 2024-09-21T14:10:23.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:23 smithi049 ceph-mon[30080]: 1.2 scrub ok 2024-09-21T14:10:23.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:23 smithi049 ceph-mon[30080]: 5.3 scrub starts 2024-09-21T14:10:23.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:23 smithi049 ceph-mon[30080]: 5.3 scrub ok 2024-09-21T14:10:23.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:23 smithi049 ceph-mon[30080]: pgmap v1100: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 497 MiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.13k op/s; 20/24426 objects misplaced (0.082%) 2024-09-21T14:10:23.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:23 smithi049 ceph-mon[30080]: 3.7 scrub starts 2024-09-21T14:10:23.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:23 smithi049 ceph-mon[30080]: 3.7 scrub ok 2024-09-21T14:10:23.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:23 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1299032469' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:10:23.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:23 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:10:23.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:23 smithi049 ceph-mon[40489]: 1.2 scrub starts 2024-09-21T14:10:23.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:23 smithi049 ceph-mon[40489]: 1.2 scrub ok 2024-09-21T14:10:23.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:23 smithi049 ceph-mon[40489]: 5.3 scrub starts 2024-09-21T14:10:23.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:23 smithi049 ceph-mon[40489]: 5.3 scrub ok 2024-09-21T14:10:23.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:23 smithi049 ceph-mon[40489]: pgmap v1100: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 497 MiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.13k op/s; 20/24426 objects misplaced (0.082%) 2024-09-21T14:10:23.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:23 smithi049 ceph-mon[40489]: 3.7 scrub starts 2024-09-21T14:10:23.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:23 smithi049 ceph-mon[40489]: 3.7 scrub ok 2024-09-21T14:10:23.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:23 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1299032469' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:10:23.756 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:23 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T14:10:23.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:23.757 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:23.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:10:23.858 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:10:23.891 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:23.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:23.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:10:23.993 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:10:24.022 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:24.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:24.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:10:24.124 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:10:24.161 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:24.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:24.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:10:24.262 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:10:24.295 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:24.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:24.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:10:24.397 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:10:24.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:24.437 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:24.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:10:24.537 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:10:24.567 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:24.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:24.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:10:24.669 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:10:24.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:24 smithi187 ceph-mon[33032]: 3.f scrub starts 2024-09-21T14:10:24.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:24 smithi187 ceph-mon[33032]: 3.f scrub ok 2024-09-21T14:10:24.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:24 smithi187 ceph-mon[33032]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:10:24.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:24 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T14:10:24.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:24 smithi187 ceph-mon[33032]: osdmap e545: 8 total, 8 up, 8 in 2024-09-21T14:10:24.708 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:24.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:24.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:10:24.739 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:24 smithi049 ceph-mon[40489]: 3.f scrub starts 2024-09-21T14:10:24.739 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:24 smithi049 ceph-mon[40489]: 3.f scrub ok 2024-09-21T14:10:24.739 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:24 smithi049 ceph-mon[40489]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:10:24.739 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:24 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T14:10:24.739 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:24 smithi049 ceph-mon[40489]: osdmap e545: 8 total, 8 up, 8 in 2024-09-21T14:10:24.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:24 smithi049 ceph-mon[30080]: 3.f scrub starts 2024-09-21T14:10:24.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:24 smithi049 ceph-mon[30080]: 3.f scrub ok 2024-09-21T14:10:24.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:24 smithi049 ceph-mon[30080]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:10:24.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:24 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T14:10:24.739 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:24 smithi049 ceph-mon[30080]: osdmap e545: 8 total, 8 up, 8 in 2024-09-21T14:10:24.810 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:10:24.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:24.850 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:24.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:10:24.951 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:10:24.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:24.990 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:24.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:10:25.091 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:10:25.122 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:25.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:25.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:10:25.224 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:10:25.241 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T14:10:25.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:25.271 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:25.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:10:25.357 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:10:25.373 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:10:25.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:25.432 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:25.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:10:25.491 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:25 smithi049 ceph-mon[30080]: pgmap v1102: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 295 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 648 B/s rd, 0 B/s wr, 1.25k op/s; 20/14739 objects misplaced (0.136%) 2024-09-21T14:10:25.492 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:25 smithi049 ceph-mon[30080]: osdmap e546: 8 total, 8 up, 8 in 2024-09-21T14:10:25.492 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:25 smithi049 ceph-mon[40489]: pgmap v1102: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 295 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 648 B/s rd, 0 B/s wr, 1.25k op/s; 20/14739 objects misplaced (0.136%) 2024-09-21T14:10:25.492 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:25 smithi049 ceph-mon[40489]: osdmap e546: 8 total, 8 up, 8 in 2024-09-21T14:10:25.533 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:10:25.558 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:25.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:25.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:10:25.659 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:10:25.684 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:25.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:25.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:10:25.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:25 smithi187 ceph-mon[33032]: pgmap v1102: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 295 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 648 B/s rd, 0 B/s wr, 1.25k op/s; 20/14739 objects misplaced (0.136%) 2024-09-21T14:10:25.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:25 smithi187 ceph-mon[33032]: osdmap e546: 8 total, 8 up, 8 in 2024-09-21T14:10:25.786 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:10:25.821 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:25.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:25.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:10:25.923 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:10:25.946 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:25.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:25.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:10:25.958 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:10:25.959 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:10:25.972 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:10:25.972 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:10:25.989 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:10:25.990 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:10:26.007 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:10:26.008 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:10:26.008 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:10:26.047 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:10:26.069 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:26.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:26.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:10:26.171 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:10:26.201 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:26.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:26.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:10:26.206 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:10:26.206 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:10:26.239 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:10:26.239 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:10:26.272 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:10:26.273 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:10:26.303 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:10:26.307 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:10:26.308 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:10:26.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:26.332 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:26.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:10:26.395 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T14:10:26.433 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:10:26.434 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:26 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1819655187' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:10:26.434 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:26 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1819655187' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:10:26.476 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:26.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:26.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:10:26.578 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:10:26.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:26.626 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:26.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:10:26.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:26 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1819655187' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:10:26.726 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:10:26.751 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:26.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:26.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:10:26.852 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:10:26.911 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T14:10:26.911 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:10:26.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:26.915 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:26.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:10:27.016 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:10:27.033 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Removed 123376 objects 2024-09-21T14:10:27.033 INFO:tasks.radosbench.radosbench.0.smithi049.stdout:Clean up completed and total clean up time :123.508 2024-09-21T14:10:27.039 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-09-21T14:10:27.039 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-09-21T14:10:27.040 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:27.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:27.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:10:27.142 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:10:27.164 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:27.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:27.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:10:27.266 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:10:27.288 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:27.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:27.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:10:27.389 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:10:27.412 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:27.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:27.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:10:27.477 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 4, 6, 2, 3, 7, 5] 2024-09-21T14:10:27.477 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T14:10:27.478 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-21T14:10:27.478 INFO:tasks.thrashosds.thrasher:Setting osd 2 primary_affinity to 1.000000 2024-09-21T14:10:27.478 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 2 1 2024-09-21T14:10:27.514 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:10:27.536 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:27.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:27.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:10:27.637 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:10:27.660 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:27.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:27.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:10:27.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:27 smithi187 ceph-mon[33032]: pgmap v1104: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 185 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.25k op/s; 7/9483 objects misplaced (0.074%) 2024-09-21T14:10:27.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:27 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:10:27.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:27 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:10:27.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:27 smithi049 ceph-mon[30080]: pgmap v1104: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 185 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.25k op/s; 7/9483 objects misplaced (0.074%) 2024-09-21T14:10:27.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:27 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:10:27.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:27 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:10:27.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:27 smithi049 ceph-mon[40489]: pgmap v1104: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 185 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.25k op/s; 7/9483 objects misplaced (0.074%) 2024-09-21T14:10:27.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:27 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:10:27.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:27 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:10:27.762 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:10:27.783 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:27.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:27.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:10:27.885 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:10:27.907 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:27.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:27.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:10:28.009 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:10:28.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:28.035 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T14:10:28.035 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:28.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:10:28.136 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:10:28.159 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:28.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:28.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:10:28.261 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:10:28.283 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:28.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:28.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:10:28.385 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:10:28.407 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:28.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:28.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:10:28.509 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:10:28.510 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:10:28.539 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:28.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:28.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:10:28.553 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:10:28.635 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:28 smithi049 ceph-mon[40489]: osdmap e547: 8 total, 8 up, 8 in 2024-09-21T14:10:28.635 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:28 smithi049 ceph-mon[30080]: osdmap e547: 8 total, 8 up, 8 in 2024-09-21T14:10:28.641 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:10:28.662 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:28.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:28.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:10:28.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:28 smithi187 ceph-mon[33032]: osdmap e547: 8 total, 8 up, 8 in 2024-09-21T14:10:28.764 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:10:28.788 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:28.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:28.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:10:28.889 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:10:28.913 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:28.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:28.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:10:29.015 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:10:29.037 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:29.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:29.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:10:29.139 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:10:29.164 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:29.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:29.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:10:29.262 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:10:29.265 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:10:29.288 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:29.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:29.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:10:29.390 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:10:29.413 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:29.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:29.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:10:29.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:29 smithi049 ceph-mon[30080]: pgmap v1106: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 144 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 942 op/s; 7/7470 objects misplaced (0.094%) 2024-09-21T14:10:29.476 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:29 smithi049 ceph-mon[40489]: pgmap v1106: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 144 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 942 op/s; 7/7470 objects misplaced (0.094%) 2024-09-21T14:10:29.515 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:10:29.537 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:29.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:29.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:10:29.639 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:10:29.661 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:29.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:29.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:10:29.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:29 smithi187 ceph-mon[33032]: pgmap v1106: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 144 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 942 op/s; 7/7470 objects misplaced (0.094%) 2024-09-21T14:10:29.762 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:10:29.785 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:29.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:29.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:10:29.887 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:10:29.910 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:29.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:29.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:10:30.011 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:10:30.017 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:10:30.035 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:30.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:30.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:10:30.136 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:10:30.158 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:30.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:30.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:10:30.260 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:10:30.269 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'unique_pool_4' removed 2024-09-21T14:10:30.274 INFO:teuthology.orchestra.run.smithi049.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-21T14:10:30.277 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is set 2024-09-21T14:10:30.309 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:30.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:30.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:10:30.402 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:30 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3161510630' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:10:30.402 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:30 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:10:30.402 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:30 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/579552390' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-21T14:10:30.402 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:30 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-21T14:10:30.402 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:30 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:10:30.402 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:30 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1374109799' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T14:10:30.402 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:30 smithi049 ceph-mon[40489]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:10:30.402 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:30 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T14:10:30.402 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:30 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-21T14:10:30.402 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:30 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1374109799' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T14:10:30.403 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:30 smithi049 ceph-mon[40489]: osdmap e548: 8 total, 8 up, 8 in 2024-09-21T14:10:30.403 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:30 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "3.d"} : dispatch 2024-09-21T14:10:30.403 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:30 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3161510630' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:10:30.403 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:30 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:10:30.403 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:30 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/579552390' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-21T14:10:30.403 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:30 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-21T14:10:30.403 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:30 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:10:30.403 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:30 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1374109799' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T14:10:30.403 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:30 smithi049 ceph-mon[30080]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:10:30.403 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:30 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T14:10:30.403 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:30 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-21T14:10:30.404 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:30 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1374109799' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T14:10:30.404 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:30 smithi049 ceph-mon[30080]: osdmap e548: 8 total, 8 up, 8 in 2024-09-21T14:10:30.404 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:30 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "3.d"} : dispatch 2024-09-21T14:10:30.411 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:10:30.432 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:30.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:30.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:10:30.534 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:10:30.550 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T14:10:30.555 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:30.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:30.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:10:30.657 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:10:30.679 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:30.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:30.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:10:30.694 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:10:30 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:10:30.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:30 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3161510630' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:10:30.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:30 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:10:30.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:30 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/579552390' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-21T14:10:30.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:30 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-21T14:10:30.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:30 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:10:30.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:30 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1374109799' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T14:10:30.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:30 smithi187 ceph-mon[33032]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:10:30.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:30 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T14:10:30.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:30 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-21T14:10:30.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:30 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1374109799' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T14:10:30.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:30 smithi187 ceph-mon[33032]: osdmap e548: 8 total, 8 up, 8 in 2024-09-21T14:10:30.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:30 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "3.d"} : dispatch 2024-09-21T14:10:30.754 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:10:30 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:10:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:10:30.780 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6 2024-09-21T14:10:30.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:30.802 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:30.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.6' 2024-09-21T14:10:30.902 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:10:30.926 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:30.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:30.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:10:31.028 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4 2024-09-21T14:10:31.048 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:31.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:31.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.4' 2024-09-21T14:10:31.150 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2 2024-09-21T14:10:31.175 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:31.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:31.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.2' 2024-09-21T14:10:31.277 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:10:31.299 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:31.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:31.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:10:31.401 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3 2024-09-21T14:10:31.423 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:31.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:31.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.3' 2024-09-21T14:10:31.524 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7 2024-09-21T14:10:31.547 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:31.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:31.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.7' 2024-09-21T14:10:31.649 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1 2024-09-21T14:10:31.651 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:31 smithi049 ceph-mon[40489]: pgmap v1107: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 27 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 741 B/s rd, 0 B/s wr, 1.09k op/s; 7/1863 objects misplaced (0.376%) 2024-09-21T14:10:31.651 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:31 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:10:31.651 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:31 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "3.d"}]': finished 2024-09-21T14:10:31.651 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:31 smithi049 ceph-mon[40489]: osdmap e549: 8 total, 8 up, 8 in 2024-09-21T14:10:31.651 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:31 smithi049 ceph-mon[30080]: pgmap v1107: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 27 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 741 B/s rd, 0 B/s wr, 1.09k op/s; 7/1863 objects misplaced (0.376%) 2024-09-21T14:10:31.651 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:31 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:10:31.651 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:31 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "3.d"}]': finished 2024-09-21T14:10:31.652 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:31 smithi049 ceph-mon[30080]: osdmap e549: 8 total, 8 up, 8 in 2024-09-21T14:10:31.672 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:31.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:31.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.1' 2024-09-21T14:10:31.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:31 smithi187 ceph-mon[33032]: pgmap v1107: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 27 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 741 B/s rd, 0 B/s wr, 1.09k op/s; 7/1863 objects misplaced (0.376%) 2024-09-21T14:10:31.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:31 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:10:31.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:31 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "3.d"}]': finished 2024-09-21T14:10:31.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:31 smithi187 ceph-mon[33032]: osdmap e549: 8 total, 8 up, 8 in 2024-09-21T14:10:31.774 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:10:31.797 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:31.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:31.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:10:31.899 DEBUG:teuthology.orchestra.run.smithi049:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0 2024-09-21T14:10:31.922 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: docker: command not found 2024-09-21T14:10:31.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:31.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi049 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.0' 2024-09-21T14:10:32.023 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5 2024-09-21T14:10:32.046 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-09-21T14:10:32.056 INFO:tasks.thrashosds:joining thrashosds 2024-09-21T14:10:32.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:10:32.057 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-21T14:10:32.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd.5' 2024-09-21T14:10:33.349 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T14:10:33.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:33 smithi187 ceph-mon[33032]: pgmap v1110: 152 pgs: 152 active+clean; 448 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-21T14:10:33.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:33 smithi187 ceph-mon[33032]: osdmap e550: 8 total, 8 up, 8 in 2024-09-21T14:10:33.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:33 smithi187 ceph-mon[33032]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-21T14:10:33.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:33 smithi049 ceph-mon[40489]: pgmap v1110: 152 pgs: 152 active+clean; 448 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-21T14:10:33.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:33 smithi049 ceph-mon[40489]: osdmap e550: 8 total, 8 up, 8 in 2024-09-21T14:10:33.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:33 smithi049 ceph-mon[40489]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-21T14:10:33.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:33 smithi049 ceph-mon[30080]: pgmap v1110: 152 pgs: 152 active+clean; 448 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-21T14:10:33.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:33 smithi049 ceph-mon[30080]: osdmap e550: 8 total, 8 up, 8 in 2024-09-21T14:10:33.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:33 smithi049 ceph-mon[30080]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-21T14:10:33.989 INFO:teuthology.orchestra.run.smithi049.stdout:ERROR: (22) Invalid argument 2024-09-21T14:10:33.989 INFO:teuthology.orchestra.run.smithi049.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T14:10:34.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:34 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:10:34.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:34 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:10:34.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:34 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:10:34.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:34 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:10:34.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:34 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:10:34.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:34 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:10:35.377 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:10:35.535 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:35 smithi049 ceph-mon[40489]: pgmap v1112: 152 pgs: 152 active+clean; 448 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-21T14:10:35.535 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:35 smithi049 ceph-mon[40489]: osdmap e551: 8 total, 8 up, 8 in 2024-09-21T14:10:35.535 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:35 smithi049 ceph-mon[40489]: osdmap e552: 8 total, 8 up, 8 in 2024-09-21T14:10:35.535 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:35 smithi049 ceph-mon[30080]: pgmap v1112: 152 pgs: 152 active+clean; 448 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-21T14:10:35.535 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:35 smithi049 ceph-mon[30080]: osdmap e551: 8 total, 8 up, 8 in 2024-09-21T14:10:35.535 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:35 smithi049 ceph-mon[30080]: osdmap e552: 8 total, 8 up, 8 in 2024-09-21T14:10:35.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:35 smithi187 ceph-mon[33032]: pgmap v1112: 152 pgs: 152 active+clean; 448 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-21T14:10:35.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:35 smithi187 ceph-mon[33032]: osdmap e551: 8 total, 8 up, 8 in 2024-09-21T14:10:35.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:35 smithi187 ceph-mon[33032]: osdmap e552: 8 total, 8 up, 8 in 2024-09-21T14:10:36.212 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T14:10:36.424 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:36 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:10:36.424 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:36 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:10:36.424 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:36 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:10:36.424 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:36 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:10:36.429 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0: {} 2024-09-21T14:10:36.429 INFO:teuthology.orchestra.run.smithi049.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T14:10:36.430 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:10:36.447 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1: {} 2024-09-21T14:10:36.447 INFO:teuthology.orchestra.run.smithi049.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T14:10:36.465 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2: {} 2024-09-21T14:10:36.465 INFO:teuthology.orchestra.run.smithi049.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T14:10:36.478 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:10:36.481 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3: {} 2024-09-21T14:10:36.482 INFO:teuthology.orchestra.run.smithi049.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T14:10:36.499 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4: {} 2024-09-21T14:10:36.499 INFO:teuthology.orchestra.run.smithi049.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T14:10:36.516 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5: {} 2024-09-21T14:10:36.516 INFO:teuthology.orchestra.run.smithi049.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T14:10:36.533 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6: {} 2024-09-21T14:10:36.533 INFO:teuthology.orchestra.run.smithi049.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T14:10:36.550 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7: {} 2024-09-21T14:10:36.550 INFO:teuthology.orchestra.run.smithi049.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T14:10:36.598 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:36 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:10:36.599 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:36 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:10:36.995 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T14:10:36.995 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-21T14:10:36.996 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 0 2024-09-21T14:10:36.996 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T14:10:37.623 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-21T14:10:37.623 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd pool set .mgr pgp_num 21 2024-09-21T14:10:37.641 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:37 smithi049 ceph-mon[40489]: pgmap v1115: 152 pgs: 152 active+clean; 448 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-21T14:10:37.641 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:37 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/152170807' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:10:37.641 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:37 smithi049 ceph-mon[30080]: pgmap v1115: 152 pgs: 152 active+clean; 448 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-21T14:10:37.641 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:37 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/152170807' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:10:37.656 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:37 smithi187 ceph-mon[33032]: pgmap v1115: 152 pgs: 152 active+clean; 448 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-21T14:10:37.656 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:37 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/152170807' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T14:10:37.684 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T14:10:37.897 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:10:37.929 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:10:38.290 INFO:teuthology.orchestra.run.smithi049.stderr:noscrub is unset 2024-09-21T14:10:38.334 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:38 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1722006871' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:10:38.442 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T14:10:38.442 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-21T14:10:38.442 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-21T14:10:38.442 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-21T14:10:38.442 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T14:10:38.622 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:38 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1722006871' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:10:38.622 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:38 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1722006871' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T14:10:39.092 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T14:10:39.298 INFO:teuthology.orchestra.run.smithi049.stderr:set pool 1 pgp_num to 21 2024-09-21T14:10:39.339 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:10:39.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:39 smithi187 ceph-mon[33032]: pgmap v1116: 152 pgs: 152 active+clean; 448 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T14:10:39.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:39 smithi187 ceph-mon[33032]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:10:39.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:39 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1722006871' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T14:10:39.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:39 smithi187 ceph-mon[33032]: osdmap e553: 8 total, 8 up, 8 in 2024-09-21T14:10:39.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:39 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3930843848' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"} : dispatch 2024-09-21T14:10:39.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:39 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"} : dispatch 2024-09-21T14:10:39.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:39 smithi049 ceph-mon[30080]: pgmap v1116: 152 pgs: 152 active+clean; 448 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T14:10:39.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:39 smithi049 ceph-mon[30080]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:10:39.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:39 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1722006871' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T14:10:39.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:39 smithi049 ceph-mon[30080]: osdmap e553: 8 total, 8 up, 8 in 2024-09-21T14:10:39.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:39 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3930843848' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"} : dispatch 2024-09-21T14:10:39.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:39 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"} : dispatch 2024-09-21T14:10:39.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:39 smithi049 ceph-mon[40489]: pgmap v1116: 152 pgs: 152 active+clean; 448 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T14:10:39.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:39 smithi049 ceph-mon[40489]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T14:10:39.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:39 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1722006871' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T14:10:39.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:39 smithi049 ceph-mon[40489]: osdmap e553: 8 total, 8 up, 8 in 2024-09-21T14:10:39.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:39 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3930843848' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"} : dispatch 2024-09-21T14:10:39.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:39 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"} : dispatch 2024-09-21T14:10:39.836 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T14:10:39.836 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-21T14:10:39.836 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-21T14:10:39.836 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-21T14:10:39.836 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.836 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:112229 10.4 10:2381a1a9:::benchmark_data_smithi049_176811_object112228:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-21T14:10:39.836 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:04:20.239436+0000", 2024-09-21T14:10:39.837 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 379.595179648, 2024-09-21T14:10:39.837 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.048690772, 2024-09-21T14:10:39.837 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:39.837 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:39.837 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:39.837 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:10:39.837 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:10:39.837 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:10:39.837 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 112229 2024-09-21T14:10:39.837 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.837 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:39.837 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.837 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:39.837 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.239436+0000", 2024-09-21T14:10:39.837 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:39.838 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.838 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.838 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:39.838 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.239436+0000", 2024-09-21T14:10:39.838 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:39.838 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.838 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.838 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:39.838 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.239437+0000", 2024-09-21T14:10:39.838 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.668e-06 2024-09-21T14:10:39.838 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.838 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.838 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:39.839 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.239558+0000", 2024-09-21T14:10:39.839 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012067599999999999 2024-09-21T14:10:39.839 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.839 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.839 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:39.839 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.239570+0000", 2024-09-21T14:10:39.839 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1581e-05 2024-09-21T14:10:39.839 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.839 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.839 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:39.839 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.239585+0000", 2024-09-21T14:10:39.839 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5241e-05 2024-09-21T14:10:39.840 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.840 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.840 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:39.840 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.239656+0000", 2024-09-21T14:10:39.840 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1248000000000006e-05 2024-09-21T14:10:39.840 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.840 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.840 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:39.840 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.239666+0000", 2024-09-21T14:10:39.840 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0068000000000001e-05 2024-09-21T14:10:39.840 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.840 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.840 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:39.841 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.194584+0000", 2024-09-21T14:10:39.841 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.95491765200000001 2024-09-21T14:10:39.841 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.841 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.841 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:39.841 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.194597+0000", 2024-09-21T14:10:39.841 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3377000000000001e-05 2024-09-21T14:10:39.841 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.841 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.841 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:39.841 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.269588+0000", 2024-09-21T14:10:39.841 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0749913129999999 2024-09-21T14:10:39.841 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.841 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.842 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:10:39.842 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.269689+0000", 2024-09-21T14:10:39.842 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000101006 2024-09-21T14:10:39.842 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.842 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.842 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1,3", 2024-09-21T14:10:39.842 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.269788+0000", 2024-09-21T14:10:39.842 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.8152000000000005e-05 2024-09-21T14:10:39.842 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.842 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.842 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:10:39.842 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.276727+0000", 2024-09-21T14:10:39.842 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0069397479999999999 2024-09-21T14:10:39.843 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.843 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.843 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:39.843 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.278656+0000", 2024-09-21T14:10:39.843 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001929123 2024-09-21T14:10:39.843 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.843 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.843 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:39.843 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.279569+0000", 2024-09-21T14:10:39.843 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00091213100000000005 2024-09-21T14:10:39.843 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.843 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.843 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:39.843 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.288003+0000", 2024-09-21T14:10:39.843 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0084348429999999992 2024-09-21T14:10:39.844 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.844 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.844 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:10:39.844 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.288045+0000", 2024-09-21T14:10:39.844 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1553e-05 2024-09-21T14:10:39.844 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.844 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.844 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:39.844 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.288126+0000", 2024-09-21T14:10:39.844 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1391999999999996e-05 2024-09-21T14:10:39.844 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:39.844 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:39.844 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:39.845 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.845 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.845 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:112231 10.4 10:2e3f0834:::benchmark_data_smithi049_176811_object112230:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-21T14:10:39.845 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:04:20.239610+0000", 2024-09-21T14:10:39.845 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 379.59500531899999, 2024-09-21T14:10:39.845 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0486707210000001, 2024-09-21T14:10:39.845 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:39.845 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:39.845 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:39.845 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:10:39.845 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:10:39.845 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:10:39.845 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 112231 2024-09-21T14:10:39.845 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.846 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:39.846 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.846 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:39.846 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.239610+0000", 2024-09-21T14:10:39.846 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:39.846 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.846 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.846 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:39.846 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.239610+0000", 2024-09-21T14:10:39.846 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:39.846 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.846 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.846 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:39.846 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.239611+0000", 2024-09-21T14:10:39.847 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3500000000000005e-07 2024-09-21T14:10:39.847 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.847 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.847 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:39.847 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.239723+0000", 2024-09-21T14:10:39.847 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000112341 2024-09-21T14:10:39.847 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.847 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.847 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:39.847 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.239733+0000", 2024-09-21T14:10:39.847 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0118999999999999e-05 2024-09-21T14:10:39.847 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.847 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.847 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:39.848 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.239740+0000", 2024-09-21T14:10:39.848 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6069999999999996e-06 2024-09-21T14:10:39.848 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.848 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.848 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:39.848 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.239792+0000", 2024-09-21T14:10:39.848 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2629e-05 2024-09-21T14:10:39.848 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.848 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.848 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:39.848 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.239801+0000", 2024-09-21T14:10:39.848 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.6500000000000002e-06 2024-09-21T14:10:39.848 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.848 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.848 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:39.848 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.194627+0000", 2024-09-21T14:10:39.849 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.95482541300000001 2024-09-21T14:10:39.849 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.849 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.849 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:39.849 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.194635+0000", 2024-09-21T14:10:39.849 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9899999999999997e-06 2024-09-21T14:10:39.849 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.849 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.849 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:39.849 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.271698+0000", 2024-09-21T14:10:39.849 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.07706378 2024-09-21T14:10:39.849 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.849 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.849 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:10:39.849 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.271785+0000", 2024-09-21T14:10:39.849 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7182000000000007e-05 2024-09-21T14:10:39.850 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.850 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.850 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1,3", 2024-09-21T14:10:39.850 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.271892+0000", 2024-09-21T14:10:39.850 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000106346 2024-09-21T14:10:39.850 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.850 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.850 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:10:39.850 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.276736+0000", 2024-09-21T14:10:39.850 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0048441659999999996 2024-09-21T14:10:39.850 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.850 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.850 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:39.850 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.279711+0000", 2024-09-21T14:10:39.850 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0029745090000000002 2024-09-21T14:10:39.851 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.851 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.851 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:39.851 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.288164+0000", 2024-09-21T14:10:39.851 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0084534599999999994 2024-09-21T14:10:39.851 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.851 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.851 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:39.851 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.288197+0000", 2024-09-21T14:10:39.851 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2592000000000002e-05 2024-09-21T14:10:39.851 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.851 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.851 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:10:39.851 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.288228+0000", 2024-09-21T14:10:39.851 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1581999999999997e-05 2024-09-21T14:10:39.851 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.852 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.852 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:39.852 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.288281+0000", 2024-09-21T14:10:39.852 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2519999999999999e-05 2024-09-21T14:10:39.852 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:39.852 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:39.852 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:39.852 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.852 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.852 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:112244 10.4 10:228aeb9f:::benchmark_data_smithi049_176811_object112243:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-21T14:10:39.852 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:04:20.250628+0000", 2024-09-21T14:10:39.852 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 379.58398727600002, 2024-09-21T14:10:39.852 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0378127940000001, 2024-09-21T14:10:39.852 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:39.853 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:39.853 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:39.853 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:10:39.853 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:10:39.853 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:10:39.853 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 112244 2024-09-21T14:10:39.853 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.853 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:39.853 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.853 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:39.853 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.250628+0000", 2024-09-21T14:10:39.853 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:39.854 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.854 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.854 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:39.854 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.250628+0000", 2024-09-21T14:10:39.854 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:39.854 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.854 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.854 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:39.854 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.250630+0000", 2024-09-21T14:10:39.854 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.841e-06 2024-09-21T14:10:39.854 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.854 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.855 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:39.855 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.250735+0000", 2024-09-21T14:10:39.855 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000105393 2024-09-21T14:10:39.855 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.855 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.855 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:39.855 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.250751+0000", 2024-09-21T14:10:39.855 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5763000000000001e-05 2024-09-21T14:10:39.855 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.855 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.855 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:39.855 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.250759+0000", 2024-09-21T14:10:39.855 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2400000000000007e-06 2024-09-21T14:10:39.855 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.855 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.855 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:39.856 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.250829+0000", 2024-09-21T14:10:39.856 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9852999999999997e-05 2024-09-21T14:10:39.856 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.856 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.856 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:39.856 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.250846+0000", 2024-09-21T14:10:39.856 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7099e-05 2024-09-21T14:10:39.856 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.856 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.856 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:39.856 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.194667+0000", 2024-09-21T14:10:39.856 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.94382083999999999 2024-09-21T14:10:39.856 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.857 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.857 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:39.857 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.194675+0000", 2024-09-21T14:10:39.857 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7230000000000004e-06 2024-09-21T14:10:39.857 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.857 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.857 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:39.857 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.273226+0000", 2024-09-21T14:10:39.857 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0785516289999999 2024-09-21T14:10:39.857 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.857 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.857 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:10:39.857 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.273299+0000", 2024-09-21T14:10:39.857 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2143000000000004e-05 2024-09-21T14:10:39.857 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.858 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.858 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1,3", 2024-09-21T14:10:39.858 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.273396+0000", 2024-09-21T14:10:39.858 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7665999999999996e-05 2024-09-21T14:10:39.858 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.858 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.858 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:10:39.858 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.276742+0000", 2024-09-21T14:10:39.858 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003345621 2024-09-21T14:10:39.858 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.858 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.858 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:39.858 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.282756+0000", 2024-09-21T14:10:39.858 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0060141680000000003 2024-09-21T14:10:39.858 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.859 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.859 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:39.859 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.288320+0000", 2024-09-21T14:10:39.859 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0055642080000000002 2024-09-21T14:10:39.859 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.859 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.859 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:39.859 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.288351+0000", 2024-09-21T14:10:39.859 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0511e-05 2024-09-21T14:10:39.859 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.859 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.859 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:10:39.859 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.288381+0000", 2024-09-21T14:10:39.859 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0270999999999999e-05 2024-09-21T14:10:39.859 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.859 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.860 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:39.860 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.288441+0000", 2024-09-21T14:10:39.860 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9824999999999997e-05 2024-09-21T14:10:39.860 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:39.860 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:39.860 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:39.860 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.860 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.860 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:112248 10.4 10:2865fdef:::benchmark_data_smithi049_176811_object112247:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-21T14:10:39.860 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:04:20.253079+0000", 2024-09-21T14:10:39.860 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 379.58153621899999, 2024-09-21T14:10:39.860 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0355127999999998, 2024-09-21T14:10:39.860 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:39.860 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:39.860 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:39.860 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:10:39.861 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:10:39.861 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:10:39.861 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 112248 2024-09-21T14:10:39.861 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.861 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:39.861 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.861 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:39.861 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.253079+0000", 2024-09-21T14:10:39.861 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:39.861 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.861 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.861 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:39.861 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.253079+0000", 2024-09-21T14:10:39.861 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:39.861 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.861 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.861 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:39.862 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.253081+0000", 2024-09-21T14:10:39.862 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7799999999999999e-06 2024-09-21T14:10:39.862 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.862 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.862 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:39.862 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.253199+0000", 2024-09-21T14:10:39.862 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011827 2024-09-21T14:10:39.862 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.862 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.862 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:39.863 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.253215+0000", 2024-09-21T14:10:39.863 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5525000000000001e-05 2024-09-21T14:10:39.863 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.863 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.863 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:39.863 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.253223+0000", 2024-09-21T14:10:39.864 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9969999999999998e-06 2024-09-21T14:10:39.864 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.864 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.864 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:39.864 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.253298+0000", 2024-09-21T14:10:39.864 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5427999999999997e-05 2024-09-21T14:10:39.864 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.864 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.864 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:39.864 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.253307+0000", 2024-09-21T14:10:39.864 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0229999999999998e-06 2024-09-21T14:10:39.864 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.864 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.864 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:39.864 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.194706+0000", 2024-09-21T14:10:39.865 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.94139865 2024-09-21T14:10:39.865 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.865 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.865 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:39.865 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.194713+0000", 2024-09-21T14:10:39.865 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5129999999999999e-06 2024-09-21T14:10:39.865 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.865 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.865 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:39.865 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.274436+0000", 2024-09-21T14:10:39.865 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0797226369999999 2024-09-21T14:10:39.865 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.866 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.866 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:10:39.866 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.274522+0000", 2024-09-21T14:10:39.866 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5945000000000005e-05 2024-09-21T14:10:39.866 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.866 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.866 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1,3", 2024-09-21T14:10:39.866 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.274669+0000", 2024-09-21T14:10:39.866 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000147245 2024-09-21T14:10:39.866 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.866 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.866 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:10:39.866 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.278678+0000", 2024-09-21T14:10:39.866 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0040084509999999997 2024-09-21T14:10:39.867 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.867 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.867 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:39.867 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.282792+0000", 2024-09-21T14:10:39.867 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0041142560000000002 2024-09-21T14:10:39.867 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.867 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.867 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:39.867 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.288475+0000", 2024-09-21T14:10:39.867 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0056830379999999996 2024-09-21T14:10:39.867 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.867 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.867 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:39.867 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.288506+0000", 2024-09-21T14:10:39.868 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1068000000000001e-05 2024-09-21T14:10:39.868 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.868 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.868 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:10:39.868 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.288535+0000", 2024-09-21T14:10:39.868 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9464000000000001e-05 2024-09-21T14:10:39.868 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.868 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.868 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:39.868 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.288592+0000", 2024-09-21T14:10:39.868 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6509999999999999e-05 2024-09-21T14:10:39.868 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:39.868 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:39.868 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:39.868 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.869 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.869 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:112252 10.4 10:25607d1e:::benchmark_data_smithi049_176811_object112251:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-21T14:10:39.869 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:04:20.256669+0000", 2024-09-21T14:10:39.869 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 379.577945833, 2024-09-21T14:10:39.869 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.032077707, 2024-09-21T14:10:39.869 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:39.869 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:39.869 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:39.869 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:10:39.869 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:10:39.869 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:10:39.869 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 112252 2024-09-21T14:10:39.869 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.869 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:39.869 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.870 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:39.870 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.256669+0000", 2024-09-21T14:10:39.870 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:39.870 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.870 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.870 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:39.870 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.256669+0000", 2024-09-21T14:10:39.870 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:39.870 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.870 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.870 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:39.870 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.256671+0000", 2024-09-21T14:10:39.870 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4920000000000001e-06 2024-09-21T14:10:39.870 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.871 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.871 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:39.871 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.256771+0000", 2024-09-21T14:10:39.871 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010003 2024-09-21T14:10:39.871 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.871 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.871 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:39.871 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.256784+0000", 2024-09-21T14:10:39.871 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3375e-05 2024-09-21T14:10:39.871 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.871 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.871 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:39.871 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.256790+0000", 2024-09-21T14:10:39.871 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.049e-06 2024-09-21T14:10:39.871 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.872 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.872 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:39.872 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.256848+0000", 2024-09-21T14:10:39.872 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7862000000000003e-05 2024-09-21T14:10:39.872 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.872 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.872 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:39.872 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.256856+0000", 2024-09-21T14:10:39.872 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0129999999999993e-06 2024-09-21T14:10:39.872 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.872 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.872 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:39.872 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.194751+0000", 2024-09-21T14:10:39.872 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.93789514600000001 2024-09-21T14:10:39.872 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.873 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.873 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:39.873 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.194852+0000", 2024-09-21T14:10:39.873 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000100234 2024-09-21T14:10:39.873 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.873 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.873 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:39.873 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.276785+0000", 2024-09-21T14:10:39.873 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0819329390000001 2024-09-21T14:10:39.873 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.873 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.873 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:10:39.874 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.276924+0000", 2024-09-21T14:10:39.874 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013901699999999999 2024-09-21T14:10:39.874 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.874 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.874 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1,3", 2024-09-21T14:10:39.874 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.277028+0000", 2024-09-21T14:10:39.874 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010436 2024-09-21T14:10:39.874 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.874 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.874 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:10:39.874 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.278898+0000", 2024-09-21T14:10:39.875 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0018704069999999999 2024-09-21T14:10:39.875 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.875 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.875 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:39.875 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.282825+0000", 2024-09-21T14:10:39.875 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0039263520000000001 2024-09-21T14:10:39.875 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.875 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.875 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:39.875 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.288628+0000", 2024-09-21T14:10:39.875 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0058029259999999999 2024-09-21T14:10:39.875 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.875 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.875 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:39.875 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.288661+0000", 2024-09-21T14:10:39.876 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3759e-05 2024-09-21T14:10:39.876 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.876 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.876 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:10:39.876 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.288693+0000", 2024-09-21T14:10:39.876 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1838000000000002e-05 2024-09-21T14:10:39.876 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.876 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.876 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:39.876 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.288747+0000", 2024-09-21T14:10:39.876 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3908e-05 2024-09-21T14:10:39.876 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:39.876 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:39.876 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:39.877 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.877 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.877 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26396.0:183774 11.e 11:78f565e1:::benchmark_data_smithi049_206132_object60395:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e521)", 2024-09-21T14:10:39.877 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:09:27.336038+0000", 2024-09-21T14:10:39.877 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 72.498577166999993, 2024-09-21T14:10:39.877 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.99663222200000001, 2024-09-21T14:10:39.877 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:39.877 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:39.877 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:39.877 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:10:39.877 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26396", 2024-09-21T14:10:39.877 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:39.877 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 183774 2024-09-21T14:10:39.877 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.878 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:39.878 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.878 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:39.878 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.336038+0000", 2024-09-21T14:10:39.878 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:39.878 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.878 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.878 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:39.878 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.336038+0000", 2024-09-21T14:10:39.878 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:39.878 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.879 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.879 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:39.879 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.336041+0000", 2024-09-21T14:10:39.879 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5299999999999999e-06 2024-09-21T14:10:39.879 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.879 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.879 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:39.879 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.336048+0000", 2024-09-21T14:10:39.879 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1679999999999998e-06 2024-09-21T14:10:39.879 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.879 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.879 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:39.879 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.336066+0000", 2024-09-21T14:10:39.879 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8212000000000001e-05 2024-09-21T14:10:39.879 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.880 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.880 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:39.880 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.336075+0000", 2024-09-21T14:10:39.880 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5690000000000005e-06 2024-09-21T14:10:39.880 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.880 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.880 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:39.880 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.336147+0000", 2024-09-21T14:10:39.880 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2389000000000001e-05 2024-09-21T14:10:39.880 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.880 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.880 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:39.880 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.336158+0000", 2024-09-21T14:10:39.880 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1017e-05 2024-09-21T14:10:39.881 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.881 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.881 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:39.881 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.323512+0000", 2024-09-21T14:10:39.881 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.98735443300000003 2024-09-21T14:10:39.881 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.881 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.881 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:10:39.881 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.323640+0000", 2024-09-21T14:10:39.881 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012804599999999999 2024-09-21T14:10:39.881 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.881 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.881 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,7", 2024-09-21T14:10:39.881 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.324010+0000", 2024-09-21T14:10:39.882 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00036933099999999998 2024-09-21T14:10:39.882 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.882 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.882 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:39.882 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.332480+0000", 2024-09-21T14:10:39.882 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0084697599999999998 2024-09-21T14:10:39.882 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.882 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.882 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:10:39.882 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.332496+0000", 2024-09-21T14:10:39.882 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6388e-05 2024-09-21T14:10:39.882 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.882 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.883 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:39.883 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.332604+0000", 2024-09-21T14:10:39.883 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000108019 2024-09-21T14:10:39.883 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.883 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.883 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:10:39.883 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.332636+0000", 2024-09-21T14:10:39.883 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1733999999999998e-05 2024-09-21T14:10:39.883 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.883 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.883 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:39.883 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.332670+0000", 2024-09-21T14:10:39.883 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4625999999999997e-05 2024-09-21T14:10:39.883 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:39.884 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:39.884 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:39.884 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.884 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.884 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26396.0:183843 11.e 11:7c4a3aa8:::benchmark_data_smithi049_206132_object60464:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e521)", 2024-09-21T14:10:39.884 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:09:27.442195+0000", 2024-09-21T14:10:39.884 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 72.392419974000006, 2024-09-21T14:10:39.884 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.89062542899999997, 2024-09-21T14:10:39.884 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:39.884 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:39.884 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:39.884 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:10:39.884 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26396", 2024-09-21T14:10:39.885 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:39.885 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 183843 2024-09-21T14:10:39.885 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.885 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:39.885 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.885 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:39.885 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.442195+0000", 2024-09-21T14:10:39.885 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:39.885 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.885 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.885 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:39.885 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.442195+0000", 2024-09-21T14:10:39.885 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:39.885 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.886 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.886 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:39.886 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.442197+0000", 2024-09-21T14:10:39.886 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7409999999999999e-06 2024-09-21T14:10:39.886 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.886 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.886 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:39.886 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.442202+0000", 2024-09-21T14:10:39.886 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2290000000000002e-06 2024-09-21T14:10:39.886 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.886 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.886 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:39.886 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.442215+0000", 2024-09-21T14:10:39.887 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3089e-05 2024-09-21T14:10:39.887 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.887 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.887 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:39.887 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.442223+0000", 2024-09-21T14:10:39.887 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5329999999999997e-06 2024-09-21T14:10:39.887 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.887 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.887 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:39.887 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.442280+0000", 2024-09-21T14:10:39.887 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6632999999999998e-05 2024-09-21T14:10:39.887 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.887 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.887 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:39.888 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.442289+0000", 2024-09-21T14:10:39.888 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.2799999999999992e-06 2024-09-21T14:10:39.888 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.888 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.888 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:39.888 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.328623+0000", 2024-09-21T14:10:39.888 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.88633455000000005 2024-09-21T14:10:39.888 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.888 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.888 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:10:39.888 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.328697+0000", 2024-09-21T14:10:39.888 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3831999999999994e-05 2024-09-21T14:10:39.888 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.888 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.889 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,7", 2024-09-21T14:10:39.889 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.328807+0000", 2024-09-21T14:10:39.889 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010946999999999999 2024-09-21T14:10:39.889 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.889 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.889 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:39.889 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.332716+0000", 2024-09-21T14:10:39.889 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003909236 2024-09-21T14:10:39.889 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.889 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.889 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:10:39.889 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.332726+0000", 2024-09-21T14:10:39.889 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0475999999999999e-05 2024-09-21T14:10:39.889 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.889 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.889 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:39.889 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.332771+0000", 2024-09-21T14:10:39.890 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4916000000000002e-05 2024-09-21T14:10:39.890 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.890 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.890 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:10:39.890 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.332799+0000", 2024-09-21T14:10:39.890 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7285e-05 2024-09-21T14:10:39.890 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.890 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.890 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:39.890 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.332821+0000", 2024-09-21T14:10:39.890 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2158999999999999e-05 2024-09-21T14:10:39.890 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:39.890 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:39.890 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:39.891 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.891 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.891 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26396.0:183861 11.e 11:7facb653:::benchmark_data_smithi049_206132_object60482:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e521)", 2024-09-21T14:10:39.891 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:09:27.453458+0000", 2024-09-21T14:10:39.891 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 72.381156970999996, 2024-09-21T14:10:39.891 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.88325181500000005, 2024-09-21T14:10:39.891 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:39.891 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:39.891 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:39.891 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:10:39.891 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26396", 2024-09-21T14:10:39.891 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:39.891 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 183861 2024-09-21T14:10:39.891 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.892 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:39.892 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.892 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:39.892 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.453458+0000", 2024-09-21T14:10:39.892 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:39.892 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.892 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.892 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:39.892 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.453458+0000", 2024-09-21T14:10:39.892 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:39.892 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.892 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.892 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:39.892 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.453460+0000", 2024-09-21T14:10:39.892 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.359e-06 2024-09-21T14:10:39.893 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.893 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.893 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:39.893 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.453463+0000", 2024-09-21T14:10:39.893 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.19e-06 2024-09-21T14:10:39.893 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.893 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.893 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:39.893 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.453472+0000", 2024-09-21T14:10:39.893 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1090000000000004e-06 2024-09-21T14:10:39.893 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.893 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.893 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:39.893 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.453478+0000", 2024-09-21T14:10:39.893 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0360000000000003e-06 2024-09-21T14:10:39.893 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.894 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.894 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:39.894 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.453577+0000", 2024-09-21T14:10:39.894 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.8819000000000004e-05 2024-09-21T14:10:39.894 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.894 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.894 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:39.894 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.453586+0000", 2024-09-21T14:10:39.894 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.816e-06 2024-09-21T14:10:39.894 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.894 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.894 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:39.894 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.330088+0000", 2024-09-21T14:10:39.895 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.87650278699999995 2024-09-21T14:10:39.895 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.895 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.895 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:10:39.895 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.330204+0000", 2024-09-21T14:10:39.895 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011601500000000001 2024-09-21T14:10:39.895 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.895 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.895 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,7", 2024-09-21T14:10:39.895 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.330321+0000", 2024-09-21T14:10:39.895 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000116637 2024-09-21T14:10:39.895 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.895 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.896 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:10:39.896 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.332732+0000", 2024-09-21T14:10:39.896 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.002410817 2024-09-21T14:10:39.896 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.896 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.896 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:39.896 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.336580+0000", 2024-09-21T14:10:39.896 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0038480210000000001 2024-09-21T14:10:39.896 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.896 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.896 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:39.896 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.336625+0000", 2024-09-21T14:10:39.896 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5238999999999998e-05 2024-09-21T14:10:39.896 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.897 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.897 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:10:39.897 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.336673+0000", 2024-09-21T14:10:39.897 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7725000000000001e-05 2024-09-21T14:10:39.897 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.897 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.897 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:39.897 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.336710+0000", 2024-09-21T14:10:39.897 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7245e-05 2024-09-21T14:10:39.897 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:39.897 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:39.897 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:39.897 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.898 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.898 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26396.0:183867 11.e 11:7fefeab6:::benchmark_data_smithi049_206132_object60488:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e521)", 2024-09-21T14:10:39.898 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:09:27.459374+0000", 2024-09-21T14:10:39.898 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 72.375240841999997, 2024-09-21T14:10:39.898 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.88155136300000003, 2024-09-21T14:10:39.898 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:39.898 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:39.898 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:39.898 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:10:39.898 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26396", 2024-09-21T14:10:39.898 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:39.898 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 183867 2024-09-21T14:10:39.898 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.898 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:39.898 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.899 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:39.899 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.459374+0000", 2024-09-21T14:10:39.899 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:39.899 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.899 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.899 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:39.899 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.459374+0000", 2024-09-21T14:10:39.899 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:39.899 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.899 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.899 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:39.899 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.459376+0000", 2024-09-21T14:10:39.899 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.72e-06 2024-09-21T14:10:39.900 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.900 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.900 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:39.900 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.459387+0000", 2024-09-21T14:10:39.900 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0573e-05 2024-09-21T14:10:39.900 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.900 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.900 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:39.900 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.459404+0000", 2024-09-21T14:10:39.900 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7699000000000001e-05 2024-09-21T14:10:39.900 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.900 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.900 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:39.900 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.459413+0000", 2024-09-21T14:10:39.901 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1249999999999993e-06 2024-09-21T14:10:39.901 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.901 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.901 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:39.901 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.460993+0000", 2024-09-21T14:10:39.901 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0015804840000000001 2024-09-21T14:10:39.901 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.901 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.901 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:39.901 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.461001+0000", 2024-09-21T14:10:39.901 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1950000000000003e-06 2024-09-21T14:10:39.901 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.901 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.902 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:39.902 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.331246+0000", 2024-09-21T14:10:39.902 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.87024519199999995 2024-09-21T14:10:39.902 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.902 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.902 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:10:39.902 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.331336+0000", 2024-09-21T14:10:39.902 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.9361e-05 2024-09-21T14:10:39.902 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.902 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.902 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,7", 2024-09-21T14:10:39.902 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.331395+0000", 2024-09-21T14:10:39.902 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9630000000000003e-05 2024-09-21T14:10:39.902 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.902 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.903 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:10:39.903 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.332738+0000", 2024-09-21T14:10:39.903 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0013430390000000001 2024-09-21T14:10:39.903 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.903 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.903 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:39.903 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.336746+0000", 2024-09-21T14:10:39.903 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0040070970000000003 2024-09-21T14:10:39.903 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.903 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.903 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:39.903 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.340815+0000", 2024-09-21T14:10:39.903 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0040696309999999998 2024-09-21T14:10:39.903 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.904 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.904 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:10:39.904 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.340885+0000", 2024-09-21T14:10:39.904 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9428e-05 2024-09-21T14:10:39.904 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.904 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.904 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:39.904 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.340926+0000", 2024-09-21T14:10:39.904 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1189000000000002e-05 2024-09-21T14:10:39.904 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:39.904 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:39.904 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:39.904 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.905 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.905 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26396.0:183877 11.e 11:7014a509:::benchmark_data_smithi049_206132_object60498:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e521)", 2024-09-21T14:10:39.905 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:09:27.465756+0000", 2024-09-21T14:10:39.905 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 72.368859357999995, 2024-09-21T14:10:39.905 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.87537897899999995, 2024-09-21T14:10:39.905 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:39.905 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:39.905 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:39.905 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:10:39.905 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26396", 2024-09-21T14:10:39.905 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:39.905 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 183877 2024-09-21T14:10:39.905 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.905 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:39.905 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.906 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:39.906 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.465756+0000", 2024-09-21T14:10:39.906 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:39.906 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.906 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.906 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:39.906 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.465756+0000", 2024-09-21T14:10:39.906 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:39.906 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.906 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.906 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:39.906 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.465758+0000", 2024-09-21T14:10:39.906 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2819999999999999e-06 2024-09-21T14:10:39.907 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.907 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.907 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:39.907 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.465763+0000", 2024-09-21T14:10:39.907 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2379999999999997e-06 2024-09-21T14:10:39.907 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.907 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.907 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:39.907 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.465776+0000", 2024-09-21T14:10:39.907 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2619999999999999e-05 2024-09-21T14:10:39.907 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.907 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.907 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:39.907 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.465784+0000", 2024-09-21T14:10:39.908 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7980000000000006e-06 2024-09-21T14:10:39.908 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.908 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.908 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:39.908 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.466103+0000", 2024-09-21T14:10:39.908 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00031869599999999999 2024-09-21T14:10:39.908 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.908 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.908 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:39.908 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.466110+0000", 2024-09-21T14:10:39.908 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7530000000000001e-06 2024-09-21T14:10:39.908 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.908 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.908 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:39.909 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.332861+0000", 2024-09-21T14:10:39.909 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.86675055199999995 2024-09-21T14:10:39.909 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.909 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.909 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:10:39.909 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.332931+0000", 2024-09-21T14:10:39.909 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9709999999999995e-05 2024-09-21T14:10:39.909 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.909 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.909 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,7", 2024-09-21T14:10:39.909 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.333036+0000", 2024-09-21T14:10:39.909 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000105574 2024-09-21T14:10:39.909 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.909 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.910 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:10:39.910 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.336594+0000", 2024-09-21T14:10:39.910 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003558276 2024-09-21T14:10:39.910 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.910 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.910 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:39.910 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.336786+0000", 2024-09-21T14:10:39.910 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0001917 2024-09-21T14:10:39.910 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.910 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.910 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:39.910 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.341055+0000", 2024-09-21T14:10:39.910 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0042684890000000003 2024-09-21T14:10:39.911 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.911 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.911 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:10:39.911 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.341100+0000", 2024-09-21T14:10:39.911 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5145999999999998e-05 2024-09-21T14:10:39.911 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.911 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.911 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:39.911 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.341135+0000", 2024-09-21T14:10:39.911 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5145000000000003e-05 2024-09-21T14:10:39.911 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:39.911 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:39.911 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:39.911 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.912 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.912 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26396.0:183887 11.e 11:71f7a282:::benchmark_data_smithi049_206132_object60508:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e521)", 2024-09-21T14:10:39.912 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:09:27.472401+0000", 2024-09-21T14:10:39.912 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 72.362214051999999, 2024-09-21T14:10:39.912 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.86886711900000002, 2024-09-21T14:10:39.912 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:39.912 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:39.912 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:39.912 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:10:39.912 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26396", 2024-09-21T14:10:39.912 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:39.912 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 183887 2024-09-21T14:10:39.912 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.913 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:39.913 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.913 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:39.913 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.472401+0000", 2024-09-21T14:10:39.913 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:39.913 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.913 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.913 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:39.913 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.472401+0000", 2024-09-21T14:10:39.913 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:39.913 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.913 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.913 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:39.914 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.472404+0000", 2024-09-21T14:10:39.914 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0280000000000001e-06 2024-09-21T14:10:39.914 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.914 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.914 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:39.914 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.472413+0000", 2024-09-21T14:10:39.914 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4540000000000007e-06 2024-09-21T14:10:39.914 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.914 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.914 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:39.914 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.472435+0000", 2024-09-21T14:10:39.914 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2475000000000001e-05 2024-09-21T14:10:39.914 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.914 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.915 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:39.915 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.472445+0000", 2024-09-21T14:10:39.915 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3330000000000003e-06 2024-09-21T14:10:39.915 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.915 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.915 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:39.915 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.472735+0000", 2024-09-21T14:10:39.915 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00029017399999999998 2024-09-21T14:10:39.915 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.915 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.915 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:39.915 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.472744+0000", 2024-09-21T14:10:39.915 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1409999999999994e-06 2024-09-21T14:10:39.916 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.916 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.916 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:39.916 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.334589+0000", 2024-09-21T14:10:39.916 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.86184512499999999 2024-09-21T14:10:39.916 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.916 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.916 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:10:39.916 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.334689+0000", 2024-09-21T14:10:39.916 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010047000000000001 2024-09-21T14:10:39.916 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.916 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.916 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,7", 2024-09-21T14:10:39.916 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.334793+0000", 2024-09-21T14:10:39.916 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010388200000000001 2024-09-21T14:10:39.916 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.917 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.917 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:10:39.917 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.336755+0000", 2024-09-21T14:10:39.917 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0019620420000000002 2024-09-21T14:10:39.917 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.917 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.917 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:39.917 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.340768+0000", 2024-09-21T14:10:39.917 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0040122320000000001 2024-09-21T14:10:39.917 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.917 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.917 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:39.917 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.341192+0000", 2024-09-21T14:10:39.917 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00042469399999999998 2024-09-21T14:10:39.917 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.918 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.918 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:10:39.918 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.341244+0000", 2024-09-21T14:10:39.918 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2132000000000002e-05 2024-09-21T14:10:39.918 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.918 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.918 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:39.918 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.341268+0000", 2024-09-21T14:10:39.918 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3937000000000001e-05 2024-09-21T14:10:39.918 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:39.918 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:39.918 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:39.918 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.919 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.919 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26396.0:183902 11.e 11:77ce24cc:::benchmark_data_smithi049_206132_object60523:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e521)", 2024-09-21T14:10:39.919 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:09:27.487659+0000", 2024-09-21T14:10:39.919 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 72.346956453000004, 2024-09-21T14:10:39.919 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.85374666499999996, 2024-09-21T14:10:39.919 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:39.919 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:39.919 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:39.919 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:10:39.919 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26396", 2024-09-21T14:10:39.919 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:39.919 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 183902 2024-09-21T14:10:39.919 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.919 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:39.920 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.920 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:39.920 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.487659+0000", 2024-09-21T14:10:39.920 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:39.920 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.920 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.920 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:39.920 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.487659+0000", 2024-09-21T14:10:39.920 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:39.920 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.920 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.920 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:39.920 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.487661+0000", 2024-09-21T14:10:39.920 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.165e-06 2024-09-21T14:10:39.921 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.921 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.921 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:39.921 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.487666+0000", 2024-09-21T14:10:39.921 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3329999999999996e-06 2024-09-21T14:10:39.921 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.921 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.921 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:39.921 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.487683+0000", 2024-09-21T14:10:39.921 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6385999999999999e-05 2024-09-21T14:10:39.921 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.921 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.921 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:39.921 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.487690+0000", 2024-09-21T14:10:39.921 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7489999999999998e-06 2024-09-21T14:10:39.921 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.922 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.922 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:39.922 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.487755+0000", 2024-09-21T14:10:39.922 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4785999999999999e-05 2024-09-21T14:10:39.922 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.922 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.922 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:39.922 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.487769+0000", 2024-09-21T14:10:39.922 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4232e-05 2024-09-21T14:10:39.922 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.922 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.922 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:39.922 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.336813+0000", 2024-09-21T14:10:39.923 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.84904370299999998 2024-09-21T14:10:39.923 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.923 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.923 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:10:39.923 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.336919+0000", 2024-09-21T14:10:39.923 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010581699999999999 2024-09-21T14:10:39.923 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.923 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.923 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,7", 2024-09-21T14:10:39.923 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.337021+0000", 2024-09-21T14:10:39.923 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000102328 2024-09-21T14:10:39.923 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.923 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.923 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:39.923 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.340996+0000", 2024-09-21T14:10:39.924 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0039748479999999996 2024-09-21T14:10:39.924 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.924 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.924 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:10:39.924 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.341014+0000", 2024-09-21T14:10:39.924 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.806e-05 2024-09-21T14:10:39.924 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.924 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.924 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:39.924 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.341316+0000", 2024-09-21T14:10:39.924 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00030193199999999998 2024-09-21T14:10:39.924 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.924 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.924 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:10:39.924 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.341364+0000", 2024-09-21T14:10:39.925 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8149000000000003e-05 2024-09-21T14:10:39.925 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.925 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.925 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:39.925 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.341405+0000", 2024-09-21T14:10:39.925 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1177000000000002e-05 2024-09-21T14:10:39.925 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:39.925 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:39.925 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:39.925 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.925 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.925 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26396.0:183905 11.e 11:7db2147a:::benchmark_data_smithi049_206132_object60526:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e521)", 2024-09-21T14:10:39.925 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:09:27.489990+0000", 2024-09-21T14:10:39.926 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 72.344625324000006, 2024-09-21T14:10:39.926 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.85580929100000003, 2024-09-21T14:10:39.926 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:39.926 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:39.926 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:39.926 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:10:39.926 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26396", 2024-09-21T14:10:39.926 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:39.926 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 183905 2024-09-21T14:10:39.926 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.926 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:39.926 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.926 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:39.926 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.489990+0000", 2024-09-21T14:10:39.926 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:39.927 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.927 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.927 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:39.927 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.489990+0000", 2024-09-21T14:10:39.927 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:39.927 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.927 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.927 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:39.927 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.489992+0000", 2024-09-21T14:10:39.927 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7680000000000001e-06 2024-09-21T14:10:39.927 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.927 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.927 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:39.927 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.489997+0000", 2024-09-21T14:10:39.928 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8130000000000003e-06 2024-09-21T14:10:39.928 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.928 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.928 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:39.928 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.490012+0000", 2024-09-21T14:10:39.928 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5247e-05 2024-09-21T14:10:39.928 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.928 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.928 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:39.928 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.490019+0000", 2024-09-21T14:10:39.928 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1350000000000003e-06 2024-09-21T14:10:39.928 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.928 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.928 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:39.928 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.490079+0000", 2024-09-21T14:10:39.928 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9673999999999998e-05 2024-09-21T14:10:39.929 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.929 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.929 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:39.929 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.490087+0000", 2024-09-21T14:10:39.929 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3319999999999992e-06 2024-09-21T14:10:39.929 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.929 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.929 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:39.929 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.338714+0000", 2024-09-21T14:10:39.929 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.84862704099999997 2024-09-21T14:10:39.929 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.930 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.930 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:10:39.930 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.338807+0000", 2024-09-21T14:10:39.930 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3331999999999993e-05 2024-09-21T14:10:39.930 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.930 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.930 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,7", 2024-09-21T14:10:39.930 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.338915+0000", 2024-09-21T14:10:39.930 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010770899999999999 2024-09-21T14:10:39.930 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.930 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.931 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:10:39.931 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.341019+0000", 2024-09-21T14:10:39.931 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0021041660000000002 2024-09-21T14:10:39.931 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.931 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.931 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:39.931 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.341452+0000", 2024-09-21T14:10:39.931 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00043269400000000001 2024-09-21T14:10:39.931 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.931 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.931 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:39.932 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.345669+0000", 2024-09-21T14:10:39.932 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0042169699999999996 2024-09-21T14:10:39.932 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.932 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.933 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:10:39.933 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.345756+0000", 2024-09-21T14:10:39.933 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7138000000000005e-05 2024-09-21T14:10:39.933 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.933 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.933 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:39.933 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.345799+0000", 2024-09-21T14:10:39.933 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3272000000000002e-05 2024-09-21T14:10:39.933 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:39.933 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:39.933 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:39.933 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.933 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.933 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26396.0:183911 11.e 11:7556b2ae:::benchmark_data_smithi049_206132_object60532:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e521)", 2024-09-21T14:10:39.933 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:09:27.495266+0000", 2024-09-21T14:10:39.934 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 72.339349085999999, 2024-09-21T14:10:39.934 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.86135666099999997, 2024-09-21T14:10:39.934 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:39.934 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:39.934 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:39.934 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:10:39.934 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26396", 2024-09-21T14:10:39.934 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:39.934 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 183911 2024-09-21T14:10:39.934 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.934 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:39.934 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.934 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:39.935 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.495266+0000", 2024-09-21T14:10:39.935 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:39.935 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.935 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.935 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:39.935 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.495266+0000", 2024-09-21T14:10:39.935 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:39.935 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.935 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.935 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:39.935 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.495268+0000", 2024-09-21T14:10:39.935 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3799999999999999e-06 2024-09-21T14:10:39.935 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.935 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.935 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:39.936 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.495272+0000", 2024-09-21T14:10:39.936 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5990000000000004e-06 2024-09-21T14:10:39.936 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.936 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.936 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:39.936 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.495284+0000", 2024-09-21T14:10:39.936 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1764e-05 2024-09-21T14:10:39.936 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.936 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.936 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:39.936 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.495292+0000", 2024-09-21T14:10:39.936 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6899999999999992e-06 2024-09-21T14:10:39.936 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.936 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.936 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:39.937 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.495351+0000", 2024-09-21T14:10:39.937 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9132999999999998e-05 2024-09-21T14:10:39.937 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.937 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.937 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:39.937 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.495359+0000", 2024-09-21T14:10:39.937 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.6959999999999994e-06 2024-09-21T14:10:39.937 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.937 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.937 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:39.937 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.341486+0000", 2024-09-21T14:10:39.937 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.84612605600000002 2024-09-21T14:10:39.937 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.937 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.937 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:10:39.938 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.341589+0000", 2024-09-21T14:10:39.938 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000103223 2024-09-21T14:10:39.938 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.938 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.938 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,7", 2024-09-21T14:10:39.938 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.341703+0000", 2024-09-21T14:10:39.938 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000114647 2024-09-21T14:10:39.938 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.938 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.938 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:10:39.938 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.350068+0000", 2024-09-21T14:10:39.938 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0083650849999999995 2024-09-21T14:10:39.938 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.938 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.938 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:39.939 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.356430+0000", 2024-09-21T14:10:39.939 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0063619310000000004 2024-09-21T14:10:39.939 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.939 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.939 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:39.939 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.356473+0000", 2024-09-21T14:10:39.939 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2698999999999999e-05 2024-09-21T14:10:39.939 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.939 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.939 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:10:39.939 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.356576+0000", 2024-09-21T14:10:39.939 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000103006 2024-09-21T14:10:39.939 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.939 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.939 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:39.939 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.356623+0000", 2024-09-21T14:10:39.940 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6752000000000002e-05 2024-09-21T14:10:39.940 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:39.940 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:39.940 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:39.940 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.940 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.940 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26396.0:183926 11.e 11:7c49e1e5:::benchmark_data_smithi049_206132_object60547:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e521)", 2024-09-21T14:10:39.940 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:09:27.509276+0000", 2024-09-21T14:10:39.940 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 72.325339209000006, 2024-09-21T14:10:39.940 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.84759120200000004, 2024-09-21T14:10:39.940 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:39.940 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:39.940 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:39.940 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:10:39.940 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26396", 2024-09-21T14:10:39.940 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:39.941 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 183926 2024-09-21T14:10:39.941 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.941 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:39.941 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.941 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:39.941 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.509276+0000", 2024-09-21T14:10:39.941 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:39.941 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.941 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.941 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:39.941 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.509276+0000", 2024-09-21T14:10:39.941 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:39.941 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.941 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.941 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:39.941 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.509278+0000", 2024-09-21T14:10:39.942 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.623e-06 2024-09-21T14:10:39.942 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.942 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.942 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:39.942 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.509284+0000", 2024-09-21T14:10:39.942 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9000000000000003e-06 2024-09-21T14:10:39.942 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.942 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.942 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:39.942 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.509299+0000", 2024-09-21T14:10:39.942 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4945e-05 2024-09-21T14:10:39.942 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.942 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.942 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:39.942 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.509307+0000", 2024-09-21T14:10:39.943 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0209999999999999e-06 2024-09-21T14:10:39.943 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.943 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.943 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:39.943 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.509370+0000", 2024-09-21T14:10:39.943 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3898000000000004e-05 2024-09-21T14:10:39.943 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.943 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.943 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:39.943 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.509380+0000", 2024-09-21T14:10:39.943 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.9750000000000002e-06 2024-09-21T14:10:39.943 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.943 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.943 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:39.943 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.343534+0000", 2024-09-21T14:10:39.943 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.83415320599999998 2024-09-21T14:10:39.944 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.944 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.944 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:10:39.944 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.343627+0000", 2024-09-21T14:10:39.944 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3639000000000003e-05 2024-09-21T14:10:39.944 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.944 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.944 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,7", 2024-09-21T14:10:39.944 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.343727+0000", 2024-09-21T14:10:39.944 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.9547000000000001e-05 2024-09-21T14:10:39.944 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.944 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.944 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:10:39.944 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.352132+0000", 2024-09-21T14:10:39.944 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0084055060000000001 2024-09-21T14:10:39.945 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.945 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.945 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:39.945 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.356671+0000", 2024-09-21T14:10:39.945 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0045387609999999997 2024-09-21T14:10:39.945 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.945 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.945 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:39.945 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.356729+0000", 2024-09-21T14:10:39.945 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7895e-05 2024-09-21T14:10:39.945 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.945 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.945 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:10:39.946 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.356820+0000", 2024-09-21T14:10:39.946 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1392999999999997e-05 2024-09-21T14:10:39.946 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.946 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.946 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:39.946 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.356867+0000", 2024-09-21T14:10:39.946 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6893e-05 2024-09-21T14:10:39.946 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:39.946 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:39.946 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:39.946 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.946 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.946 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26396.0:183945 11.e 11:734222ce:::benchmark_data_smithi049_206132_object60566:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e521)", 2024-09-21T14:10:39.946 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:09:27.531640+0000", 2024-09-21T14:10:39.947 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 72.302975748999998, 2024-09-21T14:10:39.947 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.82539555200000003, 2024-09-21T14:10:39.947 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:39.947 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:39.947 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:39.947 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:10:39.947 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26396", 2024-09-21T14:10:39.947 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:39.947 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 183945 2024-09-21T14:10:39.947 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.947 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:39.947 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.947 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:39.948 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.531640+0000", 2024-09-21T14:10:39.948 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:39.948 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.948 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.948 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:39.948 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.531640+0000", 2024-09-21T14:10:39.948 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:39.948 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.948 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.948 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:39.948 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.531641+0000", 2024-09-21T14:10:39.948 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.641e-06 2024-09-21T14:10:39.948 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.949 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.949 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:39.949 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.531646+0000", 2024-09-21T14:10:39.949 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7899999999999999e-06 2024-09-21T14:10:39.949 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.949 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.949 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:39.949 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.531660+0000", 2024-09-21T14:10:39.949 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3976e-05 2024-09-21T14:10:39.949 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.949 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.949 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:39.949 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.531668+0000", 2024-09-21T14:10:39.950 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7710000000000006e-06 2024-09-21T14:10:39.950 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.950 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.950 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:39.950 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.531717+0000", 2024-09-21T14:10:39.950 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9509000000000003e-05 2024-09-21T14:10:39.950 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.950 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.950 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:39.950 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.531723+0000", 2024-09-21T14:10:39.950 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9610000000000002e-06 2024-09-21T14:10:39.950 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.950 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.951 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:39.951 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.345867+0000", 2024-09-21T14:10:39.951 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.81414361300000004 2024-09-21T14:10:39.951 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.951 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.951 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:10:39.951 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.345985+0000", 2024-09-21T14:10:39.951 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011785699999999999 2024-09-21T14:10:39.951 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.951 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.951 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,7", 2024-09-21T14:10:39.951 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.346098+0000", 2024-09-21T14:10:39.952 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000113038 2024-09-21T14:10:39.952 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.952 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.952 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:10:39.952 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.352138+0000", 2024-09-21T14:10:39.952 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0060399340000000003 2024-09-21T14:10:39.952 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.952 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.952 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:39.952 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.356915+0000", 2024-09-21T14:10:39.952 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0047773540000000001 2024-09-21T14:10:39.952 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.953 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.953 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:39.953 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.356966+0000", 2024-09-21T14:10:39.953 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1393000000000001e-05 2024-09-21T14:10:39.953 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.953 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.953 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:10:39.953 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.356998+0000", 2024-09-21T14:10:39.953 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1516000000000001e-05 2024-09-21T14:10:39.953 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.953 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.953 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:39.953 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.357035+0000", 2024-09-21T14:10:39.953 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7199000000000001e-05 2024-09-21T14:10:39.954 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:39.954 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:39.954 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:39.954 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.954 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.954 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26396.0:183953 11.e 11:7047bbfb:::benchmark_data_smithi049_206132_object60574:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e521)", 2024-09-21T14:10:39.954 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:09:27.536517+0000", 2024-09-21T14:10:39.954 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 72.298098038000006, 2024-09-21T14:10:39.954 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.820687685, 2024-09-21T14:10:39.954 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:39.954 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:39.954 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:39.954 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:10:39.954 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26396", 2024-09-21T14:10:39.955 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:39.955 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 183953 2024-09-21T14:10:39.955 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.955 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:39.955 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.955 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:39.955 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.536517+0000", 2024-09-21T14:10:39.955 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:39.955 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.955 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.955 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:39.955 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.536517+0000", 2024-09-21T14:10:39.955 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:39.955 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.955 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.956 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:39.956 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.536519+0000", 2024-09-21T14:10:39.956 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0250000000000001e-06 2024-09-21T14:10:39.956 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.956 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.956 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:39.956 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.536525+0000", 2024-09-21T14:10:39.956 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.755e-06 2024-09-21T14:10:39.956 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.956 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.956 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:39.956 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.536541+0000", 2024-09-21T14:10:39.956 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5613000000000001e-05 2024-09-21T14:10:39.956 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.956 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.956 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:39.957 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.536549+0000", 2024-09-21T14:10:39.957 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3580000000000003e-06 2024-09-21T14:10:39.957 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.957 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.957 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:39.957 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.536608+0000", 2024-09-21T14:10:39.957 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9098000000000002e-05 2024-09-21T14:10:39.957 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.957 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.957 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:39.957 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.536617+0000", 2024-09-21T14:10:39.957 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.9090000000000006e-06 2024-09-21T14:10:39.957 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.957 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.957 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:39.958 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.347953+0000", 2024-09-21T14:10:39.958 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.81133610499999997 2024-09-21T14:10:39.958 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.958 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.958 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:10:39.958 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.348048+0000", 2024-09-21T14:10:39.958 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.4749999999999999e-05 2024-09-21T14:10:39.958 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.958 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.958 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,7", 2024-09-21T14:10:39.958 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.348146+0000", 2024-09-21T14:10:39.958 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.8289999999999996e-05 2024-09-21T14:10:39.958 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.958 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.959 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:10:39.959 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.354364+0000", 2024-09-21T14:10:39.959 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0062183129999999996 2024-09-21T14:10:39.959 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.959 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.959 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:39.959 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.357080+0000", 2024-09-21T14:10:39.959 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0027153289999999998 2024-09-21T14:10:39.959 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.959 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.959 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:39.959 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.357123+0000", 2024-09-21T14:10:39.959 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2972000000000001e-05 2024-09-21T14:10:39.959 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.959 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.960 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:10:39.960 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.357171+0000", 2024-09-21T14:10:39.960 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8306999999999997e-05 2024-09-21T14:10:39.960 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.960 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.960 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:39.960 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.357205+0000", 2024-09-21T14:10:39.960 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3861e-05 2024-09-21T14:10:39.960 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:39.960 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:39.960 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:39.960 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.960 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.960 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26396.0:183960 11.e 11:70bda135:::benchmark_data_smithi049_206132_object60581:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e521)", 2024-09-21T14:10:39.961 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:09:27.543699+0000", 2024-09-21T14:10:39.961 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 72.290916394000007, 2024-09-21T14:10:39.961 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.813649701, 2024-09-21T14:10:39.961 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:39.961 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:39.961 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:39.961 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:10:39.961 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26396", 2024-09-21T14:10:39.961 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:39.961 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 183960 2024-09-21T14:10:39.961 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.961 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:39.961 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.961 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:39.962 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.543699+0000", 2024-09-21T14:10:39.962 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:39.962 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.962 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.962 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:39.962 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.543699+0000", 2024-09-21T14:10:39.962 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:39.962 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.962 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.962 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:39.962 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.543700+0000", 2024-09-21T14:10:39.962 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6109999999999999e-06 2024-09-21T14:10:39.963 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.963 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.963 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:39.963 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.543706+0000", 2024-09-21T14:10:39.963 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6219999999999997e-06 2024-09-21T14:10:39.963 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.963 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.963 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:39.963 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.543723+0000", 2024-09-21T14:10:39.963 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6455000000000001e-05 2024-09-21T14:10:39.963 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.963 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.963 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:39.963 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.543730+0000", 2024-09-21T14:10:39.963 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1899999999999998e-06 2024-09-21T14:10:39.964 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.964 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.964 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:39.964 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.543802+0000", 2024-09-21T14:10:39.964 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1931e-05 2024-09-21T14:10:39.964 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.964 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.964 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:39.964 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.543811+0000", 2024-09-21T14:10:39.964 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1269999999999992e-06 2024-09-21T14:10:39.964 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.964 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.964 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:39.964 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.350102+0000", 2024-09-21T14:10:39.965 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.80629132400000003 2024-09-21T14:10:39.965 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.965 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.965 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:10:39.965 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.350222+0000", 2024-09-21T14:10:39.965 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000120155 2024-09-21T14:10:39.965 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.965 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.965 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,7", 2024-09-21T14:10:39.965 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.350326+0000", 2024-09-21T14:10:39.965 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010362200000000001 2024-09-21T14:10:39.965 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.965 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.966 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:10:39.966 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.356377+0000", 2024-09-21T14:10:39.966 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0060509600000000002 2024-09-21T14:10:39.966 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.966 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.966 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:39.966 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.357244+0000", 2024-09-21T14:10:39.966 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00086724600000000001 2024-09-21T14:10:39.966 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.966 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.966 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:39.967 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.357270+0000", 2024-09-21T14:10:39.967 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5537999999999998e-05 2024-09-21T14:10:39.967 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.967 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.967 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:10:39.967 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.357314+0000", 2024-09-21T14:10:39.967 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4799000000000003e-05 2024-09-21T14:10:39.967 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.967 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.967 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:39.967 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.357349+0000", 2024-09-21T14:10:39.967 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4121000000000001e-05 2024-09-21T14:10:39.967 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:39.968 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:39.968 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:39.968 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.968 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.968 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26396.0:183962 11.e 11:78830aa9:::benchmark_data_smithi049_206132_object60583:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e521)", 2024-09-21T14:10:39.968 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:09:27.546614+0000", 2024-09-21T14:10:39.968 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 72.288001346000001, 2024-09-21T14:10:39.968 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.81808440299999996, 2024-09-21T14:10:39.968 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:39.968 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:39.968 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:39.968 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:10:39.968 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26396", 2024-09-21T14:10:39.968 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:39.969 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 183962 2024-09-21T14:10:39.969 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.969 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:39.969 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.969 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:39.969 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.546614+0000", 2024-09-21T14:10:39.969 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:39.969 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.969 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.969 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:39.969 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.546614+0000", 2024-09-21T14:10:39.969 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:39.969 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.970 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.970 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:39.970 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.546615+0000", 2024-09-21T14:10:39.970 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.78e-07 2024-09-21T14:10:39.970 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.970 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.970 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:39.970 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.546619+0000", 2024-09-21T14:10:39.970 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1219999999999997e-06 2024-09-21T14:10:39.970 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.970 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.970 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:39.970 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.546627+0000", 2024-09-21T14:10:39.970 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7000000000000008e-06 2024-09-21T14:10:39.971 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.971 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.971 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:39.971 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.546631+0000", 2024-09-21T14:10:39.971 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.42e-06 2024-09-21T14:10:39.971 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.971 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.971 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:39.971 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.546670+0000", 2024-09-21T14:10:39.971 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9113000000000003e-05 2024-09-21T14:10:39.971 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.971 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.971 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:39.971 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.546676+0000", 2024-09-21T14:10:39.971 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5550000000000002e-06 2024-09-21T14:10:39.971 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.972 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.972 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:39.972 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.352166+0000", 2024-09-21T14:10:39.972 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.80548991199999997 2024-09-21T14:10:39.972 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.972 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.972 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:10:39.972 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.352255+0000", 2024-09-21T14:10:39.972 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.9639e-05 2024-09-21T14:10:39.972 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.972 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.972 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,7", 2024-09-21T14:10:39.972 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.352354+0000", 2024-09-21T14:10:39.972 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.9019999999999997e-05 2024-09-21T14:10:39.972 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.972 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.973 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:10:39.973 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.356880+0000", 2024-09-21T14:10:39.973 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0045254329999999997 2024-09-21T14:10:39.973 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.973 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.973 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:39.973 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.357395+0000", 2024-09-21T14:10:39.973 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000514852 2024-09-21T14:10:39.973 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.973 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.973 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:39.973 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.364613+0000", 2024-09-21T14:10:39.973 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0072185169999999998 2024-09-21T14:10:39.974 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.974 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.974 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:10:39.974 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.364659+0000", 2024-09-21T14:10:39.974 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6272e-05 2024-09-21T14:10:39.974 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.974 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.974 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:39.974 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.364698+0000", 2024-09-21T14:10:39.974 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8970000000000001e-05 2024-09-21T14:10:39.974 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:39.974 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:39.974 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:39.975 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.975 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.975 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26396.0:183968 11.e 11:7edba757:::benchmark_data_smithi049_206132_object60589:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e521)", 2024-09-21T14:10:39.975 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:09:27.555939+0000", 2024-09-21T14:10:39.975 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 72.278676461000003, 2024-09-21T14:10:39.975 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.80915244099999994, 2024-09-21T14:10:39.975 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:39.975 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:39.975 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:39.975 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:10:39.975 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26396", 2024-09-21T14:10:39.975 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:39.975 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 183968 2024-09-21T14:10:39.975 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.975 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:39.976 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.976 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:39.976 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.555939+0000", 2024-09-21T14:10:39.976 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:39.976 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.976 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.976 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:39.976 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.555939+0000", 2024-09-21T14:10:39.976 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:39.976 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.976 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.976 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:39.976 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.555941+0000", 2024-09-21T14:10:39.976 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0310000000000001e-06 2024-09-21T14:10:39.976 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.976 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.977 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:39.977 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.555951+0000", 2024-09-21T14:10:39.977 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.9299999999999998e-06 2024-09-21T14:10:39.977 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.977 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.977 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:39.977 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.555964+0000", 2024-09-21T14:10:39.977 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.367e-05 2024-09-21T14:10:39.977 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.977 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.977 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:39.977 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.555972+0000", 2024-09-21T14:10:39.977 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.413e-06 2024-09-21T14:10:39.977 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.977 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.977 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:39.978 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.556034+0000", 2024-09-21T14:10:39.978 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1774999999999997e-05 2024-09-21T14:10:39.978 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.978 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.978 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:39.978 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:27.556043+0000", 2024-09-21T14:10:39.978 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.5049999999999996e-06 2024-09-21T14:10:39.978 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.978 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.978 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:39.978 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.354395+0000", 2024-09-21T14:10:39.978 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.79835209600000001 2024-09-21T14:10:39.978 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.978 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.979 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:10:39.979 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.354509+0000", 2024-09-21T14:10:39.979 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011370300000000001 2024-09-21T14:10:39.979 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.979 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.979 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,7", 2024-09-21T14:10:39.979 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.354649+0000", 2024-09-21T14:10:39.979 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000140111 2024-09-21T14:10:39.979 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.979 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.979 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:10:39.979 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.356925+0000", 2024-09-21T14:10:39.979 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.002276307 2024-09-21T14:10:39.979 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.979 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.979 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:39.980 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.364754+0000", 2024-09-21T14:10:39.980 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0078281749999999997 2024-09-21T14:10:39.980 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.980 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.980 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:39.980 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.365010+0000", 2024-09-21T14:10:39.980 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00025618499999999997 2024-09-21T14:10:39.980 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.980 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.980 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:10:39.980 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.365055+0000", 2024-09-21T14:10:39.980 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5386999999999998e-05 2024-09-21T14:10:39.980 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:39.980 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:39.980 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:39.980 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:09:28.365091+0000", 2024-09-21T14:10:39.981 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6152999999999998e-05 2024-09-21T14:10:39.981 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:39.981 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:39.981 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:39.981 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:39.981 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:39.981 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T14:10:40.058 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T14:10:40.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:40 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-09-21T14:10:40.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:40 smithi187 ceph-mon[33032]: osdmap e554: 8 total, 8 up, 8 in 2024-09-21T14:10:40.444 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:10:40 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:10:40.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:40 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-09-21T14:10:40.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:40 smithi049 ceph-mon[40489]: osdmap e554: 8 total, 8 up, 8 in 2024-09-21T14:10:40.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:40 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-09-21T14:10:40.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:40 smithi049 ceph-mon[30080]: osdmap e554: 8 total, 8 up, 8 in 2024-09-21T14:10:40.584 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-21T14:10:40.677 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T14:10:40.696 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:10:40.817 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:10:40 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:10:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:10:40.920 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:10:41.411 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:10:41.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:41 smithi049 ceph-mon[40489]: 1.c scrub starts 2024-09-21T14:10:41.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:41 smithi049 ceph-mon[40489]: 1.c scrub ok 2024-09-21T14:10:41.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:41 smithi049 ceph-mon[40489]: pgmap v1119: 152 pgs: 152 active+clean; 448 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 865 B/s rd, 0 op/s 2024-09-21T14:10:41.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:41 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:10:41.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:41 smithi049 ceph-mon[40489]: 1.9 scrub starts 2024-09-21T14:10:41.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:41 smithi049 ceph-mon[40489]: 1.9 scrub ok 2024-09-21T14:10:41.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:41 smithi049 ceph-mon[30080]: 1.c scrub starts 2024-09-21T14:10:41.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:41 smithi049 ceph-mon[30080]: 1.c scrub ok 2024-09-21T14:10:41.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:41 smithi049 ceph-mon[30080]: pgmap v1119: 152 pgs: 152 active+clean; 448 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 865 B/s rd, 0 op/s 2024-09-21T14:10:41.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:41 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:10:41.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:41 smithi049 ceph-mon[30080]: 1.9 scrub starts 2024-09-21T14:10:41.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:41 smithi049 ceph-mon[30080]: 1.9 scrub ok 2024-09-21T14:10:41.543 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T14:10:41.543 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-21T14:10:41.543 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 0 2024-09-21T14:10:41.543 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T14:10:41.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:41 smithi187 ceph-mon[33032]: 1.c scrub starts 2024-09-21T14:10:41.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:41 smithi187 ceph-mon[33032]: 1.c scrub ok 2024-09-21T14:10:41.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:41 smithi187 ceph-mon[33032]: pgmap v1119: 152 pgs: 152 active+clean; 448 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 865 B/s rd, 0 op/s 2024-09-21T14:10:41.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:41 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:10:41.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:41 smithi187 ceph-mon[33032]: 1.9 scrub starts 2024-09-21T14:10:41.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:41 smithi187 ceph-mon[33032]: 1.9 scrub ok 2024-09-21T14:10:41.929 INFO:teuthology.orchestra.run.smithi049.stdout:mon.a: {} 2024-09-21T14:10:41.929 INFO:teuthology.orchestra.run.smithi049.stdout:mon.b: {} 2024-09-21T14:10:41.929 INFO:teuthology.orchestra.run.smithi049.stdout:mon.c: {} 2024-09-21T14:10:41.929 INFO:teuthology.orchestra.run.smithi049.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T14:10:41.929 INFO:teuthology.orchestra.run.smithi049.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T14:10:41.929 INFO:teuthology.orchestra.run.smithi049.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T14:10:42.201 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T14:10:42.311 INFO:teuthology.orchestra.run.smithi049.stderr:nodeep-scrub is unset 2024-09-21T14:10:42.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:42 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/304295032' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:10:42.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:42 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:10:42.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:42 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2052237660' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T14:10:42.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:42 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:10:42.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:42 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:10:42.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:42 smithi187 ceph-mon[33032]: 5.c scrub starts 2024-09-21T14:10:42.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:42 smithi187 ceph-mon[33032]: 5.c scrub ok 2024-09-21T14:10:42.451 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:10:42.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:42 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/304295032' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:10:42.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:42 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:10:42.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:42 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2052237660' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T14:10:42.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:42 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:10:42.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:42 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:10:42.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:42 smithi049 ceph-mon[30080]: 5.c scrub starts 2024-09-21T14:10:42.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:42 smithi049 ceph-mon[30080]: 5.c scrub ok 2024-09-21T14:10:42.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:42 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/304295032' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:10:42.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:42 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T14:10:42.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:42 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2052237660' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T14:10:42.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:42 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:10:42.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:42 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:10:42.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:42 smithi049 ceph-mon[40489]: 5.c scrub starts 2024-09-21T14:10:42.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:42 smithi049 ceph-mon[40489]: 5.c scrub ok 2024-09-21T14:10:42.902 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T14:10:42.902 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-21T14:10:42.902 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-21T14:10:42.902 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-21T14:10:42.903 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T14:10:43.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:43 smithi049 ceph-mon[40489]: 1.7 scrub starts 2024-09-21T14:10:43.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:43 smithi049 ceph-mon[40489]: 1.7 scrub ok 2024-09-21T14:10:43.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:43 smithi049 ceph-mon[40489]: pgmap v1120: 152 pgs: 152 active+clean; 448 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-21T14:10:43.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:43 smithi049 ceph-mon[40489]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T14:10:43.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:43 smithi049 ceph-mon[40489]: Cluster is now healthy 2024-09-21T14:10:43.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:43 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T14:10:43.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:43 smithi049 ceph-mon[40489]: osdmap e555: 8 total, 8 up, 8 in 2024-09-21T14:10:43.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:43 smithi049 ceph-mon[40489]: 4.15 scrub starts 2024-09-21T14:10:43.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:43 smithi049 ceph-mon[40489]: 4.15 scrub ok 2024-09-21T14:10:43.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:43 smithi049 ceph-mon[40489]: 6.1d scrub starts 2024-09-21T14:10:43.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:43 smithi049 ceph-mon[40489]: 6.1d scrub ok 2024-09-21T14:10:43.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:43 smithi049 ceph-mon[30080]: 1.7 scrub starts 2024-09-21T14:10:43.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:43 smithi049 ceph-mon[30080]: 1.7 scrub ok 2024-09-21T14:10:43.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:43 smithi049 ceph-mon[30080]: pgmap v1120: 152 pgs: 152 active+clean; 448 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-21T14:10:43.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:43 smithi049 ceph-mon[30080]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T14:10:43.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:43 smithi049 ceph-mon[30080]: Cluster is now healthy 2024-09-21T14:10:43.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:43 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T14:10:43.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:43 smithi049 ceph-mon[30080]: osdmap e555: 8 total, 8 up, 8 in 2024-09-21T14:10:43.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:43 smithi049 ceph-mon[30080]: 4.15 scrub starts 2024-09-21T14:10:43.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:43 smithi049 ceph-mon[30080]: 4.15 scrub ok 2024-09-21T14:10:43.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:43 smithi049 ceph-mon[30080]: 6.1d scrub starts 2024-09-21T14:10:43.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:43 smithi049 ceph-mon[30080]: 6.1d scrub ok 2024-09-21T14:10:43.597 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:43 smithi187 ceph-mon[33032]: 1.7 scrub starts 2024-09-21T14:10:43.598 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:43 smithi187 ceph-mon[33032]: 1.7 scrub ok 2024-09-21T14:10:43.598 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:43 smithi187 ceph-mon[33032]: pgmap v1120: 152 pgs: 152 active+clean; 448 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-21T14:10:43.598 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:43 smithi187 ceph-mon[33032]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T14:10:43.598 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:43 smithi187 ceph-mon[33032]: Cluster is now healthy 2024-09-21T14:10:43.598 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:43 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T14:10:43.598 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:43 smithi187 ceph-mon[33032]: osdmap e555: 8 total, 8 up, 8 in 2024-09-21T14:10:43.598 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:43 smithi187 ceph-mon[33032]: 4.15 scrub starts 2024-09-21T14:10:43.598 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:43 smithi187 ceph-mon[33032]: 4.15 scrub ok 2024-09-21T14:10:43.598 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:43 smithi187 ceph-mon[33032]: 6.1d scrub starts 2024-09-21T14:10:43.598 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:43 smithi187 ceph-mon[33032]: 6.1d scrub ok 2024-09-21T14:10:43.627 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T14:10:43.767 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd reweight 0 1 2024-09-21T14:10:43.884 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:10:44.034 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:10:44.385 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T14:10:44.385 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-21T14:10:44.385 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-21T14:10:44.385 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-21T14:10:44.385 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.385 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:112204 10.a 10:5c740623:::benchmark_data_smithi049_176811_object112203:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-21T14:10:44.385 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:04:20.206510+0000", 2024-09-21T14:10:44.386 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 384.16911908899999, 2024-09-21T14:10:44.386 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1818630240000001, 2024-09-21T14:10:44.386 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:44.386 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:44.386 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:44.386 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:10:44.386 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:10:44.386 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:10:44.386 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 112204 2024-09-21T14:10:44.386 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.386 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:44.386 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.386 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:44.387 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.206510+0000", 2024-09-21T14:10:44.387 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:44.387 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.387 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.387 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:44.387 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.206510+0000", 2024-09-21T14:10:44.387 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:44.387 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.387 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.387 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:44.387 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.206511+0000", 2024-09-21T14:10:44.387 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0900000000000001e-07 2024-09-21T14:10:44.387 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.387 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.388 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:44.388 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.206541+0000", 2024-09-21T14:10:44.388 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0082000000000001e-05 2024-09-21T14:10:44.388 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.388 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.388 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:44.388 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.206548+0000", 2024-09-21T14:10:44.388 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.0110000000000003e-06 2024-09-21T14:10:44.388 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.388 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.388 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:44.388 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.206553+0000", 2024-09-21T14:10:44.388 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1529999999999996e-06 2024-09-21T14:10:44.388 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.389 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.389 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.389 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.206601+0000", 2024-09-21T14:10:44.389 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8393000000000002e-05 2024-09-21T14:10:44.389 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.389 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.389 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:44.389 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.206610+0000", 2024-09-21T14:10:44.389 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.507e-06 2024-09-21T14:10:44.389 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.389 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.389 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.389 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.188308+0000", 2024-09-21T14:10:44.390 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.98169809900000005 2024-09-21T14:10:44.390 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.390 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.390 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:44.390 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.188318+0000", 2024-09-21T14:10:44.390 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0179e-05 2024-09-21T14:10:44.390 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.390 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.390 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.390 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.370141+0000", 2024-09-21T14:10:44.390 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1818224210000001 2024-09-21T14:10:44.390 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.390 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.391 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:10:44.391 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.370277+0000", 2024-09-21T14:10:44.391 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013640399999999999 2024-09-21T14:10:44.391 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.391 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.391 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2,5", 2024-09-21T14:10:44.391 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.370711+0000", 2024-09-21T14:10:44.391 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000433826 2024-09-21T14:10:44.391 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.391 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.391 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:44.391 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.377442+0000", 2024-09-21T14:10:44.391 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0067311239999999998 2024-09-21T14:10:44.392 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.392 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.392 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:10:44.392 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.380659+0000", 2024-09-21T14:10:44.392 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0032168869999999999 2024-09-21T14:10:44.392 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.392 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.392 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:44.392 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.388142+0000", 2024-09-21T14:10:44.392 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0074832570000000001 2024-09-21T14:10:44.392 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.392 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.392 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:44.392 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.388212+0000", 2024-09-21T14:10:44.392 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9850000000000004e-05 2024-09-21T14:10:44.393 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.393 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.393 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:10:44.393 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.388269+0000", 2024-09-21T14:10:44.393 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7224e-05 2024-09-21T14:10:44.393 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.393 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.393 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:44.393 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.388373+0000", 2024-09-21T14:10:44.393 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000103998 2024-09-21T14:10:44.393 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:44.393 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:44.393 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:44.393 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.394 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.394 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:112222 10.d 10:bb42e81a:::benchmark_data_smithi049_176811_object112221:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-21T14:10:44.394 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:04:20.235680+0000", 2024-09-21T14:10:44.394 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 384.13994962800001, 2024-09-21T14:10:44.394 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0718219699999998, 2024-09-21T14:10:44.394 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:44.394 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:44.394 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:44.394 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:10:44.394 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:10:44.394 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:10:44.394 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 112222 2024-09-21T14:10:44.394 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.395 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:44.395 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.395 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:44.395 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235680+0000", 2024-09-21T14:10:44.395 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:44.395 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.395 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.395 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:44.395 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235680+0000", 2024-09-21T14:10:44.395 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:44.395 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.395 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.395 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:44.395 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235681+0000", 2024-09-21T14:10:44.396 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.285e-06 2024-09-21T14:10:44.396 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.396 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.396 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:44.396 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235734+0000", 2024-09-21T14:10:44.396 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2580000000000001e-05 2024-09-21T14:10:44.396 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.396 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.396 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:44.396 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235744+0000", 2024-09-21T14:10:44.396 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0197000000000001e-05 2024-09-21T14:10:44.396 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.396 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.396 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:44.396 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235749+0000", 2024-09-21T14:10:44.397 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3809999999999998e-06 2024-09-21T14:10:44.397 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.397 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.397 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.397 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235804+0000", 2024-09-21T14:10:44.397 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4592999999999997e-05 2024-09-21T14:10:44.397 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.397 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.397 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:44.397 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235814+0000", 2024-09-21T14:10:44.397 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0545e-05 2024-09-21T14:10:44.397 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.397 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.397 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.398 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.187322+0000", 2024-09-21T14:10:44.398 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.95150779399999996 2024-09-21T14:10:44.398 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.398 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.398 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:44.398 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.187331+0000", 2024-09-21T14:10:44.398 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.6929999999999996e-06 2024-09-21T14:10:44.398 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.398 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.398 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.398 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.285689+0000", 2024-09-21T14:10:44.398 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.098357961 2024-09-21T14:10:44.398 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.398 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.399 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:10:44.399 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.285795+0000", 2024-09-21T14:10:44.399 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010604599999999999 2024-09-21T14:10:44.399 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.399 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.399 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3,4", 2024-09-21T14:10:44.399 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.286188+0000", 2024-09-21T14:10:44.399 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00039307900000000001 2024-09-21T14:10:44.399 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.399 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.399 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:44.399 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.297975+0000", 2024-09-21T14:10:44.399 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.011787446 2024-09-21T14:10:44.400 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.400 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.400 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:44.400 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.298048+0000", 2024-09-21T14:10:44.400 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3146999999999996e-05 2024-09-21T14:10:44.400 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.400 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.400 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:10:44.400 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.298141+0000", 2024-09-21T14:10:44.400 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.2391000000000004e-05 2024-09-21T14:10:44.400 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.400 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.400 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:44.400 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.307285+0000", 2024-09-21T14:10:44.401 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0091442880000000004 2024-09-21T14:10:44.401 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.401 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.401 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:10:44.401 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.307366+0000", 2024-09-21T14:10:44.401 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1155999999999993e-05 2024-09-21T14:10:44.401 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.401 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.401 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:44.401 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.307502+0000", 2024-09-21T14:10:44.401 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013538800000000001 2024-09-21T14:10:44.401 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:44.401 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:44.402 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:44.402 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.402 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.402 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:112223 10.0 10:05563d3b:::benchmark_data_smithi049_176811_object112222:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-21T14:10:44.402 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:04:20.235762+0000", 2024-09-21T14:10:44.402 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 384.139867756, 2024-09-21T14:10:44.402 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1264867669999998, 2024-09-21T14:10:44.402 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:44.402 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:44.402 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:44.402 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:10:44.402 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:10:44.402 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:10:44.402 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 112223 2024-09-21T14:10:44.402 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.403 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:44.403 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.403 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:44.403 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235762+0000", 2024-09-21T14:10:44.403 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:44.403 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.403 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.403 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:44.403 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235762+0000", 2024-09-21T14:10:44.403 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:44.403 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.403 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.403 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:44.403 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235762+0000", 2024-09-21T14:10:44.403 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2799999999999996e-07 2024-09-21T14:10:44.404 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.404 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.404 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:44.404 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235796+0000", 2024-09-21T14:10:44.404 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3402000000000003e-05 2024-09-21T14:10:44.404 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.404 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.404 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:44.404 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235803+0000", 2024-09-21T14:10:44.404 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9929999999999998e-06 2024-09-21T14:10:44.404 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.404 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.404 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:44.404 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235808+0000", 2024-09-21T14:10:44.405 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9699999999999998e-06 2024-09-21T14:10:44.405 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.405 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.405 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.405 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235863+0000", 2024-09-21T14:10:44.405 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5877000000000001e-05 2024-09-21T14:10:44.405 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.405 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.405 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:44.405 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235870+0000", 2024-09-21T14:10:44.405 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.007e-06 2024-09-21T14:10:44.405 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.405 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.406 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.406 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.186804+0000", 2024-09-21T14:10:44.406 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.95093328600000004 2024-09-21T14:10:44.406 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.406 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.406 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:44.406 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.186815+0000", 2024-09-21T14:10:44.406 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1265e-05 2024-09-21T14:10:44.406 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.406 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.406 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.406 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.336944+0000", 2024-09-21T14:10:44.406 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1501292430000001 2024-09-21T14:10:44.406 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.407 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.407 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:10:44.407 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.337058+0000", 2024-09-21T14:10:44.407 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000113881 2024-09-21T14:10:44.407 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.407 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.407 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,4,7", 2024-09-21T14:10:44.407 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.337547+0000", 2024-09-21T14:10:44.407 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00048919999999999996 2024-09-21T14:10:44.407 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.407 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.407 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:44.407 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.346097+0000", 2024-09-21T14:10:44.407 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0085500969999999996 2024-09-21T14:10:44.407 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.408 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.408 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:44.408 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.346145+0000", 2024-09-21T14:10:44.408 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7700000000000001e-05 2024-09-21T14:10:44.408 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.408 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.408 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:10:44.408 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.362046+0000", 2024-09-21T14:10:44.408 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.015900935000000001 2024-09-21T14:10:44.408 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.408 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.408 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:44.408 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.362138+0000", 2024-09-21T14:10:44.409 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1525000000000002e-05 2024-09-21T14:10:44.409 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.409 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.409 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:10:44.409 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.362175+0000", 2024-09-21T14:10:44.409 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6928999999999998e-05 2024-09-21T14:10:44.409 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.409 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.409 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:44.409 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.362248+0000", 2024-09-21T14:10:44.409 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3829000000000001e-05 2024-09-21T14:10:44.409 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:44.409 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:44.409 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:44.409 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.410 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.410 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:112226 10.0 10:0f5734d7:::benchmark_data_smithi049_176811_object112225:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-21T14:10:44.410 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:04:20.235905+0000", 2024-09-21T14:10:44.410 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 384.13972428599999, 2024-09-21T14:10:44.410 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1266882329999999, 2024-09-21T14:10:44.410 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:44.410 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:44.410 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:44.410 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:10:44.410 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:10:44.410 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:10:44.410 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 112226 2024-09-21T14:10:44.410 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.410 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:44.411 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.411 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:44.411 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235905+0000", 2024-09-21T14:10:44.411 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:44.411 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.411 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.411 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:44.411 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235905+0000", 2024-09-21T14:10:44.411 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:44.411 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.411 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.411 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:44.411 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235906+0000", 2024-09-21T14:10:44.411 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4e-07 2024-09-21T14:10:44.412 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.412 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.412 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:44.412 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235975+0000", 2024-09-21T14:10:44.412 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9052000000000002e-05 2024-09-21T14:10:44.412 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.412 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.412 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:44.412 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235985+0000", 2024-09-21T14:10:44.412 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.9690000000000006e-06 2024-09-21T14:10:44.412 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.412 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.412 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:44.412 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.235991+0000", 2024-09-21T14:10:44.413 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9939999999999997e-06 2024-09-21T14:10:44.413 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.413 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.413 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.413 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.236048+0000", 2024-09-21T14:10:44.413 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6943e-05 2024-09-21T14:10:44.413 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.413 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.413 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:44.413 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.236057+0000", 2024-09-21T14:10:44.413 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.4790000000000002e-06 2024-09-21T14:10:44.413 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.413 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.413 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.414 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.187660+0000", 2024-09-21T14:10:44.414 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.95160278700000001 2024-09-21T14:10:44.414 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.414 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.414 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:44.414 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.187669+0000", 2024-09-21T14:10:44.414 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.2979999999999997e-06 2024-09-21T14:10:44.414 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.414 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.414 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.414 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.346204+0000", 2024-09-21T14:10:44.414 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1585341869999999 2024-09-21T14:10:44.414 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.414 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.415 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:10:44.415 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.346382+0000", 2024-09-21T14:10:44.415 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00017858500000000001 2024-09-21T14:10:44.415 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.415 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.415 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,4,7", 2024-09-21T14:10:44.415 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.346555+0000", 2024-09-21T14:10:44.415 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0001728 2024-09-21T14:10:44.415 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.415 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.415 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:44.415 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.353773+0000", 2024-09-21T14:10:44.415 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0072175240000000003 2024-09-21T14:10:44.415 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.415 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.416 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:44.416 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.362086+0000", 2024-09-21T14:10:44.416 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0083139189999999995 2024-09-21T14:10:44.416 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.416 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.416 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:10:44.416 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.362257+0000", 2024-09-21T14:10:44.416 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00017090200000000001 2024-09-21T14:10:44.416 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.416 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.416 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:44.416 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.362405+0000", 2024-09-21T14:10:44.416 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014748800000000001 2024-09-21T14:10:44.416 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.417 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.417 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:10:44.417 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.362477+0000", 2024-09-21T14:10:44.417 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2214999999999999e-05 2024-09-21T14:10:44.417 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.417 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.417 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:44.417 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.362593+0000", 2024-09-21T14:10:44.417 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000116251 2024-09-21T14:10:44.417 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:44.417 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:44.417 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:44.417 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.417 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.418 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:112230 10.d 10:b278ca69:::benchmark_data_smithi049_176811_object112229:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-21T14:10:44.418 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:04:20.239727+0000", 2024-09-21T14:10:44.418 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 384.13590203899997, 2024-09-21T14:10:44.418 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0797783679999999, 2024-09-21T14:10:44.418 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:44.418 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:44.418 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:44.418 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:10:44.418 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:10:44.418 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:10:44.418 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 112230 2024-09-21T14:10:44.418 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.418 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:44.419 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.419 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:44.419 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.239727+0000", 2024-09-21T14:10:44.419 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:44.419 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.419 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.419 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:44.419 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.239727+0000", 2024-09-21T14:10:44.419 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:44.419 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.419 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.419 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:44.419 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.239728+0000", 2024-09-21T14:10:44.419 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8199999999999999e-07 2024-09-21T14:10:44.419 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.420 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.420 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:44.420 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.239763+0000", 2024-09-21T14:10:44.420 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4799999999999999e-05 2024-09-21T14:10:44.420 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.420 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.420 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:44.420 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.239772+0000", 2024-09-21T14:10:44.420 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1099999999999992e-06 2024-09-21T14:10:44.420 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.420 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.420 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:44.420 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.239778+0000", 2024-09-21T14:10:44.420 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0920000000000003e-06 2024-09-21T14:10:44.420 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.421 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.421 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.421 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.239835+0000", 2024-09-21T14:10:44.421 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6758000000000001e-05 2024-09-21T14:10:44.421 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.421 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.421 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:44.421 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.239861+0000", 2024-09-21T14:10:44.421 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.6290000000000001e-05 2024-09-21T14:10:44.421 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.421 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.421 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.421 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.187397+0000", 2024-09-21T14:10:44.421 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.94753606300000004 2024-09-21T14:10:44.421 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.422 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.422 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:44.422 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.187407+0000", 2024-09-21T14:10:44.422 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6600000000000007e-06 2024-09-21T14:10:44.422 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.422 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.422 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.422 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.290801+0000", 2024-09-21T14:10:44.422 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1033946459999999 2024-09-21T14:10:44.422 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.422 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.422 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:10:44.422 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.290936+0000", 2024-09-21T14:10:44.422 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013488899999999999 2024-09-21T14:10:44.423 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.423 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.423 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3,4", 2024-09-21T14:10:44.423 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.291093+0000", 2024-09-21T14:10:44.423 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015686300000000001 2024-09-21T14:10:44.423 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.423 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.423 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:44.423 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.298120+0000", 2024-09-21T14:10:44.423 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0070268209999999999 2024-09-21T14:10:44.423 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.423 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.423 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:10:44.424 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.298149+0000", 2024-09-21T14:10:44.424 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8609999999999999e-05 2024-09-21T14:10:44.424 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.424 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.424 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:44.424 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.298207+0000", 2024-09-21T14:10:44.424 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7859000000000003e-05 2024-09-21T14:10:44.424 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.424 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.424 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:44.424 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.319309+0000", 2024-09-21T14:10:44.424 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.021102584000000001 2024-09-21T14:10:44.424 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.424 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.424 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:10:44.425 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.319379+0000", 2024-09-21T14:10:44.425 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9858999999999996e-05 2024-09-21T14:10:44.425 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.425 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.425 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:44.425 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.319506+0000", 2024-09-21T14:10:44.425 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000126682 2024-09-21T14:10:44.425 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:44.425 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:44.425 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:44.425 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.425 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.425 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:112234 10.a 10:5cc5be6b:::benchmark_data_smithi049_176811_object112233:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-21T14:10:44.425 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:04:20.240298+0000", 2024-09-21T14:10:44.426 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 384.13533156199998, 2024-09-21T14:10:44.426 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.148374665, 2024-09-21T14:10:44.426 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:44.426 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:44.426 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:44.426 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:10:44.426 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:10:44.426 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:10:44.426 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 112234 2024-09-21T14:10:44.426 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.426 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:44.426 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.426 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:44.427 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.240298+0000", 2024-09-21T14:10:44.427 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:44.427 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.427 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.427 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:44.427 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.240298+0000", 2024-09-21T14:10:44.427 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:44.427 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.427 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.427 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:44.427 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.240299+0000", 2024-09-21T14:10:44.427 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4199999999999995e-07 2024-09-21T14:10:44.427 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.427 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.428 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:44.428 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.240385+0000", 2024-09-21T14:10:44.428 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.6550000000000003e-05 2024-09-21T14:10:44.428 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.428 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.428 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:44.428 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.240395+0000", 2024-09-21T14:10:44.428 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.8579999999999995e-06 2024-09-21T14:10:44.428 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.428 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.428 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:44.428 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.240400+0000", 2024-09-21T14:10:44.428 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3040000000000004e-06 2024-09-21T14:10:44.428 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.428 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.429 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.429 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.240444+0000", 2024-09-21T14:10:44.429 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3594000000000003e-05 2024-09-21T14:10:44.429 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.429 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.429 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:44.429 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.240451+0000", 2024-09-21T14:10:44.429 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.0319999999999998e-06 2024-09-21T14:10:44.429 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.429 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.429 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.429 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.188345+0000", 2024-09-21T14:10:44.429 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.94789378400000002 2024-09-21T14:10:44.429 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.430 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.430 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:44.430 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.188351+0000", 2024-09-21T14:10:44.430 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7619999999999998e-06 2024-09-21T14:10:44.430 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.430 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.430 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.430 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.377491+0000", 2024-09-21T14:10:44.430 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.189139642 2024-09-21T14:10:44.430 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.430 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.430 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:10:44.430 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.377612+0000", 2024-09-21T14:10:44.430 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000120528 2024-09-21T14:10:44.431 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.431 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.431 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2,5", 2024-09-21T14:10:44.431 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.377713+0000", 2024-09-21T14:10:44.431 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010192 2024-09-21T14:10:44.431 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.431 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.431 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:44.431 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.388079+0000", 2024-09-21T14:10:44.431 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.010365298 2024-09-21T14:10:44.431 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.431 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.431 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:10:44.431 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.388158+0000", 2024-09-21T14:10:44.432 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8726000000000005e-05 2024-09-21T14:10:44.432 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.432 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.432 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:44.432 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.388482+0000", 2024-09-21T14:10:44.432 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00032423 2024-09-21T14:10:44.432 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.432 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.432 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:44.432 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.388524+0000", 2024-09-21T14:10:44.432 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1782000000000001e-05 2024-09-21T14:10:44.432 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.432 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.432 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:10:44.433 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.388577+0000", 2024-09-21T14:10:44.433 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3965999999999998e-05 2024-09-21T14:10:44.433 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.433 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.433 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:44.433 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.388672+0000", 2024-09-21T14:10:44.433 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.4946999999999997e-05 2024-09-21T14:10:44.433 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:44.433 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:44.433 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:44.433 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.433 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.433 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:112240 10.a 10:59634543:::benchmark_data_smithi049_176811_object112239:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-21T14:10:44.433 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:04:20.245943+0000", 2024-09-21T14:10:44.433 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 384.129686306, 2024-09-21T14:10:44.434 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1573033979999998, 2024-09-21T14:10:44.434 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:44.434 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:44.434 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:44.434 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:10:44.434 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:10:44.434 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:10:44.434 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 112240 2024-09-21T14:10:44.434 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.434 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:44.434 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.434 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:44.434 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.245943+0000", 2024-09-21T14:10:44.434 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:44.435 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.435 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.435 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:44.435 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.245943+0000", 2024-09-21T14:10:44.435 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:44.435 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.435 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.435 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:44.435 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.245944+0000", 2024-09-21T14:10:44.435 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3540000000000001e-06 2024-09-21T14:10:44.435 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.435 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.435 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:44.435 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.246007+0000", 2024-09-21T14:10:44.436 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2155000000000004e-05 2024-09-21T14:10:44.436 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.436 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.436 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:44.436 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.246019+0000", 2024-09-21T14:10:44.436 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2814e-05 2024-09-21T14:10:44.436 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.436 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.436 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:44.436 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.246026+0000", 2024-09-21T14:10:44.436 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1470000000000004e-06 2024-09-21T14:10:44.436 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.436 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.436 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.436 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.246084+0000", 2024-09-21T14:10:44.437 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7731e-05 2024-09-21T14:10:44.437 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.437 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.437 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:44.437 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.246095+0000", 2024-09-21T14:10:44.437 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0964e-05 2024-09-21T14:10:44.437 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.437 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.437 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.437 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.188376+0000", 2024-09-21T14:10:44.437 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.94228104199999996 2024-09-21T14:10:44.437 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.437 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.437 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:44.438 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.188383+0000", 2024-09-21T14:10:44.438 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8979999999999999e-06 2024-09-21T14:10:44.438 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.438 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.438 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.438 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.380685+0000", 2024-09-21T14:10:44.438 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.192302239 2024-09-21T14:10:44.438 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.438 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.438 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:10:44.438 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.381343+0000", 2024-09-21T14:10:44.438 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00065746299999999995 2024-09-21T14:10:44.438 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.440 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.440 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2,5", 2024-09-21T14:10:44.440 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.381520+0000", 2024-09-21T14:10:44.440 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00017690300000000001 2024-09-21T14:10:44.440 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.440 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.441 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:10:44.441 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.388164+0000", 2024-09-21T14:10:44.441 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0066440359999999999 2024-09-21T14:10:44.441 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.441 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.441 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:44.441 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.388424+0000", 2024-09-21T14:10:44.441 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00026048999999999999 2024-09-21T14:10:44.441 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.441 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.441 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:44.441 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.388788+0000", 2024-09-21T14:10:44.441 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00036423599999999999 2024-09-21T14:10:44.442 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.442 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.442 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:44.442 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.403084+0000", 2024-09-21T14:10:44.442 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.014295491 2024-09-21T14:10:44.442 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.442 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.442 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:10:44.442 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.403141+0000", 2024-09-21T14:10:44.442 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7290999999999997e-05 2024-09-21T14:10:44.442 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.442 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.442 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:44.443 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.403246+0000", 2024-09-21T14:10:44.443 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010514399999999999 2024-09-21T14:10:44.443 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:44.443 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:44.443 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:44.443 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.443 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.443 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:112251 10.a 10:59ed2c1b:::benchmark_data_smithi049_176811_object112250:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-21T14:10:44.443 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:04:20.255435+0000", 2024-09-21T14:10:44.443 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 384.120194223, 2024-09-21T14:10:44.443 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1536580829999998, 2024-09-21T14:10:44.443 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:44.443 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:44.443 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:44.443 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:10:44.444 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:10:44.444 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:10:44.444 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 112251 2024-09-21T14:10:44.444 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.444 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:44.444 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.444 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:44.444 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.255435+0000", 2024-09-21T14:10:44.444 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:44.444 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.444 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.444 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:44.444 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.255435+0000", 2024-09-21T14:10:44.445 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:44.445 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.445 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.445 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:44.445 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.255436+0000", 2024-09-21T14:10:44.445 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.184e-06 2024-09-21T14:10:44.445 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.445 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.445 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:44.445 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.255522+0000", 2024-09-21T14:10:44.445 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5778000000000004e-05 2024-09-21T14:10:44.445 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.445 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.446 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:44.446 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.255532+0000", 2024-09-21T14:10:44.446 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.8199999999999992e-06 2024-09-21T14:10:44.446 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.446 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.446 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:44.446 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.255538+0000", 2024-09-21T14:10:44.446 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8710000000000002e-06 2024-09-21T14:10:44.446 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.446 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.446 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.446 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.255589+0000", 2024-09-21T14:10:44.446 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1644000000000002e-05 2024-09-21T14:10:44.446 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.447 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.447 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:44.447 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.255596+0000", 2024-09-21T14:10:44.447 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6499999999999999e-06 2024-09-21T14:10:44.447 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.447 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.447 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.447 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.188406+0000", 2024-09-21T14:10:44.447 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.932810425 2024-09-21T14:10:44.447 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.447 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.447 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:44.447 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.188413+0000", 2024-09-21T14:10:44.447 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9859999999999997e-06 2024-09-21T14:10:44.447 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.448 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.448 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.448 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.384443+0000", 2024-09-21T14:10:44.448 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.196029257 2024-09-21T14:10:44.448 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.448 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.448 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:10:44.448 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.384560+0000", 2024-09-21T14:10:44.448 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000117766 2024-09-21T14:10:44.448 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.448 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.448 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2,5", 2024-09-21T14:10:44.448 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.384711+0000", 2024-09-21T14:10:44.448 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015052899999999999 2024-09-21T14:10:44.449 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.449 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.449 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:10:44.449 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.388438+0000", 2024-09-21T14:10:44.449 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00372669 2024-09-21T14:10:44.449 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.449 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.449 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:44.449 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.388729+0000", 2024-09-21T14:10:44.449 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00029102899999999999 2024-09-21T14:10:44.449 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.449 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.449 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:44.449 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.397078+0000", 2024-09-21T14:10:44.450 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0083490219999999993 2024-09-21T14:10:44.450 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.450 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.450 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:44.450 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.408920+0000", 2024-09-21T14:10:44.450 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.011842065000000001 2024-09-21T14:10:44.450 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.450 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.450 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:10:44.450 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.408981+0000", 2024-09-21T14:10:44.450 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1569999999999995e-05 2024-09-21T14:10:44.450 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.450 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.451 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:44.451 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.409093+0000", 2024-09-21T14:10:44.451 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000111797 2024-09-21T14:10:44.451 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:44.451 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:44.451 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:44.451 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.451 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.451 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:112253 10.d 10:be7afa8e:::benchmark_data_smithi049_176811_object112252:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-21T14:10:44.451 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:04:20.263232+0000", 2024-09-21T14:10:44.451 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 384.11239744800002, 2024-09-21T14:10:44.451 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0592742639999999, 2024-09-21T14:10:44.451 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:44.451 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:44.452 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:44.452 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:10:44.452 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:10:44.452 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:10:44.452 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 112253 2024-09-21T14:10:44.452 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.452 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:44.452 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.452 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:44.452 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.263232+0000", 2024-09-21T14:10:44.452 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:44.452 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.452 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.452 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:44.453 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.263232+0000", 2024-09-21T14:10:44.453 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:44.453 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.453 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.453 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:44.453 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.263233+0000", 2024-09-21T14:10:44.453 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.488e-06 2024-09-21T14:10:44.453 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.453 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.453 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:44.453 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.263305+0000", 2024-09-21T14:10:44.453 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1552999999999997e-05 2024-09-21T14:10:44.453 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.453 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.454 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:44.454 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.263319+0000", 2024-09-21T14:10:44.454 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4233000000000001e-05 2024-09-21T14:10:44.454 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.454 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.454 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:44.454 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.263325+0000", 2024-09-21T14:10:44.454 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9950000000000002e-06 2024-09-21T14:10:44.454 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.454 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.454 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.454 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.263389+0000", 2024-09-21T14:10:44.454 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3567999999999999e-05 2024-09-21T14:10:44.454 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.455 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.455 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:44.455 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.263400+0000", 2024-09-21T14:10:44.455 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1206000000000001e-05 2024-09-21T14:10:44.455 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.455 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.455 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.455 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.187432+0000", 2024-09-21T14:10:44.455 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.92403234099999998 2024-09-21T14:10:44.455 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.455 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.455 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:44.455 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.187440+0000", 2024-09-21T14:10:44.455 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0260000000000007e-06 2024-09-21T14:10:44.455 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.456 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.456 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.456 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.295005+0000", 2024-09-21T14:10:44.456 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1075643630000001 2024-09-21T14:10:44.456 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.456 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.456 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:10:44.456 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.295169+0000", 2024-09-21T14:10:44.456 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000163859 2024-09-21T14:10:44.456 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.456 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.456 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,3,4", 2024-09-21T14:10:44.456 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.295394+0000", 2024-09-21T14:10:44.456 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000225353 2024-09-21T14:10:44.456 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.457 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.457 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:10:44.457 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.298164+0000", 2024-09-21T14:10:44.457 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0027705170000000001 2024-09-21T14:10:44.457 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.457 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.457 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:44.457 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.298919+0000", 2024-09-21T14:10:44.457 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00075427899999999995 2024-09-21T14:10:44.457 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.457 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.457 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:44.457 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.316063+0000", 2024-09-21T14:10:44.457 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.017144309 2024-09-21T14:10:44.457 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.458 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.458 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:44.458 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.322394+0000", 2024-09-21T14:10:44.458 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0063310720000000001 2024-09-21T14:10:44.458 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.458 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.458 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:10:44.458 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.322436+0000", 2024-09-21T14:10:44.458 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1897999999999998e-05 2024-09-21T14:10:44.458 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.458 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.458 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:44.458 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.322506+0000", 2024-09-21T14:10:44.458 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.0203999999999994e-05 2024-09-21T14:10:44.458 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:44.459 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:44.459 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:44.459 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.459 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.459 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:112256 10.a 10:5949a1c7:::benchmark_data_smithi049_176811_object112255:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-21T14:10:44.459 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:04:20.267531+0000", 2024-09-21T14:10:44.459 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 384.10809878800001, 2024-09-21T14:10:44.459 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.141982547, 2024-09-21T14:10:44.459 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:44.459 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:44.459 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:44.459 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:10:44.459 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:10:44.460 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:10:44.460 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 112256 2024-09-21T14:10:44.460 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.460 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:44.460 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.460 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:44.460 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.267531+0000", 2024-09-21T14:10:44.460 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:44.460 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.460 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.460 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:44.460 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.267531+0000", 2024-09-21T14:10:44.460 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:44.460 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.461 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.461 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:44.461 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.267531+0000", 2024-09-21T14:10:44.461 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1999999999999998e-07 2024-09-21T14:10:44.461 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.461 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.461 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:44.461 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.267570+0000", 2024-09-21T14:10:44.461 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8336e-05 2024-09-21T14:10:44.461 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.461 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.461 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:44.461 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.267579+0000", 2024-09-21T14:10:44.461 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.9930000000000001e-06 2024-09-21T14:10:44.461 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.461 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.462 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:44.462 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.267583+0000", 2024-09-21T14:10:44.462 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3830000000000002e-06 2024-09-21T14:10:44.462 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.462 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.462 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.462 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.267632+0000", 2024-09-21T14:10:44.462 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8822000000000001e-05 2024-09-21T14:10:44.462 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.462 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.462 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:44.462 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.267640+0000", 2024-09-21T14:10:44.462 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2300000000000008e-06 2024-09-21T14:10:44.462 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.462 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.463 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.463 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.188439+0000", 2024-09-21T14:10:44.463 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.92079873300000004 2024-09-21T14:10:44.463 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.463 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.463 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:44.463 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.188446+0000", 2024-09-21T14:10:44.463 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.255e-06 2024-09-21T14:10:44.463 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.463 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.463 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.463 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.388852+0000", 2024-09-21T14:10:44.463 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.200405897 2024-09-21T14:10:44.463 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.464 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.464 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:10:44.464 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.389001+0000", 2024-09-21T14:10:44.464 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000148641 2024-09-21T14:10:44.464 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.464 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.464 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2,5", 2024-09-21T14:10:44.464 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.389171+0000", 2024-09-21T14:10:44.464 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00017001399999999999 2024-09-21T14:10:44.464 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.464 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.464 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:44.464 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.396980+0000", 2024-09-21T14:10:44.464 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0078089149999999996 2024-09-21T14:10:44.464 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.464 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.465 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:44.465 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.397128+0000", 2024-09-21T14:10:44.465 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014819799999999999 2024-09-21T14:10:44.465 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.465 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.465 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:10:44.465 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.399986+0000", 2024-09-21T14:10:44.465 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0028577210000000001 2024-09-21T14:10:44.465 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.465 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.465 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:44.465 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.409327+0000", 2024-09-21T14:10:44.465 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0093417110000000008 2024-09-21T14:10:44.465 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.466 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.466 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:10:44.466 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.409390+0000", 2024-09-21T14:10:44.466 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2636000000000001e-05 2024-09-21T14:10:44.466 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.466 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.466 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:44.466 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.409513+0000", 2024-09-21T14:10:44.466 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012314199999999999 2024-09-21T14:10:44.466 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:44.466 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:44.466 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:44.466 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.466 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.467 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.16068.0:112262 10.a 10:5144907d:::benchmark_data_smithi049_176811_object112261:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-09-21T14:10:44.467 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:04:20.285420+0000", 2024-09-21T14:10:44.467 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 384.09020902899999, 2024-09-21T14:10:44.467 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1243977709999999, 2024-09-21T14:10:44.467 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:44.467 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:44.467 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:44.467 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:10:44.467 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.16068", 2024-09-21T14:10:44.467 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:10:44.467 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 112262 2024-09-21T14:10:44.467 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.467 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:44.467 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.468 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:44.468 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.285420+0000", 2024-09-21T14:10:44.468 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:44.468 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.468 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.468 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:44.468 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.285420+0000", 2024-09-21T14:10:44.468 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:44.468 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.468 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.468 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:44.468 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.285422+0000", 2024-09-21T14:10:44.468 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4449999999999999e-06 2024-09-21T14:10:44.468 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.468 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.469 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:44.469 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.285450+0000", 2024-09-21T14:10:44.469 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7985999999999999e-05 2024-09-21T14:10:44.469 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.469 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.469 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:44.469 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.285462+0000", 2024-09-21T14:10:44.469 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.185e-05 2024-09-21T14:10:44.469 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.469 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.469 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:44.469 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.285468+0000", 2024-09-21T14:10:44.469 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3570000000000003e-06 2024-09-21T14:10:44.470 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.470 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.470 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.470 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.285519+0000", 2024-09-21T14:10:44.470 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0912999999999999e-05 2024-09-21T14:10:44.470 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.470 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.470 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:44.470 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:20.285526+0000", 2024-09-21T14:10:44.470 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1069999999999999e-06 2024-09-21T14:10:44.470 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.470 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.470 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.471 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.188522+0000", 2024-09-21T14:10:44.471 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.90299566200000003 2024-09-21T14:10:44.471 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.471 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.471 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:44.471 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:21.188533+0000", 2024-09-21T14:10:44.471 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0903999999999999e-05 2024-09-21T14:10:44.471 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.471 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.471 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.471 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.391763+0000", 2024-09-21T14:10:44.471 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2032303799999999 2024-09-21T14:10:44.471 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.471 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.472 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:10:44.472 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.391895+0000", 2024-09-21T14:10:44.472 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013225300000000001 2024-09-21T14:10:44.472 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.472 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.472 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2,5", 2024-09-21T14:10:44.472 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.392058+0000", 2024-09-21T14:10:44.472 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000162619 2024-09-21T14:10:44.472 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.472 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.472 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:44.472 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.397025+0000", 2024-09-21T14:10:44.472 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0049667569999999996 2024-09-21T14:10:44.472 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.473 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.473 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:10:44.473 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.402990+0000", 2024-09-21T14:10:44.473 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0059658200000000002 2024-09-21T14:10:44.473 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.473 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.473 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:44.473 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.403310+0000", 2024-09-21T14:10:44.473 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00031964599999999998 2024-09-21T14:10:44.473 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.473 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.473 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:44.473 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.409700+0000", 2024-09-21T14:10:44.474 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0063900240000000002 2024-09-21T14:10:44.474 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.474 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.474 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:10:44.474 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.409743+0000", 2024-09-21T14:10:44.474 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2670000000000003e-05 2024-09-21T14:10:44.474 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.474 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.474 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:44.474 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:04:22.409818+0000", 2024-09-21T14:10:44.474 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5377999999999996e-05 2024-09-21T14:10:44.474 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:44.474 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:44.475 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:44.475 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.475 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.475 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26396.0:114618 11.9 11:93cd8964:::benchmark_data_smithi049_206132_object114617:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e496)", 2024-09-21T14:10:44.475 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:08:13.395853+0000", 2024-09-21T14:10:44.475 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 150.97977674099999, 2024-09-21T14:10:44.475 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.082294047, 2024-09-21T14:10:44.475 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:44.475 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:44.475 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-21T14:10:44.475 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:10:44.475 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26396", 2024-09-21T14:10:44.475 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:44.475 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 114618 2024-09-21T14:10:44.476 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.476 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:44.476 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.476 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:44.476 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.395853+0000", 2024-09-21T14:10:44.476 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:44.476 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.476 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.476 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:44.476 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.395853+0000", 2024-09-21T14:10:44.476 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:44.476 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.476 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.476 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:44.477 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.395854+0000", 2024-09-21T14:10:44.477 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5740000000000001e-06 2024-09-21T14:10:44.477 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.477 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.477 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:44.477 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.395890+0000", 2024-09-21T14:10:44.477 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5565999999999998e-05 2024-09-21T14:10:44.477 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.477 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.477 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:44.477 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.395913+0000", 2024-09-21T14:10:44.477 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3283000000000001e-05 2024-09-21T14:10:44.477 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.477 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.478 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:44.478 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.395921+0000", 2024-09-21T14:10:44.478 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3080000000000008e-06 2024-09-21T14:10:44.478 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.478 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.478 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.478 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.396057+0000", 2024-09-21T14:10:44.478 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013601600000000001 2024-09-21T14:10:44.478 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.478 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.478 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:44.478 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.396066+0000", 2024-09-21T14:10:44.478 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1260000000000004e-06 2024-09-21T14:10:44.478 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.479 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.479 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.479 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:14.382158+0000", 2024-09-21T14:10:44.479 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.98609193299999998 2024-09-21T14:10:44.479 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.479 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.479 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:44.479 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:14.382166+0000", 2024-09-21T14:10:44.479 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4420000000000001e-06 2024-09-21T14:10:44.479 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.479 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.479 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.479 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.477272+0000", 2024-09-21T14:10:44.480 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.095106286 2024-09-21T14:10:44.480 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.480 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.480 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:44.480 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.478147+0000", 2024-09-21T14:10:44.480 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00087451300000000005 2024-09-21T14:10:44.480 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:44.480 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:44.480 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:44.480 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.480 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.480 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26396.0:114621 11.9 11:908cded0:::benchmark_data_smithi049_206132_object114620:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e496)", 2024-09-21T14:10:44.480 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:08:13.395943+0000", 2024-09-21T14:10:44.480 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 150.979686531, 2024-09-21T14:10:44.481 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0826702849999998, 2024-09-21T14:10:44.481 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:44.481 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:44.481 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-21T14:10:44.481 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:10:44.481 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26396", 2024-09-21T14:10:44.481 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:44.481 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 114621 2024-09-21T14:10:44.481 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.481 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:44.481 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.481 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:44.481 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.395943+0000", 2024-09-21T14:10:44.481 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:44.482 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.482 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.482 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:44.482 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.395943+0000", 2024-09-21T14:10:44.482 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:44.482 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.482 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.482 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:44.482 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.395943+0000", 2024-09-21T14:10:44.482 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1900000000000002e-07 2024-09-21T14:10:44.482 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.482 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.482 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:44.482 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.395971+0000", 2024-09-21T14:10:44.483 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7152e-05 2024-09-21T14:10:44.483 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.483 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.483 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:44.483 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.395978+0000", 2024-09-21T14:10:44.483 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1269999999999997e-06 2024-09-21T14:10:44.483 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.483 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.483 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:44.483 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.395983+0000", 2024-09-21T14:10:44.483 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0640000000000001e-06 2024-09-21T14:10:44.483 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.483 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.484 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.484 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.396087+0000", 2024-09-21T14:10:44.484 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000104096 2024-09-21T14:10:44.484 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.484 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.484 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:44.484 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.396093+0000", 2024-09-21T14:10:44.484 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9209999999999997e-06 2024-09-21T14:10:44.484 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.484 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.484 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.484 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:14.382224+0000", 2024-09-21T14:10:44.484 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.98613127199999995 2024-09-21T14:10:44.484 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.484 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.485 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:44.485 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:14.382231+0000", 2024-09-21T14:10:44.485 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.951e-06 2024-09-21T14:10:44.485 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.485 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.485 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.485 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.478549+0000", 2024-09-21T14:10:44.485 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.096318106 2024-09-21T14:10:44.485 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.485 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.485 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:44.485 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.478613+0000", 2024-09-21T14:10:44.485 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3977000000000001e-05 2024-09-21T14:10:44.485 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:44.486 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:44.486 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:44.486 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.486 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.486 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26396.0:114641 11.1 11:877bfe6e:::benchmark_data_smithi049_206132_object114640:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e496)", 2024-09-21T14:10:44.486 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:08:13.423568+0000", 2024-09-21T14:10:44.486 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 150.952061257, 2024-09-21T14:10:44.486 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0805294160000001, 2024-09-21T14:10:44.486 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:44.486 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:44.486 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:44.486 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:10:44.486 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26396", 2024-09-21T14:10:44.486 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:44.487 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 114641 2024-09-21T14:10:44.487 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.487 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:44.487 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.487 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:44.487 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.423568+0000", 2024-09-21T14:10:44.487 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:44.487 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.487 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.487 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:44.487 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.423568+0000", 2024-09-21T14:10:44.487 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:44.487 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.487 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.488 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:44.488 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.423569+0000", 2024-09-21T14:10:44.488 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.142e-06 2024-09-21T14:10:44.488 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.488 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.488 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:44.488 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.423598+0000", 2024-09-21T14:10:44.488 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9127000000000001e-05 2024-09-21T14:10:44.488 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.488 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.488 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:44.488 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.423607+0000", 2024-09-21T14:10:44.488 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8610000000000004e-06 2024-09-21T14:10:44.488 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.488 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.489 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:44.489 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.423613+0000", 2024-09-21T14:10:44.489 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1850000000000001e-06 2024-09-21T14:10:44.489 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.489 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.489 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.489 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.423811+0000", 2024-09-21T14:10:44.489 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00019768300000000001 2024-09-21T14:10:44.489 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.489 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.489 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:44.489 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.423820+0000", 2024-09-21T14:10:44.489 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8689999999999993e-06 2024-09-21T14:10:44.489 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.489 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.490 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.490 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:14.382121+0000", 2024-09-21T14:10:44.490 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.95830109399999996 2024-09-21T14:10:44.490 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.490 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.490 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:44.490 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:14.382130+0000", 2024-09-21T14:10:44.490 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5939999999999994e-06 2024-09-21T14:10:44.490 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.490 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.490 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.490 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.485606+0000", 2024-09-21T14:10:44.490 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.103476015 2024-09-21T14:10:44.491 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.491 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.491 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:10:44.491 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.485724+0000", 2024-09-21T14:10:44.491 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000117935 2024-09-21T14:10:44.491 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.491 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.491 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2,3", 2024-09-21T14:10:44.491 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.486171+0000", 2024-09-21T14:10:44.491 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00044764300000000001 2024-09-21T14:10:44.491 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.491 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.491 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:10:44.491 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.503749+0000", 2024-09-21T14:10:44.492 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.017577406 2024-09-21T14:10:44.492 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.492 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.492 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:44.492 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.503794+0000", 2024-09-21T14:10:44.492 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5247999999999998e-05 2024-09-21T14:10:44.492 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.492 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.492 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:44.492 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.503863+0000", 2024-09-21T14:10:44.492 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9047000000000005e-05 2024-09-21T14:10:44.492 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.492 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.492 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:44.493 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.503900+0000", 2024-09-21T14:10:44.493 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6609000000000001e-05 2024-09-21T14:10:44.493 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.493 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.493 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:10:44.493 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.503962+0000", 2024-09-21T14:10:44.493 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2712999999999994e-05 2024-09-21T14:10:44.493 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.493 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.493 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:44.493 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.504097+0000", 2024-09-21T14:10:44.493 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013524500000000001 2024-09-21T14:10:44.493 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:44.493 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:44.493 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:44.494 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.494 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.494 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26396.0:114655 11.1 11:81711ddc:::benchmark_data_smithi049_206132_object114654:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e496)", 2024-09-21T14:10:44.494 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:08:13.438595+0000", 2024-09-21T14:10:44.494 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 150.93703461300001, 2024-09-21T14:10:44.494 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0658325770000001, 2024-09-21T14:10:44.494 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:44.494 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:44.494 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:44.494 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:10:44.494 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26396", 2024-09-21T14:10:44.494 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:44.494 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 114655 2024-09-21T14:10:44.494 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.495 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:44.495 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.495 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:44.495 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.438595+0000", 2024-09-21T14:10:44.495 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:44.495 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.495 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.495 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:44.495 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.438595+0000", 2024-09-21T14:10:44.495 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:44.495 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.495 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.495 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:44.495 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.438597+0000", 2024-09-21T14:10:44.495 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8279999999999999e-06 2024-09-21T14:10:44.496 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.496 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.496 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:44.496 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.438653+0000", 2024-09-21T14:10:44.496 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6721999999999997e-05 2024-09-21T14:10:44.496 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.496 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.496 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:44.496 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.438667+0000", 2024-09-21T14:10:44.496 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3458000000000001e-05 2024-09-21T14:10:44.496 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.496 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.496 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:44.496 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.438674+0000", 2024-09-21T14:10:44.496 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.661e-06 2024-09-21T14:10:44.497 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.497 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.497 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.497 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.438739+0000", 2024-09-21T14:10:44.497 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4886000000000002e-05 2024-09-21T14:10:44.497 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.497 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.497 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:44.497 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.438749+0000", 2024-09-21T14:10:44.497 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.285e-06 2024-09-21T14:10:44.497 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.497 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.497 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.497 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:14.382193+0000", 2024-09-21T14:10:44.497 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.94344481099999999 2024-09-21T14:10:44.498 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.498 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.498 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:44.498 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:14.382201+0000", 2024-09-21T14:10:44.498 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2729999999999997e-06 2024-09-21T14:10:44.498 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.498 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.498 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.498 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.490327+0000", 2024-09-21T14:10:44.498 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1081260070000001 2024-09-21T14:10:44.498 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.498 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.498 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:10:44.498 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.490433+0000", 2024-09-21T14:10:44.499 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000106208 2024-09-21T14:10:44.499 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.499 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.499 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2,3", 2024-09-21T14:10:44.499 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.490538+0000", 2024-09-21T14:10:44.499 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000105078 2024-09-21T14:10:44.499 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.499 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.499 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:10:44.499 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.503754+0000", 2024-09-21T14:10:44.499 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.013216462999999999 2024-09-21T14:10:44.499 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.499 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.499 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:44.499 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.504143+0000", 2024-09-21T14:10:44.499 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00038873100000000002 2024-09-21T14:10:44.499 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.500 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.500 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:44.500 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.504183+0000", 2024-09-21T14:10:44.500 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9845999999999998e-05 2024-09-21T14:10:44.500 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.500 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.500 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:44.500 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.504224+0000", 2024-09-21T14:10:44.500 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0930999999999998e-05 2024-09-21T14:10:44.500 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.500 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.500 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:10:44.500 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.504296+0000", 2024-09-21T14:10:44.500 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2177999999999999e-05 2024-09-21T14:10:44.501 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.501 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.501 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:44.501 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.504427+0000", 2024-09-21T14:10:44.501 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000131211 2024-09-21T14:10:44.501 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:44.501 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:44.501 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:44.501 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.501 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.501 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26396.0:114658 11.8 11:14301b05:::benchmark_data_smithi049_206132_object114657:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e496)", 2024-09-21T14:10:44.501 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:08:13.439803+0000", 2024-09-21T14:10:44.501 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 150.93582636100001, 2024-09-21T14:10:44.502 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0403675209999999, 2024-09-21T14:10:44.502 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:44.502 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:44.502 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:44.502 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:10:44.502 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26396", 2024-09-21T14:10:44.502 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:44.502 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 114658 2024-09-21T14:10:44.502 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.502 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:44.502 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.502 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:44.502 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.439803+0000", 2024-09-21T14:10:44.502 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:44.503 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.503 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.503 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:44.503 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.439803+0000", 2024-09-21T14:10:44.503 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:44.503 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.503 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.503 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:44.503 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.439804+0000", 2024-09-21T14:10:44.503 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0440000000000001e-06 2024-09-21T14:10:44.503 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.503 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.503 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:44.503 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.439976+0000", 2024-09-21T14:10:44.504 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000172246 2024-09-21T14:10:44.504 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.504 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.504 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:44.504 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.439995+0000", 2024-09-21T14:10:44.504 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8457999999999998e-05 2024-09-21T14:10:44.504 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.504 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.504 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:44.504 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.440003+0000", 2024-09-21T14:10:44.504 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3280000000000006e-06 2024-09-21T14:10:44.504 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.504 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.504 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.505 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.440069+0000", 2024-09-21T14:10:44.505 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5916999999999999e-05 2024-09-21T14:10:44.505 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.505 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.505 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:44.505 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.440079+0000", 2024-09-21T14:10:44.505 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6569999999999992e-06 2024-09-21T14:10:44.505 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.505 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.505 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.505 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:14.381910+0000", 2024-09-21T14:10:44.505 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.94183137100000003 2024-09-21T14:10:44.505 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.505 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.506 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:44.506 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:14.381921+0000", 2024-09-21T14:10:44.506 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1504e-05 2024-09-21T14:10:44.506 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.506 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.506 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.506 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.470599+0000", 2024-09-21T14:10:44.506 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0886771740000001 2024-09-21T14:10:44.506 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.506 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.506 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:10:44.506 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.472108+0000", 2024-09-21T14:10:44.506 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001509327 2024-09-21T14:10:44.507 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.507 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.507 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,3,5", 2024-09-21T14:10:44.507 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.472305+0000", 2024-09-21T14:10:44.507 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0001973 2024-09-21T14:10:44.507 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.507 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.507 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:10:44.507 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.478156+0000", 2024-09-21T14:10:44.507 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0058505679999999996 2024-09-21T14:10:44.507 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.507 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.507 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:44.507 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.478430+0000", 2024-09-21T14:10:44.508 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000274438 2024-09-21T14:10:44.508 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.508 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.508 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:44.508 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.478926+0000", 2024-09-21T14:10:44.508 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00049566400000000004 2024-09-21T14:10:44.508 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.508 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.508 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:44.508 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.479986+0000", 2024-09-21T14:10:44.508 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0010602039999999999 2024-09-21T14:10:44.508 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.508 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.508 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:10:44.509 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.480049+0000", 2024-09-21T14:10:44.509 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2459999999999995e-05 2024-09-21T14:10:44.509 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.509 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.509 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:44.509 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.480170+0000", 2024-09-21T14:10:44.509 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000121861 2024-09-21T14:10:44.509 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:44.509 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:44.509 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:44.509 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.509 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.509 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26396.0:114665 11.1 11:83a286ac:::benchmark_data_smithi049_206132_object114664:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e496)", 2024-09-21T14:10:44.509 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:08:13.447546+0000", 2024-09-21T14:10:44.509 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 150.92808306399999, 2024-09-21T14:10:44.510 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.057220273, 2024-09-21T14:10:44.510 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:44.510 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:44.510 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:44.510 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:10:44.510 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26396", 2024-09-21T14:10:44.510 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:44.510 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 114665 2024-09-21T14:10:44.510 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.510 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:44.510 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.510 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:44.510 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.447546+0000", 2024-09-21T14:10:44.510 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:44.511 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.511 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.511 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:44.511 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.447546+0000", 2024-09-21T14:10:44.511 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:44.511 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.511 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.511 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:44.511 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.447547+0000", 2024-09-21T14:10:44.511 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.057e-06 2024-09-21T14:10:44.511 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.511 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.511 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:44.511 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.447573+0000", 2024-09-21T14:10:44.511 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5749999999999999e-05 2024-09-21T14:10:44.512 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.512 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.512 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:44.512 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.447584+0000", 2024-09-21T14:10:44.512 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0672e-05 2024-09-21T14:10:44.512 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.512 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.512 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:44.512 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.447591+0000", 2024-09-21T14:10:44.512 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7800000000000003e-06 2024-09-21T14:10:44.512 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.512 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.512 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.512 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.447659+0000", 2024-09-21T14:10:44.513 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8356999999999996e-05 2024-09-21T14:10:44.513 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.513 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.513 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:44.513 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.447668+0000", 2024-09-21T14:10:44.513 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1670000000000005e-06 2024-09-21T14:10:44.513 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.513 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.513 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.513 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:14.382257+0000", 2024-09-21T14:10:44.513 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.93458886500000005 2024-09-21T14:10:44.513 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.513 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.513 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:44.514 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:14.382264+0000", 2024-09-21T14:10:44.514 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3209999999999999e-06 2024-09-21T14:10:44.514 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.514 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.514 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.514 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.492016+0000", 2024-09-21T14:10:44.514 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1097516160000001 2024-09-21T14:10:44.514 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.514 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.514 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:10:44.514 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.492143+0000", 2024-09-21T14:10:44.514 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012762400000000001 2024-09-21T14:10:44.514 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.514 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.515 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2,3", 2024-09-21T14:10:44.515 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.492248+0000", 2024-09-21T14:10:44.515 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010458699999999999 2024-09-21T14:10:44.515 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.515 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.515 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:10:44.515 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.503808+0000", 2024-09-21T14:10:44.515 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.011559764 2024-09-21T14:10:44.515 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.515 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.515 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:44.515 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.504471+0000", 2024-09-21T14:10:44.515 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00066317899999999996 2024-09-21T14:10:44.515 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.516 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.516 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:44.516 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.504522+0000", 2024-09-21T14:10:44.516 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0837999999999997e-05 2024-09-21T14:10:44.516 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.516 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.516 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:44.516 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.504580+0000", 2024-09-21T14:10:44.516 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8279999999999998e-05 2024-09-21T14:10:44.516 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.516 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.516 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:10:44.516 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.504654+0000", 2024-09-21T14:10:44.516 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4173999999999998e-05 2024-09-21T14:10:44.517 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.517 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.517 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:44.517 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.504767+0000", 2024-09-21T14:10:44.517 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000112242 2024-09-21T14:10:44.517 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:44.517 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:44.517 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:44.517 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.517 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.517 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26396.0:114667 11.9 11:9a028036:::benchmark_data_smithi049_206132_object114666:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e496)", 2024-09-21T14:10:44.517 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:08:13.448989+0000", 2024-09-21T14:10:44.517 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 150.926640787, 2024-09-21T14:10:44.517 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0554569699999998, 2024-09-21T14:10:44.518 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:44.518 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:44.518 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:44.518 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:10:44.518 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26396", 2024-09-21T14:10:44.518 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:44.518 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 114667 2024-09-21T14:10:44.518 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.518 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:44.518 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.518 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:44.518 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.448989+0000", 2024-09-21T14:10:44.518 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:44.518 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.519 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.519 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:44.519 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.448989+0000", 2024-09-21T14:10:44.519 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:44.519 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.519 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.519 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:44.519 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.448990+0000", 2024-09-21T14:10:44.519 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.048e-06 2024-09-21T14:10:44.519 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.519 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.519 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:44.519 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.449014+0000", 2024-09-21T14:10:44.519 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4239e-05 2024-09-21T14:10:44.519 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.520 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.520 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:44.520 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.449023+0000", 2024-09-21T14:10:44.520 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0019999999999995e-06 2024-09-21T14:10:44.520 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.520 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.520 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:44.520 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.449029+0000", 2024-09-21T14:10:44.520 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1539999999999999e-06 2024-09-21T14:10:44.520 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.520 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.520 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.520 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.449083+0000", 2024-09-21T14:10:44.520 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3826000000000003e-05 2024-09-21T14:10:44.521 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.521 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.521 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:44.521 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.449093+0000", 2024-09-21T14:10:44.521 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7289999999999995e-06 2024-09-21T14:10:44.521 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.521 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.521 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.521 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:14.382287+0000", 2024-09-21T14:10:44.521 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.93319489799999999 2024-09-21T14:10:44.521 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.521 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.521 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:44.521 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:14.382294+0000", 2024-09-21T14:10:44.522 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9720000000000003e-06 2024-09-21T14:10:44.522 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.522 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.522 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.522 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.478634+0000", 2024-09-21T14:10:44.522 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.096339365 2024-09-21T14:10:44.522 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.522 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.522 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:10:44.522 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.478925+0000", 2024-09-21T14:10:44.522 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00029151600000000002 2024-09-21T14:10:44.522 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.522 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.522 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3,7", 2024-09-21T14:10:44.523 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.479198+0000", 2024-09-21T14:10:44.523 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00027239499999999998 2024-09-21T14:10:44.523 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.523 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.523 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:44.523 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.485480+0000", 2024-09-21T14:10:44.523 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0062823159999999996 2024-09-21T14:10:44.523 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.523 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.523 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:10:44.523 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.490308+0000", 2024-09-21T14:10:44.523 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0048277040000000004 2024-09-21T14:10:44.523 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.524 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.524 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:44.524 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.491855+0000", 2024-09-21T14:10:44.524 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00154737 2024-09-21T14:10:44.524 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.524 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.524 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:44.524 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.504276+0000", 2024-09-21T14:10:44.524 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.012421129 2024-09-21T14:10:44.524 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.524 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.524 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:10:44.524 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.504340+0000", 2024-09-21T14:10:44.524 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3678999999999998e-05 2024-09-21T14:10:44.525 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.525 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.525 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:44.525 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.504446+0000", 2024-09-21T14:10:44.525 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000105628 2024-09-21T14:10:44.525 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:44.525 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:44.525 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:44.525 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.525 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.525 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26396.0:114671 11.9 11:95b30c50:::benchmark_data_smithi049_206132_object114670:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e496)", 2024-09-21T14:10:44.525 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:08:13.451969+0000", 2024-09-21T14:10:44.525 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 150.92366082999999, 2024-09-21T14:10:44.525 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.077318972, 2024-09-21T14:10:44.526 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:44.526 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:44.526 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:44.526 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:10:44.526 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26396", 2024-09-21T14:10:44.526 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:44.526 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 114671 2024-09-21T14:10:44.526 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.526 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:44.526 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.526 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:44.526 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.451969+0000", 2024-09-21T14:10:44.526 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:44.526 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.527 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.527 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:44.527 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.451969+0000", 2024-09-21T14:10:44.527 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:44.527 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.527 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.527 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:44.527 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.451969+0000", 2024-09-21T14:10:44.527 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7999999999999993e-07 2024-09-21T14:10:44.527 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.527 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.527 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:44.527 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.452029+0000", 2024-09-21T14:10:44.527 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9088000000000001e-05 2024-09-21T14:10:44.527 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.528 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.528 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:44.528 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.452039+0000", 2024-09-21T14:10:44.528 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.012e-05 2024-09-21T14:10:44.528 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.528 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.528 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:44.528 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.452046+0000", 2024-09-21T14:10:44.528 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.0870000000000001e-06 2024-09-21T14:10:44.528 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.528 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.528 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.528 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.452098+0000", 2024-09-21T14:10:44.528 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2012999999999998e-05 2024-09-21T14:10:44.529 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.529 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.529 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:44.529 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.452105+0000", 2024-09-21T14:10:44.529 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9929999999999998e-06 2024-09-21T14:10:44.529 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.529 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.529 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.529 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:14.382365+0000", 2024-09-21T14:10:44.529 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.93026017699999997 2024-09-21T14:10:44.529 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.529 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.529 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:44.529 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:14.382373+0000", 2024-09-21T14:10:44.529 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6260000000000003e-06 2024-09-21T14:10:44.529 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.530 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.530 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.530 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.505275+0000", 2024-09-21T14:10:44.530 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1229028670000001 2024-09-21T14:10:44.530 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.530 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.530 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:10:44.530 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.505402+0000", 2024-09-21T14:10:44.530 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012643299999999999 2024-09-21T14:10:44.530 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.530 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.530 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3,7", 2024-09-21T14:10:44.530 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.505573+0000", 2024-09-21T14:10:44.530 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00017150299999999999 2024-09-21T14:10:44.531 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.531 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.531 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:44.531 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.507545+0000", 2024-09-21T14:10:44.531 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0019714939999999999 2024-09-21T14:10:44.531 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.531 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.531 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:10:44.531 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.513978+0000", 2024-09-21T14:10:44.531 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0064327050000000004 2024-09-21T14:10:44.531 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.531 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.531 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:44.531 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.525537+0000", 2024-09-21T14:10:44.532 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.011559135 2024-09-21T14:10:44.532 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.532 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.532 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:44.532 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.529080+0000", 2024-09-21T14:10:44.532 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0035429580000000001 2024-09-21T14:10:44.532 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.532 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.532 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:10:44.532 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.529159+0000", 2024-09-21T14:10:44.532 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8903e-05 2024-09-21T14:10:44.532 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.532 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.532 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:44.533 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.529287+0000", 2024-09-21T14:10:44.533 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012888999999999999 2024-09-21T14:10:44.533 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:44.533 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:44.533 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:44.533 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.533 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.533 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26396.0:114673 11.1 11:8d58e7bd:::benchmark_data_smithi049_206132_object114672:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e496)", 2024-09-21T14:10:44.533 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:08:13.458726+0000", 2024-09-21T14:10:44.533 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 150.916903781, 2024-09-21T14:10:44.533 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0464583840000001, 2024-09-21T14:10:44.533 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:10:44.533 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:10:44.533 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:44.534 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:10:44.534 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26396", 2024-09-21T14:10:44.534 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:44.534 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 114673 2024-09-21T14:10:44.534 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.534 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:10:44.534 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.534 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:10:44.534 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.458726+0000", 2024-09-21T14:10:44.534 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:44.534 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.534 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.534 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:10:44.534 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.458726+0000", 2024-09-21T14:10:44.535 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:10:44.535 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.535 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.535 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:10:44.535 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.458732+0000", 2024-09-21T14:10:44.535 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6089999999999997e-06 2024-09-21T14:10:44.535 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.535 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.535 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:10:44.535 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.458762+0000", 2024-09-21T14:10:44.535 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0312e-05 2024-09-21T14:10:44.535 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.535 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.535 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:10:44.536 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.458777+0000", 2024-09-21T14:10:44.536 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4078e-05 2024-09-21T14:10:44.536 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.536 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.536 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:10:44.536 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.458784+0000", 2024-09-21T14:10:44.536 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6590000000000007e-06 2024-09-21T14:10:44.536 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.536 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.536 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.536 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.458844+0000", 2024-09-21T14:10:44.536 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0198000000000002e-05 2024-09-21T14:10:44.536 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.536 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.537 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:44.537 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:13.458855+0000", 2024-09-21T14:10:44.537 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0926e-05 2024-09-21T14:10:44.537 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.537 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.537 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.537 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:14.382334+0000", 2024-09-21T14:10:44.537 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.92347837899999996 2024-09-21T14:10:44.537 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.537 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.537 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:10:44.537 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:14.382342+0000", 2024-09-21T14:10:44.537 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5199999999999997e-06 2024-09-21T14:10:44.537 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.538 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.538 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:10:44.538 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.493596+0000", 2024-09-21T14:10:44.538 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1112538190000001 2024-09-21T14:10:44.538 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.538 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.538 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:10:44.538 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.493665+0000", 2024-09-21T14:10:44.538 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9076000000000001e-05 2024-09-21T14:10:44.538 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.538 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.538 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2,3", 2024-09-21T14:10:44.538 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.493770+0000", 2024-09-21T14:10:44.538 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000104502 2024-09-21T14:10:44.539 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.539 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.539 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:10:44.539 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.503813+0000", 2024-09-21T14:10:44.539 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.010043557999999999 2024-09-21T14:10:44.539 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.539 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.539 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:44.539 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.504820+0000", 2024-09-21T14:10:44.539 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0010069899999999999 2024-09-21T14:10:44.539 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.539 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.539 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:44.540 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.504875+0000", 2024-09-21T14:10:44.540 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5282999999999999e-05 2024-09-21T14:10:44.540 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.540 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.540 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:44.540 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.504974+0000", 2024-09-21T14:10:44.540 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.8987000000000006e-05 2024-09-21T14:10:44.540 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.540 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.540 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:10:44.540 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.505032+0000", 2024-09-21T14:10:44.540 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7970000000000002e-05 2024-09-21T14:10:44.540 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:10:44.540 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:10:44.540 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:10:44.541 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:08:15.505184+0000", 2024-09-21T14:10:44.541 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015151800000000001 2024-09-21T14:10:44.541 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:44.541 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:44.541 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:44.541 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:10:44.541 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:10:44.541 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T14:10:44.542 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:44 smithi049 ceph-mon[40489]: 5.6 scrub starts 2024-09-21T14:10:44.542 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:44 smithi049 ceph-mon[40489]: 5.6 scrub ok 2024-09-21T14:10:44.542 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:44 smithi049 ceph-mon[40489]: 4.10 scrub starts 2024-09-21T14:10:44.542 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:44 smithi049 ceph-mon[40489]: 4.10 scrub ok 2024-09-21T14:10:44.542 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:44 smithi049 ceph-mon[30080]: 5.6 scrub starts 2024-09-21T14:10:44.542 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:44 smithi049 ceph-mon[30080]: 5.6 scrub ok 2024-09-21T14:10:44.542 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:44 smithi049 ceph-mon[30080]: 4.10 scrub starts 2024-09-21T14:10:44.542 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:44 smithi049 ceph-mon[30080]: 4.10 scrub ok 2024-09-21T14:10:44.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:44 smithi187 ceph-mon[33032]: 5.6 scrub starts 2024-09-21T14:10:44.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:44 smithi187 ceph-mon[33032]: 5.6 scrub ok 2024-09-21T14:10:44.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:44 smithi187 ceph-mon[33032]: 4.10 scrub starts 2024-09-21T14:10:44.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:44 smithi187 ceph-mon[33032]: 4.10 scrub ok 2024-09-21T14:10:45.237 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T14:10:45.277 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.0 to 1 (10000) 2024-09-21T14:10:45.587 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:10:45.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:45 smithi187 ceph-mon[33032]: pgmap v1122: 152 pgs: 152 active+clean; 609 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-09-21T14:10:45.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:45 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1180673846' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T14:10:45.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:45 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T14:10:45.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:45 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:10:45.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:45 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:10:45.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:45 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T14:10:45.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:45 smithi187 ceph-mon[33032]: osdmap e556: 8 total, 8 up, 8 in 2024-09-21T14:10:45.707 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:45 smithi049 ceph-mon[40489]: pgmap v1122: 152 pgs: 152 active+clean; 609 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-09-21T14:10:45.707 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:45 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1180673846' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T14:10:45.707 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:45 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T14:10:45.707 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:45 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:10:45.707 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:45 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:10:45.707 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:45 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T14:10:45.708 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:45 smithi049 ceph-mon[40489]: osdmap e556: 8 total, 8 up, 8 in 2024-09-21T14:10:45.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:45 smithi049 ceph-mon[30080]: pgmap v1122: 152 pgs: 152 active+clean; 609 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-09-21T14:10:45.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:45 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1180673846' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T14:10:45.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:45 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T14:10:45.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:45 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:10:45.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:45 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:10:45.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:45 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T14:10:45.708 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:45 smithi049 ceph-mon[30080]: osdmap e556: 8 total, 8 up, 8 in 2024-09-21T14:10:46.433 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 0 1 2024-09-21T14:10:46.660 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T14:10:46.661 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-09-21T14:10:46.661 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 0 2024-09-21T14:10:46.661 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T14:10:47.355 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T14:10:47.454 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:10:47.576 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:47 smithi049 ceph-mon[40489]: 1.f scrub starts 2024-09-21T14:10:47.576 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:47 smithi049 ceph-mon[40489]: 1.f scrub ok 2024-09-21T14:10:47.576 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:47 smithi049 ceph-mon[40489]: pgmap v1124: 152 pgs: 152 active+clean; 609 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-09-21T14:10:47.576 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:47 smithi049 ceph-mon[40489]: 5.4 scrub starts 2024-09-21T14:10:47.576 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:47 smithi049 ceph-mon[40489]: 5.4 scrub ok 2024-09-21T14:10:47.576 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:47 smithi049 ceph-mon[30080]: 1.f scrub starts 2024-09-21T14:10:47.577 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:47 smithi049 ceph-mon[30080]: 1.f scrub ok 2024-09-21T14:10:47.577 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:47 smithi049 ceph-mon[30080]: pgmap v1124: 152 pgs: 152 active+clean; 609 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-09-21T14:10:47.577 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:47 smithi049 ceph-mon[30080]: 5.4 scrub starts 2024-09-21T14:10:47.577 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:47 smithi049 ceph-mon[30080]: 5.4 scrub ok 2024-09-21T14:10:47.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:47 smithi187 ceph-mon[33032]: 1.f scrub starts 2024-09-21T14:10:47.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:47 smithi187 ceph-mon[33032]: 1.f scrub ok 2024-09-21T14:10:47.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:47 smithi187 ceph-mon[33032]: pgmap v1124: 152 pgs: 152 active+clean; 609 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-09-21T14:10:47.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:47 smithi187 ceph-mon[33032]: 5.4 scrub starts 2024-09-21T14:10:47.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:47 smithi187 ceph-mon[33032]: 5.4 scrub ok 2024-09-21T14:10:48.048 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:10:48.320 INFO:teuthology.orchestra.run.smithi049.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-21T14:10:48.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:48 smithi049 ceph-mon[40489]: 1.e scrub starts 2024-09-21T14:10:48.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:48 smithi049 ceph-mon[40489]: 1.e scrub ok 2024-09-21T14:10:48.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:48 smithi049 ceph-mon[40489]: 5.9 scrub starts 2024-09-21T14:10:48.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:48 smithi049 ceph-mon[40489]: 5.9 scrub ok 2024-09-21T14:10:48.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:48 smithi049 ceph-mon[40489]: 4.12 scrub starts 2024-09-21T14:10:48.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:48 smithi049 ceph-mon[40489]: 4.12 scrub ok 2024-09-21T14:10:48.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:48 smithi049 ceph-mon[40489]: 5.1b scrub starts 2024-09-21T14:10:48.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:48 smithi049 ceph-mon[40489]: 5.1b scrub ok 2024-09-21T14:10:48.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:48 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1367264013' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-21T14:10:48.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:48 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-21T14:10:48.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:48 smithi049 ceph-mon[30080]: 1.e scrub starts 2024-09-21T14:10:48.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:48 smithi049 ceph-mon[30080]: 1.e scrub ok 2024-09-21T14:10:48.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:48 smithi049 ceph-mon[30080]: 5.9 scrub starts 2024-09-21T14:10:48.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:48 smithi049 ceph-mon[30080]: 5.9 scrub ok 2024-09-21T14:10:48.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:48 smithi049 ceph-mon[30080]: 4.12 scrub starts 2024-09-21T14:10:48.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:48 smithi049 ceph-mon[30080]: 4.12 scrub ok 2024-09-21T14:10:48.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:48 smithi049 ceph-mon[30080]: 5.1b scrub starts 2024-09-21T14:10:48.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:48 smithi049 ceph-mon[30080]: 5.1b scrub ok 2024-09-21T14:10:48.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:48 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1367264013' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-21T14:10:48.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:48 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-21T14:10:48.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:48 smithi187 ceph-mon[33032]: 1.e scrub starts 2024-09-21T14:10:48.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:48 smithi187 ceph-mon[33032]: 1.e scrub ok 2024-09-21T14:10:48.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:48 smithi187 ceph-mon[33032]: 5.9 scrub starts 2024-09-21T14:10:48.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:48 smithi187 ceph-mon[33032]: 5.9 scrub ok 2024-09-21T14:10:48.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:48 smithi187 ceph-mon[33032]: 4.12 scrub starts 2024-09-21T14:10:48.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:48 smithi187 ceph-mon[33032]: 4.12 scrub ok 2024-09-21T14:10:48.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:48 smithi187 ceph-mon[33032]: 5.1b scrub starts 2024-09-21T14:10:48.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:48 smithi187 ceph-mon[33032]: 5.1b scrub ok 2024-09-21T14:10:48.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:48 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1367264013' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-21T14:10:48.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:48 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-21T14:10:48.813 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T14:10:48.813 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-09-21T14:10:48.813 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-09-21T14:10:48.813 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-09-21T14:10:48.813 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T14:10:49.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:49 smithi049 ceph-mon[30080]: 1.5 scrub starts 2024-09-21T14:10:49.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:49 smithi049 ceph-mon[30080]: 1.5 scrub ok 2024-09-21T14:10:49.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:49 smithi049 ceph-mon[30080]: pgmap v1125: 152 pgs: 152 active+clean; 609 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-09-21T14:10:49.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:49 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-21T14:10:49.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:49 smithi049 ceph-mon[30080]: osdmap e557: 8 total, 8 up, 8 in 2024-09-21T14:10:49.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:49 smithi049 ceph-mon[30080]: 3.5 scrub starts 2024-09-21T14:10:49.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:49 smithi049 ceph-mon[30080]: 3.5 scrub ok 2024-09-21T14:10:49.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:49 smithi049 ceph-mon[40489]: 1.5 scrub starts 2024-09-21T14:10:49.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:49 smithi049 ceph-mon[40489]: 1.5 scrub ok 2024-09-21T14:10:49.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:49 smithi049 ceph-mon[40489]: pgmap v1125: 152 pgs: 152 active+clean; 609 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-09-21T14:10:49.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:49 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-21T14:10:49.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:49 smithi049 ceph-mon[40489]: osdmap e557: 8 total, 8 up, 8 in 2024-09-21T14:10:49.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:49 smithi049 ceph-mon[40489]: 3.5 scrub starts 2024-09-21T14:10:49.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:49 smithi049 ceph-mon[40489]: 3.5 scrub ok 2024-09-21T14:10:49.601 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T14:10:49.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:49 smithi187 ceph-mon[33032]: 1.5 scrub starts 2024-09-21T14:10:49.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:49 smithi187 ceph-mon[33032]: 1.5 scrub ok 2024-09-21T14:10:49.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:49 smithi187 ceph-mon[33032]: pgmap v1125: 152 pgs: 152 active+clean; 609 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-09-21T14:10:49.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:49 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-21T14:10:49.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:49 smithi187 ceph-mon[33032]: osdmap e557: 8 total, 8 up, 8 in 2024-09-21T14:10:49.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:49 smithi187 ceph-mon[33032]: 3.5 scrub starts 2024-09-21T14:10:49.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:49 smithi187 ceph-mon[33032]: 3.5 scrub ok 2024-09-21T14:10:50.149 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd reweight 1 1 2024-09-21T14:10:50.190 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:10:50.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:50 smithi049 ceph-mon[30080]: 3.1e scrub starts 2024-09-21T14:10:50.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:50 smithi049 ceph-mon[30080]: 3.1e scrub ok 2024-09-21T14:10:50.456 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:50 smithi049 ceph-mon[30080]: osdmap e558: 8 total, 8 up, 8 in 2024-09-21T14:10:50.456 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:50 smithi049 ceph-mon[40489]: 3.1e scrub starts 2024-09-21T14:10:50.456 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:50 smithi049 ceph-mon[40489]: 3.1e scrub ok 2024-09-21T14:10:50.456 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:50 smithi049 ceph-mon[40489]: osdmap e558: 8 total, 8 up, 8 in 2024-09-21T14:10:50.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:50 smithi187 ceph-mon[33032]: 3.1e scrub starts 2024-09-21T14:10:50.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:50 smithi187 ceph-mon[33032]: 3.1e scrub ok 2024-09-21T14:10:50.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:50 smithi187 ceph-mon[33032]: osdmap e558: 8 total, 8 up, 8 in 2024-09-21T14:10:50.695 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:10:50 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:10:50.720 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:10:50 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:10:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:10:50.726 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T14:10:50.727 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-09-21T14:10:50.727 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-09-21T14:10:50.727 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-09-21T14:10:50.727 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.727 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:70935 10.5 10:a1fdc64d:::benchmark_data_smithi049_176811_object70934:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-09-21T14:10:50.727 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:03:44.044851+0000", 2024-09-21T14:10:50.727 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 426.681099077, 2024-09-21T14:10:50.727 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.31126390999999998, 2024-09-21T14:10:50.727 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:50.727 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:50.727 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:50.728 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:10:50.728 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:10:50.728 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:10:50.728 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 70935 2024-09-21T14:10:50.728 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.728 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:50.728 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.728 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:50.728 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044851+0000", 2024-09-21T14:10:50.728 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:50.729 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.729 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.729 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:50.729 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044851+0000", 2024-09-21T14:10:50.729 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:50.729 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.729 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.729 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:50.729 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044852+0000", 2024-09-21T14:10:50.729 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0800000000000004e-07 2024-09-21T14:10:50.729 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.729 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.729 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:50.729 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044873+0000", 2024-09-21T14:10:50.730 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.0999e-05 2024-09-21T14:10:50.730 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.730 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.730 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:50.730 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044882+0000", 2024-09-21T14:10:50.730 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6950000000000006e-06 2024-09-21T14:10:50.730 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.730 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.731 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:50.731 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.044888+0000", 2024-09-21T14:10:50.731 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2770000000000002e-06 2024-09-21T14:10:50.731 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.731 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.731 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:50.731 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.053728+0000", 2024-09-21T14:10:50.731 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.008840311 2024-09-21T14:10:50.731 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.731 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.731 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:50.731 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.053855+0000", 2024-09-21T14:10:50.731 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000126174 2024-09-21T14:10:50.731 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.732 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.732 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,3", 2024-09-21T14:10:50.732 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.053941+0000", 2024-09-21T14:10:50.732 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6009999999999998e-05 2024-09-21T14:10:50.732 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.732 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.732 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:10:50.732 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.056428+0000", 2024-09-21T14:10:50.732 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0024874340000000002 2024-09-21T14:10:50.732 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.732 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.732 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:50.732 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.060518+0000", 2024-09-21T14:10:50.732 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.004089606 2024-09-21T14:10:50.732 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.732 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.733 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:50.733 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.356032+0000", 2024-09-21T14:10:50.733 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.29551429600000001 2024-09-21T14:10:50.733 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.733 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.733 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:10:50.733 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.356082+0000", 2024-09-21T14:10:50.733 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9679000000000003e-05 2024-09-21T14:10:50.733 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.733 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.733 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:50.733 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:03:44.356115+0000", 2024-09-21T14:10:50.733 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3821e-05 2024-09-21T14:10:50.734 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:50.734 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:50.734 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:50.734 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.734 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.734 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:208866 10.3 10:c8242cfb:::benchmark_data_smithi049_176811_object81289:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e441)", 2024-09-21T14:10:50.735 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:05:54.047026+0000", 2024-09-21T14:10:50.735 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 296.67892463700002, 2024-09-21T14:10:50.735 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.31484621499999998, 2024-09-21T14:10:50.735 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:50.735 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:50.735 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:50.735 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:10:50.735 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:10:50.735 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:10:50.735 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 208866 2024-09-21T14:10:50.735 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.735 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:50.735 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.735 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:50.735 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047026+0000", 2024-09-21T14:10:50.736 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:50.736 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.736 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.736 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:50.736 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047026+0000", 2024-09-21T14:10:50.736 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:50.736 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.736 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.736 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:50.736 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047027+0000", 2024-09-21T14:10:50.736 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3820000000000001e-06 2024-09-21T14:10:50.736 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.736 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.736 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:50.736 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047033+0000", 2024-09-21T14:10:50.737 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3859999999999998e-06 2024-09-21T14:10:50.737 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.737 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.737 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:50.737 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047046+0000", 2024-09-21T14:10:50.737 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3482999999999999e-05 2024-09-21T14:10:50.737 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.737 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.737 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:50.737 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047053+0000", 2024-09-21T14:10:50.737 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.25e-06 2024-09-21T14:10:50.737 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.737 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.737 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:50.737 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047124+0000", 2024-09-21T14:10:50.738 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0309999999999996e-05 2024-09-21T14:10:50.738 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.738 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.738 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:50.738 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047243+0000", 2024-09-21T14:10:50.738 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011944599999999999 2024-09-21T14:10:50.738 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.738 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.738 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,5", 2024-09-21T14:10:50.738 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047366+0000", 2024-09-21T14:10:50.738 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012313100000000001 2024-09-21T14:10:50.739 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.739 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.739 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:10:50.739 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.052472+0000", 2024-09-21T14:10:50.739 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0051057760000000002 2024-09-21T14:10:50.739 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.739 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.739 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:50.739 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.055019+0000", 2024-09-21T14:10:50.739 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0025469709999999999 2024-09-21T14:10:50.739 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.740 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.740 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:50.740 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.361767+0000", 2024-09-21T14:10:50.740 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.30674835499999997 2024-09-21T14:10:50.740 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.740 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.740 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:10:50.740 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.361826+0000", 2024-09-21T14:10:50.740 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8084999999999997e-05 2024-09-21T14:10:50.740 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.740 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.740 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:50.740 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.361872+0000", 2024-09-21T14:10:50.740 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6640000000000001e-05 2024-09-21T14:10:50.740 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:50.741 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:50.741 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:50.741 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.741 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.741 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:208867 10.3 10:c6755fb4:::benchmark_data_smithi049_176811_object81290:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e441)", 2024-09-21T14:10:50.741 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:05:54.047593+0000", 2024-09-21T14:10:50.741 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 296.67835738999997, 2024-09-21T14:10:50.741 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.31790776100000001, 2024-09-21T14:10:50.741 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:50.741 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:50.741 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:50.741 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:10:50.741 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:10:50.742 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:10:50.742 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 208867 2024-09-21T14:10:50.742 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.742 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:50.742 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.742 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:50.743 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047593+0000", 2024-09-21T14:10:50.743 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:50.743 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.743 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.743 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:50.743 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047593+0000", 2024-09-21T14:10:50.743 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:50.743 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.743 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.743 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:50.743 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047594+0000", 2024-09-21T14:10:50.743 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9100000000000003e-07 2024-09-21T14:10:50.743 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.744 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.744 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:50.744 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047598+0000", 2024-09-21T14:10:50.744 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0199999999999996e-06 2024-09-21T14:10:50.744 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.744 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.744 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:50.744 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047609+0000", 2024-09-21T14:10:50.744 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0858e-05 2024-09-21T14:10:50.744 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.744 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.744 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:50.744 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047616+0000", 2024-09-21T14:10:50.745 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3629999999999996e-06 2024-09-21T14:10:50.745 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.745 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.745 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:50.745 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.050406+0000", 2024-09-21T14:10:50.745 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0027899639999999998 2024-09-21T14:10:50.745 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.745 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.745 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:50.745 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.050525+0000", 2024-09-21T14:10:50.745 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011861 2024-09-21T14:10:50.745 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.745 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.745 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 3,5", 2024-09-21T14:10:50.746 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.050650+0000", 2024-09-21T14:10:50.746 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000124881 2024-09-21T14:10:50.746 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.746 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.746 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:10:50.746 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.052609+0000", 2024-09-21T14:10:50.746 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0019588499999999998 2024-09-21T14:10:50.746 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.746 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.746 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:50.746 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.057528+0000", 2024-09-21T14:10:50.746 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0049191590000000002 2024-09-21T14:10:50.746 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.747 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.747 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:50.747 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.365390+0000", 2024-09-21T14:10:50.747 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.30786228399999999 2024-09-21T14:10:50.747 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.747 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.747 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:10:50.747 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.365449+0000", 2024-09-21T14:10:50.747 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9497999999999999e-05 2024-09-21T14:10:50.747 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.747 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.747 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:50.748 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.365501+0000", 2024-09-21T14:10:50.748 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1483000000000002e-05 2024-09-21T14:10:50.748 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:50.748 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:50.748 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:50.748 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.748 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.748 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.16068.0:208869 10.5 10:a17fb6ee:::benchmark_data_smithi049_176811_object81292:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e441)", 2024-09-21T14:10:50.748 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:05:54.048229+0000", 2024-09-21T14:10:50.748 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 296.67772169800003, 2024-09-21T14:10:50.748 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.31834309799999999, 2024-09-21T14:10:50.748 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:50.748 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:50.749 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:50.749 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:10:50.749 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.16068", 2024-09-21T14:10:50.749 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/2447595313", 2024-09-21T14:10:50.749 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 208869 2024-09-21T14:10:50.749 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.749 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:50.749 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.749 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:50.749 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.048229+0000", 2024-09-21T14:10:50.749 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:50.749 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.749 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.749 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:50.750 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.048229+0000", 2024-09-21T14:10:50.750 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:50.750 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.750 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.750 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:50.750 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.048230+0000", 2024-09-21T14:10:50.750 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3799999999999996e-07 2024-09-21T14:10:50.750 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.750 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.750 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:50.750 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.048233+0000", 2024-09-21T14:10:50.750 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.602e-06 2024-09-21T14:10:50.751 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.751 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.751 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:50.751 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.048242+0000", 2024-09-21T14:10:50.751 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0389999999999993e-06 2024-09-21T14:10:50.751 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.751 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.751 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:50.751 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.048249+0000", 2024-09-21T14:10:50.751 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7599999999999997e-06 2024-09-21T14:10:50.751 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.751 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.751 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:50.752 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.048305+0000", 2024-09-21T14:10:50.752 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6085999999999998e-05 2024-09-21T14:10:50.752 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.752 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.752 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:50.752 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.048397+0000", 2024-09-21T14:10:50.752 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2069000000000002e-05 2024-09-21T14:10:50.752 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.752 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.752 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,3", 2024-09-21T14:10:50.752 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.048465+0000", 2024-09-21T14:10:50.752 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7998999999999997e-05 2024-09-21T14:10:50.752 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.753 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.753 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:10:50.753 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.050592+0000", 2024-09-21T14:10:50.753 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0021271509999999999 2024-09-21T14:10:50.753 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.753 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.753 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:50.753 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.051296+0000", 2024-09-21T14:10:50.753 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00070396900000000004 2024-09-21T14:10:50.753 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.753 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.753 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:50.753 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.366467+0000", 2024-09-21T14:10:50.753 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.31517033700000002 2024-09-21T14:10:50.754 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.754 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.754 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:10:50.754 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.366534+0000", 2024-09-21T14:10:50.754 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7045999999999995e-05 2024-09-21T14:10:50.754 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.754 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.754 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:50.754 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.366572+0000", 2024-09-21T14:10:50.754 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8201999999999998e-05 2024-09-21T14:10:50.754 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:50.754 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:50.754 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:50.755 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.755 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.755 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:209190 11.2 11:48c4ebab:::benchmark_data_smithi049_206132_object85811:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e531)", 2024-09-21T14:10:50.755 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:09:52.495398+0000", 2024-09-21T14:10:50.755 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 58.230552979999999, 2024-09-21T14:10:50.755 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0207089579999999, 2024-09-21T14:10:50.755 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:50.755 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:50.756 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T14:10:50.756 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:10:50.756 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:10:50.756 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:50.756 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 209190 2024-09-21T14:10:50.756 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.756 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:50.756 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.756 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:50.756 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.495398+0000", 2024-09-21T14:10:50.757 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:50.757 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.757 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.757 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:50.757 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.495398+0000", 2024-09-21T14:10:50.757 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:50.757 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.757 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.757 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:50.757 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.495399+0000", 2024-09-21T14:10:50.757 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.832e-06 2024-09-21T14:10:50.757 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.758 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.758 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:50.758 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.495407+0000", 2024-09-21T14:10:50.758 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5220000000000001e-06 2024-09-21T14:10:50.758 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.758 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.758 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:50.758 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.495422+0000", 2024-09-21T14:10:50.758 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.502e-05 2024-09-21T14:10:50.758 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.758 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.758 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:50.759 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.495436+0000", 2024-09-21T14:10:50.759 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3929e-05 2024-09-21T14:10:50.759 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.759 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.759 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:50.759 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.496286+0000", 2024-09-21T14:10:50.759 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00084996599999999996 2024-09-21T14:10:50.759 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.759 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.759 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:10:50.759 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.496296+0000", 2024-09-21T14:10:50.759 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0468999999999999e-05 2024-09-21T14:10:50.760 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.760 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.760 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:50.760 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.514917+0000", 2024-09-21T14:10:50.760 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.018620458 2024-09-21T14:10:50.760 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.760 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.760 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:50.760 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.516107+0000", 2024-09-21T14:10:50.760 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001189762 2024-09-21T14:10:50.760 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:50.760 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:50.760 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:50.760 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.760 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.761 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:209197 11.6 11:6f0fe2c4:::benchmark_data_smithi049_206132_object85818:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e531)", 2024-09-21T14:10:50.761 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:09:52.495460+0000", 2024-09-21T14:10:50.761 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 58.230491016999999, 2024-09-21T14:10:50.761 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0260732859999999, 2024-09-21T14:10:50.761 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:50.761 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:50.761 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:50.761 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:10:50.761 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:10:50.761 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:50.761 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 209197 2024-09-21T14:10:50.761 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.761 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:50.761 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.762 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:50.762 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.495460+0000", 2024-09-21T14:10:50.762 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:50.762 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.762 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.762 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:50.762 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.495460+0000", 2024-09-21T14:10:50.762 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:50.762 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.762 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.762 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:50.762 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.495460+0000", 2024-09-21T14:10:50.762 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9599999999999999e-07 2024-09-21T14:10:50.763 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.763 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.763 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:50.763 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.495463+0000", 2024-09-21T14:10:50.763 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.818e-06 2024-09-21T14:10:50.763 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.763 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.763 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:50.763 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.495470+0000", 2024-09-21T14:10:50.763 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8739999999999997e-06 2024-09-21T14:10:50.763 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.763 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.763 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:50.764 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.495476+0000", 2024-09-21T14:10:50.764 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9549999999999997e-06 2024-09-21T14:10:50.764 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.764 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.764 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:50.764 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.495616+0000", 2024-09-21T14:10:50.764 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014016699999999999 2024-09-21T14:10:50.764 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.764 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.764 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:10:50.764 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.495627+0000", 2024-09-21T14:10:50.764 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0994e-05 2024-09-21T14:10:50.764 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.764 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.765 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:50.765 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.513859+0000", 2024-09-21T14:10:50.765 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.018232238 2024-09-21T14:10:50.765 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.765 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.765 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:50.765 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.513947+0000", 2024-09-21T14:10:50.765 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.7767000000000002e-05 2024-09-21T14:10:50.765 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.765 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.765 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,1", 2024-09-21T14:10:50.765 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.514034+0000", 2024-09-21T14:10:50.765 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.7287000000000006e-05 2024-09-21T14:10:50.766 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.766 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.766 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:50.766 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.516989+0000", 2024-09-21T14:10:50.766 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002954903 2024-09-21T14:10:50.766 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.766 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.766 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:10:50.766 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.521368+0000", 2024-09-21T14:10:50.766 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0043789809999999997 2024-09-21T14:10:50.766 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.766 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.766 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:50.767 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.521420+0000", 2024-09-21T14:10:50.767 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1678999999999998e-05 2024-09-21T14:10:50.767 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.767 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.767 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:10:50.767 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.521473+0000", 2024-09-21T14:10:50.767 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.3523999999999998e-05 2024-09-21T14:10:50.767 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.767 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.767 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:50.767 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.521533+0000", 2024-09-21T14:10:50.767 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9602999999999998e-05 2024-09-21T14:10:50.767 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:50.767 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:50.768 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:50.768 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.768 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.768 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:209200 11.2 11:4b6066f2:::benchmark_data_smithi049_206132_object85821:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e531)", 2024-09-21T14:10:50.768 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:09:52.495501+0000", 2024-09-21T14:10:50.768 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 58.23044986, 2024-09-21T14:10:50.768 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0333213649999999, 2024-09-21T14:10:50.768 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:50.768 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:50.768 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:50.768 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:10:50.768 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:10:50.768 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:50.769 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 209200 2024-09-21T14:10:50.769 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.769 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:50.769 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.769 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:50.769 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.495501+0000", 2024-09-21T14:10:50.769 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:50.770 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.770 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.770 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:50.770 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.495501+0000", 2024-09-21T14:10:50.770 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:50.771 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.771 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.771 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:50.771 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.495501+0000", 2024-09-21T14:10:50.771 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2099999999999997e-07 2024-09-21T14:10:50.771 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.771 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.771 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:50.771 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.495504+0000", 2024-09-21T14:10:50.771 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5600000000000001e-06 2024-09-21T14:10:50.771 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.771 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.772 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:50.772 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.495511+0000", 2024-09-21T14:10:50.772 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9550000000000003e-06 2024-09-21T14:10:50.772 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.772 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.772 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:50.772 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.495517+0000", 2024-09-21T14:10:50.772 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6730000000000003e-06 2024-09-21T14:10:50.773 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.773 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.773 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:50.774 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.496325+0000", 2024-09-21T14:10:50.774 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00080740800000000002 2024-09-21T14:10:50.774 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.774 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.774 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:10:50.774 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.496334+0000", 2024-09-21T14:10:50.774 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9439999999999994e-06 2024-09-21T14:10:50.774 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.774 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.774 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:50.774 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.516697+0000", 2024-09-21T14:10:50.775 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.02036354 2024-09-21T14:10:50.775 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.775 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.775 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:50.775 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.517161+0000", 2024-09-21T14:10:50.775 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00046412999999999999 2024-09-21T14:10:50.775 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.775 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.775 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:10:50.775 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.517267+0000", 2024-09-21T14:10:50.775 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000105813 2024-09-21T14:10:50.775 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.775 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.776 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:50.776 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.523693+0000", 2024-09-21T14:10:50.776 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0064257180000000004 2024-09-21T14:10:50.776 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.776 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.776 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:10:50.776 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.528648+0000", 2024-09-21T14:10:50.776 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0049553870000000003 2024-09-21T14:10:50.776 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.776 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.777 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:50.777 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.528737+0000", 2024-09-21T14:10:50.777 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.8572999999999994e-05 2024-09-21T14:10:50.777 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.777 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.777 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:10:50.777 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.528785+0000", 2024-09-21T14:10:50.777 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7799000000000001e-05 2024-09-21T14:10:50.777 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.777 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.777 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:50.777 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.528822+0000", 2024-09-21T14:10:50.777 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7344e-05 2024-09-21T14:10:50.777 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:50.777 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:50.778 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:50.778 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.778 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.778 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:209217 11.2 11:44aaea61:::benchmark_data_smithi049_206132_object85838:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531)", 2024-09-21T14:10:50.778 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:09:52.513184+0000", 2024-09-21T14:10:50.778 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 58.212767018999998, 2024-09-21T14:10:50.778 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0157760929999999, 2024-09-21T14:10:50.778 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:50.778 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:50.778 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:50.778 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:10:50.778 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:10:50.778 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:50.779 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 209217 2024-09-21T14:10:50.779 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.779 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:50.779 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.779 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:50.779 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.513184+0000", 2024-09-21T14:10:50.779 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:50.779 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.779 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.779 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:50.779 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.513184+0000", 2024-09-21T14:10:50.779 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:50.779 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.779 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.780 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:50.780 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.513185+0000", 2024-09-21T14:10:50.780 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.903e-06 2024-09-21T14:10:50.780 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.780 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.780 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:50.780 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.513193+0000", 2024-09-21T14:10:50.780 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2389999999999996e-06 2024-09-21T14:10:50.780 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.780 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.780 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:50.780 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.513205+0000", 2024-09-21T14:10:50.780 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2323999999999999e-05 2024-09-21T14:10:50.780 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.780 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.781 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:50.781 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.513211+0000", 2024-09-21T14:10:50.781 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1970000000000001e-06 2024-09-21T14:10:50.781 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.781 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.781 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:50.781 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.513288+0000", 2024-09-21T14:10:50.781 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6719999999999997e-05 2024-09-21T14:10:50.781 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.781 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.781 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:10:50.781 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.513299+0000", 2024-09-21T14:10:50.781 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1308000000000001e-05 2024-09-21T14:10:50.781 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.782 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.782 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:50.782 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.519175+0000", 2024-09-21T14:10:50.782 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.005875268 2024-09-21T14:10:50.782 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.782 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.782 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:50.783 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.519254+0000", 2024-09-21T14:10:50.783 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9301000000000005e-05 2024-09-21T14:10:50.783 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.783 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.783 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:10:50.783 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.519336+0000", 2024-09-21T14:10:50.784 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2225000000000006e-05 2024-09-21T14:10:50.784 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.784 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.784 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:50.784 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.523731+0000", 2024-09-21T14:10:50.784 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0043954629999999996 2024-09-21T14:10:50.785 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.785 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.785 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:10:50.785 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.528836+0000", 2024-09-21T14:10:50.785 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0051045960000000003 2024-09-21T14:10:50.785 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.785 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.786 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:50.786 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.528882+0000", 2024-09-21T14:10:50.786 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5524e-05 2024-09-21T14:10:50.786 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.786 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.786 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:10:50.787 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.528926+0000", 2024-09-21T14:10:50.787 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4124e-05 2024-09-21T14:10:50.787 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.787 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.787 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:50.787 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.528960+0000", 2024-09-21T14:10:50.787 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3900999999999999e-05 2024-09-21T14:10:50.788 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:50.788 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:50.788 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:50.788 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.788 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.788 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:209230 11.6 11:6489345e:::benchmark_data_smithi049_206132_object85851:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531)", 2024-09-21T14:10:50.789 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:09:52.525520+0000", 2024-09-21T14:10:50.789 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 58.200430371000003, 2024-09-21T14:10:50.789 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0008969729999999, 2024-09-21T14:10:50.789 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:50.789 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:50.789 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:50.789 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:10:50.790 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:10:50.790 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:50.790 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 209230 2024-09-21T14:10:50.790 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.790 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:50.790 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.791 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:50.791 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.525520+0000", 2024-09-21T14:10:50.791 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:50.791 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.791 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.791 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:50.792 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.525520+0000", 2024-09-21T14:10:50.792 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:50.792 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.792 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.792 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:50.792 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.525522+0000", 2024-09-21T14:10:50.793 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.745e-06 2024-09-21T14:10:50.793 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.793 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.793 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:50.793 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.525527+0000", 2024-09-21T14:10:50.793 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5219999999999998e-06 2024-09-21T14:10:50.794 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.794 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.794 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:50.794 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.525547+0000", 2024-09-21T14:10:50.794 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9100999999999999e-05 2024-09-21T14:10:50.794 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.794 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.795 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:50.795 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.525556+0000", 2024-09-21T14:10:50.795 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.5869999999999999e-06 2024-09-21T14:10:50.795 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.795 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.795 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:50.796 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.525696+0000", 2024-09-21T14:10:50.796 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014013 2024-09-21T14:10:50.796 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.796 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.796 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:10:50.798 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.525709+0000", 2024-09-21T14:10:50.798 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3169e-05 2024-09-21T14:10:50.798 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.798 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.799 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:50.799 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.517031+0000", 2024-09-21T14:10:50.799 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.991321542 2024-09-21T14:10:50.799 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.799 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.799 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:50.800 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.517141+0000", 2024-09-21T14:10:50.800 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000109701 2024-09-21T14:10:50.800 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.801 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.801 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,1", 2024-09-21T14:10:50.801 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.517205+0000", 2024-09-21T14:10:50.801 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4251000000000005e-05 2024-09-21T14:10:50.801 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.802 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.802 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:50.802 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.521590+0000", 2024-09-21T14:10:50.802 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0043846800000000002 2024-09-21T14:10:50.802 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.802 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.803 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:10:50.803 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.521713+0000", 2024-09-21T14:10:50.803 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012304500000000001 2024-09-21T14:10:50.803 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.803 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.803 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:50.804 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.526314+0000", 2024-09-21T14:10:50.804 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0046017430000000002 2024-09-21T14:10:50.804 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.804 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.804 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:10:50.805 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.526372+0000", 2024-09-21T14:10:50.805 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7265999999999997e-05 2024-09-21T14:10:50.805 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.805 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.805 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:50.805 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.526417+0000", 2024-09-21T14:10:50.806 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5491000000000002e-05 2024-09-21T14:10:50.806 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:50.806 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:50.806 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:50.806 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.806 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.807 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:209231 11.2 11:4d8ba023:::benchmark_data_smithi049_206132_object85852:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531)", 2024-09-21T14:10:50.807 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:09:52.525583+0000", 2024-09-21T14:10:50.807 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 58.200367313000001, 2024-09-21T14:10:50.807 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.00359139, 2024-09-21T14:10:50.807 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:50.807 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:50.808 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:50.808 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:10:50.808 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:10:50.808 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:50.808 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 209231 2024-09-21T14:10:50.808 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.809 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:50.809 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.809 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:50.809 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.525583+0000", 2024-09-21T14:10:50.809 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:50.809 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.810 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.810 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:50.810 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.525583+0000", 2024-09-21T14:10:50.810 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:50.810 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.810 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.811 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:50.811 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.525584+0000", 2024-09-21T14:10:50.811 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4799999999999998e-07 2024-09-21T14:10:50.811 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.811 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.811 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:50.812 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.525587+0000", 2024-09-21T14:10:50.812 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1209999999999998e-06 2024-09-21T14:10:50.812 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.812 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.812 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:50.812 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.525595+0000", 2024-09-21T14:10:50.813 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.3319999999999992e-06 2024-09-21T14:10:50.813 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.813 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.813 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:50.813 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.525602+0000", 2024-09-21T14:10:50.813 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6379999999999998e-06 2024-09-21T14:10:50.814 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.814 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.814 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:50.814 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.525759+0000", 2024-09-21T14:10:50.814 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00015704099999999999 2024-09-21T14:10:50.814 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.815 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.815 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:10:50.815 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.525768+0000", 2024-09-21T14:10:50.816 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5739999999999996e-06 2024-09-21T14:10:50.816 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.816 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.816 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:50.816 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.521258+0000", 2024-09-21T14:10:50.816 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.99549076999999997 2024-09-21T14:10:50.816 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.817 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.817 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:50.817 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.521372+0000", 2024-09-21T14:10:50.817 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011392600000000001 2024-09-21T14:10:50.817 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.817 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.818 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:10:50.818 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.521495+0000", 2024-09-21T14:10:50.818 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000123096 2024-09-21T14:10:50.818 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.818 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.818 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:50.819 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.528690+0000", 2024-09-21T14:10:50.819 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0071944859999999999 2024-09-21T14:10:50.819 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.819 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.819 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:10:50.819 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.528840+0000", 2024-09-21T14:10:50.819 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00015053000000000001 2024-09-21T14:10:50.819 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.820 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.820 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:50.820 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.529050+0000", 2024-09-21T14:10:50.820 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00020950999999999999 2024-09-21T14:10:50.820 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.820 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.820 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:10:50.821 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.529137+0000", 2024-09-21T14:10:50.821 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.7062000000000001e-05 2024-09-21T14:10:50.821 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.821 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.821 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:50.821 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.529175+0000", 2024-09-21T14:10:50.821 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7656e-05 2024-09-21T14:10:50.821 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:50.821 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:50.821 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:50.821 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.821 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.821 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:209232 11.2 11:4e263032:::benchmark_data_smithi049_206132_object85853:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531)", 2024-09-21T14:10:50.822 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:09:52.527394+0000", 2024-09-21T14:10:50.822 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 58.198556543000002, 2024-09-21T14:10:50.822 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0019058279999999, 2024-09-21T14:10:50.822 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:50.822 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:50.822 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:50.822 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:10:50.822 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:10:50.822 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:50.822 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 209232 2024-09-21T14:10:50.822 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.822 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:50.822 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.823 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:50.823 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.527394+0000", 2024-09-21T14:10:50.823 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:50.823 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.823 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.823 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:50.823 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.527394+0000", 2024-09-21T14:10:50.823 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:50.823 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.823 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.823 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:50.823 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.527395+0000", 2024-09-21T14:10:50.823 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.384e-06 2024-09-21T14:10:50.824 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.824 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.824 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:50.824 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.527401+0000", 2024-09-21T14:10:50.824 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2360000000000004e-06 2024-09-21T14:10:50.824 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.824 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.824 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:50.824 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.527416+0000", 2024-09-21T14:10:50.824 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5193e-05 2024-09-21T14:10:50.824 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.824 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.824 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:50.825 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.527423+0000", 2024-09-21T14:10:50.825 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.605e-06 2024-09-21T14:10:50.825 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.825 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.825 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:50.825 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.527482+0000", 2024-09-21T14:10:50.825 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8712000000000003e-05 2024-09-21T14:10:50.825 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.826 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.826 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:10:50.826 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.527502+0000", 2024-09-21T14:10:50.826 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9771000000000001e-05 2024-09-21T14:10:50.826 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.826 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.826 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:50.826 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.523761+0000", 2024-09-21T14:10:50.826 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.99625940300000004 2024-09-21T14:10:50.826 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.826 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.826 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:50.826 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.523866+0000", 2024-09-21T14:10:50.827 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000104375 2024-09-21T14:10:50.827 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.827 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.827 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:10:50.827 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.523964+0000", 2024-09-21T14:10:50.827 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.8527e-05 2024-09-21T14:10:50.827 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.827 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.827 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:10:50.827 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.528844+0000", 2024-09-21T14:10:50.827 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0048799459999999996 2024-09-21T14:10:50.827 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.827 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.828 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:50.828 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.529006+0000", 2024-09-21T14:10:50.828 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000162251 2024-09-21T14:10:50.828 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.828 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.828 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:50.828 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.529218+0000", 2024-09-21T14:10:50.828 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00021191999999999999 2024-09-21T14:10:50.828 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.828 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.828 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:10:50.828 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.529265+0000", 2024-09-21T14:10:50.828 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6788e-05 2024-09-21T14:10:50.828 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.829 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.829 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:50.829 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.529300+0000", 2024-09-21T14:10:50.829 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4717e-05 2024-09-21T14:10:50.829 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:50.829 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:50.829 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:50.829 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.829 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.830 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:209234 11.2 11:4bbdc344:::benchmark_data_smithi049_206132_object85855:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531)", 2024-09-21T14:10:50.830 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:09:52.527916+0000", 2024-09-21T14:10:50.830 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 58.198034143999998, 2024-09-21T14:10:50.830 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0040576569999999, 2024-09-21T14:10:50.830 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:50.830 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:50.830 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:50.830 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:10:50.831 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:10:50.831 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:50.831 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 209234 2024-09-21T14:10:50.831 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.831 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:50.831 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.832 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:50.832 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.527916+0000", 2024-09-21T14:10:50.832 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:50.832 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.832 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.832 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:50.832 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.527916+0000", 2024-09-21T14:10:50.832 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:50.832 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.833 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.833 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:50.833 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.527917+0000", 2024-09-21T14:10:50.833 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.0599999999999999e-07 2024-09-21T14:10:50.833 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.833 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.833 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:50.833 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.527922+0000", 2024-09-21T14:10:50.833 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6890000000000003e-06 2024-09-21T14:10:50.833 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.833 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.834 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:50.834 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.527936+0000", 2024-09-21T14:10:50.834 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4086e-05 2024-09-21T14:10:50.834 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.834 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.834 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:50.834 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.527943+0000", 2024-09-21T14:10:50.834 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9589999999999998e-06 2024-09-21T14:10:50.834 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.834 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.835 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:50.835 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.528022+0000", 2024-09-21T14:10:50.835 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9294000000000003e-05 2024-09-21T14:10:50.835 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.835 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.835 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:10:50.835 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.528030+0000", 2024-09-21T14:10:50.835 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.0779999999999996e-06 2024-09-21T14:10:50.835 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.835 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.835 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:50.835 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.526461+0000", 2024-09-21T14:10:50.835 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.99843068700000004 2024-09-21T14:10:50.835 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.835 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.836 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:50.836 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.526575+0000", 2024-09-21T14:10:50.836 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011430300000000001 2024-09-21T14:10:50.836 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.836 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.836 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:10:50.836 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.526678+0000", 2024-09-21T14:10:50.836 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000102376 2024-09-21T14:10:50.836 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.836 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.837 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:10:50.837 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.528969+0000", 2024-09-21T14:10:50.837 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0022908009999999999 2024-09-21T14:10:50.837 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.837 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.837 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:50.837 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.531845+0000", 2024-09-21T14:10:50.837 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002876048 2024-09-21T14:10:50.837 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.837 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.838 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:50.838 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.531887+0000", 2024-09-21T14:10:50.838 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2079000000000001e-05 2024-09-21T14:10:50.838 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.838 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.838 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:10:50.838 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.531933+0000", 2024-09-21T14:10:50.838 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5864e-05 2024-09-21T14:10:50.838 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.838 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.838 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:50.838 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.531974+0000", 2024-09-21T14:10:50.838 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1486999999999998e-05 2024-09-21T14:10:50.839 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:50.839 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:50.839 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:50.839 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.839 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.839 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:209243 11.6 11:683a5e8b:::benchmark_data_smithi049_206132_object85864:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531)", 2024-09-21T14:10:50.839 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:09:52.535678+0000", 2024-09-21T14:10:50.839 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 58.190272587999999, 2024-09-21T14:10:50.839 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.99088372499999999, 2024-09-21T14:10:50.839 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:50.839 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:50.839 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:50.839 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:10:50.840 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:10:50.840 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:50.840 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 209243 2024-09-21T14:10:50.840 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.840 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:50.840 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.840 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:50.840 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.535678+0000", 2024-09-21T14:10:50.840 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:50.840 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.840 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.840 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:50.840 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.535678+0000", 2024-09-21T14:10:50.841 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:50.841 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.841 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.841 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:50.841 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.535679+0000", 2024-09-21T14:10:50.841 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.232e-06 2024-09-21T14:10:50.841 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.841 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.841 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:50.841 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.535694+0000", 2024-09-21T14:10:50.841 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4632e-05 2024-09-21T14:10:50.841 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.842 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.842 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:50.842 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.535707+0000", 2024-09-21T14:10:50.842 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2697999999999999e-05 2024-09-21T14:10:50.842 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.842 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.842 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:50.842 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.535714+0000", 2024-09-21T14:10:50.842 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5379999999999997e-06 2024-09-21T14:10:50.842 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.842 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.843 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:50.843 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.535783+0000", 2024-09-21T14:10:50.843 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9002000000000001e-05 2024-09-21T14:10:50.843 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.843 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.843 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:10:50.843 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.535794+0000", 2024-09-21T14:10:50.843 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0727000000000001e-05 2024-09-21T14:10:50.843 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.843 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.843 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:50.843 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.519064+0000", 2024-09-21T14:10:50.844 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.98327018600000005 2024-09-21T14:10:50.844 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.844 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.844 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:50.844 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.519145+0000", 2024-09-21T14:10:50.844 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.1211999999999994e-05 2024-09-21T14:10:50.844 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.844 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.844 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,1", 2024-09-21T14:10:50.844 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.519238+0000", 2024-09-21T14:10:50.845 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2757000000000007e-05 2024-09-21T14:10:50.845 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.845 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.845 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:50.845 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.521764+0000", 2024-09-21T14:10:50.845 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002526403 2024-09-21T14:10:50.845 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.845 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.845 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:10:50.846 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.523901+0000", 2024-09-21T14:10:50.846 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0021364829999999998 2024-09-21T14:10:50.846 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.846 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.846 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:50.846 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.526457+0000", 2024-09-21T14:10:50.847 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0025565000000000002 2024-09-21T14:10:50.847 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.847 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.847 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:10:50.847 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.526524+0000", 2024-09-21T14:10:50.847 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6365000000000006e-05 2024-09-21T14:10:50.848 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.848 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.848 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:50.848 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.526562+0000", 2024-09-21T14:10:50.848 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.799e-05 2024-09-21T14:10:50.848 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:50.848 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:50.849 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:50.849 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.849 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.849 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:209248 11.6 11:68b6aab8:::benchmark_data_smithi049_206132_object85869:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531)", 2024-09-21T14:10:50.849 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:09:52.543088+0000", 2024-09-21T14:10:50.849 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 58.182862567999997, 2024-09-21T14:10:50.850 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.99161255500000001, 2024-09-21T14:10:50.850 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:50.850 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:50.850 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:50.850 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:10:50.850 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:10:50.851 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:50.851 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 209248 2024-09-21T14:10:50.851 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.851 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:50.851 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.851 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:50.852 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.543088+0000", 2024-09-21T14:10:50.852 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:50.852 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.852 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.852 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:50.852 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.543088+0000", 2024-09-21T14:10:50.853 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:50.853 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.853 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.853 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:50.853 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.543090+0000", 2024-09-21T14:10:50.853 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9920000000000002e-06 2024-09-21T14:10:50.854 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.854 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.854 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:50.854 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.543096+0000", 2024-09-21T14:10:50.854 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9869999999999996e-06 2024-09-21T14:10:50.854 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.855 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.855 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:50.855 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.543114+0000", 2024-09-21T14:10:50.855 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7920000000000001e-05 2024-09-21T14:10:50.855 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.855 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.856 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:50.856 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.543123+0000", 2024-09-21T14:10:50.856 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.7940000000000008e-06 2024-09-21T14:10:50.856 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.856 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.856 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:50.856 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.543189+0000", 2024-09-21T14:10:50.857 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6118999999999995e-05 2024-09-21T14:10:50.857 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.857 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.857 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:10:50.857 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.543196+0000", 2024-09-21T14:10:50.857 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9739999999999996e-06 2024-09-21T14:10:50.858 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.858 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.858 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:50.858 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.521797+0000", 2024-09-21T14:10:50.858 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97860139199999996 2024-09-21T14:10:50.858 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.859 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.859 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:50.859 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.521938+0000", 2024-09-21T14:10:50.859 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014095199999999999 2024-09-21T14:10:50.859 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.859 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.859 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,1", 2024-09-21T14:10:50.860 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.522046+0000", 2024-09-21T14:10:50.860 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000107784 2024-09-21T14:10:50.860 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.860 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.860 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:50.860 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.526615+0000", 2024-09-21T14:10:50.861 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0045687499999999999 2024-09-21T14:10:50.861 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.861 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.861 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:50.861 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.526902+0000", 2024-09-21T14:10:50.861 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00028717199999999998 2024-09-21T14:10:50.861 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.862 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.862 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:10:50.862 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.534619+0000", 2024-09-21T14:10:50.862 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0077169279999999996 2024-09-21T14:10:50.862 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.862 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.863 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:10:50.863 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.534662+0000", 2024-09-21T14:10:50.863 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2815000000000003e-05 2024-09-21T14:10:50.863 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.863 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.863 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:50.864 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.534701+0000", 2024-09-21T14:10:50.864 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8976000000000001e-05 2024-09-21T14:10:50.864 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:50.864 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:50.864 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:50.864 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.865 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.865 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:209253 11.6 11:65872687:::benchmark_data_smithi049_206132_object85874:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531)", 2024-09-21T14:10:50.865 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:09:52.550608+0000", 2024-09-21T14:10:50.865 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 58.175342180000001, 2024-09-21T14:10:50.865 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.98416842599999999, 2024-09-21T14:10:50.865 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:50.865 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:50.866 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:50.866 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:10:50.866 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:10:50.866 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:50.866 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 209253 2024-09-21T14:10:50.866 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.867 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:50.867 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.867 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:50.867 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.550608+0000", 2024-09-21T14:10:50.867 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:50.867 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.868 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.868 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:50.868 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.550608+0000", 2024-09-21T14:10:50.868 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:50.868 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.868 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.868 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:50.869 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.550609+0000", 2024-09-21T14:10:50.869 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1069999999999999e-06 2024-09-21T14:10:50.869 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.869 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.869 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:50.869 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.550614+0000", 2024-09-21T14:10:50.870 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6229999999999996e-06 2024-09-21T14:10:50.870 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.870 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.870 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:50.870 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.550628+0000", 2024-09-21T14:10:50.870 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4086e-05 2024-09-21T14:10:50.871 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.871 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.871 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:50.872 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:10:50.872 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.550636+0000", 2024-09-21T14:10:50.873 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9959999999999993e-06 2024-09-21T14:10:50.873 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.873 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.873 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:50.873 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.550714+0000", 2024-09-21T14:10:50.873 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8064000000000003e-05 2024-09-21T14:10:50.874 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.874 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.874 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:10:50.874 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.550725+0000", 2024-09-21T14:10:50.874 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0705e-05 2024-09-21T14:10:50.874 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.875 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.875 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:50.875 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.523932+0000", 2024-09-21T14:10:50.875 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.97320744000000003 2024-09-21T14:10:50.875 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.875 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.875 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:50.876 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.524025+0000", 2024-09-21T14:10:50.876 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2553999999999995e-05 2024-09-21T14:10:50.876 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.876 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.876 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,1", 2024-09-21T14:10:50.876 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.524100+0000", 2024-09-21T14:10:50.877 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.4836e-05 2024-09-21T14:10:50.877 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.877 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.877 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:50.877 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.526689+0000", 2024-09-21T14:10:50.877 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0025892189999999998 2024-09-21T14:10:50.877 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.877 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.878 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:50.878 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.531849+0000", 2024-09-21T14:10:50.878 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0051594989999999997 2024-09-21T14:10:50.878 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.878 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.878 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:10:50.879 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.534707+0000", 2024-09-21T14:10:50.879 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.002858251 2024-09-21T14:10:50.879 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.879 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.879 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:10:50.879 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.534745+0000", 2024-09-21T14:10:50.879 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7755e-05 2024-09-21T14:10:50.879 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.880 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.880 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:50.880 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.534777+0000", 2024-09-21T14:10:50.880 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2291e-05 2024-09-21T14:10:50.880 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:50.880 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:50.880 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:50.880 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.880 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.880 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:209262 11.6 11:6f295219:::benchmark_data_smithi049_206132_object85883:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531)", 2024-09-21T14:10:50.880 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:09:52.559844+0000", 2024-09-21T14:10:50.881 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 58.166106917999997, 2024-09-21T14:10:50.881 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.979369198, 2024-09-21T14:10:50.881 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:50.881 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:50.881 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:50.881 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:10:50.881 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:10:50.881 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:50.881 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 209262 2024-09-21T14:10:50.881 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.881 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:50.881 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.881 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:50.882 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.559844+0000", 2024-09-21T14:10:50.882 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:50.882 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.882 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.882 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:50.882 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.559844+0000", 2024-09-21T14:10:50.882 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:50.882 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.882 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.882 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:50.882 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.559845+0000", 2024-09-21T14:10:50.882 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7239999999999999e-06 2024-09-21T14:10:50.883 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.883 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.883 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:50.883 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.559851+0000", 2024-09-21T14:10:50.883 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6799999999999998e-06 2024-09-21T14:10:50.883 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.883 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.883 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:50.883 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.559869+0000", 2024-09-21T14:10:50.883 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7972999999999999e-05 2024-09-21T14:10:50.883 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.883 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.883 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:50.884 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.559878+0000", 2024-09-21T14:10:50.884 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.85e-06 2024-09-21T14:10:50.884 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.884 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.884 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:50.884 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.559955+0000", 2024-09-21T14:10:50.884 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7161000000000002e-05 2024-09-21T14:10:50.884 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.884 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.884 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:10:50.884 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.559968+0000", 2024-09-21T14:10:50.884 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2707e-05 2024-09-21T14:10:50.884 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.884 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.885 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:50.885 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.526966+0000", 2024-09-21T14:10:50.885 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.96699796500000001 2024-09-21T14:10:50.885 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.885 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.885 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:50.885 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.527034+0000", 2024-09-21T14:10:50.885 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8553000000000006e-05 2024-09-21T14:10:50.885 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.885 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.885 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,1", 2024-09-21T14:10:50.885 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.527144+0000", 2024-09-21T14:10:50.885 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010964900000000001 2024-09-21T14:10:50.885 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.886 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.886 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:50.886 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.534560+0000", 2024-09-21T14:10:50.886 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0074165079999999996 2024-09-21T14:10:50.886 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.886 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.886 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:10:50.886 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.534783+0000", 2024-09-21T14:10:50.886 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00022304599999999999 2024-09-21T14:10:50.886 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.886 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.886 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:50.887 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.539115+0000", 2024-09-21T14:10:50.887 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.004331161 2024-09-21T14:10:50.887 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.887 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.887 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:10:50.887 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.539165+0000", 2024-09-21T14:10:50.887 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9962e-05 2024-09-21T14:10:50.887 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.887 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.887 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:50.887 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.539213+0000", 2024-09-21T14:10:50.887 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8259e-05 2024-09-21T14:10:50.887 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:50.887 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:50.888 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:50.888 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.888 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.888 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:209273 11.6 11:6286fd93:::benchmark_data_smithi049_206132_object85894:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531)", 2024-09-21T14:10:50.888 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:09:52.572667+0000", 2024-09-21T14:10:50.888 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 58.153283256000002, 2024-09-21T14:10:50.888 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.96666920599999995, 2024-09-21T14:10:50.888 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:50.888 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:50.888 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:50.888 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:10:50.888 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:10:50.889 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:50.889 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 209273 2024-09-21T14:10:50.889 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.889 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:50.889 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.889 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:50.889 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.572667+0000", 2024-09-21T14:10:50.889 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:50.889 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.889 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.889 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:50.889 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.572667+0000", 2024-09-21T14:10:50.889 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:50.889 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.889 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.890 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:50.890 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.572669+0000", 2024-09-21T14:10:50.890 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.26e-06 2024-09-21T14:10:50.890 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.890 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.890 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:50.890 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.572674+0000", 2024-09-21T14:10:50.890 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8860000000000001e-06 2024-09-21T14:10:50.890 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.890 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.890 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:50.890 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.572690+0000", 2024-09-21T14:10:50.891 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5322e-05 2024-09-21T14:10:50.891 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.891 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.891 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:50.891 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.572699+0000", 2024-09-21T14:10:50.891 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.7730000000000005e-06 2024-09-21T14:10:50.891 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.891 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.891 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:50.891 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.572775+0000", 2024-09-21T14:10:50.891 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.6384000000000006e-05 2024-09-21T14:10:50.891 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.891 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.892 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:10:50.892 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.572785+0000", 2024-09-21T14:10:50.892 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0259e-05 2024-09-21T14:10:50.892 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.892 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.892 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:50.892 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.529411+0000", 2024-09-21T14:10:50.892 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.95662623499999999 2024-09-21T14:10:50.893 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.893 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.893 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:50.893 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.529533+0000", 2024-09-21T14:10:50.893 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012112600000000001 2024-09-21T14:10:50.893 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.893 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.893 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,1", 2024-09-21T14:10:50.893 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.529627+0000", 2024-09-21T14:10:50.893 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4350000000000003e-05 2024-09-21T14:10:50.893 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.893 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.894 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:50.894 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.534600+0000", 2024-09-21T14:10:50.894 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0049734469999999998 2024-09-21T14:10:50.894 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.894 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.894 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:10:50.894 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.534788+0000", 2024-09-21T14:10:50.894 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000187367 2024-09-21T14:10:50.894 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.894 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.894 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:50.894 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.539250+0000", 2024-09-21T14:10:50.894 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0044619000000000004 2024-09-21T14:10:50.894 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.895 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.895 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:10:50.895 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.539300+0000", 2024-09-21T14:10:50.895 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9928e-05 2024-09-21T14:10:50.895 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.895 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.895 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:50.895 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.539337+0000", 2024-09-21T14:10:50.895 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.6968999999999998e-05 2024-09-21T14:10:50.895 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:50.895 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:50.896 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:50.896 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.896 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.896 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:209279 11.2 11:47fbbcd4:::benchmark_data_smithi049_206132_object85900:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531)", 2024-09-21T14:10:50.896 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:09:52.581128+0000", 2024-09-21T14:10:50.896 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 58.144822484999999, 2024-09-21T14:10:50.896 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.96403417999999996, 2024-09-21T14:10:50.896 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:50.896 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:50.896 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:50.896 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:10:50.896 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:10:50.897 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:50.897 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 209279 2024-09-21T14:10:50.897 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.897 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:50.897 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.897 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:50.897 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.581128+0000", 2024-09-21T14:10:50.897 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:50.897 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.897 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.897 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:50.898 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.581128+0000", 2024-09-21T14:10:50.898 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:50.898 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.898 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.898 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:50.898 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.581130+0000", 2024-09-21T14:10:50.898 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.742e-06 2024-09-21T14:10:50.898 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.898 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.898 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:50.898 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.581135+0000", 2024-09-21T14:10:50.898 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0860000000000001e-06 2024-09-21T14:10:50.899 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.899 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.899 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:50.899 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.581150+0000", 2024-09-21T14:10:50.899 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4603000000000001e-05 2024-09-21T14:10:50.899 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.899 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.899 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:50.899 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.581157+0000", 2024-09-21T14:10:50.899 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7670000000000003e-06 2024-09-21T14:10:50.899 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.899 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.899 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:50.900 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.581212+0000", 2024-09-21T14:10:50.900 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4469000000000003e-05 2024-09-21T14:10:50.900 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.900 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.900 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:10:50.900 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.581218+0000", 2024-09-21T14:10:50.900 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5180000000000001e-06 2024-09-21T14:10:50.900 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.900 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.900 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:50.900 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.529333+0000", 2024-09-21T14:10:50.900 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.94811520800000004 2024-09-21T14:10:50.900 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.901 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.901 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:50.901 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.529435+0000", 2024-09-21T14:10:50.901 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00010154600000000001 2024-09-21T14:10:50.901 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.901 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.901 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:10:50.901 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.529565+0000", 2024-09-21T14:10:50.901 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000130388 2024-09-21T14:10:50.901 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.901 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.901 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:10:50.901 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.536987+0000", 2024-09-21T14:10:50.901 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0074214449999999996 2024-09-21T14:10:50.902 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.902 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.902 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:50.902 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.537025+0000", 2024-09-21T14:10:50.902 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7673000000000003e-05 2024-09-21T14:10:50.902 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.902 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.902 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:50.902 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.545054+0000", 2024-09-21T14:10:50.902 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0080296470000000009 2024-09-21T14:10:50.902 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.902 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.903 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:10:50.903 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.545105+0000", 2024-09-21T14:10:50.903 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0503000000000001e-05 2024-09-21T14:10:50.903 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.903 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.903 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:50.903 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.545162+0000", 2024-09-21T14:10:50.903 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7584999999999998e-05 2024-09-21T14:10:50.903 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:50.903 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:50.903 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:50.903 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.903 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.903 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:209280 11.6 11:6fe9c422:::benchmark_data_smithi049_206132_object85901:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531)", 2024-09-21T14:10:50.903 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:09:52.582304+0000", 2024-09-21T14:10:50.904 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 58.143646414999999, 2024-09-21T14:10:50.904 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.95715086999999999, 2024-09-21T14:10:50.904 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:50.904 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:50.904 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:50.904 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:10:50.904 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:10:50.904 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:50.904 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 209280 2024-09-21T14:10:50.904 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.904 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:50.904 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.904 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:50.904 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.582304+0000", 2024-09-21T14:10:50.905 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:50.905 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.905 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.905 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:50.905 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.582304+0000", 2024-09-21T14:10:50.905 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:50.905 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.905 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.905 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:50.905 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.582305+0000", 2024-09-21T14:10:50.905 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2699999999999999e-07 2024-09-21T14:10:50.905 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.905 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.905 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:50.905 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.582308+0000", 2024-09-21T14:10:50.906 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.3009999999999998e-06 2024-09-21T14:10:50.906 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.906 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.906 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:50.906 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.582319+0000", 2024-09-21T14:10:50.906 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0369e-05 2024-09-21T14:10:50.906 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.906 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.906 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:50.906 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.582324+0000", 2024-09-21T14:10:50.906 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0180000000000001e-06 2024-09-21T14:10:50.906 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.906 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.907 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:50.907 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.582370+0000", 2024-09-21T14:10:50.907 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6202000000000003e-05 2024-09-21T14:10:50.907 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.907 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.907 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:10:50.907 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.582376+0000", 2024-09-21T14:10:50.907 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3980000000000004e-06 2024-09-21T14:10:50.907 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.907 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.907 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:50.907 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.531880+0000", 2024-09-21T14:10:50.907 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.94950334700000005 2024-09-21T14:10:50.907 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.908 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.908 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:50.908 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.532008+0000", 2024-09-21T14:10:50.908 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000128121 2024-09-21T14:10:50.908 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.908 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.908 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,1", 2024-09-21T14:10:50.908 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.532112+0000", 2024-09-21T14:10:50.908 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000104472 2024-09-21T14:10:50.908 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.908 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.908 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:10:50.908 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.535182+0000", 2024-09-21T14:10:50.909 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0030703129999999999 2024-09-21T14:10:50.909 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.909 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.909 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:50.909 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.536064+0000", 2024-09-21T14:10:50.909 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00088110599999999999 2024-09-21T14:10:50.909 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.909 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.909 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:50.909 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.539374+0000", 2024-09-21T14:10:50.909 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0033099789999999998 2024-09-21T14:10:50.909 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.909 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.909 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:10:50.910 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.539420+0000", 2024-09-21T14:10:50.910 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6152999999999997e-05 2024-09-21T14:10:50.910 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.910 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.910 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:50.910 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.539455+0000", 2024-09-21T14:10:50.910 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5364000000000002e-05 2024-09-21T14:10:50.910 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:50.910 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:50.910 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:50.910 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.910 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.911 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:209282 11.2 11:4a63f481:::benchmark_data_smithi049_206132_object85903:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e531)", 2024-09-21T14:10:50.911 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:09:52.584867+0000", 2024-09-21T14:10:50.911 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 58.141083537999997, 2024-09-21T14:10:50.911 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.960401379, 2024-09-21T14:10:50.911 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:50.911 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:50.911 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:50.911 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:10:50.911 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:10:50.911 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:50.911 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 209282 2024-09-21T14:10:50.911 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.911 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:50.912 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.912 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:50.912 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.584867+0000", 2024-09-21T14:10:50.912 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:50.912 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.912 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.912 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:50.912 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.584867+0000", 2024-09-21T14:10:50.912 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:50.912 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.912 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.912 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:50.912 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.584868+0000", 2024-09-21T14:10:50.913 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9800000000000003e-07 2024-09-21T14:10:50.913 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.913 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.913 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:50.913 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.584871+0000", 2024-09-21T14:10:50.913 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4769999999999999e-06 2024-09-21T14:10:50.913 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.913 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.913 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:50.913 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.584881+0000", 2024-09-21T14:10:50.913 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9310000000000003e-06 2024-09-21T14:10:50.913 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.913 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.913 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:50.914 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.584886+0000", 2024-09-21T14:10:50.914 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8980000000000004e-06 2024-09-21T14:10:50.914 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.914 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.914 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:50.914 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.584930+0000", 2024-09-21T14:10:50.914 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4252000000000003e-05 2024-09-21T14:10:50.914 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.914 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.914 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for peered", 2024-09-21T14:10:50.914 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:52.584936+0000", 2024-09-21T14:10:50.914 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.8799999999999996e-06 2024-09-21T14:10:50.914 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.915 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.915 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:50.915 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.532009+0000", 2024-09-21T14:10:50.915 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.94707254699999999 2024-09-21T14:10:50.915 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.915 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.915 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:50.915 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.532104+0000", 2024-09-21T14:10:50.915 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.5728000000000002e-05 2024-09-21T14:10:50.915 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.915 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.915 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:10:50.915 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.532203+0000", 2024-09-21T14:10:50.916 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.9056999999999997e-05 2024-09-21T14:10:50.916 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.916 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.916 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:10:50.916 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.537035+0000", 2024-09-21T14:10:50.916 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0048314969999999997 2024-09-21T14:10:50.916 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.916 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.916 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:50.916 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.537070+0000", 2024-09-21T14:10:50.916 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.4666999999999999e-05 2024-09-21T14:10:50.916 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.916 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.916 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:50.916 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.545196+0000", 2024-09-21T14:10:50.916 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0081266729999999992 2024-09-21T14:10:50.917 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.917 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.917 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:10:50.917 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.545237+0000", 2024-09-21T14:10:50.917 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0917000000000001e-05 2024-09-21T14:10:50.917 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:50.917 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:50.917 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:50.917 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:53.545268+0000", 2024-09-21T14:10:50.917 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1157e-05 2024-09-21T14:10:50.917 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:50.917 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:50.917 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:50.917 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:50.917 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:50.918 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T14:10:51.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:51 smithi187 ceph-mon[33032]: 5.1f scrub starts 2024-09-21T14:10:51.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:51 smithi187 ceph-mon[33032]: 5.1f scrub ok 2024-09-21T14:10:51.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:51 smithi187 ceph-mon[33032]: pgmap v1128: 152 pgs: 152 active+clean; 583 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 0 B/s rd, 0 op/s; 0 B/s, 2 objects/s recovering 2024-09-21T14:10:51.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:51 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:10:51.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:51 smithi187 ceph-mon[33032]: 6.14 scrub starts 2024-09-21T14:10:51.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:51 smithi187 ceph-mon[33032]: 6.14 scrub ok 2024-09-21T14:10:51.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:51 smithi187 ceph-mon[33032]: 5.1e scrub starts 2024-09-21T14:10:51.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:51 smithi187 ceph-mon[33032]: 5.1e scrub ok 2024-09-21T14:10:51.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:51 smithi187 ceph-mon[33032]: 6.15 scrub starts 2024-09-21T14:10:51.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:51 smithi187 ceph-mon[33032]: 6.15 scrub ok 2024-09-21T14:10:51.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:51 smithi187 ceph-mon[33032]: 4.7 scrub starts 2024-09-21T14:10:51.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:51 smithi187 ceph-mon[33032]: 4.7 scrub ok 2024-09-21T14:10:51.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:51 smithi049 ceph-mon[40489]: 5.1f scrub starts 2024-09-21T14:10:51.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:51 smithi049 ceph-mon[40489]: 5.1f scrub ok 2024-09-21T14:10:51.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:51 smithi049 ceph-mon[40489]: pgmap v1128: 152 pgs: 152 active+clean; 583 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 0 B/s rd, 0 op/s; 0 B/s, 2 objects/s recovering 2024-09-21T14:10:51.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:51 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:10:51.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:51 smithi049 ceph-mon[40489]: 6.14 scrub starts 2024-09-21T14:10:51.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:51 smithi049 ceph-mon[40489]: 6.14 scrub ok 2024-09-21T14:10:51.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:51 smithi049 ceph-mon[40489]: 5.1e scrub starts 2024-09-21T14:10:51.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:51 smithi049 ceph-mon[40489]: 5.1e scrub ok 2024-09-21T14:10:51.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:51 smithi049 ceph-mon[40489]: 6.15 scrub starts 2024-09-21T14:10:51.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:51 smithi049 ceph-mon[40489]: 6.15 scrub ok 2024-09-21T14:10:51.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:51 smithi049 ceph-mon[40489]: 4.7 scrub starts 2024-09-21T14:10:51.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:51 smithi049 ceph-mon[40489]: 4.7 scrub ok 2024-09-21T14:10:51.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:51 smithi049 ceph-mon[30080]: 5.1f scrub starts 2024-09-21T14:10:51.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:51 smithi049 ceph-mon[30080]: 5.1f scrub ok 2024-09-21T14:10:51.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:51 smithi049 ceph-mon[30080]: pgmap v1128: 152 pgs: 152 active+clean; 583 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 0 B/s rd, 0 op/s; 0 B/s, 2 objects/s recovering 2024-09-21T14:10:51.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:51 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:10:51.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:51 smithi049 ceph-mon[30080]: 6.14 scrub starts 2024-09-21T14:10:51.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:51 smithi049 ceph-mon[30080]: 6.14 scrub ok 2024-09-21T14:10:51.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:51 smithi049 ceph-mon[30080]: 5.1e scrub starts 2024-09-21T14:10:51.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:51 smithi049 ceph-mon[30080]: 5.1e scrub ok 2024-09-21T14:10:51.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:51 smithi049 ceph-mon[30080]: 6.15 scrub starts 2024-09-21T14:10:51.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:51 smithi049 ceph-mon[30080]: 6.15 scrub ok 2024-09-21T14:10:51.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:51 smithi049 ceph-mon[30080]: 4.7 scrub starts 2024-09-21T14:10:51.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:51 smithi049 ceph-mon[30080]: 4.7 scrub ok 2024-09-21T14:10:52.216 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T14:10:52.433 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:52 smithi049 ceph-mon[40489]: 1.12 scrub starts 2024-09-21T14:10:52.433 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:52 smithi049 ceph-mon[40489]: 1.12 scrub ok 2024-09-21T14:10:52.433 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:52 smithi049 ceph-mon[40489]: 3.14 scrub starts 2024-09-21T14:10:52.433 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:52 smithi049 ceph-mon[40489]: 3.14 scrub ok 2024-09-21T14:10:52.434 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:52 smithi049 ceph-mon[40489]: 3.8 scrub starts 2024-09-21T14:10:52.434 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:52 smithi049 ceph-mon[40489]: 3.8 scrub ok 2024-09-21T14:10:52.434 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:52 smithi049 ceph-mon[30080]: 1.12 scrub starts 2024-09-21T14:10:52.434 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:52 smithi049 ceph-mon[30080]: 1.12 scrub ok 2024-09-21T14:10:52.434 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:52 smithi049 ceph-mon[30080]: 3.14 scrub starts 2024-09-21T14:10:52.434 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:52 smithi049 ceph-mon[30080]: 3.14 scrub ok 2024-09-21T14:10:52.434 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:52 smithi049 ceph-mon[30080]: 3.8 scrub starts 2024-09-21T14:10:52.434 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:52 smithi049 ceph-mon[30080]: 3.8 scrub ok 2024-09-21T14:10:52.558 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:10:52.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:52 smithi187 ceph-mon[33032]: 1.12 scrub starts 2024-09-21T14:10:52.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:52 smithi187 ceph-mon[33032]: 1.12 scrub ok 2024-09-21T14:10:52.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:52 smithi187 ceph-mon[33032]: 3.14 scrub starts 2024-09-21T14:10:52.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:52 smithi187 ceph-mon[33032]: 3.14 scrub ok 2024-09-21T14:10:52.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:52 smithi187 ceph-mon[33032]: 3.8 scrub starts 2024-09-21T14:10:52.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:52 smithi187 ceph-mon[33032]: 3.8 scrub ok 2024-09-21T14:10:53.039 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T14:10:53.039 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-09-21T14:10:53.040 INFO:teuthology.orchestra.run.smithi049.stdout: "num_ops": 0 2024-09-21T14:10:53.040 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T14:10:53.348 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.1 to 1 (10000) 2024-09-21T14:10:53.666 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:53 smithi049 ceph-mon[30080]: pgmap v1129: 152 pgs: 152 active+clean; 583 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 443 B/s rd, 1 op/s; 0 B/s, 2 objects/s recovering 2024-09-21T14:10:53.666 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:53 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2656575368' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T14:10:53.666 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:53 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T14:10:53.666 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:53 smithi049 ceph-mon[30080]: 3.1f scrub starts 2024-09-21T14:10:53.666 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:53 smithi049 ceph-mon[30080]: 3.1f scrub ok 2024-09-21T14:10:53.666 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:53 smithi049 ceph-mon[40489]: pgmap v1129: 152 pgs: 152 active+clean; 583 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 443 B/s rd, 1 op/s; 0 B/s, 2 objects/s recovering 2024-09-21T14:10:53.667 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:53 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2656575368' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T14:10:53.667 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:53 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T14:10:53.667 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:53 smithi049 ceph-mon[40489]: 3.1f scrub starts 2024-09-21T14:10:53.667 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:53 smithi049 ceph-mon[40489]: 3.1f scrub ok 2024-09-21T14:10:53.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:53 smithi187 ceph-mon[33032]: pgmap v1129: 152 pgs: 152 active+clean; 583 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 443 B/s rd, 1 op/s; 0 B/s, 2 objects/s recovering 2024-09-21T14:10:53.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:53 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2656575368' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T14:10:53.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:53 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T14:10:53.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:53 smithi187 ceph-mon[33032]: 3.1f scrub starts 2024-09-21T14:10:53.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:53 smithi187 ceph-mon[33032]: 3.1f scrub ok 2024-09-21T14:10:53.696 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T14:10:53.988 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:10:54.482 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:54 smithi049 ceph-mon[30080]: 1.d scrub starts 2024-09-21T14:10:54.482 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:54 smithi049 ceph-mon[30080]: 1.d scrub ok 2024-09-21T14:10:54.482 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:54 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T14:10:54.482 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:54 smithi049 ceph-mon[30080]: osdmap e559: 8 total, 8 up, 8 in 2024-09-21T14:10:54.482 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:54 smithi049 ceph-mon[30080]: 4.11 scrub starts 2024-09-21T14:10:54.482 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:54 smithi049 ceph-mon[30080]: 4.11 scrub ok 2024-09-21T14:10:54.482 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:54 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T14:10:54.482 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:54 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T14:10:54.482 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:54 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T14:10:54.483 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:54 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:10:54.483 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:54 smithi049 ceph-mon[40489]: 1.d scrub starts 2024-09-21T14:10:54.483 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:54 smithi049 ceph-mon[40489]: 1.d scrub ok 2024-09-21T14:10:54.483 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:54 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T14:10:54.483 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:54 smithi049 ceph-mon[40489]: osdmap e559: 8 total, 8 up, 8 in 2024-09-21T14:10:54.483 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:54 smithi049 ceph-mon[40489]: 4.11 scrub starts 2024-09-21T14:10:54.483 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:54 smithi049 ceph-mon[40489]: 4.11 scrub ok 2024-09-21T14:10:54.483 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:54 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T14:10:54.483 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:54 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T14:10:54.483 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:54 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T14:10:54.483 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:54 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:10:54.508 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 1 1 2024-09-21T14:10:54.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:54 smithi187 ceph-mon[33032]: 1.d scrub starts 2024-09-21T14:10:54.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:54 smithi187 ceph-mon[33032]: 1.d scrub ok 2024-09-21T14:10:54.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:54 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T14:10:54.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:54 smithi187 ceph-mon[33032]: osdmap e559: 8 total, 8 up, 8 in 2024-09-21T14:10:54.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:54 smithi187 ceph-mon[33032]: 4.11 scrub starts 2024-09-21T14:10:54.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:54 smithi187 ceph-mon[33032]: 4.11 scrub ok 2024-09-21T14:10:54.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:54 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T14:10:54.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:54 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T14:10:54.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:54 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T14:10:54.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:54 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:10:55.049 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T14:10:55.049 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [], 2024-09-21T14:10:55.049 INFO:teuthology.orchestra.run.smithi049.stdout: "complaint_time": 30, 2024-09-21T14:10:55.049 INFO:teuthology.orchestra.run.smithi049.stdout: "num_blocked_ops": 0 2024-09-21T14:10:55.049 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T14:10:55.284 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:10:55.675 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:55 smithi049 ceph-mon[40489]: 3.11 scrub starts 2024-09-21T14:10:55.675 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:55 smithi049 ceph-mon[40489]: 3.11 scrub ok 2024-09-21T14:10:55.675 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:55 smithi049 ceph-mon[40489]: pgmap v1131: 152 pgs: 152 active+clean; 583 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 0 B/s rd, 0 op/s; 0 B/s, 2 objects/s recovering 2024-09-21T14:10:55.675 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:55 smithi049 ceph-mon[40489]: osdmap e560: 8 total, 8 up, 8 in 2024-09-21T14:10:55.676 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:55 smithi049 ceph-mon[30080]: 3.11 scrub starts 2024-09-21T14:10:55.676 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:55 smithi049 ceph-mon[30080]: 3.11 scrub ok 2024-09-21T14:10:55.676 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:55 smithi049 ceph-mon[30080]: pgmap v1131: 152 pgs: 152 active+clean; 583 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 0 B/s rd, 0 op/s; 0 B/s, 2 objects/s recovering 2024-09-21T14:10:55.676 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:55 smithi049 ceph-mon[30080]: osdmap e560: 8 total, 8 up, 8 in 2024-09-21T14:10:55.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:55 smithi187 ceph-mon[33032]: 3.11 scrub starts 2024-09-21T14:10:55.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:55 smithi187 ceph-mon[33032]: 3.11 scrub ok 2024-09-21T14:10:55.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:55 smithi187 ceph-mon[33032]: pgmap v1131: 152 pgs: 152 active+clean; 583 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 0 B/s rd, 0 op/s; 0 B/s, 2 objects/s recovering 2024-09-21T14:10:55.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:55 smithi187 ceph-mon[33032]: osdmap e560: 8 total, 8 up, 8 in 2024-09-21T14:10:56.213 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T14:10:56.282 INFO:teuthology.orchestra.run.smithi049.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-21T14:10:56.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:56 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:10:56.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:56 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:10:56.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:56 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/492461924' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-21T14:10:56.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:56 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-21T14:10:56.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:56 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-21T14:10:56.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:56 smithi049 ceph-mon[40489]: osdmap e561: 8 total, 8 up, 8 in 2024-09-21T14:10:56.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:56 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:10:56.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:56 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:10:56.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:56 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/492461924' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-21T14:10:56.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:56 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-21T14:10:56.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:56 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-21T14:10:56.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:56 smithi049 ceph-mon[30080]: osdmap e561: 8 total, 8 up, 8 in 2024-09-21T14:10:56.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:56 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:10:56.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:56 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:10:56.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:56 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/492461924' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-21T14:10:56.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:56 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-21T14:10:56.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:56 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-21T14:10:56.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:56 smithi187 ceph-mon[33032]: osdmap e561: 8 total, 8 up, 8 in 2024-09-21T14:10:57.014 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:10:57.025 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd reweight 4 1 2024-09-21T14:10:57.569 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:57 smithi049 ceph-mon[40489]: 4.c scrub starts 2024-09-21T14:10:57.569 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:57 smithi049 ceph-mon[40489]: 4.c scrub ok 2024-09-21T14:10:57.569 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:57 smithi049 ceph-mon[40489]: pgmap v1133: 152 pgs: 152 active+clean; 583 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.9 KiB/s rd, 1 op/s; 38 B/s, 2 objects/s recovering 2024-09-21T14:10:57.569 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:57 smithi049 ceph-mon[40489]: 6.1c scrub starts 2024-09-21T14:10:57.569 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:57 smithi049 ceph-mon[40489]: 6.1c scrub ok 2024-09-21T14:10:57.569 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:57 smithi049 ceph-mon[40489]: 4.1d scrub starts 2024-09-21T14:10:57.569 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:57 smithi049 ceph-mon[40489]: 4.1d scrub ok 2024-09-21T14:10:57.570 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:57 smithi049 ceph-mon[30080]: 4.c scrub starts 2024-09-21T14:10:57.570 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:57 smithi049 ceph-mon[30080]: 4.c scrub ok 2024-09-21T14:10:57.570 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:57 smithi049 ceph-mon[30080]: pgmap v1133: 152 pgs: 152 active+clean; 583 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.9 KiB/s rd, 1 op/s; 38 B/s, 2 objects/s recovering 2024-09-21T14:10:57.570 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:57 smithi049 ceph-mon[30080]: 6.1c scrub starts 2024-09-21T14:10:57.570 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:57 smithi049 ceph-mon[30080]: 6.1c scrub ok 2024-09-21T14:10:57.570 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:57 smithi049 ceph-mon[30080]: 4.1d scrub starts 2024-09-21T14:10:57.570 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:57 smithi049 ceph-mon[30080]: 4.1d scrub ok 2024-09-21T14:10:57.574 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-09-21T14:10:57.574 INFO:teuthology.orchestra.run.smithi049.stdout: "size": 20, 2024-09-21T14:10:57.574 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 600, 2024-09-21T14:10:57.574 INFO:teuthology.orchestra.run.smithi049.stdout: "ops": [ 2024-09-21T14:10:57.575 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.575 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.16068.0:208862 10.d e441/434 10:b40252c3:::benchmark_data_smithi049_176811_object81285:head v 441'13126, mlcod=441'13122)", 2024-09-21T14:10:57.575 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:05:54.047328+0000", 2024-09-21T14:10:57.575 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 303.52568709299999, 2024-09-21T14:10:57.575 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.31640931900000002, 2024-09-21T14:10:57.575 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:57.575 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:57.575 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:57.575 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:57.575 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.575 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:57.575 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047328+0000", 2024-09-21T14:10:57.575 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:57.575 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.576 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.576 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:57.576 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047328+0000", 2024-09-21T14:10:57.576 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:57.576 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.576 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.576 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:57.576 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047329+0000", 2024-09-21T14:10:57.576 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0410000000000001e-06 2024-09-21T14:10:57.576 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.576 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.576 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:57.576 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047338+0000", 2024-09-21T14:10:57.576 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6160000000000002e-06 2024-09-21T14:10:57.577 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.577 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.577 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:57.577 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047348+0000", 2024-09-21T14:10:57.577 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0688e-05 2024-09-21T14:10:57.577 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.577 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.577 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:57.577 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047356+0000", 2024-09-21T14:10:57.577 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3350000000000001e-06 2024-09-21T14:10:57.577 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.577 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.577 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:57.577 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047399+0000", 2024-09-21T14:10:57.578 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.2833000000000001e-05 2024-09-21T14:10:57.578 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.578 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.578 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:57.578 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047415+0000", 2024-09-21T14:10:57.578 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6662000000000001e-05 2024-09-21T14:10:57.578 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.578 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.578 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:10:57.578 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.363700+0000", 2024-09-21T14:10:57.578 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.31628446799999999 2024-09-21T14:10:57.578 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.578 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.578 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:57.579 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.363737+0000", 2024-09-21T14:10:57.579 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.7676000000000003e-05 2024-09-21T14:10:57.579 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:57.579 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:57.579 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:57.579 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.579 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.579 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_repop(client.16068.0:208866 10.3 e441/381 10:c8242cfb:::benchmark_data_smithi049_176811_object81289:head v 441'12975, mlcod=441'12973)", 2024-09-21T14:10:57.579 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:05:54.047688+0000", 2024-09-21T14:10:57.579 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 303.52532754100002, 2024-09-21T14:10:57.579 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.31391443499999999, 2024-09-21T14:10:57.579 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:57.579 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:57.579 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:57.579 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:57.580 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.580 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:57.580 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047688+0000", 2024-09-21T14:10:57.580 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:57.580 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.580 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.580 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:57.580 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047688+0000", 2024-09-21T14:10:57.580 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:57.580 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.580 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.580 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:57.580 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047688+0000", 2024-09-21T14:10:57.580 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9999999999999997e-07 2024-09-21T14:10:57.581 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.581 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.581 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:57.581 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.047693+0000", 2024-09-21T14:10:57.581 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7759999999999997e-06 2024-09-21T14:10:57.581 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.581 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.581 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:57.581 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.357055+0000", 2024-09-21T14:10:57.581 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.309361422 2024-09-21T14:10:57.581 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.581 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.581 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:57.581 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.357070+0000", 2024-09-21T14:10:57.582 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5311e-05 2024-09-21T14:10:57.582 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.582 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.582 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:57.582 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.357133+0000", 2024-09-21T14:10:57.582 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.2611000000000001e-05 2024-09-21T14:10:57.582 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.582 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.582 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:57.582 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.357154+0000", 2024-09-21T14:10:57.582 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.1376e-05 2024-09-21T14:10:57.582 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.582 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.582 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:10:57.582 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.361561+0000", 2024-09-21T14:10:57.583 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0044073599999999999 2024-09-21T14:10:57.583 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.583 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.583 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:57.583 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:05:54.361602+0000", 2024-09-21T14:10:57.583 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0878999999999999e-05 2024-09-21T14:10:57.583 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:57.583 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:57.583 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:57.583 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.583 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.583 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:61790 11.b 11:d012fa8a:::benchmark_data_smithi049_206132_object61789:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e476)", 2024-09-21T14:10:57.583 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:07:27.526621+0000", 2024-09-21T14:10:57.583 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 210.04639410300001, 2024-09-21T14:10:57.584 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.21106169299999999, 2024-09-21T14:10:57.584 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:57.584 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:57.584 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:57.584 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:10:57.584 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:10:57.584 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:57.584 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 61790 2024-09-21T14:10:57.584 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.584 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:57.584 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.584 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:57.584 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.526621+0000", 2024-09-21T14:10:57.584 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:57.584 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.585 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.585 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:57.585 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.526621+0000", 2024-09-21T14:10:57.585 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:57.585 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.585 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.585 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:57.585 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.526623+0000", 2024-09-21T14:10:57.585 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.725e-06 2024-09-21T14:10:57.585 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.585 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.585 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:57.585 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.526655+0000", 2024-09-21T14:10:57.585 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.1717000000000001e-05 2024-09-21T14:10:57.585 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.586 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.586 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:57.586 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.526667+0000", 2024-09-21T14:10:57.586 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2121e-05 2024-09-21T14:10:57.586 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.586 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.586 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:57.586 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.526674+0000", 2024-09-21T14:10:57.586 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.7330000000000003e-06 2024-09-21T14:10:57.586 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.586 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.586 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:57.586 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.527240+0000", 2024-09-21T14:10:57.586 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00056578599999999998 2024-09-21T14:10:57.587 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.587 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.587 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:57.587 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.527424+0000", 2024-09-21T14:10:57.587 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00018419199999999999 2024-09-21T14:10:57.587 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.587 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.587 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,1", 2024-09-21T14:10:57.587 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.527566+0000", 2024-09-21T14:10:57.587 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014196599999999999 2024-09-21T14:10:57.587 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.587 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.587 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:10:57.587 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.532651+0000", 2024-09-21T14:10:57.588 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0050843640000000001 2024-09-21T14:10:57.588 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.588 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.588 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:57.588 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.533057+0000", 2024-09-21T14:10:57.589 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00040586600000000001 2024-09-21T14:10:57.589 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.589 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.589 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:57.589 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.737591+0000", 2024-09-21T14:10:57.589 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.204534191 2024-09-21T14:10:57.589 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.589 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.589 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:10:57.589 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.737643+0000", 2024-09-21T14:10:57.589 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2555000000000001e-05 2024-09-21T14:10:57.589 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.589 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.589 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:57.590 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.737683+0000", 2024-09-21T14:10:57.590 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9477000000000002e-05 2024-09-21T14:10:57.590 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:57.590 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:57.590 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:57.590 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.590 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.590 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:61806 11.b 11:d5ce9cd2:::benchmark_data_smithi049_206132_object61805:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e476)", 2024-09-21T14:10:57.590 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:07:27.535767+0000", 2024-09-21T14:10:57.590 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 210.03724780100001, 2024-09-21T14:10:57.590 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.206897624, 2024-09-21T14:10:57.590 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:57.590 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:57.591 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:57.591 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:10:57.591 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:10:57.591 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:57.591 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 61806 2024-09-21T14:10:57.591 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.591 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:57.591 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.591 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:57.591 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.535767+0000", 2024-09-21T14:10:57.591 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:57.591 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.591 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.591 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:57.592 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.535767+0000", 2024-09-21T14:10:57.592 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:57.592 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.592 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.592 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:57.592 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.535769+0000", 2024-09-21T14:10:57.592 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2640000000000001e-06 2024-09-21T14:10:57.592 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.592 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.592 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:57.592 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.535792+0000", 2024-09-21T14:10:57.592 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.3393000000000001e-05 2024-09-21T14:10:57.592 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.593 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.593 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:57.593 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.535808+0000", 2024-09-21T14:10:57.593 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5736e-05 2024-09-21T14:10:57.593 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.593 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.593 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:57.593 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.535817+0000", 2024-09-21T14:10:57.593 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4690000000000003e-06 2024-09-21T14:10:57.593 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.593 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.593 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:57.593 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.535874+0000", 2024-09-21T14:10:57.594 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6696000000000001e-05 2024-09-21T14:10:57.594 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.594 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.594 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:57.594 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.535985+0000", 2024-09-21T14:10:57.594 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000110908 2024-09-21T14:10:57.594 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.594 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.594 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,1", 2024-09-21T14:10:57.594 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.536105+0000", 2024-09-21T14:10:57.594 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000119753 2024-09-21T14:10:57.594 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.594 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.595 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:10:57.595 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.537905+0000", 2024-09-21T14:10:57.595 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001800779 2024-09-21T14:10:57.595 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.595 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.595 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:57.595 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.538385+0000", 2024-09-21T14:10:57.595 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00047917499999999998 2024-09-21T14:10:57.595 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.595 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.595 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:57.595 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.742566+0000", 2024-09-21T14:10:57.595 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.20418138899999999 2024-09-21T14:10:57.595 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.596 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.596 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:10:57.596 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.742621+0000", 2024-09-21T14:10:57.596 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4951999999999998e-05 2024-09-21T14:10:57.596 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.596 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.596 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:57.596 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.742665+0000", 2024-09-21T14:10:57.596 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4110000000000003e-05 2024-09-21T14:10:57.596 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:57.596 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:57.596 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:57.596 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.596 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.596 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:61815 11.9 11:9bfc68e2:::benchmark_data_smithi049_206132_object61814:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e476)", 2024-09-21T14:10:57.597 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:07:27.542890+0000", 2024-09-21T14:10:57.597 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 210.03012497099999, 2024-09-21T14:10:57.597 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.19371184399999999, 2024-09-21T14:10:57.597 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:57.597 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:57.597 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:57.597 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:10:57.597 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:10:57.597 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:57.597 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 61815 2024-09-21T14:10:57.597 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.597 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:57.597 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.597 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:57.598 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.542890+0000", 2024-09-21T14:10:57.598 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:57.598 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.598 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.598 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:57.598 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.542890+0000", 2024-09-21T14:10:57.598 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:57.598 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.598 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.598 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:57.598 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.542891+0000", 2024-09-21T14:10:57.598 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4300000000000001e-07 2024-09-21T14:10:57.598 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.598 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.599 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:57.599 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.542910+0000", 2024-09-21T14:10:57.599 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9029e-05 2024-09-21T14:10:57.599 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.599 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.599 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:57.599 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.542921+0000", 2024-09-21T14:10:57.599 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1037e-05 2024-09-21T14:10:57.599 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.599 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.599 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:57.599 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.542927+0000", 2024-09-21T14:10:57.599 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6879999999999996e-06 2024-09-21T14:10:57.599 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.600 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.600 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:57.600 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.542966+0000", 2024-09-21T14:10:57.600 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.8954e-05 2024-09-21T14:10:57.600 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.600 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.600 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:57.600 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.543073+0000", 2024-09-21T14:10:57.600 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000107032 2024-09-21T14:10:57.600 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.600 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.600 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,6", 2024-09-21T14:10:57.600 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.543165+0000", 2024-09-21T14:10:57.600 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.1959000000000005e-05 2024-09-21T14:10:57.601 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.601 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.601 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:10:57.601 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.544789+0000", 2024-09-21T14:10:57.601 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0016236499999999999 2024-09-21T14:10:57.601 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.601 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.601 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:57.601 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.545917+0000", 2024-09-21T14:10:57.601 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0011286169999999999 2024-09-21T14:10:57.601 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.601 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.601 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:57.601 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.736484+0000", 2024-09-21T14:10:57.601 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.19056731700000001 2024-09-21T14:10:57.602 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.602 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.602 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:10:57.602 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.736554+0000", 2024-09-21T14:10:57.602 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.9183999999999994e-05 2024-09-21T14:10:57.602 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.602 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.602 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:57.602 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.736602+0000", 2024-09-21T14:10:57.602 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.8433999999999997e-05 2024-09-21T14:10:57.602 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:57.602 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:57.602 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:57.603 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.603 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.603 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:61817 11.9 11:9eabf162:::benchmark_data_smithi049_206132_object61816:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e476)", 2024-09-21T14:10:57.603 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:07:27.548213+0000", 2024-09-21T14:10:57.603 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 210.024802163, 2024-09-21T14:10:57.603 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.19485654799999999, 2024-09-21T14:10:57.603 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:57.603 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:57.603 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:57.603 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:10:57.603 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:10:57.603 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:57.603 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 61817 2024-09-21T14:10:57.603 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.604 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:57.604 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.604 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:57.604 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.548213+0000", 2024-09-21T14:10:57.604 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:57.604 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.604 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.604 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:57.604 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.548213+0000", 2024-09-21T14:10:57.604 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:57.604 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.604 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.604 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:57.604 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.548214+0000", 2024-09-21T14:10:57.605 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0100000000000001e-06 2024-09-21T14:10:57.605 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.605 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.605 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:57.605 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.548233+0000", 2024-09-21T14:10:57.605 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.9388000000000001e-05 2024-09-21T14:10:57.605 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.605 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.605 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:57.605 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.548244+0000", 2024-09-21T14:10:57.605 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0694e-05 2024-09-21T14:10:57.605 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.605 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.605 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:57.606 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.548251+0000", 2024-09-21T14:10:57.606 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.3320000000000003e-06 2024-09-21T14:10:57.606 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.606 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.606 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:57.606 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.548302+0000", 2024-09-21T14:10:57.606 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.0180999999999999e-05 2024-09-21T14:10:57.606 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.606 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.606 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:57.606 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.548384+0000", 2024-09-21T14:10:57.606 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.2410000000000005e-05 2024-09-21T14:10:57.606 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.606 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.607 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 1,6", 2024-09-21T14:10:57.607 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.548470+0000", 2024-09-21T14:10:57.607 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.5562000000000005e-05 2024-09-21T14:10:57.607 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.607 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.607 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:10:57.607 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.549773+0000", 2024-09-21T14:10:57.607 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001303331 2024-09-21T14:10:57.607 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.607 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.607 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:57.607 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.550545+0000", 2024-09-21T14:10:57.607 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00077198800000000004 2024-09-21T14:10:57.607 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.608 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.608 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:57.608 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.742904+0000", 2024-09-21T14:10:57.608 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.192359264 2024-09-21T14:10:57.608 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.608 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.608 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:10:57.608 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.743029+0000", 2024-09-21T14:10:57.608 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012438299999999999 2024-09-21T14:10:57.608 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.608 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.608 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:57.608 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:07:27.743070+0000", 2024-09-21T14:10:57.608 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.1004999999999998e-05 2024-09-21T14:10:57.608 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:57.609 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:57.609 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:57.609 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.609 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.609 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:150900 11.b 11:d3b5caa5:::benchmark_data_smithi049_206132_object27521:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e506)", 2024-09-21T14:10:57.609 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:08:50.049533+0000", 2024-09-21T14:10:57.609 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 127.523482183, 2024-09-21T14:10:57.609 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.138400934, 2024-09-21T14:10:57.609 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:57.609 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:57.609 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:57.609 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:10:57.609 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:10:57.609 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:57.610 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 150900 2024-09-21T14:10:57.610 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.610 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:57.610 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.610 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:57.610 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:50.049533+0000", 2024-09-21T14:10:57.610 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:57.610 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.610 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.610 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:57.610 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:50.049533+0000", 2024-09-21T14:10:57.610 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:57.610 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.610 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.610 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:57.611 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:50.049535+0000", 2024-09-21T14:10:57.611 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.7069999999999999e-06 2024-09-21T14:10:57.611 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.611 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.611 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:57.611 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:50.049540+0000", 2024-09-21T14:10:57.611 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.7060000000000001e-06 2024-09-21T14:10:57.611 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.611 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.611 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:57.611 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:50.049620+0000", 2024-09-21T14:10:57.611 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.9747999999999996e-05 2024-09-21T14:10:57.611 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.611 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.612 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:57.612 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:50.049628+0000", 2024-09-21T14:10:57.612 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5000000000000002e-06 2024-09-21T14:10:57.612 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.612 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.612 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:57.612 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:50.049777+0000", 2024-09-21T14:10:57.612 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014942199999999999 2024-09-21T14:10:57.612 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.612 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.612 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:57.612 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:50.049882+0000", 2024-09-21T14:10:57.612 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000104586 2024-09-21T14:10:57.612 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.612 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.613 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,1", 2024-09-21T14:10:57.613 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:50.050005+0000", 2024-09-21T14:10:57.613 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012326399999999999 2024-09-21T14:10:57.613 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.613 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.613 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:10:57.613 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:50.053617+0000", 2024-09-21T14:10:57.613 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0036119070000000001 2024-09-21T14:10:57.613 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.613 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.613 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:57.613 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:50.054763+0000", 2024-09-21T14:10:57.613 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001146553 2024-09-21T14:10:57.613 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.613 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.614 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:57.614 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:50.187653+0000", 2024-09-21T14:10:57.614 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.13288966699999999 2024-09-21T14:10:57.614 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.614 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.614 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:10:57.614 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:50.187858+0000", 2024-09-21T14:10:57.614 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00020505799999999999 2024-09-21T14:10:57.614 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.614 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.614 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:57.614 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:08:50.187934+0000", 2024-09-21T14:10:57.614 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5815999999999994e-05 2024-09-21T14:10:57.614 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:57.615 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:57.615 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:57.615 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.615 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.615 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:165838 11.a 11:52af266b:::benchmark_data_smithi049_206132_object42459:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e516)", 2024-09-21T14:10:57.615 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:09:07.131235+0000", 2024-09-21T14:10:57.615 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 110.441780303, 2024-09-21T14:10:57.615 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.16957782099999999, 2024-09-21T14:10:57.615 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:57.615 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:57.615 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:57.615 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:10:57.615 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:10:57.615 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:57.616 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 165838 2024-09-21T14:10:57.616 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.616 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:57.616 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.616 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:57.616 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.131235+0000", 2024-09-21T14:10:57.616 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:57.616 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.616 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.616 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:57.616 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.131235+0000", 2024-09-21T14:10:57.616 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:57.616 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.616 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.616 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:57.617 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.131236+0000", 2024-09-21T14:10:57.617 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3859999999999999e-06 2024-09-21T14:10:57.617 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.617 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.617 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:57.617 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.131241+0000", 2024-09-21T14:10:57.617 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1100000000000002e-06 2024-09-21T14:10:57.617 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.617 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.617 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:57.617 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.131256+0000", 2024-09-21T14:10:57.617 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4250000000000001e-05 2024-09-21T14:10:57.617 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.617 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.617 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:57.618 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.131288+0000", 2024-09-21T14:10:57.618 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.2449000000000001e-05 2024-09-21T14:10:57.618 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.618 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.618 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:57.618 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.131352+0000", 2024-09-21T14:10:57.618 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3961999999999995e-05 2024-09-21T14:10:57.618 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.618 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.618 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:57.618 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.131472+0000", 2024-09-21T14:10:57.618 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000119431 2024-09-21T14:10:57.618 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.618 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.619 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 5,6", 2024-09-21T14:10:57.619 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.131583+0000", 2024-09-21T14:10:57.619 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000111369 2024-09-21T14:10:57.619 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.619 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.619 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:10:57.619 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.135654+0000", 2024-09-21T14:10:57.619 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0040712389999999999 2024-09-21T14:10:57.619 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.619 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.619 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:57.619 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.136288+0000", 2024-09-21T14:10:57.619 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00063429000000000003 2024-09-21T14:10:57.619 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.620 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.620 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:57.620 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.300533+0000", 2024-09-21T14:10:57.620 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.164244749 2024-09-21T14:10:57.620 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.620 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.620 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:10:57.620 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.300763+0000", 2024-09-21T14:10:57.620 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00022982100000000001 2024-09-21T14:10:57.620 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.620 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.620 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:57.620 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:07.300813+0000", 2024-09-21T14:10:57.620 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9765000000000002e-05 2024-09-21T14:10:57.620 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:57.621 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:57.621 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:57.621 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.621 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.621 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:170987 11.b 11:db9999e3:::benchmark_data_smithi049_206132_object47608:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e516)", 2024-09-21T14:10:57.621 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:09:12.710859+0000", 2024-09-21T14:10:57.621 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 104.862156512, 2024-09-21T14:10:57.621 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.24559587799999999, 2024-09-21T14:10:57.621 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:57.621 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:57.621 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:57.621 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:10:57.621 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:10:57.622 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:57.622 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 170987 2024-09-21T14:10:57.622 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.622 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:57.622 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.622 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:57.622 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:12.710859+0000", 2024-09-21T14:10:57.622 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:57.622 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.622 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.622 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:57.622 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:12.710859+0000", 2024-09-21T14:10:57.622 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:57.622 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.622 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.623 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:57.623 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:12.710860+0000", 2024-09-21T14:10:57.623 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.466e-06 2024-09-21T14:10:57.623 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.623 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.623 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:57.623 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:12.710865+0000", 2024-09-21T14:10:57.623 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.9320000000000004e-06 2024-09-21T14:10:57.623 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.623 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.623 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:57.623 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:12.710875+0000", 2024-09-21T14:10:57.623 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0236e-05 2024-09-21T14:10:57.623 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.624 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.624 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:57.624 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:12.710881+0000", 2024-09-21T14:10:57.624 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2630000000000003e-06 2024-09-21T14:10:57.624 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.624 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.624 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:57.624 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:12.710926+0000", 2024-09-21T14:10:57.624 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5389999999999997e-05 2024-09-21T14:10:57.624 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.624 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.624 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:57.624 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:12.711001+0000", 2024-09-21T14:10:57.624 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5327999999999994e-05 2024-09-21T14:10:57.625 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.625 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.625 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,1", 2024-09-21T14:10:57.625 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:12.711065+0000", 2024-09-21T14:10:57.625 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.3565000000000006e-05 2024-09-21T14:10:57.625 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.625 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.625 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:10:57.625 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:12.714828+0000", 2024-09-21T14:10:57.625 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0037632579999999998 2024-09-21T14:10:57.625 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.625 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.625 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:57.625 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:12.715018+0000", 2024-09-21T14:10:57.625 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00019009300000000001 2024-09-21T14:10:57.626 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.626 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.626 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:57.626 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:12.956347+0000", 2024-09-21T14:10:57.626 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.24132851 2024-09-21T14:10:57.626 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.626 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.626 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:10:57.626 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:12.956408+0000", 2024-09-21T14:10:57.626 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.1094999999999997e-05 2024-09-21T14:10:57.626 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.626 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.626 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:57.626 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:12.956455+0000", 2024-09-21T14:10:57.627 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.6742000000000001e-05 2024-09-21T14:10:57.627 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:57.627 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:57.627 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:57.627 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.627 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.627 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:185618 11.b 11:d05bdf60:::benchmark_data_smithi049_206132_object62239:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e523)", 2024-09-21T14:10:57.627 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:09:29.848714+0000", 2024-09-21T14:10:57.627 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 87.724301229000005, 2024-09-21T14:10:57.627 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.137084123, 2024-09-21T14:10:57.627 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:57.627 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:57.627 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:57.627 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:10:57.627 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:10:57.628 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:57.628 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 185618 2024-09-21T14:10:57.628 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.628 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:57.628 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.628 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:57.628 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.848714+0000", 2024-09-21T14:10:57.628 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:57.628 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.628 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.628 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:57.628 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.848714+0000", 2024-09-21T14:10:57.628 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:57.628 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.629 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.629 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:57.630 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.848715+0000", 2024-09-21T14:10:57.630 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.389e-06 2024-09-21T14:10:57.630 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.630 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.630 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:57.630 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.848721+0000", 2024-09-21T14:10:57.630 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.1919999999999996e-06 2024-09-21T14:10:57.630 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.630 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.630 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:57.630 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.848733+0000", 2024-09-21T14:10:57.630 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.2347000000000001e-05 2024-09-21T14:10:57.630 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.631 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.631 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:57.631 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.848741+0000", 2024-09-21T14:10:57.631 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8429999999999993e-06 2024-09-21T14:10:57.631 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.631 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.631 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:57.631 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.853611+0000", 2024-09-21T14:10:57.631 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.004870259 2024-09-21T14:10:57.631 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.631 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.631 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:57.631 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.853736+0000", 2024-09-21T14:10:57.631 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012495900000000001 2024-09-21T14:10:57.632 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.632 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.632 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,1", 2024-09-21T14:10:57.632 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.853851+0000", 2024-09-21T14:10:57.632 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000115259 2024-09-21T14:10:57.632 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.632 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.632 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:10:57.632 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.863396+0000", 2024-09-21T14:10:57.632 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0095450810000000004 2024-09-21T14:10:57.632 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.632 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.632 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:57.632 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.952660+0000", 2024-09-21T14:10:57.633 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.089263268000000007 2024-09-21T14:10:57.633 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.633 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.633 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:57.633 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.985698+0000", 2024-09-21T14:10:57.633 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.033038784000000002 2024-09-21T14:10:57.633 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.633 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.633 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:10:57.633 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.985754+0000", 2024-09-21T14:10:57.633 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.5130999999999998e-05 2024-09-21T14:10:57.633 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.633 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.633 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:57.634 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.985798+0000", 2024-09-21T14:10:57.634 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4610999999999997e-05 2024-09-21T14:10:57.634 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:57.634 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:57.634 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:57.634 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.634 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.634 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:185623 11.b 11:dece0c01:::benchmark_data_smithi049_206132_object62244:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e523)", 2024-09-21T14:10:57.634 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:09:29.854418+0000", 2024-09-21T14:10:57.634 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 87.718597728000006, 2024-09-21T14:10:57.634 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.13361891000000001, 2024-09-21T14:10:57.634 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:57.634 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:57.634 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:57.635 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:10:57.635 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:10:57.635 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:57.635 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 185623 2024-09-21T14:10:57.635 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.635 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:57.635 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.635 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:57.635 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.854418+0000", 2024-09-21T14:10:57.635 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:57.635 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.635 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.635 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:57.635 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.854418+0000", 2024-09-21T14:10:57.636 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:57.636 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.636 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.636 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:57.636 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.854418+0000", 2024-09-21T14:10:57.636 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.5300000000000003e-07 2024-09-21T14:10:57.636 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.636 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.636 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:57.636 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.854422+0000", 2024-09-21T14:10:57.636 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.9580000000000001e-06 2024-09-21T14:10:57.636 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.636 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.637 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:57.637 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.854431+0000", 2024-09-21T14:10:57.637 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.32e-06 2024-09-21T14:10:57.637 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.637 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.637 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:57.637 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.854437+0000", 2024-09-21T14:10:57.637 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.4320000000000004e-06 2024-09-21T14:10:57.637 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.637 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.637 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:57.637 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.858487+0000", 2024-09-21T14:10:57.637 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0040498460000000002 2024-09-21T14:10:57.637 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.637 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.638 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:57.638 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.858611+0000", 2024-09-21T14:10:57.638 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012403499999999999 2024-09-21T14:10:57.638 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.638 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.638 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,1", 2024-09-21T14:10:57.638 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.858722+0000", 2024-09-21T14:10:57.638 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011086699999999999 2024-09-21T14:10:57.638 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.638 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.638 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:10:57.638 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.863710+0000", 2024-09-21T14:10:57.638 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00498804 2024-09-21T14:10:57.638 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.639 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.639 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:57.639 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.952709+0000", 2024-09-21T14:10:57.639 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.088999522999999997 2024-09-21T14:10:57.639 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.639 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.639 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:57.639 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.986072+0000", 2024-09-21T14:10:57.639 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.033363057000000002 2024-09-21T14:10:57.639 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.639 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.639 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:10:57.639 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.987984+0000", 2024-09-21T14:10:57.639 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.001911825 2024-09-21T14:10:57.640 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.640 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.640 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:57.640 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:09:29.988036+0000", 2024-09-21T14:10:57.640 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.2253999999999998e-05 2024-09-21T14:10:57.640 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:57.640 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:57.640 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:57.640 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.640 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.640 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:227158 11.b 11:d583f9c3:::benchmark_data_smithi049_206132_object103779:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e540)", 2024-09-21T14:10:57.640 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:10:09.247236+0000", 2024-09-21T14:10:57.640 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 48.325778747999998, 2024-09-21T14:10:57.640 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.15082462499999999, 2024-09-21T14:10:57.640 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:57.641 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:57.641 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:57.641 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:10:57.641 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:10:57.641 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:57.641 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 227158 2024-09-21T14:10:57.641 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.641 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:57.641 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.641 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:57.641 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:09.247236+0000", 2024-09-21T14:10:57.641 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:57.641 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.641 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.641 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:57.642 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:09.247236+0000", 2024-09-21T14:10:57.642 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:57.642 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.642 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.642 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:57.642 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:09.247237+0000", 2024-09-21T14:10:57.642 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.4799999999999997e-07 2024-09-21T14:10:57.642 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.642 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.642 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:57.642 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:09.247241+0000", 2024-09-21T14:10:57.642 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.5810000000000001e-06 2024-09-21T14:10:57.642 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.642 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.643 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:57.643 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:09.247248+0000", 2024-09-21T14:10:57.643 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.1790000000000002e-06 2024-09-21T14:10:57.643 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.643 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.643 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:57.643 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:09.247253+0000", 2024-09-21T14:10:57.643 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.079e-06 2024-09-21T14:10:57.643 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.643 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.643 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:57.643 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:09.252971+0000", 2024-09-21T14:10:57.643 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.005718257 2024-09-21T14:10:57.643 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.644 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.644 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:57.644 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:09.253134+0000", 2024-09-21T14:10:57.644 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016273200000000001 2024-09-21T14:10:57.644 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.644 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.644 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,1", 2024-09-21T14:10:57.644 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:09.253253+0000", 2024-09-21T14:10:57.644 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000118591 2024-09-21T14:10:57.644 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.644 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.644 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:10:57.644 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:09.258894+0000", 2024-09-21T14:10:57.645 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0056414209999999998 2024-09-21T14:10:57.645 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.645 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.645 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:57.645 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:09.262975+0000", 2024-09-21T14:10:57.645 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0040806569999999997 2024-09-21T14:10:57.645 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.645 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.645 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:57.645 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:09.397944+0000", 2024-09-21T14:10:57.645 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.13496966399999999 2024-09-21T14:10:57.645 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.645 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.645 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:10:57.645 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:09.398001+0000", 2024-09-21T14:10:57.646 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6651999999999999e-05 2024-09-21T14:10:57.646 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.646 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.646 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:57.646 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:09.398061+0000", 2024-09-21T14:10:57.646 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.9963999999999997e-05 2024-09-21T14:10:57.646 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:57.646 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:57.646 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:57.646 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.646 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.646 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:227179 11.b 11:dbd60be3:::benchmark_data_smithi049_206132_object103800:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e540)", 2024-09-21T14:10:57.646 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:10:09.264857+0000", 2024-09-21T14:10:57.646 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 48.30815819, 2024-09-21T14:10:57.647 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.13336240699999999, 2024-09-21T14:10:57.647 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:57.647 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:57.647 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:57.647 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:10:57.647 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:10:57.647 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:57.647 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 227179 2024-09-21T14:10:57.647 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.647 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:57.647 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.647 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:57.647 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:09.264857+0000", 2024-09-21T14:10:57.647 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:57.648 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.648 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.648 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:57.648 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:09.264857+0000", 2024-09-21T14:10:57.648 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:57.648 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.648 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.648 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:57.648 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:09.264859+0000", 2024-09-21T14:10:57.648 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6780000000000001e-06 2024-09-21T14:10:57.648 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.648 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.648 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:57.648 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:09.264864+0000", 2024-09-21T14:10:57.649 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4650000000000002e-06 2024-09-21T14:10:57.649 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.649 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.649 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:57.649 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:09.264878+0000", 2024-09-21T14:10:57.649 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3587999999999999e-05 2024-09-21T14:10:57.649 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.649 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.649 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:57.649 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:09.264883+0000", 2024-09-21T14:10:57.649 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6049999999999997e-06 2024-09-21T14:10:57.650 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.650 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.650 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:57.650 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:09.264940+0000", 2024-09-21T14:10:57.650 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6502000000000002e-05 2024-09-21T14:10:57.650 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.650 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.650 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:57.650 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:09.265027+0000", 2024-09-21T14:10:57.650 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.7516999999999996e-05 2024-09-21T14:10:57.650 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.650 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.650 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,1", 2024-09-21T14:10:57.651 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:09.265098+0000", 2024-09-21T14:10:57.651 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.0362000000000002e-05 2024-09-21T14:10:57.651 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.651 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.651 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:10:57.651 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:09.274200+0000", 2024-09-21T14:10:57.651 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0091017909999999997 2024-09-21T14:10:57.651 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.651 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.651 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:57.651 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:09.275319+0000", 2024-09-21T14:10:57.651 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0011193010000000001 2024-09-21T14:10:57.651 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.651 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.651 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:57.652 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:09.398122+0000", 2024-09-21T14:10:57.652 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.12280350900000001 2024-09-21T14:10:57.652 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.652 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.652 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:10:57.652 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:09.398179+0000", 2024-09-21T14:10:57.652 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.6509999999999999e-05 2024-09-21T14:10:57.652 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.652 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.652 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:57.652 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:09.398219+0000", 2024-09-21T14:10:57.652 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.0578999999999999e-05 2024-09-21T14:10:57.652 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:57.652 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:57.653 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:57.653 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.653 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.653 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:229424 11.b 11:ddfd6caf:::benchmark_data_smithi049_206132_object106045:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e540)", 2024-09-21T14:10:57.653 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:10:11.399720+0000", 2024-09-21T14:10:57.653 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 46.173294759999997, 2024-09-21T14:10:57.653 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.14463946799999999, 2024-09-21T14:10:57.653 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:57.653 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:57.653 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:57.653 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:10:57.653 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:10:57.653 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:57.653 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 229424 2024-09-21T14:10:57.653 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.654 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:57.654 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.654 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:57.654 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:11.399720+0000", 2024-09-21T14:10:57.654 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:57.654 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.654 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.654 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:57.654 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:11.399720+0000", 2024-09-21T14:10:57.654 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:57.654 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.654 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.654 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:57.654 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:11.399721+0000", 2024-09-21T14:10:57.655 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.4300000000000001e-07 2024-09-21T14:10:57.655 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.655 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.655 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:57.655 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:11.399726+0000", 2024-09-21T14:10:57.655 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.5360000000000003e-06 2024-09-21T14:10:57.655 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.655 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.655 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:57.655 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:11.399739+0000", 2024-09-21T14:10:57.655 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.3247999999999999e-05 2024-09-21T14:10:57.655 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.655 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.655 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:57.656 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:11.399746+0000", 2024-09-21T14:10:57.656 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.2400000000000001e-06 2024-09-21T14:10:57.656 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.656 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.656 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:57.656 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:11.399814+0000", 2024-09-21T14:10:57.656 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7754000000000002e-05 2024-09-21T14:10:57.656 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.656 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.656 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:57.656 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:11.399958+0000", 2024-09-21T14:10:57.656 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00014361800000000001 2024-09-21T14:10:57.656 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.656 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.656 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,1", 2024-09-21T14:10:57.657 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:11.400077+0000", 2024-09-21T14:10:57.657 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00011944500000000001 2024-09-21T14:10:57.657 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.657 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.657 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:10:57.657 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:11.405123+0000", 2024-09-21T14:10:57.657 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0050454059999999997 2024-09-21T14:10:57.657 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.657 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.657 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:57.657 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:11.407125+0000", 2024-09-21T14:10:57.657 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.0020020010000000002 2024-09-21T14:10:57.657 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.657 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.658 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:57.658 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:11.544245+0000", 2024-09-21T14:10:57.658 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.137120458 2024-09-21T14:10:57.658 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.658 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.658 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:10:57.658 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:11.544313+0000", 2024-09-21T14:10:57.658 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.7391999999999994e-05 2024-09-21T14:10:57.658 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.658 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.658 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:57.658 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:11.544360+0000", 2024-09-21T14:10:57.658 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.7426999999999998e-05 2024-09-21T14:10:57.658 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:57.658 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:57.659 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:57.659 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.659 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.659 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.26396.0:229442 11.b 11:da8be574:::benchmark_data_smithi049_206132_object106063:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e540)", 2024-09-21T14:10:57.659 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:10:11.413771+0000", 2024-09-21T14:10:57.659 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 46.159243877999998, 2024-09-21T14:10:57.659 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.137960154, 2024-09-21T14:10:57.659 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:57.659 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:57.659 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:10:57.659 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:10:57.659 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.26396", 2024-09-21T14:10:57.659 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/13910262", 2024-09-21T14:10:57.659 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 229442 2024-09-21T14:10:57.660 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.660 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:57.660 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.660 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:57.660 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:11.413771+0000", 2024-09-21T14:10:57.660 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:57.660 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.660 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.660 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:57.660 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:11.413771+0000", 2024-09-21T14:10:57.660 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:57.660 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.660 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.660 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:57.660 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:11.413773+0000", 2024-09-21T14:10:57.661 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.158e-06 2024-09-21T14:10:57.661 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.661 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.661 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:57.661 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:11.413777+0000", 2024-09-21T14:10:57.661 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.4580000000000003e-06 2024-09-21T14:10:57.661 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.661 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.661 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:57.661 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:11.413789+0000", 2024-09-21T14:10:57.661 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1769e-05 2024-09-21T14:10:57.661 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.661 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.661 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:57.661 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:11.413795+0000", 2024-09-21T14:10:57.662 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.0719999999999996e-06 2024-09-21T14:10:57.662 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.662 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.662 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:57.662 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:11.413861+0000", 2024-09-21T14:10:57.662 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.6612000000000005e-05 2024-09-21T14:10:57.662 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.662 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.662 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:57.662 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:11.413997+0000", 2024-09-21T14:10:57.662 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000135289 2024-09-21T14:10:57.662 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.662 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.662 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "waiting for subops from 0,1", 2024-09-21T14:10:57.663 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:11.414134+0000", 2024-09-21T14:10:57.663 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00013746799999999999 2024-09-21T14:10:57.663 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.663 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.663 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "op_commit", 2024-09-21T14:10:57.663 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:11.419306+0000", 2024-09-21T14:10:57.663 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.005172238 2024-09-21T14:10:57.663 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.663 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.663 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:57.663 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:11.420173+0000", 2024-09-21T14:10:57.663 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00086700000000000004 2024-09-21T14:10:57.663 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.663 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.663 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:10:57.664 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:11.551524+0000", 2024-09-21T14:10:57.664 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.13135044500000001 2024-09-21T14:10:57.664 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.664 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.664 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "commit_sent", 2024-09-21T14:10:57.664 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:11.551677+0000", 2024-09-21T14:10:57.664 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000153616 2024-09-21T14:10:57.664 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.664 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.664 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:57.664 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:11.551732+0000", 2024-09-21T14:10:57.664 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 5.4029000000000001e-05 2024-09-21T14:10:57.664 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:57.664 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:57.664 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:57.665 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.665 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.665 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "replica_scrub(pg: 4.15,from:0'0,to:516'21,epoch:555/345,start:4:a8000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-21T14:10:57.665 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:10:42.555583+0000", 2024-09-21T14:10:57.665 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 15.017431776, 2024-09-21T14:10:57.665 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00034925800000000002, 2024-09-21T14:10:57.665 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:57.665 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:57.665 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T14:10:57.665 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:57.665 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.665 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:57.665 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:42.555583+0000", 2024-09-21T14:10:57.665 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:57.666 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.666 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.666 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:57.666 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:42.555583+0000", 2024-09-21T14:10:57.666 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:57.666 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.666 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.666 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:57.666 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:42.555585+0000", 2024-09-21T14:10:57.666 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.4470000000000001e-06 2024-09-21T14:10:57.666 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.666 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.666 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:57.666 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:42.555596+0000", 2024-09-21T14:10:57.666 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0815999999999999e-05 2024-09-21T14:10:57.667 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.667 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.667 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:57.667 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:42.555626+0000", 2024-09-21T14:10:57.667 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.0148999999999999e-05 2024-09-21T14:10:57.667 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.667 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.667 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:57.667 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:42.555641+0000", 2024-09-21T14:10:57.667 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.501e-05 2024-09-21T14:10:57.667 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.667 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.667 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:57.667 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:42.555803+0000", 2024-09-21T14:10:57.668 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00016178899999999999 2024-09-21T14:10:57.668 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.668 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.668 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:57.668 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:42.555846+0000", 2024-09-21T14:10:57.668 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3282999999999999e-05 2024-09-21T14:10:57.668 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.668 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.668 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:57.668 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:42.555933+0000", 2024-09-21T14:10:57.668 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.6763999999999998e-05 2024-09-21T14:10:57.668 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:57.668 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:57.668 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:57.668 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.668 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.669 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDScrubReserve(4.10,GRANT,e:555,from:6,reservation_nonce:0)", 2024-09-21T14:10:57.669 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:10:43.493376+0000", 2024-09-21T14:10:57.669 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 14.079638975, 2024-09-21T14:10:57.669 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000241704, 2024-09-21T14:10:57.669 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:57.669 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:57.669 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T14:10:57.669 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:57.669 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.669 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:57.669 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:43.493376+0000", 2024-09-21T14:10:57.669 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:57.669 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.669 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.669 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:57.670 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:43.493376+0000", 2024-09-21T14:10:57.670 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:57.670 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.670 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.670 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:10:57.670 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:57.670 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:43.493377+0000", 2024-09-21T14:10:57.671 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.8400000000000004e-07 2024-09-21T14:10:57.671 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.671 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.671 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:57.671 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:43.493379+0000", 2024-09-21T14:10:57.671 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.5170000000000002e-06 2024-09-21T14:10:57.671 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.671 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.671 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:57.671 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:43.493390+0000", 2024-09-21T14:10:57.671 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0716999999999999e-05 2024-09-21T14:10:57.671 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.671 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.671 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:57.672 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:43.493394+0000", 2024-09-21T14:10:57.672 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 4.3100000000000002e-06 2024-09-21T14:10:57.672 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.672 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.672 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:57.672 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:43.493473+0000", 2024-09-21T14:10:57.672 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 7.8237000000000003e-05 2024-09-21T14:10:57.672 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.672 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.672 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:57.672 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:43.493488+0000", 2024-09-21T14:10:57.672 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.5469e-05 2024-09-21T14:10:57.672 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.672 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.673 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:57.673 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:43.493618+0000", 2024-09-21T14:10:57.673 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00012977 2024-09-21T14:10:57.673 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:57.673 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:57.673 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:57.673 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.673 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.673 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "osd_op(client.24365.0:4521 4.16 4:6fe6cc47:notif::queues_list_object:head [omap-get-keys] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e555)", 2024-09-21T14:10:57.673 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:10:47.793330+0000", 2024-09-21T14:10:57.673 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 9.7796853709999993, 2024-09-21T14:10:57.673 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00026203499999999998, 2024-09-21T14:10:57.673 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:57.673 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:57.673 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "reached pg", 2024-09-21T14:10:57.674 INFO:teuthology.orchestra.run.smithi049.stdout: "client_info": { 2024-09-21T14:10:57.674 INFO:teuthology.orchestra.run.smithi049.stdout: "client": "client.24365", 2024-09-21T14:10:57.674 INFO:teuthology.orchestra.run.smithi049.stdout: "client_addr": "172.21.15.49:0/3404764347", 2024-09-21T14:10:57.674 INFO:teuthology.orchestra.run.smithi049.stdout: "tid": 4521 2024-09-21T14:10:57.674 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.674 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:57.674 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.674 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:57.674 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:47.793330+0000", 2024-09-21T14:10:57.674 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:57.674 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.674 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.674 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:57.674 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:47.793330+0000", 2024-09-21T14:10:57.675 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:57.675 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.675 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.675 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:57.675 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:47.793332+0000", 2024-09-21T14:10:57.675 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4219999999999999e-06 2024-09-21T14:10:57.675 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.675 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.675 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:57.675 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:47.793343+0000", 2024-09-21T14:10:57.675 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.0684e-05 2024-09-21T14:10:57.675 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.675 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.675 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:57.675 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:47.793368+0000", 2024-09-21T14:10:57.676 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.4542999999999999e-05 2024-09-21T14:10:57.676 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.676 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.676 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:57.676 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:47.793377+0000", 2024-09-21T14:10:57.676 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.2830000000000007e-06 2024-09-21T14:10:57.676 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.676 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.676 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:57.676 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:47.793478+0000", 2024-09-21T14:10:57.676 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000101279 2024-09-21T14:10:57.676 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.676 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.676 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:57.677 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:47.793592+0000", 2024-09-21T14:10:57.677 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000113824 2024-09-21T14:10:57.677 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:57.677 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:57.677 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:57.677 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.677 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.677 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDScrubReserve(4.11,GRANT,e:559,from:7,reservation_nonce:0)", 2024-09-21T14:10:57.677 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:10:53.413070+0000", 2024-09-21T14:10:57.677 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 4.1599450129999997, 2024-09-21T14:10:57.677 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00043460999999999999, 2024-09-21T14:10:57.677 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:57.677 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:57.677 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T14:10:57.677 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:57.678 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.678 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:57.678 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:53.413070+0000", 2024-09-21T14:10:57.678 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:57.678 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.678 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.678 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:57.678 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:53.413070+0000", 2024-09-21T14:10:57.678 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:57.678 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.678 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.678 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:57.678 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:53.413079+0000", 2024-09-21T14:10:57.678 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.9369999999999993e-06 2024-09-21T14:10:57.679 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.679 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.679 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:57.679 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:53.413088+0000", 2024-09-21T14:10:57.679 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 8.969e-06 2024-09-21T14:10:57.679 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.679 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.679 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:57.679 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:53.413115+0000", 2024-09-21T14:10:57.679 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.7336000000000001e-05 2024-09-21T14:10:57.679 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.679 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.679 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:57.679 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:53.413127+0000", 2024-09-21T14:10:57.679 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.1926000000000001e-05 2024-09-21T14:10:57.680 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.680 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.680 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:57.680 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:53.413236+0000", 2024-09-21T14:10:57.680 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000108655 2024-09-21T14:10:57.680 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.680 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.680 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:57.680 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:53.413267+0000", 2024-09-21T14:10:57.680 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 3.0552000000000002e-05 2024-09-21T14:10:57.680 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.680 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.680 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:57.680 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:53.413505+0000", 2024-09-21T14:10:57.680 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000238235 2024-09-21T14:10:57.681 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:57.681 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:57.681 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:57.681 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.681 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.681 INFO:teuthology.orchestra.run.smithi049.stdout: "description": "MOSDScrubReserve(4.c,RELEASE,e:560,from:4,reservation_nonce:0)", 2024-09-21T14:10:57.681 INFO:teuthology.orchestra.run.smithi049.stdout: "initiated_at": "2024-09-21T14:10:55.718901+0000", 2024-09-21T14:10:57.681 INFO:teuthology.orchestra.run.smithi049.stdout: "age": 1.854113906, 2024-09-21T14:10:57.681 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.00028067599999999999, 2024-09-21T14:10:57.681 INFO:teuthology.orchestra.run.smithi049.stdout: "continuous": false, 2024-09-21T14:10:57.681 INFO:teuthology.orchestra.run.smithi049.stdout: "type_data": { 2024-09-21T14:10:57.681 INFO:teuthology.orchestra.run.smithi049.stdout: "flag_point": "started", 2024-09-21T14:10:57.681 INFO:teuthology.orchestra.run.smithi049.stdout: "events": [ 2024-09-21T14:10:57.681 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.682 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "initiated", 2024-09-21T14:10:57.682 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:55.718901+0000", 2024-09-21T14:10:57.682 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:57.682 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.682 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.682 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "header_read", 2024-09-21T14:10:57.682 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:55.718901+0000", 2024-09-21T14:10:57.682 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0 2024-09-21T14:10:57.682 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.682 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.682 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "throttled", 2024-09-21T14:10:57.682 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:55.718902+0000", 2024-09-21T14:10:57.682 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 9.7699999999999992e-07 2024-09-21T14:10:57.682 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.683 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.683 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "all_read", 2024-09-21T14:10:57.683 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:55.718909+0000", 2024-09-21T14:10:57.683 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 6.5830000000000003e-06 2024-09-21T14:10:57.683 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.683 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.683 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "dispatched", 2024-09-21T14:10:57.683 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:55.718927+0000", 2024-09-21T14:10:57.683 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.8298999999999999e-05 2024-09-21T14:10:57.683 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.683 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.683 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "queued_for_pg", 2024-09-21T14:10:57.683 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:55.718943+0000", 2024-09-21T14:10:57.683 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 1.6226999999999999e-05 2024-09-21T14:10:57.683 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.684 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.684 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "reached_pg", 2024-09-21T14:10:57.684 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:55.719125+0000", 2024-09-21T14:10:57.684 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 0.000181688 2024-09-21T14:10:57.684 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.684 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.684 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "started", 2024-09-21T14:10:57.684 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:55.719155+0000", 2024-09-21T14:10:57.684 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.9981999999999999e-05 2024-09-21T14:10:57.684 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-09-21T14:10:57.684 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-09-21T14:10:57.684 INFO:teuthology.orchestra.run.smithi049.stdout: "event": "done", 2024-09-21T14:10:57.684 INFO:teuthology.orchestra.run.smithi049.stdout: "time": "2024-09-21T14:10:55.719182+0000", 2024-09-21T14:10:57.684 INFO:teuthology.orchestra.run.smithi049.stdout: "duration": 2.692e-05 2024-09-21T14:10:57.684 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:57.685 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:57.685 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:57.685 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-09-21T14:10:57.685 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-09-21T14:10:57.685 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-09-21T14:10:57.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:57 smithi187 ceph-mon[33032]: 4.c scrub starts 2024-09-21T14:10:57.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:57 smithi187 ceph-mon[33032]: 4.c scrub ok 2024-09-21T14:10:57.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:57 smithi187 ceph-mon[33032]: pgmap v1133: 152 pgs: 152 active+clean; 583 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.9 KiB/s rd, 1 op/s; 38 B/s, 2 objects/s recovering 2024-09-21T14:10:57.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:57 smithi187 ceph-mon[33032]: 6.1c scrub starts 2024-09-21T14:10:57.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:57 smithi187 ceph-mon[33032]: 6.1c scrub ok 2024-09-21T14:10:57.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:57 smithi187 ceph-mon[33032]: 4.1d scrub starts 2024-09-21T14:10:57.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:57 smithi187 ceph-mon[33032]: 4.1d scrub ok 2024-09-21T14:10:58.618 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:58 smithi049 ceph-mon[40489]: 3.1a scrub starts 2024-09-21T14:10:58.618 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:58 smithi049 ceph-mon[40489]: 3.1a scrub ok 2024-09-21T14:10:58.619 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:58 smithi049 ceph-mon[30080]: 3.1a scrub starts 2024-09-21T14:10:58.619 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:58 smithi049 ceph-mon[30080]: 3.1a scrub ok 2024-09-21T14:10:58.654 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T14:10:58.682 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:58 smithi187 ceph-mon[33032]: 3.1a scrub starts 2024-09-21T14:10:58.682 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:58 smithi187 ceph-mon[33032]: 3.1a scrub ok 2024-09-21T14:10:58.913 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:10:59.437 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T14:10:59.437 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-21T14:10:59.437 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 0 2024-09-21T14:10:59.437 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T14:10:59.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:59 smithi049 ceph-mon[40489]: pgmap v1135: 152 pgs: 152 active+clean; 583 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T14:10:59.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:59 smithi049 ceph-mon[40489]: 4.19 scrub starts 2024-09-21T14:10:59.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:10:59 smithi049 ceph-mon[40489]: 4.19 scrub ok 2024-09-21T14:10:59.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:59 smithi049 ceph-mon[30080]: pgmap v1135: 152 pgs: 152 active+clean; 583 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T14:10:59.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:59 smithi049 ceph-mon[30080]: 4.19 scrub starts 2024-09-21T14:10:59.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:10:59 smithi049 ceph-mon[30080]: 4.19 scrub ok 2024-09-21T14:10:59.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:59 smithi187 ceph-mon[33032]: pgmap v1135: 152 pgs: 152 active+clean; 583 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T14:10:59.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:59 smithi187 ceph-mon[33032]: 4.19 scrub starts 2024-09-21T14:10:59.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:10:59 smithi187 ceph-mon[33032]: 4.19 scrub ok 2024-09-21T14:11:00.069 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T14:11:00.278 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.4 to 1 (10000) 2024-09-21T14:11:00.368 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:11:00.579 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:00 smithi049 ceph-mon[40489]: 5.0 scrub starts 2024-09-21T14:11:00.579 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:00 smithi049 ceph-mon[40489]: 5.0 scrub ok 2024-09-21T14:11:00.579 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:00 smithi049 ceph-mon[40489]: 3.16 scrub starts 2024-09-21T14:11:00.579 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:00 smithi049 ceph-mon[40489]: 3.16 scrub ok 2024-09-21T14:11:00.579 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:00 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1749028211' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T14:11:00.579 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:00 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:11:00.580 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:00 smithi049 ceph-mon[40489]: 3.a scrub starts 2024-09-21T14:11:00.580 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:00 smithi049 ceph-mon[40489]: 3.a scrub ok 2024-09-21T14:11:00.580 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:00 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1749028211' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T14:11:00.580 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:00 smithi049 ceph-mon[40489]: osdmap e562: 8 total, 8 up, 8 in 2024-09-21T14:11:00.580 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:00 smithi049 ceph-mon[30080]: 5.0 scrub starts 2024-09-21T14:11:00.580 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:00 smithi049 ceph-mon[30080]: 5.0 scrub ok 2024-09-21T14:11:00.580 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:00 smithi049 ceph-mon[30080]: 3.16 scrub starts 2024-09-21T14:11:00.580 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:00 smithi049 ceph-mon[30080]: 3.16 scrub ok 2024-09-21T14:11:00.580 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:00 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1749028211' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T14:11:00.580 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:00 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:11:00.580 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:00 smithi049 ceph-mon[30080]: 3.a scrub starts 2024-09-21T14:11:00.581 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:00 smithi049 ceph-mon[30080]: 3.a scrub ok 2024-09-21T14:11:00.581 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:00 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1749028211' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T14:11:00.581 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:00 smithi049 ceph-mon[30080]: osdmap e562: 8 total, 8 up, 8 in 2024-09-21T14:11:00.598 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:00 smithi187 ceph-mon[33032]: 5.0 scrub starts 2024-09-21T14:11:00.598 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:00 smithi187 ceph-mon[33032]: 5.0 scrub ok 2024-09-21T14:11:00.598 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:00 smithi187 ceph-mon[33032]: 3.16 scrub starts 2024-09-21T14:11:00.598 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:00 smithi187 ceph-mon[33032]: 3.16 scrub ok 2024-09-21T14:11:00.598 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:00 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1749028211' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T14:11:00.598 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:00 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:11:00.598 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:00 smithi187 ceph-mon[33032]: 3.a scrub starts 2024-09-21T14:11:00.599 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:00 smithi187 ceph-mon[33032]: 3.a scrub ok 2024-09-21T14:11:00.599 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:00 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1749028211' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T14:11:00.599 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:00 smithi187 ceph-mon[33032]: osdmap e562: 8 total, 8 up, 8 in 2024-09-21T14:11:00.599 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:11:00 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:11:00.835 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T14:11:00.835 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-21T14:11:00.835 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-21T14:11:00.835 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-21T14:11:00.836 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T14:11:01.004 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:11:00 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:11:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:11:01.223 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 4 1 2024-09-21T14:11:01.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:01 smithi187 ceph-mon[33032]: 6.0 scrub starts 2024-09-21T14:11:01.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:01 smithi187 ceph-mon[33032]: 6.0 scrub ok 2024-09-21T14:11:01.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:01 smithi187 ceph-mon[33032]: pgmap v1136: 152 pgs: 152 active+clean; 583 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 746 B/s rd, 0 op/s 2024-09-21T14:11:01.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:01 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:11:01.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:01 smithi187 ceph-mon[33032]: 6.3 scrub starts 2024-09-21T14:11:01.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:01 smithi187 ceph-mon[33032]: 6.3 scrub ok 2024-09-21T14:11:01.492 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:11:01.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:01 smithi049 ceph-mon[40489]: 6.0 scrub starts 2024-09-21T14:11:01.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:01 smithi049 ceph-mon[40489]: 6.0 scrub ok 2024-09-21T14:11:01.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:01 smithi049 ceph-mon[40489]: pgmap v1136: 152 pgs: 152 active+clean; 583 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 746 B/s rd, 0 op/s 2024-09-21T14:11:01.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:01 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:11:01.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:01 smithi049 ceph-mon[40489]: 6.3 scrub starts 2024-09-21T14:11:01.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:01 smithi049 ceph-mon[40489]: 6.3 scrub ok 2024-09-21T14:11:01.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:01 smithi049 ceph-mon[30080]: 6.0 scrub starts 2024-09-21T14:11:01.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:01 smithi049 ceph-mon[30080]: 6.0 scrub ok 2024-09-21T14:11:01.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:01 smithi049 ceph-mon[30080]: pgmap v1136: 152 pgs: 152 active+clean; 583 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 746 B/s rd, 0 op/s 2024-09-21T14:11:01.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:01 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:11:01.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:01 smithi049 ceph-mon[30080]: 6.3 scrub starts 2024-09-21T14:11:01.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:01 smithi049 ceph-mon[30080]: 6.3 scrub ok 2024-09-21T14:11:01.559 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T14:11:01.802 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:11:02.341 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T14:11:02.341 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-21T14:11:02.342 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-21T14:11:02.342 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-21T14:11:02.342 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.342 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "replica_scrub(pg: 5.4,from:0'0,to:0'0,epoch:556/491,start:5:20000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-21T14:11:02.342 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:10:46.492480+0000", 2024-09-21T14:11:02.342 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 15.792322806, 2024-09-21T14:11:02.342 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00022520600000000001, 2024-09-21T14:11:02.342 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:11:02.342 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:11:02.342 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T14:11:02.342 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:11:02.342 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.342 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:11:02.343 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:46.492480+0000", 2024-09-21T14:11:02.343 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:02.343 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.343 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.343 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:11:02.343 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:46.492480+0000", 2024-09-21T14:11:02.343 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:02.343 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.343 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.343 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:11:02.343 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:46.492482+0000", 2024-09-21T14:11:02.343 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.75e-06 2024-09-21T14:11:02.343 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.343 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.344 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:11:02.344 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:46.492493+0000", 2024-09-21T14:11:02.344 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0993e-05 2024-09-21T14:11:02.344 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.344 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.344 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:11:02.344 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:46.492524+0000", 2024-09-21T14:11:02.344 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1106000000000003e-05 2024-09-21T14:11:02.344 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.344 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.344 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:11:02.344 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:46.492539+0000", 2024-09-21T14:11:02.344 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5786000000000001e-05 2024-09-21T14:11:02.344 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.345 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.345 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:11:02.345 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:46.492674+0000", 2024-09-21T14:11:02.345 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000134692 2024-09-21T14:11:02.345 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.345 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.345 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:11:02.345 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:46.492682+0000", 2024-09-21T14:11:02.345 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.96e-06 2024-09-21T14:11:02.345 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.345 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.345 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:11:02.345 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:46.492705+0000", 2024-09-21T14:11:02.345 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2918999999999999e-05 2024-09-21T14:11:02.345 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:02.346 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:11:02.346 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:02.346 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.346 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.346 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "MOSDScrubReserve(5.1f,RELEASE,e:558,from:6,reservation_nonce:0)", 2024-09-21T14:11:02.346 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:10:50.069750+0000", 2024-09-21T14:11:02.346 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 12.215052163999999, 2024-09-21T14:11:02.346 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000184787, 2024-09-21T14:11:02.346 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:11:02.346 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:11:02.346 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T14:11:02.346 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:11:02.346 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.346 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:11:02.347 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:50.069750+0000", 2024-09-21T14:11:02.347 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:02.347 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.347 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.347 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:11:02.347 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:50.069750+0000", 2024-09-21T14:11:02.347 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:02.347 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.347 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.347 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:11:02.347 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:50.069751+0000", 2024-09-21T14:11:02.347 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6499999999999999e-07 2024-09-21T14:11:02.347 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.348 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.348 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:11:02.348 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:50.069755+0000", 2024-09-21T14:11:02.348 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4939999999999999e-06 2024-09-21T14:11:02.348 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.348 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.348 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:11:02.348 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:50.069760+0000", 2024-09-21T14:11:02.348 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7060000000000001e-06 2024-09-21T14:11:02.348 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.349 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.349 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:11:02.349 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:50.069767+0000", 2024-09-21T14:11:02.349 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6590000000000001e-06 2024-09-21T14:11:02.349 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.349 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.349 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:11:02.349 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:50.069898+0000", 2024-09-21T14:11:02.349 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013155900000000001 2024-09-21T14:11:02.349 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.349 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.349 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:11:02.349 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:50.069923+0000", 2024-09-21T14:11:02.349 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4844000000000001e-05 2024-09-21T14:11:02.349 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.350 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.350 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:11:02.350 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:50.069935+0000", 2024-09-21T14:11:02.350 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1960000000000001e-05 2024-09-21T14:11:02.350 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:02.350 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:11:02.350 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:02.350 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.350 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.350 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "MOSDScrubReserve(3.14,QREQUEST,e:558,from:6,reservation_nonce:5)", 2024-09-21T14:11:02.350 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:10:52.040803+0000", 2024-09-21T14:11:02.350 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 10.243999282000001, 2024-09-21T14:11:02.350 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000145446, 2024-09-21T14:11:02.350 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:11:02.351 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:11:02.351 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T14:11:02.351 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:11:02.351 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.351 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:11:02.351 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.040803+0000", 2024-09-21T14:11:02.351 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:02.351 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.351 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.351 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:11:02.351 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.040803+0000", 2024-09-21T14:11:02.351 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:02.351 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.351 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.351 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:11:02.352 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.040805+0000", 2024-09-21T14:11:02.352 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.31e-06 2024-09-21T14:11:02.352 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.352 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.352 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:11:02.352 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.040810+0000", 2024-09-21T14:11:02.352 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5629999999999999e-06 2024-09-21T14:11:02.352 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.352 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.352 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:11:02.352 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.040826+0000", 2024-09-21T14:11:02.352 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6191000000000001e-05 2024-09-21T14:11:02.352 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.352 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.353 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:11:02.353 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.040835+0000", 2024-09-21T14:11:02.353 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.6850000000000007e-06 2024-09-21T14:11:02.353 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.353 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.353 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:11:02.353 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.040904+0000", 2024-09-21T14:11:02.353 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8454000000000006e-05 2024-09-21T14:11:02.353 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.353 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.353 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:11:02.353 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.040920+0000", 2024-09-21T14:11:02.353 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6528000000000002e-05 2024-09-21T14:11:02.353 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.354 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.354 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:11:02.354 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.040949+0000", 2024-09-21T14:11:02.354 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8714999999999999e-05 2024-09-21T14:11:02.354 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:02.354 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:11:02.354 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:02.354 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.354 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.354 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "MOSDScrubReserve(3.14,RELEASE,e:558,from:6,reservation_nonce:0)", 2024-09-21T14:11:02.354 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:10:52.042519+0000", 2024-09-21T14:11:02.354 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 10.242283277, 2024-09-21T14:11:02.354 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014281100000000001, 2024-09-21T14:11:02.355 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:11:02.355 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:11:02.355 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T14:11:02.355 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:11:02.355 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.355 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:11:02.355 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.042519+0000", 2024-09-21T14:11:02.355 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:02.355 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.355 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.355 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:11:02.356 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.042519+0000", 2024-09-21T14:11:02.356 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:02.356 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.356 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.356 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:11:02.356 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.042520+0000", 2024-09-21T14:11:02.356 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7399999999999998e-07 2024-09-21T14:11:02.356 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.356 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.356 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:11:02.356 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.042523+0000", 2024-09-21T14:11:02.356 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2109999999999998e-06 2024-09-21T14:11:02.356 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.356 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.357 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:11:02.357 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.042539+0000", 2024-09-21T14:11:02.357 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5829e-05 2024-09-21T14:11:02.357 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.357 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.357 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:11:02.357 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.042548+0000", 2024-09-21T14:11:02.357 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.901e-06 2024-09-21T14:11:02.357 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.357 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.357 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:11:02.357 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.042629+0000", 2024-09-21T14:11:02.357 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0724999999999997e-05 2024-09-21T14:11:02.357 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.358 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.358 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:11:02.358 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.042646+0000", 2024-09-21T14:11:02.358 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7334e-05 2024-09-21T14:11:02.358 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.358 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.358 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:11:02.358 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.042662+0000", 2024-09-21T14:11:02.358 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6337e-05 2024-09-21T14:11:02.358 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:02.358 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:11:02.358 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:02.358 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.359 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.359 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "MOSDScrubReserve(3.8,QREQUEST,e:558,from:1,reservation_nonce:3)", 2024-09-21T14:11:02.359 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:10:52.259738+0000", 2024-09-21T14:11:02.359 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 10.025064742, 2024-09-21T14:11:02.359 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000145556, 2024-09-21T14:11:02.359 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:11:02.359 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:11:02.359 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T14:11:02.359 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:11:02.359 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.359 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:11:02.359 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.259738+0000", 2024-09-21T14:11:02.359 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:02.359 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.359 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.360 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:11:02.360 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.259738+0000", 2024-09-21T14:11:02.360 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:02.360 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.360 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.360 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:11:02.360 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.259739+0000", 2024-09-21T14:11:02.360 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.003e-06 2024-09-21T14:11:02.360 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.360 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.360 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:11:02.360 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.259744+0000", 2024-09-21T14:11:02.360 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4450000000000004e-06 2024-09-21T14:11:02.361 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.361 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.361 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:11:02.361 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.259759+0000", 2024-09-21T14:11:02.361 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4574e-05 2024-09-21T14:11:02.361 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.361 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.361 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:11:02.361 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.259768+0000", 2024-09-21T14:11:02.361 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7110000000000001e-06 2024-09-21T14:11:02.361 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.361 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.361 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:11:02.361 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.259825+0000", 2024-09-21T14:11:02.362 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7258e-05 2024-09-21T14:11:02.362 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.362 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.362 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:11:02.362 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.259849+0000", 2024-09-21T14:11:02.362 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4491999999999999e-05 2024-09-21T14:11:02.362 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.362 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.362 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:11:02.362 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.259883+0000", 2024-09-21T14:11:02.362 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4072999999999997e-05 2024-09-21T14:11:02.362 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:02.362 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:11:02.363 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:02.363 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.363 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.363 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "replica_scrub(pg: 3.8,from:0'0,to:0'0,epoch:558/496,start:3:10000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-21T14:11:02.363 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:10:52.260623+0000", 2024-09-21T14:11:02.363 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 10.02417925, 2024-09-21T14:11:02.363 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000152764, 2024-09-21T14:11:02.363 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:11:02.363 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:11:02.363 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T14:11:02.363 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:11:02.363 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.363 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:11:02.363 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.260623+0000", 2024-09-21T14:11:02.363 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:02.364 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.364 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.364 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:11:02.364 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.260623+0000", 2024-09-21T14:11:02.364 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:02.364 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.364 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.364 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:11:02.364 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.260624+0000", 2024-09-21T14:11:02.364 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8899999999999998e-07 2024-09-21T14:11:02.364 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.364 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.364 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:11:02.364 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.260635+0000", 2024-09-21T14:11:02.365 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0477e-05 2024-09-21T14:11:02.365 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.365 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.365 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:11:02.365 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.260648+0000", 2024-09-21T14:11:02.365 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3455000000000001e-05 2024-09-21T14:11:02.365 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.365 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.365 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:11:02.365 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.260658+0000", 2024-09-21T14:11:02.365 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.4669999999999993e-06 2024-09-21T14:11:02.365 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.365 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.365 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:11:02.365 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.260718+0000", 2024-09-21T14:11:02.366 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9865999999999999e-05 2024-09-21T14:11:02.366 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.366 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.366 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:11:02.366 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.260745+0000", 2024-09-21T14:11:02.366 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7664000000000001e-05 2024-09-21T14:11:02.366 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.366 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.366 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:11:02.366 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.260776+0000", 2024-09-21T14:11:02.366 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0945999999999998e-05 2024-09-21T14:11:02.366 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:02.366 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:11:02.366 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:02.366 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.367 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.367 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "MOSDScrubReserve(3.8,RELEASE,e:558,from:1,reservation_nonce:0)", 2024-09-21T14:11:02.367 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:10:52.261879+0000", 2024-09-21T14:11:02.367 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 10.022923463, 2024-09-21T14:11:02.367 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014642100000000001, 2024-09-21T14:11:02.367 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:11:02.367 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:11:02.367 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T14:11:02.367 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:11:02.367 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.367 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:11:02.367 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.261879+0000", 2024-09-21T14:11:02.367 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:02.367 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.367 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.368 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:11:02.368 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.261879+0000", 2024-09-21T14:11:02.368 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:02.368 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.368 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.368 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:11:02.368 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.261880+0000", 2024-09-21T14:11:02.368 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7300000000000001e-07 2024-09-21T14:11:02.368 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.368 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.368 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:11:02.368 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.261882+0000", 2024-09-21T14:11:02.369 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3870000000000002e-06 2024-09-21T14:11:02.369 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.369 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.369 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:11:02.369 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.261887+0000", 2024-09-21T14:11:02.369 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.412e-06 2024-09-21T14:11:02.369 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.369 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.369 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:11:02.369 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.261894+0000", 2024-09-21T14:11:02.369 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6919999999999996e-06 2024-09-21T14:11:02.369 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.369 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.369 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:11:02.369 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.261990+0000", 2024-09-21T14:11:02.370 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.5779999999999994e-05 2024-09-21T14:11:02.370 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.370 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.370 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:11:02.370 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.262008+0000", 2024-09-21T14:11:02.370 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8275999999999999e-05 2024-09-21T14:11:02.370 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.370 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.370 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:11:02.370 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.262026+0000", 2024-09-21T14:11:02.370 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7401000000000002e-05 2024-09-21T14:11:02.370 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:02.370 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:11:02.370 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:02.371 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.371 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.371 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "MOSDScrubReserve(1.d,QREQUEST,e:558,from:4,reservation_nonce:5)", 2024-09-21T14:11:02.371 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:10:52.637443+0000", 2024-09-21T14:11:02.371 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 9.6473600350000002, 2024-09-21T14:11:02.371 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00016173000000000001, 2024-09-21T14:11:02.371 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:11:02.371 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:11:02.371 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T14:11:02.371 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:11:02.371 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.372 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:11:02.372 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.637443+0000", 2024-09-21T14:11:02.372 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:02.372 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.372 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.372 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:11:02.372 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.637443+0000", 2024-09-21T14:11:02.372 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:02.372 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.372 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.372 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:11:02.372 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.637444+0000", 2024-09-21T14:11:02.372 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.327e-06 2024-09-21T14:11:02.372 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.373 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.373 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:11:02.373 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.637451+0000", 2024-09-21T14:11:02.373 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5340000000000002e-06 2024-09-21T14:11:02.373 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.373 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.373 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:11:02.373 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.637468+0000", 2024-09-21T14:11:02.373 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6399e-05 2024-09-21T14:11:02.373 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.373 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.373 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:11:02.373 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.637475+0000", 2024-09-21T14:11:02.373 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3189999999999997e-06 2024-09-21T14:11:02.374 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.374 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.374 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:11:02.374 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.637535+0000", 2024-09-21T14:11:02.374 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0100999999999999e-05 2024-09-21T14:11:02.374 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.374 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.374 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:11:02.374 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.637560+0000", 2024-09-21T14:11:02.374 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5099000000000001e-05 2024-09-21T14:11:02.374 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.374 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.374 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:11:02.374 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.637604+0000", 2024-09-21T14:11:02.374 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3951e-05 2024-09-21T14:11:02.375 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:02.375 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:11:02.375 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:02.375 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.375 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.375 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "replica_scrub(pg: 1.d,from:0'0,to:0'0,epoch:558/491,start:1:b0000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-21T14:11:02.375 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:10:52.638268+0000", 2024-09-21T14:11:02.375 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 9.6465349870000008, 2024-09-21T14:11:02.375 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014253700000000001, 2024-09-21T14:11:02.375 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:11:02.375 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:11:02.375 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T14:11:02.375 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:11:02.375 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.375 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:11:02.376 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.638268+0000", 2024-09-21T14:11:02.376 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:02.376 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.376 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.376 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:11:02.376 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.638268+0000", 2024-09-21T14:11:02.376 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:02.376 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.376 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.376 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:11:02.376 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.638269+0000", 2024-09-21T14:11:02.376 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.2999999999999999e-07 2024-09-21T14:11:02.376 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.376 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.376 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:11:02.377 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.638282+0000", 2024-09-21T14:11:02.377 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.324e-05 2024-09-21T14:11:02.377 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.377 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.377 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:11:02.377 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.638296+0000", 2024-09-21T14:11:02.377 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4173e-05 2024-09-21T14:11:02.377 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.377 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.377 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:11:02.377 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.638303+0000", 2024-09-21T14:11:02.377 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4510000000000003e-06 2024-09-21T14:11:02.377 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.377 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.377 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:11:02.378 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.638369+0000", 2024-09-21T14:11:02.378 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5513999999999996e-05 2024-09-21T14:11:02.378 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.378 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.378 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:11:02.378 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.638385+0000", 2024-09-21T14:11:02.378 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6130999999999999e-05 2024-09-21T14:11:02.378 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.378 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.378 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:11:02.378 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.638410+0000", 2024-09-21T14:11:02.378 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5097999999999999e-05 2024-09-21T14:11:02.378 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:02.379 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:11:02.379 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:02.379 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.379 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.379 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "MOSDScrubReserve(1.d,RELEASE,e:558,from:4,reservation_nonce:0)", 2024-09-21T14:11:02.379 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:10:52.639341+0000", 2024-09-21T14:11:02.379 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 9.6454618530000005, 2024-09-21T14:11:02.379 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014400099999999999, 2024-09-21T14:11:02.379 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:11:02.379 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:11:02.379 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T14:11:02.379 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:11:02.379 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.379 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:11:02.380 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.639341+0000", 2024-09-21T14:11:02.380 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:02.380 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.380 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.380 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:11:02.380 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.639341+0000", 2024-09-21T14:11:02.380 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:02.380 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.380 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.380 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:11:02.380 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.639341+0000", 2024-09-21T14:11:02.380 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5000000000000003e-07 2024-09-21T14:11:02.380 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.381 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.381 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:11:02.381 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.639345+0000", 2024-09-21T14:11:02.381 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.326e-06 2024-09-21T14:11:02.381 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.381 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.381 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:11:02.381 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.639353+0000", 2024-09-21T14:11:02.381 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9149999999999996e-06 2024-09-21T14:11:02.381 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.381 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.381 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:11:02.381 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.639360+0000", 2024-09-21T14:11:02.382 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.0319999999999998e-06 2024-09-21T14:11:02.382 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.382 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.382 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:11:02.382 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.639442+0000", 2024-09-21T14:11:02.382 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2322000000000002e-05 2024-09-21T14:11:02.382 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.382 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.382 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:11:02.382 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.639464+0000", 2024-09-21T14:11:02.382 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1974e-05 2024-09-21T14:11:02.382 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.383 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.383 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:11:02.383 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:52.639485+0000", 2024-09-21T14:11:02.383 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0882000000000001e-05 2024-09-21T14:11:02.383 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:02.383 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:11:02.383 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:02.383 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.383 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.383 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "MOSDScrubReserve(4.11,QREQUEST,e:559,from:3,reservation_nonce:9)", 2024-09-21T14:11:02.383 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:10:53.410916+0000", 2024-09-21T14:11:02.383 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 8.8738869909999991, 2024-09-21T14:11:02.383 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0017350620000000001, 2024-09-21T14:11:02.383 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:11:02.384 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:11:02.384 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T14:11:02.384 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:11:02.384 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.384 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:11:02.384 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:53.410916+0000", 2024-09-21T14:11:02.384 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:02.384 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.384 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.384 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:11:02.384 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:53.410916+0000", 2024-09-21T14:11:02.384 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:02.384 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.384 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.385 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:11:02.385 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:53.410917+0000", 2024-09-21T14:11:02.385 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1650000000000001e-06 2024-09-21T14:11:02.385 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.385 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.385 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:11:02.385 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:53.410924+0000", 2024-09-21T14:11:02.385 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9070000000000001e-06 2024-09-21T14:11:02.385 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.385 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.385 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:11:02.385 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:53.410939+0000", 2024-09-21T14:11:02.385 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5586e-05 2024-09-21T14:11:02.385 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.386 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.386 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:11:02.386 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:53.410948+0000", 2024-09-21T14:11:02.386 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8619999999999992e-06 2024-09-21T14:11:02.386 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.386 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.386 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:11:02.386 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:53.411023+0000", 2024-09-21T14:11:02.386 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4740000000000006e-05 2024-09-21T14:11:02.386 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.386 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.386 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op must wait for map", 2024-09-21T14:11:02.386 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:53.411033+0000", 2024-09-21T14:11:02.387 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.5950000000000005e-06 2024-09-21T14:11:02.387 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.387 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.387 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:11:02.387 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:53.412604+0000", 2024-09-21T14:11:02.387 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0015712879999999999 2024-09-21T14:11:02.387 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.387 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.387 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:11:02.387 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:53.412619+0000", 2024-09-21T14:11:02.387 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5170999999999999e-05 2024-09-21T14:11:02.387 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.387 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.387 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:11:02.388 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:53.412651+0000", 2024-09-21T14:11:02.388 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1748000000000002e-05 2024-09-21T14:11:02.388 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:02.388 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:11:02.388 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:02.388 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.388 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.388 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "MOSDScrubReserve(4.11,RELEASE,e:559,from:3,reservation_nonce:0)", 2024-09-21T14:11:02.388 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:10:53.415803+0000", 2024-09-21T14:11:02.388 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 8.8689993319999996, 2024-09-21T14:11:02.388 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00016775000000000001, 2024-09-21T14:11:02.388 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:11:02.388 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:11:02.389 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T14:11:02.389 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:11:02.389 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.389 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:11:02.389 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:53.415803+0000", 2024-09-21T14:11:02.389 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:02.389 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.389 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.389 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:11:02.389 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:53.415803+0000", 2024-09-21T14:11:02.389 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:02.389 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.389 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.389 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:11:02.390 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:53.415804+0000", 2024-09-21T14:11:02.390 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9000000000000002e-07 2024-09-21T14:11:02.390 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.390 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.390 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:11:02.390 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:53.415807+0000", 2024-09-21T14:11:02.390 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1209999999999998e-06 2024-09-21T14:11:02.390 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.390 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.390 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:11:02.390 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:53.415823+0000", 2024-09-21T14:11:02.390 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5682e-05 2024-09-21T14:11:02.390 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.390 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.391 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:11:02.391 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:53.415830+0000", 2024-09-21T14:11:02.391 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7680000000000008e-06 2024-09-21T14:11:02.391 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.391 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.391 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:11:02.391 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:53.415931+0000", 2024-09-21T14:11:02.391 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010073000000000001 2024-09-21T14:11:02.391 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.391 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.391 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:11:02.391 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:53.415952+0000", 2024-09-21T14:11:02.391 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0687999999999999e-05 2024-09-21T14:11:02.392 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.392 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.392 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:11:02.392 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:53.415971+0000", 2024-09-21T14:11:02.392 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9371000000000001e-05 2024-09-21T14:11:02.392 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:02.392 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:11:02.392 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:02.392 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.392 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.392 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "replica_scrub(pg: 3.11,from:0'0,to:61'1,epoch:559/491,start:3:88000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-21T14:11:02.392 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:10:53.655452+0000", 2024-09-21T14:11:02.392 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 8.6293506909999991, 2024-09-21T14:11:02.392 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015199500000000001, 2024-09-21T14:11:02.392 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:11:02.393 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:11:02.393 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T14:11:02.393 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:11:02.393 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.393 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:11:02.393 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:53.655452+0000", 2024-09-21T14:11:02.393 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:02.393 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.393 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.393 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:11:02.393 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:53.655452+0000", 2024-09-21T14:11:02.393 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:02.393 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.394 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.394 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:11:02.394 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:53.655453+0000", 2024-09-21T14:11:02.394 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2699999999999999e-07 2024-09-21T14:11:02.394 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.394 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.394 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:11:02.394 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:53.655457+0000", 2024-09-21T14:11:02.394 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6709999999999998e-06 2024-09-21T14:11:02.394 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.394 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.394 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:11:02.394 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:53.655472+0000", 2024-09-21T14:11:02.394 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4438e-05 2024-09-21T14:11:02.395 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.395 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.395 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:11:02.395 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:53.655480+0000", 2024-09-21T14:11:02.395 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3399999999999998e-06 2024-09-21T14:11:02.395 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.395 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.395 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:11:02.395 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:53.655562+0000", 2024-09-21T14:11:02.395 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1602999999999998e-05 2024-09-21T14:11:02.395 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.395 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.395 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:11:02.395 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:53.655573+0000", 2024-09-21T14:11:02.395 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1459e-05 2024-09-21T14:11:02.396 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.396 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.396 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:11:02.396 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:53.655604+0000", 2024-09-21T14:11:02.396 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0857e-05 2024-09-21T14:11:02.396 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:02.396 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:11:02.396 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:02.396 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.396 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.396 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "MOSDScrubReserve(6.1c,QREQUEST,e:559,from:5,reservation_nonce:5)", 2024-09-21T14:11:02.396 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:10:56.316458+0000", 2024-09-21T14:11:02.396 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 5.9683447110000003, 2024-09-21T14:11:02.396 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000224219, 2024-09-21T14:11:02.396 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:11:02.397 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:11:02.397 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T14:11:02.397 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:11:02.397 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.397 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:11:02.397 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:56.316458+0000", 2024-09-21T14:11:02.397 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:02.397 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.397 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.397 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:11:02.397 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:56.316458+0000", 2024-09-21T14:11:02.397 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:02.397 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.397 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.397 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:11:02.398 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:56.316459+0000", 2024-09-21T14:11:02.398 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4410000000000001e-06 2024-09-21T14:11:02.398 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.398 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.398 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:11:02.398 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:56.316467+0000", 2024-09-21T14:11:02.398 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0490000000000004e-06 2024-09-21T14:11:02.398 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.398 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.398 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:11:02.398 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:56.316486+0000", 2024-09-21T14:11:02.398 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8357000000000001e-05 2024-09-21T14:11:02.398 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.398 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.399 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:11:02.399 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:56.316494+0000", 2024-09-21T14:11:02.399 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5210000000000002e-06 2024-09-21T14:11:02.399 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.399 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.399 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:11:02.399 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:56.316618+0000", 2024-09-21T14:11:02.399 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000123287 2024-09-21T14:11:02.399 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.399 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.399 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:11:02.399 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:56.316635+0000", 2024-09-21T14:11:02.399 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7549000000000001e-05 2024-09-21T14:11:02.399 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.400 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.400 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:11:02.400 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:56.316682+0000", 2024-09-21T14:11:02.400 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7015000000000003e-05 2024-09-21T14:11:02.400 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:02.400 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:11:02.400 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:02.400 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.400 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.400 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "MOSDScrubReserve(6.1c,RELEASE,e:561,from:5,reservation_nonce:0)", 2024-09-21T14:11:02.400 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:10:56.320615+0000", 2024-09-21T14:11:02.400 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 5.9641872009999997, 2024-09-21T14:11:02.400 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00069374499999999995, 2024-09-21T14:11:02.400 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:11:02.401 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:11:02.401 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T14:11:02.401 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:11:02.401 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.401 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:11:02.401 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:56.320615+0000", 2024-09-21T14:11:02.401 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:02.401 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.401 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.401 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:11:02.401 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:56.320615+0000", 2024-09-21T14:11:02.401 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:02.401 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.401 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.402 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:11:02.402 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:56.320616+0000", 2024-09-21T14:11:02.402 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6199999999999998e-07 2024-09-21T14:11:02.402 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.402 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.402 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:11:02.402 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:56.320619+0000", 2024-09-21T14:11:02.402 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3330000000000001e-06 2024-09-21T14:11:02.402 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.402 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.402 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:11:02.402 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:56.320626+0000", 2024-09-21T14:11:02.402 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5849999999999996e-06 2024-09-21T14:11:02.403 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.403 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.403 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:11:02.403 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:56.320633+0000", 2024-09-21T14:11:02.403 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7789999999999998e-06 2024-09-21T14:11:02.403 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.403 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.403 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:11:02.403 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:56.321273+0000", 2024-09-21T14:11:02.403 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00064061800000000003 2024-09-21T14:11:02.403 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.403 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.403 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:11:02.403 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:56.321294+0000", 2024-09-21T14:11:02.404 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0874999999999999e-05 2024-09-21T14:11:02.404 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.404 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.404 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:11:02.404 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:10:56.321309+0000", 2024-09-21T14:11:02.404 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5092999999999999e-05 2024-09-21T14:11:02.404 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:02.404 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:11:02.404 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:02.404 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.404 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.404 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "MOSDScrubReserve(6.3,QREQUEST,e:561,from:6,reservation_nonce:3)", 2024-09-21T14:11:02.404 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:11:01.095950+0000", 2024-09-21T14:11:02.404 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 1.188852255, 2024-09-21T14:11:02.405 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00018415599999999999, 2024-09-21T14:11:02.405 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:11:02.405 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:11:02.405 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T14:11:02.405 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:11:02.405 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.405 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:11:02.405 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:11:01.095950+0000", 2024-09-21T14:11:02.405 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:02.405 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.405 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.405 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:11:02.405 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:11:01.095950+0000", 2024-09-21T14:11:02.405 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:02.406 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.406 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.406 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:11:02.406 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:11:01.095951+0000", 2024-09-21T14:11:02.406 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0999999999999997e-07 2024-09-21T14:11:02.406 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.406 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.406 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:11:02.406 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:11:01.095956+0000", 2024-09-21T14:11:02.406 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9359999999999998e-06 2024-09-21T14:11:02.406 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.406 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.406 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:11:02.407 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:11:01.095969+0000", 2024-09-21T14:11:02.407 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2924e-05 2024-09-21T14:11:02.407 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.407 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.407 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:11:02.407 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:11:01.095977+0000", 2024-09-21T14:11:02.407 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8990000000000001e-06 2024-09-21T14:11:02.407 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.407 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.407 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:11:02.407 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:11:01.096081+0000", 2024-09-21T14:11:02.407 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010353000000000001 2024-09-21T14:11:02.407 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.408 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.408 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:11:02.408 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:11:01.096098+0000", 2024-09-21T14:11:02.408 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6918999999999998e-05 2024-09-21T14:11:02.408 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.408 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.408 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:11:02.408 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:11:01.096135+0000", 2024-09-21T14:11:02.408 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7038e-05 2024-09-21T14:11:02.408 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:02.408 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:11:02.408 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:02.409 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.409 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.409 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "MOSDScrubReserve(5.17,QREQUEST,e:562,from:1,reservation_nonce:3)", 2024-09-21T14:11:02.409 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:11:01.300528+0000", 2024-09-21T14:11:02.409 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.98427447800000001, 2024-09-21T14:11:02.409 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000372383, 2024-09-21T14:11:02.409 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:11:02.409 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:11:02.409 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T14:11:02.409 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:11:02.409 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.409 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:11:02.409 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:11:01.300528+0000", 2024-09-21T14:11:02.410 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:02.410 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.410 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.410 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:11:02.410 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:11:01.300528+0000", 2024-09-21T14:11:02.410 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:02.410 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.410 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.410 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:11:02.410 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:11:01.300530+0000", 2024-09-21T14:11:02.410 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6470000000000001e-06 2024-09-21T14:11:02.410 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.410 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.410 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:11:02.410 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:11:01.300539+0000", 2024-09-21T14:11:02.411 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.5599999999999999e-06 2024-09-21T14:11:02.411 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.411 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.411 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:11:02.411 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:11:01.300565+0000", 2024-09-21T14:11:02.411 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.567e-05 2024-09-21T14:11:02.411 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.411 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.411 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:11:02.411 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:11:01.300578+0000", 2024-09-21T14:11:02.411 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3086e-05 2024-09-21T14:11:02.411 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.411 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.411 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:11:02.412 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:11:01.300738+0000", 2024-09-21T14:11:02.412 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015976400000000001 2024-09-21T14:11:02.412 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.412 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.412 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:11:02.412 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:11:01.300788+0000", 2024-09-21T14:11:02.412 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9660000000000002e-05 2024-09-21T14:11:02.412 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.412 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.412 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:11:02.412 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:11:01.300901+0000", 2024-09-21T14:11:02.412 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000112996 2024-09-21T14:11:02.412 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:02.412 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:11:02.412 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:02.413 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.413 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.413 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "replica_scrub(pg: 5.17,from:0'0,to:0'0,epoch:562/496,start:5:e8000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-21T14:11:02.413 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:11:01.301715+0000", 2024-09-21T14:11:02.413 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.983087767, 2024-09-21T14:11:02.413 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00031615199999999999, 2024-09-21T14:11:02.413 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:11:02.413 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:11:02.413 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T14:11:02.413 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:11:02.413 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.413 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:11:02.413 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:11:01.301715+0000", 2024-09-21T14:11:02.413 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:02.413 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.414 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.414 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:11:02.414 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:11:01.301715+0000", 2024-09-21T14:11:02.414 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:02.414 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.414 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.414 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:11:02.414 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:11:01.301716+0000", 2024-09-21T14:11:02.414 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6029999999999999e-06 2024-09-21T14:11:02.414 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.414 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.414 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:11:02.414 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:11:01.301728+0000", 2024-09-21T14:11:02.414 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1086e-05 2024-09-21T14:11:02.415 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.415 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.415 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:11:02.415 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:11:01.301757+0000", 2024-09-21T14:11:02.415 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.987e-05 2024-09-21T14:11:02.415 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.415 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.415 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:11:02.415 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:11:01.301773+0000", 2024-09-21T14:11:02.415 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5233e-05 2024-09-21T14:11:02.415 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.415 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.415 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:11:02.415 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:11:01.301937+0000", 2024-09-21T14:11:02.416 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00016404500000000001 2024-09-21T14:11:02.416 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.416 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.416 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:11:02.416 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:11:01.301972+0000", 2024-09-21T14:11:02.416 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5661999999999998e-05 2024-09-21T14:11:02.416 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.416 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.416 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:11:02.416 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:11:01.302031+0000", 2024-09-21T14:11:02.416 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8653000000000002e-05 2024-09-21T14:11:02.416 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:02.416 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:11:02.416 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:02.416 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.417 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.417 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "MOSDScrubReserve(5.d,RELEASE,e:562,from:2,reservation_nonce:0)", 2024-09-21T14:11:02.417 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:11:01.591326+0000", 2024-09-21T14:11:02.417 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.69347653099999995, 2024-09-21T14:11:02.417 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015420000000000001, 2024-09-21T14:11:02.417 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:11:02.417 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:11:02.417 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T14:11:02.417 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:11:02.417 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.417 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:11:02.417 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:11:01.591326+0000", 2024-09-21T14:11:02.417 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:02.417 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.417 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.418 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:11:02.418 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:11:01.591326+0000", 2024-09-21T14:11:02.418 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:02.418 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.418 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.418 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:11:02.418 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:11:01.591327+0000", 2024-09-21T14:11:02.418 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7500000000000001e-07 2024-09-21T14:11:02.418 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.418 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.418 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:11:02.418 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:11:01.591329+0000", 2024-09-21T14:11:02.418 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.39e-06 2024-09-21T14:11:02.418 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.419 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.419 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:11:02.419 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:11:01.591343+0000", 2024-09-21T14:11:02.419 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4579999999999999e-05 2024-09-21T14:11:02.419 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.419 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.419 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:11:02.419 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:11:01.591350+0000", 2024-09-21T14:11:02.419 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2149999999999998e-06 2024-09-21T14:11:02.419 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.419 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.419 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:11:02.419 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:11:01.591443+0000", 2024-09-21T14:11:02.419 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3541000000000004e-05 2024-09-21T14:11:02.419 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.420 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.420 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:11:02.420 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:11:01.591461+0000", 2024-09-21T14:11:02.420 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7878999999999999e-05 2024-09-21T14:11:02.420 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.420 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.420 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:11:02.420 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:11:01.591480+0000", 2024-09-21T14:11:02.420 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9219999999999999e-05 2024-09-21T14:11:02.420 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:02.420 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:11:02.420 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:02.420 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.420 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.420 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "MOSDScrubReserve(4.1e,QREQUEST,e:562,from:0,reservation_nonce:3)", 2024-09-21T14:11:02.421 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:11:02.067235+0000", 2024-09-21T14:11:02.421 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.21756731300000001, 2024-09-21T14:11:02.421 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014882799999999999, 2024-09-21T14:11:02.421 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:11:02.421 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:11:02.421 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T14:11:02.421 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:11:02.421 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.421 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:11:02.421 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:11:02.067235+0000", 2024-09-21T14:11:02.421 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:02.421 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.421 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.421 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:11:02.421 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:11:02.067235+0000", 2024-09-21T14:11:02.422 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:02.422 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.422 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.422 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:11:02.422 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:11:02.067237+0000", 2024-09-21T14:11:02.422 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6619999999999999e-06 2024-09-21T14:11:02.422 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.422 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.422 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:11:02.422 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:11:02.067245+0000", 2024-09-21T14:11:02.422 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2460000000000003e-06 2024-09-21T14:11:02.422 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.422 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.423 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:11:02.423 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:11:02.067264+0000", 2024-09-21T14:11:02.423 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8258999999999999e-05 2024-09-21T14:11:02.423 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.423 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.423 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:11:02.423 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:11:02.067271+0000", 2024-09-21T14:11:02.423 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7330000000000003e-06 2024-09-21T14:11:02.423 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.423 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.423 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:11:02.423 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:11:02.067338+0000", 2024-09-21T14:11:02.423 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6988000000000003e-05 2024-09-21T14:11:02.423 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.423 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.424 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:11:02.424 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:11:02.067353+0000", 2024-09-21T14:11:02.424 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5162e-05 2024-09-21T14:11:02.424 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:02.424 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:02.424 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:11:02.424 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:11:02.067384+0000", 2024-09-21T14:11:02.424 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0778000000000002e-05 2024-09-21T14:11:02.424 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:02.424 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:11:02.424 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:02.424 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:02.424 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:11:02.425 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T14:11:02.425 INFO:teuthology.orchestra.run.smithi049.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-09-21T14:11:02.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:02 smithi187 ceph-mon[33032]: 5.17 scrub starts 2024-09-21T14:11:02.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:02 smithi187 ceph-mon[33032]: 5.17 scrub ok 2024-09-21T14:11:02.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:02 smithi187 ceph-mon[33032]: 5.d scrub starts 2024-09-21T14:11:02.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:02 smithi187 ceph-mon[33032]: 5.d scrub ok 2024-09-21T14:11:02.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:02 smithi187 ceph-mon[33032]: 4.1e scrub starts 2024-09-21T14:11:02.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:02 smithi187 ceph-mon[33032]: 4.1e scrub ok 2024-09-21T14:11:02.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:02 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1339580072' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-21T14:11:02.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:02 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-21T14:11:02.668 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:02 smithi049 ceph-mon[40489]: 5.17 scrub starts 2024-09-21T14:11:02.668 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:02 smithi049 ceph-mon[40489]: 5.17 scrub ok 2024-09-21T14:11:02.668 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:02 smithi049 ceph-mon[40489]: 5.d scrub starts 2024-09-21T14:11:02.668 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:02 smithi049 ceph-mon[40489]: 5.d scrub ok 2024-09-21T14:11:02.668 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:02 smithi049 ceph-mon[40489]: 4.1e scrub starts 2024-09-21T14:11:02.668 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:02 smithi049 ceph-mon[40489]: 4.1e scrub ok 2024-09-21T14:11:02.668 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:02 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1339580072' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-21T14:11:02.668 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:02 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-21T14:11:02.668 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:02 smithi049 ceph-mon[30080]: 5.17 scrub starts 2024-09-21T14:11:02.669 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:02 smithi049 ceph-mon[30080]: 5.17 scrub ok 2024-09-21T14:11:02.669 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:02 smithi049 ceph-mon[30080]: 5.d scrub starts 2024-09-21T14:11:02.669 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:02 smithi049 ceph-mon[30080]: 5.d scrub ok 2024-09-21T14:11:02.669 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:02 smithi049 ceph-mon[30080]: 4.1e scrub starts 2024-09-21T14:11:02.669 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:02 smithi049 ceph-mon[30080]: 4.1e scrub ok 2024-09-21T14:11:02.669 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:02 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1339580072' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-21T14:11:02.669 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:02 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-21T14:11:02.981 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd reweight 6 1 2024-09-21T14:11:03.240 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:11:03.430 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T14:11:03.675 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:11:03.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:03 smithi187 ceph-mon[33032]: pgmap v1138: 152 pgs: 152 active+clean; 583 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-21T14:11:03.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:03 smithi187 ceph-mon[33032]: 5.15 scrub starts 2024-09-21T14:11:03.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:03 smithi187 ceph-mon[33032]: 5.15 scrub ok 2024-09-21T14:11:03.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:03 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-21T14:11:03.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:03 smithi187 ceph-mon[33032]: osdmap e563: 8 total, 8 up, 8 in 2024-09-21T14:11:03.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:03 smithi187 ceph-mon[33032]: 4.16 scrub starts 2024-09-21T14:11:03.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:03 smithi187 ceph-mon[33032]: 4.16 scrub ok 2024-09-21T14:11:03.706 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:03 smithi049 ceph-mon[40489]: pgmap v1138: 152 pgs: 152 active+clean; 583 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-21T14:11:03.706 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:03 smithi049 ceph-mon[40489]: 5.15 scrub starts 2024-09-21T14:11:03.706 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:03 smithi049 ceph-mon[40489]: 5.15 scrub ok 2024-09-21T14:11:03.706 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:03 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-21T14:11:03.706 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:03 smithi049 ceph-mon[40489]: osdmap e563: 8 total, 8 up, 8 in 2024-09-21T14:11:03.706 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:03 smithi049 ceph-mon[40489]: 4.16 scrub starts 2024-09-21T14:11:03.706 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:03 smithi049 ceph-mon[40489]: 4.16 scrub ok 2024-09-21T14:11:03.706 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:03 smithi049 ceph-mon[30080]: pgmap v1138: 152 pgs: 152 active+clean; 583 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-21T14:11:03.706 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:03 smithi049 ceph-mon[30080]: 5.15 scrub starts 2024-09-21T14:11:03.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:03 smithi049 ceph-mon[30080]: 5.15 scrub ok 2024-09-21T14:11:03.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:03 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-21T14:11:03.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:03 smithi049 ceph-mon[30080]: osdmap e563: 8 total, 8 up, 8 in 2024-09-21T14:11:03.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:03 smithi049 ceph-mon[30080]: 4.16 scrub starts 2024-09-21T14:11:03.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:03 smithi049 ceph-mon[30080]: 4.16 scrub ok 2024-09-21T14:11:04.347 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T14:11:04.347 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-21T14:11:04.347 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 0 2024-09-21T14:11:04.347 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T14:11:04.383 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.6 to 1 (10000) 2024-09-21T14:11:04.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:04 smithi187 ceph-mon[33032]: 3.4 scrub starts 2024-09-21T14:11:04.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:04 smithi187 ceph-mon[33032]: 3.4 scrub ok 2024-09-21T14:11:04.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:04 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1588095828' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T14:11:04.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:04 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T14:11:04.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:04 smithi049 ceph-mon[30080]: 3.4 scrub starts 2024-09-21T14:11:04.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:04 smithi049 ceph-mon[30080]: 3.4 scrub ok 2024-09-21T14:11:04.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:04 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1588095828' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T14:11:04.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:04 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T14:11:04.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:04 smithi049 ceph-mon[40489]: 3.4 scrub starts 2024-09-21T14:11:04.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:04 smithi049 ceph-mon[40489]: 3.4 scrub ok 2024-09-21T14:11:04.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:04 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1588095828' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T14:11:04.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:04 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T14:11:05.082 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T14:11:05.236 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 6 1 2024-09-21T14:11:05.335 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:11:05.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:05 smithi049 ceph-mon[30080]: pgmap v1140: 152 pgs: 152 active+clean; 583 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 646 B/s rd, 0 op/s 2024-09-21T14:11:05.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:05 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T14:11:05.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:05 smithi049 ceph-mon[30080]: osdmap e564: 8 total, 8 up, 8 in 2024-09-21T14:11:05.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:05 smithi049 ceph-mon[30080]: 1.4 scrub starts 2024-09-21T14:11:05.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:05 smithi049 ceph-mon[30080]: 1.4 scrub ok 2024-09-21T14:11:05.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:05 smithi049 ceph-mon[30080]: 2.1 scrub starts 2024-09-21T14:11:05.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:05 smithi049 ceph-mon[30080]: 2.1 scrub ok 2024-09-21T14:11:05.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:05 smithi049 ceph-mon[40489]: pgmap v1140: 152 pgs: 152 active+clean; 583 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 646 B/s rd, 0 op/s 2024-09-21T14:11:05.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:05 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T14:11:05.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:05 smithi049 ceph-mon[40489]: osdmap e564: 8 total, 8 up, 8 in 2024-09-21T14:11:05.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:05 smithi049 ceph-mon[40489]: 1.4 scrub starts 2024-09-21T14:11:05.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:05 smithi049 ceph-mon[40489]: 1.4 scrub ok 2024-09-21T14:11:05.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:05 smithi049 ceph-mon[40489]: 2.1 scrub starts 2024-09-21T14:11:05.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:05 smithi049 ceph-mon[40489]: 2.1 scrub ok 2024-09-21T14:11:05.508 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:11:05.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:05 smithi187 ceph-mon[33032]: pgmap v1140: 152 pgs: 152 active+clean; 583 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 646 B/s rd, 0 op/s 2024-09-21T14:11:05.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:05 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T14:11:05.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:05 smithi187 ceph-mon[33032]: osdmap e564: 8 total, 8 up, 8 in 2024-09-21T14:11:05.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:05 smithi187 ceph-mon[33032]: 1.4 scrub starts 2024-09-21T14:11:05.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:05 smithi187 ceph-mon[33032]: 1.4 scrub ok 2024-09-21T14:11:05.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:05 smithi187 ceph-mon[33032]: 2.1 scrub starts 2024-09-21T14:11:05.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:05 smithi187 ceph-mon[33032]: 2.1 scrub ok 2024-09-21T14:11:05.813 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T14:11:05.814 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-21T14:11:05.814 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-21T14:11:05.814 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-21T14:11:05.814 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T14:11:06.394 INFO:teuthology.orchestra.run.smithi049.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-09-21T14:11:06.513 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T14:11:06.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:06 smithi187 ceph-mon[33032]: osdmap e565: 8 total, 8 up, 8 in 2024-09-21T14:11:06.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:06 smithi187 ceph-mon[33032]: 3.b scrub starts 2024-09-21T14:11:06.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:06 smithi187 ceph-mon[33032]: 3.b scrub ok 2024-09-21T14:11:06.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:06 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/718018380' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-21T14:11:06.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:06 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-21T14:11:06.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:06 smithi049 ceph-mon[40489]: osdmap e565: 8 total, 8 up, 8 in 2024-09-21T14:11:06.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:06 smithi049 ceph-mon[40489]: 3.b scrub starts 2024-09-21T14:11:06.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:06 smithi049 ceph-mon[40489]: 3.b scrub ok 2024-09-21T14:11:06.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:06 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/718018380' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-21T14:11:06.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:06 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-21T14:11:06.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:06 smithi049 ceph-mon[30080]: osdmap e565: 8 total, 8 up, 8 in 2024-09-21T14:11:06.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:06 smithi049 ceph-mon[30080]: 3.b scrub starts 2024-09-21T14:11:06.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:06 smithi049 ceph-mon[30080]: 3.b scrub ok 2024-09-21T14:11:06.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:06 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/718018380' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-21T14:11:06.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:06 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-21T14:11:06.758 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T14:11:07.247 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd reweight 2 1 2024-09-21T14:11:07.271 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-21T14:11:07.272 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-21T14:11:07.272 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-21T14:11:07.272 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-21T14:11:07.272 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.272 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187592 9.5 9:a5cf25e0:::benchmark_data_smithi049_138504_object66184:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:11:07.272 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:41.268428+0000", 2024-09-21T14:11:07.272 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 566.00238141199998, 2024-09-21T14:11:07.272 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3375785200000001, 2024-09-21T14:11:07.272 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:11:07.272 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:11:07.272 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:11:07.272 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:11:07.273 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:11:07.273 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:11:07.273 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187592 2024-09-21T14:11:07.273 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.273 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:11:07.273 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.273 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:11:07.273 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268428+0000", 2024-09-21T14:11:07.273 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:07.273 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.273 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.273 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:11:07.273 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268428+0000", 2024-09-21T14:11:07.273 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:07.274 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.274 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.274 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:11:07.274 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268429+0000", 2024-09-21T14:11:07.274 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.466e-06 2024-09-21T14:11:07.274 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.274 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.274 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:11:07.274 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268435+0000", 2024-09-21T14:11:07.274 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5539999999999997e-06 2024-09-21T14:11:07.274 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.274 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.274 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:11:07.275 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268445+0000", 2024-09-21T14:11:07.275 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0366999999999999e-05 2024-09-21T14:11:07.275 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.275 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.275 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:11:07.275 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268452+0000", 2024-09-21T14:11:07.275 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8870000000000003e-06 2024-09-21T14:11:07.275 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.275 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.275 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:11:07.275 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.289931+0000", 2024-09-21T14:11:07.275 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.021479166000000001 2024-09-21T14:11:07.275 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.275 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.276 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:11:07.276 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.289937+0000", 2024-09-21T14:11:07.276 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4140000000000002e-06 2024-09-21T14:11:07.276 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.276 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.276 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:11:07.276 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.566606+0000", 2024-09-21T14:11:07.276 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2766695319999997 2024-09-21T14:11:07.276 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.276 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.276 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:11:07.276 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.569571+0000", 2024-09-21T14:11:07.276 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0029643489999999998 2024-09-21T14:11:07.276 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.277 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.277 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:11:07.277 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.573938+0000", 2024-09-21T14:11:07.277 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0043675099999999998 2024-09-21T14:11:07.277 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.277 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.277 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:11:07.277 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.589876+0000", 2024-09-21T14:11:07.277 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.015938249000000002 2024-09-21T14:11:07.277 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.277 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.277 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:11:07.277 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.597274+0000", 2024-09-21T14:11:07.277 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0073979509999999998 2024-09-21T14:11:07.277 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.278 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.278 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:11:07.278 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.598324+0000", 2024-09-21T14:11:07.278 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001050156 2024-09-21T14:11:07.278 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.278 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.278 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:11:07.278 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.599807+0000", 2024-09-21T14:11:07.278 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0014825439999999999 2024-09-21T14:11:07.278 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.278 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.278 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:11:07.278 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.606006+0000", 2024-09-21T14:11:07.279 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.006199375 2024-09-21T14:11:07.279 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:07.279 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:11:07.279 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:07.279 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.279 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.279 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187594 9.5 9:a4584f0e:::benchmark_data_smithi049_138504_object66186:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:11:07.279 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:41.268527+0000", 2024-09-21T14:11:07.279 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 566.00228189699999, 2024-09-21T14:11:07.279 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3614060480000001, 2024-09-21T14:11:07.279 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:11:07.279 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:11:07.279 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:11:07.279 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:11:07.279 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:11:07.280 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:11:07.280 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187594 2024-09-21T14:11:07.280 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.280 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:11:07.280 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.280 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:11:07.280 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268527+0000", 2024-09-21T14:11:07.280 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:07.280 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.280 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.280 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:11:07.280 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268527+0000", 2024-09-21T14:11:07.280 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:07.280 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.280 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.281 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:11:07.281 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268528+0000", 2024-09-21T14:11:07.281 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5299999999999999e-07 2024-09-21T14:11:07.281 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.281 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.281 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:11:07.281 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268531+0000", 2024-09-21T14:11:07.281 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.157e-06 2024-09-21T14:11:07.281 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.281 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.281 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:11:07.281 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268538+0000", 2024-09-21T14:11:07.281 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9290000000000001e-06 2024-09-21T14:11:07.281 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.282 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.282 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:11:07.282 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.268543+0000", 2024-09-21T14:11:07.282 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0409999999999997e-06 2024-09-21T14:11:07.282 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.282 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.282 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:11:07.282 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.289953+0000", 2024-09-21T14:11:07.282 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.021409811000000001 2024-09-21T14:11:07.282 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.282 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.282 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:11:07.282 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.289957+0000", 2024-09-21T14:11:07.282 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9489999999999998e-06 2024-09-21T14:11:07.283 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.283 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.283 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:11:07.283 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.606351+0000", 2024-09-21T14:11:07.283 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3163947599999997 2024-09-21T14:11:07.283 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.283 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.283 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:11:07.283 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.610368+0000", 2024-09-21T14:11:07.283 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0040163289999999999 2024-09-21T14:11:07.283 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.283 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.283 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:11:07.283 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.615560+0000", 2024-09-21T14:11:07.284 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0051918939999999998 2024-09-21T14:11:07.284 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.284 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.284 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:11:07.284 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.622425+0000", 2024-09-21T14:11:07.284 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0068655959999999999 2024-09-21T14:11:07.284 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.284 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.284 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:11:07.284 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.623509+0000", 2024-09-21T14:11:07.284 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001083669 2024-09-21T14:11:07.284 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.284 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.284 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:11:07.285 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.624009+0000", 2024-09-21T14:11:07.285 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00050001600000000005 2024-09-21T14:11:07.285 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.285 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.285 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:11:07.285 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.625282+0000", 2024-09-21T14:11:07.285 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0012730599999999999 2024-09-21T14:11:07.285 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.285 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.286 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:11:07.286 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.629933+0000", 2024-09-21T14:11:07.286 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0046513839999999997 2024-09-21T14:11:07.286 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:07.286 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:11:07.286 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:07.286 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.287 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.287 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187607 9.7 9:ea996e28:::benchmark_data_smithi049_138504_object66199:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:11:07.287 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:41.286116+0000", 2024-09-21T14:11:07.287 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 565.984693247, 2024-09-21T14:11:07.287 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0827087349999998, 2024-09-21T14:11:07.287 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:11:07.287 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:11:07.287 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:11:07.287 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:11:07.287 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:11:07.287 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:11:07.287 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187607 2024-09-21T14:11:07.287 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.288 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:11:07.288 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.288 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:11:07.288 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.286116+0000", 2024-09-21T14:11:07.288 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:07.288 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.288 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.288 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:11:07.288 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.286116+0000", 2024-09-21T14:11:07.288 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:07.288 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.288 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.288 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:11:07.288 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.286117+0000", 2024-09-21T14:11:07.289 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.097e-06 2024-09-21T14:11:07.289 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.289 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.289 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:11:07.289 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.286120+0000", 2024-09-21T14:11:07.289 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.224e-06 2024-09-21T14:11:07.289 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.289 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.289 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:11:07.289 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.286129+0000", 2024-09-21T14:11:07.289 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3659999999999992e-06 2024-09-21T14:11:07.289 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.289 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.289 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:11:07.290 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.286134+0000", 2024-09-21T14:11:07.290 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6130000000000003e-06 2024-09-21T14:11:07.290 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.290 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.290 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:11:07.290 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.286627+0000", 2024-09-21T14:11:07.290 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00049266000000000002 2024-09-21T14:11:07.290 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.290 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.290 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:11:07.290 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.286663+0000", 2024-09-21T14:11:07.290 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6164999999999997e-05 2024-09-21T14:11:07.290 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.291 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.291 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:11:07.291 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.330333+0000", 2024-09-21T14:11:07.291 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0436700029999999 2024-09-21T14:11:07.291 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.291 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.291 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:11:07.291 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.333462+0000", 2024-09-21T14:11:07.291 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003128881 2024-09-21T14:11:07.291 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.291 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.291 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,7", 2024-09-21T14:11:07.291 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.337950+0000", 2024-09-21T14:11:07.291 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0044878089999999997 2024-09-21T14:11:07.292 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.292 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.292 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:11:07.292 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353871+0000", 2024-09-21T14:11:07.292 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.015921270000000001 2024-09-21T14:11:07.292 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.292 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.292 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:11:07.292 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.360705+0000", 2024-09-21T14:11:07.292 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0068337110000000001 2024-09-21T14:11:07.292 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.292 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.292 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:11:07.292 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.362017+0000", 2024-09-21T14:11:07.292 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0013119749999999999 2024-09-21T14:11:07.293 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.293 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.293 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:11:07.293 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.364166+0000", 2024-09-21T14:11:07.293 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0021495030000000001 2024-09-21T14:11:07.293 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.293 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.293 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:11:07.293 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.368825+0000", 2024-09-21T14:11:07.293 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0046584579999999999 2024-09-21T14:11:07.293 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:07.293 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:11:07.294 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:07.294 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.294 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.294 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187609 9.5 9:a0a81d79:::benchmark_data_smithi049_138504_object66201:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:11:07.294 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:41.287003+0000", 2024-09-21T14:11:07.294 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 565.98380663600005, 2024-09-21T14:11:07.294 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3857519199999997, 2024-09-21T14:11:07.294 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:11:07.294 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:11:07.294 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:11:07.294 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:11:07.294 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:11:07.294 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:11:07.294 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187609 2024-09-21T14:11:07.295 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.295 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:11:07.295 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.295 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:11:07.295 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.287003+0000", 2024-09-21T14:11:07.295 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:07.295 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.295 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.295 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:11:07.295 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.287003+0000", 2024-09-21T14:11:07.295 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:07.295 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.295 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.295 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:11:07.296 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.287003+0000", 2024-09-21T14:11:07.296 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8400000000000003e-07 2024-09-21T14:11:07.296 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.296 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.296 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:11:07.296 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.287017+0000", 2024-09-21T14:11:07.296 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3831e-05 2024-09-21T14:11:07.296 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.296 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.296 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:11:07.296 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.287026+0000", 2024-09-21T14:11:07.296 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8389999999999996e-06 2024-09-21T14:11:07.296 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.296 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.297 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:11:07.297 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.287032+0000", 2024-09-21T14:11:07.297 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3509999999999998e-06 2024-09-21T14:11:07.297 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.297 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.297 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:11:07.297 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.289969+0000", 2024-09-21T14:11:07.297 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0029364399999999998 2024-09-21T14:11:07.297 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.297 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.297 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:11:07.297 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.289973+0000", 2024-09-21T14:11:07.297 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6339999999999999e-06 2024-09-21T14:11:07.298 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.298 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.298 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:11:07.298 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.631003+0000", 2024-09-21T14:11:07.298 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3410304320000002 2024-09-21T14:11:07.298 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.298 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.298 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:11:07.298 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.634397+0000", 2024-09-21T14:11:07.298 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0033938369999999998 2024-09-21T14:11:07.298 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.298 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.299 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:11:07.299 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.639017+0000", 2024-09-21T14:11:07.299 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0046205889999999996 2024-09-21T14:11:07.299 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.299 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.299 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:11:07.299 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.662137+0000", 2024-09-21T14:11:07.299 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.023119475 2024-09-21T14:11:07.299 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.299 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.299 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:11:07.299 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.663533+0000", 2024-09-21T14:11:07.299 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0013956159999999999 2024-09-21T14:11:07.299 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.300 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.300 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:11:07.300 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.664730+0000", 2024-09-21T14:11:07.300 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0011970139999999999 2024-09-21T14:11:07.300 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.300 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.300 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:11:07.300 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.666523+0000", 2024-09-21T14:11:07.300 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001793046 2024-09-21T14:11:07.300 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.300 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.300 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:11:07.300 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.672754+0000", 2024-09-21T14:11:07.300 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0062319319999999999 2024-09-21T14:11:07.301 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:07.301 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:11:07.301 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:07.301 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.301 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.301 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187621 9.5 9:a3e366d7:::benchmark_data_smithi049_138504_object66213:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:11:07.301 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:41.300495+0000", 2024-09-21T14:11:07.301 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 565.97031398800004, 2024-09-21T14:11:07.301 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3801383449999998, 2024-09-21T14:11:07.301 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:11:07.301 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:11:07.301 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:11:07.301 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:11:07.301 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:11:07.302 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:11:07.302 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187621 2024-09-21T14:11:07.302 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.302 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:11:07.302 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.302 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:11:07.302 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300495+0000", 2024-09-21T14:11:07.302 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:07.302 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.302 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.302 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:11:07.302 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300495+0000", 2024-09-21T14:11:07.302 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:07.302 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.303 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.303 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:11:07.303 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300497+0000", 2024-09-21T14:11:07.303 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3710000000000001e-06 2024-09-21T14:11:07.303 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.303 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.303 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:11:07.303 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300502+0000", 2024-09-21T14:11:07.303 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3569999999999997e-06 2024-09-21T14:11:07.303 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.303 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.303 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:11:07.303 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300511+0000", 2024-09-21T14:11:07.303 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.9509999999999995e-06 2024-09-21T14:11:07.304 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.304 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.304 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:11:07.304 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300517+0000", 2024-09-21T14:11:07.304 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2489999999999998e-06 2024-09-21T14:11:07.304 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.304 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.304 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:11:07.304 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300606+0000", 2024-09-21T14:11:07.304 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8794000000000004e-05 2024-09-21T14:11:07.304 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.304 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.304 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:11:07.304 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300616+0000", 2024-09-21T14:11:07.304 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0338e-05 2024-09-21T14:11:07.305 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.305 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.305 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:11:07.305 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.643702+0000", 2024-09-21T14:11:07.305 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.343085759 2024-09-21T14:11:07.305 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.305 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.305 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:11:07.305 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.648030+0000", 2024-09-21T14:11:07.305 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0043275859999999996 2024-09-21T14:11:07.305 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.305 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.305 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:11:07.305 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.653318+0000", 2024-09-21T14:11:07.306 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.005287936 2024-09-21T14:11:07.306 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.306 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.306 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:11:07.306 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.662422+0000", 2024-09-21T14:11:07.306 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0091040340000000004 2024-09-21T14:11:07.306 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.306 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.306 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:11:07.306 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.673842+0000", 2024-09-21T14:11:07.306 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.011420382999999999 2024-09-21T14:11:07.306 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.306 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.306 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:11:07.307 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.675052+0000", 2024-09-21T14:11:07.307 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0012104290000000001 2024-09-21T14:11:07.307 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.307 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.307 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:11:07.307 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.676489+0000", 2024-09-21T14:11:07.307 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0014364309999999999 2024-09-21T14:11:07.307 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.307 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.307 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:11:07.307 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.680634+0000", 2024-09-21T14:11:07.307 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.004144727 2024-09-21T14:11:07.307 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:07.307 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:11:07.308 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:07.308 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.308 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.308 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187623 9.7 9:e79a683b:::benchmark_data_smithi049_138504_object66215:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:11:07.308 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:41.300818+0000", 2024-09-21T14:11:07.308 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 565.96999158400001, 2024-09-21T14:11:07.308 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1014428919999997, 2024-09-21T14:11:07.308 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:11:07.308 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:11:07.308 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:11:07.308 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:11:07.308 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:11:07.308 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:11:07.308 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187623 2024-09-21T14:11:07.308 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.309 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:11:07.309 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.309 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:11:07.309 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300818+0000", 2024-09-21T14:11:07.309 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:07.309 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.309 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.309 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:11:07.309 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300818+0000", 2024-09-21T14:11:07.309 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:07.309 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.309 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.309 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:11:07.309 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300818+0000", 2024-09-21T14:11:07.310 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.54e-07 2024-09-21T14:11:07.310 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.310 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.310 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:11:07.310 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300822+0000", 2024-09-21T14:11:07.310 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6770000000000001e-06 2024-09-21T14:11:07.310 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.310 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.310 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:11:07.310 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300832+0000", 2024-09-21T14:11:07.310 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0202e-05 2024-09-21T14:11:07.310 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.310 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.310 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:11:07.311 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300847+0000", 2024-09-21T14:11:07.311 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4992e-05 2024-09-21T14:11:07.311 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.311 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.311 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:11:07.311 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300931+0000", 2024-09-21T14:11:07.311 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3577000000000003e-05 2024-09-21T14:11:07.311 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.311 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.311 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:11:07.311 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300940+0000", 2024-09-21T14:11:07.311 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8009999999999993e-06 2024-09-21T14:11:07.311 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.311 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.311 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:11:07.312 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.370531+0000", 2024-09-21T14:11:07.312 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0695910870000001 2024-09-21T14:11:07.312 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.312 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.312 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:11:07.312 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.373842+0000", 2024-09-21T14:11:07.312 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0033110259999999999 2024-09-21T14:11:07.312 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.312 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.312 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,7", 2024-09-21T14:11:07.312 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.378337+0000", 2024-09-21T14:11:07.312 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0044951100000000001 2024-09-21T14:11:07.312 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.312 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.312 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:11:07.313 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.394514+0000", 2024-09-21T14:11:07.313 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.016176877999999999 2024-09-21T14:11:07.313 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.313 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.313 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:11:07.313 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.395030+0000", 2024-09-21T14:11:07.313 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00051615000000000005 2024-09-21T14:11:07.313 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.313 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.313 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:11:07.313 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.395803+0000", 2024-09-21T14:11:07.313 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00077358799999999997 2024-09-21T14:11:07.313 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.313 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.314 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:11:07.314 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.397305+0000", 2024-09-21T14:11:07.314 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001501811 2024-09-21T14:11:07.314 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.314 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.314 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:11:07.314 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.402261+0000", 2024-09-21T14:11:07.314 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0049552390000000002 2024-09-21T14:11:07.314 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:07.314 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:11:07.314 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:07.314 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.314 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.314 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187624 9.5 9:a181e4d0:::benchmark_data_smithi049_138504_object66216:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:11:07.315 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:41.300871+0000", 2024-09-21T14:11:07.315 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 565.96993858099995, 2024-09-21T14:11:07.315 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4164464030000001, 2024-09-21T14:11:07.315 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:11:07.315 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:11:07.315 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:11:07.315 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:11:07.315 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:11:07.315 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:11:07.315 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187624 2024-09-21T14:11:07.315 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.315 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:11:07.315 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.315 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:11:07.315 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300871+0000", 2024-09-21T14:11:07.316 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:07.316 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.316 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.316 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:11:07.316 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300871+0000", 2024-09-21T14:11:07.316 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:07.316 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.316 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.316 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:11:07.316 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300871+0000", 2024-09-21T14:11:07.316 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8599999999999998e-07 2024-09-21T14:11:07.316 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.316 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.316 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:11:07.317 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300875+0000", 2024-09-21T14:11:07.317 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7720000000000001e-06 2024-09-21T14:11:07.317 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.317 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.317 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:11:07.317 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300886+0000", 2024-09-21T14:11:07.317 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1058e-05 2024-09-21T14:11:07.317 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.317 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.317 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:11:07.317 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.300892+0000", 2024-09-21T14:11:07.317 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.161e-06 2024-09-21T14:11:07.317 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.317 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.317 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:11:07.318 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.301287+0000", 2024-09-21T14:11:07.318 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00039522800000000002 2024-09-21T14:11:07.318 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.318 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.318 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:11:07.318 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.301297+0000", 2024-09-21T14:11:07.318 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6209999999999999e-06 2024-09-21T14:11:07.318 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.318 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.318 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:11:07.318 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.681021+0000", 2024-09-21T14:11:07.318 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3797241769999999 2024-09-21T14:11:07.318 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.319 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.319 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:11:07.319 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.685054+0000", 2024-09-21T14:11:07.319 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0040323909999999998 2024-09-21T14:11:07.319 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.319 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.319 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:11:07.319 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.689479+0000", 2024-09-21T14:11:07.319 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0044253740000000001 2024-09-21T14:11:07.319 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.319 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.319 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:11:07.319 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.710060+0000", 2024-09-21T14:11:07.319 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.020581051999999999 2024-09-21T14:11:07.320 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.320 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.320 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:11:07.320 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.710641+0000", 2024-09-21T14:11:07.320 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00058055800000000001 2024-09-21T14:11:07.320 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.320 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.320 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:11:07.320 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.711399+0000", 2024-09-21T14:11:07.320 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00075836299999999996 2024-09-21T14:11:07.320 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.320 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.320 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:11:07.320 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.712857+0000", 2024-09-21T14:11:07.320 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001458373 2024-09-21T14:11:07.321 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.321 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.321 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:11:07.321 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.717317+0000", 2024-09-21T14:11:07.321 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0044596890000000002 2024-09-21T14:11:07.321 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:07.321 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:11:07.321 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:07.321 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.321 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.321 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187626 9.7 9:e3df99cb:::benchmark_data_smithi049_138504_object66218:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:11:07.321 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:41.304965+0000", 2024-09-21T14:11:07.321 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 565.96584444500002, 2024-09-21T14:11:07.321 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1061633610000001, 2024-09-21T14:11:07.322 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:11:07.322 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:11:07.322 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:11:07.322 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:11:07.322 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:11:07.322 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:11:07.322 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187626 2024-09-21T14:11:07.322 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.322 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:11:07.322 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.322 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:11:07.322 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.304965+0000", 2024-09-21T14:11:07.322 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:07.322 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.322 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.323 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:11:07.323 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.304965+0000", 2024-09-21T14:11:07.323 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:07.323 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.323 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.323 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:11:07.323 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.304966+0000", 2024-09-21T14:11:07.323 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6619999999999999e-06 2024-09-21T14:11:07.323 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.323 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.323 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:11:07.323 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.304973+0000", 2024-09-21T14:11:07.324 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1759999999999998e-06 2024-09-21T14:11:07.324 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.324 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.324 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:11:07.324 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.304987+0000", 2024-09-21T14:11:07.324 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4372999999999999e-05 2024-09-21T14:11:07.324 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.324 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.324 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:11:07.324 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.304995+0000", 2024-09-21T14:11:07.324 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4109999999999996e-06 2024-09-21T14:11:07.324 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.324 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.324 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:11:07.325 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.305067+0000", 2024-09-21T14:11:07.325 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1833000000000002e-05 2024-09-21T14:11:07.325 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.325 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.325 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:11:07.325 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.305077+0000", 2024-09-21T14:11:07.325 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0161e-05 2024-09-21T14:11:07.325 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.325 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.325 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:11:07.325 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.382147+0000", 2024-09-21T14:11:07.325 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0770691709999998 2024-09-21T14:11:07.325 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.325 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.326 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:11:07.326 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.385209+0000", 2024-09-21T14:11:07.326 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0030624139999999998 2024-09-21T14:11:07.326 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.326 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.326 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,7", 2024-09-21T14:11:07.326 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.389597+0000", 2024-09-21T14:11:07.326 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.004388477 2024-09-21T14:11:07.326 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.326 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.326 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:11:07.326 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.403080+0000", 2024-09-21T14:11:07.326 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.013482984 2024-09-21T14:11:07.326 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.327 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.327 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:11:07.327 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.403583+0000", 2024-09-21T14:11:07.327 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00050300099999999999 2024-09-21T14:11:07.327 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.327 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.327 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:11:07.327 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.404464+0000", 2024-09-21T14:11:07.327 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00088034599999999995 2024-09-21T14:11:07.327 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.327 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.327 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:11:07.327 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.406114+0000", 2024-09-21T14:11:07.327 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0016504499999999999 2024-09-21T14:11:07.328 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.328 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.328 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:11:07.328 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.411128+0000", 2024-09-21T14:11:07.328 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0050139019999999998 2024-09-21T14:11:07.328 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:07.328 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:11:07.328 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:07.328 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.328 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.328 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187630 9.5 9:afb0e788:::benchmark_data_smithi049_138504_object66222:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:11:07.328 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:41.309934+0000", 2024-09-21T14:11:07.328 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 565.96087486399995, 2024-09-21T14:11:07.328 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4164949509999998, 2024-09-21T14:11:07.329 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:11:07.329 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:11:07.329 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:11:07.329 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:11:07.329 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:11:07.329 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:11:07.329 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187630 2024-09-21T14:11:07.329 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.329 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:11:07.329 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.329 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:11:07.329 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.309934+0000", 2024-09-21T14:11:07.329 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:07.329 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.329 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.330 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:11:07.330 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.309934+0000", 2024-09-21T14:11:07.330 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:07.330 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.330 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.330 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:11:07.330 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.309936+0000", 2024-09-21T14:11:07.330 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7460000000000001e-06 2024-09-21T14:11:07.330 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.330 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.330 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:11:07.330 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.309943+0000", 2024-09-21T14:11:07.330 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6579999999999996e-06 2024-09-21T14:11:07.330 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.331 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.331 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:11:07.331 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.309957+0000", 2024-09-21T14:11:07.331 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4496e-05 2024-09-21T14:11:07.331 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.331 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.331 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:11:07.331 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.309964+0000", 2024-09-21T14:11:07.331 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2219999999999996e-06 2024-09-21T14:11:07.331 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.331 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.331 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:11:07.331 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.310043+0000", 2024-09-21T14:11:07.332 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8801999999999995e-05 2024-09-21T14:11:07.332 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.332 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.332 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:11:07.332 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.310055+0000", 2024-09-21T14:11:07.332 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1637999999999999e-05 2024-09-21T14:11:07.332 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.332 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.332 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:11:07.332 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.693253+0000", 2024-09-21T14:11:07.332 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3831976189999997 2024-09-21T14:11:07.332 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.332 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.332 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:11:07.333 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.697465+0000", 2024-09-21T14:11:07.333 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0042119770000000004 2024-09-21T14:11:07.333 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.333 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.333 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:11:07.333 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.703657+0000", 2024-09-21T14:11:07.333 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0061928870000000002 2024-09-21T14:11:07.333 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.333 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.333 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:11:07.333 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.717616+0000", 2024-09-21T14:11:07.333 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.013958435999999999 2024-09-21T14:11:07.333 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.333 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.334 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:11:07.334 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.718555+0000", 2024-09-21T14:11:07.334 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00093888999999999995 2024-09-21T14:11:07.334 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.334 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.334 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:11:07.334 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.719633+0000", 2024-09-21T14:11:07.334 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0010787150000000001 2024-09-21T14:11:07.334 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.334 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.334 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:11:07.334 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.721631+0000", 2024-09-21T14:11:07.334 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0019973220000000002 2024-09-21T14:11:07.334 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.334 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.335 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:11:07.335 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.726429+0000", 2024-09-21T14:11:07.335 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0047985429999999997 2024-09-21T14:11:07.335 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:07.335 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:11:07.335 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:07.335 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.335 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.335 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187631 9.7 9:eb4378cd:::benchmark_data_smithi049_138504_object66223:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:11:07.335 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:41.310203+0000", 2024-09-21T14:11:07.335 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 565.96060635599997, 2024-09-21T14:11:07.335 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1434362220000001, 2024-09-21T14:11:07.335 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:11:07.335 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:11:07.336 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:11:07.336 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:11:07.336 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:11:07.336 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:11:07.336 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187631 2024-09-21T14:11:07.336 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.336 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:11:07.336 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.336 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:11:07.336 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.310203+0000", 2024-09-21T14:11:07.336 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:07.336 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.336 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.336 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:11:07.336 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.310203+0000", 2024-09-21T14:11:07.337 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:07.337 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.337 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.337 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:11:07.337 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.310204+0000", 2024-09-21T14:11:07.337 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3699999999999999e-07 2024-09-21T14:11:07.337 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.337 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.337 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:11:07.337 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.310209+0000", 2024-09-21T14:11:07.337 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7039999999999999e-06 2024-09-21T14:11:07.337 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.337 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.337 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:11:07.337 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.310219+0000", 2024-09-21T14:11:07.338 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7319999999999993e-06 2024-09-21T14:11:07.338 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.338 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.338 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:11:07.338 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.310226+0000", 2024-09-21T14:11:07.338 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4239999999999998e-06 2024-09-21T14:11:07.338 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.338 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.338 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:11:07.338 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.310279+0000", 2024-09-21T14:11:07.338 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3124999999999997e-05 2024-09-21T14:11:07.338 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.338 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.338 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:11:07.339 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.310288+0000", 2024-09-21T14:11:07.339 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3149999999999998e-06 2024-09-21T14:11:07.339 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.339 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.339 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:11:07.339 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.416040+0000", 2024-09-21T14:11:07.339 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1057518990000004 2024-09-21T14:11:07.339 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.339 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.339 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:11:07.339 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.419285+0000", 2024-09-21T14:11:07.339 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0032447399999999999 2024-09-21T14:11:07.339 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.339 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.339 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,7", 2024-09-21T14:11:07.340 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.423815+0000", 2024-09-21T14:11:07.340 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0045301509999999996 2024-09-21T14:11:07.340 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.340 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.340 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:11:07.340 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.439415+0000", 2024-09-21T14:11:07.340 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.015600334 2024-09-21T14:11:07.340 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.341 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.341 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:11:07.341 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.440566+0000", 2024-09-21T14:11:07.341 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0011505129999999999 2024-09-21T14:11:07.341 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.341 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.342 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:11:07.342 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.447676+0000", 2024-09-21T14:11:07.342 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0071103659999999999 2024-09-21T14:11:07.342 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.342 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.342 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:11:07.342 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.448927+0000", 2024-09-21T14:11:07.342 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0012508759999999999 2024-09-21T14:11:07.342 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.342 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.342 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:11:07.342 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.453639+0000", 2024-09-21T14:11:07.342 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0047121059999999998 2024-09-21T14:11:07.342 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:07.343 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:11:07.343 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:07.343 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.343 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.343 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187650 9.5 9:ab0f6d84:::benchmark_data_smithi049_138504_object66242:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:11:07.343 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:41.359224+0000", 2024-09-21T14:11:07.343 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 565.911584855, 2024-09-21T14:11:07.343 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4044763060000003, 2024-09-21T14:11:07.343 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:11:07.343 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:11:07.343 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:11:07.343 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:11:07.343 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:11:07.343 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:11:07.343 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187650 2024-09-21T14:11:07.344 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.344 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:11:07.344 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.344 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:11:07.344 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.359224+0000", 2024-09-21T14:11:07.344 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:07.344 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.344 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.344 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:11:07.344 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.359224+0000", 2024-09-21T14:11:07.344 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:07.344 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.344 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.344 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:11:07.345 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.359226+0000", 2024-09-21T14:11:07.345 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.423e-06 2024-09-21T14:11:07.345 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.345 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.345 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:11:07.345 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.359231+0000", 2024-09-21T14:11:07.345 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0980000000000001e-06 2024-09-21T14:11:07.345 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.345 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.345 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:11:07.345 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.359243+0000", 2024-09-21T14:11:07.345 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2316e-05 2024-09-21T14:11:07.345 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.345 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.346 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:11:07.346 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.359250+0000", 2024-09-21T14:11:07.346 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8820000000000003e-06 2024-09-21T14:11:07.346 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.346 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.346 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:11:07.346 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.359329+0000", 2024-09-21T14:11:07.346 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8541000000000006e-05 2024-09-21T14:11:07.346 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.346 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.346 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-21T14:11:07.346 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:41.359341+0000", 2024-09-21T14:11:07.346 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2238e-05 2024-09-21T14:11:07.346 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.346 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.347 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:11:07.347 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.726850+0000", 2024-09-21T14:11:07.347 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3675093619999998 2024-09-21T14:11:07.347 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.347 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.347 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:11:07.347 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.731153+0000", 2024-09-21T14:11:07.347 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0043027860000000003 2024-09-21T14:11:07.347 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.347 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.347 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:11:07.347 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.737448+0000", 2024-09-21T14:11:07.347 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0062951559999999997 2024-09-21T14:11:07.347 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.348 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.348 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:11:07.348 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.754776+0000", 2024-09-21T14:11:07.348 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.017328300000000001 2024-09-21T14:11:07.348 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.348 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.348 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:11:07.348 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.756004+0000", 2024-09-21T14:11:07.348 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001227337 2024-09-21T14:11:07.348 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.348 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.348 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:11:07.348 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.757990+0000", 2024-09-21T14:11:07.348 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0019857809999999998 2024-09-21T14:11:07.348 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.349 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.349 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:11:07.349 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.759321+0000", 2024-09-21T14:11:07.349 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001331203 2024-09-21T14:11:07.349 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.349 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.349 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:11:07.349 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.763701+0000", 2024-09-21T14:11:07.349 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0043798830000000002 2024-09-21T14:11:07.349 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:07.349 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:11:07.349 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:07.349 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.349 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.350 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187651 9.f 9:fed2c87a:::benchmark_data_smithi049_138504_object66243:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e347)", 2024-09-21T14:11:07.350 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:42.222057+0000", 2024-09-21T14:11:07.350 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 565.04875172599998, 2024-09-21T14:11:07.350 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1315866940000001, 2024-09-21T14:11:07.350 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:11:07.350 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:11:07.350 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:11:07.350 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:11:07.350 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:11:07.350 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:11:07.350 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187651 2024-09-21T14:11:07.350 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.350 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:11:07.350 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.350 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:11:07.351 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.222057+0000", 2024-09-21T14:11:07.351 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:07.351 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.351 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.351 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:11:07.351 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.222057+0000", 2024-09-21T14:11:07.351 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:07.351 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.351 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.351 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:11:07.351 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.222059+0000", 2024-09-21T14:11:07.351 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8029999999999999e-06 2024-09-21T14:11:07.351 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.351 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.352 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:11:07.352 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.222066+0000", 2024-09-21T14:11:07.352 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2670000000000003e-06 2024-09-21T14:11:07.352 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.352 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.352 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:11:07.352 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.222078+0000", 2024-09-21T14:11:07.352 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2899e-05 2024-09-21T14:11:07.352 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.352 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.352 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:11:07.352 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.222086+0000", 2024-09-21T14:11:07.352 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8080000000000005e-06 2024-09-21T14:11:07.352 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.353 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.353 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:11:07.353 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.233188+0000", 2024-09-21T14:11:07.353 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0111019990000001 2024-09-21T14:11:07.353 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.353 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.353 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:11:07.353 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.233317+0000", 2024-09-21T14:11:07.353 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012857999999999999 2024-09-21T14:11:07.353 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.353 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.353 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3", 2024-09-21T14:11:07.353 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.233386+0000", 2024-09-21T14:11:07.353 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8901999999999999e-05 2024-09-21T14:11:07.353 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.354 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.354 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:11:07.354 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.342687+0000", 2024-09-21T14:11:07.354 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.109301519 2024-09-21T14:11:07.354 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.354 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.354 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:11:07.354 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.342738+0000", 2024-09-21T14:11:07.354 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0756e-05 2024-09-21T14:11:07.354 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.354 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.354 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:11:07.354 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353571+0000", 2024-09-21T14:11:07.354 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.01083253 2024-09-21T14:11:07.355 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.355 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.355 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:11:07.355 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353614+0000", 2024-09-21T14:11:07.355 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3129e-05 2024-09-21T14:11:07.355 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.355 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.355 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:11:07.355 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353644+0000", 2024-09-21T14:11:07.355 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0502e-05 2024-09-21T14:11:07.355 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:07.355 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:11:07.355 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:07.355 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.356 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.356 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187654 9.5 9:a73ea609:::benchmark_data_smithi049_138504_object66246:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e347)", 2024-09-21T14:11:07.356 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:42.234021+0000", 2024-09-21T14:11:07.356 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 565.036787871, 2024-09-21T14:11:07.356 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5379708970000001, 2024-09-21T14:11:07.356 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:11:07.356 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:11:07.356 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:11:07.356 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:11:07.356 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:11:07.356 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:11:07.356 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187654 2024-09-21T14:11:07.356 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.356 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:11:07.356 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.357 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:11:07.357 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.234021+0000", 2024-09-21T14:11:07.357 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:07.357 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.357 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.357 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:11:07.357 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.234021+0000", 2024-09-21T14:11:07.357 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:07.357 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.357 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.357 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:11:07.357 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.234023+0000", 2024-09-21T14:11:07.357 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5460000000000001e-06 2024-09-21T14:11:07.357 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.358 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.358 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:11:07.358 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.234029+0000", 2024-09-21T14:11:07.358 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8069999999999996e-06 2024-09-21T14:11:07.358 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.358 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.358 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:11:07.358 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.234041+0000", 2024-09-21T14:11:07.358 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2189e-05 2024-09-21T14:11:07.358 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.358 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.358 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:11:07.358 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.234052+0000", 2024-09-21T14:11:07.358 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0713999999999999e-05 2024-09-21T14:11:07.359 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.359 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.359 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:11:07.359 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.743160+0000", 2024-09-21T14:11:07.359 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5091079629999999 2024-09-21T14:11:07.359 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.359 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.359 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:11:07.359 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.746100+0000", 2024-09-21T14:11:07.359 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0029400559999999999 2024-09-21T14:11:07.359 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.359 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.359 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:11:07.359 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.750393+0000", 2024-09-21T14:11:07.359 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0042938280000000004 2024-09-21T14:11:07.360 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.360 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.360 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:11:07.360 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.757289+0000", 2024-09-21T14:11:07.360 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0068951739999999996 2024-09-21T14:11:07.360 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.360 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.360 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:11:07.360 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.764759+0000", 2024-09-21T14:11:07.360 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0074705689999999998 2024-09-21T14:11:07.360 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.360 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.360 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:11:07.360 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.765446+0000", 2024-09-21T14:11:07.361 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00068706799999999999 2024-09-21T14:11:07.361 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.361 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.361 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:11:07.361 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.766945+0000", 2024-09-21T14:11:07.361 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0014990509999999999 2024-09-21T14:11:07.361 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.361 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.361 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:11:07.361 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.771992+0000", 2024-09-21T14:11:07.361 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0050469319999999996 2024-09-21T14:11:07.361 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:07.361 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:11:07.361 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:07.362 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.362 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.362 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187662 9.f 9:f2269e64:::benchmark_data_smithi049_138504_object66254:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e347)", 2024-09-21T14:11:07.362 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:42.244949+0000", 2024-09-21T14:11:07.362 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 565.02586008200001, 2024-09-21T14:11:07.362 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1087973170000001, 2024-09-21T14:11:07.362 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:11:07.362 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:11:07.362 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:11:07.362 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:11:07.362 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:11:07.362 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:11:07.362 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187662 2024-09-21T14:11:07.362 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.363 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:11:07.363 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.363 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:11:07.363 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.244949+0000", 2024-09-21T14:11:07.363 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:07.363 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.363 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.363 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:11:07.363 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.244949+0000", 2024-09-21T14:11:07.363 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:07.363 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.363 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.363 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:11:07.363 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.244951+0000", 2024-09-21T14:11:07.363 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.962e-06 2024-09-21T14:11:07.364 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.364 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.364 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:11:07.364 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.244957+0000", 2024-09-21T14:11:07.364 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8170000000000004e-06 2024-09-21T14:11:07.364 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.364 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.364 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:11:07.364 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.244972+0000", 2024-09-21T14:11:07.364 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5204999999999999e-05 2024-09-21T14:11:07.364 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.364 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.364 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:11:07.364 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.244981+0000", 2024-09-21T14:11:07.365 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5990000000000002e-06 2024-09-21T14:11:07.365 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.365 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.365 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:11:07.365 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.234563+0000", 2024-09-21T14:11:07.365 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.989582548 2024-09-21T14:11:07.365 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.365 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.365 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:11:07.365 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.234696+0000", 2024-09-21T14:11:07.365 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013280200000000001 2024-09-21T14:11:07.365 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.365 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.365 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3", 2024-09-21T14:11:07.366 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.234783+0000", 2024-09-21T14:11:07.366 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.6990999999999994e-05 2024-09-21T14:11:07.366 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.366 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.366 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:11:07.366 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.342749+0000", 2024-09-21T14:11:07.366 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1079658480000001 2024-09-21T14:11:07.366 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.366 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.366 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:11:07.366 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353673+0000", 2024-09-21T14:11:07.366 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.010924473000000001 2024-09-21T14:11:07.366 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.366 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.366 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:11:07.367 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353696+0000", 2024-09-21T14:11:07.367 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2792000000000001e-05 2024-09-21T14:11:07.367 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.367 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.367 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:11:07.367 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353726+0000", 2024-09-21T14:11:07.367 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9509000000000002e-05 2024-09-21T14:11:07.367 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.367 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.367 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:11:07.367 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353746+0000", 2024-09-21T14:11:07.367 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0771000000000002e-05 2024-09-21T14:11:07.367 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:07.367 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:11:07.368 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:07.368 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.368 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.368 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187665 9.f 9:fde6db3c:::benchmark_data_smithi049_138504_object66257:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e347)", 2024-09-21T14:11:07.368 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:42.254849+0000", 2024-09-21T14:11:07.368 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 565.01596022000001, 2024-09-21T14:11:07.368 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.186045939, 2024-09-21T14:11:07.368 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:11:07.368 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:11:07.368 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:11:07.368 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:11:07.368 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:11:07.368 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:11:07.368 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187665 2024-09-21T14:11:07.368 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.369 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:11:07.369 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.369 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:11:07.369 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.254849+0000", 2024-09-21T14:11:07.369 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:07.369 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.369 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.369 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:11:07.369 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.254849+0000", 2024-09-21T14:11:07.369 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:07.369 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.369 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.369 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:11:07.369 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.254851+0000", 2024-09-21T14:11:07.370 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9209999999999999e-06 2024-09-21T14:11:07.370 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.370 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.370 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:11:07.370 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.254857+0000", 2024-09-21T14:11:07.370 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3659999999999997e-06 2024-09-21T14:11:07.370 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.370 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.370 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:11:07.370 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.254870+0000", 2024-09-21T14:11:07.370 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2892e-05 2024-09-21T14:11:07.370 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.370 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.370 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:11:07.371 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.254882+0000", 2024-09-21T14:11:07.371 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1647999999999999e-05 2024-09-21T14:11:07.371 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.371 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.371 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:11:07.371 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.415623+0000", 2024-09-21T14:11:07.371 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.160741582 2024-09-21T14:11:07.371 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.371 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.371 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:11:07.371 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.415771+0000", 2024-09-21T14:11:07.371 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014724000000000001 2024-09-21T14:11:07.371 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.371 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.371 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,3", 2024-09-21T14:11:07.372 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.415898+0000", 2024-09-21T14:11:07.372 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012708399999999999 2024-09-21T14:11:07.372 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.372 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.372 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:11:07.372 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.427284+0000", 2024-09-21T14:11:07.372 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.011386624 2024-09-21T14:11:07.372 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.372 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.372 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:11:07.372 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.427305+0000", 2024-09-21T14:11:07.372 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0956000000000001e-05 2024-09-21T14:11:07.372 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.372 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.373 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:11:07.373 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.440820+0000", 2024-09-21T14:11:07.373 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.013514511999999999 2024-09-21T14:11:07.373 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.373 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.373 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:11:07.373 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.440860+0000", 2024-09-21T14:11:07.373 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0448000000000003e-05 2024-09-21T14:11:07.373 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.373 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.373 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:11:07.373 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.440895+0000", 2024-09-21T14:11:07.373 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4666000000000003e-05 2024-09-21T14:11:07.373 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:07.374 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:11:07.374 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:07.374 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.374 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.374 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25421.0:187664 9.d e347/345 9:bfa1ff03:::benchmark_data_smithi049_138504_object66256:head v 347'11740, mlcod=346'11738)", 2024-09-21T14:11:07.374 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:42.255288+0000", 2024-09-21T14:11:07.374 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 565.01552134099995, 2024-09-21T14:11:07.374 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2055402320000002, 2024-09-21T14:11:07.374 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:11:07.374 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:11:07.374 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:11:07.374 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:11:07.374 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.374 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:11:07.375 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.255288+0000", 2024-09-21T14:11:07.375 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:07.375 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.375 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.375 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:11:07.375 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.255288+0000", 2024-09-21T14:11:07.375 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:07.375 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.375 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.375 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:11:07.375 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.255289+0000", 2024-09-21T14:11:07.375 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2200000000000003e-07 2024-09-21T14:11:07.375 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.375 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.375 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:11:07.376 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.255296+0000", 2024-09-21T14:11:07.376 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3440000000000003e-06 2024-09-21T14:11:07.377 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.377 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.377 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:11:07.377 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.255312+0000", 2024-09-21T14:11:07.377 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5968999999999999e-05 2024-09-21T14:11:07.377 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.378 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.378 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:11:07.378 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:42.255319+0000", 2024-09-21T14:11:07.378 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9199999999999998e-06 2024-09-21T14:11:07.378 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.378 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.378 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:11:07.378 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.459355+0000", 2024-09-21T14:11:07.378 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2040357249999998 2024-09-21T14:11:07.378 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.378 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.378 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:11:07.378 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.459378+0000", 2024-09-21T14:11:07.378 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3757e-05 2024-09-21T14:11:07.379 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.379 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.379 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:11:07.379 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.460796+0000", 2024-09-21T14:11:07.379 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001417721 2024-09-21T14:11:07.379 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.379 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.379 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:11:07.379 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.460828+0000", 2024-09-21T14:11:07.379 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1974000000000003e-05 2024-09-21T14:11:07.379 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:07.379 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:11:07.379 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:07.379 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.379 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.380 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24365.0:2978 5.15 5:a93a5511:::notify.2:head [watch ping cookie 94025139204608 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-21T14:11:07.380 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:43.276775+0000", 2024-09-21T14:11:07.380 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 563.99403381900004, 2024-09-21T14:11:07.380 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4885784160000002, 2024-09-21T14:11:07.380 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:11:07.380 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:11:07.380 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T14:11:07.380 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:11:07.380 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24365", 2024-09-21T14:11:07.380 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3404764347", 2024-09-21T14:11:07.380 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 2978 2024-09-21T14:11:07.380 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.380 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:11:07.380 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.380 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:11:07.381 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:43.276775+0000", 2024-09-21T14:11:07.381 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:07.381 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.381 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.381 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:11:07.381 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:43.276775+0000", 2024-09-21T14:11:07.381 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:07.381 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.381 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.381 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:11:07.381 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:43.276777+0000", 2024-09-21T14:11:07.381 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9630000000000001e-06 2024-09-21T14:11:07.381 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.381 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.382 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:11:07.382 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:43.276784+0000", 2024-09-21T14:11:07.382 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1219999999999997e-06 2024-09-21T14:11:07.382 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.382 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.382 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:11:07.382 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:43.276802+0000", 2024-09-21T14:11:07.382 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7096999999999999e-05 2024-09-21T14:11:07.382 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.382 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.382 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:11:07.382 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:43.276811+0000", 2024-09-21T14:11:07.382 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7780000000000002e-06 2024-09-21T14:11:07.382 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.383 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.383 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:11:07.383 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.765168+0000", 2024-09-21T14:11:07.383 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4883561849999998 2024-09-21T14:11:07.383 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.383 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.383 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:11:07.383 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.765252+0000", 2024-09-21T14:11:07.383 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4201000000000002e-05 2024-09-21T14:11:07.383 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.383 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.383 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:11:07.383 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.765354+0000", 2024-09-21T14:11:07.383 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010207 2024-09-21T14:11:07.383 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:07.384 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:11:07.384 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:07.384 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.384 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.384 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.25421.0:187651 9.f e349/345 ondisk, result = 0)", 2024-09-21T14:11:07.384 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:44.238795+0000", 2024-09-21T14:11:07.384 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 563.03201453199995, 2024-09-21T14:11:07.384 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.115079242, 2024-09-21T14:11:07.384 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:11:07.384 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:11:07.384 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T14:11:07.384 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:11:07.384 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.384 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:11:07.385 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238795+0000", 2024-09-21T14:11:07.385 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:07.385 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.385 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.385 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:11:07.385 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238795+0000", 2024-09-21T14:11:07.385 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:07.385 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.385 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.385 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:11:07.385 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238795+0000", 2024-09-21T14:11:07.385 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9800000000000003e-07 2024-09-21T14:11:07.385 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.385 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.385 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:11:07.386 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238799+0000", 2024-09-21T14:11:07.386 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9920000000000001e-06 2024-09-21T14:11:07.386 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.386 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.386 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:11:07.386 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238809+0000", 2024-09-21T14:11:07.386 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6120000000000005e-06 2024-09-21T14:11:07.386 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.386 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.386 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:11:07.386 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238816+0000", 2024-09-21T14:11:07.386 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6229999999999999e-06 2024-09-21T14:11:07.386 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.386 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.387 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:11:07.387 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.342725+0000", 2024-09-21T14:11:07.387 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1039093099999999 2024-09-21T14:11:07.387 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.387 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.387 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:11:07.387 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.342733+0000", 2024-09-21T14:11:07.387 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9189999999999999e-06 2024-09-21T14:11:07.387 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.387 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.387 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:11:07.387 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353874+0000", 2024-09-21T14:11:07.387 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.011141088 2024-09-21T14:11:07.387 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:07.388 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:11:07.388 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:07.388 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.388 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.388 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.25421.0:187651 9.f e349/345 ondisk, result = 0)", 2024-09-21T14:11:07.388 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:44.238813+0000", 2024-09-21T14:11:07.388 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 563.03199618999997, 2024-09-21T14:11:07.388 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1148360479999999, 2024-09-21T14:11:07.388 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:11:07.388 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:11:07.388 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-21T14:11:07.388 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:11:07.388 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.388 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:11:07.389 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238813+0000", 2024-09-21T14:11:07.389 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:07.389 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.389 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.389 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:11:07.389 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238813+0000", 2024-09-21T14:11:07.389 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:07.389 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.389 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.389 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:11:07.389 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238814+0000", 2024-09-21T14:11:07.389 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4700000000000001e-07 2024-09-21T14:11:07.389 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.389 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.390 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:11:07.390 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238817+0000", 2024-09-21T14:11:07.390 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3469999999999999e-06 2024-09-21T14:11:07.390 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.390 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.390 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:11:07.390 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238827+0000", 2024-09-21T14:11:07.390 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0011e-05 2024-09-21T14:11:07.390 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.390 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.390 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:11:07.390 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.238832+0000", 2024-09-21T14:11:07.390 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1139999999999997e-06 2024-09-21T14:11:07.390 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.391 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.391 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:11:07.391 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353559+0000", 2024-09-21T14:11:07.391 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1147265909999999 2024-09-21T14:11:07.391 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.391 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.391 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:11:07.391 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353566+0000", 2024-09-21T14:11:07.391 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8669999999999994e-06 2024-09-21T14:11:07.391 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.391 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.391 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:11:07.391 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.353649+0000", 2024-09-21T14:11:07.391 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2571000000000006e-05 2024-09-21T14:11:07.392 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:07.392 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:11:07.392 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:07.392 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.392 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.392 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25421.0:187668 9.5 9:acffd95c:::benchmark_data_smithi049_138504_object66260:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e349)", 2024-09-21T14:11:07.392 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-21T14:01:44.468347+0000", 2024-09-21T14:11:07.392 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 562.80246269300005, 2024-09-21T14:11:07.392 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3419073799999999, 2024-09-21T14:11:07.392 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-21T14:11:07.392 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-21T14:11:07.392 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T14:11:07.392 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-21T14:11:07.392 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25421", 2024-09-21T14:11:07.393 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.49:0/3746179683", 2024-09-21T14:11:07.393 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 187668 2024-09-21T14:11:07.393 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.393 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-21T14:11:07.393 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.393 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-21T14:11:07.393 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.468347+0000", 2024-09-21T14:11:07.393 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:07.393 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.393 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.393 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-21T14:11:07.393 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.468347+0000", 2024-09-21T14:11:07.393 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-21T14:11:07.393 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.394 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.394 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-21T14:11:07.394 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.468347+0000", 2024-09-21T14:11:07.394 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2399999999999997e-07 2024-09-21T14:11:07.394 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.394 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.394 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-21T14:11:07.394 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.468351+0000", 2024-09-21T14:11:07.394 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6049999999999998e-06 2024-09-21T14:11:07.394 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.394 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.394 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-21T14:11:07.394 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.468358+0000", 2024-09-21T14:11:07.395 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4510000000000003e-06 2024-09-21T14:11:07.395 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.395 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.395 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-21T14:11:07.395 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:44.468366+0000", 2024-09-21T14:11:07.395 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6140000000000003e-06 2024-09-21T14:11:07.395 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.395 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.395 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-21T14:11:07.395 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.772366+0000", 2024-09-21T14:11:07.395 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3040005530000001 2024-09-21T14:11:07.395 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.395 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.395 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-21T14:11:07.396 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.775717+0000", 2024-09-21T14:11:07.396 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0033501080000000001 2024-09-21T14:11:07.396 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.396 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.396 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-21T14:11:07.396 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.781090+0000", 2024-09-21T14:11:07.396 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0053731990000000004 2024-09-21T14:11:07.396 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.396 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.396 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:11:07.396 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.798846+0000", 2024-09-21T14:11:07.396 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.017756042999999999 2024-09-21T14:11:07.396 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.396 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.397 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-21T14:11:07.397 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.800017+0000", 2024-09-21T14:11:07.397 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0011714430000000001 2024-09-21T14:11:07.397 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.397 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.397 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-21T14:11:07.397 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.800552+0000", 2024-09-21T14:11:07.397 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00053442500000000005 2024-09-21T14:11:07.397 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.397 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.397 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-21T14:11:07.397 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.801930+0000", 2024-09-21T14:11:07.397 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0013783109999999999 2024-09-21T14:11:07.397 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-21T14:11:07.398 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-21T14:11:07.398 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-21T14:11:07.398 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-21T14:01:45.810254+0000", 2024-09-21T14:11:07.398 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.008323904 2024-09-21T14:11:07.398 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:07.398 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:11:07.398 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:07.398 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-21T14:11:07.398 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-21T14:11:07.398 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-21T14:11:07.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:07 smithi187 ceph-mon[33032]: pgmap v1143: 152 pgs: 1 activating+degraded, 1 activating, 150 active+clean; 583 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 10/597 objects degraded (1.675%) 2024-09-21T14:11:07.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:07 smithi187 ceph-mon[33032]: Health check failed: Degraded data redundancy: 10/597 objects degraded (1.675%), 1 pg degraded (PG_DEGRADED) 2024-09-21T14:11:07.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:07 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-21T14:11:07.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:07 smithi187 ceph-mon[33032]: osdmap e566: 8 total, 8 up, 8 in 2024-09-21T14:11:07.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:07 smithi187 ceph-mon[33032]: 3.19 scrub starts 2024-09-21T14:11:07.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:07 smithi187 ceph-mon[33032]: 3.19 scrub ok 2024-09-21T14:11:07.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:07 smithi187 ceph-mon[33032]: 5.f scrub starts 2024-09-21T14:11:07.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:07 smithi187 ceph-mon[33032]: 5.f scrub ok 2024-09-21T14:11:07.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:07 smithi187 ceph-mon[33032]: 6.16 scrub starts 2024-09-21T14:11:07.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:07 smithi187 ceph-mon[33032]: 6.16 scrub ok 2024-09-21T14:11:07.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:07 smithi187 ceph-mon[33032]: 4.5 scrub starts 2024-09-21T14:11:07.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:07 smithi187 ceph-mon[33032]: 4.5 scrub ok 2024-09-21T14:11:07.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:07 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:11:07.445 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:07 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:11:07.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:07 smithi049 ceph-mon[40489]: pgmap v1143: 152 pgs: 1 activating+degraded, 1 activating, 150 active+clean; 583 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 10/597 objects degraded (1.675%) 2024-09-21T14:11:07.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:07 smithi049 ceph-mon[40489]: Health check failed: Degraded data redundancy: 10/597 objects degraded (1.675%), 1 pg degraded (PG_DEGRADED) 2024-09-21T14:11:07.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:07 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-21T14:11:07.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:07 smithi049 ceph-mon[40489]: osdmap e566: 8 total, 8 up, 8 in 2024-09-21T14:11:07.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:07 smithi049 ceph-mon[40489]: 3.19 scrub starts 2024-09-21T14:11:07.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:07 smithi049 ceph-mon[40489]: 3.19 scrub ok 2024-09-21T14:11:07.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:07 smithi049 ceph-mon[40489]: 5.f scrub starts 2024-09-21T14:11:07.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:07 smithi049 ceph-mon[40489]: 5.f scrub ok 2024-09-21T14:11:07.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:07 smithi049 ceph-mon[40489]: 6.16 scrub starts 2024-09-21T14:11:07.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:07 smithi049 ceph-mon[40489]: 6.16 scrub ok 2024-09-21T14:11:07.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:07 smithi049 ceph-mon[40489]: 4.5 scrub starts 2024-09-21T14:11:07.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:07 smithi049 ceph-mon[40489]: 4.5 scrub ok 2024-09-21T14:11:07.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:07 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:11:07.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:07 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:11:07.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:07 smithi049 ceph-mon[30080]: pgmap v1143: 152 pgs: 1 activating+degraded, 1 activating, 150 active+clean; 583 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 10/597 objects degraded (1.675%) 2024-09-21T14:11:07.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:07 smithi049 ceph-mon[30080]: Health check failed: Degraded data redundancy: 10/597 objects degraded (1.675%), 1 pg degraded (PG_DEGRADED) 2024-09-21T14:11:07.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:07 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-21T14:11:07.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:07 smithi049 ceph-mon[30080]: osdmap e566: 8 total, 8 up, 8 in 2024-09-21T14:11:07.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:07 smithi049 ceph-mon[30080]: 3.19 scrub starts 2024-09-21T14:11:07.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:07 smithi049 ceph-mon[30080]: 3.19 scrub ok 2024-09-21T14:11:07.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:07 smithi049 ceph-mon[30080]: 5.f scrub starts 2024-09-21T14:11:07.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:07 smithi049 ceph-mon[30080]: 5.f scrub ok 2024-09-21T14:11:07.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:07 smithi049 ceph-mon[30080]: 6.16 scrub starts 2024-09-21T14:11:07.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:07 smithi049 ceph-mon[30080]: 6.16 scrub ok 2024-09-21T14:11:07.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:07 smithi049 ceph-mon[30080]: 4.5 scrub starts 2024-09-21T14:11:07.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:07 smithi049 ceph-mon[30080]: 4.5 scrub ok 2024-09-21T14:11:07.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:07 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:11:07.506 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:07 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:11:07.517 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:11:08.397 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.2 to 1 (10000) 2024-09-21T14:11:08.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:08 smithi187 ceph-mon[33032]: 6.1e scrub starts 2024-09-21T14:11:08.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:08 smithi187 ceph-mon[33032]: 6.1e scrub ok 2024-09-21T14:11:08.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:08 smithi187 ceph-mon[33032]: osdmap e567: 8 total, 8 up, 8 in 2024-09-21T14:11:08.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:08 smithi187 ceph-mon[33032]: 1.1 scrub starts 2024-09-21T14:11:08.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:08 smithi187 ceph-mon[33032]: 1.1 scrub ok 2024-09-21T14:11:08.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:08 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2579675021' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T14:11:08.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:08 smithi049 ceph-mon[30080]: 6.1e scrub starts 2024-09-21T14:11:08.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:08 smithi049 ceph-mon[30080]: 6.1e scrub ok 2024-09-21T14:11:08.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:08 smithi049 ceph-mon[30080]: osdmap e567: 8 total, 8 up, 8 in 2024-09-21T14:11:08.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:08 smithi049 ceph-mon[30080]: 1.1 scrub starts 2024-09-21T14:11:08.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:08 smithi049 ceph-mon[30080]: 1.1 scrub ok 2024-09-21T14:11:08.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:08 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2579675021' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T14:11:08.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:08 smithi049 ceph-mon[40489]: 6.1e scrub starts 2024-09-21T14:11:08.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:08 smithi049 ceph-mon[40489]: 6.1e scrub ok 2024-09-21T14:11:08.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:08 smithi049 ceph-mon[40489]: osdmap e567: 8 total, 8 up, 8 in 2024-09-21T14:11:08.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:08 smithi049 ceph-mon[40489]: 1.1 scrub starts 2024-09-21T14:11:08.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:08 smithi049 ceph-mon[40489]: 1.1 scrub ok 2024-09-21T14:11:08.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:08 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2579675021' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T14:11:09.255 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 2 1 2024-09-21T14:11:09.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:09 smithi049 ceph-mon[40489]: pgmap v1146: 152 pgs: 1 activating+degraded, 1 activating, 150 active+clean; 583 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 10/597 objects degraded (1.675%) 2024-09-21T14:11:09.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:09 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2579675021' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T14:11:09.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:09 smithi049 ceph-mon[40489]: osdmap e568: 8 total, 8 up, 8 in 2024-09-21T14:11:09.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:09 smithi049 ceph-mon[40489]: 4.1c scrub starts 2024-09-21T14:11:09.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:09 smithi049 ceph-mon[40489]: 4.1c scrub ok 2024-09-21T14:11:09.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:09 smithi049 ceph-mon[30080]: pgmap v1146: 152 pgs: 1 activating+degraded, 1 activating, 150 active+clean; 583 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 10/597 objects degraded (1.675%) 2024-09-21T14:11:09.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:09 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2579675021' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T14:11:09.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:09 smithi049 ceph-mon[30080]: osdmap e568: 8 total, 8 up, 8 in 2024-09-21T14:11:09.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:09 smithi049 ceph-mon[30080]: 4.1c scrub starts 2024-09-21T14:11:09.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:09 smithi049 ceph-mon[30080]: 4.1c scrub ok 2024-09-21T14:11:09.519 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:11:09.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:09 smithi187 ceph-mon[33032]: pgmap v1146: 152 pgs: 1 activating+degraded, 1 activating, 150 active+clean; 583 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 10/597 objects degraded (1.675%) 2024-09-21T14:11:09.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:09 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2579675021' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T14:11:09.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:09 smithi187 ceph-mon[33032]: osdmap e568: 8 total, 8 up, 8 in 2024-09-21T14:11:09.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:09 smithi187 ceph-mon[33032]: 4.1c scrub starts 2024-09-21T14:11:09.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:09 smithi187 ceph-mon[33032]: 4.1c scrub ok 2024-09-21T14:11:10.405 INFO:teuthology.orchestra.run.smithi049.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-21T14:11:10.694 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:11:10 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:11:10.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:10 smithi187 ceph-mon[33032]: 4.f scrub starts 2024-09-21T14:11:10.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:10 smithi187 ceph-mon[33032]: 4.f scrub ok 2024-09-21T14:11:10.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:10 smithi187 ceph-mon[33032]: 5.14 scrub starts 2024-09-21T14:11:10.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:10 smithi187 ceph-mon[33032]: 5.14 scrub ok 2024-09-21T14:11:10.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:10 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3850134458' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-21T14:11:10.754 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:11:10 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:11:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:11:10.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:10 smithi049 ceph-mon[30080]: 4.f scrub starts 2024-09-21T14:11:10.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:10 smithi049 ceph-mon[30080]: 4.f scrub ok 2024-09-21T14:11:10.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:10 smithi049 ceph-mon[30080]: 5.14 scrub starts 2024-09-21T14:11:10.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:10 smithi049 ceph-mon[30080]: 5.14 scrub ok 2024-09-21T14:11:10.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:10 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3850134458' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-21T14:11:10.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:10 smithi049 ceph-mon[40489]: 4.f scrub starts 2024-09-21T14:11:10.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:10 smithi049 ceph-mon[40489]: 4.f scrub ok 2024-09-21T14:11:10.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:10 smithi049 ceph-mon[40489]: 5.14 scrub starts 2024-09-21T14:11:10.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:10 smithi049 ceph-mon[40489]: 5.14 scrub ok 2024-09-21T14:11:10.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:10 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3850134458' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-21T14:11:11.262 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd reweight 3 1 2024-09-21T14:11:11.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:11 smithi049 ceph-mon[40489]: pgmap v1148: 152 pgs: 1 activating+degraded, 1 activating, 150 active+clean; 583 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 10/597 objects degraded (1.675%) 2024-09-21T14:11:11.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:11 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:11:11.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:11 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3850134458' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-21T14:11:11.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:11 smithi049 ceph-mon[40489]: osdmap e569: 8 total, 8 up, 8 in 2024-09-21T14:11:11.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:11 smithi049 ceph-mon[40489]: 4.8 scrub starts 2024-09-21T14:11:11.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:11 smithi049 ceph-mon[40489]: 4.8 scrub ok 2024-09-21T14:11:11.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:11 smithi049 ceph-mon[30080]: pgmap v1148: 152 pgs: 1 activating+degraded, 1 activating, 150 active+clean; 583 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 10/597 objects degraded (1.675%) 2024-09-21T14:11:11.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:11 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:11:11.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:11 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3850134458' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-21T14:11:11.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:11 smithi049 ceph-mon[30080]: osdmap e569: 8 total, 8 up, 8 in 2024-09-21T14:11:11.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:11 smithi049 ceph-mon[30080]: 4.8 scrub starts 2024-09-21T14:11:11.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:11 smithi049 ceph-mon[30080]: 4.8 scrub ok 2024-09-21T14:11:11.524 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:11:11.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:11 smithi187 ceph-mon[33032]: pgmap v1148: 152 pgs: 1 activating+degraded, 1 activating, 150 active+clean; 583 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 10/597 objects degraded (1.675%) 2024-09-21T14:11:11.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:11 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:11:11.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:11 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3850134458' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-21T14:11:11.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:11 smithi187 ceph-mon[33032]: osdmap e569: 8 total, 8 up, 8 in 2024-09-21T14:11:11.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:11 smithi187 ceph-mon[33032]: 4.8 scrub starts 2024-09-21T14:11:11.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:11 smithi187 ceph-mon[33032]: 4.8 scrub ok 2024-09-21T14:11:12.418 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.3 to 1 (10000) 2024-09-21T14:11:12.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:12 smithi187 ceph-mon[33032]: 4.e scrub starts 2024-09-21T14:11:12.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:12 smithi187 ceph-mon[33032]: 4.e scrub ok 2024-09-21T14:11:12.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:12 smithi187 ceph-mon[33032]: 5.7 scrub starts 2024-09-21T14:11:12.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:12 smithi187 ceph-mon[33032]: 5.7 scrub ok 2024-09-21T14:11:12.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:12 smithi187 ceph-mon[33032]: 5.8 scrub starts 2024-09-21T14:11:12.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:12 smithi187 ceph-mon[33032]: 5.8 scrub ok 2024-09-21T14:11:12.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:12 smithi187 ceph-mon[33032]: 6.1f scrub starts 2024-09-21T14:11:12.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:12 smithi187 ceph-mon[33032]: 6.1f scrub ok 2024-09-21T14:11:12.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:12 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1530379168' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T14:11:12.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:12 smithi049 ceph-mon[30080]: 4.e scrub starts 2024-09-21T14:11:12.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:12 smithi049 ceph-mon[30080]: 4.e scrub ok 2024-09-21T14:11:12.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:12 smithi049 ceph-mon[30080]: 5.7 scrub starts 2024-09-21T14:11:12.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:12 smithi049 ceph-mon[30080]: 5.7 scrub ok 2024-09-21T14:11:12.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:12 smithi049 ceph-mon[30080]: 5.8 scrub starts 2024-09-21T14:11:12.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:12 smithi049 ceph-mon[30080]: 5.8 scrub ok 2024-09-21T14:11:12.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:12 smithi049 ceph-mon[30080]: 6.1f scrub starts 2024-09-21T14:11:12.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:12 smithi049 ceph-mon[30080]: 6.1f scrub ok 2024-09-21T14:11:12.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:12 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1530379168' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T14:11:12.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:12 smithi049 ceph-mon[40489]: 4.e scrub starts 2024-09-21T14:11:12.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:12 smithi049 ceph-mon[40489]: 4.e scrub ok 2024-09-21T14:11:12.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:12 smithi049 ceph-mon[40489]: 5.7 scrub starts 2024-09-21T14:11:12.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:12 smithi049 ceph-mon[40489]: 5.7 scrub ok 2024-09-21T14:11:12.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:12 smithi049 ceph-mon[40489]: 5.8 scrub starts 2024-09-21T14:11:12.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:12 smithi049 ceph-mon[40489]: 5.8 scrub ok 2024-09-21T14:11:12.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:12 smithi049 ceph-mon[40489]: 6.1f scrub starts 2024-09-21T14:11:12.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:12 smithi049 ceph-mon[40489]: 6.1f scrub ok 2024-09-21T14:11:12.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:12 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1530379168' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T14:11:13.471 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 3 1 2024-09-21T14:11:13.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:13 smithi187 ceph-mon[33032]: pgmap v1150: 152 pgs: 152 active+clean; 583 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 880 B/s rd, 0 op/s; 61 B/s, 1 objects/s recovering 2024-09-21T14:11:13.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:13 smithi187 ceph-mon[33032]: 4.14 scrub starts 2024-09-21T14:11:13.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:13 smithi187 ceph-mon[33032]: 4.14 scrub ok 2024-09-21T14:11:13.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:13 smithi187 ceph-mon[33032]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 10/597 objects degraded (1.675%), 1 pg degraded) 2024-09-21T14:11:13.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:13 smithi187 ceph-mon[33032]: Cluster is now healthy 2024-09-21T14:11:13.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:13 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1530379168' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T14:11:13.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:13 smithi187 ceph-mon[33032]: osdmap e570: 8 total, 8 up, 8 in 2024-09-21T14:11:13.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:13 smithi187 ceph-mon[33032]: 3.1d scrub starts 2024-09-21T14:11:13.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:13 smithi187 ceph-mon[33032]: 3.1d scrub ok 2024-09-21T14:11:13.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:13 smithi187 ceph-mon[33032]: 6.17 scrub starts 2024-09-21T14:11:13.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:13 smithi187 ceph-mon[33032]: 6.17 scrub ok 2024-09-21T14:11:13.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:13 smithi187 ceph-mon[33032]: 5.5 scrub starts 2024-09-21T14:11:13.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:13 smithi187 ceph-mon[33032]: 5.5 scrub ok 2024-09-21T14:11:13.736 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:11:13.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:13 smithi049 ceph-mon[40489]: pgmap v1150: 152 pgs: 152 active+clean; 583 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 880 B/s rd, 0 op/s; 61 B/s, 1 objects/s recovering 2024-09-21T14:11:13.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:13 smithi049 ceph-mon[40489]: 4.14 scrub starts 2024-09-21T14:11:13.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:13 smithi049 ceph-mon[40489]: 4.14 scrub ok 2024-09-21T14:11:13.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:13 smithi049 ceph-mon[40489]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 10/597 objects degraded (1.675%), 1 pg degraded) 2024-09-21T14:11:13.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:13 smithi049 ceph-mon[40489]: Cluster is now healthy 2024-09-21T14:11:13.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:13 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1530379168' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T14:11:13.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:13 smithi049 ceph-mon[40489]: osdmap e570: 8 total, 8 up, 8 in 2024-09-21T14:11:13.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:13 smithi049 ceph-mon[40489]: 3.1d scrub starts 2024-09-21T14:11:13.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:13 smithi049 ceph-mon[40489]: 3.1d scrub ok 2024-09-21T14:11:13.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:13 smithi049 ceph-mon[40489]: 6.17 scrub starts 2024-09-21T14:11:13.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:13 smithi049 ceph-mon[40489]: 6.17 scrub ok 2024-09-21T14:11:13.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:13 smithi049 ceph-mon[40489]: 5.5 scrub starts 2024-09-21T14:11:13.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:13 smithi049 ceph-mon[40489]: 5.5 scrub ok 2024-09-21T14:11:13.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:13 smithi049 ceph-mon[30080]: pgmap v1150: 152 pgs: 152 active+clean; 583 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 880 B/s rd, 0 op/s; 61 B/s, 1 objects/s recovering 2024-09-21T14:11:13.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:13 smithi049 ceph-mon[30080]: 4.14 scrub starts 2024-09-21T14:11:13.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:13 smithi049 ceph-mon[30080]: 4.14 scrub ok 2024-09-21T14:11:13.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:13 smithi049 ceph-mon[30080]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 10/597 objects degraded (1.675%), 1 pg degraded) 2024-09-21T14:11:13.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:13 smithi049 ceph-mon[30080]: Cluster is now healthy 2024-09-21T14:11:13.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:13 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1530379168' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T14:11:13.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:13 smithi049 ceph-mon[30080]: osdmap e570: 8 total, 8 up, 8 in 2024-09-21T14:11:13.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:13 smithi049 ceph-mon[30080]: 3.1d scrub starts 2024-09-21T14:11:13.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:13 smithi049 ceph-mon[30080]: 3.1d scrub ok 2024-09-21T14:11:13.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:13 smithi049 ceph-mon[30080]: 6.17 scrub starts 2024-09-21T14:11:13.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:13 smithi049 ceph-mon[30080]: 6.17 scrub ok 2024-09-21T14:11:13.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:13 smithi049 ceph-mon[30080]: 5.5 scrub starts 2024-09-21T14:11:13.756 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:13 smithi049 ceph-mon[30080]: 5.5 scrub ok 2024-09-21T14:11:14.435 INFO:teuthology.orchestra.run.smithi049.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-21T14:11:14.448 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:14 smithi049 ceph-mon[40489]: osdmap e571: 8 total, 8 up, 8 in 2024-09-21T14:11:14.448 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:14 smithi049 ceph-mon[40489]: 5.a scrub starts 2024-09-21T14:11:14.448 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:14 smithi049 ceph-mon[40489]: 5.a scrub ok 2024-09-21T14:11:14.448 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:14 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2325896226' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-21T14:11:14.448 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:14 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-21T14:11:14.448 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:14 smithi049 ceph-mon[30080]: osdmap e571: 8 total, 8 up, 8 in 2024-09-21T14:11:14.448 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:14 smithi049 ceph-mon[30080]: 5.a scrub starts 2024-09-21T14:11:14.449 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:14 smithi049 ceph-mon[30080]: 5.a scrub ok 2024-09-21T14:11:14.449 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:14 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2325896226' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-21T14:11:14.449 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:14 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-21T14:11:14.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:14 smithi187 ceph-mon[33032]: osdmap e571: 8 total, 8 up, 8 in 2024-09-21T14:11:14.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:14 smithi187 ceph-mon[33032]: 5.a scrub starts 2024-09-21T14:11:14.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:14 smithi187 ceph-mon[33032]: 5.a scrub ok 2024-09-21T14:11:14.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:14 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2325896226' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-21T14:11:14.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:14 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-21T14:11:15.130 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd reweight 7 1 2024-09-21T14:11:15.393 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:11:15.693 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:15 smithi049 ceph-mon[40489]: pgmap v1153: 152 pgs: 4 peering, 148 active+clean; 583 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 881 B/s rd, 0 op/s; 99 B/s, 2 objects/s recovering 2024-09-21T14:11:15.693 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:15 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-21T14:11:15.693 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:15 smithi049 ceph-mon[40489]: osdmap e572: 8 total, 8 up, 8 in 2024-09-21T14:11:15.693 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:15 smithi049 ceph-mon[40489]: 2.0 scrub starts 2024-09-21T14:11:15.693 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:15 smithi049 ceph-mon[40489]: 2.0 scrub ok 2024-09-21T14:11:15.693 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:15 smithi049 ceph-mon[40489]: 4.2 scrub starts 2024-09-21T14:11:15.693 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:15 smithi049 ceph-mon[40489]: 4.2 scrub ok 2024-09-21T14:11:15.693 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:15 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:11:15.693 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:15 smithi049 ceph-mon[40489]: 3.6 scrub starts 2024-09-21T14:11:15.693 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:15 smithi049 ceph-mon[40489]: 3.6 scrub ok 2024-09-21T14:11:15.694 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:15 smithi049 ceph-mon[40489]: 1.6 scrub starts 2024-09-21T14:11:15.694 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:15 smithi049 ceph-mon[40489]: 1.6 scrub ok 2024-09-21T14:11:15.694 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:15 smithi049 ceph-mon[30080]: pgmap v1153: 152 pgs: 4 peering, 148 active+clean; 583 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 881 B/s rd, 0 op/s; 99 B/s, 2 objects/s recovering 2024-09-21T14:11:15.694 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:15 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-21T14:11:15.694 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:15 smithi049 ceph-mon[30080]: osdmap e572: 8 total, 8 up, 8 in 2024-09-21T14:11:15.694 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:15 smithi049 ceph-mon[30080]: 2.0 scrub starts 2024-09-21T14:11:15.694 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:15 smithi049 ceph-mon[30080]: 2.0 scrub ok 2024-09-21T14:11:15.694 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:15 smithi049 ceph-mon[30080]: 4.2 scrub starts 2024-09-21T14:11:15.694 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:15 smithi049 ceph-mon[30080]: 4.2 scrub ok 2024-09-21T14:11:15.694 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:15 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:11:15.695 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:15 smithi049 ceph-mon[30080]: 3.6 scrub starts 2024-09-21T14:11:15.695 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:15 smithi049 ceph-mon[30080]: 3.6 scrub ok 2024-09-21T14:11:15.695 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:15 smithi049 ceph-mon[30080]: 1.6 scrub starts 2024-09-21T14:11:15.695 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:15 smithi049 ceph-mon[30080]: 1.6 scrub ok 2024-09-21T14:11:15.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:15 smithi187 ceph-mon[33032]: pgmap v1153: 152 pgs: 4 peering, 148 active+clean; 583 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 881 B/s rd, 0 op/s; 99 B/s, 2 objects/s recovering 2024-09-21T14:11:15.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:15 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-21T14:11:15.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:15 smithi187 ceph-mon[33032]: osdmap e572: 8 total, 8 up, 8 in 2024-09-21T14:11:15.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:15 smithi187 ceph-mon[33032]: 2.0 scrub starts 2024-09-21T14:11:15.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:15 smithi187 ceph-mon[33032]: 2.0 scrub ok 2024-09-21T14:11:15.696 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:15 smithi187 ceph-mon[33032]: 4.2 scrub starts 2024-09-21T14:11:15.696 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:15 smithi187 ceph-mon[33032]: 4.2 scrub ok 2024-09-21T14:11:15.696 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:15 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:11:15.696 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:15 smithi187 ceph-mon[33032]: 3.6 scrub starts 2024-09-21T14:11:15.696 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:15 smithi187 ceph-mon[33032]: 3.6 scrub ok 2024-09-21T14:11:15.696 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:15 smithi187 ceph-mon[33032]: 1.6 scrub starts 2024-09-21T14:11:15.696 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:15 smithi187 ceph-mon[33032]: 1.6 scrub ok 2024-09-21T14:11:16.438 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.7 to 1 (10000) 2024-09-21T14:11:16.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:16 smithi187 ceph-mon[33032]: 5.19 scrub starts 2024-09-21T14:11:16.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:16 smithi187 ceph-mon[33032]: 5.19 scrub ok 2024-09-21T14:11:16.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:16 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3734325236' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T14:11:16.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:16 smithi049 ceph-mon[40489]: 5.19 scrub starts 2024-09-21T14:11:16.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:16 smithi049 ceph-mon[40489]: 5.19 scrub ok 2024-09-21T14:11:16.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:16 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3734325236' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T14:11:16.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:16 smithi049 ceph-mon[30080]: 5.19 scrub starts 2024-09-21T14:11:16.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:16 smithi049 ceph-mon[30080]: 5.19 scrub ok 2024-09-21T14:11:16.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:16 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3734325236' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T14:11:17.280 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 7 1 2024-09-21T14:11:17.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:17 smithi049 ceph-mon[40489]: pgmap v1155: 152 pgs: 4 peering, 148 active+clean; 583 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 45 KiB/s, 5 objects/s recovering 2024-09-21T14:11:17.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:17 smithi049 ceph-mon[40489]: 6.c scrub starts 2024-09-21T14:11:17.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:17 smithi049 ceph-mon[40489]: 6.c scrub ok 2024-09-21T14:11:17.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:17 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3734325236' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T14:11:17.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:17 smithi049 ceph-mon[40489]: osdmap e573: 8 total, 8 up, 8 in 2024-09-21T14:11:17.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:17 smithi049 ceph-mon[40489]: 4.4 scrub starts 2024-09-21T14:11:17.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:17 smithi049 ceph-mon[40489]: 4.4 scrub ok 2024-09-21T14:11:17.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:17 smithi049 ceph-mon[40489]: 4.3 scrub starts 2024-09-21T14:11:17.505 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:17 smithi049 ceph-mon[40489]: 4.3 scrub ok 2024-09-21T14:11:17.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:17 smithi049 ceph-mon[30080]: pgmap v1155: 152 pgs: 4 peering, 148 active+clean; 583 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 45 KiB/s, 5 objects/s recovering 2024-09-21T14:11:17.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:17 smithi049 ceph-mon[30080]: 6.c scrub starts 2024-09-21T14:11:17.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:17 smithi049 ceph-mon[30080]: 6.c scrub ok 2024-09-21T14:11:17.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:17 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3734325236' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T14:11:17.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:17 smithi049 ceph-mon[30080]: osdmap e573: 8 total, 8 up, 8 in 2024-09-21T14:11:17.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:17 smithi049 ceph-mon[30080]: 4.4 scrub starts 2024-09-21T14:11:17.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:17 smithi049 ceph-mon[30080]: 4.4 scrub ok 2024-09-21T14:11:17.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:17 smithi049 ceph-mon[30080]: 4.3 scrub starts 2024-09-21T14:11:17.505 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:17 smithi049 ceph-mon[30080]: 4.3 scrub ok 2024-09-21T14:11:17.544 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:11:17.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:17 smithi187 ceph-mon[33032]: pgmap v1155: 152 pgs: 4 peering, 148 active+clean; 583 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 45 KiB/s, 5 objects/s recovering 2024-09-21T14:11:17.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:17 smithi187 ceph-mon[33032]: 6.c scrub starts 2024-09-21T14:11:17.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:17 smithi187 ceph-mon[33032]: 6.c scrub ok 2024-09-21T14:11:17.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:17 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3734325236' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T14:11:17.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:17 smithi187 ceph-mon[33032]: osdmap e573: 8 total, 8 up, 8 in 2024-09-21T14:11:17.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:17 smithi187 ceph-mon[33032]: 4.4 scrub starts 2024-09-21T14:11:17.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:17 smithi187 ceph-mon[33032]: 4.4 scrub ok 2024-09-21T14:11:17.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:17 smithi187 ceph-mon[33032]: 4.3 scrub starts 2024-09-21T14:11:17.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:17 smithi187 ceph-mon[33032]: 4.3 scrub ok 2024-09-21T14:11:18.441 INFO:teuthology.orchestra.run.smithi049.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-09-21T14:11:18.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:18 smithi049 ceph-mon[40489]: 1.b scrub starts 2024-09-21T14:11:18.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:18 smithi049 ceph-mon[40489]: 1.b scrub ok 2024-09-21T14:11:18.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:18 smithi049 ceph-mon[40489]: 3.1 scrub starts 2024-09-21T14:11:18.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:18 smithi049 ceph-mon[40489]: 3.1 scrub ok 2024-09-21T14:11:18.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:18 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/4255947840' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-21T14:11:18.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:18 smithi049 ceph-mon[30080]: 1.b scrub starts 2024-09-21T14:11:18.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:18 smithi049 ceph-mon[30080]: 1.b scrub ok 2024-09-21T14:11:18.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:18 smithi049 ceph-mon[30080]: 3.1 scrub starts 2024-09-21T14:11:18.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:18 smithi049 ceph-mon[30080]: 3.1 scrub ok 2024-09-21T14:11:18.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:18 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/4255947840' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-21T14:11:18.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:18 smithi187 ceph-mon[33032]: 1.b scrub starts 2024-09-21T14:11:18.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:18 smithi187 ceph-mon[33032]: 1.b scrub ok 2024-09-21T14:11:18.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:18 smithi187 ceph-mon[33032]: 3.1 scrub starts 2024-09-21T14:11:18.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:18 smithi187 ceph-mon[33032]: 3.1 scrub ok 2024-09-21T14:11:18.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:18 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/4255947840' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-21T14:11:19.183 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd reweight 5 1 2024-09-21T14:11:19.446 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:11:19.720 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:19 smithi049 ceph-mon[40489]: pgmap v1157: 152 pgs: 4 peering, 148 active+clean; 583 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 707 B/s rd, 0 op/s; 100 KiB/s, 7 objects/s recovering 2024-09-21T14:11:19.720 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:19 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/4255947840' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-21T14:11:19.721 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:19 smithi049 ceph-mon[40489]: osdmap e574: 8 total, 8 up, 8 in 2024-09-21T14:11:19.721 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:19 smithi049 ceph-mon[40489]: 3.1b scrub starts 2024-09-21T14:11:19.721 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:19 smithi049 ceph-mon[40489]: 3.1b scrub ok 2024-09-21T14:11:19.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:19 smithi049 ceph-mon[30080]: pgmap v1157: 152 pgs: 4 peering, 148 active+clean; 583 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 707 B/s rd, 0 op/s; 100 KiB/s, 7 objects/s recovering 2024-09-21T14:11:19.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:19 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/4255947840' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-21T14:11:19.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:19 smithi049 ceph-mon[30080]: osdmap e574: 8 total, 8 up, 8 in 2024-09-21T14:11:19.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:19 smithi049 ceph-mon[30080]: 3.1b scrub starts 2024-09-21T14:11:19.721 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:19 smithi049 ceph-mon[30080]: 3.1b scrub ok 2024-09-21T14:11:19.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:19 smithi187 ceph-mon[33032]: pgmap v1157: 152 pgs: 4 peering, 148 active+clean; 583 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 707 B/s rd, 0 op/s; 100 KiB/s, 7 objects/s recovering 2024-09-21T14:11:19.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:19 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/4255947840' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-21T14:11:19.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:19 smithi187 ceph-mon[33032]: osdmap e574: 8 total, 8 up, 8 in 2024-09-21T14:11:19.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:19 smithi187 ceph-mon[33032]: 3.1b scrub starts 2024-09-21T14:11:19.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:19 smithi187 ceph-mon[33032]: 3.1b scrub ok 2024-09-21T14:11:20.283 INFO:teuthology.orchestra.run.smithi049.stderr:reweighted osd.5 to 1 (10000) 2024-09-21T14:11:20.578 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:20 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:11:20.578 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:20 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:11:20.578 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:20 smithi049 ceph-mon[40489]: osdmap e575: 8 total, 8 up, 8 in 2024-09-21T14:11:20.578 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:20 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3789762954' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T14:11:20.578 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:20 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3789762954' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T14:11:20.578 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:20 smithi049 ceph-mon[40489]: osdmap e576: 8 total, 8 up, 8 in 2024-09-21T14:11:20.579 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:20 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:11:20.579 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:20 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:11:20.579 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:20 smithi049 ceph-mon[30080]: osdmap e575: 8 total, 8 up, 8 in 2024-09-21T14:11:20.579 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:20 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3789762954' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T14:11:20.579 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:20 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3789762954' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T14:11:20.579 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:20 smithi049 ceph-mon[30080]: osdmap e576: 8 total, 8 up, 8 in 2024-09-21T14:11:20.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:20 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:11:20.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:20 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:11:20.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:20 smithi187 ceph-mon[33032]: osdmap e575: 8 total, 8 up, 8 in 2024-09-21T14:11:20.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:20 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3789762954' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T14:11:20.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:20 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3789762954' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T14:11:20.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:20 smithi187 ceph-mon[33032]: osdmap e576: 8 total, 8 up, 8 in 2024-09-21T14:11:20.695 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:11:20 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:11:21.004 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:11:20 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:11:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:11:21.234 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 5 1 2024-09-21T14:11:21.459 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:21 smithi049 ceph-mon[30080]: pgmap v1160: 152 pgs: 152 active+clean; 583 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 709 B/s rd, 0 op/s; 100 KiB/s, 8 objects/s recovering 2024-09-21T14:11:21.459 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:21 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:11:21.505 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:11:21.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:21 smithi049 ceph-mon[30080]: 4.0 scrub starts 2024-09-21T14:11:21.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:21 smithi049 ceph-mon[30080]: 4.0 scrub ok 2024-09-21T14:11:21.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:21 smithi049 ceph-mon[30080]: 3.17 scrub starts 2024-09-21T14:11:21.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:21 smithi049 ceph-mon[30080]: 3.17 scrub ok 2024-09-21T14:11:21.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:21 smithi049 ceph-mon[40489]: pgmap v1160: 152 pgs: 152 active+clean; 583 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 709 B/s rd, 0 op/s; 100 KiB/s, 8 objects/s recovering 2024-09-21T14:11:21.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:21 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:11:21.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:21 smithi049 ceph-mon[40489]: 4.0 scrub starts 2024-09-21T14:11:21.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:21 smithi049 ceph-mon[40489]: 4.0 scrub ok 2024-09-21T14:11:21.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:21 smithi049 ceph-mon[40489]: 3.17 scrub starts 2024-09-21T14:11:21.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:21 smithi049 ceph-mon[40489]: 3.17 scrub ok 2024-09-21T14:11:21.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:21 smithi187 ceph-mon[33032]: pgmap v1160: 152 pgs: 152 active+clean; 583 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 709 B/s rd, 0 op/s; 100 KiB/s, 8 objects/s recovering 2024-09-21T14:11:21.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:21 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:11:21.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:21 smithi187 ceph-mon[33032]: 4.0 scrub starts 2024-09-21T14:11:21.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:21 smithi187 ceph-mon[33032]: 4.0 scrub ok 2024-09-21T14:11:21.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:21 smithi187 ceph-mon[33032]: 3.17 scrub starts 2024-09-21T14:11:21.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:21 smithi187 ceph-mon[33032]: 3.17 scrub ok 2024-09-21T14:11:22.461 INFO:teuthology.orchestra.run.smithi049.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-21T14:11:22.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:22 smithi049 ceph-mon[30080]: 6.5 scrub starts 2024-09-21T14:11:22.819 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:22 smithi049 ceph-mon[30080]: 6.5 scrub ok 2024-09-21T14:11:22.819 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:22 smithi049 ceph-mon[30080]: 3.0 scrub starts 2024-09-21T14:11:22.819 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:22 smithi049 ceph-mon[30080]: 3.0 scrub ok 2024-09-21T14:11:22.819 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:22 smithi049 ceph-mon[30080]: 1.14 scrub starts 2024-09-21T14:11:22.819 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:22 smithi049 ceph-mon[30080]: 1.14 scrub ok 2024-09-21T14:11:22.819 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:22 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/201512309' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-21T14:11:22.820 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:22 smithi049 ceph-mon[40489]: 6.5 scrub starts 2024-09-21T14:11:22.820 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:22 smithi049 ceph-mon[40489]: 6.5 scrub ok 2024-09-21T14:11:22.820 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:22 smithi049 ceph-mon[40489]: 3.0 scrub starts 2024-09-21T14:11:22.820 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:22 smithi049 ceph-mon[40489]: 3.0 scrub ok 2024-09-21T14:11:22.820 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:22 smithi049 ceph-mon[40489]: 1.14 scrub starts 2024-09-21T14:11:22.820 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:22 smithi049 ceph-mon[40489]: 1.14 scrub ok 2024-09-21T14:11:22.820 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:22 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/201512309' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-21T14:11:22.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:22 smithi187 ceph-mon[33032]: 6.5 scrub starts 2024-09-21T14:11:22.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:22 smithi187 ceph-mon[33032]: 6.5 scrub ok 2024-09-21T14:11:22.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:22 smithi187 ceph-mon[33032]: 3.0 scrub starts 2024-09-21T14:11:22.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:22 smithi187 ceph-mon[33032]: 3.0 scrub ok 2024-09-21T14:11:22.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:22 smithi187 ceph-mon[33032]: 1.14 scrub starts 2024-09-21T14:11:22.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:22 smithi187 ceph-mon[33032]: 1.14 scrub ok 2024-09-21T14:11:22.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:22 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/201512309' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-21T14:11:23.292 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-09-21T14:11:23.293 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-21T14:11:23.293 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-09-21T14:11:23.293 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-09-21T14:11:23.293 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-21T14:11:23.293 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd dump --format=json 2024-09-21T14:11:23.559 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:11:23.680 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:23 smithi049 ceph-mon[40489]: pgmap v1162: 152 pgs: 152 active+clean; 583 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 44 KiB/s rd, 43 op/s; 56 KiB/s, 5 objects/s recovering 2024-09-21T14:11:23.681 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:23 smithi049 ceph-mon[40489]: 5.1a scrub starts 2024-09-21T14:11:23.681 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:23 smithi049 ceph-mon[40489]: 5.1a scrub ok 2024-09-21T14:11:23.681 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:23 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/201512309' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-21T14:11:23.681 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:23 smithi049 ceph-mon[40489]: osdmap e577: 8 total, 8 up, 8 in 2024-09-21T14:11:23.681 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:23 smithi049 ceph-mon[40489]: 6.e scrub starts 2024-09-21T14:11:23.681 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:23 smithi049 ceph-mon[40489]: 6.e scrub ok 2024-09-21T14:11:23.681 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:23 smithi049 ceph-mon[30080]: pgmap v1162: 152 pgs: 152 active+clean; 583 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 44 KiB/s rd, 43 op/s; 56 KiB/s, 5 objects/s recovering 2024-09-21T14:11:23.681 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:23 smithi049 ceph-mon[30080]: 5.1a scrub starts 2024-09-21T14:11:23.681 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:23 smithi049 ceph-mon[30080]: 5.1a scrub ok 2024-09-21T14:11:23.681 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:23 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/201512309' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-21T14:11:23.681 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:23 smithi049 ceph-mon[30080]: osdmap e577: 8 total, 8 up, 8 in 2024-09-21T14:11:23.682 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:23 smithi049 ceph-mon[30080]: 6.e scrub starts 2024-09-21T14:11:23.682 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:23 smithi049 ceph-mon[30080]: 6.e scrub ok 2024-09-21T14:11:23.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:23 smithi187 ceph-mon[33032]: pgmap v1162: 152 pgs: 152 active+clean; 583 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 44 KiB/s rd, 43 op/s; 56 KiB/s, 5 objects/s recovering 2024-09-21T14:11:23.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:23 smithi187 ceph-mon[33032]: 5.1a scrub starts 2024-09-21T14:11:23.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:23 smithi187 ceph-mon[33032]: 5.1a scrub ok 2024-09-21T14:11:23.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:23 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/201512309' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-21T14:11:23.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:23 smithi187 ceph-mon[33032]: osdmap e577: 8 total, 8 up, 8 in 2024-09-21T14:11:23.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:23 smithi187 ceph-mon[33032]: 6.e scrub starts 2024-09-21T14:11:23.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:23 smithi187 ceph-mon[33032]: 6.e scrub ok 2024-09-21T14:11:24.260 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T14:11:24.260 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":577,"fsid":"40a3d2e8-781f-11ef-baf6-efdc52797490","created":"2024-09-21T13:42:57.838275+0000","modified":"2024-09-21T14:11:22.459649+0000","last_up_change":"2024-09-21T14:01:40.158454+0000","last_in_change":"2024-09-21T13:58:16.730861+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":22,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T13:46:21.096244+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":21,"pg_placement_num":21,"pg_placement_num_target":21,"pg_num_target":21,"pg_num_pending":21,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"554","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"436","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8999999761581421,"score_stable":1.8999999761581421,"optimal_score":1,"raw_score_acting":1.8999999761581421,"raw_score_stable":1.8999999761581421,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-09-21T13:48:55.540712+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":63,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-09-21T13:48:56.417456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-21T13:48:58.128455+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-21T13:49:00.069361+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-21T13:49:02.127399+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2595dca9-6219-4e0c-8728-751eb102e73a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":570,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":2095561412},{"type":"v1","addr":"172.21.15.49:6803","nonce":2095561412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":2095561412},{"type":"v1","addr":"172.21.15.49:6805","nonce":2095561412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":2095561412},{"type":"v1","addr":"172.21.15.49:6809","nonce":2095561412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":2095561412},{"type":"v1","addr":"172.21.15.49:6807","nonce":2095561412}]},"public_addr":"172.21.15.49:6803/2095561412","cluster_addr":"172.21.15.49:6805/2095561412","heartbeat_back_addr":"172.21.15.49:6809/2095561412","heartbeat_front_addr":"172.21.15.49:6807/2095561412","state":["exists","up"]},{"osd":1,"uuid":"0ba327f9-5524-4a40-abcc-729ded0b1917","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":570,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":1382840527},{"type":"v1","addr":"172.21.15.49:6811","nonce":1382840527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":1382840527},{"type":"v1","addr":"172.21.15.49:6813","nonce":1382840527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":1382840527},{"type":"v1","addr":"172.21.15.49:6817","nonce":1382840527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":1382840527},{"type":"v1","addr":"172.21.15.49:6815","nonce":1382840527}]},"public_addr":"172.21.15.49:6811/1382840527","cluster_addr":"172.21.15.49:6813/1382840527","heartbeat_back_addr":"172.21.15.49:6817/1382840527","heartbeat_front_addr":"172.21.15.49:6815/1382840527","state":["exists","up"]},{"osd":2,"uuid":"27a45750-204d-48fd-bb8f-ebb01273343d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":100,"up_from":130,"up_thru":570,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":1945314741},{"type":"v1","addr":"172.21.15.49:6819","nonce":1945314741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":1945314741},{"type":"v1","addr":"172.21.15.49:6821","nonce":1945314741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":1945314741},{"type":"v1","addr":"172.21.15.49:6825","nonce":1945314741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":1945314741},{"type":"v1","addr":"172.21.15.49:6823","nonce":1945314741}]},"public_addr":"172.21.15.49:6819/1945314741","cluster_addr":"172.21.15.49:6821/1945314741","heartbeat_back_addr":"172.21.15.49:6825/1945314741","heartbeat_front_addr":"172.21.15.49:6823/1945314741","state":["exists","up"]},{"osd":3,"uuid":"528bc390-3dc9-4f3b-ab75-d7412c9625db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":27,"last_clean_end":214,"up_from":240,"up_thru":570,"down_at":215,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":886735721},{"type":"v1","addr":"172.21.15.49:6827","nonce":886735721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":886735721},{"type":"v1","addr":"172.21.15.49:6829","nonce":886735721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":886735721},{"type":"v1","addr":"172.21.15.49:6833","nonce":886735721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":886735721},{"type":"v1","addr":"172.21.15.49:6831","nonce":886735721}]},"public_addr":"172.21.15.49:6827/886735721","cluster_addr":"172.21.15.49:6829/886735721","heartbeat_back_addr":"172.21.15.49:6833/886735721","heartbeat_front_addr":"172.21.15.49:6831/886735721","state":["exists","up"]},{"osd":4,"uuid":"6fc64698-2d37-4db0-bf3d-c7e612c45163","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":570,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":1702447776},{"type":"v1","addr":"172.21.15.187:6801","nonce":1702447776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":1702447776},{"type":"v1","addr":"172.21.15.187:6803","nonce":1702447776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":1702447776},{"type":"v1","addr":"172.21.15.187:6807","nonce":1702447776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":1702447776},{"type":"v1","addr":"172.21.15.187:6805","nonce":1702447776}]},"public_addr":"172.21.15.187:6801/1702447776","cluster_addr":"172.21.15.187:6803/1702447776","heartbeat_back_addr":"172.21.15.187:6807/1702447776","heartbeat_front_addr":"172.21.15.187:6805/1702447776","state":["exists","up"]},{"osd":5,"uuid":"8d7f62a1-7e92-4e06-9cad-86b96be514d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":39,"last_clean_end":324,"up_from":345,"up_thru":570,"down_at":325,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1595353965},{"type":"v1","addr":"172.21.15.187:6809","nonce":1595353965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1595353965},{"type":"v1","addr":"172.21.15.187:6811","nonce":1595353965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1595353965},{"type":"v1","addr":"172.21.15.187:6815","nonce":1595353965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1595353965},{"type":"v1","addr":"172.21.15.187:6813","nonce":1595353965}]},"public_addr":"172.21.15.187:6809/1595353965","cluster_addr":"172.21.15.187:6811/1595353965","heartbeat_back_addr":"172.21.15.187:6815/1595353965","heartbeat_front_addr":"172.21.15.187:6813/1595353965","state":["exists","up"]},{"osd":6,"uuid":"3a7afd13-26ce-444c-9cc0-81ce2eab6ce5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":570,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":2728426525},{"type":"v1","addr":"172.21.15.187:6817","nonce":2728426525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":2728426525},{"type":"v1","addr":"172.21.15.187:6819","nonce":2728426525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":2728426525},{"type":"v1","addr":"172.21.15.187:6823","nonce":2728426525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":2728426525},{"type":"v1","addr":"172.21.15.187:6821","nonce":2728426525}]},"public_addr":"172.21.15.187:6817/2728426525","cluster_addr":"172.21.15.187:6819/2728426525","heartbeat_back_addr":"172.21.15.187:6823/2728426525","heartbeat_front_addr":"172.21.15.187:6821/2728426525","state":["exists","up"]},{"osd":7,"uuid":"1e08d891-8747-402e-b4c2-aa3802ce8537","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":54,"last_clean_end":272,"up_from":298,"up_thru":574,"down_at":273,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":13894964},{"type":"v1","addr":"172.21.15.187:6825","nonce":13894964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":13894964},{"type":"v1","addr":"172.21.15.187:6827","nonce":13894964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":13894964},{"type":"v1","addr":"172.21.15.187:6831","nonce":13894964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":13894964},{"type":"v1","addr":"172.21.15.187:6829","nonce":13894964}]},"public_addr":"172.21.15.187:6825/13894964","cluster_addr":"172.21.15.187:6827/13894964","heartbeat_back_addr":"172.21.15.187:6831/13894964","heartbeat_front_addr":"172.21.15.187:6829/13894964","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T14:10:08.769377+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T14:11:19.341223+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T14:10:26.407454+0000","dead_epoch":100},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T14:10:35.089888+0000","dead_epoch":214},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T14:10:55.308181+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T14:10:33.625384+0000","dead_epoch":324},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T14:11:07.152349+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T14:10:42.109154+0000","dead_epoch":272}],"pg_upmap":[{"pgid":"1.9","osds":[2,3,0]},{"pgid":"3.14","osds":[7,6,0]}],"pg_upmap_items":[{"pgid":"3.1c","mappings":[{"from":4,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.1c","mappings":[{"from":1,"to":5}]},{"pgid":"4.1d","mappings":[{"from":1,"to":2}]},{"pgid":"5.16","mappings":[{"from":1,"to":6},{"from":3,"to":5}]},{"pgid":"6.7","mappings":[{"from":3,"to":7}]},{"pgid":"6.b","mappings":[{"from":3,"to":1},{"from":7,"to":6}]},{"pgid":"6.1f","mappings":[{"from":3,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.49:0/2731793917":"2024-09-22T13:49:27.311313+0000","172.21.15.49:0/799831425":"2024-09-22T13:49:27.311313+0000","172.21.15.49:6801/2518197280":"2024-09-22T13:43:28.110499+0000","172.21.15.49:0/3556829486":"2024-09-22T13:43:28.110499+0000","172.21.15.49:0/1249926115":"2024-09-22T13:49:27.311313+0000","172.21.15.49:0/231530138":"2024-09-22T13:43:28.110499+0000","172.21.15.49:0/4220781134":"2024-09-22T13:49:27.311313+0000","172.21.15.49:0/202325455":"2024-09-22T13:43:28.110499+0000","172.21.15.49:6800/4129636473":"2024-09-22T13:43:56.633723+0000","172.21.15.49:6800/2518197280":"2024-09-22T13:43:28.110499+0000","172.21.15.49:0/3514741273":"2024-09-22T13:43:56.633723+0000","172.21.15.49:6801/4129636473":"2024-09-22T13:43:56.633723+0000","172.21.15.49:0/3125743776":"2024-09-22T13:43:56.633723+0000","172.21.15.49:0/3600977363":"2024-09-22T13:43:56.633723+0000","172.21.15.49:6801/3453571598":"2024-09-22T13:49:27.311313+0000","172.21.15.49:6800/3453571598":"2024-09-22T13:49:27.311313+0000","172.21.15.49:0/3553734432":"2024-09-22T13:49:27.311313+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T14:11:24.606 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:24 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3083122206' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T14:11:24.606 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:24 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3083122206' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T14:11:24.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:24 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3083122206' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T14:11:25.046 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-21T14:11:25.046 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd dump --format=json 2024-09-21T14:11:25.307 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:11:25.753 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:25 smithi049 ceph-mon[40489]: pgmap v1164: 152 pgs: 152 active+clean; 583 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 44 KiB/s rd, 43 op/s 2024-09-21T14:11:25.753 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:25 smithi049 ceph-mon[40489]: 6.11 scrub starts 2024-09-21T14:11:25.753 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:25 smithi049 ceph-mon[40489]: 6.11 scrub ok 2024-09-21T14:11:25.753 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:25 smithi049 ceph-mon[40489]: 5.1d scrub starts 2024-09-21T14:11:25.753 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:25 smithi049 ceph-mon[40489]: 5.1d scrub ok 2024-09-21T14:11:25.753 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:25 smithi049 ceph-mon[40489]: 4.13 scrub starts 2024-09-21T14:11:25.753 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:25 smithi049 ceph-mon[40489]: 4.13 scrub ok 2024-09-21T14:11:25.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:25 smithi049 ceph-mon[40489]: osdmap e578: 8 total, 8 up, 8 in 2024-09-21T14:11:25.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:25 smithi049 ceph-mon[30080]: pgmap v1164: 152 pgs: 152 active+clean; 583 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 44 KiB/s rd, 43 op/s 2024-09-21T14:11:25.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:25 smithi049 ceph-mon[30080]: 6.11 scrub starts 2024-09-21T14:11:25.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:25 smithi049 ceph-mon[30080]: 6.11 scrub ok 2024-09-21T14:11:25.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:25 smithi049 ceph-mon[30080]: 5.1d scrub starts 2024-09-21T14:11:25.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:25 smithi049 ceph-mon[30080]: 5.1d scrub ok 2024-09-21T14:11:25.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:25 smithi049 ceph-mon[30080]: 4.13 scrub starts 2024-09-21T14:11:25.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:25 smithi049 ceph-mon[30080]: 4.13 scrub ok 2024-09-21T14:11:25.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:25 smithi049 ceph-mon[30080]: osdmap e578: 8 total, 8 up, 8 in 2024-09-21T14:11:25.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:25 smithi187 ceph-mon[33032]: pgmap v1164: 152 pgs: 152 active+clean; 583 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 44 KiB/s rd, 43 op/s 2024-09-21T14:11:25.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:25 smithi187 ceph-mon[33032]: 6.11 scrub starts 2024-09-21T14:11:25.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:25 smithi187 ceph-mon[33032]: 6.11 scrub ok 2024-09-21T14:11:25.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:25 smithi187 ceph-mon[33032]: 5.1d scrub starts 2024-09-21T14:11:25.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:25 smithi187 ceph-mon[33032]: 5.1d scrub ok 2024-09-21T14:11:25.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:25 smithi187 ceph-mon[33032]: 4.13 scrub starts 2024-09-21T14:11:25.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:25 smithi187 ceph-mon[33032]: 4.13 scrub ok 2024-09-21T14:11:25.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:25 smithi187 ceph-mon[33032]: osdmap e578: 8 total, 8 up, 8 in 2024-09-21T14:11:26.034 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T14:11:26.035 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":578,"fsid":"40a3d2e8-781f-11ef-baf6-efdc52797490","created":"2024-09-21T13:42:57.838275+0000","modified":"2024-09-21T14:11:25.284017+0000","last_up_change":"2024-09-21T14:01:40.158454+0000","last_in_change":"2024-09-21T13:58:16.730861+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":22,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T13:46:21.096244+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":21,"pg_placement_num":21,"pg_placement_num_target":21,"pg_num_target":21,"pg_num_pending":21,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"554","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"436","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8999999761581421,"score_stable":1.8999999761581421,"optimal_score":1,"raw_score_acting":1.8999999761581421,"raw_score_stable":1.8999999761581421,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-09-21T13:48:55.540712+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":63,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-09-21T13:48:56.417456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-21T13:48:58.128455+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-21T13:49:00.069361+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-21T13:49:02.127399+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2595dca9-6219-4e0c-8728-751eb102e73a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":570,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":2095561412},{"type":"v1","addr":"172.21.15.49:6803","nonce":2095561412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":2095561412},{"type":"v1","addr":"172.21.15.49:6805","nonce":2095561412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":2095561412},{"type":"v1","addr":"172.21.15.49:6809","nonce":2095561412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":2095561412},{"type":"v1","addr":"172.21.15.49:6807","nonce":2095561412}]},"public_addr":"172.21.15.49:6803/2095561412","cluster_addr":"172.21.15.49:6805/2095561412","heartbeat_back_addr":"172.21.15.49:6809/2095561412","heartbeat_front_addr":"172.21.15.49:6807/2095561412","state":["exists","up"]},{"osd":1,"uuid":"0ba327f9-5524-4a40-abcc-729ded0b1917","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":570,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":1382840527},{"type":"v1","addr":"172.21.15.49:6811","nonce":1382840527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":1382840527},{"type":"v1","addr":"172.21.15.49:6813","nonce":1382840527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":1382840527},{"type":"v1","addr":"172.21.15.49:6817","nonce":1382840527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":1382840527},{"type":"v1","addr":"172.21.15.49:6815","nonce":1382840527}]},"public_addr":"172.21.15.49:6811/1382840527","cluster_addr":"172.21.15.49:6813/1382840527","heartbeat_back_addr":"172.21.15.49:6817/1382840527","heartbeat_front_addr":"172.21.15.49:6815/1382840527","state":["exists","up"]},{"osd":2,"uuid":"27a45750-204d-48fd-bb8f-ebb01273343d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":100,"up_from":130,"up_thru":570,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":1945314741},{"type":"v1","addr":"172.21.15.49:6819","nonce":1945314741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":1945314741},{"type":"v1","addr":"172.21.15.49:6821","nonce":1945314741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":1945314741},{"type":"v1","addr":"172.21.15.49:6825","nonce":1945314741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":1945314741},{"type":"v1","addr":"172.21.15.49:6823","nonce":1945314741}]},"public_addr":"172.21.15.49:6819/1945314741","cluster_addr":"172.21.15.49:6821/1945314741","heartbeat_back_addr":"172.21.15.49:6825/1945314741","heartbeat_front_addr":"172.21.15.49:6823/1945314741","state":["exists","up"]},{"osd":3,"uuid":"528bc390-3dc9-4f3b-ab75-d7412c9625db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":27,"last_clean_end":214,"up_from":240,"up_thru":570,"down_at":215,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":886735721},{"type":"v1","addr":"172.21.15.49:6827","nonce":886735721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":886735721},{"type":"v1","addr":"172.21.15.49:6829","nonce":886735721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":886735721},{"type":"v1","addr":"172.21.15.49:6833","nonce":886735721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":886735721},{"type":"v1","addr":"172.21.15.49:6831","nonce":886735721}]},"public_addr":"172.21.15.49:6827/886735721","cluster_addr":"172.21.15.49:6829/886735721","heartbeat_back_addr":"172.21.15.49:6833/886735721","heartbeat_front_addr":"172.21.15.49:6831/886735721","state":["exists","up"]},{"osd":4,"uuid":"6fc64698-2d37-4db0-bf3d-c7e612c45163","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":570,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":1702447776},{"type":"v1","addr":"172.21.15.187:6801","nonce":1702447776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":1702447776},{"type":"v1","addr":"172.21.15.187:6803","nonce":1702447776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":1702447776},{"type":"v1","addr":"172.21.15.187:6807","nonce":1702447776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":1702447776},{"type":"v1","addr":"172.21.15.187:6805","nonce":1702447776}]},"public_addr":"172.21.15.187:6801/1702447776","cluster_addr":"172.21.15.187:6803/1702447776","heartbeat_back_addr":"172.21.15.187:6807/1702447776","heartbeat_front_addr":"172.21.15.187:6805/1702447776","state":["exists","up"]},{"osd":5,"uuid":"8d7f62a1-7e92-4e06-9cad-86b96be514d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":39,"last_clean_end":324,"up_from":345,"up_thru":570,"down_at":325,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":1595353965},{"type":"v1","addr":"172.21.15.187:6809","nonce":1595353965}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":1595353965},{"type":"v1","addr":"172.21.15.187:6811","nonce":1595353965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":1595353965},{"type":"v1","addr":"172.21.15.187:6815","nonce":1595353965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":1595353965},{"type":"v1","addr":"172.21.15.187:6813","nonce":1595353965}]},"public_addr":"172.21.15.187:6809/1595353965","cluster_addr":"172.21.15.187:6811/1595353965","heartbeat_back_addr":"172.21.15.187:6815/1595353965","heartbeat_front_addr":"172.21.15.187:6813/1595353965","state":["exists","up"]},{"osd":6,"uuid":"3a7afd13-26ce-444c-9cc0-81ce2eab6ce5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":570,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":2728426525},{"type":"v1","addr":"172.21.15.187:6817","nonce":2728426525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":2728426525},{"type":"v1","addr":"172.21.15.187:6819","nonce":2728426525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":2728426525},{"type":"v1","addr":"172.21.15.187:6823","nonce":2728426525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":2728426525},{"type":"v1","addr":"172.21.15.187:6821","nonce":2728426525}]},"public_addr":"172.21.15.187:6817/2728426525","cluster_addr":"172.21.15.187:6819/2728426525","heartbeat_back_addr":"172.21.15.187:6823/2728426525","heartbeat_front_addr":"172.21.15.187:6821/2728426525","state":["exists","up"]},{"osd":7,"uuid":"1e08d891-8747-402e-b4c2-aa3802ce8537","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":54,"last_clean_end":272,"up_from":298,"up_thru":574,"down_at":273,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":13894964},{"type":"v1","addr":"172.21.15.187:6825","nonce":13894964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":13894964},{"type":"v1","addr":"172.21.15.187:6827","nonce":13894964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":13894964},{"type":"v1","addr":"172.21.15.187:6831","nonce":13894964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":13894964},{"type":"v1","addr":"172.21.15.187:6829","nonce":13894964}]},"public_addr":"172.21.15.187:6825/13894964","cluster_addr":"172.21.15.187:6827/13894964","heartbeat_back_addr":"172.21.15.187:6831/13894964","heartbeat_front_addr":"172.21.15.187:6829/13894964","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T14:10:08.769377+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T14:11:19.341223+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T14:10:26.407454+0000","dead_epoch":100},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T14:10:35.089888+0000","dead_epoch":214},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T14:10:55.308181+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T14:10:33.625384+0000","dead_epoch":324},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T14:11:07.152349+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T14:10:42.109154+0000","dead_epoch":272}],"pg_upmap":[{"pgid":"1.9","osds":[2,3,0]},{"pgid":"3.14","osds":[7,6,0]}],"pg_upmap_items":[{"pgid":"3.1c","mappings":[{"from":4,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.1c","mappings":[{"from":1,"to":5}]},{"pgid":"4.1d","mappings":[{"from":1,"to":2}]},{"pgid":"5.16","mappings":[{"from":1,"to":6},{"from":3,"to":5}]},{"pgid":"6.7","mappings":[{"from":3,"to":7}]},{"pgid":"6.b","mappings":[{"from":3,"to":1},{"from":7,"to":6}]},{"pgid":"6.1f","mappings":[{"from":3,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.49:0/2731793917":"2024-09-22T13:49:27.311313+0000","172.21.15.49:0/799831425":"2024-09-22T13:49:27.311313+0000","172.21.15.49:6801/2518197280":"2024-09-22T13:43:28.110499+0000","172.21.15.49:0/3556829486":"2024-09-22T13:43:28.110499+0000","172.21.15.49:0/1249926115":"2024-09-22T13:49:27.311313+0000","172.21.15.49:0/231530138":"2024-09-22T13:43:28.110499+0000","172.21.15.49:0/4220781134":"2024-09-22T13:49:27.311313+0000","172.21.15.49:0/202325455":"2024-09-22T13:43:28.110499+0000","172.21.15.49:6800/4129636473":"2024-09-22T13:43:56.633723+0000","172.21.15.49:6800/2518197280":"2024-09-22T13:43:28.110499+0000","172.21.15.49:0/3514741273":"2024-09-22T13:43:56.633723+0000","172.21.15.49:6801/4129636473":"2024-09-22T13:43:56.633723+0000","172.21.15.49:0/3125743776":"2024-09-22T13:43:56.633723+0000","172.21.15.49:0/3600977363":"2024-09-22T13:43:56.633723+0000","172.21.15.49:6801/3453571598":"2024-09-22T13:49:27.311313+0000","172.21.15.49:6800/3453571598":"2024-09-22T13:49:27.311313+0000","172.21.15.49:0/3553734432":"2024-09-22T13:49:27.311313+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T14:11:26.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:26 smithi049 ceph-mon[40489]: 6.13 scrub starts 2024-09-21T14:11:26.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:26 smithi049 ceph-mon[40489]: 6.13 scrub ok 2024-09-21T14:11:26.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:26 smithi049 ceph-mon[40489]: 3.c scrub starts 2024-09-21T14:11:26.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:26 smithi049 ceph-mon[40489]: 3.c scrub ok 2024-09-21T14:11:26.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:26 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2022481382' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T14:11:26.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:26 smithi049 ceph-mon[40489]: 6.10 scrub starts 2024-09-21T14:11:26.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:26 smithi049 ceph-mon[40489]: 6.10 scrub ok 2024-09-21T14:11:26.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:26 smithi049 ceph-mon[30080]: 6.13 scrub starts 2024-09-21T14:11:26.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:26 smithi049 ceph-mon[30080]: 6.13 scrub ok 2024-09-21T14:11:26.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:26 smithi049 ceph-mon[30080]: 3.c scrub starts 2024-09-21T14:11:26.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:26 smithi049 ceph-mon[30080]: 3.c scrub ok 2024-09-21T14:11:26.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:26 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2022481382' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T14:11:26.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:26 smithi049 ceph-mon[30080]: 6.10 scrub starts 2024-09-21T14:11:26.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:26 smithi049 ceph-mon[30080]: 6.10 scrub ok 2024-09-21T14:11:26.893 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph tell osd.0 flush_pg_stats 2024-09-21T14:11:26.894 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph tell osd.1 flush_pg_stats 2024-09-21T14:11:26.894 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph tell osd.2 flush_pg_stats 2024-09-21T14:11:26.895 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph tell osd.3 flush_pg_stats 2024-09-21T14:11:26.895 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph tell osd.4 flush_pg_stats 2024-09-21T14:11:26.895 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph tell osd.5 flush_pg_stats 2024-09-21T14:11:26.896 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph tell osd.6 flush_pg_stats 2024-09-21T14:11:26.896 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph tell osd.7 flush_pg_stats 2024-09-21T14:11:26.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:26 smithi187 ceph-mon[33032]: 6.13 scrub starts 2024-09-21T14:11:26.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:26 smithi187 ceph-mon[33032]: 6.13 scrub ok 2024-09-21T14:11:26.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:26 smithi187 ceph-mon[33032]: 3.c scrub starts 2024-09-21T14:11:26.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:26 smithi187 ceph-mon[33032]: 3.c scrub ok 2024-09-21T14:11:26.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:26 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2022481382' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T14:11:26.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:26 smithi187 ceph-mon[33032]: 6.10 scrub starts 2024-09-21T14:11:26.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:26 smithi187 ceph-mon[33032]: 6.10 scrub ok 2024-09-21T14:11:27.429 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:11:27.429 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:11:27.451 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:11:27.453 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:11:27.459 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:11:27.461 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:11:27.464 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:11:27.488 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:11:27.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:27 smithi049 ceph-mon[30080]: pgmap v1166: 152 pgs: 152 active+clean; 583 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 42 KiB/s rd, 42 op/s 2024-09-21T14:11:27.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:27 smithi049 ceph-mon[30080]: 6.6 scrub starts 2024-09-21T14:11:27.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:27 smithi049 ceph-mon[30080]: 6.6 scrub ok 2024-09-21T14:11:27.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:27 smithi049 ceph-mon[30080]: 6.f scrub starts 2024-09-21T14:11:27.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:27 smithi049 ceph-mon[30080]: 6.f scrub ok 2024-09-21T14:11:27.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:27 smithi049 ceph-mon[30080]: 4.18 scrub starts 2024-09-21T14:11:27.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:27 smithi049 ceph-mon[30080]: 4.18 scrub ok 2024-09-21T14:11:27.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:27 smithi049 ceph-mon[40489]: pgmap v1166: 152 pgs: 152 active+clean; 583 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 42 KiB/s rd, 42 op/s 2024-09-21T14:11:27.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:27 smithi049 ceph-mon[40489]: 6.6 scrub starts 2024-09-21T14:11:27.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:27 smithi049 ceph-mon[40489]: 6.6 scrub ok 2024-09-21T14:11:27.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:27 smithi049 ceph-mon[40489]: 6.f scrub starts 2024-09-21T14:11:27.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:27 smithi049 ceph-mon[40489]: 6.f scrub ok 2024-09-21T14:11:27.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:27 smithi049 ceph-mon[40489]: 4.18 scrub starts 2024-09-21T14:11:27.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:27 smithi049 ceph-mon[40489]: 4.18 scrub ok 2024-09-21T14:11:27.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:27 smithi187 ceph-mon[33032]: pgmap v1166: 152 pgs: 152 active+clean; 583 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 42 KiB/s rd, 42 op/s 2024-09-21T14:11:27.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:27 smithi187 ceph-mon[33032]: 6.6 scrub starts 2024-09-21T14:11:27.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:27 smithi187 ceph-mon[33032]: 6.6 scrub ok 2024-09-21T14:11:27.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:27 smithi187 ceph-mon[33032]: 6.f scrub starts 2024-09-21T14:11:27.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:27 smithi187 ceph-mon[33032]: 6.f scrub ok 2024-09-21T14:11:27.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:27 smithi187 ceph-mon[33032]: 4.18 scrub starts 2024-09-21T14:11:27.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:27 smithi187 ceph-mon[33032]: 4.18 scrub ok 2024-09-21T14:11:28.747 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:28 smithi049 ceph-mon[40489]: 1.13 scrub starts 2024-09-21T14:11:28.748 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:28 smithi049 ceph-mon[40489]: 1.13 scrub ok 2024-09-21T14:11:28.748 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:28 smithi049 ceph-mon[40489]: 1.11 scrub starts 2024-09-21T14:11:28.748 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:28 smithi049 ceph-mon[40489]: 1.11 scrub ok 2024-09-21T14:11:28.748 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:28 smithi049 ceph-mon[40489]: 3.10 scrub starts 2024-09-21T14:11:28.748 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:28 smithi049 ceph-mon[40489]: 3.10 scrub ok 2024-09-21T14:11:28.748 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:28 smithi049 ceph-mon[30080]: 1.13 scrub starts 2024-09-21T14:11:28.748 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:28 smithi049 ceph-mon[30080]: 1.13 scrub ok 2024-09-21T14:11:28.748 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:28 smithi049 ceph-mon[30080]: 1.11 scrub starts 2024-09-21T14:11:28.748 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:28 smithi049 ceph-mon[30080]: 1.11 scrub ok 2024-09-21T14:11:28.748 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:28 smithi049 ceph-mon[30080]: 3.10 scrub starts 2024-09-21T14:11:28.748 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:28 smithi049 ceph-mon[30080]: 3.10 scrub ok 2024-09-21T14:11:28.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:28 smithi187 ceph-mon[33032]: 1.13 scrub starts 2024-09-21T14:11:28.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:28 smithi187 ceph-mon[33032]: 1.13 scrub ok 2024-09-21T14:11:28.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:28 smithi187 ceph-mon[33032]: 1.11 scrub starts 2024-09-21T14:11:28.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:28 smithi187 ceph-mon[33032]: 1.11 scrub ok 2024-09-21T14:11:28.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:28 smithi187 ceph-mon[33032]: 3.10 scrub starts 2024-09-21T14:11:28.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:28 smithi187 ceph-mon[33032]: 3.10 scrub ok 2024-09-21T14:11:29.729 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:29 smithi049 ceph-mon[40489]: pgmap v1167: 152 pgs: 152 active+clean; 583 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 33 KiB/s rd, 32 op/s 2024-09-21T14:11:29.729 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:29 smithi049 ceph-mon[40489]: 6.1b scrub starts 2024-09-21T14:11:29.729 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:29 smithi049 ceph-mon[40489]: 6.1b scrub ok 2024-09-21T14:11:29.729 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:29 smithi049 ceph-mon[40489]: 6.d scrub starts 2024-09-21T14:11:29.729 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:29 smithi049 ceph-mon[40489]: 6.d scrub ok 2024-09-21T14:11:29.729 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:29 smithi049 ceph-mon[40489]: 5.11 scrub starts 2024-09-21T14:11:29.729 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:29 smithi049 ceph-mon[40489]: 5.11 scrub ok 2024-09-21T14:11:29.730 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:29 smithi049 ceph-mon[30080]: pgmap v1167: 152 pgs: 152 active+clean; 583 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 33 KiB/s rd, 32 op/s 2024-09-21T14:11:29.730 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:29 smithi049 ceph-mon[30080]: 6.1b scrub starts 2024-09-21T14:11:29.730 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:29 smithi049 ceph-mon[30080]: 6.1b scrub ok 2024-09-21T14:11:29.730 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:29 smithi049 ceph-mon[30080]: 6.d scrub starts 2024-09-21T14:11:29.730 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:29 smithi049 ceph-mon[30080]: 6.d scrub ok 2024-09-21T14:11:29.730 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:29 smithi049 ceph-mon[30080]: 5.11 scrub starts 2024-09-21T14:11:29.730 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:29 smithi049 ceph-mon[30080]: 5.11 scrub ok 2024-09-21T14:11:29.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:29 smithi187 ceph-mon[33032]: pgmap v1167: 152 pgs: 152 active+clean; 583 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 33 KiB/s rd, 32 op/s 2024-09-21T14:11:29.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:29 smithi187 ceph-mon[33032]: 6.1b scrub starts 2024-09-21T14:11:29.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:29 smithi187 ceph-mon[33032]: 6.1b scrub ok 2024-09-21T14:11:29.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:29 smithi187 ceph-mon[33032]: 6.d scrub starts 2024-09-21T14:11:29.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:29 smithi187 ceph-mon[33032]: 6.d scrub ok 2024-09-21T14:11:29.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:29 smithi187 ceph-mon[33032]: 5.11 scrub starts 2024-09-21T14:11:29.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:29 smithi187 ceph-mon[33032]: 5.11 scrub ok 2024-09-21T14:11:30.172 INFO:teuthology.orchestra.run.smithi049.stdout:558345748715 2024-09-21T14:11:30.173 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.2 2024-09-21T14:11:30.694 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:11:30 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:11:30.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:30 smithi187 ceph-mon[33032]: 3.15 scrub starts 2024-09-21T14:11:30.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:30 smithi187 ceph-mon[33032]: 3.15 scrub ok 2024-09-21T14:11:30.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:30 smithi187 ceph-mon[33032]: 5.b scrub starts 2024-09-21T14:11:30.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:30 smithi187 ceph-mon[33032]: 5.b scrub ok 2024-09-21T14:11:30.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:30 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:11:30.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:30 smithi187 ceph-mon[33032]: 4.6 scrub starts 2024-09-21T14:11:30.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:30 smithi187 ceph-mon[33032]: 4.6 scrub ok 2024-09-21T14:11:30.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:30 smithi187 ceph-mon[33032]: 4.a scrub starts 2024-09-21T14:11:30.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:30 smithi187 ceph-mon[33032]: 4.a scrub ok 2024-09-21T14:11:30.754 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:11:30 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:11:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:11:30.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:30 smithi049 ceph-mon[40489]: 3.15 scrub starts 2024-09-21T14:11:30.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:30 smithi049 ceph-mon[40489]: 3.15 scrub ok 2024-09-21T14:11:30.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:30 smithi049 ceph-mon[40489]: 5.b scrub starts 2024-09-21T14:11:30.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:30 smithi049 ceph-mon[40489]: 5.b scrub ok 2024-09-21T14:11:30.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:30 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:11:30.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:30 smithi049 ceph-mon[40489]: 4.6 scrub starts 2024-09-21T14:11:30.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:30 smithi049 ceph-mon[40489]: 4.6 scrub ok 2024-09-21T14:11:30.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:30 smithi049 ceph-mon[40489]: 4.a scrub starts 2024-09-21T14:11:30.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:30 smithi049 ceph-mon[40489]: 4.a scrub ok 2024-09-21T14:11:30.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:30 smithi049 ceph-mon[30080]: 3.15 scrub starts 2024-09-21T14:11:30.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:30 smithi049 ceph-mon[30080]: 3.15 scrub ok 2024-09-21T14:11:30.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:30 smithi049 ceph-mon[30080]: 5.b scrub starts 2024-09-21T14:11:30.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:30 smithi049 ceph-mon[30080]: 5.b scrub ok 2024-09-21T14:11:30.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:30 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:11:30.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:30 smithi049 ceph-mon[30080]: 4.6 scrub starts 2024-09-21T14:11:30.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:30 smithi049 ceph-mon[30080]: 4.6 scrub ok 2024-09-21T14:11:30.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:30 smithi049 ceph-mon[30080]: 4.a scrub starts 2024-09-21T14:11:30.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:30 smithi049 ceph-mon[30080]: 4.a scrub ok 2024-09-21T14:11:31.627 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:31 smithi049 ceph-mon[40489]: pgmap v1168: 152 pgs: 152 active+clean; 583 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 895 B/s rd, 0 op/s 2024-09-21T14:11:31.627 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:31 smithi049 ceph-mon[40489]: 6.b scrub starts 2024-09-21T14:11:31.628 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:31 smithi049 ceph-mon[40489]: 6.b scrub ok 2024-09-21T14:11:31.628 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:31 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:11:31.628 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:31 smithi049 ceph-mon[40489]: 6.4 scrub starts 2024-09-21T14:11:31.628 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:31 smithi049 ceph-mon[40489]: 6.4 scrub ok 2024-09-21T14:11:31.628 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:31 smithi049 ceph-mon[40489]: 5.18 scrub starts 2024-09-21T14:11:31.628 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:31 smithi049 ceph-mon[40489]: 5.18 scrub ok 2024-09-21T14:11:31.628 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:31 smithi049 ceph-mon[40489]: 1.0 scrub starts 2024-09-21T14:11:31.628 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:31 smithi049 ceph-mon[40489]: 1.0 scrub ok 2024-09-21T14:11:31.628 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:31 smithi049 ceph-mon[30080]: pgmap v1168: 152 pgs: 152 active+clean; 583 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 895 B/s rd, 0 op/s 2024-09-21T14:11:31.628 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:31 smithi049 ceph-mon[30080]: 6.b scrub starts 2024-09-21T14:11:31.628 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:31 smithi049 ceph-mon[30080]: 6.b scrub ok 2024-09-21T14:11:31.629 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:31 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:11:31.629 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:31 smithi049 ceph-mon[30080]: 6.4 scrub starts 2024-09-21T14:11:31.629 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:31 smithi049 ceph-mon[30080]: 6.4 scrub ok 2024-09-21T14:11:31.629 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:31 smithi049 ceph-mon[30080]: 5.18 scrub starts 2024-09-21T14:11:31.629 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:31 smithi049 ceph-mon[30080]: 5.18 scrub ok 2024-09-21T14:11:31.629 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:31 smithi049 ceph-mon[30080]: 1.0 scrub starts 2024-09-21T14:11:31.629 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:31 smithi049 ceph-mon[30080]: 1.0 scrub ok 2024-09-21T14:11:31.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:31 smithi187 ceph-mon[33032]: pgmap v1168: 152 pgs: 152 active+clean; 583 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 895 B/s rd, 0 op/s 2024-09-21T14:11:31.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:31 smithi187 ceph-mon[33032]: 6.b scrub starts 2024-09-21T14:11:31.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:31 smithi187 ceph-mon[33032]: 6.b scrub ok 2024-09-21T14:11:31.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:31 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:11:31.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:31 smithi187 ceph-mon[33032]: 6.4 scrub starts 2024-09-21T14:11:31.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:31 smithi187 ceph-mon[33032]: 6.4 scrub ok 2024-09-21T14:11:31.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:31 smithi187 ceph-mon[33032]: 5.18 scrub starts 2024-09-21T14:11:31.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:31 smithi187 ceph-mon[33032]: 5.18 scrub ok 2024-09-21T14:11:31.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:31 smithi187 ceph-mon[33032]: 1.0 scrub starts 2024-09-21T14:11:31.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:31 smithi187 ceph-mon[33032]: 1.0 scrub ok 2024-09-21T14:11:32.045 INFO:teuthology.orchestra.run.smithi049.stdout:55834575160 2024-09-21T14:11:32.046 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.1 2024-09-21T14:11:32.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:32 smithi049 ceph-mon[40489]: 4.1f scrub starts 2024-09-21T14:11:32.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:32 smithi049 ceph-mon[40489]: 4.1f scrub ok 2024-09-21T14:11:32.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:32 smithi049 ceph-mon[30080]: 4.1f scrub starts 2024-09-21T14:11:32.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:32 smithi049 ceph-mon[30080]: 4.1f scrub ok 2024-09-21T14:11:32.938 INFO:teuthology.orchestra.run.smithi049.stdout:1030792151213 2024-09-21T14:11:32.938 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.3 2024-09-21T14:11:32.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:32 smithi187 ceph-mon[33032]: 4.1f scrub starts 2024-09-21T14:11:32.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:32 smithi187 ceph-mon[33032]: 4.1f scrub ok 2024-09-21T14:11:33.508 INFO:teuthology.orchestra.run.smithi049.stdout:1279900254353 2024-09-21T14:11:33.509 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.7 2024-09-21T14:11:33.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:33 smithi049 ceph-mon[40489]: pgmap v1169: 152 pgs: 152 active+clean; 583 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-21T14:11:33.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:33 smithi049 ceph-mon[40489]: 6.19 scrub starts 2024-09-21T14:11:33.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:33 smithi049 ceph-mon[40489]: 6.19 scrub ok 2024-09-21T14:11:33.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:33 smithi049 ceph-mon[40489]: 3.12 scrub starts 2024-09-21T14:11:33.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:33 smithi049 ceph-mon[40489]: 3.12 scrub ok 2024-09-21T14:11:33.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:33 smithi049 ceph-mon[30080]: pgmap v1169: 152 pgs: 152 active+clean; 583 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-21T14:11:33.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:33 smithi049 ceph-mon[30080]: 6.19 scrub starts 2024-09-21T14:11:33.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:33 smithi049 ceph-mon[30080]: 6.19 scrub ok 2024-09-21T14:11:33.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:33 smithi049 ceph-mon[30080]: 3.12 scrub starts 2024-09-21T14:11:33.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:33 smithi049 ceph-mon[30080]: 3.12 scrub ok 2024-09-21T14:11:33.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:33 smithi187 ceph-mon[33032]: pgmap v1169: 152 pgs: 152 active+clean; 583 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-21T14:11:33.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:33 smithi187 ceph-mon[33032]: 6.19 scrub starts 2024-09-21T14:11:33.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:33 smithi187 ceph-mon[33032]: 6.19 scrub ok 2024-09-21T14:11:33.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:33 smithi187 ceph-mon[33032]: 3.12 scrub starts 2024-09-21T14:11:33.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:33 smithi187 ceph-mon[33032]: 3.12 scrub ok 2024-09-21T14:11:34.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:34 smithi049 ceph-mon[40489]: 3.18 scrub starts 2024-09-21T14:11:34.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:34 smithi049 ceph-mon[40489]: 3.18 scrub ok 2024-09-21T14:11:34.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:34 smithi049 ceph-mon[40489]: 6.a scrub starts 2024-09-21T14:11:34.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:34 smithi049 ceph-mon[40489]: 6.a scrub ok 2024-09-21T14:11:34.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:34 smithi049 ceph-mon[40489]: 6.1 scrub starts 2024-09-21T14:11:34.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:34 smithi049 ceph-mon[40489]: 6.1 scrub ok 2024-09-21T14:11:34.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:34 smithi049 ceph-mon[40489]: 5.2 scrub starts 2024-09-21T14:11:34.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:34 smithi049 ceph-mon[40489]: 5.2 scrub ok 2024-09-21T14:11:34.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:34 smithi049 ceph-mon[30080]: 3.18 scrub starts 2024-09-21T14:11:34.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:34 smithi049 ceph-mon[30080]: 3.18 scrub ok 2024-09-21T14:11:34.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:34 smithi049 ceph-mon[30080]: 6.a scrub starts 2024-09-21T14:11:34.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:34 smithi049 ceph-mon[30080]: 6.a scrub ok 2024-09-21T14:11:34.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:34 smithi049 ceph-mon[30080]: 6.1 scrub starts 2024-09-21T14:11:34.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:34 smithi049 ceph-mon[30080]: 6.1 scrub ok 2024-09-21T14:11:34.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:34 smithi049 ceph-mon[30080]: 5.2 scrub starts 2024-09-21T14:11:34.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:34 smithi049 ceph-mon[30080]: 5.2 scrub ok 2024-09-21T14:11:34.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:34 smithi187 ceph-mon[33032]: 3.18 scrub starts 2024-09-21T14:11:34.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:34 smithi187 ceph-mon[33032]: 3.18 scrub ok 2024-09-21T14:11:34.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:34 smithi187 ceph-mon[33032]: 6.a scrub starts 2024-09-21T14:11:34.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:34 smithi187 ceph-mon[33032]: 6.a scrub ok 2024-09-21T14:11:34.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:34 smithi187 ceph-mon[33032]: 6.1 scrub starts 2024-09-21T14:11:34.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:34 smithi187 ceph-mon[33032]: 6.1 scrub ok 2024-09-21T14:11:34.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:34 smithi187 ceph-mon[33032]: 5.2 scrub starts 2024-09-21T14:11:34.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:34 smithi187 ceph-mon[33032]: 5.2 scrub ok 2024-09-21T14:11:35.372 INFO:teuthology.orchestra.run.smithi049.stdout:201863463194 2024-09-21T14:11:35.373 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.6 2024-09-21T14:11:35.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:35 smithi049 ceph-mon[30080]: pgmap v1170: 152 pgs: 152 active+clean; 583 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-21T14:11:35.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:35 smithi049 ceph-mon[30080]: 6.8 scrub starts 2024-09-21T14:11:35.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:35 smithi049 ceph-mon[30080]: 6.8 scrub ok 2024-09-21T14:11:35.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:35 smithi049 ceph-mon[40489]: pgmap v1170: 152 pgs: 152 active+clean; 583 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-21T14:11:35.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:35 smithi049 ceph-mon[40489]: 6.8 scrub starts 2024-09-21T14:11:35.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:35 smithi049 ceph-mon[40489]: 6.8 scrub ok 2024-09-21T14:11:35.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:35 smithi187 ceph-mon[33032]: pgmap v1170: 152 pgs: 152 active+clean; 583 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-21T14:11:35.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:35 smithi187 ceph-mon[33032]: 6.8 scrub starts 2024-09-21T14:11:35.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:35 smithi187 ceph-mon[33032]: 6.8 scrub ok 2024-09-21T14:11:36.673 INFO:teuthology.orchestra.run.smithi049.stdout:1481763717240 2024-09-21T14:11:36.673 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.5 2024-09-21T14:11:36.683 INFO:teuthology.orchestra.run.smithi049.stdout:34359738686 2024-09-21T14:11:36.683 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.0 2024-09-21T14:11:36.826 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:36 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:11:36.827 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:36 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:11:36.827 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:36 smithi049 ceph-mon[40489]: 5.12 scrub starts 2024-09-21T14:11:36.827 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:36 smithi049 ceph-mon[40489]: 5.12 scrub ok 2024-09-21T14:11:36.827 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:36 smithi049 ceph-mon[40489]: 6.18 scrub starts 2024-09-21T14:11:36.827 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:36 smithi049 ceph-mon[40489]: 6.18 scrub ok 2024-09-21T14:11:36.827 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:36 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:11:36.827 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:36 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:11:36.827 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:36 smithi049 ceph-mon[30080]: 5.12 scrub starts 2024-09-21T14:11:36.827 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:36 smithi049 ceph-mon[30080]: 5.12 scrub ok 2024-09-21T14:11:36.828 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:36 smithi049 ceph-mon[30080]: 6.18 scrub starts 2024-09-21T14:11:36.828 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:36 smithi049 ceph-mon[30080]: 6.18 scrub ok 2024-09-21T14:11:36.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:36 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:11:36.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:36 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:11:36.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:36 smithi187 ceph-mon[33032]: 5.12 scrub starts 2024-09-21T14:11:36.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:36 smithi187 ceph-mon[33032]: 5.12 scrub ok 2024-09-21T14:11:36.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:36 smithi187 ceph-mon[33032]: 6.18 scrub starts 2024-09-21T14:11:36.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:36 smithi187 ceph-mon[33032]: 6.18 scrub ok 2024-09-21T14:11:37.539 INFO:teuthology.orchestra.run.smithi049.stdout:141733921062 2024-09-21T14:11:37.539 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.4 2024-09-21T14:11:37.677 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:11:37.678 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:11:37.679 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:11:37.686 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:11:37.767 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:11:37.769 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:11:37.834 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:37 smithi049 ceph-mon[40489]: pgmap v1171: 152 pgs: 152 active+clean; 583 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 937 B/s rd, 0 op/s 2024-09-21T14:11:37.835 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:37 smithi049 ceph-mon[40489]: osdmap e579: 8 total, 8 up, 8 in 2024-09-21T14:11:37.835 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:37 smithi049 ceph-mon[30080]: pgmap v1171: 152 pgs: 152 active+clean; 583 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 937 B/s rd, 0 op/s 2024-09-21T14:11:37.835 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:37 smithi049 ceph-mon[30080]: osdmap e579: 8 total, 8 up, 8 in 2024-09-21T14:11:37.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:37 smithi187 ceph-mon[33032]: pgmap v1171: 152 pgs: 152 active+clean; 583 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 937 B/s rd, 0 op/s 2024-09-21T14:11:37.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:37 smithi187 ceph-mon[33032]: osdmap e579: 8 total, 8 up, 8 in 2024-09-21T14:11:38.476 INFO:teuthology.orchestra.run.smithi049.stdout:1030792151214 2024-09-21T14:11:38.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:38 smithi049 ceph-mon[40489]: 3.2 scrub starts 2024-09-21T14:11:38.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:38 smithi049 ceph-mon[40489]: 3.2 scrub ok 2024-09-21T14:11:38.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:38 smithi049 ceph-mon[40489]: 1.3 scrub starts 2024-09-21T14:11:38.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:38 smithi049 ceph-mon[40489]: 1.3 scrub ok 2024-09-21T14:11:38.754 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:38 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3027131329' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-21T14:11:38.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:38 smithi049 ceph-mon[30080]: 3.2 scrub starts 2024-09-21T14:11:38.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:38 smithi049 ceph-mon[30080]: 3.2 scrub ok 2024-09-21T14:11:38.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:38 smithi049 ceph-mon[30080]: 1.3 scrub starts 2024-09-21T14:11:38.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:38 smithi049 ceph-mon[30080]: 1.3 scrub ok 2024-09-21T14:11:38.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:38 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3027131329' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-21T14:11:38.798 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:11:38.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:38 smithi187 ceph-mon[33032]: 3.2 scrub starts 2024-09-21T14:11:38.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:38 smithi187 ceph-mon[33032]: 3.2 scrub ok 2024-09-21T14:11:38.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:38 smithi187 ceph-mon[33032]: 1.3 scrub starts 2024-09-21T14:11:38.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:38 smithi187 ceph-mon[33032]: 1.3 scrub ok 2024-09-21T14:11:38.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:38 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3027131329' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-21T14:11:39.582 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:39 smithi049 ceph-mon[40489]: 5.3 scrub starts 2024-09-21T14:11:39.583 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:39 smithi049 ceph-mon[40489]: 5.3 scrub ok 2024-09-21T14:11:39.583 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:39 smithi049 ceph-mon[40489]: pgmap v1173: 152 pgs: 152 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-21T14:11:39.583 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:39 smithi049 ceph-mon[40489]: 3.7 scrub starts 2024-09-21T14:11:39.583 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:39 smithi049 ceph-mon[40489]: 3.7 scrub ok 2024-09-21T14:11:39.583 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:39 smithi049 ceph-mon[40489]: 3.d scrub starts 2024-09-21T14:11:39.583 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:39 smithi049 ceph-mon[40489]: 3.d scrub ok 2024-09-21T14:11:39.583 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:39 smithi049 ceph-mon[30080]: 5.3 scrub starts 2024-09-21T14:11:39.583 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:39 smithi049 ceph-mon[30080]: 5.3 scrub ok 2024-09-21T14:11:39.583 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:39 smithi049 ceph-mon[30080]: pgmap v1173: 152 pgs: 152 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-21T14:11:39.583 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:39 smithi049 ceph-mon[30080]: 3.7 scrub starts 2024-09-21T14:11:39.584 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:39 smithi049 ceph-mon[30080]: 3.7 scrub ok 2024-09-21T14:11:39.584 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:39 smithi049 ceph-mon[30080]: 3.d scrub starts 2024-09-21T14:11:39.584 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:39 smithi049 ceph-mon[30080]: 3.d scrub ok 2024-09-21T14:11:39.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:39 smithi187 ceph-mon[33032]: 5.3 scrub starts 2024-09-21T14:11:39.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:39 smithi187 ceph-mon[33032]: 5.3 scrub ok 2024-09-21T14:11:39.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:39 smithi187 ceph-mon[33032]: pgmap v1173: 152 pgs: 152 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-21T14:11:39.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:39 smithi187 ceph-mon[33032]: 3.7 scrub starts 2024-09-21T14:11:39.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:39 smithi187 ceph-mon[33032]: 3.7 scrub ok 2024-09-21T14:11:39.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:39 smithi187 ceph-mon[33032]: 3.d scrub starts 2024-09-21T14:11:39.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:39 smithi187 ceph-mon[33032]: 3.d scrub ok 2024-09-21T14:11:39.978 INFO:teuthology.orchestra.run.smithi049.stdout:1279900254355 2024-09-21T14:11:40.231 INFO:teuthology.orchestra.run.smithi049.stdout:34359738687 2024-09-21T14:11:40.417 INFO:teuthology.orchestra.run.smithi049.stdout:201863463194 2024-09-21T14:11:40.694 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:11:40 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:11:40.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:40 smithi187 ceph-mon[33032]: 4.17 scrub starts 2024-09-21T14:11:40.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:40 smithi187 ceph-mon[33032]: 4.17 scrub ok 2024-09-21T14:11:40.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:40 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:11:40.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:40 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:11:40.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:40 smithi187 ceph-mon[33032]: osdmap e580: 8 total, 8 up, 8 in 2024-09-21T14:11:40.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:40 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/1326729315' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-21T14:11:40.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:40 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2448790805' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-21T14:11:40.695 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:40 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3146669471' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-21T14:11:40.754 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:11:40 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:11:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:11:40.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:40 smithi049 ceph-mon[30080]: 4.17 scrub starts 2024-09-21T14:11:40.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:40 smithi049 ceph-mon[30080]: 4.17 scrub ok 2024-09-21T14:11:40.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:40 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:11:40.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:40 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:11:40.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:40 smithi049 ceph-mon[30080]: osdmap e580: 8 total, 8 up, 8 in 2024-09-21T14:11:40.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:40 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/1326729315' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-21T14:11:40.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:40 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2448790805' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-21T14:11:40.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:40 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3146669471' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-21T14:11:40.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:40 smithi049 ceph-mon[40489]: 4.17 scrub starts 2024-09-21T14:11:40.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:40 smithi049 ceph-mon[40489]: 4.17 scrub ok 2024-09-21T14:11:40.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:40 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:11:40.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:40 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:11:40.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:40 smithi049 ceph-mon[40489]: osdmap e580: 8 total, 8 up, 8 in 2024-09-21T14:11:40.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:40 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/1326729315' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-21T14:11:40.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:40 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2448790805' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-21T14:11:40.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:40 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3146669471' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-21T14:11:41.215 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863463194 got 201863463194 for osd.6 2024-09-21T14:11:41.216 DEBUG:teuthology.parallel:result is None 2024-09-21T14:11:41.837 INFO:teuthology.orchestra.run.smithi049.stdout:558345748717 2024-09-21T14:11:41.838 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:41 smithi049 ceph-mon[40489]: pgmap v1175: 152 pgs: 152 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-21T14:11:41.838 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:41 smithi049 ceph-mon[40489]: 5.10 scrub starts 2024-09-21T14:11:41.838 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:41 smithi049 ceph-mon[40489]: 5.10 scrub ok 2024-09-21T14:11:41.838 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:41 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:11:41.838 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:41 smithi049 ceph-mon[30080]: pgmap v1175: 152 pgs: 152 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-21T14:11:41.838 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:41 smithi049 ceph-mon[30080]: 5.10 scrub starts 2024-09-21T14:11:41.838 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:41 smithi049 ceph-mon[30080]: 5.10 scrub ok 2024-09-21T14:11:41.839 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:41 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:11:41.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:41 smithi187 ceph-mon[33032]: pgmap v1175: 152 pgs: 152 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-21T14:11:41.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:41 smithi187 ceph-mon[33032]: 5.10 scrub starts 2024-09-21T14:11:41.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:41 smithi187 ceph-mon[33032]: 5.10 scrub ok 2024-09-21T14:11:41.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:41 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:11:42.619 INFO:teuthology.orchestra.run.smithi049.stdout:55834575162 2024-09-21T14:11:42.859 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:42 smithi049 ceph-mon[40489]: 5.16 scrub starts 2024-09-21T14:11:42.859 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:42 smithi049 ceph-mon[40489]: 5.16 scrub ok 2024-09-21T14:11:42.859 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:42 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3012838633' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-21T14:11:42.859 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:42 smithi049 ceph-mon[30080]: 5.16 scrub starts 2024-09-21T14:11:42.859 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:42 smithi049 ceph-mon[30080]: 5.16 scrub ok 2024-09-21T14:11:42.859 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:42 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3012838633' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-21T14:11:42.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:42 smithi187 ceph-mon[33032]: 5.16 scrub starts 2024-09-21T14:11:42.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:42 smithi187 ceph-mon[33032]: 5.16 scrub ok 2024-09-21T14:11:42.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:42 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3012838633' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-21T14:11:43.318 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738686 got 34359738687 for osd.0 2024-09-21T14:11:43.319 DEBUG:teuthology.parallel:result is None 2024-09-21T14:11:43.786 INFO:teuthology.orchestra.run.smithi049.stdout:1481763717242 2024-09-21T14:11:43.798 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:43 smithi049 ceph-mon[40489]: pgmap v1176: 152 pgs: 152 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T14:11:43.798 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:43 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/283280080' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-21T14:11:43.798 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:43 smithi049 ceph-mon[40489]: 3.9 scrub starts 2024-09-21T14:11:43.798 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:43 smithi049 ceph-mon[40489]: 3.9 scrub ok 2024-09-21T14:11:43.798 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:43 smithi049 ceph-mon[30080]: pgmap v1176: 152 pgs: 152 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T14:11:43.798 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:43 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/283280080' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-21T14:11:43.798 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:43 smithi049 ceph-mon[30080]: 3.9 scrub starts 2024-09-21T14:11:43.798 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:43 smithi049 ceph-mon[30080]: 3.9 scrub ok 2024-09-21T14:11:43.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:43 smithi187 ceph-mon[33032]: pgmap v1176: 152 pgs: 152 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T14:11:43.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:43 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/283280080' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-21T14:11:43.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:43 smithi187 ceph-mon[33032]: 3.9 scrub starts 2024-09-21T14:11:43.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:43 smithi187 ceph-mon[33032]: 3.9 scrub ok 2024-09-21T14:11:44.127 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834575160 got 55834575162 for osd.1 2024-09-21T14:11:44.127 DEBUG:teuthology.parallel:result is None 2024-09-21T14:11:44.594 INFO:tasks.cephadm.ceph_manager.ceph:need seq 1030792151213 got 1030792151214 for osd.3 2024-09-21T14:11:44.594 DEBUG:teuthology.parallel:result is None 2024-09-21T14:11:44.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:44 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2472256848' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-21T14:11:44.979 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:44 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2472256848' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-21T14:11:44.979 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:44 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2472256848' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-21T14:11:45.013 INFO:tasks.cephadm.ceph_manager.ceph:need seq 1279900254353 got 1279900254355 for osd.7 2024-09-21T14:11:45.013 DEBUG:teuthology.parallel:result is None 2024-09-21T14:11:45.459 INFO:tasks.cephadm.ceph_manager.ceph:need seq 558345748715 got 558345748717 for osd.2 2024-09-21T14:11:45.459 DEBUG:teuthology.parallel:result is None 2024-09-21T14:11:45.674 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:11:45.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:45 smithi049 ceph-mon[30080]: pgmap v1177: 152 pgs: 152 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T14:11:45.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:45 smithi049 ceph-mon[30080]: 3.1c scrub starts 2024-09-21T14:11:45.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:45 smithi049 ceph-mon[30080]: 3.1c scrub ok 2024-09-21T14:11:45.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:45 smithi049 ceph-mon[30080]: 5.13 scrub starts 2024-09-21T14:11:45.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:45 smithi049 ceph-mon[30080]: 5.13 scrub ok 2024-09-21T14:11:45.754 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:45 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:11:45.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:45 smithi049 ceph-mon[30080]: 3.e scrub starts 2024-09-21T14:11:45.755 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:45 smithi049 ceph-mon[30080]: 3.e scrub ok 2024-09-21T14:11:45.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:45 smithi049 ceph-mon[40489]: pgmap v1177: 152 pgs: 152 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T14:11:45.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:45 smithi049 ceph-mon[40489]: 3.1c scrub starts 2024-09-21T14:11:45.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:45 smithi049 ceph-mon[40489]: 3.1c scrub ok 2024-09-21T14:11:45.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:45 smithi049 ceph-mon[40489]: 5.13 scrub starts 2024-09-21T14:11:45.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:45 smithi049 ceph-mon[40489]: 5.13 scrub ok 2024-09-21T14:11:45.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:45 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:11:45.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:45 smithi049 ceph-mon[40489]: 3.e scrub starts 2024-09-21T14:11:45.755 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:45 smithi049 ceph-mon[40489]: 3.e scrub ok 2024-09-21T14:11:45.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:45 smithi187 ceph-mon[33032]: pgmap v1177: 152 pgs: 152 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T14:11:45.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:45 smithi187 ceph-mon[33032]: 3.1c scrub starts 2024-09-21T14:11:45.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:45 smithi187 ceph-mon[33032]: 3.1c scrub ok 2024-09-21T14:11:45.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:45 smithi187 ceph-mon[33032]: 5.13 scrub starts 2024-09-21T14:11:45.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:45 smithi187 ceph-mon[33032]: 5.13 scrub ok 2024-09-21T14:11:45.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:45 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T14:11:45.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:45 smithi187 ceph-mon[33032]: 3.e scrub starts 2024-09-21T14:11:45.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:45 smithi187 ceph-mon[33032]: 3.e scrub ok 2024-09-21T14:11:46.099 INFO:tasks.cephadm.ceph_manager.ceph:need seq 1481763717240 got 1481763717242 for osd.5 2024-09-21T14:11:46.100 DEBUG:teuthology.parallel:result is None 2024-09-21T14:11:46.642 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:46 smithi049 ceph-mon[40489]: 4.b scrub starts 2024-09-21T14:11:46.642 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:46 smithi049 ceph-mon[40489]: 4.b scrub ok 2024-09-21T14:11:46.642 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:46 smithi049 ceph-mon[40489]: 1.2 scrub starts 2024-09-21T14:11:46.642 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:46 smithi049 ceph-mon[40489]: 1.2 scrub ok 2024-09-21T14:11:46.642 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:46 smithi049 ceph-mon[30080]: 4.b scrub starts 2024-09-21T14:11:46.643 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:46 smithi049 ceph-mon[30080]: 4.b scrub ok 2024-09-21T14:11:46.643 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:46 smithi049 ceph-mon[30080]: 1.2 scrub starts 2024-09-21T14:11:46.643 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:46 smithi049 ceph-mon[30080]: 1.2 scrub ok 2024-09-21T14:11:46.823 INFO:teuthology.orchestra.run.smithi049.stdout:141733921065 2024-09-21T14:11:46.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:46 smithi187 ceph-mon[33032]: 4.b scrub starts 2024-09-21T14:11:46.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:46 smithi187 ceph-mon[33032]: 4.b scrub ok 2024-09-21T14:11:46.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:46 smithi187 ceph-mon[33032]: 1.2 scrub starts 2024-09-21T14:11:46.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:46 smithi187 ceph-mon[33032]: 1.2 scrub ok 2024-09-21T14:11:47.626 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733921062 got 141733921065 for osd.4 2024-09-21T14:11:47.626 DEBUG:teuthology.parallel:result is None 2024-09-21T14:11:47.626 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-21T14:11:47.626 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T14:11:47.886 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:11:47.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:47 smithi187 ceph-mon[33032]: pgmap v1178: 152 pgs: 152 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 527 B/s rd, 0 op/s 2024-09-21T14:11:47.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:47 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/2263241924' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-21T14:11:47.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:47 smithi187 ceph-mon[33032]: 6.9 scrub starts 2024-09-21T14:11:47.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:47 smithi187 ceph-mon[33032]: 6.9 scrub ok 2024-09-21T14:11:48.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:47 smithi049 ceph-mon[30080]: pgmap v1178: 152 pgs: 152 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 527 B/s rd, 0 op/s 2024-09-21T14:11:48.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:47 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/2263241924' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-21T14:11:48.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:47 smithi049 ceph-mon[30080]: 6.9 scrub starts 2024-09-21T14:11:48.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:47 smithi049 ceph-mon[30080]: 6.9 scrub ok 2024-09-21T14:11:48.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:47 smithi049 ceph-mon[40489]: pgmap v1178: 152 pgs: 152 active+clean; 583 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 527 B/s rd, 0 op/s 2024-09-21T14:11:48.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:47 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/2263241924' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-21T14:11:48.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:47 smithi049 ceph-mon[40489]: 6.9 scrub starts 2024-09-21T14:11:48.005 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:47 smithi049 ceph-mon[40489]: 6.9 scrub ok 2024-09-21T14:11:48.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:48 smithi187 ceph-mon[33032]: 4.1 scrub starts 2024-09-21T14:11:48.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:48 smithi187 ceph-mon[33032]: 4.1 scrub ok 2024-09-21T14:11:48.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:48 smithi187 ceph-mon[33032]: 6.12 scrub starts 2024-09-21T14:11:48.694 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:48 smithi187 ceph-mon[33032]: 6.12 scrub ok 2024-09-21T14:11:48.698 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T14:11:48.700 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-09-21T14:11:49.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:48 smithi049 ceph-mon[30080]: 4.1 scrub starts 2024-09-21T14:11:49.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:48 smithi049 ceph-mon[30080]: 4.1 scrub ok 2024-09-21T14:11:49.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:48 smithi049 ceph-mon[30080]: 6.12 scrub starts 2024-09-21T14:11:49.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:48 smithi049 ceph-mon[30080]: 6.12 scrub ok 2024-09-21T14:11:49.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:48 smithi049 ceph-mon[40489]: 4.1 scrub starts 2024-09-21T14:11:49.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:48 smithi049 ceph-mon[40489]: 4.1 scrub ok 2024-09-21T14:11:49.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:48 smithi049 ceph-mon[40489]: 6.12 scrub starts 2024-09-21T14:11:49.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:48 smithi049 ceph-mon[40489]: 6.12 scrub ok 2024-09-21T14:11:49.538 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":1179,"stamp":"2024-09-21T14:11:48.215756+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":2073,"num_read_kb":1825,"num_write":1008,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":365,"num_bytes_recovered":861826,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2764,"ondisk_log_size":2764,"up":456,"acting":456,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":459,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6005604,"kb_used_data":243356,"kb_used_omap":23,"kb_used_meta":5762088,"kb_avail":743955612,"statfs":{"total":767960285184,"available":761810546688,"internally_reserved":0,"allocated":249196544,"data_stored":244084777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24148,"internal_metadata":5900378540},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002733"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":941,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741293+0000","last_change":"2024-09-21T14:10:53.033673+0000","last_active":"2024-09-21T14:11:39.741293+0000","last_peered":"2024-09-21T14:11:39.741293+0000","last_clean":"2024-09-21T14:11:39.741293+0000","last_became_active":"2024-09-21T14:10:49.340676+0000","last_became_peered":"2024-09-21T14:10:49.340676+0000","last_unstale":"2024-09-21T14:11:39.741293+0000","last_undegraded":"2024-09-21T14:11:39.741293+0000","last_fullsized":"2024-09-21T14:11:39.741293+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:10:53.033628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:10:53.033628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:15.576784+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"516'21","reported_seq":1188,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.740798+0000","last_change":"2024-09-21T14:11:26.806163+0000","last_active":"2024-09-21T14:11:39.740798+0000","last_peered":"2024-09-21T14:11:39.740798+0000","last_clean":"2024-09-21T14:11:39.740798+0000","last_became_active":"2024-09-21T13:58:17.771220+0000","last_became_peered":"2024-09-21T13:58:17.771220+0000","last_unstale":"2024-09-21T14:11:39.740798+0000","last_undegraded":"2024-09-21T14:11:39.740798+0000","last_fullsized":"2024-09-21T14:11:39.740798+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'21","last_scrub_stamp":"2024-09-21T14:11:26.806087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:11:26.806087+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:37.946810+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1120,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.526374+0000","last_change":"2024-09-21T14:11:15.566060+0000","last_active":"2024-09-21T14:11:39.526374+0000","last_peered":"2024-09-21T14:11:39.526374+0000","last_clean":"2024-09-21T14:11:39.526374+0000","last_became_active":"2024-09-21T14:01:41.202168+0000","last_became_peered":"2024-09-21T14:01:41.202168+0000","last_unstale":"2024-09-21T14:11:39.526374+0000","last_undegraded":"2024-09-21T14:11:39.526374+0000","last_fullsized":"2024-09-21T14:11:39.526374+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:15.566000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:11:15.566000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:27.460647+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1096,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741338+0000","last_change":"2024-09-21T14:10:12.684731+0000","last_active":"2024-09-21T14:11:39.741338+0000","last_peered":"2024-09-21T14:11:39.741338+0000","last_clean":"2024-09-21T14:11:39.741338+0000","last_became_active":"2024-09-21T14:01:41.203275+0000","last_became_peered":"2024-09-21T14:01:41.203275+0000","last_unstale":"2024-09-21T14:11:39.741338+0000","last_undegraded":"2024-09-21T14:11:39.741338+0000","last_fullsized":"2024-09-21T14:11:39.741338+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:10:12.684656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:10:12.684656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":38,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.531441+0000","last_change":"2024-09-21T14:11:28.412019+0000","last_active":"2024-09-21T14:11:39.531441+0000","last_peered":"2024-09-21T14:11:39.531441+0000","last_clean":"2024-09-21T14:11:39.531441+0000","last_became_active":"2024-09-21T14:11:13.451144+0000","last_became_peered":"2024-09-21T14:11:13.451144+0000","last_unstale":"2024-09-21T14:11:39.531441+0000","last_undegraded":"2024-09-21T14:11:39.531441+0000","last_fullsized":"2024-09-21T14:11:39.531441+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:28.411959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:11:28.411959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:49.664370+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":30,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.532132+0000","last_change":"2024-09-21T14:11:13.456954+0000","last_active":"2024-09-21T14:11:39.532132+0000","last_peered":"2024-09-21T14:11:39.532132+0000","last_clean":"2024-09-21T14:11:39.532132+0000","last_became_active":"2024-09-21T14:11:13.456666+0000","last_became_peered":"2024-09-21T14:11:13.456666+0000","last_unstale":"2024-09-21T14:11:39.532132+0000","last_undegraded":"2024-09-21T14:11:39.532132+0000","last_fullsized":"2024-09-21T14:11:39.532132+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:10:49.045362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:10:49.045362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:14.009955+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"516'39","reported_seq":46,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.531343+0000","last_change":"2024-09-21T14:11:13.489169+0000","last_active":"2024-09-21T14:11:39.531343+0000","last_peered":"2024-09-21T14:11:39.531343+0000","last_clean":"2024-09-21T14:11:39.531343+0000","last_became_active":"2024-09-21T14:11:13.462350+0000","last_became_peered":"2024-09-21T14:11:13.462350+0000","last_unstale":"2024-09-21T14:11:39.531343+0000","last_undegraded":"2024-09-21T14:11:39.531343+0000","last_fullsized":"2024-09-21T14:11:39.531343+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'39","last_scrub_stamp":"2024-09-21T14:10:58.615240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:10:58.615240+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:18.908576+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":59,"num_read_kb":50,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":167,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.740565+0000","last_change":"2024-09-21T14:11:30.836654+0000","last_active":"2024-09-21T14:11:39.740565+0000","last_peered":"2024-09-21T14:11:39.740565+0000","last_clean":"2024-09-21T14:11:39.740565+0000","last_became_active":"2024-09-21T14:09:00.253451+0000","last_became_peered":"2024-09-21T14:09:00.253451+0000","last_unstale":"2024-09-21T14:11:39.740565+0000","last_undegraded":"2024-09-21T14:11:39.740565+0000","last_fullsized":"2024-09-21T14:11:39.740565+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:30.836593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:11:30.836593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:33.460589+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":1077,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.744025+0000","last_change":"2024-09-21T14:11:12.546029+0000","last_active":"2024-09-21T14:11:39.744025+0000","last_peered":"2024-09-21T14:11:39.744025+0000","last_clean":"2024-09-21T14:11:39.744025+0000","last_became_active":"2024-09-21T14:01:41.194517+0000","last_became_peered":"2024-09-21T14:01:41.194517+0000","last_unstale":"2024-09-21T14:11:39.744025+0000","last_undegraded":"2024-09-21T14:11:39.744025+0000","last_fullsized":"2024-09-21T14:11:39.744025+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:12.545976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:11:12.545976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:14.332461+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"516'21","reported_seq":178,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741185+0000","last_change":"2024-09-21T14:10:13.694713+0000","last_active":"2024-09-21T14:11:39.741185+0000","last_peered":"2024-09-21T14:11:39.741185+0000","last_clean":"2024-09-21T14:11:39.741185+0000","last_became_active":"2024-09-21T14:09:00.251527+0000","last_became_peered":"2024-09-21T14:09:00.251527+0000","last_unstale":"2024-09-21T14:11:39.741185+0000","last_undegraded":"2024-09-21T14:11:39.741185+0000","last_fullsized":"2024-09-21T14:11:39.741185+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'21","last_scrub_stamp":"2024-09-21T14:10:13.694626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:10:13.694626+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1026,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741483+0000","last_change":"2024-09-21T14:10:47.486485+0000","last_active":"2024-09-21T14:11:39.741483+0000","last_peered":"2024-09-21T14:11:39.741483+0000","last_clean":"2024-09-21T14:11:39.741483+0000","last_became_active":"2024-09-21T14:01:41.188248+0000","last_became_peered":"2024-09-21T14:01:41.188248+0000","last_unstale":"2024-09-21T14:11:39.741483+0000","last_undegraded":"2024-09-21T14:11:39.741483+0000","last_fullsized":"2024-09-21T14:11:39.741483+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:10:47.486425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:10:47.486425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:11:49.215493+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":987,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.740370+0000","last_change":"2024-09-21T14:11:36.078219+0000","last_active":"2024-09-21T14:11:39.740370+0000","last_peered":"2024-09-21T14:11:39.740370+0000","last_clean":"2024-09-21T14:11:39.740370+0000","last_became_active":"2024-09-21T14:10:49.341656+0000","last_became_peered":"2024-09-21T14:10:49.341656+0000","last_unstale":"2024-09-21T14:11:39.740370+0000","last_undegraded":"2024-09-21T14:11:39.740370+0000","last_fullsized":"2024-09-21T14:11:39.740370+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:36.078168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:11:36.078168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:13:05.720118+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":1061,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:44.352404+0000","last_change":"2024-09-21T14:11:44.352404+0000","last_active":"2024-09-21T14:11:44.352404+0000","last_peered":"2024-09-21T14:11:44.352404+0000","last_clean":"2024-09-21T14:11:44.352404+0000","last_became_active":"2024-09-21T14:09:31.257075+0000","last_became_peered":"2024-09-21T14:09:31.257075+0000","last_unstale":"2024-09-21T14:11:44.352404+0000","last_undegraded":"2024-09-21T14:11:44.352404+0000","last_fullsized":"2024-09-21T14:11:44.352404+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:44.352339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:11:44.352339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:49.432595+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"516'9","reported_seq":168,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741580+0000","last_change":"2024-09-21T14:09:41.805340+0000","last_active":"2024-09-21T14:11:39.741580+0000","last_peered":"2024-09-21T14:11:39.741580+0000","last_clean":"2024-09-21T14:11:39.741580+0000","last_became_active":"2024-09-21T14:09:00.253147+0000","last_became_peered":"2024-09-21T14:09:00.253147+0000","last_unstale":"2024-09-21T14:11:39.741580+0000","last_undegraded":"2024-09-21T14:11:39.741580+0000","last_fullsized":"2024-09-21T14:11:39.741580+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'9","last_scrub_stamp":"2024-09-21T14:09:41.805273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:09:41.805273+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":291,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":872,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741111+0000","last_change":"2024-09-21T14:11:22.288215+0000","last_active":"2024-09-21T14:11:39.741111+0000","last_peered":"2024-09-21T14:11:39.741111+0000","last_clean":"2024-09-21T14:11:39.741111+0000","last_became_active":"2024-09-21T14:11:19.469736+0000","last_became_peered":"2024-09-21T14:11:19.469736+0000","last_unstale":"2024-09-21T14:11:39.741111+0000","last_undegraded":"2024-09-21T14:11:39.741111+0000","last_fullsized":"2024-09-21T14:11:39.741111+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:22.288153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:11:22.288153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:27.949208+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1094,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741079+0000","last_change":"2024-09-21T14:11:32.467937+0000","last_active":"2024-09-21T14:11:39.741079+0000","last_peered":"2024-09-21T14:11:39.741079+0000","last_clean":"2024-09-21T14:11:39.741079+0000","last_became_active":"2024-09-21T14:11:13.449167+0000","last_became_peered":"2024-09-21T14:11:13.449167+0000","last_unstale":"2024-09-21T14:11:39.741079+0000","last_undegraded":"2024-09-21T14:11:39.741079+0000","last_fullsized":"2024-09-21T14:11:39.741079+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:32.467874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:11:32.467874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:49.176031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1139,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.743455+0000","last_change":"2024-09-21T14:11:06.676347+0000","last_active":"2024-09-21T14:11:39.743455+0000","last_peered":"2024-09-21T14:11:39.743455+0000","last_clean":"2024-09-21T14:11:39.743455+0000","last_became_active":"2024-09-21T14:01:41.200360+0000","last_became_peered":"2024-09-21T14:01:41.200360+0000","last_unstale":"2024-09-21T14:11:39.743455+0000","last_undegraded":"2024-09-21T14:11:39.743455+0000","last_fullsized":"2024-09-21T14:11:39.743455+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:06.676300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:11:06.676300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:28.826088+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":993,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.740729+0000","last_change":"2024-09-21T14:11:19.043091+0000","last_active":"2024-09-21T14:11:39.740729+0000","last_peered":"2024-09-21T14:11:39.740729+0000","last_clean":"2024-09-21T14:11:39.740729+0000","last_became_active":"2024-09-21T14:10:49.343587+0000","last_became_peered":"2024-09-21T14:10:49.343587+0000","last_unstale":"2024-09-21T14:11:39.740729+0000","last_undegraded":"2024-09-21T14:11:39.740729+0000","last_fullsized":"2024-09-21T14:11:39.740729+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:19.043035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:11:19.043035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:35.297874+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"516'39","reported_seq":1246,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.528582+0000","last_change":"2024-09-21T14:11:13.487805+0000","last_active":"2024-09-21T14:11:39.528582+0000","last_peered":"2024-09-21T14:11:39.528582+0000","last_clean":"2024-09-21T14:11:39.528582+0000","last_became_active":"2024-09-21T14:11:13.451125+0000","last_became_peered":"2024-09-21T14:11:13.451125+0000","last_unstale":"2024-09-21T14:11:39.528582+0000","last_undegraded":"2024-09-21T14:11:39.528582+0000","last_fullsized":"2024-09-21T14:11:39.528582+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'39","last_scrub_stamp":"2024-09-21T14:11:08.500104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:11:08.500104+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:36.293401+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1128,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.527443+0000","last_change":"2024-09-21T14:11:24.500304+0000","last_active":"2024-09-21T14:11:39.527443+0000","last_peered":"2024-09-21T14:11:39.527443+0000","last_clean":"2024-09-21T14:11:39.527443+0000","last_became_active":"2024-09-21T14:09:00.254258+0000","last_became_peered":"2024-09-21T14:09:00.254258+0000","last_unstale":"2024-09-21T14:11:39.527443+0000","last_undegraded":"2024-09-21T14:11:39.527443+0000","last_fullsized":"2024-09-21T14:11:39.527443+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:24.500257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:11:24.500257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:44.518044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":619,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741279+0000","last_change":"2024-09-21T14:11:11.985555+0000","last_active":"2024-09-21T14:11:39.741279+0000","last_peered":"2024-09-21T14:11:39.741279+0000","last_clean":"2024-09-21T14:11:39.741279+0000","last_became_active":"2024-09-21T14:10:49.341143+0000","last_became_peered":"2024-09-21T14:10:49.341143+0000","last_unstale":"2024-09-21T14:11:39.741279+0000","last_undegraded":"2024-09-21T14:11:39.741279+0000","last_fullsized":"2024-09-21T14:11:39.741279+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:11.985506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:11:11.985506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:31.972909+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":1116,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741717+0000","last_change":"2024-09-21T14:10:56.719940+0000","last_active":"2024-09-21T14:11:39.741717+0000","last_peered":"2024-09-21T14:11:39.741717+0000","last_clean":"2024-09-21T14:11:39.741717+0000","last_became_active":"2024-09-21T13:52:04.316016+0000","last_became_peered":"2024-09-21T13:52:04.316016+0000","last_unstale":"2024-09-21T14:11:39.741717+0000","last_undegraded":"2024-09-21T14:11:39.741717+0000","last_fullsized":"2024-09-21T14:11:39.741717+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:10:56.719874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:10:56.719874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:18.203148+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"516'24","reported_seq":42,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.531855+0000","last_change":"2024-09-21T14:11:13.478263+0000","last_active":"2024-09-21T14:11:39.531855+0000","last_peered":"2024-09-21T14:11:39.531855+0000","last_clean":"2024-09-21T14:11:39.531855+0000","last_became_active":"2024-09-21T14:11:13.450207+0000","last_became_peered":"2024-09-21T14:11:13.450207+0000","last_unstale":"2024-09-21T14:11:39.531855+0000","last_undegraded":"2024-09-21T14:11:39.531855+0000","last_fullsized":"2024-09-21T14:11:39.531855+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'24","last_scrub_stamp":"2024-09-21T14:10:56.585018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:10:56.585018+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:11:58.674191+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1103,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.742213+0000","last_change":"2024-09-21T14:10:08.586191+0000","last_active":"2024-09-21T14:11:39.742213+0000","last_peered":"2024-09-21T14:11:39.742213+0000","last_clean":"2024-09-21T14:11:39.742213+0000","last_became_active":"2024-09-21T13:57:12.681897+0000","last_became_peered":"2024-09-21T13:57:12.681897+0000","last_unstale":"2024-09-21T14:11:39.742213+0000","last_undegraded":"2024-09-21T14:11:39.742213+0000","last_fullsized":"2024-09-21T14:11:39.742213+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:10:08.586123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:10:08.586123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":856,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741918+0000","last_change":"2024-09-21T14:11:19.463135+0000","last_active":"2024-09-21T14:11:39.741918+0000","last_peered":"2024-09-21T14:11:39.741918+0000","last_clean":"2024-09-21T14:11:39.741918+0000","last_became_active":"2024-09-21T14:11:19.462783+0000","last_became_peered":"2024-09-21T14:11:19.462783+0000","last_unstale":"2024-09-21T14:11:39.741918+0000","last_undegraded":"2024-09-21T14:11:39.741918+0000","last_fullsized":"2024-09-21T14:11:39.741918+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T14:10:56.320207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:10:56.320207+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:26.039111+0000","scrub_duration":225,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":1140,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.526470+0000","last_change":"2024-09-21T14:11:06.401536+0000","last_active":"2024-09-21T14:11:39.526470+0000","last_peered":"2024-09-21T14:11:39.526470+0000","last_clean":"2024-09-21T14:11:39.526470+0000","last_became_active":"2024-09-21T14:09:00.257421+0000","last_became_peered":"2024-09-21T14:09:00.257421+0000","last_unstale":"2024-09-21T14:11:39.526470+0000","last_undegraded":"2024-09-21T14:11:39.526470+0000","last_fullsized":"2024-09-21T14:11:39.526470+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T14:11:06.401472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:11:06.401472+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:36.277047+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"516'30","reported_seq":454,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741319+0000","last_change":"2024-09-21T14:11:13.471749+0000","last_active":"2024-09-21T14:11:39.741319+0000","last_peered":"2024-09-21T14:11:39.741319+0000","last_clean":"2024-09-21T14:11:39.741319+0000","last_became_active":"2024-09-21T14:11:13.454129+0000","last_became_peered":"2024-09-21T14:11:13.454129+0000","last_unstale":"2024-09-21T14:11:39.741319+0000","last_undegraded":"2024-09-21T14:11:39.741319+0000","last_fullsized":"2024-09-21T14:11:39.741319+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'30","last_scrub_stamp":"2024-09-21T14:11:02.069275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:11:02.069275+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:13.248489+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"513'14","reported_seq":864,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.743991+0000","last_change":"2024-09-21T14:10:50.069522+0000","last_active":"2024-09-21T14:11:39.743991+0000","last_peered":"2024-09-21T14:11:39.743991+0000","last_clean":"2024-09-21T14:11:39.743991+0000","last_became_active":"2024-09-21T14:09:00.258760+0000","last_became_peered":"2024-09-21T14:09:00.258760+0000","last_unstale":"2024-09-21T14:11:39.743991+0000","last_undegraded":"2024-09-21T14:11:39.743991+0000","last_fullsized":"2024-09-21T14:11:39.743991+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"513'14","last_scrub_stamp":"2024-09-21T14:10:50.069455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:10:50.069455+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:17.425683+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"75'79","reported_seq":523,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741743+0000","last_change":"2024-09-21T14:09:32.389941+0000","last_active":"2024-09-21T14:11:39.741743+0000","last_peered":"2024-09-21T14:11:39.741743+0000","last_clean":"2024-09-21T14:11:39.741743+0000","last_became_active":"2024-09-21T14:01:41.246888+0000","last_became_peered":"2024-09-21T14:01:41.246888+0000","last_unstale":"2024-09-21T14:11:39.741743+0000","last_undegraded":"2024-09-21T14:11:39.741743+0000","last_fullsized":"2024-09-21T14:11:39.741743+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:09:32.389738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:09:32.389738+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"516'39","reported_seq":1191,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.527684+0000","last_change":"2024-09-21T14:11:09.466808+0000","last_active":"2024-09-21T14:11:39.527684+0000","last_peered":"2024-09-21T14:11:39.527684+0000","last_clean":"2024-09-21T14:11:39.527684+0000","last_became_active":"2024-09-21T13:57:12.695609+0000","last_became_peered":"2024-09-21T13:57:12.695609+0000","last_unstale":"2024-09-21T14:11:39.527684+0000","last_undegraded":"2024-09-21T14:11:39.527684+0000","last_fullsized":"2024-09-21T14:11:39.527684+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'39","last_scrub_stamp":"2024-09-21T14:11:09.466746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:11:09.466746+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:31.610870+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":30,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.531230+0000","last_change":"2024-09-21T14:11:13.442365+0000","last_active":"2024-09-21T14:11:39.531230+0000","last_peered":"2024-09-21T14:11:39.531230+0000","last_clean":"2024-09-21T14:11:39.531230+0000","last_became_active":"2024-09-21T14:11:13.441990+0000","last_became_peered":"2024-09-21T14:11:13.441990+0000","last_unstale":"2024-09-21T14:11:39.531230+0000","last_undegraded":"2024-09-21T14:11:39.531230+0000","last_fullsized":"2024-09-21T14:11:39.531230+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:10:52.261724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:10:52.261724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:11:52.466748+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"513'12","reported_seq":196,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.743405+0000","last_change":"2024-09-21T14:09:35.393938+0000","last_active":"2024-09-21T14:11:39.743405+0000","last_peered":"2024-09-21T14:11:39.743405+0000","last_clean":"2024-09-21T14:11:39.743405+0000","last_became_active":"2024-09-21T14:09:00.253456+0000","last_became_peered":"2024-09-21T14:09:00.253456+0000","last_unstale":"2024-09-21T14:11:39.743405+0000","last_undegraded":"2024-09-21T14:11:39.743405+0000","last_fullsized":"2024-09-21T14:11:39.743405+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"513'12","last_scrub_stamp":"2024-09-21T14:09:35.393859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:09:35.393859+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":989,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741887+0000","last_change":"2024-09-21T14:11:28.536633+0000","last_active":"2024-09-21T14:11:39.741887+0000","last_peered":"2024-09-21T14:11:39.741887+0000","last_clean":"2024-09-21T14:11:39.741887+0000","last_became_active":"2024-09-21T14:09:20.049571+0000","last_became_peered":"2024-09-21T14:09:20.049571+0000","last_unstale":"2024-09-21T14:11:39.741887+0000","last_undegraded":"2024-09-21T14:11:39.741887+0000","last_fullsized":"2024-09-21T14:11:39.741887+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:28.536572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:11:28.536572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:28.914913+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"75'79","reported_seq":585,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.743609+0000","last_change":"2024-09-21T14:10:47.671255+0000","last_active":"2024-09-21T14:11:39.743609+0000","last_peered":"2024-09-21T14:11:39.743609+0000","last_clean":"2024-09-21T14:11:39.743609+0000","last_became_active":"2024-09-21T14:05:39.296209+0000","last_became_peered":"2024-09-21T14:05:39.296209+0000","last_unstale":"2024-09-21T14:11:39.743609+0000","last_undegraded":"2024-09-21T14:11:39.743609+0000","last_fullsized":"2024-09-21T14:11:39.743609+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:10:47.671185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:10:47.671185+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:11:54.870523+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"516'42","reported_seq":56,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.531301+0000","last_change":"2024-09-21T14:11:20.355764+0000","last_active":"2024-09-21T14:11:39.531301+0000","last_peered":"2024-09-21T14:11:39.531301+0000","last_clean":"2024-09-21T14:11:39.531301+0000","last_became_active":"2024-09-21T14:11:13.460234+0000","last_became_peered":"2024-09-21T14:11:13.460234+0000","last_unstale":"2024-09-21T14:11:39.531301+0000","last_undegraded":"2024-09-21T14:11:39.531301+0000","last_fullsized":"2024-09-21T14:11:39.531301+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'42","last_scrub_stamp":"2024-09-21T14:11:20.355687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:11:20.355687+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:39.079841+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":1018,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.532305+0000","last_change":"2024-09-21T14:11:38.291252+0000","last_active":"2024-09-21T14:11:39.532305+0000","last_peered":"2024-09-21T14:11:39.532305+0000","last_clean":"2024-09-21T14:11:39.532305+0000","last_became_active":"2024-09-21T13:59:32.211004+0000","last_became_peered":"2024-09-21T13:59:32.211004+0000","last_unstale":"2024-09-21T14:11:39.532305+0000","last_undegraded":"2024-09-21T14:11:39.532305+0000","last_fullsized":"2024-09-21T14:11:39.532305+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:38.291200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:11:38.291200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:13:07.584626+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1133,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.740941+0000","last_change":"2024-09-21T14:11:13.450533+0000","last_active":"2024-09-21T14:11:39.740941+0000","last_peered":"2024-09-21T14:11:39.740941+0000","last_clean":"2024-09-21T14:11:39.740941+0000","last_became_active":"2024-09-21T14:11:13.450153+0000","last_became_peered":"2024-09-21T14:11:13.450153+0000","last_unstale":"2024-09-21T14:11:39.740941+0000","last_undegraded":"2024-09-21T14:11:39.740941+0000","last_fullsized":"2024-09-21T14:11:39.740941+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:09:43.482393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:09:43.482393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":166,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741087+0000","last_change":"2024-09-21T14:11:13.448191+0000","last_active":"2024-09-21T14:11:39.741087+0000","last_peered":"2024-09-21T14:11:39.741087+0000","last_clean":"2024-09-21T14:11:39.741087+0000","last_became_active":"2024-09-21T14:11:13.447822+0000","last_became_peered":"2024-09-21T14:11:13.447822+0000","last_unstale":"2024-09-21T14:11:39.741087+0000","last_undegraded":"2024-09-21T14:11:39.741087+0000","last_fullsized":"2024-09-21T14:11:39.741087+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:09:34.433449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:09:34.433449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"75'79","reported_seq":30,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.531773+0000","last_change":"2024-09-21T14:11:13.453415+0000","last_active":"2024-09-21T14:11:39.531773+0000","last_peered":"2024-09-21T14:11:39.531773+0000","last_clean":"2024-09-21T14:11:39.531773+0000","last_became_active":"2024-09-21T14:11:13.453029+0000","last_became_peered":"2024-09-21T14:11:13.453029+0000","last_unstale":"2024-09-21T14:11:39.531773+0000","last_undegraded":"2024-09-21T14:11:39.531773+0000","last_fullsized":"2024-09-21T14:11:39.531773+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:11:04.428090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:11:04.428090+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:31.425271+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"516'42","reported_seq":1230,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741037+0000","last_change":"2024-09-21T14:08:13.140559+0000","last_active":"2024-09-21T14:11:39.741037+0000","last_peered":"2024-09-21T14:11:39.741037+0000","last_clean":"2024-09-21T14:11:39.741037+0000","last_became_active":"2024-09-21T14:01:41.200859+0000","last_became_peered":"2024-09-21T14:01:41.200859+0000","last_unstale":"2024-09-21T14:11:39.741037+0000","last_undegraded":"2024-09-21T14:11:39.741037+0000","last_fullsized":"2024-09-21T14:11:39.741037+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'28","last_scrub_stamp":"2024-09-21T14:08:13.140470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:08:13.140470+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":118,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741144+0000","last_change":"2024-09-21T14:11:14.950193+0000","last_active":"2024-09-21T14:11:39.741144+0000","last_peered":"2024-09-21T14:11:39.741144+0000","last_clean":"2024-09-21T14:11:39.741144+0000","last_became_active":"2024-09-21T14:10:49.346769+0000","last_became_peered":"2024-09-21T14:10:49.346769+0000","last_unstale":"2024-09-21T14:11:39.741144+0000","last_undegraded":"2024-09-21T14:11:39.741144+0000","last_fullsized":"2024-09-21T14:11:39.741144+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T14:11:14.950130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:11:14.950130+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:36.810130+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"571'13","reported_seq":41,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.531244+0000","last_change":"2024-09-21T14:11:13.466255+0000","last_active":"2024-09-21T14:11:39.531244+0000","last_peered":"2024-09-21T14:11:39.531244+0000","last_clean":"2024-09-21T14:11:39.531244+0000","last_became_active":"2024-09-21T14:11:13.450354+0000","last_became_peered":"2024-09-21T14:11:13.450354+0000","last_unstale":"2024-09-21T14:11:39.531244+0000","last_undegraded":"2024-09-21T14:11:39.531244+0000","last_fullsized":"2024-09-21T14:11:39.531244+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"519'12","last_scrub_stamp":"2024-09-21T14:10:59.305542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:10:59.305542+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:04.230445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":890,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741736+0000","last_change":"2024-09-21T14:11:19.465850+0000","last_active":"2024-09-21T14:11:39.741736+0000","last_peered":"2024-09-21T14:11:39.741736+0000","last_clean":"2024-09-21T14:11:39.741736+0000","last_became_active":"2024-09-21T14:11:19.465511+0000","last_became_peered":"2024-09-21T14:11:19.465511+0000","last_unstale":"2024-09-21T14:11:39.741736+0000","last_undegraded":"2024-09-21T14:11:39.741736+0000","last_fullsized":"2024-09-21T14:11:39.741736+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:01.098264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:11:01.098264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:25.240453+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"75'79","reported_seq":1172,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.743404+0000","last_change":"2024-09-21T14:11:13.449007+0000","last_active":"2024-09-21T14:11:39.743404+0000","last_peered":"2024-09-21T14:11:39.743404+0000","last_clean":"2024-09-21T14:11:39.743404+0000","last_became_active":"2024-09-21T14:11:13.448515+0000","last_became_peered":"2024-09-21T14:11:13.448515+0000","last_unstale":"2024-09-21T14:11:39.743404+0000","last_undegraded":"2024-09-21T14:11:39.743404+0000","last_fullsized":"2024-09-21T14:11:39.743404+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:10:42.074254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:10:42.074254+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:08.096912+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"516'30","reported_seq":456,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.526823+0000","last_change":"2024-09-21T14:11:14.666842+0000","last_active":"2024-09-21T14:11:39.526823+0000","last_peered":"2024-09-21T14:11:39.526823+0000","last_clean":"2024-09-21T14:11:39.526823+0000","last_became_active":"2024-09-21T14:03:29.158218+0000","last_became_peered":"2024-09-21T14:03:29.158218+0000","last_unstale":"2024-09-21T14:11:39.526823+0000","last_undegraded":"2024-09-21T14:11:39.526823+0000","last_fullsized":"2024-09-21T14:11:39.526823+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'30","last_scrub_stamp":"2024-09-21T14:11:14.666787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:11:14.666787+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:39.337317+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"61'1","reported_seq":1067,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.742005+0000","last_change":"2024-09-21T14:11:13.465249+0000","last_active":"2024-09-21T14:11:39.742005+0000","last_peered":"2024-09-21T14:11:39.742005+0000","last_clean":"2024-09-21T14:11:39.742005+0000","last_became_active":"2024-09-21T14:11:13.448485+0000","last_became_peered":"2024-09-21T14:11:13.448485+0000","last_unstale":"2024-09-21T14:11:39.742005+0000","last_undegraded":"2024-09-21T14:11:39.742005+0000","last_fullsized":"2024-09-21T14:11:39.742005+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T14:10:48.487905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:10:48.487905+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:09.644047+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"558'18","reported_seq":476,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741008+0000","last_change":"2024-09-21T14:11:38.141037+0000","last_active":"2024-09-21T14:11:39.741008+0000","last_peered":"2024-09-21T14:11:39.741008+0000","last_clean":"2024-09-21T14:11:39.741008+0000","last_became_active":"2024-09-21T14:10:49.340748+0000","last_became_peered":"2024-09-21T14:10:49.340748+0000","last_unstale":"2024-09-21T14:11:39.741008+0000","last_undegraded":"2024-09-21T14:11:39.741008+0000","last_fullsized":"2024-09-21T14:11:39.741008+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"558'18","last_scrub_stamp":"2024-09-21T14:11:38.140984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:11:38.140984+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:41.730924+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":639,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.740580+0000","last_change":"2024-09-21T14:11:00.127972+0000","last_active":"2024-09-21T14:11:39.740580+0000","last_peered":"2024-09-21T14:11:39.740580+0000","last_clean":"2024-09-21T14:11:39.740580+0000","last_became_active":"2024-09-21T14:10:49.339816+0000","last_became_peered":"2024-09-21T14:10:49.339816+0000","last_unstale":"2024-09-21T14:11:39.740580+0000","last_undegraded":"2024-09-21T14:11:39.740580+0000","last_fullsized":"2024-09-21T14:11:39.740580+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:00.127907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:11:00.127907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:16.581999+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"75'79","reported_seq":1188,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.742541+0000","last_change":"2024-09-21T14:11:15.309594+0000","last_active":"2024-09-21T14:11:39.742541+0000","last_peered":"2024-09-21T14:11:39.742541+0000","last_clean":"2024-09-21T14:11:39.742541+0000","last_became_active":"2024-09-21T14:11:13.448110+0000","last_became_peered":"2024-09-21T14:11:13.448110+0000","last_unstale":"2024-09-21T14:11:39.742541+0000","last_undegraded":"2024-09-21T14:11:39.742541+0000","last_fullsized":"2024-09-21T14:11:39.742541+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:11:15.309533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:11:15.309533+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:42.160279+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"516'39","reported_seq":829,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741067+0000","last_change":"2024-09-21T14:11:17.042351+0000","last_active":"2024-09-21T14:11:39.741067+0000","last_peered":"2024-09-21T14:11:39.741067+0000","last_clean":"2024-09-21T14:11:39.741067+0000","last_became_active":"2024-09-21T14:10:49.344815+0000","last_became_peered":"2024-09-21T14:10:49.344815+0000","last_unstale":"2024-09-21T14:11:39.741067+0000","last_undegraded":"2024-09-21T14:11:39.741067+0000","last_fullsized":"2024-09-21T14:11:39.741067+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'39","last_scrub_stamp":"2024-09-21T14:11:17.042282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:11:17.042282+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:36.887641+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":67,"num_read_kb":53,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1113,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.527208+0000","last_change":"2024-09-21T14:11:03.365061+0000","last_active":"2024-09-21T14:11:39.527208+0000","last_peered":"2024-09-21T14:11:39.527208+0000","last_clean":"2024-09-21T14:11:39.527208+0000","last_became_active":"2024-09-21T14:01:41.185400+0000","last_became_peered":"2024-09-21T14:01:41.185400+0000","last_unstale":"2024-09-21T14:11:39.527208+0000","last_undegraded":"2024-09-21T14:11:39.527208+0000","last_fullsized":"2024-09-21T14:11:39.527208+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:03.365015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:11:03.365015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:27.060566+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":697,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.742638+0000","last_change":"2024-09-21T14:11:34.130231+0000","last_active":"2024-09-21T14:11:39.742638+0000","last_peered":"2024-09-21T14:11:39.742638+0000","last_clean":"2024-09-21T14:11:39.742638+0000","last_became_active":"2024-09-21T14:01:41.193973+0000","last_became_peered":"2024-09-21T14:01:41.193973+0000","last_unstale":"2024-09-21T14:11:39.742638+0000","last_undegraded":"2024-09-21T14:11:39.742638+0000","last_fullsized":"2024-09-21T14:11:39.742638+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:34.130174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:11:34.130174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:56.256298+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1104,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.526325+0000","last_change":"2024-09-21T14:11:33.578125+0000","last_active":"2024-09-21T14:11:39.526325+0000","last_peered":"2024-09-21T14:11:39.526325+0000","last_clean":"2024-09-21T14:11:39.526325+0000","last_became_active":"2024-09-21T13:58:17.757628+0000","last_became_peered":"2024-09-21T13:58:17.757628+0000","last_unstale":"2024-09-21T14:11:39.526325+0000","last_undegraded":"2024-09-21T14:11:39.526325+0000","last_fullsized":"2024-09-21T14:11:39.526325+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:33.578075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:11:33.578075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:35.700951+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"75'79","reported_seq":528,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741138+0000","last_change":"2024-09-21T14:11:07.480703+0000","last_active":"2024-09-21T14:11:39.741138+0000","last_peered":"2024-09-21T14:11:39.741138+0000","last_clean":"2024-09-21T14:11:39.741138+0000","last_became_active":"2024-09-21T14:05:39.310615+0000","last_became_peered":"2024-09-21T14:05:39.310615+0000","last_unstale":"2024-09-21T14:11:39.741138+0000","last_undegraded":"2024-09-21T14:11:39.741138+0000","last_fullsized":"2024-09-21T14:11:39.741138+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:11:07.480617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:11:07.480617+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:35.036408+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"571'59","reported_seq":1583,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.527111+0000","last_change":"2024-09-21T14:11:16.530926+0000","last_active":"2024-09-21T14:11:39.527111+0000","last_peered":"2024-09-21T14:11:39.527111+0000","last_clean":"2024-09-21T14:11:39.527111+0000","last_became_active":"2024-09-21T14:11:13.444928+0000","last_became_peered":"2024-09-21T14:11:13.444928+0000","last_unstale":"2024-09-21T14:11:39.527111+0000","last_undegraded":"2024-09-21T14:11:39.527111+0000","last_fullsized":"2024-09-21T14:11:39.527111+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"571'59","last_scrub_stamp":"2024-09-21T14:11:16.530870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:11:16.530870+0000","objects_scrubbed":10,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:28.890899+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":84,"num_read_kb":69,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1128,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.742088+0000","last_change":"2024-09-21T14:09:38.447915+0000","last_active":"2024-09-21T14:11:39.742088+0000","last_peered":"2024-09-21T14:11:39.742088+0000","last_clean":"2024-09-21T14:11:39.742088+0000","last_became_active":"2024-09-21T13:58:17.770937+0000","last_became_peered":"2024-09-21T13:58:17.770937+0000","last_unstale":"2024-09-21T14:11:39.742088+0000","last_undegraded":"2024-09-21T14:11:39.742088+0000","last_fullsized":"2024-09-21T14:11:39.742088+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:09:38.447835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:09:38.447835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":1064,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741639+0000","last_change":"2024-09-21T14:09:50.282494+0000","last_active":"2024-09-21T14:11:39.741639+0000","last_peered":"2024-09-21T14:11:39.741639+0000","last_clean":"2024-09-21T14:11:39.741639+0000","last_became_active":"2024-09-21T14:01:41.247642+0000","last_became_peered":"2024-09-21T14:01:41.247642+0000","last_unstale":"2024-09-21T14:11:39.741639+0000","last_undegraded":"2024-09-21T14:11:39.741639+0000","last_fullsized":"2024-09-21T14:11:39.741639+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-21T14:09:50.282440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T14:09:50.282440+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":986,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.740913+0000","last_change":"2024-09-21T14:11:12.991300+0000","last_active":"2024-09-21T14:11:39.740913+0000","last_peered":"2024-09-21T14:11:39.740913+0000","last_clean":"2024-09-21T14:11:39.740913+0000","last_became_active":"2024-09-21T14:10:49.341777+0000","last_became_peered":"2024-09-21T14:10:49.341777+0000","last_unstale":"2024-09-21T14:11:39.740913+0000","last_undegraded":"2024-09-21T14:11:39.740913+0000","last_fullsized":"2024-09-21T14:11:39.740913+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:12.991242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:11:12.991242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:20.335492+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":38,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.532032+0000","last_change":"2024-09-21T14:11:26.409856+0000","last_active":"2024-09-21T14:11:39.532032+0000","last_peered":"2024-09-21T14:11:39.532032+0000","last_clean":"2024-09-21T14:11:39.532032+0000","last_became_active":"2024-09-21T14:11:13.454970+0000","last_became_peered":"2024-09-21T14:11:13.454970+0000","last_unstale":"2024-09-21T14:11:39.532032+0000","last_undegraded":"2024-09-21T14:11:39.532032+0000","last_fullsized":"2024-09-21T14:11:39.532032+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:26.409797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:11:26.409797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:32.125500+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"75'79","reported_seq":1001,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:46.208177+0000","last_change":"2024-09-21T14:11:46.208177+0000","last_active":"2024-09-21T14:11:46.208177+0000","last_peered":"2024-09-21T14:11:46.208177+0000","last_clean":"2024-09-21T14:11:46.208177+0000","last_became_active":"2024-09-21T14:11:19.470686+0000","last_became_peered":"2024-09-21T14:11:19.470686+0000","last_unstale":"2024-09-21T14:11:46.208177+0000","last_undegraded":"2024-09-21T14:11:46.208177+0000","last_fullsized":"2024-09-21T14:11:46.208177+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:11:46.208113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:11:46.208113+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:13:16.060204+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"516'21","reported_seq":1183,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.527669+0000","last_change":"2024-09-21T14:10:51.251070+0000","last_active":"2024-09-21T14:11:39.527669+0000","last_peered":"2024-09-21T14:11:39.527669+0000","last_clean":"2024-09-21T14:11:39.527669+0000","last_became_active":"2024-09-21T14:09:00.258874+0000","last_became_peered":"2024-09-21T14:09:00.258874+0000","last_unstale":"2024-09-21T14:11:39.527669+0000","last_undegraded":"2024-09-21T14:11:39.527669+0000","last_fullsized":"2024-09-21T14:11:39.527669+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'21","last_scrub_stamp":"2024-09-21T14:10:51.251013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:10:51.251013+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:06.205275+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1157,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.526213+0000","last_change":"2024-09-21T14:11:21.516006+0000","last_active":"2024-09-21T14:11:39.526213+0000","last_peered":"2024-09-21T14:11:39.526213+0000","last_clean":"2024-09-21T14:11:39.526213+0000","last_became_active":"2024-09-21T13:58:17.759671+0000","last_became_peered":"2024-09-21T13:58:17.759671+0000","last_unstale":"2024-09-21T14:11:39.526213+0000","last_undegraded":"2024-09-21T14:11:39.526213+0000","last_fullsized":"2024-09-21T14:11:39.526213+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:21.515959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:11:21.515959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:42.463169+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":1052,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.527923+0000","last_change":"2024-09-21T14:11:13.479751+0000","last_active":"2024-09-21T14:11:39.527923+0000","last_peered":"2024-09-21T14:11:39.527923+0000","last_clean":"2024-09-21T14:11:39.527923+0000","last_became_active":"2024-09-21T14:11:13.452505+0000","last_became_peered":"2024-09-21T14:11:13.452505+0000","last_unstale":"2024-09-21T14:11:39.527923+0000","last_undegraded":"2024-09-21T14:11:39.527923+0000","last_fullsized":"2024-09-21T14:11:39.527923+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T14:11:04.629569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T14:11:04.629569+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:24.582268+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1025,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.527634+0000","last_change":"2024-09-21T14:10:43.473799+0000","last_active":"2024-09-21T14:11:39.527634+0000","last_peered":"2024-09-21T14:11:39.527634+0000","last_clean":"2024-09-21T14:11:39.527634+0000","last_became_active":"2024-09-21T14:01:41.182000+0000","last_became_peered":"2024-09-21T14:01:41.182000+0000","last_unstale":"2024-09-21T14:11:39.527634+0000","last_undegraded":"2024-09-21T14:11:39.527634+0000","last_fullsized":"2024-09-21T14:11:39.527634+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:10:43.473741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:10:43.473741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:11:58.368712+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":906,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.742661+0000","last_change":"2024-09-21T14:11:21.252559+0000","last_active":"2024-09-21T14:11:39.742661+0000","last_peered":"2024-09-21T14:11:39.742661+0000","last_clean":"2024-09-21T14:11:39.742661+0000","last_became_active":"2024-09-21T14:11:19.465954+0000","last_became_peered":"2024-09-21T14:11:19.465954+0000","last_unstale":"2024-09-21T14:11:39.742661+0000","last_undegraded":"2024-09-21T14:11:39.742661+0000","last_fullsized":"2024-09-21T14:11:39.742661+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:21.252502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:11:21.252502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:25.626368+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"75'79","reported_seq":599,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.527323+0000","last_change":"2024-09-21T14:11:37.678851+0000","last_active":"2024-09-21T14:11:39.527323+0000","last_peered":"2024-09-21T14:11:39.527323+0000","last_clean":"2024-09-21T14:11:39.527323+0000","last_became_active":"2024-09-21T14:11:05.584920+0000","last_became_peered":"2024-09-21T14:11:05.584920+0000","last_unstale":"2024-09-21T14:11:39.527323+0000","last_undegraded":"2024-09-21T14:11:39.527323+0000","last_fullsized":"2024-09-21T14:11:39.527323+0000","mapping_epoch":564,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":565,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:11:37.678757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:11:37.678757+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:40.667696+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"516'36","reported_seq":120,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.740983+0000","last_change":"2024-09-21T14:11:30.078177+0000","last_active":"2024-09-21T14:11:39.740983+0000","last_peered":"2024-09-21T14:11:39.740983+0000","last_clean":"2024-09-21T14:11:39.740983+0000","last_became_active":"2024-09-21T14:10:49.340725+0000","last_became_peered":"2024-09-21T14:10:49.340725+0000","last_unstale":"2024-09-21T14:11:39.740983+0000","last_undegraded":"2024-09-21T14:11:39.740983+0000","last_fullsized":"2024-09-21T14:11:39.740983+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'36","last_scrub_stamp":"2024-09-21T14:11:30.078110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:11:30.078110+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:42.054806+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":46,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":650,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.740680+0000","last_change":"2024-09-21T14:11:18.056925+0000","last_active":"2024-09-21T14:11:39.740680+0000","last_peered":"2024-09-21T14:11:39.740680+0000","last_clean":"2024-09-21T14:11:39.740680+0000","last_became_active":"2024-09-21T14:11:13.455641+0000","last_became_peered":"2024-09-21T14:11:13.455641+0000","last_unstale":"2024-09-21T14:11:39.740680+0000","last_undegraded":"2024-09-21T14:11:39.740680+0000","last_fullsized":"2024-09-21T14:11:39.740680+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:18.056877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:11:18.056877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:20.873051+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":1970,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:44.502191+0000","last_change":"2024-09-21T14:11:19.465191+0000","last_active":"2024-09-21T14:11:44.502191+0000","last_peered":"2024-09-21T14:11:44.502191+0000","last_clean":"2024-09-21T14:11:44.502191+0000","last_became_active":"2024-09-21T14:11:19.464763+0000","last_became_peered":"2024-09-21T14:11:19.464763+0000","last_unstale":"2024-09-21T14:11:44.502191+0000","last_undegraded":"2024-09-21T14:11:44.502191+0000","last_fullsized":"2024-09-21T14:11:44.502191+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-21T14:11:14.522856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T14:11:14.522856+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:15.654784+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0.00047048000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":541,"num_read_kb":536,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1043,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.743779+0000","last_change":"2024-09-21T14:11:11.566821+0000","last_active":"2024-09-21T14:11:39.743779+0000","last_peered":"2024-09-21T14:11:39.743779+0000","last_clean":"2024-09-21T14:11:39.743779+0000","last_became_active":"2024-09-21T14:01:41.200531+0000","last_became_peered":"2024-09-21T14:01:41.200531+0000","last_unstale":"2024-09-21T14:11:39.743779+0000","last_undegraded":"2024-09-21T14:11:39.743779+0000","last_fullsized":"2024-09-21T14:11:39.743779+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:11.566762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:11:11.566762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:37.548378+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1155,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.526827+0000","last_change":"2024-09-21T14:11:30.633960+0000","last_active":"2024-09-21T14:11:39.526827+0000","last_peered":"2024-09-21T14:11:39.526827+0000","last_clean":"2024-09-21T14:11:39.526827+0000","last_became_active":"2024-09-21T14:11:13.442994+0000","last_became_peered":"2024-09-21T14:11:13.442994+0000","last_unstale":"2024-09-21T14:11:39.526827+0000","last_undegraded":"2024-09-21T14:11:39.526827+0000","last_fullsized":"2024-09-21T14:11:39.526827+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:30.633910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:11:30.633910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:47.538708+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"516'44","reported_seq":905,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741968+0000","last_change":"2024-09-21T14:11:07.149781+0000","last_active":"2024-09-21T14:11:39.741968+0000","last_peered":"2024-09-21T14:11:39.741968+0000","last_clean":"2024-09-21T14:11:39.741968+0000","last_became_active":"2024-09-21T13:59:32.204624+0000","last_became_peered":"2024-09-21T13:59:32.204624+0000","last_unstale":"2024-09-21T14:11:39.741968+0000","last_undegraded":"2024-09-21T14:11:39.741968+0000","last_fullsized":"2024-09-21T14:11:39.741968+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'44","last_scrub_stamp":"2024-09-21T14:11:07.149727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:11:07.149727+0000","objects_scrubbed":8,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:09.590664+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":38,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.531383+0000","last_change":"2024-09-21T14:11:37.271458+0000","last_active":"2024-09-21T14:11:39.531383+0000","last_peered":"2024-09-21T14:11:39.531383+0000","last_clean":"2024-09-21T14:11:39.531383+0000","last_became_active":"2024-09-21T14:11:13.451947+0000","last_became_peered":"2024-09-21T14:11:13.451947+0000","last_unstale":"2024-09-21T14:11:39.531383+0000","last_undegraded":"2024-09-21T14:11:39.531383+0000","last_fullsized":"2024-09-21T14:11:39.531383+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:37.271400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:11:37.271400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:39.415667+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"75'79","reported_seq":996,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.742538+0000","last_change":"2024-09-21T14:11:31.223417+0000","last_active":"2024-09-21T14:11:39.742538+0000","last_peered":"2024-09-21T14:11:39.742538+0000","last_clean":"2024-09-21T14:11:39.742538+0000","last_became_active":"2024-09-21T14:11:19.466477+0000","last_became_peered":"2024-09-21T14:11:19.466477+0000","last_unstale":"2024-09-21T14:11:39.742538+0000","last_undegraded":"2024-09-21T14:11:39.742538+0000","last_fullsized":"2024-09-21T14:11:39.742538+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:11:31.223349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:11:31.223349+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:35.391856+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":878,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.742021+0000","last_change":"2024-09-21T14:11:19.463022+0000","last_active":"2024-09-21T14:11:39.742021+0000","last_peered":"2024-09-21T14:11:39.742021+0000","last_clean":"2024-09-21T14:11:39.742021+0000","last_became_active":"2024-09-21T14:11:19.462693+0000","last_became_peered":"2024-09-21T14:11:19.462693+0000","last_unstale":"2024-09-21T14:11:39.742021+0000","last_undegraded":"2024-09-21T14:11:39.742021+0000","last_fullsized":"2024-09-21T14:11:39.742021+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:10:46.493469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:10:46.493469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:11:56.122037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":992,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.743513+0000","last_change":"2024-09-21T14:09:37.303313+0000","last_active":"2024-09-21T14:11:39.743513+0000","last_peered":"2024-09-21T14:11:39.743513+0000","last_clean":"2024-09-21T14:11:39.743513+0000","last_became_active":"2024-09-21T14:03:57.346453+0000","last_became_peered":"2024-09-21T14:03:57.346453+0000","last_unstale":"2024-09-21T14:11:39.743513+0000","last_undegraded":"2024-09-21T14:11:39.743513+0000","last_fullsized":"2024-09-21T14:11:39.743513+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:09:37.303243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:09:37.303243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"75'79","reported_seq":589,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.527169+0000","last_change":"2024-09-21T14:11:17.490554+0000","last_active":"2024-09-21T14:11:39.527169+0000","last_peered":"2024-09-21T14:11:39.527169+0000","last_clean":"2024-09-21T14:11:39.527169+0000","last_became_active":"2024-09-21T14:05:41.345392+0000","last_became_peered":"2024-09-21T14:05:41.345392+0000","last_unstale":"2024-09-21T14:11:39.527169+0000","last_undegraded":"2024-09-21T14:11:39.527169+0000","last_fullsized":"2024-09-21T14:11:39.527169+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:11:17.490501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:11:17.490501+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:47.055039+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"516'27","reported_seq":1186,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.743546+0000","last_change":"2024-09-21T14:11:10.614517+0000","last_active":"2024-09-21T14:11:39.743546+0000","last_peered":"2024-09-21T14:11:39.743546+0000","last_clean":"2024-09-21T14:11:39.743546+0000","last_became_active":"2024-09-21T13:58:17.776310+0000","last_became_peered":"2024-09-21T13:58:17.776310+0000","last_unstale":"2024-09-21T14:11:39.743546+0000","last_undegraded":"2024-09-21T14:11:39.743546+0000","last_fullsized":"2024-09-21T14:11:39.743546+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'27","last_scrub_stamp":"2024-09-21T14:11:10.614438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:11:10.614438+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:24.560178+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1144,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:42.910606+0000","last_change":"2024-09-21T14:11:42.910606+0000","last_active":"2024-09-21T14:11:42.910606+0000","last_peered":"2024-09-21T14:11:42.910606+0000","last_clean":"2024-09-21T14:11:42.910606+0000","last_became_active":"2024-09-21T13:59:32.223247+0000","last_became_peered":"2024-09-21T13:59:32.223247+0000","last_unstale":"2024-09-21T14:11:42.910606+0000","last_undegraded":"2024-09-21T14:11:42.910606+0000","last_fullsized":"2024-09-21T14:11:42.910606+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:42.910548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:11:42.910548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:13:00.872306+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1069,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741987+0000","last_change":"2024-09-21T14:11:06.461105+0000","last_active":"2024-09-21T14:11:39.741987+0000","last_peered":"2024-09-21T14:11:39.741987+0000","last_clean":"2024-09-21T14:11:39.741987+0000","last_became_active":"2024-09-21T14:01:41.193941+0000","last_became_peered":"2024-09-21T14:01:41.193941+0000","last_unstale":"2024-09-21T14:11:39.741987+0000","last_undegraded":"2024-09-21T14:11:39.741987+0000","last_fullsized":"2024-09-21T14:11:39.741987+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:06.461058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:11:06.461058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:22.790875+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":38,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.531790+0000","last_change":"2024-09-21T14:11:16.412877+0000","last_active":"2024-09-21T14:11:39.531790+0000","last_peered":"2024-09-21T14:11:39.531790+0000","last_clean":"2024-09-21T14:11:39.531790+0000","last_became_active":"2024-09-21T14:11:13.452915+0000","last_became_peered":"2024-09-21T14:11:13.452915+0000","last_unstale":"2024-09-21T14:11:39.531790+0000","last_undegraded":"2024-09-21T14:11:39.531790+0000","last_fullsized":"2024-09-21T14:11:39.531790+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:16.412831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:11:16.412831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:35.952542+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"75'79","reported_seq":560,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741458+0000","last_change":"2024-09-21T14:11:13.474582+0000","last_active":"2024-09-21T14:11:39.741458+0000","last_peered":"2024-09-21T14:11:39.741458+0000","last_clean":"2024-09-21T14:11:39.741458+0000","last_became_active":"2024-09-21T14:11:13.450234+0000","last_became_peered":"2024-09-21T14:11:13.450234+0000","last_unstale":"2024-09-21T14:11:39.741458+0000","last_undegraded":"2024-09-21T14:11:39.741458+0000","last_fullsized":"2024-09-21T14:11:39.741458+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:09:44.277011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:09:44.277011+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"516'45","reported_seq":198,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.743559+0000","last_change":"2024-09-21T14:10:06.596332+0000","last_active":"2024-09-21T14:11:39.743559+0000","last_peered":"2024-09-21T14:11:39.743559+0000","last_clean":"2024-09-21T14:11:39.743559+0000","last_became_active":"2024-09-21T14:09:00.257088+0000","last_became_peered":"2024-09-21T14:09:00.257088+0000","last_unstale":"2024-09-21T14:11:39.743559+0000","last_undegraded":"2024-09-21T14:11:39.743559+0000","last_fullsized":"2024-09-21T14:11:39.743559+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'45","last_scrub_stamp":"2024-09-21T14:10:06.596239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:10:06.596239+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":60,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":670,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.742379+0000","last_change":"2024-09-21T14:11:00.134030+0000","last_active":"2024-09-21T14:11:39.742379+0000","last_peered":"2024-09-21T14:11:39.742379+0000","last_clean":"2024-09-21T14:11:39.742379+0000","last_became_active":"2024-09-21T13:58:17.762520+0000","last_became_peered":"2024-09-21T13:58:17.762520+0000","last_unstale":"2024-09-21T14:11:39.742379+0000","last_undegraded":"2024-09-21T14:11:39.742379+0000","last_fullsized":"2024-09-21T14:11:39.742379+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:00.133978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:11:00.133978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:08.169028+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1083,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.526919+0000","last_change":"2024-09-21T14:10:42.270730+0000","last_active":"2024-09-21T14:11:39.526919+0000","last_peered":"2024-09-21T14:11:39.526919+0000","last_clean":"2024-09-21T14:11:39.526919+0000","last_became_active":"2024-09-21T13:49:02.067472+0000","last_became_peered":"2024-09-21T13:49:02.067472+0000","last_unstale":"2024-09-21T14:11:39.526919+0000","last_undegraded":"2024-09-21T14:11:39.526919+0000","last_fullsized":"2024-09-21T14:11:39.526919+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:10:42.270672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:10:42.270672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:01.425062+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":987,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.527901+0000","last_change":"2024-09-21T14:11:26.669787+0000","last_active":"2024-09-21T14:11:39.527901+0000","last_peered":"2024-09-21T14:11:39.527901+0000","last_clean":"2024-09-21T14:11:39.527901+0000","last_became_active":"2024-09-21T14:11:13.448947+0000","last_became_peered":"2024-09-21T14:11:13.448947+0000","last_unstale":"2024-09-21T14:11:39.527901+0000","last_undegraded":"2024-09-21T14:11:39.527901+0000","last_fullsized":"2024-09-21T14:11:39.527901+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:26.669736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:11:26.669736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:39.580627+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"75'79","reported_seq":768,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.527656+0000","last_change":"2024-09-21T14:10:40.398254+0000","last_active":"2024-09-21T14:11:39.527656+0000","last_peered":"2024-09-21T14:11:39.527656+0000","last_clean":"2024-09-21T14:11:39.527656+0000","last_became_active":"2024-09-21T14:08:47.765045+0000","last_became_peered":"2024-09-21T14:08:47.765045+0000","last_unstale":"2024-09-21T14:11:39.527656+0000","last_undegraded":"2024-09-21T14:11:39.527656+0000","last_fullsized":"2024-09-21T14:11:39.527656+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:10:40.398204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:10:40.398204+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:05.552007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"516'30","reported_seq":1196,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.742298+0000","last_change":"2024-09-21T14:10:55.718375+0000","last_active":"2024-09-21T14:11:39.742298+0000","last_peered":"2024-09-21T14:11:39.742298+0000","last_clean":"2024-09-21T14:11:39.742298+0000","last_became_active":"2024-09-21T13:58:17.765427+0000","last_became_peered":"2024-09-21T13:58:17.765427+0000","last_unstale":"2024-09-21T14:11:39.742298+0000","last_undegraded":"2024-09-21T14:11:39.742298+0000","last_fullsized":"2024-09-21T14:11:39.742298+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'30","last_scrub_stamp":"2024-09-21T14:10:55.718305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:10:55.718305+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:07.425610+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1052,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.531594+0000","last_change":"2024-09-21T14:11:05.426846+0000","last_active":"2024-09-21T14:11:39.531594+0000","last_peered":"2024-09-21T14:11:39.531594+0000","last_clean":"2024-09-21T14:11:39.531594+0000","last_became_active":"2024-09-21T14:09:00.258570+0000","last_became_peered":"2024-09-21T14:09:00.258570+0000","last_unstale":"2024-09-21T14:11:39.531594+0000","last_undegraded":"2024-09-21T14:11:39.531594+0000","last_fullsized":"2024-09-21T14:11:39.531594+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:05.426790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:11:05.426790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:17.540835+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"346'14","reported_seq":1267,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.528265+0000","last_change":"2024-09-21T14:11:01.591298+0000","last_active":"2024-09-21T14:11:39.528265+0000","last_peered":"2024-09-21T14:11:39.528265+0000","last_clean":"2024-09-21T14:11:39.528265+0000","last_became_active":"2024-09-21T14:01:41.186564+0000","last_became_peered":"2024-09-21T14:01:41.186564+0000","last_unstale":"2024-09-21T14:11:39.528265+0000","last_undegraded":"2024-09-21T14:11:39.528265+0000","last_fullsized":"2024-09-21T14:11:39.528265+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'14","last_scrub_stamp":"2024-09-21T14:11:01.591239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:11:01.591239+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:07.167679+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"68'1","reported_seq":1109,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.743268+0000","last_change":"2024-09-21T14:11:22.706336+0000","last_active":"2024-09-21T14:11:39.743268+0000","last_peered":"2024-09-21T14:11:39.743268+0000","last_clean":"2024-09-21T14:11:39.743268+0000","last_became_active":"2024-09-21T13:52:04.313210+0000","last_became_peered":"2024-09-21T13:52:04.313210+0000","last_unstale":"2024-09-21T14:11:39.743268+0000","last_undegraded":"2024-09-21T14:11:39.743268+0000","last_fullsized":"2024-09-21T14:11:39.743268+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T14:11:22.706276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:11:22.706276+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:43.511209+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.e","version":"75'79","reported_seq":160,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.743174+0000","last_change":"2024-09-21T14:10:46.639536+0000","last_active":"2024-09-21T14:11:39.743174+0000","last_peered":"2024-09-21T14:11:39.743174+0000","last_clean":"2024-09-21T14:11:39.743174+0000","last_became_active":"2024-09-21T14:09:00.255631+0000","last_became_peered":"2024-09-21T14:09:00.255631+0000","last_unstale":"2024-09-21T14:11:39.743174+0000","last_undegraded":"2024-09-21T14:11:39.743174+0000","last_fullsized":"2024-09-21T14:11:39.743174+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:10:46.639473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:10:46.639473+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:11:59.281908+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"516'21","reported_seq":1084,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.740982+0000","last_change":"2024-09-21T14:10:49.346684+0000","last_active":"2024-09-21T14:11:39.740982+0000","last_peered":"2024-09-21T14:11:39.740982+0000","last_clean":"2024-09-21T14:11:39.740982+0000","last_became_active":"2024-09-21T14:10:49.345321+0000","last_became_peered":"2024-09-21T14:10:49.345321+0000","last_unstale":"2024-09-21T14:11:39.740982+0000","last_undegraded":"2024-09-21T14:11:39.740982+0000","last_fullsized":"2024-09-21T14:11:39.740982+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'21","last_scrub_stamp":"2024-09-21T14:10:19.157797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:10:19.157797+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:11:42.000711+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1079,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.742050+0000","last_change":"2024-09-21T14:11:25.524630+0000","last_active":"2024-09-21T14:11:39.742050+0000","last_peered":"2024-09-21T14:11:39.742050+0000","last_clean":"2024-09-21T14:11:39.742050+0000","last_became_active":"2024-09-21T14:11:13.448302+0000","last_became_peered":"2024-09-21T14:11:13.448302+0000","last_unstale":"2024-09-21T14:11:39.742050+0000","last_undegraded":"2024-09-21T14:11:39.742050+0000","last_fullsized":"2024-09-21T14:11:39.742050+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:25.524544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:11:25.524544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:44.752546+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":959,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.527047+0000","last_change":"2024-09-21T14:11:13.626721+0000","last_active":"2024-09-21T14:11:39.527047+0000","last_peered":"2024-09-21T14:11:39.527047+0000","last_clean":"2024-09-21T14:11:39.527047+0000","last_became_active":"2024-09-21T14:11:13.449586+0000","last_became_peered":"2024-09-21T14:11:13.449586+0000","last_unstale":"2024-09-21T14:11:39.527047+0000","last_undegraded":"2024-09-21T14:11:39.527047+0000","last_fullsized":"2024-09-21T14:11:39.527047+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:13.626673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:11:13.626673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:21.448316+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":738,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.740625+0000","last_change":"2024-09-21T14:10:49.344617+0000","last_active":"2024-09-21T14:11:39.740625+0000","last_peered":"2024-09-21T14:11:39.740625+0000","last_clean":"2024-09-21T14:11:39.740625+0000","last_became_active":"2024-09-21T14:10:49.343216+0000","last_became_peered":"2024-09-21T14:10:49.343216+0000","last_unstale":"2024-09-21T14:11:39.740625+0000","last_undegraded":"2024-09-21T14:11:39.740625+0000","last_fullsized":"2024-09-21T14:11:39.740625+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:10:12.438685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:10:12.438685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:11:42.429394+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"75'79","reported_seq":1171,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.743636+0000","last_change":"2024-09-21T14:10:46.039287+0000","last_active":"2024-09-21T14:11:39.743636+0000","last_peered":"2024-09-21T14:11:39.743636+0000","last_clean":"2024-09-21T14:11:39.743636+0000","last_became_active":"2024-09-21T14:05:41.327757+0000","last_became_peered":"2024-09-21T14:05:41.327757+0000","last_unstale":"2024-09-21T14:11:39.743636+0000","last_undegraded":"2024-09-21T14:11:39.743636+0000","last_fullsized":"2024-09-21T14:11:39.743636+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:10:46.039232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:10:46.039232+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:11:51.759284+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"516'51","reported_seq":768,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.742531+0000","last_change":"2024-09-21T14:11:30.158765+0000","last_active":"2024-09-21T14:11:39.742531+0000","last_peered":"2024-09-21T14:11:39.742531+0000","last_clean":"2024-09-21T14:11:39.742531+0000","last_became_active":"2024-09-21T13:59:32.204184+0000","last_became_peered":"2024-09-21T13:59:32.204184+0000","last_unstale":"2024-09-21T14:11:39.742531+0000","last_undegraded":"2024-09-21T14:11:39.742531+0000","last_fullsized":"2024-09-21T14:11:39.742531+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'51","last_scrub_stamp":"2024-09-21T14:11:30.158653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:11:30.158653+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:46.448132+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":928,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.743054+0000","last_change":"2024-09-21T14:11:38.301955+0000","last_active":"2024-09-21T14:11:39.743054+0000","last_peered":"2024-09-21T14:11:39.743054+0000","last_clean":"2024-09-21T14:11:39.743054+0000","last_became_active":"2024-09-21T14:11:19.472878+0000","last_became_peered":"2024-09-21T14:11:19.472878+0000","last_unstale":"2024-09-21T14:11:39.743054+0000","last_undegraded":"2024-09-21T14:11:39.743054+0000","last_fullsized":"2024-09-21T14:11:39.743054+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:38.301896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:11:38.301896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:51.654993+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1075,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.527286+0000","last_change":"2024-09-21T14:11:29.721058+0000","last_active":"2024-09-21T14:11:39.527286+0000","last_peered":"2024-09-21T14:11:39.527286+0000","last_clean":"2024-09-21T14:11:39.527286+0000","last_became_active":"2024-09-21T14:01:41.181911+0000","last_became_peered":"2024-09-21T14:01:41.181911+0000","last_unstale":"2024-09-21T14:11:39.527286+0000","last_undegraded":"2024-09-21T14:11:39.527286+0000","last_fullsized":"2024-09-21T14:11:39.527286+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:29.721000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:11:29.721000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:44.023515+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":855,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.742477+0000","last_change":"2024-09-21T14:11:34.247767+0000","last_active":"2024-09-21T14:11:39.742477+0000","last_peered":"2024-09-21T14:11:39.742477+0000","last_clean":"2024-09-21T14:11:39.742477+0000","last_became_active":"2024-09-21T14:11:19.463664+0000","last_became_peered":"2024-09-21T14:11:19.463664+0000","last_unstale":"2024-09-21T14:11:39.742477+0000","last_undegraded":"2024-09-21T14:11:39.742477+0000","last_fullsized":"2024-09-21T14:11:39.742477+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:34.247706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:11:34.247706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:36.622023+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"75'79","reported_seq":34,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.531943+0000","last_change":"2024-09-21T14:11:13.472293+0000","last_active":"2024-09-21T14:11:39.531943+0000","last_peered":"2024-09-21T14:11:39.531943+0000","last_clean":"2024-09-21T14:11:39.531943+0000","last_became_active":"2024-09-21T14:11:13.460428+0000","last_became_peered":"2024-09-21T14:11:13.460428+0000","last_unstale":"2024-09-21T14:11:39.531943+0000","last_undegraded":"2024-09-21T14:11:39.531943+0000","last_fullsized":"2024-09-21T14:11:39.531943+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:10:40.067459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:10:40.067459+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:05.510502+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"516'24","reported_seq":1200,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741242+0000","last_change":"2024-09-21T14:11:13.476010+0000","last_active":"2024-09-21T14:11:39.741242+0000","last_peered":"2024-09-21T14:11:39.741242+0000","last_clean":"2024-09-21T14:11:39.741242+0000","last_became_active":"2024-09-21T14:11:13.450947+0000","last_became_peered":"2024-09-21T14:11:13.450947+0000","last_unstale":"2024-09-21T14:11:39.741242+0000","last_undegraded":"2024-09-21T14:11:39.741242+0000","last_fullsized":"2024-09-21T14:11:39.741242+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'24","last_scrub_stamp":"2024-09-21T14:09:49.533719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:09:49.533719+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":882,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:45.190139+0000","last_change":"2024-09-21T14:11:45.190139+0000","last_active":"2024-09-21T14:11:45.190139+0000","last_peered":"2024-09-21T14:11:45.190139+0000","last_clean":"2024-09-21T14:11:45.190139+0000","last_became_active":"2024-09-21T14:11:19.469222+0000","last_became_peered":"2024-09-21T14:11:19.469222+0000","last_unstale":"2024-09-21T14:11:45.190139+0000","last_undegraded":"2024-09-21T14:11:45.190139+0000","last_fullsized":"2024-09-21T14:11:45.190139+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:45.190081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:11:45.190081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:49.941162+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1030,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.526650+0000","last_change":"2024-09-21T14:11:11.576821+0000","last_active":"2024-09-21T14:11:39.526650+0000","last_peered":"2024-09-21T14:11:39.526650+0000","last_clean":"2024-09-21T14:11:39.526650+0000","last_became_active":"2024-09-21T13:52:04.322147+0000","last_became_peered":"2024-09-21T13:52:04.322147+0000","last_unstale":"2024-09-21T14:11:39.526650+0000","last_undegraded":"2024-09-21T14:11:39.526650+0000","last_fullsized":"2024-09-21T14:11:39.526650+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:11.576777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:11:11.576777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:21.835297+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1053,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.531787+0000","last_change":"2024-09-21T14:11:30.321924+0000","last_active":"2024-09-21T14:11:39.531787+0000","last_peered":"2024-09-21T14:11:39.531787+0000","last_clean":"2024-09-21T14:11:39.531787+0000","last_became_active":"2024-09-21T14:05:30.243052+0000","last_became_peered":"2024-09-21T14:05:30.243052+0000","last_unstale":"2024-09-21T14:11:39.531787+0000","last_undegraded":"2024-09-21T14:11:39.531787+0000","last_fullsized":"2024-09-21T14:11:39.531787+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:30.321877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:11:30.321877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:45.647118+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"75'79","reported_seq":144,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.742524+0000","last_change":"2024-09-21T14:11:19.471949+0000","last_active":"2024-09-21T14:11:39.742524+0000","last_peered":"2024-09-21T14:11:39.742524+0000","last_clean":"2024-09-21T14:11:39.742524+0000","last_became_active":"2024-09-21T14:11:19.470642+0000","last_became_peered":"2024-09-21T14:11:19.470642+0000","last_unstale":"2024-09-21T14:11:39.742524+0000","last_undegraded":"2024-09-21T14:11:39.742524+0000","last_fullsized":"2024-09-21T14:11:39.742524+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:10:52.639044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:10:52.639044+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:14.834448+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"516'39","reported_seq":1175,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741610+0000","last_change":"2024-09-21T14:11:10.540588+0000","last_active":"2024-09-21T14:11:39.741610+0000","last_peered":"2024-09-21T14:11:39.741610+0000","last_clean":"2024-09-21T14:11:39.741610+0000","last_became_active":"2024-09-21T14:01:41.192481+0000","last_became_peered":"2024-09-21T14:01:41.192481+0000","last_unstale":"2024-09-21T14:11:39.741610+0000","last_undegraded":"2024-09-21T14:11:39.741610+0000","last_fullsized":"2024-09-21T14:11:39.741610+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'39","last_scrub_stamp":"2024-09-21T14:11:10.540491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:11:10.540491+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:14.680730+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":860,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.743213+0000","last_change":"2024-09-21T14:11:19.470913+0000","last_active":"2024-09-21T14:11:39.743213+0000","last_peered":"2024-09-21T14:11:39.743213+0000","last_clean":"2024-09-21T14:11:39.743213+0000","last_became_active":"2024-09-21T14:11:19.470526+0000","last_became_peered":"2024-09-21T14:11:19.470526+0000","last_unstale":"2024-09-21T14:11:39.743213+0000","last_undegraded":"2024-09-21T14:11:39.743213+0000","last_fullsized":"2024-09-21T14:11:39.743213+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T14:10:22.619411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:10:22.619411+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"575'16","reported_seq":1121,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.742423+0000","last_change":"2024-09-21T14:11:19.470104+0000","last_active":"2024-09-21T14:11:39.742423+0000","last_peered":"2024-09-21T14:11:39.742423+0000","last_clean":"2024-09-21T14:11:39.742423+0000","last_became_active":"2024-09-21T14:11:19.469532+0000","last_became_peered":"2024-09-21T14:11:19.469532+0000","last_unstale":"2024-09-21T14:11:39.742423+0000","last_undegraded":"2024-09-21T14:11:39.742423+0000","last_fullsized":"2024-09-21T14:11:39.742423+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":0,"last_scrub":"492'15","last_scrub_stamp":"2024-09-21T14:10:47.022190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:10:47.022190+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:09.968975+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1105,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741467+0000","last_change":"2024-09-21T14:11:33.434354+0000","last_active":"2024-09-21T14:11:39.741467+0000","last_peered":"2024-09-21T14:11:39.741467+0000","last_clean":"2024-09-21T14:11:39.741467+0000","last_became_active":"2024-09-21T14:01:41.246106+0000","last_became_peered":"2024-09-21T14:01:41.246106+0000","last_unstale":"2024-09-21T14:11:39.741467+0000","last_undegraded":"2024-09-21T14:11:39.741467+0000","last_fullsized":"2024-09-21T14:11:39.741467+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:33.434297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:11:33.434297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:41.150173+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"75'79","reported_seq":319,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741315+0000","last_change":"2024-09-21T14:10:51.421789+0000","last_active":"2024-09-21T14:11:39.741315+0000","last_peered":"2024-09-21T14:11:39.741315+0000","last_clean":"2024-09-21T14:11:39.741315+0000","last_became_active":"2024-09-21T14:05:41.352416+0000","last_became_peered":"2024-09-21T14:05:41.352416+0000","last_unstale":"2024-09-21T14:11:39.741315+0000","last_undegraded":"2024-09-21T14:11:39.741315+0000","last_fullsized":"2024-09-21T14:11:39.741315+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":5,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:10:51.421721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:10:51.421721+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:02.587500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":693,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.743268+0000","last_change":"2024-09-21T14:11:28.188525+0000","last_active":"2024-09-21T14:11:39.743268+0000","last_peered":"2024-09-21T14:11:39.743268+0000","last_clean":"2024-09-21T14:11:39.743268+0000","last_became_active":"2024-09-21T14:09:00.257598+0000","last_became_peered":"2024-09-21T14:09:00.257598+0000","last_unstale":"2024-09-21T14:11:39.743268+0000","last_undegraded":"2024-09-21T14:11:39.743268+0000","last_fullsized":"2024-09-21T14:11:39.743268+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:28.188466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:11:28.188466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:45.114740+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"516'18","reported_seq":46,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.532215+0000","last_change":"2024-09-21T14:11:39.272172+0000","last_active":"2024-09-21T14:11:39.532215+0000","last_peered":"2024-09-21T14:11:39.532215+0000","last_clean":"2024-09-21T14:11:39.532215+0000","last_became_active":"2024-09-21T14:11:13.439951+0000","last_became_peered":"2024-09-21T14:11:13.439951+0000","last_unstale":"2024-09-21T14:11:39.532215+0000","last_undegraded":"2024-09-21T14:11:39.532215+0000","last_fullsized":"2024-09-21T14:11:39.532215+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'18","last_scrub_stamp":"2024-09-21T14:11:39.272111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:11:39.272111+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:43.321429+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":22,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1028,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:41.357429+0000","last_change":"2024-09-21T14:11:41.357429+0000","last_active":"2024-09-21T14:11:41.357429+0000","last_peered":"2024-09-21T14:11:41.357429+0000","last_clean":"2024-09-21T14:11:41.357429+0000","last_became_active":"2024-09-21T14:07:13.246170+0000","last_became_peered":"2024-09-21T14:07:13.246170+0000","last_unstale":"2024-09-21T14:11:41.357429+0000","last_undegraded":"2024-09-21T14:11:41.357429+0000","last_fullsized":"2024-09-21T14:11:41.357429+0000","mapping_epoch":471,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:41.357369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:11:41.357369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:13:03.023343+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":885,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.742935+0000","last_change":"2024-09-21T14:11:19.472529+0000","last_active":"2024-09-21T14:11:39.742935+0000","last_peered":"2024-09-21T14:11:39.742935+0000","last_clean":"2024-09-21T14:11:39.742935+0000","last_became_active":"2024-09-21T14:11:19.471238+0000","last_became_peered":"2024-09-21T14:11:19.471238+0000","last_unstale":"2024-09-21T14:11:39.742935+0000","last_undegraded":"2024-09-21T14:11:39.742935+0000","last_fullsized":"2024-09-21T14:11:39.742935+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:10:51.064772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:10:51.064772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:11:51.528077+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"75'79","reported_seq":610,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.526878+0000","last_change":"2024-09-21T14:11:27.578700+0000","last_active":"2024-09-21T14:11:39.526878+0000","last_peered":"2024-09-21T14:11:39.526878+0000","last_clean":"2024-09-21T14:11:39.526878+0000","last_became_active":"2024-09-21T14:11:13.445287+0000","last_became_peered":"2024-09-21T14:11:13.445287+0000","last_unstale":"2024-09-21T14:11:39.526878+0000","last_undegraded":"2024-09-21T14:11:39.526878+0000","last_fullsized":"2024-09-21T14:11:39.526878+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":5,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:11:27.578640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:11:27.578640+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:48.755021+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"516'21","reported_seq":133,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741240+0000","last_change":"2024-09-21T14:11:03.050702+0000","last_active":"2024-09-21T14:11:39.741240+0000","last_peered":"2024-09-21T14:11:39.741240+0000","last_clean":"2024-09-21T14:11:39.741240+0000","last_became_active":"2024-09-21T14:10:49.341923+0000","last_became_peered":"2024-09-21T14:10:49.341923+0000","last_unstale":"2024-09-21T14:11:39.741240+0000","last_undegraded":"2024-09-21T14:11:39.741240+0000","last_fullsized":"2024-09-21T14:11:39.741240+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'21","last_scrub_stamp":"2024-09-21T14:11:03.050650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:11:03.050650+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:08.199738+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"61'1","reported_seq":911,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.742834+0000","last_change":"2024-09-21T14:11:19.470234+0000","last_active":"2024-09-21T14:11:39.742834+0000","last_peered":"2024-09-21T14:11:39.742834+0000","last_clean":"2024-09-21T14:11:39.742834+0000","last_became_active":"2024-09-21T14:11:19.469709+0000","last_became_peered":"2024-09-21T14:11:19.469709+0000","last_unstale":"2024-09-21T14:11:39.742834+0000","last_undegraded":"2024-09-21T14:11:39.742834+0000","last_fullsized":"2024-09-21T14:11:39.742834+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T14:10:53.656323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:10:53.656323+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:16.023272+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":30,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.532267+0000","last_change":"2024-09-21T14:11:13.443970+0000","last_active":"2024-09-21T14:11:39.532267+0000","last_peered":"2024-09-21T14:11:39.532267+0000","last_clean":"2024-09-21T14:11:39.532267+0000","last_became_active":"2024-09-21T14:11:13.443647+0000","last_became_peered":"2024-09-21T14:11:13.443647+0000","last_unstale":"2024-09-21T14:11:39.532267+0000","last_undegraded":"2024-09-21T14:11:39.532267+0000","last_fullsized":"2024-09-21T14:11:39.532267+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:01.302748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:11:01.302748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:09.696317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1071,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.528321+0000","last_change":"2024-09-21T14:10:50.471544+0000","last_active":"2024-09-21T14:11:39.528321+0000","last_peered":"2024-09-21T14:11:39.528321+0000","last_clean":"2024-09-21T14:11:39.528321+0000","last_became_active":"2024-09-21T13:59:32.205803+0000","last_became_peered":"2024-09-21T13:59:32.205803+0000","last_unstale":"2024-09-21T14:11:39.528321+0000","last_undegraded":"2024-09-21T14:11:39.528321+0000","last_fullsized":"2024-09-21T14:11:39.528321+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:10:50.471483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:10:50.471483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:15.819295+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"75'79","reported_seq":311,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.740826+0000","last_change":"2024-09-21T14:09:42.199233+0000","last_active":"2024-09-21T14:11:39.740826+0000","last_peered":"2024-09-21T14:11:39.740826+0000","last_clean":"2024-09-21T14:11:39.740826+0000","last_became_active":"2024-09-21T14:05:41.325618+0000","last_became_peered":"2024-09-21T14:05:41.325618+0000","last_unstale":"2024-09-21T14:11:39.740826+0000","last_undegraded":"2024-09-21T14:11:39.740826+0000","last_fullsized":"2024-09-21T14:11:39.740826+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":5,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:09:42.199185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:09:42.199185+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"516'21","reported_seq":1067,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.743794+0000","last_change":"2024-09-21T14:10:42.557080+0000","last_active":"2024-09-21T14:11:39.743794+0000","last_peered":"2024-09-21T14:11:39.743794+0000","last_clean":"2024-09-21T14:11:39.743794+0000","last_became_active":"2024-09-21T14:01:41.187904+0000","last_became_peered":"2024-09-21T14:01:41.187904+0000","last_unstale":"2024-09-21T14:11:39.743794+0000","last_undegraded":"2024-09-21T14:11:39.743794+0000","last_fullsized":"2024-09-21T14:11:39.743794+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'21","last_scrub_stamp":"2024-09-21T14:10:42.557023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:10:42.557023+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:10.798670+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":672,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.740754+0000","last_change":"2024-09-21T14:11:33.019928+0000","last_active":"2024-09-21T14:11:39.740754+0000","last_peered":"2024-09-21T14:11:39.740754+0000","last_clean":"2024-09-21T14:11:39.740754+0000","last_became_active":"2024-09-21T14:11:13.451798+0000","last_became_peered":"2024-09-21T14:11:13.451798+0000","last_unstale":"2024-09-21T14:11:39.740754+0000","last_undegraded":"2024-09-21T14:11:39.740754+0000","last_fullsized":"2024-09-21T14:11:39.740754+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:33.019878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:11:33.019878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:54.945357+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"571'16","reported_seq":41,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.532041+0000","last_change":"2024-09-21T14:11:13.476028+0000","last_active":"2024-09-21T14:11:39.532041+0000","last_peered":"2024-09-21T14:11:39.532041+0000","last_clean":"2024-09-21T14:11:39.532041+0000","last_became_active":"2024-09-21T14:11:13.462346+0000","last_became_peered":"2024-09-21T14:11:13.462346+0000","last_unstale":"2024-09-21T14:11:39.532041+0000","last_undegraded":"2024-09-21T14:11:39.532041+0000","last_fullsized":"2024-09-21T14:11:39.532041+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"498'15","last_scrub_stamp":"2024-09-21T14:11:09.539231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:11:09.539231+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:16.052628+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1123,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.742859+0000","last_change":"2024-09-21T14:11:12.676444+0000","last_active":"2024-09-21T14:11:39.742859+0000","last_peered":"2024-09-21T14:11:39.742859+0000","last_clean":"2024-09-21T14:11:39.742859+0000","last_became_active":"2024-09-21T14:01:41.200113+0000","last_became_peered":"2024-09-21T14:01:41.200113+0000","last_unstale":"2024-09-21T14:11:39.742859+0000","last_undegraded":"2024-09-21T14:11:39.742859+0000","last_fullsized":"2024-09-21T14:11:39.742859+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:12.676373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:11:12.676373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:29.745437+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"75'79","reported_seq":129,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.740515+0000","last_change":"2024-09-21T14:11:28.099284+0000","last_active":"2024-09-21T14:11:39.740515+0000","last_peered":"2024-09-21T14:11:39.740515+0000","last_clean":"2024-09-21T14:11:39.740515+0000","last_became_active":"2024-09-21T14:11:13.451849+0000","last_became_peered":"2024-09-21T14:11:13.451849+0000","last_unstale":"2024-09-21T14:11:39.740515+0000","last_undegraded":"2024-09-21T14:11:39.740515+0000","last_fullsized":"2024-09-21T14:11:39.740515+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":5,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:11:28.099231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:11:28.099231+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:52.970057+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"516'30","reported_seq":1087,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.532170+0000","last_change":"2024-09-21T14:11:12.354250+0000","last_active":"2024-09-21T14:11:39.532170+0000","last_peered":"2024-09-21T14:11:39.532170+0000","last_clean":"2024-09-21T14:11:39.532170+0000","last_became_active":"2024-09-21T13:59:32.208332+0000","last_became_peered":"2024-09-21T13:59:32.208332+0000","last_unstale":"2024-09-21T14:11:39.532170+0000","last_undegraded":"2024-09-21T14:11:39.532170+0000","last_fullsized":"2024-09-21T14:11:39.532170+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'30","last_scrub_stamp":"2024-09-21T14:11:12.354195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:11:12.354195+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:20.890317+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":879,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741599+0000","last_change":"2024-09-21T14:11:19.471330+0000","last_active":"2024-09-21T14:11:39.741599+0000","last_peered":"2024-09-21T14:11:39.741599+0000","last_clean":"2024-09-21T14:11:39.741599+0000","last_became_active":"2024-09-21T14:11:19.471040+0000","last_became_peered":"2024-09-21T14:11:19.471040+0000","last_unstale":"2024-09-21T14:11:39.741599+0000","last_undegraded":"2024-09-21T14:11:39.741599+0000","last_fullsized":"2024-09-21T14:11:39.741599+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:10:19.686004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:10:19.686004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"346'10","reported_seq":1284,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.743745+0000","last_change":"2024-09-21T14:11:02.349223+0000","last_active":"2024-09-21T14:11:39.743745+0000","last_peered":"2024-09-21T14:11:39.743745+0000","last_clean":"2024-09-21T14:11:39.743745+0000","last_became_active":"2024-09-21T14:01:41.206303+0000","last_became_peered":"2024-09-21T14:01:41.206303+0000","last_unstale":"2024-09-21T14:11:39.743745+0000","last_undegraded":"2024-09-21T14:11:39.743745+0000","last_fullsized":"2024-09-21T14:11:39.743745+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'10","last_scrub_stamp":"2024-09-21T14:11:02.349171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:11:02.349171+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:23.300972+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":716,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741179+0000","last_change":"2024-09-21T14:11:13.453684+0000","last_active":"2024-09-21T14:11:39.741179+0000","last_peered":"2024-09-21T14:11:39.741179+0000","last_clean":"2024-09-21T14:11:39.741179+0000","last_became_active":"2024-09-21T14:11:13.453350+0000","last_became_peered":"2024-09-21T14:11:13.453350+0000","last_unstale":"2024-09-21T14:11:39.741179+0000","last_undegraded":"2024-09-21T14:11:39.741179+0000","last_fullsized":"2024-09-21T14:11:39.741179+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:07.043612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:11:07.043612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:28.590410+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"516'27","reported_seq":1190,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741814+0000","last_change":"2024-09-21T14:11:24.722760+0000","last_active":"2024-09-21T14:11:39.741814+0000","last_peered":"2024-09-21T14:11:39.741814+0000","last_clean":"2024-09-21T14:11:39.741814+0000","last_became_active":"2024-09-21T13:58:17.771497+0000","last_became_peered":"2024-09-21T13:58:17.771497+0000","last_unstale":"2024-09-21T14:11:39.741814+0000","last_undegraded":"2024-09-21T14:11:39.741814+0000","last_fullsized":"2024-09-21T14:11:39.741814+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'27","last_scrub_stamp":"2024-09-21T14:11:24.722644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:11:24.722644+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:54.057781+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":513,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.742108+0000","last_change":"2024-09-21T14:11:19.466314+0000","last_active":"2024-09-21T14:11:39.742108+0000","last_peered":"2024-09-21T14:11:39.742108+0000","last_clean":"2024-09-21T14:11:39.742108+0000","last_became_active":"2024-09-21T14:11:19.465996+0000","last_became_peered":"2024-09-21T14:11:19.465996+0000","last_unstale":"2024-09-21T14:11:39.742108+0000","last_undegraded":"2024-09-21T14:11:39.742108+0000","last_fullsized":"2024-09-21T14:11:39.742108+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:10:52.042228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:10:52.042228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:21.748017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1150,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.526511+0000","last_change":"2024-09-21T14:11:35.617965+0000","last_active":"2024-09-21T14:11:39.526511+0000","last_peered":"2024-09-21T14:11:39.526511+0000","last_clean":"2024-09-21T14:11:39.526511+0000","last_became_active":"2024-09-21T14:11:13.441313+0000","last_became_peered":"2024-09-21T14:11:13.441313+0000","last_unstale":"2024-09-21T14:11:39.526511+0000","last_undegraded":"2024-09-21T14:11:39.526511+0000","last_fullsized":"2024-09-21T14:11:39.526511+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:35.617914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:11:35.617914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:35.925993+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":38,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.531386+0000","last_change":"2024-09-21T14:11:24.414702+0000","last_active":"2024-09-21T14:11:39.531386+0000","last_peered":"2024-09-21T14:11:39.531386+0000","last_clean":"2024-09-21T14:11:39.531386+0000","last_became_active":"2024-09-21T14:11:13.459760+0000","last_became_peered":"2024-09-21T14:11:13.459760+0000","last_unstale":"2024-09-21T14:11:39.531386+0000","last_undegraded":"2024-09-21T14:11:39.531386+0000","last_fullsized":"2024-09-21T14:11:39.531386+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:24.414653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:11:24.414653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:50.482709+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"516'21","reported_seq":1197,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.526621+0000","last_change":"2024-09-21T14:10:47.307322+0000","last_active":"2024-09-21T14:11:39.526621+0000","last_peered":"2024-09-21T14:11:39.526621+0000","last_clean":"2024-09-21T14:11:39.526621+0000","last_became_active":"2024-09-21T13:58:17.764120+0000","last_became_peered":"2024-09-21T13:58:17.764120+0000","last_unstale":"2024-09-21T14:11:39.526621+0000","last_undegraded":"2024-09-21T14:11:39.526621+0000","last_fullsized":"2024-09-21T14:11:39.526621+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'21","last_scrub_stamp":"2024-09-21T14:10:47.307251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:10:47.307251+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:13.879687+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":899,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.742761+0000","last_change":"2024-09-21T14:11:29.244714+0000","last_active":"2024-09-21T14:11:39.742761+0000","last_peered":"2024-09-21T14:11:39.742761+0000","last_clean":"2024-09-21T14:11:39.742761+0000","last_became_active":"2024-09-21T14:11:19.470391+0000","last_became_peered":"2024-09-21T14:11:19.470391+0000","last_unstale":"2024-09-21T14:11:39.742761+0000","last_undegraded":"2024-09-21T14:11:39.742761+0000","last_fullsized":"2024-09-21T14:11:39.742761+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:29.244663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:11:29.244663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:54.090608+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1012,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:44.390424+0000","last_change":"2024-09-21T14:11:44.390424+0000","last_active":"2024-09-21T14:11:44.390424+0000","last_peered":"2024-09-21T14:11:44.390424+0000","last_clean":"2024-09-21T14:11:44.390424+0000","last_became_active":"2024-09-21T13:57:12.703441+0000","last_became_peered":"2024-09-21T13:57:12.703441+0000","last_unstale":"2024-09-21T14:11:44.390424+0000","last_undegraded":"2024-09-21T14:11:44.390424+0000","last_fullsized":"2024-09-21T14:11:44.390424+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:44.390375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:11:44.390375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:13:03.876615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":722,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.740504+0000","last_change":"2024-09-21T14:11:26.075362+0000","last_active":"2024-09-21T14:11:39.740504+0000","last_peered":"2024-09-21T14:11:39.740504+0000","last_clean":"2024-09-21T14:11:39.740504+0000","last_became_active":"2024-09-21T14:10:49.340390+0000","last_became_peered":"2024-09-21T14:10:49.340390+0000","last_unstale":"2024-09-21T14:11:39.740504+0000","last_undegraded":"2024-09-21T14:11:39.740504+0000","last_fullsized":"2024-09-21T14:11:39.740504+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:26.075312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:11:26.075312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:41.431629+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"75'79","reported_seq":336,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.528263+0000","last_change":"2024-09-21T14:11:21.565075+0000","last_active":"2024-09-21T14:11:39.528263+0000","last_peered":"2024-09-21T14:11:39.528263+0000","last_clean":"2024-09-21T14:11:39.528263+0000","last_became_active":"2024-09-21T14:09:00.260220+0000","last_became_peered":"2024-09-21T14:09:00.260220+0000","last_unstale":"2024-09-21T14:11:39.528263+0000","last_undegraded":"2024-09-21T14:11:39.528263+0000","last_fullsized":"2024-09-21T14:11:39.528263+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":5,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:11:21.565024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:11:21.565024+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:45.505021+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"516'27","reported_seq":1160,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.531777+0000","last_change":"2024-09-21T14:10:53.415737+0000","last_active":"2024-09-21T14:11:39.531777+0000","last_peered":"2024-09-21T14:11:39.531777+0000","last_clean":"2024-09-21T14:11:39.531777+0000","last_became_active":"2024-09-21T13:59:32.231567+0000","last_became_peered":"2024-09-21T13:59:32.231567+0000","last_unstale":"2024-09-21T14:11:39.531777+0000","last_undegraded":"2024-09-21T14:11:39.531777+0000","last_fullsized":"2024-09-21T14:11:39.531777+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'27","last_scrub_stamp":"2024-09-21T14:10:53.415682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:10:53.415682+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:11:57.472353+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1026,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741076+0000","last_change":"2024-09-21T14:10:59.326076+0000","last_active":"2024-09-21T14:11:39.741076+0000","last_peered":"2024-09-21T14:11:39.741076+0000","last_clean":"2024-09-21T14:11:39.741076+0000","last_became_active":"2024-09-21T14:01:41.190760+0000","last_became_peered":"2024-09-21T14:01:41.190760+0000","last_unstale":"2024-09-21T14:11:39.741076+0000","last_undegraded":"2024-09-21T14:11:39.741076+0000","last_fullsized":"2024-09-21T14:11:39.741076+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:10:59.326031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:10:59.326031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:27.455413+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":905,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:40.266721+0000","last_change":"2024-09-21T14:11:40.266721+0000","last_active":"2024-09-21T14:11:40.266721+0000","last_peered":"2024-09-21T14:11:40.266721+0000","last_clean":"2024-09-21T14:11:40.266721+0000","last_became_active":"2024-09-21T14:11:19.468570+0000","last_became_peered":"2024-09-21T14:11:19.468570+0000","last_unstale":"2024-09-21T14:11:40.266721+0000","last_undegraded":"2024-09-21T14:11:40.266721+0000","last_fullsized":"2024-09-21T14:11:40.266721+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:40.266656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:11:40.266656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:13:03.164277+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":38,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.531570+0000","last_change":"2024-09-21T14:11:25.438297+0000","last_active":"2024-09-21T14:11:39.531570+0000","last_peered":"2024-09-21T14:11:39.531570+0000","last_clean":"2024-09-21T14:11:39.531570+0000","last_became_active":"2024-09-21T14:11:13.455917+0000","last_became_peered":"2024-09-21T14:11:13.455917+0000","last_unstale":"2024-09-21T14:11:39.531570+0000","last_undegraded":"2024-09-21T14:11:39.531570+0000","last_fullsized":"2024-09-21T14:11:39.531570+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:25.438247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:11:25.438247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:35.273105+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"516'12","reported_seq":1105,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.531417+0000","last_change":"2024-09-21T14:10:43.495107+0000","last_active":"2024-09-21T14:11:39.531417+0000","last_peered":"2024-09-21T14:11:39.531417+0000","last_clean":"2024-09-21T14:11:39.531417+0000","last_became_active":"2024-09-21T14:09:31.261004+0000","last_became_peered":"2024-09-21T14:09:31.261004+0000","last_unstale":"2024-09-21T14:11:39.531417+0000","last_undegraded":"2024-09-21T14:11:39.531417+0000","last_fullsized":"2024-09-21T14:11:39.531417+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'12","last_scrub_stamp":"2024-09-21T14:10:43.495045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:10:43.495045+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:02.257308+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":823,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741385+0000","last_change":"2024-09-21T14:11:21.093192+0000","last_active":"2024-09-21T14:11:39.741385+0000","last_peered":"2024-09-21T14:11:39.741385+0000","last_clean":"2024-09-21T14:11:39.741385+0000","last_became_active":"2024-09-21T14:11:13.453444+0000","last_became_peered":"2024-09-21T14:11:13.453444+0000","last_unstale":"2024-09-21T14:11:39.741385+0000","last_undegraded":"2024-09-21T14:11:39.741385+0000","last_fullsized":"2024-09-21T14:11:39.741385+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:21.093138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:11:21.093138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:29.250375+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":702,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741501+0000","last_change":"2024-09-21T14:11:29.175124+0000","last_active":"2024-09-21T14:11:39.741501+0000","last_peered":"2024-09-21T14:11:39.741501+0000","last_clean":"2024-09-21T14:11:39.741501+0000","last_became_active":"2024-09-21T14:09:00.257432+0000","last_became_peered":"2024-09-21T14:09:00.257432+0000","last_unstale":"2024-09-21T14:11:39.741501+0000","last_undegraded":"2024-09-21T14:11:39.741501+0000","last_fullsized":"2024-09-21T14:11:39.741501+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:29.175063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:11:29.175063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:29.902783+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":830,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741195+0000","last_change":"2024-09-21T14:11:19.470112+0000","last_active":"2024-09-21T14:11:39.741195+0000","last_peered":"2024-09-21T14:11:39.741195+0000","last_clean":"2024-09-21T14:11:39.741195+0000","last_became_active":"2024-09-21T14:11:19.469802+0000","last_became_peered":"2024-09-21T14:11:19.469802+0000","last_unstale":"2024-09-21T14:11:39.741195+0000","last_undegraded":"2024-09-21T14:11:39.741195+0000","last_fullsized":"2024-09-21T14:11:39.741195+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:10:15.727357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:10:15.727357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1105,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.527075+0000","last_change":"2024-09-21T14:10:43.235227+0000","last_active":"2024-09-21T14:11:39.527075+0000","last_peered":"2024-09-21T14:11:39.527075+0000","last_clean":"2024-09-21T14:11:39.527075+0000","last_became_active":"2024-09-21T14:01:41.187438+0000","last_became_peered":"2024-09-21T14:01:41.187438+0000","last_unstale":"2024-09-21T14:11:39.527075+0000","last_undegraded":"2024-09-21T14:11:39.527075+0000","last_fullsized":"2024-09-21T14:11:39.527075+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:10:43.235173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:10:43.235173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:10.679058+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":38,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.531454+0000","last_change":"2024-09-21T14:11:33.274104+0000","last_active":"2024-09-21T14:11:39.531454+0000","last_peered":"2024-09-21T14:11:39.531454+0000","last_clean":"2024-09-21T14:11:39.531454+0000","last_became_active":"2024-09-21T14:11:13.460267+0000","last_became_peered":"2024-09-21T14:11:13.460267+0000","last_unstale":"2024-09-21T14:11:39.531454+0000","last_undegraded":"2024-09-21T14:11:39.531454+0000","last_fullsized":"2024-09-21T14:11:39.531454+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:33.274053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:11:33.274053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:44.042337+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"516'27","reported_seq":733,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.743736+0000","last_change":"2024-09-21T14:11:32.143120+0000","last_active":"2024-09-21T14:11:39.743736+0000","last_peered":"2024-09-21T14:11:39.743736+0000","last_clean":"2024-09-21T14:11:39.743736+0000","last_became_active":"2024-09-21T14:01:41.194606+0000","last_became_peered":"2024-09-21T14:01:41.194606+0000","last_unstale":"2024-09-21T14:11:39.743736+0000","last_undegraded":"2024-09-21T14:11:39.743736+0000","last_fullsized":"2024-09-21T14:11:39.743736+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'27","last_scrub_stamp":"2024-09-21T14:11:32.143037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:11:32.143037+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:38.418630+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":670,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741300+0000","last_change":"2024-09-21T14:10:51.061937+0000","last_active":"2024-09-21T14:11:39.741300+0000","last_peered":"2024-09-21T14:11:39.741300+0000","last_clean":"2024-09-21T14:11:39.741300+0000","last_became_active":"2024-09-21T14:10:49.343665+0000","last_became_peered":"2024-09-21T14:10:49.343665+0000","last_unstale":"2024-09-21T14:11:39.741300+0000","last_undegraded":"2024-09-21T14:11:39.741300+0000","last_fullsized":"2024-09-21T14:11:39.741300+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:10:51.061888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:10:51.061888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:11:56.379877+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113,"ondisk_log_size":113,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1532,"num_read_kb":1289,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":344,"num_bytes_recovered":6508,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":976,"ondisk_log_size":976,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2262,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":541,"num_read_kb":536,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":853056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1659,"ondisk_log_size":1659,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":298,"seq":1279900254357,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":696944,"kb_used_data":1568,"kb_used_omap":4,"kb_used_meta":695355,"kb_avail":93048208,"statfs":{"total":95995035648,"available":95281364992,"internally_reserved":0,"allocated":1605632,"data_stored":970313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":712044200},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463196,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":682784,"kb_used_data":1868,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":93062368,"statfs":{"total":95995035648,"available":95295864832,"internally_reserved":0,"allocated":1912832,"data_stored":1296908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":697235910},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":345,"seq":1481763717243,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":349412,"kb_used_data":1816,"kb_used_omap":4,"kb_used_meta":347579,"kb_avail":93395740,"statfs":{"total":95995035648,"available":95637237760,"internally_reserved":0,"allocated":1859584,"data_stored":1232942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":355921571},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921065,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":660284,"kb_used_data":1580,"kb_used_omap":1,"kb_used_meta":658686,"kb_avail":93084868,"statfs":{"total":95995035648,"available":95318904832,"internally_reserved":0,"allocated":1617920,"data_stored":969846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":674494917},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":240,"seq":1030792151217,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":854916,"kb_used_data":231220,"kb_used_omap":4,"kb_used_meta":623675,"kb_avail":92890236,"statfs":{"total":95995035648,"available":95119601664,"internally_reserved":0,"allocated":236769280,"data_stored":236114388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4443,"internal_metadata":638643877},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":130,"seq":558345748718,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":940588,"kb_used_data":1816,"kb_used_omap":4,"kb_used_meta":938747,"kb_avail":92804564,"statfs":{"total":95995035648,"available":95031873536,"internally_reserved":0,"allocated":1859584,"data_stored":1232854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4441,"internal_metadata":961277607},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575163,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":903248,"kb_used_data":1600,"kb_used_omap":1,"kb_used_meta":901630,"kb_avail":92841904,"statfs":{"total":95995035648,"available":95070109696,"internally_reserved":0,"allocated":1638400,"data_stored":970260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":923269573},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738689,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":917428,"kb_used_data":1888,"kb_used_omap":1,"kb_used_meta":915518,"kb_avail":92827724,"statfs":{"total":95995035648,"available":95055589376,"internally_reserved":0,"allocated":1933312,"data_stored":1297266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":937490885},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T14:11:49.545 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T14:11:49.806 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:11:49.927 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:49 smithi049 ceph-mon[40489]: pgmap v1179: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-21T14:11:49.928 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:49 smithi049 ceph-mon[40489]: from='client.26939 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:11:49.928 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:49 smithi049 ceph-mon[40489]: 3.f scrub starts 2024-09-21T14:11:49.928 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:49 smithi049 ceph-mon[40489]: 3.f scrub ok 2024-09-21T14:11:49.928 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:49 smithi049 ceph-mon[30080]: pgmap v1179: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-21T14:11:49.928 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:49 smithi049 ceph-mon[30080]: from='client.26939 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:11:49.928 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:49 smithi049 ceph-mon[30080]: 3.f scrub starts 2024-09-21T14:11:49.928 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:49 smithi049 ceph-mon[30080]: 3.f scrub ok 2024-09-21T14:11:49.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:49 smithi187 ceph-mon[33032]: pgmap v1179: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-21T14:11:49.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:49 smithi187 ceph-mon[33032]: from='client.26939 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:11:49.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:49 smithi187 ceph-mon[33032]: 3.f scrub starts 2024-09-21T14:11:49.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:49 smithi187 ceph-mon[33032]: 3.f scrub ok 2024-09-21T14:11:50.501 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-09-21T14:11:50.503 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-09-21T14:11:50.694 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:11:50 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug there is no tcmu-runner data available 2024-09-21T14:11:50.754 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:11:50 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: ::ffff:172.21.15.187 - - [21/Sep/2024:14:11:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-09-21T14:11:51.304 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":1180,"stamp":"2024-09-21T14:11:50.216238+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":2073,"num_read_kb":1825,"num_write":1008,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":365,"num_bytes_recovered":861826,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2764,"ondisk_log_size":2764,"up":456,"acting":456,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":459,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6005596,"kb_used_data":243356,"kb_used_omap":23,"kb_used_meta":5762088,"kb_avail":743955620,"statfs":{"total":767960285184,"available":761810554880,"internally_reserved":0,"allocated":249196544,"data_stored":244084777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24148,"internal_metadata":5900378540},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.691044"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":941,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741293+0000","last_change":"2024-09-21T14:10:53.033673+0000","last_active":"2024-09-21T14:11:39.741293+0000","last_peered":"2024-09-21T14:11:39.741293+0000","last_clean":"2024-09-21T14:11:39.741293+0000","last_became_active":"2024-09-21T14:10:49.340676+0000","last_became_peered":"2024-09-21T14:10:49.340676+0000","last_unstale":"2024-09-21T14:11:39.741293+0000","last_undegraded":"2024-09-21T14:11:39.741293+0000","last_fullsized":"2024-09-21T14:11:39.741293+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:10:53.033628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:10:53.033628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:15.576784+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"516'21","reported_seq":1188,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.740798+0000","last_change":"2024-09-21T14:11:26.806163+0000","last_active":"2024-09-21T14:11:39.740798+0000","last_peered":"2024-09-21T14:11:39.740798+0000","last_clean":"2024-09-21T14:11:39.740798+0000","last_became_active":"2024-09-21T13:58:17.771220+0000","last_became_peered":"2024-09-21T13:58:17.771220+0000","last_unstale":"2024-09-21T14:11:39.740798+0000","last_undegraded":"2024-09-21T14:11:39.740798+0000","last_fullsized":"2024-09-21T14:11:39.740798+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'21","last_scrub_stamp":"2024-09-21T14:11:26.806087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:11:26.806087+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:37.946810+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1120,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.526374+0000","last_change":"2024-09-21T14:11:15.566060+0000","last_active":"2024-09-21T14:11:39.526374+0000","last_peered":"2024-09-21T14:11:39.526374+0000","last_clean":"2024-09-21T14:11:39.526374+0000","last_became_active":"2024-09-21T14:01:41.202168+0000","last_became_peered":"2024-09-21T14:01:41.202168+0000","last_unstale":"2024-09-21T14:11:39.526374+0000","last_undegraded":"2024-09-21T14:11:39.526374+0000","last_fullsized":"2024-09-21T14:11:39.526374+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:15.566000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:11:15.566000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:27.460647+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1096,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741338+0000","last_change":"2024-09-21T14:10:12.684731+0000","last_active":"2024-09-21T14:11:39.741338+0000","last_peered":"2024-09-21T14:11:39.741338+0000","last_clean":"2024-09-21T14:11:39.741338+0000","last_became_active":"2024-09-21T14:01:41.203275+0000","last_became_peered":"2024-09-21T14:01:41.203275+0000","last_unstale":"2024-09-21T14:11:39.741338+0000","last_undegraded":"2024-09-21T14:11:39.741338+0000","last_fullsized":"2024-09-21T14:11:39.741338+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:10:12.684656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:10:12.684656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":38,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.531441+0000","last_change":"2024-09-21T14:11:28.412019+0000","last_active":"2024-09-21T14:11:39.531441+0000","last_peered":"2024-09-21T14:11:39.531441+0000","last_clean":"2024-09-21T14:11:39.531441+0000","last_became_active":"2024-09-21T14:11:13.451144+0000","last_became_peered":"2024-09-21T14:11:13.451144+0000","last_unstale":"2024-09-21T14:11:39.531441+0000","last_undegraded":"2024-09-21T14:11:39.531441+0000","last_fullsized":"2024-09-21T14:11:39.531441+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:28.411959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:11:28.411959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:49.664370+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":30,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.532132+0000","last_change":"2024-09-21T14:11:13.456954+0000","last_active":"2024-09-21T14:11:39.532132+0000","last_peered":"2024-09-21T14:11:39.532132+0000","last_clean":"2024-09-21T14:11:39.532132+0000","last_became_active":"2024-09-21T14:11:13.456666+0000","last_became_peered":"2024-09-21T14:11:13.456666+0000","last_unstale":"2024-09-21T14:11:39.532132+0000","last_undegraded":"2024-09-21T14:11:39.532132+0000","last_fullsized":"2024-09-21T14:11:39.532132+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:10:49.045362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:10:49.045362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:14.009955+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"516'39","reported_seq":46,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.531343+0000","last_change":"2024-09-21T14:11:13.489169+0000","last_active":"2024-09-21T14:11:39.531343+0000","last_peered":"2024-09-21T14:11:39.531343+0000","last_clean":"2024-09-21T14:11:39.531343+0000","last_became_active":"2024-09-21T14:11:13.462350+0000","last_became_peered":"2024-09-21T14:11:13.462350+0000","last_unstale":"2024-09-21T14:11:39.531343+0000","last_undegraded":"2024-09-21T14:11:39.531343+0000","last_fullsized":"2024-09-21T14:11:39.531343+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'39","last_scrub_stamp":"2024-09-21T14:10:58.615240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:10:58.615240+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:18.908576+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":59,"num_read_kb":50,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":167,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.740565+0000","last_change":"2024-09-21T14:11:30.836654+0000","last_active":"2024-09-21T14:11:39.740565+0000","last_peered":"2024-09-21T14:11:39.740565+0000","last_clean":"2024-09-21T14:11:39.740565+0000","last_became_active":"2024-09-21T14:09:00.253451+0000","last_became_peered":"2024-09-21T14:09:00.253451+0000","last_unstale":"2024-09-21T14:11:39.740565+0000","last_undegraded":"2024-09-21T14:11:39.740565+0000","last_fullsized":"2024-09-21T14:11:39.740565+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:30.836593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:11:30.836593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:33.460589+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":1077,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.744025+0000","last_change":"2024-09-21T14:11:12.546029+0000","last_active":"2024-09-21T14:11:39.744025+0000","last_peered":"2024-09-21T14:11:39.744025+0000","last_clean":"2024-09-21T14:11:39.744025+0000","last_became_active":"2024-09-21T14:01:41.194517+0000","last_became_peered":"2024-09-21T14:01:41.194517+0000","last_unstale":"2024-09-21T14:11:39.744025+0000","last_undegraded":"2024-09-21T14:11:39.744025+0000","last_fullsized":"2024-09-21T14:11:39.744025+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:12.545976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:11:12.545976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:14.332461+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"516'21","reported_seq":178,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741185+0000","last_change":"2024-09-21T14:10:13.694713+0000","last_active":"2024-09-21T14:11:39.741185+0000","last_peered":"2024-09-21T14:11:39.741185+0000","last_clean":"2024-09-21T14:11:39.741185+0000","last_became_active":"2024-09-21T14:09:00.251527+0000","last_became_peered":"2024-09-21T14:09:00.251527+0000","last_unstale":"2024-09-21T14:11:39.741185+0000","last_undegraded":"2024-09-21T14:11:39.741185+0000","last_fullsized":"2024-09-21T14:11:39.741185+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'21","last_scrub_stamp":"2024-09-21T14:10:13.694626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:10:13.694626+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1026,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741483+0000","last_change":"2024-09-21T14:10:47.486485+0000","last_active":"2024-09-21T14:11:39.741483+0000","last_peered":"2024-09-21T14:11:39.741483+0000","last_clean":"2024-09-21T14:11:39.741483+0000","last_became_active":"2024-09-21T14:01:41.188248+0000","last_became_peered":"2024-09-21T14:01:41.188248+0000","last_unstale":"2024-09-21T14:11:39.741483+0000","last_undegraded":"2024-09-21T14:11:39.741483+0000","last_fullsized":"2024-09-21T14:11:39.741483+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:10:47.486425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:10:47.486425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:11:49.215493+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":987,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.740370+0000","last_change":"2024-09-21T14:11:36.078219+0000","last_active":"2024-09-21T14:11:39.740370+0000","last_peered":"2024-09-21T14:11:39.740370+0000","last_clean":"2024-09-21T14:11:39.740370+0000","last_became_active":"2024-09-21T14:10:49.341656+0000","last_became_peered":"2024-09-21T14:10:49.341656+0000","last_unstale":"2024-09-21T14:11:39.740370+0000","last_undegraded":"2024-09-21T14:11:39.740370+0000","last_fullsized":"2024-09-21T14:11:39.740370+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:36.078168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:11:36.078168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:13:05.720118+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":1061,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:44.352404+0000","last_change":"2024-09-21T14:11:44.352404+0000","last_active":"2024-09-21T14:11:44.352404+0000","last_peered":"2024-09-21T14:11:44.352404+0000","last_clean":"2024-09-21T14:11:44.352404+0000","last_became_active":"2024-09-21T14:09:31.257075+0000","last_became_peered":"2024-09-21T14:09:31.257075+0000","last_unstale":"2024-09-21T14:11:44.352404+0000","last_undegraded":"2024-09-21T14:11:44.352404+0000","last_fullsized":"2024-09-21T14:11:44.352404+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:44.352339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:11:44.352339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:49.432595+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"516'9","reported_seq":168,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741580+0000","last_change":"2024-09-21T14:09:41.805340+0000","last_active":"2024-09-21T14:11:39.741580+0000","last_peered":"2024-09-21T14:11:39.741580+0000","last_clean":"2024-09-21T14:11:39.741580+0000","last_became_active":"2024-09-21T14:09:00.253147+0000","last_became_peered":"2024-09-21T14:09:00.253147+0000","last_unstale":"2024-09-21T14:11:39.741580+0000","last_undegraded":"2024-09-21T14:11:39.741580+0000","last_fullsized":"2024-09-21T14:11:39.741580+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'9","last_scrub_stamp":"2024-09-21T14:09:41.805273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:09:41.805273+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":291,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":872,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741111+0000","last_change":"2024-09-21T14:11:22.288215+0000","last_active":"2024-09-21T14:11:39.741111+0000","last_peered":"2024-09-21T14:11:39.741111+0000","last_clean":"2024-09-21T14:11:39.741111+0000","last_became_active":"2024-09-21T14:11:19.469736+0000","last_became_peered":"2024-09-21T14:11:19.469736+0000","last_unstale":"2024-09-21T14:11:39.741111+0000","last_undegraded":"2024-09-21T14:11:39.741111+0000","last_fullsized":"2024-09-21T14:11:39.741111+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:22.288153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:11:22.288153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:27.949208+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1094,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741079+0000","last_change":"2024-09-21T14:11:32.467937+0000","last_active":"2024-09-21T14:11:39.741079+0000","last_peered":"2024-09-21T14:11:39.741079+0000","last_clean":"2024-09-21T14:11:39.741079+0000","last_became_active":"2024-09-21T14:11:13.449167+0000","last_became_peered":"2024-09-21T14:11:13.449167+0000","last_unstale":"2024-09-21T14:11:39.741079+0000","last_undegraded":"2024-09-21T14:11:39.741079+0000","last_fullsized":"2024-09-21T14:11:39.741079+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:32.467874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:11:32.467874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:49.176031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1139,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.743455+0000","last_change":"2024-09-21T14:11:06.676347+0000","last_active":"2024-09-21T14:11:39.743455+0000","last_peered":"2024-09-21T14:11:39.743455+0000","last_clean":"2024-09-21T14:11:39.743455+0000","last_became_active":"2024-09-21T14:01:41.200360+0000","last_became_peered":"2024-09-21T14:01:41.200360+0000","last_unstale":"2024-09-21T14:11:39.743455+0000","last_undegraded":"2024-09-21T14:11:39.743455+0000","last_fullsized":"2024-09-21T14:11:39.743455+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:06.676300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:11:06.676300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:28.826088+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":994,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.740777+0000","last_change":"2024-09-21T14:11:19.043091+0000","last_active":"2024-09-21T14:11:39.740777+0000","last_peered":"2024-09-21T14:11:39.740777+0000","last_clean":"2024-09-21T14:11:39.740777+0000","last_became_active":"2024-09-21T14:10:49.343587+0000","last_became_peered":"2024-09-21T14:10:49.343587+0000","last_unstale":"2024-09-21T14:11:39.740777+0000","last_undegraded":"2024-09-21T14:11:39.740777+0000","last_fullsized":"2024-09-21T14:11:39.740777+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:19.043035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:11:19.043035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:35.297874+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"516'39","reported_seq":1246,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.528582+0000","last_change":"2024-09-21T14:11:13.487805+0000","last_active":"2024-09-21T14:11:39.528582+0000","last_peered":"2024-09-21T14:11:39.528582+0000","last_clean":"2024-09-21T14:11:39.528582+0000","last_became_active":"2024-09-21T14:11:13.451125+0000","last_became_peered":"2024-09-21T14:11:13.451125+0000","last_unstale":"2024-09-21T14:11:39.528582+0000","last_undegraded":"2024-09-21T14:11:39.528582+0000","last_fullsized":"2024-09-21T14:11:39.528582+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'39","last_scrub_stamp":"2024-09-21T14:11:08.500104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:11:08.500104+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:36.293401+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1128,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.527443+0000","last_change":"2024-09-21T14:11:24.500304+0000","last_active":"2024-09-21T14:11:39.527443+0000","last_peered":"2024-09-21T14:11:39.527443+0000","last_clean":"2024-09-21T14:11:39.527443+0000","last_became_active":"2024-09-21T14:09:00.254258+0000","last_became_peered":"2024-09-21T14:09:00.254258+0000","last_unstale":"2024-09-21T14:11:39.527443+0000","last_undegraded":"2024-09-21T14:11:39.527443+0000","last_fullsized":"2024-09-21T14:11:39.527443+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:24.500257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:11:24.500257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:44.518044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":619,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741279+0000","last_change":"2024-09-21T14:11:11.985555+0000","last_active":"2024-09-21T14:11:39.741279+0000","last_peered":"2024-09-21T14:11:39.741279+0000","last_clean":"2024-09-21T14:11:39.741279+0000","last_became_active":"2024-09-21T14:10:49.341143+0000","last_became_peered":"2024-09-21T14:10:49.341143+0000","last_unstale":"2024-09-21T14:11:39.741279+0000","last_undegraded":"2024-09-21T14:11:39.741279+0000","last_fullsized":"2024-09-21T14:11:39.741279+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:11.985506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:11:11.985506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:31.972909+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":1116,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741717+0000","last_change":"2024-09-21T14:10:56.719940+0000","last_active":"2024-09-21T14:11:39.741717+0000","last_peered":"2024-09-21T14:11:39.741717+0000","last_clean":"2024-09-21T14:11:39.741717+0000","last_became_active":"2024-09-21T13:52:04.316016+0000","last_became_peered":"2024-09-21T13:52:04.316016+0000","last_unstale":"2024-09-21T14:11:39.741717+0000","last_undegraded":"2024-09-21T14:11:39.741717+0000","last_fullsized":"2024-09-21T14:11:39.741717+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:10:56.719874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:10:56.719874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:18.203148+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"516'24","reported_seq":42,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.531855+0000","last_change":"2024-09-21T14:11:13.478263+0000","last_active":"2024-09-21T14:11:39.531855+0000","last_peered":"2024-09-21T14:11:39.531855+0000","last_clean":"2024-09-21T14:11:39.531855+0000","last_became_active":"2024-09-21T14:11:13.450207+0000","last_became_peered":"2024-09-21T14:11:13.450207+0000","last_unstale":"2024-09-21T14:11:39.531855+0000","last_undegraded":"2024-09-21T14:11:39.531855+0000","last_fullsized":"2024-09-21T14:11:39.531855+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'24","last_scrub_stamp":"2024-09-21T14:10:56.585018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:10:56.585018+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:11:58.674191+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1103,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.742213+0000","last_change":"2024-09-21T14:10:08.586191+0000","last_active":"2024-09-21T14:11:39.742213+0000","last_peered":"2024-09-21T14:11:39.742213+0000","last_clean":"2024-09-21T14:11:39.742213+0000","last_became_active":"2024-09-21T13:57:12.681897+0000","last_became_peered":"2024-09-21T13:57:12.681897+0000","last_unstale":"2024-09-21T14:11:39.742213+0000","last_undegraded":"2024-09-21T14:11:39.742213+0000","last_fullsized":"2024-09-21T14:11:39.742213+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:10:08.586123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:10:08.586123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":856,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741918+0000","last_change":"2024-09-21T14:11:19.463135+0000","last_active":"2024-09-21T14:11:39.741918+0000","last_peered":"2024-09-21T14:11:39.741918+0000","last_clean":"2024-09-21T14:11:39.741918+0000","last_became_active":"2024-09-21T14:11:19.462783+0000","last_became_peered":"2024-09-21T14:11:19.462783+0000","last_unstale":"2024-09-21T14:11:39.741918+0000","last_undegraded":"2024-09-21T14:11:39.741918+0000","last_fullsized":"2024-09-21T14:11:39.741918+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T14:10:56.320207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:10:56.320207+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:26.039111+0000","scrub_duration":225,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":1140,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.526470+0000","last_change":"2024-09-21T14:11:06.401536+0000","last_active":"2024-09-21T14:11:39.526470+0000","last_peered":"2024-09-21T14:11:39.526470+0000","last_clean":"2024-09-21T14:11:39.526470+0000","last_became_active":"2024-09-21T14:09:00.257421+0000","last_became_peered":"2024-09-21T14:09:00.257421+0000","last_unstale":"2024-09-21T14:11:39.526470+0000","last_undegraded":"2024-09-21T14:11:39.526470+0000","last_fullsized":"2024-09-21T14:11:39.526470+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T14:11:06.401472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:11:06.401472+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:36.277047+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"516'30","reported_seq":454,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741319+0000","last_change":"2024-09-21T14:11:13.471749+0000","last_active":"2024-09-21T14:11:39.741319+0000","last_peered":"2024-09-21T14:11:39.741319+0000","last_clean":"2024-09-21T14:11:39.741319+0000","last_became_active":"2024-09-21T14:11:13.454129+0000","last_became_peered":"2024-09-21T14:11:13.454129+0000","last_unstale":"2024-09-21T14:11:39.741319+0000","last_undegraded":"2024-09-21T14:11:39.741319+0000","last_fullsized":"2024-09-21T14:11:39.741319+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'30","last_scrub_stamp":"2024-09-21T14:11:02.069275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:11:02.069275+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:13.248489+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"513'14","reported_seq":865,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:43.401128+0000","last_change":"2024-09-21T14:10:50.069522+0000","last_active":"2024-09-21T14:11:43.401128+0000","last_peered":"2024-09-21T14:11:43.401128+0000","last_clean":"2024-09-21T14:11:43.401128+0000","last_became_active":"2024-09-21T14:09:00.258760+0000","last_became_peered":"2024-09-21T14:09:00.258760+0000","last_unstale":"2024-09-21T14:11:43.401128+0000","last_undegraded":"2024-09-21T14:11:43.401128+0000","last_fullsized":"2024-09-21T14:11:43.401128+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"513'14","last_scrub_stamp":"2024-09-21T14:10:50.069455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:10:50.069455+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:17.425683+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"75'79","reported_seq":523,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741743+0000","last_change":"2024-09-21T14:09:32.389941+0000","last_active":"2024-09-21T14:11:39.741743+0000","last_peered":"2024-09-21T14:11:39.741743+0000","last_clean":"2024-09-21T14:11:39.741743+0000","last_became_active":"2024-09-21T14:01:41.246888+0000","last_became_peered":"2024-09-21T14:01:41.246888+0000","last_unstale":"2024-09-21T14:11:39.741743+0000","last_undegraded":"2024-09-21T14:11:39.741743+0000","last_fullsized":"2024-09-21T14:11:39.741743+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:09:32.389738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:09:32.389738+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"516'39","reported_seq":1191,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.527684+0000","last_change":"2024-09-21T14:11:09.466808+0000","last_active":"2024-09-21T14:11:39.527684+0000","last_peered":"2024-09-21T14:11:39.527684+0000","last_clean":"2024-09-21T14:11:39.527684+0000","last_became_active":"2024-09-21T13:57:12.695609+0000","last_became_peered":"2024-09-21T13:57:12.695609+0000","last_unstale":"2024-09-21T14:11:39.527684+0000","last_undegraded":"2024-09-21T14:11:39.527684+0000","last_fullsized":"2024-09-21T14:11:39.527684+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'39","last_scrub_stamp":"2024-09-21T14:11:09.466746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:11:09.466746+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:31.610870+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":30,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.531230+0000","last_change":"2024-09-21T14:11:13.442365+0000","last_active":"2024-09-21T14:11:39.531230+0000","last_peered":"2024-09-21T14:11:39.531230+0000","last_clean":"2024-09-21T14:11:39.531230+0000","last_became_active":"2024-09-21T14:11:13.441990+0000","last_became_peered":"2024-09-21T14:11:13.441990+0000","last_unstale":"2024-09-21T14:11:39.531230+0000","last_undegraded":"2024-09-21T14:11:39.531230+0000","last_fullsized":"2024-09-21T14:11:39.531230+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:10:52.261724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:10:52.261724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:11:52.466748+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"513'12","reported_seq":197,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:43.401085+0000","last_change":"2024-09-21T14:09:35.393938+0000","last_active":"2024-09-21T14:11:43.401085+0000","last_peered":"2024-09-21T14:11:43.401085+0000","last_clean":"2024-09-21T14:11:43.401085+0000","last_became_active":"2024-09-21T14:09:00.253456+0000","last_became_peered":"2024-09-21T14:09:00.253456+0000","last_unstale":"2024-09-21T14:11:43.401085+0000","last_undegraded":"2024-09-21T14:11:43.401085+0000","last_fullsized":"2024-09-21T14:11:43.401085+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"513'12","last_scrub_stamp":"2024-09-21T14:09:35.393859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:09:35.393859+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":989,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741887+0000","last_change":"2024-09-21T14:11:28.536633+0000","last_active":"2024-09-21T14:11:39.741887+0000","last_peered":"2024-09-21T14:11:39.741887+0000","last_clean":"2024-09-21T14:11:39.741887+0000","last_became_active":"2024-09-21T14:09:20.049571+0000","last_became_peered":"2024-09-21T14:09:20.049571+0000","last_unstale":"2024-09-21T14:11:39.741887+0000","last_undegraded":"2024-09-21T14:11:39.741887+0000","last_fullsized":"2024-09-21T14:11:39.741887+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:28.536572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:11:28.536572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:28.914913+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"75'79","reported_seq":585,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.743609+0000","last_change":"2024-09-21T14:10:47.671255+0000","last_active":"2024-09-21T14:11:39.743609+0000","last_peered":"2024-09-21T14:11:39.743609+0000","last_clean":"2024-09-21T14:11:39.743609+0000","last_became_active":"2024-09-21T14:05:39.296209+0000","last_became_peered":"2024-09-21T14:05:39.296209+0000","last_unstale":"2024-09-21T14:11:39.743609+0000","last_undegraded":"2024-09-21T14:11:39.743609+0000","last_fullsized":"2024-09-21T14:11:39.743609+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:10:47.671185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:10:47.671185+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:11:54.870523+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"516'42","reported_seq":56,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.531301+0000","last_change":"2024-09-21T14:11:20.355764+0000","last_active":"2024-09-21T14:11:39.531301+0000","last_peered":"2024-09-21T14:11:39.531301+0000","last_clean":"2024-09-21T14:11:39.531301+0000","last_became_active":"2024-09-21T14:11:13.460234+0000","last_became_peered":"2024-09-21T14:11:13.460234+0000","last_unstale":"2024-09-21T14:11:39.531301+0000","last_undegraded":"2024-09-21T14:11:39.531301+0000","last_fullsized":"2024-09-21T14:11:39.531301+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'42","last_scrub_stamp":"2024-09-21T14:11:20.355687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:11:20.355687+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:39.079841+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":1018,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.532305+0000","last_change":"2024-09-21T14:11:38.291252+0000","last_active":"2024-09-21T14:11:39.532305+0000","last_peered":"2024-09-21T14:11:39.532305+0000","last_clean":"2024-09-21T14:11:39.532305+0000","last_became_active":"2024-09-21T13:59:32.211004+0000","last_became_peered":"2024-09-21T13:59:32.211004+0000","last_unstale":"2024-09-21T14:11:39.532305+0000","last_undegraded":"2024-09-21T14:11:39.532305+0000","last_fullsized":"2024-09-21T14:11:39.532305+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:38.291200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:11:38.291200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:13:07.584626+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1133,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.740941+0000","last_change":"2024-09-21T14:11:13.450533+0000","last_active":"2024-09-21T14:11:39.740941+0000","last_peered":"2024-09-21T14:11:39.740941+0000","last_clean":"2024-09-21T14:11:39.740941+0000","last_became_active":"2024-09-21T14:11:13.450153+0000","last_became_peered":"2024-09-21T14:11:13.450153+0000","last_unstale":"2024-09-21T14:11:39.740941+0000","last_undegraded":"2024-09-21T14:11:39.740941+0000","last_fullsized":"2024-09-21T14:11:39.740941+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:09:43.482393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:09:43.482393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":166,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741087+0000","last_change":"2024-09-21T14:11:13.448191+0000","last_active":"2024-09-21T14:11:39.741087+0000","last_peered":"2024-09-21T14:11:39.741087+0000","last_clean":"2024-09-21T14:11:39.741087+0000","last_became_active":"2024-09-21T14:11:13.447822+0000","last_became_peered":"2024-09-21T14:11:13.447822+0000","last_unstale":"2024-09-21T14:11:39.741087+0000","last_undegraded":"2024-09-21T14:11:39.741087+0000","last_fullsized":"2024-09-21T14:11:39.741087+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:09:34.433449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:09:34.433449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"75'79","reported_seq":30,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.531773+0000","last_change":"2024-09-21T14:11:13.453415+0000","last_active":"2024-09-21T14:11:39.531773+0000","last_peered":"2024-09-21T14:11:39.531773+0000","last_clean":"2024-09-21T14:11:39.531773+0000","last_became_active":"2024-09-21T14:11:13.453029+0000","last_became_peered":"2024-09-21T14:11:13.453029+0000","last_unstale":"2024-09-21T14:11:39.531773+0000","last_undegraded":"2024-09-21T14:11:39.531773+0000","last_fullsized":"2024-09-21T14:11:39.531773+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:11:04.428090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:11:04.428090+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:31.425271+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"516'42","reported_seq":1238,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:48.027870+0000","last_change":"2024-09-21T14:11:48.027870+0000","last_active":"2024-09-21T14:11:48.027870+0000","last_peered":"2024-09-21T14:11:48.027870+0000","last_clean":"2024-09-21T14:11:48.027870+0000","last_became_active":"2024-09-21T14:01:41.200859+0000","last_became_peered":"2024-09-21T14:01:41.200859+0000","last_unstale":"2024-09-21T14:11:48.027870+0000","last_undegraded":"2024-09-21T14:11:48.027870+0000","last_fullsized":"2024-09-21T14:11:48.027870+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'42","last_scrub_stamp":"2024-09-21T14:11:48.027765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:11:48.027765+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:51.074131+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":118,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741144+0000","last_change":"2024-09-21T14:11:14.950193+0000","last_active":"2024-09-21T14:11:39.741144+0000","last_peered":"2024-09-21T14:11:39.741144+0000","last_clean":"2024-09-21T14:11:39.741144+0000","last_became_active":"2024-09-21T14:10:49.346769+0000","last_became_peered":"2024-09-21T14:10:49.346769+0000","last_unstale":"2024-09-21T14:11:39.741144+0000","last_undegraded":"2024-09-21T14:11:39.741144+0000","last_fullsized":"2024-09-21T14:11:39.741144+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T14:11:14.950130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:11:14.950130+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:36.810130+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"571'13","reported_seq":41,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.531244+0000","last_change":"2024-09-21T14:11:13.466255+0000","last_active":"2024-09-21T14:11:39.531244+0000","last_peered":"2024-09-21T14:11:39.531244+0000","last_clean":"2024-09-21T14:11:39.531244+0000","last_became_active":"2024-09-21T14:11:13.450354+0000","last_became_peered":"2024-09-21T14:11:13.450354+0000","last_unstale":"2024-09-21T14:11:39.531244+0000","last_undegraded":"2024-09-21T14:11:39.531244+0000","last_fullsized":"2024-09-21T14:11:39.531244+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"519'12","last_scrub_stamp":"2024-09-21T14:10:59.305542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:10:59.305542+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:04.230445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":890,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741736+0000","last_change":"2024-09-21T14:11:19.465850+0000","last_active":"2024-09-21T14:11:39.741736+0000","last_peered":"2024-09-21T14:11:39.741736+0000","last_clean":"2024-09-21T14:11:39.741736+0000","last_became_active":"2024-09-21T14:11:19.465511+0000","last_became_peered":"2024-09-21T14:11:19.465511+0000","last_unstale":"2024-09-21T14:11:39.741736+0000","last_undegraded":"2024-09-21T14:11:39.741736+0000","last_fullsized":"2024-09-21T14:11:39.741736+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:01.098264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:11:01.098264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:25.240453+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"75'79","reported_seq":1172,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.743404+0000","last_change":"2024-09-21T14:11:13.449007+0000","last_active":"2024-09-21T14:11:39.743404+0000","last_peered":"2024-09-21T14:11:39.743404+0000","last_clean":"2024-09-21T14:11:39.743404+0000","last_became_active":"2024-09-21T14:11:13.448515+0000","last_became_peered":"2024-09-21T14:11:13.448515+0000","last_unstale":"2024-09-21T14:11:39.743404+0000","last_undegraded":"2024-09-21T14:11:39.743404+0000","last_fullsized":"2024-09-21T14:11:39.743404+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:10:42.074254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:10:42.074254+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:08.096912+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"516'30","reported_seq":456,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.526823+0000","last_change":"2024-09-21T14:11:14.666842+0000","last_active":"2024-09-21T14:11:39.526823+0000","last_peered":"2024-09-21T14:11:39.526823+0000","last_clean":"2024-09-21T14:11:39.526823+0000","last_became_active":"2024-09-21T14:03:29.158218+0000","last_became_peered":"2024-09-21T14:03:29.158218+0000","last_unstale":"2024-09-21T14:11:39.526823+0000","last_undegraded":"2024-09-21T14:11:39.526823+0000","last_fullsized":"2024-09-21T14:11:39.526823+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'30","last_scrub_stamp":"2024-09-21T14:11:14.666787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:11:14.666787+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:39.337317+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"61'1","reported_seq":1067,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.742005+0000","last_change":"2024-09-21T14:11:13.465249+0000","last_active":"2024-09-21T14:11:39.742005+0000","last_peered":"2024-09-21T14:11:39.742005+0000","last_clean":"2024-09-21T14:11:39.742005+0000","last_became_active":"2024-09-21T14:11:13.448485+0000","last_became_peered":"2024-09-21T14:11:13.448485+0000","last_unstale":"2024-09-21T14:11:39.742005+0000","last_undegraded":"2024-09-21T14:11:39.742005+0000","last_fullsized":"2024-09-21T14:11:39.742005+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T14:10:48.487905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:10:48.487905+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:09.644047+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"558'18","reported_seq":477,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:43.401104+0000","last_change":"2024-09-21T14:11:38.141037+0000","last_active":"2024-09-21T14:11:43.401104+0000","last_peered":"2024-09-21T14:11:43.401104+0000","last_clean":"2024-09-21T14:11:43.401104+0000","last_became_active":"2024-09-21T14:10:49.340748+0000","last_became_peered":"2024-09-21T14:10:49.340748+0000","last_unstale":"2024-09-21T14:11:43.401104+0000","last_undegraded":"2024-09-21T14:11:43.401104+0000","last_fullsized":"2024-09-21T14:11:43.401104+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"558'18","last_scrub_stamp":"2024-09-21T14:11:38.140984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:11:38.140984+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:41.730924+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":639,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.740580+0000","last_change":"2024-09-21T14:11:00.127972+0000","last_active":"2024-09-21T14:11:39.740580+0000","last_peered":"2024-09-21T14:11:39.740580+0000","last_clean":"2024-09-21T14:11:39.740580+0000","last_became_active":"2024-09-21T14:10:49.339816+0000","last_became_peered":"2024-09-21T14:10:49.339816+0000","last_unstale":"2024-09-21T14:11:39.740580+0000","last_undegraded":"2024-09-21T14:11:39.740580+0000","last_fullsized":"2024-09-21T14:11:39.740580+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:00.127907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:11:00.127907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:16.581999+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"75'79","reported_seq":1188,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.742541+0000","last_change":"2024-09-21T14:11:15.309594+0000","last_active":"2024-09-21T14:11:39.742541+0000","last_peered":"2024-09-21T14:11:39.742541+0000","last_clean":"2024-09-21T14:11:39.742541+0000","last_became_active":"2024-09-21T14:11:13.448110+0000","last_became_peered":"2024-09-21T14:11:13.448110+0000","last_unstale":"2024-09-21T14:11:39.742541+0000","last_undegraded":"2024-09-21T14:11:39.742541+0000","last_fullsized":"2024-09-21T14:11:39.742541+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:11:15.309533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:11:15.309533+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:42.160279+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"516'39","reported_seq":829,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741067+0000","last_change":"2024-09-21T14:11:17.042351+0000","last_active":"2024-09-21T14:11:39.741067+0000","last_peered":"2024-09-21T14:11:39.741067+0000","last_clean":"2024-09-21T14:11:39.741067+0000","last_became_active":"2024-09-21T14:10:49.344815+0000","last_became_peered":"2024-09-21T14:10:49.344815+0000","last_unstale":"2024-09-21T14:11:39.741067+0000","last_undegraded":"2024-09-21T14:11:39.741067+0000","last_fullsized":"2024-09-21T14:11:39.741067+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'39","last_scrub_stamp":"2024-09-21T14:11:17.042282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:11:17.042282+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:36.887641+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":67,"num_read_kb":53,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1113,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.527208+0000","last_change":"2024-09-21T14:11:03.365061+0000","last_active":"2024-09-21T14:11:39.527208+0000","last_peered":"2024-09-21T14:11:39.527208+0000","last_clean":"2024-09-21T14:11:39.527208+0000","last_became_active":"2024-09-21T14:01:41.185400+0000","last_became_peered":"2024-09-21T14:01:41.185400+0000","last_unstale":"2024-09-21T14:11:39.527208+0000","last_undegraded":"2024-09-21T14:11:39.527208+0000","last_fullsized":"2024-09-21T14:11:39.527208+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:03.365015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:11:03.365015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:27.060566+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":697,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.742638+0000","last_change":"2024-09-21T14:11:34.130231+0000","last_active":"2024-09-21T14:11:39.742638+0000","last_peered":"2024-09-21T14:11:39.742638+0000","last_clean":"2024-09-21T14:11:39.742638+0000","last_became_active":"2024-09-21T14:01:41.193973+0000","last_became_peered":"2024-09-21T14:01:41.193973+0000","last_unstale":"2024-09-21T14:11:39.742638+0000","last_undegraded":"2024-09-21T14:11:39.742638+0000","last_fullsized":"2024-09-21T14:11:39.742638+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:34.130174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:11:34.130174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:56.256298+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1104,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.526325+0000","last_change":"2024-09-21T14:11:33.578125+0000","last_active":"2024-09-21T14:11:39.526325+0000","last_peered":"2024-09-21T14:11:39.526325+0000","last_clean":"2024-09-21T14:11:39.526325+0000","last_became_active":"2024-09-21T13:58:17.757628+0000","last_became_peered":"2024-09-21T13:58:17.757628+0000","last_unstale":"2024-09-21T14:11:39.526325+0000","last_undegraded":"2024-09-21T14:11:39.526325+0000","last_fullsized":"2024-09-21T14:11:39.526325+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:33.578075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:11:33.578075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:35.700951+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"75'79","reported_seq":528,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741138+0000","last_change":"2024-09-21T14:11:07.480703+0000","last_active":"2024-09-21T14:11:39.741138+0000","last_peered":"2024-09-21T14:11:39.741138+0000","last_clean":"2024-09-21T14:11:39.741138+0000","last_became_active":"2024-09-21T14:05:39.310615+0000","last_became_peered":"2024-09-21T14:05:39.310615+0000","last_unstale":"2024-09-21T14:11:39.741138+0000","last_undegraded":"2024-09-21T14:11:39.741138+0000","last_fullsized":"2024-09-21T14:11:39.741138+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:11:07.480617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:11:07.480617+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:35.036408+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"571'59","reported_seq":1583,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.527111+0000","last_change":"2024-09-21T14:11:16.530926+0000","last_active":"2024-09-21T14:11:39.527111+0000","last_peered":"2024-09-21T14:11:39.527111+0000","last_clean":"2024-09-21T14:11:39.527111+0000","last_became_active":"2024-09-21T14:11:13.444928+0000","last_became_peered":"2024-09-21T14:11:13.444928+0000","last_unstale":"2024-09-21T14:11:39.527111+0000","last_undegraded":"2024-09-21T14:11:39.527111+0000","last_fullsized":"2024-09-21T14:11:39.527111+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"571'59","last_scrub_stamp":"2024-09-21T14:11:16.530870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:11:16.530870+0000","objects_scrubbed":10,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:28.890899+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":84,"num_read_kb":69,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1128,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.742088+0000","last_change":"2024-09-21T14:09:38.447915+0000","last_active":"2024-09-21T14:11:39.742088+0000","last_peered":"2024-09-21T14:11:39.742088+0000","last_clean":"2024-09-21T14:11:39.742088+0000","last_became_active":"2024-09-21T13:58:17.770937+0000","last_became_peered":"2024-09-21T13:58:17.770937+0000","last_unstale":"2024-09-21T14:11:39.742088+0000","last_undegraded":"2024-09-21T14:11:39.742088+0000","last_fullsized":"2024-09-21T14:11:39.742088+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:09:38.447835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:09:38.447835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"63'2","reported_seq":1064,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741639+0000","last_change":"2024-09-21T14:09:50.282494+0000","last_active":"2024-09-21T14:11:39.741639+0000","last_peered":"2024-09-21T14:11:39.741639+0000","last_clean":"2024-09-21T14:11:39.741639+0000","last_became_active":"2024-09-21T14:01:41.247642+0000","last_became_peered":"2024-09-21T14:01:41.247642+0000","last_unstale":"2024-09-21T14:11:39.741639+0000","last_undegraded":"2024-09-21T14:11:39.741639+0000","last_fullsized":"2024-09-21T14:11:39.741639+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-21T14:09:50.282440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T14:09:50.282440+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":986,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.740913+0000","last_change":"2024-09-21T14:11:12.991300+0000","last_active":"2024-09-21T14:11:39.740913+0000","last_peered":"2024-09-21T14:11:39.740913+0000","last_clean":"2024-09-21T14:11:39.740913+0000","last_became_active":"2024-09-21T14:10:49.341777+0000","last_became_peered":"2024-09-21T14:10:49.341777+0000","last_unstale":"2024-09-21T14:11:39.740913+0000","last_undegraded":"2024-09-21T14:11:39.740913+0000","last_fullsized":"2024-09-21T14:11:39.740913+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:12.991242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:11:12.991242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:20.335492+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":38,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.532032+0000","last_change":"2024-09-21T14:11:26.409856+0000","last_active":"2024-09-21T14:11:39.532032+0000","last_peered":"2024-09-21T14:11:39.532032+0000","last_clean":"2024-09-21T14:11:39.532032+0000","last_became_active":"2024-09-21T14:11:13.454970+0000","last_became_peered":"2024-09-21T14:11:13.454970+0000","last_unstale":"2024-09-21T14:11:39.532032+0000","last_undegraded":"2024-09-21T14:11:39.532032+0000","last_fullsized":"2024-09-21T14:11:39.532032+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:26.409797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:11:26.409797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:32.125500+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"75'79","reported_seq":1001,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:46.208177+0000","last_change":"2024-09-21T14:11:46.208177+0000","last_active":"2024-09-21T14:11:46.208177+0000","last_peered":"2024-09-21T14:11:46.208177+0000","last_clean":"2024-09-21T14:11:46.208177+0000","last_became_active":"2024-09-21T14:11:19.470686+0000","last_became_peered":"2024-09-21T14:11:19.470686+0000","last_unstale":"2024-09-21T14:11:46.208177+0000","last_undegraded":"2024-09-21T14:11:46.208177+0000","last_fullsized":"2024-09-21T14:11:46.208177+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:11:46.208113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:11:46.208113+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:13:16.060204+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"516'21","reported_seq":1183,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.527669+0000","last_change":"2024-09-21T14:10:51.251070+0000","last_active":"2024-09-21T14:11:39.527669+0000","last_peered":"2024-09-21T14:11:39.527669+0000","last_clean":"2024-09-21T14:11:39.527669+0000","last_became_active":"2024-09-21T14:09:00.258874+0000","last_became_peered":"2024-09-21T14:09:00.258874+0000","last_unstale":"2024-09-21T14:11:39.527669+0000","last_undegraded":"2024-09-21T14:11:39.527669+0000","last_fullsized":"2024-09-21T14:11:39.527669+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'21","last_scrub_stamp":"2024-09-21T14:10:51.251013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:10:51.251013+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:06.205275+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1157,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.526213+0000","last_change":"2024-09-21T14:11:21.516006+0000","last_active":"2024-09-21T14:11:39.526213+0000","last_peered":"2024-09-21T14:11:39.526213+0000","last_clean":"2024-09-21T14:11:39.526213+0000","last_became_active":"2024-09-21T13:58:17.759671+0000","last_became_peered":"2024-09-21T13:58:17.759671+0000","last_unstale":"2024-09-21T14:11:39.526213+0000","last_undegraded":"2024-09-21T14:11:39.526213+0000","last_fullsized":"2024-09-21T14:11:39.526213+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:21.515959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:11:21.515959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:42.463169+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"61'1","reported_seq":1052,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.527923+0000","last_change":"2024-09-21T14:11:13.479751+0000","last_active":"2024-09-21T14:11:39.527923+0000","last_peered":"2024-09-21T14:11:39.527923+0000","last_clean":"2024-09-21T14:11:39.527923+0000","last_became_active":"2024-09-21T14:11:13.452505+0000","last_became_peered":"2024-09-21T14:11:13.452505+0000","last_unstale":"2024-09-21T14:11:39.527923+0000","last_undegraded":"2024-09-21T14:11:39.527923+0000","last_fullsized":"2024-09-21T14:11:39.527923+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T14:11:04.629569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T14:11:04.629569+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:24.582268+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1025,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.527634+0000","last_change":"2024-09-21T14:10:43.473799+0000","last_active":"2024-09-21T14:11:39.527634+0000","last_peered":"2024-09-21T14:11:39.527634+0000","last_clean":"2024-09-21T14:11:39.527634+0000","last_became_active":"2024-09-21T14:01:41.182000+0000","last_became_peered":"2024-09-21T14:01:41.182000+0000","last_unstale":"2024-09-21T14:11:39.527634+0000","last_undegraded":"2024-09-21T14:11:39.527634+0000","last_fullsized":"2024-09-21T14:11:39.527634+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:10:43.473741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:10:43.473741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:11:58.368712+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":906,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.742661+0000","last_change":"2024-09-21T14:11:21.252559+0000","last_active":"2024-09-21T14:11:39.742661+0000","last_peered":"2024-09-21T14:11:39.742661+0000","last_clean":"2024-09-21T14:11:39.742661+0000","last_became_active":"2024-09-21T14:11:19.465954+0000","last_became_peered":"2024-09-21T14:11:19.465954+0000","last_unstale":"2024-09-21T14:11:39.742661+0000","last_undegraded":"2024-09-21T14:11:39.742661+0000","last_fullsized":"2024-09-21T14:11:39.742661+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:21.252502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:11:21.252502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:25.626368+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"75'79","reported_seq":599,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.527323+0000","last_change":"2024-09-21T14:11:37.678851+0000","last_active":"2024-09-21T14:11:39.527323+0000","last_peered":"2024-09-21T14:11:39.527323+0000","last_clean":"2024-09-21T14:11:39.527323+0000","last_became_active":"2024-09-21T14:11:05.584920+0000","last_became_peered":"2024-09-21T14:11:05.584920+0000","last_unstale":"2024-09-21T14:11:39.527323+0000","last_undegraded":"2024-09-21T14:11:39.527323+0000","last_fullsized":"2024-09-21T14:11:39.527323+0000","mapping_epoch":564,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":565,"parent":"0.0","parent_split_bits":3,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:11:37.678757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:11:37.678757+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:40.667696+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"516'36","reported_seq":120,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.740983+0000","last_change":"2024-09-21T14:11:30.078177+0000","last_active":"2024-09-21T14:11:39.740983+0000","last_peered":"2024-09-21T14:11:39.740983+0000","last_clean":"2024-09-21T14:11:39.740983+0000","last_became_active":"2024-09-21T14:10:49.340725+0000","last_became_peered":"2024-09-21T14:10:49.340725+0000","last_unstale":"2024-09-21T14:11:39.740983+0000","last_undegraded":"2024-09-21T14:11:39.740983+0000","last_fullsized":"2024-09-21T14:11:39.740983+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'36","last_scrub_stamp":"2024-09-21T14:11:30.078110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:11:30.078110+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:42.054806+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":46,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":650,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.740680+0000","last_change":"2024-09-21T14:11:18.056925+0000","last_active":"2024-09-21T14:11:39.740680+0000","last_peered":"2024-09-21T14:11:39.740680+0000","last_clean":"2024-09-21T14:11:39.740680+0000","last_became_active":"2024-09-21T14:11:13.455641+0000","last_became_peered":"2024-09-21T14:11:13.455641+0000","last_unstale":"2024-09-21T14:11:39.740680+0000","last_undegraded":"2024-09-21T14:11:39.740680+0000","last_fullsized":"2024-09-21T14:11:39.740680+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:18.056877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:11:18.056877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:20.873051+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":1970,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:44.502191+0000","last_change":"2024-09-21T14:11:19.465191+0000","last_active":"2024-09-21T14:11:44.502191+0000","last_peered":"2024-09-21T14:11:44.502191+0000","last_clean":"2024-09-21T14:11:44.502191+0000","last_became_active":"2024-09-21T14:11:19.464763+0000","last_became_peered":"2024-09-21T14:11:19.464763+0000","last_unstale":"2024-09-21T14:11:44.502191+0000","last_undegraded":"2024-09-21T14:11:44.502191+0000","last_fullsized":"2024-09-21T14:11:44.502191+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-21T14:11:14.522856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:56.029323+0000","last_clean_scrub_stamp":"2024-09-21T14:11:14.522856+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:15.654784+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0.00047048000000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":541,"num_read_kb":536,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1043,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.743779+0000","last_change":"2024-09-21T14:11:11.566821+0000","last_active":"2024-09-21T14:11:39.743779+0000","last_peered":"2024-09-21T14:11:39.743779+0000","last_clean":"2024-09-21T14:11:39.743779+0000","last_became_active":"2024-09-21T14:01:41.200531+0000","last_became_peered":"2024-09-21T14:01:41.200531+0000","last_unstale":"2024-09-21T14:11:39.743779+0000","last_undegraded":"2024-09-21T14:11:39.743779+0000","last_fullsized":"2024-09-21T14:11:39.743779+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:11.566762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:11:11.566762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:37.548378+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1155,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.526827+0000","last_change":"2024-09-21T14:11:30.633960+0000","last_active":"2024-09-21T14:11:39.526827+0000","last_peered":"2024-09-21T14:11:39.526827+0000","last_clean":"2024-09-21T14:11:39.526827+0000","last_became_active":"2024-09-21T14:11:13.442994+0000","last_became_peered":"2024-09-21T14:11:13.442994+0000","last_unstale":"2024-09-21T14:11:39.526827+0000","last_undegraded":"2024-09-21T14:11:39.526827+0000","last_fullsized":"2024-09-21T14:11:39.526827+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:30.633910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:11:30.633910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:47.538708+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"516'44","reported_seq":906,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:43.401384+0000","last_change":"2024-09-21T14:11:07.149781+0000","last_active":"2024-09-21T14:11:43.401384+0000","last_peered":"2024-09-21T14:11:43.401384+0000","last_clean":"2024-09-21T14:11:43.401384+0000","last_became_active":"2024-09-21T13:59:32.204624+0000","last_became_peered":"2024-09-21T13:59:32.204624+0000","last_unstale":"2024-09-21T14:11:43.401384+0000","last_undegraded":"2024-09-21T14:11:43.401384+0000","last_fullsized":"2024-09-21T14:11:43.401384+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'44","last_scrub_stamp":"2024-09-21T14:11:07.149727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:11:07.149727+0000","objects_scrubbed":8,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:09.590664+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":38,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.531383+0000","last_change":"2024-09-21T14:11:37.271458+0000","last_active":"2024-09-21T14:11:39.531383+0000","last_peered":"2024-09-21T14:11:39.531383+0000","last_clean":"2024-09-21T14:11:39.531383+0000","last_became_active":"2024-09-21T14:11:13.451947+0000","last_became_peered":"2024-09-21T14:11:13.451947+0000","last_unstale":"2024-09-21T14:11:39.531383+0000","last_undegraded":"2024-09-21T14:11:39.531383+0000","last_fullsized":"2024-09-21T14:11:39.531383+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:37.271400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:11:37.271400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:39.415667+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"75'79","reported_seq":996,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.742538+0000","last_change":"2024-09-21T14:11:31.223417+0000","last_active":"2024-09-21T14:11:39.742538+0000","last_peered":"2024-09-21T14:11:39.742538+0000","last_clean":"2024-09-21T14:11:39.742538+0000","last_became_active":"2024-09-21T14:11:19.466477+0000","last_became_peered":"2024-09-21T14:11:19.466477+0000","last_unstale":"2024-09-21T14:11:39.742538+0000","last_undegraded":"2024-09-21T14:11:39.742538+0000","last_fullsized":"2024-09-21T14:11:39.742538+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:11:31.223349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:11:31.223349+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:35.391856+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":878,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.742021+0000","last_change":"2024-09-21T14:11:19.463022+0000","last_active":"2024-09-21T14:11:39.742021+0000","last_peered":"2024-09-21T14:11:39.742021+0000","last_clean":"2024-09-21T14:11:39.742021+0000","last_became_active":"2024-09-21T14:11:19.462693+0000","last_became_peered":"2024-09-21T14:11:19.462693+0000","last_unstale":"2024-09-21T14:11:39.742021+0000","last_undegraded":"2024-09-21T14:11:39.742021+0000","last_fullsized":"2024-09-21T14:11:39.742021+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:10:46.493469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:10:46.493469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:11:56.122037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":992,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.743513+0000","last_change":"2024-09-21T14:09:37.303313+0000","last_active":"2024-09-21T14:11:39.743513+0000","last_peered":"2024-09-21T14:11:39.743513+0000","last_clean":"2024-09-21T14:11:39.743513+0000","last_became_active":"2024-09-21T14:03:57.346453+0000","last_became_peered":"2024-09-21T14:03:57.346453+0000","last_unstale":"2024-09-21T14:11:39.743513+0000","last_undegraded":"2024-09-21T14:11:39.743513+0000","last_fullsized":"2024-09-21T14:11:39.743513+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:09:37.303243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:09:37.303243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"75'79","reported_seq":589,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.527169+0000","last_change":"2024-09-21T14:11:17.490554+0000","last_active":"2024-09-21T14:11:39.527169+0000","last_peered":"2024-09-21T14:11:39.527169+0000","last_clean":"2024-09-21T14:11:39.527169+0000","last_became_active":"2024-09-21T14:05:41.345392+0000","last_became_peered":"2024-09-21T14:05:41.345392+0000","last_unstale":"2024-09-21T14:11:39.527169+0000","last_undegraded":"2024-09-21T14:11:39.527169+0000","last_fullsized":"2024-09-21T14:11:39.527169+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:11:17.490501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:11:17.490501+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:47.055039+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"516'27","reported_seq":1186,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.743546+0000","last_change":"2024-09-21T14:11:10.614517+0000","last_active":"2024-09-21T14:11:39.743546+0000","last_peered":"2024-09-21T14:11:39.743546+0000","last_clean":"2024-09-21T14:11:39.743546+0000","last_became_active":"2024-09-21T13:58:17.776310+0000","last_became_peered":"2024-09-21T13:58:17.776310+0000","last_unstale":"2024-09-21T14:11:39.743546+0000","last_undegraded":"2024-09-21T14:11:39.743546+0000","last_fullsized":"2024-09-21T14:11:39.743546+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'27","last_scrub_stamp":"2024-09-21T14:11:10.614438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:11:10.614438+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:24.560178+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1144,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:42.910606+0000","last_change":"2024-09-21T14:11:42.910606+0000","last_active":"2024-09-21T14:11:42.910606+0000","last_peered":"2024-09-21T14:11:42.910606+0000","last_clean":"2024-09-21T14:11:42.910606+0000","last_became_active":"2024-09-21T13:59:32.223247+0000","last_became_peered":"2024-09-21T13:59:32.223247+0000","last_unstale":"2024-09-21T14:11:42.910606+0000","last_undegraded":"2024-09-21T14:11:42.910606+0000","last_fullsized":"2024-09-21T14:11:42.910606+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:42.910548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:11:42.910548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:13:00.872306+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1069,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741987+0000","last_change":"2024-09-21T14:11:06.461105+0000","last_active":"2024-09-21T14:11:39.741987+0000","last_peered":"2024-09-21T14:11:39.741987+0000","last_clean":"2024-09-21T14:11:39.741987+0000","last_became_active":"2024-09-21T14:01:41.193941+0000","last_became_peered":"2024-09-21T14:01:41.193941+0000","last_unstale":"2024-09-21T14:11:39.741987+0000","last_undegraded":"2024-09-21T14:11:39.741987+0000","last_fullsized":"2024-09-21T14:11:39.741987+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:06.461058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:11:06.461058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:22.790875+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":38,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.531790+0000","last_change":"2024-09-21T14:11:16.412877+0000","last_active":"2024-09-21T14:11:39.531790+0000","last_peered":"2024-09-21T14:11:39.531790+0000","last_clean":"2024-09-21T14:11:39.531790+0000","last_became_active":"2024-09-21T14:11:13.452915+0000","last_became_peered":"2024-09-21T14:11:13.452915+0000","last_unstale":"2024-09-21T14:11:39.531790+0000","last_undegraded":"2024-09-21T14:11:39.531790+0000","last_fullsized":"2024-09-21T14:11:39.531790+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:16.412831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:11:16.412831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:35.952542+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"75'79","reported_seq":560,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741458+0000","last_change":"2024-09-21T14:11:13.474582+0000","last_active":"2024-09-21T14:11:39.741458+0000","last_peered":"2024-09-21T14:11:39.741458+0000","last_clean":"2024-09-21T14:11:39.741458+0000","last_became_active":"2024-09-21T14:11:13.450234+0000","last_became_peered":"2024-09-21T14:11:13.450234+0000","last_unstale":"2024-09-21T14:11:39.741458+0000","last_undegraded":"2024-09-21T14:11:39.741458+0000","last_fullsized":"2024-09-21T14:11:39.741458+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:09:44.277011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:09:44.277011+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"516'45","reported_seq":198,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.743559+0000","last_change":"2024-09-21T14:10:06.596332+0000","last_active":"2024-09-21T14:11:39.743559+0000","last_peered":"2024-09-21T14:11:39.743559+0000","last_clean":"2024-09-21T14:11:39.743559+0000","last_became_active":"2024-09-21T14:09:00.257088+0000","last_became_peered":"2024-09-21T14:09:00.257088+0000","last_unstale":"2024-09-21T14:11:39.743559+0000","last_undegraded":"2024-09-21T14:11:39.743559+0000","last_fullsized":"2024-09-21T14:11:39.743559+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'45","last_scrub_stamp":"2024-09-21T14:10:06.596239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:10:06.596239+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":60,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":670,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.742379+0000","last_change":"2024-09-21T14:11:00.134030+0000","last_active":"2024-09-21T14:11:39.742379+0000","last_peered":"2024-09-21T14:11:39.742379+0000","last_clean":"2024-09-21T14:11:39.742379+0000","last_became_active":"2024-09-21T13:58:17.762520+0000","last_became_peered":"2024-09-21T13:58:17.762520+0000","last_unstale":"2024-09-21T14:11:39.742379+0000","last_undegraded":"2024-09-21T14:11:39.742379+0000","last_fullsized":"2024-09-21T14:11:39.742379+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:00.133978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:11:00.133978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:08.169028+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1083,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.526919+0000","last_change":"2024-09-21T14:10:42.270730+0000","last_active":"2024-09-21T14:11:39.526919+0000","last_peered":"2024-09-21T14:11:39.526919+0000","last_clean":"2024-09-21T14:11:39.526919+0000","last_became_active":"2024-09-21T13:49:02.067472+0000","last_became_peered":"2024-09-21T13:49:02.067472+0000","last_unstale":"2024-09-21T14:11:39.526919+0000","last_undegraded":"2024-09-21T14:11:39.526919+0000","last_fullsized":"2024-09-21T14:11:39.526919+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:10:42.270672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:10:42.270672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:01.425062+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":987,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.527901+0000","last_change":"2024-09-21T14:11:26.669787+0000","last_active":"2024-09-21T14:11:39.527901+0000","last_peered":"2024-09-21T14:11:39.527901+0000","last_clean":"2024-09-21T14:11:39.527901+0000","last_became_active":"2024-09-21T14:11:13.448947+0000","last_became_peered":"2024-09-21T14:11:13.448947+0000","last_unstale":"2024-09-21T14:11:39.527901+0000","last_undegraded":"2024-09-21T14:11:39.527901+0000","last_fullsized":"2024-09-21T14:11:39.527901+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:26.669736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:11:26.669736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:39.580627+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"75'79","reported_seq":768,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.527656+0000","last_change":"2024-09-21T14:10:40.398254+0000","last_active":"2024-09-21T14:11:39.527656+0000","last_peered":"2024-09-21T14:11:39.527656+0000","last_clean":"2024-09-21T14:11:39.527656+0000","last_became_active":"2024-09-21T14:08:47.765045+0000","last_became_peered":"2024-09-21T14:08:47.765045+0000","last_unstale":"2024-09-21T14:11:39.527656+0000","last_undegraded":"2024-09-21T14:11:39.527656+0000","last_fullsized":"2024-09-21T14:11:39.527656+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:10:40.398204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:10:40.398204+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:05.552007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"516'30","reported_seq":1196,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.742298+0000","last_change":"2024-09-21T14:10:55.718375+0000","last_active":"2024-09-21T14:11:39.742298+0000","last_peered":"2024-09-21T14:11:39.742298+0000","last_clean":"2024-09-21T14:11:39.742298+0000","last_became_active":"2024-09-21T13:58:17.765427+0000","last_became_peered":"2024-09-21T13:58:17.765427+0000","last_unstale":"2024-09-21T14:11:39.742298+0000","last_undegraded":"2024-09-21T14:11:39.742298+0000","last_fullsized":"2024-09-21T14:11:39.742298+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'30","last_scrub_stamp":"2024-09-21T14:10:55.718305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:10:55.718305+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:07.425610+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1052,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.531594+0000","last_change":"2024-09-21T14:11:05.426846+0000","last_active":"2024-09-21T14:11:39.531594+0000","last_peered":"2024-09-21T14:11:39.531594+0000","last_clean":"2024-09-21T14:11:39.531594+0000","last_became_active":"2024-09-21T14:09:00.258570+0000","last_became_peered":"2024-09-21T14:09:00.258570+0000","last_unstale":"2024-09-21T14:11:39.531594+0000","last_undegraded":"2024-09-21T14:11:39.531594+0000","last_fullsized":"2024-09-21T14:11:39.531594+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:05.426790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:11:05.426790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:17.540835+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"346'14","reported_seq":1268,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.528339+0000","last_change":"2024-09-21T14:11:01.591298+0000","last_active":"2024-09-21T14:11:39.528339+0000","last_peered":"2024-09-21T14:11:39.528339+0000","last_clean":"2024-09-21T14:11:39.528339+0000","last_became_active":"2024-09-21T14:01:41.186564+0000","last_became_peered":"2024-09-21T14:01:41.186564+0000","last_unstale":"2024-09-21T14:11:39.528339+0000","last_undegraded":"2024-09-21T14:11:39.528339+0000","last_fullsized":"2024-09-21T14:11:39.528339+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'14","last_scrub_stamp":"2024-09-21T14:11:01.591239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:11:01.591239+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:07.167679+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"68'1","reported_seq":1109,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.743268+0000","last_change":"2024-09-21T14:11:22.706336+0000","last_active":"2024-09-21T14:11:39.743268+0000","last_peered":"2024-09-21T14:11:39.743268+0000","last_clean":"2024-09-21T14:11:39.743268+0000","last_became_active":"2024-09-21T13:52:04.313210+0000","last_became_peered":"2024-09-21T13:52:04.313210+0000","last_unstale":"2024-09-21T14:11:39.743268+0000","last_undegraded":"2024-09-21T14:11:39.743268+0000","last_fullsized":"2024-09-21T14:11:39.743268+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-09-21T14:11:22.706276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:11:22.706276+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:43.511209+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.e","version":"75'79","reported_seq":160,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.743174+0000","last_change":"2024-09-21T14:10:46.639536+0000","last_active":"2024-09-21T14:11:39.743174+0000","last_peered":"2024-09-21T14:11:39.743174+0000","last_clean":"2024-09-21T14:11:39.743174+0000","last_became_active":"2024-09-21T14:09:00.255631+0000","last_became_peered":"2024-09-21T14:09:00.255631+0000","last_unstale":"2024-09-21T14:11:39.743174+0000","last_undegraded":"2024-09-21T14:11:39.743174+0000","last_fullsized":"2024-09-21T14:11:39.743174+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:10:46.639473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:10:46.639473+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:11:59.281908+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"516'21","reported_seq":1092,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:45.946964+0000","last_change":"2024-09-21T14:11:45.946964+0000","last_active":"2024-09-21T14:11:45.946964+0000","last_peered":"2024-09-21T14:11:45.946964+0000","last_clean":"2024-09-21T14:11:45.946964+0000","last_became_active":"2024-09-21T14:10:49.345321+0000","last_became_peered":"2024-09-21T14:10:49.345321+0000","last_unstale":"2024-09-21T14:11:45.946964+0000","last_undegraded":"2024-09-21T14:11:45.946964+0000","last_fullsized":"2024-09-21T14:11:45.946964+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'21","last_scrub_stamp":"2024-09-21T14:11:45.946874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:11:45.946874+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:13:04.361140+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1079,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.742050+0000","last_change":"2024-09-21T14:11:25.524630+0000","last_active":"2024-09-21T14:11:39.742050+0000","last_peered":"2024-09-21T14:11:39.742050+0000","last_clean":"2024-09-21T14:11:39.742050+0000","last_became_active":"2024-09-21T14:11:13.448302+0000","last_became_peered":"2024-09-21T14:11:13.448302+0000","last_unstale":"2024-09-21T14:11:39.742050+0000","last_undegraded":"2024-09-21T14:11:39.742050+0000","last_fullsized":"2024-09-21T14:11:39.742050+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:25.524544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:11:25.524544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:44.752546+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":959,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.527047+0000","last_change":"2024-09-21T14:11:13.626721+0000","last_active":"2024-09-21T14:11:39.527047+0000","last_peered":"2024-09-21T14:11:39.527047+0000","last_clean":"2024-09-21T14:11:39.527047+0000","last_became_active":"2024-09-21T14:11:13.449586+0000","last_became_peered":"2024-09-21T14:11:13.449586+0000","last_unstale":"2024-09-21T14:11:39.527047+0000","last_undegraded":"2024-09-21T14:11:39.527047+0000","last_fullsized":"2024-09-21T14:11:39.527047+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:13.626673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:11:13.626673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:21.448316+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":746,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:46.957850+0000","last_change":"2024-09-21T14:11:46.957850+0000","last_active":"2024-09-21T14:11:46.957850+0000","last_peered":"2024-09-21T14:11:46.957850+0000","last_clean":"2024-09-21T14:11:46.957850+0000","last_became_active":"2024-09-21T14:10:49.343216+0000","last_became_peered":"2024-09-21T14:10:49.343216+0000","last_unstale":"2024-09-21T14:11:46.957850+0000","last_undegraded":"2024-09-21T14:11:46.957850+0000","last_fullsized":"2024-09-21T14:11:46.957850+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:46.957802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:11:46.957802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:53.586957+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"75'79","reported_seq":1171,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.743636+0000","last_change":"2024-09-21T14:10:46.039287+0000","last_active":"2024-09-21T14:11:39.743636+0000","last_peered":"2024-09-21T14:11:39.743636+0000","last_clean":"2024-09-21T14:11:39.743636+0000","last_became_active":"2024-09-21T14:05:41.327757+0000","last_became_peered":"2024-09-21T14:05:41.327757+0000","last_unstale":"2024-09-21T14:11:39.743636+0000","last_undegraded":"2024-09-21T14:11:39.743636+0000","last_fullsized":"2024-09-21T14:11:39.743636+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:10:46.039232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:10:46.039232+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:11:51.759284+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"516'51","reported_seq":768,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.742531+0000","last_change":"2024-09-21T14:11:30.158765+0000","last_active":"2024-09-21T14:11:39.742531+0000","last_peered":"2024-09-21T14:11:39.742531+0000","last_clean":"2024-09-21T14:11:39.742531+0000","last_became_active":"2024-09-21T13:59:32.204184+0000","last_became_peered":"2024-09-21T13:59:32.204184+0000","last_unstale":"2024-09-21T14:11:39.742531+0000","last_undegraded":"2024-09-21T14:11:39.742531+0000","last_fullsized":"2024-09-21T14:11:39.742531+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'51","last_scrub_stamp":"2024-09-21T14:11:30.158653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:11:30.158653+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:46.448132+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":928,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.743054+0000","last_change":"2024-09-21T14:11:38.301955+0000","last_active":"2024-09-21T14:11:39.743054+0000","last_peered":"2024-09-21T14:11:39.743054+0000","last_clean":"2024-09-21T14:11:39.743054+0000","last_became_active":"2024-09-21T14:11:19.472878+0000","last_became_peered":"2024-09-21T14:11:19.472878+0000","last_unstale":"2024-09-21T14:11:39.743054+0000","last_undegraded":"2024-09-21T14:11:39.743054+0000","last_fullsized":"2024-09-21T14:11:39.743054+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:38.301896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:11:38.301896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:51.654993+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1075,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.527286+0000","last_change":"2024-09-21T14:11:29.721058+0000","last_active":"2024-09-21T14:11:39.527286+0000","last_peered":"2024-09-21T14:11:39.527286+0000","last_clean":"2024-09-21T14:11:39.527286+0000","last_became_active":"2024-09-21T14:01:41.181911+0000","last_became_peered":"2024-09-21T14:01:41.181911+0000","last_unstale":"2024-09-21T14:11:39.527286+0000","last_undegraded":"2024-09-21T14:11:39.527286+0000","last_fullsized":"2024-09-21T14:11:39.527286+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:29.721000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:11:29.721000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:44.023515+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":855,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.742477+0000","last_change":"2024-09-21T14:11:34.247767+0000","last_active":"2024-09-21T14:11:39.742477+0000","last_peered":"2024-09-21T14:11:39.742477+0000","last_clean":"2024-09-21T14:11:39.742477+0000","last_became_active":"2024-09-21T14:11:19.463664+0000","last_became_peered":"2024-09-21T14:11:19.463664+0000","last_unstale":"2024-09-21T14:11:39.742477+0000","last_undegraded":"2024-09-21T14:11:39.742477+0000","last_fullsized":"2024-09-21T14:11:39.742477+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:34.247706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:11:34.247706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:36.622023+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"75'79","reported_seq":34,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.531943+0000","last_change":"2024-09-21T14:11:13.472293+0000","last_active":"2024-09-21T14:11:39.531943+0000","last_peered":"2024-09-21T14:11:39.531943+0000","last_clean":"2024-09-21T14:11:39.531943+0000","last_became_active":"2024-09-21T14:11:13.460428+0000","last_became_peered":"2024-09-21T14:11:13.460428+0000","last_unstale":"2024-09-21T14:11:39.531943+0000","last_undegraded":"2024-09-21T14:11:39.531943+0000","last_fullsized":"2024-09-21T14:11:39.531943+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:10:40.067459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:10:40.067459+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:05.510502+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"516'24","reported_seq":1200,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741242+0000","last_change":"2024-09-21T14:11:13.476010+0000","last_active":"2024-09-21T14:11:39.741242+0000","last_peered":"2024-09-21T14:11:39.741242+0000","last_clean":"2024-09-21T14:11:39.741242+0000","last_became_active":"2024-09-21T14:11:13.450947+0000","last_became_peered":"2024-09-21T14:11:13.450947+0000","last_unstale":"2024-09-21T14:11:39.741242+0000","last_undegraded":"2024-09-21T14:11:39.741242+0000","last_fullsized":"2024-09-21T14:11:39.741242+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'24","last_scrub_stamp":"2024-09-21T14:09:49.533719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:09:49.533719+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":882,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:45.190139+0000","last_change":"2024-09-21T14:11:45.190139+0000","last_active":"2024-09-21T14:11:45.190139+0000","last_peered":"2024-09-21T14:11:45.190139+0000","last_clean":"2024-09-21T14:11:45.190139+0000","last_became_active":"2024-09-21T14:11:19.469222+0000","last_became_peered":"2024-09-21T14:11:19.469222+0000","last_unstale":"2024-09-21T14:11:45.190139+0000","last_undegraded":"2024-09-21T14:11:45.190139+0000","last_fullsized":"2024-09-21T14:11:45.190139+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:45.190081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:11:45.190081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:49.941162+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1030,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.526650+0000","last_change":"2024-09-21T14:11:11.576821+0000","last_active":"2024-09-21T14:11:39.526650+0000","last_peered":"2024-09-21T14:11:39.526650+0000","last_clean":"2024-09-21T14:11:39.526650+0000","last_became_active":"2024-09-21T13:52:04.322147+0000","last_became_peered":"2024-09-21T13:52:04.322147+0000","last_unstale":"2024-09-21T14:11:39.526650+0000","last_undegraded":"2024-09-21T14:11:39.526650+0000","last_fullsized":"2024-09-21T14:11:39.526650+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:11.576777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:11:11.576777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:21.835297+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1053,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.531787+0000","last_change":"2024-09-21T14:11:30.321924+0000","last_active":"2024-09-21T14:11:39.531787+0000","last_peered":"2024-09-21T14:11:39.531787+0000","last_clean":"2024-09-21T14:11:39.531787+0000","last_became_active":"2024-09-21T14:05:30.243052+0000","last_became_peered":"2024-09-21T14:05:30.243052+0000","last_unstale":"2024-09-21T14:11:39.531787+0000","last_undegraded":"2024-09-21T14:11:39.531787+0000","last_fullsized":"2024-09-21T14:11:39.531787+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:30.321877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:11:30.321877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:45.647118+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"75'79","reported_seq":144,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.742524+0000","last_change":"2024-09-21T14:11:19.471949+0000","last_active":"2024-09-21T14:11:39.742524+0000","last_peered":"2024-09-21T14:11:39.742524+0000","last_clean":"2024-09-21T14:11:39.742524+0000","last_became_active":"2024-09-21T14:11:19.470642+0000","last_became_peered":"2024-09-21T14:11:19.470642+0000","last_unstale":"2024-09-21T14:11:39.742524+0000","last_undegraded":"2024-09-21T14:11:39.742524+0000","last_fullsized":"2024-09-21T14:11:39.742524+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":4,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:10:52.639044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:10:52.639044+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:14.834448+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"516'39","reported_seq":1175,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741610+0000","last_change":"2024-09-21T14:11:10.540588+0000","last_active":"2024-09-21T14:11:39.741610+0000","last_peered":"2024-09-21T14:11:39.741610+0000","last_clean":"2024-09-21T14:11:39.741610+0000","last_became_active":"2024-09-21T14:01:41.192481+0000","last_became_peered":"2024-09-21T14:01:41.192481+0000","last_unstale":"2024-09-21T14:11:39.741610+0000","last_undegraded":"2024-09-21T14:11:39.741610+0000","last_fullsized":"2024-09-21T14:11:39.741610+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'39","last_scrub_stamp":"2024-09-21T14:11:10.540491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:11:10.540491+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:14.680730+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":860,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.743213+0000","last_change":"2024-09-21T14:11:19.470913+0000","last_active":"2024-09-21T14:11:39.743213+0000","last_peered":"2024-09-21T14:11:39.743213+0000","last_clean":"2024-09-21T14:11:39.743213+0000","last_became_active":"2024-09-21T14:11:19.470526+0000","last_became_peered":"2024-09-21T14:11:19.470526+0000","last_unstale":"2024-09-21T14:11:39.743213+0000","last_undegraded":"2024-09-21T14:11:39.743213+0000","last_fullsized":"2024-09-21T14:11:39.743213+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2024-09-21T14:10:22.619411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:10:22.619411+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"575'16","reported_seq":1121,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.742423+0000","last_change":"2024-09-21T14:11:19.470104+0000","last_active":"2024-09-21T14:11:39.742423+0000","last_peered":"2024-09-21T14:11:39.742423+0000","last_clean":"2024-09-21T14:11:39.742423+0000","last_became_active":"2024-09-21T14:11:19.469532+0000","last_became_peered":"2024-09-21T14:11:19.469532+0000","last_unstale":"2024-09-21T14:11:39.742423+0000","last_undegraded":"2024-09-21T14:11:39.742423+0000","last_fullsized":"2024-09-21T14:11:39.742423+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":0,"last_scrub":"492'15","last_scrub_stamp":"2024-09-21T14:10:47.022190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:10:47.022190+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:09.968975+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1105,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741467+0000","last_change":"2024-09-21T14:11:33.434354+0000","last_active":"2024-09-21T14:11:39.741467+0000","last_peered":"2024-09-21T14:11:39.741467+0000","last_clean":"2024-09-21T14:11:39.741467+0000","last_became_active":"2024-09-21T14:01:41.246106+0000","last_became_peered":"2024-09-21T14:01:41.246106+0000","last_unstale":"2024-09-21T14:11:39.741467+0000","last_undegraded":"2024-09-21T14:11:39.741467+0000","last_fullsized":"2024-09-21T14:11:39.741467+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:33.434297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:11:33.434297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:41.150173+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"75'79","reported_seq":319,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741315+0000","last_change":"2024-09-21T14:10:51.421789+0000","last_active":"2024-09-21T14:11:39.741315+0000","last_peered":"2024-09-21T14:11:39.741315+0000","last_clean":"2024-09-21T14:11:39.741315+0000","last_became_active":"2024-09-21T14:05:41.352416+0000","last_became_peered":"2024-09-21T14:05:41.352416+0000","last_unstale":"2024-09-21T14:11:39.741315+0000","last_undegraded":"2024-09-21T14:11:39.741315+0000","last_fullsized":"2024-09-21T14:11:39.741315+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":5,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:10:51.421721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:10:51.421721+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:02.587500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":693,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.743268+0000","last_change":"2024-09-21T14:11:28.188525+0000","last_active":"2024-09-21T14:11:39.743268+0000","last_peered":"2024-09-21T14:11:39.743268+0000","last_clean":"2024-09-21T14:11:39.743268+0000","last_became_active":"2024-09-21T14:09:00.257598+0000","last_became_peered":"2024-09-21T14:09:00.257598+0000","last_unstale":"2024-09-21T14:11:39.743268+0000","last_undegraded":"2024-09-21T14:11:39.743268+0000","last_fullsized":"2024-09-21T14:11:39.743268+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:28.188466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:11:28.188466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:45.114740+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"516'18","reported_seq":46,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.532215+0000","last_change":"2024-09-21T14:11:39.272172+0000","last_active":"2024-09-21T14:11:39.532215+0000","last_peered":"2024-09-21T14:11:39.532215+0000","last_clean":"2024-09-21T14:11:39.532215+0000","last_became_active":"2024-09-21T14:11:13.439951+0000","last_became_peered":"2024-09-21T14:11:13.439951+0000","last_unstale":"2024-09-21T14:11:39.532215+0000","last_undegraded":"2024-09-21T14:11:39.532215+0000","last_fullsized":"2024-09-21T14:11:39.532215+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'18","last_scrub_stamp":"2024-09-21T14:11:39.272111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:11:39.272111+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:43.321429+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":22,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1028,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:41.357429+0000","last_change":"2024-09-21T14:11:41.357429+0000","last_active":"2024-09-21T14:11:41.357429+0000","last_peered":"2024-09-21T14:11:41.357429+0000","last_clean":"2024-09-21T14:11:41.357429+0000","last_became_active":"2024-09-21T14:07:13.246170+0000","last_became_peered":"2024-09-21T14:07:13.246170+0000","last_unstale":"2024-09-21T14:11:41.357429+0000","last_undegraded":"2024-09-21T14:11:41.357429+0000","last_fullsized":"2024-09-21T14:11:41.357429+0000","mapping_epoch":471,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:41.357369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:11:41.357369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:13:03.023343+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":885,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.742935+0000","last_change":"2024-09-21T14:11:19.472529+0000","last_active":"2024-09-21T14:11:39.742935+0000","last_peered":"2024-09-21T14:11:39.742935+0000","last_clean":"2024-09-21T14:11:39.742935+0000","last_became_active":"2024-09-21T14:11:19.471238+0000","last_became_peered":"2024-09-21T14:11:19.471238+0000","last_unstale":"2024-09-21T14:11:39.742935+0000","last_undegraded":"2024-09-21T14:11:39.742935+0000","last_fullsized":"2024-09-21T14:11:39.742935+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:10:51.064772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:10:51.064772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:11:51.528077+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"75'79","reported_seq":610,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.526878+0000","last_change":"2024-09-21T14:11:27.578700+0000","last_active":"2024-09-21T14:11:39.526878+0000","last_peered":"2024-09-21T14:11:39.526878+0000","last_clean":"2024-09-21T14:11:39.526878+0000","last_became_active":"2024-09-21T14:11:13.445287+0000","last_became_peered":"2024-09-21T14:11:13.445287+0000","last_unstale":"2024-09-21T14:11:39.526878+0000","last_undegraded":"2024-09-21T14:11:39.526878+0000","last_fullsized":"2024-09-21T14:11:39.526878+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":5,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:11:27.578640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:11:27.578640+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:48.755021+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"516'21","reported_seq":133,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741240+0000","last_change":"2024-09-21T14:11:03.050702+0000","last_active":"2024-09-21T14:11:39.741240+0000","last_peered":"2024-09-21T14:11:39.741240+0000","last_clean":"2024-09-21T14:11:39.741240+0000","last_became_active":"2024-09-21T14:10:49.341923+0000","last_became_peered":"2024-09-21T14:10:49.341923+0000","last_unstale":"2024-09-21T14:11:39.741240+0000","last_undegraded":"2024-09-21T14:11:39.741240+0000","last_fullsized":"2024-09-21T14:11:39.741240+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'21","last_scrub_stamp":"2024-09-21T14:11:03.050650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:11:03.050650+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:08.199738+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"61'1","reported_seq":911,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.742834+0000","last_change":"2024-09-21T14:11:19.470234+0000","last_active":"2024-09-21T14:11:39.742834+0000","last_peered":"2024-09-21T14:11:39.742834+0000","last_clean":"2024-09-21T14:11:39.742834+0000","last_became_active":"2024-09-21T14:11:19.469709+0000","last_became_peered":"2024-09-21T14:11:19.469709+0000","last_unstale":"2024-09-21T14:11:39.742834+0000","last_undegraded":"2024-09-21T14:11:39.742834+0000","last_fullsized":"2024-09-21T14:11:39.742834+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-21T14:10:53.656323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:10:53.656323+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:16.023272+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":30,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.532267+0000","last_change":"2024-09-21T14:11:13.443970+0000","last_active":"2024-09-21T14:11:39.532267+0000","last_peered":"2024-09-21T14:11:39.532267+0000","last_clean":"2024-09-21T14:11:39.532267+0000","last_became_active":"2024-09-21T14:11:13.443647+0000","last_became_peered":"2024-09-21T14:11:13.443647+0000","last_unstale":"2024-09-21T14:11:39.532267+0000","last_undegraded":"2024-09-21T14:11:39.532267+0000","last_fullsized":"2024-09-21T14:11:39.532267+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:01.302748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:11:01.302748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:09.696317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1071,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.528321+0000","last_change":"2024-09-21T14:10:50.471544+0000","last_active":"2024-09-21T14:11:39.528321+0000","last_peered":"2024-09-21T14:11:39.528321+0000","last_clean":"2024-09-21T14:11:39.528321+0000","last_became_active":"2024-09-21T13:59:32.205803+0000","last_became_peered":"2024-09-21T13:59:32.205803+0000","last_unstale":"2024-09-21T14:11:39.528321+0000","last_undegraded":"2024-09-21T14:11:39.528321+0000","last_fullsized":"2024-09-21T14:11:39.528321+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:10:50.471483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:10:50.471483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:15.819295+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"75'79","reported_seq":311,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.740826+0000","last_change":"2024-09-21T14:09:42.199233+0000","last_active":"2024-09-21T14:11:39.740826+0000","last_peered":"2024-09-21T14:11:39.740826+0000","last_clean":"2024-09-21T14:11:39.740826+0000","last_became_active":"2024-09-21T14:05:41.325618+0000","last_became_peered":"2024-09-21T14:05:41.325618+0000","last_unstale":"2024-09-21T14:11:39.740826+0000","last_undegraded":"2024-09-21T14:11:39.740826+0000","last_fullsized":"2024-09-21T14:11:39.740826+0000","mapping_epoch":438,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":5,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:09:42.199185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:09:42.199185+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"516'21","reported_seq":1067,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.743794+0000","last_change":"2024-09-21T14:10:42.557080+0000","last_active":"2024-09-21T14:11:39.743794+0000","last_peered":"2024-09-21T14:11:39.743794+0000","last_clean":"2024-09-21T14:11:39.743794+0000","last_became_active":"2024-09-21T14:01:41.187904+0000","last_became_peered":"2024-09-21T14:01:41.187904+0000","last_unstale":"2024-09-21T14:11:39.743794+0000","last_undegraded":"2024-09-21T14:11:39.743794+0000","last_fullsized":"2024-09-21T14:11:39.743794+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'21","last_scrub_stamp":"2024-09-21T14:10:42.557023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:10:42.557023+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:10.798670+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":672,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.740754+0000","last_change":"2024-09-21T14:11:33.019928+0000","last_active":"2024-09-21T14:11:39.740754+0000","last_peered":"2024-09-21T14:11:39.740754+0000","last_clean":"2024-09-21T14:11:39.740754+0000","last_became_active":"2024-09-21T14:11:13.451798+0000","last_became_peered":"2024-09-21T14:11:13.451798+0000","last_unstale":"2024-09-21T14:11:39.740754+0000","last_undegraded":"2024-09-21T14:11:39.740754+0000","last_fullsized":"2024-09-21T14:11:39.740754+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:33.019878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:11:33.019878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:54.945357+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"571'16","reported_seq":41,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.532041+0000","last_change":"2024-09-21T14:11:13.476028+0000","last_active":"2024-09-21T14:11:39.532041+0000","last_peered":"2024-09-21T14:11:39.532041+0000","last_clean":"2024-09-21T14:11:39.532041+0000","last_became_active":"2024-09-21T14:11:13.462346+0000","last_became_peered":"2024-09-21T14:11:13.462346+0000","last_unstale":"2024-09-21T14:11:39.532041+0000","last_undegraded":"2024-09-21T14:11:39.532041+0000","last_fullsized":"2024-09-21T14:11:39.532041+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"498'15","last_scrub_stamp":"2024-09-21T14:11:09.539231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:11:09.539231+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:16.052628+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1123,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.742859+0000","last_change":"2024-09-21T14:11:12.676444+0000","last_active":"2024-09-21T14:11:39.742859+0000","last_peered":"2024-09-21T14:11:39.742859+0000","last_clean":"2024-09-21T14:11:39.742859+0000","last_became_active":"2024-09-21T14:01:41.200113+0000","last_became_peered":"2024-09-21T14:01:41.200113+0000","last_unstale":"2024-09-21T14:11:39.742859+0000","last_undegraded":"2024-09-21T14:11:39.742859+0000","last_fullsized":"2024-09-21T14:11:39.742859+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:12.676373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:11:12.676373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:29.745437+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"75'79","reported_seq":129,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.740515+0000","last_change":"2024-09-21T14:11:28.099284+0000","last_active":"2024-09-21T14:11:39.740515+0000","last_peered":"2024-09-21T14:11:39.740515+0000","last_clean":"2024-09-21T14:11:39.740515+0000","last_became_active":"2024-09-21T14:11:13.451849+0000","last_became_peered":"2024-09-21T14:11:13.451849+0000","last_unstale":"2024-09-21T14:11:39.740515+0000","last_undegraded":"2024-09-21T14:11:39.740515+0000","last_fullsized":"2024-09-21T14:11:39.740515+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":5,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:11:28.099231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:11:28.099231+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:52.970057+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"516'30","reported_seq":1087,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.532170+0000","last_change":"2024-09-21T14:11:12.354250+0000","last_active":"2024-09-21T14:11:39.532170+0000","last_peered":"2024-09-21T14:11:39.532170+0000","last_clean":"2024-09-21T14:11:39.532170+0000","last_became_active":"2024-09-21T13:59:32.208332+0000","last_became_peered":"2024-09-21T13:59:32.208332+0000","last_unstale":"2024-09-21T14:11:39.532170+0000","last_undegraded":"2024-09-21T14:11:39.532170+0000","last_fullsized":"2024-09-21T14:11:39.532170+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'30","last_scrub_stamp":"2024-09-21T14:11:12.354195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:11:12.354195+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:20.890317+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":879,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741599+0000","last_change":"2024-09-21T14:11:19.471330+0000","last_active":"2024-09-21T14:11:39.741599+0000","last_peered":"2024-09-21T14:11:39.741599+0000","last_clean":"2024-09-21T14:11:39.741599+0000","last_became_active":"2024-09-21T14:11:19.471040+0000","last_became_peered":"2024-09-21T14:11:19.471040+0000","last_unstale":"2024-09-21T14:11:39.741599+0000","last_undegraded":"2024-09-21T14:11:39.741599+0000","last_fullsized":"2024-09-21T14:11:39.741599+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:10:19.686004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:10:19.686004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"346'10","reported_seq":1284,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.743745+0000","last_change":"2024-09-21T14:11:02.349223+0000","last_active":"2024-09-21T14:11:39.743745+0000","last_peered":"2024-09-21T14:11:39.743745+0000","last_clean":"2024-09-21T14:11:39.743745+0000","last_became_active":"2024-09-21T14:01:41.206303+0000","last_became_peered":"2024-09-21T14:01:41.206303+0000","last_unstale":"2024-09-21T14:11:39.743745+0000","last_undegraded":"2024-09-21T14:11:39.743745+0000","last_fullsized":"2024-09-21T14:11:39.743745+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'10","last_scrub_stamp":"2024-09-21T14:11:02.349171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:11:02.349171+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:23.300972+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":716,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741179+0000","last_change":"2024-09-21T14:11:13.453684+0000","last_active":"2024-09-21T14:11:39.741179+0000","last_peered":"2024-09-21T14:11:39.741179+0000","last_clean":"2024-09-21T14:11:39.741179+0000","last_became_active":"2024-09-21T14:11:13.453350+0000","last_became_peered":"2024-09-21T14:11:13.453350+0000","last_unstale":"2024-09-21T14:11:39.741179+0000","last_undegraded":"2024-09-21T14:11:39.741179+0000","last_fullsized":"2024-09-21T14:11:39.741179+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:07.043612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:11:07.043612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:28.590410+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"516'27","reported_seq":1190,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741814+0000","last_change":"2024-09-21T14:11:24.722760+0000","last_active":"2024-09-21T14:11:39.741814+0000","last_peered":"2024-09-21T14:11:39.741814+0000","last_clean":"2024-09-21T14:11:39.741814+0000","last_became_active":"2024-09-21T13:58:17.771497+0000","last_became_peered":"2024-09-21T13:58:17.771497+0000","last_unstale":"2024-09-21T14:11:39.741814+0000","last_undegraded":"2024-09-21T14:11:39.741814+0000","last_fullsized":"2024-09-21T14:11:39.741814+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'27","last_scrub_stamp":"2024-09-21T14:11:24.722644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:11:24.722644+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:54.057781+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":513,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.742108+0000","last_change":"2024-09-21T14:11:19.466314+0000","last_active":"2024-09-21T14:11:39.742108+0000","last_peered":"2024-09-21T14:11:39.742108+0000","last_clean":"2024-09-21T14:11:39.742108+0000","last_became_active":"2024-09-21T14:11:19.465996+0000","last_became_peered":"2024-09-21T14:11:19.465996+0000","last_unstale":"2024-09-21T14:11:39.742108+0000","last_undegraded":"2024-09-21T14:11:39.742108+0000","last_fullsized":"2024-09-21T14:11:39.742108+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:10:52.042228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:10:52.042228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:21.748017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1150,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.526511+0000","last_change":"2024-09-21T14:11:35.617965+0000","last_active":"2024-09-21T14:11:39.526511+0000","last_peered":"2024-09-21T14:11:39.526511+0000","last_clean":"2024-09-21T14:11:39.526511+0000","last_became_active":"2024-09-21T14:11:13.441313+0000","last_became_peered":"2024-09-21T14:11:13.441313+0000","last_unstale":"2024-09-21T14:11:39.526511+0000","last_undegraded":"2024-09-21T14:11:39.526511+0000","last_fullsized":"2024-09-21T14:11:39.526511+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:35.617914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:11:35.617914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:35.925993+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":38,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.531386+0000","last_change":"2024-09-21T14:11:24.414702+0000","last_active":"2024-09-21T14:11:39.531386+0000","last_peered":"2024-09-21T14:11:39.531386+0000","last_clean":"2024-09-21T14:11:39.531386+0000","last_became_active":"2024-09-21T14:11:13.459760+0000","last_became_peered":"2024-09-21T14:11:13.459760+0000","last_unstale":"2024-09-21T14:11:39.531386+0000","last_undegraded":"2024-09-21T14:11:39.531386+0000","last_fullsized":"2024-09-21T14:11:39.531386+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:24.414653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:11:24.414653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:50.482709+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"516'21","reported_seq":1197,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.526621+0000","last_change":"2024-09-21T14:10:47.307322+0000","last_active":"2024-09-21T14:11:39.526621+0000","last_peered":"2024-09-21T14:11:39.526621+0000","last_clean":"2024-09-21T14:11:39.526621+0000","last_became_active":"2024-09-21T13:58:17.764120+0000","last_became_peered":"2024-09-21T13:58:17.764120+0000","last_unstale":"2024-09-21T14:11:39.526621+0000","last_undegraded":"2024-09-21T14:11:39.526621+0000","last_fullsized":"2024-09-21T14:11:39.526621+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'21","last_scrub_stamp":"2024-09-21T14:10:47.307251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:10:47.307251+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:13.879687+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":899,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.742761+0000","last_change":"2024-09-21T14:11:29.244714+0000","last_active":"2024-09-21T14:11:39.742761+0000","last_peered":"2024-09-21T14:11:39.742761+0000","last_clean":"2024-09-21T14:11:39.742761+0000","last_became_active":"2024-09-21T14:11:19.470391+0000","last_became_peered":"2024-09-21T14:11:19.470391+0000","last_unstale":"2024-09-21T14:11:39.742761+0000","last_undegraded":"2024-09-21T14:11:39.742761+0000","last_fullsized":"2024-09-21T14:11:39.742761+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:29.244663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:11:29.244663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:54.090608+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1012,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:44.390424+0000","last_change":"2024-09-21T14:11:44.390424+0000","last_active":"2024-09-21T14:11:44.390424+0000","last_peered":"2024-09-21T14:11:44.390424+0000","last_clean":"2024-09-21T14:11:44.390424+0000","last_became_active":"2024-09-21T13:57:12.703441+0000","last_became_peered":"2024-09-21T13:57:12.703441+0000","last_unstale":"2024-09-21T14:11:44.390424+0000","last_undegraded":"2024-09-21T14:11:44.390424+0000","last_fullsized":"2024-09-21T14:11:44.390424+0000","mapping_epoch":240,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":241,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:44.390375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:11:44.390375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:13:03.876615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":722,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.740504+0000","last_change":"2024-09-21T14:11:26.075362+0000","last_active":"2024-09-21T14:11:39.740504+0000","last_peered":"2024-09-21T14:11:39.740504+0000","last_clean":"2024-09-21T14:11:39.740504+0000","last_became_active":"2024-09-21T14:10:49.340390+0000","last_became_peered":"2024-09-21T14:10:49.340390+0000","last_unstale":"2024-09-21T14:11:39.740504+0000","last_undegraded":"2024-09-21T14:11:39.740504+0000","last_fullsized":"2024-09-21T14:11:39.740504+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:26.075312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:11:26.075312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:41.431629+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"75'79","reported_seq":336,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.528263+0000","last_change":"2024-09-21T14:11:21.565075+0000","last_active":"2024-09-21T14:11:39.528263+0000","last_peered":"2024-09-21T14:11:39.528263+0000","last_clean":"2024-09-21T14:11:39.528263+0000","last_became_active":"2024-09-21T14:09:00.260220+0000","last_became_peered":"2024-09-21T14:09:00.260220+0000","last_unstale":"2024-09-21T14:11:39.528263+0000","last_undegraded":"2024-09-21T14:11:39.528263+0000","last_fullsized":"2024-09-21T14:11:39.528263+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":436,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":5,"last_scrub":"75'79","last_scrub_stamp":"2024-09-21T14:11:21.565024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:46:21.921969+0000","last_clean_scrub_stamp":"2024-09-21T14:11:21.565024+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:45.505021+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"516'27","reported_seq":1160,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.531777+0000","last_change":"2024-09-21T14:10:53.415737+0000","last_active":"2024-09-21T14:11:39.531777+0000","last_peered":"2024-09-21T14:11:39.531777+0000","last_clean":"2024-09-21T14:11:39.531777+0000","last_became_active":"2024-09-21T13:59:32.231567+0000","last_became_peered":"2024-09-21T13:59:32.231567+0000","last_unstale":"2024-09-21T14:11:39.531777+0000","last_undegraded":"2024-09-21T14:11:39.531777+0000","last_fullsized":"2024-09-21T14:11:39.531777+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'27","last_scrub_stamp":"2024-09-21T14:10:53.415682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:10:53.415682+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:11:57.472353+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1026,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741076+0000","last_change":"2024-09-21T14:10:59.326076+0000","last_active":"2024-09-21T14:11:39.741076+0000","last_peered":"2024-09-21T14:11:39.741076+0000","last_clean":"2024-09-21T14:11:39.741076+0000","last_became_active":"2024-09-21T14:01:41.190760+0000","last_became_peered":"2024-09-21T14:01:41.190760+0000","last_unstale":"2024-09-21T14:11:39.741076+0000","last_undegraded":"2024-09-21T14:11:39.741076+0000","last_fullsized":"2024-09-21T14:11:39.741076+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:10:59.326031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:10:59.326031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:27.455413+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":905,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:40.266721+0000","last_change":"2024-09-21T14:11:40.266721+0000","last_active":"2024-09-21T14:11:40.266721+0000","last_peered":"2024-09-21T14:11:40.266721+0000","last_clean":"2024-09-21T14:11:40.266721+0000","last_became_active":"2024-09-21T14:11:19.468570+0000","last_became_peered":"2024-09-21T14:11:19.468570+0000","last_unstale":"2024-09-21T14:11:40.266721+0000","last_undegraded":"2024-09-21T14:11:40.266721+0000","last_fullsized":"2024-09-21T14:11:40.266721+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:40.266656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:11:40.266656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:13:03.164277+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":38,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.531570+0000","last_change":"2024-09-21T14:11:25.438297+0000","last_active":"2024-09-21T14:11:39.531570+0000","last_peered":"2024-09-21T14:11:39.531570+0000","last_clean":"2024-09-21T14:11:39.531570+0000","last_became_active":"2024-09-21T14:11:13.455917+0000","last_became_peered":"2024-09-21T14:11:13.455917+0000","last_unstale":"2024-09-21T14:11:39.531570+0000","last_undegraded":"2024-09-21T14:11:39.531570+0000","last_fullsized":"2024-09-21T14:11:39.531570+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:25.438247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:11:25.438247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:35.273105+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"516'12","reported_seq":1105,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.531417+0000","last_change":"2024-09-21T14:10:43.495107+0000","last_active":"2024-09-21T14:11:39.531417+0000","last_peered":"2024-09-21T14:11:39.531417+0000","last_clean":"2024-09-21T14:11:39.531417+0000","last_became_active":"2024-09-21T14:09:31.261004+0000","last_became_peered":"2024-09-21T14:09:31.261004+0000","last_unstale":"2024-09-21T14:11:39.531417+0000","last_undegraded":"2024-09-21T14:11:39.531417+0000","last_fullsized":"2024-09-21T14:11:39.531417+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'12","last_scrub_stamp":"2024-09-21T14:10:43.495045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:10:43.495045+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:02.257308+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":823,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741385+0000","last_change":"2024-09-21T14:11:21.093192+0000","last_active":"2024-09-21T14:11:39.741385+0000","last_peered":"2024-09-21T14:11:39.741385+0000","last_clean":"2024-09-21T14:11:39.741385+0000","last_became_active":"2024-09-21T14:11:13.453444+0000","last_became_peered":"2024-09-21T14:11:13.453444+0000","last_unstale":"2024-09-21T14:11:39.741385+0000","last_undegraded":"2024-09-21T14:11:39.741385+0000","last_fullsized":"2024-09-21T14:11:39.741385+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:21.093138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:11:21.093138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:29.250375+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":702,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741501+0000","last_change":"2024-09-21T14:11:29.175124+0000","last_active":"2024-09-21T14:11:39.741501+0000","last_peered":"2024-09-21T14:11:39.741501+0000","last_clean":"2024-09-21T14:11:39.741501+0000","last_became_active":"2024-09-21T14:09:00.257432+0000","last_became_peered":"2024-09-21T14:09:00.257432+0000","last_unstale":"2024-09-21T14:11:39.741501+0000","last_undegraded":"2024-09-21T14:11:39.741501+0000","last_fullsized":"2024-09-21T14:11:39.741501+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:29.175063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:11:29.175063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:29.902783+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":830,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741195+0000","last_change":"2024-09-21T14:11:19.470112+0000","last_active":"2024-09-21T14:11:39.741195+0000","last_peered":"2024-09-21T14:11:39.741195+0000","last_clean":"2024-09-21T14:11:39.741195+0000","last_became_active":"2024-09-21T14:11:19.469802+0000","last_became_peered":"2024-09-21T14:11:19.469802+0000","last_unstale":"2024-09-21T14:11:39.741195+0000","last_undegraded":"2024-09-21T14:11:39.741195+0000","last_fullsized":"2024-09-21T14:11:39.741195+0000","mapping_epoch":574,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":575,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:10:15.727357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:10:15.727357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1105,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.527075+0000","last_change":"2024-09-21T14:10:43.235227+0000","last_active":"2024-09-21T14:11:39.527075+0000","last_peered":"2024-09-21T14:11:39.527075+0000","last_clean":"2024-09-21T14:11:39.527075+0000","last_became_active":"2024-09-21T14:01:41.187438+0000","last_became_peered":"2024-09-21T14:01:41.187438+0000","last_unstale":"2024-09-21T14:11:39.527075+0000","last_undegraded":"2024-09-21T14:11:39.527075+0000","last_fullsized":"2024-09-21T14:11:39.527075+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:10:43.235173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:03.054313+0000","last_clean_scrub_stamp":"2024-09-21T14:10:43.235173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:10.679058+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":38,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.531454+0000","last_change":"2024-09-21T14:11:33.274104+0000","last_active":"2024-09-21T14:11:39.531454+0000","last_peered":"2024-09-21T14:11:39.531454+0000","last_clean":"2024-09-21T14:11:39.531454+0000","last_became_active":"2024-09-21T14:11:13.460267+0000","last_became_peered":"2024-09-21T14:11:13.460267+0000","last_unstale":"2024-09-21T14:11:39.531454+0000","last_undegraded":"2024-09-21T14:11:39.531454+0000","last_fullsized":"2024-09-21T14:11:39.531454+0000","mapping_epoch":570,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":571,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:11:33.274053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:57.035903+0000","last_clean_scrub_stamp":"2024-09-21T14:11:33.274053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:44.042337+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"516'27","reported_seq":733,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.743736+0000","last_change":"2024-09-21T14:11:32.143120+0000","last_active":"2024-09-21T14:11:39.743736+0000","last_peered":"2024-09-21T14:11:39.743736+0000","last_clean":"2024-09-21T14:11:39.743736+0000","last_became_active":"2024-09-21T14:01:41.194606+0000","last_became_peered":"2024-09-21T14:01:41.194606+0000","last_unstale":"2024-09-21T14:11:39.743736+0000","last_undegraded":"2024-09-21T14:11:39.743736+0000","last_fullsized":"2024-09-21T14:11:39.743736+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'27","last_scrub_stamp":"2024-09-21T14:11:32.143037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:48:59.038826+0000","last_clean_scrub_stamp":"2024-09-21T14:11:32.143037+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:12:38.418630+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":670,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-21T14:11:39.741300+0000","last_change":"2024-09-21T14:10:51.061937+0000","last_active":"2024-09-21T14:11:39.741300+0000","last_peered":"2024-09-21T14:11:39.741300+0000","last_clean":"2024-09-21T14:11:39.741300+0000","last_became_active":"2024-09-21T14:10:49.343665+0000","last_became_peered":"2024-09-21T14:10:49.343665+0000","last_unstale":"2024-09-21T14:11:39.741300+0000","last_undegraded":"2024-09-21T14:11:39.741300+0000","last_fullsized":"2024-09-21T14:11:39.741300+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T14:10:51.061888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T13:49:01.042371+0000","last_clean_scrub_stamp":"2024-09-21T14:10:51.061888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T14:11:56.379877+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113,"ondisk_log_size":113,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1532,"num_read_kb":1289,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":344,"num_bytes_recovered":6508,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":976,"ondisk_log_size":976,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2262,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":541,"num_read_kb":536,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":853056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1659,"ondisk_log_size":1659,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":298,"seq":1279900254357,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":696944,"kb_used_data":1568,"kb_used_omap":4,"kb_used_meta":695355,"kb_avail":93048208,"statfs":{"total":95995035648,"available":95281364992,"internally_reserved":0,"allocated":1605632,"data_stored":970313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":712044200},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463197,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":682784,"kb_used_data":1868,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":93062368,"statfs":{"total":95995035648,"available":95295864832,"internally_reserved":0,"allocated":1912832,"data_stored":1296908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":697235910},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":345,"seq":1481763717243,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":349412,"kb_used_data":1816,"kb_used_omap":4,"kb_used_meta":347579,"kb_avail":93395740,"statfs":{"total":95995035648,"available":95637237760,"internally_reserved":0,"allocated":1859584,"data_stored":1232942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4445,"internal_metadata":355921571},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921066,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":660284,"kb_used_data":1580,"kb_used_omap":1,"kb_used_meta":658686,"kb_avail":93084868,"statfs":{"total":95995035648,"available":95318904832,"internally_reserved":0,"allocated":1617920,"data_stored":969846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":674494917},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":240,"seq":1030792151217,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":854916,"kb_used_data":231220,"kb_used_omap":4,"kb_used_meta":623675,"kb_avail":92890236,"statfs":{"total":95995035648,"available":95119601664,"internally_reserved":0,"allocated":236769280,"data_stored":236114388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4443,"internal_metadata":638643877},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":130,"seq":558345748719,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":940584,"kb_used_data":1816,"kb_used_omap":4,"kb_used_meta":938747,"kb_avail":92804568,"statfs":{"total":95995035648,"available":95031877632,"internally_reserved":0,"allocated":1859584,"data_stored":1232854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4441,"internal_metadata":961277607},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575163,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":903248,"kb_used_data":1600,"kb_used_omap":1,"kb_used_meta":901630,"kb_avail":92841904,"statfs":{"total":95995035648,"available":95070109696,"internally_reserved":0,"allocated":1638400,"data_stored":970260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":923269573},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738690,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":917424,"kb_used_data":1888,"kb_used_omap":1,"kb_used_meta":915518,"kb_avail":92827728,"statfs":{"total":95995035648,"available":95055593472,"internally_reserved":0,"allocated":1933312,"data_stored":1297266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":937490885},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T14:11:51.311 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-21T14:11:51.311 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a shell --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-09-21T14:11:51.575 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T14:11:51.847 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:51 smithi049 ceph-mon[40489]: pgmap v1180: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 957 B/s rd, 0 op/s 2024-09-21T14:11:51.847 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:51 smithi049 ceph-mon[40489]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:11:51.848 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:51 smithi049 ceph-mon[40489]: from='client.26945 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:11:51.848 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:51 smithi049 ceph-mon[30080]: pgmap v1180: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 957 B/s rd, 0 op/s 2024-09-21T14:11:51.848 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:51 smithi049 ceph-mon[30080]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:11:51.848 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:51 smithi049 ceph-mon[30080]: from='client.26945 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:11:51.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:51 smithi187 ceph-mon[33032]: pgmap v1180: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 957 B/s rd, 0 op/s 2024-09-21T14:11:51.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:51 smithi187 ceph-mon[33032]: from='client.14547 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T14:11:51.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:51 smithi187 ceph-mon[33032]: from='client.26945 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T14:11:52.567 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:11:52 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: [21/Sep/2024:14:11:52] ENGINE Bus STOPPING 2024-09-21T14:11:52.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:52 smithi187 ceph-mon[33032]: purged_snaps scrub starts 2024-09-21T14:11:52.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:52 smithi187 ceph-mon[33032]: purged_snaps scrub ok 2024-09-21T14:11:52.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:52 smithi187 ceph-mon[33032]: 3.13 scrub starts 2024-09-21T14:11:52.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:52 smithi187 ceph-mon[33032]: 3.13 scrub ok 2024-09-21T14:11:52.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:52 smithi187 ceph-mon[33032]: from='client.? 172.21.15.49:0/3005252560' entity='client.admin' cmd={"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} : dispatch 2024-09-21T14:11:52.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:52 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd={"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} : dispatch 2024-09-21T14:11:52.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:52 smithi187 ceph-mon[33032]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-21T14:11:52.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:52 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T14:11:52.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:52 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T14:11:52.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:52 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T14:11:52.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:52 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:11:53.004 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:52 smithi049 ceph-mon[30080]: purged_snaps scrub starts 2024-09-21T14:11:53.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:52 smithi049 ceph-mon[30080]: purged_snaps scrub ok 2024-09-21T14:11:53.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:52 smithi049 ceph-mon[30080]: 3.13 scrub starts 2024-09-21T14:11:53.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:52 smithi049 ceph-mon[30080]: 3.13 scrub ok 2024-09-21T14:11:53.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:52 smithi049 ceph-mon[30080]: from='client.? 172.21.15.49:0/3005252560' entity='client.admin' cmd={"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} : dispatch 2024-09-21T14:11:53.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:52 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd={"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} : dispatch 2024-09-21T14:11:53.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:52 smithi049 ceph-mon[30080]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-21T14:11:53.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:52 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T14:11:53.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:52 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T14:11:53.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:52 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T14:11:53.005 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:52 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:11:53.006 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:11:52 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: [21/Sep/2024:14:11:52] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T14:11:53.006 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:11:52 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: [21/Sep/2024:14:11:52] ENGINE Bus STOPPED 2024-09-21T14:11:53.006 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:11:52 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: [21/Sep/2024:14:11:52] ENGINE Bus STARTING 2024-09-21T14:11:53.006 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:11:52 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: [21/Sep/2024:14:11:52] ENGINE Serving on http://:::9283 2024-09-21T14:11:53.006 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:11:52 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: [21/Sep/2024:14:11:52] ENGINE Bus STARTED 2024-09-21T14:11:53.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:52 smithi049 ceph-mon[40489]: purged_snaps scrub starts 2024-09-21T14:11:53.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:52 smithi049 ceph-mon[40489]: purged_snaps scrub ok 2024-09-21T14:11:53.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:52 smithi049 ceph-mon[40489]: 3.13 scrub starts 2024-09-21T14:11:53.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:52 smithi049 ceph-mon[40489]: 3.13 scrub ok 2024-09-21T14:11:53.006 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:52 smithi049 ceph-mon[40489]: from='client.? 172.21.15.49:0/3005252560' entity='client.admin' cmd={"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} : dispatch 2024-09-21T14:11:53.007 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:52 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd={"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} : dispatch 2024-09-21T14:11:53.007 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:52 smithi049 ceph-mon[40489]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-21T14:11:53.007 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:52 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T14:11:53.007 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:52 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T14:11:53.007 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:52 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T14:11:53.007 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:52 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' 2024-09-21T14:11:53.085 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-09-21T14:11:53.098 INFO:tasks.cephadm:Teardown begin 2024-09-21T14:11:53.098 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T14:11:53.137 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T14:11:53.184 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-09-21T14:11:53.184 DEBUG:teuthology.orchestra.run.smithi049:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-21T14:11:53.200 DEBUG:teuthology.orchestra.run.smithi187:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-21T14:11:53.226 INFO:tasks.cephadm:Stopping all daemons... 2024-09-21T14:11:53.227 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-09-21T14:11:53.227 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@mon.a 2024-09-21T14:11:53.504 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:53 smithi049 systemd[1]: Stopping Ceph mon.a for 40a3d2e8-781f-11ef-baf6-efdc52797490... 2024-09-21T14:11:53.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:53 smithi187 ceph-mon[33032]: pgmap v1181: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T14:11:53.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:53 smithi187 ceph-mon[33032]: 6.7 scrub starts 2024-09-21T14:11:53.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:53 smithi187 ceph-mon[33032]: 6.7 scrub ok 2024-09-21T14:11:53.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:53 smithi187 ceph-mon[33032]: osdmap e581: 8 total, 8 up, 8 in 2024-09-21T14:11:53.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:53 smithi187 ceph-mon[33032]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T14:11:53.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:53 smithi187 ceph-mon[33032]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T14:11:53.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:53 smithi187 ceph-mon[33032]: 5.e scrub starts 2024-09-21T14:11:53.945 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:53 smithi187 ceph-mon[33032]: 5.e scrub ok 2024-09-21T14:11:53.996 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:53 smithi049 ceph-mon[40489]: pgmap v1181: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T14:11:53.996 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:53 smithi049 ceph-mon[40489]: 6.7 scrub starts 2024-09-21T14:11:53.996 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:53 smithi049 ceph-mon[40489]: 6.7 scrub ok 2024-09-21T14:11:53.996 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:53 smithi049 ceph-mon[40489]: osdmap e581: 8 total, 8 up, 8 in 2024-09-21T14:11:53.996 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:53 smithi049 ceph-mon[40489]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T14:11:53.996 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:53 smithi049 ceph-mon[40489]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T14:11:53.996 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:53 smithi049 ceph-mon[40489]: 5.e scrub starts 2024-09-21T14:11:53.996 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:53 smithi049 ceph-mon[40489]: 5.e scrub ok 2024-09-21T14:11:53.997 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:11:53 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: [21/Sep/2024:14:11:53] ENGINE Bus STOPPING 2024-09-21T14:11:53.997 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:11:53 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: [21/Sep/2024:14:11:53] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T14:11:53.997 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:11:53 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: [21/Sep/2024:14:11:53] ENGINE Bus STOPPED 2024-09-21T14:11:53.997 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:11:53 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: [21/Sep/2024:14:11:53] ENGINE Bus STARTING 2024-09-21T14:11:53.997 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:11:53 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: [21/Sep/2024:14:11:53] ENGINE Serving on http://:::9283 2024-09-21T14:11:53.997 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:11:53 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: [21/Sep/2024:14:11:53] ENGINE Bus STARTED 2024-09-21T14:11:53.997 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:53 smithi049 ceph-mon[30080]: pgmap v1181: 152 pgs: 152 active+clean; 583 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T14:11:53.997 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:53 smithi049 ceph-mon[30080]: 6.7 scrub starts 2024-09-21T14:11:53.997 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:53 smithi049 ceph-mon[30080]: 6.7 scrub ok 2024-09-21T14:11:53.997 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:53 smithi049 ceph-mon[30080]: osdmap e581: 8 total, 8 up, 8 in 2024-09-21T14:11:53.997 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:53 smithi049 ceph-mon[30080]: from='mgr.14601 172.21.15.49:0/3327314754' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T14:11:53.998 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:53 smithi049 ceph-mon[30080]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T14:11:53.998 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:53 smithi049 ceph-mon[30080]: 5.e scrub starts 2024-09-21T14:11:53.998 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:53 smithi049 ceph-mon[30080]: 5.e scrub ok 2024-09-21T14:11:53.998 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:53 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mon-a[30057]: 2024-09-21T14:11:53.580+0000 7fc8bc857640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T14:11:53.998 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:53 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mon-a[30057]: 2024-09-21T14:11:53.580+0000 7fc8bc857640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-09-21T14:11:53.998 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:53 smithi049 podman[237842]: 2024-09-21 14:11:53.68365792 +0000 UTC m=+0.233752957 container died 5ef18ce25de2fa63e69656e4aff529955e8cc0a86e7f783e183cc4634d08d806 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mon-a, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.schema-version=1.0) 2024-09-21T14:11:54.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:54 smithi049 podman[237842]: 2024-09-21 14:11:54.203974225 +0000 UTC m=+0.754069261 container cleanup 5ef18ce25de2fa63e69656e4aff529955e8cc0a86e7f783e183cc4634d08d806 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mon-a, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T14:11:54.254 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:54 smithi049 bash[237842]: ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mon-a 2024-09-21T14:11:54.704 DEBUG:teuthology.orchestra.run.smithi049:> sudo pkill -f 'journalctl -f -n 0 -u ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@mon.a.service' 2024-09-21T14:11:54.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:54 smithi049 podman[237861]: 2024-09-21 14:11:54.464152279 +0000 UTC m=+0.781959130 container remove 5ef18ce25de2fa63e69656e4aff529955e8cc0a86e7f783e183cc4634d08d806 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mon-a, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-21-0842-4417849, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-09-21T14:11:54.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:54 smithi049 systemd[1]: ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@mon.a.service: Deactivated successfully. 2024-09-21T14:11:54.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:54 smithi049 systemd[1]: Stopped Ceph mon.a for 40a3d2e8-781f-11ef-baf6-efdc52797490. 2024-09-21T14:11:54.727 INFO:journalctl@ceph.mon.a.smithi049.stdout:Sep 21 14:11:54 smithi049 systemd[1]: ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@mon.a.service: Consumed 34.954s CPU time. 2024-09-21T14:11:54.743 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T14:11:54.744 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-09-21T14:11:54.744 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-09-21T14:11:54.744 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@mon.c 2024-09-21T14:11:55.004 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:54 smithi049 systemd[1]: Stopping Ceph mon.c for 40a3d2e8-781f-11ef-baf6-efdc52797490... 2024-09-21T14:11:55.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:55 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mon-c[40466]: 2024-09-21T14:11:55.127+0000 7f9044050640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T14:11:55.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:55 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mon-c[40466]: 2024-09-21T14:11:55.127+0000 7f9044050640 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-09-21T14:11:55.504 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:55 smithi049 podman[237984]: 2024-09-21 14:11:55.22072686 +0000 UTC m=+0.243497949 container died cd9887925ce4b247db9da2687f910e68500abf41fdcbc3c776120dd25f835f91 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mon-c, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-guits-main-2024-09-21-0842-4417849, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-09-21T14:11:55.986 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:55 smithi049 podman[237984]: 2024-09-21 14:11:55.718848918 +0000 UTC m=+0.741620006 container cleanup cd9887925ce4b247db9da2687f910e68500abf41fdcbc3c776120dd25f835f91 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mon-c, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T14:11:55.986 INFO:journalctl@ceph.mon.c.smithi049.stdout:Sep 21 14:11:55 smithi049 bash[237984]: ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mon-c 2024-09-21T14:11:56.116 DEBUG:teuthology.orchestra.run.smithi049:> sudo pkill -f 'journalctl -f -n 0 -u ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@mon.c.service' 2024-09-21T14:11:56.152 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T14:11:56.152 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-09-21T14:11:56.152 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-09-21T14:11:56.153 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@mon.b 2024-09-21T14:11:56.444 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:56 smithi187 systemd[1]: Stopping Ceph mon.b for 40a3d2e8-781f-11ef-baf6-efdc52797490... 2024-09-21T14:11:56.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:56 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mon-b[33009]: 2024-09-21T14:11:56.479+0000 7f7416eb9640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T14:11:56.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:56 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mon-b[33009]: 2024-09-21T14:11:56.479+0000 7f7416eb9640 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-09-21T14:11:56.944 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:56 smithi187 podman[169317]: 2024-09-21 14:11:56.725028387 +0000 UTC m=+0.366298686 container died f5f4713510d2b18e55f23a9e0332d00d119cc061ac1e4407c2215fc553576ab4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mon-b, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-21-0842-4417849, ceph=True, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-09-21T14:11:57.371 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:57 smithi187 podman[169317]: 2024-09-21 14:11:57.136783974 +0000 UTC m=+0.778054271 container cleanup f5f4713510d2b18e55f23a9e0332d00d119cc061ac1e4407c2215fc553576ab4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mon-b, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-21-0842-4417849, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T14:11:57.371 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:57 smithi187 bash[169317]: ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mon-b 2024-09-21T14:11:57.619 DEBUG:teuthology.orchestra.run.smithi187:> sudo pkill -f 'journalctl -f -n 0 -u ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@mon.b.service' 2024-09-21T14:11:57.641 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:57 smithi187 podman[169334]: 2024-09-21 14:11:57.371728784 +0000 UTC m=+0.650105221 container remove f5f4713510d2b18e55f23a9e0332d00d119cc061ac1e4407c2215fc553576ab4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mon-b, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-21-0842-4417849, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T14:11:57.641 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:57 smithi187 systemd[1]: ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@mon.b.service: Deactivated successfully. 2024-09-21T14:11:57.641 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:57 smithi187 systemd[1]: Stopped Ceph mon.b for 40a3d2e8-781f-11ef-baf6-efdc52797490. 2024-09-21T14:11:57.641 INFO:journalctl@ceph.mon.b.smithi187.stdout:Sep 21 14:11:57 smithi187 systemd[1]: ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@mon.b.service: Consumed 23.806s CPU time. 2024-09-21T14:11:57.665 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T14:11:57.666 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-09-21T14:11:57.666 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-09-21T14:11:57.666 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@mgr.y 2024-09-21T14:11:57.983 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:11:57 smithi049 systemd[1]: Stopping Ceph mgr.y for 40a3d2e8-781f-11ef-baf6-efdc52797490... 2024-09-21T14:11:58.254 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:11:57 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: 2024-09-21T14:11:57.982+0000 7f6ae9a2d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T14:11:58.254 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:11:57 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y[30332]: 2024-09-21T14:11:57.982+0000 7f6ae9a2d640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-09-21T14:11:58.254 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:11:58 smithi049 podman[238116]: 2024-09-21 14:11:58.015443489 +0000 UTC m=+0.150133151 container died 0bf9f1993273ed20cf3091d6c7746dde435ba745f53b058916a879c6b6067c50 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True) 2024-09-21T14:11:59.004 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:11:58 smithi049 podman[238116]: 2024-09-21 14:11:58.716413991 +0000 UTC m=+0.851103646 container cleanup 0bf9f1993273ed20cf3091d6c7746dde435ba745f53b058916a879c6b6067c50 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True) 2024-09-21T14:11:59.004 INFO:journalctl@ceph.mgr.y.smithi049.stdout:Sep 21 14:11:58 smithi049 bash[238116]: ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-y 2024-09-21T14:11:59.148 DEBUG:teuthology.orchestra.run.smithi049:> sudo pkill -f 'journalctl -f -n 0 -u ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@mgr.y.service' 2024-09-21T14:11:59.184 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T14:11:59.184 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-09-21T14:11:59.184 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-09-21T14:11:59.184 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@mgr.x 2024-09-21T14:11:59.532 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Sep 21 14:11:59 smithi187 systemd[1]: Stopping Ceph mgr.x for 40a3d2e8-781f-11ef-baf6-efdc52797490... 2024-09-21T14:11:59.842 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Sep 21 14:11:59 smithi187 podman[169450]: 2024-09-21 14:11:59.53305556 +0000 UTC m=+0.153943744 container died 3c06c2000419200307e04376a168148d371d904454d105011517df8eab755906 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T14:12:00.194 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Sep 21 14:12:00 smithi187 podman[169450]: 2024-09-21 14:12:00.025067357 +0000 UTC m=+0.645955577 container cleanup 3c06c2000419200307e04376a168148d371d904454d105011517df8eab755906 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-x, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-21-0842-4417849, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T14:12:00.194 INFO:journalctl@ceph.mgr.x.smithi187.stdout:Sep 21 14:12:00 smithi187 bash[169450]: ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-mgr-x 2024-09-21T14:12:00.501 DEBUG:teuthology.orchestra.run.smithi187:> sudo pkill -f 'journalctl -f -n 0 -u ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@mgr.x.service' 2024-09-21T14:12:00.536 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T14:12:00.536 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-09-21T14:12:00.537 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-21T14:12:00.537 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@osd.0 2024-09-21T14:12:00.881 INFO:journalctl@ceph.osd.0.smithi049.stdout:Sep 21 14:12:00 smithi049 systemd[1]: Stopping Ceph osd.0 for 40a3d2e8-781f-11ef-baf6-efdc52797490... 2024-09-21T14:12:01.254 INFO:journalctl@ceph.osd.0.smithi049.stdout:Sep 21 14:12:00 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-0[44192]: 2024-09-21T14:12:00.880+0000 7f3ad3217640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T14:12:01.255 INFO:journalctl@ceph.osd.0.smithi049.stdout:Sep 21 14:12:00 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-0[44192]: 2024-09-21T14:12:00.880+0000 7f3ad3217640 -1 osd.0 582 *** Got signal Terminated *** 2024-09-21T14:12:01.255 INFO:journalctl@ceph.osd.0.smithi049.stdout:Sep 21 14:12:00 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-0[44192]: 2024-09-21T14:12:00.880+0000 7f3ad3217640 -1 osd.0 582 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T14:12:06.683 INFO:journalctl@ceph.osd.0.smithi049.stdout:Sep 21 14:12:06 smithi049 podman[238245]: 2024-09-21 14:12:06.403881121 +0000 UTC m=+5.660937932 container died 03ce896824472aa92d80139736237f458feda618a94918c9d972cbabaef418a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-guits-main-2024-09-21-0842-4417849, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-21T14:12:07.004 INFO:journalctl@ceph.osd.0.smithi049.stdout:Sep 21 14:12:06 smithi049 podman[238245]: 2024-09-21 14:12:06.888637452 +0000 UTC m=+6.145694289 container cleanup 03ce896824472aa92d80139736237f458feda618a94918c9d972cbabaef418a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-21-0842-4417849, ceph=True) 2024-09-21T14:12:07.004 INFO:journalctl@ceph.osd.0.smithi049.stdout:Sep 21 14:12:06 smithi049 bash[238245]: ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-0 2024-09-21T14:12:07.504 INFO:journalctl@ceph.osd.0.smithi049.stdout:Sep 21 14:12:07 smithi049 podman[238265]: 2024-09-21 14:12:07.148536693 +0000 UTC m=+0.743777983 container remove 03ce896824472aa92d80139736237f458feda618a94918c9d972cbabaef418a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-main-2024-09-21-0842-4417849, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-21T14:12:07.884 INFO:journalctl@ceph.osd.0.smithi049.stdout:Sep 21 14:12:07 smithi049 podman[238362]: 2024-09-21 14:12:07.415977058 +0000 UTC m=+0.016619520 image pull b415816db8b41f544cdfdc71d193ff8b1ee008bf7b0ec4f64da8eb8f71b62e85 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5 2024-09-21T14:12:07.884 INFO:journalctl@ceph.osd.0.smithi049.stdout:Sep 21 14:12:07 smithi049 podman[238362]: 2024-09-21 14:12:07.53290971 +0000 UTC m=+0.133552189 container create 596dde465e2f5d68c6c193d683e99ce10ef178b8a77ace208ff961e521b8c7b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-0-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True) 2024-09-21T14:12:07.885 INFO:journalctl@ceph.osd.0.smithi049.stdout:Sep 21 14:12:07 smithi049 podman[238362]: 2024-09-21 14:12:07.752050523 +0000 UTC m=+0.352692979 container init 596dde465e2f5d68c6c193d683e99ce10ef178b8a77ace208ff961e521b8c7b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-21-0842-4417849, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS) 2024-09-21T14:12:07.885 INFO:journalctl@ceph.osd.0.smithi049.stdout:Sep 21 14:12:07 smithi049 podman[238362]: 2024-09-21 14:12:07.757049656 +0000 UTC m=+0.357692093 container start 596dde465e2f5d68c6c193d683e99ce10ef178b8a77ace208ff961e521b8c7b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-0-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-guits-main-2024-09-21-0842-4417849, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-09-21T14:12:07.885 INFO:journalctl@ceph.osd.0.smithi049.stdout:Sep 21 14:12:07 smithi049 podman[238362]: 2024-09-21 14:12:07.885299421 +0000 UTC m=+0.485941865 container attach 596dde465e2f5d68c6c193d683e99ce10ef178b8a77ace208ff961e521b8c7b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-0-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-21T14:12:08.504 INFO:journalctl@ceph.osd.0.smithi049.stdout:Sep 21 14:12:08 smithi049 podman[238362]: 2024-09-21 14:12:08.143769668 +0000 UTC m=+0.744412123 container died 596dde465e2f5d68c6c193d683e99ce10ef178b8a77ace208ff961e521b8c7b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-0-deactivate, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-21-0842-4417849, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2) 2024-09-21T14:12:08.853 DEBUG:teuthology.orchestra.run.smithi049:> sudo pkill -f 'journalctl -f -n 0 -u ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@osd.0.service' 2024-09-21T14:12:08.918 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T14:12:08.918 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-21T14:12:08.918 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-21T14:12:08.918 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@osd.1 2024-09-21T14:12:09.254 INFO:journalctl@ceph.osd.1.smithi049.stdout:Sep 21 14:12:09 smithi049 systemd[1]: Stopping Ceph osd.1 for 40a3d2e8-781f-11ef-baf6-efdc52797490... 2024-09-21T14:12:09.754 INFO:journalctl@ceph.osd.1.smithi049.stdout:Sep 21 14:12:09 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-1[49970]: 2024-09-21T14:12:09.261+0000 7fe19d68e640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T14:12:09.754 INFO:journalctl@ceph.osd.1.smithi049.stdout:Sep 21 14:12:09 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-1[49970]: 2024-09-21T14:12:09.261+0000 7fe19d68e640 -1 osd.1 582 *** Got signal Terminated *** 2024-09-21T14:12:09.754 INFO:journalctl@ceph.osd.1.smithi049.stdout:Sep 21 14:12:09 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-1[49970]: 2024-09-21T14:12:09.261+0000 7fe19d68e640 -1 osd.1 582 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T14:12:15.076 INFO:journalctl@ceph.osd.1.smithi049.stdout:Sep 21 14:12:14 smithi049 podman[238490]: 2024-09-21 14:12:14.778977974 +0000 UTC m=+5.630177811 container died 583a67317952184d7eb46411b4d233407df90b72937d10af2d8db984f08cd2a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-1, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-guits-main-2024-09-21-0842-4417849, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS) 2024-09-21T14:12:15.504 INFO:journalctl@ceph.osd.1.smithi049.stdout:Sep 21 14:12:15 smithi049 podman[238490]: 2024-09-21 14:12:15.284009199 +0000 UTC m=+6.135209033 container cleanup 583a67317952184d7eb46411b4d233407df90b72937d10af2d8db984f08cd2a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-1, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-guits-main-2024-09-21-0842-4417849, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-21T14:12:15.504 INFO:journalctl@ceph.osd.1.smithi049.stdout:Sep 21 14:12:15 smithi049 bash[238490]: ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-1 2024-09-21T14:12:15.860 INFO:journalctl@ceph.osd.1.smithi049.stdout:Sep 21 14:12:15 smithi049 podman[238510]: 2024-09-21 14:12:15.546129516 +0000 UTC m=+0.760237663 container remove 583a67317952184d7eb46411b4d233407df90b72937d10af2d8db984f08cd2a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-1, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-21T14:12:16.122 INFO:journalctl@ceph.osd.1.smithi049.stdout:Sep 21 14:12:15 smithi049 podman[238605]: 2024-09-21 14:12:15.76448087 +0000 UTC m=+0.016179877 image pull b415816db8b41f544cdfdc71d193ff8b1ee008bf7b0ec4f64da8eb8f71b62e85 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5 2024-09-21T14:12:16.122 INFO:journalctl@ceph.osd.1.smithi049.stdout:Sep 21 14:12:15 smithi049 podman[238605]: 2024-09-21 14:12:15.895485615 +0000 UTC m=+0.147184608 container create 0a2d75f3fc7661acaa4a0fd65bbbe069b8901f4ecaf4b0810b168fcf76e88a2b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=) 2024-09-21T14:12:16.504 INFO:journalctl@ceph.osd.1.smithi049.stdout:Sep 21 14:12:16 smithi049 podman[238605]: 2024-09-21 14:12:16.122929209 +0000 UTC m=+0.374628211 container init 0a2d75f3fc7661acaa4a0fd65bbbe069b8901f4ecaf4b0810b168fcf76e88a2b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-1-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.license=GPLv2) 2024-09-21T14:12:16.504 INFO:journalctl@ceph.osd.1.smithi049.stdout:Sep 21 14:12:16 smithi049 podman[238605]: 2024-09-21 14:12:16.127967315 +0000 UTC m=+0.379666310 container start 0a2d75f3fc7661acaa4a0fd65bbbe069b8901f4ecaf4b0810b168fcf76e88a2b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-1-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-21T14:12:16.504 INFO:journalctl@ceph.osd.1.smithi049.stdout:Sep 21 14:12:16 smithi049 podman[238605]: 2024-09-21 14:12:16.222699377 +0000 UTC m=+0.474398374 container attach 0a2d75f3fc7661acaa4a0fd65bbbe069b8901f4ecaf4b0810b168fcf76e88a2b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-1-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-21-0842-4417849, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=) 2024-09-21T14:12:16.504 INFO:journalctl@ceph.osd.1.smithi049.stdout:Sep 21 14:12:16 smithi049 podman[238605]: 2024-09-21 14:12:16.325479583 +0000 UTC m=+0.577178578 container died 0a2d75f3fc7661acaa4a0fd65bbbe069b8901f4ecaf4b0810b168fcf76e88a2b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-1-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T14:12:16.988 DEBUG:teuthology.orchestra.run.smithi049:> sudo pkill -f 'journalctl -f -n 0 -u ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@osd.1.service' 2024-09-21T14:12:17.035 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T14:12:17.036 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-21T14:12:17.036 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-21T14:12:17.036 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@osd.2 2024-09-21T14:12:17.254 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 14:12:17 smithi049 systemd[1]: Stopping Ceph osd.2 for 40a3d2e8-781f-11ef-baf6-efdc52797490... 2024-09-21T14:12:17.754 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 14:12:17 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-2[90617]: 2024-09-21T14:12:17.364+0000 7fe75ce44640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T14:12:17.754 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 14:12:17 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-2[90617]: 2024-09-21T14:12:17.364+0000 7fe75ce44640 -1 osd.2 582 *** Got signal Terminated *** 2024-09-21T14:12:17.754 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 14:12:17 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-2[90617]: 2024-09-21T14:12:17.364+0000 7fe75ce44640 -1 osd.2 582 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T14:12:23.161 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 14:12:22 smithi049 podman[238732]: 2024-09-21 14:12:22.838553339 +0000 UTC m=+5.608988682 container died 3237a2aba8854d698c9a5eaf24faee6e29558c9ddf9a48b90f332e76bee835c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-2, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T14:12:23.504 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 14:12:23 smithi049 podman[238732]: 2024-09-21 14:12:23.32123977 +0000 UTC m=+6.091675113 container cleanup 3237a2aba8854d698c9a5eaf24faee6e29558c9ddf9a48b90f332e76bee835c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-21T14:12:23.504 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 14:12:23 smithi049 bash[238732]: ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-2 2024-09-21T14:12:23.951 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 14:12:23 smithi049 podman[238754]: 2024-09-21 14:12:23.639701273 +0000 UTC m=+0.785776517 container remove 3237a2aba8854d698c9a5eaf24faee6e29558c9ddf9a48b90f332e76bee835c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-21T14:12:24.206 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 14:12:23 smithi049 podman[238849]: 2024-09-21 14:12:23.85533821 +0000 UTC m=+0.016676188 image pull b415816db8b41f544cdfdc71d193ff8b1ee008bf7b0ec4f64da8eb8f71b62e85 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5 2024-09-21T14:12:24.206 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 14:12:23 smithi049 podman[238849]: 2024-09-21 14:12:23.965697371 +0000 UTC m=+0.127035375 container create 6ed324549a57a37a658704160bcef610db6f4dcf7949ef4937d6c30f733bb020 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-2-deactivate, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-09-21T14:12:24.206 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 14:12:24 smithi049 podman[238849]: 2024-09-21 14:12:24.201697895 +0000 UTC m=+0.363035862 container init 6ed324549a57a37a658704160bcef610db6f4dcf7949ef4937d6c30f733bb020 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-2-deactivate, ceph=True, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-21-0842-4417849) 2024-09-21T14:12:24.206 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 14:12:24 smithi049 podman[238849]: 2024-09-21 14:12:24.207001598 +0000 UTC m=+0.368339560 container start 6ed324549a57a37a658704160bcef610db6f4dcf7949ef4937d6c30f733bb020 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2) 2024-09-21T14:12:24.504 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 14:12:24 smithi049 podman[238849]: 2024-09-21 14:12:24.318151567 +0000 UTC m=+0.479489532 container attach 6ed324549a57a37a658704160bcef610db6f4dcf7949ef4937d6c30f733bb020 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-2-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-guits-main-2024-09-21-0842-4417849, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.37.2) 2024-09-21T14:12:24.504 INFO:journalctl@ceph.osd.2.smithi049.stdout:Sep 21 14:12:24 smithi049 podman[238849]: 2024-09-21 14:12:24.401519552 +0000 UTC m=+0.562857525 container died 6ed324549a57a37a658704160bcef610db6f4dcf7949ef4937d6c30f733bb020 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-2-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-guits-main-2024-09-21-0842-4417849, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_CLEAN=True) 2024-09-21T14:12:25.117 DEBUG:teuthology.orchestra.run.smithi049:> sudo pkill -f 'journalctl -f -n 0 -u ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@osd.2.service' 2024-09-21T14:12:25.165 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T14:12:25.165 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-21T14:12:25.165 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-09-21T14:12:25.165 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@osd.3 2024-09-21T14:12:25.501 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 14:12:25 smithi049 systemd[1]: Stopping Ceph osd.3 for 40a3d2e8-781f-11ef-baf6-efdc52797490... 2024-09-21T14:12:25.754 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 14:12:25 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-3[129752]: 2024-09-21T14:12:25.500+0000 7ff0d73f4640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T14:12:25.754 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 14:12:25 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-3[129752]: 2024-09-21T14:12:25.500+0000 7ff0d73f4640 -1 osd.3 582 *** Got signal Terminated *** 2024-09-21T14:12:25.754 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 14:12:25 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-3[129752]: 2024-09-21T14:12:25.500+0000 7ff0d73f4640 -1 osd.3 582 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T14:12:26.945 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 14:12:26 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[71382]: ts=2024-09-21T14:12:26.500Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=nfs msg="Unable to refresh target groups" err="Get \"http://172.21.15.49:8765/sd/prometheus/sd-config?service=nfs\": dial tcp 172.21.15.49:8765: connect: connection refused" 2024-09-21T14:12:26.945 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 14:12:26 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[71382]: ts=2024-09-21T14:12:26.500Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.49:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.49:8765: connect: connection refused" 2024-09-21T14:12:26.945 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 14:12:26 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[71382]: ts=2024-09-21T14:12:26.504Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=smb msg="Unable to refresh target groups" err="Get \"http://172.21.15.49:8765/sd/prometheus/sd-config?service=smb\": dial tcp 172.21.15.49:8765: connect: connection refused" 2024-09-21T14:12:26.945 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 14:12:26 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[71382]: ts=2024-09-21T14:12:26.504Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.49:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.49:8765: connect: connection refused" 2024-09-21T14:12:26.945 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 14:12:26 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[71382]: ts=2024-09-21T14:12:26.504Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.49:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.49:8765: connect: connection refused" 2024-09-21T14:12:26.945 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 14:12:26 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[71382]: ts=2024-09-21T14:12:26.504Z caller=refresh.go:90 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.49:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.49:8765: connect: connection refused" 2024-09-21T14:12:26.945 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 14:12:26 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[71382]: ts=2024-09-21T14:12:26.505Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.49:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.49:8765: connect: connection refused" 2024-09-21T14:12:28.444 INFO:journalctl@ceph.osd.4.smithi187.stdout:Sep 21 14:12:28 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-4[37666]: 2024-09-21T14:12:28.124+0000 7fdf1d043640 -1 osd.4 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:03.289750+0000 front 2024-09-21T14:12:03.290474+0000 (oldest deadline 2024-09-21T14:12:27.989684+0000) 2024-09-21T14:12:28.754 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 14:12:28 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-3[129752]: 2024-09-21T14:12:28.372+0000 7ff0d3a09640 -1 osd.3 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:03.860930+0000 front 2024-09-21T14:12:03.860951+0000 (oldest deadline 2024-09-21T14:12:27.961024+0000) 2024-09-21T14:12:29.444 INFO:journalctl@ceph.osd.4.smithi187.stdout:Sep 21 14:12:29 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-4[37666]: 2024-09-21T14:12:29.152+0000 7fdf1d043640 -1 osd.4 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:03.289750+0000 front 2024-09-21T14:12:03.290474+0000 (oldest deadline 2024-09-21T14:12:27.989684+0000) 2024-09-21T14:12:29.444 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:29 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:29.067+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:05.454830+0000 front 2024-09-21T14:12:05.454963+0000 (oldest deadline 2024-09-21T14:12:28.355054+0000) 2024-09-21T14:12:29.754 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 14:12:29 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-3[129752]: 2024-09-21T14:12:29.390+0000 7ff0d3a09640 -1 osd.3 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:03.860930+0000 front 2024-09-21T14:12:03.860951+0000 (oldest deadline 2024-09-21T14:12:27.961024+0000) 2024-09-21T14:12:29.944 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:12:29 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5[125175]: 2024-09-21T14:12:29.474+0000 7f29e26d7640 -1 osd.5 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:03.765259+0000 front 2024-09-21T14:12:03.765323+0000 (oldest deadline 2024-09-21T14:12:29.065105+0000) 2024-09-21T14:12:30.444 INFO:journalctl@ceph.osd.4.smithi187.stdout:Sep 21 14:12:30 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-4[37666]: 2024-09-21T14:12:30.184+0000 7fdf1d043640 -1 osd.4 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:03.289750+0000 front 2024-09-21T14:12:03.290474+0000 (oldest deadline 2024-09-21T14:12:27.989684+0000) 2024-09-21T14:12:30.444 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:30 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:30.084+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:05.454830+0000 front 2024-09-21T14:12:05.454963+0000 (oldest deadline 2024-09-21T14:12:28.355054+0000) 2024-09-21T14:12:30.754 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 14:12:30 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-3[129752]: 2024-09-21T14:12:30.378+0000 7ff0d3a09640 -1 osd.3 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:03.860930+0000 front 2024-09-21T14:12:03.860951+0000 (oldest deadline 2024-09-21T14:12:27.961024+0000) 2024-09-21T14:12:30.944 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:12:30 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5[125175]: 2024-09-21T14:12:30.445+0000 7f29e26d7640 -1 osd.5 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:03.765259+0000 front 2024-09-21T14:12:03.765323+0000 (oldest deadline 2024-09-21T14:12:29.065105+0000) 2024-09-21T14:12:31.250 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 14:12:30 smithi049 podman[238976]: 2024-09-21 14:12:30.949045114 +0000 UTC m=+5.580232995 container died 48889efcabc233953851259bbae7c1dfdd5a57b89411580d7df6cbdf2137b3f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-3, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-21-0842-4417849, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T14:12:31.396 INFO:journalctl@ceph.osd.4.smithi187.stdout:Sep 21 14:12:31 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-4[37666]: 2024-09-21T14:12:31.144+0000 7fdf1d043640 -1 osd.4 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:03.289750+0000 front 2024-09-21T14:12:03.290474+0000 (oldest deadline 2024-09-21T14:12:27.989684+0000) 2024-09-21T14:12:31.396 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:31 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:31.069+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:05.454830+0000 front 2024-09-21T14:12:05.454963+0000 (oldest deadline 2024-09-21T14:12:28.355054+0000) 2024-09-21T14:12:31.397 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:30 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T14:12:30.947+0000 7f3a3bf19640 -1 osd.6 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:04.825816+0000 front 2024-09-21T14:12:04.826007+0000 (oldest deadline 2024-09-21T14:12:30.125898+0000) 2024-09-21T14:12:31.504 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 14:12:31 smithi049 podman[238976]: 2024-09-21 14:12:31.416619666 +0000 UTC m=+6.047807534 container cleanup 48889efcabc233953851259bbae7c1dfdd5a57b89411580d7df6cbdf2137b3f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-3, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-21-0842-4417849, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.37.2, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2) 2024-09-21T14:12:31.504 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 14:12:31 smithi049 bash[238976]: ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-3 2024-09-21T14:12:31.694 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:12:31 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5[125175]: 2024-09-21T14:12:31.396+0000 7f29e26d7640 -1 osd.5 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:03.765259+0000 front 2024-09-21T14:12:03.765323+0000 (oldest deadline 2024-09-21T14:12:29.065105+0000) 2024-09-21T14:12:32.004 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 14:12:31 smithi049 podman[238996]: 2024-09-21 14:12:31.668707622 +0000 UTC m=+0.664345137 container remove 48889efcabc233953851259bbae7c1dfdd5a57b89411580d7df6cbdf2137b3f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-3, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-21-0842-4417849, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.vendor=CentOS) 2024-09-21T14:12:32.387 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 14:12:32 smithi049 podman[239088]: 2024-09-21 14:12:31.991866163 +0000 UTC m=+0.016651663 image pull b415816db8b41f544cdfdc71d193ff8b1ee008bf7b0ec4f64da8eb8f71b62e85 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5 2024-09-21T14:12:32.387 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 14:12:32 smithi049 podman[239088]: 2024-09-21 14:12:32.094462796 +0000 UTC m=+0.119248329 container create 9578de0b2b4b34ec9db11ea39af44185df0f1f08eed55ece14e38d5213c5da63 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-3-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-21-0842-4417849, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240731) 2024-09-21T14:12:32.387 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 14:12:32 smithi049 podman[239088]: 2024-09-21 14:12:32.296697047 +0000 UTC m=+0.321482555 container init 9578de0b2b4b34ec9db11ea39af44185df0f1f08eed55ece14e38d5213c5da63 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-21-0842-4417849, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-09-21T14:12:32.387 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 14:12:32 smithi049 podman[239088]: 2024-09-21 14:12:32.301634599 +0000 UTC m=+0.326420089 container start 9578de0b2b4b34ec9db11ea39af44185df0f1f08eed55ece14e38d5213c5da63 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-3-deactivate, RELEASE=wip-guits-main-2024-09-21-0842-4417849, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-09-21T14:12:32.388 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 14:12:32 smithi049 podman[239088]: 2024-09-21 14:12:32.388244612 +0000 UTC m=+0.413030121 container attach 9578de0b2b4b34ec9db11ea39af44185df0f1f08eed55ece14e38d5213c5da63 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-3-deactivate, io.buildah.version=1.37.2, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-guits-main-2024-09-21-0842-4417849, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-09-21T14:12:32.409 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:31 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T14:12:31.994+0000 7f3a3bf19640 -1 osd.6 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:04.825816+0000 front 2024-09-21T14:12:04.826007+0000 (oldest deadline 2024-09-21T14:12:30.125898+0000) 2024-09-21T14:12:32.409 INFO:journalctl@ceph.osd.4.smithi187.stdout:Sep 21 14:12:32 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-4[37666]: 2024-09-21T14:12:32.192+0000 7fdf1d043640 -1 osd.4 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:03.289750+0000 front 2024-09-21T14:12:03.290474+0000 (oldest deadline 2024-09-21T14:12:27.989684+0000) 2024-09-21T14:12:32.409 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:32 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:32.027+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:05.454830+0000 front 2024-09-21T14:12:05.454963+0000 (oldest deadline 2024-09-21T14:12:28.355054+0000) 2024-09-21T14:12:32.694 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:12:32 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5[125175]: 2024-09-21T14:12:32.409+0000 7f29e26d7640 -1 osd.5 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:03.765259+0000 front 2024-09-21T14:12:03.765323+0000 (oldest deadline 2024-09-21T14:12:29.065105+0000) 2024-09-21T14:12:32.754 INFO:journalctl@ceph.osd.3.smithi049.stdout:Sep 21 14:12:32 smithi049 podman[239131]: 2024-09-21 14:12:32.51621557 +0000 UTC m=+0.016593858 container died 9578de0b2b4b34ec9db11ea39af44185df0f1f08eed55ece14e38d5213c5da63 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-3-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-guits-main-2024-09-21-0842-4417849, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T14:12:33.080 DEBUG:teuthology.orchestra.run.smithi049:> sudo pkill -f 'journalctl -f -n 0 -u ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@osd.3.service' 2024-09-21T14:12:33.121 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T14:12:33.121 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-09-21T14:12:33.121 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-21T14:12:33.121 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@osd.4 2024-09-21T14:12:33.419 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:33 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T14:12:33.026+0000 7f3a3bf19640 -1 osd.6 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:04.825816+0000 front 2024-09-21T14:12:04.826007+0000 (oldest deadline 2024-09-21T14:12:30.125898+0000) 2024-09-21T14:12:33.422 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:33 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:33.002+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:05.454830+0000 front 2024-09-21T14:12:05.454963+0000 (oldest deadline 2024-09-21T14:12:28.355054+0000) 2024-09-21T14:12:33.422 INFO:journalctl@ceph.osd.4.smithi187.stdout:Sep 21 14:12:33 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-4[37666]: 2024-09-21T14:12:33.148+0000 7fdf1d043640 -1 osd.4 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:03.289750+0000 front 2024-09-21T14:12:03.290474+0000 (oldest deadline 2024-09-21T14:12:27.989684+0000) 2024-09-21T14:12:33.422 INFO:journalctl@ceph.osd.4.smithi187.stdout:Sep 21 14:12:33 smithi187 systemd[1]: Stopping Ceph osd.4 for 40a3d2e8-781f-11ef-baf6-efdc52797490... 2024-09-21T14:12:33.694 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:12:33 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5[125175]: 2024-09-21T14:12:33.435+0000 7f29e26d7640 -1 osd.5 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:03.765259+0000 front 2024-09-21T14:12:03.765323+0000 (oldest deadline 2024-09-21T14:12:29.065105+0000) 2024-09-21T14:12:33.694 INFO:journalctl@ceph.osd.4.smithi187.stdout:Sep 21 14:12:33 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-4[37666]: 2024-09-21T14:12:33.419+0000 7fdf20a2e640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T14:12:33.694 INFO:journalctl@ceph.osd.4.smithi187.stdout:Sep 21 14:12:33 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-4[37666]: 2024-09-21T14:12:33.419+0000 7fdf20a2e640 -1 osd.4 582 *** Got signal Terminated *** 2024-09-21T14:12:33.695 INFO:journalctl@ceph.osd.4.smithi187.stdout:Sep 21 14:12:33 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-4[37666]: 2024-09-21T14:12:33.419+0000 7fdf20a2e640 -1 osd.4 582 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T14:12:34.446 INFO:journalctl@ceph.osd.4.smithi187.stdout:Sep 21 14:12:34 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-4[37666]: 2024-09-21T14:12:34.115+0000 7fdf1d043640 -1 osd.4 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:03.289750+0000 front 2024-09-21T14:12:03.290474+0000 (oldest deadline 2024-09-21T14:12:27.989684+0000) 2024-09-21T14:12:34.446 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:33 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T14:12:33.981+0000 7f3a3bf19640 -1 osd.6 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:04.825816+0000 front 2024-09-21T14:12:04.826007+0000 (oldest deadline 2024-09-21T14:12:30.125898+0000) 2024-09-21T14:12:34.446 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:33 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:33.994+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:05.454830+0000 front 2024-09-21T14:12:05.454963+0000 (oldest deadline 2024-09-21T14:12:28.355054+0000) 2024-09-21T14:12:34.944 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:12:34 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5[125175]: 2024-09-21T14:12:34.451+0000 7f29e26d7640 -1 osd.5 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:03.765259+0000 front 2024-09-21T14:12:03.765323+0000 (oldest deadline 2024-09-21T14:12:29.065105+0000) 2024-09-21T14:12:35.444 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:34 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T14:12:34.947+0000 7f3a3bf19640 -1 osd.6 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:04.825816+0000 front 2024-09-21T14:12:04.826007+0000 (oldest deadline 2024-09-21T14:12:30.125898+0000) 2024-09-21T14:12:35.444 INFO:journalctl@ceph.osd.4.smithi187.stdout:Sep 21 14:12:35 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-4[37666]: 2024-09-21T14:12:35.159+0000 7fdf1d043640 -1 osd.4 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:03.289750+0000 front 2024-09-21T14:12:03.290474+0000 (oldest deadline 2024-09-21T14:12:27.989684+0000) 2024-09-21T14:12:35.445 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:34 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:34.972+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:05.454830+0000 front 2024-09-21T14:12:05.454963+0000 (oldest deadline 2024-09-21T14:12:28.355054+0000) 2024-09-21T14:12:35.913 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:12:35 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5[125175]: 2024-09-21T14:12:35.493+0000 7f29e26d7640 -1 osd.5 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:03.765259+0000 front 2024-09-21T14:12:03.765323+0000 (oldest deadline 2024-09-21T14:12:29.065105+0000) 2024-09-21T14:12:35.913 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:12:35 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5[125175]: 2024-09-21T14:12:35.493+0000 7f29e26d7640 -1 osd.5 582 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-09-21T14:12:09.065605+0000 front 2024-09-21T14:12:09.065685+0000 (oldest deadline 2024-09-21T14:12:34.965557+0000) 2024-09-21T14:12:36.194 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:35 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T14:12:35.913+0000 7f3a3bf19640 -1 osd.6 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:04.825816+0000 front 2024-09-21T14:12:04.826007+0000 (oldest deadline 2024-09-21T14:12:30.125898+0000) 2024-09-21T14:12:36.194 INFO:journalctl@ceph.osd.4.smithi187.stdout:Sep 21 14:12:36 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-4[37666]: 2024-09-21T14:12:36.152+0000 7fdf1d043640 -1 osd.4 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:03.289750+0000 front 2024-09-21T14:12:03.290474+0000 (oldest deadline 2024-09-21T14:12:27.989684+0000) 2024-09-21T14:12:36.195 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:35 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:35.992+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:05.454830+0000 front 2024-09-21T14:12:05.454963+0000 (oldest deadline 2024-09-21T14:12:28.355054+0000) 2024-09-21T14:12:36.889 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:12:36 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5[125175]: 2024-09-21T14:12:36.486+0000 7f29e26d7640 -1 osd.5 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:03.765259+0000 front 2024-09-21T14:12:03.765323+0000 (oldest deadline 2024-09-21T14:12:29.065105+0000) 2024-09-21T14:12:36.889 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:12:36 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5[125175]: 2024-09-21T14:12:36.486+0000 7f29e26d7640 -1 osd.5 582 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-09-21T14:12:09.065605+0000 front 2024-09-21T14:12:09.065685+0000 (oldest deadline 2024-09-21T14:12:34.965557+0000) 2024-09-21T14:12:37.194 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:36 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T14:12:36.888+0000 7f3a3bf19640 -1 osd.6 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:04.825816+0000 front 2024-09-21T14:12:04.826007+0000 (oldest deadline 2024-09-21T14:12:30.125898+0000) 2024-09-21T14:12:37.194 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:36 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T14:12:36.888+0000 7f3a3bf19640 -1 osd.6 582 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-09-21T14:12:10.126808+0000 front 2024-09-21T14:12:10.126851+0000 (oldest deadline 2024-09-21T14:12:36.026360+0000) 2024-09-21T14:12:37.194 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:36 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:36.975+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:05.454830+0000 front 2024-09-21T14:12:05.454963+0000 (oldest deadline 2024-09-21T14:12:28.355054+0000) 2024-09-21T14:12:37.194 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:36 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:36.975+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-09-21T14:12:10.655909+0000 front 2024-09-21T14:12:10.656015+0000 (oldest deadline 2024-09-21T14:12:36.555950+0000) 2024-09-21T14:12:37.518 INFO:journalctl@ceph.osd.4.smithi187.stdout:Sep 21 14:12:37 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-4[37666]: 2024-09-21T14:12:37.200+0000 7fdf1d043640 -1 osd.4 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:03.289750+0000 front 2024-09-21T14:12:03.290474+0000 (oldest deadline 2024-09-21T14:12:27.989684+0000) 2024-09-21T14:12:37.850 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:12:37 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5[125175]: 2024-09-21T14:12:37.517+0000 7f29e26d7640 -1 osd.5 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:03.765259+0000 front 2024-09-21T14:12:03.765323+0000 (oldest deadline 2024-09-21T14:12:29.065105+0000) 2024-09-21T14:12:37.850 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:12:37 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5[125175]: 2024-09-21T14:12:37.517+0000 7f29e26d7640 -1 osd.5 582 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-09-21T14:12:09.065605+0000 front 2024-09-21T14:12:09.065685+0000 (oldest deadline 2024-09-21T14:12:34.965557+0000) 2024-09-21T14:12:38.167 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:37 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T14:12:37.850+0000 7f3a3bf19640 -1 osd.6 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:04.825816+0000 front 2024-09-21T14:12:04.826007+0000 (oldest deadline 2024-09-21T14:12:30.125898+0000) 2024-09-21T14:12:38.168 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:37 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T14:12:37.850+0000 7f3a3bf19640 -1 osd.6 582 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-09-21T14:12:10.126808+0000 front 2024-09-21T14:12:10.126851+0000 (oldest deadline 2024-09-21T14:12:36.026360+0000) 2024-09-21T14:12:38.168 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:38 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:38.022+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:05.454830+0000 front 2024-09-21T14:12:05.454963+0000 (oldest deadline 2024-09-21T14:12:28.355054+0000) 2024-09-21T14:12:38.168 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:38 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:38.022+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-09-21T14:12:10.655909+0000 front 2024-09-21T14:12:10.656015+0000 (oldest deadline 2024-09-21T14:12:36.555950+0000) 2024-09-21T14:12:38.444 INFO:journalctl@ceph.osd.4.smithi187.stdout:Sep 21 14:12:38 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-4[37666]: 2024-09-21T14:12:38.167+0000 7fdf1d043640 -1 osd.4 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:03.289750+0000 front 2024-09-21T14:12:03.290474+0000 (oldest deadline 2024-09-21T14:12:27.989684+0000) 2024-09-21T14:12:38.444 INFO:journalctl@ceph.osd.4.smithi187.stdout:Sep 21 14:12:38 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-4[37666]: 2024-09-21T14:12:38.167+0000 7fdf1d043640 -1 osd.4 582 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-09-21T14:12:12.090872+0000 front 2024-09-21T14:12:12.090697+0000 (oldest deadline 2024-09-21T14:12:37.990735+0000) 2024-09-21T14:12:38.808 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:12:38 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5[125175]: 2024-09-21T14:12:38.533+0000 7f29e26d7640 -1 osd.5 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:03.765259+0000 front 2024-09-21T14:12:03.765323+0000 (oldest deadline 2024-09-21T14:12:29.065105+0000) 2024-09-21T14:12:38.808 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:12:38 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5[125175]: 2024-09-21T14:12:38.534+0000 7f29e26d7640 -1 osd.5 582 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-09-21T14:12:09.065605+0000 front 2024-09-21T14:12:09.065685+0000 (oldest deadline 2024-09-21T14:12:34.965557+0000) 2024-09-21T14:12:39.069 INFO:journalctl@ceph.osd.4.smithi187.stdout:Sep 21 14:12:38 smithi187 podman[169590]: 2024-09-21 14:12:38.971339059 +0000 UTC m=+5.661434161 container died 148bbc28a3758fafa5a7c48a1b0a531a15f33d6c0dc890a254c2eb41f5abf41f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-4, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-09-21T14:12:39.070 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:38 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T14:12:38.807+0000 7f3a3bf19640 -1 osd.6 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:04.825816+0000 front 2024-09-21T14:12:04.826007+0000 (oldest deadline 2024-09-21T14:12:30.125898+0000) 2024-09-21T14:12:39.070 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:38 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T14:12:38.807+0000 7f3a3bf19640 -1 osd.6 582 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-09-21T14:12:10.126808+0000 front 2024-09-21T14:12:10.126851+0000 (oldest deadline 2024-09-21T14:12:36.026360+0000) 2024-09-21T14:12:39.434 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:39 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:39.069+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:05.454830+0000 front 2024-09-21T14:12:05.454963+0000 (oldest deadline 2024-09-21T14:12:28.355054+0000) 2024-09-21T14:12:39.434 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:39 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:39.069+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-09-21T14:12:10.655909+0000 front 2024-09-21T14:12:10.656015+0000 (oldest deadline 2024-09-21T14:12:36.555950+0000) 2024-09-21T14:12:39.694 INFO:journalctl@ceph.osd.4.smithi187.stdout:Sep 21 14:12:39 smithi187 podman[169590]: 2024-09-21 14:12:39.434234082 +0000 UTC m=+6.124329183 container cleanup 148bbc28a3758fafa5a7c48a1b0a531a15f33d6c0dc890a254c2eb41f5abf41f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, ceph=True, GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.vendor=CentOS) 2024-09-21T14:12:39.694 INFO:journalctl@ceph.osd.4.smithi187.stdout:Sep 21 14:12:39 smithi187 bash[169590]: ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-4 2024-09-21T14:12:39.694 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:12:39 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5[125175]: 2024-09-21T14:12:39.567+0000 7f29e26d7640 -1 osd.5 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:03.765259+0000 front 2024-09-21T14:12:03.765323+0000 (oldest deadline 2024-09-21T14:12:29.065105+0000) 2024-09-21T14:12:39.694 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:12:39 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5[125175]: 2024-09-21T14:12:39.567+0000 7f29e26d7640 -1 osd.5 582 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-09-21T14:12:09.065605+0000 front 2024-09-21T14:12:09.065685+0000 (oldest deadline 2024-09-21T14:12:34.965557+0000) 2024-09-21T14:12:40.035 INFO:journalctl@ceph.osd.4.smithi187.stdout:Sep 21 14:12:39 smithi187 podman[169613]: 2024-09-21 14:12:39.725397851 +0000 UTC m=+0.748444915 container remove 148bbc28a3758fafa5a7c48a1b0a531a15f33d6c0dc890a254c2eb41f5abf41f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-4, io.buildah.version=1.37.2, RELEASE=wip-guits-main-2024-09-21-0842-4417849, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-21T14:12:40.035 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:39 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T14:12:39.787+0000 7f3a3bf19640 -1 osd.6 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:04.825816+0000 front 2024-09-21T14:12:04.826007+0000 (oldest deadline 2024-09-21T14:12:30.125898+0000) 2024-09-21T14:12:40.035 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:39 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T14:12:39.787+0000 7f3a3bf19640 -1 osd.6 582 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-09-21T14:12:10.126808+0000 front 2024-09-21T14:12:10.126851+0000 (oldest deadline 2024-09-21T14:12:36.026360+0000) 2024-09-21T14:12:40.286 INFO:journalctl@ceph.osd.4.smithi187.stdout:Sep 21 14:12:40 smithi187 podman[169710]: 2024-09-21 14:12:39.938759576 +0000 UTC m=+0.016155670 image pull b415816db8b41f544cdfdc71d193ff8b1ee008bf7b0ec4f64da8eb8f71b62e85 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5 2024-09-21T14:12:40.286 INFO:journalctl@ceph.osd.4.smithi187.stdout:Sep 21 14:12:40 smithi187 podman[169710]: 2024-09-21 14:12:40.04546386 +0000 UTC m=+0.122859978 container create 9175d6cef1a3ab2cde3284d6be243c2228c192338f34425c6612b492bcd820d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-4-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-main-2024-09-21-0842-4417849) 2024-09-21T14:12:40.286 INFO:journalctl@ceph.osd.4.smithi187.stdout:Sep 21 14:12:40 smithi187 podman[169710]: 2024-09-21 14:12:40.281237019 +0000 UTC m=+0.358633104 container init 9175d6cef1a3ab2cde3284d6be243c2228c192338f34425c6612b492bcd820d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-21-0842-4417849) 2024-09-21T14:12:40.286 INFO:journalctl@ceph.osd.4.smithi187.stdout:Sep 21 14:12:40 smithi187 podman[169710]: 2024-09-21 14:12:40.286418345 +0000 UTC m=+0.363814422 container start 9175d6cef1a3ab2cde3284d6be243c2228c192338f34425c6612b492bcd820d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-4-deactivate, GIT_BRANCH=HEAD, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-09-21T14:12:40.286 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:40 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:40.092+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:05.454830+0000 front 2024-09-21T14:12:05.454963+0000 (oldest deadline 2024-09-21T14:12:28.355054+0000) 2024-09-21T14:12:40.287 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:40 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:40.092+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-09-21T14:12:10.655909+0000 front 2024-09-21T14:12:10.656015+0000 (oldest deadline 2024-09-21T14:12:36.555950+0000) 2024-09-21T14:12:40.541 INFO:journalctl@ceph.osd.4.smithi187.stdout:Sep 21 14:12:40 smithi187 podman[169710]: 2024-09-21 14:12:40.414749294 +0000 UTC m=+0.492145379 container attach 9175d6cef1a3ab2cde3284d6be243c2228c192338f34425c6612b492bcd820d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-guits-main-2024-09-21-0842-4417849, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T14:12:40.813 INFO:journalctl@ceph.osd.4.smithi187.stdout:Sep 21 14:12:40 smithi187 podman[169710]: 2024-09-21 14:12:40.542023242 +0000 UTC m=+0.619419326 container died 9175d6cef1a3ab2cde3284d6be243c2228c192338f34425c6612b492bcd820d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-4-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-main-2024-09-21-0842-4417849, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2) 2024-09-21T14:12:40.813 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:12:40 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5[125175]: 2024-09-21T14:12:40.610+0000 7f29e26d7640 -1 osd.5 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:03.765259+0000 front 2024-09-21T14:12:03.765323+0000 (oldest deadline 2024-09-21T14:12:29.065105+0000) 2024-09-21T14:12:40.813 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:12:40 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5[125175]: 2024-09-21T14:12:40.610+0000 7f29e26d7640 -1 osd.5 582 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-09-21T14:12:09.065605+0000 front 2024-09-21T14:12:09.065685+0000 (oldest deadline 2024-09-21T14:12:34.965557+0000) 2024-09-21T14:12:41.142 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:40 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T14:12:40.812+0000 7f3a3bf19640 -1 osd.6 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:04.825816+0000 front 2024-09-21T14:12:04.826007+0000 (oldest deadline 2024-09-21T14:12:30.125898+0000) 2024-09-21T14:12:41.142 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:40 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T14:12:40.812+0000 7f3a3bf19640 -1 osd.6 582 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-09-21T14:12:10.126808+0000 front 2024-09-21T14:12:10.126851+0000 (oldest deadline 2024-09-21T14:12:36.026360+0000) 2024-09-21T14:12:41.143 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:41 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:41.054+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:05.454830+0000 front 2024-09-21T14:12:05.454963+0000 (oldest deadline 2024-09-21T14:12:28.355054+0000) 2024-09-21T14:12:41.143 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:41 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:41.054+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-09-21T14:12:10.655909+0000 front 2024-09-21T14:12:10.656015+0000 (oldest deadline 2024-09-21T14:12:36.555950+0000) 2024-09-21T14:12:41.166 DEBUG:teuthology.orchestra.run.smithi187:> sudo pkill -f 'journalctl -f -n 0 -u ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@osd.4.service' 2024-09-21T14:12:41.207 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T14:12:41.207 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-21T14:12:41.207 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-21T14:12:41.207 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@osd.5 2024-09-21T14:12:41.444 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:12:41 smithi187 systemd[1]: Stopping Ceph osd.5 for 40a3d2e8-781f-11ef-baf6-efdc52797490... 2024-09-21T14:12:41.944 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:41 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T14:12:41.781+0000 7f3a3bf19640 -1 osd.6 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:04.825816+0000 front 2024-09-21T14:12:04.826007+0000 (oldest deadline 2024-09-21T14:12:30.125898+0000) 2024-09-21T14:12:41.944 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:41 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T14:12:41.781+0000 7f3a3bf19640 -1 osd.6 582 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-09-21T14:12:10.126808+0000 front 2024-09-21T14:12:10.126851+0000 (oldest deadline 2024-09-21T14:12:36.026360+0000) 2024-09-21T14:12:41.944 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:12:41 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5[125175]: 2024-09-21T14:12:41.565+0000 7f29ea0ca640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T14:12:41.945 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:12:41 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5[125175]: 2024-09-21T14:12:41.565+0000 7f29ea0ca640 -1 osd.5 582 *** Got signal Terminated *** 2024-09-21T14:12:41.945 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:12:41 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5[125175]: 2024-09-21T14:12:41.565+0000 7f29ea0ca640 -1 osd.5 582 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T14:12:41.945 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:12:41 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5[125175]: 2024-09-21T14:12:41.629+0000 7f29e26d7640 -1 osd.5 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:03.765259+0000 front 2024-09-21T14:12:03.765323+0000 (oldest deadline 2024-09-21T14:12:29.065105+0000) 2024-09-21T14:12:41.945 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:12:41 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5[125175]: 2024-09-21T14:12:41.629+0000 7f29e26d7640 -1 osd.5 582 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-09-21T14:12:09.065605+0000 front 2024-09-21T14:12:09.065685+0000 (oldest deadline 2024-09-21T14:12:34.965557+0000) 2024-09-21T14:12:42.444 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:42 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:42.072+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:05.454830+0000 front 2024-09-21T14:12:05.454963+0000 (oldest deadline 2024-09-21T14:12:28.355054+0000) 2024-09-21T14:12:42.444 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:42 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:42.072+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-09-21T14:12:10.655909+0000 front 2024-09-21T14:12:10.656015+0000 (oldest deadline 2024-09-21T14:12:36.555950+0000) 2024-09-21T14:12:42.944 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:42 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T14:12:42.777+0000 7f3a3bf19640 -1 osd.6 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:04.825816+0000 front 2024-09-21T14:12:04.826007+0000 (oldest deadline 2024-09-21T14:12:30.125898+0000) 2024-09-21T14:12:42.944 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:42 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T14:12:42.777+0000 7f3a3bf19640 -1 osd.6 582 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-09-21T14:12:10.126808+0000 front 2024-09-21T14:12:10.126851+0000 (oldest deadline 2024-09-21T14:12:36.026360+0000) 2024-09-21T14:12:42.944 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:12:42 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5[125175]: 2024-09-21T14:12:42.596+0000 7f29e26d7640 -1 osd.5 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:03.765259+0000 front 2024-09-21T14:12:03.765323+0000 (oldest deadline 2024-09-21T14:12:29.065105+0000) 2024-09-21T14:12:42.945 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:12:42 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5[125175]: 2024-09-21T14:12:42.596+0000 7f29e26d7640 -1 osd.5 582 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-09-21T14:12:09.065605+0000 front 2024-09-21T14:12:09.065685+0000 (oldest deadline 2024-09-21T14:12:34.965557+0000) 2024-09-21T14:12:43.444 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:43 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:43.088+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:05.454830+0000 front 2024-09-21T14:12:05.454963+0000 (oldest deadline 2024-09-21T14:12:28.355054+0000) 2024-09-21T14:12:43.444 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:43 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:43.088+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-09-21T14:12:10.655909+0000 front 2024-09-21T14:12:10.656015+0000 (oldest deadline 2024-09-21T14:12:36.555950+0000) 2024-09-21T14:12:43.444 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:43 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:43.088+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-09-21T14:12:17.656866+0000 front 2024-09-21T14:12:17.656877+0000 (oldest deadline 2024-09-21T14:12:42.956816+0000) 2024-09-21T14:12:43.944 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:12:43 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5[125175]: 2024-09-21T14:12:43.618+0000 7f29e26d7640 -1 osd.5 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:03.765259+0000 front 2024-09-21T14:12:03.765323+0000 (oldest deadline 2024-09-21T14:12:29.065105+0000) 2024-09-21T14:12:43.944 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:12:43 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5[125175]: 2024-09-21T14:12:43.618+0000 7f29e26d7640 -1 osd.5 582 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-09-21T14:12:09.065605+0000 front 2024-09-21T14:12:09.065685+0000 (oldest deadline 2024-09-21T14:12:34.965557+0000) 2024-09-21T14:12:43.945 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:43 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T14:12:43.729+0000 7f3a3bf19640 -1 osd.6 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:04.825816+0000 front 2024-09-21T14:12:04.826007+0000 (oldest deadline 2024-09-21T14:12:30.125898+0000) 2024-09-21T14:12:43.945 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:43 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T14:12:43.729+0000 7f3a3bf19640 -1 osd.6 582 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-09-21T14:12:10.126808+0000 front 2024-09-21T14:12:10.126851+0000 (oldest deadline 2024-09-21T14:12:36.026360+0000) 2024-09-21T14:12:44.444 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:44 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:44.055+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:05.454830+0000 front 2024-09-21T14:12:05.454963+0000 (oldest deadline 2024-09-21T14:12:28.355054+0000) 2024-09-21T14:12:44.444 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:44 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:44.055+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-09-21T14:12:10.655909+0000 front 2024-09-21T14:12:10.656015+0000 (oldest deadline 2024-09-21T14:12:36.555950+0000) 2024-09-21T14:12:44.445 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:44 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:44.055+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-09-21T14:12:17.656866+0000 front 2024-09-21T14:12:17.656877+0000 (oldest deadline 2024-09-21T14:12:42.956816+0000) 2024-09-21T14:12:44.944 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:12:44 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5[125175]: 2024-09-21T14:12:44.662+0000 7f29e26d7640 -1 osd.5 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:03.765259+0000 front 2024-09-21T14:12:03.765323+0000 (oldest deadline 2024-09-21T14:12:29.065105+0000) 2024-09-21T14:12:44.944 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:12:44 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5[125175]: 2024-09-21T14:12:44.662+0000 7f29e26d7640 -1 osd.5 582 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-09-21T14:12:09.065605+0000 front 2024-09-21T14:12:09.065685+0000 (oldest deadline 2024-09-21T14:12:34.965557+0000) 2024-09-21T14:12:44.945 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:44 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T14:12:44.735+0000 7f3a3bf19640 -1 osd.6 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:04.825816+0000 front 2024-09-21T14:12:04.826007+0000 (oldest deadline 2024-09-21T14:12:30.125898+0000) 2024-09-21T14:12:44.945 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:44 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T14:12:44.735+0000 7f3a3bf19640 -1 osd.6 582 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-09-21T14:12:10.126808+0000 front 2024-09-21T14:12:10.126851+0000 (oldest deadline 2024-09-21T14:12:36.026360+0000) 2024-09-21T14:12:44.945 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:44 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T14:12:44.735+0000 7f3a3bf19640 -1 osd.6 582 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-09-21T14:12:18.927470+0000 front 2024-09-21T14:12:18.927486+0000 (oldest deadline 2024-09-21T14:12:44.227224+0000) 2024-09-21T14:12:45.444 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:45 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:45.027+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:05.454830+0000 front 2024-09-21T14:12:05.454963+0000 (oldest deadline 2024-09-21T14:12:28.355054+0000) 2024-09-21T14:12:45.444 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:45 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:45.027+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-09-21T14:12:10.655909+0000 front 2024-09-21T14:12:10.656015+0000 (oldest deadline 2024-09-21T14:12:36.555950+0000) 2024-09-21T14:12:45.444 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:45 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:45.027+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-09-21T14:12:17.656866+0000 front 2024-09-21T14:12:17.656877+0000 (oldest deadline 2024-09-21T14:12:42.956816+0000) 2024-09-21T14:12:45.944 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:12:45 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5[125175]: 2024-09-21T14:12:45.679+0000 7f29e26d7640 -1 osd.5 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:03.765259+0000 front 2024-09-21T14:12:03.765323+0000 (oldest deadline 2024-09-21T14:12:29.065105+0000) 2024-09-21T14:12:45.944 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:12:45 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5[125175]: 2024-09-21T14:12:45.679+0000 7f29e26d7640 -1 osd.5 582 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-09-21T14:12:09.065605+0000 front 2024-09-21T14:12:09.065685+0000 (oldest deadline 2024-09-21T14:12:34.965557+0000) 2024-09-21T14:12:45.944 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:45 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T14:12:45.689+0000 7f3a3bf19640 -1 osd.6 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:04.825816+0000 front 2024-09-21T14:12:04.826007+0000 (oldest deadline 2024-09-21T14:12:30.125898+0000) 2024-09-21T14:12:45.945 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:45 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T14:12:45.689+0000 7f3a3bf19640 -1 osd.6 582 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-09-21T14:12:10.126808+0000 front 2024-09-21T14:12:10.126851+0000 (oldest deadline 2024-09-21T14:12:36.026360+0000) 2024-09-21T14:12:45.945 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:45 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T14:12:45.689+0000 7f3a3bf19640 -1 osd.6 582 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-09-21T14:12:18.927470+0000 front 2024-09-21T14:12:18.927486+0000 (oldest deadline 2024-09-21T14:12:44.227224+0000) 2024-09-21T14:12:46.444 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:46 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:45.999+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:05.454830+0000 front 2024-09-21T14:12:05.454963+0000 (oldest deadline 2024-09-21T14:12:28.355054+0000) 2024-09-21T14:12:46.444 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:46 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:45.999+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-09-21T14:12:10.655909+0000 front 2024-09-21T14:12:10.656015+0000 (oldest deadline 2024-09-21T14:12:36.555950+0000) 2024-09-21T14:12:46.444 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:46 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:45.999+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-09-21T14:12:17.656866+0000 front 2024-09-21T14:12:17.656877+0000 (oldest deadline 2024-09-21T14:12:42.956816+0000) 2024-09-21T14:12:46.445 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:46 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:45.999+0000 7f5484fe7640 -1 osd.7 582 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24442.0:1383 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e582) 2024-09-21T14:12:46.944 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:46 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T14:12:46.678+0000 7f3a3bf19640 -1 osd.6 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:04.825816+0000 front 2024-09-21T14:12:04.826007+0000 (oldest deadline 2024-09-21T14:12:30.125898+0000) 2024-09-21T14:12:46.944 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:46 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T14:12:46.678+0000 7f3a3bf19640 -1 osd.6 582 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-09-21T14:12:10.126808+0000 front 2024-09-21T14:12:10.126851+0000 (oldest deadline 2024-09-21T14:12:36.026360+0000) 2024-09-21T14:12:46.944 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:46 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T14:12:46.678+0000 7f3a3bf19640 -1 osd.6 582 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-09-21T14:12:18.927470+0000 front 2024-09-21T14:12:18.927486+0000 (oldest deadline 2024-09-21T14:12:44.227224+0000) 2024-09-21T14:12:47.254 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:12:46 smithi187 podman[169840]: 2024-09-21 14:12:46.972856751 +0000 UTC m=+5.535770920 container died 1fc4fc5f79021f071b26a40fca744526db585ab2a34443ff1cbbccb5272ff339 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-guits-main-2024-09-21-0842-4417849, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T14:12:47.255 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:47 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:47.042+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:05.454830+0000 front 2024-09-21T14:12:05.454963+0000 (oldest deadline 2024-09-21T14:12:28.355054+0000) 2024-09-21T14:12:47.255 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:47 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:47.042+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-09-21T14:12:10.655909+0000 front 2024-09-21T14:12:10.656015+0000 (oldest deadline 2024-09-21T14:12:36.555950+0000) 2024-09-21T14:12:47.255 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:47 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:47.042+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-09-21T14:12:17.656866+0000 front 2024-09-21T14:12:17.656877+0000 (oldest deadline 2024-09-21T14:12:42.956816+0000) 2024-09-21T14:12:47.255 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:47 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:47.043+0000 7f5484fe7640 -1 osd.7 582 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24442.0:1383 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e582) 2024-09-21T14:12:47.639 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:12:47 smithi187 podman[169840]: 2024-09-21 14:12:47.438613572 +0000 UTC m=+6.001527727 container cleanup 1fc4fc5f79021f071b26a40fca744526db585ab2a34443ff1cbbccb5272ff339 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T14:12:47.639 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:12:47 smithi187 bash[169840]: ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5 2024-09-21T14:12:47.944 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:12:47 smithi187 podman[169865]: 2024-09-21 14:12:47.640005035 +0000 UTC m=+0.634977833 container remove 1fc4fc5f79021f071b26a40fca744526db585ab2a34443ff1cbbccb5272ff339 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, ceph=True) 2024-09-21T14:12:47.944 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:47 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T14:12:47.652+0000 7f3a3bf19640 -1 osd.6 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:04.825816+0000 front 2024-09-21T14:12:04.826007+0000 (oldest deadline 2024-09-21T14:12:30.125898+0000) 2024-09-21T14:12:47.944 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:47 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T14:12:47.652+0000 7f3a3bf19640 -1 osd.6 582 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-09-21T14:12:10.126808+0000 front 2024-09-21T14:12:10.126851+0000 (oldest deadline 2024-09-21T14:12:36.026360+0000) 2024-09-21T14:12:47.944 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:47 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T14:12:47.652+0000 7f3a3bf19640 -1 osd.6 582 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-09-21T14:12:18.927470+0000 front 2024-09-21T14:12:18.927486+0000 (oldest deadline 2024-09-21T14:12:44.227224+0000) 2024-09-21T14:12:48.292 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:47 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:47.994+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:05.454830+0000 front 2024-09-21T14:12:05.454963+0000 (oldest deadline 2024-09-21T14:12:28.355054+0000) 2024-09-21T14:12:48.293 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:47 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:47.994+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-09-21T14:12:10.655909+0000 front 2024-09-21T14:12:10.656015+0000 (oldest deadline 2024-09-21T14:12:36.555950+0000) 2024-09-21T14:12:48.293 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:47 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:47.994+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-09-21T14:12:17.656866+0000 front 2024-09-21T14:12:17.656877+0000 (oldest deadline 2024-09-21T14:12:42.956816+0000) 2024-09-21T14:12:48.293 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:47 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:47.994+0000 7f5484fe7640 -1 osd.7 582 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24442.0:1383 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e582) 2024-09-21T14:12:48.293 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:12:48 smithi187 podman[169958]: 2024-09-21 14:12:48.04927598 +0000 UTC m=+0.111170903 container create 811d18803a34e2a8049eb4c6071ffc09c67bada80847088dd36e6032ae234d3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, ceph=True, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T14:12:48.293 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:12:48 smithi187 podman[169958]: 2024-09-21 14:12:47.954224935 +0000 UTC m=+0.016119825 image pull b415816db8b41f544cdfdc71d193ff8b1ee008bf7b0ec4f64da8eb8f71b62e85 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5 2024-09-21T14:12:48.293 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:12:48 smithi187 podman[169958]: 2024-09-21 14:12:48.293082738 +0000 UTC m=+0.354977620 container init 811d18803a34e2a8049eb4c6071ffc09c67bada80847088dd36e6032ae234d3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, ceph=True, RELEASE=wip-guits-main-2024-09-21-0842-4417849) 2024-09-21T14:12:48.636 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:12:48 smithi187 podman[169958]: 2024-09-21 14:12:48.298482134 +0000 UTC m=+0.360377022 container start 811d18803a34e2a8049eb4c6071ffc09c67bada80847088dd36e6032ae234d3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5-deactivate, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2) 2024-09-21T14:12:48.636 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:12:48 smithi187 podman[169958]: 2024-09-21 14:12:48.393251126 +0000 UTC m=+0.455146021 container attach 811d18803a34e2a8049eb4c6071ffc09c67bada80847088dd36e6032ae234d3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-09-21T14:12:48.636 INFO:journalctl@ceph.osd.5.smithi187.stdout:Sep 21 14:12:48 smithi187 podman[169958]: 2024-09-21 14:12:48.499858114 +0000 UTC m=+0.561753007 container died 811d18803a34e2a8049eb4c6071ffc09c67bada80847088dd36e6032ae234d3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-5-deactivate, CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, ceph=True, GIT_CLEAN=True) 2024-09-21T14:12:48.944 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:48 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T14:12:48.635+0000 7f3a3bf19640 -1 osd.6 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:04.825816+0000 front 2024-09-21T14:12:04.826007+0000 (oldest deadline 2024-09-21T14:12:30.125898+0000) 2024-09-21T14:12:48.944 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:48 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T14:12:48.635+0000 7f3a3bf19640 -1 osd.6 582 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-09-21T14:12:10.126808+0000 front 2024-09-21T14:12:10.126851+0000 (oldest deadline 2024-09-21T14:12:36.026360+0000) 2024-09-21T14:12:48.944 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:48 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T14:12:48.635+0000 7f3a3bf19640 -1 osd.6 582 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-09-21T14:12:18.927470+0000 front 2024-09-21T14:12:18.927486+0000 (oldest deadline 2024-09-21T14:12:44.227224+0000) 2024-09-21T14:12:49.183 DEBUG:teuthology.orchestra.run.smithi187:> sudo pkill -f 'journalctl -f -n 0 -u ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@osd.5.service' 2024-09-21T14:12:49.224 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T14:12:49.224 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-21T14:12:49.224 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-21T14:12:49.224 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@osd.6 2024-09-21T14:12:49.286 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:48 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:48.981+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:05.454830+0000 front 2024-09-21T14:12:05.454963+0000 (oldest deadline 2024-09-21T14:12:28.355054+0000) 2024-09-21T14:12:49.286 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:48 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:48.981+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-09-21T14:12:10.655909+0000 front 2024-09-21T14:12:10.656015+0000 (oldest deadline 2024-09-21T14:12:36.555950+0000) 2024-09-21T14:12:49.286 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:48 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:48.981+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-09-21T14:12:17.656866+0000 front 2024-09-21T14:12:17.656877+0000 (oldest deadline 2024-09-21T14:12:42.956816+0000) 2024-09-21T14:12:49.552 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:49 smithi187 systemd[1]: Stopping Ceph osd.6 for 40a3d2e8-781f-11ef-baf6-efdc52797490... 2024-09-21T14:12:49.944 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:49 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T14:12:49.551+0000 7f3a3f904640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T14:12:49.944 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:49 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T14:12:49.551+0000 7f3a3f904640 -1 osd.6 582 *** Got signal Terminated *** 2024-09-21T14:12:49.944 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:49 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T14:12:49.551+0000 7f3a3f904640 -1 osd.6 582 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T14:12:49.944 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:49 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T14:12:49.679+0000 7f3a3bf19640 -1 osd.6 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:04.825816+0000 front 2024-09-21T14:12:04.826007+0000 (oldest deadline 2024-09-21T14:12:30.125898+0000) 2024-09-21T14:12:49.944 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:49 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T14:12:49.679+0000 7f3a3bf19640 -1 osd.6 582 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-09-21T14:12:10.126808+0000 front 2024-09-21T14:12:10.126851+0000 (oldest deadline 2024-09-21T14:12:36.026360+0000) 2024-09-21T14:12:49.944 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:49 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T14:12:49.679+0000 7f3a3bf19640 -1 osd.6 582 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-09-21T14:12:18.927470+0000 front 2024-09-21T14:12:18.927486+0000 (oldest deadline 2024-09-21T14:12:44.227224+0000) 2024-09-21T14:12:50.444 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:49 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:49.950+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:05.454830+0000 front 2024-09-21T14:12:05.454963+0000 (oldest deadline 2024-09-21T14:12:28.355054+0000) 2024-09-21T14:12:50.444 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:49 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:49.950+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-09-21T14:12:10.655909+0000 front 2024-09-21T14:12:10.656015+0000 (oldest deadline 2024-09-21T14:12:36.555950+0000) 2024-09-21T14:12:50.444 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:49 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:49.950+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-09-21T14:12:17.656866+0000 front 2024-09-21T14:12:17.656877+0000 (oldest deadline 2024-09-21T14:12:42.956816+0000) 2024-09-21T14:12:50.944 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:50 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T14:12:50.674+0000 7f3a3bf19640 -1 osd.6 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:04.825816+0000 front 2024-09-21T14:12:04.826007+0000 (oldest deadline 2024-09-21T14:12:30.125898+0000) 2024-09-21T14:12:50.944 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:50 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T14:12:50.674+0000 7f3a3bf19640 -1 osd.6 582 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-09-21T14:12:10.126808+0000 front 2024-09-21T14:12:10.126851+0000 (oldest deadline 2024-09-21T14:12:36.026360+0000) 2024-09-21T14:12:50.944 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:50 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T14:12:50.674+0000 7f3a3bf19640 -1 osd.6 582 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-09-21T14:12:18.927470+0000 front 2024-09-21T14:12:18.927486+0000 (oldest deadline 2024-09-21T14:12:44.227224+0000) 2024-09-21T14:12:50.944 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:50 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:50.905+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:05.454830+0000 front 2024-09-21T14:12:05.454963+0000 (oldest deadline 2024-09-21T14:12:28.355054+0000) 2024-09-21T14:12:50.945 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:50 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:50.905+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-09-21T14:12:10.655909+0000 front 2024-09-21T14:12:10.656015+0000 (oldest deadline 2024-09-21T14:12:36.555950+0000) 2024-09-21T14:12:50.945 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:50 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:50.905+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-09-21T14:12:17.656866+0000 front 2024-09-21T14:12:17.656877+0000 (oldest deadline 2024-09-21T14:12:42.956816+0000) 2024-09-21T14:12:51.944 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:51 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T14:12:51.669+0000 7f3a3bf19640 -1 osd.6 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:04.825816+0000 front 2024-09-21T14:12:04.826007+0000 (oldest deadline 2024-09-21T14:12:30.125898+0000) 2024-09-21T14:12:51.944 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:51 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T14:12:51.669+0000 7f3a3bf19640 -1 osd.6 582 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-09-21T14:12:10.126808+0000 front 2024-09-21T14:12:10.126851+0000 (oldest deadline 2024-09-21T14:12:36.026360+0000) 2024-09-21T14:12:51.944 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:51 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T14:12:51.669+0000 7f3a3bf19640 -1 osd.6 582 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-09-21T14:12:18.927470+0000 front 2024-09-21T14:12:18.927486+0000 (oldest deadline 2024-09-21T14:12:44.227224+0000) 2024-09-21T14:12:51.944 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:51 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T14:12:51.669+0000 7f3a3bf19640 -1 osd.6 582 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-09-21T14:12:29.428808+0000 front 2024-09-21T14:12:29.428794+0000 (oldest deadline 2024-09-21T14:12:51.128472+0000) 2024-09-21T14:12:52.444 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:51 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:51.946+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:05.454830+0000 front 2024-09-21T14:12:05.454963+0000 (oldest deadline 2024-09-21T14:12:28.355054+0000) 2024-09-21T14:12:52.444 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:51 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:51.946+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-09-21T14:12:10.655909+0000 front 2024-09-21T14:12:10.656015+0000 (oldest deadline 2024-09-21T14:12:36.555950+0000) 2024-09-21T14:12:52.444 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:51 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:51.946+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-09-21T14:12:17.656866+0000 front 2024-09-21T14:12:17.656877+0000 (oldest deadline 2024-09-21T14:12:42.956816+0000) 2024-09-21T14:12:52.444 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:51 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:51.946+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-09-21T14:12:28.658748+0000 front 2024-09-21T14:12:28.658762+0000 (oldest deadline 2024-09-21T14:12:50.958669+0000) 2024-09-21T14:12:52.944 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:52 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T14:12:52.668+0000 7f3a3bf19640 -1 osd.6 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:04.825816+0000 front 2024-09-21T14:12:04.826007+0000 (oldest deadline 2024-09-21T14:12:30.125898+0000) 2024-09-21T14:12:52.944 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:52 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T14:12:52.668+0000 7f3a3bf19640 -1 osd.6 582 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-09-21T14:12:10.126808+0000 front 2024-09-21T14:12:10.126851+0000 (oldest deadline 2024-09-21T14:12:36.026360+0000) 2024-09-21T14:12:52.944 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:52 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T14:12:52.668+0000 7f3a3bf19640 -1 osd.6 582 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-09-21T14:12:18.927470+0000 front 2024-09-21T14:12:18.927486+0000 (oldest deadline 2024-09-21T14:12:44.227224+0000) 2024-09-21T14:12:52.944 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:52 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T14:12:52.668+0000 7f3a3bf19640 -1 osd.6 582 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-09-21T14:12:29.428808+0000 front 2024-09-21T14:12:29.428794+0000 (oldest deadline 2024-09-21T14:12:51.128472+0000) 2024-09-21T14:12:53.444 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:52 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:52.986+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:05.454830+0000 front 2024-09-21T14:12:05.454963+0000 (oldest deadline 2024-09-21T14:12:28.355054+0000) 2024-09-21T14:12:53.444 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:52 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:52.986+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-09-21T14:12:10.655909+0000 front 2024-09-21T14:12:10.656015+0000 (oldest deadline 2024-09-21T14:12:36.555950+0000) 2024-09-21T14:12:53.445 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:52 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:52.986+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-09-21T14:12:17.656866+0000 front 2024-09-21T14:12:17.656877+0000 (oldest deadline 2024-09-21T14:12:42.956816+0000) 2024-09-21T14:12:53.445 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:52 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:52.986+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-09-21T14:12:28.658748+0000 front 2024-09-21T14:12:28.658762+0000 (oldest deadline 2024-09-21T14:12:50.958669+0000) 2024-09-21T14:12:54.018 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:53 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T14:12:53.712+0000 7f3a3bf19640 -1 osd.6 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:04.825816+0000 front 2024-09-21T14:12:04.826007+0000 (oldest deadline 2024-09-21T14:12:30.125898+0000) 2024-09-21T14:12:54.018 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:53 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T14:12:53.712+0000 7f3a3bf19640 -1 osd.6 582 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-09-21T14:12:10.126808+0000 front 2024-09-21T14:12:10.126851+0000 (oldest deadline 2024-09-21T14:12:36.026360+0000) 2024-09-21T14:12:54.018 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:53 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T14:12:53.712+0000 7f3a3bf19640 -1 osd.6 582 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-09-21T14:12:18.927470+0000 front 2024-09-21T14:12:18.927486+0000 (oldest deadline 2024-09-21T14:12:44.227224+0000) 2024-09-21T14:12:54.018 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:53 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6[49583]: 2024-09-21T14:12:53.712+0000 7f3a3bf19640 -1 osd.6 582 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-09-21T14:12:29.428808+0000 front 2024-09-21T14:12:29.428794+0000 (oldest deadline 2024-09-21T14:12:51.128472+0000) 2024-09-21T14:12:54.444 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:54 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:54.017+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:05.454830+0000 front 2024-09-21T14:12:05.454963+0000 (oldest deadline 2024-09-21T14:12:28.355054+0000) 2024-09-21T14:12:54.444 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:54 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:54.017+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-09-21T14:12:10.655909+0000 front 2024-09-21T14:12:10.656015+0000 (oldest deadline 2024-09-21T14:12:36.555950+0000) 2024-09-21T14:12:54.444 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:54 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:54.017+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-09-21T14:12:17.656866+0000 front 2024-09-21T14:12:17.656877+0000 (oldest deadline 2024-09-21T14:12:42.956816+0000) 2024-09-21T14:12:54.444 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:54 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:54.017+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-09-21T14:12:28.658748+0000 front 2024-09-21T14:12:28.658762+0000 (oldest deadline 2024-09-21T14:12:50.958669+0000) 2024-09-21T14:12:55.379 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:55 smithi187 podman[170085]: 2024-09-21 14:12:55.088202201 +0000 UTC m=+5.639351927 container died 331720c3d778d193371fafa4242dba7e253b87ca840e19ceb0aaafa73546acc5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-guits-main-2024-09-21-0842-4417849, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-21T14:12:55.379 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:54 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:54.974+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:05.454830+0000 front 2024-09-21T14:12:05.454963+0000 (oldest deadline 2024-09-21T14:12:28.355054+0000) 2024-09-21T14:12:55.379 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:54 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:54.974+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-09-21T14:12:10.655909+0000 front 2024-09-21T14:12:10.656015+0000 (oldest deadline 2024-09-21T14:12:36.555950+0000) 2024-09-21T14:12:55.379 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:54 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:54.974+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-09-21T14:12:17.656866+0000 front 2024-09-21T14:12:17.656877+0000 (oldest deadline 2024-09-21T14:12:42.956816+0000) 2024-09-21T14:12:55.379 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:54 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:54.974+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-09-21T14:12:28.658748+0000 front 2024-09-21T14:12:28.658762+0000 (oldest deadline 2024-09-21T14:12:50.958669+0000) 2024-09-21T14:12:55.694 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:55 smithi187 podman[170085]: 2024-09-21 14:12:55.583187392 +0000 UTC m=+6.134337117 container cleanup 331720c3d778d193371fafa4242dba7e253b87ca840e19ceb0aaafa73546acc5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-21-0842-4417849, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-21T14:12:55.694 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:55 smithi187 bash[170085]: ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6 2024-09-21T14:12:56.185 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:55 smithi187 podman[170104]: 2024-09-21 14:12:55.851601771 +0000 UTC m=+0.755465184 container remove 331720c3d778d193371fafa4242dba7e253b87ca840e19ceb0aaafa73546acc5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-21-0842-4417849) 2024-09-21T14:12:56.186 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:55 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:55.996+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:05.454830+0000 front 2024-09-21T14:12:05.454963+0000 (oldest deadline 2024-09-21T14:12:28.355054+0000) 2024-09-21T14:12:56.186 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:55 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:55.996+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-09-21T14:12:10.655909+0000 front 2024-09-21T14:12:10.656015+0000 (oldest deadline 2024-09-21T14:12:36.555950+0000) 2024-09-21T14:12:56.186 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:55 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:55.996+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-09-21T14:12:17.656866+0000 front 2024-09-21T14:12:17.656877+0000 (oldest deadline 2024-09-21T14:12:42.956816+0000) 2024-09-21T14:12:56.186 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:55 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:55.996+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-09-21T14:12:28.658748+0000 front 2024-09-21T14:12:28.658762+0000 (oldest deadline 2024-09-21T14:12:50.958669+0000) 2024-09-21T14:12:56.444 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:56 smithi187 podman[170199]: 2024-09-21 14:12:56.185885783 +0000 UTC m=+0.105462843 container create 9fb6c74c588eb04a4a6adfba4ccc52371836550d4539a3944c548d079a953198 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6-deactivate, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-guits-main-2024-09-21-0842-4417849) 2024-09-21T14:12:56.444 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:56 smithi187 podman[170199]: 2024-09-21 14:12:56.096608294 +0000 UTC m=+0.016185326 image pull b415816db8b41f544cdfdc71d193ff8b1ee008bf7b0ec4f64da8eb8f71b62e85 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5 2024-09-21T14:12:56.945 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:56 smithi187 podman[170199]: 2024-09-21 14:12:56.454821374 +0000 UTC m=+0.374398399 container init 9fb6c74c588eb04a4a6adfba4ccc52371836550d4539a3944c548d079a953198 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-21-0842-4417849, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-09-21T14:12:56.945 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:56 smithi187 podman[170199]: 2024-09-21 14:12:56.459890804 +0000 UTC m=+0.379467822 container start 9fb6c74c588eb04a4a6adfba4ccc52371836550d4539a3944c548d079a953198 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6-deactivate, RELEASE=wip-guits-main-2024-09-21-0842-4417849, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True) 2024-09-21T14:12:56.945 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:56 smithi187 podman[170199]: 2024-09-21 14:12:56.546565238 +0000 UTC m=+0.466142265 container attach 9fb6c74c588eb04a4a6adfba4ccc52371836550d4539a3944c548d079a953198 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6-deactivate, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-09-21T14:12:56.945 INFO:journalctl@ceph.osd.6.smithi187.stdout:Sep 21 14:12:56 smithi187 podman[170199]: 2024-09-21 14:12:56.647433201 +0000 UTC m=+0.567010221 container died 9fb6c74c588eb04a4a6adfba4ccc52371836550d4539a3944c548d079a953198 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-6-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-21T14:12:57.249 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:56 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:56.982+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:05.454830+0000 front 2024-09-21T14:12:05.454963+0000 (oldest deadline 2024-09-21T14:12:28.355054+0000) 2024-09-21T14:12:57.249 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:56 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:56.982+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-09-21T14:12:10.655909+0000 front 2024-09-21T14:12:10.656015+0000 (oldest deadline 2024-09-21T14:12:36.555950+0000) 2024-09-21T14:12:57.249 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:56 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:56.982+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-09-21T14:12:17.656866+0000 front 2024-09-21T14:12:17.656877+0000 (oldest deadline 2024-09-21T14:12:42.956816+0000) 2024-09-21T14:12:57.249 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:56 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:56.982+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-09-21T14:12:28.658748+0000 front 2024-09-21T14:12:28.658762+0000 (oldest deadline 2024-09-21T14:12:50.958669+0000) 2024-09-21T14:12:57.263 DEBUG:teuthology.orchestra.run.smithi187:> sudo pkill -f 'journalctl -f -n 0 -u ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@osd.6.service' 2024-09-21T14:12:57.342 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T14:12:57.342 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-21T14:12:57.342 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-21T14:12:57.343 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@osd.7 2024-09-21T14:12:57.663 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:57 smithi187 systemd[1]: Stopping Ceph osd.7 for 40a3d2e8-781f-11ef-baf6-efdc52797490... 2024-09-21T14:12:57.944 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:57 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:57.663+0000 7f54889d2640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T14:12:57.944 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:57 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:57.663+0000 7f54889d2640 -1 osd.7 582 *** Got signal Terminated *** 2024-09-21T14:12:57.944 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:57 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:57.663+0000 7f54889d2640 -1 osd.7 582 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T14:12:58.444 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:57 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:57.971+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:05.454830+0000 front 2024-09-21T14:12:05.454963+0000 (oldest deadline 2024-09-21T14:12:28.355054+0000) 2024-09-21T14:12:58.444 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:57 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:57.971+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-09-21T14:12:10.655909+0000 front 2024-09-21T14:12:10.656015+0000 (oldest deadline 2024-09-21T14:12:36.555950+0000) 2024-09-21T14:12:58.444 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:57 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:57.971+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-09-21T14:12:17.656866+0000 front 2024-09-21T14:12:17.656877+0000 (oldest deadline 2024-09-21T14:12:42.956816+0000) 2024-09-21T14:12:58.445 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:57 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:57.971+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-09-21T14:12:28.658748+0000 front 2024-09-21T14:12:28.658762+0000 (oldest deadline 2024-09-21T14:12:50.958669+0000) 2024-09-21T14:12:59.444 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:58 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:58.993+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:05.454830+0000 front 2024-09-21T14:12:05.454963+0000 (oldest deadline 2024-09-21T14:12:28.355054+0000) 2024-09-21T14:12:59.444 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:58 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:58.993+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-09-21T14:12:10.655909+0000 front 2024-09-21T14:12:10.656015+0000 (oldest deadline 2024-09-21T14:12:36.555950+0000) 2024-09-21T14:12:59.444 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:58 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:58.993+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-09-21T14:12:17.656866+0000 front 2024-09-21T14:12:17.656877+0000 (oldest deadline 2024-09-21T14:12:42.956816+0000) 2024-09-21T14:12:59.444 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:58 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:58.993+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-09-21T14:12:28.658748+0000 front 2024-09-21T14:12:28.658762+0000 (oldest deadline 2024-09-21T14:12:50.958669+0000) 2024-09-21T14:13:00.444 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:59 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:59.975+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:05.454830+0000 front 2024-09-21T14:12:05.454963+0000 (oldest deadline 2024-09-21T14:12:28.355054+0000) 2024-09-21T14:13:00.444 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:59 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:59.975+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-09-21T14:12:10.655909+0000 front 2024-09-21T14:12:10.656015+0000 (oldest deadline 2024-09-21T14:12:36.555950+0000) 2024-09-21T14:13:00.444 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:59 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:59.975+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-09-21T14:12:17.656866+0000 front 2024-09-21T14:12:17.656877+0000 (oldest deadline 2024-09-21T14:12:42.956816+0000) 2024-09-21T14:13:00.444 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:59 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:59.975+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-09-21T14:12:28.658748+0000 front 2024-09-21T14:12:28.658762+0000 (oldest deadline 2024-09-21T14:12:50.958669+0000) 2024-09-21T14:13:00.444 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:12:59 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:12:59.975+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.187:6804 osd.4 since back 2024-09-21T14:12:36.859427+0000 front 2024-09-21T14:12:36.859481+0000 (oldest deadline 2024-09-21T14:12:59.159412+0000) 2024-09-21T14:13:01.444 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:13:00 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:13:00.964+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:05.454830+0000 front 2024-09-21T14:12:05.454963+0000 (oldest deadline 2024-09-21T14:12:28.355054+0000) 2024-09-21T14:13:01.444 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:13:00 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:13:00.964+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-09-21T14:12:10.655909+0000 front 2024-09-21T14:12:10.656015+0000 (oldest deadline 2024-09-21T14:12:36.555950+0000) 2024-09-21T14:13:01.444 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:13:00 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:13:00.964+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-09-21T14:12:17.656866+0000 front 2024-09-21T14:12:17.656877+0000 (oldest deadline 2024-09-21T14:12:42.956816+0000) 2024-09-21T14:13:01.444 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:13:00 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:13:00.964+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-09-21T14:12:28.658748+0000 front 2024-09-21T14:12:28.658762+0000 (oldest deadline 2024-09-21T14:12:50.958669+0000) 2024-09-21T14:13:01.445 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:13:00 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:13:00.964+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.187:6804 osd.4 since back 2024-09-21T14:12:36.859427+0000 front 2024-09-21T14:12:36.859481+0000 (oldest deadline 2024-09-21T14:12:59.159412+0000) 2024-09-21T14:13:02.444 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:13:01 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:13:01.982+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-09-21T14:12:05.454830+0000 front 2024-09-21T14:12:05.454963+0000 (oldest deadline 2024-09-21T14:12:28.355054+0000) 2024-09-21T14:13:02.444 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:13:01 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:13:01.982+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6814 osd.1 since back 2024-09-21T14:12:10.655909+0000 front 2024-09-21T14:12:10.656015+0000 (oldest deadline 2024-09-21T14:12:36.555950+0000) 2024-09-21T14:13:02.445 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:13:01 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:13:01.982+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6822 osd.2 since back 2024-09-21T14:12:17.656866+0000 front 2024-09-21T14:12:17.656877+0000 (oldest deadline 2024-09-21T14:12:42.956816+0000) 2024-09-21T14:13:02.445 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:13:01 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:13:01.982+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.49:6830 osd.3 since back 2024-09-21T14:12:28.658748+0000 front 2024-09-21T14:12:28.658762+0000 (oldest deadline 2024-09-21T14:12:50.958669+0000) 2024-09-21T14:13:02.445 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:13:01 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7[112971]: 2024-09-21T14:13:01.982+0000 7f5484fe7640 -1 osd.7 582 heartbeat_check: no reply from 172.21.15.187:6804 osd.4 since back 2024-09-21T14:12:36.859427+0000 front 2024-09-21T14:12:36.859481+0000 (oldest deadline 2024-09-21T14:12:59.159412+0000) 2024-09-21T14:13:03.436 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:13:03 smithi187 podman[170326]: 2024-09-21 14:13:03.172227251 +0000 UTC m=+5.611892875 container died c1d12429d1b8305b6384fa8fe1242c1cf4de491ffc089af38043c0077038ac86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7, RELEASE=wip-guits-main-2024-09-21-0842-4417849, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True) 2024-09-21T14:13:03.694 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:13:03 smithi187 podman[170326]: 2024-09-21 14:13:03.510769066 +0000 UTC m=+5.950434687 container cleanup c1d12429d1b8305b6384fa8fe1242c1cf4de491ffc089af38043c0077038ac86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-guits-main-2024-09-21-0842-4417849, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T14:13:03.694 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:13:03 smithi187 bash[170326]: ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7 2024-09-21T14:13:04.134 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:13:03 smithi187 podman[170348]: 2024-09-21 14:13:03.737111017 +0000 UTC m=+0.558556546 container remove c1d12429d1b8305b6384fa8fe1242c1cf4de491ffc089af38043c0077038ac86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux ) 2024-09-21T14:13:04.444 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:13:04 smithi187 podman[170444]: 2024-09-21 14:13:04.037274766 +0000 UTC m=+0.016043370 image pull b415816db8b41f544cdfdc71d193ff8b1ee008bf7b0ec4f64da8eb8f71b62e85 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5 2024-09-21T14:13:04.444 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:13:04 smithi187 podman[170444]: 2024-09-21 14:13:04.1461105 +0000 UTC m=+0.124879148 container create f9874785c3223548ffc4d0edc5ee4aee643f75a1b950266a4586cfd808ba26eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-main-2024-09-21-0842-4417849, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-21T14:13:04.444 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:13:04 smithi187 podman[170444]: 2024-09-21 14:13:04.365457643 +0000 UTC m=+0.344226242 container init f9874785c3223548ffc4d0edc5ee4aee643f75a1b950266a4586cfd808ba26eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-21T14:13:04.444 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:13:04 smithi187 podman[170444]: 2024-09-21 14:13:04.37154397 +0000 UTC m=+0.350312560 container start f9874785c3223548ffc4d0edc5ee4aee643f75a1b950266a4586cfd808ba26eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-21T14:13:04.859 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:13:04 smithi187 podman[170444]: 2024-09-21 14:13:04.459979178 +0000 UTC m=+0.438747771 container attach f9874785c3223548ffc4d0edc5ee4aee643f75a1b950266a4586cfd808ba26eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, RELEASE=wip-guits-main-2024-09-21-0842-4417849, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-09-21T14:13:04.859 INFO:journalctl@ceph.osd.7.smithi187.stdout:Sep 21 14:13:04 smithi187 podman[170444]: 2024-09-21 14:13:04.564083788 +0000 UTC m=+0.542852389 container died f9874785c3223548ffc4d0edc5ee4aee643f75a1b950266a4586cfd808ba26eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-osd-7-deactivate, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-09-21T14:13:05.145 DEBUG:teuthology.orchestra.run.smithi187:> sudo pkill -f 'journalctl -f -n 0 -u ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@osd.7.service' 2024-09-21T14:13:05.205 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T14:13:05.205 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-21T14:13:05.206 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-09-21T14:13:05.206 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@rgw.foo.a 2024-09-21T14:13:05.504 INFO:journalctl@ceph.rgw.foo.a.smithi049.stdout:Sep 21 14:13:05 smithi049 systemd[1]: Stopping Ceph rgw.foo.a for 40a3d2e8-781f-11ef-baf6-efdc52797490... 2024-09-21T14:13:06.004 INFO:journalctl@ceph.rgw.foo.a.smithi049.stdout:Sep 21 14:13:05 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-rgw-foo-a[66367]: 2024-09-21T14:13:05.524+0000 7fe1dc520640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T14:13:06.004 INFO:journalctl@ceph.rgw.foo.a.smithi049.stdout:Sep 21 14:13:05 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-rgw-foo-a[66367]: 2024-09-21T14:13:05.524+0000 7fe1dfd8c880 -1 shutting down 2024-09-21T14:13:15.823 INFO:journalctl@ceph.rgw.foo.a.smithi049.stdout:Sep 21 14:13:15 smithi049 bash[239216]: time="2024-09-21T14:13:15Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-09-21T14:13:15.823 INFO:journalctl@ceph.rgw.foo.a.smithi049.stdout:Sep 21 14:13:15 smithi049 podman[239216]: 2024-09-21 14:13:15.553621283 +0000 UTC m=+10.148610709 container died 0a792214d2a97ae87eb69b0a046de7ca87bfc08396f1dd1e54383930fe649673 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-rgw-foo-a, org.label-schema.build-date=20240731, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-21T14:13:16.254 INFO:journalctl@ceph.rgw.foo.a.smithi049.stdout:Sep 21 14:13:16 smithi049 podman[239216]: 2024-09-21 14:13:16.022153281 +0000 UTC m=+10.617142695 container cleanup 0a792214d2a97ae87eb69b0a046de7ca87bfc08396f1dd1e54383930fe649673 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-rgw-foo-a, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T14:13:16.254 INFO:journalctl@ceph.rgw.foo.a.smithi049.stdout:Sep 21 14:13:16 smithi049 bash[239216]: ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-rgw-foo-a 2024-09-21T14:13:16.442 DEBUG:teuthology.orchestra.run.smithi049:> sudo pkill -f 'journalctl -f -n 0 -u ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@rgw.foo.a.service' 2024-09-21T14:13:16.478 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T14:13:16.478 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-09-21T14:13:16.478 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-09-21T14:13:16.479 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@prometheus.a 2024-09-21T14:13:16.804 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 14:13:16 smithi187 systemd[1]: Stopping Ceph prometheus.a for 40a3d2e8-781f-11ef-baf6-efdc52797490... 2024-09-21T14:13:17.095 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 14:13:16 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[71382]: ts=2024-09-21T14:13:16.804Z caller=main.go:964 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-21T14:13:17.095 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 14:13:16 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[71382]: ts=2024-09-21T14:13:16.804Z caller=main.go:988 level=info msg="Stopping scrape discovery manager..." 2024-09-21T14:13:17.095 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 14:13:16 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[71382]: ts=2024-09-21T14:13:16.804Z caller=main.go:1002 level=info msg="Stopping notify discovery manager..." 2024-09-21T14:13:17.095 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 14:13:16 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[71382]: ts=2024-09-21T14:13:16.804Z caller=manager.go:177 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-21T14:13:17.095 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 14:13:16 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[71382]: ts=2024-09-21T14:13:16.804Z caller=main.go:984 level=info msg="Scrape discovery manager stopped" 2024-09-21T14:13:17.095 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 14:13:16 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[71382]: ts=2024-09-21T14:13:16.804Z caller=main.go:998 level=info msg="Notify discovery manager stopped" 2024-09-21T14:13:17.095 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 14:13:16 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[71382]: ts=2024-09-21T14:13:16.804Z caller=manager.go:187 level=info component="rule manager" msg="Rule manager stopped" 2024-09-21T14:13:17.095 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 14:13:16 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[71382]: ts=2024-09-21T14:13:16.804Z caller=main.go:1039 level=info msg="Stopping scrape manager..." 2024-09-21T14:13:17.095 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 14:13:16 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[71382]: ts=2024-09-21T14:13:16.805Z caller=main.go:1031 level=info msg="Scrape manager stopped" 2024-09-21T14:13:17.096 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 14:13:16 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[71382]: ts=2024-09-21T14:13:16.806Z caller=notifier.go:618 level=info component=notifier msg="Stopping notification manager..." 2024-09-21T14:13:17.096 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 14:13:16 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[71382]: ts=2024-09-21T14:13:16.806Z caller=main.go:1261 level=info msg="Notifier manager stopped" 2024-09-21T14:13:17.096 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 14:13:16 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a[71382]: ts=2024-09-21T14:13:16.806Z caller=main.go:1273 level=info msg="See you next time!" 2024-09-21T14:13:17.096 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 14:13:16 smithi187 podman[170572]: 2024-09-21 14:13:16.81592302 +0000 UTC m=+0.113062809 container died fb308ba7f84fd9687f72f2d2ce6f0d19af7167503647515356f5b8627f4798d9 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T14:13:17.436 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 14:13:17 smithi187 podman[170572]: 2024-09-21 14:13:17.194449838 +0000 UTC m=+0.491589620 container cleanup fb308ba7f84fd9687f72f2d2ce6f0d19af7167503647515356f5b8627f4798d9 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T14:13:17.436 INFO:journalctl@ceph.prometheus.a.smithi187.stdout:Sep 21 14:13:17 smithi187 bash[170572]: ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-prometheus-a 2024-09-21T14:13:17.561 DEBUG:teuthology.orchestra.run.smithi187:> sudo pkill -f 'journalctl -f -n 0 -u ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@prometheus.a.service' 2024-09-21T14:13:17.596 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T14:13:17.596 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-09-21T14:13:17.596 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 --force --keep-logs 2024-09-21T14:13:17.810 INFO:teuthology.orchestra.run.smithi049.stdout:Deleting cluster with fsid: 40a3d2e8-781f-11ef-baf6-efdc52797490 2024-09-21T14:13:19.721 INFO:journalctl@ceph.alertmanager.a.smithi049.stdout:Sep 21 14:13:19 smithi049 systemd[1]: Stopping Ceph alertmanager.a for 40a3d2e8-781f-11ef-baf6-efdc52797490... 2024-09-21T14:13:20.004 INFO:journalctl@ceph.alertmanager.a.smithi049.stdout:Sep 21 14:13:19 smithi049 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-alertmanager-a[83730]: ts=2024-09-21T14:13:19.721Z caller=main.go:542 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-21T14:13:20.004 INFO:journalctl@ceph.alertmanager.a.smithi049.stdout:Sep 21 14:13:19 smithi049 podman[239721]: 2024-09-21 14:13:19.762596191 +0000 UTC m=+0.148841253 container died 4adfef62b5cbc923096c89843940f55d787405b3944d75c91b1a3462382d1e7f (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T14:13:20.387 INFO:journalctl@ceph.alertmanager.a.smithi049.stdout:Sep 21 14:13:20 smithi049 podman[239721]: 2024-09-21 14:13:20.195178747 +0000 UTC m=+0.581423841 container cleanup 4adfef62b5cbc923096c89843940f55d787405b3944d75c91b1a3462382d1e7f (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T14:13:20.387 INFO:journalctl@ceph.alertmanager.a.smithi049.stdout:Sep 21 14:13:20 smithi049 bash[239721]: ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-alertmanager-a 2024-09-21T14:13:20.754 INFO:journalctl@ceph.alertmanager.a.smithi049.stdout:Sep 21 14:13:20 smithi049 podman[239738]: 2024-09-21 14:13:20.38810274 +0000 UTC m=+0.618839185 container remove 4adfef62b5cbc923096c89843940f55d787405b3944d75c91b1a3462382d1e7f (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T14:13:20.754 INFO:journalctl@ceph.alertmanager.a.smithi049.stdout:Sep 21 14:13:20 smithi049 podman[239738]: 2024-09-21 14:13:20.463820024 +0000 UTC m=+0.694556470 volume remove 4deb6f5be0a9fd7049fd917ef1002640663e30d11df69dbe0274465f50711c52 2024-09-21T14:13:21.228 INFO:journalctl@ceph.alertmanager.a.smithi049.stdout:Sep 21 14:13:20 smithi049 systemd[1]: ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@alertmanager.a.service: Deactivated successfully. 2024-09-21T14:13:21.229 INFO:journalctl@ceph.alertmanager.a.smithi049.stdout:Sep 21 14:13:20 smithi049 systemd[1]: Stopped Ceph alertmanager.a for 40a3d2e8-781f-11ef-baf6-efdc52797490. 2024-09-21T14:13:21.229 INFO:journalctl@ceph.alertmanager.a.smithi049.stdout:Sep 21 14:13:20 smithi049 systemd[1]: ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@alertmanager.a.service: Consumed 1.826s CPU time. 2024-09-21T14:13:21.229 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 14:13:20 smithi049 systemd[1]: Stopping Ceph node-exporter.a for 40a3d2e8-781f-11ef-baf6-efdc52797490... 2024-09-21T14:13:21.504 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 14:13:21 smithi049 podman[239858]: 2024-09-21 14:13:21.229341816 +0000 UTC m=+0.107883250 container died 556e964b68ca8f011a5e0cb8305e166ab59cd8d53c0710b605a382512d907fad (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T14:13:21.858 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 14:13:21 smithi049 podman[239858]: 2024-09-21 14:13:21.599970183 +0000 UTC m=+0.478511618 container cleanup 556e964b68ca8f011a5e0cb8305e166ab59cd8d53c0710b605a382512d907fad (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T14:13:21.859 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 14:13:21 smithi049 bash[239858]: ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a 2024-09-21T14:13:22.172 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 14:13:21 smithi049 podman[239876]: 2024-09-21 14:13:21.859522828 +0000 UTC m=+0.630352053 container remove 556e964b68ca8f011a5e0cb8305e166ab59cd8d53c0710b605a382512d907fad (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T14:13:22.172 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 14:13:21 smithi049 systemd[1]: ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-09-21T14:13:22.173 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 14:13:21 smithi049 systemd[1]: ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@node-exporter.a.service: Failed with result 'exit-code'. 2024-09-21T14:13:22.173 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 14:13:21 smithi049 systemd[1]: Stopped Ceph node-exporter.a for 40a3d2e8-781f-11ef-baf6-efdc52797490. 2024-09-21T14:13:22.173 INFO:journalctl@ceph.node-exporter.a.smithi049.stdout:Sep 21 14:13:21 smithi049 systemd[1]: ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@node-exporter.a.service: Consumed 5.478s CPU time. 2024-09-21T14:13:22.681 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 --force --keep-logs 2024-09-21T14:13:22.883 INFO:teuthology.orchestra.run.smithi187.stdout:Deleting cluster with fsid: 40a3d2e8-781f-11ef-baf6-efdc52797490 2024-09-21T14:13:24.444 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:13:24 smithi187 systemd[1]: Stopping Ceph iscsi.iscsi.a for 40a3d2e8-781f-11ef-baf6-efdc52797490... 2024-09-21T14:13:24.444 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:13:24 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a[60801]: debug Shutdown received 2024-09-21T14:13:34.694 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:13:34 smithi187 bash[170992]: time="2024-09-21T14:13:34Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-09-21T14:13:34.694 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:13:34 smithi187 podman[170992]: 2024-09-21 14:13:34.424662961 +0000 UTC m=+10.123004524 container died 660ba721898d9859f179ca37d531124ce7ccfacc12af9c252f61bcf28aa2a2b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T14:13:35.047 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:13:34 smithi187 podman[170992]: 2024-09-21 14:13:34.838686283 +0000 UTC m=+10.537027844 container cleanup 660ba721898d9859f179ca37d531124ce7ccfacc12af9c252f61bcf28aa2a2b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-guits-main-2024-09-21-0842-4417849, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T14:13:35.047 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:13:34 smithi187 bash[170992]: ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a 2024-09-21T14:13:35.300 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:13:35 smithi187 podman[171017]: 2024-09-21 14:13:35.048185634 +0000 UTC m=+0.622554254 container remove 660ba721898d9859f179ca37d531124ce7ccfacc12af9c252f61bcf28aa2a2b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-iscsi-iscsi-a, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-guits-main-2024-09-21-0842-4417849, GIT_BRANCH=HEAD) 2024-09-21T14:13:35.300 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:13:35 smithi187 systemd[1]: ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-09-21T14:13:35.300 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:13:35 smithi187 systemd[1]: ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-09-21T14:13:35.300 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:13:35 smithi187 systemd[1]: Stopped Ceph iscsi.iscsi.a for 40a3d2e8-781f-11ef-baf6-efdc52797490. 2024-09-21T14:13:35.300 INFO:journalctl@ceph.iscsi.iscsi.a.smithi187.stdout:Sep 21 14:13:35 smithi187 systemd[1]: ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@iscsi.iscsi.a.service: Consumed 2.807s CPU time. 2024-09-21T14:13:36.933 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 14:13:36 smithi187 systemd[1]: Stopping Ceph grafana.a for 40a3d2e8-781f-11ef-baf6-efdc52797490... 2024-09-21T14:13:37.194 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 14:13:36 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=server t=2024-09-21T14:13:36.932626975Z level=info msg="Shutdown started" reason="System signal: terminated" 2024-09-21T14:13:37.195 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 14:13:36 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=tracing t=2024-09-21T14:13:36.933030948Z level=info msg="Closing tracing" 2024-09-21T14:13:37.195 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 14:13:36 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=ticker t=2024-09-21T14:13:36.933075978Z level=info msg=stopped last_tick=2024-09-21T14:13:30Z 2024-09-21T14:13:37.195 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 14:13:36 smithi187 ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a[63882]: logger=grafana-apiserver t=2024-09-21T14:13:36.933099712Z level=info msg="StorageObjectCountTracker pruner is exiting" 2024-09-21T14:13:37.195 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 14:13:36 smithi187 podman[171291]: 2024-09-21 14:13:36.943210916 +0000 UTC m=+0.111664324 container died bef1e288687e5c07030d19e57743fa0825b3890b53a3d8874b18d99c050f0be8 (image=quay.io/ceph/grafana:10.4.8, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a, maintainer=Grafana Labs ) 2024-09-21T14:13:37.513 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 14:13:37 smithi187 podman[171291]: 2024-09-21 14:13:37.288494225 +0000 UTC m=+0.456947634 container cleanup bef1e288687e5c07030d19e57743fa0825b3890b53a3d8874b18d99c050f0be8 (image=quay.io/ceph/grafana:10.4.8, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a, maintainer=Grafana Labs ) 2024-09-21T14:13:37.514 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 14:13:37 smithi187 bash[171291]: ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a 2024-09-21T14:13:37.780 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 14:13:37 smithi187 podman[171308]: 2024-09-21 14:13:37.514635511 +0000 UTC m=+0.565022363 container remove bef1e288687e5c07030d19e57743fa0825b3890b53a3d8874b18d99c050f0be8 (image=quay.io/ceph/grafana:10.4.8, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-grafana-a, maintainer=Grafana Labs ) 2024-09-21T14:13:37.781 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 14:13:37 smithi187 systemd[1]: ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@grafana.a.service: Deactivated successfully. 2024-09-21T14:13:37.781 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 14:13:37 smithi187 systemd[1]: Stopped Ceph grafana.a for 40a3d2e8-781f-11ef-baf6-efdc52797490. 2024-09-21T14:13:37.781 INFO:journalctl@ceph.grafana.a.smithi187.stdout:Sep 21 14:13:37 smithi187 systemd[1]: ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@grafana.a.service: Consumed 16.342s CPU time. 2024-09-21T14:13:38.194 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 14:13:37 smithi187 systemd[1]: Stopping Ceph node-exporter.b for 40a3d2e8-781f-11ef-baf6-efdc52797490... 2024-09-21T14:13:38.467 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 14:13:38 smithi187 podman[171429]: 2024-09-21 14:13:38.213302412 +0000 UTC m=+0.112341238 container died 5520efe30bc09115edf596ac2923e898d5bcac34c846319d1ede36be5169ff23 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T14:13:38.935 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 14:13:38 smithi187 podman[171429]: 2024-09-21 14:13:38.542279227 +0000 UTC m=+0.441318040 container cleanup 5520efe30bc09115edf596ac2923e898d5bcac34c846319d1ede36be5169ff23 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T14:13:38.935 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 14:13:38 smithi187 bash[171429]: ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b 2024-09-21T14:13:38.935 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 14:13:38 smithi187 podman[171449]: 2024-09-21 14:13:38.701674542 +0000 UTC m=+0.492422098 container remove 5520efe30bc09115edf596ac2923e898d5bcac34c846319d1ede36be5169ff23 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-40a3d2e8-781f-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T14:13:38.935 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 14:13:38 smithi187 systemd[1]: ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-09-21T14:13:39.420 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 14:13:38 smithi187 systemd[1]: ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@node-exporter.b.service: Failed with result 'exit-code'. 2024-09-21T14:13:39.420 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 14:13:38 smithi187 systemd[1]: Stopped Ceph node-exporter.b for 40a3d2e8-781f-11ef-baf6-efdc52797490. 2024-09-21T14:13:39.420 INFO:journalctl@ceph.node-exporter.b.smithi187.stdout:Sep 21 14:13:38 smithi187 systemd[1]: ceph-40a3d2e8-781f-11ef-baf6-efdc52797490@node-exporter.b.service: Consumed 4.764s CPU time. 2024-09-21T14:13:39.743 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T14:13:39.777 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T14:13:39.804 INFO:tasks.cephadm:Archiving crash dumps... 2024-09-21T14:13:39.806 DEBUG:teuthology.misc:Transferring archived files from smithi049:/var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/crash to /home/teuthworker/archive/gabrioux-2024-09-21_11:23:40-orch:cephadm-wip-guits-main-2024-09-21-0842-distro-default-smithi/7914033/remote/smithi049/crash 2024-09-21T14:13:39.806 DEBUG:teuthology.orchestra.run.smithi049:> sudo tar c -f - -C /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/crash -- . 2024-09-21T14:13:39.861 INFO:teuthology.orchestra.run.smithi049.stderr:tar: /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/crash: Cannot open: No such file or directory 2024-09-21T14:13:39.862 INFO:teuthology.orchestra.run.smithi049.stderr:tar: Error is not recoverable: exiting now 2024-09-21T14:13:39.864 DEBUG:teuthology.misc:Transferring archived files from smithi187:/var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/crash to /home/teuthworker/archive/gabrioux-2024-09-21_11:23:40-orch:cephadm-wip-guits-main-2024-09-21-0842-distro-default-smithi/7914033/remote/smithi187/crash 2024-09-21T14:13:39.865 DEBUG:teuthology.orchestra.run.smithi187:> sudo tar c -f - -C /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/crash -- . 2024-09-21T14:13:39.902 INFO:teuthology.orchestra.run.smithi187.stderr:tar: /var/lib/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/crash: Cannot open: No such file or directory 2024-09-21T14:13:39.902 INFO:teuthology.orchestra.run.smithi187.stderr:tar: Error is not recoverable: exiting now 2024-09-21T14:13:39.904 INFO:tasks.cephadm:Checking cluster log for badness... 2024-09-21T14:13:39.904 DEBUG:teuthology.orchestra.run.smithi049:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v CEPHADM_STRAY_DAEMON | egrep -v CEPHADM_FAILED_DAEMON | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-09-21T14:13:39.978 INFO:tasks.cephadm:Compressing logs... 2024-09-21T14:13:39.978 DEBUG:teuthology.orchestra.run.smithi049:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-21T14:13:39.980 DEBUG:teuthology.orchestra.run.smithi187:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-21T14:13:40.016 INFO:teuthology.orchestra.run.smithi187.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/tcmu-runner.log 2024-09-21T14:13:40.017 INFO:teuthology.orchestra.run.smithi187.stderr:: No such file or directory 2024-09-21T14:13:40.017 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph-osd.7.log 2024-09-21T14:13:40.017 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/tcmu-runner.log: gzip -5 --verbose -- /var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph-volume.log 2024-09-21T14:13:40.017 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph-osd.4.log 2024-09-21T14:13:40.017 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph.audit.log 2024-09-21T14:13:40.017 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph.cephadm.log 2024-09-21T14:13:40.018 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph-osd.6.log 2024-09-21T14:13:40.018 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph-osd.5.log 2024-09-21T14:13:40.018 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph.log 2024-09-21T14:13:40.018 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph-mgr.x.log 2024-09-21T14:13:40.018 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph.log: gzip -5 --verbose -- /var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph-mon.b.log 2024-09-21T14:13:40.018 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/objectstore_tool.7.log 2024-09-21T14:13:40.018 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-21T14:13:40.019 INFO:teuthology.orchestra.run.smithi049.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph-osd.0.log 2024-09-21T14:13:40.019 INFO:teuthology.orchestra.run.smithi049.stderr:: No such file or directory 2024-09-21T14:13:40.019 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph-osd.1.log 2024-09-21T14:13:40.020 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph-volume.log 2024-09-21T14:13:40.020 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph.audit.log 2024-09-21T14:13:40.022 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph-mon.c.log 2024-09-21T14:13:40.023 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph.cephadm.log 2024-09-21T14:13:40.023 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph-mon.a.log 2024-09-21T14:13:40.023 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph-mon.c.log: /var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/objectstore_tool.6.log 2024-09-21T14:13:40.023 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph-osd.2.log 2024-09-21T14:13:40.023 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/objectstore_tool.6.log: gzip -5 --verbose -- /var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph-mgr.y.log 2024-09-21T14:13:40.023 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph.log 2024-09-21T14:13:40.023 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph-client.rgw.foo.a.log 2024-09-21T14:13:40.023 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph.log: gzip -5 --verbose -- /var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph-osd.3.log 2024-09-21T14:13:40.023 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph-client.rgw.foo.a.log: gzip -5 --verbose -- /var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/objectstore_tool.7.log 2024-09-21T14:13:40.023 INFO:teuthology.orchestra.run.smithi049.stderr: 58.2% -- replaced with /var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph-client.rgw.foo.a.log.gz 2024-09-21T14:13:40.023 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-21T14:13:40.033 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/objectstore_tool.7.log: /var/log/ceph/cephadm.log: 91.9% -- replaced with /var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph.log.gz 2024-09-21T14:13:40.061 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/objectstore_tool.7.log: /var/log/ceph/cephadm.log: 63.5% -- replaced with /var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/tcmu-runner.log.gz 2024-09-21T14:13:40.071 INFO:teuthology.orchestra.run.smithi187.stderr: 79.7% -- replaced with /var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph.cephadm.log.gz 2024-09-21T14:13:40.122 INFO:teuthology.orchestra.run.smithi187.stderr: 91.9% -- replaced with /var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph-mgr.x.log.gz 2024-09-21T14:13:40.146 INFO:teuthology.orchestra.run.smithi049.stderr: 91.5% -- replaced with /var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph-volume.log.gz 2024-09-21T14:13:40.157 INFO:teuthology.orchestra.run.smithi049.stderr: 89.2% -- replaced with /var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph.cephadm.log.gz 2024-09-21T14:13:40.163 INFO:teuthology.orchestra.run.smithi049.stderr: 95.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-21T14:13:40.237 INFO:teuthology.orchestra.run.smithi187.stderr: 89.9% -- replaced with /var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph.audit.log.gz 2024-09-21T14:13:40.268 INFO:teuthology.orchestra.run.smithi187.stderr: 94.5% -- replaced with /var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/objectstore_tool.7.log.gz 2024-09-21T14:13:40.295 INFO:teuthology.orchestra.run.smithi187.stderr: 91.5% -- replaced with /var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph-volume.log.gz 2024-09-21T14:13:40.334 INFO:teuthology.orchestra.run.smithi187.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-21T14:13:40.361 INFO:teuthology.orchestra.run.smithi187.stderr: 85.7% -- replaced with /var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph.log.gz 2024-09-21T14:13:40.420 INFO:teuthology.orchestra.run.smithi049.stderr: 94.4% -- replaced with /var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/objectstore_tool.7.log.gz 2024-09-21T14:13:40.486 INFO:teuthology.orchestra.run.smithi049.stderr: 94.0% -- replaced with /var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/objectstore_tool.6.log.gz 2024-09-21T14:13:40.536 INFO:teuthology.orchestra.run.smithi049.stderr: 93.9% -- replaced with /var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph.audit.log.gz 2024-09-21T14:13:41.827 INFO:teuthology.orchestra.run.smithi049.stderr: 90.0% -- replaced with /var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph-mgr.y.log.gz 2024-09-21T14:13:43.850 INFO:teuthology.orchestra.run.smithi049.stderr: 90.2% -- replaced with /var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph-mon.a.log.gz 2024-09-21T14:13:44.969 INFO:teuthology.orchestra.run.smithi187.stderr: 91.0% -- replaced with /var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph-mon.b.log.gz 2024-09-21T14:13:51.791 INFO:teuthology.orchestra.run.smithi049.stderr: 91.1% -- replaced with /var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph-mon.c.log.gz 2024-09-21T14:18:52.661 INFO:teuthology.orchestra.run.smithi187.stderr: 92.9% -- replaced with /var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph-osd.4.log.gz 2024-09-21T14:20:44.370 INFO:teuthology.orchestra.run.smithi049.stderr: 94.1% -- replaced with /var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph-osd.2.log.gz 2024-09-21T14:21:24.298 INFO:teuthology.orchestra.run.smithi187.stderr: 92.9% -- replaced with /var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph-osd.6.log.gz 2024-09-21T14:22:45.278 INFO:teuthology.orchestra.run.smithi049.stderr: 92.6% -- replaced with /var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph-osd.0.log.gz 2024-09-21T14:23:22.278 INFO:teuthology.orchestra.run.smithi187.stderr: 93.4% -- replaced with /var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph-osd.7.log.gz 2024-09-21T14:24:08.654 INFO:teuthology.orchestra.run.smithi187.stderr: 92.6% -- replaced with /var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph-osd.5.log.gz 2024-09-21T14:24:08.656 INFO:teuthology.orchestra.run.smithi187.stderr: 2024-09-21T14:24:08.657 INFO:teuthology.orchestra.run.smithi187.stderr:real 10m28.662s 2024-09-21T14:24:08.657 INFO:teuthology.orchestra.run.smithi187.stderr:user 6m24.419s 2024-09-21T14:24:08.657 INFO:teuthology.orchestra.run.smithi187.stderr:sys 0m17.808s 2024-09-21T14:25:43.810 INFO:teuthology.orchestra.run.smithi049.stderr: 92.6% -- replaced with /var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph-osd.3.log.gz 2024-09-21T14:26:14.521 INFO:teuthology.orchestra.run.smithi049.stderr: 91.8% -- replaced with /var/log/ceph/40a3d2e8-781f-11ef-baf6-efdc52797490/ceph-osd.1.log.gz 2024-09-21T14:26:14.523 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-09-21T14:26:14.523 INFO:teuthology.orchestra.run.smithi049.stderr:real 12m34.529s 2024-09-21T14:26:14.523 INFO:teuthology.orchestra.run.smithi049.stderr:user 7m21.651s 2024-09-21T14:26:14.523 INFO:teuthology.orchestra.run.smithi049.stderr:sys 0m18.490s 2024-09-21T14:26:14.523 INFO:tasks.cephadm:Archiving logs... 2024-09-21T14:26:14.524 DEBUG:teuthology.misc:Transferring archived files from smithi049:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-09-21_11:23:40-orch:cephadm-wip-guits-main-2024-09-21-0842-distro-default-smithi/7914033/remote/smithi049/log 2024-09-21T14:26:14.525 DEBUG:teuthology.orchestra.run.smithi049:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-21T14:27:19.194 DEBUG:teuthology.misc:Transferring archived files from smithi187:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-09-21_11:23:40-orch:cephadm-wip-guits-main-2024-09-21-0842-distro-default-smithi/7914033/remote/smithi187/log 2024-09-21T14:27:19.195 DEBUG:teuthology.orchestra.run.smithi187:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-21T14:28:17.910 INFO:tasks.cephadm:Removing cluster... 2024-09-21T14:28:17.910 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 --force 2024-09-21T14:28:18.149 INFO:teuthology.orchestra.run.smithi049.stdout:Deleting cluster with fsid: 40a3d2e8-781f-11ef-baf6-efdc52797490 2024-09-21T14:28:18.910 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 40a3d2e8-781f-11ef-baf6-efdc52797490 --force 2024-09-21T14:28:19.183 INFO:teuthology.orchestra.run.smithi187.stdout:Deleting cluster with fsid: 40a3d2e8-781f-11ef-baf6-efdc52797490 2024-09-21T14:28:19.911 INFO:tasks.cephadm:Removing cephadm ... 2024-09-21T14:28:19.911 DEBUG:teuthology.orchestra.run.smithi049:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-21T14:28:19.931 DEBUG:teuthology.orchestra.run.smithi187:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-21T14:28:19.947 INFO:tasks.cephadm:Teardown complete 2024-09-21T14:28:19.947 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-21T14:28:19.955 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-21T14:28:19.956 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-21T14:28:19.974 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-21T14:28:20.064 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-21T14:28:20.073 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-21T14:28:20.073 DEBUG:teuthology.orchestra.run.smithi049:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T14:28:20.075 DEBUG:teuthology.orchestra.run.smithi187:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T14:28:20.090 INFO:teuthology.orchestra.run.smithi187.stderr:bash: line 1: ntpq: command not found 2024-09-21T14:28:20.090 INFO:teuthology.orchestra.run.smithi049.stderr:bash: line 1: ntpq: command not found 2024-09-21T14:28:20.102 INFO:teuthology.orchestra.run.smithi049.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T14:28:20.102 INFO:teuthology.orchestra.run.smithi049.stdout:=============================================================================== 2024-09-21T14:28:20.102 INFO:teuthology.orchestra.run.smithi049.stdout:^* hv01.front.sepia.ceph.com 3 6 377 28 +801us[ +817us] +/- 37ms 2024-09-21T14:28:20.102 INFO:teuthology.orchestra.run.smithi049.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 23 -3232us[-3232us] +/- 50ms 2024-09-21T14:28:20.102 INFO:teuthology.orchestra.run.smithi049.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 30 +1543us[+1559us] +/- 64ms 2024-09-21T14:28:20.102 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T14:28:20.105 INFO:teuthology.orchestra.run.smithi187.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T14:28:20.105 INFO:teuthology.orchestra.run.smithi187.stdout:=============================================================================== 2024-09-21T14:28:20.105 INFO:teuthology.orchestra.run.smithi187.stdout:^* hv01.front.sepia.ceph.com 3 6 377 25 +777us[ +829us] +/- 37ms 2024-09-21T14:28:20.106 INFO:teuthology.orchestra.run.smithi187.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 19 -3250us[-3250us] +/- 50ms 2024-09-21T14:28:20.106 INFO:teuthology.orchestra.run.smithi187.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 21 +1524us[+1524us] +/- 64ms 2024-09-21T14:28:20.106 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T14:28:20.106 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-21T14:28:20.115 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-21T14:28:20.115 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-21T14:28:20.156 DEBUG:teuthology.orchestra.run.smithi049:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-21T14:28:20.267 DEBUG:teuthology.orchestra.run.smithi187:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-21T14:28:20.357 DEBUG:teuthology.orchestra.run.smithi049:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-21T14:28:20.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:28:20.388 DEBUG:teuthology.orchestra.run.smithi187:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-21T14:28:20.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:28:20.415 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-21T14:28:20.427 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-21T14:28:20.440 INFO:teuthology.task.internal:Duration was 3165.198893 seconds 2024-09-21T14:28:20.440 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-21T14:28:20.448 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-21T14:28:20.448 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-21T14:28:20.451 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-21T14:28:20.515 INFO:teuthology.orchestra.run.smithi049.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T14:28:20.518 INFO:teuthology.orchestra.run.smithi187.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T14:28:20.810 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-21T14:28:20.810 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi049.front.sepia.ceph.com 2024-09-21T14:28:20.810 DEBUG:teuthology.orchestra.run.smithi049:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-21T14:28:20.834 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi187.front.sepia.ceph.com 2024-09-21T14:28:20.834 DEBUG:teuthology.orchestra.run.smithi187:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-21T14:28:20.865 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-21T14:28:20.866 DEBUG:teuthology.orchestra.run.smithi049:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-21T14:28:20.877 DEBUG:teuthology.orchestra.run.smithi187:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-21T14:28:20.931 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-21T14:28:20.932 DEBUG:teuthology.orchestra.run.smithi049:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-21T14:28:20.944 DEBUG:teuthology.orchestra.run.smithi187:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-21T14:28:23.289 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-21T14:28:23.301 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-21T14:28:23.301 DEBUG:teuthology.orchestra.run.smithi049:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-21T14:28:23.327 DEBUG:teuthology.orchestra.run.smithi187:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-21T14:28:23.387 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-21T14:28:23.397 DEBUG:teuthology.orchestra.run.smithi049:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-21T14:28:23.399 DEBUG:teuthology.orchestra.run.smithi187:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-21T14:28:23.421 INFO:teuthology.orchestra.run.smithi049.stdout:kernel.core_pattern = core 2024-09-21T14:28:23.432 INFO:teuthology.orchestra.run.smithi187.stdout:kernel.core_pattern = core 2024-09-21T14:28:23.466 DEBUG:teuthology.orchestra.run.smithi049:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-21T14:28:23.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:28:23.522 DEBUG:teuthology.orchestra.run.smithi187:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-21T14:28:23.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T14:28:23.538 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-21T14:28:23.548 INFO:teuthology.task.internal:Transferring archived files... 2024-09-21T14:28:23.548 DEBUG:teuthology.misc:Transferring archived files from smithi049:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-09-21_11:23:40-orch:cephadm-wip-guits-main-2024-09-21-0842-distro-default-smithi/7914033/remote/smithi049 2024-09-21T14:28:23.549 DEBUG:teuthology.orchestra.run.smithi049:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-21T14:28:23.634 DEBUG:teuthology.misc:Transferring archived files from smithi187:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-09-21_11:23:40-orch:cephadm-wip-guits-main-2024-09-21-0842-distro-default-smithi/7914033/remote/smithi187 2024-09-21T14:28:23.634 DEBUG:teuthology.orchestra.run.smithi187:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-21T14:28:23.677 INFO:teuthology.task.internal:Removing archive directory... 2024-09-21T14:28:23.677 DEBUG:teuthology.orchestra.run.smithi049:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-21T14:28:23.679 DEBUG:teuthology.orchestra.run.smithi187:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-21T14:28:23.725 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-21T14:28:23.743 INFO:teuthology.task.internal:Not uploading archives. 2024-09-21T14:28:23.744 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-21T14:28:23.760 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-21T14:28:23.760 DEBUG:teuthology.orchestra.run.smithi049:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-21T14:28:23.768 DEBUG:teuthology.orchestra.run.smithi187:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-21T14:28:23.781 INFO:teuthology.orchestra.run.smithi049.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 21 14:28 /home/ubuntu/cephtest 2024-09-21T14:28:23.796 INFO:teuthology.orchestra.run.smithi187.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 21 14:28 /home/ubuntu/cephtest 2024-09-21T14:28:23.797 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-21T14:28:23.820 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-21T14:28:23.864 INFO:teuthology.run:Summary data: description: orch:cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root} duration: 3165.1988933086395 flavor: default owner: scheduled_gabrioux@teuthology success: true 2024-09-21T14:28:23.865 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-21T14:28:23.964 INFO:teuthology.run:pass